--- /dev/null
+What: /sys/bus/i2c/.../idle_state
+Date: January 2019
+KernelVersion: 5.2
+Contact: Robert Shearman <robert.shearman@att.com>
+Description:
+ Value that exists only for mux devices that can be
+ written to control the behaviour of the multiplexer on
+ idle. Possible values:
+ -2 - disconnect on idle, i.e. deselect the last used
+ channel, which is useful when there is a device
+ with an address that conflicts with another
+ device on another mux on the same parent bus.
+ -1 - leave the mux as-is, which is the most optimal
+ setting in terms of I2C operations and is the
+ default mode.
+ 0..<nchans> - set the mux to a predetermined channel,
+ which is useful if there is one channel that is
+ used almost always, and you want to reduce the
+ latency for normal operations after rare
+ transactions on other channels
driver needs regions sized smaller than a page, you may prefer using
the dma_pool interface, described below.
-The consistent DMA mapping interfaces, for non-NULL dev, will by
-default return a DMA address which is 32-bit addressable. Even if the
-device indicates (via DMA mask) that it may address the upper 32-bits,
-consistent allocation will only return > 32-bit addresses for DMA if
-the consistent DMA mask has been explicitly changed via
-dma_set_coherent_mask(). This is true of the dma_pool interface as
-well.
+The consistent DMA mapping interfaces, will by default return a DMA address
+which is 32-bit addressable. Even if the device indicates (via the DMA mask)
+that it may address the upper 32-bits, consistent allocation will only
+return > 32-bit addresses for DMA if the consistent DMA mask has been
+explicitly changed via dma_set_coherent_mask(). This is true of the
+dma_pool interface as well.
dma_alloc_coherent() returns two values: the virtual address which you
can use to access it from the CPU and dma_handle which you pass to the
populated
1 if the cgroup or its descendants contains any live
processes; otherwise, 0.
+ frozen
+ 1 if the cgroup is frozen; otherwise, 0.
cgroup.max.descendants
A read-write single value files. The default is "max".
A dying cgroup can consume system resources not exceeding
limits, which were active at the moment of cgroup deletion.
+ cgroup.freeze
+ A read-write single value file which exists on non-root cgroups.
+ Allowed values are "0" and "1". The default is "0".
+
+ Writing "1" to the file causes freezing of the cgroup and all
+ descendant cgroups. This means that all belonging processes will
+ be stopped and will not run until the cgroup will be explicitly
+ unfrozen. Freezing of the cgroup may take some time; when this action
+ is completed, the "frozen" value in the cgroup.events control file
+ will be updated to "1" and the corresponding notification will be
+ issued.
+
+ A cgroup can be frozen either by its own settings, or by settings
+ of any ancestor cgroups. If any of ancestor cgroups is frozen, the
+ cgroup will remain frozen.
+
+ Processes in the frozen cgroup can be killed by a fatal signal.
+ They also can enter and leave a frozen cgroup: either by an explicit
+ move by a user, or if freezing of the cgroup races with fork().
+ If a process is moved to a frozen cgroup, it stops. If a process is
+ moved out of a frozen cgroup, it becomes running.
+
+ Frozen status of a cgroup doesn't affect any cgroup tree operations:
+ it's possible to delete a frozen (and empty) cgroup, as well as
+ create new sub-cgroups.
Controllers
===========
Format: { "off" | "enforce" | "fix" | "log" }
default: "enforce"
- ima_appraise_tcb [IMA]
+ ima_appraise_tcb [IMA] Deprecated. Use ima_policy= instead.
The builtin appraise policy appraises all files
owned by uid=0.
uid=0.
The "appraise_tcb" policy appraises the integrity of
- all files owned by root. (This is the equivalent
- of ima_appraise_tcb.)
+ all files owned by root.
The "secure_boot" policy appraises the integrity
of files (eg. kexec kernel image, kernel modules,
- "mediatek,mt7629-apmixedsys"
- "mediatek,mt8135-apmixedsys"
- "mediatek,mt8173-apmixedsys"
+ - "mediatek,mt8183-apmixedsys", "syscon"
+ - "mediatek,mt8516-apmixedsys"
- #clock-cells: Must be 1
The apmixedsys controller uses the common clk binding from
- "mediatek,mt2701-audsys", "syscon"
- "mediatek,mt7622-audsys", "syscon"
- "mediatek,mt7623-audsys", "mediatek,mt2701-audsys", "syscon"
+ - "mediatek,mt8183-audiosys", "syscon"
- #clock-cells: Must be 1
The AUDSYS controller uses the common clk binding from
--- /dev/null
+MediaTek CAMSYS controller
+============================
+
+The MediaTek camsys controller provides various clocks to the system.
+
+Required Properties:
+
+- compatible: Should be one of:
+ - "mediatek,mt8183-camsys", "syscon"
+- #clock-cells: Must be 1
+
+The camsys controller uses the common clk binding from
+Documentation/devicetree/bindings/clock/clock-bindings.txt
+The available clocks are defined in dt-bindings/clock/mt*-clk.h.
+
+Example:
+
+camsys: camsys@1a000000 {
+ compatible = "mediatek,mt8183-camsys", "syscon";
+ reg = <0 0x1a000000 0 0x1000>;
+ #clock-cells = <1>;
+};
- "mediatek,mt6797-imgsys", "syscon"
- "mediatek,mt7623-imgsys", "mediatek,mt2701-imgsys", "syscon"
- "mediatek,mt8173-imgsys", "syscon"
+ - "mediatek,mt8183-imgsys", "syscon"
- #clock-cells: Must be 1
The imgsys controller uses the common clk binding from
- "mediatek,mt7629-infracfg", "syscon"
- "mediatek,mt8135-infracfg", "syscon"
- "mediatek,mt8173-infracfg", "syscon"
+ - "mediatek,mt8183-infracfg", "syscon"
+ - "mediatek,mt8516-infracfg", "syscon"
- #clock-cells: Must be 1
- #reset-cells: Must be 1
--- /dev/null
+Mediatek IPU controller
+============================
+
+The Mediatek ipu controller provides various clocks to the system.
+
+Required Properties:
+
+- compatible: Should be one of:
+ - "mediatek,mt8183-ipu_conn", "syscon"
+ - "mediatek,mt8183-ipu_adl", "syscon"
+ - "mediatek,mt8183-ipu_core0", "syscon"
+ - "mediatek,mt8183-ipu_core1", "syscon"
+- #clock-cells: Must be 1
+
+The ipu controller uses the common clk binding from
+Documentation/devicetree/bindings/clock/clock-bindings.txt
+The available clocks are defined in dt-bindings/clock/mt*-clk.h.
+
+Example:
+
+ipu_conn: syscon@19000000 {
+ compatible = "mediatek,mt8183-ipu_conn", "syscon";
+ reg = <0 0x19000000 0 0x1000>;
+ #clock-cells = <1>;
+};
+
+ipu_adl: syscon@19010000 {
+ compatible = "mediatek,mt8183-ipu_adl", "syscon";
+ reg = <0 0x19010000 0 0x1000>;
+ #clock-cells = <1>;
+};
+
+ipu_core0: syscon@19180000 {
+ compatible = "mediatek,mt8183-ipu_core0", "syscon";
+ reg = <0 0x19180000 0 0x1000>;
+ #clock-cells = <1>;
+};
+
+ipu_core1: syscon@19280000 {
+ compatible = "mediatek,mt8183-ipu_core1", "syscon";
+ reg = <0 0x19280000 0 0x1000>;
+ #clock-cells = <1>;
+};
- compatible: Should be one of:
- "mediatek,mt2712-mcucfg", "syscon"
+ - "mediatek,mt8183-mcucfg", "syscon"
- #clock-cells: Must be 1
The mcucfg controller uses the common clk binding from
- compatible: Should be one of:
- "mediatek,mt2712-mfgcfg", "syscon"
+ - "mediatek,mt8183-mfgcfg", "syscon"
- #clock-cells: Must be 1
The mfgcfg controller uses the common clk binding from
- "mediatek,mt6797-mmsys", "syscon"
- "mediatek,mt7623-mmsys", "mediatek,mt2701-mmsys", "syscon"
- "mediatek,mt8173-mmsys", "syscon"
+ - "mediatek,mt8183-mmsys", "syscon"
- #clock-cells: Must be 1
The mmsys controller uses the common clk binding from
- "mediatek,mt7629-topckgen"
- "mediatek,mt8135-topckgen"
- "mediatek,mt8173-topckgen"
+ - "mediatek,mt8183-topckgen", "syscon"
+ - "mediatek,mt8516-topckgen"
- #clock-cells: Must be 1
The topckgen controller uses the common clk binding from
- "mediatek,mt6797-vdecsys", "syscon"
- "mediatek,mt7623-vdecsys", "mediatek,mt2701-vdecsys", "syscon"
- "mediatek,mt8173-vdecsys", "syscon"
+ - "mediatek,mt8183-vdecsys", "syscon"
- #clock-cells: Must be 1
The vdecsys controller uses the common clk binding from
- "mediatek,mt2712-vencsys", "syscon"
- "mediatek,mt6797-vencsys", "syscon"
- "mediatek,mt8173-vencsys", "syscon"
+ - "mediatek,mt8183-vencsys", "syscon"
- #clock-cells: Must be 1
The vencsys controller uses the common clk binding from
Required Properties:
-- compatible : should be "amlogic,axg-audio-clkc" for the A113X and A113D
+- compatible : should be "amlogic,axg-audio-clkc" for the A113X and A113D,
+ "amlogic,g12a-audio-clkc" for G12A.
- reg : physical base address of the clock controller and length of
memory mapped region.
- clocks : a list of phandle + clock-specifier pairs for the clocks listed
Required properties:
- compatible : shall be one of the following:
- "atmel,at91sam9x5-sckc" or
+ "atmel,at91sam9x5-sckc",
+ "atmel,sama5d3-sckc" or
"atmel,sama5d4-sckc":
at91 SCKC (Slow Clock Controller)
- This node contains the slow clock definitions.
-
- "atmel,at91sam9x5-clk-slow-osc":
- at91 slow oscillator
-
- "atmel,at91sam9x5-clk-slow-rc-osc":
- at91 internal slow RC oscillator
-- reg : defines the IO memory reserved for the SCKC.
-- #size-cells : shall be 0 (reg is used to encode clk id).
-- #address-cells : shall be 1 (reg is used to encode clk id).
+- #clock-cells : shall be 0.
+- clocks : shall be the input parent clock phandle for the clock.
+Optional properties:
+- atmel,osc-bypass : boolean property. Set this when a clock signal is directly
+ provided on XIN.
For example:
- sckc: sckc@fffffe50 {
- compatible = "atmel,sama5d3-pmc";
- reg = <0xfffffe50 0x4>
- #size-cells = <0>;
- #address-cells = <1>;
-
- /* put at91 slow clocks here */
+ sckc@fffffe50 {
+ compatible = "atmel,at91sam9x5-sckc";
+ reg = <0xfffffe50 0x4>;
+ clocks = <&slow_xtal>;
+ #clock-cells = <0>;
};
Power Management Controller (PMC):
Required properties:
-- compatible : shall be "atmel,<chip>-pmc", "syscon":
+- compatible : shall be "atmel,<chip>-pmc", "syscon" or
+ "microchip,sam9x60-pmc"
<chip> can be: at91rm9200, at91sam9260, at91sam9261,
at91sam9263, at91sam9g45, at91sam9n12, at91sam9rl, at91sam9g15,
at91sam9g25, at91sam9g35, at91sam9x25, at91sam9x35, at91sam9x5,
--- /dev/null
+Cirrus Logic Lochnagar Audio Development Board
+
+Lochnagar is an evaluation and development board for Cirrus Logic
+Smart CODEC and Amp devices. It allows the connection of most Cirrus
+Logic devices on mini-cards, as well as allowing connection of
+various application processor systems to provide a full evaluation
+platform. Audio system topology, clocking and power can all be
+controlled through the Lochnagar, allowing the device under test
+to be used in a variety of possible use cases.
+
+This binding document describes the binding for the clock portion of
+the driver.
+
+Also see these documents for generic binding information:
+ [1] Clock : ../clock/clock-bindings.txt
+
+And these for relevant defines:
+ [2] include/dt-bindings/clock/lochnagar.h
+
+This binding must be part of the Lochnagar MFD binding:
+ [3] ../mfd/cirrus,lochnagar.txt
+
+Required properties:
+
+ - compatible : One of the following strings:
+ "cirrus,lochnagar1-clk"
+ "cirrus,lochnagar2-clk"
+
+ - #clock-cells : Must be 1. The first cell indicates the clock
+ number, see [2] for available clocks and [1].
+
+Optional properties:
+
+ - clocks : Must contain an entry for each clock in clock-names.
+ - clock-names : May contain entries for each of the following
+ clocks:
+ - ln-cdc-clkout : Output clock from CODEC card.
+ - ln-dsp-clkout : Output clock from DSP card.
+ - ln-gf-mclk1,ln-gf-mclk2,ln-gf-mclk3,ln-gf-mclk4 : Optional
+ input audio clocks from host system.
+ - ln-psia1-mclk, ln-psia2-mclk : Optional input audio clocks from
+ external connector.
+ - ln-spdif-clkout : Optional input audio clock from SPDIF.
+ - ln-adat-mclk : Optional input audio clock from ADAT.
+ - ln-pmic-32k : On board fixed clock.
+ - ln-clk-12m : On board fixed clock.
+ - ln-clk-11m : On board fixed clock.
+ - ln-clk-24m : On board fixed clock.
+ - ln-clk-22m : On board fixed clock.
+ - ln-clk-8m : On board fixed clock.
+ - ln-usb-clk-24m : On board fixed clock.
+ - ln-usb-clk-12m : On board fixed clock.
+
+ - assigned-clocks : A list of Lochnagar clocks to be reparented, see
+ [2] for available clocks.
+ - assigned-clock-parents : Parents to be assigned to the clocks
+ listed in "assigned-clocks".
+
+Optional nodes:
+
+ - fixed-clock nodes may be registered for the following on board clocks:
+ - ln-pmic-32k : 32768 Hz
+ - ln-clk-12m : 12288000 Hz
+ - ln-clk-11m : 11298600 Hz
+ - ln-clk-24m : 24576000 Hz
+ - ln-clk-22m : 22579200 Hz
+ - ln-clk-8m : 8192000 Hz
+ - ln-usb-clk-24m : 24576000 Hz
+ - ln-usb-clk-12m : 12288000 Hz
+
+Example:
+
+lochnagar {
+ lochnagar-clk {
+ compatible = "cirrus,lochnagar2-clk";
+
+ #clock-cells = <1>;
+
+ clocks = <&clk-audio>, <&clk_pmic>;
+ clock-names = "ln-gf-mclk2", "ln-pmic-32k";
+
+ assigned-clocks = <&lochnagar-clk LOCHNAGAR_CDC_MCLK1>,
+ <&lochnagar-clk LOCHNAGAR_CDC_MCLK2>;
+ assigned-clock-parents = <&clk-audio>,
+ <&clk-pmic>;
+ };
+
+ clk-pmic: clk-pmic {
+ compatible = "fixed-clock";
+ clock-cells = <0>;
+ clock-frequency = <32768>;
+ };
+};
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/bindings/clock/milbeaut-clock.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Milbeaut SoCs Clock Controller Binding
+
+maintainers:
+ - Taichi Sugaya <sugaya.taichi@socionext.com>
+
+description: |
+ Milbeaut SoCs Clock controller is an integrated clock controller, which
+ generates and supplies to all modules.
+
+ This binding uses common clock bindings
+ [1] Documentation/devicetree/bindings/clock/clock-bindings.txt
+
+properties:
+ compatible:
+ oneOf:
+ - items:
+ - enum:
+ - socionext,milbeaut-m10v-ccu
+ clocks:
+ maxItems: 1
+ description: external clock
+
+ '#clock-cells':
+ const: 1
+
+required:
+ - compatible
+ - reg
+ - clocks
+ - '#clock-cells'
+
+examples:
+ # Clock controller node:
+ - |
+ m10v-clk-ctrl@1d021000 {
+ compatible = "socionext,milbeaut-m10v-clk-ccu";
+ reg = <0x1d021000 0x4000>;
+ #clock-cells = <1>;
+ clocks = <&clki40mhz>;
+ };
+
+ # Required an external clock for Clock controller node:
+ - |
+ clocks {
+ clki40mhz: clki40mhz {
+ compatible = "fixed-clock";
+ #clock-cells = <0>;
+ clock-frequency = <40000000>;
+ };
+ /* other clocks */
+ };
+
+ # The clock consumer shall specify the desired clock-output of the clock
+ # controller as below by specifying output-id in its "clk" phandle cell.
+ # 2: uart
+ # 4: 32-bit timer
+ # 7: UHS-I/II
+ - |
+ serial@1e700010 {
+ compatible = "socionext,milbeaut-usio-uart";
+ reg = <0x1e700010 0x10>;
+ interrupts = <0 141 0x4>, <0 149 0x4>;
+ interrupt-names = "rx", "tx";
+ clocks = <&clk 2>;
+ };
+
+...
--- /dev/null
+Qualcomm Turing Clock & Reset Controller Binding
+------------------------------------------------
+
+Required properties :
+- compatible: shall contain "qcom,qcs404-turingcc".
+- reg: shall contain base register location and length.
+- clocks: ahb clock for the TuringCC
+- #clock-cells: from common clock binding, shall contain 1.
+- #reset-cells: from common reset binding, shall contain 1.
+
+Example:
+ turingcc: clock-controller@800000 {
+ compatible = "qcom,qcs404-turingcc";
+ reg = <0x00800000 0x30000>;
+ clocks = <&gcc GCC_CDSP_CFG_AHB_CLK>;
+
+ #clock-cells = <1>;
+ #reset-cells = <1>;
+ };
* "fsl,b4860-clockgen"
* "fsl,ls1012a-clockgen"
* "fsl,ls1021a-clockgen"
+ * "fsl,ls1028a-clockgen"
* "fsl,ls1043a-clockgen"
* "fsl,ls1046a-clockgen"
* "fsl,ls1088a-clockgen"
1 cmux index (n in CLKCnCSR)
2 hwaccel index (n in CLKCGnHWACSR)
3 fman 0 for fm1, 1 for fm2
- 4 platform pll 0=pll, 1=pll/2, 2=pll/3, 3=pll/4
- 4=pll/5, 5=pll/6, 6=pll/7, 7=pll/8
+ 4 platform pll n=pll/(n+1). For example, when n=1,
+ that means output_freq=PLL_freq/2.
5 coreclk must be 0
3. Example
--- /dev/null
+SiFive FU540 PRCI bindings
+
+On the FU540 family of SoCs, most system-wide clock and reset integration
+is via the PRCI IP block.
+
+Required properties:
+- compatible: Should be "sifive,<chip>-prci". Only one value is
+ supported: "sifive,fu540-c000-prci"
+- reg: Should describe the PRCI's register target physical address region
+- clocks: Should point to the hfclk device tree node and the rtcclk
+ device tree node. The RTC clock here is not a time-of-day clock,
+ but is instead a high-stability clock source for system timers
+ and cycle counters.
+- #clock-cells: Should be <1>
+
+The clock consumer should specify the desired clock via the clock ID
+macros defined in include/dt-bindings/clock/sifive-fu540-prci.h.
+These macros begin with PRCI_CLK_.
+
+The hfclk and rtcclk nodes are required, and represent physical
+crystals or resonators located on the PCB. These nodes should be present
+underneath /, rather than /soc.
+
+Examples:
+
+/* under /, in PCB-specific DT data */
+hfclk: hfclk {
+ #clock-cells = <0>;
+ compatible = "fixed-clock";
+ clock-frequency = <33333333>;
+ clock-output-names = "hfclk";
+};
+rtcclk: rtcclk {
+ #clock-cells = <0>;
+ compatible = "fixed-clock";
+ clock-frequency = <1000000>;
+ clock-output-names = "rtcclk";
+};
+
+/* under /soc, in SoC-specific DT data */
+prci: clock-controller@10000000 {
+ compatible = "sifive,fu540-c000-prci";
+ reg = <0x0 0x10000000 0x0 0x1000>;
+ clocks = <&hfclk>, <&rtcclk>;
+ #clock-cells = <1>;
+};
"st,stm32f42xx-rcc"
"st,stm32f469-rcc"
"st,stm32f746-rcc"
+ "st,stm32f769-rcc"
+
- reg: should be register base and length as documented in the
datasheet
- #reset-cells: 1, see below
28 CLK_I2C3
29 CLK_I2C4
30 CLK_LPTIMER (LPTimer1 clock)
+ 31 CLK_PLL_SRC
+ 32 CLK_DFSDM1
+ 33 CLK_ADFSDM1
+ 34 CLK_F769_DSI
)
Example:
- GXL (S905X, S905D) : "amlogic,meson-gxl-dw-hdmi"
- GXM (S912) : "amlogic,meson-gxm-dw-hdmi"
followed by the common "amlogic,meson-gx-dw-hdmi"
+ - G12A (S905X2, S905Y2, S905D2) : "amlogic,meson-g12a-dw-hdmi"
- reg: Physical base address and length of the controller's registers.
- interrupts: The HDMI interrupt number
- clocks, clock-names : must have the phandles to the HDMI iahb and isfr clocks,
S905X (GXL) VENC Input TMDS Output
S905D (GXL) VENC Input TMDS Output
S912 (GXM) VENC Input TMDS Output
+ S905X2 (G12A) VENC Input TMDS Output
+ S905Y2 (G12A) VENC Input TMDS Output
+ S905D2 (G12A) VENC Input TMDS Output
Example:
- GXL (S905X, S905D) : "amlogic,meson-gxl-vpu"
- GXM (S912) : "amlogic,meson-gxm-vpu"
followed by the common "amlogic,meson-gx-vpu"
+ - G12A (S905X2, S905Y2, S905D2) : "amlogic,meson-g12a-vpu"
- reg: base address and size of he following memory-mapped regions :
- vpu
- hhi
- - dmc
- reg-names: should contain the names of the previous memory regions
- interrupts: should contain the VENC Vsync interrupt number
+- amlogic,canvas: phandle to canvas provider node as described in the file
+ ../soc/amlogic/amlogic,canvas.txt
Optional properties:
- power-domains: Optional phandle to associated power domain as described in
the file ../power/power_domain.txt
-- amlogic,canvas: phandle to canvas provider node as described in the file
- ../soc/amlogic/amlogic,canvas.txt
Required nodes:
S905X (GXL) CVBS VDAC HDMI-TX
S905D (GXL) CVBS VDAC HDMI-TX
S912 (GXM) CVBS VDAC HDMI-TX
+ S905X2 (G12A) CVBS VDAC HDMI-TX
+ S905Y2 (G12A) CVBS VDAC HDMI-TX
+ S905D2 (G12A) CVBS VDAC HDMI-TX
Example:
Optional properties:
- powerdown-gpios: power-down gpio
-- reg: I2C address. If and only if present the device node
- should be placed into the i2c controller node where the
- tfp410 i2c is connected to.
+- reg: I2C address. If and only if present the device node should be placed
+ into the I2C controller node where the TFP410 I2C is connected to.
+- ti,deskew: data de-skew in 350ps increments, from -4 to +3, as configured
+ through th DK[3:1] pins. This property shall be present only if the TFP410
+ is not connected through I2C.
Required nodes:
-- Video port 0 for DPI input [1].
-- Video port 1 for DVI output [1].
-[1]: Documentation/devicetree/bindings/media/video-interfaces.txt
+This device has two video ports. Their connections are modeled using the OF
+graph bindings specified in [1]. Each port node shall have a single endpoint.
+
+- Port 0 is the DPI input port. Its endpoint subnode shall contain a
+ pclk-sample and bus-width property and a remote-endpoint property as specified
+ in [1].
+ - If pclk-sample is not defined, pclk-sample = 0 should be assumed for
+ backward compatibility.
+ - If bus-width is not defined then bus-width = 24 should be assumed for
+ backward compatibility.
+ bus-width = 24: 24 data lines are connected and single-edge mode
+ bus-width = 12: 12 data lines are connected and dual-edge mode
+
+- Port 1 is the DVI output port. Its endpoint subnode shall contain a
+ remote-endpoint property is specified in [1].
+
+[1] Documentation/devicetree/bindings/media/video-interfaces.txt
+
Example
-------
tfp410: encoder@0 {
compatible = "ti,tfp410";
powerdown-gpios = <&twl_gpio 2 GPIO_ACTIVE_LOW>;
+ ti,deskew = <4>;
ports {
#address-cells = <1>;
reg = <0>;
tfp410_in: endpoint@0 {
+ pclk-sample = <1>;
+ bus-width = <24>;
remote-endpoint = <&dpi_out>;
};
};
* "cxo"
* "axi"
* "mnoc"
-- power-domains: should be <&clock_gpucc GPU_CX_GDSC>
+- power-domains: should be:
+ <&clock_gpucc GPU_CX_GDSC>
+ <&clock_gpucc GPU_GX_GDSC>
+- power-domain-names: Matching names for the power domains
- iommus: phandle to the adreno iommu
- operating-points-v2: phandle to the OPP operating points
<&gcc GCC_GPU_MEMNOC_GFX_CLK>;
clock-names = "gmu", "cxo", "axi", "memnoc";
- power-domains = <&gpucc GPU_CX_GDSC>;
+ power-domains = <&gpucc GPU_CX_GDSC>,
+ <&gpucc GPU_GX_GDSC>;
+ power-domain-names = "cx", "gx";
+
iommus = <&adreno_smmu 5>;
operating-points-v2 = <&gmu_opp_table>;
- qcom,adreno-630.2
- iommus: optional phandle to an adreno iommu instance
- operating-points-v2: optional phandle to the OPP operating points
+- interconnects: optional phandle to an interconnect provider. See
+ ../interconnect/interconnect.txt for details.
- qcom,gmu: For GMU attached devices a phandle to the GMU device that will
control the power for the GPU. Applicable targets:
- qcom,adreno-630.2
+- zap-shader: For a5xx and a6xx devices this node contains a memory-region that
+ points to reserved memory to store the zap shader that can be used to help
+ bring the GPU out of secure mode.
Example 3xx/4xx/a5xx:
operating-points-v2 = <&gpu_opp_table>;
+ interconnects = <&rsc_hlos MASTER_GFX3D &rsc_hlos SLAVE_EBI1>;
+
qcom,gmu = <&gmu>;
+
+ zap-shader {
+ memory-region = <&zap_shader_region>;
+ };
};
};
--- /dev/null
+Feiyang FY07024DI26A30-D 7" MIPI-DSI LCD Panel
+
+Required properties:
+- compatible: must be "feiyang,fy07024di26a30d"
+- reg: DSI virtual channel used by that screen
+- avdd-supply: analog regulator dc1 switch
+- dvdd-supply: 3v3 digital regulator
+- reset-gpios: a GPIO phandle for the reset pin
+
+Optional properties:
+- backlight: phandle for the backlight control.
+
+panel@0 {
+ compatible = "feiyang,fy07024di26a30d";
+ reg = <0>;
+ avdd-supply = <®_dc1sw>;
+ dvdd-supply = <®_dldo2>;
+ reset-gpios = <&pio 3 24 GPIO_ACTIVE_HIGH>; /* LCD-RST: PD24 */
+ backlight = <&backlight>;
+};
Example:
&mipi_dsi {
- panel {
+ panel@0 {
compatible = "innolux,p079zca";
reg = <0>;
power-supply = <...>;
Example:
&mipi_dsi {
- panel {
+ panel@0 {
compatible = "innolux,p079zca";
reg = <0>;
avdd-supply = <...>;
Example:
&mipi_dsi {
- panel {
+ panel@0 {
compatible = "kingdisplay,kd097d04";
reg = <0>;
power-supply = <...>;
--- /dev/null
+LG ACX467AKM-7 4.95" 1080×1920 LCD Panel
+
+Required properties:
+- compatible: must be "lg,acx467akm-7"
+
+This binding is compatible with the simple-panel binding, which is specified
+in simple-panel.txt in this directory.
--- /dev/null
+OSD Displays OSD070T1718-19TS 7" WVGA TFT LCD panel
+
+Required properties:
+- compatible: shall be "osddisplays,osd070t1718-19ts"
+- power-supply: see simple-panel.txt
+
+Optional properties:
+- backlight: see simple-panel.txt
+
+This binding is compatible with the simple-panel binding, which is specified
+in simple-panel.txt in this directory. No other simple-panel properties than
+the ones specified herein are valid.
--- /dev/null
+Rocktech jh057n00900 5.5" 720x1440 TFT LCD panel
+
+Required properties:
+- compatible: should be "rocktech,jh057n00900"
+- reg: DSI virtual channel of the peripheral
+- reset-gpios: panel reset gpio
+- backlight: phandle of the backlight device attached to the panel
+
+Example:
+
+ &mipi_dsi {
+ panel@0 {
+ compatible = "rocktech,jh057n00900";
+ reg = <0>;
+ backlight = <&backlight>;
+ reset-gpios = <&gpio3 13 GPIO_ACTIVE_LOW>;
+ };
+ };
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0+ OR X11)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/display/panel/ronbo,rb070d30.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Ronbo RB070D30 DSI Display Panel
+
+maintainers:
+ - Maxime Ripard <maxime.ripard@bootlin.com>
+
+properties:
+ compatible:
+ const: ronbo,rb070d30
+
+ reg:
+ description: MIPI-DSI virtual channel
+
+ power-gpios:
+ description: GPIO used for the power pin
+ maxItems: 1
+
+ reset-gpios:
+ description: GPIO used for the reset pin
+ maxItems: 1
+
+ shlr-gpios:
+ description: GPIO used for the shlr pin (horizontal flip)
+ maxItems: 1
+
+ updn-gpios:
+ description: GPIO used for the updn pin (vertical flip)
+ maxItems: 1
+
+ vcc-lcd-supply:
+ description: Power regulator
+
+ backlight:
+ description: Backlight used by the panel
+ $ref: "/schemas/types.yaml#/definitions/phandle"
+
+required:
+ - compatible
+ - power-gpios
+ - reg
+ - reset-gpios
+ - shlr-gpios
+ - updn-gpios
+ - vcc-lcd-supply
+
+additionalProperties: false
Optional properties:
- label: a symbolic name for the panel
+- backlight: phandle of the backlight device
Required nodes:
- Video port for DPI input
spi-cpha;
label = "lcd";
+ backlight = <&backlight>;
port {
lcd_in: endpoint {
remote-endpoint = <&dpi_out>;
--- /dev/null
+Rockchip specific extensions for rk3066 HDMI
+============================================
+
+Required properties:
+- compatible:
+ "rockchip,rk3066-hdmi";
+- reg:
+ Physical base address and length of the controller's registers.
+- clocks, clock-names:
+ Phandle to HDMI controller clock, name should be "hclk".
+- interrupts:
+ HDMI interrupt number.
+- power-domains:
+ Phandle to the RK3066_PD_VIO power domain.
+- rockchip,grf:
+ This soc uses GRF regs to switch the HDMI TX input between vop0 and vop1.
+- ports:
+ Contains one port node with two endpoints, numbered 0 and 1,
+ connected respectively to vop0 and vop1.
+ Contains one port node with one endpoint
+ connected to a hdmi-connector node.
+- pinctrl-0, pinctrl-name:
+ Switch the iomux for the HPD/I2C pins to HDMI function.
+
+Example:
+ hdmi: hdmi@10116000 {
+ compatible = "rockchip,rk3066-hdmi";
+ reg = <0x10116000 0x2000>;
+ interrupts = <GIC_SPI 64 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&cru HCLK_HDMI>;
+ clock-names = "hclk";
+ power-domains = <&power RK3066_PD_VIO>;
+ rockchip,grf = <&grf>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&hdmii2c_xfer>, <&hdmi_hpd>;
+
+ ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ hdmi_in: port@0 {
+ reg = <0>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ hdmi_in_vop0: endpoint@0 {
+ reg = <0>;
+ remote-endpoint = <&vop0_out_hdmi>;
+ };
+ hdmi_in_vop1: endpoint@1 {
+ reg = <1>;
+ remote-endpoint = <&vop1_out_hdmi>;
+ };
+ };
+ hdmi_out: port@1 {
+ reg = <1>;
+ hdmi_out_con: endpoint {
+ remote-endpoint = <&hdmi_con_in>;
+ };
+ };
+ };
+ };
+
+&pinctrl {
+ hdmi {
+ hdmi_hpd: hdmi-hpd {
+ rockchip,pins = <0 RK_PA0 1 &pcfg_pull_default>;
+ };
+ hdmii2c_xfer: hdmii2c-xfer {
+ rockchip,pins = <0 RK_PA1 1 &pcfg_pull_none>,
+ <0 RK_PA2 1 &pcfg_pull_none>;
+ };
+ };
+};
--- /dev/null
+ST-Ericsson Multi Channel Display Engine MCDE
+
+The ST-Ericsson MCDE is a display controller with support for compositing
+and displaying several channels memory resident graphics data on DSI or
+LCD displays or bridges. It is used in the ST-Ericsson U8500 platform.
+
+Required properties:
+
+- compatible: must be:
+ "ste,mcde"
+- reg: register base for the main MCDE control registers, should be
+ 0x1000 in size
+- interrupts: the interrupt line for the MCDE
+- epod-supply: a phandle to the EPOD regulator
+- vana-supply: a phandle to the analog voltage regulator
+- clocks: an array of the MCDE clocks in this strict order:
+ MCDECLK (main MCDE clock), LCDCLK (LCD clock), PLLDSI
+ (HDMI clock), DSI0ESCLK (DSI0 energy save clock),
+ DSI1ESCLK (DSI1 energy save clock), DSI2ESCLK (DSI2 energy
+ save clock)
+- clock-names: must be the following array:
+ "mcde", "lcd", "hdmi"
+ to match the required clock inputs above.
+- #address-cells: should be <1> (for the DSI hosts that will be children)
+- #size-cells: should be <1> (for the DSI hosts that will be children)
+- ranges: this should always be stated
+
+Required subnodes:
+
+The devicetree must specify subnodes for the DSI host adapters.
+These must have the following characteristics:
+
+- compatible: must be:
+ "ste,mcde-dsi"
+- reg: must specify the register range for the DSI host
+- vana-supply: phandle to the VANA voltage regulator
+- clocks: phandles to the high speed and low power (energy save) clocks
+ the high speed clock is not present on the third (dsi2) block, so it
+ should only have the "lp" clock
+- clock-names: "hs" for the high speed clock and "lp" for the low power
+ (energy save) clock
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+
+Display panels and bridges will appear as children on the DSI hosts, and
+the displays are connected to the DSI hosts using the common binding
+for video transmitter interfaces; see
+Documentation/devicetree/bindings/media/video-interfaces.txt
+
+If a DSI host is unused (not connected) it will have no children defined.
+
+Example:
+
+mcde@a0350000 {
+ compatible = "ste,mcde";
+ reg = <0xa0350000 0x1000>;
+ interrupts = <GIC_SPI 48 IRQ_TYPE_LEVEL_HIGH>;
+ epod-supply = <&db8500_b2r2_mcde_reg>;
+ vana-supply = <&ab8500_ldo_ana_reg>;
+ clocks = <&prcmu_clk PRCMU_MCDECLK>, /* Main MCDE clock */
+ <&prcmu_clk PRCMU_LCDCLK>, /* LCD clock */
+ <&prcmu_clk PRCMU_PLLDSI>; /* HDMI clock */
+ clock-names = "mcde", "lcd", "hdmi";
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges;
+
+ dsi0: dsi@a0351000 {
+ compatible = "ste,mcde-dsi";
+ reg = <0xa0351000 0x1000>;
+ vana-supply = <&ab8500_ldo_ana_reg>;
+ clocks = <&prcmu_clk PRCMU_DSI0CLK>, <&prcmu_clk PRCMU_DSI0ESCCLK>;
+ clock-names = "hs", "lp";
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ panel {
+ compatible = "samsung,s6d16d0";
+ reg = <0>;
+ vdd1-supply = <&ab8500_ldo_aux1_reg>;
+ reset-gpios = <&gpio2 1 GPIO_ACTIVE_LOW>;
+ };
+
+ };
+ dsi1: dsi@a0352000 {
+ compatible = "ste,mcde-dsi";
+ reg = <0xa0352000 0x1000>;
+ vana-supply = <&ab8500_ldo_ana_reg>;
+ clocks = <&prcmu_clk PRCMU_DSI1CLK>, <&prcmu_clk PRCMU_DSI1ESCCLK>;
+ clock-names = "hs", "lp";
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
+ dsi2: dsi@a0353000 {
+ compatible = "ste,mcde-dsi";
+ reg = <0xa0353000 0x1000>;
+ vana-supply = <&ab8500_ldo_ana_reg>;
+ /* This DSI port only has the Low Power / Energy Save clock */
+ clocks = <&prcmu_clk PRCMU_DSI2ESCCLK>;
+ clock-names = "lp";
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
+};
Required channel sub-node properties:
- reg: Which channel this node refers to.
- - adi,length-width: Width of the DMA transfer length register.
- adi,source-bus-width,
adi,destination-bus-width: Width of the source or destination bus in bits.
- adi,source-bus-type,
1 (AXI_DMAC_TYPE_AXI_STREAM): Streaming AXI interface
2 (AXI_DMAC_TYPE_AXI_FIFO): FIFO interface
-Optional channel properties:
+Deprecated optional channel properties:
+ - adi,length-width: Width of the DMA transfer length register.
- adi,cyclic: Must be set if the channel supports hardware cyclic DMA
transfers.
- adi,2d: Must be set if the channel supports hardware 2D DMA transfers.
between system memory and the Audio Processing Engine (APE).
Required properties:
-- compatible: Must be "nvidia,tegra210-adma".
+- compatible: Should contain one of the following:
+ - "nvidia,tegra210-adma": for Tegra210
+ - "nvidia,tegra186-adma": for Tegra186 and Tegra194
- reg: Should contain DMA registers location and length. This should be
a single entry that includes all of the per-channel registers in one
contiguous bank.
"nxp,se97b" - the fallback is "atmel,24c02",
"renesas,r1ex24002" - the fallback is "atmel,24c02"
+ "renesas,r1ex24016" - the fallback is "atmel,24c16"
"renesas,r1ex24128" - the fallback is "atmel,24c128"
"rohm,br24t01" - the fallback is "atmel,24c01"
--- /dev/null
+ARM Mali Bifrost GPU
+====================
+
+Required properties:
+
+- compatible :
+ * Since Mali Bifrost GPU model/revision is fully discoverable by reading
+ some determined registers, must contain the following:
+ + "arm,mali-bifrost"
+ * which must be preceded by one of the following vendor specifics:
+ + "amlogic,meson-g12a-mali"
+
+- reg : Physical base address of the device and length of the register area.
+
+- interrupts : Contains the three IRQ lines required by Mali Bifrost devices,
+ in the following defined order.
+
+- interrupt-names : Contains the names of IRQ resources in this exact defined
+ order: "job", "mmu", "gpu".
+
+Optional properties:
+
+- clocks : Phandle to clock for the Mali Bifrost device.
+
+- mali-supply : Phandle to regulator for the Mali device. Refer to
+ Documentation/devicetree/bindings/regulator/regulator.txt for details.
+
+- operating-points-v2 : Refer to Documentation/devicetree/bindings/opp/opp.txt
+ for details.
+
+- resets : Phandle of the GPU reset line.
+
+Vendor-specific bindings
+------------------------
+
+The Mali GPU is integrated very differently from one SoC to
+another. In order to accommodate those differences, you have the option
+to specify one more vendor-specific compatible, among:
+
+- "amlogic,meson-g12a-mali"
+ Required properties:
+ - resets : Should contain phandles of :
+ + GPU reset line
+ + GPU APB glue reset line
+
+Example for a Mali-G31:
+
+gpu@ffa30000 {
+ compatible = "amlogic,meson-g12a-mali", "arm,mali-bifrost";
+ reg = <0xffe40000 0x10000>;
+ interrupts = <GIC_SPI 160 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 161 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 162 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-names = "job", "mmu", "gpu";
+ clocks = <&clk CLKID_MALI>;
+ mali-supply = <&vdd_gpu>;
+ operating-points-v2 = <&gpu_opp_table>;
+ resets = <&reset RESET_DVALIN_CAPB3>, <&reset RESET_DVALIN>;
+};
+
+gpu_opp_table: opp_table0 {
+ compatible = "operating-points-v2";
+
+ opp@533000000 {
+ opp-hz = /bits/ 64 <533000000>;
+ opp-microvolt = <1250000>;
+ };
+ opp@450000000 {
+ opp-hz = /bits/ 64 <450000000>;
+ opp-microvolt = <1150000>;
+ };
+ opp@400000000 {
+ opp-hz = /bits/ 64 <400000000>;
+ opp-microvolt = <1125000>;
+ };
+ opp@350000000 {
+ opp-hz = /bits/ 64 <350000000>;
+ opp-microvolt = <1075000>;
+ };
+ opp@266000000 {
+ opp-hz = /bits/ 64 <266000000>;
+ opp-microvolt = <1025000>;
+ };
+ opp@160000000 {
+ opp-hz = /bits/ 64 <160000000>;
+ opp-microvolt = <925000>;
+ };
+ opp@100000000 {
+ opp-hz = /bits/ 64 <100000000>;
+ opp-microvolt = <912500>;
+ };
+};
--- /dev/null
+Device tree configuration for the GFX display device on the ASPEED SoCs
+
+Required properties:
+ - compatible
+ * Must be one of the following:
+ + aspeed,ast2500-gfx
+ + aspeed,ast2400-gfx
+ * In addition, the ASPEED pinctrl bindings require the 'syscon' property to
+ be present
+
+ - reg: Physical base address and length of the GFX registers
+
+ - interrupts: interrupt number for the GFX device
+
+ - clocks: clock number used to generate the pixel clock
+
+ - resets: reset line that must be released to use the GFX device
+
+ - memory-region:
+ Phandle to a memory region to allocate from, as defined in
+ Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
+
+
+Example:
+
+gfx: display@1e6e6000 {
+ compatible = "aspeed,ast2500-gfx", "syscon";
+ reg = <0x1e6e6000 0x1000>;
+ reg-io-width = <4>;
+ clocks = <&syscon ASPEED_CLK_GATE_D1CLK>;
+ resets = <&syscon ASPEED_RESET_CRT1>;
+ interrupts = <0x19>;
+ memory-region = <&gfx_memory>;
+};
+
+gfx_memory: framebuffer {
+ size = <0x01000000>;
+ alignment = <0x01000000>;
+ compatible = "shared-dma-pool";
+ reusable;
+};
Required properties:
- compatible: Should be "brcm,7268-v3d" or "brcm,7278-v3d"
- reg: Physical base addresses and lengths of the register areas
-- reg-names: Names for the register areas. The "hub", "bridge", and "core0"
+- reg-names: Names for the register areas. The "hub" and "core0"
register areas are always required. The "gca" register area
- is required if the GCA cache controller is present.
+ is required if the GCA cache controller is present. The
+ "bridge" register area is required if an external reset
+ controller is not present.
- interrupts: The interrupt numbers. The first interrupt is for the hub,
- while the following interrupts are for the cores.
+ while the following interrupts are separate interrupt lines
+ for the cores (if they don't share the hub's interrupt).
See bindings/interrupt-controller/interrupts.txt
Optional properties:
- clocks: The core clock the unit runs on
+- resets: The reset line for v3d, if not using a mapping of the bridge
+ See bindings/reset/reset.txt
v3d {
compatible = "brcm,7268-v3d";
Required properties:
- compatible:
- Must be "brcm,iproc-i2c"
+ Must be "brcm,iproc-i2c" or "brcm,iproc-nic-i2c"
- reg:
Define the base and range of the I/O address space that contain the iProc
I2C controller registers
-- interrupts:
- Should contain the I2C interrupt
-
- clock-frequency:
This is the I2C bus clock. Need to be either 100000 or 400000
- #size-cells:
Always 0
+Optional properties:
+
+- interrupts:
+ Should contain the I2C interrupt. For certain revisions of the I2C
+ controller, I2C interrupt is unwired to the interrupt controller. In such
+ case, this property should be left unspecified, and driver will fall back
+ to polling mode
+
+- brcm,ape-hsls-addr-mask:
+ Required for "brcm,iproc-nic-i2c". Host view of address mask into the
+ 'APE' co-processor. Value must be unsigned, 32-bit
+
Example:
i2c0: i2c@18008000 {
compatible = "brcm,iproc-i2c";
or "mscc,ocelot-i2c" with "snps,designware-i2c" for fallback
- reg : Offset and length of the register set for the device
- interrupts : <IRQ> where IRQ is the interrupt number.
+ - clocks : phandles for the clocks, see the description of clock-names below.
+ The phandle for the "ic_clk" clock is required. The phandle for the "pclk"
+ clock is optional. If a single clock is specified but no clock-name, it is
+ the "ic_clk" clock. If both clocks are listed, the "ic_clk" must be first.
Recommended properties :
- clock-frequency : desired I2C bus clock frequency in Hz.
Optional properties :
+
+ - clock-names : Contains the names of the clocks:
+ "ic_clk", for the core clock used to generate the external I2C clock.
+ "pclk", the interface clock, required for register access.
+
- reg : for "mscc,ocelot-i2c", a second register set to configure the SDA hold
time, named ICPU_CFG:TWI_DELAY in the datasheet.
"mediatek,mt7623-i2c", "mediatek,mt6577-i2c": for MediaTek MT7623
"mediatek,mt7629-i2c", "mediatek,mt2712-i2c": for MediaTek MT7629
"mediatek,mt8173-i2c": for MediaTek MT8173
+ "mediatek,mt8183-i2c": for MediaTek MT8183
+ "mediatek,mt8516-i2c", "mediatek,mt2712-i2c": for MediaTek MT8516
- reg: physical base address of the controller and dma base, length of memory
mapped region.
- interrupts: interrupt number to the cpu.
- clock-div: the fixed value for frequency divider of clock source in i2c
module. Each IC may be different.
- clocks: clock name from clock manager
- - clock-names: Must include "main" and "dma", if enable have-pmic need include
+ - clock-names: Must include "main" and "dma", "arb" is for multi-master that
+ one bus has more than two i2c controllers, if enable have-pmic need include
"pmic" extra.
Optional properties:
Device tree configuration for Renesas RIIC driver
Required properties:
-- compatible : "renesas,riic-<soctype>". "renesas,riic-rz" as fallback
+- compatible :
+ "renesas,riic-r7s72100" if the device is a part of a R7S72100 SoC.
+ "renesas,riic-r7s9210" if the device is a part of a R7S9210 SoC.
+ "renesas,riic-rz" for a generic RZ/A compatible device.
- reg : address start and address range size of device
- interrupts : 8 interrupts (TEI, RI, TI, SPI, STI, NAKI, ALI, TMOI)
- clock-frequency : frequency of bus clock in Hz
* I2C controller embedded in STMicroelectronics STM32 I2C platform
-Required properties :
-- compatible : Must be one of the following
+Required properties:
+- compatible: Must be one of the following
- "st,stm32f4-i2c"
- "st,stm32f7-i2c"
-- reg : Offset and length of the register set for the device
-- interrupts : Must contain the interrupt id for I2C event and then the
+- reg: Offset and length of the register set for the device
+- interrupts: Must contain the interrupt id for I2C event and then the
interrupt id for I2C error.
- resets: Must contain the phandle to the reset controller.
- clocks: Must contain the input clock of the I2C instance.
- #address-cells = <1>;
- #size-cells = <0>;
-Optional properties :
-- clock-frequency : Desired I2C bus clock frequency in Hz. If not specified,
+Optional properties:
+- clock-frequency: Desired I2C bus clock frequency in Hz. If not specified,
the default 100 kHz frequency will be used.
For STM32F4 SoC Standard-mode and Fast-mode are supported, possible values are
100000 and 400000.
- For STM32F7 SoC, Standard-mode, Fast-mode and Fast-mode Plus are supported,
- possible values are 100000, 400000 and 1000000.
-- i2c-scl-rising-time-ns : Only for STM32F7, I2C SCL Rising time for the board
- (default: 25)
-- i2c-scl-falling-time-ns : Only for STM32F7, I2C SCL Falling time for the board
- (default: 10)
+ For STM32F7, STM32H7 and STM32MP1 SoCs, Standard-mode, Fast-mode and Fast-mode
+ Plus are supported, possible values are 100000, 400000 and 1000000.
+- i2c-scl-rising-time-ns: I2C SCL Rising time for the board (default: 25)
+ For STM32F7, STM32H7 and STM32MP1 only.
+- i2c-scl-falling-time-ns: I2C SCL Falling time for the board (default: 10)
+ For STM32F7, STM32H7 and STM32MP1 only.
I2C Timings are derived from these 2 values
-- st,syscfg-fmp: Only for STM32F7, use to set Fast Mode Plus bit within SYSCFG
- whether Fast Mode Plus speed is selected by slave.
- 1st cell : phandle to syscfg
- 2nd cell : register offset within SYSCFG
- 3rd cell : register bitmask for FMP bit
+- st,syscfg-fmp: Use to set Fast Mode Plus bit within SYSCFG when Fast Mode
+ Plus speed is selected by slave.
+ 1st cell: phandle to syscfg
+ 2nd cell: register offset within SYSCFG
+ 3rd cell: register bitmask for FMP bit
+ For STM32F7, STM32H7 and STM32MP1 only.
-Example :
+Example:
i2c@40005400 {
compatible = "st,stm32f4-i2c";
* NXP PCF85063 Real Time Clock
Required properties:
-- compatible: Should contain "nxp,pcf85063".
+- compatible: Should one of contain:
+ "nxp,pcf85063",
+ "nxp,pcf85063a",
+ "nxp,pcf85063tp",
+ "microcrystal,rv8263"
- reg: I2C address for chip.
Optional property:
--- /dev/null
+ASPEED BMC RTC
+==============
+
+Required properties:
+ - compatible: should be one of the following
+ * aspeed,ast2400-rtc for the ast2400
+ * aspeed,ast2500-rtc for the ast2500
+ * aspeed,ast2600-rtc for the ast2600
+
+ - reg: physical base address of the controller and length of memory mapped
+ region
+
+ - interrupts: The interrupt number
+
+Example:
+
+ rtc@1e781000 {
+ compatible = "aspeed,ast2400-rtc";
+ reg = <0x1e781000 0x18>;
+ interrupts = <22>;
+ status = "disabled";
+ };
ricoh,rv5c387a I2C bus SERIAL INTERFACE REAL-TIME CLOCK IC
sii,s35390a 2-wire CMOS real-time clock
whwave,sd3078 I2C bus SERIAL INTERFACE REAL-TIME CLOCK IC
+xircom,x1205 Xircom X1205 I2C RTC
ADI AXI-I2S controller
+The core can be generated with transmit (playback), only receive
+(capture) or both directions enabled.
+
Required properties:
- compatible : Must be "adi,axi-i2s-1.00.a"
- reg : Must contain I2S core's registers location and length
- clock-names : "axi" for the clock to the AXI interface, "ref" for the sample
rate reference clock.
- dmas: Pairs of phandle and specifier for the DMA channels that are used by
- the core. The core expects two dma channels, one for transmit and one for
- receive.
+ the core. The core expects two dma channels if both transmit and receive are
+ enabled, one channel otherwise.
- dma-names : "tx" for the transmit channel, "rx" for the receive channel.
For more details on the 'dma', 'dma-names', 'clock' and 'clock-names' properties
Required properties:
- compatible: 'amlogic,axg-toddr' or
- 'amlogic,axg-frddr'
+ 'amlogic,axg-toddr' or
+ 'amlogic,g12a-frddr' or
+ 'amlogic,g12a-toddr'
- reg: physical base address of the controller and length of memory
mapped region.
- interrupts: interrupt specifier for the fifo.
* Amlogic Audio PDM input
Required properties:
-- compatible: 'amlogic,axg-pdm'
+- compatible: 'amlogic,axg-pdm' or
+ 'amlogic,g12a-pdm'
- reg: physical base address of the controller and length of memory
mapped region.
- clocks: list of clock phandle, one for each entry clock-names.
* Amlogic Audio SPDIF Input
Required properties:
-- compatible: 'amlogic,axg-spdifin'
+- compatible: 'amlogic,axg-spdifin' or
+ 'amlogic,g12a-spdifin'
- interrupts: interrupt specifier for the spdif input.
- clocks: list of clock phandle, one for each entry clock-names.
- clock-names: should contain the following:
* Amlogic Audio SPDIF Output
Required properties:
-- compatible: 'amlogic,axg-spdifout'
+- compatible: 'amlogic,axg-spdifout' or
+ 'amlogic,g12a-spdifout'
- clocks: list of clock phandle, one for each entry clock-names.
- clock-names: should contain the following:
* "pclk" : peripheral clock.
Required properties:
- compatible: 'amlogic,axg-tdmin' or
- 'amlogic,axg-tdmout'
+ 'amlogic,axg-tdmout' or
+ 'amlogic,g12a-tdmin' or
+ 'amlogic,g12a-tdmout'
- reg: physical base address of the controller and length of memory
mapped region.
- clocks: list of clock phandle, one for each entry clock-names.
--- /dev/null
+Cirrus Logic Lochnagar Audio Development Board
+
+Lochnagar is an evaluation and development board for Cirrus Logic
+Smart CODEC and Amp devices. It allows the connection of most Cirrus
+Logic devices on mini-cards, as well as allowing connection of
+various application processor systems to provide a full evaluation
+platform. Audio system topology, clocking and power can all be
+controlled through the Lochnagar, allowing the device under test
+to be used in a variety of possible use cases.
+
+This binding document describes the binding for the audio portion
+of the driver.
+
+This binding must be part of the Lochnagar MFD binding:
+ [4] ../mfd/cirrus,lochnagar.txt
+
+Required properties:
+
+ - compatible : One of the following strings:
+ "cirrus,lochnagar2-soundcard"
+
+ - #sound-dai-cells : Must be set to 1.
+
+ - clocks : Contains an entry for each entry in clock-names.
+ - clock-names : Must include the following clocks:
+ "mclk" Master clock source for the sound card, should normally
+ be set to LOCHNAGAR_SOUNDCARD_MCLK provided by the Lochnagar
+ clock driver.
+
+Example:
+
+lochnagar-sc {
+ compatible = "cirrus,lochnagar2-soundcard";
+
+ #sound-dai-cells = <1>;
+
+ clocks = <&lochnagar_clk LOCHNAGAR_SOUNDCARD_MCLK>;
+ clock-names = "mclk";
+};
CS42L51 audio CODEC
+Required properties:
+
+ - compatible : "cirrus,cs42l51"
+
+ - reg : the I2C address of the device for I2C.
+
Optional properties:
+ - VL-supply, VD-supply, VA-supply, VAHP-supply: power supplies for the device,
+ as covered in Documentation/devicetree/bindings/regulator/regulator.txt.
+
+ - reset-gpios : GPIO specification for the reset pin. If specified, it will be
+ deasserted before starting the communication with the codec.
- clocks : a list of phandles + clock-specifiers, one for each entry in
clock-names
reg = <0x4a>;
clocks = <&mclk_prov>;
clock-names = "MCLK";
+ VL-supply = <®_audio>;
+ VD-supply = <®_audio>;
+ VA-supply = <®_audio>;
+ VAHP-supply = <®_audio>;
+ reset-gpios = <&gpiog 9 GPIO_ACTIVE_LOW>;
};
interrupt is to be used to wake system, otherwise "irq" should be used.
- wakeup-source: Flag to indicate this device can wake system (suspend/resume).
-- #clock-cells : Should be set to '<0>', only one clock source provided;
-- clock-output-names : Name given for DAI clocks output;
+- #clock-cells : Should be set to '<1>', two clock sources provided;
+- clock-output-names : Names given for DAI clock outputs (WCLK & BCLK);
- clocks : phandle and clock specifier for codec MCLK.
- clock-names : Clock name string for 'clocks' attribute, should be "mclk".
VDDMIC-supply = <®_audio>;
VDDIO-supply = <®_audio>;
- #clock-cells = <0>;
- clock-output-names = "dai-clks";
+ #clock-cells = <1>;
+ clock-output-names = "dai-wclk", "dai-bclk";
clocks = <&clks 201>;
clock-names = "mclk";
--- /dev/null
+NXP Audio Mixer (AUDMIX).
+
+The Audio Mixer is a on-chip functional module that allows mixing of two
+audio streams into a single audio stream. Audio Mixer has two input serial
+audio interfaces. These are driven by two Synchronous Audio interface
+modules (SAI). Each input serial interface carries 8 audio channels in its
+frame in TDM manner. Mixer mixes audio samples of corresponding channels
+from two interfaces into a single sample. Before mixing, audio samples of
+two inputs can be attenuated based on configuration. The output of the
+Audio Mixer is also a serial audio interface. Like input interfaces it has
+the same TDM frame format. This output is used to drive the serial DAC TDM
+interface of audio codec and also sent to the external pins along with the
+receive path of normal audio SAI module for readback by the CPU.
+
+The output of Audio Mixer can be selected from any of the three streams
+ - serial audio input 1
+ - serial audio input 2
+ - mixed audio
+
+Mixing operation is independent of audio sample rate but the two audio
+input streams must have same audio sample rate with same number of channels
+in TDM frame to be eligible for mixing.
+
+Device driver required properties:
+=================================
+ - compatible : Compatible list, contains "fsl,imx8qm-audmix"
+
+ - reg : Offset and length of the register set for the device.
+
+ - clocks : Must contain an entry for each entry in clock-names.
+
+ - clock-names : Must include the "ipg" for register access.
+
+ - power-domains : Must contain the phandle to AUDMIX power domain node
+
+ - dais : Must contain a list of phandles to AUDMIX connected
+ DAIs. The current implementation requires two phandles
+ to SAI interfaces to be provided, the first SAI in the
+ list being used to route the AUDMIX output.
+
+Device driver configuration example:
+======================================
+ audmix: audmix@59840000 {
+ compatible = "fsl,imx8qm-audmix";
+ reg = <0x0 0x59840000 0x0 0x10000>;
+ clocks = <&clk IMX8QXP_AUD_AUDMIX_IPG>;
+ clock-names = "ipg";
+ power-domains = <&pd_audmix>;
+ dais = <&sai4>, <&sai5>;
+ };
--- /dev/null
+* Microchip I2S Multi-Channel Controller
+
+Required properties:
+- compatible: Should be "microchip,sam9x60-i2smcc".
+- reg: Should be the physical base address of the controller and the
+ length of memory mapped region.
+- interrupts: Should contain the interrupt for the controller.
+- dmas: Should be one per channel name listed in the dma-names property,
+ as described in atmel-dma.txt and dma.txt files.
+- dma-names: Identifier string for each DMA request line in the dmas property.
+ Two dmas have to be defined, "tx" and "rx".
+- clocks: Must contain an entry for each entry in clock-names.
+ Please refer to clock-bindings.txt.
+- clock-names: Should be one of each entry matching the clocks phandles list:
+ - "pclk" (peripheral clock) Required.
+ - "gclk" (generated clock) Optional (1).
+
+Optional properties:
+- pinctrl-0: Should specify pin control groups used for this controller.
+- princtrl-names: Should contain only one value - "default".
+
+
+(1) : Only the peripheral clock is required. The generated clock is optional
+ and should be set mostly when Master Mode is required.
+
+Example:
+
+ i2s@f001c000 {
+ compatible = "microchip,sam9x60-i2smcc";
+ reg = <0xf001c000 0x100>;
+ interrupts = <34 IRQ_TYPE_LEVEL_HIGH 7>;
+ dmas = <&dma0
+ (AT91_XDMAC_DT_MEM_IF(0) | AT91_XDMAC_DT_PER_IF(1) |
+ AT91_XDMAC_DT_PERID(36))>,
+ <&dma0
+ (AT91_XDMAC_DT_MEM_IF(0) | AT91_XDMAC_DT_PER_IF(1) |
+ AT91_XDMAC_DT_PERID(37))>;
+ dma-names = "tx", "rx";
+ clocks = <&i2s_clk>, <&i2s_gclk>;
+ clock-names = "pclk", "gclk";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_i2s_default>;
+ };
--- /dev/null
+MT8183 with MT6358, DA7219 and MAX98357 CODECS
+
+Required properties:
+- compatible : "mediatek,mt8183_da7219_max98357"
+- mediatek,headset-codec: the phandles of da7219 codecs
+- mediatek,platform: the phandle of MT8183 ASoC platform
+
+Example:
+
+ sound {
+ compatible = "mediatek,mt8183_da7219_max98357";
+ mediatek,headset-codec = <&da7219>;
+ mediatek,platform = <&afe>;
+ };
+
--- /dev/null
+MT8183 with MT6358, TS3A227 and MAX98357 CODECS
+
+Required properties:
+- compatible : "mediatek,mt8183_mt6358_ts3a227_max98357"
+- mediatek,headset-codec: the phandles of ts3a227 codecs
+- mediatek,platform: the phandle of MT8183 ASoC platform
+
+Example:
+
+ sound {
+ compatible = "mediatek,mt8183_mt6358_ts3a227_max98357";
+ mediatek,headset-codec = <&ts3a227>;
+ mediatek,platform = <&afe>;
+ };
+
- "renesas,rcar_sound-r8a7743" (RZ/G1M)
- "renesas,rcar_sound-r8a7744" (RZ/G1N)
- "renesas,rcar_sound-r8a7745" (RZ/G1E)
+ - "renesas,rcar_sound-r8a77470" (RZ/G1C)
- "renesas,rcar_sound-r8a774a1" (RZ/G2M)
- "renesas,rcar_sound-r8a774c0" (RZ/G2E)
- "renesas,rcar_sound-r8a7778" (R-Car M1A)
- reg : Should contain the register physical address.
required register is
SRU/ADG/SSI if generation1
- SRU/ADG/SSIU/SSI if generation2
+ SRU/ADG/SSIU/SSI/AUDIO-DMAC-periperi if generation2/generation3
+ Select extended AUDIO-DMAC-periperi address if SoC has it,
+ otherwise select normal AUDIO-DMAC-periperi address.
+- reg-names : Should contain the register names.
+ scu/adg/ssi if generation1
+ scu/adg/ssiu/ssi/audmapp if generation2/generation3
- rcar_sound,ssi : Should contain SSI feature.
The number of SSI subnode should be same as HW.
see below for detail.
Required properties:
- compatible: "rockchip,pdm"
+ - "rockchip,px30-pdm"
+ - "rockchip,rk1808-pdm"
+ - "rockchip,rk3308-pdm"
- reg: physical base address of the controller and length of memory mapped
region.
- dmas: DMA specifiers for rx dma. See the DMA client binding,
- clock-names: should contain following:
- "pdm_hclk": clock for PDM BUS
- "pdm_clk" : clock for PDM controller
+- resets: a list of phandle + reset-specifer paris, one for each entry in reset-names.
+- reset-names: reset names, should include "pdm-m".
- pinctrl-names: Must contain a "default" entry.
- pinctrl-N: One property must exist for each entry in
pinctrl-names. See ../pinctrl/pinctrl-bindings.txt
2: Use JD1_2 pin for jack-detect
3: Use JD2 pin for jack-detect
+- realtek,jack-detect-not-inverted
+ bool. Normal jack-detect switches give an inverted (active-low) signal,
+ set this bool in the rare case you've a jack-detect switch which is not
+ inverted.
+
- realtek,over-current-threshold-microamp
u32, micbias over-current detection threshold in µA, valid values are
600, 1500 and 2000µA.
Required properties:
- compatible : "dioo,dio2125" or "simple-audio-amplifier"
-- enable-gpios : the gpio connected to the enable pin of the simple amplifier
Optional properties:
+- enable-gpios : the gpio connected to the enable pin of the simple amplifier
- VCC-supply : power supply for the device, as covered
in Documentation/devicetree/bindings/regulator/regulator.txt
a microphone is attached.
- simple-audio-card,aux-devs : List of phandles pointing to auxiliary devices, such
as amplifiers, to be added to the sound card.
+- simple-audio-card,pin-switches : List of strings containing the widget names for
+ which pin switches must be created.
Optional subnodes:
--- /dev/null
+Spreadtrum Multi-Channel Data Transfer Binding
+
+The Multi-channel data transfer controller is used for sound stream
+transmission between audio subsystem and other AP/CP subsystem. It
+supports 10 DAC channel and 10 ADC channel, and each channel can be
+configured with DMA mode or interrupt mode.
+
+Required properties:
+- compatible: Should be "sprd,sc9860-mcdt".
+- reg: Should contain registers address and length.
+- interrupts: Should contain one interrupt shared by all channel.
+
+Example:
+
+mcdt@41490000 {
+ compatible = "sprd,sc9860-mcdt";
+ reg = <0 0x41490000 0 0x170>;
+ interrupts = <GIC_SPI 48 IRQ_TYPE_LEVEL_HIGH>;
+};
ORCL Oracle Corporation
orisetech Orise Technology
ortustech Ortus Technology Co., Ltd.
+osddisplays OSD Displays
ovti OmniVision Technologies
oxsemi Oxford Semiconductor, Ltd.
panasonic Panasonic Corporation
rikomagic Rikomagic Tech Corp. Ltd
riscv RISC-V Foundation
rockchip Fuzhou Rockchip Electronics Co., Ltd
+rocktech ROCKTECH DISPLAYS LIMITED
rohm ROHM Semiconductor Co., Ltd
+ronbo Ronbo Electronics
roofull Shenzhen Roofull Technology Co, Ltd
samsung Samsung Semiconductor
samtec Samtec/Softing company
+.. _component:
+
======================================
Component Helper for Aggregate Drivers
======================================
dmam_pool_create()
dmam_pool_destroy()
+DRM
+ devm_drm_dev_init()
+
GPIO
devm_gpiod_get()
devm_gpiod_get_index()
Driver Load
-----------
+Component Helper Usage
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. kernel-doc:: drivers/gpu/drm/drm_drv.c
+ :doc: component helper usage recommendations
IRQ Helper Library
~~~~~~~~~~~~~~~~~~
.. kernel-doc:: drivers/gpu/drm/drm_fb_helper.c
:export:
+format Helper Functions Reference
+=================================
+
+.. kernel-doc:: drivers/gpu/drm/drm_format_helper.c
+ :export:
+
Framebuffer CMA Helper Functions Reference
==========================================
.. kernel-doc:: drivers/gpu/drm/drm_crtc_helper.c
:export:
+
+SHMEM GEM Helper Reference
+==========================
+
+.. kernel-doc:: drivers/gpu/drm/drm_gem_shmem_helper.c
+ :doc: overview
+
+.. kernel-doc:: include/drm/drm_gem_shmem_helper.h
+ :internal:
+
+.. kernel-doc:: drivers/gpu/drm/drm_gem_shmem_helper.c
+ :export:
,Virtual GPU,“suggested X”,RANGE,"Min=0, Max=0xffffffff",Connector,property to suggest an X offset for a connector
,,“suggested Y”,RANGE,"Min=0, Max=0xffffffff",Connector,property to suggest an Y offset for a connector
,Optional,"""aspect ratio""",ENUM,"{ ""None"", ""4:3"", ""16:9"" }",Connector,TDB
-,Optional,"""content type""",ENUM,"{ ""No Data"", ""Graphics"", ""Photo"", ""Cinema"", ""Game"" }",Connector,TBD
i915,Generic,"""Broadcast RGB""",ENUM,"{ ""Automatic"", ""Full"", ""Limited 16:235"" }",Connector,"When this property is set to Limited 16:235 and CTM is set, the hardware will be programmed with the result of the multiplication of CTM by the limited range matrix to ensure the pixels normaly in the range 0..1.0 are remapped to the range 16/255..235/255."
,,“audio”,ENUM,"{ ""force-dvi"", ""off"", ""auto"", ""on"" }",Connector,TBD
,SDVO-TV,“mode”,ENUM,"{ ""NTSC_M"", ""NTSC_J"", ""NTSC_443"", ""PAL_B"" } etc.",Connector,TBD
.. kernel-doc:: drivers/gpu/drm/meson/meson_venc.c
:doc: Video Encoder
-Video Canvas Management
-=======================
-
-.. kernel-doc:: drivers/gpu/drm/meson/meson_canvas.c
- :doc: Canvas
-
Video Clocks
============
-==========================
-drm/tinydrm Driver library
-==========================
+============================
+drm/tinydrm Tiny DRM drivers
+============================
-.. kernel-doc:: drivers/gpu/drm/tinydrm/core/tinydrm-core.c
- :doc: overview
-
-Core functionality
-==================
+tinydrm is a collection of DRM drivers that are so small they can fit in a
+single source file.
-.. kernel-doc:: drivers/gpu/drm/tinydrm/core/tinydrm-core.c
- :doc: core
+Helpers
+=======
-.. kernel-doc:: include/drm/tinydrm/tinydrm.h
+.. kernel-doc:: include/drm/tinydrm/tinydrm-helpers.h
:internal:
-.. kernel-doc:: drivers/gpu/drm/tinydrm/core/tinydrm-core.c
+.. kernel-doc:: drivers/gpu/drm/tinydrm/core/tinydrm-helpers.c
:export:
.. kernel-doc:: drivers/gpu/drm/tinydrm/core/tinydrm-pipe.c
:export:
-Additional helpers
-==================
-
-.. kernel-doc:: include/drm/tinydrm/tinydrm-helpers.h
- :internal:
-
-.. kernel-doc:: drivers/gpu/drm/tinydrm/core/tinydrm-helpers.c
- :export:
-
MIPI DBI Compatible Controllers
===============================
Contact: Daniel Vetter, Noralf Tronnes
-Put a reservation_object into drm_gem_object
+Remove the ->gem_prime_res_obj callback
--------------------------------------------
-This would remove the need for the ->gem_prime_res_obj callback. It would also
-allow us to implement generic helpers for waiting for a bo, allowing for quite a
-bit of refactoring in the various wait ioctl implementations.
+The ->gem_prime_res_obj callback can be removed from drivers by using the
+reservation_object in the drm_gem_object. It may also be possible to use the
+generic drm_gem_reservation_object_wait helper for waiting for a bo.
Contact: Daniel Vetter
one of the ideas for having a shared dsi/dbi helper, abstracting away the
transport details more.
-- Quick aside: The unregister devm stuff is kinda getting the lifetimes of
- a drm_device wrong. Doesn't matter, since everyone else gets it wrong
- too :-)
-
Contact: Noralf Trønnes, Daniel Vetter
AMD DC Display Driver
--- /dev/null
+Kernel driver i2c-amd-mp2
+
+Supported adapters:
+ * AMD MP2 PCIe interface
+
+Datasheet: not publicly available.
+
+Authors:
+ Shyam Sundar S K <Shyam-sundar.S-k@amd.com>
+ Nehal Shah <nehal-bakulchandra.shah@amd.com>
+ Elie Morisse <syniurge@gmail.com>
+
+Description
+-----------
+
+The MP2 is an ARM processor programmed as an I2C controller and communicating
+with the x86 host through PCI.
+
+If you see something like this:
+
+03:00.7 MP2 I2C controller: Advanced Micro Devices, Inc. [AMD] Device 15e6
+
+in your 'lspci -v', then this driver is for your device.
http://support.amd.com/us/Embedded_TechDocs/44413.pdf
* AMD Hudson-2, ML, CZ
Datasheet: Not publicly available
+ * Hygon CZ
+ Datasheet: Not publicly available
* Standard Microsystems (SMSC) SLC90E66 (Victory66) southbridge
Datasheet: Publicly available at the SMSC website http://www.smsc.com
strcpy(card->driver, "My Chip");
strcpy(card->shortname, "My Own Chip 123");
sprintf(card->longname, "%s at 0x%lx irq %i",
- card->shortname, chip->ioport, chip->irq);
+ card->shortname, chip->port, chip->irq);
/* (5) */
.... /* implemented later */
strcpy(card->driver, "My Chip");
strcpy(card->shortname, "My Own Chip 123");
sprintf(card->longname, "%s at 0x%lx irq %i",
- card->shortname, chip->ioport, chip->irq);
+ card->shortname, chip->port, chip->irq);
The driver field holds the minimal ID string of the chip. This is used
by alsa-lib's configurator, so keep it simple but unique. Even the
F: Documentation/networking/device_drivers/amazon/ena.txt
F: drivers/net/ethernet/amazon/
+AMAZON RDMA EFA DRIVER
+M: Gal Pressman <galpress@amazon.com>
+R: Yossi Leybovich <sleybo@amazon.com>
+L: linux-rdma@vger.kernel.org
+Q: https://patchwork.kernel.org/project/linux-rdma/list/
+S: Supported
+F: drivers/infiniband/hw/efa/
+F: include/uapi/rdma/efa-abi.h
+
AMD CRYPTOGRAPHIC COPROCESSOR (CCP) DRIVER
M: Tom Lendacky <thomas.lendacky@amd.com>
M: Gary Hook <gary.hook@amd.com>
F: drivers/gpu/drm/amd/include/v9_structs.h
F: include/uapi/linux/kfd_ioctl.h
+AMD MP2 I2C DRIVER
+M: Elie Morisse <syniurge@gmail.com>
+M: Nehal Shah <nehal-bakulchandra.shah@amd.com>
+M: Shyam Sundar S K <shyam-sundar.s-k@amd.com>
+L: linux-i2c@vger.kernel.org
+S: Maintained
+F: drivers/i2c/busses/i2c-amd-mp2*
+
AMD POWERPLAY
M: Rex Zhu <rex.zhu@amd.com>
M: Evan Quan <evan.quan@amd.com>
X: drivers/iio/*/adjd*
F: drivers/staging/iio/*/ad*
+ANALOGBITS PLL LIBRARIES
+M: Paul Walmsley <paul.walmsley@sifive.com>
+S: Supported
+F: drivers/clk/analogbits/*
+F: include/linux/clk/analogbits*
+
ANDES ARCHITECTURE
M: Greentime Hu <green.hu@gmail.com>
M: Vincent Chen <deanbo422@gmail.com>
T: git git://linux-arm.org/linux-ld.git for-upstream/mali-dp
F: drivers/gpu/drm/arm/display/include/
F: drivers/gpu/drm/arm/display/komeda/
-F: Documentation/devicetree/bindings/display/arm/arm,komeda.txt
+F: Documentation/devicetree/bindings/display/arm,komeda.txt
F: Documentation/gpu/komeda-kms.rst
ARM MALI-DP DRM DRIVER
F: Documentation/devicetree/bindings/display/arm,malidp.txt
F: Documentation/gpu/afbc.rst
+ARM MALI PANFROST DRM DRIVER
+M: Rob Herring <robh@kernel.org>
+M: Tomeu Vizoso <tomeu.vizoso@collabora.com>
+L: dri-devel@lists.freedesktop.org
+S: Supported
+T: git git://anongit.freedesktop.org/drm/drm-misc
+F: drivers/gpu/drm/panfrost/
+F: include/uapi/drm/panfrost_drm.h
+
ARM MFM AND FLOPPY DRIVERS
M: Ian Molton <spyro@f2s.com>
S: Maintained
F: include/linux/async_tx.h
AT24 EEPROM DRIVER
-M: Bartosz Golaszewski <brgl@bgdev.pl>
+M: Bartosz Golaszewski <bgolaszewski@baylibre.com>
L: linux-i2c@vger.kernel.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux.git
S: Maintained
BT87X AUDIO DRIVER
M: Clemens Ladisch <clemens@ladisch.de>
L: alsa-devel@alsa-project.org (moderated for non-subscribers)
-T: git git://git.alsa-project.org/alsa-kernel.git
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git
S: Maintained
F: Documentation/sound/cards/bt87x.rst
F: sound/pci/bt87x.c
C-MEDIA CMI8788 DRIVER
M: Clemens Ladisch <clemens@ladisch.de>
L: alsa-devel@alsa-project.org (moderated for non-subscribers)
-T: git git://git.alsa-project.org/alsa-kernel.git
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git
S: Maintained
F: sound/pci/oxygen/
F: drivers/mfd/lochnagar-i2c.c
F: drivers/pinctrl/cirrus/pinctrl-lochnagar.c
F: drivers/regulator/lochnagar-regulator.c
+F: sound/soc/codecs/lochnagar-sc.c
F: include/dt-bindings/clk/lochnagar.h
F: include/dt-bindings/pinctrl/lochnagar.h
F: include/linux/mfd/lochnagar*
F: Documentation/devicetree/bindings/hwmon/cirrus,lochnagar.txt
F: Documentation/devicetree/bindings/pinctrl/cirrus,lochnagar.txt
F: Documentation/devicetree/bindings/regulator/cirrus,lochnagar.txt
+F: Documentation/devicetree/bindings/sound/cirrus,lochnagar.txt
F: Documentation/hwmon/lochnagar
CISCO FCOE HBA DRIVER
F: drivers/scsi/cxgbi/cxgb3i
CXGB3 IWARP RNIC DRIVER (IW_CXGB3)
-M: Steve Wise <swise@chelsio.com>
+M: Potnuri Bharat Teja <bharat@chelsio.com>
L: linux-rdma@vger.kernel.org
W: http://www.openfabrics.org
S: Supported
F: drivers/scsi/cxgbi/cxgb4i
CXGB4 IWARP RNIC DRIVER (IW_CXGB4)
-M: Steve Wise <swise@chelsio.com>
+M: Potnuri Bharat Teja <bharat@chelsio.com>
L: linux-rdma@vger.kernel.org
W: http://www.openfabrics.org
S: Supported
S: Odd Fixes
F: drivers/gpu/drm/ast/
+DRM DRIVER FOR ASPEED BMC GFX
+M: Joel Stanley <joel@jms.id.au>
+L: linux-aspeed@lists.ozlabs.org
+T: git git://anongit.freedesktop.org/drm/drm-misc
+S: Supported
+F: drivers/gpu/drm/aspeed/
+F: Documentation/devicetree/bindings/gpu/aspeed-gfx.txt
+
DRM DRIVER FOR BOCHS VIRTUAL GPU
M: Gerd Hoffmann <kraxel@redhat.com>
L: virtualization@lists.linux-foundation.org
S: Maintained
F: drivers/gpu/drm/tve200/
+DRM DRIVER FOR FEIYANG FY07024DI26A30-D MIPI-DSI LCD PANELS
+M: Jagan Teki <jagan@amarulasolutions.com>
+S: Maintained
+F: drivers/gpu/drm/panel/panel-feiyang-fy07024di26a30d.c
+F: Documentation/devicetree/bindings/display/panel/feiyang,fy07024di26a30d.txt
+
DRM DRIVER FOR ILITEK ILI9225 PANELS
M: David Lechner <david@lechnology.com>
S: Maintained
F: drivers/gpu/drm/r128/
F: include/uapi/drm/r128_drm.h
+DRM DRIVER FOR ROCKTECH JH057N00900 PANELS
+M: Guido Günther <agx@sigxcpu.org>
+S: Maintained
+F: drivers/gpu/drm/panel/panel-rocktech-jh057n00900.c
+F: Documentation/devicetree/bindings/display/panel/rocktech,jh057n00900.txt
+
DRM DRIVER FOR SAVAGE VIDEO CARDS
S: Orphan / Obsolete
F: drivers/gpu/drm/savage/
F: drivers/gpu/drm/udl/
T: git git://anongit.freedesktop.org/drm/drm-misc
+DRM DRIVER FOR VIRTUALBOX VIRTUAL GPU
+M: Hans de Goede <hdegoede@redhat.com>
+L: dri-devel@lists.freedesktop.org
+S: Maintained
+F: drivers/gpu/drm/vboxvideo/
+T: git git://anongit.freedesktop.org/drm/drm-misc
+
DRM DRIVER FOR VIRTUAL KERNEL MODESETTING (VKMS)
M: Rodrigo Siqueira <rodrigosiqueiramelo@gmail.com>
R: Haneen Mohammed <hamohammed.sa@gmail.com>
F: drivers/gpu/drm/hisilicon/
F: Documentation/devicetree/bindings/display/hisilicon/
+DRM DRIVERS FOR LIMA
+M: Qiang Yu <yuq825@gmail.com>
+L: dri-devel@lists.freedesktop.org
+L: lima@lists.freedesktop.org (moderated for non-subscribers)
+S: Maintained
+F: drivers/gpu/drm/lima/
+F: include/uapi/drm/lima_drm.h
+T: git git://anongit.freedesktop.org/drm/drm-misc
+
DRM DRIVERS FOR MEDIATEK
M: CK Hu <ck.hu@mediatek.com>
M: Philipp Zabel <p.zabel@pengutronix.de>
EDIROL UA-101/UA-1000 DRIVER
M: Clemens Ladisch <clemens@ladisch.de>
L: alsa-devel@alsa-project.org (moderated for non-subscribers)
-T: git git://git.alsa-project.org/alsa-kernel.git
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git
S: Maintained
F: sound/usb/misc/ua101.c
FIREWIRE AUDIO DRIVERS
M: Clemens Ladisch <clemens@ladisch.de>
L: alsa-devel@alsa-project.org (moderated for non-subscribers)
-T: git git://git.alsa-project.org/alsa-kernel.git
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git
S: Maintained
F: sound/firewire/
F: include/uapi/linux/if_infiniband.h
F: include/uapi/rdma/
F: include/rdma/
+F: include/trace/events/ib_mad.h
+F: include/trace/events/ib_umad.h
+F: samples/bpf/ibumad_kern.c
+F: samples/bpf/ibumad_user.c
INGENIC JZ4780 DMA Driver
M: Zubair Lutfullah Kakakhel <Zubair.Kakakhel@imgtec.com>
M: Ludovic Desroches <ludovic.desroches@microchip.com>
L: linux-i2c@vger.kernel.org
S: Supported
-F: drivers/i2c/busses/i2c-at91.c
+F: drivers/i2c/busses/i2c-at91.h
+F: drivers/i2c/busses/i2c-at91-*.c
MICROCHIP ISC DRIVER
M: Eugen Hristev <eugen.hristev@microchip.com>
OPL4 DRIVER
M: Clemens Ladisch <clemens@ladisch.de>
L: alsa-devel@alsa-project.org (moderated for non-subscribers)
-T: git git://git.alsa-project.org/alsa-kernel.git
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git
S: Maintained
F: sound/drivers/opl4/
L: alsa-devel@alsa-project.org (moderated for non-subscribers)
W: http://www.alsa-project.org/
T: git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git
-T: git git://git.alsa-project.org/alsa-kernel.git
Q: http://patchwork.kernel.org/project/alsa-devel/list/
S: Maintained
F: Documentation/sound/
USB MIDI DRIVER
M: Clemens Ladisch <clemens@ladisch.de>
L: alsa-devel@alsa-project.org (moderated for non-subscribers)
-T: git git://git.alsa-project.org/alsa-kernel.git
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git
S: Maintained
F: sound/usb/midi.*
return chan < 4 ? count : (count << 1);
}
+static struct device isa_dma_dev = {
+ .init_name = "fallback device",
+ .coherent_dma_mask = ~(dma_addr_t)0,
+ .dma_mask = &isa_dma_dev.coherent_dma_mask,
+};
+
static void isa_enable_dma(unsigned int chan, dma_t *dma)
{
if (dma->invalid) {
dma->sg = &dma->buf;
dma->sgcount = 1;
dma->buf.length = dma->count;
- dma->buf.dma_address = dma_map_single(NULL,
+ dma->buf.dma_address = dma_map_single(&isa_dma_dev,
dma->addr, dma->count,
direction);
}
if (cpu_is_omap343x())
features.flags |= TI_CLK_DPLL_HAS_FREQSEL;
+ if (omap_type() == OMAP2_DEVICE_TYPE_GP)
+ features.flags |= TI_CLK_DEVICE_TYPE_GP;
+
/* Idlest value for interface clocks.
* 24xx uses 0 to indicate not ready, and 1 to indicate ready.
* 34xx reverses this, just to keep us on our toes
if (oh->clkdm) {
return oh->clkdm;
} else if (oh->_clk) {
- if (__clk_get_flags(oh->_clk) & CLK_IS_BASIC)
+ if (!omap2_clk_is_hw_omap(__clk_get_hw(oh->_clk)))
return NULL;
clk = to_clk_hw_omap(__clk_get_hw(oh->_clk));
- return clk->clkdm;
+ return clk->clkdm;
}
return NULL;
}
free_irq(idma->irq, idma);
}
+static struct device isa_dma_dev = {
+ .init_name = "fallback device",
+ .coherent_dma_mask = ~(dma_addr_t)0,
+ .dma_mask = &isa_dma_dev.coherent_dma_mask,
+};
+
static void iomd_enable_dma(unsigned int chan, dma_t *dma)
{
struct iomd_dma *idma = container_of(dma, struct iomd_dma, dma);
idma->dma.sg = &idma->dma.buf;
idma->dma.sgcount = 1;
idma->dma.buf.length = idma->dma.count;
- idma->dma.buf.dma_address = dma_map_single(NULL,
+ idma->dma.buf.dma_address = dma_map_single(&isa_dma_dev,
idma->dma.addr, idma->dma.count,
idma->dma.dma_mode == DMA_MODE_READ ?
DMA_FROM_DEVICE : DMA_TO_DEVICE);
select ARCH_HAS_DEVMEM_IS_ALLOWED
select ARCH_HAS_DMA_COHERENT_TO_PFN
select ARCH_HAS_DMA_MMAP_PGPROT
+ select ARCH_HAS_DMA_PREP_COHERENT
select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI
select ARCH_HAS_ELF_RANDOMIZE
select ARCH_HAS_FAST_MULTIPLIER
config CSKY
def_bool y
select ARCH_32BIT_OFF_T
+ select ARCH_HAS_DMA_PREP_COHERENT
select ARCH_HAS_SYNC_DMA_FOR_CPU
select ARCH_HAS_SYNC_DMA_FOR_DEVICE
select ARCH_USE_BUILTIN_BSWAP
id.name = ALCHEMY_CPU_CLK;
id.parent_names = &parent_name;
id.num_parents = 1;
- id.flags = CLK_IS_BASIC;
+ id.flags = 0;
id.ops = &alchemy_clkops_cpu;
h->init = &id;
const char *name, const char *parent_name, u8 clkflags,
u32 __iomem *reg, u8 pos, u8 len, int divflags)
{
+ divflags |= CLK_DIVIDER_BIG_ENDIAN;
return clk_register_divider(NULL, name, parent_name, clkflags,
reg, pos, len, divflags, &clklock);
}
{
u8 divflags;
- divflags = 0;
+ divflags = CLK_DIVIDER_BIG_ENDIAN;
return clk_register_divider_table(NULL, name, parent_name, 0,
reg, pos, len, divflags,
divtab, &clklock);
u32 __iomem *reg, u8 pos)
{
int clkflags;
+ u8 gateflags;
clkflags = CLK_SET_RATE_PARENT;
+ gateflags = CLK_GATE_BIG_ENDIAN;
return clk_register_gate(NULL, name, parent_name, clkflags,
- reg, pos, 0, &clklock);
+ reg, pos, gateflags, &clklock);
}
static inline struct clk *mpc512x_clk_muxed(const char *name,
u8 muxflags;
clkflags = CLK_SET_RATE_PARENT;
- muxflags = 0;
+ muxflags = CLK_MUX_BIG_ENDIAN;
return clk_register_mux(NULL, name,
parent_names, parent_count, clkflags,
reg, pos, len, muxflags, &clklock);
#include <asm/swiotlb.h>
#include <linux/dma-contiguous.h>
-#ifdef CONFIG_ISA
-# define ISA_DMA_BIT_MASK DMA_BIT_MASK(24)
-#else
-# define ISA_DMA_BIT_MASK DMA_BIT_MASK(32)
-#endif
-
extern int iommu_merge;
-extern struct device x86_dma_fallback_dev;
extern int panic_on_overflow;
extern const struct dma_map_ops *dma_ops;
return dma_ops;
}
-bool arch_dma_alloc_attrs(struct device **dev);
-#define arch_dma_alloc_attrs arch_dma_alloc_attrs
-
#endif
unsigned long bus;
phys_addr_t paddr = page_to_phys(page) + offset;
- if (!dev)
- dev = &x86_dma_fallback_dev;
-
if (!need_iommu(dev, paddr, size))
return paddr;
if (nents == 0)
return 0;
- if (!dev)
- dev = &x86_dma_fallback_dev;
-
out = 0;
start = 0;
start_sg = sg;
INTEL_I945G_IDS(&gen3_early_ops),
INTEL_I945GM_IDS(&gen3_early_ops),
INTEL_VLV_IDS(&gen6_early_ops),
- INTEL_PINEVIEW_IDS(&gen3_early_ops),
+ INTEL_PINEVIEW_G_IDS(&gen3_early_ops),
+ INTEL_PINEVIEW_M_IDS(&gen3_early_ops),
INTEL_I965G_IDS(&gen3_early_ops),
INTEL_G33_IDS(&gen3_early_ops),
INTEL_I965GM_IDS(&gen3_early_ops),
INTEL_GLK_IDS(&gen9_early_ops),
INTEL_CNL_IDS(&gen9_early_ops),
INTEL_ICL_11_IDS(&gen11_early_ops),
+ INTEL_EHL_IDS(&gen11_early_ops),
};
struct resource intel_graphics_stolen_res __ro_after_init = DEFINE_RES_MEM(0, 0);
* Copyright (C) 2018 IBM Corporation
*/
#include <linux/efi.h>
+#include <linux/module.h>
#include <linux/ima.h>
extern struct boot_params boot_params;
"appraise func=KEXEC_KERNEL_CHECK appraise_type=imasig",
#endif /* CONFIG_KEXEC_VERIFY_SIG */
"measure func=KEXEC_KERNEL_CHECK",
+#if !IS_ENABLED(CONFIG_MODULE_SIG)
+ "appraise func=MODULE_CHECK appraise_type=imasig",
+#endif
+ "measure func=MODULE_CHECK",
NULL
};
const char * const *arch_get_ima_policy(void)
{
- if (IS_ENABLED(CONFIG_IMA_ARCH_POLICY) && arch_ima_get_secureboot())
+ if (IS_ENABLED(CONFIG_IMA_ARCH_POLICY) && arch_ima_get_secureboot()) {
+ if (IS_ENABLED(CONFIG_MODULE_SIG))
+ set_module_sig_enforced();
return sb_arch_rules;
+ }
return NULL;
}
extern struct iommu_table_entry __iommu_table[], __iommu_table_end[];
-/* Dummy device used for NULL arguments (normally ISA). */
-struct device x86_dma_fallback_dev = {
- .init_name = "fallback device",
- .coherent_dma_mask = ISA_DMA_BIT_MASK,
- .dma_mask = &x86_dma_fallback_dev.coherent_dma_mask,
-};
-EXPORT_SYMBOL(x86_dma_fallback_dev);
-
void __init pci_iommu_alloc(void)
{
struct iommu_table_entry *p;
}
}
-bool arch_dma_alloc_attrs(struct device **dev)
-{
- if (!*dev)
- *dev = &x86_dma_fallback_dev;
-
- if (!is_device_dma_capable(*dev))
- return false;
- return true;
-
-}
-EXPORT_SYMBOL(arch_dma_alloc_attrs);
-
/*
* See <Documentation/x86/x86_64/boot-options.txt> for the iommu kernel
* parameter documentation.
+# SPDX-License-Identifier: GPL-2.0
config CLKDEV_LOOKUP
bool
---help---
Sypport for the APM X-Gene SoC reference, PLL, and device clocks.
+config COMMON_CLK_LOCHNAGAR
+ tristate "Cirrus Logic Lochnagar clock driver"
+ depends on MFD_LOCHNAGAR
+ help
+ This driver supports the clocking features of the Cirrus Logic
+ Lochnagar audio development board.
+
config COMMON_CLK_NXP
def_bool COMMON_CLK && (ARCH_LPC18XX || ARCH_LPC32XX)
select REGMAP_MMIO if ARCH_LPC32XX
Support for Memory Mapped IO Fixed clocks
source "drivers/clk/actions/Kconfig"
+source "drivers/clk/analogbits/Kconfig"
source "drivers/clk/bcm/Kconfig"
source "drivers/clk/hisilicon/Kconfig"
source "drivers/clk/imgtec/Kconfig"
source "drivers/clk/qcom/Kconfig"
source "drivers/clk/renesas/Kconfig"
source "drivers/clk/samsung/Kconfig"
+source "drivers/clk/sifive/Kconfig"
source "drivers/clk/sprd/Kconfig"
+source "drivers/clk/sunxi/Kconfig"
source "drivers/clk/sunxi-ng/Kconfig"
source "drivers/clk/tegra/Kconfig"
source "drivers/clk/ti/Kconfig"
obj-$(CONFIG_COMMON_CLK_ASPEED) += clk-aspeed.o
obj-$(CONFIG_ARCH_HIGHBANK) += clk-highbank.o
obj-$(CONFIG_CLK_HSDK) += clk-hsdk-pll.o
+obj-$(CONFIG_COMMON_CLK_LOCHNAGAR) += clk-lochnagar.o
obj-$(CONFIG_COMMON_CLK_MAX77686) += clk-max77686.o
obj-$(CONFIG_COMMON_CLK_MAX9485) += clk-max9485.o
+obj-$(CONFIG_ARCH_MILBEAUT_M10V) += clk-milbeaut.o
obj-$(CONFIG_ARCH_MOXART) += clk-moxart.o
obj-$(CONFIG_ARCH_NOMADIK) += clk-nomadik.o
obj-$(CONFIG_ARCH_NPCM7XX) += clk-npcm7xx.o
# please keep this section sorted lexicographically by directory path name
obj-y += actions/
+obj-y += analogbits/
obj-$(CONFIG_COMMON_CLK_AT91) += at91/
obj-$(CONFIG_ARCH_ARTPEC) += axis/
obj-$(CONFIG_ARC_PLAT_AXS10X) += axs10x/
obj-y += renesas/
obj-$(CONFIG_ARCH_ROCKCHIP) += rockchip/
obj-$(CONFIG_COMMON_CLK_SAMSUNG) += samsung/
+obj-$(CONFIG_CLK_SIFIVE) += sifive/
obj-$(CONFIG_ARCH_SIRF) += sirf/
obj-$(CONFIG_ARCH_SOCFPGA) += socfpga/
obj-$(CONFIG_PLAT_SPEAR) += spear/
-// SPDX-License-Identifier: GPL-2.0+
+/* SPDX-License-Identifier: GPL-2.0+ */
//
// OWL common clock driver
//
-// SPDX-License-Identifier: GPL-2.0+
+/* SPDX-License-Identifier: GPL-2.0+ */
//
// OWL composite clock driver
//
-// SPDX-License-Identifier: GPL-2.0+
+/* SPDX-License-Identifier: GPL-2.0+ */
//
// OWL divider clock driver
//
-// SPDX-License-Identifier: GPL-2.0+
+/* SPDX-License-Identifier: GPL-2.0+ */
//
// OWL factor clock driver
//
-// SPDX-License-Identifier: GPL-2.0+
+/* SPDX-License-Identifier: GPL-2.0+ */
//
// OWL fixed factor clock driver
//
-// SPDX-License-Identifier: GPL-2.0+
+/* SPDX-License-Identifier: GPL-2.0+ */
//
// OWL gate clock driver
//
-// SPDX-License-Identifier: GPL-2.0+
+/* SPDX-License-Identifier: GPL-2.0+ */
//
// OWL mux clock driver
//
-// SPDX-License-Identifier: GPL-2.0+
+/* SPDX-License-Identifier: GPL-2.0+ */
//
// OWL pll clock driver
//
-// SPDX-License-Identifier: GPL-2.0-or-later
+/* SPDX-License-Identifier: GPL-2.0-or-later */
//
// Actions Semi Owl SoCs Reset Management Unit driver
//
--- /dev/null
+config CLK_ANALOGBITS_WRPLL_CLN28HPC
+ bool
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_CLK_ANALOGBITS_WRPLL_CLN28HPC) += wrpll-cln28hpc.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2018-2019 SiFive, Inc.
+ * Wesley Terpstra
+ * Paul Walmsley
+ *
+ * This library supports configuration parsing and reprogramming of
+ * the CLN28HPC variant of the Analog Bits Wide Range PLL. The
+ * intention is for this library to be reusable for any device that
+ * integrates this PLL; thus the register structure and programming
+ * details are expected to be provided by a separate IP block driver.
+ *
+ * The bulk of this code is primarily useful for clock configurations
+ * that must operate at arbitrary rates, as opposed to clock configurations
+ * that are restricted by software or manufacturer guidance to a small,
+ * pre-determined set of performance points.
+ *
+ * References:
+ * - Analog Bits "Wide Range PLL Datasheet", version 2015.10.01
+ * - SiFive FU540-C000 Manual v1p0, Chapter 7 "Clocking and Reset"
+ * https://static.dev.sifive.com/FU540-C000-v1.0.pdf
+ */
+
+#include <linux/bug.h>
+#include <linux/err.h>
+#include <linux/log2.h>
+#include <linux/math64.h>
+#include <linux/clk/analogbits-wrpll-cln28hpc.h>
+
+/* MIN_INPUT_FREQ: minimum input clock frequency, in Hz (Fref_min) */
+#define MIN_INPUT_FREQ 7000000
+
+/* MAX_INPUT_FREQ: maximum input clock frequency, in Hz (Fref_max) */
+#define MAX_INPUT_FREQ 600000000
+
+/* MIN_POST_DIVIDE_REF_FREQ: minimum post-divider reference frequency, in Hz */
+#define MIN_POST_DIVR_FREQ 7000000
+
+/* MAX_POST_DIVIDE_REF_FREQ: maximum post-divider reference frequency, in Hz */
+#define MAX_POST_DIVR_FREQ 200000000
+
+/* MIN_VCO_FREQ: minimum VCO frequency, in Hz (Fvco_min) */
+#define MIN_VCO_FREQ 2400000000UL
+
+/* MAX_VCO_FREQ: maximum VCO frequency, in Hz (Fvco_max) */
+#define MAX_VCO_FREQ 4800000000ULL
+
+/* MAX_DIVQ_DIVISOR: maximum output divisor. Selected by DIVQ = 6 */
+#define MAX_DIVQ_DIVISOR 64
+
+/* MAX_DIVR_DIVISOR: maximum reference divisor. Selected by DIVR = 63 */
+#define MAX_DIVR_DIVISOR 64
+
+/* MAX_LOCK_US: maximum PLL lock time, in microseconds (tLOCK_max) */
+#define MAX_LOCK_US 70
+
+/*
+ * ROUND_SHIFT: number of bits to shift to avoid precision loss in the rounding
+ * algorithm
+ */
+#define ROUND_SHIFT 20
+
+/*
+ * Private functions
+ */
+
+/**
+ * __wrpll_calc_filter_range() - determine PLL loop filter bandwidth
+ * @post_divr_freq: input clock rate after the R divider
+ *
+ * Select the value to be presented to the PLL RANGE input signals, based
+ * on the input clock frequency after the post-R-divider @post_divr_freq.
+ * This code follows the recommendations in the PLL datasheet for filter
+ * range selection.
+ *
+ * Return: The RANGE value to be presented to the PLL configuration inputs,
+ * or a negative return code upon error.
+ */
+static int __wrpll_calc_filter_range(unsigned long post_divr_freq)
+{
+ if (post_divr_freq < MIN_POST_DIVR_FREQ ||
+ post_divr_freq > MAX_POST_DIVR_FREQ) {
+ WARN(1, "%s: post-divider reference freq out of range: %lu",
+ __func__, post_divr_freq);
+ return -ERANGE;
+ }
+
+ switch (post_divr_freq) {
+ case 0 ... 10999999:
+ return 1;
+ case 11000000 ... 17999999:
+ return 2;
+ case 18000000 ... 29999999:
+ return 3;
+ case 30000000 ... 49999999:
+ return 4;
+ case 50000000 ... 79999999:
+ return 5;
+ case 80000000 ... 129999999:
+ return 6;
+ }
+
+ return 7;
+}
+
+/**
+ * __wrpll_calc_fbdiv() - return feedback fixed divide value
+ * @c: ptr to a struct wrpll_cfg record to read from
+ *
+ * The internal feedback path includes a fixed by-two divider; the
+ * external feedback path does not. Return the appropriate divider
+ * value (2 or 1) depending on whether internal or external feedback
+ * is enabled. This code doesn't test for invalid configurations
+ * (e.g. both or neither of WRPLL_FLAGS_*_FEEDBACK are set); it relies
+ * on the caller to do so.
+ *
+ * Context: Any context. Caller must protect the memory pointed to by
+ * @c from simultaneous modification.
+ *
+ * Return: 2 if internal feedback is enabled or 1 if external feedback
+ * is enabled.
+ */
+static u8 __wrpll_calc_fbdiv(const struct wrpll_cfg *c)
+{
+ return (c->flags & WRPLL_FLAGS_INT_FEEDBACK_MASK) ? 2 : 1;
+}
+
+/**
+ * __wrpll_calc_divq() - determine DIVQ based on target PLL output clock rate
+ * @target_rate: target PLL output clock rate
+ * @vco_rate: pointer to a u64 to store the computed VCO rate into
+ *
+ * Determine a reasonable value for the PLL Q post-divider, based on the
+ * target output rate @target_rate for the PLL. Along with returning the
+ * computed Q divider value as the return value, this function stores the
+ * desired target VCO rate into the variable pointed to by @vco_rate.
+ *
+ * Context: Any context. Caller must protect the memory pointed to by
+ * @vco_rate from simultaneous access or modification.
+ *
+ * Return: a positive integer DIVQ value to be programmed into the hardware
+ * upon success, or 0 upon error (since 0 is an invalid DIVQ value)
+ */
+static u8 __wrpll_calc_divq(u32 target_rate, u64 *vco_rate)
+{
+ u64 s;
+ u8 divq = 0;
+
+ if (!vco_rate) {
+ WARN_ON(1);
+ goto wcd_out;
+ }
+
+ s = div_u64(MAX_VCO_FREQ, target_rate);
+ if (s <= 1) {
+ divq = 1;
+ *vco_rate = MAX_VCO_FREQ;
+ } else if (s > MAX_DIVQ_DIVISOR) {
+ divq = ilog2(MAX_DIVQ_DIVISOR);
+ *vco_rate = MIN_VCO_FREQ;
+ } else {
+ divq = ilog2(s);
+ *vco_rate = (u64)target_rate << divq;
+ }
+
+wcd_out:
+ return divq;
+}
+
+/**
+ * __wrpll_update_parent_rate() - update PLL data when parent rate changes
+ * @c: ptr to a struct wrpll_cfg record to write PLL data to
+ * @parent_rate: PLL input refclk rate (pre-R-divider)
+ *
+ * Pre-compute some data used by the PLL configuration algorithm when
+ * the PLL's reference clock rate changes. The intention is to avoid
+ * computation when the parent rate remains constant - expected to be
+ * the common case.
+ *
+ * Returns: 0 upon success or -ERANGE if the reference clock rate is
+ * out of range.
+ */
+static int __wrpll_update_parent_rate(struct wrpll_cfg *c,
+ unsigned long parent_rate)
+{
+ u8 max_r_for_parent;
+
+ if (parent_rate > MAX_INPUT_FREQ || parent_rate < MIN_POST_DIVR_FREQ)
+ return -ERANGE;
+
+ c->parent_rate = parent_rate;
+ max_r_for_parent = div_u64(parent_rate, MIN_POST_DIVR_FREQ);
+ c->max_r = min_t(u8, MAX_DIVR_DIVISOR, max_r_for_parent);
+
+ c->init_r = DIV_ROUND_UP_ULL(parent_rate, MAX_POST_DIVR_FREQ);
+
+ return 0;
+}
+
+/**
+ * wrpll_configure() - compute PLL configuration for a target rate
+ * @c: ptr to a struct wrpll_cfg record to write into
+ * @target_rate: target PLL output clock rate (post-Q-divider)
+ * @parent_rate: PLL input refclk rate (pre-R-divider)
+ *
+ * Compute the appropriate PLL signal configuration values and store
+ * in PLL context @c. PLL reprogramming is not glitchless, so the
+ * caller should switch any downstream logic to a different clock
+ * source or clock-gate it before presenting these values to the PLL
+ * configuration signals.
+ *
+ * The caller must pass this function a pre-initialized struct
+ * wrpll_cfg record: either initialized to zero (with the
+ * exception of the .name and .flags fields) or read from the PLL.
+ *
+ * Context: Any context. Caller must protect the memory pointed to by @c
+ * from simultaneous access or modification.
+ *
+ * Return: 0 upon success; anything else upon failure.
+ */
+int wrpll_configure_for_rate(struct wrpll_cfg *c, u32 target_rate,
+ unsigned long parent_rate)
+{
+ unsigned long ratio;
+ u64 target_vco_rate, delta, best_delta, f_pre_div, vco, vco_pre;
+ u32 best_f, f, post_divr_freq;
+ u8 fbdiv, divq, best_r, r;
+ int range;
+
+ if (c->flags == 0) {
+ WARN(1, "%s called with uninitialized PLL config", __func__);
+ return -EINVAL;
+ }
+
+ /* Initialize rounding data if it hasn't been initialized already */
+ if (parent_rate != c->parent_rate) {
+ if (__wrpll_update_parent_rate(c, parent_rate)) {
+ pr_err("%s: PLL input rate is out of range\n",
+ __func__);
+ return -ERANGE;
+ }
+ }
+
+ c->flags &= ~WRPLL_FLAGS_RESET_MASK;
+
+ /* Put the PLL into bypass if the user requests the parent clock rate */
+ if (target_rate == parent_rate) {
+ c->flags |= WRPLL_FLAGS_BYPASS_MASK;
+ return 0;
+ }
+
+ c->flags &= ~WRPLL_FLAGS_BYPASS_MASK;
+
+ /* Calculate the Q shift and target VCO rate */
+ divq = __wrpll_calc_divq(target_rate, &target_vco_rate);
+ if (!divq)
+ return -1;
+ c->divq = divq;
+
+ /* Precalculate the pre-Q divider target ratio */
+ ratio = div64_u64((target_vco_rate << ROUND_SHIFT), parent_rate);
+
+ fbdiv = __wrpll_calc_fbdiv(c);
+ best_r = 0;
+ best_f = 0;
+ best_delta = MAX_VCO_FREQ;
+
+ /*
+ * Consider all values for R which land within
+ * [MIN_POST_DIVR_FREQ, MAX_POST_DIVR_FREQ]; prefer smaller R
+ */
+ for (r = c->init_r; r <= c->max_r; ++r) {
+ f_pre_div = ratio * r;
+ f = (f_pre_div + (1 << ROUND_SHIFT)) >> ROUND_SHIFT;
+ f >>= (fbdiv - 1);
+
+ post_divr_freq = div_u64(parent_rate, r);
+ vco_pre = fbdiv * post_divr_freq;
+ vco = vco_pre * f;
+
+ /* Ensure rounding didn't take us out of range */
+ if (vco > target_vco_rate) {
+ --f;
+ vco = vco_pre * f;
+ } else if (vco < MIN_VCO_FREQ) {
+ ++f;
+ vco = vco_pre * f;
+ }
+
+ delta = abs(target_rate - vco);
+ if (delta < best_delta) {
+ best_delta = delta;
+ best_r = r;
+ best_f = f;
+ }
+ }
+
+ c->divr = best_r - 1;
+ c->divf = best_f - 1;
+
+ post_divr_freq = div_u64(parent_rate, best_r);
+
+ /* Pick the best PLL jitter filter */
+ range = __wrpll_calc_filter_range(post_divr_freq);
+ if (range < 0)
+ return range;
+ c->range = range;
+
+ return 0;
+}
+
+/**
+ * wrpll_calc_output_rate() - calculate the PLL's target output rate
+ * @c: ptr to a struct wrpll_cfg record to read from
+ * @parent_rate: PLL refclk rate
+ *
+ * Given a pointer to the PLL's current input configuration @c and the
+ * PLL's input reference clock rate @parent_rate (before the R
+ * pre-divider), calculate the PLL's output clock rate (after the Q
+ * post-divider).
+ *
+ * Context: Any context. Caller must protect the memory pointed to by @c
+ * from simultaneous modification.
+ *
+ * Return: the PLL's output clock rate, in Hz. The return value from
+ * this function is intended to be convenient to pass directly
+ * to the Linux clock framework; thus there is no explicit
+ * error return value.
+ */
+unsigned long wrpll_calc_output_rate(const struct wrpll_cfg *c,
+ unsigned long parent_rate)
+{
+ u8 fbdiv;
+ u64 n;
+
+ if (c->flags & WRPLL_FLAGS_EXT_FEEDBACK_MASK) {
+ WARN(1, "external feedback mode not yet supported");
+ return ULONG_MAX;
+ }
+
+ fbdiv = __wrpll_calc_fbdiv(c);
+ n = parent_rate * fbdiv * (c->divf + 1);
+ n = div_u64(n, c->divr + 1);
+ n >>= c->divq;
+
+ return n;
+}
+
+/**
+ * wrpll_calc_max_lock_us() - return the time for the PLL to lock
+ * @c: ptr to a struct wrpll_cfg record to read from
+ *
+ * Return the minimum amount of time (in microseconds) that the caller
+ * must wait after reprogramming the PLL to ensure that it is locked
+ * to the input frequency and stable. This is likely to depend on the DIVR
+ * value; this is under discussion with the manufacturer.
+ *
+ * Return: the minimum amount of time the caller must wait for the PLL
+ * to lock (in microseconds)
+ */
+unsigned int wrpll_calc_max_lock_us(const struct wrpll_cfg *c)
+{
+ return MAX_LOCK_US;
+}
obj-$(CONFIG_HAVE_AT91_H32MX) += clk-h32mx.o
obj-$(CONFIG_HAVE_AT91_GENERATED_CLK) += clk-generated.o
obj-$(CONFIG_HAVE_AT91_I2S_MUX_CLK) += clk-i2s-mux.o
+obj-$(CONFIG_HAVE_AT91_SAM9X60_PLL) += clk-sam9x60-pll.o
obj-$(CONFIG_SOC_AT91SAM9) += at91sam9260.o at91sam9rl.o at91sam9x5.o
+obj-$(CONFIG_SOC_SAM9X60) += sam9x60.o
obj-$(CONFIG_SOC_SAMA5D4) += sama5d4.o
obj-$(CONFIG_SOC_SAMA5D2) += sama5d2.o
static u16 sam9260_plla_icpll[] = { 1, 1 };
-static struct clk_range sam9260_plla_outputs[] = {
+static const struct clk_range sam9260_plla_outputs[] = {
{ .min = 80000000, .max = 160000000 },
{ .min = 150000000, .max = 240000000 },
};
static u16 sam9260_pllb_icpll[] = { 1 };
-static struct clk_range sam9260_pllb_outputs[] = {
+static const struct clk_range sam9260_pllb_outputs[] = {
{ .min = 70000000, .max = 130000000 },
};
static u16 sam9g20_plla_icpll[] = { 0, 0, 0, 0, 1, 1, 1, 1 };
-static struct clk_range sam9g20_plla_outputs[] = {
+static const struct clk_range sam9g20_plla_outputs[] = {
{ .min = 745000000, .max = 800000000 },
{ .min = 695000000, .max = 750000000 },
{ .min = 645000000, .max = 700000000 },
static u16 sam9g20_pllb_icpll[] = { 0 };
-static struct clk_range sam9g20_pllb_outputs[] = {
+static const struct clk_range sam9g20_pllb_outputs[] = {
{ .min = 30000000, .max = 100000000 },
};
.divisors = { 1, 2, 4, 0 },
};
-static struct clk_range sam9261_plla_outputs[] = {
+static const struct clk_range sam9261_plla_outputs[] = {
{ .min = 80000000, .max = 200000000 },
{ .min = 190000000, .max = 240000000 },
};
static u16 sam9261_pllb_icpll[] = { 1 };
-static struct clk_range sam9261_pllb_outputs[] = {
+static const struct clk_range sam9261_pllb_outputs[] = {
{ .min = 70000000, .max = 130000000 },
};
.divisors = { 1, 2, 4, 0 },
};
-static struct clk_range sam9263_pll_outputs[] = {
+static const struct clk_range sam9263_pll_outputs[] = {
{ .min = 80000000, .max = 200000000 },
{ .min = 190000000, .max = 240000000 },
};
static u8 sam9rl_plla_out[] = { 0, 2 };
-static struct clk_range sam9rl_plla_outputs[] = {
+static const struct clk_range sam9rl_plla_outputs[] = {
{ .min = 80000000, .max = 200000000 },
{ .min = 190000000, .max = 240000000 },
};
static u16 plla_icpll[] = { 0, 0, 0, 0, 1, 1, 1, 1 };
-static struct clk_range plla_outputs[] = {
+static const struct clk_range plla_outputs[] = {
{ .min = 745000000, .max = 800000000 },
{ .min = 695000000, .max = 750000000 },
{ .min = 645000000, .max = 700000000 },
{ .n = "pck1", .p = "prog1", .id = 9 },
};
+static const struct clk_pcr_layout at91sam9x5_pcr_layout = {
+ .offset = 0x10c,
+ .cmd = BIT(12),
+ .pid_mask = GENMASK(5, 0),
+ .div_mask = GENMASK(17, 16),
+};
+
struct pck {
char *n;
u8 id;
for (i = 0; i < ARRAY_SIZE(at91sam9x5_periphck); i++) {
hw = at91_clk_register_sam9x5_peripheral(regmap, &pmc_pcr_lock,
+ &at91sam9x5_pcr_layout,
at91sam9x5_periphck[i].n,
"masterck",
at91sam9x5_periphck[i].id,
for (i = 0; extra_pcks[i].id; i++) {
hw = at91_clk_register_sam9x5_peripheral(regmap, &pmc_pcr_lock,
+ &at91sam9x5_pcr_layout,
extra_pcks[i].n,
"masterck",
extra_pcks[i].id,
*
*/
+#include <linux/bitfield.h>
#include <linux/clk-provider.h>
#include <linux/clkdev.h>
#include <linux/clk/at91_pmc.h>
spinlock_t *lock;
u32 id;
u32 gckdiv;
+ const struct clk_pcr_layout *layout;
u8 parent_id;
bool audio_pll_allowed;
};
__func__, gck->gckdiv, gck->parent_id);
spin_lock_irqsave(gck->lock, flags);
- regmap_write(gck->regmap, AT91_PMC_PCR,
- (gck->id & AT91_PMC_PCR_PID_MASK));
- regmap_update_bits(gck->regmap, AT91_PMC_PCR,
- AT91_PMC_PCR_GCKDIV_MASK | AT91_PMC_PCR_GCKCSS_MASK |
- AT91_PMC_PCR_CMD | AT91_PMC_PCR_GCKEN,
- AT91_PMC_PCR_GCKCSS(gck->parent_id) |
- AT91_PMC_PCR_CMD |
- AT91_PMC_PCR_GCKDIV(gck->gckdiv) |
+ regmap_write(gck->regmap, gck->layout->offset,
+ (gck->id & gck->layout->pid_mask));
+ regmap_update_bits(gck->regmap, gck->layout->offset,
+ AT91_PMC_PCR_GCKDIV_MASK | gck->layout->gckcss_mask |
+ gck->layout->cmd | AT91_PMC_PCR_GCKEN,
+ field_prep(gck->layout->gckcss_mask, gck->parent_id) |
+ gck->layout->cmd |
+ FIELD_PREP(AT91_PMC_PCR_GCKDIV_MASK, gck->gckdiv) |
AT91_PMC_PCR_GCKEN);
spin_unlock_irqrestore(gck->lock, flags);
return 0;
unsigned long flags;
spin_lock_irqsave(gck->lock, flags);
- regmap_write(gck->regmap, AT91_PMC_PCR,
- (gck->id & AT91_PMC_PCR_PID_MASK));
- regmap_update_bits(gck->regmap, AT91_PMC_PCR,
- AT91_PMC_PCR_CMD | AT91_PMC_PCR_GCKEN,
- AT91_PMC_PCR_CMD);
+ regmap_write(gck->regmap, gck->layout->offset,
+ (gck->id & gck->layout->pid_mask));
+ regmap_update_bits(gck->regmap, gck->layout->offset,
+ gck->layout->cmd | AT91_PMC_PCR_GCKEN,
+ gck->layout->cmd);
spin_unlock_irqrestore(gck->lock, flags);
}
unsigned int status;
spin_lock_irqsave(gck->lock, flags);
- regmap_write(gck->regmap, AT91_PMC_PCR,
- (gck->id & AT91_PMC_PCR_PID_MASK));
- regmap_read(gck->regmap, AT91_PMC_PCR, &status);
+ regmap_write(gck->regmap, gck->layout->offset,
+ (gck->id & gck->layout->pid_mask));
+ regmap_read(gck->regmap, gck->layout->offset, &status);
spin_unlock_irqrestore(gck->lock, flags);
return status & AT91_PMC_PCR_GCKEN ? 1 : 0;
unsigned long flags;
spin_lock_irqsave(gck->lock, flags);
- regmap_write(gck->regmap, AT91_PMC_PCR,
- (gck->id & AT91_PMC_PCR_PID_MASK));
- regmap_read(gck->regmap, AT91_PMC_PCR, &tmp);
+ regmap_write(gck->regmap, gck->layout->offset,
+ (gck->id & gck->layout->pid_mask));
+ regmap_read(gck->regmap, gck->layout->offset, &tmp);
spin_unlock_irqrestore(gck->lock, flags);
- gck->parent_id = (tmp & AT91_PMC_PCR_GCKCSS_MASK)
- >> AT91_PMC_PCR_GCKCSS_OFFSET;
- gck->gckdiv = (tmp & AT91_PMC_PCR_GCKDIV_MASK)
- >> AT91_PMC_PCR_GCKDIV_OFFSET;
+ gck->parent_id = field_get(gck->layout->gckcss_mask, tmp);
+ gck->gckdiv = FIELD_GET(AT91_PMC_PCR_GCKDIV_MASK, tmp);
}
struct clk_hw * __init
at91_clk_register_generated(struct regmap *regmap, spinlock_t *lock,
+ const struct clk_pcr_layout *layout,
const char *name, const char **parent_names,
u8 num_parents, u8 id, bool pll_audio,
const struct clk_range *range)
gck->lock = lock;
gck->range = *range;
gck->audio_pll_allowed = pll_audio;
+ gck->layout = layout;
clk_generated_startup(gck);
hw = &gck->hw;
struct regmap *regmap;
const struct clk_master_layout *layout;
const struct clk_master_characteristics *characteristics;
+ u32 mckr;
};
static inline bool clk_master_ready(struct regmap *regmap)
master->characteristics;
unsigned int mckr;
- regmap_read(master->regmap, AT91_PMC_MCKR, &mckr);
+ regmap_read(master->regmap, master->layout->offset, &mckr);
mckr &= layout->mask;
pres = (mckr >> layout->pres_shift) & MASTER_PRES_MASK;
struct clk_master *master = to_clk_master(hw);
unsigned int mckr;
- regmap_read(master->regmap, AT91_PMC_MCKR, &mckr);
+ regmap_read(master->regmap, master->layout->offset, &mckr);
return mckr & AT91_PMC_CSS;
}
return hw;
}
-
const struct clk_master_layout at91rm9200_master_layout = {
.mask = 0x31F,
.pres_shift = 2,
+ .offset = AT91_PMC_MCKR,
};
const struct clk_master_layout at91sam9x5_master_layout = {
.mask = 0x373,
.pres_shift = 4,
+ .offset = AT91_PMC_MCKR,
};
*
*/
+#include <linux/bitops.h>
#include <linux/clk-provider.h>
#include <linux/clkdev.h>
#include <linux/clk/at91_pmc.h>
#define PERIPHERAL_ID_MAX 31
#define PERIPHERAL_MASK(id) (1 << ((id) & PERIPHERAL_ID_MAX))
-#define PERIPHERAL_RSHIFT_MASK 0x3
-#define PERIPHERAL_RSHIFT(val) (((val) >> 16) & PERIPHERAL_RSHIFT_MASK)
-
#define PERIPHERAL_MAX_SHIFT 3
struct clk_peripheral {
spinlock_t *lock;
u32 id;
u32 div;
+ const struct clk_pcr_layout *layout;
bool auto_div;
};
return 0;
spin_lock_irqsave(periph->lock, flags);
- regmap_write(periph->regmap, AT91_PMC_PCR,
- (periph->id & AT91_PMC_PCR_PID_MASK));
- regmap_update_bits(periph->regmap, AT91_PMC_PCR,
- AT91_PMC_PCR_DIV_MASK | AT91_PMC_PCR_CMD |
+ regmap_write(periph->regmap, periph->layout->offset,
+ (periph->id & periph->layout->pid_mask));
+ regmap_update_bits(periph->regmap, periph->layout->offset,
+ periph->layout->div_mask | periph->layout->cmd |
AT91_PMC_PCR_EN,
- AT91_PMC_PCR_DIV(periph->div) |
- AT91_PMC_PCR_CMD |
+ field_prep(periph->layout->div_mask, periph->div) |
+ periph->layout->cmd |
AT91_PMC_PCR_EN);
spin_unlock_irqrestore(periph->lock, flags);
return;
spin_lock_irqsave(periph->lock, flags);
- regmap_write(periph->regmap, AT91_PMC_PCR,
- (periph->id & AT91_PMC_PCR_PID_MASK));
- regmap_update_bits(periph->regmap, AT91_PMC_PCR,
- AT91_PMC_PCR_EN | AT91_PMC_PCR_CMD,
- AT91_PMC_PCR_CMD);
+ regmap_write(periph->regmap, periph->layout->offset,
+ (periph->id & periph->layout->pid_mask));
+ regmap_update_bits(periph->regmap, periph->layout->offset,
+ AT91_PMC_PCR_EN | periph->layout->cmd,
+ periph->layout->cmd);
spin_unlock_irqrestore(periph->lock, flags);
}
return 1;
spin_lock_irqsave(periph->lock, flags);
- regmap_write(periph->regmap, AT91_PMC_PCR,
- (periph->id & AT91_PMC_PCR_PID_MASK));
- regmap_read(periph->regmap, AT91_PMC_PCR, &status);
+ regmap_write(periph->regmap, periph->layout->offset,
+ (periph->id & periph->layout->pid_mask));
+ regmap_read(periph->regmap, periph->layout->offset, &status);
spin_unlock_irqrestore(periph->lock, flags);
return status & AT91_PMC_PCR_EN ? 1 : 0;
return parent_rate;
spin_lock_irqsave(periph->lock, flags);
- regmap_write(periph->regmap, AT91_PMC_PCR,
- (periph->id & AT91_PMC_PCR_PID_MASK));
- regmap_read(periph->regmap, AT91_PMC_PCR, &status);
+ regmap_write(periph->regmap, periph->layout->offset,
+ (periph->id & periph->layout->pid_mask));
+ regmap_read(periph->regmap, periph->layout->offset, &status);
spin_unlock_irqrestore(periph->lock, flags);
if (status & AT91_PMC_PCR_EN) {
- periph->div = PERIPHERAL_RSHIFT(status);
+ periph->div = field_get(periph->layout->div_mask, status);
periph->auto_div = false;
} else {
clk_sam9x5_peripheral_autodiv(periph);
struct clk_hw * __init
at91_clk_register_sam9x5_peripheral(struct regmap *regmap, spinlock_t *lock,
+ const struct clk_pcr_layout *layout,
const char *name, const char *parent_name,
u32 id, const struct clk_range *range)
{
periph->div = 0;
periph->regmap = regmap;
periph->lock = lock;
- periph->auto_div = true;
+ if (layout->div_mask)
+ periph->auto_div = true;
+ periph->layout = layout;
periph->range = *range;
hw = &periph->hw;
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2019 Microchip Technology Inc.
+ *
+ */
+
+#include <linux/bitfield.h>
+#include <linux/clk-provider.h>
+#include <linux/clkdev.h>
+#include <linux/clk/at91_pmc.h>
+#include <linux/of.h>
+#include <linux/mfd/syscon.h>
+#include <linux/regmap.h>
+
+#include "pmc.h"
+
+#define PMC_PLL_CTRL0 0xc
+#define PMC_PLL_CTRL0_DIV_MSK GENMASK(7, 0)
+#define PMC_PLL_CTRL0_ENPLL BIT(28)
+#define PMC_PLL_CTRL0_ENPLLCK BIT(29)
+#define PMC_PLL_CTRL0_ENLOCK BIT(31)
+
+#define PMC_PLL_CTRL1 0x10
+#define PMC_PLL_CTRL1_FRACR_MSK GENMASK(21, 0)
+#define PMC_PLL_CTRL1_MUL_MSK GENMASK(30, 24)
+
+#define PMC_PLL_ACR 0x18
+#define PMC_PLL_ACR_DEFAULT 0x1b040010UL
+#define PMC_PLL_ACR_UTMIVR BIT(12)
+#define PMC_PLL_ACR_UTMIBG BIT(13)
+#define PMC_PLL_ACR_LOOP_FILTER_MSK GENMASK(31, 24)
+
+#define PMC_PLL_UPDT 0x1c
+#define PMC_PLL_UPDT_UPDATE BIT(8)
+
+#define PMC_PLL_ISR0 0xec
+
+#define PLL_DIV_MAX (FIELD_GET(PMC_PLL_CTRL0_DIV_MSK, UINT_MAX) + 1)
+#define UPLL_DIV 2
+#define PLL_MUL_MAX (FIELD_GET(PMC_PLL_CTRL1_MUL_MSK, UINT_MAX) + 1)
+
+#define PLL_MAX_ID 1
+
+struct sam9x60_pll {
+ struct clk_hw hw;
+ struct regmap *regmap;
+ spinlock_t *lock;
+ const struct clk_pll_characteristics *characteristics;
+ u32 frac;
+ u8 id;
+ u8 div;
+ u16 mul;
+};
+
+#define to_sam9x60_pll(hw) container_of(hw, struct sam9x60_pll, hw)
+
+static inline bool sam9x60_pll_ready(struct regmap *regmap, int id)
+{
+ unsigned int status;
+
+ regmap_read(regmap, PMC_PLL_ISR0, &status);
+
+ return !!(status & BIT(id));
+}
+
+static int sam9x60_pll_prepare(struct clk_hw *hw)
+{
+ struct sam9x60_pll *pll = to_sam9x60_pll(hw);
+ struct regmap *regmap = pll->regmap;
+ unsigned long flags;
+ u8 div;
+ u16 mul;
+ u32 val;
+
+ spin_lock_irqsave(pll->lock, flags);
+ regmap_write(regmap, PMC_PLL_UPDT, pll->id);
+
+ regmap_read(regmap, PMC_PLL_CTRL0, &val);
+ div = FIELD_GET(PMC_PLL_CTRL0_DIV_MSK, val);
+
+ regmap_read(regmap, PMC_PLL_CTRL1, &val);
+ mul = FIELD_GET(PMC_PLL_CTRL1_MUL_MSK, val);
+
+ if (sam9x60_pll_ready(regmap, pll->id) &&
+ (div == pll->div && mul == pll->mul)) {
+ spin_unlock_irqrestore(pll->lock, flags);
+ return 0;
+ }
+
+ /* Recommended value for PMC_PLL_ACR */
+ val = PMC_PLL_ACR_DEFAULT;
+ regmap_write(regmap, PMC_PLL_ACR, val);
+
+ regmap_write(regmap, PMC_PLL_CTRL1,
+ FIELD_PREP(PMC_PLL_CTRL1_MUL_MSK, pll->mul));
+
+ if (pll->characteristics->upll) {
+ /* Enable the UTMI internal bandgap */
+ val |= PMC_PLL_ACR_UTMIBG;
+ regmap_write(regmap, PMC_PLL_ACR, val);
+
+ udelay(10);
+
+ /* Enable the UTMI internal regulator */
+ val |= PMC_PLL_ACR_UTMIVR;
+ regmap_write(regmap, PMC_PLL_ACR, val);
+
+ udelay(10);
+ }
+
+ regmap_update_bits(regmap, PMC_PLL_UPDT,
+ PMC_PLL_UPDT_UPDATE, PMC_PLL_UPDT_UPDATE);
+
+ regmap_write(regmap, PMC_PLL_CTRL0,
+ PMC_PLL_CTRL0_ENLOCK | PMC_PLL_CTRL0_ENPLL |
+ PMC_PLL_CTRL0_ENPLLCK | pll->div);
+
+ regmap_update_bits(regmap, PMC_PLL_UPDT,
+ PMC_PLL_UPDT_UPDATE, PMC_PLL_UPDT_UPDATE);
+
+ while (!sam9x60_pll_ready(regmap, pll->id))
+ cpu_relax();
+
+ spin_unlock_irqrestore(pll->lock, flags);
+
+ return 0;
+}
+
+static int sam9x60_pll_is_prepared(struct clk_hw *hw)
+{
+ struct sam9x60_pll *pll = to_sam9x60_pll(hw);
+
+ return sam9x60_pll_ready(pll->regmap, pll->id);
+}
+
+static void sam9x60_pll_unprepare(struct clk_hw *hw)
+{
+ struct sam9x60_pll *pll = to_sam9x60_pll(hw);
+ unsigned long flags;
+
+ spin_lock_irqsave(pll->lock, flags);
+
+ regmap_write(pll->regmap, PMC_PLL_UPDT, pll->id);
+
+ regmap_update_bits(pll->regmap, PMC_PLL_CTRL0,
+ PMC_PLL_CTRL0_ENPLLCK, 0);
+
+ regmap_update_bits(pll->regmap, PMC_PLL_UPDT,
+ PMC_PLL_UPDT_UPDATE, PMC_PLL_UPDT_UPDATE);
+
+ regmap_update_bits(pll->regmap, PMC_PLL_CTRL0, PMC_PLL_CTRL0_ENPLL, 0);
+
+ if (pll->characteristics->upll)
+ regmap_update_bits(pll->regmap, PMC_PLL_ACR,
+ PMC_PLL_ACR_UTMIBG | PMC_PLL_ACR_UTMIVR, 0);
+
+ regmap_update_bits(pll->regmap, PMC_PLL_UPDT,
+ PMC_PLL_UPDT_UPDATE, PMC_PLL_UPDT_UPDATE);
+
+ spin_unlock_irqrestore(pll->lock, flags);
+}
+
+static unsigned long sam9x60_pll_recalc_rate(struct clk_hw *hw,
+ unsigned long parent_rate)
+{
+ struct sam9x60_pll *pll = to_sam9x60_pll(hw);
+
+ return (parent_rate * (pll->mul + 1)) / (pll->div + 1);
+}
+
+static long sam9x60_pll_get_best_div_mul(struct sam9x60_pll *pll,
+ unsigned long rate,
+ unsigned long parent_rate,
+ bool update)
+{
+ const struct clk_pll_characteristics *characteristics =
+ pll->characteristics;
+ unsigned long bestremainder = ULONG_MAX;
+ unsigned long maxdiv, mindiv, tmpdiv;
+ long bestrate = -ERANGE;
+ unsigned long bestdiv = 0;
+ unsigned long bestmul = 0;
+ unsigned long bestfrac = 0;
+
+ if (rate < characteristics->output[0].min ||
+ rate > characteristics->output[0].max)
+ return -ERANGE;
+
+ if (!pll->characteristics->upll) {
+ mindiv = parent_rate / rate;
+ if (mindiv < 2)
+ mindiv = 2;
+
+ maxdiv = DIV_ROUND_UP(parent_rate * PLL_MUL_MAX, rate);
+ if (maxdiv > PLL_DIV_MAX)
+ maxdiv = PLL_DIV_MAX;
+ } else {
+ mindiv = maxdiv = UPLL_DIV;
+ }
+
+ for (tmpdiv = mindiv; tmpdiv <= maxdiv; tmpdiv++) {
+ unsigned long remainder;
+ unsigned long tmprate;
+ unsigned long tmpmul;
+ unsigned long tmpfrac = 0;
+
+ /*
+ * Calculate the multiplier associated with the current
+ * divider that provide the closest rate to the requested one.
+ */
+ tmpmul = mult_frac(rate, tmpdiv, parent_rate);
+ tmprate = mult_frac(parent_rate, tmpmul, tmpdiv);
+ remainder = rate - tmprate;
+
+ if (remainder) {
+ tmpfrac = DIV_ROUND_CLOSEST_ULL((u64)remainder * tmpdiv * (1 << 22),
+ parent_rate);
+
+ tmprate += DIV_ROUND_CLOSEST_ULL((u64)tmpfrac * parent_rate,
+ tmpdiv * (1 << 22));
+
+ if (tmprate > rate)
+ remainder = tmprate - rate;
+ else
+ remainder = rate - tmprate;
+ }
+
+ /*
+ * Compare the remainder with the best remainder found until
+ * now and elect a new best multiplier/divider pair if the
+ * current remainder is smaller than the best one.
+ */
+ if (remainder < bestremainder) {
+ bestremainder = remainder;
+ bestdiv = tmpdiv;
+ bestmul = tmpmul;
+ bestrate = tmprate;
+ bestfrac = tmpfrac;
+ }
+
+ /* We've found a perfect match! */
+ if (!remainder)
+ break;
+ }
+
+ /* Check if bestrate is a valid output rate */
+ if (bestrate < characteristics->output[0].min &&
+ bestrate > characteristics->output[0].max)
+ return -ERANGE;
+
+ if (update) {
+ pll->div = bestdiv - 1;
+ pll->mul = bestmul - 1;
+ pll->frac = bestfrac;
+ }
+
+ return bestrate;
+}
+
+static long sam9x60_pll_round_rate(struct clk_hw *hw, unsigned long rate,
+ unsigned long *parent_rate)
+{
+ struct sam9x60_pll *pll = to_sam9x60_pll(hw);
+
+ return sam9x60_pll_get_best_div_mul(pll, rate, *parent_rate, false);
+}
+
+static int sam9x60_pll_set_rate(struct clk_hw *hw, unsigned long rate,
+ unsigned long parent_rate)
+{
+ struct sam9x60_pll *pll = to_sam9x60_pll(hw);
+
+ return sam9x60_pll_get_best_div_mul(pll, rate, parent_rate, true);
+}
+
+static const struct clk_ops pll_ops = {
+ .prepare = sam9x60_pll_prepare,
+ .unprepare = sam9x60_pll_unprepare,
+ .is_prepared = sam9x60_pll_is_prepared,
+ .recalc_rate = sam9x60_pll_recalc_rate,
+ .round_rate = sam9x60_pll_round_rate,
+ .set_rate = sam9x60_pll_set_rate,
+};
+
+struct clk_hw * __init
+sam9x60_clk_register_pll(struct regmap *regmap, spinlock_t *lock,
+ const char *name, const char *parent_name, u8 id,
+ const struct clk_pll_characteristics *characteristics)
+{
+ struct sam9x60_pll *pll;
+ struct clk_hw *hw;
+ struct clk_init_data init;
+ unsigned int pllr;
+ int ret;
+
+ if (id > PLL_MAX_ID)
+ return ERR_PTR(-EINVAL);
+
+ pll = kzalloc(sizeof(*pll), GFP_KERNEL);
+ if (!pll)
+ return ERR_PTR(-ENOMEM);
+
+ init.name = name;
+ init.ops = &pll_ops;
+ init.parent_names = &parent_name;
+ init.num_parents = 1;
+ init.flags = CLK_SET_RATE_GATE;
+
+ pll->id = id;
+ pll->hw.init = &init;
+ pll->characteristics = characteristics;
+ pll->regmap = regmap;
+ pll->lock = lock;
+
+ regmap_write(regmap, PMC_PLL_UPDT, id);
+ regmap_read(regmap, PMC_PLL_CTRL0, &pllr);
+ pll->div = FIELD_GET(PMC_PLL_CTRL0_DIV_MSK, pllr);
+ regmap_read(regmap, PMC_PLL_CTRL1, &pllr);
+ pll->mul = FIELD_GET(PMC_PLL_CTRL1_MUL_MSK, pllr);
+
+ hw = &pll->hw;
+ ret = clk_hw_register(NULL, hw);
+ if (ret) {
+ kfree(pll);
+ hw = ERR_PTR(ret);
+ }
+
+ return hw;
+}
+
#define RM9200_USB_DIV_SHIFT 28
#define RM9200_USB_DIV_TAB_SIZE 4
+#define SAM9X5_USBS_MASK GENMASK(0, 0)
+#define SAM9X60_USBS_MASK GENMASK(1, 0)
+
struct at91sam9x5_clk_usb {
struct clk_hw hw;
struct regmap *regmap;
+ u32 usbs_mask;
};
#define to_at91sam9x5_clk_usb(hw) \
if (index > 1)
return -EINVAL;
- regmap_update_bits(usb->regmap, AT91_PMC_USB, AT91_PMC_USBS,
- index ? AT91_PMC_USBS : 0);
+ regmap_update_bits(usb->regmap, AT91_PMC_USB, usb->usbs_mask, index);
return 0;
}
regmap_read(usb->regmap, AT91_PMC_USB, &usbr);
- return usbr & AT91_PMC_USBS;
+ return usbr & usb->usbs_mask;
}
static int at91sam9x5_clk_usb_set_rate(struct clk_hw *hw, unsigned long rate,
.set_rate = at91sam9x5_clk_usb_set_rate,
};
-struct clk_hw * __init
-at91sam9x5_clk_register_usb(struct regmap *regmap, const char *name,
- const char **parent_names, u8 num_parents)
+static struct clk_hw * __init
+_at91sam9x5_clk_register_usb(struct regmap *regmap, const char *name,
+ const char **parent_names, u8 num_parents,
+ u32 usbs_mask)
{
struct at91sam9x5_clk_usb *usb;
struct clk_hw *hw;
usb->hw.init = &init;
usb->regmap = regmap;
+ usb->usbs_mask = SAM9X5_USBS_MASK;
hw = &usb->hw;
ret = clk_hw_register(NULL, &usb->hw);
return hw;
}
+struct clk_hw * __init
+at91sam9x5_clk_register_usb(struct regmap *regmap, const char *name,
+ const char **parent_names, u8 num_parents)
+{
+ return _at91sam9x5_clk_register_usb(regmap, name, parent_names,
+ num_parents, SAM9X5_USBS_MASK);
+}
+
+struct clk_hw * __init
+sam9x60_clk_register_usb(struct regmap *regmap, const char *name,
+ const char **parent_names, u8 num_parents)
+{
+ return _at91sam9x5_clk_register_usb(regmap, name, parent_names,
+ num_parents, SAM9X60_USBS_MASK);
+}
+
struct clk_hw * __init
at91sam9n12_clk_register_usb(struct regmap *regmap, const char *name,
const char *parent_name)
of_sama5d2_clk_audio_pll_pmc_setup);
#endif /* CONFIG_HAVE_AT91_AUDIO_PLL */
+static const struct clk_pcr_layout dt_pcr_layout = {
+ .offset = 0x10c,
+ .cmd = BIT(12),
+ .pid_mask = GENMASK(5, 0),
+ .div_mask = GENMASK(17, 16),
+ .gckcss_mask = GENMASK(10, 8),
+};
+
#ifdef CONFIG_HAVE_AT91_GENERATED_CLK
#define GENERATED_SOURCE_MAX 6
id == GCK_ID_CLASSD))
pll_audio = true;
- hw = at91_clk_register_generated(regmap, &pmc_pcr_lock, name,
+ hw = at91_clk_register_generated(regmap, &pmc_pcr_lock,
+ &dt_pcr_layout, name,
parent_names, num_parents,
id, pll_audio, &range);
if (IS_ERR(hw))
hw = at91_clk_register_sam9x5_peripheral(regmap,
&pmc_pcr_lock,
+ &dt_pcr_layout,
name,
parent_name,
id, &range);
#define CLK_RANGE(MIN, MAX) {.min = MIN, .max = MAX,}
struct clk_master_layout {
+ u32 offset;
u32 mask;
u8 pres_shift;
};
struct clk_pll_characteristics {
struct clk_range input;
int num_output;
- struct clk_range *output;
+ const struct clk_range *output;
u16 *icpll;
u8 *out;
+ u8 upll : 1;
};
struct clk_programmable_layout {
extern const struct clk_programmable_layout at91sam9g45_programmable_layout;
extern const struct clk_programmable_layout at91sam9x5_programmable_layout;
+struct clk_pcr_layout {
+ u32 offset;
+ u32 cmd;
+ u32 div_mask;
+ u32 gckcss_mask;
+ u32 pid_mask;
+};
+
+#define field_get(_mask, _reg) (((_reg) & (_mask)) >> (ffs(_mask) - 1))
+#define field_prep(_mask, _val) (((_val) << (ffs(_mask) - 1)) & (_mask))
+
#define ndck(a, s) (a[s - 1].id + 1)
#define nck(a) (a[ARRAY_SIZE(a) - 1].id + 1)
struct pmc_data *pmc_data_allocate(unsigned int ncore, unsigned int nsystem,
struct clk_hw * __init
at91_clk_register_generated(struct regmap *regmap, spinlock_t *lock,
+ const struct clk_pcr_layout *layout,
const char *name, const char **parent_names,
u8 num_parents, u8 id, bool pll_audio,
const struct clk_range *range);
const char *parent_name, u32 id);
struct clk_hw * __init
at91_clk_register_sam9x5_peripheral(struct regmap *regmap, spinlock_t *lock,
+ const struct clk_pcr_layout *layout,
const char *name, const char *parent_name,
u32 id, const struct clk_range *range);
at91_clk_register_plldiv(struct regmap *regmap, const char *name,
const char *parent_name);
+struct clk_hw * __init
+sam9x60_clk_register_pll(struct regmap *regmap, spinlock_t *lock,
+ const char *name, const char *parent_name, u8 id,
+ const struct clk_pll_characteristics *characteristics);
+
struct clk_hw * __init
at91_clk_register_programmable(struct regmap *regmap, const char *name,
const char **parent_names, u8 num_parents, u8 id,
at91sam9n12_clk_register_usb(struct regmap *regmap, const char *name,
const char *parent_name);
struct clk_hw * __init
+sam9x60_clk_register_usb(struct regmap *regmap, const char *name,
+ const char **parent_names, u8 num_parents);
+struct clk_hw * __init
at91rm9200_clk_register_usb(struct regmap *regmap, const char *name,
const char *parent_name, const u32 *divisors);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/clk-provider.h>
+#include <linux/mfd/syscon.h>
+#include <linux/slab.h>
+
+#include <dt-bindings/clock/at91.h>
+
+#include "pmc.h"
+
+static DEFINE_SPINLOCK(pmc_pll_lock);
+
+static const struct clk_master_characteristics mck_characteristics = {
+ .output = { .min = 140000000, .max = 200000000 },
+ .divisors = { 1, 2, 4, 3 },
+ .have_div3_pres = 1,
+};
+
+static const struct clk_master_layout sam9x60_master_layout = {
+ .mask = 0x373,
+ .pres_shift = 4,
+ .offset = 0x28,
+};
+
+static const struct clk_range plla_outputs[] = {
+ { .min = 300000000, .max = 600000000 },
+};
+
+static const struct clk_pll_characteristics plla_characteristics = {
+ .input = { .min = 12000000, .max = 48000000 },
+ .num_output = ARRAY_SIZE(plla_outputs),
+ .output = plla_outputs,
+};
+
+static const struct clk_range upll_outputs[] = {
+ { .min = 300000000, .max = 500000000 },
+};
+
+static const struct clk_pll_characteristics upll_characteristics = {
+ .input = { .min = 12000000, .max = 48000000 },
+ .num_output = ARRAY_SIZE(upll_outputs),
+ .output = upll_outputs,
+ .upll = true,
+};
+
+static const struct clk_programmable_layout sam9x60_programmable_layout = {
+ .pres_shift = 8,
+ .css_mask = 0x1f,
+ .have_slck_mck = 0,
+};
+
+static const struct clk_pcr_layout sam9x60_pcr_layout = {
+ .offset = 0x88,
+ .cmd = BIT(31),
+ .gckcss_mask = GENMASK(12, 8),
+ .pid_mask = GENMASK(6, 0),
+};
+
+static const struct {
+ char *n;
+ char *p;
+ u8 id;
+} sam9x60_systemck[] = {
+ { .n = "ddrck", .p = "masterck", .id = 2 },
+ { .n = "uhpck", .p = "usbck", .id = 6 },
+ { .n = "pck0", .p = "prog0", .id = 8 },
+ { .n = "pck1", .p = "prog1", .id = 9 },
+ { .n = "qspick", .p = "masterck", .id = 19 },
+};
+
+static const struct {
+ char *n;
+ u8 id;
+} sam9x60_periphck[] = {
+ { .n = "pioA_clk", .id = 2, },
+ { .n = "pioB_clk", .id = 3, },
+ { .n = "pioC_clk", .id = 4, },
+ { .n = "flex0_clk", .id = 5, },
+ { .n = "flex1_clk", .id = 6, },
+ { .n = "flex2_clk", .id = 7, },
+ { .n = "flex3_clk", .id = 8, },
+ { .n = "flex6_clk", .id = 9, },
+ { .n = "flex7_clk", .id = 10, },
+ { .n = "flex8_clk", .id = 11, },
+ { .n = "sdmmc0_clk", .id = 12, },
+ { .n = "flex4_clk", .id = 13, },
+ { .n = "flex5_clk", .id = 14, },
+ { .n = "flex9_clk", .id = 15, },
+ { .n = "flex10_clk", .id = 16, },
+ { .n = "tcb0_clk", .id = 17, },
+ { .n = "pwm_clk", .id = 18, },
+ { .n = "adc_clk", .id = 19, },
+ { .n = "dma0_clk", .id = 20, },
+ { .n = "matrix_clk", .id = 21, },
+ { .n = "uhphs_clk", .id = 22, },
+ { .n = "udphs_clk", .id = 23, },
+ { .n = "macb0_clk", .id = 24, },
+ { .n = "lcd_clk", .id = 25, },
+ { .n = "sdmmc1_clk", .id = 26, },
+ { .n = "macb1_clk", .id = 27, },
+ { .n = "ssc_clk", .id = 28, },
+ { .n = "can0_clk", .id = 29, },
+ { .n = "can1_clk", .id = 30, },
+ { .n = "flex11_clk", .id = 32, },
+ { .n = "flex12_clk", .id = 33, },
+ { .n = "i2s_clk", .id = 34, },
+ { .n = "qspi_clk", .id = 35, },
+ { .n = "gfx2d_clk", .id = 36, },
+ { .n = "pit64b_clk", .id = 37, },
+ { .n = "trng_clk", .id = 38, },
+ { .n = "aes_clk", .id = 39, },
+ { .n = "tdes_clk", .id = 40, },
+ { .n = "sha_clk", .id = 41, },
+ { .n = "classd_clk", .id = 42, },
+ { .n = "isi_clk", .id = 43, },
+ { .n = "pioD_clk", .id = 44, },
+ { .n = "tcb1_clk", .id = 45, },
+ { .n = "dbgu_clk", .id = 47, },
+ { .n = "mpddr_clk", .id = 49, },
+};
+
+static const struct {
+ char *n;
+ u8 id;
+ struct clk_range r;
+ bool pll;
+} sam9x60_gck[] = {
+ { .n = "flex0_gclk", .id = 5, },
+ { .n = "flex1_gclk", .id = 6, },
+ { .n = "flex2_gclk", .id = 7, },
+ { .n = "flex3_gclk", .id = 8, },
+ { .n = "flex6_gclk", .id = 9, },
+ { .n = "flex7_gclk", .id = 10, },
+ { .n = "flex8_gclk", .id = 11, },
+ { .n = "sdmmc0_gclk", .id = 12, .r = { .min = 0, .max = 105000000 }, },
+ { .n = "flex4_gclk", .id = 13, },
+ { .n = "flex5_gclk", .id = 14, },
+ { .n = "flex9_gclk", .id = 15, },
+ { .n = "flex10_gclk", .id = 16, },
+ { .n = "tcb0_gclk", .id = 17, },
+ { .n = "adc_gclk", .id = 19, },
+ { .n = "lcd_gclk", .id = 25, .r = { .min = 0, .max = 140000000 }, },
+ { .n = "sdmmc1_gclk", .id = 26, .r = { .min = 0, .max = 105000000 }, },
+ { .n = "flex11_gclk", .id = 32, },
+ { .n = "flex12_gclk", .id = 33, },
+ { .n = "i2s_gclk", .id = 34, .r = { .min = 0, .max = 105000000 },
+ .pll = true, },
+ { .n = "pit64b_gclk", .id = 37, },
+ { .n = "classd_gclk", .id = 42, .r = { .min = 0, .max = 100000000 },
+ .pll = true, },
+ { .n = "tcb1_gclk", .id = 45, },
+ { .n = "dbgu_gclk", .id = 47, },
+};
+
+static void __init sam9x60_pmc_setup(struct device_node *np)
+{
+ struct clk_range range = CLK_RANGE(0, 0);
+ const char *td_slck_name, *md_slck_name, *mainxtal_name;
+ struct pmc_data *sam9x60_pmc;
+ const char *parent_names[6];
+ struct regmap *regmap;
+ struct clk_hw *hw;
+ int i;
+ bool bypass;
+
+ i = of_property_match_string(np, "clock-names", "td_slck");
+ if (i < 0)
+ return;
+
+ td_slck_name = of_clk_get_parent_name(np, i);
+
+ i = of_property_match_string(np, "clock-names", "md_slck");
+ if (i < 0)
+ return;
+
+ md_slck_name = of_clk_get_parent_name(np, i);
+
+ i = of_property_match_string(np, "clock-names", "main_xtal");
+ if (i < 0)
+ return;
+ mainxtal_name = of_clk_get_parent_name(np, i);
+
+ regmap = syscon_node_to_regmap(np);
+ if (IS_ERR(regmap))
+ return;
+
+ sam9x60_pmc = pmc_data_allocate(PMC_MAIN + 1,
+ nck(sam9x60_systemck),
+ nck(sam9x60_periphck),
+ nck(sam9x60_gck));
+ if (!sam9x60_pmc)
+ return;
+
+ hw = at91_clk_register_main_rc_osc(regmap, "main_rc_osc", 24000000,
+ 50000000);
+ if (IS_ERR(hw))
+ goto err_free;
+
+ bypass = of_property_read_bool(np, "atmel,osc-bypass");
+
+ hw = at91_clk_register_main_osc(regmap, "main_osc", mainxtal_name,
+ bypass);
+ if (IS_ERR(hw))
+ goto err_free;
+
+ parent_names[0] = "main_rc_osc";
+ parent_names[1] = "main_osc";
+ hw = at91_clk_register_sam9x5_main(regmap, "mainck", parent_names, 2);
+ if (IS_ERR(hw))
+ goto err_free;
+
+ sam9x60_pmc->chws[PMC_MAIN] = hw;
+
+ hw = sam9x60_clk_register_pll(regmap, &pmc_pll_lock, "pllack",
+ "mainck", 0, &plla_characteristics);
+ if (IS_ERR(hw))
+ goto err_free;
+
+ hw = sam9x60_clk_register_pll(regmap, &pmc_pll_lock, "upllck",
+ "main_osc", 1, &upll_characteristics);
+ if (IS_ERR(hw))
+ goto err_free;
+
+ sam9x60_pmc->chws[PMC_UTMI] = hw;
+
+ parent_names[0] = md_slck_name;
+ parent_names[1] = "mainck";
+ parent_names[2] = "pllack";
+ hw = at91_clk_register_master(regmap, "masterck", 3, parent_names,
+ &sam9x60_master_layout,
+ &mck_characteristics);
+ if (IS_ERR(hw))
+ goto err_free;
+
+ sam9x60_pmc->chws[PMC_MCK] = hw;
+
+ parent_names[0] = "pllack";
+ parent_names[1] = "upllck";
+ parent_names[2] = "mainck";
+ parent_names[3] = "mainck";
+ hw = sam9x60_clk_register_usb(regmap, "usbck", parent_names, 4);
+ if (IS_ERR(hw))
+ goto err_free;
+
+ parent_names[0] = md_slck_name;
+ parent_names[1] = td_slck_name;
+ parent_names[2] = "mainck";
+ parent_names[3] = "masterck";
+ parent_names[4] = "pllack";
+ parent_names[5] = "upllck";
+ for (i = 0; i < 8; i++) {
+ char name[6];
+
+ snprintf(name, sizeof(name), "prog%d", i);
+
+ hw = at91_clk_register_programmable(regmap, name,
+ parent_names, 6, i,
+ &sam9x60_programmable_layout);
+ if (IS_ERR(hw))
+ goto err_free;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(sam9x60_systemck); i++) {
+ hw = at91_clk_register_system(regmap, sam9x60_systemck[i].n,
+ sam9x60_systemck[i].p,
+ sam9x60_systemck[i].id);
+ if (IS_ERR(hw))
+ goto err_free;
+
+ sam9x60_pmc->shws[sam9x60_systemck[i].id] = hw;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(sam9x60_periphck); i++) {
+ hw = at91_clk_register_sam9x5_peripheral(regmap, &pmc_pcr_lock,
+ &sam9x60_pcr_layout,
+ sam9x60_periphck[i].n,
+ "masterck",
+ sam9x60_periphck[i].id,
+ &range);
+ if (IS_ERR(hw))
+ goto err_free;
+
+ sam9x60_pmc->phws[sam9x60_periphck[i].id] = hw;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(sam9x60_gck); i++) {
+ hw = at91_clk_register_generated(regmap, &pmc_pcr_lock,
+ &sam9x60_pcr_layout,
+ sam9x60_gck[i].n,
+ parent_names, 6,
+ sam9x60_gck[i].id,
+ sam9x60_gck[i].pll,
+ &sam9x60_gck[i].r);
+ if (IS_ERR(hw))
+ goto err_free;
+
+ sam9x60_pmc->ghws[sam9x60_gck[i].id] = hw;
+ }
+
+ of_clk_add_hw_provider(np, of_clk_hw_pmc_get, sam9x60_pmc);
+
+ return;
+
+err_free:
+ pmc_data_free(sam9x60_pmc);
+}
+/* Some clks are used for a clocksource */
+CLK_OF_DECLARE(sam9x60_pmc, "microchip,sam9x60-pmc", sam9x60_pmc_setup);
static u16 plla_icpll[] = { 0 };
-static struct clk_range plla_outputs[] = {
+static const struct clk_range plla_outputs[] = {
{ .min = 600000000, .max = 1200000000 },
};
.out = plla_out,
};
+static const struct clk_pcr_layout sama5d2_pcr_layout = {
+ .offset = 0x10c,
+ .cmd = BIT(12),
+ .gckcss_mask = GENMASK(10, 8),
+ .pid_mask = GENMASK(6, 0),
+};
+
static const struct {
char *n;
char *p;
for (i = 0; i < ARRAY_SIZE(sama5d2_periphck); i++) {
hw = at91_clk_register_sam9x5_peripheral(regmap, &pmc_pcr_lock,
+ &sama5d2_pcr_layout,
sama5d2_periphck[i].n,
"masterck",
sama5d2_periphck[i].id,
for (i = 0; i < ARRAY_SIZE(sama5d2_periph32ck); i++) {
hw = at91_clk_register_sam9x5_peripheral(regmap, &pmc_pcr_lock,
+ &sama5d2_pcr_layout,
sama5d2_periph32ck[i].n,
"h32mxck",
sama5d2_periph32ck[i].id,
parent_names[5] = "audiopll_pmcck";
for (i = 0; i < ARRAY_SIZE(sama5d2_gck); i++) {
hw = at91_clk_register_generated(regmap, &pmc_pcr_lock,
+ &sama5d2_pcr_layout,
sama5d2_gck[i].n,
parent_names, 6,
sama5d2_gck[i].id,
static u16 plla_icpll[] = { 0 };
-static struct clk_range plla_outputs[] = {
+static const struct clk_range plla_outputs[] = {
{ .min = 600000000, .max = 1200000000 },
};
.out = plla_out,
};
+static const struct clk_pcr_layout sama5d4_pcr_layout = {
+ .offset = 0x10c,
+ .cmd = BIT(12),
+ .pid_mask = GENMASK(6, 0),
+};
+
static const struct {
char *n;
char *p;
for (i = 0; i < ARRAY_SIZE(sama5d4_periphck); i++) {
hw = at91_clk_register_sam9x5_peripheral(regmap, &pmc_pcr_lock,
+ &sama5d4_pcr_layout,
sama5d4_periphck[i].n,
"masterck",
sama5d4_periphck[i].id,
for (i = 0; i < ARRAY_SIZE(sama5d4_periph32ck); i++) {
hw = at91_clk_register_sam9x5_peripheral(regmap, &pmc_pcr_lock,
+ &sama5d4_pcr_layout,
sama5d4_periph32ck[i].n,
"h32mxck",
sama5d4_periph32ck[i].id,
return hw;
}
-static void __init
-of_at91sam9x5_clk_slow_osc_setup(struct device_node *np, void __iomem *sckcr)
-{
- struct clk_hw *hw;
- const char *parent_name;
- const char *name = np->name;
- u32 startup;
- bool bypass;
-
- parent_name = of_clk_get_parent_name(np, 0);
- of_property_read_string(np, "clock-output-names", &name);
- of_property_read_u32(np, "atmel,startup-time-usec", &startup);
- bypass = of_property_read_bool(np, "atmel,osc-bypass");
-
- hw = at91_clk_register_slow_osc(sckcr, name, parent_name, startup,
- bypass);
- if (IS_ERR(hw))
- return;
-
- of_clk_add_hw_provider(np, of_clk_hw_simple_get, hw);
-}
-
static unsigned long clk_slow_rc_osc_recalc_rate(struct clk_hw *hw,
unsigned long parent_rate)
{
return hw;
}
-static void __init
-of_at91sam9x5_clk_slow_rc_osc_setup(struct device_node *np, void __iomem *sckcr)
-{
- struct clk_hw *hw;
- u32 frequency = 0;
- u32 accuracy = 0;
- u32 startup = 0;
- const char *name = np->name;
-
- of_property_read_string(np, "clock-output-names", &name);
- of_property_read_u32(np, "clock-frequency", &frequency);
- of_property_read_u32(np, "clock-accuracy", &accuracy);
- of_property_read_u32(np, "atmel,startup-time-usec", &startup);
-
- hw = at91_clk_register_slow_rc_osc(sckcr, name, frequency, accuracy,
- startup);
- if (IS_ERR(hw))
- return;
-
- of_clk_add_hw_provider(np, of_clk_hw_simple_get, hw);
-}
-
static int clk_sam9x5_slow_set_parent(struct clk_hw *hw, u8 index)
{
struct clk_sam9x5_slow *slowck = to_clk_sam9x5_slow(hw);
return hw;
}
-static void __init
-of_at91sam9x5_clk_slow_setup(struct device_node *np, void __iomem *sckcr)
+static void __init at91sam9x5_sckc_register(struct device_node *np,
+ unsigned int rc_osc_startup_us)
{
+ const char *parent_names[2] = { "slow_rc_osc", "slow_osc" };
+ void __iomem *regbase = of_iomap(np, 0);
+ struct device_node *child = NULL;
+ const char *xtal_name;
struct clk_hw *hw;
- const char *parent_names[2];
- unsigned int num_parents;
- const char *name = np->name;
+ bool bypass;
- num_parents = of_clk_get_parent_count(np);
- if (num_parents == 0 || num_parents > 2)
+ if (!regbase)
+ return;
+
+ hw = at91_clk_register_slow_rc_osc(regbase, parent_names[0], 32768,
+ 50000000, rc_osc_startup_us);
+ if (IS_ERR(hw))
return;
- of_clk_parent_fill(np, parent_names, num_parents);
+ xtal_name = of_clk_get_parent_name(np, 0);
+ if (!xtal_name) {
+ /* DT backward compatibility */
+ child = of_get_compatible_child(np, "atmel,at91sam9x5-clk-slow-osc");
+ if (!child)
+ return;
+
+ xtal_name = of_clk_get_parent_name(child, 0);
+ bypass = of_property_read_bool(child, "atmel,osc-bypass");
+
+ child = of_get_compatible_child(np, "atmel,at91sam9x5-clk-slow");
+ } else {
+ bypass = of_property_read_bool(np, "atmel,osc-bypass");
+ }
+
+ if (!xtal_name)
+ return;
- of_property_read_string(np, "clock-output-names", &name);
+ hw = at91_clk_register_slow_osc(regbase, parent_names[1], xtal_name,
+ 1200000, bypass);
+ if (IS_ERR(hw))
+ return;
- hw = at91_clk_register_sam9x5_slow(sckcr, name, parent_names,
- num_parents);
+ hw = at91_clk_register_sam9x5_slow(regbase, "slowck", parent_names, 2);
if (IS_ERR(hw))
return;
of_clk_add_hw_provider(np, of_clk_hw_simple_get, hw);
-}
-static const struct of_device_id sckc_clk_ids[] __initconst = {
- /* Slow clock */
- {
- .compatible = "atmel,at91sam9x5-clk-slow-osc",
- .data = of_at91sam9x5_clk_slow_osc_setup,
- },
- {
- .compatible = "atmel,at91sam9x5-clk-slow-rc-osc",
- .data = of_at91sam9x5_clk_slow_rc_osc_setup,
- },
- {
- .compatible = "atmel,at91sam9x5-clk-slow",
- .data = of_at91sam9x5_clk_slow_setup,
- },
- { /*sentinel*/ }
-};
+ /* DT backward compatibility */
+ if (child)
+ of_clk_add_hw_provider(child, of_clk_hw_simple_get, hw);
+}
static void __init of_at91sam9x5_sckc_setup(struct device_node *np)
{
- struct device_node *childnp;
- void (*clk_setup)(struct device_node *, void __iomem *);
- const struct of_device_id *clk_id;
- void __iomem *regbase = of_iomap(np, 0);
-
- if (!regbase)
- return;
-
- for_each_child_of_node(np, childnp) {
- clk_id = of_match_node(sckc_clk_ids, childnp);
- if (!clk_id)
- continue;
- clk_setup = clk_id->data;
- clk_setup(childnp, regbase);
- }
+ at91sam9x5_sckc_register(np, 75);
}
CLK_OF_DECLARE(at91sam9x5_clk_sckc, "atmel,at91sam9x5-sckc",
of_at91sam9x5_sckc_setup);
+static void __init of_sama5d3_sckc_setup(struct device_node *np)
+{
+ at91sam9x5_sckc_register(np, 500);
+}
+CLK_OF_DECLARE(sama5d3_clk_sckc, "atmel,sama5d3-sckc",
+ of_sama5d3_sckc_setup);
+
static int clk_sama5d4_slow_osc_prepare(struct clk_hw *hw)
{
struct clk_sama5d4_slow_osc *osc = to_clk_sama5d4_slow_osc(hw);
/* TODO: ask Aspeed about the actual parent data */
static const struct aspeed_gate_data aspeed_gates[] = {
/* clk rst name parent flags */
- [ASPEED_CLK_GATE_ECLK] = { 0, -1, "eclk-gate", "eclk", 0 }, /* Video Engine */
+ [ASPEED_CLK_GATE_ECLK] = { 0, 6, "eclk-gate", "eclk", 0 }, /* Video Engine */
[ASPEED_CLK_GATE_GCLK] = { 1, 7, "gclk-gate", NULL, 0 }, /* 2D engine */
[ASPEED_CLK_GATE_MCLK] = { 2, -1, "mclk-gate", "mpll", CLK_IS_CRITICAL }, /* SDRAM */
- [ASPEED_CLK_GATE_VCLK] = { 3, 6, "vclk-gate", NULL, 0 }, /* Video Capture */
+ [ASPEED_CLK_GATE_VCLK] = { 3, -1, "vclk-gate", NULL, 0 }, /* Video Capture */
[ASPEED_CLK_GATE_BCLK] = { 4, 8, "bclk-gate", "bclk", CLK_IS_CRITICAL }, /* PCIe/PCI */
[ASPEED_CLK_GATE_DCLK] = { 5, -1, "dclk-gate", NULL, CLK_IS_CRITICAL }, /* DAC */
[ASPEED_CLK_GATE_REFCLK] = { 6, -1, "refclk-gate", "clkin", CLK_IS_CRITICAL },
[ASPEED_CLK_GATE_LHCCLK] = { 28, -1, "lhclk-gate", "lhclk", 0 }, /* LPC master/LPC+ */
};
+static const char * const eclk_parent_names[] = {
+ "mpll",
+ "hpll",
+ "dpll",
+};
+
+static const struct clk_div_table ast2500_eclk_div_table[] = {
+ { 0x0, 2 },
+ { 0x1, 2 },
+ { 0x2, 3 },
+ { 0x3, 4 },
+ { 0x4, 5 },
+ { 0x5, 6 },
+ { 0x6, 7 },
+ { 0x7, 8 },
+ { 0 }
+};
+
static const struct clk_div_table ast2500_mac_div_table[] = {
{ 0x0, 4 }, /* Yep, really. Aspeed confirmed this is correct */
{ 0x1, 4 },
struct aspeed_clk_soc_data {
const struct clk_div_table *div_table;
+ const struct clk_div_table *eclk_div_table;
const struct clk_div_table *mac_div_table;
struct clk_hw *(*calc_pll)(const char *name, u32 val);
};
static const struct aspeed_clk_soc_data ast2500_data = {
.div_table = ast2500_div_table,
+ .eclk_div_table = ast2500_eclk_div_table,
.mac_div_table = ast2500_mac_div_table,
.calc_pll = aspeed_ast2500_calc_pll,
};
static const struct aspeed_clk_soc_data ast2400_data = {
.div_table = ast2400_div_table,
+ .eclk_div_table = ast2400_div_table,
.mac_div_table = ast2400_div_table,
.calc_pll = aspeed_ast2400_calc_pll,
};
return PTR_ERR(hw);
aspeed_clk_data->hws[ASPEED_CLK_24M] = hw;
+ hw = clk_hw_register_mux(dev, "eclk-mux", eclk_parent_names,
+ ARRAY_SIZE(eclk_parent_names), 0,
+ scu_base + ASPEED_CLK_SELECTION, 2, 0x3, 0,
+ &aspeed_clk_lock);
+ if (IS_ERR(hw))
+ return PTR_ERR(hw);
+ aspeed_clk_data->hws[ASPEED_CLK_ECLK_MUX] = hw;
+
+ hw = clk_hw_register_divider_table(dev, "eclk", "eclk-mux", 0,
+ scu_base + ASPEED_CLK_SELECTION, 28,
+ 3, 0, soc_data->eclk_div_table,
+ &aspeed_clk_lock);
+ if (IS_ERR(hw))
+ return PTR_ERR(hw);
+ aspeed_clk_data->hws[ASPEED_CLK_ECLK] = hw;
+
/*
* TODO: There are a number of clocks that not included in this driver
* as more information is required:
* RGMII
* RMII
* UART[1..5] clock source mux
- * Video Engine (ECLK) mux and clock divider
*/
for (i = 0; i < ARRAY_SIZE(aspeed_gates); i++) {
return ERR_PTR(-ENOMEM);
init.name = name;
- init.flags = flags | CLK_IS_BASIC;
+ init.flags = flags;
init.parent_names = parent_names;
init.num_parents = num_parents;
hw = &composite->hw;
* parent - fixed parent. No clk_set_parent support
*/
+static inline u32 clk_div_readl(struct clk_divider *divider)
+{
+ if (divider->flags & CLK_DIVIDER_BIG_ENDIAN)
+ return ioread32be(divider->reg);
+
+ return readl(divider->reg);
+}
+
+static inline void clk_div_writel(struct clk_divider *divider, u32 val)
+{
+ if (divider->flags & CLK_DIVIDER_BIG_ENDIAN)
+ iowrite32be(val, divider->reg);
+ else
+ writel(val, divider->reg);
+}
+
static unsigned int _get_table_maxdiv(const struct clk_div_table *table,
u8 width)
{
struct clk_divider *divider = to_clk_divider(hw);
unsigned int val;
- val = clk_readl(divider->reg) >> divider->shift;
+ val = clk_div_readl(divider) >> divider->shift;
val &= clk_div_mask(divider->width);
return divider_recalc_rate(hw, parent_rate, val, divider->table,
if (divider->flags & CLK_DIVIDER_READ_ONLY) {
u32 val;
- val = clk_readl(divider->reg) >> divider->shift;
+ val = clk_div_readl(divider) >> divider->shift;
val &= clk_div_mask(divider->width);
return divider_ro_round_rate(hw, rate, prate, divider->table,
if (divider->flags & CLK_DIVIDER_HIWORD_MASK) {
val = clk_div_mask(divider->width) << (divider->shift + 16);
} else {
- val = clk_readl(divider->reg);
+ val = clk_div_readl(divider);
val &= ~(clk_div_mask(divider->width) << divider->shift);
}
val |= (u32)value << divider->shift;
- clk_writel(val, divider->reg);
+ clk_div_writel(divider, val);
if (divider->lock)
spin_unlock_irqrestore(divider->lock, flags);
init.ops = &clk_divider_ro_ops;
else
init.ops = &clk_divider_ops;
- init.flags = flags | CLK_IS_BASIC;
+ init.flags = flags;
init.parent_names = (parent_name ? &parent_name: NULL);
init.num_parents = (parent_name ? 1 : 0);
};
EXPORT_SYMBOL_GPL(clk_fixed_factor_ops);
-struct clk_hw *clk_hw_register_fixed_factor(struct device *dev,
- const char *name, const char *parent_name, unsigned long flags,
- unsigned int mult, unsigned int div)
+static struct clk_hw *
+__clk_hw_register_fixed_factor(struct device *dev, struct device_node *np,
+ const char *name, const char *parent_name, int index,
+ unsigned long flags, unsigned int mult, unsigned int div)
{
struct clk_fixed_factor *fix;
- struct clk_init_data init;
+ struct clk_init_data init = { };
+ struct clk_parent_data pdata = { .index = index };
struct clk_hw *hw;
int ret;
init.name = name;
init.ops = &clk_fixed_factor_ops;
- init.flags = flags | CLK_IS_BASIC;
- init.parent_names = &parent_name;
+ init.flags = flags;
+ if (parent_name)
+ init.parent_names = &parent_name;
+ else
+ init.parent_data = &pdata;
init.num_parents = 1;
hw = &fix->hw;
- ret = clk_hw_register(dev, hw);
+ if (dev)
+ ret = clk_hw_register(dev, hw);
+ else
+ ret = of_clk_hw_register(np, hw);
if (ret) {
kfree(fix);
hw = ERR_PTR(ret);
return hw;
}
+
+struct clk_hw *clk_hw_register_fixed_factor(struct device *dev,
+ const char *name, const char *parent_name, unsigned long flags,
+ unsigned int mult, unsigned int div)
+{
+ return __clk_hw_register_fixed_factor(dev, NULL, name, parent_name, -1,
+ flags, mult, div);
+}
EXPORT_SYMBOL_GPL(clk_hw_register_fixed_factor);
struct clk *clk_register_fixed_factor(struct device *dev, const char *name,
{ /* Sentinel */ },
};
-static struct clk *_of_fixed_factor_clk_setup(struct device_node *node)
+static struct clk_hw *_of_fixed_factor_clk_setup(struct device_node *node)
{
- struct clk *clk;
+ struct clk_hw *hw;
const char *clk_name = node->name;
- const char *parent_name;
unsigned long flags = 0;
u32 div, mult;
int ret;
}
of_property_read_string(node, "clock-output-names", &clk_name);
- parent_name = of_clk_get_parent_name(node, 0);
if (of_match_node(set_rate_parent_matches, node))
flags |= CLK_SET_RATE_PARENT;
- clk = clk_register_fixed_factor(NULL, clk_name, parent_name, flags,
- mult, div);
- if (IS_ERR(clk)) {
+ hw = __clk_hw_register_fixed_factor(NULL, node, clk_name, NULL, 0,
+ flags, mult, div);
+ if (IS_ERR(hw)) {
/*
- * If parent clock is not registered, registration would fail.
* Clear OF_POPULATED flag so that clock registration can be
* attempted again from probe function.
*/
of_node_clear_flag(node, OF_POPULATED);
- return clk;
+ return ERR_CAST(hw);
}
- ret = of_clk_add_provider(node, of_clk_src_simple_get, clk);
+ ret = of_clk_add_hw_provider(node, of_clk_hw_simple_get, hw);
if (ret) {
- clk_unregister(clk);
+ clk_hw_unregister_fixed_factor(hw);
return ERR_PTR(ret);
}
- return clk;
+ return hw;
}
/**
static int of_fixed_factor_clk_remove(struct platform_device *pdev)
{
- struct clk *clk = platform_get_drvdata(pdev);
+ struct clk_hw *clk = platform_get_drvdata(pdev);
of_clk_del_provider(pdev->dev.of_node);
- clk_unregister_fixed_factor(clk);
+ clk_hw_unregister_fixed_factor(clk);
return 0;
}
static int of_fixed_factor_clk_probe(struct platform_device *pdev)
{
- struct clk *clk;
+ struct clk_hw *clk;
/*
* This function is not executed when of_fixed_factor_clk_setup
init.name = name;
init.ops = &clk_fixed_rate_ops;
- init.flags = flags | CLK_IS_BASIC;
+ init.flags = flags;
init.parent_names = (parent_name ? &parent_name: NULL);
init.num_parents = (parent_name ? 1 : 0);
#include <linux/slab.h>
#include <linux/rational.h>
+static inline u32 clk_fd_readl(struct clk_fractional_divider *fd)
+{
+ if (fd->flags & CLK_FRAC_DIVIDER_BIG_ENDIAN)
+ return ioread32be(fd->reg);
+
+ return readl(fd->reg);
+}
+
+static inline void clk_fd_writel(struct clk_fractional_divider *fd, u32 val)
+{
+ if (fd->flags & CLK_FRAC_DIVIDER_BIG_ENDIAN)
+ iowrite32be(val, fd->reg);
+ else
+ writel(val, fd->reg);
+}
+
static unsigned long clk_fd_recalc_rate(struct clk_hw *hw,
unsigned long parent_rate)
{
else
__acquire(fd->lock);
- val = clk_readl(fd->reg);
+ val = clk_fd_readl(fd);
if (fd->lock)
spin_unlock_irqrestore(fd->lock, flags);
else
__acquire(fd->lock);
- val = clk_readl(fd->reg);
+ val = clk_fd_readl(fd);
val &= ~(fd->mmask | fd->nmask);
val |= (m << fd->mshift) | (n << fd->nshift);
- clk_writel(val, fd->reg);
+ clk_fd_writel(fd, val);
if (fd->lock)
spin_unlock_irqrestore(fd->lock, flags);
init.name = name;
init.ops = &clk_fractional_divider_ops;
- init.flags = flags | CLK_IS_BASIC;
+ init.flags = flags;
init.parent_names = parent_name ? &parent_name : NULL;
init.num_parents = parent_name ? 1 : 0;
* parent - fixed parent. No clk_set_parent support
*/
+static inline u32 clk_gate_readl(struct clk_gate *gate)
+{
+ if (gate->flags & CLK_GATE_BIG_ENDIAN)
+ return ioread32be(gate->reg);
+
+ return readl(gate->reg);
+}
+
+static inline void clk_gate_writel(struct clk_gate *gate, u32 val)
+{
+ if (gate->flags & CLK_GATE_BIG_ENDIAN)
+ iowrite32be(val, gate->reg);
+ else
+ writel(val, gate->reg);
+}
+
/*
* It works on following logic:
*
if (set)
reg |= BIT(gate->bit_idx);
} else {
- reg = clk_readl(gate->reg);
+ reg = clk_gate_readl(gate);
if (set)
reg |= BIT(gate->bit_idx);
reg &= ~BIT(gate->bit_idx);
}
- clk_writel(reg, gate->reg);
+ clk_gate_writel(gate, reg);
if (gate->lock)
spin_unlock_irqrestore(gate->lock, flags);
u32 reg;
struct clk_gate *gate = to_clk_gate(hw);
- reg = clk_readl(gate->reg);
+ reg = clk_gate_readl(gate);
/* if a set bit disables this clk, flip it before masking */
if (gate->flags & CLK_GATE_SET_TO_DISABLE)
init.name = name;
init.ops = &clk_gate_ops;
- init.flags = flags | CLK_IS_BASIC;
+ init.flags = flags;
init.parent_names = parent_name ? &parent_name : NULL;
init.num_parents = parent_name ? 1 : 0;
init.name = name;
init.ops = clk_gpio_ops;
- init.flags = flags | CLK_IS_BASIC;
+ init.flags = flags;
init.parent_names = parent_names;
init.num_parents = num_parents;
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/err.h>
-#include <linux/clk.h>
#include <linux/clk-provider.h>
#include <linux/io.h>
#include <linux/of.h>
.set_rate = clk_periclk_set_rate,
};
-static __init struct clk *hb_clk_init(struct device_node *node, const struct clk_ops *ops)
+static void __init hb_clk_init(struct device_node *node, const struct clk_ops *ops, unsigned long clkflags)
{
u32 reg;
struct hb_clk *hb_clk;
rc = of_property_read_u32(node, "reg", ®);
if (WARN_ON(rc))
- return NULL;
+ return;
hb_clk = kzalloc(sizeof(*hb_clk), GFP_KERNEL);
if (WARN_ON(!hb_clk))
- return NULL;
+ return;
/* Map system registers */
srnp = of_find_compatible_node(NULL, NULL, "calxeda,hb-sregs");
init.name = clk_name;
init.ops = ops;
- init.flags = 0;
+ init.flags = clkflags;
parent_name = of_clk_get_parent_name(node, 0);
init.parent_names = &parent_name;
init.num_parents = 1;
rc = clk_hw_register(NULL, &hb_clk->hw);
if (WARN_ON(rc)) {
kfree(hb_clk);
- return NULL;
+ return;
}
- rc = of_clk_add_hw_provider(node, of_clk_hw_simple_get, &hb_clk->hw);
- return hb_clk->hw.clk;
+ of_clk_add_hw_provider(node, of_clk_hw_simple_get, &hb_clk->hw);
}
static void __init hb_pll_init(struct device_node *node)
{
- hb_clk_init(node, &clk_pll_ops);
+ hb_clk_init(node, &clk_pll_ops, 0);
}
CLK_OF_DECLARE(hb_pll, "calxeda,hb-pll-clock", hb_pll_init);
static void __init hb_a9periph_init(struct device_node *node)
{
- hb_clk_init(node, &a9periphclk_ops);
+ hb_clk_init(node, &a9periphclk_ops, 0);
}
CLK_OF_DECLARE(hb_a9periph, "calxeda,hb-a9periph-clock", hb_a9periph_init);
static void __init hb_a9bus_init(struct device_node *node)
{
- struct clk *clk = hb_clk_init(node, &a9bclk_ops);
- clk_prepare_enable(clk);
+ hb_clk_init(node, &a9bclk_ops, CLK_IS_CRITICAL);
}
CLK_OF_DECLARE(hb_a9bus, "calxeda,hb-a9bus-clock", hb_a9bus_init);
static void __init hb_emmc_init(struct device_node *node)
{
- hb_clk_init(node, &periclk_ops);
+ hb_clk_init(node, &periclk_ops, 0);
}
CLK_OF_DECLARE(hb_emmc, "calxeda,hb-emmc-clock", hb_emmc_init);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Lochnagar clock control
+ *
+ * Copyright (c) 2017-2018 Cirrus Logic, Inc. and
+ * Cirrus Logic International Semiconductor Ltd.
+ *
+ * Author: Charles Keepax <ckeepax@opensource.cirrus.com>
+ */
+
+#include <linux/clk-provider.h>
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+
+#include <linux/mfd/lochnagar.h>
+#include <linux/mfd/lochnagar1_regs.h>
+#include <linux/mfd/lochnagar2_regs.h>
+
+#include <dt-bindings/clk/lochnagar.h>
+
+#define LOCHNAGAR_NUM_CLOCKS (LOCHNAGAR_SPDIF_CLKOUT + 1)
+
+struct lochnagar_clk {
+ const char * const name;
+ struct clk_hw hw;
+
+ struct lochnagar_clk_priv *priv;
+
+ u16 cfg_reg;
+ u16 ena_mask;
+
+ u16 src_reg;
+ u16 src_mask;
+};
+
+struct lochnagar_clk_priv {
+ struct device *dev;
+ struct regmap *regmap;
+ enum lochnagar_type type;
+
+ const char **parents;
+ unsigned int nparents;
+
+ struct lochnagar_clk lclks[LOCHNAGAR_NUM_CLOCKS];
+};
+
+static const char * const lochnagar1_clk_parents[] = {
+ "ln-none",
+ "ln-spdif-mclk",
+ "ln-psia1-mclk",
+ "ln-psia2-mclk",
+ "ln-cdc-clkout",
+ "ln-dsp-clkout",
+ "ln-pmic-32k",
+ "ln-gf-mclk1",
+ "ln-gf-mclk3",
+ "ln-gf-mclk2",
+ "ln-gf-mclk4",
+};
+
+static const char * const lochnagar2_clk_parents[] = {
+ "ln-none",
+ "ln-cdc-clkout",
+ "ln-dsp-clkout",
+ "ln-pmic-32k",
+ "ln-spdif-mclk",
+ "ln-clk-12m",
+ "ln-clk-11m",
+ "ln-clk-24m",
+ "ln-clk-22m",
+ "ln-clk-8m",
+ "ln-usb-clk-24m",
+ "ln-gf-mclk1",
+ "ln-gf-mclk3",
+ "ln-gf-mclk2",
+ "ln-psia1-mclk",
+ "ln-psia2-mclk",
+ "ln-spdif-clkout",
+ "ln-adat-mclk",
+ "ln-usb-clk-12m",
+};
+
+#define LN1_CLK(ID, NAME, REG) \
+ [LOCHNAGAR_##ID] = { \
+ .name = NAME, \
+ .cfg_reg = LOCHNAGAR1_##REG, \
+ .ena_mask = LOCHNAGAR1_##ID##_ENA_MASK, \
+ .src_reg = LOCHNAGAR1_##ID##_SEL, \
+ .src_mask = LOCHNAGAR1_SRC_MASK, \
+ }
+
+#define LN2_CLK(ID, NAME) \
+ [LOCHNAGAR_##ID] = { \
+ .name = NAME, \
+ .cfg_reg = LOCHNAGAR2_##ID##_CTRL, \
+ .src_reg = LOCHNAGAR2_##ID##_CTRL, \
+ .ena_mask = LOCHNAGAR2_CLK_ENA_MASK, \
+ .src_mask = LOCHNAGAR2_CLK_SRC_MASK, \
+ }
+
+static const struct lochnagar_clk lochnagar1_clks[LOCHNAGAR_NUM_CLOCKS] = {
+ LN1_CLK(CDC_MCLK1, "ln-cdc-mclk1", CDC_AIF_CTRL2),
+ LN1_CLK(CDC_MCLK2, "ln-cdc-mclk2", CDC_AIF_CTRL2),
+ LN1_CLK(DSP_CLKIN, "ln-dsp-clkin", DSP_AIF),
+ LN1_CLK(GF_CLKOUT1, "ln-gf-clkout1", GF_AIF1),
+};
+
+static const struct lochnagar_clk lochnagar2_clks[LOCHNAGAR_NUM_CLOCKS] = {
+ LN2_CLK(CDC_MCLK1, "ln-cdc-mclk1"),
+ LN2_CLK(CDC_MCLK2, "ln-cdc-mclk2"),
+ LN2_CLK(DSP_CLKIN, "ln-dsp-clkin"),
+ LN2_CLK(GF_CLKOUT1, "ln-gf-clkout1"),
+ LN2_CLK(GF_CLKOUT2, "ln-gf-clkout2"),
+ LN2_CLK(PSIA1_MCLK, "ln-psia1-mclk"),
+ LN2_CLK(PSIA2_MCLK, "ln-psia2-mclk"),
+ LN2_CLK(SPDIF_MCLK, "ln-spdif-mclk"),
+ LN2_CLK(ADAT_MCLK, "ln-adat-mclk"),
+ LN2_CLK(SOUNDCARD_MCLK, "ln-soundcard-mclk"),
+};
+
+static inline struct lochnagar_clk *lochnagar_hw_to_lclk(struct clk_hw *hw)
+{
+ return container_of(hw, struct lochnagar_clk, hw);
+}
+
+static int lochnagar_clk_prepare(struct clk_hw *hw)
+{
+ struct lochnagar_clk *lclk = lochnagar_hw_to_lclk(hw);
+ struct lochnagar_clk_priv *priv = lclk->priv;
+ struct regmap *regmap = priv->regmap;
+ int ret;
+
+ ret = regmap_update_bits(regmap, lclk->cfg_reg,
+ lclk->ena_mask, lclk->ena_mask);
+ if (ret < 0)
+ dev_dbg(priv->dev, "Failed to prepare %s: %d\n",
+ lclk->name, ret);
+
+ return ret;
+}
+
+static void lochnagar_clk_unprepare(struct clk_hw *hw)
+{
+ struct lochnagar_clk *lclk = lochnagar_hw_to_lclk(hw);
+ struct lochnagar_clk_priv *priv = lclk->priv;
+ struct regmap *regmap = priv->regmap;
+ int ret;
+
+ ret = regmap_update_bits(regmap, lclk->cfg_reg, lclk->ena_mask, 0);
+ if (ret < 0)
+ dev_dbg(priv->dev, "Failed to unprepare %s: %d\n",
+ lclk->name, ret);
+}
+
+static int lochnagar_clk_set_parent(struct clk_hw *hw, u8 index)
+{
+ struct lochnagar_clk *lclk = lochnagar_hw_to_lclk(hw);
+ struct lochnagar_clk_priv *priv = lclk->priv;
+ struct regmap *regmap = priv->regmap;
+ int ret;
+
+ ret = regmap_update_bits(regmap, lclk->src_reg, lclk->src_mask, index);
+ if (ret < 0)
+ dev_dbg(priv->dev, "Failed to reparent %s: %d\n",
+ lclk->name, ret);
+
+ return ret;
+}
+
+static u8 lochnagar_clk_get_parent(struct clk_hw *hw)
+{
+ struct lochnagar_clk *lclk = lochnagar_hw_to_lclk(hw);
+ struct lochnagar_clk_priv *priv = lclk->priv;
+ struct regmap *regmap = priv->regmap;
+ unsigned int val;
+ int ret;
+
+ ret = regmap_read(regmap, lclk->src_reg, &val);
+ if (ret < 0) {
+ dev_dbg(priv->dev, "Failed to read parent of %s: %d\n",
+ lclk->name, ret);
+ return priv->nparents;
+ }
+
+ val &= lclk->src_mask;
+
+ return val;
+}
+
+static const struct clk_ops lochnagar_clk_ops = {
+ .prepare = lochnagar_clk_prepare,
+ .unprepare = lochnagar_clk_unprepare,
+ .set_parent = lochnagar_clk_set_parent,
+ .get_parent = lochnagar_clk_get_parent,
+};
+
+static int lochnagar_init_parents(struct lochnagar_clk_priv *priv)
+{
+ struct device_node *np = priv->dev->of_node;
+ int i, j;
+
+ switch (priv->type) {
+ case LOCHNAGAR1:
+ memcpy(priv->lclks, lochnagar1_clks, sizeof(lochnagar1_clks));
+
+ priv->nparents = ARRAY_SIZE(lochnagar1_clk_parents);
+ priv->parents = devm_kmemdup(priv->dev, lochnagar1_clk_parents,
+ sizeof(lochnagar1_clk_parents),
+ GFP_KERNEL);
+ break;
+ case LOCHNAGAR2:
+ memcpy(priv->lclks, lochnagar2_clks, sizeof(lochnagar2_clks));
+
+ priv->nparents = ARRAY_SIZE(lochnagar2_clk_parents);
+ priv->parents = devm_kmemdup(priv->dev, lochnagar2_clk_parents,
+ sizeof(lochnagar2_clk_parents),
+ GFP_KERNEL);
+ break;
+ default:
+ dev_err(priv->dev, "Unknown Lochnagar type: %d\n", priv->type);
+ return -EINVAL;
+ }
+
+ if (!priv->parents)
+ return -ENOMEM;
+
+ for (i = 0; i < priv->nparents; i++) {
+ j = of_property_match_string(np, "clock-names",
+ priv->parents[i]);
+ if (j >= 0)
+ priv->parents[i] = of_clk_get_parent_name(np, j);
+ }
+
+ return 0;
+}
+
+static struct clk_hw *
+lochnagar_of_clk_hw_get(struct of_phandle_args *clkspec, void *data)
+{
+ struct lochnagar_clk_priv *priv = data;
+ unsigned int idx = clkspec->args[0];
+
+ if (idx >= ARRAY_SIZE(priv->lclks)) {
+ dev_err(priv->dev, "Invalid index %u\n", idx);
+ return ERR_PTR(-EINVAL);
+ }
+
+ return &priv->lclks[idx].hw;
+}
+
+static int lochnagar_init_clks(struct lochnagar_clk_priv *priv)
+{
+ struct clk_init_data clk_init = {
+ .ops = &lochnagar_clk_ops,
+ .parent_names = priv->parents,
+ .num_parents = priv->nparents,
+ };
+ struct lochnagar_clk *lclk;
+ int ret, i;
+
+ for (i = 0; i < ARRAY_SIZE(priv->lclks); i++) {
+ lclk = &priv->lclks[i];
+
+ if (!lclk->name)
+ continue;
+
+ clk_init.name = lclk->name;
+
+ lclk->priv = priv;
+ lclk->hw.init = &clk_init;
+
+ ret = devm_clk_hw_register(priv->dev, &lclk->hw);
+ if (ret) {
+ dev_err(priv->dev, "Failed to register %s: %d\n",
+ lclk->name, ret);
+ return ret;
+ }
+ }
+
+ ret = devm_of_clk_add_hw_provider(priv->dev, lochnagar_of_clk_hw_get,
+ priv);
+ if (ret < 0)
+ dev_err(priv->dev, "Failed to register provider: %d\n", ret);
+
+ return ret;
+}
+
+static const struct of_device_id lochnagar_of_match[] = {
+ { .compatible = "cirrus,lochnagar1-clk", .data = (void *)LOCHNAGAR1 },
+ { .compatible = "cirrus,lochnagar2-clk", .data = (void *)LOCHNAGAR2 },
+ {}
+};
+MODULE_DEVICE_TABLE(of, lochnagar_of_match);
+
+static int lochnagar_clk_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct lochnagar_clk_priv *priv;
+ const struct of_device_id *of_id;
+ int ret;
+
+ of_id = of_match_device(lochnagar_of_match, dev);
+ if (!of_id)
+ return -EINVAL;
+
+ priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->dev = dev;
+ priv->regmap = dev_get_regmap(dev->parent, NULL);
+ priv->type = (enum lochnagar_type)of_id->data;
+
+ ret = lochnagar_init_parents(priv);
+ if (ret)
+ return ret;
+
+ return lochnagar_init_clks(priv);
+}
+
+static struct platform_driver lochnagar_clk_driver = {
+ .driver = {
+ .name = "lochnagar-clk",
+ .of_match_table = lochnagar_of_match,
+ },
+ .probe = lochnagar_clk_probe,
+};
+module_platform_driver(lochnagar_clk_driver);
+
+MODULE_AUTHOR("Charles Keepax <ckeepax@opensource.cirrus.com>");
+MODULE_DESCRIPTION("Clock driver for Cirrus Logic Lochnagar Board");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2018 Socionext Inc.
+ * Copyright (C) 2016 Linaro Ltd.
+ */
+
+#include <linux/clk-provider.h>
+#include <linux/err.h>
+#include <linux/io.h>
+#include <linux/iopoll.h>
+#include <linux/of_address.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+
+#define M10V_CLKSEL1 0x0
+#define CLKSEL(n) (((n) - 1) * 4 + M10V_CLKSEL1)
+
+#define M10V_PLL1 "pll1"
+#define M10V_PLL1DIV2 "pll1-2"
+#define M10V_PLL2 "pll2"
+#define M10V_PLL2DIV2 "pll2-2"
+#define M10V_PLL6 "pll6"
+#define M10V_PLL6DIV2 "pll6-2"
+#define M10V_PLL6DIV3 "pll6-3"
+#define M10V_PLL7 "pll7"
+#define M10V_PLL7DIV2 "pll7-2"
+#define M10V_PLL7DIV5 "pll7-5"
+#define M10V_PLL9 "pll9"
+#define M10V_PLL10 "pll10"
+#define M10V_PLL10DIV2 "pll10-2"
+#define M10V_PLL11 "pll11"
+
+#define M10V_SPI_PARENT0 "spi-parent0"
+#define M10V_SPI_PARENT1 "spi-parent1"
+#define M10V_SPI_PARENT2 "spi-parent2"
+#define M10V_UHS1CLK2_PARENT0 "uhs1clk2-parent0"
+#define M10V_UHS1CLK2_PARENT1 "uhs1clk2-parent1"
+#define M10V_UHS1CLK2_PARENT2 "uhs1clk2-parent2"
+#define M10V_UHS1CLK1_PARENT0 "uhs1clk1-parent0"
+#define M10V_UHS1CLK1_PARENT1 "uhs1clk1-parent1"
+#define M10V_NFCLK_PARENT0 "nfclk-parent0"
+#define M10V_NFCLK_PARENT1 "nfclk-parent1"
+#define M10V_NFCLK_PARENT2 "nfclk-parent2"
+#define M10V_NFCLK_PARENT3 "nfclk-parent3"
+#define M10V_NFCLK_PARENT4 "nfclk-parent4"
+#define M10V_NFCLK_PARENT5 "nfclk-parent5"
+
+#define M10V_DCHREQ 1
+#define M10V_UPOLL_RATE 1
+#define M10V_UTIMEOUT 250
+
+#define M10V_EMMCCLK_ID 0
+#define M10V_ACLK_ID 1
+#define M10V_HCLK_ID 2
+#define M10V_PCLK_ID 3
+#define M10V_RCLK_ID 4
+#define M10V_SPICLK_ID 5
+#define M10V_NFCLK_ID 6
+#define M10V_UHS1CLK2_ID 7
+#define M10V_NUM_CLKS 8
+
+#define to_m10v_div(_hw) container_of(_hw, struct m10v_clk_divider, hw)
+
+static struct clk_hw_onecell_data *m10v_clk_data;
+
+static DEFINE_SPINLOCK(m10v_crglock);
+
+struct m10v_clk_div_factors {
+ const char *name;
+ const char *parent_name;
+ u32 offset;
+ u8 shift;
+ u8 width;
+ const struct clk_div_table *table;
+ unsigned long div_flags;
+ int onecell_idx;
+};
+
+struct m10v_clk_div_fixed_data {
+ const char *name;
+ const char *parent_name;
+ u8 div;
+ u8 mult;
+ int onecell_idx;
+};
+
+struct m10v_clk_mux_factors {
+ const char *name;
+ const char * const *parent_names;
+ u8 num_parents;
+ u32 offset;
+ u8 shift;
+ u8 mask;
+ u32 *table;
+ unsigned long mux_flags;
+ int onecell_idx;
+};
+
+static const struct clk_div_table emmcclk_table[] = {
+ { .val = 0, .div = 8 },
+ { .val = 1, .div = 9 },
+ { .val = 2, .div = 10 },
+ { .val = 3, .div = 15 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table mclk400_table[] = {
+ { .val = 1, .div = 2 },
+ { .val = 3, .div = 4 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table mclk200_table[] = {
+ { .val = 3, .div = 4 },
+ { .val = 7, .div = 8 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table aclk400_table[] = {
+ { .val = 1, .div = 2 },
+ { .val = 3, .div = 4 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table aclk300_table[] = {
+ { .val = 0, .div = 2 },
+ { .val = 1, .div = 3 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table aclk_table[] = {
+ { .val = 3, .div = 4 },
+ { .val = 7, .div = 8 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table aclkexs_table[] = {
+ { .val = 3, .div = 4 },
+ { .val = 4, .div = 5 },
+ { .val = 5, .div = 6 },
+ { .val = 7, .div = 8 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table hclk_table[] = {
+ { .val = 7, .div = 8 },
+ { .val = 15, .div = 16 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table hclkbmh_table[] = {
+ { .val = 3, .div = 4 },
+ { .val = 7, .div = 8 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table pclk_table[] = {
+ { .val = 15, .div = 16 },
+ { .val = 31, .div = 32 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table rclk_table[] = {
+ { .val = 0, .div = 8 },
+ { .val = 1, .div = 16 },
+ { .val = 2, .div = 24 },
+ { .val = 3, .div = 32 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table uhs1clk0_table[] = {
+ { .val = 0, .div = 2 },
+ { .val = 1, .div = 3 },
+ { .val = 2, .div = 4 },
+ { .val = 3, .div = 8 },
+ { .val = 4, .div = 16 },
+ { .div = 0 },
+};
+
+static const struct clk_div_table uhs2clk_table[] = {
+ { .val = 0, .div = 9 },
+ { .val = 1, .div = 10 },
+ { .val = 2, .div = 11 },
+ { .val = 3, .div = 12 },
+ { .val = 4, .div = 13 },
+ { .val = 5, .div = 14 },
+ { .val = 6, .div = 16 },
+ { .val = 7, .div = 18 },
+ { .div = 0 },
+};
+
+static u32 spi_mux_table[] = {0, 1, 2};
+static const char * const spi_mux_names[] = {
+ M10V_SPI_PARENT0, M10V_SPI_PARENT1, M10V_SPI_PARENT2
+};
+
+static u32 uhs1clk2_mux_table[] = {2, 3, 4, 8};
+static const char * const uhs1clk2_mux_names[] = {
+ M10V_UHS1CLK2_PARENT0, M10V_UHS1CLK2_PARENT1,
+ M10V_UHS1CLK2_PARENT2, M10V_PLL6DIV2
+};
+
+static u32 uhs1clk1_mux_table[] = {3, 4, 8};
+static const char * const uhs1clk1_mux_names[] = {
+ M10V_UHS1CLK1_PARENT0, M10V_UHS1CLK1_PARENT1, M10V_PLL6DIV2
+};
+
+static u32 nfclk_mux_table[] = {0, 1, 2, 3, 4, 8};
+static const char * const nfclk_mux_names[] = {
+ M10V_NFCLK_PARENT0, M10V_NFCLK_PARENT1, M10V_NFCLK_PARENT2,
+ M10V_NFCLK_PARENT3, M10V_NFCLK_PARENT4, M10V_NFCLK_PARENT5
+};
+
+static const struct m10v_clk_div_fixed_data m10v_pll_fixed_data[] = {
+ {M10V_PLL1, NULL, 1, 40, -1},
+ {M10V_PLL2, NULL, 1, 30, -1},
+ {M10V_PLL6, NULL, 1, 35, -1},
+ {M10V_PLL7, NULL, 1, 40, -1},
+ {M10V_PLL9, NULL, 1, 33, -1},
+ {M10V_PLL10, NULL, 5, 108, -1},
+ {M10V_PLL10DIV2, M10V_PLL10, 2, 1, -1},
+ {M10V_PLL11, NULL, 2, 75, -1},
+};
+
+static const struct m10v_clk_div_fixed_data m10v_div_fixed_data[] = {
+ {"usb2", NULL, 2, 1, -1},
+ {"pcisuppclk", NULL, 20, 1, -1},
+ {M10V_PLL1DIV2, M10V_PLL1, 2, 1, -1},
+ {M10V_PLL2DIV2, M10V_PLL2, 2, 1, -1},
+ {M10V_PLL6DIV2, M10V_PLL6, 2, 1, -1},
+ {M10V_PLL6DIV3, M10V_PLL6, 3, 1, -1},
+ {M10V_PLL7DIV2, M10V_PLL7, 2, 1, -1},
+ {M10V_PLL7DIV5, M10V_PLL7, 5, 1, -1},
+ {"ca7wd", M10V_PLL2DIV2, 12, 1, -1},
+ {"pclkca7wd", M10V_PLL1DIV2, 16, 1, -1},
+ {M10V_SPI_PARENT0, M10V_PLL10DIV2, 2, 1, -1},
+ {M10V_SPI_PARENT1, M10V_PLL10DIV2, 4, 1, -1},
+ {M10V_SPI_PARENT2, M10V_PLL7DIV2, 8, 1, -1},
+ {M10V_UHS1CLK2_PARENT0, M10V_PLL7, 4, 1, -1},
+ {M10V_UHS1CLK2_PARENT1, M10V_PLL7, 8, 1, -1},
+ {M10V_UHS1CLK2_PARENT2, M10V_PLL7, 16, 1, -1},
+ {M10V_UHS1CLK1_PARENT0, M10V_PLL7, 8, 1, -1},
+ {M10V_UHS1CLK1_PARENT1, M10V_PLL7, 16, 1, -1},
+ {M10V_NFCLK_PARENT0, M10V_PLL7DIV2, 8, 1, -1},
+ {M10V_NFCLK_PARENT1, M10V_PLL7DIV2, 10, 1, -1},
+ {M10V_NFCLK_PARENT2, M10V_PLL7DIV2, 13, 1, -1},
+ {M10V_NFCLK_PARENT3, M10V_PLL7DIV2, 16, 1, -1},
+ {M10V_NFCLK_PARENT4, M10V_PLL7DIV2, 40, 1, -1},
+ {M10V_NFCLK_PARENT5, M10V_PLL7DIV5, 10, 1, -1},
+};
+
+static const struct m10v_clk_div_factors m10v_div_factor_data[] = {
+ {"emmc", M10V_PLL11, CLKSEL(1), 28, 3, emmcclk_table, 0,
+ M10V_EMMCCLK_ID},
+ {"mclk400", M10V_PLL1DIV2, CLKSEL(10), 7, 3, mclk400_table, 0, -1},
+ {"mclk200", M10V_PLL1DIV2, CLKSEL(10), 3, 4, mclk200_table, 0, -1},
+ {"aclk400", M10V_PLL1DIV2, CLKSEL(10), 0, 3, aclk400_table, 0, -1},
+ {"aclk300", M10V_PLL2DIV2, CLKSEL(12), 0, 2, aclk300_table, 0, -1},
+ {"aclk", M10V_PLL1DIV2, CLKSEL(9), 20, 4, aclk_table, 0, M10V_ACLK_ID},
+ {"aclkexs", M10V_PLL1DIV2, CLKSEL(9), 16, 4, aclkexs_table, 0, -1},
+ {"hclk", M10V_PLL1DIV2, CLKSEL(9), 7, 5, hclk_table, 0, M10V_HCLK_ID},
+ {"hclkbmh", M10V_PLL1DIV2, CLKSEL(9), 12, 4, hclkbmh_table, 0, -1},
+ {"pclk", M10V_PLL1DIV2, CLKSEL(9), 0, 7, pclk_table, 0, M10V_PCLK_ID},
+ {"uhs1clk0", M10V_PLL7, CLKSEL(1), 3, 5, uhs1clk0_table, 0, -1},
+ {"uhs2clk", M10V_PLL6DIV3, CLKSEL(1), 18, 4, uhs2clk_table, 0, -1},
+};
+
+static const struct m10v_clk_mux_factors m10v_mux_factor_data[] = {
+ {"spi", spi_mux_names, ARRAY_SIZE(spi_mux_names),
+ CLKSEL(8), 3, 7, spi_mux_table, 0, M10V_SPICLK_ID},
+ {"uhs1clk2", uhs1clk2_mux_names, ARRAY_SIZE(uhs1clk2_mux_names),
+ CLKSEL(1), 13, 31, uhs1clk2_mux_table, 0, M10V_UHS1CLK2_ID},
+ {"uhs1clk1", uhs1clk1_mux_names, ARRAY_SIZE(uhs1clk1_mux_names),
+ CLKSEL(1), 8, 31, uhs1clk1_mux_table, 0, -1},
+ {"nfclk", nfclk_mux_names, ARRAY_SIZE(nfclk_mux_names),
+ CLKSEL(1), 22, 127, nfclk_mux_table, 0, M10V_NFCLK_ID},
+};
+
+static u8 m10v_mux_get_parent(struct clk_hw *hw)
+{
+ struct clk_mux *mux = to_clk_mux(hw);
+ u32 val;
+
+ val = readl(mux->reg) >> mux->shift;
+ val &= mux->mask;
+
+ return clk_mux_val_to_index(hw, mux->table, mux->flags, val);
+}
+
+static int m10v_mux_set_parent(struct clk_hw *hw, u8 index)
+{
+ struct clk_mux *mux = to_clk_mux(hw);
+ u32 val = clk_mux_index_to_val(mux->table, mux->flags, index);
+ unsigned long flags = 0;
+ u32 reg;
+ u32 write_en = BIT(fls(mux->mask) - 1);
+
+ if (mux->lock)
+ spin_lock_irqsave(mux->lock, flags);
+ else
+ __acquire(mux->lock);
+
+ reg = readl(mux->reg);
+ reg &= ~(mux->mask << mux->shift);
+
+ val = (val | write_en) << mux->shift;
+ reg |= val;
+ writel(reg, mux->reg);
+
+ if (mux->lock)
+ spin_unlock_irqrestore(mux->lock, flags);
+ else
+ __release(mux->lock);
+
+ return 0;
+}
+
+static const struct clk_ops m10v_mux_ops = {
+ .get_parent = m10v_mux_get_parent,
+ .set_parent = m10v_mux_set_parent,
+ .determine_rate = __clk_mux_determine_rate,
+};
+
+static struct clk_hw *m10v_clk_hw_register_mux(struct device *dev,
+ const char *name, const char * const *parent_names,
+ u8 num_parents, unsigned long flags, void __iomem *reg,
+ u8 shift, u32 mask, u8 clk_mux_flags, u32 *table,
+ spinlock_t *lock)
+{
+ struct clk_mux *mux;
+ struct clk_hw *hw;
+ struct clk_init_data init;
+ int ret;
+
+ mux = kzalloc(sizeof(*mux), GFP_KERNEL);
+ if (!mux)
+ return ERR_PTR(-ENOMEM);
+
+ init.name = name;
+ init.ops = &m10v_mux_ops;
+ init.flags = flags;
+ init.parent_names = parent_names;
+ init.num_parents = num_parents;
+
+ mux->reg = reg;
+ mux->shift = shift;
+ mux->mask = mask;
+ mux->flags = clk_mux_flags;
+ mux->lock = lock;
+ mux->table = table;
+ mux->hw.init = &init;
+
+ hw = &mux->hw;
+ ret = clk_hw_register(dev, hw);
+ if (ret) {
+ kfree(mux);
+ hw = ERR_PTR(ret);
+ }
+
+ return hw;
+
+}
+
+struct m10v_clk_divider {
+ struct clk_hw hw;
+ void __iomem *reg;
+ u8 shift;
+ u8 width;
+ u8 flags;
+ const struct clk_div_table *table;
+ spinlock_t *lock;
+ void __iomem *write_valid_reg;
+};
+
+static unsigned long m10v_clk_divider_recalc_rate(struct clk_hw *hw,
+ unsigned long parent_rate)
+{
+ struct m10v_clk_divider *divider = to_m10v_div(hw);
+ unsigned int val;
+
+ val = readl(divider->reg) >> divider->shift;
+ val &= clk_div_mask(divider->width);
+
+ return divider_recalc_rate(hw, parent_rate, val, divider->table,
+ divider->flags, divider->width);
+}
+
+static long m10v_clk_divider_round_rate(struct clk_hw *hw, unsigned long rate,
+ unsigned long *prate)
+{
+ struct m10v_clk_divider *divider = to_m10v_div(hw);
+
+ /* if read only, just return current value */
+ if (divider->flags & CLK_DIVIDER_READ_ONLY) {
+ u32 val;
+
+ val = readl(divider->reg) >> divider->shift;
+ val &= clk_div_mask(divider->width);
+
+ return divider_ro_round_rate(hw, rate, prate, divider->table,
+ divider->width, divider->flags,
+ val);
+ }
+
+ return divider_round_rate(hw, rate, prate, divider->table,
+ divider->width, divider->flags);
+}
+
+static int m10v_clk_divider_set_rate(struct clk_hw *hw, unsigned long rate,
+ unsigned long parent_rate)
+{
+ struct m10v_clk_divider *divider = to_m10v_div(hw);
+ int value;
+ unsigned long flags = 0;
+ u32 val;
+ u32 write_en = BIT(divider->width - 1);
+
+ value = divider_get_val(rate, parent_rate, divider->table,
+ divider->width, divider->flags);
+ if (value < 0)
+ return value;
+
+ if (divider->lock)
+ spin_lock_irqsave(divider->lock, flags);
+ else
+ __acquire(divider->lock);
+
+ val = readl(divider->reg);
+ val &= ~(clk_div_mask(divider->width) << divider->shift);
+
+ val |= ((u32)value | write_en) << divider->shift;
+ writel(val, divider->reg);
+
+ if (divider->write_valid_reg) {
+ writel(M10V_DCHREQ, divider->write_valid_reg);
+ if (readl_poll_timeout(divider->write_valid_reg, val,
+ !val, M10V_UPOLL_RATE, M10V_UTIMEOUT))
+ pr_err("%s:%s couldn't stabilize\n",
+ __func__, divider->hw.init->name);
+ }
+
+ if (divider->lock)
+ spin_unlock_irqrestore(divider->lock, flags);
+ else
+ __release(divider->lock);
+
+ return 0;
+}
+
+static const struct clk_ops m10v_clk_divider_ops = {
+ .recalc_rate = m10v_clk_divider_recalc_rate,
+ .round_rate = m10v_clk_divider_round_rate,
+ .set_rate = m10v_clk_divider_set_rate,
+};
+
+static struct clk_hw *m10v_clk_hw_register_divider(struct device *dev,
+ const char *name, const char *parent_name, unsigned long flags,
+ void __iomem *reg, u8 shift, u8 width,
+ u8 clk_divider_flags, const struct clk_div_table *table,
+ spinlock_t *lock, void __iomem *write_valid_reg)
+{
+ struct m10v_clk_divider *div;
+ struct clk_hw *hw;
+ struct clk_init_data init;
+ int ret;
+
+ div = kzalloc(sizeof(*div), GFP_KERNEL);
+ if (!div)
+ return ERR_PTR(-ENOMEM);
+
+ init.name = name;
+ init.ops = &m10v_clk_divider_ops;
+ init.flags = flags;
+ init.parent_names = &parent_name;
+ init.num_parents = 1;
+
+ div->reg = reg;
+ div->shift = shift;
+ div->width = width;
+ div->flags = clk_divider_flags;
+ div->lock = lock;
+ div->hw.init = &init;
+ div->table = table;
+ div->write_valid_reg = write_valid_reg;
+
+ /* register the clock */
+ hw = &div->hw;
+ ret = clk_hw_register(dev, hw);
+ if (ret) {
+ kfree(div);
+ hw = ERR_PTR(ret);
+ }
+
+ return hw;
+}
+
+static void m10v_reg_div_pre(const struct m10v_clk_div_factors *factors,
+ struct clk_hw_onecell_data *clk_data,
+ void __iomem *base)
+{
+ struct clk_hw *hw;
+ void __iomem *write_valid_reg;
+
+ /*
+ * The registers on CLKSEL(9) or CLKSEL(10) need additional
+ * writing to become valid.
+ */
+ if ((factors->offset == CLKSEL(9)) || (factors->offset == CLKSEL(10)))
+ write_valid_reg = base + CLKSEL(11);
+ else
+ write_valid_reg = NULL;
+
+ hw = m10v_clk_hw_register_divider(NULL, factors->name,
+ factors->parent_name,
+ CLK_SET_RATE_PARENT,
+ base + factors->offset,
+ factors->shift,
+ factors->width, factors->div_flags,
+ factors->table,
+ &m10v_crglock, write_valid_reg);
+
+ if (factors->onecell_idx >= 0)
+ clk_data->hws[factors->onecell_idx] = hw;
+}
+
+static void m10v_reg_fixed_pre(const struct m10v_clk_div_fixed_data *factors,
+ struct clk_hw_onecell_data *clk_data,
+ const char *parent_name)
+{
+ struct clk_hw *hw;
+ const char *pn = factors->parent_name ?
+ factors->parent_name : parent_name;
+
+ hw = clk_hw_register_fixed_factor(NULL, factors->name, pn, 0,
+ factors->mult, factors->div);
+
+ if (factors->onecell_idx >= 0)
+ clk_data->hws[factors->onecell_idx] = hw;
+}
+
+static void m10v_reg_mux_pre(const struct m10v_clk_mux_factors *factors,
+ struct clk_hw_onecell_data *clk_data,
+ void __iomem *base)
+{
+ struct clk_hw *hw;
+
+ hw = m10v_clk_hw_register_mux(NULL, factors->name,
+ factors->parent_names,
+ factors->num_parents,
+ CLK_SET_RATE_PARENT,
+ base + factors->offset, factors->shift,
+ factors->mask, factors->mux_flags,
+ factors->table, &m10v_crglock);
+
+ if (factors->onecell_idx >= 0)
+ clk_data->hws[factors->onecell_idx] = hw;
+}
+
+static int m10v_clk_probe(struct platform_device *pdev)
+{
+ int id;
+ struct resource *res;
+ struct device *dev = &pdev->dev;
+ struct device_node *np = dev->of_node;
+ void __iomem *base;
+ const char *parent_name;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ base = devm_ioremap_resource(dev, res);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ parent_name = of_clk_get_parent_name(np, 0);
+
+ for (id = 0; id < ARRAY_SIZE(m10v_div_factor_data); ++id)
+ m10v_reg_div_pre(&m10v_div_factor_data[id],
+ m10v_clk_data, base);
+
+ for (id = 0; id < ARRAY_SIZE(m10v_div_fixed_data); ++id)
+ m10v_reg_fixed_pre(&m10v_div_fixed_data[id],
+ m10v_clk_data, parent_name);
+
+ for (id = 0; id < ARRAY_SIZE(m10v_mux_factor_data); ++id)
+ m10v_reg_mux_pre(&m10v_mux_factor_data[id],
+ m10v_clk_data, base);
+
+ for (id = 0; id < M10V_NUM_CLKS; id++) {
+ if (IS_ERR(m10v_clk_data->hws[id]))
+ return PTR_ERR(m10v_clk_data->hws[id]);
+ }
+
+ return 0;
+}
+
+static const struct of_device_id m10v_clk_dt_ids[] = {
+ { .compatible = "socionext,milbeaut-m10v-ccu", },
+ { }
+};
+
+static struct platform_driver m10v_clk_driver = {
+ .probe = m10v_clk_probe,
+ .driver = {
+ .name = "m10v-ccu",
+ .of_match_table = m10v_clk_dt_ids,
+ },
+};
+builtin_platform_driver(m10v_clk_driver);
+
+static void __init m10v_cc_init(struct device_node *np)
+{
+ int id;
+ void __iomem *base;
+ const char *parent_name;
+ struct clk_hw *hw;
+
+ m10v_clk_data = kzalloc(struct_size(m10v_clk_data, hws,
+ M10V_NUM_CLKS),
+ GFP_KERNEL);
+
+ if (!m10v_clk_data)
+ return;
+
+ base = of_iomap(np, 0);
+ if (!base) {
+ kfree(m10v_clk_data);
+ return;
+ }
+
+ parent_name = of_clk_get_parent_name(np, 0);
+ if (!parent_name) {
+ kfree(m10v_clk_data);
+ iounmap(base);
+ return;
+ }
+
+ /*
+ * This way all clocks fetched before the platform device probes,
+ * except those we assign here for early use, will be deferred.
+ */
+ for (id = 0; id < M10V_NUM_CLKS; id++)
+ m10v_clk_data->hws[id] = ERR_PTR(-EPROBE_DEFER);
+
+ /*
+ * PLLs are set by bootloader so this driver registers them as the
+ * fixed factor.
+ */
+ for (id = 0; id < ARRAY_SIZE(m10v_pll_fixed_data); ++id)
+ m10v_reg_fixed_pre(&m10v_pll_fixed_data[id],
+ m10v_clk_data, parent_name);
+
+ /*
+ * timer consumes "rclk" so it needs to register here.
+ */
+ hw = m10v_clk_hw_register_divider(NULL, "rclk", M10V_PLL10DIV2, 0,
+ base + CLKSEL(1), 0, 3, 0, rclk_table,
+ &m10v_crglock, NULL);
+ m10v_clk_data->hws[M10V_RCLK_ID] = hw;
+
+ m10v_clk_data->num = M10V_NUM_CLKS;
+ of_clk_add_hw_provider(np, of_clk_hw_onecell_get, m10v_clk_data);
+}
+CLK_OF_DECLARE_DRIVER(m10v_cc, "socionext,milbeaut-m10v-ccu", m10v_cc_init);
#include <linux/of.h>
#include <linux/slab.h>
+static inline u32 clk_mult_readl(struct clk_multiplier *mult)
+{
+ if (mult->flags & CLK_MULTIPLIER_BIG_ENDIAN)
+ return ioread32be(mult->reg);
+
+ return readl(mult->reg);
+}
+
+static inline void clk_mult_writel(struct clk_multiplier *mult, u32 val)
+{
+ if (mult->flags & CLK_MULTIPLIER_BIG_ENDIAN)
+ iowrite32be(val, mult->reg);
+ else
+ writel(val, mult->reg);
+}
+
static unsigned long __get_mult(struct clk_multiplier *mult,
unsigned long rate,
unsigned long parent_rate)
struct clk_multiplier *mult = to_clk_multiplier(hw);
unsigned long val;
- val = clk_readl(mult->reg) >> mult->shift;
+ val = clk_mult_readl(mult) >> mult->shift;
val &= GENMASK(mult->width - 1, 0);
if (!val && mult->flags & CLK_MULTIPLIER_ZERO_BYPASS)
else
__acquire(mult->lock);
- val = clk_readl(mult->reg);
+ val = clk_mult_readl(mult);
val &= ~GENMASK(mult->width + mult->shift - 1, mult->shift);
val |= factor << mult->shift;
- clk_writel(val, mult->reg);
+ clk_mult_writel(mult, val);
if (mult->lock)
spin_unlock_irqrestore(mult->lock, flags);
* parent - parent is adjustable through clk_set_parent
*/
+static inline u32 clk_mux_readl(struct clk_mux *mux)
+{
+ if (mux->flags & CLK_MUX_BIG_ENDIAN)
+ return ioread32be(mux->reg);
+
+ return readl(mux->reg);
+}
+
+static inline void clk_mux_writel(struct clk_mux *mux, u32 val)
+{
+ if (mux->flags & CLK_MUX_BIG_ENDIAN)
+ iowrite32be(val, mux->reg);
+ else
+ writel(val, mux->reg);
+}
+
int clk_mux_val_to_index(struct clk_hw *hw, u32 *table, unsigned int flags,
unsigned int val)
{
struct clk_mux *mux = to_clk_mux(hw);
u32 val;
- val = clk_readl(mux->reg) >> mux->shift;
+ val = clk_mux_readl(mux) >> mux->shift;
val &= mux->mask;
return clk_mux_val_to_index(hw, mux->table, mux->flags, val);
if (mux->flags & CLK_MUX_HIWORD_MASK) {
reg = mux->mask << (mux->shift + 16);
} else {
- reg = clk_readl(mux->reg);
+ reg = clk_mux_readl(mux);
reg &= ~(mux->mask << mux->shift);
}
val = val << mux->shift;
reg |= val;
- clk_writel(reg, mux->reg);
+ clk_mux_writel(mux, reg);
if (mux->lock)
spin_unlock_irqrestore(mux->lock, flags);
init.ops = &clk_mux_ro_ops;
else
init.ops = &clk_mux_ops;
- init.flags = flags | CLK_IS_BASIC;
+ init.flags = flags;
init.parent_names = parent_names;
init.num_parents = num_parents;
init.name = clk_name;
init.ops = &clk_pwm_ops;
- init.flags = CLK_IS_BASIC;
+ init.flags = 0;
init.num_parents = 0;
clk_pwm->pwm = pwm;
#define CGA_PLL4 4 /* only on clockgen-1.0, which lacks CGB */
#define CGB_PLL1 4
#define CGB_PLL2 5
+#define MAX_PLL_DIV 16
struct clockgen_pll_div {
struct clk *clk;
};
struct clockgen_pll {
- struct clockgen_pll_div div[8];
+ struct clockgen_pll_div div[MAX_PLL_DIV];
};
#define CLKSEL_VALID 1
const struct clockgen_muxinfo *cmux_groups[2];
const struct clockgen_muxinfo *hwaccel[NUM_HWACCEL];
void (*init_periph)(struct clockgen *cg);
- int cmux_to_group[NUM_CMUX]; /* -1 terminates if fewer than NUM_CMUX */
+ int cmux_to_group[NUM_CMUX + 1]; /* array should be -1 terminated */
u32 pll_mask; /* 1 << n bit set if PLL n is valid */
u32 flags; /* CG_xxx */
};
},
};
+static const struct clockgen_muxinfo ls1028a_hwa1 = {
+ {
+ { CLKSEL_VALID, PLATFORM_PLL, PLL_DIV1 },
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV1 },
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV2 },
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV3 },
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV4 },
+ {},
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV2 },
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV3 },
+ },
+};
+
+static const struct clockgen_muxinfo ls1028a_hwa2 = {
+ {
+ { CLKSEL_VALID, PLATFORM_PLL, PLL_DIV1 },
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV1 },
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV2 },
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV3 },
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV4 },
+ {},
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV2 },
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV3 },
+ },
+};
+
+static const struct clockgen_muxinfo ls1028a_hwa3 = {
+ {
+ { CLKSEL_VALID, PLATFORM_PLL, PLL_DIV1 },
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV1 },
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV2 },
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV3 },
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV4 },
+ {},
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV2 },
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV3 },
+ },
+};
+
+static const struct clockgen_muxinfo ls1028a_hwa4 = {
+ {
+ { CLKSEL_VALID, PLATFORM_PLL, PLL_DIV1 },
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV1 },
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV2 },
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV3 },
+ { CLKSEL_VALID, CGA_PLL2, PLL_DIV4 },
+ {},
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV2 },
+ { CLKSEL_VALID, CGA_PLL1, PLL_DIV3 },
+ },
+};
+
static const struct clockgen_muxinfo ls1043a_hwa1 = {
{
{},
},
.pll_mask = 0x03,
},
+ {
+ .compat = "fsl,ls1028a-clockgen",
+ .cmux_groups = {
+ &clockgen2_cmux_cga12
+ },
+ .hwaccel = {
+ &ls1028a_hwa1, &ls1028a_hwa2,
+ &ls1028a_hwa3, &ls1028a_hwa4
+ },
+ .cmux_to_group = {
+ 0, 0, 0, 0, -1
+ },
+ .pll_mask = 0x07,
+ .flags = CG_VER3 | CG_LITTLE_ENDIAN,
+ },
{
.compat = "fsl,ls1043a-clockgen",
.init_periph = t2080_init_periph,
&p4080_cmux_grp1, &p4080_cmux_grp2
},
.cmux_to_group = {
- 0, 0, 0, 0, 1, 1, 1, 1
+ 0, 0, 0, 0, 1, 1, 1, 1, -1
},
.pll_mask = 0x1f,
},
int ret;
/*
- * For platform PLL, there are 8 divider clocks.
+ * For platform PLL, there are MAX_PLL_DIV divider clocks.
* For core PLL, there are 4 divider clocks at most.
*/
if (idx != PLATFORM_PLL && i >= 4)
CLK_OF_DECLARE(qoriq_clockgen_b4860, "fsl,b4860-clockgen", clockgen_init);
CLK_OF_DECLARE(qoriq_clockgen_ls1012a, "fsl,ls1012a-clockgen", clockgen_init);
CLK_OF_DECLARE(qoriq_clockgen_ls1021a, "fsl,ls1021a-clockgen", clockgen_init);
+CLK_OF_DECLARE(qoriq_clockgen_ls1028a, "fsl,ls1028a-clockgen", clockgen_init);
CLK_OF_DECLARE(qoriq_clockgen_ls1043a, "fsl,ls1043a-clockgen", clockgen_init);
CLK_OF_DECLARE(qoriq_clockgen_ls1046a, "fsl,ls1046a-clockgen", clockgen_init);
CLK_OF_DECLARE(qoriq_clockgen_ls1088a, "fsl,ls1088a-clockgen", clockgen_init);
{ STM32F4_RCC_APB2ENR, 26, "ltdc", "apb2_div" },
};
+static const struct stm32f4_gate_data stm32f769_gates[] __initconst = {
+ { STM32F4_RCC_AHB1ENR, 0, "gpioa", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 1, "gpiob", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 2, "gpioc", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 3, "gpiod", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 4, "gpioe", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 5, "gpiof", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 6, "gpiog", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 7, "gpioh", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 8, "gpioi", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 9, "gpioj", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 10, "gpiok", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 12, "crc", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 18, "bkpsra", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 20, "dtcmram", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 21, "dma1", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 22, "dma2", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 23, "dma2d", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 25, "ethmac", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 26, "ethmactx", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 27, "ethmacrx", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 28, "ethmacptp", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 29, "otghs", "ahb_div" },
+ { STM32F4_RCC_AHB1ENR, 30, "otghsulpi", "ahb_div" },
+
+ { STM32F4_RCC_AHB2ENR, 0, "dcmi", "ahb_div" },
+ { STM32F4_RCC_AHB2ENR, 1, "jpeg", "ahb_div" },
+ { STM32F4_RCC_AHB2ENR, 4, "cryp", "ahb_div" },
+ { STM32F4_RCC_AHB2ENR, 5, "hash", "ahb_div" },
+ { STM32F4_RCC_AHB2ENR, 6, "rng", "pll48" },
+ { STM32F4_RCC_AHB2ENR, 7, "otgfs", "pll48" },
+
+ { STM32F4_RCC_AHB3ENR, 0, "fmc", "ahb_div",
+ CLK_IGNORE_UNUSED },
+ { STM32F4_RCC_AHB3ENR, 1, "qspi", "ahb_div",
+ CLK_IGNORE_UNUSED },
+
+ { STM32F4_RCC_APB1ENR, 0, "tim2", "apb1_mul" },
+ { STM32F4_RCC_APB1ENR, 1, "tim3", "apb1_mul" },
+ { STM32F4_RCC_APB1ENR, 2, "tim4", "apb1_mul" },
+ { STM32F4_RCC_APB1ENR, 3, "tim5", "apb1_mul" },
+ { STM32F4_RCC_APB1ENR, 4, "tim6", "apb1_mul" },
+ { STM32F4_RCC_APB1ENR, 5, "tim7", "apb1_mul" },
+ { STM32F4_RCC_APB1ENR, 6, "tim12", "apb1_mul" },
+ { STM32F4_RCC_APB1ENR, 7, "tim13", "apb1_mul" },
+ { STM32F4_RCC_APB1ENR, 8, "tim14", "apb1_mul" },
+ { STM32F4_RCC_APB1ENR, 10, "rtcapb", "apb1_mul" },
+ { STM32F4_RCC_APB1ENR, 11, "wwdg", "apb1_div" },
+ { STM32F4_RCC_APB1ENR, 13, "can3", "apb1_div" },
+ { STM32F4_RCC_APB1ENR, 14, "spi2", "apb1_div" },
+ { STM32F4_RCC_APB1ENR, 15, "spi3", "apb1_div" },
+ { STM32F4_RCC_APB1ENR, 16, "spdifrx", "apb1_div" },
+ { STM32F4_RCC_APB1ENR, 25, "can1", "apb1_div" },
+ { STM32F4_RCC_APB1ENR, 26, "can2", "apb1_div" },
+ { STM32F4_RCC_APB1ENR, 27, "cec", "apb1_div" },
+ { STM32F4_RCC_APB1ENR, 28, "pwr", "apb1_div" },
+ { STM32F4_RCC_APB1ENR, 29, "dac", "apb1_div" },
+
+ { STM32F4_RCC_APB2ENR, 0, "tim1", "apb2_mul" },
+ { STM32F4_RCC_APB2ENR, 1, "tim8", "apb2_mul" },
+ { STM32F4_RCC_APB2ENR, 7, "sdmmc2", "sdmux2" },
+ { STM32F4_RCC_APB2ENR, 8, "adc1", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 9, "adc2", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 10, "adc3", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 11, "sdmmc1", "sdmux1" },
+ { STM32F4_RCC_APB2ENR, 12, "spi1", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 13, "spi4", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 14, "syscfg", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 16, "tim9", "apb2_mul" },
+ { STM32F4_RCC_APB2ENR, 17, "tim10", "apb2_mul" },
+ { STM32F4_RCC_APB2ENR, 18, "tim11", "apb2_mul" },
+ { STM32F4_RCC_APB2ENR, 20, "spi5", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 21, "spi6", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 22, "sai1", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 23, "sai2", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 26, "ltdc", "apb2_div" },
+ { STM32F4_RCC_APB2ENR, 30, "mdio", "apb2_div" },
+};
+
/*
* This bitmask tells us which bit offsets (0..192) on STM32F4[23]xxx
* have gate bits associated with them. Its combined hweight is 71.
0x0000000000000003ull,
0x04f77f833e01c9ffull };
+static const u64 stm32f769_gate_map[MAX_GATE_MAP] = { 0x000000f37ef417ffull,
+ 0x0000000000000003ull,
+ 0x44F77F833E01EDFFull };
+
static const u64 *stm32f4_gate_map;
static struct clk_hw **clks;
"no-clock", "lse", "lsi", "hse-rtc"
};
+static const char *pll_src = "pll-src";
+
+static const char *pllsrc_parent[2] = { "hsi", NULL };
+
static const char *dsi_parent[2] = { NULL, "pll-r" };
static const char *lcd_parent[1] = { "pllsai-r-div" };
static const char *i2c_parents[4] = { "apb1_div", "sys", "hsi", "no-clock" };
+static const char * const dfsdm1_src[] = { "apb2_div", "sys" };
+static const char * const adsfdm1_parent[] = { "sai1_clk", "sai2_clk" };
+
struct stm32_aux_clk {
int idx;
const char *name;
},
};
+static const struct stm32_aux_clk stm32f769_aux_clk[] = {
+ {
+ CLK_LCD, "lcd-tft", lcd_parent, ARRAY_SIZE(lcd_parent),
+ NO_MUX, 0, 0,
+ STM32F4_RCC_APB2ENR, 26,
+ CLK_SET_RATE_PARENT
+ },
+ {
+ CLK_I2S, "i2s", i2s_parents, ARRAY_SIZE(i2s_parents),
+ STM32F4_RCC_CFGR, 23, 1,
+ NO_GATE, 0,
+ CLK_SET_RATE_PARENT
+ },
+ {
+ CLK_SAI1, "sai1_clk", sai_parents, ARRAY_SIZE(sai_parents),
+ STM32F4_RCC_DCKCFGR, 20, 3,
+ STM32F4_RCC_APB2ENR, 22,
+ CLK_SET_RATE_PARENT
+ },
+ {
+ CLK_SAI2, "sai2_clk", sai_parents, ARRAY_SIZE(sai_parents),
+ STM32F4_RCC_DCKCFGR, 22, 3,
+ STM32F4_RCC_APB2ENR, 23,
+ CLK_SET_RATE_PARENT
+ },
+ {
+ NO_IDX, "pll48", pll48_parents, ARRAY_SIZE(pll48_parents),
+ STM32F7_RCC_DCKCFGR2, 27, 1,
+ NO_GATE, 0,
+ 0
+ },
+ {
+ NO_IDX, "sdmux1", sdmux_parents, ARRAY_SIZE(sdmux_parents),
+ STM32F7_RCC_DCKCFGR2, 28, 1,
+ NO_GATE, 0,
+ 0
+ },
+ {
+ NO_IDX, "sdmux2", sdmux_parents, ARRAY_SIZE(sdmux_parents),
+ STM32F7_RCC_DCKCFGR2, 29, 1,
+ NO_GATE, 0,
+ 0
+ },
+ {
+ CLK_HDMI_CEC, "hdmi-cec",
+ hdmi_parents, ARRAY_SIZE(hdmi_parents),
+ STM32F7_RCC_DCKCFGR2, 26, 1,
+ NO_GATE, 0,
+ 0
+ },
+ {
+ CLK_SPDIF, "spdif-rx",
+ spdif_parent, ARRAY_SIZE(spdif_parent),
+ STM32F7_RCC_DCKCFGR2, 22, 3,
+ STM32F4_RCC_APB2ENR, 23,
+ CLK_SET_RATE_PARENT
+ },
+ {
+ CLK_USART1, "usart1",
+ uart_parents1, ARRAY_SIZE(uart_parents1),
+ STM32F7_RCC_DCKCFGR2, 0, 3,
+ STM32F4_RCC_APB2ENR, 4,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_USART2, "usart2",
+ uart_parents2, ARRAY_SIZE(uart_parents1),
+ STM32F7_RCC_DCKCFGR2, 2, 3,
+ STM32F4_RCC_APB1ENR, 17,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_USART3, "usart3",
+ uart_parents2, ARRAY_SIZE(uart_parents1),
+ STM32F7_RCC_DCKCFGR2, 4, 3,
+ STM32F4_RCC_APB1ENR, 18,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_UART4, "uart4",
+ uart_parents2, ARRAY_SIZE(uart_parents1),
+ STM32F7_RCC_DCKCFGR2, 6, 3,
+ STM32F4_RCC_APB1ENR, 19,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_UART5, "uart5",
+ uart_parents2, ARRAY_SIZE(uart_parents1),
+ STM32F7_RCC_DCKCFGR2, 8, 3,
+ STM32F4_RCC_APB1ENR, 20,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_USART6, "usart6",
+ uart_parents1, ARRAY_SIZE(uart_parents1),
+ STM32F7_RCC_DCKCFGR2, 10, 3,
+ STM32F4_RCC_APB2ENR, 5,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_UART7, "uart7",
+ uart_parents2, ARRAY_SIZE(uart_parents1),
+ STM32F7_RCC_DCKCFGR2, 12, 3,
+ STM32F4_RCC_APB1ENR, 30,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_UART8, "uart8",
+ uart_parents2, ARRAY_SIZE(uart_parents1),
+ STM32F7_RCC_DCKCFGR2, 14, 3,
+ STM32F4_RCC_APB1ENR, 31,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_I2C1, "i2c1",
+ i2c_parents, ARRAY_SIZE(i2c_parents),
+ STM32F7_RCC_DCKCFGR2, 16, 3,
+ STM32F4_RCC_APB1ENR, 21,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_I2C2, "i2c2",
+ i2c_parents, ARRAY_SIZE(i2c_parents),
+ STM32F7_RCC_DCKCFGR2, 18, 3,
+ STM32F4_RCC_APB1ENR, 22,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_I2C3, "i2c3",
+ i2c_parents, ARRAY_SIZE(i2c_parents),
+ STM32F7_RCC_DCKCFGR2, 20, 3,
+ STM32F4_RCC_APB1ENR, 23,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_I2C4, "i2c4",
+ i2c_parents, ARRAY_SIZE(i2c_parents),
+ STM32F7_RCC_DCKCFGR2, 22, 3,
+ STM32F4_RCC_APB1ENR, 24,
+ CLK_SET_RATE_PARENT,
+ },
+ {
+ CLK_LPTIMER, "lptim1",
+ lptim_parent, ARRAY_SIZE(lptim_parent),
+ STM32F7_RCC_DCKCFGR2, 24, 3,
+ STM32F4_RCC_APB1ENR, 9,
+ CLK_SET_RATE_PARENT
+ },
+ {
+ CLK_F769_DSI, "dsi",
+ dsi_parent, ARRAY_SIZE(dsi_parent),
+ STM32F7_RCC_DCKCFGR2, 0, 1,
+ STM32F4_RCC_APB2ENR, 27,
+ CLK_SET_RATE_PARENT
+ },
+ {
+ CLK_DFSDM1, "dfsdm1",
+ dfsdm1_src, ARRAY_SIZE(dfsdm1_src),
+ STM32F4_RCC_DCKCFGR, 25, 1,
+ STM32F4_RCC_APB2ENR, 29,
+ CLK_SET_RATE_PARENT
+ },
+ {
+ CLK_ADFSDM1, "adfsdm1",
+ adsfdm1_parent, ARRAY_SIZE(adsfdm1_parent),
+ STM32F4_RCC_DCKCFGR, 26, 1,
+ STM32F4_RCC_APB2ENR, 29,
+ CLK_SET_RATE_PARENT
+ },
+};
+
static const struct stm32f4_clk_data stm32f429_clk_data = {
.end_primary = END_PRIMARY_CLK,
.gates_data = stm32f429_gates,
.aux_clk_num = ARRAY_SIZE(stm32f746_aux_clk),
};
+static const struct stm32f4_clk_data stm32f769_clk_data = {
+ .end_primary = END_PRIMARY_CLK_F7,
+ .gates_data = stm32f769_gates,
+ .gates_map = stm32f769_gate_map,
+ .gates_num = ARRAY_SIZE(stm32f769_gates),
+ .pll_data = stm32f469_pll,
+ .aux_clk = stm32f769_aux_clk,
+ .aux_clk_num = ARRAY_SIZE(stm32f769_aux_clk),
+};
+
static const struct of_device_id stm32f4_of_match[] = {
{
.compatible = "st,stm32f42xx-rcc",
.compatible = "st,stm32f746-rcc",
.data = &stm32f746_clk_data
},
+ {
+ .compatible = "st,stm32f769-rcc",
+ .data = &stm32f769_clk_data
+ },
{}
};
int n;
const struct of_device_id *match;
const struct stm32f4_clk_data *data;
- unsigned long pllcfgr;
- const char *pllsrc;
unsigned long pllm;
+ struct clk_hw *pll_src_hw;
base = of_iomap(np, 0);
if (!base) {
hse_clk = of_clk_get_parent_name(np, 0);
dsi_parent[0] = hse_clk;
+ pllsrc_parent[1] = hse_clk;
i2s_in_clk = of_clk_get_parent_name(np, 1);
i2s_parents[1] = i2s_in_clk;
sai_parents[2] = i2s_in_clk;
+ if (of_device_is_compatible(np, "st,stm32f769-rcc")) {
+ clk_hw_register_gate(NULL, "dfsdm1_apb", "apb2_div", 0,
+ base + STM32F4_RCC_APB2ENR, 29,
+ CLK_IGNORE_UNUSED, &stm32f4_clk_lock);
+ dsi_parent[0] = pll_src;
+ sai_parents[3] = pll_src;
+ }
+
clks[CLK_HSI] = clk_hw_register_fixed_rate_with_accuracy(NULL, "hsi",
NULL, 0, 16000000, 160000);
- pllcfgr = readl(base + STM32F4_RCC_PLLCFGR);
- pllsrc = pllcfgr & BIT(22) ? hse_clk : "hsi";
- pllm = pllcfgr & 0x3f;
+ pll_src_hw = clk_hw_register_mux(NULL, pll_src, pllsrc_parent,
+ ARRAY_SIZE(pllsrc_parent), 0,
+ base + STM32F4_RCC_PLLCFGR, 22, 1, 0,
+ &stm32f4_clk_lock);
+
+ pllm = readl(base + STM32F4_RCC_PLLCFGR) & 0x3f;
- clk_hw_register_fixed_factor(NULL, "vco_in", pllsrc,
- 0, 1, pllm);
+ clk_hw_register_fixed_factor(NULL, "vco_in", pll_src,
+ 0, 1, pllm);
stm32f4_rcc_register_pll("vco_in", &data->pll_data[0],
&stm32f4_clk_lock);
clks[aux_clk->idx] = hw;
}
- if (of_device_is_compatible(np, "st,stm32f746-rcc"))
+ if (of_device_is_compatible(np, "st,stm32f746-rcc")) {
clk_hw_register_fixed_factor(NULL, "hsi_div488", "hsi", 0,
1, 488);
+ clks[CLK_PLL_SRC] = pll_src_hw;
+ }
+
of_clk_add_hw_provider(np, stm32f4_rcc_lookup_clk, NULL);
+
return;
fail:
kfree(clks);
CLK_OF_DECLARE_DRIVER(stm32f42xx_rcc, "st,stm32f42xx-rcc", stm32f4_rcc_init);
CLK_OF_DECLARE_DRIVER(stm32f46xx_rcc, "st,stm32f469-rcc", stm32f4_rcc_init);
CLK_OF_DECLARE_DRIVER(stm32f746_rcc, "st,stm32f746-rcc", stm32f4_rcc_init);
+CLK_OF_DECLARE_DRIVER(stm32f769_rcc, "st,stm32f769-rcc", stm32f4_rcc_init);
G_CRYP1,
G_HASH1,
G_BKPSRAM,
+ G_DDRPERFM,
G_LAST
};
K_GATE(G_STGENRO, RCC_APB4ENSETR, 20, 0),
K_MGATE(G_USBPHY, RCC_APB4ENSETR, 16, 0),
K_GATE(G_IWDG2, RCC_APB4ENSETR, 15, 0),
+ K_GATE(G_DDRPERFM, RCC_APB4ENSETR, 8, 0),
K_MGATE(G_DSI, RCC_APB4ENSETR, 4, 0),
K_MGATE(G_LTDC, RCC_APB4ENSETR, 0, 0),
PCLK(CRC1, "crc1", "ck_axi", 0, G_CRC1),
PCLK(USBH, "usbh", "ck_axi", 0, G_USBH),
PCLK(ETHSTP, "ethstp", "ck_axi", 0, G_ETHSTP),
+ PCLK(DDRPERFM, "ddrperfm", "pclk4", 0, G_DDRPERFM),
/* Kernel clocks */
KCLK(SDMMC1_K, "sdmmc1_k", sdmmc12_src, 0, G_SDMMC1, M_SDMMC12),
else
__acquire(fd->lock);
- val = clk_readl(fd->reg);
+ val = readl(fd->reg);
if (fd->lock)
spin_unlock_irqrestore(fd->lock, flags);
else
__acquire(fd->lock);
- val = clk_readl(fd->reg);
+ val = readl(fd->reg);
val &= ~fd->mask;
val |= (scale << fd->shift);
- clk_writel(val, fd->reg);
+ writel(val, fd->reg);
if (fd->lock)
spin_unlock_irqrestore(fd->lock, flags);
/*** private data structures ***/
+struct clk_parent_map {
+ const struct clk_hw *hw;
+ struct clk_core *core;
+ const char *fw_name;
+ const char *name;
+ int index;
+};
+
struct clk_core {
const char *name;
const struct clk_ops *ops;
struct clk_hw *hw;
struct module *owner;
struct device *dev;
+ struct device_node *of_node;
struct clk_core *parent;
- const char **parent_names;
- struct clk_core **parents;
+ struct clk_parent_map *parents;
u8 num_parents;
u8 new_parent_index;
unsigned long rate;
return NULL;
}
+/**
+ * clk_core_get - Find the clk_core parent of a clk
+ * @core: clk to find parent of
+ * @p_index: parent index to search for
+ *
+ * This is the preferred method for clk providers to find the parent of a
+ * clk when that parent is external to the clk controller. The parent_names
+ * array is indexed and treated as a local name matching a string in the device
+ * node's 'clock-names' property or as the 'con_id' matching the device's
+ * dev_name() in a clk_lookup. This allows clk providers to use their own
+ * namespace instead of looking for a globally unique parent string.
+ *
+ * For example the following DT snippet would allow a clock registered by the
+ * clock-controller@c001 that has a clk_init_data::parent_data array
+ * with 'xtal' in the 'name' member to find the clock provided by the
+ * clock-controller@f00abcd without needing to get the globally unique name of
+ * the xtal clk.
+ *
+ * parent: clock-controller@f00abcd {
+ * reg = <0xf00abcd 0xabcd>;
+ * #clock-cells = <0>;
+ * };
+ *
+ * clock-controller@c001 {
+ * reg = <0xc001 0xf00d>;
+ * clocks = <&parent>;
+ * clock-names = "xtal";
+ * #clock-cells = <1>;
+ * };
+ *
+ * Returns: -ENOENT when the provider can't be found or the clk doesn't
+ * exist in the provider. -EINVAL when the name can't be found. NULL when the
+ * provider knows about the clk but it isn't provided on this system.
+ * A valid clk_core pointer when the clk can be found in the provider.
+ */
+static struct clk_core *clk_core_get(struct clk_core *core, u8 p_index)
+{
+ const char *name = core->parents[p_index].fw_name;
+ int index = core->parents[p_index].index;
+ struct clk_hw *hw = ERR_PTR(-ENOENT);
+ struct device *dev = core->dev;
+ const char *dev_id = dev ? dev_name(dev) : NULL;
+ struct device_node *np = core->of_node;
+
+ if (np && index >= 0)
+ hw = of_clk_get_hw(np, index, name);
+
+ /*
+ * If the DT search above couldn't find the provider or the provider
+ * didn't know about this clk, fallback to looking up via clkdev based
+ * clk_lookups
+ */
+ if (PTR_ERR(hw) == -ENOENT && name)
+ hw = clk_find_hw(dev_id, name);
+
+ if (IS_ERR(hw))
+ return ERR_CAST(hw);
+
+ return hw->core;
+}
+
+static void clk_core_fill_parent_index(struct clk_core *core, u8 index)
+{
+ struct clk_parent_map *entry = &core->parents[index];
+ struct clk_core *parent = ERR_PTR(-ENOENT);
+
+ if (entry->hw) {
+ parent = entry->hw->core;
+ /*
+ * We have a direct reference but it isn't registered yet?
+ * Orphan it and let clk_reparent() update the orphan status
+ * when the parent is registered.
+ */
+ if (!parent)
+ parent = ERR_PTR(-EPROBE_DEFER);
+ } else {
+ parent = clk_core_get(core, index);
+ if (IS_ERR(parent) && PTR_ERR(parent) == -ENOENT)
+ parent = clk_core_lookup(entry->name);
+ }
+
+ /* Only cache it if it's not an error */
+ if (!IS_ERR(parent))
+ entry->core = parent;
+}
+
static struct clk_core *clk_core_get_parent_by_index(struct clk_core *core,
u8 index)
{
- if (!core || index >= core->num_parents)
+ if (!core || index >= core->num_parents || !core->parents)
return NULL;
- if (!core->parents[index])
- core->parents[index] =
- clk_core_lookup(core->parent_names[index]);
+ if (!core->parents[index].core)
+ clk_core_fill_parent_index(core, index);
- return core->parents[index];
+ return core->parents[index].core;
}
struct clk_hw *
static unsigned long clk_core_get_rate_nolock(struct clk_core *core)
{
- unsigned long ret;
-
- if (!core) {
- ret = 0;
- goto out;
- }
-
- ret = core->rate;
-
- if (!core->num_parents)
- goto out;
+ if (!core)
+ return 0;
- if (!core->parent)
- ret = 0;
+ if (!core->num_parents || core->parent)
+ return core->rate;
-out:
- return ret;
+ /*
+ * Clk must have a parent because num_parents > 0 but the parent isn't
+ * known yet. Best to return 0 as the rate of this clk until we can
+ * properly recalc the rate based on the parent's rate.
+ */
+ return 0;
}
unsigned long clk_hw_get_rate(const struct clk_hw *hw)
EXPORT_SYMBOL_GPL(clk_hw_set_rate_range);
/*
+ * __clk_mux_determine_rate - clk_ops::determine_rate implementation for a mux type clk
+ * @hw: mux type clk to determine rate on
+ * @req: rate request, also used to return preferred parent and frequencies
+ *
* Helper for finding best parent to provide a given frequency. This can be used
* directly as a determine_rate callback (e.g. for a mux), or from a more
* complex clock that may combine a mux with other operations.
+ *
+ * Returns: 0 on success, -EERROR value on error
*/
int __clk_mux_determine_rate(struct clk_hw *hw,
struct clk_rate_request *req)
return -EINVAL;
for (i = 0; i < core->num_parents; i++) {
- if (core->parents[i] == parent)
+ /* Found it first try! */
+ if (core->parents[i].core == parent)
return i;
- if (core->parents[i])
+ /* Something else is here, so keep looking */
+ if (core->parents[i].core)
continue;
- /* Fallback to comparing globally unique names */
- if (!strcmp(parent->name, core->parent_names[i])) {
- core->parents[i] = parent;
- return i;
+ /* Maybe core hasn't been cached but the hw is all we know? */
+ if (core->parents[i].hw) {
+ if (core->parents[i].hw == parent->hw)
+ break;
+
+ /* Didn't match, but we're expecting a clk_hw */
+ continue;
}
+
+ /* Maybe it hasn't been cached (clk_set_parent() path) */
+ if (parent == clk_core_get(core, i))
+ break;
+
+ /* Fallback to comparing globally unique names */
+ if (!strcmp(parent->name, core->parents[i].name))
+ break;
}
- return -EINVAL;
+ if (i == core->num_parents)
+ return -EINVAL;
+
+ core->parents[i].core = parent;
+ return i;
}
/*
bool clk_has_parent(struct clk *clk, struct clk *parent)
{
struct clk_core *core, *parent_core;
+ int i;
/* NULL clocks should be nops, so return success if either is NULL. */
if (!clk || !parent)
if (core->parent == parent_core)
return true;
- return match_string(core->parent_names, core->num_parents,
- parent_core->name) >= 0;
+ for (i = 0; i < core->num_parents; i++)
+ if (!strcmp(core->parents[i].name, parent_core->name))
+ return true;
+
+ return false;
}
EXPORT_SYMBOL_GPL(clk_has_parent);
ENTRY(CLK_SET_PARENT_GATE),
ENTRY(CLK_SET_RATE_PARENT),
ENTRY(CLK_IGNORE_UNUSED),
- ENTRY(CLK_IS_BASIC),
ENTRY(CLK_GET_RATE_NOCACHE),
ENTRY(CLK_SET_RATE_NO_REPARENT),
ENTRY(CLK_GET_ACCURACY_NOCACHE),
int i;
for (i = 0; i < core->num_parents - 1; i++)
- seq_printf(s, "%s ", core->parent_names[i]);
+ seq_printf(s, "%s ", core->parents[i].name);
- seq_printf(s, "%s\n", core->parent_names[i]);
+ seq_printf(s, "%s\n", core->parents[i].name);
return 0;
}
*/
static int __clk_core_init(struct clk_core *core)
{
- int i, ret;
+ int ret;
struct clk_core *orphan;
struct hlist_node *tmp2;
unsigned long rate;
goto out;
}
- /* throw a WARN if any entries in parent_names are NULL */
- for (i = 0; i < core->num_parents; i++)
- WARN(!core->parent_names[i],
- "%s: invalid NULL in %s's .parent_names\n",
- __func__, core->name);
-
core->parent = __clk_init_parent(core);
/*
return clk;
}
-/**
- * clk_register - allocate a new clock, register it and return an opaque cookie
- * @dev: device that is registering this clock
- * @hw: link to hardware-specific clock data
- *
- * clk_register is the primary interface for populating the clock tree with new
- * clock nodes. It returns a pointer to the newly allocated struct clk which
- * cannot be dereferenced by driver code but may be used in conjunction with the
- * rest of the clock API. In the event of an error clk_register will return an
- * error code; drivers must test for an error code after calling clk_register.
- */
-struct clk *clk_register(struct device *dev, struct clk_hw *hw)
+static int clk_cpy_name(const char **dst_p, const char *src, bool must_exist)
+{
+ const char *dst;
+
+ if (!src) {
+ if (must_exist)
+ return -EINVAL;
+ return 0;
+ }
+
+ *dst_p = dst = kstrdup_const(src, GFP_KERNEL);
+ if (!dst)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static int clk_core_populate_parent_map(struct clk_core *core)
+{
+ const struct clk_init_data *init = core->hw->init;
+ u8 num_parents = init->num_parents;
+ const char * const *parent_names = init->parent_names;
+ const struct clk_hw **parent_hws = init->parent_hws;
+ const struct clk_parent_data *parent_data = init->parent_data;
+ int i, ret = 0;
+ struct clk_parent_map *parents, *parent;
+
+ if (!num_parents)
+ return 0;
+
+ /*
+ * Avoid unnecessary string look-ups of clk_core's possible parents by
+ * having a cache of names/clk_hw pointers to clk_core pointers.
+ */
+ parents = kcalloc(num_parents, sizeof(*parents), GFP_KERNEL);
+ core->parents = parents;
+ if (!parents)
+ return -ENOMEM;
+
+ /* Copy everything over because it might be __initdata */
+ for (i = 0, parent = parents; i < num_parents; i++, parent++) {
+ parent->index = -1;
+ if (parent_names) {
+ /* throw a WARN if any entries are NULL */
+ WARN(!parent_names[i],
+ "%s: invalid NULL in %s's .parent_names\n",
+ __func__, core->name);
+ ret = clk_cpy_name(&parent->name, parent_names[i],
+ true);
+ } else if (parent_data) {
+ parent->hw = parent_data[i].hw;
+ parent->index = parent_data[i].index;
+ ret = clk_cpy_name(&parent->fw_name,
+ parent_data[i].fw_name, false);
+ if (!ret)
+ ret = clk_cpy_name(&parent->name,
+ parent_data[i].name,
+ false);
+ } else if (parent_hws) {
+ parent->hw = parent_hws[i];
+ } else {
+ ret = -EINVAL;
+ WARN(1, "Must specify parents if num_parents > 0\n");
+ }
+
+ if (ret) {
+ do {
+ kfree_const(parents[i].name);
+ kfree_const(parents[i].fw_name);
+ } while (--i >= 0);
+ kfree(parents);
+
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static void clk_core_free_parent_map(struct clk_core *core)
+{
+ int i = core->num_parents;
+
+ if (!core->num_parents)
+ return;
+
+ while (--i >= 0) {
+ kfree_const(core->parents[i].name);
+ kfree_const(core->parents[i].fw_name);
+ }
+
+ kfree(core->parents);
+}
+
+static struct clk *
+__clk_register(struct device *dev, struct device_node *np, struct clk_hw *hw)
{
- int i, ret;
+ int ret;
struct clk_core *core;
core = kzalloc(sizeof(*core), GFP_KERNEL);
if (dev && pm_runtime_enabled(dev))
core->rpm_enabled = true;
core->dev = dev;
+ core->of_node = np;
if (dev && dev->driver)
core->owner = dev->driver->owner;
core->hw = hw;
core->max_rate = ULONG_MAX;
hw->core = core;
- /* allocate local copy in case parent_names is __initdata */
- core->parent_names = kcalloc(core->num_parents, sizeof(char *),
- GFP_KERNEL);
-
- if (!core->parent_names) {
- ret = -ENOMEM;
- goto fail_parent_names;
- }
-
-
- /* copy each string name in case parent_names is __initdata */
- for (i = 0; i < core->num_parents; i++) {
- core->parent_names[i] = kstrdup_const(hw->init->parent_names[i],
- GFP_KERNEL);
- if (!core->parent_names[i]) {
- ret = -ENOMEM;
- goto fail_parent_names_copy;
- }
- }
-
- /* avoid unnecessary string look-ups of clk_core's possible parents. */
- core->parents = kcalloc(core->num_parents, sizeof(*core->parents),
- GFP_KERNEL);
- if (!core->parents) {
- ret = -ENOMEM;
+ ret = clk_core_populate_parent_map(core);
+ if (ret)
goto fail_parents;
- };
INIT_HLIST_HEAD(&core->clks);
hw->clk = alloc_clk(core, NULL, NULL);
if (IS_ERR(hw->clk)) {
ret = PTR_ERR(hw->clk);
- goto fail_parents;
+ goto fail_create_clk;
}
clk_core_link_consumer(hw->core, hw->clk);
free_clk(hw->clk);
hw->clk = NULL;
+fail_create_clk:
+ clk_core_free_parent_map(core);
fail_parents:
- kfree(core->parents);
-fail_parent_names_copy:
- while (--i >= 0)
- kfree_const(core->parent_names[i]);
- kfree(core->parent_names);
-fail_parent_names:
fail_ops:
kfree_const(core->name);
fail_name:
fail_out:
return ERR_PTR(ret);
}
+
+/**
+ * clk_register - allocate a new clock, register it and return an opaque cookie
+ * @dev: device that is registering this clock
+ * @hw: link to hardware-specific clock data
+ *
+ * clk_register is the *deprecated* interface for populating the clock tree with
+ * new clock nodes. Use clk_hw_register() instead.
+ *
+ * Returns: a pointer to the newly allocated struct clk which
+ * cannot be dereferenced by driver code but may be used in conjunction with the
+ * rest of the clock API. In the event of an error clk_register will return an
+ * error code; drivers must test for an error code after calling clk_register.
+ */
+struct clk *clk_register(struct device *dev, struct clk_hw *hw)
+{
+ return __clk_register(dev, dev_of_node(dev), hw);
+}
EXPORT_SYMBOL_GPL(clk_register);
/**
*/
int clk_hw_register(struct device *dev, struct clk_hw *hw)
{
- return PTR_ERR_OR_ZERO(clk_register(dev, hw));
+ return PTR_ERR_OR_ZERO(__clk_register(dev, dev_of_node(dev), hw));
}
EXPORT_SYMBOL_GPL(clk_hw_register);
+/*
+ * of_clk_hw_register - register a clk_hw and return an error code
+ * @node: device_node of device that is registering this clock
+ * @hw: link to hardware-specific clock data
+ *
+ * of_clk_hw_register() is the primary interface for populating the clock tree
+ * with new clock nodes when a struct device is not available, but a struct
+ * device_node is. It returns an integer equal to zero indicating success or
+ * less than zero indicating failure. Drivers must test for an error code after
+ * calling of_clk_hw_register().
+ */
+int of_clk_hw_register(struct device_node *node, struct clk_hw *hw)
+{
+ return PTR_ERR_OR_ZERO(__clk_register(NULL, node, hw));
+}
+EXPORT_SYMBOL_GPL(of_clk_hw_register);
+
/* Free memory allocated for a clock. */
static void __clk_release(struct kref *ref)
{
struct clk_core *core = container_of(ref, struct clk_core, ref);
- int i = core->num_parents;
lockdep_assert_held(&prepare_lock);
- kfree(core->parents);
- while (--i >= 0)
- kfree_const(core->parent_names[i]);
-
- kfree(core->parent_names);
+ clk_core_free_parent_map(core);
kfree_const(core->name);
kfree(core);
}
* @dev: device that is registering this clock
* @hw: link to hardware-specific clock data
*
- * Managed clk_register(). Clocks returned from this function are
- * automatically clk_unregister()ed on driver detach. See clk_register() for
- * more information.
+ * Managed clk_register(). This function is *deprecated*, use devm_clk_hw_register() instead.
+ *
+ * Clocks returned from this function are automatically clk_unregister()ed on
+ * driver detach. See clk_register() for more information.
*/
struct clk *devm_clk_register(struct device *dev, struct clk_hw *hw)
{
* @np: Device node pointer associated with clock provider
* @clk_src_get: callback for decoding clock
* @data: context pointer for @clk_src_get callback.
+ *
+ * This function is *deprecated*. Use of_clk_add_hw_provider() instead.
*/
int of_clk_add_provider(struct device_node *np,
struct clk *(*clk_src_get)(struct of_phandle_args *clkspec,
}
#endif
+struct clk_hw *clk_find_hw(const char *dev_id, const char *con_id);
+
#ifdef CONFIG_COMMON_CLK
struct clk *clk_hw_create_clk(struct device *dev, struct clk_hw *hw,
const char *dev_id, const char *con_id);
return cl;
}
-static struct clk *__clk_get_sys(struct device *dev, const char *dev_id,
- const char *con_id)
+struct clk_hw *clk_find_hw(const char *dev_id, const char *con_id)
{
struct clk_lookup *cl;
- struct clk *clk = NULL;
+ struct clk_hw *hw = ERR_PTR(-ENOENT);
mutex_lock(&clocks_mutex);
-
cl = clk_find(dev_id, con_id);
- if (!cl)
- goto out;
-
- clk = clk_hw_create_clk(dev, cl->clk_hw, dev_id, con_id);
- if (IS_ERR(clk))
- cl = NULL;
-out:
+ if (cl)
+ hw = cl->clk_hw;
mutex_unlock(&clocks_mutex);
- return cl ? clk : ERR_PTR(-ENOENT);
+ return hw;
+}
+
+static struct clk *__clk_get_sys(struct device *dev, const char *dev_id,
+ const char *con_id)
+{
+ struct clk_hw *hw = clk_find_hw(dev_id, con_id);
+
+ return clk_hw_create_clk(dev, hw, dev_id, con_id);
}
struct clk *clk_get_sys(const char *dev_id, const char *con_id)
struct da8xx_cfgchip_gate_clk *gate;
gate = da8xx_cfgchip_gate_clk_register(dev, &da8xx_div4p5ena_info, regmap);
- if (IS_ERR(gate))
- return PTR_ERR(gate);
- return 0;
+ return PTR_ERR_OR_ZERO(gate);
}
static int __init
-// SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-2.0 */
/*
* Clock driver for TI Davinci PSC controllers
*
-// SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-2.0 */
/*
* Clock driver for TI Davinci PSC controllers
*
"clk_isp_snclk_mux", CLK_SET_RATE_PARENT, 0x50, 17, 0, },
{ HI3660_CLK_GATE_ISP_SNCLK2, "clk_gate_isp_snclk2",
"clk_isp_snclk_mux", CLK_SET_RATE_PARENT, 0x50, 18, 0, },
+ /*
+ * clk_gate_ufs_subsys is a system bus clock, mark it as critical
+ * clock and keep it on for system suspend and resume.
+ */
{ HI3660_CLK_GATE_UFS_SUBSYS, "clk_gate_ufs_subsys", "clk_div_sysbus",
- CLK_SET_RATE_PARENT, 0x50, 21, 0, },
+ CLK_SET_RATE_PARENT | CLK_IS_CRITICAL, 0x50, 21, 0, },
{ HI3660_PCLK_GATE_DSI0, "pclk_gate_dsi0", "clk_div_cfgbus",
CLK_SET_RATE_PARENT, 0x50, 28, 0, },
{ HI3660_PCLK_GATE_DSI1, "pclk_gate_dsi1", "clk_div_cfgbus",
spin_lock_irqsave(phase->lock, flags);
- val = clk_readl(phase->reg);
+ val = readl(phase->reg);
val &= ~phase->mask;
val |= regval << phase->shift;
- clk_writel(val, phase->reg);
+ writel(val, phase->reg);
spin_unlock_irqrestore(phase->lock, flags);
obj-$(CONFIG_SOC_IMX27) += clk-imx27.o
obj-$(CONFIG_SOC_IMX31) += clk-imx31.o
obj-$(CONFIG_SOC_IMX35) += clk-imx35.o
-obj-$(CONFIG_SOC_IMX5) += clk-imx51-imx53.o
+obj-$(CONFIG_SOC_IMX5) += clk-imx5.o
obj-$(CONFIG_SOC_IMX6Q) += clk-imx6q.o
obj-$(CONFIG_SOC_IMX6SL) += clk-imx6sl.o
obj-$(CONFIG_SOC_IMX6SLL) += clk-imx6sll.o
struct clk_divider *div = to_clk_divider(hw);
unsigned int val;
- val = clk_readl(div->reg) >> div->shift;
+ val = readl(div->reg) >> div->shift;
val &= clk_div_mask(div->width);
if (!val)
return 0;
if (!clk_hw_is_enabled(hw)) {
val = div_gate->cached_val;
} else {
- val = clk_readl(div->reg) >> div->shift;
+ val = readl(div->reg) >> div->shift;
val &= clk_div_mask(div->width);
}
spin_lock_irqsave(div->lock, flags);
if (clk_hw_is_enabled(hw)) {
- val = clk_readl(div->reg);
+ val = readl(div->reg);
val &= ~(clk_div_mask(div->width) << div->shift);
val |= (u32)value << div->shift;
- clk_writel(val, div->reg);
+ writel(val, div->reg);
} else {
div_gate->cached_val = value;
}
spin_lock_irqsave(div->lock, flags);
/* restore div val */
- val = clk_readl(div->reg);
+ val = readl(div->reg);
val |= div_gate->cached_val << div->shift;
- clk_writel(val, div->reg);
+ writel(val, div->reg);
spin_unlock_irqrestore(div->lock, flags);
spin_lock_irqsave(div->lock, flags);
/* store the current div val */
- val = clk_readl(div->reg) >> div->shift;
+ val = readl(div->reg) >> div->shift;
val &= clk_div_mask(div->width);
div_gate->cached_val = val;
- clk_writel(0, div->reg);
+ writel(0, div->reg);
spin_unlock_irqrestore(div->lock, flags);
}
struct clk_divider *div = to_clk_divider(hw);
u32 val;
- val = clk_readl(div->reg) >> div->shift;
+ val = readl(div->reg) >> div->shift;
val &= clk_div_mask(div->width);
return val ? 1 : 0;
div_gate->divider.hw.init = &init;
div_gate->divider.flags = CLK_DIVIDER_ONE_BASED | clk_divider_flags;
/* cache gate status */
- val = clk_readl(reg) >> shift;
+ val = readl(reg) >> shift;
val &= clk_div_mask(width);
div_gate->cached_val = val;
--- /dev/null
+/*
+ * Copyright (C) 2011 Sascha Hauer, Pengutronix <s.hauer@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+#include <linux/mm.h>
+#include <linux/delay.h>
+#include <linux/clk.h>
+#include <linux/io.h>
+#include <linux/clkdev.h>
+#include <linux/clk-provider.h>
+#include <linux/err.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/sizes.h>
+#include <soc/imx/revision.h>
+#include <dt-bindings/clock/imx5-clock.h>
+
+#include "clk.h"
+
+#define MX51_DPLL1_BASE 0x83f80000
+#define MX51_DPLL2_BASE 0x83f84000
+#define MX51_DPLL3_BASE 0x83f88000
+
+#define MX53_DPLL1_BASE 0x63f80000
+#define MX53_DPLL2_BASE 0x63f84000
+#define MX53_DPLL3_BASE 0x63f88000
+#define MX53_DPLL4_BASE 0x63f8c000
+
+#define MXC_CCM_CCR (ccm_base + 0x00)
+#define MXC_CCM_CCDR (ccm_base + 0x04)
+#define MXC_CCM_CSR (ccm_base + 0x08)
+#define MXC_CCM_CCSR (ccm_base + 0x0c)
+#define MXC_CCM_CACRR (ccm_base + 0x10)
+#define MXC_CCM_CBCDR (ccm_base + 0x14)
+#define MXC_CCM_CBCMR (ccm_base + 0x18)
+#define MXC_CCM_CSCMR1 (ccm_base + 0x1c)
+#define MXC_CCM_CSCMR2 (ccm_base + 0x20)
+#define MXC_CCM_CSCDR1 (ccm_base + 0x24)
+#define MXC_CCM_CS1CDR (ccm_base + 0x28)
+#define MXC_CCM_CS2CDR (ccm_base + 0x2c)
+#define MXC_CCM_CDCDR (ccm_base + 0x30)
+#define MXC_CCM_CHSCDR (ccm_base + 0x34)
+#define MXC_CCM_CSCDR2 (ccm_base + 0x38)
+#define MXC_CCM_CSCDR3 (ccm_base + 0x3c)
+#define MXC_CCM_CSCDR4 (ccm_base + 0x40)
+#define MXC_CCM_CWDR (ccm_base + 0x44)
+#define MXC_CCM_CDHIPR (ccm_base + 0x48)
+#define MXC_CCM_CDCR (ccm_base + 0x4c)
+#define MXC_CCM_CTOR (ccm_base + 0x50)
+#define MXC_CCM_CLPCR (ccm_base + 0x54)
+#define MXC_CCM_CISR (ccm_base + 0x58)
+#define MXC_CCM_CIMR (ccm_base + 0x5c)
+#define MXC_CCM_CCOSR (ccm_base + 0x60)
+#define MXC_CCM_CGPR (ccm_base + 0x64)
+#define MXC_CCM_CCGR0 (ccm_base + 0x68)
+#define MXC_CCM_CCGR1 (ccm_base + 0x6c)
+#define MXC_CCM_CCGR2 (ccm_base + 0x70)
+#define MXC_CCM_CCGR3 (ccm_base + 0x74)
+#define MXC_CCM_CCGR4 (ccm_base + 0x78)
+#define MXC_CCM_CCGR5 (ccm_base + 0x7c)
+#define MXC_CCM_CCGR6 (ccm_base + 0x80)
+#define MXC_CCM_CCGR7 (ccm_base + 0x84)
+
+/* Low-power Audio Playback Mode clock */
+static const char *lp_apm_sel[] = { "osc", };
+
+/* This is used multiple times */
+static const char *standard_pll_sel[] = { "pll1_sw", "pll2_sw", "pll3_sw", "lp_apm", };
+static const char *periph_apm_sel[] = { "pll1_sw", "pll3_sw", "lp_apm", };
+static const char *main_bus_sel[] = { "pll2_sw", "periph_apm", };
+static const char *per_lp_apm_sel[] = { "main_bus", "lp_apm", };
+static const char *per_root_sel[] = { "per_podf", "ipg", };
+static const char *esdhc_c_sel[] = { "esdhc_a_podf", "esdhc_b_podf", };
+static const char *esdhc_d_sel[] = { "esdhc_a_podf", "esdhc_b_podf", };
+static const char *ssi_apm_sels[] = { "ckih1", "lp_amp", "ckih2", };
+static const char *ssi_clk_sels[] = { "pll1_sw", "pll2_sw", "pll3_sw", "ssi_apm", };
+static const char *ssi3_clk_sels[] = { "ssi1_root_gate", "ssi2_root_gate", };
+static const char *ssi_ext1_com_sels[] = { "ssi_ext1_podf", "ssi1_root_gate", };
+static const char *ssi_ext2_com_sels[] = { "ssi_ext2_podf", "ssi2_root_gate", };
+static const char *emi_slow_sel[] = { "main_bus", "ahb", };
+static const char *usb_phy_sel_str[] = { "osc", "usb_phy_podf", };
+static const char *mx51_ipu_di0_sel[] = { "di_pred", "osc", "ckih1", "tve_di", };
+static const char *mx53_ipu_di0_sel[] = { "di_pred", "osc", "ckih1", "di_pll4_podf", "dummy", "ldb_di0_gate", };
+static const char *mx53_ldb_di0_sel[] = { "pll3_sw", "pll4_sw", };
+static const char *mx51_ipu_di1_sel[] = { "di_pred", "osc", "ckih1", "tve_di", "ipp_di1", };
+static const char *mx53_ipu_di1_sel[] = { "di_pred", "osc", "ckih1", "tve_di", "ipp_di1", "ldb_di1_gate", };
+static const char *mx53_ldb_di1_sel[] = { "pll3_sw", "pll4_sw", };
+static const char *mx51_tve_ext_sel[] = { "osc", "ckih1", };
+static const char *mx53_tve_ext_sel[] = { "pll4_sw", "ckih1", };
+static const char *mx51_tve_sel[] = { "tve_pred", "tve_ext_sel", };
+static const char *ipu_sel[] = { "axi_a", "axi_b", "emi_slow_gate", "ahb", };
+static const char *gpu3d_sel[] = { "axi_a", "axi_b", "emi_slow_gate", "ahb" };
+static const char *gpu2d_sel[] = { "axi_a", "axi_b", "emi_slow_gate", "ahb" };
+static const char *vpu_sel[] = { "axi_a", "axi_b", "emi_slow_gate", "ahb", };
+static const char *mx53_can_sel[] = { "ipg", "ckih1", "ckih2", "lp_apm", };
+static const char *mx53_cko1_sel[] = {
+ "cpu_podf", "pll1_sw", "pll2_sw", "pll3_sw",
+ "emi_slow_podf", "pll4_sw", "nfc_podf", "dummy",
+ "di_pred", "dummy", "dummy", "ahb",
+ "ipg", "per_root", "ckil", "dummy",};
+static const char *mx53_cko2_sel[] = {
+ "dummy"/* dptc_core */, "dummy"/* dptc_perich */,
+ "dummy", "esdhc_a_podf",
+ "usboh3_podf", "dummy"/* wrck_clk_root */,
+ "ecspi_podf", "dummy"/* pll1_ref_clk */,
+ "esdhc_b_podf", "dummy"/* ddr_clk_root */,
+ "dummy"/* arm_axi_clk_root */, "dummy"/* usb_phy_out */,
+ "vpu_sel", "ipu_sel",
+ "osc", "ckih1",
+ "dummy", "esdhc_c_sel",
+ "ssi1_root_podf", "ssi2_root_podf",
+ "dummy", "dummy",
+ "dummy"/* lpsr_clk_root */, "dummy"/* pgc_clk_root */,
+ "dummy"/* tve_out */, "usb_phy_sel",
+ "tve_sel", "lp_apm",
+ "uart_root", "dummy"/* spdif0_clk_root */,
+ "dummy", "dummy", };
+static const char *mx51_spdif_xtal_sel[] = { "osc", "ckih", "ckih2", };
+static const char *mx53_spdif_xtal_sel[] = { "osc", "ckih", "ckih2", "pll4_sw", };
+static const char *spdif_sel[] = { "pll1_sw", "pll2_sw", "pll3_sw", "spdif_xtal_sel", };
+static const char *spdif0_com_sel[] = { "spdif0_podf", "ssi1_root_gate", };
+static const char *mx51_spdif1_com_sel[] = { "spdif1_podf", "ssi2_root_gate", };
+static const char *step_sels[] = { "lp_apm", };
+static const char *cpu_podf_sels[] = { "pll1_sw", "step_sel" };
+static const char *ieee1588_sels[] = { "pll3_sw", "pll4_sw", "dummy" /* usbphy2_clk */, "dummy" /* fec_phy_clk */ };
+
+static struct clk *clk[IMX5_CLK_END];
+static struct clk_onecell_data clk_data;
+
+static struct clk ** const uart_clks_mx51[] __initconst = {
+ &clk[IMX5_CLK_UART1_IPG_GATE],
+ &clk[IMX5_CLK_UART1_PER_GATE],
+ &clk[IMX5_CLK_UART2_IPG_GATE],
+ &clk[IMX5_CLK_UART2_PER_GATE],
+ &clk[IMX5_CLK_UART3_IPG_GATE],
+ &clk[IMX5_CLK_UART3_PER_GATE],
+ NULL
+};
+
+static struct clk ** const uart_clks_mx50_mx53[] __initconst = {
+ &clk[IMX5_CLK_UART1_IPG_GATE],
+ &clk[IMX5_CLK_UART1_PER_GATE],
+ &clk[IMX5_CLK_UART2_IPG_GATE],
+ &clk[IMX5_CLK_UART2_PER_GATE],
+ &clk[IMX5_CLK_UART3_IPG_GATE],
+ &clk[IMX5_CLK_UART3_PER_GATE],
+ &clk[IMX5_CLK_UART4_IPG_GATE],
+ &clk[IMX5_CLK_UART4_PER_GATE],
+ &clk[IMX5_CLK_UART5_IPG_GATE],
+ &clk[IMX5_CLK_UART5_PER_GATE],
+ NULL
+};
+
+static void __init mx5_clocks_common_init(void __iomem *ccm_base)
+{
+ clk[IMX5_CLK_DUMMY] = imx_clk_fixed("dummy", 0);
+ clk[IMX5_CLK_CKIL] = imx_obtain_fixed_clock("ckil", 0);
+ clk[IMX5_CLK_OSC] = imx_obtain_fixed_clock("osc", 0);
+ clk[IMX5_CLK_CKIH1] = imx_obtain_fixed_clock("ckih1", 0);
+ clk[IMX5_CLK_CKIH2] = imx_obtain_fixed_clock("ckih2", 0);
+
+ clk[IMX5_CLK_PER_LP_APM] = imx_clk_mux("per_lp_apm", MXC_CCM_CBCMR, 1, 1,
+ per_lp_apm_sel, ARRAY_SIZE(per_lp_apm_sel));
+ clk[IMX5_CLK_PER_PRED1] = imx_clk_divider("per_pred1", "per_lp_apm", MXC_CCM_CBCDR, 6, 2);
+ clk[IMX5_CLK_PER_PRED2] = imx_clk_divider("per_pred2", "per_pred1", MXC_CCM_CBCDR, 3, 3);
+ clk[IMX5_CLK_PER_PODF] = imx_clk_divider("per_podf", "per_pred2", MXC_CCM_CBCDR, 0, 3);
+ clk[IMX5_CLK_PER_ROOT] = imx_clk_mux("per_root", MXC_CCM_CBCMR, 0, 1,
+ per_root_sel, ARRAY_SIZE(per_root_sel));
+ clk[IMX5_CLK_AHB] = imx_clk_divider("ahb", "main_bus", MXC_CCM_CBCDR, 10, 3);
+ clk[IMX5_CLK_AHB_MAX] = imx_clk_gate2_flags("ahb_max", "ahb", MXC_CCM_CCGR0, 28, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_AIPS_TZ1] = imx_clk_gate2_flags("aips_tz1", "ahb", MXC_CCM_CCGR0, 24, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_AIPS_TZ2] = imx_clk_gate2_flags("aips_tz2", "ahb", MXC_CCM_CCGR0, 26, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_TMAX1] = imx_clk_gate2_flags("tmax1", "ahb", MXC_CCM_CCGR1, 0, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_TMAX2] = imx_clk_gate2_flags("tmax2", "ahb", MXC_CCM_CCGR1, 2, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_TMAX3] = imx_clk_gate2_flags("tmax3", "ahb", MXC_CCM_CCGR1, 4, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_SPBA] = imx_clk_gate2_flags("spba", "ipg", MXC_CCM_CCGR5, 0, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_IPG] = imx_clk_divider("ipg", "ahb", MXC_CCM_CBCDR, 8, 2);
+ clk[IMX5_CLK_AXI_A] = imx_clk_divider("axi_a", "main_bus", MXC_CCM_CBCDR, 16, 3);
+ clk[IMX5_CLK_AXI_B] = imx_clk_divider("axi_b", "main_bus", MXC_CCM_CBCDR, 19, 3);
+ clk[IMX5_CLK_UART_SEL] = imx_clk_mux("uart_sel", MXC_CCM_CSCMR1, 24, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_UART_PRED] = imx_clk_divider("uart_pred", "uart_sel", MXC_CCM_CSCDR1, 3, 3);
+ clk[IMX5_CLK_UART_ROOT] = imx_clk_divider("uart_root", "uart_pred", MXC_CCM_CSCDR1, 0, 3);
+
+ clk[IMX5_CLK_ESDHC_A_PRED] = imx_clk_divider("esdhc_a_pred", "esdhc_a_sel", MXC_CCM_CSCDR1, 16, 3);
+ clk[IMX5_CLK_ESDHC_A_PODF] = imx_clk_divider("esdhc_a_podf", "esdhc_a_pred", MXC_CCM_CSCDR1, 11, 3);
+ clk[IMX5_CLK_ESDHC_B_PRED] = imx_clk_divider("esdhc_b_pred", "esdhc_b_sel", MXC_CCM_CSCDR1, 22, 3);
+ clk[IMX5_CLK_ESDHC_B_PODF] = imx_clk_divider("esdhc_b_podf", "esdhc_b_pred", MXC_CCM_CSCDR1, 19, 3);
+
+ clk[IMX5_CLK_EMI_SEL] = imx_clk_mux("emi_sel", MXC_CCM_CBCDR, 26, 1,
+ emi_slow_sel, ARRAY_SIZE(emi_slow_sel));
+ clk[IMX5_CLK_EMI_SLOW_PODF] = imx_clk_divider("emi_slow_podf", "emi_sel", MXC_CCM_CBCDR, 22, 3);
+ clk[IMX5_CLK_NFC_PODF] = imx_clk_divider("nfc_podf", "emi_slow_podf", MXC_CCM_CBCDR, 13, 3);
+ clk[IMX5_CLK_ECSPI_SEL] = imx_clk_mux("ecspi_sel", MXC_CCM_CSCMR1, 4, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_ECSPI_PRED] = imx_clk_divider("ecspi_pred", "ecspi_sel", MXC_CCM_CSCDR2, 25, 3);
+ clk[IMX5_CLK_ECSPI_PODF] = imx_clk_divider("ecspi_podf", "ecspi_pred", MXC_CCM_CSCDR2, 19, 6);
+ clk[IMX5_CLK_USBOH3_SEL] = imx_clk_mux("usboh3_sel", MXC_CCM_CSCMR1, 22, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_USBOH3_PRED] = imx_clk_divider("usboh3_pred", "usboh3_sel", MXC_CCM_CSCDR1, 8, 3);
+ clk[IMX5_CLK_USBOH3_PODF] = imx_clk_divider("usboh3_podf", "usboh3_pred", MXC_CCM_CSCDR1, 6, 2);
+ clk[IMX5_CLK_USB_PHY_PRED] = imx_clk_divider("usb_phy_pred", "pll3_sw", MXC_CCM_CDCDR, 3, 3);
+ clk[IMX5_CLK_USB_PHY_PODF] = imx_clk_divider("usb_phy_podf", "usb_phy_pred", MXC_CCM_CDCDR, 0, 3);
+ clk[IMX5_CLK_USB_PHY_SEL] = imx_clk_mux("usb_phy_sel", MXC_CCM_CSCMR1, 26, 1,
+ usb_phy_sel_str, ARRAY_SIZE(usb_phy_sel_str));
+ clk[IMX5_CLK_STEP_SEL] = imx_clk_mux("step_sel", MXC_CCM_CCSR, 7, 2, step_sels, ARRAY_SIZE(step_sels));
+ clk[IMX5_CLK_CPU_PODF_SEL] = imx_clk_mux("cpu_podf_sel", MXC_CCM_CCSR, 2, 1, cpu_podf_sels, ARRAY_SIZE(cpu_podf_sels));
+ clk[IMX5_CLK_CPU_PODF] = imx_clk_divider("cpu_podf", "cpu_podf_sel", MXC_CCM_CACRR, 0, 3);
+ clk[IMX5_CLK_DI_PRED] = imx_clk_divider("di_pred", "pll3_sw", MXC_CCM_CDCDR, 6, 3);
+ clk[IMX5_CLK_IIM_GATE] = imx_clk_gate2("iim_gate", "ipg", MXC_CCM_CCGR0, 30);
+ clk[IMX5_CLK_UART1_IPG_GATE] = imx_clk_gate2("uart1_ipg_gate", "ipg", MXC_CCM_CCGR1, 6);
+ clk[IMX5_CLK_UART1_PER_GATE] = imx_clk_gate2("uart1_per_gate", "uart_root", MXC_CCM_CCGR1, 8);
+ clk[IMX5_CLK_UART2_IPG_GATE] = imx_clk_gate2("uart2_ipg_gate", "ipg", MXC_CCM_CCGR1, 10);
+ clk[IMX5_CLK_UART2_PER_GATE] = imx_clk_gate2("uart2_per_gate", "uart_root", MXC_CCM_CCGR1, 12);
+ clk[IMX5_CLK_UART3_IPG_GATE] = imx_clk_gate2("uart3_ipg_gate", "ipg", MXC_CCM_CCGR1, 14);
+ clk[IMX5_CLK_UART3_PER_GATE] = imx_clk_gate2("uart3_per_gate", "uart_root", MXC_CCM_CCGR1, 16);
+ clk[IMX5_CLK_I2C1_GATE] = imx_clk_gate2("i2c1_gate", "per_root", MXC_CCM_CCGR1, 18);
+ clk[IMX5_CLK_I2C2_GATE] = imx_clk_gate2("i2c2_gate", "per_root", MXC_CCM_CCGR1, 20);
+ clk[IMX5_CLK_PWM1_IPG_GATE] = imx_clk_gate2("pwm1_ipg_gate", "ipg", MXC_CCM_CCGR2, 10);
+ clk[IMX5_CLK_PWM1_HF_GATE] = imx_clk_gate2("pwm1_hf_gate", "per_root", MXC_CCM_CCGR2, 12);
+ clk[IMX5_CLK_PWM2_IPG_GATE] = imx_clk_gate2("pwm2_ipg_gate", "ipg", MXC_CCM_CCGR2, 14);
+ clk[IMX5_CLK_PWM2_HF_GATE] = imx_clk_gate2("pwm2_hf_gate", "per_root", MXC_CCM_CCGR2, 16);
+ clk[IMX5_CLK_GPT_IPG_GATE] = imx_clk_gate2("gpt_ipg_gate", "ipg", MXC_CCM_CCGR2, 18);
+ clk[IMX5_CLK_GPT_HF_GATE] = imx_clk_gate2("gpt_hf_gate", "per_root", MXC_CCM_CCGR2, 20);
+ clk[IMX5_CLK_FEC_GATE] = imx_clk_gate2("fec_gate", "ipg", MXC_CCM_CCGR2, 24);
+ clk[IMX5_CLK_USBOH3_GATE] = imx_clk_gate2("usboh3_gate", "ipg", MXC_CCM_CCGR2, 26);
+ clk[IMX5_CLK_USBOH3_PER_GATE] = imx_clk_gate2("usboh3_per_gate", "usboh3_podf", MXC_CCM_CCGR2, 28);
+ clk[IMX5_CLK_ESDHC1_IPG_GATE] = imx_clk_gate2("esdhc1_ipg_gate", "ipg", MXC_CCM_CCGR3, 0);
+ clk[IMX5_CLK_ESDHC2_IPG_GATE] = imx_clk_gate2("esdhc2_ipg_gate", "ipg", MXC_CCM_CCGR3, 4);
+ clk[IMX5_CLK_ESDHC3_IPG_GATE] = imx_clk_gate2("esdhc3_ipg_gate", "ipg", MXC_CCM_CCGR3, 8);
+ clk[IMX5_CLK_ESDHC4_IPG_GATE] = imx_clk_gate2("esdhc4_ipg_gate", "ipg", MXC_CCM_CCGR3, 12);
+ clk[IMX5_CLK_SSI1_IPG_GATE] = imx_clk_gate2("ssi1_ipg_gate", "ipg", MXC_CCM_CCGR3, 16);
+ clk[IMX5_CLK_SSI2_IPG_GATE] = imx_clk_gate2("ssi2_ipg_gate", "ipg", MXC_CCM_CCGR3, 20);
+ clk[IMX5_CLK_SSI3_IPG_GATE] = imx_clk_gate2("ssi3_ipg_gate", "ipg", MXC_CCM_CCGR3, 24);
+ clk[IMX5_CLK_ECSPI1_IPG_GATE] = imx_clk_gate2("ecspi1_ipg_gate", "ipg", MXC_CCM_CCGR4, 18);
+ clk[IMX5_CLK_ECSPI1_PER_GATE] = imx_clk_gate2("ecspi1_per_gate", "ecspi_podf", MXC_CCM_CCGR4, 20);
+ clk[IMX5_CLK_ECSPI2_IPG_GATE] = imx_clk_gate2("ecspi2_ipg_gate", "ipg", MXC_CCM_CCGR4, 22);
+ clk[IMX5_CLK_ECSPI2_PER_GATE] = imx_clk_gate2("ecspi2_per_gate", "ecspi_podf", MXC_CCM_CCGR4, 24);
+ clk[IMX5_CLK_CSPI_IPG_GATE] = imx_clk_gate2("cspi_ipg_gate", "ipg", MXC_CCM_CCGR4, 26);
+ clk[IMX5_CLK_SDMA_GATE] = imx_clk_gate2("sdma_gate", "ipg", MXC_CCM_CCGR4, 30);
+ clk[IMX5_CLK_EMI_FAST_GATE] = imx_clk_gate2_flags("emi_fast_gate", "dummy", MXC_CCM_CCGR5, 14, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_EMI_SLOW_GATE] = imx_clk_gate2_flags("emi_slow_gate", "emi_slow_podf", MXC_CCM_CCGR5, 16, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_IPU_SEL] = imx_clk_mux("ipu_sel", MXC_CCM_CBCMR, 6, 2, ipu_sel, ARRAY_SIZE(ipu_sel));
+ clk[IMX5_CLK_IPU_GATE] = imx_clk_gate2("ipu_gate", "ipu_sel", MXC_CCM_CCGR5, 10);
+ clk[IMX5_CLK_NFC_GATE] = imx_clk_gate2("nfc_gate", "nfc_podf", MXC_CCM_CCGR5, 20);
+ clk[IMX5_CLK_IPU_DI0_GATE] = imx_clk_gate2("ipu_di0_gate", "ipu_di0_sel", MXC_CCM_CCGR6, 10);
+ clk[IMX5_CLK_IPU_DI1_GATE] = imx_clk_gate2("ipu_di1_gate", "ipu_di1_sel", MXC_CCM_CCGR6, 12);
+ clk[IMX5_CLK_GPU3D_SEL] = imx_clk_mux("gpu3d_sel", MXC_CCM_CBCMR, 4, 2, gpu3d_sel, ARRAY_SIZE(gpu3d_sel));
+ clk[IMX5_CLK_GPU2D_SEL] = imx_clk_mux("gpu2d_sel", MXC_CCM_CBCMR, 16, 2, gpu2d_sel, ARRAY_SIZE(gpu2d_sel));
+ clk[IMX5_CLK_GPU3D_GATE] = imx_clk_gate2("gpu3d_gate", "gpu3d_sel", MXC_CCM_CCGR5, 2);
+ clk[IMX5_CLK_GARB_GATE] = imx_clk_gate2("garb_gate", "axi_a", MXC_CCM_CCGR5, 4);
+ clk[IMX5_CLK_GPU2D_GATE] = imx_clk_gate2("gpu2d_gate", "gpu2d_sel", MXC_CCM_CCGR6, 14);
+ clk[IMX5_CLK_VPU_SEL] = imx_clk_mux("vpu_sel", MXC_CCM_CBCMR, 14, 2, vpu_sel, ARRAY_SIZE(vpu_sel));
+ clk[IMX5_CLK_VPU_GATE] = imx_clk_gate2("vpu_gate", "vpu_sel", MXC_CCM_CCGR5, 6);
+ clk[IMX5_CLK_VPU_REFERENCE_GATE] = imx_clk_gate2("vpu_reference_gate", "osc", MXC_CCM_CCGR5, 8);
+ clk[IMX5_CLK_GPC_DVFS] = imx_clk_gate2_flags("gpc_dvfs", "dummy", MXC_CCM_CCGR5, 24, CLK_IS_CRITICAL);
+
+ clk[IMX5_CLK_SSI_APM] = imx_clk_mux("ssi_apm", MXC_CCM_CSCMR1, 8, 2, ssi_apm_sels, ARRAY_SIZE(ssi_apm_sels));
+ clk[IMX5_CLK_SSI1_ROOT_SEL] = imx_clk_mux("ssi1_root_sel", MXC_CCM_CSCMR1, 14, 2, ssi_clk_sels, ARRAY_SIZE(ssi_clk_sels));
+ clk[IMX5_CLK_SSI2_ROOT_SEL] = imx_clk_mux("ssi2_root_sel", MXC_CCM_CSCMR1, 12, 2, ssi_clk_sels, ARRAY_SIZE(ssi_clk_sels));
+ clk[IMX5_CLK_SSI3_ROOT_SEL] = imx_clk_mux("ssi3_root_sel", MXC_CCM_CSCMR1, 11, 1, ssi3_clk_sels, ARRAY_SIZE(ssi3_clk_sels));
+ clk[IMX5_CLK_SSI_EXT1_SEL] = imx_clk_mux("ssi_ext1_sel", MXC_CCM_CSCMR1, 28, 2, ssi_clk_sels, ARRAY_SIZE(ssi_clk_sels));
+ clk[IMX5_CLK_SSI_EXT2_SEL] = imx_clk_mux("ssi_ext2_sel", MXC_CCM_CSCMR1, 30, 2, ssi_clk_sels, ARRAY_SIZE(ssi_clk_sels));
+ clk[IMX5_CLK_SSI_EXT1_COM_SEL] = imx_clk_mux("ssi_ext1_com_sel", MXC_CCM_CSCMR1, 0, 1, ssi_ext1_com_sels, ARRAY_SIZE(ssi_ext1_com_sels));
+ clk[IMX5_CLK_SSI_EXT2_COM_SEL] = imx_clk_mux("ssi_ext2_com_sel", MXC_CCM_CSCMR1, 1, 1, ssi_ext2_com_sels, ARRAY_SIZE(ssi_ext2_com_sels));
+ clk[IMX5_CLK_SSI1_ROOT_PRED] = imx_clk_divider("ssi1_root_pred", "ssi1_root_sel", MXC_CCM_CS1CDR, 6, 3);
+ clk[IMX5_CLK_SSI1_ROOT_PODF] = imx_clk_divider("ssi1_root_podf", "ssi1_root_pred", MXC_CCM_CS1CDR, 0, 6);
+ clk[IMX5_CLK_SSI2_ROOT_PRED] = imx_clk_divider("ssi2_root_pred", "ssi2_root_sel", MXC_CCM_CS2CDR, 6, 3);
+ clk[IMX5_CLK_SSI2_ROOT_PODF] = imx_clk_divider("ssi2_root_podf", "ssi2_root_pred", MXC_CCM_CS2CDR, 0, 6);
+ clk[IMX5_CLK_SSI_EXT1_PRED] = imx_clk_divider("ssi_ext1_pred", "ssi_ext1_sel", MXC_CCM_CS1CDR, 22, 3);
+ clk[IMX5_CLK_SSI_EXT1_PODF] = imx_clk_divider("ssi_ext1_podf", "ssi_ext1_pred", MXC_CCM_CS1CDR, 16, 6);
+ clk[IMX5_CLK_SSI_EXT2_PRED] = imx_clk_divider("ssi_ext2_pred", "ssi_ext2_sel", MXC_CCM_CS2CDR, 22, 3);
+ clk[IMX5_CLK_SSI_EXT2_PODF] = imx_clk_divider("ssi_ext2_podf", "ssi_ext2_pred", MXC_CCM_CS2CDR, 16, 6);
+ clk[IMX5_CLK_SSI1_ROOT_GATE] = imx_clk_gate2("ssi1_root_gate", "ssi1_root_podf", MXC_CCM_CCGR3, 18);
+ clk[IMX5_CLK_SSI2_ROOT_GATE] = imx_clk_gate2("ssi2_root_gate", "ssi2_root_podf", MXC_CCM_CCGR3, 22);
+ clk[IMX5_CLK_SSI3_ROOT_GATE] = imx_clk_gate2("ssi3_root_gate", "ssi3_root_sel", MXC_CCM_CCGR3, 26);
+ clk[IMX5_CLK_SSI_EXT1_GATE] = imx_clk_gate2("ssi_ext1_gate", "ssi_ext1_com_sel", MXC_CCM_CCGR3, 28);
+ clk[IMX5_CLK_SSI_EXT2_GATE] = imx_clk_gate2("ssi_ext2_gate", "ssi_ext2_com_sel", MXC_CCM_CCGR3, 30);
+ clk[IMX5_CLK_EPIT1_IPG_GATE] = imx_clk_gate2("epit1_ipg_gate", "ipg", MXC_CCM_CCGR2, 2);
+ clk[IMX5_CLK_EPIT1_HF_GATE] = imx_clk_gate2("epit1_hf_gate", "per_root", MXC_CCM_CCGR2, 4);
+ clk[IMX5_CLK_EPIT2_IPG_GATE] = imx_clk_gate2("epit2_ipg_gate", "ipg", MXC_CCM_CCGR2, 6);
+ clk[IMX5_CLK_EPIT2_HF_GATE] = imx_clk_gate2("epit2_hf_gate", "per_root", MXC_CCM_CCGR2, 8);
+ clk[IMX5_CLK_OWIRE_GATE] = imx_clk_gate2("owire_gate", "per_root", MXC_CCM_CCGR2, 22);
+ clk[IMX5_CLK_SRTC_GATE] = imx_clk_gate2("srtc_gate", "per_root", MXC_CCM_CCGR4, 28);
+ clk[IMX5_CLK_PATA_GATE] = imx_clk_gate2("pata_gate", "ipg", MXC_CCM_CCGR4, 0);
+ clk[IMX5_CLK_SPDIF0_SEL] = imx_clk_mux("spdif0_sel", MXC_CCM_CSCMR2, 0, 2, spdif_sel, ARRAY_SIZE(spdif_sel));
+ clk[IMX5_CLK_SPDIF0_PRED] = imx_clk_divider("spdif0_pred", "spdif0_sel", MXC_CCM_CDCDR, 25, 3);
+ clk[IMX5_CLK_SPDIF0_PODF] = imx_clk_divider("spdif0_podf", "spdif0_pred", MXC_CCM_CDCDR, 19, 6);
+ clk[IMX5_CLK_SPDIF0_COM_SEL] = imx_clk_mux_flags("spdif0_com_sel", MXC_CCM_CSCMR2, 4, 1,
+ spdif0_com_sel, ARRAY_SIZE(spdif0_com_sel), CLK_SET_RATE_PARENT);
+ clk[IMX5_CLK_SPDIF0_GATE] = imx_clk_gate2("spdif0_gate", "spdif0_com_sel", MXC_CCM_CCGR5, 26);
+ clk[IMX5_CLK_SPDIF_IPG_GATE] = imx_clk_gate2("spdif_ipg_gate", "ipg", MXC_CCM_CCGR5, 30);
+ clk[IMX5_CLK_SAHARA_IPG_GATE] = imx_clk_gate2("sahara_ipg_gate", "ipg", MXC_CCM_CCGR4, 14);
+ clk[IMX5_CLK_SATA_REF] = imx_clk_fixed_factor("sata_ref", "usb_phy1_gate", 1, 1);
+
+ clk_register_clkdev(clk[IMX5_CLK_CPU_PODF], NULL, "cpu0");
+ clk_register_clkdev(clk[IMX5_CLK_GPC_DVFS], "gpc_dvfs", NULL);
+
+ /* move usb phy clk to 24MHz */
+ clk_set_parent(clk[IMX5_CLK_USB_PHY_SEL], clk[IMX5_CLK_OSC]);
+}
+
+static void __init mx50_clocks_init(struct device_node *np)
+{
+ void __iomem *ccm_base;
+ void __iomem *pll_base;
+ unsigned long r;
+
+ pll_base = ioremap(MX53_DPLL1_BASE, SZ_16K);
+ WARN_ON(!pll_base);
+ clk[IMX5_CLK_PLL1_SW] = imx_clk_pllv2("pll1_sw", "osc", pll_base);
+
+ pll_base = ioremap(MX53_DPLL2_BASE, SZ_16K);
+ WARN_ON(!pll_base);
+ clk[IMX5_CLK_PLL2_SW] = imx_clk_pllv2("pll2_sw", "osc", pll_base);
+
+ pll_base = ioremap(MX53_DPLL3_BASE, SZ_16K);
+ WARN_ON(!pll_base);
+ clk[IMX5_CLK_PLL3_SW] = imx_clk_pllv2("pll3_sw", "osc", pll_base);
+
+ ccm_base = of_iomap(np, 0);
+ WARN_ON(!ccm_base);
+
+ mx5_clocks_common_init(ccm_base);
+
+ /*
+ * This clock is called periph_clk in the i.MX50 Reference Manual, but
+ * it comes closest in scope to the main_bus_clk of i.MX51 and i.MX53
+ */
+ clk[IMX5_CLK_MAIN_BUS] = imx_clk_mux("main_bus", MXC_CCM_CBCDR, 25, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+
+ clk[IMX5_CLK_LP_APM] = imx_clk_mux("lp_apm", MXC_CCM_CCSR, 10, 1,
+ lp_apm_sel, ARRAY_SIZE(lp_apm_sel));
+ clk[IMX5_CLK_ESDHC_A_SEL] = imx_clk_mux("esdhc_a_sel", MXC_CCM_CSCMR1, 21, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_ESDHC_B_SEL] = imx_clk_mux("esdhc_b_sel", MXC_CCM_CSCMR1, 16, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_ESDHC_C_SEL] = imx_clk_mux("esdhc_c_sel", MXC_CCM_CSCMR1, 20, 1, esdhc_c_sel, ARRAY_SIZE(esdhc_c_sel));
+ clk[IMX5_CLK_ESDHC_D_SEL] = imx_clk_mux("esdhc_d_sel", MXC_CCM_CSCMR1, 19, 1, esdhc_d_sel, ARRAY_SIZE(esdhc_d_sel));
+ clk[IMX5_CLK_ESDHC1_PER_GATE] = imx_clk_gate2("esdhc1_per_gate", "esdhc_a_podf", MXC_CCM_CCGR3, 2);
+ clk[IMX5_CLK_ESDHC2_PER_GATE] = imx_clk_gate2("esdhc2_per_gate", "esdhc_c_sel", MXC_CCM_CCGR3, 6);
+ clk[IMX5_CLK_ESDHC3_PER_GATE] = imx_clk_gate2("esdhc3_per_gate", "esdhc_b_podf", MXC_CCM_CCGR3, 10);
+ clk[IMX5_CLK_ESDHC4_PER_GATE] = imx_clk_gate2("esdhc4_per_gate", "esdhc_d_sel", MXC_CCM_CCGR3, 14);
+ clk[IMX5_CLK_USB_PHY1_GATE] = imx_clk_gate2("usb_phy1_gate", "usb_phy_sel", MXC_CCM_CCGR4, 10);
+ clk[IMX5_CLK_USB_PHY2_GATE] = imx_clk_gate2("usb_phy2_gate", "usb_phy_sel", MXC_CCM_CCGR4, 12);
+ clk[IMX5_CLK_I2C3_GATE] = imx_clk_gate2("i2c3_gate", "per_root", MXC_CCM_CCGR1, 22);
+ clk[IMX5_CLK_UART4_IPG_GATE] = imx_clk_gate2("uart4_ipg_gate", "ipg", MXC_CCM_CCGR7, 8);
+ clk[IMX5_CLK_UART4_PER_GATE] = imx_clk_gate2("uart4_per_gate", "uart_root", MXC_CCM_CCGR7, 10);
+ clk[IMX5_CLK_UART5_IPG_GATE] = imx_clk_gate2("uart5_ipg_gate", "ipg", MXC_CCM_CCGR7, 12);
+ clk[IMX5_CLK_UART5_PER_GATE] = imx_clk_gate2("uart5_per_gate", "uart_root", MXC_CCM_CCGR7, 14);
+
+ clk[IMX5_CLK_CKO1_SEL] = imx_clk_mux("cko1_sel", MXC_CCM_CCOSR, 0, 4,
+ mx53_cko1_sel, ARRAY_SIZE(mx53_cko1_sel));
+ clk[IMX5_CLK_CKO1_PODF] = imx_clk_divider("cko1_podf", "cko1_sel", MXC_CCM_CCOSR, 4, 3);
+ clk[IMX5_CLK_CKO1] = imx_clk_gate2("cko1", "cko1_podf", MXC_CCM_CCOSR, 7);
+
+ clk[IMX5_CLK_CKO2_SEL] = imx_clk_mux("cko2_sel", MXC_CCM_CCOSR, 16, 5,
+ mx53_cko2_sel, ARRAY_SIZE(mx53_cko2_sel));
+ clk[IMX5_CLK_CKO2_PODF] = imx_clk_divider("cko2_podf", "cko2_sel", MXC_CCM_CCOSR, 21, 3);
+ clk[IMX5_CLK_CKO2] = imx_clk_gate2("cko2", "cko2_podf", MXC_CCM_CCOSR, 24);
+
+ imx_check_clocks(clk, ARRAY_SIZE(clk));
+
+ clk_data.clks = clk;
+ clk_data.clk_num = ARRAY_SIZE(clk);
+ of_clk_add_provider(np, of_clk_src_onecell_get, &clk_data);
+
+ /* Set SDHC parents to be PLL2 */
+ clk_set_parent(clk[IMX5_CLK_ESDHC_A_SEL], clk[IMX5_CLK_PLL2_SW]);
+ clk_set_parent(clk[IMX5_CLK_ESDHC_B_SEL], clk[IMX5_CLK_PLL2_SW]);
+
+ /* set SDHC root clock to 200MHZ*/
+ clk_set_rate(clk[IMX5_CLK_ESDHC_A_PODF], 200000000);
+ clk_set_rate(clk[IMX5_CLK_ESDHC_B_PODF], 200000000);
+
+ clk_prepare_enable(clk[IMX5_CLK_IIM_GATE]);
+ imx_print_silicon_rev("i.MX50", IMX_CHIP_REVISION_1_1);
+ clk_disable_unprepare(clk[IMX5_CLK_IIM_GATE]);
+
+ r = clk_round_rate(clk[IMX5_CLK_USBOH3_PER_GATE], 54000000);
+ clk_set_rate(clk[IMX5_CLK_USBOH3_PER_GATE], r);
+
+ imx_register_uart_clocks(uart_clks_mx50_mx53);
+}
+CLK_OF_DECLARE(imx50_ccm, "fsl,imx50-ccm", mx50_clocks_init);
+
+static void __init mx51_clocks_init(struct device_node *np)
+{
+ void __iomem *ccm_base;
+ void __iomem *pll_base;
+ u32 val;
+
+ pll_base = ioremap(MX51_DPLL1_BASE, SZ_16K);
+ WARN_ON(!pll_base);
+ clk[IMX5_CLK_PLL1_SW] = imx_clk_pllv2("pll1_sw", "osc", pll_base);
+
+ pll_base = ioremap(MX51_DPLL2_BASE, SZ_16K);
+ WARN_ON(!pll_base);
+ clk[IMX5_CLK_PLL2_SW] = imx_clk_pllv2("pll2_sw", "osc", pll_base);
+
+ pll_base = ioremap(MX51_DPLL3_BASE, SZ_16K);
+ WARN_ON(!pll_base);
+ clk[IMX5_CLK_PLL3_SW] = imx_clk_pllv2("pll3_sw", "osc", pll_base);
+
+ ccm_base = of_iomap(np, 0);
+ WARN_ON(!ccm_base);
+
+ mx5_clocks_common_init(ccm_base);
+
+ clk[IMX5_CLK_PERIPH_APM] = imx_clk_mux("periph_apm", MXC_CCM_CBCMR, 12, 2,
+ periph_apm_sel, ARRAY_SIZE(periph_apm_sel));
+ clk[IMX5_CLK_MAIN_BUS] = imx_clk_mux("main_bus", MXC_CCM_CBCDR, 25, 1,
+ main_bus_sel, ARRAY_SIZE(main_bus_sel));
+ clk[IMX5_CLK_LP_APM] = imx_clk_mux("lp_apm", MXC_CCM_CCSR, 9, 1,
+ lp_apm_sel, ARRAY_SIZE(lp_apm_sel));
+ clk[IMX5_CLK_IPU_DI0_SEL] = imx_clk_mux_flags("ipu_di0_sel", MXC_CCM_CSCMR2, 26, 3,
+ mx51_ipu_di0_sel, ARRAY_SIZE(mx51_ipu_di0_sel), CLK_SET_RATE_PARENT);
+ clk[IMX5_CLK_IPU_DI1_SEL] = imx_clk_mux_flags("ipu_di1_sel", MXC_CCM_CSCMR2, 29, 3,
+ mx51_ipu_di1_sel, ARRAY_SIZE(mx51_ipu_di1_sel), CLK_SET_RATE_PARENT);
+ clk[IMX5_CLK_TVE_EXT_SEL] = imx_clk_mux_flags("tve_ext_sel", MXC_CCM_CSCMR1, 6, 1,
+ mx51_tve_ext_sel, ARRAY_SIZE(mx51_tve_ext_sel), CLK_SET_RATE_PARENT);
+ clk[IMX5_CLK_TVE_SEL] = imx_clk_mux("tve_sel", MXC_CCM_CSCMR1, 7, 1,
+ mx51_tve_sel, ARRAY_SIZE(mx51_tve_sel));
+ clk[IMX5_CLK_TVE_GATE] = imx_clk_gate2("tve_gate", "tve_sel", MXC_CCM_CCGR2, 30);
+ clk[IMX5_CLK_TVE_PRED] = imx_clk_divider("tve_pred", "pll3_sw", MXC_CCM_CDCDR, 28, 3);
+ clk[IMX5_CLK_ESDHC_A_SEL] = imx_clk_mux("esdhc_a_sel", MXC_CCM_CSCMR1, 20, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_ESDHC_B_SEL] = imx_clk_mux("esdhc_b_sel", MXC_CCM_CSCMR1, 16, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_ESDHC_C_SEL] = imx_clk_mux("esdhc_c_sel", MXC_CCM_CSCMR1, 19, 1, esdhc_c_sel, ARRAY_SIZE(esdhc_c_sel));
+ clk[IMX5_CLK_ESDHC_D_SEL] = imx_clk_mux("esdhc_d_sel", MXC_CCM_CSCMR1, 18, 1, esdhc_d_sel, ARRAY_SIZE(esdhc_d_sel));
+ clk[IMX5_CLK_ESDHC1_PER_GATE] = imx_clk_gate2("esdhc1_per_gate", "esdhc_a_podf", MXC_CCM_CCGR3, 2);
+ clk[IMX5_CLK_ESDHC2_PER_GATE] = imx_clk_gate2("esdhc2_per_gate", "esdhc_b_podf", MXC_CCM_CCGR3, 6);
+ clk[IMX5_CLK_ESDHC3_PER_GATE] = imx_clk_gate2("esdhc3_per_gate", "esdhc_c_sel", MXC_CCM_CCGR3, 10);
+ clk[IMX5_CLK_ESDHC4_PER_GATE] = imx_clk_gate2("esdhc4_per_gate", "esdhc_d_sel", MXC_CCM_CCGR3, 14);
+ clk[IMX5_CLK_USB_PHY_GATE] = imx_clk_gate2("usb_phy_gate", "usb_phy_sel", MXC_CCM_CCGR2, 0);
+ clk[IMX5_CLK_HSI2C_GATE] = imx_clk_gate2("hsi2c_gate", "ipg", MXC_CCM_CCGR1, 22);
+ clk[IMX5_CLK_SCC2_IPG_GATE] = imx_clk_gate2("scc2_gate", "ipg", MXC_CCM_CCGR1, 30);
+ clk[IMX5_CLK_MIPI_HSC1_GATE] = imx_clk_gate2_flags("mipi_hsc1_gate", "ipg", MXC_CCM_CCGR4, 6, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_MIPI_HSC2_GATE] = imx_clk_gate2_flags("mipi_hsc2_gate", "ipg", MXC_CCM_CCGR4, 8, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_MIPI_ESC_GATE] = imx_clk_gate2_flags("mipi_esc_gate", "ipg", MXC_CCM_CCGR4, 10, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_MIPI_HSP_GATE] = imx_clk_gate2_flags("mipi_hsp_gate", "ipg", MXC_CCM_CCGR4, 12, CLK_IS_CRITICAL);
+ clk[IMX5_CLK_SPDIF_XTAL_SEL] = imx_clk_mux("spdif_xtal_sel", MXC_CCM_CSCMR1, 2, 2,
+ mx51_spdif_xtal_sel, ARRAY_SIZE(mx51_spdif_xtal_sel));
+ clk[IMX5_CLK_SPDIF1_SEL] = imx_clk_mux("spdif1_sel", MXC_CCM_CSCMR2, 2, 2,
+ spdif_sel, ARRAY_SIZE(spdif_sel));
+ clk[IMX5_CLK_SPDIF1_PRED] = imx_clk_divider("spdif1_pred", "spdif1_sel", MXC_CCM_CDCDR, 16, 3);
+ clk[IMX5_CLK_SPDIF1_PODF] = imx_clk_divider("spdif1_podf", "spdif1_pred", MXC_CCM_CDCDR, 9, 6);
+ clk[IMX5_CLK_SPDIF1_COM_SEL] = imx_clk_mux("spdif1_com_sel", MXC_CCM_CSCMR2, 5, 1,
+ mx51_spdif1_com_sel, ARRAY_SIZE(mx51_spdif1_com_sel));
+ clk[IMX5_CLK_SPDIF1_GATE] = imx_clk_gate2("spdif1_gate", "spdif1_com_sel", MXC_CCM_CCGR5, 28);
+
+ imx_check_clocks(clk, ARRAY_SIZE(clk));
+
+ clk_data.clks = clk;
+ clk_data.clk_num = ARRAY_SIZE(clk);
+ of_clk_add_provider(np, of_clk_src_onecell_get, &clk_data);
+
+ /* set the usboh3 parent to pll2_sw */
+ clk_set_parent(clk[IMX5_CLK_USBOH3_SEL], clk[IMX5_CLK_PLL2_SW]);
+
+ /* Set SDHC parents to be PLL2 */
+ clk_set_parent(clk[IMX5_CLK_ESDHC_A_SEL], clk[IMX5_CLK_PLL2_SW]);
+ clk_set_parent(clk[IMX5_CLK_ESDHC_B_SEL], clk[IMX5_CLK_PLL2_SW]);
+
+ /* set SDHC root clock to 166.25MHZ*/
+ clk_set_rate(clk[IMX5_CLK_ESDHC_A_PODF], 166250000);
+ clk_set_rate(clk[IMX5_CLK_ESDHC_B_PODF], 166250000);
+
+ clk_prepare_enable(clk[IMX5_CLK_IIM_GATE]);
+ imx_print_silicon_rev("i.MX51", mx51_revision());
+ clk_disable_unprepare(clk[IMX5_CLK_IIM_GATE]);
+
+ /*
+ * Reference Manual says: Functionality of CCDR[18] and CLPCR[23] is no
+ * longer supported. Set to one for better power saving.
+ *
+ * The effect of not setting these bits is that MIPI clocks can't be
+ * enabled without the IPU clock being enabled aswell.
+ */
+ val = readl(MXC_CCM_CCDR);
+ val |= 1 << 18;
+ writel(val, MXC_CCM_CCDR);
+
+ val = readl(MXC_CCM_CLPCR);
+ val |= 1 << 23;
+ writel(val, MXC_CCM_CLPCR);
+
+ imx_register_uart_clocks(uart_clks_mx51);
+}
+CLK_OF_DECLARE(imx51_ccm, "fsl,imx51-ccm", mx51_clocks_init);
+
+static void __init mx53_clocks_init(struct device_node *np)
+{
+ void __iomem *ccm_base;
+ void __iomem *pll_base;
+ unsigned long r;
+
+ pll_base = ioremap(MX53_DPLL1_BASE, SZ_16K);
+ WARN_ON(!pll_base);
+ clk[IMX5_CLK_PLL1_SW] = imx_clk_pllv2("pll1_sw", "osc", pll_base);
+
+ pll_base = ioremap(MX53_DPLL2_BASE, SZ_16K);
+ WARN_ON(!pll_base);
+ clk[IMX5_CLK_PLL2_SW] = imx_clk_pllv2("pll2_sw", "osc", pll_base);
+
+ pll_base = ioremap(MX53_DPLL3_BASE, SZ_16K);
+ WARN_ON(!pll_base);
+ clk[IMX5_CLK_PLL3_SW] = imx_clk_pllv2("pll3_sw", "osc", pll_base);
+
+ pll_base = ioremap(MX53_DPLL4_BASE, SZ_16K);
+ WARN_ON(!pll_base);
+ clk[IMX5_CLK_PLL4_SW] = imx_clk_pllv2("pll4_sw", "osc", pll_base);
+
+ ccm_base = of_iomap(np, 0);
+ WARN_ON(!ccm_base);
+
+ mx5_clocks_common_init(ccm_base);
+
+ clk[IMX5_CLK_PERIPH_APM] = imx_clk_mux("periph_apm", MXC_CCM_CBCMR, 12, 2,
+ periph_apm_sel, ARRAY_SIZE(periph_apm_sel));
+ clk[IMX5_CLK_MAIN_BUS] = imx_clk_mux("main_bus", MXC_CCM_CBCDR, 25, 1,
+ main_bus_sel, ARRAY_SIZE(main_bus_sel));
+ clk[IMX5_CLK_LP_APM] = imx_clk_mux("lp_apm", MXC_CCM_CCSR, 10, 1,
+ lp_apm_sel, ARRAY_SIZE(lp_apm_sel));
+ clk[IMX5_CLK_LDB_DI1_DIV_3_5] = imx_clk_fixed_factor("ldb_di1_div_3_5", "ldb_di1_sel", 2, 7);
+ clk[IMX5_CLK_LDB_DI1_DIV] = imx_clk_divider_flags("ldb_di1_div", "ldb_di1_div_3_5", MXC_CCM_CSCMR2, 11, 1, 0);
+ clk[IMX5_CLK_LDB_DI1_SEL] = imx_clk_mux_flags("ldb_di1_sel", MXC_CCM_CSCMR2, 9, 1,
+ mx53_ldb_di1_sel, ARRAY_SIZE(mx53_ldb_di1_sel), CLK_SET_RATE_PARENT);
+ clk[IMX5_CLK_DI_PLL4_PODF] = imx_clk_divider("di_pll4_podf", "pll4_sw", MXC_CCM_CDCDR, 16, 3);
+ clk[IMX5_CLK_LDB_DI0_DIV_3_5] = imx_clk_fixed_factor("ldb_di0_div_3_5", "ldb_di0_sel", 2, 7);
+ clk[IMX5_CLK_LDB_DI0_DIV] = imx_clk_divider_flags("ldb_di0_div", "ldb_di0_div_3_5", MXC_CCM_CSCMR2, 10, 1, 0);
+ clk[IMX5_CLK_LDB_DI0_SEL] = imx_clk_mux_flags("ldb_di0_sel", MXC_CCM_CSCMR2, 8, 1,
+ mx53_ldb_di0_sel, ARRAY_SIZE(mx53_ldb_di0_sel), CLK_SET_RATE_PARENT);
+ clk[IMX5_CLK_LDB_DI0_GATE] = imx_clk_gate2("ldb_di0_gate", "ldb_di0_div", MXC_CCM_CCGR6, 28);
+ clk[IMX5_CLK_LDB_DI1_GATE] = imx_clk_gate2("ldb_di1_gate", "ldb_di1_div", MXC_CCM_CCGR6, 30);
+ clk[IMX5_CLK_IPU_DI0_SEL] = imx_clk_mux_flags("ipu_di0_sel", MXC_CCM_CSCMR2, 26, 3,
+ mx53_ipu_di0_sel, ARRAY_SIZE(mx53_ipu_di0_sel), CLK_SET_RATE_PARENT);
+ clk[IMX5_CLK_IPU_DI1_SEL] = imx_clk_mux_flags("ipu_di1_sel", MXC_CCM_CSCMR2, 29, 3,
+ mx53_ipu_di1_sel, ARRAY_SIZE(mx53_ipu_di1_sel), CLK_SET_RATE_PARENT);
+ clk[IMX5_CLK_TVE_EXT_SEL] = imx_clk_mux_flags("tve_ext_sel", MXC_CCM_CSCMR1, 6, 1,
+ mx53_tve_ext_sel, ARRAY_SIZE(mx53_tve_ext_sel), CLK_SET_RATE_PARENT);
+ clk[IMX5_CLK_TVE_GATE] = imx_clk_gate2("tve_gate", "tve_pred", MXC_CCM_CCGR2, 30);
+ clk[IMX5_CLK_TVE_PRED] = imx_clk_divider("tve_pred", "tve_ext_sel", MXC_CCM_CDCDR, 28, 3);
+ clk[IMX5_CLK_ESDHC_A_SEL] = imx_clk_mux("esdhc_a_sel", MXC_CCM_CSCMR1, 20, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_ESDHC_B_SEL] = imx_clk_mux("esdhc_b_sel", MXC_CCM_CSCMR1, 16, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_ESDHC_C_SEL] = imx_clk_mux("esdhc_c_sel", MXC_CCM_CSCMR1, 19, 1, esdhc_c_sel, ARRAY_SIZE(esdhc_c_sel));
+ clk[IMX5_CLK_ESDHC_D_SEL] = imx_clk_mux("esdhc_d_sel", MXC_CCM_CSCMR1, 18, 1, esdhc_d_sel, ARRAY_SIZE(esdhc_d_sel));
+ clk[IMX5_CLK_ESDHC1_PER_GATE] = imx_clk_gate2("esdhc1_per_gate", "esdhc_a_podf", MXC_CCM_CCGR3, 2);
+ clk[IMX5_CLK_ESDHC2_PER_GATE] = imx_clk_gate2("esdhc2_per_gate", "esdhc_c_sel", MXC_CCM_CCGR3, 6);
+ clk[IMX5_CLK_ESDHC3_PER_GATE] = imx_clk_gate2("esdhc3_per_gate", "esdhc_b_podf", MXC_CCM_CCGR3, 10);
+ clk[IMX5_CLK_ESDHC4_PER_GATE] = imx_clk_gate2("esdhc4_per_gate", "esdhc_d_sel", MXC_CCM_CCGR3, 14);
+ clk[IMX5_CLK_USB_PHY1_GATE] = imx_clk_gate2("usb_phy1_gate", "usb_phy_sel", MXC_CCM_CCGR4, 10);
+ clk[IMX5_CLK_USB_PHY2_GATE] = imx_clk_gate2("usb_phy2_gate", "usb_phy_sel", MXC_CCM_CCGR4, 12);
+ clk[IMX5_CLK_CAN_SEL] = imx_clk_mux("can_sel", MXC_CCM_CSCMR2, 6, 2,
+ mx53_can_sel, ARRAY_SIZE(mx53_can_sel));
+ clk[IMX5_CLK_CAN1_SERIAL_GATE] = imx_clk_gate2("can1_serial_gate", "can_sel", MXC_CCM_CCGR6, 22);
+ clk[IMX5_CLK_CAN1_IPG_GATE] = imx_clk_gate2("can1_ipg_gate", "ipg", MXC_CCM_CCGR6, 20);
+ clk[IMX5_CLK_OCRAM] = imx_clk_gate2("ocram", "ahb", MXC_CCM_CCGR6, 2);
+ clk[IMX5_CLK_CAN2_SERIAL_GATE] = imx_clk_gate2("can2_serial_gate", "can_sel", MXC_CCM_CCGR4, 8);
+ clk[IMX5_CLK_CAN2_IPG_GATE] = imx_clk_gate2("can2_ipg_gate", "ipg", MXC_CCM_CCGR4, 6);
+ clk[IMX5_CLK_I2C3_GATE] = imx_clk_gate2("i2c3_gate", "per_root", MXC_CCM_CCGR1, 22);
+ clk[IMX5_CLK_SATA_GATE] = imx_clk_gate2("sata_gate", "ipg", MXC_CCM_CCGR4, 2);
+
+ clk[IMX5_CLK_FIRI_SEL] = imx_clk_mux("firi_sel", MXC_CCM_CSCMR2, 12, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_FIRI_PRED] = imx_clk_divider("firi_pred", "firi_sel", MXC_CCM_CSCDR3, 6, 3);
+ clk[IMX5_CLK_FIRI_PODF] = imx_clk_divider("firi_podf", "firi_pred", MXC_CCM_CSCDR3, 0, 6);
+ clk[IMX5_CLK_FIRI_SERIAL_GATE] = imx_clk_gate2("firi_serial_gate", "firi_podf", MXC_CCM_CCGR1, 28);
+ clk[IMX5_CLK_FIRI_IPG_GATE] = imx_clk_gate2("firi_ipg_gate", "ipg", MXC_CCM_CCGR1, 26);
+
+ clk[IMX5_CLK_CSI0_MCLK1_SEL] = imx_clk_mux("csi0_mclk1_sel", MXC_CCM_CSCMR2, 22, 2,
+ standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
+ clk[IMX5_CLK_CSI0_MCLK1_PRED] = imx_clk_divider("csi0_mclk1_pred", "csi0_mclk1_sel", MXC_CCM_CSCDR4, 6, 3);
+ clk[IMX5_CLK_CSI0_MCLK1_PODF] = imx_clk_divider("csi0_mclk1_podf", "csi0_mclk1_pred", MXC_CCM_CSCDR4, 0, 6);
+ clk[IMX5_CLK_CSI0_MCLK1_GATE] = imx_clk_gate2("csi0_mclk1_serial_gate", "csi0_mclk1_podf", MXC_CCM_CCGR6, 4);
+
+ clk[IMX5_CLK_IEEE1588_SEL] = imx_clk_mux("ieee1588_sel", MXC_CCM_CSCMR2, 14, 2,
+ ieee1588_sels, ARRAY_SIZE(ieee1588_sels));
+ clk[IMX5_CLK_IEEE1588_PRED] = imx_clk_divider("ieee1588_pred", "ieee1588_sel", MXC_CCM_CSCDR2, 6, 3);
+ clk[IMX5_CLK_IEEE1588_PODF] = imx_clk_divider("ieee1588_podf", "ieee1588_pred", MXC_CCM_CSCDR2, 0, 6);
+ clk[IMX5_CLK_IEEE1588_GATE] = imx_clk_gate2("ieee1588_serial_gate", "ieee1588_podf", MXC_CCM_CCGR7, 6);
+ clk[IMX5_CLK_UART4_IPG_GATE] = imx_clk_gate2("uart4_ipg_gate", "ipg", MXC_CCM_CCGR7, 8);
+ clk[IMX5_CLK_UART4_PER_GATE] = imx_clk_gate2("uart4_per_gate", "uart_root", MXC_CCM_CCGR7, 10);
+ clk[IMX5_CLK_UART5_IPG_GATE] = imx_clk_gate2("uart5_ipg_gate", "ipg", MXC_CCM_CCGR7, 12);
+ clk[IMX5_CLK_UART5_PER_GATE] = imx_clk_gate2("uart5_per_gate", "uart_root", MXC_CCM_CCGR7, 14);
+
+ clk[IMX5_CLK_CKO1_SEL] = imx_clk_mux("cko1_sel", MXC_CCM_CCOSR, 0, 4,
+ mx53_cko1_sel, ARRAY_SIZE(mx53_cko1_sel));
+ clk[IMX5_CLK_CKO1_PODF] = imx_clk_divider("cko1_podf", "cko1_sel", MXC_CCM_CCOSR, 4, 3);
+ clk[IMX5_CLK_CKO1] = imx_clk_gate2("cko1", "cko1_podf", MXC_CCM_CCOSR, 7);
+
+ clk[IMX5_CLK_CKO2_SEL] = imx_clk_mux("cko2_sel", MXC_CCM_CCOSR, 16, 5,
+ mx53_cko2_sel, ARRAY_SIZE(mx53_cko2_sel));
+ clk[IMX5_CLK_CKO2_PODF] = imx_clk_divider("cko2_podf", "cko2_sel", MXC_CCM_CCOSR, 21, 3);
+ clk[IMX5_CLK_CKO2] = imx_clk_gate2("cko2", "cko2_podf", MXC_CCM_CCOSR, 24);
+ clk[IMX5_CLK_SPDIF_XTAL_SEL] = imx_clk_mux("spdif_xtal_sel", MXC_CCM_CSCMR1, 2, 2,
+ mx53_spdif_xtal_sel, ARRAY_SIZE(mx53_spdif_xtal_sel));
+ clk[IMX5_CLK_ARM] = imx_clk_cpu("arm", "cpu_podf",
+ clk[IMX5_CLK_CPU_PODF],
+ clk[IMX5_CLK_CPU_PODF_SEL],
+ clk[IMX5_CLK_PLL1_SW],
+ clk[IMX5_CLK_STEP_SEL]);
+
+ imx_check_clocks(clk, ARRAY_SIZE(clk));
+
+ clk_data.clks = clk;
+ clk_data.clk_num = ARRAY_SIZE(clk);
+ of_clk_add_provider(np, of_clk_src_onecell_get, &clk_data);
+
+ /* Set SDHC parents to be PLL2 */
+ clk_set_parent(clk[IMX5_CLK_ESDHC_A_SEL], clk[IMX5_CLK_PLL2_SW]);
+ clk_set_parent(clk[IMX5_CLK_ESDHC_B_SEL], clk[IMX5_CLK_PLL2_SW]);
+
+ /* set SDHC root clock to 200MHZ*/
+ clk_set_rate(clk[IMX5_CLK_ESDHC_A_PODF], 200000000);
+ clk_set_rate(clk[IMX5_CLK_ESDHC_B_PODF], 200000000);
+
+ /* move can bus clk to 24MHz */
+ clk_set_parent(clk[IMX5_CLK_CAN_SEL], clk[IMX5_CLK_LP_APM]);
+
+ /* make sure step clock is running from 24MHz */
+ clk_set_parent(clk[IMX5_CLK_STEP_SEL], clk[IMX5_CLK_LP_APM]);
+
+ clk_prepare_enable(clk[IMX5_CLK_IIM_GATE]);
+ imx_print_silicon_rev("i.MX53", mx53_revision());
+ clk_disable_unprepare(clk[IMX5_CLK_IIM_GATE]);
+
+ r = clk_round_rate(clk[IMX5_CLK_USBOH3_PER_GATE], 54000000);
+ clk_set_rate(clk[IMX5_CLK_USBOH3_PER_GATE], r);
+
+ imx_register_uart_clocks(uart_clks_mx50_mx53);
+}
+CLK_OF_DECLARE(imx53_ccm, "fsl,imx53-ccm", mx53_clocks_init);
+++ /dev/null
-/*
- * Copyright (C) 2011 Sascha Hauer, Pengutronix <s.hauer@pengutronix.de>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- */
-#include <linux/mm.h>
-#include <linux/delay.h>
-#include <linux/clk.h>
-#include <linux/io.h>
-#include <linux/clkdev.h>
-#include <linux/clk-provider.h>
-#include <linux/err.h>
-#include <linux/of.h>
-#include <linux/of_address.h>
-#include <linux/of_irq.h>
-#include <linux/sizes.h>
-#include <soc/imx/revision.h>
-#include <dt-bindings/clock/imx5-clock.h>
-
-#include "clk.h"
-
-#define MX51_DPLL1_BASE 0x83f80000
-#define MX51_DPLL2_BASE 0x83f84000
-#define MX51_DPLL3_BASE 0x83f88000
-
-#define MX53_DPLL1_BASE 0x63f80000
-#define MX53_DPLL2_BASE 0x63f84000
-#define MX53_DPLL3_BASE 0x63f88000
-#define MX53_DPLL4_BASE 0x63f8c000
-
-#define MXC_CCM_CCR (ccm_base + 0x00)
-#define MXC_CCM_CCDR (ccm_base + 0x04)
-#define MXC_CCM_CSR (ccm_base + 0x08)
-#define MXC_CCM_CCSR (ccm_base + 0x0c)
-#define MXC_CCM_CACRR (ccm_base + 0x10)
-#define MXC_CCM_CBCDR (ccm_base + 0x14)
-#define MXC_CCM_CBCMR (ccm_base + 0x18)
-#define MXC_CCM_CSCMR1 (ccm_base + 0x1c)
-#define MXC_CCM_CSCMR2 (ccm_base + 0x20)
-#define MXC_CCM_CSCDR1 (ccm_base + 0x24)
-#define MXC_CCM_CS1CDR (ccm_base + 0x28)
-#define MXC_CCM_CS2CDR (ccm_base + 0x2c)
-#define MXC_CCM_CDCDR (ccm_base + 0x30)
-#define MXC_CCM_CHSCDR (ccm_base + 0x34)
-#define MXC_CCM_CSCDR2 (ccm_base + 0x38)
-#define MXC_CCM_CSCDR3 (ccm_base + 0x3c)
-#define MXC_CCM_CSCDR4 (ccm_base + 0x40)
-#define MXC_CCM_CWDR (ccm_base + 0x44)
-#define MXC_CCM_CDHIPR (ccm_base + 0x48)
-#define MXC_CCM_CDCR (ccm_base + 0x4c)
-#define MXC_CCM_CTOR (ccm_base + 0x50)
-#define MXC_CCM_CLPCR (ccm_base + 0x54)
-#define MXC_CCM_CISR (ccm_base + 0x58)
-#define MXC_CCM_CIMR (ccm_base + 0x5c)
-#define MXC_CCM_CCOSR (ccm_base + 0x60)
-#define MXC_CCM_CGPR (ccm_base + 0x64)
-#define MXC_CCM_CCGR0 (ccm_base + 0x68)
-#define MXC_CCM_CCGR1 (ccm_base + 0x6c)
-#define MXC_CCM_CCGR2 (ccm_base + 0x70)
-#define MXC_CCM_CCGR3 (ccm_base + 0x74)
-#define MXC_CCM_CCGR4 (ccm_base + 0x78)
-#define MXC_CCM_CCGR5 (ccm_base + 0x7c)
-#define MXC_CCM_CCGR6 (ccm_base + 0x80)
-#define MXC_CCM_CCGR7 (ccm_base + 0x84)
-
-/* Low-power Audio Playback Mode clock */
-static const char *lp_apm_sel[] = { "osc", };
-
-/* This is used multiple times */
-static const char *standard_pll_sel[] = { "pll1_sw", "pll2_sw", "pll3_sw", "lp_apm", };
-static const char *periph_apm_sel[] = { "pll1_sw", "pll3_sw", "lp_apm", };
-static const char *main_bus_sel[] = { "pll2_sw", "periph_apm", };
-static const char *per_lp_apm_sel[] = { "main_bus", "lp_apm", };
-static const char *per_root_sel[] = { "per_podf", "ipg", };
-static const char *esdhc_c_sel[] = { "esdhc_a_podf", "esdhc_b_podf", };
-static const char *esdhc_d_sel[] = { "esdhc_a_podf", "esdhc_b_podf", };
-static const char *ssi_apm_sels[] = { "ckih1", "lp_amp", "ckih2", };
-static const char *ssi_clk_sels[] = { "pll1_sw", "pll2_sw", "pll3_sw", "ssi_apm", };
-static const char *ssi3_clk_sels[] = { "ssi1_root_gate", "ssi2_root_gate", };
-static const char *ssi_ext1_com_sels[] = { "ssi_ext1_podf", "ssi1_root_gate", };
-static const char *ssi_ext2_com_sels[] = { "ssi_ext2_podf", "ssi2_root_gate", };
-static const char *emi_slow_sel[] = { "main_bus", "ahb", };
-static const char *usb_phy_sel_str[] = { "osc", "usb_phy_podf", };
-static const char *mx51_ipu_di0_sel[] = { "di_pred", "osc", "ckih1", "tve_di", };
-static const char *mx53_ipu_di0_sel[] = { "di_pred", "osc", "ckih1", "di_pll4_podf", "dummy", "ldb_di0_gate", };
-static const char *mx53_ldb_di0_sel[] = { "pll3_sw", "pll4_sw", };
-static const char *mx51_ipu_di1_sel[] = { "di_pred", "osc", "ckih1", "tve_di", "ipp_di1", };
-static const char *mx53_ipu_di1_sel[] = { "di_pred", "osc", "ckih1", "tve_di", "ipp_di1", "ldb_di1_gate", };
-static const char *mx53_ldb_di1_sel[] = { "pll3_sw", "pll4_sw", };
-static const char *mx51_tve_ext_sel[] = { "osc", "ckih1", };
-static const char *mx53_tve_ext_sel[] = { "pll4_sw", "ckih1", };
-static const char *mx51_tve_sel[] = { "tve_pred", "tve_ext_sel", };
-static const char *ipu_sel[] = { "axi_a", "axi_b", "emi_slow_gate", "ahb", };
-static const char *gpu3d_sel[] = { "axi_a", "axi_b", "emi_slow_gate", "ahb" };
-static const char *gpu2d_sel[] = { "axi_a", "axi_b", "emi_slow_gate", "ahb" };
-static const char *vpu_sel[] = { "axi_a", "axi_b", "emi_slow_gate", "ahb", };
-static const char *mx53_can_sel[] = { "ipg", "ckih1", "ckih2", "lp_apm", };
-static const char *mx53_cko1_sel[] = {
- "cpu_podf", "pll1_sw", "pll2_sw", "pll3_sw",
- "emi_slow_podf", "pll4_sw", "nfc_podf", "dummy",
- "di_pred", "dummy", "dummy", "ahb",
- "ipg", "per_root", "ckil", "dummy",};
-static const char *mx53_cko2_sel[] = {
- "dummy"/* dptc_core */, "dummy"/* dptc_perich */,
- "dummy", "esdhc_a_podf",
- "usboh3_podf", "dummy"/* wrck_clk_root */,
- "ecspi_podf", "dummy"/* pll1_ref_clk */,
- "esdhc_b_podf", "dummy"/* ddr_clk_root */,
- "dummy"/* arm_axi_clk_root */, "dummy"/* usb_phy_out */,
- "vpu_sel", "ipu_sel",
- "osc", "ckih1",
- "dummy", "esdhc_c_sel",
- "ssi1_root_podf", "ssi2_root_podf",
- "dummy", "dummy",
- "dummy"/* lpsr_clk_root */, "dummy"/* pgc_clk_root */,
- "dummy"/* tve_out */, "usb_phy_sel",
- "tve_sel", "lp_apm",
- "uart_root", "dummy"/* spdif0_clk_root */,
- "dummy", "dummy", };
-static const char *mx51_spdif_xtal_sel[] = { "osc", "ckih", "ckih2", };
-static const char *mx53_spdif_xtal_sel[] = { "osc", "ckih", "ckih2", "pll4_sw", };
-static const char *spdif_sel[] = { "pll1_sw", "pll2_sw", "pll3_sw", "spdif_xtal_sel", };
-static const char *spdif0_com_sel[] = { "spdif0_podf", "ssi1_root_gate", };
-static const char *mx51_spdif1_com_sel[] = { "spdif1_podf", "ssi2_root_gate", };
-static const char *step_sels[] = { "lp_apm", };
-static const char *cpu_podf_sels[] = { "pll1_sw", "step_sel" };
-static const char *ieee1588_sels[] = { "pll3_sw", "pll4_sw", "dummy" /* usbphy2_clk */, "dummy" /* fec_phy_clk */ };
-
-static struct clk *clk[IMX5_CLK_END];
-static struct clk_onecell_data clk_data;
-
-static struct clk ** const uart_clks_mx51[] __initconst = {
- &clk[IMX5_CLK_UART1_IPG_GATE],
- &clk[IMX5_CLK_UART1_PER_GATE],
- &clk[IMX5_CLK_UART2_IPG_GATE],
- &clk[IMX5_CLK_UART2_PER_GATE],
- &clk[IMX5_CLK_UART3_IPG_GATE],
- &clk[IMX5_CLK_UART3_PER_GATE],
- NULL
-};
-
-static struct clk ** const uart_clks_mx50_mx53[] __initconst = {
- &clk[IMX5_CLK_UART1_IPG_GATE],
- &clk[IMX5_CLK_UART1_PER_GATE],
- &clk[IMX5_CLK_UART2_IPG_GATE],
- &clk[IMX5_CLK_UART2_PER_GATE],
- &clk[IMX5_CLK_UART3_IPG_GATE],
- &clk[IMX5_CLK_UART3_PER_GATE],
- &clk[IMX5_CLK_UART4_IPG_GATE],
- &clk[IMX5_CLK_UART4_PER_GATE],
- &clk[IMX5_CLK_UART5_IPG_GATE],
- &clk[IMX5_CLK_UART5_PER_GATE],
- NULL
-};
-
-static void __init mx5_clocks_common_init(void __iomem *ccm_base)
-{
- clk[IMX5_CLK_DUMMY] = imx_clk_fixed("dummy", 0);
- clk[IMX5_CLK_CKIL] = imx_obtain_fixed_clock("ckil", 0);
- clk[IMX5_CLK_OSC] = imx_obtain_fixed_clock("osc", 0);
- clk[IMX5_CLK_CKIH1] = imx_obtain_fixed_clock("ckih1", 0);
- clk[IMX5_CLK_CKIH2] = imx_obtain_fixed_clock("ckih2", 0);
-
- clk[IMX5_CLK_PERIPH_APM] = imx_clk_mux("periph_apm", MXC_CCM_CBCMR, 12, 2,
- periph_apm_sel, ARRAY_SIZE(periph_apm_sel));
- clk[IMX5_CLK_MAIN_BUS] = imx_clk_mux("main_bus", MXC_CCM_CBCDR, 25, 1,
- main_bus_sel, ARRAY_SIZE(main_bus_sel));
- clk[IMX5_CLK_PER_LP_APM] = imx_clk_mux("per_lp_apm", MXC_CCM_CBCMR, 1, 1,
- per_lp_apm_sel, ARRAY_SIZE(per_lp_apm_sel));
- clk[IMX5_CLK_PER_PRED1] = imx_clk_divider("per_pred1", "per_lp_apm", MXC_CCM_CBCDR, 6, 2);
- clk[IMX5_CLK_PER_PRED2] = imx_clk_divider("per_pred2", "per_pred1", MXC_CCM_CBCDR, 3, 3);
- clk[IMX5_CLK_PER_PODF] = imx_clk_divider("per_podf", "per_pred2", MXC_CCM_CBCDR, 0, 3);
- clk[IMX5_CLK_PER_ROOT] = imx_clk_mux("per_root", MXC_CCM_CBCMR, 0, 1,
- per_root_sel, ARRAY_SIZE(per_root_sel));
- clk[IMX5_CLK_AHB] = imx_clk_divider("ahb", "main_bus", MXC_CCM_CBCDR, 10, 3);
- clk[IMX5_CLK_AHB_MAX] = imx_clk_gate2_flags("ahb_max", "ahb", MXC_CCM_CCGR0, 28, CLK_IS_CRITICAL);
- clk[IMX5_CLK_AIPS_TZ1] = imx_clk_gate2_flags("aips_tz1", "ahb", MXC_CCM_CCGR0, 24, CLK_IS_CRITICAL);
- clk[IMX5_CLK_AIPS_TZ2] = imx_clk_gate2_flags("aips_tz2", "ahb", MXC_CCM_CCGR0, 26, CLK_IS_CRITICAL);
- clk[IMX5_CLK_TMAX1] = imx_clk_gate2_flags("tmax1", "ahb", MXC_CCM_CCGR1, 0, CLK_IS_CRITICAL);
- clk[IMX5_CLK_TMAX2] = imx_clk_gate2_flags("tmax2", "ahb", MXC_CCM_CCGR1, 2, CLK_IS_CRITICAL);
- clk[IMX5_CLK_TMAX3] = imx_clk_gate2_flags("tmax3", "ahb", MXC_CCM_CCGR1, 4, CLK_IS_CRITICAL);
- clk[IMX5_CLK_SPBA] = imx_clk_gate2_flags("spba", "ipg", MXC_CCM_CCGR5, 0, CLK_IS_CRITICAL);
- clk[IMX5_CLK_IPG] = imx_clk_divider("ipg", "ahb", MXC_CCM_CBCDR, 8, 2);
- clk[IMX5_CLK_AXI_A] = imx_clk_divider("axi_a", "main_bus", MXC_CCM_CBCDR, 16, 3);
- clk[IMX5_CLK_AXI_B] = imx_clk_divider("axi_b", "main_bus", MXC_CCM_CBCDR, 19, 3);
- clk[IMX5_CLK_UART_SEL] = imx_clk_mux("uart_sel", MXC_CCM_CSCMR1, 24, 2,
- standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
- clk[IMX5_CLK_UART_PRED] = imx_clk_divider("uart_pred", "uart_sel", MXC_CCM_CSCDR1, 3, 3);
- clk[IMX5_CLK_UART_ROOT] = imx_clk_divider("uart_root", "uart_pred", MXC_CCM_CSCDR1, 0, 3);
-
- clk[IMX5_CLK_ESDHC_A_SEL] = imx_clk_mux("esdhc_a_sel", MXC_CCM_CSCMR1, 20, 2,
- standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
- clk[IMX5_CLK_ESDHC_B_SEL] = imx_clk_mux("esdhc_b_sel", MXC_CCM_CSCMR1, 16, 2,
- standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
- clk[IMX5_CLK_ESDHC_A_PRED] = imx_clk_divider("esdhc_a_pred", "esdhc_a_sel", MXC_CCM_CSCDR1, 16, 3);
- clk[IMX5_CLK_ESDHC_A_PODF] = imx_clk_divider("esdhc_a_podf", "esdhc_a_pred", MXC_CCM_CSCDR1, 11, 3);
- clk[IMX5_CLK_ESDHC_B_PRED] = imx_clk_divider("esdhc_b_pred", "esdhc_b_sel", MXC_CCM_CSCDR1, 22, 3);
- clk[IMX5_CLK_ESDHC_B_PODF] = imx_clk_divider("esdhc_b_podf", "esdhc_b_pred", MXC_CCM_CSCDR1, 19, 3);
- clk[IMX5_CLK_ESDHC_C_SEL] = imx_clk_mux("esdhc_c_sel", MXC_CCM_CSCMR1, 19, 1, esdhc_c_sel, ARRAY_SIZE(esdhc_c_sel));
- clk[IMX5_CLK_ESDHC_D_SEL] = imx_clk_mux("esdhc_d_sel", MXC_CCM_CSCMR1, 18, 1, esdhc_d_sel, ARRAY_SIZE(esdhc_d_sel));
-
- clk[IMX5_CLK_EMI_SEL] = imx_clk_mux("emi_sel", MXC_CCM_CBCDR, 26, 1,
- emi_slow_sel, ARRAY_SIZE(emi_slow_sel));
- clk[IMX5_CLK_EMI_SLOW_PODF] = imx_clk_divider("emi_slow_podf", "emi_sel", MXC_CCM_CBCDR, 22, 3);
- clk[IMX5_CLK_NFC_PODF] = imx_clk_divider("nfc_podf", "emi_slow_podf", MXC_CCM_CBCDR, 13, 3);
- clk[IMX5_CLK_ECSPI_SEL] = imx_clk_mux("ecspi_sel", MXC_CCM_CSCMR1, 4, 2,
- standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
- clk[IMX5_CLK_ECSPI_PRED] = imx_clk_divider("ecspi_pred", "ecspi_sel", MXC_CCM_CSCDR2, 25, 3);
- clk[IMX5_CLK_ECSPI_PODF] = imx_clk_divider("ecspi_podf", "ecspi_pred", MXC_CCM_CSCDR2, 19, 6);
- clk[IMX5_CLK_USBOH3_SEL] = imx_clk_mux("usboh3_sel", MXC_CCM_CSCMR1, 22, 2,
- standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
- clk[IMX5_CLK_USBOH3_PRED] = imx_clk_divider("usboh3_pred", "usboh3_sel", MXC_CCM_CSCDR1, 8, 3);
- clk[IMX5_CLK_USBOH3_PODF] = imx_clk_divider("usboh3_podf", "usboh3_pred", MXC_CCM_CSCDR1, 6, 2);
- clk[IMX5_CLK_USB_PHY_PRED] = imx_clk_divider("usb_phy_pred", "pll3_sw", MXC_CCM_CDCDR, 3, 3);
- clk[IMX5_CLK_USB_PHY_PODF] = imx_clk_divider("usb_phy_podf", "usb_phy_pred", MXC_CCM_CDCDR, 0, 3);
- clk[IMX5_CLK_USB_PHY_SEL] = imx_clk_mux("usb_phy_sel", MXC_CCM_CSCMR1, 26, 1,
- usb_phy_sel_str, ARRAY_SIZE(usb_phy_sel_str));
- clk[IMX5_CLK_STEP_SEL] = imx_clk_mux("step_sel", MXC_CCM_CCSR, 7, 2, step_sels, ARRAY_SIZE(step_sels));
- clk[IMX5_CLK_CPU_PODF_SEL] = imx_clk_mux("cpu_podf_sel", MXC_CCM_CCSR, 2, 1, cpu_podf_sels, ARRAY_SIZE(cpu_podf_sels));
- clk[IMX5_CLK_CPU_PODF] = imx_clk_divider("cpu_podf", "cpu_podf_sel", MXC_CCM_CACRR, 0, 3);
- clk[IMX5_CLK_DI_PRED] = imx_clk_divider("di_pred", "pll3_sw", MXC_CCM_CDCDR, 6, 3);
- clk[IMX5_CLK_IIM_GATE] = imx_clk_gate2("iim_gate", "ipg", MXC_CCM_CCGR0, 30);
- clk[IMX5_CLK_UART1_IPG_GATE] = imx_clk_gate2("uart1_ipg_gate", "ipg", MXC_CCM_CCGR1, 6);
- clk[IMX5_CLK_UART1_PER_GATE] = imx_clk_gate2("uart1_per_gate", "uart_root", MXC_CCM_CCGR1, 8);
- clk[IMX5_CLK_UART2_IPG_GATE] = imx_clk_gate2("uart2_ipg_gate", "ipg", MXC_CCM_CCGR1, 10);
- clk[IMX5_CLK_UART2_PER_GATE] = imx_clk_gate2("uart2_per_gate", "uart_root", MXC_CCM_CCGR1, 12);
- clk[IMX5_CLK_UART3_IPG_GATE] = imx_clk_gate2("uart3_ipg_gate", "ipg", MXC_CCM_CCGR1, 14);
- clk[IMX5_CLK_UART3_PER_GATE] = imx_clk_gate2("uart3_per_gate", "uart_root", MXC_CCM_CCGR1, 16);
- clk[IMX5_CLK_I2C1_GATE] = imx_clk_gate2("i2c1_gate", "per_root", MXC_CCM_CCGR1, 18);
- clk[IMX5_CLK_I2C2_GATE] = imx_clk_gate2("i2c2_gate", "per_root", MXC_CCM_CCGR1, 20);
- clk[IMX5_CLK_PWM1_IPG_GATE] = imx_clk_gate2("pwm1_ipg_gate", "ipg", MXC_CCM_CCGR2, 10);
- clk[IMX5_CLK_PWM1_HF_GATE] = imx_clk_gate2("pwm1_hf_gate", "per_root", MXC_CCM_CCGR2, 12);
- clk[IMX5_CLK_PWM2_IPG_GATE] = imx_clk_gate2("pwm2_ipg_gate", "ipg", MXC_CCM_CCGR2, 14);
- clk[IMX5_CLK_PWM2_HF_GATE] = imx_clk_gate2("pwm2_hf_gate", "per_root", MXC_CCM_CCGR2, 16);
- clk[IMX5_CLK_GPT_IPG_GATE] = imx_clk_gate2("gpt_ipg_gate", "ipg", MXC_CCM_CCGR2, 18);
- clk[IMX5_CLK_GPT_HF_GATE] = imx_clk_gate2("gpt_hf_gate", "per_root", MXC_CCM_CCGR2, 20);
- clk[IMX5_CLK_FEC_GATE] = imx_clk_gate2("fec_gate", "ipg", MXC_CCM_CCGR2, 24);
- clk[IMX5_CLK_USBOH3_GATE] = imx_clk_gate2("usboh3_gate", "ipg", MXC_CCM_CCGR2, 26);
- clk[IMX5_CLK_USBOH3_PER_GATE] = imx_clk_gate2("usboh3_per_gate", "usboh3_podf", MXC_CCM_CCGR2, 28);
- clk[IMX5_CLK_ESDHC1_IPG_GATE] = imx_clk_gate2("esdhc1_ipg_gate", "ipg", MXC_CCM_CCGR3, 0);
- clk[IMX5_CLK_ESDHC2_IPG_GATE] = imx_clk_gate2("esdhc2_ipg_gate", "ipg", MXC_CCM_CCGR3, 4);
- clk[IMX5_CLK_ESDHC3_IPG_GATE] = imx_clk_gate2("esdhc3_ipg_gate", "ipg", MXC_CCM_CCGR3, 8);
- clk[IMX5_CLK_ESDHC4_IPG_GATE] = imx_clk_gate2("esdhc4_ipg_gate", "ipg", MXC_CCM_CCGR3, 12);
- clk[IMX5_CLK_SSI1_IPG_GATE] = imx_clk_gate2("ssi1_ipg_gate", "ipg", MXC_CCM_CCGR3, 16);
- clk[IMX5_CLK_SSI2_IPG_GATE] = imx_clk_gate2("ssi2_ipg_gate", "ipg", MXC_CCM_CCGR3, 20);
- clk[IMX5_CLK_SSI3_IPG_GATE] = imx_clk_gate2("ssi3_ipg_gate", "ipg", MXC_CCM_CCGR3, 24);
- clk[IMX5_CLK_ECSPI1_IPG_GATE] = imx_clk_gate2("ecspi1_ipg_gate", "ipg", MXC_CCM_CCGR4, 18);
- clk[IMX5_CLK_ECSPI1_PER_GATE] = imx_clk_gate2("ecspi1_per_gate", "ecspi_podf", MXC_CCM_CCGR4, 20);
- clk[IMX5_CLK_ECSPI2_IPG_GATE] = imx_clk_gate2("ecspi2_ipg_gate", "ipg", MXC_CCM_CCGR4, 22);
- clk[IMX5_CLK_ECSPI2_PER_GATE] = imx_clk_gate2("ecspi2_per_gate", "ecspi_podf", MXC_CCM_CCGR4, 24);
- clk[IMX5_CLK_CSPI_IPG_GATE] = imx_clk_gate2("cspi_ipg_gate", "ipg", MXC_CCM_CCGR4, 26);
- clk[IMX5_CLK_SDMA_GATE] = imx_clk_gate2("sdma_gate", "ipg", MXC_CCM_CCGR4, 30);
- clk[IMX5_CLK_EMI_FAST_GATE] = imx_clk_gate2_flags("emi_fast_gate", "dummy", MXC_CCM_CCGR5, 14, CLK_IS_CRITICAL);
- clk[IMX5_CLK_EMI_SLOW_GATE] = imx_clk_gate2_flags("emi_slow_gate", "emi_slow_podf", MXC_CCM_CCGR5, 16, CLK_IS_CRITICAL);
- clk[IMX5_CLK_IPU_SEL] = imx_clk_mux("ipu_sel", MXC_CCM_CBCMR, 6, 2, ipu_sel, ARRAY_SIZE(ipu_sel));
- clk[IMX5_CLK_IPU_GATE] = imx_clk_gate2("ipu_gate", "ipu_sel", MXC_CCM_CCGR5, 10);
- clk[IMX5_CLK_NFC_GATE] = imx_clk_gate2("nfc_gate", "nfc_podf", MXC_CCM_CCGR5, 20);
- clk[IMX5_CLK_IPU_DI0_GATE] = imx_clk_gate2("ipu_di0_gate", "ipu_di0_sel", MXC_CCM_CCGR6, 10);
- clk[IMX5_CLK_IPU_DI1_GATE] = imx_clk_gate2("ipu_di1_gate", "ipu_di1_sel", MXC_CCM_CCGR6, 12);
- clk[IMX5_CLK_GPU3D_SEL] = imx_clk_mux("gpu3d_sel", MXC_CCM_CBCMR, 4, 2, gpu3d_sel, ARRAY_SIZE(gpu3d_sel));
- clk[IMX5_CLK_GPU2D_SEL] = imx_clk_mux("gpu2d_sel", MXC_CCM_CBCMR, 16, 2, gpu2d_sel, ARRAY_SIZE(gpu2d_sel));
- clk[IMX5_CLK_GPU3D_GATE] = imx_clk_gate2("gpu3d_gate", "gpu3d_sel", MXC_CCM_CCGR5, 2);
- clk[IMX5_CLK_GARB_GATE] = imx_clk_gate2("garb_gate", "axi_a", MXC_CCM_CCGR5, 4);
- clk[IMX5_CLK_GPU2D_GATE] = imx_clk_gate2("gpu2d_gate", "gpu2d_sel", MXC_CCM_CCGR6, 14);
- clk[IMX5_CLK_VPU_SEL] = imx_clk_mux("vpu_sel", MXC_CCM_CBCMR, 14, 2, vpu_sel, ARRAY_SIZE(vpu_sel));
- clk[IMX5_CLK_VPU_GATE] = imx_clk_gate2("vpu_gate", "vpu_sel", MXC_CCM_CCGR5, 6);
- clk[IMX5_CLK_VPU_REFERENCE_GATE] = imx_clk_gate2("vpu_reference_gate", "osc", MXC_CCM_CCGR5, 8);
- clk[IMX5_CLK_GPC_DVFS] = imx_clk_gate2_flags("gpc_dvfs", "dummy", MXC_CCM_CCGR5, 24, CLK_IS_CRITICAL);
-
- clk[IMX5_CLK_SSI_APM] = imx_clk_mux("ssi_apm", MXC_CCM_CSCMR1, 8, 2, ssi_apm_sels, ARRAY_SIZE(ssi_apm_sels));
- clk[IMX5_CLK_SSI1_ROOT_SEL] = imx_clk_mux("ssi1_root_sel", MXC_CCM_CSCMR1, 14, 2, ssi_clk_sels, ARRAY_SIZE(ssi_clk_sels));
- clk[IMX5_CLK_SSI2_ROOT_SEL] = imx_clk_mux("ssi2_root_sel", MXC_CCM_CSCMR1, 12, 2, ssi_clk_sels, ARRAY_SIZE(ssi_clk_sels));
- clk[IMX5_CLK_SSI3_ROOT_SEL] = imx_clk_mux("ssi3_root_sel", MXC_CCM_CSCMR1, 11, 1, ssi3_clk_sels, ARRAY_SIZE(ssi3_clk_sels));
- clk[IMX5_CLK_SSI_EXT1_SEL] = imx_clk_mux("ssi_ext1_sel", MXC_CCM_CSCMR1, 28, 2, ssi_clk_sels, ARRAY_SIZE(ssi_clk_sels));
- clk[IMX5_CLK_SSI_EXT2_SEL] = imx_clk_mux("ssi_ext2_sel", MXC_CCM_CSCMR1, 30, 2, ssi_clk_sels, ARRAY_SIZE(ssi_clk_sels));
- clk[IMX5_CLK_SSI_EXT1_COM_SEL] = imx_clk_mux("ssi_ext1_com_sel", MXC_CCM_CSCMR1, 0, 1, ssi_ext1_com_sels, ARRAY_SIZE(ssi_ext1_com_sels));
- clk[IMX5_CLK_SSI_EXT2_COM_SEL] = imx_clk_mux("ssi_ext2_com_sel", MXC_CCM_CSCMR1, 1, 1, ssi_ext2_com_sels, ARRAY_SIZE(ssi_ext2_com_sels));
- clk[IMX5_CLK_SSI1_ROOT_PRED] = imx_clk_divider("ssi1_root_pred", "ssi1_root_sel", MXC_CCM_CS1CDR, 6, 3);
- clk[IMX5_CLK_SSI1_ROOT_PODF] = imx_clk_divider("ssi1_root_podf", "ssi1_root_pred", MXC_CCM_CS1CDR, 0, 6);
- clk[IMX5_CLK_SSI2_ROOT_PRED] = imx_clk_divider("ssi2_root_pred", "ssi2_root_sel", MXC_CCM_CS2CDR, 6, 3);
- clk[IMX5_CLK_SSI2_ROOT_PODF] = imx_clk_divider("ssi2_root_podf", "ssi2_root_pred", MXC_CCM_CS2CDR, 0, 6);
- clk[IMX5_CLK_SSI_EXT1_PRED] = imx_clk_divider("ssi_ext1_pred", "ssi_ext1_sel", MXC_CCM_CS1CDR, 22, 3);
- clk[IMX5_CLK_SSI_EXT1_PODF] = imx_clk_divider("ssi_ext1_podf", "ssi_ext1_pred", MXC_CCM_CS1CDR, 16, 6);
- clk[IMX5_CLK_SSI_EXT2_PRED] = imx_clk_divider("ssi_ext2_pred", "ssi_ext2_sel", MXC_CCM_CS2CDR, 22, 3);
- clk[IMX5_CLK_SSI_EXT2_PODF] = imx_clk_divider("ssi_ext2_podf", "ssi_ext2_pred", MXC_CCM_CS2CDR, 16, 6);
- clk[IMX5_CLK_SSI1_ROOT_GATE] = imx_clk_gate2("ssi1_root_gate", "ssi1_root_podf", MXC_CCM_CCGR3, 18);
- clk[IMX5_CLK_SSI2_ROOT_GATE] = imx_clk_gate2("ssi2_root_gate", "ssi2_root_podf", MXC_CCM_CCGR3, 22);
- clk[IMX5_CLK_SSI3_ROOT_GATE] = imx_clk_gate2("ssi3_root_gate", "ssi3_root_sel", MXC_CCM_CCGR3, 26);
- clk[IMX5_CLK_SSI_EXT1_GATE] = imx_clk_gate2("ssi_ext1_gate", "ssi_ext1_com_sel", MXC_CCM_CCGR3, 28);
- clk[IMX5_CLK_SSI_EXT2_GATE] = imx_clk_gate2("ssi_ext2_gate", "ssi_ext2_com_sel", MXC_CCM_CCGR3, 30);
- clk[IMX5_CLK_EPIT1_IPG_GATE] = imx_clk_gate2("epit1_ipg_gate", "ipg", MXC_CCM_CCGR2, 2);
- clk[IMX5_CLK_EPIT1_HF_GATE] = imx_clk_gate2("epit1_hf_gate", "per_root", MXC_CCM_CCGR2, 4);
- clk[IMX5_CLK_EPIT2_IPG_GATE] = imx_clk_gate2("epit2_ipg_gate", "ipg", MXC_CCM_CCGR2, 6);
- clk[IMX5_CLK_EPIT2_HF_GATE] = imx_clk_gate2("epit2_hf_gate", "per_root", MXC_CCM_CCGR2, 8);
- clk[IMX5_CLK_OWIRE_GATE] = imx_clk_gate2("owire_gate", "per_root", MXC_CCM_CCGR2, 22);
- clk[IMX5_CLK_SRTC_GATE] = imx_clk_gate2("srtc_gate", "per_root", MXC_CCM_CCGR4, 28);
- clk[IMX5_CLK_PATA_GATE] = imx_clk_gate2("pata_gate", "ipg", MXC_CCM_CCGR4, 0);
- clk[IMX5_CLK_SPDIF0_SEL] = imx_clk_mux("spdif0_sel", MXC_CCM_CSCMR2, 0, 2, spdif_sel, ARRAY_SIZE(spdif_sel));
- clk[IMX5_CLK_SPDIF0_PRED] = imx_clk_divider("spdif0_pred", "spdif0_sel", MXC_CCM_CDCDR, 25, 3);
- clk[IMX5_CLK_SPDIF0_PODF] = imx_clk_divider("spdif0_podf", "spdif0_pred", MXC_CCM_CDCDR, 19, 6);
- clk[IMX5_CLK_SPDIF0_COM_SEL] = imx_clk_mux_flags("spdif0_com_sel", MXC_CCM_CSCMR2, 4, 1,
- spdif0_com_sel, ARRAY_SIZE(spdif0_com_sel), CLK_SET_RATE_PARENT);
- clk[IMX5_CLK_SPDIF0_GATE] = imx_clk_gate2("spdif0_gate", "spdif0_com_sel", MXC_CCM_CCGR5, 26);
- clk[IMX5_CLK_SPDIF_IPG_GATE] = imx_clk_gate2("spdif_ipg_gate", "ipg", MXC_CCM_CCGR5, 30);
- clk[IMX5_CLK_SAHARA_IPG_GATE] = imx_clk_gate2("sahara_ipg_gate", "ipg", MXC_CCM_CCGR4, 14);
- clk[IMX5_CLK_SATA_REF] = imx_clk_fixed_factor("sata_ref", "usb_phy1_gate", 1, 1);
-
- clk_register_clkdev(clk[IMX5_CLK_CPU_PODF], NULL, "cpu0");
- clk_register_clkdev(clk[IMX5_CLK_GPC_DVFS], "gpc_dvfs", NULL);
-
- /* Set SDHC parents to be PLL2 */
- clk_set_parent(clk[IMX5_CLK_ESDHC_A_SEL], clk[IMX5_CLK_PLL2_SW]);
- clk_set_parent(clk[IMX5_CLK_ESDHC_B_SEL], clk[IMX5_CLK_PLL2_SW]);
-
- /* move usb phy clk to 24MHz */
- clk_set_parent(clk[IMX5_CLK_USB_PHY_SEL], clk[IMX5_CLK_OSC]);
-}
-
-static void __init mx50_clocks_init(struct device_node *np)
-{
- void __iomem *ccm_base;
- void __iomem *pll_base;
- unsigned long r;
-
- pll_base = ioremap(MX53_DPLL1_BASE, SZ_16K);
- WARN_ON(!pll_base);
- clk[IMX5_CLK_PLL1_SW] = imx_clk_pllv2("pll1_sw", "osc", pll_base);
-
- pll_base = ioremap(MX53_DPLL2_BASE, SZ_16K);
- WARN_ON(!pll_base);
- clk[IMX5_CLK_PLL2_SW] = imx_clk_pllv2("pll2_sw", "osc", pll_base);
-
- pll_base = ioremap(MX53_DPLL3_BASE, SZ_16K);
- WARN_ON(!pll_base);
- clk[IMX5_CLK_PLL3_SW] = imx_clk_pllv2("pll3_sw", "osc", pll_base);
-
- ccm_base = of_iomap(np, 0);
- WARN_ON(!ccm_base);
-
- mx5_clocks_common_init(ccm_base);
-
- clk[IMX5_CLK_LP_APM] = imx_clk_mux("lp_apm", MXC_CCM_CCSR, 10, 1,
- lp_apm_sel, ARRAY_SIZE(lp_apm_sel));
- clk[IMX5_CLK_ESDHC1_PER_GATE] = imx_clk_gate2("esdhc1_per_gate", "esdhc_a_podf", MXC_CCM_CCGR3, 2);
- clk[IMX5_CLK_ESDHC2_PER_GATE] = imx_clk_gate2("esdhc2_per_gate", "esdhc_c_sel", MXC_CCM_CCGR3, 6);
- clk[IMX5_CLK_ESDHC3_PER_GATE] = imx_clk_gate2("esdhc3_per_gate", "esdhc_b_podf", MXC_CCM_CCGR3, 10);
- clk[IMX5_CLK_ESDHC4_PER_GATE] = imx_clk_gate2("esdhc4_per_gate", "esdhc_d_sel", MXC_CCM_CCGR3, 14);
- clk[IMX5_CLK_USB_PHY1_GATE] = imx_clk_gate2("usb_phy1_gate", "usb_phy_sel", MXC_CCM_CCGR4, 10);
- clk[IMX5_CLK_USB_PHY2_GATE] = imx_clk_gate2("usb_phy2_gate", "usb_phy_sel", MXC_CCM_CCGR4, 12);
- clk[IMX5_CLK_I2C3_GATE] = imx_clk_gate2("i2c3_gate", "per_root", MXC_CCM_CCGR1, 22);
- clk[IMX5_CLK_UART4_IPG_GATE] = imx_clk_gate2("uart4_ipg_gate", "ipg", MXC_CCM_CCGR7, 8);
- clk[IMX5_CLK_UART4_PER_GATE] = imx_clk_gate2("uart4_per_gate", "uart_root", MXC_CCM_CCGR7, 10);
- clk[IMX5_CLK_UART5_IPG_GATE] = imx_clk_gate2("uart5_ipg_gate", "ipg", MXC_CCM_CCGR7, 12);
- clk[IMX5_CLK_UART5_PER_GATE] = imx_clk_gate2("uart5_per_gate", "uart_root", MXC_CCM_CCGR7, 14);
-
- clk[IMX5_CLK_CKO1_SEL] = imx_clk_mux("cko1_sel", MXC_CCM_CCOSR, 0, 4,
- mx53_cko1_sel, ARRAY_SIZE(mx53_cko1_sel));
- clk[IMX5_CLK_CKO1_PODF] = imx_clk_divider("cko1_podf", "cko1_sel", MXC_CCM_CCOSR, 4, 3);
- clk[IMX5_CLK_CKO1] = imx_clk_gate2("cko1", "cko1_podf", MXC_CCM_CCOSR, 7);
-
- clk[IMX5_CLK_CKO2_SEL] = imx_clk_mux("cko2_sel", MXC_CCM_CCOSR, 16, 5,
- mx53_cko2_sel, ARRAY_SIZE(mx53_cko2_sel));
- clk[IMX5_CLK_CKO2_PODF] = imx_clk_divider("cko2_podf", "cko2_sel", MXC_CCM_CCOSR, 21, 3);
- clk[IMX5_CLK_CKO2] = imx_clk_gate2("cko2", "cko2_podf", MXC_CCM_CCOSR, 24);
-
- imx_check_clocks(clk, ARRAY_SIZE(clk));
-
- clk_data.clks = clk;
- clk_data.clk_num = ARRAY_SIZE(clk);
- of_clk_add_provider(np, of_clk_src_onecell_get, &clk_data);
-
- /* set SDHC root clock to 200MHZ*/
- clk_set_rate(clk[IMX5_CLK_ESDHC_A_PODF], 200000000);
- clk_set_rate(clk[IMX5_CLK_ESDHC_B_PODF], 200000000);
-
- clk_prepare_enable(clk[IMX5_CLK_IIM_GATE]);
- imx_print_silicon_rev("i.MX50", IMX_CHIP_REVISION_1_1);
- clk_disable_unprepare(clk[IMX5_CLK_IIM_GATE]);
-
- r = clk_round_rate(clk[IMX5_CLK_USBOH3_PER_GATE], 54000000);
- clk_set_rate(clk[IMX5_CLK_USBOH3_PER_GATE], r);
-
- imx_register_uart_clocks(uart_clks_mx50_mx53);
-}
-CLK_OF_DECLARE(imx50_ccm, "fsl,imx50-ccm", mx50_clocks_init);
-
-static void __init mx51_clocks_init(struct device_node *np)
-{
- void __iomem *ccm_base;
- void __iomem *pll_base;
- u32 val;
-
- pll_base = ioremap(MX51_DPLL1_BASE, SZ_16K);
- WARN_ON(!pll_base);
- clk[IMX5_CLK_PLL1_SW] = imx_clk_pllv2("pll1_sw", "osc", pll_base);
-
- pll_base = ioremap(MX51_DPLL2_BASE, SZ_16K);
- WARN_ON(!pll_base);
- clk[IMX5_CLK_PLL2_SW] = imx_clk_pllv2("pll2_sw", "osc", pll_base);
-
- pll_base = ioremap(MX51_DPLL3_BASE, SZ_16K);
- WARN_ON(!pll_base);
- clk[IMX5_CLK_PLL3_SW] = imx_clk_pllv2("pll3_sw", "osc", pll_base);
-
- ccm_base = of_iomap(np, 0);
- WARN_ON(!ccm_base);
-
- mx5_clocks_common_init(ccm_base);
-
- clk[IMX5_CLK_LP_APM] = imx_clk_mux("lp_apm", MXC_CCM_CCSR, 9, 1,
- lp_apm_sel, ARRAY_SIZE(lp_apm_sel));
- clk[IMX5_CLK_IPU_DI0_SEL] = imx_clk_mux_flags("ipu_di0_sel", MXC_CCM_CSCMR2, 26, 3,
- mx51_ipu_di0_sel, ARRAY_SIZE(mx51_ipu_di0_sel), CLK_SET_RATE_PARENT);
- clk[IMX5_CLK_IPU_DI1_SEL] = imx_clk_mux_flags("ipu_di1_sel", MXC_CCM_CSCMR2, 29, 3,
- mx51_ipu_di1_sel, ARRAY_SIZE(mx51_ipu_di1_sel), CLK_SET_RATE_PARENT);
- clk[IMX5_CLK_TVE_EXT_SEL] = imx_clk_mux_flags("tve_ext_sel", MXC_CCM_CSCMR1, 6, 1,
- mx51_tve_ext_sel, ARRAY_SIZE(mx51_tve_ext_sel), CLK_SET_RATE_PARENT);
- clk[IMX5_CLK_TVE_SEL] = imx_clk_mux("tve_sel", MXC_CCM_CSCMR1, 7, 1,
- mx51_tve_sel, ARRAY_SIZE(mx51_tve_sel));
- clk[IMX5_CLK_TVE_GATE] = imx_clk_gate2("tve_gate", "tve_sel", MXC_CCM_CCGR2, 30);
- clk[IMX5_CLK_TVE_PRED] = imx_clk_divider("tve_pred", "pll3_sw", MXC_CCM_CDCDR, 28, 3);
- clk[IMX5_CLK_ESDHC1_PER_GATE] = imx_clk_gate2("esdhc1_per_gate", "esdhc_a_podf", MXC_CCM_CCGR3, 2);
- clk[IMX5_CLK_ESDHC2_PER_GATE] = imx_clk_gate2("esdhc2_per_gate", "esdhc_b_podf", MXC_CCM_CCGR3, 6);
- clk[IMX5_CLK_ESDHC3_PER_GATE] = imx_clk_gate2("esdhc3_per_gate", "esdhc_c_sel", MXC_CCM_CCGR3, 10);
- clk[IMX5_CLK_ESDHC4_PER_GATE] = imx_clk_gate2("esdhc4_per_gate", "esdhc_d_sel", MXC_CCM_CCGR3, 14);
- clk[IMX5_CLK_USB_PHY_GATE] = imx_clk_gate2("usb_phy_gate", "usb_phy_sel", MXC_CCM_CCGR2, 0);
- clk[IMX5_CLK_HSI2C_GATE] = imx_clk_gate2("hsi2c_gate", "ipg", MXC_CCM_CCGR1, 22);
- clk[IMX5_CLK_SCC2_IPG_GATE] = imx_clk_gate2("scc2_gate", "ipg", MXC_CCM_CCGR1, 30);
- clk[IMX5_CLK_MIPI_HSC1_GATE] = imx_clk_gate2_flags("mipi_hsc1_gate", "ipg", MXC_CCM_CCGR4, 6, CLK_IS_CRITICAL);
- clk[IMX5_CLK_MIPI_HSC2_GATE] = imx_clk_gate2_flags("mipi_hsc2_gate", "ipg", MXC_CCM_CCGR4, 8, CLK_IS_CRITICAL);
- clk[IMX5_CLK_MIPI_ESC_GATE] = imx_clk_gate2_flags("mipi_esc_gate", "ipg", MXC_CCM_CCGR4, 10, CLK_IS_CRITICAL);
- clk[IMX5_CLK_MIPI_HSP_GATE] = imx_clk_gate2_flags("mipi_hsp_gate", "ipg", MXC_CCM_CCGR4, 12, CLK_IS_CRITICAL);
- clk[IMX5_CLK_SPDIF_XTAL_SEL] = imx_clk_mux("spdif_xtal_sel", MXC_CCM_CSCMR1, 2, 2,
- mx51_spdif_xtal_sel, ARRAY_SIZE(mx51_spdif_xtal_sel));
- clk[IMX5_CLK_SPDIF1_SEL] = imx_clk_mux("spdif1_sel", MXC_CCM_CSCMR2, 2, 2,
- spdif_sel, ARRAY_SIZE(spdif_sel));
- clk[IMX5_CLK_SPDIF1_PRED] = imx_clk_divider("spdif1_pred", "spdif1_sel", MXC_CCM_CDCDR, 16, 3);
- clk[IMX5_CLK_SPDIF1_PODF] = imx_clk_divider("spdif1_podf", "spdif1_pred", MXC_CCM_CDCDR, 9, 6);
- clk[IMX5_CLK_SPDIF1_COM_SEL] = imx_clk_mux("spdif1_com_sel", MXC_CCM_CSCMR2, 5, 1,
- mx51_spdif1_com_sel, ARRAY_SIZE(mx51_spdif1_com_sel));
- clk[IMX5_CLK_SPDIF1_GATE] = imx_clk_gate2("spdif1_gate", "spdif1_com_sel", MXC_CCM_CCGR5, 28);
-
- imx_check_clocks(clk, ARRAY_SIZE(clk));
-
- clk_data.clks = clk;
- clk_data.clk_num = ARRAY_SIZE(clk);
- of_clk_add_provider(np, of_clk_src_onecell_get, &clk_data);
-
- /* set the usboh3 parent to pll2_sw */
- clk_set_parent(clk[IMX5_CLK_USBOH3_SEL], clk[IMX5_CLK_PLL2_SW]);
-
- /* set SDHC root clock to 166.25MHZ*/
- clk_set_rate(clk[IMX5_CLK_ESDHC_A_PODF], 166250000);
- clk_set_rate(clk[IMX5_CLK_ESDHC_B_PODF], 166250000);
-
- clk_prepare_enable(clk[IMX5_CLK_IIM_GATE]);
- imx_print_silicon_rev("i.MX51", mx51_revision());
- clk_disable_unprepare(clk[IMX5_CLK_IIM_GATE]);
-
- /*
- * Reference Manual says: Functionality of CCDR[18] and CLPCR[23] is no
- * longer supported. Set to one for better power saving.
- *
- * The effect of not setting these bits is that MIPI clocks can't be
- * enabled without the IPU clock being enabled aswell.
- */
- val = readl(MXC_CCM_CCDR);
- val |= 1 << 18;
- writel(val, MXC_CCM_CCDR);
-
- val = readl(MXC_CCM_CLPCR);
- val |= 1 << 23;
- writel(val, MXC_CCM_CLPCR);
-
- imx_register_uart_clocks(uart_clks_mx51);
-}
-CLK_OF_DECLARE(imx51_ccm, "fsl,imx51-ccm", mx51_clocks_init);
-
-static void __init mx53_clocks_init(struct device_node *np)
-{
- void __iomem *ccm_base;
- void __iomem *pll_base;
- unsigned long r;
-
- pll_base = ioremap(MX53_DPLL1_BASE, SZ_16K);
- WARN_ON(!pll_base);
- clk[IMX5_CLK_PLL1_SW] = imx_clk_pllv2("pll1_sw", "osc", pll_base);
-
- pll_base = ioremap(MX53_DPLL2_BASE, SZ_16K);
- WARN_ON(!pll_base);
- clk[IMX5_CLK_PLL2_SW] = imx_clk_pllv2("pll2_sw", "osc", pll_base);
-
- pll_base = ioremap(MX53_DPLL3_BASE, SZ_16K);
- WARN_ON(!pll_base);
- clk[IMX5_CLK_PLL3_SW] = imx_clk_pllv2("pll3_sw", "osc", pll_base);
-
- pll_base = ioremap(MX53_DPLL4_BASE, SZ_16K);
- WARN_ON(!pll_base);
- clk[IMX5_CLK_PLL4_SW] = imx_clk_pllv2("pll4_sw", "osc", pll_base);
-
- ccm_base = of_iomap(np, 0);
- WARN_ON(!ccm_base);
-
- mx5_clocks_common_init(ccm_base);
-
- clk[IMX5_CLK_LP_APM] = imx_clk_mux("lp_apm", MXC_CCM_CCSR, 10, 1,
- lp_apm_sel, ARRAY_SIZE(lp_apm_sel));
- clk[IMX5_CLK_LDB_DI1_DIV_3_5] = imx_clk_fixed_factor("ldb_di1_div_3_5", "ldb_di1_sel", 2, 7);
- clk[IMX5_CLK_LDB_DI1_DIV] = imx_clk_divider_flags("ldb_di1_div", "ldb_di1_div_3_5", MXC_CCM_CSCMR2, 11, 1, 0);
- clk[IMX5_CLK_LDB_DI1_SEL] = imx_clk_mux_flags("ldb_di1_sel", MXC_CCM_CSCMR2, 9, 1,
- mx53_ldb_di1_sel, ARRAY_SIZE(mx53_ldb_di1_sel), CLK_SET_RATE_PARENT);
- clk[IMX5_CLK_DI_PLL4_PODF] = imx_clk_divider("di_pll4_podf", "pll4_sw", MXC_CCM_CDCDR, 16, 3);
- clk[IMX5_CLK_LDB_DI0_DIV_3_5] = imx_clk_fixed_factor("ldb_di0_div_3_5", "ldb_di0_sel", 2, 7);
- clk[IMX5_CLK_LDB_DI0_DIV] = imx_clk_divider_flags("ldb_di0_div", "ldb_di0_div_3_5", MXC_CCM_CSCMR2, 10, 1, 0);
- clk[IMX5_CLK_LDB_DI0_SEL] = imx_clk_mux_flags("ldb_di0_sel", MXC_CCM_CSCMR2, 8, 1,
- mx53_ldb_di0_sel, ARRAY_SIZE(mx53_ldb_di0_sel), CLK_SET_RATE_PARENT);
- clk[IMX5_CLK_LDB_DI0_GATE] = imx_clk_gate2("ldb_di0_gate", "ldb_di0_div", MXC_CCM_CCGR6, 28);
- clk[IMX5_CLK_LDB_DI1_GATE] = imx_clk_gate2("ldb_di1_gate", "ldb_di1_div", MXC_CCM_CCGR6, 30);
- clk[IMX5_CLK_IPU_DI0_SEL] = imx_clk_mux_flags("ipu_di0_sel", MXC_CCM_CSCMR2, 26, 3,
- mx53_ipu_di0_sel, ARRAY_SIZE(mx53_ipu_di0_sel), CLK_SET_RATE_PARENT);
- clk[IMX5_CLK_IPU_DI1_SEL] = imx_clk_mux_flags("ipu_di1_sel", MXC_CCM_CSCMR2, 29, 3,
- mx53_ipu_di1_sel, ARRAY_SIZE(mx53_ipu_di1_sel), CLK_SET_RATE_PARENT);
- clk[IMX5_CLK_TVE_EXT_SEL] = imx_clk_mux_flags("tve_ext_sel", MXC_CCM_CSCMR1, 6, 1,
- mx53_tve_ext_sel, ARRAY_SIZE(mx53_tve_ext_sel), CLK_SET_RATE_PARENT);
- clk[IMX5_CLK_TVE_GATE] = imx_clk_gate2("tve_gate", "tve_pred", MXC_CCM_CCGR2, 30);
- clk[IMX5_CLK_TVE_PRED] = imx_clk_divider("tve_pred", "tve_ext_sel", MXC_CCM_CDCDR, 28, 3);
- clk[IMX5_CLK_ESDHC1_PER_GATE] = imx_clk_gate2("esdhc1_per_gate", "esdhc_a_podf", MXC_CCM_CCGR3, 2);
- clk[IMX5_CLK_ESDHC2_PER_GATE] = imx_clk_gate2("esdhc2_per_gate", "esdhc_c_sel", MXC_CCM_CCGR3, 6);
- clk[IMX5_CLK_ESDHC3_PER_GATE] = imx_clk_gate2("esdhc3_per_gate", "esdhc_b_podf", MXC_CCM_CCGR3, 10);
- clk[IMX5_CLK_ESDHC4_PER_GATE] = imx_clk_gate2("esdhc4_per_gate", "esdhc_d_sel", MXC_CCM_CCGR3, 14);
- clk[IMX5_CLK_USB_PHY1_GATE] = imx_clk_gate2("usb_phy1_gate", "usb_phy_sel", MXC_CCM_CCGR4, 10);
- clk[IMX5_CLK_USB_PHY2_GATE] = imx_clk_gate2("usb_phy2_gate", "usb_phy_sel", MXC_CCM_CCGR4, 12);
- clk[IMX5_CLK_CAN_SEL] = imx_clk_mux("can_sel", MXC_CCM_CSCMR2, 6, 2,
- mx53_can_sel, ARRAY_SIZE(mx53_can_sel));
- clk[IMX5_CLK_CAN1_SERIAL_GATE] = imx_clk_gate2("can1_serial_gate", "can_sel", MXC_CCM_CCGR6, 22);
- clk[IMX5_CLK_CAN1_IPG_GATE] = imx_clk_gate2("can1_ipg_gate", "ipg", MXC_CCM_CCGR6, 20);
- clk[IMX5_CLK_OCRAM] = imx_clk_gate2("ocram", "ahb", MXC_CCM_CCGR6, 2);
- clk[IMX5_CLK_CAN2_SERIAL_GATE] = imx_clk_gate2("can2_serial_gate", "can_sel", MXC_CCM_CCGR4, 8);
- clk[IMX5_CLK_CAN2_IPG_GATE] = imx_clk_gate2("can2_ipg_gate", "ipg", MXC_CCM_CCGR4, 6);
- clk[IMX5_CLK_I2C3_GATE] = imx_clk_gate2("i2c3_gate", "per_root", MXC_CCM_CCGR1, 22);
- clk[IMX5_CLK_SATA_GATE] = imx_clk_gate2("sata_gate", "ipg", MXC_CCM_CCGR4, 2);
-
- clk[IMX5_CLK_FIRI_SEL] = imx_clk_mux("firi_sel", MXC_CCM_CSCMR2, 12, 2,
- standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
- clk[IMX5_CLK_FIRI_PRED] = imx_clk_divider("firi_pred", "firi_sel", MXC_CCM_CSCDR3, 6, 3);
- clk[IMX5_CLK_FIRI_PODF] = imx_clk_divider("firi_podf", "firi_pred", MXC_CCM_CSCDR3, 0, 6);
- clk[IMX5_CLK_FIRI_SERIAL_GATE] = imx_clk_gate2("firi_serial_gate", "firi_podf", MXC_CCM_CCGR1, 28);
- clk[IMX5_CLK_FIRI_IPG_GATE] = imx_clk_gate2("firi_ipg_gate", "ipg", MXC_CCM_CCGR1, 26);
-
- clk[IMX5_CLK_CSI0_MCLK1_SEL] = imx_clk_mux("csi0_mclk1_sel", MXC_CCM_CSCMR2, 22, 2,
- standard_pll_sel, ARRAY_SIZE(standard_pll_sel));
- clk[IMX5_CLK_CSI0_MCLK1_PRED] = imx_clk_divider("csi0_mclk1_pred", "csi0_mclk1_sel", MXC_CCM_CSCDR4, 6, 3);
- clk[IMX5_CLK_CSI0_MCLK1_PODF] = imx_clk_divider("csi0_mclk1_podf", "csi0_mclk1_pred", MXC_CCM_CSCDR4, 0, 6);
- clk[IMX5_CLK_CSI0_MCLK1_GATE] = imx_clk_gate2("csi0_mclk1_serial_gate", "csi0_mclk1_podf", MXC_CCM_CCGR6, 4);
-
- clk[IMX5_CLK_IEEE1588_SEL] = imx_clk_mux("ieee1588_sel", MXC_CCM_CSCMR2, 14, 2,
- ieee1588_sels, ARRAY_SIZE(ieee1588_sels));
- clk[IMX5_CLK_IEEE1588_PRED] = imx_clk_divider("ieee1588_pred", "ieee1588_sel", MXC_CCM_CSCDR2, 6, 3);
- clk[IMX5_CLK_IEEE1588_PODF] = imx_clk_divider("ieee1588_podf", "ieee1588_pred", MXC_CCM_CSCDR2, 0, 6);
- clk[IMX5_CLK_IEEE1588_GATE] = imx_clk_gate2("ieee1588_serial_gate", "ieee1588_podf", MXC_CCM_CCGR7, 6);
- clk[IMX5_CLK_UART4_IPG_GATE] = imx_clk_gate2("uart4_ipg_gate", "ipg", MXC_CCM_CCGR7, 8);
- clk[IMX5_CLK_UART4_PER_GATE] = imx_clk_gate2("uart4_per_gate", "uart_root", MXC_CCM_CCGR7, 10);
- clk[IMX5_CLK_UART5_IPG_GATE] = imx_clk_gate2("uart5_ipg_gate", "ipg", MXC_CCM_CCGR7, 12);
- clk[IMX5_CLK_UART5_PER_GATE] = imx_clk_gate2("uart5_per_gate", "uart_root", MXC_CCM_CCGR7, 14);
-
- clk[IMX5_CLK_CKO1_SEL] = imx_clk_mux("cko1_sel", MXC_CCM_CCOSR, 0, 4,
- mx53_cko1_sel, ARRAY_SIZE(mx53_cko1_sel));
- clk[IMX5_CLK_CKO1_PODF] = imx_clk_divider("cko1_podf", "cko1_sel", MXC_CCM_CCOSR, 4, 3);
- clk[IMX5_CLK_CKO1] = imx_clk_gate2("cko1", "cko1_podf", MXC_CCM_CCOSR, 7);
-
- clk[IMX5_CLK_CKO2_SEL] = imx_clk_mux("cko2_sel", MXC_CCM_CCOSR, 16, 5,
- mx53_cko2_sel, ARRAY_SIZE(mx53_cko2_sel));
- clk[IMX5_CLK_CKO2_PODF] = imx_clk_divider("cko2_podf", "cko2_sel", MXC_CCM_CCOSR, 21, 3);
- clk[IMX5_CLK_CKO2] = imx_clk_gate2("cko2", "cko2_podf", MXC_CCM_CCOSR, 24);
- clk[IMX5_CLK_SPDIF_XTAL_SEL] = imx_clk_mux("spdif_xtal_sel", MXC_CCM_CSCMR1, 2, 2,
- mx53_spdif_xtal_sel, ARRAY_SIZE(mx53_spdif_xtal_sel));
- clk[IMX5_CLK_ARM] = imx_clk_cpu("arm", "cpu_podf",
- clk[IMX5_CLK_CPU_PODF],
- clk[IMX5_CLK_CPU_PODF_SEL],
- clk[IMX5_CLK_PLL1_SW],
- clk[IMX5_CLK_STEP_SEL]);
-
- imx_check_clocks(clk, ARRAY_SIZE(clk));
-
- clk_data.clks = clk;
- clk_data.clk_num = ARRAY_SIZE(clk);
- of_clk_add_provider(np, of_clk_src_onecell_get, &clk_data);
-
- /* set SDHC root clock to 200MHZ*/
- clk_set_rate(clk[IMX5_CLK_ESDHC_A_PODF], 200000000);
- clk_set_rate(clk[IMX5_CLK_ESDHC_B_PODF], 200000000);
-
- /* move can bus clk to 24MHz */
- clk_set_parent(clk[IMX5_CLK_CAN_SEL], clk[IMX5_CLK_LP_APM]);
-
- /* make sure step clock is running from 24MHz */
- clk_set_parent(clk[IMX5_CLK_STEP_SEL], clk[IMX5_CLK_LP_APM]);
-
- clk_prepare_enable(clk[IMX5_CLK_IIM_GATE]);
- imx_print_silicon_rev("i.MX53", mx53_revision());
- clk_disable_unprepare(clk[IMX5_CLK_IIM_GATE]);
-
- r = clk_round_rate(clk[IMX5_CLK_USBOH3_PER_GATE], 54000000);
- clk_set_rate(clk[IMX5_CLK_USBOH3_PER_GATE], r);
-
- imx_register_uart_clocks(uart_clks_mx50_mx53);
-}
-CLK_OF_DECLARE(imx53_ccm, "fsl,imx53-ccm", mx53_clocks_init);
static u32 share_count_ssi2;
static u32 share_count_ssi3;
+static struct clk ** const uart_clks[] __initconst = {
+ &clks[IMX6SLL_CLK_UART1_IPG],
+ &clks[IMX6SLL_CLK_UART1_SERIAL],
+ &clks[IMX6SLL_CLK_UART2_IPG],
+ &clks[IMX6SLL_CLK_UART2_SERIAL],
+ &clks[IMX6SLL_CLK_UART3_IPG],
+ &clks[IMX6SLL_CLK_UART3_SERIAL],
+ &clks[IMX6SLL_CLK_UART4_IPG],
+ &clks[IMX6SLL_CLK_UART4_SERIAL],
+ &clks[IMX6SLL_CLK_UART5_IPG],
+ &clks[IMX6SLL_CLK_UART5_SERIAL],
+ NULL
+};
+
static void __init imx6sll_clocks_init(struct device_node *ccm_node)
{
struct device_node *np;
clks[IMX6SLL_CLK_GPT_BUS] = imx_clk_gate2("gpt1_bus", "perclk", base + 0x6c, 20);
clks[IMX6SLL_CLK_GPT_SERIAL] = imx_clk_gate2("gpt1_serial", "perclk", base + 0x6c, 22);
clks[IMX6SLL_CLK_UART4_IPG] = imx_clk_gate2("uart4_ipg", "ipg", base + 0x6c, 24);
- clks[IMX6SLL_CLK_UART4_SERIAL] = imx_clk_gate2("uart4_serail", "uart_podf", base + 0x6c, 24);
+ clks[IMX6SLL_CLK_UART4_SERIAL] = imx_clk_gate2("uart4_serial", "uart_podf", base + 0x6c, 24);
clks[IMX6SLL_CLK_GPIO1] = imx_clk_gate2("gpio1", "ipg", base + 0x6c, 26);
clks[IMX6SLL_CLK_GPIO5] = imx_clk_gate2("gpio5", "ipg", base + 0x6c, 30);
clk_data.clk_num = ARRAY_SIZE(clks);
of_clk_add_provider(np, of_clk_src_onecell_get, &clk_data);
+ imx_register_uart_clocks(uart_clks);
+
/* Lower the AHB clock rate before changing the clock source. */
clk_set_rate(clks[IMX6SLL_CLK_AHB], 99000000);
clks[IMX7D_PLL_DRAM_MAIN] = imx_clk_pllv3(IMX_PLLV3_DDR_IMX7, "pll_dram_main", "osc", base + 0x70, 0x7f);
clks[IMX7D_PLL_SYS_MAIN] = imx_clk_pllv3(IMX_PLLV3_GENERIC, "pll_sys_main", "osc", base + 0xb0, 0x1);
clks[IMX7D_PLL_ENET_MAIN] = imx_clk_pllv3(IMX_PLLV3_ENET_IMX7, "pll_enet_main", "osc", base + 0xe0, 0x0);
- clks[IMX7D_PLL_AUDIO_MAIN] = imx_clk_pllv3(IMX_PLLV3_AV, "pll_audio_main", "osc", base + 0xf0, 0x7f);
- clks[IMX7D_PLL_VIDEO_MAIN] = imx_clk_pllv3(IMX_PLLV3_AV, "pll_video_main", "osc", base + 0x130, 0x7f);
+ clks[IMX7D_PLL_AUDIO_MAIN] = imx_clk_pllv3(IMX_PLLV3_AV_IMX7, "pll_audio_main", "osc", base + 0xf0, 0x7f);
+ clks[IMX7D_PLL_VIDEO_MAIN] = imx_clk_pllv3(IMX_PLLV3_AV_IMX7, "pll_video_main", "osc", base + 0x130, 0x7f);
clks[IMX7D_PLL_ARM_MAIN_BYPASS] = imx_clk_mux_flags("pll_arm_main_bypass", base + 0x60, 16, 1, pll_arm_bypass_sel, ARRAY_SIZE(pll_arm_bypass_sel), CLK_SET_RATE_PARENT);
clks[IMX7D_PLL_DRAM_MAIN_BYPASS] = imx_clk_mux_flags("pll_dram_main_bypass", base + 0x70, 16, 1, pll_dram_bypass_sel, ARRAY_SIZE(pll_dram_bypass_sel), CLK_SET_RATE_PARENT);
clks[IMX7ULP_CLK_DMA1] = imx_clk_hw_gate("dma1", "nic1_clk", base + 0x20, 30);
clks[IMX7ULP_CLK_RGPIO2P1] = imx_clk_hw_gate("rgpio2p1", "nic1_bus_clk", base + 0x3c, 30);
clks[IMX7ULP_CLK_DMA_MUX1] = imx_clk_hw_gate("dma_mux1", "nic1_bus_clk", base + 0x84, 30);
- clks[IMX7ULP_CLK_SNVS] = imx_clk_hw_gate("snvs", "nic1_bus_clk", base + 0x8c, 30);
clks[IMX7ULP_CLK_CAAM] = imx_clk_hw_gate("caam", "nic1_clk", base + 0x90, 30);
clks[IMX7ULP_CLK_LPTPM4] = imx7ulp_clk_composite("lptpm4", periph_bus_sels, ARRAY_SIZE(periph_bus_sels), true, false, true, base + 0x94);
clks[IMX7ULP_CLK_LPTPM5] = imx7ulp_clk_composite("lptpm5", periph_bus_sels, ARRAY_SIZE(periph_bus_sels), true, false, true, base + 0x98);
clks[IMX8MQ_CLK_DSI_DBI] = imx8m_clk_composite("dsi_dbi", imx8mq_dsi_dbi_sels, base + 0xbc00);
clks[IMX8MQ_CLK_DSI_ESC] = imx8m_clk_composite("dsi_esc", imx8mq_dsi_esc_sels, base + 0xbc80);
clks[IMX8MQ_CLK_DSI_AHB] = imx8m_clk_composite("dsi_ahb", imx8mq_dsi_ahb_sels, base + 0x9200);
+ clks[IMX8MQ_CLK_DSI_IPG_DIV] = imx_clk_divider2("dsi_ipg_div", "dsi_ahb", base + 0x9280, 0, 6);
clks[IMX8MQ_CLK_CSI1_CORE] = imx8m_clk_composite("csi1_core", imx8mq_csi1_core_sels, base + 0xbd00);
clks[IMX8MQ_CLK_CSI1_PHY_REF] = imx8m_clk_composite("csi1_phy_ref", imx8mq_csi1_phy_sels, base + 0xbd80);
clks[IMX8MQ_CLK_CSI1_ESC] = imx8m_clk_composite("csi1_esc", imx8mq_csi1_esc_sels, base + 0xbe00);
{
u32 val;
- return readl_poll_timeout(pfd->reg, val, val & pfd->vld_bit,
+ return readl_poll_timeout(pfd->reg, val, val & (1 << pfd->vld_bit),
0, LOCK_TIMEOUT_US);
}
spin_lock_irqsave(&pfd_lock, flags);
val = readl_relaxed(pfd->reg);
- val &= ~pfd->gate_bit;
+ val &= ~(1 << pfd->gate_bit);
writel_relaxed(val, pfd->reg);
spin_unlock_irqrestore(&pfd_lock, flags);
spin_lock_irqsave(&pfd_lock, flags);
val = readl_relaxed(pfd->reg);
- val |= pfd->gate_bit;
+ val |= (1 << pfd->gate_bit);
writel_relaxed(val, pfd->reg);
spin_unlock_irqrestore(&pfd_lock, flags);
}
{
struct clk_pfdv2 *pfd = to_clk_pfdv2(hw);
- if (readl_relaxed(pfd->reg) & pfd->gate_bit)
+ if (readl_relaxed(pfd->reg) & (1 << pfd->gate_bit))
return 0;
return 1;
return ERR_PTR(-ENOMEM);
pfd->reg = reg;
- pfd->gate_bit = 1 << ((idx + 1) * 8 - 1);
+ pfd->gate_bit = (idx + 1) * 8 - 1;
pfd->vld_bit = pfd->gate_bit - 1;
pfd->frac_off = idx * 8;
unsigned long parent_rate)
{
struct clk_pll14xx *pll = to_clk_pll14xx(hw);
- u32 mdiv, pdiv, sdiv, pll_gnrl, pll_div;
+ u32 mdiv, pdiv, sdiv, pll_div;
u64 fvco = parent_rate;
- pll_gnrl = readl_relaxed(pll->base);
pll_div = readl_relaxed(pll->base + 4);
mdiv = (pll_div & MDIV_MASK) >> MDIV_SHIFT;
pdiv = (pll_div & PDIV_MASK) >> PDIV_SHIFT;
unsigned long parent_rate)
{
struct clk_pll14xx *pll = to_clk_pll14xx(hw);
- u32 mdiv, pdiv, sdiv, pll_gnrl, pll_div_ctl0, pll_div_ctl1;
+ u32 mdiv, pdiv, sdiv, pll_div_ctl0, pll_div_ctl1;
short int kdiv;
u64 fvco = parent_rate;
- pll_gnrl = readl_relaxed(pll->base);
pll_div_ctl0 = readl_relaxed(pll->base + 4);
pll_div_ctl1 = readl_relaxed(pll->base + 8);
mdiv = (pll_div_ctl0 & MDIV_MASK) >> MDIV_SHIFT;
#define PLL_NUM_OFFSET 0x10
#define PLL_DENOM_OFFSET 0x20
+#define PLL_IMX7_NUM_OFFSET 0x20
+#define PLL_IMX7_DENOM_OFFSET 0x30
#define PLL_VF610_NUM_OFFSET 0x20
#define PLL_VF610_DENOM_OFFSET 0x30
u32 div_mask;
u32 div_shift;
unsigned long ref_clock;
+ u32 num_offset;
+ u32 denom_offset;
};
#define to_clk_pllv3(_hw) container_of(_hw, struct clk_pllv3, hw)
unsigned long parent_rate)
{
struct clk_pllv3 *pll = to_clk_pllv3(hw);
- u32 mfn = readl_relaxed(pll->base + PLL_NUM_OFFSET);
- u32 mfd = readl_relaxed(pll->base + PLL_DENOM_OFFSET);
+ u32 mfn = readl_relaxed(pll->base + pll->num_offset);
+ u32 mfd = readl_relaxed(pll->base + pll->denom_offset);
u32 div = readl_relaxed(pll->base) & pll->div_mask;
u64 temp64 = (u64)parent_rate;
val &= ~pll->div_mask;
val |= div;
writel_relaxed(val, pll->base);
- writel_relaxed(mfn, pll->base + PLL_NUM_OFFSET);
- writel_relaxed(mfd, pll->base + PLL_DENOM_OFFSET);
+ writel_relaxed(mfn, pll->base + pll->num_offset);
+ writel_relaxed(mfd, pll->base + pll->denom_offset);
return clk_pllv3_wait_lock(pll);
}
struct clk_pllv3 *pll = to_clk_pllv3(hw);
struct clk_pllv3_vf610_mf mf;
- mf.mfn = readl_relaxed(pll->base + PLL_VF610_NUM_OFFSET);
- mf.mfd = readl_relaxed(pll->base + PLL_VF610_DENOM_OFFSET);
+ mf.mfn = readl_relaxed(pll->base + pll->num_offset);
+ mf.mfd = readl_relaxed(pll->base + pll->denom_offset);
mf.mfi = (readl_relaxed(pll->base) & pll->div_mask) ? 22 : 20;
return clk_pllv3_vf610_mf_to_rate(parent_rate, mf);
val |= pll->div_mask; /* set bit for mfi=22 */
writel_relaxed(val, pll->base);
- writel_relaxed(mf.mfn, pll->base + PLL_VF610_NUM_OFFSET);
- writel_relaxed(mf.mfd, pll->base + PLL_VF610_DENOM_OFFSET);
+ writel_relaxed(mf.mfn, pll->base + pll->num_offset);
+ writel_relaxed(mf.mfd, pll->base + pll->denom_offset);
return clk_pllv3_wait_lock(pll);
}
return ERR_PTR(-ENOMEM);
pll->power_bit = BM_PLL_POWER;
+ pll->num_offset = PLL_NUM_OFFSET;
+ pll->denom_offset = PLL_DENOM_OFFSET;
switch (type) {
case IMX_PLLV3_SYS:
break;
case IMX_PLLV3_SYS_VF610:
ops = &clk_pllv3_vf610_ops;
+ pll->num_offset = PLL_VF610_NUM_OFFSET;
+ pll->denom_offset = PLL_VF610_DENOM_OFFSET;
break;
case IMX_PLLV3_USB_VF610:
pll->div_shift = 1;
+ /* fall through */
case IMX_PLLV3_USB:
ops = &clk_pllv3_ops;
pll->powerup_set = true;
break;
+ case IMX_PLLV3_AV_IMX7:
+ pll->num_offset = PLL_IMX7_NUM_OFFSET;
+ pll->denom_offset = PLL_IMX7_DENOM_OFFSET;
+ /* fall through */
case IMX_PLLV3_AV:
ops = &clk_pllv3_av_ops;
break;
break;
case IMX_PLLV3_DDR_IMX7:
pll->power_bit = IMX7_DDR_PLL_POWER;
+ pll->num_offset = PLL_IMX7_NUM_OFFSET;
+ pll->denom_offset = PLL_IMX7_DENOM_OFFSET;
ops = &clk_pllv3_av_ops;
break;
default:
/* PLL Denominator Register (xPLLDENOM) */
#define PLL_DENOM_OFFSET 0x14
+#define MAX_MFD 0x3fffffff
+#define DEFAULT_MFD 1000000
+
struct clk_pllv4 {
struct clk_hw hw;
void __iomem *base;
unsigned long parent_rate)
{
struct clk_pllv4 *pll = to_clk_pllv4(hw);
- u32 div;
+ u32 mult, mfn, mfd;
+ u64 temp64;
+
+ mult = readl_relaxed(pll->base + PLL_CFG_OFFSET);
+ mult &= BM_PLL_MULT;
+ mult >>= BP_PLL_MULT;
- div = readl_relaxed(pll->base + PLL_CFG_OFFSET);
- div &= BM_PLL_MULT;
- div >>= BP_PLL_MULT;
+ mfn = readl_relaxed(pll->base + PLL_NUM_OFFSET);
+ mfd = readl_relaxed(pll->base + PLL_DENOM_OFFSET);
+ temp64 = parent_rate;
+ temp64 *= mfn;
+ do_div(temp64, mfd);
- return parent_rate * div;
+ return (parent_rate * mult) + (u32)temp64;
}
static long clk_pllv4_round_rate(struct clk_hw *hw, unsigned long rate,
{
unsigned long parent_rate = *prate;
unsigned long round_rate, i;
+ u32 mfn, mfd = DEFAULT_MFD;
+ bool found = false;
+ u64 temp64;
for (i = 0; i < ARRAY_SIZE(pllv4_mult_table); i++) {
round_rate = parent_rate * pllv4_mult_table[i];
- if (rate >= round_rate)
- return round_rate;
+ if (rate >= round_rate) {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found) {
+ pr_warn("%s: unable to round rate %lu, parent rate %lu\n",
+ clk_hw_get_name(hw), rate, parent_rate);
+ return 0;
}
- return round_rate;
+ if (parent_rate <= MAX_MFD)
+ mfd = parent_rate;
+
+ temp64 = (u64)(rate - round_rate);
+ temp64 *= mfd;
+ do_div(temp64, parent_rate);
+ mfn = temp64;
+
+ /*
+ * NOTE: The value of numerator must always be configured to be
+ * less than the value of the denominator. If we can't get a proper
+ * pair of mfn/mfd, we simply return the round_rate without using
+ * the frac part.
+ */
+ if (mfn >= mfd)
+ return round_rate;
+
+ temp64 = (u64)parent_rate;
+ temp64 *= mfn;
+ do_div(temp64, mfd);
+
+ return round_rate + (u32)temp64;
}
static bool clk_pllv4_is_valid_mult(unsigned int mult)
unsigned long parent_rate)
{
struct clk_pllv4 *pll = to_clk_pllv4(hw);
- u32 val, mult;
+ u32 val, mult, mfn, mfd = DEFAULT_MFD;
+ u64 temp64;
mult = rate / parent_rate;
if (!clk_pllv4_is_valid_mult(mult))
return -EINVAL;
+ if (parent_rate <= MAX_MFD)
+ mfd = parent_rate;
+
+ temp64 = (u64)(rate - mult * parent_rate);
+ temp64 *= mfd;
+ do_div(temp64, parent_rate);
+ mfn = temp64;
+
val = readl_relaxed(pll->base + PLL_CFG_OFFSET);
val &= ~BM_PLL_MULT;
val |= mult << BP_PLL_MULT;
writel_relaxed(val, pll->base + PLL_CFG_OFFSET);
+ writel_relaxed(mfn, pll->base + PLL_NUM_OFFSET);
+ writel_relaxed(mfd, pll->base + PLL_DENOM_OFFSET);
+
return 0;
}
temp64 = parent_rate;
- val = clk_readl(pll->base + PLL_CFG0);
+ val = readl(pll->base + PLL_CFG0);
if (val & SSCG_PLL_BYPASS2_MASK) {
temp64 = parent_rate;
} else if (val & SSCG_PLL_BYPASS1_MASK) {
u32 val;
/* set bypass here too since the parent might be the same */
- val = clk_readl(pll->base + PLL_CFG0);
+ val = readl(pll->base + PLL_CFG0);
val &= ~SSCG_PLL_BYPASS_MASK;
val |= FIELD_PREP(SSCG_PLL_BYPASS_MASK, setup->bypass);
- clk_writel(val, pll->base + PLL_CFG0);
+ writel(val, pll->base + PLL_CFG0);
val = readl_relaxed(pll->base + PLL_CFG2);
val &= ~(PLL_DIVF1_MASK | PLL_DIVF2_MASK);
u32 val;
u8 ret = pll->parent;
- val = clk_readl(pll->base + PLL_CFG0);
+ val = readl(pll->base + PLL_CFG0);
if (val & SSCG_PLL_BYPASS2_MASK)
ret = pll->bypass2;
else if (val & SSCG_PLL_BYPASS1_MASK)
struct clk_sccg_pll *pll = to_clk_sccg_pll(hw);
u32 val;
- val = clk_readl(pll->base + PLL_CFG0);
+ val = readl(pll->base + PLL_CFG0);
val &= ~SSCG_PLL_BYPASS_MASK;
val |= FIELD_PREP(SSCG_PLL_BYPASS_MASK, pll->setup.bypass);
- clk_writel(val, pll->base + PLL_CFG0);
+ writel(val, pll->base + PLL_CFG0);
return clk_sccg_pll_wait_lock(pll);
}
IMX_PLLV3_ENET_IMX7,
IMX_PLLV3_SYS_VF610,
IMX_PLLV3_DDR_IMX7,
+ IMX_PLLV3_AV_IMX7,
};
struct clk *imx_clk_pllv3(enum imx_pllv3_type type, const char *name,
return clk_hw_register_fixed_rate(NULL, name, NULL, 0, rate);
}
-static inline struct clk_hw *imx_get_clk_hw_fixed(const char *name, int rate)
-{
- return clk_hw_register_fixed_rate(NULL, name, NULL, 0, rate);
-}
-
static inline struct clk *imx_clk_mux_ldb(const char *name, void __iomem *reg,
u8 shift, u8 width, const char * const *parents,
int num_parents)
.parents = { JZ4725B_CLK_EXT512, JZ4725B_CLK_OSC32K, -1, -1 },
.mux = { CGU_REG_OPCR, 2, 1},
},
+
+ [JZ4725B_CLK_UDC_PHY] = {
+ "udc_phy", CGU_CLK_GATE,
+ .parents = { JZ4725B_CLK_EXT, -1, -1, -1 },
+ .gate = { CGU_REG_OPCR, 6, true },
+ },
};
static void __init jz4725b_cgu_init(struct device_node *np)
default ARCH_MEDIATEK
---help---
This driver supports MediaTek MT8173 clocks.
+
+config COMMON_CLK_MT8183
+ bool "Clock driver for MediaTek MT8183"
+ depends on (ARCH_MEDIATEK && ARM64) || COMPILE_TEST
+ select COMMON_CLK_MEDIATEK
+ default ARCH_MEDIATEK && ARM64
+ help
+ This driver supports MediaTek MT8183 basic clocks.
+
+config COMMON_CLK_MT8183_AUDIOSYS
+ bool "Clock driver for MediaTek MT8183 audiosys"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 audiosys clocks.
+
+config COMMON_CLK_MT8183_CAMSYS
+ bool "Clock driver for MediaTek MT8183 camsys"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 camsys clocks.
+
+config COMMON_CLK_MT8183_IMGSYS
+ bool "Clock driver for MediaTek MT8183 imgsys"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 imgsys clocks.
+
+config COMMON_CLK_MT8183_IPU_CORE0
+ bool "Clock driver for MediaTek MT8183 ipu_core0"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 ipu_core0 clocks.
+
+config COMMON_CLK_MT8183_IPU_CORE1
+ bool "Clock driver for MediaTek MT8183 ipu_core1"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 ipu_core1 clocks.
+
+config COMMON_CLK_MT8183_IPU_ADL
+ bool "Clock driver for MediaTek MT8183 ipu_adl"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 ipu_adl clocks.
+
+config COMMON_CLK_MT8183_IPU_CONN
+ bool "Clock driver for MediaTek MT8183 ipu_conn"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 ipu_conn clocks.
+
+config COMMON_CLK_MT8183_MFGCFG
+ bool "Clock driver for MediaTek MT8183 mfgcfg"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 mfgcfg clocks.
+
+config COMMON_CLK_MT8183_MMSYS
+ bool "Clock driver for MediaTek MT8183 mmsys"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 mmsys clocks.
+
+config COMMON_CLK_MT8183_VDECSYS
+ bool "Clock driver for MediaTek MT8183 vdecsys"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 vdecsys clocks.
+
+config COMMON_CLK_MT8183_VENCSYS
+ bool "Clock driver for MediaTek MT8183 vencsys"
+ depends on COMMON_CLK_MT8183
+ help
+ This driver supports MediaTek MT8183 vencsys clocks.
+
+config COMMON_CLK_MT8516
+ bool "Clock driver for MediaTek MT8516"
+ depends on ARCH_MEDIATEK || COMPILE_TEST
+ select COMMON_CLK_MEDIATEK
+ default ARCH_MEDIATEK
+ help
+ This driver supports MediaTek MT8516 clocks.
+
endmenu
# SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_COMMON_CLK_MEDIATEK) += clk-mtk.o clk-pll.o clk-gate.o clk-apmixed.o clk-cpumux.o reset.o
+obj-$(CONFIG_COMMON_CLK_MEDIATEK) += clk-mtk.o clk-pll.o clk-gate.o clk-apmixed.o clk-cpumux.o reset.o clk-mux.o
+
obj-$(CONFIG_COMMON_CLK_MT6797) += clk-mt6797.o
obj-$(CONFIG_COMMON_CLK_MT6797_IMGSYS) += clk-mt6797-img.o
obj-$(CONFIG_COMMON_CLK_MT6797_MMSYS) += clk-mt6797-mm.o
obj-$(CONFIG_COMMON_CLK_MT7629_HIFSYS) += clk-mt7629-hif.o
obj-$(CONFIG_COMMON_CLK_MT8135) += clk-mt8135.o
obj-$(CONFIG_COMMON_CLK_MT8173) += clk-mt8173.o
+obj-$(CONFIG_COMMON_CLK_MT8183) += clk-mt8183.o
+obj-$(CONFIG_COMMON_CLK_MT8183_AUDIOSYS) += clk-mt8183-audio.o
+obj-$(CONFIG_COMMON_CLK_MT8183_CAMSYS) += clk-mt8183-cam.o
+obj-$(CONFIG_COMMON_CLK_MT8183_IMGSYS) += clk-mt8183-img.o
+obj-$(CONFIG_COMMON_CLK_MT8183_IPU_CORE0) += clk-mt8183-ipu0.o
+obj-$(CONFIG_COMMON_CLK_MT8183_IPU_CORE1) += clk-mt8183-ipu1.o
+obj-$(CONFIG_COMMON_CLK_MT8183_IPU_ADL) += clk-mt8183-ipu_adl.o
+obj-$(CONFIG_COMMON_CLK_MT8183_IPU_CONN) += clk-mt8183-ipu_conn.o
+obj-$(CONFIG_COMMON_CLK_MT8183_MFGCFG) += clk-mt8183-mfgcfg.o
+obj-$(CONFIG_COMMON_CLK_MT8183_MMSYS) += clk-mt8183-mm.o
+obj-$(CONFIG_COMMON_CLK_MT8183_VDECSYS) += clk-mt8183-vdec.o
+obj-$(CONFIG_COMMON_CLK_MT8183_VENCSYS) += clk-mt8183-venc.o
+obj-$(CONFIG_COMMON_CLK_MT8516) += clk-mt8516.o
const struct clk_ops *ops,
unsigned long flags);
+#define GATE_MTK_FLAGS(_id, _name, _parent, _regs, _shift, \
+ _ops, _flags) { \
+ .id = _id, \
+ .name = _name, \
+ .parent_name = _parent, \
+ .regs = _regs, \
+ .shift = _shift, \
+ .ops = _ops, \
+ .flags = _flags, \
+ }
+
+#define GATE_MTK(_id, _name, _parent, _regs, _shift, _ops) \
+ GATE_MTK_FLAGS(_id, _name, _parent, _regs, _shift, _ops, 0)
+
#endif /* __DRV_CLK_GATE_H */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs audio0_cg_regs = {
+ .set_ofs = 0x0,
+ .clr_ofs = 0x0,
+ .sta_ofs = 0x0,
+};
+
+static const struct mtk_gate_regs audio1_cg_regs = {
+ .set_ofs = 0x4,
+ .clr_ofs = 0x4,
+ .sta_ofs = 0x4,
+};
+
+#define GATE_AUDIO0(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &audio0_cg_regs, _shift, \
+ &mtk_clk_gate_ops_no_setclr)
+
+#define GATE_AUDIO1(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &audio1_cg_regs, _shift, \
+ &mtk_clk_gate_ops_no_setclr)
+
+static const struct mtk_gate audio_clks[] = {
+ /* AUDIO0 */
+ GATE_AUDIO0(CLK_AUDIO_AFE, "aud_afe", "audio_sel",
+ 2),
+ GATE_AUDIO0(CLK_AUDIO_22M, "aud_22m", "aud_eng1_sel",
+ 8),
+ GATE_AUDIO0(CLK_AUDIO_24M, "aud_24m", "aud_eng2_sel",
+ 9),
+ GATE_AUDIO0(CLK_AUDIO_APLL2_TUNER, "aud_apll2_tuner", "aud_eng2_sel",
+ 18),
+ GATE_AUDIO0(CLK_AUDIO_APLL_TUNER, "aud_apll_tuner", "aud_eng1_sel",
+ 19),
+ GATE_AUDIO0(CLK_AUDIO_TDM, "aud_tdm", "apll12_divb",
+ 20),
+ GATE_AUDIO0(CLK_AUDIO_ADC, "aud_adc", "audio_sel",
+ 24),
+ GATE_AUDIO0(CLK_AUDIO_DAC, "aud_dac", "audio_sel",
+ 25),
+ GATE_AUDIO0(CLK_AUDIO_DAC_PREDIS, "aud_dac_predis", "audio_sel",
+ 26),
+ GATE_AUDIO0(CLK_AUDIO_TML, "aud_tml", "audio_sel",
+ 27),
+ /* AUDIO1 */
+ GATE_AUDIO1(CLK_AUDIO_I2S1, "aud_i2s1", "audio_sel",
+ 4),
+ GATE_AUDIO1(CLK_AUDIO_I2S2, "aud_i2s2", "audio_sel",
+ 5),
+ GATE_AUDIO1(CLK_AUDIO_I2S3, "aud_i2s3", "audio_sel",
+ 6),
+ GATE_AUDIO1(CLK_AUDIO_I2S4, "aud_i2s4", "audio_sel",
+ 7),
+ GATE_AUDIO1(CLK_AUDIO_PDN_ADDA6_ADC, "aud_pdn_adda6_adc", "audio_sel",
+ 20),
+};
+
+static int clk_mt8183_audio_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ int r;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_AUDIO_NR_CLK);
+
+ mtk_clk_register_gates(node, audio_clks, ARRAY_SIZE(audio_clks),
+ clk_data);
+
+ r = of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+ if (r)
+ return r;
+
+ r = devm_of_platform_populate(&pdev->dev);
+ if (r)
+ of_clk_del_provider(node);
+
+ return r;
+}
+
+static const struct of_device_id of_match_clk_mt8183_audio[] = {
+ { .compatible = "mediatek,mt8183-audiosys", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_audio_drv = {
+ .probe = clk_mt8183_audio_probe,
+ .driver = {
+ .name = "clk-mt8183-audio",
+ .of_match_table = of_match_clk_mt8183_audio,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_audio_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs cam_cg_regs = {
+ .set_ofs = 0x4,
+ .clr_ofs = 0x8,
+ .sta_ofs = 0x0,
+};
+
+#define GATE_CAM(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &cam_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+static const struct mtk_gate cam_clks[] = {
+ GATE_CAM(CLK_CAM_LARB6, "cam_larb6", "cam_sel", 0),
+ GATE_CAM(CLK_CAM_DFP_VAD, "cam_dfp_vad", "cam_sel", 1),
+ GATE_CAM(CLK_CAM_LARB3, "cam_larb3", "cam_sel", 2),
+ GATE_CAM(CLK_CAM_CAM, "cam_cam", "cam_sel", 6),
+ GATE_CAM(CLK_CAM_CAMTG, "cam_camtg", "cam_sel", 7),
+ GATE_CAM(CLK_CAM_SENINF, "cam_seninf", "cam_sel", 8),
+ GATE_CAM(CLK_CAM_CAMSV0, "cam_camsv0", "cam_sel", 9),
+ GATE_CAM(CLK_CAM_CAMSV1, "cam_camsv1", "cam_sel", 10),
+ GATE_CAM(CLK_CAM_CAMSV2, "cam_camsv2", "cam_sel", 11),
+ GATE_CAM(CLK_CAM_CCU, "cam_ccu", "cam_sel", 12),
+};
+
+static int clk_mt8183_cam_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_CAM_NR_CLK);
+
+ mtk_clk_register_gates(node, cam_clks, ARRAY_SIZE(cam_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183_cam[] = {
+ { .compatible = "mediatek,mt8183-camsys", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_cam_drv = {
+ .probe = clk_mt8183_cam_probe,
+ .driver = {
+ .name = "clk-mt8183-cam",
+ .of_match_table = of_match_clk_mt8183_cam,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_cam_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs img_cg_regs = {
+ .set_ofs = 0x4,
+ .clr_ofs = 0x8,
+ .sta_ofs = 0x0,
+};
+
+#define GATE_IMG(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &img_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+static const struct mtk_gate img_clks[] = {
+ GATE_IMG(CLK_IMG_LARB5, "img_larb5", "img_sel", 0),
+ GATE_IMG(CLK_IMG_LARB2, "img_larb2", "img_sel", 1),
+ GATE_IMG(CLK_IMG_DIP, "img_dip", "img_sel", 2),
+ GATE_IMG(CLK_IMG_FDVT, "img_fdvt", "img_sel", 3),
+ GATE_IMG(CLK_IMG_DPE, "img_dpe", "img_sel", 4),
+ GATE_IMG(CLK_IMG_RSC, "img_rsc", "img_sel", 5),
+ GATE_IMG(CLK_IMG_MFB, "img_mfb", "img_sel", 6),
+ GATE_IMG(CLK_IMG_WPE_A, "img_wpe_a", "img_sel", 7),
+ GATE_IMG(CLK_IMG_WPE_B, "img_wpe_b", "img_sel", 8),
+ GATE_IMG(CLK_IMG_OWE, "img_owe", "img_sel", 9),
+};
+
+static int clk_mt8183_img_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_IMG_NR_CLK);
+
+ mtk_clk_register_gates(node, img_clks, ARRAY_SIZE(img_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183_img[] = {
+ { .compatible = "mediatek,mt8183-imgsys", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_img_drv = {
+ .probe = clk_mt8183_img_probe,
+ .driver = {
+ .name = "clk-mt8183-img",
+ .of_match_table = of_match_clk_mt8183_img,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_img_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs ipu_core0_cg_regs = {
+ .set_ofs = 0x4,
+ .clr_ofs = 0x8,
+ .sta_ofs = 0x0,
+};
+
+#define GATE_IPU_CORE0(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &ipu_core0_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+static const struct mtk_gate ipu_core0_clks[] = {
+ GATE_IPU_CORE0(CLK_IPU_CORE0_JTAG, "ipu_core0_jtag", "dsp_sel", 0),
+ GATE_IPU_CORE0(CLK_IPU_CORE0_AXI, "ipu_core0_axi", "dsp_sel", 1),
+ GATE_IPU_CORE0(CLK_IPU_CORE0_IPU, "ipu_core0_ipu", "dsp_sel", 2),
+};
+
+static int clk_mt8183_ipu_core0_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_IPU_CORE0_NR_CLK);
+
+ mtk_clk_register_gates(node, ipu_core0_clks, ARRAY_SIZE(ipu_core0_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183_ipu_core0[] = {
+ { .compatible = "mediatek,mt8183-ipu_core0", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_ipu_core0_drv = {
+ .probe = clk_mt8183_ipu_core0_probe,
+ .driver = {
+ .name = "clk-mt8183-ipu_core0",
+ .of_match_table = of_match_clk_mt8183_ipu_core0,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_ipu_core0_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs ipu_core1_cg_regs = {
+ .set_ofs = 0x4,
+ .clr_ofs = 0x8,
+ .sta_ofs = 0x0,
+};
+
+#define GATE_IPU_CORE1(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &ipu_core1_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+static const struct mtk_gate ipu_core1_clks[] = {
+ GATE_IPU_CORE1(CLK_IPU_CORE1_JTAG, "ipu_core1_jtag", "dsp_sel", 0),
+ GATE_IPU_CORE1(CLK_IPU_CORE1_AXI, "ipu_core1_axi", "dsp_sel", 1),
+ GATE_IPU_CORE1(CLK_IPU_CORE1_IPU, "ipu_core1_ipu", "dsp_sel", 2),
+};
+
+static int clk_mt8183_ipu_core1_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_IPU_CORE1_NR_CLK);
+
+ mtk_clk_register_gates(node, ipu_core1_clks, ARRAY_SIZE(ipu_core1_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183_ipu_core1[] = {
+ { .compatible = "mediatek,mt8183-ipu_core1", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_ipu_core1_drv = {
+ .probe = clk_mt8183_ipu_core1_probe,
+ .driver = {
+ .name = "clk-mt8183-ipu_core1",
+ .of_match_table = of_match_clk_mt8183_ipu_core1,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_ipu_core1_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs ipu_adl_cg_regs = {
+ .set_ofs = 0x204,
+ .clr_ofs = 0x204,
+ .sta_ofs = 0x204,
+};
+
+#define GATE_IPU_ADL_I(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &ipu_adl_cg_regs, _shift, \
+ &mtk_clk_gate_ops_no_setclr_inv)
+
+static const struct mtk_gate ipu_adl_clks[] = {
+ GATE_IPU_ADL_I(CLK_IPU_ADL_CABGEN, "ipu_adl_cabgen", "dsp_sel", 24),
+};
+
+static int clk_mt8183_ipu_adl_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_IPU_ADL_NR_CLK);
+
+ mtk_clk_register_gates(node, ipu_adl_clks, ARRAY_SIZE(ipu_adl_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183_ipu_adl[] = {
+ { .compatible = "mediatek,mt8183-ipu_adl", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_ipu_adl_drv = {
+ .probe = clk_mt8183_ipu_adl_probe,
+ .driver = {
+ .name = "clk-mt8183-ipu_adl",
+ .of_match_table = of_match_clk_mt8183_ipu_adl,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_ipu_adl_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs ipu_conn_cg_regs = {
+ .set_ofs = 0x4,
+ .clr_ofs = 0x8,
+ .sta_ofs = 0x0,
+};
+
+static const struct mtk_gate_regs ipu_conn_apb_cg_regs = {
+ .set_ofs = 0x10,
+ .clr_ofs = 0x10,
+ .sta_ofs = 0x10,
+};
+
+static const struct mtk_gate_regs ipu_conn_axi_cg_regs = {
+ .set_ofs = 0x18,
+ .clr_ofs = 0x18,
+ .sta_ofs = 0x18,
+};
+
+static const struct mtk_gate_regs ipu_conn_axi1_cg_regs = {
+ .set_ofs = 0x1c,
+ .clr_ofs = 0x1c,
+ .sta_ofs = 0x1c,
+};
+
+static const struct mtk_gate_regs ipu_conn_axi2_cg_regs = {
+ .set_ofs = 0x20,
+ .clr_ofs = 0x20,
+ .sta_ofs = 0x20,
+};
+
+#define GATE_IPU_CONN(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &ipu_conn_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+#define GATE_IPU_CONN_APB(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &ipu_conn_apb_cg_regs, _shift, \
+ &mtk_clk_gate_ops_no_setclr)
+
+#define GATE_IPU_CONN_AXI_I(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &ipu_conn_axi_cg_regs, _shift, \
+ &mtk_clk_gate_ops_no_setclr_inv)
+
+#define GATE_IPU_CONN_AXI1_I(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &ipu_conn_axi1_cg_regs, _shift, \
+ &mtk_clk_gate_ops_no_setclr_inv)
+
+#define GATE_IPU_CONN_AXI2_I(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &ipu_conn_axi2_cg_regs, _shift, \
+ &mtk_clk_gate_ops_no_setclr_inv)
+
+static const struct mtk_gate ipu_conn_clks[] = {
+ GATE_IPU_CONN(CLK_IPU_CONN_IPU,
+ "ipu_conn_ipu", "dsp_sel", 0),
+ GATE_IPU_CONN(CLK_IPU_CONN_AHB,
+ "ipu_conn_ahb", "dsp_sel", 1),
+ GATE_IPU_CONN(CLK_IPU_CONN_AXI,
+ "ipu_conn_axi", "dsp_sel", 2),
+ GATE_IPU_CONN(CLK_IPU_CONN_ISP,
+ "ipu_conn_isp", "dsp_sel", 3),
+ GATE_IPU_CONN(CLK_IPU_CONN_CAM_ADL,
+ "ipu_conn_cam_adl", "dsp_sel", 4),
+ GATE_IPU_CONN(CLK_IPU_CONN_IMG_ADL,
+ "ipu_conn_img_adl", "dsp_sel", 5),
+ GATE_IPU_CONN_APB(CLK_IPU_CONN_DAP_RX,
+ "ipu_conn_dap_rx", "dsp1_sel", 0),
+ GATE_IPU_CONN_APB(CLK_IPU_CONN_APB2AXI,
+ "ipu_conn_apb2axi", "dsp1_sel", 3),
+ GATE_IPU_CONN_APB(CLK_IPU_CONN_APB2AHB,
+ "ipu_conn_apb2ahb", "dsp1_sel", 20),
+ GATE_IPU_CONN_AXI_I(CLK_IPU_CONN_IPU_CAB1TO2,
+ "ipu_conn_ipu_cab1to2", "dsp1_sel", 6),
+ GATE_IPU_CONN_AXI_I(CLK_IPU_CONN_IPU1_CAB1TO2,
+ "ipu_conn_ipu1_cab1to2", "dsp1_sel", 13),
+ GATE_IPU_CONN_AXI_I(CLK_IPU_CONN_IPU2_CAB1TO2,
+ "ipu_conn_ipu2_cab1to2", "dsp1_sel", 20),
+ GATE_IPU_CONN_AXI1_I(CLK_IPU_CONN_CAB3TO3,
+ "ipu_conn_cab3to3", "dsp1_sel", 0),
+ GATE_IPU_CONN_AXI2_I(CLK_IPU_CONN_CAB2TO1,
+ "ipu_conn_cab2to1", "dsp1_sel", 14),
+ GATE_IPU_CONN_AXI2_I(CLK_IPU_CONN_CAB3TO1_SLICE,
+ "ipu_conn_cab3to1_slice", "dsp1_sel", 17),
+};
+
+static int clk_mt8183_ipu_conn_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_IPU_CONN_NR_CLK);
+
+ mtk_clk_register_gates(node, ipu_conn_clks, ARRAY_SIZE(ipu_conn_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183_ipu_conn[] = {
+ { .compatible = "mediatek,mt8183-ipu_conn", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_ipu_conn_drv = {
+ .probe = clk_mt8183_ipu_conn_probe,
+ .driver = {
+ .name = "clk-mt8183-ipu_conn",
+ .of_match_table = of_match_clk_mt8183_ipu_conn,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_ipu_conn_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs mfg_cg_regs = {
+ .set_ofs = 0x4,
+ .clr_ofs = 0x8,
+ .sta_ofs = 0x0,
+};
+
+#define GATE_MFG(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &mfg_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+static const struct mtk_gate mfg_clks[] = {
+ GATE_MFG(CLK_MFG_BG3D, "mfg_bg3d", "mfg_sel", 0)
+};
+
+static int clk_mt8183_mfg_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_MFG_NR_CLK);
+
+ mtk_clk_register_gates(node, mfg_clks, ARRAY_SIZE(mfg_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183_mfg[] = {
+ { .compatible = "mediatek,mt8183-mfgcfg", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_mfg_drv = {
+ .probe = clk_mt8183_mfg_probe,
+ .driver = {
+ .name = "clk-mt8183-mfg",
+ .of_match_table = of_match_clk_mt8183_mfg,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_mfg_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs mm0_cg_regs = {
+ .set_ofs = 0x104,
+ .clr_ofs = 0x108,
+ .sta_ofs = 0x100,
+};
+
+static const struct mtk_gate_regs mm1_cg_regs = {
+ .set_ofs = 0x114,
+ .clr_ofs = 0x118,
+ .sta_ofs = 0x110,
+};
+
+#define GATE_MM0(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &mm0_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+#define GATE_MM1(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &mm1_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+static const struct mtk_gate mm_clks[] = {
+ /* MM0 */
+ GATE_MM0(CLK_MM_SMI_COMMON, "mm_smi_common", "mm_sel", 0),
+ GATE_MM0(CLK_MM_SMI_LARB0, "mm_smi_larb0", "mm_sel", 1),
+ GATE_MM0(CLK_MM_SMI_LARB1, "mm_smi_larb1", "mm_sel", 2),
+ GATE_MM0(CLK_MM_GALS_COMM0, "mm_gals_comm0", "mm_sel", 3),
+ GATE_MM0(CLK_MM_GALS_COMM1, "mm_gals_comm1", "mm_sel", 4),
+ GATE_MM0(CLK_MM_GALS_CCU2MM, "mm_gals_ccu2mm", "mm_sel", 5),
+ GATE_MM0(CLK_MM_GALS_IPU12MM, "mm_gals_ipu12mm", "mm_sel", 6),
+ GATE_MM0(CLK_MM_GALS_IMG2MM, "mm_gals_img2mm", "mm_sel", 7),
+ GATE_MM0(CLK_MM_GALS_CAM2MM, "mm_gals_cam2mm", "mm_sel", 8),
+ GATE_MM0(CLK_MM_GALS_IPU2MM, "mm_gals_ipu2mm", "mm_sel", 9),
+ GATE_MM0(CLK_MM_MDP_DL_TXCK, "mm_mdp_dl_txck", "mm_sel", 10),
+ GATE_MM0(CLK_MM_IPU_DL_TXCK, "mm_ipu_dl_txck", "mm_sel", 11),
+ GATE_MM0(CLK_MM_MDP_RDMA0, "mm_mdp_rdma0", "mm_sel", 12),
+ GATE_MM0(CLK_MM_MDP_RDMA1, "mm_mdp_rdma1", "mm_sel", 13),
+ GATE_MM0(CLK_MM_MDP_RSZ0, "mm_mdp_rsz0", "mm_sel", 14),
+ GATE_MM0(CLK_MM_MDP_RSZ1, "mm_mdp_rsz1", "mm_sel", 15),
+ GATE_MM0(CLK_MM_MDP_TDSHP, "mm_mdp_tdshp", "mm_sel", 16),
+ GATE_MM0(CLK_MM_MDP_WROT0, "mm_mdp_wrot0", "mm_sel", 17),
+ GATE_MM0(CLK_MM_MDP_WDMA0, "mm_mdp_wdma0", "mm_sel", 18),
+ GATE_MM0(CLK_MM_FAKE_ENG, "mm_fake_eng", "mm_sel", 19),
+ GATE_MM0(CLK_MM_DISP_OVL0, "mm_disp_ovl0", "mm_sel", 20),
+ GATE_MM0(CLK_MM_DISP_OVL0_2L, "mm_disp_ovl0_2l", "mm_sel", 21),
+ GATE_MM0(CLK_MM_DISP_OVL1_2L, "mm_disp_ovl1_2l", "mm_sel", 22),
+ GATE_MM0(CLK_MM_DISP_RDMA0, "mm_disp_rdma0", "mm_sel", 23),
+ GATE_MM0(CLK_MM_DISP_RDMA1, "mm_disp_rdma1", "mm_sel", 24),
+ GATE_MM0(CLK_MM_DISP_WDMA0, "mm_disp_wdma0", "mm_sel", 25),
+ GATE_MM0(CLK_MM_DISP_COLOR0, "mm_disp_color0", "mm_sel", 26),
+ GATE_MM0(CLK_MM_DISP_CCORR0, "mm_disp_ccorr0", "mm_sel", 27),
+ GATE_MM0(CLK_MM_DISP_AAL0, "mm_disp_aal0", "mm_sel", 28),
+ GATE_MM0(CLK_MM_DISP_GAMMA0, "mm_disp_gamma0", "mm_sel", 29),
+ GATE_MM0(CLK_MM_DISP_DITHER0, "mm_disp_dither0", "mm_sel", 30),
+ GATE_MM0(CLK_MM_DISP_SPLIT, "mm_disp_split", "mm_sel", 31),
+ /* MM1 */
+ GATE_MM1(CLK_MM_DSI0_MM, "mm_dsi0_mm", "mm_sel", 0),
+ GATE_MM1(CLK_MM_DSI0_IF, "mm_dsi0_if", "mm_sel", 1),
+ GATE_MM1(CLK_MM_DPI_MM, "mm_dpi_mm", "mm_sel", 2),
+ GATE_MM1(CLK_MM_DPI_IF, "mm_dpi_if", "dpi0_sel", 3),
+ GATE_MM1(CLK_MM_FAKE_ENG2, "mm_fake_eng2", "mm_sel", 4),
+ GATE_MM1(CLK_MM_MDP_DL_RX, "mm_mdp_dl_rx", "mm_sel", 5),
+ GATE_MM1(CLK_MM_IPU_DL_RX, "mm_ipu_dl_rx", "mm_sel", 6),
+ GATE_MM1(CLK_MM_26M, "mm_26m", "f_f26m_ck", 7),
+ GATE_MM1(CLK_MM_MMSYS_R2Y, "mm_mmsys_r2y", "mm_sel", 8),
+ GATE_MM1(CLK_MM_DISP_RSZ, "mm_disp_rsz", "mm_sel", 9),
+ GATE_MM1(CLK_MM_MDP_AAL, "mm_mdp_aal", "mm_sel", 10),
+ GATE_MM1(CLK_MM_MDP_CCORR, "mm_mdp_ccorr", "mm_sel", 11),
+ GATE_MM1(CLK_MM_DBI_MM, "mm_dbi_mm", "mm_sel", 12),
+ GATE_MM1(CLK_MM_DBI_IF, "mm_dbi_if", "dpi0_sel", 13),
+};
+
+static int clk_mt8183_mm_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_MM_NR_CLK);
+
+ mtk_clk_register_gates(node, mm_clks, ARRAY_SIZE(mm_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183_mm[] = {
+ { .compatible = "mediatek,mt8183-mmsys", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_mm_drv = {
+ .probe = clk_mt8183_mm_probe,
+ .driver = {
+ .name = "clk-mt8183-mm",
+ .of_match_table = of_match_clk_mt8183_mm,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_mm_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs vdec0_cg_regs = {
+ .set_ofs = 0x0,
+ .clr_ofs = 0x4,
+ .sta_ofs = 0x0,
+};
+
+static const struct mtk_gate_regs vdec1_cg_regs = {
+ .set_ofs = 0x8,
+ .clr_ofs = 0xc,
+ .sta_ofs = 0x8,
+};
+
+#define GATE_VDEC0_I(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &vdec0_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr_inv)
+
+#define GATE_VDEC1_I(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &vdec1_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr_inv)
+
+static const struct mtk_gate vdec_clks[] = {
+ /* VDEC0 */
+ GATE_VDEC0_I(CLK_VDEC_VDEC, "vdec_vdec", "mm_sel", 0),
+ /* VDEC1 */
+ GATE_VDEC1_I(CLK_VDEC_LARB1, "vdec_larb1", "mm_sel", 0),
+};
+
+static int clk_mt8183_vdec_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_VDEC_NR_CLK);
+
+ mtk_clk_register_gates(node, vdec_clks, ARRAY_SIZE(vdec_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183_vdec[] = {
+ { .compatible = "mediatek,mt8183-vdecsys", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_vdec_drv = {
+ .probe = clk_mt8183_vdec_probe,
+ .driver = {
+ .name = "clk-mt8183-vdec",
+ .of_match_table = of_match_clk_mt8183_vdec,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_vdec_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/clk-provider.h>
+#include <linux/platform_device.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static const struct mtk_gate_regs venc_cg_regs = {
+ .set_ofs = 0x4,
+ .clr_ofs = 0x8,
+ .sta_ofs = 0x0,
+};
+
+#define GATE_VENC_I(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &venc_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr_inv)
+
+static const struct mtk_gate venc_clks[] = {
+ GATE_VENC_I(CLK_VENC_LARB, "venc_larb",
+ "mm_sel", 0),
+ GATE_VENC_I(CLK_VENC_VENC, "venc_venc",
+ "mm_sel", 4),
+ GATE_VENC_I(CLK_VENC_JPGENC, "venc_jpgenc",
+ "mm_sel", 8),
+};
+
+static int clk_mt8183_venc_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_VENC_NR_CLK);
+
+ mtk_clk_register_gates(node, venc_clks, ARRAY_SIZE(venc_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183_venc[] = {
+ { .compatible = "mediatek,mt8183-vencsys", },
+ {}
+};
+
+static struct platform_driver clk_mt8183_venc_drv = {
+ .probe = clk_mt8183_venc_probe,
+ .driver = {
+ .name = "clk-mt8183-venc",
+ .of_match_table = of_match_clk_mt8183_venc,
+ },
+};
+
+builtin_platform_driver(clk_mt8183_venc_drv);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Weiyi Lu <weiyi.lu@mediatek.com>
+
+#include <linux/delay.h>
+#include <linux/mfd/syscon.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+
+#include "clk-mtk.h"
+#include "clk-mux.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8183-clk.h>
+
+static DEFINE_SPINLOCK(mt8183_clk_lock);
+
+static const struct mtk_fixed_clk top_fixed_clks[] = {
+ FIXED_CLK(CLK_TOP_CLK26M, "f_f26m_ck", "clk26m", 26000000),
+ FIXED_CLK(CLK_TOP_ULPOSC, "osc", NULL, 250000),
+ FIXED_CLK(CLK_TOP_UNIVP_192M, "univpll_192m", "univpll", 192000000),
+};
+
+static const struct mtk_fixed_factor top_divs[] = {
+ FACTOR(CLK_TOP_CLK13M, "clk13m", "clk26m", 1,
+ 2),
+ FACTOR(CLK_TOP_F26M_CK_D2, "csw_f26m_ck_d2", "clk26m", 1,
+ 2),
+ FACTOR(CLK_TOP_SYSPLL_CK, "syspll_ck", "mainpll", 1,
+ 1),
+ FACTOR(CLK_TOP_SYSPLL_D2, "syspll_d2", "syspll_ck", 1,
+ 2),
+ FACTOR(CLK_TOP_SYSPLL_D2_D2, "syspll_d2_d2", "syspll_d2", 1,
+ 2),
+ FACTOR(CLK_TOP_SYSPLL_D2_D4, "syspll_d2_d4", "syspll_d2", 1,
+ 4),
+ FACTOR(CLK_TOP_SYSPLL_D2_D8, "syspll_d2_d8", "syspll_d2", 1,
+ 8),
+ FACTOR(CLK_TOP_SYSPLL_D2_D16, "syspll_d2_d16", "syspll_d2", 1,
+ 16),
+ FACTOR(CLK_TOP_SYSPLL_D3, "syspll_d3", "mainpll", 1,
+ 3),
+ FACTOR(CLK_TOP_SYSPLL_D3_D2, "syspll_d3_d2", "syspll_d3", 1,
+ 2),
+ FACTOR(CLK_TOP_SYSPLL_D3_D4, "syspll_d3_d4", "syspll_d3", 1,
+ 4),
+ FACTOR(CLK_TOP_SYSPLL_D3_D8, "syspll_d3_d8", "syspll_d3", 1,
+ 8),
+ FACTOR(CLK_TOP_SYSPLL_D5, "syspll_d5", "mainpll", 1,
+ 5),
+ FACTOR(CLK_TOP_SYSPLL_D5_D2, "syspll_d5_d2", "syspll_d5", 1,
+ 2),
+ FACTOR(CLK_TOP_SYSPLL_D5_D4, "syspll_d5_d4", "syspll_d5", 1,
+ 4),
+ FACTOR(CLK_TOP_SYSPLL_D7, "syspll_d7", "mainpll", 1,
+ 7),
+ FACTOR(CLK_TOP_SYSPLL_D7_D2, "syspll_d7_d2", "syspll_d7", 1,
+ 2),
+ FACTOR(CLK_TOP_SYSPLL_D7_D4, "syspll_d7_d4", "syspll_d7", 1,
+ 4),
+ FACTOR(CLK_TOP_UNIVPLL_CK, "univpll_ck", "univpll", 1,
+ 1),
+ FACTOR(CLK_TOP_UNIVPLL_D2, "univpll_d2", "univpll_ck", 1,
+ 2),
+ FACTOR(CLK_TOP_UNIVPLL_D2_D2, "univpll_d2_d2", "univpll_d2", 1,
+ 2),
+ FACTOR(CLK_TOP_UNIVPLL_D2_D4, "univpll_d2_d4", "univpll_d2", 1,
+ 4),
+ FACTOR(CLK_TOP_UNIVPLL_D2_D8, "univpll_d2_d8", "univpll_d2", 1,
+ 8),
+ FACTOR(CLK_TOP_UNIVPLL_D3, "univpll_d3", "univpll", 1,
+ 3),
+ FACTOR(CLK_TOP_UNIVPLL_D3_D2, "univpll_d3_d2", "univpll_d3", 1,
+ 2),
+ FACTOR(CLK_TOP_UNIVPLL_D3_D4, "univpll_d3_d4", "univpll_d3", 1,
+ 4),
+ FACTOR(CLK_TOP_UNIVPLL_D3_D8, "univpll_d3_d8", "univpll_d3", 1,
+ 8),
+ FACTOR(CLK_TOP_UNIVPLL_D5, "univpll_d5", "univpll", 1,
+ 5),
+ FACTOR(CLK_TOP_UNIVPLL_D5_D2, "univpll_d5_d2", "univpll_d5", 1,
+ 2),
+ FACTOR(CLK_TOP_UNIVPLL_D5_D4, "univpll_d5_d4", "univpll_d5", 1,
+ 4),
+ FACTOR(CLK_TOP_UNIVPLL_D5_D8, "univpll_d5_d8", "univpll_d5", 1,
+ 8),
+ FACTOR(CLK_TOP_UNIVPLL_D7, "univpll_d7", "univpll", 1,
+ 7),
+ FACTOR(CLK_TOP_UNIVP_192M_CK, "univ_192m_ck", "univpll_192m", 1,
+ 1),
+ FACTOR(CLK_TOP_UNIVP_192M_D2, "univ_192m_d2", "univ_192m_ck", 1,
+ 2),
+ FACTOR(CLK_TOP_UNIVP_192M_D4, "univ_192m_d4", "univ_192m_ck", 1,
+ 4),
+ FACTOR(CLK_TOP_UNIVP_192M_D8, "univ_192m_d8", "univ_192m_ck", 1,
+ 8),
+ FACTOR(CLK_TOP_UNIVP_192M_D16, "univ_192m_d16", "univ_192m_ck", 1,
+ 16),
+ FACTOR(CLK_TOP_UNIVP_192M_D32, "univ_192m_d32", "univ_192m_ck", 1,
+ 32),
+ FACTOR(CLK_TOP_APLL1_CK, "apll1_ck", "apll1", 1,
+ 1),
+ FACTOR(CLK_TOP_APLL1_D2, "apll1_d2", "apll1", 1,
+ 2),
+ FACTOR(CLK_TOP_APLL1_D4, "apll1_d4", "apll1", 1,
+ 4),
+ FACTOR(CLK_TOP_APLL1_D8, "apll1_d8", "apll1", 1,
+ 8),
+ FACTOR(CLK_TOP_APLL2_CK, "apll2_ck", "apll2", 1,
+ 1),
+ FACTOR(CLK_TOP_APLL2_D2, "apll2_d2", "apll2", 1,
+ 2),
+ FACTOR(CLK_TOP_APLL2_D4, "apll2_d4", "apll2", 1,
+ 4),
+ FACTOR(CLK_TOP_APLL2_D8, "apll2_d8", "apll2", 1,
+ 8),
+ FACTOR(CLK_TOP_TVDPLL_CK, "tvdpll_ck", "tvdpll", 1,
+ 1),
+ FACTOR(CLK_TOP_TVDPLL_D2, "tvdpll_d2", "tvdpll_ck", 1,
+ 2),
+ FACTOR(CLK_TOP_TVDPLL_D4, "tvdpll_d4", "tvdpll", 1,
+ 4),
+ FACTOR(CLK_TOP_TVDPLL_D8, "tvdpll_d8", "tvdpll", 1,
+ 8),
+ FACTOR(CLK_TOP_TVDPLL_D16, "tvdpll_d16", "tvdpll", 1,
+ 16),
+ FACTOR(CLK_TOP_MMPLL_CK, "mmpll_ck", "mmpll", 1,
+ 1),
+ FACTOR(CLK_TOP_MMPLL_D4, "mmpll_d4", "mmpll", 1,
+ 4),
+ FACTOR(CLK_TOP_MMPLL_D4_D2, "mmpll_d4_d2", "mmpll_d4", 1,
+ 2),
+ FACTOR(CLK_TOP_MMPLL_D4_D4, "mmpll_d4_d4", "mmpll_d4", 1,
+ 4),
+ FACTOR(CLK_TOP_MMPLL_D5, "mmpll_d5", "mmpll", 1,
+ 5),
+ FACTOR(CLK_TOP_MMPLL_D5_D2, "mmpll_d5_d2", "mmpll_d5", 1,
+ 2),
+ FACTOR(CLK_TOP_MMPLL_D5_D4, "mmpll_d5_d4", "mmpll_d5", 1,
+ 4),
+ FACTOR(CLK_TOP_MMPLL_D6, "mmpll_d6", "mmpll", 1,
+ 6),
+ FACTOR(CLK_TOP_MMPLL_D7, "mmpll_d7", "mmpll", 1,
+ 7),
+ FACTOR(CLK_TOP_MFGPLL_CK, "mfgpll_ck", "mfgpll", 1,
+ 1),
+ FACTOR(CLK_TOP_MSDCPLL_CK, "msdcpll_ck", "msdcpll", 1,
+ 1),
+ FACTOR(CLK_TOP_MSDCPLL_D2, "msdcpll_d2", "msdcpll", 1,
+ 2),
+ FACTOR(CLK_TOP_MSDCPLL_D4, "msdcpll_d4", "msdcpll", 1,
+ 4),
+ FACTOR(CLK_TOP_MSDCPLL_D8, "msdcpll_d8", "msdcpll", 1,
+ 8),
+ FACTOR(CLK_TOP_MSDCPLL_D16, "msdcpll_d16", "msdcpll", 1,
+ 16),
+ FACTOR(CLK_TOP_AD_OSC_CK, "ad_osc_ck", "osc", 1,
+ 1),
+ FACTOR(CLK_TOP_OSC_D2, "osc_d2", "osc", 1,
+ 2),
+ FACTOR(CLK_TOP_OSC_D4, "osc_d4", "osc", 1,
+ 4),
+ FACTOR(CLK_TOP_OSC_D8, "osc_d8", "osc", 1,
+ 8),
+ FACTOR(CLK_TOP_OSC_D16, "osc_d16", "osc", 1,
+ 16),
+ FACTOR(CLK_TOP_UNIVPLL, "univpll", "univ2pll", 1,
+ 2),
+ FACTOR(CLK_TOP_UNIVPLL_D3_D16, "univpll_d3_d16", "univpll_d3", 1,
+ 16),
+};
+
+static const char * const axi_parents[] = {
+ "clk26m",
+ "syspll_d2_d4",
+ "syspll_d7",
+ "osc_d4"
+};
+
+static const char * const mm_parents[] = {
+ "clk26m",
+ "mmpll_d7",
+ "syspll_d3",
+ "univpll_d2_d2",
+ "syspll_d2_d2",
+ "syspll_d3_d2"
+};
+
+static const char * const img_parents[] = {
+ "clk26m",
+ "mmpll_d6",
+ "univpll_d3",
+ "syspll_d3",
+ "univpll_d2_d2",
+ "syspll_d2_d2",
+ "univpll_d3_d2",
+ "syspll_d3_d2"
+};
+
+static const char * const cam_parents[] = {
+ "clk26m",
+ "syspll_d2",
+ "mmpll_d6",
+ "syspll_d3",
+ "mmpll_d7",
+ "univpll_d3",
+ "univpll_d2_d2",
+ "syspll_d2_d2",
+ "syspll_d3_d2",
+ "univpll_d3_d2"
+};
+
+static const char * const dsp_parents[] = {
+ "clk26m",
+ "mmpll_d6",
+ "mmpll_d7",
+ "univpll_d3",
+ "syspll_d3",
+ "univpll_d2_d2",
+ "syspll_d2_d2",
+ "univpll_d3_d2",
+ "syspll_d3_d2"
+};
+
+static const char * const dsp1_parents[] = {
+ "clk26m",
+ "mmpll_d6",
+ "mmpll_d7",
+ "univpll_d3",
+ "syspll_d3",
+ "univpll_d2_d2",
+ "syspll_d2_d2",
+ "univpll_d3_d2",
+ "syspll_d3_d2"
+};
+
+static const char * const dsp2_parents[] = {
+ "clk26m",
+ "mmpll_d6",
+ "mmpll_d7",
+ "univpll_d3",
+ "syspll_d3",
+ "univpll_d2_d2",
+ "syspll_d2_d2",
+ "univpll_d3_d2",
+ "syspll_d3_d2"
+};
+
+static const char * const ipu_if_parents[] = {
+ "clk26m",
+ "mmpll_d6",
+ "mmpll_d7",
+ "univpll_d3",
+ "syspll_d3",
+ "univpll_d2_d2",
+ "syspll_d2_d2",
+ "univpll_d3_d2",
+ "syspll_d3_d2"
+};
+
+static const char * const mfg_parents[] = {
+ "clk26m",
+ "mfgpll_ck",
+ "univpll_d3",
+ "syspll_d3"
+};
+
+static const char * const f52m_mfg_parents[] = {
+ "clk26m",
+ "univpll_d3_d2",
+ "univpll_d3_d4",
+ "univpll_d3_d8"
+};
+
+static const char * const camtg_parents[] = {
+ "clk26m",
+ "univ_192m_d8",
+ "univpll_d3_d8",
+ "univ_192m_d4",
+ "univpll_d3_d16",
+ "csw_f26m_ck_d2",
+ "univ_192m_d16",
+ "univ_192m_d32"
+};
+
+static const char * const camtg2_parents[] = {
+ "clk26m",
+ "univ_192m_d8",
+ "univpll_d3_d8",
+ "univ_192m_d4",
+ "univpll_d3_d16",
+ "csw_f26m_ck_d2",
+ "univ_192m_d16",
+ "univ_192m_d32"
+};
+
+static const char * const camtg3_parents[] = {
+ "clk26m",
+ "univ_192m_d8",
+ "univpll_d3_d8",
+ "univ_192m_d4",
+ "univpll_d3_d16",
+ "csw_f26m_ck_d2",
+ "univ_192m_d16",
+ "univ_192m_d32"
+};
+
+static const char * const camtg4_parents[] = {
+ "clk26m",
+ "univ_192m_d8",
+ "univpll_d3_d8",
+ "univ_192m_d4",
+ "univpll_d3_d16",
+ "csw_f26m_ck_d2",
+ "univ_192m_d16",
+ "univ_192m_d32"
+};
+
+static const char * const uart_parents[] = {
+ "clk26m",
+ "univpll_d3_d8"
+};
+
+static const char * const spi_parents[] = {
+ "clk26m",
+ "syspll_d5_d2",
+ "syspll_d3_d4",
+ "msdcpll_d4"
+};
+
+static const char * const msdc50_hclk_parents[] = {
+ "clk26m",
+ "syspll_d2_d2",
+ "syspll_d3_d2"
+};
+
+static const char * const msdc50_0_parents[] = {
+ "clk26m",
+ "msdcpll_ck",
+ "msdcpll_d2",
+ "univpll_d2_d4",
+ "syspll_d3_d2",
+ "univpll_d2_d2"
+};
+
+static const char * const msdc30_1_parents[] = {
+ "clk26m",
+ "univpll_d3_d2",
+ "syspll_d3_d2",
+ "syspll_d7",
+ "msdcpll_d2"
+};
+
+static const char * const msdc30_2_parents[] = {
+ "clk26m",
+ "univpll_d3_d2",
+ "syspll_d3_d2",
+ "syspll_d7",
+ "msdcpll_d2"
+};
+
+static const char * const audio_parents[] = {
+ "clk26m",
+ "syspll_d5_d4",
+ "syspll_d7_d4",
+ "syspll_d2_d16"
+};
+
+static const char * const aud_intbus_parents[] = {
+ "clk26m",
+ "syspll_d2_d4",
+ "syspll_d7_d2"
+};
+
+static const char * const pmicspi_parents[] = {
+ "clk26m",
+ "syspll_d2_d8",
+ "osc_d8"
+};
+
+static const char * const fpwrap_ulposc_parents[] = {
+ "clk26m",
+ "osc_d16",
+ "osc_d4",
+ "osc_d8"
+};
+
+static const char * const atb_parents[] = {
+ "clk26m",
+ "syspll_d2_d2",
+ "syspll_d5"
+};
+
+static const char * const sspm_parents[] = {
+ "clk26m",
+ "univpll_d2_d4",
+ "syspll_d2_d2",
+ "univpll_d2_d2",
+ "syspll_d3"
+};
+
+static const char * const dpi0_parents[] = {
+ "clk26m",
+ "tvdpll_d2",
+ "tvdpll_d4",
+ "tvdpll_d8",
+ "tvdpll_d16",
+ "univpll_d5_d2",
+ "univpll_d3_d4",
+ "syspll_d3_d4",
+ "univpll_d3_d8"
+};
+
+static const char * const scam_parents[] = {
+ "clk26m",
+ "syspll_d5_d2"
+};
+
+static const char * const disppwm_parents[] = {
+ "clk26m",
+ "univpll_d3_d4",
+ "osc_d2",
+ "osc_d4",
+ "osc_d16"
+};
+
+static const char * const usb_top_parents[] = {
+ "clk26m",
+ "univpll_d5_d4",
+ "univpll_d3_d4",
+ "univpll_d5_d2"
+};
+
+
+static const char * const ssusb_top_xhci_parents[] = {
+ "clk26m",
+ "univpll_d5_d4",
+ "univpll_d3_d4",
+ "univpll_d5_d2"
+};
+
+static const char * const spm_parents[] = {
+ "clk26m",
+ "syspll_d2_d8"
+};
+
+static const char * const i2c_parents[] = {
+ "clk26m",
+ "syspll_d2_d8",
+ "univpll_d5_d2"
+};
+
+static const char * const scp_parents[] = {
+ "clk26m",
+ "univpll_d2_d8",
+ "syspll_d5",
+ "syspll_d2_d2",
+ "univpll_d2_d2",
+ "syspll_d3",
+ "univpll_d3"
+};
+
+static const char * const seninf_parents[] = {
+ "clk26m",
+ "univpll_d2_d2",
+ "univpll_d3_d2",
+ "univpll_d2_d4"
+};
+
+static const char * const dxcc_parents[] = {
+ "clk26m",
+ "syspll_d2_d2",
+ "syspll_d2_d4",
+ "syspll_d2_d8"
+};
+
+static const char * const aud_engen1_parents[] = {
+ "clk26m",
+ "apll1_d2",
+ "apll1_d4",
+ "apll1_d8"
+};
+
+static const char * const aud_engen2_parents[] = {
+ "clk26m",
+ "apll2_d2",
+ "apll2_d4",
+ "apll2_d8"
+};
+
+static const char * const faes_ufsfde_parents[] = {
+ "clk26m",
+ "syspll_d2",
+ "syspll_d2_d2",
+ "syspll_d3",
+ "syspll_d2_d4",
+ "univpll_d3"
+};
+
+static const char * const fufs_parents[] = {
+ "clk26m",
+ "syspll_d2_d4",
+ "syspll_d2_d8",
+ "syspll_d2_d16"
+};
+
+static const char * const aud_1_parents[] = {
+ "clk26m",
+ "apll1_ck"
+};
+
+static const char * const aud_2_parents[] = {
+ "clk26m",
+ "apll2_ck"
+};
+
+/*
+ * CRITICAL CLOCK:
+ * axi_sel is the main bus clock of whole SOC.
+ * spm_sel is the clock of the always-on co-processor.
+ */
+static const struct mtk_mux top_muxes[] = {
+ /* CLK_CFG_0 */
+ MUX_GATE_CLR_SET_UPD_FLAGS(CLK_TOP_MUX_AXI, "axi_sel",
+ axi_parents, 0x40,
+ 0x44, 0x48, 0, 2, 7, 0x004, 0, CLK_IS_CRITICAL),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_MM, "mm_sel",
+ mm_parents, 0x40,
+ 0x44, 0x48, 8, 3, 15, 0x004, 1),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_IMG, "img_sel",
+ img_parents, 0x40,
+ 0x44, 0x48, 16, 3, 23, 0x004, 2),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_CAM, "cam_sel",
+ cam_parents, 0x40,
+ 0x44, 0x48, 24, 4, 31, 0x004, 3),
+ /* CLK_CFG_1 */
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_DSP, "dsp_sel",
+ dsp_parents, 0x50,
+ 0x54, 0x58, 0, 4, 7, 0x004, 4),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_DSP1, "dsp1_sel",
+ dsp1_parents, 0x50,
+ 0x54, 0x58, 8, 4, 15, 0x004, 5),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_DSP2, "dsp2_sel",
+ dsp2_parents, 0x50,
+ 0x54, 0x58, 16, 4, 23, 0x004, 6),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_IPU_IF, "ipu_if_sel",
+ ipu_if_parents, 0x50,
+ 0x54, 0x58, 24, 4, 31, 0x004, 7),
+ /* CLK_CFG_2 */
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_MFG, "mfg_sel",
+ mfg_parents, 0x60,
+ 0x64, 0x68, 0, 2, 7, 0x004, 8),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_F52M_MFG, "f52m_mfg_sel",
+ f52m_mfg_parents, 0x60,
+ 0x64, 0x68, 8, 2, 15, 0x004, 9),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_CAMTG, "camtg_sel",
+ camtg_parents, 0x60,
+ 0x64, 0x68, 16, 3, 23, 0x004, 10),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_CAMTG2, "camtg2_sel",
+ camtg2_parents, 0x60,
+ 0x64, 0x68, 24, 3, 31, 0x004, 11),
+ /* CLK_CFG_3 */
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_CAMTG3, "camtg3_sel",
+ camtg3_parents, 0x70,
+ 0x74, 0x78, 0, 3, 7, 0x004, 12),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_CAMTG4, "camtg4_sel",
+ camtg4_parents, 0x70,
+ 0x74, 0x78, 8, 3, 15, 0x004, 13),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_UART, "uart_sel",
+ uart_parents, 0x70,
+ 0x74, 0x78, 16, 1, 23, 0x004, 14),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_SPI, "spi_sel",
+ spi_parents, 0x70,
+ 0x74, 0x78, 24, 2, 31, 0x004, 15),
+ /* CLK_CFG_4 */
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_MSDC50_0_HCLK, "msdc50_hclk_sel",
+ msdc50_hclk_parents, 0x80,
+ 0x84, 0x88, 0, 2, 7, 0x004, 16),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_MSDC50_0, "msdc50_0_sel",
+ msdc50_0_parents, 0x80,
+ 0x84, 0x88, 8, 3, 15, 0x004, 17),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_MSDC30_1, "msdc30_1_sel",
+ msdc30_1_parents, 0x80,
+ 0x84, 0x88, 16, 3, 23, 0x004, 18),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_MSDC30_2, "msdc30_2_sel",
+ msdc30_2_parents, 0x80,
+ 0x84, 0x88, 24, 3, 31, 0x004, 19),
+ /* CLK_CFG_5 */
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_AUDIO, "audio_sel",
+ audio_parents, 0x90,
+ 0x94, 0x98, 0, 2, 7, 0x004, 20),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_AUD_INTBUS, "aud_intbus_sel",
+ aud_intbus_parents, 0x90,
+ 0x94, 0x98, 8, 2, 15, 0x004, 21),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_PMICSPI, "pmicspi_sel",
+ pmicspi_parents, 0x90,
+ 0x94, 0x98, 16, 2, 23, 0x004, 22),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_FPWRAP_ULPOSC, "fpwrap_ulposc_sel",
+ fpwrap_ulposc_parents, 0x90,
+ 0x94, 0x98, 24, 2, 31, 0x004, 23),
+ /* CLK_CFG_6 */
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_ATB, "atb_sel",
+ atb_parents, 0xa0,
+ 0xa4, 0xa8, 0, 2, 7, 0x004, 24),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_SSPM, "sspm_sel",
+ sspm_parents, 0xa0,
+ 0xa4, 0xa8, 8, 3, 15, 0x004, 25),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_DPI0, "dpi0_sel",
+ dpi0_parents, 0xa0,
+ 0xa4, 0xa8, 16, 4, 23, 0x004, 26),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_SCAM, "scam_sel",
+ scam_parents, 0xa0,
+ 0xa4, 0xa8, 24, 1, 31, 0x004, 27),
+ /* CLK_CFG_7 */
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_DISP_PWM, "disppwm_sel",
+ disppwm_parents, 0xb0,
+ 0xb4, 0xb8, 0, 3, 7, 0x004, 28),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_USB_TOP, "usb_top_sel",
+ usb_top_parents, 0xb0,
+ 0xb4, 0xb8, 8, 2, 15, 0x004, 29),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_SSUSB_TOP_XHCI, "ssusb_top_xhci_sel",
+ ssusb_top_xhci_parents, 0xb0,
+ 0xb4, 0xb8, 16, 2, 23, 0x004, 30),
+ MUX_GATE_CLR_SET_UPD_FLAGS(CLK_TOP_MUX_SPM, "spm_sel",
+ spm_parents, 0xb0,
+ 0xb4, 0xb8, 24, 1, 31, 0x008, 0, CLK_IS_CRITICAL),
+ /* CLK_CFG_8 */
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_I2C, "i2c_sel",
+ i2c_parents, 0xc0,
+ 0xc4, 0xc8, 0, 2, 7, 0x008, 1),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_SCP, "scp_sel",
+ scp_parents, 0xc0,
+ 0xc4, 0xc8, 8, 3, 15, 0x008, 2),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_SENINF, "seninf_sel",
+ seninf_parents, 0xc0,
+ 0xc4, 0xc8, 16, 2, 23, 0x008, 3),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_DXCC, "dxcc_sel",
+ dxcc_parents, 0xc0,
+ 0xc4, 0xc8, 24, 2, 31, 0x008, 4),
+ /* CLK_CFG_9 */
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_AUD_ENG1, "aud_eng1_sel",
+ aud_engen1_parents, 0xd0,
+ 0xd4, 0xd8, 0, 2, 7, 0x008, 5),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_AUD_ENG2, "aud_eng2_sel",
+ aud_engen2_parents, 0xd0,
+ 0xd4, 0xd8, 8, 2, 15, 0x008, 6),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_FAES_UFSFDE, "faes_ufsfde_sel",
+ faes_ufsfde_parents, 0xd0,
+ 0xd4, 0xd8, 16, 3, 23, 0x008, 7),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_FUFS, "fufs_sel",
+ fufs_parents, 0xd0,
+ 0xd4, 0xd8, 24, 2, 31, 0x008, 8),
+ /* CLK_CFG_10 */
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_AUD_1, "aud_1_sel",
+ aud_1_parents, 0xe0,
+ 0xe4, 0xe8, 0, 1, 7, 0x008, 9),
+ MUX_GATE_CLR_SET_UPD(CLK_TOP_MUX_AUD_2, "aud_2_sel",
+ aud_2_parents, 0xe0,
+ 0xe4, 0xe8, 8, 1, 15, 0x008, 10),
+};
+
+static const char * const apll_i2s0_parents[] = {
+ "aud_1_sel",
+ "aud_2_sel"
+};
+
+static const char * const apll_i2s1_parents[] = {
+ "aud_1_sel",
+ "aud_2_sel"
+};
+
+static const char * const apll_i2s2_parents[] = {
+ "aud_1_sel",
+ "aud_2_sel"
+};
+
+static const char * const apll_i2s3_parents[] = {
+ "aud_1_sel",
+ "aud_2_sel"
+};
+
+static const char * const apll_i2s4_parents[] = {
+ "aud_1_sel",
+ "aud_2_sel"
+};
+
+static const char * const apll_i2s5_parents[] = {
+ "aud_1_sel",
+ "aud_2_sel"
+};
+
+static struct mtk_composite top_aud_muxes[] = {
+ MUX(CLK_TOP_MUX_APLL_I2S0, "apll_i2s0_sel", apll_i2s0_parents,
+ 0x320, 8, 1),
+ MUX(CLK_TOP_MUX_APLL_I2S1, "apll_i2s1_sel", apll_i2s1_parents,
+ 0x320, 9, 1),
+ MUX(CLK_TOP_MUX_APLL_I2S2, "apll_i2s2_sel", apll_i2s2_parents,
+ 0x320, 10, 1),
+ MUX(CLK_TOP_MUX_APLL_I2S3, "apll_i2s3_sel", apll_i2s3_parents,
+ 0x320, 11, 1),
+ MUX(CLK_TOP_MUX_APLL_I2S4, "apll_i2s4_sel", apll_i2s4_parents,
+ 0x320, 12, 1),
+ MUX(CLK_TOP_MUX_APLL_I2S5, "apll_i2s5_sel", apll_i2s5_parents,
+ 0x328, 20, 1),
+};
+
+static const char * const mcu_mp0_parents[] = {
+ "clk26m",
+ "armpll_ll",
+ "armpll_div_pll1",
+ "armpll_div_pll2"
+};
+
+static const char * const mcu_mp2_parents[] = {
+ "clk26m",
+ "armpll_l",
+ "armpll_div_pll1",
+ "armpll_div_pll2"
+};
+
+static const char * const mcu_bus_parents[] = {
+ "clk26m",
+ "ccipll",
+ "armpll_div_pll1",
+ "armpll_div_pll2"
+};
+
+static struct mtk_composite mcu_muxes[] = {
+ /* mp0_pll_divider_cfg */
+ MUX(CLK_MCU_MP0_SEL, "mcu_mp0_sel", mcu_mp0_parents, 0x7A0, 9, 2),
+ /* mp2_pll_divider_cfg */
+ MUX(CLK_MCU_MP2_SEL, "mcu_mp2_sel", mcu_mp2_parents, 0x7A8, 9, 2),
+ /* bus_pll_divider_cfg */
+ MUX(CLK_MCU_BUS_SEL, "mcu_bus_sel", mcu_bus_parents, 0x7C0, 9, 2),
+};
+
+static struct mtk_composite top_aud_divs[] = {
+ DIV_GATE(CLK_TOP_APLL12_DIV0, "apll12_div0", "apll_i2s0_sel",
+ 0x320, 2, 0x324, 8, 0),
+ DIV_GATE(CLK_TOP_APLL12_DIV1, "apll12_div1", "apll_i2s1_sel",
+ 0x320, 3, 0x324, 8, 8),
+ DIV_GATE(CLK_TOP_APLL12_DIV2, "apll12_div2", "apll_i2s2_sel",
+ 0x320, 4, 0x324, 8, 16),
+ DIV_GATE(CLK_TOP_APLL12_DIV3, "apll12_div3", "apll_i2s3_sel",
+ 0x320, 5, 0x324, 8, 24),
+ DIV_GATE(CLK_TOP_APLL12_DIV4, "apll12_div4", "apll_i2s4_sel",
+ 0x320, 6, 0x328, 8, 0),
+ DIV_GATE(CLK_TOP_APLL12_DIVB, "apll12_divb", "apll12_div4",
+ 0x320, 7, 0x328, 8, 8),
+};
+
+static const struct mtk_gate_regs top_cg_regs = {
+ .set_ofs = 0x104,
+ .clr_ofs = 0x104,
+ .sta_ofs = 0x104,
+};
+
+#define GATE_TOP(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &top_cg_regs, _shift, \
+ &mtk_clk_gate_ops_no_setclr_inv)
+
+static const struct mtk_gate top_clks[] = {
+ /* TOP */
+ GATE_TOP(CLK_TOP_ARMPLL_DIV_PLL1, "armpll_div_pll1", "mainpll", 4),
+ GATE_TOP(CLK_TOP_ARMPLL_DIV_PLL2, "armpll_div_pll2", "univpll", 5),
+};
+
+static const struct mtk_gate_regs infra0_cg_regs = {
+ .set_ofs = 0x80,
+ .clr_ofs = 0x84,
+ .sta_ofs = 0x90,
+};
+
+static const struct mtk_gate_regs infra1_cg_regs = {
+ .set_ofs = 0x88,
+ .clr_ofs = 0x8c,
+ .sta_ofs = 0x94,
+};
+
+static const struct mtk_gate_regs infra2_cg_regs = {
+ .set_ofs = 0xa4,
+ .clr_ofs = 0xa8,
+ .sta_ofs = 0xac,
+};
+
+static const struct mtk_gate_regs infra3_cg_regs = {
+ .set_ofs = 0xc0,
+ .clr_ofs = 0xc4,
+ .sta_ofs = 0xc8,
+};
+
+#define GATE_INFRA0(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &infra0_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+#define GATE_INFRA1(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &infra1_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+#define GATE_INFRA2(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &infra2_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+#define GATE_INFRA3(_id, _name, _parent, _shift) \
+ GATE_MTK(_id, _name, _parent, &infra3_cg_regs, _shift, \
+ &mtk_clk_gate_ops_setclr)
+
+static const struct mtk_gate infra_clks[] = {
+ /* INFRA0 */
+ GATE_INFRA0(CLK_INFRA_PMIC_TMR, "infra_pmic_tmr",
+ "axi_sel", 0),
+ GATE_INFRA0(CLK_INFRA_PMIC_AP, "infra_pmic_ap",
+ "axi_sel", 1),
+ GATE_INFRA0(CLK_INFRA_PMIC_MD, "infra_pmic_md",
+ "axi_sel", 2),
+ GATE_INFRA0(CLK_INFRA_PMIC_CONN, "infra_pmic_conn",
+ "axi_sel", 3),
+ GATE_INFRA0(CLK_INFRA_SCPSYS, "infra_scp",
+ "scp_sel", 4),
+ GATE_INFRA0(CLK_INFRA_SEJ, "infra_sej",
+ "f_f26m_ck", 5),
+ GATE_INFRA0(CLK_INFRA_APXGPT, "infra_apxgpt",
+ "axi_sel", 6),
+ GATE_INFRA0(CLK_INFRA_ICUSB, "infra_icusb",
+ "axi_sel", 8),
+ GATE_INFRA0(CLK_INFRA_GCE, "infra_gce",
+ "axi_sel", 9),
+ GATE_INFRA0(CLK_INFRA_THERM, "infra_therm",
+ "axi_sel", 10),
+ GATE_INFRA0(CLK_INFRA_I2C0, "infra_i2c0",
+ "i2c_sel", 11),
+ GATE_INFRA0(CLK_INFRA_I2C1, "infra_i2c1",
+ "i2c_sel", 12),
+ GATE_INFRA0(CLK_INFRA_I2C2, "infra_i2c2",
+ "i2c_sel", 13),
+ GATE_INFRA0(CLK_INFRA_I2C3, "infra_i2c3",
+ "i2c_sel", 14),
+ GATE_INFRA0(CLK_INFRA_PWM_HCLK, "infra_pwm_hclk",
+ "axi_sel", 15),
+ GATE_INFRA0(CLK_INFRA_PWM1, "infra_pwm1",
+ "i2c_sel", 16),
+ GATE_INFRA0(CLK_INFRA_PWM2, "infra_pwm2",
+ "i2c_sel", 17),
+ GATE_INFRA0(CLK_INFRA_PWM3, "infra_pwm3",
+ "i2c_sel", 18),
+ GATE_INFRA0(CLK_INFRA_PWM4, "infra_pwm4",
+ "i2c_sel", 19),
+ GATE_INFRA0(CLK_INFRA_PWM, "infra_pwm",
+ "i2c_sel", 21),
+ GATE_INFRA0(CLK_INFRA_UART0, "infra_uart0",
+ "uart_sel", 22),
+ GATE_INFRA0(CLK_INFRA_UART1, "infra_uart1",
+ "uart_sel", 23),
+ GATE_INFRA0(CLK_INFRA_UART2, "infra_uart2",
+ "uart_sel", 24),
+ GATE_INFRA0(CLK_INFRA_UART3, "infra_uart3",
+ "uart_sel", 25),
+ GATE_INFRA0(CLK_INFRA_GCE_26M, "infra_gce_26m",
+ "axi_sel", 27),
+ GATE_INFRA0(CLK_INFRA_CQ_DMA_FPC, "infra_cqdma_fpc",
+ "axi_sel", 28),
+ GATE_INFRA0(CLK_INFRA_BTIF, "infra_btif",
+ "axi_sel", 31),
+ /* INFRA1 */
+ GATE_INFRA1(CLK_INFRA_SPI0, "infra_spi0",
+ "spi_sel", 1),
+ GATE_INFRA1(CLK_INFRA_MSDC0, "infra_msdc0",
+ "msdc50_hclk_sel", 2),
+ GATE_INFRA1(CLK_INFRA_MSDC1, "infra_msdc1",
+ "axi_sel", 4),
+ GATE_INFRA1(CLK_INFRA_MSDC2, "infra_msdc2",
+ "axi_sel", 5),
+ GATE_INFRA1(CLK_INFRA_MSDC0_SCK, "infra_msdc0_sck",
+ "msdc50_0_sel", 6),
+ GATE_INFRA1(CLK_INFRA_DVFSRC, "infra_dvfsrc",
+ "f_f26m_ck", 7),
+ GATE_INFRA1(CLK_INFRA_GCPU, "infra_gcpu",
+ "axi_sel", 8),
+ GATE_INFRA1(CLK_INFRA_TRNG, "infra_trng",
+ "axi_sel", 9),
+ GATE_INFRA1(CLK_INFRA_AUXADC, "infra_auxadc",
+ "f_f26m_ck", 10),
+ GATE_INFRA1(CLK_INFRA_CPUM, "infra_cpum",
+ "axi_sel", 11),
+ GATE_INFRA1(CLK_INFRA_CCIF1_AP, "infra_ccif1_ap",
+ "axi_sel", 12),
+ GATE_INFRA1(CLK_INFRA_CCIF1_MD, "infra_ccif1_md",
+ "axi_sel", 13),
+ GATE_INFRA1(CLK_INFRA_AUXADC_MD, "infra_auxadc_md",
+ "f_f26m_ck", 14),
+ GATE_INFRA1(CLK_INFRA_MSDC1_SCK, "infra_msdc1_sck",
+ "msdc30_1_sel", 16),
+ GATE_INFRA1(CLK_INFRA_MSDC2_SCK, "infra_msdc2_sck",
+ "msdc30_2_sel", 17),
+ GATE_INFRA1(CLK_INFRA_AP_DMA, "infra_apdma",
+ "axi_sel", 18),
+ GATE_INFRA1(CLK_INFRA_XIU, "infra_xiu",
+ "axi_sel", 19),
+ GATE_INFRA1(CLK_INFRA_DEVICE_APC, "infra_device_apc",
+ "axi_sel", 20),
+ GATE_INFRA1(CLK_INFRA_CCIF_AP, "infra_ccif_ap",
+ "axi_sel", 23),
+ GATE_INFRA1(CLK_INFRA_DEBUGSYS, "infra_debugsys",
+ "axi_sel", 24),
+ GATE_INFRA1(CLK_INFRA_AUDIO, "infra_audio",
+ "axi_sel", 25),
+ GATE_INFRA1(CLK_INFRA_CCIF_MD, "infra_ccif_md",
+ "axi_sel", 26),
+ GATE_INFRA1(CLK_INFRA_DXCC_SEC_CORE, "infra_dxcc_sec_core",
+ "dxcc_sel", 27),
+ GATE_INFRA1(CLK_INFRA_DXCC_AO, "infra_dxcc_ao",
+ "dxcc_sel", 28),
+ GATE_INFRA1(CLK_INFRA_DEVMPU_BCLK, "infra_devmpu_bclk",
+ "axi_sel", 30),
+ GATE_INFRA1(CLK_INFRA_DRAMC_F26M, "infra_dramc_f26m",
+ "f_f26m_ck", 31),
+ /* INFRA2 */
+ GATE_INFRA2(CLK_INFRA_IRTX, "infra_irtx",
+ "f_f26m_ck", 0),
+ GATE_INFRA2(CLK_INFRA_USB, "infra_usb",
+ "usb_top_sel", 1),
+ GATE_INFRA2(CLK_INFRA_DISP_PWM, "infra_disppwm",
+ "axi_sel", 2),
+ GATE_INFRA2(CLK_INFRA_CLDMA_BCLK, "infra_cldma_bclk",
+ "axi_sel", 3),
+ GATE_INFRA2(CLK_INFRA_AUDIO_26M_BCLK, "infra_audio_26m_bclk",
+ "f_f26m_ck", 4),
+ GATE_INFRA2(CLK_INFRA_SPI1, "infra_spi1",
+ "spi_sel", 6),
+ GATE_INFRA2(CLK_INFRA_I2C4, "infra_i2c4",
+ "i2c_sel", 7),
+ GATE_INFRA2(CLK_INFRA_MODEM_TEMP_SHARE, "infra_md_tmp_share",
+ "f_f26m_ck", 8),
+ GATE_INFRA2(CLK_INFRA_SPI2, "infra_spi2",
+ "spi_sel", 9),
+ GATE_INFRA2(CLK_INFRA_SPI3, "infra_spi3",
+ "spi_sel", 10),
+ GATE_INFRA2(CLK_INFRA_UNIPRO_SCK, "infra_unipro_sck",
+ "ssusb_top_xhci_sel", 11),
+ GATE_INFRA2(CLK_INFRA_UNIPRO_TICK, "infra_unipro_tick",
+ "fufs_sel", 12),
+ GATE_INFRA2(CLK_INFRA_UFS_MP_SAP_BCLK, "infra_ufs_mp_sap_bck",
+ "fufs_sel", 13),
+ GATE_INFRA2(CLK_INFRA_MD32_BCLK, "infra_md32_bclk",
+ "axi_sel", 14),
+ GATE_INFRA2(CLK_INFRA_SSPM, "infra_sspm",
+ "sspm_sel", 15),
+ GATE_INFRA2(CLK_INFRA_UNIPRO_MBIST, "infra_unipro_mbist",
+ "axi_sel", 16),
+ GATE_INFRA2(CLK_INFRA_SSPM_BUS_HCLK, "infra_sspm_bus_hclk",
+ "axi_sel", 17),
+ GATE_INFRA2(CLK_INFRA_I2C5, "infra_i2c5",
+ "i2c_sel", 18),
+ GATE_INFRA2(CLK_INFRA_I2C5_ARBITER, "infra_i2c5_arbiter",
+ "i2c_sel", 19),
+ GATE_INFRA2(CLK_INFRA_I2C5_IMM, "infra_i2c5_imm",
+ "i2c_sel", 20),
+ GATE_INFRA2(CLK_INFRA_I2C1_ARBITER, "infra_i2c1_arbiter",
+ "i2c_sel", 21),
+ GATE_INFRA2(CLK_INFRA_I2C1_IMM, "infra_i2c1_imm",
+ "i2c_sel", 22),
+ GATE_INFRA2(CLK_INFRA_I2C2_ARBITER, "infra_i2c2_arbiter",
+ "i2c_sel", 23),
+ GATE_INFRA2(CLK_INFRA_I2C2_IMM, "infra_i2c2_imm",
+ "i2c_sel", 24),
+ GATE_INFRA2(CLK_INFRA_SPI4, "infra_spi4",
+ "spi_sel", 25),
+ GATE_INFRA2(CLK_INFRA_SPI5, "infra_spi5",
+ "spi_sel", 26),
+ GATE_INFRA2(CLK_INFRA_CQ_DMA, "infra_cqdma",
+ "axi_sel", 27),
+ GATE_INFRA2(CLK_INFRA_UFS, "infra_ufs",
+ "fufs_sel", 28),
+ GATE_INFRA2(CLK_INFRA_AES_UFSFDE, "infra_aes_ufsfde",
+ "faes_ufsfde_sel", 29),
+ GATE_INFRA2(CLK_INFRA_UFS_TICK, "infra_ufs_tick",
+ "fufs_sel", 30),
+ /* INFRA3 */
+ GATE_INFRA3(CLK_INFRA_MSDC0_SELF, "infra_msdc0_self",
+ "msdc50_0_sel", 0),
+ GATE_INFRA3(CLK_INFRA_MSDC1_SELF, "infra_msdc1_self",
+ "msdc50_0_sel", 1),
+ GATE_INFRA3(CLK_INFRA_MSDC2_SELF, "infra_msdc2_self",
+ "msdc50_0_sel", 2),
+ GATE_INFRA3(CLK_INFRA_SSPM_26M_SELF, "infra_sspm_26m_self",
+ "f_f26m_ck", 3),
+ GATE_INFRA3(CLK_INFRA_SSPM_32K_SELF, "infra_sspm_32k_self",
+ "f_f26m_ck", 4),
+ GATE_INFRA3(CLK_INFRA_UFS_AXI, "infra_ufs_axi",
+ "axi_sel", 5),
+ GATE_INFRA3(CLK_INFRA_I2C6, "infra_i2c6",
+ "i2c_sel", 6),
+ GATE_INFRA3(CLK_INFRA_AP_MSDC0, "infra_ap_msdc0",
+ "msdc50_hclk_sel", 7),
+ GATE_INFRA3(CLK_INFRA_MD_MSDC0, "infra_md_msdc0",
+ "msdc50_hclk_sel", 8),
+ GATE_INFRA3(CLK_INFRA_CCIF2_AP, "infra_ccif2_ap",
+ "axi_sel", 16),
+ GATE_INFRA3(CLK_INFRA_CCIF2_MD, "infra_ccif2_md",
+ "axi_sel", 17),
+ GATE_INFRA3(CLK_INFRA_CCIF3_AP, "infra_ccif3_ap",
+ "axi_sel", 18),
+ GATE_INFRA3(CLK_INFRA_CCIF3_MD, "infra_ccif3_md",
+ "axi_sel", 19),
+ GATE_INFRA3(CLK_INFRA_SEJ_F13M, "infra_sej_f13m",
+ "f_f26m_ck", 20),
+ GATE_INFRA3(CLK_INFRA_AES_BCLK, "infra_aes_bclk",
+ "axi_sel", 21),
+ GATE_INFRA3(CLK_INFRA_I2C7, "infra_i2c7",
+ "i2c_sel", 22),
+ GATE_INFRA3(CLK_INFRA_I2C8, "infra_i2c8",
+ "i2c_sel", 23),
+ GATE_INFRA3(CLK_INFRA_FBIST2FPC, "infra_fbist2fpc",
+ "msdc50_0_sel", 24),
+};
+
+static const struct mtk_gate_regs apmixed_cg_regs = {
+ .set_ofs = 0x20,
+ .clr_ofs = 0x20,
+ .sta_ofs = 0x20,
+};
+
+#define GATE_APMIXED_FLAGS(_id, _name, _parent, _shift, _flags) \
+ GATE_MTK_FLAGS(_id, _name, _parent, &apmixed_cg_regs, \
+ _shift, &mtk_clk_gate_ops_no_setclr_inv, _flags)
+
+#define GATE_APMIXED(_id, _name, _parent, _shift) \
+ GATE_APMIXED_FLAGS(_id, _name, _parent, _shift, 0)
+
+/*
+ * CRITICAL CLOCK:
+ * apmixed_appll26m is the toppest clock gate of all PLLs.
+ */
+static const struct mtk_gate apmixed_clks[] = {
+ /* AUDIO0 */
+ GATE_APMIXED(CLK_APMIXED_SSUSB_26M, "apmixed_ssusb26m",
+ "f_f26m_ck", 4),
+ GATE_APMIXED_FLAGS(CLK_APMIXED_APPLL_26M, "apmixed_appll26m",
+ "f_f26m_ck", 5, CLK_IS_CRITICAL),
+ GATE_APMIXED(CLK_APMIXED_MIPIC0_26M, "apmixed_mipic026m",
+ "f_f26m_ck", 6),
+ GATE_APMIXED(CLK_APMIXED_MDPLLGP_26M, "apmixed_mdpll26m",
+ "f_f26m_ck", 7),
+ GATE_APMIXED(CLK_APMIXED_MMSYS_26M, "apmixed_mmsys26m",
+ "f_f26m_ck", 8),
+ GATE_APMIXED(CLK_APMIXED_UFS_26M, "apmixed_ufs26m",
+ "f_f26m_ck", 9),
+ GATE_APMIXED(CLK_APMIXED_MIPIC1_26M, "apmixed_mipic126m",
+ "f_f26m_ck", 11),
+ GATE_APMIXED(CLK_APMIXED_MEMPLL_26M, "apmixed_mempll26m",
+ "f_f26m_ck", 13),
+ GATE_APMIXED(CLK_APMIXED_CLKSQ_LVPLL_26M, "apmixed_lvpll26m",
+ "f_f26m_ck", 14),
+ GATE_APMIXED(CLK_APMIXED_MIPID0_26M, "apmixed_mipid026m",
+ "f_f26m_ck", 16),
+ GATE_APMIXED(CLK_APMIXED_MIPID1_26M, "apmixed_mipid126m",
+ "f_f26m_ck", 17),
+};
+
+#define MT8183_PLL_FMAX (3800UL * MHZ)
+#define MT8183_PLL_FMIN (1500UL * MHZ)
+
+#define PLL_B(_id, _name, _reg, _pwr_reg, _en_mask, _flags, \
+ _rst_bar_mask, _pcwbits, _pcwibits, _pd_reg, \
+ _pd_shift, _tuner_reg, _tuner_en_reg, \
+ _tuner_en_bit, _pcw_reg, _pcw_shift, \
+ _pcw_chg_reg, _div_table) { \
+ .id = _id, \
+ .name = _name, \
+ .reg = _reg, \
+ .pwr_reg = _pwr_reg, \
+ .en_mask = _en_mask, \
+ .flags = _flags, \
+ .rst_bar_mask = _rst_bar_mask, \
+ .fmax = MT8183_PLL_FMAX, \
+ .fmin = MT8183_PLL_FMIN, \
+ .pcwbits = _pcwbits, \
+ .pcwibits = _pcwibits, \
+ .pd_reg = _pd_reg, \
+ .pd_shift = _pd_shift, \
+ .tuner_reg = _tuner_reg, \
+ .tuner_en_reg = _tuner_en_reg, \
+ .tuner_en_bit = _tuner_en_bit, \
+ .pcw_reg = _pcw_reg, \
+ .pcw_shift = _pcw_shift, \
+ .pcw_chg_reg = _pcw_chg_reg, \
+ .div_table = _div_table, \
+ }
+
+#define PLL(_id, _name, _reg, _pwr_reg, _en_mask, _flags, \
+ _rst_bar_mask, _pcwbits, _pcwibits, _pd_reg, \
+ _pd_shift, _tuner_reg, _tuner_en_reg, \
+ _tuner_en_bit, _pcw_reg, _pcw_shift, \
+ _pcw_chg_reg) \
+ PLL_B(_id, _name, _reg, _pwr_reg, _en_mask, _flags, \
+ _rst_bar_mask, _pcwbits, _pcwibits, _pd_reg, \
+ _pd_shift, _tuner_reg, _tuner_en_reg, \
+ _tuner_en_bit, _pcw_reg, _pcw_shift, \
+ _pcw_chg_reg, NULL)
+
+static const struct mtk_pll_div_table armpll_div_table[] = {
+ { .div = 0, .freq = MT8183_PLL_FMAX },
+ { .div = 1, .freq = 1500 * MHZ },
+ { .div = 2, .freq = 750 * MHZ },
+ { .div = 3, .freq = 375 * MHZ },
+ { .div = 4, .freq = 187500000 },
+ { } /* sentinel */
+};
+
+static const struct mtk_pll_div_table mfgpll_div_table[] = {
+ { .div = 0, .freq = MT8183_PLL_FMAX },
+ { .div = 1, .freq = 1600 * MHZ },
+ { .div = 2, .freq = 800 * MHZ },
+ { .div = 3, .freq = 400 * MHZ },
+ { .div = 4, .freq = 200 * MHZ },
+ { } /* sentinel */
+};
+
+static const struct mtk_pll_data plls[] = {
+ PLL_B(CLK_APMIXED_ARMPLL_LL, "armpll_ll", 0x0200, 0x020C, 0x00000001,
+ HAVE_RST_BAR | PLL_AO, BIT(24), 22, 8, 0x0204, 24, 0x0, 0x0, 0,
+ 0x0204, 0, 0, armpll_div_table),
+ PLL_B(CLK_APMIXED_ARMPLL_L, "armpll_l", 0x0210, 0x021C, 0x00000001,
+ HAVE_RST_BAR | PLL_AO, BIT(24), 22, 8, 0x0214, 24, 0x0, 0x0, 0,
+ 0x0214, 0, 0, armpll_div_table),
+ PLL(CLK_APMIXED_CCIPLL, "ccipll", 0x0290, 0x029C, 0x00000001,
+ HAVE_RST_BAR | PLL_AO, BIT(24), 22, 8, 0x0294, 24, 0x0, 0x0, 0,
+ 0x0294, 0, 0),
+ PLL(CLK_APMIXED_MAINPLL, "mainpll", 0x0220, 0x022C, 0x00000001,
+ HAVE_RST_BAR, BIT(24), 22, 8, 0x0224, 24, 0x0, 0x0, 0,
+ 0x0224, 0, 0),
+ PLL(CLK_APMIXED_UNIV2PLL, "univ2pll", 0x0230, 0x023C, 0x00000001,
+ HAVE_RST_BAR, BIT(24), 22, 8, 0x0234, 24, 0x0, 0x0, 0,
+ 0x0234, 0, 0),
+ PLL_B(CLK_APMIXED_MFGPLL, "mfgpll", 0x0240, 0x024C, 0x00000001,
+ 0, 0, 22, 8, 0x0244, 24, 0x0, 0x0, 0, 0x0244, 0, 0,
+ mfgpll_div_table),
+ PLL(CLK_APMIXED_MSDCPLL, "msdcpll", 0x0250, 0x025C, 0x00000001,
+ 0, 0, 22, 8, 0x0254, 24, 0x0, 0x0, 0, 0x0254, 0, 0),
+ PLL(CLK_APMIXED_TVDPLL, "tvdpll", 0x0260, 0x026C, 0x00000001,
+ 0, 0, 22, 8, 0x0264, 24, 0x0, 0x0, 0, 0x0264, 0, 0),
+ PLL(CLK_APMIXED_MMPLL, "mmpll", 0x0270, 0x027C, 0x00000001,
+ HAVE_RST_BAR, BIT(23), 22, 8, 0x0274, 24, 0x0, 0x0, 0,
+ 0x0274, 0, 0),
+ PLL(CLK_APMIXED_APLL1, "apll1", 0x02A0, 0x02B0, 0x00000001,
+ 0, 0, 32, 8, 0x02A0, 1, 0x02A8, 0x0014, 0, 0x02A4, 0, 0x02A0),
+ PLL(CLK_APMIXED_APLL2, "apll2", 0x02b4, 0x02c4, 0x00000001,
+ 0, 0, 32, 8, 0x02B4, 1, 0x02BC, 0x0014, 1, 0x02B8, 0, 0x02B4),
+};
+
+static int clk_mt8183_apmixed_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_APMIXED_NR_CLK);
+
+ mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data);
+
+ mtk_clk_register_gates(node, apmixed_clks, ARRAY_SIZE(apmixed_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static int clk_mt8183_top_probe(struct platform_device *pdev)
+{
+ struct resource *res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ void __iomem *base;
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ clk_data = mtk_alloc_clk_data(CLK_TOP_NR_CLK);
+
+ mtk_clk_register_fixed_clks(top_fixed_clks, ARRAY_SIZE(top_fixed_clks),
+ clk_data);
+
+ mtk_clk_register_factors(top_divs, ARRAY_SIZE(top_divs), clk_data);
+
+ mtk_clk_register_muxes(top_muxes, ARRAY_SIZE(top_muxes),
+ node, &mt8183_clk_lock, clk_data);
+
+ mtk_clk_register_composites(top_aud_muxes, ARRAY_SIZE(top_aud_muxes),
+ base, &mt8183_clk_lock, clk_data);
+
+ mtk_clk_register_composites(top_aud_divs, ARRAY_SIZE(top_aud_divs),
+ base, &mt8183_clk_lock, clk_data);
+
+ mtk_clk_register_gates(node, top_clks, ARRAY_SIZE(top_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static int clk_mt8183_infra_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+
+ clk_data = mtk_alloc_clk_data(CLK_INFRA_NR_CLK);
+
+ mtk_clk_register_gates(node, infra_clks, ARRAY_SIZE(infra_clks),
+ clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static int clk_mt8183_mcu_probe(struct platform_device *pdev)
+{
+ struct clk_onecell_data *clk_data;
+ struct device_node *node = pdev->dev.of_node;
+ void __iomem *base;
+ struct resource *res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+
+ base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ clk_data = mtk_alloc_clk_data(CLK_MCU_NR_CLK);
+
+ mtk_clk_register_composites(mcu_muxes, ARRAY_SIZE(mcu_muxes), base,
+ &mt8183_clk_lock, clk_data);
+
+ return of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+}
+
+static const struct of_device_id of_match_clk_mt8183[] = {
+ {
+ .compatible = "mediatek,mt8183-apmixedsys",
+ .data = clk_mt8183_apmixed_probe,
+ }, {
+ .compatible = "mediatek,mt8183-topckgen",
+ .data = clk_mt8183_top_probe,
+ }, {
+ .compatible = "mediatek,mt8183-infracfg",
+ .data = clk_mt8183_infra_probe,
+ }, {
+ .compatible = "mediatek,mt8183-mcucfg",
+ .data = clk_mt8183_mcu_probe,
+ }, {
+ /* sentinel */
+ }
+};
+
+static int clk_mt8183_probe(struct platform_device *pdev)
+{
+ int (*clk_probe)(struct platform_device *pdev);
+ int r;
+
+ clk_probe = of_device_get_match_data(&pdev->dev);
+ if (!clk_probe)
+ return -EINVAL;
+
+ r = clk_probe(pdev);
+ if (r)
+ dev_err(&pdev->dev,
+ "could not register clock provider: %s: %d\n",
+ pdev->name, r);
+
+ return r;
+}
+
+static struct platform_driver clk_mt8183_drv = {
+ .probe = clk_mt8183_probe,
+ .driver = {
+ .name = "clk-mt8183",
+ .of_match_table = of_match_clk_mt8183,
+ },
+};
+
+static int __init clk_mt8183_init(void)
+{
+ return platform_driver_register(&clk_mt8183_drv);
+}
+
+arch_initcall(clk_mt8183_init);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2019 MediaTek Inc.
+ * Author: James Liao <jamesjj.liao@mediatek.com>
+ * Fabien Parent <fparent@baylibre.com>
+ */
+
+#include <linux/delay.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/slab.h>
+#include <linux/mfd/syscon.h>
+
+#include "clk-mtk.h"
+#include "clk-gate.h"
+
+#include <dt-bindings/clock/mt8516-clk.h>
+
+static DEFINE_SPINLOCK(mt8516_clk_lock);
+
+static const struct mtk_fixed_clk fixed_clks[] __initconst = {
+ FIXED_CLK(CLK_TOP_CLK_NULL, "clk_null", NULL, 0),
+ FIXED_CLK(CLK_TOP_I2S_INFRA_BCK, "i2s_infra_bck", "clk_null", 26000000),
+ FIXED_CLK(CLK_TOP_MEMPLL, "mempll", "clk26m", 800000000),
+};
+
+static const struct mtk_fixed_factor top_divs[] __initconst = {
+ FACTOR(CLK_TOP_DMPLL, "dmpll_ck", "mempll", 1, 1),
+ FACTOR(CLK_TOP_MAINPLL_D2, "mainpll_d2", "mainpll", 1, 2),
+ FACTOR(CLK_TOP_MAINPLL_D4, "mainpll_d4", "mainpll", 1, 4),
+ FACTOR(CLK_TOP_MAINPLL_D8, "mainpll_d8", "mainpll", 1, 8),
+ FACTOR(CLK_TOP_MAINPLL_D16, "mainpll_d16", "mainpll", 1, 16),
+ FACTOR(CLK_TOP_MAINPLL_D11, "mainpll_d11", "mainpll", 1, 11),
+ FACTOR(CLK_TOP_MAINPLL_D22, "mainpll_d22", "mainpll", 1, 22),
+ FACTOR(CLK_TOP_MAINPLL_D3, "mainpll_d3", "mainpll", 1, 3),
+ FACTOR(CLK_TOP_MAINPLL_D6, "mainpll_d6", "mainpll", 1, 6),
+ FACTOR(CLK_TOP_MAINPLL_D12, "mainpll_d12", "mainpll", 1, 12),
+ FACTOR(CLK_TOP_MAINPLL_D5, "mainpll_d5", "mainpll", 1, 5),
+ FACTOR(CLK_TOP_MAINPLL_D10, "mainpll_d10", "mainpll", 1, 10),
+ FACTOR(CLK_TOP_MAINPLL_D20, "mainpll_d20", "mainpll", 1, 20),
+ FACTOR(CLK_TOP_MAINPLL_D40, "mainpll_d40", "mainpll", 1, 40),
+ FACTOR(CLK_TOP_MAINPLL_D7, "mainpll_d7", "mainpll", 1, 7),
+ FACTOR(CLK_TOP_MAINPLL_D14, "mainpll_d14", "mainpll", 1, 14),
+ FACTOR(CLK_TOP_UNIVPLL_D2, "univpll_d2", "univpll", 1, 2),
+ FACTOR(CLK_TOP_UNIVPLL_D4, "univpll_d4", "univpll", 1, 4),
+ FACTOR(CLK_TOP_UNIVPLL_D8, "univpll_d8", "univpll", 1, 8),
+ FACTOR(CLK_TOP_UNIVPLL_D16, "univpll_d16", "univpll", 1, 16),
+ FACTOR(CLK_TOP_UNIVPLL_D3, "univpll_d3", "univpll", 1, 3),
+ FACTOR(CLK_TOP_UNIVPLL_D6, "univpll_d6", "univpll", 1, 6),
+ FACTOR(CLK_TOP_UNIVPLL_D12, "univpll_d12", "univpll", 1, 12),
+ FACTOR(CLK_TOP_UNIVPLL_D24, "univpll_d24", "univpll", 1, 24),
+ FACTOR(CLK_TOP_UNIVPLL_D5, "univpll_d5", "univpll", 1, 5),
+ FACTOR(CLK_TOP_UNIVPLL_D20, "univpll_d20", "univpll", 1, 20),
+ FACTOR(CLK_TOP_MMPLL380M, "mmpll380m", "mmpll", 1, 1),
+ FACTOR(CLK_TOP_MMPLL_D2, "mmpll_d2", "mmpll", 1, 2),
+ FACTOR(CLK_TOP_MMPLL_200M, "mmpll_200m", "mmpll", 1, 3),
+ FACTOR(CLK_TOP_USB_PHY48M, "usb_phy48m_ck", "univpll", 1, 26),
+ FACTOR(CLK_TOP_APLL1, "apll1_ck", "apll1", 1, 1),
+ FACTOR(CLK_TOP_APLL1_D2, "apll1_d2", "apll1_ck", 1, 2),
+ FACTOR(CLK_TOP_APLL1_D4, "apll1_d4", "rg_apll1_d2_en", 1, 2),
+ FACTOR(CLK_TOP_APLL1_D8, "apll1_d8", "rg_apll1_d4_en", 1, 2),
+ FACTOR(CLK_TOP_APLL2, "apll2_ck", "apll2", 1, 1),
+ FACTOR(CLK_TOP_APLL2_D2, "apll2_d2", "apll2_ck", 1, 2),
+ FACTOR(CLK_TOP_APLL2_D4, "apll2_d4", "rg_apll2_d2_en", 1, 2),
+ FACTOR(CLK_TOP_APLL2_D8, "apll2_d8", "rg_apll2_d4_en", 1, 2),
+ FACTOR(CLK_TOP_CLK26M, "clk26m_ck", "clk26m", 1, 1),
+ FACTOR(CLK_TOP_CLK26M_D2, "clk26m_d2", "clk26m", 1, 2),
+ FACTOR(CLK_TOP_AHB_INFRA_D2, "ahb_infra_d2", "ahb_infra_sel", 1, 2),
+ FACTOR(CLK_TOP_NFI1X, "nfi1x_ck", "nfi2x_pad_sel", 1, 2),
+ FACTOR(CLK_TOP_ETH_D2, "eth_d2_ck", "eth_sel", 1, 2),
+};
+
+static const char * const uart0_parents[] __initconst = {
+ "clk26m_ck",
+ "univpll_d24"
+};
+
+static const char * const ahb_infra_parents[] __initconst = {
+ "clk_null",
+ "clk26m_ck",
+ "mainpll_d11",
+ "clk_null",
+ "mainpll_d12",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "mainpll_d10"
+};
+
+static const char * const msdc0_parents[] __initconst = {
+ "clk26m_ck",
+ "univpll_d6",
+ "mainpll_d8",
+ "univpll_d8",
+ "mainpll_d16",
+ "mmpll_200m",
+ "mainpll_d12",
+ "mmpll_d2"
+};
+
+static const char * const uart1_parents[] __initconst = {
+ "clk26m_ck",
+ "univpll_d24"
+};
+
+static const char * const msdc1_parents[] __initconst = {
+ "clk26m_ck",
+ "univpll_d6",
+ "mainpll_d8",
+ "univpll_d8",
+ "mainpll_d16",
+ "mmpll_200m",
+ "mainpll_d12",
+ "mmpll_d2"
+};
+
+static const char * const pmicspi_parents[] __initconst = {
+ "univpll_d20",
+ "usb_phy48m_ck",
+ "univpll_d16",
+ "clk26m_ck"
+};
+
+static const char * const qaxi_aud26m_parents[] __initconst = {
+ "clk26m_ck",
+ "ahb_infra_sel"
+};
+
+static const char * const aud_intbus_parents[] __initconst = {
+ "clk_null",
+ "clk26m_ck",
+ "mainpll_d22",
+ "clk_null",
+ "mainpll_d11"
+};
+
+static const char * const nfi2x_pad_parents[] __initconst = {
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk26m_ck",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "mainpll_d12",
+ "mainpll_d8",
+ "clk_null",
+ "mainpll_d6",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "mainpll_d4",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "clk_null",
+ "mainpll_d10",
+ "mainpll_d7",
+ "clk_null",
+ "mainpll_d5"
+};
+
+static const char * const nfi1x_pad_parents[] __initconst = {
+ "ahb_infra_sel",
+ "nfi1x_ck"
+};
+
+static const char * const ddrphycfg_parents[] __initconst = {
+ "clk26m_ck",
+ "mainpll_d16"
+};
+
+static const char * const usb_78m_parents[] __initconst = {
+ "clk_null",
+ "clk26m_ck",
+ "univpll_d16",
+ "clk_null",
+ "mainpll_d20"
+};
+
+static const char * const spinor_parents[] __initconst = {
+ "clk26m_d2",
+ "clk26m_ck",
+ "mainpll_d40",
+ "univpll_d24",
+ "univpll_d20",
+ "mainpll_d20",
+ "mainpll_d16",
+ "univpll_d12"
+};
+
+static const char * const msdc2_parents[] __initconst = {
+ "clk26m_ck",
+ "univpll_d6",
+ "mainpll_d8",
+ "univpll_d8",
+ "mainpll_d16",
+ "mmpll_200m",
+ "mainpll_d12",
+ "mmpll_d2"
+};
+
+static const char * const eth_parents[] __initconst = {
+ "clk26m_ck",
+ "mainpll_d40",
+ "univpll_d24",
+ "univpll_d20",
+ "mainpll_d20"
+};
+
+static const char * const aud1_parents[] __initconst = {
+ "clk26m_ck",
+ "apll1_ck"
+};
+
+static const char * const aud2_parents[] __initconst = {
+ "clk26m_ck",
+ "apll2_ck"
+};
+
+static const char * const aud_engen1_parents[] __initconst = {
+ "clk26m_ck",
+ "rg_apll1_d2_en",
+ "rg_apll1_d4_en",
+ "rg_apll1_d8_en"
+};
+
+static const char * const aud_engen2_parents[] __initconst = {
+ "clk26m_ck",
+ "rg_apll2_d2_en",
+ "rg_apll2_d4_en",
+ "rg_apll2_d8_en"
+};
+
+static const char * const i2c_parents[] __initconst = {
+ "clk26m_ck",
+ "univpll_d20",
+ "univpll_d16",
+ "univpll_d12"
+};
+
+static const char * const aud_i2s0_m_parents[] __initconst = {
+ "rg_aud1",
+ "rg_aud2"
+};
+
+static const char * const pwm_parents[] __initconst = {
+ "clk26m_ck",
+ "univpll_d12"
+};
+
+static const char * const spi_parents[] __initconst = {
+ "clk26m_ck",
+ "univpll_d12",
+ "univpll_d8",
+ "univpll_d6"
+};
+
+static const char * const aud_spdifin_parents[] __initconst = {
+ "clk26m_ck",
+ "univpll_d2"
+};
+
+static const char * const uart2_parents[] __initconst = {
+ "clk26m_ck",
+ "univpll_d24"
+};
+
+static const char * const bsi_parents[] __initconst = {
+ "clk26m_ck",
+ "mainpll_d10",
+ "mainpll_d12",
+ "mainpll_d20"
+};
+
+static const char * const dbg_atclk_parents[] __initconst = {
+ "clk_null",
+ "clk26m_ck",
+ "mainpll_d5",
+ "clk_null",
+ "univpll_d5"
+};
+
+static const char * const csw_nfiecc_parents[] __initconst = {
+ "clk_null",
+ "mainpll_d7",
+ "mainpll_d6",
+ "clk_null",
+ "mainpll_d5"
+};
+
+static const char * const nfiecc_parents[] __initconst = {
+ "clk_null",
+ "nfi2x_pad_sel",
+ "mainpll_d4",
+ "clk_null",
+ "csw_nfiecc_sel"
+};
+
+static struct mtk_composite top_muxes[] __initdata = {
+ /* CLK_MUX_SEL0 */
+ MUX(CLK_TOP_UART0_SEL, "uart0_sel", uart0_parents,
+ 0x000, 0, 1),
+ MUX(CLK_TOP_AHB_INFRA_SEL, "ahb_infra_sel", ahb_infra_parents,
+ 0x000, 4, 4),
+ MUX(CLK_TOP_MSDC0_SEL, "msdc0_sel", msdc0_parents,
+ 0x000, 11, 3),
+ MUX(CLK_TOP_UART1_SEL, "uart1_sel", uart1_parents,
+ 0x000, 19, 1),
+ MUX(CLK_TOP_MSDC1_SEL, "msdc1_sel", msdc1_parents,
+ 0x000, 20, 3),
+ MUX(CLK_TOP_PMICSPI_SEL, "pmicspi_sel", pmicspi_parents,
+ 0x000, 24, 2),
+ MUX(CLK_TOP_QAXI_AUD26M_SEL, "qaxi_aud26m_sel", qaxi_aud26m_parents,
+ 0x000, 26, 1),
+ MUX(CLK_TOP_AUD_INTBUS_SEL, "aud_intbus_sel", aud_intbus_parents,
+ 0x000, 27, 3),
+ /* CLK_MUX_SEL1 */
+ MUX(CLK_TOP_NFI2X_PAD_SEL, "nfi2x_pad_sel", nfi2x_pad_parents,
+ 0x004, 0, 7),
+ MUX(CLK_TOP_NFI1X_PAD_SEL, "nfi1x_pad_sel", nfi1x_pad_parents,
+ 0x004, 7, 1),
+ MUX(CLK_TOP_USB_78M_SEL, "usb_78m_sel", usb_78m_parents,
+ 0x004, 20, 3),
+ /* CLK_MUX_SEL8 */
+ MUX(CLK_TOP_SPINOR_SEL, "spinor_sel", spinor_parents,
+ 0x040, 0, 3),
+ MUX(CLK_TOP_MSDC2_SEL, "msdc2_sel", msdc2_parents,
+ 0x040, 3, 3),
+ MUX(CLK_TOP_ETH_SEL, "eth_sel", eth_parents,
+ 0x040, 6, 3),
+ MUX(CLK_TOP_AUD1_SEL, "aud1_sel", aud1_parents,
+ 0x040, 22, 1),
+ MUX(CLK_TOP_AUD2_SEL, "aud2_sel", aud2_parents,
+ 0x040, 23, 1),
+ MUX(CLK_TOP_AUD_ENGEN1_SEL, "aud_engen1_sel", aud_engen1_parents,
+ 0x040, 24, 2),
+ MUX(CLK_TOP_AUD_ENGEN2_SEL, "aud_engen2_sel", aud_engen2_parents,
+ 0x040, 26, 2),
+ MUX(CLK_TOP_I2C_SEL, "i2c_sel", i2c_parents,
+ 0x040, 28, 2),
+ /* CLK_SEL_9 */
+ MUX(CLK_TOP_AUD_I2S0_M_SEL, "aud_i2s0_m_sel", aud_i2s0_m_parents,
+ 0x044, 12, 1),
+ MUX(CLK_TOP_AUD_I2S1_M_SEL, "aud_i2s1_m_sel", aud_i2s0_m_parents,
+ 0x044, 13, 1),
+ MUX(CLK_TOP_AUD_I2S2_M_SEL, "aud_i2s2_m_sel", aud_i2s0_m_parents,
+ 0x044, 14, 1),
+ MUX(CLK_TOP_AUD_I2S3_M_SEL, "aud_i2s3_m_sel", aud_i2s0_m_parents,
+ 0x044, 15, 1),
+ MUX(CLK_TOP_AUD_I2S4_M_SEL, "aud_i2s4_m_sel", aud_i2s0_m_parents,
+ 0x044, 16, 1),
+ MUX(CLK_TOP_AUD_I2S5_M_SEL, "aud_i2s5_m_sel", aud_i2s0_m_parents,
+ 0x044, 17, 1),
+ MUX(CLK_TOP_AUD_SPDIF_B_SEL, "aud_spdif_b_sel", aud_i2s0_m_parents,
+ 0x044, 18, 1),
+ /* CLK_MUX_SEL13 */
+ MUX(CLK_TOP_PWM_SEL, "pwm_sel", pwm_parents,
+ 0x07c, 0, 1),
+ MUX(CLK_TOP_SPI_SEL, "spi_sel", spi_parents,
+ 0x07c, 1, 2),
+ MUX(CLK_TOP_AUD_SPDIFIN_SEL, "aud_spdifin_sel", aud_spdifin_parents,
+ 0x07c, 3, 1),
+ MUX(CLK_TOP_UART2_SEL, "uart2_sel", uart2_parents,
+ 0x07c, 4, 1),
+ MUX(CLK_TOP_BSI_SEL, "bsi_sel", bsi_parents,
+ 0x07c, 5, 2),
+ MUX(CLK_TOP_DBG_ATCLK_SEL, "dbg_atclk_sel", dbg_atclk_parents,
+ 0x07c, 7, 3),
+ MUX(CLK_TOP_CSW_NFIECC_SEL, "csw_nfiecc_sel", csw_nfiecc_parents,
+ 0x07c, 10, 3),
+ MUX(CLK_TOP_NFIECC_SEL, "nfiecc_sel", nfiecc_parents,
+ 0x07c, 13, 3),
+};
+
+static const char * const ifr_mux1_parents[] __initconst = {
+ "clk26m_ck",
+ "armpll",
+ "univpll",
+ "mainpll_d2"
+};
+
+static const char * const ifr_eth_25m_parents[] __initconst = {
+ "eth_d2_ck",
+ "rg_eth"
+};
+
+static const char * const ifr_i2c0_parents[] __initconst = {
+ "ahb_infra_d2",
+ "rg_i2c"
+};
+
+static const struct mtk_composite ifr_muxes[] __initconst = {
+ MUX(CLK_IFR_MUX1_SEL, "ifr_mux1_sel", ifr_mux1_parents, 0x000,
+ 2, 2),
+ MUX(CLK_IFR_ETH_25M_SEL, "ifr_eth_25m_sel", ifr_eth_25m_parents, 0x080,
+ 0, 1),
+ MUX(CLK_IFR_I2C0_SEL, "ifr_i2c0_sel", ifr_i2c0_parents, 0x080,
+ 1, 1),
+ MUX(CLK_IFR_I2C1_SEL, "ifr_i2c1_sel", ifr_i2c0_parents, 0x080,
+ 2, 1),
+ MUX(CLK_IFR_I2C2_SEL, "ifr_i2c2_sel", ifr_i2c0_parents, 0x080,
+ 3, 1),
+};
+
+#define DIV_ADJ(_id, _name, _parent, _reg, _shift, _width) { \
+ .id = _id, \
+ .name = _name, \
+ .parent_name = _parent, \
+ .div_reg = _reg, \
+ .div_shift = _shift, \
+ .div_width = _width, \
+}
+
+static const struct mtk_clk_divider top_adj_divs[] = {
+ DIV_ADJ(CLK_TOP_APLL12_CK_DIV0, "apll12_ck_div0", "aud_i2s0_m_sel",
+ 0x0048, 0, 8),
+ DIV_ADJ(CLK_TOP_APLL12_CK_DIV1, "apll12_ck_div1", "aud_i2s1_m_sel",
+ 0x0048, 8, 8),
+ DIV_ADJ(CLK_TOP_APLL12_CK_DIV2, "apll12_ck_div2", "aud_i2s2_m_sel",
+ 0x0048, 16, 8),
+ DIV_ADJ(CLK_TOP_APLL12_CK_DIV3, "apll12_ck_div3", "aud_i2s3_m_sel",
+ 0x0048, 24, 8),
+ DIV_ADJ(CLK_TOP_APLL12_CK_DIV4, "apll12_ck_div4", "aud_i2s4_m_sel",
+ 0x004c, 0, 8),
+ DIV_ADJ(CLK_TOP_APLL12_CK_DIV4B, "apll12_ck_div4b", "apll12_div4",
+ 0x004c, 8, 8),
+ DIV_ADJ(CLK_TOP_APLL12_CK_DIV5, "apll12_ck_div5", "aud_i2s5_m_sel",
+ 0x004c, 16, 8),
+ DIV_ADJ(CLK_TOP_APLL12_CK_DIV5B, "apll12_ck_div5b", "apll12_div5",
+ 0x004c, 24, 8),
+ DIV_ADJ(CLK_TOP_APLL12_CK_DIV6, "apll12_ck_div6", "aud_spdif_b_sel",
+ 0x0078, 0, 8),
+};
+
+static const struct mtk_gate_regs top1_cg_regs = {
+ .set_ofs = 0x54,
+ .clr_ofs = 0x84,
+ .sta_ofs = 0x24,
+};
+
+static const struct mtk_gate_regs top2_cg_regs = {
+ .set_ofs = 0x6c,
+ .clr_ofs = 0x9c,
+ .sta_ofs = 0x3c,
+};
+
+static const struct mtk_gate_regs top3_cg_regs = {
+ .set_ofs = 0xa0,
+ .clr_ofs = 0xb0,
+ .sta_ofs = 0x70,
+};
+
+static const struct mtk_gate_regs top4_cg_regs = {
+ .set_ofs = 0xa4,
+ .clr_ofs = 0xb4,
+ .sta_ofs = 0x74,
+};
+
+static const struct mtk_gate_regs top5_cg_regs = {
+ .set_ofs = 0x44,
+ .clr_ofs = 0x44,
+ .sta_ofs = 0x44,
+};
+
+#define GATE_TOP1(_id, _name, _parent, _shift) { \
+ .id = _id, \
+ .name = _name, \
+ .parent_name = _parent, \
+ .regs = &top1_cg_regs, \
+ .shift = _shift, \
+ .ops = &mtk_clk_gate_ops_setclr, \
+ }
+
+#define GATE_TOP2(_id, _name, _parent, _shift) { \
+ .id = _id, \
+ .name = _name, \
+ .parent_name = _parent, \
+ .regs = &top2_cg_regs, \
+ .shift = _shift, \
+ .ops = &mtk_clk_gate_ops_setclr, \
+ }
+
+#define GATE_TOP2_I(_id, _name, _parent, _shift) { \
+ .id = _id, \
+ .name = _name, \
+ .parent_name = _parent, \
+ .regs = &top2_cg_regs, \
+ .shift = _shift, \
+ .ops = &mtk_clk_gate_ops_setclr_inv, \
+ }
+
+#define GATE_TOP3(_id, _name, _parent, _shift) { \
+ .id = _id, \
+ .name = _name, \
+ .parent_name = _parent, \
+ .regs = &top3_cg_regs, \
+ .shift = _shift, \
+ .ops = &mtk_clk_gate_ops_setclr, \
+ }
+
+#define GATE_TOP4_I(_id, _name, _parent, _shift) { \
+ .id = _id, \
+ .name = _name, \
+ .parent_name = _parent, \
+ .regs = &top4_cg_regs, \
+ .shift = _shift, \
+ .ops = &mtk_clk_gate_ops_setclr_inv, \
+ }
+
+#define GATE_TOP5(_id, _name, _parent, _shift) { \
+ .id = _id, \
+ .name = _name, \
+ .parent_name = _parent, \
+ .regs = &top5_cg_regs, \
+ .shift = _shift, \
+ .ops = &mtk_clk_gate_ops_no_setclr, \
+ }
+
+static const struct mtk_gate top_clks[] __initconst = {
+ /* TOP1 */
+ GATE_TOP1(CLK_TOP_THEM, "them", "ahb_infra_sel", 1),
+ GATE_TOP1(CLK_TOP_APDMA, "apdma", "ahb_infra_sel", 2),
+ GATE_TOP1(CLK_TOP_I2C0, "i2c0", "ifr_i2c0_sel", 3),
+ GATE_TOP1(CLK_TOP_I2C1, "i2c1", "ifr_i2c1_sel", 4),
+ GATE_TOP1(CLK_TOP_AUXADC1, "auxadc1", "ahb_infra_sel", 5),
+ GATE_TOP1(CLK_TOP_NFI, "nfi", "nfi1x_pad_sel", 6),
+ GATE_TOP1(CLK_TOP_NFIECC, "nfiecc", "rg_nfiecc", 7),
+ GATE_TOP1(CLK_TOP_DEBUGSYS, "debugsys", "rg_dbg_atclk", 8),
+ GATE_TOP1(CLK_TOP_PWM, "pwm", "ahb_infra_sel", 9),
+ GATE_TOP1(CLK_TOP_UART0, "uart0", "uart0_sel", 10),
+ GATE_TOP1(CLK_TOP_UART1, "uart1", "uart1_sel", 11),
+ GATE_TOP1(CLK_TOP_BTIF, "btif", "ahb_infra_sel", 12),
+ GATE_TOP1(CLK_TOP_USB, "usb", "usb_78m", 13),
+ GATE_TOP1(CLK_TOP_FLASHIF_26M, "flashif_26m", "clk26m_ck", 14),
+ GATE_TOP1(CLK_TOP_AUXADC2, "auxadc2", "ahb_infra_sel", 15),
+ GATE_TOP1(CLK_TOP_I2C2, "i2c2", "ifr_i2c2_sel", 16),
+ GATE_TOP1(CLK_TOP_MSDC0, "msdc0", "msdc0_sel", 17),
+ GATE_TOP1(CLK_TOP_MSDC1, "msdc1", "msdc1_sel", 18),
+ GATE_TOP1(CLK_TOP_NFI2X, "nfi2x", "nfi2x_pad_sel", 19),
+ GATE_TOP1(CLK_TOP_PMICWRAP_AP, "pwrap_ap", "clk26m_ck", 20),
+ GATE_TOP1(CLK_TOP_SEJ, "sej", "ahb_infra_sel", 21),
+ GATE_TOP1(CLK_TOP_MEMSLP_DLYER, "memslp_dlyer", "clk26m_ck", 22),
+ GATE_TOP1(CLK_TOP_SPI, "spi", "spi_sel", 23),
+ GATE_TOP1(CLK_TOP_APXGPT, "apxgpt", "clk26m_ck", 24),
+ GATE_TOP1(CLK_TOP_AUDIO, "audio", "clk26m_ck", 25),
+ GATE_TOP1(CLK_TOP_PMICWRAP_MD, "pwrap_md", "clk26m_ck", 27),
+ GATE_TOP1(CLK_TOP_PMICWRAP_CONN, "pwrap_conn", "clk26m_ck", 28),
+ GATE_TOP1(CLK_TOP_PMICWRAP_26M, "pwrap_26m", "clk26m_ck", 29),
+ GATE_TOP1(CLK_TOP_AUX_ADC, "aux_adc", "clk26m_ck", 30),
+ GATE_TOP1(CLK_TOP_AUX_TP, "aux_tp", "clk26m_ck", 31),
+ /* TOP2 */
+ GATE_TOP2(CLK_TOP_MSDC2, "msdc2", "ahb_infra_sel", 0),
+ GATE_TOP2(CLK_TOP_RBIST, "rbist", "univpll_d12", 1),
+ GATE_TOP2(CLK_TOP_NFI_BUS, "nfi_bus", "ahb_infra_sel", 2),
+ GATE_TOP2(CLK_TOP_GCE, "gce", "ahb_infra_sel", 4),
+ GATE_TOP2(CLK_TOP_TRNG, "trng", "ahb_infra_sel", 5),
+ GATE_TOP2(CLK_TOP_SEJ_13M, "sej_13m", "clk26m_ck", 6),
+ GATE_TOP2(CLK_TOP_AES, "aes", "ahb_infra_sel", 7),
+ GATE_TOP2(CLK_TOP_PWM_B, "pwm_b", "rg_pwm_infra", 8),
+ GATE_TOP2(CLK_TOP_PWM1_FB, "pwm1_fb", "rg_pwm_infra", 9),
+ GATE_TOP2(CLK_TOP_PWM2_FB, "pwm2_fb", "rg_pwm_infra", 10),
+ GATE_TOP2(CLK_TOP_PWM3_FB, "pwm3_fb", "rg_pwm_infra", 11),
+ GATE_TOP2(CLK_TOP_PWM4_FB, "pwm4_fb", "rg_pwm_infra", 12),
+ GATE_TOP2(CLK_TOP_PWM5_FB, "pwm5_fb", "rg_pwm_infra", 13),
+ GATE_TOP2(CLK_TOP_USB_1P, "usb_1p", "usb_78m", 14),
+ GATE_TOP2(CLK_TOP_FLASHIF_FREERUN, "flashif_freerun", "ahb_infra_sel",
+ 15),
+ GATE_TOP2(CLK_TOP_66M_ETH, "eth_66m", "ahb_infra_d2", 19),
+ GATE_TOP2(CLK_TOP_133M_ETH, "eth_133m", "ahb_infra_sel", 20),
+ GATE_TOP2(CLK_TOP_FETH_25M, "feth_25m", "ifr_eth_25m_sel", 21),
+ GATE_TOP2(CLK_TOP_FETH_50M, "feth_50m", "rg_eth", 22),
+ GATE_TOP2(CLK_TOP_FLASHIF_AXI, "flashif_axi", "ahb_infra_sel", 23),
+ GATE_TOP2(CLK_TOP_USBIF, "usbif", "ahb_infra_sel", 24),
+ GATE_TOP2(CLK_TOP_UART2, "uart2", "rg_uart2", 25),
+ GATE_TOP2(CLK_TOP_BSI, "bsi", "ahb_infra_sel", 26),
+ GATE_TOP2_I(CLK_TOP_MSDC0_INFRA, "msdc0_infra", "msdc0", 28),
+ GATE_TOP2_I(CLK_TOP_MSDC1_INFRA, "msdc1_infra", "msdc1", 29),
+ GATE_TOP2_I(CLK_TOP_MSDC2_INFRA, "msdc2_infra", "rg_msdc2", 30),
+ GATE_TOP2(CLK_TOP_USB_78M, "usb_78m", "usb_78m_sel", 31),
+ /* TOP3 */
+ GATE_TOP3(CLK_TOP_RG_SPINOR, "rg_spinor", "spinor_sel", 0),
+ GATE_TOP3(CLK_TOP_RG_MSDC2, "rg_msdc2", "msdc2_sel", 1),
+ GATE_TOP3(CLK_TOP_RG_ETH, "rg_eth", "eth_sel", 2),
+ GATE_TOP3(CLK_TOP_RG_AUD1, "rg_aud1", "aud1_sel", 8),
+ GATE_TOP3(CLK_TOP_RG_AUD2, "rg_aud2", "aud2_sel", 9),
+ GATE_TOP3(CLK_TOP_RG_AUD_ENGEN1, "rg_aud_engen1", "aud_engen1_sel", 10),
+ GATE_TOP3(CLK_TOP_RG_AUD_ENGEN2, "rg_aud_engen2", "aud_engen2_sel", 11),
+ GATE_TOP3(CLK_TOP_RG_I2C, "rg_i2c", "i2c_sel", 12),
+ GATE_TOP3(CLK_TOP_RG_PWM_INFRA, "rg_pwm_infra", "pwm_sel", 13),
+ GATE_TOP3(CLK_TOP_RG_AUD_SPDIF_IN, "rg_aud_spdif_in", "aud_spdifin_sel",
+ 14),
+ GATE_TOP3(CLK_TOP_RG_UART2, "rg_uart2", "uart2_sel", 15),
+ GATE_TOP3(CLK_TOP_RG_BSI, "rg_bsi", "bsi_sel", 16),
+ GATE_TOP3(CLK_TOP_RG_DBG_ATCLK, "rg_dbg_atclk", "dbg_atclk_sel", 17),
+ GATE_TOP3(CLK_TOP_RG_NFIECC, "rg_nfiecc", "nfiecc_sel", 18),
+ /* TOP4 */
+ GATE_TOP4_I(CLK_TOP_RG_APLL1_D2_EN, "rg_apll1_d2_en", "apll1_d2", 8),
+ GATE_TOP4_I(CLK_TOP_RG_APLL1_D4_EN, "rg_apll1_d4_en", "apll1_d4", 9),
+ GATE_TOP4_I(CLK_TOP_RG_APLL1_D8_EN, "rg_apll1_d8_en", "apll1_d8", 10),
+ GATE_TOP4_I(CLK_TOP_RG_APLL2_D2_EN, "rg_apll2_d2_en", "apll2_d2", 11),
+ GATE_TOP4_I(CLK_TOP_RG_APLL2_D4_EN, "rg_apll2_d4_en", "apll2_d4", 12),
+ GATE_TOP4_I(CLK_TOP_RG_APLL2_D8_EN, "rg_apll2_d8_en", "apll2_d8", 13),
+ /* TOP5 */
+ GATE_TOP5(CLK_TOP_APLL12_DIV0, "apll12_div0", "apll12_ck_div0", 0),
+ GATE_TOP5(CLK_TOP_APLL12_DIV1, "apll12_div1", "apll12_ck_div1", 1),
+ GATE_TOP5(CLK_TOP_APLL12_DIV2, "apll12_div2", "apll12_ck_div2", 2),
+ GATE_TOP5(CLK_TOP_APLL12_DIV3, "apll12_div3", "apll12_ck_div3", 3),
+ GATE_TOP5(CLK_TOP_APLL12_DIV4, "apll12_div4", "apll12_ck_div4", 4),
+ GATE_TOP5(CLK_TOP_APLL12_DIV4B, "apll12_div4b", "apll12_ck_div4b", 5),
+ GATE_TOP5(CLK_TOP_APLL12_DIV5, "apll12_div5", "apll12_ck_div5", 6),
+ GATE_TOP5(CLK_TOP_APLL12_DIV5B, "apll12_div5b", "apll12_ck_div5b", 7),
+ GATE_TOP5(CLK_TOP_APLL12_DIV6, "apll12_div6", "apll12_ck_div6", 8),
+};
+
+static void __init mtk_topckgen_init(struct device_node *node)
+{
+ struct clk_onecell_data *clk_data;
+ int r;
+ void __iomem *base;
+
+ base = of_iomap(node, 0);
+ if (!base) {
+ pr_err("%s(): ioremap failed\n", __func__);
+ return;
+ }
+
+ clk_data = mtk_alloc_clk_data(CLK_TOP_NR_CLK);
+
+ mtk_clk_register_fixed_clks(fixed_clks, ARRAY_SIZE(fixed_clks),
+ clk_data);
+ mtk_clk_register_gates(node, top_clks, ARRAY_SIZE(top_clks), clk_data);
+
+ mtk_clk_register_factors(top_divs, ARRAY_SIZE(top_divs), clk_data);
+ mtk_clk_register_composites(top_muxes, ARRAY_SIZE(top_muxes), base,
+ &mt8516_clk_lock, clk_data);
+ mtk_clk_register_dividers(top_adj_divs, ARRAY_SIZE(top_adj_divs),
+ base, &mt8516_clk_lock, clk_data);
+
+ r = of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+ if (r)
+ pr_err("%s(): could not register clock provider: %d\n",
+ __func__, r);
+}
+CLK_OF_DECLARE(mtk_topckgen, "mediatek,mt8516-topckgen", mtk_topckgen_init);
+
+static void __init mtk_infracfg_init(struct device_node *node)
+{
+ struct clk_onecell_data *clk_data;
+ int r;
+ void __iomem *base;
+
+ base = of_iomap(node, 0);
+ if (!base) {
+ pr_err("%s(): ioremap failed\n", __func__);
+ return;
+ }
+
+ clk_data = mtk_alloc_clk_data(CLK_IFR_NR_CLK);
+
+ mtk_clk_register_composites(ifr_muxes, ARRAY_SIZE(ifr_muxes), base,
+ &mt8516_clk_lock, clk_data);
+
+ r = of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+ if (r)
+ pr_err("%s(): could not register clock provider: %d\n",
+ __func__, r);
+}
+CLK_OF_DECLARE(mtk_infracfg, "mediatek,mt8516-infracfg", mtk_infracfg_init);
+
+#define MT8516_PLL_FMAX (1502UL * MHZ)
+
+#define CON0_MT8516_RST_BAR BIT(27)
+
+#define PLL_B(_id, _name, _reg, _pwr_reg, _en_mask, _flags, _pcwbits, \
+ _pd_reg, _pd_shift, _tuner_reg, _pcw_reg, \
+ _pcw_shift, _div_table) { \
+ .id = _id, \
+ .name = _name, \
+ .reg = _reg, \
+ .pwr_reg = _pwr_reg, \
+ .en_mask = _en_mask, \
+ .flags = _flags, \
+ .rst_bar_mask = CON0_MT8516_RST_BAR, \
+ .fmax = MT8516_PLL_FMAX, \
+ .pcwbits = _pcwbits, \
+ .pd_reg = _pd_reg, \
+ .pd_shift = _pd_shift, \
+ .tuner_reg = _tuner_reg, \
+ .pcw_reg = _pcw_reg, \
+ .pcw_shift = _pcw_shift, \
+ .div_table = _div_table, \
+ }
+
+#define PLL(_id, _name, _reg, _pwr_reg, _en_mask, _flags, _pcwbits, \
+ _pd_reg, _pd_shift, _tuner_reg, _pcw_reg, \
+ _pcw_shift) \
+ PLL_B(_id, _name, _reg, _pwr_reg, _en_mask, _flags, _pcwbits, \
+ _pd_reg, _pd_shift, _tuner_reg, _pcw_reg, _pcw_shift, \
+ NULL)
+
+static const struct mtk_pll_div_table mmpll_div_table[] = {
+ { .div = 0, .freq = MT8516_PLL_FMAX },
+ { .div = 1, .freq = 1000000000 },
+ { .div = 2, .freq = 604500000 },
+ { .div = 3, .freq = 253500000 },
+ { .div = 4, .freq = 126750000 },
+ { } /* sentinel */
+};
+
+static const struct mtk_pll_data plls[] = {
+ PLL(CLK_APMIXED_ARMPLL, "armpll", 0x0100, 0x0110, 0x00000001, 0,
+ 21, 0x0104, 24, 0, 0x0104, 0),
+ PLL(CLK_APMIXED_MAINPLL, "mainpll", 0x0120, 0x0130, 0x00000001,
+ HAVE_RST_BAR, 21, 0x0124, 24, 0, 0x0124, 0),
+ PLL(CLK_APMIXED_UNIVPLL, "univpll", 0x0140, 0x0150, 0x30000001,
+ HAVE_RST_BAR, 7, 0x0144, 24, 0, 0x0144, 0),
+ PLL_B(CLK_APMIXED_MMPLL, "mmpll", 0x0160, 0x0170, 0x00000001, 0,
+ 21, 0x0164, 24, 0, 0x0164, 0, mmpll_div_table),
+ PLL(CLK_APMIXED_APLL1, "apll1", 0x0180, 0x0190, 0x00000001, 0,
+ 31, 0x0180, 1, 0x0194, 0x0184, 0),
+ PLL(CLK_APMIXED_APLL2, "apll2", 0x01A0, 0x01B0, 0x00000001, 0,
+ 31, 0x01A0, 1, 0x01B4, 0x01A4, 0),
+};
+
+static void __init mtk_apmixedsys_init(struct device_node *node)
+{
+ struct clk_onecell_data *clk_data;
+ void __iomem *base;
+ int r;
+
+ base = of_iomap(node, 0);
+ if (!base) {
+ pr_err("%s(): ioremap failed\n", __func__);
+ return;
+ }
+
+ clk_data = mtk_alloc_clk_data(CLK_APMIXED_NR_CLK);
+
+ mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data);
+
+ r = of_clk_add_provider(node, of_clk_src_onecell_get, clk_data);
+ if (r)
+ pr_err("%s(): could not register clock provider: %d\n",
+ __func__, r);
+
+}
+CLK_OF_DECLARE(mtk_apmixedsys, "mediatek,mt8516-apmixedsys",
+ mtk_apmixedsys_init);
unsigned int flags;
const struct clk_ops *ops;
u32 rst_bar_mask;
+ unsigned long fmin;
unsigned long fmax;
int pcwbits;
+ int pcwibits;
uint32_t pcw_reg;
int pcw_shift;
+ uint32_t pcw_chg_reg;
const struct mtk_pll_div_table *div_table;
const char *parent_name;
};
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Owen Chen <owen.chen@mediatek.com>
+ */
+
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/slab.h>
+#include <linux/mfd/syscon.h>
+
+#include "clk-mtk.h"
+#include "clk-mux.h"
+
+static inline struct mtk_clk_mux *to_mtk_clk_mux(struct clk_hw *hw)
+{
+ return container_of(hw, struct mtk_clk_mux, hw);
+}
+
+static int mtk_clk_mux_enable(struct clk_hw *hw)
+{
+ struct mtk_clk_mux *mux = to_mtk_clk_mux(hw);
+ u32 mask = BIT(mux->data->gate_shift);
+
+ return regmap_update_bits(mux->regmap, mux->data->mux_ofs,
+ mask, ~mask);
+}
+
+static void mtk_clk_mux_disable(struct clk_hw *hw)
+{
+ struct mtk_clk_mux *mux = to_mtk_clk_mux(hw);
+ u32 mask = BIT(mux->data->gate_shift);
+
+ regmap_update_bits(mux->regmap, mux->data->mux_ofs, mask, mask);
+}
+
+static int mtk_clk_mux_enable_setclr(struct clk_hw *hw)
+{
+ struct mtk_clk_mux *mux = to_mtk_clk_mux(hw);
+
+ return regmap_write(mux->regmap, mux->data->clr_ofs,
+ BIT(mux->data->gate_shift));
+}
+
+static void mtk_clk_mux_disable_setclr(struct clk_hw *hw)
+{
+ struct mtk_clk_mux *mux = to_mtk_clk_mux(hw);
+
+ regmap_write(mux->regmap, mux->data->set_ofs,
+ BIT(mux->data->gate_shift));
+}
+
+static int mtk_clk_mux_is_enabled(struct clk_hw *hw)
+{
+ struct mtk_clk_mux *mux = to_mtk_clk_mux(hw);
+ u32 val;
+
+ regmap_read(mux->regmap, mux->data->mux_ofs, &val);
+
+ return (val & BIT(mux->data->gate_shift)) == 0;
+}
+
+static u8 mtk_clk_mux_get_parent(struct clk_hw *hw)
+{
+ struct mtk_clk_mux *mux = to_mtk_clk_mux(hw);
+ u32 mask = GENMASK(mux->data->mux_width - 1, 0);
+ u32 val;
+
+ regmap_read(mux->regmap, mux->data->mux_ofs, &val);
+ val = (val >> mux->data->mux_shift) & mask;
+
+ return val;
+}
+
+static int mtk_clk_mux_set_parent_lock(struct clk_hw *hw, u8 index)
+{
+ struct mtk_clk_mux *mux = to_mtk_clk_mux(hw);
+ u32 mask = GENMASK(mux->data->mux_width - 1, 0);
+ unsigned long flags = 0;
+
+ if (mux->lock)
+ spin_lock_irqsave(mux->lock, flags);
+ else
+ __acquire(mux->lock);
+
+ regmap_update_bits(mux->regmap, mux->data->mux_ofs, mask,
+ index << mux->data->mux_shift);
+
+ if (mux->lock)
+ spin_unlock_irqrestore(mux->lock, flags);
+ else
+ __release(mux->lock);
+
+ return 0;
+}
+
+static int mtk_clk_mux_set_parent_setclr_lock(struct clk_hw *hw, u8 index)
+{
+ struct mtk_clk_mux *mux = to_mtk_clk_mux(hw);
+ u32 mask = GENMASK(mux->data->mux_width - 1, 0);
+ u32 val, orig;
+ unsigned long flags = 0;
+
+ if (mux->lock)
+ spin_lock_irqsave(mux->lock, flags);
+ else
+ __acquire(mux->lock);
+
+ regmap_read(mux->regmap, mux->data->mux_ofs, &orig);
+ val = (orig & ~(mask << mux->data->mux_shift))
+ | (index << mux->data->mux_shift);
+
+ if (val != orig) {
+ regmap_write(mux->regmap, mux->data->clr_ofs,
+ mask << mux->data->mux_shift);
+ regmap_write(mux->regmap, mux->data->set_ofs,
+ index << mux->data->mux_shift);
+
+ if (mux->data->upd_shift >= 0)
+ regmap_write(mux->regmap, mux->data->upd_ofs,
+ BIT(mux->data->upd_shift));
+ }
+
+ if (mux->lock)
+ spin_unlock_irqrestore(mux->lock, flags);
+ else
+ __release(mux->lock);
+
+ return 0;
+}
+
+const struct clk_ops mtk_mux_ops = {
+ .get_parent = mtk_clk_mux_get_parent,
+ .set_parent = mtk_clk_mux_set_parent_lock,
+};
+
+const struct clk_ops mtk_mux_clr_set_upd_ops = {
+ .get_parent = mtk_clk_mux_get_parent,
+ .set_parent = mtk_clk_mux_set_parent_setclr_lock,
+};
+
+const struct clk_ops mtk_mux_gate_ops = {
+ .enable = mtk_clk_mux_enable,
+ .disable = mtk_clk_mux_disable,
+ .is_enabled = mtk_clk_mux_is_enabled,
+ .get_parent = mtk_clk_mux_get_parent,
+ .set_parent = mtk_clk_mux_set_parent_lock,
+};
+
+const struct clk_ops mtk_mux_gate_clr_set_upd_ops = {
+ .enable = mtk_clk_mux_enable_setclr,
+ .disable = mtk_clk_mux_disable_setclr,
+ .is_enabled = mtk_clk_mux_is_enabled,
+ .get_parent = mtk_clk_mux_get_parent,
+ .set_parent = mtk_clk_mux_set_parent_setclr_lock,
+};
+
+struct clk *mtk_clk_register_mux(const struct mtk_mux *mux,
+ struct regmap *regmap,
+ spinlock_t *lock)
+{
+ struct mtk_clk_mux *clk_mux;
+ struct clk_init_data init;
+ struct clk *clk;
+
+ clk_mux = kzalloc(sizeof(*clk_mux), GFP_KERNEL);
+ if (!clk_mux)
+ return ERR_PTR(-ENOMEM);
+
+ init.name = mux->name;
+ init.flags = mux->flags | CLK_SET_RATE_PARENT;
+ init.parent_names = mux->parent_names;
+ init.num_parents = mux->num_parents;
+ init.ops = mux->ops;
+
+ clk_mux->regmap = regmap;
+ clk_mux->data = mux;
+ clk_mux->lock = lock;
+ clk_mux->hw.init = &init;
+
+ clk = clk_register(NULL, &clk_mux->hw);
+ if (IS_ERR(clk)) {
+ kfree(clk_mux);
+ return clk;
+ }
+
+ return clk;
+}
+
+int mtk_clk_register_muxes(const struct mtk_mux *muxes,
+ int num, struct device_node *node,
+ spinlock_t *lock,
+ struct clk_onecell_data *clk_data)
+{
+ struct regmap *regmap;
+ struct clk *clk;
+ int i;
+
+ regmap = syscon_node_to_regmap(node);
+ if (IS_ERR(regmap)) {
+ pr_err("Cannot find regmap for %pOF: %ld\n", node,
+ PTR_ERR(regmap));
+ return PTR_ERR(regmap);
+ }
+
+ for (i = 0; i < num; i++) {
+ const struct mtk_mux *mux = &muxes[i];
+
+ if (IS_ERR_OR_NULL(clk_data->clks[mux->id])) {
+ clk = mtk_clk_register_mux(mux, regmap, lock);
+
+ if (IS_ERR(clk)) {
+ pr_err("Failed to register clk %s: %ld\n",
+ mux->name, PTR_ERR(clk));
+ continue;
+ }
+
+ clk_data->clks[mux->id] = clk;
+ }
+ }
+
+ return 0;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Owen Chen <owen.chen@mediatek.com>
+ */
+
+#ifndef __DRV_CLK_MTK_MUX_H
+#define __DRV_CLK_MTK_MUX_H
+
+#include <linux/clk-provider.h>
+
+struct mtk_clk_mux {
+ struct clk_hw hw;
+ struct regmap *regmap;
+ const struct mtk_mux *data;
+ spinlock_t *lock;
+};
+
+struct mtk_mux {
+ int id;
+ const char *name;
+ const char * const *parent_names;
+ unsigned int flags;
+
+ u32 mux_ofs;
+ u32 set_ofs;
+ u32 clr_ofs;
+ u32 upd_ofs;
+
+ u8 mux_shift;
+ u8 mux_width;
+ u8 gate_shift;
+ s8 upd_shift;
+
+ const struct clk_ops *ops;
+
+ signed char num_parents;
+};
+
+extern const struct clk_ops mtk_mux_ops;
+extern const struct clk_ops mtk_mux_clr_set_upd_ops;
+extern const struct clk_ops mtk_mux_gate_ops;
+extern const struct clk_ops mtk_mux_gate_clr_set_upd_ops;
+
+#define GATE_CLR_SET_UPD_FLAGS(_id, _name, _parents, _mux_ofs, \
+ _mux_set_ofs, _mux_clr_ofs, _shift, _width, \
+ _gate, _upd_ofs, _upd, _flags, _ops) { \
+ .id = _id, \
+ .name = _name, \
+ .mux_ofs = _mux_ofs, \
+ .set_ofs = _mux_set_ofs, \
+ .clr_ofs = _mux_clr_ofs, \
+ .upd_ofs = _upd_ofs, \
+ .mux_shift = _shift, \
+ .mux_width = _width, \
+ .gate_shift = _gate, \
+ .upd_shift = _upd, \
+ .parent_names = _parents, \
+ .num_parents = ARRAY_SIZE(_parents), \
+ .flags = _flags, \
+ .ops = &_ops, \
+ }
+
+#define MUX_GATE_CLR_SET_UPD_FLAGS(_id, _name, _parents, _mux_ofs, \
+ _mux_set_ofs, _mux_clr_ofs, _shift, _width, \
+ _gate, _upd_ofs, _upd, _flags) \
+ GATE_CLR_SET_UPD_FLAGS(_id, _name, _parents, _mux_ofs, \
+ _mux_set_ofs, _mux_clr_ofs, _shift, _width, \
+ _gate, _upd_ofs, _upd, _flags, \
+ mtk_mux_gate_clr_set_upd_ops)
+
+#define MUX_GATE_CLR_SET_UPD(_id, _name, _parents, _mux_ofs, \
+ _mux_set_ofs, _mux_clr_ofs, _shift, _width, \
+ _gate, _upd_ofs, _upd) \
+ MUX_GATE_CLR_SET_UPD_FLAGS(_id, _name, _parents, \
+ _mux_ofs, _mux_set_ofs, _mux_clr_ofs, _shift, \
+ _width, _gate, _upd_ofs, _upd, \
+ CLK_SET_RATE_PARENT)
+
+struct clk *mtk_clk_register_mux(const struct mtk_mux *mux,
+ struct regmap *regmap,
+ spinlock_t *lock);
+
+int mtk_clk_register_muxes(const struct mtk_mux *muxes,
+ int num, struct device_node *node,
+ spinlock_t *lock,
+ struct clk_onecell_data *clk_data);
+
+#endif /* __DRV_CLK_MTK_MUX_H */
#define CON0_BASE_EN BIT(0)
#define CON0_PWR_ON BIT(0)
#define CON0_ISO_EN BIT(1)
-#define CON0_PCW_CHG BIT(31)
+#define PCW_CHG_MASK BIT(31)
#define AUDPLL_TUNER_EN BIT(31)
#define POSTDIV_MASK 0x7
+
+/* default 7 bits integer, can be overridden with pcwibits. */
#define INTEGER_BITS 7
/*
void __iomem *tuner_addr;
void __iomem *tuner_en_addr;
void __iomem *pcw_addr;
+ void __iomem *pcw_chg_addr;
const struct mtk_pll_data *data;
};
u32 pcw, int postdiv)
{
int pcwbits = pll->data->pcwbits;
- int pcwfbits;
+ int pcwfbits = 0;
+ int ibits;
u64 vco;
u8 c = 0;
/* The fractional part of the PLL divider. */
- pcwfbits = pcwbits > INTEGER_BITS ? pcwbits - INTEGER_BITS : 0;
+ ibits = pll->data->pcwibits ? pll->data->pcwibits : INTEGER_BITS;
+ if (pcwbits > ibits)
+ pcwfbits = pcwbits - ibits;
vco = (u64)fin * pcw;
return ((unsigned long)vco + postdiv - 1) / postdiv;
}
+static void __mtk_pll_tuner_enable(struct mtk_clk_pll *pll)
+{
+ u32 r;
+
+ if (pll->tuner_en_addr) {
+ r = readl(pll->tuner_en_addr) | BIT(pll->data->tuner_en_bit);
+ writel(r, pll->tuner_en_addr);
+ } else if (pll->tuner_addr) {
+ r = readl(pll->tuner_addr) | AUDPLL_TUNER_EN;
+ writel(r, pll->tuner_addr);
+ }
+}
+
+static void __mtk_pll_tuner_disable(struct mtk_clk_pll *pll)
+{
+ u32 r;
+
+ if (pll->tuner_en_addr) {
+ r = readl(pll->tuner_en_addr) & ~BIT(pll->data->tuner_en_bit);
+ writel(r, pll->tuner_en_addr);
+ } else if (pll->tuner_addr) {
+ r = readl(pll->tuner_addr) & ~AUDPLL_TUNER_EN;
+ writel(r, pll->tuner_addr);
+ }
+}
+
static void mtk_pll_set_rate_regs(struct mtk_clk_pll *pll, u32 pcw,
int postdiv)
{
- u32 con1, val;
- int pll_en;
+ u32 chg, val;
- pll_en = readl(pll->base_addr + REG_CON0) & CON0_BASE_EN;
+ /* disable tuner */
+ __mtk_pll_tuner_disable(pll);
/* set postdiv */
val = readl(pll->pd_addr);
pll->data->pcw_shift);
val |= pcw << pll->data->pcw_shift;
writel(val, pll->pcw_addr);
-
- con1 = readl(pll->base_addr + REG_CON1);
-
- if (pll_en)
- con1 |= CON0_PCW_CHG;
-
- writel(con1, pll->base_addr + REG_CON1);
+ chg = readl(pll->pcw_chg_addr) | PCW_CHG_MASK;
+ writel(chg, pll->pcw_chg_addr);
if (pll->tuner_addr)
- writel(con1 + 1, pll->tuner_addr);
+ writel(val + 1, pll->tuner_addr);
+
+ /* restore tuner_en */
+ __mtk_pll_tuner_enable(pll);
- if (pll_en)
- udelay(20);
+ udelay(20);
}
/*
static void mtk_pll_calc_values(struct mtk_clk_pll *pll, u32 *pcw, u32 *postdiv,
u32 freq, u32 fin)
{
- unsigned long fmin = 1000 * MHZ;
+ unsigned long fmin = pll->data->fmin ? pll->data->fmin : (1000 * MHZ);
const struct mtk_pll_div_table *div_table = pll->data->div_table;
u64 _pcw;
+ int ibits;
u32 val;
if (freq > pll->data->fmax)
}
/* _pcw = freq * postdiv / fin * 2^pcwfbits */
- _pcw = ((u64)freq << val) << (pll->data->pcwbits - INTEGER_BITS);
+ ibits = pll->data->pcwibits ? pll->data->pcwibits : INTEGER_BITS;
+ _pcw = ((u64)freq << val) << (pll->data->pcwbits - ibits);
do_div(_pcw, fin);
*pcw = (u32)_pcw;
r |= pll->data->en_mask;
writel(r, pll->base_addr + REG_CON0);
- if (pll->tuner_en_addr) {
- r = readl(pll->tuner_en_addr) | BIT(pll->data->tuner_en_bit);
- writel(r, pll->tuner_en_addr);
- } else if (pll->tuner_addr) {
- r = readl(pll->tuner_addr) | AUDPLL_TUNER_EN;
- writel(r, pll->tuner_addr);
- }
+ __mtk_pll_tuner_enable(pll);
udelay(20);
writel(r, pll->base_addr + REG_CON0);
}
- if (pll->tuner_en_addr) {
- r = readl(pll->tuner_en_addr) & ~BIT(pll->data->tuner_en_bit);
- writel(r, pll->tuner_en_addr);
- } else if (pll->tuner_addr) {
- r = readl(pll->tuner_addr) & ~AUDPLL_TUNER_EN;
- writel(r, pll->tuner_addr);
- }
+ __mtk_pll_tuner_disable(pll);
r = readl(pll->base_addr + REG_CON0);
r &= ~CON0_BASE_EN;
pll->pwr_addr = base + data->pwr_reg;
pll->pd_addr = base + data->pd_reg;
pll->pcw_addr = base + data->pcw_reg;
+ if (data->pcw_chg_reg)
+ pll->pcw_chg_addr = base + data->pcw_chg_reg;
+ else
+ pll->pcw_chg_addr = pll->base_addr + REG_CON1;
if (data->tuner_reg)
pll->tuner_addr = base + data->tuner_reg;
if (data->tuner_en_reg)
#include "clk-phase.h"
#include "sclk-div.h"
-#define AXG_MST_IN_COUNT 8
-#define AXG_SLV_SCLK_COUNT 10
-#define AXG_SLV_LRCLK_COUNT 10
+#define AUD_MST_IN_COUNT 8
+#define AUD_SLV_SCLK_COUNT 10
+#define AUD_SLV_LRCLK_COUNT 10
-#define AXG_AUD_GATE(_name, _reg, _bit, _pname, _iflags) \
-struct clk_regmap axg_##_name = { \
+#define AUD_GATE(_name, _reg, _bit, _pname, _iflags) \
+struct clk_regmap aud_##_name = { \
.data = &(struct clk_regmap_gate_data){ \
.offset = (_reg), \
.bit_idx = (_bit), \
}, \
.hw.init = &(struct clk_init_data) { \
- .name = "axg_"#_name, \
+ .name = "aud_"#_name, \
.ops = &clk_regmap_gate_ops, \
.parent_names = (const char *[]){ _pname }, \
.num_parents = 1, \
}, \
}
-#define AXG_AUD_MUX(_name, _reg, _mask, _shift, _dflags, _pnames, _iflags) \
-struct clk_regmap axg_##_name = { \
+#define AUD_MUX(_name, _reg, _mask, _shift, _dflags, _pnames, _iflags) \
+struct clk_regmap aud_##_name = { \
.data = &(struct clk_regmap_mux_data){ \
.offset = (_reg), \
.mask = (_mask), \
.flags = (_dflags), \
}, \
.hw.init = &(struct clk_init_data){ \
- .name = "axg_"#_name, \
+ .name = "aud_"#_name, \
.ops = &clk_regmap_mux_ops, \
.parent_names = (_pnames), \
.num_parents = ARRAY_SIZE(_pnames), \
}, \
}
-#define AXG_AUD_DIV(_name, _reg, _shift, _width, _dflags, _pname, _iflags) \
-struct clk_regmap axg_##_name = { \
+#define AUD_DIV(_name, _reg, _shift, _width, _dflags, _pname, _iflags) \
+struct clk_regmap aud_##_name = { \
.data = &(struct clk_regmap_div_data){ \
.offset = (_reg), \
.shift = (_shift), \
.flags = (_dflags), \
}, \
.hw.init = &(struct clk_init_data){ \
- .name = "axg_"#_name, \
+ .name = "aud_"#_name, \
.ops = &clk_regmap_divider_ops, \
.parent_names = (const char *[]) { _pname }, \
.num_parents = 1, \
}, \
}
-#define AXG_PCLK_GATE(_name, _bit) \
- AXG_AUD_GATE(_name, AUDIO_CLK_GATE_EN, _bit, "axg_audio_pclk", 0)
+#define AUD_PCLK_GATE(_name, _bit) \
+ AUD_GATE(_name, AUDIO_CLK_GATE_EN, _bit, "audio_pclk", 0)
/* Audio peripheral clocks */
-static AXG_PCLK_GATE(ddr_arb, 0);
-static AXG_PCLK_GATE(pdm, 1);
-static AXG_PCLK_GATE(tdmin_a, 2);
-static AXG_PCLK_GATE(tdmin_b, 3);
-static AXG_PCLK_GATE(tdmin_c, 4);
-static AXG_PCLK_GATE(tdmin_lb, 5);
-static AXG_PCLK_GATE(tdmout_a, 6);
-static AXG_PCLK_GATE(tdmout_b, 7);
-static AXG_PCLK_GATE(tdmout_c, 8);
-static AXG_PCLK_GATE(frddr_a, 9);
-static AXG_PCLK_GATE(frddr_b, 10);
-static AXG_PCLK_GATE(frddr_c, 11);
-static AXG_PCLK_GATE(toddr_a, 12);
-static AXG_PCLK_GATE(toddr_b, 13);
-static AXG_PCLK_GATE(toddr_c, 14);
-static AXG_PCLK_GATE(loopback, 15);
-static AXG_PCLK_GATE(spdifin, 16);
-static AXG_PCLK_GATE(spdifout, 17);
-static AXG_PCLK_GATE(resample, 18);
-static AXG_PCLK_GATE(power_detect, 19);
+static AUD_PCLK_GATE(ddr_arb, 0);
+static AUD_PCLK_GATE(pdm, 1);
+static AUD_PCLK_GATE(tdmin_a, 2);
+static AUD_PCLK_GATE(tdmin_b, 3);
+static AUD_PCLK_GATE(tdmin_c, 4);
+static AUD_PCLK_GATE(tdmin_lb, 5);
+static AUD_PCLK_GATE(tdmout_a, 6);
+static AUD_PCLK_GATE(tdmout_b, 7);
+static AUD_PCLK_GATE(tdmout_c, 8);
+static AUD_PCLK_GATE(frddr_a, 9);
+static AUD_PCLK_GATE(frddr_b, 10);
+static AUD_PCLK_GATE(frddr_c, 11);
+static AUD_PCLK_GATE(toddr_a, 12);
+static AUD_PCLK_GATE(toddr_b, 13);
+static AUD_PCLK_GATE(toddr_c, 14);
+static AUD_PCLK_GATE(loopback, 15);
+static AUD_PCLK_GATE(spdifin, 16);
+static AUD_PCLK_GATE(spdifout, 17);
+static AUD_PCLK_GATE(resample, 18);
+static AUD_PCLK_GATE(power_detect, 19);
+static AUD_PCLK_GATE(spdifout_b, 21);
/* Audio Master Clocks */
static const char * const mst_mux_parent_names[] = {
- "axg_mst_in0", "axg_mst_in1", "axg_mst_in2", "axg_mst_in3",
- "axg_mst_in4", "axg_mst_in5", "axg_mst_in6", "axg_mst_in7",
+ "aud_mst_in0", "aud_mst_in1", "aud_mst_in2", "aud_mst_in3",
+ "aud_mst_in4", "aud_mst_in5", "aud_mst_in6", "aud_mst_in7",
};
-#define AXG_MST_MUX(_name, _reg, _flag) \
- AXG_AUD_MUX(_name##_sel, _reg, 0x7, 24, _flag, \
- mst_mux_parent_names, CLK_SET_RATE_PARENT)
-
-#define AXG_MST_MCLK_MUX(_name, _reg) \
- AXG_MST_MUX(_name, _reg, CLK_MUX_ROUND_CLOSEST)
-
-#define AXG_MST_SYS_MUX(_name, _reg) \
- AXG_MST_MUX(_name, _reg, 0)
-
-static AXG_MST_MCLK_MUX(mst_a_mclk, AUDIO_MCLK_A_CTRL);
-static AXG_MST_MCLK_MUX(mst_b_mclk, AUDIO_MCLK_B_CTRL);
-static AXG_MST_MCLK_MUX(mst_c_mclk, AUDIO_MCLK_C_CTRL);
-static AXG_MST_MCLK_MUX(mst_d_mclk, AUDIO_MCLK_D_CTRL);
-static AXG_MST_MCLK_MUX(mst_e_mclk, AUDIO_MCLK_E_CTRL);
-static AXG_MST_MCLK_MUX(mst_f_mclk, AUDIO_MCLK_F_CTRL);
-static AXG_MST_MCLK_MUX(spdifout_clk, AUDIO_CLK_SPDIFOUT_CTRL);
-static AXG_MST_MCLK_MUX(pdm_dclk, AUDIO_CLK_PDMIN_CTRL0);
-static AXG_MST_SYS_MUX(spdifin_clk, AUDIO_CLK_SPDIFIN_CTRL);
-static AXG_MST_SYS_MUX(pdm_sysclk, AUDIO_CLK_PDMIN_CTRL1);
-
-#define AXG_MST_DIV(_name, _reg, _flag) \
- AXG_AUD_DIV(_name##_div, _reg, 0, 16, _flag, \
- "axg_"#_name"_sel", CLK_SET_RATE_PARENT) \
-
-#define AXG_MST_MCLK_DIV(_name, _reg) \
- AXG_MST_DIV(_name, _reg, CLK_DIVIDER_ROUND_CLOSEST)
-
-#define AXG_MST_SYS_DIV(_name, _reg) \
- AXG_MST_DIV(_name, _reg, 0)
-
-static AXG_MST_MCLK_DIV(mst_a_mclk, AUDIO_MCLK_A_CTRL);
-static AXG_MST_MCLK_DIV(mst_b_mclk, AUDIO_MCLK_B_CTRL);
-static AXG_MST_MCLK_DIV(mst_c_mclk, AUDIO_MCLK_C_CTRL);
-static AXG_MST_MCLK_DIV(mst_d_mclk, AUDIO_MCLK_D_CTRL);
-static AXG_MST_MCLK_DIV(mst_e_mclk, AUDIO_MCLK_E_CTRL);
-static AXG_MST_MCLK_DIV(mst_f_mclk, AUDIO_MCLK_F_CTRL);
-static AXG_MST_MCLK_DIV(spdifout_clk, AUDIO_CLK_SPDIFOUT_CTRL);
-static AXG_MST_MCLK_DIV(pdm_dclk, AUDIO_CLK_PDMIN_CTRL0);
-static AXG_MST_SYS_DIV(spdifin_clk, AUDIO_CLK_SPDIFIN_CTRL);
-static AXG_MST_SYS_DIV(pdm_sysclk, AUDIO_CLK_PDMIN_CTRL1);
-
-#define AXG_MST_MCLK_GATE(_name, _reg) \
- AXG_AUD_GATE(_name, _reg, 31, "axg_"#_name"_div", \
- CLK_SET_RATE_PARENT)
-
-static AXG_MST_MCLK_GATE(mst_a_mclk, AUDIO_MCLK_A_CTRL);
-static AXG_MST_MCLK_GATE(mst_b_mclk, AUDIO_MCLK_B_CTRL);
-static AXG_MST_MCLK_GATE(mst_c_mclk, AUDIO_MCLK_C_CTRL);
-static AXG_MST_MCLK_GATE(mst_d_mclk, AUDIO_MCLK_D_CTRL);
-static AXG_MST_MCLK_GATE(mst_e_mclk, AUDIO_MCLK_E_CTRL);
-static AXG_MST_MCLK_GATE(mst_f_mclk, AUDIO_MCLK_F_CTRL);
-static AXG_MST_MCLK_GATE(spdifout_clk, AUDIO_CLK_SPDIFOUT_CTRL);
-static AXG_MST_MCLK_GATE(spdifin_clk, AUDIO_CLK_SPDIFIN_CTRL);
-static AXG_MST_MCLK_GATE(pdm_dclk, AUDIO_CLK_PDMIN_CTRL0);
-static AXG_MST_MCLK_GATE(pdm_sysclk, AUDIO_CLK_PDMIN_CTRL1);
+#define AUD_MST_MUX(_name, _reg, _flag) \
+ AUD_MUX(_name##_sel, _reg, 0x7, 24, _flag, \
+ mst_mux_parent_names, CLK_SET_RATE_PARENT)
+
+#define AUD_MST_MCLK_MUX(_name, _reg) \
+ AUD_MST_MUX(_name, _reg, CLK_MUX_ROUND_CLOSEST)
+
+#define AUD_MST_SYS_MUX(_name, _reg) \
+ AUD_MST_MUX(_name, _reg, 0)
+
+static AUD_MST_MCLK_MUX(mst_a_mclk, AUDIO_MCLK_A_CTRL);
+static AUD_MST_MCLK_MUX(mst_b_mclk, AUDIO_MCLK_B_CTRL);
+static AUD_MST_MCLK_MUX(mst_c_mclk, AUDIO_MCLK_C_CTRL);
+static AUD_MST_MCLK_MUX(mst_d_mclk, AUDIO_MCLK_D_CTRL);
+static AUD_MST_MCLK_MUX(mst_e_mclk, AUDIO_MCLK_E_CTRL);
+static AUD_MST_MCLK_MUX(mst_f_mclk, AUDIO_MCLK_F_CTRL);
+static AUD_MST_MCLK_MUX(spdifout_clk, AUDIO_CLK_SPDIFOUT_CTRL);
+static AUD_MST_MCLK_MUX(pdm_dclk, AUDIO_CLK_PDMIN_CTRL0);
+static AUD_MST_SYS_MUX(spdifin_clk, AUDIO_CLK_SPDIFIN_CTRL);
+static AUD_MST_SYS_MUX(pdm_sysclk, AUDIO_CLK_PDMIN_CTRL1);
+static AUD_MST_MCLK_MUX(spdifout_b_clk, AUDIO_CLK_SPDIFOUT_B_CTRL);
+
+#define AUD_MST_DIV(_name, _reg, _flag) \
+ AUD_DIV(_name##_div, _reg, 0, 16, _flag, \
+ "aud_"#_name"_sel", CLK_SET_RATE_PARENT) \
+
+#define AUD_MST_MCLK_DIV(_name, _reg) \
+ AUD_MST_DIV(_name, _reg, CLK_DIVIDER_ROUND_CLOSEST)
+
+#define AUD_MST_SYS_DIV(_name, _reg) \
+ AUD_MST_DIV(_name, _reg, 0)
+
+static AUD_MST_MCLK_DIV(mst_a_mclk, AUDIO_MCLK_A_CTRL);
+static AUD_MST_MCLK_DIV(mst_b_mclk, AUDIO_MCLK_B_CTRL);
+static AUD_MST_MCLK_DIV(mst_c_mclk, AUDIO_MCLK_C_CTRL);
+static AUD_MST_MCLK_DIV(mst_d_mclk, AUDIO_MCLK_D_CTRL);
+static AUD_MST_MCLK_DIV(mst_e_mclk, AUDIO_MCLK_E_CTRL);
+static AUD_MST_MCLK_DIV(mst_f_mclk, AUDIO_MCLK_F_CTRL);
+static AUD_MST_MCLK_DIV(spdifout_clk, AUDIO_CLK_SPDIFOUT_CTRL);
+static AUD_MST_MCLK_DIV(pdm_dclk, AUDIO_CLK_PDMIN_CTRL0);
+static AUD_MST_SYS_DIV(spdifin_clk, AUDIO_CLK_SPDIFIN_CTRL);
+static AUD_MST_SYS_DIV(pdm_sysclk, AUDIO_CLK_PDMIN_CTRL1);
+static AUD_MST_MCLK_DIV(spdifout_b_clk, AUDIO_CLK_SPDIFOUT_B_CTRL);
+
+#define AUD_MST_MCLK_GATE(_name, _reg) \
+ AUD_GATE(_name, _reg, 31, "aud_"#_name"_div", \
+ CLK_SET_RATE_PARENT)
+
+static AUD_MST_MCLK_GATE(mst_a_mclk, AUDIO_MCLK_A_CTRL);
+static AUD_MST_MCLK_GATE(mst_b_mclk, AUDIO_MCLK_B_CTRL);
+static AUD_MST_MCLK_GATE(mst_c_mclk, AUDIO_MCLK_C_CTRL);
+static AUD_MST_MCLK_GATE(mst_d_mclk, AUDIO_MCLK_D_CTRL);
+static AUD_MST_MCLK_GATE(mst_e_mclk, AUDIO_MCLK_E_CTRL);
+static AUD_MST_MCLK_GATE(mst_f_mclk, AUDIO_MCLK_F_CTRL);
+static AUD_MST_MCLK_GATE(spdifout_clk, AUDIO_CLK_SPDIFOUT_CTRL);
+static AUD_MST_MCLK_GATE(spdifin_clk, AUDIO_CLK_SPDIFIN_CTRL);
+static AUD_MST_MCLK_GATE(pdm_dclk, AUDIO_CLK_PDMIN_CTRL0);
+static AUD_MST_MCLK_GATE(pdm_sysclk, AUDIO_CLK_PDMIN_CTRL1);
+static AUD_MST_MCLK_GATE(spdifout_b_clk, AUDIO_CLK_SPDIFOUT_B_CTRL);
/* Sample Clocks */
-#define AXG_MST_SCLK_PRE_EN(_name, _reg) \
- AXG_AUD_GATE(mst_##_name##_sclk_pre_en, _reg, 31, \
- "axg_mst_"#_name"_mclk", 0)
-
-static AXG_MST_SCLK_PRE_EN(a, AUDIO_MST_A_SCLK_CTRL0);
-static AXG_MST_SCLK_PRE_EN(b, AUDIO_MST_B_SCLK_CTRL0);
-static AXG_MST_SCLK_PRE_EN(c, AUDIO_MST_C_SCLK_CTRL0);
-static AXG_MST_SCLK_PRE_EN(d, AUDIO_MST_D_SCLK_CTRL0);
-static AXG_MST_SCLK_PRE_EN(e, AUDIO_MST_E_SCLK_CTRL0);
-static AXG_MST_SCLK_PRE_EN(f, AUDIO_MST_F_SCLK_CTRL0);
-
-#define AXG_AUD_SCLK_DIV(_name, _reg, _div_shift, _div_width, \
+#define AUD_MST_SCLK_PRE_EN(_name, _reg) \
+ AUD_GATE(mst_##_name##_sclk_pre_en, _reg, 31, \
+ "aud_mst_"#_name"_mclk", 0)
+
+static AUD_MST_SCLK_PRE_EN(a, AUDIO_MST_A_SCLK_CTRL0);
+static AUD_MST_SCLK_PRE_EN(b, AUDIO_MST_B_SCLK_CTRL0);
+static AUD_MST_SCLK_PRE_EN(c, AUDIO_MST_C_SCLK_CTRL0);
+static AUD_MST_SCLK_PRE_EN(d, AUDIO_MST_D_SCLK_CTRL0);
+static AUD_MST_SCLK_PRE_EN(e, AUDIO_MST_E_SCLK_CTRL0);
+static AUD_MST_SCLK_PRE_EN(f, AUDIO_MST_F_SCLK_CTRL0);
+
+#define AUD_SCLK_DIV(_name, _reg, _div_shift, _div_width, \
_hi_shift, _hi_width, _pname, _iflags) \
-struct clk_regmap axg_##_name = { \
+struct clk_regmap aud_##_name = { \
.data = &(struct meson_sclk_div_data) { \
.div = { \
.reg_off = (_reg), \
}, \
}, \
.hw.init = &(struct clk_init_data) { \
- .name = "axg_"#_name, \
+ .name = "aud_"#_name, \
.ops = &meson_sclk_div_ops, \
.parent_names = (const char *[]) { _pname }, \
.num_parents = 1, \
}, \
}
-#define AXG_MST_SCLK_DIV(_name, _reg) \
- AXG_AUD_SCLK_DIV(mst_##_name##_sclk_div, _reg, 20, 10, 0, 0, \
- "axg_mst_"#_name"_sclk_pre_en", \
- CLK_SET_RATE_PARENT)
-
-static AXG_MST_SCLK_DIV(a, AUDIO_MST_A_SCLK_CTRL0);
-static AXG_MST_SCLK_DIV(b, AUDIO_MST_B_SCLK_CTRL0);
-static AXG_MST_SCLK_DIV(c, AUDIO_MST_C_SCLK_CTRL0);
-static AXG_MST_SCLK_DIV(d, AUDIO_MST_D_SCLK_CTRL0);
-static AXG_MST_SCLK_DIV(e, AUDIO_MST_E_SCLK_CTRL0);
-static AXG_MST_SCLK_DIV(f, AUDIO_MST_F_SCLK_CTRL0);
-
-#define AXG_MST_SCLK_POST_EN(_name, _reg) \
- AXG_AUD_GATE(mst_##_name##_sclk_post_en, _reg, 30, \
- "axg_mst_"#_name"_sclk_div", CLK_SET_RATE_PARENT)
-
-static AXG_MST_SCLK_POST_EN(a, AUDIO_MST_A_SCLK_CTRL0);
-static AXG_MST_SCLK_POST_EN(b, AUDIO_MST_B_SCLK_CTRL0);
-static AXG_MST_SCLK_POST_EN(c, AUDIO_MST_C_SCLK_CTRL0);
-static AXG_MST_SCLK_POST_EN(d, AUDIO_MST_D_SCLK_CTRL0);
-static AXG_MST_SCLK_POST_EN(e, AUDIO_MST_E_SCLK_CTRL0);
-static AXG_MST_SCLK_POST_EN(f, AUDIO_MST_F_SCLK_CTRL0);
-
-#define AXG_AUD_TRIPHASE(_name, _reg, _width, _shift0, _shift1, _shift2, \
+#define AUD_MST_SCLK_DIV(_name, _reg) \
+ AUD_SCLK_DIV(mst_##_name##_sclk_div, _reg, 20, 10, 0, 0, \
+ "aud_mst_"#_name"_sclk_pre_en", \
+ CLK_SET_RATE_PARENT)
+
+static AUD_MST_SCLK_DIV(a, AUDIO_MST_A_SCLK_CTRL0);
+static AUD_MST_SCLK_DIV(b, AUDIO_MST_B_SCLK_CTRL0);
+static AUD_MST_SCLK_DIV(c, AUDIO_MST_C_SCLK_CTRL0);
+static AUD_MST_SCLK_DIV(d, AUDIO_MST_D_SCLK_CTRL0);
+static AUD_MST_SCLK_DIV(e, AUDIO_MST_E_SCLK_CTRL0);
+static AUD_MST_SCLK_DIV(f, AUDIO_MST_F_SCLK_CTRL0);
+
+#define AUD_MST_SCLK_POST_EN(_name, _reg) \
+ AUD_GATE(mst_##_name##_sclk_post_en, _reg, 30, \
+ "aud_mst_"#_name"_sclk_div", CLK_SET_RATE_PARENT)
+
+static AUD_MST_SCLK_POST_EN(a, AUDIO_MST_A_SCLK_CTRL0);
+static AUD_MST_SCLK_POST_EN(b, AUDIO_MST_B_SCLK_CTRL0);
+static AUD_MST_SCLK_POST_EN(c, AUDIO_MST_C_SCLK_CTRL0);
+static AUD_MST_SCLK_POST_EN(d, AUDIO_MST_D_SCLK_CTRL0);
+static AUD_MST_SCLK_POST_EN(e, AUDIO_MST_E_SCLK_CTRL0);
+static AUD_MST_SCLK_POST_EN(f, AUDIO_MST_F_SCLK_CTRL0);
+
+#define AUD_TRIPHASE(_name, _reg, _width, _shift0, _shift1, _shift2, \
_pname, _iflags) \
-struct clk_regmap axg_##_name = { \
+struct clk_regmap aud_##_name = { \
.data = &(struct meson_clk_triphase_data) { \
.ph0 = { \
.reg_off = (_reg), \
}, \
}, \
.hw.init = &(struct clk_init_data) { \
- .name = "axg_"#_name, \
+ .name = "aud_"#_name, \
.ops = &meson_clk_triphase_ops, \
.parent_names = (const char *[]) { _pname }, \
.num_parents = 1, \
}, \
}
-#define AXG_MST_SCLK(_name, _reg) \
- AXG_AUD_TRIPHASE(mst_##_name##_sclk, _reg, 1, 0, 2, 4, \
- "axg_mst_"#_name"_sclk_post_en", CLK_SET_RATE_PARENT)
-
-static AXG_MST_SCLK(a, AUDIO_MST_A_SCLK_CTRL1);
-static AXG_MST_SCLK(b, AUDIO_MST_B_SCLK_CTRL1);
-static AXG_MST_SCLK(c, AUDIO_MST_C_SCLK_CTRL1);
-static AXG_MST_SCLK(d, AUDIO_MST_D_SCLK_CTRL1);
-static AXG_MST_SCLK(e, AUDIO_MST_E_SCLK_CTRL1);
-static AXG_MST_SCLK(f, AUDIO_MST_F_SCLK_CTRL1);
-
-#define AXG_MST_LRCLK_DIV(_name, _reg) \
- AXG_AUD_SCLK_DIV(mst_##_name##_lrclk_div, _reg, 0, 10, 10, 10, \
- "axg_mst_"#_name"_sclk_post_en", 0) \
-
-static AXG_MST_LRCLK_DIV(a, AUDIO_MST_A_SCLK_CTRL0);
-static AXG_MST_LRCLK_DIV(b, AUDIO_MST_B_SCLK_CTRL0);
-static AXG_MST_LRCLK_DIV(c, AUDIO_MST_C_SCLK_CTRL0);
-static AXG_MST_LRCLK_DIV(d, AUDIO_MST_D_SCLK_CTRL0);
-static AXG_MST_LRCLK_DIV(e, AUDIO_MST_E_SCLK_CTRL0);
-static AXG_MST_LRCLK_DIV(f, AUDIO_MST_F_SCLK_CTRL0);
-
-#define AXG_MST_LRCLK(_name, _reg) \
- AXG_AUD_TRIPHASE(mst_##_name##_lrclk, _reg, 1, 1, 3, 5, \
- "axg_mst_"#_name"_lrclk_div", CLK_SET_RATE_PARENT)
-
-static AXG_MST_LRCLK(a, AUDIO_MST_A_SCLK_CTRL1);
-static AXG_MST_LRCLK(b, AUDIO_MST_B_SCLK_CTRL1);
-static AXG_MST_LRCLK(c, AUDIO_MST_C_SCLK_CTRL1);
-static AXG_MST_LRCLK(d, AUDIO_MST_D_SCLK_CTRL1);
-static AXG_MST_LRCLK(e, AUDIO_MST_E_SCLK_CTRL1);
-static AXG_MST_LRCLK(f, AUDIO_MST_F_SCLK_CTRL1);
+#define AUD_MST_SCLK(_name, _reg) \
+ AUD_TRIPHASE(mst_##_name##_sclk, _reg, 1, 0, 2, 4, \
+ "aud_mst_"#_name"_sclk_post_en", CLK_SET_RATE_PARENT)
+
+static AUD_MST_SCLK(a, AUDIO_MST_A_SCLK_CTRL1);
+static AUD_MST_SCLK(b, AUDIO_MST_B_SCLK_CTRL1);
+static AUD_MST_SCLK(c, AUDIO_MST_C_SCLK_CTRL1);
+static AUD_MST_SCLK(d, AUDIO_MST_D_SCLK_CTRL1);
+static AUD_MST_SCLK(e, AUDIO_MST_E_SCLK_CTRL1);
+static AUD_MST_SCLK(f, AUDIO_MST_F_SCLK_CTRL1);
+
+#define AUD_MST_LRCLK_DIV(_name, _reg) \
+ AUD_SCLK_DIV(mst_##_name##_lrclk_div, _reg, 0, 10, 10, 10, \
+ "aud_mst_"#_name"_sclk_post_en", 0) \
+
+static AUD_MST_LRCLK_DIV(a, AUDIO_MST_A_SCLK_CTRL0);
+static AUD_MST_LRCLK_DIV(b, AUDIO_MST_B_SCLK_CTRL0);
+static AUD_MST_LRCLK_DIV(c, AUDIO_MST_C_SCLK_CTRL0);
+static AUD_MST_LRCLK_DIV(d, AUDIO_MST_D_SCLK_CTRL0);
+static AUD_MST_LRCLK_DIV(e, AUDIO_MST_E_SCLK_CTRL0);
+static AUD_MST_LRCLK_DIV(f, AUDIO_MST_F_SCLK_CTRL0);
+
+#define AUD_MST_LRCLK(_name, _reg) \
+ AUD_TRIPHASE(mst_##_name##_lrclk, _reg, 1, 1, 3, 5, \
+ "aud_mst_"#_name"_lrclk_div", CLK_SET_RATE_PARENT)
+
+static AUD_MST_LRCLK(a, AUDIO_MST_A_SCLK_CTRL1);
+static AUD_MST_LRCLK(b, AUDIO_MST_B_SCLK_CTRL1);
+static AUD_MST_LRCLK(c, AUDIO_MST_C_SCLK_CTRL1);
+static AUD_MST_LRCLK(d, AUDIO_MST_D_SCLK_CTRL1);
+static AUD_MST_LRCLK(e, AUDIO_MST_E_SCLK_CTRL1);
+static AUD_MST_LRCLK(f, AUDIO_MST_F_SCLK_CTRL1);
static const char * const tdm_sclk_parent_names[] = {
- "axg_mst_a_sclk", "axg_mst_b_sclk", "axg_mst_c_sclk",
- "axg_mst_d_sclk", "axg_mst_e_sclk", "axg_mst_f_sclk",
- "axg_slv_sclk0", "axg_slv_sclk1", "axg_slv_sclk2",
- "axg_slv_sclk3", "axg_slv_sclk4", "axg_slv_sclk5",
- "axg_slv_sclk6", "axg_slv_sclk7", "axg_slv_sclk8",
- "axg_slv_sclk9"
+ "aud_mst_a_sclk", "aud_mst_b_sclk", "aud_mst_c_sclk",
+ "aud_mst_d_sclk", "aud_mst_e_sclk", "aud_mst_f_sclk",
+ "aud_slv_sclk0", "aud_slv_sclk1", "aud_slv_sclk2",
+ "aud_slv_sclk3", "aud_slv_sclk4", "aud_slv_sclk5",
+ "aud_slv_sclk6", "aud_slv_sclk7", "aud_slv_sclk8",
+ "aud_slv_sclk9"
};
-#define AXG_TDM_SCLK_MUX(_name, _reg) \
- AXG_AUD_MUX(tdm##_name##_sclk_sel, _reg, 0xf, 24, \
+#define AUD_TDM_SCLK_MUX(_name, _reg) \
+ AUD_MUX(tdm##_name##_sclk_sel, _reg, 0xf, 24, \
CLK_MUX_ROUND_CLOSEST, \
tdm_sclk_parent_names, 0)
-static AXG_TDM_SCLK_MUX(in_a, AUDIO_CLK_TDMIN_A_CTRL);
-static AXG_TDM_SCLK_MUX(in_b, AUDIO_CLK_TDMIN_B_CTRL);
-static AXG_TDM_SCLK_MUX(in_c, AUDIO_CLK_TDMIN_C_CTRL);
-static AXG_TDM_SCLK_MUX(in_lb, AUDIO_CLK_TDMIN_LB_CTRL);
-static AXG_TDM_SCLK_MUX(out_a, AUDIO_CLK_TDMOUT_A_CTRL);
-static AXG_TDM_SCLK_MUX(out_b, AUDIO_CLK_TDMOUT_B_CTRL);
-static AXG_TDM_SCLK_MUX(out_c, AUDIO_CLK_TDMOUT_C_CTRL);
-
-#define AXG_TDM_SCLK_PRE_EN(_name, _reg) \
- AXG_AUD_GATE(tdm##_name##_sclk_pre_en, _reg, 31, \
- "axg_tdm"#_name"_sclk_sel", CLK_SET_RATE_PARENT)
-
-static AXG_TDM_SCLK_PRE_EN(in_a, AUDIO_CLK_TDMIN_A_CTRL);
-static AXG_TDM_SCLK_PRE_EN(in_b, AUDIO_CLK_TDMIN_B_CTRL);
-static AXG_TDM_SCLK_PRE_EN(in_c, AUDIO_CLK_TDMIN_C_CTRL);
-static AXG_TDM_SCLK_PRE_EN(in_lb, AUDIO_CLK_TDMIN_LB_CTRL);
-static AXG_TDM_SCLK_PRE_EN(out_a, AUDIO_CLK_TDMOUT_A_CTRL);
-static AXG_TDM_SCLK_PRE_EN(out_b, AUDIO_CLK_TDMOUT_B_CTRL);
-static AXG_TDM_SCLK_PRE_EN(out_c, AUDIO_CLK_TDMOUT_C_CTRL);
-
-#define AXG_TDM_SCLK_POST_EN(_name, _reg) \
- AXG_AUD_GATE(tdm##_name##_sclk_post_en, _reg, 30, \
- "axg_tdm"#_name"_sclk_pre_en", CLK_SET_RATE_PARENT)
-
-static AXG_TDM_SCLK_POST_EN(in_a, AUDIO_CLK_TDMIN_A_CTRL);
-static AXG_TDM_SCLK_POST_EN(in_b, AUDIO_CLK_TDMIN_B_CTRL);
-static AXG_TDM_SCLK_POST_EN(in_c, AUDIO_CLK_TDMIN_C_CTRL);
-static AXG_TDM_SCLK_POST_EN(in_lb, AUDIO_CLK_TDMIN_LB_CTRL);
-static AXG_TDM_SCLK_POST_EN(out_a, AUDIO_CLK_TDMOUT_A_CTRL);
-static AXG_TDM_SCLK_POST_EN(out_b, AUDIO_CLK_TDMOUT_B_CTRL);
-static AXG_TDM_SCLK_POST_EN(out_c, AUDIO_CLK_TDMOUT_C_CTRL);
-
-#define AXG_TDM_SCLK(_name, _reg) \
- struct clk_regmap axg_tdm##_name##_sclk = { \
+static AUD_TDM_SCLK_MUX(in_a, AUDIO_CLK_TDMIN_A_CTRL);
+static AUD_TDM_SCLK_MUX(in_b, AUDIO_CLK_TDMIN_B_CTRL);
+static AUD_TDM_SCLK_MUX(in_c, AUDIO_CLK_TDMIN_C_CTRL);
+static AUD_TDM_SCLK_MUX(in_lb, AUDIO_CLK_TDMIN_LB_CTRL);
+static AUD_TDM_SCLK_MUX(out_a, AUDIO_CLK_TDMOUT_A_CTRL);
+static AUD_TDM_SCLK_MUX(out_b, AUDIO_CLK_TDMOUT_B_CTRL);
+static AUD_TDM_SCLK_MUX(out_c, AUDIO_CLK_TDMOUT_C_CTRL);
+
+#define AUD_TDM_SCLK_PRE_EN(_name, _reg) \
+ AUD_GATE(tdm##_name##_sclk_pre_en, _reg, 31, \
+ "aud_tdm"#_name"_sclk_sel", CLK_SET_RATE_PARENT)
+
+static AUD_TDM_SCLK_PRE_EN(in_a, AUDIO_CLK_TDMIN_A_CTRL);
+static AUD_TDM_SCLK_PRE_EN(in_b, AUDIO_CLK_TDMIN_B_CTRL);
+static AUD_TDM_SCLK_PRE_EN(in_c, AUDIO_CLK_TDMIN_C_CTRL);
+static AUD_TDM_SCLK_PRE_EN(in_lb, AUDIO_CLK_TDMIN_LB_CTRL);
+static AUD_TDM_SCLK_PRE_EN(out_a, AUDIO_CLK_TDMOUT_A_CTRL);
+static AUD_TDM_SCLK_PRE_EN(out_b, AUDIO_CLK_TDMOUT_B_CTRL);
+static AUD_TDM_SCLK_PRE_EN(out_c, AUDIO_CLK_TDMOUT_C_CTRL);
+
+#define AUD_TDM_SCLK_POST_EN(_name, _reg) \
+ AUD_GATE(tdm##_name##_sclk_post_en, _reg, 30, \
+ "aud_tdm"#_name"_sclk_pre_en", CLK_SET_RATE_PARENT)
+
+static AUD_TDM_SCLK_POST_EN(in_a, AUDIO_CLK_TDMIN_A_CTRL);
+static AUD_TDM_SCLK_POST_EN(in_b, AUDIO_CLK_TDMIN_B_CTRL);
+static AUD_TDM_SCLK_POST_EN(in_c, AUDIO_CLK_TDMIN_C_CTRL);
+static AUD_TDM_SCLK_POST_EN(in_lb, AUDIO_CLK_TDMIN_LB_CTRL);
+static AUD_TDM_SCLK_POST_EN(out_a, AUDIO_CLK_TDMOUT_A_CTRL);
+static AUD_TDM_SCLK_POST_EN(out_b, AUDIO_CLK_TDMOUT_B_CTRL);
+static AUD_TDM_SCLK_POST_EN(out_c, AUDIO_CLK_TDMOUT_C_CTRL);
+
+#define AUD_TDM_SCLK(_name, _reg) \
+ struct clk_regmap aud_tdm##_name##_sclk = { \
.data = &(struct meson_clk_phase_data) { \
.ph = { \
.reg_off = (_reg), \
}, \
}, \
.hw.init = &(struct clk_init_data) { \
- .name = "axg_tdm"#_name"_sclk", \
+ .name = "aud_tdm"#_name"_sclk", \
.ops = &meson_clk_phase_ops, \
.parent_names = (const char *[]) \
- { "axg_tdm"#_name"_sclk_post_en" }, \
+ { "aud_tdm"#_name"_sclk_post_en" }, \
.num_parents = 1, \
.flags = CLK_DUTY_CYCLE_PARENT | CLK_SET_RATE_PARENT, \
}, \
}
-static AXG_TDM_SCLK(in_a, AUDIO_CLK_TDMIN_A_CTRL);
-static AXG_TDM_SCLK(in_b, AUDIO_CLK_TDMIN_B_CTRL);
-static AXG_TDM_SCLK(in_c, AUDIO_CLK_TDMIN_C_CTRL);
-static AXG_TDM_SCLK(in_lb, AUDIO_CLK_TDMIN_LB_CTRL);
-static AXG_TDM_SCLK(out_a, AUDIO_CLK_TDMOUT_A_CTRL);
-static AXG_TDM_SCLK(out_b, AUDIO_CLK_TDMOUT_B_CTRL);
-static AXG_TDM_SCLK(out_c, AUDIO_CLK_TDMOUT_C_CTRL);
+static AUD_TDM_SCLK(in_a, AUDIO_CLK_TDMIN_A_CTRL);
+static AUD_TDM_SCLK(in_b, AUDIO_CLK_TDMIN_B_CTRL);
+static AUD_TDM_SCLK(in_c, AUDIO_CLK_TDMIN_C_CTRL);
+static AUD_TDM_SCLK(in_lb, AUDIO_CLK_TDMIN_LB_CTRL);
+static AUD_TDM_SCLK(out_a, AUDIO_CLK_TDMOUT_A_CTRL);
+static AUD_TDM_SCLK(out_b, AUDIO_CLK_TDMOUT_B_CTRL);
+static AUD_TDM_SCLK(out_c, AUDIO_CLK_TDMOUT_C_CTRL);
static const char * const tdm_lrclk_parent_names[] = {
- "axg_mst_a_lrclk", "axg_mst_b_lrclk", "axg_mst_c_lrclk",
- "axg_mst_d_lrclk", "axg_mst_e_lrclk", "axg_mst_f_lrclk",
- "axg_slv_lrclk0", "axg_slv_lrclk1", "axg_slv_lrclk2",
- "axg_slv_lrclk3", "axg_slv_lrclk4", "axg_slv_lrclk5",
- "axg_slv_lrclk6", "axg_slv_lrclk7", "axg_slv_lrclk8",
- "axg_slv_lrclk9"
+ "aud_mst_a_lrclk", "aud_mst_b_lrclk", "aud_mst_c_lrclk",
+ "aud_mst_d_lrclk", "aud_mst_e_lrclk", "aud_mst_f_lrclk",
+ "aud_slv_lrclk0", "aud_slv_lrclk1", "aud_slv_lrclk2",
+ "aud_slv_lrclk3", "aud_slv_lrclk4", "aud_slv_lrclk5",
+ "aud_slv_lrclk6", "aud_slv_lrclk7", "aud_slv_lrclk8",
+ "aud_slv_lrclk9"
};
-#define AXG_TDM_LRLCK(_name, _reg) \
- AXG_AUD_MUX(tdm##_name##_lrclk, _reg, 0xf, 20, \
- CLK_MUX_ROUND_CLOSEST, \
- tdm_lrclk_parent_names, 0)
+#define AUD_TDM_LRLCK(_name, _reg) \
+ AUD_MUX(tdm##_name##_lrclk, _reg, 0xf, 20, \
+ CLK_MUX_ROUND_CLOSEST, \
+ tdm_lrclk_parent_names, 0)
+
+static AUD_TDM_LRLCK(in_a, AUDIO_CLK_TDMIN_A_CTRL);
+static AUD_TDM_LRLCK(in_b, AUDIO_CLK_TDMIN_B_CTRL);
+static AUD_TDM_LRLCK(in_c, AUDIO_CLK_TDMIN_C_CTRL);
+static AUD_TDM_LRLCK(in_lb, AUDIO_CLK_TDMIN_LB_CTRL);
+static AUD_TDM_LRLCK(out_a, AUDIO_CLK_TDMOUT_A_CTRL);
+static AUD_TDM_LRLCK(out_b, AUDIO_CLK_TDMOUT_B_CTRL);
+static AUD_TDM_LRLCK(out_c, AUDIO_CLK_TDMOUT_C_CTRL);
+
+/* G12a Pad control */
+#define AUD_TDM_PAD_CTRL(_name, _reg, _shift, _parents) \
+ AUD_MUX(tdm_##_name, _reg, 0x7, _shift, 0, _parents, \
+ CLK_SET_RATE_NO_REPARENT)
+
+static const char * const mclk_pad_ctrl_parent_names[] = {
+ "aud_mst_a_mclk", "aud_mst_b_mclk", "aud_mst_c_mclk",
+ "aud_mst_d_mclk", "aud_mst_e_mclk", "aud_mst_f_mclk",
+};
+
+static AUD_TDM_PAD_CTRL(mclk_pad_0, AUDIO_MST_PAD_CTRL0, 0,
+ mclk_pad_ctrl_parent_names);
+static AUD_TDM_PAD_CTRL(mclk_pad_1, AUDIO_MST_PAD_CTRL0, 4,
+ mclk_pad_ctrl_parent_names);
+
+static const char * const lrclk_pad_ctrl_parent_names[] = {
+ "aud_mst_a_lrclk", "aud_mst_b_lrclk", "aud_mst_c_lrclk",
+ "aud_mst_d_lrclk", "aud_mst_e_lrclk", "aud_mst_f_lrclk",
+};
-static AXG_TDM_LRLCK(in_a, AUDIO_CLK_TDMIN_A_CTRL);
-static AXG_TDM_LRLCK(in_b, AUDIO_CLK_TDMIN_B_CTRL);
-static AXG_TDM_LRLCK(in_c, AUDIO_CLK_TDMIN_C_CTRL);
-static AXG_TDM_LRLCK(in_lb, AUDIO_CLK_TDMIN_LB_CTRL);
-static AXG_TDM_LRLCK(out_a, AUDIO_CLK_TDMOUT_A_CTRL);
-static AXG_TDM_LRLCK(out_b, AUDIO_CLK_TDMOUT_B_CTRL);
-static AXG_TDM_LRLCK(out_c, AUDIO_CLK_TDMOUT_C_CTRL);
+static AUD_TDM_PAD_CTRL(lrclk_pad_0, AUDIO_MST_PAD_CTRL1, 16,
+ lrclk_pad_ctrl_parent_names);
+static AUD_TDM_PAD_CTRL(lrclk_pad_1, AUDIO_MST_PAD_CTRL1, 20,
+ lrclk_pad_ctrl_parent_names);
+static AUD_TDM_PAD_CTRL(lrclk_pad_2, AUDIO_MST_PAD_CTRL1, 24,
+ lrclk_pad_ctrl_parent_names);
+
+static const char * const sclk_pad_ctrl_parent_names[] = {
+ "aud_mst_a_sclk", "aud_mst_b_sclk", "aud_mst_c_sclk",
+ "aud_mst_d_sclk", "aud_mst_e_sclk", "aud_mst_f_sclk",
+};
+
+static AUD_TDM_PAD_CTRL(sclk_pad_0, AUDIO_MST_PAD_CTRL1, 0,
+ sclk_pad_ctrl_parent_names);
+static AUD_TDM_PAD_CTRL(sclk_pad_1, AUDIO_MST_PAD_CTRL1, 4,
+ sclk_pad_ctrl_parent_names);
+static AUD_TDM_PAD_CTRL(sclk_pad_2, AUDIO_MST_PAD_CTRL1, 8,
+ sclk_pad_ctrl_parent_names);
/*
* Array of all clocks provided by this provider
*/
static struct clk_hw_onecell_data axg_audio_hw_onecell_data = {
.hws = {
- [AUD_CLKID_DDR_ARB] = &axg_ddr_arb.hw,
- [AUD_CLKID_PDM] = &axg_pdm.hw,
- [AUD_CLKID_TDMIN_A] = &axg_tdmin_a.hw,
- [AUD_CLKID_TDMIN_B] = &axg_tdmin_b.hw,
- [AUD_CLKID_TDMIN_C] = &axg_tdmin_c.hw,
- [AUD_CLKID_TDMIN_LB] = &axg_tdmin_lb.hw,
- [AUD_CLKID_TDMOUT_A] = &axg_tdmout_a.hw,
- [AUD_CLKID_TDMOUT_B] = &axg_tdmout_b.hw,
- [AUD_CLKID_TDMOUT_C] = &axg_tdmout_c.hw,
- [AUD_CLKID_FRDDR_A] = &axg_frddr_a.hw,
- [AUD_CLKID_FRDDR_B] = &axg_frddr_b.hw,
- [AUD_CLKID_FRDDR_C] = &axg_frddr_c.hw,
- [AUD_CLKID_TODDR_A] = &axg_toddr_a.hw,
- [AUD_CLKID_TODDR_B] = &axg_toddr_b.hw,
- [AUD_CLKID_TODDR_C] = &axg_toddr_c.hw,
- [AUD_CLKID_LOOPBACK] = &axg_loopback.hw,
- [AUD_CLKID_SPDIFIN] = &axg_spdifin.hw,
- [AUD_CLKID_SPDIFOUT] = &axg_spdifout.hw,
- [AUD_CLKID_RESAMPLE] = &axg_resample.hw,
- [AUD_CLKID_POWER_DETECT] = &axg_power_detect.hw,
- [AUD_CLKID_MST_A_MCLK_SEL] = &axg_mst_a_mclk_sel.hw,
- [AUD_CLKID_MST_B_MCLK_SEL] = &axg_mst_b_mclk_sel.hw,
- [AUD_CLKID_MST_C_MCLK_SEL] = &axg_mst_c_mclk_sel.hw,
- [AUD_CLKID_MST_D_MCLK_SEL] = &axg_mst_d_mclk_sel.hw,
- [AUD_CLKID_MST_E_MCLK_SEL] = &axg_mst_e_mclk_sel.hw,
- [AUD_CLKID_MST_F_MCLK_SEL] = &axg_mst_f_mclk_sel.hw,
- [AUD_CLKID_MST_A_MCLK_DIV] = &axg_mst_a_mclk_div.hw,
- [AUD_CLKID_MST_B_MCLK_DIV] = &axg_mst_b_mclk_div.hw,
- [AUD_CLKID_MST_C_MCLK_DIV] = &axg_mst_c_mclk_div.hw,
- [AUD_CLKID_MST_D_MCLK_DIV] = &axg_mst_d_mclk_div.hw,
- [AUD_CLKID_MST_E_MCLK_DIV] = &axg_mst_e_mclk_div.hw,
- [AUD_CLKID_MST_F_MCLK_DIV] = &axg_mst_f_mclk_div.hw,
- [AUD_CLKID_MST_A_MCLK] = &axg_mst_a_mclk.hw,
- [AUD_CLKID_MST_B_MCLK] = &axg_mst_b_mclk.hw,
- [AUD_CLKID_MST_C_MCLK] = &axg_mst_c_mclk.hw,
- [AUD_CLKID_MST_D_MCLK] = &axg_mst_d_mclk.hw,
- [AUD_CLKID_MST_E_MCLK] = &axg_mst_e_mclk.hw,
- [AUD_CLKID_MST_F_MCLK] = &axg_mst_f_mclk.hw,
- [AUD_CLKID_SPDIFOUT_CLK_SEL] = &axg_spdifout_clk_sel.hw,
- [AUD_CLKID_SPDIFOUT_CLK_DIV] = &axg_spdifout_clk_div.hw,
- [AUD_CLKID_SPDIFOUT_CLK] = &axg_spdifout_clk.hw,
- [AUD_CLKID_SPDIFIN_CLK_SEL] = &axg_spdifin_clk_sel.hw,
- [AUD_CLKID_SPDIFIN_CLK_DIV] = &axg_spdifin_clk_div.hw,
- [AUD_CLKID_SPDIFIN_CLK] = &axg_spdifin_clk.hw,
- [AUD_CLKID_PDM_DCLK_SEL] = &axg_pdm_dclk_sel.hw,
- [AUD_CLKID_PDM_DCLK_DIV] = &axg_pdm_dclk_div.hw,
- [AUD_CLKID_PDM_DCLK] = &axg_pdm_dclk.hw,
- [AUD_CLKID_PDM_SYSCLK_SEL] = &axg_pdm_sysclk_sel.hw,
- [AUD_CLKID_PDM_SYSCLK_DIV] = &axg_pdm_sysclk_div.hw,
- [AUD_CLKID_PDM_SYSCLK] = &axg_pdm_sysclk.hw,
- [AUD_CLKID_MST_A_SCLK_PRE_EN] = &axg_mst_a_sclk_pre_en.hw,
- [AUD_CLKID_MST_B_SCLK_PRE_EN] = &axg_mst_b_sclk_pre_en.hw,
- [AUD_CLKID_MST_C_SCLK_PRE_EN] = &axg_mst_c_sclk_pre_en.hw,
- [AUD_CLKID_MST_D_SCLK_PRE_EN] = &axg_mst_d_sclk_pre_en.hw,
- [AUD_CLKID_MST_E_SCLK_PRE_EN] = &axg_mst_e_sclk_pre_en.hw,
- [AUD_CLKID_MST_F_SCLK_PRE_EN] = &axg_mst_f_sclk_pre_en.hw,
- [AUD_CLKID_MST_A_SCLK_DIV] = &axg_mst_a_sclk_div.hw,
- [AUD_CLKID_MST_B_SCLK_DIV] = &axg_mst_b_sclk_div.hw,
- [AUD_CLKID_MST_C_SCLK_DIV] = &axg_mst_c_sclk_div.hw,
- [AUD_CLKID_MST_D_SCLK_DIV] = &axg_mst_d_sclk_div.hw,
- [AUD_CLKID_MST_E_SCLK_DIV] = &axg_mst_e_sclk_div.hw,
- [AUD_CLKID_MST_F_SCLK_DIV] = &axg_mst_f_sclk_div.hw,
- [AUD_CLKID_MST_A_SCLK_POST_EN] = &axg_mst_a_sclk_post_en.hw,
- [AUD_CLKID_MST_B_SCLK_POST_EN] = &axg_mst_b_sclk_post_en.hw,
- [AUD_CLKID_MST_C_SCLK_POST_EN] = &axg_mst_c_sclk_post_en.hw,
- [AUD_CLKID_MST_D_SCLK_POST_EN] = &axg_mst_d_sclk_post_en.hw,
- [AUD_CLKID_MST_E_SCLK_POST_EN] = &axg_mst_e_sclk_post_en.hw,
- [AUD_CLKID_MST_F_SCLK_POST_EN] = &axg_mst_f_sclk_post_en.hw,
- [AUD_CLKID_MST_A_SCLK] = &axg_mst_a_sclk.hw,
- [AUD_CLKID_MST_B_SCLK] = &axg_mst_b_sclk.hw,
- [AUD_CLKID_MST_C_SCLK] = &axg_mst_c_sclk.hw,
- [AUD_CLKID_MST_D_SCLK] = &axg_mst_d_sclk.hw,
- [AUD_CLKID_MST_E_SCLK] = &axg_mst_e_sclk.hw,
- [AUD_CLKID_MST_F_SCLK] = &axg_mst_f_sclk.hw,
- [AUD_CLKID_MST_A_LRCLK_DIV] = &axg_mst_a_lrclk_div.hw,
- [AUD_CLKID_MST_B_LRCLK_DIV] = &axg_mst_b_lrclk_div.hw,
- [AUD_CLKID_MST_C_LRCLK_DIV] = &axg_mst_c_lrclk_div.hw,
- [AUD_CLKID_MST_D_LRCLK_DIV] = &axg_mst_d_lrclk_div.hw,
- [AUD_CLKID_MST_E_LRCLK_DIV] = &axg_mst_e_lrclk_div.hw,
- [AUD_CLKID_MST_F_LRCLK_DIV] = &axg_mst_f_lrclk_div.hw,
- [AUD_CLKID_MST_A_LRCLK] = &axg_mst_a_lrclk.hw,
- [AUD_CLKID_MST_B_LRCLK] = &axg_mst_b_lrclk.hw,
- [AUD_CLKID_MST_C_LRCLK] = &axg_mst_c_lrclk.hw,
- [AUD_CLKID_MST_D_LRCLK] = &axg_mst_d_lrclk.hw,
- [AUD_CLKID_MST_E_LRCLK] = &axg_mst_e_lrclk.hw,
- [AUD_CLKID_MST_F_LRCLK] = &axg_mst_f_lrclk.hw,
- [AUD_CLKID_TDMIN_A_SCLK_SEL] = &axg_tdmin_a_sclk_sel.hw,
- [AUD_CLKID_TDMIN_B_SCLK_SEL] = &axg_tdmin_b_sclk_sel.hw,
- [AUD_CLKID_TDMIN_C_SCLK_SEL] = &axg_tdmin_c_sclk_sel.hw,
- [AUD_CLKID_TDMIN_LB_SCLK_SEL] = &axg_tdmin_lb_sclk_sel.hw,
- [AUD_CLKID_TDMOUT_A_SCLK_SEL] = &axg_tdmout_a_sclk_sel.hw,
- [AUD_CLKID_TDMOUT_B_SCLK_SEL] = &axg_tdmout_b_sclk_sel.hw,
- [AUD_CLKID_TDMOUT_C_SCLK_SEL] = &axg_tdmout_c_sclk_sel.hw,
- [AUD_CLKID_TDMIN_A_SCLK_PRE_EN] = &axg_tdmin_a_sclk_pre_en.hw,
- [AUD_CLKID_TDMIN_B_SCLK_PRE_EN] = &axg_tdmin_b_sclk_pre_en.hw,
- [AUD_CLKID_TDMIN_C_SCLK_PRE_EN] = &axg_tdmin_c_sclk_pre_en.hw,
- [AUD_CLKID_TDMIN_LB_SCLK_PRE_EN] = &axg_tdmin_lb_sclk_pre_en.hw,
- [AUD_CLKID_TDMOUT_A_SCLK_PRE_EN] = &axg_tdmout_a_sclk_pre_en.hw,
- [AUD_CLKID_TDMOUT_B_SCLK_PRE_EN] = &axg_tdmout_b_sclk_pre_en.hw,
- [AUD_CLKID_TDMOUT_C_SCLK_PRE_EN] = &axg_tdmout_c_sclk_pre_en.hw,
- [AUD_CLKID_TDMIN_A_SCLK_POST_EN] = &axg_tdmin_a_sclk_post_en.hw,
- [AUD_CLKID_TDMIN_B_SCLK_POST_EN] = &axg_tdmin_b_sclk_post_en.hw,
- [AUD_CLKID_TDMIN_C_SCLK_POST_EN] = &axg_tdmin_c_sclk_post_en.hw,
- [AUD_CLKID_TDMIN_LB_SCLK_POST_EN] = &axg_tdmin_lb_sclk_post_en.hw,
- [AUD_CLKID_TDMOUT_A_SCLK_POST_EN] = &axg_tdmout_a_sclk_post_en.hw,
- [AUD_CLKID_TDMOUT_B_SCLK_POST_EN] = &axg_tdmout_b_sclk_post_en.hw,
- [AUD_CLKID_TDMOUT_C_SCLK_POST_EN] = &axg_tdmout_c_sclk_post_en.hw,
- [AUD_CLKID_TDMIN_A_SCLK] = &axg_tdmin_a_sclk.hw,
- [AUD_CLKID_TDMIN_B_SCLK] = &axg_tdmin_b_sclk.hw,
- [AUD_CLKID_TDMIN_C_SCLK] = &axg_tdmin_c_sclk.hw,
- [AUD_CLKID_TDMIN_LB_SCLK] = &axg_tdmin_lb_sclk.hw,
- [AUD_CLKID_TDMOUT_A_SCLK] = &axg_tdmout_a_sclk.hw,
- [AUD_CLKID_TDMOUT_B_SCLK] = &axg_tdmout_b_sclk.hw,
- [AUD_CLKID_TDMOUT_C_SCLK] = &axg_tdmout_c_sclk.hw,
- [AUD_CLKID_TDMIN_A_LRCLK] = &axg_tdmin_a_lrclk.hw,
- [AUD_CLKID_TDMIN_B_LRCLK] = &axg_tdmin_b_lrclk.hw,
- [AUD_CLKID_TDMIN_C_LRCLK] = &axg_tdmin_c_lrclk.hw,
- [AUD_CLKID_TDMIN_LB_LRCLK] = &axg_tdmin_lb_lrclk.hw,
- [AUD_CLKID_TDMOUT_A_LRCLK] = &axg_tdmout_a_lrclk.hw,
- [AUD_CLKID_TDMOUT_B_LRCLK] = &axg_tdmout_b_lrclk.hw,
- [AUD_CLKID_TDMOUT_C_LRCLK] = &axg_tdmout_c_lrclk.hw,
+ [AUD_CLKID_DDR_ARB] = &aud_ddr_arb.hw,
+ [AUD_CLKID_PDM] = &aud_pdm.hw,
+ [AUD_CLKID_TDMIN_A] = &aud_tdmin_a.hw,
+ [AUD_CLKID_TDMIN_B] = &aud_tdmin_b.hw,
+ [AUD_CLKID_TDMIN_C] = &aud_tdmin_c.hw,
+ [AUD_CLKID_TDMIN_LB] = &aud_tdmin_lb.hw,
+ [AUD_CLKID_TDMOUT_A] = &aud_tdmout_a.hw,
+ [AUD_CLKID_TDMOUT_B] = &aud_tdmout_b.hw,
+ [AUD_CLKID_TDMOUT_C] = &aud_tdmout_c.hw,
+ [AUD_CLKID_FRDDR_A] = &aud_frddr_a.hw,
+ [AUD_CLKID_FRDDR_B] = &aud_frddr_b.hw,
+ [AUD_CLKID_FRDDR_C] = &aud_frddr_c.hw,
+ [AUD_CLKID_TODDR_A] = &aud_toddr_a.hw,
+ [AUD_CLKID_TODDR_B] = &aud_toddr_b.hw,
+ [AUD_CLKID_TODDR_C] = &aud_toddr_c.hw,
+ [AUD_CLKID_LOOPBACK] = &aud_loopback.hw,
+ [AUD_CLKID_SPDIFIN] = &aud_spdifin.hw,
+ [AUD_CLKID_SPDIFOUT] = &aud_spdifout.hw,
+ [AUD_CLKID_RESAMPLE] = &aud_resample.hw,
+ [AUD_CLKID_POWER_DETECT] = &aud_power_detect.hw,
+ [AUD_CLKID_MST_A_MCLK_SEL] = &aud_mst_a_mclk_sel.hw,
+ [AUD_CLKID_MST_B_MCLK_SEL] = &aud_mst_b_mclk_sel.hw,
+ [AUD_CLKID_MST_C_MCLK_SEL] = &aud_mst_c_mclk_sel.hw,
+ [AUD_CLKID_MST_D_MCLK_SEL] = &aud_mst_d_mclk_sel.hw,
+ [AUD_CLKID_MST_E_MCLK_SEL] = &aud_mst_e_mclk_sel.hw,
+ [AUD_CLKID_MST_F_MCLK_SEL] = &aud_mst_f_mclk_sel.hw,
+ [AUD_CLKID_MST_A_MCLK_DIV] = &aud_mst_a_mclk_div.hw,
+ [AUD_CLKID_MST_B_MCLK_DIV] = &aud_mst_b_mclk_div.hw,
+ [AUD_CLKID_MST_C_MCLK_DIV] = &aud_mst_c_mclk_div.hw,
+ [AUD_CLKID_MST_D_MCLK_DIV] = &aud_mst_d_mclk_div.hw,
+ [AUD_CLKID_MST_E_MCLK_DIV] = &aud_mst_e_mclk_div.hw,
+ [AUD_CLKID_MST_F_MCLK_DIV] = &aud_mst_f_mclk_div.hw,
+ [AUD_CLKID_MST_A_MCLK] = &aud_mst_a_mclk.hw,
+ [AUD_CLKID_MST_B_MCLK] = &aud_mst_b_mclk.hw,
+ [AUD_CLKID_MST_C_MCLK] = &aud_mst_c_mclk.hw,
+ [AUD_CLKID_MST_D_MCLK] = &aud_mst_d_mclk.hw,
+ [AUD_CLKID_MST_E_MCLK] = &aud_mst_e_mclk.hw,
+ [AUD_CLKID_MST_F_MCLK] = &aud_mst_f_mclk.hw,
+ [AUD_CLKID_SPDIFOUT_CLK_SEL] = &aud_spdifout_clk_sel.hw,
+ [AUD_CLKID_SPDIFOUT_CLK_DIV] = &aud_spdifout_clk_div.hw,
+ [AUD_CLKID_SPDIFOUT_CLK] = &aud_spdifout_clk.hw,
+ [AUD_CLKID_SPDIFIN_CLK_SEL] = &aud_spdifin_clk_sel.hw,
+ [AUD_CLKID_SPDIFIN_CLK_DIV] = &aud_spdifin_clk_div.hw,
+ [AUD_CLKID_SPDIFIN_CLK] = &aud_spdifin_clk.hw,
+ [AUD_CLKID_PDM_DCLK_SEL] = &aud_pdm_dclk_sel.hw,
+ [AUD_CLKID_PDM_DCLK_DIV] = &aud_pdm_dclk_div.hw,
+ [AUD_CLKID_PDM_DCLK] = &aud_pdm_dclk.hw,
+ [AUD_CLKID_PDM_SYSCLK_SEL] = &aud_pdm_sysclk_sel.hw,
+ [AUD_CLKID_PDM_SYSCLK_DIV] = &aud_pdm_sysclk_div.hw,
+ [AUD_CLKID_PDM_SYSCLK] = &aud_pdm_sysclk.hw,
+ [AUD_CLKID_MST_A_SCLK_PRE_EN] = &aud_mst_a_sclk_pre_en.hw,
+ [AUD_CLKID_MST_B_SCLK_PRE_EN] = &aud_mst_b_sclk_pre_en.hw,
+ [AUD_CLKID_MST_C_SCLK_PRE_EN] = &aud_mst_c_sclk_pre_en.hw,
+ [AUD_CLKID_MST_D_SCLK_PRE_EN] = &aud_mst_d_sclk_pre_en.hw,
+ [AUD_CLKID_MST_E_SCLK_PRE_EN] = &aud_mst_e_sclk_pre_en.hw,
+ [AUD_CLKID_MST_F_SCLK_PRE_EN] = &aud_mst_f_sclk_pre_en.hw,
+ [AUD_CLKID_MST_A_SCLK_DIV] = &aud_mst_a_sclk_div.hw,
+ [AUD_CLKID_MST_B_SCLK_DIV] = &aud_mst_b_sclk_div.hw,
+ [AUD_CLKID_MST_C_SCLK_DIV] = &aud_mst_c_sclk_div.hw,
+ [AUD_CLKID_MST_D_SCLK_DIV] = &aud_mst_d_sclk_div.hw,
+ [AUD_CLKID_MST_E_SCLK_DIV] = &aud_mst_e_sclk_div.hw,
+ [AUD_CLKID_MST_F_SCLK_DIV] = &aud_mst_f_sclk_div.hw,
+ [AUD_CLKID_MST_A_SCLK_POST_EN] = &aud_mst_a_sclk_post_en.hw,
+ [AUD_CLKID_MST_B_SCLK_POST_EN] = &aud_mst_b_sclk_post_en.hw,
+ [AUD_CLKID_MST_C_SCLK_POST_EN] = &aud_mst_c_sclk_post_en.hw,
+ [AUD_CLKID_MST_D_SCLK_POST_EN] = &aud_mst_d_sclk_post_en.hw,
+ [AUD_CLKID_MST_E_SCLK_POST_EN] = &aud_mst_e_sclk_post_en.hw,
+ [AUD_CLKID_MST_F_SCLK_POST_EN] = &aud_mst_f_sclk_post_en.hw,
+ [AUD_CLKID_MST_A_SCLK] = &aud_mst_a_sclk.hw,
+ [AUD_CLKID_MST_B_SCLK] = &aud_mst_b_sclk.hw,
+ [AUD_CLKID_MST_C_SCLK] = &aud_mst_c_sclk.hw,
+ [AUD_CLKID_MST_D_SCLK] = &aud_mst_d_sclk.hw,
+ [AUD_CLKID_MST_E_SCLK] = &aud_mst_e_sclk.hw,
+ [AUD_CLKID_MST_F_SCLK] = &aud_mst_f_sclk.hw,
+ [AUD_CLKID_MST_A_LRCLK_DIV] = &aud_mst_a_lrclk_div.hw,
+ [AUD_CLKID_MST_B_LRCLK_DIV] = &aud_mst_b_lrclk_div.hw,
+ [AUD_CLKID_MST_C_LRCLK_DIV] = &aud_mst_c_lrclk_div.hw,
+ [AUD_CLKID_MST_D_LRCLK_DIV] = &aud_mst_d_lrclk_div.hw,
+ [AUD_CLKID_MST_E_LRCLK_DIV] = &aud_mst_e_lrclk_div.hw,
+ [AUD_CLKID_MST_F_LRCLK_DIV] = &aud_mst_f_lrclk_div.hw,
+ [AUD_CLKID_MST_A_LRCLK] = &aud_mst_a_lrclk.hw,
+ [AUD_CLKID_MST_B_LRCLK] = &aud_mst_b_lrclk.hw,
+ [AUD_CLKID_MST_C_LRCLK] = &aud_mst_c_lrclk.hw,
+ [AUD_CLKID_MST_D_LRCLK] = &aud_mst_d_lrclk.hw,
+ [AUD_CLKID_MST_E_LRCLK] = &aud_mst_e_lrclk.hw,
+ [AUD_CLKID_MST_F_LRCLK] = &aud_mst_f_lrclk.hw,
+ [AUD_CLKID_TDMIN_A_SCLK_SEL] = &aud_tdmin_a_sclk_sel.hw,
+ [AUD_CLKID_TDMIN_B_SCLK_SEL] = &aud_tdmin_b_sclk_sel.hw,
+ [AUD_CLKID_TDMIN_C_SCLK_SEL] = &aud_tdmin_c_sclk_sel.hw,
+ [AUD_CLKID_TDMIN_LB_SCLK_SEL] = &aud_tdmin_lb_sclk_sel.hw,
+ [AUD_CLKID_TDMOUT_A_SCLK_SEL] = &aud_tdmout_a_sclk_sel.hw,
+ [AUD_CLKID_TDMOUT_B_SCLK_SEL] = &aud_tdmout_b_sclk_sel.hw,
+ [AUD_CLKID_TDMOUT_C_SCLK_SEL] = &aud_tdmout_c_sclk_sel.hw,
+ [AUD_CLKID_TDMIN_A_SCLK_PRE_EN] = &aud_tdmin_a_sclk_pre_en.hw,
+ [AUD_CLKID_TDMIN_B_SCLK_PRE_EN] = &aud_tdmin_b_sclk_pre_en.hw,
+ [AUD_CLKID_TDMIN_C_SCLK_PRE_EN] = &aud_tdmin_c_sclk_pre_en.hw,
+ [AUD_CLKID_TDMIN_LB_SCLK_PRE_EN] = &aud_tdmin_lb_sclk_pre_en.hw,
+ [AUD_CLKID_TDMOUT_A_SCLK_PRE_EN] = &aud_tdmout_a_sclk_pre_en.hw,
+ [AUD_CLKID_TDMOUT_B_SCLK_PRE_EN] = &aud_tdmout_b_sclk_pre_en.hw,
+ [AUD_CLKID_TDMOUT_C_SCLK_PRE_EN] = &aud_tdmout_c_sclk_pre_en.hw,
+ [AUD_CLKID_TDMIN_A_SCLK_POST_EN] = &aud_tdmin_a_sclk_post_en.hw,
+ [AUD_CLKID_TDMIN_B_SCLK_POST_EN] = &aud_tdmin_b_sclk_post_en.hw,
+ [AUD_CLKID_TDMIN_C_SCLK_POST_EN] = &aud_tdmin_c_sclk_post_en.hw,
+ [AUD_CLKID_TDMIN_LB_SCLK_POST_EN] = &aud_tdmin_lb_sclk_post_en.hw,
+ [AUD_CLKID_TDMOUT_A_SCLK_POST_EN] = &aud_tdmout_a_sclk_post_en.hw,
+ [AUD_CLKID_TDMOUT_B_SCLK_POST_EN] = &aud_tdmout_b_sclk_post_en.hw,
+ [AUD_CLKID_TDMOUT_C_SCLK_POST_EN] = &aud_tdmout_c_sclk_post_en.hw,
+ [AUD_CLKID_TDMIN_A_SCLK] = &aud_tdmin_a_sclk.hw,
+ [AUD_CLKID_TDMIN_B_SCLK] = &aud_tdmin_b_sclk.hw,
+ [AUD_CLKID_TDMIN_C_SCLK] = &aud_tdmin_c_sclk.hw,
+ [AUD_CLKID_TDMIN_LB_SCLK] = &aud_tdmin_lb_sclk.hw,
+ [AUD_CLKID_TDMOUT_A_SCLK] = &aud_tdmout_a_sclk.hw,
+ [AUD_CLKID_TDMOUT_B_SCLK] = &aud_tdmout_b_sclk.hw,
+ [AUD_CLKID_TDMOUT_C_SCLK] = &aud_tdmout_c_sclk.hw,
+ [AUD_CLKID_TDMIN_A_LRCLK] = &aud_tdmin_a_lrclk.hw,
+ [AUD_CLKID_TDMIN_B_LRCLK] = &aud_tdmin_b_lrclk.hw,
+ [AUD_CLKID_TDMIN_C_LRCLK] = &aud_tdmin_c_lrclk.hw,
+ [AUD_CLKID_TDMIN_LB_LRCLK] = &aud_tdmin_lb_lrclk.hw,
+ [AUD_CLKID_TDMOUT_A_LRCLK] = &aud_tdmout_a_lrclk.hw,
+ [AUD_CLKID_TDMOUT_B_LRCLK] = &aud_tdmout_b_lrclk.hw,
+ [AUD_CLKID_TDMOUT_C_LRCLK] = &aud_tdmout_c_lrclk.hw,
[NR_CLKS] = NULL,
},
.num = NR_CLKS,
};
-/* Convenience table to populate regmap in .probe() */
-static struct clk_regmap *const axg_audio_clk_regmaps[] = {
- &axg_ddr_arb,
- &axg_pdm,
- &axg_tdmin_a,
- &axg_tdmin_b,
- &axg_tdmin_c,
- &axg_tdmin_lb,
- &axg_tdmout_a,
- &axg_tdmout_b,
- &axg_tdmout_c,
- &axg_frddr_a,
- &axg_frddr_b,
- &axg_frddr_c,
- &axg_toddr_a,
- &axg_toddr_b,
- &axg_toddr_c,
- &axg_loopback,
- &axg_spdifin,
- &axg_spdifout,
- &axg_resample,
- &axg_power_detect,
- &axg_mst_a_mclk_sel,
- &axg_mst_b_mclk_sel,
- &axg_mst_c_mclk_sel,
- &axg_mst_d_mclk_sel,
- &axg_mst_e_mclk_sel,
- &axg_mst_f_mclk_sel,
- &axg_mst_a_mclk_div,
- &axg_mst_b_mclk_div,
- &axg_mst_c_mclk_div,
- &axg_mst_d_mclk_div,
- &axg_mst_e_mclk_div,
- &axg_mst_f_mclk_div,
- &axg_mst_a_mclk,
- &axg_mst_b_mclk,
- &axg_mst_c_mclk,
- &axg_mst_d_mclk,
- &axg_mst_e_mclk,
- &axg_mst_f_mclk,
- &axg_spdifout_clk_sel,
- &axg_spdifout_clk_div,
- &axg_spdifout_clk,
- &axg_spdifin_clk_sel,
- &axg_spdifin_clk_div,
- &axg_spdifin_clk,
- &axg_pdm_dclk_sel,
- &axg_pdm_dclk_div,
- &axg_pdm_dclk,
- &axg_pdm_sysclk_sel,
- &axg_pdm_sysclk_div,
- &axg_pdm_sysclk,
- &axg_mst_a_sclk_pre_en,
- &axg_mst_b_sclk_pre_en,
- &axg_mst_c_sclk_pre_en,
- &axg_mst_d_sclk_pre_en,
- &axg_mst_e_sclk_pre_en,
- &axg_mst_f_sclk_pre_en,
- &axg_mst_a_sclk_div,
- &axg_mst_b_sclk_div,
- &axg_mst_c_sclk_div,
- &axg_mst_d_sclk_div,
- &axg_mst_e_sclk_div,
- &axg_mst_f_sclk_div,
- &axg_mst_a_sclk_post_en,
- &axg_mst_b_sclk_post_en,
- &axg_mst_c_sclk_post_en,
- &axg_mst_d_sclk_post_en,
- &axg_mst_e_sclk_post_en,
- &axg_mst_f_sclk_post_en,
- &axg_mst_a_sclk,
- &axg_mst_b_sclk,
- &axg_mst_c_sclk,
- &axg_mst_d_sclk,
- &axg_mst_e_sclk,
- &axg_mst_f_sclk,
- &axg_mst_a_lrclk_div,
- &axg_mst_b_lrclk_div,
- &axg_mst_c_lrclk_div,
- &axg_mst_d_lrclk_div,
- &axg_mst_e_lrclk_div,
- &axg_mst_f_lrclk_div,
- &axg_mst_a_lrclk,
- &axg_mst_b_lrclk,
- &axg_mst_c_lrclk,
- &axg_mst_d_lrclk,
- &axg_mst_e_lrclk,
- &axg_mst_f_lrclk,
- &axg_tdmin_a_sclk_sel,
- &axg_tdmin_b_sclk_sel,
- &axg_tdmin_c_sclk_sel,
- &axg_tdmin_lb_sclk_sel,
- &axg_tdmout_a_sclk_sel,
- &axg_tdmout_b_sclk_sel,
- &axg_tdmout_c_sclk_sel,
- &axg_tdmin_a_sclk_pre_en,
- &axg_tdmin_b_sclk_pre_en,
- &axg_tdmin_c_sclk_pre_en,
- &axg_tdmin_lb_sclk_pre_en,
- &axg_tdmout_a_sclk_pre_en,
- &axg_tdmout_b_sclk_pre_en,
- &axg_tdmout_c_sclk_pre_en,
- &axg_tdmin_a_sclk_post_en,
- &axg_tdmin_b_sclk_post_en,
- &axg_tdmin_c_sclk_post_en,
- &axg_tdmin_lb_sclk_post_en,
- &axg_tdmout_a_sclk_post_en,
- &axg_tdmout_b_sclk_post_en,
- &axg_tdmout_c_sclk_post_en,
- &axg_tdmin_a_sclk,
- &axg_tdmin_b_sclk,
- &axg_tdmin_c_sclk,
- &axg_tdmin_lb_sclk,
- &axg_tdmout_a_sclk,
- &axg_tdmout_b_sclk,
- &axg_tdmout_c_sclk,
- &axg_tdmin_a_lrclk,
- &axg_tdmin_b_lrclk,
- &axg_tdmin_c_lrclk,
- &axg_tdmin_lb_lrclk,
- &axg_tdmout_a_lrclk,
- &axg_tdmout_b_lrclk,
- &axg_tdmout_c_lrclk,
+/*
+ * Array of all G12A clocks provided by this provider
+ * The input clocks of the controller will be populated at runtime
+ */
+static struct clk_hw_onecell_data g12a_audio_hw_onecell_data = {
+ .hws = {
+ [AUD_CLKID_DDR_ARB] = &aud_ddr_arb.hw,
+ [AUD_CLKID_PDM] = &aud_pdm.hw,
+ [AUD_CLKID_TDMIN_A] = &aud_tdmin_a.hw,
+ [AUD_CLKID_TDMIN_B] = &aud_tdmin_b.hw,
+ [AUD_CLKID_TDMIN_C] = &aud_tdmin_c.hw,
+ [AUD_CLKID_TDMIN_LB] = &aud_tdmin_lb.hw,
+ [AUD_CLKID_TDMOUT_A] = &aud_tdmout_a.hw,
+ [AUD_CLKID_TDMOUT_B] = &aud_tdmout_b.hw,
+ [AUD_CLKID_TDMOUT_C] = &aud_tdmout_c.hw,
+ [AUD_CLKID_FRDDR_A] = &aud_frddr_a.hw,
+ [AUD_CLKID_FRDDR_B] = &aud_frddr_b.hw,
+ [AUD_CLKID_FRDDR_C] = &aud_frddr_c.hw,
+ [AUD_CLKID_TODDR_A] = &aud_toddr_a.hw,
+ [AUD_CLKID_TODDR_B] = &aud_toddr_b.hw,
+ [AUD_CLKID_TODDR_C] = &aud_toddr_c.hw,
+ [AUD_CLKID_LOOPBACK] = &aud_loopback.hw,
+ [AUD_CLKID_SPDIFIN] = &aud_spdifin.hw,
+ [AUD_CLKID_SPDIFOUT] = &aud_spdifout.hw,
+ [AUD_CLKID_RESAMPLE] = &aud_resample.hw,
+ [AUD_CLKID_POWER_DETECT] = &aud_power_detect.hw,
+ [AUD_CLKID_SPDIFOUT_B] = &aud_spdifout_b.hw,
+ [AUD_CLKID_MST_A_MCLK_SEL] = &aud_mst_a_mclk_sel.hw,
+ [AUD_CLKID_MST_B_MCLK_SEL] = &aud_mst_b_mclk_sel.hw,
+ [AUD_CLKID_MST_C_MCLK_SEL] = &aud_mst_c_mclk_sel.hw,
+ [AUD_CLKID_MST_D_MCLK_SEL] = &aud_mst_d_mclk_sel.hw,
+ [AUD_CLKID_MST_E_MCLK_SEL] = &aud_mst_e_mclk_sel.hw,
+ [AUD_CLKID_MST_F_MCLK_SEL] = &aud_mst_f_mclk_sel.hw,
+ [AUD_CLKID_MST_A_MCLK_DIV] = &aud_mst_a_mclk_div.hw,
+ [AUD_CLKID_MST_B_MCLK_DIV] = &aud_mst_b_mclk_div.hw,
+ [AUD_CLKID_MST_C_MCLK_DIV] = &aud_mst_c_mclk_div.hw,
+ [AUD_CLKID_MST_D_MCLK_DIV] = &aud_mst_d_mclk_div.hw,
+ [AUD_CLKID_MST_E_MCLK_DIV] = &aud_mst_e_mclk_div.hw,
+ [AUD_CLKID_MST_F_MCLK_DIV] = &aud_mst_f_mclk_div.hw,
+ [AUD_CLKID_MST_A_MCLK] = &aud_mst_a_mclk.hw,
+ [AUD_CLKID_MST_B_MCLK] = &aud_mst_b_mclk.hw,
+ [AUD_CLKID_MST_C_MCLK] = &aud_mst_c_mclk.hw,
+ [AUD_CLKID_MST_D_MCLK] = &aud_mst_d_mclk.hw,
+ [AUD_CLKID_MST_E_MCLK] = &aud_mst_e_mclk.hw,
+ [AUD_CLKID_MST_F_MCLK] = &aud_mst_f_mclk.hw,
+ [AUD_CLKID_SPDIFOUT_CLK_SEL] = &aud_spdifout_clk_sel.hw,
+ [AUD_CLKID_SPDIFOUT_CLK_DIV] = &aud_spdifout_clk_div.hw,
+ [AUD_CLKID_SPDIFOUT_CLK] = &aud_spdifout_clk.hw,
+ [AUD_CLKID_SPDIFOUT_B_CLK_SEL] = &aud_spdifout_b_clk_sel.hw,
+ [AUD_CLKID_SPDIFOUT_B_CLK_DIV] = &aud_spdifout_b_clk_div.hw,
+ [AUD_CLKID_SPDIFOUT_B_CLK] = &aud_spdifout_b_clk.hw,
+ [AUD_CLKID_SPDIFIN_CLK_SEL] = &aud_spdifin_clk_sel.hw,
+ [AUD_CLKID_SPDIFIN_CLK_DIV] = &aud_spdifin_clk_div.hw,
+ [AUD_CLKID_SPDIFIN_CLK] = &aud_spdifin_clk.hw,
+ [AUD_CLKID_PDM_DCLK_SEL] = &aud_pdm_dclk_sel.hw,
+ [AUD_CLKID_PDM_DCLK_DIV] = &aud_pdm_dclk_div.hw,
+ [AUD_CLKID_PDM_DCLK] = &aud_pdm_dclk.hw,
+ [AUD_CLKID_PDM_SYSCLK_SEL] = &aud_pdm_sysclk_sel.hw,
+ [AUD_CLKID_PDM_SYSCLK_DIV] = &aud_pdm_sysclk_div.hw,
+ [AUD_CLKID_PDM_SYSCLK] = &aud_pdm_sysclk.hw,
+ [AUD_CLKID_MST_A_SCLK_PRE_EN] = &aud_mst_a_sclk_pre_en.hw,
+ [AUD_CLKID_MST_B_SCLK_PRE_EN] = &aud_mst_b_sclk_pre_en.hw,
+ [AUD_CLKID_MST_C_SCLK_PRE_EN] = &aud_mst_c_sclk_pre_en.hw,
+ [AUD_CLKID_MST_D_SCLK_PRE_EN] = &aud_mst_d_sclk_pre_en.hw,
+ [AUD_CLKID_MST_E_SCLK_PRE_EN] = &aud_mst_e_sclk_pre_en.hw,
+ [AUD_CLKID_MST_F_SCLK_PRE_EN] = &aud_mst_f_sclk_pre_en.hw,
+ [AUD_CLKID_MST_A_SCLK_DIV] = &aud_mst_a_sclk_div.hw,
+ [AUD_CLKID_MST_B_SCLK_DIV] = &aud_mst_b_sclk_div.hw,
+ [AUD_CLKID_MST_C_SCLK_DIV] = &aud_mst_c_sclk_div.hw,
+ [AUD_CLKID_MST_D_SCLK_DIV] = &aud_mst_d_sclk_div.hw,
+ [AUD_CLKID_MST_E_SCLK_DIV] = &aud_mst_e_sclk_div.hw,
+ [AUD_CLKID_MST_F_SCLK_DIV] = &aud_mst_f_sclk_div.hw,
+ [AUD_CLKID_MST_A_SCLK_POST_EN] = &aud_mst_a_sclk_post_en.hw,
+ [AUD_CLKID_MST_B_SCLK_POST_EN] = &aud_mst_b_sclk_post_en.hw,
+ [AUD_CLKID_MST_C_SCLK_POST_EN] = &aud_mst_c_sclk_post_en.hw,
+ [AUD_CLKID_MST_D_SCLK_POST_EN] = &aud_mst_d_sclk_post_en.hw,
+ [AUD_CLKID_MST_E_SCLK_POST_EN] = &aud_mst_e_sclk_post_en.hw,
+ [AUD_CLKID_MST_F_SCLK_POST_EN] = &aud_mst_f_sclk_post_en.hw,
+ [AUD_CLKID_MST_A_SCLK] = &aud_mst_a_sclk.hw,
+ [AUD_CLKID_MST_B_SCLK] = &aud_mst_b_sclk.hw,
+ [AUD_CLKID_MST_C_SCLK] = &aud_mst_c_sclk.hw,
+ [AUD_CLKID_MST_D_SCLK] = &aud_mst_d_sclk.hw,
+ [AUD_CLKID_MST_E_SCLK] = &aud_mst_e_sclk.hw,
+ [AUD_CLKID_MST_F_SCLK] = &aud_mst_f_sclk.hw,
+ [AUD_CLKID_MST_A_LRCLK_DIV] = &aud_mst_a_lrclk_div.hw,
+ [AUD_CLKID_MST_B_LRCLK_DIV] = &aud_mst_b_lrclk_div.hw,
+ [AUD_CLKID_MST_C_LRCLK_DIV] = &aud_mst_c_lrclk_div.hw,
+ [AUD_CLKID_MST_D_LRCLK_DIV] = &aud_mst_d_lrclk_div.hw,
+ [AUD_CLKID_MST_E_LRCLK_DIV] = &aud_mst_e_lrclk_div.hw,
+ [AUD_CLKID_MST_F_LRCLK_DIV] = &aud_mst_f_lrclk_div.hw,
+ [AUD_CLKID_MST_A_LRCLK] = &aud_mst_a_lrclk.hw,
+ [AUD_CLKID_MST_B_LRCLK] = &aud_mst_b_lrclk.hw,
+ [AUD_CLKID_MST_C_LRCLK] = &aud_mst_c_lrclk.hw,
+ [AUD_CLKID_MST_D_LRCLK] = &aud_mst_d_lrclk.hw,
+ [AUD_CLKID_MST_E_LRCLK] = &aud_mst_e_lrclk.hw,
+ [AUD_CLKID_MST_F_LRCLK] = &aud_mst_f_lrclk.hw,
+ [AUD_CLKID_TDMIN_A_SCLK_SEL] = &aud_tdmin_a_sclk_sel.hw,
+ [AUD_CLKID_TDMIN_B_SCLK_SEL] = &aud_tdmin_b_sclk_sel.hw,
+ [AUD_CLKID_TDMIN_C_SCLK_SEL] = &aud_tdmin_c_sclk_sel.hw,
+ [AUD_CLKID_TDMIN_LB_SCLK_SEL] = &aud_tdmin_lb_sclk_sel.hw,
+ [AUD_CLKID_TDMOUT_A_SCLK_SEL] = &aud_tdmout_a_sclk_sel.hw,
+ [AUD_CLKID_TDMOUT_B_SCLK_SEL] = &aud_tdmout_b_sclk_sel.hw,
+ [AUD_CLKID_TDMOUT_C_SCLK_SEL] = &aud_tdmout_c_sclk_sel.hw,
+ [AUD_CLKID_TDMIN_A_SCLK_PRE_EN] = &aud_tdmin_a_sclk_pre_en.hw,
+ [AUD_CLKID_TDMIN_B_SCLK_PRE_EN] = &aud_tdmin_b_sclk_pre_en.hw,
+ [AUD_CLKID_TDMIN_C_SCLK_PRE_EN] = &aud_tdmin_c_sclk_pre_en.hw,
+ [AUD_CLKID_TDMIN_LB_SCLK_PRE_EN] = &aud_tdmin_lb_sclk_pre_en.hw,
+ [AUD_CLKID_TDMOUT_A_SCLK_PRE_EN] = &aud_tdmout_a_sclk_pre_en.hw,
+ [AUD_CLKID_TDMOUT_B_SCLK_PRE_EN] = &aud_tdmout_b_sclk_pre_en.hw,
+ [AUD_CLKID_TDMOUT_C_SCLK_PRE_EN] = &aud_tdmout_c_sclk_pre_en.hw,
+ [AUD_CLKID_TDMIN_A_SCLK_POST_EN] = &aud_tdmin_a_sclk_post_en.hw,
+ [AUD_CLKID_TDMIN_B_SCLK_POST_EN] = &aud_tdmin_b_sclk_post_en.hw,
+ [AUD_CLKID_TDMIN_C_SCLK_POST_EN] = &aud_tdmin_c_sclk_post_en.hw,
+ [AUD_CLKID_TDMIN_LB_SCLK_POST_EN] = &aud_tdmin_lb_sclk_post_en.hw,
+ [AUD_CLKID_TDMOUT_A_SCLK_POST_EN] = &aud_tdmout_a_sclk_post_en.hw,
+ [AUD_CLKID_TDMOUT_B_SCLK_POST_EN] = &aud_tdmout_b_sclk_post_en.hw,
+ [AUD_CLKID_TDMOUT_C_SCLK_POST_EN] = &aud_tdmout_c_sclk_post_en.hw,
+ [AUD_CLKID_TDMIN_A_SCLK] = &aud_tdmin_a_sclk.hw,
+ [AUD_CLKID_TDMIN_B_SCLK] = &aud_tdmin_b_sclk.hw,
+ [AUD_CLKID_TDMIN_C_SCLK] = &aud_tdmin_c_sclk.hw,
+ [AUD_CLKID_TDMIN_LB_SCLK] = &aud_tdmin_lb_sclk.hw,
+ [AUD_CLKID_TDMOUT_A_SCLK] = &aud_tdmout_a_sclk.hw,
+ [AUD_CLKID_TDMOUT_B_SCLK] = &aud_tdmout_b_sclk.hw,
+ [AUD_CLKID_TDMOUT_C_SCLK] = &aud_tdmout_c_sclk.hw,
+ [AUD_CLKID_TDMIN_A_LRCLK] = &aud_tdmin_a_lrclk.hw,
+ [AUD_CLKID_TDMIN_B_LRCLK] = &aud_tdmin_b_lrclk.hw,
+ [AUD_CLKID_TDMIN_C_LRCLK] = &aud_tdmin_c_lrclk.hw,
+ [AUD_CLKID_TDMIN_LB_LRCLK] = &aud_tdmin_lb_lrclk.hw,
+ [AUD_CLKID_TDMOUT_A_LRCLK] = &aud_tdmout_a_lrclk.hw,
+ [AUD_CLKID_TDMOUT_B_LRCLK] = &aud_tdmout_b_lrclk.hw,
+ [AUD_CLKID_TDMOUT_C_LRCLK] = &aud_tdmout_c_lrclk.hw,
+ [AUD_CLKID_TDM_MCLK_PAD0] = &aud_tdm_mclk_pad_0.hw,
+ [AUD_CLKID_TDM_MCLK_PAD1] = &aud_tdm_mclk_pad_1.hw,
+ [AUD_CLKID_TDM_LRCLK_PAD0] = &aud_tdm_lrclk_pad_0.hw,
+ [AUD_CLKID_TDM_LRCLK_PAD1] = &aud_tdm_lrclk_pad_1.hw,
+ [AUD_CLKID_TDM_LRCLK_PAD2] = &aud_tdm_lrclk_pad_2.hw,
+ [AUD_CLKID_TDM_SCLK_PAD0] = &aud_tdm_sclk_pad_0.hw,
+ [AUD_CLKID_TDM_SCLK_PAD1] = &aud_tdm_sclk_pad_1.hw,
+ [AUD_CLKID_TDM_SCLK_PAD2] = &aud_tdm_sclk_pad_2.hw,
+ [NR_CLKS] = NULL,
+ },
+ .num = NR_CLKS,
+};
+
+/* Convenience table to populate regmap in .probe()
+ * Note that this table is shared between both AXG and G12A,
+ * with spdifout_b clocks being exclusive to G12A. Since those
+ * clocks are not declared within the AXG onecell table, we do not
+ * feel the need to have separate AXG/G12A regmap tables.
+ */
+static struct clk_regmap *const aud_clk_regmaps[] = {
+ &aud_ddr_arb,
+ &aud_pdm,
+ &aud_tdmin_a,
+ &aud_tdmin_b,
+ &aud_tdmin_c,
+ &aud_tdmin_lb,
+ &aud_tdmout_a,
+ &aud_tdmout_b,
+ &aud_tdmout_c,
+ &aud_frddr_a,
+ &aud_frddr_b,
+ &aud_frddr_c,
+ &aud_toddr_a,
+ &aud_toddr_b,
+ &aud_toddr_c,
+ &aud_loopback,
+ &aud_spdifin,
+ &aud_spdifout,
+ &aud_resample,
+ &aud_power_detect,
+ &aud_spdifout_b,
+ &aud_mst_a_mclk_sel,
+ &aud_mst_b_mclk_sel,
+ &aud_mst_c_mclk_sel,
+ &aud_mst_d_mclk_sel,
+ &aud_mst_e_mclk_sel,
+ &aud_mst_f_mclk_sel,
+ &aud_mst_a_mclk_div,
+ &aud_mst_b_mclk_div,
+ &aud_mst_c_mclk_div,
+ &aud_mst_d_mclk_div,
+ &aud_mst_e_mclk_div,
+ &aud_mst_f_mclk_div,
+ &aud_mst_a_mclk,
+ &aud_mst_b_mclk,
+ &aud_mst_c_mclk,
+ &aud_mst_d_mclk,
+ &aud_mst_e_mclk,
+ &aud_mst_f_mclk,
+ &aud_spdifout_clk_sel,
+ &aud_spdifout_clk_div,
+ &aud_spdifout_clk,
+ &aud_spdifin_clk_sel,
+ &aud_spdifin_clk_div,
+ &aud_spdifin_clk,
+ &aud_pdm_dclk_sel,
+ &aud_pdm_dclk_div,
+ &aud_pdm_dclk,
+ &aud_pdm_sysclk_sel,
+ &aud_pdm_sysclk_div,
+ &aud_pdm_sysclk,
+ &aud_mst_a_sclk_pre_en,
+ &aud_mst_b_sclk_pre_en,
+ &aud_mst_c_sclk_pre_en,
+ &aud_mst_d_sclk_pre_en,
+ &aud_mst_e_sclk_pre_en,
+ &aud_mst_f_sclk_pre_en,
+ &aud_mst_a_sclk_div,
+ &aud_mst_b_sclk_div,
+ &aud_mst_c_sclk_div,
+ &aud_mst_d_sclk_div,
+ &aud_mst_e_sclk_div,
+ &aud_mst_f_sclk_div,
+ &aud_mst_a_sclk_post_en,
+ &aud_mst_b_sclk_post_en,
+ &aud_mst_c_sclk_post_en,
+ &aud_mst_d_sclk_post_en,
+ &aud_mst_e_sclk_post_en,
+ &aud_mst_f_sclk_post_en,
+ &aud_mst_a_sclk,
+ &aud_mst_b_sclk,
+ &aud_mst_c_sclk,
+ &aud_mst_d_sclk,
+ &aud_mst_e_sclk,
+ &aud_mst_f_sclk,
+ &aud_mst_a_lrclk_div,
+ &aud_mst_b_lrclk_div,
+ &aud_mst_c_lrclk_div,
+ &aud_mst_d_lrclk_div,
+ &aud_mst_e_lrclk_div,
+ &aud_mst_f_lrclk_div,
+ &aud_mst_a_lrclk,
+ &aud_mst_b_lrclk,
+ &aud_mst_c_lrclk,
+ &aud_mst_d_lrclk,
+ &aud_mst_e_lrclk,
+ &aud_mst_f_lrclk,
+ &aud_tdmin_a_sclk_sel,
+ &aud_tdmin_b_sclk_sel,
+ &aud_tdmin_c_sclk_sel,
+ &aud_tdmin_lb_sclk_sel,
+ &aud_tdmout_a_sclk_sel,
+ &aud_tdmout_b_sclk_sel,
+ &aud_tdmout_c_sclk_sel,
+ &aud_tdmin_a_sclk_pre_en,
+ &aud_tdmin_b_sclk_pre_en,
+ &aud_tdmin_c_sclk_pre_en,
+ &aud_tdmin_lb_sclk_pre_en,
+ &aud_tdmout_a_sclk_pre_en,
+ &aud_tdmout_b_sclk_pre_en,
+ &aud_tdmout_c_sclk_pre_en,
+ &aud_tdmin_a_sclk_post_en,
+ &aud_tdmin_b_sclk_post_en,
+ &aud_tdmin_c_sclk_post_en,
+ &aud_tdmin_lb_sclk_post_en,
+ &aud_tdmout_a_sclk_post_en,
+ &aud_tdmout_b_sclk_post_en,
+ &aud_tdmout_c_sclk_post_en,
+ &aud_tdmin_a_sclk,
+ &aud_tdmin_b_sclk,
+ &aud_tdmin_c_sclk,
+ &aud_tdmin_lb_sclk,
+ &aud_tdmout_a_sclk,
+ &aud_tdmout_b_sclk,
+ &aud_tdmout_c_sclk,
+ &aud_tdmin_a_lrclk,
+ &aud_tdmin_b_lrclk,
+ &aud_tdmin_c_lrclk,
+ &aud_tdmin_lb_lrclk,
+ &aud_tdmout_a_lrclk,
+ &aud_tdmout_b_lrclk,
+ &aud_tdmout_c_lrclk,
+ &aud_spdifout_b_clk_sel,
+ &aud_spdifout_b_clk_div,
+ &aud_spdifout_b_clk,
+ &aud_tdm_mclk_pad_0,
+ &aud_tdm_mclk_pad_1,
+ &aud_tdm_lrclk_pad_0,
+ &aud_tdm_lrclk_pad_1,
+ &aud_tdm_lrclk_pad_2,
+ &aud_tdm_sclk_pad_0,
+ &aud_tdm_sclk_pad_1,
+ &aud_tdm_sclk_pad_2,
};
static int devm_clk_get_enable(struct device *dev, char *id)
}
static int axg_register_clk_hw_input(struct device *dev,
- const char *name,
- unsigned int clkid)
+ const char *name)
{
char *clk_name;
struct clk_hw *hw;
int err = 0;
- clk_name = kasprintf(GFP_KERNEL, "axg_%s", name);
+ clk_name = kasprintf(GFP_KERNEL, "aud_%s", name);
if (!clk_name)
return -ENOMEM;
if (err != -EPROBE_DEFER)
dev_err(dev, "failed to get %s clock", name);
}
- } else {
- axg_audio_hw_onecell_data.hws[clkid] = hw;
}
kfree(clk_name);
static int axg_register_clk_hw_inputs(struct device *dev,
const char *basename,
- unsigned int count,
- unsigned int clkid)
+ unsigned int count)
{
char *name;
int i, ret;
if (!name)
return -ENOMEM;
- ret = axg_register_clk_hw_input(dev, name, clkid + i);
+ ret = axg_register_clk_hw_input(dev, name);
kfree(name);
if (ret)
return ret;
.max_register = AUDIO_CLK_PDMIN_CTRL1,
};
+struct audioclk_data {
+ struct clk_hw_onecell_data *hw_onecell_data;
+};
+
static int axg_audio_clkc_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
+ const struct audioclk_data *data;
struct regmap *map;
struct resource *res;
void __iomem *regs;
struct clk_hw *hw;
int ret, i;
+ data = of_device_get_match_data(dev);
+ if (!data)
+ return -EINVAL;
+
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
regs = devm_ioremap_resource(dev, res);
if (IS_ERR(regs))
}
/* Register the peripheral input clock */
- hw = meson_clk_hw_register_input(dev, "pclk", "axg_audio_pclk", 0);
+ hw = meson_clk_hw_register_input(dev, "pclk", "audio_pclk", 0);
if (IS_ERR(hw))
return PTR_ERR(hw);
- axg_audio_hw_onecell_data.hws[AUD_CLKID_PCLK] = hw;
-
/* Register optional input master clocks */
ret = axg_register_clk_hw_inputs(dev, "mst_in",
- AXG_MST_IN_COUNT,
- AUD_CLKID_MST0);
+ AUD_MST_IN_COUNT);
if (ret)
return ret;
/* Register optional input slave sclks */
ret = axg_register_clk_hw_inputs(dev, "slv_sclk",
- AXG_SLV_SCLK_COUNT,
- AUD_CLKID_SLV_SCLK0);
+ AUD_SLV_SCLK_COUNT);
if (ret)
return ret;
/* Register optional input slave lrclks */
ret = axg_register_clk_hw_inputs(dev, "slv_lrclk",
- AXG_SLV_LRCLK_COUNT,
- AUD_CLKID_SLV_LRCLK0);
+ AUD_SLV_LRCLK_COUNT);
if (ret)
return ret;
/* Populate regmap for the regmap backed clocks */
- for (i = 0; i < ARRAY_SIZE(axg_audio_clk_regmaps); i++)
- axg_audio_clk_regmaps[i]->map = map;
+ for (i = 0; i < ARRAY_SIZE(aud_clk_regmaps); i++)
+ aud_clk_regmaps[i]->map = map;
/* Take care to skip the registered input clocks */
- for (i = AUD_CLKID_DDR_ARB; i < axg_audio_hw_onecell_data.num; i++) {
- hw = axg_audio_hw_onecell_data.hws[i];
+ for (i = AUD_CLKID_DDR_ARB; i < data->hw_onecell_data->num; i++) {
+ hw = data->hw_onecell_data->hws[i];
/* array might be sparse */
if (!hw)
continue;
}
return devm_of_clk_add_hw_provider(dev, of_clk_hw_onecell_get,
- &axg_audio_hw_onecell_data);
+ data->hw_onecell_data);
}
+static const struct audioclk_data axg_audioclk_data = {
+ .hw_onecell_data = &axg_audio_hw_onecell_data,
+};
+
+static const struct audioclk_data g12a_audioclk_data = {
+ .hw_onecell_data = &g12a_audio_hw_onecell_data,
+};
+
static const struct of_device_id clkc_match_table[] = {
- { .compatible = "amlogic,axg-audio-clkc" },
- {}
+ {
+ .compatible = "amlogic,axg-audio-clkc",
+ .data = &axg_audioclk_data
+ }, {
+ .compatible = "amlogic,g12a-audio-clkc",
+ .data = &g12a_audioclk_data
+ }, {}
};
MODULE_DEVICE_TABLE(of, clkc_match_table);
};
module_platform_driver(axg_audio_driver);
-MODULE_DESCRIPTION("Amlogic A113x Audio Clock driver");
+MODULE_DESCRIPTION("Amlogic AXG/G12A Audio Clock driver");
MODULE_AUTHOR("Jerome Brunet <jbrunet@baylibre.com>");
MODULE_LICENSE("GPL v2");
#define AUDIO_MCLK_D_CTRL 0x010
#define AUDIO_MCLK_E_CTRL 0x014
#define AUDIO_MCLK_F_CTRL 0x018
+#define AUDIO_MST_PAD_CTRL0 0x01c
+#define AUDIO_MST_PAD_CTRL1 0x020
#define AUDIO_MST_A_SCLK_CTRL0 0x040
#define AUDIO_MST_A_SCLK_CTRL1 0x044
#define AUDIO_MST_B_SCLK_CTRL0 0x048
#define AUDIO_CLK_LOCKER_CTRL 0x0A8
#define AUDIO_CLK_PDMIN_CTRL0 0x0AC
#define AUDIO_CLK_PDMIN_CTRL1 0x0B0
+#define AUDIO_CLK_SPDIFOUT_B_CTRL 0x0B4
/*
* CLKID index values
* These indices are entirely contrived and do not map onto the hardware.
*/
-#define AUD_CLKID_PCLK 0
-#define AUD_CLKID_MST0 1
-#define AUD_CLKID_MST1 2
-#define AUD_CLKID_MST2 3
-#define AUD_CLKID_MST3 4
-#define AUD_CLKID_MST4 5
-#define AUD_CLKID_MST5 6
-#define AUD_CLKID_MST6 7
-#define AUD_CLKID_MST7 8
#define AUD_CLKID_MST_A_MCLK_SEL 59
#define AUD_CLKID_MST_B_MCLK_SEL 60
#define AUD_CLKID_MST_C_MCLK_SEL 61
#define AUD_CLKID_TDMOUT_A_SCLK_POST_EN 148
#define AUD_CLKID_TDMOUT_B_SCLK_POST_EN 149
#define AUD_CLKID_TDMOUT_C_SCLK_POST_EN 150
+#define AUD_CLKID_SPDIFOUT_B_CLK_SEL 153
+#define AUD_CLKID_SPDIFOUT_B_CLK_DIV 154
/* include the CLKIDs which are part of the DT bindings */
#include <dt-bindings/clock/axg-audio-clkc.h>
-#define NR_CLKS 151
+#define NR_CLKS 163
#endif /*__AXG_AUDIO_CLKC_H */
return 1;
}
+static int meson_clk_pcie_pll_enable(struct clk_hw *hw)
+{
+ meson_clk_pll_init(hw);
+
+ if (meson_clk_pll_wait_lock(hw))
+ return -EIO;
+
+ return 0;
+}
+
static int meson_clk_pll_enable(struct clk_hw *hw)
{
struct clk_regmap *clk = to_clk_regmap(hw);
return 0;
}
+/*
+ * The Meson G12A PCIE PLL is fined tuned to deliver a very precise
+ * 100MHz reference clock for the PCIe Analog PHY, and thus requires
+ * a strict register sequence to enable the PLL.
+ * To simplify, re-use the _init() op to enable the PLL and keep
+ * the other ops except set_rate since the rate is fixed.
+ */
+const struct clk_ops meson_clk_pcie_pll_ops = {
+ .recalc_rate = meson_clk_pll_recalc_rate,
+ .round_rate = meson_clk_pll_round_rate,
+ .is_enabled = meson_clk_pll_is_enabled,
+ .enable = meson_clk_pcie_pll_enable,
+ .disable = meson_clk_pll_disable
+};
+EXPORT_SYMBOL_GPL(meson_clk_pcie_pll_ops);
+
const struct clk_ops meson_clk_pll_ops = {
.init = meson_clk_pll_init,
.recalc_rate = meson_clk_pll_recalc_rate,
extern const struct clk_ops meson_clk_pll_ro_ops;
extern const struct clk_ops meson_clk_pll_ops;
+extern const struct clk_ops meson_clk_pcie_pll_ops;
#endif /* __MESON_CLK_PLL_H */
* to expose, such as the internal muxes and dividers of composite clocks,
* will remain defined here.
*/
-#define CLKID_AO_SAR_ADC_SEL 16
#define CLKID_AO_SAR_ADC_DIV 17
-#define CLKID_AO_CTS_OSCIN 19
#define CLKID_AO_32K_PRE 20
#define CLKID_AO_32K_DIV 21
#define CLKID_AO_32K_SEL 22
},
};
+static struct clk_regmap g12a_sys_pll_div16_en = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL1,
+ .bit_idx = 24,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "sys_pll_div16_en",
+ .ops = &clk_regmap_gate_ro_ops,
+ .parent_names = (const char *[]){ "sys_pll" },
+ .num_parents = 1,
+ /*
+ * This clock is used to debug the sys_pll range
+ * Linux should not change it at runtime
+ */
+ },
+};
+
+static struct clk_fixed_factor g12a_sys_pll_div16 = {
+ .mult = 1,
+ .div = 16,
+ .hw.init = &(struct clk_init_data){
+ .name = "sys_pll_div16",
+ .ops = &clk_fixed_factor_ops,
+ .parent_names = (const char *[]){ "sys_pll_div16_en" },
+ .num_parents = 1,
+ },
+};
+
+/* Datasheet names this field as "premux0" */
+static struct clk_regmap g12a_cpu_clk_premux0 = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL0,
+ .mask = 0x3,
+ .shift = 0,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_dyn0_sel",
+ .ops = &clk_regmap_mux_ro_ops,
+ .parent_names = (const char *[]){ IN_PREFIX "xtal",
+ "fclk_div2",
+ "fclk_div3" },
+ .num_parents = 3,
+ },
+};
+
+/* Datasheet names this field as "mux0_divn_tcnt" */
+static struct clk_regmap g12a_cpu_clk_mux0_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL0,
+ .shift = 4,
+ .width = 6,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_dyn0_div",
+ .ops = &clk_regmap_divider_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk_dyn0_sel" },
+ .num_parents = 1,
+ },
+};
+
+/* Datasheet names this field as "postmux0" */
+static struct clk_regmap g12a_cpu_clk_postmux0 = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL0,
+ .mask = 0x1,
+ .shift = 2,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_dyn0",
+ .ops = &clk_regmap_mux_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk_dyn0_sel",
+ "cpu_clk_dyn0_div" },
+ .num_parents = 2,
+ },
+};
+
+/* Datasheet names this field as "premux1" */
+static struct clk_regmap g12a_cpu_clk_premux1 = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL0,
+ .mask = 0x3,
+ .shift = 16,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_dyn1_sel",
+ .ops = &clk_regmap_mux_ro_ops,
+ .parent_names = (const char *[]){ IN_PREFIX "xtal",
+ "fclk_div2",
+ "fclk_div3" },
+ .num_parents = 3,
+ },
+};
+
+/* Datasheet names this field as "Mux1_divn_tcnt" */
+static struct clk_regmap g12a_cpu_clk_mux1_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL0,
+ .shift = 20,
+ .width = 6,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_dyn1_div",
+ .ops = &clk_regmap_divider_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk_dyn1_sel" },
+ .num_parents = 1,
+ },
+};
+
+/* Datasheet names this field as "postmux1" */
+static struct clk_regmap g12a_cpu_clk_postmux1 = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL0,
+ .mask = 0x1,
+ .shift = 18,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_dyn1",
+ .ops = &clk_regmap_mux_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk_dyn1_sel",
+ "cpu_clk_dyn1_div" },
+ .num_parents = 2,
+ },
+};
+
+/* Datasheet names this field as "Final_dyn_mux_sel" */
+static struct clk_regmap g12a_cpu_clk_dyn = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL0,
+ .mask = 0x1,
+ .shift = 10,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_dyn",
+ .ops = &clk_regmap_mux_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk_dyn0",
+ "cpu_clk_dyn1" },
+ .num_parents = 2,
+ },
+};
+
+/* Datasheet names this field as "Final_mux_sel" */
+static struct clk_regmap g12a_cpu_clk = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL0,
+ .mask = 0x1,
+ .shift = 11,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk",
+ .ops = &clk_regmap_mux_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk_dyn",
+ "sys_pll" },
+ .num_parents = 2,
+ },
+};
+
+static struct clk_regmap g12a_cpu_clk_div16_en = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL1,
+ .bit_idx = 1,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "cpu_clk_div16_en",
+ .ops = &clk_regmap_gate_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk" },
+ .num_parents = 1,
+ /*
+ * This clock is used to debug the cpu_clk range
+ * Linux should not change it at runtime
+ */
+ },
+};
+
+static struct clk_fixed_factor g12a_cpu_clk_div16 = {
+ .mult = 1,
+ .div = 16,
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_div16",
+ .ops = &clk_fixed_factor_ops,
+ .parent_names = (const char *[]){ "cpu_clk_div16_en" },
+ .num_parents = 1,
+ },
+};
+
+static struct clk_regmap g12a_cpu_clk_apb_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL1,
+ .shift = 3,
+ .width = 3,
+ .flags = CLK_DIVIDER_POWER_OF_TWO,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_apb_div",
+ .ops = &clk_regmap_divider_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk" },
+ .num_parents = 1,
+ },
+};
+
+static struct clk_regmap g12a_cpu_clk_apb = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL1,
+ .bit_idx = 1,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "cpu_clk_apb",
+ .ops = &clk_regmap_gate_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk_apb_div" },
+ .num_parents = 1,
+ /*
+ * This clock is set by the ROM monitor code,
+ * Linux should not change it at runtime
+ */
+ },
+};
+
+static struct clk_regmap g12a_cpu_clk_atb_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL1,
+ .shift = 6,
+ .width = 3,
+ .flags = CLK_DIVIDER_POWER_OF_TWO,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_atb_div",
+ .ops = &clk_regmap_divider_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk" },
+ .num_parents = 1,
+ },
+};
+
+static struct clk_regmap g12a_cpu_clk_atb = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL1,
+ .bit_idx = 17,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "cpu_clk_atb",
+ .ops = &clk_regmap_gate_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk_atb_div" },
+ .num_parents = 1,
+ /*
+ * This clock is set by the ROM monitor code,
+ * Linux should not change it at runtime
+ */
+ },
+};
+
+static struct clk_regmap g12a_cpu_clk_axi_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL1,
+ .shift = 9,
+ .width = 3,
+ .flags = CLK_DIVIDER_POWER_OF_TWO,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_axi_div",
+ .ops = &clk_regmap_divider_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk" },
+ .num_parents = 1,
+ },
+};
+
+static struct clk_regmap g12a_cpu_clk_axi = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL1,
+ .bit_idx = 18,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "cpu_clk_axi",
+ .ops = &clk_regmap_gate_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk_axi_div" },
+ .num_parents = 1,
+ /*
+ * This clock is set by the ROM monitor code,
+ * Linux should not change it at runtime
+ */
+ },
+};
+
+static struct clk_regmap g12a_cpu_clk_trace_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL1,
+ .shift = 20,
+ .width = 3,
+ .flags = CLK_DIVIDER_POWER_OF_TWO,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "cpu_clk_trace_div",
+ .ops = &clk_regmap_divider_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk" },
+ .num_parents = 1,
+ },
+};
+
+static struct clk_regmap g12a_cpu_clk_trace = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_SYS_CPU_CLK_CNTL1,
+ .bit_idx = 23,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "cpu_clk_trace",
+ .ops = &clk_regmap_gate_ro_ops,
+ .parent_names = (const char *[]){ "cpu_clk_trace_div" },
+ .num_parents = 1,
+ /*
+ * This clock is set by the ROM monitor code,
+ * Linux should not change it at runtime
+ */
+ },
+};
+
static const struct pll_mult_range g12a_gp0_pll_mult_range = {
.min = 55,
.max = 255,
},
};
+/*
+ * The Meson G12A PCIE PLL is fined tuned to deliver a very precise
+ * 100MHz reference clock for the PCIe Analog PHY, and thus requires
+ * a strict register sequence to enable the PLL.
+ */
+static const struct reg_sequence g12a_pcie_pll_init_regs[] = {
+ { .reg = HHI_PCIE_PLL_CNTL0, .def = 0x20090496 },
+ { .reg = HHI_PCIE_PLL_CNTL0, .def = 0x30090496 },
+ { .reg = HHI_PCIE_PLL_CNTL1, .def = 0x00000000 },
+ { .reg = HHI_PCIE_PLL_CNTL2, .def = 0x00001100 },
+ { .reg = HHI_PCIE_PLL_CNTL3, .def = 0x10058e00 },
+ { .reg = HHI_PCIE_PLL_CNTL4, .def = 0x000100c0 },
+ { .reg = HHI_PCIE_PLL_CNTL5, .def = 0x68000048 },
+ { .reg = HHI_PCIE_PLL_CNTL5, .def = 0x68000068, .delay_us = 20 },
+ { .reg = HHI_PCIE_PLL_CNTL4, .def = 0x008100c0, .delay_us = 10 },
+ { .reg = HHI_PCIE_PLL_CNTL0, .def = 0x34090496 },
+ { .reg = HHI_PCIE_PLL_CNTL0, .def = 0x14090496, .delay_us = 10 },
+ { .reg = HHI_PCIE_PLL_CNTL2, .def = 0x00001000 },
+};
+
+/* Keep a single entry table for recalc/round_rate() ops */
+static const struct pll_params_table g12a_pcie_pll_table[] = {
+ PLL_PARAMS(150, 1),
+ {0, 0},
+};
+
+static struct clk_regmap g12a_pcie_pll_dco = {
+ .data = &(struct meson_clk_pll_data){
+ .en = {
+ .reg_off = HHI_PCIE_PLL_CNTL0,
+ .shift = 28,
+ .width = 1,
+ },
+ .m = {
+ .reg_off = HHI_PCIE_PLL_CNTL0,
+ .shift = 0,
+ .width = 8,
+ },
+ .n = {
+ .reg_off = HHI_PCIE_PLL_CNTL0,
+ .shift = 10,
+ .width = 5,
+ },
+ .frac = {
+ .reg_off = HHI_PCIE_PLL_CNTL1,
+ .shift = 0,
+ .width = 12,
+ },
+ .l = {
+ .reg_off = HHI_PCIE_PLL_CNTL0,
+ .shift = 31,
+ .width = 1,
+ },
+ .rst = {
+ .reg_off = HHI_PCIE_PLL_CNTL0,
+ .shift = 29,
+ .width = 1,
+ },
+ .table = g12a_pcie_pll_table,
+ .init_regs = g12a_pcie_pll_init_regs,
+ .init_count = ARRAY_SIZE(g12a_pcie_pll_init_regs),
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "pcie_pll_dco",
+ .ops = &meson_clk_pcie_pll_ops,
+ .parent_names = (const char *[]){ IN_PREFIX "xtal" },
+ .num_parents = 1,
+ },
+};
+
+static struct clk_fixed_factor g12a_pcie_pll_dco_div2 = {
+ .mult = 1,
+ .div = 2,
+ .hw.init = &(struct clk_init_data){
+ .name = "pcie_pll_dco_div2",
+ .ops = &clk_fixed_factor_ops,
+ .parent_names = (const char *[]){ "pcie_pll_dco" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap g12a_pcie_pll_od = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_PCIE_PLL_CNTL0,
+ .shift = 16,
+ .width = 5,
+ .flags = CLK_DIVIDER_ROUND_CLOSEST |
+ CLK_DIVIDER_ONE_BASED |
+ CLK_DIVIDER_ALLOW_ZERO,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "pcie_pll_od",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "pcie_pll_dco_div2" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_fixed_factor g12a_pcie_pll = {
+ .mult = 1,
+ .div = 2,
+ .hw.init = &(struct clk_init_data){
+ .name = "pcie_pll_pll",
+ .ops = &clk_fixed_factor_ops,
+ .parent_names = (const char *[]){ "pcie_pll_od" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
static struct clk_regmap g12a_hdmi_pll_dco = {
.data = &(struct meson_clk_pll_data){
.en = {
},
};
+/* VDEC clocks */
+
+static const char * const g12a_vdec_parent_names[] = {
+ "fclk_div2p5", "fclk_div3", "fclk_div4", "fclk_div5", "fclk_div7",
+ "hifi_pll", "gp0_pll",
+};
+
+static struct clk_regmap g12a_vdec_1_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VDEC_CLK_CNTL,
+ .mask = 0x7,
+ .shift = 9,
+ .flags = CLK_MUX_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_1_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = g12a_vdec_parent_names,
+ .num_parents = ARRAY_SIZE(g12a_vdec_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap g12a_vdec_1_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_VDEC_CLK_CNTL,
+ .shift = 0,
+ .width = 7,
+ .flags = CLK_DIVIDER_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_1_div",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "vdec_1_sel" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap g12a_vdec_1 = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_VDEC_CLK_CNTL,
+ .bit_idx = 8,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "vdec_1",
+ .ops = &clk_regmap_gate_ops,
+ .parent_names = (const char *[]){ "vdec_1_div" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap g12a_vdec_hevcf_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .mask = 0x7,
+ .shift = 9,
+ .flags = CLK_MUX_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_hevcf_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = g12a_vdec_parent_names,
+ .num_parents = ARRAY_SIZE(g12a_vdec_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap g12a_vdec_hevcf_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .shift = 0,
+ .width = 7,
+ .flags = CLK_DIVIDER_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_hevcf_div",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "vdec_hevcf_sel" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap g12a_vdec_hevcf = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .bit_idx = 8,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "vdec_hevcf",
+ .ops = &clk_regmap_gate_ops,
+ .parent_names = (const char *[]){ "vdec_hevcf_div" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap g12a_vdec_hevc_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .mask = 0x7,
+ .shift = 25,
+ .flags = CLK_MUX_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_hevc_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = g12a_vdec_parent_names,
+ .num_parents = ARRAY_SIZE(g12a_vdec_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap g12a_vdec_hevc_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .shift = 16,
+ .width = 7,
+ .flags = CLK_DIVIDER_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_hevc_div",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "vdec_hevc_sel" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap g12a_vdec_hevc = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .bit_idx = 24,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "vdec_hevc",
+ .ops = &clk_regmap_gate_ops,
+ .parent_names = (const char *[]){ "vdec_hevc_div" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
/* VAPB Clock */
static const char * const g12a_vapb_parent_names[] = {
[CLKID_MALI] = &g12a_mali.hw,
[CLKID_MPLL_5OM_DIV] = &g12a_mpll_50m_div.hw,
[CLKID_MPLL_5OM] = &g12a_mpll_50m.hw,
+ [CLKID_SYS_PLL_DIV16_EN] = &g12a_sys_pll_div16_en.hw,
+ [CLKID_SYS_PLL_DIV16] = &g12a_sys_pll_div16.hw,
+ [CLKID_CPU_CLK_DYN0_SEL] = &g12a_cpu_clk_premux0.hw,
+ [CLKID_CPU_CLK_DYN0_DIV] = &g12a_cpu_clk_mux0_div.hw,
+ [CLKID_CPU_CLK_DYN0] = &g12a_cpu_clk_postmux0.hw,
+ [CLKID_CPU_CLK_DYN1_SEL] = &g12a_cpu_clk_premux1.hw,
+ [CLKID_CPU_CLK_DYN1_DIV] = &g12a_cpu_clk_mux1_div.hw,
+ [CLKID_CPU_CLK_DYN1] = &g12a_cpu_clk_postmux1.hw,
+ [CLKID_CPU_CLK_DYN] = &g12a_cpu_clk_dyn.hw,
+ [CLKID_CPU_CLK] = &g12a_cpu_clk.hw,
+ [CLKID_CPU_CLK_DIV16_EN] = &g12a_cpu_clk_div16_en.hw,
+ [CLKID_CPU_CLK_DIV16] = &g12a_cpu_clk_div16.hw,
+ [CLKID_CPU_CLK_APB_DIV] = &g12a_cpu_clk_apb_div.hw,
+ [CLKID_CPU_CLK_APB] = &g12a_cpu_clk_apb.hw,
+ [CLKID_CPU_CLK_ATB_DIV] = &g12a_cpu_clk_atb_div.hw,
+ [CLKID_CPU_CLK_ATB] = &g12a_cpu_clk_atb.hw,
+ [CLKID_CPU_CLK_AXI_DIV] = &g12a_cpu_clk_axi_div.hw,
+ [CLKID_CPU_CLK_AXI] = &g12a_cpu_clk_axi.hw,
+ [CLKID_CPU_CLK_TRACE_DIV] = &g12a_cpu_clk_trace_div.hw,
+ [CLKID_CPU_CLK_TRACE] = &g12a_cpu_clk_trace.hw,
+ [CLKID_PCIE_PLL_DCO] = &g12a_pcie_pll_dco.hw,
+ [CLKID_PCIE_PLL_DCO_DIV2] = &g12a_pcie_pll_dco_div2.hw,
+ [CLKID_PCIE_PLL_OD] = &g12a_pcie_pll_od.hw,
+ [CLKID_PCIE_PLL] = &g12a_pcie_pll.hw,
+ [CLKID_VDEC_1_SEL] = &g12a_vdec_1_sel.hw,
+ [CLKID_VDEC_1_DIV] = &g12a_vdec_1_div.hw,
+ [CLKID_VDEC_1] = &g12a_vdec_1.hw,
+ [CLKID_VDEC_HEVC_SEL] = &g12a_vdec_hevc_sel.hw,
+ [CLKID_VDEC_HEVC_DIV] = &g12a_vdec_hevc_div.hw,
+ [CLKID_VDEC_HEVC] = &g12a_vdec_hevc.hw,
+ [CLKID_VDEC_HEVCF_SEL] = &g12a_vdec_hevcf_sel.hw,
+ [CLKID_VDEC_HEVCF_DIV] = &g12a_vdec_hevcf_div.hw,
+ [CLKID_VDEC_HEVCF] = &g12a_vdec_hevcf.hw,
[NR_CLKS] = NULL,
},
.num = NR_CLKS,
&g12a_mali_1,
&g12a_mali,
&g12a_mpll_50m,
+ &g12a_sys_pll_div16_en,
+ &g12a_cpu_clk_premux0,
+ &g12a_cpu_clk_mux0_div,
+ &g12a_cpu_clk_postmux0,
+ &g12a_cpu_clk_premux1,
+ &g12a_cpu_clk_mux1_div,
+ &g12a_cpu_clk_postmux1,
+ &g12a_cpu_clk_dyn,
+ &g12a_cpu_clk,
+ &g12a_cpu_clk_div16_en,
+ &g12a_cpu_clk_apb_div,
+ &g12a_cpu_clk_apb,
+ &g12a_cpu_clk_atb_div,
+ &g12a_cpu_clk_atb,
+ &g12a_cpu_clk_axi_div,
+ &g12a_cpu_clk_axi,
+ &g12a_cpu_clk_trace_div,
+ &g12a_cpu_clk_trace,
+ &g12a_pcie_pll_od,
+ &g12a_pcie_pll_dco,
+ &g12a_vdec_1_sel,
+ &g12a_vdec_1_div,
+ &g12a_vdec_1,
+ &g12a_vdec_hevc_sel,
+ &g12a_vdec_hevc_div,
+ &g12a_vdec_hevc,
+ &g12a_vdec_hevcf_sel,
+ &g12a_vdec_hevcf_div,
+ &g12a_vdec_hevcf,
};
static const struct meson_eeclkc_data g12a_clkc_data = {
#define HHI_GCLK_MPEG2 0x148
#define HHI_GCLK_OTHER 0x150
#define HHI_GCLK_OTHER2 0x154
+#define HHI_SYS_CPU_CLK_CNTL1 0x15c
#define HHI_VID_CLK_DIV 0x164
#define HHI_MPEG_CLK_CNTL 0x174
#define HHI_AUD_CLK_CNTL 0x178
#define CLKID_MALI_0_DIV 170
#define CLKID_MALI_1_DIV 173
#define CLKID_MPLL_5OM_DIV 176
+#define CLKID_SYS_PLL_DIV16_EN 178
+#define CLKID_SYS_PLL_DIV16 179
+#define CLKID_CPU_CLK_DYN0_SEL 180
+#define CLKID_CPU_CLK_DYN0_DIV 181
+#define CLKID_CPU_CLK_DYN0 182
+#define CLKID_CPU_CLK_DYN1_SEL 183
+#define CLKID_CPU_CLK_DYN1_DIV 184
+#define CLKID_CPU_CLK_DYN1 185
+#define CLKID_CPU_CLK_DYN 186
+#define CLKID_CPU_CLK_DIV16_EN 188
+#define CLKID_CPU_CLK_DIV16 189
+#define CLKID_CPU_CLK_APB_DIV 190
+#define CLKID_CPU_CLK_APB 191
+#define CLKID_CPU_CLK_ATB_DIV 192
+#define CLKID_CPU_CLK_ATB 193
+#define CLKID_CPU_CLK_AXI_DIV 194
+#define CLKID_CPU_CLK_AXI 195
+#define CLKID_CPU_CLK_TRACE_DIV 196
+#define CLKID_CPU_CLK_TRACE 197
+#define CLKID_PCIE_PLL_DCO 198
+#define CLKID_PCIE_PLL_DCO_DIV2 199
+#define CLKID_PCIE_PLL_OD 200
+#define CLKID_VDEC_1_SEL 202
+#define CLKID_VDEC_1_DIV 203
+#define CLKID_VDEC_HEVC_SEL 205
+#define CLKID_VDEC_HEVC_DIV 206
+#define CLKID_VDEC_HEVCF_SEL 208
+#define CLKID_VDEC_HEVCF_DIV 209
-#define NR_CLKS 178
+#define NR_CLKS 211
/* include the CLKIDs that have been made part of the DT binding */
#include <dt-bindings/clock/g12a-clkc.h>
},
};
+static const struct pll_params_table meson8m2_gp_pll_params_table[] = {
+ PLL_PARAMS(182, 3),
+ { /* sentinel */ },
+};
+
+static struct clk_regmap meson8m2_gp_pll_dco = {
+ .data = &(struct meson_clk_pll_data){
+ .en = {
+ .reg_off = HHI_GP_PLL_CNTL,
+ .shift = 30,
+ .width = 1,
+ },
+ .m = {
+ .reg_off = HHI_GP_PLL_CNTL,
+ .shift = 0,
+ .width = 9,
+ },
+ .n = {
+ .reg_off = HHI_GP_PLL_CNTL,
+ .shift = 9,
+ .width = 5,
+ },
+ .l = {
+ .reg_off = HHI_GP_PLL_CNTL,
+ .shift = 31,
+ .width = 1,
+ },
+ .rst = {
+ .reg_off = HHI_GP_PLL_CNTL,
+ .shift = 29,
+ .width = 1,
+ },
+ .table = meson8m2_gp_pll_params_table,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "gp_pll_dco",
+ .ops = &meson_clk_pll_ops,
+ .parent_names = (const char *[]){ "xtal" },
+ .num_parents = 1,
+ },
+};
+
+static struct clk_regmap meson8m2_gp_pll = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_GP_PLL_CNTL,
+ .shift = 16,
+ .width = 2,
+ .flags = CLK_DIVIDER_POWER_OF_TWO,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "gp_pll",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "gp_pll_dco" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static const char * const mmeson8b_vpu_0_1_parent_names[] = {
+ "fclk_div4", "fclk_div3", "fclk_div5", "fclk_div7"
+};
+
+static const char * const mmeson8m2_vpu_0_1_parent_names[] = {
+ "fclk_div4", "fclk_div3", "fclk_div5", "gp_pll"
+};
+
+static struct clk_regmap meson8b_vpu_0_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VPU_CLK_CNTL,
+ .mask = 0x3,
+ .shift = 9,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vpu_0_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = mmeson8b_vpu_0_1_parent_names,
+ .num_parents = ARRAY_SIZE(mmeson8b_vpu_0_1_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8m2_vpu_0_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VPU_CLK_CNTL,
+ .mask = 0x3,
+ .shift = 9,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vpu_0_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = mmeson8m2_vpu_0_1_parent_names,
+ .num_parents = ARRAY_SIZE(mmeson8m2_vpu_0_1_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vpu_0_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_VPU_CLK_CNTL,
+ .shift = 0,
+ .width = 7,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vpu_0_div",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "vpu_0_sel" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vpu_0 = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_VPU_CLK_CNTL,
+ .bit_idx = 8,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "vpu_0",
+ .ops = &clk_regmap_gate_ops,
+ .parent_names = (const char *[]){ "vpu_0_div" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vpu_1_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VPU_CLK_CNTL,
+ .mask = 0x3,
+ .shift = 25,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vpu_1_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = mmeson8b_vpu_0_1_parent_names,
+ .num_parents = ARRAY_SIZE(mmeson8b_vpu_0_1_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8m2_vpu_1_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VPU_CLK_CNTL,
+ .mask = 0x3,
+ .shift = 25,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vpu_1_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = mmeson8m2_vpu_0_1_parent_names,
+ .num_parents = ARRAY_SIZE(mmeson8m2_vpu_0_1_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vpu_1_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_VPU_CLK_CNTL,
+ .shift = 16,
+ .width = 7,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vpu_1_div",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "vpu_1_sel" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vpu_1 = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_VPU_CLK_CNTL,
+ .bit_idx = 24,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "vpu_1",
+ .ops = &clk_regmap_gate_ops,
+ .parent_names = (const char *[]){ "vpu_1_div" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vpu = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VPU_CLK_CNTL,
+ .mask = 1,
+ .shift = 31,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vpu",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = (const char *[]){ "vpu_0", "vpu_1" },
+ .num_parents = 2,
+ .flags = CLK_SET_RATE_NO_REPARENT,
+ },
+};
+
+static const char * const meson8b_vdec_parent_names[] = {
+ "fclk_div4", "fclk_div3", "fclk_div5", "fclk_div7", "mpll2", "mpll1"
+};
+
+static struct clk_regmap meson8b_vdec_1_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VDEC_CLK_CNTL,
+ .mask = 0x3,
+ .shift = 9,
+ .flags = CLK_MUX_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_1_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = meson8b_vdec_parent_names,
+ .num_parents = ARRAY_SIZE(meson8b_vdec_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_1_1_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_VDEC_CLK_CNTL,
+ .shift = 0,
+ .width = 7,
+ .flags = CLK_DIVIDER_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_1_1_div",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "vdec_1_sel" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_1_1 = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_VDEC_CLK_CNTL,
+ .bit_idx = 8,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "vdec_1_1",
+ .ops = &clk_regmap_gate_ops,
+ .parent_names = (const char *[]){ "vdec_1_1_div" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_1_2_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_VDEC3_CLK_CNTL,
+ .shift = 0,
+ .width = 7,
+ .flags = CLK_DIVIDER_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_1_2_div",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "vdec_1_sel" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_1_2 = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_VDEC3_CLK_CNTL,
+ .bit_idx = 8,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "vdec_1_2",
+ .ops = &clk_regmap_gate_ops,
+ .parent_names = (const char *[]){ "vdec_1_2_div" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_1 = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VDEC3_CLK_CNTL,
+ .mask = 0x1,
+ .shift = 15,
+ .flags = CLK_MUX_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_1",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = (const char *[]){ "vdec_1_1", "vdec_1_2" },
+ .num_parents = 2,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_hcodec_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VDEC_CLK_CNTL,
+ .mask = 0x3,
+ .shift = 25,
+ .flags = CLK_MUX_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_hcodec_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = meson8b_vdec_parent_names,
+ .num_parents = ARRAY_SIZE(meson8b_vdec_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_hcodec_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_VDEC_CLK_CNTL,
+ .shift = 16,
+ .width = 7,
+ .flags = CLK_DIVIDER_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_hcodec_div",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "vdec_hcodec_sel" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_hcodec = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_VDEC_CLK_CNTL,
+ .bit_idx = 24,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "vdec_hcodec",
+ .ops = &clk_regmap_gate_ops,
+ .parent_names = (const char *[]){ "vdec_hcodec_div" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_2_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .mask = 0x3,
+ .shift = 9,
+ .flags = CLK_MUX_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_2_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = meson8b_vdec_parent_names,
+ .num_parents = ARRAY_SIZE(meson8b_vdec_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_2_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .shift = 0,
+ .width = 7,
+ .flags = CLK_DIVIDER_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_2_div",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "vdec_2_sel" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_2 = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .bit_idx = 8,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "vdec_2",
+ .ops = &clk_regmap_gate_ops,
+ .parent_names = (const char *[]){ "vdec_2_div" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_hevc_sel = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .mask = 0x3,
+ .shift = 25,
+ .flags = CLK_MUX_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_hevc_sel",
+ .ops = &clk_regmap_mux_ops,
+ .parent_names = meson8b_vdec_parent_names,
+ .num_parents = ARRAY_SIZE(meson8b_vdec_parent_names),
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_hevc_div = {
+ .data = &(struct clk_regmap_div_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .shift = 16,
+ .width = 7,
+ .flags = CLK_DIVIDER_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_hevc_div",
+ .ops = &clk_regmap_divider_ops,
+ .parent_names = (const char *[]){ "vdec_hevc_sel" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_hevc_en = {
+ .data = &(struct clk_regmap_gate_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .bit_idx = 24,
+ },
+ .hw.init = &(struct clk_init_data) {
+ .name = "vdec_hevc_en",
+ .ops = &clk_regmap_gate_ops,
+ .parent_names = (const char *[]){ "vdec_hevc_div" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
+static struct clk_regmap meson8b_vdec_hevc = {
+ .data = &(struct clk_regmap_mux_data){
+ .offset = HHI_VDEC2_CLK_CNTL,
+ .mask = 0x1,
+ .shift = 31,
+ .flags = CLK_MUX_ROUND_CLOSEST,
+ },
+ .hw.init = &(struct clk_init_data){
+ .name = "vdec_hevc",
+ .ops = &clk_regmap_mux_ops,
+ /* TODO: The second parent is currently unknown */
+ .parent_names = (const char *[]){ "vdec_hevc_en" },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ },
+};
+
/* Everything Else (EE) domain gates */
static MESON_GATE(meson8b_ddr, HHI_GCLK_MPEG0, 0);
[CLKID_MALI_0_SEL] = &meson8b_mali_0_sel.hw,
[CLKID_MALI_0_DIV] = &meson8b_mali_0_div.hw,
[CLKID_MALI] = &meson8b_mali_0.hw,
+ [CLKID_VPU_0_SEL] = &meson8b_vpu_0_sel.hw,
+ [CLKID_VPU_0_DIV] = &meson8b_vpu_0_div.hw,
+ [CLKID_VPU] = &meson8b_vpu_0.hw,
+ [CLKID_VDEC_1_SEL] = &meson8b_vdec_1_sel.hw,
+ [CLKID_VDEC_1_1_DIV] = &meson8b_vdec_1_1_div.hw,
+ [CLKID_VDEC_1] = &meson8b_vdec_1_1.hw,
+ [CLKID_VDEC_HCODEC_SEL] = &meson8b_vdec_hcodec_sel.hw,
+ [CLKID_VDEC_HCODEC_DIV] = &meson8b_vdec_hcodec_div.hw,
+ [CLKID_VDEC_HCODEC] = &meson8b_vdec_hcodec.hw,
+ [CLKID_VDEC_2_SEL] = &meson8b_vdec_2_sel.hw,
+ [CLKID_VDEC_2_DIV] = &meson8b_vdec_2_div.hw,
+ [CLKID_VDEC_2] = &meson8b_vdec_2.hw,
+ [CLKID_VDEC_HEVC_SEL] = &meson8b_vdec_hevc_sel.hw,
+ [CLKID_VDEC_HEVC_DIV] = &meson8b_vdec_hevc_div.hw,
+ [CLKID_VDEC_HEVC_EN] = &meson8b_vdec_hevc_en.hw,
+ [CLKID_VDEC_HEVC] = &meson8b_vdec_hevc.hw,
[CLK_NR_CLKS] = NULL,
},
.num = CLK_NR_CLKS,
[CLKID_MALI_1_DIV] = &meson8b_mali_1_div.hw,
[CLKID_MALI_1] = &meson8b_mali_1.hw,
[CLKID_MALI] = &meson8b_mali.hw,
+ [CLKID_VPU_0_SEL] = &meson8b_vpu_0_sel.hw,
+ [CLKID_VPU_0_DIV] = &meson8b_vpu_0_div.hw,
+ [CLKID_VPU_0] = &meson8b_vpu_0.hw,
+ [CLKID_VPU_1_SEL] = &meson8b_vpu_1_sel.hw,
+ [CLKID_VPU_1_DIV] = &meson8b_vpu_1_div.hw,
+ [CLKID_VPU_1] = &meson8b_vpu_1.hw,
+ [CLKID_VPU] = &meson8b_vpu.hw,
+ [CLKID_VDEC_1_SEL] = &meson8b_vdec_1_sel.hw,
+ [CLKID_VDEC_1_1_DIV] = &meson8b_vdec_1_1_div.hw,
+ [CLKID_VDEC_1_1] = &meson8b_vdec_1_1.hw,
+ [CLKID_VDEC_1_2_DIV] = &meson8b_vdec_1_2_div.hw,
+ [CLKID_VDEC_1_2] = &meson8b_vdec_1_2.hw,
+ [CLKID_VDEC_1] = &meson8b_vdec_1.hw,
+ [CLKID_VDEC_HCODEC_SEL] = &meson8b_vdec_hcodec_sel.hw,
+ [CLKID_VDEC_HCODEC_DIV] = &meson8b_vdec_hcodec_div.hw,
+ [CLKID_VDEC_HCODEC] = &meson8b_vdec_hcodec.hw,
+ [CLKID_VDEC_2_SEL] = &meson8b_vdec_2_sel.hw,
+ [CLKID_VDEC_2_DIV] = &meson8b_vdec_2_div.hw,
+ [CLKID_VDEC_2] = &meson8b_vdec_2.hw,
+ [CLKID_VDEC_HEVC_SEL] = &meson8b_vdec_hevc_sel.hw,
+ [CLKID_VDEC_HEVC_DIV] = &meson8b_vdec_hevc_div.hw,
+ [CLKID_VDEC_HEVC_EN] = &meson8b_vdec_hevc_en.hw,
+ [CLKID_VDEC_HEVC] = &meson8b_vdec_hevc.hw,
+ [CLK_NR_CLKS] = NULL,
+ },
+ .num = CLK_NR_CLKS,
+};
+
+static struct clk_hw_onecell_data meson8m2_hw_onecell_data = {
+ .hws = {
+ [CLKID_XTAL] = &meson8b_xtal.hw,
+ [CLKID_PLL_FIXED] = &meson8b_fixed_pll.hw,
+ [CLKID_PLL_VID] = &meson8b_vid_pll.hw,
+ [CLKID_PLL_SYS] = &meson8b_sys_pll.hw,
+ [CLKID_FCLK_DIV2] = &meson8b_fclk_div2.hw,
+ [CLKID_FCLK_DIV3] = &meson8b_fclk_div3.hw,
+ [CLKID_FCLK_DIV4] = &meson8b_fclk_div4.hw,
+ [CLKID_FCLK_DIV5] = &meson8b_fclk_div5.hw,
+ [CLKID_FCLK_DIV7] = &meson8b_fclk_div7.hw,
+ [CLKID_CPUCLK] = &meson8b_cpu_clk.hw,
+ [CLKID_MPEG_SEL] = &meson8b_mpeg_clk_sel.hw,
+ [CLKID_MPEG_DIV] = &meson8b_mpeg_clk_div.hw,
+ [CLKID_CLK81] = &meson8b_clk81.hw,
+ [CLKID_DDR] = &meson8b_ddr.hw,
+ [CLKID_DOS] = &meson8b_dos.hw,
+ [CLKID_ISA] = &meson8b_isa.hw,
+ [CLKID_PL301] = &meson8b_pl301.hw,
+ [CLKID_PERIPHS] = &meson8b_periphs.hw,
+ [CLKID_SPICC] = &meson8b_spicc.hw,
+ [CLKID_I2C] = &meson8b_i2c.hw,
+ [CLKID_SAR_ADC] = &meson8b_sar_adc.hw,
+ [CLKID_SMART_CARD] = &meson8b_smart_card.hw,
+ [CLKID_RNG0] = &meson8b_rng0.hw,
+ [CLKID_UART0] = &meson8b_uart0.hw,
+ [CLKID_SDHC] = &meson8b_sdhc.hw,
+ [CLKID_STREAM] = &meson8b_stream.hw,
+ [CLKID_ASYNC_FIFO] = &meson8b_async_fifo.hw,
+ [CLKID_SDIO] = &meson8b_sdio.hw,
+ [CLKID_ABUF] = &meson8b_abuf.hw,
+ [CLKID_HIU_IFACE] = &meson8b_hiu_iface.hw,
+ [CLKID_ASSIST_MISC] = &meson8b_assist_misc.hw,
+ [CLKID_SPI] = &meson8b_spi.hw,
+ [CLKID_I2S_SPDIF] = &meson8b_i2s_spdif.hw,
+ [CLKID_ETH] = &meson8b_eth.hw,
+ [CLKID_DEMUX] = &meson8b_demux.hw,
+ [CLKID_AIU_GLUE] = &meson8b_aiu_glue.hw,
+ [CLKID_IEC958] = &meson8b_iec958.hw,
+ [CLKID_I2S_OUT] = &meson8b_i2s_out.hw,
+ [CLKID_AMCLK] = &meson8b_amclk.hw,
+ [CLKID_AIFIFO2] = &meson8b_aififo2.hw,
+ [CLKID_MIXER] = &meson8b_mixer.hw,
+ [CLKID_MIXER_IFACE] = &meson8b_mixer_iface.hw,
+ [CLKID_ADC] = &meson8b_adc.hw,
+ [CLKID_BLKMV] = &meson8b_blkmv.hw,
+ [CLKID_AIU] = &meson8b_aiu.hw,
+ [CLKID_UART1] = &meson8b_uart1.hw,
+ [CLKID_G2D] = &meson8b_g2d.hw,
+ [CLKID_USB0] = &meson8b_usb0.hw,
+ [CLKID_USB1] = &meson8b_usb1.hw,
+ [CLKID_RESET] = &meson8b_reset.hw,
+ [CLKID_NAND] = &meson8b_nand.hw,
+ [CLKID_DOS_PARSER] = &meson8b_dos_parser.hw,
+ [CLKID_USB] = &meson8b_usb.hw,
+ [CLKID_VDIN1] = &meson8b_vdin1.hw,
+ [CLKID_AHB_ARB0] = &meson8b_ahb_arb0.hw,
+ [CLKID_EFUSE] = &meson8b_efuse.hw,
+ [CLKID_BOOT_ROM] = &meson8b_boot_rom.hw,
+ [CLKID_AHB_DATA_BUS] = &meson8b_ahb_data_bus.hw,
+ [CLKID_AHB_CTRL_BUS] = &meson8b_ahb_ctrl_bus.hw,
+ [CLKID_HDMI_INTR_SYNC] = &meson8b_hdmi_intr_sync.hw,
+ [CLKID_HDMI_PCLK] = &meson8b_hdmi_pclk.hw,
+ [CLKID_USB1_DDR_BRIDGE] = &meson8b_usb1_ddr_bridge.hw,
+ [CLKID_USB0_DDR_BRIDGE] = &meson8b_usb0_ddr_bridge.hw,
+ [CLKID_MMC_PCLK] = &meson8b_mmc_pclk.hw,
+ [CLKID_DVIN] = &meson8b_dvin.hw,
+ [CLKID_UART2] = &meson8b_uart2.hw,
+ [CLKID_SANA] = &meson8b_sana.hw,
+ [CLKID_VPU_INTR] = &meson8b_vpu_intr.hw,
+ [CLKID_SEC_AHB_AHB3_BRIDGE] = &meson8b_sec_ahb_ahb3_bridge.hw,
+ [CLKID_CLK81_A9] = &meson8b_clk81_a9.hw,
+ [CLKID_VCLK2_VENCI0] = &meson8b_vclk2_venci0.hw,
+ [CLKID_VCLK2_VENCI1] = &meson8b_vclk2_venci1.hw,
+ [CLKID_VCLK2_VENCP0] = &meson8b_vclk2_vencp0.hw,
+ [CLKID_VCLK2_VENCP1] = &meson8b_vclk2_vencp1.hw,
+ [CLKID_GCLK_VENCI_INT] = &meson8b_gclk_venci_int.hw,
+ [CLKID_GCLK_VENCP_INT] = &meson8b_gclk_vencp_int.hw,
+ [CLKID_DAC_CLK] = &meson8b_dac_clk.hw,
+ [CLKID_AOCLK_GATE] = &meson8b_aoclk_gate.hw,
+ [CLKID_IEC958_GATE] = &meson8b_iec958_gate.hw,
+ [CLKID_ENC480P] = &meson8b_enc480p.hw,
+ [CLKID_RNG1] = &meson8b_rng1.hw,
+ [CLKID_GCLK_VENCL_INT] = &meson8b_gclk_vencl_int.hw,
+ [CLKID_VCLK2_VENCLMCC] = &meson8b_vclk2_venclmcc.hw,
+ [CLKID_VCLK2_VENCL] = &meson8b_vclk2_vencl.hw,
+ [CLKID_VCLK2_OTHER] = &meson8b_vclk2_other.hw,
+ [CLKID_EDP] = &meson8b_edp.hw,
+ [CLKID_AO_MEDIA_CPU] = &meson8b_ao_media_cpu.hw,
+ [CLKID_AO_AHB_SRAM] = &meson8b_ao_ahb_sram.hw,
+ [CLKID_AO_AHB_BUS] = &meson8b_ao_ahb_bus.hw,
+ [CLKID_AO_IFACE] = &meson8b_ao_iface.hw,
+ [CLKID_MPLL0] = &meson8b_mpll0.hw,
+ [CLKID_MPLL1] = &meson8b_mpll1.hw,
+ [CLKID_MPLL2] = &meson8b_mpll2.hw,
+ [CLKID_MPLL0_DIV] = &meson8b_mpll0_div.hw,
+ [CLKID_MPLL1_DIV] = &meson8b_mpll1_div.hw,
+ [CLKID_MPLL2_DIV] = &meson8b_mpll2_div.hw,
+ [CLKID_CPU_IN_SEL] = &meson8b_cpu_in_sel.hw,
+ [CLKID_CPU_IN_DIV2] = &meson8b_cpu_in_div2.hw,
+ [CLKID_CPU_IN_DIV3] = &meson8b_cpu_in_div3.hw,
+ [CLKID_CPU_SCALE_DIV] = &meson8b_cpu_scale_div.hw,
+ [CLKID_CPU_SCALE_OUT_SEL] = &meson8b_cpu_scale_out_sel.hw,
+ [CLKID_MPLL_PREDIV] = &meson8b_mpll_prediv.hw,
+ [CLKID_FCLK_DIV2_DIV] = &meson8b_fclk_div2_div.hw,
+ [CLKID_FCLK_DIV3_DIV] = &meson8b_fclk_div3_div.hw,
+ [CLKID_FCLK_DIV4_DIV] = &meson8b_fclk_div4_div.hw,
+ [CLKID_FCLK_DIV5_DIV] = &meson8b_fclk_div5_div.hw,
+ [CLKID_FCLK_DIV7_DIV] = &meson8b_fclk_div7_div.hw,
+ [CLKID_NAND_SEL] = &meson8b_nand_clk_sel.hw,
+ [CLKID_NAND_DIV] = &meson8b_nand_clk_div.hw,
+ [CLKID_NAND_CLK] = &meson8b_nand_clk_gate.hw,
+ [CLKID_PLL_FIXED_DCO] = &meson8b_fixed_pll_dco.hw,
+ [CLKID_HDMI_PLL_DCO] = &meson8b_hdmi_pll_dco.hw,
+ [CLKID_PLL_SYS_DCO] = &meson8b_sys_pll_dco.hw,
+ [CLKID_CPU_CLK_DIV2] = &meson8b_cpu_clk_div2.hw,
+ [CLKID_CPU_CLK_DIV3] = &meson8b_cpu_clk_div3.hw,
+ [CLKID_CPU_CLK_DIV4] = &meson8b_cpu_clk_div4.hw,
+ [CLKID_CPU_CLK_DIV5] = &meson8b_cpu_clk_div5.hw,
+ [CLKID_CPU_CLK_DIV6] = &meson8b_cpu_clk_div6.hw,
+ [CLKID_CPU_CLK_DIV7] = &meson8b_cpu_clk_div7.hw,
+ [CLKID_CPU_CLK_DIV8] = &meson8b_cpu_clk_div8.hw,
+ [CLKID_APB_SEL] = &meson8b_apb_clk_sel.hw,
+ [CLKID_APB] = &meson8b_apb_clk_gate.hw,
+ [CLKID_PERIPH_SEL] = &meson8b_periph_clk_sel.hw,
+ [CLKID_PERIPH] = &meson8b_periph_clk_gate.hw,
+ [CLKID_AXI_SEL] = &meson8b_axi_clk_sel.hw,
+ [CLKID_AXI] = &meson8b_axi_clk_gate.hw,
+ [CLKID_L2_DRAM_SEL] = &meson8b_l2_dram_clk_sel.hw,
+ [CLKID_L2_DRAM] = &meson8b_l2_dram_clk_gate.hw,
+ [CLKID_HDMI_PLL_LVDS_OUT] = &meson8b_hdmi_pll_lvds_out.hw,
+ [CLKID_HDMI_PLL_HDMI_OUT] = &meson8b_hdmi_pll_hdmi_out.hw,
+ [CLKID_VID_PLL_IN_SEL] = &meson8b_vid_pll_in_sel.hw,
+ [CLKID_VID_PLL_IN_EN] = &meson8b_vid_pll_in_en.hw,
+ [CLKID_VID_PLL_PRE_DIV] = &meson8b_vid_pll_pre_div.hw,
+ [CLKID_VID_PLL_POST_DIV] = &meson8b_vid_pll_post_div.hw,
+ [CLKID_VID_PLL_FINAL_DIV] = &meson8b_vid_pll_final_div.hw,
+ [CLKID_VCLK_IN_SEL] = &meson8b_vclk_in_sel.hw,
+ [CLKID_VCLK_IN_EN] = &meson8b_vclk_in_en.hw,
+ [CLKID_VCLK_DIV1] = &meson8b_vclk_div1_gate.hw,
+ [CLKID_VCLK_DIV2_DIV] = &meson8b_vclk_div2_div.hw,
+ [CLKID_VCLK_DIV2] = &meson8b_vclk_div2_div_gate.hw,
+ [CLKID_VCLK_DIV4_DIV] = &meson8b_vclk_div4_div.hw,
+ [CLKID_VCLK_DIV4] = &meson8b_vclk_div4_div_gate.hw,
+ [CLKID_VCLK_DIV6_DIV] = &meson8b_vclk_div6_div.hw,
+ [CLKID_VCLK_DIV6] = &meson8b_vclk_div6_div_gate.hw,
+ [CLKID_VCLK_DIV12_DIV] = &meson8b_vclk_div12_div.hw,
+ [CLKID_VCLK_DIV12] = &meson8b_vclk_div12_div_gate.hw,
+ [CLKID_VCLK2_IN_SEL] = &meson8b_vclk2_in_sel.hw,
+ [CLKID_VCLK2_IN_EN] = &meson8b_vclk2_clk_in_en.hw,
+ [CLKID_VCLK2_DIV1] = &meson8b_vclk2_div1_gate.hw,
+ [CLKID_VCLK2_DIV2_DIV] = &meson8b_vclk2_div2_div.hw,
+ [CLKID_VCLK2_DIV2] = &meson8b_vclk2_div2_div_gate.hw,
+ [CLKID_VCLK2_DIV4_DIV] = &meson8b_vclk2_div4_div.hw,
+ [CLKID_VCLK2_DIV4] = &meson8b_vclk2_div4_div_gate.hw,
+ [CLKID_VCLK2_DIV6_DIV] = &meson8b_vclk2_div6_div.hw,
+ [CLKID_VCLK2_DIV6] = &meson8b_vclk2_div6_div_gate.hw,
+ [CLKID_VCLK2_DIV12_DIV] = &meson8b_vclk2_div12_div.hw,
+ [CLKID_VCLK2_DIV12] = &meson8b_vclk2_div12_div_gate.hw,
+ [CLKID_CTS_ENCT_SEL] = &meson8b_cts_enct_sel.hw,
+ [CLKID_CTS_ENCT] = &meson8b_cts_enct.hw,
+ [CLKID_CTS_ENCP_SEL] = &meson8b_cts_encp_sel.hw,
+ [CLKID_CTS_ENCP] = &meson8b_cts_encp.hw,
+ [CLKID_CTS_ENCI_SEL] = &meson8b_cts_enci_sel.hw,
+ [CLKID_CTS_ENCI] = &meson8b_cts_enci.hw,
+ [CLKID_HDMI_TX_PIXEL_SEL] = &meson8b_hdmi_tx_pixel_sel.hw,
+ [CLKID_HDMI_TX_PIXEL] = &meson8b_hdmi_tx_pixel.hw,
+ [CLKID_CTS_ENCL_SEL] = &meson8b_cts_encl_sel.hw,
+ [CLKID_CTS_ENCL] = &meson8b_cts_encl.hw,
+ [CLKID_CTS_VDAC0_SEL] = &meson8b_cts_vdac0_sel.hw,
+ [CLKID_CTS_VDAC0] = &meson8b_cts_vdac0.hw,
+ [CLKID_HDMI_SYS_SEL] = &meson8b_hdmi_sys_sel.hw,
+ [CLKID_HDMI_SYS_DIV] = &meson8b_hdmi_sys_div.hw,
+ [CLKID_HDMI_SYS] = &meson8b_hdmi_sys.hw,
+ [CLKID_MALI_0_SEL] = &meson8b_mali_0_sel.hw,
+ [CLKID_MALI_0_DIV] = &meson8b_mali_0_div.hw,
+ [CLKID_MALI_0] = &meson8b_mali_0.hw,
+ [CLKID_MALI_1_SEL] = &meson8b_mali_1_sel.hw,
+ [CLKID_MALI_1_DIV] = &meson8b_mali_1_div.hw,
+ [CLKID_MALI_1] = &meson8b_mali_1.hw,
+ [CLKID_MALI] = &meson8b_mali.hw,
+ [CLKID_GP_PLL_DCO] = &meson8m2_gp_pll_dco.hw,
+ [CLKID_GP_PLL] = &meson8m2_gp_pll.hw,
+ [CLKID_VPU_0_SEL] = &meson8m2_vpu_0_sel.hw,
+ [CLKID_VPU_0_DIV] = &meson8b_vpu_0_div.hw,
+ [CLKID_VPU_0] = &meson8b_vpu_0.hw,
+ [CLKID_VPU_1_SEL] = &meson8m2_vpu_1_sel.hw,
+ [CLKID_VPU_1_DIV] = &meson8b_vpu_1_div.hw,
+ [CLKID_VPU_1] = &meson8b_vpu_1.hw,
+ [CLKID_VPU] = &meson8b_vpu.hw,
+ [CLKID_VDEC_1_SEL] = &meson8b_vdec_1_sel.hw,
+ [CLKID_VDEC_1_1_DIV] = &meson8b_vdec_1_1_div.hw,
+ [CLKID_VDEC_1_1] = &meson8b_vdec_1_1.hw,
+ [CLKID_VDEC_1_2_DIV] = &meson8b_vdec_1_2_div.hw,
+ [CLKID_VDEC_1_2] = &meson8b_vdec_1_2.hw,
+ [CLKID_VDEC_1] = &meson8b_vdec_1.hw,
+ [CLKID_VDEC_HCODEC_SEL] = &meson8b_vdec_hcodec_sel.hw,
+ [CLKID_VDEC_HCODEC_DIV] = &meson8b_vdec_hcodec_div.hw,
+ [CLKID_VDEC_HCODEC] = &meson8b_vdec_hcodec.hw,
+ [CLKID_VDEC_2_SEL] = &meson8b_vdec_2_sel.hw,
+ [CLKID_VDEC_2_DIV] = &meson8b_vdec_2_div.hw,
+ [CLKID_VDEC_2] = &meson8b_vdec_2.hw,
+ [CLKID_VDEC_HEVC_SEL] = &meson8b_vdec_hevc_sel.hw,
+ [CLKID_VDEC_HEVC_DIV] = &meson8b_vdec_hevc_div.hw,
+ [CLKID_VDEC_HEVC_EN] = &meson8b_vdec_hevc_en.hw,
+ [CLKID_VDEC_HEVC] = &meson8b_vdec_hevc.hw,
[CLK_NR_CLKS] = NULL,
},
.num = CLK_NR_CLKS,
&meson8b_mali_1_div,
&meson8b_mali_1,
&meson8b_mali,
+ &meson8m2_gp_pll_dco,
+ &meson8m2_gp_pll,
+ &meson8b_vpu_0_sel,
+ &meson8m2_vpu_0_sel,
+ &meson8b_vpu_0_div,
+ &meson8b_vpu_0,
+ &meson8b_vpu_1_sel,
+ &meson8m2_vpu_1_sel,
+ &meson8b_vpu_1_div,
+ &meson8b_vpu_1,
+ &meson8b_vpu,
+ &meson8b_vdec_1_sel,
+ &meson8b_vdec_1_1_div,
+ &meson8b_vdec_1_1,
+ &meson8b_vdec_1_2_div,
+ &meson8b_vdec_1_2,
+ &meson8b_vdec_1,
+ &meson8b_vdec_hcodec_sel,
+ &meson8b_vdec_hcodec_div,
+ &meson8b_vdec_hcodec,
+ &meson8b_vdec_2_sel,
+ &meson8b_vdec_2_div,
+ &meson8b_vdec_2,
+ &meson8b_vdec_hevc_sel,
+ &meson8b_vdec_hevc_div,
+ &meson8b_vdec_hevc_en,
+ &meson8b_vdec_hevc,
};
static const struct meson8b_clk_reset_line {
return meson8b_clkc_init_common(np, &meson8b_hw_onecell_data);
}
+static void __init meson8m2_clkc_init(struct device_node *np)
+{
+ return meson8b_clkc_init_common(np, &meson8m2_hw_onecell_data);
+}
+
CLK_OF_DECLARE_DRIVER(meson8_clkc, "amlogic,meson8-clkc",
meson8_clkc_init);
CLK_OF_DECLARE_DRIVER(meson8b_clkc, "amlogic,meson8b-clkc",
meson8b_clkc_init);
CLK_OF_DECLARE_DRIVER(meson8m2_clkc, "amlogic,meson8m2-clkc",
- meson8b_clkc_init);
+ meson8m2_clkc_init);
*
* [0] http://dn.odroid.com/S805/Datasheet/S805_Datasheet%20V0.8%2020150126.pdf
*/
+#define HHI_GP_PLL_CNTL 0x40 /* 0x10 offset in data sheet */
#define HHI_VIID_CLK_DIV 0x128 /* 0x4a offset in data sheet */
#define HHI_VIID_CLK_CNTL 0x12c /* 0x4b offset in data sheet */
#define HHI_GCLK_MPEG0 0x140 /* 0x50 offset in data sheet */
#define HHI_VID_DIVIDER_CNTL 0x198 /* 0x66 offset in data sheet */
#define HHI_SYS_CPU_CLK_CNTL0 0x19c /* 0x67 offset in data sheet */
#define HHI_MALI_CLK_CNTL 0x1b0 /* 0x6c offset in data sheet */
+#define HHI_VPU_CLK_CNTL 0x1bc /* 0x6f offset in data sheet */
#define HHI_HDMI_CLK_CNTL 0x1cc /* 0x73 offset in data sheet */
+#define HHI_VDEC_CLK_CNTL 0x1e0 /* 0x78 offset in data sheet */
+#define HHI_VDEC2_CLK_CNTL 0x1e4 /* 0x79 offset in data sheet */
+#define HHI_VDEC3_CLK_CNTL 0x1e8 /* 0x7a offset in data sheet */
#define HHI_NAND_CLK_CNTL 0x25c /* 0x97 offset in data sheet */
#define HHI_MPLL_CNTL 0x280 /* 0xa0 offset in data sheet */
#define HHI_SYS_PLL_CNTL 0x300 /* 0xc0 offset in data sheet */
#define CLKID_MALI_1_SEL 178
#define CLKID_MALI_1_DIV 179
#define CLKID_MALI_1 180
+#define CLKID_GP_PLL_DCO 181
+#define CLKID_GP_PLL 182
+#define CLKID_VPU_0_SEL 183
+#define CLKID_VPU_0_DIV 184
+#define CLKID_VPU_0 185
+#define CLKID_VPU_1_SEL 186
+#define CLKID_VPU_1_DIV 187
+#define CLKID_VPU_1 189
+#define CLKID_VDEC_1_SEL 191
+#define CLKID_VDEC_1_1_DIV 192
+#define CLKID_VDEC_1_1 193
+#define CLKID_VDEC_1_2_DIV 194
+#define CLKID_VDEC_1_2 195
+#define CLKID_VDEC_HCODEC_SEL 197
+#define CLKID_VDEC_HCODEC_DIV 198
+#define CLKID_VDEC_2_SEL 200
+#define CLKID_VDEC_2_DIV 201
+#define CLKID_VDEC_HEVC_SEL 203
+#define CLKID_VDEC_HEVC_DIV 204
+#define CLKID_VDEC_HEVC_EN 205
-#define CLK_NR_CLKS 181
+#define CLK_NR_CLKS 207
/*
* include the CLKID and RESETID that have
init.name = name;
init.ops = &mmp_clk_gate_ops;
- init.flags = flags | CLK_IS_BASIC;
+ init.flags = flags;
init.parent_names = (parent_name ? &parent_name : NULL);
init.num_parents = (parent_name ? 1 : 0);
int n;
if (ctrl) {
- pr_err("mvebu-clk-gating: cannot instantiate more than one gatable clock device\n");
+ pr_err("mvebu-clk-gating: cannot instantiate more than one gateable clock device\n");
return;
}
* - Equal to SDIO clock
* - 2/5 PLL0
*
- * CP110 has 32 gatable clocks, for the various peripherals in the IP.
+ * CP110 has 32 gateable clocks, for the various peripherals in the IP.
*/
#define pr_fmt(fmt) "cp110-system-controller: " fmt
#define CP110_CORE_NAND 4
#define CP110_CORE_SDIO 5
-/* A number of gatable clocks need special handling */
+/* A number of gateable clocks need special handling */
#define CP110_GATE_AUDIO 0
#define CP110_GATE_COMM_UNIT 1
#define CP110_GATE_NAND 2
* Divider field is write only, so divider stat field must
* be read so divider field can be set accordingly.
*/
- val = clk_readl(gate->reg);
+ val = readl(gate->reg);
if (val & LPC18XX_CCU_DIVSTAT)
val |= LPC18XX_CCU_DIV;
* and the next write should clear the RUN bit.
*/
val |= LPC18XX_CCU_AUTO;
- clk_writel(val, gate->reg);
+ writel(val, gate->reg);
val &= ~LPC18XX_CCU_RUN;
}
- clk_writel(val, gate->reg);
+ writel(val, gate->reg);
return 0;
}
struct lpc18xx_pll *pll = to_lpc_pll(hw);
u32 ctrl, mdiv, msel, npdiv;
- ctrl = clk_readl(pll->reg + LPC18XX_CGU_PLL0USB_CTRL);
- mdiv = clk_readl(pll->reg + LPC18XX_CGU_PLL0USB_MDIV);
- npdiv = clk_readl(pll->reg + LPC18XX_CGU_PLL0USB_NP_DIV);
+ ctrl = readl(pll->reg + LPC18XX_CGU_PLL0USB_CTRL);
+ mdiv = readl(pll->reg + LPC18XX_CGU_PLL0USB_MDIV);
+ npdiv = readl(pll->reg + LPC18XX_CGU_PLL0USB_NP_DIV);
if (ctrl & LPC18XX_PLL0_CTRL_BYPASS)
return parent_rate;
m |= lpc18xx_pll0_msel2seli(m) << LPC18XX_PLL0_MDIV_SELI_SHIFT;
/* Power down PLL, disable clk output and dividers */
- ctrl = clk_readl(pll->reg + LPC18XX_CGU_PLL0USB_CTRL);
+ ctrl = readl(pll->reg + LPC18XX_CGU_PLL0USB_CTRL);
ctrl |= LPC18XX_PLL0_CTRL_PD;
ctrl &= ~(LPC18XX_PLL0_CTRL_BYPASS | LPC18XX_PLL0_CTRL_DIRECTI |
LPC18XX_PLL0_CTRL_DIRECTO | LPC18XX_PLL0_CTRL_CLKEN);
- clk_writel(ctrl, pll->reg + LPC18XX_CGU_PLL0USB_CTRL);
+ writel(ctrl, pll->reg + LPC18XX_CGU_PLL0USB_CTRL);
/* Configure new PLL settings */
- clk_writel(m, pll->reg + LPC18XX_CGU_PLL0USB_MDIV);
- clk_writel(LPC18XX_PLL0_NP_DIVS_1, pll->reg + LPC18XX_CGU_PLL0USB_NP_DIV);
+ writel(m, pll->reg + LPC18XX_CGU_PLL0USB_MDIV);
+ writel(LPC18XX_PLL0_NP_DIVS_1, pll->reg + LPC18XX_CGU_PLL0USB_NP_DIV);
/* Power up PLL and wait for lock */
ctrl &= ~LPC18XX_PLL0_CTRL_PD;
- clk_writel(ctrl, pll->reg + LPC18XX_CGU_PLL0USB_CTRL);
+ writel(ctrl, pll->reg + LPC18XX_CGU_PLL0USB_CTRL);
do {
udelay(10);
- stat = clk_readl(pll->reg + LPC18XX_CGU_PLL0USB_STAT);
+ stat = readl(pll->reg + LPC18XX_CGU_PLL0USB_STAT);
if (stat & LPC18XX_PLL0_STAT_LOCK) {
ctrl |= LPC18XX_PLL0_CTRL_CLKEN;
- clk_writel(ctrl, pll->reg + LPC18XX_CGU_PLL0USB_CTRL);
+ writel(ctrl, pll->reg + LPC18XX_CGU_PLL0USB_CTRL);
return 0;
}
bool direct, fbsel;
u32 stat, ctrl;
- stat = clk_readl(pll->reg + LPC18XX_CGU_PLL1_STAT);
- ctrl = clk_readl(pll->reg + LPC18XX_CGU_PLL1_CTRL);
+ stat = readl(pll->reg + LPC18XX_CGU_PLL1_STAT);
+ ctrl = readl(pll->reg + LPC18XX_CGU_PLL1_CTRL);
direct = (ctrl & LPC18XX_PLL1_CTRL_DIRECT) ? true : false;
fbsel = (ctrl & LPC18XX_PLL1_CTRL_FBSEL) ? true : false;
};
};
-#define LPC32XX_DEFINE_FIXED(_idx, _rate, _flags) \
+#define LPC32XX_DEFINE_FIXED(_idx, _rate) \
[CLK_PREFIX(_idx)] = { \
.type = CLK_FIXED, \
{ \
.f = { \
.fixed_rate = (_rate), \
- .flags = (_flags), \
}, \
}, \
}
}
static struct clk_hw_proto clk_hw_proto[LPC32XX_CLK_HW_MAX] = {
- LPC32XX_DEFINE_FIXED(RTC, 32768, 0),
+ LPC32XX_DEFINE_FIXED(RTC, 32768),
LPC32XX_DEFINE_PLL(PLL397X, pll_397x, HCLKPLL_CTRL, BIT(1)),
LPC32XX_DEFINE_PLL(HCLK_PLL, hclk_pll, HCLKPLL_CTRL, PLL_CTRL_ENABLE),
LPC32XX_DEFINE_PLL(USB_PLL, usb_pll, USB_CTRL, PLL_CTRL_ENABLE),
struct clk_fixed_rate *fixed = &clk_hw->f;
clk = clk_register_fixed_rate(NULL, lpc32xx_clk->name,
- parents[0], fixed->flags, fixed->fixed_rate);
+ parents[0], 0, fixed->fixed_rate);
break;
}
default:
Say Y if you want to use peripheral devices such as UART, SPI,
i2C, USB, UFS, SDDC, PCIe, etc.
+config QCS_TURING_404
+ tristate "QCS404 Turing Clock Controller"
+ help
+ Support for the Turing Clock Controller on QCS404, provides clocks
+ and resets for the Turing subsystem.
+
config SDM_GCC_845
tristate "SDM845 Global Clock Controller"
select QCOM_GDSC
obj-$(CONFIG_QCOM_CLK_RPMH) += clk-rpmh.o
obj-$(CONFIG_QCOM_CLK_SMD_RPM) += clk-smd-rpm.o
obj-$(CONFIG_QCS_GCC_404) += gcc-qcs404.o
+obj-$(CONFIG_QCS_TURING_404) += turingcc-qcs404.o
obj-$(CONFIG_SDM_CAMCC_845) += camcc-sdm845.o
obj-$(CONFIG_SDM_DISPCC_845) += dispcc-sdm845.o
obj-$(CONFIG_SDM_GCC_660) += gcc-sdm660.o
};
EXPORT_SYMBOL_GPL(clk_branch2_ops);
+const struct clk_ops clk_branch2_aon_ops = {
+ .enable = clk_branch2_enable,
+ .is_enabled = clk_is_enabled_regmap,
+};
+EXPORT_SYMBOL_GPL(clk_branch2_aon_ops);
+
const struct clk_ops clk_branch_simple_ops = {
.enable = clk_enable_regmap,
.disable = clk_disable_regmap,
extern const struct clk_ops clk_branch_ops;
extern const struct clk_ops clk_branch2_ops;
extern const struct clk_ops clk_branch_simple_ops;
+extern const struct clk_ops clk_branch2_aon_ops;
#define to_clk_branch(_hw) \
container_of(to_clk_regmap(_hw), struct clk_branch, clkr)
-// SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (c) 2017, Linaro Limited
* Author: Georgi Djakov <georgi.djakov@linaro.org>
static struct clk_branch gcc_pcie_0_pipe_clk = {
.halt_reg = 0x6b018,
- .halt_check = BRANCH_HALT,
+ .halt_check = BRANCH_HALT_SKIP,
.clkr = {
.enable_reg = 0x6b018,
.enable_mask = BIT(0),
"core_bi_pll_test_se",
};
+static const struct parent_map gcc_parent_map_16[] = {
+ { P_XO, 0 },
+ { P_GPLL0_OUT_MAIN, 1 },
+ { P_GPLL0_OUT_AUX, 2 },
+ { P_CORE_BI_PLL_TEST_SE, 7 },
+};
+
+static const char * const gcc_parent_names_16[] = {
+ "cxo",
+ "gpll0_out_main",
+ "gpll0_out_aux",
+ "core_bi_pll_test_se",
+};
+
static struct clk_fixed_factor cxo = {
.mult = 1,
.div = 1,
},
};
+static const struct freq_tbl ftbl_cdsp_bimc_clk_src[] = {
+ F(19200000, P_XO, 1, 0, 0),
+ F(133333333, P_GPLL0_OUT_MAIN, 6, 0, 0),
+ F(266666667, P_GPLL0_OUT_MAIN, 3, 0, 0),
+ F(320000000, P_GPLL0_OUT_MAIN, 2.5, 0, 0),
+ { }
+};
+
+static struct clk_rcg2 cdsp_bimc_clk_src = {
+ .cmd_rcgr = 0x5e010,
+ .mnd_width = 0,
+ .hid_width = 5,
+ .parent_map = gcc_parent_map_16,
+ .freq_tbl = ftbl_cdsp_bimc_clk_src,
+ .clkr.hw.init = &(struct clk_init_data) {
+ .name = "cdsp_bimc_clk_src",
+ .parent_names = gcc_parent_names_16,
+ .num_parents = 4,
+ .ops = &clk_rcg2_ops,
+ },
+};
+
static struct clk_branch gcc_apss_ahb_clk = {
.halt_reg = 0x4601c,
.halt_check = BRANCH_HALT_VOTED,
},
};
+static struct clk_branch gcc_bimc_cdsp_clk = {
+ .halt_reg = 0x31030,
+ .halt_check = BRANCH_HALT,
+ .clkr = {
+ .enable_reg = 0x31030,
+ .enable_mask = BIT(0),
+ .hw.init = &(struct clk_init_data) {
+ .name = "gcc_bimc_cdsp_clk",
+ .parent_names = (const char *[]) {
+ "cdsp_bimc_clk_src",
+ },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ .ops = &clk_branch2_ops,
+ },
+ },
+};
+
static struct clk_branch gcc_bimc_mdss_clk = {
.halt_reg = 0x31038,
.halt_check = BRANCH_HALT,
},
};
+static struct clk_branch gcc_cdsp_tbu_clk = {
+ .halt_reg = 0x1203c,
+ .halt_check = BRANCH_VOTED,
+ .clkr = {
+ .enable_reg = 0x13020,
+ .enable_mask = BIT(9),
+ .hw.init = &(struct clk_init_data) {
+ .name = "gcc_cdsp_tbu_clk",
+ .parent_names = (const char *[]) {
+ "cdsp_bimc_clk_src",
+ },
+ .num_parents = 1,
+ .flags = CLK_SET_RATE_PARENT,
+ .ops = &clk_branch2_ops,
+ },
+ },
+};
+
static struct clk_branch gcc_gp1_clk = {
.halt_reg = 0x8000,
.halt_check = BRANCH_HALT,
},
};
+static struct clk_branch gcc_cdsp_cfg_ahb_clk = {
+ .halt_reg = 0x5e004,
+ .halt_check = BRANCH_HALT,
+ .clkr = {
+ .enable_reg = 0x5e004,
+ .enable_mask = BIT(0),
+ .hw.init = &(struct clk_init_data) {
+ .name = "gcc_cdsp_cfg_ahb_cbcr",
+ .ops = &clk_branch2_ops,
+ },
+ },
+};
+
static struct clk_branch gcc_sdcc2_ahb_clk = {
.halt_reg = 0x4301c,
.halt_check = BRANCH_HALT,
[GCC_ESC0_CLK_SRC] = &esc0_clk_src.clkr,
[GCC_APSS_AHB_CLK] = &gcc_apss_ahb_clk.clkr,
[GCC_BIMC_GFX_CLK] = &gcc_bimc_gfx_clk.clkr,
+ [GCC_BIMC_CDSP_CLK] = &gcc_bimc_cdsp_clk.clkr,
[GCC_BIMC_MDSS_CLK] = &gcc_bimc_mdss_clk.clkr,
[GCC_BLSP1_AHB_CLK] = &gcc_blsp1_ahb_clk.clkr,
[GCC_BLSP1_QUP0_I2C_APPS_CLK] = &gcc_blsp1_qup0_i2c_apps_clk.clkr,
[GCC_SDCC1_AHB_CLK] = &gcc_sdcc1_ahb_clk.clkr,
[GCC_SDCC1_APPS_CLK] = &gcc_sdcc1_apps_clk.clkr,
[GCC_SDCC1_ICE_CORE_CLK] = &gcc_sdcc1_ice_core_clk.clkr,
+ [GCC_CDSP_CFG_AHB_CLK] = &gcc_cdsp_cfg_ahb_clk.clkr,
[GCC_SDCC2_AHB_CLK] = &gcc_sdcc2_ahb_clk.clkr,
[GCC_SDCC2_APPS_CLK] = &gcc_sdcc2_apps_clk.clkr,
[GCC_SYS_NOC_USB3_CLK] = &gcc_sys_noc_usb3_clk.clkr,
[GCC_USB3_PHY_AUX_CLK_SRC] = &usb3_phy_aux_clk_src.clkr,
[GCC_USB_HS_SYSTEM_CLK_SRC] = &usb_hs_system_clk_src.clkr,
[GCC_VSYNC_CLK_SRC] = &vsync_clk_src.clkr,
+ [GCC_CDSP_BIMC_CLK_SRC] = &cdsp_bimc_clk_src.clkr,
[GCC_USB_HS_INACTIVITY_TIMERS_CLK] =
&gcc_usb_hs_inactivity_timers_clk.clkr,
[GCC_BIMC_GPU_CLK] = &gcc_bimc_gpu_clk.clkr,
[GCC_GFX_TBU_CLK] = &gcc_gfx_tbu_clk.clkr,
[GCC_SMMU_CFG_CLK] = &gcc_smmu_cfg_clk.clkr,
[GCC_APSS_TCU_CLK] = &gcc_apss_tcu_clk.clkr,
+ [GCC_CDSP_TBU_CLK] = &gcc_cdsp_tbu_clk.clkr,
[GCC_CRYPTO_AHB_CLK] = &gcc_crypto_ahb_clk.clkr,
[GCC_CRYPTO_AXI_CLK] = &gcc_crypto_axi_clk.clkr,
[GCC_CRYPTO_CLK] = &gcc_crypto_clk.clkr,
static const struct qcom_reset_map gcc_qcs404_resets[] = {
[GCC_GENI_IR_BCR] = { 0x0F000 },
+ [GCC_CDSP_RESTART] = { 0x18000 },
[GCC_USB_HS_BCR] = { 0x41000 },
[GCC_USB2_HS_PHY_ONLY_BCR] = { 0x41034 },
[GCC_QUSB2_PHY_BCR] = { 0x4103c },
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2019, Linaro Ltd.
+ */
+
+#include <linux/bitops.h>
+#include <linux/err.h>
+#include <linux/platform_device.h>
+#include <linux/module.h>
+#include <linux/of_address.h>
+#include <linux/pm_clock.h>
+#include <linux/pm_runtime.h>
+#include <linux/regmap.h>
+
+#include <dt-bindings/clock/qcom,turingcc-qcs404.h>
+
+#include "clk-regmap.h"
+#include "clk-branch.h"
+#include "common.h"
+#include "reset.h"
+
+static struct clk_branch turing_wrapper_aon_cbcr = {
+ .halt_reg = 0x5098,
+ .halt_check = BRANCH_HALT,
+ .clkr = {
+ .enable_reg = 0x5098,
+ .enable_mask = BIT(0),
+ .hw.init = &(struct clk_init_data) {
+ .name = "turing_wrapper_aon_clk",
+ .ops = &clk_branch2_aon_ops,
+ },
+ },
+};
+
+static struct clk_branch turing_q6ss_ahbm_aon_cbcr = {
+ .halt_reg = 0x9000,
+ .halt_check = BRANCH_HALT,
+ .clkr = {
+ .enable_reg = 0x9000,
+ .enable_mask = BIT(0),
+ .hw.init = &(struct clk_init_data) {
+ .name = "turing_q6ss_ahbm_aon_cbcr",
+ .ops = &clk_branch2_ops,
+ },
+ },
+};
+
+static struct clk_branch turing_q6ss_q6_axim_clk = {
+ .halt_reg = 0xb000,
+ .halt_check = BRANCH_HALT,
+ .clkr = {
+ .enable_reg = 0xb000,
+ .enable_mask = BIT(0),
+ .hw.init = &(struct clk_init_data) {
+ .name = "turing_q6ss_q6_axim_clk",
+ .ops = &clk_branch2_aon_ops,
+ },
+ },
+};
+
+static struct clk_branch turing_q6ss_ahbs_aon_cbcr = {
+ .halt_reg = 0x10000,
+ .halt_check = BRANCH_HALT,
+ .clkr = {
+ .enable_reg = 0x10000,
+ .enable_mask = BIT(0),
+ .hw.init = &(struct clk_init_data) {
+ .name = "turing_q6ss_ahbs_aon_clk",
+ .ops = &clk_branch2_aon_ops,
+ },
+ },
+};
+
+static struct clk_branch turing_wrapper_qos_ahbs_aon_cbcr = {
+ .halt_reg = 0x11014,
+ .halt_check = BRANCH_HALT,
+ .clkr = {
+ .enable_reg = 0x11014,
+ .enable_mask = BIT(0),
+ .hw.init = &(struct clk_init_data) {
+ .name = "turing_wrapper_qos_ahbs_aon_clk",
+ .ops = &clk_branch2_aon_ops,
+ },
+ },
+};
+
+static struct clk_regmap *turingcc_clocks[] = {
+ [TURING_WRAPPER_AON_CLK] = &turing_wrapper_aon_cbcr.clkr,
+ [TURING_Q6SS_AHBM_AON_CLK] = &turing_q6ss_ahbm_aon_cbcr.clkr,
+ [TURING_Q6SS_Q6_AXIM_CLK] = &turing_q6ss_q6_axim_clk.clkr,
+ [TURING_Q6SS_AHBS_AON_CLK] = &turing_q6ss_ahbs_aon_cbcr.clkr,
+ [TURING_WRAPPER_QOS_AHBS_AON_CLK] = &turing_wrapper_qos_ahbs_aon_cbcr.clkr,
+};
+
+static const struct regmap_config turingcc_regmap_config = {
+ .reg_bits = 32,
+ .reg_stride = 4,
+ .val_bits = 32,
+ .max_register = 0x30000,
+ .fast_io = true,
+};
+
+static const struct qcom_cc_desc turingcc_desc = {
+ .config = &turingcc_regmap_config,
+ .clks = turingcc_clocks,
+ .num_clks = ARRAY_SIZE(turingcc_clocks),
+};
+
+static int turingcc_probe(struct platform_device *pdev)
+{
+ int ret;
+
+ pm_runtime_enable(&pdev->dev);
+ ret = pm_clk_create(&pdev->dev);
+ if (ret)
+ goto disable_pm_runtime;
+
+ ret = pm_clk_add(&pdev->dev, NULL);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "failed to acquire iface clock\n");
+ goto destroy_pm_clk;
+ }
+
+ ret = qcom_cc_probe(pdev, &turingcc_desc);
+ if (ret < 0)
+ goto destroy_pm_clk;
+
+ return 0;
+
+destroy_pm_clk:
+ pm_clk_destroy(&pdev->dev);
+
+disable_pm_runtime:
+ pm_runtime_disable(&pdev->dev);
+
+ return ret;
+}
+
+static int turingcc_remove(struct platform_device *pdev)
+{
+ pm_clk_destroy(&pdev->dev);
+ pm_runtime_disable(&pdev->dev);
+
+ return 0;
+}
+
+static const struct dev_pm_ops turingcc_pm_ops = {
+ SET_RUNTIME_PM_OPS(pm_clk_suspend, pm_clk_resume, NULL)
+};
+
+static const struct of_device_id turingcc_match_table[] = {
+ { .compatible = "qcom,qcs404-turingcc" },
+ { }
+};
+MODULE_DEVICE_TABLE(of, turingcc_match_table);
+
+static struct platform_driver turingcc_driver = {
+ .probe = turingcc_probe,
+ .remove = turingcc_remove,
+ .driver = {
+ .name = "qcs404-turingcc",
+ .of_match_table = turingcc_match_table,
+ .pm = &turingcc_pm_ops,
+ },
+};
+
+module_platform_driver(turingcc_driver);
+
+MODULE_DESCRIPTION("Qualcomm QCS404 Turing Clock Controller");
+MODULE_LICENSE("GPL v2");
#include <linux/clk.h>
#include <linux/clk-provider.h>
+#include <linux/io.h>
#include <dt-bindings/clock/r7s9210-cpg-mssr.h>
#include "renesas-cpg-mssr.h"
if (clk_get_rate(extal_clk) > 12000000)
cpg_mode = 1;
- frqcr = clk_readl(base + CPG_FRQCR) & 0xFFF;
+ frqcr = readl(base + CPG_FRQCR) & 0xFFF;
if (frqcr == 0x012)
index = 0;
else if (frqcr == 0x112)
DEF_GEN3_OSC(".r", CLK_RINT, CLK_EXTAL, 32),
/* Core Clock Outputs */
- DEF_BASE("z", R8A774A1_CLK_Z, CLK_TYPE_GEN3_Z, CLK_PLL0),
- DEF_BASE("z2", R8A774A1_CLK_Z2, CLK_TYPE_GEN3_Z2, CLK_PLL2),
+ DEF_GEN3_Z("z", R8A774A1_CLK_Z, CLK_TYPE_GEN3_Z, CLK_PLL0, 2, 8),
+ DEF_GEN3_Z("z2", R8A774A1_CLK_Z2, CLK_TYPE_GEN3_Z, CLK_PLL2, 2, 0),
DEF_FIXED("ztr", R8A774A1_CLK_ZTR, CLK_PLL1_DIV2, 6, 1),
DEF_FIXED("ztrd2", R8A774A1_CLK_ZTRD2, CLK_PLL1_DIV2, 12, 1),
DEF_FIXED("zt", R8A774A1_CLK_ZT, CLK_PLL1_DIV2, 4, 1),
DEF_MOD("msiof2", 209, R8A774A1_CLK_MSO),
DEF_MOD("msiof1", 210, R8A774A1_CLK_MSO),
DEF_MOD("msiof0", 211, R8A774A1_CLK_MSO),
- DEF_MOD("sys-dmac2", 217, R8A774A1_CLK_S0D3),
- DEF_MOD("sys-dmac1", 218, R8A774A1_CLK_S0D3),
+ DEF_MOD("sys-dmac2", 217, R8A774A1_CLK_S3D1),
+ DEF_MOD("sys-dmac1", 218, R8A774A1_CLK_S3D1),
DEF_MOD("sys-dmac0", 219, R8A774A1_CLK_S0D3),
DEF_MOD("cmt3", 300, R8A774A1_CLK_R),
DEF_MOD("cmt2", 301, R8A774A1_CLK_R),
DEF_MOD("rwdt", 402, R8A774A1_CLK_R),
DEF_MOD("intc-ex", 407, R8A774A1_CLK_CP),
DEF_MOD("intc-ap", 408, R8A774A1_CLK_S0D3),
- DEF_MOD("audmac1", 501, R8A774A1_CLK_S0D3),
- DEF_MOD("audmac0", 502, R8A774A1_CLK_S0D3),
+ DEF_MOD("audmac1", 501, R8A774A1_CLK_S1D2),
+ DEF_MOD("audmac0", 502, R8A774A1_CLK_S1D2),
DEF_MOD("hscif4", 516, R8A774A1_CLK_S3D1),
DEF_MOD("hscif3", 517, R8A774A1_CLK_S3D1),
DEF_MOD("hscif2", 518, R8A774A1_CLK_S3D1),
DEF_MOD("vspd0", 623, R8A774A1_CLK_S0D2),
DEF_MOD("vspb", 626, R8A774A1_CLK_S0D1),
DEF_MOD("vspi0", 631, R8A774A1_CLK_S0D1),
- DEF_MOD("ehci1", 702, R8A774A1_CLK_S3D4),
- DEF_MOD("ehci0", 703, R8A774A1_CLK_S3D4),
- DEF_MOD("hsusb", 704, R8A774A1_CLK_S3D4),
+ DEF_MOD("ehci1", 702, R8A774A1_CLK_S3D2),
+ DEF_MOD("ehci0", 703, R8A774A1_CLK_S3D2),
+ DEF_MOD("hsusb", 704, R8A774A1_CLK_S3D2),
DEF_MOD("csi20", 714, R8A774A1_CLK_CSI0),
DEF_MOD("csi40", 716, R8A774A1_CLK_CSI0),
DEF_MOD("du2", 722, R8A774A1_CLK_S2D1),
/* Core Clock Outputs */
DEF_FIXED("za2", R8A774C0_CLK_ZA2, CLK_PLL0D24, 1, 1),
DEF_FIXED("za8", R8A774C0_CLK_ZA8, CLK_PLL0D8, 1, 1),
+ DEF_GEN3_Z("z2", R8A774C0_CLK_Z2, CLK_TYPE_GEN3_Z, CLK_PLL0, 4, 8),
DEF_FIXED("ztr", R8A774C0_CLK_ZTR, CLK_PLL1, 6, 1),
DEF_FIXED("zt", R8A774C0_CLK_ZT, CLK_PLL1, 4, 1),
DEF_FIXED("zx", R8A774C0_CLK_ZX, CLK_PLL1, 3, 1),
DEF_MOD("intc-ex", 407, R8A774C0_CLK_CP),
DEF_MOD("intc-ap", 408, R8A774C0_CLK_S0D3),
- DEF_MOD("audmac0", 502, R8A774C0_CLK_S3D4),
+ DEF_MOD("audmac0", 502, R8A774C0_CLK_S1D2),
DEF_MOD("hscif4", 516, R8A774C0_CLK_S3D1C),
DEF_MOD("hscif3", 517, R8A774C0_CLK_S3D1C),
DEF_MOD("hscif2", 518, R8A774C0_CLK_S3D1C),
DEF_MOD("vspb", 626, R8A774C0_CLK_S0D1),
DEF_MOD("vspi0", 631, R8A774C0_CLK_S0D1),
- DEF_MOD("ehci0", 703, R8A774C0_CLK_S3D4),
- DEF_MOD("hsusb", 704, R8A774C0_CLK_S3D4),
+ DEF_MOD("ehci0", 703, R8A774C0_CLK_S3D2),
+ DEF_MOD("hsusb", 704, R8A774C0_CLK_S3D2),
DEF_MOD("csi40", 716, R8A774C0_CLK_CSI0),
DEF_MOD("du1", 723, R8A774C0_CLK_S1D1),
DEF_MOD("du0", 724, R8A774C0_CLK_S1D1),
* r8a7795 Clock Pulse Generator / Module Standby and Software Reset
*
* Copyright (C) 2015 Glider bvba
+ * Copyright (C) 2018-2019 Renesas Electronics Corp.
*
* Based on clk-rcar-gen3.c
*
DEF_GEN3_OSC(".r", CLK_RINT, CLK_EXTAL, 32),
/* Core Clock Outputs */
- DEF_BASE("z", R8A7795_CLK_Z, CLK_TYPE_GEN3_Z, CLK_PLL0),
- DEF_BASE("z2", R8A7795_CLK_Z2, CLK_TYPE_GEN3_Z2, CLK_PLL2),
+ DEF_GEN3_Z("z", R8A7795_CLK_Z, CLK_TYPE_GEN3_Z, CLK_PLL0, 2, 8),
+ DEF_GEN3_Z("z2", R8A7795_CLK_Z2, CLK_TYPE_GEN3_Z, CLK_PLL2, 2, 0),
DEF_FIXED("ztr", R8A7795_CLK_ZTR, CLK_PLL1_DIV2, 6, 1),
DEF_FIXED("ztrd2", R8A7795_CLK_ZTRD2, CLK_PLL1_DIV2, 12, 1),
DEF_FIXED("zt", R8A7795_CLK_ZT, CLK_PLL1_DIV2, 4, 1),
DEF_MOD("msiof2", 209, R8A7795_CLK_MSO),
DEF_MOD("msiof1", 210, R8A7795_CLK_MSO),
DEF_MOD("msiof0", 211, R8A7795_CLK_MSO),
- DEF_MOD("sys-dmac2", 217, R8A7795_CLK_S0D3),
- DEF_MOD("sys-dmac1", 218, R8A7795_CLK_S0D3),
+ DEF_MOD("sys-dmac2", 217, R8A7795_CLK_S3D1),
+ DEF_MOD("sys-dmac1", 218, R8A7795_CLK_S3D1),
DEF_MOD("sys-dmac0", 219, R8A7795_CLK_S0D3),
DEF_MOD("sceg-pub", 229, R8A7795_CLK_CR),
DEF_MOD("cmt3", 300, R8A7795_CLK_R),
DEF_MOD("rwdt", 402, R8A7795_CLK_R),
DEF_MOD("intc-ex", 407, R8A7795_CLK_CP),
DEF_MOD("intc-ap", 408, R8A7795_CLK_S0D3),
- DEF_MOD("audmac1", 501, R8A7795_CLK_S0D3),
- DEF_MOD("audmac0", 502, R8A7795_CLK_S0D3),
- DEF_MOD("drif7", 508, R8A7795_CLK_S3D2),
- DEF_MOD("drif6", 509, R8A7795_CLK_S3D2),
- DEF_MOD("drif5", 510, R8A7795_CLK_S3D2),
- DEF_MOD("drif4", 511, R8A7795_CLK_S3D2),
- DEF_MOD("drif3", 512, R8A7795_CLK_S3D2),
- DEF_MOD("drif2", 513, R8A7795_CLK_S3D2),
- DEF_MOD("drif1", 514, R8A7795_CLK_S3D2),
- DEF_MOD("drif0", 515, R8A7795_CLK_S3D2),
+ DEF_MOD("audmac1", 501, R8A7795_CLK_S1D2),
+ DEF_MOD("audmac0", 502, R8A7795_CLK_S1D2),
+ DEF_MOD("drif31", 508, R8A7795_CLK_S3D2),
+ DEF_MOD("drif30", 509, R8A7795_CLK_S3D2),
+ DEF_MOD("drif21", 510, R8A7795_CLK_S3D2),
+ DEF_MOD("drif20", 511, R8A7795_CLK_S3D2),
+ DEF_MOD("drif11", 512, R8A7795_CLK_S3D2),
+ DEF_MOD("drif10", 513, R8A7795_CLK_S3D2),
+ DEF_MOD("drif01", 514, R8A7795_CLK_S3D2),
+ DEF_MOD("drif00", 515, R8A7795_CLK_S3D2),
DEF_MOD("hscif4", 516, R8A7795_CLK_S3D1),
DEF_MOD("hscif3", 517, R8A7795_CLK_S3D1),
DEF_MOD("hscif2", 518, R8A7795_CLK_S3D1),
DEF_MOD("vspi2", 629, R8A7795_CLK_S2D1), /* ES1.x */
DEF_MOD("vspi1", 630, R8A7795_CLK_S0D1),
DEF_MOD("vspi0", 631, R8A7795_CLK_S0D1),
- DEF_MOD("ehci3", 700, R8A7795_CLK_S3D4),
- DEF_MOD("ehci2", 701, R8A7795_CLK_S3D4),
- DEF_MOD("ehci1", 702, R8A7795_CLK_S3D4),
- DEF_MOD("ehci0", 703, R8A7795_CLK_S3D4),
- DEF_MOD("hsusb", 704, R8A7795_CLK_S3D4),
- DEF_MOD("hsusb3", 705, R8A7795_CLK_S3D4),
+ DEF_MOD("ehci3", 700, R8A7795_CLK_S3D2),
+ DEF_MOD("ehci2", 701, R8A7795_CLK_S3D2),
+ DEF_MOD("ehci1", 702, R8A7795_CLK_S3D2),
+ DEF_MOD("ehci0", 703, R8A7795_CLK_S3D2),
+ DEF_MOD("hsusb", 704, R8A7795_CLK_S3D2),
+ DEF_MOD("hsusb3", 705, R8A7795_CLK_S3D2),
DEF_MOD("csi21", 713, R8A7795_CLK_CSI0), /* ES1.x */
DEF_MOD("csi20", 714, R8A7795_CLK_CSI0),
DEF_MOD("csi41", 715, R8A7795_CLK_CSI0),
* r8a7796 Clock Pulse Generator / Module Standby and Software Reset
*
* Copyright (C) 2016 Glider bvba
+ * Copyright (C) 2018 Renesas Electronics Corp.
*
* Based on r8a7795-cpg-mssr.c
*
DEF_GEN3_OSC(".r", CLK_RINT, CLK_EXTAL, 32),
/* Core Clock Outputs */
- DEF_BASE("z", R8A7796_CLK_Z, CLK_TYPE_GEN3_Z, CLK_PLL0),
- DEF_BASE("z2", R8A7796_CLK_Z2, CLK_TYPE_GEN3_Z2, CLK_PLL2),
+ DEF_GEN3_Z("z", R8A7796_CLK_Z, CLK_TYPE_GEN3_Z, CLK_PLL0, 2, 8),
+ DEF_GEN3_Z("z2", R8A7796_CLK_Z2, CLK_TYPE_GEN3_Z, CLK_PLL2, 2, 0),
DEF_FIXED("ztr", R8A7796_CLK_ZTR, CLK_PLL1_DIV2, 6, 1),
DEF_FIXED("ztrd2", R8A7796_CLK_ZTRD2, CLK_PLL1_DIV2, 12, 1),
DEF_FIXED("zt", R8A7796_CLK_ZT, CLK_PLL1_DIV2, 4, 1),
DEF_MOD("msiof2", 209, R8A7796_CLK_MSO),
DEF_MOD("msiof1", 210, R8A7796_CLK_MSO),
DEF_MOD("msiof0", 211, R8A7796_CLK_MSO),
- DEF_MOD("sys-dmac2", 217, R8A7796_CLK_S0D3),
- DEF_MOD("sys-dmac1", 218, R8A7796_CLK_S0D3),
+ DEF_MOD("sys-dmac2", 217, R8A7796_CLK_S3D1),
+ DEF_MOD("sys-dmac1", 218, R8A7796_CLK_S3D1),
DEF_MOD("sys-dmac0", 219, R8A7796_CLK_S0D3),
DEF_MOD("cmt3", 300, R8A7796_CLK_R),
DEF_MOD("cmt2", 301, R8A7796_CLK_R),
DEF_MOD("rwdt", 402, R8A7796_CLK_R),
DEF_MOD("intc-ex", 407, R8A7796_CLK_CP),
DEF_MOD("intc-ap", 408, R8A7796_CLK_S0D3),
- DEF_MOD("audmac1", 501, R8A7796_CLK_S0D3),
- DEF_MOD("audmac0", 502, R8A7796_CLK_S0D3),
- DEF_MOD("drif7", 508, R8A7796_CLK_S3D2),
- DEF_MOD("drif6", 509, R8A7796_CLK_S3D2),
- DEF_MOD("drif5", 510, R8A7796_CLK_S3D2),
- DEF_MOD("drif4", 511, R8A7796_CLK_S3D2),
- DEF_MOD("drif3", 512, R8A7796_CLK_S3D2),
- DEF_MOD("drif2", 513, R8A7796_CLK_S3D2),
- DEF_MOD("drif1", 514, R8A7796_CLK_S3D2),
- DEF_MOD("drif0", 515, R8A7796_CLK_S3D2),
+ DEF_MOD("audmac1", 501, R8A7796_CLK_S1D2),
+ DEF_MOD("audmac0", 502, R8A7796_CLK_S1D2),
+ DEF_MOD("drif31", 508, R8A7796_CLK_S3D2),
+ DEF_MOD("drif30", 509, R8A7796_CLK_S3D2),
+ DEF_MOD("drif21", 510, R8A7796_CLK_S3D2),
+ DEF_MOD("drif20", 511, R8A7796_CLK_S3D2),
+ DEF_MOD("drif11", 512, R8A7796_CLK_S3D2),
+ DEF_MOD("drif10", 513, R8A7796_CLK_S3D2),
+ DEF_MOD("drif01", 514, R8A7796_CLK_S3D2),
+ DEF_MOD("drif00", 515, R8A7796_CLK_S3D2),
DEF_MOD("hscif4", 516, R8A7796_CLK_S3D1),
DEF_MOD("hscif3", 517, R8A7796_CLK_S3D1),
DEF_MOD("hscif2", 518, R8A7796_CLK_S3D1),
DEF_MOD("vspd0", 623, R8A7796_CLK_S0D2),
DEF_MOD("vspb", 626, R8A7796_CLK_S0D1),
DEF_MOD("vspi0", 631, R8A7796_CLK_S0D1),
- DEF_MOD("ehci1", 702, R8A7796_CLK_S3D4),
- DEF_MOD("ehci0", 703, R8A7796_CLK_S3D4),
- DEF_MOD("hsusb", 704, R8A7796_CLK_S3D4),
+ DEF_MOD("ehci1", 702, R8A7796_CLK_S3D2),
+ DEF_MOD("ehci0", 703, R8A7796_CLK_S3D2),
+ DEF_MOD("hsusb", 704, R8A7796_CLK_S3D2),
DEF_MOD("csi20", 714, R8A7796_CLK_CSI0),
DEF_MOD("csi40", 716, R8A7796_CLK_CSI0),
DEF_MOD("du2", 722, R8A7796_CLK_S2D1),
* r8a77965 Clock Pulse Generator / Module Standby and Software Reset
*
* Copyright (C) 2018 Jacopo Mondi <jacopo+renesas@jmondi.org>
+ * Copyright (C) 2019 Renesas Electronics Corp.
*
* Based on r8a7795-cpg-mssr.c
*
DEF_GEN3_OSC(".r", CLK_RINT, CLK_EXTAL, 32),
/* Core Clock Outputs */
- DEF_BASE("z", R8A77965_CLK_Z, CLK_TYPE_GEN3_Z, CLK_PLL0),
+ DEF_GEN3_Z("z", R8A77965_CLK_Z, CLK_TYPE_GEN3_Z, CLK_PLL0, 2, 8),
DEF_FIXED("ztr", R8A77965_CLK_ZTR, CLK_PLL1_DIV2, 6, 1),
DEF_FIXED("ztrd2", R8A77965_CLK_ZTRD2, CLK_PLL1_DIV2, 12, 1),
DEF_FIXED("zt", R8A77965_CLK_ZT, CLK_PLL1_DIV2, 4, 1),
DEF_MOD("msiof2", 209, R8A77965_CLK_MSO),
DEF_MOD("msiof1", 210, R8A77965_CLK_MSO),
DEF_MOD("msiof0", 211, R8A77965_CLK_MSO),
- DEF_MOD("sys-dmac2", 217, R8A77965_CLK_S0D3),
- DEF_MOD("sys-dmac1", 218, R8A77965_CLK_S0D3),
+ DEF_MOD("sys-dmac2", 217, R8A77965_CLK_S3D1),
+ DEF_MOD("sys-dmac1", 218, R8A77965_CLK_S3D1),
DEF_MOD("sys-dmac0", 219, R8A77965_CLK_S0D3),
DEF_MOD("cmt3", 300, R8A77965_CLK_R),
DEF_MOD("intc-ex", 407, R8A77965_CLK_CP),
DEF_MOD("intc-ap", 408, R8A77965_CLK_S0D3),
- DEF_MOD("audmac1", 501, R8A77965_CLK_S0D3),
- DEF_MOD("audmac0", 502, R8A77965_CLK_S0D3),
- DEF_MOD("drif7", 508, R8A77965_CLK_S3D2),
- DEF_MOD("drif6", 509, R8A77965_CLK_S3D2),
- DEF_MOD("drif5", 510, R8A77965_CLK_S3D2),
- DEF_MOD("drif4", 511, R8A77965_CLK_S3D2),
- DEF_MOD("drif3", 512, R8A77965_CLK_S3D2),
- DEF_MOD("drif2", 513, R8A77965_CLK_S3D2),
- DEF_MOD("drif1", 514, R8A77965_CLK_S3D2),
- DEF_MOD("drif0", 515, R8A77965_CLK_S3D2),
+ DEF_MOD("audmac1", 501, R8A77965_CLK_S1D2),
+ DEF_MOD("audmac0", 502, R8A77965_CLK_S1D2),
+ DEF_MOD("drif31", 508, R8A77965_CLK_S3D2),
+ DEF_MOD("drif30", 509, R8A77965_CLK_S3D2),
+ DEF_MOD("drif21", 510, R8A77965_CLK_S3D2),
+ DEF_MOD("drif20", 511, R8A77965_CLK_S3D2),
+ DEF_MOD("drif11", 512, R8A77965_CLK_S3D2),
+ DEF_MOD("drif10", 513, R8A77965_CLK_S3D2),
+ DEF_MOD("drif01", 514, R8A77965_CLK_S3D2),
+ DEF_MOD("drif00", 515, R8A77965_CLK_S3D2),
DEF_MOD("hscif4", 516, R8A77965_CLK_S3D1),
DEF_MOD("hscif3", 517, R8A77965_CLK_S3D1),
DEF_MOD("hscif2", 518, R8A77965_CLK_S3D1),
DEF_MOD("vspb", 626, R8A77965_CLK_S0D1),
DEF_MOD("vspi0", 631, R8A77965_CLK_S0D1),
- DEF_MOD("ehci1", 702, R8A77965_CLK_S3D4),
- DEF_MOD("ehci0", 703, R8A77965_CLK_S3D4),
- DEF_MOD("hsusb", 704, R8A77965_CLK_S3D4),
+ DEF_MOD("ehci1", 702, R8A77965_CLK_S3D2),
+ DEF_MOD("ehci0", 703, R8A77965_CLK_S3D2),
+ DEF_MOD("hsusb", 704, R8A77965_CLK_S3D2),
DEF_MOD("csi20", 714, R8A77965_CLK_CSI0),
DEF_MOD("csi40", 716, R8A77965_CLK_CSI0),
DEF_MOD("du3", 721, R8A77965_CLK_S2D1),
DEF_MOD("gpio1", 911, R8A77980_CLK_CP),
DEF_MOD("gpio0", 912, R8A77980_CLK_CP),
DEF_MOD("can-fd", 914, R8A77980_CLK_S3D2),
- DEF_MOD("rpc-if", 917, R8A77980_CLK_RPC),
+ DEF_MOD("rpc-if", 917, R8A77980_CLK_RPCD2),
DEF_MOD("i2c4", 927, R8A77980_CLK_S0D6),
DEF_MOD("i2c3", 928, R8A77980_CLK_S0D6),
DEF_MOD("i2c2", 929, R8A77980_CLK_S3D2),
/*
* r8a77990 Clock Pulse Generator / Module Standby and Software Reset
*
- * Copyright (C) 2018 Renesas Electronics Corp.
+ * Copyright (C) 2018-2019 Renesas Electronics Corp.
*
* Based on r8a7795-cpg-mssr.c
*
/* Core Clock Outputs */
DEF_FIXED("za2", R8A77990_CLK_ZA2, CLK_PLL0D24, 1, 1),
DEF_FIXED("za8", R8A77990_CLK_ZA8, CLK_PLL0D8, 1, 1),
+ DEF_GEN3_Z("z2", R8A77990_CLK_Z2, CLK_TYPE_GEN3_Z, CLK_PLL0, 4, 8),
DEF_FIXED("ztr", R8A77990_CLK_ZTR, CLK_PLL1, 6, 1),
DEF_FIXED("zt", R8A77990_CLK_ZT, CLK_PLL1, 4, 1),
DEF_FIXED("zx", R8A77990_CLK_ZX, CLK_PLL1, 3, 1),
DEF_MOD("intc-ex", 407, R8A77990_CLK_CP),
DEF_MOD("intc-ap", 408, R8A77990_CLK_S0D3),
- DEF_MOD("audmac0", 502, R8A77990_CLK_S3D4),
- DEF_MOD("drif7", 508, R8A77990_CLK_S3D2),
- DEF_MOD("drif6", 509, R8A77990_CLK_S3D2),
- DEF_MOD("drif5", 510, R8A77990_CLK_S3D2),
- DEF_MOD("drif4", 511, R8A77990_CLK_S3D2),
- DEF_MOD("drif3", 512, R8A77990_CLK_S3D2),
- DEF_MOD("drif2", 513, R8A77990_CLK_S3D2),
- DEF_MOD("drif1", 514, R8A77990_CLK_S3D2),
- DEF_MOD("drif0", 515, R8A77990_CLK_S3D2),
+ DEF_MOD("audmac0", 502, R8A77990_CLK_S1D2),
+ DEF_MOD("drif31", 508, R8A77990_CLK_S3D2),
+ DEF_MOD("drif30", 509, R8A77990_CLK_S3D2),
+ DEF_MOD("drif21", 510, R8A77990_CLK_S3D2),
+ DEF_MOD("drif20", 511, R8A77990_CLK_S3D2),
+ DEF_MOD("drif11", 512, R8A77990_CLK_S3D2),
+ DEF_MOD("drif10", 513, R8A77990_CLK_S3D2),
+ DEF_MOD("drif01", 514, R8A77990_CLK_S3D2),
+ DEF_MOD("drif00", 515, R8A77990_CLK_S3D2),
DEF_MOD("hscif4", 516, R8A77990_CLK_S3D1C),
DEF_MOD("hscif3", 517, R8A77990_CLK_S3D1C),
DEF_MOD("hscif2", 518, R8A77990_CLK_S3D1C),
DEF_MOD("vspb", 626, R8A77990_CLK_S0D1),
DEF_MOD("vspi0", 631, R8A77990_CLK_S0D1),
- DEF_MOD("ehci0", 703, R8A77990_CLK_S3D4),
- DEF_MOD("hsusb", 704, R8A77990_CLK_S3D4),
+ DEF_MOD("ehci0", 703, R8A77990_CLK_S3D2),
+ DEF_MOD("hsusb", 704, R8A77990_CLK_S3D2),
DEF_MOD("csi40", 716, R8A77990_CLK_CSI0),
DEF_MOD("du1", 723, R8A77990_CLK_S1D1),
DEF_MOD("du0", 724, R8A77990_CLK_S1D1),
DEF_MOD("rwdt", 402, R8A77995_CLK_R),
DEF_MOD("intc-ex", 407, R8A77995_CLK_CP),
DEF_MOD("intc-ap", 408, R8A77995_CLK_S1D2),
- DEF_MOD("audmac0", 502, R8A77995_CLK_S3D1),
+ DEF_MOD("audmac0", 502, R8A77995_CLK_S1D2),
DEF_MOD("hscif3", 517, R8A77995_CLK_S3D1C),
DEF_MOD("hscif0", 520, R8A77995_CLK_S3D1C),
DEF_MOD("thermal", 522, R8A77995_CLK_CP),
D_GATE(CLK_P6_PG2, "clk_p6_pg2", DIV_P6_PG, 0x8a3, 0x8a4, 0x8a5, 0, 0xb61, 0, 0),
D_GATE(CLK_P6_PG3, "clk_p6_pg3", DIV_P6_PG, 0x8a6, 0x8a7, 0x8a8, 0, 0xb62, 0, 0),
D_GATE(CLK_P6_PG4, "clk_p6_pg4", DIV_P6_PG, 0x8a9, 0x8aa, 0x8ab, 0, 0xb63, 0, 0),
+ D_GATE(CLK_PCI_USB, "clk_pci_usb", CLKOUT_D40, 0xe6, 0, 0, 0, 0, 0, 0),
D_GATE(CLK_QSPI0, "clk_qspi0", DIV_QSPI0, 0x2a4, 0x2a5, 0, 0, 0, 0, 0),
D_GATE(CLK_QSPI1, "clk_qspi1", DIV_QSPI1, 0x484, 0x485, 0, 0, 0, 0, 0),
D_GATE(CLK_RGMII_REF, "clk_rgmii_ref", CLKOUT_D8, 0x340, 0, 0, 0, 0, 0, 0),
-/* SPDX-License-Identifier: GPL-2.0
- *
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
* R-Car Gen2 Clock Pulse Generator
*
* Copyright (C) 2016 Cogent Embedded Inc.
* R-Car Gen3 Clock Pulse Generator
*
* Copyright (C) 2015-2018 Glider bvba
+ * Copyright (C) 2019 Renesas Electronics Corp.
*
* Based on clk-rcar-gen3.c
*
#define CPG_FRQCRB 0x00000004
#define CPG_FRQCRB_KICK BIT(31)
#define CPG_FRQCRC 0x000000e0
-#define CPG_FRQCRC_ZFC_MASK GENMASK(12, 8)
-#define CPG_FRQCRC_Z2FC_MASK GENMASK(4, 0)
struct cpg_z_clk {
struct clk_hw hw;
void __iomem *reg;
void __iomem *kick_reg;
unsigned long mask;
+ unsigned int fixed_div;
};
#define to_z_clk(_hw) container_of(_hw, struct cpg_z_clk, hw)
val = readl(zclk->reg) & zclk->mask;
mult = 32 - (val >> __ffs(zclk->mask));
- /* Factor of 2 is for fixed divider */
- return DIV_ROUND_CLOSEST_ULL((u64)parent_rate * mult, 32 * 2);
+ return DIV_ROUND_CLOSEST_ULL((u64)parent_rate * mult,
+ 32 * zclk->fixed_div);
}
static long cpg_z_clk_round_rate(struct clk_hw *hw, unsigned long rate,
unsigned long *parent_rate)
{
- /* Factor of 2 is for fixed divider */
- unsigned long prate = *parent_rate / 2;
+ struct cpg_z_clk *zclk = to_z_clk(hw);
+ unsigned long prate;
unsigned int mult;
+ prate = *parent_rate / zclk->fixed_div;
mult = div_u64(rate * 32ULL, prate);
mult = clamp(mult, 1U, 32U);
unsigned int mult;
unsigned int i;
- /* Factor of 2 is for fixed divider */
- mult = DIV_ROUND_CLOSEST_ULL(rate * 32ULL * 2, parent_rate);
+ mult = DIV64_U64_ROUND_CLOSEST(rate * 32ULL * zclk->fixed_div,
+ parent_rate);
mult = clamp(mult, 1U, 32U);
if (readl(zclk->kick_reg) & CPG_FRQCRB_KICK)
static struct clk * __init cpg_z_clk_register(const char *name,
const char *parent_name,
void __iomem *reg,
- unsigned long mask)
+ unsigned int div,
+ unsigned int offset)
{
struct clk_init_data init;
struct cpg_z_clk *zclk;
zclk->reg = reg + CPG_FRQCRC;
zclk->kick_reg = reg + CPG_FRQCRB;
zclk->hw.init = &init;
- zclk->mask = mask;
+ zclk->mask = GENMASK(offset + 4, offset);
+ zclk->fixed_div = div; /* PLLVCO x 1/div x SYS-CPU divider */
clk = clk_register(NULL, &zclk->hw);
if (IS_ERR(clk))
const struct sd_div_table *div_table;
struct cpg_simple_notifier csn;
unsigned int div_num;
- unsigned int div_min;
- unsigned int div_max;
unsigned int cur_div_idx;
};
unsigned long rate,
unsigned long parent_rate)
{
- unsigned int div;
-
- if (!rate)
- rate = 1;
-
- div = DIV_ROUND_CLOSEST(parent_rate, rate);
+ unsigned long calc_rate, diff, diff_min = ULONG_MAX;
+ unsigned int i, best_div = 0;
+
+ for (i = 0; i < clock->div_num; i++) {
+ calc_rate = DIV_ROUND_CLOSEST(parent_rate,
+ clock->div_table[i].div);
+ diff = calc_rate > rate ? calc_rate - rate : rate - calc_rate;
+ if (diff < diff_min) {
+ best_div = clock->div_table[i].div;
+ diff_min = diff;
+ }
+ }
- return clamp_t(unsigned int, div, clock->div_min, clock->div_max);
+ return best_div;
}
static long cpg_sd_clock_round_rate(struct clk_hw *hw, unsigned long rate,
#define RCKCR_CKSEL BIT(1) /* Manual RCLK parent selection */
#define SD_SKIP_FIRST BIT(2) /* Skip first clock in SD table */
-static struct clk * __init cpg_sd_clk_register(const struct cpg_core_clk *core,
- void __iomem *base, const char *parent_name,
+static struct clk * __init cpg_sd_clk_register(const char *name,
+ void __iomem *base, unsigned int offset, const char *parent_name,
struct raw_notifier_head *notifiers)
{
struct clk_init_data init;
struct sd_clock *clock;
struct clk *clk;
- unsigned int i;
u32 val;
clock = kzalloc(sizeof(*clock), GFP_KERNEL);
if (!clock)
return ERR_PTR(-ENOMEM);
- init.name = core->name;
+ init.name = name;
init.ops = &cpg_sd_clock_ops;
init.flags = CLK_SET_RATE_PARENT;
init.parent_names = &parent_name;
init.num_parents = 1;
- clock->csn.reg = base + core->offset;
+ clock->csn.reg = base + offset;
clock->hw.init = &init;
clock->div_table = cpg_sd_div_table;
clock->div_num = ARRAY_SIZE(cpg_sd_div_table);
val |= CPG_SD_STP_MASK | (clock->div_table[0].val & CPG_SD_FC_MASK);
writel(val, clock->csn.reg);
- clock->div_max = clock->div_table[0].div;
- clock->div_min = clock->div_max;
- for (i = 1; i < clock->div_num; i++) {
- clock->div_max = max(clock->div_max, clock->div_table[i].div);
- clock->div_min = min(clock->div_min, clock->div_table[i].div);
- }
-
clk = clk_register(NULL, &clock->hw);
if (IS_ERR(clk))
goto free_clock;
break;
case CLK_TYPE_GEN3_SD:
- return cpg_sd_clk_register(core, base, __clk_get_name(parent),
- notifiers);
+ return cpg_sd_clk_register(core->name, base, core->offset,
+ __clk_get_name(parent), notifiers);
case CLK_TYPE_GEN3_R:
if (cpg_quirks & RCKCR_CKSEL) {
case CLK_TYPE_GEN3_Z:
return cpg_z_clk_register(core->name, __clk_get_name(parent),
- base, CPG_FRQCRC_ZFC_MASK);
-
- case CLK_TYPE_GEN3_Z2:
- return cpg_z_clk_register(core->name, __clk_get_name(parent),
- base, CPG_FRQCRC_Z2FC_MASK);
+ base, core->div, core->offset);
case CLK_TYPE_GEN3_OSC:
/*
-/* SPDX-License-Identifier: GPL-2.0
- *
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
* R-Car Gen3 Clock Pulse Generator
*
* Copyright (C) 2015-2018 Glider bvba
+ * Copyright (C) 2018 Renesas Electronics Corp.
*
*/
CLK_TYPE_GEN3_R,
CLK_TYPE_GEN3_MDSEL, /* Select parent/divider using mode pin */
CLK_TYPE_GEN3_Z,
- CLK_TYPE_GEN3_Z2,
CLK_TYPE_GEN3_OSC, /* OSC EXTAL predivider and fixed divider */
CLK_TYPE_GEN3_RCKSEL, /* Select parent/divider using RCKCR.CKSEL */
CLK_TYPE_GEN3_RPCSRC,
DEF_BASE(_name, _id, CLK_TYPE_GEN3_RCKSEL, \
(_parent0) << 16 | (_parent1), .div = (_div0) << 16 | (_div1))
+#define DEF_GEN3_Z(_name, _id, _type, _parent, _div, _offset) \
+ DEF_BASE(_name, _id, _type, _parent, .div = _div, .offset = _offset)
+
struct rcar_gen3_cpg_pll_config {
u8 extal_div;
u8 pll1_mult;
-/* SPDX-License-Identifier: GPL-2.0
- *
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
* Renesas Clock Pulse Generator / Module Standby and Software Reset
*
* Copyright (C) 2015 Glider bvba
struct rockchip_ddrclk *ddrclk = to_rockchip_ddrclk_hw(hw);
u32 val;
- val = clk_readl(ddrclk->reg_base +
+ val = readl(ddrclk->reg_base +
ddrclk->mux_offset) >> ddrclk->mux_shift;
val &= GENMASK(ddrclk->mux_width - 1, 0);
struct clk_divider *divider = to_clk_divider(hw);
unsigned int val;
- val = clk_readl(divider->reg) >> divider->shift;
+ val = readl(divider->reg) >> divider->shift;
val &= div_mask(divider->width);
val = val * 2 + 3;
if (divider->flags & CLK_DIVIDER_HIWORD_MASK) {
val = div_mask(divider->width) << (divider->shift + 16);
} else {
- val = clk_readl(divider->reg);
+ val = readl(divider->reg);
val &= ~(div_mask(divider->width) << divider->shift);
}
val |= value << divider->shift;
- clk_writel(val, divider->reg);
+ writel(val, divider->reg);
if (divider->lock)
spin_unlock_irqrestore(divider->lock, flags);
PNAME(mux_pll_src_cpll_gpll_p) = { "cpll", "gpll" };
PNAME(mux_pll_src_npll_cpll_gpll_p) = { "npll", "cpll", "gpll" };
PNAME(mux_pll_src_cpll_gpll_npll_p) = { "cpll", "gpll", "npll" };
-PNAME(mux_pll_src_cpll_gpll_usb480m_p) = { "cpll", "gpll", "usbphy480m_src" };
-PNAME(mux_pll_src_cpll_gll_usb_npll_p) = { "cpll", "gpll", "usbphy480m_src", "npll" };
+PNAME(mux_pll_src_cpll_gpll_usb480m_p) = { "cpll", "gpll", "unstable:usbphy480m_src" };
+PNAME(mux_pll_src_cpll_gll_usb_npll_p) = { "cpll", "gpll", "unstable:usbphy480m_src", "npll" };
PNAME(mux_mmc_src_p) = { "cpll", "gpll", "xin24m", "xin24m" };
PNAME(mux_i2s_pre_p) = { "i2s_src", "i2s_frac", "ext_i2s", "xin12m" };
PNAME(mux_edp_24m_p) = { "ext_edp_24m", "xin24m" };
PNAME(mux_tspout_p) = { "cpll", "gpll", "npll", "xin27m" };
-PNAME(mux_aclk_vcodec_pre_p) = { "aclk_vepu", "aclk_vdpu" };
+PNAME(mux_aclk_vcodec_pre_p) = { "aclk_vdpu", "aclk_vepu" };
PNAME(mux_usbphy480m_p) = { "sclk_otgphy1_480m", "sclk_otgphy2_480m",
"sclk_otgphy0_480m" };
PNAME(mux_hsicphy480m_p) = { "cpll", "gpll", "usbphy480m_src" };
COMPOSITE_NOMUX(0, "aclk_core_mp", "armclk", CLK_IGNORE_UNUSED,
RK3288_CLKSEL_CON(0), 4, 4, DFLAGS | CLK_DIVIDER_READ_ONLY,
RK3288_CLKGATE_CON(12), 6, GFLAGS),
- COMPOSITE_NOMUX(0, "atclk", "armclk", CLK_IGNORE_UNUSED,
+ COMPOSITE_NOMUX(0, "atclk", "armclk", 0,
RK3288_CLKSEL_CON(37), 4, 5, DFLAGS | CLK_DIVIDER_READ_ONLY,
RK3288_CLKGATE_CON(12), 7, GFLAGS),
COMPOSITE_NOMUX(0, "pclk_dbg_pre", "armclk", CLK_IGNORE_UNUSED,
RK3288_CLKSEL_CON(37), 9, 5, DFLAGS | CLK_DIVIDER_READ_ONLY,
RK3288_CLKGATE_CON(12), 8, GFLAGS),
- GATE(0, "pclk_dbg", "pclk_dbg_pre", CLK_IGNORE_UNUSED,
+ GATE(0, "pclk_dbg", "pclk_dbg_pre", 0,
RK3288_CLKGATE_CON(12), 9, GFLAGS),
GATE(0, "cs_dbg", "pclk_dbg_pre", CLK_IGNORE_UNUSED,
RK3288_CLKGATE_CON(12), 10, GFLAGS),
COMPOSITE(0, "aclk_vdpu", mux_pll_src_cpll_gpll_usb480m_p, 0,
RK3288_CLKSEL_CON(32), 14, 2, MFLAGS, 8, 5, DFLAGS,
RK3288_CLKGATE_CON(3), 11, GFLAGS),
- MUXGRF(0, "aclk_vcodec_pre", mux_aclk_vcodec_pre_p, 0,
+ MUXGRF(0, "aclk_vcodec_pre", mux_aclk_vcodec_pre_p, CLK_SET_RATE_PARENT,
RK3288_GRF_SOC_CON(0), 7, 1, MFLAGS),
GATE(ACLK_VCODEC, "aclk_vcodec", "aclk_vcodec_pre", 0,
RK3288_CLKGATE_CON(9), 0, GFLAGS),
INVERTER(SCLK_HSADC, "sclk_hsadc", "sclk_hsadc_out",
RK3288_CLKSEL_CON(22), 7, IFLAGS),
- GATE(0, "jtag", "ext_jtag", CLK_IGNORE_UNUSED,
+ GATE(0, "jtag", "ext_jtag", 0,
RK3288_CLKGATE_CON(4), 14, GFLAGS),
COMPOSITE_NODIV(SCLK_USBPHY480M_SRC, "usbphy480m_src", mux_usbphy480m_p, 0,
COMPOSITE_NODIV(SCLK_HSICPHY480M, "sclk_hsicphy480m", mux_hsicphy480m_p, 0,
RK3288_CLKSEL_CON(29), 0, 2, MFLAGS,
RK3288_CLKGATE_CON(3), 6, GFLAGS),
- GATE(0, "hsicphy12m_xin12m", "xin12m", CLK_IGNORE_UNUSED,
+ GATE(0, "hsicphy12m_xin12m", "xin12m", 0,
RK3288_CLKGATE_CON(13), 9, GFLAGS),
DIV(0, "hsicphy12m_usbphy", "sclk_hsicphy480m", 0,
RK3288_CLKSEL_CON(11), 8, 6, DFLAGS),
GATE(PCLK_TZPC, "pclk_tzpc", "pclk_cpu", 0, RK3288_CLKGATE_CON(11), 3, GFLAGS),
GATE(PCLK_UART2, "pclk_uart2", "pclk_cpu", 0, RK3288_CLKGATE_CON(11), 9, GFLAGS),
GATE(PCLK_EFUSE256, "pclk_efuse_256", "pclk_cpu", 0, RK3288_CLKGATE_CON(11), 10, GFLAGS),
- GATE(PCLK_RKPWM, "pclk_rkpwm", "pclk_cpu", CLK_IGNORE_UNUSED, RK3288_CLKGATE_CON(11), 11, GFLAGS),
+ GATE(PCLK_RKPWM, "pclk_rkpwm", "pclk_cpu", 0, RK3288_CLKGATE_CON(11), 11, GFLAGS),
/* ddrctrl [DDR Controller PHY clock] gates */
GATE(0, "nclk_ddrupctl0", "ddrphy", CLK_IGNORE_UNUSED, RK3288_CLKGATE_CON(11), 4, GFLAGS),
"pclk_alive_niu",
"pclk_pd_pmu",
"pclk_pmu_niu",
- "pclk_core_niu",
- "pclk_ddrupctl0",
- "pclk_publ0",
- "pclk_ddrupctl1",
- "pclk_publ1",
"pmu_hclk_otg0",
+ /* pwm-regulators on some boards, so handoff-critical later */
+ "pclk_rkpwm",
};
static void __iomem *rk3288_cru_base;
RK3288_CLKSEL_CON(10),
RK3288_CLKSEL_CON(33),
RK3288_CLKSEL_CON(37),
+
+ /* We turn aclk_dmac1 on for suspend; this will restore it */
+ RK3288_CLKGATE_CON(10),
};
static u32 rk3288_saved_cru_regs[ARRAY_SIZE(rk3288_saved_cru_reg_ids)];
readl_relaxed(rk3288_cru_base + reg_id);
}
+ /*
+ * Going into deep sleep (specifically setting PMU_CLR_DMA in
+ * RK3288_PMU_PWRMODE_CON1) appears to fail unless
+ * "aclk_dmac1" is on.
+ */
+ writel_relaxed(1 << (12 + 16),
+ rk3288_cru_base + RK3288_CLKGATE_CON(10));
+
/*
* Switch PLLs other than DPLL (for SDRAM) to slow mode to
* avoid crashes on resume. The Mask ROM on the system will
RK3328_CLKSEL_CON(35), 15, 1, MFLAGS, 8, 7, DFLAGS,
RK3328_CLKGATE_CON(2), 12, GFLAGS),
COMPOSITE(SCLK_CRYPTO, "clk_crypto", mux_2plls_p, 0,
- RK3328_CLKSEL_CON(20), 7, 1, MFLAGS, 0, 7, DFLAGS,
+ RK3328_CLKSEL_CON(20), 7, 1, MFLAGS, 0, 5, DFLAGS,
RK3328_CLKGATE_CON(2), 4, GFLAGS),
COMPOSITE_NOMUX(SCLK_TSADC, "clk_tsadc", "clk_24m", 0,
RK3328_CLKSEL_CON(22), 0, 10, DFLAGS,
GATE(0, "hclk_rkvenc_niu", "hclk_rkvenc", 0,
RK3328_CLKGATE_CON(25), 1, GFLAGS),
GATE(ACLK_H265, "aclk_h265", "aclk_rkvenc", 0,
- RK3328_CLKGATE_CON(25), 0, GFLAGS),
+ RK3328_CLKGATE_CON(25), 2, GFLAGS),
GATE(PCLK_H265, "pclk_h265", "hclk_rkvenc", 0,
- RK3328_CLKGATE_CON(25), 1, GFLAGS),
+ RK3328_CLKGATE_CON(25), 3, GFLAGS),
GATE(ACLK_H264, "aclk_h264", "aclk_rkvenc", 0,
- RK3328_CLKGATE_CON(25), 0, GFLAGS),
+ RK3328_CLKGATE_CON(25), 4, GFLAGS),
GATE(HCLK_H264, "hclk_h264", "hclk_rkvenc", 0,
- RK3328_CLKGATE_CON(25), 1, GFLAGS),
+ RK3328_CLKGATE_CON(25), 5, GFLAGS),
GATE(ACLK_AXISRAM, "aclk_axisram", "aclk_rkvenc", CLK_IGNORE_UNUSED,
- RK3328_CLKGATE_CON(25), 0, GFLAGS),
+ RK3328_CLKGATE_CON(25), 6, GFLAGS),
COMPOSITE(SCLK_VENC_CORE, "sclk_venc_core", mux_4plls_p, 0,
RK3328_CLKSEL_CON(51), 14, 2, MFLAGS, 8, 5, DFLAGS,
/* PD_GMAC */
COMPOSITE(ACLK_GMAC, "aclk_gmac", mux_2plls_hdmiphy_p, 0,
- RK3328_CLKSEL_CON(35), 6, 2, MFLAGS, 0, 5, DFLAGS,
+ RK3328_CLKSEL_CON(25), 6, 2, MFLAGS, 0, 5, DFLAGS,
RK3328_CLKGATE_CON(3), 2, GFLAGS),
COMPOSITE_NOMUX(PCLK_GMAC, "pclk_gmac", "aclk_gmac", 0,
RK3328_CLKSEL_CON(25), 8, 3, DFLAGS,
/* PD_PERI */
GATE(0, "aclk_peri_noc", "aclk_peri", CLK_IGNORE_UNUSED, RK3328_CLKGATE_CON(19), 11, GFLAGS),
- GATE(ACLK_USB3OTG, "aclk_usb3otg", "aclk_peri", 0, RK3328_CLKGATE_CON(19), 4, GFLAGS),
+ GATE(ACLK_USB3OTG, "aclk_usb3otg", "aclk_peri", 0, RK3328_CLKGATE_CON(19), 14, GFLAGS),
GATE(HCLK_SDMMC, "hclk_sdmmc", "hclk_peri", 0, RK3328_CLKGATE_CON(19), 0, GFLAGS),
GATE(HCLK_SDIO, "hclk_sdio", "hclk_peri", 0, RK3328_CLKGATE_CON(19), 1, GFLAGS),
&rk3328_cpuclk_data, rk3328_cpuclk_rates,
ARRAY_SIZE(rk3328_cpuclk_rates));
- rockchip_register_softrst(np, 11, reg_base + RK3328_SOFTRST_CON(0),
+ rockchip_register_softrst(np, 12, reg_base + RK3328_SOFTRST_CON(0),
ROCKCHIP_SOFTRST_HIWORD_MASK);
rockchip_register_restart_notifier(ctx, RK3328_GLB_SRST_FST, NULL);
const char *const *parent_names, u8 num_parents,
void __iomem *base,
int muxdiv_offset, u8 mux_shift, u8 mux_width, u8 mux_flags,
- u8 div_shift, u8 div_width, u8 div_flags,
+ int div_offset, u8 div_shift, u8 div_width, u8 div_flags,
struct clk_div_table *div_table, int gate_offset,
u8 gate_shift, u8 gate_flags, unsigned long flags,
spinlock_t *lock)
}
div->flags = div_flags;
- div->reg = base + muxdiv_offset;
+ if (div_offset)
+ div->reg = base + div_offset;
+ else
+ div->reg = base + muxdiv_offset;
div->shift = div_shift;
div->width = div_width;
div->lock = lock;
ctx->reg_base, list->muxdiv_offset,
list->mux_shift,
list->mux_width, list->mux_flags,
- list->div_shift, list->div_width,
+ list->div_offset, list->div_shift, list->div_width,
list->div_flags, list->div_table,
list->gate_offset, list->gate_shift,
list->gate_flags, flags, &ctx->lock);
u8 mux_shift;
u8 mux_width;
u8 mux_flags;
+ int div_offset;
u8 div_shift;
u8 div_width;
u8 div_flags;
.gate_flags = gf, \
}
+#define COMPOSITE_DIV_OFFSET(_id, cname, pnames, f, mo, ms, mw, \
+ mf, do, ds, dw, df, go, gs, gf) \
+ { \
+ .id = _id, \
+ .branch_type = branch_composite, \
+ .name = cname, \
+ .parent_names = pnames, \
+ .num_parents = ARRAY_SIZE(pnames), \
+ .flags = f, \
+ .muxdiv_offset = mo, \
+ .mux_shift = ms, \
+ .mux_width = mw, \
+ .mux_flags = mf, \
+ .div_offset = do, \
+ .div_shift = ds, \
+ .div_width = dw, \
+ .div_flags = df, \
+ .gate_offset = go, \
+ .gate_shift = gs, \
+ .gate_flags = gf, \
+ }
+
#define COMPOSITE_NOMUX(_id, cname, pname, f, mo, ds, dw, df, \
go, gs, gf) \
{ \
GATE(CLK_USI1, "usi1", "aclk66", GATE_IP_PERIC, 11, 0, 0),
GATE(CLK_USI2, "usi2", "aclk66", GATE_IP_PERIC, 12, 0, 0),
GATE(CLK_USI3, "usi3", "aclk66", GATE_IP_PERIC, 13, 0, 0),
+ GATE(CLK_TSADC, "tsadc", "aclk66", GATE_IP_PERIC, 15, 0, 0),
GATE(CLK_PWM, "pwm", "aclk66", GATE_IP_PERIC, 24, 0, 0),
GATE(CLK_SCLK_UART0, "sclk_uart0", "div_uart0",
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+
+menuconfig CLK_SIFIVE
+ bool "SiFive SoC driver support"
+ help
+ SoC drivers for SiFive Linux-capable SoCs.
+
+if CLK_SIFIVE
+
+config CLK_SIFIVE_FU540_PRCI
+ bool "PRCI driver for SiFive FU540 SoCs"
+ select CLK_ANALOGBITS_WRPLL_CLN28HPC
+ help
+ Supports the Power Reset Clock interface (PRCI) IP block found in
+ FU540 SoCs. If this kernel is meant to run on a SiFive FU540 SoC,
+ enable this driver.
+
+endif
--- /dev/null
+obj-$(CONFIG_CLK_SIFIVE_FU540_PRCI) += fu540-prci.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2018-2019 SiFive, Inc.
+ * Wesley Terpstra
+ * Paul Walmsley
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * The FU540 PRCI implements clock and reset control for the SiFive
+ * FU540-C000 chip. This driver assumes that it has sole control
+ * over all PRCI resources.
+ *
+ * This driver is based on the PRCI driver written by Wesley Terpstra:
+ * https://github.com/riscv/riscv-linux/commit/999529edf517ed75b56659d456d221b2ee56bb60
+ *
+ * References:
+ * - SiFive FU540-C000 manual v1p0, Chapter 7 "Clocking and Reset"
+ */
+
+#include <dt-bindings/clock/sifive-fu540-prci.h>
+#include <linux/clkdev.h>
+#include <linux/clk-provider.h>
+#include <linux/clk/analogbits-wrpll-cln28hpc.h>
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_clk.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+
+/*
+ * EXPECTED_CLK_PARENT_COUNT: how many parent clocks this driver expects:
+ * hfclk and rtcclk
+ */
+#define EXPECTED_CLK_PARENT_COUNT 2
+
+/*
+ * Register offsets and bitmasks
+ */
+
+/* COREPLLCFG0 */
+#define PRCI_COREPLLCFG0_OFFSET 0x4
+# define PRCI_COREPLLCFG0_DIVR_SHIFT 0
+# define PRCI_COREPLLCFG0_DIVR_MASK (0x3f << PRCI_COREPLLCFG0_DIVR_SHIFT)
+# define PRCI_COREPLLCFG0_DIVF_SHIFT 6
+# define PRCI_COREPLLCFG0_DIVF_MASK (0x1ff << PRCI_COREPLLCFG0_DIVF_SHIFT)
+# define PRCI_COREPLLCFG0_DIVQ_SHIFT 15
+# define PRCI_COREPLLCFG0_DIVQ_MASK (0x7 << PRCI_COREPLLCFG0_DIVQ_SHIFT)
+# define PRCI_COREPLLCFG0_RANGE_SHIFT 18
+# define PRCI_COREPLLCFG0_RANGE_MASK (0x7 << PRCI_COREPLLCFG0_RANGE_SHIFT)
+# define PRCI_COREPLLCFG0_BYPASS_SHIFT 24
+# define PRCI_COREPLLCFG0_BYPASS_MASK (0x1 << PRCI_COREPLLCFG0_BYPASS_SHIFT)
+# define PRCI_COREPLLCFG0_FSE_SHIFT 25
+# define PRCI_COREPLLCFG0_FSE_MASK (0x1 << PRCI_COREPLLCFG0_FSE_SHIFT)
+# define PRCI_COREPLLCFG0_LOCK_SHIFT 31
+# define PRCI_COREPLLCFG0_LOCK_MASK (0x1 << PRCI_COREPLLCFG0_LOCK_SHIFT)
+
+/* DDRPLLCFG0 */
+#define PRCI_DDRPLLCFG0_OFFSET 0xc
+# define PRCI_DDRPLLCFG0_DIVR_SHIFT 0
+# define PRCI_DDRPLLCFG0_DIVR_MASK (0x3f << PRCI_DDRPLLCFG0_DIVR_SHIFT)
+# define PRCI_DDRPLLCFG0_DIVF_SHIFT 6
+# define PRCI_DDRPLLCFG0_DIVF_MASK (0x1ff << PRCI_DDRPLLCFG0_DIVF_SHIFT)
+# define PRCI_DDRPLLCFG0_DIVQ_SHIFT 15
+# define PRCI_DDRPLLCFG0_DIVQ_MASK (0x7 << PRCI_DDRPLLCFG0_DIVQ_SHIFT)
+# define PRCI_DDRPLLCFG0_RANGE_SHIFT 18
+# define PRCI_DDRPLLCFG0_RANGE_MASK (0x7 << PRCI_DDRPLLCFG0_RANGE_SHIFT)
+# define PRCI_DDRPLLCFG0_BYPASS_SHIFT 24
+# define PRCI_DDRPLLCFG0_BYPASS_MASK (0x1 << PRCI_DDRPLLCFG0_BYPASS_SHIFT)
+# define PRCI_DDRPLLCFG0_FSE_SHIFT 25
+# define PRCI_DDRPLLCFG0_FSE_MASK (0x1 << PRCI_DDRPLLCFG0_FSE_SHIFT)
+# define PRCI_DDRPLLCFG0_LOCK_SHIFT 31
+# define PRCI_DDRPLLCFG0_LOCK_MASK (0x1 << PRCI_DDRPLLCFG0_LOCK_SHIFT)
+
+/* DDRPLLCFG1 */
+#define PRCI_DDRPLLCFG1_OFFSET 0x10
+# define PRCI_DDRPLLCFG1_CKE_SHIFT 24
+# define PRCI_DDRPLLCFG1_CKE_MASK (0x1 << PRCI_DDRPLLCFG1_CKE_SHIFT)
+
+/* GEMGXLPLLCFG0 */
+#define PRCI_GEMGXLPLLCFG0_OFFSET 0x1c
+# define PRCI_GEMGXLPLLCFG0_DIVR_SHIFT 0
+# define PRCI_GEMGXLPLLCFG0_DIVR_MASK (0x3f << PRCI_GEMGXLPLLCFG0_DIVR_SHIFT)
+# define PRCI_GEMGXLPLLCFG0_DIVF_SHIFT 6
+# define PRCI_GEMGXLPLLCFG0_DIVF_MASK (0x1ff << PRCI_GEMGXLPLLCFG0_DIVF_SHIFT)
+# define PRCI_GEMGXLPLLCFG0_DIVQ_SHIFT 15
+# define PRCI_GEMGXLPLLCFG0_DIVQ_MASK (0x7 << PRCI_GEMGXLPLLCFG0_DIVQ_SHIFT)
+# define PRCI_GEMGXLPLLCFG0_RANGE_SHIFT 18
+# define PRCI_GEMGXLPLLCFG0_RANGE_MASK (0x7 << PRCI_GEMGXLPLLCFG0_RANGE_SHIFT)
+# define PRCI_GEMGXLPLLCFG0_BYPASS_SHIFT 24
+# define PRCI_GEMGXLPLLCFG0_BYPASS_MASK (0x1 << PRCI_GEMGXLPLLCFG0_BYPASS_SHIFT)
+# define PRCI_GEMGXLPLLCFG0_FSE_SHIFT 25
+# define PRCI_GEMGXLPLLCFG0_FSE_MASK (0x1 << PRCI_GEMGXLPLLCFG0_FSE_SHIFT)
+# define PRCI_GEMGXLPLLCFG0_LOCK_SHIFT 31
+# define PRCI_GEMGXLPLLCFG0_LOCK_MASK (0x1 << PRCI_GEMGXLPLLCFG0_LOCK_SHIFT)
+
+/* GEMGXLPLLCFG1 */
+#define PRCI_GEMGXLPLLCFG1_OFFSET 0x20
+# define PRCI_GEMGXLPLLCFG1_CKE_SHIFT 24
+# define PRCI_GEMGXLPLLCFG1_CKE_MASK (0x1 << PRCI_GEMGXLPLLCFG1_CKE_SHIFT)
+
+/* CORECLKSEL */
+#define PRCI_CORECLKSEL_OFFSET 0x24
+# define PRCI_CORECLKSEL_CORECLKSEL_SHIFT 0
+# define PRCI_CORECLKSEL_CORECLKSEL_MASK (0x1 << PRCI_CORECLKSEL_CORECLKSEL_SHIFT)
+
+/* DEVICESRESETREG */
+#define PRCI_DEVICESRESETREG_OFFSET 0x28
+# define PRCI_DEVICESRESETREG_DDR_CTRL_RST_N_SHIFT 0
+# define PRCI_DEVICESRESETREG_DDR_CTRL_RST_N_MASK (0x1 << PRCI_DEVICESRESETREG_DDR_CTRL_RST_N_SHIFT)
+# define PRCI_DEVICESRESETREG_DDR_AXI_RST_N_SHIFT 1
+# define PRCI_DEVICESRESETREG_DDR_AXI_RST_N_MASK (0x1 << PRCI_DEVICESRESETREG_DDR_AXI_RST_N_SHIFT)
+# define PRCI_DEVICESRESETREG_DDR_AHB_RST_N_SHIFT 2
+# define PRCI_DEVICESRESETREG_DDR_AHB_RST_N_MASK (0x1 << PRCI_DEVICESRESETREG_DDR_AHB_RST_N_SHIFT)
+# define PRCI_DEVICESRESETREG_DDR_PHY_RST_N_SHIFT 3
+# define PRCI_DEVICESRESETREG_DDR_PHY_RST_N_MASK (0x1 << PRCI_DEVICESRESETREG_DDR_PHY_RST_N_SHIFT)
+# define PRCI_DEVICESRESETREG_GEMGXL_RST_N_SHIFT 5
+# define PRCI_DEVICESRESETREG_GEMGXL_RST_N_MASK (0x1 << PRCI_DEVICESRESETREG_GEMGXL_RST_N_SHIFT)
+
+/* CLKMUXSTATUSREG */
+#define PRCI_CLKMUXSTATUSREG_OFFSET 0x2c
+# define PRCI_CLKMUXSTATUSREG_TLCLKSEL_STATUS_SHIFT 1
+# define PRCI_CLKMUXSTATUSREG_TLCLKSEL_STATUS_MASK (0x1 << PRCI_CLKMUXSTATUSREG_TLCLKSEL_STATUS_SHIFT)
+
+/*
+ * Private structures
+ */
+
+/**
+ * struct __prci_data - per-device-instance data
+ * @va: base virtual address of the PRCI IP block
+ * @hw_clks: encapsulates struct clk_hw records
+ *
+ * PRCI per-device instance data
+ */
+struct __prci_data {
+ void __iomem *va;
+ struct clk_hw_onecell_data hw_clks;
+};
+
+/**
+ * struct __prci_wrpll_data - WRPLL configuration and integration data
+ * @c: WRPLL current configuration record
+ * @enable_bypass: fn ptr to code to bypass the WRPLL (if applicable; else NULL)
+ * @disable_bypass: fn ptr to code to not bypass the WRPLL (or NULL)
+ * @cfg0_offs: WRPLL CFG0 register offset (in bytes) from the PRCI base address
+ *
+ * @enable_bypass and @disable_bypass are used for WRPLL instances
+ * that contain a separate external glitchless clock mux downstream
+ * from the PLL. The WRPLL internal bypass mux is not glitchless.
+ */
+struct __prci_wrpll_data {
+ struct wrpll_cfg c;
+ void (*enable_bypass)(struct __prci_data *pd);
+ void (*disable_bypass)(struct __prci_data *pd);
+ u8 cfg0_offs;
+};
+
+/**
+ * struct __prci_clock - describes a clock device managed by PRCI
+ * @name: user-readable clock name string - should match the manual
+ * @parent_name: parent name for this clock
+ * @ops: struct clk_ops for the Linux clock framework to use for control
+ * @hw: Linux-private clock data
+ * @pwd: WRPLL-specific data, associated with this clock (if not NULL)
+ * @pd: PRCI-specific data associated with this clock (if not NULL)
+ *
+ * PRCI clock data. Used by the PRCI driver to register PRCI-provided
+ * clocks to the Linux clock infrastructure.
+ */
+struct __prci_clock {
+ const char *name;
+ const char *parent_name;
+ const struct clk_ops *ops;
+ struct clk_hw hw;
+ struct __prci_wrpll_data *pwd;
+ struct __prci_data *pd;
+};
+
+#define clk_hw_to_prci_clock(pwd) container_of(pwd, struct __prci_clock, hw)
+
+/*
+ * Private functions
+ */
+
+/**
+ * __prci_readl() - read from a PRCI register
+ * @pd: PRCI context
+ * @offs: register offset to read from (in bytes, from PRCI base address)
+ *
+ * Read the register located at offset @offs from the base virtual
+ * address of the PRCI register target described by @pd, and return
+ * the value to the caller.
+ *
+ * Context: Any context.
+ *
+ * Return: the contents of the register described by @pd and @offs.
+ */
+static u32 __prci_readl(struct __prci_data *pd, u32 offs)
+{
+ return readl_relaxed(pd->va + offs);
+}
+
+static void __prci_writel(u32 v, u32 offs, struct __prci_data *pd)
+{
+ writel_relaxed(v, pd->va + offs);
+}
+
+/* WRPLL-related private functions */
+
+/**
+ * __prci_wrpll_unpack() - unpack WRPLL configuration registers into parameters
+ * @c: ptr to a struct wrpll_cfg record to write config into
+ * @r: value read from the PRCI PLL configuration register
+ *
+ * Given a value @r read from an FU540 PRCI PLL configuration register,
+ * split it into fields and populate it into the WRPLL configuration record
+ * pointed to by @c.
+ *
+ * The COREPLLCFG0 macros are used below, but the other *PLLCFG0 macros
+ * have the same register layout.
+ *
+ * Context: Any context.
+ */
+static void __prci_wrpll_unpack(struct wrpll_cfg *c, u32 r)
+{
+ u32 v;
+
+ v = r & PRCI_COREPLLCFG0_DIVR_MASK;
+ v >>= PRCI_COREPLLCFG0_DIVR_SHIFT;
+ c->divr = v;
+
+ v = r & PRCI_COREPLLCFG0_DIVF_MASK;
+ v >>= PRCI_COREPLLCFG0_DIVF_SHIFT;
+ c->divf = v;
+
+ v = r & PRCI_COREPLLCFG0_DIVQ_MASK;
+ v >>= PRCI_COREPLLCFG0_DIVQ_SHIFT;
+ c->divq = v;
+
+ v = r & PRCI_COREPLLCFG0_RANGE_MASK;
+ v >>= PRCI_COREPLLCFG0_RANGE_SHIFT;
+ c->range = v;
+
+ c->flags &= (WRPLL_FLAGS_INT_FEEDBACK_MASK |
+ WRPLL_FLAGS_EXT_FEEDBACK_MASK);
+
+ /* external feedback mode not supported */
+ c->flags |= WRPLL_FLAGS_INT_FEEDBACK_MASK;
+}
+
+/**
+ * __prci_wrpll_pack() - pack PLL configuration parameters into a register value
+ * @c: pointer to a struct wrpll_cfg record containing the PLL's cfg
+ *
+ * Using a set of WRPLL configuration values pointed to by @c,
+ * assemble a PRCI PLL configuration register value, and return it to
+ * the caller.
+ *
+ * Context: Any context. Caller must ensure that the contents of the
+ * record pointed to by @c do not change during the execution
+ * of this function.
+ *
+ * Returns: a value suitable for writing into a PRCI PLL configuration
+ * register
+ */
+static u32 __prci_wrpll_pack(const struct wrpll_cfg *c)
+{
+ u32 r = 0;
+
+ r |= c->divr << PRCI_COREPLLCFG0_DIVR_SHIFT;
+ r |= c->divf << PRCI_COREPLLCFG0_DIVF_SHIFT;
+ r |= c->divq << PRCI_COREPLLCFG0_DIVQ_SHIFT;
+ r |= c->range << PRCI_COREPLLCFG0_RANGE_SHIFT;
+
+ /* external feedback mode not supported */
+ r |= PRCI_COREPLLCFG0_FSE_MASK;
+
+ return r;
+}
+
+/**
+ * __prci_wrpll_read_cfg() - read the WRPLL configuration from the PRCI
+ * @pd: PRCI context
+ * @pwd: PRCI WRPLL metadata
+ *
+ * Read the current configuration of the PLL identified by @pwd from
+ * the PRCI identified by @pd, and store it into the local configuration
+ * cache in @pwd.
+ *
+ * Context: Any context. Caller must prevent the records pointed to by
+ * @pd and @pwd from changing during execution.
+ */
+static void __prci_wrpll_read_cfg(struct __prci_data *pd,
+ struct __prci_wrpll_data *pwd)
+{
+ __prci_wrpll_unpack(&pwd->c, __prci_readl(pd, pwd->cfg0_offs));
+}
+
+/**
+ * __prci_wrpll_write_cfg() - write WRPLL configuration into the PRCI
+ * @pd: PRCI context
+ * @pwd: PRCI WRPLL metadata
+ * @c: WRPLL configuration record to write
+ *
+ * Write the WRPLL configuration described by @c into the WRPLL
+ * configuration register identified by @pwd in the PRCI instance
+ * described by @c. Make a cached copy of the WRPLL's current
+ * configuration so it can be used by other code.
+ *
+ * Context: Any context. Caller must prevent the records pointed to by
+ * @pd and @pwd from changing during execution.
+ */
+static void __prci_wrpll_write_cfg(struct __prci_data *pd,
+ struct __prci_wrpll_data *pwd,
+ struct wrpll_cfg *c)
+{
+ __prci_writel(__prci_wrpll_pack(c), pwd->cfg0_offs, pd);
+
+ memcpy(&pwd->c, c, sizeof(*c));
+}
+
+/* Core clock mux control */
+
+/**
+ * __prci_coreclksel_use_hfclk() - switch the CORECLK mux to output HFCLK
+ * @pd: struct __prci_data * for the PRCI containing the CORECLK mux reg
+ *
+ * Switch the CORECLK mux to the HFCLK input source; return once complete.
+ *
+ * Context: Any context. Caller must prevent concurrent changes to the
+ * PRCI_CORECLKSEL_OFFSET register.
+ */
+static void __prci_coreclksel_use_hfclk(struct __prci_data *pd)
+{
+ u32 r;
+
+ r = __prci_readl(pd, PRCI_CORECLKSEL_OFFSET);
+ r |= PRCI_CORECLKSEL_CORECLKSEL_MASK;
+ __prci_writel(r, PRCI_CORECLKSEL_OFFSET, pd);
+
+ r = __prci_readl(pd, PRCI_CORECLKSEL_OFFSET); /* barrier */
+}
+
+/**
+ * __prci_coreclksel_use_corepll() - switch the CORECLK mux to output COREPLL
+ * @pd: struct __prci_data * for the PRCI containing the CORECLK mux reg
+ *
+ * Switch the CORECLK mux to the PLL output clock; return once complete.
+ *
+ * Context: Any context. Caller must prevent concurrent changes to the
+ * PRCI_CORECLKSEL_OFFSET register.
+ */
+static void __prci_coreclksel_use_corepll(struct __prci_data *pd)
+{
+ u32 r;
+
+ r = __prci_readl(pd, PRCI_CORECLKSEL_OFFSET);
+ r &= ~PRCI_CORECLKSEL_CORECLKSEL_MASK;
+ __prci_writel(r, PRCI_CORECLKSEL_OFFSET, pd);
+
+ r = __prci_readl(pd, PRCI_CORECLKSEL_OFFSET); /* barrier */
+}
+
+/*
+ * Linux clock framework integration
+ *
+ * See the Linux clock framework documentation for more information on
+ * these functions.
+ */
+
+static unsigned long sifive_fu540_prci_wrpll_recalc_rate(struct clk_hw *hw,
+ unsigned long parent_rate)
+{
+ struct __prci_clock *pc = clk_hw_to_prci_clock(hw);
+ struct __prci_wrpll_data *pwd = pc->pwd;
+
+ return wrpll_calc_output_rate(&pwd->c, parent_rate);
+}
+
+static long sifive_fu540_prci_wrpll_round_rate(struct clk_hw *hw,
+ unsigned long rate,
+ unsigned long *parent_rate)
+{
+ struct __prci_clock *pc = clk_hw_to_prci_clock(hw);
+ struct __prci_wrpll_data *pwd = pc->pwd;
+ struct wrpll_cfg c;
+
+ memcpy(&c, &pwd->c, sizeof(c));
+
+ wrpll_configure_for_rate(&c, rate, *parent_rate);
+
+ return wrpll_calc_output_rate(&c, *parent_rate);
+}
+
+static int sifive_fu540_prci_wrpll_set_rate(struct clk_hw *hw,
+ unsigned long rate,
+ unsigned long parent_rate)
+{
+ struct __prci_clock *pc = clk_hw_to_prci_clock(hw);
+ struct __prci_wrpll_data *pwd = pc->pwd;
+ struct __prci_data *pd = pc->pd;
+ int r;
+
+ r = wrpll_configure_for_rate(&pwd->c, rate, parent_rate);
+ if (r)
+ return r;
+
+ if (pwd->enable_bypass)
+ pwd->enable_bypass(pd);
+
+ __prci_wrpll_write_cfg(pd, pwd, &pwd->c);
+
+ udelay(wrpll_calc_max_lock_us(&pwd->c));
+
+ if (pwd->disable_bypass)
+ pwd->disable_bypass(pd);
+
+ return 0;
+}
+
+static const struct clk_ops sifive_fu540_prci_wrpll_clk_ops = {
+ .set_rate = sifive_fu540_prci_wrpll_set_rate,
+ .round_rate = sifive_fu540_prci_wrpll_round_rate,
+ .recalc_rate = sifive_fu540_prci_wrpll_recalc_rate,
+};
+
+static const struct clk_ops sifive_fu540_prci_wrpll_ro_clk_ops = {
+ .recalc_rate = sifive_fu540_prci_wrpll_recalc_rate,
+};
+
+/* TLCLKSEL clock integration */
+
+static unsigned long sifive_fu540_prci_tlclksel_recalc_rate(struct clk_hw *hw,
+ unsigned long parent_rate)
+{
+ struct __prci_clock *pc = clk_hw_to_prci_clock(hw);
+ struct __prci_data *pd = pc->pd;
+ u32 v;
+ u8 div;
+
+ v = __prci_readl(pd, PRCI_CLKMUXSTATUSREG_OFFSET);
+ v &= PRCI_CLKMUXSTATUSREG_TLCLKSEL_STATUS_MASK;
+ div = v ? 1 : 2;
+
+ return div_u64(parent_rate, div);
+}
+
+static const struct clk_ops sifive_fu540_prci_tlclksel_clk_ops = {
+ .recalc_rate = sifive_fu540_prci_tlclksel_recalc_rate,
+};
+
+/*
+ * PRCI integration data for each WRPLL instance
+ */
+
+static struct __prci_wrpll_data __prci_corepll_data = {
+ .cfg0_offs = PRCI_COREPLLCFG0_OFFSET,
+ .enable_bypass = __prci_coreclksel_use_hfclk,
+ .disable_bypass = __prci_coreclksel_use_corepll,
+};
+
+static struct __prci_wrpll_data __prci_ddrpll_data = {
+ .cfg0_offs = PRCI_DDRPLLCFG0_OFFSET,
+};
+
+static struct __prci_wrpll_data __prci_gemgxlpll_data = {
+ .cfg0_offs = PRCI_GEMGXLPLLCFG0_OFFSET,
+};
+
+/*
+ * List of clock controls provided by the PRCI
+ */
+
+static struct __prci_clock __prci_init_clocks[] = {
+ [PRCI_CLK_COREPLL] = {
+ .name = "corepll",
+ .parent_name = "hfclk",
+ .ops = &sifive_fu540_prci_wrpll_clk_ops,
+ .pwd = &__prci_corepll_data,
+ },
+ [PRCI_CLK_DDRPLL] = {
+ .name = "ddrpll",
+ .parent_name = "hfclk",
+ .ops = &sifive_fu540_prci_wrpll_ro_clk_ops,
+ .pwd = &__prci_ddrpll_data,
+ },
+ [PRCI_CLK_GEMGXLPLL] = {
+ .name = "gemgxlpll",
+ .parent_name = "hfclk",
+ .ops = &sifive_fu540_prci_wrpll_clk_ops,
+ .pwd = &__prci_gemgxlpll_data,
+ },
+ [PRCI_CLK_TLCLK] = {
+ .name = "tlclk",
+ .parent_name = "corepll",
+ .ops = &sifive_fu540_prci_tlclksel_clk_ops,
+ },
+};
+
+/**
+ * __prci_register_clocks() - register clock controls in the PRCI with Linux
+ * @dev: Linux struct device *
+ *
+ * Register the list of clock controls described in __prci_init_plls[] with
+ * the Linux clock framework.
+ *
+ * Return: 0 upon success or a negative error code upon failure.
+ */
+static int __prci_register_clocks(struct device *dev, struct __prci_data *pd)
+{
+ struct clk_init_data init = { };
+ struct __prci_clock *pic;
+ int parent_count, i, r;
+
+ parent_count = of_clk_get_parent_count(dev->of_node);
+ if (parent_count != EXPECTED_CLK_PARENT_COUNT) {
+ dev_err(dev, "expected only two parent clocks, found %d\n",
+ parent_count);
+ return -EINVAL;
+ }
+
+ /* Register PLLs */
+ for (i = 0; i < ARRAY_SIZE(__prci_init_clocks); ++i) {
+ pic = &__prci_init_clocks[i];
+
+ init.name = pic->name;
+ init.parent_names = &pic->parent_name;
+ init.num_parents = 1;
+ init.ops = pic->ops;
+ pic->hw.init = &init;
+
+ pic->pd = pd;
+
+ if (pic->pwd)
+ __prci_wrpll_read_cfg(pd, pic->pwd);
+
+ r = devm_clk_hw_register(dev, &pic->hw);
+ if (r) {
+ dev_warn(dev, "Failed to register clock %s: %d\n",
+ init.name, r);
+ return r;
+ }
+
+ r = clk_hw_register_clkdev(&pic->hw, pic->name, dev_name(dev));
+ if (r) {
+ dev_warn(dev, "Failed to register clkdev for %s: %d\n",
+ init.name, r);
+ return r;
+ }
+
+ pd->hw_clks.hws[i] = &pic->hw;
+ }
+
+ pd->hw_clks.num = i;
+
+ r = devm_of_clk_add_hw_provider(dev, of_clk_hw_onecell_get,
+ &pd->hw_clks);
+ if (r) {
+ dev_err(dev, "could not add hw_provider: %d\n", r);
+ return r;
+ }
+
+ return 0;
+}
+
+/*
+ * Linux device model integration
+ *
+ * See the Linux device model documentation for more information about
+ * these functions.
+ */
+static int sifive_fu540_prci_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct resource *res;
+ struct __prci_data *pd;
+ int r;
+
+ pd = devm_kzalloc(dev, sizeof(*pd), GFP_KERNEL);
+ if (!pd)
+ return -ENOMEM;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ pd->va = devm_ioremap_resource(dev, res);
+ if (IS_ERR(pd->va))
+ return PTR_ERR(pd->va);
+
+ r = __prci_register_clocks(dev, pd);
+ if (r) {
+ dev_err(dev, "could not register clocks: %d\n", r);
+ return r;
+ }
+
+ dev_dbg(dev, "SiFive FU540 PRCI probed\n");
+
+ return 0;
+}
+
+static const struct of_device_id sifive_fu540_prci_of_match[] = {
+ { .compatible = "sifive,fu540-c000-prci", },
+ {}
+};
+MODULE_DEVICE_TABLE(of, sifive_fu540_prci_of_match);
+
+static struct platform_driver sifive_fu540_prci_driver = {
+ .driver = {
+ .name = "sifive-fu540-prci",
+ .of_match_table = sifive_fu540_prci_of_match,
+ },
+ .probe = sifive_fu540_prci_probe,
+};
+
+static int __init sifive_fu540_prci_init(void)
+{
+ return platform_driver_register(&sifive_fu540_prci_driver);
+}
+core_initcall(sifive_fu540_prci_init);
-// SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-2.0 */
//
// Spreadtrum clock infrastructure
//
-// SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-2.0 */
//
// Spreadtrum composite clock driver
//
-// SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-2.0 */
//
// Spreadtrum divider clock driver
//
-// SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-2.0 */
//
// Spreadtrum gate clock driver
//
-// SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-2.0 */
//
// Spreadtrum multiplexer clock driver
//
-// SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-2.0 */
//
// Spreadtrum pll clock driver
//
static const char * const de_parents[] = { "pll-periph0-2x", "pll-de" };
static SUNXI_CCU_M_WITH_MUX_GATE(de_clk, "de", de_parents,
- 0x104, 0, 4, 24, 3, BIT(31), 0);
+ 0x104, 0, 4, 24, 3, BIT(31),
+ CLK_SET_RATE_PARENT);
static const char * const tcon0_parents[] = { "pll-mipi", "pll-video0-2x" };
static const u8 tcon0_table[] = { 0, 2, };
0, 4, /* M */
24, 1, /* mux */
BIT(31), /* gate */
- 0);
+ CLK_SET_RATE_PARENT);
static SUNXI_CCU_GATE(bus_de_clk, "bus-de", "psi-ahb1-ahb2",
0x60c, BIT(0), 0);
0, 3, /* M */
24, 1, /* mux */
BIT(31), /* gate */
- 0);
+ CLK_SET_RATE_PARENT);
static SUNXI_CCU_GATE(bus_ve_clk, "bus-ve", "psi-ahb1-ahb2",
0x69c, BIT(0), 0);
static const struct ccu_mux_fixed_prediv hdmi_cec_predivs[] = {
{ .index = 1, .div = 36621 },
};
+
+#define SUN50I_H6_HDMI_CEC_CLK_REG 0xb10
static struct ccu_mux hdmi_cec_clk = {
.enable = BIT(31),
tcon_lcd0_parents, 0xb60,
24, 3, /* mux */
BIT(31), /* gate */
- 0);
+ CLK_SET_RATE_PARENT);
static SUNXI_CCU_GATE(bus_tcon_lcd0_clk, "bus-tcon-lcd0", "ahb3",
0xb7c, BIT(0), 0);
8, 2, /* P */
24, 3, /* mux */
BIT(31), /* gate */
- 0);
+ CLK_SET_RATE_PARENT);
static SUNXI_CCU_GATE(bus_tcon_tv0_clk, "bus-tcon-tv0", "ahb3",
0xb9c, BIT(0), 0);
val &= ~(GENMASK(21, 16) | BIT(0));
writel(val | (7 << 16), reg + SUN50I_H6_PLL_AUDIO_REG);
+ /*
+ * First clock parent (osc32K) is unusable for CEC. But since there
+ * is no good way to force parent switch (both run with same frequency),
+ * just set second clock parent here.
+ */
+ val = readl(reg + SUN50I_H6_HDMI_CEC_CLK_REG);
+ val |= BIT(24);
+ writel(val, reg + SUN50I_H6_HDMI_CEC_CLK_REG);
+
return sunxi_ccu_probe(pdev->dev.of_node, reg, &sun50i_h6_ccu_desc);
}
-// SPDX-License-Identifier: GPL-2.0
+/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright 2016 Icenowy Zheng <icenowy@aosc.io>
*/
/* The rest of the module clocks are exported */
-#define CLK_MBUS 99
-
-/* And finally the IEP clock */
-
#define CLK_NUMBER (CLK_IEP + 1)
#endif /* _CCU_SUN5I_H_ */
static SUNXI_CCU_GATE(mipi_csi_clk, "mipi-csi", "osc24M", 0x130, BIT(31), 0);
-static const char * const csi_mclk_parents[] = { "pll-de", "osc24M" };
-static const u8 csi_mclk_table[] = { 3, 5 };
+static const char * const csi_mclk_parents[] = { "pll-video0", "pll-de",
+ "osc24M" };
+static const u8 csi_mclk_table[] = { 0, 3, 5 };
static SUNXI_CCU_M_WITH_MUX_TABLE_GATE(csi_mclk_clk, "csi-mclk",
csi_mclk_parents, csi_mclk_table,
0x134,
static const char * const de_parents[] = { "pll-video", "pll-periph0" };
static SUNXI_CCU_M_WITH_MUX_GATE(de_clk, "de", de_parents,
- 0x104, 0, 4, 24, 2, BIT(31), 0);
+ 0x104, 0, 4, 24, 2, BIT(31),
+ CLK_SET_RATE_PARENT);
static const char * const tcon_parents[] = { "pll-video" };
static SUNXI_CCU_M_WITH_MUX_GATE(tcon_clk, "tcon", tcon_parents,
/* The BSP header file has a CIR_CFG, but no mod clock uses this definition */
static SUNXI_CCU_GATE(usb_phy0_clk, "usb-phy0", "osc24M",
- 0x0cc, BIT(8), 0);
+ 0x0cc, BIT(1), 0);
static SUNXI_CCU_GATE(dram_ve_clk, "dram-ve", "pll-ddr",
0x100, BIT(0), 0);
-/* SPDX-License-Identifier: GPL-2.0+
- *
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
* Copyright 2017 Icenowy Zheng <icenowy@aosc.io>
*
*/
--- /dev/null
+menuconfig CLK_SUNXI
+ bool "Legacy clock support for Allwinner SoCs"
+ depends on ARCH_SUNXI || COMPILE_TEST
+ default y
+
+if CLK_SUNXI
+
+config CLK_SUNXI_CLOCKS
+ bool "Legacy clock drivers"
+ default y
+ help
+ Legacy clock drivers being used on older (A10, A13, A20,
+ A23, A31, A80) SoCs. These drivers are kept around for
+ Device Tree backward compatibility issues, in case one would
+ still use a Device Tree with one clock provider by
+ node. Newer Device Trees and newer SoCs use the drivers
+ controlled by CONFIG_SUNXI_CCU.
+
+config CLK_SUNXI_PRCM_SUN6I
+ bool "Legacy A31 PRCM driver"
+ select MFD_SUN6I_PRCM
+ default y
+ help
+ Legacy clock driver for the A31 PRCM clocks. Those are
+ usually needed for the PMIC communication, mostly.
+
+config CLK_SUNXI_PRCM_SUN8I
+ bool "Legacy sun8i PRCM driver"
+ select MFD_SUN6I_PRCM
+ default y
+ help
+ Legacy clock driver for the sun8i family PRCM clocks.
+ Those are usually needed for the PMIC communication,
+ mostly.
+
+config CLK_SUNXI_PRCM_SUN9I
+ bool "Legacy A80 PRCM driver"
+ default y
+ help
+ Legacy clock driver for the A80 PRCM clocks. Those are
+ usually needed for the PMIC communication, mostly.
+
+endif
# Makefile for sunxi specific clk
#
-obj-y += clk-sunxi.o clk-factors.o
-obj-y += clk-a10-codec.o
-obj-y += clk-a10-hosc.o
-obj-y += clk-a10-mod1.o
-obj-y += clk-a10-pll2.o
-obj-y += clk-a10-ve.o
-obj-y += clk-a20-gmac.o
-obj-y += clk-mod0.o
-obj-y += clk-simple-gates.o
-obj-y += clk-sun4i-display.o
-obj-y += clk-sun4i-pll3.o
-obj-y += clk-sun4i-tcon-ch1.o
-obj-y += clk-sun8i-bus-gates.o
-obj-y += clk-sun8i-mbus.o
-obj-y += clk-sun9i-core.o
-obj-y += clk-sun9i-mmc.o
-obj-y += clk-usb.o
+obj-$(CONFIG_CLK_SUNXI) += clk-factors.o
-obj-$(CONFIG_MACH_SUN9I) += clk-sun8i-apb0.o
-obj-$(CONFIG_MACH_SUN9I) += clk-sun9i-cpus.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-sunxi.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-a10-codec.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-a10-hosc.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-a10-mod1.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-a10-pll2.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-a10-ve.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-a20-gmac.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-mod0.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-simple-gates.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-sun4i-display.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-sun4i-pll3.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-sun4i-tcon-ch1.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-sun8i-bus-gates.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-sun8i-mbus.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-sun9i-core.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-sun9i-mmc.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-usb.o
-obj-$(CONFIG_MFD_SUN6I_PRCM) += \
- clk-sun6i-ar100.o clk-sun6i-apb0.o clk-sun6i-apb0-gates.o \
- clk-sun8i-apb0.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-sun8i-apb0.o
+obj-$(CONFIG_CLK_SUNXI_CLOCKS) += clk-sun9i-cpus.o
+
+obj-$(CONFIG_CLK_SUNXI_PRCM_SUN6I) += clk-sun6i-apb0.o
+obj-$(CONFIG_CLK_SUNXI_PRCM_SUN6I) += clk-sun6i-apb0-gates.o
+obj-$(CONFIG_CLK_SUNXI_PRCM_SUN6I) += clk-sun6i-ar100.o
+
+obj-$(CONFIG_CLK_SUNXI_PRCM_SUN8I) += clk-sun8i-apb0.o
+obj-$(CONFIG_CLK_SUNXI_PRCM_SUN8I) += clk-sun6i-apb0-gates.o
void __iomem *reg, spinlock_t *lock)
{
return clk_register_divider_table(NULL, name, parent_name,
- CLK_IS_CRITICAL, reg, 16, 1, 0,
+ CLK_IS_CRITICAL,
+ reg, 16, 1, CLK_DIVIDER_READ_ONLY,
mc_div_table, lock);
}
struct tegra_clk_emc *tegra;
u8 ram_code = tegra_read_ram_code();
struct emc_timing *timing = NULL;
- int i;
+ int i, k, t;
tegra = container_of(hw, struct tegra_clk_emc, hw);
- for (i = 0; i < tegra->num_timings; i++) {
- if (tegra->timings[i].ram_code != ram_code)
- continue;
+ for (k = 0; k < tegra->num_timings; k++) {
+ if (tegra->timings[k].ram_code == ram_code)
+ break;
+ }
+
+ for (t = k; t < tegra->num_timings; t++) {
+ if (tegra->timings[t].ram_code != ram_code)
+ break;
+ }
+ for (i = k; i < t; i++) {
timing = tegra->timings + i;
+ if (timing->rate < req->rate && i != t - 1)
+ continue;
+
if (timing->rate > req->max_rate) {
- i = max(i, 1);
+ i = max(i, k + 1);
req->rate = tegra->timings[i - 1].rate;
return 0;
}
if (timing->rate < req->min_rate)
continue;
- if (timing->rate >= req->rate) {
- req->rate = timing->rate;
- return 0;
- }
+ req->rate = timing->rate;
+ return 0;
}
if (timing) {
if (emc_get_parent(&tegra->hw) == timing->parent_index &&
clk_get_rate(timing->parent) != timing->parent_rate) {
- BUG();
+ WARN_ONCE(1, "parent %s rate mismatch %lu %lu\n",
+ __clk_get_name(timing->parent),
+ clk_get_rate(timing->parent),
+ timing->parent_rate);
return -EINVAL;
}
for (i = timing_index+1; i < tegra->num_timings; i++) {
timing = tegra->timings + i;
if (timing->ram_code != ram_code)
- continue;
+ break;
if (emc_parent_clk_sources[timing->parent_index] !=
emc_parent_clk_sources[
for (i = timing_index-1; i >= 0; --i) {
timing = tegra->timings + i;
if (timing->ram_code != ram_code)
- continue;
+ break;
if (emc_parent_clk_sources[timing->parent_index] !=
emc_parent_clk_sources[
struct device_node *node,
u32 ram_code)
{
+ struct emc_timing *timings_ptr;
struct device_node *child;
int child_count = of_get_child_count(node);
int i = 0, err;
+ size_t size;
- tegra->timings = kcalloc(child_count, sizeof(struct emc_timing),
- GFP_KERNEL);
+ size = (tegra->num_timings + child_count) * sizeof(struct emc_timing);
+
+ tegra->timings = krealloc(tegra->timings, size, GFP_KERNEL);
if (!tegra->timings)
return -ENOMEM;
- tegra->num_timings = child_count;
+ timings_ptr = tegra->timings + tegra->num_timings;
+ tegra->num_timings += child_count;
for_each_child_of_node(node, child) {
- struct emc_timing *timing = tegra->timings + (i++);
+ struct emc_timing *timing = timings_ptr + (i++);
err = load_one_timing_from_dt(tegra, timing, child);
if (err) {
timing->ram_code = ram_code;
}
- sort(tegra->timings, tegra->num_timings, sizeof(struct emc_timing),
+ sort(timings_ptr, child_count, sizeof(struct emc_timing),
cmp_timings, NULL);
return 0;
* fuses until the apbmisc driver is loaded.
*/
err = load_timings_from_dt(tegra, node, node_ram_code);
- of_node_put(node);
- if (err)
+ if (err) {
+ of_node_put(node);
return ERR_PTR(err);
- break;
+ }
}
if (tegra->num_timings == 0)
/* Allow debugging tools to see the EMC clock */
clk_register_clkdev(clk, "emc", "tegra-clk-debug");
- clk_prepare_enable(clk);
-
return clk;
};
unsigned long flags = 0;
int ret;
+ if (clk_pll_is_enabled(hw))
+ return 0;
+
if (pll->lock)
spin_lock_irqsave(pll->lock, flags);
pll_override_writel(val, params->pmc_divp_reg, pll);
val = pll_override_readl(params->pmc_divnm_reg, pll);
- val &= ~(divm_mask(pll) << div_nmp->override_divm_shift) |
- ~(divn_mask(pll) << div_nmp->override_divn_shift);
+ val &= ~((divm_mask(pll) << div_nmp->override_divm_shift) |
+ (divn_mask(pll) << div_nmp->override_divn_shift));
val |= (cfg->m << div_nmp->override_divm_shift) |
(cfg->n << div_nmp->override_divn_shift);
pll_override_writel(val, params->pmc_divnm_reg, pll);
static int clk_plle_enable(struct clk_hw *hw)
{
struct tegra_clk_pll *pll = to_clk_pll(hw);
- unsigned long input_rate = clk_hw_get_rate(clk_hw_get_parent(hw));
struct tegra_clk_pll_freq_table sel;
+ unsigned long input_rate;
u32 val;
int err;
+ if (clk_pll_is_enabled(hw))
+ return 0;
+
+ input_rate = clk_hw_get_rate(clk_hw_get_parent(hw));
+
if (_get_table_rate(hw, &sel, pll->params->fixed_rate, input_rate))
return -EINVAL;
int ret;
unsigned long flags = 0;
+ if (clk_pll_is_enabled(hw))
+ return 0;
+
if (pll->lock)
spin_lock_irqsave(pll->lock, flags);
u32 val;
int ret;
unsigned long flags = 0;
- unsigned long input_rate = clk_hw_get_rate(clk_hw_get_parent(hw));
+ unsigned long input_rate;
+
+ if (clk_pll_is_enabled(hw))
+ return 0;
+
+ input_rate = clk_hw_get_rate(clk_hw_get_parent(hw));
if (_get_table_rate(hw, &sel, pll->params->fixed_rate, input_rate))
return -EINVAL;
return -EINVAL;
}
+ if (clk_pll_is_enabled(hw))
+ return 0;
+
input_rate = clk_hw_get_rate(__clk_get_hw(osc));
if (pll->lock)
return clk;
}
+static int clk_plle_tegra210_is_enabled(struct clk_hw *hw)
+{
+ struct tegra_clk_pll *pll = to_clk_pll(hw);
+ u32 val;
+
+ val = pll_readl_base(pll);
+
+ return val & PLLE_BASE_ENABLE ? 1 : 0;
+}
+
static int clk_plle_tegra210_enable(struct clk_hw *hw)
{
struct tegra_clk_pll *pll = to_clk_pll(hw);
u32 val;
int ret = 0;
unsigned long flags = 0;
- unsigned long input_rate = clk_hw_get_rate(clk_hw_get_parent(hw));
+ unsigned long input_rate;
+
+ if (clk_plle_tegra210_is_enabled(hw))
+ return 0;
+
+ input_rate = clk_hw_get_rate(clk_hw_get_parent(hw));
if (_get_table_rate(hw, &sel, pll->params->fixed_rate, input_rate))
return -EINVAL;
spin_unlock_irqrestore(pll->lock, flags);
}
-static int clk_plle_tegra210_is_enabled(struct clk_hw *hw)
-{
- struct tegra_clk_pll *pll = to_clk_pll(hw);
- u32 val;
-
- val = pll_readl_base(pll);
-
- return val & PLLE_BASE_ENABLE ? 1 : 0;
-}
-
static const struct clk_ops tegra_clk_plle_tegra210_ops = {
.is_enabled = clk_plle_tegra210_is_enabled,
.enable = clk_plle_tegra210_enable,
return err;
}
-const struct clk_ops tegra_clk_super_mux_ops = {
+static const struct clk_ops tegra_clk_super_mux_ops = {
.get_parent = clk_super_get_parent,
.set_parent = clk_super_set_parent,
};
.base_reg = PLLM_BASE,
.misc_reg = PLLM_MISC,
.lock_mask = PLL_BASE_LOCK,
- .lock_enable_bit_idx = PLL_MISC_LOCK_ENABLE,
.lock_delay = 300,
.max_p = 5,
.pdiv_tohw = pllm_p,
.pmc_divnm_reg = PMC_PLLM_WB0_OVERRIDE,
.pmc_divp_reg = PMC_PLLM_WB0_OVERRIDE_2,
.freq_table = pll_m_freq_table,
- .flags = TEGRA_PLL_USE_LOCK | TEGRA_PLL_HAS_LOCK_ENABLE,
+ .flags = TEGRA_PLL_USE_LOCK,
};
static struct tegra_clk_pll_freq_table pll_e_freq_table[] = {
tegra_pmc_clk_init(pmc_base, tegra124_clks);
/* For Tegra124 & Tegra132, PLLD is the only source for DSIA & DSIB */
- plld_base = clk_readl(clk_base + PLLD_BASE);
+ plld_base = readl(clk_base + PLLD_BASE);
plld_base &= ~BIT(25);
- clk_writel(plld_base, clk_base + PLLD_BASE);
+ writel(plld_base, clk_base + PLLD_BASE);
}
/**
if (!clks)
return;
- value = clk_readl(clk_base + SPARE_REG0) >> CLK_M_DIVISOR_SHIFT;
+ value = readl(clk_base + SPARE_REG0) >> CLK_M_DIVISOR_SHIFT;
clk_m_div = (value & CLK_M_DIVISOR_MASK) + 1;
if (tegra_osc_clk_init(clk_base, tegra210_clks, tegra210_input_freq,
tegra_pmc_clk_init(pmc_base, tegra210_clks);
/* For Tegra210, PLLD is the only source for DSIA & DSIB */
- value = clk_readl(clk_base + PLLD_BASE);
+ value = readl(clk_base + PLLD_BASE);
value &= ~BIT(25);
- clk_writel(value, clk_base + PLLD_BASE);
+ writel(value, clk_base + PLLD_BASE);
tegra_clk_apply_init_table = tegra210_clock_apply_init_table;
{ DRA7_MMC2_CLKCTRL, dra7_mmc2_bit_data, CLKF_SW_SUP, "l3init_cm:clk:0010:25" },
{ DRA7_USB_OTG_SS2_CLKCTRL, dra7_usb_otg_ss2_bit_data, CLKF_HW_SUP, "dpll_core_h13x2_ck" },
{ DRA7_USB_OTG_SS3_CLKCTRL, NULL, CLKF_HW_SUP, "dpll_core_h13x2_ck" },
- { DRA7_USB_OTG_SS4_CLKCTRL, NULL, CLKF_HW_SUP, "dpll_core_h13x2_ck" },
+ { DRA7_USB_OTG_SS4_CLKCTRL, NULL, CLKF_HW_SUP | CLKF_SOC_DRA74 | CLKF_SOC_DRA76, "dpll_core_h13x2_ck" },
{ DRA7_SATA_CLKCTRL, dra7_sata_bit_data, CLKF_SW_SUP, "func_48m_fclk" },
{ DRA7_PCIE1_CLKCTRL, dra7_pcie1_bit_data, CLKF_SW_SUP, "l4_root_clk_div", "pcie_clkdm" },
{ DRA7_PCIE2_CLKCTRL, dra7_pcie2_bit_data, CLKF_SW_SUP, "l4_root_clk_div", "pcie_clkdm" },
{ DRA7_AES1_CLKCTRL, NULL, CLKF_HW_SUP, "l3_iclk_div", "l4sec_clkdm" },
{ DRA7_AES2_CLKCTRL, NULL, CLKF_HW_SUP, "l3_iclk_div", "l4sec_clkdm" },
{ DRA7_DES_CLKCTRL, NULL, CLKF_HW_SUP, "l3_iclk_div", "l4sec_clkdm" },
- { DRA7_RNG_CLKCTRL, NULL, CLKF_HW_SUP, "l3_iclk_div", "l4sec_clkdm" },
+ { DRA7_RNG_CLKCTRL, NULL, CLKF_HW_SUP | CLKF_SOC_NONSEC, "l3_iclk_div", "l4sec_clkdm" },
{ DRA7_SHAM_CLKCTRL, NULL, CLKF_HW_SUP, "l3_iclk_div", "l4sec_clkdm" },
{ DRA7_UART7_CLKCTRL, dra7_uart7_bit_data, CLKF_SW_SUP, "l4per_cm:clk:01d0:24", "l4per2_clkdm" },
{ DRA7_UART8_CLKCTRL, dra7_uart8_bit_data, CLKF_SW_SUP, "l4per_cm:clk:01e0:24", "l4per2_clkdm" },
{ DRA7_WD_TIMER2_CLKCTRL, NULL, CLKF_SW_SUP, "sys_32k_ck" },
{ DRA7_GPIO1_CLKCTRL, dra7_gpio1_bit_data, CLKF_HW_SUP, "wkupaon_iclk_mux" },
{ DRA7_TIMER1_CLKCTRL, dra7_timer1_bit_data, CLKF_SW_SUP, "wkupaon_cm:clk:0020:24" },
- { DRA7_TIMER12_CLKCTRL, NULL, 0, "secure_32k_clk_src_ck" },
+ { DRA7_TIMER12_CLKCTRL, NULL, CLKF_SOC_NONSEC, "secure_32k_clk_src_ck" },
{ DRA7_COUNTER_32K_CLKCTRL, NULL, 0, "wkupaon_iclk_mux" },
{ DRA7_UART10_CLKCTRL, dra7_uart10_bit_data, CLKF_SW_SUP, "wkupaon_cm:clk:0060:24" },
{ DRA7_DCAN1_CLKCTRL, dra7_dcan1_bit_data, CLKF_SW_SUP, "wkupaon_cm:clk:0068:24" },
{ DRA7_L3INIT_MMC2_CLKCTRL, dra7_mmc2_bit_data, CLKF_SW_SUP, "l3init-clkctrl:0010:25" },
{ DRA7_L3INIT_USB_OTG_SS2_CLKCTRL, dra7_usb_otg_ss2_bit_data, CLKF_HW_SUP, "dpll_core_h13x2_ck" },
{ DRA7_L3INIT_USB_OTG_SS3_CLKCTRL, NULL, CLKF_HW_SUP, "dpll_core_h13x2_ck" },
- { DRA7_L3INIT_USB_OTG_SS4_CLKCTRL, NULL, CLKF_HW_SUP, "dpll_core_h13x2_ck" },
+ { DRA7_L3INIT_USB_OTG_SS4_CLKCTRL, NULL, CLKF_HW_SUP | CLKF_SOC_DRA74 | CLKF_SOC_DRA76, "dpll_core_h13x2_ck" },
{ DRA7_L3INIT_SATA_CLKCTRL, dra7_sata_bit_data, CLKF_SW_SUP, "func_48m_fclk" },
{ DRA7_L3INIT_OCP2SCP1_CLKCTRL, NULL, CLKF_HW_SUP, "l4_root_clk_div" },
{ DRA7_L3INIT_OCP2SCP3_CLKCTRL, NULL, CLKF_HW_SUP, "l4_root_clk_div" },
{ DRA7_L4SEC_AES1_CLKCTRL, NULL, CLKF_HW_SUP, "l3_iclk_div" },
{ DRA7_L4SEC_AES2_CLKCTRL, NULL, CLKF_HW_SUP, "l3_iclk_div" },
{ DRA7_L4SEC_DES_CLKCTRL, NULL, CLKF_HW_SUP, "l3_iclk_div" },
- { DRA7_L4SEC_RNG_CLKCTRL, NULL, CLKF_HW_SUP, "" },
+ { DRA7_L4SEC_RNG_CLKCTRL, NULL, CLKF_HW_SUP | CLKF_SOC_NONSEC, "" },
{ DRA7_L4SEC_SHAM_CLKCTRL, NULL, CLKF_HW_SUP, "l3_iclk_div" },
{ 0 },
};
{ DRA7_WKUPAON_WD_TIMER2_CLKCTRL, NULL, CLKF_SW_SUP, "sys_32k_ck" },
{ DRA7_WKUPAON_GPIO1_CLKCTRL, dra7_gpio1_bit_data, CLKF_HW_SUP, "wkupaon_iclk_mux" },
{ DRA7_WKUPAON_TIMER1_CLKCTRL, dra7_timer1_bit_data, CLKF_SW_SUP, "wkupaon-clkctrl:0020:24" },
- { DRA7_WKUPAON_TIMER12_CLKCTRL, NULL, 0, "secure_32k_clk_src_ck" },
+ { DRA7_WKUPAON_TIMER12_CLKCTRL, NULL, CLKF_SOC_NONSEC, "secure_32k_clk_src_ck" },
{ DRA7_WKUPAON_COUNTER_32K_CLKCTRL, NULL, 0, "wkupaon_iclk_mux" },
{ DRA7_WKUPAON_UART10_CLKCTRL, dra7_uart10_bit_data, CLKF_SW_SUP, "wkupaon-clkctrl:0060:24" },
{ DRA7_WKUPAON_DCAN1_CLKCTRL, dra7_dcan1_bit_data, CLKF_SW_SUP, "wkupaon-clkctrl:0068:24" },
u32 addr;
int ret;
char *c;
+ u16 soc_mask = 0;
if (!(ti_clk_get_features()->flags & TI_CLK_CLKCTRL_COMPAT) &&
of_node_name_eq(node, "clk"))
else
data = dra7_clkctrl_data;
}
+
+ if (of_machine_is_compatible("ti,dra72"))
+ soc_mask = CLKF_SOC_DRA72;
+ if (of_machine_is_compatible("ti,dra74"))
+ soc_mask = CLKF_SOC_DRA74;
+ if (of_machine_is_compatible("ti,dra76"))
+ soc_mask = CLKF_SOC_DRA76;
#endif
#ifdef CONFIG_SOC_AM33XX
if (of_machine_is_compatible("ti,am33xx")) {
data = dm816_clkctrl_data;
#endif
+ if (ti_clk_get_features()->flags & TI_CLK_DEVICE_TYPE_GP)
+ soc_mask |= CLKF_SOC_NONSEC;
+
while (data->addr) {
if (addr == data->addr)
break;
reg_data = data->regs;
while (reg_data->parent) {
+ if ((reg_data->flags & CLKF_SOC_MASK) &&
+ (reg_data->flags & soc_mask) == 0) {
+ reg_data++;
+ continue;
+ }
+
hw = kzalloc(sizeof(*hw), GFP_KERNEL);
if (!hw)
return;
#define CLKF_HW_SUP BIT(6)
#define CLKF_NO_IDLEST BIT(7)
+#define CLKF_SOC_MASK GENMASK(11, 8)
+
+#define CLKF_SOC_NONSEC BIT(8)
+#define CLKF_SOC_DRA72 BIT(9)
+#define CLKF_SOC_DRA74 BIT(10)
+#define CLKF_SOC_DRA76 BIT(11)
+
#define CLK(dev, con, ck) \
{ \
.lk = { \
int omap4_dpll_regm4xen_determine_rate(struct clk_hw *hw,
struct clk_rate_request *req);
int omap2_clk_for_each(int (*fn)(struct clk_hw_omap *hw));
-bool omap2_clk_is_hw_omap(struct clk_hw *hw);
extern struct ti_clk_ll_ops *ti_clk_ll_ops;
clk->reg_bits[0]);
if (!ret && clk->enable_delay_us)
- usleep_range(clk->enable_delay_us, clk->enable_delay_us);
+ usleep_range(clk->enable_delay_us, clk->enable_delay_us +
+ (clk->enable_delay_us >> 2));
return ret;
}
clks[fclk] = clk_register_gate(NULL, clk_name,
div1_name, CLK_SET_RATE_PARENT, fclk_gate_reg,
0, CLK_GATE_SET_TO_DISABLE, fclk_gate_lock);
- enable_reg = clk_readl(fclk_gate_reg) & 1;
+ enable_reg = readl(fclk_gate_reg) & 1;
if (enable && !enable_reg) {
if (clk_prepare_enable(clks[fclk]))
pr_warn("%s: FCLK%u enable failed\n", __func__,
SLCR_IOPLL_CTRL, 4, 1, 0, &iopll_lock);
/* CPU clocks */
- tmp = clk_readl(SLCR_621_TRUE) & 1;
+ tmp = readl(SLCR_621_TRUE) & 1;
clk = clk_register_mux(NULL, "cpu_mux", cpu_parents, 4,
CLK_SET_RATE_NO_REPARENT, SLCR_ARM_CLK_CTRL, 4, 2, 0,
&armclk_lock);
&dbgclk_lock);
/* leave debug clocks in the state the bootloader set them up to */
- tmp = clk_readl(SLCR_DBG_CLK_CTRL);
+ tmp = readl(SLCR_DBG_CLK_CTRL);
if (tmp & DBG_CLK_CTRL_CLKACT_TRC)
if (clk_prepare_enable(clks[dbg_trc]))
pr_warn("%s: trace clk enable failed\n", __func__);
* makes probably sense to redundantly save fbdiv in the struct
* zynq_pll to save the IO access.
*/
- fbdiv = (clk_readl(clk->pll_ctrl) & PLLCTRL_FBDIV_MASK) >>
+ fbdiv = (readl(clk->pll_ctrl) & PLLCTRL_FBDIV_MASK) >>
PLLCTRL_FBDIV_SHIFT;
return parent_rate * fbdiv;
spin_lock_irqsave(clk->lock, flags);
- reg = clk_readl(clk->pll_ctrl);
+ reg = readl(clk->pll_ctrl);
spin_unlock_irqrestore(clk->lock, flags);
/* Power up PLL and wait for lock */
spin_lock_irqsave(clk->lock, flags);
- reg = clk_readl(clk->pll_ctrl);
+ reg = readl(clk->pll_ctrl);
reg &= ~(PLLCTRL_RESET_MASK | PLLCTRL_PWRDWN_MASK);
- clk_writel(reg, clk->pll_ctrl);
- while (!(clk_readl(clk->pll_status) & (1 << clk->lockbit)))
+ writel(reg, clk->pll_ctrl);
+ while (!(readl(clk->pll_status) & (1 << clk->lockbit)))
;
spin_unlock_irqrestore(clk->lock, flags);
/* shut down PLL */
spin_lock_irqsave(clk->lock, flags);
- reg = clk_readl(clk->pll_ctrl);
+ reg = readl(clk->pll_ctrl);
reg |= PLLCTRL_RESET_MASK | PLLCTRL_PWRDWN_MASK;
- clk_writel(reg, clk->pll_ctrl);
+ writel(reg, clk->pll_ctrl);
spin_unlock_irqrestore(clk->lock, flags);
}
spin_lock_irqsave(pll->lock, flags);
- reg = clk_readl(pll->pll_ctrl);
+ reg = readl(pll->pll_ctrl);
reg &= ~PLLCTRL_BPQUAL_MASK;
- clk_writel(reg, pll->pll_ctrl);
+ writel(reg, pll->pll_ctrl);
spin_unlock_irqrestore(pll->lock, flags);
return hw;
}
-EXPORT_SYMBOL_GPL(zynqmp_clk_register_mux);
#include <linux/firmware/xlnx-zynqmp.h>
-/* Clock APIs payload parameters */
-#define CLK_GET_NAME_RESP_LEN 16
-#define CLK_GET_TOPOLOGY_RESP_WORDS 3
-#define CLK_GET_PARENTS_RESP_WORDS 3
-#define CLK_GET_ATTR_RESP_WORDS 1
-
enum topology_type {
TYPE_INVALID,
TYPE_MUX,
#define MAX_NODES 6
#define MAX_NAME_LEN 50
-#define CLK_TYPE_SHIFT 2
-
-#define PM_API_PAYLOAD_LEN 3
-
-#define NA_PARENT 0xFFFFFFFF
-#define DUMMY_PARENT 0xFFFFFFFE
-
-#define CLK_TYPE_FIELD_LEN 4
-#define CLK_TOPOLOGY_NODE_OFFSET 16
-#define NODES_PER_RESP 3
-
-#define CLK_TYPE_FIELD_MASK 0xF
-#define CLK_FLAG_FIELD_MASK GENMASK(21, 8)
-#define CLK_TYPE_FLAG_FIELD_MASK GENMASK(31, 24)
-
-#define CLK_PARENTS_ID_LEN 16
-#define CLK_PARENTS_ID_MASK 0xFFFF
-
/* Flags for parents */
#define PARENT_CLK_SELF 0
#define PARENT_CLK_NODE1 1
#define END_OF_PARENTS 1
#define RESERVED_CLK_NAME ""
-#define CLK_VALID_MASK 0x1
+#define CLK_GET_NAME_RESP_LEN 16
+#define CLK_GET_TOPOLOGY_RESP_WORDS 3
+#define CLK_GET_PARENTS_RESP_WORDS 3
+#define CLK_GET_ATTR_RESP_WORDS 1
enum clk_type {
CLK_TYPE_OUTPUT,
* @num_nodes: Number of nodes present in topology
* @parent: Parent of clock
* @num_parents: Number of parents of clock
+ * @clk_id: Clock id
*/
struct zynqmp_clock {
char clk_name[MAX_NAME_LEN];
u32 num_nodes;
struct clock_parent parent[MAX_PARENT];
u32 num_parents;
+ u32 clk_id;
+};
+
+struct name_resp {
+ char name[CLK_GET_NAME_RESP_LEN];
+};
+
+struct topology_resp {
+#define CLK_TOPOLOGY_TYPE GENMASK(3, 0)
+#define CLK_TOPOLOGY_FLAGS GENMASK(23, 8)
+#define CLK_TOPOLOGY_TYPE_FLAGS GENMASK(31, 24)
+ u32 topology[CLK_GET_TOPOLOGY_RESP_WORDS];
+};
+
+struct parents_resp {
+#define NA_PARENT 0xFFFFFFFF
+#define DUMMY_PARENT 0xFFFFFFFE
+#define CLK_PARENTS_ID GENMASK(15, 0)
+#define CLK_PARENTS_FLAGS GENMASK(31, 16)
+ u32 parents[CLK_GET_PARENTS_RESP_WORDS];
+};
+
+struct attr_resp {
+#define CLK_ATTR_VALID BIT(0)
+#define CLK_ATTR_TYPE BIT(2)
+#define CLK_ATTR_NODE_INDEX GENMASK(13, 0)
+#define CLK_ATTR_NODE_TYPE GENMASK(19, 14)
+#define CLK_ATTR_NODE_SUBCLASS GENMASK(25, 20)
+#define CLK_ATTR_NODE_CLASS GENMASK(31, 26)
+ u32 attr[CLK_GET_ATTR_RESP_WORDS];
};
static const char clk_type_postfix[][10] = {
/**
* zynqmp_pm_clock_get_name() - Get the name of clock for given id
* @clock_id: ID of the clock to be queried
- * @name: Name of given clock
+ * @response: Name of the clock with the given id
*
* This function is used to get name of clock specified by given
* clock ID.
*
- * Return: Returns 0, in case of error name would be 0
+ * Return: Returns 0
*/
-static int zynqmp_pm_clock_get_name(u32 clock_id, char *name)
+static int zynqmp_pm_clock_get_name(u32 clock_id,
+ struct name_resp *response)
{
struct zynqmp_pm_query_data qdata = {0};
u32 ret_payload[PAYLOAD_ARG_CNT];
qdata.arg1 = clock_id;
eemi_ops->query_data(qdata, ret_payload);
- memcpy(name, ret_payload, CLK_GET_NAME_RESP_LEN);
+ memcpy(response, ret_payload, sizeof(*response));
return 0;
}
* zynqmp_pm_clock_get_topology() - Get the topology of clock for given id
* @clock_id: ID of the clock to be queried
* @index: Node index of clock topology
- * @topology: Buffer to store nodes in topology and flags
+ * @response: Buffer used for the topology response
*
* This function is used to get topology information for the clock
* specified by given clock ID.
*
* Return: 0 on success else error+reason
*/
-static int zynqmp_pm_clock_get_topology(u32 clock_id, u32 index, u32 *topology)
+static int zynqmp_pm_clock_get_topology(u32 clock_id, u32 index,
+ struct topology_resp *response)
{
struct zynqmp_pm_query_data qdata = {0};
u32 ret_payload[PAYLOAD_ARG_CNT];
qdata.arg2 = index;
ret = eemi_ops->query_data(qdata, ret_payload);
- memcpy(topology, &ret_payload[1], CLK_GET_TOPOLOGY_RESP_WORDS * 4);
+ memcpy(response, &ret_payload[1], sizeof(*response));
return ret;
}
* zynqmp_pm_clock_get_parents() - Get the first 3 parents of clock for given id
* @clock_id: Clock ID
* @index: Parent index
- * @parents: 3 parents of the given clock
+ * @response: Parents of the given clock
*
* This function is used to get 3 parents for the clock specified by
* given clock ID.
*
* Return: 0 on success else error+reason
*/
-static int zynqmp_pm_clock_get_parents(u32 clock_id, u32 index, u32 *parents)
+static int zynqmp_pm_clock_get_parents(u32 clock_id, u32 index,
+ struct parents_resp *response)
{
struct zynqmp_pm_query_data qdata = {0};
u32 ret_payload[PAYLOAD_ARG_CNT];
qdata.arg2 = index;
ret = eemi_ops->query_data(qdata, ret_payload);
- memcpy(parents, &ret_payload[1], CLK_GET_PARENTS_RESP_WORDS * 4);
+ memcpy(response, &ret_payload[1], sizeof(*response));
return ret;
}
/**
* zynqmp_pm_clock_get_attributes() - Get the attributes of clock for given id
* @clock_id: Clock ID
- * @attr: Clock attributes
+ * @response: Clock attributes response
*
* This function is used to get clock's attributes(e.g. valid, clock type, etc).
*
* Return: 0 on success else error+reason
*/
-static int zynqmp_pm_clock_get_attributes(u32 clock_id, u32 *attr)
+static int zynqmp_pm_clock_get_attributes(u32 clock_id,
+ struct attr_resp *response)
{
struct zynqmp_pm_query_data qdata = {0};
u32 ret_payload[PAYLOAD_ARG_CNT];
qdata.arg1 = clock_id;
ret = eemi_ops->query_data(qdata, ret_payload);
- memcpy(attr, &ret_payload[1], CLK_GET_ATTR_RESP_WORDS * 4);
+ memcpy(response, &ret_payload[1], sizeof(*response));
return ret;
}
* __zynqmp_clock_get_topology() - Get topology data of clock from firmware
* response data
* @topology: Clock topology
- * @data: Clock topology data received from firmware
+ * @response: Clock topology data received from firmware
* @nnodes: Number of nodes
*
* Return: 0 on success else error+reason
*/
static int __zynqmp_clock_get_topology(struct clock_topology *topology,
- u32 *data, u32 *nnodes)
+ struct topology_resp *response,
+ u32 *nnodes)
{
int i;
+ u32 type;
- for (i = 0; i < PM_API_PAYLOAD_LEN; i++) {
- if (!(data[i] & CLK_TYPE_FIELD_MASK))
+ for (i = 0; i < ARRAY_SIZE(response->topology); i++) {
+ type = FIELD_GET(CLK_TOPOLOGY_TYPE, response->topology[i]);
+ if (type == TYPE_INVALID)
return END_OF_TOPOLOGY_NODE;
- topology[*nnodes].type = data[i] & CLK_TYPE_FIELD_MASK;
- topology[*nnodes].flag = FIELD_GET(CLK_FLAG_FIELD_MASK,
- data[i]);
+ topology[*nnodes].type = type;
+ topology[*nnodes].flag = FIELD_GET(CLK_TOPOLOGY_FLAGS,
+ response->topology[i]);
topology[*nnodes].type_flag =
- FIELD_GET(CLK_TYPE_FLAG_FIELD_MASK, data[i]);
+ FIELD_GET(CLK_TOPOLOGY_TYPE_FLAGS,
+ response->topology[i]);
(*nnodes)++;
}
u32 *num_nodes)
{
int j, ret;
- u32 pm_resp[PM_API_PAYLOAD_LEN] = {0};
+ struct topology_resp response = { };
*num_nodes = 0;
- for (j = 0; j <= MAX_NODES; j += 3) {
- ret = zynqmp_pm_clock_get_topology(clk_id, j, pm_resp);
+ for (j = 0; j <= MAX_NODES; j += ARRAY_SIZE(response.topology)) {
+ ret = zynqmp_pm_clock_get_topology(clock[clk_id].clk_id, j,
+ &response);
if (ret)
return ret;
- ret = __zynqmp_clock_get_topology(topology, pm_resp, num_nodes);
+ ret = __zynqmp_clock_get_topology(topology, &response,
+ num_nodes);
if (ret == END_OF_TOPOLOGY_NODE)
return 0;
}
}
/**
- * __zynqmp_clock_get_topology() - Get parents info of clock from firmware
+ * __zynqmp_clock_get_parents() - Get parents info of clock from firmware
* response data
* @parents: Clock parents
- * @data: Clock parents data received from firmware
+ * @response: Clock parents data received from firmware
* @nparent: Number of parent
*
* Return: 0 on success else error+reason
*/
-static int __zynqmp_clock_get_parents(struct clock_parent *parents, u32 *data,
+static int __zynqmp_clock_get_parents(struct clock_parent *parents,
+ struct parents_resp *response,
u32 *nparent)
{
int i;
struct clock_parent *parent;
- for (i = 0; i < PM_API_PAYLOAD_LEN; i++) {
- if (data[i] == NA_PARENT)
+ for (i = 0; i < ARRAY_SIZE(response->parents); i++) {
+ if (response->parents[i] == NA_PARENT)
return END_OF_PARENTS;
parent = &parents[i];
- parent->id = data[i] & CLK_PARENTS_ID_MASK;
- if (data[i] == DUMMY_PARENT) {
+ parent->id = FIELD_GET(CLK_PARENTS_ID, response->parents[i]);
+ if (response->parents[i] == DUMMY_PARENT) {
strcpy(parent->name, "dummy_name");
parent->flag = 0;
} else {
- parent->flag = data[i] >> CLK_PARENTS_ID_LEN;
+ parent->flag = FIELD_GET(CLK_PARENTS_FLAGS,
+ response->parents[i]);
if (zynqmp_get_clock_name(parent->id, parent->name))
continue;
}
u32 *num_parents)
{
int j = 0, ret;
- u32 pm_resp[PM_API_PAYLOAD_LEN] = {0};
+ struct parents_resp response = { };
*num_parents = 0;
do {
/* Get parents from firmware */
- ret = zynqmp_pm_clock_get_parents(clk_id, j, pm_resp);
+ ret = zynqmp_pm_clock_get_parents(clock[clk_id].clk_id, j,
+ &response);
if (ret)
return ret;
- ret = __zynqmp_clock_get_parents(&parents[j], pm_resp,
+ ret = __zynqmp_clock_get_parents(&parents[j], &response,
num_parents);
if (ret == END_OF_PARENTS)
return 0;
- j += PM_API_PAYLOAD_LEN;
+ j += ARRAY_SIZE(response.parents);
} while (*num_parents <= MAX_PARENT);
return 0;
const char **parent_names)
{
int j;
- u32 num_nodes;
+ u32 num_nodes, clk_dev_id;
char *clk_out = NULL;
struct clock_topology *nodes;
struct clk_hw *hw = NULL;
nodes = clock[clk_id].node;
num_nodes = clock[clk_id].num_nodes;
+ clk_dev_id = clock[clk_id].clk_id;
for (j = 0; j < num_nodes; j++) {
/*
if (!clk_topology[nodes[j].type])
continue;
- hw = (*clk_topology[nodes[j].type])(clk_out, clk_id,
+ hw = (*clk_topology[nodes[j].type])(clk_out, clk_dev_id,
parent_names,
num_parents,
&nodes[j]);
if (IS_ERR(hw))
- pr_warn_once("%s() %s register fail with %ld\n",
- __func__, clk_name, PTR_ERR(hw));
+ pr_warn_once("%s() 0x%x: %s register fail with %ld\n",
+ __func__, clk_dev_id, clk_name,
+ PTR_ERR(hw));
parent_names[0] = clk_out;
}
static void zynqmp_get_clock_info(void)
{
int i, ret;
- u32 attr, type = 0;
+ u32 type = 0;
+ u32 nodetype, subclass, class;
+ struct attr_resp attr;
+ struct name_resp name;
for (i = 0; i < clock_max_idx; i++) {
- zynqmp_pm_clock_get_name(i, clock[i].clk_name);
- if (!strcmp(clock[i].clk_name, RESERVED_CLK_NAME))
- continue;
-
ret = zynqmp_pm_clock_get_attributes(i, &attr);
if (ret)
continue;
- clock[i].valid = attr & CLK_VALID_MASK;
- clock[i].type = attr >> CLK_TYPE_SHIFT ? CLK_TYPE_EXTERNAL :
- CLK_TYPE_OUTPUT;
+ clock[i].valid = FIELD_GET(CLK_ATTR_VALID, attr.attr[0]);
+ clock[i].type = FIELD_GET(CLK_ATTR_TYPE, attr.attr[0]) ?
+ CLK_TYPE_EXTERNAL : CLK_TYPE_OUTPUT;
+
+ nodetype = FIELD_GET(CLK_ATTR_NODE_TYPE, attr.attr[0]);
+ subclass = FIELD_GET(CLK_ATTR_NODE_SUBCLASS, attr.attr[0]);
+ class = FIELD_GET(CLK_ATTR_NODE_CLASS, attr.attr[0]);
+
+ clock[i].clk_id = FIELD_PREP(CLK_ATTR_NODE_CLASS, class) |
+ FIELD_PREP(CLK_ATTR_NODE_SUBCLASS, subclass) |
+ FIELD_PREP(CLK_ATTR_NODE_TYPE, nodetype) |
+ FIELD_PREP(CLK_ATTR_NODE_INDEX, i);
+
+ zynqmp_pm_clock_get_name(clock[i].clk_id, &name);
+ if (!strcmp(name.name, RESERVED_CLK_NAME))
+ continue;
+ strncpy(clock[i].clk_name, name.name, MAX_NAME_LEN);
}
/* Get topology of all clock */
* struct zynqmp_clk_divider - adjustable divider clock
* @hw: handle between common and hardware-specific interfaces
* @flags: Hardware specific flags
+ * @is_frac: The divider is a fractional divider
* @clk_id: Id of clock
* @div_type: divisor type (TYPE_DIV1 or TYPE_DIV2)
*/
struct zynqmp_clk_divider {
struct clk_hw hw;
u8 flags;
+ bool is_frac;
u32 clk_id;
u32 div_type;
};
else
value = div >> 16;
+ if (!value) {
+ WARN(!(divider->flags & CLK_DIVIDER_ALLOW_ZERO),
+ "%s: Zero divisor and CLK_DIVIDER_ALLOW_ZERO not set\n",
+ clk_name);
+ return parent_rate;
+ }
+
return DIV_ROUND_UP_ULL(parent_rate, value);
}
bestdiv = zynqmp_divider_get_val(*prate, rate);
- if ((clk_hw_get_flags(hw) & CLK_SET_RATE_PARENT) &&
- (divider->flags & CLK_FRAC))
+ if ((clk_hw_get_flags(hw) & CLK_SET_RATE_PARENT) && divider->is_frac)
bestdiv = rate % *prate ? 1 : bestdiv;
*prate = rate * bestdiv;
init.name = name;
init.ops = &zynqmp_clk_divider_ops;
- init.flags = nodes->flag;
+ /* CLK_FRAC is not defined in the common clk framework */
+ init.flags = nodes->flag & ~CLK_FRAC;
init.parent_names = parents;
init.num_parents = 1;
/* struct clk_divider assignments */
+ div->is_frac = !!(nodes->flag & CLK_FRAC);
div->flags = nodes->type_flag;
div->hw.init = &init;
div->clk_id = clk_id;
return hw;
}
-EXPORT_SYMBOL_GPL(zynqmp_clk_register_divider);
-obj-y := dma-buf.o dma-fence.o dma-fence-array.o reservation.o seqno-fence.o
+obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \
+ reservation.o seqno-fence.o
obj-$(CONFIG_SYNC_FILE) += sync_file.o
obj-$(CONFIG_SW_SYNC) += sw_sync.o sync_debug.o
obj-$(CONFIG_UDMABUF) += udmabuf.o
--- /dev/null
+/*
+ * fence-chain: chain fences together in a timeline
+ *
+ * Copyright (C) 2018 Advanced Micro Devices, Inc.
+ * Authors:
+ * Christian König <christian.koenig@amd.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/dma-fence-chain.h>
+
+static bool dma_fence_chain_enable_signaling(struct dma_fence *fence);
+
+/**
+ * dma_fence_chain_get_prev - use RCU to get a reference to the previous fence
+ * @chain: chain node to get the previous node from
+ *
+ * Use dma_fence_get_rcu_safe to get a reference to the previous fence of the
+ * chain node.
+ */
+static struct dma_fence *dma_fence_chain_get_prev(struct dma_fence_chain *chain)
+{
+ struct dma_fence *prev;
+
+ rcu_read_lock();
+ prev = dma_fence_get_rcu_safe(&chain->prev);
+ rcu_read_unlock();
+ return prev;
+}
+
+/**
+ * dma_fence_chain_walk - chain walking function
+ * @fence: current chain node
+ *
+ * Walk the chain to the next node. Returns the next fence or NULL if we are at
+ * the end of the chain. Garbage collects chain nodes which are already
+ * signaled.
+ */
+struct dma_fence *dma_fence_chain_walk(struct dma_fence *fence)
+{
+ struct dma_fence_chain *chain, *prev_chain;
+ struct dma_fence *prev, *replacement, *tmp;
+
+ chain = to_dma_fence_chain(fence);
+ if (!chain) {
+ dma_fence_put(fence);
+ return NULL;
+ }
+
+ while ((prev = dma_fence_chain_get_prev(chain))) {
+
+ prev_chain = to_dma_fence_chain(prev);
+ if (prev_chain) {
+ if (!dma_fence_is_signaled(prev_chain->fence))
+ break;
+
+ replacement = dma_fence_chain_get_prev(prev_chain);
+ } else {
+ if (!dma_fence_is_signaled(prev))
+ break;
+
+ replacement = NULL;
+ }
+
+ tmp = cmpxchg((void **)&chain->prev, (void *)prev, (void *)replacement);
+ if (tmp == prev)
+ dma_fence_put(tmp);
+ else
+ dma_fence_put(replacement);
+ dma_fence_put(prev);
+ }
+
+ dma_fence_put(fence);
+ return prev;
+}
+EXPORT_SYMBOL(dma_fence_chain_walk);
+
+/**
+ * dma_fence_chain_find_seqno - find fence chain node by seqno
+ * @pfence: pointer to the chain node where to start
+ * @seqno: the sequence number to search for
+ *
+ * Advance the fence pointer to the chain node which will signal this sequence
+ * number. If no sequence number is provided then this is a no-op.
+ *
+ * Returns EINVAL if the fence is not a chain node or the sequence number has
+ * not yet advanced far enough.
+ */
+int dma_fence_chain_find_seqno(struct dma_fence **pfence, uint64_t seqno)
+{
+ struct dma_fence_chain *chain;
+
+ if (!seqno)
+ return 0;
+
+ chain = to_dma_fence_chain(*pfence);
+ if (!chain || chain->base.seqno < seqno)
+ return -EINVAL;
+
+ dma_fence_chain_for_each(*pfence, &chain->base) {
+ if ((*pfence)->context != chain->base.context ||
+ to_dma_fence_chain(*pfence)->prev_seqno < seqno)
+ break;
+ }
+ dma_fence_put(&chain->base);
+
+ return 0;
+}
+EXPORT_SYMBOL(dma_fence_chain_find_seqno);
+
+static const char *dma_fence_chain_get_driver_name(struct dma_fence *fence)
+{
+ return "dma_fence_chain";
+}
+
+static const char *dma_fence_chain_get_timeline_name(struct dma_fence *fence)
+{
+ return "unbound";
+}
+
+static void dma_fence_chain_irq_work(struct irq_work *work)
+{
+ struct dma_fence_chain *chain;
+
+ chain = container_of(work, typeof(*chain), work);
+
+ /* Try to rearm the callback */
+ if (!dma_fence_chain_enable_signaling(&chain->base))
+ /* Ok, we are done. No more unsignaled fences left */
+ dma_fence_signal(&chain->base);
+ dma_fence_put(&chain->base);
+}
+
+static void dma_fence_chain_cb(struct dma_fence *f, struct dma_fence_cb *cb)
+{
+ struct dma_fence_chain *chain;
+
+ chain = container_of(cb, typeof(*chain), cb);
+ irq_work_queue(&chain->work);
+ dma_fence_put(f);
+}
+
+static bool dma_fence_chain_enable_signaling(struct dma_fence *fence)
+{
+ struct dma_fence_chain *head = to_dma_fence_chain(fence);
+
+ dma_fence_get(&head->base);
+ dma_fence_chain_for_each(fence, &head->base) {
+ struct dma_fence_chain *chain = to_dma_fence_chain(fence);
+ struct dma_fence *f = chain ? chain->fence : fence;
+
+ dma_fence_get(f);
+ if (!dma_fence_add_callback(f, &head->cb, dma_fence_chain_cb)) {
+ dma_fence_put(fence);
+ return true;
+ }
+ dma_fence_put(f);
+ }
+ dma_fence_put(&head->base);
+ return false;
+}
+
+static bool dma_fence_chain_signaled(struct dma_fence *fence)
+{
+ dma_fence_chain_for_each(fence, fence) {
+ struct dma_fence_chain *chain = to_dma_fence_chain(fence);
+ struct dma_fence *f = chain ? chain->fence : fence;
+
+ if (!dma_fence_is_signaled(f)) {
+ dma_fence_put(fence);
+ return false;
+ }
+ }
+
+ return true;
+}
+
+static void dma_fence_chain_release(struct dma_fence *fence)
+{
+ struct dma_fence_chain *chain = to_dma_fence_chain(fence);
+
+ dma_fence_put(rcu_dereference_protected(chain->prev, true));
+ dma_fence_put(chain->fence);
+ dma_fence_free(fence);
+}
+
+const struct dma_fence_ops dma_fence_chain_ops = {
+ .use_64bit_seqno = true,
+ .get_driver_name = dma_fence_chain_get_driver_name,
+ .get_timeline_name = dma_fence_chain_get_timeline_name,
+ .enable_signaling = dma_fence_chain_enable_signaling,
+ .signaled = dma_fence_chain_signaled,
+ .release = dma_fence_chain_release,
+};
+EXPORT_SYMBOL(dma_fence_chain_ops);
+
+/**
+ * dma_fence_chain_init - initialize a fence chain
+ * @chain: the chain node to initialize
+ * @prev: the previous fence
+ * @fence: the current fence
+ *
+ * Initialize a new chain node and either start a new chain or add the node to
+ * the existing chain of the previous fence.
+ */
+void dma_fence_chain_init(struct dma_fence_chain *chain,
+ struct dma_fence *prev,
+ struct dma_fence *fence,
+ uint64_t seqno)
+{
+ struct dma_fence_chain *prev_chain = to_dma_fence_chain(prev);
+ uint64_t context;
+
+ spin_lock_init(&chain->lock);
+ rcu_assign_pointer(chain->prev, prev);
+ chain->fence = fence;
+ chain->prev_seqno = 0;
+ init_irq_work(&chain->work, dma_fence_chain_irq_work);
+
+ /* Try to reuse the context of the previous chain node. */
+ if (prev_chain && __dma_fence_is_later(seqno, prev->seqno, prev->ops)) {
+ context = prev->context;
+ chain->prev_seqno = prev->seqno;
+ } else {
+ context = dma_fence_context_alloc(1);
+ /* Make sure that we always have a valid sequence number. */
+ if (prev_chain)
+ seqno = max(prev->seqno, seqno);
+ }
+
+ dma_fence_init(&chain->base, &dma_fence_chain_ops,
+ &chain->lock, context, seqno);
+}
+EXPORT_SYMBOL(dma_fence_chain_init);
struct reservation_object_list *old, *new;
unsigned int i, j, k, max;
+ reservation_object_assert_held(obj);
+
old = reservation_object_get_list(obj);
if (old && old->shared_max) {
dma_fence_get(fence);
+ reservation_object_assert_held(obj);
+
fobj = reservation_object_get_list(obj);
count = fobj->shared_count;
struct reservation_object_list *old;
u32 i = 0;
+ reservation_object_assert_held(obj);
+
old = reservation_object_get_list(obj);
if (old)
i = old->shared_count;
size_t size;
unsigned i;
+ reservation_object_assert_held(dst);
+
rcu_read_lock();
src_list = rcu_dereference(src->fence);
{
struct sync_timeline *parent = dma_fence_parent(fence);
- return !__dma_fence_is_later(fence->seqno, parent->value);
+ return !__dma_fence_is_later(fence->seqno, parent->value, fence->ops);
}
static bool timeline_fence_enable_signaling(struct dma_fence *fence)
i_b++;
} else {
- if (__dma_fence_is_later(pt_a->seqno, pt_b->seqno))
+ if (__dma_fence_is_later(pt_a->seqno, pt_b->seqno,
+ pt_a->ops))
add_fence(fences, &i, pt_a);
else
add_fence(fences, &i, pt_b);
config AXI_DMAC
tristate "Analog Devices AXI-DMAC DMA support"
- depends on MICROBLAZE || NIOS2 || ARCH_ZYNQ || ARCH_SOCFPGA || COMPILE_TEST
+ depends on MICROBLAZE || NIOS2 || ARCH_ZYNQ || ARCH_ZYNQMP || ARCH_SOCFPGA || COMPILE_TEST
select DMA_ENGINE
select DMA_VIRTUAL_CHANNELS
help
* @slave: whether this channel is a device (slave) or for memcpy
* @signal: the physical DMA request signal which this channel is using
* @mux_use: count of descriptors using this DMA request signal setting
+ * @waiting_at: time in jiffies when this channel moved to waiting state
*/
struct pl08x_dma_chan {
struct virt_dma_chan vc;
bool slave;
int signal;
unsigned mux_use;
+ unsigned long waiting_at;
};
/**
if (!ch) {
dev_dbg(&pl08x->adev->dev, "no physical channel available for xfer on %s\n", plchan->name);
plchan->state = PL08X_CHAN_WAITING;
+ plchan->waiting_at = jiffies;
return;
}
{
struct pl08x_driver_data *pl08x = plchan->host;
struct pl08x_dma_chan *p, *next;
-
+ unsigned long waiting_at;
retry:
next = NULL;
+ waiting_at = jiffies;
- /* Find a waiting virtual channel for the next transfer. */
+ /*
+ * Find a waiting virtual channel for the next transfer.
+ * To be fair, time when each channel reached waiting state is compared
+ * to select channel that is waiting for the longest time.
+ */
list_for_each_entry(p, &pl08x->memcpy.channels, vc.chan.device_node)
- if (p->state == PL08X_CHAN_WAITING) {
+ if (p->state == PL08X_CHAN_WAITING &&
+ p->waiting_at <= waiting_at) {
next = p;
- break;
+ waiting_at = p->waiting_at;
}
if (!next && pl08x->has_slave) {
list_for_each_entry(p, &pl08x->slave.channels, vc.chan.device_node)
- if (p->state == PL08X_CHAN_WAITING) {
+ if (p->state == PL08X_CHAN_WAITING &&
+ p->waiting_at <= waiting_at) {
next = p;
- break;
+ waiting_at = p->waiting_at;
}
}
return csize;
};
+static inline bool at_xdmac_chan_is_peripheral_xfer(u32 cfg)
+{
+ return cfg & AT_XDMAC_CC_TYPE_PER_TRAN;
+}
+
static inline u8 at_xdmac_get_dwidth(u32 cfg)
{
return (cfg & AT_XDMAC_CC_DWIDTH_MASK) >> AT_XDMAC_CC_DWIDTH_OFFSET;
at_xdmac_chan_read(atchan, AT_XDMAC_CUBC));
at_xdmac_chan_write(atchan, AT_XDMAC_CID, 0xffffffff);
- reg = AT_XDMAC_CIE_RBEIE | AT_XDMAC_CIE_WBEIE | AT_XDMAC_CIE_ROIE;
+ reg = AT_XDMAC_CIE_RBEIE | AT_XDMAC_CIE_WBEIE;
+ /*
+ * Request Overflow Error is only for peripheral synchronized transfers
+ */
+ if (at_xdmac_chan_is_peripheral_xfer(first->lld.mbr_cfg))
+ reg |= AT_XDMAC_CIE_ROIE;
+
/*
* There is no end of list when doing cyclic dma, we need to get
* an interrupt after each periods.
dmaengine_desc_get_callback_invoke(txd, NULL);
}
+static void at_xdmac_handle_error(struct at_xdmac_chan *atchan)
+{
+ struct at_xdmac *atxdmac = to_at_xdmac(atchan->chan.device);
+ struct at_xdmac_desc *bad_desc;
+
+ /*
+ * The descriptor currently at the head of the active list is
+ * broken. Since we don't have any way to report errors, we'll
+ * just have to scream loudly and try to continue with other
+ * descriptors queued (if any).
+ */
+ if (atchan->irq_status & AT_XDMAC_CIS_RBEIS)
+ dev_err(chan2dev(&atchan->chan), "read bus error!!!");
+ if (atchan->irq_status & AT_XDMAC_CIS_WBEIS)
+ dev_err(chan2dev(&atchan->chan), "write bus error!!!");
+ if (atchan->irq_status & AT_XDMAC_CIS_ROIS)
+ dev_err(chan2dev(&atchan->chan), "request overflow error!!!");
+
+ spin_lock_bh(&atchan->lock);
+
+ /* Channel must be disabled first as it's not done automatically */
+ at_xdmac_write(atxdmac, AT_XDMAC_GD, atchan->mask);
+ while (at_xdmac_read(atxdmac, AT_XDMAC_GS) & atchan->mask)
+ cpu_relax();
+
+ bad_desc = list_first_entry(&atchan->xfers_list,
+ struct at_xdmac_desc,
+ xfer_node);
+
+ spin_unlock_bh(&atchan->lock);
+
+ /* Print bad descriptor's details if needed */
+ dev_dbg(chan2dev(&atchan->chan),
+ "%s: lld: mbr_sa=%pad, mbr_da=%pad, mbr_ubc=0x%08x\n",
+ __func__, &bad_desc->lld.mbr_sa, &bad_desc->lld.mbr_da,
+ bad_desc->lld.mbr_ubc);
+
+ /* Then continue with usual descriptor management */
+}
+
static void at_xdmac_tasklet(unsigned long data)
{
struct at_xdmac_chan *atchan = (struct at_xdmac_chan *)data;
|| (atchan->irq_status & error_mask)) {
struct dma_async_tx_descriptor *txd;
- if (atchan->irq_status & AT_XDMAC_CIS_RBEIS)
- dev_err(chan2dev(&atchan->chan), "read bus error!!!");
- if (atchan->irq_status & AT_XDMAC_CIS_WBEIS)
- dev_err(chan2dev(&atchan->chan), "write bus error!!!");
- if (atchan->irq_status & AT_XDMAC_CIS_ROIS)
- dev_err(chan2dev(&atchan->chan), "request overflow error!!!");
+ if (atchan->irq_status & error_mask)
+ at_xdmac_handle_error(atchan);
spin_lock(&atchan->lock);
desc = list_first_entry(&atchan->xfers_list,
struct at_xdmac_desc,
xfer_node);
dev_vdbg(chan2dev(&atchan->chan), "%s: desc 0x%p\n", __func__, desc);
- BUG_ON(!desc->active_xfer);
+ if (!desc->active_xfer) {
+ dev_err(chan2dev(&atchan->chan), "Xfer not active: exiting");
+ spin_unlock(&atchan->lock);
+ return;
+ }
txd = &desc->tx_dma_desc;
static int sba_debugfs_stats_show(struct seq_file *file, void *offset)
{
- struct platform_device *pdev = to_platform_device(file->private);
- struct sba_device *sba = platform_get_drvdata(pdev);
+ struct sba_device *sba = dev_get_drvdata(file->private);
/* Write stats in file */
sba_write_stats_in_seqfile(sba, file);
dma_cap_set(DMA_SLAVE, od->ddev.cap_mask);
dma_cap_set(DMA_PRIVATE, od->ddev.cap_mask);
dma_cap_set(DMA_CYCLIC, od->ddev.cap_mask);
- dma_cap_set(DMA_SLAVE, od->ddev.cap_mask);
dma_cap_set(DMA_MEMCPY, od->ddev.cap_mask);
od->ddev.device_alloc_chan_resources = bcm2835_dma_alloc_chan_resources;
od->ddev.device_free_chan_resources = bcm2835_dma_free_chan_resources;
static bool axi_dmac_check_len(struct axi_dmac_chan *chan, unsigned int len)
{
- if (len == 0 || len > chan->max_length)
+ if (len == 0)
return false;
if ((len & chan->align_mask) != 0) /* Not aligned */
return false;
return desc;
}
+static struct axi_dmac_sg *axi_dmac_fill_linear_sg(struct axi_dmac_chan *chan,
+ enum dma_transfer_direction direction, dma_addr_t addr,
+ unsigned int num_periods, unsigned int period_len,
+ struct axi_dmac_sg *sg)
+{
+ unsigned int num_segments, i;
+ unsigned int segment_size;
+ unsigned int len;
+
+ /* Split into multiple equally sized segments if necessary */
+ num_segments = DIV_ROUND_UP(period_len, chan->max_length);
+ segment_size = DIV_ROUND_UP(period_len, num_segments);
+ /* Take care of alignment */
+ segment_size = ((segment_size - 1) | chan->align_mask) + 1;
+
+ for (i = 0; i < num_periods; i++) {
+ len = period_len;
+
+ while (len > segment_size) {
+ if (direction == DMA_DEV_TO_MEM)
+ sg->dest_addr = addr;
+ else
+ sg->src_addr = addr;
+ sg->x_len = segment_size;
+ sg->y_len = 1;
+ sg++;
+ addr += segment_size;
+ len -= segment_size;
+ }
+
+ if (direction == DMA_DEV_TO_MEM)
+ sg->dest_addr = addr;
+ else
+ sg->src_addr = addr;
+ sg->x_len = len;
+ sg->y_len = 1;
+ sg++;
+ addr += len;
+ }
+
+ return sg;
+}
+
static struct dma_async_tx_descriptor *axi_dmac_prep_slave_sg(
struct dma_chan *c, struct scatterlist *sgl,
unsigned int sg_len, enum dma_transfer_direction direction,
{
struct axi_dmac_chan *chan = to_axi_dmac_chan(c);
struct axi_dmac_desc *desc;
+ struct axi_dmac_sg *dsg;
struct scatterlist *sg;
+ unsigned int num_sgs;
unsigned int i;
if (direction != chan->direction)
return NULL;
- desc = axi_dmac_alloc_desc(sg_len);
+ num_sgs = 0;
+ for_each_sg(sgl, sg, sg_len, i)
+ num_sgs += DIV_ROUND_UP(sg_dma_len(sg), chan->max_length);
+
+ desc = axi_dmac_alloc_desc(num_sgs);
if (!desc)
return NULL;
+ dsg = desc->sg;
+
for_each_sg(sgl, sg, sg_len, i) {
if (!axi_dmac_check_addr(chan, sg_dma_address(sg)) ||
!axi_dmac_check_len(chan, sg_dma_len(sg))) {
return NULL;
}
- if (direction == DMA_DEV_TO_MEM)
- desc->sg[i].dest_addr = sg_dma_address(sg);
- else
- desc->sg[i].src_addr = sg_dma_address(sg);
- desc->sg[i].x_len = sg_dma_len(sg);
- desc->sg[i].y_len = 1;
+ dsg = axi_dmac_fill_linear_sg(chan, direction, sg_dma_address(sg), 1,
+ sg_dma_len(sg), dsg);
}
desc->cyclic = false;
{
struct axi_dmac_chan *chan = to_axi_dmac_chan(c);
struct axi_dmac_desc *desc;
- unsigned int num_periods, i;
+ unsigned int num_periods, num_segments;
if (direction != chan->direction)
return NULL;
return NULL;
num_periods = buf_len / period_len;
+ num_segments = DIV_ROUND_UP(period_len, chan->max_length);
- desc = axi_dmac_alloc_desc(num_periods);
+ desc = axi_dmac_alloc_desc(num_periods * num_segments);
if (!desc)
return NULL;
- for (i = 0; i < num_periods; i++) {
- if (direction == DMA_DEV_TO_MEM)
- desc->sg[i].dest_addr = buf_addr;
- else
- desc->sg[i].src_addr = buf_addr;
- desc->sg[i].x_len = period_len;
- desc->sg[i].y_len = 1;
- buf_addr += period_len;
- }
+ axi_dmac_fill_linear_sg(chan, direction, buf_addr, num_periods,
+ period_len, desc->sg);
desc->cyclic = true;
if (chan->hw_2d) {
if (!axi_dmac_check_len(chan, xt->sgl[0].size) ||
- !axi_dmac_check_len(chan, xt->numf))
+ xt->numf == 0)
return NULL;
if (xt->sgl[0].size + dst_icg > chan->max_length ||
xt->sgl[0].size + src_icg > chan->max_length)
return ret;
chan->dest_width = val / 8;
- ret = of_property_read_u32(of_chan, "adi,length-width", &val);
- if (ret)
- return ret;
-
- if (val >= 32)
- chan->max_length = UINT_MAX;
- else
- chan->max_length = (1ULL << val) - 1;
-
chan->align_mask = max(chan->dest_width, chan->src_width) - 1;
if (axi_dmac_dest_is_mem(chan) && axi_dmac_src_is_mem(chan))
else
chan->direction = DMA_DEV_TO_DEV;
- chan->hw_cyclic = of_property_read_bool(of_chan, "adi,cyclic");
- chan->hw_2d = of_property_read_bool(of_chan, "adi,2d");
-
return 0;
}
+static void axi_dmac_detect_caps(struct axi_dmac *dmac)
+{
+ struct axi_dmac_chan *chan = &dmac->chan;
+
+ axi_dmac_write(dmac, AXI_DMAC_REG_FLAGS, AXI_DMAC_FLAG_CYCLIC);
+ if (axi_dmac_read(dmac, AXI_DMAC_REG_FLAGS) == AXI_DMAC_FLAG_CYCLIC)
+ chan->hw_cyclic = true;
+
+ axi_dmac_write(dmac, AXI_DMAC_REG_Y_LENGTH, 1);
+ if (axi_dmac_read(dmac, AXI_DMAC_REG_Y_LENGTH) == 1)
+ chan->hw_2d = true;
+
+ axi_dmac_write(dmac, AXI_DMAC_REG_X_LENGTH, 0xffffffff);
+ chan->max_length = axi_dmac_read(dmac, AXI_DMAC_REG_X_LENGTH);
+ if (chan->max_length != UINT_MAX)
+ chan->max_length++;
+}
+
static int axi_dmac_probe(struct platform_device *pdev)
{
struct device_node *of_channels, *of_chan;
of_node_put(of_channels);
pdev->dev.dma_parms = &dmac->dma_parms;
- dma_set_max_seg_size(&pdev->dev, dmac->chan.max_length);
+ dma_set_max_seg_size(&pdev->dev, UINT_MAX);
dma_dev = &dmac->dma_dev;
dma_cap_set(DMA_SLAVE, dma_dev->cap_mask);
dma_cap_set(DMA_CYCLIC, dma_dev->cap_mask);
+ dma_cap_set(DMA_INTERLEAVE, dma_dev->cap_mask);
dma_dev->device_free_chan_resources = axi_dmac_free_chan_resources;
dma_dev->device_tx_status = dma_cookie_status;
dma_dev->device_issue_pending = axi_dmac_issue_pending;
if (ret < 0)
return ret;
+ axi_dmac_detect_caps(dmac);
+
axi_dmac_write(dmac, AXI_DMAC_REG_IRQ_MASK, 0x00);
ret = dma_async_device_register(dma_dev);
};
enum edma_version {
- v1, /* 32ch, Vybdir, mpc57x, etc */
+ v1, /* 32ch, Vybrid, mpc57x, etc */
v2, /* 64ch Coldfire */
};
fsl_edma_irq_handler, 0, "eDMA", fsl_edma);
if (ret) {
dev_err(&pdev->dev, "Can't register eDMA IRQ.\n");
- return ret;
+ return ret;
}
} else {
ret = devm_request_irq(&pdev->dev, fsl_edma->txirq,
fsl_edma_tx_handler, 0, "eDMA tx", fsl_edma);
if (ret) {
dev_err(&pdev->dev, "Can't register eDMA tx IRQ.\n");
- return ret;
+ return ret;
}
ret = devm_request_irq(&pdev->dev, fsl_edma->errirq,
fsl_edma_err_handler, 0, "eDMA err", fsl_edma);
if (ret) {
dev_err(&pdev->dev, "Can't register eDMA err IRQ.\n");
- return ret;
+ return ret;
}
}
#include <linux/platform_device.h>
#include <linux/slab.h>
-#include "idma64.h"
+#include <linux/dma/idma64.h>
-/* Platform driver name */
-#define DRV_NAME "idma64"
+#include "idma64.h"
/* For now we support only two channels */
#define IDMA64_NR_CHAN 2
idma64->dma.directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
idma64->dma.residue_granularity = DMA_RESIDUE_GRANULARITY_BURST;
- idma64->dma.dev = chip->dev;
+ idma64->dma.dev = chip->sysdev;
dma_set_max_seg_size(idma64->dma.dev, IDMA64C_CTLH_BLOCK_TS_MASK);
{
struct idma64_chip *chip;
struct device *dev = &pdev->dev;
+ struct device *sysdev = dev->parent;
struct resource *mem;
int ret;
if (IS_ERR(chip->regs))
return PTR_ERR(chip->regs);
- ret = dma_coerce_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+ ret = dma_coerce_mask_and_coherent(sysdev, DMA_BIT_MASK(64));
if (ret)
return ret;
chip->dev = dev;
+ chip->sysdev = sysdev;
ret = idma64_probe(chip);
if (ret)
.probe = idma64_platform_probe,
.remove = idma64_platform_remove,
.driver = {
- .name = DRV_NAME,
+ .name = LPSS_IDMA64_DRIVER_NAME,
.pm = &idma64_dev_pm_ops,
},
};
MODULE_LICENSE("GPL v2");
MODULE_DESCRIPTION("iDMA64 core driver");
MODULE_AUTHOR("Andy Shevchenko <andriy.shevchenko@linux.intel.com>");
-MODULE_ALIAS("platform:" DRV_NAME);
+MODULE_ALIAS("platform:" LPSS_IDMA64_DRIVER_NAME);
/**
* struct idma64_chip - representation of iDMA 64-bit controller hardware
* @dev: struct device of the DMA controller
+ * @sysdev: struct device of the physical device that does DMA
* @irq: irq line
* @regs: memory mapped I/O space
* @idma64: struct idma64 that is filed by idma64_probe()
*/
struct idma64_chip {
struct device *dev;
+ struct device *sysdev;
int irq;
void __iomem *regs;
struct idma64 *idma64;
int chnenbl0;
int num_events;
struct sdma_script_start_addrs *script_addrs;
+ bool check_ratio;
};
struct sdma_engine {
.script_addrs = &sdma_script_imx7d,
};
+static struct sdma_driver_data sdma_imx8mq = {
+ .chnenbl0 = SDMA_CHNENBL0_IMX35,
+ .num_events = 48,
+ .script_addrs = &sdma_script_imx7d,
+ .check_ratio = 1,
+};
+
static const struct platform_device_id sdma_devtypes[] = {
{
.name = "imx25-sdma",
}, {
.name = "imx7d-sdma",
.driver_data = (unsigned long)&sdma_imx7d,
+ }, {
+ .name = "imx8mq-sdma",
+ .driver_data = (unsigned long)&sdma_imx8mq,
}, {
/* sentinel */
}
{ .compatible = "fsl,imx31-sdma", .data = &sdma_imx31, },
{ .compatible = "fsl,imx25-sdma", .data = &sdma_imx25, },
{ .compatible = "fsl,imx7d-sdma", .data = &sdma_imx7d, },
+ { .compatible = "fsl,imx8mq-sdma", .data = &sdma_imx8mq, },
{ /* sentinel */ }
};
MODULE_DEVICE_TABLE(of, sdma_dt_ids);
if (ret)
goto disable_clk_ipg;
- if (clk_get_rate(sdma->clk_ahb) == clk_get_rate(sdma->clk_ipg))
+ if (sdma->drvdata->check_ratio &&
+ (clk_get_rate(sdma->clk_ahb) == clk_get_rate(sdma->clk_ipg)))
sdma->clk_ratio = 1;
/* Be sure SDMA has not started yet */
#ifdef CONFIG_PM
static int nbpf_runtime_suspend(struct device *dev)
{
- struct nbpf_device *nbpf = platform_get_drvdata(to_platform_device(dev));
+ struct nbpf_device *nbpf = dev_get_drvdata(dev);
clk_disable_unprepare(nbpf->clk);
return 0;
}
static int nbpf_runtime_resume(struct device *dev)
{
- struct nbpf_device *nbpf = platform_get_drvdata(to_platform_device(dev));
+ struct nbpf_device *nbpf = dev_get_drvdata(dev);
return clk_prepare_enable(nbpf->clk);
}
#endif
* (at your option) any later version.
*/
+#include <linux/debugfs.h>
#include <linux/kernel.h>
#include <linux/io.h>
#include <linux/init.h>
{
void __iomem *regs = thrd->dmac->base;
u8 insn[6] = {0, 0, 0, 0, 0, 0};
+ u32 inten = readl(regs + INTEN);
if (_state(thrd) == PL330_STATE_FAULT_COMPLETING)
UNTIL(thrd, PL330_STATE_FAULTING | PL330_STATE_KILLING);
_emit_KILL(0, insn);
- /* Stop generating interrupts for SEV */
- writel(readl(regs + INTEN) & ~(1 << thrd->ev), regs + INTEN);
-
_execute_DBGINSN(thrd, insn, is_manager(thrd));
+
+ /* clear the event */
+ if (inten & (1 << thrd->ev))
+ writel(1 << thrd->ev, regs + INTCLR);
+ /* Stop generating interrupts for SEV */
+ writel(inten & ~(1 << thrd->ev), regs + INTEN);
}
/* Start doing req 'idx' of thread 'thrd' */
BIT(DMA_SLAVE_BUSWIDTH_4_BYTES) | \
BIT(DMA_SLAVE_BUSWIDTH_8_BYTES)
+#ifdef CONFIG_DEBUG_FS
+static int pl330_debugfs_show(struct seq_file *s, void *data)
+{
+ struct pl330_dmac *pl330 = s->private;
+ int chans, pchs, ch, pr;
+
+ chans = pl330->pcfg.num_chan;
+ pchs = pl330->num_peripherals;
+
+ seq_puts(s, "PL330 physical channels:\n");
+ seq_puts(s, "THREAD:\t\tCHANNEL:\n");
+ seq_puts(s, "--------\t-----\n");
+ for (ch = 0; ch < chans; ch++) {
+ struct pl330_thread *thrd = &pl330->channels[ch];
+ int found = -1;
+
+ for (pr = 0; pr < pchs; pr++) {
+ struct dma_pl330_chan *pch = &pl330->peripherals[pr];
+
+ if (!pch->thread || thrd->id != pch->thread->id)
+ continue;
+
+ found = pr;
+ }
+
+ seq_printf(s, "%d\t\t", thrd->id);
+ if (found == -1)
+ seq_puts(s, "--\n");
+ else
+ seq_printf(s, "%d\n", found);
+ }
+
+ return 0;
+}
+
+DEFINE_SHOW_ATTRIBUTE(pl330_debugfs);
+
+static inline void init_pl330_debugfs(struct pl330_dmac *pl330)
+{
+ debugfs_create_file(dev_name(pl330->ddma.dev),
+ S_IFREG | 0444, NULL, pl330,
+ &pl330_debugfs_fops);
+}
+#else
+static inline void init_pl330_debugfs(struct pl330_dmac *pl330)
+{
+}
+#endif
+
/*
* Runtime PM callbacks are provided by amba/bus.c driver.
*
dev_err(&adev->dev, "unable to set the seg size\n");
+ init_pl330_debugfs(pl330);
dev_info(&adev->dev,
"Loaded driver for PL330 DMAC-%x\n", adev->periphid);
dev_info(&adev->dev,
// SPDX-License-Identifier: GPL-2.0
/*
- * Renesas R-Car Gen2 DMA Controller Driver
+ * Renesas R-Car Gen2/Gen3 DMA Controller Driver
*
- * Copyright (C) 2014 Renesas Electronics Inc.
+ * Copyright (C) 2014-2019 Renesas Electronics Inc.
*
* Author: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
*/
return ndtr << width;
}
+/**
+ * stm32_dma_is_current_sg - check that expected sg_req is currently transferred
+ * @chan: dma channel
+ *
+ * This function called when IRQ are disable, checks that the hardware has not
+ * switched on the next transfer in double buffer mode. The test is done by
+ * comparing the next_sg memory address with the hardware related register
+ * (based on CT bit value).
+ *
+ * Returns true if expected current transfer is still running or double
+ * buffer mode is not activated.
+ */
+static bool stm32_dma_is_current_sg(struct stm32_dma_chan *chan)
+{
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ struct stm32_dma_sg_req *sg_req;
+ u32 dma_scr, dma_smar, id;
+
+ id = chan->id;
+ dma_scr = stm32_dma_read(dmadev, STM32_DMA_SCR(id));
+
+ if (!(dma_scr & STM32_DMA_SCR_DBM))
+ return true;
+
+ sg_req = &chan->desc->sg_req[chan->next_sg];
+
+ if (dma_scr & STM32_DMA_SCR_CT) {
+ dma_smar = stm32_dma_read(dmadev, STM32_DMA_SM0AR(id));
+ return (dma_smar == sg_req->chan_reg.dma_sm0ar);
+ }
+
+ dma_smar = stm32_dma_read(dmadev, STM32_DMA_SM1AR(id));
+
+ return (dma_smar == sg_req->chan_reg.dma_sm1ar);
+}
+
static size_t stm32_dma_desc_residue(struct stm32_dma_chan *chan,
struct stm32_dma_desc *desc,
u32 next_sg)
{
u32 modulo, burst_size;
- u32 residue = 0;
+ u32 residue;
+ u32 n_sg = next_sg;
+ struct stm32_dma_sg_req *sg_req = &chan->desc->sg_req[chan->next_sg];
int i;
/*
- * In cyclic mode, for the last period, residue = remaining bytes from
- * NDTR
+ * Calculate the residue means compute the descriptors
+ * information:
+ * - the sg_req currently transferred
+ * - the Hardware remaining position in this sg (NDTR bits field).
+ *
+ * A race condition may occur if DMA is running in cyclic or double
+ * buffer mode, since the DMA register are automatically reloaded at end
+ * of period transfer. The hardware may have switched to the next
+ * transfer (CT bit updated) just before the position (SxNDTR reg) is
+ * read.
+ * In this case the SxNDTR reg could (or not) correspond to the new
+ * transfer position, and not the expected one.
+ * The strategy implemented in the stm32 driver is to:
+ * - read the SxNDTR register
+ * - crosscheck that hardware is still in current transfer.
+ * In case of switch, we can assume that the DMA is at the beginning of
+ * the next transfer. So we approximate the residue in consequence, by
+ * pointing on the beginning of next transfer.
+ *
+ * This race condition doesn't apply for none cyclic mode, as double
+ * buffer is not used. In such situation registers are updated by the
+ * software.
*/
- if (chan->desc->cyclic && next_sg == 0) {
- residue = stm32_dma_get_remaining_bytes(chan);
- goto end;
+
+ residue = stm32_dma_get_remaining_bytes(chan);
+
+ if (!stm32_dma_is_current_sg(chan)) {
+ n_sg++;
+ if (n_sg == chan->desc->num_sgs)
+ n_sg = 0;
+ residue = sg_req->len;
}
/*
- * For all other periods in cyclic mode, and in sg mode,
- * residue = remaining bytes from NDTR + remaining periods/sg to be
- * transferred
+ * In cyclic mode, for the last period, residue = remaining bytes
+ * from NDTR,
+ * else for all other periods in cyclic mode, and in sg mode,
+ * residue = remaining bytes from NDTR + remaining
+ * periods/sg to be transferred
*/
- for (i = next_sg; i < desc->num_sgs; i++)
- residue += desc->sg_req[i].len;
- residue += stm32_dma_get_remaining_bytes(chan);
+ if (!chan->desc->cyclic || n_sg != 0)
+ for (i = n_sg; i < desc->num_sgs; i++)
+ residue += desc->sg_req[i].len;
-end:
if (!chan->mem_burst)
return residue;
for (i = 0; i < STM32_DMA_MAX_CHANNELS; i++) {
chan = &dmadev->chan[i];
- res = platform_get_resource(pdev, IORESOURCE_IRQ, i);
- if (!res) {
- ret = -EINVAL;
- dev_err(&pdev->dev, "No irq resource for chan %d\n", i);
+ chan->irq = platform_get_irq(pdev, i);
+ ret = platform_get_irq(pdev, i);
+ if (ret < 0) {
+ if (ret != -EPROBE_DEFER)
+ dev_err(&pdev->dev,
+ "No irq resource for chan %d\n", i);
goto err_unregister;
}
- chan->irq = res->start;
+ chan->irq = ret;
+
ret = devm_request_irq(&pdev->dev, chan->irq,
stm32_dma_chan_irq, 0,
dev_name(chan2dev(chan)), chan);
#include <linux/of_device.h>
#include <linux/of_dma.h>
#include <linux/of_irq.h>
-#include <linux/pm_clock.h>
#include <linux/pm_runtime.h>
#include <linux/slab.h>
#define ADMA_CH_CMD 0x00
#define ADMA_CH_STATUS 0x0c
#define ADMA_CH_STATUS_XFER_EN BIT(0)
+#define ADMA_CH_STATUS_XFER_PAUSED BIT(1)
#define ADMA_CH_INT_STATUS 0x10
#define ADMA_CH_INT_STATUS_XFER_DONE BIT(0)
#define ADMA_CH_INT_CLEAR 0x1c
#define ADMA_CH_CTRL 0x24
-#define ADMA_CH_CTRL_TX_REQ(val) (((val) & 0xf) << 28)
-#define ADMA_CH_CTRL_TX_REQ_MAX 10
-#define ADMA_CH_CTRL_RX_REQ(val) (((val) & 0xf) << 24)
-#define ADMA_CH_CTRL_RX_REQ_MAX 10
#define ADMA_CH_CTRL_DIR(val) (((val) & 0xf) << 12)
#define ADMA_CH_CTRL_DIR_AHUB2MEM 2
#define ADMA_CH_CTRL_DIR_MEM2AHUB 4
#define ADMA_CH_CTRL_MODE_CONTINUOUS (2 << 8)
#define ADMA_CH_CTRL_FLOWCTRL_EN BIT(1)
+#define ADMA_CH_CTRL_XFER_PAUSE_SHIFT 0
#define ADMA_CH_CONFIG 0x28
#define ADMA_CH_CONFIG_SRC_BUF(val) (((val) & 0x7) << 28)
#define ADMA_CH_CONFIG_TRG_BUF(val) (((val) & 0x7) << 24)
-#define ADMA_CH_CONFIG_BURST_SIZE(val) (((val) & 0x7) << 20)
-#define ADMA_CH_CONFIG_BURST_16 5
+#define ADMA_CH_CONFIG_BURST_SIZE_SHIFT 20
+#define ADMA_CH_CONFIG_MAX_BURST_SIZE 16
#define ADMA_CH_CONFIG_WEIGHT_FOR_WRR(val) ((val) & 0xf)
#define ADMA_CH_CONFIG_MAX_BUFS 8
#define ADMA_CH_FIFO_CTRL 0x2c
#define ADMA_CH_FIFO_CTRL_OVRFW_THRES(val) (((val) & 0xf) << 24)
#define ADMA_CH_FIFO_CTRL_STARV_THRES(val) (((val) & 0xf) << 16)
-#define ADMA_CH_FIFO_CTRL_TX_SIZE(val) (((val) & 0xf) << 8)
-#define ADMA_CH_FIFO_CTRL_RX_SIZE(val) ((val) & 0xf)
+#define ADMA_CH_FIFO_CTRL_TX_FIFO_SIZE_SHIFT 8
+#define ADMA_CH_FIFO_CTRL_RX_FIFO_SIZE_SHIFT 0
#define ADMA_CH_LOWER_SRC_ADDR 0x34
#define ADMA_CH_LOWER_TRG_ADDR 0x3c
#define ADMA_CH_XFER_STATUS 0x54
#define ADMA_CH_XFER_STATUS_COUNT_MASK 0xffff
-#define ADMA_GLOBAL_CMD 0xc00
-#define ADMA_GLOBAL_SOFT_RESET 0xc04
-#define ADMA_GLOBAL_INT_CLEAR 0xc20
-#define ADMA_GLOBAL_CTRL 0xc24
+#define ADMA_GLOBAL_CMD 0x00
+#define ADMA_GLOBAL_SOFT_RESET 0x04
-#define ADMA_CH_REG_OFFSET(a) (a * 0x80)
+#define TEGRA_ADMA_BURST_COMPLETE_TIME 20
#define ADMA_CH_FIFO_CTRL_DEFAULT (ADMA_CH_FIFO_CTRL_OVRFW_THRES(1) | \
- ADMA_CH_FIFO_CTRL_STARV_THRES(1) | \
- ADMA_CH_FIFO_CTRL_TX_SIZE(3) | \
- ADMA_CH_FIFO_CTRL_RX_SIZE(3))
+ ADMA_CH_FIFO_CTRL_STARV_THRES(1))
+
+#define ADMA_CH_REG_FIELD_VAL(val, mask, shift) (((val) & mask) << shift)
+
struct tegra_adma;
/*
* struct tegra_adma_chip_data - Tegra chip specific data
+ * @global_reg_offset: Register offset of DMA global register.
+ * @global_int_clear: Register offset of DMA global interrupt clear.
+ * @ch_req_tx_shift: Register offset for AHUB transmit channel select.
+ * @ch_req_rx_shift: Register offset for AHUB receive channel select.
+ * @ch_base_offset: Reister offset of DMA channel registers.
+ * @ch_req_mask: Mask for Tx or Rx channel select.
+ * @ch_req_max: Maximum number of Tx or Rx channels available.
+ * @ch_reg_size: Size of DMA channel register space.
* @nr_channels: Number of DMA channels available.
*/
struct tegra_adma_chip_data {
- int nr_channels;
+ unsigned int (*adma_get_burst_config)(unsigned int burst_size);
+ unsigned int global_reg_offset;
+ unsigned int global_int_clear;
+ unsigned int ch_req_tx_shift;
+ unsigned int ch_req_rx_shift;
+ unsigned int ch_base_offset;
+ unsigned int ch_req_mask;
+ unsigned int ch_req_max;
+ unsigned int ch_reg_size;
+ unsigned int nr_channels;
};
/*
unsigned int src_addr;
unsigned int trg_addr;
unsigned int fifo_ctrl;
+ unsigned int cmd;
unsigned int tc;
};
enum dma_transfer_direction sreq_dir;
unsigned int sreq_index;
bool sreq_reserved;
+ struct tegra_adma_chan_regs ch_regs;
/* Transfer count and position info */
unsigned int tx_buf_count;
struct dma_device dma_dev;
struct device *dev;
void __iomem *base_addr;
+ struct clk *ahub_clk;
unsigned int nr_channels;
unsigned long rx_requests_reserved;
unsigned long tx_requests_reserved;
/* Used to store global command register state when suspending */
unsigned int global_cmd;
+ const struct tegra_adma_chip_data *cdata;
+
/* Last member of the structure */
struct tegra_adma_chan channels[0];
};
static inline void tdma_write(struct tegra_adma *tdma, u32 reg, u32 val)
{
- writel(val, tdma->base_addr + reg);
+ writel(val, tdma->base_addr + tdma->cdata->global_reg_offset + reg);
}
static inline u32 tdma_read(struct tegra_adma *tdma, u32 reg)
{
- return readl(tdma->base_addr + reg);
+ return readl(tdma->base_addr + tdma->cdata->global_reg_offset + reg);
}
static inline void tdma_ch_write(struct tegra_adma_chan *tdc, u32 reg, u32 val)
int ret;
/* Clear any interrupts */
- tdma_write(tdma, ADMA_GLOBAL_INT_CLEAR, 0x1);
+ tdma_write(tdma, tdma->cdata->global_int_clear, 0x1);
/* Assert soft reset */
tdma_write(tdma, ADMA_GLOBAL_SOFT_RESET, 0x1);
/* Wait for reset to clear */
ret = readx_poll_timeout(readl,
- tdma->base_addr + ADMA_GLOBAL_SOFT_RESET,
+ tdma->base_addr +
+ tdma->cdata->global_reg_offset +
+ ADMA_GLOBAL_SOFT_RESET,
status, status == 0, 20, 10000);
if (ret)
return ret;
if (tdc->sreq_reserved)
return tdc->sreq_dir == direction ? 0 : -EINVAL;
+ if (sreq_index > tdma->cdata->ch_req_max) {
+ dev_err(tdma->dev, "invalid DMA request\n");
+ return -EINVAL;
+ }
+
switch (direction) {
case DMA_MEM_TO_DEV:
- if (sreq_index > ADMA_CH_CTRL_TX_REQ_MAX) {
- dev_err(tdma->dev, "invalid DMA request\n");
- return -EINVAL;
- }
-
if (test_and_set_bit(sreq_index, &tdma->tx_requests_reserved)) {
dev_err(tdma->dev, "DMA request reserved\n");
return -EINVAL;
break;
case DMA_DEV_TO_MEM:
- if (sreq_index > ADMA_CH_CTRL_RX_REQ_MAX) {
- dev_err(tdma->dev, "invalid DMA request\n");
- return -EINVAL;
- }
-
if (test_and_set_bit(sreq_index, &tdma->rx_requests_reserved)) {
dev_err(tdma->dev, "DMA request reserved\n");
return -EINVAL;
spin_unlock_irqrestore(&tdc->vc.lock, flags);
}
+static bool tegra_adma_is_paused(struct tegra_adma_chan *tdc)
+{
+ u32 csts;
+
+ csts = tdma_ch_read(tdc, ADMA_CH_STATUS);
+ csts &= ADMA_CH_STATUS_XFER_PAUSED;
+
+ return csts ? true : false;
+}
+
+static int tegra_adma_pause(struct dma_chan *dc)
+{
+ struct tegra_adma_chan *tdc = to_tegra_adma_chan(dc);
+ struct tegra_adma_desc *desc = tdc->desc;
+ struct tegra_adma_chan_regs *ch_regs = &desc->ch_regs;
+ int dcnt = 10;
+
+ ch_regs->ctrl = tdma_ch_read(tdc, ADMA_CH_CTRL);
+ ch_regs->ctrl |= (1 << ADMA_CH_CTRL_XFER_PAUSE_SHIFT);
+ tdma_ch_write(tdc, ADMA_CH_CTRL, ch_regs->ctrl);
+
+ while (dcnt-- && !tegra_adma_is_paused(tdc))
+ udelay(TEGRA_ADMA_BURST_COMPLETE_TIME);
+
+ if (dcnt < 0) {
+ dev_err(tdc2dev(tdc), "unable to pause DMA channel\n");
+ return -EBUSY;
+ }
+
+ return 0;
+}
+
+static int tegra_adma_resume(struct dma_chan *dc)
+{
+ struct tegra_adma_chan *tdc = to_tegra_adma_chan(dc);
+ struct tegra_adma_desc *desc = tdc->desc;
+ struct tegra_adma_chan_regs *ch_regs = &desc->ch_regs;
+
+ ch_regs->ctrl = tdma_ch_read(tdc, ADMA_CH_CTRL);
+ ch_regs->ctrl &= ~(1 << ADMA_CH_CTRL_XFER_PAUSE_SHIFT);
+ tdma_ch_write(tdc, ADMA_CH_CTRL, ch_regs->ctrl);
+
+ return 0;
+}
+
static int tegra_adma_terminate_all(struct dma_chan *dc)
{
struct tegra_adma_chan *tdc = to_tegra_adma_chan(dc);
return ret;
}
+static unsigned int tegra210_adma_get_burst_config(unsigned int burst_size)
+{
+ if (!burst_size || burst_size > ADMA_CH_CONFIG_MAX_BURST_SIZE)
+ burst_size = ADMA_CH_CONFIG_MAX_BURST_SIZE;
+
+ return fls(burst_size) << ADMA_CH_CONFIG_BURST_SIZE_SHIFT;
+}
+
+static unsigned int tegra186_adma_get_burst_config(unsigned int burst_size)
+{
+ if (!burst_size || burst_size > ADMA_CH_CONFIG_MAX_BURST_SIZE)
+ burst_size = ADMA_CH_CONFIG_MAX_BURST_SIZE;
+
+ return (burst_size - 1) << ADMA_CH_CONFIG_BURST_SIZE_SHIFT;
+}
+
static int tegra_adma_set_xfer_params(struct tegra_adma_chan *tdc,
struct tegra_adma_desc *desc,
dma_addr_t buf_addr,
enum dma_transfer_direction direction)
{
struct tegra_adma_chan_regs *ch_regs = &desc->ch_regs;
+ const struct tegra_adma_chip_data *cdata = tdc->tdma->cdata;
unsigned int burst_size, adma_dir;
if (desc->num_periods > ADMA_CH_CONFIG_MAX_BUFS)
switch (direction) {
case DMA_MEM_TO_DEV:
adma_dir = ADMA_CH_CTRL_DIR_MEM2AHUB;
- burst_size = fls(tdc->sconfig.dst_maxburst);
+ burst_size = tdc->sconfig.dst_maxburst;
ch_regs->config = ADMA_CH_CONFIG_SRC_BUF(desc->num_periods - 1);
- ch_regs->ctrl = ADMA_CH_CTRL_TX_REQ(tdc->sreq_index);
+ ch_regs->ctrl = ADMA_CH_REG_FIELD_VAL(tdc->sreq_index,
+ cdata->ch_req_mask,
+ cdata->ch_req_tx_shift);
ch_regs->src_addr = buf_addr;
break;
case DMA_DEV_TO_MEM:
adma_dir = ADMA_CH_CTRL_DIR_AHUB2MEM;
- burst_size = fls(tdc->sconfig.src_maxburst);
+ burst_size = tdc->sconfig.src_maxburst;
ch_regs->config = ADMA_CH_CONFIG_TRG_BUF(desc->num_periods - 1);
- ch_regs->ctrl = ADMA_CH_CTRL_RX_REQ(tdc->sreq_index);
+ ch_regs->ctrl = ADMA_CH_REG_FIELD_VAL(tdc->sreq_index,
+ cdata->ch_req_mask,
+ cdata->ch_req_rx_shift);
ch_regs->trg_addr = buf_addr;
break;
return -EINVAL;
}
- if (!burst_size || burst_size > ADMA_CH_CONFIG_BURST_16)
- burst_size = ADMA_CH_CONFIG_BURST_16;
-
ch_regs->ctrl |= ADMA_CH_CTRL_DIR(adma_dir) |
ADMA_CH_CTRL_MODE_CONTINUOUS |
ADMA_CH_CTRL_FLOWCTRL_EN;
- ch_regs->config |= ADMA_CH_CONFIG_BURST_SIZE(burst_size);
+ ch_regs->config |= cdata->adma_get_burst_config(burst_size);
ch_regs->config |= ADMA_CH_CONFIG_WEIGHT_FOR_WRR(1);
ch_regs->fifo_ctrl = ADMA_CH_FIFO_CTRL_DEFAULT;
ch_regs->tc = desc->period_len & ADMA_CH_TC_COUNT_MASK;
static int tegra_adma_runtime_suspend(struct device *dev)
{
struct tegra_adma *tdma = dev_get_drvdata(dev);
+ struct tegra_adma_chan_regs *ch_reg;
+ struct tegra_adma_chan *tdc;
+ int i;
tdma->global_cmd = tdma_read(tdma, ADMA_GLOBAL_CMD);
+ if (!tdma->global_cmd)
+ goto clk_disable;
+
+ for (i = 0; i < tdma->nr_channels; i++) {
+ tdc = &tdma->channels[i];
+ ch_reg = &tdc->ch_regs;
+ ch_reg->cmd = tdma_ch_read(tdc, ADMA_CH_CMD);
+ /* skip if channel is not active */
+ if (!ch_reg->cmd)
+ continue;
+ ch_reg->tc = tdma_ch_read(tdc, ADMA_CH_TC);
+ ch_reg->src_addr = tdma_ch_read(tdc, ADMA_CH_LOWER_SRC_ADDR);
+ ch_reg->trg_addr = tdma_ch_read(tdc, ADMA_CH_LOWER_TRG_ADDR);
+ ch_reg->ctrl = tdma_ch_read(tdc, ADMA_CH_CTRL);
+ ch_reg->fifo_ctrl = tdma_ch_read(tdc, ADMA_CH_FIFO_CTRL);
+ ch_reg->config = tdma_ch_read(tdc, ADMA_CH_CONFIG);
+ }
+
+clk_disable:
+ clk_disable_unprepare(tdma->ahub_clk);
- return pm_clk_suspend(dev);
+ return 0;
}
static int tegra_adma_runtime_resume(struct device *dev)
{
struct tegra_adma *tdma = dev_get_drvdata(dev);
- int ret;
+ struct tegra_adma_chan_regs *ch_reg;
+ struct tegra_adma_chan *tdc;
+ int ret, i;
- ret = pm_clk_resume(dev);
- if (ret)
+ ret = clk_prepare_enable(tdma->ahub_clk);
+ if (ret) {
+ dev_err(dev, "ahub clk_enable failed: %d\n", ret);
return ret;
-
+ }
tdma_write(tdma, ADMA_GLOBAL_CMD, tdma->global_cmd);
+ if (!tdma->global_cmd)
+ return 0;
+
+ for (i = 0; i < tdma->nr_channels; i++) {
+ tdc = &tdma->channels[i];
+ ch_reg = &tdc->ch_regs;
+ /* skip if channel was not active earlier */
+ if (!ch_reg->cmd)
+ continue;
+ tdma_ch_write(tdc, ADMA_CH_TC, ch_reg->tc);
+ tdma_ch_write(tdc, ADMA_CH_LOWER_SRC_ADDR, ch_reg->src_addr);
+ tdma_ch_write(tdc, ADMA_CH_LOWER_TRG_ADDR, ch_reg->trg_addr);
+ tdma_ch_write(tdc, ADMA_CH_CTRL, ch_reg->ctrl);
+ tdma_ch_write(tdc, ADMA_CH_FIFO_CTRL, ch_reg->fifo_ctrl);
+ tdma_ch_write(tdc, ADMA_CH_CONFIG, ch_reg->config);
+ tdma_ch_write(tdc, ADMA_CH_CMD, ch_reg->cmd);
+ }
+
return 0;
}
static const struct tegra_adma_chip_data tegra210_chip_data = {
- .nr_channels = 22,
+ .adma_get_burst_config = tegra210_adma_get_burst_config,
+ .global_reg_offset = 0xc00,
+ .global_int_clear = 0x20,
+ .ch_req_tx_shift = 28,
+ .ch_req_rx_shift = 24,
+ .ch_base_offset = 0,
+ .ch_req_mask = 0xf,
+ .ch_req_max = 10,
+ .ch_reg_size = 0x80,
+ .nr_channels = 22,
+};
+
+static const struct tegra_adma_chip_data tegra186_chip_data = {
+ .adma_get_burst_config = tegra186_adma_get_burst_config,
+ .global_reg_offset = 0,
+ .global_int_clear = 0x402c,
+ .ch_req_tx_shift = 27,
+ .ch_req_rx_shift = 22,
+ .ch_base_offset = 0x10000,
+ .ch_req_mask = 0x1f,
+ .ch_req_max = 20,
+ .ch_reg_size = 0x100,
+ .nr_channels = 32,
};
static const struct of_device_id tegra_adma_of_match[] = {
{ .compatible = "nvidia,tegra210-adma", .data = &tegra210_chip_data },
+ { .compatible = "nvidia,tegra186-adma", .data = &tegra186_chip_data },
{ },
};
MODULE_DEVICE_TABLE(of, tegra_adma_of_match);
return -ENOMEM;
tdma->dev = &pdev->dev;
+ tdma->cdata = cdata;
tdma->nr_channels = cdata->nr_channels;
platform_set_drvdata(pdev, tdma);
if (IS_ERR(tdma->base_addr))
return PTR_ERR(tdma->base_addr);
- ret = pm_clk_create(&pdev->dev);
- if (ret)
- return ret;
-
- ret = of_pm_clk_add_clk(&pdev->dev, "d_audio");
- if (ret)
- goto clk_destroy;
+ tdma->ahub_clk = devm_clk_get(&pdev->dev, "d_audio");
+ if (IS_ERR(tdma->ahub_clk)) {
+ dev_err(&pdev->dev, "Error: Missing ahub controller clock\n");
+ return PTR_ERR(tdma->ahub_clk);
+ }
pm_runtime_enable(&pdev->dev);
for (i = 0; i < tdma->nr_channels; i++) {
struct tegra_adma_chan *tdc = &tdma->channels[i];
- tdc->chan_addr = tdma->base_addr + ADMA_CH_REG_OFFSET(i);
+ tdc->chan_addr = tdma->base_addr + cdata->ch_base_offset
+ + (cdata->ch_reg_size * i);
tdc->irq = of_irq_get(pdev->dev.of_node, i);
if (tdc->irq <= 0) {
tdma->dma_dev.dst_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_4_BYTES);
tdma->dma_dev.directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
tdma->dma_dev.residue_granularity = DMA_RESIDUE_GRANULARITY_SEGMENT;
+ tdma->dma_dev.device_pause = tegra_adma_pause;
+ tdma->dma_dev.device_resume = tegra_adma_resume;
ret = dma_async_device_register(&tdma->dma_dev);
if (ret < 0) {
pm_runtime_put_sync(&pdev->dev);
rpm_disable:
pm_runtime_disable(&pdev->dev);
-clk_destroy:
- pm_clk_destroy(&pdev->dev);
return ret;
}
struct tegra_adma *tdma = platform_get_drvdata(pdev);
int i;
+ of_dma_controller_free(pdev->dev.of_node);
dma_async_device_unregister(&tdma->dma_dev);
for (i = 0; i < tdma->nr_channels; ++i)
pm_runtime_put_sync(&pdev->dev);
pm_runtime_disable(&pdev->dev);
- pm_clk_destroy(&pdev->dev);
return 0;
}
-#ifdef CONFIG_PM_SLEEP
-static int tegra_adma_pm_suspend(struct device *dev)
-{
- return pm_runtime_suspended(dev) == false;
-}
-#endif
-
static const struct dev_pm_ops tegra_adma_dev_pm_ops = {
SET_RUNTIME_PM_OPS(tegra_adma_runtime_suspend,
tegra_adma_runtime_resume, NULL)
- SET_SYSTEM_SLEEP_PM_OPS(tegra_adma_pm_suspend, NULL)
+ SET_LATE_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
+ pm_runtime_force_resume)
};
static struct platform_driver tegra_admac_driver = {
INIT_LIST_HEAD(&ld_completed);
- spin_lock_bh(&chan->lock);
+ spin_lock(&chan->lock);
/* Clean already completed and acked descriptors */
xgene_dma_clean_completed_descriptor(chan);
*/
xgene_chan_xfer_ld_pending(chan);
- spin_unlock_bh(&chan->lock);
+ spin_unlock(&chan->lock);
/* Run the callback for each descriptor, in order */
list_for_each_entry_safe(desc_sw, _desc_sw, &ld_completed, node) {
return -ENOMEM;
}
- chan_dbg(chan, "Allocate descripto pool\n");
+ chan_dbg(chan, "Allocate descriptor pool\n");
return 1;
}
help
Choose this if you need the KMS CMA helper functions
+config DRM_GEM_SHMEM_HELPER
+ bool
+ depends on DRM
+ help
+ Choose this if you need the GEM shmem helper functions
+
config DRM_VM
bool
depends on DRM && MMU
source "drivers/gpu/drm/amd/amdgpu/Kconfig"
-source "drivers/gpu/drm/amd/lib/Kconfig"
-
source "drivers/gpu/drm/nouveau/Kconfig"
source "drivers/gpu/drm/i915/Kconfig"
If M is selected the module will be called vkms.
+config DRM_ATI_PCIGART
+ bool
+
source "drivers/gpu/drm/exynos/Kconfig"
source "drivers/gpu/drm/rockchip/Kconfig"
source "drivers/gpu/drm/xen/Kconfig"
+source "drivers/gpu/drm/vboxvideo/Kconfig"
+
+source "drivers/gpu/drm/lima/Kconfig"
+
+source "drivers/gpu/drm/panfrost/Kconfig"
+
+source "drivers/gpu/drm/aspeed/Kconfig"
+
# Keep legacy drivers last
menuconfig DRM_LEGACY
bool "Enable legacy drivers (DANGEROUS)"
depends on DRM && MMU
select DRM_VM
+ select DRM_ATI_PCIGART if PCI
help
Enable legacy DRI1 drivers. Those drivers expose unsafe and dangerous
APIs to user-space, which can be used to circumvent access
# Makefile for the drm device driver. This driver provides support for the
# Direct Rendering Infrastructure (DRI) in XFree86 4.1.0 and higher.
-drm-y := drm_auth.o drm_bufs.o drm_cache.o \
- drm_context.o drm_dma.o \
+drm-y := drm_auth.o drm_cache.o \
drm_file.o drm_gem.o drm_ioctl.o drm_irq.o \
- drm_lock.o drm_memory.o drm_drv.o \
- drm_scatter.o drm_pci.o \
+ drm_memory.o drm_drv.o drm_pci.o \
drm_sysfs.o drm_hashtab.o drm_mm.o \
drm_crtc.o drm_fourcc.o drm_modes.o drm_edid.o \
drm_encoder_slave.o \
drm_syncobj.o drm_lease.o drm_writeback.o drm_client.o \
drm_atomic_uapi.o
+drm-$(CONFIG_DRM_LEGACY) += drm_legacy_misc.o drm_bufs.o drm_context.o drm_dma.o drm_scatter.o drm_lock.o
drm-$(CONFIG_DRM_LIB_RANDOM) += lib/drm_random.o
drm-$(CONFIG_DRM_VM) += drm_vm.o
drm-$(CONFIG_COMPAT) += drm_ioc32.o
drm-$(CONFIG_DRM_GEM_CMA_HELPER) += drm_gem_cma_helper.o
-drm-$(CONFIG_PCI) += ati_pcigart.o
+drm-$(CONFIG_DRM_GEM_SHMEM_HELPER) += drm_gem_shmem_helper.o
+drm-$(CONFIG_DRM_ATI_PCIGART) += ati_pcigart.o
drm-$(CONFIG_DRM_PANEL) += drm_panel.o
drm-$(CONFIG_OF) += drm_of.o
drm-$(CONFIG_AGP) += drm_agpsupport.o
drm_kms_helper_common.o drm_dp_dual_mode_helper.o \
drm_simple_kms_helper.o drm_modeset_helper.o \
drm_scdc_helper.o drm_gem_framebuffer_helper.o \
- drm_atomic_state_helper.o drm_damage_helper.o
+ drm_atomic_state_helper.o drm_damage_helper.o \
+ drm_format_helper.o
drm_kms_helper-$(CONFIG_DRM_PANEL_BRIDGE) += bridge/panel.o
drm_kms_helper-$(CONFIG_DRM_FBDEV_EMULATION) += drm_fb_helper.o
obj-$(CONFIG_DRM_SCHED) += scheduler/
obj-$(CONFIG_DRM_TDFX) += tdfx/
obj-$(CONFIG_DRM_R128) += r128/
-obj-y += amd/lib/
obj-$(CONFIG_HSA_AMD) += amd/amdkfd/
obj-$(CONFIG_DRM_RADEON)+= radeon/
obj-$(CONFIG_DRM_AMDGPU)+= amd/amdgpu/
obj-$(CONFIG_DRM_PL111) += pl111/
obj-$(CONFIG_DRM_TVE200) += tve200/
obj-$(CONFIG_DRM_XEN) += xen/
+obj-$(CONFIG_DRM_VBOXVIDEO) += vboxvideo/
+obj-$(CONFIG_DRM_LIMA) += lima/
+obj-$(CONFIG_DRM_PANFROST) += panfrost/
+obj-$(CONFIG_DRM_ASPEED_GFX) += aspeed/
# Makefile for the drm device driver. This driver provides support for the
# Direct Rendering Infrastructure (DRI) in XFree86 4.1.0 and higher.
-FULL_AMD_PATH=$(src)/..
+FULL_AMD_PATH=$(srctree)/$(src)/..
DISPLAY_FOLDER_NAME=display
FULL_AMD_DISPLAY_PATH = $(FULL_AMD_PATH)/$(DISPLAY_FOLDER_NAME)
amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \
amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_virt.o amdgpu_atomfirmware.o \
amdgpu_vf_error.o amdgpu_sched.o amdgpu_debugfs.o amdgpu_ids.o \
- amdgpu_gmc.o amdgpu_xgmi.o amdgpu_csa.o
+ amdgpu_gmc.o amdgpu_xgmi.o amdgpu_csa.o amdgpu_ras.o amdgpu_vm_cpu.o \
+ amdgpu_vm_sdma.o
# add asic specific block
amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o kv_dpm.o \
#include "amdgpu_gem.h"
#include "amdgpu_doorbell.h"
#include "amdgpu_amdkfd.h"
+#include "amdgpu_smu.h"
#define MAX_GPU_INSTANCE 16
extern uint amdgpu_smu_memory_pool_size;
extern uint amdgpu_dc_feature_mask;
extern struct amdgpu_mgpu_info mgpu_info;
+extern int amdgpu_ras_enable;
+extern uint amdgpu_ras_mask;
#ifdef CONFIG_DRM_AMDGPU_SI
extern int amdgpu_si_support;
void *kdata;
};
+struct amdgpu_cs_post_dep {
+ struct drm_syncobj *syncobj;
+ struct dma_fence_chain *chain;
+ u64 point;
+};
+
struct amdgpu_cs_parser {
struct amdgpu_device *adev;
struct drm_file *filp;
/* user fence */
struct amdgpu_bo_list_entry uf_entry;
- unsigned num_post_dep_syncobjs;
- struct drm_syncobj **post_dep_syncobjs;
+ unsigned num_post_deps;
+ struct amdgpu_cs_post_dep *post_deps;
};
static inline u32 amdgpu_get_ib_value(struct amdgpu_cs_parser *p,
struct amd_powerplay {
void *pp_handle;
const struct amd_pm_funcs *pp_funcs;
- uint32_t pp_feature;
};
#define AMDGPU_RESET_MAGIC_NUM 64
/* For pre-DCE11. DCE11 and later are in "struct amdgpu_device->dm" */
struct work_struct hotplug_work;
struct amdgpu_irq_src crtc_irq;
+ struct amdgpu_irq_src vupdate_irq;
struct amdgpu_irq_src pageflip_irq;
struct amdgpu_irq_src hpd_irq;
struct amd_powerplay powerplay;
bool pp_force_state_enabled;
+ /* smu */
+ struct smu_context smu;
+
/* dpm */
struct amdgpu_pm pm;
u32 cg_flags;
int asic_reset_res;
struct work_struct xgmi_reset_work;
+
+ bool in_baco_reset;
};
static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
amdgpu_bo_unref(&(bo));
}
+uint32_t amdgpu_amdkfd_get_fw_version(struct kgd_dev *kgd,
+ enum kgd_engine_type type)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)kgd;
+
+ switch (type) {
+ case KGD_ENGINE_PFP:
+ return adev->gfx.pfp_fw_version;
+
+ case KGD_ENGINE_ME:
+ return adev->gfx.me_fw_version;
+
+ case KGD_ENGINE_CE:
+ return adev->gfx.ce_fw_version;
+
+ case KGD_ENGINE_MEC1:
+ return adev->gfx.mec_fw_version;
+
+ case KGD_ENGINE_MEC2:
+ return adev->gfx.mec2_fw_version;
+
+ case KGD_ENGINE_RLC:
+ return adev->gfx.rlc_fw_version;
+
+ case KGD_ENGINE_SDMA1:
+ return adev->sdma.instance[0].fw_version;
+
+ case KGD_ENGINE_SDMA2:
+ return adev->sdma.instance[1].fw_version;
+
+ default:
+ return 0;
+ }
+
+ return 0;
+}
+
void amdgpu_amdkfd_get_local_mem_info(struct kgd_dev *kgd,
struct kfd_local_mem_info *mem_info)
{
void kgd2kfd_interrupt(struct kfd_dev *kfd, const void *ih_ring_entry)
{
}
+
+void kgd2kfd_set_sram_ecc_flag(struct kfd_dev *kfd)
+{
+}
#endif
uint64_t vram_used;
};
+enum kgd_engine_type {
+ KGD_ENGINE_PFP = 1,
+ KGD_ENGINE_ME,
+ KGD_ENGINE_CE,
+ KGD_ENGINE_MEC1,
+ KGD_ENGINE_MEC2,
+ KGD_ENGINE_RLC,
+ KGD_ENGINE_SDMA1,
+ KGD_ENGINE_SDMA2,
+ KGD_ENGINE_MAX
+};
+
struct amdgpu_amdkfd_fence *amdgpu_amdkfd_fence_create(u64 context,
struct mm_struct *mm);
bool amdkfd_fence_check_mm(struct dma_fence *f, struct mm_struct *mm);
void **mem_obj, uint64_t *gpu_addr,
void **cpu_ptr, bool mqd_gfx9);
void amdgpu_amdkfd_free_gtt_mem(struct kgd_dev *kgd, void *mem_obj);
+uint32_t amdgpu_amdkfd_get_fw_version(struct kgd_dev *kgd,
+ enum kgd_engine_type type);
void amdgpu_amdkfd_get_local_mem_info(struct kgd_dev *kgd,
struct kfd_local_mem_info *mem_info);
uint64_t amdgpu_amdkfd_get_gpu_clock_counter(struct kgd_dev *kgd);
int kgd2kfd_resume_mm(struct mm_struct *mm);
int kgd2kfd_schedule_evict_and_restore_process(struct mm_struct *mm,
struct dma_fence *fence);
+void kgd2kfd_set_sram_ecc_flag(struct kfd_dev *kfd);
#endif /* AMDGPU_AMDKFD_H_INCLUDED */
#include <linux/fdtable.h>
#include <linux/uaccess.h>
-#include <linux/firmware.h>
#include <linux/mmu_context.h>
#include <drm/drmP.h>
#include "amdgpu.h"
#include "amdgpu_amdkfd.h"
#include "cikd.h"
#include "cik_sdma.h"
-#include "amdgpu_ucode.h"
#include "gfx_v7_0.h"
#include "gca/gfx_7_2_d.h"
#include "gca/gfx_7_2_enum.h"
static uint16_t get_atc_vmid_pasid_mapping_pasid(struct kgd_dev *kgd,
uint8_t vmid);
-static uint16_t get_fw_version(struct kgd_dev *kgd, enum kgd_engine_type type);
static void set_scratch_backing_va(struct kgd_dev *kgd,
uint64_t va, uint32_t vmid);
static void set_vm_context_page_table_base(struct kgd_dev *kgd, uint32_t vmid,
.address_watch_get_offset = kgd_address_watch_get_offset,
.get_atc_vmid_pasid_mapping_pasid = get_atc_vmid_pasid_mapping_pasid,
.get_atc_vmid_pasid_mapping_valid = get_atc_vmid_pasid_mapping_valid,
- .get_fw_version = get_fw_version,
.set_scratch_backing_va = set_scratch_backing_va,
.get_tile_config = get_tile_config,
.set_vm_context_page_table_base = set_vm_context_page_table_base,
unlock_srbm(kgd);
}
-static uint16_t get_fw_version(struct kgd_dev *kgd, enum kgd_engine_type type)
-{
- struct amdgpu_device *adev = (struct amdgpu_device *) kgd;
- const union amdgpu_firmware_header *hdr;
-
- switch (type) {
- case KGD_ENGINE_PFP:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.pfp_fw->data;
- break;
-
- case KGD_ENGINE_ME:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.me_fw->data;
- break;
-
- case KGD_ENGINE_CE:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.ce_fw->data;
- break;
-
- case KGD_ENGINE_MEC1:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.mec_fw->data;
- break;
-
- case KGD_ENGINE_MEC2:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.mec2_fw->data;
- break;
-
- case KGD_ENGINE_RLC:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.rlc_fw->data;
- break;
-
- case KGD_ENGINE_SDMA1:
- hdr = (const union amdgpu_firmware_header *)
- adev->sdma.instance[0].fw->data;
- break;
-
- case KGD_ENGINE_SDMA2:
- hdr = (const union amdgpu_firmware_header *)
- adev->sdma.instance[1].fw->data;
- break;
-
- default:
- return 0;
- }
-
- if (hdr == NULL)
- return 0;
-
- /* Only 12 bit in use*/
- return hdr->common.ucode_version;
-}
-
static void set_vm_context_page_table_base(struct kgd_dev *kgd, uint32_t vmid,
uint64_t page_table_base)
{
#include <linux/module.h>
#include <linux/fdtable.h>
#include <linux/uaccess.h>
-#include <linux/firmware.h>
#include <linux/mmu_context.h>
#include <drm/drmP.h>
#include "amdgpu.h"
#include "amdgpu_amdkfd.h"
-#include "amdgpu_ucode.h"
#include "gfx_v8_0.h"
#include "gca/gfx_8_0_sh_mask.h"
#include "gca/gfx_8_0_d.h"
uint8_t vmid);
static uint16_t get_atc_vmid_pasid_mapping_pasid(struct kgd_dev *kgd,
uint8_t vmid);
-static uint16_t get_fw_version(struct kgd_dev *kgd, enum kgd_engine_type type);
static void set_scratch_backing_va(struct kgd_dev *kgd,
uint64_t va, uint32_t vmid);
static void set_vm_context_page_table_base(struct kgd_dev *kgd, uint32_t vmid,
get_atc_vmid_pasid_mapping_pasid,
.get_atc_vmid_pasid_mapping_valid =
get_atc_vmid_pasid_mapping_valid,
- .get_fw_version = get_fw_version,
.set_scratch_backing_va = set_scratch_backing_va,
.get_tile_config = get_tile_config,
.set_vm_context_page_table_base = set_vm_context_page_table_base,
unlock_srbm(kgd);
}
-static uint16_t get_fw_version(struct kgd_dev *kgd, enum kgd_engine_type type)
-{
- struct amdgpu_device *adev = (struct amdgpu_device *) kgd;
- const union amdgpu_firmware_header *hdr;
-
- switch (type) {
- case KGD_ENGINE_PFP:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.pfp_fw->data;
- break;
-
- case KGD_ENGINE_ME:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.me_fw->data;
- break;
-
- case KGD_ENGINE_CE:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.ce_fw->data;
- break;
-
- case KGD_ENGINE_MEC1:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.mec_fw->data;
- break;
-
- case KGD_ENGINE_MEC2:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.mec2_fw->data;
- break;
-
- case KGD_ENGINE_RLC:
- hdr = (const union amdgpu_firmware_header *)
- adev->gfx.rlc_fw->data;
- break;
-
- case KGD_ENGINE_SDMA1:
- hdr = (const union amdgpu_firmware_header *)
- adev->sdma.instance[0].fw->data;
- break;
-
- case KGD_ENGINE_SDMA2:
- hdr = (const union amdgpu_firmware_header *)
- adev->sdma.instance[1].fw->data;
- break;
-
- default:
- return 0;
- }
-
- if (hdr == NULL)
- return 0;
-
- /* Only 12 bit in use*/
- return hdr->common.ucode_version;
-}
-
static void set_vm_context_page_table_base(struct kgd_dev *kgd, uint32_t vmid,
uint64_t page_table_base)
{
#include <linux/module.h>
#include <linux/fdtable.h>
#include <linux/uaccess.h>
-#include <linux/firmware.h>
#include <linux/mmu_context.h>
#include <drm/drmP.h>
#include "amdgpu.h"
#include "amdgpu_amdkfd.h"
-#include "amdgpu_ucode.h"
#include "soc15_hw_ip.h"
#include "gc/gc_9_0_offset.h"
#include "gc/gc_9_0_sh_mask.h"
uint8_t vmid);
static void set_vm_context_page_table_base(struct kgd_dev *kgd, uint32_t vmid,
uint64_t page_table_base);
-static uint16_t get_fw_version(struct kgd_dev *kgd, enum kgd_engine_type type);
static void set_scratch_backing_va(struct kgd_dev *kgd,
uint64_t va, uint32_t vmid);
static int invalidate_tlbs(struct kgd_dev *kgd, uint16_t pasid);
get_atc_vmid_pasid_mapping_pasid,
.get_atc_vmid_pasid_mapping_valid =
get_atc_vmid_pasid_mapping_valid,
- .get_fw_version = get_fw_version,
.set_scratch_backing_va = set_scratch_backing_va,
.get_tile_config = amdgpu_amdkfd_get_tile_config,
.set_vm_context_page_table_base = set_vm_context_page_table_base,
*/
}
-/* FIXME: Does this need to be ASIC-specific code? */
-static uint16_t get_fw_version(struct kgd_dev *kgd, enum kgd_engine_type type)
-{
- struct amdgpu_device *adev = (struct amdgpu_device *) kgd;
- const union amdgpu_firmware_header *hdr;
-
- switch (type) {
- case KGD_ENGINE_PFP:
- hdr = (const union amdgpu_firmware_header *)adev->gfx.pfp_fw->data;
- break;
-
- case KGD_ENGINE_ME:
- hdr = (const union amdgpu_firmware_header *)adev->gfx.me_fw->data;
- break;
-
- case KGD_ENGINE_CE:
- hdr = (const union amdgpu_firmware_header *)adev->gfx.ce_fw->data;
- break;
-
- case KGD_ENGINE_MEC1:
- hdr = (const union amdgpu_firmware_header *)adev->gfx.mec_fw->data;
- break;
-
- case KGD_ENGINE_MEC2:
- hdr = (const union amdgpu_firmware_header *)adev->gfx.mec2_fw->data;
- break;
-
- case KGD_ENGINE_RLC:
- hdr = (const union amdgpu_firmware_header *)adev->gfx.rlc_fw->data;
- break;
-
- case KGD_ENGINE_SDMA1:
- hdr = (const union amdgpu_firmware_header *)adev->sdma.instance[0].fw->data;
- break;
-
- case KGD_ENGINE_SDMA2:
- hdr = (const union amdgpu_firmware_header *)adev->sdma.instance[1].fw->data;
- break;
-
- default:
- return 0;
- }
-
- if (hdr == NULL)
- return 0;
-
- /* Only 12 bit in use*/
- return hdr->common.ucode_version;
-}
-
static void set_vm_context_page_table_base(struct kgd_dev *kgd, uint32_t vmid,
uint64_t page_table_base)
{
if (p_bo_va_entry)
*p_bo_va_entry = bo_va_entry;
- /* Allocate new page tables if needed and validate
- * them.
- */
- ret = amdgpu_vm_alloc_pts(adev, vm, va, amdgpu_bo_size(bo));
- if (ret) {
- pr_err("Failed to allocate pts, err=%d\n", ret);
- goto err_alloc_pts;
- }
-
+ /* Allocate validate page tables if needed */
ret = vm_validate_pt_pd_bos(vm);
if (ret) {
pr_err("validate_pt_pd_bos() failed\n");
struct amdgpu_sync *sync)
{
int ret;
- struct amdgpu_vm *vm;
- struct amdgpu_bo_va *bo_va;
- struct amdgpu_bo *bo;
-
- bo_va = entry->bo_va;
- vm = bo_va->base.vm;
- bo = bo_va->base.bo;
+ struct amdgpu_bo_va *bo_va = entry->bo_va;
/* Update the page tables */
ret = amdgpu_vm_bo_update(adev, bo_va, false);
pr_err("validate_pt_pd_bos() failed\n");
goto validate_pd_fail;
}
- amdgpu_bo_sync_wait(vm->root.base.bo, AMDGPU_FENCE_OWNER_KFD, false);
+ ret = amdgpu_bo_sync_wait(vm->root.base.bo,
+ AMDGPU_FENCE_OWNER_KFD, false);
if (ret)
goto wait_pd_fail;
amdgpu_bo_fence(vm->root.base.bo,
#include "atom.h"
#include "atombios.h"
-#define get_index_into_master_table(master_table, table_name) (offsetof(struct master_table, table_name) / sizeof(uint16_t))
-
bool amdgpu_atomfirmware_gpu_supports_virtualization(struct amdgpu_device *adev)
{
int index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1,
return 0;
}
+/*
+ * Return true if vbios enabled ecc by default, if umc info table is available
+ * or false if ecc is not enabled or umc info table is not available
+ */
+bool amdgpu_atomfirmware_mem_ecc_supported(struct amdgpu_device *adev)
+{
+ struct amdgpu_mode_info *mode_info = &adev->mode_info;
+ int index;
+ u16 data_offset, size;
+ union umc_info *umc_info;
+ u8 frev, crev;
+ bool ecc_default_enabled = false;
+
+ index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1,
+ umc_info);
+
+ if (amdgpu_atom_parse_data_header(mode_info->atom_context,
+ index, &size, &frev, &crev, &data_offset)) {
+ /* support umc_info 3.1+ */
+ if ((frev == 3 && crev >= 1) || (frev > 3)) {
+ umc_info = (union umc_info *)
+ (mode_info->atom_context->bios + data_offset);
+ ecc_default_enabled =
+ (le32_to_cpu(umc_info->v31.umc_config) &
+ UMC_CONFIG__DEFAULT_MEM_ECC_ENABLE) ? true : false;
+ }
+ }
+
+ return ecc_default_enabled;
+}
+
union firmware_info {
struct atom_firmware_info_v3_1 v31;
};
+/*
+ * Return true if vbios supports sram ecc or false if not
+ */
+bool amdgpu_atomfirmware_sram_ecc_supported(struct amdgpu_device *adev)
+{
+ struct amdgpu_mode_info *mode_info = &adev->mode_info;
+ int index;
+ u16 data_offset, size;
+ union firmware_info *firmware_info;
+ u8 frev, crev;
+ bool sram_ecc_supported = false;
+
+ index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1,
+ firmwareinfo);
+
+ if (amdgpu_atom_parse_data_header(adev->mode_info.atom_context,
+ index, &size, &frev, &crev, &data_offset)) {
+ /* support firmware_info 3.1 + */
+ if ((frev == 3 && crev >=1) || (frev > 3)) {
+ firmware_info = (union firmware_info *)
+ (mode_info->atom_context->bios + data_offset);
+ sram_ecc_supported =
+ (le32_to_cpu(firmware_info->v31.firmware_capability) &
+ ATOM_FIRMWARE_CAP_SRAM_ECC) ? true : false;
+ }
+ }
+
+ return sram_ecc_supported;
+}
+
union smu_info {
struct atom_smu_info_v3_1 v31;
};
(mode_info->atom_context->bios + data_offset);
switch (crev) {
case 4:
- adev->gfx.config.max_shader_engines = gfx_info->v24.gc_num_se;
- adev->gfx.config.max_cu_per_sh = gfx_info->v24.gc_num_cu_per_sh;
- adev->gfx.config.max_sh_per_se = gfx_info->v24.gc_num_sh_per_se;
- adev->gfx.config.max_backends_per_se = gfx_info->v24.gc_num_rb_per_se;
- adev->gfx.config.max_texture_channel_caches = gfx_info->v24.gc_num_tccs;
+ adev->gfx.config.max_shader_engines = gfx_info->v24.max_shader_engines;
+ adev->gfx.config.max_cu_per_sh = gfx_info->v24.max_cu_per_sh;
+ adev->gfx.config.max_sh_per_se = gfx_info->v24.max_sh_per_se;
+ adev->gfx.config.max_backends_per_se = gfx_info->v24.max_backends_per_se;
+ adev->gfx.config.max_texture_channel_caches = gfx_info->v24.max_texture_channel_caches;
adev->gfx.config.max_gprs = le16_to_cpu(gfx_info->v24.gc_num_gprs);
adev->gfx.config.max_gs_threads = gfx_info->v24.gc_num_max_gs_thds;
adev->gfx.config.gs_vgt_table_depth = gfx_info->v24.gc_gs_table_depth;
#ifndef __AMDGPU_ATOMFIRMWARE_H__
#define __AMDGPU_ATOMFIRMWARE_H__
+#define get_index_into_master_table(master_table, table_name) (offsetof(struct master_table, table_name) / sizeof(uint16_t))
+
bool amdgpu_atomfirmware_gpu_supports_virtualization(struct amdgpu_device *adev);
void amdgpu_atomfirmware_scratch_regs_init(struct amdgpu_device *adev);
int amdgpu_atomfirmware_allocate_fb_scratch(struct amdgpu_device *adev);
int amdgpu_atomfirmware_get_vram_type(struct amdgpu_device *adev);
int amdgpu_atomfirmware_get_clock_info(struct amdgpu_device *adev);
int amdgpu_atomfirmware_get_gfx_info(struct amdgpu_device *adev);
+bool amdgpu_atomfirmware_mem_ecc_supported(struct amdgpu_device *adev);
+bool amdgpu_atomfirmware_sram_ecc_supported(struct amdgpu_device *adev);
#endif
case AMDGPU_CHUNK_ID_SYNCOBJ_IN:
case AMDGPU_CHUNK_ID_SYNCOBJ_OUT:
case AMDGPU_CHUNK_ID_SCHEDULED_DEPENDENCIES:
+ case AMDGPU_CHUNK_ID_SYNCOBJ_TIMELINE_WAIT:
+ case AMDGPU_CHUNK_ID_SYNCOBJ_TIMELINE_SIGNAL:
break;
default:
ttm_eu_backoff_reservation(&parser->ticket,
&parser->validated);
- for (i = 0; i < parser->num_post_dep_syncobjs; i++)
- drm_syncobj_put(parser->post_dep_syncobjs[i]);
- kfree(parser->post_dep_syncobjs);
+ for (i = 0; i < parser->num_post_deps; i++) {
+ drm_syncobj_put(parser->post_deps[i].syncobj);
+ kfree(parser->post_deps[i].chain);
+ }
+ kfree(parser->post_deps);
dma_fence_put(parser->fence);
}
static int amdgpu_syncobj_lookup_and_add_to_sync(struct amdgpu_cs_parser *p,
- uint32_t handle)
+ uint32_t handle, u64 point,
+ u64 flags)
{
- int r;
struct dma_fence *fence;
- r = drm_syncobj_find_fence(p->filp, handle, 0, 0, &fence);
- if (r)
+ int r;
+
+ r = drm_syncobj_find_fence(p->filp, handle, point, flags, &fence);
+ if (r) {
+ DRM_ERROR("syncobj %u failed to find fence @ %llu (%d)!\n",
+ handle, point, r);
return r;
+ }
r = amdgpu_sync_fence(p->adev, &p->job->sync, fence, true);
dma_fence_put(fence);
static int amdgpu_cs_process_syncobj_in_dep(struct amdgpu_cs_parser *p,
struct amdgpu_cs_chunk *chunk)
{
+ struct drm_amdgpu_cs_chunk_sem *deps;
unsigned num_deps;
int i, r;
- struct drm_amdgpu_cs_chunk_sem *deps;
deps = (struct drm_amdgpu_cs_chunk_sem *)chunk->kdata;
num_deps = chunk->length_dw * 4 /
sizeof(struct drm_amdgpu_cs_chunk_sem);
+ for (i = 0; i < num_deps; ++i) {
+ r = amdgpu_syncobj_lookup_and_add_to_sync(p, deps[i].handle,
+ 0, 0);
+ if (r)
+ return r;
+ }
+
+ return 0;
+}
+
+static int amdgpu_cs_process_syncobj_timeline_in_dep(struct amdgpu_cs_parser *p,
+ struct amdgpu_cs_chunk *chunk)
+{
+ struct drm_amdgpu_cs_chunk_syncobj *syncobj_deps;
+ unsigned num_deps;
+ int i, r;
+
+ syncobj_deps = (struct drm_amdgpu_cs_chunk_syncobj *)chunk->kdata;
+ num_deps = chunk->length_dw * 4 /
+ sizeof(struct drm_amdgpu_cs_chunk_syncobj);
for (i = 0; i < num_deps; ++i) {
- r = amdgpu_syncobj_lookup_and_add_to_sync(p, deps[i].handle);
+ r = amdgpu_syncobj_lookup_and_add_to_sync(p,
+ syncobj_deps[i].handle,
+ syncobj_deps[i].point,
+ syncobj_deps[i].flags);
if (r)
return r;
}
+
return 0;
}
static int amdgpu_cs_process_syncobj_out_dep(struct amdgpu_cs_parser *p,
struct amdgpu_cs_chunk *chunk)
{
+ struct drm_amdgpu_cs_chunk_sem *deps;
unsigned num_deps;
int i;
- struct drm_amdgpu_cs_chunk_sem *deps;
+
deps = (struct drm_amdgpu_cs_chunk_sem *)chunk->kdata;
num_deps = chunk->length_dw * 4 /
sizeof(struct drm_amdgpu_cs_chunk_sem);
- p->post_dep_syncobjs = kmalloc_array(num_deps,
- sizeof(struct drm_syncobj *),
- GFP_KERNEL);
- p->num_post_dep_syncobjs = 0;
+ p->post_deps = kmalloc_array(num_deps, sizeof(*p->post_deps),
+ GFP_KERNEL);
+ p->num_post_deps = 0;
- if (!p->post_dep_syncobjs)
+ if (!p->post_deps)
return -ENOMEM;
+
for (i = 0; i < num_deps; ++i) {
- p->post_dep_syncobjs[i] = drm_syncobj_find(p->filp, deps[i].handle);
- if (!p->post_dep_syncobjs[i])
+ p->post_deps[i].syncobj =
+ drm_syncobj_find(p->filp, deps[i].handle);
+ if (!p->post_deps[i].syncobj)
return -EINVAL;
- p->num_post_dep_syncobjs++;
+ p->post_deps[i].chain = NULL;
+ p->post_deps[i].point = 0;
+ p->num_post_deps++;
}
+
+ return 0;
+}
+
+
+static int amdgpu_cs_process_syncobj_timeline_out_dep(struct amdgpu_cs_parser *p,
+ struct amdgpu_cs_chunk
+ *chunk)
+{
+ struct drm_amdgpu_cs_chunk_syncobj *syncobj_deps;
+ unsigned num_deps;
+ int i;
+
+ syncobj_deps = (struct drm_amdgpu_cs_chunk_syncobj *)chunk->kdata;
+ num_deps = chunk->length_dw * 4 /
+ sizeof(struct drm_amdgpu_cs_chunk_syncobj);
+
+ p->post_deps = kmalloc_array(num_deps, sizeof(*p->post_deps),
+ GFP_KERNEL);
+ p->num_post_deps = 0;
+
+ if (!p->post_deps)
+ return -ENOMEM;
+
+ for (i = 0; i < num_deps; ++i) {
+ struct amdgpu_cs_post_dep *dep = &p->post_deps[i];
+
+ dep->chain = NULL;
+ if (syncobj_deps[i].point) {
+ dep->chain = kmalloc(sizeof(*dep->chain), GFP_KERNEL);
+ if (!dep->chain)
+ return -ENOMEM;
+ }
+
+ dep->syncobj = drm_syncobj_find(p->filp,
+ syncobj_deps[i].handle);
+ if (!dep->syncobj) {
+ kfree(dep->chain);
+ return -EINVAL;
+ }
+ dep->point = syncobj_deps[i].point;
+ p->num_post_deps++;
+ }
+
return 0;
}
chunk = &p->chunks[i];
- if (chunk->chunk_id == AMDGPU_CHUNK_ID_DEPENDENCIES ||
- chunk->chunk_id == AMDGPU_CHUNK_ID_SCHEDULED_DEPENDENCIES) {
+ switch (chunk->chunk_id) {
+ case AMDGPU_CHUNK_ID_DEPENDENCIES:
+ case AMDGPU_CHUNK_ID_SCHEDULED_DEPENDENCIES:
r = amdgpu_cs_process_fence_dep(p, chunk);
if (r)
return r;
- } else if (chunk->chunk_id == AMDGPU_CHUNK_ID_SYNCOBJ_IN) {
+ break;
+ case AMDGPU_CHUNK_ID_SYNCOBJ_IN:
r = amdgpu_cs_process_syncobj_in_dep(p, chunk);
if (r)
return r;
- } else if (chunk->chunk_id == AMDGPU_CHUNK_ID_SYNCOBJ_OUT) {
+ break;
+ case AMDGPU_CHUNK_ID_SYNCOBJ_OUT:
r = amdgpu_cs_process_syncobj_out_dep(p, chunk);
if (r)
return r;
+ break;
+ case AMDGPU_CHUNK_ID_SYNCOBJ_TIMELINE_WAIT:
+ r = amdgpu_cs_process_syncobj_timeline_in_dep(p, chunk);
+ if (r)
+ return r;
+ break;
+ case AMDGPU_CHUNK_ID_SYNCOBJ_TIMELINE_SIGNAL:
+ r = amdgpu_cs_process_syncobj_timeline_out_dep(p, chunk);
+ if (r)
+ return r;
+ break;
}
}
{
int i;
- for (i = 0; i < p->num_post_dep_syncobjs; ++i)
- drm_syncobj_replace_fence(p->post_dep_syncobjs[i], p->fence);
+ for (i = 0; i < p->num_post_deps; ++i) {
+ if (p->post_deps[i].chain && p->post_deps[i].point) {
+ drm_syncobj_add_point(p->post_deps[i].syncobj,
+ p->post_deps[i].chain,
+ p->fence, p->post_deps[i].point);
+ p->post_deps[i].chain = NULL;
+ } else {
+ drm_syncobj_replace_fence(p->post_deps[i].syncobj,
+ p->fence);
+ }
+ }
}
static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,
return -ENOMEM;
}
- r = amdgpu_vm_alloc_pts(adev, (*bo_va)->base.vm, csa_addr,
- size);
- if (r) {
- DRM_ERROR("failed to allocate pts for static CSA, err=%d\n", r);
- amdgpu_vm_bo_rmv(adev, *bo_va);
- ttm_eu_backoff_reservation(&ticket, &list);
- return r;
- }
-
r = amdgpu_vm_bo_map(adev, *bo_va, csa_addr, 0, size,
AMDGPU_PTE_READABLE | AMDGPU_PTE_WRITEABLE |
AMDGPU_PTE_EXECUTABLE);
#include <drm/drm_auth.h>
#include "amdgpu.h"
#include "amdgpu_sched.h"
+#include "amdgpu_ras.h"
#define to_amdgpu_ctx_entity(e) \
container_of((e), struct amdgpu_ctx_entity, entity)
{
struct amdgpu_ctx *ctx;
struct amdgpu_ctx_mgr *mgr;
+ uint32_t ras_counter;
if (!fpriv)
return -EINVAL;
if (atomic_read(&ctx->guilty))
out->state.flags |= AMDGPU_CTX_QUERY2_FLAGS_GUILTY;
+ /*query ue count*/
+ ras_counter = amdgpu_ras_query_error_count(adev, false);
+ /*ras counter is monotonic increasing*/
+ if (ras_counter != ctx->ras_counter_ue) {
+ out->state.flags |= AMDGPU_CTX_QUERY2_FLAGS_RAS_UE;
+ ctx->ras_counter_ue = ras_counter;
+ }
+
+ /*query ce count*/
+ ras_counter = amdgpu_ras_query_error_count(adev, true);
+ if (ras_counter != ctx->ras_counter_ce) {
+ out->state.flags |= AMDGPU_CTX_QUERY2_FLAGS_RAS_CE;
+ ctx->ras_counter_ce = ras_counter;
+ }
+
mutex_unlock(&mgr->lock);
return 0;
}
idr_init(&mgr->ctx_handles);
}
-void amdgpu_ctx_mgr_entity_flush(struct amdgpu_ctx_mgr *mgr)
+long amdgpu_ctx_mgr_entity_flush(struct amdgpu_ctx_mgr *mgr, long timeout)
{
unsigned num_entities = amdgput_ctx_total_num_entities();
struct amdgpu_ctx *ctx;
struct idr *idp;
uint32_t id, i;
- long max_wait = MAX_WAIT_SCHED_ENTITY_Q_EMPTY;
idp = &mgr->ctx_handles;
mutex_lock(&mgr->lock);
idr_for_each_entry(idp, ctx, id) {
-
- if (!ctx->adev) {
- mutex_unlock(&mgr->lock);
- return;
- }
-
for (i = 0; i < num_entities; i++) {
struct drm_sched_entity *entity;
entity = &ctx->entities[0][i].entity;
- max_wait = drm_sched_entity_flush(entity, max_wait);
+ timeout = drm_sched_entity_flush(entity, timeout);
}
}
mutex_unlock(&mgr->lock);
+ return timeout;
}
void amdgpu_ctx_mgr_entity_fini(struct amdgpu_ctx_mgr *mgr)
idp = &mgr->ctx_handles;
idr_for_each_entry(idp, ctx, id) {
-
- if (!ctx->adev)
- return;
-
if (kref_read(&ctx->refcount) != 1) {
DRM_ERROR("ctx %p is still alive\n", ctx);
continue;
enum drm_sched_priority override_priority;
struct mutex lock;
atomic_t guilty;
+ uint32_t ras_counter_ce;
+ uint32_t ras_counter_ue;
};
struct amdgpu_ctx_mgr {
void amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr);
void amdgpu_ctx_mgr_entity_fini(struct amdgpu_ctx_mgr *mgr);
-void amdgpu_ctx_mgr_entity_flush(struct amdgpu_ctx_mgr *mgr);
+long amdgpu_ctx_mgr_entity_flush(struct amdgpu_ctx_mgr *mgr, long timeout);
void amdgpu_ctx_mgr_fini(struct amdgpu_ctx_mgr *mgr);
#endif
idx = *pos >> 2;
valuesize = sizeof(values);
- if (adev->powerplay.pp_funcs && adev->powerplay.pp_funcs->read_sensor)
- r = amdgpu_dpm_read_sensor(adev, idx, &values[0], &valuesize);
- else
- return -EINVAL;
+ r = amdgpu_dpm_read_sensor(adev, idx, &values[0], &valuesize);
+ if (r)
+ return r;
if (size > valuesize)
return -EINVAL;
#include "amdgpu_pm.h"
#include "amdgpu_xgmi.h"
+#include "amdgpu_ras.h"
MODULE_FIRMWARE("amdgpu/vega10_gpu_info.bin");
MODULE_FIRMWARE("amdgpu/vega12_gpu_info.bin");
return -EAGAIN;
}
- adev->powerplay.pp_feature = amdgpu_pp_feature_mask;
+ adev->pm.pp_feature = amdgpu_pp_feature_mask;
+ if (amdgpu_sriov_vf(adev))
+ adev->pm.pp_feature &= ~PP_GFXOFF_MASK;
for (i = 0; i < adev->num_ip_blocks; i++) {
if ((amdgpu_ip_block_mask & (1 << i)) == 0) {
{
int i, r;
+ r = amdgpu_ras_init(adev);
+ if (r)
+ return r;
+
for (i = 0; i < adev->num_ip_blocks; i++) {
if (!adev->ip_blocks[i].status.valid)
continue;
}
}
+ r = amdgpu_ib_pool_init(adev);
+ if (r) {
+ dev_err(adev->dev, "IB initialization failed (%d).\n", r);
+ amdgpu_vf_error_put(adev, AMDGIM_ERROR_VF_IB_INIT_FAIL, 0, r);
+ goto init_failed;
+ }
+
r = amdgpu_ucode_create_bo(adev); /* create ucode bo when sw_init complete*/
if (r)
goto init_failed;
{
int i, r;
+ amdgpu_ras_pre_fini(adev);
+
if (adev->gmc.xgmi.num_physical_nodes > 1)
amdgpu_xgmi_remove_device(adev);
amdgpu_free_static_csa(&adev->virt.csa_obj);
amdgpu_device_wb_fini(adev);
amdgpu_device_vram_scratch_fini(adev);
+ amdgpu_ib_pool_fini(adev);
}
r = adev->ip_blocks[i].version->funcs->sw_fini((void *)adev);
adev->ip_blocks[i].status.late_initialized = false;
}
+ amdgpu_ras_fini(adev);
+
if (amdgpu_sriov_vf(adev))
if (amdgpu_virt_release_full_gpu(adev, false))
DRM_ERROR("failed to release exclusive mode on fini\n");
r = amdgpu_device_enable_mgpu_fan_boost();
if (r)
DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
+
+ /*set to low pstate by default */
+ amdgpu_xgmi_set_pstate(adev, 0);
+
}
static void amdgpu_device_delay_enable_gfx_off(struct work_struct *work)
adev->asic_reset_res = amdgpu_asic_reset(adev);
if (adev->asic_reset_res)
- DRM_WARN("ASIC reset failed with err r, %d for drm dev, %s",
+ DRM_WARN("ASIC reset failed with error, %d for drm dev, %s",
adev->asic_reset_res, adev->ddev->unique);
}
mutex_init(&adev->virt.vf_errors.lock);
hash_init(adev->mn_hash);
mutex_init(&adev->lock_reset);
+ mutex_init(&adev->virt.dpm_mutex);
amdgpu_device_check_arguments(adev);
/* Get a log2 for easy divisions. */
adev->mm_stats.log2_max_MBps = ilog2(max(1u, max_MBps));
- r = amdgpu_ib_pool_init(adev);
- if (r) {
- dev_err(adev->dev, "IB initialization failed (%d).\n", r);
- amdgpu_vf_error_put(adev, AMDGIM_ERROR_VF_IB_INIT_FAIL, 0, r);
- goto failed;
- }
-
amdgpu_fbdev_init(adev);
r = amdgpu_pm_sysfs_init(adev);
goto failed;
}
+ /* must succeed. */
+ amdgpu_ras_post_init(adev);
+
return 0;
failed:
else
drm_atomic_helper_shutdown(adev->ddev);
}
- amdgpu_ib_pool_fini(adev);
amdgpu_fence_driver_fini(adev);
amdgpu_pm_sysfs_fini(adev);
amdgpu_fbdev_fini(adev);
if (r)
return r;
+ amdgpu_amdkfd_pre_reset(adev);
+
/* Resume IP prior to SMC */
r = amdgpu_device_ip_reinit_early_sriov(adev);
if (r)
amdgpu_irq_gpu_reset_resume_helper(adev);
r = amdgpu_ib_ring_tests(adev);
+ amdgpu_amdkfd_post_reset(adev);
error:
amdgpu_virt_init_data_exchange(adev);
r = amdgpu_asic_reset(tmp_adev);
if (r) {
- DRM_ERROR("ASIC reset failed with err r, %d for drm dev, %s",
+ DRM_ERROR("ASIC reset failed with error, %d for drm dev, %s",
r, tmp_adev->ddev->unique);
break;
}
break;
}
}
+
+ list_for_each_entry(tmp_adev, device_list_handle,
+ gmc.xgmi.head) {
+ amdgpu_ras_reserve_bad_pages(tmp_adev);
+ }
}
}
vram_lost = amdgpu_device_check_vram_lost(tmp_adev);
if (vram_lost) {
- DRM_ERROR("VRAM is lost!\n");
+ DRM_INFO("VRAM is lost due to GPU reset!\n");
atomic_inc(&tmp_adev->vram_lost_counter);
}
return NULL;
}
+
+int amdgpu_dpm_get_sclk(struct amdgpu_device *adev, bool low)
+{
+ if (is_support_sw_smu(adev))
+ return smu_get_sclk(&adev->smu, low);
+ else
+ return (adev)->powerplay.pp_funcs->get_sclk((adev)->powerplay.pp_handle, (low));
+}
+
+int amdgpu_dpm_get_mclk(struct amdgpu_device *adev, bool low)
+{
+ if (is_support_sw_smu(adev))
+ return smu_get_mclk(&adev->smu, low);
+ else
+ return (adev)->powerplay.pp_funcs->get_mclk((adev)->powerplay.pp_handle, (low));
+}
#define amdgpu_dpm_enable_bapm(adev, e) \
((adev)->powerplay.pp_funcs->enable_bapm((adev)->powerplay.pp_handle, (e)))
-#define amdgpu_dpm_read_sensor(adev, idx, value, size) \
- ((adev)->powerplay.pp_funcs->read_sensor((adev)->powerplay.pp_handle, (idx), (value), (size)))
-
#define amdgpu_dpm_set_fan_control_mode(adev, m) \
((adev)->powerplay.pp_funcs->set_fan_control_mode((adev)->powerplay.pp_handle, (m)))
#define amdgpu_dpm_set_fan_speed_rpm(adev, s) \
((adev)->powerplay.pp_funcs->set_fan_speed_rpm)((adev)->powerplay.pp_handle, (s))
-#define amdgpu_dpm_get_sclk(adev, l) \
- ((adev)->powerplay.pp_funcs->get_sclk((adev)->powerplay.pp_handle, (l)))
-
-#define amdgpu_dpm_get_mclk(adev, l) \
- ((adev)->powerplay.pp_funcs->get_mclk((adev)->powerplay.pp_handle, (l)))
-
#define amdgpu_dpm_force_performance_level(adev, l) \
((adev)->powerplay.pp_funcs->force_performance_level((adev)->powerplay.pp_handle, (l)))
#define amdgpu_dpm_get_current_power_state(adev) \
((adev)->powerplay.pp_funcs->get_current_power_state((adev)->powerplay.pp_handle))
+#define amdgpu_smu_get_current_power_state(adev) \
+ ((adev)->smu.ppt_funcs->get_current_power_state(&((adev)->smu)))
+
+#define amdgpu_smu_set_power_state(adev) \
+ ((adev)->smu.ppt_funcs->set_power_state(&((adev)->smu)))
+
#define amdgpu_dpm_get_pp_num_states(adev, data) \
((adev)->powerplay.pp_funcs->get_pp_num_states((adev)->powerplay.pp_handle, data))
uint32_t smu_prv_buffer_size;
struct amdgpu_bo *smu_prv_buffer;
bool ac_power;
+ /* powerplay feature */
+ uint32_t pp_feature;
+
};
#define R600_SSTU_DFLT 0
u32 amdgpu_dpm_get_vblank_time(struct amdgpu_device *adev);
u32 amdgpu_dpm_get_vrefresh(struct amdgpu_device *adev);
void amdgpu_dpm_get_active_displays(struct amdgpu_device *adev);
+int amdgpu_dpm_read_sensor(struct amdgpu_device *adev, enum amd_pp_sensors sensor,
+ void *data, uint32_t *size);
bool amdgpu_is_internal_thermal_sensor(enum amdgpu_int_thermal_type sensor);
struct amd_vce_state*
amdgpu_get_vce_clock_state(void *handle, u32 idx);
+extern int amdgpu_dpm_get_sclk(struct amdgpu_device *adev, bool low);
+
+extern int amdgpu_dpm_get_mclk(struct amdgpu_device *adev, bool low);
+
#endif
* - 3.28.0 - Add AMDGPU_CHUNK_ID_SCHEDULED_DEPENDENCIES
* - 3.29.0 - Add AMDGPU_IB_FLAG_RESET_GDS_MAX_WAVE_ID
* - 3.30.0 - Add AMDGPU_SCHED_OP_CONTEXT_PRIORITY_OVERRIDE.
+ * - 3.31.0 - Add support for per-flip tiling attribute changes with DC
+ * - 3.32.0 - Add syncobj timeline support to AMDGPU_CS.
*/
#define KMS_DRIVER_MAJOR 3
-#define KMS_DRIVER_MINOR 30
+#define KMS_DRIVER_MINOR 32
#define KMS_DRIVER_PATCHLEVEL 0
int amdgpu_vram_limit = 0;
uint amdgpu_sdma_phase_quantum = 32;
char *amdgpu_disable_cu = NULL;
char *amdgpu_virtual_display = NULL;
-/* OverDrive(bit 14),gfxoff(bit 15),stutter mode(bit 17) disabled by default*/
-uint amdgpu_pp_feature_mask = 0xfffd3fff;
+/* OverDrive(bit 14) disabled by default*/
+uint amdgpu_pp_feature_mask = 0xffffbfff;
int amdgpu_ngg = 0;
int amdgpu_prim_buf_per_se = 0;
int amdgpu_pos_buf_per_se = 0;
struct amdgpu_mgpu_info mgpu_info = {
.mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
};
+int amdgpu_ras_enable = -1;
+uint amdgpu_ras_mask = 0xffffffff;
/**
* DOC: vramlimit (int)
MODULE_PARM_DESC(emu_mode, "Emulation mode, (1 = enable, 0 = disable)");
module_param_named(emu_mode, amdgpu_emu_mode, int, 0444);
+/**
+ * DOC: ras_enable (int)
+ * Enable RAS features on the GPU (0 = disable, 1 = enable, -1 = auto (default))
+ */
+MODULE_PARM_DESC(ras_enable, "Enable RAS features on the GPU (0 = disable, 1 = enable, -1 = auto (default))");
+module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);
+
+/**
+ * DOC: ras_mask (uint)
+ * Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1
+ * See the flags in drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h
+ */
+MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1");
+module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
+
/**
* DOC: si_support (int)
* Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
DRM_ERROR("Device removal is currently not supported outside of fbcon\n");
drm_dev_unplug(dev);
+ drm_dev_put(dev);
pci_disable_device(pdev);
pci_set_drvdata(pdev, NULL);
}
{
struct drm_file *file_priv = f->private_data;
struct amdgpu_fpriv *fpriv = file_priv->driver_priv;
+ long timeout = MAX_WAIT_SCHED_ENTITY_Q_EMPTY;
- amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr);
+ timeout = amdgpu_ctx_mgr_entity_flush(&fpriv->ctx_mgr, timeout);
+ timeout = amdgpu_vm_wait_idle(&fpriv->vm, timeout);
- return 0;
+ return timeout >= 0 ? 0 : timeout;
}
-
static const struct file_operations amdgpu_driver_kms_fops = {
.owner = THIS_MODULE,
.open = drm_open,
static int
amdgpufb_open(struct fb_info *info, int user)
{
- struct amdgpu_fbdev *rfbdev = info->par;
- struct amdgpu_device *adev = rfbdev->adev;
- int ret = pm_runtime_get_sync(adev->ddev->dev);
+ struct drm_fb_helper *fb_helper = info->par;
+ int ret = pm_runtime_get_sync(fb_helper->dev->dev);
if (ret < 0 && ret != -EACCES) {
- pm_runtime_mark_last_busy(adev->ddev->dev);
- pm_runtime_put_autosuspend(adev->ddev->dev);
+ pm_runtime_mark_last_busy(fb_helper->dev->dev);
+ pm_runtime_put_autosuspend(fb_helper->dev->dev);
return ret;
}
return 0;
static int
amdgpufb_release(struct fb_info *info, int user)
{
- struct amdgpu_fbdev *rfbdev = info->par;
- struct amdgpu_device *adev = rfbdev->adev;
+ struct drm_fb_helper *fb_helper = info->par;
- pm_runtime_mark_last_busy(adev->ddev->dev);
- pm_runtime_put_autosuspend(adev->ddev->dev);
+ pm_runtime_mark_last_busy(fb_helper->dev->dev);
+ pm_runtime_put_autosuspend(fb_helper->dev->dev);
return 0;
}
goto out;
}
- info->par = rfbdev;
- info->skip_vt_switch = true;
-
ret = amdgpu_display_framebuffer_init(adev->ddev, &rfbdev->rfb,
&mode_cmd, gobj);
if (ret) {
/* setup helper */
rfbdev->helper.fb = fb;
- strcpy(info->fix.id, "amdgpudrmfb");
-
- drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
-
info->fbops = &amdgpufb_ops;
tmp = amdgpu_bo_gpu_offset(abo) - adev->gmc.vram_start;
info->screen_base = amdgpu_bo_kptr(abo);
info->screen_size = amdgpu_bo_size(abo);
- drm_fb_helper_fill_var(info, &rfbdev->helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(info, &rfbdev->helper, sizes);
/* setup aperture base/size for vesafb takeover */
info->apertures->ranges[0].base = adev->ddev->mode_config.fb_base;
{
struct amdgpu_device *adev = ring->adev;
struct amdgpu_fence *fence;
- struct dma_fence *old, **ptr;
+ struct dma_fence __rcu **ptr;
uint32_t seq;
+ int r;
fence = kmem_cache_alloc(amdgpu_fence_slab, GFP_KERNEL);
if (fence == NULL)
seq, flags | AMDGPU_FENCE_FLAG_INT);
ptr = &ring->fence_drv.fences[seq & ring->fence_drv.num_fences_mask];
+ if (unlikely(rcu_dereference_protected(*ptr, 1))) {
+ struct dma_fence *old;
+
+ rcu_read_lock();
+ old = dma_fence_get_rcu_safe(ptr);
+ rcu_read_unlock();
+
+ if (old) {
+ r = dma_fence_wait(old, false);
+ dma_fence_put(old);
+ if (r)
+ return r;
+ }
+ }
+
/* This function can't be called concurrently anyway, otherwise
* emitting the fence would mess up the hardware ring buffer.
*/
- old = rcu_dereference_protected(*ptr, 1);
- if (old && !dma_fence_is_signaled(old)) {
- DRM_INFO("rcu slot is busy\n");
- dma_fence_wait(old, false);
- }
-
rcu_assign_pointer(*ptr, dma_fence_get(&fence->base));
*f = &fence->base;
#include <drm/amdgpu_drm.h>
#include "amdgpu.h"
#include "amdgpu_display.h"
+#include "amdgpu_xgmi.h"
void amdgpu_gem_object_free(struct drm_gem_object *gobj)
{
switch (args->operation) {
case AMDGPU_VA_OP_MAP:
- r = amdgpu_vm_alloc_pts(adev, bo_va->base.vm, args->va_address,
- args->map_size);
- if (r)
- goto error_backoff;
-
va_flags = amdgpu_gmc_get_pte_flags(adev, args->flags);
r = amdgpu_vm_bo_map(adev, bo_va, args->va_address,
args->offset_in_bo, args->map_size,
args->map_size);
break;
case AMDGPU_VA_OP_REPLACE:
- r = amdgpu_vm_alloc_pts(adev, bo_va->base.vm, args->va_address,
- args->map_size);
- if (r)
- goto error_backoff;
-
va_flags = amdgpu_gmc_get_pte_flags(adev, args->flags);
r = amdgpu_vm_bo_replace_map(adev, bo_va, args->va_address,
args->offset_in_bo, args->map_size,
struct amdgpu_device *adev = dev->dev_private;
struct drm_amdgpu_gem_op *args = data;
struct drm_gem_object *gobj;
+ struct amdgpu_vm_bo_base *base;
struct amdgpu_bo *robj;
int r;
amdgpu_bo_unreserve(robj);
break;
}
+ for (base = robj->vm_bo; base; base = base->next)
+ if (amdgpu_xgmi_same_hive(amdgpu_ttm_adev(robj->tbo.bdev),
+ amdgpu_ttm_adev(base->vm->root.base.bo->tbo.bdev))) {
+ r = -EINVAL;
+ amdgpu_bo_unreserve(robj);
+ goto out;
+ }
+
+
robj->preferred_domains = args->value & (AMDGPU_GEM_DOMAIN_VRAM |
AMDGPU_GEM_DOMAIN_GTT |
AMDGPU_GEM_DOMAIN_CPU);
struct amdgpu_device *adev = dev->dev_private;
struct drm_gem_object *gobj;
uint32_t handle;
+ u64 flags = AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
u32 domain;
int r;
+ /*
+ * The buffer returned from this function should be cleared, but
+ * it can only be done if the ring is enabled or we'll fail to
+ * create the buffer.
+ */
+ if (adev->mman.buffer_funcs_enabled)
+ flags |= AMDGPU_GEM_CREATE_VRAM_CLEARED;
+
args->pitch = amdgpu_align_pitch(adev, args->width,
DIV_ROUND_UP(args->bpp, 8), 0);
args->size = (u64)args->pitch * args->height;
args->size = ALIGN(args->size, PAGE_SIZE);
domain = amdgpu_bo_get_preferred_pin_domain(adev,
amdgpu_display_supported_domains(adev));
- r = amdgpu_gem_object_create(adev, args->size, 0, domain,
- AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED,
+ r = amdgpu_gem_object_create(adev, args->size, 0, domain, flags,
ttm_bo_type_device, NULL, &gobj);
if (r)
return -ENOMEM;
void amdgpu_gfx_off_ctrl(struct amdgpu_device *adev, bool enable)
{
- if (!(adev->powerplay.pp_feature & PP_GFXOFF_MASK))
+ if (!(adev->pm.pp_feature & PP_GFXOFF_MASK))
return;
if (!adev->powerplay.pp_funcs || !adev->powerplay.pp_funcs->set_powergating_by_smu)
/* pipe reservation */
struct mutex pipe_reserve_mutex;
DECLARE_BITMAP (pipe_reserve_bitmap, AMDGPU_MAX_COMPUTE_QUEUES);
+
+ /*ras */
+ struct ras_common_if *ras_if;
};
#define amdgpu_gfx_get_gpu_clock_counter(adev) (adev)->gfx.funcs->get_gpu_clock_counter((adev))
return pd_addr;
}
+/**
+ * amdgpu_gmc_set_pte_pde - update the page tables using CPU
+ *
+ * @adev: amdgpu_device pointer
+ * @cpu_pt_addr: cpu address of the page table
+ * @gpu_page_idx: entry in the page table to update
+ * @addr: dst addr to write into pte/pde
+ * @flags: access flags
+ *
+ * Update the page tables using CPU.
+ */
+int amdgpu_gmc_set_pte_pde(struct amdgpu_device *adev, void *cpu_pt_addr,
+ uint32_t gpu_page_idx, uint64_t addr,
+ uint64_t flags)
+{
+ void __iomem *ptr = (void *)cpu_pt_addr;
+ uint64_t value;
+
+ /*
+ * The following is for PTE only. GART does not have PDEs.
+ */
+ value = addr & 0x0000FFFFFFFFF000ULL;
+ value |= flags;
+ writeq(value, ptr + (gpu_page_idx * 8));
+ return 0;
+}
+
/**
* amdgpu_gmc_agp_addr - return the address in the AGP address space
*
dev_info(adev->dev, "AGP: %lluM 0x%016llX - 0x%016llX\n",
mc->agp_size >> 20, mc->agp_start, mc->agp_end);
}
+
+/**
+ * amdgpu_gmc_filter_faults - filter VM faults
+ *
+ * @adev: amdgpu device structure
+ * @addr: address of the VM fault
+ * @pasid: PASID of the process causing the fault
+ * @timestamp: timestamp of the fault
+ *
+ * Returns:
+ * True if the fault was filtered and should not be processed further.
+ * False if the fault is a new one and needs to be handled.
+ */
+bool amdgpu_gmc_filter_faults(struct amdgpu_device *adev, uint64_t addr,
+ uint16_t pasid, uint64_t timestamp)
+{
+ struct amdgpu_gmc *gmc = &adev->gmc;
+
+ uint64_t stamp, key = addr << 4 | pasid;
+ struct amdgpu_gmc_fault *fault;
+ uint32_t hash;
+
+ /* If we don't have space left in the ring buffer return immediately */
+ stamp = max(timestamp, AMDGPU_GMC_FAULT_TIMEOUT + 1) -
+ AMDGPU_GMC_FAULT_TIMEOUT;
+ if (gmc->fault_ring[gmc->last_fault].timestamp >= stamp)
+ return true;
+
+ /* Try to find the fault in the hash */
+ hash = hash_64(key, AMDGPU_GMC_FAULT_HASH_ORDER);
+ fault = &gmc->fault_ring[gmc->fault_hash[hash].idx];
+ while (fault->timestamp >= stamp) {
+ uint64_t tmp;
+
+ if (fault->key == key)
+ return true;
+
+ tmp = fault->timestamp;
+ fault = &gmc->fault_ring[fault->next];
+
+ /* Check if the entry was reused */
+ if (fault->timestamp >= tmp)
+ break;
+ }
+
+ /* Add the fault to the ring */
+ fault = &gmc->fault_ring[gmc->last_fault];
+ fault->key = key;
+ fault->timestamp = timestamp;
+
+ /* And update the hash */
+ fault->next = gmc->fault_hash[hash].idx;
+ gmc->fault_hash[hash].idx = gmc->last_fault++;
+ return false;
+}
*/
#define AMDGPU_GMC_HOLE_MASK 0x0000ffffffffffffULL
+/*
+ * Ring size as power of two for the log of recent faults.
+ */
+#define AMDGPU_GMC_FAULT_RING_ORDER 8
+#define AMDGPU_GMC_FAULT_RING_SIZE (1 << AMDGPU_GMC_FAULT_RING_ORDER)
+
+/*
+ * Hash size as power of two for the log of recent faults
+ */
+#define AMDGPU_GMC_FAULT_HASH_ORDER 8
+#define AMDGPU_GMC_FAULT_HASH_SIZE (1 << AMDGPU_GMC_FAULT_HASH_ORDER)
+
+/*
+ * Number of IH timestamp ticks until a fault is considered handled
+ */
+#define AMDGPU_GMC_FAULT_TIMEOUT 5000ULL
+
struct firmware;
+/*
+ * GMC page fault information
+ */
+struct amdgpu_gmc_fault {
+ uint64_t timestamp;
+ uint64_t next:AMDGPU_GMC_FAULT_RING_ORDER;
+ uint64_t key:52;
+};
+
/*
* VMHUB structures, functions & helpers
*/
/* Change the VMID -> PASID mapping */
void (*emit_pasid_mapping)(struct amdgpu_ring *ring, unsigned vmid,
unsigned pasid);
- /* write pte/pde updates using the cpu */
- int (*set_pte_pde)(struct amdgpu_device *adev,
- void *cpu_pt_addr, /* cpu addr of page table */
- uint32_t gpu_page_idx, /* pte/pde to update */
- uint64_t addr, /* addr to write into pte/pde */
- uint64_t flags); /* access flags */
/* enable/disable PRT support */
void (*set_prt)(struct amdgpu_device *adev, bool enable);
/* set pte flags based per asic */
struct kfd_vm_fault_info *vm_fault_info;
atomic_t vm_fault_info_updated;
+ struct amdgpu_gmc_fault fault_ring[AMDGPU_GMC_FAULT_RING_SIZE];
+ struct {
+ uint64_t idx:AMDGPU_GMC_FAULT_RING_ORDER;
+ } fault_hash[AMDGPU_GMC_FAULT_HASH_SIZE];
+ uint64_t last_fault:AMDGPU_GMC_FAULT_RING_ORDER;
+
const struct amdgpu_gmc_funcs *gmc_funcs;
struct amdgpu_xgmi xgmi;
+ struct amdgpu_irq_src ecc_irq;
+ struct ras_common_if *ras_if;
};
#define amdgpu_gmc_flush_gpu_tlb(adev, vmid, type) (adev)->gmc.gmc_funcs->flush_gpu_tlb((adev), (vmid), (type))
#define amdgpu_gmc_emit_flush_gpu_tlb(r, vmid, addr) (r)->adev->gmc.gmc_funcs->emit_flush_gpu_tlb((r), (vmid), (addr))
#define amdgpu_gmc_emit_pasid_mapping(r, vmid, pasid) (r)->adev->gmc.gmc_funcs->emit_pasid_mapping((r), (vmid), (pasid))
-#define amdgpu_gmc_set_pte_pde(adev, pt, idx, addr, flags) (adev)->gmc.gmc_funcs->set_pte_pde((adev), (pt), (idx), (addr), (flags))
#define amdgpu_gmc_get_vm_pde(adev, level, dst, flags) (adev)->gmc.gmc_funcs->get_vm_pde((adev), (level), (dst), (flags))
#define amdgpu_gmc_get_pte_flags(adev, flags) (adev)->gmc.gmc_funcs->get_vm_pte_flags((adev),(flags))
void amdgpu_gmc_get_pde_for_bo(struct amdgpu_bo *bo, int level,
uint64_t *addr, uint64_t *flags);
+int amdgpu_gmc_set_pte_pde(struct amdgpu_device *adev, void *cpu_pt_addr,
+ uint32_t gpu_page_idx, uint64_t addr,
+ uint64_t flags);
uint64_t amdgpu_gmc_pd_addr(struct amdgpu_bo *bo);
uint64_t amdgpu_gmc_agp_addr(struct ttm_buffer_object *bo);
void amdgpu_gmc_vram_location(struct amdgpu_device *adev, struct amdgpu_gmc *mc,
struct amdgpu_gmc *mc);
void amdgpu_gmc_agp_location(struct amdgpu_device *adev,
struct amdgpu_gmc *mc);
+bool amdgpu_gmc_filter_faults(struct amdgpu_device *adev, uint64_t addr,
+ uint16_t pasid, uint64_t timestamp);
#endif
struct ttm_buffer_object *tbo;
};
+/**
+ * DOC: mem_info_gtt_total
+ *
+ * The amdgpu driver provides a sysfs API for reporting current total size of
+ * the GTT.
+ * The file mem_info_gtt_total is used for this, and returns the total size of
+ * the GTT block, in bytes
+ */
+static ssize_t amdgpu_mem_info_gtt_total_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct amdgpu_device *adev = ddev->dev_private;
+
+ return snprintf(buf, PAGE_SIZE, "%llu\n",
+ (adev->mman.bdev.man[TTM_PL_TT].size) * PAGE_SIZE);
+}
+
+/**
+ * DOC: mem_info_gtt_used
+ *
+ * The amdgpu driver provides a sysfs API for reporting current total amount of
+ * used GTT.
+ * The file mem_info_gtt_used is used for this, and returns the current used
+ * size of the GTT block, in bytes
+ */
+static ssize_t amdgpu_mem_info_gtt_used_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct amdgpu_device *adev = ddev->dev_private;
+
+ return snprintf(buf, PAGE_SIZE, "%llu\n",
+ amdgpu_gtt_mgr_usage(&adev->mman.bdev.man[TTM_PL_TT]));
+}
+
+static DEVICE_ATTR(mem_info_gtt_total, S_IRUGO,
+ amdgpu_mem_info_gtt_total_show, NULL);
+static DEVICE_ATTR(mem_info_gtt_used, S_IRUGO,
+ amdgpu_mem_info_gtt_used_show, NULL);
+
/**
* amdgpu_gtt_mgr_init - init GTT manager and DRM MM
*
struct amdgpu_device *adev = amdgpu_ttm_adev(man->bdev);
struct amdgpu_gtt_mgr *mgr;
uint64_t start, size;
+ int ret;
mgr = kzalloc(sizeof(*mgr), GFP_KERNEL);
if (!mgr)
spin_lock_init(&mgr->lock);
atomic64_set(&mgr->available, p_size);
man->priv = mgr;
+
+ ret = device_create_file(adev->dev, &dev_attr_mem_info_gtt_total);
+ if (ret) {
+ DRM_ERROR("Failed to create device file mem_info_gtt_total\n");
+ return ret;
+ }
+ ret = device_create_file(adev->dev, &dev_attr_mem_info_gtt_used);
+ if (ret) {
+ DRM_ERROR("Failed to create device file mem_info_gtt_used\n");
+ return ret;
+ }
+
return 0;
}
*/
static int amdgpu_gtt_mgr_fini(struct ttm_mem_type_manager *man)
{
+ struct amdgpu_device *adev = amdgpu_ttm_adev(man->bdev);
struct amdgpu_gtt_mgr *mgr = man->priv;
spin_lock(&mgr->lock);
drm_mm_takedown(&mgr->mm);
spin_unlock(&mgr->lock);
kfree(mgr);
man->priv = NULL;
+
+ device_remove_file(adev->dev, &dev_attr_mem_info_gtt_total);
+ device_remove_file(adev->dev, &dev_attr_mem_info_gtt_used);
+
return 0;
}
*/
int amdgpu_ih_process(struct amdgpu_device *adev, struct amdgpu_ih_ring *ih)
{
+ unsigned int count = AMDGPU_IH_MAX_NUM_IVS;
u32 wptr;
if (!ih->enabled || adev->shutdown)
/* Order reading of wptr vs. reading of IH ring data */
rmb();
- while (ih->rptr != wptr) {
+ while (ih->rptr != wptr && --count) {
amdgpu_irq_dispatch(adev, ih);
ih->rptr &= ih->ptr_mask;
}
#ifndef __AMDGPU_IH_H__
#define __AMDGPU_IH_H__
+/* Maximum number of IVs processed at once */
+#define AMDGPU_IH_MAX_NUM_IVS 32
+
struct amdgpu_device;
struct amdgpu_iv_entry;
#include "amdgpu_amdkfd.h"
#include "amdgpu_gem.h"
#include "amdgpu_display.h"
+#include "amdgpu_ras.h"
static void amdgpu_unregister_gpu_instance(struct amdgpu_device *adev)
{
fw_info->ver = adev->pm.fw_version;
fw_info->feature = 0;
break;
+ case AMDGPU_INFO_FW_TA:
+ if (query_fw->index > 1)
+ return -EINVAL;
+ if (query_fw->index == 0) {
+ fw_info->ver = adev->psp.ta_fw_version;
+ fw_info->feature = adev->psp.ta_xgmi_ucode_version;
+ } else {
+ fw_info->ver = adev->psp.ta_fw_version;
+ fw_info->feature = adev->psp.ta_ras_ucode_version;
+ }
+ break;
case AMDGPU_INFO_FW_SDMA:
if (query_fw->index >= adev->sdma.num_instances)
return -EINVAL;
if (adev->pm.dpm_enabled) {
dev_info.max_engine_clock = amdgpu_dpm_get_sclk(adev, false) * 10;
dev_info.max_memory_clock = amdgpu_dpm_get_mclk(adev, false) * 10;
+ } else if (amdgpu_sriov_vf(adev) && amdgim_is_hwperf(adev) &&
+ adev->virt.ops->get_pp_clk) {
+ dev_info.max_engine_clock = amdgpu_virt_get_sclk(adev, false) * 10;
+ dev_info.max_memory_clock = amdgpu_virt_get_mclk(adev, false) * 10;
} else {
dev_info.max_engine_clock = adev->clock.default_sclk * 10;
dev_info.max_memory_clock = adev->clock.default_mclk * 10;
case AMDGPU_INFO_VRAM_LOST_COUNTER:
ui32 = atomic_read(&adev->vram_lost_counter);
return copy_to_user(out, &ui32, min(size, 4u)) ? -EFAULT : 0;
+ case AMDGPU_INFO_RAS_ENABLED_FEATURES: {
+ struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
+ uint64_t ras_mask;
+
+ if (!ras)
+ return -EINVAL;
+ ras_mask = (uint64_t)ras->supported << 32 | ras->features;
+
+ return copy_to_user(out, &ras_mask,
+ min_t(u64, size, sizeof(ras_mask))) ?
+ -EFAULT : 0;
+ }
default:
DRM_DEBUG_KMS("Invalid request %d\n", info->query);
return -EINVAL;
seq_printf(m, "ASD feature version: %u, firmware version: 0x%08x\n",
fw_info.feature, fw_info.ver);
+ query_fw.fw_type = AMDGPU_INFO_FW_TA;
+ for (i = 0; i < 2; i++) {
+ query_fw.index = i;
+ ret = amdgpu_firmware_info(&fw_info, &query_fw, adev);
+ if (ret)
+ continue;
+ seq_printf(m, "TA %s feature version: %u, firmware version: 0x%08x\n",
+ i ? "RAS" : "XGMI", fw_info.feature, fw_info.ver);
+ }
+
/* SMC */
query_fw.fw_type = AMDGPU_INFO_FW_SMC;
ret = amdgpu_firmware_info(&fw_info, &query_fw, adev);
#define to_amdgpu_encoder(x) container_of(x, struct amdgpu_encoder, base)
#define to_amdgpu_framebuffer(x) container_of(x, struct amdgpu_framebuffer, base)
-#define to_dm_plane_state(x) container_of(x, struct dm_plane_state, base);
+#define to_dm_plane_state(x) container_of(x, struct dm_plane_state, base)
#define AMDGPU_MAX_HPD_PINS 6
#define AMDGPU_MAX_CRTCS 6
struct amdgpu_flip_work *pflip_works;
enum amdgpu_flip_status pflip_status;
int deferred_flip_completion;
- u64 last_flip_vblank;
+ u32 last_flip_vblank;
/* pll sharing */
struct amdgpu_atom_ss ss;
bool ss_enabled;
if (bo->gem_base.import_attach)
drm_prime_gem_destroy(&bo->gem_base, bo->tbo.sg);
drm_gem_object_release(&bo->gem_base);
- amdgpu_bo_unref(&bo->parent);
+ /* in case amdgpu_device_recover_vram got NULL of bo->parent */
if (!list_empty(&bo->shadow_list)) {
mutex_lock(&adev->shadow_list_lock);
list_del_init(&bo->shadow_list);
mutex_unlock(&adev->shadow_list_lock);
}
+ amdgpu_bo_unref(&bo->parent);
+
kfree(bo->metadata);
kfree(bo);
}
/* If the mappings are cleared or filled */
bool cleared;
+
+ bool is_xgmi;
};
struct amdgpu_bo {
#include "amdgpu_pm.h"
#include "amdgpu_dpm.h"
#include "amdgpu_display.h"
+#include "amdgpu_smu.h"
#include "atom.h"
#include <linux/power_supply.h>
#include <linux/hwmon.h>
}
}
+int amdgpu_dpm_read_sensor(struct amdgpu_device *adev, enum amd_pp_sensors sensor,
+ void *data, uint32_t *size)
+{
+ int ret = 0;
+
+ if (!data || !size)
+ return -EINVAL;
+
+ if (is_support_sw_smu(adev))
+ ret = smu_read_sensor(&adev->smu, sensor, data, size);
+ else {
+ if (adev->powerplay.pp_funcs && adev->powerplay.pp_funcs->read_sensor)
+ ret = adev->powerplay.pp_funcs->read_sensor((adev)->powerplay.pp_handle,
+ sensor, data, size);
+ else
+ ret = -EINVAL;
+ }
+
+ return ret;
+}
+
/**
* DOC: power_dpm_state
*
struct amdgpu_device *adev = ddev->dev_private;
enum amd_pm_state_type pm;
- if (adev->powerplay.pp_funcs->get_current_power_state)
+ if (is_support_sw_smu(adev) && adev->smu.ppt_funcs->get_current_power_state)
+ pm = amdgpu_smu_get_current_power_state(adev);
+ else if (adev->powerplay.pp_funcs->get_current_power_state)
pm = amdgpu_dpm_get_current_power_state(adev);
else
pm = adev->pm.dpm.user_state;
(ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return snprintf(buf, PAGE_SIZE, "off\n");
- if (adev->powerplay.pp_funcs->get_performance_level)
+ if (is_support_sw_smu(adev))
+ level = smu_get_performance_level(&adev->smu);
+ else if (adev->powerplay.pp_funcs->get_performance_level)
level = amdgpu_dpm_get_performance_level(adev);
else
level = adev->pm.dpm.forced_level;
(ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
- if (adev->powerplay.pp_funcs->get_performance_level)
+ if (is_support_sw_smu(adev))
+ current_level = smu_get_performance_level(&adev->smu);
+ else if (adev->powerplay.pp_funcs->get_performance_level)
current_level = amdgpu_dpm_get_performance_level(adev);
if (strncmp("low", buf, strlen("low")) == 0) {
goto fail;
}
+ if (amdgpu_sriov_vf(adev)) {
+ if (amdgim_is_hwperf(adev) &&
+ adev->virt.ops->force_dpm_level) {
+ mutex_lock(&adev->pm.mutex);
+ adev->virt.ops->force_dpm_level(adev, level);
+ mutex_unlock(&adev->pm.mutex);
+ return count;
+ } else {
+ return -EINVAL;
+ }
+ }
+
if (current_level == level)
return count;
- if (adev->powerplay.pp_funcs->force_performance_level) {
+ if (is_support_sw_smu(adev)) {
+ mutex_lock(&adev->pm.mutex);
+ if (adev->pm.dpm.thermal_active) {
+ count = -EINVAL;
+ mutex_unlock(&adev->pm.mutex);
+ goto fail;
+ }
+ ret = smu_force_performance_level(&adev->smu, level);
+ if (ret)
+ count = -EINVAL;
+ else
+ adev->pm.dpm.forced_level = level;
+ mutex_unlock(&adev->pm.mutex);
+ } else if (adev->powerplay.pp_funcs->force_performance_level) {
mutex_lock(&adev->pm.mutex);
if (adev->pm.dpm.thermal_active) {
count = -EINVAL;
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = ddev->dev_private;
struct pp_states_info data;
- int i, buf_len;
+ int i, buf_len, ret;
- if (adev->powerplay.pp_funcs->get_pp_num_states)
+ if (is_support_sw_smu(adev)) {
+ ret = smu_get_power_num_states(&adev->smu, &data);
+ if (ret)
+ return ret;
+ } else if (adev->powerplay.pp_funcs->get_pp_num_states)
amdgpu_dpm_get_pp_num_states(adev, &data);
buf_len = snprintf(buf, PAGE_SIZE, "states: %d\n", data.nums);
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = ddev->dev_private;
struct pp_states_info data;
+ struct smu_context *smu = &adev->smu;
enum amd_pm_state_type pm = 0;
- int i = 0;
+ int i = 0, ret = 0;
- if (adev->powerplay.pp_funcs->get_current_power_state
+ if (is_support_sw_smu(adev)) {
+ pm = smu_get_current_power_state(smu);
+ ret = smu_get_power_num_states(smu, &data);
+ if (ret)
+ return ret;
+ } else if (adev->powerplay.pp_funcs->get_current_power_state
&& adev->powerplay.pp_funcs->get_pp_num_states) {
pm = amdgpu_dpm_get_current_power_state(adev);
amdgpu_dpm_get_pp_num_states(adev, &data);
+ }
- for (i = 0; i < data.nums; i++) {
- if (pm == data.states[i])
- break;
- }
-
- if (i == data.nums)
- i = -EINVAL;
+ for (i = 0; i < data.nums; i++) {
+ if (pm == data.states[i])
+ break;
}
+ if (i == data.nums)
+ i = -EINVAL;
+
return snprintf(buf, PAGE_SIZE, "%d\n", i);
}
if (strlen(buf) == 1)
adev->pp_force_state_enabled = false;
+ else if (is_support_sw_smu(adev))
+ adev->pp_force_state_enabled = false;
else if (adev->powerplay.pp_funcs->dispatch_tasks &&
adev->powerplay.pp_funcs->get_pp_num_states) {
struct pp_states_info data;
char *table = NULL;
int size;
- if (adev->powerplay.pp_funcs->get_pp_table)
+ if (is_support_sw_smu(adev)) {
+ size = smu_sys_get_pp_table(&adev->smu, (void **)&table);
+ if (size < 0)
+ return size;
+ }
+ else if (adev->powerplay.pp_funcs->get_pp_table)
size = amdgpu_dpm_get_pp_table(adev, &table);
else
return 0;
{
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = ddev->dev_private;
+ int ret = 0;
- if (adev->powerplay.pp_funcs->set_pp_table)
+ if (is_support_sw_smu(adev)) {
+ ret = smu_sys_set_pp_table(&adev->smu, (void *)buf, count);
+ if (ret)
+ return ret;
+ } else if (adev->powerplay.pp_funcs->set_pp_table)
amdgpu_dpm_set_pp_table(adev, buf, count);
return count;
tmp_str++;
}
- if (adev->powerplay.pp_funcs->odn_edit_dpm_table)
- ret = amdgpu_dpm_odn_edit_dpm_table(adev, type,
- parameter, parameter_size);
+ if (is_support_sw_smu(adev)) {
+ ret = smu_od_edit_dpm_table(&adev->smu, type,
+ parameter, parameter_size);
- if (ret)
- return -EINVAL;
+ if (ret)
+ return -EINVAL;
+ } else {
+ if (adev->powerplay.pp_funcs->odn_edit_dpm_table)
+ ret = amdgpu_dpm_odn_edit_dpm_table(adev, type,
+ parameter, parameter_size);
- if (type == PP_OD_COMMIT_DPM_TABLE) {
- if (adev->powerplay.pp_funcs->dispatch_tasks) {
- amdgpu_dpm_dispatch_task(adev, AMD_PP_TASK_READJUST_POWER_STATE, NULL);
- return count;
- } else {
+ if (ret)
return -EINVAL;
+
+ if (type == PP_OD_COMMIT_DPM_TABLE) {
+ if (adev->powerplay.pp_funcs->dispatch_tasks) {
+ amdgpu_dpm_dispatch_task(adev,
+ AMD_PP_TASK_READJUST_POWER_STATE,
+ NULL);
+ return count;
+ } else {
+ return -EINVAL;
+ }
}
}
struct amdgpu_device *adev = ddev->dev_private;
uint32_t size = 0;
- if (adev->powerplay.pp_funcs->print_clock_levels) {
+ if (is_support_sw_smu(adev)) {
+ size = smu_print_clk_levels(&adev->smu, OD_SCLK, buf);
+ size += smu_print_clk_levels(&adev->smu, OD_MCLK, buf+size);
+ size += smu_print_clk_levels(&adev->smu, OD_VDDC_CURVE, buf+size);
+ size += smu_print_clk_levels(&adev->smu, OD_RANGE, buf+size);
+ return size;
+ } else if (adev->powerplay.pp_funcs->print_clock_levels) {
size = amdgpu_dpm_print_clock_levels(adev, OD_SCLK, buf);
size += amdgpu_dpm_print_clock_levels(adev, OD_MCLK, buf+size);
size += amdgpu_dpm_print_clock_levels(adev, OD_VDDC_CURVE, buf+size);
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = ddev->dev_private;
- if (adev->powerplay.pp_funcs->print_clock_levels)
+ if (amdgpu_sriov_vf(adev) && amdgim_is_hwperf(adev) &&
+ adev->virt.ops->get_pp_clk)
+ return adev->virt.ops->get_pp_clk(adev, PP_SCLK, buf);
+
+ if (is_support_sw_smu(adev))
+ return smu_print_clk_levels(&adev->smu, PP_SCLK, buf);
+ else if (adev->powerplay.pp_funcs->print_clock_levels)
return amdgpu_dpm_print_clock_levels(adev, PP_SCLK, buf);
else
return snprintf(buf, PAGE_SIZE, "\n");
if (ret)
return ret;
- if (adev->powerplay.pp_funcs->force_clock_level)
+ if (is_support_sw_smu(adev))
+ ret = smu_force_clk_levels(&adev->smu, PP_SCLK, mask);
+ else if (adev->powerplay.pp_funcs->force_clock_level)
ret = amdgpu_dpm_force_clock_level(adev, PP_SCLK, mask);
if (ret)
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = ddev->dev_private;
- if (adev->powerplay.pp_funcs->print_clock_levels)
+ if (is_support_sw_smu(adev))
+ return smu_print_clk_levels(&adev->smu, PP_MCLK, buf);
+ else if (adev->powerplay.pp_funcs->print_clock_levels)
return amdgpu_dpm_print_clock_levels(adev, PP_MCLK, buf);
else
return snprintf(buf, PAGE_SIZE, "\n");
if (ret)
return ret;
- if (adev->powerplay.pp_funcs->force_clock_level)
+ if (is_support_sw_smu(adev))
+ ret = smu_force_clk_levels(&adev->smu, PP_MCLK, mask);
+ else if (adev->powerplay.pp_funcs->force_clock_level)
ret = amdgpu_dpm_force_clock_level(adev, PP_MCLK, mask);
if (ret)
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = ddev->dev_private;
- if (adev->powerplay.pp_funcs->print_clock_levels)
+ if (is_support_sw_smu(adev))
+ return smu_print_clk_levels(&adev->smu, PP_SOCCLK, buf);
+ else if (adev->powerplay.pp_funcs->print_clock_levels)
return amdgpu_dpm_print_clock_levels(adev, PP_SOCCLK, buf);
else
return snprintf(buf, PAGE_SIZE, "\n");
if (ret)
return ret;
- if (adev->powerplay.pp_funcs->force_clock_level)
+ if (is_support_sw_smu(adev))
+ ret = smu_force_clk_levels(&adev->smu, PP_SOCCLK, mask);
+ else if (adev->powerplay.pp_funcs->force_clock_level)
ret = amdgpu_dpm_force_clock_level(adev, PP_SOCCLK, mask);
if (ret)
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = ddev->dev_private;
- if (adev->powerplay.pp_funcs->print_clock_levels)
+ if (is_support_sw_smu(adev))
+ return smu_print_clk_levels(&adev->smu, PP_FCLK, buf);
+ else if (adev->powerplay.pp_funcs->print_clock_levels)
return amdgpu_dpm_print_clock_levels(adev, PP_FCLK, buf);
else
return snprintf(buf, PAGE_SIZE, "\n");
if (ret)
return ret;
- if (adev->powerplay.pp_funcs->force_clock_level)
+ if (is_support_sw_smu(adev))
+ ret = smu_force_clk_levels(&adev->smu, PP_FCLK, mask);
+ else if (adev->powerplay.pp_funcs->force_clock_level)
ret = amdgpu_dpm_force_clock_level(adev, PP_FCLK, mask);
if (ret)
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = ddev->dev_private;
- if (adev->powerplay.pp_funcs->print_clock_levels)
+ if (is_support_sw_smu(adev))
+ return smu_print_clk_levels(&adev->smu, PP_DCEFCLK, buf);
+ else if (adev->powerplay.pp_funcs->print_clock_levels)
return amdgpu_dpm_print_clock_levels(adev, PP_DCEFCLK, buf);
else
return snprintf(buf, PAGE_SIZE, "\n");
if (ret)
return ret;
- if (adev->powerplay.pp_funcs->force_clock_level)
+ if (is_support_sw_smu(adev))
+ ret = smu_force_clk_levels(&adev->smu, PP_DCEFCLK, mask);
+ else if (adev->powerplay.pp_funcs->force_clock_level)
ret = amdgpu_dpm_force_clock_level(adev, PP_DCEFCLK, mask);
if (ret)
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = ddev->dev_private;
- if (adev->powerplay.pp_funcs->print_clock_levels)
+ if (is_support_sw_smu(adev))
+ return smu_print_clk_levels(&adev->smu, PP_PCIE, buf);
+ else if (adev->powerplay.pp_funcs->print_clock_levels)
return amdgpu_dpm_print_clock_levels(adev, PP_PCIE, buf);
else
return snprintf(buf, PAGE_SIZE, "\n");
if (ret)
return ret;
- if (adev->powerplay.pp_funcs->force_clock_level)
+ if (is_support_sw_smu(adev))
+ ret = smu_force_clk_levels(&adev->smu, PP_PCIE, mask);
+ else if (adev->powerplay.pp_funcs->force_clock_level)
ret = amdgpu_dpm_force_clock_level(adev, PP_PCIE, mask);
if (ret)
struct amdgpu_device *adev = ddev->dev_private;
uint32_t value = 0;
- if (adev->powerplay.pp_funcs->get_sclk_od)
+ if (is_support_sw_smu(adev))
+ value = smu_get_od_percentage(&(adev->smu), OD_SCLK);
+ else if (adev->powerplay.pp_funcs->get_sclk_od)
value = amdgpu_dpm_get_sclk_od(adev);
return snprintf(buf, PAGE_SIZE, "%d\n", value);
count = -EINVAL;
goto fail;
}
- if (adev->powerplay.pp_funcs->set_sclk_od)
- amdgpu_dpm_set_sclk_od(adev, (uint32_t)value);
- if (adev->powerplay.pp_funcs->dispatch_tasks) {
- amdgpu_dpm_dispatch_task(adev, AMD_PP_TASK_READJUST_POWER_STATE, NULL);
+ if (is_support_sw_smu(adev)) {
+ value = smu_set_od_percentage(&(adev->smu), OD_SCLK, (uint32_t)value);
} else {
- adev->pm.dpm.current_ps = adev->pm.dpm.boot_ps;
- amdgpu_pm_compute_clocks(adev);
+ if (adev->powerplay.pp_funcs->set_sclk_od)
+ amdgpu_dpm_set_sclk_od(adev, (uint32_t)value);
+
+ if (adev->powerplay.pp_funcs->dispatch_tasks) {
+ amdgpu_dpm_dispatch_task(adev, AMD_PP_TASK_READJUST_POWER_STATE, NULL);
+ } else {
+ adev->pm.dpm.current_ps = adev->pm.dpm.boot_ps;
+ amdgpu_pm_compute_clocks(adev);
+ }
}
fail:
struct amdgpu_device *adev = ddev->dev_private;
uint32_t value = 0;
- if (adev->powerplay.pp_funcs->get_mclk_od)
+ if (is_support_sw_smu(adev))
+ value = smu_get_od_percentage(&(adev->smu), OD_MCLK);
+ else if (adev->powerplay.pp_funcs->get_mclk_od)
value = amdgpu_dpm_get_mclk_od(adev);
return snprintf(buf, PAGE_SIZE, "%d\n", value);
count = -EINVAL;
goto fail;
}
- if (adev->powerplay.pp_funcs->set_mclk_od)
- amdgpu_dpm_set_mclk_od(adev, (uint32_t)value);
- if (adev->powerplay.pp_funcs->dispatch_tasks) {
- amdgpu_dpm_dispatch_task(adev, AMD_PP_TASK_READJUST_POWER_STATE, NULL);
+ if (is_support_sw_smu(adev)) {
+ value = smu_set_od_percentage(&(adev->smu), OD_MCLK, (uint32_t)value);
} else {
- adev->pm.dpm.current_ps = adev->pm.dpm.boot_ps;
- amdgpu_pm_compute_clocks(adev);
+ if (adev->powerplay.pp_funcs->set_mclk_od)
+ amdgpu_dpm_set_mclk_od(adev, (uint32_t)value);
+
+ if (adev->powerplay.pp_funcs->dispatch_tasks) {
+ amdgpu_dpm_dispatch_task(adev, AMD_PP_TASK_READJUST_POWER_STATE, NULL);
+ } else {
+ adev->pm.dpm.current_ps = adev->pm.dpm.boot_ps;
+ amdgpu_pm_compute_clocks(adev);
+ }
}
fail:
struct drm_device *ddev = dev_get_drvdata(dev);
struct amdgpu_device *adev = ddev->dev_private;
- if (adev->powerplay.pp_funcs->get_power_profile_mode)
+ if (is_support_sw_smu(adev))
+ return smu_get_power_profile_mode(&adev->smu, buf);
+ else if (adev->powerplay.pp_funcs->get_power_profile_mode)
return amdgpu_dpm_get_power_profile_mode(adev, buf);
return snprintf(buf, PAGE_SIZE, "\n");
}
}
parameter[parameter_size] = profile_mode;
- if (adev->powerplay.pp_funcs->set_power_profile_mode)
+ if (is_support_sw_smu(adev))
+ ret = smu_set_power_profile_mode(&adev->smu, parameter, parameter_size);
+ else if (adev->powerplay.pp_funcs->set_power_profile_mode)
ret = amdgpu_dpm_set_power_profile_mode(adev, parameter, parameter_size);
-
if (!ret)
return count;
fail:
struct amdgpu_device *adev = ddev->dev_private;
int r, value, size = sizeof(value);
- /* sanity check PP is enabled */
- if (!(adev->powerplay.pp_funcs &&
- adev->powerplay.pp_funcs->read_sensor))
- return -EINVAL;
-
/* read the IP busy sensor */
r = amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_GPU_LOAD,
(void *)&value, &size);
+
if (r)
return r;
(ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
- /* sanity check PP is enabled */
- if (!(adev->powerplay.pp_funcs &&
- adev->powerplay.pp_funcs->read_sensor))
- return -EINVAL;
-
/* get the temperature */
r = amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_GPU_TEMP,
(void *)&temp, &size);
{
struct amdgpu_device *adev = dev_get_drvdata(dev);
u32 pwm_mode = 0;
+ if (is_support_sw_smu(adev)) {
+ pwm_mode = smu_get_fan_control_mode(&adev->smu);
+ } else {
+ if (!adev->powerplay.pp_funcs->get_fan_control_mode)
+ return -EINVAL;
- if (!adev->powerplay.pp_funcs->get_fan_control_mode)
- return -EINVAL;
-
- pwm_mode = amdgpu_dpm_get_fan_control_mode(adev);
+ pwm_mode = amdgpu_dpm_get_fan_control_mode(adev);
+ }
return sprintf(buf, "%i\n", pwm_mode);
}
(adev->ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
- if (!adev->powerplay.pp_funcs->set_fan_control_mode)
- return -EINVAL;
+ if (is_support_sw_smu(adev)) {
+ err = kstrtoint(buf, 10, &value);
+ if (err)
+ return err;
- err = kstrtoint(buf, 10, &value);
- if (err)
- return err;
+ smu_set_fan_control_mode(&adev->smu, value);
+ } else {
+ if (!adev->powerplay.pp_funcs->set_fan_control_mode)
+ return -EINVAL;
+
+ err = kstrtoint(buf, 10, &value);
+ if (err)
+ return err;
- amdgpu_dpm_set_fan_control_mode(adev, value);
+ amdgpu_dpm_set_fan_control_mode(adev, value);
+ }
return count;
}
if ((adev->flags & AMD_IS_PX) &&
(adev->ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
-
- pwm_mode = amdgpu_dpm_get_fan_control_mode(adev);
+ if (is_support_sw_smu(adev))
+ pwm_mode = smu_get_fan_control_mode(&adev->smu);
+ else
+ pwm_mode = amdgpu_dpm_get_fan_control_mode(adev);
if (pwm_mode != AMD_FAN_CTRL_MANUAL) {
pr_info("manual fan speed control should be enabled first\n");
return -EINVAL;
value = (value * 100) / 255;
- if (adev->powerplay.pp_funcs->set_fan_speed_percent) {
+ if (is_support_sw_smu(adev)) {
+ err = smu_set_fan_speed_percent(&adev->smu, value);
+ if (err)
+ return err;
+ } else if (adev->powerplay.pp_funcs->set_fan_speed_percent) {
err = amdgpu_dpm_set_fan_speed_percent(adev, value);
if (err)
return err;
(adev->ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
- if (adev->powerplay.pp_funcs->get_fan_speed_percent) {
+ if (is_support_sw_smu(adev)) {
+ err = smu_get_fan_speed_percent(&adev->smu, &speed);
+ if (err)
+ return err;
+ } else if (adev->powerplay.pp_funcs->get_fan_speed_percent) {
err = amdgpu_dpm_get_fan_speed_percent(adev, &speed);
if (err)
return err;
(adev->ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
- if (adev->powerplay.pp_funcs->get_fan_speed_rpm) {
+ if (is_support_sw_smu(adev)) {
+ err = smu_get_current_rpm(&adev->smu, &speed);
+ if (err)
+ return err;
+ } else if (adev->powerplay.pp_funcs->get_fan_speed_rpm) {
err = amdgpu_dpm_get_fan_speed_rpm(adev, &speed);
if (err)
return err;
u32 size = sizeof(min_rpm);
int r;
- if (!adev->powerplay.pp_funcs->read_sensor)
- return -EINVAL;
-
r = amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_MIN_FAN_RPM,
(void *)&min_rpm, &size);
if (r)
u32 size = sizeof(max_rpm);
int r;
- if (!adev->powerplay.pp_funcs->read_sensor)
- return -EINVAL;
-
r = amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_MAX_FAN_RPM,
(void *)&max_rpm, &size);
if (r)
(adev->ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
- if (adev->powerplay.pp_funcs->get_fan_speed_rpm) {
+ if (is_support_sw_smu(adev)) {
+ err = smu_get_current_rpm(&adev->smu, &rpm);
+ if (err)
+ return err;
+ } else if (adev->powerplay.pp_funcs->get_fan_speed_rpm) {
err = amdgpu_dpm_get_fan_speed_rpm(adev, &rpm);
if (err)
return err;
u32 value;
u32 pwm_mode;
- pwm_mode = amdgpu_dpm_get_fan_control_mode(adev);
+ if (is_support_sw_smu(adev))
+ pwm_mode = smu_get_fan_control_mode(&adev->smu);
+ else
+ pwm_mode = amdgpu_dpm_get_fan_control_mode(adev);
+
if (pwm_mode != AMD_FAN_CTRL_MANUAL)
return -ENODATA;
if (err)
return err;
- if (adev->powerplay.pp_funcs->set_fan_speed_rpm) {
+ if (is_support_sw_smu(adev)) {
+ err = smu_set_fan_speed_rpm(&adev->smu, value);
+ if (err)
+ return err;
+ } else if (adev->powerplay.pp_funcs->set_fan_speed_rpm) {
err = amdgpu_dpm_set_fan_speed_rpm(adev, value);
if (err)
return err;
struct amdgpu_device *adev = dev_get_drvdata(dev);
u32 pwm_mode = 0;
- if (!adev->powerplay.pp_funcs->get_fan_control_mode)
- return -EINVAL;
-
- pwm_mode = amdgpu_dpm_get_fan_control_mode(adev);
+ if (is_support_sw_smu(adev)) {
+ pwm_mode = smu_get_fan_control_mode(&adev->smu);
+ } else {
+ if (!adev->powerplay.pp_funcs->get_fan_control_mode)
+ return -EINVAL;
+ pwm_mode = amdgpu_dpm_get_fan_control_mode(adev);
+ }
return sprintf(buf, "%i\n", pwm_mode == AMD_FAN_CTRL_AUTO ? 0 : 1);
}
(adev->ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
- if (!adev->powerplay.pp_funcs->set_fan_control_mode)
- return -EINVAL;
err = kstrtoint(buf, 10, &value);
if (err)
else
return -EINVAL;
- amdgpu_dpm_set_fan_control_mode(adev, pwm_mode);
+ if (is_support_sw_smu(adev)) {
+ smu_set_fan_control_mode(&adev->smu, pwm_mode);
+ } else {
+ if (!adev->powerplay.pp_funcs->set_fan_control_mode)
+ return -EINVAL;
+ amdgpu_dpm_set_fan_control_mode(adev, pwm_mode);
+ }
return count;
}
(ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
- /* sanity check PP is enabled */
- if (!(adev->powerplay.pp_funcs &&
- adev->powerplay.pp_funcs->read_sensor))
- return -EINVAL;
-
/* get the voltage */
r = amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_VDDGFX,
(void *)&vddgfx, &size);
(ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
- /* sanity check PP is enabled */
- if (!(adev->powerplay.pp_funcs &&
- adev->powerplay.pp_funcs->read_sensor))
- return -EINVAL;
-
/* get the voltage */
r = amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_VDDNB,
(void *)&vddnb, &size);
(ddev->switch_power_state != DRM_SWITCH_POWER_ON))
return -EINVAL;
- /* sanity check PP is enabled */
- if (!(adev->powerplay.pp_funcs &&
- adev->powerplay.pp_funcs->read_sensor))
- return -EINVAL;
-
/* get the voltage */
r = amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_GPU_POWER,
(void *)&query, &size);
struct amdgpu_device *adev = dev_get_drvdata(dev);
uint32_t limit = 0;
- if (adev->powerplay.pp_funcs && adev->powerplay.pp_funcs->get_power_limit) {
+ if (is_support_sw_smu(adev)) {
+ smu_get_power_limit(&adev->smu, &limit, true);
+ return snprintf(buf, PAGE_SIZE, "%u\n", limit * 1000000);
+ } else if (adev->powerplay.pp_funcs && adev->powerplay.pp_funcs->get_power_limit) {
adev->powerplay.pp_funcs->get_power_limit(adev->powerplay.pp_handle, &limit, true);
return snprintf(buf, PAGE_SIZE, "%u\n", limit * 1000000);
} else {
struct amdgpu_device *adev = dev_get_drvdata(dev);
uint32_t limit = 0;
- if (adev->powerplay.pp_funcs && adev->powerplay.pp_funcs->get_power_limit) {
+ if (is_support_sw_smu(adev)) {
+ smu_get_power_limit(&adev->smu, &limit, false);
+ return snprintf(buf, PAGE_SIZE, "%u\n", limit * 1000000);
+ } else if (adev->powerplay.pp_funcs && adev->powerplay.pp_funcs->get_power_limit) {
adev->powerplay.pp_funcs->get_power_limit(adev->powerplay.pp_handle, &limit, false);
return snprintf(buf, PAGE_SIZE, "%u\n", limit * 1000000);
} else {
return err;
value = value / 1000000; /* convert to Watt */
- if (adev->powerplay.pp_funcs && adev->powerplay.pp_funcs->set_power_limit) {
+ if (is_support_sw_smu(adev)) {
+ adev->smu.funcs->set_power_limit(&adev->smu, value);
+ } else if (adev->powerplay.pp_funcs && adev->powerplay.pp_funcs->set_power_limit) {
err = adev->powerplay.pp_funcs->set_power_limit(adev->powerplay.pp_handle, value);
if (err)
return err;
attr == &sensor_dev_attr_fan1_enable.dev_attr.attr))
return 0;
- /* mask fan attributes if we have no bindings for this asic to expose */
- if ((!adev->powerplay.pp_funcs->get_fan_speed_percent &&
- attr == &sensor_dev_attr_pwm1.dev_attr.attr) || /* can't query fan */
- (!adev->powerplay.pp_funcs->get_fan_control_mode &&
- attr == &sensor_dev_attr_pwm1_enable.dev_attr.attr)) /* can't query state */
- effective_mode &= ~S_IRUGO;
+ if (!is_support_sw_smu(adev)) {
+ /* mask fan attributes if we have no bindings for this asic to expose */
+ if ((!adev->powerplay.pp_funcs->get_fan_speed_percent &&
+ attr == &sensor_dev_attr_pwm1.dev_attr.attr) || /* can't query fan */
+ (!adev->powerplay.pp_funcs->get_fan_control_mode &&
+ attr == &sensor_dev_attr_pwm1_enable.dev_attr.attr)) /* can't query state */
+ effective_mode &= ~S_IRUGO;
- if ((!adev->powerplay.pp_funcs->set_fan_speed_percent &&
- attr == &sensor_dev_attr_pwm1.dev_attr.attr) || /* can't manage fan */
- (!adev->powerplay.pp_funcs->set_fan_control_mode &&
- attr == &sensor_dev_attr_pwm1_enable.dev_attr.attr)) /* can't manage state */
- effective_mode &= ~S_IWUSR;
+ if ((!adev->powerplay.pp_funcs->set_fan_speed_percent &&
+ attr == &sensor_dev_attr_pwm1.dev_attr.attr) || /* can't manage fan */
+ (!adev->powerplay.pp_funcs->set_fan_control_mode &&
+ attr == &sensor_dev_attr_pwm1_enable.dev_attr.attr)) /* can't manage state */
+ effective_mode &= ~S_IWUSR;
+ }
if ((adev->flags & AMD_IS_APU) &&
(attr == &sensor_dev_attr_power1_average.dev_attr.attr ||
attr == &sensor_dev_attr_power1_cap.dev_attr.attr))
return 0;
- /* hide max/min values if we can't both query and manage the fan */
- if ((!adev->powerplay.pp_funcs->set_fan_speed_percent &&
- !adev->powerplay.pp_funcs->get_fan_speed_percent) &&
- (!adev->powerplay.pp_funcs->set_fan_speed_rpm &&
- !adev->powerplay.pp_funcs->get_fan_speed_rpm) &&
- (attr == &sensor_dev_attr_pwm1_max.dev_attr.attr ||
- attr == &sensor_dev_attr_pwm1_min.dev_attr.attr))
- return 0;
+ if (!is_support_sw_smu(adev)) {
+ /* hide max/min values if we can't both query and manage the fan */
+ if ((!adev->powerplay.pp_funcs->set_fan_speed_percent &&
+ !adev->powerplay.pp_funcs->get_fan_speed_percent) &&
+ (!adev->powerplay.pp_funcs->set_fan_speed_rpm &&
+ !adev->powerplay.pp_funcs->get_fan_speed_rpm) &&
+ (attr == &sensor_dev_attr_pwm1_max.dev_attr.attr ||
+ attr == &sensor_dev_attr_pwm1_min.dev_attr.attr))
+ return 0;
- if ((!adev->powerplay.pp_funcs->set_fan_speed_rpm &&
- !adev->powerplay.pp_funcs->get_fan_speed_rpm) &&
- (attr == &sensor_dev_attr_fan1_max.dev_attr.attr ||
- attr == &sensor_dev_attr_fan1_min.dev_attr.attr))
- return 0;
+ if ((!adev->powerplay.pp_funcs->set_fan_speed_rpm &&
+ !adev->powerplay.pp_funcs->get_fan_speed_rpm) &&
+ (attr == &sensor_dev_attr_fan1_max.dev_attr.attr ||
+ attr == &sensor_dev_attr_fan1_min.dev_attr.attr))
+ return 0;
+ }
/* only APUs have vddnb */
if (!(adev->flags & AMD_IS_APU) &&
if (!adev->pm.dpm_enabled)
return;
- if (adev->powerplay.pp_funcs &&
- adev->powerplay.pp_funcs->read_sensor &&
- !amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_GPU_TEMP,
+ if (!amdgpu_dpm_read_sensor(adev, AMDGPU_PP_SENSOR_GPU_TEMP,
(void *)&temp, &size)) {
if (temp < adev->pm.dpm.thermal.min_temp)
/* switch back the user state */
void amdgpu_dpm_enable_uvd(struct amdgpu_device *adev, bool enable)
{
- if (adev->powerplay.pp_funcs->set_powergating_by_smu) {
+ int ret = 0;
+ if (is_support_sw_smu(adev)) {
+ ret = smu_dpm_set_power_gate(&adev->smu, AMD_IP_BLOCK_TYPE_UVD, enable);
+ if (ret)
+ DRM_ERROR("[SW SMU]: dpm enable uvd failed, state = %s, ret = %d. \n",
+ enable ? "true" : "false", ret);
+ } else if (adev->powerplay.pp_funcs->set_powergating_by_smu) {
/* enable/disable UVD */
mutex_lock(&adev->pm.mutex);
amdgpu_dpm_set_powergating_by_smu(adev, AMD_IP_BLOCK_TYPE_UVD, !enable);
void amdgpu_dpm_enable_vce(struct amdgpu_device *adev, bool enable)
{
- if (adev->powerplay.pp_funcs->set_powergating_by_smu) {
+ int ret = 0;
+ if (is_support_sw_smu(adev)) {
+ ret = smu_dpm_set_power_gate(&adev->smu, AMD_IP_BLOCK_TYPE_VCE, enable);
+ if (ret)
+ DRM_ERROR("[SW SMU]: dpm enable vce failed, state = %s, ret = %d. \n",
+ enable ? "true" : "false", ret);
+ } else if (adev->powerplay.pp_funcs->set_powergating_by_smu) {
/* enable/disable VCE */
mutex_lock(&adev->pm.mutex);
amdgpu_dpm_set_powergating_by_smu(adev, AMD_IP_BLOCK_TYPE_VCE, !enable);
"pp_power_profile_mode\n");
return ret;
}
- if (hwmgr->od_enabled) {
+ if ((is_support_sw_smu(adev) && adev->smu.od_enabled) ||
+ (!is_support_sw_smu(adev) && hwmgr->od_enabled)) {
ret = device_create_file(adev->dev,
&dev_attr_pp_od_clk_voltage);
if (ret) {
device_remove_file(adev->dev, &dev_attr_pp_mclk_od);
device_remove_file(adev->dev,
&dev_attr_pp_power_profile_mode);
- if (hwmgr->od_enabled)
+ if ((is_support_sw_smu(adev) && adev->smu.od_enabled) ||
+ (!is_support_sw_smu(adev) && hwmgr->od_enabled))
device_remove_file(adev->dev,
&dev_attr_pp_od_clk_voltage);
device_remove_file(adev->dev, &dev_attr_gpu_busy_percent);
amdgpu_fence_wait_empty(ring);
}
- if (adev->powerplay.pp_funcs->dispatch_tasks) {
- if (!amdgpu_device_has_dc_support(adev)) {
+ if (is_support_sw_smu(adev)) {
+ struct smu_context *smu = &adev->smu;
+ struct smu_dpm_context *smu_dpm = &adev->smu.smu_dpm;
+ mutex_lock(&(smu->mutex));
+ smu_handle_task(&adev->smu,
+ smu_dpm->dpm_level,
+ AMD_PP_TASK_DISPLAY_CONFIG_CHANGE);
+ mutex_unlock(&(smu->mutex));
+ } else {
+ if (adev->powerplay.pp_funcs->dispatch_tasks) {
+ if (!amdgpu_device_has_dc_support(adev)) {
+ mutex_lock(&adev->pm.mutex);
+ amdgpu_dpm_get_active_displays(adev);
+ adev->pm.pm_display_cfg.num_display = adev->pm.dpm.new_active_crtc_count;
+ adev->pm.pm_display_cfg.vrefresh = amdgpu_dpm_get_vrefresh(adev);
+ adev->pm.pm_display_cfg.min_vblank_time = amdgpu_dpm_get_vblank_time(adev);
+ /* we have issues with mclk switching with refresh rates over 120 hz on the non-DC code. */
+ if (adev->pm.pm_display_cfg.vrefresh > 120)
+ adev->pm.pm_display_cfg.min_vblank_time = 0;
+ if (adev->powerplay.pp_funcs->display_configuration_change)
+ adev->powerplay.pp_funcs->display_configuration_change(
+ adev->powerplay.pp_handle,
+ &adev->pm.pm_display_cfg);
+ mutex_unlock(&adev->pm.mutex);
+ }
+ amdgpu_dpm_dispatch_task(adev, AMD_PP_TASK_DISPLAY_CONFIG_CHANGE, NULL);
+ } else {
mutex_lock(&adev->pm.mutex);
amdgpu_dpm_get_active_displays(adev);
- adev->pm.pm_display_cfg.num_display = adev->pm.dpm.new_active_crtc_count;
- adev->pm.pm_display_cfg.vrefresh = amdgpu_dpm_get_vrefresh(adev);
- adev->pm.pm_display_cfg.min_vblank_time = amdgpu_dpm_get_vblank_time(adev);
- /* we have issues with mclk switching with refresh rates over 120 hz on the non-DC code. */
- if (adev->pm.pm_display_cfg.vrefresh > 120)
- adev->pm.pm_display_cfg.min_vblank_time = 0;
- if (adev->powerplay.pp_funcs->display_configuration_change)
- adev->powerplay.pp_funcs->display_configuration_change(
- adev->powerplay.pp_handle,
- &adev->pm.pm_display_cfg);
+ amdgpu_dpm_change_power_state_locked(adev);
mutex_unlock(&adev->pm.mutex);
}
- amdgpu_dpm_dispatch_task(adev, AMD_PP_TASK_DISPLAY_CONFIG_CHANGE, NULL);
- } else {
- mutex_lock(&adev->pm.mutex);
- amdgpu_dpm_get_active_displays(adev);
- amdgpu_dpm_change_power_state_locked(adev);
- mutex_unlock(&adev->pm.mutex);
}
}
uint32_t query = 0;
int size;
- /* sanity check PP is enabled */
- if (!(adev->powerplay.pp_funcs &&
- adev->powerplay.pp_funcs->read_sensor))
- return -EINVAL;
-
/* GPU Clocks */
size = sizeof(value);
seq_printf(m, "GFX Clocks and Power:\n");
if ((adev->flags & AMD_IS_PX) &&
(ddev->switch_power_state != DRM_SWITCH_POWER_ON)) {
seq_printf(m, "PX asic powered off\n");
- } else if (adev->powerplay.pp_funcs->debugfs_print_current_performance_level) {
+ } else if (!is_support_sw_smu(adev) && adev->powerplay.pp_funcs->debugfs_print_current_performance_level) {
mutex_lock(&adev->pm.mutex);
if (adev->powerplay.pp_funcs->debugfs_print_current_performance_level)
adev->powerplay.pp_funcs->debugfs_print_current_performance_level(adev, m);
{
int ret;
int index;
+ int timeout = 2000;
memset(psp->cmd_buf_mem, 0, PSP_CMD_BUFFER_SIZE);
return ret;
}
- while (*((unsigned int *)psp->fence_buf) != index)
+ while (*((unsigned int *)psp->fence_buf) != index) {
+ if (--timeout == 0)
+ break;
msleep(1);
+ }
/* In some cases, psp response status is not 0 even there is no
* problem while the command is submitted. Some version of PSP FW
* during psp initialization to avoid breaking hw_init and it doesn't
* return -EINVAL.
*/
- if (psp->cmd_buf_mem->resp.status) {
+ if (psp->cmd_buf_mem->resp.status || !timeout) {
if (ucode)
DRM_WARN("failed to load ucode id (%d) ",
ucode->ucode_id);
DRM_WARN("psp command failed and response status is (%d)\n",
psp->cmd_buf_mem->resp.status);
+ if (!timeout)
+ return -EINVAL;
}
/* get xGMI session id from response buffer */
int ret;
/*
- * Allocate 3M memory aligned to 1M from Frame Buffer (local
- * physical).
+ * According to HW engineer, they prefer the TMR address be "naturally
+ * aligned" , e.g. the start address be an integer divide of TMR size.
*
* Note: this memory need be reserved till the driver
* uninitializes.
*/
- ret = amdgpu_bo_create_kernel(psp->adev, PSP_TMR_SIZE, 0x100000,
+ ret = amdgpu_bo_create_kernel(psp->adev, PSP_TMR_SIZE, PSP_TMR_SIZE,
AMDGPU_GEM_DOMAIN_VRAM,
&psp->tmr_bo, &psp->tmr_mc_addr, &psp->tmr_buf);
return ret;
}
+// ras begin
+static void psp_prep_ras_ta_load_cmd_buf(struct psp_gfx_cmd_resp *cmd,
+ uint64_t ras_ta_mc, uint64_t ras_mc_shared,
+ uint32_t ras_ta_size, uint32_t shared_size)
+{
+ cmd->cmd_id = GFX_CMD_ID_LOAD_TA;
+ cmd->cmd.cmd_load_ta.app_phy_addr_lo = lower_32_bits(ras_ta_mc);
+ cmd->cmd.cmd_load_ta.app_phy_addr_hi = upper_32_bits(ras_ta_mc);
+ cmd->cmd.cmd_load_ta.app_len = ras_ta_size;
+
+ cmd->cmd.cmd_load_ta.cmd_buf_phy_addr_lo = lower_32_bits(ras_mc_shared);
+ cmd->cmd.cmd_load_ta.cmd_buf_phy_addr_hi = upper_32_bits(ras_mc_shared);
+ cmd->cmd.cmd_load_ta.cmd_buf_len = shared_size;
+}
+
+static int psp_ras_init_shared_buf(struct psp_context *psp)
+{
+ int ret;
+
+ /*
+ * Allocate 16k memory aligned to 4k from Frame Buffer (local
+ * physical) for ras ta <-> Driver
+ */
+ ret = amdgpu_bo_create_kernel(psp->adev, PSP_RAS_SHARED_MEM_SIZE,
+ PAGE_SIZE, AMDGPU_GEM_DOMAIN_VRAM,
+ &psp->ras.ras_shared_bo,
+ &psp->ras.ras_shared_mc_addr,
+ &psp->ras.ras_shared_buf);
+
+ return ret;
+}
+
+static int psp_ras_load(struct psp_context *psp)
+{
+ int ret;
+ struct psp_gfx_cmd_resp *cmd;
+
+ /*
+ * TODO: bypass the loading in sriov for now
+ */
+ if (amdgpu_sriov_vf(psp->adev))
+ return 0;
+
+ cmd = kzalloc(sizeof(struct psp_gfx_cmd_resp), GFP_KERNEL);
+ if (!cmd)
+ return -ENOMEM;
+
+ memset(psp->fw_pri_buf, 0, PSP_1_MEG);
+ memcpy(psp->fw_pri_buf, psp->ta_ras_start_addr, psp->ta_ras_ucode_size);
+
+ psp_prep_ras_ta_load_cmd_buf(cmd, psp->fw_pri_mc_addr,
+ psp->ras.ras_shared_mc_addr,
+ psp->ta_ras_ucode_size, PSP_RAS_SHARED_MEM_SIZE);
+
+ ret = psp_cmd_submit_buf(psp, NULL, cmd,
+ psp->fence_buf_mc_addr);
+
+ if (!ret) {
+ psp->ras.ras_initialized = 1;
+ psp->ras.session_id = cmd->resp.session_id;
+ }
+
+ kfree(cmd);
+
+ return ret;
+}
+
+static void psp_prep_ras_ta_unload_cmd_buf(struct psp_gfx_cmd_resp *cmd,
+ uint32_t ras_session_id)
+{
+ cmd->cmd_id = GFX_CMD_ID_UNLOAD_TA;
+ cmd->cmd.cmd_unload_ta.session_id = ras_session_id;
+}
+
+static int psp_ras_unload(struct psp_context *psp)
+{
+ int ret;
+ struct psp_gfx_cmd_resp *cmd;
+
+ /*
+ * TODO: bypass the unloading in sriov for now
+ */
+ if (amdgpu_sriov_vf(psp->adev))
+ return 0;
+
+ cmd = kzalloc(sizeof(struct psp_gfx_cmd_resp), GFP_KERNEL);
+ if (!cmd)
+ return -ENOMEM;
+
+ psp_prep_ras_ta_unload_cmd_buf(cmd, psp->ras.session_id);
+
+ ret = psp_cmd_submit_buf(psp, NULL, cmd,
+ psp->fence_buf_mc_addr);
+
+ kfree(cmd);
+
+ return ret;
+}
+
+static void psp_prep_ras_ta_invoke_cmd_buf(struct psp_gfx_cmd_resp *cmd,
+ uint32_t ta_cmd_id,
+ uint32_t ras_session_id)
+{
+ cmd->cmd_id = GFX_CMD_ID_INVOKE_CMD;
+ cmd->cmd.cmd_invoke_cmd.session_id = ras_session_id;
+ cmd->cmd.cmd_invoke_cmd.ta_cmd_id = ta_cmd_id;
+ /* Note: cmd_invoke_cmd.buf is not used for now */
+}
+
+int psp_ras_invoke(struct psp_context *psp, uint32_t ta_cmd_id)
+{
+ int ret;
+ struct psp_gfx_cmd_resp *cmd;
+
+ /*
+ * TODO: bypass the loading in sriov for now
+ */
+ if (amdgpu_sriov_vf(psp->adev))
+ return 0;
+
+ cmd = kzalloc(sizeof(struct psp_gfx_cmd_resp), GFP_KERNEL);
+ if (!cmd)
+ return -ENOMEM;
+
+ psp_prep_ras_ta_invoke_cmd_buf(cmd, ta_cmd_id,
+ psp->ras.session_id);
+
+ ret = psp_cmd_submit_buf(psp, NULL, cmd,
+ psp->fence_buf_mc_addr);
+
+ kfree(cmd);
+
+ return ret;
+}
+
+int psp_ras_enable_features(struct psp_context *psp,
+ union ta_ras_cmd_input *info, bool enable)
+{
+ struct ta_ras_shared_memory *ras_cmd;
+ int ret;
+
+ if (!psp->ras.ras_initialized)
+ return -EINVAL;
+
+ ras_cmd = (struct ta_ras_shared_memory *)psp->ras.ras_shared_buf;
+ memset(ras_cmd, 0, sizeof(struct ta_ras_shared_memory));
+
+ if (enable)
+ ras_cmd->cmd_id = TA_RAS_COMMAND__ENABLE_FEATURES;
+ else
+ ras_cmd->cmd_id = TA_RAS_COMMAND__DISABLE_FEATURES;
+
+ ras_cmd->ras_in_message = *info;
+
+ ret = psp_ras_invoke(psp, ras_cmd->cmd_id);
+ if (ret)
+ return -EINVAL;
+
+ return ras_cmd->ras_status;
+}
+
+static int psp_ras_terminate(struct psp_context *psp)
+{
+ int ret;
+
+ if (!psp->ras.ras_initialized)
+ return 0;
+
+ ret = psp_ras_unload(psp);
+ if (ret)
+ return ret;
+
+ psp->ras.ras_initialized = 0;
+
+ /* free ras shared memory */
+ amdgpu_bo_free_kernel(&psp->ras.ras_shared_bo,
+ &psp->ras.ras_shared_mc_addr,
+ &psp->ras.ras_shared_buf);
+
+ return 0;
+}
+
+static int psp_ras_initialize(struct psp_context *psp)
+{
+ int ret;
+
+ if (!psp->ras.ras_initialized) {
+ ret = psp_ras_init_shared_buf(psp);
+ if (ret)
+ return ret;
+ }
+
+ ret = psp_ras_load(psp);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+// ras end
+
static int psp_hw_start(struct psp_context *psp)
{
struct amdgpu_device *adev = psp->adev;
if (!amdgpu_sriov_vf(adev) || !adev->in_gpu_reset) {
ret = psp_bootloader_load_sysdrv(psp);
- if (ret)
+ if (ret) {
+ DRM_ERROR("PSP load sysdrv failed!\n");
return ret;
+ }
ret = psp_bootloader_load_sos(psp);
- if (ret)
+ if (ret) {
+ DRM_ERROR("PSP load sos failed!\n");
return ret;
+ }
}
ret = psp_ring_create(psp, PSP_RING_TYPE__KM);
- if (ret)
+ if (ret) {
+ DRM_ERROR("PSP create ring failed!\n");
return ret;
+ }
ret = psp_tmr_load(psp);
- if (ret)
+ if (ret) {
+ DRM_ERROR("PSP load tmr failed!\n");
return ret;
+ }
ret = psp_asd_load(psp);
- if (ret)
+ if (ret) {
+ DRM_ERROR("PSP load asd failed!\n");
return ret;
+ }
if (adev->gmc.xgmi.num_physical_nodes > 1) {
ret = psp_xgmi_initialize(psp);
dev_err(psp->adev->dev,
"XGMI: Failed to initialize XGMI session\n");
}
+
+
+ if (psp->adev->psp.ta_fw) {
+ ret = psp_ras_initialize(psp);
+ if (ret)
+ dev_err(psp->adev->dev,
+ "RAS: Failed to initialize RAS\n");
+ }
+
return 0;
}
&psp->fence_buf_mc_addr,
&psp->fence_buf);
if (ret)
- goto failed_mem2;
+ goto failed;
ret = amdgpu_bo_create_kernel(adev, PSP_CMD_BUFFER_SIZE, PAGE_SIZE,
AMDGPU_GEM_DOMAIN_VRAM,
&psp->cmd_buf_bo, &psp->cmd_buf_mc_addr,
(void **)&psp->cmd_buf_mem);
if (ret)
- goto failed_mem1;
+ goto failed;
memset(psp->fence_buf, 0, PSP_FENCE_BUFFER_SIZE);
ret = psp_ring_init(psp, PSP_RING_TYPE__KM);
- if (ret)
- goto failed_mem;
+ if (ret) {
+ DRM_ERROR("PSP ring init failed!\n");
+ goto failed;
+ }
ret = psp_tmr_init(psp);
- if (ret)
- goto failed_mem;
+ if (ret) {
+ DRM_ERROR("PSP tmr init failed!\n");
+ goto failed;
+ }
ret = psp_asd_init(psp);
- if (ret)
- goto failed_mem;
+ if (ret) {
+ DRM_ERROR("PSP asd init failed!\n");
+ goto failed;
+ }
skip_memalloc:
ret = psp_hw_start(psp);
if (ret)
- goto failed_mem;
+ goto failed;
ret = psp_np_fw_load(psp);
if (ret)
- goto failed_mem;
+ goto failed;
return 0;
-failed_mem:
- amdgpu_bo_free_kernel(&psp->cmd_buf_bo,
- &psp->cmd_buf_mc_addr,
- (void **)&psp->cmd_buf_mem);
-failed_mem1:
- amdgpu_bo_free_kernel(&psp->fence_buf_bo,
- &psp->fence_buf_mc_addr, &psp->fence_buf);
-failed_mem2:
- amdgpu_bo_free_kernel(&psp->fw_pri_bo,
- &psp->fw_pri_mc_addr, &psp->fw_pri_buf);
failed:
- kfree(psp->cmd);
- psp->cmd = NULL;
+ /*
+ * all cleanup jobs (xgmi terminate, ras terminate,
+ * ring destroy, cmd/fence/fw buffers destory,
+ * psp->cmd destory) are delayed to psp_hw_fini
+ */
return ret;
}
psp->xgmi_context.initialized == 1)
psp_xgmi_terminate(psp);
+ if (psp->adev->psp.ta_fw)
+ psp_ras_terminate(psp);
+
psp_ring_destroy(psp, PSP_RING_TYPE__KM);
amdgpu_bo_free_kernel(&psp->tmr_bo, &psp->tmr_mc_addr, &psp->tmr_buf);
}
}
+ if (psp->adev->psp.ta_fw) {
+ ret = psp_ras_terminate(psp);
+ if (ret) {
+ DRM_ERROR("Failed to terminate ras ta\n");
+ return ret;
+ }
+ }
+
ret = psp_ring_stop(psp, PSP_RING_TYPE__KM);
if (ret) {
DRM_ERROR("PSP ring stop failed\n");
#include "amdgpu.h"
#include "psp_gfx_if.h"
#include "ta_xgmi_if.h"
+#include "ta_ras_if.h"
#define PSP_FENCE_BUFFER_SIZE 0x1000
#define PSP_CMD_BUFFER_SIZE 0x1000
#define PSP_ASD_SHARED_MEM_SIZE 0x4000
#define PSP_XGMI_SHARED_MEM_SIZE 0x4000
+#define PSP_RAS_SHARED_MEM_SIZE 0x4000
#define PSP_1_MEG 0x100000
#define PSP_TMR_SIZE 0x400000
int (*xgmi_set_topology_info)(struct psp_context *psp, int number_devices,
struct psp_xgmi_topology_info *topology);
bool (*support_vmr_ring)(struct psp_context *psp);
+ int (*ras_trigger_error)(struct psp_context *psp,
+ struct ta_ras_trigger_error_input *info);
+ int (*ras_cure_posion)(struct psp_context *psp, uint64_t *mode_ptr);
};
struct psp_xgmi_context {
void *xgmi_shared_buf;
};
+struct psp_ras_context {
+ /*ras fw*/
+ bool ras_initialized;
+ uint32_t session_id;
+ struct amdgpu_bo *ras_shared_bo;
+ uint64_t ras_shared_mc_addr;
+ void *ras_shared_buf;
+ struct amdgpu_ras *ras;
+};
+
struct psp_context
{
struct amdgpu_device *adev;
/* xgmi ta firmware and buffer */
const struct firmware *ta_fw;
+ uint32_t ta_fw_version;
uint32_t ta_xgmi_ucode_version;
uint32_t ta_xgmi_ucode_size;
uint8_t *ta_xgmi_start_addr;
+ uint32_t ta_ras_ucode_version;
+ uint32_t ta_ras_ucode_size;
+ uint8_t *ta_ras_start_addr;
struct psp_xgmi_context xgmi_context;
+ struct psp_ras_context ras;
};
struct amdgpu_psp_funcs {
#define amdgpu_psp_check_fw_loading_status(adev, i) (adev)->firmware.funcs->check_fw_loading_status((adev), (i))
+#define psp_ras_trigger_error(psp, info) \
+ ((psp)->funcs->ras_trigger_error ? \
+ (psp)->funcs->ras_trigger_error((psp), (info)) : -EINVAL)
+#define psp_ras_cure_posion(psp, addr) \
+ ((psp)->funcs->ras_cure_posion ? \
+ (psp)->funcs->ras_cure_posion(psp, (addr)) : -EINVAL)
+
extern const struct amd_ip_funcs psp_ip_funcs;
extern const struct amdgpu_ip_block_version psp_v3_1_ip_block;
int psp_gpu_reset(struct amdgpu_device *adev);
int psp_xgmi_invoke(struct psp_context *psp, uint32_t ta_cmd_id);
+
+int psp_ras_invoke(struct psp_context *psp, uint32_t ta_cmd_id);
+int psp_ras_enable_features(struct psp_context *psp,
+ union ta_ras_cmd_input *info, bool enable);
+
extern const struct amdgpu_ip_block_version psp_v11_0_ip_block;
#endif
--- /dev/null
+/*
+ * Copyright 2018 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ *
+ */
+#include <linux/debugfs.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include "amdgpu.h"
+#include "amdgpu_ras.h"
+#include "amdgpu_atomfirmware.h"
+
+struct ras_ih_data {
+ /* interrupt bottom half */
+ struct work_struct ih_work;
+ int inuse;
+ /* IP callback */
+ ras_ih_cb cb;
+ /* full of entries */
+ unsigned char *ring;
+ unsigned int ring_size;
+ unsigned int element_size;
+ unsigned int aligned_element_size;
+ unsigned int rptr;
+ unsigned int wptr;
+};
+
+struct ras_fs_data {
+ char sysfs_name[32];
+ char debugfs_name[32];
+};
+
+struct ras_err_data {
+ unsigned long ue_count;
+ unsigned long ce_count;
+};
+
+struct ras_err_handler_data {
+ /* point to bad pages array */
+ struct {
+ unsigned long bp;
+ struct amdgpu_bo *bo;
+ } *bps;
+ /* the count of entries */
+ int count;
+ /* the space can place new entries */
+ int space_left;
+ /* last reserved entry's index + 1 */
+ int last_reserved;
+};
+
+struct ras_manager {
+ struct ras_common_if head;
+ /* reference count */
+ int use;
+ /* ras block link */
+ struct list_head node;
+ /* the device */
+ struct amdgpu_device *adev;
+ /* debugfs */
+ struct dentry *ent;
+ /* sysfs */
+ struct device_attribute sysfs_attr;
+ int attr_inuse;
+
+ /* fs node name */
+ struct ras_fs_data fs_data;
+
+ /* IH data */
+ struct ras_ih_data ih_data;
+
+ struct ras_err_data err_data;
+};
+
+const char *ras_error_string[] = {
+ "none",
+ "parity",
+ "single_correctable",
+ "multi_uncorrectable",
+ "poison",
+};
+
+const char *ras_block_string[] = {
+ "umc",
+ "sdma",
+ "gfx",
+ "mmhub",
+ "athub",
+ "pcie_bif",
+ "hdp",
+ "xgmi_wafl",
+ "df",
+ "smn",
+ "sem",
+ "mp0",
+ "mp1",
+ "fuse",
+};
+
+#define ras_err_str(i) (ras_error_string[ffs(i)])
+#define ras_block_str(i) (ras_block_string[i])
+
+#define AMDGPU_RAS_FLAG_INIT_BY_VBIOS 1
+#define RAS_DEFAULT_FLAGS (AMDGPU_RAS_FLAG_INIT_BY_VBIOS)
+
+static void amdgpu_ras_self_test(struct amdgpu_device *adev)
+{
+ /* TODO */
+}
+
+static ssize_t amdgpu_ras_debugfs_read(struct file *f, char __user *buf,
+ size_t size, loff_t *pos)
+{
+ struct ras_manager *obj = (struct ras_manager *)file_inode(f)->i_private;
+ struct ras_query_if info = {
+ .head = obj->head,
+ };
+ ssize_t s;
+ char val[128];
+
+ if (amdgpu_ras_error_query(obj->adev, &info))
+ return -EINVAL;
+
+ s = snprintf(val, sizeof(val), "%s: %lu\n%s: %lu\n",
+ "ue", info.ue_count,
+ "ce", info.ce_count);
+ if (*pos >= s)
+ return 0;
+
+ s -= *pos;
+ s = min_t(u64, s, size);
+
+
+ if (copy_to_user(buf, &val[*pos], s))
+ return -EINVAL;
+
+ *pos += s;
+
+ return s;
+}
+
+static const struct file_operations amdgpu_ras_debugfs_ops = {
+ .owner = THIS_MODULE,
+ .read = amdgpu_ras_debugfs_read,
+ .write = NULL,
+ .llseek = default_llseek
+};
+
+static int amdgpu_ras_find_block_id_by_name(const char *name, int *block_id)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(ras_block_string); i++) {
+ *block_id = i;
+ if (strcmp(name, ras_block_str(i)) == 0)
+ return 0;
+ }
+ return -EINVAL;
+}
+
+static int amdgpu_ras_debugfs_ctrl_parse_data(struct file *f,
+ const char __user *buf, size_t size,
+ loff_t *pos, struct ras_debug_if *data)
+{
+ ssize_t s = min_t(u64, 64, size);
+ char str[65];
+ char block_name[33];
+ char err[9] = "ue";
+ int op = -1;
+ int block_id;
+ u64 address, value;
+
+ if (*pos)
+ return -EINVAL;
+ *pos = size;
+
+ memset(str, 0, sizeof(str));
+ memset(data, 0, sizeof(*data));
+
+ if (copy_from_user(str, buf, s))
+ return -EINVAL;
+
+ if (sscanf(str, "disable %32s", block_name) == 1)
+ op = 0;
+ else if (sscanf(str, "enable %32s %8s", block_name, err) == 2)
+ op = 1;
+ else if (sscanf(str, "inject %32s %8s", block_name, err) == 2)
+ op = 2;
+ else if (str[0] && str[1] && str[2] && str[3])
+ /* ascii string, but commands are not matched. */
+ return -EINVAL;
+
+ if (op != -1) {
+ if (amdgpu_ras_find_block_id_by_name(block_name, &block_id))
+ return -EINVAL;
+
+ data->head.block = block_id;
+ data->head.type = memcmp("ue", err, 2) == 0 ?
+ AMDGPU_RAS_ERROR__MULTI_UNCORRECTABLE :
+ AMDGPU_RAS_ERROR__SINGLE_CORRECTABLE;
+ data->op = op;
+
+ if (op == 2) {
+ if (sscanf(str, "%*s %*s %*s %llu %llu",
+ &address, &value) != 2)
+ if (sscanf(str, "%*s %*s %*s 0x%llx 0x%llx",
+ &address, &value) != 2)
+ return -EINVAL;
+ data->inject.address = address;
+ data->inject.value = value;
+ }
+ } else {
+ if (size < sizeof(*data))
+ return -EINVAL;
+
+ if (copy_from_user(data, buf, sizeof(*data)))
+ return -EINVAL;
+ }
+
+ return 0;
+}
+/*
+ * DOC: ras debugfs control interface
+ *
+ * It accepts struct ras_debug_if who has two members.
+ *
+ * First member: ras_debug_if::head or ras_debug_if::inject.
+ *
+ * head is used to indicate which IP block will be under control.
+ *
+ * head has four members, they are block, type, sub_block_index, name.
+ * block: which IP will be under control.
+ * type: what kind of error will be enabled/disabled/injected.
+ * sub_block_index: some IPs have subcomponets. say, GFX, sDMA.
+ * name: the name of IP.
+ *
+ * inject has two more members than head, they are address, value.
+ * As their names indicate, inject operation will write the
+ * value to the address.
+ *
+ * Second member: struct ras_debug_if::op.
+ * It has three kinds of operations.
+ * 0: disable RAS on the block. Take ::head as its data.
+ * 1: enable RAS on the block. Take ::head as its data.
+ * 2: inject errors on the block. Take ::inject as its data.
+ *
+ * How to use the interface?
+ * programs:
+ * copy the struct ras_debug_if in your codes and initialize it.
+ * write the struct to the control node.
+ *
+ * bash:
+ * echo op block [error [address value]] > .../ras/ras_ctrl
+ * op: disable, enable, inject
+ * disable: only block is needed
+ * enable: block and error are needed
+ * inject: error, address, value are needed
+ * block: umc, smda, gfx, .........
+ * see ras_block_string[] for details
+ * error: ue, ce
+ * ue: multi_uncorrectable
+ * ce: single_correctable
+ *
+ * here are some examples for bash commands,
+ * echo inject umc ue 0x0 0x0 > /sys/kernel/debug/dri/0/ras/ras_ctrl
+ * echo inject umc ce 0 0 > /sys/kernel/debug/dri/0/ras/ras_ctrl
+ * echo disable umc > /sys/kernel/debug/dri/0/ras/ras_ctrl
+ *
+ * How to check the result?
+ *
+ * For disable/enable, please check ras features at
+ * /sys/class/drm/card[0/1/2...]/device/ras/features
+ *
+ * For inject, please check corresponding err count at
+ * /sys/class/drm/card[0/1/2...]/device/ras/[gfx/sdma/...]_err_count
+ *
+ * NOTE: operation is only allowed on blocks which are supported.
+ * Please check ras mask at /sys/module/amdgpu/parameters/ras_mask
+ */
+static ssize_t amdgpu_ras_debugfs_ctrl_write(struct file *f, const char __user *buf,
+ size_t size, loff_t *pos)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)file_inode(f)->i_private;
+ struct ras_debug_if data;
+ int ret = 0;
+
+ ret = amdgpu_ras_debugfs_ctrl_parse_data(f, buf, size, pos, &data);
+ if (ret)
+ return -EINVAL;
+
+ if (!amdgpu_ras_is_supported(adev, data.head.block))
+ return -EINVAL;
+
+ switch (data.op) {
+ case 0:
+ ret = amdgpu_ras_feature_enable(adev, &data.head, 0);
+ break;
+ case 1:
+ ret = amdgpu_ras_feature_enable(adev, &data.head, 1);
+ break;
+ case 2:
+ ret = amdgpu_ras_error_inject(adev, &data.inject);
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ };
+
+ if (ret)
+ return -EINVAL;
+
+ return size;
+}
+
+static const struct file_operations amdgpu_ras_debugfs_ctrl_ops = {
+ .owner = THIS_MODULE,
+ .read = NULL,
+ .write = amdgpu_ras_debugfs_ctrl_write,
+ .llseek = default_llseek
+};
+
+static ssize_t amdgpu_ras_sysfs_read(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ras_manager *obj = container_of(attr, struct ras_manager, sysfs_attr);
+ struct ras_query_if info = {
+ .head = obj->head,
+ };
+
+ if (amdgpu_ras_error_query(obj->adev, &info))
+ return -EINVAL;
+
+ return snprintf(buf, PAGE_SIZE, "%s: %lu\n%s: %lu\n",
+ "ue", info.ue_count,
+ "ce", info.ce_count);
+}
+
+/* obj begin */
+
+#define get_obj(obj) do { (obj)->use++; } while (0)
+#define alive_obj(obj) ((obj)->use)
+
+static inline void put_obj(struct ras_manager *obj)
+{
+ if (obj && --obj->use == 0)
+ list_del(&obj->node);
+ if (obj && obj->use < 0) {
+ DRM_ERROR("RAS ERROR: Unbalance obj(%s) use\n", obj->head.name);
+ }
+}
+
+/* make one obj and return it. */
+static struct ras_manager *amdgpu_ras_create_obj(struct amdgpu_device *adev,
+ struct ras_common_if *head)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_manager *obj;
+
+ if (!con)
+ return NULL;
+
+ if (head->block >= AMDGPU_RAS_BLOCK_COUNT)
+ return NULL;
+
+ obj = &con->objs[head->block];
+ /* already exist. return obj? */
+ if (alive_obj(obj))
+ return NULL;
+
+ obj->head = *head;
+ obj->adev = adev;
+ list_add(&obj->node, &con->head);
+ get_obj(obj);
+
+ return obj;
+}
+
+/* return an obj equal to head, or the first when head is NULL */
+static struct ras_manager *amdgpu_ras_find_obj(struct amdgpu_device *adev,
+ struct ras_common_if *head)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_manager *obj;
+ int i;
+
+ if (!con)
+ return NULL;
+
+ if (head) {
+ if (head->block >= AMDGPU_RAS_BLOCK_COUNT)
+ return NULL;
+
+ obj = &con->objs[head->block];
+
+ if (alive_obj(obj)) {
+ WARN_ON(head->block != obj->head.block);
+ return obj;
+ }
+ } else {
+ for (i = 0; i < AMDGPU_RAS_BLOCK_COUNT; i++) {
+ obj = &con->objs[i];
+ if (alive_obj(obj)) {
+ WARN_ON(i != obj->head.block);
+ return obj;
+ }
+ }
+ }
+
+ return NULL;
+}
+/* obj end */
+
+/* feature ctl begin */
+static int amdgpu_ras_is_feature_allowed(struct amdgpu_device *adev,
+ struct ras_common_if *head)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+
+ return con->hw_supported & BIT(head->block);
+}
+
+static int amdgpu_ras_is_feature_enabled(struct amdgpu_device *adev,
+ struct ras_common_if *head)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+
+ return con->features & BIT(head->block);
+}
+
+/*
+ * if obj is not created, then create one.
+ * set feature enable flag.
+ */
+static int __amdgpu_ras_feature_enable(struct amdgpu_device *adev,
+ struct ras_common_if *head, int enable)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_manager *obj = amdgpu_ras_find_obj(adev, head);
+
+ /* If hardware does not support ras, then do not create obj.
+ * But if hardware support ras, we can create the obj.
+ * Ras framework checks con->hw_supported to see if it need do
+ * corresponding initialization.
+ * IP checks con->support to see if it need disable ras.
+ */
+ if (!amdgpu_ras_is_feature_allowed(adev, head))
+ return 0;
+ if (!(!!enable ^ !!amdgpu_ras_is_feature_enabled(adev, head)))
+ return 0;
+
+ if (enable) {
+ if (!obj) {
+ obj = amdgpu_ras_create_obj(adev, head);
+ if (!obj)
+ return -EINVAL;
+ } else {
+ /* In case we create obj somewhere else */
+ get_obj(obj);
+ }
+ con->features |= BIT(head->block);
+ } else {
+ if (obj && amdgpu_ras_is_feature_enabled(adev, head)) {
+ con->features &= ~BIT(head->block);
+ put_obj(obj);
+ }
+ }
+
+ return 0;
+}
+
+/* wrapper of psp_ras_enable_features */
+int amdgpu_ras_feature_enable(struct amdgpu_device *adev,
+ struct ras_common_if *head, bool enable)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ union ta_ras_cmd_input info;
+ int ret;
+
+ if (!con)
+ return -EINVAL;
+
+ if (!enable) {
+ info.disable_features = (struct ta_ras_disable_features_input) {
+ .block_id = amdgpu_ras_block_to_ta(head->block),
+ .error_type = amdgpu_ras_error_to_ta(head->type),
+ };
+ } else {
+ info.enable_features = (struct ta_ras_enable_features_input) {
+ .block_id = amdgpu_ras_block_to_ta(head->block),
+ .error_type = amdgpu_ras_error_to_ta(head->type),
+ };
+ }
+
+ /* Do not enable if it is not allowed. */
+ WARN_ON(enable && !amdgpu_ras_is_feature_allowed(adev, head));
+ /* Are we alerady in that state we are going to set? */
+ if (!(!!enable ^ !!amdgpu_ras_is_feature_enabled(adev, head)))
+ return 0;
+
+ ret = psp_ras_enable_features(&adev->psp, &info, enable);
+ if (ret) {
+ DRM_ERROR("RAS ERROR: %s %s feature failed ret %d\n",
+ enable ? "enable":"disable",
+ ras_block_str(head->block),
+ ret);
+ return -EINVAL;
+ }
+
+ /* setup the obj */
+ __amdgpu_ras_feature_enable(adev, head, enable);
+
+ return 0;
+}
+
+/* Only used in device probe stage and called only once. */
+int amdgpu_ras_feature_enable_on_boot(struct amdgpu_device *adev,
+ struct ras_common_if *head, bool enable)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ int ret;
+
+ if (!con)
+ return -EINVAL;
+
+ if (con->flags & AMDGPU_RAS_FLAG_INIT_BY_VBIOS) {
+ /* If ras is enabled by vbios, we set up ras object first in
+ * both case. For enable, that is all what we need do. For
+ * disable, we need perform a ras TA disable cmd after that.
+ */
+ ret = __amdgpu_ras_feature_enable(adev, head, 1);
+ if (ret)
+ return ret;
+
+ if (!enable)
+ ret = amdgpu_ras_feature_enable(adev, head, 0);
+ } else
+ ret = amdgpu_ras_feature_enable(adev, head, enable);
+
+ return ret;
+}
+
+static int amdgpu_ras_disable_all_features(struct amdgpu_device *adev,
+ bool bypass)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_manager *obj, *tmp;
+
+ list_for_each_entry_safe(obj, tmp, &con->head, node) {
+ /* bypass psp.
+ * aka just release the obj and corresponding flags
+ */
+ if (bypass) {
+ if (__amdgpu_ras_feature_enable(adev, &obj->head, 0))
+ break;
+ } else {
+ if (amdgpu_ras_feature_enable(adev, &obj->head, 0))
+ break;
+ }
+ }
+
+ return con->features;
+}
+
+static int amdgpu_ras_enable_all_features(struct amdgpu_device *adev,
+ bool bypass)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ int ras_block_count = AMDGPU_RAS_BLOCK_COUNT;
+ int i;
+ const enum amdgpu_ras_error_type default_ras_type =
+ AMDGPU_RAS_ERROR__NONE;
+
+ for (i = 0; i < ras_block_count; i++) {
+ struct ras_common_if head = {
+ .block = i,
+ .type = default_ras_type,
+ .sub_block_index = 0,
+ };
+ strcpy(head.name, ras_block_str(i));
+ if (bypass) {
+ /*
+ * bypass psp. vbios enable ras for us.
+ * so just create the obj
+ */
+ if (__amdgpu_ras_feature_enable(adev, &head, 1))
+ break;
+ } else {
+ if (amdgpu_ras_feature_enable(adev, &head, 1))
+ break;
+ }
+ }
+
+ return con->features;
+}
+/* feature ctl end */
+
+/* query/inject/cure begin */
+int amdgpu_ras_error_query(struct amdgpu_device *adev,
+ struct ras_query_if *info)
+{
+ struct ras_manager *obj = amdgpu_ras_find_obj(adev, &info->head);
+
+ if (!obj)
+ return -EINVAL;
+ /* TODO might read the register to read the count */
+
+ info->ue_count = obj->err_data.ue_count;
+ info->ce_count = obj->err_data.ce_count;
+
+ return 0;
+}
+
+/* wrapper of psp_ras_trigger_error */
+int amdgpu_ras_error_inject(struct amdgpu_device *adev,
+ struct ras_inject_if *info)
+{
+ struct ras_manager *obj = amdgpu_ras_find_obj(adev, &info->head);
+ struct ta_ras_trigger_error_input block_info = {
+ .block_id = amdgpu_ras_block_to_ta(info->head.block),
+ .inject_error_type = amdgpu_ras_error_to_ta(info->head.type),
+ .sub_block_index = info->head.sub_block_index,
+ .address = info->address,
+ .value = info->value,
+ };
+ int ret = 0;
+
+ if (!obj)
+ return -EINVAL;
+
+ ret = psp_ras_trigger_error(&adev->psp, &block_info);
+ if (ret)
+ DRM_ERROR("RAS ERROR: inject %s error failed ret %d\n",
+ ras_block_str(info->head.block),
+ ret);
+
+ return ret;
+}
+
+int amdgpu_ras_error_cure(struct amdgpu_device *adev,
+ struct ras_cure_if *info)
+{
+ /* psp fw has no cure interface for now. */
+ return 0;
+}
+
+/* get the total error counts on all IPs */
+int amdgpu_ras_query_error_count(struct amdgpu_device *adev,
+ bool is_ce)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_manager *obj;
+ struct ras_err_data data = {0, 0};
+
+ if (!con)
+ return -EINVAL;
+
+ list_for_each_entry(obj, &con->head, node) {
+ struct ras_query_if info = {
+ .head = obj->head,
+ };
+
+ if (amdgpu_ras_error_query(adev, &info))
+ return -EINVAL;
+
+ data.ce_count += info.ce_count;
+ data.ue_count += info.ue_count;
+ }
+
+ return is_ce ? data.ce_count : data.ue_count;
+}
+/* query/inject/cure end */
+
+
+/* sysfs begin */
+
+static ssize_t amdgpu_ras_sysfs_features_read(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct amdgpu_ras *con =
+ container_of(attr, struct amdgpu_ras, features_attr);
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct amdgpu_device *adev = ddev->dev_private;
+ struct ras_common_if head;
+ int ras_block_count = AMDGPU_RAS_BLOCK_COUNT;
+ int i;
+ ssize_t s;
+ struct ras_manager *obj;
+
+ s = scnprintf(buf, PAGE_SIZE, "feature mask: 0x%x\n", con->features);
+
+ for (i = 0; i < ras_block_count; i++) {
+ head.block = i;
+
+ if (amdgpu_ras_is_feature_enabled(adev, &head)) {
+ obj = amdgpu_ras_find_obj(adev, &head);
+ s += scnprintf(&buf[s], PAGE_SIZE - s,
+ "%s: %s\n",
+ ras_block_str(i),
+ ras_err_str(obj->head.type));
+ } else
+ s += scnprintf(&buf[s], PAGE_SIZE - s,
+ "%s: disabled\n",
+ ras_block_str(i));
+ }
+
+ return s;
+}
+
+static int amdgpu_ras_sysfs_create_feature_node(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct attribute *attrs[] = {
+ &con->features_attr.attr,
+ NULL
+ };
+ struct attribute_group group = {
+ .name = "ras",
+ .attrs = attrs,
+ };
+
+ con->features_attr = (struct device_attribute) {
+ .attr = {
+ .name = "features",
+ .mode = S_IRUGO,
+ },
+ .show = amdgpu_ras_sysfs_features_read,
+ };
+ sysfs_attr_init(attrs[0]);
+
+ return sysfs_create_group(&adev->dev->kobj, &group);
+}
+
+static int amdgpu_ras_sysfs_remove_feature_node(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct attribute *attrs[] = {
+ &con->features_attr.attr,
+ NULL
+ };
+ struct attribute_group group = {
+ .name = "ras",
+ .attrs = attrs,
+ };
+
+ sysfs_remove_group(&adev->dev->kobj, &group);
+
+ return 0;
+}
+
+int amdgpu_ras_sysfs_create(struct amdgpu_device *adev,
+ struct ras_fs_if *head)
+{
+ struct ras_manager *obj = amdgpu_ras_find_obj(adev, &head->head);
+
+ if (!obj || obj->attr_inuse)
+ return -EINVAL;
+
+ get_obj(obj);
+
+ memcpy(obj->fs_data.sysfs_name,
+ head->sysfs_name,
+ sizeof(obj->fs_data.sysfs_name));
+
+ obj->sysfs_attr = (struct device_attribute){
+ .attr = {
+ .name = obj->fs_data.sysfs_name,
+ .mode = S_IRUGO,
+ },
+ .show = amdgpu_ras_sysfs_read,
+ };
+ sysfs_attr_init(&obj->sysfs_attr.attr);
+
+ if (sysfs_add_file_to_group(&adev->dev->kobj,
+ &obj->sysfs_attr.attr,
+ "ras")) {
+ put_obj(obj);
+ return -EINVAL;
+ }
+
+ obj->attr_inuse = 1;
+
+ return 0;
+}
+
+int amdgpu_ras_sysfs_remove(struct amdgpu_device *adev,
+ struct ras_common_if *head)
+{
+ struct ras_manager *obj = amdgpu_ras_find_obj(adev, head);
+
+ if (!obj || !obj->attr_inuse)
+ return -EINVAL;
+
+ sysfs_remove_file_from_group(&adev->dev->kobj,
+ &obj->sysfs_attr.attr,
+ "ras");
+ obj->attr_inuse = 0;
+ put_obj(obj);
+
+ return 0;
+}
+
+static int amdgpu_ras_sysfs_remove_all(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_manager *obj, *tmp;
+
+ list_for_each_entry_safe(obj, tmp, &con->head, node) {
+ amdgpu_ras_sysfs_remove(adev, &obj->head);
+ }
+
+ amdgpu_ras_sysfs_remove_feature_node(adev);
+
+ return 0;
+}
+/* sysfs end */
+
+/* debugfs begin */
+static int amdgpu_ras_debugfs_create_ctrl_node(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct drm_minor *minor = adev->ddev->primary;
+ struct dentry *root = minor->debugfs_root, *dir;
+ struct dentry *ent;
+
+ dir = debugfs_create_dir("ras", root);
+ if (IS_ERR(dir))
+ return -EINVAL;
+
+ con->dir = dir;
+
+ ent = debugfs_create_file("ras_ctrl",
+ S_IWUGO | S_IRUGO, con->dir,
+ adev, &amdgpu_ras_debugfs_ctrl_ops);
+ if (IS_ERR(ent)) {
+ debugfs_remove(con->dir);
+ return -EINVAL;
+ }
+
+ con->ent = ent;
+ return 0;
+}
+
+int amdgpu_ras_debugfs_create(struct amdgpu_device *adev,
+ struct ras_fs_if *head)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_manager *obj = amdgpu_ras_find_obj(adev, &head->head);
+ struct dentry *ent;
+
+ if (!obj || obj->ent)
+ return -EINVAL;
+
+ get_obj(obj);
+
+ memcpy(obj->fs_data.debugfs_name,
+ head->debugfs_name,
+ sizeof(obj->fs_data.debugfs_name));
+
+ ent = debugfs_create_file(obj->fs_data.debugfs_name,
+ S_IWUGO | S_IRUGO, con->dir,
+ obj, &amdgpu_ras_debugfs_ops);
+
+ if (IS_ERR(ent))
+ return -EINVAL;
+
+ obj->ent = ent;
+
+ return 0;
+}
+
+int amdgpu_ras_debugfs_remove(struct amdgpu_device *adev,
+ struct ras_common_if *head)
+{
+ struct ras_manager *obj = amdgpu_ras_find_obj(adev, head);
+
+ if (!obj || !obj->ent)
+ return 0;
+
+ debugfs_remove(obj->ent);
+ obj->ent = NULL;
+ put_obj(obj);
+
+ return 0;
+}
+
+static int amdgpu_ras_debugfs_remove_all(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_manager *obj, *tmp;
+
+ list_for_each_entry_safe(obj, tmp, &con->head, node) {
+ amdgpu_ras_debugfs_remove(adev, &obj->head);
+ }
+
+ debugfs_remove(con->ent);
+ debugfs_remove(con->dir);
+ con->dir = NULL;
+ con->ent = NULL;
+
+ return 0;
+}
+/* debugfs end */
+
+/* ras fs */
+
+static int amdgpu_ras_fs_init(struct amdgpu_device *adev)
+{
+ amdgpu_ras_sysfs_create_feature_node(adev);
+ amdgpu_ras_debugfs_create_ctrl_node(adev);
+
+ return 0;
+}
+
+static int amdgpu_ras_fs_fini(struct amdgpu_device *adev)
+{
+ amdgpu_ras_debugfs_remove_all(adev);
+ amdgpu_ras_sysfs_remove_all(adev);
+ return 0;
+}
+/* ras fs end */
+
+/* ih begin */
+static void amdgpu_ras_interrupt_handler(struct ras_manager *obj)
+{
+ struct ras_ih_data *data = &obj->ih_data;
+ struct amdgpu_iv_entry entry;
+ int ret;
+
+ while (data->rptr != data->wptr) {
+ rmb();
+ memcpy(&entry, &data->ring[data->rptr],
+ data->element_size);
+
+ wmb();
+ data->rptr = (data->aligned_element_size +
+ data->rptr) % data->ring_size;
+
+ /* Let IP handle its data, maybe we need get the output
+ * from the callback to udpate the error type/count, etc
+ */
+ if (data->cb) {
+ ret = data->cb(obj->adev, &entry);
+ /* ue will trigger an interrupt, and in that case
+ * we need do a reset to recovery the whole system.
+ * But leave IP do that recovery, here we just dispatch
+ * the error.
+ */
+ if (ret == AMDGPU_RAS_UE) {
+ obj->err_data.ue_count++;
+ }
+ /* Might need get ce count by register, but not all IP
+ * saves ce count, some IP just use one bit or two bits
+ * to indicate ce happened.
+ */
+ }
+ }
+}
+
+static void amdgpu_ras_interrupt_process_handler(struct work_struct *work)
+{
+ struct ras_ih_data *data =
+ container_of(work, struct ras_ih_data, ih_work);
+ struct ras_manager *obj =
+ container_of(data, struct ras_manager, ih_data);
+
+ amdgpu_ras_interrupt_handler(obj);
+}
+
+int amdgpu_ras_interrupt_dispatch(struct amdgpu_device *adev,
+ struct ras_dispatch_if *info)
+{
+ struct ras_manager *obj = amdgpu_ras_find_obj(adev, &info->head);
+ struct ras_ih_data *data = &obj->ih_data;
+
+ if (!obj)
+ return -EINVAL;
+
+ if (data->inuse == 0)
+ return 0;
+
+ /* Might be overflow... */
+ memcpy(&data->ring[data->wptr], info->entry,
+ data->element_size);
+
+ wmb();
+ data->wptr = (data->aligned_element_size +
+ data->wptr) % data->ring_size;
+
+ schedule_work(&data->ih_work);
+
+ return 0;
+}
+
+int amdgpu_ras_interrupt_remove_handler(struct amdgpu_device *adev,
+ struct ras_ih_if *info)
+{
+ struct ras_manager *obj = amdgpu_ras_find_obj(adev, &info->head);
+ struct ras_ih_data *data;
+
+ if (!obj)
+ return -EINVAL;
+
+ data = &obj->ih_data;
+ if (data->inuse == 0)
+ return 0;
+
+ cancel_work_sync(&data->ih_work);
+
+ kfree(data->ring);
+ memset(data, 0, sizeof(*data));
+ put_obj(obj);
+
+ return 0;
+}
+
+int amdgpu_ras_interrupt_add_handler(struct amdgpu_device *adev,
+ struct ras_ih_if *info)
+{
+ struct ras_manager *obj = amdgpu_ras_find_obj(adev, &info->head);
+ struct ras_ih_data *data;
+
+ if (!obj) {
+ /* in case we registe the IH before enable ras feature */
+ obj = amdgpu_ras_create_obj(adev, &info->head);
+ if (!obj)
+ return -EINVAL;
+ } else
+ get_obj(obj);
+
+ data = &obj->ih_data;
+ /* add the callback.etc */
+ *data = (struct ras_ih_data) {
+ .inuse = 0,
+ .cb = info->cb,
+ .element_size = sizeof(struct amdgpu_iv_entry),
+ .rptr = 0,
+ .wptr = 0,
+ };
+
+ INIT_WORK(&data->ih_work, amdgpu_ras_interrupt_process_handler);
+
+ data->aligned_element_size = ALIGN(data->element_size, 8);
+ /* the ring can store 64 iv entries. */
+ data->ring_size = 64 * data->aligned_element_size;
+ data->ring = kmalloc(data->ring_size, GFP_KERNEL);
+ if (!data->ring) {
+ put_obj(obj);
+ return -ENOMEM;
+ }
+
+ /* IH is ready */
+ data->inuse = 1;
+
+ return 0;
+}
+
+static int amdgpu_ras_interrupt_remove_all(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_manager *obj, *tmp;
+
+ list_for_each_entry_safe(obj, tmp, &con->head, node) {
+ struct ras_ih_if info = {
+ .head = obj->head,
+ };
+ amdgpu_ras_interrupt_remove_handler(adev, &info);
+ }
+
+ return 0;
+}
+/* ih end */
+
+/* recovery begin */
+static void amdgpu_ras_do_recovery(struct work_struct *work)
+{
+ struct amdgpu_ras *ras =
+ container_of(work, struct amdgpu_ras, recovery_work);
+
+ amdgpu_device_gpu_recover(ras->adev, 0);
+ atomic_set(&ras->in_recovery, 0);
+}
+
+static int amdgpu_ras_release_vram(struct amdgpu_device *adev,
+ struct amdgpu_bo **bo_ptr)
+{
+ /* no need to free it actually. */
+ amdgpu_bo_free_kernel(bo_ptr, NULL, NULL);
+ return 0;
+}
+
+/* reserve vram with size@offset */
+static int amdgpu_ras_reserve_vram(struct amdgpu_device *adev,
+ uint64_t offset, uint64_t size,
+ struct amdgpu_bo **bo_ptr)
+{
+ struct ttm_operation_ctx ctx = { false, false };
+ struct amdgpu_bo_param bp;
+ int r = 0;
+ int i;
+ struct amdgpu_bo *bo;
+
+ if (bo_ptr)
+ *bo_ptr = NULL;
+ memset(&bp, 0, sizeof(bp));
+ bp.size = size;
+ bp.byte_align = PAGE_SIZE;
+ bp.domain = AMDGPU_GEM_DOMAIN_VRAM;
+ bp.flags = AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS |
+ AMDGPU_GEM_CREATE_NO_CPU_ACCESS;
+ bp.type = ttm_bo_type_kernel;
+ bp.resv = NULL;
+
+ r = amdgpu_bo_create(adev, &bp, &bo);
+ if (r)
+ return -EINVAL;
+
+ r = amdgpu_bo_reserve(bo, false);
+ if (r)
+ goto error_reserve;
+
+ offset = ALIGN(offset, PAGE_SIZE);
+ for (i = 0; i < bo->placement.num_placement; ++i) {
+ bo->placements[i].fpfn = offset >> PAGE_SHIFT;
+ bo->placements[i].lpfn = (offset + size) >> PAGE_SHIFT;
+ }
+
+ ttm_bo_mem_put(&bo->tbo, &bo->tbo.mem);
+ r = ttm_bo_mem_space(&bo->tbo, &bo->placement, &bo->tbo.mem, &ctx);
+ if (r)
+ goto error_pin;
+
+ r = amdgpu_bo_pin_restricted(bo,
+ AMDGPU_GEM_DOMAIN_VRAM,
+ offset,
+ offset + size);
+ if (r)
+ goto error_pin;
+
+ if (bo_ptr)
+ *bo_ptr = bo;
+
+ amdgpu_bo_unreserve(bo);
+ return r;
+
+error_pin:
+ amdgpu_bo_unreserve(bo);
+error_reserve:
+ amdgpu_bo_unref(&bo);
+ return r;
+}
+
+/* alloc/realloc bps array */
+static int amdgpu_ras_realloc_eh_data_space(struct amdgpu_device *adev,
+ struct ras_err_handler_data *data, int pages)
+{
+ unsigned int old_space = data->count + data->space_left;
+ unsigned int new_space = old_space + pages;
+ unsigned int align_space = ALIGN(new_space, 1024);
+ void *tmp = kmalloc(align_space * sizeof(*data->bps), GFP_KERNEL);
+
+ if (!tmp)
+ return -ENOMEM;
+
+ if (data->bps) {
+ memcpy(tmp, data->bps,
+ data->count * sizeof(*data->bps));
+ kfree(data->bps);
+ }
+
+ data->bps = tmp;
+ data->space_left += align_space - old_space;
+ return 0;
+}
+
+/* it deal with vram only. */
+int amdgpu_ras_add_bad_pages(struct amdgpu_device *adev,
+ unsigned long *bps, int pages)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_err_handler_data *data;
+ int i = pages;
+ int ret = 0;
+
+ if (!con || !con->eh_data || !bps || pages <= 0)
+ return 0;
+
+ mutex_lock(&con->recovery_lock);
+ data = con->eh_data;
+ if (!data)
+ goto out;
+
+ if (data->space_left <= pages)
+ if (amdgpu_ras_realloc_eh_data_space(adev, data, pages)) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ while (i--)
+ data->bps[data->count++].bp = bps[i];
+
+ data->space_left -= pages;
+out:
+ mutex_unlock(&con->recovery_lock);
+
+ return ret;
+}
+
+/* called in gpu recovery/init */
+int amdgpu_ras_reserve_bad_pages(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_err_handler_data *data;
+ uint64_t bp;
+ struct amdgpu_bo *bo;
+ int i;
+
+ if (!con || !con->eh_data)
+ return 0;
+
+ mutex_lock(&con->recovery_lock);
+ data = con->eh_data;
+ if (!data)
+ goto out;
+ /* reserve vram at driver post stage. */
+ for (i = data->last_reserved; i < data->count; i++) {
+ bp = data->bps[i].bp;
+
+ if (amdgpu_ras_reserve_vram(adev, bp << PAGE_SHIFT,
+ PAGE_SIZE, &bo))
+ DRM_ERROR("RAS ERROR: reserve vram %llx fail\n", bp);
+
+ data->bps[i].bo = bo;
+ data->last_reserved = i + 1;
+ }
+out:
+ mutex_unlock(&con->recovery_lock);
+ return 0;
+}
+
+/* called when driver unload */
+static int amdgpu_ras_release_bad_pages(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_err_handler_data *data;
+ struct amdgpu_bo *bo;
+ int i;
+
+ if (!con || !con->eh_data)
+ return 0;
+
+ mutex_lock(&con->recovery_lock);
+ data = con->eh_data;
+ if (!data)
+ goto out;
+
+ for (i = data->last_reserved - 1; i >= 0; i--) {
+ bo = data->bps[i].bo;
+
+ amdgpu_ras_release_vram(adev, &bo);
+
+ data->bps[i].bo = bo;
+ data->last_reserved = i;
+ }
+out:
+ mutex_unlock(&con->recovery_lock);
+ return 0;
+}
+
+static int amdgpu_ras_save_bad_pages(struct amdgpu_device *adev)
+{
+ /* TODO
+ * write the array to eeprom when SMU disabled.
+ */
+ return 0;
+}
+
+static int amdgpu_ras_load_bad_pages(struct amdgpu_device *adev)
+{
+ /* TODO
+ * read the array to eeprom when SMU disabled.
+ */
+ return 0;
+}
+
+static int amdgpu_ras_recovery_init(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_err_handler_data **data = &con->eh_data;
+
+ *data = kmalloc(sizeof(**data),
+ GFP_KERNEL|__GFP_ZERO);
+ if (!*data)
+ return -ENOMEM;
+
+ mutex_init(&con->recovery_lock);
+ INIT_WORK(&con->recovery_work, amdgpu_ras_do_recovery);
+ atomic_set(&con->in_recovery, 0);
+ con->adev = adev;
+
+ amdgpu_ras_load_bad_pages(adev);
+ amdgpu_ras_reserve_bad_pages(adev);
+
+ return 0;
+}
+
+static int amdgpu_ras_recovery_fini(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_err_handler_data *data = con->eh_data;
+
+ cancel_work_sync(&con->recovery_work);
+ amdgpu_ras_save_bad_pages(adev);
+ amdgpu_ras_release_bad_pages(adev);
+
+ mutex_lock(&con->recovery_lock);
+ con->eh_data = NULL;
+ kfree(data->bps);
+ kfree(data);
+ mutex_unlock(&con->recovery_lock);
+
+ return 0;
+}
+/* recovery end */
+
+/*
+ * check hardware's ras ability which will be saved in hw_supported.
+ * if hardware does not support ras, we can skip some ras initializtion and
+ * forbid some ras operations from IP.
+ * if software itself, say boot parameter, limit the ras ability. We still
+ * need allow IP do some limited operations, like disable. In such case,
+ * we have to initialize ras as normal. but need check if operation is
+ * allowed or not in each function.
+ */
+static void amdgpu_ras_check_supported(struct amdgpu_device *adev,
+ uint32_t *hw_supported, uint32_t *supported)
+{
+ *hw_supported = 0;
+ *supported = 0;
+
+ if (amdgpu_sriov_vf(adev) ||
+ adev->asic_type != CHIP_VEGA20)
+ return;
+
+ if (adev->is_atom_fw &&
+ (amdgpu_atomfirmware_mem_ecc_supported(adev) ||
+ amdgpu_atomfirmware_sram_ecc_supported(adev)))
+ *hw_supported = AMDGPU_RAS_BLOCK_MASK;
+
+ *supported = amdgpu_ras_enable == 0 ?
+ 0 : *hw_supported & amdgpu_ras_mask;
+}
+
+int amdgpu_ras_init(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+
+ if (con)
+ return 0;
+
+ con = kmalloc(sizeof(struct amdgpu_ras) +
+ sizeof(struct ras_manager) * AMDGPU_RAS_BLOCK_COUNT,
+ GFP_KERNEL|__GFP_ZERO);
+ if (!con)
+ return -ENOMEM;
+
+ con->objs = (struct ras_manager *)(con + 1);
+
+ amdgpu_ras_set_context(adev, con);
+
+ amdgpu_ras_check_supported(adev, &con->hw_supported,
+ &con->supported);
+ con->features = 0;
+ INIT_LIST_HEAD(&con->head);
+ /* Might need get this flag from vbios. */
+ con->flags = RAS_DEFAULT_FLAGS;
+
+ if (amdgpu_ras_recovery_init(adev))
+ goto recovery_out;
+
+ amdgpu_ras_mask &= AMDGPU_RAS_BLOCK_MASK;
+
+ if (amdgpu_ras_fs_init(adev))
+ goto fs_out;
+
+ amdgpu_ras_self_test(adev);
+
+ DRM_INFO("RAS INFO: ras initialized successfully, "
+ "hardware ability[%x] ras_mask[%x]\n",
+ con->hw_supported, con->supported);
+ return 0;
+fs_out:
+ amdgpu_ras_recovery_fini(adev);
+recovery_out:
+ amdgpu_ras_set_context(adev, NULL);
+ kfree(con);
+
+ return -EINVAL;
+}
+
+/* do some init work after IP late init as dependence */
+void amdgpu_ras_post_init(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct ras_manager *obj, *tmp;
+
+ if (!con)
+ return;
+
+ if (con->flags & AMDGPU_RAS_FLAG_INIT_BY_VBIOS) {
+ /* Set up all other IPs which are not implemented. There is a
+ * tricky thing that IP's actual ras error type should be
+ * MULTI_UNCORRECTABLE, but as driver does not handle it, so
+ * ERROR_NONE make sense anyway.
+ */
+ amdgpu_ras_enable_all_features(adev, 1);
+
+ /* We enable ras on all hw_supported block, but as boot
+ * parameter might disable some of them and one or more IP has
+ * not implemented yet. So we disable them on behalf.
+ */
+ list_for_each_entry_safe(obj, tmp, &con->head, node) {
+ if (!amdgpu_ras_is_supported(adev, obj->head.block)) {
+ amdgpu_ras_feature_enable(adev, &obj->head, 0);
+ /* there should be no any reference. */
+ WARN_ON(alive_obj(obj));
+ }
+ }
+ }
+}
+
+/* do some fini work before IP fini as dependence */
+int amdgpu_ras_pre_fini(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+
+ if (!con)
+ return 0;
+
+ /* Need disable ras on all IPs here before ip [hw/sw]fini */
+ amdgpu_ras_disable_all_features(adev, 0);
+ amdgpu_ras_recovery_fini(adev);
+ return 0;
+}
+
+int amdgpu_ras_fini(struct amdgpu_device *adev)
+{
+ struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+
+ if (!con)
+ return 0;
+
+ amdgpu_ras_fs_fini(adev);
+ amdgpu_ras_interrupt_remove_all(adev);
+
+ WARN(con->features, "Feature mask is not cleared");
+
+ if (con->features)
+ amdgpu_ras_disable_all_features(adev, 1);
+
+ amdgpu_ras_set_context(adev, NULL);
+ kfree(con);
+
+ return 0;
+}
--- /dev/null
+/*
+ * Copyright 2018 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ *
+ */
+#ifndef _AMDGPU_RAS_H
+#define _AMDGPU_RAS_H
+
+#include <linux/debugfs.h>
+#include <linux/list.h>
+#include "amdgpu.h"
+#include "amdgpu_psp.h"
+#include "ta_ras_if.h"
+
+enum amdgpu_ras_block {
+ AMDGPU_RAS_BLOCK__UMC = 0,
+ AMDGPU_RAS_BLOCK__SDMA,
+ AMDGPU_RAS_BLOCK__GFX,
+ AMDGPU_RAS_BLOCK__MMHUB,
+ AMDGPU_RAS_BLOCK__ATHUB,
+ AMDGPU_RAS_BLOCK__PCIE_BIF,
+ AMDGPU_RAS_BLOCK__HDP,
+ AMDGPU_RAS_BLOCK__XGMI_WAFL,
+ AMDGPU_RAS_BLOCK__DF,
+ AMDGPU_RAS_BLOCK__SMN,
+ AMDGPU_RAS_BLOCK__SEM,
+ AMDGPU_RAS_BLOCK__MP0,
+ AMDGPU_RAS_BLOCK__MP1,
+ AMDGPU_RAS_BLOCK__FUSE,
+
+ AMDGPU_RAS_BLOCK__LAST
+};
+
+#define AMDGPU_RAS_BLOCK_COUNT AMDGPU_RAS_BLOCK__LAST
+#define AMDGPU_RAS_BLOCK_MASK ((1ULL << AMDGPU_RAS_BLOCK_COUNT) - 1)
+
+enum amdgpu_ras_error_type {
+ AMDGPU_RAS_ERROR__NONE = 0,
+ AMDGPU_RAS_ERROR__PARITY = 1,
+ AMDGPU_RAS_ERROR__SINGLE_CORRECTABLE = 2,
+ AMDGPU_RAS_ERROR__MULTI_UNCORRECTABLE = 4,
+ AMDGPU_RAS_ERROR__POISON = 8,
+};
+
+enum amdgpu_ras_ret {
+ AMDGPU_RAS_SUCCESS = 0,
+ AMDGPU_RAS_FAIL,
+ AMDGPU_RAS_UE,
+ AMDGPU_RAS_CE,
+ AMDGPU_RAS_PT,
+};
+
+struct ras_common_if {
+ enum amdgpu_ras_block block;
+ enum amdgpu_ras_error_type type;
+ uint32_t sub_block_index;
+ /* block name */
+ char name[32];
+};
+
+typedef int (*ras_ih_cb)(struct amdgpu_device *adev,
+ struct amdgpu_iv_entry *entry);
+
+struct amdgpu_ras {
+ /* ras infrastructure */
+ /* for ras itself. */
+ uint32_t hw_supported;
+ /* for IP to check its ras ability. */
+ uint32_t supported;
+ uint32_t features;
+ struct list_head head;
+ /* debugfs */
+ struct dentry *dir;
+ /* debugfs ctrl */
+ struct dentry *ent;
+ /* sysfs */
+ struct device_attribute features_attr;
+ /* block array */
+ struct ras_manager *objs;
+
+ /* gpu recovery */
+ struct work_struct recovery_work;
+ atomic_t in_recovery;
+ struct amdgpu_device *adev;
+ /* error handler data */
+ struct ras_err_handler_data *eh_data;
+ struct mutex recovery_lock;
+
+ uint32_t flags;
+};
+
+/* interfaces for IP */
+
+struct ras_fs_if {
+ struct ras_common_if head;
+ char sysfs_name[32];
+ char debugfs_name[32];
+};
+
+struct ras_query_if {
+ struct ras_common_if head;
+ unsigned long ue_count;
+ unsigned long ce_count;
+};
+
+struct ras_inject_if {
+ struct ras_common_if head;
+ uint64_t address;
+ uint64_t value;
+};
+
+struct ras_cure_if {
+ struct ras_common_if head;
+ uint64_t address;
+};
+
+struct ras_ih_if {
+ struct ras_common_if head;
+ ras_ih_cb cb;
+};
+
+struct ras_dispatch_if {
+ struct ras_common_if head;
+ struct amdgpu_iv_entry *entry;
+};
+
+struct ras_debug_if {
+ union {
+ struct ras_common_if head;
+ struct ras_inject_if inject;
+ };
+ int op;
+};
+/* work flow
+ * vbios
+ * 1: ras feature enable (enabled by default)
+ * psp
+ * 2: ras framework init (in ip_init)
+ * IP
+ * 3: IH add
+ * 4: debugfs/sysfs create
+ * 5: query/inject
+ * 6: debugfs/sysfs remove
+ * 7: IH remove
+ * 8: feature disable
+ */
+
+#define amdgpu_ras_get_context(adev) ((adev)->psp.ras.ras)
+#define amdgpu_ras_set_context(adev, ras_con) ((adev)->psp.ras.ras = (ras_con))
+
+/* check if ras is supported on block, say, sdma, gfx */
+static inline int amdgpu_ras_is_supported(struct amdgpu_device *adev,
+ unsigned int block)
+{
+ struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
+
+ return ras && (ras->supported & (1 << block));
+}
+
+int amdgpu_ras_query_error_count(struct amdgpu_device *adev,
+ bool is_ce);
+
+/* error handling functions */
+int amdgpu_ras_add_bad_pages(struct amdgpu_device *adev,
+ unsigned long *bps, int pages);
+
+int amdgpu_ras_reserve_bad_pages(struct amdgpu_device *adev);
+
+static inline int amdgpu_ras_reset_gpu(struct amdgpu_device *adev,
+ bool is_baco)
+{
+ /* remove me when gpu reset works on vega20 A1. */
+#if 0
+ struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
+
+ if (atomic_cmpxchg(&ras->in_recovery, 0, 1) == 0)
+ schedule_work(&ras->recovery_work);
+#endif
+ return 0;
+}
+
+static inline enum ta_ras_block
+amdgpu_ras_block_to_ta(enum amdgpu_ras_block block) {
+ switch (block) {
+ case AMDGPU_RAS_BLOCK__UMC:
+ return TA_RAS_BLOCK__UMC;
+ case AMDGPU_RAS_BLOCK__SDMA:
+ return TA_RAS_BLOCK__SDMA;
+ case AMDGPU_RAS_BLOCK__GFX:
+ return TA_RAS_BLOCK__GFX;
+ case AMDGPU_RAS_BLOCK__MMHUB:
+ return TA_RAS_BLOCK__MMHUB;
+ case AMDGPU_RAS_BLOCK__ATHUB:
+ return TA_RAS_BLOCK__ATHUB;
+ case AMDGPU_RAS_BLOCK__PCIE_BIF:
+ return TA_RAS_BLOCK__PCIE_BIF;
+ case AMDGPU_RAS_BLOCK__HDP:
+ return TA_RAS_BLOCK__HDP;
+ case AMDGPU_RAS_BLOCK__XGMI_WAFL:
+ return TA_RAS_BLOCK__XGMI_WAFL;
+ case AMDGPU_RAS_BLOCK__DF:
+ return TA_RAS_BLOCK__DF;
+ case AMDGPU_RAS_BLOCK__SMN:
+ return TA_RAS_BLOCK__SMN;
+ case AMDGPU_RAS_BLOCK__SEM:
+ return TA_RAS_BLOCK__SEM;
+ case AMDGPU_RAS_BLOCK__MP0:
+ return TA_RAS_BLOCK__MP0;
+ case AMDGPU_RAS_BLOCK__MP1:
+ return TA_RAS_BLOCK__MP1;
+ case AMDGPU_RAS_BLOCK__FUSE:
+ return TA_RAS_BLOCK__FUSE;
+ default:
+ WARN_ONCE(1, "RAS ERROR: unexpected block id %d\n", block);
+ return TA_RAS_BLOCK__UMC;
+ }
+}
+
+static inline enum ta_ras_error_type
+amdgpu_ras_error_to_ta(enum amdgpu_ras_error_type error) {
+ switch (error) {
+ case AMDGPU_RAS_ERROR__NONE:
+ return TA_RAS_ERROR__NONE;
+ case AMDGPU_RAS_ERROR__PARITY:
+ return TA_RAS_ERROR__PARITY;
+ case AMDGPU_RAS_ERROR__SINGLE_CORRECTABLE:
+ return TA_RAS_ERROR__SINGLE_CORRECTABLE;
+ case AMDGPU_RAS_ERROR__MULTI_UNCORRECTABLE:
+ return TA_RAS_ERROR__MULTI_UNCORRECTABLE;
+ case AMDGPU_RAS_ERROR__POISON:
+ return TA_RAS_ERROR__POISON;
+ default:
+ WARN_ONCE(1, "RAS ERROR: unexpected error type %d\n", error);
+ return TA_RAS_ERROR__NONE;
+ }
+}
+
+/* called in ip_init and ip_fini */
+int amdgpu_ras_init(struct amdgpu_device *adev);
+void amdgpu_ras_post_init(struct amdgpu_device *adev);
+int amdgpu_ras_fini(struct amdgpu_device *adev);
+int amdgpu_ras_pre_fini(struct amdgpu_device *adev);
+
+int amdgpu_ras_feature_enable(struct amdgpu_device *adev,
+ struct ras_common_if *head, bool enable);
+
+int amdgpu_ras_feature_enable_on_boot(struct amdgpu_device *adev,
+ struct ras_common_if *head, bool enable);
+
+int amdgpu_ras_sysfs_create(struct amdgpu_device *adev,
+ struct ras_fs_if *head);
+
+int amdgpu_ras_sysfs_remove(struct amdgpu_device *adev,
+ struct ras_common_if *head);
+
+int amdgpu_ras_debugfs_create(struct amdgpu_device *adev,
+ struct ras_fs_if *head);
+
+int amdgpu_ras_debugfs_remove(struct amdgpu_device *adev,
+ struct ras_common_if *head);
+
+int amdgpu_ras_error_query(struct amdgpu_device *adev,
+ struct ras_query_if *info);
+
+int amdgpu_ras_error_inject(struct amdgpu_device *adev,
+ struct ras_inject_if *info);
+
+int amdgpu_ras_interrupt_add_handler(struct amdgpu_device *adev,
+ struct ras_ih_if *info);
+
+int amdgpu_ras_interrupt_remove_handler(struct amdgpu_device *adev,
+ struct ras_ih_if *info);
+
+int amdgpu_ras_interrupt_dispatch(struct amdgpu_device *adev,
+ struct ras_dispatch_if *info);
+#endif
*/
if (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
sched_hw_submission = max(sched_hw_submission, 256);
+ else if (ring == &adev->sdma.instance[0].page)
+ sched_hw_submission = 256;
if (ring->adev == NULL) {
if (adev->num_rings >= AMDGPU_MAX_RINGS)
#define AMDGPU_MAX_SDMA_INSTANCES 2
enum amdgpu_sdma_irq {
- AMDGPU_SDMA_IRQ_TRAP0 = 0,
- AMDGPU_SDMA_IRQ_TRAP1,
-
+ AMDGPU_SDMA_IRQ_INSTANCE0 = 0,
+ AMDGPU_SDMA_IRQ_INSTANCE1,
AMDGPU_SDMA_IRQ_LAST
};
struct amdgpu_sdma_instance instance[AMDGPU_MAX_SDMA_INSTANCES];
struct amdgpu_irq_src trap_irq;
struct amdgpu_irq_src illegal_inst_irq;
+ struct amdgpu_irq_src ecc_irq;
int num_instances;
uint32_t srbm_soft_reset;
bool has_page_queue;
+ struct ras_common_if *ras_if;
};
/*
#include "amdgpu_sdma.h"
#include "bif/bif_4_1_d.h"
-#define DRM_FILE_PAGE_OFFSET (0x100000000ULL >> PAGE_SHIFT)
-
static int amdgpu_map_buffer(struct ttm_buffer_object *bo,
struct ttm_mem_reg *mem, unsigned num_pages,
uint64_t offset, unsigned window,
struct dma_fence *f;
int i;
+ /* Don't evict VM page tables while they are busy, otherwise we can't
+ * cleanly handle page faults.
+ */
+ if (bo->type == ttm_bo_type_kernel &&
+ !reservation_object_test_signaled_rcu(bo->resv, true))
+ return false;
+
/* If bo is a KFD BO, check if the bo belongs to the current process.
* If true, then return false as any KFD process needs all its BOs to
* be resident to run successfully
r = ttm_bo_device_init(&adev->mman.bdev,
&amdgpu_bo_driver,
adev->ddev->anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET,
adev->need_dma32);
if (r) {
DRM_ERROR("failed initializing buffer object driver(%d).\n", r);
int amdgpu_mmap(struct file *filp, struct vm_area_struct *vma)
{
- struct drm_file *file_priv;
- struct amdgpu_device *adev;
-
- if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET))
- return -EINVAL;
+ struct drm_file *file_priv = filp->private_data;
+ struct amdgpu_device *adev = file_priv->minor->dev->dev_private;
- file_priv = filp->private_data;
- adev = file_priv->minor->dev->dev_private;
if (adev == NULL)
return -EINVAL;
/* enable virtual display */
adev->mode_info.num_crtc = 1;
adev->enable_virtual_display = true;
+ adev->ddev->driver->driver_features &= ~DRIVER_ATOMIC;
adev->cg_flags = 0;
adev->pg_flags = 0;
}
}
}
+static uint32_t parse_clk(char *buf, bool min)
+{
+ char *ptr = buf;
+ uint32_t clk = 0;
+
+ do {
+ ptr = strchr(ptr, ':');
+ if (!ptr)
+ break;
+ ptr+=2;
+ clk = simple_strtoul(ptr, NULL, 10);
+ } while (!min);
+
+ return clk * 100;
+}
+
+uint32_t amdgpu_virt_get_sclk(struct amdgpu_device *adev, bool lowest)
+{
+ char *buf = NULL;
+ uint32_t clk = 0;
+
+ buf = kzalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ adev->virt.ops->get_pp_clk(adev, PP_SCLK, buf);
+ clk = parse_clk(buf, lowest);
+
+ kfree(buf);
+
+ return clk;
+}
+
+uint32_t amdgpu_virt_get_mclk(struct amdgpu_device *adev, bool lowest)
+{
+ char *buf = NULL;
+ uint32_t clk = 0;
+
+ buf = kzalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ adev->virt.ops->get_pp_clk(adev, PP_MCLK, buf);
+ clk = parse_clk(buf, lowest);
+
+ kfree(buf);
+
+ return clk;
+}
int (*reset_gpu)(struct amdgpu_device *adev);
int (*wait_reset)(struct amdgpu_device *adev);
void (*trans_msg)(struct amdgpu_device *adev, u32 req, u32 data1, u32 data2, u32 data3);
+ int (*get_pp_clk)(struct amdgpu_device *adev, u32 type, char *buf);
+ int (*force_dpm_level)(struct amdgpu_device *adev, u32 level);
};
/*
AMDGIM_FEATURE_GIM_LOAD_UCODES = 0x2,
/* VRAM LOST by GIM */
AMDGIM_FEATURE_GIM_FLR_VRAMLOST = 0x4,
+ /* HW PERF SIM in GIM */
+ AMDGIM_FEATURE_HW_PERF_SIMULATION = (1 << 3),
};
struct amd_sriov_msg_pf2vf_info_header {
struct amdgpu_vf_error_buffer vf_errors;
struct amdgpu_virt_fw_reserve fw_reserve;
uint32_t gim_feature;
+ /* protect DPM events to GIM */
+ struct mutex dpm_mutex;
};
#define amdgpu_sriov_enabled(adev) \
#endif
}
+#define amdgim_is_hwperf(adev) \
+ ((adev)->virt.gim_feature & AMDGIM_FEATURE_HW_PERF_SIMULATION)
+
bool amdgpu_virt_mmio_blocked(struct amdgpu_device *adev);
void amdgpu_virt_init_setting(struct amdgpu_device *adev);
uint32_t amdgpu_virt_kiq_rreg(struct amdgpu_device *adev, uint32_t reg);
unsigned int key,
unsigned int chksum);
void amdgpu_virt_init_data_exchange(struct amdgpu_device *adev);
+uint32_t amdgpu_virt_get_sclk(struct amdgpu_device *adev, bool lowest);
+uint32_t amdgpu_virt_get_mclk(struct amdgpu_device *adev, bool lowest);
#endif
#include "amdgpu_trace.h"
#include "amdgpu_amdkfd.h"
#include "amdgpu_gmc.h"
+#include "amdgpu_xgmi.h"
/**
* DOC: GPUVM
#undef START
#undef LAST
-/**
- * struct amdgpu_pte_update_params - Local structure
- *
- * Encapsulate some VM table update parameters to reduce
- * the number of function parameters
- *
- */
-struct amdgpu_pte_update_params {
-
- /**
- * @adev: amdgpu device we do this update for
- */
- struct amdgpu_device *adev;
-
- /**
- * @vm: optional amdgpu_vm we do this update for
- */
- struct amdgpu_vm *vm;
-
- /**
- * @src: address where to copy page table entries from
- */
- uint64_t src;
-
- /**
- * @ib: indirect buffer to fill with commands
- */
- struct amdgpu_ib *ib;
-
- /**
- * @func: Function which actually does the update
- */
- void (*func)(struct amdgpu_pte_update_params *params,
- struct amdgpu_bo *bo, uint64_t pe,
- uint64_t addr, unsigned count, uint32_t incr,
- uint64_t flags);
- /**
- * @pages_addr:
- *
- * DMA addresses to use for mapping, used during VM update by CPU
- */
- dma_addr_t *pages_addr;
-};
-
/**
* struct amdgpu_prt_cb - Helper to disable partial resident texture feature from a fence callback
*/
return AMDGPU_VM_PTE_COUNT(adev);
}
+/**
+ * amdgpu_vm_num_ats_entries - return the number of ATS entries in the root PD
+ *
+ * @adev: amdgpu_device pointer
+ *
+ * Returns:
+ * The number of entries in the root page directory which needs the ATS setting.
+ */
+static unsigned amdgpu_vm_num_ats_entries(struct amdgpu_device *adev)
+{
+ unsigned shift;
+
+ shift = amdgpu_vm_level_shift(adev, adev->vm_manager.root_level);
+ return AMDGPU_GMC_HOLE_START >> (shift + AMDGPU_GPU_PAGE_SHIFT);
+}
+
/**
* amdgpu_vm_entries_mask - the mask to get the entry number of a PD/PT
*
return;
vm->bulk_moveable = false;
- if (bo->tbo.type == ttm_bo_type_kernel)
+ if (bo->tbo.type == ttm_bo_type_kernel && bo->parent)
amdgpu_vm_bo_relocated(base);
else
amdgpu_vm_bo_idle(base);
}
/**
- * amdgpu_vm_pt_first_leaf - get first leaf PD/PT
+ * amdgpu_vm_pt_first_dfs - start a deep first search
*
- * @adev: amdgpu_device pointer
+ * @adev: amdgpu_device structure
* @vm: amdgpu_vm structure
- * @start: start addr of the walk
* @cursor: state to initialize
*
- * Start a walk and go directly to the leaf node.
- */
-static void amdgpu_vm_pt_first_leaf(struct amdgpu_device *adev,
- struct amdgpu_vm *vm, uint64_t start,
- struct amdgpu_vm_pt_cursor *cursor)
-{
- amdgpu_vm_pt_start(adev, vm, start, cursor);
- while (amdgpu_vm_pt_descendant(adev, cursor));
-}
-
-/**
- * amdgpu_vm_pt_next_leaf - get next leaf PD/PT
- *
- * @adev: amdgpu_device pointer
- * @cursor: current state
- *
- * Walk the PD/PT tree to the next leaf node.
+ * Starts a deep first traversal of the PD/PT tree.
*/
-static void amdgpu_vm_pt_next_leaf(struct amdgpu_device *adev,
+static void amdgpu_vm_pt_first_dfs(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm,
+ struct amdgpu_vm_pt_cursor *start,
struct amdgpu_vm_pt_cursor *cursor)
{
- amdgpu_vm_pt_next(adev, cursor);
- if (cursor->pfn != ~0ll)
- while (amdgpu_vm_pt_descendant(adev, cursor));
+ if (start)
+ *cursor = *start;
+ else
+ amdgpu_vm_pt_start(adev, vm, 0, cursor);
+ while (amdgpu_vm_pt_descendant(adev, cursor));
}
/**
- * for_each_amdgpu_vm_pt_leaf - walk over all leaf PDs/PTs in the hierarchy
- */
-#define for_each_amdgpu_vm_pt_leaf(adev, vm, start, end, cursor) \
- for (amdgpu_vm_pt_first_leaf((adev), (vm), (start), &(cursor)); \
- (cursor).pfn <= end; amdgpu_vm_pt_next_leaf((adev), &(cursor)))
-
-/**
- * amdgpu_vm_pt_first_dfs - start a deep first search
+ * amdgpu_vm_pt_continue_dfs - check if the deep first search should continue
*
- * @adev: amdgpu_device structure
- * @vm: amdgpu_vm structure
- * @cursor: state to initialize
+ * @start: starting point for the search
+ * @entry: current entry
*
- * Starts a deep first traversal of the PD/PT tree.
+ * Returns:
+ * True when the search should continue, false otherwise.
*/
-static void amdgpu_vm_pt_first_dfs(struct amdgpu_device *adev,
- struct amdgpu_vm *vm,
- struct amdgpu_vm_pt_cursor *cursor)
+static bool amdgpu_vm_pt_continue_dfs(struct amdgpu_vm_pt_cursor *start,
+ struct amdgpu_vm_pt *entry)
{
- amdgpu_vm_pt_start(adev, vm, 0, cursor);
- while (amdgpu_vm_pt_descendant(adev, cursor));
+ return entry && (!start || entry != start->entry);
}
/**
/**
* for_each_amdgpu_vm_pt_dfs_safe - safe deep first search of all PDs/PTs
*/
-#define for_each_amdgpu_vm_pt_dfs_safe(adev, vm, cursor, entry) \
- for (amdgpu_vm_pt_first_dfs((adev), (vm), &(cursor)), \
+#define for_each_amdgpu_vm_pt_dfs_safe(adev, vm, start, cursor, entry) \
+ for (amdgpu_vm_pt_first_dfs((adev), (vm), (start), &(cursor)), \
(entry) = (cursor).entry, amdgpu_vm_pt_next_dfs((adev), &(cursor));\
- (entry); (entry) = (cursor).entry, \
- amdgpu_vm_pt_next_dfs((adev), &(cursor)))
+ amdgpu_vm_pt_continue_dfs((start), (entry)); \
+ (entry) = (cursor).entry, amdgpu_vm_pt_next_dfs((adev), &(cursor)))
/**
* amdgpu_vm_get_pd_bo - add the VM PD to a validation list
if (bo->tbo.type != ttm_bo_type_kernel) {
amdgpu_vm_bo_moved(bo_base);
} else {
- if (vm->use_cpu_for_update)
- r = amdgpu_bo_kmap(bo, NULL);
+ vm->update_funcs->map_table(bo);
+ if (bo->parent)
+ amdgpu_vm_bo_relocated(bo_base);
else
- r = amdgpu_ttm_alloc_gart(&bo->tbo);
- if (r)
- break;
- if (bo->shadow) {
- r = amdgpu_ttm_alloc_gart(&bo->shadow->tbo);
- if (r)
- break;
- }
- amdgpu_vm_bo_relocated(bo_base);
+ amdgpu_vm_bo_idle(bo_base);
}
}
* @adev: amdgpu_device pointer
* @vm: VM to clear BO from
* @bo: BO to clear
- * @level: level this BO is at
- * @pte_support_ats: indicate ATS support from PTE
*
* Root PD needs to be reserved when calling this.
*
* 0 on success, errno otherwise.
*/
static int amdgpu_vm_clear_bo(struct amdgpu_device *adev,
- struct amdgpu_vm *vm, struct amdgpu_bo *bo,
- unsigned level, bool pte_support_ats)
+ struct amdgpu_vm *vm,
+ struct amdgpu_bo *bo)
{
struct ttm_operation_ctx ctx = { true, false };
- struct dma_fence *fence = NULL;
+ unsigned level = adev->vm_manager.root_level;
+ struct amdgpu_vm_update_params params;
+ struct amdgpu_bo *ancestor = bo;
unsigned entries, ats_entries;
- struct amdgpu_ring *ring;
- struct amdgpu_job *job;
uint64_t addr;
int r;
+ /* Figure out our place in the hierarchy */
+ if (ancestor->parent) {
+ ++level;
+ while (ancestor->parent->parent) {
+ ++level;
+ ancestor = ancestor->parent;
+ }
+ }
+
entries = amdgpu_bo_size(bo) / 8;
+ if (!vm->pte_support_ats) {
+ ats_entries = 0;
+
+ } else if (!bo->parent) {
+ ats_entries = amdgpu_vm_num_ats_entries(adev);
+ ats_entries = min(ats_entries, entries);
+ entries -= ats_entries;
+
+ } else {
+ struct amdgpu_vm_pt *pt;
- if (pte_support_ats) {
- if (level == adev->vm_manager.root_level) {
- ats_entries = amdgpu_vm_level_shift(adev, level);
- ats_entries += AMDGPU_GPU_PAGE_SHIFT;
- ats_entries = AMDGPU_GMC_HOLE_START >> ats_entries;
- ats_entries = min(ats_entries, entries);
- entries -= ats_entries;
+ pt = container_of(ancestor->vm_bo, struct amdgpu_vm_pt, base);
+ ats_entries = amdgpu_vm_num_ats_entries(adev);
+ if ((pt - vm->root.entries) >= ats_entries) {
+ ats_entries = 0;
} else {
ats_entries = entries;
entries = 0;
}
- } else {
- ats_entries = 0;
}
- ring = container_of(vm->entity.rq->sched, struct amdgpu_ring, sched);
-
r = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
if (r)
- goto error;
+ return r;
+
+ if (bo->shadow) {
+ r = ttm_bo_validate(&bo->shadow->tbo, &bo->shadow->placement,
+ &ctx);
+ if (r)
+ return r;
+ }
- r = amdgpu_ttm_alloc_gart(&bo->tbo);
+ r = vm->update_funcs->map_table(bo);
if (r)
return r;
- r = amdgpu_job_alloc_with_ib(adev, 64, &job);
+ memset(¶ms, 0, sizeof(params));
+ params.adev = adev;
+ params.vm = vm;
+
+ r = vm->update_funcs->prepare(¶ms, AMDGPU_FENCE_OWNER_KFD, NULL);
if (r)
- goto error;
+ return r;
- addr = amdgpu_bo_gpu_offset(bo);
+ addr = 0;
if (ats_entries) {
- uint64_t ats_value;
+ uint64_t value = 0, flags;
- ats_value = AMDGPU_PTE_DEFAULT_ATC;
- if (level != AMDGPU_VM_PTB)
- ats_value |= AMDGPU_PDE_PTE;
+ flags = AMDGPU_PTE_DEFAULT_ATC;
+ if (level != AMDGPU_VM_PTB) {
+ /* Handle leaf PDEs as PTEs */
+ flags |= AMDGPU_PDE_PTE;
+ amdgpu_gmc_get_vm_pde(adev, level, &value, &flags);
+ }
+
+ r = vm->update_funcs->update(¶ms, bo, addr, 0, ats_entries,
+ value, flags);
+ if (r)
+ return r;
- amdgpu_vm_set_pte_pde(adev, &job->ibs[0], addr, 0,
- ats_entries, 0, ats_value);
addr += ats_entries * 8;
}
if (entries) {
- uint64_t value = 0;
-
- /* Workaround for fault priority problem on GMC9 */
- if (level == AMDGPU_VM_PTB && adev->asic_type >= CHIP_VEGA10)
- value = AMDGPU_PTE_EXECUTABLE;
+ uint64_t value = 0, flags = 0;
+
+ if (adev->asic_type >= CHIP_VEGA10) {
+ if (level != AMDGPU_VM_PTB) {
+ /* Handle leaf PDEs as PTEs */
+ flags |= AMDGPU_PDE_PTE;
+ amdgpu_gmc_get_vm_pde(adev, level,
+ &value, &flags);
+ } else {
+ /* Workaround for fault priority problem on GMC9 */
+ flags = AMDGPU_PTE_EXECUTABLE;
+ }
+ }
- amdgpu_vm_set_pte_pde(adev, &job->ibs[0], addr, 0,
- entries, 0, value);
+ r = vm->update_funcs->update(¶ms, bo, addr, 0, entries,
+ value, flags);
+ if (r)
+ return r;
}
- amdgpu_ring_pad_ib(ring, &job->ibs[0]);
-
- WARN_ON(job->ibs[0].length_dw > 64);
- r = amdgpu_sync_resv(adev, &job->sync, bo->tbo.resv,
- AMDGPU_FENCE_OWNER_KFD, false);
- if (r)
- goto error_free;
-
- r = amdgpu_job_submit(job, &vm->entity, AMDGPU_FENCE_OWNER_UNDEFINED,
- &fence);
- if (r)
- goto error_free;
-
- amdgpu_bo_fence(bo, fence, true);
- dma_fence_put(fence);
-
- if (bo->shadow)
- return amdgpu_vm_clear_bo(adev, vm, bo->shadow,
- level, pte_support_ats);
-
- return 0;
-
-error_free:
- amdgpu_job_free(job);
-
-error:
- return r;
+ return vm->update_funcs->commit(¶ms, NULL);
}
/**
}
/**
- * amdgpu_vm_alloc_pts - Allocate page tables.
+ * amdgpu_vm_alloc_pts - Allocate a specific page table
*
* @adev: amdgpu_device pointer
* @vm: VM to allocate page tables for
- * @saddr: Start address which needs to be allocated
- * @size: Size from start address we need.
+ * @cursor: Which page table to allocate
*
- * Make sure the page directories and page tables are allocated
+ * Make sure a specific page table or directory is allocated.
*
* Returns:
- * 0 on success, errno otherwise.
+ * 1 if page table needed to be allocated, 0 if page table was already
+ * allocated, negative errno if an error occurred.
*/
-int amdgpu_vm_alloc_pts(struct amdgpu_device *adev,
- struct amdgpu_vm *vm,
- uint64_t saddr, uint64_t size)
+static int amdgpu_vm_alloc_pts(struct amdgpu_device *adev,
+ struct amdgpu_vm *vm,
+ struct amdgpu_vm_pt_cursor *cursor)
{
- struct amdgpu_vm_pt_cursor cursor;
+ struct amdgpu_vm_pt *entry = cursor->entry;
+ struct amdgpu_bo_param bp;
struct amdgpu_bo *pt;
- bool ats = false;
- uint64_t eaddr;
int r;
- /* validate the parameters */
- if (saddr & AMDGPU_GPU_PAGE_MASK || size & AMDGPU_GPU_PAGE_MASK)
- return -EINVAL;
-
- eaddr = saddr + size - 1;
-
- if (vm->pte_support_ats)
- ats = saddr < AMDGPU_GMC_HOLE_START;
-
- saddr /= AMDGPU_GPU_PAGE_SIZE;
- eaddr /= AMDGPU_GPU_PAGE_SIZE;
+ if (cursor->level < AMDGPU_VM_PTB && !entry->entries) {
+ unsigned num_entries;
- if (eaddr >= adev->vm_manager.max_pfn) {
- dev_err(adev->dev, "va above limit (0x%08llX >= 0x%08llX)\n",
- eaddr, adev->vm_manager.max_pfn);
- return -EINVAL;
+ num_entries = amdgpu_vm_num_entries(adev, cursor->level);
+ entry->entries = kvmalloc_array(num_entries,
+ sizeof(*entry->entries),
+ GFP_KERNEL | __GFP_ZERO);
+ if (!entry->entries)
+ return -ENOMEM;
}
- for_each_amdgpu_vm_pt_leaf(adev, vm, saddr, eaddr, cursor) {
- struct amdgpu_vm_pt *entry = cursor.entry;
- struct amdgpu_bo_param bp;
-
- if (cursor.level < AMDGPU_VM_PTB) {
- unsigned num_entries;
-
- num_entries = amdgpu_vm_num_entries(adev, cursor.level);
- entry->entries = kvmalloc_array(num_entries,
- sizeof(*entry->entries),
- GFP_KERNEL |
- __GFP_ZERO);
- if (!entry->entries)
- return -ENOMEM;
- }
-
-
- if (entry->base.bo)
- continue;
-
- amdgpu_vm_bo_param(adev, vm, cursor.level, &bp);
+ if (entry->base.bo)
+ return 0;
- r = amdgpu_bo_create(adev, &bp, &pt);
- if (r)
- return r;
+ amdgpu_vm_bo_param(adev, vm, cursor->level, &bp);
- if (vm->use_cpu_for_update) {
- r = amdgpu_bo_kmap(pt, NULL);
- if (r)
- goto error_free_pt;
- }
-
- /* Keep a reference to the root directory to avoid
- * freeing them up in the wrong order.
- */
- pt->parent = amdgpu_bo_ref(cursor.parent->base.bo);
+ r = amdgpu_bo_create(adev, &bp, &pt);
+ if (r)
+ return r;
- amdgpu_vm_bo_base_init(&entry->base, vm, pt);
+ /* Keep a reference to the root directory to avoid
+ * freeing them up in the wrong order.
+ */
+ pt->parent = amdgpu_bo_ref(cursor->parent->base.bo);
+ amdgpu_vm_bo_base_init(&entry->base, vm, pt);
- r = amdgpu_vm_clear_bo(adev, vm, pt, cursor.level, ats);
- if (r)
- goto error_free_pt;
- }
+ r = amdgpu_vm_clear_bo(adev, vm, pt);
+ if (r)
+ goto error_free_pt;
return 0;
return r;
}
+/**
+ * amdgpu_vm_free_table - fre one PD/PT
+ *
+ * @entry: PDE to free
+ */
+static void amdgpu_vm_free_table(struct amdgpu_vm_pt *entry)
+{
+ if (entry->base.bo) {
+ entry->base.bo->vm_bo = NULL;
+ list_del(&entry->base.vm_status);
+ amdgpu_bo_unref(&entry->base.bo->shadow);
+ amdgpu_bo_unref(&entry->base.bo);
+ }
+ kvfree(entry->entries);
+ entry->entries = NULL;
+}
+
/**
* amdgpu_vm_free_pts - free PD/PT levels
*
* @adev: amdgpu device structure
* @vm: amdgpu vm structure
+ * @start: optional cursor where to start freeing PDs/PTs
*
* Free the page directory or page table level and all sub levels.
*/
static void amdgpu_vm_free_pts(struct amdgpu_device *adev,
- struct amdgpu_vm *vm)
+ struct amdgpu_vm *vm,
+ struct amdgpu_vm_pt_cursor *start)
{
struct amdgpu_vm_pt_cursor cursor;
struct amdgpu_vm_pt *entry;
- for_each_amdgpu_vm_pt_dfs_safe(adev, vm, cursor, entry) {
+ vm->bulk_moveable = false;
- if (entry->base.bo) {
- entry->base.bo->vm_bo = NULL;
- list_del(&entry->base.vm_status);
- amdgpu_bo_unref(&entry->base.bo->shadow);
- amdgpu_bo_unref(&entry->base.bo);
- }
- kvfree(entry->entries);
- }
+ for_each_amdgpu_vm_pt_dfs_safe(adev, vm, start, cursor, entry)
+ amdgpu_vm_free_table(entry);
- BUG_ON(vm->root.base.bo);
+ if (start)
+ amdgpu_vm_free_table(start->entry);
}
/**
return NULL;
}
-/**
- * amdgpu_vm_do_set_ptes - helper to call the right asic function
- *
- * @params: see amdgpu_pte_update_params definition
- * @bo: PD/PT to update
- * @pe: addr of the page entry
- * @addr: dst addr to write into pe
- * @count: number of page entries to update
- * @incr: increase next addr by incr bytes
- * @flags: hw access flags
- *
- * Traces the parameters and calls the right asic functions
- * to setup the page table using the DMA.
- */
-static void amdgpu_vm_do_set_ptes(struct amdgpu_pte_update_params *params,
- struct amdgpu_bo *bo,
- uint64_t pe, uint64_t addr,
- unsigned count, uint32_t incr,
- uint64_t flags)
-{
- pe += amdgpu_bo_gpu_offset(bo);
- trace_amdgpu_vm_set_ptes(pe, addr, count, incr, flags);
-
- if (count < 3) {
- amdgpu_vm_write_pte(params->adev, params->ib, pe,
- addr | flags, count, incr);
-
- } else {
- amdgpu_vm_set_pte_pde(params->adev, params->ib, pe, addr,
- count, incr, flags);
- }
-}
-
-/**
- * amdgpu_vm_do_copy_ptes - copy the PTEs from the GART
- *
- * @params: see amdgpu_pte_update_params definition
- * @bo: PD/PT to update
- * @pe: addr of the page entry
- * @addr: dst addr to write into pe
- * @count: number of page entries to update
- * @incr: increase next addr by incr bytes
- * @flags: hw access flags
- *
- * Traces the parameters and calls the DMA function to copy the PTEs.
- */
-static void amdgpu_vm_do_copy_ptes(struct amdgpu_pte_update_params *params,
- struct amdgpu_bo *bo,
- uint64_t pe, uint64_t addr,
- unsigned count, uint32_t incr,
- uint64_t flags)
-{
- uint64_t src = (params->src + (addr >> 12) * 8);
-
- pe += amdgpu_bo_gpu_offset(bo);
- trace_amdgpu_vm_copy_ptes(pe, src, count);
-
- amdgpu_vm_copy_pte(params->adev, params->ib, pe, src, count);
-}
-
/**
* amdgpu_vm_map_gart - Resolve gart mapping of addr
*
* Returns:
* The pointer for the page table entry.
*/
-static uint64_t amdgpu_vm_map_gart(const dma_addr_t *pages_addr, uint64_t addr)
+uint64_t amdgpu_vm_map_gart(const dma_addr_t *pages_addr, uint64_t addr)
{
uint64_t result;
return result;
}
-/**
- * amdgpu_vm_cpu_set_ptes - helper to update page tables via CPU
- *
- * @params: see amdgpu_pte_update_params definition
- * @bo: PD/PT to update
- * @pe: kmap addr of the page entry
- * @addr: dst addr to write into pe
- * @count: number of page entries to update
- * @incr: increase next addr by incr bytes
- * @flags: hw access flags
- *
- * Write count number of PT/PD entries directly.
- */
-static void amdgpu_vm_cpu_set_ptes(struct amdgpu_pte_update_params *params,
- struct amdgpu_bo *bo,
- uint64_t pe, uint64_t addr,
- unsigned count, uint32_t incr,
- uint64_t flags)
-{
- unsigned int i;
- uint64_t value;
-
- pe += (unsigned long)amdgpu_bo_kptr(bo);
-
- trace_amdgpu_vm_set_ptes(pe, addr, count, incr, flags);
-
- for (i = 0; i < count; i++) {
- value = params->pages_addr ?
- amdgpu_vm_map_gart(params->pages_addr, addr) :
- addr;
- amdgpu_gmc_set_pte_pde(params->adev, (void *)(uintptr_t)pe,
- i, value, flags);
- addr += incr;
- }
-}
-
-/**
- * amdgpu_vm_update_func - helper to call update function
- *
- * Calls the update function for both the given BO as well as its shadow.
- */
-static void amdgpu_vm_update_func(struct amdgpu_pte_update_params *params,
- struct amdgpu_bo *bo,
- uint64_t pe, uint64_t addr,
- unsigned count, uint32_t incr,
- uint64_t flags)
-{
- if (bo->shadow)
- params->func(params, bo->shadow, pe, addr, count, incr, flags);
- params->func(params, bo, pe, addr, count, incr, flags);
-}
-
/*
* amdgpu_vm_update_pde - update a single level in the hierarchy
*
* @param: parameters for the update
* @vm: requested vm
- * @parent: parent directory
* @entry: entry to update
*
* Makes sure the requested entry in parent is up to date.
*/
-static void amdgpu_vm_update_pde(struct amdgpu_pte_update_params *params,
- struct amdgpu_vm *vm,
- struct amdgpu_vm_pt *parent,
- struct amdgpu_vm_pt *entry)
+static int amdgpu_vm_update_pde(struct amdgpu_vm_update_params *params,
+ struct amdgpu_vm *vm,
+ struct amdgpu_vm_pt *entry)
{
+ struct amdgpu_vm_pt *parent = amdgpu_vm_pt_parent(entry);
struct amdgpu_bo *bo = parent->base.bo, *pbo;
uint64_t pde, pt, flags;
unsigned level;
- /* Don't update huge pages here */
- if (entry->huge)
- return;
-
for (level = 0, pbo = bo->parent; pbo; ++level)
pbo = pbo->parent;
level += params->adev->vm_manager.root_level;
amdgpu_gmc_get_pde_for_bo(entry->base.bo, level, &pt, &flags);
pde = (entry - parent->entries) * 8;
- amdgpu_vm_update_func(params, bo, pde, pt, 1, 0, flags);
+ return vm->update_funcs->update(params, bo, pde, pt, 1, 0, flags);
}
/*
struct amdgpu_vm_pt_cursor cursor;
struct amdgpu_vm_pt *entry;
- for_each_amdgpu_vm_pt_dfs_safe(adev, vm, cursor, entry)
+ for_each_amdgpu_vm_pt_dfs_safe(adev, vm, NULL, cursor, entry)
if (entry->base.bo && !entry->base.moved)
amdgpu_vm_bo_relocated(&entry->base);
}
int amdgpu_vm_update_directories(struct amdgpu_device *adev,
struct amdgpu_vm *vm)
{
- struct amdgpu_pte_update_params params;
- struct amdgpu_job *job;
- unsigned ndw = 0;
- int r = 0;
+ struct amdgpu_vm_update_params params;
+ int r;
if (list_empty(&vm->relocated))
return 0;
-restart:
memset(¶ms, 0, sizeof(params));
params.adev = adev;
+ params.vm = vm;
- if (vm->use_cpu_for_update) {
- r = amdgpu_bo_sync_wait(vm->root.base.bo,
- AMDGPU_FENCE_OWNER_VM, true);
- if (unlikely(r))
- return r;
-
- params.func = amdgpu_vm_cpu_set_ptes;
- } else {
- ndw = 512 * 8;
- r = amdgpu_job_alloc_with_ib(adev, ndw * 4, &job);
- if (r)
- return r;
-
- params.ib = &job->ibs[0];
- params.func = amdgpu_vm_do_set_ptes;
- }
+ r = vm->update_funcs->prepare(¶ms, AMDGPU_FENCE_OWNER_VM, NULL);
+ if (r)
+ return r;
while (!list_empty(&vm->relocated)) {
- struct amdgpu_vm_pt *pt, *entry;
+ struct amdgpu_vm_pt *entry;
entry = list_first_entry(&vm->relocated, struct amdgpu_vm_pt,
base.vm_status);
amdgpu_vm_bo_idle(&entry->base);
- pt = amdgpu_vm_pt_parent(entry);
- if (!pt)
- continue;
-
- amdgpu_vm_update_pde(¶ms, vm, pt, entry);
-
- if (!vm->use_cpu_for_update &&
- (ndw - params.ib->length_dw) < 32)
- break;
- }
-
- if (vm->use_cpu_for_update) {
- /* Flush HDP */
- mb();
- amdgpu_asic_flush_hdp(adev, NULL);
- } else if (params.ib->length_dw == 0) {
- amdgpu_job_free(job);
- } else {
- struct amdgpu_bo *root = vm->root.base.bo;
- struct amdgpu_ring *ring;
- struct dma_fence *fence;
-
- ring = container_of(vm->entity.rq->sched, struct amdgpu_ring,
- sched);
-
- amdgpu_ring_pad_ib(ring, params.ib);
- amdgpu_sync_resv(adev, &job->sync, root->tbo.resv,
- AMDGPU_FENCE_OWNER_VM, false);
- WARN_ON(params.ib->length_dw > ndw);
- r = amdgpu_job_submit(job, &vm->entity, AMDGPU_FENCE_OWNER_VM,
- &fence);
+ r = amdgpu_vm_update_pde(¶ms, vm, entry);
if (r)
goto error;
-
- amdgpu_bo_fence(root, fence, true);
- dma_fence_put(vm->last_update);
- vm->last_update = fence;
}
- if (!list_empty(&vm->relocated))
- goto restart;
-
+ r = vm->update_funcs->commit(¶ms, &vm->last_update);
+ if (r)
+ goto error;
return 0;
error:
amdgpu_vm_invalidate_pds(adev, vm);
- amdgpu_job_free(job);
return r;
}
*
* Make sure to set the right flags for the PTEs at the desired level.
*/
-static void amdgpu_vm_update_flags(struct amdgpu_pte_update_params *params,
+static void amdgpu_vm_update_flags(struct amdgpu_vm_update_params *params,
struct amdgpu_bo *bo, unsigned level,
uint64_t pe, uint64_t addr,
unsigned count, uint32_t incr,
flags |= AMDGPU_PTE_EXECUTABLE;
}
- amdgpu_vm_update_func(params, bo, pe, addr, count, incr, flags);
+ params->vm->update_funcs->update(params, bo, pe, addr, count, incr,
+ flags);
}
/**
* amdgpu_vm_fragment - get fragment for PTEs
*
- * @params: see amdgpu_pte_update_params definition
+ * @params: see amdgpu_vm_update_params definition
* @start: first PTE to handle
* @end: last PTE to handle
* @flags: hw mapping flags
*
* Returns the first possible fragment for the start and end address.
*/
-static void amdgpu_vm_fragment(struct amdgpu_pte_update_params *params,
+static void amdgpu_vm_fragment(struct amdgpu_vm_update_params *params,
uint64_t start, uint64_t end, uint64_t flags,
unsigned int *frag, uint64_t *frag_end)
{
max_frag = 31;
/* system pages are non continuously */
- if (params->src) {
+ if (params->pages_addr) {
*frag = 0;
*frag_end = end;
return;
/**
* amdgpu_vm_update_ptes - make sure that page tables are valid
*
- * @params: see amdgpu_pte_update_params definition
+ * @params: see amdgpu_vm_update_params definition
* @start: start of GPU address range
* @end: end of GPU address range
* @dst: destination address to map to, the next dst inside the function
* Returns:
* 0 for success, -EINVAL for failure.
*/
-static int amdgpu_vm_update_ptes(struct amdgpu_pte_update_params *params,
+static int amdgpu_vm_update_ptes(struct amdgpu_vm_update_params *params,
uint64_t start, uint64_t end,
uint64_t dst, uint64_t flags)
{
struct amdgpu_vm_pt_cursor cursor;
uint64_t frag_start = start, frag_end;
unsigned int frag;
+ int r;
/* figure out the initial fragment */
amdgpu_vm_fragment(params, frag_start, end, flags, &frag, &frag_end);
/* walk over the address space and update the PTs */
amdgpu_vm_pt_start(adev, params->vm, start, &cursor);
while (cursor.pfn < end) {
- struct amdgpu_bo *pt = cursor.entry->base.bo;
unsigned shift, parent_shift, mask;
uint64_t incr, entry_end, pe_start;
+ struct amdgpu_bo *pt;
- if (!pt)
- return -ENOENT;
+ r = amdgpu_vm_alloc_pts(params->adev, params->vm, &cursor);
+ if (r)
+ return r;
+
+ pt = cursor.entry->base.bo;
/* The root level can't be a huge page */
if (cursor.level == adev->vm_manager.root_level) {
continue;
}
- /* If it isn't already handled it can't be a huge page */
- if (cursor.entry->huge) {
- /* Add the entry to the relocated list to update it. */
- cursor.entry->huge = false;
- amdgpu_vm_bo_relocated(&cursor.entry->base);
- }
-
shift = amdgpu_vm_level_shift(adev, cursor.level);
parent_shift = amdgpu_vm_level_shift(adev, cursor.level - 1);
- if (adev->asic_type < CHIP_VEGA10) {
+ if (adev->asic_type < CHIP_VEGA10 &&
+ (flags & AMDGPU_PTE_VALID)) {
/* No huge page support before GMC v9 */
if (cursor.level != AMDGPU_VM_PTB) {
if (!amdgpu_vm_pt_descendant(adev, &cursor))
} while (frag_start < entry_end);
if (amdgpu_vm_pt_descendant(adev, &cursor)) {
- /* Mark all child entries as huge */
+ /* Free all child entries */
while (cursor.pfn < frag_start) {
- cursor.entry->huge = true;
+ amdgpu_vm_free_pts(adev, params->vm, &cursor);
amdgpu_vm_pt_next(adev, &cursor);
}
uint64_t flags, uint64_t addr,
struct dma_fence **fence)
{
- struct amdgpu_ring *ring;
+ struct amdgpu_vm_update_params params;
void *owner = AMDGPU_FENCE_OWNER_VM;
- unsigned nptes, ncmds, ndw;
- struct amdgpu_job *job;
- struct amdgpu_pte_update_params params;
- struct dma_fence *f = NULL;
int r;
memset(¶ms, 0, sizeof(params));
params.adev = adev;
params.vm = vm;
+ params.pages_addr = pages_addr;
/* sync to everything except eviction fences on unmapping */
if (!(flags & AMDGPU_PTE_VALID))
owner = AMDGPU_FENCE_OWNER_KFD;
- if (vm->use_cpu_for_update) {
- /* params.src is used as flag to indicate system Memory */
- if (pages_addr)
- params.src = ~0;
-
- /* Wait for PT BOs to be idle. PTs share the same resv. object
- * as the root PD BO
- */
- r = amdgpu_bo_sync_wait(vm->root.base.bo, owner, true);
- if (unlikely(r))
- return r;
-
- /* Wait for any BO move to be completed */
- if (exclusive) {
- r = dma_fence_wait(exclusive, true);
- if (unlikely(r))
- return r;
- }
-
- params.func = amdgpu_vm_cpu_set_ptes;
- params.pages_addr = pages_addr;
- return amdgpu_vm_update_ptes(¶ms, start, last + 1,
- addr, flags);
- }
-
- ring = container_of(vm->entity.rq->sched, struct amdgpu_ring, sched);
-
- nptes = last - start + 1;
-
- /*
- * reserve space for two commands every (1 << BLOCK_SIZE)
- * entries or 2k dwords (whatever is smaller)
- */
- ncmds = ((nptes >> min(adev->vm_manager.block_size, 11u)) + 1);
-
- /* The second command is for the shadow pagetables. */
- if (vm->root.base.bo->shadow)
- ncmds *= 2;
-
- /* padding, etc. */
- ndw = 64;
-
- if (pages_addr) {
- /* copy commands needed */
- ndw += ncmds * adev->vm_manager.vm_pte_funcs->copy_pte_num_dw;
-
- /* and also PTEs */
- ndw += nptes * 2;
-
- params.func = amdgpu_vm_do_copy_ptes;
-
- } else {
- /* set page commands needed */
- ndw += ncmds * 10;
-
- /* extra commands for begin/end fragments */
- ncmds = 2 * adev->vm_manager.fragment_size;
- if (vm->root.base.bo->shadow)
- ncmds *= 2;
-
- ndw += 10 * ncmds;
-
- params.func = amdgpu_vm_do_set_ptes;
- }
-
- r = amdgpu_job_alloc_with_ib(adev, ndw * 4, &job);
+ r = vm->update_funcs->prepare(¶ms, owner, exclusive);
if (r)
return r;
- params.ib = &job->ibs[0];
-
- if (pages_addr) {
- uint64_t *pte;
- unsigned i;
-
- /* Put the PTEs at the end of the IB. */
- i = ndw - nptes * 2;
- pte= (uint64_t *)&(job->ibs->ptr[i]);
- params.src = job->ibs->gpu_addr + i * 4;
-
- for (i = 0; i < nptes; ++i) {
- pte[i] = amdgpu_vm_map_gart(pages_addr, addr + i *
- AMDGPU_GPU_PAGE_SIZE);
- pte[i] |= flags;
- }
- addr = 0;
- }
-
- r = amdgpu_sync_fence(adev, &job->sync, exclusive, false);
- if (r)
- goto error_free;
-
- r = amdgpu_sync_resv(adev, &job->sync, vm->root.base.bo->tbo.resv,
- owner, false);
- if (r)
- goto error_free;
-
r = amdgpu_vm_update_ptes(¶ms, start, last + 1, addr, flags);
if (r)
- goto error_free;
-
- amdgpu_ring_pad_ib(ring, params.ib);
- WARN_ON(params.ib->length_dw > ndw);
- r = amdgpu_job_submit(job, &vm->entity, AMDGPU_FENCE_OWNER_VM, &f);
- if (r)
- goto error_free;
-
- amdgpu_bo_fence(vm->root.base.bo, f, true);
- dma_fence_put(*fence);
- *fence = f;
- return 0;
+ return r;
-error_free:
- amdgpu_job_free(job);
- return r;
+ return vm->update_funcs->commit(¶ms, fence);
}
/**
* @vm: requested vm
* @mapping: mapped range and flags to use for the update
* @flags: HW flags for the mapping
+ * @bo_adev: amdgpu_device pointer that bo actually been allocated
* @nodes: array of drm_mm_nodes with the MC addresses
* @fence: optional resulting fence
*
struct amdgpu_vm *vm,
struct amdgpu_bo_va_mapping *mapping,
uint64_t flags,
+ struct amdgpu_device *bo_adev,
struct drm_mm_node *nodes,
struct dma_fence **fence)
{
if (pages_addr) {
uint64_t count;
- max_entries = min(max_entries, 16ull * 1024ull);
for (count = 1;
count < max_entries / AMDGPU_GPU_PAGES_IN_CPU_PAGE;
++count) {
}
} else if (flags & AMDGPU_PTE_VALID) {
- addr += adev->vm_manager.vram_base_offset;
+ addr += bo_adev->vm_manager.vram_base_offset;
addr += pfn << PAGE_SHIFT;
}
struct drm_mm_node *nodes;
struct dma_fence *exclusive, **last_update;
uint64_t flags;
+ struct amdgpu_device *bo_adev = adev;
int r;
if (clear || !bo) {
exclusive = reservation_object_get_excl(bo->tbo.resv);
}
- if (bo)
+ if (bo) {
flags = amdgpu_ttm_tt_pte_flags(adev, bo->tbo.ttm, mem);
- else
+ bo_adev = amdgpu_ttm_adev(bo->tbo.bdev);
+ } else {
flags = 0x0;
+ }
if (clear || (bo && bo->tbo.resv == vm->root.base.bo->tbo.resv))
last_update = &vm->last_update;
list_for_each_entry(mapping, &bo_va->invalids, list) {
r = amdgpu_vm_bo_split_mapping(adev, exclusive, pages_addr, vm,
- mapping, flags, nodes,
+ mapping, flags, bo_adev, nodes,
last_update);
if (r)
return r;
INIT_LIST_HEAD(&bo_va->valids);
INIT_LIST_HEAD(&bo_va->invalids);
+ if (bo && amdgpu_xgmi_same_hive(adev, amdgpu_ttm_adev(bo->tbo.bdev)) &&
+ (bo->preferred_domains & AMDGPU_GEM_DOMAIN_VRAM)) {
+ bo_va->is_xgmi = true;
+ mutex_lock(&adev->vm_manager.lock_pstate);
+ /* Power up XGMI if it can be potentially used */
+ if (++adev->vm_manager.xgmi_map_counter == 1)
+ amdgpu_xgmi_set_pstate(adev, 1);
+ mutex_unlock(&adev->vm_manager.lock_pstate);
+ }
+
return bo_va;
}
}
dma_fence_put(bo_va->last_pt_update);
+
+ if (bo && bo_va->is_xgmi) {
+ mutex_lock(&adev->vm_manager.lock_pstate);
+ if (--adev->vm_manager.xgmi_map_counter == 0)
+ amdgpu_xgmi_set_pstate(adev, 0);
+ mutex_unlock(&adev->vm_manager.lock_pstate);
+ }
+
kfree(bo_va);
}
adev->vm_manager.fragment_size);
}
-static struct amdgpu_retryfault_hashtable *init_fault_hash(void)
+/**
+ * amdgpu_vm_wait_idle - wait for the VM to become idle
+ *
+ * @vm: VM object to wait for
+ * @timeout: timeout to wait for VM to become idle
+ */
+long amdgpu_vm_wait_idle(struct amdgpu_vm *vm, long timeout)
{
- struct amdgpu_retryfault_hashtable *fault_hash;
-
- fault_hash = kmalloc(sizeof(*fault_hash), GFP_KERNEL);
- if (!fault_hash)
- return fault_hash;
-
- INIT_CHASH_TABLE(fault_hash->hash,
- AMDGPU_PAGEFAULT_HASH_BITS, 8, 0);
- spin_lock_init(&fault_hash->lock);
- fault_hash->count = 0;
-
- return fault_hash;
+ return reservation_object_wait_timeout_rcu(vm->root.base.bo->tbo.resv,
+ true, true, timeout);
}
/**
vm->use_cpu_for_update ? "CPU" : "SDMA");
WARN_ONCE((vm->use_cpu_for_update && !amdgpu_gmc_vram_full_visible(&adev->gmc)),
"CPU update of VM recommended only for large BAR system\n");
+
+ if (vm->use_cpu_for_update)
+ vm->update_funcs = &amdgpu_vm_cpu_funcs;
+ else
+ vm->update_funcs = &amdgpu_vm_sdma_funcs;
vm->last_update = NULL;
amdgpu_vm_bo_param(adev, vm, adev->vm_manager.root_level, &bp);
amdgpu_vm_bo_base_init(&vm->root.base, vm, root);
- r = amdgpu_vm_clear_bo(adev, vm, root,
- adev->vm_manager.root_level,
- vm->pte_support_ats);
+ r = amdgpu_vm_clear_bo(adev, vm, root);
if (r)
goto error_unreserve;
vm->pasid = pasid;
}
- vm->fault_hash = init_fault_hash();
- if (!vm->fault_hash) {
- r = -ENOMEM;
- goto error_free_root;
- }
-
INIT_KFIFO(vm->faults);
return 0;
* changing any other state, in case it fails.
*/
if (pte_support_ats != vm->pte_support_ats) {
- r = amdgpu_vm_clear_bo(adev, vm, vm->root.base.bo,
- adev->vm_manager.root_level,
- pte_support_ats);
+ vm->pte_support_ats = pte_support_ats;
+ r = amdgpu_vm_clear_bo(adev, vm, vm->root.base.bo);
if (r)
goto free_idr;
}
/* Update VM state */
vm->use_cpu_for_update = !!(adev->vm_manager.vm_update_mode &
AMDGPU_VM_USE_CPU_FOR_COMPUTE);
- vm->pte_support_ats = pte_support_ats;
DRM_DEBUG_DRIVER("VM update mode is %s\n",
vm->use_cpu_for_update ? "CPU" : "SDMA");
WARN_ONCE((vm->use_cpu_for_update && !amdgpu_gmc_vram_full_visible(&adev->gmc)),
struct amdgpu_bo_va_mapping *mapping, *tmp;
bool prt_fini_needed = !!adev->gmc.gmc_funcs->set_prt;
struct amdgpu_bo *root;
- u64 fault;
int i, r;
amdgpu_amdkfd_gpuvm_destroy_cb(adev, vm);
- /* Clear pending page faults from IH when the VM is destroyed */
- while (kfifo_get(&vm->faults, &fault))
- amdgpu_vm_clear_fault(vm->fault_hash, fault);
-
if (vm->pasid) {
unsigned long flags;
spin_unlock_irqrestore(&adev->vm_manager.pasid_lock, flags);
}
- kfree(vm->fault_hash);
- vm->fault_hash = NULL;
-
drm_sched_entity_destroy(&vm->entity);
if (!RB_EMPTY_ROOT(&vm->va.rb_root)) {
if (r) {
dev_err(adev->dev, "Leaking page tables because BO reservation failed\n");
} else {
- amdgpu_vm_free_pts(adev, vm);
+ amdgpu_vm_free_pts(adev, vm, NULL);
amdgpu_bo_unreserve(root);
}
amdgpu_bo_unref(&root);
+ WARN_ON(vm->root.base.bo);
dma_fence_put(vm->last_update);
for (i = 0; i < AMDGPU_MAX_VMHUBS; i++)
amdgpu_vmid_free_reserved(adev, vm, i);
idr_init(&adev->vm_manager.pasid_idr);
spin_lock_init(&adev->vm_manager.pasid_lock);
+
+ adev->vm_manager.xgmi_map_counter = 0;
+ mutex_init(&adev->vm_manager.lock_pstate);
}
/**
}
}
}
-
-/**
- * amdgpu_vm_add_fault - Add a page fault record to fault hash table
- *
- * @fault_hash: fault hash table
- * @key: 64-bit encoding of PASID and address
- *
- * This should be called when a retry page fault interrupt is
- * received. If this is a new page fault, it will be added to a hash
- * table. The return value indicates whether this is a new fault, or
- * a fault that was already known and is already being handled.
- *
- * If there are too many pending page faults, this will fail. Retry
- * interrupts should be ignored in this case until there is enough
- * free space.
- *
- * Returns 0 if the fault was added, 1 if the fault was already known,
- * -ENOSPC if there are too many pending faults.
- */
-int amdgpu_vm_add_fault(struct amdgpu_retryfault_hashtable *fault_hash, u64 key)
-{
- unsigned long flags;
- int r = -ENOSPC;
-
- if (WARN_ON_ONCE(!fault_hash))
- /* Should be allocated in amdgpu_vm_init
- */
- return r;
-
- spin_lock_irqsave(&fault_hash->lock, flags);
-
- /* Only let the hash table fill up to 50% for best performance */
- if (fault_hash->count >= (1 << (AMDGPU_PAGEFAULT_HASH_BITS-1)))
- goto unlock_out;
-
- r = chash_table_copy_in(&fault_hash->hash, key, NULL);
- if (!r)
- fault_hash->count++;
-
- /* chash_table_copy_in should never fail unless we're losing count */
- WARN_ON_ONCE(r < 0);
-
-unlock_out:
- spin_unlock_irqrestore(&fault_hash->lock, flags);
- return r;
-}
-
-/**
- * amdgpu_vm_clear_fault - Remove a page fault record
- *
- * @fault_hash: fault hash table
- * @key: 64-bit encoding of PASID and address
- *
- * This should be called when a page fault has been handled. Any
- * future interrupt with this key will be processed as a new
- * page fault.
- */
-void amdgpu_vm_clear_fault(struct amdgpu_retryfault_hashtable *fault_hash, u64 key)
-{
- unsigned long flags;
- int r;
-
- if (!fault_hash)
- return;
-
- spin_lock_irqsave(&fault_hash->lock, flags);
-
- r = chash_table_remove(&fault_hash->hash, key, NULL);
- if (!WARN_ON_ONCE(r < 0)) {
- fault_hash->count--;
- WARN_ON_ONCE(fault_hash->count < 0);
- }
-
- spin_unlock_irqrestore(&fault_hash->lock, flags);
-}
#include <drm/gpu_scheduler.h>
#include <drm/drm_file.h>
#include <drm/ttm/ttm_bo_driver.h>
-#include <linux/chash.h>
#include "amdgpu_sync.h"
#include "amdgpu_ring.h"
struct amdgpu_vm_pt {
struct amdgpu_vm_bo_base base;
- bool huge;
/* array of page tables, one for each directory entry */
struct amdgpu_vm_pt *entries;
uint32_t incr, uint64_t flags);
};
-#define AMDGPU_VM_FAULT(pasid, addr) (((u64)(pasid) << 48) | (addr))
-#define AMDGPU_VM_FAULT_PASID(fault) ((u64)(fault) >> 48)
-#define AMDGPU_VM_FAULT_ADDR(fault) ((u64)(fault) & 0xfffffffff000ULL)
-
-
struct amdgpu_task_info {
char process_name[TASK_COMM_LEN];
char task_name[TASK_COMM_LEN];
pid_t tgid;
};
-#define AMDGPU_PAGEFAULT_HASH_BITS 8
-struct amdgpu_retryfault_hashtable {
- DECLARE_CHASH_TABLE(hash, AMDGPU_PAGEFAULT_HASH_BITS, 8, 0);
- spinlock_t lock;
- int count;
+/**
+ * struct amdgpu_vm_update_params
+ *
+ * Encapsulate some VM table update parameters to reduce
+ * the number of function parameters
+ *
+ */
+struct amdgpu_vm_update_params {
+
+ /**
+ * @adev: amdgpu device we do this update for
+ */
+ struct amdgpu_device *adev;
+
+ /**
+ * @vm: optional amdgpu_vm we do this update for
+ */
+ struct amdgpu_vm *vm;
+
+ /**
+ * @pages_addr:
+ *
+ * DMA addresses to use for mapping
+ */
+ dma_addr_t *pages_addr;
+
+ /**
+ * @job: job to used for hw submission
+ */
+ struct amdgpu_job *job;
+
+ /**
+ * @num_dw_left: number of dw left for the IB
+ */
+ unsigned int num_dw_left;
+};
+
+struct amdgpu_vm_update_funcs {
+ int (*map_table)(struct amdgpu_bo *bo);
+ int (*prepare)(struct amdgpu_vm_update_params *p, void * owner,
+ struct dma_fence *exclusive);
+ int (*update)(struct amdgpu_vm_update_params *p,
+ struct amdgpu_bo *bo, uint64_t pe, uint64_t addr,
+ unsigned count, uint32_t incr, uint64_t flags);
+ int (*commit)(struct amdgpu_vm_update_params *p,
+ struct dma_fence **fence);
};
struct amdgpu_vm {
struct amdgpu_vmid *reserved_vmid[AMDGPU_MAX_VMHUBS];
/* Flag to indicate if VM tables are updated by CPU or GPU (SDMA) */
- bool use_cpu_for_update;
+ bool use_cpu_for_update;
+
+ /* Functions to use for VM table updates */
+ const struct amdgpu_vm_update_funcs *update_funcs;
/* Flag to indicate ATS support from PTE for GFX9 */
bool pte_support_ats;
struct ttm_lru_bulk_move lru_bulk_move;
/* mark whether can do the bulk move */
bool bulk_moveable;
- struct amdgpu_retryfault_hashtable *fault_hash;
};
struct amdgpu_vm_manager {
const struct amdgpu_vm_pte_funcs *vm_pte_funcs;
struct drm_sched_rq *vm_pte_rqs[AMDGPU_MAX_RINGS];
unsigned vm_pte_num_rqs;
+ struct amdgpu_ring *page_fault;
/* partial resident texture handling */
spinlock_t prt_lock;
*/
struct idr pasid_idr;
spinlock_t pasid_lock;
+
+ /* counter of mapped memory through xgmi */
+ uint32_t xgmi_map_counter;
+ struct mutex lock_pstate;
};
#define amdgpu_vm_copy_pte(adev, ib, pe, src, count) ((adev)->vm_manager.vm_pte_funcs->copy_pte((ib), (pe), (src), (count)))
#define amdgpu_vm_write_pte(adev, ib, pe, value, count, incr) ((adev)->vm_manager.vm_pte_funcs->write_pte((ib), (pe), (value), (count), (incr)))
#define amdgpu_vm_set_pte_pde(adev, ib, pe, addr, count, incr, flags) ((adev)->vm_manager.vm_pte_funcs->set_pte_pde((ib), (pe), (addr), (count), (incr), (flags)))
+extern const struct amdgpu_vm_update_funcs amdgpu_vm_cpu_funcs;
+extern const struct amdgpu_vm_update_funcs amdgpu_vm_sdma_funcs;
+
void amdgpu_vm_manager_init(struct amdgpu_device *adev);
void amdgpu_vm_manager_fini(struct amdgpu_device *adev);
+
+long amdgpu_vm_wait_idle(struct amdgpu_vm *vm, long timeout);
int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
int vm_context, unsigned int pasid);
int amdgpu_vm_make_compute(struct amdgpu_device *adev, struct amdgpu_vm *vm, unsigned int pasid);
int amdgpu_vm_validate_pt_bos(struct amdgpu_device *adev, struct amdgpu_vm *vm,
int (*callback)(void *p, struct amdgpu_bo *bo),
void *param);
-int amdgpu_vm_alloc_pts(struct amdgpu_device *adev,
- struct amdgpu_vm *vm,
- uint64_t saddr, uint64_t size);
int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, bool need_pipe_sync);
int amdgpu_vm_update_directories(struct amdgpu_device *adev,
struct amdgpu_vm *vm);
bool clear);
void amdgpu_vm_bo_invalidate(struct amdgpu_device *adev,
struct amdgpu_bo *bo, bool evicted);
+uint64_t amdgpu_vm_map_gart(const dma_addr_t *pages_addr, uint64_t addr);
struct amdgpu_bo_va *amdgpu_vm_bo_find(struct amdgpu_vm *vm,
struct amdgpu_bo *bo);
struct amdgpu_bo_va *amdgpu_vm_bo_add(struct amdgpu_device *adev,
void amdgpu_vm_move_to_lru_tail(struct amdgpu_device *adev,
struct amdgpu_vm *vm);
-
-int amdgpu_vm_add_fault(struct amdgpu_retryfault_hashtable *fault_hash, u64 key);
-
-void amdgpu_vm_clear_fault(struct amdgpu_retryfault_hashtable *fault_hash, u64 key);
-
void amdgpu_vm_del_from_lru_notify(struct ttm_buffer_object *bo);
#endif
--- /dev/null
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include "amdgpu_vm.h"
+#include "amdgpu_object.h"
+#include "amdgpu_trace.h"
+
+/**
+ * amdgpu_vm_cpu_map_table - make sure new PDs/PTs are kmapped
+ *
+ * @table: newly allocated or validated PD/PT
+ */
+static int amdgpu_vm_cpu_map_table(struct amdgpu_bo *table)
+{
+ return amdgpu_bo_kmap(table, NULL);
+}
+
+/**
+ * amdgpu_vm_cpu_prepare - prepare page table update with the CPU
+ *
+ * @p: see amdgpu_vm_update_params definition
+ * @owner: owner we need to sync to
+ * @exclusive: exclusive move fence we need to sync to
+ *
+ * Returns:
+ * Negativ errno, 0 for success.
+ */
+static int amdgpu_vm_cpu_prepare(struct amdgpu_vm_update_params *p, void *owner,
+ struct dma_fence *exclusive)
+{
+ int r;
+
+ /* Wait for PT BOs to be idle. PTs share the same resv. object
+ * as the root PD BO
+ */
+ r = amdgpu_bo_sync_wait(p->vm->root.base.bo, owner, true);
+ if (unlikely(r))
+ return r;
+
+ /* Wait for any BO move to be completed */
+ if (exclusive) {
+ r = dma_fence_wait(exclusive, true);
+ if (unlikely(r))
+ return r;
+ }
+
+ return 0;
+}
+
+/**
+ * amdgpu_vm_cpu_update - helper to update page tables via CPU
+ *
+ * @p: see amdgpu_vm_update_params definition
+ * @bo: PD/PT to update
+ * @pe: kmap addr of the page entry
+ * @addr: dst addr to write into pe
+ * @count: number of page entries to update
+ * @incr: increase next addr by incr bytes
+ * @flags: hw access flags
+ *
+ * Write count number of PT/PD entries directly.
+ */
+static int amdgpu_vm_cpu_update(struct amdgpu_vm_update_params *p,
+ struct amdgpu_bo *bo, uint64_t pe,
+ uint64_t addr, unsigned count, uint32_t incr,
+ uint64_t flags)
+{
+ unsigned int i;
+ uint64_t value;
+
+ pe += (unsigned long)amdgpu_bo_kptr(bo);
+
+ trace_amdgpu_vm_set_ptes(pe, addr, count, incr, flags);
+
+ for (i = 0; i < count; i++) {
+ value = p->pages_addr ?
+ amdgpu_vm_map_gart(p->pages_addr, addr) :
+ addr;
+ amdgpu_gmc_set_pte_pde(p->adev, (void *)(uintptr_t)pe,
+ i, value, flags);
+ addr += incr;
+ }
+ return 0;
+}
+
+/**
+ * amdgpu_vm_cpu_commit - commit page table update to the HW
+ *
+ * @p: see amdgpu_vm_update_params definition
+ * @fence: unused
+ *
+ * Make sure that the hardware sees the page table updates.
+ */
+static int amdgpu_vm_cpu_commit(struct amdgpu_vm_update_params *p,
+ struct dma_fence **fence)
+{
+ /* Flush HDP */
+ mb();
+ amdgpu_asic_flush_hdp(p->adev, NULL);
+ return 0;
+}
+
+const struct amdgpu_vm_update_funcs amdgpu_vm_cpu_funcs = {
+ .map_table = amdgpu_vm_cpu_map_table,
+ .prepare = amdgpu_vm_cpu_prepare,
+ .update = amdgpu_vm_cpu_update,
+ .commit = amdgpu_vm_cpu_commit
+};
--- /dev/null
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include "amdgpu_vm.h"
+#include "amdgpu_job.h"
+#include "amdgpu_object.h"
+#include "amdgpu_trace.h"
+
+#define AMDGPU_VM_SDMA_MIN_NUM_DW 256u
+#define AMDGPU_VM_SDMA_MAX_NUM_DW (16u * 1024u)
+
+/**
+ * amdgpu_vm_sdma_map_table - make sure new PDs/PTs are GTT mapped
+ *
+ * @table: newly allocated or validated PD/PT
+ */
+static int amdgpu_vm_sdma_map_table(struct amdgpu_bo *table)
+{
+ int r;
+
+ r = amdgpu_ttm_alloc_gart(&table->tbo);
+ if (r)
+ return r;
+
+ if (table->shadow)
+ r = amdgpu_ttm_alloc_gart(&table->shadow->tbo);
+
+ return r;
+}
+
+/**
+ * amdgpu_vm_sdma_prepare - prepare SDMA command submission
+ *
+ * @p: see amdgpu_vm_update_params definition
+ * @owner: owner we need to sync to
+ * @exclusive: exclusive move fence we need to sync to
+ *
+ * Returns:
+ * Negativ errno, 0 for success.
+ */
+static int amdgpu_vm_sdma_prepare(struct amdgpu_vm_update_params *p,
+ void *owner, struct dma_fence *exclusive)
+{
+ struct amdgpu_bo *root = p->vm->root.base.bo;
+ unsigned int ndw = AMDGPU_VM_SDMA_MIN_NUM_DW;
+ int r;
+
+ r = amdgpu_job_alloc_with_ib(p->adev, ndw * 4, &p->job);
+ if (r)
+ return r;
+
+ r = amdgpu_sync_fence(p->adev, &p->job->sync, exclusive, false);
+ if (r)
+ return r;
+
+ r = amdgpu_sync_resv(p->adev, &p->job->sync, root->tbo.resv,
+ owner, false);
+ if (r)
+ return r;
+
+ p->num_dw_left = ndw;
+ return 0;
+}
+
+/**
+ * amdgpu_vm_sdma_commit - commit SDMA command submission
+ *
+ * @p: see amdgpu_vm_update_params definition
+ * @fence: resulting fence
+ *
+ * Returns:
+ * Negativ errno, 0 for success.
+ */
+static int amdgpu_vm_sdma_commit(struct amdgpu_vm_update_params *p,
+ struct dma_fence **fence)
+{
+ struct amdgpu_bo *root = p->vm->root.base.bo;
+ struct amdgpu_ib *ib = p->job->ibs;
+ struct amdgpu_ring *ring;
+ struct dma_fence *f;
+ int r;
+
+ ring = container_of(p->vm->entity.rq->sched, struct amdgpu_ring, sched);
+
+ WARN_ON(ib->length_dw == 0);
+ amdgpu_ring_pad_ib(ring, ib);
+ WARN_ON(ib->length_dw > p->num_dw_left);
+ r = amdgpu_job_submit(p->job, &p->vm->entity,
+ AMDGPU_FENCE_OWNER_VM, &f);
+ if (r)
+ goto error;
+
+ amdgpu_bo_fence(root, f, true);
+ if (fence)
+ swap(*fence, f);
+ dma_fence_put(f);
+ return 0;
+
+error:
+ amdgpu_job_free(p->job);
+ return r;
+}
+
+
+/**
+ * amdgpu_vm_sdma_copy_ptes - copy the PTEs from mapping
+ *
+ * @p: see amdgpu_vm_update_params definition
+ * @bo: PD/PT to update
+ * @pe: addr of the page entry
+ * @count: number of page entries to copy
+ *
+ * Traces the parameters and calls the DMA function to copy the PTEs.
+ */
+static void amdgpu_vm_sdma_copy_ptes(struct amdgpu_vm_update_params *p,
+ struct amdgpu_bo *bo, uint64_t pe,
+ unsigned count)
+{
+ struct amdgpu_ib *ib = p->job->ibs;
+ uint64_t src = ib->gpu_addr;
+
+ src += p->num_dw_left * 4;
+
+ pe += amdgpu_bo_gpu_offset(bo);
+ trace_amdgpu_vm_copy_ptes(pe, src, count);
+
+ amdgpu_vm_copy_pte(p->adev, ib, pe, src, count);
+}
+
+/**
+ * amdgpu_vm_sdma_set_ptes - helper to call the right asic function
+ *
+ * @p: see amdgpu_vm_update_params definition
+ * @bo: PD/PT to update
+ * @pe: addr of the page entry
+ * @addr: dst addr to write into pe
+ * @count: number of page entries to update
+ * @incr: increase next addr by incr bytes
+ * @flags: hw access flags
+ *
+ * Traces the parameters and calls the right asic functions
+ * to setup the page table using the DMA.
+ */
+static void amdgpu_vm_sdma_set_ptes(struct amdgpu_vm_update_params *p,
+ struct amdgpu_bo *bo, uint64_t pe,
+ uint64_t addr, unsigned count,
+ uint32_t incr, uint64_t flags)
+{
+ struct amdgpu_ib *ib = p->job->ibs;
+
+ pe += amdgpu_bo_gpu_offset(bo);
+ trace_amdgpu_vm_set_ptes(pe, addr, count, incr, flags);
+ if (count < 3) {
+ amdgpu_vm_write_pte(p->adev, ib, pe, addr | flags,
+ count, incr);
+ } else {
+ amdgpu_vm_set_pte_pde(p->adev, ib, pe, addr,
+ count, incr, flags);
+ }
+}
+
+/**
+ * amdgpu_vm_sdma_update - execute VM update
+ *
+ * @p: see amdgpu_vm_update_params definition
+ * @bo: PD/PT to update
+ * @pe: addr of the page entry
+ * @addr: dst addr to write into pe
+ * @count: number of page entries to update
+ * @incr: increase next addr by incr bytes
+ * @flags: hw access flags
+ *
+ * Reserve space in the IB, setup mapping buffer on demand and write commands to
+ * the IB.
+ */
+static int amdgpu_vm_sdma_update(struct amdgpu_vm_update_params *p,
+ struct amdgpu_bo *bo, uint64_t pe,
+ uint64_t addr, unsigned count, uint32_t incr,
+ uint64_t flags)
+{
+ unsigned int i, ndw, nptes;
+ uint64_t *pte;
+ int r;
+
+ do {
+ ndw = p->num_dw_left;
+ ndw -= p->job->ibs->length_dw;
+
+ if (ndw < 32) {
+ r = amdgpu_vm_sdma_commit(p, NULL);
+ if (r)
+ return r;
+
+ /* estimate how many dw we need */
+ ndw = 32;
+ if (p->pages_addr)
+ ndw += count * 2;
+ ndw = max(ndw, AMDGPU_VM_SDMA_MIN_NUM_DW);
+ ndw = min(ndw, AMDGPU_VM_SDMA_MAX_NUM_DW);
+
+ r = amdgpu_job_alloc_with_ib(p->adev, ndw * 4, &p->job);
+ if (r)
+ return r;
+
+ p->num_dw_left = ndw;
+ }
+
+ if (!p->pages_addr) {
+ /* set page commands needed */
+ if (bo->shadow)
+ amdgpu_vm_sdma_set_ptes(p, bo->shadow, pe, addr,
+ count, incr, flags);
+ amdgpu_vm_sdma_set_ptes(p, bo, pe, addr, count,
+ incr, flags);
+ return 0;
+ }
+
+ /* copy commands needed */
+ ndw -= p->adev->vm_manager.vm_pte_funcs->copy_pte_num_dw *
+ (bo->shadow ? 2 : 1);
+
+ /* for padding */
+ ndw -= 7;
+
+ nptes = min(count, ndw / 2);
+
+ /* Put the PTEs at the end of the IB. */
+ p->num_dw_left -= nptes * 2;
+ pte = (uint64_t *)&(p->job->ibs->ptr[p->num_dw_left]);
+ for (i = 0; i < nptes; ++i, addr += incr) {
+ pte[i] = amdgpu_vm_map_gart(p->pages_addr, addr);
+ pte[i] |= flags;
+ }
+
+ if (bo->shadow)
+ amdgpu_vm_sdma_copy_ptes(p, bo->shadow, pe, nptes);
+ amdgpu_vm_sdma_copy_ptes(p, bo, pe, nptes);
+
+ pe += nptes * 8;
+ count -= nptes;
+ } while (count);
+
+ return 0;
+}
+
+const struct amdgpu_vm_update_funcs amdgpu_vm_sdma_funcs = {
+ .map_table = amdgpu_vm_sdma_map_table,
+ .prepare = amdgpu_vm_sdma_prepare,
+ .update = amdgpu_vm_sdma_update,
+ .commit = amdgpu_vm_sdma_commit
+};
atomic64_t vis_usage;
};
+/**
+ * DOC: mem_info_vram_total
+ *
+ * The amdgpu driver provides a sysfs API for reporting current total VRAM
+ * available on the device
+ * The file mem_info_vram_total is used for this and returns the total
+ * amount of VRAM in bytes
+ */
+static ssize_t amdgpu_mem_info_vram_total_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct amdgpu_device *adev = ddev->dev_private;
+
+ return snprintf(buf, PAGE_SIZE, "%llu\n", adev->gmc.real_vram_size);
+}
+
+/**
+ * DOC: mem_info_vis_vram_total
+ *
+ * The amdgpu driver provides a sysfs API for reporting current total
+ * visible VRAM available on the device
+ * The file mem_info_vis_vram_total is used for this and returns the total
+ * amount of visible VRAM in bytes
+ */
+static ssize_t amdgpu_mem_info_vis_vram_total_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct amdgpu_device *adev = ddev->dev_private;
+
+ return snprintf(buf, PAGE_SIZE, "%llu\n", adev->gmc.visible_vram_size);
+}
+
+/**
+ * DOC: mem_info_vram_used
+ *
+ * The amdgpu driver provides a sysfs API for reporting current total VRAM
+ * available on the device
+ * The file mem_info_vram_used is used for this and returns the total
+ * amount of currently used VRAM in bytes
+ */
+static ssize_t amdgpu_mem_info_vram_used_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct amdgpu_device *adev = ddev->dev_private;
+
+ return snprintf(buf, PAGE_SIZE, "%llu\n",
+ amdgpu_vram_mgr_usage(&adev->mman.bdev.man[TTM_PL_VRAM]));
+}
+
+/**
+ * DOC: mem_info_vis_vram_used
+ *
+ * The amdgpu driver provides a sysfs API for reporting current total of
+ * used visible VRAM
+ * The file mem_info_vis_vram_used is used for this and returns the total
+ * amount of currently used visible VRAM in bytes
+ */
+static ssize_t amdgpu_mem_info_vis_vram_used_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct amdgpu_device *adev = ddev->dev_private;
+
+ return snprintf(buf, PAGE_SIZE, "%llu\n",
+ amdgpu_vram_mgr_vis_usage(&adev->mman.bdev.man[TTM_PL_VRAM]));
+}
+
+static DEVICE_ATTR(mem_info_vram_total, S_IRUGO,
+ amdgpu_mem_info_vram_total_show, NULL);
+static DEVICE_ATTR(mem_info_vis_vram_total, S_IRUGO,
+ amdgpu_mem_info_vis_vram_total_show,NULL);
+static DEVICE_ATTR(mem_info_vram_used, S_IRUGO,
+ amdgpu_mem_info_vram_used_show, NULL);
+static DEVICE_ATTR(mem_info_vis_vram_used, S_IRUGO,
+ amdgpu_mem_info_vis_vram_used_show, NULL);
+
/**
* amdgpu_vram_mgr_init - init VRAM manager and DRM MM
*
static int amdgpu_vram_mgr_init(struct ttm_mem_type_manager *man,
unsigned long p_size)
{
+ struct amdgpu_device *adev = amdgpu_ttm_adev(man->bdev);
struct amdgpu_vram_mgr *mgr;
+ int ret;
mgr = kzalloc(sizeof(*mgr), GFP_KERNEL);
if (!mgr)
drm_mm_init(&mgr->mm, 0, p_size);
spin_lock_init(&mgr->lock);
man->priv = mgr;
+
+ /* Add the two VRAM-related sysfs files */
+ ret = device_create_file(adev->dev, &dev_attr_mem_info_vram_total);
+ if (ret) {
+ DRM_ERROR("Failed to create device file mem_info_vram_total\n");
+ return ret;
+ }
+ ret = device_create_file(adev->dev, &dev_attr_mem_info_vis_vram_total);
+ if (ret) {
+ DRM_ERROR("Failed to create device file mem_info_vis_vram_total\n");
+ return ret;
+ }
+ ret = device_create_file(adev->dev, &dev_attr_mem_info_vram_used);
+ if (ret) {
+ DRM_ERROR("Failed to create device file mem_info_vram_used\n");
+ return ret;
+ }
+ ret = device_create_file(adev->dev, &dev_attr_mem_info_vis_vram_used);
+ if (ret) {
+ DRM_ERROR("Failed to create device file mem_info_vis_vram_used\n");
+ return ret;
+ }
+
return 0;
}
*/
static int amdgpu_vram_mgr_fini(struct ttm_mem_type_manager *man)
{
+ struct amdgpu_device *adev = amdgpu_ttm_adev(man->bdev);
struct amdgpu_vram_mgr *mgr = man->priv;
spin_lock(&mgr->lock);
spin_unlock(&mgr->lock);
kfree(mgr);
man->priv = NULL;
+ device_remove_file(adev->dev, &dev_attr_mem_info_vram_total);
+ device_remove_file(adev->dev, &dev_attr_mem_info_vis_vram_total);
+ device_remove_file(adev->dev, &dev_attr_mem_info_vram_used);
+ device_remove_file(adev->dev, &dev_attr_mem_info_vis_vram_used);
return 0;
}
#include <linux/list.h>
#include "amdgpu.h"
#include "amdgpu_xgmi.h"
+#include "amdgpu_smu.h"
static DEFINE_MUTEX(xgmi_mutex);
static struct amdgpu_hive_info xgmi_hives[AMDGPU_MAX_XGMI_HIVE];
static unsigned hive_count = 0;
-
void *amdgpu_xgmi_hive_try_lock(struct amdgpu_hive_info *hive)
{
return &hive->device_list;
}
+static ssize_t amdgpu_xgmi_show_hive_id(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct amdgpu_hive_info *hive =
+ container_of(attr, struct amdgpu_hive_info, dev_attr);
+
+ return snprintf(buf, PAGE_SIZE, "%llu\n", hive->hive_id);
+}
+
+static int amdgpu_xgmi_sysfs_create(struct amdgpu_device *adev,
+ struct amdgpu_hive_info *hive)
+{
+ int ret = 0;
+
+ if (WARN_ON(hive->kobj))
+ return -EINVAL;
+
+ hive->kobj = kobject_create_and_add("xgmi_hive_info", &adev->dev->kobj);
+ if (!hive->kobj) {
+ dev_err(adev->dev, "XGMI: Failed to allocate sysfs entry!\n");
+ return -EINVAL;
+ }
+
+ hive->dev_attr = (struct device_attribute) {
+ .attr = {
+ .name = "xgmi_hive_id",
+ .mode = S_IRUGO,
+
+ },
+ .show = amdgpu_xgmi_show_hive_id,
+ };
+
+ ret = sysfs_create_file(hive->kobj, &hive->dev_attr.attr);
+ if (ret) {
+ dev_err(adev->dev, "XGMI: Failed to create device file xgmi_hive_id\n");
+ kobject_del(hive->kobj);
+ kobject_put(hive->kobj);
+ hive->kobj = NULL;
+ }
+
+ return ret;
+}
+
+static void amdgpu_xgmi_sysfs_destroy(struct amdgpu_device *adev,
+ struct amdgpu_hive_info *hive)
+{
+ sysfs_remove_file(hive->kobj, &hive->dev_attr.attr);
+ kobject_del(hive->kobj);
+ kobject_put(hive->kobj);
+ hive->kobj = NULL;
+}
+
+static ssize_t amdgpu_xgmi_show_device_id(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct amdgpu_device *adev = ddev->dev_private;
+
+ return snprintf(buf, PAGE_SIZE, "%llu\n", adev->gmc.xgmi.node_id);
+
+}
+
+
+static DEVICE_ATTR(xgmi_device_id, S_IRUGO, amdgpu_xgmi_show_device_id, NULL);
+
+
+static int amdgpu_xgmi_sysfs_add_dev_info(struct amdgpu_device *adev,
+ struct amdgpu_hive_info *hive)
+{
+ int ret = 0;
+ char node[10] = { 0 };
+
+ /* Create xgmi device id file */
+ ret = device_create_file(adev->dev, &dev_attr_xgmi_device_id);
+ if (ret) {
+ dev_err(adev->dev, "XGMI: Failed to create device file xgmi_device_id\n");
+ return ret;
+ }
+
+ /* Create sysfs link to hive info folder on the first device */
+ if (adev != hive->adev) {
+ ret = sysfs_create_link(&adev->dev->kobj, hive->kobj,
+ "xgmi_hive_info");
+ if (ret) {
+ dev_err(adev->dev, "XGMI: Failed to create link to hive info");
+ goto remove_file;
+ }
+ }
+
+ sprintf(node, "node%d", hive->number_devices);
+ /* Create sysfs link form the hive folder to yourself */
+ ret = sysfs_create_link(hive->kobj, &adev->dev->kobj, node);
+ if (ret) {
+ dev_err(adev->dev, "XGMI: Failed to create link from hive info");
+ goto remove_link;
+ }
+
+ goto success;
+
+
+remove_link:
+ sysfs_remove_link(&adev->dev->kobj, adev->ddev->unique);
+
+remove_file:
+ device_remove_file(adev->dev, &dev_attr_xgmi_device_id);
+
+success:
+ return ret;
+}
+
+static void amdgpu_xgmi_sysfs_rem_dev_info(struct amdgpu_device *adev,
+ struct amdgpu_hive_info *hive)
+{
+ device_remove_file(adev->dev, &dev_attr_xgmi_device_id);
+ sysfs_remove_link(&adev->dev->kobj, adev->ddev->unique);
+ sysfs_remove_link(hive->kobj, adev->ddev->unique);
+}
+
+
+
struct amdgpu_hive_info *amdgpu_get_xgmi_hive(struct amdgpu_device *adev, int lock)
{
int i;
/* initialize new hive if not exist */
tmp = &xgmi_hives[hive_count++];
+
+ if (amdgpu_xgmi_sysfs_create(adev, tmp)) {
+ mutex_unlock(&xgmi_mutex);
+ return NULL;
+ }
+
+ tmp->adev = adev;
tmp->hive_id = adev->gmc.xgmi.hive_id;
INIT_LIST_HEAD(&tmp->device_list);
mutex_init(&tmp->hive_lock);
mutex_init(&tmp->reset_lock);
+
if (lock)
mutex_lock(&tmp->hive_lock);
-
+ tmp->pstate = -1;
mutex_unlock(&xgmi_mutex);
return tmp;
}
+int amdgpu_xgmi_set_pstate(struct amdgpu_device *adev, int pstate)
+{
+ int ret = 0;
+ struct amdgpu_hive_info *hive = amdgpu_get_xgmi_hive(adev, 0);
+
+ if (!hive)
+ return 0;
+
+ if (hive->pstate == pstate)
+ return 0;
+
+ dev_dbg(adev->dev, "Set xgmi pstate %d.\n", pstate);
+
+ if (is_support_sw_smu(adev))
+ ret = smu_set_xgmi_pstate(&adev->smu, pstate);
+ if (ret)
+ dev_err(adev->dev,
+ "XGMI: Set pstate failure on device %llx, hive %llx, ret %d",
+ adev->gmc.xgmi.node_id,
+ adev->gmc.xgmi.hive_id, ret);
+
+ return ret;
+}
+
int amdgpu_xgmi_update_topology(struct amdgpu_hive_info *hive, struct amdgpu_device *adev)
{
int ret = -EINVAL;
break;
}
- dev_info(adev->dev, "XGMI: Add node %d, hive 0x%llx.\n",
- adev->gmc.xgmi.physical_node_id, adev->gmc.xgmi.hive_id);
+ if (!ret)
+ ret = amdgpu_xgmi_sysfs_add_dev_info(adev, hive);
+
+ if (!ret)
+ dev_info(adev->dev, "XGMI: Add node %d, hive 0x%llx.\n",
+ adev->gmc.xgmi.physical_node_id, adev->gmc.xgmi.hive_id);
+ else
+ dev_err(adev->dev, "XGMI: Failed to add node %d, hive 0x%llx ret: %d\n",
+ adev->gmc.xgmi.physical_node_id, adev->gmc.xgmi.hive_id,
+ ret);
+
mutex_unlock(&hive->hive_lock);
exit:
return;
if (!(hive->number_devices--)) {
+ amdgpu_xgmi_sysfs_destroy(adev, hive);
mutex_destroy(&hive->hive_lock);
mutex_destroy(&hive->reset_lock);
} else {
+ amdgpu_xgmi_sysfs_rem_dev_info(adev, hive);
mutex_unlock(&hive->hive_lock);
}
}
struct list_head device_list;
struct psp_xgmi_topology_info topology_info;
int number_devices;
- struct mutex hive_lock,
- reset_lock;
+ struct mutex hive_lock, reset_lock;
+ struct kobject *kobj;
+ struct device_attribute dev_attr;
+ struct amdgpu_device *adev;
+ int pstate; /*0 -- low , 1 -- high , -1 unknown*/
};
struct amdgpu_hive_info *amdgpu_get_xgmi_hive(struct amdgpu_device *adev, int lock);
int amdgpu_xgmi_update_topology(struct amdgpu_hive_info *hive, struct amdgpu_device *adev);
int amdgpu_xgmi_add_device(struct amdgpu_device *adev);
void amdgpu_xgmi_remove_device(struct amdgpu_device *adev);
+int amdgpu_xgmi_set_pstate(struct amdgpu_device *adev, int pstate);
+
+static inline bool amdgpu_xgmi_same_hive(struct amdgpu_device *adev,
+ struct amdgpu_device *bo_adev)
+{
+ return (adev != bo_adev &&
+ adev->gmc.xgmi.hive_id &&
+ adev->gmc.xgmi.hive_id == bo_adev->gmc.xgmi.hive_id);
+}
#endif
r = amdgpu_ring_init(adev, ring, 1024,
&adev->sdma.trap_irq,
(i == 0) ?
- AMDGPU_SDMA_IRQ_TRAP0 :
- AMDGPU_SDMA_IRQ_TRAP1);
+ AMDGPU_SDMA_IRQ_INSTANCE0 :
+ AMDGPU_SDMA_IRQ_INSTANCE1);
if (r)
return r;
}
u32 sdma_cntl;
switch (type) {
- case AMDGPU_SDMA_IRQ_TRAP0:
+ case AMDGPU_SDMA_IRQ_INSTANCE0:
switch (state) {
case AMDGPU_IRQ_STATE_DISABLE:
sdma_cntl = RREG32(mmSDMA0_CNTL + SDMA0_REGISTER_OFFSET);
break;
}
break;
- case AMDGPU_SDMA_IRQ_TRAP1:
+ case AMDGPU_SDMA_IRQ_INSTANCE1:
switch (state) {
case AMDGPU_IRQ_STATE_DISABLE:
sdma_cntl = RREG32(mmSDMA0_CNTL + SDMA1_REGISTER_OFFSET);
BANK_WIDTH(ADDR_SURF_BANK_WIDTH_1) |
BANK_HEIGHT(ADDR_SURF_BANK_HEIGHT_1) |
MACRO_TILE_ASPECT(ADDR_SURF_MACRO_ASPECT_2);
+ tilemode[18] = MICRO_TILE_MODE(ADDR_SURF_THIN_MICRO_TILING) |
+ ARRAY_MODE(ARRAY_1D_TILED_THICK) |
+ PIPE_CONFIG(ADDR_SURF_P4_8x16);
+ tilemode[19] = MICRO_TILE_MODE(ADDR_SURF_THIN_MICRO_TILING) |
+ ARRAY_MODE(ARRAY_2D_TILED_XTHICK) |
+ PIPE_CONFIG(ADDR_SURF_P4_8x16) |
+ BANK_WIDTH(ADDR_SURF_BANK_WIDTH_1) |
+ BANK_HEIGHT(ADDR_SURF_BANK_HEIGHT_1) |
+ MACRO_TILE_ASPECT(ADDR_SURF_MACRO_ASPECT_2) |
+ NUM_BANKS(ADDR_SURF_16_BANK) |
+ TILE_SPLIT(split_equal_to_row_size);
+ tilemode[20] = MICRO_TILE_MODE(ADDR_SURF_THIN_MICRO_TILING) |
+ ARRAY_MODE(ARRAY_2D_TILED_THICK) |
+ PIPE_CONFIG(ADDR_SURF_P4_8x16) |
+ BANK_WIDTH(ADDR_SURF_BANK_WIDTH_1) |
+ BANK_HEIGHT(ADDR_SURF_BANK_HEIGHT_1) |
+ MACRO_TILE_ASPECT(ADDR_SURF_MACRO_ASPECT_2) |
+ NUM_BANKS(ADDR_SURF_16_BANK) |
+ TILE_SPLIT(split_equal_to_row_size);
tilemode[21] = MICRO_TILE_MODE(ADDR_SURF_THIN_MICRO_TILING) |
ARRAY_MODE(ARRAY_2D_TILED_THIN1) |
PIPE_CONFIG(ADDR_SURF_P8_32x32_8x16) |
dev_warn(adev->dev,
"Unknown chip type (%d) in function gfx_v8_0_tiling_mode_table_init() falling through to CHIP_CARRIZO\n",
adev->asic_type);
+ /* fall through */
case CHIP_CARRIZO:
modearray[0] = (ARRAY_MODE(ARRAY_2D_TILED_THIN1) |
#include "ivsrcid/gfx/irqsrcs_gfx_9_0.h"
+#include "amdgpu_ras.h"
+
#define GFX9_NUM_GFX_RINGS 1
#define GFX9_MEC_HPD_SIZE 4096
#define RLCG_UCODE_LOADING_START_ADDRESS 0x00002000L
}
}
+static void gfx_v9_0_check_if_need_gfxoff(struct amdgpu_device *adev)
+{
+ switch (adev->asic_type) {
+ case CHIP_VEGA10:
+ case CHIP_VEGA12:
+ case CHIP_VEGA20:
+ break;
+ case CHIP_RAVEN:
+ if (adev->rev_id >= 0x8 || adev->pdev->device == 0x15d8)
+ break;
+ if ((adev->gfx.rlc_fw_version < 531) ||
+ (adev->gfx.rlc_fw_version == 53815) ||
+ (adev->gfx.rlc_feature_version < 1) ||
+ !adev->gfx.rlc.is_rlc_v2_1)
+ adev->pm.pp_feature &= ~PP_GFXOFF_MASK;
+ break;
+ default:
+ break;
+ }
+}
+
static int gfx_v9_0_init_microcode(struct amdgpu_device *adev)
{
const char *chip_name;
}
out:
+ gfx_v9_0_check_if_need_gfxoff(adev);
gfx_v9_0_check_fw_write_wait(adev);
if (err) {
dev_err(adev->dev,
if (r)
return r;
+ /* ECC error */
+ r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_GRBM_CP, GFX_9_0__SRCID__CP_ECC_ERROR,
+ &adev->gfx.cp_ecc_error_irq);
+ if (r)
+ return r;
+
+ /* FUE error */
+ r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_GRBM_CP, GFX_9_0__SRCID__CP_FUE_ERROR,
+ &adev->gfx.cp_ecc_error_irq);
+ if (r)
+ return r;
+
adev->gfx.gfx_current_status = AMDGPU_GFX_NORMAL_MODE;
gfx_v9_0_scratch_init(adev);
int i;
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ if (amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__GFX) &&
+ adev->gfx.ras_if) {
+ struct ras_common_if *ras_if = adev->gfx.ras_if;
+ struct ras_ih_if ih_info = {
+ .head = *ras_if,
+ };
+
+ amdgpu_ras_debugfs_remove(adev, ras_if);
+ amdgpu_ras_sysfs_remove(adev, ras_if);
+ amdgpu_ras_interrupt_remove_handler(adev, &ih_info);
+ amdgpu_ras_feature_enable(adev, ras_if, 0);
+ kfree(ras_if);
+ }
+
amdgpu_bo_free_kernel(&adev->gds.oa_gfx_bo, NULL, NULL);
amdgpu_bo_free_kernel(&adev->gds.gws_gfx_bo, NULL, NULL);
amdgpu_bo_free_kernel(&adev->gds.gds_gfx_bo, NULL, NULL);
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ amdgpu_irq_put(adev, &adev->gfx.cp_ecc_error_irq, 0);
amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);
amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
return 0;
}
+static int gfx_v9_0_process_ras_data_cb(struct amdgpu_device *adev,
+ struct amdgpu_iv_entry *entry);
+
+static int gfx_v9_0_ecc_late_init(void *handle)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct ras_common_if **ras_if = &adev->gfx.ras_if;
+ struct ras_ih_if ih_info = {
+ .cb = gfx_v9_0_process_ras_data_cb,
+ };
+ struct ras_fs_if fs_info = {
+ .sysfs_name = "gfx_err_count",
+ .debugfs_name = "gfx_err_inject",
+ };
+ struct ras_common_if ras_block = {
+ .block = AMDGPU_RAS_BLOCK__GFX,
+ .type = AMDGPU_RAS_ERROR__MULTI_UNCORRECTABLE,
+ .sub_block_index = 0,
+ .name = "gfx",
+ };
+ int r;
+
+ if (!amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__GFX)) {
+ amdgpu_ras_feature_enable_on_boot(adev, &ras_block, 0);
+ return 0;
+ }
+
+ if (*ras_if)
+ goto resume;
+
+ *ras_if = kmalloc(sizeof(**ras_if), GFP_KERNEL);
+ if (!*ras_if)
+ return -ENOMEM;
+
+ **ras_if = ras_block;
+
+ r = amdgpu_ras_feature_enable_on_boot(adev, *ras_if, 1);
+ if (r)
+ goto feature;
+
+ ih_info.head = **ras_if;
+ fs_info.head = **ras_if;
+
+ r = amdgpu_ras_interrupt_add_handler(adev, &ih_info);
+ if (r)
+ goto interrupt;
+
+ r = amdgpu_ras_debugfs_create(adev, &fs_info);
+ if (r)
+ goto debugfs;
+
+ r = amdgpu_ras_sysfs_create(adev, &fs_info);
+ if (r)
+ goto sysfs;
+resume:
+ r = amdgpu_irq_get(adev, &adev->gfx.cp_ecc_error_irq, 0);
+ if (r)
+ goto irq;
+
+ return 0;
+irq:
+ amdgpu_ras_sysfs_remove(adev, *ras_if);
+sysfs:
+ amdgpu_ras_debugfs_remove(adev, *ras_if);
+debugfs:
+ amdgpu_ras_interrupt_remove_handler(adev, &ih_info);
+interrupt:
+ amdgpu_ras_feature_enable(adev, *ras_if, 0);
+feature:
+ kfree(*ras_if);
+ *ras_if = NULL;
+ return -EINVAL;
+}
+
static int gfx_v9_0_late_init(void *handle)
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
if (r)
return r;
+ r = gfx_v9_0_ecc_late_init(handle);
+ if (r)
+ return r;
+
return 0;
}
return 0;
}
+#define ENABLE_ECC_ON_ME_PIPE(me, pipe) \
+ WREG32_FIELD15(GC, 0, CP_ME##me##_PIPE##pipe##_INT_CNTL,\
+ CP_ECC_ERROR_INT_ENABLE, 1)
+
+#define DISABLE_ECC_ON_ME_PIPE(me, pipe) \
+ WREG32_FIELD15(GC, 0, CP_ME##me##_PIPE##pipe##_INT_CNTL,\
+ CP_ECC_ERROR_INT_ENABLE, 0)
+
+static int gfx_v9_0_set_cp_ecc_error_state(struct amdgpu_device *adev,
+ struct amdgpu_irq_src *source,
+ unsigned type,
+ enum amdgpu_interrupt_state state)
+{
+ switch (state) {
+ case AMDGPU_IRQ_STATE_DISABLE:
+ WREG32_FIELD15(GC, 0, CP_INT_CNTL_RING0,
+ CP_ECC_ERROR_INT_ENABLE, 0);
+ DISABLE_ECC_ON_ME_PIPE(1, 0);
+ DISABLE_ECC_ON_ME_PIPE(1, 1);
+ DISABLE_ECC_ON_ME_PIPE(1, 2);
+ DISABLE_ECC_ON_ME_PIPE(1, 3);
+ break;
+
+ case AMDGPU_IRQ_STATE_ENABLE:
+ WREG32_FIELD15(GC, 0, CP_INT_CNTL_RING0,
+ CP_ECC_ERROR_INT_ENABLE, 1);
+ ENABLE_ECC_ON_ME_PIPE(1, 0);
+ ENABLE_ECC_ON_ME_PIPE(1, 1);
+ ENABLE_ECC_ON_ME_PIPE(1, 2);
+ ENABLE_ECC_ON_ME_PIPE(1, 3);
+ break;
+ default:
+ break;
+ }
+
+ return 0;
+}
+
+
static int gfx_v9_0_set_eop_interrupt_state(struct amdgpu_device *adev,
struct amdgpu_irq_src *src,
unsigned type,
return 0;
}
+static int gfx_v9_0_process_ras_data_cb(struct amdgpu_device *adev,
+ struct amdgpu_iv_entry *entry)
+{
+ /* TODO ue will trigger an interrupt. */
+ kgd2kfd_set_sram_ecc_flag(adev->kfd.dev);
+ amdgpu_ras_reset_gpu(adev, 0);
+ return AMDGPU_RAS_UE;
+}
+
+static int gfx_v9_0_cp_ecc_error_irq(struct amdgpu_device *adev,
+ struct amdgpu_irq_src *source,
+ struct amdgpu_iv_entry *entry)
+{
+ struct ras_common_if *ras_if = adev->gfx.ras_if;
+ struct ras_dispatch_if ih_data = {
+ .entry = entry,
+ };
+
+ if (!ras_if)
+ return 0;
+
+ ih_data.head = *ras_if;
+
+ DRM_ERROR("CP ECC ERROR IRQ\n");
+ amdgpu_ras_interrupt_dispatch(adev, &ih_data);
+ return 0;
+}
+
static const struct amd_ip_funcs gfx_v9_0_ip_funcs = {
.name = "gfx_v9_0",
.early_init = gfx_v9_0_early_init,
.process = gfx_v9_0_priv_inst_irq,
};
+static const struct amdgpu_irq_src_funcs gfx_v9_0_cp_ecc_error_irq_funcs = {
+ .set = gfx_v9_0_set_cp_ecc_error_state,
+ .process = gfx_v9_0_cp_ecc_error_irq,
+};
+
+
static void gfx_v9_0_set_irq_funcs(struct amdgpu_device *adev)
{
adev->gfx.eop_irq.num_types = AMDGPU_CP_IRQ_LAST;
adev->gfx.priv_inst_irq.num_types = 1;
adev->gfx.priv_inst_irq.funcs = &gfx_v9_0_priv_inst_irq_funcs;
+
+ adev->gfx.cp_ecc_error_irq.num_types = 2; /*C5 ECC error and C9 FUE error*/
+ adev->gfx.cp_ecc_error_irq.funcs = &gfx_v9_0_cp_ecc_error_irq_funcs;
}
static void gfx_v9_0_set_rlc_funcs(struct amdgpu_device *adev)
/* XXX for emulation, Refer to closed source code.*/
tmp = REG_SET_FIELD(tmp, VM_L2_CNTL, L2_PDE0_CACHE_TAG_GENERATION_MODE,
0);
- tmp = REG_SET_FIELD(tmp, VM_L2_CNTL, PDE_FAULT_CLASSIFICATION, 1);
+ tmp = REG_SET_FIELD(tmp, VM_L2_CNTL, PDE_FAULT_CLASSIFICATION, 0);
tmp = REG_SET_FIELD(tmp, VM_L2_CNTL, CONTEXT1_IDENTITY_ACCESS_MODE, 1);
tmp = REG_SET_FIELD(tmp, VM_L2_CNTL, IDENTITY_MODE_FRAGMENT_SIZE, 0);
WREG32_SOC15(GC, 0, mmVM_L2_CNTL, tmp);
block_size);
/* Send no-retry XNACK on fault to suppress VM fault storm. */
tmp = REG_SET_FIELD(tmp, VM_CONTEXT1_CNTL,
- RETRY_PERMISSION_OR_INVALID_PAGE_FAULT, 0);
+ RETRY_PERMISSION_OR_INVALID_PAGE_FAULT, 1);
WREG32_SOC15_OFFSET(GC, 0, mmVM_CONTEXT1_CNTL, i, tmp);
WREG32_SOC15_OFFSET(GC, 0, mmVM_CONTEXT1_PAGE_TABLE_START_ADDR_LO32, i*2, 0);
WREG32_SOC15_OFFSET(GC, 0, mmVM_CONTEXT1_PAGE_TABLE_START_ADDR_HI32, i*2, 0);
u64 base = RREG32(mmMC_VM_FB_LOCATION) & 0xFFFF;
base <<= 24;
- amdgpu_gmc_vram_location(adev, &adev->gmc, base);
+ amdgpu_gmc_vram_location(adev, mc, base);
amdgpu_gmc_gart_location(adev, mc);
}
return pd_addr;
}
-static int gmc_v6_0_set_pte_pde(struct amdgpu_device *adev, void *cpu_pt_addr,
- uint32_t gpu_page_idx, uint64_t addr,
- uint64_t flags)
-{
- void __iomem *ptr = (void *)cpu_pt_addr;
- uint64_t value;
-
- value = addr & 0xFFFFFFFFFFFFF000ULL;
- value |= flags;
- writeq(value, ptr + (gpu_page_idx * 8));
-
- return 0;
-}
-
static uint64_t gmc_v6_0_get_vm_pte_flags(struct amdgpu_device *adev,
uint32_t flags)
{
pci_set_consistent_dma_mask(adev->pdev, DMA_BIT_MASK(32));
dev_warn(adev->dev, "amdgpu: No coherent DMA available.\n");
}
- adev->need_swiotlb = drm_get_max_iomem() > ((u64)1 << dma_bits);
+ adev->need_swiotlb = drm_need_swiotlb(dma_bits);
r = gmc_v6_0_init_microcode(adev);
if (r) {
static const struct amdgpu_gmc_funcs gmc_v6_0_gmc_funcs = {
.flush_gpu_tlb = gmc_v6_0_flush_gpu_tlb,
.emit_flush_gpu_tlb = gmc_v6_0_emit_flush_gpu_tlb,
- .set_pte_pde = gmc_v6_0_set_pte_pde,
.set_prt = gmc_v6_0_set_prt,
.get_vm_pde = gmc_v6_0_get_vm_pde,
.get_vm_pte_flags = gmc_v6_0_get_vm_pte_flags
u64 base = RREG32(mmMC_VM_FB_LOCATION) & 0xFFFF;
base <<= 24;
- amdgpu_gmc_vram_location(adev, &adev->gmc, base);
+ amdgpu_gmc_vram_location(adev, mc, base);
amdgpu_gmc_gart_location(adev, mc);
}
amdgpu_ring_emit_wreg(ring, mmIH_VMID_0_LUT + vmid, pasid);
}
-/**
- * gmc_v7_0_set_pte_pde - update the page tables using MMIO
- *
- * @adev: amdgpu_device pointer
- * @cpu_pt_addr: cpu address of the page table
- * @gpu_page_idx: entry in the page table to update
- * @addr: dst addr to write into pte/pde
- * @flags: access flags
- *
- * Update the page tables using the CPU.
- */
-static int gmc_v7_0_set_pte_pde(struct amdgpu_device *adev, void *cpu_pt_addr,
- uint32_t gpu_page_idx, uint64_t addr,
- uint64_t flags)
-{
- void __iomem *ptr = (void *)cpu_pt_addr;
- uint64_t value;
-
- value = addr & 0xFFFFFFFFFFFFF000ULL;
- value |= flags;
- writeq(value, ptr + (gpu_page_idx * 8));
-
- return 0;
-}
-
static uint64_t gmc_v7_0_get_vm_pte_flags(struct amdgpu_device *adev,
uint32_t flags)
{
pci_set_consistent_dma_mask(adev->pdev, DMA_BIT_MASK(32));
pr_warn("amdgpu: No coherent DMA available\n");
}
- adev->need_swiotlb = drm_get_max_iomem() > ((u64)1 << dma_bits);
+ adev->need_swiotlb = drm_need_swiotlb(dma_bits);
r = gmc_v7_0_init_microcode(adev);
if (r) {
.flush_gpu_tlb = gmc_v7_0_flush_gpu_tlb,
.emit_flush_gpu_tlb = gmc_v7_0_emit_flush_gpu_tlb,
.emit_pasid_mapping = gmc_v7_0_emit_pasid_mapping,
- .set_pte_pde = gmc_v7_0_set_pte_pde,
.set_prt = gmc_v7_0_set_prt,
.get_vm_pte_flags = gmc_v7_0_get_vm_pte_flags,
.get_vm_pde = gmc_v7_0_get_vm_pde
base = RREG32(mmMC_VM_FB_LOCATION) & 0xFFFF;
base <<= 24;
- amdgpu_gmc_vram_location(adev, &adev->gmc, base);
+ amdgpu_gmc_vram_location(adev, mc, base);
amdgpu_gmc_gart_location(adev, mc);
}
amdgpu_ring_emit_wreg(ring, mmIH_VMID_0_LUT + vmid, pasid);
}
-/**
- * gmc_v8_0_set_pte_pde - update the page tables using MMIO
- *
- * @adev: amdgpu_device pointer
- * @cpu_pt_addr: cpu address of the page table
- * @gpu_page_idx: entry in the page table to update
- * @addr: dst addr to write into pte/pde
- * @flags: access flags
+/*
+ * PTE format on VI:
+ * 63:40 reserved
+ * 39:12 4k physical page base address
+ * 11:7 fragment
+ * 6 write
+ * 5 read
+ * 4 exe
+ * 3 reserved
+ * 2 snooped
+ * 1 system
+ * 0 valid
*
- * Update the page tables using the CPU.
+ * PDE format on VI:
+ * 63:59 block fragment size
+ * 58:40 reserved
+ * 39:1 physical base address of PTE
+ * bits 5:1 must be 0.
+ * 0 valid
*/
-static int gmc_v8_0_set_pte_pde(struct amdgpu_device *adev, void *cpu_pt_addr,
- uint32_t gpu_page_idx, uint64_t addr,
- uint64_t flags)
-{
- void __iomem *ptr = (void *)cpu_pt_addr;
- uint64_t value;
-
- /*
- * PTE format on VI:
- * 63:40 reserved
- * 39:12 4k physical page base address
- * 11:7 fragment
- * 6 write
- * 5 read
- * 4 exe
- * 3 reserved
- * 2 snooped
- * 1 system
- * 0 valid
- *
- * PDE format on VI:
- * 63:59 block fragment size
- * 58:40 reserved
- * 39:1 physical base address of PTE
- * bits 5:1 must be 0.
- * 0 valid
- */
- value = addr & 0x000000FFFFFFF000ULL;
- value |= flags;
- writeq(value, ptr + (gpu_page_idx * 8));
-
- return 0;
-}
static uint64_t gmc_v8_0_get_vm_pte_flags(struct amdgpu_device *adev,
uint32_t flags)
pci_set_consistent_dma_mask(adev->pdev, DMA_BIT_MASK(32));
pr_warn("amdgpu: No coherent DMA available\n");
}
- adev->need_swiotlb = drm_get_max_iomem() > ((u64)1 << dma_bits);
+ adev->need_swiotlb = drm_need_swiotlb(dma_bits);
r = gmc_v8_0_init_microcode(adev);
if (r) {
.flush_gpu_tlb = gmc_v8_0_flush_gpu_tlb,
.emit_flush_gpu_tlb = gmc_v8_0_emit_flush_gpu_tlb,
.emit_pasid_mapping = gmc_v8_0_emit_pasid_mapping,
- .set_pte_pde = gmc_v8_0_set_pte_pde,
.set_prt = gmc_v8_0_set_prt,
.get_vm_pte_flags = gmc_v8_0_get_vm_pte_flags,
.get_vm_pde = gmc_v8_0_get_vm_pde
#include "ivsrcid/vmc/irqsrcs_vmc_1_0.h"
+#include "amdgpu_ras.h"
+
/* add these here since we already include dce12 headers and these are for DCN */
#define mmHUBP0_DCSURF_PRI_VIEWPORT_DIMENSION 0x055d
#define mmHUBP0_DCSURF_PRI_VIEWPORT_DIMENSION_BASE_IDX 2
SOC15_REG_GOLDEN_VALUE(ATHUB, 0, mmRPB_ARB_CNTL2, 0x00ff00ff, 0x00080008)
};
-/* Ecc related register addresses, (BASE + reg offset) */
-/* Universal Memory Controller caps (may be fused). */
-/* UMCCH:UmcLocalCap */
-#define UMCLOCALCAPS_ADDR0 (0x00014306 + 0x00000000)
-#define UMCLOCALCAPS_ADDR1 (0x00014306 + 0x00000800)
-#define UMCLOCALCAPS_ADDR2 (0x00014306 + 0x00001000)
-#define UMCLOCALCAPS_ADDR3 (0x00014306 + 0x00001800)
-#define UMCLOCALCAPS_ADDR4 (0x00054306 + 0x00000000)
-#define UMCLOCALCAPS_ADDR5 (0x00054306 + 0x00000800)
-#define UMCLOCALCAPS_ADDR6 (0x00054306 + 0x00001000)
-#define UMCLOCALCAPS_ADDR7 (0x00054306 + 0x00001800)
-#define UMCLOCALCAPS_ADDR8 (0x00094306 + 0x00000000)
-#define UMCLOCALCAPS_ADDR9 (0x00094306 + 0x00000800)
-#define UMCLOCALCAPS_ADDR10 (0x00094306 + 0x00001000)
-#define UMCLOCALCAPS_ADDR11 (0x00094306 + 0x00001800)
-#define UMCLOCALCAPS_ADDR12 (0x000d4306 + 0x00000000)
-#define UMCLOCALCAPS_ADDR13 (0x000d4306 + 0x00000800)
-#define UMCLOCALCAPS_ADDR14 (0x000d4306 + 0x00001000)
-#define UMCLOCALCAPS_ADDR15 (0x000d4306 + 0x00001800)
-
-/* Universal Memory Controller Channel config. */
-/* UMCCH:UMC_CONFIG */
-#define UMCCH_UMC_CONFIG_ADDR0 (0x00014040 + 0x00000000)
-#define UMCCH_UMC_CONFIG_ADDR1 (0x00014040 + 0x00000800)
-#define UMCCH_UMC_CONFIG_ADDR2 (0x00014040 + 0x00001000)
-#define UMCCH_UMC_CONFIG_ADDR3 (0x00014040 + 0x00001800)
-#define UMCCH_UMC_CONFIG_ADDR4 (0x00054040 + 0x00000000)
-#define UMCCH_UMC_CONFIG_ADDR5 (0x00054040 + 0x00000800)
-#define UMCCH_UMC_CONFIG_ADDR6 (0x00054040 + 0x00001000)
-#define UMCCH_UMC_CONFIG_ADDR7 (0x00054040 + 0x00001800)
-#define UMCCH_UMC_CONFIG_ADDR8 (0x00094040 + 0x00000000)
-#define UMCCH_UMC_CONFIG_ADDR9 (0x00094040 + 0x00000800)
-#define UMCCH_UMC_CONFIG_ADDR10 (0x00094040 + 0x00001000)
-#define UMCCH_UMC_CONFIG_ADDR11 (0x00094040 + 0x00001800)
-#define UMCCH_UMC_CONFIG_ADDR12 (0x000d4040 + 0x00000000)
-#define UMCCH_UMC_CONFIG_ADDR13 (0x000d4040 + 0x00000800)
-#define UMCCH_UMC_CONFIG_ADDR14 (0x000d4040 + 0x00001000)
-#define UMCCH_UMC_CONFIG_ADDR15 (0x000d4040 + 0x00001800)
-
-/* Universal Memory Controller Channel Ecc config. */
-/* UMCCH:EccCtrl */
-#define UMCCH_ECCCTRL_ADDR0 (0x00014053 + 0x00000000)
-#define UMCCH_ECCCTRL_ADDR1 (0x00014053 + 0x00000800)
-#define UMCCH_ECCCTRL_ADDR2 (0x00014053 + 0x00001000)
-#define UMCCH_ECCCTRL_ADDR3 (0x00014053 + 0x00001800)
-#define UMCCH_ECCCTRL_ADDR4 (0x00054053 + 0x00000000)
-#define UMCCH_ECCCTRL_ADDR5 (0x00054053 + 0x00000800)
-#define UMCCH_ECCCTRL_ADDR6 (0x00054053 + 0x00001000)
-#define UMCCH_ECCCTRL_ADDR7 (0x00054053 + 0x00001800)
-#define UMCCH_ECCCTRL_ADDR8 (0x00094053 + 0x00000000)
-#define UMCCH_ECCCTRL_ADDR9 (0x00094053 + 0x00000800)
-#define UMCCH_ECCCTRL_ADDR10 (0x00094053 + 0x00001000)
-#define UMCCH_ECCCTRL_ADDR11 (0x00094053 + 0x00001800)
-#define UMCCH_ECCCTRL_ADDR12 (0x000d4053 + 0x00000000)
-#define UMCCH_ECCCTRL_ADDR13 (0x000d4053 + 0x00000800)
-#define UMCCH_ECCCTRL_ADDR14 (0x000d4053 + 0x00001000)
-#define UMCCH_ECCCTRL_ADDR15 (0x000d4053 + 0x00001800)
-
-static const uint32_t ecc_umclocalcap_addrs[] = {
- UMCLOCALCAPS_ADDR0,
- UMCLOCALCAPS_ADDR1,
- UMCLOCALCAPS_ADDR2,
- UMCLOCALCAPS_ADDR3,
- UMCLOCALCAPS_ADDR4,
- UMCLOCALCAPS_ADDR5,
- UMCLOCALCAPS_ADDR6,
- UMCLOCALCAPS_ADDR7,
- UMCLOCALCAPS_ADDR8,
- UMCLOCALCAPS_ADDR9,
- UMCLOCALCAPS_ADDR10,
- UMCLOCALCAPS_ADDR11,
- UMCLOCALCAPS_ADDR12,
- UMCLOCALCAPS_ADDR13,
- UMCLOCALCAPS_ADDR14,
- UMCLOCALCAPS_ADDR15,
+static const uint32_t ecc_umc_mcumc_ctrl_addrs[] = {
+ (0x000143c0 + 0x00000000),
+ (0x000143c0 + 0x00000800),
+ (0x000143c0 + 0x00001000),
+ (0x000143c0 + 0x00001800),
+ (0x000543c0 + 0x00000000),
+ (0x000543c0 + 0x00000800),
+ (0x000543c0 + 0x00001000),
+ (0x000543c0 + 0x00001800),
+ (0x000943c0 + 0x00000000),
+ (0x000943c0 + 0x00000800),
+ (0x000943c0 + 0x00001000),
+ (0x000943c0 + 0x00001800),
+ (0x000d43c0 + 0x00000000),
+ (0x000d43c0 + 0x00000800),
+ (0x000d43c0 + 0x00001000),
+ (0x000d43c0 + 0x00001800),
+ (0x001143c0 + 0x00000000),
+ (0x001143c0 + 0x00000800),
+ (0x001143c0 + 0x00001000),
+ (0x001143c0 + 0x00001800),
+ (0x001543c0 + 0x00000000),
+ (0x001543c0 + 0x00000800),
+ (0x001543c0 + 0x00001000),
+ (0x001543c0 + 0x00001800),
+ (0x001943c0 + 0x00000000),
+ (0x001943c0 + 0x00000800),
+ (0x001943c0 + 0x00001000),
+ (0x001943c0 + 0x00001800),
+ (0x001d43c0 + 0x00000000),
+ (0x001d43c0 + 0x00000800),
+ (0x001d43c0 + 0x00001000),
+ (0x001d43c0 + 0x00001800),
};
-static const uint32_t ecc_umcch_umc_config_addrs[] = {
- UMCCH_UMC_CONFIG_ADDR0,
- UMCCH_UMC_CONFIG_ADDR1,
- UMCCH_UMC_CONFIG_ADDR2,
- UMCCH_UMC_CONFIG_ADDR3,
- UMCCH_UMC_CONFIG_ADDR4,
- UMCCH_UMC_CONFIG_ADDR5,
- UMCCH_UMC_CONFIG_ADDR6,
- UMCCH_UMC_CONFIG_ADDR7,
- UMCCH_UMC_CONFIG_ADDR8,
- UMCCH_UMC_CONFIG_ADDR9,
- UMCCH_UMC_CONFIG_ADDR10,
- UMCCH_UMC_CONFIG_ADDR11,
- UMCCH_UMC_CONFIG_ADDR12,
- UMCCH_UMC_CONFIG_ADDR13,
- UMCCH_UMC_CONFIG_ADDR14,
- UMCCH_UMC_CONFIG_ADDR15,
+static const uint32_t ecc_umc_mcumc_ctrl_mask_addrs[] = {
+ (0x000143e0 + 0x00000000),
+ (0x000143e0 + 0x00000800),
+ (0x000143e0 + 0x00001000),
+ (0x000143e0 + 0x00001800),
+ (0x000543e0 + 0x00000000),
+ (0x000543e0 + 0x00000800),
+ (0x000543e0 + 0x00001000),
+ (0x000543e0 + 0x00001800),
+ (0x000943e0 + 0x00000000),
+ (0x000943e0 + 0x00000800),
+ (0x000943e0 + 0x00001000),
+ (0x000943e0 + 0x00001800),
+ (0x000d43e0 + 0x00000000),
+ (0x000d43e0 + 0x00000800),
+ (0x000d43e0 + 0x00001000),
+ (0x000d43e0 + 0x00001800),
+ (0x001143e0 + 0x00000000),
+ (0x001143e0 + 0x00000800),
+ (0x001143e0 + 0x00001000),
+ (0x001143e0 + 0x00001800),
+ (0x001543e0 + 0x00000000),
+ (0x001543e0 + 0x00000800),
+ (0x001543e0 + 0x00001000),
+ (0x001543e0 + 0x00001800),
+ (0x001943e0 + 0x00000000),
+ (0x001943e0 + 0x00000800),
+ (0x001943e0 + 0x00001000),
+ (0x001943e0 + 0x00001800),
+ (0x001d43e0 + 0x00000000),
+ (0x001d43e0 + 0x00000800),
+ (0x001d43e0 + 0x00001000),
+ (0x001d43e0 + 0x00001800),
};
-static const uint32_t ecc_umcch_eccctrl_addrs[] = {
- UMCCH_ECCCTRL_ADDR0,
- UMCCH_ECCCTRL_ADDR1,
- UMCCH_ECCCTRL_ADDR2,
- UMCCH_ECCCTRL_ADDR3,
- UMCCH_ECCCTRL_ADDR4,
- UMCCH_ECCCTRL_ADDR5,
- UMCCH_ECCCTRL_ADDR6,
- UMCCH_ECCCTRL_ADDR7,
- UMCCH_ECCCTRL_ADDR8,
- UMCCH_ECCCTRL_ADDR9,
- UMCCH_ECCCTRL_ADDR10,
- UMCCH_ECCCTRL_ADDR11,
- UMCCH_ECCCTRL_ADDR12,
- UMCCH_ECCCTRL_ADDR13,
- UMCCH_ECCCTRL_ADDR14,
- UMCCH_ECCCTRL_ADDR15,
+static const uint32_t ecc_umc_mcumc_status_addrs[] = {
+ (0x000143c2 + 0x00000000),
+ (0x000143c2 + 0x00000800),
+ (0x000143c2 + 0x00001000),
+ (0x000143c2 + 0x00001800),
+ (0x000543c2 + 0x00000000),
+ (0x000543c2 + 0x00000800),
+ (0x000543c2 + 0x00001000),
+ (0x000543c2 + 0x00001800),
+ (0x000943c2 + 0x00000000),
+ (0x000943c2 + 0x00000800),
+ (0x000943c2 + 0x00001000),
+ (0x000943c2 + 0x00001800),
+ (0x000d43c2 + 0x00000000),
+ (0x000d43c2 + 0x00000800),
+ (0x000d43c2 + 0x00001000),
+ (0x000d43c2 + 0x00001800),
+ (0x001143c2 + 0x00000000),
+ (0x001143c2 + 0x00000800),
+ (0x001143c2 + 0x00001000),
+ (0x001143c2 + 0x00001800),
+ (0x001543c2 + 0x00000000),
+ (0x001543c2 + 0x00000800),
+ (0x001543c2 + 0x00001000),
+ (0x001543c2 + 0x00001800),
+ (0x001943c2 + 0x00000000),
+ (0x001943c2 + 0x00000800),
+ (0x001943c2 + 0x00001000),
+ (0x001943c2 + 0x00001800),
+ (0x001d43c2 + 0x00000000),
+ (0x001d43c2 + 0x00000800),
+ (0x001d43c2 + 0x00001000),
+ (0x001d43c2 + 0x00001800),
};
+static int gmc_v9_0_ecc_interrupt_state(struct amdgpu_device *adev,
+ struct amdgpu_irq_src *src,
+ unsigned type,
+ enum amdgpu_interrupt_state state)
+{
+ u32 bits, i, tmp, reg;
+
+ bits = 0x7f;
+
+ switch (state) {
+ case AMDGPU_IRQ_STATE_DISABLE:
+ for (i = 0; i < ARRAY_SIZE(ecc_umc_mcumc_ctrl_addrs); i++) {
+ reg = ecc_umc_mcumc_ctrl_addrs[i];
+ tmp = RREG32(reg);
+ tmp &= ~bits;
+ WREG32(reg, tmp);
+ }
+ for (i = 0; i < ARRAY_SIZE(ecc_umc_mcumc_ctrl_mask_addrs); i++) {
+ reg = ecc_umc_mcumc_ctrl_mask_addrs[i];
+ tmp = RREG32(reg);
+ tmp &= ~bits;
+ WREG32(reg, tmp);
+ }
+ break;
+ case AMDGPU_IRQ_STATE_ENABLE:
+ for (i = 0; i < ARRAY_SIZE(ecc_umc_mcumc_ctrl_addrs); i++) {
+ reg = ecc_umc_mcumc_ctrl_addrs[i];
+ tmp = RREG32(reg);
+ tmp |= bits;
+ WREG32(reg, tmp);
+ }
+ for (i = 0; i < ARRAY_SIZE(ecc_umc_mcumc_ctrl_mask_addrs); i++) {
+ reg = ecc_umc_mcumc_ctrl_mask_addrs[i];
+ tmp = RREG32(reg);
+ tmp |= bits;
+ WREG32(reg, tmp);
+ }
+ break;
+ default:
+ break;
+ }
+
+ return 0;
+}
+
+static int gmc_v9_0_process_ras_data_cb(struct amdgpu_device *adev,
+ struct amdgpu_iv_entry *entry)
+{
+ kgd2kfd_set_sram_ecc_flag(adev->kfd.dev);
+ amdgpu_ras_reset_gpu(adev, 0);
+ return AMDGPU_RAS_UE;
+}
+
+static int gmc_v9_0_process_ecc_irq(struct amdgpu_device *adev,
+ struct amdgpu_irq_src *source,
+ struct amdgpu_iv_entry *entry)
+{
+ struct ras_common_if *ras_if = adev->gmc.ras_if;
+ struct ras_dispatch_if ih_data = {
+ .entry = entry,
+ };
+
+ if (!ras_if)
+ return 0;
+
+ ih_data.head = *ras_if;
+
+ amdgpu_ras_interrupt_dispatch(adev, &ih_data);
+ return 0;
+}
+
static int gmc_v9_0_vm_fault_interrupt_state(struct amdgpu_device *adev,
struct amdgpu_irq_src *src,
unsigned type,
return 0;
}
-/**
- * vega10_ih_prescreen_iv - prescreen an interrupt vector
- *
- * @adev: amdgpu_device pointer
- *
- * Returns true if the interrupt vector should be further processed.
- */
-static bool gmc_v9_0_prescreen_iv(struct amdgpu_device *adev,
- struct amdgpu_iv_entry *entry,
- uint64_t addr)
-{
- struct amdgpu_vm *vm;
- u64 key;
- int r;
-
- /* No PASID, can't identify faulting process */
- if (!entry->pasid)
- return true;
-
- /* Not a retry fault */
- if (!(entry->src_data[1] & 0x80))
- return true;
-
- /* Track retry faults in per-VM fault FIFO. */
- spin_lock(&adev->vm_manager.pasid_lock);
- vm = idr_find(&adev->vm_manager.pasid_idr, entry->pasid);
- if (!vm) {
- /* VM not found, process it normally */
- spin_unlock(&adev->vm_manager.pasid_lock);
- return true;
- }
-
- key = AMDGPU_VM_FAULT(entry->pasid, addr);
- r = amdgpu_vm_add_fault(vm->fault_hash, key);
-
- /* Hash table is full or the fault is already being processed,
- * ignore further page faults
- */
- if (r != 0) {
- spin_unlock(&adev->vm_manager.pasid_lock);
- return false;
- }
- /* No locking required with single writer and single reader */
- r = kfifo_put(&vm->faults, key);
- if (!r) {
- /* FIFO is full. Ignore it until there is space */
- amdgpu_vm_clear_fault(vm->fault_hash, key);
- spin_unlock(&adev->vm_manager.pasid_lock);
- return false;
- }
-
- spin_unlock(&adev->vm_manager.pasid_lock);
- /* It's the first fault for this address, process it normally */
- return true;
-}
-
static int gmc_v9_0_process_interrupt(struct amdgpu_device *adev,
struct amdgpu_irq_src *source,
struct amdgpu_iv_entry *entry)
addr = (u64)entry->src_data[0] << 12;
addr |= ((u64)entry->src_data[1] & 0xf) << 44;
- if (!gmc_v9_0_prescreen_iv(adev, entry, addr))
+ if (retry_fault && amdgpu_gmc_filter_faults(adev, addr, entry->pasid,
+ entry->timestamp))
return 1; /* This also prevents sending it to KFD */
+ /* If it's the first fault for this address, process it normally */
if (!amdgpu_sriov_vf(adev)) {
status = RREG32(hub->vm_l2_pro_fault_status);
WREG32_P(hub->vm_l2_pro_fault_cntl, 1, ~1);
.process = gmc_v9_0_process_interrupt,
};
+
+static const struct amdgpu_irq_src_funcs gmc_v9_0_ecc_funcs = {
+ .set = gmc_v9_0_ecc_interrupt_state,
+ .process = gmc_v9_0_process_ecc_irq,
+};
+
static void gmc_v9_0_set_irq_funcs(struct amdgpu_device *adev)
{
adev->gmc.vm_fault.num_types = 1;
adev->gmc.vm_fault.funcs = &gmc_v9_0_irq_funcs;
+
+ adev->gmc.ecc_irq.num_types = 1;
+ adev->gmc.ecc_irq.funcs = &gmc_v9_0_ecc_funcs;
}
static uint32_t gmc_v9_0_get_invalidate_req(unsigned int vmid,
amdgpu_ring_emit_wreg(ring, reg, pasid);
}
-/**
- * gmc_v9_0_set_pte_pde - update the page tables using MMIO
- *
- * @adev: amdgpu_device pointer
- * @cpu_pt_addr: cpu address of the page table
- * @gpu_page_idx: entry in the page table to update
- * @addr: dst addr to write into pte/pde
- * @flags: access flags
+/*
+ * PTE format on VEGA 10:
+ * 63:59 reserved
+ * 58:57 mtype
+ * 56 F
+ * 55 L
+ * 54 P
+ * 53 SW
+ * 52 T
+ * 50:48 reserved
+ * 47:12 4k physical page base address
+ * 11:7 fragment
+ * 6 write
+ * 5 read
+ * 4 exe
+ * 3 Z
+ * 2 snooped
+ * 1 system
+ * 0 valid
*
- * Update the page tables using the CPU.
+ * PDE format on VEGA 10:
+ * 63:59 block fragment size
+ * 58:55 reserved
+ * 54 P
+ * 53:48 reserved
+ * 47:6 physical base address of PD or PTE
+ * 5:3 reserved
+ * 2 C
+ * 1 system
+ * 0 valid
*/
-static int gmc_v9_0_set_pte_pde(struct amdgpu_device *adev, void *cpu_pt_addr,
- uint32_t gpu_page_idx, uint64_t addr,
- uint64_t flags)
-{
- void __iomem *ptr = (void *)cpu_pt_addr;
- uint64_t value;
-
- /*
- * PTE format on VEGA 10:
- * 63:59 reserved
- * 58:57 mtype
- * 56 F
- * 55 L
- * 54 P
- * 53 SW
- * 52 T
- * 50:48 reserved
- * 47:12 4k physical page base address
- * 11:7 fragment
- * 6 write
- * 5 read
- * 4 exe
- * 3 Z
- * 2 snooped
- * 1 system
- * 0 valid
- *
- * PDE format on VEGA 10:
- * 63:59 block fragment size
- * 58:55 reserved
- * 54 P
- * 53:48 reserved
- * 47:6 physical base address of PD or PTE
- * 5:3 reserved
- * 2 C
- * 1 system
- * 0 valid
- */
-
- /*
- * The following is for PTE only. GART does not have PDEs.
- */
- value = addr & 0x0000FFFFFFFFF000ULL;
- value |= flags;
- writeq(value, ptr + (gpu_page_idx * 8));
- return 0;
-}
static uint64_t gmc_v9_0_get_vm_pte_flags(struct amdgpu_device *adev,
uint32_t flags)
.flush_gpu_tlb = gmc_v9_0_flush_gpu_tlb,
.emit_flush_gpu_tlb = gmc_v9_0_emit_flush_gpu_tlb,
.emit_pasid_mapping = gmc_v9_0_emit_pasid_mapping,
- .set_pte_pde = gmc_v9_0_set_pte_pde,
.get_vm_pte_flags = gmc_v9_0_get_vm_pte_flags,
.get_vm_pde = gmc_v9_0_get_vm_pde
};
return 0;
}
-static int gmc_v9_0_ecc_available(struct amdgpu_device *adev)
-{
- uint32_t reg_val;
- uint32_t reg_addr;
- uint32_t field_val;
- size_t i;
- uint32_t fv2;
- size_t lost_sheep;
-
- DRM_DEBUG("ecc: gmc_v9_0_ecc_available()\n");
-
- lost_sheep = 0;
- for (i = 0; i < ARRAY_SIZE(ecc_umclocalcap_addrs); ++i) {
- reg_addr = ecc_umclocalcap_addrs[i];
- DRM_DEBUG("ecc: "
- "UMCCH_UmcLocalCap[%zu]: reg_addr: 0x%08x\n",
- i, reg_addr);
- reg_val = RREG32(reg_addr);
- field_val = REG_GET_FIELD(reg_val, UMCCH0_0_UmcLocalCap,
- EccDis);
- DRM_DEBUG("ecc: "
- "reg_val: 0x%08x, "
- "EccDis: 0x%08x, ",
- reg_val, field_val);
- if (field_val) {
- DRM_ERROR("ecc: UmcLocalCap:EccDis is set.\n");
- ++lost_sheep;
- }
- }
-
- for (i = 0; i < ARRAY_SIZE(ecc_umcch_umc_config_addrs); ++i) {
- reg_addr = ecc_umcch_umc_config_addrs[i];
- DRM_DEBUG("ecc: "
- "UMCCH0_0_UMC_CONFIG[%zu]: reg_addr: 0x%08x",
- i, reg_addr);
- reg_val = RREG32(reg_addr);
- field_val = REG_GET_FIELD(reg_val, UMCCH0_0_UMC_CONFIG,
- DramReady);
- DRM_DEBUG("ecc: "
- "reg_val: 0x%08x, "
- "DramReady: 0x%08x\n",
- reg_val, field_val);
-
- if (!field_val) {
- DRM_ERROR("ecc: UMC_CONFIG:DramReady is not set.\n");
- ++lost_sheep;
- }
- }
-
- for (i = 0; i < ARRAY_SIZE(ecc_umcch_eccctrl_addrs); ++i) {
- reg_addr = ecc_umcch_eccctrl_addrs[i];
- DRM_DEBUG("ecc: "
- "UMCCH_EccCtrl[%zu]: reg_addr: 0x%08x, ",
- i, reg_addr);
- reg_val = RREG32(reg_addr);
- field_val = REG_GET_FIELD(reg_val, UMCCH0_0_EccCtrl,
- WrEccEn);
- fv2 = REG_GET_FIELD(reg_val, UMCCH0_0_EccCtrl,
- RdEccEn);
- DRM_DEBUG("ecc: "
- "reg_val: 0x%08x, "
- "WrEccEn: 0x%08x, "
- "RdEccEn: 0x%08x\n",
- reg_val, field_val, fv2);
-
- if (!field_val) {
- DRM_DEBUG("ecc: WrEccEn is not set\n");
- ++lost_sheep;
- }
- if (!fv2) {
- DRM_DEBUG("ecc: RdEccEn is not set\n");
- ++lost_sheep;
- }
- }
-
- DRM_DEBUG("ecc: lost_sheep: %zu\n", lost_sheep);
- return lost_sheep == 0;
-}
-
static bool gmc_v9_0_keep_stolen_memory(struct amdgpu_device *adev)
{
return 0;
}
-static int gmc_v9_0_late_init(void *handle)
+static int gmc_v9_0_ecc_late_init(void *handle)
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct ras_common_if **ras_if = &adev->gmc.ras_if;
+ struct ras_ih_if ih_info = {
+ .cb = gmc_v9_0_process_ras_data_cb,
+ };
+ struct ras_fs_if fs_info = {
+ .sysfs_name = "umc_err_count",
+ .debugfs_name = "umc_err_inject",
+ };
+ struct ras_common_if ras_block = {
+ .block = AMDGPU_RAS_BLOCK__UMC,
+ .type = AMDGPU_RAS_ERROR__MULTI_UNCORRECTABLE,
+ .sub_block_index = 0,
+ .name = "umc",
+ };
int r;
+ if (!amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__UMC)) {
+ amdgpu_ras_feature_enable_on_boot(adev, &ras_block, 0);
+ return 0;
+ }
+ /* handle resume path. */
+ if (*ras_if)
+ goto resume;
+
+ *ras_if = kmalloc(sizeof(**ras_if), GFP_KERNEL);
+ if (!*ras_if)
+ return -ENOMEM;
+
+ **ras_if = ras_block;
+
+ r = amdgpu_ras_feature_enable_on_boot(adev, *ras_if, 1);
+ if (r)
+ goto feature;
+
+ ih_info.head = **ras_if;
+ fs_info.head = **ras_if;
+
+ r = amdgpu_ras_interrupt_add_handler(adev, &ih_info);
+ if (r)
+ goto interrupt;
+
+ r = amdgpu_ras_debugfs_create(adev, &fs_info);
+ if (r)
+ goto debugfs;
+
+ r = amdgpu_ras_sysfs_create(adev, &fs_info);
+ if (r)
+ goto sysfs;
+resume:
+ r = amdgpu_irq_get(adev, &adev->gmc.ecc_irq, 0);
+ if (r)
+ goto irq;
+
+ return 0;
+irq:
+ amdgpu_ras_sysfs_remove(adev, *ras_if);
+sysfs:
+ amdgpu_ras_debugfs_remove(adev, *ras_if);
+debugfs:
+ amdgpu_ras_interrupt_remove_handler(adev, &ih_info);
+interrupt:
+ amdgpu_ras_feature_enable(adev, *ras_if, 0);
+feature:
+ kfree(*ras_if);
+ *ras_if = NULL;
+ return -EINVAL;
+}
+
+
+static int gmc_v9_0_late_init(void *handle)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ bool r;
+
if (!gmc_v9_0_keep_stolen_memory(adev))
amdgpu_bo_late_init(adev);
r = gmc_v9_0_allocate_vm_inv_eng(adev);
if (r)
return r;
+ /* Check if ecc is available */
+ if (!amdgpu_sriov_vf(adev)) {
+ switch (adev->asic_type) {
+ case CHIP_VEGA10:
+ case CHIP_VEGA20:
+ r = amdgpu_atomfirmware_mem_ecc_supported(adev);
+ if (!r) {
+ DRM_INFO("ECC is not present.\n");
+ if (adev->df_funcs->enable_ecc_force_par_wr_rmw)
+ adev->df_funcs->enable_ecc_force_par_wr_rmw(adev, false);
+ } else {
+ DRM_INFO("ECC is active.\n");
+ }
- if (adev->asic_type == CHIP_VEGA10 && !amdgpu_sriov_vf(adev)) {
- r = gmc_v9_0_ecc_available(adev);
- if (r == 1) {
- DRM_INFO("ECC is active.\n");
- } else if (r == 0) {
- DRM_INFO("ECC is not present.\n");
- adev->df_funcs->enable_ecc_force_par_wr_rmw(adev, false);
- } else {
- DRM_ERROR("gmc_v9_0_ecc_available() failed. r: %d\n", r);
- return r;
+ r = amdgpu_atomfirmware_sram_ecc_supported(adev);
+ if (!r) {
+ DRM_INFO("SRAM ECC is not present.\n");
+ } else {
+ DRM_INFO("SRAM ECC is active.\n");
+ }
+ break;
+ default:
+ break;
}
}
+ r = gmc_v9_0_ecc_late_init(handle);
+ if (r)
+ return r;
+
return amdgpu_irq_get(adev, &adev->gmc.vm_fault, 0);
}
base = mmhub_v1_0_get_fb_location(adev);
/* add the xgmi offset of the physical node */
base += adev->gmc.xgmi.physical_node_id * adev->gmc.xgmi.node_segment_size;
- amdgpu_gmc_vram_location(adev, &adev->gmc, base);
+ amdgpu_gmc_vram_location(adev, mc, base);
amdgpu_gmc_gart_location(adev, mc);
if (!amdgpu_sriov_vf(adev))
amdgpu_gmc_agp_location(adev, mc);
if (r)
return r;
+ /* interrupt sent to DF. */
+ r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_DF, 0,
+ &adev->gmc.ecc_irq);
+ if (r)
+ return r;
+
/* Set the internal MC address mask
* This is the max address of the GPU's
* internal address space.
pci_set_consistent_dma_mask(adev->pdev, DMA_BIT_MASK(32));
printk(KERN_WARNING "amdgpu: No coherent DMA available.\n");
}
- adev->need_swiotlb = drm_get_max_iomem() > ((u64)1 << dma_bits);
+ adev->need_swiotlb = drm_need_swiotlb(dma_bits);
if (adev->gmc.xgmi.supported) {
r = gfxhub_v1_1_get_xgmi_info(adev);
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ if (amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__UMC) &&
+ adev->gmc.ras_if) {
+ struct ras_common_if *ras_if = adev->gmc.ras_if;
+ struct ras_ih_if ih_info = {
+ .head = *ras_if,
+ };
+
+ /*remove fs first*/
+ amdgpu_ras_debugfs_remove(adev, ras_if);
+ amdgpu_ras_sysfs_remove(adev, ras_if);
+ /*remove the IH*/
+ amdgpu_ras_interrupt_remove_handler(adev, &ih_info);
+ amdgpu_ras_feature_enable(adev, ras_if, 0);
+ kfree(ras_if);
+ }
+
amdgpu_gem_force_release(adev);
amdgpu_vm_manager_fini(adev);
return 0;
}
+ amdgpu_irq_put(adev, &adev->gmc.ecc_irq, 0);
amdgpu_irq_put(adev, &adev->gmc.vm_fault, 0);
gmc_v9_0_gart_disable(adev);
pi->caps_tcp_ramping = true;
}
- if (adev->powerplay.pp_feature & PP_SCLK_DEEP_SLEEP_MASK)
+ if (adev->pm.pp_feature & PP_SCLK_DEEP_SLEEP_MASK)
pi->caps_sclk_ds = true;
else
pi->caps_sclk_ds = false;
/* XXX for emulation, Refer to closed source code.*/
tmp = REG_SET_FIELD(tmp, VM_L2_CNTL, L2_PDE0_CACHE_TAG_GENERATION_MODE,
0);
- tmp = REG_SET_FIELD(tmp, VM_L2_CNTL, PDE_FAULT_CLASSIFICATION, 1);
+ tmp = REG_SET_FIELD(tmp, VM_L2_CNTL, PDE_FAULT_CLASSIFICATION, 0);
tmp = REG_SET_FIELD(tmp, VM_L2_CNTL, CONTEXT1_IDENTITY_ACCESS_MODE, 1);
tmp = REG_SET_FIELD(tmp, VM_L2_CNTL, IDENTITY_MODE_FRAGMENT_SIZE, 0);
WREG32_SOC15(MMHUB, 0, mmVM_L2_CNTL, tmp);
block_size);
/* Send no-retry XNACK on fault to suppress VM fault storm. */
tmp = REG_SET_FIELD(tmp, VM_CONTEXT1_CNTL,
- RETRY_PERMISSION_OR_INVALID_PAGE_FAULT, 0);
+ RETRY_PERMISSION_OR_INVALID_PAGE_FAULT, 1);
WREG32_SOC15_OFFSET(MMHUB, 0, mmVM_CONTEXT1_CNTL, i, tmp);
WREG32_SOC15_OFFSET(MMHUB, 0, mmVM_CONTEXT1_PAGE_TABLE_START_ADDR_LO32, i*2, 0);
WREG32_SOC15_OFFSET(MMHUB, 0, mmVM_CONTEXT1_PAGE_TABLE_START_ADDR_HI32, i*2, 0);
xgpu_ai_mailbox_set_valid(adev, false);
}
+static int xgpu_ai_get_pp_clk(struct amdgpu_device *adev, u32 type, char *buf)
+{
+ int r = 0;
+ u32 req, val, size;
+
+ if (!amdgim_is_hwperf(adev) || buf == NULL)
+ return -EBADRQC;
+
+ switch(type) {
+ case PP_SCLK:
+ req = IDH_IRQ_GET_PP_SCLK;
+ break;
+ case PP_MCLK:
+ req = IDH_IRQ_GET_PP_MCLK;
+ break;
+ default:
+ return -EBADRQC;
+ }
+
+ mutex_lock(&adev->virt.dpm_mutex);
+
+ xgpu_ai_mailbox_trans_msg(adev, req, 0, 0, 0);
+
+ r = xgpu_ai_poll_msg(adev, IDH_SUCCESS);
+ if (!r && adev->fw_vram_usage.va != NULL) {
+ val = RREG32_NO_KIQ(
+ SOC15_REG_OFFSET(NBIO, 0,
+ mmBIF_BX_PF0_MAILBOX_MSGBUF_RCV_DW1));
+ size = strnlen((((char *)adev->virt.fw_reserve.p_pf2vf) +
+ val), PAGE_SIZE);
+
+ if (size < PAGE_SIZE)
+ strcpy(buf,((char *)adev->virt.fw_reserve.p_pf2vf + val));
+ else
+ size = 0;
+
+ r = size;
+ goto out;
+ }
+
+ r = xgpu_ai_poll_msg(adev, IDH_FAIL);
+ if(r)
+ pr_info("%s DPM request failed",
+ (type == PP_SCLK)? "SCLK" : "MCLK");
+
+out:
+ mutex_unlock(&adev->virt.dpm_mutex);
+ return r;
+}
+
+static int xgpu_ai_force_dpm_level(struct amdgpu_device *adev, u32 level)
+{
+ int r = 0;
+ u32 req = IDH_IRQ_FORCE_DPM_LEVEL;
+
+ if (!amdgim_is_hwperf(adev))
+ return -EBADRQC;
+
+ mutex_lock(&adev->virt.dpm_mutex);
+ xgpu_ai_mailbox_trans_msg(adev, req, level, 0, 0);
+
+ r = xgpu_ai_poll_msg(adev, IDH_SUCCESS);
+ if (!r)
+ goto out;
+
+ r = xgpu_ai_poll_msg(adev, IDH_FAIL);
+ if (!r)
+ pr_info("DPM request failed");
+ else
+ pr_info("Mailbox is broken");
+
+out:
+ mutex_unlock(&adev->virt.dpm_mutex);
+ return r;
+}
+
static int xgpu_ai_send_access_requests(struct amdgpu_device *adev,
enum idh_request req)
{
.reset_gpu = xgpu_ai_request_reset,
.wait_reset = NULL,
.trans_msg = xgpu_ai_mailbox_trans_msg,
+ .get_pp_clk = xgpu_ai_get_pp_clk,
+ .force_dpm_level = xgpu_ai_force_dpm_level,
};
IDH_REL_GPU_FINI_ACCESS,
IDH_REQ_GPU_RESET_ACCESS,
+ IDH_IRQ_FORCE_DPM_LEVEL = 10,
+ IDH_IRQ_GET_PP_SCLK,
+ IDH_IRQ_GET_PP_MCLK,
+
IDH_LOG_VF_ERROR = 200,
};
IDH_READY_TO_ACCESS_GPU,
IDH_FLR_NOTIFICATION,
IDH_FLR_NOTIFICATION_CMPL,
+ IDH_SUCCESS,
+ IDH_FAIL,
IDH_EVENT_MAX
};
/* wait until RCV_MSG become 3 */
if (xgpu_vi_poll_msg(adev, IDH_FLR_NOTIFICATION_CMPL)) {
- pr_err("failed to recieve FLR_CMPL\n");
+ pr_err("failed to receive FLR_CMPL\n");
return;
}
if (use_doorbell) {
ih_doorbell_range = REG_SET_FIELD(ih_doorbell_range, BIF_IH_DOORBELL_RANGE, OFFSET, doorbell_index);
- ih_doorbell_range = REG_SET_FIELD(ih_doorbell_range, BIF_IH_DOORBELL_RANGE, SIZE, 2);
+ ih_doorbell_range = REG_SET_FIELD(ih_doorbell_range,
+ BIF_IH_DOORBELL_RANGE, SIZE, 6);
} else
ih_doorbell_range = REG_SET_FIELD(ih_doorbell_range, BIF_IH_DOORBELL_RANGE, SIZE, 0);
GFX_CTRL_CMD_ID_ENABLE_INT = 0x00050000, /* enable PSP-to-Gfx interrupt */
GFX_CTRL_CMD_ID_DISABLE_INT = 0x00060000, /* disable PSP-to-Gfx interrupt */
GFX_CTRL_CMD_ID_MODE1_RST = 0x00070000, /* trigger the Mode 1 reset */
+ GFX_CTRL_CMD_ID_GBR_IH_SET = 0x00080000, /* set Gbr IH_RB_CNTL registers */
GFX_CTRL_CMD_ID_CONSUME_CMD = 0x000A0000, /* send interrupt to psp for updating write pointer of vf */
GFX_CTRL_CMD_ID_DESTROY_GPCOM_RING = 0x000C0000, /* destroy GPCOM ring */
#include "sdma0/sdma0_4_0_offset.h"
#include "nbio/nbio_7_4_offset.h"
+#include "oss/osssys_4_0_offset.h"
+#include "oss/osssys_4_0_sh_mask.h"
+
MODULE_FIRMWARE("amdgpu/vega20_sos.bin");
MODULE_FIRMWARE("amdgpu/vega20_asd.bin");
MODULE_FIRMWARE("amdgpu/vega20_ta.bin");
adev->psp.ta_xgmi_ucode_size = le32_to_cpu(ta_hdr->ta_xgmi_size_bytes);
adev->psp.ta_xgmi_start_addr = (uint8_t *)ta_hdr +
le32_to_cpu(ta_hdr->header.ucode_array_offset_bytes);
+
+ adev->psp.ta_fw_version = le32_to_cpu(ta_hdr->header.ucode_version);
+
+ adev->psp.ta_ras_ucode_version = le32_to_cpu(ta_hdr->ta_ras_ucode_version);
+ adev->psp.ta_ras_ucode_size = le32_to_cpu(ta_hdr->ta_ras_size_bytes);
+ adev->psp.ta_ras_start_addr = (uint8_t *)adev->psp.ta_xgmi_start_addr +
+ le32_to_cpu(ta_hdr->ta_ras_offset_bytes);
}
return 0;
return ret;
}
+static void psp_v11_0_reroute_ih(struct psp_context *psp)
+{
+ struct amdgpu_device *adev = psp->adev;
+ uint32_t tmp;
+
+ /* Change IH ring for VMC */
+ tmp = REG_SET_FIELD(0, IH_CLIENT_CFG_DATA, CREDIT_RETURN_ADDR, 0x1244b);
+ tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, CLIENT_TYPE, 1);
+ tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, RING_ID, 1);
+
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, 3);
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, tmp);
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, GFX_CTRL_CMD_ID_GBR_IH_SET);
+
+ mdelay(20);
+ psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
+ 0x80000000, 0x8000FFFF, false);
+
+ /* Change IH ring for UMC */
+ tmp = REG_SET_FIELD(0, IH_CLIENT_CFG_DATA, CREDIT_RETURN_ADDR, 0x1216b);
+ tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, RING_ID, 1);
+
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, 4);
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, tmp);
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, GFX_CTRL_CMD_ID_GBR_IH_SET);
+
+ mdelay(20);
+ psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
+ 0x80000000, 0x8000FFFF, false);
+}
+
static int psp_v11_0_ring_init(struct psp_context *psp,
enum psp_ring_type ring_type)
{
struct psp_ring *ring;
struct amdgpu_device *adev = psp->adev;
+ psp_v11_0_reroute_ih(psp);
+
ring = &psp->km_ring;
ring->ring_type = ring_type;
for (i = 0; i < topology_info_input->num_nodes; i++) {
topology_info_input->nodes[i].node_id = topology->nodes[i].node_id;
topology_info_input->nodes[i].num_hops = topology->nodes[i].num_hops;
- topology_info_input->nodes[i].is_sharing_enabled = topology->nodes[i].is_sharing_enabled;
+ topology_info_input->nodes[i].is_sharing_enabled = 1;
topology_info_input->nodes[i].sdma_engine = topology->nodes[i].sdma_engine;
}
return 0;
}
+static int psp_v11_0_ras_trigger_error(struct psp_context *psp,
+ struct ta_ras_trigger_error_input *info)
+{
+ struct ta_ras_shared_memory *ras_cmd;
+ int ret;
+
+ if (!psp->ras.ras_initialized)
+ return -EINVAL;
+
+ ras_cmd = (struct ta_ras_shared_memory *)psp->ras.ras_shared_buf;
+ memset(ras_cmd, 0, sizeof(struct ta_ras_shared_memory));
+
+ ras_cmd->cmd_id = TA_RAS_COMMAND__TRIGGER_ERROR;
+ ras_cmd->ras_in_message.trigger_error = *info;
+
+ ret = psp_ras_invoke(psp, ras_cmd->cmd_id);
+ if (ret)
+ return -EINVAL;
+
+ return ras_cmd->ras_status;
+}
+
+static int psp_v11_0_ras_cure_posion(struct psp_context *psp, uint64_t *mode_ptr)
+{
+#if 0
+ // not support yet.
+ struct ta_ras_shared_memory *ras_cmd;
+ int ret;
+
+ if (!psp->ras.ras_initialized)
+ return -EINVAL;
+
+ ras_cmd = (struct ta_ras_shared_memory *)psp->ras.ras_shared_buf;
+ memset(ras_cmd, 0, sizeof(struct ta_ras_shared_memory));
+
+ ras_cmd->cmd_id = TA_RAS_COMMAND__CURE_POISON;
+ ras_cmd->ras_in_message.cure_poison.mode_ptr = mode_ptr;
+
+ ret = psp_ras_invoke(psp, ras_cmd->cmd_id);
+ if (ret)
+ return -EINVAL;
+
+ return ras_cmd->ras_status;
+#else
+ return -EINVAL;
+#endif
+}
+
static const struct psp_funcs psp_v11_0_funcs = {
.init_microcode = psp_v11_0_init_microcode,
.bootloader_load_sysdrv = psp_v11_0_bootloader_load_sysdrv,
.xgmi_get_hive_id = psp_v11_0_xgmi_get_hive_id,
.xgmi_get_node_id = psp_v11_0_xgmi_get_node_id,
.support_vmr_ring = psp_v11_0_support_vmr_ring,
+ .ras_trigger_error = psp_v11_0_ras_trigger_error,
+ .ras_cure_posion = psp_v11_0_ras_cure_posion,
};
void psp_v11_0_set_psp_funcs(struct psp_context *psp)
#include "sdma0/sdma0_4_0_offset.h"
#include "nbio/nbio_6_1_offset.h"
+#include "oss/osssys_4_0_offset.h"
+#include "oss/osssys_4_0_sh_mask.h"
+
MODULE_FIRMWARE("amdgpu/vega10_sos.bin");
MODULE_FIRMWARE("amdgpu/vega10_asd.bin");
MODULE_FIRMWARE("amdgpu/vega12_sos.bin");
return 0;
}
+static void psp_v3_1_reroute_ih(struct psp_context *psp)
+{
+ struct amdgpu_device *adev = psp->adev;
+ uint32_t tmp;
+
+ /* Change IH ring for VMC */
+ tmp = REG_SET_FIELD(0, IH_CLIENT_CFG_DATA, CREDIT_RETURN_ADDR, 0x1244b);
+ tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, CLIENT_TYPE, 1);
+ tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, RING_ID, 1);
+
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, 3);
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, tmp);
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, GFX_CTRL_CMD_ID_GBR_IH_SET);
+
+ mdelay(20);
+ psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
+ 0x80000000, 0x8000FFFF, false);
+
+ /* Change IH ring for UMC */
+ tmp = REG_SET_FIELD(0, IH_CLIENT_CFG_DATA, CREDIT_RETURN_ADDR, 0x1216b);
+ tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, RING_ID, 1);
+
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, 4);
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, tmp);
+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, GFX_CTRL_CMD_ID_GBR_IH_SET);
+
+ mdelay(20);
+ psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
+ 0x80000000, 0x8000FFFF, false);
+}
+
static int psp_v3_1_ring_create(struct psp_context *psp,
enum psp_ring_type ring_type)
{
struct psp_ring *ring = &psp->km_ring;
struct amdgpu_device *adev = psp->adev;
+ psp_v3_1_reroute_ih(psp);
+
/* Write low address of the ring to C2PMSG_69 */
psp_ring_reg = lower_32_bits(ring->ring_mem_mc_addr);
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, psp_ring_reg);
r = amdgpu_ring_init(adev, ring, 1024,
&adev->sdma.trap_irq,
(i == 0) ?
- AMDGPU_SDMA_IRQ_TRAP0 :
- AMDGPU_SDMA_IRQ_TRAP1);
+ AMDGPU_SDMA_IRQ_INSTANCE0 :
+ AMDGPU_SDMA_IRQ_INSTANCE1);
if (r)
return r;
}
u32 sdma_cntl;
switch (type) {
- case AMDGPU_SDMA_IRQ_TRAP0:
+ case AMDGPU_SDMA_IRQ_INSTANCE0:
switch (state) {
case AMDGPU_IRQ_STATE_DISABLE:
sdma_cntl = RREG32(mmSDMA0_CNTL + SDMA0_REGISTER_OFFSET);
break;
}
break;
- case AMDGPU_SDMA_IRQ_TRAP1:
+ case AMDGPU_SDMA_IRQ_INSTANCE1:
switch (state) {
case AMDGPU_IRQ_STATE_DISABLE:
sdma_cntl = RREG32(mmSDMA0_CNTL + SDMA1_REGISTER_OFFSET);
r = amdgpu_ring_init(adev, ring, 1024,
&adev->sdma.trap_irq,
(i == 0) ?
- AMDGPU_SDMA_IRQ_TRAP0 :
- AMDGPU_SDMA_IRQ_TRAP1);
+ AMDGPU_SDMA_IRQ_INSTANCE0 :
+ AMDGPU_SDMA_IRQ_INSTANCE1);
if (r)
return r;
}
u32 sdma_cntl;
switch (type) {
- case AMDGPU_SDMA_IRQ_TRAP0:
+ case AMDGPU_SDMA_IRQ_INSTANCE0:
switch (state) {
case AMDGPU_IRQ_STATE_DISABLE:
sdma_cntl = RREG32(mmSDMA0_CNTL + SDMA0_REGISTER_OFFSET);
break;
}
break;
- case AMDGPU_SDMA_IRQ_TRAP1:
+ case AMDGPU_SDMA_IRQ_INSTANCE1:
switch (state) {
case AMDGPU_IRQ_STATE_DISABLE:
sdma_cntl = RREG32(mmSDMA0_CNTL + SDMA1_REGISTER_OFFSET);
#include "ivsrcid/sdma0/irqsrcs_sdma0_4_0.h"
#include "ivsrcid/sdma1/irqsrcs_sdma1_4_0.h"
+#include "amdgpu_ras.h"
+
MODULE_FIRMWARE("amdgpu/vega10_sdma.bin");
MODULE_FIRMWARE("amdgpu/vega10_sdma1.bin");
MODULE_FIRMWARE("amdgpu/vega12_sdma.bin");
SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_RLC7_RB_RPTR_ADDR_LO, 0xfffffffd, 0x00000001),
SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_RLC7_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000),
SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_UTCL1_PAGE, 0x000003ff, 0x000003c0),
- SOC15_REG_GOLDEN_VALUE(SDMA0, 0, mmSDMA0_UTCL1_WATERMK, 0xFE000000, 0x00000000),
};
static const struct soc15_reg_golden golden_settings_sdma1_4_2[] = {
SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_RLC7_RB_RPTR_ADDR_LO, 0xfffffffd, 0x00000001),
SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_RLC7_RB_WPTR_POLL_CNTL, 0xfffffff7, 0x00403000),
SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_UTCL1_PAGE, 0x000003ff, 0x000003c0),
- SOC15_REG_GOLDEN_VALUE(SDMA1, 0, mmSDMA1_UTCL1_WATERMK, 0xFE000000, 0x00000000),
};
static const struct soc15_reg_golden golden_settings_sdma_rv1[] =
wptr_poll_cntl = RREG32_SDMA(i, mmSDMA0_GFX_RB_WPTR_POLL_CNTL);
wptr_poll_cntl = REG_SET_FIELD(wptr_poll_cntl,
SDMA0_GFX_RB_WPTR_POLL_CNTL,
- F32_POLL_ENABLE, amdgpu_sriov_vf(adev));
+ F32_POLL_ENABLE, amdgpu_sriov_vf(adev)? 1 : 0);
WREG32_SDMA(i, mmSDMA0_GFX_RB_WPTR_POLL_CNTL, wptr_poll_cntl);
/* enable DMA RB */
wptr_poll_cntl = RREG32_SDMA(i, mmSDMA0_PAGE_RB_WPTR_POLL_CNTL);
wptr_poll_cntl = REG_SET_FIELD(wptr_poll_cntl,
SDMA0_PAGE_RB_WPTR_POLL_CNTL,
- F32_POLL_ENABLE, amdgpu_sriov_vf(adev));
+ F32_POLL_ENABLE, amdgpu_sriov_vf(adev)? 1 : 0);
WREG32_SDMA(i, mmSDMA0_PAGE_RB_WPTR_POLL_CNTL, wptr_poll_cntl);
/* enable DMA RB */
return 0;
}
+static int sdma_v4_0_process_ras_data_cb(struct amdgpu_device *adev,
+ struct amdgpu_iv_entry *entry);
+
+static int sdma_v4_0_late_init(void *handle)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct ras_common_if **ras_if = &adev->sdma.ras_if;
+ struct ras_ih_if ih_info = {
+ .cb = sdma_v4_0_process_ras_data_cb,
+ };
+ struct ras_fs_if fs_info = {
+ .sysfs_name = "sdma_err_count",
+ .debugfs_name = "sdma_err_inject",
+ };
+ struct ras_common_if ras_block = {
+ .block = AMDGPU_RAS_BLOCK__SDMA,
+ .type = AMDGPU_RAS_ERROR__MULTI_UNCORRECTABLE,
+ .sub_block_index = 0,
+ .name = "sdma",
+ };
+ int r;
+
+ if (!amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__SDMA)) {
+ amdgpu_ras_feature_enable_on_boot(adev, &ras_block, 0);
+ return 0;
+ }
+
+ /* handle resume path. */
+ if (*ras_if)
+ goto resume;
+
+ *ras_if = kmalloc(sizeof(**ras_if), GFP_KERNEL);
+ if (!*ras_if)
+ return -ENOMEM;
+
+ **ras_if = ras_block;
+
+ r = amdgpu_ras_feature_enable_on_boot(adev, *ras_if, 1);
+ if (r)
+ goto feature;
+
+ ih_info.head = **ras_if;
+ fs_info.head = **ras_if;
+
+ r = amdgpu_ras_interrupt_add_handler(adev, &ih_info);
+ if (r)
+ goto interrupt;
+
+ r = amdgpu_ras_debugfs_create(adev, &fs_info);
+ if (r)
+ goto debugfs;
+
+ r = amdgpu_ras_sysfs_create(adev, &fs_info);
+ if (r)
+ goto sysfs;
+resume:
+ r = amdgpu_irq_get(adev, &adev->sdma.ecc_irq, AMDGPU_SDMA_IRQ_INSTANCE0);
+ if (r)
+ goto irq;
+
+ r = amdgpu_irq_get(adev, &adev->sdma.ecc_irq, AMDGPU_SDMA_IRQ_INSTANCE1);
+ if (r) {
+ amdgpu_irq_put(adev, &adev->sdma.ecc_irq, AMDGPU_SDMA_IRQ_INSTANCE0);
+ goto irq;
+ }
+
+ return 0;
+irq:
+ amdgpu_ras_sysfs_remove(adev, *ras_if);
+sysfs:
+ amdgpu_ras_debugfs_remove(adev, *ras_if);
+debugfs:
+ amdgpu_ras_interrupt_remove_handler(adev, &ih_info);
+interrupt:
+ amdgpu_ras_feature_enable(adev, *ras_if, 0);
+feature:
+ kfree(*ras_if);
+ *ras_if = NULL;
+ return -EINVAL;
+}
+
static int sdma_v4_0_sw_init(void *handle)
{
struct amdgpu_ring *ring;
if (r)
return r;
+ /* SDMA SRAM ECC event */
+ r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_SDMA0, SDMA0_4_0__SRCID__SDMA_SRAM_ECC,
+ &adev->sdma.ecc_irq);
+ if (r)
+ return r;
+
+ /* SDMA SRAM ECC event */
+ r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_SDMA1, SDMA1_4_0__SRCID__SDMA_SRAM_ECC,
+ &adev->sdma.ecc_irq);
+ if (r)
+ return r;
+
for (i = 0; i < adev->sdma.num_instances; i++) {
ring = &adev->sdma.instance[i].ring;
ring->ring_obj = NULL;
r = amdgpu_ring_init(adev, ring, 1024,
&adev->sdma.trap_irq,
(i == 0) ?
- AMDGPU_SDMA_IRQ_TRAP0 :
- AMDGPU_SDMA_IRQ_TRAP1);
+ AMDGPU_SDMA_IRQ_INSTANCE0 :
+ AMDGPU_SDMA_IRQ_INSTANCE1);
if (r)
return r;
r = amdgpu_ring_init(adev, ring, 1024,
&adev->sdma.trap_irq,
(i == 0) ?
- AMDGPU_SDMA_IRQ_TRAP0 :
- AMDGPU_SDMA_IRQ_TRAP1);
+ AMDGPU_SDMA_IRQ_INSTANCE0 :
+ AMDGPU_SDMA_IRQ_INSTANCE1);
if (r)
return r;
}
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
int i;
+ if (amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__SDMA) &&
+ adev->sdma.ras_if) {
+ struct ras_common_if *ras_if = adev->sdma.ras_if;
+ struct ras_ih_if ih_info = {
+ .head = *ras_if,
+ };
+
+ /*remove fs first*/
+ amdgpu_ras_debugfs_remove(adev, ras_if);
+ amdgpu_ras_sysfs_remove(adev, ras_if);
+ /*remove the IH*/
+ amdgpu_ras_interrupt_remove_handler(adev, &ih_info);
+ amdgpu_ras_feature_enable(adev, ras_if, 0);
+ kfree(ras_if);
+ }
+
for (i = 0; i < adev->sdma.num_instances; i++) {
amdgpu_ring_fini(&adev->sdma.instance[i].ring);
if (adev->sdma.has_page_queue)
if (amdgpu_sriov_vf(adev))
return 0;
+ amdgpu_irq_put(adev, &adev->sdma.ecc_irq, AMDGPU_SDMA_IRQ_INSTANCE0);
+ amdgpu_irq_put(adev, &adev->sdma.ecc_irq, AMDGPU_SDMA_IRQ_INSTANCE1);
+
sdma_v4_0_ctx_switch_enable(adev, false);
sdma_v4_0_enable(adev, false);
unsigned type,
enum amdgpu_interrupt_state state)
{
- unsigned int instance = (type == AMDGPU_SDMA_IRQ_TRAP0) ? 0 : 1;
u32 sdma_cntl;
- sdma_cntl = RREG32_SDMA(instance, mmSDMA0_CNTL);
+ sdma_cntl = RREG32_SDMA(type, mmSDMA0_CNTL);
sdma_cntl = REG_SET_FIELD(sdma_cntl, SDMA0_CNTL, TRAP_ENABLE,
state == AMDGPU_IRQ_STATE_ENABLE ? 1 : 0);
- WREG32_SDMA(instance, mmSDMA0_CNTL, sdma_cntl);
+ WREG32_SDMA(type, mmSDMA0_CNTL, sdma_cntl);
return 0;
}
return 0;
}
+static int sdma_v4_0_process_ras_data_cb(struct amdgpu_device *adev,
+ struct amdgpu_iv_entry *entry)
+{
+ uint32_t instance, err_source;
+
+ switch (entry->client_id) {
+ case SOC15_IH_CLIENTID_SDMA0:
+ instance = 0;
+ break;
+ case SOC15_IH_CLIENTID_SDMA1:
+ instance = 1;
+ break;
+ default:
+ return 0;
+ }
+
+ switch (entry->src_id) {
+ case SDMA0_4_0__SRCID__SDMA_SRAM_ECC:
+ err_source = 0;
+ break;
+ case SDMA0_4_0__SRCID__SDMA_ECC:
+ err_source = 1;
+ break;
+ default:
+ return 0;
+ }
+
+ kgd2kfd_set_sram_ecc_flag(adev->kfd.dev);
+
+ amdgpu_ras_reset_gpu(adev, 0);
+
+ return AMDGPU_RAS_UE;
+}
+
+static int sdma_v4_0_process_ecc_irq(struct amdgpu_device *adev,
+ struct amdgpu_irq_src *source,
+ struct amdgpu_iv_entry *entry)
+{
+ struct ras_common_if *ras_if = adev->sdma.ras_if;
+ struct ras_dispatch_if ih_data = {
+ .entry = entry,
+ };
+
+ if (!ras_if)
+ return 0;
+
+ ih_data.head = *ras_if;
+
+ amdgpu_ras_interrupt_dispatch(adev, &ih_data);
+ return 0;
+}
+
static int sdma_v4_0_process_illegal_inst_irq(struct amdgpu_device *adev,
struct amdgpu_irq_src *source,
struct amdgpu_iv_entry *entry)
return 0;
}
+static int sdma_v4_0_set_ecc_irq_state(struct amdgpu_device *adev,
+ struct amdgpu_irq_src *source,
+ unsigned type,
+ enum amdgpu_interrupt_state state)
+{
+ u32 sdma_edc_config;
+
+ u32 reg_offset = (type == AMDGPU_SDMA_IRQ_INSTANCE0) ?
+ sdma_v4_0_get_reg_offset(adev, 0, mmSDMA0_EDC_CONFIG) :
+ sdma_v4_0_get_reg_offset(adev, 1, mmSDMA0_EDC_CONFIG);
+
+ sdma_edc_config = RREG32(reg_offset);
+ sdma_edc_config = REG_SET_FIELD(sdma_edc_config, SDMA0_EDC_CONFIG, ECC_INT_ENABLE,
+ state == AMDGPU_IRQ_STATE_ENABLE ? 1 : 0);
+ WREG32(reg_offset, sdma_edc_config);
+
+ return 0;
+}
+
static void sdma_v4_0_update_medium_grain_clock_gating(
struct amdgpu_device *adev,
bool enable)
const struct amd_ip_funcs sdma_v4_0_ip_funcs = {
.name = "sdma_v4_0",
.early_init = sdma_v4_0_early_init,
- .late_init = NULL,
+ .late_init = sdma_v4_0_late_init,
.sw_init = sdma_v4_0_sw_init,
.sw_fini = sdma_v4_0_sw_fini,
.hw_init = sdma_v4_0_hw_init,
.process = sdma_v4_0_process_illegal_inst_irq,
};
+static const struct amdgpu_irq_src_funcs sdma_v4_0_ecc_irq_funcs = {
+ .set = sdma_v4_0_set_ecc_irq_state,
+ .process = sdma_v4_0_process_ecc_irq,
+};
+
+
+
static void sdma_v4_0_set_irq_funcs(struct amdgpu_device *adev)
{
adev->sdma.trap_irq.num_types = AMDGPU_SDMA_IRQ_LAST;
adev->sdma.trap_irq.funcs = &sdma_v4_0_trap_irq_funcs;
adev->sdma.illegal_inst_irq.funcs = &sdma_v4_0_illegal_inst_irq_funcs;
+ adev->sdma.ecc_irq.num_types = AMDGPU_SDMA_IRQ_LAST;
+ adev->sdma.ecc_irq.funcs = &sdma_v4_0_ecc_irq_funcs;
}
/**
static void sdma_v4_0_set_buffer_funcs(struct amdgpu_device *adev)
{
adev->mman.buffer_funcs = &sdma_v4_0_buffer_funcs;
- if (adev->sdma.has_page_queue)
- adev->mman.buffer_funcs_ring = &adev->sdma.instance[0].page;
+ if (adev->sdma.has_page_queue && adev->sdma.num_instances > 1)
+ adev->mman.buffer_funcs_ring = &adev->sdma.instance[1].page;
else
adev->mman.buffer_funcs_ring = &adev->sdma.instance[0].ring;
}
unsigned i;
adev->vm_manager.vm_pte_funcs = &sdma_v4_0_vm_pte_funcs;
- for (i = 0; i < adev->sdma.num_instances; i++) {
- if (adev->sdma.has_page_queue)
+ if (adev->sdma.has_page_queue && adev->sdma.num_instances > 1) {
+ for (i = 1; i < adev->sdma.num_instances; i++) {
sched = &adev->sdma.instance[i].page.sched;
- else
+ adev->vm_manager.vm_pte_rqs[i - 1] =
+ &sched->sched_rq[DRM_SCHED_PRIORITY_KERNEL];
+ }
+ adev->vm_manager.vm_pte_num_rqs = adev->sdma.num_instances - 1;
+ adev->vm_manager.page_fault = &adev->sdma.instance[0].page;
+ } else {
+ for (i = 0; i < adev->sdma.num_instances; i++) {
sched = &adev->sdma.instance[i].ring.sched;
- adev->vm_manager.vm_pte_rqs[i] =
- &sched->sched_rq[DRM_SCHED_PRIORITY_KERNEL];
+ adev->vm_manager.vm_pte_rqs[i] =
+ &sched->sched_rq[DRM_SCHED_PRIORITY_KERNEL];
+ }
+ adev->vm_manager.vm_pte_num_rqs = adev->sdma.num_instances;
}
- adev->vm_manager.vm_pte_num_rqs = adev->sdma.num_instances;
}
const struct amdgpu_ip_block_version sdma_v4_0_ip_block = {
r = amdgpu_ring_init(adev, ring, 1024,
&adev->sdma.trap_irq,
(i == 0) ?
- AMDGPU_SDMA_IRQ_TRAP0 :
- AMDGPU_SDMA_IRQ_TRAP1);
+ AMDGPU_SDMA_IRQ_INSTANCE0 :
+ AMDGPU_SDMA_IRQ_INSTANCE1);
if (r)
return r;
}
u32 sdma_cntl;
switch (type) {
- case AMDGPU_SDMA_IRQ_TRAP0:
+ case AMDGPU_SDMA_IRQ_INSTANCE0:
switch (state) {
case AMDGPU_IRQ_STATE_DISABLE:
sdma_cntl = RREG32(DMA_CNTL + DMA0_REGISTER_OFFSET);
break;
}
break;
- case AMDGPU_SDMA_IRQ_TRAP1:
+ case AMDGPU_SDMA_IRQ_INSTANCE1:
switch (state) {
case AMDGPU_IRQ_STATE_DISABLE:
sdma_cntl = RREG32(DMA_CNTL + DMA1_REGISTER_OFFSET);
static void si_program_response_times(struct amdgpu_device *adev)
{
- u32 voltage_response_time, backbias_response_time, acpi_delay_time, vbi_time_out;
+ u32 voltage_response_time, acpi_delay_time, vbi_time_out;
u32 vddc_dly, acpi_dly, vbi_dly;
u32 reference_clock;
si_write_smc_soft_register(adev, SI_SMC_SOFT_REGISTER_mvdd_chg_time, 1);
voltage_response_time = (u32)adev->pm.dpm.voltage_response_time;
- backbias_response_time = (u32)adev->pm.dpm.backbias_response_time;
if (voltage_response_time == 0)
voltage_response_time = 1000;
#include "vcn_v1_0.h"
#include "dce_virtual.h"
#include "mxgpu_ai.h"
+#include "amdgpu_smu.h"
#define mmMP0_MISC_CGTT_CTRL0 0x01b9
#define mmMP0_MISC_CGTT_CTRL0_BASE_IDX 0
static int soc15_asic_mode1_reset(struct amdgpu_device *adev)
{
u32 i;
+ int ret = 0;
amdgpu_atombios_scratch_regs_engine_hung(adev, true);
pci_save_state(adev->pdev);
- psp_gpu_reset(adev);
+ ret = psp_gpu_reset(adev);
+ if (ret)
+ dev_err(adev->dev, "GPU mode1 reset failed\n");
pci_restore_state(adev->pdev);
amdgpu_atombios_scratch_regs_engine_hung(adev, false);
- return 0;
+ return ret;
}
static int soc15_asic_get_baco_capability(struct amdgpu_device *adev, bool *cap)
dev_info(adev->dev, "GPU BACO reset\n");
+ adev->in_baco_reset = 1;
+
return 0;
}
switch (adev->asic_type) {
case CHIP_VEGA10:
+ case CHIP_VEGA12:
soc15_asic_get_baco_capability(adev, &baco_reset);
break;
+ case CHIP_VEGA20:
+ if (adev->psp.sos_fw_version >= 0x80067)
+ soc15_asic_get_baco_capability(adev, &baco_reset);
+ else
+ baco_reset = false;
+ break;
default:
baco_reset = false;
break;
}
amdgpu_device_ip_block_add(adev, &gfx_v9_0_ip_block);
amdgpu_device_ip_block_add(adev, &sdma_v4_0_ip_block);
- if (!amdgpu_sriov_vf(adev))
- amdgpu_device_ip_block_add(adev, &pp_smu_ip_block);
+ if (!amdgpu_sriov_vf(adev)) {
+ if (is_support_sw_smu(adev))
+ amdgpu_device_ip_block_add(adev, &smu_v11_0_ip_block);
+ else
+ amdgpu_device_ip_block_add(adev, &pp_smu_ip_block);
+ }
if (adev->enable_virtual_display || amdgpu_sriov_vf(adev))
amdgpu_device_ip_block_add(adev, &dce_virtual_ip_block);
#if defined(CONFIG_DRM_AMD_DC)
adev->pg_flags = AMD_PG_SUPPORT_SDMA | AMD_PG_SUPPORT_VCN;
} else if (adev->pdev->device == 0x15d8) {
- adev->cg_flags = AMD_CG_SUPPORT_GFX_MGLS |
+ adev->cg_flags = AMD_CG_SUPPORT_GFX_MGCG |
+ AMD_CG_SUPPORT_GFX_MGLS |
AMD_CG_SUPPORT_GFX_CP_LS |
AMD_CG_SUPPORT_GFX_3D_CGCG |
AMD_CG_SUPPORT_GFX_3D_CGLS |
adev->pg_flags = AMD_PG_SUPPORT_SDMA | AMD_PG_SUPPORT_VCN;
}
- if (adev->powerplay.pp_feature & PP_GFXOFF_MASK)
+ if (adev->pm.pp_feature & PP_GFXOFF_MASK)
adev->pg_flags |= AMD_PG_SUPPORT_GFX_PG |
AMD_PG_SUPPORT_CP |
AMD_PG_SUPPORT_RLC_SMU_HS;
--- /dev/null
+/****************************************************************************\
+*
+* File Name ta_ras_if.h
+* Project AMD PSP SW IP Module
+*
+* Description Interface to the RAS Trusted Application
+*
+* Copyright 2019 Advanced Micro Devices, Inc.
+*
+* Permission is hereby granted, free of charge, to any person obtaining a copy of this software
+* and associated documentation files (the "Software"), to deal in the Software without restriction,
+* including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
+* and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so,
+* subject to the following conditions:
+*
+* The above copyright notice and this permission notice shall be included in all copies or substantial
+* portions of the Software.
+*
+* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+* OTHER DEALINGS IN THE SOFTWARE.
+*/
+#ifndef _TA_RAS_IF_H
+#define _TA_RAS_IF_H
+
+/* Responses have bit 31 set */
+#define RSP_ID_MASK (1U << 31)
+#define RSP_ID(cmdId) (((uint32_t)(cmdId)) | RSP_ID_MASK)
+
+#define TA_NUM_BLOCK_MAX 14
+
+enum ras_command {
+ TA_RAS_COMMAND__ENABLE_FEATURES = 0,
+ TA_RAS_COMMAND__DISABLE_FEATURES,
+ TA_RAS_COMMAND__TRIGGER_ERROR,
+};
+
+enum ta_ras_status {
+ TA_RAS_STATUS__SUCCESS = 0x00,
+ TA_RAS_STATUS__RESET_NEEDED = 0x01,
+ TA_RAS_STATUS__ERROR_INVALID_PARAMETER = 0x02,
+ TA_RAS_STATUS__ERROR_RAS_NOT_AVAILABLE = 0x03,
+ TA_RAS_STATUS__ERROR_RAS_DUPLICATE_CMD = 0x04,
+ TA_RAS_STATUS__ERROR_INJECTION_FAILED = 0x05
+};
+
+enum ta_ras_block {
+ TA_RAS_BLOCK__UMC = 0,
+ TA_RAS_BLOCK__SDMA,
+ TA_RAS_BLOCK__GFX,
+ TA_RAS_BLOCK__MMHUB,
+ TA_RAS_BLOCK__ATHUB,
+ TA_RAS_BLOCK__PCIE_BIF,
+ TA_RAS_BLOCK__HDP,
+ TA_RAS_BLOCK__XGMI_WAFL,
+ TA_RAS_BLOCK__DF,
+ TA_RAS_BLOCK__SMN,
+ TA_RAS_BLOCK__SEM,
+ TA_RAS_BLOCK__MP0,
+ TA_RAS_BLOCK__MP1,
+ TA_RAS_BLOCK__FUSE = (TA_NUM_BLOCK_MAX - 1),
+};
+
+enum ta_ras_error_type {
+ TA_RAS_ERROR__NONE = 0,
+ TA_RAS_ERROR__PARITY = 1,
+ TA_RAS_ERROR__SINGLE_CORRECTABLE = 2,
+ TA_RAS_ERROR__MULTI_UNCORRECTABLE = 4,
+ TA_RAS_ERROR__POISON = 8
+};
+
+struct ta_ras_enable_features_input {
+ enum ta_ras_block block_id;
+ enum ta_ras_error_type error_type;
+};
+
+struct ta_ras_disable_features_input {
+ enum ta_ras_block block_id;
+ enum ta_ras_error_type error_type;
+};
+
+struct ta_ras_trigger_error_input {
+ enum ta_ras_block block_id;
+ enum ta_ras_error_type inject_error_type;
+ uint32_t sub_block_index;
+ uint64_t address;
+ uint64_t value;
+};
+
+union ta_ras_cmd_input {
+ struct ta_ras_enable_features_input enable_features;
+ struct ta_ras_disable_features_input disable_features;
+ struct ta_ras_trigger_error_input trigger_error;
+};
+
+struct ta_ras_shared_memory {
+ uint32_t cmd_id;
+ uint32_t resp_id;
+ enum ta_ras_status ras_status;
+ uint32_t reserved;
+ union ta_ras_cmd_input ras_in_message;
+};
+
+#endif // TL_RAS_IF_H_
}
if (vce_v2_0_wait_for_idle(adev)) {
- DRM_INFO("VCE is busy, Can't set clock gateing");
+ DRM_INFO("VCE is busy, Can't set clock gating");
return 0;
}
static int vce_v4_0_stop(struct amdgpu_device *adev)
{
+ /* Disable VCPU */
WREG32_P(SOC15_REG_OFFSET(VCE, 0, mmVCE_VCPU_CNTL), 0, ~0x200001);
/* hold on ECPU */
VCE_SOFT_RESET__ECPU_SOFT_RESET_MASK,
~VCE_SOFT_RESET__ECPU_SOFT_RESET_MASK);
- /* clear BUSY flag */
- WREG32_P(SOC15_REG_OFFSET(VCE, 0, mmVCE_STATUS), 0, ~VCE_STATUS__JOB_BUSY_MASK);
+ /* clear VCE_STATUS */
+ WREG32(SOC15_REG_OFFSET(VCE, 0, mmVCE_STATUS), 0);
/* Set Clock-Gating off */
/* if (adev->cg_flags & AMD_CG_SUPPORT_VCE_MGCG)
return 0;
}
+#endif
static int vce_v4_0_set_powergating_state(void *handle,
enum amd_powergating_state state)
*/
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
- if (!(adev->pg_flags & AMD_PG_SUPPORT_VCE))
- return 0;
-
if (state == AMD_PG_STATE_GATE)
- /* XXX do we need a vce_v4_0_stop()? */
- return 0;
+ return vce_v4_0_stop(adev);
else
return vce_v4_0_start(adev);
}
-#endif
static void vce_v4_0_ring_emit_ib(struct amdgpu_ring *ring, struct amdgpu_job *job,
struct amdgpu_ib *ib, uint32_t flags)
.soft_reset = NULL /* vce_v4_0_soft_reset */,
.post_soft_reset = NULL /* vce_v4_0_post_soft_reset */,
.set_clockgating_state = vce_v4_0_set_clockgating_state,
- .set_powergating_state = NULL /* vce_v4_0_set_powergating_state */,
+ .set_powergating_state = vce_v4_0_set_powergating_state,
};
static const struct amdgpu_ring_funcs vce_v4_0_ring_vm_funcs = {
return ih_rb_cntl;
}
+static uint32_t vega10_ih_doorbell_rptr(struct amdgpu_ih_ring *ih)
+{
+ u32 ih_doorbell_rtpr = 0;
+
+ if (ih->use_doorbell) {
+ ih_doorbell_rtpr = REG_SET_FIELD(ih_doorbell_rtpr,
+ IH_DOORBELL_RPTR, OFFSET,
+ ih->doorbell_index);
+ ih_doorbell_rtpr = REG_SET_FIELD(ih_doorbell_rtpr,
+ IH_DOORBELL_RPTR,
+ ENABLE, 1);
+ } else {
+ ih_doorbell_rtpr = REG_SET_FIELD(ih_doorbell_rtpr,
+ IH_DOORBELL_RPTR,
+ ENABLE, 0);
+ }
+ return ih_doorbell_rtpr;
+}
+
/**
* vega10_ih_irq_init - init and enable the interrupt ring
*
static int vega10_ih_irq_init(struct amdgpu_device *adev)
{
struct amdgpu_ih_ring *ih;
+ u32 ih_rb_cntl;
int ret = 0;
- u32 ih_rb_cntl, ih_doorbell_rtpr;
u32 tmp;
/* disable irqs */
upper_32_bits(ih->wptr_addr) & 0xFFFF);
/* set rptr, wptr to 0 */
- WREG32_SOC15(OSSSYS, 0, mmIH_RB_RPTR, 0);
WREG32_SOC15(OSSSYS, 0, mmIH_RB_WPTR, 0);
+ WREG32_SOC15(OSSSYS, 0, mmIH_RB_RPTR, 0);
- ih_doorbell_rtpr = RREG32_SOC15(OSSSYS, 0, mmIH_DOORBELL_RPTR);
- if (adev->irq.ih.use_doorbell) {
- ih_doorbell_rtpr = REG_SET_FIELD(ih_doorbell_rtpr,
- IH_DOORBELL_RPTR, OFFSET,
- adev->irq.ih.doorbell_index);
- ih_doorbell_rtpr = REG_SET_FIELD(ih_doorbell_rtpr,
- IH_DOORBELL_RPTR,
- ENABLE, 1);
- } else {
- ih_doorbell_rtpr = REG_SET_FIELD(ih_doorbell_rtpr,
- IH_DOORBELL_RPTR,
- ENABLE, 0);
- }
- WREG32_SOC15(OSSSYS, 0, mmIH_DOORBELL_RPTR, ih_doorbell_rtpr);
+ WREG32_SOC15(OSSSYS, 0, mmIH_DOORBELL_RPTR,
+ vega10_ih_doorbell_rptr(ih));
ih = &adev->irq.ih1;
if (ih->ring_size) {
ih_rb_cntl = RREG32_SOC15(OSSSYS, 0, mmIH_RB_CNTL_RING1);
ih_rb_cntl = vega10_ih_rb_cntl(ih, ih_rb_cntl);
+ ih_rb_cntl = REG_SET_FIELD(ih_rb_cntl, IH_RB_CNTL,
+ WPTR_OVERFLOW_ENABLE, 0);
+ ih_rb_cntl = REG_SET_FIELD(ih_rb_cntl, IH_RB_CNTL,
+ RB_FULL_DRAIN_ENABLE, 1);
WREG32_SOC15(OSSSYS, 0, mmIH_RB_CNTL_RING1, ih_rb_cntl);
/* set rptr, wptr to 0 */
- WREG32_SOC15(OSSSYS, 0, mmIH_RB_RPTR_RING1, 0);
WREG32_SOC15(OSSSYS, 0, mmIH_RB_WPTR_RING1, 0);
+ WREG32_SOC15(OSSSYS, 0, mmIH_RB_RPTR_RING1, 0);
+
+ WREG32_SOC15(OSSSYS, 0, mmIH_DOORBELL_RPTR_RING1,
+ vega10_ih_doorbell_rptr(ih));
}
ih = &adev->irq.ih2;
WREG32_SOC15(OSSSYS, 0, mmIH_RB_BASE_HI_RING2,
(ih->gpu_addr >> 40) & 0xff);
- ih_rb_cntl = RREG32_SOC15(OSSSYS, 0, mmIH_RB_CNTL_RING1);
+ ih_rb_cntl = RREG32_SOC15(OSSSYS, 0, mmIH_RB_CNTL_RING2);
ih_rb_cntl = vega10_ih_rb_cntl(ih, ih_rb_cntl);
WREG32_SOC15(OSSSYS, 0, mmIH_RB_CNTL_RING2, ih_rb_cntl);
/* set rptr, wptr to 0 */
- WREG32_SOC15(OSSSYS, 0, mmIH_RB_RPTR_RING2, 0);
WREG32_SOC15(OSSSYS, 0, mmIH_RB_WPTR_RING2, 0);
+ WREG32_SOC15(OSSSYS, 0, mmIH_RB_RPTR_RING2, 0);
+
+ WREG32_SOC15(OSSSYS, 0, mmIH_DOORBELL_RPTR_RING2,
+ vega10_ih_doorbell_rptr(ih));
}
tmp = RREG32_SOC15(OSSSYS, 0, mmIH_STORM_CLIENT_LIST_CNTL);
if (r)
return r;
- if (adev->asic_type == CHIP_VEGA10) {
- r = amdgpu_ih_ring_init(adev, &adev->irq.ih1, PAGE_SIZE, true);
- if (r)
- return r;
-
- r = amdgpu_ih_ring_init(adev, &adev->irq.ih2, PAGE_SIZE, true);
- if (r)
- return r;
- }
-
- /* TODO add doorbell for IH1 & IH2 as well */
adev->irq.ih.use_doorbell = true;
adev->irq.ih.doorbell_index = adev->doorbell_index.ih << 1;
+ r = amdgpu_ih_ring_init(adev, &adev->irq.ih1, PAGE_SIZE, true);
+ if (r)
+ return r;
+
+ adev->irq.ih1.use_doorbell = true;
+ adev->irq.ih1.doorbell_index = (adev->doorbell_index.ih + 1) << 1;
+
+ r = amdgpu_ih_ring_init(adev, &adev->irq.ih2, PAGE_SIZE, true);
+ if (r)
+ return r;
+
+ adev->irq.ih2.use_doorbell = true;
+ adev->irq.ih2.doorbell_index = (adev->doorbell_index.ih + 2) << 1;
+
r = amdgpu_irq_init(adev);
return r;
memset(&kfd->doorbell_available_index, 0,
sizeof(kfd->doorbell_available_index));
+ atomic_set(&kfd->sram_ecc_flag, 0);
+
return kfd;
}
{
unsigned int size;
- kfd->mec_fw_version = kfd->kfd2kgd->get_fw_version(kfd->kgd,
+ kfd->mec_fw_version = amdgpu_amdkfd_get_fw_version(kfd->kgd,
KGD_ENGINE_MEC1);
- kfd->sdma_fw_version = kfd->kfd2kgd->get_fw_version(kfd->kgd,
+ kfd->sdma_fw_version = amdgpu_amdkfd_get_fw_version(kfd->kgd,
KGD_ENGINE_SDMA1);
kfd->shared_resources = *gpu_resources;
return ret;
count = atomic_dec_return(&kfd_locked);
WARN_ONCE(count != 0, "KFD reset ref. error");
+
+ atomic_set(&kfd->sram_ecc_flag, 0);
+
return 0;
}
return 0;
}
+void kgd2kfd_set_sram_ecc_flag(struct kfd_dev *kfd)
+{
+ if (kfd)
+ atomic_inc(&kfd->sram_ecc_flag);
+}
+
#if defined(CONFIG_DEBUG_FS)
/* This function will send a package to HIQ to hang the HWS
void kfd_signal_reset_event(struct kfd_dev *dev)
{
struct kfd_hsa_hw_exception_data hw_exception_data;
+ struct kfd_hsa_memory_exception_data memory_exception_data;
struct kfd_process *p;
struct kfd_event *ev;
unsigned int temp;
uint32_t id, idx;
+ int reset_cause = atomic_read(&dev->sram_ecc_flag) ?
+ KFD_HW_EXCEPTION_ECC :
+ KFD_HW_EXCEPTION_GPU_HANG;
/* Whole gpu reset caused by GPU hang and memory is lost */
memset(&hw_exception_data, 0, sizeof(hw_exception_data));
hw_exception_data.gpu_id = dev->id;
hw_exception_data.memory_lost = 1;
+ hw_exception_data.reset_cause = reset_cause;
+
+ memset(&memory_exception_data, 0, sizeof(memory_exception_data));
+ memory_exception_data.ErrorType = KFD_MEM_ERR_SRAM_ECC;
+ memory_exception_data.gpu_id = dev->id;
+ memory_exception_data.failure.imprecise = true;
idx = srcu_read_lock(&kfd_processes_srcu);
hash_for_each_rcu(kfd_processes_table, temp, p, kfd_processes) {
mutex_lock(&p->event_mutex);
id = KFD_FIRST_NONSIGNAL_EVENT_ID;
- idr_for_each_entry_continue(&p->event_idr, ev, id)
+ idr_for_each_entry_continue(&p->event_idr, ev, id) {
if (ev->type == KFD_EVENT_TYPE_HW_EXCEPTION) {
ev->hw_exception_data = hw_exception_data;
set_event(ev);
}
+ if (ev->type == KFD_EVENT_TYPE_MEMORY &&
+ reset_cause == KFD_HW_EXCEPTION_ECC) {
+ ev->memory_exception_data = memory_exception_data;
+ set_event(ev);
+ }
+ }
mutex_unlock(&p->event_mutex);
}
srcu_read_unlock(&kfd_processes_srcu, idx);
uint64_t hive_id;
bool pci_atomic_requested;
+
+ /* SRAM ECC flag */
+ atomic_t sram_ecc_flag;
};
enum kfd_mempool {
#include "kfd_device_queue_manager.h"
#include "kfd_iommu.h"
#include "amdgpu_amdkfd.h"
+#include "amdgpu_ras.h"
/* topology_device_list - Master list of all topology devices */
static struct list_head topology_device_list;
void *crat_image = NULL;
size_t image_size = 0;
int proximity_domain;
+ struct amdgpu_ras *ctx;
INIT_LIST_HEAD(&temp_topology_device_list);
dev->node_props.capability |= HSA_CAP_ATS_PRESENT;
}
+ ctx = amdgpu_ras_get_context((struct amdgpu_device *)(dev->gpu->kgd));
+ if (ctx) {
+ /* kfd only concerns sram ecc on GFX/SDMA and HBM ecc on UMC */
+ dev->node_props.capability |=
+ (((ctx->features & BIT(AMDGPU_RAS_BLOCK__SDMA)) != 0) ||
+ ((ctx->features & BIT(AMDGPU_RAS_BLOCK__GFX)) != 0)) ?
+ HSA_CAP_SRAM_EDCSUPPORTED : 0;
+ dev->node_props.capability |= ((ctx->features & BIT(AMDGPU_RAS_BLOCK__UMC)) != 0) ?
+ HSA_CAP_MEM_EDCSUPPORTED : 0;
+
+ dev->node_props.capability |= (ctx->features != 0) ?
+ HSA_CAP_RASEVENTNOTIFY : 0;
+ }
+
kfd_debug_print_topology();
if (!res)
#define HSA_CAP_DOORBELL_TYPE_2_0 0x2
#define HSA_CAP_AQL_QUEUE_DOUBLE_MAP 0x00004000
+#define HSA_CAP_SRAM_EDCSUPPORTED 0x00080000
+#define HSA_CAP_MEM_EDCSUPPORTED 0x00100000
+#define HSA_CAP_RASEVENTNOTIFY 0x00200000
+
struct kfd_node_properties {
uint64_t hive_id;
uint32_t cpu_cores_count;
static int amdgpu_dm_plane_init(struct amdgpu_display_manager *dm,
struct drm_plane *plane,
- unsigned long possible_crtcs);
+ unsigned long possible_crtcs,
+ const struct dc_plane_cap *plane_cap);
static int amdgpu_dm_crtc_init(struct amdgpu_display_manager *dm,
struct drm_plane *plane,
uint32_t link_index);
static void handle_cursor_update(struct drm_plane *plane,
struct drm_plane_state *old_plane_state);
-
-
-static const enum drm_plane_type dm_plane_type_default[AMDGPU_MAX_PLANES] = {
- DRM_PLANE_TYPE_PRIMARY,
- DRM_PLANE_TYPE_PRIMARY,
- DRM_PLANE_TYPE_PRIMARY,
- DRM_PLANE_TYPE_PRIMARY,
- DRM_PLANE_TYPE_PRIMARY,
- DRM_PLANE_TYPE_PRIMARY,
-};
-
-static const enum drm_plane_type dm_plane_type_carizzo[AMDGPU_MAX_PLANES] = {
- DRM_PLANE_TYPE_PRIMARY,
- DRM_PLANE_TYPE_PRIMARY,
- DRM_PLANE_TYPE_PRIMARY,
- DRM_PLANE_TYPE_OVERLAY,/* YUV Capable Underlay */
-};
-
-static const enum drm_plane_type dm_plane_type_stoney[AMDGPU_MAX_PLANES] = {
- DRM_PLANE_TYPE_PRIMARY,
- DRM_PLANE_TYPE_PRIMARY,
- DRM_PLANE_TYPE_OVERLAY, /* YUV Capable Underlay */
-};
-
/*
* dm_vblank_get_counter
*
return NULL;
}
+static inline bool amdgpu_dm_vrr_active(struct dm_crtc_state *dm_state)
+{
+ return dm_state->freesync_config.state == VRR_STATE_ACTIVE_VARIABLE ||
+ dm_state->freesync_config.state == VRR_STATE_ACTIVE_FIXED;
+}
+
static void dm_pflip_high_irq(void *interrupt_params)
{
struct amdgpu_crtc *amdgpu_crtc;
struct common_irq_params *irq_params = interrupt_params;
struct amdgpu_device *adev = irq_params->adev;
unsigned long flags;
+ struct drm_pending_vblank_event *e;
+ struct dm_crtc_state *acrtc_state;
+ uint32_t vpos, hpos, v_blank_start, v_blank_end;
+ bool vrr_active;
amdgpu_crtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_PFLIP);
return;
}
- /* Update to correct count(s) if racing with vblank irq */
- amdgpu_crtc->last_flip_vblank = drm_crtc_accurate_vblank_count(&amdgpu_crtc->base);
+ /* page flip completed. */
+ e = amdgpu_crtc->event;
+ amdgpu_crtc->event = NULL;
- /* wake up userspace */
- if (amdgpu_crtc->event) {
- drm_crtc_send_vblank_event(&amdgpu_crtc->base, amdgpu_crtc->event);
+ if (!e)
+ WARN_ON(1);
- /* page flip completed. clean up */
- amdgpu_crtc->event = NULL;
+ acrtc_state = to_dm_crtc_state(amdgpu_crtc->base.state);
+ vrr_active = amdgpu_dm_vrr_active(acrtc_state);
+
+ /* Fixed refresh rate, or VRR scanout position outside front-porch? */
+ if (!vrr_active ||
+ !dc_stream_get_scanoutpos(acrtc_state->stream, &v_blank_start,
+ &v_blank_end, &hpos, &vpos) ||
+ (vpos < v_blank_start)) {
+ /* Update to correct count and vblank timestamp if racing with
+ * vblank irq. This also updates to the correct vblank timestamp
+ * even in VRR mode, as scanout is past the front-porch atm.
+ */
+ drm_crtc_accurate_vblank_count(&amdgpu_crtc->base);
- } else
- WARN_ON(1);
+ /* Wake up userspace by sending the pageflip event with proper
+ * count and timestamp of vblank of flip completion.
+ */
+ if (e) {
+ drm_crtc_send_vblank_event(&amdgpu_crtc->base, e);
+
+ /* Event sent, so done with vblank for this flip */
+ drm_crtc_vblank_put(&amdgpu_crtc->base);
+ }
+ } else if (e) {
+ /* VRR active and inside front-porch: vblank count and
+ * timestamp for pageflip event will only be up to date after
+ * drm_crtc_handle_vblank() has been executed from late vblank
+ * irq handler after start of back-porch (vline 0). We queue the
+ * pageflip event for send-out by drm_crtc_handle_vblank() with
+ * updated timestamp and count, once it runs after us.
+ *
+ * We need to open-code this instead of using the helper
+ * drm_crtc_arm_vblank_event(), as that helper would
+ * call drm_crtc_accurate_vblank_count(), which we must
+ * not call in VRR mode while we are in front-porch!
+ */
+
+ /* sequence will be replaced by real count during send-out. */
+ e->sequence = drm_crtc_vblank_count(&amdgpu_crtc->base);
+ e->pipe = amdgpu_crtc->crtc_id;
+
+ list_add_tail(&e->base.link, &adev->ddev->vblank_event_list);
+ e = NULL;
+ }
+
+ /* Keep track of vblank of this flip for flip throttling. We use the
+ * cooked hw counter, as that one incremented at start of this vblank
+ * of pageflip completion, so last_flip_vblank is the forbidden count
+ * for queueing new pageflips if vsync + VRR is enabled.
+ */
+ amdgpu_crtc->last_flip_vblank = amdgpu_get_vblank_counter_kms(adev->ddev,
+ amdgpu_crtc->crtc_id);
amdgpu_crtc->pflip_status = AMDGPU_FLIP_NONE;
spin_unlock_irqrestore(&adev->ddev->event_lock, flags);
- DRM_DEBUG_DRIVER("%s - crtc :%d[%p], pflip_stat:AMDGPU_FLIP_NONE\n",
- __func__, amdgpu_crtc->crtc_id, amdgpu_crtc);
+ DRM_DEBUG_DRIVER("crtc:%d[%p], pflip_stat:AMDGPU_FLIP_NONE, vrr[%d]-fp %d\n",
+ amdgpu_crtc->crtc_id, amdgpu_crtc,
+ vrr_active, (int) !e);
+}
+
+static void dm_vupdate_high_irq(void *interrupt_params)
+{
+ struct common_irq_params *irq_params = interrupt_params;
+ struct amdgpu_device *adev = irq_params->adev;
+ struct amdgpu_crtc *acrtc;
+ struct dm_crtc_state *acrtc_state;
+ unsigned long flags;
+
+ acrtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_VUPDATE);
+
+ if (acrtc) {
+ acrtc_state = to_dm_crtc_state(acrtc->base.state);
+
+ DRM_DEBUG_DRIVER("crtc:%d, vupdate-vrr:%d\n", acrtc->crtc_id,
+ amdgpu_dm_vrr_active(acrtc_state));
- drm_crtc_vblank_put(&amdgpu_crtc->base);
+ /* Core vblank handling is done here after end of front-porch in
+ * vrr mode, as vblank timestamping will give valid results
+ * while now done after front-porch. This will also deliver
+ * page-flip completion events that have been queued to us
+ * if a pageflip happened inside front-porch.
+ */
+ if (amdgpu_dm_vrr_active(acrtc_state)) {
+ drm_crtc_handle_vblank(&acrtc->base);
+
+ /* BTR processing for pre-DCE12 ASICs */
+ if (acrtc_state->stream &&
+ adev->family < AMDGPU_FAMILY_AI) {
+ spin_lock_irqsave(&adev->ddev->event_lock, flags);
+ mod_freesync_handle_v_update(
+ adev->dm.freesync_module,
+ acrtc_state->stream,
+ &acrtc_state->vrr_params);
+
+ dc_stream_adjust_vmin_vmax(
+ adev->dm.dc,
+ acrtc_state->stream,
+ &acrtc_state->vrr_params.adjust);
+ spin_unlock_irqrestore(&adev->ddev->event_lock, flags);
+ }
+ }
+ }
}
static void dm_crtc_high_irq(void *interrupt_params)
struct amdgpu_device *adev = irq_params->adev;
struct amdgpu_crtc *acrtc;
struct dm_crtc_state *acrtc_state;
+ unsigned long flags;
acrtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_VBLANK);
if (acrtc) {
- drm_crtc_handle_vblank(&acrtc->base);
- amdgpu_dm_crtc_handle_crc_irq(&acrtc->base);
-
acrtc_state = to_dm_crtc_state(acrtc->base.state);
- if (acrtc_state->stream &&
+ DRM_DEBUG_DRIVER("crtc:%d, vupdate-vrr:%d\n", acrtc->crtc_id,
+ amdgpu_dm_vrr_active(acrtc_state));
+
+ /* Core vblank handling at start of front-porch is only possible
+ * in non-vrr mode, as only there vblank timestamping will give
+ * valid results while done in front-porch. Otherwise defer it
+ * to dm_vupdate_high_irq after end of front-porch.
+ */
+ if (!amdgpu_dm_vrr_active(acrtc_state))
+ drm_crtc_handle_vblank(&acrtc->base);
+
+ /* Following stuff must happen at start of vblank, for crc
+ * computation and below-the-range btr support in vrr mode.
+ */
+ amdgpu_dm_crtc_handle_crc_irq(&acrtc->base);
+
+ if (acrtc_state->stream && adev->family >= AMDGPU_FAMILY_AI &&
acrtc_state->vrr_params.supported &&
acrtc_state->freesync_config.state == VRR_STATE_ACTIVE_VARIABLE) {
+ spin_lock_irqsave(&adev->ddev->event_lock, flags);
mod_freesync_handle_v_update(
adev->dm.freesync_module,
acrtc_state->stream,
adev->dm.dc,
acrtc_state->stream,
&acrtc_state->vrr_params.adjust);
+ spin_unlock_irqrestore(&adev->ddev->event_lock, flags);
}
}
}
if (amdgpu_dc_feature_mask & DC_FBC_MASK)
init_data.flags.fbc_support = true;
+ init_data.flags.power_down_display_on_boot = true;
+
/* Display Core create. */
adev->dm.dc = dc_create(&init_data);
struct drm_plane *plane;
struct drm_plane_state *new_plane_state;
struct dm_plane_state *dm_new_plane_state;
+ struct dm_atomic_state *dm_state = to_dm_atomic_state(dm->atomic_obj.state);
enum dc_connection_type new_connection_type = dc_connection_none;
int i;
+ /* Recreate dc_state - DC invalidates it when setting power state to S3. */
+ dc_release_state(dm_state->context);
+ dm_state->context = dc_create_state(dm->dc);
+ /* TODO: Remove dc_state->dccg, use dc->dccg directly. */
+ dc_resource_state_construct(dm->dc, dm_state->context);
+
/* power on hardware */
dc_set_power_state(dm->dc, DC_ACPI_CM_POWER_STATE_D0);
dm_crtc_high_irq, c_irq_params);
}
+ /* Use VUPDATE interrupt */
+ for (i = VISLANDS30_IV_SRCID_D1_V_UPDATE_INT; i <= VISLANDS30_IV_SRCID_D6_V_UPDATE_INT; i += 2) {
+ r = amdgpu_irq_add_id(adev, client_id, i, &adev->vupdate_irq);
+ if (r) {
+ DRM_ERROR("Failed to add vupdate irq id!\n");
+ return r;
+ }
+
+ int_params.int_context = INTERRUPT_HIGH_IRQ_CONTEXT;
+ int_params.irq_source =
+ dc_interrupt_to_irq_source(dc, i, 0);
+
+ c_irq_params = &adev->dm.vupdate_params[int_params.irq_source - DC_IRQ_SOURCE_VUPDATE1];
+
+ c_irq_params->adev = adev;
+ c_irq_params->irq_src = int_params.irq_source;
+
+ amdgpu_dm_irq_register_interrupt(adev, &int_params,
+ dm_vupdate_high_irq, c_irq_params);
+ }
+
/* Use GRPH_PFLIP interrupt */
for (i = VISLANDS30_IV_SRCID_D1_GRPH_PFLIP;
i <= VISLANDS30_IV_SRCID_D6_GRPH_PFLIP; i += 2) {
dm_crtc_high_irq, c_irq_params);
}
+ /* Use VUPDATE_NO_LOCK interrupt on DCN, which seems to correspond to
+ * the regular VUPDATE interrupt on DCE. We want DC_IRQ_SOURCE_VUPDATEx
+ * to trigger at end of each vblank, regardless of state of the lock,
+ * matching DCE behaviour.
+ */
+ for (i = DCN_1_0__SRCID__OTG0_IHC_V_UPDATE_NO_LOCK_INTERRUPT;
+ i <= DCN_1_0__SRCID__OTG0_IHC_V_UPDATE_NO_LOCK_INTERRUPT + adev->mode_info.num_crtc - 1;
+ i++) {
+ r = amdgpu_irq_add_id(adev, SOC15_IH_CLIENTID_DCE, i, &adev->vupdate_irq);
+
+ if (r) {
+ DRM_ERROR("Failed to add vupdate irq id!\n");
+ return r;
+ }
+
+ int_params.int_context = INTERRUPT_HIGH_IRQ_CONTEXT;
+ int_params.irq_source =
+ dc_interrupt_to_irq_source(dc, i, 0);
+
+ c_irq_params = &adev->dm.vupdate_params[int_params.irq_source - DC_IRQ_SOURCE_VUPDATE1];
+
+ c_irq_params->adev = adev;
+ c_irq_params->irq_src = int_params.irq_source;
+
+ amdgpu_dm_irq_register_interrupt(adev, &int_params,
+ dm_vupdate_high_irq, c_irq_params);
+ }
+
/* Use GRPH_PFLIP interrupt */
for (i = DCN_1_0__SRCID__HUBP0_FLIP_INTERRUPT;
i <= DCN_1_0__SRCID__HUBP0_FLIP_INTERRUPT + adev->mode_info.num_crtc - 1;
struct amdgpu_device *adev = dev->dev_private;
struct amdgpu_display_manager *dm = &adev->dm;
struct drm_private_state *priv_state;
- int ret;
if (*dm_state)
return 0;
- ret = drm_modeset_lock(&dm->atomic_obj_lock, state->acquire_ctx);
- if (ret)
- return ret;
-
priv_state = drm_atomic_get_private_obj_state(state, &dm->atomic_obj);
if (IS_ERR(priv_state))
return PTR_ERR(priv_state);
__drm_atomic_helper_private_obj_duplicate_state(obj, &new_state->base);
- new_state->context = dc_create_state();
+ old_state = to_dm_atomic_state(obj->state);
+
+ if (old_state && old_state->context)
+ new_state->context = dc_copy_state(old_state->context);
+
if (!new_state->context) {
kfree(new_state);
return NULL;
}
- old_state = to_dm_atomic_state(obj->state);
- if (old_state && old_state->context)
- dc_resource_state_copy_construct(old_state->context,
- new_state->context);
-
return &new_state->base;
}
adev->ddev->mode_config.fb_base = adev->gmc.aper_base;
- drm_modeset_lock_init(&adev->dm.atomic_obj_lock);
-
state = kzalloc(sizeof(*state), GFP_KERNEL);
if (!state)
return -ENOMEM;
- state->context = dc_create_state();
+ state->context = dc_create_state(adev->dm.dc);
if (!state->context) {
kfree(state);
return -ENOMEM;
#endif
static int initialize_plane(struct amdgpu_display_manager *dm,
- struct amdgpu_mode_info *mode_info,
- int plane_id)
+ struct amdgpu_mode_info *mode_info, int plane_id,
+ enum drm_plane_type plane_type,
+ const struct dc_plane_cap *plane_cap)
{
struct drm_plane *plane;
unsigned long possible_crtcs;
int ret = 0;
plane = kzalloc(sizeof(struct drm_plane), GFP_KERNEL);
- mode_info->planes[plane_id] = plane;
-
if (!plane) {
DRM_ERROR("KMS: Failed to allocate plane\n");
return -ENOMEM;
}
- plane->type = mode_info->plane_type[plane_id];
+ plane->type = plane_type;
/*
- * HACK: IGT tests expect that each plane can only have
- * one possible CRTC. For now, set one CRTC for each
- * plane that is not an underlay, but still allow multiple
- * CRTCs for underlay planes.
+ * HACK: IGT tests expect that the primary plane for a CRTC
+ * can only have one possible CRTC. Only expose support for
+ * any CRTC if they're not going to be used as a primary plane
+ * for a CRTC - like overlay or underlay planes.
*/
possible_crtcs = 1 << plane_id;
if (plane_id >= dm->dc->caps.max_streams)
possible_crtcs = 0xff;
- ret = amdgpu_dm_plane_init(dm, mode_info->planes[plane_id], possible_crtcs);
+ ret = amdgpu_dm_plane_init(dm, plane, possible_crtcs, plane_cap);
if (ret) {
DRM_ERROR("KMS: Failed to initialize plane\n");
+ kfree(plane);
return ret;
}
+ if (mode_info)
+ mode_info->planes[plane_id] = plane;
+
return ret;
}
struct amdgpu_encoder *aencoder = NULL;
struct amdgpu_mode_info *mode_info = &adev->mode_info;
uint32_t link_cnt;
- int32_t total_overlay_planes, total_primary_planes;
+ int32_t primary_planes;
enum dc_connection_type new_connection_type = dc_connection_none;
+ const struct dc_plane_cap *plane;
link_cnt = dm->dc->caps.max_links;
if (amdgpu_dm_mode_config_init(dm->adev)) {
return -EINVAL;
}
- /* Identify the number of planes to be initialized */
- total_overlay_planes = dm->dc->caps.max_slave_planes;
- total_primary_planes = dm->dc->caps.max_planes - dm->dc->caps.max_slave_planes;
+ /* There is one primary plane per CRTC */
+ primary_planes = dm->dc->caps.max_streams;
+ ASSERT(primary_planes <= AMDGPU_MAX_PLANES);
- /* First initialize overlay planes, index starting after primary planes */
- for (i = (total_overlay_planes - 1); i >= 0; i--) {
- if (initialize_plane(dm, mode_info, (total_primary_planes + i))) {
- DRM_ERROR("KMS: Failed to initialize overlay plane\n");
+ /*
+ * Initialize primary planes, implicit planes for legacy IOCTLS.
+ * Order is reversed to match iteration order in atomic check.
+ */
+ for (i = (primary_planes - 1); i >= 0; i--) {
+ plane = &dm->dc->caps.planes[i];
+
+ if (initialize_plane(dm, mode_info, i,
+ DRM_PLANE_TYPE_PRIMARY, plane)) {
+ DRM_ERROR("KMS: Failed to initialize primary plane\n");
goto fail;
}
}
- /* Initialize primary planes */
- for (i = (total_primary_planes - 1); i >= 0; i--) {
- if (initialize_plane(dm, mode_info, i)) {
- DRM_ERROR("KMS: Failed to initialize primary plane\n");
+ /*
+ * Initialize overlay planes, index starting after primary planes.
+ * These planes have a higher DRM index than the primary planes since
+ * they should be considered as having a higher z-order.
+ * Order is reversed to match iteration order in atomic check.
+ *
+ * Only support DCN for now, and only expose one so we don't encourage
+ * userspace to use up all the pipes.
+ */
+ for (i = 0; i < dm->dc->caps.max_planes; ++i) {
+ struct dc_plane_cap *plane = &dm->dc->caps.planes[i];
+
+ if (plane->type != DC_PLANE_TYPE_DCN_UNIVERSAL)
+ continue;
+
+ if (!plane->blends_with_above || !plane->blends_with_below)
+ continue;
+
+ if (!plane->pixel_format_support.argb8888)
+ continue;
+
+ if (initialize_plane(dm, NULL, primary_planes + i,
+ DRM_PLANE_TYPE_OVERLAY, plane)) {
+ DRM_ERROR("KMS: Failed to initialize overlay plane\n");
goto fail;
}
+
+ /* Only create one overlay plane. */
+ break;
}
for (i = 0; i < dm->dc->caps.max_streams; i++)
fail:
kfree(aencoder);
kfree(aconnector);
- for (i = 0; i < dm->dc->caps.max_planes; i++)
- kfree(mode_info->planes[i]);
+
return -EINVAL;
}
adev->mode_info.num_crtc = 6;
adev->mode_info.num_hpd = 6;
adev->mode_info.num_dig = 6;
- adev->mode_info.plane_type = dm_plane_type_default;
break;
case CHIP_KAVERI:
adev->mode_info.num_crtc = 4;
adev->mode_info.num_hpd = 6;
adev->mode_info.num_dig = 7;
- adev->mode_info.plane_type = dm_plane_type_default;
break;
case CHIP_KABINI:
case CHIP_MULLINS:
adev->mode_info.num_crtc = 2;
adev->mode_info.num_hpd = 6;
adev->mode_info.num_dig = 6;
- adev->mode_info.plane_type = dm_plane_type_default;
break;
case CHIP_FIJI:
case CHIP_TONGA:
adev->mode_info.num_crtc = 6;
adev->mode_info.num_hpd = 6;
adev->mode_info.num_dig = 7;
- adev->mode_info.plane_type = dm_plane_type_default;
break;
case CHIP_CARRIZO:
adev->mode_info.num_crtc = 3;
adev->mode_info.num_hpd = 6;
adev->mode_info.num_dig = 9;
- adev->mode_info.plane_type = dm_plane_type_carizzo;
break;
case CHIP_STONEY:
adev->mode_info.num_crtc = 2;
adev->mode_info.num_hpd = 6;
adev->mode_info.num_dig = 9;
- adev->mode_info.plane_type = dm_plane_type_stoney;
break;
case CHIP_POLARIS11:
case CHIP_POLARIS12:
adev->mode_info.num_crtc = 5;
adev->mode_info.num_hpd = 5;
adev->mode_info.num_dig = 5;
- adev->mode_info.plane_type = dm_plane_type_default;
break;
case CHIP_POLARIS10:
case CHIP_VEGAM:
adev->mode_info.num_crtc = 6;
adev->mode_info.num_hpd = 6;
adev->mode_info.num_dig = 6;
- adev->mode_info.plane_type = dm_plane_type_default;
break;
case CHIP_VEGA10:
case CHIP_VEGA12:
adev->mode_info.num_crtc = 6;
adev->mode_info.num_hpd = 6;
adev->mode_info.num_dig = 6;
- adev->mode_info.plane_type = dm_plane_type_default;
break;
#if defined(CONFIG_DRM_AMD_DC_DCN1_0)
case CHIP_RAVEN:
adev->mode_info.num_crtc = 4;
adev->mode_info.num_hpd = 4;
adev->mode_info.num_dig = 4;
- adev->mode_info.plane_type = dm_plane_type_default;
break;
#endif
default:
.destroy = amdgpu_dm_encoder_destroy,
};
-static bool fill_rects_from_plane_state(const struct drm_plane_state *state,
- struct dc_plane_state *plane_state)
+
+static int fill_dc_scaling_info(const struct drm_plane_state *state,
+ struct dc_scaling_info *scaling_info)
{
- plane_state->src_rect.x = state->src_x >> 16;
- plane_state->src_rect.y = state->src_y >> 16;
- /* we ignore the mantissa for now and do not deal with floating pixels :( */
- plane_state->src_rect.width = state->src_w >> 16;
+ int scale_w, scale_h;
- if (plane_state->src_rect.width == 0)
- return false;
+ memset(scaling_info, 0, sizeof(*scaling_info));
- plane_state->src_rect.height = state->src_h >> 16;
- if (plane_state->src_rect.height == 0)
- return false;
+ /* Source is fixed 16.16 but we ignore mantissa for now... */
+ scaling_info->src_rect.x = state->src_x >> 16;
+ scaling_info->src_rect.y = state->src_y >> 16;
+
+ scaling_info->src_rect.width = state->src_w >> 16;
+ if (scaling_info->src_rect.width == 0)
+ return -EINVAL;
- plane_state->dst_rect.x = state->crtc_x;
- plane_state->dst_rect.y = state->crtc_y;
+ scaling_info->src_rect.height = state->src_h >> 16;
+ if (scaling_info->src_rect.height == 0)
+ return -EINVAL;
+
+ scaling_info->dst_rect.x = state->crtc_x;
+ scaling_info->dst_rect.y = state->crtc_y;
if (state->crtc_w == 0)
- return false;
+ return -EINVAL;
- plane_state->dst_rect.width = state->crtc_w;
+ scaling_info->dst_rect.width = state->crtc_w;
if (state->crtc_h == 0)
- return false;
+ return -EINVAL;
- plane_state->dst_rect.height = state->crtc_h;
+ scaling_info->dst_rect.height = state->crtc_h;
- plane_state->clip_rect = plane_state->dst_rect;
+ /* DRM doesn't specify clipping on destination output. */
+ scaling_info->clip_rect = scaling_info->dst_rect;
- switch (state->rotation & DRM_MODE_ROTATE_MASK) {
- case DRM_MODE_ROTATE_0:
- plane_state->rotation = ROTATION_ANGLE_0;
- break;
- case DRM_MODE_ROTATE_90:
- plane_state->rotation = ROTATION_ANGLE_90;
- break;
- case DRM_MODE_ROTATE_180:
- plane_state->rotation = ROTATION_ANGLE_180;
- break;
- case DRM_MODE_ROTATE_270:
- plane_state->rotation = ROTATION_ANGLE_270;
- break;
- default:
- plane_state->rotation = ROTATION_ANGLE_0;
- break;
- }
+ /* TODO: Validate scaling per-format with DC plane caps */
+ scale_w = scaling_info->dst_rect.width * 1000 /
+ scaling_info->src_rect.width;
- return true;
+ if (scale_w < 250 || scale_w > 16000)
+ return -EINVAL;
+
+ scale_h = scaling_info->dst_rect.height * 1000 /
+ scaling_info->src_rect.height;
+
+ if (scale_h < 250 || scale_h > 16000)
+ return -EINVAL;
+
+ /*
+ * The "scaling_quality" can be ignored for now, quality = 0 has DC
+ * assume reasonable defaults based on the format.
+ */
+
+ return 0;
}
+
static int get_fb_info(const struct amdgpu_framebuffer *amdgpu_fb,
uint64_t *tiling_flags)
{
return offset ? (address + offset * 256) : 0;
}
-static bool fill_plane_dcc_attributes(struct amdgpu_device *adev,
- const struct amdgpu_framebuffer *afb,
- struct dc_plane_state *plane_state,
- uint64_t info)
+static int
+fill_plane_dcc_attributes(struct amdgpu_device *adev,
+ const struct amdgpu_framebuffer *afb,
+ const enum surface_pixel_format format,
+ const enum dc_rotation_angle rotation,
+ const union plane_size *plane_size,
+ const union dc_tiling_info *tiling_info,
+ const uint64_t info,
+ struct dc_plane_dcc_param *dcc,
+ struct dc_plane_address *address)
{
struct dc *dc = adev->dm.dc;
struct dc_dcc_surface_param input;
memset(&output, 0, sizeof(output));
if (!offset)
- return false;
+ return 0;
+
+ if (format >= SURFACE_PIXEL_FORMAT_VIDEO_BEGIN)
+ return 0;
if (!dc->cap_funcs.get_dcc_compression_cap)
- return false;
+ return -EINVAL;
- input.format = plane_state->format;
- input.surface_size.width =
- plane_state->plane_size.grph.surface_size.width;
- input.surface_size.height =
- plane_state->plane_size.grph.surface_size.height;
- input.swizzle_mode = plane_state->tiling_info.gfx9.swizzle;
+ input.format = format;
+ input.surface_size.width = plane_size->grph.surface_size.width;
+ input.surface_size.height = plane_size->grph.surface_size.height;
+ input.swizzle_mode = tiling_info->gfx9.swizzle;
- if (plane_state->rotation == ROTATION_ANGLE_0 ||
- plane_state->rotation == ROTATION_ANGLE_180)
+ if (rotation == ROTATION_ANGLE_0 || rotation == ROTATION_ANGLE_180)
input.scan = SCAN_DIRECTION_HORIZONTAL;
- else if (plane_state->rotation == ROTATION_ANGLE_90 ||
- plane_state->rotation == ROTATION_ANGLE_270)
+ else if (rotation == ROTATION_ANGLE_90 || rotation == ROTATION_ANGLE_270)
input.scan = SCAN_DIRECTION_VERTICAL;
if (!dc->cap_funcs.get_dcc_compression_cap(dc, &input, &output))
- return false;
+ return -EINVAL;
if (!output.capable)
- return false;
+ return -EINVAL;
if (i64b == 0 && output.grph.rgb.independent_64b_blks != 0)
- return false;
+ return -EINVAL;
- plane_state->dcc.enable = 1;
- plane_state->dcc.grph.meta_pitch =
+ dcc->enable = 1;
+ dcc->grph.meta_pitch =
AMDGPU_TILING_GET(info, DCC_PITCH_MAX) + 1;
- plane_state->dcc.grph.independent_64b_blks = i64b;
+ dcc->grph.independent_64b_blks = i64b;
dcc_address = get_dcc_address(afb->address, info);
- plane_state->address.grph.meta_addr.low_part =
- lower_32_bits(dcc_address);
- plane_state->address.grph.meta_addr.high_part =
- upper_32_bits(dcc_address);
+ address->grph.meta_addr.low_part = lower_32_bits(dcc_address);
+ address->grph.meta_addr.high_part = upper_32_bits(dcc_address);
- return true;
+ return 0;
}
-static int fill_plane_attributes_from_fb(struct amdgpu_device *adev,
- struct dc_plane_state *plane_state,
- const struct amdgpu_framebuffer *amdgpu_fb)
-{
- uint64_t tiling_flags;
- unsigned int awidth;
- const struct drm_framebuffer *fb = &amdgpu_fb->base;
- int ret = 0;
- struct drm_format_name_buf format_name;
-
- ret = get_fb_info(
- amdgpu_fb,
- &tiling_flags);
-
- if (ret)
- return ret;
-
- switch (fb->format->format) {
- case DRM_FORMAT_C8:
- plane_state->format = SURFACE_PIXEL_FORMAT_GRPH_PALETA_256_COLORS;
- break;
- case DRM_FORMAT_RGB565:
- plane_state->format = SURFACE_PIXEL_FORMAT_GRPH_RGB565;
- break;
- case DRM_FORMAT_XRGB8888:
- case DRM_FORMAT_ARGB8888:
- plane_state->format = SURFACE_PIXEL_FORMAT_GRPH_ARGB8888;
- break;
- case DRM_FORMAT_XRGB2101010:
- case DRM_FORMAT_ARGB2101010:
- plane_state->format = SURFACE_PIXEL_FORMAT_GRPH_ARGB2101010;
- break;
- case DRM_FORMAT_XBGR2101010:
- case DRM_FORMAT_ABGR2101010:
- plane_state->format = SURFACE_PIXEL_FORMAT_GRPH_ABGR2101010;
- break;
- case DRM_FORMAT_XBGR8888:
- case DRM_FORMAT_ABGR8888:
- plane_state->format = SURFACE_PIXEL_FORMAT_GRPH_ABGR8888;
- break;
- case DRM_FORMAT_NV21:
- plane_state->format = SURFACE_PIXEL_FORMAT_VIDEO_420_YCbCr;
- break;
- case DRM_FORMAT_NV12:
- plane_state->format = SURFACE_PIXEL_FORMAT_VIDEO_420_YCrCb;
- break;
- default:
- DRM_ERROR("Unsupported screen format %s\n",
- drm_get_format_name(fb->format->format, &format_name));
- return -EINVAL;
- }
-
- memset(&plane_state->address, 0, sizeof(plane_state->address));
- memset(&plane_state->tiling_info, 0, sizeof(plane_state->tiling_info));
- memset(&plane_state->dcc, 0, sizeof(plane_state->dcc));
-
- if (plane_state->format < SURFACE_PIXEL_FORMAT_VIDEO_BEGIN) {
- plane_state->address.type = PLN_ADDR_TYPE_GRAPHICS;
- plane_state->plane_size.grph.surface_size.x = 0;
- plane_state->plane_size.grph.surface_size.y = 0;
- plane_state->plane_size.grph.surface_size.width = fb->width;
- plane_state->plane_size.grph.surface_size.height = fb->height;
- plane_state->plane_size.grph.surface_pitch =
- fb->pitches[0] / fb->format->cpp[0];
- /* TODO: unhardcode */
- plane_state->color_space = COLOR_SPACE_SRGB;
+static int
+fill_plane_buffer_attributes(struct amdgpu_device *adev,
+ const struct amdgpu_framebuffer *afb,
+ const enum surface_pixel_format format,
+ const enum dc_rotation_angle rotation,
+ const uint64_t tiling_flags,
+ union dc_tiling_info *tiling_info,
+ union plane_size *plane_size,
+ struct dc_plane_dcc_param *dcc,
+ struct dc_plane_address *address)
+{
+ const struct drm_framebuffer *fb = &afb->base;
+ int ret;
+ memset(tiling_info, 0, sizeof(*tiling_info));
+ memset(plane_size, 0, sizeof(*plane_size));
+ memset(dcc, 0, sizeof(*dcc));
+ memset(address, 0, sizeof(*address));
+
+ if (format < SURFACE_PIXEL_FORMAT_VIDEO_BEGIN) {
+ plane_size->grph.surface_size.x = 0;
+ plane_size->grph.surface_size.y = 0;
+ plane_size->grph.surface_size.width = fb->width;
+ plane_size->grph.surface_size.height = fb->height;
+ plane_size->grph.surface_pitch =
+ fb->pitches[0] / fb->format->cpp[0];
+
+ address->type = PLN_ADDR_TYPE_GRAPHICS;
+ address->grph.addr.low_part = lower_32_bits(afb->address);
+ address->grph.addr.high_part = upper_32_bits(afb->address);
} else {
- awidth = ALIGN(fb->width, 64);
- plane_state->address.type = PLN_ADDR_TYPE_VIDEO_PROGRESSIVE;
- plane_state->plane_size.video.luma_size.x = 0;
- plane_state->plane_size.video.luma_size.y = 0;
- plane_state->plane_size.video.luma_size.width = awidth;
- plane_state->plane_size.video.luma_size.height = fb->height;
- /* TODO: unhardcode */
- plane_state->plane_size.video.luma_pitch = awidth;
-
- plane_state->plane_size.video.chroma_size.x = 0;
- plane_state->plane_size.video.chroma_size.y = 0;
- plane_state->plane_size.video.chroma_size.width = awidth;
- plane_state->plane_size.video.chroma_size.height = fb->height;
- plane_state->plane_size.video.chroma_pitch = awidth / 2;
-
- /* TODO: unhardcode */
- plane_state->color_space = COLOR_SPACE_YCBCR709;
+ uint64_t chroma_addr = afb->address + fb->offsets[1];
+
+ plane_size->video.luma_size.x = 0;
+ plane_size->video.luma_size.y = 0;
+ plane_size->video.luma_size.width = fb->width;
+ plane_size->video.luma_size.height = fb->height;
+ plane_size->video.luma_pitch =
+ fb->pitches[0] / fb->format->cpp[0];
+
+ plane_size->video.chroma_size.x = 0;
+ plane_size->video.chroma_size.y = 0;
+ /* TODO: set these based on surface format */
+ plane_size->video.chroma_size.width = fb->width / 2;
+ plane_size->video.chroma_size.height = fb->height / 2;
+
+ plane_size->video.chroma_pitch =
+ fb->pitches[1] / fb->format->cpp[1];
+
+ address->type = PLN_ADDR_TYPE_VIDEO_PROGRESSIVE;
+ address->video_progressive.luma_addr.low_part =
+ lower_32_bits(afb->address);
+ address->video_progressive.luma_addr.high_part =
+ upper_32_bits(afb->address);
+ address->video_progressive.chroma_addr.low_part =
+ lower_32_bits(chroma_addr);
+ address->video_progressive.chroma_addr.high_part =
+ upper_32_bits(chroma_addr);
}
/* Fill GFX8 params */
num_banks = AMDGPU_TILING_GET(tiling_flags, NUM_BANKS);
/* XXX fix me for VI */
- plane_state->tiling_info.gfx8.num_banks = num_banks;
- plane_state->tiling_info.gfx8.array_mode =
+ tiling_info->gfx8.num_banks = num_banks;
+ tiling_info->gfx8.array_mode =
DC_ARRAY_2D_TILED_THIN1;
- plane_state->tiling_info.gfx8.tile_split = tile_split;
- plane_state->tiling_info.gfx8.bank_width = bankw;
- plane_state->tiling_info.gfx8.bank_height = bankh;
- plane_state->tiling_info.gfx8.tile_aspect = mtaspect;
- plane_state->tiling_info.gfx8.tile_mode =
+ tiling_info->gfx8.tile_split = tile_split;
+ tiling_info->gfx8.bank_width = bankw;
+ tiling_info->gfx8.bank_height = bankh;
+ tiling_info->gfx8.tile_aspect = mtaspect;
+ tiling_info->gfx8.tile_mode =
DC_ADDR_SURF_MICRO_TILING_DISPLAY;
} else if (AMDGPU_TILING_GET(tiling_flags, ARRAY_MODE)
== DC_ARRAY_1D_TILED_THIN1) {
- plane_state->tiling_info.gfx8.array_mode = DC_ARRAY_1D_TILED_THIN1;
+ tiling_info->gfx8.array_mode = DC_ARRAY_1D_TILED_THIN1;
}
- plane_state->tiling_info.gfx8.pipe_config =
+ tiling_info->gfx8.pipe_config =
AMDGPU_TILING_GET(tiling_flags, PIPE_CONFIG);
if (adev->asic_type == CHIP_VEGA10 ||
adev->asic_type == CHIP_VEGA20 ||
adev->asic_type == CHIP_RAVEN) {
/* Fill GFX9 params */
- plane_state->tiling_info.gfx9.num_pipes =
+ tiling_info->gfx9.num_pipes =
adev->gfx.config.gb_addr_config_fields.num_pipes;
- plane_state->tiling_info.gfx9.num_banks =
+ tiling_info->gfx9.num_banks =
adev->gfx.config.gb_addr_config_fields.num_banks;
- plane_state->tiling_info.gfx9.pipe_interleave =
+ tiling_info->gfx9.pipe_interleave =
adev->gfx.config.gb_addr_config_fields.pipe_interleave_size;
- plane_state->tiling_info.gfx9.num_shader_engines =
+ tiling_info->gfx9.num_shader_engines =
adev->gfx.config.gb_addr_config_fields.num_se;
- plane_state->tiling_info.gfx9.max_compressed_frags =
+ tiling_info->gfx9.max_compressed_frags =
adev->gfx.config.gb_addr_config_fields.max_compress_frags;
- plane_state->tiling_info.gfx9.num_rb_per_se =
+ tiling_info->gfx9.num_rb_per_se =
adev->gfx.config.gb_addr_config_fields.num_rb_per_se;
- plane_state->tiling_info.gfx9.swizzle =
+ tiling_info->gfx9.swizzle =
AMDGPU_TILING_GET(tiling_flags, SWIZZLE_MODE);
- plane_state->tiling_info.gfx9.shaderEnable = 1;
+ tiling_info->gfx9.shaderEnable = 1;
- fill_plane_dcc_attributes(adev, amdgpu_fb, plane_state,
- tiling_flags);
+ ret = fill_plane_dcc_attributes(adev, afb, format, rotation,
+ plane_size, tiling_info,
+ tiling_flags, dcc, address);
+ if (ret)
+ return ret;
}
- plane_state->visible = true;
- plane_state->scaling_quality.h_taps_c = 0;
- plane_state->scaling_quality.v_taps_c = 0;
-
- /* is this needed? is plane_state zeroed at allocation? */
- plane_state->scaling_quality.h_taps = 0;
- plane_state->scaling_quality.v_taps = 0;
- plane_state->stereo_format = PLANE_STEREO_FORMAT_NONE;
-
- return ret;
-
+ return 0;
}
-static int fill_plane_attributes(struct amdgpu_device *adev,
- struct dc_plane_state *dc_plane_state,
- struct drm_plane_state *plane_state,
- struct drm_crtc_state *crtc_state)
+static void
+fill_blending_from_plane_state(const struct drm_plane_state *plane_state,
+ bool *per_pixel_alpha, bool *global_alpha,
+ int *global_alpha_value)
{
- const struct amdgpu_framebuffer *amdgpu_fb =
- to_amdgpu_framebuffer(plane_state->fb);
- const struct drm_crtc *crtc = plane_state->crtc;
- int ret = 0;
-
- if (!fill_rects_from_plane_state(plane_state, dc_plane_state))
- return -EINVAL;
-
- ret = fill_plane_attributes_from_fb(
- crtc->dev->dev_private,
- dc_plane_state,
- amdgpu_fb);
+ *per_pixel_alpha = false;
+ *global_alpha = false;
+ *global_alpha_value = 0xff;
- if (ret)
- return ret;
+ if (plane_state->plane->type != DRM_PLANE_TYPE_OVERLAY)
+ return;
- /*
- * Always set input transfer function, since plane state is refreshed
- * every time.
- */
- ret = amdgpu_dm_set_degamma_lut(crtc_state, dc_plane_state);
- if (ret) {
- dc_transfer_func_release(dc_plane_state->in_transfer_func);
- dc_plane_state->in_transfer_func = NULL;
+ if (plane_state->pixel_blend_mode == DRM_MODE_BLEND_PREMULTI) {
+ static const uint32_t alpha_formats[] = {
+ DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_RGBA8888,
+ DRM_FORMAT_ABGR8888,
+ };
+ uint32_t format = plane_state->fb->format->format;
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(alpha_formats); ++i) {
+ if (format == alpha_formats[i]) {
+ *per_pixel_alpha = true;
+ break;
+ }
+ }
}
- return ret;
+ if (plane_state->alpha < 0xffff) {
+ *global_alpha = true;
+ *global_alpha_value = plane_state->alpha >> 8;
+ }
}
-static void update_stream_scaling_settings(const struct drm_display_mode *mode,
- const struct dm_connector_state *dm_state,
- struct dc_stream_state *stream)
+static int
+fill_plane_color_attributes(const struct drm_plane_state *plane_state,
+ const enum surface_pixel_format format,
+ enum dc_color_space *color_space)
{
- enum amdgpu_rmx_type rmx_type;
+ bool full_range;
- struct rect src = { 0 }; /* viewport in composition space*/
- struct rect dst = { 0 }; /* stream addressable area */
+ *color_space = COLOR_SPACE_SRGB;
- /* no mode. nothing to be done */
- if (!mode)
- return;
+ /* DRM color properties only affect non-RGB formats. */
+ if (format < SURFACE_PIXEL_FORMAT_VIDEO_BEGIN)
+ return 0;
- /* Full screen scaling by default */
- src.width = mode->hdisplay;
- src.height = mode->vdisplay;
- dst.width = stream->timing.h_addressable;
- dst.height = stream->timing.v_addressable;
+ full_range = (plane_state->color_range == DRM_COLOR_YCBCR_FULL_RANGE);
- if (dm_state) {
- rmx_type = dm_state->scaling;
- if (rmx_type == RMX_ASPECT || rmx_type == RMX_OFF) {
- if (src.width * dst.height <
+ switch (plane_state->color_encoding) {
+ case DRM_COLOR_YCBCR_BT601:
+ if (full_range)
+ *color_space = COLOR_SPACE_YCBCR601;
+ else
+ *color_space = COLOR_SPACE_YCBCR601_LIMITED;
+ break;
+
+ case DRM_COLOR_YCBCR_BT709:
+ if (full_range)
+ *color_space = COLOR_SPACE_YCBCR709;
+ else
+ *color_space = COLOR_SPACE_YCBCR709_LIMITED;
+ break;
+
+ case DRM_COLOR_YCBCR_BT2020:
+ if (full_range)
+ *color_space = COLOR_SPACE_2020_YCBCR;
+ else
+ return -EINVAL;
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int
+fill_dc_plane_info_and_addr(struct amdgpu_device *adev,
+ const struct drm_plane_state *plane_state,
+ const uint64_t tiling_flags,
+ struct dc_plane_info *plane_info,
+ struct dc_plane_address *address)
+{
+ const struct drm_framebuffer *fb = plane_state->fb;
+ const struct amdgpu_framebuffer *afb =
+ to_amdgpu_framebuffer(plane_state->fb);
+ struct drm_format_name_buf format_name;
+ int ret;
+
+ memset(plane_info, 0, sizeof(*plane_info));
+
+ switch (fb->format->format) {
+ case DRM_FORMAT_C8:
+ plane_info->format =
+ SURFACE_PIXEL_FORMAT_GRPH_PALETA_256_COLORS;
+ break;
+ case DRM_FORMAT_RGB565:
+ plane_info->format = SURFACE_PIXEL_FORMAT_GRPH_RGB565;
+ break;
+ case DRM_FORMAT_XRGB8888:
+ case DRM_FORMAT_ARGB8888:
+ plane_info->format = SURFACE_PIXEL_FORMAT_GRPH_ARGB8888;
+ break;
+ case DRM_FORMAT_XRGB2101010:
+ case DRM_FORMAT_ARGB2101010:
+ plane_info->format = SURFACE_PIXEL_FORMAT_GRPH_ARGB2101010;
+ break;
+ case DRM_FORMAT_XBGR2101010:
+ case DRM_FORMAT_ABGR2101010:
+ plane_info->format = SURFACE_PIXEL_FORMAT_GRPH_ABGR2101010;
+ break;
+ case DRM_FORMAT_XBGR8888:
+ case DRM_FORMAT_ABGR8888:
+ plane_info->format = SURFACE_PIXEL_FORMAT_GRPH_ABGR8888;
+ break;
+ case DRM_FORMAT_NV21:
+ plane_info->format = SURFACE_PIXEL_FORMAT_VIDEO_420_YCbCr;
+ break;
+ case DRM_FORMAT_NV12:
+ plane_info->format = SURFACE_PIXEL_FORMAT_VIDEO_420_YCrCb;
+ break;
+ default:
+ DRM_ERROR(
+ "Unsupported screen format %s\n",
+ drm_get_format_name(fb->format->format, &format_name));
+ return -EINVAL;
+ }
+
+ switch (plane_state->rotation & DRM_MODE_ROTATE_MASK) {
+ case DRM_MODE_ROTATE_0:
+ plane_info->rotation = ROTATION_ANGLE_0;
+ break;
+ case DRM_MODE_ROTATE_90:
+ plane_info->rotation = ROTATION_ANGLE_90;
+ break;
+ case DRM_MODE_ROTATE_180:
+ plane_info->rotation = ROTATION_ANGLE_180;
+ break;
+ case DRM_MODE_ROTATE_270:
+ plane_info->rotation = ROTATION_ANGLE_270;
+ break;
+ default:
+ plane_info->rotation = ROTATION_ANGLE_0;
+ break;
+ }
+
+ plane_info->visible = true;
+ plane_info->stereo_format = PLANE_STEREO_FORMAT_NONE;
+
+ ret = fill_plane_color_attributes(plane_state, plane_info->format,
+ &plane_info->color_space);
+ if (ret)
+ return ret;
+
+ ret = fill_plane_buffer_attributes(adev, afb, plane_info->format,
+ plane_info->rotation, tiling_flags,
+ &plane_info->tiling_info,
+ &plane_info->plane_size,
+ &plane_info->dcc, address);
+ if (ret)
+ return ret;
+
+ fill_blending_from_plane_state(
+ plane_state, &plane_info->per_pixel_alpha,
+ &plane_info->global_alpha, &plane_info->global_alpha_value);
+
+ return 0;
+}
+
+static int fill_dc_plane_attributes(struct amdgpu_device *adev,
+ struct dc_plane_state *dc_plane_state,
+ struct drm_plane_state *plane_state,
+ struct drm_crtc_state *crtc_state)
+{
+ const struct amdgpu_framebuffer *amdgpu_fb =
+ to_amdgpu_framebuffer(plane_state->fb);
+ struct dc_scaling_info scaling_info;
+ struct dc_plane_info plane_info;
+ uint64_t tiling_flags;
+ int ret;
+
+ ret = fill_dc_scaling_info(plane_state, &scaling_info);
+ if (ret)
+ return ret;
+
+ dc_plane_state->src_rect = scaling_info.src_rect;
+ dc_plane_state->dst_rect = scaling_info.dst_rect;
+ dc_plane_state->clip_rect = scaling_info.clip_rect;
+ dc_plane_state->scaling_quality = scaling_info.scaling_quality;
+
+ ret = get_fb_info(amdgpu_fb, &tiling_flags);
+ if (ret)
+ return ret;
+
+ ret = fill_dc_plane_info_and_addr(adev, plane_state, tiling_flags,
+ &plane_info,
+ &dc_plane_state->address);
+ if (ret)
+ return ret;
+
+ dc_plane_state->format = plane_info.format;
+ dc_plane_state->color_space = plane_info.color_space;
+ dc_plane_state->format = plane_info.format;
+ dc_plane_state->plane_size = plane_info.plane_size;
+ dc_plane_state->rotation = plane_info.rotation;
+ dc_plane_state->horizontal_mirror = plane_info.horizontal_mirror;
+ dc_plane_state->stereo_format = plane_info.stereo_format;
+ dc_plane_state->tiling_info = plane_info.tiling_info;
+ dc_plane_state->visible = plane_info.visible;
+ dc_plane_state->per_pixel_alpha = plane_info.per_pixel_alpha;
+ dc_plane_state->global_alpha = plane_info.global_alpha;
+ dc_plane_state->global_alpha_value = plane_info.global_alpha_value;
+ dc_plane_state->dcc = plane_info.dcc;
+
+ /*
+ * Always set input transfer function, since plane state is refreshed
+ * every time.
+ */
+ ret = amdgpu_dm_set_degamma_lut(crtc_state, dc_plane_state);
+ if (ret) {
+ dc_transfer_func_release(dc_plane_state->in_transfer_func);
+ dc_plane_state->in_transfer_func = NULL;
+ }
+
+ return ret;
+}
+
+static void update_stream_scaling_settings(const struct drm_display_mode *mode,
+ const struct dm_connector_state *dm_state,
+ struct dc_stream_state *stream)
+{
+ enum amdgpu_rmx_type rmx_type;
+
+ struct rect src = { 0 }; /* viewport in composition space*/
+ struct rect dst = { 0 }; /* stream addressable area */
+
+ /* no mode. nothing to be done */
+ if (!mode)
+ return;
+
+ /* Full screen scaling by default */
+ src.width = mode->hdisplay;
+ src.height = mode->vdisplay;
+ dst.width = stream->timing.h_addressable;
+ dst.height = stream->timing.v_addressable;
+
+ if (dm_state) {
+ rmx_type = dm_state->scaling;
+ if (rmx_type == RMX_ASPECT || rmx_type == RMX_OFF) {
+ if (src.width * dst.height <
src.height * dst.width) {
/* height needs less upscaling/more downscaling */
dst.width = src.width *
dc_stream_retain(state->stream);
}
+ state->active_planes = cur->active_planes;
+ state->interrupts_enabled = cur->interrupts_enabled;
state->vrr_params = cur->vrr_params;
state->vrr_infopacket = cur->vrr_infopacket;
state->abm_level = cur->abm_level;
return &state->base;
}
+static inline int dm_set_vupdate_irq(struct drm_crtc *crtc, bool enable)
+{
+ enum dc_irq_source irq_source;
+ struct amdgpu_crtc *acrtc = to_amdgpu_crtc(crtc);
+ struct amdgpu_device *adev = crtc->dev->dev_private;
+ int rc;
+
+ irq_source = IRQ_TYPE_VUPDATE + acrtc->otg_inst;
+
+ rc = dc_interrupt_set(adev->dm.dc, irq_source, enable) ? 0 : -EBUSY;
+
+ DRM_DEBUG_DRIVER("crtc %d - vupdate irq %sabling: r=%d\n",
+ acrtc->crtc_id, enable ? "en" : "dis", rc);
+ return rc;
+}
static inline int dm_set_vblank(struct drm_crtc *crtc, bool enable)
{
enum dc_irq_source irq_source;
struct amdgpu_crtc *acrtc = to_amdgpu_crtc(crtc);
struct amdgpu_device *adev = crtc->dev->dev_private;
+ struct dm_crtc_state *acrtc_state = to_dm_crtc_state(crtc->state);
+ int rc = 0;
+
+ if (enable) {
+ /* vblank irq on -> Only need vupdate irq in vrr mode */
+ if (amdgpu_dm_vrr_active(acrtc_state))
+ rc = dm_set_vupdate_irq(crtc, true);
+ } else {
+ /* vblank irq off -> vupdate irq off */
+ rc = dm_set_vupdate_irq(crtc, false);
+ }
+
+ if (rc)
+ return rc;
irq_source = IRQ_TYPE_VBLANK + acrtc->otg_inst;
return dc_interrupt_set(adev->dm.dc, irq_source, enable) ? 0 : -EBUSY;
{
}
+static bool does_crtc_have_active_cursor(struct drm_crtc_state *new_crtc_state)
+{
+ struct drm_device *dev = new_crtc_state->crtc->dev;
+ struct drm_plane *plane;
+
+ drm_for_each_plane_mask(plane, dev, new_crtc_state->plane_mask) {
+ if (plane->type == DRM_PLANE_TYPE_CURSOR)
+ return true;
+ }
+
+ return false;
+}
+
+static int count_crtc_active_planes(struct drm_crtc_state *new_crtc_state)
+{
+ struct drm_atomic_state *state = new_crtc_state->state;
+ struct drm_plane *plane;
+ int num_active = 0;
+
+ drm_for_each_plane_mask(plane, state->dev, new_crtc_state->plane_mask) {
+ struct drm_plane_state *new_plane_state;
+
+ /* Cursor planes are "fake". */
+ if (plane->type == DRM_PLANE_TYPE_CURSOR)
+ continue;
+
+ new_plane_state = drm_atomic_get_new_plane_state(state, plane);
+
+ if (!new_plane_state) {
+ /*
+ * The plane is enable on the CRTC and hasn't changed
+ * state. This means that it previously passed
+ * validation and is therefore enabled.
+ */
+ num_active += 1;
+ continue;
+ }
+
+ /* We need a framebuffer to be considered enabled. */
+ num_active += (new_plane_state->fb != NULL);
+ }
+
+ return num_active;
+}
+
+/*
+ * Sets whether interrupts should be enabled on a specific CRTC.
+ * We require that the stream be enabled and that there exist active
+ * DC planes on the stream.
+ */
+static void
+dm_update_crtc_interrupt_state(struct drm_crtc *crtc,
+ struct drm_crtc_state *new_crtc_state)
+{
+ struct dm_crtc_state *dm_new_crtc_state =
+ to_dm_crtc_state(new_crtc_state);
+
+ dm_new_crtc_state->active_planes = 0;
+ dm_new_crtc_state->interrupts_enabled = false;
+
+ if (!dm_new_crtc_state->stream)
+ return;
+
+ dm_new_crtc_state->active_planes =
+ count_crtc_active_planes(new_crtc_state);
+
+ dm_new_crtc_state->interrupts_enabled =
+ dm_new_crtc_state->active_planes > 0;
+}
+
static int dm_crtc_helper_atomic_check(struct drm_crtc *crtc,
struct drm_crtc_state *state)
{
struct dm_crtc_state *dm_crtc_state = to_dm_crtc_state(state);
int ret = -EINVAL;
+ /*
+ * Update interrupt state for the CRTC. This needs to happen whenever
+ * the CRTC has changed or whenever any of its planes have changed.
+ * Atomic check satisfies both of these requirements since the CRTC
+ * is added to the state by DRM during drm_atomic_helper_check_planes.
+ */
+ dm_update_crtc_interrupt_state(crtc, state);
+
if (unlikely(!dm_crtc_state->stream &&
modeset_required(state, NULL, dm_crtc_state->stream))) {
WARN_ON(1);
if (!dm_crtc_state->stream)
return 0;
+ /*
+ * We want at least one hardware plane enabled to use
+ * the stream with a cursor enabled.
+ */
+ if (state->enable && state->active &&
+ does_crtc_have_active_cursor(state) &&
+ dm_crtc_state->active_planes == 0)
+ return -EINVAL;
+
if (dc_validate_stream(dc, dm_crtc_state->stream) == DC_OK)
return 0;
amdgpu_state = kzalloc(sizeof(*amdgpu_state), GFP_KERNEL);
WARN_ON(amdgpu_state == NULL);
- if (amdgpu_state) {
- plane->state = &amdgpu_state->base;
- plane->state->plane = plane;
- plane->state->rotation = DRM_MODE_ROTATE_0;
- }
+ if (amdgpu_state)
+ __drm_atomic_helper_plane_reset(plane, &amdgpu_state->base);
}
static struct drm_plane_state *
struct drm_gem_object *obj;
struct amdgpu_device *adev;
struct amdgpu_bo *rbo;
- uint64_t chroma_addr = 0;
struct dm_plane_state *dm_plane_state_new, *dm_plane_state_old;
- uint64_t tiling_flags, dcc_address;
- unsigned int awidth;
+ uint64_t tiling_flags;
uint32_t domain;
int r;
dm_plane_state_old->dc_state != dm_plane_state_new->dc_state) {
struct dc_plane_state *plane_state = dm_plane_state_new->dc_state;
- if (plane_state->format < SURFACE_PIXEL_FORMAT_VIDEO_BEGIN) {
- plane_state->address.grph.addr.low_part = lower_32_bits(afb->address);
- plane_state->address.grph.addr.high_part = upper_32_bits(afb->address);
-
- dcc_address =
- get_dcc_address(afb->address, tiling_flags);
- plane_state->address.grph.meta_addr.low_part =
- lower_32_bits(dcc_address);
- plane_state->address.grph.meta_addr.high_part =
- upper_32_bits(dcc_address);
- } else {
- awidth = ALIGN(new_state->fb->width, 64);
- plane_state->address.type = PLN_ADDR_TYPE_VIDEO_PROGRESSIVE;
- plane_state->address.video_progressive.luma_addr.low_part
- = lower_32_bits(afb->address);
- plane_state->address.video_progressive.luma_addr.high_part
- = upper_32_bits(afb->address);
- chroma_addr = afb->address + (u64)awidth * new_state->fb->height;
- plane_state->address.video_progressive.chroma_addr.low_part
- = lower_32_bits(chroma_addr);
- plane_state->address.video_progressive.chroma_addr.high_part
- = upper_32_bits(chroma_addr);
- }
+ fill_plane_buffer_attributes(
+ adev, afb, plane_state->format, plane_state->rotation,
+ tiling_flags, &plane_state->tiling_info,
+ &plane_state->plane_size, &plane_state->dcc,
+ &plane_state->address);
}
return 0;
{
struct amdgpu_device *adev = plane->dev->dev_private;
struct dc *dc = adev->dm.dc;
- struct dm_plane_state *dm_plane_state = to_dm_plane_state(state);
+ struct dm_plane_state *dm_plane_state;
+ struct dc_scaling_info scaling_info;
+ int ret;
+
+ dm_plane_state = to_dm_plane_state(state);
if (!dm_plane_state->dc_state)
return 0;
- if (!fill_rects_from_plane_state(state, dm_plane_state->dc_state))
- return -EINVAL;
+ ret = fill_dc_scaling_info(state, &scaling_info);
+ if (ret)
+ return ret;
if (dc_validate_plane(dc, dm_plane_state->dc_state) == DC_OK)
return 0;
DRM_FORMAT_ABGR2101010,
DRM_FORMAT_XBGR8888,
DRM_FORMAT_ABGR8888,
+ DRM_FORMAT_RGB565,
};
-static const uint32_t yuv_formats[] = {
- DRM_FORMAT_NV12,
- DRM_FORMAT_NV21,
+static const uint32_t overlay_formats[] = {
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_RGBA8888,
+ DRM_FORMAT_XBGR8888,
+ DRM_FORMAT_ABGR8888,
+ DRM_FORMAT_RGB565
};
static const u32 cursor_formats[] = {
DRM_FORMAT_ARGB8888
};
-static int amdgpu_dm_plane_init(struct amdgpu_display_manager *dm,
- struct drm_plane *plane,
- unsigned long possible_crtcs)
+static int get_plane_formats(const struct drm_plane *plane,
+ const struct dc_plane_cap *plane_cap,
+ uint32_t *formats, int max_formats)
{
- int res = -EPERM;
+ int i, num_formats = 0;
+
+ /*
+ * TODO: Query support for each group of formats directly from
+ * DC plane caps. This will require adding more formats to the
+ * caps list.
+ */
switch (plane->type) {
case DRM_PLANE_TYPE_PRIMARY:
- res = drm_universal_plane_init(
- dm->adev->ddev,
- plane,
- possible_crtcs,
- &dm_plane_funcs,
- rgb_formats,
- ARRAY_SIZE(rgb_formats),
- NULL, plane->type, NULL);
+ for (i = 0; i < ARRAY_SIZE(rgb_formats); ++i) {
+ if (num_formats >= max_formats)
+ break;
+
+ formats[num_formats++] = rgb_formats[i];
+ }
+
+ if (plane_cap && plane_cap->pixel_format_support.nv12)
+ formats[num_formats++] = DRM_FORMAT_NV12;
break;
+
case DRM_PLANE_TYPE_OVERLAY:
- res = drm_universal_plane_init(
- dm->adev->ddev,
- plane,
- possible_crtcs,
- &dm_plane_funcs,
- yuv_formats,
- ARRAY_SIZE(yuv_formats),
- NULL, plane->type, NULL);
+ for (i = 0; i < ARRAY_SIZE(overlay_formats); ++i) {
+ if (num_formats >= max_formats)
+ break;
+
+ formats[num_formats++] = overlay_formats[i];
+ }
break;
+
case DRM_PLANE_TYPE_CURSOR:
- res = drm_universal_plane_init(
- dm->adev->ddev,
- plane,
- possible_crtcs,
- &dm_plane_funcs,
- cursor_formats,
- ARRAY_SIZE(cursor_formats),
- NULL, plane->type, NULL);
+ for (i = 0; i < ARRAY_SIZE(cursor_formats); ++i) {
+ if (num_formats >= max_formats)
+ break;
+
+ formats[num_formats++] = cursor_formats[i];
+ }
break;
}
+ return num_formats;
+}
+
+static int amdgpu_dm_plane_init(struct amdgpu_display_manager *dm,
+ struct drm_plane *plane,
+ unsigned long possible_crtcs,
+ const struct dc_plane_cap *plane_cap)
+{
+ uint32_t formats[32];
+ int num_formats;
+ int res = -EPERM;
+
+ num_formats = get_plane_formats(plane, plane_cap, formats,
+ ARRAY_SIZE(formats));
+
+ res = drm_universal_plane_init(dm->adev->ddev, plane, possible_crtcs,
+ &dm_plane_funcs, formats, num_formats,
+ NULL, plane->type, NULL);
+ if (res)
+ return res;
+
+ if (plane->type == DRM_PLANE_TYPE_OVERLAY &&
+ plane_cap && plane_cap->per_pixel_alpha) {
+ unsigned int blend_caps = BIT(DRM_MODE_BLEND_PIXEL_NONE) |
+ BIT(DRM_MODE_BLEND_PREMULTI);
+
+ drm_plane_create_alpha_property(plane);
+ drm_plane_create_blend_mode_property(plane, blend_caps);
+ }
+
+ if (plane->type == DRM_PLANE_TYPE_PRIMARY &&
+ plane_cap && plane_cap->pixel_format_support.nv12) {
+ /* This only affects YUV formats. */
+ drm_plane_create_color_properties(
+ plane,
+ BIT(DRM_COLOR_YCBCR_BT601) |
+ BIT(DRM_COLOR_YCBCR_BT709),
+ BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) |
+ BIT(DRM_COLOR_YCBCR_FULL_RANGE),
+ DRM_COLOR_YCBCR_BT709, DRM_COLOR_YCBCR_LIMITED_RANGE);
+ }
+
drm_plane_helper_add(plane, &dm_plane_helper_funcs);
/* Create (reset) the plane state */
if (plane->funcs->reset)
plane->funcs->reset(plane);
-
- return res;
+ return 0;
}
static int amdgpu_dm_crtc_init(struct amdgpu_display_manager *dm,
goto fail;
cursor_plane->type = DRM_PLANE_TYPE_CURSOR;
- res = amdgpu_dm_plane_init(dm, cursor_plane, 0);
+ res = amdgpu_dm_plane_init(dm, cursor_plane, 0, NULL);
acrtc = kzalloc(sizeof(struct amdgpu_crtc), GFP_KERNEL);
if (!acrtc)
DRM_ERROR("Failed to create debugfs for connector");
goto out_free;
}
+ aconnector->debugfs_dpcd_address = 0;
+ aconnector->debugfs_dpcd_size = 0;
#endif
if (connector_type == DRM_MODE_CONNECTOR_DisplayPort
x = plane->state->crtc_x;
y = plane->state->crtc_y;
- /* avivo cursor are offset into the total surface */
- x += crtc->primary->state->src_x >> 16;
- y += crtc->primary->state->src_y >> 16;
+
+ if (crtc->primary->state) {
+ /* avivo cursor are offset into the total surface */
+ x += crtc->primary->state->src_x >> 16;
+ y += crtc->primary->state->src_y >> 16;
+ }
+
if (x < 0) {
xorigin = min(-x, amdgpu_crtc->max_cursor_width - 1);
x = 0;
struct dc_plane_state *surface,
u32 flip_timestamp_in_us)
{
- struct mod_vrr_params vrr_params = new_crtc_state->vrr_params;
+ struct mod_vrr_params vrr_params;
struct dc_info_packet vrr_infopacket = {0};
- struct mod_freesync_config config = new_crtc_state->freesync_config;
+ struct amdgpu_device *adev = dm->adev;
+ unsigned long flags;
if (!new_stream)
return;
if (!new_stream->timing.h_total || !new_stream->timing.v_total)
return;
- if (new_crtc_state->vrr_supported &&
- config.min_refresh_in_uhz &&
- config.max_refresh_in_uhz) {
- config.state = new_crtc_state->base.vrr_enabled ?
- VRR_STATE_ACTIVE_VARIABLE :
- VRR_STATE_INACTIVE;
- } else {
- config.state = VRR_STATE_UNSUPPORTED;
- }
-
- mod_freesync_build_vrr_params(dm->freesync_module,
- new_stream,
- &config, &vrr_params);
+ spin_lock_irqsave(&adev->ddev->event_lock, flags);
+ vrr_params = new_crtc_state->vrr_params;
if (surface) {
mod_freesync_handle_preflip(
new_stream,
flip_timestamp_in_us,
&vrr_params);
+
+ if (adev->family < AMDGPU_FAMILY_AI &&
+ amdgpu_dm_vrr_active(new_crtc_state)) {
+ mod_freesync_handle_v_update(dm->freesync_module,
+ new_stream, &vrr_params);
+ }
}
mod_freesync_build_vrr_infopacket(
new_crtc_state->base.crtc->base.id,
(int)new_crtc_state->base.vrr_enabled,
(int)vrr_params.state);
+
+ spin_unlock_irqrestore(&adev->ddev->event_lock, flags);
+}
+
+static void pre_update_freesync_state_on_stream(
+ struct amdgpu_display_manager *dm,
+ struct dm_crtc_state *new_crtc_state)
+{
+ struct dc_stream_state *new_stream = new_crtc_state->stream;
+ struct mod_vrr_params vrr_params;
+ struct mod_freesync_config config = new_crtc_state->freesync_config;
+ struct amdgpu_device *adev = dm->adev;
+ unsigned long flags;
+
+ if (!new_stream)
+ return;
+
+ /*
+ * TODO: Determine why min/max totals and vrefresh can be 0 here.
+ * For now it's sufficient to just guard against these conditions.
+ */
+ if (!new_stream->timing.h_total || !new_stream->timing.v_total)
+ return;
+
+ spin_lock_irqsave(&adev->ddev->event_lock, flags);
+ vrr_params = new_crtc_state->vrr_params;
+
+ if (new_crtc_state->vrr_supported &&
+ config.min_refresh_in_uhz &&
+ config.max_refresh_in_uhz) {
+ config.state = new_crtc_state->base.vrr_enabled ?
+ VRR_STATE_ACTIVE_VARIABLE :
+ VRR_STATE_INACTIVE;
+ } else {
+ config.state = VRR_STATE_UNSUPPORTED;
+ }
+
+ mod_freesync_build_vrr_params(dm->freesync_module,
+ new_stream,
+ &config, &vrr_params);
+
+ new_crtc_state->freesync_timing_changed |=
+ (memcmp(&new_crtc_state->vrr_params.adjust,
+ &vrr_params.adjust,
+ sizeof(vrr_params.adjust)) != 0);
+
+ new_crtc_state->vrr_params = vrr_params;
+ spin_unlock_irqrestore(&adev->ddev->event_lock, flags);
+}
+
+static void amdgpu_dm_handle_vrr_transition(struct dm_crtc_state *old_state,
+ struct dm_crtc_state *new_state)
+{
+ bool old_vrr_active = amdgpu_dm_vrr_active(old_state);
+ bool new_vrr_active = amdgpu_dm_vrr_active(new_state);
+
+ if (!old_vrr_active && new_vrr_active) {
+ /* Transition VRR inactive -> active:
+ * While VRR is active, we must not disable vblank irq, as a
+ * reenable after disable would compute bogus vblank/pflip
+ * timestamps if it likely happened inside display front-porch.
+ *
+ * We also need vupdate irq for the actual core vblank handling
+ * at end of vblank.
+ */
+ dm_set_vupdate_irq(new_state->base.crtc, true);
+ drm_crtc_vblank_get(new_state->base.crtc);
+ DRM_DEBUG_DRIVER("%s: crtc=%u VRR off->on: Get vblank ref\n",
+ __func__, new_state->base.crtc->base.id);
+ } else if (old_vrr_active && !new_vrr_active) {
+ /* Transition VRR active -> inactive:
+ * Allow vblank irq disable again for fixed refresh rate.
+ */
+ dm_set_vupdate_irq(new_state->base.crtc, false);
+ drm_crtc_vblank_put(new_state->base.crtc);
+ DRM_DEBUG_DRIVER("%s: crtc=%u VRR on->off: Drop vblank ref\n",
+ __func__, new_state->base.crtc->base.id);
+ }
+}
+
+static void amdgpu_dm_commit_cursors(struct drm_atomic_state *state)
+{
+ struct drm_plane *plane;
+ struct drm_plane_state *old_plane_state, *new_plane_state;
+ int i;
+
+ /*
+ * TODO: Make this per-stream so we don't issue redundant updates for
+ * commits with multiple streams.
+ */
+ for_each_oldnew_plane_in_state(state, plane, old_plane_state,
+ new_plane_state, i)
+ if (plane->type == DRM_PLANE_TYPE_CURSOR)
+ handle_cursor_update(plane, old_plane_state);
}
static void amdgpu_dm_commit_planes(struct drm_atomic_state *state,
struct drm_device *dev,
struct amdgpu_display_manager *dm,
struct drm_crtc *pcrtc,
- bool *wait_for_vblank)
+ bool wait_for_vblank)
{
uint32_t i, r;
uint64_t timestamp_ns;
struct dm_crtc_state *acrtc_state = to_dm_crtc_state(new_pcrtc_state);
struct dm_crtc_state *dm_old_crtc_state =
to_dm_crtc_state(drm_atomic_get_old_crtc_state(state, pcrtc));
- int flip_count = 0, planes_count = 0, vpos, hpos;
+ int planes_count = 0, vpos, hpos;
unsigned long flags;
struct amdgpu_bo *abo;
- uint64_t tiling_flags, dcc_address;
- uint32_t target, target_vblank;
- uint64_t last_flip_vblank;
- bool vrr_active = acrtc_state->freesync_config.state == VRR_STATE_ACTIVE_VARIABLE;
-
- struct {
- struct dc_surface_update surface_updates[MAX_SURFACES];
- struct dc_flip_addrs flip_addrs[MAX_SURFACES];
- struct dc_stream_update stream_update;
- } *flip;
-
+ uint64_t tiling_flags;
+ uint32_t target_vblank, last_flip_vblank;
+ bool vrr_active = amdgpu_dm_vrr_active(acrtc_state);
+ bool pflip_present = false;
struct {
struct dc_surface_update surface_updates[MAX_SURFACES];
struct dc_plane_info plane_infos[MAX_SURFACES];
struct dc_scaling_info scaling_infos[MAX_SURFACES];
+ struct dc_flip_addrs flip_addrs[MAX_SURFACES];
struct dc_stream_update stream_update;
- } *full;
+ } *bundle;
- flip = kzalloc(sizeof(*flip), GFP_KERNEL);
- full = kzalloc(sizeof(*full), GFP_KERNEL);
+ bundle = kzalloc(sizeof(*bundle), GFP_KERNEL);
- if (!flip || !full) {
- dm_error("Failed to allocate update bundles\n");
+ if (!bundle) {
+ dm_error("Failed to allocate update bundle\n");
goto cleanup;
}
+ /*
+ * Disable the cursor first if we're disabling all the planes.
+ * It'll remain on the screen after the planes are re-enabled
+ * if we don't.
+ */
+ if (acrtc_state->active_planes == 0)
+ amdgpu_dm_commit_cursors(state);
+
/* update planes when needed */
for_each_oldnew_plane_in_state(state, plane, old_plane_state, new_plane_state, i) {
struct drm_crtc *crtc = new_plane_state->crtc;
struct drm_crtc_state *new_crtc_state;
struct drm_framebuffer *fb = new_plane_state->fb;
- struct amdgpu_framebuffer *afb = to_amdgpu_framebuffer(fb);
- bool pflip_needed;
+ bool plane_needs_flip;
struct dc_plane_state *dc_plane;
struct dm_plane_state *dm_new_plane_state = to_dm_plane_state(new_plane_state);
if (!new_crtc_state->active)
continue;
- pflip_needed = old_plane_state->fb &&
- old_plane_state->fb != new_plane_state->fb;
-
dc_plane = dm_new_plane_state->dc_state;
- if (pflip_needed) {
- /*
- * Assume even ONE crtc with immediate flip means
- * entire can't wait for VBLANK
- * TODO Check if it's correct
- */
- if (new_pcrtc_state->pageflip_flags & DRM_MODE_PAGE_FLIP_ASYNC)
- *wait_for_vblank = false;
+ bundle->surface_updates[planes_count].surface = dc_plane;
+ if (new_pcrtc_state->color_mgmt_changed) {
+ bundle->surface_updates[planes_count].gamma = dc_plane->gamma_correction;
+ bundle->surface_updates[planes_count].in_transfer_func = dc_plane->in_transfer_func;
+ }
- /*
- * TODO This might fail and hence better not used, wait
- * explicitly on fences instead
- * and in general should be called for
- * blocking commit to as per framework helpers
- */
- abo = gem_to_amdgpu_bo(fb->obj[0]);
- r = amdgpu_bo_reserve(abo, true);
- if (unlikely(r != 0))
- DRM_ERROR("failed to reserve buffer before flip\n");
+ fill_dc_scaling_info(new_plane_state,
+ &bundle->scaling_infos[planes_count]);
- /*
- * Wait for all fences on this FB. Do limited wait to avoid
- * deadlock during GPU reset when this fence will not signal
- * but we hold reservation lock for the BO.
- */
- r = reservation_object_wait_timeout_rcu(abo->tbo.resv,
- true, false,
- msecs_to_jiffies(5000));
- if (unlikely(r == 0))
- DRM_ERROR("Waiting for fences timed out.");
+ bundle->surface_updates[planes_count].scaling_info =
+ &bundle->scaling_infos[planes_count];
+ plane_needs_flip = old_plane_state->fb && new_plane_state->fb;
+ pflip_present = pflip_present || plane_needs_flip;
- amdgpu_bo_get_tiling_flags(abo, &tiling_flags);
+ if (!plane_needs_flip) {
+ planes_count += 1;
+ continue;
+ }
- amdgpu_bo_unreserve(abo);
+ abo = gem_to_amdgpu_bo(fb->obj[0]);
- flip->flip_addrs[flip_count].address.grph.addr.low_part = lower_32_bits(afb->address);
- flip->flip_addrs[flip_count].address.grph.addr.high_part = upper_32_bits(afb->address);
+ /*
+ * Wait for all fences on this FB. Do limited wait to avoid
+ * deadlock during GPU reset when this fence will not signal
+ * but we hold reservation lock for the BO.
+ */
+ r = reservation_object_wait_timeout_rcu(abo->tbo.resv, true,
+ false,
+ msecs_to_jiffies(5000));
+ if (unlikely(r <= 0))
+ DRM_ERROR("Waiting for fences timed out or interrupted!");
- dcc_address = get_dcc_address(afb->address, tiling_flags);
- flip->flip_addrs[flip_count].address.grph.meta_addr.low_part = lower_32_bits(dcc_address);
- flip->flip_addrs[flip_count].address.grph.meta_addr.high_part = upper_32_bits(dcc_address);
+ /*
+ * TODO This might fail and hence better not used, wait
+ * explicitly on fences instead
+ * and in general should be called for
+ * blocking commit to as per framework helpers
+ */
+ r = amdgpu_bo_reserve(abo, true);
+ if (unlikely(r != 0))
+ DRM_ERROR("failed to reserve buffer before flip\n");
- flip->flip_addrs[flip_count].flip_immediate =
- (crtc->state->pageflip_flags & DRM_MODE_PAGE_FLIP_ASYNC) != 0;
+ amdgpu_bo_get_tiling_flags(abo, &tiling_flags);
- timestamp_ns = ktime_get_ns();
- flip->flip_addrs[flip_count].flip_timestamp_in_us = div_u64(timestamp_ns, 1000);
- flip->surface_updates[flip_count].flip_addr = &flip->flip_addrs[flip_count];
- flip->surface_updates[flip_count].surface = dc_plane;
+ amdgpu_bo_unreserve(abo);
- if (!flip->surface_updates[flip_count].surface) {
- DRM_ERROR("No surface for CRTC: id=%d\n",
- acrtc_attach->crtc_id);
- continue;
- }
+ fill_dc_plane_info_and_addr(
+ dm->adev, new_plane_state, tiling_flags,
+ &bundle->plane_infos[planes_count],
+ &bundle->flip_addrs[planes_count].address);
- if (plane == pcrtc->primary)
- update_freesync_state_on_stream(
- dm,
- acrtc_state,
- acrtc_state->stream,
- dc_plane,
- flip->flip_addrs[flip_count].flip_timestamp_in_us);
+ bundle->surface_updates[planes_count].plane_info =
+ &bundle->plane_infos[planes_count];
- DRM_DEBUG_DRIVER("%s Flipping to hi: 0x%x, low: 0x%x\n",
- __func__,
- flip->flip_addrs[flip_count].address.grph.addr.high_part,
- flip->flip_addrs[flip_count].address.grph.addr.low_part);
+ bundle->flip_addrs[planes_count].flip_immediate =
+ (crtc->state->pageflip_flags & DRM_MODE_PAGE_FLIP_ASYNC) != 0;
- flip_count += 1;
- }
+ timestamp_ns = ktime_get_ns();
+ bundle->flip_addrs[planes_count].flip_timestamp_in_us = div_u64(timestamp_ns, 1000);
+ bundle->surface_updates[planes_count].flip_addr = &bundle->flip_addrs[planes_count];
+ bundle->surface_updates[planes_count].surface = dc_plane;
- full->surface_updates[planes_count].surface = dc_plane;
- if (new_pcrtc_state->color_mgmt_changed) {
- full->surface_updates[planes_count].gamma = dc_plane->gamma_correction;
- full->surface_updates[planes_count].in_transfer_func = dc_plane->in_transfer_func;
+ if (!bundle->surface_updates[planes_count].surface) {
+ DRM_ERROR("No surface for CRTC: id=%d\n",
+ acrtc_attach->crtc_id);
+ continue;
}
+ if (plane == pcrtc->primary)
+ update_freesync_state_on_stream(
+ dm,
+ acrtc_state,
+ acrtc_state->stream,
+ dc_plane,
+ bundle->flip_addrs[planes_count].flip_timestamp_in_us);
- full->scaling_infos[planes_count].scaling_quality = dc_plane->scaling_quality;
- full->scaling_infos[planes_count].src_rect = dc_plane->src_rect;
- full->scaling_infos[planes_count].dst_rect = dc_plane->dst_rect;
- full->scaling_infos[planes_count].clip_rect = dc_plane->clip_rect;
- full->surface_updates[planes_count].scaling_info = &full->scaling_infos[planes_count];
-
-
- full->plane_infos[planes_count].color_space = dc_plane->color_space;
- full->plane_infos[planes_count].format = dc_plane->format;
- full->plane_infos[planes_count].plane_size = dc_plane->plane_size;
- full->plane_infos[planes_count].rotation = dc_plane->rotation;
- full->plane_infos[planes_count].horizontal_mirror = dc_plane->horizontal_mirror;
- full->plane_infos[planes_count].stereo_format = dc_plane->stereo_format;
- full->plane_infos[planes_count].tiling_info = dc_plane->tiling_info;
- full->plane_infos[planes_count].visible = dc_plane->visible;
- full->plane_infos[planes_count].per_pixel_alpha = dc_plane->per_pixel_alpha;
- full->plane_infos[planes_count].dcc = dc_plane->dcc;
- full->surface_updates[planes_count].plane_info = &full->plane_infos[planes_count];
+ DRM_DEBUG_DRIVER("%s Flipping to hi: 0x%x, low: 0x%x\n",
+ __func__,
+ bundle->flip_addrs[planes_count].address.grph.addr.high_part,
+ bundle->flip_addrs[planes_count].address.grph.addr.low_part);
planes_count += 1;
}
- /*
- * TODO: For proper atomic behaviour, we should be calling into DC once with
- * all the changes. However, DC refuses to do pageflips and non-pageflip
- * changes in the same call. Change DC to respect atomic behaviour,
- * hopefully eliminating dc_*_update structs in their entirety.
- */
- if (flip_count) {
+ if (pflip_present) {
if (!vrr_active) {
/* Use old throttling in non-vrr fixed refresh rate mode
* to keep flip scheduling based on target vblank counts
* clients using the GLX_OML_sync_control extension or
* DRI3/Present extension with defined target_msc.
*/
- last_flip_vblank = drm_crtc_vblank_count(pcrtc);
+ last_flip_vblank = amdgpu_get_vblank_counter_kms(dm->ddev, acrtc_attach->crtc_id);
}
else {
/* For variable refresh rate mode only:
spin_unlock_irqrestore(&pcrtc->dev->event_lock, flags);
}
- target = (uint32_t)last_flip_vblank + *wait_for_vblank;
-
- /* Prepare wait for target vblank early - before the fence-waits */
- target_vblank = target - (uint32_t)drm_crtc_vblank_count(pcrtc) +
- amdgpu_get_vblank_counter_kms(pcrtc->dev, acrtc_attach->crtc_id);
+ target_vblank = last_flip_vblank + wait_for_vblank;
/*
* Wait until we're out of the vertical blank period before the one
if (acrtc_state->stream) {
if (acrtc_state->freesync_timing_changed)
- flip->stream_update.adjust =
+ bundle->stream_update.adjust =
&acrtc_state->stream->adjust;
if (acrtc_state->freesync_vrr_info_changed)
- flip->stream_update.vrr_infopacket =
+ bundle->stream_update.vrr_infopacket =
&acrtc_state->stream->vrr_infopacket;
}
-
- mutex_lock(&dm->dc_lock);
- dc_commit_updates_for_stream(dm->dc,
- flip->surface_updates,
- flip_count,
- acrtc_state->stream,
- &flip->stream_update,
- dc_state);
- mutex_unlock(&dm->dc_lock);
}
- if (planes_count) {
+ /* Update the planes if changed or disable if we don't have any. */
+ if (planes_count || acrtc_state->active_planes == 0) {
if (new_pcrtc_state->mode_changed) {
- full->stream_update.src = acrtc_state->stream->src;
- full->stream_update.dst = acrtc_state->stream->dst;
+ bundle->stream_update.src = acrtc_state->stream->src;
+ bundle->stream_update.dst = acrtc_state->stream->dst;
}
if (new_pcrtc_state->color_mgmt_changed)
- full->stream_update.out_transfer_func = acrtc_state->stream->out_transfer_func;
+ bundle->stream_update.out_transfer_func = acrtc_state->stream->out_transfer_func;
acrtc_state->stream->abm_level = acrtc_state->abm_level;
if (acrtc_state->abm_level != dm_old_crtc_state->abm_level)
- full->stream_update.abm_level = &acrtc_state->abm_level;
+ bundle->stream_update.abm_level = &acrtc_state->abm_level;
mutex_lock(&dm->dc_lock);
dc_commit_updates_for_stream(dm->dc,
- full->surface_updates,
+ bundle->surface_updates,
planes_count,
acrtc_state->stream,
- &full->stream_update,
+ &bundle->stream_update,
dc_state);
mutex_unlock(&dm->dc_lock);
}
- for_each_oldnew_plane_in_state(state, plane, old_plane_state, new_plane_state, i)
- if (plane->type == DRM_PLANE_TYPE_CURSOR)
- handle_cursor_update(plane, old_plane_state);
+ /*
+ * Update cursor state *after* programming all the planes.
+ * This avoids redundant programming in the case where we're going
+ * to be disabling a single plane - those pipes are being disabled.
+ */
+ if (acrtc_state->active_planes)
+ amdgpu_dm_commit_cursors(state);
cleanup:
- kfree(flip);
- kfree(full);
+ kfree(bundle);
+}
+
+/*
+ * Enable interrupts on CRTCs that are newly active, undergone
+ * a modeset, or have active planes again.
+ *
+ * Done in two passes, based on the for_modeset flag:
+ * Pass 1: For CRTCs going through modeset
+ * Pass 2: For CRTCs going from 0 to n active planes
+ *
+ * Interrupts can only be enabled after the planes are programmed,
+ * so this requires a two-pass approach since we don't want to
+ * just defer the interrupts until after commit planes every time.
+ */
+static void amdgpu_dm_enable_crtc_interrupts(struct drm_device *dev,
+ struct drm_atomic_state *state,
+ bool for_modeset)
+{
+ struct amdgpu_device *adev = dev->dev_private;
+ struct drm_crtc *crtc;
+ struct drm_crtc_state *old_crtc_state, *new_crtc_state;
+ int i;
+
+ for_each_oldnew_crtc_in_state(state, crtc, old_crtc_state,
+ new_crtc_state, i) {
+ struct amdgpu_crtc *acrtc = to_amdgpu_crtc(crtc);
+ struct dm_crtc_state *dm_new_crtc_state =
+ to_dm_crtc_state(new_crtc_state);
+ struct dm_crtc_state *dm_old_crtc_state =
+ to_dm_crtc_state(old_crtc_state);
+ bool modeset = drm_atomic_crtc_needs_modeset(new_crtc_state);
+ bool run_pass;
+
+ run_pass = (for_modeset && modeset) ||
+ (!for_modeset && !modeset &&
+ !dm_old_crtc_state->interrupts_enabled);
+
+ if (!run_pass)
+ continue;
+
+ if (!dm_new_crtc_state->interrupts_enabled)
+ continue;
+
+ manage_dm_interrupts(adev, acrtc, true);
+
+#ifdef CONFIG_DEBUG_FS
+ /* The stream has changed so CRC capture needs to re-enabled. */
+ if (dm_new_crtc_state->crc_enabled) {
+ dm_new_crtc_state->crc_enabled = false;
+ amdgpu_dm_crtc_set_crc_source(crtc, "auto");
+ }
+#endif
+ }
}
/*
static void amdgpu_dm_crtc_copy_transient_flags(struct drm_crtc_state *crtc_state,
struct dc_stream_state *stream_state)
{
- stream_state->mode_changed =
- crtc_state->mode_changed || crtc_state->active_changed;
+ stream_state->mode_changed = drm_atomic_crtc_needs_modeset(crtc_state);
}
static int amdgpu_dm_atomic_commit(struct drm_device *dev,
int i;
/*
- * We evade vblanks and pflips on crtc that
- * should be changed. We do it here to flush & disable
- * interrupts before drm_swap_state is called in drm_atomic_helper_commit
- * it will update crtc->dm_crtc_state->stream pointer which is used in
- * the ISRs.
+ * We evade vblank and pflip interrupts on CRTCs that are undergoing
+ * a modeset, being disabled, or have no active planes.
+ *
+ * It's done in atomic commit rather than commit tail for now since
+ * some of these interrupt handlers access the current CRTC state and
+ * potentially the stream pointer itself.
+ *
+ * Since the atomic state is swapped within atomic commit and not within
+ * commit tail this would leave to new state (that hasn't been committed yet)
+ * being accesssed from within the handlers.
+ *
+ * TODO: Fix this so we can do this in commit tail and not have to block
+ * in atomic check.
*/
for_each_oldnew_crtc_in_state(state, crtc, old_crtc_state, new_crtc_state, i) {
struct dm_crtc_state *dm_old_crtc_state = to_dm_crtc_state(old_crtc_state);
struct dm_crtc_state *dm_new_crtc_state = to_dm_crtc_state(new_crtc_state);
struct amdgpu_crtc *acrtc = to_amdgpu_crtc(crtc);
- if (drm_atomic_crtc_needs_modeset(new_crtc_state)
- && dm_old_crtc_state->stream) {
+ if (dm_old_crtc_state->interrupts_enabled &&
+ (!dm_new_crtc_state->interrupts_enabled ||
+ drm_atomic_crtc_needs_modeset(new_crtc_state))) {
/*
- * If the stream is removed and CRC capture was
- * enabled on the CRTC the extra vblank reference
- * needs to be dropped since CRC capture will be
- * disabled.
+ * Drop the extra vblank reference added by CRC
+ * capture if applicable.
*/
- if (!dm_new_crtc_state->stream
- && dm_new_crtc_state->crc_enabled) {
+ if (dm_new_crtc_state->crc_enabled)
drm_crtc_vblank_put(crtc);
+
+ /*
+ * Only keep CRC capture enabled if there's
+ * still a stream for the CRTC.
+ */
+ if (!dm_new_crtc_state->stream)
dm_new_crtc_state->crc_enabled = false;
- }
manage_dm_interrupts(adev, acrtc, false);
}
dc_state = dm_state->context;
} else {
/* No state changes, retain current state. */
- dc_state_temp = dc_create_state();
+ dc_state_temp = dc_create_state(dm->dc);
ASSERT(dc_state_temp);
dc_state = dc_state_temp;
dc_resource_state_copy_construct_current(dm->dc, dc_state);
mutex_unlock(&dm->dc_lock);
}
+ /* Count number of newly disabled CRTCs for dropping PM refs later. */
for_each_oldnew_crtc_in_state(state, crtc, old_crtc_state,
- new_crtc_state, i) {
- /*
- * loop to enable interrupts on newly arrived crtc
- */
- struct amdgpu_crtc *acrtc = to_amdgpu_crtc(crtc);
- bool modeset_needed;
-
+ new_crtc_state, i) {
if (old_crtc_state->active && !new_crtc_state->active)
crtc_disable_count++;
dm_new_crtc_state = to_dm_crtc_state(new_crtc_state);
dm_old_crtc_state = to_dm_crtc_state(old_crtc_state);
- modeset_needed = modeset_required(
- new_crtc_state,
- dm_new_crtc_state->stream,
- dm_old_crtc_state->stream);
- if (dm_new_crtc_state->stream == NULL || !modeset_needed)
- continue;
-
- manage_dm_interrupts(adev, acrtc, true);
+ /* Update freesync active state. */
+ pre_update_freesync_state_on_stream(dm, dm_new_crtc_state);
-#ifdef CONFIG_DEBUG_FS
- /* The stream has changed so CRC capture needs to re-enabled. */
- if (dm_new_crtc_state->crc_enabled)
- amdgpu_dm_crtc_set_crc_source(crtc, "auto");
-#endif
+ /* Handle vrr on->off / off->on transitions */
+ amdgpu_dm_handle_vrr_transition(dm_old_crtc_state,
+ dm_new_crtc_state);
}
+ /* Enable interrupts for CRTCs going through a modeset. */
+ amdgpu_dm_enable_crtc_interrupts(dev, state, true);
+
+ for_each_new_crtc_in_state(state, crtc, new_crtc_state, j)
+ if (new_crtc_state->pageflip_flags & DRM_MODE_PAGE_FLIP_ASYNC)
+ wait_for_vblank = false;
+
/* update planes when needed per crtc*/
for_each_new_crtc_in_state(state, crtc, new_crtc_state, j) {
dm_new_crtc_state = to_dm_crtc_state(new_crtc_state);
if (dm_new_crtc_state->stream)
amdgpu_dm_commit_planes(state, dc_state, dev,
- dm, crtc, &wait_for_vblank);
+ dm, crtc, wait_for_vblank);
}
+ /* Enable interrupts for CRTCs going from 0 to n active planes. */
+ amdgpu_dm_enable_crtc_interrupts(dev, state, false);
/*
* send vblank event on all events not handled in flip and
struct amdgpu_dm_connector *aconnector = NULL;
struct drm_connector_state *drm_new_conn_state = NULL, *drm_old_conn_state = NULL;
struct dm_connector_state *dm_new_conn_state = NULL, *dm_old_conn_state = NULL;
- struct drm_plane_state *new_plane_state = NULL;
new_stream = NULL;
dm_old_crtc_state = to_dm_crtc_state(old_crtc_state);
dm_new_crtc_state = to_dm_crtc_state(new_crtc_state);
acrtc = to_amdgpu_crtc(crtc);
-
- new_plane_state = drm_atomic_get_new_plane_state(state, new_crtc_state->crtc->primary);
-
- if (new_crtc_state->enable && new_plane_state && !new_plane_state->fb) {
- ret = -EINVAL;
- goto fail;
- }
-
aconnector = amdgpu_dm_find_first_crtc_matching_connector(state, crtc);
/* TODO This hack should go away */
update_stream_scaling_settings(
&new_crtc_state->mode, dm_new_conn_state, dm_new_crtc_state->stream);
+ /* ABM settings */
+ dm_new_crtc_state->abm_level = dm_new_conn_state->abm_level;
+
/*
* Color management settings. We also update color properties
* when a modeset is needed, to ensure it gets reprogrammed.
return ret;
}
+static bool should_reset_plane(struct drm_atomic_state *state,
+ struct drm_plane *plane,
+ struct drm_plane_state *old_plane_state,
+ struct drm_plane_state *new_plane_state)
+{
+ struct drm_plane *other;
+ struct drm_plane_state *old_other_state, *new_other_state;
+ struct drm_crtc_state *new_crtc_state;
+ int i;
+
+ /*
+ * TODO: Remove this hack once the checks below are sufficient
+ * enough to determine when we need to reset all the planes on
+ * the stream.
+ */
+ if (state->allow_modeset)
+ return true;
+
+ /* Exit early if we know that we're adding or removing the plane. */
+ if (old_plane_state->crtc != new_plane_state->crtc)
+ return true;
+
+ /* old crtc == new_crtc == NULL, plane not in context. */
+ if (!new_plane_state->crtc)
+ return false;
+
+ new_crtc_state =
+ drm_atomic_get_new_crtc_state(state, new_plane_state->crtc);
+
+ if (!new_crtc_state)
+ return true;
+
+ if (drm_atomic_crtc_needs_modeset(new_crtc_state))
+ return true;
+
+ /*
+ * If there are any new primary or overlay planes being added or
+ * removed then the z-order can potentially change. To ensure
+ * correct z-order and pipe acquisition the current DC architecture
+ * requires us to remove and recreate all existing planes.
+ *
+ * TODO: Come up with a more elegant solution for this.
+ */
+ for_each_oldnew_plane_in_state(state, other, old_other_state, new_other_state, i) {
+ if (other->type == DRM_PLANE_TYPE_CURSOR)
+ continue;
+
+ if (old_other_state->crtc != new_plane_state->crtc &&
+ new_other_state->crtc != new_plane_state->crtc)
+ continue;
+
+ if (old_other_state->crtc != new_other_state->crtc)
+ return true;
+
+ /* TODO: Remove this once we can handle fast format changes. */
+ if (old_other_state->fb && new_other_state->fb &&
+ old_other_state->fb->format != new_other_state->fb->format)
+ return true;
+ }
+
+ return false;
+}
+
static int dm_update_plane_state(struct dc *dc,
struct drm_atomic_state *state,
struct drm_plane *plane,
struct drm_crtc_state *old_crtc_state, *new_crtc_state;
struct dm_crtc_state *dm_new_crtc_state, *dm_old_crtc_state;
struct dm_plane_state *dm_new_plane_state, *dm_old_plane_state;
- /* TODO return page_flip_needed() function */
- bool pflip_needed = !state->allow_modeset;
+ bool needs_reset;
int ret = 0;
if (plane->type == DRM_PLANE_TYPE_CURSOR)
return 0;
+ needs_reset = should_reset_plane(state, plane, old_plane_state,
+ new_plane_state);
+
/* Remove any changed/removed planes */
if (!enable) {
- if (pflip_needed &&
- plane->type != DRM_PLANE_TYPE_OVERLAY)
+ if (!needs_reset)
return 0;
if (!old_plane_crtc)
if (!dm_new_crtc_state->stream)
return 0;
- if (pflip_needed && plane->type != DRM_PLANE_TYPE_OVERLAY)
+ if (!needs_reset)
return 0;
WARN_ON(dm_new_plane_state->dc_state);
DRM_DEBUG_DRIVER("Enabling DRM plane: %d on DRM crtc %d\n",
plane->base.id, new_plane_crtc->base.id);
- ret = fill_plane_attributes(
+ ret = fill_dc_plane_attributes(
new_plane_crtc->dev->dev_private,
dc_new_plane_state,
new_plane_state,
}
static int
-dm_determine_update_type_for_commit(struct dc *dc,
+dm_determine_update_type_for_commit(struct amdgpu_display_manager *dm,
struct drm_atomic_state *state,
enum surface_update_type *out_type)
{
+ struct dc *dc = dm->dc;
struct dm_atomic_state *dm_state = NULL, *old_dm_state = NULL;
int i, j, num_plane, ret = 0;
struct drm_plane_state *old_plane_state, *new_plane_state;
struct dc_stream_status *status = NULL;
struct dc_surface_update *updates;
- struct dc_plane_state *surface;
enum surface_update_type update_type = UPDATE_TYPE_FAST;
updates = kcalloc(MAX_SURFACES, sizeof(*updates), GFP_KERNEL);
- surface = kcalloc(MAX_SURFACES, sizeof(*surface), GFP_KERNEL);
- if (!updates || !surface) {
- DRM_ERROR("Plane or surface update failed to allocate");
+ if (!updates) {
+ DRM_ERROR("Failed to allocate plane updates\n");
/* Set type to FULL to avoid crashing in DC*/
update_type = UPDATE_TYPE_FULL;
goto cleanup;
}
for_each_oldnew_crtc_in_state(state, crtc, old_crtc_state, new_crtc_state, i) {
- struct dc_stream_update stream_update = { 0 };
+ struct dc_scaling_info scaling_info;
+ struct dc_stream_update stream_update;
+
+ memset(&stream_update, 0, sizeof(stream_update));
new_dm_crtc_state = to_dm_crtc_state(new_crtc_state);
old_dm_crtc_state = to_dm_crtc_state(old_crtc_state);
goto cleanup;
}
- if (!state->allow_modeset)
- continue;
-
if (crtc != new_plane_crtc)
continue;
- updates[num_plane].surface = &surface[num_plane];
+ updates[num_plane].surface = new_dm_plane_state->dc_state;
if (new_crtc_state->mode_changed) {
- updates[num_plane].surface->src_rect =
- new_dm_plane_state->dc_state->src_rect;
- updates[num_plane].surface->dst_rect =
- new_dm_plane_state->dc_state->dst_rect;
- updates[num_plane].surface->rotation =
- new_dm_plane_state->dc_state->rotation;
- updates[num_plane].surface->in_transfer_func =
- new_dm_plane_state->dc_state->in_transfer_func;
stream_update.dst = new_dm_crtc_state->stream->dst;
stream_update.src = new_dm_crtc_state->stream->src;
}
new_dm_crtc_state->stream->out_transfer_func;
}
+ ret = fill_dc_scaling_info(new_plane_state,
+ &scaling_info);
+ if (ret)
+ goto cleanup;
+
+ updates[num_plane].scaling_info = &scaling_info;
+
num_plane++;
}
status = dc_stream_get_status_from_state(old_dm_state->context,
new_dm_crtc_state->stream);
+ /*
+ * TODO: DC modifies the surface during this call so we need
+ * to lock here - find a way to do this without locking.
+ */
+ mutex_lock(&dm->dc_lock);
update_type = dc_check_update_surfaces_for_stream(dc, updates, num_plane,
&stream_update, status);
+ mutex_unlock(&dm->dc_lock);
if (update_type > UPDATE_TYPE_MED) {
update_type = UPDATE_TYPE_FULL;
cleanup:
kfree(updates);
- kfree(surface);
*out_type = update_type;
return ret;
lock_and_validation_needed = true;
}
- ret = dm_determine_update_type_for_commit(dc, state, &update_type);
+ ret = dm_determine_update_type_for_commit(&adev->dm, state, &update_type);
if (ret)
goto fail;
*/
if (lock_and_validation_needed && overall_update_type <= UPDATE_TYPE_FAST)
WARN(1, "Global lock should be Set, overall_update_type should be UPDATE_TYPE_MED or UPDATE_TYPE_FULL");
- else if (!lock_and_validation_needed && overall_update_type > UPDATE_TYPE_FAST)
- WARN(1, "Global lock should NOT be set, overall_update_type should be UPDATE_TYPE_FAST");
-
if (overall_update_type > UPDATE_TYPE_FAST) {
ret = dm_atomic_get_state(state, &dm_state);
if (ret)
goto fail;
- if (dc_validate_global_state(dc, dm_state->context) != DC_OK) {
+ if (dc_validate_global_state(dc, dm_state->context, false) != DC_OK) {
ret = -EINVAL;
goto fail;
}
*/
struct drm_private_obj atomic_obj;
- struct drm_modeset_lock atomic_obj_lock;
-
/**
* @dc_lock:
*
struct common_irq_params
vblank_params[DC_IRQ_SOURCE_VBLANK6 - DC_IRQ_SOURCE_VBLANK1 + 1];
+ /**
+ * @vupdate_params:
+ *
+ * Vertical update IRQ parameters, passed to registered handlers when
+ * triggered.
+ */
+ struct common_irq_params
+ vupdate_params[DC_IRQ_SOURCE_VUPDATE6 - DC_IRQ_SOURCE_VUPDATE1 + 1];
+
spinlock_t irq_handler_list_table_lock;
struct backlight_device *backlight_dev;
struct mutex hpd_lock;
bool fake_enable;
+#ifdef CONFIG_DEBUG_FS
+ uint32_t debugfs_dpcd_address;
+ uint32_t debugfs_dpcd_size;
+#endif
};
#define to_amdgpu_dm_connector(x) container_of(x, struct amdgpu_dm_connector, base)
struct drm_crtc_state base;
struct dc_stream_state *stream;
+ int active_planes;
+ bool interrupts_enabled;
+
int crc_skip_count;
bool crc_enabled;
crtc->base.state->dev->dev_private;
struct drm_color_lut *lut;
uint32_t lut_size;
- struct dc_gamma *gamma;
+ struct dc_gamma *gamma = NULL;
enum dc_transfer_func_type old_type = stream->out_transfer_func->type;
bool ret;
- if (!blob) {
+ if (!blob && adev->asic_type <= CHIP_RAVEN) {
/* By default, use the SRGB predefined curve.*/
stream->out_transfer_func->type = TF_TYPE_PREDEFINED;
stream->out_transfer_func->tf = TRANSFER_FUNCTION_SRGB;
return 0;
}
- lut = (struct drm_color_lut *)blob->data;
- lut_size = blob->length / sizeof(struct drm_color_lut);
-
- gamma = dc_create_gamma();
- if (!gamma)
- return -ENOMEM;
+ if (blob) {
+ lut = (struct drm_color_lut *)blob->data;
+ lut_size = blob->length / sizeof(struct drm_color_lut);
+
+ gamma = dc_create_gamma();
+ if (!gamma)
+ return -ENOMEM;
+
+ gamma->num_entries = lut_size;
+ if (gamma->num_entries == MAX_COLOR_LEGACY_LUT_ENTRIES)
+ gamma->type = GAMMA_RGB_256;
+ else if (gamma->num_entries == MAX_COLOR_LUT_ENTRIES)
+ gamma->type = GAMMA_CS_TFM_1D;
+ else {
+ /* Invalid lut size */
+ dc_gamma_release(&gamma);
+ return -EINVAL;
+ }
- gamma->num_entries = lut_size;
- if (gamma->num_entries == MAX_COLOR_LEGACY_LUT_ENTRIES)
- gamma->type = GAMMA_RGB_256;
- else if (gamma->num_entries == MAX_COLOR_LUT_ENTRIES)
- gamma->type = GAMMA_CS_TFM_1D;
- else {
- /* Invalid lut size */
- dc_gamma_release(&gamma);
- return -EINVAL;
+ /* Convert drm_lut into dc_gamma */
+ __drm_lut_to_dc_gamma(lut, gamma, gamma->type == GAMMA_RGB_256);
}
- /* Convert drm_lut into dc_gamma */
- __drm_lut_to_dc_gamma(lut, gamma, gamma->type == GAMMA_RGB_256);
-
- /* Call color module to translate into something DC understands. Namely
- * a transfer function.
+ /* predefined gamma ROM only exist for RAVEN and pre-RAVEN ASIC,
+ * set canRomBeUsed accordingly
*/
stream->out_transfer_func->type = TF_TYPE_DISTRIBUTED_POINTS;
ret = mod_color_calculate_regamma_params(stream->out_transfer_func,
- gamma, true, adev->asic_type <= CHIP_RAVEN, NULL);
- dc_gamma_release(&gamma);
+ gamma, true, adev->asic_type <= CHIP_RAVEN, NULL);
+
+ if (gamma)
+ dc_gamma_release(&gamma);
+
if (!ret) {
stream->out_transfer_func->type = old_type;
DRM_ERROR("Out of memory when calculating regamma params\n");
#include "amdgpu.h"
#include "amdgpu_dm.h"
#include "amdgpu_dm_debugfs.h"
+#include "dm_helpers.h"
/* function description
* get/ set DP configuration: lane_count, link_rate, spread_spectrum
return 0;
}
+
+/* function description
+ *
+ * generic SDP message access for testing
+ *
+ * debugfs sdp_message is located at /syskernel/debug/dri/0/DP-x
+ *
+ * SDP header
+ * Hb0 : Secondary-Data Packet ID
+ * Hb1 : Secondary-Data Packet type
+ * Hb2 : Secondary-Data-packet-specific header, Byte 0
+ * Hb3 : Secondary-Data-packet-specific header, Byte 1
+ *
+ * for using custom sdp message: input 4 bytes SDP header and 32 bytes raw data
+ */
+static ssize_t dp_sdp_message_debugfs_write(struct file *f, const char __user *buf,
+ size_t size, loff_t *pos)
+{
+ int r;
+ uint8_t data[36];
+ struct amdgpu_dm_connector *connector = file_inode(f)->i_private;
+ struct dm_crtc_state *acrtc_state;
+ uint32_t write_size = 36;
+
+ if (connector->base.status != connector_status_connected)
+ return -ENODEV;
+
+ if (size == 0)
+ return 0;
+
+ acrtc_state = to_dm_crtc_state(connector->base.state->crtc->state);
+
+ r = copy_from_user(data, buf, write_size);
+
+ write_size -= r;
+
+ dc_stream_send_dp_sdp(acrtc_state->stream, data, write_size);
+
+ return write_size;
+}
+
DEFINE_SHOW_ATTRIBUTE(vrr_range);
+static ssize_t dp_dpcd_address_write(struct file *f, const char __user *buf,
+ size_t size, loff_t *pos)
+{
+ int r;
+ struct amdgpu_dm_connector *connector = file_inode(f)->i_private;
+
+ if (size < sizeof(connector->debugfs_dpcd_address))
+ return 0;
+
+ r = copy_from_user(&connector->debugfs_dpcd_address,
+ buf, sizeof(connector->debugfs_dpcd_address));
+
+ return size - r;
+}
+
+static ssize_t dp_dpcd_size_write(struct file *f, const char __user *buf,
+ size_t size, loff_t *pos)
+{
+ int r;
+ struct amdgpu_dm_connector *connector = file_inode(f)->i_private;
+
+ if (size < sizeof(connector->debugfs_dpcd_size))
+ return 0;
+
+ r = copy_from_user(&connector->debugfs_dpcd_size,
+ buf, sizeof(connector->debugfs_dpcd_size));
+
+ if (connector->debugfs_dpcd_size > 256)
+ connector->debugfs_dpcd_size = 0;
+
+ return size - r;
+}
+
+static ssize_t dp_dpcd_data_write(struct file *f, const char __user *buf,
+ size_t size, loff_t *pos)
+{
+ int r;
+ char *data;
+ struct amdgpu_dm_connector *connector = file_inode(f)->i_private;
+ struct dc_link *link = connector->dc_link;
+ uint32_t write_size = connector->debugfs_dpcd_size;
+
+ if (size < write_size)
+ return 0;
+
+ data = kzalloc(write_size, GFP_KERNEL);
+ if (!data)
+ return 0;
+
+ r = copy_from_user(data, buf, write_size);
+
+ dm_helpers_dp_write_dpcd(link->ctx, link,
+ connector->debugfs_dpcd_address, data, write_size - r);
+ kfree(data);
+ return write_size - r;
+}
+
+static ssize_t dp_dpcd_data_read(struct file *f, char __user *buf,
+ size_t size, loff_t *pos)
+{
+ int r;
+ char *data;
+ struct amdgpu_dm_connector *connector = file_inode(f)->i_private;
+ struct dc_link *link = connector->dc_link;
+ uint32_t read_size = connector->debugfs_dpcd_size;
+
+ if (size < read_size)
+ return 0;
+
+ data = kzalloc(read_size, GFP_KERNEL);
+ if (!data)
+ return 0;
+
+ dm_helpers_dp_read_dpcd(link->ctx, link,
+ connector->debugfs_dpcd_address, data, read_size);
+
+ r = copy_to_user(buf, data, read_size);
+
+ kfree(data);
+ return read_size - r;
+}
+
static const struct file_operations dp_link_settings_debugfs_fops = {
.owner = THIS_MODULE,
.read = dp_link_settings_read,
.llseek = default_llseek
};
+static const struct file_operations sdp_message_fops = {
+ .owner = THIS_MODULE,
+ .write = dp_sdp_message_debugfs_write,
+ .llseek = default_llseek
+};
+
+static const struct file_operations dp_dpcd_address_debugfs_fops = {
+ .owner = THIS_MODULE,
+ .write = dp_dpcd_address_write,
+ .llseek = default_llseek
+};
+
+static const struct file_operations dp_dpcd_size_debugfs_fops = {
+ .owner = THIS_MODULE,
+ .write = dp_dpcd_size_write,
+ .llseek = default_llseek
+};
+
+static const struct file_operations dp_dpcd_data_debugfs_fops = {
+ .owner = THIS_MODULE,
+ .read = dp_dpcd_data_read,
+ .write = dp_dpcd_data_write,
+ .llseek = default_llseek
+};
+
static const struct {
char *name;
const struct file_operations *fops;
{"link_settings", &dp_link_settings_debugfs_fops},
{"phy_settings", &dp_phy_settings_debugfs_fop},
{"test_pattern", &dp_phy_test_pattern_fops},
- {"vrr_range", &vrr_range_fops}
+ {"vrr_range", &vrr_range_fops},
+ {"sdp_message", &sdp_message_fops},
+ {"aux_dpcd_address", &dp_dpcd_address_debugfs_fops},
+ {"aux_dpcd_size", &dp_dpcd_size_debugfs_fops},
+ {"aux_dpcd_data", &dp_dpcd_data_debugfs_fops}
};
int connector_debugfs_init(struct amdgpu_dm_connector *connector)
{"amdgpu_target_backlight_pwm", &target_backlight_read},
};
+/*
+ * Sets the DC visual confirm debug option from the given string.
+ * Example usage: echo 1 > /sys/kernel/debug/dri/0/amdgpu_visual_confirm
+ */
+static int visual_confirm_set(void *data, u64 val)
+{
+ struct amdgpu_device *adev = data;
+
+ adev->dm.dc->debug.visual_confirm = (enum visual_confirm)val;
+
+ return 0;
+}
+
+/*
+ * Reads the DC visual confirm debug option value into the given buffer.
+ * Example usage: cat /sys/kernel/debug/dri/0/amdgpu_dm_visual_confirm
+ */
+static int visual_confirm_get(void *data, u64 *val)
+{
+ struct amdgpu_device *adev = data;
+
+ *val = adev->dm.dc->debug.visual_confirm;
+
+ return 0;
+}
+
+DEFINE_DEBUGFS_ATTRIBUTE(visual_confirm_fops, visual_confirm_get,
+ visual_confirm_set, "%llu\n");
+
int dtn_debugfs_init(struct amdgpu_device *adev)
{
static const struct file_operations dtn_log_fops = {
adev,
&dtn_log_fops);
- return PTR_ERR_OR_ZERO(ent);
+ if (IS_ERR(ent))
+ return PTR_ERR(ent);
+
+ ent = debugfs_create_file_unsafe("amdgpu_dm_visual_confirm", 0644, root,
+ adev, &visual_confirm_fops);
+ if (IS_ERR(ent))
+ return PTR_ERR(ent);
+
+ return 0;
}
}
/*
- * poll pending down reply before clear payload allocation table
+ * poll pending down reply
*/
void dm_helpers_dp_mst_poll_pending_down_reply(
struct dc_context *ctx,
__func__);
}
+static int amdgpu_dm_set_vupdate_irq_state(struct amdgpu_device *adev,
+ struct amdgpu_irq_src *source,
+ unsigned int crtc_id,
+ enum amdgpu_interrupt_state state)
+{
+ return dm_irq_state(
+ adev,
+ source,
+ crtc_id,
+ state,
+ IRQ_TYPE_VUPDATE,
+ __func__);
+}
+
static const struct amdgpu_irq_src_funcs dm_crtc_irq_funcs = {
.set = amdgpu_dm_set_crtc_irq_state,
.process = amdgpu_dm_irq_handler,
};
+static const struct amdgpu_irq_src_funcs dm_vupdate_irq_funcs = {
+ .set = amdgpu_dm_set_vupdate_irq_state,
+ .process = amdgpu_dm_irq_handler,
+};
+
static const struct amdgpu_irq_src_funcs dm_pageflip_irq_funcs = {
.set = amdgpu_dm_set_pflip_irq_state,
.process = amdgpu_dm_irq_handler,
adev->crtc_irq.num_types = adev->mode_info.num_crtc;
adev->crtc_irq.funcs = &dm_crtc_irq_funcs;
+ adev->vupdate_irq.num_types = adev->mode_info.num_crtc;
+ adev->vupdate_irq.funcs = &dm_vupdate_irq_funcs;
+
adev->pageflip_irq.num_types = adev->mode_info.num_crtc;
adev->pageflip_irq.funcs = &dm_pageflip_irq_funcs;
{
ssize_t result = 0;
struct aux_payload payload;
+ enum aux_channel_operation_result operation_result;
if (WARN_ON(msg->size > 16))
return -E2BIG;
payload.mot = (msg->request & DP_AUX_I2C_MOT) != 0;
payload.defer_delay = 0;
- result = dc_link_aux_transfer(TO_DM_AUX(aux)->ddc_service, &payload);
+ result = dc_link_aux_transfer_raw(TO_DM_AUX(aux)->ddc_service, &payload,
+ &operation_result);
if (payload.write)
result = msg->size;
- if (result < 0) /* DC doesn't know about kernel error codes */
- result = -EIO;
+ if (result < 0)
+ switch (operation_result) {
+ case AUX_CHANNEL_OPERATION_SUCCEEDED:
+ break;
+ case AUX_CHANNEL_OPERATION_FAILED_HPD_DISCON:
+ case AUX_CHANNEL_OPERATION_FAILED_REASON_UNKNOWN:
+ result = -EIO;
+ break;
+ case AUX_CHANNEL_OPERATION_FAILED_INVALID_REPLY:
+ result = -EBUSY;
+ break;
+ case AUX_CHANNEL_OPERATION_FAILED_TIMEOUT:
+ result = -ETIMEDOUT;
+ break;
+ }
return result;
}
#include "amdgpu_dm_irq.h"
#include "amdgpu_pm.h"
#include "dm_pp_smu.h"
+#include "amdgpu_smu.h"
bool dm_pp_apply_display_requirements(
const struct dm_pp_display_configuration *pp_display_cfg)
{
struct amdgpu_device *adev = ctx->driver_context;
+ struct smu_context *smu = &adev->smu;
int i;
if (adev->pm.dpm_enabled) {
adev->powerplay.pp_funcs->display_configuration_change(
adev->powerplay.pp_handle,
&adev->pm.pm_display_cfg);
+ else
+ smu_display_configuration_change(smu,
+ &adev->pm.pm_display_cfg);
amdgpu_pm_compute_clocks(adev);
}
if (adev->powerplay.pp_funcs->get_clock_by_type(pp_handle,
dc_to_pp_clock_type(clk_type), &pp_clks)) {
/* Error in pplib. Provide default values. */
+ return true;
+ }
+ } else if (adev->smu.funcs && adev->smu.funcs->get_clock_by_type) {
+ if (smu_get_clock_by_type(&adev->smu,
+ dc_to_pp_clock_type(clk_type),
+ &pp_clks)) {
get_default_clock_levels(clk_type, dc_clks);
return true;
}
validation_clks.memory_max_clock = 80000;
validation_clks.level = 0;
}
+ } else if (adev->smu.funcs && adev->smu.funcs->get_max_high_clocks) {
+ if (smu_get_max_high_clocks(&adev->smu, &validation_clks)) {
+ DRM_INFO("DM_PPLIB: Warning: using default validation clocks!\n");
+ validation_clks.engine_max_clock = 72000;
+ validation_clks.memory_max_clock = 80000;
+ validation_clks.level = 0;
+ }
}
DRM_INFO("DM_PPLIB: Validation clocks:\n");
void *pp_handle = adev->powerplay.pp_handle;
struct pp_clock_levels_with_latency pp_clks = { 0 };
const struct amd_pm_funcs *pp_funcs = adev->powerplay.pp_funcs;
+ int ret;
+
+ if (pp_funcs && pp_funcs->get_clock_by_type_with_latency) {
+ ret = pp_funcs->get_clock_by_type_with_latency(pp_handle,
+ dc_to_pp_clock_type(clk_type),
+ &pp_clks);
+ if (ret)
+ return false;
+ } else if (adev->smu.ppt_funcs && adev->smu.ppt_funcs->get_clock_by_type_with_latency) {
+ if (smu_get_clock_by_type_with_latency(&adev->smu,
+ dc_to_pp_clock_type(clk_type),
+ &pp_clks))
+ return false;
+ }
- if (!pp_funcs || !pp_funcs->get_clock_by_type_with_latency)
- return false;
-
- if (pp_funcs->get_clock_by_type_with_latency(pp_handle,
- dc_to_pp_clock_type(clk_type),
- &pp_clks))
- return false;
pp_to_dc_clock_levels_with_latency(&pp_clks, clk_level_info, clk_type);
void *pp_handle = adev->powerplay.pp_handle;
struct pp_clock_levels_with_voltage pp_clk_info = {0};
const struct amd_pm_funcs *pp_funcs = adev->powerplay.pp_funcs;
-
- if (!pp_funcs || !pp_funcs->get_clock_by_type_with_voltage)
- return false;
-
- if (pp_funcs->get_clock_by_type_with_voltage(pp_handle,
- dc_to_pp_clock_type(clk_type),
- &pp_clk_info))
- return false;
+ int ret;
+
+ if (pp_funcs && pp_funcs->get_clock_by_type_with_voltage) {
+ ret = pp_funcs->get_clock_by_type_with_voltage(pp_handle,
+ dc_to_pp_clock_type(clk_type),
+ &pp_clk_info);
+ if (ret)
+ return false;
+ } else if (adev->smu.ppt_funcs && adev->smu.ppt_funcs->get_clock_by_type_with_voltage) {
+ if (smu_get_clock_by_type_with_voltage(&adev->smu,
+ dc_to_pp_clock_type(clk_type),
+ &pp_clk_info))
+ return false;
+ }
pp_to_dc_clock_levels_with_voltage(&pp_clk_info, clk_level_info, clk_type);
ret = adev->powerplay.pp_funcs->display_clock_voltage_request(
adev->powerplay.pp_handle,
&pp_clock_request);
+ else if (adev->smu.funcs &&
+ adev->smu.funcs->display_clock_voltage_request)
+ ret = smu_display_clock_voltage_request(&adev->smu,
+ &pp_clock_request);
if (ret)
return false;
return true;
ret = adev->powerplay.pp_funcs->get_current_clocks(
adev->powerplay.pp_handle,
&pp_clk_info);
+ else if (adev->smu.funcs)
+ ret = smu_get_current_clocks(&adev->smu, &pp_clk_info);
if (ret)
return false;
return true;
}
-void pp_rv_set_display_requirement(struct pp_smu *pp,
- struct pp_smu_display_requirement_rv *req)
-{
- const struct dc_context *ctx = pp->dm;
- struct amdgpu_device *adev = ctx->driver_context;
- void *pp_handle = adev->powerplay.pp_handle;
- const struct amd_pm_funcs *pp_funcs = adev->powerplay.pp_funcs;
- struct pp_display_clock_request clock = {0};
-
- if (!pp_funcs || !pp_funcs->display_clock_voltage_request)
- return;
-
- clock.clock_type = amd_pp_dcf_clock;
- clock.clock_freq_in_khz = req->hard_min_dcefclk_mhz * 1000;
- pp_funcs->display_clock_voltage_request(pp_handle, &clock);
-
- clock.clock_type = amd_pp_f_clock;
- clock.clock_freq_in_khz = req->hard_min_fclk_mhz * 1000;
- pp_funcs->display_clock_voltage_request(pp_handle, &clock);
-}
-
void pp_rv_set_wm_ranges(struct pp_smu *pp,
struct pp_smu_wm_range_sets *ranges)
{
wm_with_clock_ranges.num_wm_dmif_sets = ranges->num_reader_wm_sets;
wm_with_clock_ranges.num_wm_mcif_sets = ranges->num_writer_wm_sets;
- if (!pp_funcs || !pp_funcs->set_watermarks_for_clocks_ranges)
- return;
-
for (i = 0; i < wm_with_clock_ranges.num_wm_dmif_sets; i++) {
if (ranges->reader_wm_sets[i].wm_inst > 3)
wm_dce_clocks[i].wm_set_id = WM_SET_A;
ranges->writer_wm_sets[i].min_drain_clk_mhz * 1000;
}
- pp_funcs->set_watermarks_for_clocks_ranges(pp_handle, &wm_with_clock_ranges);
+ if (pp_funcs && pp_funcs->set_watermarks_for_clocks_ranges)
+ pp_funcs->set_watermarks_for_clocks_ranges(pp_handle,
+ &wm_with_clock_ranges);
+ else if (adev->smu.funcs &&
+ adev->smu.funcs->set_watermarks_for_clock_ranges)
+ smu_set_watermarks_for_clock_ranges(&adev->smu,
+ &wm_with_clock_ranges);
}
void pp_rv_set_pme_wa_enable(struct pp_smu *pp)
void *pp_handle = adev->powerplay.pp_handle;
const struct amd_pm_funcs *pp_funcs = adev->powerplay.pp_funcs;
- if (!pp_funcs || !pp_funcs->notify_smu_enable_pwe)
- return;
-
- pp_funcs->notify_smu_enable_pwe(pp_handle);
+ if (pp_funcs && pp_funcs->notify_smu_enable_pwe)
+ pp_funcs->notify_smu_enable_pwe(pp_handle);
+ else if (adev->smu.funcs)
+ smu_notify_smu_enable_pwe(&adev->smu);
}
void pp_rv_set_active_display_count(struct pp_smu *pp, int count)
pp_funcs->set_hard_min_fclk_by_freq(pp_handle, mhz);
}
-void dm_pp_get_funcs_rv(
+void dm_pp_get_funcs(
struct dc_context *ctx,
- struct pp_smu_funcs_rv *funcs)
+ struct pp_smu_funcs *funcs)
{
- funcs->pp_smu.dm = ctx;
- funcs->set_display_requirement = pp_rv_set_display_requirement;
- funcs->set_wm_ranges = pp_rv_set_wm_ranges;
- funcs->set_pme_wa_enable = pp_rv_set_pme_wa_enable;
- funcs->set_display_count = pp_rv_set_active_display_count;
- funcs->set_min_deep_sleep_dcfclk = pp_rv_set_min_deep_sleep_dcfclk;
- funcs->set_hard_min_dcfclk_by_freq = pp_rv_set_hard_min_dcefclk_by_freq;
- funcs->set_hard_min_fclk_by_freq = pp_rv_set_hard_min_fclk_by_freq;
+ funcs->rv_funcs.pp_smu.dm = ctx;
+ funcs->rv_funcs.set_wm_ranges = pp_rv_set_wm_ranges;
+ funcs->rv_funcs.set_pme_wa_enable = pp_rv_set_pme_wa_enable;
+ funcs->rv_funcs.set_display_count = pp_rv_set_active_display_count;
+ funcs->rv_funcs.set_min_deep_sleep_dcfclk = pp_rv_set_min_deep_sleep_dcfclk;
+ funcs->rv_funcs.set_hard_min_dcfclk_by_freq = pp_rv_set_hard_min_dcefclk_by_freq;
+ funcs->rv_funcs.set_hard_min_fclk_by_freq = pp_rv_set_hard_min_fclk_by_freq;
}
return min_clamp;
}
+unsigned int dc_fixpt_u4d19(struct fixed31_32 arg)
+{
+ return ux_dy(arg.value, 4, 19);
+}
+
unsigned int dc_fixpt_u3d19(struct fixed31_32 arg)
{
return ux_dy(arg.value, 3, 19);
}
}
+enum source_macro_tile_size swizzle_mode_to_macro_tile_size(enum swizzle_mode_values sw_mode)
+{
+ switch (sw_mode) {
+ /* for 4/8/16 high tiles */
+ case DC_SW_LINEAR:
+ return dm_4k_tile;
+ case DC_SW_4KB_S:
+ case DC_SW_4KB_S_X:
+ return dm_4k_tile;
+ case DC_SW_64KB_S:
+ case DC_SW_64KB_S_X:
+ case DC_SW_64KB_S_T:
+ return dm_64k_tile;
+ case DC_SW_VAR_S:
+ case DC_SW_VAR_S_X:
+ return dm_256k_tile;
+
+ /* For 64bpp 2 high tiles */
+ case DC_SW_4KB_D:
+ case DC_SW_4KB_D_X:
+ return dm_4k_tile;
+ case DC_SW_64KB_D:
+ case DC_SW_64KB_D_X:
+ case DC_SW_64KB_D_T:
+ return dm_64k_tile;
+ case DC_SW_VAR_D:
+ case DC_SW_VAR_D_X:
+ return dm_256k_tile;
+
+ case DC_SW_4KB_R:
+ case DC_SW_4KB_R_X:
+ return dm_4k_tile;
+ case DC_SW_64KB_R:
+ case DC_SW_64KB_R_X:
+ return dm_64k_tile;
+ case DC_SW_VAR_R:
+ case DC_SW_VAR_R_X:
+ return dm_256k_tile;
+
+ /* Unsupported swizzle modes for dcn */
+ case DC_SW_256B_S:
+ default:
+ ASSERT(0); /* Not supported */
+ return 0;
+ }
+}
+
static void pipe_ctx_to_e2e_pipe_params (
const struct pipe_ctx *pipe,
struct _vcs_dpi_display_pipe_params_st *input)
input->src.cur0_src_width = 128; /* TODO: Cursor calcs, not curently stored */
input->src.cur0_bpp = 32;
- switch (pipe->plane_state->tiling_info.gfx9.swizzle) {
- /* for 4/8/16 high tiles */
- case DC_SW_LINEAR:
- input->src.macro_tile_size = dm_4k_tile;
- break;
- case DC_SW_4KB_S:
- case DC_SW_4KB_S_X:
- input->src.macro_tile_size = dm_4k_tile;
- break;
- case DC_SW_64KB_S:
- case DC_SW_64KB_S_X:
- case DC_SW_64KB_S_T:
- input->src.macro_tile_size = dm_64k_tile;
- break;
- case DC_SW_VAR_S:
- case DC_SW_VAR_S_X:
- input->src.macro_tile_size = dm_256k_tile;
- break;
-
- /* For 64bpp 2 high tiles */
- case DC_SW_4KB_D:
- case DC_SW_4KB_D_X:
- input->src.macro_tile_size = dm_4k_tile;
- break;
- case DC_SW_64KB_D:
- case DC_SW_64KB_D_X:
- case DC_SW_64KB_D_T:
- input->src.macro_tile_size = dm_64k_tile;
- break;
- case DC_SW_VAR_D:
- case DC_SW_VAR_D_X:
- input->src.macro_tile_size = dm_256k_tile;
- break;
-
- /* Unsupported swizzle modes for dcn */
- case DC_SW_256B_S:
- default:
- ASSERT(0); /* Not supported */
- break;
- }
+ input->src.macro_tile_size = swizzle_mode_to_macro_tile_size(pipe->plane_state->tiling_info.gfx9.swizzle);
switch (pipe->plane_state->rotation) {
case ROTATION_ANGLE_0:
input.clks_cfg.dcfclk_mhz = v->dcfclk;
input.clks_cfg.dispclk_mhz = v->dispclk;
input.clks_cfg.dppclk_mhz = v->dppclk;
- input.clks_cfg.refclk_mhz = dc->res_pool->ref_clock_inKhz / 1000.0;
+ input.clks_cfg.refclk_mhz = dc->res_pool->ref_clocks.dchub_ref_clock_inKhz / 1000.0;
input.clks_cfg.socclk_mhz = v->socclk;
input.clks_cfg.voltage = v->voltage_level;
// dc->dml.logger = pool->base.logger;
v->fabric_and_dram_bandwidth = v->fabric_and_dram_bandwidth_vnom0p8;
dispclkdppclkdcfclk_deep_sleep_prefetch_parameters_watermarks_and_performance_calculation(v);
- context->bw.dcn.watermarks.b.cstate_pstate.cstate_exit_ns =
+ context->bw_ctx.bw.dcn.watermarks.b.cstate_pstate.cstate_exit_ns =
v->stutter_exit_watermark * 1000;
- context->bw.dcn.watermarks.b.cstate_pstate.cstate_enter_plus_exit_ns =
+ context->bw_ctx.bw.dcn.watermarks.b.cstate_pstate.cstate_enter_plus_exit_ns =
v->stutter_enter_plus_exit_watermark * 1000;
- context->bw.dcn.watermarks.b.cstate_pstate.pstate_change_ns =
+ context->bw_ctx.bw.dcn.watermarks.b.cstate_pstate.pstate_change_ns =
v->dram_clock_change_watermark * 1000;
- context->bw.dcn.watermarks.b.pte_meta_urgent_ns = v->ptemeta_urgent_watermark * 1000;
- context->bw.dcn.watermarks.b.urgent_ns = v->urgent_watermark * 1000;
+ context->bw_ctx.bw.dcn.watermarks.b.pte_meta_urgent_ns = v->ptemeta_urgent_watermark * 1000;
+ context->bw_ctx.bw.dcn.watermarks.b.urgent_ns = v->urgent_watermark * 1000;
v->dcfclk_per_state[1] = v->dcfclkv_nom0p8;
v->dcfclk_per_state[0] = v->dcfclkv_nom0p8;
v->dcfclk = v->dcfclkv_nom0p8;
dispclkdppclkdcfclk_deep_sleep_prefetch_parameters_watermarks_and_performance_calculation(v);
- context->bw.dcn.watermarks.c.cstate_pstate.cstate_exit_ns =
+ context->bw_ctx.bw.dcn.watermarks.c.cstate_pstate.cstate_exit_ns =
v->stutter_exit_watermark * 1000;
- context->bw.dcn.watermarks.c.cstate_pstate.cstate_enter_plus_exit_ns =
+ context->bw_ctx.bw.dcn.watermarks.c.cstate_pstate.cstate_enter_plus_exit_ns =
v->stutter_enter_plus_exit_watermark * 1000;
- context->bw.dcn.watermarks.c.cstate_pstate.pstate_change_ns =
+ context->bw_ctx.bw.dcn.watermarks.c.cstate_pstate.pstate_change_ns =
v->dram_clock_change_watermark * 1000;
- context->bw.dcn.watermarks.c.pte_meta_urgent_ns = v->ptemeta_urgent_watermark * 1000;
- context->bw.dcn.watermarks.c.urgent_ns = v->urgent_watermark * 1000;
+ context->bw_ctx.bw.dcn.watermarks.c.pte_meta_urgent_ns = v->ptemeta_urgent_watermark * 1000;
+ context->bw_ctx.bw.dcn.watermarks.c.urgent_ns = v->urgent_watermark * 1000;
}
if (v->voltage_level < 3) {
v->dcfclk = v->dcfclkv_max0p9;
dispclkdppclkdcfclk_deep_sleep_prefetch_parameters_watermarks_and_performance_calculation(v);
- context->bw.dcn.watermarks.d.cstate_pstate.cstate_exit_ns =
+ context->bw_ctx.bw.dcn.watermarks.d.cstate_pstate.cstate_exit_ns =
v->stutter_exit_watermark * 1000;
- context->bw.dcn.watermarks.d.cstate_pstate.cstate_enter_plus_exit_ns =
+ context->bw_ctx.bw.dcn.watermarks.d.cstate_pstate.cstate_enter_plus_exit_ns =
v->stutter_enter_plus_exit_watermark * 1000;
- context->bw.dcn.watermarks.d.cstate_pstate.pstate_change_ns =
+ context->bw_ctx.bw.dcn.watermarks.d.cstate_pstate.pstate_change_ns =
v->dram_clock_change_watermark * 1000;
- context->bw.dcn.watermarks.d.pte_meta_urgent_ns = v->ptemeta_urgent_watermark * 1000;
- context->bw.dcn.watermarks.d.urgent_ns = v->urgent_watermark * 1000;
+ context->bw_ctx.bw.dcn.watermarks.d.pte_meta_urgent_ns = v->ptemeta_urgent_watermark * 1000;
+ context->bw_ctx.bw.dcn.watermarks.d.urgent_ns = v->urgent_watermark * 1000;
}
v->fabric_and_dram_bandwidth_per_state[2] = v->fabric_and_dram_bandwidth_vnom0p8;
v->dcfclk = v->dcfclk_per_state[v->voltage_level];
dispclkdppclkdcfclk_deep_sleep_prefetch_parameters_watermarks_and_performance_calculation(v);
- context->bw.dcn.watermarks.a.cstate_pstate.cstate_exit_ns =
+ context->bw_ctx.bw.dcn.watermarks.a.cstate_pstate.cstate_exit_ns =
v->stutter_exit_watermark * 1000;
- context->bw.dcn.watermarks.a.cstate_pstate.cstate_enter_plus_exit_ns =
+ context->bw_ctx.bw.dcn.watermarks.a.cstate_pstate.cstate_enter_plus_exit_ns =
v->stutter_enter_plus_exit_watermark * 1000;
- context->bw.dcn.watermarks.a.cstate_pstate.pstate_change_ns =
+ context->bw_ctx.bw.dcn.watermarks.a.cstate_pstate.pstate_change_ns =
v->dram_clock_change_watermark * 1000;
- context->bw.dcn.watermarks.a.pte_meta_urgent_ns = v->ptemeta_urgent_watermark * 1000;
- context->bw.dcn.watermarks.a.urgent_ns = v->urgent_watermark * 1000;
+ context->bw_ctx.bw.dcn.watermarks.a.pte_meta_urgent_ns = v->ptemeta_urgent_watermark * 1000;
+ context->bw_ctx.bw.dcn.watermarks.a.urgent_ns = v->urgent_watermark * 1000;
if (v->voltage_level >= 2) {
- context->bw.dcn.watermarks.b = context->bw.dcn.watermarks.a;
- context->bw.dcn.watermarks.c = context->bw.dcn.watermarks.a;
+ context->bw_ctx.bw.dcn.watermarks.b = context->bw_ctx.bw.dcn.watermarks.a;
+ context->bw_ctx.bw.dcn.watermarks.c = context->bw_ctx.bw.dcn.watermarks.a;
}
if (v->voltage_level >= 3)
- context->bw.dcn.watermarks.d = context->bw.dcn.watermarks.a;
+ context->bw_ctx.bw.dcn.watermarks.d = context->bw_ctx.bw.dcn.watermarks.a;
}
#endif
bool dcn_validate_bandwidth(
struct dc *dc,
- struct dc_state *context)
+ struct dc_state *context,
+ bool fast_validate)
{
+ /*
+ * we want a breakdown of the various stages of validation, which the
+ * perf_trace macro doesn't support
+ */
+ BW_VAL_TRACE_SETUP();
+
const struct resource_pool *pool = dc->res_pool;
struct dcn_bw_internal_vars *v = &context->dcn_bw_vars;
int i, input_idx;
float bw_limit;
PERFORMANCE_TRACE_START();
+
+ BW_VAL_TRACE_COUNT();
+
if (dcn_bw_apply_registry_override(dc))
dcn_bw_sync_calcs_and_dml(dc);
dc->debug.sr_enter_plus_exit_time_dpm0_ns / 1000.0f;
if (dc->debug.sr_exit_time_dpm0_ns)
v->sr_exit_time = dc->debug.sr_exit_time_dpm0_ns / 1000.0f;
- dc->dml.soc.sr_enter_plus_exit_time_us = v->sr_enter_plus_exit_time;
- dc->dml.soc.sr_exit_time_us = v->sr_exit_time;
+ context->bw_ctx.dml.soc.sr_enter_plus_exit_time_us = v->sr_enter_plus_exit_time;
+ context->bw_ctx.dml.soc.sr_exit_time_us = v->sr_exit_time;
mode_support_and_system_configuration(v);
}
- if (v->voltage_level != 5) {
+ BW_VAL_TRACE_END_VOLTAGE_LEVEL();
+
+ if (v->voltage_level != number_of_states_plus_one && !fast_validate) {
float bw_consumed = v->total_bandwidth_consumed_gbyte_per_second;
+
if (bw_consumed < v->fabric_and_dram_bandwidth_vmin0p65)
bw_consumed = v->fabric_and_dram_bandwidth_vmin0p65;
else if (bw_consumed < v->fabric_and_dram_bandwidth_vmid0p72)
*/
dispclkdppclkdcfclk_deep_sleep_prefetch_parameters_watermarks_and_performance_calculation(v);
- context->bw.dcn.watermarks.a.cstate_pstate.cstate_exit_ns =
+ context->bw_ctx.bw.dcn.watermarks.a.cstate_pstate.cstate_exit_ns =
v->stutter_exit_watermark * 1000;
- context->bw.dcn.watermarks.a.cstate_pstate.cstate_enter_plus_exit_ns =
+ context->bw_ctx.bw.dcn.watermarks.a.cstate_pstate.cstate_enter_plus_exit_ns =
v->stutter_enter_plus_exit_watermark * 1000;
- context->bw.dcn.watermarks.a.cstate_pstate.pstate_change_ns =
+ context->bw_ctx.bw.dcn.watermarks.a.cstate_pstate.pstate_change_ns =
v->dram_clock_change_watermark * 1000;
- context->bw.dcn.watermarks.a.pte_meta_urgent_ns = v->ptemeta_urgent_watermark * 1000;
- context->bw.dcn.watermarks.a.urgent_ns = v->urgent_watermark * 1000;
- context->bw.dcn.watermarks.b = context->bw.dcn.watermarks.a;
- context->bw.dcn.watermarks.c = context->bw.dcn.watermarks.a;
- context->bw.dcn.watermarks.d = context->bw.dcn.watermarks.a;
+ context->bw_ctx.bw.dcn.watermarks.a.pte_meta_urgent_ns = v->ptemeta_urgent_watermark * 1000;
+ context->bw_ctx.bw.dcn.watermarks.a.urgent_ns = v->urgent_watermark * 1000;
+ context->bw_ctx.bw.dcn.watermarks.b = context->bw_ctx.bw.dcn.watermarks.a;
+ context->bw_ctx.bw.dcn.watermarks.c = context->bw_ctx.bw.dcn.watermarks.a;
+ context->bw_ctx.bw.dcn.watermarks.d = context->bw_ctx.bw.dcn.watermarks.a;
- context->bw.dcn.clk.fclk_khz = (int)(bw_consumed * 1000000 /
+ context->bw_ctx.bw.dcn.clk.fclk_khz = (int)(bw_consumed * 1000000 /
(ddr4_dram_factor_single_Channel * v->number_of_channels));
if (bw_consumed == v->fabric_and_dram_bandwidth_vmin0p65) {
- context->bw.dcn.clk.fclk_khz = (int)(bw_consumed * 1000000 / 32);
+ context->bw_ctx.bw.dcn.clk.fclk_khz = (int)(bw_consumed * 1000000 / 32);
}
- context->bw.dcn.clk.dcfclk_deep_sleep_khz = (int)(v->dcf_clk_deep_sleep * 1000);
- context->bw.dcn.clk.dcfclk_khz = (int)(v->dcfclk * 1000);
+ context->bw_ctx.bw.dcn.clk.dcfclk_deep_sleep_khz = (int)(v->dcf_clk_deep_sleep * 1000);
+ context->bw_ctx.bw.dcn.clk.dcfclk_khz = (int)(v->dcfclk * 1000);
- context->bw.dcn.clk.dispclk_khz = (int)(v->dispclk * 1000);
+ context->bw_ctx.bw.dcn.clk.dispclk_khz = (int)(v->dispclk * 1000);
if (dc->debug.max_disp_clk == true)
- context->bw.dcn.clk.dispclk_khz = (int)(dc->dcn_soc->max_dispclk_vmax0p9 * 1000);
+ context->bw_ctx.bw.dcn.clk.dispclk_khz = (int)(dc->dcn_soc->max_dispclk_vmax0p9 * 1000);
- if (context->bw.dcn.clk.dispclk_khz <
+ if (context->bw_ctx.bw.dcn.clk.dispclk_khz <
dc->debug.min_disp_clk_khz) {
- context->bw.dcn.clk.dispclk_khz =
+ context->bw_ctx.bw.dcn.clk.dispclk_khz =
dc->debug.min_disp_clk_khz;
}
- context->bw.dcn.clk.dppclk_khz = context->bw.dcn.clk.dispclk_khz / v->dispclk_dppclk_ratio;
- context->bw.dcn.clk.phyclk_khz = v->phyclk_per_state[v->voltage_level];
+ context->bw_ctx.bw.dcn.clk.dppclk_khz = context->bw_ctx.bw.dcn.clk.dispclk_khz / v->dispclk_dppclk_ratio;
+ context->bw_ctx.bw.dcn.clk.phyclk_khz = v->phyclk_per_state[v->voltage_level];
switch (v->voltage_level) {
case 0:
- context->bw.dcn.clk.max_supported_dppclk_khz =
+ context->bw_ctx.bw.dcn.clk.max_supported_dppclk_khz =
(int)(dc->dcn_soc->max_dppclk_vmin0p65 * 1000);
break;
case 1:
- context->bw.dcn.clk.max_supported_dppclk_khz =
+ context->bw_ctx.bw.dcn.clk.max_supported_dppclk_khz =
(int)(dc->dcn_soc->max_dppclk_vmid0p72 * 1000);
break;
case 2:
- context->bw.dcn.clk.max_supported_dppclk_khz =
+ context->bw_ctx.bw.dcn.clk.max_supported_dppclk_khz =
(int)(dc->dcn_soc->max_dppclk_vnom0p8 * 1000);
break;
default:
- context->bw.dcn.clk.max_supported_dppclk_khz =
+ context->bw_ctx.bw.dcn.clk.max_supported_dppclk_khz =
(int)(dc->dcn_soc->max_dppclk_vmax0p9 * 1000);
break;
}
+ BW_VAL_TRACE_END_WATERMARKS();
+
for (i = 0, input_idx = 0; i < pool->pipe_count; i++) {
struct pipe_ctx *pipe = &context->res_ctx.pipe_ctx[i];
hsplit_pipe->pipe_dlg_param.vblank_end = pipe->pipe_dlg_param.vblank_end;
} else {
/* pipe not split previously needs split */
- hsplit_pipe = find_idle_secondary_pipe(&context->res_ctx, pool);
+ hsplit_pipe = find_idle_secondary_pipe(&context->res_ctx, pool, pipe);
ASSERT(hsplit_pipe);
split_stream_across_pipes(
&context->res_ctx, pool,
input_idx++;
}
+ } else if (v->voltage_level == number_of_states_plus_one) {
+ BW_VAL_TRACE_SKIP(fail);
+ } else if (fast_validate) {
+ BW_VAL_TRACE_SKIP(fast);
}
if (v->voltage_level == 0) {
- dc->dml.soc.sr_enter_plus_exit_time_us =
+ context->bw_ctx.dml.soc.sr_enter_plus_exit_time_us =
dc->dcn_soc->sr_enter_plus_exit_time;
- dc->dml.soc.sr_exit_time_us = dc->dcn_soc->sr_exit_time;
+ context->bw_ctx.dml.soc.sr_exit_time_us = dc->dcn_soc->sr_exit_time;
}
/*
kernel_fpu_end();
PERFORMANCE_TRACE_END();
+ BW_VAL_TRACE_FINISH();
if (bw_limit_pass && v->voltage_level != 5)
return true;
void dcn_bw_notify_pplib_of_wm_ranges(struct dc *dc)
{
- struct pp_smu_funcs_rv *pp = dc->res_pool->pp_smu;
+ struct pp_smu_funcs_rv *pp = NULL;
struct pp_smu_wm_range_sets ranges = {0};
int min_fclk_khz, min_dcfclk_khz, socclk_khz;
const int overdrive = 5000000; /* 5 GHz to cover Overdrive */
- if (!pp->set_wm_ranges)
+ if (dc->res_pool->pp_smu)
+ pp = &dc->res_pool->pp_smu->rv_funcs;
+ if (!pp || !pp->set_wm_ranges)
return;
kernel_fpu_begin();
struct dc_stream_state *link_stream;
struct dc_link_settings store_settings = *link_setting;
+ link->preferred_link_setting = store_settings;
+
+ /* Retrain with preferred link settings only relevant for
+ * DP signal type
+ */
+ if (!dc_is_dp_signal(link->connector_signal))
+ return;
+
for (i = 0; i < MAX_PIPES; i++) {
pipe = &dc->current_state->res_ctx.pipe_ctx[i];
if (pipe->stream && pipe->stream->link) {
link_stream = link->dc->current_state->res_ctx.pipe_ctx[i].stream;
- link->preferred_link_setting = store_settings;
+ /* Cannot retrain link if backend is off */
+ if (link_stream->dpms_off)
+ return;
+
if (link_stream)
decide_link_settings(link_stream, &store_settings);
cust_pattern_size);
}
+uint32_t dc_link_bandwidth_kbps(
+ const struct dc_link *link,
+ const struct dc_link_settings *link_setting)
+{
+ uint32_t link_bw_kbps = link_setting->link_rate * LINK_RATE_REF_FREQ_IN_KHZ; /* bytes per sec */
+
+ link_bw_kbps *= 8; /* 8 bits per byte*/
+ link_bw_kbps *= link_setting->lane_count;
+
+ return link_bw_kbps;
+
+}
+
+const struct dc_link_settings *dc_link_get_link_cap(
+ const struct dc_link *link)
+{
+ if (link->preferred_link_setting.lane_count != LANE_COUNT_UNKNOWN &&
+ link->preferred_link_setting.link_rate != LINK_RATE_UNKNOWN)
+ return &link->preferred_link_setting;
+ return &link->verified_link_cap;
+}
+
static void destruct(struct dc *dc)
{
dc_release_state(dc->current_state);
#endif
enum dce_version dc_version = DCE_VERSION_UNKNOWN;
+ dc->config = init_params->flags;
+
+ memcpy(&dc->bb_overrides, &init_params->bb_overrides, sizeof(dc->bb_overrides));
+
dc_dceip = kzalloc(sizeof(*dc_dceip), GFP_KERNEL);
if (!dc_dceip) {
dm_error("%s: failed to create dceip\n", __func__);
dc_ctx->dc_stream_id_count = 0;
dc->ctx = dc_ctx;
- dc->current_state = dc_create_state();
-
- if (!dc->current_state) {
- dm_error("%s: failed to create validate ctx\n", __func__);
- goto fail;
- }
-
/* Create logger */
dc_ctx->dce_environment = init_params->dce_environment;
goto fail;
}
- dc->res_pool = dc_create_resource_pool(
- dc,
- init_params->num_virtual_links,
- dc_version,
- init_params->asic_id);
+ dc->res_pool = dc_create_resource_pool(dc, init_params, dc_version);
if (!dc->res_pool)
goto fail;
+ /* Creation of current_state must occur after dc->dml
+ * is initialized in dc_create_resource_pool because
+ * on creation it copies the contents of dc->dml
+ */
+
+ dc->current_state = dc_create_state(dc);
+
+ if (!dc->current_state) {
+ dm_error("%s: failed to create validate ctx\n", __func__);
+ goto fail;
+ }
+
dc_resource_state_construct(dc, dc->current_state);
if (!create_links(dc, init_params->num_virtual_links))
static void disable_dangling_plane(struct dc *dc, struct dc_state *context)
{
int i, j;
- struct dc_state *dangling_context = dc_create_state();
+ struct dc_state *dangling_context = dc_create_state(dc);
struct dc_state *current_ctx;
if (dangling_context == NULL)
if (dc->res_pool->dmcu != NULL)
dc->versions.dmcu_version = dc->res_pool->dmcu->dmcu_version;
- dc->config = init_params->flags;
-
dc->build_id = DC_BUILD_ID;
DC_LOG_DC("Display Core initialized\n");
return false;
}
-bool dc_validate_seamless_boot_timing(struct dc *dc,
+bool dc_validate_seamless_boot_timing(const struct dc *dc,
const struct dc_sink *sink,
struct dc_crtc_timing *crtc_timing)
{
if (!dcb->funcs->is_accelerated_mode(dcb))
dc->hwss.enable_accelerated_mode(dc, context);
- dc->hwss.prepare_bandwidth(dc, context);
+ for (i = 0; i < context->stream_count; i++) {
+ if (context->streams[i]->apply_seamless_boot_optimization)
+ dc->optimize_seamless_boot = true;
+ }
+
+ if (!dc->optimize_seamless_boot)
+ dc->hwss.prepare_bandwidth(dc, context);
/* re-program planes for existing stream, in case we need to
* free up plane resource for later use
dc_enable_stereo(dc, context, dc_streams, context->stream_count);
- /* pplib is notified if disp_num changed */
- dc->hwss.optimize_bandwidth(dc, context);
+ if (!dc->optimize_seamless_boot)
+ /* pplib is notified if disp_num changed */
+ dc->hwss.optimize_bandwidth(dc, context);
for (i = 0; i < context->stream_count; i++)
context->streams[i]->mode_changed = false;
+ memset(&context->commit_hints, 0, sizeof(context->commit_hints));
+
dc_release_state(dc->current_state);
dc->current_state = context;
int i;
struct dc_state *context = dc->current_state;
- if (dc->optimized_required == false)
+ if (!dc->optimized_required || dc->optimize_seamless_boot)
return true;
post_surface_trace(dc);
return true;
}
-struct dc_state *dc_create_state(void)
+struct dc_state *dc_create_state(struct dc *dc)
{
struct dc_state *context = kzalloc(sizeof(struct dc_state),
GFP_KERNEL);
if (!context)
return NULL;
+ /* Each context must have their own instance of VBA and in order to
+ * initialize and obtain IP and SOC the base DML instance from DC is
+ * initially copied into every context
+ */
+#ifdef CONFIG_DRM_AMD_DC_DCN1_0
+ memcpy(&context->bw_ctx.dml, &dc->dml, sizeof(struct display_mode_lib));
+#endif
kref_init(&context->refcount);
+
return context;
}
+struct dc_state *dc_copy_state(struct dc_state *src_ctx)
+{
+ int i, j;
+ struct dc_state *new_ctx = kzalloc(sizeof(struct dc_state),
+ GFP_KERNEL);
+
+ if (!new_ctx)
+ return NULL;
+
+ memcpy(new_ctx, src_ctx, sizeof(struct dc_state));
+
+ for (i = 0; i < MAX_PIPES; i++) {
+ struct pipe_ctx *cur_pipe = &new_ctx->res_ctx.pipe_ctx[i];
+
+ if (cur_pipe->top_pipe)
+ cur_pipe->top_pipe = &new_ctx->res_ctx.pipe_ctx[cur_pipe->top_pipe->pipe_idx];
+
+ if (cur_pipe->bottom_pipe)
+ cur_pipe->bottom_pipe = &new_ctx->res_ctx.pipe_ctx[cur_pipe->bottom_pipe->pipe_idx];
+
+ }
+
+ for (i = 0; i < new_ctx->stream_count; i++) {
+ dc_stream_retain(new_ctx->streams[i]);
+ for (j = 0; j < new_ctx->stream_status[i].plane_count; j++)
+ dc_plane_state_retain(
+ new_ctx->stream_status[i].plane_states[j]);
+ }
+
+ kref_init(&new_ctx->refcount);
+
+ return new_ctx;
+}
+
void dc_retain_state(struct dc_state *context)
{
kref_get(&context->refcount);
continue;
if (stream_update->dpms_off) {
+ dc->hwss.pipe_control_lock(dc, pipe_ctx, true);
if (*stream_update->dpms_off) {
core_link_disable_stream(pipe_ctx, KEEP_ACQUIRED_RESOURCE);
dc->hwss.optimize_bandwidth(dc, dc->current_state);
dc->hwss.prepare_bandwidth(dc, dc->current_state);
core_link_enable_stream(dc->current_state, pipe_ctx);
}
+ dc->hwss.pipe_control_lock(dc, pipe_ctx, false);
}
if (stream_update->abm_level && pipe_ctx->stream_res.abm) {
int i, j;
struct pipe_ctx *top_pipe_to_program = NULL;
- if (update_type == UPDATE_TYPE_FULL) {
+ if (dc->optimize_seamless_boot && surface_count > 0) {
+ /* Optimize seamless boot flag keeps clocks and watermarks high until
+ * first flip. After first flip, optimization is required to lower
+ * bandwidth.
+ */
+ dc->optimize_seamless_boot = false;
+ dc->optimized_required = true;
+ }
+
+ if (update_type == UPDATE_TYPE_FULL && !dc->optimize_seamless_boot) {
dc->hwss.prepare_bandwidth(dc, context);
context_clock_trace(dc, context);
}
if (update_type >= UPDATE_TYPE_FULL) {
/* initialize scratch memory for building context */
- context = dc_create_state();
+ context = dc_create_state(dc);
if (context == NULL) {
DC_ERROR("Failed to allocate new validate context!\n");
return;
void get_clock_requirements_for_state(struct dc_state *state, struct AsicStateEx *info)
{
- info->displayClock = (unsigned int)state->bw.dcn.clk.dispclk_khz;
- info->engineClock = (unsigned int)state->bw.dcn.clk.dcfclk_khz;
- info->memoryClock = (unsigned int)state->bw.dcn.clk.dramclk_khz;
- info->maxSupportedDppClock = (unsigned int)state->bw.dcn.clk.max_supported_dppclk_khz;
- info->dppClock = (unsigned int)state->bw.dcn.clk.dppclk_khz;
- info->socClock = (unsigned int)state->bw.dcn.clk.socclk_khz;
- info->dcfClockDeepSleep = (unsigned int)state->bw.dcn.clk.dcfclk_deep_sleep_khz;
- info->fClock = (unsigned int)state->bw.dcn.clk.fclk_khz;
- info->phyClock = (unsigned int)state->bw.dcn.clk.phyclk_khz;
+ info->displayClock = (unsigned int)state->bw_ctx.bw.dcn.clk.dispclk_khz;
+ info->engineClock = (unsigned int)state->bw_ctx.bw.dcn.clk.dcfclk_khz;
+ info->memoryClock = (unsigned int)state->bw_ctx.bw.dcn.clk.dramclk_khz;
+ info->maxSupportedDppClock = (unsigned int)state->bw_ctx.bw.dcn.clk.max_supported_dppclk_khz;
+ info->dppClock = (unsigned int)state->bw_ctx.bw.dcn.clk.dppclk_khz;
+ info->socClock = (unsigned int)state->bw_ctx.bw.dcn.clk.socclk_khz;
+ info->dcfClockDeepSleep = (unsigned int)state->bw_ctx.bw.dcn.clk.dcfclk_deep_sleep_khz;
+ info->fClock = (unsigned int)state->bw_ctx.bw.dcn.clk.fclk_khz;
+ info->phyClock = (unsigned int)state->bw_ctx.bw.dcn.clk.phyclk_khz;
}
DC_LOGGER_INIT(dc->ctx->logger);
CLOCK_TRACE("Current: dispclk_khz:%d max_dppclk_khz:%d dcfclk_khz:%d\n"
"dcfclk_deep_sleep_khz:%d fclk_khz:%d socclk_khz:%d\n",
- context->bw.dcn.clk.dispclk_khz,
- context->bw.dcn.clk.dppclk_khz,
- context->bw.dcn.clk.dcfclk_khz,
- context->bw.dcn.clk.dcfclk_deep_sleep_khz,
- context->bw.dcn.clk.fclk_khz,
- context->bw.dcn.clk.socclk_khz);
+ context->bw_ctx.bw.dcn.clk.dispclk_khz,
+ context->bw_ctx.bw.dcn.clk.dppclk_khz,
+ context->bw_ctx.bw.dcn.clk.dcfclk_khz,
+ context->bw_ctx.bw.dcn.clk.dcfclk_deep_sleep_khz,
+ context->bw_ctx.bw.dcn.clk.fclk_khz,
+ context->bw_ctx.bw.dcn.clk.socclk_khz);
CLOCK_TRACE("Calculated: dispclk_khz:%d max_dppclk_khz:%d dcfclk_khz:%d\n"
"dcfclk_deep_sleep_khz:%d fclk_khz:%d socclk_khz:%d\n",
- context->bw.dcn.clk.dispclk_khz,
- context->bw.dcn.clk.dppclk_khz,
- context->bw.dcn.clk.dcfclk_khz,
- context->bw.dcn.clk.dcfclk_deep_sleep_khz,
- context->bw.dcn.clk.fclk_khz,
- context->bw.dcn.clk.socclk_khz);
+ context->bw_ctx.bw.dcn.clk.dispclk_khz,
+ context->bw_ctx.bw.dcn.clk.dppclk_khz,
+ context->bw_ctx.bw.dcn.clk.dcfclk_khz,
+ context->bw_ctx.bw.dcn.clk.dcfclk_deep_sleep_khz,
+ context->bw_ctx.bw.dcn.clk.fclk_khz,
+ context->bw_ctx.bw.dcn.clk.socclk_khz);
#endif
}
******************************************************************************/
enum {
- LINK_RATE_REF_FREQ_IN_MHZ = 27,
PEAK_FACTOR_X1000 = 1006,
/*
* Some receivers fail to train on first try and are good
}
+static void read_edp_current_link_settings_on_detect(struct dc_link *link)
+{
+ union lane_count_set lane_count_set = { {0} };
+ uint8_t link_bw_set;
+ uint8_t link_rate_set;
+
+ // Read DPCD 00101h to find out the number of lanes currently set
+ core_link_read_dpcd(link, DP_LANE_COUNT_SET,
+ &lane_count_set.raw, sizeof(lane_count_set));
+ link->cur_link_settings.lane_count = lane_count_set.bits.LANE_COUNT_SET;
+
+ // Read DPCD 00100h to find if standard link rates are set
+ core_link_read_dpcd(link, DP_LINK_BW_SET,
+ &link_bw_set, sizeof(link_bw_set));
+
+ if (link_bw_set == 0) {
+ /* If standard link rates are not being used,
+ * Read DPCD 00115h to find the link rate set used
+ */
+ core_link_read_dpcd(link, DP_LINK_RATE_SET,
+ &link_rate_set, sizeof(link_rate_set));
+
+ if (link_rate_set < link->dpcd_caps.edp_supported_link_rates_count) {
+ link->cur_link_settings.link_rate =
+ link->dpcd_caps.edp_supported_link_rates[link_rate_set];
+ link->cur_link_settings.link_rate_set = link_rate_set;
+ link->cur_link_settings.use_link_rate_set = true;
+ }
+ } else {
+ link->cur_link_settings.link_rate = link_bw_set;
+ link->cur_link_settings.use_link_rate_set = false;
+ }
+}
+
static bool detect_dp(
struct dc_link *link,
struct display_sink_capability *sink_caps,
bool same_dpcd = true;
enum dc_connection_type new_connection_type = dc_connection_none;
DC_LOGGER_INIT(link->ctx->logger);
- if (link->connector_signal == SIGNAL_TYPE_VIRTUAL)
+
+ if (dc_is_virtual_signal(link->connector_signal))
return false;
if (false == dc_link_detect_sink(link, &new_connection_type)) {
return false;
}
- if (link->connector_signal == SIGNAL_TYPE_EDP &&
- link->local_sink)
- return true;
+ if (link->connector_signal == SIGNAL_TYPE_EDP) {
+ /* On detect, we want to make sure current link settings are
+ * up to date, especially if link was powered on by GOP.
+ */
+ read_edp_current_link_settings_on_detect(link);
+ if (link->local_sink)
+ return true;
+ }
if (link->connector_signal == SIGNAL_TYPE_LVDS &&
link->local_sink)
same_dpcd = false;
}
/* Active dongle plug in without display or downstream unplug*/
- if (link->type == dc_connection_active_dongle
- && link->dpcd_caps.sink_count.
- bits.SINK_COUNT == 0) {
+ if (link->type == dc_connection_active_dongle &&
+ link->dpcd_caps.sink_count.bits.SINK_COUNT == 0) {
if (prev_sink != NULL) {
/* Downstream unplug */
dc_sink_release(prev_sink);
goto create_fail;
}
-
-
/* TODO: #DAL3 Implement id to str function.*/
LINK_INFO("Connector[%d] description:"
"signal %d\n",
link->link_enc = link->dc->res_pool->funcs->link_enc_create(
&enc_init_data);
- if( link->link_enc == NULL) {
+ if (link->link_enc == NULL) {
DC_ERROR("Failed to create link encoder!\n");
goto link_enc_create_fail;
}
/* get link settings for video mode timing */
decide_link_settings(stream, &link_settings);
+ if (pipe_ctx->stream->signal == SIGNAL_TYPE_EDP) {
+ /* If link settings are different than current and link already enabled
+ * then need to disable before programming to new rate.
+ */
+ if (link->link_status.link_active &&
+ (link->cur_link_settings.lane_count != link_settings.lane_count ||
+ link->cur_link_settings.link_rate != link_settings.link_rate)) {
+ dp_disable_link_phy(link, pipe_ctx->stream->signal);
+ }
+
+ /*in case it is not on*/
+ link->dc->hwss.edp_power_control(link, true);
+ link->dc->hwss.edp_wait_for_hpd_ready(link, true);
+ }
+
pipe_ctx->stream_res.pix_clk_params.requested_sym_clk =
link_settings.link_rate * LINK_RATE_REF_FREQ_IN_KHZ;
- state->dccg->funcs->update_clocks(state->dccg, state, false);
+ state->clk_mgr->funcs->update_clocks(state->clk_mgr, state, false);
dp_enable_link_phy(
link,
struct pipe_ctx *pipe_ctx)
{
enum dc_status status;
- struct dc_stream_state *stream = pipe_ctx->stream;
- struct dc_link *link = stream->link;
- /*in case it is not on*/
- link->dc->hwss.edp_power_control(link, true);
- link->dc->hwss.edp_wait_for_hpd_ready(link, true);
status = enable_link_dp(state, pipe_ctx);
-
return status;
}
if (link->cur_link_settings.lane_count != LANE_COUNT_UNKNOWN)
return DC_OK;
+ /* clear payload table */
+ dm_helpers_dp_mst_clear_payload_allocation_table(link->ctx, link);
+
/* to make sure the pending down rep can be processed
- * before clear payload table
+ * before enabling the link
*/
dm_helpers_dp_mst_poll_pending_down_reply(link->ctx, link);
- /* clear payload table */
- dm_helpers_dp_mst_clear_payload_allocation_table(link->ctx, link);
-
/* set the sink to MST mode before enabling the link */
dp_enable_mst_on_sink(link, true);
pipe_ctx->stream->signal,
stream->phy_pix_clk);
- if (pipe_ctx->stream->signal == SIGNAL_TYPE_HDMI_TYPE_A)
+ if (dc_is_hdmi_signal(pipe_ctx->stream->signal))
dal_ddc_service_read_scdc_data(link->ddc);
}
}
}
+static uint32_t get_timing_pixel_clock_100hz(const struct dc_crtc_timing *timing)
+{
+
+ uint32_t pxl_clk = timing->pix_clk_100hz;
+
+ if (timing->pixel_encoding == PIXEL_ENCODING_YCBCR420)
+ pxl_clk /= 2;
+ else if (timing->pixel_encoding == PIXEL_ENCODING_YCBCR422)
+ pxl_clk = pxl_clk * 2 / 3;
+
+ if (timing->display_color_depth == COLOR_DEPTH_101010)
+ pxl_clk = pxl_clk * 10 / 8;
+ else if (timing->display_color_depth == COLOR_DEPTH_121212)
+ pxl_clk = pxl_clk * 12 / 8;
+
+ return pxl_clk;
+}
+
static bool dp_active_dongle_validate_timing(
const struct dc_crtc_timing *timing,
const struct dpcd_caps *dpcd_caps)
{
- unsigned int required_pix_clk_100hz = timing->pix_clk_100hz;
const struct dc_dongle_caps *dongle_caps = &dpcd_caps->dongle_caps;
switch (dpcd_caps->dongle_type) {
return false;
}
-
- /* Check Color Depth and Pixel Clock */
- if (timing->pixel_encoding == PIXEL_ENCODING_YCBCR420)
- required_pix_clk_100hz /= 2;
- else if (timing->pixel_encoding == PIXEL_ENCODING_YCBCR422)
- required_pix_clk_100hz = required_pix_clk_100hz * 2 / 3;
-
switch (timing->display_color_depth) {
case COLOR_DEPTH_666:
case COLOR_DEPTH_888:
case COLOR_DEPTH_101010:
if (dongle_caps->dp_hdmi_max_bpc < 10)
return false;
- required_pix_clk_100hz = required_pix_clk_100hz * 10 / 8;
break;
case COLOR_DEPTH_121212:
if (dongle_caps->dp_hdmi_max_bpc < 12)
return false;
- required_pix_clk_100hz = required_pix_clk_100hz * 12 / 8;
break;
-
case COLOR_DEPTH_141414:
case COLOR_DEPTH_161616:
default:
return false;
}
- if (required_pix_clk_100hz > (dongle_caps->dp_hdmi_max_pixel_clk * 10))
+ if (get_timing_pixel_clock_100hz(timing) > (dongle_caps->dp_hdmi_max_pixel_clk_in_khz * 10))
return false;
return true;
return DC_OK;
/* Passive Dongle */
- if (0 != max_pix_clk && timing->pix_clk_100hz > max_pix_clk)
+ if (max_pix_clk != 0 && get_timing_pixel_clock_100hz(timing) > max_pix_clk)
return DC_EXCEED_DONGLE_CAP;
/* Active Dongle*/
static struct fixed31_32 get_pbn_per_slot(struct dc_stream_state *stream)
{
- struct dc_link_settings *link_settings =
- &stream->link->cur_link_settings;
- uint32_t link_rate_in_mbps =
- link_settings->link_rate * LINK_RATE_REF_FREQ_IN_MHZ;
- struct fixed31_32 mbps = dc_fixpt_from_int(
- link_rate_in_mbps * link_settings->lane_count);
-
- return dc_fixpt_div_int(mbps, 54);
+ struct fixed31_32 mbytes_per_sec;
+ uint32_t link_rate_in_mbytes_per_sec = dc_link_bandwidth_kbps(stream->link, &stream->link->cur_link_settings);
+ link_rate_in_mbytes_per_sec /= 8000; /* Kbits to MBytes */
+
+ mbytes_per_sec = dc_fixpt_from_int(link_rate_in_mbytes_per_sec);
+
+ return dc_fixpt_div_int(mbytes_per_sec, 54);
}
static int get_color_depth(enum dc_color_depth color_depth)
uint32_t denominator;
bpc = get_color_depth(pipe_ctx->stream_res.pix_clk_params.color_depth);
- kbps = pipe_ctx->stream_res.pix_clk_params.requested_pix_clk_100hz / 10 * bpc * 3;
+ kbps = dc_bandwidth_in_kbps_from_timing(&pipe_ctx->stream->timing);
/*
* margin 5300ppm + 300ppm ~ 0.6% as per spec, factor is 1.006
struct dc_state *state,
struct pipe_ctx *pipe_ctx)
{
- struct dc *core_dc = pipe_ctx->stream->ctx->dc;
+ struct dc *core_dc = pipe_ctx->stream->ctx->dc;
struct dc_stream_state *stream = pipe_ctx->stream;
enum dc_status status;
DC_LOGGER_INIT(pipe_ctx->stream->ctx->logger);
- if (pipe_ctx->stream->signal != SIGNAL_TYPE_VIRTUAL) {
+ if (!dc_is_virtual_signal(pipe_ctx->stream->signal)) {
stream->link->link_enc->funcs->setup(
stream->link->link_enc,
pipe_ctx->stream->signal);
pipe_ctx->stream_res.stream_enc->funcs->dp_set_stream_attribute(
pipe_ctx->stream_res.stream_enc,
&stream->timing,
- stream->output_color_space);
+ stream->output_color_space,
+ stream->link->dpcd_caps.dprx_feature.bits.SST_SPLIT_SDP_CAP);
- if (dc_is_hdmi_signal(pipe_ctx->stream->signal))
+ if (dc_is_hdmi_tmds_signal(pipe_ctx->stream->signal))
pipe_ctx->stream_res.stream_enc->funcs->hdmi_set_stream_attribute(
pipe_ctx->stream_res.stream_enc,
&stream->timing,
}
}
+uint32_t dc_bandwidth_in_kbps_from_timing(
+ const struct dc_crtc_timing *timing)
+{
+ uint32_t bits_per_channel = 0;
+ uint32_t kbps;
+
+ switch (timing->display_color_depth) {
+ case COLOR_DEPTH_666:
+ bits_per_channel = 6;
+ break;
+ case COLOR_DEPTH_888:
+ bits_per_channel = 8;
+ break;
+ case COLOR_DEPTH_101010:
+ bits_per_channel = 10;
+ break;
+ case COLOR_DEPTH_121212:
+ bits_per_channel = 12;
+ break;
+ case COLOR_DEPTH_141414:
+ bits_per_channel = 14;
+ break;
+ case COLOR_DEPTH_161616:
+ bits_per_channel = 16;
+ break;
+ default:
+ break;
+ }
+
+ ASSERT(bits_per_channel != 0);
+
+ kbps = timing->pix_clk_100hz / 10;
+ kbps *= bits_per_channel;
+
+ if (timing->flags.Y_ONLY != 1) {
+ /*Only YOnly make reduce bandwidth by 1/3 compares to RGB*/
+ kbps *= 3;
+ if (timing->pixel_encoding == PIXEL_ENCODING_YCBCR420)
+ kbps /= 2;
+ else if (timing->pixel_encoding == PIXEL_ENCODING_YCBCR422)
+ kbps = kbps * 2 / 3;
+ }
+
+ return kbps;
+
+}
return ret;
}
-int dc_link_aux_transfer(struct ddc_service *ddc,
- struct aux_payload *payload)
+/* dc_link_aux_transfer_raw() - Attempt to transfer
+ * the given aux payload. This function does not perform
+ * retries or handle error states. The reply is returned
+ * in the payload->reply and the result through
+ * *operation_result. Returns the number of bytes transferred,
+ * or -1 on a failure.
+ */
+int dc_link_aux_transfer_raw(struct ddc_service *ddc,
+ struct aux_payload *payload,
+ enum aux_channel_operation_result *operation_result)
{
- return dce_aux_transfer(ddc, payload);
+ return dce_aux_transfer_raw(ddc, payload, operation_result);
}
+/* dc_link_aux_transfer_with_retries() - Attempt to submit an
+ * aux payload, retrying on timeouts, defers, and busy states
+ * as outlined in the DP spec. Returns true if the request
+ * was successful.
+ *
+ * Unless you want to implement your own retry semantics, this
+ * is probably the one you want.
+ */
bool dc_link_aux_transfer_with_retries(struct ddc_service *ddc,
struct aux_payload *payload)
{
struct dc_link *link,
const struct link_training_settings *lt_settings)
{
- uint8_t rate = (uint8_t)
- (lt_settings->link_settings.link_rate);
+ uint8_t rate;
union down_spread_ctrl downspread = { {0} };
union lane_count_set lane_count_set = { {0} };
- uint8_t link_set_buffer[2];
downspread.raw = (uint8_t)
(lt_settings->link_settings.link_spread);
lane_count_set.bits.POST_LT_ADJ_REQ_GRANTED =
link->dpcd_caps.max_ln_count.bits.POST_LT_ADJ_REQ_SUPPORTED;
- link_set_buffer[0] = rate;
- link_set_buffer[1] = lane_count_set.raw;
-
- core_link_write_dpcd(link, DP_LINK_BW_SET,
- link_set_buffer, 2);
core_link_write_dpcd(link, DP_DOWNSPREAD_CTRL,
&downspread.raw, sizeof(downspread));
+ core_link_write_dpcd(link, DP_LANE_COUNT_SET,
+ &lane_count_set.raw, 1);
+
if (link->dpcd_caps.dpcd_rev.raw >= DPCD_REV_14 &&
- (link->dpcd_caps.link_rate_set >= 1 &&
- link->dpcd_caps.link_rate_set <= 8)) {
+ lt_settings->link_settings.use_link_rate_set == true) {
+ rate = 0;
+ core_link_write_dpcd(link, DP_LINK_BW_SET, &rate, 1);
core_link_write_dpcd(link, DP_LINK_RATE_SET,
- &link->dpcd_caps.link_rate_set, 1);
+ <_settings->link_settings.link_rate_set, 1);
+ } else {
+ rate = (uint8_t) (lt_settings->link_settings.link_rate);
+ core_link_write_dpcd(link, DP_LINK_BW_SET, &rate, 1);
}
- DC_LOG_HW_LINK_TRAINING("%s\n %x rate = %x\n %x lane = %x\n %x spread = %x\n",
- __func__,
- DP_LINK_BW_SET,
- lt_settings->link_settings.link_rate,
- DP_LANE_COUNT_SET,
- lt_settings->link_settings.lane_count,
- DP_DOWNSPREAD_CTRL,
- lt_settings->link_settings.link_spread);
+ if (rate) {
+ DC_LOG_HW_LINK_TRAINING("%s\n %x rate = %x\n %x lane = %x\n %x spread = %x\n",
+ __func__,
+ DP_LINK_BW_SET,
+ lt_settings->link_settings.link_rate,
+ DP_LANE_COUNT_SET,
+ lt_settings->link_settings.lane_count,
+ DP_DOWNSPREAD_CTRL,
+ lt_settings->link_settings.link_spread);
+ } else {
+ DC_LOG_HW_LINK_TRAINING("%s\n %x rate set = %x\n %x lane = %x\n %x spread = %x\n",
+ __func__,
+ DP_LINK_RATE_SET,
+ lt_settings->link_settings.link_rate_set,
+ DP_LANE_COUNT_SET,
+ lt_settings->link_settings.lane_count,
+ DP_DOWNSPREAD_CTRL,
+ lt_settings->link_settings.link_spread);
+ }
}
lt_settings.link_settings.link_rate = link_setting->link_rate;
lt_settings.link_settings.lane_count = link_setting->lane_count;
+ lt_settings.link_settings.use_link_rate_set = link_setting->use_link_rate_set;
+ lt_settings.link_settings.link_rate_set = link_setting->link_rate_set;
/*@todo[vdevulap] move SS to LS, should not be handled by displaypath*/
{
/* Set Default link settings */
struct dc_link_settings max_link_cap = {LANE_COUNT_FOUR, LINK_RATE_HIGH,
- LINK_SPREAD_05_DOWNSPREAD_30KHZ};
+ LINK_SPREAD_05_DOWNSPREAD_30KHZ, false, 0};
/* Higher link settings based on feature supported */
if (link->link_enc->features.flags.bits.IS_HBR2_CAPABLE)
return true;
}
-static uint32_t bandwidth_in_kbps_from_timing(
- const struct dc_crtc_timing *timing)
-{
- uint32_t bits_per_channel = 0;
- uint32_t kbps;
-
- switch (timing->display_color_depth) {
- case COLOR_DEPTH_666:
- bits_per_channel = 6;
- break;
- case COLOR_DEPTH_888:
- bits_per_channel = 8;
- break;
- case COLOR_DEPTH_101010:
- bits_per_channel = 10;
- break;
- case COLOR_DEPTH_121212:
- bits_per_channel = 12;
- break;
- case COLOR_DEPTH_141414:
- bits_per_channel = 14;
- break;
- case COLOR_DEPTH_161616:
- bits_per_channel = 16;
- break;
- default:
- break;
- }
-
- ASSERT(bits_per_channel != 0);
-
- kbps = timing->pix_clk_100hz / 10;
- kbps *= bits_per_channel;
-
- if (timing->flags.Y_ONLY != 1) {
- /*Only YOnly make reduce bandwidth by 1/3 compares to RGB*/
- kbps *= 3;
- if (timing->pixel_encoding == PIXEL_ENCODING_YCBCR420)
- kbps /= 2;
- else if (timing->pixel_encoding == PIXEL_ENCODING_YCBCR422)
- kbps = kbps * 2 / 3;
- }
-
- return kbps;
-
-}
-
-static uint32_t bandwidth_in_kbps_from_link_settings(
- const struct dc_link_settings *link_setting)
-{
- uint32_t link_rate_in_kbps = link_setting->link_rate *
- LINK_RATE_REF_FREQ_IN_KHZ;
-
- uint32_t lane_count = link_setting->lane_count;
- uint32_t kbps = link_rate_in_kbps;
-
- kbps *= lane_count;
- kbps *= 8; /* 8 bits per byte*/
-
- return kbps;
-
-}
-
bool dp_validate_mode_timing(
struct dc_link *link,
const struct dc_crtc_timing *timing)
timing->v_addressable == (uint32_t) 480)
return true;
- /* We always use verified link settings */
- link_setting = &link->verified_link_cap;
+ link_setting = dc_link_get_link_cap(link);
/* TODO: DYNAMIC_VALIDATION needs to be implemented */
/*if (flags.DYNAMIC_VALIDATION == 1 &&
link_setting = &link->verified_link_cap;
*/
- req_bw = bandwidth_in_kbps_from_timing(timing);
- max_bw = bandwidth_in_kbps_from_link_settings(link_setting);
+ req_bw = dc_bandwidth_in_kbps_from_timing(timing);
+ max_bw = dc_link_bandwidth_kbps(link, link_setting);
if (req_bw <= max_bw) {
/* remember the biggest mode here, during
return false;
}
-void decide_link_settings(struct dc_stream_state *stream,
- struct dc_link_settings *link_setting)
+static bool decide_dp_link_settings(struct dc_link *link, struct dc_link_settings *link_setting, uint32_t req_bw)
{
-
struct dc_link_settings initial_link_setting = {
- LANE_COUNT_ONE, LINK_RATE_LOW, LINK_SPREAD_DISABLED};
+ LANE_COUNT_ONE, LINK_RATE_LOW, LINK_SPREAD_DISABLED, false, 0};
struct dc_link_settings current_link_setting =
initial_link_setting;
- struct dc_link *link;
- uint32_t req_bw;
uint32_t link_bw;
- req_bw = bandwidth_in_kbps_from_timing(&stream->timing);
-
- link = stream->link;
-
- /* if preferred is specified through AMDDP, use it, if it's enough
- * to drive the mode
+ /* search for the minimum link setting that:
+ * 1. is supported according to the link training result
+ * 2. could support the b/w requested by the timing
*/
- if (link->preferred_link_setting.lane_count !=
- LANE_COUNT_UNKNOWN &&
- link->preferred_link_setting.link_rate !=
- LINK_RATE_UNKNOWN) {
- *link_setting = link->preferred_link_setting;
- return;
- }
+ while (current_link_setting.link_rate <=
+ link->verified_link_cap.link_rate) {
+ link_bw = dc_link_bandwidth_kbps(
+ link,
+ ¤t_link_setting);
+ if (req_bw <= link_bw) {
+ *link_setting = current_link_setting;
+ return true;
+ }
- /* MST doesn't perform link training for now
- * TODO: add MST specific link training routine
- */
- if (stream->signal == SIGNAL_TYPE_DISPLAY_PORT_MST) {
- *link_setting = link->verified_link_cap;
- return;
+ if (current_link_setting.lane_count <
+ link->verified_link_cap.lane_count) {
+ current_link_setting.lane_count =
+ increase_lane_count(
+ current_link_setting.lane_count);
+ } else {
+ current_link_setting.link_rate =
+ increase_link_rate(
+ current_link_setting.link_rate);
+ current_link_setting.lane_count =
+ initial_link_setting.lane_count;
+ }
}
- /* EDP use the link cap setting */
- if (link->connector_signal == SIGNAL_TYPE_EDP) {
+ return false;
+}
+
+static bool decide_edp_link_settings(struct dc_link *link, struct dc_link_settings *link_setting, uint32_t req_bw)
+{
+ struct dc_link_settings initial_link_setting;
+ struct dc_link_settings current_link_setting;
+ uint32_t link_bw;
+
+ if (link->dpcd_caps.dpcd_rev.raw < DPCD_REV_14 ||
+ link->dpcd_caps.edp_supported_link_rates_count == 0 ||
+ link->dc->config.optimize_edp_link_rate == false) {
*link_setting = link->verified_link_cap;
- return;
+ return true;
}
+ memset(&initial_link_setting, 0, sizeof(initial_link_setting));
+ initial_link_setting.lane_count = LANE_COUNT_ONE;
+ initial_link_setting.link_rate = link->dpcd_caps.edp_supported_link_rates[0];
+ initial_link_setting.link_spread = LINK_SPREAD_DISABLED;
+ initial_link_setting.use_link_rate_set = true;
+ initial_link_setting.link_rate_set = 0;
+ current_link_setting = initial_link_setting;
+
/* search for the minimum link setting that:
* 1. is supported according to the link training result
* 2. could support the b/w requested by the timing
*/
while (current_link_setting.link_rate <=
link->verified_link_cap.link_rate) {
- link_bw = bandwidth_in_kbps_from_link_settings(
+ link_bw = dc_link_bandwidth_kbps(
+ link,
¤t_link_setting);
if (req_bw <= link_bw) {
*link_setting = current_link_setting;
- return;
+ return true;
}
if (current_link_setting.lane_count <
increase_lane_count(
current_link_setting.lane_count);
} else {
- current_link_setting.link_rate =
- increase_link_rate(
- current_link_setting.link_rate);
- current_link_setting.lane_count =
- initial_link_setting.lane_count;
+ if (current_link_setting.link_rate_set < link->dpcd_caps.edp_supported_link_rates_count) {
+ current_link_setting.link_rate_set++;
+ current_link_setting.link_rate =
+ link->dpcd_caps.edp_supported_link_rates[current_link_setting.link_rate_set];
+ current_link_setting.lane_count =
+ initial_link_setting.lane_count;
+ } else
+ break;
}
}
+ return false;
+}
+
+void decide_link_settings(struct dc_stream_state *stream,
+ struct dc_link_settings *link_setting)
+{
+ struct dc_link *link;
+ uint32_t req_bw;
+
+ req_bw = dc_bandwidth_in_kbps_from_timing(&stream->timing);
+
+ link = stream->link;
+
+ /* if preferred is specified through AMDDP, use it, if it's enough
+ * to drive the mode
+ */
+ if (link->preferred_link_setting.lane_count !=
+ LANE_COUNT_UNKNOWN &&
+ link->preferred_link_setting.link_rate !=
+ LINK_RATE_UNKNOWN) {
+ *link_setting = link->preferred_link_setting;
+ return;
+ }
+
+ /* MST doesn't perform link training for now
+ * TODO: add MST specific link training routine
+ */
+ if (stream->signal == SIGNAL_TYPE_DISPLAY_PORT_MST) {
+ *link_setting = link->verified_link_cap;
+ return;
+ }
+
+ if (link->connector_signal == SIGNAL_TYPE_EDP) {
+ if (decide_edp_link_settings(link, link_setting, req_bw))
+ return;
+ } else if (decide_dp_link_settings(link, link_setting, req_bw))
+ return;
BREAK_TO_DEBUGGER();
ASSERT(link->verified_link_cap.lane_count != LANE_COUNT_UNKNOWN);
bool is_dp_active_dongle(const struct dc_link *link)
{
- enum display_dongle_type dongle_type = link->dpcd_caps.dongle_type;
-
- return (dongle_type == DISPLAY_DONGLE_DP_VGA_CONVERTER) ||
- (dongle_type == DISPLAY_DONGLE_DP_DVI_CONVERTER) ||
- (dongle_type == DISPLAY_DONGLE_DP_HDMI_CONVERTER);
+ return link->dpcd_caps.is_branch_dev;
}
static int translate_dpcd_max_bpc(enum dpcd_downstream_port_max_bpc bpc)
return -1;
}
+static void read_dp_device_vendor_id(struct dc_link *link)
+{
+ struct dp_device_vendor_id dp_id;
+
+ /* read IEEE branch device id */
+ core_link_read_dpcd(
+ link,
+ DP_BRANCH_OUI,
+ (uint8_t *)&dp_id,
+ sizeof(dp_id));
+
+ link->dpcd_caps.branch_dev_id =
+ (dp_id.ieee_oui[0] << 16) +
+ (dp_id.ieee_oui[1] << 8) +
+ dp_id.ieee_oui[2];
+
+ memmove(
+ link->dpcd_caps.branch_dev_name,
+ dp_id.ieee_device_id,
+ sizeof(dp_id.ieee_device_id));
+}
+
+
+
static void get_active_converter_info(
uint8_t data, struct dc_link *link)
{
return;
}
+ /* DPCD 0x5 bit 0 = 1, it indicate it's branch device */
+ link->dpcd_caps.is_branch_dev = ds_port.fields.PORT_PRESENT;
+
switch (ds_port.fields.PORT_TYPE) {
case DOWNSTREAM_VGA:
link->dpcd_caps.dongle_type = DISPLAY_DONGLE_DP_VGA_CONVERTER;
hdmi_caps = {.raw = det_caps[3] };
union dwnstream_port_caps_byte2
hdmi_color_caps = {.raw = det_caps[2] };
- link->dpcd_caps.dongle_caps.dp_hdmi_max_pixel_clk =
- det_caps[1] * 25000;
+ link->dpcd_caps.dongle_caps.dp_hdmi_max_pixel_clk_in_khz =
+ det_caps[1] * 2500;
link->dpcd_caps.dongle_caps.is_dp_hdmi_s3d_converter =
hdmi_caps.bits.FRAME_SEQ_TO_FRAME_PACK;
translate_dpcd_max_bpc(
hdmi_color_caps.bits.MAX_BITS_PER_COLOR_COMPONENT);
- if (link->dpcd_caps.dongle_caps.dp_hdmi_max_pixel_clk != 0)
+ if (link->dpcd_caps.dongle_caps.dp_hdmi_max_pixel_clk_in_khz != 0)
link->dpcd_caps.dongle_caps.extendedCapValid = true;
}
ddc_service_set_dongle_type(link->ddc, link->dpcd_caps.dongle_type);
- {
- struct dp_device_vendor_id dp_id;
-
- /* read IEEE branch device id */
- core_link_read_dpcd(
- link,
- DP_BRANCH_OUI,
- (uint8_t *)&dp_id,
- sizeof(dp_id));
-
- link->dpcd_caps.branch_dev_id =
- (dp_id.ieee_oui[0] << 16) +
- (dp_id.ieee_oui[1] << 8) +
- dp_id.ieee_oui[2];
-
- memmove(
- link->dpcd_caps.branch_dev_name,
- dp_id.ieee_device_id,
- sizeof(dp_id.ieee_device_id));
- }
-
{
struct dp_sink_hw_fw_revision dp_hw_fw_revision;
{
uint8_t dpcd_data[DP_ADAPTER_CAP - DP_DPCD_REV + 1];
+ /*Only need to read 1 byte starting from DP_DPRX_FEATURE_ENUMERATION_LIST.
+ */
+ uint8_t dpcd_dprx_data = '\0';
+
struct dp_device_vendor_id sink_id;
union down_stream_port_count down_strm_port_count;
union edp_configuration_cap edp_config_cap;
aux_rd_interval.raw =
dpcd_data[DP_TRAINING_AUX_RD_INTERVAL];
- if (aux_rd_interval.bits.EXT_RECIEVER_CAP_FIELD_PRESENT == 1) {
+ link->dpcd_caps.ext_receiver_cap_field_present =
+ aux_rd_interval.bits.EXT_RECEIVER_CAP_FIELD_PRESENT == 1 ? true:false;
+
+ if (aux_rd_interval.bits.EXT_RECEIVER_CAP_FIELD_PRESENT == 1) {
uint8_t ext_cap_data[16];
memset(ext_cap_data, '\0', sizeof(ext_cap_data));
}
link->dpcd_caps.dpcd_rev.raw =
- dpcd_data[DP_DPCD_REV - DP_DPCD_REV];
+ dpcd_data[DP_DPCD_REV - DP_DPCD_REV];
+
+ if (link->dpcd_caps.dpcd_rev.raw >= 0x14) {
+ for (i = 0; i < read_dpcd_retry_cnt; i++) {
+ status = core_link_read_dpcd(
+ link,
+ DP_DPRX_FEATURE_ENUMERATION_LIST,
+ &dpcd_dprx_data,
+ sizeof(dpcd_dprx_data));
+ if (status == DC_OK)
+ break;
+ }
+
+ link->dpcd_caps.dprx_feature.raw = dpcd_dprx_data;
+
+ if (status != DC_OK)
+ dm_error("%s: Read DPRX caps data failed.\n", __func__);
+ }
+
+ else {
+ link->dpcd_caps.dprx_feature.raw = 0;
+ }
+
+
+ /* Error condition checking...
+ * It is impossible for Sink to report Max Lane Count = 0.
+ * It is possible for Sink to report Max Link Rate = 0, if it is
+ * an eDP device that is reporting specialized link rates in the
+ * SUPPORTED_LINK_RATE table.
+ */
+ if (dpcd_data[DP_MAX_LANE_COUNT - DP_DPCD_REV] == 0)
+ return false;
ds_port.byte = dpcd_data[DP_DOWNSTREAMPORT_PRESENT -
DP_DPCD_REV];
+ read_dp_device_vendor_id(link);
+
get_active_converter_info(ds_port.byte, link);
dp_wa_power_up_0010FA(link, dpcd_data, sizeof(dpcd_data));
void detect_edp_sink_caps(struct dc_link *link)
{
- uint8_t supported_link_rates[16] = {0};
+ uint8_t supported_link_rates[16];
uint32_t entry;
uint32_t link_rate_in_khz;
enum dc_link_rate link_rate = LINK_RATE_UNKNOWN;
retrieve_link_cap(link);
+ link->dpcd_caps.edp_supported_link_rates_count = 0;
+ memset(supported_link_rates, 0, sizeof(supported_link_rates));
- if (link->dpcd_caps.dpcd_rev.raw >= DPCD_REV_14) {
+ if (link->dpcd_caps.dpcd_rev.raw >= DPCD_REV_14 &&
+ link->dc->config.optimize_edp_link_rate) {
// Read DPCD 00010h - 0001Fh 16 bytes at one shot
core_link_read_dpcd(link, DP_SUPPORTED_LINK_RATES,
supported_link_rates, sizeof(supported_link_rates));
- link->dpcd_caps.link_rate_set = 0;
for (entry = 0; entry < 16; entry += 2) {
// DPCD register reports per-lane link rate = 16-bit link rate capability
- // value X 200 kHz. Need multipler to find link rate in kHz.
+ // value X 200 kHz. Need multiplier to find link rate in kHz.
link_rate_in_khz = (supported_link_rates[entry+1] * 0x100 +
supported_link_rates[entry]) * 200;
if (link_rate_in_khz != 0) {
link_rate = linkRateInKHzToLinkRateMultiplier(link_rate_in_khz);
- if (link->reported_link_cap.link_rate < link_rate) {
- link->reported_link_cap.link_rate = link_rate;
- link->dpcd_caps.link_rate_set = entry;
- }
+ link->dpcd_caps.edp_supported_link_rates[link->dpcd_caps.edp_supported_link_rates_count] = link_rate;
+ link->dpcd_caps.edp_supported_link_rates_count++;
}
}
}
enum dc_color_depth color_depth = pipe_ctx->
stream->timing.display_color_depth;
struct bit_depth_reduction_params params;
+ struct output_pixel_processor *opp = pipe_ctx->stream_res.opp;
memset(¶ms, 0, sizeof(params));
{
/* disable bit depth reduction */
pipe_ctx->stream->bit_depth_params = params;
- pipe_ctx->stream_res.opp->funcs->
- opp_program_bit_depth_reduction(pipe_ctx->stream_res.opp, ¶ms);
+ opp->funcs->opp_program_bit_depth_reduction(opp, ¶ms);
if (pipe_ctx->stream_res.tg->funcs->set_test_pattern)
pipe_ctx->stream_res.tg->funcs->set_test_pattern(pipe_ctx->stream_res.tg,
controller_test_pattern, color_depth);
case DP_TEST_PATTERN_VIDEO_MODE:
{
/* restore bitdepth reduction */
- resource_build_bit_depth_reduction_params(pipe_ctx->stream,
- ¶ms);
+ resource_build_bit_depth_reduction_params(pipe_ctx->stream, ¶ms);
pipe_ctx->stream->bit_depth_params = params;
- pipe_ctx->stream_res.opp->funcs->
- opp_program_bit_depth_reduction(pipe_ctx->stream_res.opp, ¶ms);
+ opp->funcs->opp_program_bit_depth_reduction(opp, ¶ms);
if (pipe_ctx->stream_res.tg->funcs->set_test_pattern)
pipe_ctx->stream_res.tg->funcs->set_test_pattern(pipe_ctx->stream_res.tg,
CONTROLLER_DP_TEST_PATTERN_VIDEOMODE,
const struct dc_link_settings *link_settings)
{
struct link_encoder *link_enc = link->link_enc;
+ struct dc *core_dc = link->ctx->dc;
+ struct dmcu *dmcu = core_dc->res_pool->dmcu;
struct pipe_ctx *pipes =
link->dc->current_state->res_ctx.pipe_ctx;
}
}
+ if (dmcu != NULL && dmcu->funcs->lock_phy)
+ dmcu->funcs->lock_phy(dmcu);
+
if (dc_is_dp_sst_signal(signal)) {
link_enc->funcs->enable_dp_output(
link_enc,
link_settings,
clock_source);
}
+
+ if (dmcu != NULL && dmcu->funcs->unlock_phy)
+ dmcu->funcs->unlock_phy(dmcu);
+
link->cur_link_settings = *link_settings;
dp_receiver_power_ctrl(link, true);
void dp_disable_link_phy(struct dc_link *link, enum signal_type signal)
{
+ struct dc *core_dc = link->ctx->dc;
+ struct dmcu *dmcu = core_dc->res_pool->dmcu;
+
if (!link->wa_flags.dp_keep_receiver_powered)
dp_receiver_power_ctrl(link, false);
if (signal == SIGNAL_TYPE_EDP) {
link->link_enc->funcs->disable_output(link->link_enc, signal);
link->dc->hwss.edp_power_control(link, false);
- } else
+ } else {
+ if (dmcu != NULL && dmcu->funcs->lock_phy)
+ dmcu->funcs->lock_phy(dmcu);
+
link->link_enc->funcs->disable_output(link->link_enc, signal);
+ if (dmcu != NULL && dmcu->funcs->unlock_phy)
+ dmcu->funcs->unlock_phy(dmcu);
+ }
+
/* Clear current link setting.*/
memset(&link->cur_link_settings, 0,
sizeof(link->cur_link_settings));
#include "opp.h"
#include "timing_generator.h"
#include "transform.h"
+#include "dccg.h"
+#include "dchubbub.h"
#include "dpp.h"
#include "core_types.h"
#include "set_mode_types.h"
return dc_version;
}
-struct resource_pool *dc_create_resource_pool(
- struct dc *dc,
- int num_virtual_links,
- enum dce_version dc_version,
- struct hw_asic_id asic_id)
+struct resource_pool *dc_create_resource_pool(struct dc *dc,
+ const struct dc_init_data *init_data,
+ enum dce_version dc_version)
{
struct resource_pool *res_pool = NULL;
switch (dc_version) {
case DCE_VERSION_8_0:
res_pool = dce80_create_resource_pool(
- num_virtual_links, dc);
+ init_data->num_virtual_links, dc);
break;
case DCE_VERSION_8_1:
res_pool = dce81_create_resource_pool(
- num_virtual_links, dc);
+ init_data->num_virtual_links, dc);
break;
case DCE_VERSION_8_3:
res_pool = dce83_create_resource_pool(
- num_virtual_links, dc);
+ init_data->num_virtual_links, dc);
break;
case DCE_VERSION_10_0:
res_pool = dce100_create_resource_pool(
- num_virtual_links, dc);
+ init_data->num_virtual_links, dc);
break;
case DCE_VERSION_11_0:
res_pool = dce110_create_resource_pool(
- num_virtual_links, dc, asic_id);
+ init_data->num_virtual_links, dc,
+ init_data->asic_id);
break;
case DCE_VERSION_11_2:
case DCE_VERSION_11_22:
res_pool = dce112_create_resource_pool(
- num_virtual_links, dc);
+ init_data->num_virtual_links, dc);
break;
case DCE_VERSION_12_0:
case DCE_VERSION_12_1:
res_pool = dce120_create_resource_pool(
- num_virtual_links, dc);
+ init_data->num_virtual_links, dc);
break;
#if defined(CONFIG_DRM_AMD_DC_DCN1_0)
#if defined(CONFIG_DRM_AMD_DC_DCN1_01)
case DCN_VERSION_1_01:
#endif
- res_pool = dcn10_create_resource_pool(
- num_virtual_links, dc);
+ res_pool = dcn10_create_resource_pool(init_data, dc);
break;
#endif
if (dc->ctx->dc_bios->funcs->get_firmware_info(
dc->ctx->dc_bios, &fw_info) == BP_RESULT_OK) {
- res_pool->ref_clock_inKhz = fw_info.pll_info.crystal_frequency;
+ res_pool->ref_clocks.xtalin_clock_inKhz = fw_info.pll_info.crystal_frequency;
+
+ if (IS_FPGA_MAXIMUS_DC(dc->ctx->dce_environment)) {
+ // On FPGA these dividers are currently not configured by GDB
+ res_pool->ref_clocks.dccg_ref_clock_inKhz = res_pool->ref_clocks.xtalin_clock_inKhz;
+ res_pool->ref_clocks.dchub_ref_clock_inKhz = res_pool->ref_clocks.xtalin_clock_inKhz;
+ } else if (res_pool->dccg && res_pool->hubbub) {
+ // If DCCG reference frequency cannot be determined (usually means not set to xtalin) then this is a critical error
+ // as this value must be known for DCHUB programming
+ (res_pool->dccg->funcs->get_dccg_ref_freq)(res_pool->dccg,
+ fw_info.pll_info.crystal_frequency,
+ &res_pool->ref_clocks.dccg_ref_clock_inKhz);
+
+ // Similarly, if DCHUB reference frequency cannot be determined, then it is also a critical error
+ (res_pool->hubbub->funcs->get_dchub_ref_freq)(res_pool->hubbub,
+ res_pool->ref_clocks.dccg_ref_clock_inKhz,
+ &res_pool->ref_clocks.dchub_ref_clock_inKhz);
+ } else {
+ // Not all ASICs have DCCG sw component
+ res_pool->ref_clocks.dccg_ref_clock_inKhz = res_pool->ref_clocks.xtalin_clock_inKhz;
+ res_pool->ref_clocks.dchub_ref_clock_inKhz = res_pool->ref_clocks.xtalin_clock_inKhz;
+ }
} else
ASSERT_CRITICAL(false);
}
pool->stream_enc_count++;
}
}
+
dc->caps.dynamic_audio = false;
if (pool->audio_count < pool->stream_enc_count) {
dc->caps.dynamic_audio = true;
struct pipe_ctx *find_idle_secondary_pipe(
struct resource_context *res_ctx,
- const struct resource_pool *pool)
+ const struct resource_pool *pool,
+ const struct pipe_ctx *primary_pipe)
{
int i;
struct pipe_ctx *secondary_pipe = NULL;
/*
- * search backwards for the second pipe to keep pipe
- * assignment more consistent
+ * We add a preferred pipe mapping to avoid the chance that
+ * MPCCs already in use will need to be reassigned to other trees.
+ * For example, if we went with the strict, assign backwards logic:
+ *
+ * (State 1)
+ * Display A on, no surface, top pipe = 0
+ * Display B on, no surface, top pipe = 1
+ *
+ * (State 2)
+ * Display A on, no surface, top pipe = 0
+ * Display B on, surface enable, top pipe = 1, bottom pipe = 5
+ *
+ * (State 3)
+ * Display A on, surface enable, top pipe = 0, bottom pipe = 5
+ * Display B on, surface enable, top pipe = 1, bottom pipe = 4
+ *
+ * The state 2->3 transition requires remapping MPCC 5 from display B
+ * to display A.
+ *
+ * However, with the preferred pipe logic, state 2 would look like:
+ *
+ * (State 2)
+ * Display A on, no surface, top pipe = 0
+ * Display B on, surface enable, top pipe = 1, bottom pipe = 4
+ *
+ * This would then cause 2->3 to not require remapping any MPCCs.
*/
-
- for (i = pool->pipe_count - 1; i >= 0; i--) {
- if (res_ctx->pipe_ctx[i].stream == NULL) {
- secondary_pipe = &res_ctx->pipe_ctx[i];
- secondary_pipe->pipe_idx = i;
- break;
+ if (primary_pipe) {
+ int preferred_pipe_idx = (pool->pipe_count - 1) - primary_pipe->pipe_idx;
+ if (res_ctx->pipe_ctx[preferred_pipe_idx].stream == NULL) {
+ secondary_pipe = &res_ctx->pipe_ctx[preferred_pipe_idx];
+ secondary_pipe->pipe_idx = preferred_pipe_idx;
}
}
+ /*
+ * search backwards for the second pipe to keep pipe
+ * assignment more consistent
+ */
+ if (!secondary_pipe)
+ for (i = pool->pipe_count - 1; i >= 0; i--) {
+ if (res_ctx->pipe_ctx[i].stream == NULL) {
+ secondary_pipe = &res_ctx->pipe_ctx[i];
+ secondary_pipe->pipe_idx = i;
+ break;
+ }
+ }
return secondary_pipe;
}
free_pipe->clock_source = tail_pipe->clock_source;
free_pipe->top_pipe = tail_pipe;
tail_pipe->bottom_pipe = free_pipe;
+ } else if (free_pipe->bottom_pipe && free_pipe->bottom_pipe->plane_state == NULL) {
+ ASSERT(free_pipe->bottom_pipe->stream_res.opp != free_pipe->stream_res.opp);
+ free_pipe->bottom_pipe->plane_state = plane_state;
}
/* assign new surfaces*/
return true;
}
+struct pipe_ctx *dc_res_get_odm_bottom_pipe(struct pipe_ctx *pipe_ctx)
+{
+ struct pipe_ctx *bottom_pipe = pipe_ctx->bottom_pipe;
+
+ /* ODM should only be updated once per otg */
+ if (pipe_ctx->top_pipe)
+ return NULL;
+
+ while (bottom_pipe) {
+ if (bottom_pipe->stream_res.opp != pipe_ctx->stream_res.opp)
+ break;
+ bottom_pipe = bottom_pipe->bottom_pipe;
+ }
+
+ return bottom_pipe;
+}
+
+bool dc_res_is_odm_head_pipe(struct pipe_ctx *pipe_ctx)
+{
+ struct pipe_ctx *top_pipe = pipe_ctx->top_pipe;
+
+ if (!top_pipe)
+ return false;
+ if (top_pipe && top_pipe->stream_res.opp == pipe_ctx->stream_res.opp)
+ return false;
+
+ return true;
+}
+
bool dc_remove_plane_from_context(
const struct dc *dc,
struct dc_stream_state *stream,
/* release pipe for plane*/
for (i = pool->pipe_count - 1; i >= 0; i--) {
- struct pipe_ctx *pipe_ctx;
+ struct pipe_ctx *pipe_ctx = &context->res_ctx.pipe_ctx[i];
- if (context->res_ctx.pipe_ctx[i].plane_state == plane_state) {
- pipe_ctx = &context->res_ctx.pipe_ctx[i];
+ if (pipe_ctx->plane_state == plane_state) {
+ if (dc_res_is_odm_head_pipe(pipe_ctx)) {
+ pipe_ctx->plane_state = NULL;
+ pipe_ctx->bottom_pipe = NULL;
+ continue;
+ }
if (pipe_ctx->top_pipe)
pipe_ctx->top_pipe->bottom_pipe = pipe_ctx->bottom_pipe;
*/
if (!pipe_ctx->top_pipe) {
pipe_ctx->plane_state = NULL;
- pipe_ctx->bottom_pipe = NULL;
- } else {
+ if (!dc_res_get_odm_bottom_pipe(pipe_ctx))
+ pipe_ctx->bottom_pipe = NULL;
+ } else {
memset(pipe_ctx, 0, sizeof(*pipe_ctx));
}
}
for (i = 0; i < MAX_PIPES; i++) {
if (new_ctx->res_ctx.pipe_ctx[i].stream == stream &&
!new_ctx->res_ctx.pipe_ctx[i].top_pipe) {
+ struct pipe_ctx *odm_pipe =
+ dc_res_get_odm_bottom_pipe(&new_ctx->res_ctx.pipe_ctx[i]);
+
del_pipe = &new_ctx->res_ctx.pipe_ctx[i];
ASSERT(del_pipe->stream_res.stream_enc);
dc->res_pool->funcs->remove_stream_from_ctx(dc, new_ctx, stream);
memset(del_pipe, 0, sizeof(*del_pipe));
+ if (odm_pipe)
+ memset(odm_pipe, 0, sizeof(*odm_pipe));
break;
}
struct dc_context *dc_ctx = dc->ctx;
struct pipe_ctx *pipe_ctx = NULL;
int pipe_idx = -1;
+ struct dc_bios *dcb = dc->ctx->dc_bios;
/* TODO Check if this is needed */
/*if (!resource_is_stream_unchanged(old_context, stream)) {
calculate_phy_pix_clks(stream);
+ /* TODO: Check Linux */
+ if (dc->config.allow_seamless_boot_optimization &&
+ !dcb->funcs->is_accelerated_mode(dcb)) {
+ if (dc_validate_seamless_boot_timing(dc, stream->sink, &stream->timing))
+ stream->apply_seamless_boot_optimization = true;
+ }
+
if (stream->apply_seamless_boot_optimization)
pipe_idx = acquire_resource_from_hw_enabled_state(
&context->res_ctx,
const struct dc *dc,
struct dc_state *dst_ctx)
{
- dst_ctx->dccg = dc->res_pool->clk_mgr;
+ dst_ctx->clk_mgr = dc->res_pool->clk_mgr;
}
/**
* Checks HW resource availability and bandwidth requirement.
* @dc: dc struct for this driver
* @new_ctx: state to be validated
+ * @fast_validate: set to true if only yes/no to support matters
*
* Return: DC_OK if the result can be programmed. Otherwise, an error code.
*/
enum dc_status dc_validate_global_state(
struct dc *dc,
- struct dc_state *new_ctx)
+ struct dc_state *new_ctx,
+ bool fast_validate)
{
enum dc_status result = DC_ERROR_UNEXPECTED;
int i, j;
result = resource_build_scaling_params_for_context(dc, new_ctx);
if (result == DC_OK)
- if (!dc->res_pool->funcs->validate_bandwidth(dc, new_ctx))
+ if (!dc->res_pool->funcs->validate_bandwidth(dc, new_ctx, fast_validate))
result = DC_FAIL_BANDWIDTH_VALIDATE;
return result;
*info_packet = stream->vrr_infopacket;
}
+static void set_dp_sdp_info_packet(
+ struct dc_info_packet *info_packet,
+ struct dc_stream_state *stream)
+{
+ /* SPD info packet for custom sdp message */
+
+ /* Return if false. If true,
+ * set the corresponding bit in the info packet
+ */
+ if (!stream->dpsdp_infopacket.valid)
+ return;
+
+ *info_packet = stream->dpsdp_infopacket;
+}
+
static void set_hdr_static_info_packet(
struct dc_info_packet *info_packet,
struct dc_stream_state *stream)
info->spd.valid = false;
info->hdrsmd.valid = false;
info->vsc.valid = false;
+ info->dpsdp.valid = false;
signal = pipe_ctx->stream->signal;
set_spd_info_packet(&info->spd, pipe_ctx->stream);
set_hdr_static_info_packet(&info->hdrsmd, pipe_ctx->stream);
+
+ set_dp_sdp_info_packet(&info->dpsdp, pipe_ctx->stream);
}
patch_gamut_packet_checksum(&info->gamut);
if (!tg->funcs->validate_timing(tg, &stream->timing))
res = DC_FAIL_CONTROLLER_VALIDATE;
- if (res == DC_OK)
+ if (res == DC_OK) {
if (!link->link_enc->funcs->validate_output_with_stream(
link->link_enc, stream))
res = DC_FAIL_ENC_VALIDATE;
+ }
/* TODO: validate audio ASIC caps, encoder */
#include "resource.h"
#include "ipp.h"
#include "timing_generator.h"
+#if defined(CONFIG_DRM_AMD_DC_DCN1_0)
+#include "dcn10/dcn10_hw_sequencer.h"
+#endif
#define DC_LOGGER dc->ctx->logger
return stream;
}
+struct dc_stream_state *dc_copy_stream(const struct dc_stream_state *stream)
+{
+ struct dc_stream_state *new_stream;
+
+ new_stream = kzalloc(sizeof(struct dc_stream_state), GFP_KERNEL);
+ if (!new_stream)
+ return NULL;
+
+ memcpy(new_stream, stream, sizeof(struct dc_stream_state));
+
+ if (new_stream->sink)
+ dc_sink_retain(new_stream->sink);
+
+ if (new_stream->out_transfer_func)
+ dc_transfer_func_retain(new_stream->out_transfer_func);
+
+ kref_init(&new_stream->refcount);
+
+ return new_stream;
+}
+
/**
* dc_stream_get_status_from_state - Get stream status from given dc state
* @state: DC state to find the stream status in
return dc_stream_get_status_from_state(dc->current_state, stream);
}
+static void delay_cursor_until_vupdate(struct pipe_ctx *pipe_ctx, struct dc *dc)
+{
+#if defined(CONFIG_DRM_AMD_DC_DCN1_0)
+ unsigned int vupdate_line;
+ unsigned int lines_to_vupdate, us_to_vupdate, vpos, nvpos;
+ struct dc_stream_state *stream = pipe_ctx->stream;
+ unsigned int us_per_line;
+
+ if (stream->ctx->asic_id.chip_family == FAMILY_RV &&
+ ASIC_REV_IS_RAVEN(stream->ctx->asic_id.hw_internal_rev)) {
+
+ vupdate_line = get_vupdate_offset_from_vsync(pipe_ctx);
+ if (!dc_stream_get_crtc_position(dc, &stream, 1, &vpos, &nvpos))
+ return;
+
+ if (vpos >= vupdate_line)
+ return;
+
+ us_per_line = stream->timing.h_total * 10000 / stream->timing.pix_clk_100hz;
+ lines_to_vupdate = vupdate_line - vpos;
+ us_to_vupdate = lines_to_vupdate * us_per_line;
+
+ /* 70 us is a conservative estimate of cursor update time*/
+ if (us_to_vupdate < 70)
+ udelay(us_to_vupdate);
+ }
+#endif
+}
+
/**
* dc_stream_set_cursor_attributes() - Update cursor attributes and set cursor surface address
*/
if (!pipe_to_program) {
pipe_to_program = pipe_ctx;
+
+ delay_cursor_until_vupdate(pipe_ctx, core_dc);
core_dc->hwss.pipe_control_lock(core_dc, pipe_to_program, true);
}
(!pipe_ctx->plane_res.mi && !pipe_ctx->plane_res.hubp) ||
!pipe_ctx->plane_state ||
(!pipe_ctx->plane_res.xfm && !pipe_ctx->plane_res.dpp) ||
- !pipe_ctx->plane_res.ipp)
+ (!pipe_ctx->plane_res.ipp && !pipe_ctx->plane_res.dpp))
continue;
if (!pipe_to_program) {
pipe_to_program = pipe_ctx;
+
+ delay_cursor_until_vupdate(pipe_ctx, core_dc);
core_dc->hwss.pipe_control_lock(core_dc, pipe_to_program, true);
}
return 0;
}
+static void build_dp_sdp_info_frame(struct pipe_ctx *pipe_ctx,
+ const uint8_t *custom_sdp_message,
+ unsigned int sdp_message_size)
+{
+ uint8_t i;
+ struct encoder_info_frame *info = &pipe_ctx->stream_res.encoder_info_frame;
+
+ /* set valid info */
+ info->dpsdp.valid = true;
+
+ /* set sdp message header */
+ info->dpsdp.hb0 = custom_sdp_message[0]; /* package id */
+ info->dpsdp.hb1 = custom_sdp_message[1]; /* package type */
+ info->dpsdp.hb2 = custom_sdp_message[2]; /* package specific byte 0 any data */
+ info->dpsdp.hb3 = custom_sdp_message[3]; /* package specific byte 0 any data */
+
+ /* set sdp message data */
+ for (i = 0; i < 32; i++)
+ info->dpsdp.sb[i] = (custom_sdp_message[i+4]);
+
+}
+
+static void invalid_dp_sdp_info_frame(struct pipe_ctx *pipe_ctx)
+{
+ struct encoder_info_frame *info = &pipe_ctx->stream_res.encoder_info_frame;
+
+ /* in-valid info */
+ info->dpsdp.valid = false;
+}
+
+bool dc_stream_send_dp_sdp(const struct dc_stream_state *stream,
+ const uint8_t *custom_sdp_message,
+ unsigned int sdp_message_size)
+{
+ int i;
+ struct dc *core_dc;
+ struct resource_context *res_ctx;
+
+ if (stream == NULL) {
+ dm_error("DC: dc_stream is NULL!\n");
+ return false;
+ }
+
+ core_dc = stream->ctx->dc;
+ res_ctx = &core_dc->current_state->res_ctx;
+
+ for (i = 0; i < MAX_PIPES; i++) {
+ struct pipe_ctx *pipe_ctx = &res_ctx->pipe_ctx[i];
+
+ if (pipe_ctx->stream != stream)
+ continue;
+
+ build_dp_sdp_info_frame(pipe_ctx, custom_sdp_message, sdp_message_size);
+
+ core_dc->hwss.update_info_frame(pipe_ctx);
+
+ invalid_dp_sdp_info_frame(pipe_ctx);
+ }
+
+ return true;
+}
+
bool dc_stream_get_scanoutpos(const struct dc_stream_state *stream,
uint32_t *v_blank_start,
uint32_t *v_blank_end,
if (core_dc->current_state == NULL)
return NULL;
+ /* Find the current plane state and set its pending bit to false */
+ for (i = 0; i < core_dc->res_pool->pipe_count; i++) {
+ struct pipe_ctx *pipe_ctx =
+ &core_dc->current_state->res_ctx.pipe_ctx[i];
+
+ if (pipe_ctx->plane_state != plane_state)
+ continue;
+
+ pipe_ctx->plane_state->status.is_flip_pending = false;
+
+ break;
+ }
+
for (i = 0; i < core_dc->res_pool->pipe_count; i++) {
struct pipe_ctx *pipe_ctx =
&core_dc->current_state->res_ctx.pipe_ctx[i];
#include "inc/hw/dmcu.h"
#include "dml/display_mode_lib.h"
-#define DC_VER "3.2.17"
+#define DC_VER "3.2.27"
#define MAX_SURFACES 3
+#define MAX_PLANES 6
#define MAX_STREAMS 6
#define MAX_SINKS_PER_LINK 4
struct dmcu_version dmcu_version;
};
+enum dc_plane_type {
+ DC_PLANE_TYPE_INVALID,
+ DC_PLANE_TYPE_DCE_RGB,
+ DC_PLANE_TYPE_DCE_UNDERLAY,
+ DC_PLANE_TYPE_DCN_UNIVERSAL,
+};
+
+struct dc_plane_cap {
+ enum dc_plane_type type;
+ uint32_t blends_with_above : 1;
+ uint32_t blends_with_below : 1;
+ uint32_t per_pixel_alpha : 1;
+ struct {
+ uint32_t argb8888 : 1;
+ uint32_t nv12 : 1;
+ uint32_t fp16 : 1;
+ } pixel_format_support;
+ // max upscaling factor x1000
+ // upscaling factors are always >= 1
+ // for example, 1080p -> 8K is 4.0, or 4000 raw value
+ struct {
+ uint32_t argb8888;
+ uint32_t nv12;
+ uint32_t fp16;
+ } max_upscale_factor;
+ // max downscale factor x1000
+ // downscale factors are always <= 1
+ // for example, 8K -> 1080p is 0.25, or 250 raw value
+ struct {
+ uint32_t argb8888;
+ uint32_t nv12;
+ uint32_t fp16;
+ } max_downscale_factor;
+};
+
struct dc_caps {
uint32_t max_streams;
uint32_t max_links;
bool force_dp_tps4_for_cp2520;
bool disable_dp_clk_share;
bool psp_setup_panel_mode;
+ struct dc_plane_cap planes[MAX_PLANES];
};
struct dc_dcc_surface_param {
bool gpu_vm_support;
bool disable_disp_pll_sharing;
bool fbc_support;
+ bool optimize_edp_link_rate;
+ bool disable_fractional_pwm;
+ bool allow_seamless_boot_optimization;
+ bool power_down_display_on_boot;
};
enum visual_confirm {
int fclk_khz;
int phyclk_khz;
int dramclk_khz;
-};
+ bool p_state_change_support;
+};
+
+struct dc_bw_validation_profile {
+ bool enable;
+
+ unsigned long long total_ticks;
+ unsigned long long voltage_level_ticks;
+ unsigned long long watermark_ticks;
+ unsigned long long rq_dlg_ticks;
+
+ unsigned long long total_count;
+ unsigned long long skip_fast_count;
+ unsigned long long skip_pass_count;
+ unsigned long long skip_fail_count;
+};
+
+#define BW_VAL_TRACE_SETUP() \
+ unsigned long long end_tick = 0; \
+ unsigned long long voltage_level_tick = 0; \
+ unsigned long long watermark_tick = 0; \
+ unsigned long long start_tick = dc->debug.bw_val_profile.enable ? \
+ dm_get_timestamp(dc->ctx) : 0
+
+#define BW_VAL_TRACE_COUNT() \
+ if (dc->debug.bw_val_profile.enable) \
+ dc->debug.bw_val_profile.total_count++
+
+#define BW_VAL_TRACE_SKIP(status) \
+ if (dc->debug.bw_val_profile.enable) { \
+ if (!voltage_level_tick) \
+ voltage_level_tick = dm_get_timestamp(dc->ctx); \
+ dc->debug.bw_val_profile.skip_ ## status ## _count++; \
+ }
+
+#define BW_VAL_TRACE_END_VOLTAGE_LEVEL() \
+ if (dc->debug.bw_val_profile.enable) \
+ voltage_level_tick = dm_get_timestamp(dc->ctx)
+
+#define BW_VAL_TRACE_END_WATERMARKS() \
+ if (dc->debug.bw_val_profile.enable) \
+ watermark_tick = dm_get_timestamp(dc->ctx)
+
+#define BW_VAL_TRACE_FINISH() \
+ if (dc->debug.bw_val_profile.enable) { \
+ end_tick = dm_get_timestamp(dc->ctx); \
+ dc->debug.bw_val_profile.total_ticks += end_tick - start_tick; \
+ dc->debug.bw_val_profile.voltage_level_ticks += voltage_level_tick - start_tick; \
+ if (watermark_tick) { \
+ dc->debug.bw_val_profile.watermark_ticks += watermark_tick - voltage_level_tick; \
+ dc->debug.bw_val_profile.rq_dlg_ticks += end_tick - watermark_tick; \
+ } \
+ }
struct dc_debug_options {
enum visual_confirm visual_confirm;
bool skip_detection_link_training;
unsigned int force_odm_combine; //bit vector based on otg inst
unsigned int force_fclk_khz;
+ bool disable_tri_buf;
+ struct dc_bw_validation_profile bw_val_profile;
};
struct dc_debug_data {
uint32_t auxErrorCount;
};
+struct dc_bounding_box_overrides {
+ int sr_exit_time_ns;
+ int sr_enter_plus_exit_time_ns;
+ int urgent_latency_ns;
+ int percent_of_ideal_drambw;
+ int dram_clock_change_latency_ns;
+};
+
struct dc_state;
struct resource_pool;
struct dce_hwseq;
struct dc_cap_funcs cap_funcs;
struct dc_config config;
struct dc_debug_options debug;
+ struct dc_bounding_box_overrides bb_overrides;
struct dc_context *ctx;
uint8_t link_count;
struct hw_sequencer_funcs hwss;
struct dce_hwseq *hwseq;
+ /* Require to optimize clocks and bandwidth for added/removed planes */
bool optimized_required;
+ /* Require to maintain clocks and bandwidth for UEFI enabled HW */
+ bool optimize_seamless_boot;
+
/* FBC compressor */
struct compressor *fbc_compressor;
struct hw_asic_id asic_id;
void *driver; /* ctx */
struct cgs_device *cgs_device;
+ struct dc_bounding_box_overrides bb_overrides;
int num_virtual_links;
/*
uint8_t plane_count;
};
-bool dc_validate_seamless_boot_timing(struct dc *dc,
+bool dc_validate_seamless_boot_timing(const struct dc *dc,
const struct dc_sink *sink,
struct dc_crtc_timing *crtc_timing);
void get_clock_requirements_for_state(struct dc_state *state, struct AsicStateEx *info);
+/*
+ * fast_validate: we return after determining if we can support the new state,
+ * but before we populate the programming info
+ */
enum dc_status dc_validate_global_state(
struct dc *dc,
- struct dc_state *new_ctx);
+ struct dc_state *new_ctx,
+ bool fast_validate);
void dc_resource_state_construct(
bool dc_commit_state(struct dc *dc, struct dc_state *context);
-struct dc_state *dc_create_state(void);
+struct dc_state *dc_create_state(struct dc *dc);
+struct dc_state *dc_copy_state(struct dc_state *src_ctx);
void dc_retain_state(struct dc_state *context);
void dc_release_state(struct dc_state *context);
union dpcd_rev dpcd_rev;
union max_lane_count max_ln_count;
union max_down_spread max_down_spread;
+ union dprx_feature dprx_feature;
+
+ /* valid only for eDP v1.4 or higher*/
+ uint8_t edp_supported_link_rates_count;
+ enum dc_link_rate edp_supported_link_rates[8];
/* dongle type (DP converter, CV smart dongle) */
enum display_dongle_type dongle_type;
+ /* branch device or sink device */
+ bool is_branch_dev;
/* Dongle's downstream count. */
union sink_count sink_count;
/* If dongle_type == DISPLAY_DONGLE_DP_HDMI_CONVERTER,
int8_t branch_dev_name[6];
int8_t branch_hw_revision;
int8_t branch_fw_revision[2];
- uint8_t link_rate_set;
bool allow_invalid_MSA_timing_param;
bool panel_mode_edp;
bool dpcd_display_control_capable;
+ bool ext_receiver_cap_field_present;
};
#include "dc_link.h"
AUX_TRANSACTION_REPLY_AUX_ACK = 0x00,
AUX_TRANSACTION_REPLY_AUX_NACK = 0x01,
AUX_TRANSACTION_REPLY_AUX_DEFER = 0x02,
+ AUX_TRANSACTION_REPLY_I2C_OVER_AUX_NACK = 0x04,
+ AUX_TRANSACTION_REPLY_I2C_OVER_AUX_DEFER = 0x08,
AUX_TRANSACTION_REPLY_I2C_ACK = 0x00,
AUX_TRANSACTION_REPLY_I2C_NACK = 0x10,
enum dc_lane_count lane_count;
enum dc_link_rate link_rate;
enum dc_link_spread link_spread;
+ bool use_link_rate_set;
+ uint8_t link_rate_set;
};
struct dc_lane_settings {
uint8_t raw;
};
+union dprx_feature {
+ struct {
+ uint8_t GTC_CAP:1; // bit 0: DP 1.3+
+ uint8_t SST_SPLIT_SDP_CAP:1; // bit 1: DP 1.4
+ uint8_t AV_SYNC_CAP:1; // bit 2: DP 1.3+
+ uint8_t VSC_SDP_COLORIMETRY_SUPPORTED:1; // bit 3: DP 1.3+
+ uint8_t VSC_EXT_VESA_SDP_SUPPORTED:1; // bit 4: DP 1.4
+ uint8_t VSC_EXT_VESA_SDP_CHAINING_SUPPORTED:1; // bit 5: DP 1.4
+ uint8_t VSC_EXT_CEA_SDP_SUPPORTED:1; // bit 6: DP 1.4
+ uint8_t VSC_EXT_CEA_SDP_CHAINING_SUPPORTED:1; // bit 7: DP 1.4
+ } bits;
+ uint8_t raw;
+};
+
union training_aux_rd_interval {
struct {
uint8_t TRAINIG_AUX_RD_INTERVAL:7;
- uint8_t EXT_RECIEVER_CAP_FIELD_PRESENT:1;
+ uint8_t EXT_RECEIVER_CAP_FIELD_PRESENT:1;
} bits;
uint8_t raw;
};
field_value_mask->mask = field_value_mask->mask | mask;
}
-uint32_t generic_reg_update_ex(const struct dc_context *ctx,
- uint32_t addr, uint32_t reg_val, int n,
+static void set_reg_field_values(struct dc_reg_value_masks *field_value_mask,
+ uint32_t addr, int n,
uint8_t shift1, uint32_t mask1, uint32_t field_value1,
- ...)
+ va_list ap)
{
- struct dc_reg_value_masks field_value_mask = {0};
uint32_t shift, mask, field_value;
int i = 1;
- va_list ap;
- va_start(ap, field_value1);
-
/* gather all bits value/mask getting updated in this register */
- set_reg_field_value_masks(&field_value_mask,
+ set_reg_field_value_masks(field_value_mask,
field_value1, mask1, shift1);
while (i < n) {
mask = va_arg(ap, uint32_t);
field_value = va_arg(ap, uint32_t);
- set_reg_field_value_masks(&field_value_mask,
+ set_reg_field_value_masks(field_value_mask,
field_value, mask, shift);
i++;
}
+}
+
+uint32_t generic_reg_update_ex(const struct dc_context *ctx,
+ uint32_t addr, int n,
+ uint8_t shift1, uint32_t mask1, uint32_t field_value1,
+ ...)
+{
+ struct dc_reg_value_masks field_value_mask = {0};
+ uint32_t reg_val;
+ va_list ap;
+
+ va_start(ap, field_value1);
+
+ set_reg_field_values(&field_value_mask, addr, n, shift1, mask1,
+ field_value1, ap);
+
+ va_end(ap);
+
+ /* mmio write directly */
+ reg_val = dm_read_reg(ctx, addr);
+ reg_val = (reg_val & ~field_value_mask.mask) | field_value_mask.value;
+ dm_write_reg(ctx, addr, reg_val);
+ return reg_val;
+}
+
+uint32_t generic_reg_set_ex(const struct dc_context *ctx,
+ uint32_t addr, uint32_t reg_val, int n,
+ uint8_t shift1, uint32_t mask1, uint32_t field_value1,
+ ...)
+{
+ struct dc_reg_value_masks field_value_mask = {0};
+ va_list ap;
+
+ va_start(ap, field_value1);
+
+ set_reg_field_values(&field_value_mask, addr, n, shift1, mask1,
+ field_value1, ap);
+
va_end(ap);
return reg_val;
}
+uint32_t dm_read_reg_func(
+ const struct dc_context *ctx,
+ uint32_t address,
+ const char *func_name)
+{
+ uint32_t value;
+#ifdef DM_CHECK_ADDR_0
+ if (address == 0) {
+ DC_ERR("invalid register read; address = 0\n");
+ return 0;
+ }
+#endif
+ value = cgs_read_register(ctx->cgs_device, address);
+ trace_amdgpu_dc_rreg(&ctx->perf_trace->read_count, address, value);
+
+ return value;
+}
+
uint32_t generic_reg_get(const struct dc_context *ctx, uint32_t addr,
uint8_t shift, uint32_t mask, uint32_t *field_value)
{
}
*/
-uint32_t generic_reg_wait(const struct dc_context *ctx,
+void generic_reg_wait(const struct dc_context *ctx,
uint32_t addr, uint32_t shift, uint32_t mask, uint32_t condition_value,
unsigned int delay_between_poll_us, unsigned int time_out_num_tries,
const char *func_name, int line)
DC_LOG_DC("REG_WAIT taking a while: %dms in %s line:%d\n",
delay_between_poll_us * i / 1000,
func_name, line);
- return reg_val;
+ return;
}
}
if (!IS_FPGA_MAXIMUS_DC(ctx->dce_environment))
BREAK_TO_DEBUGGER();
-
- return reg_val;
}
void generic_write_indirect_reg(const struct dc_context *ctx,
/* MST record stream using this link */
struct link_flags {
bool dp_keep_receiver_powered;
+ bool dp_skip_DID2;
} wa_flags;
struct link_mst_stream_allocation_table mst_stream_alloc_table;
const struct link_training_settings *p_link_settings,
const unsigned char *p_custom_pattern,
unsigned int cust_pattern_size);
+uint32_t dc_link_bandwidth_kbps(
+ const struct dc_link *link,
+ const struct dc_link_settings *link_setting);
+
+const struct dc_link_settings *dc_link_get_link_cap(
+ const struct dc_link *link);
bool dc_submit_i2c(
struct dc *dc,
uint32_t link_index,
struct i2c_command *cmd);
+uint32_t dc_bandwidth_in_kbps_from_timing(
+ const struct dc_crtc_timing *timing);
#endif /* DC_LINK_H_ */
struct dc_info_packet vrr_infopacket;
struct dc_info_packet vsc_infopacket;
struct dc_info_packet vsp_infopacket;
+ struct dc_info_packet dpsdp_infopacket;
struct rect src; /* composition area */
struct rect dst; /* stream addressable area */
*/
uint32_t dc_stream_get_vblank_counter(const struct dc_stream_state *stream);
+/*
+ * Send dp sdp message.
+ */
+bool dc_stream_send_dp_sdp(const struct dc_stream_state *stream,
+ const uint8_t *custom_sdp_message,
+ unsigned int sdp_message_size);
+
/* TODO: Return parsed values rather than direct register read
* This has a dependency on the caller (amdgpu_display_get_crtc_scanoutpos)
* being refactored properly to be dce-specific
*/
struct dc_stream_state *dc_create_stream_for_sink(struct dc_sink *dc_sink);
+struct dc_stream_state *dc_copy_stream(const struct dc_stream_state *stream);
+
void update_stream_signal(struct dc_stream_state *stream, struct dc_sink *sink);
void dc_stream_retain(struct dc_stream_state *dc_stream);
};
-#define DC_MAX_EDID_BUFFER_SIZE 512
+#define DC_MAX_EDID_BUFFER_SIZE 1024
#define EDID_BLOCK_SIZE 128
#define MAX_SURFACE_NUM 4
#define NUM_PIXEL_FORMATS 10
bool is_dp_hdmi_ycbcr422_converter;
bool is_dp_hdmi_ycbcr420_converter;
uint32_t dp_hdmi_max_bpc;
- uint32_t dp_hdmi_max_pixel_clk;
+ uint32_t dp_hdmi_max_pixel_clk_in_khz;
};
/* Scaling format */
enum scaling_transformation {
unsigned char psr_version;
unsigned int psr_rfb_setup_time;
bool psr_exit_link_training_required;
-
bool psr_frame_capture_indication_req;
unsigned int psr_sdp_transmit_line_num_deadline;
+ bool allow_smu_optimizations;
};
union dmcu_psr_level {
* continuing powerd own
*/
unsigned int frame_delay;
+ bool allow_smu_optimizations;
};
struct colorspace_transform {
(request->action == I2CAUX_TRANSACTION_ACTION_I2C_WRITE_MOT)));
if (REG(AUXN_IMPCAL)) {
/* clear_aux_error */
- REG_UPDATE_SEQ(AUXN_IMPCAL, AUXN_CALOUT_ERROR_AK,
- 1,
- 0);
+ REG_UPDATE_SEQ_2(AUXN_IMPCAL,
+ AUXN_CALOUT_ERROR_AK, 1,
+ AUXN_CALOUT_ERROR_AK, 0);
- REG_UPDATE_SEQ(AUXP_IMPCAL, AUXP_CALOUT_ERROR_AK,
- 1,
- 0);
+ REG_UPDATE_SEQ_2(AUXP_IMPCAL,
+ AUXP_CALOUT_ERROR_AK, 1,
+ AUXP_CALOUT_ERROR_AK, 0);
/* force_default_calibrate */
- REG_UPDATE_1BY1_2(AUXN_IMPCAL,
+ REG_UPDATE_SEQ_2(AUXN_IMPCAL,
AUXN_IMPCAL_ENABLE, 1,
AUXN_IMPCAL_OVERRIDE_ENABLE, 0);
/* bug? why AUXN update EN and OVERRIDE_EN 1 by 1 while AUX P toggles OVERRIDE? */
- REG_UPDATE_SEQ(AUXP_IMPCAL, AUXP_IMPCAL_OVERRIDE_ENABLE,
- 1,
- 0);
+ REG_UPDATE_SEQ_2(AUXP_IMPCAL,
+ AUXP_IMPCAL_OVERRIDE_ENABLE, 1,
+ AUXP_IMPCAL_OVERRIDE_ENABLE, 0);
}
REG_UPDATE(AUX_INTERRUPT_CONTROL, AUX_SW_DONE_ACK, 1);
if (!bytes_replied)
return -1;
- REG_UPDATE_1BY1_3(AUX_SW_DATA,
+ REG_UPDATE_SEQ_3(AUX_SW_DATA,
AUX_SW_INDEX, 0,
AUX_SW_AUTOINCREMENT_DISABLE, 1,
AUX_SW_DATA_RW, 1);
*returned_bytes = 0;
/* poll to make sure that SW_DONE is asserted */
- value = REG_WAIT(AUX_SW_STATUS, AUX_SW_DONE, 1,
+ REG_WAIT(AUX_SW_STATUS, AUX_SW_DONE, 1,
10, aux110->timeout_period/10);
+ value = REG_READ(AUX_SW_STATUS);
/* in case HPD is LOW, exit AUX transaction */
if ((value & AUX_SW_STATUS__AUX_SW_HPD_DISCON_MASK))
return AUX_CHANNEL_OPERATION_FAILED_HPD_DISCON;
struct dce_aux *engine,
struct ddc *ddc)
{
-
enum gpio_result result;
if (!is_engine_available(engine))
return I2CAUX_TRANSACTION_ACTION_DP_READ;
}
-int dce_aux_transfer(struct ddc_service *ddc,
- struct aux_payload *payload)
+int dce_aux_transfer_raw(struct ddc_service *ddc,
+ struct aux_payload *payload,
+ enum aux_channel_operation_result *operation_result)
{
struct ddc *ddc_pin = ddc->ddc_pin;
struct dce_aux *aux_engine;
- enum aux_channel_operation_result operation_result;
struct aux_request_transaction_data aux_req;
struct aux_reply_transaction_data aux_rep;
uint8_t returned_bytes = 0;
memset(&aux_rep, 0, sizeof(aux_rep));
aux_engine = ddc->ctx->dc->res_pool->engines[ddc_pin->pin_data->en];
- acquire(aux_engine, ddc_pin);
+ if (!acquire(aux_engine, ddc_pin))
+ return -1;
if (payload->i2c_over_aux)
aux_req.type = AUX_TRANSACTION_TYPE_I2C;
aux_req.data = payload->data;
submit_channel_request(aux_engine, &aux_req);
- operation_result = get_channel_status(aux_engine, &returned_bytes);
-
- switch (operation_result) {
- case AUX_CHANNEL_OPERATION_SUCCEEDED:
- res = read_channel_reply(aux_engine, payload->length,
- payload->data, payload->reply,
- &status);
- break;
- case AUX_CHANNEL_OPERATION_FAILED_HPD_DISCON:
- res = 0;
- break;
- case AUX_CHANNEL_OPERATION_FAILED_REASON_UNKNOWN:
- case AUX_CHANNEL_OPERATION_FAILED_INVALID_REPLY:
- case AUX_CHANNEL_OPERATION_FAILED_TIMEOUT:
+ *operation_result = get_channel_status(aux_engine, &returned_bytes);
+
+ if (*operation_result == AUX_CHANNEL_OPERATION_SUCCEEDED) {
+ read_channel_reply(aux_engine, payload->length,
+ payload->data, payload->reply,
+ &status);
+ res = returned_bytes;
+ } else {
res = -1;
- break;
}
+
release_engine(aux_engine);
return res;
}
-#define AUX_RETRY_MAX 7
+#define AUX_MAX_RETRIES 7
+#define AUX_MAX_DEFER_RETRIES 7
+#define AUX_MAX_I2C_DEFER_RETRIES 7
+#define AUX_MAX_INVALID_REPLY_RETRIES 2
+#define AUX_MAX_TIMEOUT_RETRIES 3
bool dce_aux_transfer_with_retries(struct ddc_service *ddc,
struct aux_payload *payload)
int i, ret = 0;
uint8_t reply;
bool payload_reply = true;
+ enum aux_channel_operation_result operation_result;
+ int aux_ack_retries = 0,
+ aux_defer_retries = 0,
+ aux_i2c_defer_retries = 0,
+ aux_timeout_retries = 0,
+ aux_invalid_reply_retries = 0;
if (!payload->reply) {
payload_reply = false;
payload->reply = &reply;
}
- for (i = 0; i < AUX_RETRY_MAX; i++) {
- ret = dce_aux_transfer(ddc, payload);
-
- if (ret >= 0) {
- if (*payload->reply == 0) {
- if (!payload_reply)
- payload->reply = NULL;
- return true;
+ for (i = 0; i < AUX_MAX_RETRIES; i++) {
+ ret = dce_aux_transfer_raw(ddc, payload, &operation_result);
+ switch (operation_result) {
+ case AUX_CHANNEL_OPERATION_SUCCEEDED:
+ aux_timeout_retries = 0;
+ aux_invalid_reply_retries = 0;
+
+ switch (*payload->reply) {
+ case AUX_TRANSACTION_REPLY_AUX_ACK:
+ if (!payload->write && payload->length != ret) {
+ if (++aux_ack_retries >= AUX_MAX_RETRIES)
+ goto fail;
+ else
+ udelay(300);
+ } else
+ return true;
+ break;
+
+ case AUX_TRANSACTION_REPLY_AUX_DEFER:
+ case AUX_TRANSACTION_REPLY_I2C_OVER_AUX_NACK:
+ case AUX_TRANSACTION_REPLY_I2C_OVER_AUX_DEFER:
+ if (++aux_defer_retries >= AUX_MAX_DEFER_RETRIES)
+ goto fail;
+ break;
+
+ case AUX_TRANSACTION_REPLY_I2C_DEFER:
+ aux_defer_retries = 0;
+ if (++aux_i2c_defer_retries >= AUX_MAX_I2C_DEFER_RETRIES)
+ goto fail;
+ break;
+
+ case AUX_TRANSACTION_REPLY_AUX_NACK:
+ case AUX_TRANSACTION_REPLY_HPD_DISCON:
+ default:
+ goto fail;
}
- }
+ break;
+
+ case AUX_CHANNEL_OPERATION_FAILED_INVALID_REPLY:
+ if (++aux_invalid_reply_retries >= AUX_MAX_INVALID_REPLY_RETRIES)
+ goto fail;
+ else
+ udelay(400);
+ break;
+
+ case AUX_CHANNEL_OPERATION_FAILED_TIMEOUT:
+ if (++aux_timeout_retries >= AUX_MAX_TIMEOUT_RETRIES)
+ goto fail;
+ else {
+ /*
+ * DP 1.4, 2.8.2: AUX Transaction Response/Reply Timeouts
+ * According to the DP spec there should be 3 retries total
+ * with a 400us wait inbetween each. Hardware already waits
+ * for 550us therefore no wait is required here.
+ */
+ }
+ break;
- udelay(1000);
+ case AUX_CHANNEL_OPERATION_FAILED_HPD_DISCON:
+ case AUX_CHANNEL_OPERATION_FAILED_REASON_UNKNOWN:
+ default:
+ goto fail;
+ }
}
+
+fail:
+ if (!payload_reply)
+ payload->reply = NULL;
return false;
}
struct dce_aux *aux_engine,
struct ddc *ddc);
-int dce_aux_transfer(struct ddc_service *ddc,
- struct aux_payload *cmd);
+int dce_aux_transfer_raw(struct ddc_service *ddc,
+ struct aux_payload *cmd,
+ enum aux_channel_operation_result *operation_result);
bool dce_aux_transfer_with_retries(struct ddc_service *ddc,
struct aux_payload *cmd);
* all required clocks
*/
for (i = clk_mgr_dce->max_clks_state; i >= DM_PP_CLOCKS_STATE_ULTRA_LOW; i--)
- if (context->bw.dce.dispclk_khz >
+ if (context->bw_ctx.bw.dce.dispclk_khz >
clk_mgr_dce->max_clks_by_state[i].display_clk_khz
|| max_pix_clk >
clk_mgr_dce->max_clks_by_state[i].pixel_clk_khz)
if (low_req_clk > clk_mgr_dce->max_clks_state) {
/* set max clock state for high phyclock, invalid on exceeding display clock */
if (clk_mgr_dce->max_clks_by_state[clk_mgr_dce->max_clks_state].display_clk_khz
- < context->bw.dce.dispclk_khz)
+ < context->bw_ctx.bw.dce.dispclk_khz)
low_req_clk = DM_PP_CLOCKS_STATE_INVALID;
else
low_req_clk = clk_mgr_dce->max_clks_state;
struct dm_pp_display_configuration *pp_display_cfg = &context->pp_display_cfg;
pp_display_cfg->all_displays_in_sync =
- context->bw.dce.all_displays_in_sync;
+ context->bw_ctx.bw.dce.all_displays_in_sync;
pp_display_cfg->nb_pstate_switch_disable =
- context->bw.dce.nbp_state_change_enable == false;
+ context->bw_ctx.bw.dce.nbp_state_change_enable == false;
pp_display_cfg->cpu_cc6_disable =
- context->bw.dce.cpuc_state_change_enable == false;
+ context->bw_ctx.bw.dce.cpuc_state_change_enable == false;
pp_display_cfg->cpu_pstate_disable =
- context->bw.dce.cpup_state_change_enable == false;
+ context->bw_ctx.bw.dce.cpup_state_change_enable == false;
pp_display_cfg->cpu_pstate_separation_time =
- context->bw.dce.blackout_recovery_time_us;
+ context->bw_ctx.bw.dce.blackout_recovery_time_us;
- pp_display_cfg->min_memory_clock_khz = context->bw.dce.yclk_khz
+ pp_display_cfg->min_memory_clock_khz = context->bw_ctx.bw.dce.yclk_khz
/ MEMORY_TYPE_MULTIPLIER_CZ;
pp_display_cfg->min_engine_clock_khz = determine_sclk_from_bounding_box(
dc,
- context->bw.dce.sclk_khz);
+ context->bw_ctx.bw.dce.sclk_khz);
/*
* As workaround for >4x4K lightup set dcfclock to min_engine_clock value.
pp_display_cfg->min_engine_clock_khz : 0;
pp_display_cfg->min_engine_clock_deep_sleep_khz
- = context->bw.dce.sclk_deep_sleep_khz;
+ = context->bw_ctx.bw.dce.sclk_deep_sleep_khz;
pp_display_cfg->avail_mclk_switch_time_us =
dce110_get_min_vblank_time_us(context);
{
struct dce_clk_mgr *clk_mgr_dce = TO_DCE_CLK_MGR(clk_mgr);
struct dm_pp_power_level_change_request level_change_req;
- int patched_disp_clk = context->bw.dce.dispclk_khz;
+ int patched_disp_clk = context->bw_ctx.bw.dce.dispclk_khz;
/*TODO: W/A for dal3 linux, investigate why this works */
if (!clk_mgr_dce->dfs_bypass_active)
{
struct dce_clk_mgr *clk_mgr_dce = TO_DCE_CLK_MGR(clk_mgr);
struct dm_pp_power_level_change_request level_change_req;
- int patched_disp_clk = context->bw.dce.dispclk_khz;
+ int patched_disp_clk = context->bw_ctx.bw.dce.dispclk_khz;
/*TODO: W/A for dal3 linux, investigate why this works */
if (!clk_mgr_dce->dfs_bypass_active)
}
if (should_set_clock(safe_to_lower, patched_disp_clk, clk_mgr->clks.dispclk_khz)) {
- context->bw.dce.dispclk_khz = dce_set_clock(clk_mgr, patched_disp_clk);
+ context->bw_ctx.bw.dce.dispclk_khz = dce_set_clock(clk_mgr, patched_disp_clk);
clk_mgr->clks.dispclk_khz = patched_disp_clk;
}
dce11_pplib_apply_display_requirements(clk_mgr->ctx->dc, context);
{
struct dce_clk_mgr *clk_mgr_dce = TO_DCE_CLK_MGR(clk_mgr);
struct dm_pp_power_level_change_request level_change_req;
- int patched_disp_clk = context->bw.dce.dispclk_khz;
+ int patched_disp_clk = context->bw_ctx.bw.dce.dispclk_khz;
/*TODO: W/A for dal3 linux, investigate why this works */
if (!clk_mgr_dce->dfs_bypass_active)
struct dce_clk_mgr *clk_mgr_dce = TO_DCE_CLK_MGR(clk_mgr);
struct dm_pp_clock_for_voltage_req clock_voltage_req = {0};
int max_pix_clk = get_max_pixel_clock_for_all_paths(context);
- int patched_disp_clk = context->bw.dce.dispclk_khz;
+ int patched_disp_clk = context->bw_ctx.bw.dce.dispclk_khz;
/*TODO: W/A for dal3 linux, investigate why this works */
if (!clk_mgr_dce->dfs_bypass_active)
}
static bool get_pixel_clk_frequency_100hz(
- struct clock_source *clock_source,
+ const struct clock_source *clock_source,
unsigned int inst,
unsigned int *pixel_clk_khz)
{
#define PSR_SET_WAITLOOP 0x31
#define MCP_INIT_DMCU 0x88
#define MCP_INIT_IRAM 0x89
+#define MCP_SYNC_PHY_LOCK 0x90
+#define MCP_SYNC_PHY_UNLOCK 0x91
+#define MCP_BL_SET_PWM_FRAC 0x6A /* Enable or disable Fractional PWM */
#define MASTER_COMM_CNTL_REG__MASTER_COMM_INTERRUPT_MASK 0x00000001L
static bool dce_dmcu_init(struct dmcu *dmcu)
link->link_enc->funcs->psr_program_secondary_packet(link->link_enc,
psr_context->sdpTransmitLineNumDeadline);
- if (psr_context->psr_level.bits.SKIP_SMU_NOTIFICATION)
- REG_UPDATE(SMU_INTERRUPT_CONTROL, DC_SMU_INT_ENABLE, 1);
-
/* waitDMCUReadyForCmd */
REG_WAIT(MASTER_COMM_CNTL_REG, MASTER_COMM_INTERRUPT, 0,
dmcu_wait_reg_ready_interval,
IRAM_RD_ADDR_AUTO_INC, 0);
}
+static void dcn10_dmcu_enable_fractional_pwm(struct dmcu *dmcu,
+ uint32_t fractional_pwm)
+{
+ struct dce_dmcu *dmcu_dce = TO_DCE_DMCU(dmcu);
+
+ /* Wait until microcontroller is ready to process interrupt */
+ REG_WAIT(MASTER_COMM_CNTL_REG, MASTER_COMM_INTERRUPT, 0, 100, 800);
+
+ /* Set PWM fractional enable/disable */
+ REG_WRITE(MASTER_COMM_DATA_REG1, fractional_pwm);
+
+ /* Set command to enable or disable fractional PWM microcontroller */
+ REG_UPDATE(MASTER_COMM_CMD_REG, MASTER_COMM_CMD_REG_BYTE0,
+ MCP_BL_SET_PWM_FRAC);
+
+ /* Notify microcontroller of new command */
+ REG_UPDATE(MASTER_COMM_CNTL_REG, MASTER_COMM_INTERRUPT, 1);
+
+ /* Ensure command has been executed before continuing */
+ REG_WAIT(MASTER_COMM_CNTL_REG, MASTER_COMM_INTERRUPT, 0, 100, 800);
+}
+
static bool dcn10_dmcu_init(struct dmcu *dmcu)
{
struct dce_dmcu *dmcu_dce = TO_DCE_DMCU(dmcu);
+ const struct dc_config *config = &dmcu->ctx->dc->config;
bool status = false;
/* Definition of DC_DMCU_SCRATCH
if (dmcu->dmcu_state == DMCU_RUNNING) {
/* Retrieve and cache the DMCU firmware version. */
dcn10_get_dmcu_version(dmcu);
+
+ /* Initialize DMCU to use fractional PWM or not */
+ dcn10_dmcu_enable_fractional_pwm(dmcu,
+ (config->disable_fractional_pwm == false) ? 1 : 0);
status = true;
- } else
+ } else {
status = false;
+ }
break;
case DMCU_RUNNING:
link->link_enc->funcs->psr_program_secondary_packet(link->link_enc,
psr_context->sdpTransmitLineNumDeadline);
- if (psr_context->psr_level.bits.SKIP_SMU_NOTIFICATION)
+ if (psr_context->allow_smu_optimizations)
REG_UPDATE(SMU_INTERRUPT_CONTROL, DC_SMU_INT_ENABLE, 1);
/* waitDMCUReadyForCmd */
psr_context->psrFrameCaptureIndicationReq;
masterCmdData1.bits.aux_chan = psr_context->channel;
masterCmdData1.bits.aux_repeat = psr_context->aux_repeats;
+ masterCmdData1.bits.allow_smu_optimizations = psr_context->allow_smu_optimizations;
dm_write_reg(dmcu->ctx, REG(MASTER_COMM_DATA_REG1),
masterCmdData1.u32All);
dm_write_reg(dmcu->ctx, REG(MASTER_COMM_DATA_REG3),
masterCmdData3.u32All);
+
/* setDMCUParam_Cmd */
REG_UPDATE(MASTER_COMM_CMD_REG,
MASTER_COMM_CMD_REG_BYTE0, PSR_SET);
return true;
}
-#endif
+#endif //(CONFIG_DRM_AMD_DC_DCN1_0)
static const struct dmcu_funcs dce_funcs = {
.dmcu_init = dce_dmcu_init,
******************************************************************/
union dce_dmcu_psr_config_data_reg1 {
struct {
- unsigned int timehyst_frames:8; /*[7:0]*/
- unsigned int hyst_lines:7; /*[14:8]*/
- unsigned int rfb_update_auto_en:1; /*[15:15]*/
- unsigned int dp_port_num:3; /*[18:16]*/
- unsigned int dcp_sel:3; /*[21:19]*/
- unsigned int phy_type:1; /*[22:22]*/
- unsigned int frame_cap_ind:1; /*[23:23]*/
- unsigned int aux_chan:3; /*[26:24]*/
- unsigned int aux_repeat:4; /*[30:27]*/
- unsigned int reserved:1; /*[31:31]*/
+ unsigned int timehyst_frames:8; /*[7:0]*/
+ unsigned int hyst_lines:7; /*[14:8]*/
+ unsigned int rfb_update_auto_en:1; /*[15:15]*/
+ unsigned int dp_port_num:3; /*[18:16]*/
+ unsigned int dcp_sel:3; /*[21:19]*/
+ unsigned int phy_type:1; /*[22:22]*/
+ unsigned int frame_cap_ind:1; /*[23:23]*/
+ unsigned int aux_chan:3; /*[26:24]*/
+ unsigned int aux_repeat:4; /*[30:27]*/
+ unsigned int allow_smu_optimizations:1; /*[31:31]*/
} bits;
unsigned int u32All;
};
struct {
unsigned int psr_level:16; /*[15:0]*/
unsigned int link_rate:4; /*[19:16]*/
- unsigned int reserved:12; /*[31:20]*/
+ unsigned int reserved:12; /*[31:20]*/
} bits;
unsigned int u32All;
};
}
+static bool is_engine_available(struct dce_i2c_hw *dce_i2c_hw)
+{
+ unsigned int arbitrate;
+
+ REG_GET(DC_I2C_ARBITRATION, DC_I2C_REG_RW_CNTL_STATUS, &arbitrate);
+ if (arbitrate == DC_I2C_REG_RW_CNTL_STATUS_DMCU_ONLY)
+ return false;
+ return true;
+}
+
struct dce_i2c_hw *acquire_i2c_hw_engine(
struct resource_pool *pool,
struct ddc *ddc)
if (!dce_i2c_hw)
return NULL;
- if (pool->i2c_hw_buffer_in_use)
+ if (pool->i2c_hw_buffer_in_use || !is_engine_available(dce_i2c_hw))
return NULL;
do {
enum dc_i2c_status {
DC_I2C_STATUS__DC_I2C_STATUS_IDLE,
DC_I2C_STATUS__DC_I2C_STATUS_USED_BY_SW,
- DC_I2C_STATUS__DC_I2C_STATUS_USED_BY_HW
+ DC_I2C_STATUS__DC_I2C_STATUS_USED_BY_HW,
+ DC_I2C_REG_RW_CNTL_STATUS_DMCU_ONLY = 2,
};
enum dc_i2c_arbitration {
I2C_SF(DC_I2C_DATA, DC_I2C_DATA, mask_sh),\
I2C_SF(DC_I2C_DATA, DC_I2C_INDEX, mask_sh),\
I2C_SF(DC_I2C_DATA, DC_I2C_INDEX_WRITE, mask_sh),\
- I2C_SF(MICROSECOND_TIME_BASE_DIV, XTAL_REF_DIV, mask_sh)
+ I2C_SF(MICROSECOND_TIME_BASE_DIV, XTAL_REF_DIV, mask_sh),\
+ I2C_SF(DC_I2C_ARBITRATION, DC_I2C_REG_RW_CNTL_STATUS, mask_sh)
#define I2C_COMMON_MASK_SH_LIST_DCE110(mask_sh)\
I2C_COMMON_MASK_SH_LIST_DCE_COMMON_BASE(mask_sh),\
uint8_t DC_I2C_INDEX;
uint8_t DC_I2C_INDEX_WRITE;
uint8_t XTAL_REF_DIV;
+ uint8_t DC_I2C_REG_RW_CNTL_STATUS;
};
struct dce_i2c_mask {
uint32_t DC_I2C_INDEX;
uint32_t DC_I2C_INDEX_WRITE;
uint32_t XTAL_REF_DIV;
+ uint32_t DC_I2C_REG_RW_CNTL_STATUS;
};
struct dce_i2c_registers {
static void dce110_stream_encoder_dp_set_stream_attribute(
struct stream_encoder *enc,
struct dc_crtc_timing *crtc_timing,
- enum dc_color_space output_color_space)
+ enum dc_color_space output_color_space,
+ uint32_t enable_sdp_splitting)
{
#if defined(CONFIG_DRM_AMD_DC_DCN1_0)
uint32_t h_active_start;
uint64_t m_vid_l = n_vid;
- m_vid_l *= param->pixel_clk_khz;
+ m_vid_l *= param->timing.pix_clk_100hz / 10;
m_vid_l = div_u64(m_vid_l,
param->link_settings.link_rate
* LINK_RATE_REF_FREQ_IN_KHZ);
.num_ddc = 6,
};
+static const struct dc_plane_cap plane_cap = {
+ .type = DC_PLANE_TYPE_DCE_RGB,
+
+ .pixel_format_support = {
+ .argb8888 = true,
+ .nv12 = false,
+ .fp16 = false
+ },
+
+ .max_upscale_factor = {
+ .argb8888 = 16000,
+ .nv12 = 1,
+ .fp16 = 1
+ },
+
+ .max_downscale_factor = {
+ .argb8888 = 250,
+ .nv12 = 1,
+ .fp16 = 1
+ }
+};
+
#define CTX ctx
#define REG(reg) mm ## reg
bool dce100_validate_bandwidth(
struct dc *dc,
- struct dc_state *context)
+ struct dc_state *context,
+ bool fast_validate)
{
int i;
bool at_least_one_pipe = false;
if (at_least_one_pipe) {
/* TODO implement when needed but for now hardcode max value*/
- context->bw.dce.dispclk_khz = 681000;
- context->bw.dce.yclk_khz = 250000 * MEMORY_TYPE_MULTIPLIER_CZ;
+ context->bw_ctx.bw.dce.dispclk_khz = 681000;
+ context->bw_ctx.bw.dce.yclk_khz = 250000 * MEMORY_TYPE_MULTIPLIER_CZ;
} else {
- context->bw.dce.dispclk_khz = 0;
- context->bw.dce.yclk_khz = 0;
+ context->bw_ctx.bw.dce.dispclk_khz = 0;
+ context->bw_ctx.bw.dce.yclk_khz = 0;
}
return true;
dc->caps.max_planes = pool->base.pipe_count;
+ for (i = 0; i < dc->caps.max_planes; ++i)
+ dc->caps.planes[i] = plane_cap;
+
if (!resource_construct(num_virtual_links, dc, &pool->base,
&res_create_funcs))
goto res_create_fail;
void dce110_update_info_frame(struct pipe_ctx *pipe_ctx)
{
- bool is_hdmi;
+ bool is_hdmi_tmds;
bool is_dp;
ASSERT(pipe_ctx->stream);
if (pipe_ctx->stream_res.stream_enc == NULL)
return; /* this is not root pipe */
- is_hdmi = dc_is_hdmi_signal(pipe_ctx->stream->signal);
+ is_hdmi_tmds = dc_is_hdmi_tmds_signal(pipe_ctx->stream->signal);
is_dp = dc_is_dp_signal(pipe_ctx->stream->signal);
- if (!is_hdmi && !is_dp)
+ if (!is_hdmi_tmds && !is_dp)
return;
- if (is_hdmi)
+ if (is_hdmi_tmds)
pipe_ctx->stream_res.stream_enc->funcs->update_hdmi_info_packets(
pipe_ctx->stream_res.stream_enc,
&pipe_ctx->stream_res.encoder_info_frame);
edp_receiver_ready_T9(link);
}
+// Static helper function which calls the correct function
+// based on pp_smu version
+static void set_pme_wa_enable_by_version(struct dc *dc)
+{
+ struct pp_smu_funcs *pp_smu = NULL;
+
+ if (dc->res_pool->pp_smu)
+ pp_smu = dc->res_pool->pp_smu;
+
+ if (pp_smu) {
+ if (pp_smu->ctx.ver == PP_SMU_VER_RV && pp_smu->rv_funcs.set_pme_wa_enable)
+ pp_smu->rv_funcs.set_pme_wa_enable(&(pp_smu->ctx));
+ }
+}
+
void dce110_enable_audio_stream(struct pipe_ctx *pipe_ctx)
{
- struct dc *core_dc = pipe_ctx->stream->ctx->dc;
/* notify audio driver for audio modes of monitor */
- struct pp_smu_funcs_rv *pp_smu = core_dc->res_pool->pp_smu;
+ struct dc *core_dc = pipe_ctx->stream->ctx->dc;
+ struct pp_smu_funcs *pp_smu = NULL;
unsigned int i, num_audio = 1;
+ if (core_dc->res_pool->pp_smu)
+ pp_smu = core_dc->res_pool->pp_smu;
+
if (pipe_ctx->stream_res.audio) {
for (i = 0; i < MAX_PIPES; i++) {
/*current_state not updated yet*/
pipe_ctx->stream_res.audio->funcs->az_enable(pipe_ctx->stream_res.audio);
- if (num_audio >= 1 && pp_smu != NULL && pp_smu->set_pme_wa_enable != NULL)
+ if (num_audio >= 1 && pp_smu != NULL)
/*this is the first audio. apply the PME w/a in order to wake AZ from D3*/
- pp_smu->set_pme_wa_enable(&pp_smu->pp_smu);
+ set_pme_wa_enable_by_version(core_dc);
/* un-mute audio */
/* TODO: audio should be per stream rather than per link */
pipe_ctx->stream_res.stream_enc->funcs->audio_mute_control(
- pipe_ctx->stream_res.stream_enc, false);
+ pipe_ctx->stream_res.stream_enc, false);
}
}
void dce110_disable_audio_stream(struct pipe_ctx *pipe_ctx, int option)
{
struct dc *dc = pipe_ctx->stream->ctx->dc;
+ struct pp_smu_funcs *pp_smu = NULL;
pipe_ctx->stream_res.stream_enc->funcs->audio_mute_control(
pipe_ctx->stream_res.stream_enc, true);
if (pipe_ctx->stream_res.audio) {
- struct pp_smu_funcs_rv *pp_smu = dc->res_pool->pp_smu;
+ if (dc->res_pool->pp_smu)
+ pp_smu = dc->res_pool->pp_smu;
if (option != KEEP_ACQUIRED_RESOURCE ||
- !dc->debug.az_endpoint_mute_only) {
+ !dc->debug.az_endpoint_mute_only)
/*only disalbe az_endpoint if power down or free*/
pipe_ctx->stream_res.audio->funcs->az_disable(pipe_ctx->stream_res.audio);
- }
if (dc_is_dp_signal(pipe_ctx->stream->signal))
pipe_ctx->stream_res.stream_enc->funcs->dp_audio_disable(
update_audio_usage(&dc->current_state->res_ctx, dc->res_pool, pipe_ctx->stream_res.audio, false);
pipe_ctx->stream_res.audio = NULL;
}
- if (pp_smu != NULL && pp_smu->set_pme_wa_enable != NULL)
+ if (pp_smu != NULL)
/*this is the first audio. apply the PME w/a in order to wake AZ from D3*/
- pp_smu->set_pme_wa_enable(&pp_smu->pp_smu);
+ set_pme_wa_enable_by_version(dc);
/* TODO: notify audio driver for if audio modes list changed
* add audio mode list change flag */
struct dc_link *link = stream->link;
struct dc *dc = pipe_ctx->stream->ctx->dc;
- if (dc_is_hdmi_signal(pipe_ctx->stream->signal))
+ if (dc_is_hdmi_tmds_signal(pipe_ctx->stream->signal))
pipe_ctx->stream_res.stream_enc->funcs->stop_hdmi_info_packets(
pipe_ctx->stream_res.stream_enc);
struct dc_link *link = stream->link;
/* only 3 items below are used by unblank */
- params.pixel_clk_khz = pipe_ctx->stream->timing.pix_clk_100hz / 10;
+ params.timing = pipe_ctx->stream->timing;
params.link_settings.link_rate = link_settings->link_rate;
if (dc_is_dp_signal(pipe_ctx->stream->signal))
if (pipe_ctx->stream->signal == SIGNAL_TYPE_DISPLAY_PORT ||
pipe_ctx->stream->signal == SIGNAL_TYPE_DISPLAY_PORT_MST) {
audio_output->pll_info.dp_dto_source_clock_in_khz =
- state->dccg->funcs->get_dp_ref_clk_frequency(
- state->dccg);
+ state->clk_mgr->funcs->get_dp_ref_clk_frequency(
+ state->clk_mgr);
}
audio_output->pll_info.feed_back_divider =
pipe_ctx->stream_res.tg->funcs->set_static_screen_control(
pipe_ctx->stream_res.tg, event_triggers);
- if (pipe_ctx->stream->signal != SIGNAL_TYPE_VIRTUAL)
+ if (!dc_is_virtual_signal(pipe_ctx->stream->signal))
pipe_ctx->stream_res.stream_enc->funcs->dig_connect_to_otg(
pipe_ctx->stream_res.stream_enc,
pipe_ctx->stream_res.tg->inst);
pipe_ctx->stream_res.opp,
COLOR_SPACE_YCBCR601,
stream->timing.display_color_depth,
- pipe_ctx->stream->signal);
+ stream->signal);
pipe_ctx->stream_res.opp->funcs->opp_program_fmt(
pipe_ctx->stream_res.opp,
}
}
+ if (dc->hwss.init_pipes)
+ dc->hwss.init_pipes(dc, context);
+
if (edp_link) {
/* this seems to cause blank screens on DCE8 */
if ((dc->ctx->dce_version == DCE_VERSION_8_0) ||
dc->bw_vbios->blackout_duration, pipe_ctx->stream);
pipe_ctx->plane_res.mi->funcs->mem_input_program_display_marks(
pipe_ctx->plane_res.mi,
- context->bw.dce.nbp_state_change_wm_ns[num_pipes],
- context->bw.dce.stutter_exit_wm_ns[num_pipes],
- context->bw.dce.stutter_entry_wm_ns[num_pipes],
- context->bw.dce.urgent_wm_ns[num_pipes],
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[num_pipes],
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[num_pipes],
+ context->bw_ctx.bw.dce.stutter_entry_wm_ns[num_pipes],
+ context->bw_ctx.bw.dce.urgent_wm_ns[num_pipes],
total_dest_line_time_ns);
if (i == underlay_idx) {
num_pipes++;
pipe_ctx->plane_res.mi->funcs->mem_input_program_chroma_display_marks(
pipe_ctx->plane_res.mi,
- context->bw.dce.nbp_state_change_wm_ns[num_pipes],
- context->bw.dce.stutter_exit_wm_ns[num_pipes],
- context->bw.dce.urgent_wm_ns[num_pipes],
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[num_pipes],
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[num_pipes],
+ context->bw_ctx.bw.dce.urgent_wm_ns[num_pipes],
total_dest_line_time_ns);
}
num_pipes++;
struct mem_input *mi = pipe_ctx->plane_res.mi;
struct dc_cursor_mi_param param = {
.pixel_clk_khz = pipe_ctx->stream->timing.pix_clk_100hz / 10,
- .ref_clk_khz = pipe_ctx->stream->ctx->dc->res_pool->ref_clock_inKhz,
+ .ref_clk_khz = pipe_ctx->stream->ctx->dc->res_pool->ref_clocks.xtalin_clock_inKhz,
.viewport = pipe_ctx->plane_res.scl_data.viewport,
.h_scale_ratio = pipe_ctx->plane_res.scl_data.ratios.horz,
.v_scale_ratio = pipe_ctx->plane_res.scl_data.ratios.vert,
.num_ddc = 3,
};
+static const struct dc_plane_cap plane_cap = {
+ .type = DC_PLANE_TYPE_DCE_RGB,
+ .blends_with_below = true,
+ .blends_with_above = true,
+ .per_pixel_alpha = 1,
+
+ .pixel_format_support = {
+ .argb8888 = true,
+ .nv12 = false,
+ .fp16 = false
+ },
+
+ .max_upscale_factor = {
+ .argb8888 = 16000,
+ .nv12 = 1,
+ .fp16 = 1
+ },
+
+ .max_downscale_factor = {
+ .argb8888 = 250,
+ .nv12 = 1,
+ .fp16 = 1
+ }
+};
+
+static const struct dc_plane_cap underlay_plane_cap = {
+ .type = DC_PLANE_TYPE_DCE_UNDERLAY,
+ .blends_with_above = true,
+ .per_pixel_alpha = 1,
+
+ .pixel_format_support = {
+ .argb8888 = false,
+ .nv12 = true,
+ .fp16 = false
+ },
+
+ .max_upscale_factor = {
+ .argb8888 = 1,
+ .nv12 = 16000,
+ .fp16 = 1
+ },
+
+ .max_downscale_factor = {
+ .argb8888 = 1,
+ .nv12 = 250,
+ .fp16 = 1
+ }
+};
+
#define CTX ctx
#define REG(reg) mm ## reg
static bool dce110_validate_bandwidth(
struct dc *dc,
- struct dc_state *context)
+ struct dc_state *context,
+ bool fast_validate)
{
bool result = false;
dc->bw_vbios,
context->res_ctx.pipe_ctx,
dc->res_pool->pipe_count,
- &context->bw.dce))
+ &context->bw_ctx.bw.dce))
result = true;
if (!result)
context->streams[0]->timing.v_addressable,
context->streams[0]->timing.pix_clk_100hz / 10);
- if (memcmp(&dc->current_state->bw.dce,
- &context->bw.dce, sizeof(context->bw.dce))) {
+ if (memcmp(&dc->current_state->bw_ctx.bw.dce,
+ &context->bw_ctx.bw.dce, sizeof(context->bw_ctx.bw.dce))) {
DC_LOG_BANDWIDTH_CALCS(
"%s: finish,\n"
"sclk: %d sclk_sleep: %d yclk: %d blackout_recovery_time_us: %d\n"
,
__func__,
- context->bw.dce.nbp_state_change_wm_ns[0].b_mark,
- context->bw.dce.nbp_state_change_wm_ns[0].a_mark,
- context->bw.dce.urgent_wm_ns[0].b_mark,
- context->bw.dce.urgent_wm_ns[0].a_mark,
- context->bw.dce.stutter_exit_wm_ns[0].b_mark,
- context->bw.dce.stutter_exit_wm_ns[0].a_mark,
- context->bw.dce.nbp_state_change_wm_ns[1].b_mark,
- context->bw.dce.nbp_state_change_wm_ns[1].a_mark,
- context->bw.dce.urgent_wm_ns[1].b_mark,
- context->bw.dce.urgent_wm_ns[1].a_mark,
- context->bw.dce.stutter_exit_wm_ns[1].b_mark,
- context->bw.dce.stutter_exit_wm_ns[1].a_mark,
- context->bw.dce.nbp_state_change_wm_ns[2].b_mark,
- context->bw.dce.nbp_state_change_wm_ns[2].a_mark,
- context->bw.dce.urgent_wm_ns[2].b_mark,
- context->bw.dce.urgent_wm_ns[2].a_mark,
- context->bw.dce.stutter_exit_wm_ns[2].b_mark,
- context->bw.dce.stutter_exit_wm_ns[2].a_mark,
- context->bw.dce.stutter_mode_enable,
- context->bw.dce.cpuc_state_change_enable,
- context->bw.dce.cpup_state_change_enable,
- context->bw.dce.nbp_state_change_enable,
- context->bw.dce.all_displays_in_sync,
- context->bw.dce.dispclk_khz,
- context->bw.dce.sclk_khz,
- context->bw.dce.sclk_deep_sleep_khz,
- context->bw.dce.yclk_khz,
- context->bw.dce.blackout_recovery_time_us);
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[0].b_mark,
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[0].a_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[0].b_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[0].a_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[0].b_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[0].a_mark,
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[1].b_mark,
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[1].a_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[1].b_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[1].a_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[1].b_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[1].a_mark,
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[2].b_mark,
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[2].a_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[2].b_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[2].a_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[2].b_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[2].a_mark,
+ context->bw_ctx.bw.dce.stutter_mode_enable,
+ context->bw_ctx.bw.dce.cpuc_state_change_enable,
+ context->bw_ctx.bw.dce.cpup_state_change_enable,
+ context->bw_ctx.bw.dce.nbp_state_change_enable,
+ context->bw_ctx.bw.dce.all_displays_in_sync,
+ context->bw_ctx.bw.dce.dispclk_khz,
+ context->bw_ctx.bw.dce.sclk_khz,
+ context->bw_ctx.bw.dce.sclk_deep_sleep_khz,
+ context->bw_ctx.bw.dce.yclk_khz,
+ context->bw_ctx.bw.dce.blackout_recovery_time_us);
}
return result;
}
dc->caps.max_planes = pool->base.pipe_count;
+ for (i = 0; i < pool->base.underlay_pipe_index; ++i)
+ dc->caps.planes[i] = plane_cap;
+
+ dc->caps.planes[pool->base.underlay_pipe_index] = underlay_plane_cap;
+
bw_calcs_init(dc->bw_dceip, dc->bw_vbios, dc->ctx->asic_id);
bw_calcs_data_update_from_pplib(dc);
.num_ddc = 5,
};
+static const struct dc_plane_cap plane_cap = {
+ .type = DC_PLANE_TYPE_DCE_RGB,
+
+ .pixel_format_support = {
+ .argb8888 = true,
+ .nv12 = false,
+ .fp16 = false
+ },
+
+ .max_upscale_factor = {
+ .argb8888 = 16000,
+ .nv12 = 1,
+ .fp16 = 1
+ },
+
+ .max_downscale_factor = {
+ .argb8888 = 250,
+ .nv12 = 1,
+ .fp16 = 1
+ }
+};
+
#define CTX ctx
#define REG(reg) mm ## reg
bool dce112_validate_bandwidth(
struct dc *dc,
- struct dc_state *context)
+ struct dc_state *context,
+ bool fast_validate)
{
bool result = false;
dc->bw_vbios,
context->res_ctx.pipe_ctx,
dc->res_pool->pipe_count,
- &context->bw.dce))
+ &context->bw_ctx.bw.dce))
result = true;
if (!result)
"%s: Bandwidth validation failed!",
__func__);
- if (memcmp(&dc->current_state->bw.dce,
- &context->bw.dce, sizeof(context->bw.dce))) {
+ if (memcmp(&dc->current_state->bw_ctx.bw.dce,
+ &context->bw_ctx.bw.dce, sizeof(context->bw_ctx.bw.dce))) {
DC_LOG_BANDWIDTH_CALCS(
"%s: finish,\n"
"sclk: %d sclk_sleep: %d yclk: %d blackout_recovery_time_us: %d\n"
,
__func__,
- context->bw.dce.nbp_state_change_wm_ns[0].b_mark,
- context->bw.dce.nbp_state_change_wm_ns[0].a_mark,
- context->bw.dce.urgent_wm_ns[0].b_mark,
- context->bw.dce.urgent_wm_ns[0].a_mark,
- context->bw.dce.stutter_exit_wm_ns[0].b_mark,
- context->bw.dce.stutter_exit_wm_ns[0].a_mark,
- context->bw.dce.nbp_state_change_wm_ns[1].b_mark,
- context->bw.dce.nbp_state_change_wm_ns[1].a_mark,
- context->bw.dce.urgent_wm_ns[1].b_mark,
- context->bw.dce.urgent_wm_ns[1].a_mark,
- context->bw.dce.stutter_exit_wm_ns[1].b_mark,
- context->bw.dce.stutter_exit_wm_ns[1].a_mark,
- context->bw.dce.nbp_state_change_wm_ns[2].b_mark,
- context->bw.dce.nbp_state_change_wm_ns[2].a_mark,
- context->bw.dce.urgent_wm_ns[2].b_mark,
- context->bw.dce.urgent_wm_ns[2].a_mark,
- context->bw.dce.stutter_exit_wm_ns[2].b_mark,
- context->bw.dce.stutter_exit_wm_ns[2].a_mark,
- context->bw.dce.stutter_mode_enable,
- context->bw.dce.cpuc_state_change_enable,
- context->bw.dce.cpup_state_change_enable,
- context->bw.dce.nbp_state_change_enable,
- context->bw.dce.all_displays_in_sync,
- context->bw.dce.dispclk_khz,
- context->bw.dce.sclk_khz,
- context->bw.dce.sclk_deep_sleep_khz,
- context->bw.dce.yclk_khz,
- context->bw.dce.blackout_recovery_time_us);
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[0].b_mark,
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[0].a_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[0].b_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[0].a_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[0].b_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[0].a_mark,
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[1].b_mark,
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[1].a_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[1].b_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[1].a_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[1].b_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[1].a_mark,
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[2].b_mark,
+ context->bw_ctx.bw.dce.nbp_state_change_wm_ns[2].a_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[2].b_mark,
+ context->bw_ctx.bw.dce.urgent_wm_ns[2].a_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[2].b_mark,
+ context->bw_ctx.bw.dce.stutter_exit_wm_ns[2].a_mark,
+ context->bw_ctx.bw.dce.stutter_mode_enable,
+ context->bw_ctx.bw.dce.cpuc_state_change_enable,
+ context->bw_ctx.bw.dce.cpup_state_change_enable,
+ context->bw_ctx.bw.dce.nbp_state_change_enable,
+ context->bw_ctx.bw.dce.all_displays_in_sync,
+ context->bw_ctx.bw.dce.dispclk_khz,
+ context->bw_ctx.bw.dce.sclk_khz,
+ context->bw_ctx.bw.dce.sclk_deep_sleep_khz,
+ context->bw_ctx.bw.dce.yclk_khz,
+ context->bw_ctx.bw.dce.blackout_recovery_time_us);
}
return result;
}
return DC_ERROR_UNEXPECTED;
if (dc_is_dp_signal(pipe_ctx->stream->signal)
- || pipe_ctx->stream->signal == SIGNAL_TYPE_VIRTUAL)
+ || dc_is_virtual_signal(pipe_ctx->stream->signal))
pipe_ctx->clock_source =
dc->res_pool->dp_clock_source;
else
dc->caps.max_planes = pool->base.pipe_count;
+ for (i = 0; i < dc->caps.max_planes; ++i)
+ dc->caps.planes[i] = plane_cap;
+
/* Create hardware sequencer */
dce112_hw_sequencer_construct(dc);
bool dce112_validate_bandwidth(
struct dc *dc,
- struct dc_state *context);
+ struct dc_state *context,
+ bool fast_validate);
enum dc_status dce112_add_stream_to_ctx(
struct dc *dc,
.num_ddc = 6,
};
+static const struct dc_plane_cap plane_cap = {
+ .type = DC_PLANE_TYPE_DCE_RGB,
+
+ .pixel_format_support = {
+ .argb8888 = true,
+ .nv12 = false,
+ .fp16 = false
+ },
+
+ .max_upscale_factor = {
+ .argb8888 = 16000,
+ .nv12 = 1,
+ .fp16 = 1
+ },
+
+ .max_downscale_factor = {
+ .argb8888 = 250,
+ .nv12 = 1,
+ .fp16 = 1
+ }
+};
+
static const struct dc_debug_options debug_defaults = {
.disable_clock_gate = true,
};
dc->caps.max_planes = pool->base.pipe_count;
+ for (i = 0; i < dc->caps.max_planes; ++i)
+ dc->caps.planes[i] = plane_cap;
+
bw_calcs_init(dc->bw_dceip, dc->bw_vbios, dc->ctx->asic_id);
bw_calcs_data_update_from_pplib(dc);
.num_ddc = 2,
};
+static const struct dc_plane_cap plane_cap = {
+ .type = DC_PLANE_TYPE_DCE_RGB,
+
+ .pixel_format_support = {
+ .argb8888 = true,
+ .nv12 = false,
+ .fp16 = false
+ },
+
+ .max_upscale_factor = {
+ .argb8888 = 16000,
+ .nv12 = 1,
+ .fp16 = 1
+ },
+
+ .max_downscale_factor = {
+ .argb8888 = 250,
+ .nv12 = 1,
+ .fp16 = 1
+ }
+};
+
static const struct dce_dmcu_registers dmcu_regs = {
DMCU_DCE80_REG_LIST()
};
bool dce80_validate_bandwidth(
struct dc *dc,
- struct dc_state *context)
+ struct dc_state *context,
+ bool fast_validate)
{
int i;
bool at_least_one_pipe = false;
if (at_least_one_pipe) {
/* TODO implement when needed but for now hardcode max value*/
- context->bw.dce.dispclk_khz = 681000;
- context->bw.dce.yclk_khz = 250000 * MEMORY_TYPE_MULTIPLIER_CZ;
+ context->bw_ctx.bw.dce.dispclk_khz = 681000;
+ context->bw_ctx.bw.dce.yclk_khz = 250000 * MEMORY_TYPE_MULTIPLIER_CZ;
} else {
- context->bw.dce.dispclk_khz = 0;
- context->bw.dce.yclk_khz = 0;
+ context->bw_ctx.bw.dce.dispclk_khz = 0;
+ context->bw_ctx.bw.dce.yclk_khz = 0;
}
return true;
}
dc->caps.max_planes = pool->base.pipe_count;
+
+ for (i = 0; i < dc->caps.max_planes; ++i)
+ dc->caps.planes[i] = plane_cap;
+
dc->caps.disable_dp_clk_share = true;
if (!resource_construct(num_virtual_links, dc, &pool->base,
}
dc->caps.max_planes = pool->base.pipe_count;
+
+ for (i = 0; i < dc->caps.max_planes; ++i)
+ dc->caps.planes[i] = plane_cap;
+
dc->caps.disable_dp_clk_share = true;
if (!resource_construct(num_virtual_links, dc, &pool->base,
}
dc->caps.max_planes = pool->base.pipe_count;
+
+ for (i = 0; i < dc->caps.max_planes; ++i)
+ dc->caps.planes[i] = plane_cap;
+
dc->caps.disable_dp_clk_share = true;
if (!resource_construct(num_virtual_links, dc, &pool->base,
#define DC_LOGGER \
clk_mgr->ctx->logger
-void dcn1_pplib_apply_display_requirements(
- struct dc *dc,
- struct dc_state *context)
-{
- struct dm_pp_display_configuration *pp_display_cfg = &context->pp_display_cfg;
-
- pp_display_cfg->min_engine_clock_khz = dc->res_pool->clk_mgr->clks.dcfclk_khz;
- pp_display_cfg->min_memory_clock_khz = dc->res_pool->clk_mgr->clks.fclk_khz;
- pp_display_cfg->min_engine_clock_deep_sleep_khz = dc->res_pool->clk_mgr->clks.dcfclk_deep_sleep_khz;
- pp_display_cfg->min_dcfc_deep_sleep_clock_khz = dc->res_pool->clk_mgr->clks.dcfclk_deep_sleep_khz;
- pp_display_cfg->min_dcfclock_khz = dc->res_pool->clk_mgr->clks.dcfclk_khz;
- pp_display_cfg->disp_clk_khz = dc->res_pool->clk_mgr->clks.dispclk_khz;
- dce110_fill_display_configs(context, pp_display_cfg);
-
- dm_pp_apply_display_requirements(dc->ctx, pp_display_cfg);
-}
-
static int dcn1_determine_dppclk_threshold(struct clk_mgr *clk_mgr, struct dc_clocks *new_clocks)
{
bool request_dpp_div = new_clocks->dispclk_khz > new_clocks->dppclk_khz;
{
struct dc *dc = clk_mgr->ctx->dc;
struct dc_debug_options *debug = &dc->debug;
- struct dc_clocks *new_clocks = &context->bw.dcn.clk;
- struct pp_smu_display_requirement_rv *smu_req_cur =
- &dc->res_pool->pp_smu_req;
- struct pp_smu_display_requirement_rv smu_req = *smu_req_cur;
- struct pp_smu_funcs_rv *pp_smu = dc->res_pool->pp_smu;
+ struct dc_clocks *new_clocks = &context->bw_ctx.bw.dcn.clk;
+ struct pp_smu_funcs_rv *pp_smu = NULL;
bool send_request_to_increase = false;
bool send_request_to_lower = false;
int display_count;
bool enter_display_off = false;
display_count = get_active_display_cnt(dc, context);
-
+ if (dc->res_pool->pp_smu)
+ pp_smu = &dc->res_pool->pp_smu->rv_funcs;
if (display_count == 0)
enter_display_off = true;
* if function pointer not set up, this message is
* sent as part of pplib_apply_display_requirements.
*/
- if (pp_smu->set_display_count)
+ if (pp_smu && pp_smu->set_display_count)
pp_smu->set_display_count(&pp_smu->pp_smu, display_count);
-
- smu_req.display_count = display_count;
}
if (new_clocks->dispclk_khz > clk_mgr->clks.dispclk_khz
if (should_set_clock(safe_to_lower, new_clocks->phyclk_khz, clk_mgr->clks.phyclk_khz)) {
clk_mgr->clks.phyclk_khz = new_clocks->phyclk_khz;
-
send_request_to_lower = true;
}
if (should_set_clock(safe_to_lower, new_clocks->fclk_khz, clk_mgr->clks.fclk_khz)) {
clk_mgr->clks.fclk_khz = new_clocks->fclk_khz;
- smu_req.hard_min_fclk_mhz = new_clocks->fclk_khz / 1000;
-
send_request_to_lower = true;
}
//DCF Clock
if (should_set_clock(safe_to_lower, new_clocks->dcfclk_khz, clk_mgr->clks.dcfclk_khz)) {
clk_mgr->clks.dcfclk_khz = new_clocks->dcfclk_khz;
- smu_req.hard_min_dcefclk_mhz = new_clocks->dcfclk_khz / 1000;
-
send_request_to_lower = true;
}
if (should_set_clock(safe_to_lower,
new_clocks->dcfclk_deep_sleep_khz, clk_mgr->clks.dcfclk_deep_sleep_khz)) {
clk_mgr->clks.dcfclk_deep_sleep_khz = new_clocks->dcfclk_deep_sleep_khz;
- smu_req.min_deep_sleep_dcefclk_mhz = (new_clocks->dcfclk_deep_sleep_khz + 999) / 1000;
-
send_request_to_lower = true;
}
*/
if (send_request_to_increase) {
/*use dcfclk to request voltage*/
- if (pp_smu->set_hard_min_fclk_by_freq &&
+ if (pp_smu && pp_smu->set_hard_min_fclk_by_freq &&
pp_smu->set_hard_min_dcfclk_by_freq &&
pp_smu->set_min_deep_sleep_dcfclk) {
- pp_smu->set_hard_min_fclk_by_freq(&pp_smu->pp_smu, smu_req.hard_min_fclk_mhz);
- pp_smu->set_hard_min_dcfclk_by_freq(&pp_smu->pp_smu, smu_req.hard_min_dcefclk_mhz);
- pp_smu->set_min_deep_sleep_dcfclk(&pp_smu->pp_smu, smu_req.min_deep_sleep_dcefclk_mhz);
- } else {
- if (pp_smu->set_display_requirement)
- pp_smu->set_display_requirement(&pp_smu->pp_smu, &smu_req);
- dcn1_pplib_apply_display_requirements(dc, context);
+ pp_smu->set_hard_min_fclk_by_freq(&pp_smu->pp_smu, new_clocks->fclk_khz / 1000);
+ pp_smu->set_hard_min_dcfclk_by_freq(&pp_smu->pp_smu, new_clocks->dcfclk_khz / 1000);
+ pp_smu->set_min_deep_sleep_dcfclk(&pp_smu->pp_smu, (new_clocks->dcfclk_deep_sleep_khz + 999) / 1000);
}
}
|| new_clocks->dispclk_khz == clk_mgr->clks.dispclk_khz) {
dcn1_ramp_up_dispclk_with_dpp(clk_mgr, new_clocks);
clk_mgr->clks.dispclk_khz = new_clocks->dispclk_khz;
-
send_request_to_lower = true;
}
if (!send_request_to_increase && send_request_to_lower) {
/*use dcfclk to request voltage*/
- if (pp_smu->set_hard_min_fclk_by_freq &&
+ if (pp_smu && pp_smu->set_hard_min_fclk_by_freq &&
pp_smu->set_hard_min_dcfclk_by_freq &&
pp_smu->set_min_deep_sleep_dcfclk) {
- pp_smu->set_hard_min_fclk_by_freq(&pp_smu->pp_smu, smu_req.hard_min_fclk_mhz);
- pp_smu->set_hard_min_dcfclk_by_freq(&pp_smu->pp_smu, smu_req.hard_min_dcefclk_mhz);
- pp_smu->set_min_deep_sleep_dcfclk(&pp_smu->pp_smu, smu_req.min_deep_sleep_dcefclk_mhz);
- } else {
- if (pp_smu->set_display_requirement)
- pp_smu->set_display_requirement(&pp_smu->pp_smu, &smu_req);
- dcn1_pplib_apply_display_requirements(dc, context);
+ pp_smu->set_hard_min_fclk_by_freq(&pp_smu->pp_smu, new_clocks->fclk_khz / 1000);
+ pp_smu->set_hard_min_dcfclk_by_freq(&pp_smu->pp_smu, new_clocks->dcfclk_khz / 1000);
+ pp_smu->set_min_deep_sleep_dcfclk(&pp_smu->pp_smu, (new_clocks->dcfclk_deep_sleep_khz + 999) / 1000);
}
}
-
- *smu_req_cur = smu_req;
}
static const struct clk_mgr_funcs dcn1_funcs = {
.get_dp_ref_clk_frequency = dce12_get_dp_ref_freq_khz,
uint32_t dprefclk_bypass;
};
-void dcn1_pplib_apply_display_requirements(
- struct dc *dc,
- struct dc_state *context);
-
struct clk_mgr *dcn1_clk_mgr_create(struct dc_context *ctx);
#endif //__DCN10_CLK_MGR_H__
DSCL_MODE_DSCL_BYPASS = 6
};
-enum gamut_remap_select {
- GAMUT_REMAP_BYPASS = 0,
- GAMUT_REMAP_COEFF,
- GAMUT_REMAP_COMA_COEFF,
- GAMUT_REMAP_COMB_COEFF
-};
-
void dpp_read_state(struct dpp *dpp_base,
struct dcn_dpp_state *s)
{
default:
break;
}
+
+ /* Set default color space based on format if none is given. */
+ color_space = input_color_space ? input_color_space : color_space;
+
REG_SET(CNVC_SURFACE_PIXEL_FORMAT, 0,
CNVC_SURFACE_PIXEL_FORMAT, pixel_format);
REG_UPDATE(FORMAT_CONTROL, FORMAT_CONTROL__ALPHA_EN, alpha_en);
for (i = 0; i < 12; i++)
tbl_entry.regval[i] = input_csc_color_matrix.matrix[i];
- tbl_entry.color_space = input_color_space;
+ tbl_entry.color_space = color_space;
if (color_space >= COLOR_SPACE_YCBCR601)
select = INPUT_CSC_SELECT_ICSC;
DSCL_MODE_DSCL_BYPASS = 6
};
-enum gamut_remap_select {
- GAMUT_REMAP_BYPASS = 0,
- GAMUT_REMAP_COEFF,
- GAMUT_REMAP_COMA_COEFF,
- GAMUT_REMAP_COMB_COEFF
-};
-
static const struct dpp_input_csc_matrix dpp_input_csc_matrix[] = {
{COLOR_SPACE_SRGB,
{0x2000, 0, 0, 0, 0, 0x2000, 0, 0, 0, 0, 0x2000, 0} },
int *num_part_y,
int *num_part_c)
{
+ int lb_memory_size, lb_memory_size_c, lb_memory_size_a, num_partitions_a,
+ lb_bpc, memory_line_size_y, memory_line_size_c, memory_line_size_a;
+
int line_size = scl_data->viewport.width < scl_data->recout.width ?
scl_data->viewport.width : scl_data->recout.width;
int line_size_c = scl_data->viewport_c.width < scl_data->recout.width ?
scl_data->viewport_c.width : scl_data->recout.width;
- int lb_bpc = dpp1_dscl_get_lb_depth_bpc(scl_data->lb_params.depth);
- int memory_line_size_y = (line_size * lb_bpc + 71) / 72; /* +71 to ceil */
- int memory_line_size_c = (line_size_c * lb_bpc + 71) / 72; /* +71 to ceil */
- int memory_line_size_a = (line_size + 5) / 6; /* +5 to ceil */
- int lb_memory_size, lb_memory_size_c, lb_memory_size_a, num_partitions_a;
+
+ if (line_size == 0)
+ line_size = 1;
+
+ if (line_size_c == 0)
+ line_size_c = 1;
+
+
+ lb_bpc = dpp1_dscl_get_lb_depth_bpc(scl_data->lb_params.depth);
+ memory_line_size_y = (line_size * lb_bpc + 71) / 72; /* +71 to ceil */
+ memory_line_size_c = (line_size_c * lb_bpc + 71) / 72; /* +71 to ceil */
+ memory_line_size_a = (line_size + 5) / 6; /* +5 to ceil */
if (lb_config == LB_MEMORY_CONFIG_1) {
lb_memory_size = 816;
{
struct dcn10_hubbub *hubbub1 = TO_DCN10_HUBBUB(hubbub);
- REG_UPDATE_SEQ(DCHUBBUB_ARB_WATERMARK_CHANGE_CNTL,
- DCHUBBUB_ARB_WATERMARK_CHANGE_REQUEST, 0, 1);
+ REG_UPDATE_SEQ_2(DCHUBBUB_ARB_WATERMARK_CHANGE_CNTL,
+ DCHUBBUB_ARB_WATERMARK_CHANGE_REQUEST, 0,
+ DCHUBBUB_ARB_WATERMARK_CHANGE_REQUEST, 1);
}
void hubbub1_program_watermarks(
hubbub1->watermarks.a.urgent_ns = watermarks->a.urgent_ns;
prog_wm_value = convert_and_clamp(watermarks->a.urgent_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_A, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_A, 0,
+ DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_A, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("URGENCY_WATERMARK_A calculated =%d\n"
"HW register value = 0x%x\n",
prog_wm_value = convert_and_clamp(
watermarks->a.cstate_pstate.cstate_enter_plus_exit_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_A, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_A, 0,
+ DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_A, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("SR_ENTER_EXIT_WATERMARK_A calculated =%d\n"
"HW register value = 0x%x\n",
watermarks->a.cstate_pstate.cstate_enter_plus_exit_ns, prog_wm_value);
prog_wm_value = convert_and_clamp(
watermarks->a.cstate_pstate.cstate_exit_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_A, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_A, 0,
+ DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_A, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("SR_EXIT_WATERMARK_A calculated =%d\n"
"HW register value = 0x%x\n",
watermarks->a.cstate_pstate.cstate_exit_ns, prog_wm_value);
prog_wm_value = convert_and_clamp(
watermarks->a.cstate_pstate.pstate_change_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_A, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_A, 0,
+ DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_A, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("DRAM_CLK_CHANGE_WATERMARK_A calculated =%d\n"
"HW register value = 0x%x\n\n",
watermarks->a.cstate_pstate.pstate_change_ns, prog_wm_value);
hubbub1->watermarks.b.urgent_ns = watermarks->b.urgent_ns;
prog_wm_value = convert_and_clamp(watermarks->b.urgent_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_B, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_B, 0,
+ DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_B, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("URGENCY_WATERMARK_B calculated =%d\n"
"HW register value = 0x%x\n",
prog_wm_value = convert_and_clamp(
watermarks->b.cstate_pstate.cstate_enter_plus_exit_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_B, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_B, 0,
+ DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_B, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("SR_ENTER_EXIT_WATERMARK_B calculated =%d\n"
"HW register value = 0x%x\n",
watermarks->b.cstate_pstate.cstate_enter_plus_exit_ns, prog_wm_value);
prog_wm_value = convert_and_clamp(
watermarks->b.cstate_pstate.cstate_exit_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_B, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_B, 0,
+ DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_B, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("SR_EXIT_WATERMARK_B calculated =%d\n"
"HW register value = 0x%x\n",
watermarks->b.cstate_pstate.cstate_exit_ns, prog_wm_value);
prog_wm_value = convert_and_clamp(
watermarks->b.cstate_pstate.pstate_change_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_B, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_B, 0,
+ DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_B, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("DRAM_CLK_CHANGE_WATERMARK_B calculated =%d\n"
"HW register value = 0x%x\n\n",
watermarks->b.cstate_pstate.pstate_change_ns, prog_wm_value);
hubbub1->watermarks.c.urgent_ns = watermarks->c.urgent_ns;
prog_wm_value = convert_and_clamp(watermarks->c.urgent_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_C, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_C, 0,
+ DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_C, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("URGENCY_WATERMARK_C calculated =%d\n"
"HW register value = 0x%x\n",
prog_wm_value = convert_and_clamp(
watermarks->c.cstate_pstate.cstate_enter_plus_exit_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_C, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_C, 0,
+ DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_C, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("SR_ENTER_EXIT_WATERMARK_C calculated =%d\n"
"HW register value = 0x%x\n",
watermarks->c.cstate_pstate.cstate_enter_plus_exit_ns, prog_wm_value);
prog_wm_value = convert_and_clamp(
watermarks->c.cstate_pstate.cstate_exit_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_C, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_C, 0,
+ DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_C, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("SR_EXIT_WATERMARK_C calculated =%d\n"
"HW register value = 0x%x\n",
watermarks->c.cstate_pstate.cstate_exit_ns, prog_wm_value);
prog_wm_value = convert_and_clamp(
watermarks->c.cstate_pstate.pstate_change_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_C, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_C, 0,
+ DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_C, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("DRAM_CLK_CHANGE_WATERMARK_C calculated =%d\n"
"HW register value = 0x%x\n\n",
watermarks->c.cstate_pstate.pstate_change_ns, prog_wm_value);
hubbub1->watermarks.d.urgent_ns = watermarks->d.urgent_ns;
prog_wm_value = convert_and_clamp(watermarks->d.urgent_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_D, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_D, 0,
+ DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_D, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("URGENCY_WATERMARK_D calculated =%d\n"
"HW register value = 0x%x\n",
prog_wm_value = convert_and_clamp(
watermarks->d.cstate_pstate.cstate_enter_plus_exit_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_D, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_D, 0,
+ DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_D, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("SR_ENTER_EXIT_WATERMARK_D calculated =%d\n"
"HW register value = 0x%x\n",
watermarks->d.cstate_pstate.cstate_enter_plus_exit_ns, prog_wm_value);
prog_wm_value = convert_and_clamp(
watermarks->d.cstate_pstate.cstate_exit_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_D, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_D, 0,
+ DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_D, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("SR_EXIT_WATERMARK_D calculated =%d\n"
"HW register value = 0x%x\n",
watermarks->d.cstate_pstate.cstate_exit_ns, prog_wm_value);
prog_wm_value = convert_and_clamp(
watermarks->d.cstate_pstate.pstate_change_ns,
refclk_mhz, 0x1fffff);
- REG_WRITE(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_D, prog_wm_value);
+ REG_SET(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_D, 0,
+ DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_D, prog_wm_value);
DC_LOG_BANDWIDTH_CALCS("DRAM_CLK_CHANGE_WATERMARK_D calculated =%d\n"
"HW register value = 0x%x\n\n",
watermarks->d.cstate_pstate.pstate_change_ns, prog_wm_value);
.dcc_support_pixel_format = hubbub1_dcc_support_pixel_format,
.get_dcc_compression_cap = hubbub1_get_dcc_compression_cap,
.wm_read_state = hubbub1_wm_read_state,
+ .program_watermarks = hubbub1_program_watermarks,
};
void hubbub1_construct(struct hubbub *hubbub,
#define TO_DCN10_HUBBUB(hubbub)\
container_of(hubbub, struct dcn10_hubbub, base)
-#define HUBHUB_REG_LIST_DCN()\
+#define HUBBUB_REG_LIST_DCN_COMMON()\
SR(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_A),\
- SR(DCHUBBUB_ARB_PTE_META_URGENCY_WATERMARK_A),\
SR(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_A),\
SR(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_B),\
- SR(DCHUBBUB_ARB_PTE_META_URGENCY_WATERMARK_B),\
SR(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_B),\
SR(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_C),\
- SR(DCHUBBUB_ARB_PTE_META_URGENCY_WATERMARK_C),\
SR(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_C),\
SR(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_D),\
- SR(DCHUBBUB_ARB_PTE_META_URGENCY_WATERMARK_D),\
SR(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_D),\
SR(DCHUBBUB_ARB_WATERMARK_CHANGE_CNTL),\
SR(DCHUBBUB_ARB_DRAM_STATE_CNTL),\
SR(DCHUBBUB_TEST_DEBUG_DATA),\
SR(DCHUBBUB_SOFT_RESET)
+#define HUBBUB_VM_REG_LIST() \
+ SR(DCHUBBUB_ARB_PTE_META_URGENCY_WATERMARK_A),\
+ SR(DCHUBBUB_ARB_PTE_META_URGENCY_WATERMARK_B),\
+ SR(DCHUBBUB_ARB_PTE_META_URGENCY_WATERMARK_C),\
+ SR(DCHUBBUB_ARB_PTE_META_URGENCY_WATERMARK_D)
+
#define HUBBUB_SR_WATERMARK_REG_LIST()\
SR(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_A),\
SR(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_A),\
SR(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_D)
#define HUBBUB_REG_LIST_DCN10(id)\
- HUBHUB_REG_LIST_DCN(), \
+ HUBBUB_REG_LIST_DCN_COMMON(), \
+ HUBBUB_VM_REG_LIST(), \
HUBBUB_SR_WATERMARK_REG_LIST(), \
SR(DCHUBBUB_SDPIF_FB_TOP),\
SR(DCHUBBUB_SDPIF_FB_BASE),\
#define HUBBUB_SF(reg_name, field_name, post_fix)\
.field_name = reg_name ## __ ## field_name ## post_fix
-
-#define HUBBUB_MASK_SH_LIST_DCN(mask_sh)\
+#define HUBBUB_MASK_SH_LIST_DCN_COMMON(mask_sh)\
HUBBUB_SF(DCHUBBUB_GLOBAL_TIMER_CNTL, DCHUBBUB_GLOBAL_TIMER_ENABLE, mask_sh), \
HUBBUB_SF(DCHUBBUB_SOFT_RESET, DCHUBBUB_GLOBAL_SOFT_RESET, mask_sh), \
HUBBUB_SF(DCHUBBUB_ARB_WATERMARK_CHANGE_CNTL, DCHUBBUB_ARB_WATERMARK_CHANGE_REQUEST, mask_sh), \
HUBBUB_SF(DCHUBBUB_ARB_DRAM_STATE_CNTL, DCHUBBUB_ARB_ALLOW_PSTATE_CHANGE_FORCE_VALUE, mask_sh), \
HUBBUB_SF(DCHUBBUB_ARB_DRAM_STATE_CNTL, DCHUBBUB_ARB_ALLOW_PSTATE_CHANGE_FORCE_ENABLE, mask_sh), \
HUBBUB_SF(DCHUBBUB_ARB_SAT_LEVEL, DCHUBBUB_ARB_SAT_LEVEL, mask_sh), \
- HUBBUB_SF(DCHUBBUB_ARB_DF_REQ_OUTSTAND, DCHUBBUB_ARB_MIN_REQ_OUTSTAND, mask_sh)
+ HUBBUB_SF(DCHUBBUB_ARB_DF_REQ_OUTSTAND, DCHUBBUB_ARB_MIN_REQ_OUTSTAND, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_A, DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_A, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_B, DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_B, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_C, DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_C, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_D, DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_D, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_A, DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_A, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_B, DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_B, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_C, DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_C, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_D, DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_D, mask_sh)
+
+#define HUBBUB_MASK_SH_LIST_STUTTER(mask_sh) \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_A, DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_A, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_B, DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_B, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_C, DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_C, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_D, DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_D, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_A, DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_A, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_B, DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_B, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_C, DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_C, mask_sh), \
+ HUBBUB_SF(DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_D, DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_D, mask_sh)
#define HUBBUB_MASK_SH_LIST_DCN10(mask_sh)\
- HUBBUB_MASK_SH_LIST_DCN(mask_sh), \
+ HUBBUB_MASK_SH_LIST_DCN_COMMON(mask_sh), \
+ HUBBUB_MASK_SH_LIST_STUTTER(mask_sh), \
HUBBUB_SF(DCHUBBUB_SDPIF_FB_TOP, SDPIF_FB_TOP, mask_sh), \
HUBBUB_SF(DCHUBBUB_SDPIF_FB_BASE, SDPIF_FB_BASE, mask_sh), \
HUBBUB_SF(DCHUBBUB_SDPIF_FB_OFFSET, SDPIF_FB_OFFSET, mask_sh), \
type FB_OFFSET;\
type AGP_BOT;\
type AGP_TOP;\
- type AGP_BASE
+ type AGP_BASE;\
+ type DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_A;\
+ type DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_B;\
+ type DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_C;\
+ type DCHUBBUB_ARB_DATA_URGENCY_WATERMARK_D;\
+ type DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_A;\
+ type DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_B;\
+ type DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_C;\
+ type DCHUBBUB_ARB_ALLOW_DRAM_CLK_CHANGE_WATERMARK_D
+
+#define HUBBUB_STUTTER_REG_FIELD_LIST(type) \
+ type DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_A;\
+ type DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_B;\
+ type DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_C;\
+ type DCHUBBUB_ARB_ALLOW_SR_ENTER_WATERMARK_D;\
+ type DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_A;\
+ type DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_B;\
+ type DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_C;\
+ type DCHUBBUB_ARB_ALLOW_SR_EXIT_WATERMARK_D
struct dcn_hubbub_shift {
DCN_HUBBUB_REG_FIELD_LIST(uint8_t);
+ HUBBUB_STUTTER_REG_FIELD_LIST(uint8_t);
};
struct dcn_hubbub_mask {
DCN_HUBBUB_REG_FIELD_LIST(uint32_t);
+ HUBBUB_STUTTER_REG_FIELD_LIST(uint32_t);
};
struct dc;
REG_UPDATE(DCHUBP_CNTL, HUBP_VTG_SEL, otg_inst);
}
+void hubp1_init(struct hubp *hubp)
+{
+ //do nothing
+}
static const struct hubp_funcs dcn10_hubp_funcs = {
.hubp_program_surface_flip_and_addr =
hubp1_program_surface_flip_and_addr,
.hubp_clear_underflow = hubp1_clear_underflow,
.hubp_disable_control = hubp1_disable_control,
.hubp_get_underflow_status = hubp1_get_underflow_status,
-
+ .hubp_init = hubp1_init,
};
/*****************************************/
#define HUBP_REG_LIST_DCN(id)\
SRI(DCHUBP_CNTL, HUBP, id),\
SRI(HUBPREQ_DEBUG_DB, HUBP, id),\
+ SRI(HUBPREQ_DEBUG, HUBP, id),\
SRI(DCSURF_ADDR_CONFIG, HUBP, id),\
SRI(DCSURF_TILING_CONFIG, HUBP, id),\
SRI(DCSURF_SURFACE_PITCH, HUBPREQ, id),\
#define HUBP_COMMON_REG_VARIABLE_LIST \
uint32_t DCHUBP_CNTL; \
uint32_t HUBPREQ_DEBUG_DB; \
+ uint32_t HUBPREQ_DEBUG; \
uint32_t DCSURF_ADDR_CONFIG; \
uint32_t DCSURF_TILING_CONFIG; \
uint32_t DCSURF_SURFACE_PITCH; \
.field_name = reg_name ## __ ## field_name ## post_fix
/* Mask/shift struct generation macro for all ASICs (including those with reduced functionality) */
-#define HUBP_MASK_SH_LIST_DCN(mask_sh)\
+#define HUBP_MASK_SH_LIST_DCN_COMMON(mask_sh)\
HUBP_SF(HUBP0_DCHUBP_CNTL, HUBP_BLANK_EN, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_CNTL, HUBP_TTU_DISABLE, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_CNTL, HUBP_UNDERFLOW_STATUS, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG, META_CHUNK_SIZE, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG, MIN_META_CHUNK_SIZE, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG, DPTE_GROUP_SIZE, mask_sh),\
- HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG, MPTE_GROUP_SIZE, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG, SWATH_HEIGHT, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG, PTE_ROW_HEIGHT_LINEAR, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG_C, CHUNK_SIZE_C, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG_C, META_CHUNK_SIZE_C, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG_C, MIN_META_CHUNK_SIZE_C, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG_C, DPTE_GROUP_SIZE_C, mask_sh),\
- HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG_C, MPTE_GROUP_SIZE_C, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG_C, SWATH_HEIGHT_C, mask_sh),\
HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG_C, PTE_ROW_HEIGHT_LINEAR_C, mask_sh),\
HUBP_SF(HUBPREQ0_BLANK_OFFSET_0, REFCYC_H_BLANK_END, mask_sh),\
HUBP_SF(HUBPREQ0_DCN_SURF0_TTU_CNTL1, REFCYC_PER_REQ_DELIVERY_PRE, mask_sh),\
HUBP_SF(HUBP0_HUBP_CLK_CNTL, HUBP_CLOCK_ENABLE, mask_sh)
+#define HUBP_MASK_SH_LIST_DCN(mask_sh)\
+ HUBP_MASK_SH_LIST_DCN_COMMON(mask_sh),\
+ HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG, MPTE_GROUP_SIZE, mask_sh),\
+ HUBP_SF(HUBP0_DCHUBP_REQ_SIZE_CONFIG_C, MPTE_GROUP_SIZE_C, mask_sh)
+
/* Mask/shift struct generation macro for ASICs with VM */
#define HUBP_MASK_SH_LIST_DCN_VM(mask_sh)\
HUBP_SF(HUBPREQ0_NOM_PARAMETERS_0, DST_Y_PER_PTE_ROW_NOM_L, mask_sh),\
type AGP_BASE;\
type AGP_BOT;\
type AGP_TOP;\
+ type DCN_VM_SYSTEM_APERTURE_DEFAULT_SYSTEM;\
+ type DCN_VM_SYSTEM_APERTURE_DEFAULT_ADDR_MSB;\
+ type DCN_VM_SYSTEM_APERTURE_DEFAULT_ADDR_LSB;\
/* todo: get these from GVM instead of reading registers ourselves */\
type PAGE_DIRECTORY_ENTRY_HI32;\
type PAGE_DIRECTORY_ENTRY_LO32;\
void hubp1_vready_workaround(struct hubp *hubp,
struct _vcs_dpi_display_pipe_dest_params_st *pipe_dest);
+void hubp1_init(struct hubp *hubp);
+
#endif
struct dc_log_buffer_ctx *log_ctx,
uint32_t ref_cycle)
{
- const uint32_t ref_clk_mhz = dc_ctx->dc->res_pool->ref_clock_inKhz / 1000;
+ const uint32_t ref_clk_mhz = dc_ctx->dc->res_pool->ref_clocks.dchub_ref_clock_inKhz / 1000;
static const unsigned int frac = 1000;
uint32_t us_x10 = (ref_cycle * frac) / ref_clk_mhz;
DTN_INFO("\nCALCULATED Clocks: dcfclk_khz:%d dcfclk_deep_sleep_khz:%d dispclk_khz:%d\n"
"dppclk_khz:%d max_supported_dppclk_khz:%d fclk_khz:%d socclk_khz:%d\n\n",
- dc->current_state->bw.dcn.clk.dcfclk_khz,
- dc->current_state->bw.dcn.clk.dcfclk_deep_sleep_khz,
- dc->current_state->bw.dcn.clk.dispclk_khz,
- dc->current_state->bw.dcn.clk.dppclk_khz,
- dc->current_state->bw.dcn.clk.max_supported_dppclk_khz,
- dc->current_state->bw.dcn.clk.fclk_khz,
- dc->current_state->bw.dcn.clk.socclk_khz);
+ dc->current_state->bw_ctx.bw.dcn.clk.dcfclk_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.dcfclk_deep_sleep_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.dispclk_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.dppclk_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.max_supported_dppclk_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.fclk_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.socclk_khz);
log_mpc_crc(dc, log_ctx);
return DC_OK;
}
-static void reset_back_end_for_pipe(
+static void dcn10_reset_back_end_for_pipe(
struct dc *dc,
struct pipe_ctx *pipe_ctx,
struct dc_state *context)
dcn10_verify_allow_pstate_change_high(dc);
}
-static void plane_atomic_power_down(struct dc *dc, struct pipe_ctx *pipe_ctx)
+static void plane_atomic_power_down(struct dc *dc,
+ struct dpp *dpp,
+ struct hubp *hubp)
{
struct dce_hwseq *hws = dc->hwseq;
- struct dpp *dpp = pipe_ctx->plane_res.dpp;
DC_LOGGER_INIT(dc->ctx->logger);
if (REG(DC_IP_REQUEST_CNTL)) {
REG_SET(DC_IP_REQUEST_CNTL, 0,
IP_REQUEST_EN, 1);
dpp_pg_control(hws, dpp->inst, false);
- hubp_pg_control(hws, pipe_ctx->plane_res.hubp->inst, false);
+ hubp_pg_control(hws, hubp->inst, false);
dpp->funcs->dpp_reset(dpp);
REG_SET(DC_IP_REQUEST_CNTL, 0,
IP_REQUEST_EN, 0);
DC_LOG_DEBUG(
- "Power gated front end %d\n", pipe_ctx->pipe_idx);
+ "Power gated front end %d\n", hubp->inst);
}
}
hubp->power_gated = true;
dc->optimized_required = false; /* We're powering off, no need to optimize */
- plane_atomic_power_down(dc, pipe_ctx);
+ plane_atomic_power_down(dc,
+ pipe_ctx->plane_res.dpp,
+ pipe_ctx->plane_res.hubp);
pipe_ctx->stream = NULL;
memset(&pipe_ctx->stream_res, 0, sizeof(pipe_ctx->stream_res));
* to non-preferred front end. If pipe_ctx->stream is not NULL,
* we will use the pipe, so don't disable
*/
- if (pipe_ctx->stream != NULL)
+ if (pipe_ctx->stream != NULL && can_apply_seamless_boot)
continue;
- if (tg->funcs->is_tg_enabled(tg))
- tg->funcs->lock(tg);
-
/* Blank controller using driver code instead of
* command table.
*/
if (tg->funcs->is_tg_enabled(tg)) {
+ tg->funcs->lock(tg);
tg->funcs->set_blank(tg, true);
hwss_wait_for_blank_complete(tg);
}
struct dpp *dpp = dc->res_pool->dpps[i];
struct pipe_ctx *pipe_ctx = &context->res_ctx.pipe_ctx[i];
- // W/A for issue with dc_post_update_surfaces_to_stream
- hubp->power_gated = true;
-
/* There is assumption that pipe_ctx is not mapping irregularly
* to non-preferred front end. If pipe_ctx->stream is not NULL,
* we will use the pipe, so don't disable
*/
- if (pipe_ctx->stream != NULL)
+ if (can_apply_seamless_boot &&
+ pipe_ctx->stream != NULL &&
+ pipe_ctx->stream_res.tg->funcs->is_tg_enabled(
+ pipe_ctx->stream_res.tg))
continue;
+ /* Disable on the current state so the new one isn't cleared. */
+ pipe_ctx = &dc->current_state->res_ctx.pipe_ctx[i];
+
dpp->funcs->dpp_reset(dpp);
pipe_ctx->stream_res.tg = tg;
link->link_status.link_active = true;
}
+ /* If taking control over from VBIOS, we may want to optimize our first
+ * mode set, so we need to skip powering down pipes until we know which
+ * pipes we want to use.
+ * Otherwise, if taking control is not possible, we need to power
+ * everything down.
+ */
+ if (dcb->funcs->is_accelerated_mode(dcb) || dc->config.power_down_display_on_boot) {
+ for (i = 0; i < dc->res_pool->pipe_count; i++) {
+ struct hubp *hubp = dc->res_pool->hubps[i];
+ struct dpp *dpp = dc->res_pool->dpps[i];
+
+ hubp->funcs->hubp_init(hubp);
+ dc->res_pool->opps[i]->mpc_tree_params.opp_id = dc->res_pool->opps[i]->inst;
+ plane_atomic_power_down(dc, dpp, hubp);
+ }
+
+ apply_DEGVIDCN10_253_wa(dc);
+ }
+
for (i = 0; i < dc->res_pool->audio_count; i++) {
struct audio *audio = dc->res_pool->audios[i];
enable_power_gating_plane(dc->hwseq, true);
memset(&dc->res_pool->clk_mgr->clks, 0, sizeof(dc->res_pool->clk_mgr->clks));
-
- if (dc->hwss.init_pipes)
- dc->hwss.init_pipes(dc, dc->current_state);
}
-static void reset_hw_ctx_wrap(
+static void dcn10_reset_hw_ctx_wrap(
struct dc *dc,
struct dc_state *context)
{
pipe_need_reprogram(pipe_ctx_old, pipe_ctx)) {
struct clock_source *old_clk = pipe_ctx_old->clock_source;
- reset_back_end_for_pipe(dc, pipe_ctx_old, dc->current_state);
- if (dc->hwss.enable_stream_gating) {
+ dcn10_reset_back_end_for_pipe(dc, pipe_ctx_old, dc->current_state);
+ if (dc->hwss.enable_stream_gating)
dc->hwss.enable_stream_gating(dc, pipe_ctx);
- }
if (old_clk)
old_clk->funcs->cs_power_down(old_clk);
}
switch (top_pipe_ctx->plane_res.scl_data.format) {
case PIXEL_FORMAT_ARGB2101010:
- if (top_pipe_ctx->stream->out_transfer_func->tf == TRANSFER_FUNCTION_UNITY) {
+ if (top_pipe_ctx->stream->out_transfer_func->tf == TRANSFER_FUNCTION_PQ) {
/* HDR10, ARGB2101010 - set boarder color to red */
color->color_r_cr = color_value;
}
plane_state->format,
EXPANSION_MODE_ZERO,
plane_state->input_csc_color_matrix,
- COLOR_SPACE_YCBCR601_LIMITED);
+ plane_state->color_space);
//set scale and bias registers
build_prescale_params(&bns_params, plane_state);
* divided by 2
*/
if (plane_state->update_flags.bits.full_update) {
- bool should_divided_by_2 = context->bw.dcn.clk.dppclk_khz <=
+ bool should_divided_by_2 = context->bw_ctx.bw.dcn.clk.dppclk_khz <=
dc->res_pool->clk_mgr->clks.dispclk_khz / 2;
dpp->funcs->dpp_dppclk_control(
if (pipe_ctx->stream->cursor_attributes.address.quad_part != 0) {
dc->hwss.set_cursor_position(pipe_ctx);
dc->hwss.set_cursor_attribute(pipe_ctx);
+
+ if (dc->hwss.set_cursor_sdr_white_level)
+ dc->hwss.set_cursor_sdr_white_level(pipe_ctx);
}
if (plane_state->update_flags.bits.full_update) {
int i;
struct timing_generator *tg;
bool removed_pipe[4] = { false };
+ bool interdependent_update = false;
struct pipe_ctx *top_pipe_to_program =
find_top_pipe_for_stream(dc, context, stream);
DC_LOGGER_INIT(dc->ctx->logger);
tg = top_pipe_to_program->stream_res.tg;
- dcn10_pipe_control_lock(dc, top_pipe_to_program, true);
+ interdependent_update = top_pipe_to_program->plane_state &&
+ top_pipe_to_program->plane_state->update_flags.bits.full_update;
+
+ if (interdependent_update)
+ lock_all_pipes(dc, context, true);
+ else
+ dcn10_pipe_control_lock(dc, top_pipe_to_program, true);
if (num_planes == 0) {
/* OTG blank before remove all front end */
*/
if (pipe_ctx->plane_state && !old_pipe_ctx->plane_state) {
if (old_pipe_ctx->stream_res.tg == tg &&
- old_pipe_ctx->plane_res.hubp &&
- old_pipe_ctx->plane_res.hubp->opp_id != 0xf) {
+ old_pipe_ctx->plane_res.hubp &&
+ old_pipe_ctx->plane_res.hubp->opp_id != 0xf)
dcn10_disable_plane(dc, old_pipe_ctx);
- /*
- * power down fe will unlock when calling reset, need
- * to lock it back here. Messy, need rework.
- */
- pipe_ctx->stream_res.tg->funcs->lock(pipe_ctx->stream_res.tg);
- }
}
if ((!pipe_ctx->plane_state ||
if (num_planes > 0)
program_all_pipe_in_tree(dc, top_pipe_to_program, context);
- dcn10_pipe_control_lock(dc, top_pipe_to_program, false);
-
- if (top_pipe_to_program->plane_state &&
- top_pipe_to_program->plane_state->update_flags.bits.full_update)
+ if (interdependent_update)
for (i = 0; i < dc->res_pool->pipe_count; i++) {
struct pipe_ctx *pipe_ctx = &context->res_ctx.pipe_ctx[i];
- tg = pipe_ctx->stream_res.tg;
/* Skip inactive pipes and ones already updated */
- if (!pipe_ctx->stream || pipe_ctx->stream == stream
- || !pipe_ctx->plane_state
- || !tg->funcs->is_tg_enabled(tg))
+ if (!pipe_ctx->stream || pipe_ctx->stream == stream ||
+ !pipe_ctx->plane_state || !tg->funcs->is_tg_enabled(tg))
continue;
- tg->funcs->lock(tg);
-
pipe_ctx->plane_res.hubp->funcs->hubp_setup_interdependent(
pipe_ctx->plane_res.hubp,
&pipe_ctx->dlg_regs,
&pipe_ctx->ttu_regs);
-
- tg->funcs->unlock(tg);
}
+ if (interdependent_update)
+ lock_all_pipes(dc, context, false);
+ else
+ dcn10_pipe_control_lock(dc, top_pipe_to_program, false);
+
if (num_planes == 0)
false_optc_underflow_wa(dc, stream, tg);
struct dc *dc,
struct dc_state *context)
{
+ struct hubbub *hubbub = dc->res_pool->hubbub;
+
if (dc->debug.sanity_checks)
dcn10_verify_allow_pstate_change_high(dc);
if (!IS_FPGA_MAXIMUS_DC(dc->ctx->dce_environment)) {
if (context->stream_count == 0)
- context->bw.dcn.clk.phyclk_khz = 0;
+ context->bw_ctx.bw.dcn.clk.phyclk_khz = 0;
dc->res_pool->clk_mgr->funcs->update_clocks(
dc->res_pool->clk_mgr,
false);
}
- hubbub1_program_watermarks(dc->res_pool->hubbub,
- &context->bw.dcn.watermarks,
- dc->res_pool->ref_clock_inKhz / 1000,
+ hubbub->funcs->program_watermarks(hubbub,
+ &context->bw_ctx.bw.dcn.watermarks,
+ dc->res_pool->ref_clocks.dchub_ref_clock_inKhz / 1000,
true);
dcn10_stereo_hw_frame_pack_wa(dc, context);
struct dc *dc,
struct dc_state *context)
{
+ struct hubbub *hubbub = dc->res_pool->hubbub;
+
if (dc->debug.sanity_checks)
dcn10_verify_allow_pstate_change_high(dc);
if (!IS_FPGA_MAXIMUS_DC(dc->ctx->dce_environment)) {
if (context->stream_count == 0)
- context->bw.dcn.clk.phyclk_khz = 0;
+ context->bw_ctx.bw.dcn.clk.phyclk_khz = 0;
dc->res_pool->clk_mgr->funcs->update_clocks(
dc->res_pool->clk_mgr,
true);
}
- hubbub1_program_watermarks(dc->res_pool->hubbub,
- &context->bw.dcn.watermarks,
- dc->res_pool->ref_clock_inKhz / 1000,
+ hubbub->funcs->program_watermarks(hubbub,
+ &context->bw_ctx.bw.dcn.watermarks,
+ dc->res_pool->ref_clocks.dchub_ref_clock_inKhz / 1000,
true);
dcn10_stereo_hw_frame_pack_wa(dc, context);
flip_pending = pipe_ctx->plane_res.hubp->funcs->hubp_is_flip_pending(
pipe_ctx->plane_res.hubp);
- plane_state->status.is_flip_pending = flip_pending;
+ plane_state->status.is_flip_pending = plane_state->status.is_flip_pending || flip_pending;
if (!flip_pending)
plane_state->status.current_address = plane_state->status.requested_address;
struct dpp *dpp = pipe_ctx->plane_res.dpp;
struct dc_cursor_mi_param param = {
.pixel_clk_khz = pipe_ctx->stream->timing.pix_clk_100hz / 10,
- .ref_clk_khz = pipe_ctx->stream->ctx->dc->res_pool->ref_clock_inKhz,
+ .ref_clk_khz = pipe_ctx->stream->ctx->dc->res_pool->ref_clocks.dchub_ref_clock_inKhz,
.viewport = pipe_ctx->plane_res.scl_data.viewport,
.h_scale_ratio = pipe_ctx->plane_res.scl_data.ratios.horz,
.v_scale_ratio = pipe_ctx->plane_res.scl_data.ratios.vert,
.rotation = pipe_ctx->plane_state->rotation,
.mirror = pipe_ctx->plane_state->horizontal_mirror
};
+ uint32_t x_plane = pipe_ctx->plane_state->dst_rect.x;
+ uint32_t y_plane = pipe_ctx->plane_state->dst_rect.y;
+ uint32_t x_offset = min(x_plane, pos_cpy.x);
+ uint32_t y_offset = min(y_plane, pos_cpy.y);
- pos_cpy.x_hotspot += pipe_ctx->plane_state->dst_rect.x;
- pos_cpy.y_hotspot += pipe_ctx->plane_state->dst_rect.y;
+ pos_cpy.x -= x_offset;
+ pos_cpy.y -= y_offset;
+ pos_cpy.x_hotspot += (x_plane - x_offset);
+ pos_cpy.y_hotspot += (y_plane - y_offset);
if (pipe_ctx->plane_state->address.type
== PLN_ADDR_TYPE_VIDEO_PROGRESSIVE)
return vertical_line_start;
}
+void lock_all_pipes(struct dc *dc,
+ struct dc_state *context,
+ bool lock)
+{
+ struct pipe_ctx *pipe_ctx;
+ struct timing_generator *tg;
+ int i;
+
+ for (i = 0; i < dc->res_pool->pipe_count; i++) {
+ pipe_ctx = &context->res_ctx.pipe_ctx[i];
+ tg = pipe_ctx->stream_res.tg;
+ /*
+ * Only lock the top pipe's tg to prevent redundant
+ * (un)locking. Also skip if pipe is disabled.
+ */
+ if (pipe_ctx->top_pipe ||
+ !pipe_ctx->stream || !pipe_ctx->plane_state ||
+ !tg->funcs->is_tg_enabled(tg))
+ continue;
+
+ if (lock)
+ tg->funcs->lock(tg);
+ else
+ tg->funcs->unlock(tg);
+ }
+}
+
static void calc_vupdate_position(
struct pipe_ctx *pipe_ctx,
uint32_t *start_line,
tg->funcs->setup_vertical_interrupt2(tg, start_line);
}
+static void dcn10_unblank_stream(struct pipe_ctx *pipe_ctx,
+ struct dc_link_settings *link_settings)
+{
+ struct encoder_unblank_param params = { { 0 } };
+ struct dc_stream_state *stream = pipe_ctx->stream;
+ struct dc_link *link = stream->link;
+
+ /* only 3 items below are used by unblank */
+ params.timing = pipe_ctx->stream->timing;
+
+ params.link_settings.link_rate = link_settings->link_rate;
+
+ if (dc_is_dp_signal(pipe_ctx->stream->signal)) {
+ if (params.timing.pixel_encoding == PIXEL_ENCODING_YCBCR420)
+ params.timing.pix_clk_100hz /= 2;
+ pipe_ctx->stream_res.stream_enc->funcs->dp_unblank(pipe_ctx->stream_res.stream_enc, ¶ms);
+ }
+
+ if (link->local_sink && link->local_sink->sink_signal == SIGNAL_TYPE_EDP) {
+ link->dc->hwss.edp_backlight_control(link, true);
+ }
+}
+
static const struct hw_sequencer_funcs dcn10_funcs = {
.program_gamut_remap = program_gamut_remap,
.init_hw = dcn10_init_hw,
.update_info_frame = dce110_update_info_frame,
.enable_stream = dce110_enable_stream,
.disable_stream = dce110_disable_stream,
- .unblank_stream = dce110_unblank_stream,
+ .unblank_stream = dcn10_unblank_stream,
.blank_stream = dce110_blank_stream,
.enable_audio_stream = dce110_enable_audio_stream,
.disable_audio_stream = dce110_disable_audio_stream,
.pipe_control_lock = dcn10_pipe_control_lock,
.prepare_bandwidth = dcn10_prepare_bandwidth,
.optimize_bandwidth = dcn10_optimize_bandwidth,
- .reset_hw_ctx_wrap = reset_hw_ctx_wrap,
+ .reset_hw_ctx_wrap = dcn10_reset_hw_ctx_wrap,
.enable_stream_timing = dcn10_enable_stream_timing,
.set_drr = set_drr,
.get_position = get_position,
int get_vupdate_offset_from_vsync(struct pipe_ctx *pipe_ctx);
+void lock_all_pipes(struct dc *dc,
+ struct dc_state *context,
+ bool lock);
+
#endif /* __DC_HWSS_DCN10_H__ */
unsigned int chars_printed = 0;
unsigned int remaining_buffer = bufSize;
- const uint32_t ref_clk_mhz = dc_ctx->dc->res_pool->ref_clock_inKhz / 1000;
+ const uint32_t ref_clk_mhz = dc_ctx->dc->res_pool->ref_clocks.dchub_ref_clock_inKhz / 1000;
static const unsigned int frac = 1000;
memset(&wm, 0, sizeof(struct dcn_hubbub_wm));
unsigned int chars_printed = 0;
unsigned int remaining_buffer = bufSize;
- const uint32_t ref_clk_mhz = dc_ctx->dc->res_pool->ref_clock_inKhz / 1000;
+ const uint32_t ref_clk_mhz = dc_ctx->dc->res_pool->ref_clocks.dchub_ref_clock_inKhz / 1000;
static const unsigned int frac = 1000;
if (invarOnly)
chars_printed = snprintf_count(pBuf, bufSize, "dcfclk,dcfclk_deep_sleep,dispclk,"
"dppclk,fclk,socclk\n"
"%d,%d,%d,%d,%d,%d\n",
- dc->current_state->bw.dcn.clk.dcfclk_khz,
- dc->current_state->bw.dcn.clk.dcfclk_deep_sleep_khz,
- dc->current_state->bw.dcn.clk.dispclk_khz,
- dc->current_state->bw.dcn.clk.dppclk_khz,
- dc->current_state->bw.dcn.clk.fclk_khz,
- dc->current_state->bw.dcn.clk.socclk_khz);
+ dc->current_state->bw_ctx.bw.dcn.clk.dcfclk_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.dcfclk_deep_sleep_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.dispclk_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.dppclk_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.fclk_khz,
+ dc->current_state->bw_ctx.bw.dcn.clk.socclk_khz);
remaining_buffer -= chars_printed;
pBuf += chars_printed;
#define HPD_REG_UPDATE_N(reg_name, n, ...) \
generic_reg_update_ex(CTX, \
HPD_REG(reg_name), \
- HPD_REG_READ(reg_name), \
n, __VA_ARGS__)
#define HPD_REG_UPDATE(reg_name, field, val) \
#define AUX_REG_UPDATE_N(reg_name, n, ...) \
generic_reg_update_ex(CTX, \
AUX_REG(reg_name), \
- AUX_REG_READ(reg_name), \
n, __VA_ARGS__)
#define AUX_REG_UPDATE(reg_name, field, val) \
};
#endif
+static const struct dc_plane_cap plane_cap = {
+ .type = DC_PLANE_TYPE_DCN_UNIVERSAL,
+ .blends_with_above = true,
+ .blends_with_below = true,
+ .per_pixel_alpha = true,
+
+ .pixel_format_support = {
+ .argb8888 = true,
+ .nv12 = true,
+ .fp16 = true
+ },
+
+ .max_upscale_factor = {
+ .argb8888 = 16000,
+ .nv12 = 16000,
+ .fp16 = 1
+ },
+
+ .max_downscale_factor = {
+ .argb8888 = 250,
+ .nv12 = 250,
+ .fp16 = 1
+ }
+};
+
static const struct dc_debug_options debug_defaults_drv = {
.sanity_checks = true,
.disable_dmcu = true,
*clk_src = NULL;
}
-static struct pp_smu_funcs_rv *dcn10_pp_smu_create(struct dc_context *ctx)
+static struct pp_smu_funcs *dcn10_pp_smu_create(struct dc_context *ctx)
{
- struct pp_smu_funcs_rv *pp_smu = kzalloc(sizeof(*pp_smu), GFP_KERNEL);
+ struct pp_smu_funcs *pp_smu = kzalloc(sizeof(*pp_smu), GFP_KERNEL);
if (!pp_smu)
return pp_smu;
- dm_pp_get_funcs_rv(ctx, pp_smu);
+ dm_pp_get_funcs(ctx, pp_smu);
return pp_smu;
}
for (i = 0; i < pool->base.stream_enc_count; i++) {
if (pool->base.stream_enc[i] != NULL) {
- /* TODO: free dcn version of stream encoder once implemented
- * rather than using virtual stream encoder
- */
- kfree(pool->base.stream_enc[i]);
+ kfree(DCN10STRENC_FROM_STRENC(pool->base.stream_enc[i]));
pool->base.stream_enc[i] = NULL;
}
}
}
}
- for (i = 0; i < pool->base.stream_enc_count; i++)
- kfree(pool->base.stream_enc[i]);
-
for (i = 0; i < pool->base.audio_count; i++) {
if (pool->base.audios[i])
dce_aud_destroy(&pool->base.audios[i]);
{
struct resource_context *res_ctx = &context->res_ctx;
struct pipe_ctx *head_pipe = resource_get_head_pipe_for_stream(res_ctx, stream);
- struct pipe_ctx *idle_pipe = find_idle_secondary_pipe(res_ctx, pool);
+ struct pipe_ctx *idle_pipe = find_idle_secondary_pipe(res_ctx, pool, head_pipe);
if (!head_pipe) {
ASSERT(0);
continue;
if (context->stream_status[i].plane_count > 2)
- return false;
+ return DC_FAIL_UNSUPPORTED_1;
for (j = 0; j < context->stream_status[i].plane_count; j++) {
struct dc_plane_state *plane =
goto fail;
}
- dml_init_instance(&dc->dml, DML_PROJECT_RAVEN1);
+ dml_init_instance(&dc->dml, &dcn1_0_soc, &dcn1_0_ip, DML_PROJECT_RAVEN1);
memcpy(dc->dcn_ip, &dcn10_ip_defaults, sizeof(dcn10_ip_defaults));
memcpy(dc->dcn_soc, &dcn10_soc_defaults, sizeof(dcn10_soc_defaults));
dcn10_hw_sequencer_construct(dc);
dc->caps.max_planes = pool->base.pipe_count;
+ for (i = 0; i < dc->caps.max_planes; ++i)
+ dc->caps.planes[i] = plane_cap;
+
dc->cap_funcs = cap_funcs;
return true;
}
struct resource_pool *dcn10_create_resource_pool(
- uint8_t num_virtual_links,
+ const struct dc_init_data *init_data,
struct dc *dc)
{
struct dcn10_resource_pool *pool =
if (!pool)
return NULL;
- if (construct(num_virtual_links, dc, pool))
+ if (construct(init_data->num_virtual_links, dc, pool))
return &pool->base;
BREAK_TO_DEBUGGER();
struct resource_pool base;
};
struct resource_pool *dcn10_create_resource_pool(
- uint8_t num_virtual_links,
+ const struct dc_init_data *init_data,
struct dc *dc);
void enc1_stream_encoder_dp_set_stream_attribute(
struct stream_encoder *enc,
struct dc_crtc_timing *crtc_timing,
- enum dc_color_space output_color_space)
+ enum dc_color_space output_color_space,
+ uint32_t enable_sdp_splitting)
{
uint32_t h_active_start;
uint32_t v_active_start;
break;
case PIXEL_ENCODING_YCBCR420:
dp_pixel_encoding = DP_PIXEL_ENCODING_TYPE_YCBCR420;
- REG_UPDATE(DP_VID_TIMING, DP_VID_N_MUL, 1);
break;
default:
dp_pixel_encoding = DP_PIXEL_ENCODING_TYPE_RGB444;
3, /* packetIndex */
&info_frame->hdrsmd);
+ if (info_frame->dpsdp.valid)
+ enc1_update_generic_info_packet(
+ enc1,
+ 4,/* packetIndex */
+ &info_frame->dpsdp);
+
/* enable/disable transmission of packet(s).
* If enabled, packet transmission begins on the next frame
*/
REG_UPDATE(DP_SEC_CNTL, DP_SEC_GSP0_ENABLE, info_frame->vsc.valid);
REG_UPDATE(DP_SEC_CNTL, DP_SEC_GSP2_ENABLE, info_frame->spd.valid);
REG_UPDATE(DP_SEC_CNTL, DP_SEC_GSP3_ENABLE, info_frame->hdrsmd.valid);
-
+ REG_UPDATE(DP_SEC_CNTL, DP_SEC_GSP4_ENABLE, info_frame->dpsdp.valid);
/* This bit is the master enable bit.
* When enabling secondary stream engine,
*/
REG_UPDATE(DP_VID_STREAM_CNTL, DP_VID_STREAM_DIS_DEFER, 2);
/* Larger delay to wait until VBLANK - use max retry of
- * 10us*3000=30ms. This covers 16.6ms of typical 60 Hz mode +
+ * 10us*5000=50ms. This covers 41.7ms of minimum 24 Hz mode +
* a little more because we may not trust delay accuracy.
*/
- max_retries = DP_BLANK_MAX_RETRY * 150;
+ max_retries = DP_BLANK_MAX_RETRY * 250;
/* disable DP stream */
REG_UPDATE(DP_VID_STREAM_CNTL, DP_VID_STREAM_ENABLE, 0);
if (param->link_settings.link_rate != LINK_RATE_UNKNOWN) {
uint32_t n_vid = 0x8000;
uint32_t m_vid;
+ uint32_t n_multiply = 0;
+ uint64_t m_vid_l = n_vid;
+ /* YCbCr 4:2:0 : Computed VID_M will be 2X the input rate */
+ if (param->timing.pixel_encoding == PIXEL_ENCODING_YCBCR420) {
+ /*this param->pixel_clk_khz is half of 444 rate for 420 already*/
+ n_multiply = 1;
+ }
/* M / N = Fstream / Flink
* m_vid / n_vid = pixel rate / link rate
*/
- uint64_t m_vid_l = n_vid;
-
- m_vid_l *= param->pixel_clk_khz;
+ m_vid_l *= param->timing.pix_clk_100hz / 10;
m_vid_l = div_u64(m_vid_l,
param->link_settings.link_rate
* LINK_RATE_REF_FREQ_IN_KHZ);
REG_UPDATE(DP_VID_M, DP_VID_M, m_vid);
- REG_UPDATE(DP_VID_TIMING, DP_VID_M_N_GEN_EN, 1);
+ REG_UPDATE_2(DP_VID_TIMING,
+ DP_VID_M_N_GEN_EN, 1,
+ DP_VID_N_MUL, n_multiply);
}
/* set DIG_START to 0x1 to resync FIFO */
void enc1_stream_encoder_dp_set_stream_attribute(
struct stream_encoder *enc,
struct dc_crtc_timing *crtc_timing,
- enum dc_color_space output_color_space);
+ enum dc_color_space output_color_space,
+ uint32_t enable_sdp_splitting);
void enc1_stream_encoder_hdmi_set_stream_attribute(
struct stream_encoder *enc,
bool enable);
/*
- * poll pending down reply before clear payload allocation table
+ * poll pending down reply
*/
void dm_helpers_dp_mst_poll_pending_down_reply(
struct dc_context *ctx,
* interface to PPLIB/SMU to setup clocks and pstate requirements on SoC
*/
+typedef bool BOOLEAN;
+
enum pp_smu_ver {
/*
* PP_SMU_INTERFACE_X should be interpreted as the interface defined
struct pp_smu_wm_set_range writer_wm_sets[MAX_WATERMARK_SETS];
};
-struct pp_smu_display_requirement_rv {
- /* PPSMC_MSG_SetDisplayCount: count
- * 0 triggers S0i2 optimization
- */
- unsigned int display_count;
-
- /* PPSMC_MSG_SetHardMinFclkByFreq: mhz
- * FCLK will vary with DPM, but never below requested hard min
- */
- unsigned int hard_min_fclk_mhz;
-
- /* PPSMC_MSG_SetHardMinDcefclkByFreq: mhz
- * fixed clock at requested freq, either from FCH bypass or DFS
- */
- unsigned int hard_min_dcefclk_mhz;
-
- /* PPSMC_MSG_SetMinDeepSleepDcefclk: mhz
- * when DF is in cstate, dcf clock is further divided down
- * to just above given frequency
- */
- unsigned int min_deep_sleep_dcefclk_mhz;
-};
-
struct pp_smu_funcs_rv {
struct pp_smu pp_smu;
/* PME w/a */
void (*set_pme_wa_enable)(struct pp_smu *pp);
- /*
- * Legacy functions. Used for backwards comp. with existing
- * PPlib code.
- */
- void (*set_display_requirement)(struct pp_smu *pp,
- struct pp_smu_display_requirement_rv *req);
};
struct pp_smu_funcs {
* GPU registers access
*
*/
-
+uint32_t dm_read_reg_func(
+ const struct dc_context *ctx,
+ uint32_t address,
+ const char *func_name);
/* enable for debugging new code, this adds 50k to the driver size. */
/* #define DM_CHECK_ADDR_0 */
#define dm_read_reg(ctx, address) \
dm_read_reg_func(ctx, address, __func__)
-static inline uint32_t dm_read_reg_func(
- const struct dc_context *ctx,
- uint32_t address,
- const char *func_name)
-{
- uint32_t value;
-#ifdef DM_CHECK_ADDR_0
- if (address == 0) {
- DC_ERR("invalid register read; address = 0\n");
- return 0;
- }
-#endif
- value = cgs_read_register(ctx->cgs_device, address);
- trace_amdgpu_dc_rreg(&ctx->perf_trace->read_count, address, value);
- return value;
-}
#define dm_write_reg(ctx, address, value) \
dm_write_reg_func(ctx, address, value, __func__)
reg_name ## __ ## reg_field ## _MASK,\
reg_name ## __ ## reg_field ## __SHIFT)
-uint32_t generic_reg_update_ex(const struct dc_context *ctx,
+uint32_t generic_reg_set_ex(const struct dc_context *ctx,
uint32_t addr, uint32_t reg_val, int n,
uint8_t shift1, uint32_t mask1, uint32_t field_value1, ...);
+uint32_t generic_reg_update_ex(const struct dc_context *ctx,
+ uint32_t addr, int n,
+ uint8_t shift1, uint32_t mask1, uint32_t field_value1, ...);
+
#define FD(reg_field) reg_field ## __SHIFT, \
reg_field ## _MASK
* return number of poll before condition is met
* return 0 if condition is not meet after specified time out tries
*/
-unsigned int generic_reg_wait(const struct dc_context *ctx,
+void generic_reg_wait(const struct dc_context *ctx,
uint32_t addr, uint32_t mask, uint32_t shift, uint32_t condition_value,
unsigned int delay_between_poll_us, unsigned int time_out_num_tries,
const char *func_name, int line);
#define generic_reg_update_soc15(ctx, inst_offset, reg_name, n, ...)\
generic_reg_update_ex(ctx, DCE_BASE.instance[0].segment[mm##reg_name##_BASE_IDX] + mm##reg_name + inst_offset, \
- dm_read_reg_func(ctx, mm##reg_name + DCE_BASE.instance[0].segment[mm##reg_name##_BASE_IDX] + inst_offset, __func__), \
n, __VA_ARGS__)
#define generic_reg_set_soc15(ctx, inst_offset, reg_name, n, ...)\
- generic_reg_update_ex(ctx, DCE_BASE.instance[0].segment[mm##reg_name##_BASE_IDX] + mm##reg_name + inst_offset, 0, \
+ generic_reg_set_ex(ctx, DCE_BASE.instance[0].segment[mm##reg_name##_BASE_IDX] + mm##reg_name + inst_offset, 0, \
n, __VA_ARGS__)
#define get_reg_field_value_soc15(reg_value, block, reg_num, reg_name, reg_field)\
const struct dc_context *ctx,
struct dm_pp_wm_sets_with_clock_ranges *wm_with_clock_ranges);
-void dm_pp_get_funcs_rv(struct dc_context *ctx,
- struct pp_smu_funcs_rv *funcs);
+void dm_pp_get_funcs(struct dc_context *ctx,
+ struct pp_smu_funcs *funcs);
/* DAL calls this function to notify PP about completion of Mode Set.
* For PP it means that current DCE clocks are those which were returned
#include "os_types.h"
#include "dc_types.h"
-struct pp_smu_funcs_rv;
+struct pp_smu_funcs;
struct dm_pp_clock_range {
int min_khz;
#include "display_mode_lib.h"
#include "dc_features.h"
-extern const struct _vcs_dpi_ip_params_st dcn1_0_ip;
-extern const struct _vcs_dpi_soc_bounding_box_st dcn1_0_soc;
-
-static void set_soc_bounding_box(struct _vcs_dpi_soc_bounding_box_st *soc, enum dml_project project)
-{
- switch (project) {
- case DML_PROJECT_RAVEN1:
- *soc = dcn1_0_soc;
- break;
- default:
- ASSERT(0);
- break;
- }
-}
-
-static void set_ip_params(struct _vcs_dpi_ip_params_st *ip, enum dml_project project)
+void dml_init_instance(struct display_mode_lib *lib,
+ const struct _vcs_dpi_soc_bounding_box_st *soc_bb,
+ const struct _vcs_dpi_ip_params_st *ip_params,
+ enum dml_project project)
{
- switch (project) {
- case DML_PROJECT_RAVEN1:
- *ip = dcn1_0_ip;
- break;
- default:
- ASSERT(0);
- break;
- }
-}
-
-void dml_init_instance(struct display_mode_lib *lib, enum dml_project project)
-{
- if (lib->project != project) {
- set_soc_bounding_box(&lib->soc, project);
- set_ip_params(&lib->ip, project);
- lib->project = project;
- }
+ lib->soc = *soc_bb;
+ lib->ip = *ip_params;
+ lib->project = project;
}
const char *dml_get_status_message(enum dm_validation_status status)
struct dal_logger *logger;
};
-void dml_init_instance(struct display_mode_lib *lib, enum dml_project project);
+void dml_init_instance(struct display_mode_lib *lib,
+ const struct _vcs_dpi_soc_bounding_box_st *soc_bb,
+ const struct _vcs_dpi_ip_params_st *ip_params,
+ enum dml_project project);
const char *dml_get_status_message(enum dm_validation_status status);
#ifndef __DISPLAY_MODE_STRUCTS_H__
#define __DISPLAY_MODE_STRUCTS_H__
+#define MAX_CLOCK_LIMIT_STATES 8
+
typedef struct _vcs_dpi_voltage_scaling_st voltage_scaling_st;
typedef struct _vcs_dpi_soc_bounding_box_st soc_bounding_box_st;
typedef struct _vcs_dpi_ip_params_st ip_params_st;
double xfc_xbuf_latency_tolerance_us;
int use_urgent_burst_bw;
unsigned int num_states;
- struct _vcs_dpi_voltage_scaling_st clock_limits[8];
+ struct _vcs_dpi_voltage_scaling_st clock_limits[MAX_CLOCK_LIMIT_STATES];
};
struct _vcs_dpi_ip_params_st {
unsigned int refcyc_per_vm_group_flip;
unsigned int refcyc_per_vm_req_vblank;
unsigned int refcyc_per_vm_req_flip;
+ unsigned int refcyc_per_vm_dmdata;
};
struct _vcs_dpi_display_ttu_regs_st {
dml_print(
"DML_RQ_DLG_CALC: xfc_reg_remote_surface_flip_latency = 0x%0x\n",
dlg_regs.xfc_reg_remote_surface_flip_latency);
+ dml_print(
+ "DML_RQ_DLG_CALC: refcyc_per_vm_dmdata = 0x%0x\n",
+ dlg_regs.refcyc_per_vm_dmdata);
dml_print("DML_RQ_DLG_CALC: =====================================\n");
}
struct pixel_clk_params *,
struct pll_settings *);
bool (*get_pixel_clk_frequency_100hz)(
- struct clock_source *clock_source,
+ const struct clock_source *clock_source,
unsigned int inst,
unsigned int *pixel_clk_khz);
};
DC_FAIL_BANDWIDTH_VALIDATE = 13, /* BW and Watermark validation */
DC_FAIL_SCALING = 14,
DC_FAIL_DP_LINK_TRAINING = 15,
-
+ DC_FAIL_UNSUPPORTED_1 = 18,
DC_ERROR_UNEXPECTED = -1
};
void (*link_init)(struct dc_link *link);
struct link_encoder *(*link_enc_create)(
const struct encoder_init_data *init);
-
bool (*validate_bandwidth)(
struct dc *dc,
- struct dc_state *context);
+ struct dc_state *context,
+ bool fast_validate);
enum dc_status (*validate_global)(
struct dc *dc,
struct stream_encoder *stream_enc[MAX_PIPES * 2];
struct hubbub *hubbub;
struct mpc *mpc;
- struct pp_smu_funcs_rv *pp_smu;
- struct pp_smu_display_requirement_rv pp_smu_req;
+ struct pp_smu_funcs *pp_smu;
struct dce_aux *engines[MAX_PIPES];
struct dce_i2c_hw *hw_i2cs[MAX_PIPES];
struct dce_i2c_sw *sw_i2cs[MAX_PIPES];
unsigned int pipe_count;
unsigned int underlay_pipe_index;
unsigned int stream_enc_count;
- unsigned int ref_clock_inKhz;
+
+ struct {
+ unsigned int xtalin_clock_inKhz;
+ unsigned int dccg_ref_clock_inKhz;
+ unsigned int dchub_ref_clock_inKhz;
+ } ref_clocks;
unsigned int timing_generator_count;
/*
struct dcn_watermark_set watermarks;
};
-union bw_context {
+union bw_output {
struct dcn_bw_output dcn;
struct dce_bw_output dce;
};
+struct bw_context {
+ union bw_output bw;
+ struct display_mode_lib dml;
+};
/**
* struct dc_state - The full description of a state requested by a user
*
* @streams: Stream properties
* @stream_status: The planes on a given stream
* @res_ctx: Persistent state of resources
- * @bw: The output from bandwidth and watermark calculations
+ * @bw_ctx: The output from bandwidth and watermark calculations and the DML
* @pp_display_cfg: PowerPlay clocks and settings
* @dcn_bw_vars: non-stack memory to support bandwidth calculations
*
struct resource_context res_ctx;
- union bw_context bw;
+ struct bw_context bw_ctx;
/* Note: these are big structures, do *not* put on stack! */
struct dm_pp_display_configuration pp_display_cfg;
struct dcn_bw_internal_vars dcn_bw_vars;
#endif
- struct clk_mgr *dccg;
+ struct clk_mgr *clk_mgr;
+
+ struct {
+ bool full_update_needed : 1;
+ } commit_hints;
struct kref refcount;
};
uint8_t *read_buf,
uint32_t read_size);
-int dc_link_aux_transfer(struct ddc_service *ddc,
- struct aux_payload *payload);
+int dc_link_aux_transfer_raw(struct ddc_service *ddc,
+ struct aux_payload *payload,
+ enum aux_channel_operation_result *operation_result);
bool dc_link_aux_transfer_with_retries(struct ddc_service *ddc,
struct aux_payload *payload);
bool dcn_validate_bandwidth(
struct dc *dc,
- struct dc_state *context);
+ struct dc_state *context,
+ bool fast_validate);
unsigned int dcn_find_dcfclk_suits_all(
const struct dc *dc,
void dcn_bw_notify_pplib_of_wm_ranges(struct dc *dc);
void dcn_bw_sync_calcs_and_dml(struct dc *dc);
+enum source_macro_tile_size swizzle_mode_to_macro_tile_size(enum swizzle_mode_values sw_mode);
+
#endif /* __DCN_CALCS_H__ */
bool safe_to_lower);
int (*get_dp_ref_clk_frequency)(struct clk_mgr *clk_mgr);
+
+ void (*init_clocks)(struct clk_mgr *clk_mgr);
};
#endif /* __DAL_CLK_MGR_H__ */
void (*update_dpp_dto)(struct dccg *dccg,
int dpp_inst,
int req_dppclk);
+ void (*get_dccg_ref_freq)(struct dccg *dccg,
+ unsigned int xtalin_freq_inKhz,
+ unsigned int *dccg_ref_freq_inKhz);
+ void (*dccg_init)(struct dccg *dccg);
};
#endif //__DAL_DCCG_H__
void (*wm_read_state)(struct hubbub *hubbub,
struct dcn_hubbub_wm *wm);
+
+ void (*get_dchub_ref_freq)(struct hubbub *hubbub,
+ unsigned int dccg_ref_freq_inKhz,
+ unsigned int *dchub_ref_freq_inKhz);
+
+ void (*program_watermarks)(
+ struct hubbub *hubbub,
+ struct dcn_watermark_set *watermarks,
+ unsigned int refclk_mhz,
+ bool safe_to_lower);
};
struct hubbub {
void (*get_psr_wait_loop)(struct dmcu *dmcu,
unsigned int *psr_wait_loop_number);
bool (*is_dmcu_initialized)(struct dmcu *dmcu);
+ bool (*lock_phy)(struct dmcu *dmcu);
+ bool (*unlock_phy)(struct dmcu *dmcu);
};
#endif
void (*hubp_clear_underflow)(struct hubp *hubp);
void (*hubp_disable_control)(struct hubp *hubp, bool disable_hubp);
unsigned int (*hubp_get_underflow_status)(struct hubp *hubp);
+ void (*hubp_init)(struct hubp *hubp);
};
uint16_t regval[12];
};
+enum gamut_remap_select {
+ GAMUT_REMAP_BYPASS = 0,
+ GAMUT_REMAP_COEFF,
+ GAMUT_REMAP_COMA_COEFF,
+ GAMUT_REMAP_COMB_COEFF
+};
enum opp_regamma {
OPP_REGAMMA_BYPASS = 0,
struct dc_info_packet vsc;
/* HDR Static MetaData */
struct dc_info_packet hdrsmd;
+ /* custom sdp message */
+ struct dc_info_packet dpsdp;
};
struct encoder_unblank_param {
struct dc_link_settings link_settings;
- unsigned int pixel_clk_khz;
+ struct dc_crtc_timing timing;
};
struct encoder_set_dp_phy_pattern_param {
void (*dp_set_stream_attribute)(
struct stream_encoder *enc,
struct dc_crtc_timing *crtc_timing,
- enum dc_color_space output_color_space);
+ enum dc_color_space output_color_space,
+ uint32_t enable_sdp_splitting);
void (*hdmi_set_stream_attribute)(
struct stream_encoder *enc,
bool (*did_triggered_reset_occur)(struct timing_generator *tg);
void (*setup_global_swap_lock)(struct timing_generator *tg,
const struct dcp_gsl_params *gsl_params);
+ void (*setup_global_lock)(struct timing_generator *tg);
void (*unlock)(struct timing_generator *tg);
void (*lock)(struct timing_generator *tg);
+ void (*lock_global)(struct timing_generator *tg);
void (*enable_reset_trigger)(struct timing_generator *tg,
int source_tg_inst);
void (*enable_crtc_reset)(struct timing_generator *tg,
struct dc *dc,
struct pipe_ctx *pipe,
bool lock);
+ void (*pipe_control_lock_global)(
+ struct dc *dc,
+ struct pipe_ctx *pipe,
+ bool lock);
void (*blank_pixel_data)(
struct dc *dc,
struct pipe_ctx *pipe_ctx,
/* macro to set register fields. */
#define REG_SET_N(reg_name, n, initial_val, ...) \
- generic_reg_update_ex(CTX, \
+ generic_reg_set_ex(CTX, \
REG(reg_name), \
initial_val, \
n, __VA_ARGS__)
#define REG_UPDATE_N(reg_name, n, ...) \
generic_reg_update_ex(CTX, \
REG(reg_name), \
- REG_READ(reg_name), \
n, __VA_ARGS__)
#define REG_UPDATE(reg_name, field, val) \
/* macro to update a register field to specified values in given sequences.
* useful when toggling bits
*/
-#define REG_UPDATE_SEQ(reg, field, value1, value2) \
-{ uint32_t val = REG_UPDATE(reg, field, value1); \
- REG_SET(reg, val, field, value2); }
-
-/* macro to update fields in register 1 field at a time in given order */
-#define REG_UPDATE_1BY1_2(reg, f1, v1, f2, v2) \
+#define REG_UPDATE_SEQ_2(reg, f1, v1, f2, v2) \
{ uint32_t val = REG_UPDATE(reg, f1, v1); \
REG_SET(reg, val, f2, v2); }
-#define REG_UPDATE_1BY1_3(reg, f1, v1, f2, v2, f3, v3) \
+#define REG_UPDATE_SEQ_3(reg, f1, v1, f2, v2, f3, v3) \
{ uint32_t val = REG_UPDATE(reg, f1, v1); \
val = REG_SET(reg, val, f2, v2); \
REG_SET(reg, val, f3, v3); }
struct resource_pool *pool,
const struct resource_create_funcs *create_funcs);
-struct resource_pool *dc_create_resource_pool(
- struct dc *dc,
- int num_virtual_links,
- enum dce_version dc_version,
- struct hw_asic_id asic_id);
+struct resource_pool *dc_create_resource_pool(struct dc *dc,
+ const struct dc_init_data *init_data,
+ enum dce_version dc_version);
void dc_destroy_resource_pool(struct dc *dc);
struct pipe_ctx *find_idle_secondary_pipe(
struct resource_context *res_ctx,
- const struct resource_pool *pool);
+ const struct resource_pool *pool,
+ const struct pipe_ctx *primary_pipe);
bool resource_is_stream_unchanged(
struct dc_state *old_context, struct dc_stream_state *stream);
unsigned int resource_pixel_format_to_bpp(enum surface_pixel_format format);
+struct pipe_ctx *dc_res_get_odm_bottom_pipe(struct pipe_ctx *pipe_ctx);
+bool dc_res_is_odm_head_pipe(struct pipe_ctx *pipe_ctx);
+
#endif /* DRIVERS_GPU_DRM_AMD_DC_DEV_DC_INC_RESOURCE_H_ */
.ack = NULL
};
+static const struct irq_source_info_funcs vupdate_irq_info_funcs = {
+ .set = NULL,
+ .ack = NULL
+};
+
#define hpd_int_entry(reg_num)\
[DC_IRQ_SOURCE_HPD1 + reg_num] = {\
.enable_reg = mmHPD ## reg_num ## _DC_HPD_INT_CONTROL,\
CRTC_V_UPDATE_INT_STATUS__CRTC_V_UPDATE_INT_CLEAR_MASK,\
.ack_value =\
CRTC_V_UPDATE_INT_STATUS__CRTC_V_UPDATE_INT_CLEAR_MASK,\
- .funcs = &vblank_irq_info_funcs\
+ .funcs = &vupdate_irq_info_funcs\
}
#define vblank_int_entry(reg_num)\
.ack = NULL
};
+static const struct irq_source_info_funcs vupdate_irq_info_funcs = {
+ .set = NULL,
+ .ack = NULL
+};
+
#define BASE_INNER(seg) \
DCE_BASE__INST0_SEG ## seg
IRQ_REG_ENTRY(CRTC, reg_num,\
CRTC_INTERRUPT_CONTROL, CRTC_V_UPDATE_INT_MSK,\
CRTC_V_UPDATE_INT_STATUS, CRTC_V_UPDATE_INT_CLEAR),\
- .funcs = &vblank_irq_info_funcs\
+ .funcs = &vupdate_irq_info_funcs\
}
#define vblank_int_entry(reg_num)\
.ack = NULL
};
+static const struct irq_source_info_funcs vupdate_irq_info_funcs = {
+ .set = NULL,
+ .ack = NULL
+};
#define hpd_int_entry(reg_num)\
[DC_IRQ_SOURCE_INVALID + reg_num] = {\
CRTC_V_UPDATE_INT_STATUS__CRTC_V_UPDATE_INT_CLEAR_MASK,\
.ack_value =\
CRTC_V_UPDATE_INT_STATUS__CRTC_V_UPDATE_INT_CLEAR_MASK,\
- .funcs = &vblank_irq_info_funcs\
+ .funcs = &vupdate_irq_info_funcs\
}
#define vblank_int_entry(reg_num)\
return DC_IRQ_SOURCE_VBLANK5;
case DCN_1_0__SRCID__DC_D6_OTG_VSTARTUP:
return DC_IRQ_SOURCE_VBLANK6;
+ case DCN_1_0__SRCID__OTG0_IHC_V_UPDATE_NO_LOCK_INTERRUPT:
+ return DC_IRQ_SOURCE_VUPDATE1;
+ case DCN_1_0__SRCID__OTG1_IHC_V_UPDATE_NO_LOCK_INTERRUPT:
+ return DC_IRQ_SOURCE_VUPDATE2;
+ case DCN_1_0__SRCID__OTG2_IHC_V_UPDATE_NO_LOCK_INTERRUPT:
+ return DC_IRQ_SOURCE_VUPDATE3;
+ case DCN_1_0__SRCID__OTG3_IHC_V_UPDATE_NO_LOCK_INTERRUPT:
+ return DC_IRQ_SOURCE_VUPDATE4;
+ case DCN_1_0__SRCID__OTG4_IHC_V_UPDATE_NO_LOCK_INTERRUPT:
+ return DC_IRQ_SOURCE_VUPDATE5;
+ case DCN_1_0__SRCID__OTG5_IHC_V_UPDATE_NO_LOCK_INTERRUPT:
+ return DC_IRQ_SOURCE_VUPDATE6;
case DCN_1_0__SRCID__HUBP0_FLIP_INTERRUPT:
return DC_IRQ_SOURCE_PFLIP1;
case DCN_1_0__SRCID__HUBP1_FLIP_INTERRUPT:
.ack = NULL
};
+static const struct irq_source_info_funcs vupdate_no_lock_irq_info_funcs = {
+ .set = NULL,
+ .ack = NULL
+};
+
#define BASE_INNER(seg) \
DCE_BASE__INST0_SEG ## seg
.funcs = &pflip_irq_info_funcs\
}
-#define vupdate_int_entry(reg_num)\
+/* vupdate_no_lock_int_entry maps to DC_IRQ_SOURCE_VUPDATEx, to match semantic
+ * of DCE's DC_IRQ_SOURCE_VUPDATEx.
+ */
+#define vupdate_no_lock_int_entry(reg_num)\
[DC_IRQ_SOURCE_VUPDATE1 + reg_num] = {\
IRQ_REG_ENTRY(OTG, reg_num,\
- OTG_GLOBAL_SYNC_STATUS, VUPDATE_INT_EN,\
- OTG_GLOBAL_SYNC_STATUS, VUPDATE_EVENT_CLEAR),\
- .funcs = &vblank_irq_info_funcs\
+ OTG_GLOBAL_SYNC_STATUS, VUPDATE_NO_LOCK_INT_EN,\
+ OTG_GLOBAL_SYNC_STATUS, VUPDATE_NO_LOCK_EVENT_CLEAR),\
+ .funcs = &vupdate_no_lock_irq_info_funcs\
}
#define vblank_int_entry(reg_num)\
dc_underflow_int_entry(6),
[DC_IRQ_SOURCE_DMCU_SCP] = dummy_irq_entry(),
[DC_IRQ_SOURCE_VBIOS_SW] = dummy_irq_entry(),
- vupdate_int_entry(0),
- vupdate_int_entry(1),
- vupdate_int_entry(2),
- vupdate_int_entry(3),
- vupdate_int_entry(4),
- vupdate_int_entry(5),
+ vupdate_no_lock_int_entry(0),
+ vupdate_no_lock_int_entry(1),
+ vupdate_no_lock_int_entry(2),
+ vupdate_no_lock_int_entry(3),
+ vupdate_no_lock_int_entry(4),
+ vupdate_no_lock_int_entry(5),
vblank_int_entry(0),
vblank_int_entry(1),
vblank_int_entry(2),
static void virtual_stream_encoder_dp_set_stream_attribute(
struct stream_encoder *enc,
struct dc_crtc_timing *crtc_timing,
- enum dc_color_space output_color_space) {}
+ enum dc_color_space output_color_space,
+ uint32_t enable_sdp_splitting) {}
static void virtual_stream_encoder_hdmi_set_stream_attribute(
struct stream_encoder *enc,
* fractional
*/
+unsigned int dc_fixpt_u4d19(struct fixed31_32 arg);
+
unsigned int dc_fixpt_u3d19(struct fixed31_32 arg);
unsigned int dc_fixpt_u2d19(struct fixed31_32 arg);
};
/* help functions for signal types manipulation */
+static inline bool dc_is_hdmi_tmds_signal(enum signal_type signal)
+{
+ return (signal == SIGNAL_TYPE_HDMI_TYPE_A);
+}
+
static inline bool dc_is_hdmi_signal(enum signal_type signal)
{
return (signal == SIGNAL_TYPE_HDMI_TYPE_A);
coordinates_x, axis_x, curve,
MAX_HW_POINTS, tf_pts,
mapUserRamp && ramp && ramp->type == GAMMA_RGB_256);
+ if (ramp->type == GAMMA_CUSTOM)
+ apply_lut_1d(ramp, MAX_HW_POINTS, tf_pts);
ret = true;
#define RENDER_TIMES_MAX_COUNT 10
/* Threshold to exit BTR (to avoid frequent enter-exits at the lower limit) */
#define BTR_EXIT_MARGIN 2000
+/* Threshold to change BTR multiplier (to avoid frequent changes) */
+#define BTR_DRIFT_MARGIN 2000
/*Threshold to exit fixed refresh rate*/
#define FIXED_REFRESH_EXIT_MARGIN_IN_HZ 4
/* Number of consecutive frames to check before entering/exiting fixed refresh*/
struct dc *dc;
};
+void setFieldWithMask(unsigned char *dest, unsigned int mask, unsigned int value)
+{
+ unsigned int shift = 0;
+
+ if (!mask || !dest)
+ return;
+
+ while (!((mask >> shift) & 1))
+ shift++;
+
+ //reset
+ *dest = *dest & ~mask;
+ //set
+ //dont let value span past mask
+ value = value & (mask >> shift);
+ //insert value
+ *dest = *dest | (value << shift);
+}
+
+// VTEM Byte Offset
+#define VRR_VTEM_PB0 0
+#define VRR_VTEM_PB1 1
+#define VRR_VTEM_PB2 2
+#define VRR_VTEM_PB3 3
+#define VRR_VTEM_PB4 4
+#define VRR_VTEM_PB5 5
+#define VRR_VTEM_PB6 6
+
+#define VRR_VTEM_MD0 7
+#define VRR_VTEM_MD1 8
+#define VRR_VTEM_MD2 9
+#define VRR_VTEM_MD3 10
+
+
+// VTEM Byte Masks
+//PB0
+#define MASK__VRR_VTEM_PB0__RESERVED0 0x01
+#define MASK__VRR_VTEM_PB0__SYNC 0x02
+#define MASK__VRR_VTEM_PB0__VFR 0x04
+#define MASK__VRR_VTEM_PB0__AFR 0x08
+#define MASK__VRR_VTEM_PB0__DS_TYPE 0x30
+ //0: Periodic pseudo-static EM Data Set
+ //1: Periodic dynamic EM Data Set
+ //2: Unique EM Data Set
+ //3: Reserved
+#define MASK__VRR_VTEM_PB0__END 0x40
+#define MASK__VRR_VTEM_PB0__NEW 0x80
+
+//PB1
+#define MASK__VRR_VTEM_PB1__RESERVED1 0xFF
+
+//PB2
+#define MASK__VRR_VTEM_PB2__ORGANIZATION_ID 0xFF
+ //0: This is a Vendor Specific EM Data Set
+ //1: This EM Data Set is defined by This Specification (HDMI 2.1 r102.clean)
+ //2: This EM Data Set is defined by CTA-861-G
+ //3: This EM Data Set is defined by VESA
+//PB3
+#define MASK__VRR_VTEM_PB3__DATA_SET_TAG_MSB 0xFF
+//PB4
+#define MASK__VRR_VTEM_PB4__DATA_SET_TAG_LSB 0xFF
+//PB5
+#define MASK__VRR_VTEM_PB5__DATA_SET_LENGTH_MSB 0xFF
+//PB6
+#define MASK__VRR_VTEM_PB6__DATA_SET_LENGTH_LSB 0xFF
+
+
+
+//PB7-27 (20 bytes):
+//PB7 = MD0
+#define MASK__VRR_VTEM_MD0__VRR_EN 0x01
+#define MASK__VRR_VTEM_MD0__M_CONST 0x02
+#define MASK__VRR_VTEM_MD0__RESERVED2 0x0C
+#define MASK__VRR_VTEM_MD0__FVA_FACTOR_M1 0xF0
+
+//MD1
+#define MASK__VRR_VTEM_MD1__BASE_VFRONT 0xFF
+
+//MD2
+#define MASK__VRR_VTEM_MD2__BASE_REFRESH_RATE_98 0x03
+#define MASK__VRR_VTEM_MD2__RB 0x04
+#define MASK__VRR_VTEM_MD2__RESERVED3 0xF8
+
+//MD3
+#define MASK__VRR_VTEM_MD3__BASE_REFRESH_RATE_07 0xFF
+
+
#define MOD_FREESYNC_TO_CORE(mod_freesync)\
container_of(mod_freesync, struct core_freesync, public)
unsigned int frames_to_insert = 0;
unsigned int min_frame_duration_in_ns = 0;
unsigned int max_render_time_in_us = in_out_vrr->max_duration_in_us;
+ unsigned int delta_from_mid_point_delta_in_us;
min_frame_duration_in_ns = ((unsigned int) (div64_u64(
(1000000000ULL * 1000000),
/* Choose number of frames to insert based on how close it
* can get to the mid point of the variable range.
*/
- if (delta_from_mid_point_in_us_1 < delta_from_mid_point_in_us_2)
+ if (delta_from_mid_point_in_us_1 < delta_from_mid_point_in_us_2) {
frames_to_insert = mid_point_frames_ceil;
- else
+ delta_from_mid_point_delta_in_us = delta_from_mid_point_in_us_2 -
+ delta_from_mid_point_in_us_1;
+ } else {
frames_to_insert = mid_point_frames_floor;
+ delta_from_mid_point_delta_in_us = delta_from_mid_point_in_us_1 -
+ delta_from_mid_point_in_us_2;
+ }
+
+ /* Prefer current frame multiplier when BTR is enabled unless it drifts
+ * too far from the midpoint
+ */
+ if (in_out_vrr->btr.frames_to_insert != 0 &&
+ delta_from_mid_point_delta_in_us < BTR_DRIFT_MARGIN) {
+ if (((last_render_time_in_us / in_out_vrr->btr.frames_to_insert) <
+ in_out_vrr->max_duration_in_us) &&
+ ((last_render_time_in_us / in_out_vrr->btr.frames_to_insert) >
+ in_out_vrr->min_duration_in_us))
+ frames_to_insert = in_out_vrr->btr.frames_to_insert;
+ }
/* Either we've calculated the number of frames to insert,
* or we need to insert min duration frames
inserted_frame_duration_in_us = last_render_time_in_us /
frames_to_insert;
- if (inserted_frame_duration_in_us <
- (1000000 / in_out_vrr->max_refresh_in_uhz))
- inserted_frame_duration_in_us =
- (1000000 / in_out_vrr->max_refresh_in_uhz);
+ if (inserted_frame_duration_in_us < in_out_vrr->min_duration_in_us)
+ inserted_frame_duration_in_us = in_out_vrr->min_duration_in_us;
/* Cache the calculated variables */
in_out_vrr->btr.inserted_duration_in_us =
// HB0, HB1, HB2 indicates PacketType VTEMPacket
infopacket->hb0 = 0x7F;
infopacket->hb1 = 0xC0;
- infopacket->hb2 = 0x00;
- /* HB3 Bit Fields
- * Reserved :1 = 0
- * Sync :1 = 0
- * VFR :1 = 1
- * Ds_Type :2 = 0
- * End :1 = 0
- * New :1 = 0
- */
- infopacket->hb3 = 0x20;
+ infopacket->hb2 = 0x00; //sequence_index
+
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_PB0], MASK__VRR_VTEM_PB0__VFR, 1);
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_PB2], MASK__VRR_VTEM_PB2__ORGANIZATION_ID, 1);
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_PB3], MASK__VRR_VTEM_PB3__DATA_SET_TAG_MSB, 0);
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_PB4], MASK__VRR_VTEM_PB4__DATA_SET_TAG_LSB, 1);
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_PB5], MASK__VRR_VTEM_PB5__DATA_SET_LENGTH_MSB, 0);
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_PB6], MASK__VRR_VTEM_PB6__DATA_SET_LENGTH_LSB, 4);
}
static void build_vrr_infopacket_header_v1(enum signal_type signal,
const struct mod_vrr_params *vrr,
struct dc_info_packet *infopacket)
{
- /* dc_info_packet to VtemPacket Translation of Bit-fields,
- * SB[6]
- * unsigned char VRR_EN :1
- * unsigned char M_CONST :1
- * unsigned char Reserved2 :2
- * unsigned char FVA_Factor_M1 :4
- * SB[7]
- * unsigned char Base_Vfront :8
- * SB[8]
- * unsigned char Base_Refresh_Rate_98 :2
- * unsigned char RB :1
- * unsigned char Reserved3 :5
- * SB[9]
- * unsigned char Base_RefreshRate_07 :8
- */
unsigned int fieldRateInHz;
if (vrr->state == VRR_STATE_ACTIVE_VARIABLE ||
- vrr->state == VRR_STATE_ACTIVE_FIXED){
- infopacket->sb[6] |= 0x80; //VRR_EN Bit = 1
+ vrr->state == VRR_STATE_ACTIVE_FIXED) {
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_MD0], MASK__VRR_VTEM_MD0__VRR_EN, 1);
} else {
- infopacket->sb[6] &= 0x7F; //VRR_EN Bit = 0
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_MD0], MASK__VRR_VTEM_MD0__VRR_EN, 0);
}
if (!stream->timing.vic) {
- infopacket->sb[7] = stream->timing.v_front_porch;
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_MD1], MASK__VRR_VTEM_MD1__BASE_VFRONT,
+ stream->timing.v_front_porch);
+
/* TODO: In dal2, we check mode flags for a reduced blanking timing.
* Need a way to relay that information to this function.
* if("ReducedBlanking")
* {
- * infopacket->sb[8] |= 0x20; //Set 3rd bit to 1
+ * setFieldWithMask(&infopacket->sb[VRR_VTEM_MD2], MASK__VRR_VTEM_MD2__RB, 1;
* }
*/
+
+ //TODO: DAL2 does FixPoint and rounding. Here we might need to account for that
fieldRateInHz = (stream->timing.pix_clk_100hz * 100)/
- (stream->timing.h_total * stream->timing.v_total);
+ (stream->timing.h_total * stream->timing.v_total);
- infopacket->sb[8] |= ((fieldRateInHz & 0x300) >> 2);
- infopacket->sb[9] |= fieldRateInHz & 0xFF;
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_MD2], MASK__VRR_VTEM_MD2__BASE_REFRESH_RATE_98,
+ fieldRateInHz >> 8);
+ setFieldWithMask(&infopacket->sb[VRR_VTEM_MD3], MASK__VRR_VTEM_MD3__BASE_REFRESH_RATE_07,
+ fieldRateInHz);
}
infopacket->valid = true;
{
//VTEM info packet for HdmiVrr
+ memset(infopacket, 0, sizeof(struct dc_info_packet));
+
//VTEM Packet is structured differently
build_vrr_infopacket_header_vtem(stream->signal, infopacket);
build_vrr_vtem_infopacket_data(stream, vrr, infopacket);
static const unsigned char max_reduction_table[13] = {
0xf5, 0xe5, 0xd9, 0xcd, 0xb1, 0xa5, 0xa5, 0x80, 0x65, 0x4d, 0x4d, 0x4d, 0x32};
-/* ABM 2.2 Min Reduction effectively disabled (100% for all configs)*/
+/* Possible ABM 2.2 Min Reduction configs from least aggressive to most aggressive
+ * 0 1 2 3 4 5 6 7 8 9 10 11 12
+ * 100 100 100 100 100 100 100 100 100 92.2 83.1 75.3 75.3 %
+ */
static const unsigned char min_reduction_table_v_2_2[13] = {
-0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff};
+0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xeb, 0xd4, 0xc0, 0xc0};
/* Possible ABM 2.2 Max Reduction configs from least aggressive to most aggressive
* 0 1 2 3 4 5 6 7 8 9 10 11 12
ram_table->flags = 0x0;
ram_table->deviation_gain[0] = 0xb3;
- ram_table->deviation_gain[1] = 0xb3;
- ram_table->deviation_gain[2] = 0xb3;
- ram_table->deviation_gain[3] = 0xb3;
+ ram_table->deviation_gain[1] = 0xa8;
+ ram_table->deviation_gain[2] = 0x98;
+ ram_table->deviation_gain[3] = 0x68;
ram_table->min_reduction[0][0] = min_reduction_table_v_2_2[abm_config[set][0]];
ram_table->min_reduction[1][0] = min_reduction_table_v_2_2[abm_config[set][0]];
ram_table->contrastFactor[0] = 0x99;
ram_table->contrastFactor[1] = 0x99;
- ram_table->contrastFactor[2] = 0x99;
+ ram_table->contrastFactor[2] = 0x90;
ram_table->contrastFactor[3] = 0x80;
ram_table->iir_curve[0] = 0x65;
DC_FBC_MASK = 0x1,
};
+enum amd_dpm_forced_level;
/**
* struct amd_ip_funcs - general hooks for managing amdgpu IP Blocks
*/
enum amd_powergating_state state);
/** @get_clockgating_state: get current clockgating status */
void (*get_clockgating_state)(void *handle, u32 *flags);
+ /** @enable_umd_pstate: enable UMD powerstate */
+ int (*enable_umd_pstate)(void *handle, enum amd_dpm_forced_level *level);
};
#define mmHUBP0_DCHUBP_VMPG_CONFIG_BASE_IDX 2
#define mmHUBP0_HUBPREQ_DEBUG_DB 0x0569
#define mmHUBP0_HUBPREQ_DEBUG_DB_BASE_IDX 2
+#define mmHUBP0_HUBPREQ_DEBUG 0x056a
+#define mmHUBP0_HUBPREQ_DEBUG_BASE_IDX 2
#define mmHUBP0_HUBP_MEASURE_WIN_CTRL_DCFCLK 0x056e
#define mmHUBP0_HUBP_MEASURE_WIN_CTRL_DCFCLK_BASE_IDX 2
#define mmHUBP0_HUBP_MEASURE_WIN_CTRL_DPPCLK 0x056f
#define mmHUBP1_DCHUBP_VMPG_CONFIG_BASE_IDX 2
#define mmHUBP1_HUBPREQ_DEBUG_DB 0x062d
#define mmHUBP1_HUBPREQ_DEBUG_DB_BASE_IDX 2
+#define mmHUBP1_HUBPREQ_DEBUG 0x062e
+#define mmHUBP1_HUBPREQ_DEBUG_BASE_IDX 2
#define mmHUBP1_HUBP_MEASURE_WIN_CTRL_DCFCLK 0x0632
#define mmHUBP1_HUBP_MEASURE_WIN_CTRL_DCFCLK_BASE_IDX 2
#define mmHUBP1_HUBP_MEASURE_WIN_CTRL_DPPCLK 0x0633
#define mmHUBP2_DCHUBP_VMPG_CONFIG_BASE_IDX 2
#define mmHUBP2_HUBPREQ_DEBUG_DB 0x06f1
#define mmHUBP2_HUBPREQ_DEBUG_DB_BASE_IDX 2
+#define mmHUBP2_HUBPREQ_DEBUG 0x06f2
+#define mmHUBP2_HUBPREQ_DEBUG_BASE_IDX 2
#define mmHUBP2_HUBP_MEASURE_WIN_CTRL_DCFCLK 0x06f6
#define mmHUBP2_HUBP_MEASURE_WIN_CTRL_DCFCLK_BASE_IDX 2
#define mmHUBP2_HUBP_MEASURE_WIN_CTRL_DPPCLK 0x06f7
#define mmHUBP3_DCHUBP_VMPG_CONFIG_BASE_IDX 2
#define mmHUBP3_HUBPREQ_DEBUG_DB 0x07b5
#define mmHUBP3_HUBPREQ_DEBUG_DB_BASE_IDX 2
+#define mmHUBP3_HUBPREQ_DEBUG 0x07b6
+#define mmHUBP3_HUBPREQ_DEBUG_BASE_IDX 2
#define mmHUBP3_HUBP_MEASURE_WIN_CTRL_DCFCLK 0x07ba
#define mmHUBP3_HUBP_MEASURE_WIN_CTRL_DCFCLK_BASE_IDX 2
#define mmHUBP3_HUBP_MEASURE_WIN_CTRL_DPPCLK 0x07bb
#define mmUVD_GPCOM_VCPU_DATA0_BASE_IDX 1
#define mmUVD_GPCOM_VCPU_DATA1 0x03c5
#define mmUVD_GPCOM_VCPU_DATA1_BASE_IDX 1
+#define mmUVD_ENGINE_CNTL 0x03c6
+#define mmUVD_ENGINE_CNTL_BASE_IDX 1
#define mmUVD_UDEC_DBW_UV_ADDR_CONFIG 0x03d2
#define mmUVD_UDEC_DBW_UV_ADDR_CONFIG_BASE_IDX 1
#define mmUVD_UDEC_ADDR_CONFIG 0x03d3
//UVD_GPCOM_VCPU_DATA1
#define UVD_GPCOM_VCPU_DATA1__DATA1__SHIFT 0x0
#define UVD_GPCOM_VCPU_DATA1__DATA1_MASK 0xFFFFFFFFL
+//UVD_ENGINE_CNTL
+#define UVD_ENGINE_CNTL__ENGINE_START_MASK 0x1
+#define UVD_ENGINE_CNTL__ENGINE_START__SHIFT 0x0
+#define UVD_ENGINE_CNTL__ENGINE_START_MODE_MASK 0x2
+#define UVD_ENGINE_CNTL__ENGINE_START_MODE__SHIFT 0x1
//UVD_UDEC_DBW_UV_ADDR_CONFIG
#define UVD_UDEC_DBW_UV_ADDR_CONFIG__NUM_PIPES__SHIFT 0x0
#define UVD_UDEC_DBW_UV_ADDR_CONFIG__PIPE_INTERLEAVE_SIZE__SHIFT 0x3
ATOM_FIRMWARE_CAP_FIRMWARE_POSTED = 0x00000001,
ATOM_FIRMWARE_CAP_GPU_VIRTUALIZATION = 0x00000002,
ATOM_FIRMWARE_CAP_WMI_SUPPORT = 0x00000040,
+ ATOM_FIRMWARE_CAP_HWEMU_ENABLE = 0x00000080,
+ ATOM_FIRMWARE_CAP_HWEMU_UMC_CFG = 0x00000100,
+ ATOM_FIRMWARE_CAP_SRAM_ECC = 0x00000200,
};
enum atom_cooling_solution_id{
uint32_t reserved2[3];
};
+struct atom_firmware_info_v3_3
+{
+ struct atom_common_table_header table_header;
+ uint32_t firmware_revision;
+ uint32_t bootup_sclk_in10khz;
+ uint32_t bootup_mclk_in10khz;
+ uint32_t firmware_capability; // enum atombios_firmware_capability
+ uint32_t main_call_parser_entry; /* direct address of main parser call in VBIOS binary. */
+ uint32_t bios_scratch_reg_startaddr; // 1st bios scratch register dword address
+ uint16_t bootup_vddc_mv;
+ uint16_t bootup_vddci_mv;
+ uint16_t bootup_mvddc_mv;
+ uint16_t bootup_vddgfx_mv;
+ uint8_t mem_module_id;
+ uint8_t coolingsolution_id; /*0: Air cooling; 1: Liquid cooling ... */
+ uint8_t reserved1[2];
+ uint32_t mc_baseaddr_high;
+ uint32_t mc_baseaddr_low;
+ uint8_t board_i2c_feature_id; // enum of atom_board_i2c_feature_id_def
+ uint8_t board_i2c_feature_gpio_id; // i2c id find in gpio_lut data table gpio_id
+ uint8_t board_i2c_feature_slave_addr;
+ uint8_t reserved3;
+ uint16_t bootup_mvddq_mv;
+ uint16_t bootup_mvpp_mv;
+ uint32_t zfbstartaddrin16mb;
+ uint32_t pplib_pptable_id; // if pplib_pptable_id!=0, pplib get powerplay table inside driver instead of from VBIOS
+ uint32_t reserved2[2];
+};
+
/*
***************************************************************************
Data Table lcd_info structure
ATOM_ENCODER_CAP_RECORD_HBR2_EN =0x02, // DP1.2 HBR2 setting is qualified and HBR2 can be enabled
ATOM_ENCODER_CAP_RECORD_HDMI6Gbps_EN =0x04, // HDMI2.0 6Gbps enable or not.
ATOM_ENCODER_CAP_RECORD_HBR3_EN =0x08, // DP1.3 HBR3 is supported by board.
+ ATOM_ENCODER_CAP_RECORD_USB_C_TYPE =0x100, // the DP connector is a USB-C type.
};
struct atom_encoder_caps_record
uint32_t rm21_sram_vmin_value;
};
-struct atom_gfx_info_v2_4 {
+struct atom_gfx_info_v2_4
+{
struct atom_common_table_header table_header;
uint8_t gfxip_min_ver;
uint8_t gfxip_max_ver;
- uint8_t gc_num_se;
- uint8_t max_tile_pipes;
- uint8_t gc_num_cu_per_sh;
- uint8_t gc_num_sh_per_se;
- uint8_t gc_num_rb_per_se;
- uint8_t gc_num_tccs;
+ uint8_t max_shader_engines;
+ uint8_t reserved;
+ uint8_t max_cu_per_sh;
+ uint8_t max_sh_per_se;
+ uint8_t max_backends_per_se;
+ uint8_t max_texture_channel_caches;
uint32_t regaddr_cp_dma_src_addr;
uint32_t regaddr_cp_dma_src_addr_hi;
uint32_t regaddr_cp_dma_dst_addr;
uint32_t mem_refclk_10khz;
};
+// umc_info.umc_config
+enum atom_umc_config_def {
+ UMC_CONFIG__ENABLE_1KB_INTERLEAVE_MODE = 0x00000001,
+ UMC_CONFIG__DEFAULT_MEM_ECC_ENABLE = 0x00000002,
+ UMC_CONFIG__ENABLE_HBM_LANE_REPAIR = 0x00000004,
+ UMC_CONFIG__ENABLE_BANK_HARVESTING = 0x00000008,
+ UMC_CONFIG__ENABLE_PHY_REINIT = 0x00000010,
+ UMC_CONFIG__DISABLE_UCODE_CHKSTATUS = 0x00000020,
+};
+
+struct atom_umc_info_v3_2
+{
+ struct atom_common_table_header table_header;
+ uint32_t ucode_version;
+ uint32_t ucode_rom_startaddr;
+ uint32_t ucode_length;
+ uint16_t umc_reg_init_offset;
+ uint16_t customer_ucode_name_offset;
+ uint16_t mclk_ss_percentage;
+ uint16_t mclk_ss_rate_10hz;
+ uint8_t umcip_min_ver;
+ uint8_t umcip_max_ver;
+ uint8_t vram_type; //enum of atom_dgpu_vram_type
+ uint8_t umc_config;
+ uint32_t mem_refclk_10khz;
+ uint32_t pstate_uclk_10khz[4];
+ uint16_t umcgoldenoffset;
+ uint16_t densitygoldenoffset;
+};
+
+struct atom_umc_info_v3_3
+{
+ struct atom_common_table_header table_header;
+ uint32_t ucode_reserved;
+ uint32_t ucode_rom_startaddr;
+ uint32_t ucode_length;
+ uint16_t umc_reg_init_offset;
+ uint16_t customer_ucode_name_offset;
+ uint16_t mclk_ss_percentage;
+ uint16_t mclk_ss_rate_10hz;
+ uint8_t umcip_min_ver;
+ uint8_t umcip_max_ver;
+ uint8_t vram_type; //enum of atom_dgpu_vram_type
+ uint8_t umc_config;
+ uint32_t mem_refclk_10khz;
+ uint32_t pstate_uclk_10khz[4];
+ uint16_t umcgoldenoffset;
+ uint16_t densitygoldenoffset;
+ uint32_t reserved[4];
+};
/*
***************************************************************************
KGD_POOL_FRAMEBUFFER = 3,
};
-enum kgd_engine_type {
- KGD_ENGINE_PFP = 1,
- KGD_ENGINE_ME,
- KGD_ENGINE_CE,
- KGD_ENGINE_MEC1,
- KGD_ENGINE_MEC2,
- KGD_ENGINE_RLC,
- KGD_ENGINE_SDMA1,
- KGD_ENGINE_SDMA2,
- KGD_ENGINE_MAX
-};
-
/**
* enum kfd_sched_policy
*
* @hqd_sdma_destroy: Destructs and preempts the SDMA queue assigned to that
* SDMA hqd slot.
*
- * @get_fw_version: Returns FW versions from the header
- *
* @set_scratch_backing_va: Sets VA for scratch backing memory of a VMID.
* Only used for no cp scheduling mode
*
struct kgd_dev *kgd,
uint8_t vmid);
- uint16_t (*get_fw_version)(struct kgd_dev *kgd,
- enum kgd_engine_type type);
void (*set_scratch_backing_va)(struct kgd_dev *kgd,
uint64_t va, uint32_t vmid);
int (*get_tile_config)(struct kgd_dev *kgd, struct tile_config *config);
+++ /dev/null
-/*
- * Copyright 2017 Advanced Micro Devices, Inc.
- *
- * Permission is hereby granted, free of charge, to any person obtaining a
- * copy of this software and associated documentation files (the "Software"),
- * to deal in the Software without restriction, including without limitation
- * the rights to use, copy, modify, merge, publish, distribute, sublicense,
- * and/or sell copies of the Software, and to permit persons to whom the
- * Software is furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
- * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
- * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
- * OTHER DEALINGS IN THE SOFTWARE.
- *
- */
-
-#ifndef _LINUX_CHASH_H
-#define _LINUX_CHASH_H
-
-#include <linux/types.h>
-#include <linux/hash.h>
-#include <linux/bug.h>
-#include <asm/bitsperlong.h>
-
-#if BITS_PER_LONG == 32
-# define _CHASH_LONG_SHIFT 5
-#elif BITS_PER_LONG == 64
-# define _CHASH_LONG_SHIFT 6
-#else
-# error "Unexpected BITS_PER_LONG"
-#endif
-
-struct __chash_table {
- u8 bits;
- u8 key_size;
- unsigned int value_size;
- u32 size_mask;
- unsigned long *occup_bitmap, *valid_bitmap;
- union {
- u32 *keys32;
- u64 *keys64;
- };
- u8 *values;
-
-#ifdef CONFIG_CHASH_STATS
- u64 hits, hits_steps, hits_time_ns;
- u64 miss, miss_steps, miss_time_ns;
- u64 relocs, reloc_dist;
-#endif
-};
-
-#define __CHASH_BITMAP_SIZE(bits) \
- (((1 << (bits)) + BITS_PER_LONG - 1) / BITS_PER_LONG)
-#define __CHASH_ARRAY_SIZE(bits, size) \
- ((((size) << (bits)) + sizeof(long) - 1) / sizeof(long))
-
-#define __CHASH_DATA_SIZE(bits, key_size, value_size) \
- (__CHASH_BITMAP_SIZE(bits) * 2 + \
- __CHASH_ARRAY_SIZE(bits, key_size) + \
- __CHASH_ARRAY_SIZE(bits, value_size))
-
-#define STRUCT_CHASH_TABLE(bits, key_size, value_size) \
- struct { \
- struct __chash_table table; \
- unsigned long data \
- [__CHASH_DATA_SIZE(bits, key_size, value_size)];\
- }
-
-/**
- * struct chash_table - Dynamically allocated closed hash table
- *
- * Use this struct for dynamically allocated hash tables (using
- * chash_table_alloc and chash_table_free), where the size is
- * determined at runtime.
- */
-struct chash_table {
- struct __chash_table table;
- unsigned long *data;
-};
-
-/**
- * DECLARE_CHASH_TABLE - macro to declare a closed hash table
- * @table: name of the declared hash table
- * @bts: Table size will be 2^bits entries
- * @key_sz: Size of hash keys in bytes, 4 or 8
- * @val_sz: Size of data values in bytes, can be 0
- *
- * This declares the hash table variable with a static size.
- *
- * The closed hash table stores key-value pairs with low memory and
- * lookup overhead. In operation it performs no dynamic memory
- * management. The data being stored does not require any
- * list_heads. The hash table performs best with small @val_sz and as
- * long as some space (about 50%) is left free in the table. But the
- * table can still work reasonably efficiently even when filled up to
- * about 90%. If bigger data items need to be stored and looked up,
- * store the pointer to it as value in the hash table.
- *
- * @val_sz may be 0. This can be useful when all the stored
- * information is contained in the key itself and the fact that it is
- * in the hash table (or not).
- */
-#define DECLARE_CHASH_TABLE(table, bts, key_sz, val_sz) \
- STRUCT_CHASH_TABLE(bts, key_sz, val_sz) table
-
-#ifdef CONFIG_CHASH_STATS
-#define __CHASH_STATS_INIT(prefix), \
- prefix.hits = 0, \
- prefix.hits_steps = 0, \
- prefix.hits_time_ns = 0, \
- prefix.miss = 0, \
- prefix.miss_steps = 0, \
- prefix.miss_time_ns = 0, \
- prefix.relocs = 0, \
- prefix.reloc_dist = 0
-#else
-#define __CHASH_STATS_INIT(prefix)
-#endif
-
-#define __CHASH_TABLE_INIT(prefix, data, bts, key_sz, val_sz) \
- prefix.bits = (bts), \
- prefix.key_size = (key_sz), \
- prefix.value_size = (val_sz), \
- prefix.size_mask = ((1 << bts) - 1), \
- prefix.occup_bitmap = &data[0], \
- prefix.valid_bitmap = &data \
- [__CHASH_BITMAP_SIZE(bts)], \
- prefix.keys64 = (u64 *)&data \
- [__CHASH_BITMAP_SIZE(bts) * 2], \
- prefix.values = (u8 *)&data \
- [__CHASH_BITMAP_SIZE(bts) * 2 + \
- __CHASH_ARRAY_SIZE(bts, key_sz)] \
- __CHASH_STATS_INIT(prefix)
-
-/**
- * DEFINE_CHASH_TABLE - macro to define and initialize a closed hash table
- * @tbl: name of the declared hash table
- * @bts: Table size will be 2^bits entries
- * @key_sz: Size of hash keys in bytes, 4 or 8
- * @val_sz: Size of data values in bytes, can be 0
- *
- * Note: the macro can be used for global and local hash table variables.
- */
-#define DEFINE_CHASH_TABLE(tbl, bts, key_sz, val_sz) \
- DECLARE_CHASH_TABLE(tbl, bts, key_sz, val_sz) = { \
- .table = { \
- __CHASH_TABLE_INIT(, (tbl).data, bts, key_sz, val_sz) \
- }, \
- .data = {0} \
- }
-
-/**
- * INIT_CHASH_TABLE - Initialize a hash table declared by DECLARE_CHASH_TABLE
- * @tbl: name of the declared hash table
- * @bts: Table size will be 2^bits entries
- * @key_sz: Size of hash keys in bytes, 4 or 8
- * @val_sz: Size of data values in bytes, can be 0
- */
-#define INIT_CHASH_TABLE(tbl, bts, key_sz, val_sz) \
- __CHASH_TABLE_INIT(((tbl).table), (tbl).data, bts, key_sz, val_sz)
-
-int chash_table_alloc(struct chash_table *table, u8 bits, u8 key_size,
- unsigned int value_size, gfp_t gfp_mask);
-void chash_table_free(struct chash_table *table);
-
-/**
- * chash_table_dump_stats - Dump statistics of a closed hash table
- * @tbl: Pointer to the table structure
- *
- * Dumps some performance statistics of the table gathered in operation
- * in the kernel log using pr_debug. If CONFIG_DYNAMIC_DEBUG is enabled,
- * user must turn on messages for chash.c (file chash.c +p).
- */
-#ifdef CONFIG_CHASH_STATS
-#define chash_table_dump_stats(tbl) __chash_table_dump_stats(&(*tbl).table)
-
-void __chash_table_dump_stats(struct __chash_table *table);
-#else
-#define chash_table_dump_stats(tbl)
-#endif
-
-/**
- * chash_table_reset_stats - Reset statistics of a closed hash table
- * @tbl: Pointer to the table structure
- */
-#ifdef CONFIG_CHASH_STATS
-#define chash_table_reset_stats(tbl) __chash_table_reset_stats(&(*tbl).table)
-
-static inline void __chash_table_reset_stats(struct __chash_table *table)
-{
- (void)table __CHASH_STATS_INIT((*table));
-}
-#else
-#define chash_table_reset_stats(tbl)
-#endif
-
-/**
- * chash_table_copy_in - Copy a new value into the hash table
- * @tbl: Pointer to the table structure
- * @key: Key of the entry to add or update
- * @value: Pointer to value to copy, may be NULL
- *
- * If @key already has an entry, its value is replaced. Otherwise a
- * new entry is added. If @value is NULL, the value is left unchanged
- * or uninitialized. Returns 1 if an entry already existed, 0 if a new
- * entry was added or %-ENOMEM if there was no free space in the
- * table.
- */
-#define chash_table_copy_in(tbl, key, value) \
- __chash_table_copy_in(&(*tbl).table, key, value)
-
-int __chash_table_copy_in(struct __chash_table *table, u64 key,
- const void *value);
-
-/**
- * chash_table_copy_out - Copy a value out of the hash table
- * @tbl: Pointer to the table structure
- * @key: Key of the entry to find
- * @value: Pointer to value to copy, may be NULL
- *
- * If @value is not NULL and the table has a non-0 value_size, the
- * value at @key is copied to @value. Returns the slot index of the
- * entry or %-EINVAL if @key was not found.
- */
-#define chash_table_copy_out(tbl, key, value) \
- __chash_table_copy_out(&(*tbl).table, key, value, false)
-
-int __chash_table_copy_out(struct __chash_table *table, u64 key,
- void *value, bool remove);
-
-/**
- * chash_table_remove - Remove an entry from the hash table
- * @tbl: Pointer to the table structure
- * @key: Key of the entry to find
- * @value: Pointer to value to copy, may be NULL
- *
- * If @value is not NULL and the table has a non-0 value_size, the
- * value at @key is copied to @value. The entry is removed from the
- * table. Returns the slot index of the removed entry or %-EINVAL if
- * @key was not found.
- */
-#define chash_table_remove(tbl, key, value) \
- __chash_table_copy_out(&(*tbl).table, key, value, true)
-
-/*
- * Low level iterator API used internally by the above functions.
- */
-struct chash_iter {
- struct __chash_table *table;
- unsigned long mask;
- int slot;
-};
-
-/**
- * CHASH_ITER_INIT - Initialize a hash table iterator
- * @tbl: Pointer to hash table to iterate over
- * @s: Initial slot number
- */
-#define CHASH_ITER_INIT(table, s) { \
- table, \
- 1UL << ((s) & (BITS_PER_LONG - 1)), \
- s \
- }
-/**
- * CHASH_ITER_SET - Set hash table iterator to new slot
- * @iter: Iterator
- * @s: Slot number
- */
-#define CHASH_ITER_SET(iter, s) \
- (iter).mask = 1UL << ((s) & (BITS_PER_LONG - 1)), \
- (iter).slot = (s)
-/**
- * CHASH_ITER_INC - Increment hash table iterator
- * @table: Hash table to iterate over
- *
- * Wraps around at the end.
- */
-#define CHASH_ITER_INC(iter) do { \
- (iter).mask = (iter).mask << 1 | \
- (iter).mask >> (BITS_PER_LONG - 1); \
- (iter).slot = ((iter).slot + 1) & (iter).table->size_mask; \
- } while (0)
-
-static inline bool chash_iter_is_valid(const struct chash_iter iter)
-{
- BUG_ON((unsigned)iter.slot >= (1 << iter.table->bits));
- return !!(iter.table->valid_bitmap[iter.slot >> _CHASH_LONG_SHIFT] &
- iter.mask);
-}
-static inline bool chash_iter_is_empty(const struct chash_iter iter)
-{
- BUG_ON((unsigned)iter.slot >= (1 << iter.table->bits));
- return !(iter.table->occup_bitmap[iter.slot >> _CHASH_LONG_SHIFT] &
- iter.mask);
-}
-
-static inline void chash_iter_set_valid(const struct chash_iter iter)
-{
- BUG_ON((unsigned)iter.slot >= (1 << iter.table->bits));
- iter.table->valid_bitmap[iter.slot >> _CHASH_LONG_SHIFT] |= iter.mask;
- iter.table->occup_bitmap[iter.slot >> _CHASH_LONG_SHIFT] |= iter.mask;
-}
-static inline void chash_iter_set_invalid(const struct chash_iter iter)
-{
- BUG_ON((unsigned)iter.slot >= (1 << iter.table->bits));
- iter.table->valid_bitmap[iter.slot >> _CHASH_LONG_SHIFT] &= ~iter.mask;
-}
-static inline void chash_iter_set_empty(const struct chash_iter iter)
-{
- BUG_ON((unsigned)iter.slot >= (1 << iter.table->bits));
- iter.table->occup_bitmap[iter.slot >> _CHASH_LONG_SHIFT] &= ~iter.mask;
-}
-
-static inline u32 chash_iter_key32(const struct chash_iter iter)
-{
- BUG_ON(iter.table->key_size != 4);
- BUG_ON((unsigned)iter.slot >= (1 << iter.table->bits));
- return iter.table->keys32[iter.slot];
-}
-static inline u64 chash_iter_key64(const struct chash_iter iter)
-{
- BUG_ON(iter.table->key_size != 8);
- BUG_ON((unsigned)iter.slot >= (1 << iter.table->bits));
- return iter.table->keys64[iter.slot];
-}
-static inline u64 chash_iter_key(const struct chash_iter iter)
-{
- BUG_ON((unsigned)iter.slot >= (1 << iter.table->bits));
- return (iter.table->key_size == 4) ?
- iter.table->keys32[iter.slot] : iter.table->keys64[iter.slot];
-}
-
-static inline u32 chash_iter_hash32(const struct chash_iter iter)
-{
- BUG_ON(iter.table->key_size != 4);
- return hash_32(chash_iter_key32(iter), iter.table->bits);
-}
-
-static inline u32 chash_iter_hash64(const struct chash_iter iter)
-{
- BUG_ON(iter.table->key_size != 8);
- return hash_64(chash_iter_key64(iter), iter.table->bits);
-}
-
-static inline u32 chash_iter_hash(const struct chash_iter iter)
-{
- return (iter.table->key_size == 4) ?
- hash_32(chash_iter_key32(iter), iter.table->bits) :
- hash_64(chash_iter_key64(iter), iter.table->bits);
-}
-
-static inline void *chash_iter_value(const struct chash_iter iter)
-{
- BUG_ON((unsigned)iter.slot >= (1 << iter.table->bits));
- return iter.table->values +
- ((unsigned long)iter.slot * iter.table->value_size);
-}
-
-#endif /* _LINUX_CHASH_H */
+++ /dev/null
-menu "AMD Library routines"
-
-#
-# Closed hash table
-#
-config CHASH
- tristate
- default DRM_AMDGPU
- help
- Statically sized closed hash table implementation with low
- memory and CPU overhead.
-
-config CHASH_STATS
- bool "Closed hash table performance statistics"
- depends on CHASH
- default n
- help
- Enable collection of performance statistics for closed hash tables.
-
-config CHASH_SELFTEST
- bool "Closed hash table self test"
- depends on CHASH
- default n
- help
- Runs a selftest during module load. Several module parameters
- are available to modify the behaviour of the test.
-
-endmenu
+++ /dev/null
-#
-# Copyright 2017 Advanced Micro Devices, Inc.
-#
-# Permission is hereby granted, free of charge, to any person obtaining a
-# copy of this software and associated documentation files (the "Software"),
-# to deal in the Software without restriction, including without limitation
-# the rights to use, copy, modify, merge, publish, distribute, sublicense,
-# and/or sell copies of the Software, and to permit persons to whom the
-# Software is furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
-# THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
-# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
-# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
-# OTHER DEALINGS IN THE SOFTWARE.
-#
-#
-# Makefile for AMD library routines, which are used by AMD driver
-# components.
-#
-# This is for common library routines that can be shared between AMD
-# driver components or later moved to kernel/lib for sharing with
-# other drivers.
-
-ccflags-y := -I$(src)/../include
-
-obj-$(CONFIG_CHASH) += chash.o
+++ /dev/null
-/*
- * Copyright 2017 Advanced Micro Devices, Inc.
- *
- * Permission is hereby granted, free of charge, to any person obtaining a
- * copy of this software and associated documentation files (the "Software"),
- * to deal in the Software without restriction, including without limitation
- * the rights to use, copy, modify, merge, publish, distribute, sublicense,
- * and/or sell copies of the Software, and to permit persons to whom the
- * Software is furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
- * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
- * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
- * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
- * OTHER DEALINGS IN THE SOFTWARE.
- *
- */
-
-#include <linux/types.h>
-#include <linux/hash.h>
-#include <linux/bug.h>
-#include <linux/slab.h>
-#include <linux/module.h>
-#include <linux/sched/clock.h>
-#include <asm/div64.h>
-#include <linux/chash.h>
-
-/**
- * chash_table_alloc - Allocate closed hash table
- * @table: Pointer to the table structure
- * @bits: Table size will be 2^bits entries
- * @key_size: Size of hash keys in bytes, 4 or 8
- * @value_size: Size of data values in bytes, can be 0
- */
-int chash_table_alloc(struct chash_table *table, u8 bits, u8 key_size,
- unsigned int value_size, gfp_t gfp_mask)
-{
- if (bits > 31)
- return -EINVAL;
-
- if (key_size != 4 && key_size != 8)
- return -EINVAL;
-
- table->data = kcalloc(__CHASH_DATA_SIZE(bits, key_size, value_size),
- sizeof(long), gfp_mask);
- if (!table->data)
- return -ENOMEM;
-
- __CHASH_TABLE_INIT(table->table, table->data,
- bits, key_size, value_size);
-
- return 0;
-}
-EXPORT_SYMBOL(chash_table_alloc);
-
-/**
- * chash_table_free - Free closed hash table
- * @table: Pointer to the table structure
- */
-void chash_table_free(struct chash_table *table)
-{
- kfree(table->data);
-}
-EXPORT_SYMBOL(chash_table_free);
-
-#ifdef CONFIG_CHASH_STATS
-
-#define DIV_FRAC(nom, denom, quot, frac, frac_digits) do { \
- u64 __nom = (nom); \
- u64 __denom = (denom); \
- u64 __quot, __frac; \
- u32 __rem; \
- \
- while (__denom >> 32) { \
- __nom >>= 1; \
- __denom >>= 1; \
- } \
- __quot = __nom; \
- __rem = do_div(__quot, __denom); \
- __frac = __rem * (frac_digits) + (__denom >> 1); \
- do_div(__frac, __denom); \
- (quot) = __quot; \
- (frac) = __frac; \
- } while (0)
-
-void __chash_table_dump_stats(struct __chash_table *table)
-{
- struct chash_iter iter = CHASH_ITER_INIT(table, 0);
- u32 filled = 0, empty = 0, tombstones = 0;
- u64 quot1, quot2;
- u32 frac1, frac2;
-
- do {
- if (chash_iter_is_valid(iter))
- filled++;
- else if (chash_iter_is_empty(iter))
- empty++;
- else
- tombstones++;
- CHASH_ITER_INC(iter);
- } while (iter.slot);
-
- pr_debug("chash: key size %u, value size %u\n",
- table->key_size, table->value_size);
- pr_debug(" Slots total/filled/empty/tombstones: %u / %u / %u / %u\n",
- 1 << table->bits, filled, empty, tombstones);
- if (table->hits > 0) {
- DIV_FRAC(table->hits_steps, table->hits, quot1, frac1, 1000);
- DIV_FRAC(table->hits * 1000, table->hits_time_ns,
- quot2, frac2, 1000);
- } else {
- quot1 = quot2 = 0;
- frac1 = frac2 = 0;
- }
- pr_debug(" Hits (avg.cost, rate): %llu (%llu.%03u, %llu.%03u M/s)\n",
- table->hits, quot1, frac1, quot2, frac2);
- if (table->miss > 0) {
- DIV_FRAC(table->miss_steps, table->miss, quot1, frac1, 1000);
- DIV_FRAC(table->miss * 1000, table->miss_time_ns,
- quot2, frac2, 1000);
- } else {
- quot1 = quot2 = 0;
- frac1 = frac2 = 0;
- }
- pr_debug(" Misses (avg.cost, rate): %llu (%llu.%03u, %llu.%03u M/s)\n",
- table->miss, quot1, frac1, quot2, frac2);
- if (table->hits + table->miss > 0) {
- DIV_FRAC(table->hits_steps + table->miss_steps,
- table->hits + table->miss, quot1, frac1, 1000);
- DIV_FRAC((table->hits + table->miss) * 1000,
- (table->hits_time_ns + table->miss_time_ns),
- quot2, frac2, 1000);
- } else {
- quot1 = quot2 = 0;
- frac1 = frac2 = 0;
- }
- pr_debug(" Total (avg.cost, rate): %llu (%llu.%03u, %llu.%03u M/s)\n",
- table->hits + table->miss, quot1, frac1, quot2, frac2);
- if (table->relocs > 0) {
- DIV_FRAC(table->hits + table->miss, table->relocs,
- quot1, frac1, 1000);
- DIV_FRAC(table->reloc_dist, table->relocs, quot2, frac2, 1000);
- pr_debug(" Relocations (freq, avg.dist): %llu (1:%llu.%03u, %llu.%03u)\n",
- table->relocs, quot1, frac1, quot2, frac2);
- } else {
- pr_debug(" No relocations\n");
- }
-}
-EXPORT_SYMBOL(__chash_table_dump_stats);
-
-#undef DIV_FRAC
-#endif
-
-#define CHASH_INC(table, a) ((a) = ((a) + 1) & (table)->size_mask)
-#define CHASH_ADD(table, a, b) (((a) + (b)) & (table)->size_mask)
-#define CHASH_SUB(table, a, b) (((a) - (b)) & (table)->size_mask)
-#define CHASH_IN_RANGE(table, slot, first, last) \
- (CHASH_SUB(table, slot, first) <= CHASH_SUB(table, last, first))
-
-/*#define CHASH_DEBUG Uncomment this to enable verbose debug output*/
-#ifdef CHASH_DEBUG
-static void chash_table_dump(struct __chash_table *table)
-{
- struct chash_iter iter = CHASH_ITER_INIT(table, 0);
-
- do {
- if ((iter.slot & 3) == 0)
- pr_debug("%04x: ", iter.slot);
-
- if (chash_iter_is_valid(iter))
- pr_debug("[%016llx] ", chash_iter_key(iter));
- else if (chash_iter_is_empty(iter))
- pr_debug("[ <empty> ] ");
- else
- pr_debug("[ <tombstone> ] ");
-
- if ((iter.slot & 3) == 3)
- pr_debug("\n");
-
- CHASH_ITER_INC(iter);
- } while (iter.slot);
-
- if ((iter.slot & 3) != 0)
- pr_debug("\n");
-}
-
-static int chash_table_check(struct __chash_table *table)
-{
- u32 hash;
- struct chash_iter iter = CHASH_ITER_INIT(table, 0);
- struct chash_iter cur = CHASH_ITER_INIT(table, 0);
-
- do {
- if (!chash_iter_is_valid(iter)) {
- CHASH_ITER_INC(iter);
- continue;
- }
-
- hash = chash_iter_hash(iter);
- CHASH_ITER_SET(cur, hash);
- while (cur.slot != iter.slot) {
- if (chash_iter_is_empty(cur)) {
- pr_err("Path to element at %x with hash %x broken at slot %x\n",
- iter.slot, hash, cur.slot);
- chash_table_dump(table);
- return -EINVAL;
- }
- CHASH_ITER_INC(cur);
- }
-
- CHASH_ITER_INC(iter);
- } while (iter.slot);
-
- return 0;
-}
-#endif
-
-static void chash_iter_relocate(struct chash_iter dst, struct chash_iter src)
-{
- BUG_ON(src.table == dst.table && src.slot == dst.slot);
- BUG_ON(src.table->key_size != dst.table->key_size);
- BUG_ON(src.table->value_size != dst.table->value_size);
-
- if (dst.table->key_size == 4)
- dst.table->keys32[dst.slot] = src.table->keys32[src.slot];
- else
- dst.table->keys64[dst.slot] = src.table->keys64[src.slot];
-
- if (dst.table->value_size)
- memcpy(chash_iter_value(dst), chash_iter_value(src),
- dst.table->value_size);
-
- chash_iter_set_valid(dst);
- chash_iter_set_invalid(src);
-
-#ifdef CONFIG_CHASH_STATS
- if (src.table == dst.table) {
- dst.table->relocs++;
- dst.table->reloc_dist +=
- CHASH_SUB(dst.table, src.slot, dst.slot);
- }
-#endif
-}
-
-/**
- * __chash_table_find - Helper for looking up a hash table entry
- * @iter: Pointer to hash table iterator
- * @key: Key of the entry to find
- * @for_removal: set to true if the element will be removed soon
- *
- * Searches for an entry in the hash table with a given key. iter must
- * be initialized by the caller to point to the home position of the
- * hypothetical entry, i.e. it must be initialized with the hash table
- * and the key's hash as the initial slot for the search.
- *
- * This function also does some local clean-up to speed up future
- * look-ups by relocating entries to better slots and removing
- * tombstones that are no longer needed.
- *
- * If @for_removal is true, the function avoids relocating the entry
- * that is being returned.
- *
- * Returns 0 if the search is successful. In this case iter is updated
- * to point to the found entry. Otherwise %-EINVAL is returned and the
- * iter is updated to point to the first available slot for the given
- * key. If the table is full, the slot is set to -1.
- */
-static int chash_table_find(struct chash_iter *iter, u64 key,
- bool for_removal)
-{
-#ifdef CONFIG_CHASH_STATS
- u64 ts1 = local_clock();
-#endif
- u32 hash = iter->slot;
- struct chash_iter first_redundant = CHASH_ITER_INIT(iter->table, -1);
- int first_avail = (for_removal ? -2 : -1);
-
- while (!chash_iter_is_valid(*iter) || chash_iter_key(*iter) != key) {
- if (chash_iter_is_empty(*iter)) {
- /* Found an empty slot, which ends the
- * search. Clean up any preceding tombstones
- * that are no longer needed because they lead
- * to no-where
- */
- if ((int)first_redundant.slot < 0)
- goto not_found;
- while (first_redundant.slot != iter->slot) {
- if (!chash_iter_is_valid(first_redundant))
- chash_iter_set_empty(first_redundant);
- CHASH_ITER_INC(first_redundant);
- }
-#ifdef CHASH_DEBUG
- chash_table_check(iter->table);
-#endif
- goto not_found;
- } else if (!chash_iter_is_valid(*iter)) {
- /* Found a tombstone. Remember it as candidate
- * for relocating the entry we're looking for
- * or for adding a new entry with the given key
- */
- if (first_avail == -1)
- first_avail = iter->slot;
- /* Or mark it as the start of a series of
- * potentially redundant tombstones
- */
- else if (first_redundant.slot == -1)
- CHASH_ITER_SET(first_redundant, iter->slot);
- } else if (first_redundant.slot >= 0) {
- /* Found a valid, occupied slot with a
- * preceding series of tombstones. Relocate it
- * to a better position that no longer depends
- * on those tombstones
- */
- u32 cur_hash = chash_iter_hash(*iter);
-
- if (!CHASH_IN_RANGE(iter->table, cur_hash,
- first_redundant.slot + 1,
- iter->slot)) {
- /* This entry has a hash at or before
- * the first tombstone we found. We
- * can relocate it to that tombstone
- * and advance to the next tombstone
- */
- chash_iter_relocate(first_redundant, *iter);
- do {
- CHASH_ITER_INC(first_redundant);
- } while (chash_iter_is_valid(first_redundant));
- } else if (cur_hash != iter->slot) {
- /* Relocate entry to its home position
- * or as close as possible so it no
- * longer depends on any preceding
- * tombstones
- */
- struct chash_iter new_iter =
- CHASH_ITER_INIT(iter->table, cur_hash);
-
- while (new_iter.slot != iter->slot &&
- chash_iter_is_valid(new_iter))
- CHASH_ITER_INC(new_iter);
-
- if (new_iter.slot != iter->slot)
- chash_iter_relocate(new_iter, *iter);
- }
- }
-
- CHASH_ITER_INC(*iter);
- if (iter->slot == hash) {
- iter->slot = -1;
- goto not_found;
- }
- }
-
-#ifdef CONFIG_CHASH_STATS
- iter->table->hits++;
- iter->table->hits_steps += CHASH_SUB(iter->table, iter->slot, hash) + 1;
-#endif
-
- if (first_avail >= 0) {
- CHASH_ITER_SET(first_redundant, first_avail);
- chash_iter_relocate(first_redundant, *iter);
- iter->slot = first_redundant.slot;
- iter->mask = first_redundant.mask;
- }
-
-#ifdef CONFIG_CHASH_STATS
- iter->table->hits_time_ns += local_clock() - ts1;
-#endif
-
- return 0;
-
-not_found:
-#ifdef CONFIG_CHASH_STATS
- iter->table->miss++;
- iter->table->miss_steps += (iter->slot < 0) ?
- (1 << iter->table->bits) :
- CHASH_SUB(iter->table, iter->slot, hash) + 1;
-#endif
-
- if (first_avail >= 0)
- CHASH_ITER_SET(*iter, first_avail);
-
-#ifdef CONFIG_CHASH_STATS
- iter->table->miss_time_ns += local_clock() - ts1;
-#endif
-
- return -EINVAL;
-}
-
-int __chash_table_copy_in(struct __chash_table *table, u64 key,
- const void *value)
-{
- u32 hash = (table->key_size == 4) ?
- hash_32(key, table->bits) : hash_64(key, table->bits);
- struct chash_iter iter = CHASH_ITER_INIT(table, hash);
- int r = chash_table_find(&iter, key, false);
-
- /* Found an existing entry */
- if (!r) {
- if (value && table->value_size)
- memcpy(chash_iter_value(iter), value,
- table->value_size);
- return 1;
- }
-
- /* Is there a place to add a new entry? */
- if (iter.slot < 0) {
- pr_err("Hash table overflow\n");
- return -ENOMEM;
- }
-
- chash_iter_set_valid(iter);
-
- if (table->key_size == 4)
- table->keys32[iter.slot] = key;
- else
- table->keys64[iter.slot] = key;
- if (value && table->value_size)
- memcpy(chash_iter_value(iter), value, table->value_size);
-
- return 0;
-}
-EXPORT_SYMBOL(__chash_table_copy_in);
-
-int __chash_table_copy_out(struct __chash_table *table, u64 key,
- void *value, bool remove)
-{
- u32 hash = (table->key_size == 4) ?
- hash_32(key, table->bits) : hash_64(key, table->bits);
- struct chash_iter iter = CHASH_ITER_INIT(table, hash);
- int r = chash_table_find(&iter, key, remove);
-
- if (r < 0)
- return r;
-
- if (value && table->value_size)
- memcpy(value, chash_iter_value(iter), table->value_size);
-
- if (remove)
- chash_iter_set_invalid(iter);
-
- return iter.slot;
-}
-EXPORT_SYMBOL(__chash_table_copy_out);
-
-#ifdef CONFIG_CHASH_SELFTEST
-/**
- * chash_self_test - Run a self-test of the hash table implementation
- * @bits: Table size will be 2^bits entries
- * @key_size: Size of hash keys in bytes, 4 or 8
- * @min_fill: Minimum fill level during the test
- * @max_fill: Maximum fill level during the test
- * @iterations: Number of test iterations
- *
- * The test adds and removes entries from a hash table, cycling the
- * fill level between min_fill and max_fill entries. Also tests lookup
- * and value retrieval.
- */
-static int __init chash_self_test(u8 bits, u8 key_size,
- int min_fill, int max_fill,
- u64 iterations)
-{
- struct chash_table table;
- int ret;
- u64 add_count, rmv_count;
- u64 value;
-
- if (key_size == 4 && iterations > 0xffffffff)
- return -EINVAL;
- if (min_fill >= max_fill)
- return -EINVAL;
-
- ret = chash_table_alloc(&table, bits, key_size, sizeof(u64),
- GFP_KERNEL);
- if (ret) {
- pr_err("chash_table_alloc failed: %d\n", ret);
- return ret;
- }
-
- for (add_count = 0, rmv_count = 0; add_count < iterations;
- add_count++) {
- /* When we hit the max_fill level, remove entries down
- * to min_fill
- */
- if (add_count - rmv_count == max_fill) {
- u64 find_count = rmv_count;
-
- /* First try to find all entries that we're
- * about to remove, confirm their value, test
- * writing them back a second time.
- */
- for (; add_count - find_count > min_fill;
- find_count++) {
- ret = chash_table_copy_out(&table, find_count,
- &value);
- if (ret < 0) {
- pr_err("chash_table_copy_out failed: %d\n",
- ret);
- goto out;
- }
- if (value != ~find_count) {
- pr_err("Wrong value retrieved for key 0x%llx, expected 0x%llx got 0x%llx\n",
- find_count, ~find_count, value);
-#ifdef CHASH_DEBUG
- chash_table_dump(&table.table);
-#endif
- ret = -EFAULT;
- goto out;
- }
- ret = chash_table_copy_in(&table, find_count,
- &value);
- if (ret != 1) {
- pr_err("copy_in second time returned %d, expected 1\n",
- ret);
- ret = -EFAULT;
- goto out;
- }
- }
- /* Remove them until we hit min_fill level */
- for (; add_count - rmv_count > min_fill; rmv_count++) {
- ret = chash_table_remove(&table, rmv_count,
- NULL);
- if (ret < 0) {
- pr_err("chash_table_remove failed: %d\n",
- ret);
- goto out;
- }
- }
- }
-
- /* Add a new value */
- value = ~add_count;
- ret = chash_table_copy_in(&table, add_count, &value);
- if (ret != 0) {
- pr_err("copy_in first time returned %d, expected 0\n",
- ret);
- ret = -EFAULT;
- goto out;
- }
- }
-
- chash_table_dump_stats(&table);
- chash_table_reset_stats(&table);
-
-out:
- chash_table_free(&table);
- return ret;
-}
-
-static unsigned int chash_test_bits = 10;
-MODULE_PARM_DESC(test_bits,
- "Selftest number of hash bits ([4..20], default=10)");
-module_param_named(test_bits, chash_test_bits, uint, 0444);
-
-static unsigned int chash_test_keysize = 8;
-MODULE_PARM_DESC(test_keysize, "Selftest keysize (4 or 8, default=8)");
-module_param_named(test_keysize, chash_test_keysize, uint, 0444);
-
-static unsigned int chash_test_minfill;
-MODULE_PARM_DESC(test_minfill, "Selftest minimum #entries (default=50%)");
-module_param_named(test_minfill, chash_test_minfill, uint, 0444);
-
-static unsigned int chash_test_maxfill;
-MODULE_PARM_DESC(test_maxfill, "Selftest maximum #entries (default=80%)");
-module_param_named(test_maxfill, chash_test_maxfill, uint, 0444);
-
-static unsigned long chash_test_iters;
-MODULE_PARM_DESC(test_iters, "Selftest iterations (default=1000 x #entries)");
-module_param_named(test_iters, chash_test_iters, ulong, 0444);
-
-static int __init chash_init(void)
-{
- int ret;
- u64 ts1_ns;
-
- /* Skip self test on user errors */
- if (chash_test_bits < 4 || chash_test_bits > 20) {
- pr_err("chash: test_bits out of range [4..20].\n");
- return 0;
- }
- if (chash_test_keysize != 4 && chash_test_keysize != 8) {
- pr_err("chash: test_keysize invalid. Must be 4 or 8.\n");
- return 0;
- }
-
- if (!chash_test_minfill)
- chash_test_minfill = (1 << chash_test_bits) / 2;
- if (!chash_test_maxfill)
- chash_test_maxfill = (1 << chash_test_bits) * 4 / 5;
- if (!chash_test_iters)
- chash_test_iters = (1 << chash_test_bits) * 1000;
-
- if (chash_test_minfill >= (1 << chash_test_bits)) {
- pr_err("chash: test_minfill too big. Must be < table size.\n");
- return 0;
- }
- if (chash_test_maxfill >= (1 << chash_test_bits)) {
- pr_err("chash: test_maxfill too big. Must be < table size.\n");
- return 0;
- }
- if (chash_test_minfill >= chash_test_maxfill) {
- pr_err("chash: test_minfill must be < test_maxfill.\n");
- return 0;
- }
- if (chash_test_keysize == 4 && chash_test_iters > 0xffffffff) {
- pr_err("chash: test_iters must be < 4G for 4 byte keys.\n");
- return 0;
- }
-
- ts1_ns = local_clock();
- ret = chash_self_test(chash_test_bits, chash_test_keysize,
- chash_test_minfill, chash_test_maxfill,
- chash_test_iters);
- if (!ret) {
- u64 ts_delta_us = local_clock() - ts1_ns;
- u64 iters_per_second = (u64)chash_test_iters * 1000000;
-
- do_div(ts_delta_us, 1000);
- do_div(iters_per_second, ts_delta_us);
- pr_info("chash: self test took %llu us, %llu iterations/s\n",
- ts_delta_us, iters_per_second);
- } else {
- pr_err("chash: self test failed: %d\n", ret);
- }
-
- return ret;
-}
-
-module_init(chash_init);
-
-#endif /* CONFIG_CHASH_SELFTEST */
-
-MODULE_DESCRIPTION("Closed hash table");
-MODULE_LICENSE("GPL and additional rights");
include $(AMD_POWERPLAY)
-POWER_MGR = amd_powerplay.o
+POWER_MGR = amd_powerplay.o amdgpu_smu.o smu_v11_0.o vega20_ppt.o
AMD_PP_POWER = $(addprefix $(AMD_PP_PATH)/,$(POWER_MGR))
mutex_init(&hwmgr->smu_lock);
hwmgr->chip_family = adev->family;
hwmgr->chip_id = adev->asic_type;
- hwmgr->feature_mask = adev->powerplay.pp_feature;
+ hwmgr->feature_mask = adev->pm.pp_feature;
hwmgr->display_config = &adev->pm.pm_display_cfg;
adev->powerplay.pp_handle = hwmgr;
adev->powerplay.pp_funcs = &pp_dpm_funcs;
if (hwmgr->hwmgr_func->smus_notify_pwe == NULL) {
pr_info_ratelimited("%s was not implemented.\n", __func__);
- return -EINVAL;;
+ return -EINVAL;
}
mutex_lock(&hwmgr->smu_lock);
if (hwmgr->hwmgr_func->set_min_deep_sleep_dcefclk == NULL) {
pr_debug("%s was not implemented.\n", __func__);
- return -EINVAL;;
+ return -EINVAL;
}
mutex_lock(&hwmgr->smu_lock);
if (hwmgr->hwmgr_func->set_hard_min_dcefclk_by_freq == NULL) {
pr_debug("%s was not implemented.\n", __func__);
- return -EINVAL;;
+ return -EINVAL;
}
mutex_lock(&hwmgr->smu_lock);
if (hwmgr->hwmgr_func->set_hard_min_fclk_by_freq == NULL) {
pr_debug("%s was not implemented.\n", __func__);
- return -EINVAL;;
+ return -EINVAL;
}
mutex_lock(&hwmgr->smu_lock);
--- /dev/null
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include "pp_debug.h"
+#include <linux/firmware.h>
+#include <drm/drmP.h>
+#include "amdgpu.h"
+#include "amdgpu_smu.h"
+#include "soc15_common.h"
+#include "smu_v11_0.h"
+#include "atom.h"
+#include "amd_pcie.h"
+
+int smu_dpm_set_power_gate(struct smu_context *smu, uint32_t block_type,
+ bool gate)
+{
+ int ret = 0;
+
+ switch (block_type) {
+ case AMD_IP_BLOCK_TYPE_UVD:
+ ret = smu_dpm_set_uvd_enable(smu, gate);
+ break;
+ case AMD_IP_BLOCK_TYPE_VCE:
+ ret = smu_dpm_set_vce_enable(smu, gate);
+ break;
+ default:
+ break;
+ }
+
+ return ret;
+}
+
+enum amd_pm_state_type smu_get_current_power_state(struct smu_context *smu)
+{
+ /* not support power state */
+ return POWER_STATE_TYPE_DEFAULT;
+}
+
+int smu_get_power_num_states(struct smu_context *smu,
+ struct pp_states_info *state_info)
+{
+ if (!state_info)
+ return -EINVAL;
+
+ /* not support power state */
+ memset(state_info, 0, sizeof(struct pp_states_info));
+ state_info->nums = 0;
+
+ return 0;
+}
+
+int smu_common_read_sensor(struct smu_context *smu, enum amd_pp_sensors sensor,
+ void *data, uint32_t *size)
+{
+ int ret = 0;
+
+ switch (sensor) {
+ case AMDGPU_PP_SENSOR_STABLE_PSTATE_SCLK:
+ *((uint32_t *)data) = smu->pstate_sclk;
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_STABLE_PSTATE_MCLK:
+ *((uint32_t *)data) = smu->pstate_mclk;
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_ENABLED_SMC_FEATURES_MASK:
+ ret = smu_feature_get_enabled_mask(smu, (uint32_t *)data, 2);
+ *size = 8;
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ if (ret)
+ *size = 0;
+
+ return ret;
+}
+
+int smu_update_table_with_arg(struct smu_context *smu, uint16_t table_id, uint16_t exarg,
+ void *table_data, bool drv2smu)
+{
+ struct smu_table_context *smu_table = &smu->smu_table;
+ struct smu_table *table = NULL;
+ int ret = 0;
+ uint32_t table_index;
+
+ if (!table_data || table_id >= smu_table->table_count)
+ return -EINVAL;
+
+ table_index = (exarg << 16) | table_id;
+
+ table = &smu_table->tables[table_id];
+
+ if (drv2smu)
+ memcpy(table->cpu_addr, table_data, table->size);
+
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_SetDriverDramAddrHigh,
+ upper_32_bits(table->mc_address));
+ if (ret)
+ return ret;
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_SetDriverDramAddrLow,
+ lower_32_bits(table->mc_address));
+ if (ret)
+ return ret;
+ ret = smu_send_smc_msg_with_param(smu, drv2smu ?
+ SMU_MSG_TransferTableDram2Smu :
+ SMU_MSG_TransferTableSmu2Dram,
+ table_index);
+ if (ret)
+ return ret;
+
+ if (!drv2smu)
+ memcpy(table_data, table->cpu_addr, table->size);
+
+ return ret;
+}
+
+bool is_support_sw_smu(struct amdgpu_device *adev)
+{
+ if (amdgpu_dpm != 1)
+ return false;
+
+ if (adev->asic_type >= CHIP_VEGA20 && adev->asic_type != CHIP_RAVEN)
+ return true;
+
+ return false;
+}
+
+int smu_sys_get_pp_table(struct smu_context *smu, void **table)
+{
+ struct smu_table_context *smu_table = &smu->smu_table;
+
+ if (!smu_table->power_play_table && !smu_table->hardcode_pptable)
+ return -EINVAL;
+
+ if (smu_table->hardcode_pptable)
+ *table = smu_table->hardcode_pptable;
+ else
+ *table = smu_table->power_play_table;
+
+ return smu_table->power_play_table_size;
+}
+
+int smu_sys_set_pp_table(struct smu_context *smu, void *buf, size_t size)
+{
+ struct smu_table_context *smu_table = &smu->smu_table;
+ ATOM_COMMON_TABLE_HEADER *header = (ATOM_COMMON_TABLE_HEADER *)buf;
+ int ret = 0;
+
+ if (header->usStructureSize != size) {
+ pr_err("pp table size not matched !\n");
+ return -EIO;
+ }
+
+ mutex_lock(&smu->mutex);
+ if (!smu_table->hardcode_pptable)
+ smu_table->hardcode_pptable = kzalloc(size, GFP_KERNEL);
+ if (!smu_table->hardcode_pptable) {
+ ret = -ENOMEM;
+ goto failed;
+ }
+
+ memcpy(smu_table->hardcode_pptable, buf, size);
+ smu_table->power_play_table = smu_table->hardcode_pptable;
+ smu_table->power_play_table_size = size;
+ mutex_unlock(&smu->mutex);
+
+ ret = smu_reset(smu);
+ if (ret)
+ pr_info("smu reset failed, ret = %d\n", ret);
+
+ return ret;
+
+failed:
+ mutex_unlock(&smu->mutex);
+ return ret;
+}
+
+int smu_feature_init_dpm(struct smu_context *smu)
+{
+ struct smu_feature *feature = &smu->smu_feature;
+ int ret = 0;
+ uint32_t unallowed_feature_mask[SMU_FEATURE_MAX/32];
+
+ mutex_lock(&feature->mutex);
+ bitmap_fill(feature->allowed, SMU_FEATURE_MAX);
+ mutex_unlock(&feature->mutex);
+
+ ret = smu_get_unallowed_feature_mask(smu, unallowed_feature_mask,
+ SMU_FEATURE_MAX/32);
+ if (ret)
+ return ret;
+
+ mutex_lock(&feature->mutex);
+ bitmap_andnot(feature->allowed, feature->allowed,
+ (unsigned long *)unallowed_feature_mask,
+ feature->feature_num);
+ mutex_unlock(&feature->mutex);
+
+ return ret;
+}
+
+int smu_feature_is_enabled(struct smu_context *smu, int feature_id)
+{
+ struct smu_feature *feature = &smu->smu_feature;
+ int ret = 0;
+
+ WARN_ON(feature_id > feature->feature_num);
+
+ mutex_lock(&feature->mutex);
+ ret = test_bit(feature_id, feature->enabled);
+ mutex_unlock(&feature->mutex);
+
+ return ret;
+}
+
+int smu_feature_set_enabled(struct smu_context *smu, int feature_id, bool enable)
+{
+ struct smu_feature *feature = &smu->smu_feature;
+ int ret = 0;
+
+ WARN_ON(feature_id > feature->feature_num);
+
+ mutex_lock(&feature->mutex);
+ ret = smu_feature_update_enable_state(smu, feature_id, enable);
+ if (ret)
+ goto failed;
+
+ if (enable)
+ test_and_set_bit(feature_id, feature->enabled);
+ else
+ test_and_clear_bit(feature_id, feature->enabled);
+
+failed:
+ mutex_unlock(&feature->mutex);
+
+ return ret;
+}
+
+int smu_feature_is_supported(struct smu_context *smu, int feature_id)
+{
+ struct smu_feature *feature = &smu->smu_feature;
+ int ret = 0;
+
+ WARN_ON(feature_id > feature->feature_num);
+
+ mutex_lock(&feature->mutex);
+ ret = test_bit(feature_id, feature->supported);
+ mutex_unlock(&feature->mutex);
+
+ return ret;
+}
+
+int smu_feature_set_supported(struct smu_context *smu, int feature_id,
+ bool enable)
+{
+ struct smu_feature *feature = &smu->smu_feature;
+ int ret = 0;
+
+ WARN_ON(feature_id > feature->feature_num);
+
+ mutex_unlock(&feature->mutex);
+ if (enable)
+ test_and_set_bit(feature_id, feature->supported);
+ else
+ test_and_clear_bit(feature_id, feature->supported);
+ mutex_unlock(&feature->mutex);
+
+ return ret;
+}
+
+static int smu_set_funcs(struct amdgpu_device *adev)
+{
+ struct smu_context *smu = &adev->smu;
+
+ switch (adev->asic_type) {
+ case CHIP_VEGA20:
+ adev->pm.pp_feature &= ~PP_GFXOFF_MASK;
+ if (adev->pm.pp_feature & PP_OVERDRIVE_MASK)
+ smu->od_enabled = true;
+ smu_v11_0_set_smu_funcs(smu);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int smu_early_init(void *handle)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct smu_context *smu = &adev->smu;
+
+ smu->adev = adev;
+ mutex_init(&smu->mutex);
+
+ return smu_set_funcs(adev);
+}
+
+static int smu_late_init(void *handle)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct smu_context *smu = &adev->smu;
+ mutex_lock(&smu->mutex);
+ smu_handle_task(&adev->smu,
+ smu->smu_dpm.dpm_level,
+ AMD_PP_TASK_COMPLETE_INIT);
+ mutex_unlock(&smu->mutex);
+
+ return 0;
+}
+
+int smu_get_atom_data_table(struct smu_context *smu, uint32_t table,
+ uint16_t *size, uint8_t *frev, uint8_t *crev,
+ uint8_t **addr)
+{
+ struct amdgpu_device *adev = smu->adev;
+ uint16_t data_start;
+
+ if (!amdgpu_atom_parse_data_header(adev->mode_info.atom_context, table,
+ size, frev, crev, &data_start))
+ return -EINVAL;
+
+ *addr = (uint8_t *)adev->mode_info.atom_context->bios + data_start;
+
+ return 0;
+}
+
+static int smu_initialize_pptable(struct smu_context *smu)
+{
+ /* TODO */
+ return 0;
+}
+
+static int smu_smc_table_sw_init(struct smu_context *smu)
+{
+ int ret;
+
+ ret = smu_initialize_pptable(smu);
+ if (ret) {
+ pr_err("Failed to init smu_initialize_pptable!\n");
+ return ret;
+ }
+
+ /**
+ * Create smu_table structure, and init smc tables such as
+ * TABLE_PPTABLE, TABLE_WATERMARKS, TABLE_SMU_METRICS, and etc.
+ */
+ ret = smu_init_smc_tables(smu);
+ if (ret) {
+ pr_err("Failed to init smc tables!\n");
+ return ret;
+ }
+
+ /**
+ * Create smu_power_context structure, and allocate smu_dpm_context and
+ * context size to fill the smu_power_context data.
+ */
+ ret = smu_init_power(smu);
+ if (ret) {
+ pr_err("Failed to init smu_init_power!\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int smu_smc_table_sw_fini(struct smu_context *smu)
+{
+ int ret;
+
+ ret = smu_fini_smc_tables(smu);
+ if (ret) {
+ pr_err("Failed to smu_fini_smc_tables!\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int smu_sw_init(void *handle)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct smu_context *smu = &adev->smu;
+ int ret;
+
+ if (!is_support_sw_smu(adev))
+ return -EINVAL;
+
+ smu->pool_size = adev->pm.smu_prv_buffer_size;
+ smu->smu_feature.feature_num = SMU_FEATURE_MAX;
+ mutex_init(&smu->smu_feature.mutex);
+ bitmap_zero(smu->smu_feature.supported, SMU_FEATURE_MAX);
+ bitmap_zero(smu->smu_feature.enabled, SMU_FEATURE_MAX);
+ bitmap_zero(smu->smu_feature.allowed, SMU_FEATURE_MAX);
+ smu->watermarks_bitmap = 0;
+ smu->power_profile_mode = PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+ smu->default_power_profile_mode = PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+
+ smu->workload_mask = 1 << smu->workload_prority[PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT];
+ smu->workload_prority[PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT] = 0;
+ smu->workload_prority[PP_SMC_POWER_PROFILE_FULLSCREEN3D] = 1;
+ smu->workload_prority[PP_SMC_POWER_PROFILE_POWERSAVING] = 2;
+ smu->workload_prority[PP_SMC_POWER_PROFILE_VIDEO] = 3;
+ smu->workload_prority[PP_SMC_POWER_PROFILE_VR] = 4;
+ smu->workload_prority[PP_SMC_POWER_PROFILE_COMPUTE] = 5;
+ smu->workload_prority[PP_SMC_POWER_PROFILE_CUSTOM] = 6;
+
+ smu->workload_setting[0] = PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT;
+ smu->workload_setting[1] = PP_SMC_POWER_PROFILE_FULLSCREEN3D;
+ smu->workload_setting[2] = PP_SMC_POWER_PROFILE_POWERSAVING;
+ smu->workload_setting[3] = PP_SMC_POWER_PROFILE_VIDEO;
+ smu->workload_setting[4] = PP_SMC_POWER_PROFILE_VR;
+ smu->workload_setting[5] = PP_SMC_POWER_PROFILE_COMPUTE;
+ smu->workload_setting[6] = PP_SMC_POWER_PROFILE_CUSTOM;
+ smu->display_config = &adev->pm.pm_display_cfg;
+
+ smu->smu_dpm.dpm_level = AMD_DPM_FORCED_LEVEL_AUTO;
+ smu->smu_dpm.requested_dpm_level = AMD_DPM_FORCED_LEVEL_AUTO;
+ ret = smu_init_microcode(smu);
+ if (ret) {
+ pr_err("Failed to load smu firmware!\n");
+ return ret;
+ }
+
+ ret = smu_smc_table_sw_init(smu);
+ if (ret) {
+ pr_err("Failed to sw init smc table!\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int smu_sw_fini(void *handle)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct smu_context *smu = &adev->smu;
+ int ret;
+
+ if (!is_support_sw_smu(adev))
+ return -EINVAL;
+
+ ret = smu_smc_table_sw_fini(smu);
+ if (ret) {
+ pr_err("Failed to sw fini smc table!\n");
+ return ret;
+ }
+
+ ret = smu_fini_power(smu);
+ if (ret) {
+ pr_err("Failed to init smu_fini_power!\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int smu_init_fb_allocations(struct smu_context *smu)
+{
+ struct amdgpu_device *adev = smu->adev;
+ struct smu_table_context *smu_table = &smu->smu_table;
+ struct smu_table *tables = smu_table->tables;
+ uint32_t table_count = smu_table->table_count;
+ uint32_t i = 0;
+ int32_t ret = 0;
+
+ if (table_count <= 0)
+ return -EINVAL;
+
+ for (i = 0 ; i < table_count; i++) {
+ if (tables[i].size == 0)
+ continue;
+ ret = amdgpu_bo_create_kernel(adev,
+ tables[i].size,
+ tables[i].align,
+ tables[i].domain,
+ &tables[i].bo,
+ &tables[i].mc_address,
+ &tables[i].cpu_addr);
+ if (ret)
+ goto failed;
+ }
+
+ return 0;
+failed:
+ for (; i > 0; i--) {
+ if (tables[i].size == 0)
+ continue;
+ amdgpu_bo_free_kernel(&tables[i].bo,
+ &tables[i].mc_address,
+ &tables[i].cpu_addr);
+
+ }
+ return ret;
+}
+
+static int smu_fini_fb_allocations(struct smu_context *smu)
+{
+ struct smu_table_context *smu_table = &smu->smu_table;
+ struct smu_table *tables = smu_table->tables;
+ uint32_t table_count = smu_table->table_count;
+ uint32_t i = 0;
+
+ if (table_count == 0 || tables == NULL)
+ return 0;
+
+ for (i = 0 ; i < table_count; i++) {
+ if (tables[i].size == 0)
+ continue;
+ amdgpu_bo_free_kernel(&tables[i].bo,
+ &tables[i].mc_address,
+ &tables[i].cpu_addr);
+ }
+
+ return 0;
+}
+
+static int smu_override_pcie_parameters(struct smu_context *smu)
+{
+ struct amdgpu_device *adev = smu->adev;
+ uint32_t pcie_gen = 0, pcie_width = 0, smu_pcie_arg;
+ int ret;
+
+ if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN4)
+ pcie_gen = 3;
+ else if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3)
+ pcie_gen = 2;
+ else if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN2)
+ pcie_gen = 1;
+ else if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN1)
+ pcie_gen = 0;
+
+ /* Bit 31:16: LCLK DPM level. 0 is DPM0, and 1 is DPM1
+ * Bit 15:8: PCIE GEN, 0 to 3 corresponds to GEN1 to GEN4
+ * Bit 7:0: PCIE lane width, 1 to 7 corresponds is x1 to x32
+ */
+ if (adev->pm.pcie_mlw_mask & CAIL_PCIE_LINK_WIDTH_SUPPORT_X16)
+ pcie_width = 6;
+ else if (adev->pm.pcie_mlw_mask & CAIL_PCIE_LINK_WIDTH_SUPPORT_X12)
+ pcie_width = 5;
+ else if (adev->pm.pcie_mlw_mask & CAIL_PCIE_LINK_WIDTH_SUPPORT_X8)
+ pcie_width = 4;
+ else if (adev->pm.pcie_mlw_mask & CAIL_PCIE_LINK_WIDTH_SUPPORT_X4)
+ pcie_width = 3;
+ else if (adev->pm.pcie_mlw_mask & CAIL_PCIE_LINK_WIDTH_SUPPORT_X2)
+ pcie_width = 2;
+ else if (adev->pm.pcie_mlw_mask & CAIL_PCIE_LINK_WIDTH_SUPPORT_X1)
+ pcie_width = 1;
+
+ smu_pcie_arg = (1 << 16) | (pcie_gen << 8) | pcie_width;
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_OverridePcieParameters,
+ smu_pcie_arg);
+ if (ret)
+ pr_err("[%s] Attempt to override pcie params failed!\n", __func__);
+ return ret;
+}
+
+static int smu_smc_table_hw_init(struct smu_context *smu,
+ bool initialize)
+{
+ struct amdgpu_device *adev = smu->adev;
+ int ret;
+
+ if (smu_is_dpm_running(smu) && adev->in_suspend) {
+ pr_info("dpm has been enabled\n");
+ return 0;
+ }
+
+ ret = smu_init_display(smu);
+ if (ret)
+ return ret;
+
+ if (initialize) {
+ ret = smu_read_pptable_from_vbios(smu);
+ if (ret)
+ return ret;
+
+ /* get boot_values from vbios to set revision, gfxclk, and etc. */
+ ret = smu_get_vbios_bootup_values(smu);
+ if (ret)
+ return ret;
+
+ ret = smu_get_clk_info_from_vbios(smu);
+ if (ret)
+ return ret;
+
+ /*
+ * check if the format_revision in vbios is up to pptable header
+ * version, and the structure size is not 0.
+ */
+ ret = smu_get_clk_info_from_vbios(smu);
+ if (ret)
+ return ret;
+
+ ret = smu_check_pptable(smu);
+ if (ret)
+ return ret;
+
+ /*
+ * allocate vram bos to store smc table contents.
+ */
+ ret = smu_init_fb_allocations(smu);
+ if (ret)
+ return ret;
+
+ /*
+ * Parse pptable format and fill PPTable_t smc_pptable to
+ * smu_table_context structure. And read the smc_dpm_table from vbios,
+ * then fill it into smc_pptable.
+ */
+ ret = smu_parse_pptable(smu);
+ if (ret)
+ return ret;
+
+ /*
+ * Send msg GetDriverIfVersion to check if the return value is equal
+ * with DRIVER_IF_VERSION of smc header.
+ */
+ ret = smu_check_fw_version(smu);
+ if (ret)
+ return ret;
+ }
+
+ /*
+ * Copy pptable bo in the vram to smc with SMU MSGs such as
+ * SetDriverDramAddr and TransferTableDram2Smu.
+ */
+ ret = smu_write_pptable(smu);
+ if (ret)
+ return ret;
+
+ /* issue RunAfllBtc msg */
+ ret = smu_run_afll_btc(smu);
+ if (ret)
+ return ret;
+
+ ret = smu_feature_set_allowed_mask(smu);
+ if (ret)
+ return ret;
+
+ ret = smu_system_features_control(smu, true);
+ if (ret)
+ return ret;
+
+ ret = smu_override_pcie_parameters(smu);
+ if (ret)
+ return ret;
+
+ ret = smu_notify_display_change(smu);
+ if (ret)
+ return ret;
+
+ /*
+ * Set min deep sleep dce fclk with bootup value from vbios via
+ * SetMinDeepSleepDcefclk MSG.
+ */
+ ret = smu_set_min_dcef_deep_sleep(smu);
+ if (ret)
+ return ret;
+
+ /*
+ * Set initialized values (get from vbios) to dpm tables context such as
+ * gfxclk, memclk, dcefclk, and etc. And enable the DPM feature for each
+ * type of clks.
+ */
+ if (initialize) {
+ ret = smu_populate_smc_pptable(smu);
+ if (ret)
+ return ret;
+
+ ret = smu_init_max_sustainable_clocks(smu);
+ if (ret)
+ return ret;
+ }
+
+ ret = smu_set_od8_default_settings(smu, initialize);
+ if (ret)
+ return ret;
+
+ if (initialize) {
+ ret = smu_populate_umd_state_clk(smu);
+ if (ret)
+ return ret;
+
+ ret = smu_get_power_limit(smu, &smu->default_power_limit, false);
+ if (ret)
+ return ret;
+ }
+
+ /*
+ * Set PMSTATUSLOG table bo address with SetToolsDramAddr MSG for tools.
+ */
+ ret = smu_set_tool_table_location(smu);
+
+ return ret;
+}
+
+/**
+ * smu_alloc_memory_pool - allocate memory pool in the system memory
+ *
+ * @smu: amdgpu_device pointer
+ *
+ * This memory pool will be used for SMC use and msg SetSystemVirtualDramAddr
+ * and DramLogSetDramAddr can notify it changed.
+ *
+ * Returns 0 on success, error on failure.
+ */
+static int smu_alloc_memory_pool(struct smu_context *smu)
+{
+ struct amdgpu_device *adev = smu->adev;
+ struct smu_table_context *smu_table = &smu->smu_table;
+ struct smu_table *memory_pool = &smu_table->memory_pool;
+ uint64_t pool_size = smu->pool_size;
+ int ret = 0;
+
+ if (pool_size == SMU_MEMORY_POOL_SIZE_ZERO)
+ return ret;
+
+ memory_pool->size = pool_size;
+ memory_pool->align = PAGE_SIZE;
+ memory_pool->domain = AMDGPU_GEM_DOMAIN_GTT;
+
+ switch (pool_size) {
+ case SMU_MEMORY_POOL_SIZE_256_MB:
+ case SMU_MEMORY_POOL_SIZE_512_MB:
+ case SMU_MEMORY_POOL_SIZE_1_GB:
+ case SMU_MEMORY_POOL_SIZE_2_GB:
+ ret = amdgpu_bo_create_kernel(adev,
+ memory_pool->size,
+ memory_pool->align,
+ memory_pool->domain,
+ &memory_pool->bo,
+ &memory_pool->mc_address,
+ &memory_pool->cpu_addr);
+ break;
+ default:
+ break;
+ }
+
+ return ret;
+}
+
+static int smu_free_memory_pool(struct smu_context *smu)
+{
+ struct smu_table_context *smu_table = &smu->smu_table;
+ struct smu_table *memory_pool = &smu_table->memory_pool;
+ int ret = 0;
+
+ if (memory_pool->size == SMU_MEMORY_POOL_SIZE_ZERO)
+ return ret;
+
+ amdgpu_bo_free_kernel(&memory_pool->bo,
+ &memory_pool->mc_address,
+ &memory_pool->cpu_addr);
+
+ memset(memory_pool, 0, sizeof(struct smu_table));
+
+ return ret;
+}
+
+static int smu_hw_init(void *handle)
+{
+ int ret;
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct smu_context *smu = &adev->smu;
+
+ if (!is_support_sw_smu(adev))
+ return -EINVAL;
+
+ if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP) {
+ ret = smu_load_microcode(smu);
+ if (ret)
+ return ret;
+ }
+
+ ret = smu_check_fw_status(smu);
+ if (ret) {
+ pr_err("SMC firmware status is not correct\n");
+ return ret;
+ }
+
+ mutex_lock(&smu->mutex);
+
+ ret = smu_feature_init_dpm(smu);
+ if (ret)
+ goto failed;
+
+ ret = smu_smc_table_hw_init(smu, true);
+ if (ret)
+ goto failed;
+
+ ret = smu_alloc_memory_pool(smu);
+ if (ret)
+ goto failed;
+
+ /*
+ * Use msg SetSystemVirtualDramAddr and DramLogSetDramAddr can notify
+ * pool location.
+ */
+ ret = smu_notify_memory_pool_location(smu);
+ if (ret)
+ goto failed;
+
+ ret = smu_start_thermal_control(smu);
+ if (ret)
+ goto failed;
+
+ mutex_unlock(&smu->mutex);
+
+ adev->pm.dpm_enabled = true;
+
+ pr_info("SMU is initialized successfully!\n");
+
+ return 0;
+
+failed:
+ mutex_unlock(&smu->mutex);
+ return ret;
+}
+
+static int smu_hw_fini(void *handle)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct smu_context *smu = &adev->smu;
+ struct smu_table_context *table_context = &smu->smu_table;
+ int ret = 0;
+
+ if (!is_support_sw_smu(adev))
+ return -EINVAL;
+
+ kfree(table_context->driver_pptable);
+ table_context->driver_pptable = NULL;
+
+ kfree(table_context->max_sustainable_clocks);
+ table_context->max_sustainable_clocks = NULL;
+
+ kfree(table_context->od_feature_capabilities);
+ table_context->od_feature_capabilities = NULL;
+
+ kfree(table_context->od_settings_max);
+ table_context->od_settings_max = NULL;
+
+ kfree(table_context->od_settings_min);
+ table_context->od_settings_min = NULL;
+
+ kfree(table_context->overdrive_table);
+ table_context->overdrive_table = NULL;
+
+ kfree(table_context->od8_settings);
+ table_context->od8_settings = NULL;
+
+ ret = smu_fini_fb_allocations(smu);
+ if (ret)
+ return ret;
+
+ ret = smu_free_memory_pool(smu);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+int smu_reset(struct smu_context *smu)
+{
+ struct amdgpu_device *adev = smu->adev;
+ int ret = 0;
+
+ ret = smu_hw_fini(adev);
+ if (ret)
+ return ret;
+
+ ret = smu_hw_init(adev);
+ if (ret)
+ return ret;
+
+ return ret;
+}
+
+static int smu_suspend(void *handle)
+{
+ int ret;
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct smu_context *smu = &adev->smu;
+
+ if (!is_support_sw_smu(adev))
+ return -EINVAL;
+
+ ret = smu_system_features_control(smu, false);
+ if (ret)
+ return ret;
+
+ smu->watermarks_bitmap &= ~(WATERMARKS_LOADED);
+
+ return 0;
+}
+
+static int smu_resume(void *handle)
+{
+ int ret;
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ struct smu_context *smu = &adev->smu;
+
+ if (!is_support_sw_smu(adev))
+ return -EINVAL;
+
+ pr_info("SMU is resuming...\n");
+
+ mutex_lock(&smu->mutex);
+
+ ret = smu_smc_table_hw_init(smu, false);
+ if (ret)
+ goto failed;
+
+ ret = smu_start_thermal_control(smu);
+ if (ret)
+ goto failed;
+
+ mutex_unlock(&smu->mutex);
+
+ pr_info("SMU is resumed successfully!\n");
+
+ return 0;
+failed:
+ mutex_unlock(&smu->mutex);
+ return ret;
+}
+
+int smu_display_configuration_change(struct smu_context *smu,
+ const struct amd_pp_display_configuration *display_config)
+{
+ int index = 0;
+ int num_of_active_display = 0;
+
+ if (!is_support_sw_smu(smu->adev))
+ return -EINVAL;
+
+ if (!display_config)
+ return -EINVAL;
+
+ mutex_lock(&smu->mutex);
+
+ smu_set_deep_sleep_dcefclk(smu,
+ display_config->min_dcef_deep_sleep_set_clk / 100);
+
+ for (index = 0; index < display_config->num_path_including_non_display; index++) {
+ if (display_config->displays[index].controller_id != 0)
+ num_of_active_display++;
+ }
+
+ smu_set_active_display_count(smu, num_of_active_display);
+
+ smu_store_cc6_data(smu, display_config->cpu_pstate_separation_time,
+ display_config->cpu_cc6_disable,
+ display_config->cpu_pstate_disable,
+ display_config->nb_pstate_switch_disable);
+
+ mutex_unlock(&smu->mutex);
+
+ return 0;
+}
+
+static int smu_get_clock_info(struct smu_context *smu,
+ struct smu_clock_info *clk_info,
+ enum smu_perf_level_designation designation)
+{
+ int ret;
+ struct smu_performance_level level = {0};
+
+ if (!clk_info)
+ return -EINVAL;
+
+ ret = smu_get_perf_level(smu, PERF_LEVEL_ACTIVITY, &level);
+ if (ret)
+ return -EINVAL;
+
+ clk_info->min_mem_clk = level.memory_clock;
+ clk_info->min_eng_clk = level.core_clock;
+ clk_info->min_bus_bandwidth = level.non_local_mem_freq * level.non_local_mem_width;
+
+ ret = smu_get_perf_level(smu, designation, &level);
+ if (ret)
+ return -EINVAL;
+
+ clk_info->min_mem_clk = level.memory_clock;
+ clk_info->min_eng_clk = level.core_clock;
+ clk_info->min_bus_bandwidth = level.non_local_mem_freq * level.non_local_mem_width;
+
+ return 0;
+}
+
+int smu_get_current_clocks(struct smu_context *smu,
+ struct amd_pp_clock_info *clocks)
+{
+ struct amd_pp_simple_clock_info simple_clocks = {0};
+ struct smu_clock_info hw_clocks;
+ int ret = 0;
+
+ if (!is_support_sw_smu(smu->adev))
+ return -EINVAL;
+
+ mutex_lock(&smu->mutex);
+
+ smu_get_dal_power_level(smu, &simple_clocks);
+
+ if (smu->support_power_containment)
+ ret = smu_get_clock_info(smu, &hw_clocks,
+ PERF_LEVEL_POWER_CONTAINMENT);
+ else
+ ret = smu_get_clock_info(smu, &hw_clocks, PERF_LEVEL_ACTIVITY);
+
+ if (ret) {
+ pr_err("Error in smu_get_clock_info\n");
+ goto failed;
+ }
+
+ clocks->min_engine_clock = hw_clocks.min_eng_clk;
+ clocks->max_engine_clock = hw_clocks.max_eng_clk;
+ clocks->min_memory_clock = hw_clocks.min_mem_clk;
+ clocks->max_memory_clock = hw_clocks.max_mem_clk;
+ clocks->min_bus_bandwidth = hw_clocks.min_bus_bandwidth;
+ clocks->max_bus_bandwidth = hw_clocks.max_bus_bandwidth;
+ clocks->max_engine_clock_in_sr = hw_clocks.max_eng_clk;
+ clocks->min_engine_clock_in_sr = hw_clocks.min_eng_clk;
+
+ if (simple_clocks.level == 0)
+ clocks->max_clocks_state = PP_DAL_POWERLEVEL_7;
+ else
+ clocks->max_clocks_state = simple_clocks.level;
+
+ if (!smu_get_current_shallow_sleep_clocks(smu, &hw_clocks)) {
+ clocks->max_engine_clock_in_sr = hw_clocks.max_eng_clk;
+ clocks->min_engine_clock_in_sr = hw_clocks.min_eng_clk;
+ }
+
+failed:
+ mutex_unlock(&smu->mutex);
+ return ret;
+}
+
+static int smu_set_clockgating_state(void *handle,
+ enum amd_clockgating_state state)
+{
+ return 0;
+}
+
+static int smu_set_powergating_state(void *handle,
+ enum amd_powergating_state state)
+{
+ return 0;
+}
+
+static int smu_enable_umd_pstate(void *handle,
+ enum amd_dpm_forced_level *level)
+{
+ uint32_t profile_mode_mask = AMD_DPM_FORCED_LEVEL_PROFILE_STANDARD |
+ AMD_DPM_FORCED_LEVEL_PROFILE_MIN_SCLK |
+ AMD_DPM_FORCED_LEVEL_PROFILE_MIN_MCLK |
+ AMD_DPM_FORCED_LEVEL_PROFILE_PEAK;
+
+ struct smu_context *smu = (struct smu_context*)(handle);
+ struct smu_dpm_context *smu_dpm_ctx = &(smu->smu_dpm);
+ if (!smu_dpm_ctx->dpm_context)
+ return -EINVAL;
+
+ if (!(smu_dpm_ctx->dpm_level & profile_mode_mask)) {
+ /* enter umd pstate, save current level, disable gfx cg*/
+ if (*level & profile_mode_mask) {
+ smu_dpm_ctx->saved_dpm_level = smu_dpm_ctx->dpm_level;
+ smu_dpm_ctx->enable_umd_pstate = true;
+ amdgpu_device_ip_set_clockgating_state(smu->adev,
+ AMD_IP_BLOCK_TYPE_GFX,
+ AMD_CG_STATE_UNGATE);
+ amdgpu_device_ip_set_powergating_state(smu->adev,
+ AMD_IP_BLOCK_TYPE_GFX,
+ AMD_PG_STATE_UNGATE);
+ }
+ } else {
+ /* exit umd pstate, restore level, enable gfx cg*/
+ if (!(*level & profile_mode_mask)) {
+ if (*level == AMD_DPM_FORCED_LEVEL_PROFILE_EXIT)
+ *level = smu_dpm_ctx->saved_dpm_level;
+ smu_dpm_ctx->enable_umd_pstate = false;
+ amdgpu_device_ip_set_clockgating_state(smu->adev,
+ AMD_IP_BLOCK_TYPE_GFX,
+ AMD_CG_STATE_GATE);
+ amdgpu_device_ip_set_powergating_state(smu->adev,
+ AMD_IP_BLOCK_TYPE_GFX,
+ AMD_PG_STATE_GATE);
+ }
+ }
+
+ return 0;
+}
+
+int smu_adjust_power_state_dynamic(struct smu_context *smu,
+ enum amd_dpm_forced_level level,
+ bool skip_display_settings)
+{
+ int ret = 0;
+ int index = 0;
+ uint32_t sclk_mask, mclk_mask, soc_mask;
+ long workload;
+ struct smu_dpm_context *smu_dpm_ctx = &(smu->smu_dpm);
+
+ if (!skip_display_settings) {
+ ret = smu_display_config_changed(smu);
+ if (ret) {
+ pr_err("Failed to change display config!");
+ return ret;
+ }
+ }
+
+ ret = smu_apply_clocks_adjust_rules(smu);
+ if (ret) {
+ pr_err("Failed to apply clocks adjust rules!");
+ return ret;
+ }
+
+ if (!skip_display_settings) {
+ ret = smu_notify_smc_dispaly_config(smu);
+ if (ret) {
+ pr_err("Failed to notify smc display config!");
+ return ret;
+ }
+ }
+
+ if (smu_dpm_ctx->dpm_level != level) {
+ switch (level) {
+ case AMD_DPM_FORCED_LEVEL_HIGH:
+ ret = smu_force_dpm_limit_value(smu, true);
+ break;
+ case AMD_DPM_FORCED_LEVEL_LOW:
+ ret = smu_force_dpm_limit_value(smu, false);
+ break;
+
+ case AMD_DPM_FORCED_LEVEL_AUTO:
+ ret = smu_unforce_dpm_levels(smu);
+ break;
+
+ case AMD_DPM_FORCED_LEVEL_PROFILE_STANDARD:
+ case AMD_DPM_FORCED_LEVEL_PROFILE_MIN_SCLK:
+ case AMD_DPM_FORCED_LEVEL_PROFILE_MIN_MCLK:
+ case AMD_DPM_FORCED_LEVEL_PROFILE_PEAK:
+ ret = smu_get_profiling_clk_mask(smu, level,
+ &sclk_mask,
+ &mclk_mask,
+ &soc_mask);
+ if (ret)
+ return ret;
+ smu_force_clk_levels(smu, PP_SCLK, 1 << sclk_mask);
+ smu_force_clk_levels(smu, PP_MCLK, 1 << mclk_mask);
+ break;
+
+ case AMD_DPM_FORCED_LEVEL_MANUAL:
+ case AMD_DPM_FORCED_LEVEL_PROFILE_EXIT:
+ default:
+ break;
+ }
+
+ if (!ret)
+ smu_dpm_ctx->dpm_level = level;
+ }
+
+ if (smu_dpm_ctx->dpm_level != AMD_DPM_FORCED_LEVEL_MANUAL) {
+ index = fls(smu->workload_mask);
+ index = index > 0 && index <= WORKLOAD_POLICY_MAX ? index - 1 : 0;
+ workload = smu->workload_setting[index];
+
+ if (smu->power_profile_mode != workload)
+ smu_set_power_profile_mode(smu, &workload, 0);
+ }
+
+ return ret;
+}
+
+int smu_handle_task(struct smu_context *smu,
+ enum amd_dpm_forced_level level,
+ enum amd_pp_task task_id)
+{
+ int ret = 0;
+
+ switch (task_id) {
+ case AMD_PP_TASK_DISPLAY_CONFIG_CHANGE:
+ ret = smu_pre_display_config_changed(smu);
+ if (ret)
+ return ret;
+ ret = smu_set_cpu_power_state(smu);
+ if (ret)
+ return ret;
+ ret = smu_adjust_power_state_dynamic(smu, level, false);
+ break;
+ case AMD_PP_TASK_COMPLETE_INIT:
+ case AMD_PP_TASK_READJUST_POWER_STATE:
+ ret = smu_adjust_power_state_dynamic(smu, level, true);
+ break;
+ default:
+ break;
+ }
+
+ return ret;
+}
+
+const struct amd_ip_funcs smu_ip_funcs = {
+ .name = "smu",
+ .early_init = smu_early_init,
+ .late_init = smu_late_init,
+ .sw_init = smu_sw_init,
+ .sw_fini = smu_sw_fini,
+ .hw_init = smu_hw_init,
+ .hw_fini = smu_hw_fini,
+ .suspend = smu_suspend,
+ .resume = smu_resume,
+ .is_idle = NULL,
+ .check_soft_reset = NULL,
+ .wait_for_idle = NULL,
+ .soft_reset = NULL,
+ .set_clockgating_state = smu_set_clockgating_state,
+ .set_powergating_state = smu_set_powergating_state,
+ .enable_umd_pstate = smu_enable_umd_pstate,
+};
+
+const struct amdgpu_ip_block_version smu_v11_0_ip_block =
+{
+ .type = AMD_IP_BLOCK_TYPE_SMC,
+ .major = 11,
+ .minor = 0,
+ .rev = 0,
+ .funcs = &smu_ip_funcs,
+};
vega12_thermal.o \
pp_overdriver.o smu_helper.o \
vega20_processpptables.o vega20_hwmgr.o vega20_powertune.o \
- vega20_thermal.o common_baco.o vega10_baco.o vega20_baco.o
+ vega20_thermal.o common_baco.o vega10_baco.o vega20_baco.o \
+ vega12_baco.o smu9_baco.o
AMD_PP_HWMGR = $(addprefix $(AMD_PP_PATH)/hwmgr/,$(HARDWARE_MGR))
int phm_enable_dynamic_state_management(struct pp_hwmgr *hwmgr)
{
struct amdgpu_device *adev = NULL;
- int ret = -EINVAL;;
+ int ret = -EINVAL;
PHM_FUNC_CHECK(hwmgr);
adev = hwmgr->adev;
#include "smu10_hwmgr.h"
#include "power_state.h"
#include "soc15_common.h"
+#include "smu10.h"
#define SMU10_MAX_DEEPSLEEP_DIVIDER_ID 5
#define SMU10_MINIMUM_ENGINE_CLOCK 800 /* 8Mhz, the low boundary of engine clock allowed on this chip */
smu10_data->num_active_display = 0;
smu10_data->deep_sleep_dcefclk = 0;
- if (hwmgr->feature_mask & PP_GFXOFF_MASK)
- smu10_data->gfx_off_controled_by_driver = true;
- else
- smu10_data->gfx_off_controled_by_driver = false;
-
phm_cap_unset(hwmgr->platform_descriptor.platformCaps,
PHM_PlatformCaps_SclkDeepSleep);
return 0;
}
-static inline uint32_t convert_10k_to_mhz(uint32_t clock)
-{
- return (clock + 99) / 100;
-}
-
static int smu10_set_min_deep_sleep_dcefclk(struct pp_hwmgr *hwmgr, uint32_t clock)
{
struct smu10_hwmgr *smu10_data = (struct smu10_hwmgr *)(hwmgr->backend);
if (smu10_data->need_min_deep_sleep_dcefclk &&
- smu10_data->deep_sleep_dcefclk != convert_10k_to_mhz(clock)) {
- smu10_data->deep_sleep_dcefclk = convert_10k_to_mhz(clock);
+ smu10_data->deep_sleep_dcefclk != clock) {
+ smu10_data->deep_sleep_dcefclk = clock;
smum_send_msg_to_smc_with_parameter(hwmgr,
PPSMC_MSG_SetMinDeepSleepDcefclk,
smu10_data->deep_sleep_dcefclk);
struct smu10_hwmgr *smu10_data = (struct smu10_hwmgr *)(hwmgr->backend);
if (smu10_data->dcf_actual_hard_min_freq &&
- smu10_data->dcf_actual_hard_min_freq != convert_10k_to_mhz(clock)) {
- smu10_data->dcf_actual_hard_min_freq = convert_10k_to_mhz(clock);
+ smu10_data->dcf_actual_hard_min_freq != clock) {
+ smu10_data->dcf_actual_hard_min_freq = clock;
smum_send_msg_to_smc_with_parameter(hwmgr,
PPSMC_MSG_SetHardMinDcefclkByFreq,
smu10_data->dcf_actual_hard_min_freq);
struct smu10_hwmgr *smu10_data = (struct smu10_hwmgr *)(hwmgr->backend);
if (smu10_data->f_actual_hard_min_freq &&
- smu10_data->f_actual_hard_min_freq != convert_10k_to_mhz(clock)) {
- smu10_data->f_actual_hard_min_freq = convert_10k_to_mhz(clock);
+ smu10_data->f_actual_hard_min_freq != clock) {
+ smu10_data->f_actual_hard_min_freq = clock;
smum_send_msg_to_smc_with_parameter(hwmgr,
PPSMC_MSG_SetHardMinFclkByFreq,
smu10_data->f_actual_hard_min_freq);
static int smu10_disable_gfx_off(struct pp_hwmgr *hwmgr)
{
- struct smu10_hwmgr *smu10_data = (struct smu10_hwmgr *)(hwmgr->backend);
+ struct amdgpu_device *adev = hwmgr->adev;
- if (smu10_data->gfx_off_controled_by_driver) {
+ if (adev->pm.pp_feature & PP_GFXOFF_MASK) {
smum_send_msg_to_smc(hwmgr, PPSMC_MSG_DisableGfxOff);
/* confirm gfx is back to "on" state */
static int smu10_enable_gfx_off(struct pp_hwmgr *hwmgr)
{
- struct smu10_hwmgr *smu10_data = (struct smu10_hwmgr *)(hwmgr->backend);
+ struct amdgpu_device *adev = hwmgr->adev;
- if (smu10_data->gfx_off_controled_by_driver)
+ if (adev->pm.pp_feature & PP_GFXOFF_MASK)
smum_send_msg_to_smc(hwmgr, PPSMC_MSG_EnableGfxOff);
return 0;
enum amd_dpm_forced_level level)
{
struct smu10_hwmgr *data = hwmgr->backend;
- struct amdgpu_device *adev = hwmgr->adev;
uint32_t min_sclk = hwmgr->display_config->min_core_set_clock;
uint32_t min_mclk = hwmgr->display_config->min_mem_set_clock/100;
return 0;
}
- /* Disable UMDPSTATE support on rv2 temporarily */
- if ((adev->asic_type == CHIP_RAVEN) &&
- (adev->rev_id >= 8))
- return 0;
-
if (min_sclk < data->gfx_min_freq_limit)
min_sclk = data->gfx_min_freq_limit;
}
}
+static int conv_power_profile_to_pplib_workload(int power_profile)
+{
+ int pplib_workload = 0;
+
+ switch (power_profile) {
+ case PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT:
+ pplib_workload = WORKLOAD_DEFAULT_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_FULLSCREEN3D:
+ pplib_workload = WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_POWERSAVING:
+ pplib_workload = WORKLOAD_PPLIB_POWER_SAVING_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_VIDEO:
+ pplib_workload = WORKLOAD_PPLIB_VIDEO_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_VR:
+ pplib_workload = WORKLOAD_PPLIB_VR_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_COMPUTE:
+ pplib_workload = WORKLOAD_PPLIB_COMPUTE_BIT;
+ break;
+ }
+
+ return pplib_workload;
+}
+
+static int smu10_get_power_profile_mode(struct pp_hwmgr *hwmgr, char *buf)
+{
+ uint32_t i, size = 0;
+ static const uint8_t
+ profile_mode_setting[6][4] = {{70, 60, 0, 0,},
+ {70, 60, 1, 3,},
+ {90, 60, 0, 0,},
+ {70, 60, 0, 0,},
+ {70, 90, 0, 0,},
+ {30, 60, 0, 6,},
+ };
+ static const char *profile_name[6] = {
+ "BOOTUP_DEFAULT",
+ "3D_FULL_SCREEN",
+ "POWER_SAVING",
+ "VIDEO",
+ "VR",
+ "COMPUTE"};
+ static const char *title[6] = {"NUM",
+ "MODE_NAME",
+ "BUSY_SET_POINT",
+ "FPS",
+ "USE_RLC_BUSY",
+ "MIN_ACTIVE_LEVEL"};
+
+ if (!buf)
+ return -EINVAL;
+
+ size += sprintf(buf + size, "%s %16s %s %s %s %s\n",title[0],
+ title[1], title[2], title[3], title[4], title[5]);
+
+ for (i = 0; i <= PP_SMC_POWER_PROFILE_COMPUTE; i++)
+ size += sprintf(buf + size, "%3d %14s%s: %14d %3d %10d %14d\n",
+ i, profile_name[i], (i == hwmgr->power_profile_mode) ? "*" : " ",
+ profile_mode_setting[i][0], profile_mode_setting[i][1],
+ profile_mode_setting[i][2], profile_mode_setting[i][3]);
+
+ return size;
+}
+
+static int smu10_set_power_profile_mode(struct pp_hwmgr *hwmgr, long *input, uint32_t size)
+{
+ int workload_type = 0;
+
+ if (input[size] > PP_SMC_POWER_PROFILE_COMPUTE) {
+ pr_err("Invalid power profile mode %ld\n", input[size]);
+ return -EINVAL;
+ }
+ hwmgr->power_profile_mode = input[size];
+
+ /* conv PP_SMC_POWER_PROFILE* to WORKLOAD_PPLIB_*_BIT */
+ workload_type =
+ conv_power_profile_to_pplib_workload(hwmgr->power_profile_mode);
+ smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_ActiveProcessNotify,
+ 1 << workload_type);
+
+ return 0;
+}
+
+
static const struct pp_hwmgr_func smu10_hwmgr_funcs = {
.backend_init = smu10_hwmgr_backend_init,
.backend_fini = smu10_hwmgr_backend_fini,
.powergate_sdma = smu10_powergate_sdma,
.set_hard_min_dcefclk_by_freq = smu10_set_hard_min_dcefclk_by_freq,
.set_hard_min_fclk_by_freq = smu10_set_hard_min_fclk_by_freq,
+ .get_power_profile_mode = smu10_get_power_profile_mode,
+ .set_power_profile_mode = smu10_set_power_profile_mode,
};
int smu10_init_function_pointers(struct pp_hwmgr *hwmgr)
#define PCIE_BUS_CLK 10000
#define TCLK (PCIE_BUS_CLK / 10)
-static const struct profile_mode_setting smu7_profiling[7] =
+static struct profile_mode_setting smu7_profiling[7] =
{{0, 0, 0, 0, 0, 0, 0, 0},
{1, 0, 100, 30, 1, 0, 100, 10},
{1, 10, 0, 30, 0, 0, 0, 0},
mode = input[size];
switch (mode) {
case PP_SMC_POWER_PROFILE_CUSTOM:
- if (size < 8)
+ if (size < 8 && size != 0)
return -EINVAL;
-
- tmp.bupdate_sclk = input[0];
- tmp.sclk_up_hyst = input[1];
- tmp.sclk_down_hyst = input[2];
- tmp.sclk_activity = input[3];
- tmp.bupdate_mclk = input[4];
- tmp.mclk_up_hyst = input[5];
- tmp.mclk_down_hyst = input[6];
- tmp.mclk_activity = input[7];
+ /* If only CUSTOM is passed in, use the saved values. Check
+ * that we actually have a CUSTOM profile by ensuring that
+ * the "use sclk" or the "use mclk" bits are set
+ */
+ tmp = smu7_profiling[PP_SMC_POWER_PROFILE_CUSTOM];
+ if (size == 0) {
+ if (tmp.bupdate_sclk == 0 && tmp.bupdate_mclk == 0)
+ return -EINVAL;
+ } else {
+ tmp.bupdate_sclk = input[0];
+ tmp.sclk_up_hyst = input[1];
+ tmp.sclk_down_hyst = input[2];
+ tmp.sclk_activity = input[3];
+ tmp.bupdate_mclk = input[4];
+ tmp.mclk_up_hyst = input[5];
+ tmp.mclk_down_hyst = input[6];
+ tmp.mclk_activity = input[7];
+ smu7_profiling[PP_SMC_POWER_PROFILE_CUSTOM] = tmp;
+ }
if (!smum_update_dpm_settings(hwmgr, &tmp)) {
memcpy(&data->current_profile_setting, &tmp, sizeof(struct profile_mode_setting));
hwmgr->power_profile_mode = mode;
--- /dev/null
+/*
+ * Copyright 2018 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#include "amdgpu.h"
+#include "soc15.h"
+#include "soc15_hw_ip.h"
+#include "vega10_ip_offset.h"
+#include "soc15_common.h"
+#include "vega10_inc.h"
+#include "smu9_baco.h"
+
+int smu9_baco_get_capability(struct pp_hwmgr *hwmgr, bool *cap)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)(hwmgr->adev);
+ uint32_t reg, data;
+
+ *cap = false;
+ if (!phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_BACO))
+ return 0;
+
+ WREG32(0x12074, 0xFFF0003B);
+ data = RREG32(0x12075);
+
+ if (data == 0x1) {
+ reg = RREG32_SOC15(NBIF, 0, mmRCC_BIF_STRAP0);
+
+ if (reg & RCC_BIF_STRAP0__STRAP_PX_CAPABLE_MASK)
+ *cap = true;
+ }
+
+ return 0;
+}
+
+int smu9_baco_get_state(struct pp_hwmgr *hwmgr, enum BACO_STATE *state)
+{
+ struct amdgpu_device *adev = (struct amdgpu_device *)(hwmgr->adev);
+ uint32_t reg;
+
+ reg = RREG32_SOC15(NBIF, 0, mmBACO_CNTL);
+
+ if (reg & BACO_CNTL__BACO_MODE_MASK)
+ /* gfx has already entered BACO state */
+ *state = BACO_STATE_IN;
+ else
+ *state = BACO_STATE_OUT;
+ return 0;
+}
--- /dev/null
+/*
+ * Copyright 2018 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#ifndef __SMU9_BACO_H__
+#define __SMU9_BACO_H__
+#include "hwmgr.h"
+#include "common_baco.h"
+
+extern int smu9_baco_get_capability(struct pp_hwmgr *hwmgr, bool *cap);
+extern int smu9_baco_get_state(struct pp_hwmgr *hwmgr, enum BACO_STATE *state);
+
+#endif
{CMD_WRITE, SOC15_REG_ENTRY(NBIF, 0, mmBIOS_SCRATCH_7), 0, 0, 0, 0},
};
-int vega10_baco_get_capability(struct pp_hwmgr *hwmgr, bool *cap)
-{
- struct amdgpu_device *adev = (struct amdgpu_device *)(hwmgr->adev);
- uint32_t reg, data;
-
- *cap = false;
- if (!phm_cap_enabled(hwmgr->platform_descriptor.platformCaps, PHM_PlatformCaps_BACO))
- return 0;
-
- WREG32(0x12074, 0xFFF0003B);
- data = RREG32(0x12075);
-
- if (data == 0x1) {
- reg = RREG32_SOC15(NBIF, 0, mmRCC_BIF_STRAP0);
-
- if (reg & RCC_BIF_STRAP0__STRAP_PX_CAPABLE_MASK)
- *cap = true;
- }
-
- return 0;
-}
-
-int vega10_baco_get_state(struct pp_hwmgr *hwmgr, enum BACO_STATE *state)
-{
- struct amdgpu_device *adev = (struct amdgpu_device *)(hwmgr->adev);
- uint32_t reg;
-
- reg = RREG32_SOC15(NBIF, 0, mmBACO_CNTL);
-
- if (reg & BACO_CNTL__BACO_MODE_MASK)
- /* gfx has already entered BACO state */
- *state = BACO_STATE_IN;
- else
- *state = BACO_STATE_OUT;
- return 0;
-}
-
int vega10_baco_set_state(struct pp_hwmgr *hwmgr, enum BACO_STATE state)
{
enum BACO_STATE cur_state;
- vega10_baco_get_state(hwmgr, &cur_state);
+ smu9_baco_get_state(hwmgr, &cur_state);
if (cur_state == state)
/* aisc already in the target state */
*/
#ifndef __VEGA10_BACO_H__
#define __VEGA10_BACO_H__
-#include "hwmgr.h"
-#include "common_baco.h"
+#include "smu9_baco.h"
-extern int vega10_baco_get_capability(struct pp_hwmgr *hwmgr, bool *cap);
-extern int vega10_baco_get_state(struct pp_hwmgr *hwmgr, enum BACO_STATE *state);
extern int vega10_baco_set_state(struct pp_hwmgr *hwmgr, enum BACO_STATE state);
#endif
vega10_setup_default_pcie_table(hwmgr);
+ /* Zero out the saved copy of the CUSTOM profile
+ * This will be checked when trying to set the profile
+ * and will require that new values be passed in
+ */
+ data->custom_profile_mode[0] = 0;
+ data->custom_profile_mode[1] = 0;
+ data->custom_profile_mode[2] = 0;
+ data->custom_profile_mode[3] = 0;
+
/* save a copy of the default DPM table */
memcpy(&(data->golden_dpm_table), &(data->dpm_table),
sizeof(struct vega10_dpm_table));
uint8_t FPS;
uint8_t use_rlc_busy;
uint8_t min_active_level;
+ uint32_t power_profile_mode = input[size];
- hwmgr->power_profile_mode = input[size];
-
- smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetWorkloadMask,
- 1<<hwmgr->power_profile_mode);
-
- if (hwmgr->power_profile_mode == PP_SMC_POWER_PROFILE_CUSTOM) {
- if (size == 0 || size > 4)
+ if (power_profile_mode == PP_SMC_POWER_PROFILE_CUSTOM) {
+ if (size != 0 && size != 4)
return -EINVAL;
+ /* If size = 0 and the CUSTOM profile has been set already
+ * then just apply the profile. The copy stored in the hwmgr
+ * is zeroed out on init
+ */
+ if (size == 0) {
+ if (data->custom_profile_mode[0] != 0)
+ goto out;
+ else
+ return -EINVAL;
+ }
+
data->custom_profile_mode[0] = busy_set_point = input[0];
data->custom_profile_mode[1] = FPS = input[1];
data->custom_profile_mode[2] = use_rlc_busy = input[2];
use_rlc_busy << 16 | min_active_level<<24);
}
+out:
+ smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetWorkloadMask,
+ 1 << power_profile_mode);
+ hwmgr->power_profile_mode = power_profile_mode;
+
return 0;
}
.set_power_limit = vega10_set_power_limit,
.odn_edit_dpm_table = vega10_odn_edit_dpm_table,
.get_performance_level = vega10_get_performance_level,
- .get_asic_baco_capability = vega10_baco_get_capability,
- .get_asic_baco_state = vega10_baco_get_state,
+ .get_asic_baco_capability = smu9_baco_get_capability,
+ .get_asic_baco_state = smu9_baco_get_state,
.set_asic_baco_state = vega10_baco_set_state,
.enable_mgpu_fan_boost = vega10_enable_mgpu_fan_boost,
.get_ppfeature_status = vega10_get_ppfeature_status,
--- /dev/null
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#include "amdgpu.h"
+#include "soc15.h"
+#include "soc15_hw_ip.h"
+#include "vega10_ip_offset.h"
+#include "soc15_common.h"
+#include "vega12_inc.h"
+#include "vega12_ppsmc.h"
+#include "vega12_baco.h"
+
+static const struct soc15_baco_cmd_entry pre_baco_tbl[] =
+{
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmBIF_DOORBELL_CNTL_BASE_IDX, mmBIF_DOORBELL_CNTL, BIF_DOORBELL_CNTL__DOORBELL_MONITOR_EN_MASK, BIF_DOORBELL_CNTL__DOORBELL_MONITOR_EN__SHIFT, 0, 0 },
+ { CMD_WRITE, NBIF_HWID, 0, mmBIF_FB_EN_BASE_IDX, mmBIF_FB_EN, 0, 0, 0, 0 },
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_DSTATE_BYPASS_MASK, BACO_CNTL__BACO_DSTATE_BYPASS__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_RST_INTR_MASK_MASK, BACO_CNTL__BACO_RST_INTR_MASK__SHIFT, 0, 1 }
+};
+
+static const struct soc15_baco_cmd_entry enter_baco_tbl[] =
+{
+ { CMD_WAITFOR, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__SOC_DOMAIN_IDLE_MASK, THM_BACO_CNTL__SOC_DOMAIN_IDLE__SHIFT, 0xffffffff, 0x80000000 },
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_EN_MASK, BACO_CNTL__BACO_EN__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_BIF_LCLK_SWITCH_MASK, BACO_CNTL__BACO_BIF_LCLK_SWITCH__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_DUMMY_EN_MASK, BACO_CNTL__BACO_DUMMY_EN__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_SOC_VDCI_RESET_MASK, THM_BACO_CNTL__BACO_SOC_VDCI_RESET__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_SMNCLK_MUX_MASK, THM_BACO_CNTL__BACO_SMNCLK_MUX__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_ISO_EN_MASK, THM_BACO_CNTL__BACO_ISO_EN__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_AEB_ISO_EN_MASK, THM_BACO_CNTL__BACO_AEB_ISO_EN__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_ANA_ISO_EN_MASK, THM_BACO_CNTL__BACO_ANA_ISO_EN__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_SOC_REFCLK_OFF_MASK, THM_BACO_CNTL__BACO_SOC_REFCLK_OFF__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_POWER_OFF_MASK, BACO_CNTL__BACO_POWER_OFF__SHIFT, 0, 1 },
+ { CMD_DELAY_MS, 0, 0, 0, 5, 0 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_RESET_EN_MASK, THM_BACO_CNTL__BACO_RESET_EN__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_PWROKRAW_CNTL_MASK, THM_BACO_CNTL__BACO_PWROKRAW_CNTL__SHIFT, 0, 0 },
+ { CMD_WAITFOR, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_MODE_MASK, BACO_CNTL__BACO_MODE__SHIFT, 0xffffffff, 0x100 }
+};
+
+static const struct soc15_baco_cmd_entry exit_baco_tbl[] =
+{
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_POWER_OFF_MASK, BACO_CNTL__BACO_POWER_OFF__SHIFT, 0, 0 },
+ { CMD_DELAY_MS, 0, 0, 0, 0, 0, 0, 10, 0 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_SOC_REFCLK_OFF_MASK, THM_BACO_CNTL__BACO_SOC_REFCLK_OFF__SHIFT, 0, 0 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_ANA_ISO_EN_MASK, THM_BACO_CNTL__BACO_ANA_ISO_EN__SHIFT, 0, 0 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_AEB_ISO_EN_MASK, THM_BACO_CNTL__BACO_AEB_ISO_EN__SHIFT, 0, 0 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_ISO_EN_MASK, THM_BACO_CNTL__BACO_ISO_EN__SHIFT, 0, 0 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_PWROKRAW_CNTL_MASK, THM_BACO_CNTL__BACO_PWROKRAW_CNTL__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_SMNCLK_MUX_MASK, THM_BACO_CNTL__BACO_SMNCLK_MUX__SHIFT, 0, 0 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_SOC_VDCI_RESET_MASK, THM_BACO_CNTL__BACO_SOC_VDCI_RESET__SHIFT, 0, 0 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_EXIT_MASK, THM_BACO_CNTL__BACO_EXIT__SHIFT, 0, 1 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_RESET_EN_MASK, THM_BACO_CNTL__BACO_RESET_EN__SHIFT, 0, 0 },
+ { CMD_WAITFOR, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_EXIT_MASK, 0, 0xffffffff, 0 },
+ { CMD_READMODIFYWRITE, THM_HWID, 0, mmTHM_BACO_CNTL_BASE_IDX, mmTHM_BACO_CNTL, THM_BACO_CNTL__BACO_SB_AXI_FENCE_MASK, THM_BACO_CNTL__BACO_SB_AXI_FENCE__SHIFT, 0, 0 },
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_DUMMY_EN_MASK, BACO_CNTL__BACO_DUMMY_EN__SHIFT, 0, 0 },
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_BIF_LCLK_SWITCH_MASK, BACO_CNTL__BACO_BIF_LCLK_SWITCH__SHIFT, 0, 0 },
+ { CMD_READMODIFYWRITE, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_EN_MASK, BACO_CNTL__BACO_EN__SHIFT, 0, 0 },
+ { CMD_WAITFOR, NBIF_HWID, 0, mmRCC_BACO_CNTL_MISC_BASE_IDX, mmBACO_CNTL, BACO_CNTL__BACO_MODE_MASK, 0, 0xffffffff, 0 }
+};
+
+static const struct soc15_baco_cmd_entry clean_baco_tbl[] =
+{
+ { CMD_WRITE, NBIF_HWID, 0, mmBIOS_SCRATCH_6_BASE_IDX, mmBIOS_SCRATCH_6, 0, 0, 0, 0 },
+ { CMD_WRITE, NBIF_HWID, 0, mmBIOS_SCRATCH_7_BASE_IDX, mmBIOS_SCRATCH_7, 0, 0, 0, 0 }
+};
+
+int vega12_baco_set_state(struct pp_hwmgr *hwmgr, enum BACO_STATE state)
+{
+ enum BACO_STATE cur_state;
+
+ smu9_baco_get_state(hwmgr, &cur_state);
+
+ if (cur_state == state)
+ /* aisc already in the target state */
+ return 0;
+
+ if (state == BACO_STATE_IN) {
+ if (soc15_baco_program_registers(hwmgr, pre_baco_tbl,
+ ARRAY_SIZE(pre_baco_tbl))) {
+ if (smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_EnterBaco, 0))
+ return -EINVAL;
+
+ if (soc15_baco_program_registers(hwmgr, enter_baco_tbl,
+ ARRAY_SIZE(enter_baco_tbl)))
+ return 0;
+ }
+ } else if (state == BACO_STATE_OUT) {
+ /* HW requires at least 20ms between regulator off and on */
+ msleep(20);
+ /* Execute Hardware BACO exit sequence */
+ if (soc15_baco_program_registers(hwmgr, exit_baco_tbl,
+ ARRAY_SIZE(exit_baco_tbl))) {
+ if (soc15_baco_program_registers(hwmgr, clean_baco_tbl,
+ ARRAY_SIZE(clean_baco_tbl)))
+ return 0;
+ }
+ }
+
+ return -EINVAL;
+}
--- /dev/null
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#ifndef __VEGA12_BACO_H__
+#define __VEGA12_BACO_H__
+#include "smu9_baco.h"
+
+extern int vega12_baco_set_state(struct pp_hwmgr *hwmgr, enum BACO_STATE state);
+
+#endif
#include "ppinterrupt.h"
#include "pp_overdriver.h"
#include "pp_thermal.h"
+#include "vega12_baco.h"
static int vega12_force_clock_level(struct pp_hwmgr *hwmgr,
.start_thermal_controller = vega12_start_thermal_controller,
.powergate_gfx = vega12_gfx_off_control,
.get_performance_level = vega12_get_performance_level,
+ .get_asic_baco_capability = smu9_baco_get_capability,
+ .get_asic_baco_state = smu9_baco_get_state,
+ .set_asic_baco_state = vega12_baco_set_state,
.get_ppfeature_status = vega12_get_ppfeature_status,
.set_ppfeature_status = vega12_set_ppfeature_status,
+
};
int vega12_hwmgr_init(struct pp_hwmgr *hwmgr)
#include "asic_reg/gc/gc_9_2_1_sh_mask.h"
#include "asic_reg/nbio/nbio_6_1_offset.h"
+#include "asic_reg/nbio/nbio_6_1_offset.h"
+#include "asic_reg/nbio/nbio_6_1_sh_mask.h"
#endif
#include "vega20_inc.h"
#include "vega20_ppsmc.h"
#include "vega20_baco.h"
+#include "vega20_smumgr.h"
return 0;
}
+
+int vega20_baco_apply_vdci_flush_workaround(struct pp_hwmgr *hwmgr)
+{
+ int ret = 0;
+
+ ret = vega20_set_pptable_driver_address(hwmgr);
+ if (ret)
+ return ret;
+
+ return smum_send_msg_to_smc(hwmgr, PPSMC_MSG_BacoWorkAroundFlushVDCI);
+}
extern int vega20_baco_get_capability(struct pp_hwmgr *hwmgr, bool *cap);
extern int vega20_baco_get_state(struct pp_hwmgr *hwmgr, enum BACO_STATE *state);
extern int vega20_baco_set_state(struct pp_hwmgr *hwmgr, enum BACO_STATE state);
+extern int vega20_baco_apply_vdci_flush_workaround(struct pp_hwmgr *hwmgr);
#endif
hwmgr->platform_descriptor.clockStep.memoryClock = 500;
data->total_active_cus = adev->gfx.cu_info.number;
+ data->is_custom_profile_set = false;
return 0;
}
static int vega20_setup_asic_task(struct pp_hwmgr *hwmgr)
{
+ struct amdgpu_device *adev = (struct amdgpu_device *)(hwmgr->adev);
int ret = 0;
ret = vega20_init_sclk_threshold(hwmgr);
"Failed to init sclk threshold!",
return ret);
- return 0;
+ if (adev->in_baco_reset) {
+ adev->in_baco_reset = 0;
+
+ ret = vega20_baco_apply_vdci_flush_workaround(hwmgr);
+ if (ret)
+ pr_err("Failed to apply vega20 baco workaround!\n");
+ }
+
+ return ret;
}
/*
return ;
data->vce_power_gated = bgate;
- vega20_enable_disable_vce_dpm(hwmgr, !bgate);
+ if (bgate) {
+ vega20_enable_disable_vce_dpm(hwmgr, !bgate);
+ amdgpu_device_ip_set_powergating_state(hwmgr->adev,
+ AMD_IP_BLOCK_TYPE_VCE,
+ AMD_PG_STATE_GATE);
+ } else {
+ amdgpu_device_ip_set_powergating_state(hwmgr->adev,
+ AMD_IP_BLOCK_TYPE_VCE,
+ AMD_PG_STATE_UNGATE);
+ vega20_enable_disable_vce_dpm(hwmgr, !bgate);
+ }
+
}
static void vega20_power_gate_uvd(struct pp_hwmgr *hwmgr, bool bgate)
{
DpmActivityMonitorCoeffInt_t activity_monitor;
int workload_type, result = 0;
+ uint32_t power_profile_mode = input[size];
- hwmgr->power_profile_mode = input[size];
-
- if (hwmgr->power_profile_mode > PP_SMC_POWER_PROFILE_CUSTOM) {
- pr_err("Invalid power profile mode %d\n", hwmgr->power_profile_mode);
+ if (power_profile_mode > PP_SMC_POWER_PROFILE_CUSTOM) {
+ pr_err("Invalid power profile mode %d\n", power_profile_mode);
return -EINVAL;
}
- if (hwmgr->power_profile_mode == PP_SMC_POWER_PROFILE_CUSTOM) {
- if (size < 10)
+ if (power_profile_mode == PP_SMC_POWER_PROFILE_CUSTOM) {
+ struct vega20_hwmgr *data =
+ (struct vega20_hwmgr *)(hwmgr->backend);
+ if (size == 0 && !data->is_custom_profile_set)
+ return -EINVAL;
+ if (size < 10 && size != 0)
return -EINVAL;
result = vega20_get_activity_monitor_coeff(hwmgr,
"[SetPowerProfile] Failed to get activity monitor!",
return result);
+ /* If size==0, then we want to apply the already-configured
+ * CUSTOM profile again. Just apply it, since we checked its
+ * validity above
+ */
+ if (size == 0)
+ goto out;
+
switch (input[0]) {
case 0: /* Gfxclk */
activity_monitor.Gfx_FPS = input[1];
result = vega20_set_activity_monitor_coeff(hwmgr,
(uint8_t *)(&activity_monitor),
WORKLOAD_PPLIB_CUSTOM_BIT);
+ data->is_custom_profile_set = true;
PP_ASSERT_WITH_CODE(!result,
"[SetPowerProfile] Failed to set activity monitor!",
return result);
}
+out:
/* conv PP_SMC_POWER_PROFILE* to WORKLOAD_PPLIB_*_BIT */
workload_type =
- conv_power_profile_to_pplib_workload(hwmgr->power_profile_mode);
+ conv_power_profile_to_pplib_workload(power_profile_mode);
smum_send_msg_to_smc_with_parameter(hwmgr, PPSMC_MSG_SetWorkloadMask,
1 << workload_type);
+ hwmgr->power_profile_mode = power_profile_mode;
+
return 0;
}
bool pcie_parameters_override;
uint32_t pcie_gen_level1;
uint32_t pcie_width_level1;
+
+ bool is_custom_profile_set;
};
#define VEGA20_DPM2_NEAR_TDP_DEC 10
--- /dev/null
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+#ifndef __AMDGPU_SMU_H__
+#define __AMDGPU_SMU_H__
+
+#include "amdgpu.h"
+#include "kgd_pp_interface.h"
+#include "dm_pp_interface.h"
+
+struct smu_hw_power_state {
+ unsigned int magic;
+};
+
+struct smu_power_state;
+
+enum smu_state_ui_label {
+ SMU_STATE_UI_LABEL_NONE,
+ SMU_STATE_UI_LABEL_BATTERY,
+ SMU_STATE_UI_TABEL_MIDDLE_LOW,
+ SMU_STATE_UI_LABEL_BALLANCED,
+ SMU_STATE_UI_LABEL_MIDDLE_HIGHT,
+ SMU_STATE_UI_LABEL_PERFORMANCE,
+ SMU_STATE_UI_LABEL_BACO,
+};
+
+enum smu_state_classification_flag {
+ SMU_STATE_CLASSIFICATION_FLAG_BOOT = 0x0001,
+ SMU_STATE_CLASSIFICATION_FLAG_THERMAL = 0x0002,
+ SMU_STATE_CLASSIFICATIN_FLAG_LIMITED_POWER_SOURCE = 0x0004,
+ SMU_STATE_CLASSIFICATION_FLAG_RESET = 0x0008,
+ SMU_STATE_CLASSIFICATION_FLAG_FORCED = 0x0010,
+ SMU_STATE_CLASSIFICATION_FLAG_USER_3D_PERFORMANCE = 0x0020,
+ SMU_STATE_CLASSIFICATION_FLAG_USER_2D_PERFORMANCE = 0x0040,
+ SMU_STATE_CLASSIFICATION_FLAG_3D_PERFORMANCE = 0x0080,
+ SMU_STATE_CLASSIFICATION_FLAG_AC_OVERDIRVER_TEMPLATE = 0x0100,
+ SMU_STATE_CLASSIFICATION_FLAG_UVD = 0x0200,
+ SMU_STATE_CLASSIFICATION_FLAG_3D_PERFORMANCE_LOW = 0x0400,
+ SMU_STATE_CLASSIFICATION_FLAG_ACPI = 0x0800,
+ SMU_STATE_CLASSIFICATION_FLAG_HD2 = 0x1000,
+ SMU_STATE_CLASSIFICATION_FLAG_UVD_HD = 0x2000,
+ SMU_STATE_CLASSIFICATION_FLAG_UVD_SD = 0x4000,
+ SMU_STATE_CLASSIFICATION_FLAG_USER_DC_PERFORMANCE = 0x8000,
+ SMU_STATE_CLASSIFICATION_FLAG_DC_OVERDIRVER_TEMPLATE = 0x10000,
+ SMU_STATE_CLASSIFICATION_FLAG_BACO = 0x20000,
+ SMU_STATE_CLASSIFICATIN_FLAG_LIMITED_POWER_SOURCE2 = 0x40000,
+ SMU_STATE_CLASSIFICATION_FLAG_ULV = 0x80000,
+ SMU_STATE_CLASSIFICATION_FLAG_UVD_MVC = 0x100000,
+};
+
+struct smu_state_classification_block {
+ enum smu_state_ui_label ui_label;
+ enum smu_state_classification_flag flags;
+ int bios_index;
+ bool temporary_state;
+ bool to_be_deleted;
+};
+
+struct smu_state_pcie_block {
+ unsigned int lanes;
+};
+
+enum smu_refreshrate_source {
+ SMU_REFRESHRATE_SOURCE_EDID,
+ SMU_REFRESHRATE_SOURCE_EXPLICIT
+};
+
+struct smu_state_display_block {
+ bool disable_frame_modulation;
+ bool limit_refreshrate;
+ enum smu_refreshrate_source refreshrate_source;
+ int explicit_refreshrate;
+ int edid_refreshrate_index;
+ bool enable_vari_bright;
+};
+
+struct smu_state_memroy_block {
+ bool dll_off;
+ uint8_t m3arb;
+ uint8_t unused[3];
+};
+
+struct smu_state_software_algorithm_block {
+ bool disable_load_balancing;
+ bool enable_sleep_for_timestamps;
+};
+
+struct smu_temperature_range {
+ int min;
+ int max;
+};
+
+struct smu_state_validation_block {
+ bool single_display_only;
+ bool disallow_on_dc;
+ uint8_t supported_power_levels;
+};
+
+struct smu_uvd_clocks {
+ uint32_t vclk;
+ uint32_t dclk;
+};
+
+/**
+* Structure to hold a SMU Power State.
+*/
+struct smu_power_state {
+ uint32_t id;
+ struct list_head ordered_list;
+ struct list_head all_states_list;
+
+ struct smu_state_classification_block classification;
+ struct smu_state_validation_block validation;
+ struct smu_state_pcie_block pcie;
+ struct smu_state_display_block display;
+ struct smu_state_memroy_block memory;
+ struct smu_temperature_range temperatures;
+ struct smu_state_software_algorithm_block software;
+ struct smu_uvd_clocks uvd_clocks;
+ struct smu_hw_power_state hardware;
+};
+
+enum smu_message_type
+{
+ SMU_MSG_TestMessage = 0,
+ SMU_MSG_GetSmuVersion,
+ SMU_MSG_GetDriverIfVersion,
+ SMU_MSG_SetAllowedFeaturesMaskLow,
+ SMU_MSG_SetAllowedFeaturesMaskHigh,
+ SMU_MSG_EnableAllSmuFeatures,
+ SMU_MSG_DisableAllSmuFeatures,
+ SMU_MSG_EnableSmuFeaturesLow,
+ SMU_MSG_EnableSmuFeaturesHigh,
+ SMU_MSG_DisableSmuFeaturesLow,
+ SMU_MSG_DisableSmuFeaturesHigh,
+ SMU_MSG_GetEnabledSmuFeaturesLow,
+ SMU_MSG_GetEnabledSmuFeaturesHigh,
+ SMU_MSG_SetWorkloadMask,
+ SMU_MSG_SetPptLimit,
+ SMU_MSG_SetDriverDramAddrHigh,
+ SMU_MSG_SetDriverDramAddrLow,
+ SMU_MSG_SetToolsDramAddrHigh,
+ SMU_MSG_SetToolsDramAddrLow,
+ SMU_MSG_TransferTableSmu2Dram,
+ SMU_MSG_TransferTableDram2Smu,
+ SMU_MSG_UseDefaultPPTable,
+ SMU_MSG_UseBackupPPTable,
+ SMU_MSG_RunBtc,
+ SMU_MSG_RequestI2CBus,
+ SMU_MSG_ReleaseI2CBus,
+ SMU_MSG_SetFloorSocVoltage,
+ SMU_MSG_SoftReset,
+ SMU_MSG_StartBacoMonitor,
+ SMU_MSG_CancelBacoMonitor,
+ SMU_MSG_EnterBaco,
+ SMU_MSG_SetSoftMinByFreq,
+ SMU_MSG_SetSoftMaxByFreq,
+ SMU_MSG_SetHardMinByFreq,
+ SMU_MSG_SetHardMaxByFreq,
+ SMU_MSG_GetMinDpmFreq,
+ SMU_MSG_GetMaxDpmFreq,
+ SMU_MSG_GetDpmFreqByIndex,
+ SMU_MSG_GetDpmClockFreq,
+ SMU_MSG_GetSsVoltageByDpm,
+ SMU_MSG_SetMemoryChannelConfig,
+ SMU_MSG_SetGeminiMode,
+ SMU_MSG_SetGeminiApertureHigh,
+ SMU_MSG_SetGeminiApertureLow,
+ SMU_MSG_SetMinLinkDpmByIndex,
+ SMU_MSG_OverridePcieParameters,
+ SMU_MSG_OverDriveSetPercentage,
+ SMU_MSG_SetMinDeepSleepDcefclk,
+ SMU_MSG_ReenableAcDcInterrupt,
+ SMU_MSG_NotifyPowerSource,
+ SMU_MSG_SetUclkFastSwitch,
+ SMU_MSG_SetUclkDownHyst,
+ SMU_MSG_GfxDeviceDriverReset,
+ SMU_MSG_GetCurrentRpm,
+ SMU_MSG_SetVideoFps,
+ SMU_MSG_SetTjMax,
+ SMU_MSG_SetFanTemperatureTarget,
+ SMU_MSG_PrepareMp1ForUnload,
+ SMU_MSG_DramLogSetDramAddrHigh,
+ SMU_MSG_DramLogSetDramAddrLow,
+ SMU_MSG_DramLogSetDramSize,
+ SMU_MSG_SetFanMaxRpm,
+ SMU_MSG_SetFanMinPwm,
+ SMU_MSG_ConfigureGfxDidt,
+ SMU_MSG_NumOfDisplays,
+ SMU_MSG_RemoveMargins,
+ SMU_MSG_ReadSerialNumTop32,
+ SMU_MSG_ReadSerialNumBottom32,
+ SMU_MSG_SetSystemVirtualDramAddrHigh,
+ SMU_MSG_SetSystemVirtualDramAddrLow,
+ SMU_MSG_WaflTest,
+ SMU_MSG_SetFclkGfxClkRatio,
+ SMU_MSG_AllowGfxOff,
+ SMU_MSG_DisallowGfxOff,
+ SMU_MSG_GetPptLimit,
+ SMU_MSG_GetDcModeMaxDpmFreq,
+ SMU_MSG_GetDebugData,
+ SMU_MSG_SetXgmiMode,
+ SMU_MSG_RunAfllBtc,
+ SMU_MSG_ExitBaco,
+ SMU_MSG_PrepareMp1ForReset,
+ SMU_MSG_PrepareMp1ForShutdown,
+ SMU_MSG_SetMGpuFanBoostLimitRpm,
+ SMU_MSG_GetAVFSVoltageByDpm,
+ SMU_MSG_MAX_COUNT,
+};
+
+enum smu_memory_pool_size
+{
+ SMU_MEMORY_POOL_SIZE_ZERO = 0,
+ SMU_MEMORY_POOL_SIZE_256_MB = 0x10000000,
+ SMU_MEMORY_POOL_SIZE_512_MB = 0x20000000,
+ SMU_MEMORY_POOL_SIZE_1_GB = 0x40000000,
+ SMU_MEMORY_POOL_SIZE_2_GB = 0x80000000,
+};
+
+#define SMU_TABLE_INIT(tables, table_id, s, a, d) \
+ do { \
+ tables[table_id].size = s; \
+ tables[table_id].align = a; \
+ tables[table_id].domain = d; \
+ } while (0)
+
+struct smu_table {
+ uint64_t size;
+ uint32_t align;
+ uint8_t domain;
+ uint64_t mc_address;
+ void *cpu_addr;
+ struct amdgpu_bo *bo;
+};
+
+enum smu_perf_level_designation {
+ PERF_LEVEL_ACTIVITY,
+ PERF_LEVEL_POWER_CONTAINMENT,
+};
+
+struct smu_performance_level {
+ uint32_t core_clock;
+ uint32_t memory_clock;
+ uint32_t vddc;
+ uint32_t vddci;
+ uint32_t non_local_mem_freq;
+ uint32_t non_local_mem_width;
+};
+
+struct smu_clock_info {
+ uint32_t min_mem_clk;
+ uint32_t max_mem_clk;
+ uint32_t min_eng_clk;
+ uint32_t max_eng_clk;
+ uint32_t min_bus_bandwidth;
+ uint32_t max_bus_bandwidth;
+};
+
+struct smu_bios_boot_up_values
+{
+ uint32_t revision;
+ uint32_t gfxclk;
+ uint32_t uclk;
+ uint32_t socclk;
+ uint32_t dcefclk;
+ uint32_t eclk;
+ uint32_t vclk;
+ uint32_t dclk;
+ uint16_t vddc;
+ uint16_t vddci;
+ uint16_t mvddc;
+ uint16_t vdd_gfx;
+ uint8_t cooling_id;
+ uint32_t pp_table_id;
+};
+
+struct smu_table_context
+{
+ void *power_play_table;
+ uint32_t power_play_table_size;
+ void *hardcode_pptable;
+
+ void *max_sustainable_clocks;
+ struct smu_bios_boot_up_values boot_values;
+ void *driver_pptable;
+ struct smu_table *tables;
+ uint32_t table_count;
+ struct smu_table memory_pool;
+ uint16_t software_shutdown_temp;
+ uint8_t thermal_controller_type;
+ uint16_t TDPODLimit;
+
+ uint8_t *od_feature_capabilities;
+ uint32_t *od_settings_max;
+ uint32_t *od_settings_min;
+ void *overdrive_table;
+ void *od8_settings;
+ bool od_gfxclk_update;
+ bool od_memclk_update;
+};
+
+struct smu_dpm_context {
+ uint32_t dpm_context_size;
+ void *dpm_context;
+ void *golden_dpm_context;
+ bool enable_umd_pstate;
+ enum amd_dpm_forced_level dpm_level;
+ enum amd_dpm_forced_level saved_dpm_level;
+ enum amd_dpm_forced_level requested_dpm_level;
+ struct smu_power_state *dpm_request_power_state;
+ struct smu_power_state *dpm_current_power_state;
+ struct mclock_latency_table *mclk_latency_table;
+};
+
+struct smu_power_context {
+ void *power_context;
+ uint32_t power_context_size;
+};
+
+
+#define SMU_FEATURE_MAX (64)
+struct smu_feature
+{
+ uint32_t feature_num;
+ DECLARE_BITMAP(supported, SMU_FEATURE_MAX);
+ DECLARE_BITMAP(allowed, SMU_FEATURE_MAX);
+ DECLARE_BITMAP(enabled, SMU_FEATURE_MAX);
+ struct mutex mutex;
+};
+
+struct smu_clocks {
+ uint32_t engine_clock;
+ uint32_t memory_clock;
+ uint32_t bus_bandwidth;
+ uint32_t engine_clock_in_sr;
+ uint32_t dcef_clock;
+ uint32_t dcef_clock_in_sr;
+};
+
+#define MAX_REGULAR_DPM_NUM 16
+struct mclk_latency_entries {
+ uint32_t frequency;
+ uint32_t latency;
+};
+struct mclock_latency_table {
+ uint32_t count;
+ struct mclk_latency_entries entries[MAX_REGULAR_DPM_NUM];
+};
+
+#define WORKLOAD_POLICY_MAX 7
+struct smu_context
+{
+ struct amdgpu_device *adev;
+
+ const struct smu_funcs *funcs;
+ const struct pptable_funcs *ppt_funcs;
+ struct mutex mutex;
+ uint64_t pool_size;
+
+ struct smu_table_context smu_table;
+ struct smu_dpm_context smu_dpm;
+ struct smu_power_context smu_power;
+ struct smu_feature smu_feature;
+ struct amd_pp_display_configuration *display_config;
+
+ uint32_t pstate_sclk;
+ uint32_t pstate_mclk;
+
+ bool od_enabled;
+ uint32_t power_limit;
+ uint32_t default_power_limit;
+
+ bool support_power_containment;
+ bool disable_watermark;
+
+#define WATERMARKS_EXIST (1 << 0)
+#define WATERMARKS_LOADED (1 << 1)
+ uint32_t watermarks_bitmap;
+
+ uint32_t workload_mask;
+ uint32_t workload_prority[WORKLOAD_POLICY_MAX];
+ uint32_t workload_setting[WORKLOAD_POLICY_MAX];
+ uint32_t power_profile_mode;
+ uint32_t default_power_profile_mode;
+
+ uint32_t smc_if_version;
+};
+
+struct pptable_funcs {
+ int (*alloc_dpm_context)(struct smu_context *smu);
+ int (*store_powerplay_table)(struct smu_context *smu);
+ int (*check_powerplay_table)(struct smu_context *smu);
+ int (*append_powerplay_table)(struct smu_context *smu);
+ int (*get_smu_msg_index)(struct smu_context *smu, uint32_t index);
+ int (*run_afll_btc)(struct smu_context *smu);
+ int (*get_unallowed_feature_mask)(struct smu_context *smu, uint32_t *feature_mask, uint32_t num);
+ enum amd_pm_state_type (*get_current_power_state)(struct smu_context *smu);
+ int (*set_default_dpm_table)(struct smu_context *smu);
+ int (*set_power_state)(struct smu_context *smu);
+ int (*populate_umd_state_clk)(struct smu_context *smu);
+ int (*print_clk_levels)(struct smu_context *smu, enum pp_clock_type type, char *buf);
+ int (*force_clk_levels)(struct smu_context *smu, enum pp_clock_type type, uint32_t mask);
+ int (*set_default_od8_settings)(struct smu_context *smu);
+ int (*update_specified_od8_value)(struct smu_context *smu,
+ uint32_t index,
+ uint32_t value);
+ int (*get_od_percentage)(struct smu_context *smu, enum pp_clock_type type);
+ int (*set_od_percentage)(struct smu_context *smu,
+ enum pp_clock_type type,
+ uint32_t value);
+ int (*od_edit_dpm_table)(struct smu_context *smu,
+ enum PP_OD_DPM_TABLE_COMMAND type,
+ long *input, uint32_t size);
+ int (*get_clock_by_type_with_latency)(struct smu_context *smu,
+ enum amd_pp_clock_type type,
+ struct
+ pp_clock_levels_with_latency
+ *clocks);
+ int (*get_clock_by_type_with_voltage)(struct smu_context *smu,
+ enum amd_pp_clock_type type,
+ struct
+ pp_clock_levels_with_voltage
+ *clocks);
+ int (*get_power_profile_mode)(struct smu_context *smu, char *buf);
+ int (*set_power_profile_mode)(struct smu_context *smu, long *input, uint32_t size);
+ enum amd_dpm_forced_level (*get_performance_level)(struct smu_context *smu);
+ int (*force_performance_level)(struct smu_context *smu, enum amd_dpm_forced_level level);
+ int (*pre_display_config_changed)(struct smu_context *smu);
+ int (*display_config_changed)(struct smu_context *smu);
+ int (*apply_clocks_adjust_rules)(struct smu_context *smu);
+ int (*notify_smc_dispaly_config)(struct smu_context *smu);
+ int (*force_dpm_limit_value)(struct smu_context *smu, bool highest);
+ int (*unforce_dpm_levels)(struct smu_context *smu);
+ int (*upload_dpm_level)(struct smu_context *smu, bool max,
+ uint32_t feature_mask);
+ int (*get_profiling_clk_mask)(struct smu_context *smu,
+ enum amd_dpm_forced_level level,
+ uint32_t *sclk_mask,
+ uint32_t *mclk_mask,
+ uint32_t *soc_mask);
+ int (*set_cpu_power_state)(struct smu_context *smu);
+};
+
+struct smu_funcs
+{
+ int (*init_microcode)(struct smu_context *smu);
+ int (*init_smc_tables)(struct smu_context *smu);
+ int (*fini_smc_tables)(struct smu_context *smu);
+ int (*init_power)(struct smu_context *smu);
+ int (*fini_power)(struct smu_context *smu);
+ int (*load_microcode)(struct smu_context *smu);
+ int (*check_fw_status)(struct smu_context *smu);
+ int (*read_pptable_from_vbios)(struct smu_context *smu);
+ int (*get_vbios_bootup_values)(struct smu_context *smu);
+ int (*get_clk_info_from_vbios)(struct smu_context *smu);
+ int (*check_pptable)(struct smu_context *smu);
+ int (*parse_pptable)(struct smu_context *smu);
+ int (*populate_smc_pptable)(struct smu_context *smu);
+ int (*check_fw_version)(struct smu_context *smu);
+ int (*write_pptable)(struct smu_context *smu);
+ int (*set_min_dcef_deep_sleep)(struct smu_context *smu);
+ int (*set_tool_table_location)(struct smu_context *smu);
+ int (*notify_memory_pool_location)(struct smu_context *smu);
+ int (*write_watermarks_table)(struct smu_context *smu);
+ int (*set_last_dcef_min_deep_sleep_clk)(struct smu_context *smu);
+ int (*system_features_control)(struct smu_context *smu, bool en);
+ int (*send_smc_msg)(struct smu_context *smu, uint16_t msg);
+ int (*send_smc_msg_with_param)(struct smu_context *smu, uint16_t msg, uint32_t param);
+ int (*read_smc_arg)(struct smu_context *smu, uint32_t *arg);
+ int (*init_display)(struct smu_context *smu);
+ int (*set_allowed_mask)(struct smu_context *smu);
+ int (*get_enabled_mask)(struct smu_context *smu, uint32_t *feature_mask, uint32_t num);
+ bool (*is_dpm_running)(struct smu_context *smu);
+ int (*update_feature_enable_state)(struct smu_context *smu, uint32_t feature_id, bool enabled);
+ int (*notify_display_change)(struct smu_context *smu);
+ int (*get_power_limit)(struct smu_context *smu, uint32_t *limit, bool def);
+ int (*set_power_limit)(struct smu_context *smu, uint32_t n);
+ int (*get_current_clk_freq)(struct smu_context *smu, uint32_t clk_id, uint32_t *value);
+ int (*init_max_sustainable_clocks)(struct smu_context *smu);
+ int (*start_thermal_control)(struct smu_context *smu);
+ int (*read_sensor)(struct smu_context *smu, enum amd_pp_sensors sensor,
+ void *data, uint32_t *size);
+ int (*set_deep_sleep_dcefclk)(struct smu_context *smu, uint32_t clk);
+ int (*set_active_display_count)(struct smu_context *smu, uint32_t count);
+ int (*store_cc6_data)(struct smu_context *smu, uint32_t separation_time,
+ bool cc6_disable, bool pstate_disable,
+ bool pstate_switch_disable);
+ int (*get_clock_by_type)(struct smu_context *smu,
+ enum amd_pp_clock_type type,
+ struct amd_pp_clocks *clocks);
+ int (*get_max_high_clocks)(struct smu_context *smu,
+ struct amd_pp_simple_clock_info *clocks);
+ int (*display_clock_voltage_request)(struct smu_context *smu, struct
+ pp_display_clock_request
+ *clock_req);
+ int (*get_dal_power_level)(struct smu_context *smu,
+ struct amd_pp_simple_clock_info *clocks);
+ int (*get_perf_level)(struct smu_context *smu,
+ enum smu_perf_level_designation designation,
+ struct smu_performance_level *level);
+ int (*get_current_shallow_sleep_clocks)(struct smu_context *smu,
+ struct smu_clock_info *clocks);
+ int (*notify_smu_enable_pwe)(struct smu_context *smu);
+ int (*set_watermarks_for_clock_ranges)(struct smu_context *smu,
+ struct dm_pp_wm_sets_with_clock_ranges_soc15 *clock_ranges);
+ int (*set_od8_default_settings)(struct smu_context *smu,
+ bool initialize);
+ int (*conv_power_profile_to_pplib_workload)(int power_profile);
+ int (*get_power_profile_mode)(struct smu_context *smu, char *buf);
+ int (*set_power_profile_mode)(struct smu_context *smu, long *input, uint32_t size);
+ int (*update_od8_settings)(struct smu_context *smu,
+ uint32_t index,
+ uint32_t value);
+ int (*dpm_set_uvd_enable)(struct smu_context *smu, bool enable);
+ int (*dpm_set_vce_enable)(struct smu_context *smu, bool enable);
+ uint32_t (*get_sclk)(struct smu_context *smu, bool low);
+ uint32_t (*get_mclk)(struct smu_context *smu, bool low);
+ int (*get_current_rpm)(struct smu_context *smu, uint32_t *speed);
+ uint32_t (*get_fan_control_mode)(struct smu_context *smu);
+ int (*set_fan_control_mode)(struct smu_context *smu, uint32_t mode);
+ int (*get_fan_speed_percent)(struct smu_context *smu, uint32_t *speed);
+ int (*set_fan_speed_percent)(struct smu_context *smu, uint32_t speed);
+ int (*set_fan_speed_rpm)(struct smu_context *smu, uint32_t speed);
+ int (*set_xgmi_pstate)(struct smu_context *smu, uint32_t pstate);
+
+};
+
+#define smu_init_microcode(smu) \
+ ((smu)->funcs->init_microcode ? (smu)->funcs->init_microcode((smu)) : 0)
+#define smu_init_smc_tables(smu) \
+ ((smu)->funcs->init_smc_tables ? (smu)->funcs->init_smc_tables((smu)) : 0)
+#define smu_fini_smc_tables(smu) \
+ ((smu)->funcs->fini_smc_tables ? (smu)->funcs->fini_smc_tables((smu)) : 0)
+#define smu_init_power(smu) \
+ ((smu)->funcs->init_power ? (smu)->funcs->init_power((smu)) : 0)
+#define smu_fini_power(smu) \
+ ((smu)->funcs->fini_power ? (smu)->funcs->fini_power((smu)) : 0)
+#define smu_load_microcode(smu) \
+ ((smu)->funcs->load_microcode ? (smu)->funcs->load_microcode((smu)) : 0)
+#define smu_check_fw_status(smu) \
+ ((smu)->funcs->check_fw_status ? (smu)->funcs->check_fw_status((smu)) : 0)
+#define smu_read_pptable_from_vbios(smu) \
+ ((smu)->funcs->read_pptable_from_vbios ? (smu)->funcs->read_pptable_from_vbios((smu)) : 0)
+#define smu_get_vbios_bootup_values(smu) \
+ ((smu)->funcs->get_vbios_bootup_values ? (smu)->funcs->get_vbios_bootup_values((smu)) : 0)
+#define smu_get_clk_info_from_vbios(smu) \
+ ((smu)->funcs->get_clk_info_from_vbios ? (smu)->funcs->get_clk_info_from_vbios((smu)) : 0)
+#define smu_check_pptable(smu) \
+ ((smu)->funcs->check_pptable ? (smu)->funcs->check_pptable((smu)) : 0)
+#define smu_parse_pptable(smu) \
+ ((smu)->funcs->parse_pptable ? (smu)->funcs->parse_pptable((smu)) : 0)
+#define smu_populate_smc_pptable(smu) \
+ ((smu)->funcs->populate_smc_pptable ? (smu)->funcs->populate_smc_pptable((smu)) : 0)
+#define smu_check_fw_version(smu) \
+ ((smu)->funcs->check_fw_version ? (smu)->funcs->check_fw_version((smu)) : 0)
+#define smu_write_pptable(smu) \
+ ((smu)->funcs->write_pptable ? (smu)->funcs->write_pptable((smu)) : 0)
+#define smu_set_min_dcef_deep_sleep(smu) \
+ ((smu)->funcs->set_min_dcef_deep_sleep ? (smu)->funcs->set_min_dcef_deep_sleep((smu)) : 0)
+#define smu_set_tool_table_location(smu) \
+ ((smu)->funcs->set_tool_table_location ? (smu)->funcs->set_tool_table_location((smu)) : 0)
+#define smu_notify_memory_pool_location(smu) \
+ ((smu)->funcs->notify_memory_pool_location ? (smu)->funcs->notify_memory_pool_location((smu)) : 0)
+#define smu_write_watermarks_table(smu) \
+ ((smu)->funcs->write_watermarks_table ? (smu)->funcs->write_watermarks_table((smu)) : 0)
+#define smu_set_last_dcef_min_deep_sleep_clk(smu) \
+ ((smu)->funcs->set_last_dcef_min_deep_sleep_clk ? (smu)->funcs->set_last_dcef_min_deep_sleep_clk((smu)) : 0)
+#define smu_system_features_control(smu, en) \
+ ((smu)->funcs->system_features_control ? (smu)->funcs->system_features_control((smu), (en)) : 0)
+#define smu_init_max_sustainable_clocks(smu) \
+ ((smu)->funcs->init_max_sustainable_clocks ? (smu)->funcs->init_max_sustainable_clocks((smu)) : 0)
+#define smu_set_od8_default_settings(smu, initialize) \
+ ((smu)->funcs->set_od8_default_settings ? (smu)->funcs->set_od8_default_settings((smu), (initialize)) : 0)
+#define smu_update_od8_settings(smu, index, value) \
+ ((smu)->funcs->update_od8_settings ? (smu)->funcs->update_od8_settings((smu), (index), (value)) : 0)
+#define smu_get_current_rpm(smu, speed) \
+ ((smu)->funcs->get_current_rpm ? (smu)->funcs->get_current_rpm((smu), (speed)) : 0)
+#define smu_set_fan_speed_rpm(smu, speed) \
+ ((smu)->funcs->set_fan_speed_rpm ? (smu)->funcs->set_fan_speed_rpm((smu), (speed)) : 0)
+#define smu_send_smc_msg(smu, msg) \
+ ((smu)->funcs->send_smc_msg? (smu)->funcs->send_smc_msg((smu), (msg)) : 0)
+#define smu_send_smc_msg_with_param(smu, msg, param) \
+ ((smu)->funcs->send_smc_msg_with_param? (smu)->funcs->send_smc_msg_with_param((smu), (msg), (param)) : 0)
+#define smu_read_smc_arg(smu, arg) \
+ ((smu)->funcs->read_smc_arg? (smu)->funcs->read_smc_arg((smu), (arg)) : 0)
+#define smu_alloc_dpm_context(smu) \
+ ((smu)->ppt_funcs->alloc_dpm_context ? (smu)->ppt_funcs->alloc_dpm_context((smu)) : 0)
+#define smu_init_display(smu) \
+ ((smu)->funcs->init_display ? (smu)->funcs->init_display((smu)) : 0)
+#define smu_feature_set_allowed_mask(smu) \
+ ((smu)->funcs->set_allowed_mask? (smu)->funcs->set_allowed_mask((smu)) : 0)
+#define smu_feature_get_enabled_mask(smu, mask, num) \
+ ((smu)->funcs->get_enabled_mask? (smu)->funcs->get_enabled_mask((smu), (mask), (num)) : 0)
+#define smu_is_dpm_running(smu) \
+ ((smu)->funcs->is_dpm_running ? (smu)->funcs->is_dpm_running((smu)) : 0)
+#define smu_feature_update_enable_state(smu, feature_id, enabled) \
+ ((smu)->funcs->update_feature_enable_state? (smu)->funcs->update_feature_enable_state((smu), (feature_id), (enabled)) : 0)
+#define smu_notify_display_change(smu) \
+ ((smu)->funcs->notify_display_change? (smu)->funcs->notify_display_change((smu)) : 0)
+#define smu_store_powerplay_table(smu) \
+ ((smu)->ppt_funcs->store_powerplay_table ? (smu)->ppt_funcs->store_powerplay_table((smu)) : 0)
+#define smu_check_powerplay_table(smu) \
+ ((smu)->ppt_funcs->check_powerplay_table ? (smu)->ppt_funcs->check_powerplay_table((smu)) : 0)
+#define smu_append_powerplay_table(smu) \
+ ((smu)->ppt_funcs->append_powerplay_table ? (smu)->ppt_funcs->append_powerplay_table((smu)) : 0)
+#define smu_set_default_dpm_table(smu) \
+ ((smu)->ppt_funcs->set_default_dpm_table ? (smu)->ppt_funcs->set_default_dpm_table((smu)) : 0)
+#define smu_populate_umd_state_clk(smu) \
+ ((smu)->ppt_funcs->populate_umd_state_clk ? (smu)->ppt_funcs->populate_umd_state_clk((smu)) : 0)
+#define smu_set_default_od8_settings(smu) \
+ ((smu)->ppt_funcs->set_default_od8_settings ? (smu)->ppt_funcs->set_default_od8_settings((smu)) : 0)
+#define smu_update_specified_od8_value(smu, index, value) \
+ ((smu)->ppt_funcs->update_specified_od8_value ? (smu)->ppt_funcs->update_specified_od8_value((smu), (index), (value)) : 0)
+#define smu_get_power_limit(smu, limit, def) \
+ ((smu)->funcs->get_power_limit ? (smu)->funcs->get_power_limit((smu), (limit), (def)) : 0)
+#define smu_set_power_limit(smu, limit) \
+ ((smu)->funcs->set_power_limit ? (smu)->funcs->set_power_limit((smu), (limit)) : 0)
+#define smu_get_current_clk_freq(smu, clk_id, value) \
+ ((smu)->funcs->get_current_clk_freq? (smu)->funcs->get_current_clk_freq((smu), (clk_id), (value)) : 0)
+#define smu_print_clk_levels(smu, type, buf) \
+ ((smu)->ppt_funcs->print_clk_levels ? (smu)->ppt_funcs->print_clk_levels((smu), (type), (buf)) : 0)
+#define smu_force_clk_levels(smu, type, level) \
+ ((smu)->ppt_funcs->force_clk_levels ? (smu)->ppt_funcs->force_clk_levels((smu), (type), (level)) : 0)
+#define smu_get_od_percentage(smu, type) \
+ ((smu)->ppt_funcs->get_od_percentage ? (smu)->ppt_funcs->get_od_percentage((smu), (type)) : 0)
+#define smu_set_od_percentage(smu, type, value) \
+ ((smu)->ppt_funcs->set_od_percentage ? (smu)->ppt_funcs->set_od_percentage((smu), (type), (value)) : 0)
+#define smu_od_edit_dpm_table(smu, type, input, size) \
+ ((smu)->ppt_funcs->od_edit_dpm_table ? (smu)->ppt_funcs->od_edit_dpm_table((smu), (type), (input), (size)) : 0)
+#define smu_start_thermal_control(smu) \
+ ((smu)->funcs->start_thermal_control? (smu)->funcs->start_thermal_control((smu)) : 0)
+#define smu_read_sensor(smu, sensor, data, size) \
+ ((smu)->funcs->read_sensor? (smu)->funcs->read_sensor((smu), (sensor), (data), (size)) : 0)
+#define smu_get_power_profile_mode(smu, buf) \
+ ((smu)->funcs->get_power_profile_mode ? (smu)->funcs->get_power_profile_mode((smu), buf) : 0)
+#define smu_set_power_profile_mode(smu, param, param_size) \
+ ((smu)->funcs->set_power_profile_mode ? (smu)->funcs->set_power_profile_mode((smu), (param), (param_size)) : 0)
+#define smu_get_performance_level(smu) \
+ ((smu)->ppt_funcs->get_performance_level ? (smu)->ppt_funcs->get_performance_level((smu)) : 0)
+#define smu_force_performance_level(smu, level) \
+ ((smu)->ppt_funcs->force_performance_level ? (smu)->ppt_funcs->force_performance_level((smu), (level)) : 0)
+#define smu_pre_display_config_changed(smu) \
+ ((smu)->ppt_funcs->pre_display_config_changed ? (smu)->ppt_funcs->pre_display_config_changed((smu)) : 0)
+#define smu_display_config_changed(smu) \
+ ((smu)->ppt_funcs->display_config_changed ? (smu)->ppt_funcs->display_config_changed((smu)) : 0)
+#define smu_apply_clocks_adjust_rules(smu) \
+ ((smu)->ppt_funcs->apply_clocks_adjust_rules ? (smu)->ppt_funcs->apply_clocks_adjust_rules((smu)) : 0)
+#define smu_notify_smc_dispaly_config(smu) \
+ ((smu)->ppt_funcs->notify_smc_dispaly_config ? (smu)->ppt_funcs->notify_smc_dispaly_config((smu)) : 0)
+#define smu_force_dpm_limit_value(smu, highest) \
+ ((smu)->ppt_funcs->force_dpm_limit_value ? (smu)->ppt_funcs->force_dpm_limit_value((smu), (highest)) : 0)
+#define smu_unforce_dpm_levels(smu) \
+ ((smu)->ppt_funcs->unforce_dpm_levels ? (smu)->ppt_funcs->unforce_dpm_levels((smu)) : 0)
+#define smu_upload_dpm_level(smu, max, feature_mask) \
+ ((smu)->ppt_funcs->upload_dpm_level ? (smu)->ppt_funcs->upload_dpm_level((smu), (max), (feature_mask)) : 0)
+#define smu_get_profiling_clk_mask(smu, level, sclk_mask, mclk_mask, soc_mask) \
+ ((smu)->ppt_funcs->get_profiling_clk_mask ? (smu)->ppt_funcs->get_profiling_clk_mask((smu), (level), (sclk_mask), (mclk_mask), (soc_mask)) : 0)
+#define smu_set_cpu_power_state(smu) \
+ ((smu)->ppt_funcs->set_cpu_power_state ? (smu)->ppt_funcs->set_cpu_power_state((smu)) : 0)
+#define smu_get_fan_control_mode(smu) \
+ ((smu)->funcs->get_fan_control_mode ? (smu)->funcs->get_fan_control_mode((smu)) : 0)
+#define smu_set_fan_control_mode(smu, value) \
+ ((smu)->funcs->set_fan_control_mode ? (smu)->funcs->set_fan_control_mode((smu), (value)) : 0)
+#define smu_get_fan_speed_percent(smu, speed) \
+ ((smu)->funcs->get_fan_speed_percent ? (smu)->funcs->get_fan_speed_percent((smu), (speed)) : 0)
+#define smu_set_fan_speed_percent(smu, speed) \
+ ((smu)->funcs->set_fan_speed_percent ? (smu)->funcs->set_fan_speed_percent((smu), (speed)) : 0)
+
+#define smu_msg_get_index(smu, msg) \
+ ((smu)->ppt_funcs? ((smu)->ppt_funcs->get_smu_msg_index? (smu)->ppt_funcs->get_smu_msg_index((smu), (msg)) : -EINVAL) : -EINVAL)
+#define smu_run_afll_btc(smu) \
+ ((smu)->ppt_funcs? ((smu)->ppt_funcs->run_afll_btc? (smu)->ppt_funcs->run_afll_btc((smu)) : 0) : 0)
+#define smu_get_unallowed_feature_mask(smu, feature_mask, num) \
+ ((smu)->ppt_funcs? ((smu)->ppt_funcs->get_unallowed_feature_mask? (smu)->ppt_funcs->get_unallowed_feature_mask((smu), (feature_mask), (num)) : 0) : 0)
+#define smu_set_deep_sleep_dcefclk(smu, clk) \
+ ((smu)->funcs->set_deep_sleep_dcefclk ? (smu)->funcs->set_deep_sleep_dcefclk((smu), (clk)) : 0)
+#define smu_set_active_display_count(smu, count) \
+ ((smu)->funcs->set_active_display_count ? (smu)->funcs->set_active_display_count((smu), (count)) : 0)
+#define smu_store_cc6_data(smu, st, cc6_dis, pst_dis, pst_sw_dis) \
+ ((smu)->funcs->store_cc6_data ? (smu)->funcs->store_cc6_data((smu), (st), (cc6_dis), (pst_dis), (pst_sw_dis)) : 0)
+#define smu_get_clock_by_type(smu, type, clocks) \
+ ((smu)->funcs->get_clock_by_type ? (smu)->funcs->get_clock_by_type((smu), (type), (clocks)) : 0)
+#define smu_get_max_high_clocks(smu, clocks) \
+ ((smu)->funcs->get_max_high_clocks ? (smu)->funcs->get_max_high_clocks((smu), (clocks)) : 0)
+#define smu_get_clock_by_type_with_latency(smu, type, clocks) \
+ ((smu)->ppt_funcs->get_clock_by_type_with_latency ? (smu)->ppt_funcs->get_clock_by_type_with_latency((smu), (type), (clocks)) : 0)
+#define smu_get_clock_by_type_with_voltage(smu, type, clocks) \
+ ((smu)->ppt_funcs->get_clock_by_type_with_voltage ? (smu)->ppt_funcs->get_clock_by_type_with_voltage((smu), (type), (clocks)) : 0)
+#define smu_display_clock_voltage_request(smu, clock_req) \
+ ((smu)->funcs->display_clock_voltage_request ? (smu)->funcs->display_clock_voltage_request((smu), (clock_req)) : 0)
+#define smu_get_dal_power_level(smu, clocks) \
+ ((smu)->funcs->get_dal_power_level ? (smu)->funcs->get_dal_power_level((smu), (clocks)) : 0)
+#define smu_get_perf_level(smu, designation, level) \
+ ((smu)->funcs->get_perf_level ? (smu)->funcs->get_perf_level((smu), (designation), (level)) : 0)
+#define smu_get_current_shallow_sleep_clocks(smu, clocks) \
+ ((smu)->funcs->get_current_shallow_sleep_clocks ? (smu)->funcs->get_current_shallow_sleep_clocks((smu), (clocks)) : 0)
+#define smu_notify_smu_enable_pwe(smu) \
+ ((smu)->funcs->notify_smu_enable_pwe ? (smu)->funcs->notify_smu_enable_pwe((smu)) : 0)
+#define smu_set_watermarks_for_clock_ranges(smu, clock_ranges) \
+ ((smu)->funcs->set_watermarks_for_clock_ranges ? (smu)->funcs->set_watermarks_for_clock_ranges((smu), (clock_ranges)) : 0)
+#define smu_dpm_set_uvd_enable(smu, enable) \
+ ((smu)->funcs->dpm_set_uvd_enable ? (smu)->funcs->dpm_set_uvd_enable((smu), (enable)) : 0)
+#define smu_dpm_set_vce_enable(smu, enable) \
+ ((smu)->funcs->dpm_set_vce_enable ? (smu)->funcs->dpm_set_vce_enable((smu), (enable)) : 0)
+#define smu_get_sclk(smu, low) \
+ ((smu)->funcs->get_sclk ? (smu)->funcs->get_sclk((smu), (low)) : 0)
+#define smu_get_mclk(smu, low) \
+ ((smu)->funcs->get_mclk ? (smu)->funcs->get_mclk((smu), (low)) : 0)
+#define smu_set_xgmi_pstate(smu, pstate) \
+ ((smu)->funcs->set_xgmi_pstate ? (smu)->funcs->set_xgmi_pstate((smu), (pstate)) : 0)
+
+
+extern int smu_get_atom_data_table(struct smu_context *smu, uint32_t table,
+ uint16_t *size, uint8_t *frev, uint8_t *crev,
+ uint8_t **addr);
+
+extern const struct amd_ip_funcs smu_ip_funcs;
+
+extern const struct amdgpu_ip_block_version smu_v11_0_ip_block;
+extern int smu_feature_init_dpm(struct smu_context *smu);
+
+extern int smu_feature_is_enabled(struct smu_context *smu, int feature_id);
+extern int smu_feature_set_enabled(struct smu_context *smu, int feature_id, bool enable);
+extern int smu_feature_is_supported(struct smu_context *smu, int feature_id);
+extern int smu_feature_set_supported(struct smu_context *smu, int feature_id, bool enable);
+
+int smu_update_table_with_arg(struct smu_context *smu, uint16_t table_id, uint16_t exarg,
+ void *table_data, bool drv2smu);
+#define smu_update_table(smu, table_id, table_data, drv2smu) \
+ smu_update_table_with_arg((smu), (table_id), 0, (table_data), (drv2smu))
+
+bool is_support_sw_smu(struct amdgpu_device *adev);
+int smu_reset(struct smu_context *smu);
+int smu_common_read_sensor(struct smu_context *smu, enum amd_pp_sensors sensor,
+ void *data, uint32_t *size);
+int smu_sys_get_pp_table(struct smu_context *smu, void **table);
+int smu_sys_set_pp_table(struct smu_context *smu, void *buf, size_t size);
+int smu_get_power_num_states(struct smu_context *smu, struct pp_states_info *state_info);
+enum amd_pm_state_type smu_get_current_power_state(struct smu_context *smu);
+
+/* smu to display interface */
+extern int smu_display_configuration_change(struct smu_context *smu, const
+ struct amd_pp_display_configuration
+ *display_config);
+extern int smu_get_current_clocks(struct smu_context *smu,
+ struct amd_pp_clock_info *clocks);
+extern int smu_dpm_set_power_gate(struct smu_context *smu,uint32_t block_type, bool gate);
+extern int smu_handle_task(struct smu_context *smu,
+ enum amd_dpm_forced_level level,
+ enum amd_pp_task task_id);
+#endif
#define PPSMC_MSG_SetRccPfcPmeRestoreRegister 0x36
#define PPSMC_Message_Count 0x37
-
typedef uint16_t PPSMC_Result;
typedef int PPSMC_Msg;
#define FEATURE_CORE_CSTATES_MASK (1 << FEATURE_CORE_CSTATES_BIT)
/* Workload bits */
-#define WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT 0
-#define WORKLOAD_PPLIB_VIDEO_BIT 2
-#define WORKLOAD_PPLIB_VR_BIT 3
-#define WORKLOAD_PPLIB_COMPUTE_BIT 4
-#define WORKLOAD_PPLIB_CUSTOM_BIT 5
-#define WORKLOAD_PPLIB_COUNT 6
+#define WORKLOAD_DEFAULT_BIT 0
+#define WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT 1
+#define WORKLOAD_PPLIB_POWER_SAVING_BIT 2
+#define WORKLOAD_PPLIB_VIDEO_BIT 3
+#define WORKLOAD_PPLIB_VR_BIT 4
+#define WORKLOAD_PPLIB_COMPUTE_BIT 5
+#define WORKLOAD_PPLIB_CUSTOM_BIT 6
+#define WORKLOAD_PPLIB_COUNT 7
typedef struct {
/* MP1_EXT_SCRATCH0 */
--- /dev/null
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#ifndef __SMU_V11_0_H__
+#define __SMU_V11_0_H__
+
+#include "amdgpu_smu.h"
+
+/* MP Apertures */
+#define MP0_Public 0x03800000
+#define MP0_SRAM 0x03900000
+#define MP1_Public 0x03b00000
+#define MP1_SRAM 0x03c00004
+
+/* address block */
+#define smnMP1_FIRMWARE_FLAGS 0x3010024
+#define smnMP0_FW_INTF 0x30101c0
+#define smnMP1_PUB_CTRL 0x3010b14
+
+struct smu_11_0_max_sustainable_clocks {
+ uint32_t display_clock;
+ uint32_t phy_clock;
+ uint32_t pixel_clock;
+ uint32_t uclock;
+ uint32_t dcef_clock;
+ uint32_t soc_clock;
+};
+
+struct smu_11_0_dpm_table {
+ uint32_t min; /* MHz */
+ uint32_t max; /* MHz */
+};
+
+struct smu_11_0_dpm_tables {
+ struct smu_11_0_dpm_table soc_table;
+ struct smu_11_0_dpm_table gfx_table;
+ struct smu_11_0_dpm_table uclk_table;
+ struct smu_11_0_dpm_table eclk_table;
+ struct smu_11_0_dpm_table vclk_table;
+ struct smu_11_0_dpm_table dclk_table;
+ struct smu_11_0_dpm_table dcef_table;
+ struct smu_11_0_dpm_table pixel_table;
+ struct smu_11_0_dpm_table display_table;
+ struct smu_11_0_dpm_table phy_table;
+ struct smu_11_0_dpm_table fclk_table;
+};
+
+struct smu_11_0_dpm_context {
+ struct smu_11_0_dpm_tables dpm_tables;
+ uint32_t workload_policy_mask;
+ uint32_t dcef_min_ds_clk;
+};
+
+enum smu_11_0_power_state {
+ SMU_11_0_POWER_STATE__D0 = 0,
+ SMU_11_0_POWER_STATE__D1,
+ SMU_11_0_POWER_STATE__D3, /* Sleep*/
+ SMU_11_0_POWER_STATE__D4, /* Hibernate*/
+ SMU_11_0_POWER_STATE__D5, /* Power off*/
+};
+
+struct smu_11_0_power_context {
+ uint32_t power_source;
+ uint8_t in_power_limit_boost_mode;
+ enum smu_11_0_power_state power_state;
+};
+
+void smu_v11_0_set_smu_funcs(struct smu_context *smu);
+
+#endif
--- /dev/null
+/*
+ * Copyright 2018 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef SMU_V11_0_PPSMC_H
+#define SMU_V11_0_PPSMC_H
+
+// SMU Response Codes:
+#define PPSMC_Result_OK 0x1
+#define PPSMC_Result_Failed 0xFF
+#define PPSMC_Result_UnknownCmd 0xFE
+#define PPSMC_Result_CmdRejectedPrereq 0xFD
+#define PPSMC_Result_CmdRejectedBusy 0xFC
+
+// Message Definitions:
+// BASIC
+#define PPSMC_MSG_TestMessage 0x1
+#define PPSMC_MSG_GetSmuVersion 0x2
+#define PPSMC_MSG_GetDriverIfVersion 0x3
+#define PPSMC_MSG_SetAllowedFeaturesMaskLow 0x4
+#define PPSMC_MSG_SetAllowedFeaturesMaskHigh 0x5
+#define PPSMC_MSG_EnableAllSmuFeatures 0x6
+#define PPSMC_MSG_DisableAllSmuFeatures 0x7
+#define PPSMC_MSG_EnableSmuFeaturesLow 0x8
+#define PPSMC_MSG_EnableSmuFeaturesHigh 0x9
+#define PPSMC_MSG_DisableSmuFeaturesLow 0xA
+#define PPSMC_MSG_DisableSmuFeaturesHigh 0xB
+#define PPSMC_MSG_GetEnabledSmuFeaturesLow 0xC
+#define PPSMC_MSG_GetEnabledSmuFeaturesHigh 0xD
+#define PPSMC_MSG_SetDriverDramAddrHigh 0xE
+#define PPSMC_MSG_SetDriverDramAddrLow 0xF
+#define PPSMC_MSG_SetToolsDramAddrHigh 0x10
+#define PPSMC_MSG_SetToolsDramAddrLow 0x11
+#define PPSMC_MSG_TransferTableSmu2Dram 0x12
+#define PPSMC_MSG_TransferTableDram2Smu 0x13
+#define PPSMC_MSG_UseDefaultPPTable 0x14
+#define PPSMC_MSG_UseBackupPPTable 0x15
+#define PPSMC_MSG_SetSystemVirtualDramAddrHigh 0x16
+#define PPSMC_MSG_SetSystemVirtualDramAddrLow 0x17
+
+//BACO/BAMACO/BOMACO
+#define PPSMC_MSG_EnterBaco 0x18
+#define PPSMC_MSG_ExitBaco 0x19
+
+//DPM
+#define PPSMC_MSG_SetSoftMinByFreq 0x1A
+#define PPSMC_MSG_SetSoftMaxByFreq 0x1B
+#define PPSMC_MSG_SetHardMinByFreq 0x1C
+#define PPSMC_MSG_SetHardMaxByFreq 0x1D
+#define PPSMC_MSG_GetMinDpmFreq 0x1E
+#define PPSMC_MSG_GetMaxDpmFreq 0x1F
+#define PPSMC_MSG_GetDpmFreqByIndex 0x20
+#define PPSMC_MSG_OverridePcieParameters 0x21
+#define PPSMC_MSG_SetMinDeepSleepDcefclk 0x22
+#define PPSMC_MSG_SetWorkloadMask 0x23
+#define PPSMC_MSG_SetUclkFastSwitch 0x24
+#define PPSMC_MSG_GetAvfsVoltageByDpm 0x25
+#define PPSMC_MSG_SetVideoFps 0x26
+#define PPSMC_MSG_GetDcModeMaxDpmFreq 0x27
+
+//Power Gating
+#define PPSMC_MSG_AllowGfxOff 0x28
+#define PPSMC_MSG_DisallowGfxOff 0x29
+#define PPSMC_MSG_PowerUpVcn 0x2A
+#define PPSMC_MSG_PowerDownVcn 0x2B
+#define PPSMC_MSG_PowerUpJpeg 0x2C
+#define PPSMC_MSG_PowerDownJpeg 0x2D
+//reserve 0x2A to 0x2F for PG harvesting TBD
+
+//I2C Interface
+#define PPSMC_RequestI2cTransaction 0x30
+
+//Resets
+#define PPSMC_MSG_SoftReset 0x31 //FIXME Need confirmation from driver
+#define PPSMC_MSG_PrepareMp1ForUnload 0x32
+#define PPSMC_MSG_PrepareMp1ForReset 0x33
+#define PPSMC_MSG_PrepareMp1ForShutdown 0x34
+
+//ACDC Power Source
+#define PPSMC_MSG_SetPptLimit 0x35
+#define PPSMC_MSG_GetPptLimit 0x36
+#define PPSMC_MSG_ReenableAcDcInterrupt 0x37
+#define PPSMC_MSG_NotifyPowerSource 0x38
+//#define PPSMC_MSG_GfxDeviceDriverReset 0x39 //FIXME mode1 and 2 resets will go directly go PSP
+
+//BTC
+#define PPSMC_MSG_RunBtc 0x3A
+
+//Debug
+#define PPSMC_MSG_DramLogSetDramAddrHigh 0x3B
+#define PPSMC_MSG_DramLogSetDramAddrLow 0x3C
+#define PPSMC_MSG_DramLogSetDramSize 0x3D
+#define PPSMC_MSG_GetDebugData 0x3E
+
+//Others
+#define PPSMC_MSG_ConfigureGfxDidt 0x3F
+#define PPSMC_MSG_NumOfDisplays 0x40
+
+#define PPSMC_MSG_SetMemoryChannelConfig 0x41
+#define PPSMC_MSG_SetGeminiMode 0x42
+#define PPSMC_MSG_SetGeminiApertureHigh 0x43
+#define PPSMC_MSG_SetGeminiApertureLow 0x44
+
+#define PPSMC_Message_Count 0x45
+
+typedef uint32_t PPSMC_Result;
+typedef uint32_t PPSMC_Msg;
+
+#endif
--- /dev/null
+/*
+ * Copyright 2018 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+#ifndef SMU_11_0_PPTABLE_H
+#define SMU_11_0_PPTABLE_H
+
+
+#define SMU_11_0_TABLE_FORMAT_REVISION 12
+
+//// POWERPLAYTABLE::ulPlatformCaps
+#define SMU_11_0_PP_PLATFORM_CAP_POWERPLAY 0x1
+#define SMU_11_0_PP_PLATFORM_CAP_SBIOSPOWERSOURCE 0x2
+#define SMU_11_0_PP_PLATFORM_CAP_HARDWAREDC 0x4
+#define SMU_11_0_PP_PLATFORM_CAP_BACO 0x8
+#define SMU_11_0_PP_PLATFORM_CAP_MACO 0x10
+#define SMU_11_0_PP_PLATFORM_CAP_SHADOWPSTATE 0x20
+
+// SMU_11_0_PP_THERMALCONTROLLER - Thermal Controller Type
+#define SMU_11_0_PP_THERMALCONTROLLER_NONE 0
+
+#define SMU_11_0_PP_OVERDRIVE_VERSION 0x0800
+#define SMU_11_0_PP_POWERSAVINGCLOCK_VERSION 0x0100
+
+enum SMU_11_0_ODFEATURE_ID {
+ SMU_11_0_ODFEATURE_GFXCLK_LIMITS = 1 << 0, //GFXCLK Limit feature
+ SMU_11_0_ODFEATURE_GFXCLK_CURVE = 1 << 1, //GFXCLK Curve feature
+ SMU_11_0_ODFEATURE_UCLK_MAX = 1 << 2, //UCLK Limit feature
+ SMU_11_0_ODFEATURE_POWER_LIMIT = 1 << 3, //Power Limit feature
+ SMU_11_0_ODFEATURE_FAN_ACOUSTIC_LIMIT = 1 << 4, //Fan Acoustic RPM feature
+ SMU_11_0_ODFEATURE_FAN_SPEED_MIN = 1 << 5, //Minimum Fan Speed feature
+ SMU_11_0_ODFEATURE_TEMPERATURE_FAN = 1 << 6, //Fan Target Temperature Limit feature
+ SMU_11_0_ODFEATURE_TEMPERATURE_SYSTEM = 1 << 7, //Operating Temperature Limit feature
+ SMU_11_0_ODFEATURE_MEMORY_TIMING_TUNE = 1 << 8, //AC Timing Tuning feature
+ SMU_11_0_ODFEATURE_FAN_ZERO_RPM_CONTROL = 1 << 9, //Zero RPM feature
+ SMU_11_0_ODFEATURE_AUTO_UV_ENGINE = 1 << 10, //Auto Under Volt GFXCLK feature
+ SMU_11_0_ODFEATURE_AUTO_OC_ENGINE = 1 << 11, //Auto Over Clock GFXCLK feature
+ SMU_11_0_ODFEATURE_AUTO_OC_MEMORY = 1 << 12, //Auto Over Clock MCLK feature
+ SMU_11_0_ODFEATURE_FAN_CURVE = 1 << 13, //VICTOR TODO
+ SMU_11_0_ODFEATURE_COUNT = 14,
+};
+#define SMU_11_0_MAX_ODFEATURE 32 //Maximum Number of OD Features
+
+enum SMU_11_0_ODSETTING_ID {
+ SMU_11_0_ODSETTING_GFXCLKFMAX = 0,
+ SMU_11_0_ODSETTING_GFXCLKFMIN,
+ SMU_11_0_ODSETTING_VDDGFXCURVEFREQ_P1,
+ SMU_11_0_ODSETTING_VDDGFXCURVEVOLTAGE_P1,
+ SMU_11_0_ODSETTING_VDDGFXCURVEFREQ_P2,
+ SMU_11_0_ODSETTING_VDDGFXCURVEVOLTAGE_P2,
+ SMU_11_0_ODSETTING_VDDGFXCURVEFREQ_P3,
+ SMU_11_0_ODSETTING_VDDGFXCURVEVOLTAGE_P3,
+ SMU_11_0_ODSETTING_UCLKFMAX,
+ SMU_11_0_ODSETTING_POWERPERCENTAGE,
+ SMU_11_0_ODSETTING_FANRPMMIN,
+ SMU_11_0_ODSETTING_FANRPMACOUSTICLIMIT,
+ SMU_11_0_ODSETTING_FANTARGETTEMPERATURE,
+ SMU_11_0_ODSETTING_OPERATINGTEMPMAX,
+ SMU_11_0_ODSETTING_ACTIMING,
+ SMU_11_0_ODSETTING_FAN_ZERO_RPM_CONTROL,
+ SMU_11_0_ODSETTING_AUTOUVENGINE,
+ SMU_11_0_ODSETTING_AUTOOCENGINE,
+ SMU_11_0_ODSETTING_AUTOOCMEMORY,
+ SMU_11_0_ODSETTING_COUNT,
+};
+#define SMU_11_0_MAX_ODSETTING 32 //Maximum Number of ODSettings
+
+struct smu_11_0_overdrive_table
+{
+ uint8_t revision; //Revision = SMU_11_0_PP_OVERDRIVE_VERSION
+ uint8_t reserve[3]; //Zero filled field reserved for future use
+ uint32_t feature_count; //Total number of supported features
+ uint32_t setting_count; //Total number of supported settings
+ uint8_t cap[SMU_11_0_MAX_ODFEATURE]; //OD feature support flags
+ uint32_t max[SMU_11_0_MAX_ODSETTING]; //default maximum settings
+ uint32_t min[SMU_11_0_MAX_ODSETTING]; //default minimum settings
+} __attribute__((packed));
+
+enum SMU_11_0_PPCLOCK_ID {
+ SMU_11_0_PPCLOCK_GFXCLK = 0,
+ SMU_11_0_PPCLOCK_VCLK,
+ SMU_11_0_PPCLOCK_DCLK,
+ SMU_11_0_PPCLOCK_ECLK,
+ SMU_11_0_PPCLOCK_SOCCLK,
+ SMU_11_0_PPCLOCK_UCLK,
+ SMU_11_0_PPCLOCK_DCEFCLK,
+ SMU_11_0_PPCLOCK_DISPCLK,
+ SMU_11_0_PPCLOCK_PIXCLK,
+ SMU_11_0_PPCLOCK_PHYCLK,
+ SMU_11_0_PPCLOCK_COUNT,
+};
+#define SMU_11_0_MAX_PPCLOCK 16 //Maximum Number of PP Clocks
+
+struct smu_11_0_power_saving_clock_table
+{
+ uint8_t revision; //Revision = SMU_11_0_PP_POWERSAVINGCLOCK_VERSION
+ uint8_t reserve[3]; //Zero filled field reserved for future use
+ uint32_t count; //power_saving_clock_count = SMU_11_0_PPCLOCK_COUNT
+ uint32_t max[SMU_11_0_MAX_PPCLOCK]; //PowerSavingClock Mode Clock Maximum array In MHz
+ uint32_t min[SMU_11_0_MAX_PPCLOCK]; //PowerSavingClock Mode Clock Minimum array In MHz
+} __attribute__((packed));
+
+struct smu_11_0_powerplay_table
+{
+ struct atom_common_table_header header;
+ uint8_t table_revision;
+ uint32_t table_size; //Driver portion table size. The offset to smc_pptable including header size
+ uint32_t golden_pp_id;
+ uint32_t golden_revision;
+ uint16_t format_id;
+ uint32_t platform_caps; //POWERPLAYABLE::ulPlatformCaps
+
+ uint8_t thermal_controller_type; //one of SMU_11_0_PP_THERMALCONTROLLER
+
+ uint16_t small_power_limit1;
+ uint16_t small_power_limit2;
+ uint16_t boost_power_limit;
+ uint16_t od_turbo_power_limit; //Power limit setting for Turbo mode in Performance UI Tuning.
+ uint16_t od_power_save_power_limit; //Power limit setting for PowerSave/Optimal mode in Performance UI Tuning.
+ uint16_t software_shutdown_temp;
+
+ uint16_t reserve[6]; //Zero filled field reserved for future use
+
+ struct smu_11_0_power_saving_clock_table power_saving_clock;
+ struct smu_11_0_overdrive_table overdrive_table;
+
+ PPTable_t smc_pptable; //PPTable_t in smu11_driver_if.h
+} __attribute__((packed));
+
+#endif
#define PPSMC_MSG_PrepareMp1ForShutdown 0x5A
#define PPSMC_MSG_SetMGpuFanBoostLimitRpm 0x5D
#define PPSMC_MSG_GetAVFSVoltageByDpm 0x5F
-#define PPSMC_Message_Count 0x60
+#define PPSMC_MSG_BacoWorkAroundFlushVDCI 0x60
+#define PPSMC_Message_Count 0x61
typedef uint32_t PPSMC_Result;
typedef uint32_t PPSMC_Msg;
--- /dev/null
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include "pp_debug.h"
+#include <linux/firmware.h>
+#include "amdgpu.h"
+#include "amdgpu_smu.h"
+#include "atomfirmware.h"
+#include "amdgpu_atomfirmware.h"
+#include "smu_v11_0.h"
+#include "smu11_driver_if.h"
+#include "soc15_common.h"
+#include "atom.h"
+#include "vega20_ppt.h"
+#include "pp_thermal.h"
+
+#include "asic_reg/thm/thm_11_0_2_offset.h"
+#include "asic_reg/thm/thm_11_0_2_sh_mask.h"
+#include "asic_reg/mp/mp_9_0_offset.h"
+#include "asic_reg/mp/mp_9_0_sh_mask.h"
+#include "asic_reg/nbio/nbio_7_4_offset.h"
+#include "asic_reg/smuio/smuio_9_0_offset.h"
+#include "asic_reg/smuio/smuio_9_0_sh_mask.h"
+
+MODULE_FIRMWARE("amdgpu/vega20_smc.bin");
+
+#define SMU11_TOOL_SIZE 0x19000
+#define SMU11_THERMAL_MINIMUM_ALERT_TEMP 0
+#define SMU11_THERMAL_MAXIMUM_ALERT_TEMP 255
+
+#define SMU11_TEMPERATURE_UNITS_PER_CENTIGRADES 1000
+#define SMU11_VOLTAGE_SCALE 4
+
+#define SMC_DPM_FEATURE (FEATURE_DPM_PREFETCHER_MASK | \
+ FEATURE_DPM_GFXCLK_MASK | \
+ FEATURE_DPM_UCLK_MASK | \
+ FEATURE_DPM_SOCCLK_MASK | \
+ FEATURE_DPM_UVD_MASK | \
+ FEATURE_DPM_VCE_MASK | \
+ FEATURE_DPM_MP0CLK_MASK | \
+ FEATURE_DPM_LINK_MASK | \
+ FEATURE_DPM_DCEFCLK_MASK)
+
+static int smu_v11_0_send_msg_without_waiting(struct smu_context *smu,
+ uint16_t msg)
+{
+ struct amdgpu_device *adev = smu->adev;
+ WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_66, msg);
+ return 0;
+}
+
+static int smu_v11_0_read_arg(struct smu_context *smu, uint32_t *arg)
+{
+ struct amdgpu_device *adev = smu->adev;
+
+ *arg = RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_82);
+ return 0;
+}
+
+static int smu_v11_0_wait_for_response(struct smu_context *smu)
+{
+ struct amdgpu_device *adev = smu->adev;
+ uint32_t cur_value, i;
+
+ for (i = 0; i < adev->usec_timeout; i++) {
+ cur_value = RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90);
+ if ((cur_value & MP1_C2PMSG_90__CONTENT_MASK) != 0)
+ break;
+ udelay(1);
+ }
+
+ /* timeout means wrong logic */
+ if (i == adev->usec_timeout)
+ return -ETIME;
+
+ return RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90) == 0x1 ? 0 : -EIO;
+}
+
+static int smu_v11_0_send_msg(struct smu_context *smu, uint16_t msg)
+{
+ struct amdgpu_device *adev = smu->adev;
+ int ret = 0, index = 0;
+
+ index = smu_msg_get_index(smu, msg);
+ if (index < 0)
+ return index;
+
+ smu_v11_0_wait_for_response(smu);
+
+ WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90, 0);
+
+ smu_v11_0_send_msg_without_waiting(smu, (uint16_t)index);
+
+ ret = smu_v11_0_wait_for_response(smu);
+
+ if (ret)
+ pr_err("Failed to send message 0x%x, response 0x%x\n", index,
+ ret);
+
+ return ret;
+
+}
+
+static int
+smu_v11_0_send_msg_with_param(struct smu_context *smu, uint16_t msg,
+ uint32_t param)
+{
+
+ struct amdgpu_device *adev = smu->adev;
+ int ret = 0, index = 0;
+
+ index = smu_msg_get_index(smu, msg);
+ if (index < 0)
+ return index;
+
+ ret = smu_v11_0_wait_for_response(smu);
+ if (ret)
+ pr_err("Failed to send message 0x%x, response 0x%x, param 0x%x\n",
+ index, ret, param);
+
+ WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_90, 0);
+
+ WREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_82, param);
+
+ smu_v11_0_send_msg_without_waiting(smu, (uint16_t)index);
+
+ ret = smu_v11_0_wait_for_response(smu);
+ if (ret)
+ pr_err("Failed to send message 0x%x, response 0x%x param 0x%x\n",
+ index, ret, param);
+
+ return ret;
+}
+
+static int smu_v11_0_init_microcode(struct smu_context *smu)
+{
+ struct amdgpu_device *adev = smu->adev;
+ const char *chip_name;
+ char fw_name[30];
+ int err = 0;
+ const struct smc_firmware_header_v1_0 *hdr;
+ const struct common_firmware_header *header;
+ struct amdgpu_firmware_info *ucode = NULL;
+
+ switch (adev->asic_type) {
+ case CHIP_VEGA20:
+ chip_name = "vega20";
+ break;
+ default:
+ BUG();
+ }
+
+ snprintf(fw_name, sizeof(fw_name), "amdgpu/%s_smc.bin", chip_name);
+
+ err = request_firmware(&adev->pm.fw, fw_name, adev->dev);
+ if (err)
+ goto out;
+ err = amdgpu_ucode_validate(adev->pm.fw);
+ if (err)
+ goto out;
+
+ hdr = (const struct smc_firmware_header_v1_0 *) adev->pm.fw->data;
+ amdgpu_ucode_print_smc_hdr(&hdr->header);
+ adev->pm.fw_version = le32_to_cpu(hdr->header.ucode_version);
+
+ if (adev->firmware.load_type == AMDGPU_FW_LOAD_PSP) {
+ ucode = &adev->firmware.ucode[AMDGPU_UCODE_ID_SMC];
+ ucode->ucode_id = AMDGPU_UCODE_ID_SMC;
+ ucode->fw = adev->pm.fw;
+ header = (const struct common_firmware_header *)ucode->fw->data;
+ adev->firmware.fw_size +=
+ ALIGN(le32_to_cpu(header->ucode_size_bytes), PAGE_SIZE);
+ }
+
+out:
+ if (err) {
+ DRM_ERROR("smu_v11_0: Failed to load firmware \"%s\"\n",
+ fw_name);
+ release_firmware(adev->pm.fw);
+ adev->pm.fw = NULL;
+ }
+ return err;
+}
+
+static int smu_v11_0_load_microcode(struct smu_context *smu)
+{
+ return 0;
+}
+
+static int smu_v11_0_check_fw_status(struct smu_context *smu)
+{
+ struct amdgpu_device *adev = smu->adev;
+ uint32_t mp1_fw_flags;
+
+ mp1_fw_flags = RREG32_PCIE(MP1_Public |
+ (smnMP1_FIRMWARE_FLAGS & 0xffffffff));
+
+ if ((mp1_fw_flags & MP1_FIRMWARE_FLAGS__INTERRUPTS_ENABLED_MASK) >>
+ MP1_FIRMWARE_FLAGS__INTERRUPTS_ENABLED__SHIFT)
+ return 0;
+
+ return -EIO;
+}
+
+static int smu_v11_0_check_fw_version(struct smu_context *smu)
+{
+ uint32_t smu_version = 0xff;
+ int ret = 0;
+
+ ret = smu_send_smc_msg(smu, SMU_MSG_GetDriverIfVersion);
+ if (ret)
+ goto err;
+
+ ret = smu_read_smc_arg(smu, &smu_version);
+ if (ret)
+ goto err;
+
+ if (smu_version != smu->smc_if_version)
+ ret = -EINVAL;
+err:
+ return ret;
+}
+
+static int smu_v11_0_read_pptable_from_vbios(struct smu_context *smu)
+{
+ int ret, index;
+ uint16_t size;
+ uint8_t frev, crev;
+ void *table;
+
+ index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1,
+ powerplayinfo);
+
+ ret = smu_get_atom_data_table(smu, index, &size, &frev, &crev,
+ (uint8_t **)&table);
+ if (ret)
+ return ret;
+
+ if (!smu->smu_table.power_play_table)
+ smu->smu_table.power_play_table = table;
+ if (!smu->smu_table.power_play_table_size)
+ smu->smu_table.power_play_table_size = size;
+
+ return 0;
+}
+
+static int smu_v11_0_init_dpm_context(struct smu_context *smu)
+{
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+
+ if (smu_dpm->dpm_context || smu_dpm->dpm_context_size != 0)
+ return -EINVAL;
+
+ return smu_alloc_dpm_context(smu);
+}
+
+static int smu_v11_0_fini_dpm_context(struct smu_context *smu)
+{
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+
+ if (!smu_dpm->dpm_context || smu_dpm->dpm_context_size == 0)
+ return -EINVAL;
+
+ kfree(smu_dpm->dpm_context);
+ kfree(smu_dpm->golden_dpm_context);
+ kfree(smu_dpm->dpm_current_power_state);
+ kfree(smu_dpm->dpm_request_power_state);
+ smu_dpm->dpm_context = NULL;
+ smu_dpm->golden_dpm_context = NULL;
+ smu_dpm->dpm_context_size = 0;
+ smu_dpm->dpm_current_power_state = NULL;
+ smu_dpm->dpm_request_power_state = NULL;
+
+ return 0;
+}
+
+static int smu_v11_0_init_smc_tables(struct smu_context *smu)
+{
+ struct smu_table_context *smu_table = &smu->smu_table;
+ struct smu_table *tables = NULL;
+ int ret = 0;
+
+ if (smu_table->tables || smu_table->table_count != 0)
+ return -EINVAL;
+
+ tables = kcalloc(TABLE_COUNT, sizeof(struct smu_table), GFP_KERNEL);
+ if (!tables)
+ return -ENOMEM;
+
+ smu_table->tables = tables;
+ smu_table->table_count = TABLE_COUNT;
+
+ SMU_TABLE_INIT(tables, TABLE_PPTABLE, sizeof(PPTable_t),
+ PAGE_SIZE, AMDGPU_GEM_DOMAIN_VRAM);
+ SMU_TABLE_INIT(tables, TABLE_WATERMARKS, sizeof(Watermarks_t),
+ PAGE_SIZE, AMDGPU_GEM_DOMAIN_VRAM);
+ SMU_TABLE_INIT(tables, TABLE_SMU_METRICS, sizeof(SmuMetrics_t),
+ PAGE_SIZE, AMDGPU_GEM_DOMAIN_VRAM);
+ SMU_TABLE_INIT(tables, TABLE_OVERDRIVE, sizeof(OverDriveTable_t),
+ PAGE_SIZE, AMDGPU_GEM_DOMAIN_VRAM);
+ SMU_TABLE_INIT(tables, TABLE_PMSTATUSLOG, SMU11_TOOL_SIZE, PAGE_SIZE,
+ AMDGPU_GEM_DOMAIN_VRAM);
+ SMU_TABLE_INIT(tables, TABLE_ACTIVITY_MONITOR_COEFF,
+ sizeof(DpmActivityMonitorCoeffInt_t),
+ PAGE_SIZE,
+ AMDGPU_GEM_DOMAIN_VRAM);
+
+ ret = smu_v11_0_init_dpm_context(smu);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int smu_v11_0_fini_smc_tables(struct smu_context *smu)
+{
+ struct smu_table_context *smu_table = &smu->smu_table;
+ int ret = 0;
+
+ if (!smu_table->tables || smu_table->table_count == 0)
+ return -EINVAL;
+
+ kfree(smu_table->tables);
+ smu_table->tables = NULL;
+ smu_table->table_count = 0;
+
+ ret = smu_v11_0_fini_dpm_context(smu);
+ if (ret)
+ return ret;
+ return 0;
+}
+
+static int smu_v11_0_init_power(struct smu_context *smu)
+{
+ struct smu_power_context *smu_power = &smu->smu_power;
+
+ if (smu_power->power_context || smu_power->power_context_size != 0)
+ return -EINVAL;
+
+ smu_power->power_context = kzalloc(sizeof(struct smu_11_0_dpm_context),
+ GFP_KERNEL);
+ if (!smu_power->power_context)
+ return -ENOMEM;
+ smu_power->power_context_size = sizeof(struct smu_11_0_dpm_context);
+
+ return 0;
+}
+
+static int smu_v11_0_fini_power(struct smu_context *smu)
+{
+ struct smu_power_context *smu_power = &smu->smu_power;
+
+ if (!smu_power->power_context || smu_power->power_context_size == 0)
+ return -EINVAL;
+
+ kfree(smu_power->power_context);
+ smu_power->power_context = NULL;
+ smu_power->power_context_size = 0;
+
+ return 0;
+}
+
+int smu_v11_0_get_vbios_bootup_values(struct smu_context *smu)
+{
+ int ret, index;
+ uint16_t size;
+ uint8_t frev, crev;
+ struct atom_common_table_header *header;
+ struct atom_firmware_info_v3_3 *v_3_3;
+ struct atom_firmware_info_v3_1 *v_3_1;
+
+ index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1,
+ firmwareinfo);
+
+ ret = smu_get_atom_data_table(smu, index, &size, &frev, &crev,
+ (uint8_t **)&header);
+ if (ret)
+ return ret;
+
+ if (header->format_revision != 3) {
+ pr_err("unknown atom_firmware_info version! for smu11\n");
+ return -EINVAL;
+ }
+
+ switch (header->content_revision) {
+ case 0:
+ case 1:
+ case 2:
+ v_3_1 = (struct atom_firmware_info_v3_1 *)header;
+ smu->smu_table.boot_values.revision = v_3_1->firmware_revision;
+ smu->smu_table.boot_values.gfxclk = v_3_1->bootup_sclk_in10khz;
+ smu->smu_table.boot_values.uclk = v_3_1->bootup_mclk_in10khz;
+ smu->smu_table.boot_values.socclk = 0;
+ smu->smu_table.boot_values.dcefclk = 0;
+ smu->smu_table.boot_values.vddc = v_3_1->bootup_vddc_mv;
+ smu->smu_table.boot_values.vddci = v_3_1->bootup_vddci_mv;
+ smu->smu_table.boot_values.mvddc = v_3_1->bootup_mvddc_mv;
+ smu->smu_table.boot_values.vdd_gfx = v_3_1->bootup_vddgfx_mv;
+ smu->smu_table.boot_values.cooling_id = v_3_1->coolingsolution_id;
+ smu->smu_table.boot_values.pp_table_id = 0;
+ break;
+ case 3:
+ default:
+ v_3_3 = (struct atom_firmware_info_v3_3 *)header;
+ smu->smu_table.boot_values.revision = v_3_3->firmware_revision;
+ smu->smu_table.boot_values.gfxclk = v_3_3->bootup_sclk_in10khz;
+ smu->smu_table.boot_values.uclk = v_3_3->bootup_mclk_in10khz;
+ smu->smu_table.boot_values.socclk = 0;
+ smu->smu_table.boot_values.dcefclk = 0;
+ smu->smu_table.boot_values.vddc = v_3_3->bootup_vddc_mv;
+ smu->smu_table.boot_values.vddci = v_3_3->bootup_vddci_mv;
+ smu->smu_table.boot_values.mvddc = v_3_3->bootup_mvddc_mv;
+ smu->smu_table.boot_values.vdd_gfx = v_3_3->bootup_vddgfx_mv;
+ smu->smu_table.boot_values.cooling_id = v_3_3->coolingsolution_id;
+ smu->smu_table.boot_values.pp_table_id = v_3_3->pplib_pptable_id;
+ }
+
+ return 0;
+}
+
+static int smu_v11_0_get_clk_info_from_vbios(struct smu_context *smu)
+{
+ int ret, index;
+ struct amdgpu_device *adev = smu->adev;
+ struct atom_get_smu_clock_info_parameters_v3_1 input = {0};
+ struct atom_get_smu_clock_info_output_parameters_v3_1 *output;
+
+ input.clk_id = SMU11_SYSPLL0_SOCCLK_ID;
+ input.command = GET_SMU_CLOCK_INFO_V3_1_GET_CLOCK_FREQ;
+ index = get_index_into_master_table(atom_master_list_of_command_functions_v2_1,
+ getsmuclockinfo);
+
+ ret = amdgpu_atom_execute_table(adev->mode_info.atom_context, index,
+ (uint32_t *)&input);
+ if (ret)
+ return -EINVAL;
+
+ output = (struct atom_get_smu_clock_info_output_parameters_v3_1 *)&input;
+ smu->smu_table.boot_values.socclk = le32_to_cpu(output->atom_smu_outputclkfreq.smu_clock_freq_hz) / 10000;
+
+ memset(&input, 0, sizeof(input));
+ input.clk_id = SMU11_SYSPLL0_DCEFCLK_ID;
+ input.command = GET_SMU_CLOCK_INFO_V3_1_GET_CLOCK_FREQ;
+ index = get_index_into_master_table(atom_master_list_of_command_functions_v2_1,
+ getsmuclockinfo);
+
+ ret = amdgpu_atom_execute_table(adev->mode_info.atom_context, index,
+ (uint32_t *)&input);
+ if (ret)
+ return -EINVAL;
+
+ output = (struct atom_get_smu_clock_info_output_parameters_v3_1 *)&input;
+ smu->smu_table.boot_values.dcefclk = le32_to_cpu(output->atom_smu_outputclkfreq.smu_clock_freq_hz) / 10000;
+
+ memset(&input, 0, sizeof(input));
+ input.clk_id = SMU11_SYSPLL0_ECLK_ID;
+ input.command = GET_SMU_CLOCK_INFO_V3_1_GET_CLOCK_FREQ;
+ index = get_index_into_master_table(atom_master_list_of_command_functions_v2_1,
+ getsmuclockinfo);
+
+ ret = amdgpu_atom_execute_table(adev->mode_info.atom_context, index,
+ (uint32_t *)&input);
+ if (ret)
+ return -EINVAL;
+
+ output = (struct atom_get_smu_clock_info_output_parameters_v3_1 *)&input;
+ smu->smu_table.boot_values.eclk = le32_to_cpu(output->atom_smu_outputclkfreq.smu_clock_freq_hz) / 10000;
+
+ memset(&input, 0, sizeof(input));
+ input.clk_id = SMU11_SYSPLL0_VCLK_ID;
+ input.command = GET_SMU_CLOCK_INFO_V3_1_GET_CLOCK_FREQ;
+ index = get_index_into_master_table(atom_master_list_of_command_functions_v2_1,
+ getsmuclockinfo);
+
+ ret = amdgpu_atom_execute_table(adev->mode_info.atom_context, index,
+ (uint32_t *)&input);
+ if (ret)
+ return -EINVAL;
+
+ output = (struct atom_get_smu_clock_info_output_parameters_v3_1 *)&input;
+ smu->smu_table.boot_values.vclk = le32_to_cpu(output->atom_smu_outputclkfreq.smu_clock_freq_hz) / 10000;
+
+ memset(&input, 0, sizeof(input));
+ input.clk_id = SMU11_SYSPLL0_DCLK_ID;
+ input.command = GET_SMU_CLOCK_INFO_V3_1_GET_CLOCK_FREQ;
+ index = get_index_into_master_table(atom_master_list_of_command_functions_v2_1,
+ getsmuclockinfo);
+
+ ret = amdgpu_atom_execute_table(adev->mode_info.atom_context, index,
+ (uint32_t *)&input);
+ if (ret)
+ return -EINVAL;
+
+ output = (struct atom_get_smu_clock_info_output_parameters_v3_1 *)&input;
+ smu->smu_table.boot_values.dclk = le32_to_cpu(output->atom_smu_outputclkfreq.smu_clock_freq_hz) / 10000;
+
+ return 0;
+}
+
+static int smu_v11_0_notify_memory_pool_location(struct smu_context *smu)
+{
+ struct smu_table_context *smu_table = &smu->smu_table;
+ struct smu_table *memory_pool = &smu_table->memory_pool;
+ int ret = 0;
+ uint64_t address;
+ uint32_t address_low, address_high;
+
+ if (memory_pool->size == 0 || memory_pool->cpu_addr == NULL)
+ return ret;
+
+ address = (uintptr_t)memory_pool->cpu_addr;
+ address_high = (uint32_t)upper_32_bits(address);
+ address_low = (uint32_t)lower_32_bits(address);
+
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetSystemVirtualDramAddrHigh,
+ address_high);
+ if (ret)
+ return ret;
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetSystemVirtualDramAddrLow,
+ address_low);
+ if (ret)
+ return ret;
+
+ address = memory_pool->mc_address;
+ address_high = (uint32_t)upper_32_bits(address);
+ address_low = (uint32_t)lower_32_bits(address);
+
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_DramLogSetDramAddrHigh,
+ address_high);
+ if (ret)
+ return ret;
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_DramLogSetDramAddrLow,
+ address_low);
+ if (ret)
+ return ret;
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_DramLogSetDramSize,
+ (uint32_t)memory_pool->size);
+ if (ret)
+ return ret;
+
+ return ret;
+}
+
+static int smu_v11_0_check_pptable(struct smu_context *smu)
+{
+ int ret;
+
+ ret = smu_check_powerplay_table(smu);
+ return ret;
+}
+
+static int smu_v11_0_parse_pptable(struct smu_context *smu)
+{
+ int ret;
+
+ struct smu_table_context *table_context = &smu->smu_table;
+
+ if (table_context->driver_pptable)
+ return -EINVAL;
+
+ table_context->driver_pptable = kzalloc(sizeof(PPTable_t), GFP_KERNEL);
+
+ if (!table_context->driver_pptable)
+ return -ENOMEM;
+
+ ret = smu_store_powerplay_table(smu);
+ if (ret)
+ return -EINVAL;
+
+ ret = smu_append_powerplay_table(smu);
+
+ return ret;
+}
+
+static int smu_v11_0_populate_smc_pptable(struct smu_context *smu)
+{
+ int ret;
+
+ ret = smu_set_default_dpm_table(smu);
+
+ return ret;
+}
+
+static int smu_v11_0_write_pptable(struct smu_context *smu)
+{
+ struct smu_table_context *table_context = &smu->smu_table;
+ int ret = 0;
+
+ ret = smu_update_table(smu, TABLE_PPTABLE, table_context->driver_pptable, true);
+
+ return ret;
+}
+
+static int smu_v11_0_write_watermarks_table(struct smu_context *smu)
+{
+ return smu_update_table(smu, TABLE_WATERMARKS,
+ smu->smu_table.tables[TABLE_WATERMARKS].cpu_addr, true);
+}
+
+static int smu_v11_0_set_deep_sleep_dcefclk(struct smu_context *smu, uint32_t clk)
+{
+ int ret;
+
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetMinDeepSleepDcefclk, clk);
+ if (ret)
+ pr_err("SMU11 attempt to set divider for DCEFCLK Failed!");
+
+ return ret;
+}
+
+static int smu_v11_0_set_min_dcef_deep_sleep(struct smu_context *smu)
+{
+ struct smu_table_context *table_context = &smu->smu_table;
+
+ if (!table_context)
+ return -EINVAL;
+
+ return smu_set_deep_sleep_dcefclk(smu,
+ table_context->boot_values.dcefclk / 100);
+}
+
+static int smu_v11_0_set_tool_table_location(struct smu_context *smu)
+{
+ int ret = 0;
+ struct smu_table *tool_table = &smu->smu_table.tables[TABLE_PMSTATUSLOG];
+
+ if (tool_table->mc_address) {
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetToolsDramAddrHigh,
+ upper_32_bits(tool_table->mc_address));
+ if (!ret)
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetToolsDramAddrLow,
+ lower_32_bits(tool_table->mc_address));
+ }
+
+ return ret;
+}
+
+static int smu_v11_0_init_display(struct smu_context *smu)
+{
+ int ret = 0;
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_NumOfDisplays, 0);
+ return ret;
+}
+
+static int smu_v11_0_update_feature_enable_state(struct smu_context *smu, uint32_t feature_id, bool enabled)
+{
+ uint32_t feature_low = 0, feature_high = 0;
+ int ret = 0;
+
+ if (feature_id >= 0 && feature_id < 31)
+ feature_low = (1 << feature_id);
+ else if (feature_id > 31 && feature_id < 63)
+ feature_high = (1 << feature_id);
+ else
+ return -EINVAL;
+
+ if (enabled) {
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_EnableSmuFeaturesLow,
+ feature_low);
+ if (ret)
+ return ret;
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_EnableSmuFeaturesHigh,
+ feature_high);
+ if (ret)
+ return ret;
+
+ } else {
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_DisableSmuFeaturesLow,
+ feature_low);
+ if (ret)
+ return ret;
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_DisableSmuFeaturesHigh,
+ feature_high);
+ if (ret)
+ return ret;
+
+ }
+
+ return ret;
+}
+
+static int smu_v11_0_set_allowed_mask(struct smu_context *smu)
+{
+ struct smu_feature *feature = &smu->smu_feature;
+ int ret = 0;
+ uint32_t feature_mask[2];
+
+ mutex_lock(&feature->mutex);
+ if (bitmap_empty(feature->allowed, SMU_FEATURE_MAX) || feature->feature_num < 64)
+ goto failed;
+
+ bitmap_copy((unsigned long *)feature_mask, feature->allowed, 64);
+
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_SetAllowedFeaturesMaskHigh,
+ feature_mask[1]);
+ if (ret)
+ goto failed;
+
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_SetAllowedFeaturesMaskLow,
+ feature_mask[0]);
+ if (ret)
+ goto failed;
+
+failed:
+ mutex_unlock(&feature->mutex);
+ return ret;
+}
+
+static int smu_v11_0_get_enabled_mask(struct smu_context *smu,
+ uint32_t *feature_mask, uint32_t num)
+{
+ uint32_t feature_mask_high = 0, feature_mask_low = 0;
+ int ret = 0;
+
+ if (!feature_mask || num < 2)
+ return -EINVAL;
+
+ ret = smu_send_smc_msg(smu, SMU_MSG_GetEnabledSmuFeaturesHigh);
+ if (ret)
+ return ret;
+ ret = smu_read_smc_arg(smu, &feature_mask_high);
+ if (ret)
+ return ret;
+
+ ret = smu_send_smc_msg(smu, SMU_MSG_GetEnabledSmuFeaturesLow);
+ if (ret)
+ return ret;
+ ret = smu_read_smc_arg(smu, &feature_mask_low);
+ if (ret)
+ return ret;
+
+ feature_mask[0] = feature_mask_low;
+ feature_mask[1] = feature_mask_high;
+
+ return ret;
+}
+
+static bool smu_v11_0_is_dpm_running(struct smu_context *smu)
+{
+ int ret = 0;
+ uint32_t feature_mask[2];
+ unsigned long feature_enabled;
+ ret = smu_v11_0_get_enabled_mask(smu, feature_mask, 2);
+ feature_enabled = (unsigned long)((uint64_t)feature_mask[0] |
+ ((uint64_t)feature_mask[1] << 32));
+ return !!(feature_enabled & SMC_DPM_FEATURE);
+}
+
+static int smu_v11_0_system_features_control(struct smu_context *smu,
+ bool en)
+{
+ struct smu_feature *feature = &smu->smu_feature;
+ uint32_t feature_mask[2];
+ int ret = 0;
+
+ ret = smu_send_smc_msg(smu, (en ? SMU_MSG_EnableAllSmuFeatures :
+ SMU_MSG_DisableAllSmuFeatures));
+ if (ret)
+ return ret;
+ ret = smu_feature_get_enabled_mask(smu, feature_mask, 2);
+ if (ret)
+ return ret;
+
+ bitmap_copy(feature->enabled, (unsigned long *)&feature_mask,
+ feature->feature_num);
+ bitmap_copy(feature->supported, (unsigned long *)&feature_mask,
+ feature->feature_num);
+
+ return ret;
+}
+
+static int smu_v11_0_notify_display_change(struct smu_context *smu)
+{
+ int ret = 0;
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_UCLK_BIT))
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_SetUclkFastSwitch, 1);
+
+ return ret;
+}
+
+static int
+smu_v11_0_get_max_sustainable_clock(struct smu_context *smu, uint32_t *clock,
+ PPCLK_e clock_select)
+{
+ int ret = 0;
+
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_GetDcModeMaxDpmFreq,
+ clock_select << 16);
+ if (ret) {
+ pr_err("[GetMaxSustainableClock] Failed to get max DC clock from SMC!");
+ return ret;
+ }
+
+ ret = smu_read_smc_arg(smu, clock);
+ if (ret)
+ return ret;
+
+ if (*clock != 0)
+ return 0;
+
+ /* if DC limit is zero, return AC limit */
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_GetMaxDpmFreq,
+ clock_select << 16);
+ if (ret) {
+ pr_err("[GetMaxSustainableClock] failed to get max AC clock from SMC!");
+ return ret;
+ }
+
+ ret = smu_read_smc_arg(smu, clock);
+
+ return ret;
+}
+
+static int smu_v11_0_init_max_sustainable_clocks(struct smu_context *smu)
+{
+ struct smu_11_0_max_sustainable_clocks *max_sustainable_clocks;
+ int ret = 0;
+
+ max_sustainable_clocks = kzalloc(sizeof(struct smu_11_0_max_sustainable_clocks),
+ GFP_KERNEL);
+ smu->smu_table.max_sustainable_clocks = (void *)max_sustainable_clocks;
+
+ max_sustainable_clocks->uclock = smu->smu_table.boot_values.uclk / 100;
+ max_sustainable_clocks->soc_clock = smu->smu_table.boot_values.socclk / 100;
+ max_sustainable_clocks->dcef_clock = smu->smu_table.boot_values.dcefclk / 100;
+ max_sustainable_clocks->display_clock = 0xFFFFFFFF;
+ max_sustainable_clocks->phy_clock = 0xFFFFFFFF;
+ max_sustainable_clocks->pixel_clock = 0xFFFFFFFF;
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_UCLK_BIT)) {
+ ret = smu_v11_0_get_max_sustainable_clock(smu,
+ &(max_sustainable_clocks->uclock),
+ PPCLK_UCLK);
+ if (ret) {
+ pr_err("[%s] failed to get max UCLK from SMC!",
+ __func__);
+ return ret;
+ }
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_SOCCLK_BIT)) {
+ ret = smu_v11_0_get_max_sustainable_clock(smu,
+ &(max_sustainable_clocks->soc_clock),
+ PPCLK_SOCCLK);
+ if (ret) {
+ pr_err("[%s] failed to get max SOCCLK from SMC!",
+ __func__);
+ return ret;
+ }
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_DCEFCLK_BIT)) {
+ ret = smu_v11_0_get_max_sustainable_clock(smu,
+ &(max_sustainable_clocks->dcef_clock),
+ PPCLK_DCEFCLK);
+ if (ret) {
+ pr_err("[%s] failed to get max DCEFCLK from SMC!",
+ __func__);
+ return ret;
+ }
+
+ ret = smu_v11_0_get_max_sustainable_clock(smu,
+ &(max_sustainable_clocks->display_clock),
+ PPCLK_DISPCLK);
+ if (ret) {
+ pr_err("[%s] failed to get max DISPCLK from SMC!",
+ __func__);
+ return ret;
+ }
+ ret = smu_v11_0_get_max_sustainable_clock(smu,
+ &(max_sustainable_clocks->phy_clock),
+ PPCLK_PHYCLK);
+ if (ret) {
+ pr_err("[%s] failed to get max PHYCLK from SMC!",
+ __func__);
+ return ret;
+ }
+ ret = smu_v11_0_get_max_sustainable_clock(smu,
+ &(max_sustainable_clocks->pixel_clock),
+ PPCLK_PIXCLK);
+ if (ret) {
+ pr_err("[%s] failed to get max PIXCLK from SMC!",
+ __func__);
+ return ret;
+ }
+ }
+
+ if (max_sustainable_clocks->soc_clock < max_sustainable_clocks->uclock)
+ max_sustainable_clocks->uclock = max_sustainable_clocks->soc_clock;
+
+ return 0;
+}
+
+static int smu_v11_0_get_power_limit(struct smu_context *smu,
+ uint32_t *limit,
+ bool get_default)
+{
+ int ret = 0;
+
+ if (get_default) {
+ mutex_lock(&smu->mutex);
+ *limit = smu->default_power_limit;
+ if (smu->od_enabled) {
+ *limit *= (100 + smu->smu_table.TDPODLimit);
+ *limit /= 100;
+ }
+ mutex_unlock(&smu->mutex);
+ } else {
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_GetPptLimit,
+ POWER_SOURCE_AC << 16);
+ if (ret) {
+ pr_err("[%s] get PPT limit failed!", __func__);
+ return ret;
+ }
+ smu_read_smc_arg(smu, limit);
+ smu->power_limit = *limit;
+ }
+
+ return ret;
+}
+
+static int smu_v11_0_set_power_limit(struct smu_context *smu, uint32_t n)
+{
+ uint32_t max_power_limit;
+ int ret = 0;
+
+ if (n == 0)
+ n = smu->default_power_limit;
+
+ max_power_limit = smu->default_power_limit;
+
+ if (smu->od_enabled) {
+ max_power_limit *= (100 + smu->smu_table.TDPODLimit);
+ max_power_limit /= 100;
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_PPT_BIT))
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_SetPptLimit, n);
+ if (ret) {
+ pr_err("[%s] Set power limit Failed!", __func__);
+ return ret;
+ }
+
+ return ret;
+}
+
+static int smu_v11_0_get_current_clk_freq(struct smu_context *smu, uint32_t clk_id, uint32_t *value)
+{
+ int ret = 0;
+ uint32_t freq;
+
+ if (clk_id >= PPCLK_COUNT || !value)
+ return -EINVAL;
+
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_GetDpmClockFreq, (clk_id << 16));
+ if (ret)
+ return ret;
+
+ ret = smu_read_smc_arg(smu, &freq);
+ if (ret)
+ return ret;
+
+ freq *= 100;
+ *value = freq;
+
+ return ret;
+}
+
+static int smu_v11_0_get_thermal_range(struct smu_context *smu,
+ struct PP_TemperatureRange *range)
+{
+ memcpy(range, &SMU7ThermalWithDelayPolicy[0], sizeof(struct PP_TemperatureRange));
+
+ range->max = smu->smu_table.software_shutdown_temp *
+ PP_TEMPERATURE_UNITS_PER_CENTIGRADES;
+
+ return 0;
+}
+
+static int smu_v11_0_set_thermal_range(struct smu_context *smu,
+ struct PP_TemperatureRange *range)
+{
+ struct amdgpu_device *adev = smu->adev;
+ int low = SMU11_THERMAL_MINIMUM_ALERT_TEMP *
+ PP_TEMPERATURE_UNITS_PER_CENTIGRADES;
+ int high = SMU11_THERMAL_MAXIMUM_ALERT_TEMP *
+ PP_TEMPERATURE_UNITS_PER_CENTIGRADES;
+ uint32_t val;
+
+ if (low < range->min)
+ low = range->min;
+ if (high > range->max)
+ high = range->max;
+
+ if (low > high)
+ return -EINVAL;
+
+ val = RREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_CTRL);
+ val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, MAX_IH_CREDIT, 5);
+ val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, THERM_IH_HW_ENA, 1);
+ val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, DIG_THERM_INTH, (high / PP_TEMPERATURE_UNITS_PER_CENTIGRADES));
+ val = REG_SET_FIELD(val, THM_THERMAL_INT_CTRL, DIG_THERM_INTL, (low / PP_TEMPERATURE_UNITS_PER_CENTIGRADES));
+ val = val & (~THM_THERMAL_INT_CTRL__THERM_TRIGGER_MASK_MASK);
+
+ WREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_CTRL, val);
+
+ return 0;
+}
+
+static int smu_v11_0_enable_thermal_alert(struct smu_context *smu)
+{
+ struct amdgpu_device *adev = smu->adev;
+ uint32_t val = 0;
+
+ val |= (1 << THM_THERMAL_INT_ENA__THERM_INTH_CLR__SHIFT);
+ val |= (1 << THM_THERMAL_INT_ENA__THERM_INTL_CLR__SHIFT);
+ val |= (1 << THM_THERMAL_INT_ENA__THERM_TRIGGER_CLR__SHIFT);
+
+ WREG32_SOC15(THM, 0, mmTHM_THERMAL_INT_ENA, val);
+
+ return 0;
+}
+
+static int smu_v11_0_set_thermal_fan_table(struct smu_context *smu)
+{
+ int ret;
+ struct smu_table_context *table_context = &smu->smu_table;
+ PPTable_t *pptable = table_context->driver_pptable;
+
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_SetFanTemperatureTarget,
+ (uint32_t)pptable->FanTargetTemperature);
+
+ return ret;
+}
+
+static int smu_v11_0_start_thermal_control(struct smu_context *smu)
+{
+ int ret = 0;
+ struct PP_TemperatureRange range;
+ struct amdgpu_device *adev = smu->adev;
+
+ smu_v11_0_get_thermal_range(smu, &range);
+
+ if (smu->smu_table.thermal_controller_type) {
+ ret = smu_v11_0_set_thermal_range(smu, &range);
+ if (ret)
+ return ret;
+
+ ret = smu_v11_0_enable_thermal_alert(smu);
+ if (ret)
+ return ret;
+ ret = smu_v11_0_set_thermal_fan_table(smu);
+ if (ret)
+ return ret;
+ }
+
+ adev->pm.dpm.thermal.min_temp = range.min;
+ adev->pm.dpm.thermal.max_temp = range.max;
+
+ return ret;
+}
+
+static int smu_v11_0_get_current_activity_percent(struct smu_context *smu,
+ uint32_t *value)
+{
+ int ret = 0;
+ SmuMetrics_t metrics;
+
+ if (!value)
+ return -EINVAL;
+
+ ret = smu_update_table(smu, TABLE_SMU_METRICS, (void *)&metrics, false);
+ if (ret)
+ return ret;
+
+ *value = metrics.AverageGfxActivity;
+
+ return 0;
+}
+
+static int smu_v11_0_thermal_get_temperature(struct smu_context *smu, uint32_t *value)
+{
+ struct amdgpu_device *adev = smu->adev;
+ uint32_t temp = 0;
+
+ if (!value)
+ return -EINVAL;
+
+ temp = RREG32_SOC15(THM, 0, mmCG_MULT_THERMAL_STATUS);
+ temp = (temp & CG_MULT_THERMAL_STATUS__CTF_TEMP_MASK) >>
+ CG_MULT_THERMAL_STATUS__CTF_TEMP__SHIFT;
+
+ temp = temp & 0x1ff;
+ temp *= SMU11_TEMPERATURE_UNITS_PER_CENTIGRADES;
+
+ *value = temp;
+
+ return 0;
+}
+
+static int smu_v11_0_get_gpu_power(struct smu_context *smu, uint32_t *value)
+{
+ int ret = 0;
+ SmuMetrics_t metrics;
+
+ if (!value)
+ return -EINVAL;
+
+ ret = smu_update_table(smu, TABLE_SMU_METRICS, (void *)&metrics, false);
+ if (ret)
+ return ret;
+
+ *value = metrics.CurrSocketPower << 8;
+
+ return 0;
+}
+
+static uint16_t convert_to_vddc(uint8_t vid)
+{
+ return (uint16_t) ((6200 - (vid * 25)) / SMU11_VOLTAGE_SCALE);
+}
+
+static int smu_v11_0_get_gfx_vdd(struct smu_context *smu, uint32_t *value)
+{
+ struct amdgpu_device *adev = smu->adev;
+ uint32_t vdd = 0, val_vid = 0;
+
+ if (!value)
+ return -EINVAL;
+ val_vid = (RREG32_SOC15(SMUIO, 0, mmSMUSVI0_TEL_PLANE0) &
+ SMUSVI0_TEL_PLANE0__SVI0_PLANE0_VDDCOR_MASK) >>
+ SMUSVI0_TEL_PLANE0__SVI0_PLANE0_VDDCOR__SHIFT;
+
+ vdd = (uint32_t)convert_to_vddc((uint8_t)val_vid);
+
+ *value = vdd;
+
+ return 0;
+
+}
+
+static int smu_v11_0_read_sensor(struct smu_context *smu,
+ enum amd_pp_sensors sensor,
+ void *data, uint32_t *size)
+{
+ struct smu_table_context *table_context = &smu->smu_table;
+ PPTable_t *pptable = table_context->driver_pptable;
+ int ret = 0;
+ switch (sensor) {
+ case AMDGPU_PP_SENSOR_GPU_LOAD:
+ ret = smu_v11_0_get_current_activity_percent(smu,
+ (uint32_t *)data);
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_GFX_MCLK:
+ ret = smu_get_current_clk_freq(smu, PPCLK_UCLK, (uint32_t *)data);
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_GFX_SCLK:
+ ret = smu_get_current_clk_freq(smu, PPCLK_GFXCLK, (uint32_t *)data);
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_GPU_TEMP:
+ ret = smu_v11_0_thermal_get_temperature(smu, (uint32_t *)data);
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_GPU_POWER:
+ ret = smu_v11_0_get_gpu_power(smu, (uint32_t *)data);
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_VDDGFX:
+ ret = smu_v11_0_get_gfx_vdd(smu, (uint32_t *)data);
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_UVD_POWER:
+ *(uint32_t *)data = smu_feature_is_enabled(smu, FEATURE_DPM_UVD_BIT) ? 1 : 0;
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_VCE_POWER:
+ *(uint32_t *)data = smu_feature_is_enabled(smu, FEATURE_DPM_VCE_BIT) ? 1 : 0;
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_MIN_FAN_RPM:
+ *(uint32_t *)data = 0;
+ *size = 4;
+ break;
+ case AMDGPU_PP_SENSOR_MAX_FAN_RPM:
+ *(uint32_t *)data = pptable->FanMaximumRpm;
+ *size = 4;
+ break;
+ default:
+ ret = smu_common_read_sensor(smu, sensor, data, size);
+ break;
+ }
+
+ if (ret)
+ *size = 0;
+
+ return ret;
+}
+
+static int
+smu_v11_0_display_clock_voltage_request(struct smu_context *smu,
+ struct pp_display_clock_request
+ *clock_req)
+{
+ enum amd_pp_clock_type clk_type = clock_req->clock_type;
+ int ret = 0;
+ PPCLK_e clk_select = 0;
+ uint32_t clk_freq = clock_req->clock_freq_in_khz / 1000;
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_DCEFCLK_BIT)) {
+ switch (clk_type) {
+ case amd_pp_dcef_clock:
+ clk_select = PPCLK_DCEFCLK;
+ break;
+ case amd_pp_disp_clock:
+ clk_select = PPCLK_DISPCLK;
+ break;
+ case amd_pp_pixel_clock:
+ clk_select = PPCLK_PIXCLK;
+ break;
+ case amd_pp_phy_clock:
+ clk_select = PPCLK_PHYCLK;
+ break;
+ default:
+ pr_info("[%s] Invalid Clock Type!", __func__);
+ ret = -EINVAL;
+ break;
+ }
+
+ if (ret)
+ goto failed;
+
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_SetHardMinByFreq,
+ (clk_select << 16) | clk_freq);
+ }
+
+failed:
+ return ret;
+}
+
+static int smu_v11_0_set_watermarks_table(struct smu_context *smu,
+ Watermarks_t *table, struct
+ dm_pp_wm_sets_with_clock_ranges_soc15
+ *clock_ranges)
+{
+ int i;
+
+ if (!table || !clock_ranges)
+ return -EINVAL;
+
+ if (clock_ranges->num_wm_dmif_sets > 4 ||
+ clock_ranges->num_wm_mcif_sets > 4)
+ return -EINVAL;
+
+ for (i = 0; i < clock_ranges->num_wm_dmif_sets; i++) {
+ table->WatermarkRow[1][i].MinClock =
+ cpu_to_le16((uint16_t)
+ (clock_ranges->wm_dmif_clocks_ranges[i].wm_min_dcfclk_clk_in_khz /
+ 1000));
+ table->WatermarkRow[1][i].MaxClock =
+ cpu_to_le16((uint16_t)
+ (clock_ranges->wm_dmif_clocks_ranges[i].wm_max_dcfclk_clk_in_khz /
+ 1000));
+ table->WatermarkRow[1][i].MinUclk =
+ cpu_to_le16((uint16_t)
+ (clock_ranges->wm_dmif_clocks_ranges[i].wm_min_mem_clk_in_khz /
+ 1000));
+ table->WatermarkRow[1][i].MaxUclk =
+ cpu_to_le16((uint16_t)
+ (clock_ranges->wm_dmif_clocks_ranges[i].wm_max_mem_clk_in_khz /
+ 1000));
+ table->WatermarkRow[1][i].WmSetting = (uint8_t)
+ clock_ranges->wm_dmif_clocks_ranges[i].wm_set_id;
+ }
+
+ for (i = 0; i < clock_ranges->num_wm_mcif_sets; i++) {
+ table->WatermarkRow[0][i].MinClock =
+ cpu_to_le16((uint16_t)
+ (clock_ranges->wm_mcif_clocks_ranges[i].wm_min_socclk_clk_in_khz /
+ 1000));
+ table->WatermarkRow[0][i].MaxClock =
+ cpu_to_le16((uint16_t)
+ (clock_ranges->wm_mcif_clocks_ranges[i].wm_max_socclk_clk_in_khz /
+ 1000));
+ table->WatermarkRow[0][i].MinUclk =
+ cpu_to_le16((uint16_t)
+ (clock_ranges->wm_mcif_clocks_ranges[i].wm_min_mem_clk_in_khz /
+ 1000));
+ table->WatermarkRow[0][i].MaxUclk =
+ cpu_to_le16((uint16_t)
+ (clock_ranges->wm_mcif_clocks_ranges[i].wm_max_mem_clk_in_khz /
+ 1000));
+ table->WatermarkRow[0][i].WmSetting = (uint8_t)
+ clock_ranges->wm_mcif_clocks_ranges[i].wm_set_id;
+ }
+
+ return 0;
+}
+
+static int
+smu_v11_0_set_watermarks_for_clock_ranges(struct smu_context *smu, struct
+ dm_pp_wm_sets_with_clock_ranges_soc15
+ *clock_ranges)
+{
+ int ret = 0;
+ struct smu_table *watermarks = &smu->smu_table.tables[TABLE_WATERMARKS];
+ Watermarks_t *table = watermarks->cpu_addr;
+
+ if (!smu->disable_watermark &&
+ smu_feature_is_enabled(smu, FEATURE_DPM_DCEFCLK_BIT) &&
+ smu_feature_is_enabled(smu, FEATURE_DPM_SOCCLK_BIT)) {
+ smu_v11_0_set_watermarks_table(smu, table, clock_ranges);
+ smu->watermarks_bitmap |= WATERMARKS_EXIST;
+ smu->watermarks_bitmap &= ~WATERMARKS_LOADED;
+ }
+
+ return ret;
+}
+
+static int smu_v11_0_get_clock_ranges(struct smu_context *smu,
+ uint32_t *clock,
+ PPCLK_e clock_select,
+ bool max)
+{
+ int ret;
+ *clock = 0;
+ if (max) {
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_GetMaxDpmFreq,
+ (clock_select << 16));
+ if (ret) {
+ pr_err("[GetClockRanges] Failed to get max clock from SMC!\n");
+ return ret;
+ }
+ smu_read_smc_arg(smu, clock);
+ } else {
+ ret = smu_send_smc_msg_with_param(smu, SMU_MSG_GetMinDpmFreq,
+ (clock_select << 16));
+ if (ret) {
+ pr_err("[GetClockRanges] Failed to get min clock from SMC!\n");
+ return ret;
+ }
+ smu_read_smc_arg(smu, clock);
+ }
+
+ return 0;
+}
+
+static uint32_t smu_v11_0_dpm_get_sclk(struct smu_context *smu, bool low)
+{
+ uint32_t gfx_clk;
+ int ret;
+
+ if (!smu_feature_is_enabled(smu, FEATURE_DPM_GFXCLK_BIT)) {
+ pr_err("[GetSclks]: gfxclk dpm not enabled!\n");
+ return -EPERM;
+ }
+
+ if (low) {
+ ret = smu_v11_0_get_clock_ranges(smu, &gfx_clk, PPCLK_GFXCLK, false);
+ if (ret) {
+ pr_err("[GetSclks]: fail to get min PPCLK_GFXCLK\n");
+ return ret;
+ }
+ } else {
+ ret = smu_v11_0_get_clock_ranges(smu, &gfx_clk, PPCLK_GFXCLK, true);
+ if (ret) {
+ pr_err("[GetSclks]: fail to get max PPCLK_GFXCLK\n");
+ return ret;
+ }
+ }
+
+ return (gfx_clk * 100);
+}
+
+static uint32_t smu_v11_0_dpm_get_mclk(struct smu_context *smu, bool low)
+{
+ uint32_t mem_clk;
+ int ret;
+
+ if (!smu_feature_is_enabled(smu, FEATURE_DPM_UCLK_BIT)) {
+ pr_err("[GetMclks]: memclk dpm not enabled!\n");
+ return -EPERM;
+ }
+
+ if (low) {
+ ret = smu_v11_0_get_clock_ranges(smu, &mem_clk, PPCLK_UCLK, false);
+ if (ret) {
+ pr_err("[GetMclks]: fail to get min PPCLK_UCLK\n");
+ return ret;
+ }
+ } else {
+ ret = smu_v11_0_get_clock_ranges(smu, &mem_clk, PPCLK_GFXCLK, true);
+ if (ret) {
+ pr_err("[GetMclks]: fail to get max PPCLK_UCLK\n");
+ return ret;
+ }
+ }
+
+ return (mem_clk * 100);
+}
+
+static int smu_v11_0_set_od8_default_settings(struct smu_context *smu,
+ bool initialize)
+{
+ struct smu_table_context *table_context = &smu->smu_table;
+ int ret;
+
+ if (initialize) {
+ if (table_context->overdrive_table)
+ return -EINVAL;
+
+ table_context->overdrive_table = kzalloc(sizeof(OverDriveTable_t), GFP_KERNEL);
+
+ if (!table_context->overdrive_table)
+ return -ENOMEM;
+
+ ret = smu_update_table(smu, TABLE_OVERDRIVE, table_context->overdrive_table, false);
+ if (ret) {
+ pr_err("Failed to export over drive table!\n");
+ return ret;
+ }
+
+ smu_set_default_od8_settings(smu);
+ }
+
+ ret = smu_update_table(smu, TABLE_OVERDRIVE, table_context->overdrive_table, true);
+ if (ret) {
+ pr_err("Failed to import over drive table!\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int smu_v11_0_conv_power_profile_to_pplib_workload(int power_profile)
+{
+ int pplib_workload = 0;
+
+ switch (power_profile) {
+ case PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT:
+ pplib_workload = WORKLOAD_DEFAULT_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_FULLSCREEN3D:
+ pplib_workload = WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_POWERSAVING:
+ pplib_workload = WORKLOAD_PPLIB_POWER_SAVING_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_VIDEO:
+ pplib_workload = WORKLOAD_PPLIB_VIDEO_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_VR:
+ pplib_workload = WORKLOAD_PPLIB_VR_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_COMPUTE:
+ pplib_workload = WORKLOAD_PPLIB_COMPUTE_BIT;
+ break;
+ case PP_SMC_POWER_PROFILE_CUSTOM:
+ pplib_workload = WORKLOAD_PPLIB_CUSTOM_BIT;
+ break;
+ }
+
+ return pplib_workload;
+}
+
+static int smu_v11_0_get_power_profile_mode(struct smu_context *smu, char *buf)
+{
+ DpmActivityMonitorCoeffInt_t activity_monitor;
+ uint32_t i, size = 0;
+ uint16_t workload_type = 0;
+ static const char *profile_name[] = {
+ "BOOTUP_DEFAULT",
+ "3D_FULL_SCREEN",
+ "POWER_SAVING",
+ "VIDEO",
+ "VR",
+ "COMPUTE",
+ "CUSTOM"};
+ static const char *title[] = {
+ "PROFILE_INDEX(NAME)",
+ "CLOCK_TYPE(NAME)",
+ "FPS",
+ "UseRlcBusy",
+ "MinActiveFreqType",
+ "MinActiveFreq",
+ "BoosterFreqType",
+ "BoosterFreq",
+ "PD_Data_limit_c",
+ "PD_Data_error_coeff",
+ "PD_Data_error_rate_coeff"};
+ int result = 0;
+
+ if (!buf)
+ return -EINVAL;
+
+ size += sprintf(buf + size, "%16s %s %s %s %s %s %s %s %s %s %s\n",
+ title[0], title[1], title[2], title[3], title[4], title[5],
+ title[6], title[7], title[8], title[9], title[10]);
+
+ for (i = 0; i <= PP_SMC_POWER_PROFILE_CUSTOM; i++) {
+ /* conv PP_SMC_POWER_PROFILE* to WORKLOAD_PPLIB_*_BIT */
+ workload_type = smu_v11_0_conv_power_profile_to_pplib_workload(i);
+ result = smu_update_table_with_arg(smu, TABLE_ACTIVITY_MONITOR_COEFF,
+ workload_type, &activity_monitor, false);
+ if (result) {
+ pr_err("[%s] Failed to get activity monitor!", __func__);
+ return result;
+ }
+
+ size += sprintf(buf + size, "%2d %14s%s:\n",
+ i, profile_name[i], (i == smu->power_profile_mode) ? "*" : " ");
+
+ size += sprintf(buf + size, "%19s %d(%13s) %7d %7d %7d %7d %7d %7d %7d %7d %7d\n",
+ " ",
+ 0,
+ "GFXCLK",
+ activity_monitor.Gfx_FPS,
+ activity_monitor.Gfx_UseRlcBusy,
+ activity_monitor.Gfx_MinActiveFreqType,
+ activity_monitor.Gfx_MinActiveFreq,
+ activity_monitor.Gfx_BoosterFreqType,
+ activity_monitor.Gfx_BoosterFreq,
+ activity_monitor.Gfx_PD_Data_limit_c,
+ activity_monitor.Gfx_PD_Data_error_coeff,
+ activity_monitor.Gfx_PD_Data_error_rate_coeff);
+
+ size += sprintf(buf + size, "%19s %d(%13s) %7d %7d %7d %7d %7d %7d %7d %7d %7d\n",
+ " ",
+ 1,
+ "SOCCLK",
+ activity_monitor.Soc_FPS,
+ activity_monitor.Soc_UseRlcBusy,
+ activity_monitor.Soc_MinActiveFreqType,
+ activity_monitor.Soc_MinActiveFreq,
+ activity_monitor.Soc_BoosterFreqType,
+ activity_monitor.Soc_BoosterFreq,
+ activity_monitor.Soc_PD_Data_limit_c,
+ activity_monitor.Soc_PD_Data_error_coeff,
+ activity_monitor.Soc_PD_Data_error_rate_coeff);
+
+ size += sprintf(buf + size, "%19s %d(%13s) %7d %7d %7d %7d %7d %7d %7d %7d %7d\n",
+ " ",
+ 2,
+ "UCLK",
+ activity_monitor.Mem_FPS,
+ activity_monitor.Mem_UseRlcBusy,
+ activity_monitor.Mem_MinActiveFreqType,
+ activity_monitor.Mem_MinActiveFreq,
+ activity_monitor.Mem_BoosterFreqType,
+ activity_monitor.Mem_BoosterFreq,
+ activity_monitor.Mem_PD_Data_limit_c,
+ activity_monitor.Mem_PD_Data_error_coeff,
+ activity_monitor.Mem_PD_Data_error_rate_coeff);
+
+ size += sprintf(buf + size, "%19s %d(%13s) %7d %7d %7d %7d %7d %7d %7d %7d %7d\n",
+ " ",
+ 3,
+ "FCLK",
+ activity_monitor.Fclk_FPS,
+ activity_monitor.Fclk_UseRlcBusy,
+ activity_monitor.Fclk_MinActiveFreqType,
+ activity_monitor.Fclk_MinActiveFreq,
+ activity_monitor.Fclk_BoosterFreqType,
+ activity_monitor.Fclk_BoosterFreq,
+ activity_monitor.Fclk_PD_Data_limit_c,
+ activity_monitor.Fclk_PD_Data_error_coeff,
+ activity_monitor.Fclk_PD_Data_error_rate_coeff);
+ }
+
+ return size;
+}
+
+static int smu_v11_0_set_power_profile_mode(struct smu_context *smu, long *input, uint32_t size)
+{
+ DpmActivityMonitorCoeffInt_t activity_monitor;
+ int workload_type = 0, ret = 0;
+
+ smu->power_profile_mode = input[size];
+
+ if (smu->power_profile_mode > PP_SMC_POWER_PROFILE_CUSTOM) {
+ pr_err("Invalid power profile mode %d\n", smu->power_profile_mode);
+ return -EINVAL;
+ }
+
+ if (smu->power_profile_mode == PP_SMC_POWER_PROFILE_CUSTOM) {
+ ret = smu_update_table_with_arg(smu, TABLE_ACTIVITY_MONITOR_COEFF,
+ WORKLOAD_PPLIB_CUSTOM_BIT, &activity_monitor, false);
+ if (ret) {
+ pr_err("[%s] Failed to get activity monitor!", __func__);
+ return ret;
+ }
+
+ switch (input[0]) {
+ case 0: /* Gfxclk */
+ activity_monitor.Gfx_FPS = input[1];
+ activity_monitor.Gfx_UseRlcBusy = input[2];
+ activity_monitor.Gfx_MinActiveFreqType = input[3];
+ activity_monitor.Gfx_MinActiveFreq = input[4];
+ activity_monitor.Gfx_BoosterFreqType = input[5];
+ activity_monitor.Gfx_BoosterFreq = input[6];
+ activity_monitor.Gfx_PD_Data_limit_c = input[7];
+ activity_monitor.Gfx_PD_Data_error_coeff = input[8];
+ activity_monitor.Gfx_PD_Data_error_rate_coeff = input[9];
+ break;
+ case 1: /* Socclk */
+ activity_monitor.Soc_FPS = input[1];
+ activity_monitor.Soc_UseRlcBusy = input[2];
+ activity_monitor.Soc_MinActiveFreqType = input[3];
+ activity_monitor.Soc_MinActiveFreq = input[4];
+ activity_monitor.Soc_BoosterFreqType = input[5];
+ activity_monitor.Soc_BoosterFreq = input[6];
+ activity_monitor.Soc_PD_Data_limit_c = input[7];
+ activity_monitor.Soc_PD_Data_error_coeff = input[8];
+ activity_monitor.Soc_PD_Data_error_rate_coeff = input[9];
+ break;
+ case 2: /* Uclk */
+ activity_monitor.Mem_FPS = input[1];
+ activity_monitor.Mem_UseRlcBusy = input[2];
+ activity_monitor.Mem_MinActiveFreqType = input[3];
+ activity_monitor.Mem_MinActiveFreq = input[4];
+ activity_monitor.Mem_BoosterFreqType = input[5];
+ activity_monitor.Mem_BoosterFreq = input[6];
+ activity_monitor.Mem_PD_Data_limit_c = input[7];
+ activity_monitor.Mem_PD_Data_error_coeff = input[8];
+ activity_monitor.Mem_PD_Data_error_rate_coeff = input[9];
+ break;
+ case 3: /* Fclk */
+ activity_monitor.Fclk_FPS = input[1];
+ activity_monitor.Fclk_UseRlcBusy = input[2];
+ activity_monitor.Fclk_MinActiveFreqType = input[3];
+ activity_monitor.Fclk_MinActiveFreq = input[4];
+ activity_monitor.Fclk_BoosterFreqType = input[5];
+ activity_monitor.Fclk_BoosterFreq = input[6];
+ activity_monitor.Fclk_PD_Data_limit_c = input[7];
+ activity_monitor.Fclk_PD_Data_error_coeff = input[8];
+ activity_monitor.Fclk_PD_Data_error_rate_coeff = input[9];
+ break;
+ }
+
+ ret = smu_update_table_with_arg(smu, TABLE_ACTIVITY_MONITOR_COEFF,
+ WORKLOAD_PPLIB_COMPUTE_BIT, &activity_monitor, true);
+ if (ret) {
+ pr_err("[%s] Failed to set activity monitor!", __func__);
+ return ret;
+ }
+ }
+
+ /* conv PP_SMC_POWER_PROFILE* to WORKLOAD_PPLIB_*_BIT */
+ workload_type =
+ smu_v11_0_conv_power_profile_to_pplib_workload(smu->power_profile_mode);
+ smu_send_smc_msg_with_param(smu, SMU_MSG_SetWorkloadMask,
+ 1 << workload_type);
+
+ return ret;
+}
+
+static int smu_v11_0_update_od8_settings(struct smu_context *smu,
+ uint32_t index,
+ uint32_t value)
+{
+ struct smu_table_context *table_context = &smu->smu_table;
+ int ret;
+
+ ret = smu_update_table(smu, TABLE_OVERDRIVE,
+ table_context->overdrive_table, false);
+ if (ret) {
+ pr_err("Failed to export over drive table!\n");
+ return ret;
+ }
+
+ smu_update_specified_od8_value(smu, index, value);
+
+ ret = smu_update_table(smu, TABLE_OVERDRIVE,
+ table_context->overdrive_table, true);
+ if (ret) {
+ pr_err("Failed to import over drive table!\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int smu_v11_0_dpm_set_uvd_enable(struct smu_context *smu, bool enable)
+{
+ if (!smu_feature_is_supported(smu, FEATURE_DPM_VCE_BIT))
+ return 0;
+
+ if (enable == smu_feature_is_enabled(smu, FEATURE_DPM_VCE_BIT))
+ return 0;
+
+ return smu_feature_set_enabled(smu, FEATURE_DPM_VCE_BIT, enable);
+}
+
+static int smu_v11_0_dpm_set_vce_enable(struct smu_context *smu, bool enable)
+{
+ if (!smu_feature_is_supported(smu, FEATURE_DPM_UVD_BIT))
+ return 0;
+
+ if (enable == smu_feature_is_enabled(smu, FEATURE_DPM_UVD_BIT))
+ return 0;
+
+ return smu_feature_set_enabled(smu, FEATURE_DPM_UVD_BIT, enable);
+}
+
+static int smu_v11_0_get_current_rpm(struct smu_context *smu,
+ uint32_t *current_rpm)
+{
+ int ret;
+
+ ret = smu_send_smc_msg(smu, SMU_MSG_GetCurrentRpm);
+
+ if (ret) {
+ pr_err("Attempt to get current RPM from SMC Failed!\n");
+ return ret;
+ }
+
+ smu_read_smc_arg(smu, current_rpm);
+
+ return 0;
+}
+
+static uint32_t
+smu_v11_0_get_fan_control_mode(struct smu_context *smu)
+{
+ if (!smu_feature_is_enabled(smu, FEATURE_FAN_CONTROL_BIT))
+ return AMD_FAN_CTRL_MANUAL;
+ else
+ return AMD_FAN_CTRL_AUTO;
+}
+
+static int
+smu_v11_0_get_fan_speed_percent(struct smu_context *smu,
+ uint32_t *speed)
+{
+ int ret = 0;
+ uint32_t percent = 0;
+ uint32_t current_rpm;
+ PPTable_t *pptable = smu->smu_table.driver_pptable;
+
+ ret = smu_v11_0_get_current_rpm(smu, ¤t_rpm);
+ percent = current_rpm * 100 / pptable->FanMaximumRpm;
+ *speed = percent > 100 ? 100 : percent;
+
+ return ret;
+}
+
+static int
+smu_v11_0_smc_fan_control(struct smu_context *smu, bool start)
+{
+ int ret = 0;
+
+ if (smu_feature_is_supported(smu, FEATURE_FAN_CONTROL_BIT))
+ return 0;
+
+ ret = smu_feature_set_enabled(smu, FEATURE_FAN_CONTROL_BIT, start);
+ if (ret)
+ pr_err("[%s]%s smc FAN CONTROL feature failed!",
+ __func__, (start ? "Start" : "Stop"));
+
+ return ret;
+}
+
+static int
+smu_v11_0_set_fan_static_mode(struct smu_context *smu, uint32_t mode)
+{
+ struct amdgpu_device *adev = smu->adev;
+
+ WREG32_SOC15(THM, 0, mmCG_FDO_CTRL2,
+ REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL2),
+ CG_FDO_CTRL2, TMIN, 0));
+ WREG32_SOC15(THM, 0, mmCG_FDO_CTRL2,
+ REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL2),
+ CG_FDO_CTRL2, FDO_PWM_MODE, mode));
+
+ return 0;
+}
+
+static int
+smu_v11_0_set_fan_speed_percent(struct smu_context *smu, uint32_t speed)
+{
+ struct amdgpu_device *adev = smu->adev;
+ uint32_t duty100;
+ uint32_t duty;
+ uint64_t tmp64;
+ bool stop = 0;
+
+ if (speed > 100)
+ speed = 100;
+
+ if (smu_v11_0_smc_fan_control(smu, stop))
+ return -EINVAL;
+ duty100 = REG_GET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL1),
+ CG_FDO_CTRL1, FMAX_DUTY100);
+ if (!duty100)
+ return -EINVAL;
+
+ tmp64 = (uint64_t)speed * duty100;
+ do_div(tmp64, 100);
+ duty = (uint32_t)tmp64;
+
+ WREG32_SOC15(THM, 0, mmCG_FDO_CTRL0,
+ REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_FDO_CTRL0),
+ CG_FDO_CTRL0, FDO_STATIC_DUTY, duty));
+
+ return smu_v11_0_set_fan_static_mode(smu, FDO_PWM_MODE_STATIC);
+}
+
+static int
+smu_v11_0_set_fan_control_mode(struct smu_context *smu,
+ uint32_t mode)
+{
+ int ret = 0;
+ bool start = 1;
+ bool stop = 0;
+
+ switch (mode) {
+ case AMD_FAN_CTRL_NONE:
+ ret = smu_v11_0_set_fan_speed_percent(smu, 100);
+ break;
+ case AMD_FAN_CTRL_MANUAL:
+ ret = smu_v11_0_smc_fan_control(smu, stop);
+ break;
+ case AMD_FAN_CTRL_AUTO:
+ ret = smu_v11_0_smc_fan_control(smu, start);
+ break;
+ default:
+ break;
+ }
+
+ if (ret) {
+ pr_err("[%s]Set fan control mode failed!", __func__);
+ return -EINVAL;
+ }
+
+ return ret;
+}
+
+static int smu_v11_0_set_fan_speed_rpm(struct smu_context *smu,
+ uint32_t speed)
+{
+ struct amdgpu_device *adev = smu->adev;
+ int ret;
+ uint32_t tach_period, crystal_clock_freq;
+ bool stop = 0;
+
+ if (!speed)
+ return -EINVAL;
+
+ mutex_lock(&(smu->mutex));
+ ret = smu_v11_0_smc_fan_control(smu, stop);
+ if (ret)
+ goto set_fan_speed_rpm_failed;
+
+ crystal_clock_freq = amdgpu_asic_get_xclk(adev);
+ tach_period = 60 * crystal_clock_freq * 10000 / (8 * speed);
+ WREG32_SOC15(THM, 0, mmCG_TACH_CTRL,
+ REG_SET_FIELD(RREG32_SOC15(THM, 0, mmCG_TACH_CTRL),
+ CG_TACH_CTRL, TARGET_PERIOD,
+ tach_period));
+
+ ret = smu_v11_0_set_fan_static_mode(smu, FDO_PWM_MODE_STATIC_RPM);
+
+set_fan_speed_rpm_failed:
+ mutex_unlock(&(smu->mutex));
+ return ret;
+}
+
+static int smu_v11_0_set_xgmi_pstate(struct smu_context *smu,
+ uint32_t pstate)
+{
+ int ret = 0;
+ mutex_lock(&(smu->mutex));
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetXgmiMode,
+ pstate ? XGMI_STATE_D0 : XGMI_STATE_D3);
+ mutex_unlock(&(smu->mutex));
+ return ret;
+}
+
+static const struct smu_funcs smu_v11_0_funcs = {
+ .init_microcode = smu_v11_0_init_microcode,
+ .load_microcode = smu_v11_0_load_microcode,
+ .check_fw_status = smu_v11_0_check_fw_status,
+ .check_fw_version = smu_v11_0_check_fw_version,
+ .send_smc_msg = smu_v11_0_send_msg,
+ .send_smc_msg_with_param = smu_v11_0_send_msg_with_param,
+ .read_smc_arg = smu_v11_0_read_arg,
+ .read_pptable_from_vbios = smu_v11_0_read_pptable_from_vbios,
+ .init_smc_tables = smu_v11_0_init_smc_tables,
+ .fini_smc_tables = smu_v11_0_fini_smc_tables,
+ .init_power = smu_v11_0_init_power,
+ .fini_power = smu_v11_0_fini_power,
+ .get_vbios_bootup_values = smu_v11_0_get_vbios_bootup_values,
+ .get_clk_info_from_vbios = smu_v11_0_get_clk_info_from_vbios,
+ .notify_memory_pool_location = smu_v11_0_notify_memory_pool_location,
+ .check_pptable = smu_v11_0_check_pptable,
+ .parse_pptable = smu_v11_0_parse_pptable,
+ .populate_smc_pptable = smu_v11_0_populate_smc_pptable,
+ .write_pptable = smu_v11_0_write_pptable,
+ .write_watermarks_table = smu_v11_0_write_watermarks_table,
+ .set_min_dcef_deep_sleep = smu_v11_0_set_min_dcef_deep_sleep,
+ .set_tool_table_location = smu_v11_0_set_tool_table_location,
+ .init_display = smu_v11_0_init_display,
+ .set_allowed_mask = smu_v11_0_set_allowed_mask,
+ .get_enabled_mask = smu_v11_0_get_enabled_mask,
+ .is_dpm_running = smu_v11_0_is_dpm_running,
+ .system_features_control = smu_v11_0_system_features_control,
+ .update_feature_enable_state = smu_v11_0_update_feature_enable_state,
+ .notify_display_change = smu_v11_0_notify_display_change,
+ .get_power_limit = smu_v11_0_get_power_limit,
+ .set_power_limit = smu_v11_0_set_power_limit,
+ .get_current_clk_freq = smu_v11_0_get_current_clk_freq,
+ .init_max_sustainable_clocks = smu_v11_0_init_max_sustainable_clocks,
+ .start_thermal_control = smu_v11_0_start_thermal_control,
+ .read_sensor = smu_v11_0_read_sensor,
+ .set_deep_sleep_dcefclk = smu_v11_0_set_deep_sleep_dcefclk,
+ .display_clock_voltage_request = smu_v11_0_display_clock_voltage_request,
+ .set_watermarks_for_clock_ranges = smu_v11_0_set_watermarks_for_clock_ranges,
+ .get_sclk = smu_v11_0_dpm_get_sclk,
+ .get_mclk = smu_v11_0_dpm_get_mclk,
+ .set_od8_default_settings = smu_v11_0_set_od8_default_settings,
+ .conv_power_profile_to_pplib_workload = smu_v11_0_conv_power_profile_to_pplib_workload,
+ .get_power_profile_mode = smu_v11_0_get_power_profile_mode,
+ .set_power_profile_mode = smu_v11_0_set_power_profile_mode,
+ .update_od8_settings = smu_v11_0_update_od8_settings,
+ .dpm_set_uvd_enable = smu_v11_0_dpm_set_uvd_enable,
+ .dpm_set_vce_enable = smu_v11_0_dpm_set_vce_enable,
+ .get_current_rpm = smu_v11_0_get_current_rpm,
+ .get_fan_control_mode = smu_v11_0_get_fan_control_mode,
+ .set_fan_control_mode = smu_v11_0_set_fan_control_mode,
+ .get_fan_speed_percent = smu_v11_0_get_fan_speed_percent,
+ .set_fan_speed_percent = smu_v11_0_set_fan_speed_percent,
+ .set_fan_speed_rpm = smu_v11_0_set_fan_speed_rpm,
+ .set_xgmi_pstate = smu_v11_0_set_xgmi_pstate,
+};
+
+void smu_v11_0_set_smu_funcs(struct smu_context *smu)
+{
+ struct amdgpu_device *adev = smu->adev;
+
+ smu->funcs = &smu_v11_0_funcs;
+ switch (adev->asic_type) {
+ case CHIP_VEGA20:
+ vega20_set_ppt_funcs(smu);
+ break;
+ default:
+ pr_warn("Unknown asic for smu11\n");
+ }
+}
hwmgr->smu_version = smu10_read_arg_from_smc(hwmgr);
adev->pm.fw_version = hwmgr->smu_version >> 8;
+ if (adev->rev_id < 0x8 && adev->pdev->device != 0x15d8 &&
+ adev->pm.fw_version < 0x1e45)
+ adev->pm.pp_feature &= ~PP_GFXOFF_MASK;
+
if (smu10_verify_smc_interface(hwmgr))
return -EINVAL;
return ret;
}
+int vega20_set_pptable_driver_address(struct pp_hwmgr *hwmgr)
+{
+ struct vega20_smumgr *priv =
+ (struct vega20_smumgr *)(hwmgr->smu_backend);
+ int ret = 0;
+
+ PP_ASSERT_WITH_CODE((ret = vega20_send_msg_to_smc_with_parameter(hwmgr,
+ PPSMC_MSG_SetDriverDramAddrHigh,
+ upper_32_bits(priv->smu_tables.entry[TABLE_PPTABLE].mc_addr))) == 0,
+ "[SetPPtabeDriverAddress] Attempt to Set Dram Addr High Failed!",
+ return ret);
+ PP_ASSERT_WITH_CODE((ret = vega20_send_msg_to_smc_with_parameter(hwmgr,
+ PPSMC_MSG_SetDriverDramAddrLow,
+ lower_32_bits(priv->smu_tables.entry[TABLE_PPTABLE].mc_addr))) == 0,
+ "[SetPPtabeDriverAddress] Attempt to Set Dram Addr Low Failed!",
+ return ret);
+
+ return ret;
+}
+
static int vega20_smu_init(struct pp_hwmgr *hwmgr)
{
struct vega20_smumgr *priv;
uint8_t *table, uint16_t workload_type);
int vega20_get_activity_monitor_coeff(struct pp_hwmgr *hwmgr,
uint8_t *table, uint16_t workload_type);
+int vega20_set_pptable_driver_address(struct pp_hwmgr *hwmgr);
#endif
--- /dev/null
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#include "pp_debug.h"
+#include <linux/firmware.h>
+#include "amdgpu.h"
+#include "amdgpu_smu.h"
+#include "atomfirmware.h"
+#include "amdgpu_atomfirmware.h"
+#include "smu_v11_0.h"
+#include "smu11_driver_if.h"
+#include "soc15_common.h"
+#include "atom.h"
+#include "power_state.h"
+#include "vega20_ppt.h"
+#include "vega20_pptable.h"
+#include "vega20_ppsmc.h"
+#include "nbio/nbio_7_4_sh_mask.h"
+
+#define smnPCIE_LC_SPEED_CNTL 0x11140290
+#define smnPCIE_LC_LINK_WIDTH_CNTL 0x11140288
+
+#define MSG_MAP(msg) \
+ [SMU_MSG_##msg] = PPSMC_MSG_##msg
+
+static int vega20_message_map[SMU_MSG_MAX_COUNT] = {
+ MSG_MAP(TestMessage),
+ MSG_MAP(GetSmuVersion),
+ MSG_MAP(GetDriverIfVersion),
+ MSG_MAP(SetAllowedFeaturesMaskLow),
+ MSG_MAP(SetAllowedFeaturesMaskHigh),
+ MSG_MAP(EnableAllSmuFeatures),
+ MSG_MAP(DisableAllSmuFeatures),
+ MSG_MAP(EnableSmuFeaturesLow),
+ MSG_MAP(EnableSmuFeaturesHigh),
+ MSG_MAP(DisableSmuFeaturesLow),
+ MSG_MAP(DisableSmuFeaturesHigh),
+ MSG_MAP(GetEnabledSmuFeaturesLow),
+ MSG_MAP(GetEnabledSmuFeaturesHigh),
+ MSG_MAP(SetWorkloadMask),
+ MSG_MAP(SetPptLimit),
+ MSG_MAP(SetDriverDramAddrHigh),
+ MSG_MAP(SetDriverDramAddrLow),
+ MSG_MAP(SetToolsDramAddrHigh),
+ MSG_MAP(SetToolsDramAddrLow),
+ MSG_MAP(TransferTableSmu2Dram),
+ MSG_MAP(TransferTableDram2Smu),
+ MSG_MAP(UseDefaultPPTable),
+ MSG_MAP(UseBackupPPTable),
+ MSG_MAP(RunBtc),
+ MSG_MAP(RequestI2CBus),
+ MSG_MAP(ReleaseI2CBus),
+ MSG_MAP(SetFloorSocVoltage),
+ MSG_MAP(SoftReset),
+ MSG_MAP(StartBacoMonitor),
+ MSG_MAP(CancelBacoMonitor),
+ MSG_MAP(EnterBaco),
+ MSG_MAP(SetSoftMinByFreq),
+ MSG_MAP(SetSoftMaxByFreq),
+ MSG_MAP(SetHardMinByFreq),
+ MSG_MAP(SetHardMaxByFreq),
+ MSG_MAP(GetMinDpmFreq),
+ MSG_MAP(GetMaxDpmFreq),
+ MSG_MAP(GetDpmFreqByIndex),
+ MSG_MAP(GetDpmClockFreq),
+ MSG_MAP(GetSsVoltageByDpm),
+ MSG_MAP(SetMemoryChannelConfig),
+ MSG_MAP(SetGeminiMode),
+ MSG_MAP(SetGeminiApertureHigh),
+ MSG_MAP(SetGeminiApertureLow),
+ MSG_MAP(SetMinLinkDpmByIndex),
+ MSG_MAP(OverridePcieParameters),
+ MSG_MAP(OverDriveSetPercentage),
+ MSG_MAP(SetMinDeepSleepDcefclk),
+ MSG_MAP(ReenableAcDcInterrupt),
+ MSG_MAP(NotifyPowerSource),
+ MSG_MAP(SetUclkFastSwitch),
+ MSG_MAP(SetUclkDownHyst),
+ MSG_MAP(GetCurrentRpm),
+ MSG_MAP(SetVideoFps),
+ MSG_MAP(SetTjMax),
+ MSG_MAP(SetFanTemperatureTarget),
+ MSG_MAP(PrepareMp1ForUnload),
+ MSG_MAP(DramLogSetDramAddrHigh),
+ MSG_MAP(DramLogSetDramAddrLow),
+ MSG_MAP(DramLogSetDramSize),
+ MSG_MAP(SetFanMaxRpm),
+ MSG_MAP(SetFanMinPwm),
+ MSG_MAP(ConfigureGfxDidt),
+ MSG_MAP(NumOfDisplays),
+ MSG_MAP(RemoveMargins),
+ MSG_MAP(ReadSerialNumTop32),
+ MSG_MAP(ReadSerialNumBottom32),
+ MSG_MAP(SetSystemVirtualDramAddrHigh),
+ MSG_MAP(SetSystemVirtualDramAddrLow),
+ MSG_MAP(WaflTest),
+ MSG_MAP(SetFclkGfxClkRatio),
+ MSG_MAP(AllowGfxOff),
+ MSG_MAP(DisallowGfxOff),
+ MSG_MAP(GetPptLimit),
+ MSG_MAP(GetDcModeMaxDpmFreq),
+ MSG_MAP(GetDebugData),
+ MSG_MAP(SetXgmiMode),
+ MSG_MAP(RunAfllBtc),
+ MSG_MAP(ExitBaco),
+ MSG_MAP(PrepareMp1ForReset),
+ MSG_MAP(PrepareMp1ForShutdown),
+ MSG_MAP(SetMGpuFanBoostLimitRpm),
+ MSG_MAP(GetAVFSVoltageByDpm),
+};
+
+static int vega20_get_smu_msg_index(struct smu_context *smc, uint32_t index)
+{
+ int val;
+
+ if (index >= SMU_MSG_MAX_COUNT)
+ return -EINVAL;
+
+ val = vega20_message_map[index];
+ if (val > PPSMC_Message_Count)
+ return -EINVAL;
+
+ return val;
+}
+
+static int vega20_allocate_dpm_context(struct smu_context *smu)
+{
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+
+ if (smu_dpm->dpm_context)
+ return -EINVAL;
+
+ smu_dpm->dpm_context = kzalloc(sizeof(struct vega20_dpm_table),
+ GFP_KERNEL);
+ if (!smu_dpm->dpm_context)
+ return -ENOMEM;
+
+ if (smu_dpm->golden_dpm_context)
+ return -EINVAL;
+
+ smu_dpm->golden_dpm_context = kzalloc(sizeof(struct vega20_dpm_table),
+ GFP_KERNEL);
+ if (!smu_dpm->golden_dpm_context)
+ return -ENOMEM;
+
+ smu_dpm->dpm_context_size = sizeof(struct vega20_dpm_table);
+
+ smu_dpm->dpm_current_power_state = kzalloc(sizeof(struct smu_power_state),
+ GFP_KERNEL);
+ if (!smu_dpm->dpm_current_power_state)
+ return -ENOMEM;
+
+ smu_dpm->dpm_request_power_state = kzalloc(sizeof(struct smu_power_state),
+ GFP_KERNEL);
+ if (!smu_dpm->dpm_request_power_state)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static int vega20_setup_od8_information(struct smu_context *smu)
+{
+ ATOM_Vega20_POWERPLAYTABLE *powerplay_table = NULL;
+ struct smu_table_context *table_context = &smu->smu_table;
+
+ uint32_t od_feature_count, od_feature_array_size,
+ od_setting_count, od_setting_array_size;
+
+ if (!table_context->power_play_table)
+ return -EINVAL;
+
+ powerplay_table = table_context->power_play_table;
+
+ if (powerplay_table->OverDrive8Table.ucODTableRevision == 1) {
+ /* Setup correct ODFeatureCount, and store ODFeatureArray from
+ * powerplay table to od_feature_capabilities */
+ od_feature_count =
+ (le32_to_cpu(powerplay_table->OverDrive8Table.ODFeatureCount) >
+ ATOM_VEGA20_ODFEATURE_COUNT) ?
+ ATOM_VEGA20_ODFEATURE_COUNT :
+ le32_to_cpu(powerplay_table->OverDrive8Table.ODFeatureCount);
+
+ od_feature_array_size = sizeof(uint8_t) * od_feature_count;
+
+ if (table_context->od_feature_capabilities)
+ return -EINVAL;
+
+ table_context->od_feature_capabilities = kmemdup(&powerplay_table->OverDrive8Table.ODFeatureCapabilities,
+ od_feature_array_size,
+ GFP_KERNEL);
+ if (!table_context->od_feature_capabilities)
+ return -ENOMEM;
+
+ /* Setup correct ODSettingCount, and store ODSettingArray from
+ * powerplay table to od_settings_max and od_setting_min */
+ od_setting_count =
+ (le32_to_cpu(powerplay_table->OverDrive8Table.ODSettingCount) >
+ ATOM_VEGA20_ODSETTING_COUNT) ?
+ ATOM_VEGA20_ODSETTING_COUNT :
+ le32_to_cpu(powerplay_table->OverDrive8Table.ODSettingCount);
+
+ od_setting_array_size = sizeof(uint32_t) * od_setting_count;
+
+ if (table_context->od_settings_max)
+ return -EINVAL;
+
+ table_context->od_settings_max = kmemdup(&powerplay_table->OverDrive8Table.ODSettingsMax,
+ od_setting_array_size,
+ GFP_KERNEL);
+
+ if (!table_context->od_settings_max) {
+ kfree(table_context->od_feature_capabilities);
+ table_context->od_feature_capabilities = NULL;
+ return -ENOMEM;
+ }
+
+ if (table_context->od_settings_min)
+ return -EINVAL;
+
+ table_context->od_settings_min = kmemdup(&powerplay_table->OverDrive8Table.ODSettingsMin,
+ od_setting_array_size,
+ GFP_KERNEL);
+
+ if (!table_context->od_settings_min) {
+ kfree(table_context->od_feature_capabilities);
+ table_context->od_feature_capabilities = NULL;
+ kfree(table_context->od_settings_max);
+ table_context->od_settings_max = NULL;
+ return -ENOMEM;
+ }
+ }
+
+ return 0;
+}
+
+static int vega20_store_powerplay_table(struct smu_context *smu)
+{
+ ATOM_Vega20_POWERPLAYTABLE *powerplay_table = NULL;
+ struct smu_table_context *table_context = &smu->smu_table;
+ int ret;
+
+ if (!table_context->power_play_table)
+ return -EINVAL;
+
+ powerplay_table = table_context->power_play_table;
+
+ memcpy(table_context->driver_pptable, &powerplay_table->smcPPTable,
+ sizeof(PPTable_t));
+
+ table_context->software_shutdown_temp = powerplay_table->usSoftwareShutdownTemp;
+ table_context->thermal_controller_type = powerplay_table->ucThermalControllerType;
+ table_context->TDPODLimit = le32_to_cpu(powerplay_table->OverDrive8Table.ODSettingsMax[ATOM_VEGA20_ODSETTING_POWERPERCENTAGE]);
+
+ ret = vega20_setup_od8_information(smu);
+
+ return ret;
+}
+
+static int vega20_append_powerplay_table(struct smu_context *smu)
+{
+ struct smu_table_context *table_context = &smu->smu_table;
+ PPTable_t *smc_pptable = table_context->driver_pptable;
+ struct atom_smc_dpm_info_v4_4 *smc_dpm_table;
+ int index, i, ret;
+
+ index = get_index_into_master_table(atom_master_list_of_data_tables_v2_1,
+ smc_dpm_info);
+
+ ret = smu_get_atom_data_table(smu, index, NULL, NULL, NULL,
+ (uint8_t **)&smc_dpm_table);
+ if (ret)
+ return ret;
+
+ smc_pptable->MaxVoltageStepGfx = smc_dpm_table->maxvoltagestepgfx;
+ smc_pptable->MaxVoltageStepSoc = smc_dpm_table->maxvoltagestepsoc;
+
+ smc_pptable->VddGfxVrMapping = smc_dpm_table->vddgfxvrmapping;
+ smc_pptable->VddSocVrMapping = smc_dpm_table->vddsocvrmapping;
+ smc_pptable->VddMem0VrMapping = smc_dpm_table->vddmem0vrmapping;
+ smc_pptable->VddMem1VrMapping = smc_dpm_table->vddmem1vrmapping;
+
+ smc_pptable->GfxUlvPhaseSheddingMask = smc_dpm_table->gfxulvphasesheddingmask;
+ smc_pptable->SocUlvPhaseSheddingMask = smc_dpm_table->soculvphasesheddingmask;
+ smc_pptable->ExternalSensorPresent = smc_dpm_table->externalsensorpresent;
+
+ smc_pptable->GfxMaxCurrent = smc_dpm_table->gfxmaxcurrent;
+ smc_pptable->GfxOffset = smc_dpm_table->gfxoffset;
+ smc_pptable->Padding_TelemetryGfx = smc_dpm_table->padding_telemetrygfx;
+
+ smc_pptable->SocMaxCurrent = smc_dpm_table->socmaxcurrent;
+ smc_pptable->SocOffset = smc_dpm_table->socoffset;
+ smc_pptable->Padding_TelemetrySoc = smc_dpm_table->padding_telemetrysoc;
+
+ smc_pptable->Mem0MaxCurrent = smc_dpm_table->mem0maxcurrent;
+ smc_pptable->Mem0Offset = smc_dpm_table->mem0offset;
+ smc_pptable->Padding_TelemetryMem0 = smc_dpm_table->padding_telemetrymem0;
+
+ smc_pptable->Mem1MaxCurrent = smc_dpm_table->mem1maxcurrent;
+ smc_pptable->Mem1Offset = smc_dpm_table->mem1offset;
+ smc_pptable->Padding_TelemetryMem1 = smc_dpm_table->padding_telemetrymem1;
+
+ smc_pptable->AcDcGpio = smc_dpm_table->acdcgpio;
+ smc_pptable->AcDcPolarity = smc_dpm_table->acdcpolarity;
+ smc_pptable->VR0HotGpio = smc_dpm_table->vr0hotgpio;
+ smc_pptable->VR0HotPolarity = smc_dpm_table->vr0hotpolarity;
+
+ smc_pptable->VR1HotGpio = smc_dpm_table->vr1hotgpio;
+ smc_pptable->VR1HotPolarity = smc_dpm_table->vr1hotpolarity;
+ smc_pptable->Padding1 = smc_dpm_table->padding1;
+ smc_pptable->Padding2 = smc_dpm_table->padding2;
+
+ smc_pptable->LedPin0 = smc_dpm_table->ledpin0;
+ smc_pptable->LedPin1 = smc_dpm_table->ledpin1;
+ smc_pptable->LedPin2 = smc_dpm_table->ledpin2;
+
+ smc_pptable->PllGfxclkSpreadEnabled = smc_dpm_table->pllgfxclkspreadenabled;
+ smc_pptable->PllGfxclkSpreadPercent = smc_dpm_table->pllgfxclkspreadpercent;
+ smc_pptable->PllGfxclkSpreadFreq = smc_dpm_table->pllgfxclkspreadfreq;
+
+ smc_pptable->UclkSpreadEnabled = 0;
+ smc_pptable->UclkSpreadPercent = smc_dpm_table->uclkspreadpercent;
+ smc_pptable->UclkSpreadFreq = smc_dpm_table->uclkspreadfreq;
+
+ smc_pptable->FclkSpreadEnabled = smc_dpm_table->fclkspreadenabled;
+ smc_pptable->FclkSpreadPercent = smc_dpm_table->fclkspreadpercent;
+ smc_pptable->FclkSpreadFreq = smc_dpm_table->fclkspreadfreq;
+
+ smc_pptable->FllGfxclkSpreadEnabled = smc_dpm_table->fllgfxclkspreadenabled;
+ smc_pptable->FllGfxclkSpreadPercent = smc_dpm_table->fllgfxclkspreadpercent;
+ smc_pptable->FllGfxclkSpreadFreq = smc_dpm_table->fllgfxclkspreadfreq;
+
+ for (i = 0; i < I2C_CONTROLLER_NAME_COUNT; i++) {
+ smc_pptable->I2cControllers[i].Enabled =
+ smc_dpm_table->i2ccontrollers[i].enabled;
+ smc_pptable->I2cControllers[i].SlaveAddress =
+ smc_dpm_table->i2ccontrollers[i].slaveaddress;
+ smc_pptable->I2cControllers[i].ControllerPort =
+ smc_dpm_table->i2ccontrollers[i].controllerport;
+ smc_pptable->I2cControllers[i].ThermalThrottler =
+ smc_dpm_table->i2ccontrollers[i].thermalthrottler;
+ smc_pptable->I2cControllers[i].I2cProtocol =
+ smc_dpm_table->i2ccontrollers[i].i2cprotocol;
+ smc_pptable->I2cControllers[i].I2cSpeed =
+ smc_dpm_table->i2ccontrollers[i].i2cspeed;
+ }
+
+ return 0;
+}
+
+static int vega20_check_powerplay_table(struct smu_context *smu)
+{
+ ATOM_Vega20_POWERPLAYTABLE *powerplay_table = NULL;
+ struct smu_table_context *table_context = &smu->smu_table;
+
+ powerplay_table = table_context->power_play_table;
+
+ if (powerplay_table->sHeader.format_revision < ATOM_VEGA20_TABLE_REVISION_VEGA20) {
+ pr_err("Unsupported PPTable format!");
+ return -EINVAL;
+ }
+
+ if (!powerplay_table->sHeader.structuresize) {
+ pr_err("Invalid PowerPlay Table!");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int vega20_run_btc_afll(struct smu_context *smu)
+{
+ return smu_send_smc_msg(smu, SMU_MSG_RunAfllBtc);
+}
+
+static int
+vega20_get_unallowed_feature_mask(struct smu_context *smu,
+ uint32_t *feature_mask, uint32_t num)
+{
+ if (num > 2)
+ return -EINVAL;
+
+ feature_mask[0] = 0xE0041C00;
+ feature_mask[1] = 0xFFFFFFFE; /* bit32~bit63 is Unsupported */
+
+ return 0;
+}
+
+static enum
+amd_pm_state_type vega20_get_current_power_state(struct smu_context *smu)
+{
+ enum amd_pm_state_type pm_type;
+ struct smu_dpm_context *smu_dpm_ctx = &(smu->smu_dpm);
+
+ if (!smu_dpm_ctx->dpm_context ||
+ !smu_dpm_ctx->dpm_current_power_state)
+ return -EINVAL;
+
+ mutex_lock(&(smu->mutex));
+ switch (smu_dpm_ctx->dpm_current_power_state->classification.ui_label) {
+ case SMU_STATE_UI_LABEL_BATTERY:
+ pm_type = POWER_STATE_TYPE_BATTERY;
+ break;
+ case SMU_STATE_UI_LABEL_BALLANCED:
+ pm_type = POWER_STATE_TYPE_BALANCED;
+ break;
+ case SMU_STATE_UI_LABEL_PERFORMANCE:
+ pm_type = POWER_STATE_TYPE_PERFORMANCE;
+ break;
+ default:
+ if (smu_dpm_ctx->dpm_current_power_state->classification.flags & SMU_STATE_CLASSIFICATION_FLAG_BOOT)
+ pm_type = POWER_STATE_TYPE_INTERNAL_BOOT;
+ else
+ pm_type = POWER_STATE_TYPE_DEFAULT;
+ break;
+ }
+ mutex_unlock(&(smu->mutex));
+
+ return pm_type;
+}
+
+static int
+vega20_set_single_dpm_table(struct smu_context *smu,
+ struct vega20_single_dpm_table *single_dpm_table,
+ PPCLK_e clk_id)
+{
+ int ret = 0;
+ uint32_t i, num_of_levels = 0, clk;
+
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_GetDpmFreqByIndex,
+ (clk_id << 16 | 0xFF));
+ if (ret) {
+ pr_err("[GetNumOfDpmLevel] failed to get dpm levels!");
+ return ret;
+ }
+
+ smu_read_smc_arg(smu, &num_of_levels);
+ if (!num_of_levels) {
+ pr_err("[GetNumOfDpmLevel] number of clk levels is invalid!");
+ return -EINVAL;
+ }
+
+ single_dpm_table->count = num_of_levels;
+
+ for (i = 0; i < num_of_levels; i++) {
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_GetDpmFreqByIndex,
+ (clk_id << 16 | i));
+ if (ret) {
+ pr_err("[GetDpmFreqByIndex] failed to get dpm freq by index!");
+ return ret;
+ }
+ smu_read_smc_arg(smu, &clk);
+ if (!clk) {
+ pr_err("[GetDpmFreqByIndex] clk value is invalid!");
+ return -EINVAL;
+ }
+ single_dpm_table->dpm_levels[i].value = clk;
+ single_dpm_table->dpm_levels[i].enabled = true;
+ }
+ return 0;
+}
+
+static void vega20_init_single_dpm_state(struct vega20_dpm_state *dpm_state)
+{
+ dpm_state->soft_min_level = 0x0;
+ dpm_state->soft_max_level = 0xffff;
+ dpm_state->hard_min_level = 0x0;
+ dpm_state->hard_max_level = 0xffff;
+}
+
+static int vega20_set_default_dpm_table(struct smu_context *smu)
+{
+ int ret;
+
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+ struct vega20_dpm_table *dpm_table = NULL;
+ struct vega20_single_dpm_table *single_dpm_table;
+
+ dpm_table = smu_dpm->dpm_context;
+
+ /* socclk */
+ single_dpm_table = &(dpm_table->soc_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_SOCCLK_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table,
+ PPCLK_SOCCLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get socclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 1;
+ single_dpm_table->dpm_levels[0].value = smu->smu_table.boot_values.socclk / 100;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ /* gfxclk */
+ single_dpm_table = &(dpm_table->gfx_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_GFXCLK_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table,
+ PPCLK_GFXCLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get gfxclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 1;
+ single_dpm_table->dpm_levels[0].value = smu->smu_table.boot_values.gfxclk / 100;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ /* memclk */
+ single_dpm_table = &(dpm_table->mem_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_UCLK_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table,
+ PPCLK_UCLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get memclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 1;
+ single_dpm_table->dpm_levels[0].value = smu->smu_table.boot_values.uclk / 100;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ /* eclk */
+ single_dpm_table = &(dpm_table->eclk_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_VCE_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table, PPCLK_ECLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get eclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 1;
+ single_dpm_table->dpm_levels[0].value = smu->smu_table.boot_values.eclk / 100;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ /* vclk */
+ single_dpm_table = &(dpm_table->vclk_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_UVD_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table, PPCLK_VCLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get vclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 1;
+ single_dpm_table->dpm_levels[0].value = smu->smu_table.boot_values.vclk / 100;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ /* dclk */
+ single_dpm_table = &(dpm_table->dclk_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_UVD_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table, PPCLK_DCLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get dclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 1;
+ single_dpm_table->dpm_levels[0].value = smu->smu_table.boot_values.dclk / 100;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ /* dcefclk */
+ single_dpm_table = &(dpm_table->dcef_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_DCEFCLK_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table,
+ PPCLK_DCEFCLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get dcefclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 1;
+ single_dpm_table->dpm_levels[0].value = smu->smu_table.boot_values.dcefclk / 100;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ /* pixclk */
+ single_dpm_table = &(dpm_table->pixel_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_DCEFCLK_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table,
+ PPCLK_PIXCLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get pixclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 0;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ /* dispclk */
+ single_dpm_table = &(dpm_table->display_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_DCEFCLK_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table,
+ PPCLK_DISPCLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get dispclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 0;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ /* phyclk */
+ single_dpm_table = &(dpm_table->phy_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_DCEFCLK_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table,
+ PPCLK_PHYCLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get phyclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 0;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ /* fclk */
+ single_dpm_table = &(dpm_table->fclk_table);
+
+ if (smu_feature_is_enabled(smu,FEATURE_DPM_FCLK_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table,
+ PPCLK_FCLK);
+ if (ret) {
+ pr_err("[SetupDefaultDpmTable] failed to get fclk dpm levels!");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 0;
+ }
+ vega20_init_single_dpm_state(&(single_dpm_table->dpm_state));
+
+ memcpy(smu_dpm->golden_dpm_context, dpm_table,
+ sizeof(struct vega20_dpm_table));
+
+ return 0;
+}
+
+static int vega20_populate_umd_state_clk(struct smu_context *smu)
+{
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+ struct vega20_dpm_table *dpm_table = NULL;
+ struct vega20_single_dpm_table *gfx_table = NULL;
+ struct vega20_single_dpm_table *mem_table = NULL;
+
+ dpm_table = smu_dpm->dpm_context;
+ gfx_table = &(dpm_table->gfx_table);
+ mem_table = &(dpm_table->mem_table);
+
+ smu->pstate_sclk = gfx_table->dpm_levels[0].value;
+ smu->pstate_mclk = mem_table->dpm_levels[0].value;
+
+ if (gfx_table->count > VEGA20_UMD_PSTATE_GFXCLK_LEVEL &&
+ mem_table->count > VEGA20_UMD_PSTATE_MCLK_LEVEL) {
+ smu->pstate_sclk = gfx_table->dpm_levels[VEGA20_UMD_PSTATE_GFXCLK_LEVEL].value;
+ smu->pstate_mclk = mem_table->dpm_levels[VEGA20_UMD_PSTATE_MCLK_LEVEL].value;
+ }
+
+ smu->pstate_sclk = smu->pstate_sclk * 100;
+ smu->pstate_mclk = smu->pstate_mclk * 100;
+
+ return 0;
+}
+
+static int vega20_get_clk_table(struct smu_context *smu,
+ struct pp_clock_levels_with_latency *clocks,
+ struct vega20_single_dpm_table *dpm_table)
+{
+ int i, count;
+
+ count = (dpm_table->count > MAX_NUM_CLOCKS) ? MAX_NUM_CLOCKS : dpm_table->count;
+ clocks->num_levels = count;
+
+ for (i = 0; i < count; i++) {
+ clocks->data[i].clocks_in_khz =
+ dpm_table->dpm_levels[i].value * 1000;
+ clocks->data[i].latency_in_us = 0;
+ }
+
+ return 0;
+}
+
+static int vega20_print_clk_levels(struct smu_context *smu,
+ enum pp_clock_type type, char *buf)
+{
+ int i, now, size = 0;
+ int ret = 0;
+ uint32_t gen_speed, lane_width;
+ struct amdgpu_device *adev = smu->adev;
+ struct pp_clock_levels_with_latency clocks;
+ struct vega20_single_dpm_table *single_dpm_table;
+ struct smu_table_context *table_context = &smu->smu_table;
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+ struct vega20_dpm_table *dpm_table = NULL;
+ struct vega20_od8_settings *od8_settings =
+ (struct vega20_od8_settings *)table_context->od8_settings;
+ OverDriveTable_t *od_table =
+ (OverDriveTable_t *)(table_context->overdrive_table);
+ PPTable_t *pptable = (PPTable_t *)table_context->driver_pptable;
+
+ dpm_table = smu_dpm->dpm_context;
+
+ switch (type) {
+ case PP_SCLK:
+ ret = smu_get_current_clk_freq(smu, PPCLK_GFXCLK, &now);
+ if (ret) {
+ pr_err("Attempt to get current gfx clk Failed!");
+ return ret;
+ }
+
+ single_dpm_table = &(dpm_table->gfx_table);
+ ret = vega20_get_clk_table(smu, &clocks, single_dpm_table);
+ if (ret) {
+ pr_err("Attempt to get gfx clk levels Failed!");
+ return ret;
+ }
+
+ for (i = 0; i < clocks.num_levels; i++)
+ size += sprintf(buf + size, "%d: %uMhz %s\n", i,
+ clocks.data[i].clocks_in_khz / 1000,
+ (clocks.data[i].clocks_in_khz == now * 10)
+ ? "*" : "");
+ break;
+
+ case PP_MCLK:
+ ret = smu_get_current_clk_freq(smu, PPCLK_UCLK, &now);
+ if (ret) {
+ pr_err("Attempt to get current mclk Failed!");
+ return ret;
+ }
+
+ single_dpm_table = &(dpm_table->mem_table);
+ ret = vega20_get_clk_table(smu, &clocks, single_dpm_table);
+ if (ret) {
+ pr_err("Attempt to get memory clk levels Failed!");
+ return ret;
+ }
+
+ for (i = 0; i < clocks.num_levels; i++)
+ size += sprintf(buf + size, "%d: %uMhz %s\n",
+ i, clocks.data[i].clocks_in_khz / 1000,
+ (clocks.data[i].clocks_in_khz == now * 10)
+ ? "*" : "");
+ break;
+
+ case PP_SOCCLK:
+ ret = smu_get_current_clk_freq(smu, PPCLK_SOCCLK, &now);
+ if (ret) {
+ pr_err("Attempt to get current socclk Failed!");
+ return ret;
+ }
+
+ single_dpm_table = &(dpm_table->soc_table);
+ ret = vega20_get_clk_table(smu, &clocks, single_dpm_table);
+ if (ret) {
+ pr_err("Attempt to get socclk levels Failed!");
+ return ret;
+ }
+
+ for (i = 0; i < clocks.num_levels; i++)
+ size += sprintf(buf + size, "%d: %uMhz %s\n",
+ i, clocks.data[i].clocks_in_khz / 1000,
+ (clocks.data[i].clocks_in_khz == now * 10)
+ ? "*" : "");
+ break;
+
+ case PP_FCLK:
+ ret = smu_get_current_clk_freq(smu, PPCLK_FCLK, &now);
+ if (ret) {
+ pr_err("Attempt to get current fclk Failed!");
+ return ret;
+ }
+
+ single_dpm_table = &(dpm_table->fclk_table);
+ for (i = 0; i < single_dpm_table->count; i++)
+ size += sprintf(buf + size, "%d: %uMhz %s\n",
+ i, single_dpm_table->dpm_levels[i].value,
+ (single_dpm_table->dpm_levels[i].value == now / 100)
+ ? "*" : "");
+ break;
+
+ case PP_DCEFCLK:
+ ret = smu_get_current_clk_freq(smu, PPCLK_DCEFCLK, &now);
+ if (ret) {
+ pr_err("Attempt to get current dcefclk Failed!");
+ return ret;
+ }
+
+ single_dpm_table = &(dpm_table->dcef_table);
+ ret = vega20_get_clk_table(smu, &clocks, single_dpm_table);
+ if (ret) {
+ pr_err("Attempt to get dcefclk levels Failed!");
+ return ret;
+ }
+
+ for (i = 0; i < clocks.num_levels; i++)
+ size += sprintf(buf + size, "%d: %uMhz %s\n",
+ i, clocks.data[i].clocks_in_khz / 1000,
+ (clocks.data[i].clocks_in_khz == now * 10) ? "*" : "");
+ break;
+
+ case PP_PCIE:
+ gen_speed = (RREG32_PCIE(smnPCIE_LC_SPEED_CNTL) &
+ PSWUSP0_PCIE_LC_SPEED_CNTL__LC_CURRENT_DATA_RATE_MASK)
+ >> PSWUSP0_PCIE_LC_SPEED_CNTL__LC_CURRENT_DATA_RATE__SHIFT;
+ lane_width = (RREG32_PCIE(smnPCIE_LC_LINK_WIDTH_CNTL) &
+ PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_RD_MASK)
+ >> PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_RD__SHIFT;
+ for (i = 0; i < NUM_LINK_LEVELS; i++)
+ size += sprintf(buf + size, "%d: %s %s %dMhz %s\n", i,
+ (pptable->PcieGenSpeed[i] == 0) ? "2.5GT/s," :
+ (pptable->PcieGenSpeed[i] == 1) ? "5.0GT/s," :
+ (pptable->PcieGenSpeed[i] == 2) ? "8.0GT/s," :
+ (pptable->PcieGenSpeed[i] == 3) ? "16.0GT/s," : "",
+ (pptable->PcieLaneCount[i] == 1) ? "x1" :
+ (pptable->PcieLaneCount[i] == 2) ? "x2" :
+ (pptable->PcieLaneCount[i] == 3) ? "x4" :
+ (pptable->PcieLaneCount[i] == 4) ? "x8" :
+ (pptable->PcieLaneCount[i] == 5) ? "x12" :
+ (pptable->PcieLaneCount[i] == 6) ? "x16" : "",
+ pptable->LclkFreq[i],
+ (gen_speed == pptable->PcieGenSpeed[i]) &&
+ (lane_width == pptable->PcieLaneCount[i]) ?
+ "*" : "");
+ break;
+
+ case OD_SCLK:
+ if (od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMIN].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMAX].feature_id) {
+ size = sprintf(buf, "%s:\n", "OD_SCLK");
+ size += sprintf(buf + size, "0: %10uMhz\n",
+ od_table->GfxclkFmin);
+ size += sprintf(buf + size, "1: %10uMhz\n",
+ od_table->GfxclkFmax);
+ }
+
+ break;
+
+ case OD_MCLK:
+ if (od8_settings->od8_settings_array[OD8_SETTING_UCLK_FMAX].feature_id) {
+ size = sprintf(buf, "%s:\n", "OD_MCLK");
+ size += sprintf(buf + size, "1: %10uMhz\n",
+ od_table->UclkFmax);
+ }
+
+ break;
+
+ case OD_VDDC_CURVE:
+ if (od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ1].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ2].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ3].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE1].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE2].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE3].feature_id) {
+ size = sprintf(buf, "%s:\n", "OD_VDDC_CURVE");
+ size += sprintf(buf + size, "0: %10uMhz %10dmV\n",
+ od_table->GfxclkFreq1,
+ od_table->GfxclkVolt1 / VOLTAGE_SCALE);
+ size += sprintf(buf + size, "1: %10uMhz %10dmV\n",
+ od_table->GfxclkFreq2,
+ od_table->GfxclkVolt2 / VOLTAGE_SCALE);
+ size += sprintf(buf + size, "2: %10uMhz %10dmV\n",
+ od_table->GfxclkFreq3,
+ od_table->GfxclkVolt3 / VOLTAGE_SCALE);
+ }
+
+ break;
+
+ case OD_RANGE:
+ size = sprintf(buf, "%s:\n", "OD_RANGE");
+
+ if (od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMIN].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMAX].feature_id) {
+ size += sprintf(buf + size, "SCLK: %7uMhz %10uMhz\n",
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMIN].min_value,
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMAX].max_value);
+ }
+
+ if (od8_settings->od8_settings_array[OD8_SETTING_UCLK_FMAX].feature_id) {
+ single_dpm_table = &(dpm_table->mem_table);
+ ret = vega20_get_clk_table(smu, &clocks, single_dpm_table);
+ if (ret) {
+ pr_err("Attempt to get memory clk levels Failed!");
+ return ret;
+ }
+
+ size += sprintf(buf + size, "MCLK: %7uMhz %10uMhz\n",
+ clocks.data[0].clocks_in_khz / 1000,
+ od8_settings->od8_settings_array[OD8_SETTING_UCLK_FMAX].max_value);
+ }
+
+ if (od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ1].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ2].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ3].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE1].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE2].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE3].feature_id) {
+ size += sprintf(buf + size, "VDDC_CURVE_SCLK[0]: %7uMhz %10uMhz\n",
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ1].min_value,
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ1].max_value);
+ size += sprintf(buf + size, "VDDC_CURVE_VOLT[0]: %7dmV %11dmV\n",
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE1].min_value,
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE1].max_value);
+ size += sprintf(buf + size, "VDDC_CURVE_SCLK[1]: %7uMhz %10uMhz\n",
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ2].min_value,
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ2].max_value);
+ size += sprintf(buf + size, "VDDC_CURVE_VOLT[1]: %7dmV %11dmV\n",
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE2].min_value,
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE2].max_value);
+ size += sprintf(buf + size, "VDDC_CURVE_SCLK[2]: %7uMhz %10uMhz\n",
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ3].min_value,
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ3].max_value);
+ size += sprintf(buf + size, "VDDC_CURVE_VOLT[2]: %7dmV %11dmV\n",
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE3].min_value,
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE3].max_value);
+ }
+
+ break;
+
+ default:
+ break;
+ }
+ return size;
+}
+
+static int vega20_upload_dpm_level(struct smu_context *smu, bool max,
+ uint32_t feature_mask)
+{
+ struct vega20_dpm_table *dpm_table;
+ struct vega20_single_dpm_table *single_dpm_table;
+ uint32_t freq;
+ int ret = 0;
+
+ dpm_table = smu->smu_dpm.dpm_context;
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_GFXCLK_BIT) &&
+ (feature_mask & FEATURE_DPM_GFXCLK_MASK)) {
+ single_dpm_table = &(dpm_table->gfx_table);
+ freq = max ? single_dpm_table->dpm_state.soft_max_level :
+ single_dpm_table->dpm_state.soft_min_level;
+ ret = smu_send_smc_msg_with_param(smu,
+ (max ? SMU_MSG_SetSoftMaxByFreq : SMU_MSG_SetSoftMinByFreq),
+ (PPCLK_GFXCLK << 16) | (freq & 0xffff));
+ if (ret) {
+ pr_err("Failed to set soft %s gfxclk !\n",
+ max ? "max" : "min");
+ return ret;
+ }
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_UCLK_BIT) &&
+ (feature_mask & FEATURE_DPM_UCLK_MASK)) {
+ single_dpm_table = &(dpm_table->mem_table);
+ freq = max ? single_dpm_table->dpm_state.soft_max_level :
+ single_dpm_table->dpm_state.soft_min_level;
+ ret = smu_send_smc_msg_with_param(smu,
+ (max ? SMU_MSG_SetSoftMaxByFreq : SMU_MSG_SetSoftMinByFreq),
+ (PPCLK_UCLK << 16) | (freq & 0xffff));
+ if (ret) {
+ pr_err("Failed to set soft %s memclk !\n",
+ max ? "max" : "min");
+ return ret;
+ }
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_SOCCLK_BIT) &&
+ (feature_mask & FEATURE_DPM_SOCCLK_MASK)) {
+ single_dpm_table = &(dpm_table->soc_table);
+ freq = max ? single_dpm_table->dpm_state.soft_max_level :
+ single_dpm_table->dpm_state.soft_min_level;
+ ret = smu_send_smc_msg_with_param(smu,
+ (max ? SMU_MSG_SetSoftMaxByFreq : SMU_MSG_SetSoftMinByFreq),
+ (PPCLK_SOCCLK << 16) | (freq & 0xffff));
+ if (ret) {
+ pr_err("Failed to set soft %s socclk !\n",
+ max ? "max" : "min");
+ return ret;
+ }
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_FCLK_BIT) &&
+ (feature_mask & FEATURE_DPM_FCLK_MASK)) {
+ single_dpm_table = &(dpm_table->fclk_table);
+ freq = max ? single_dpm_table->dpm_state.soft_max_level :
+ single_dpm_table->dpm_state.soft_min_level;
+ ret = smu_send_smc_msg_with_param(smu,
+ (max ? SMU_MSG_SetSoftMaxByFreq : SMU_MSG_SetSoftMinByFreq),
+ (PPCLK_FCLK << 16) | (freq & 0xffff));
+ if (ret) {
+ pr_err("Failed to set soft %s fclk !\n",
+ max ? "max" : "min");
+ return ret;
+ }
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_DCEFCLK_BIT) &&
+ (feature_mask & FEATURE_DPM_DCEFCLK_MASK)) {
+ single_dpm_table = &(dpm_table->dcef_table);
+ freq = single_dpm_table->dpm_state.hard_min_level;
+ if (!max) {
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetHardMinByFreq,
+ (PPCLK_DCEFCLK << 16) | (freq & 0xffff));
+ if (ret) {
+ pr_err("Failed to set hard min dcefclk !\n");
+ return ret;
+ }
+ }
+ }
+
+ return ret;
+}
+
+static int vega20_force_clk_levels(struct smu_context *smu,
+ enum pp_clock_type type, uint32_t mask)
+{
+ struct vega20_dpm_table *dpm_table;
+ struct vega20_single_dpm_table *single_dpm_table;
+ uint32_t soft_min_level, soft_max_level, hard_min_level;
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+ int ret = 0;
+
+ if (smu_dpm->dpm_level != AMD_DPM_FORCED_LEVEL_MANUAL) {
+ pr_info("force clock level is for dpm manual mode only.\n");
+ return -EINVAL;
+ }
+
+ mutex_lock(&(smu->mutex));
+
+ soft_min_level = mask ? (ffs(mask) - 1) : 0;
+ soft_max_level = mask ? (fls(mask) - 1) : 0;
+
+ dpm_table = smu->smu_dpm.dpm_context;
+
+ switch (type) {
+ case PP_SCLK:
+ single_dpm_table = &(dpm_table->gfx_table);
+
+ if (soft_max_level >= single_dpm_table->count) {
+ pr_err("Clock level specified %d is over max allowed %d\n",
+ soft_max_level, single_dpm_table->count - 1);
+ ret = -EINVAL;
+ break;
+ }
+
+ single_dpm_table->dpm_state.soft_min_level =
+ single_dpm_table->dpm_levels[soft_min_level].value;
+ single_dpm_table->dpm_state.soft_max_level =
+ single_dpm_table->dpm_levels[soft_max_level].value;
+
+ ret = vega20_upload_dpm_level(smu, false, FEATURE_DPM_GFXCLK_MASK);
+ if (ret) {
+ pr_err("Failed to upload boot level to lowest!\n");
+ break;
+ }
+
+ ret = vega20_upload_dpm_level(smu, true, FEATURE_DPM_GFXCLK_MASK);
+ if (ret)
+ pr_err("Failed to upload dpm max level to highest!\n");
+
+ break;
+
+ case PP_MCLK:
+ single_dpm_table = &(dpm_table->mem_table);
+
+ if (soft_max_level >= single_dpm_table->count) {
+ pr_err("Clock level specified %d is over max allowed %d\n",
+ soft_max_level, single_dpm_table->count - 1);
+ ret = -EINVAL;
+ break;
+ }
+
+ single_dpm_table->dpm_state.soft_min_level =
+ single_dpm_table->dpm_levels[soft_min_level].value;
+ single_dpm_table->dpm_state.soft_max_level =
+ single_dpm_table->dpm_levels[soft_max_level].value;
+
+ ret = vega20_upload_dpm_level(smu, false, FEATURE_DPM_UCLK_MASK);
+ if (ret) {
+ pr_err("Failed to upload boot level to lowest!\n");
+ break;
+ }
+
+ ret = vega20_upload_dpm_level(smu, true, FEATURE_DPM_UCLK_MASK);
+ if (ret)
+ pr_err("Failed to upload dpm max level to highest!\n");
+
+ break;
+
+ case PP_SOCCLK:
+ single_dpm_table = &(dpm_table->soc_table);
+
+ if (soft_max_level >= single_dpm_table->count) {
+ pr_err("Clock level specified %d is over max allowed %d\n",
+ soft_max_level, single_dpm_table->count - 1);
+ ret = -EINVAL;
+ break;
+ }
+
+ single_dpm_table->dpm_state.soft_min_level =
+ single_dpm_table->dpm_levels[soft_min_level].value;
+ single_dpm_table->dpm_state.soft_max_level =
+ single_dpm_table->dpm_levels[soft_max_level].value;
+
+ ret = vega20_upload_dpm_level(smu, false, FEATURE_DPM_SOCCLK_MASK);
+ if (ret) {
+ pr_err("Failed to upload boot level to lowest!\n");
+ break;
+ }
+
+ ret = vega20_upload_dpm_level(smu, true, FEATURE_DPM_SOCCLK_MASK);
+ if (ret)
+ pr_err("Failed to upload dpm max level to highest!\n");
+
+ break;
+
+ case PP_FCLK:
+ single_dpm_table = &(dpm_table->fclk_table);
+
+ if (soft_max_level >= single_dpm_table->count) {
+ pr_err("Clock level specified %d is over max allowed %d\n",
+ soft_max_level, single_dpm_table->count - 1);
+ ret = -EINVAL;
+ break;
+ }
+
+ single_dpm_table->dpm_state.soft_min_level =
+ single_dpm_table->dpm_levels[soft_min_level].value;
+ single_dpm_table->dpm_state.soft_max_level =
+ single_dpm_table->dpm_levels[soft_max_level].value;
+
+ ret = vega20_upload_dpm_level(smu, false, FEATURE_DPM_FCLK_MASK);
+ if (ret) {
+ pr_err("Failed to upload boot level to lowest!\n");
+ break;
+ }
+
+ ret = vega20_upload_dpm_level(smu, true, FEATURE_DPM_FCLK_MASK);
+ if (ret)
+ pr_err("Failed to upload dpm max level to highest!\n");
+
+ break;
+
+ case PP_DCEFCLK:
+ hard_min_level = soft_min_level;
+ single_dpm_table = &(dpm_table->dcef_table);
+
+ if (hard_min_level >= single_dpm_table->count) {
+ pr_err("Clock level specified %d is over max allowed %d\n",
+ hard_min_level, single_dpm_table->count - 1);
+ ret = -EINVAL;
+ break;
+ }
+
+ single_dpm_table->dpm_state.hard_min_level =
+ single_dpm_table->dpm_levels[hard_min_level].value;
+
+ ret = vega20_upload_dpm_level(smu, false, FEATURE_DPM_DCEFCLK_MASK);
+ if (ret)
+ pr_err("Failed to upload boot level to lowest!\n");
+
+ break;
+
+ case PP_PCIE:
+ if (soft_min_level >= NUM_LINK_LEVELS ||
+ soft_max_level >= NUM_LINK_LEVELS) {
+ ret = -EINVAL;
+ break;
+ }
+
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetMinLinkDpmByIndex, soft_min_level);
+ if (ret)
+ pr_err("Failed to set min link dpm level!\n");
+
+ break;
+
+ default:
+ break;
+ }
+
+ mutex_unlock(&(smu->mutex));
+ return ret;
+}
+
+static int vega20_get_clock_by_type_with_latency(struct smu_context *smu,
+ enum amd_pp_clock_type type,
+ struct pp_clock_levels_with_latency *clocks)
+{
+ int ret;
+ struct vega20_single_dpm_table *single_dpm_table;
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+ struct vega20_dpm_table *dpm_table = NULL;
+
+ dpm_table = smu_dpm->dpm_context;
+
+ mutex_lock(&smu->mutex);
+
+ switch (type) {
+ case amd_pp_sys_clock:
+ single_dpm_table = &(dpm_table->gfx_table);
+ ret = vega20_get_clk_table(smu, clocks, single_dpm_table);
+ break;
+ case amd_pp_mem_clock:
+ single_dpm_table = &(dpm_table->mem_table);
+ ret = vega20_get_clk_table(smu, clocks, single_dpm_table);
+ break;
+ case amd_pp_dcef_clock:
+ single_dpm_table = &(dpm_table->dcef_table);
+ ret = vega20_get_clk_table(smu, clocks, single_dpm_table);
+ break;
+ case amd_pp_soc_clock:
+ single_dpm_table = &(dpm_table->soc_table);
+ ret = vega20_get_clk_table(smu, clocks, single_dpm_table);
+ break;
+ default:
+ ret = -EINVAL;
+ }
+
+ mutex_unlock(&smu->mutex);
+ return ret;
+}
+
+static int vega20_overdrive_get_gfx_clk_base_voltage(struct smu_context *smu,
+ uint32_t *voltage,
+ uint32_t freq)
+{
+ int ret;
+
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_GetAVFSVoltageByDpm,
+ ((AVFS_CURVE << 24) | (OD8_HOTCURVE_TEMPERATURE << 16) | freq));
+ if (ret) {
+ pr_err("[GetBaseVoltage] failed to get GFXCLK AVFS voltage from SMU!");
+ return ret;
+ }
+
+ smu_read_smc_arg(smu, voltage);
+ *voltage = *voltage / VOLTAGE_SCALE;
+
+ return 0;
+}
+
+static int vega20_set_default_od8_setttings(struct smu_context *smu)
+{
+ struct smu_table_context *table_context = &smu->smu_table;
+ OverDriveTable_t *od_table = (OverDriveTable_t *)(table_context->overdrive_table);
+ struct vega20_od8_settings *od8_settings = NULL;
+ PPTable_t *smc_pptable = table_context->driver_pptable;
+ int i, ret;
+
+ if (table_context->od8_settings)
+ return -EINVAL;
+
+ table_context->od8_settings = kzalloc(sizeof(struct vega20_od8_settings), GFP_KERNEL);
+
+ if (!table_context->od8_settings)
+ return -ENOMEM;
+
+ memset(table_context->od8_settings, 0, sizeof(struct vega20_od8_settings));
+ od8_settings = (struct vega20_od8_settings *)table_context->od8_settings;
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_SOCCLK_BIT)) {
+ if (table_context->od_feature_capabilities[ATOM_VEGA20_ODFEATURE_GFXCLK_LIMITS] &&
+ table_context->od_settings_max[OD8_SETTING_GFXCLK_FMAX] > 0 &&
+ table_context->od_settings_min[OD8_SETTING_GFXCLK_FMIN] > 0 &&
+ (table_context->od_settings_max[OD8_SETTING_GFXCLK_FMAX] >=
+ table_context->od_settings_min[OD8_SETTING_GFXCLK_FMIN])) {
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMIN].feature_id =
+ OD8_GFXCLK_LIMITS;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMAX].feature_id =
+ OD8_GFXCLK_LIMITS;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMIN].default_value =
+ od_table->GfxclkFmin;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMAX].default_value =
+ od_table->GfxclkFmax;
+ }
+
+ if (table_context->od_feature_capabilities[ATOM_VEGA20_ODFEATURE_GFXCLK_CURVE] &&
+ (table_context->od_settings_min[OD8_SETTING_GFXCLK_VOLTAGE1] >=
+ smc_pptable->MinVoltageGfx / VOLTAGE_SCALE) &&
+ (table_context->od_settings_max[OD8_SETTING_GFXCLK_VOLTAGE3] <=
+ smc_pptable->MaxVoltageGfx / VOLTAGE_SCALE) &&
+ (table_context->od_settings_min[OD8_SETTING_GFXCLK_VOLTAGE1] <=
+ table_context->od_settings_max[OD8_SETTING_GFXCLK_VOLTAGE3])) {
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ1].feature_id =
+ OD8_GFXCLK_CURVE;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE1].feature_id =
+ OD8_GFXCLK_CURVE;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ2].feature_id =
+ OD8_GFXCLK_CURVE;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE2].feature_id =
+ OD8_GFXCLK_CURVE;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ3].feature_id =
+ OD8_GFXCLK_CURVE;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE3].feature_id =
+ OD8_GFXCLK_CURVE;
+
+ od_table->GfxclkFreq1 = od_table->GfxclkFmin;
+ od_table->GfxclkFreq2 = (od_table->GfxclkFmin + od_table->GfxclkFmax) / 2;
+ od_table->GfxclkFreq3 = od_table->GfxclkFmax;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ1].default_value =
+ od_table->GfxclkFreq1;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ2].default_value =
+ od_table->GfxclkFreq2;
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ3].default_value =
+ od_table->GfxclkFreq3;
+
+ ret = vega20_overdrive_get_gfx_clk_base_voltage(smu,
+ &od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE1].default_value,
+ od_table->GfxclkFreq1);
+ if (ret)
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE1].default_value = 0;
+ od_table->GfxclkVolt1 =
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE1].default_value
+ * VOLTAGE_SCALE;
+ ret = vega20_overdrive_get_gfx_clk_base_voltage(smu,
+ &od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE2].default_value,
+ od_table->GfxclkFreq2);
+ if (ret)
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE2].default_value = 0;
+ od_table->GfxclkVolt2 =
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE2].default_value
+ * VOLTAGE_SCALE;
+ ret = vega20_overdrive_get_gfx_clk_base_voltage(smu,
+ &od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE3].default_value,
+ od_table->GfxclkFreq3);
+ if (ret)
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE3].default_value = 0;
+ od_table->GfxclkVolt3 =
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE3].default_value
+ * VOLTAGE_SCALE;
+ }
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_UCLK_BIT)) {
+ if (table_context->od_feature_capabilities[ATOM_VEGA20_ODFEATURE_UCLK_MAX] &&
+ table_context->od_settings_min[OD8_SETTING_UCLK_FMAX] > 0 &&
+ table_context->od_settings_max[OD8_SETTING_UCLK_FMAX] > 0 &&
+ (table_context->od_settings_max[OD8_SETTING_UCLK_FMAX] >=
+ table_context->od_settings_min[OD8_SETTING_UCLK_FMAX])) {
+ od8_settings->od8_settings_array[OD8_SETTING_UCLK_FMAX].feature_id =
+ OD8_UCLK_MAX;
+ od8_settings->od8_settings_array[OD8_SETTING_UCLK_FMAX].default_value =
+ od_table->UclkFmax;
+ }
+ }
+
+ if (table_context->od_feature_capabilities[ATOM_VEGA20_ODFEATURE_POWER_LIMIT] &&
+ table_context->od_settings_min[OD8_SETTING_POWER_PERCENTAGE] > 0 &&
+ table_context->od_settings_min[OD8_SETTING_POWER_PERCENTAGE] <= 100 &&
+ table_context->od_settings_max[OD8_SETTING_POWER_PERCENTAGE] > 0 &&
+ table_context->od_settings_max[OD8_SETTING_POWER_PERCENTAGE] <= 100) {
+ od8_settings->od8_settings_array[OD8_SETTING_POWER_PERCENTAGE].feature_id =
+ OD8_POWER_LIMIT;
+ od8_settings->od8_settings_array[OD8_SETTING_POWER_PERCENTAGE].default_value =
+ od_table->OverDrivePct;
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_FAN_CONTROL_BIT)) {
+ if (table_context->od_feature_capabilities[ATOM_VEGA20_ODFEATURE_FAN_ACOUSTIC_LIMIT] &&
+ table_context->od_settings_min[OD8_SETTING_FAN_ACOUSTIC_LIMIT] > 0 &&
+ table_context->od_settings_max[OD8_SETTING_FAN_ACOUSTIC_LIMIT] > 0 &&
+ (table_context->od_settings_max[OD8_SETTING_FAN_ACOUSTIC_LIMIT] >=
+ table_context->od_settings_min[OD8_SETTING_FAN_ACOUSTIC_LIMIT])) {
+ od8_settings->od8_settings_array[OD8_SETTING_FAN_ACOUSTIC_LIMIT].feature_id =
+ OD8_ACOUSTIC_LIMIT_SCLK;
+ od8_settings->od8_settings_array[OD8_SETTING_FAN_ACOUSTIC_LIMIT].default_value =
+ od_table->FanMaximumRpm;
+ }
+
+ if (table_context->od_feature_capabilities[ATOM_VEGA20_ODFEATURE_FAN_SPEED_MIN] &&
+ table_context->od_settings_min[OD8_SETTING_FAN_MIN_SPEED] > 0 &&
+ table_context->od_settings_max[OD8_SETTING_FAN_MIN_SPEED] > 0 &&
+ (table_context->od_settings_max[OD8_SETTING_FAN_MIN_SPEED] >=
+ table_context->od_settings_min[OD8_SETTING_FAN_MIN_SPEED])) {
+ od8_settings->od8_settings_array[OD8_SETTING_FAN_MIN_SPEED].feature_id =
+ OD8_FAN_SPEED_MIN;
+ od8_settings->od8_settings_array[OD8_SETTING_FAN_MIN_SPEED].default_value =
+ od_table->FanMinimumPwm * smc_pptable->FanMaximumRpm / 100;
+ }
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_THERMAL_BIT)) {
+ if (table_context->od_feature_capabilities[ATOM_VEGA20_ODFEATURE_TEMPERATURE_FAN] &&
+ table_context->od_settings_min[OD8_SETTING_FAN_TARGET_TEMP] > 0 &&
+ table_context->od_settings_max[OD8_SETTING_FAN_TARGET_TEMP] > 0 &&
+ (table_context->od_settings_max[OD8_SETTING_FAN_TARGET_TEMP] >=
+ table_context->od_settings_min[OD8_SETTING_FAN_TARGET_TEMP])) {
+ od8_settings->od8_settings_array[OD8_SETTING_FAN_TARGET_TEMP].feature_id =
+ OD8_TEMPERATURE_FAN;
+ od8_settings->od8_settings_array[OD8_SETTING_FAN_TARGET_TEMP].default_value =
+ od_table->FanTargetTemperature;
+ }
+
+ if (table_context->od_feature_capabilities[ATOM_VEGA20_ODFEATURE_TEMPERATURE_SYSTEM] &&
+ table_context->od_settings_min[OD8_SETTING_OPERATING_TEMP_MAX] > 0 &&
+ table_context->od_settings_max[OD8_SETTING_OPERATING_TEMP_MAX] > 0 &&
+ (table_context->od_settings_max[OD8_SETTING_OPERATING_TEMP_MAX] >=
+ table_context->od_settings_min[OD8_SETTING_OPERATING_TEMP_MAX])) {
+ od8_settings->od8_settings_array[OD8_SETTING_OPERATING_TEMP_MAX].feature_id =
+ OD8_TEMPERATURE_SYSTEM;
+ od8_settings->od8_settings_array[OD8_SETTING_OPERATING_TEMP_MAX].default_value =
+ od_table->MaxOpTemp;
+ }
+ }
+
+ for (i = 0; i < OD8_SETTING_COUNT; i++) {
+ if (od8_settings->od8_settings_array[i].feature_id) {
+ od8_settings->od8_settings_array[i].min_value =
+ table_context->od_settings_min[i];
+ od8_settings->od8_settings_array[i].max_value =
+ table_context->od_settings_max[i];
+ od8_settings->od8_settings_array[i].current_value =
+ od8_settings->od8_settings_array[i].default_value;
+ } else {
+ od8_settings->od8_settings_array[i].min_value = 0;
+ od8_settings->od8_settings_array[i].max_value = 0;
+ od8_settings->od8_settings_array[i].current_value = 0;
+ }
+ }
+
+ return 0;
+}
+
+static int vega20_get_od_percentage(struct smu_context *smu,
+ enum pp_clock_type type)
+{
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+ struct vega20_dpm_table *dpm_table = NULL;
+ struct vega20_dpm_table *golden_table = NULL;
+ struct vega20_single_dpm_table *single_dpm_table;
+ struct vega20_single_dpm_table *golden_dpm_table;
+ int value, golden_value;
+
+ dpm_table = smu_dpm->dpm_context;
+ golden_table = smu_dpm->golden_dpm_context;
+
+ switch (type) {
+ case OD_SCLK:
+ single_dpm_table = &(dpm_table->gfx_table);
+ golden_dpm_table = &(golden_table->gfx_table);
+ break;
+ case OD_MCLK:
+ single_dpm_table = &(dpm_table->mem_table);
+ golden_dpm_table = &(golden_table->mem_table);
+ break;
+ default:
+ return -EINVAL;
+ break;
+ }
+
+ value = single_dpm_table->dpm_levels[single_dpm_table->count - 1].value;
+ golden_value = golden_dpm_table->dpm_levels[golden_dpm_table->count - 1].value;
+
+ value -= golden_value;
+ value = DIV_ROUND_UP(value * 100, golden_value);
+
+ return value;
+}
+
+static int
+vega20_get_profiling_clk_mask(struct smu_context *smu,
+ enum amd_dpm_forced_level level,
+ uint32_t *sclk_mask,
+ uint32_t *mclk_mask,
+ uint32_t *soc_mask)
+{
+ struct vega20_dpm_table *dpm_table = (struct vega20_dpm_table *)smu->smu_dpm.dpm_context;
+ struct vega20_single_dpm_table *gfx_dpm_table;
+ struct vega20_single_dpm_table *mem_dpm_table;
+ struct vega20_single_dpm_table *soc_dpm_table;
+
+ if (!smu->smu_dpm.dpm_context)
+ return -EINVAL;
+
+ gfx_dpm_table = &dpm_table->gfx_table;
+ mem_dpm_table = &dpm_table->mem_table;
+ soc_dpm_table = &dpm_table->soc_table;
+
+ *sclk_mask = 0;
+ *mclk_mask = 0;
+ *soc_mask = 0;
+
+ if (gfx_dpm_table->count > VEGA20_UMD_PSTATE_GFXCLK_LEVEL &&
+ mem_dpm_table->count > VEGA20_UMD_PSTATE_MCLK_LEVEL &&
+ soc_dpm_table->count > VEGA20_UMD_PSTATE_SOCCLK_LEVEL) {
+ *sclk_mask = VEGA20_UMD_PSTATE_GFXCLK_LEVEL;
+ *mclk_mask = VEGA20_UMD_PSTATE_MCLK_LEVEL;
+ *soc_mask = VEGA20_UMD_PSTATE_SOCCLK_LEVEL;
+ }
+
+ if (level == AMD_DPM_FORCED_LEVEL_PROFILE_MIN_SCLK) {
+ *sclk_mask = 0;
+ } else if (level == AMD_DPM_FORCED_LEVEL_PROFILE_MIN_MCLK) {
+ *mclk_mask = 0;
+ } else if (level == AMD_DPM_FORCED_LEVEL_PROFILE_PEAK) {
+ *sclk_mask = gfx_dpm_table->count - 1;
+ *mclk_mask = mem_dpm_table->count - 1;
+ *soc_mask = soc_dpm_table->count - 1;
+ }
+
+ return 0;
+}
+
+static int
+vega20_set_uclk_to_highest_dpm_level(struct smu_context *smu,
+ struct vega20_single_dpm_table *dpm_table)
+{
+ int ret = 0;
+ struct smu_dpm_context *smu_dpm_ctx = &(smu->smu_dpm);
+ if (!smu_dpm_ctx->dpm_context)
+ return -EINVAL;
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_UCLK_BIT)) {
+ if (dpm_table->count <= 0) {
+ pr_err("[%s] Dpm table has no entry!", __func__);
+ return -EINVAL;
+ }
+
+ if (dpm_table->count > NUM_UCLK_DPM_LEVELS) {
+ pr_err("[%s] Dpm table has too many entries!", __func__);
+ return -EINVAL;
+ }
+
+ dpm_table->dpm_state.hard_min_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetHardMinByFreq,
+ (PPCLK_UCLK << 16) | dpm_table->dpm_state.hard_min_level);
+ if (ret) {
+ pr_err("[%s] Set hard min uclk failed!", __func__);
+ return ret;
+ }
+ }
+
+ return ret;
+}
+
+static int vega20_pre_display_config_changed(struct smu_context *smu)
+{
+ int ret = 0;
+ struct vega20_dpm_table *dpm_table = smu->smu_dpm.dpm_context;
+
+ if (!smu->smu_dpm.dpm_context)
+ return -EINVAL;
+
+ smu_send_smc_msg_with_param(smu, SMU_MSG_NumOfDisplays, 0);
+ ret = vega20_set_uclk_to_highest_dpm_level(smu,
+ &dpm_table->mem_table);
+ if (ret)
+ pr_err("Failed to set uclk to highest dpm level");
+ return ret;
+}
+
+static int vega20_display_config_changed(struct smu_context *smu)
+{
+ int ret = 0;
+
+ if (!smu->funcs)
+ return -EINVAL;
+
+ if (!smu->smu_dpm.dpm_context ||
+ !smu->smu_table.tables ||
+ !smu->smu_table.tables[TABLE_WATERMARKS].cpu_addr)
+ return -EINVAL;
+
+ if ((smu->watermarks_bitmap & WATERMARKS_EXIST) &&
+ !(smu->watermarks_bitmap & WATERMARKS_LOADED)) {
+ ret = smu->funcs->write_watermarks_table(smu);
+ if (ret) {
+ pr_err("Failed to update WMTABLE!");
+ return ret;
+ }
+ smu->watermarks_bitmap |= WATERMARKS_LOADED;
+ }
+
+ if ((smu->watermarks_bitmap & WATERMARKS_EXIST) &&
+ smu_feature_is_supported(smu, FEATURE_DPM_DCEFCLK_BIT) &&
+ smu_feature_is_supported(smu, FEATURE_DPM_SOCCLK_BIT)) {
+ smu_send_smc_msg_with_param(smu,
+ SMU_MSG_NumOfDisplays,
+ smu->display_config->num_display);
+ }
+
+ return ret;
+}
+
+static int vega20_apply_clocks_adjust_rules(struct smu_context *smu)
+{
+ struct smu_dpm_context *smu_dpm_ctx = &(smu->smu_dpm);
+ struct vega20_dpm_table *dpm_ctx = (struct vega20_dpm_table *)(smu_dpm_ctx->dpm_context);
+ struct vega20_single_dpm_table *dpm_table;
+ bool vblank_too_short = false;
+ bool disable_mclk_switching;
+ uint32_t i, latency;
+
+ disable_mclk_switching = ((1 < smu->display_config->num_display) &&
+ !smu->display_config->multi_monitor_in_sync) || vblank_too_short;
+ latency = smu->display_config->dce_tolerable_mclk_in_active_latency;
+
+ /* gfxclk */
+ dpm_table = &(dpm_ctx->gfx_table);
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.hard_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.hard_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+
+ if (VEGA20_UMD_PSTATE_GFXCLK_LEVEL < dpm_table->count) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_GFXCLK_LEVEL].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_GFXCLK_LEVEL].value;
+ }
+
+ if (smu_dpm_ctx->dpm_level == AMD_DPM_FORCED_LEVEL_PROFILE_MIN_SCLK) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[0].value;
+ }
+
+ if (smu_dpm_ctx->dpm_level == AMD_DPM_FORCED_LEVEL_PROFILE_PEAK) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ }
+
+ /* memclk */
+ dpm_table = &(dpm_ctx->mem_table);
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.hard_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.hard_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+
+ if (VEGA20_UMD_PSTATE_MCLK_LEVEL < dpm_table->count) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_MCLK_LEVEL].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_MCLK_LEVEL].value;
+ }
+
+ if (smu_dpm_ctx->dpm_level == AMD_DPM_FORCED_LEVEL_PROFILE_MIN_MCLK) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[0].value;
+ }
+
+ if (smu_dpm_ctx->dpm_level == AMD_DPM_FORCED_LEVEL_PROFILE_PEAK) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ }
+
+ /* honour DAL's UCLK Hardmin */
+ if (dpm_table->dpm_state.hard_min_level < (smu->display_config->min_mem_set_clock / 100))
+ dpm_table->dpm_state.hard_min_level = smu->display_config->min_mem_set_clock / 100;
+
+ /* Hardmin is dependent on displayconfig */
+ if (disable_mclk_switching) {
+ dpm_table->dpm_state.hard_min_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ for (i = 0; i < smu_dpm_ctx->mclk_latency_table->count - 1; i++) {
+ if (smu_dpm_ctx->mclk_latency_table->entries[i].latency <= latency) {
+ if (dpm_table->dpm_levels[i].value >= (smu->display_config->min_mem_set_clock / 100)) {
+ dpm_table->dpm_state.hard_min_level = dpm_table->dpm_levels[i].value;
+ break;
+ }
+ }
+ }
+ }
+
+ if (smu->display_config->nb_pstate_switch_disable)
+ dpm_table->dpm_state.hard_min_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+
+ /* vclk */
+ dpm_table = &(dpm_ctx->vclk_table);
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.hard_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.hard_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+
+ if (VEGA20_UMD_PSTATE_UVDCLK_LEVEL < dpm_table->count) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_UVDCLK_LEVEL].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_UVDCLK_LEVEL].value;
+ }
+
+ if (smu_dpm_ctx->dpm_level == AMD_DPM_FORCED_LEVEL_PROFILE_PEAK) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ }
+
+ /* dclk */
+ dpm_table = &(dpm_ctx->dclk_table);
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.hard_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.hard_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+
+ if (VEGA20_UMD_PSTATE_UVDCLK_LEVEL < dpm_table->count) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_UVDCLK_LEVEL].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_UVDCLK_LEVEL].value;
+ }
+
+ if (smu_dpm_ctx->dpm_level == AMD_DPM_FORCED_LEVEL_PROFILE_PEAK) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ }
+
+ /* socclk */
+ dpm_table = &(dpm_ctx->soc_table);
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.hard_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.hard_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+
+ if (VEGA20_UMD_PSTATE_SOCCLK_LEVEL < dpm_table->count) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_SOCCLK_LEVEL].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_SOCCLK_LEVEL].value;
+ }
+
+ if (smu_dpm_ctx->dpm_level == AMD_DPM_FORCED_LEVEL_PROFILE_PEAK) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ }
+
+ /* eclk */
+ dpm_table = &(dpm_ctx->eclk_table);
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.hard_min_level = dpm_table->dpm_levels[0].value;
+ dpm_table->dpm_state.hard_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+
+ if (VEGA20_UMD_PSTATE_VCEMCLK_LEVEL < dpm_table->count) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_VCEMCLK_LEVEL].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[VEGA20_UMD_PSTATE_VCEMCLK_LEVEL].value;
+ }
+
+ if (smu_dpm_ctx->dpm_level == AMD_DPM_FORCED_LEVEL_PROFILE_PEAK) {
+ dpm_table->dpm_state.soft_min_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ dpm_table->dpm_state.soft_max_level = dpm_table->dpm_levels[dpm_table->count - 1].value;
+ }
+ return 0;
+}
+
+static int
+vega20_notify_smc_dispaly_config(struct smu_context *smu)
+{
+ struct vega20_dpm_table *dpm_table = smu->smu_dpm.dpm_context;
+ struct vega20_single_dpm_table *memtable = &dpm_table->mem_table;
+ struct smu_clocks min_clocks = {0};
+ struct pp_display_clock_request clock_req;
+ int ret = 0;
+
+ min_clocks.dcef_clock = smu->display_config->min_dcef_set_clk;
+ min_clocks.dcef_clock_in_sr = smu->display_config->min_dcef_deep_sleep_set_clk;
+ min_clocks.memory_clock = smu->display_config->min_mem_set_clock;
+
+ if (smu_feature_is_supported(smu, FEATURE_DPM_DCEFCLK_BIT)) {
+ clock_req.clock_type = amd_pp_dcef_clock;
+ clock_req.clock_freq_in_khz = min_clocks.dcef_clock * 10;
+ if (!smu->funcs->display_clock_voltage_request(smu, &clock_req)) {
+ if (smu_feature_is_supported(smu, FEATURE_DS_DCEFCLK_BIT)) {
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetMinDeepSleepDcefclk,
+ min_clocks.dcef_clock_in_sr/100);
+ if (ret) {
+ pr_err("Attempt to set divider for DCEFCLK Failed!");
+ return ret;
+ }
+ }
+ } else {
+ pr_info("Attempt to set Hard Min for DCEFCLK Failed!");
+ }
+ }
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_UCLK_BIT)) {
+ memtable->dpm_state.hard_min_level = min_clocks.memory_clock/100;
+ ret = smu_send_smc_msg_with_param(smu,
+ SMU_MSG_SetHardMinByFreq,
+ (PPCLK_UCLK << 16) | memtable->dpm_state.hard_min_level);
+ if (ret) {
+ pr_err("[%s] Set hard min uclk failed!", __func__);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static uint32_t vega20_find_lowest_dpm_level(struct vega20_single_dpm_table *table)
+{
+ uint32_t i;
+
+ for (i = 0; i < table->count; i++) {
+ if (table->dpm_levels[i].enabled)
+ break;
+ }
+ if (i >= table->count) {
+ i = 0;
+ table->dpm_levels[i].enabled = true;
+ }
+
+ return i;
+}
+
+static uint32_t vega20_find_highest_dpm_level(struct vega20_single_dpm_table *table)
+{
+ int i = 0;
+
+ if (!table) {
+ pr_err("[%s] DPM Table does not exist!", __func__);
+ return 0;
+ }
+ if (table->count <= 0) {
+ pr_err("[%s] DPM Table has no entry!", __func__);
+ return 0;
+ }
+ if (table->count > MAX_REGULAR_DPM_NUMBER) {
+ pr_err("[%s] DPM Table has too many entries!", __func__);
+ return MAX_REGULAR_DPM_NUMBER - 1;
+ }
+
+ for (i = table->count - 1; i >= 0; i--) {
+ if (table->dpm_levels[i].enabled)
+ break;
+ }
+ if (i < 0) {
+ i = 0;
+ table->dpm_levels[i].enabled = true;
+ }
+
+ return i;
+}
+
+static int vega20_force_dpm_limit_value(struct smu_context *smu, bool highest)
+{
+ uint32_t soft_level;
+ int ret = 0;
+ struct vega20_dpm_table *dpm_table =
+ (struct vega20_dpm_table *)smu->smu_dpm.dpm_context;
+
+ if (highest)
+ soft_level = vega20_find_highest_dpm_level(&(dpm_table->gfx_table));
+ else
+ soft_level = vega20_find_lowest_dpm_level(&(dpm_table->gfx_table));
+
+ dpm_table->gfx_table.dpm_state.soft_min_level =
+ dpm_table->gfx_table.dpm_state.soft_max_level =
+ dpm_table->gfx_table.dpm_levels[soft_level].value;
+
+ if (highest)
+ soft_level = vega20_find_highest_dpm_level(&(dpm_table->mem_table));
+ else
+ soft_level = vega20_find_lowest_dpm_level(&(dpm_table->mem_table));
+
+ dpm_table->mem_table.dpm_state.soft_min_level =
+ dpm_table->mem_table.dpm_state.soft_max_level =
+ dpm_table->mem_table.dpm_levels[soft_level].value;
+
+ if (highest)
+ soft_level = vega20_find_highest_dpm_level(&(dpm_table->soc_table));
+ else
+ soft_level = vega20_find_lowest_dpm_level(&(dpm_table->soc_table));
+
+ dpm_table->soc_table.dpm_state.soft_min_level =
+ dpm_table->soc_table.dpm_state.soft_max_level =
+ dpm_table->soc_table.dpm_levels[soft_level].value;
+
+ ret = vega20_upload_dpm_level(smu, false, 0xFFFFFFFF);
+ if (ret) {
+ pr_err("Failed to upload boot level to %s!\n",
+ highest ? "highest" : "lowest");
+ return ret;
+ }
+
+ ret = vega20_upload_dpm_level(smu, true, 0xFFFFFFFF);
+ if (ret) {
+ pr_err("Failed to upload dpm max level to %s!\n!",
+ highest ? "highest" : "lowest");
+ return ret;
+ }
+
+ return ret;
+}
+
+static int vega20_unforce_dpm_levels(struct smu_context *smu)
+{
+ uint32_t soft_min_level, soft_max_level;
+ int ret = 0;
+ struct vega20_dpm_table *dpm_table =
+ (struct vega20_dpm_table *)smu->smu_dpm.dpm_context;
+
+ soft_min_level = vega20_find_lowest_dpm_level(&(dpm_table->gfx_table));
+ soft_max_level = vega20_find_highest_dpm_level(&(dpm_table->gfx_table));
+ dpm_table->gfx_table.dpm_state.soft_min_level =
+ dpm_table->gfx_table.dpm_levels[soft_min_level].value;
+ dpm_table->gfx_table.dpm_state.soft_max_level =
+ dpm_table->gfx_table.dpm_levels[soft_max_level].value;
+
+ soft_min_level = vega20_find_lowest_dpm_level(&(dpm_table->mem_table));
+ soft_max_level = vega20_find_highest_dpm_level(&(dpm_table->mem_table));
+ dpm_table->mem_table.dpm_state.soft_min_level =
+ dpm_table->gfx_table.dpm_levels[soft_min_level].value;
+ dpm_table->mem_table.dpm_state.soft_max_level =
+ dpm_table->gfx_table.dpm_levels[soft_max_level].value;
+
+ soft_min_level = vega20_find_lowest_dpm_level(&(dpm_table->soc_table));
+ soft_max_level = vega20_find_highest_dpm_level(&(dpm_table->soc_table));
+ dpm_table->soc_table.dpm_state.soft_min_level =
+ dpm_table->soc_table.dpm_levels[soft_min_level].value;
+ dpm_table->soc_table.dpm_state.soft_max_level =
+ dpm_table->soc_table.dpm_levels[soft_max_level].value;
+
+ ret = smu_upload_dpm_level(smu, false, 0xFFFFFFFF);
+ if (ret) {
+ pr_err("Failed to upload DPM Bootup Levels!");
+ return ret;
+ }
+
+ ret = smu_upload_dpm_level(smu, true, 0xFFFFFFFF);
+ if (ret) {
+ pr_err("Failed to upload DPM Max Levels!");
+ return ret;
+ }
+
+ return ret;
+}
+
+static enum amd_dpm_forced_level vega20_get_performance_level(struct smu_context *smu)
+{
+ struct smu_dpm_context *smu_dpm_ctx = &(smu->smu_dpm);
+ if (!smu_dpm_ctx->dpm_context)
+ return -EINVAL;
+
+ if (smu_dpm_ctx->dpm_level != smu_dpm_ctx->saved_dpm_level) {
+ mutex_lock(&(smu->mutex));
+ smu_dpm_ctx->saved_dpm_level = smu_dpm_ctx->dpm_level;
+ mutex_unlock(&(smu->mutex));
+ }
+ return smu_dpm_ctx->dpm_level;
+}
+
+static int
+vega20_force_performance_level(struct smu_context *smu, enum amd_dpm_forced_level level)
+{
+ int ret = 0;
+ int i;
+ struct smu_dpm_context *smu_dpm_ctx = &(smu->smu_dpm);
+
+ if (!smu_dpm_ctx->dpm_context)
+ return -EINVAL;
+
+ for (i = 0; i < smu->adev->num_ip_blocks; i++) {
+ if (smu->adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_SMC)
+ break;
+ }
+
+ mutex_lock(&smu->mutex);
+
+ smu->adev->ip_blocks[i].version->funcs->enable_umd_pstate(smu, &level);
+ ret = smu_handle_task(smu, level,
+ AMD_PP_TASK_READJUST_POWER_STATE);
+
+ mutex_unlock(&smu->mutex);
+
+ return ret;
+}
+
+static int vega20_update_specified_od8_value(struct smu_context *smu,
+ uint32_t index,
+ uint32_t value)
+{
+ struct smu_table_context *table_context = &smu->smu_table;
+ OverDriveTable_t *od_table =
+ (OverDriveTable_t *)(table_context->overdrive_table);
+ struct vega20_od8_settings *od8_settings =
+ (struct vega20_od8_settings *)table_context->od8_settings;
+
+ switch (index) {
+ case OD8_SETTING_GFXCLK_FMIN:
+ od_table->GfxclkFmin = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_GFXCLK_FMAX:
+ if (value < od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMAX].min_value ||
+ value > od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMAX].max_value)
+ return -EINVAL;
+ od_table->GfxclkFmax = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_GFXCLK_FREQ1:
+ od_table->GfxclkFreq1 = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_GFXCLK_VOLTAGE1:
+ od_table->GfxclkVolt1 = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_GFXCLK_FREQ2:
+ od_table->GfxclkFreq2 = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_GFXCLK_VOLTAGE2:
+ od_table->GfxclkVolt2 = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_GFXCLK_FREQ3:
+ od_table->GfxclkFreq3 = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_GFXCLK_VOLTAGE3:
+ od_table->GfxclkVolt3 = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_UCLK_FMAX:
+ if (value < od8_settings->od8_settings_array[OD8_SETTING_UCLK_FMAX].min_value ||
+ value > od8_settings->od8_settings_array[OD8_SETTING_UCLK_FMAX].max_value)
+ return -EINVAL;
+ od_table->UclkFmax = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_POWER_PERCENTAGE:
+ od_table->OverDrivePct = (int16_t)value;
+ break;
+
+ case OD8_SETTING_FAN_ACOUSTIC_LIMIT:
+ od_table->FanMaximumRpm = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_FAN_MIN_SPEED:
+ od_table->FanMinimumPwm = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_FAN_TARGET_TEMP:
+ od_table->FanTargetTemperature = (uint16_t)value;
+ break;
+
+ case OD8_SETTING_OPERATING_TEMP_MAX:
+ od_table->MaxOpTemp = (uint16_t)value;
+ break;
+ }
+
+ return 0;
+}
+
+static int vega20_set_od_percentage(struct smu_context *smu,
+ enum pp_clock_type type,
+ uint32_t value)
+{
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+ struct vega20_dpm_table *dpm_table = NULL;
+ struct vega20_dpm_table *golden_table = NULL;
+ struct vega20_single_dpm_table *single_dpm_table;
+ struct vega20_single_dpm_table *golden_dpm_table;
+ uint32_t od_clk, index;
+ int ret = 0;
+ int feature_enabled;
+ PPCLK_e clk_id;
+
+ mutex_lock(&(smu->mutex));
+
+ dpm_table = smu_dpm->dpm_context;
+ golden_table = smu_dpm->golden_dpm_context;
+
+ switch (type) {
+ case OD_SCLK:
+ single_dpm_table = &(dpm_table->gfx_table);
+ golden_dpm_table = &(golden_table->gfx_table);
+ feature_enabled = smu_feature_is_enabled(smu, FEATURE_DPM_GFXCLK_BIT);
+ clk_id = PPCLK_GFXCLK;
+ index = OD8_SETTING_GFXCLK_FMAX;
+ break;
+ case OD_MCLK:
+ single_dpm_table = &(dpm_table->mem_table);
+ golden_dpm_table = &(golden_table->mem_table);
+ feature_enabled = smu_feature_is_enabled(smu, FEATURE_DPM_UCLK_BIT);
+ clk_id = PPCLK_UCLK;
+ index = OD8_SETTING_UCLK_FMAX;
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ if (ret)
+ goto set_od_failed;
+
+ od_clk = golden_dpm_table->dpm_levels[golden_dpm_table->count - 1].value * value;
+ od_clk /= 100;
+ od_clk += golden_dpm_table->dpm_levels[golden_dpm_table->count - 1].value;
+
+ ret = smu_update_od8_settings(smu, index, od_clk);
+ if (ret) {
+ pr_err("[Setoverdrive] failed to set od clk!\n");
+ goto set_od_failed;
+ }
+
+ if (feature_enabled) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table,
+ clk_id);
+ if (ret) {
+ pr_err("[Setoverdrive] failed to refresh dpm table!\n");
+ goto set_od_failed;
+ }
+ } else {
+ single_dpm_table->count = 1;
+ single_dpm_table->dpm_levels[0].value = smu->smu_table.boot_values.gfxclk / 100;
+ }
+
+ ret = smu_handle_task(smu, smu_dpm->dpm_level,
+ AMD_PP_TASK_READJUST_POWER_STATE);
+
+set_od_failed:
+ mutex_unlock(&(smu->mutex));
+
+ return ret;
+}
+
+static int vega20_odn_edit_dpm_table(struct smu_context *smu,
+ enum PP_OD_DPM_TABLE_COMMAND type,
+ long *input, uint32_t size)
+{
+ struct smu_table_context *table_context = &smu->smu_table;
+ OverDriveTable_t *od_table =
+ (OverDriveTable_t *)(table_context->overdrive_table);
+ struct smu_dpm_context *smu_dpm = &smu->smu_dpm;
+ struct vega20_dpm_table *dpm_table = NULL;
+ struct vega20_single_dpm_table *single_dpm_table;
+ struct vega20_od8_settings *od8_settings =
+ (struct vega20_od8_settings *)table_context->od8_settings;
+ struct pp_clock_levels_with_latency clocks;
+ int32_t input_index, input_clk, input_vol, i;
+ int od8_id;
+ int ret = 0;
+
+ dpm_table = smu_dpm->dpm_context;
+
+ if (!input) {
+ pr_warn("NULL user input for clock and voltage\n");
+ return -EINVAL;
+ }
+
+ switch (type) {
+ case PP_OD_EDIT_SCLK_VDDC_TABLE:
+ if (!(od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMIN].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMAX].feature_id)) {
+ pr_info("Sclk min/max frequency overdrive not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ for (i = 0; i < size; i += 2) {
+ if (i + 2 > size) {
+ pr_info("invalid number of input parameters %d\n", size);
+ return -EINVAL;
+ }
+
+ input_index = input[i];
+ input_clk = input[i + 1];
+
+ if (input_index != 0 && input_index != 1) {
+ pr_info("Invalid index %d\n", input_index);
+ pr_info("Support min/max sclk frequency settingonly which index by 0/1\n");
+ return -EINVAL;
+ }
+
+ if (input_clk < od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMIN].min_value ||
+ input_clk > od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMAX].max_value) {
+ pr_info("clock freq %d is not within allowed range [%d - %d]\n",
+ input_clk,
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMIN].min_value,
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FMAX].max_value);
+ return -EINVAL;
+ }
+
+ if (input_index == 0 && od_table->GfxclkFmin != input_clk) {
+ od_table->GfxclkFmin = input_clk;
+ table_context->od_gfxclk_update = true;
+ } else if (input_index == 1 && od_table->GfxclkFmax != input_clk) {
+ od_table->GfxclkFmax = input_clk;
+ table_context->od_gfxclk_update = true;
+ }
+ }
+
+ break;
+
+ case PP_OD_EDIT_MCLK_VDDC_TABLE:
+ if (!od8_settings->od8_settings_array[OD8_SETTING_UCLK_FMAX].feature_id) {
+ pr_info("Mclk max frequency overdrive not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ single_dpm_table = &(dpm_table->mem_table);
+ ret = vega20_get_clk_table(smu, &clocks, single_dpm_table);
+ if (ret) {
+ pr_err("Attempt to get memory clk levels Failed!");
+ return ret;
+ }
+
+ for (i = 0; i < size; i += 2) {
+ if (i + 2 > size) {
+ pr_info("invalid number of input parameters %d\n",
+ size);
+ return -EINVAL;
+ }
+
+ input_index = input[i];
+ input_clk = input[i + 1];
+
+ if (input_index != 1) {
+ pr_info("Invalid index %d\n", input_index);
+ pr_info("Support max Mclk frequency setting only which index by 1\n");
+ return -EINVAL;
+ }
+
+ if (input_clk < clocks.data[0].clocks_in_khz / 1000 ||
+ input_clk > od8_settings->od8_settings_array[OD8_SETTING_UCLK_FMAX].max_value) {
+ pr_info("clock freq %d is not within allowed range [%d - %d]\n",
+ input_clk,
+ clocks.data[0].clocks_in_khz / 1000,
+ od8_settings->od8_settings_array[OD8_SETTING_UCLK_FMAX].max_value);
+ return -EINVAL;
+ }
+
+ if (input_index == 1 && od_table->UclkFmax != input_clk) {
+ table_context->od_gfxclk_update = true;
+ od_table->UclkFmax = input_clk;
+ }
+ }
+
+ break;
+
+ case PP_OD_EDIT_VDDC_CURVE:
+ if (!(od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ1].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ2].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_FREQ3].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE1].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE2].feature_id &&
+ od8_settings->od8_settings_array[OD8_SETTING_GFXCLK_VOLTAGE3].feature_id)) {
+ pr_info("Voltage curve calibrate not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ for (i = 0; i < size; i += 3) {
+ if (i + 3 > size) {
+ pr_info("invalid number of input parameters %d\n",
+ size);
+ return -EINVAL;
+ }
+
+ input_index = input[i];
+ input_clk = input[i + 1];
+ input_vol = input[i + 2];
+
+ if (input_index > 2) {
+ pr_info("Setting for point %d is not supported\n",
+ input_index + 1);
+ pr_info("Three supported points index by 0, 1, 2\n");
+ return -EINVAL;
+ }
+
+ od8_id = OD8_SETTING_GFXCLK_FREQ1 + 2 * input_index;
+ if (input_clk < od8_settings->od8_settings_array[od8_id].min_value ||
+ input_clk > od8_settings->od8_settings_array[od8_id].max_value) {
+ pr_info("clock freq %d is not within allowed range [%d - %d]\n",
+ input_clk,
+ od8_settings->od8_settings_array[od8_id].min_value,
+ od8_settings->od8_settings_array[od8_id].max_value);
+ return -EINVAL;
+ }
+
+ od8_id = OD8_SETTING_GFXCLK_VOLTAGE1 + 2 * input_index;
+ if (input_vol < od8_settings->od8_settings_array[od8_id].min_value ||
+ input_vol > od8_settings->od8_settings_array[od8_id].max_value) {
+ pr_info("clock voltage %d is not within allowed range [%d- %d]\n",
+ input_vol,
+ od8_settings->od8_settings_array[od8_id].min_value,
+ od8_settings->od8_settings_array[od8_id].max_value);
+ return -EINVAL;
+ }
+
+ switch (input_index) {
+ case 0:
+ od_table->GfxclkFreq1 = input_clk;
+ od_table->GfxclkVolt1 = input_vol * VOLTAGE_SCALE;
+ break;
+ case 1:
+ od_table->GfxclkFreq2 = input_clk;
+ od_table->GfxclkVolt2 = input_vol * VOLTAGE_SCALE;
+ break;
+ case 2:
+ od_table->GfxclkFreq3 = input_clk;
+ od_table->GfxclkVolt3 = input_vol * VOLTAGE_SCALE;
+ break;
+ }
+ }
+
+ break;
+
+ case PP_OD_RESTORE_DEFAULT_TABLE:
+ ret = smu_update_table(smu, TABLE_OVERDRIVE, table_context->overdrive_table, false);
+ if (ret) {
+ pr_err("Failed to export over drive table!\n");
+ return ret;
+ }
+
+ break;
+
+ case PP_OD_COMMIT_DPM_TABLE:
+ ret = smu_update_table(smu, TABLE_OVERDRIVE, table_context->overdrive_table, true);
+ if (ret) {
+ pr_err("Failed to import over drive table!\n");
+ return ret;
+ }
+
+ /* retrieve updated gfxclk table */
+ if (table_context->od_gfxclk_update) {
+ table_context->od_gfxclk_update = false;
+ single_dpm_table = &(dpm_table->gfx_table);
+
+ if (smu_feature_is_enabled(smu, FEATURE_DPM_GFXCLK_BIT)) {
+ ret = vega20_set_single_dpm_table(smu, single_dpm_table,
+ PPCLK_GFXCLK);
+ if (ret) {
+ pr_err("[Setoverdrive] failed to refresh dpm table!\n");
+ return ret;
+ }
+ } else {
+ single_dpm_table->count = 1;
+ single_dpm_table->dpm_levels[0].value = smu->smu_table.boot_values.gfxclk / 100;
+ }
+ }
+
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ if (type == PP_OD_COMMIT_DPM_TABLE) {
+ mutex_lock(&(smu->mutex));
+ ret = smu_handle_task(smu, smu_dpm->dpm_level,
+ AMD_PP_TASK_READJUST_POWER_STATE);
+ mutex_unlock(&(smu->mutex));
+ }
+
+ return ret;
+}
+
+static const struct pptable_funcs vega20_ppt_funcs = {
+ .alloc_dpm_context = vega20_allocate_dpm_context,
+ .store_powerplay_table = vega20_store_powerplay_table,
+ .check_powerplay_table = vega20_check_powerplay_table,
+ .append_powerplay_table = vega20_append_powerplay_table,
+ .get_smu_msg_index = vega20_get_smu_msg_index,
+ .run_afll_btc = vega20_run_btc_afll,
+ .get_unallowed_feature_mask = vega20_get_unallowed_feature_mask,
+ .get_current_power_state = vega20_get_current_power_state,
+ .set_default_dpm_table = vega20_set_default_dpm_table,
+ .set_power_state = NULL,
+ .populate_umd_state_clk = vega20_populate_umd_state_clk,
+ .print_clk_levels = vega20_print_clk_levels,
+ .force_clk_levels = vega20_force_clk_levels,
+ .get_clock_by_type_with_latency = vega20_get_clock_by_type_with_latency,
+ .set_default_od8_settings = vega20_set_default_od8_setttings,
+ .get_od_percentage = vega20_get_od_percentage,
+ .get_performance_level = vega20_get_performance_level,
+ .force_performance_level = vega20_force_performance_level,
+ .update_specified_od8_value = vega20_update_specified_od8_value,
+ .set_od_percentage = vega20_set_od_percentage,
+ .od_edit_dpm_table = vega20_odn_edit_dpm_table,
+ .pre_display_config_changed = vega20_pre_display_config_changed,
+ .display_config_changed = vega20_display_config_changed,
+ .apply_clocks_adjust_rules = vega20_apply_clocks_adjust_rules,
+ .notify_smc_dispaly_config = vega20_notify_smc_dispaly_config,
+ .force_dpm_limit_value = vega20_force_dpm_limit_value,
+ .unforce_dpm_levels = vega20_unforce_dpm_levels,
+ .upload_dpm_level = vega20_upload_dpm_level,
+ .get_profiling_clk_mask = vega20_get_profiling_clk_mask,
+};
+
+void vega20_set_ppt_funcs(struct smu_context *smu)
+{
+ smu->ppt_funcs = &vega20_ppt_funcs;
+ smu->smc_if_version = SMU11_DRIVER_IF_VERSION;
+}
--- /dev/null
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#ifndef __VEGA20_PPT_H__
+#define __VEGA20_PPT_H__
+
+#define VEGA20_UMD_PSTATE_GFXCLK_LEVEL 0x3
+#define VEGA20_UMD_PSTATE_SOCCLK_LEVEL 0x3
+#define VEGA20_UMD_PSTATE_MCLK_LEVEL 0x2
+#define VEGA20_UMD_PSTATE_UVDCLK_LEVEL 0x3
+#define VEGA20_UMD_PSTATE_VCEMCLK_LEVEL 0x3
+
+#define MAX_REGULAR_DPM_NUMBER 16
+#define MAX_PCIE_CONF 2
+
+#define VOLTAGE_SCALE 4
+#define AVFS_CURVE 0
+#define OD8_HOTCURVE_TEMPERATURE 85
+
+struct vega20_dpm_level {
+ bool enabled;
+ uint32_t value;
+ uint32_t param1;
+};
+
+struct vega20_dpm_state {
+ uint32_t soft_min_level;
+ uint32_t soft_max_level;
+ uint32_t hard_min_level;
+ uint32_t hard_max_level;
+};
+
+struct vega20_single_dpm_table {
+ uint32_t count;
+ struct vega20_dpm_state dpm_state;
+ struct vega20_dpm_level dpm_levels[MAX_REGULAR_DPM_NUMBER];
+};
+
+struct vega20_pcie_table {
+ uint16_t count;
+ uint8_t pcie_gen[MAX_PCIE_CONF];
+ uint8_t pcie_lane[MAX_PCIE_CONF];
+ uint32_t lclk[MAX_PCIE_CONF];
+};
+
+struct vega20_dpm_table {
+ struct vega20_single_dpm_table soc_table;
+ struct vega20_single_dpm_table gfx_table;
+ struct vega20_single_dpm_table mem_table;
+ struct vega20_single_dpm_table eclk_table;
+ struct vega20_single_dpm_table vclk_table;
+ struct vega20_single_dpm_table dclk_table;
+ struct vega20_single_dpm_table dcef_table;
+ struct vega20_single_dpm_table pixel_table;
+ struct vega20_single_dpm_table display_table;
+ struct vega20_single_dpm_table phy_table;
+ struct vega20_single_dpm_table fclk_table;
+ struct vega20_pcie_table pcie_table;
+};
+
+enum OD8_FEATURE_ID
+{
+ OD8_GFXCLK_LIMITS = 1 << 0,
+ OD8_GFXCLK_CURVE = 1 << 1,
+ OD8_UCLK_MAX = 1 << 2,
+ OD8_POWER_LIMIT = 1 << 3,
+ OD8_ACOUSTIC_LIMIT_SCLK = 1 << 4, //FanMaximumRpm
+ OD8_FAN_SPEED_MIN = 1 << 5, //FanMinimumPwm
+ OD8_TEMPERATURE_FAN = 1 << 6, //FanTargetTemperature
+ OD8_TEMPERATURE_SYSTEM = 1 << 7, //MaxOpTemp
+ OD8_MEMORY_TIMING_TUNE = 1 << 8,
+ OD8_FAN_ZERO_RPM_CONTROL = 1 << 9
+};
+
+enum OD8_SETTING_ID
+{
+ OD8_SETTING_GFXCLK_FMIN = 0,
+ OD8_SETTING_GFXCLK_FMAX,
+ OD8_SETTING_GFXCLK_FREQ1,
+ OD8_SETTING_GFXCLK_VOLTAGE1,
+ OD8_SETTING_GFXCLK_FREQ2,
+ OD8_SETTING_GFXCLK_VOLTAGE2,
+ OD8_SETTING_GFXCLK_FREQ3,
+ OD8_SETTING_GFXCLK_VOLTAGE3,
+ OD8_SETTING_UCLK_FMAX,
+ OD8_SETTING_POWER_PERCENTAGE,
+ OD8_SETTING_FAN_ACOUSTIC_LIMIT,
+ OD8_SETTING_FAN_MIN_SPEED,
+ OD8_SETTING_FAN_TARGET_TEMP,
+ OD8_SETTING_OPERATING_TEMP_MAX,
+ OD8_SETTING_AC_TIMING,
+ OD8_SETTING_FAN_ZERO_RPM_CONTROL,
+ OD8_SETTING_COUNT
+};
+
+struct vega20_od8_single_setting {
+ uint32_t feature_id;
+ int32_t min_value;
+ int32_t max_value;
+ int32_t current_value;
+ int32_t default_value;
+};
+
+struct vega20_od8_settings {
+ struct vega20_od8_single_setting od8_settings_array[OD8_SETTING_COUNT];
+};
+
+extern void vega20_set_ppt_funcs(struct smu_context *smu);
+
+#endif
/* Mali-display product IDs */
#define MALIDP_D71_PRODUCT_ID 0x0071
+union komeda_config_id {
+ struct {
+ __u32 max_line_sz:16,
+ n_pipelines:2,
+ n_scalers:2, /* number of scalers per pipeline */
+ n_layers:3, /* number of layers per pipeline */
+ n_richs:3, /* number of rich layers per pipeline */
+ reserved_bits:6;
+ };
+ __u32 value;
+};
+
#endif /* _MALIDP_PRODUCT_H_ */
#ifndef _MALIDP_UTILS_
#define _MALIDP_UTILS_
+#include <linux/delay.h>
+
#define has_bit(nr, mask) (BIT(nr) & (mask))
#define has_bits(bits, mask) (((bits) & (mask)) == (bits))
#define dp_for_each_set_bit(bit, mask) \
for_each_set_bit((bit), ((unsigned long *)&(mask)), sizeof(mask) * 8)
+#define dp_wait_cond(__cond, __tries, __min_range, __max_range) \
+({ \
+ int num_tries = __tries; \
+ while (!__cond && (num_tries > 0)) { \
+ usleep_range(__min_range, __max_range); \
+ if (__cond) \
+ break; \
+ num_tries--; \
+ } \
+ num_tries; \
+})
+
+/* the restriction of range is [start, end] */
+struct malidp_range {
+ u32 start;
+ u32 end;
+};
+
+static inline void set_range(struct malidp_range *rg, u32 start, u32 end)
+{
+ rg->start = start;
+ rg->end = end;
+}
+
+static inline bool in_range(struct malidp_range *rg, u32 v)
+{
+ return (v >= rg->start) && (v <= rg->end);
+}
+
#endif /* _MALIDP_UTILS_ */
# SPDX-License-Identifier: GPL-2.0
ccflags-y := \
- -I$(src)/../include \
- -I$(src)
+ -I $(srctree)/$(src)/../include \
+ -I $(srctree)/$(src)
komeda-y := \
komeda_drv.o \
komeda_dev.o \
komeda_format_caps.o \
komeda_pipeline.o \
+ komeda_pipeline_state.o \
komeda_framebuffer.o \
komeda_kms.o \
komeda_crtc.o \
komeda_private_obj.o
komeda-y += \
- d71/d71_dev.o
+ d71/d71_dev.o \
+ d71/d71_component.o
obj-$(CONFIG_DRM_KOMEDA) += komeda.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * (C) COPYRIGHT 2018 ARM Limited. All rights reserved.
+ * Author: James.Qian.Wang <james.qian.wang@arm.com>
+ *
+ */
+
+#include <drm/drm_print.h>
+#include "d71_dev.h"
+#include "komeda_kms.h"
+#include "malidp_io.h"
+#include "komeda_framebuffer.h"
+
+static void get_resources_id(u32 hw_id, u32 *pipe_id, u32 *comp_id)
+{
+ u32 id = BLOCK_INFO_BLK_ID(hw_id);
+ u32 pipe = id;
+
+ switch (BLOCK_INFO_BLK_TYPE(hw_id)) {
+ case D71_BLK_TYPE_LPU_WB_LAYER:
+ id = KOMEDA_COMPONENT_WB_LAYER;
+ break;
+ case D71_BLK_TYPE_CU_SPLITTER:
+ id = KOMEDA_COMPONENT_SPLITTER;
+ break;
+ case D71_BLK_TYPE_CU_SCALER:
+ pipe = id / D71_PIPELINE_MAX_SCALERS;
+ id %= D71_PIPELINE_MAX_SCALERS;
+ id += KOMEDA_COMPONENT_SCALER0;
+ break;
+ case D71_BLK_TYPE_CU:
+ id += KOMEDA_COMPONENT_COMPIZ0;
+ break;
+ case D71_BLK_TYPE_LPU_LAYER:
+ pipe = id / D71_PIPELINE_MAX_LAYERS;
+ id %= D71_PIPELINE_MAX_LAYERS;
+ id += KOMEDA_COMPONENT_LAYER0;
+ break;
+ case D71_BLK_TYPE_DOU_IPS:
+ id += KOMEDA_COMPONENT_IPS0;
+ break;
+ case D71_BLK_TYPE_CU_MERGER:
+ id = KOMEDA_COMPONENT_MERGER;
+ break;
+ case D71_BLK_TYPE_DOU:
+ id = KOMEDA_COMPONENT_TIMING_CTRLR;
+ break;
+ default:
+ id = 0xFFFFFFFF;
+ }
+
+ if (comp_id)
+ *comp_id = id;
+
+ if (pipe_id)
+ *pipe_id = pipe;
+}
+
+static u32 get_valid_inputs(struct block_header *blk)
+{
+ u32 valid_inputs = 0, comp_id;
+ int i;
+
+ for (i = 0; i < PIPELINE_INFO_N_VALID_INPUTS(blk->pipeline_info); i++) {
+ get_resources_id(blk->input_ids[i], NULL, &comp_id);
+ if (comp_id == 0xFFFFFFFF)
+ continue;
+ valid_inputs |= BIT(comp_id);
+ }
+
+ return valid_inputs;
+}
+
+static void get_values_from_reg(void __iomem *reg, u32 offset,
+ u32 count, u32 *val)
+{
+ u32 i, addr;
+
+ for (i = 0; i < count; i++) {
+ addr = offset + (i << 2);
+ /* 0xA4 is WO register */
+ if (addr != 0xA4)
+ val[i] = malidp_read32(reg, addr);
+ else
+ val[i] = 0xDEADDEAD;
+ }
+}
+
+static void dump_block_header(struct seq_file *sf, void __iomem *reg)
+{
+ struct block_header hdr;
+ u32 i, n_input, n_output;
+
+ d71_read_block_header(reg, &hdr);
+ seq_printf(sf, "BLOCK_INFO:\t\t0x%X\n", hdr.block_info);
+ seq_printf(sf, "PIPELINE_INFO:\t\t0x%X\n", hdr.pipeline_info);
+
+ n_output = PIPELINE_INFO_N_OUTPUTS(hdr.pipeline_info);
+ n_input = PIPELINE_INFO_N_VALID_INPUTS(hdr.pipeline_info);
+
+ for (i = 0; i < n_input; i++)
+ seq_printf(sf, "VALID_INPUT_ID%u:\t0x%X\n",
+ i, hdr.input_ids[i]);
+
+ for (i = 0; i < n_output; i++)
+ seq_printf(sf, "OUTPUT_ID%u:\t\t0x%X\n",
+ i, hdr.output_ids[i]);
+}
+
+static u32 to_rot_ctrl(u32 rot)
+{
+ u32 lr_ctrl = 0;
+
+ switch (rot & DRM_MODE_ROTATE_MASK) {
+ case DRM_MODE_ROTATE_0:
+ lr_ctrl |= L_ROT(L_ROT_R0);
+ break;
+ case DRM_MODE_ROTATE_90:
+ lr_ctrl |= L_ROT(L_ROT_R90);
+ break;
+ case DRM_MODE_ROTATE_180:
+ lr_ctrl |= L_ROT(L_ROT_R180);
+ break;
+ case DRM_MODE_ROTATE_270:
+ lr_ctrl |= L_ROT(L_ROT_R270);
+ break;
+ }
+
+ if (rot & DRM_MODE_REFLECT_X)
+ lr_ctrl |= L_HFLIP;
+ if (rot & DRM_MODE_REFLECT_Y)
+ lr_ctrl |= L_VFLIP;
+
+ return lr_ctrl;
+}
+
+static inline u32 to_d71_input_id(struct komeda_component_output *output)
+{
+ struct komeda_component *comp = output->component;
+
+ return comp ? (comp->hw_id + output->output_port) : 0;
+}
+
+static void d71_layer_disable(struct komeda_component *c)
+{
+ malidp_write32_mask(c->reg, BLK_CONTROL, L_EN, 0);
+}
+
+static void d71_layer_update(struct komeda_component *c,
+ struct komeda_component_state *state)
+{
+ struct komeda_layer_state *st = to_layer_st(state);
+ struct drm_plane_state *plane_st = state->plane->state;
+ struct drm_framebuffer *fb = plane_st->fb;
+ struct komeda_fb *kfb = to_kfb(fb);
+ u32 __iomem *reg = c->reg;
+ u32 ctrl_mask = L_EN | L_ROT(L_ROT_R270) | L_HFLIP | L_VFLIP | L_TBU_EN;
+ u32 ctrl = L_EN | to_rot_ctrl(st->rot);
+ int i;
+
+ for (i = 0; i < fb->format->num_planes; i++) {
+ malidp_write32(reg,
+ BLK_P0_PTR_LOW + i * LAYER_PER_PLANE_REGS * 4,
+ lower_32_bits(st->addr[i]));
+ malidp_write32(reg,
+ BLK_P0_PTR_HIGH + i * LAYER_PER_PLANE_REGS * 4,
+ upper_32_bits(st->addr[i]));
+ if (i >= 2)
+ break;
+
+ malidp_write32(reg,
+ BLK_P0_STRIDE + i * LAYER_PER_PLANE_REGS * 4,
+ fb->pitches[i] & 0xFFFF);
+ }
+
+ malidp_write32(reg, LAYER_FMT, kfb->format_caps->hw_id);
+ malidp_write32(reg, BLK_IN_SIZE, HV_SIZE(st->hsize, st->vsize));
+
+ malidp_write32_mask(reg, BLK_CONTROL, ctrl_mask, ctrl);
+}
+
+static void d71_layer_dump(struct komeda_component *c, struct seq_file *sf)
+{
+ u32 v[15], i;
+ bool rich, rgb2rgb;
+ char *prefix;
+
+ get_values_from_reg(c->reg, LAYER_INFO, 1, &v[14]);
+ if (v[14] & 0x1) {
+ rich = true;
+ prefix = "LR_";
+ } else {
+ rich = false;
+ prefix = "LS_";
+ }
+
+ rgb2rgb = !!(v[14] & L_INFO_CM);
+
+ dump_block_header(sf, c->reg);
+
+ seq_printf(sf, "%sLAYER_INFO:\t\t0x%X\n", prefix, v[14]);
+
+ get_values_from_reg(c->reg, 0xD0, 1, v);
+ seq_printf(sf, "%sCONTROL:\t\t0x%X\n", prefix, v[0]);
+ if (rich) {
+ get_values_from_reg(c->reg, 0xD4, 1, v);
+ seq_printf(sf, "LR_RICH_CONTROL:\t0x%X\n", v[0]);
+ }
+ get_values_from_reg(c->reg, 0xD8, 4, v);
+ seq_printf(sf, "%sFORMAT:\t\t0x%X\n", prefix, v[0]);
+ seq_printf(sf, "%sIT_COEFFTAB:\t\t0x%X\n", prefix, v[1]);
+ seq_printf(sf, "%sIN_SIZE:\t\t0x%X\n", prefix, v[2]);
+ seq_printf(sf, "%sPALPHA:\t\t0x%X\n", prefix, v[3]);
+
+ get_values_from_reg(c->reg, 0x100, 3, v);
+ seq_printf(sf, "%sP0_PTR_LOW:\t\t0x%X\n", prefix, v[0]);
+ seq_printf(sf, "%sP0_PTR_HIGH:\t\t0x%X\n", prefix, v[1]);
+ seq_printf(sf, "%sP0_STRIDE:\t\t0x%X\n", prefix, v[2]);
+
+ get_values_from_reg(c->reg, 0x110, 2, v);
+ seq_printf(sf, "%sP1_PTR_LOW:\t\t0x%X\n", prefix, v[0]);
+ seq_printf(sf, "%sP1_PTR_HIGH:\t\t0x%X\n", prefix, v[1]);
+ if (rich) {
+ get_values_from_reg(c->reg, 0x118, 1, v);
+ seq_printf(sf, "LR_P1_STRIDE:\t\t0x%X\n", v[0]);
+
+ get_values_from_reg(c->reg, 0x120, 2, v);
+ seq_printf(sf, "LR_P2_PTR_LOW:\t\t0x%X\n", v[0]);
+ seq_printf(sf, "LR_P2_PTR_HIGH:\t\t0x%X\n", v[1]);
+
+ get_values_from_reg(c->reg, 0x130, 12, v);
+ for (i = 0; i < 12; i++)
+ seq_printf(sf, "LR_YUV_RGB_COEFF%u:\t0x%X\n", i, v[i]);
+ }
+
+ if (rgb2rgb) {
+ get_values_from_reg(c->reg, LAYER_RGB_RGB_COEFF0, 12, v);
+ for (i = 0; i < 12; i++)
+ seq_printf(sf, "LS_RGB_RGB_COEFF%u:\t0x%X\n", i, v[i]);
+ }
+
+ get_values_from_reg(c->reg, 0x160, 3, v);
+ seq_printf(sf, "%sAD_CONTROL:\t\t0x%X\n", prefix, v[0]);
+ seq_printf(sf, "%sAD_H_CROP:\t\t0x%X\n", prefix, v[1]);
+ seq_printf(sf, "%sAD_V_CROP:\t\t0x%X\n", prefix, v[2]);
+}
+
+static struct komeda_component_funcs d71_layer_funcs = {
+ .update = d71_layer_update,
+ .disable = d71_layer_disable,
+ .dump_register = d71_layer_dump,
+};
+
+static int d71_layer_init(struct d71_dev *d71,
+ struct block_header *blk, u32 __iomem *reg)
+{
+ struct komeda_component *c;
+ struct komeda_layer *layer;
+ u32 pipe_id, layer_id, layer_info;
+
+ get_resources_id(blk->block_info, &pipe_id, &layer_id);
+ c = komeda_component_add(&d71->pipes[pipe_id]->base, sizeof(*layer),
+ layer_id,
+ BLOCK_INFO_INPUT_ID(blk->block_info),
+ &d71_layer_funcs, 0,
+ get_valid_inputs(blk),
+ 1, reg, "LPU%d_LAYER%d", pipe_id, layer_id);
+ if (IS_ERR(c)) {
+ DRM_ERROR("Failed to add layer component\n");
+ return PTR_ERR(c);
+ }
+
+ layer = to_layer(c);
+ layer_info = malidp_read32(reg, LAYER_INFO);
+
+ if (layer_info & L_INFO_RF)
+ layer->layer_type = KOMEDA_FMT_RICH_LAYER;
+ else
+ layer->layer_type = KOMEDA_FMT_SIMPLE_LAYER;
+
+ set_range(&layer->hsize_in, 4, d71->max_line_size);
+ set_range(&layer->vsize_in, 4, d71->max_vsize);
+
+ malidp_write32(reg, LAYER_PALPHA, D71_PALPHA_DEF_MAP);
+
+ layer->supported_rots = DRM_MODE_ROTATE_MASK | DRM_MODE_REFLECT_MASK;
+
+ return 0;
+}
+
+static int d71_wb_layer_init(struct d71_dev *d71,
+ struct block_header *blk, u32 __iomem *reg)
+{
+ DRM_DEBUG("Detect D71_Wb_Layer.\n");
+
+ return 0;
+}
+
+static void d71_component_disable(struct komeda_component *c)
+{
+ u32 __iomem *reg = c->reg;
+ u32 i;
+
+ malidp_write32(reg, BLK_CONTROL, 0);
+
+ for (i = 0; i < c->max_active_inputs; i++)
+ malidp_write32(reg, BLK_INPUT_ID0 + (i << 2), 0);
+}
+
+static void compiz_enable_input(u32 __iomem *id_reg,
+ u32 __iomem *cfg_reg,
+ u32 input_hw_id,
+ struct komeda_compiz_input_cfg *cin)
+{
+ u32 ctrl = CU_INPUT_CTRL_EN;
+ u8 blend = cin->pixel_blend_mode;
+
+ if (blend == DRM_MODE_BLEND_PIXEL_NONE)
+ ctrl |= CU_INPUT_CTRL_PAD;
+ else if (blend == DRM_MODE_BLEND_PREMULTI)
+ ctrl |= CU_INPUT_CTRL_PMUL;
+
+ ctrl |= CU_INPUT_CTRL_ALPHA(cin->layer_alpha);
+
+ malidp_write32(id_reg, BLK_INPUT_ID0, input_hw_id);
+
+ malidp_write32(cfg_reg, CU_INPUT0_SIZE,
+ HV_SIZE(cin->hsize, cin->vsize));
+ malidp_write32(cfg_reg, CU_INPUT0_OFFSET,
+ HV_OFFSET(cin->hoffset, cin->voffset));
+ malidp_write32(cfg_reg, CU_INPUT0_CONTROL, ctrl);
+}
+
+static void d71_compiz_update(struct komeda_component *c,
+ struct komeda_component_state *state)
+{
+ struct komeda_compiz_state *st = to_compiz_st(state);
+ u32 __iomem *reg = c->reg;
+ u32 __iomem *id_reg, *cfg_reg;
+ u32 index, input_hw_id;
+
+ for_each_changed_input(state, index) {
+ id_reg = reg + index;
+ cfg_reg = reg + index * CU_PER_INPUT_REGS;
+ input_hw_id = to_d71_input_id(&state->inputs[index]);
+ if (state->active_inputs & BIT(index)) {
+ compiz_enable_input(id_reg, cfg_reg,
+ input_hw_id, &st->cins[index]);
+ } else {
+ malidp_write32(id_reg, BLK_INPUT_ID0, 0);
+ malidp_write32(cfg_reg, CU_INPUT0_CONTROL, 0);
+ }
+ }
+
+ malidp_write32(reg, BLK_SIZE, HV_SIZE(st->hsize, st->vsize));
+}
+
+static void d71_compiz_dump(struct komeda_component *c, struct seq_file *sf)
+{
+ u32 v[8], i;
+
+ dump_block_header(sf, c->reg);
+
+ get_values_from_reg(c->reg, 0x80, 5, v);
+ for (i = 0; i < 5; i++)
+ seq_printf(sf, "CU_INPUT_ID%u:\t\t0x%X\n", i, v[i]);
+
+ get_values_from_reg(c->reg, 0xA0, 5, v);
+ seq_printf(sf, "CU_IRQ_RAW_STATUS:\t0x%X\n", v[0]);
+ seq_printf(sf, "CU_IRQ_CLEAR:\t\t0x%X\n", v[1]);
+ seq_printf(sf, "CU_IRQ_MASK:\t\t0x%X\n", v[2]);
+ seq_printf(sf, "CU_IRQ_STATUS:\t\t0x%X\n", v[3]);
+ seq_printf(sf, "CU_STATUS:\t\t0x%X\n", v[4]);
+
+ get_values_from_reg(c->reg, 0xD0, 2, v);
+ seq_printf(sf, "CU_CONTROL:\t\t0x%X\n", v[0]);
+ seq_printf(sf, "CU_SIZE:\t\t0x%X\n", v[1]);
+
+ get_values_from_reg(c->reg, 0xDC, 1, v);
+ seq_printf(sf, "CU_BG_COLOR:\t\t0x%X\n", v[0]);
+
+ for (i = 0, v[4] = 0xE0; i < 5; i++, v[4] += 0x10) {
+ get_values_from_reg(c->reg, v[4], 3, v);
+ seq_printf(sf, "CU_INPUT%u_SIZE:\t\t0x%X\n", i, v[0]);
+ seq_printf(sf, "CU_INPUT%u_OFFSET:\t0x%X\n", i, v[1]);
+ seq_printf(sf, "CU_INPUT%u_CONTROL:\t0x%X\n", i, v[2]);
+ }
+
+ get_values_from_reg(c->reg, 0x130, 2, v);
+ seq_printf(sf, "CU_USER_LOW:\t\t0x%X\n", v[0]);
+ seq_printf(sf, "CU_USER_HIGH:\t\t0x%X\n", v[1]);
+}
+
+static struct komeda_component_funcs d71_compiz_funcs = {
+ .update = d71_compiz_update,
+ .disable = d71_component_disable,
+ .dump_register = d71_compiz_dump,
+};
+
+static int d71_compiz_init(struct d71_dev *d71,
+ struct block_header *blk, u32 __iomem *reg)
+{
+ struct komeda_component *c;
+ struct komeda_compiz *compiz;
+ u32 pipe_id, comp_id;
+
+ get_resources_id(blk->block_info, &pipe_id, &comp_id);
+
+ c = komeda_component_add(&d71->pipes[pipe_id]->base, sizeof(*compiz),
+ comp_id,
+ BLOCK_INFO_INPUT_ID(blk->block_info),
+ &d71_compiz_funcs,
+ CU_NUM_INPUT_IDS, get_valid_inputs(blk),
+ CU_NUM_OUTPUT_IDS, reg,
+ "CU%d", pipe_id);
+ if (IS_ERR(c))
+ return PTR_ERR(c);
+
+ compiz = to_compiz(c);
+
+ set_range(&compiz->hsize, D71_MIN_LINE_SIZE, d71->max_line_size);
+ set_range(&compiz->vsize, D71_MIN_VERTICAL_SIZE, d71->max_vsize);
+
+ return 0;
+}
+
+static void d71_improc_update(struct komeda_component *c,
+ struct komeda_component_state *state)
+{
+ struct komeda_improc_state *st = to_improc_st(state);
+ u32 __iomem *reg = c->reg;
+ u32 index, input_hw_id;
+
+ for_each_changed_input(state, index) {
+ input_hw_id = state->active_inputs & BIT(index) ?
+ to_d71_input_id(&state->inputs[index]) : 0;
+ malidp_write32(reg, BLK_INPUT_ID0 + index * 4, input_hw_id);
+ }
+
+ malidp_write32(reg, BLK_SIZE, HV_SIZE(st->hsize, st->vsize));
+}
+
+static void d71_improc_dump(struct komeda_component *c, struct seq_file *sf)
+{
+ u32 v[12], i;
+
+ dump_block_header(sf, c->reg);
+
+ get_values_from_reg(c->reg, 0x80, 2, v);
+ seq_printf(sf, "IPS_INPUT_ID0:\t\t0x%X\n", v[0]);
+ seq_printf(sf, "IPS_INPUT_ID1:\t\t0x%X\n", v[1]);
+
+ get_values_from_reg(c->reg, 0xC0, 1, v);
+ seq_printf(sf, "IPS_INFO:\t\t0x%X\n", v[0]);
+
+ get_values_from_reg(c->reg, 0xD0, 3, v);
+ seq_printf(sf, "IPS_CONTROL:\t\t0x%X\n", v[0]);
+ seq_printf(sf, "IPS_SIZE:\t\t0x%X\n", v[1]);
+ seq_printf(sf, "IPS_DEPTH:\t\t0x%X\n", v[2]);
+
+ get_values_from_reg(c->reg, 0x130, 12, v);
+ for (i = 0; i < 12; i++)
+ seq_printf(sf, "IPS_RGB_RGB_COEFF%u:\t0x%X\n", i, v[i]);
+
+ get_values_from_reg(c->reg, 0x170, 12, v);
+ for (i = 0; i < 12; i++)
+ seq_printf(sf, "IPS_RGB_YUV_COEFF%u:\t0x%X\n", i, v[i]);
+}
+
+static struct komeda_component_funcs d71_improc_funcs = {
+ .update = d71_improc_update,
+ .disable = d71_component_disable,
+ .dump_register = d71_improc_dump,
+};
+
+static int d71_improc_init(struct d71_dev *d71,
+ struct block_header *blk, u32 __iomem *reg)
+{
+ struct komeda_component *c;
+ struct komeda_improc *improc;
+ u32 pipe_id, comp_id, value;
+
+ get_resources_id(blk->block_info, &pipe_id, &comp_id);
+
+ c = komeda_component_add(&d71->pipes[pipe_id]->base, sizeof(*improc),
+ comp_id,
+ BLOCK_INFO_INPUT_ID(blk->block_info),
+ &d71_improc_funcs, IPS_NUM_INPUT_IDS,
+ get_valid_inputs(blk),
+ IPS_NUM_OUTPUT_IDS, reg, "DOU%d_IPS", pipe_id);
+ if (IS_ERR(c)) {
+ DRM_ERROR("Failed to add improc component\n");
+ return PTR_ERR(c);
+ }
+
+ improc = to_improc(c);
+ improc->supported_color_depths = BIT(8) | BIT(10);
+ improc->supported_color_formats = DRM_COLOR_FORMAT_RGB444 |
+ DRM_COLOR_FORMAT_YCRCB444 |
+ DRM_COLOR_FORMAT_YCRCB422;
+ value = malidp_read32(reg, BLK_INFO);
+ if (value & IPS_INFO_CHD420)
+ improc->supported_color_formats |= DRM_COLOR_FORMAT_YCRCB420;
+
+ improc->supports_csc = true;
+ improc->supports_gamma = true;
+
+ return 0;
+}
+
+static void d71_timing_ctrlr_disable(struct komeda_component *c)
+{
+ malidp_write32_mask(c->reg, BLK_CONTROL, BS_CTRL_EN, 0);
+}
+
+static void d71_timing_ctrlr_update(struct komeda_component *c,
+ struct komeda_component_state *state)
+{
+ struct drm_crtc_state *crtc_st = state->crtc->state;
+ u32 __iomem *reg = c->reg;
+ struct videomode vm;
+ u32 value;
+
+ drm_display_mode_to_videomode(&crtc_st->adjusted_mode, &vm);
+
+ malidp_write32(reg, BS_ACTIVESIZE, HV_SIZE(vm.hactive, vm.vactive));
+ malidp_write32(reg, BS_HINTERVALS, BS_H_INTVALS(vm.hfront_porch,
+ vm.hback_porch));
+ malidp_write32(reg, BS_VINTERVALS, BS_V_INTVALS(vm.vfront_porch,
+ vm.vback_porch));
+
+ value = BS_SYNC_VSW(vm.vsync_len) | BS_SYNC_HSW(vm.hsync_len);
+ value |= vm.flags & DISPLAY_FLAGS_VSYNC_HIGH ? BS_SYNC_VSP : 0;
+ value |= vm.flags & DISPLAY_FLAGS_HSYNC_HIGH ? BS_SYNC_HSP : 0;
+ malidp_write32(reg, BS_SYNC, value);
+
+ malidp_write32(reg, BS_PROG_LINE, D71_DEFAULT_PREPRETCH_LINE - 1);
+ malidp_write32(reg, BS_PREFETCH_LINE, D71_DEFAULT_PREPRETCH_LINE);
+
+ /* configure bs control register */
+ value = BS_CTRL_EN | BS_CTRL_VM;
+
+ malidp_write32(reg, BLK_CONTROL, value);
+}
+
+static void d71_timing_ctrlr_dump(struct komeda_component *c,
+ struct seq_file *sf)
+{
+ u32 v[8], i;
+
+ dump_block_header(sf, c->reg);
+
+ get_values_from_reg(c->reg, 0xC0, 1, v);
+ seq_printf(sf, "BS_INFO:\t\t0x%X\n", v[0]);
+
+ get_values_from_reg(c->reg, 0xD0, 8, v);
+ seq_printf(sf, "BS_CONTROL:\t\t0x%X\n", v[0]);
+ seq_printf(sf, "BS_PROG_LINE:\t\t0x%X\n", v[1]);
+ seq_printf(sf, "BS_PREFETCH_LINE:\t0x%X\n", v[2]);
+ seq_printf(sf, "BS_BG_COLOR:\t\t0x%X\n", v[3]);
+ seq_printf(sf, "BS_ACTIVESIZE:\t\t0x%X\n", v[4]);
+ seq_printf(sf, "BS_HINTERVALS:\t\t0x%X\n", v[5]);
+ seq_printf(sf, "BS_VINTERVALS:\t\t0x%X\n", v[6]);
+ seq_printf(sf, "BS_SYNC:\t\t0x%X\n", v[7]);
+
+ get_values_from_reg(c->reg, 0x100, 3, v);
+ seq_printf(sf, "BS_DRIFT_TO:\t\t0x%X\n", v[0]);
+ seq_printf(sf, "BS_FRAME_TO:\t\t0x%X\n", v[1]);
+ seq_printf(sf, "BS_TE_TO:\t\t0x%X\n", v[2]);
+
+ get_values_from_reg(c->reg, 0x110, 3, v);
+ for (i = 0; i < 3; i++)
+ seq_printf(sf, "BS_T%u_INTERVAL:\t\t0x%X\n", i, v[i]);
+
+ get_values_from_reg(c->reg, 0x120, 5, v);
+ for (i = 0; i < 2; i++) {
+ seq_printf(sf, "BS_CRC%u_LOW:\t\t0x%X\n", i, v[i << 1]);
+ seq_printf(sf, "BS_CRC%u_HIGH:\t\t0x%X\n", i, v[(i << 1) + 1]);
+ }
+ seq_printf(sf, "BS_USER:\t\t0x%X\n", v[4]);
+}
+
+static struct komeda_component_funcs d71_timing_ctrlr_funcs = {
+ .update = d71_timing_ctrlr_update,
+ .disable = d71_timing_ctrlr_disable,
+ .dump_register = d71_timing_ctrlr_dump,
+};
+
+static int d71_timing_ctrlr_init(struct d71_dev *d71,
+ struct block_header *blk, u32 __iomem *reg)
+{
+ struct komeda_component *c;
+ struct komeda_timing_ctrlr *ctrlr;
+ u32 pipe_id, comp_id;
+
+ get_resources_id(blk->block_info, &pipe_id, &comp_id);
+
+ c = komeda_component_add(&d71->pipes[pipe_id]->base, sizeof(*ctrlr),
+ KOMEDA_COMPONENT_TIMING_CTRLR,
+ BLOCK_INFO_INPUT_ID(blk->block_info),
+ &d71_timing_ctrlr_funcs,
+ 1, BIT(KOMEDA_COMPONENT_IPS0 + pipe_id),
+ BS_NUM_OUTPUT_IDS, reg, "DOU%d_BS", pipe_id);
+ if (IS_ERR(c)) {
+ DRM_ERROR("Failed to add display_ctrl component\n");
+ return PTR_ERR(c);
+ }
+
+ ctrlr = to_ctrlr(c);
+
+ ctrlr->supports_dual_link = true;
+
+ return 0;
+}
+
+int d71_probe_block(struct d71_dev *d71,
+ struct block_header *blk, u32 __iomem *reg)
+{
+ struct d71_pipeline *pipe;
+ int blk_id = BLOCK_INFO_BLK_ID(blk->block_info);
+
+ int err = 0;
+
+ switch (BLOCK_INFO_BLK_TYPE(blk->block_info)) {
+ case D71_BLK_TYPE_GCU:
+ break;
+
+ case D71_BLK_TYPE_LPU:
+ pipe = d71->pipes[blk_id];
+ pipe->lpu_addr = reg;
+ break;
+
+ case D71_BLK_TYPE_LPU_LAYER:
+ err = d71_layer_init(d71, blk, reg);
+ break;
+
+ case D71_BLK_TYPE_LPU_WB_LAYER:
+ err = d71_wb_layer_init(d71, blk, reg);
+ break;
+
+ case D71_BLK_TYPE_CU:
+ pipe = d71->pipes[blk_id];
+ pipe->cu_addr = reg;
+ err = d71_compiz_init(d71, blk, reg);
+ break;
+
+ case D71_BLK_TYPE_CU_SPLITTER:
+ case D71_BLK_TYPE_CU_SCALER:
+ case D71_BLK_TYPE_CU_MERGER:
+ break;
+
+ case D71_BLK_TYPE_DOU:
+ pipe = d71->pipes[blk_id];
+ pipe->dou_addr = reg;
+ break;
+
+ case D71_BLK_TYPE_DOU_IPS:
+ err = d71_improc_init(d71, blk, reg);
+ break;
+
+ case D71_BLK_TYPE_DOU_FT_COEFF:
+ pipe = d71->pipes[blk_id];
+ pipe->dou_ft_coeff_addr = reg;
+ break;
+
+ case D71_BLK_TYPE_DOU_BS:
+ err = d71_timing_ctrlr_init(d71, blk, reg);
+ break;
+
+ case D71_BLK_TYPE_GLB_LT_COEFF:
+ break;
+
+ case D71_BLK_TYPE_GLB_SCL_COEFF:
+ d71->glb_scl_coeff_addr[blk_id] = reg;
+ break;
+
+ default:
+ DRM_ERROR("Unknown block (block_info: 0x%x) is found\n",
+ blk->block_info);
+ err = -EINVAL;
+ break;
+ }
+
+ return err;
+}
* Author: James.Qian.Wang <james.qian.wang@arm.com>
*
*/
+
+#include <drm/drm_print.h>
+#include "d71_dev.h"
#include "malidp_io.h"
-#include "komeda_dev.h"
+
+static u64 get_lpu_event(struct d71_pipeline *d71_pipeline)
+{
+ u32 __iomem *reg = d71_pipeline->lpu_addr;
+ u32 status, raw_status;
+ u64 evts = 0ULL;
+
+ raw_status = malidp_read32(reg, BLK_IRQ_RAW_STATUS);
+ if (raw_status & LPU_IRQ_IBSY)
+ evts |= KOMEDA_EVENT_IBSY;
+ if (raw_status & LPU_IRQ_EOW)
+ evts |= KOMEDA_EVENT_EOW;
+
+ if (raw_status & (LPU_IRQ_ERR | LPU_IRQ_IBSY)) {
+ u32 restore = 0, tbu_status;
+ /* Check error of LPU status */
+ status = malidp_read32(reg, BLK_STATUS);
+ if (status & LPU_STATUS_AXIE) {
+ restore |= LPU_STATUS_AXIE;
+ evts |= KOMEDA_ERR_AXIE;
+ }
+ if (status & LPU_STATUS_ACE0) {
+ restore |= LPU_STATUS_ACE0;
+ evts |= KOMEDA_ERR_ACE0;
+ }
+ if (status & LPU_STATUS_ACE1) {
+ restore |= LPU_STATUS_ACE1;
+ evts |= KOMEDA_ERR_ACE1;
+ }
+ if (status & LPU_STATUS_ACE2) {
+ restore |= LPU_STATUS_ACE2;
+ evts |= KOMEDA_ERR_ACE2;
+ }
+ if (status & LPU_STATUS_ACE3) {
+ restore |= LPU_STATUS_ACE3;
+ evts |= KOMEDA_ERR_ACE3;
+ }
+ if (restore != 0)
+ malidp_write32_mask(reg, BLK_STATUS, restore, 0);
+
+ restore = 0;
+ /* Check errors of TBU status */
+ tbu_status = malidp_read32(reg, LPU_TBU_STATUS);
+ if (tbu_status & LPU_TBU_STATUS_TCF) {
+ restore |= LPU_TBU_STATUS_TCF;
+ evts |= KOMEDA_ERR_TCF;
+ }
+ if (tbu_status & LPU_TBU_STATUS_TTNG) {
+ restore |= LPU_TBU_STATUS_TTNG;
+ evts |= KOMEDA_ERR_TTNG;
+ }
+ if (tbu_status & LPU_TBU_STATUS_TITR) {
+ restore |= LPU_TBU_STATUS_TITR;
+ evts |= KOMEDA_ERR_TITR;
+ }
+ if (tbu_status & LPU_TBU_STATUS_TEMR) {
+ restore |= LPU_TBU_STATUS_TEMR;
+ evts |= KOMEDA_ERR_TEMR;
+ }
+ if (tbu_status & LPU_TBU_STATUS_TTF) {
+ restore |= LPU_TBU_STATUS_TTF;
+ evts |= KOMEDA_ERR_TTF;
+ }
+ if (restore != 0)
+ malidp_write32_mask(reg, LPU_TBU_STATUS, restore, 0);
+ }
+
+ malidp_write32(reg, BLK_IRQ_CLEAR, raw_status);
+ return evts;
+}
+
+static u64 get_cu_event(struct d71_pipeline *d71_pipeline)
+{
+ u32 __iomem *reg = d71_pipeline->cu_addr;
+ u32 status, raw_status;
+ u64 evts = 0ULL;
+
+ raw_status = malidp_read32(reg, BLK_IRQ_RAW_STATUS);
+ if (raw_status & CU_IRQ_OVR)
+ evts |= KOMEDA_EVENT_OVR;
+
+ if (raw_status & (CU_IRQ_ERR | CU_IRQ_OVR)) {
+ status = malidp_read32(reg, BLK_STATUS) & 0x7FFFFFFF;
+ if (status & CU_STATUS_CPE)
+ evts |= KOMEDA_ERR_CPE;
+ if (status & CU_STATUS_ZME)
+ evts |= KOMEDA_ERR_ZME;
+ if (status & CU_STATUS_CFGE)
+ evts |= KOMEDA_ERR_CFGE;
+ if (status)
+ malidp_write32_mask(reg, BLK_STATUS, status, 0);
+ }
+
+ malidp_write32(reg, BLK_IRQ_CLEAR, raw_status);
+
+ return evts;
+}
+
+static u64 get_dou_event(struct d71_pipeline *d71_pipeline)
+{
+ u32 __iomem *reg = d71_pipeline->dou_addr;
+ u32 status, raw_status;
+ u64 evts = 0ULL;
+
+ raw_status = malidp_read32(reg, BLK_IRQ_RAW_STATUS);
+ if (raw_status & DOU_IRQ_PL0)
+ evts |= KOMEDA_EVENT_VSYNC;
+ if (raw_status & DOU_IRQ_UND)
+ evts |= KOMEDA_EVENT_URUN;
+
+ if (raw_status & (DOU_IRQ_ERR | DOU_IRQ_UND)) {
+ u32 restore = 0;
+
+ status = malidp_read32(reg, BLK_STATUS);
+ if (status & DOU_STATUS_DRIFTTO) {
+ restore |= DOU_STATUS_DRIFTTO;
+ evts |= KOMEDA_ERR_DRIFTTO;
+ }
+ if (status & DOU_STATUS_FRAMETO) {
+ restore |= DOU_STATUS_FRAMETO;
+ evts |= KOMEDA_ERR_FRAMETO;
+ }
+ if (status & DOU_STATUS_TETO) {
+ restore |= DOU_STATUS_TETO;
+ evts |= KOMEDA_ERR_TETO;
+ }
+ if (status & DOU_STATUS_CSCE) {
+ restore |= DOU_STATUS_CSCE;
+ evts |= KOMEDA_ERR_CSCE;
+ }
+
+ if (restore != 0)
+ malidp_write32_mask(reg, BLK_STATUS, restore, 0);
+ }
+
+ malidp_write32(reg, BLK_IRQ_CLEAR, raw_status);
+ return evts;
+}
+
+static u64 get_pipeline_event(struct d71_pipeline *d71_pipeline, u32 gcu_status)
+{
+ u32 evts = 0ULL;
+
+ if (gcu_status & (GLB_IRQ_STATUS_LPU0 | GLB_IRQ_STATUS_LPU1))
+ evts |= get_lpu_event(d71_pipeline);
+
+ if (gcu_status & (GLB_IRQ_STATUS_CU0 | GLB_IRQ_STATUS_CU1))
+ evts |= get_cu_event(d71_pipeline);
+
+ if (gcu_status & (GLB_IRQ_STATUS_DOU0 | GLB_IRQ_STATUS_DOU1))
+ evts |= get_dou_event(d71_pipeline);
+
+ return evts;
+}
+
+static irqreturn_t
+d71_irq_handler(struct komeda_dev *mdev, struct komeda_events *evts)
+{
+ struct d71_dev *d71 = mdev->chip_data;
+ u32 status, gcu_status, raw_status;
+
+ gcu_status = malidp_read32(d71->gcu_addr, GLB_IRQ_STATUS);
+
+ if (gcu_status & GLB_IRQ_STATUS_GCU) {
+ raw_status = malidp_read32(d71->gcu_addr, BLK_IRQ_RAW_STATUS);
+ if (raw_status & GCU_IRQ_CVAL0)
+ evts->pipes[0] |= KOMEDA_EVENT_FLIP;
+ if (raw_status & GCU_IRQ_CVAL1)
+ evts->pipes[1] |= KOMEDA_EVENT_FLIP;
+ if (raw_status & GCU_IRQ_ERR) {
+ status = malidp_read32(d71->gcu_addr, BLK_STATUS);
+ if (status & GCU_STATUS_MERR) {
+ evts->global |= KOMEDA_ERR_MERR;
+ malidp_write32_mask(d71->gcu_addr, BLK_STATUS,
+ GCU_STATUS_MERR, 0);
+ }
+ }
+
+ malidp_write32(d71->gcu_addr, BLK_IRQ_CLEAR, raw_status);
+ }
+
+ if (gcu_status & GLB_IRQ_STATUS_PIPE0)
+ evts->pipes[0] |= get_pipeline_event(d71->pipes[0], gcu_status);
+
+ if (gcu_status & GLB_IRQ_STATUS_PIPE1)
+ evts->pipes[1] |= get_pipeline_event(d71->pipes[1], gcu_status);
+
+ return gcu_status ? IRQ_HANDLED : IRQ_NONE;
+}
+
+#define ENABLED_GCU_IRQS (GCU_IRQ_CVAL0 | GCU_IRQ_CVAL1 | \
+ GCU_IRQ_MODE | GCU_IRQ_ERR)
+#define ENABLED_LPU_IRQS (LPU_IRQ_IBSY | LPU_IRQ_ERR | LPU_IRQ_EOW)
+#define ENABLED_CU_IRQS (CU_IRQ_OVR | CU_IRQ_ERR)
+#define ENABLED_DOU_IRQS (DOU_IRQ_UND | DOU_IRQ_ERR)
+
+static int d71_enable_irq(struct komeda_dev *mdev)
+{
+ struct d71_dev *d71 = mdev->chip_data;
+ struct d71_pipeline *pipe;
+ u32 i;
+
+ malidp_write32_mask(d71->gcu_addr, BLK_IRQ_MASK,
+ ENABLED_GCU_IRQS, ENABLED_GCU_IRQS);
+ for (i = 0; i < d71->num_pipelines; i++) {
+ pipe = d71->pipes[i];
+ malidp_write32_mask(pipe->cu_addr, BLK_IRQ_MASK,
+ ENABLED_CU_IRQS, ENABLED_CU_IRQS);
+ malidp_write32_mask(pipe->lpu_addr, BLK_IRQ_MASK,
+ ENABLED_LPU_IRQS, ENABLED_LPU_IRQS);
+ malidp_write32_mask(pipe->dou_addr, BLK_IRQ_MASK,
+ ENABLED_DOU_IRQS, ENABLED_DOU_IRQS);
+ }
+ return 0;
+}
+
+static int d71_disable_irq(struct komeda_dev *mdev)
+{
+ struct d71_dev *d71 = mdev->chip_data;
+ struct d71_pipeline *pipe;
+ u32 i;
+
+ malidp_write32_mask(d71->gcu_addr, BLK_IRQ_MASK, ENABLED_GCU_IRQS, 0);
+ for (i = 0; i < d71->num_pipelines; i++) {
+ pipe = d71->pipes[i];
+ malidp_write32_mask(pipe->cu_addr, BLK_IRQ_MASK,
+ ENABLED_CU_IRQS, 0);
+ malidp_write32_mask(pipe->lpu_addr, BLK_IRQ_MASK,
+ ENABLED_LPU_IRQS, 0);
+ malidp_write32_mask(pipe->dou_addr, BLK_IRQ_MASK,
+ ENABLED_DOU_IRQS, 0);
+ }
+ return 0;
+}
+
+static void d71_on_off_vblank(struct komeda_dev *mdev, int master_pipe, bool on)
+{
+ struct d71_dev *d71 = mdev->chip_data;
+ struct d71_pipeline *pipe = d71->pipes[master_pipe];
+
+ malidp_write32_mask(pipe->dou_addr, BLK_IRQ_MASK,
+ DOU_IRQ_PL0, on ? DOU_IRQ_PL0 : 0);
+}
+
+static int to_d71_opmode(int core_mode)
+{
+ switch (core_mode) {
+ case KOMEDA_MODE_DISP0:
+ return DO0_ACTIVE_MODE;
+ case KOMEDA_MODE_DISP1:
+ return DO1_ACTIVE_MODE;
+ case KOMEDA_MODE_DUAL_DISP:
+ return DO01_ACTIVE_MODE;
+ case KOMEDA_MODE_INACTIVE:
+ return INACTIVE_MODE;
+ default:
+ WARN(1, "Unknown operation mode");
+ return INACTIVE_MODE;
+ }
+}
+
+static int d71_change_opmode(struct komeda_dev *mdev, int new_mode)
+{
+ struct d71_dev *d71 = mdev->chip_data;
+ u32 opmode = to_d71_opmode(new_mode);
+ int ret;
+
+ malidp_write32_mask(d71->gcu_addr, BLK_CONTROL, 0x7, opmode);
+
+ ret = dp_wait_cond(((malidp_read32(d71->gcu_addr, BLK_CONTROL) & 0x7) == opmode),
+ 100, 1000, 10000);
+
+ return ret > 0 ? 0 : -ETIMEDOUT;
+}
+
+static void d71_flush(struct komeda_dev *mdev,
+ int master_pipe, u32 active_pipes)
+{
+ struct d71_dev *d71 = mdev->chip_data;
+ u32 reg_offset = (master_pipe == 0) ?
+ GCU_CONFIG_VALID0 : GCU_CONFIG_VALID1;
+
+ malidp_write32(d71->gcu_addr, reg_offset, GCU_CONFIG_CVAL);
+}
+
+static int d71_reset(struct d71_dev *d71)
+{
+ u32 __iomem *gcu = d71->gcu_addr;
+ int ret;
+
+ malidp_write32_mask(gcu, BLK_CONTROL,
+ GCU_CONTROL_SRST, GCU_CONTROL_SRST);
+
+ ret = dp_wait_cond(!(malidp_read32(gcu, BLK_CONTROL) & GCU_CONTROL_SRST),
+ 100, 1000, 10000);
+
+ return ret > 0 ? 0 : -ETIMEDOUT;
+}
+
+void d71_read_block_header(u32 __iomem *reg, struct block_header *blk)
+{
+ int i;
+
+ blk->block_info = malidp_read32(reg, BLK_BLOCK_INFO);
+ if (BLOCK_INFO_BLK_TYPE(blk->block_info) == D71_BLK_TYPE_RESERVED)
+ return;
+
+ blk->pipeline_info = malidp_read32(reg, BLK_PIPELINE_INFO);
+
+ /* get valid input and output ids */
+ for (i = 0; i < PIPELINE_INFO_N_VALID_INPUTS(blk->pipeline_info); i++)
+ blk->input_ids[i] = malidp_read32(reg + i, BLK_VALID_INPUT_ID0);
+ for (i = 0; i < PIPELINE_INFO_N_OUTPUTS(blk->pipeline_info); i++)
+ blk->output_ids[i] = malidp_read32(reg + i, BLK_OUTPUT_ID0);
+}
+
+static void d71_cleanup(struct komeda_dev *mdev)
+{
+ struct d71_dev *d71 = mdev->chip_data;
+
+ if (!d71)
+ return;
+
+ devm_kfree(mdev->dev, d71);
+ mdev->chip_data = NULL;
+}
static int d71_enum_resources(struct komeda_dev *mdev)
{
- /* TODO add enum resources */
- return -1;
+ struct d71_dev *d71;
+ struct komeda_pipeline *pipe;
+ struct block_header blk;
+ u32 __iomem *blk_base;
+ u32 i, value, offset;
+ int err;
+
+ d71 = devm_kzalloc(mdev->dev, sizeof(*d71), GFP_KERNEL);
+ if (!d71)
+ return -ENOMEM;
+
+ mdev->chip_data = d71;
+ d71->mdev = mdev;
+ d71->gcu_addr = mdev->reg_base;
+ d71->periph_addr = mdev->reg_base + (D71_BLOCK_OFFSET_PERIPH >> 2);
+
+ err = d71_reset(d71);
+ if (err) {
+ DRM_ERROR("Fail to reset d71 device.\n");
+ goto err_cleanup;
+ }
+
+ /* probe GCU */
+ value = malidp_read32(d71->gcu_addr, GLB_CORE_INFO);
+ d71->num_blocks = value & 0xFF;
+ d71->num_pipelines = (value >> 8) & 0x7;
+
+ if (d71->num_pipelines > D71_MAX_PIPELINE) {
+ DRM_ERROR("d71 supports %d pipelines, but got: %d.\n",
+ D71_MAX_PIPELINE, d71->num_pipelines);
+ err = -EINVAL;
+ goto err_cleanup;
+ }
+
+ /* probe PERIPH */
+ value = malidp_read32(d71->periph_addr, BLK_BLOCK_INFO);
+ if (BLOCK_INFO_BLK_TYPE(value) != D71_BLK_TYPE_PERIPH) {
+ DRM_ERROR("access blk periph but got blk: %d.\n",
+ BLOCK_INFO_BLK_TYPE(value));
+ err = -EINVAL;
+ goto err_cleanup;
+ }
+
+ value = malidp_read32(d71->periph_addr, PERIPH_CONFIGURATION_ID);
+
+ d71->max_line_size = value & PERIPH_MAX_LINE_SIZE ? 4096 : 2048;
+ d71->max_vsize = 4096;
+ d71->num_rich_layers = value & PERIPH_NUM_RICH_LAYERS ? 2 : 1;
+ d71->supports_dual_link = value & PERIPH_SPLIT_EN ? true : false;
+ d71->integrates_tbu = value & PERIPH_TBU_EN ? true : false;
+
+ for (i = 0; i < d71->num_pipelines; i++) {
+ pipe = komeda_pipeline_add(mdev, sizeof(struct d71_pipeline),
+ NULL);
+ if (IS_ERR(pipe)) {
+ err = PTR_ERR(pipe);
+ goto err_cleanup;
+ }
+ d71->pipes[i] = to_d71_pipeline(pipe);
+ }
+
+ /* loop the register blks and probe */
+ i = 2; /* exclude GCU and PERIPH */
+ offset = D71_BLOCK_SIZE; /* skip GCU */
+ while (i < d71->num_blocks) {
+ blk_base = mdev->reg_base + (offset >> 2);
+
+ d71_read_block_header(blk_base, &blk);
+ if (BLOCK_INFO_BLK_TYPE(blk.block_info) != D71_BLK_TYPE_RESERVED) {
+ err = d71_probe_block(d71, &blk, blk_base);
+ if (err)
+ goto err_cleanup;
+ i++;
+ }
+
+ offset += D71_BLOCK_SIZE;
+ }
+
+ DRM_DEBUG("total %d (out of %d) blocks are found.\n",
+ i, d71->num_blocks);
+
+ return 0;
+
+err_cleanup:
+ d71_cleanup(mdev);
+ return err;
}
#define __HW_ID(__group, __format) \
static struct komeda_dev_funcs d71_chip_funcs = {
.init_format_table = d71_init_fmt_tbl,
.enum_resources = d71_enum_resources,
- .cleanup = NULL,
+ .cleanup = d71_cleanup,
+ .irq_handler = d71_irq_handler,
+ .enable_irq = d71_enable_irq,
+ .disable_irq = d71_disable_irq,
+ .on_off_vblank = d71_on_off_vblank,
+ .change_opmode = d71_change_opmode,
+ .flush = d71_flush,
};
-#define GLB_ARCH_ID 0x000
-#define GLB_CORE_ID 0x004
-#define GLB_CORE_INFO 0x008
-
struct komeda_dev_funcs *
d71_identify(u32 __iomem *reg_base, struct komeda_chip_info *chip)
{
chip->arch_id = malidp_read32(reg_base, GLB_ARCH_ID);
chip->core_id = malidp_read32(reg_base, GLB_CORE_ID);
chip->core_info = malidp_read32(reg_base, GLB_CORE_INFO);
+ chip->bus_width = D71_BUS_WIDTH_16_BYTES;
return &d71_chip_funcs;
}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * (C) COPYRIGHT 2018 ARM Limited. All rights reserved.
+ * Author: James.Qian.Wang <james.qian.wang@arm.com>
+ *
+ */
+#ifndef _D71_DEV_H_
+#define _D71_DEV_H_
+
+#include "komeda_dev.h"
+#include "komeda_pipeline.h"
+#include "d71_regs.h"
+
+struct d71_pipeline {
+ struct komeda_pipeline base;
+
+ /* d71 private pipeline blocks */
+ u32 __iomem *lpu_addr;
+ u32 __iomem *cu_addr;
+ u32 __iomem *dou_addr;
+ u32 __iomem *dou_ft_coeff_addr; /* forward transform coeffs table */
+};
+
+struct d71_dev {
+ struct komeda_dev *mdev;
+
+ int num_blocks;
+ int num_pipelines;
+ int num_rich_layers;
+ u32 max_line_size;
+ u32 max_vsize;
+ u32 supports_dual_link : 1;
+ u32 integrates_tbu : 1;
+
+ /* global register blocks */
+ u32 __iomem *gcu_addr;
+ /* scaling coeffs table */
+ u32 __iomem *glb_scl_coeff_addr[D71_MAX_GLB_SCL_COEFF];
+ u32 __iomem *periph_addr;
+
+ struct d71_pipeline *pipes[D71_MAX_PIPELINE];
+};
+
+#define to_d71_pipeline(x) container_of(x, struct d71_pipeline, base)
+
+int d71_probe_block(struct d71_dev *d71,
+ struct block_header *blk, u32 __iomem *reg);
+void d71_read_block_header(u32 __iomem *reg, struct block_header *blk);
+
+#endif /* !_D71_DEV_H_ */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * (C) COPYRIGHT 2018 ARM Limited. All rights reserved.
+ * Author: James.Qian.Wang <james.qian.wang@arm.com>
+ *
+ */
+#ifndef _D71_REG_H_
+#define _D71_REG_H_
+
+/* Common block registers offset */
+#define BLK_BLOCK_INFO 0x000
+#define BLK_PIPELINE_INFO 0x004
+#define BLK_VALID_INPUT_ID0 0x020
+#define BLK_OUTPUT_ID0 0x060
+#define BLK_INPUT_ID0 0x080
+#define BLK_IRQ_RAW_STATUS 0x0A0
+#define BLK_IRQ_CLEAR 0x0A4
+#define BLK_IRQ_MASK 0x0A8
+#define BLK_IRQ_STATUS 0x0AC
+#define BLK_STATUS 0x0B0
+#define BLK_INFO 0x0C0
+#define BLK_CONTROL 0x0D0
+#define BLK_SIZE 0x0D4
+#define BLK_IN_SIZE 0x0E0
+
+#define BLK_P0_PTR_LOW 0x100
+#define BLK_P0_PTR_HIGH 0x104
+#define BLK_P0_STRIDE 0x108
+#define BLK_P1_PTR_LOW 0x110
+#define BLK_P1_PTR_HIGH 0x114
+#define BLK_P1_STRIDE 0x118
+#define BLK_P2_PTR_LOW 0x120
+#define BLK_P2_PTR_HIGH 0x124
+
+#define BLOCK_INFO_N_SUBBLKS(x) ((x) & 0x000F)
+#define BLOCK_INFO_BLK_ID(x) (((x) & 0x00F0) >> 4)
+#define BLOCK_INFO_BLK_TYPE(x) (((x) & 0xFF00) >> 8)
+#define BLOCK_INFO_INPUT_ID(x) ((x) & 0xFFF0)
+#define BLOCK_INFO_TYPE_ID(x) (((x) & 0x0FF0) >> 4)
+
+#define PIPELINE_INFO_N_OUTPUTS(x) ((x) & 0x000F)
+#define PIPELINE_INFO_N_VALID_INPUTS(x) (((x) & 0x0F00) >> 8)
+
+/* Common block control register bits */
+#define BLK_CTRL_EN BIT(0)
+/* Common size macro */
+#define HV_SIZE(h, v) (((h) & 0x1FFF) + (((v) & 0x1FFF) << 16))
+#define HV_OFFSET(h, v) (((h) & 0xFFF) + (((v) & 0xFFF) << 16))
+#define HV_CROP(h, v) (((h) & 0xFFF) + (((v) & 0xFFF) << 16))
+
+/* AD_CONTROL register */
+#define AD_CONTROL 0x160
+
+/* AD_CONTROL register bits */
+#define AD_AEN BIT(0)
+#define AD_YT BIT(1)
+#define AD_BS BIT(2)
+#define AD_WB BIT(3)
+#define AD_TH BIT(4)
+
+/* Global Control Unit */
+#define GLB_ARCH_ID 0x000
+#define GLB_CORE_ID 0x004
+#define GLB_CORE_INFO 0x008
+#define GLB_IRQ_STATUS 0x010
+
+#define GCU_CONFIG_VALID0 0x0D4
+#define GCU_CONFIG_VALID1 0x0D8
+
+/* GCU_CONTROL_BITS */
+#define GCU_CONTROL_MODE(x) ((x) & 0x7)
+#define GCU_CONTROL_SRST BIT(16)
+
+/* GCU opmode */
+#define INACTIVE_MODE 0
+#define TBU_CONNECT_MODE 1
+#define TBU_DISCONNECT_MODE 2
+#define DO0_ACTIVE_MODE 3
+#define DO1_ACTIVE_MODE 4
+#define DO01_ACTIVE_MODE 5
+
+/* GLB_IRQ_STATUS bits */
+#define GLB_IRQ_STATUS_GCU BIT(0)
+#define GLB_IRQ_STATUS_LPU0 BIT(8)
+#define GLB_IRQ_STATUS_LPU1 BIT(9)
+#define GLB_IRQ_STATUS_ATU0 BIT(10)
+#define GLB_IRQ_STATUS_ATU1 BIT(11)
+#define GLB_IRQ_STATUS_ATU2 BIT(12)
+#define GLB_IRQ_STATUS_ATU3 BIT(13)
+#define GLB_IRQ_STATUS_CU0 BIT(16)
+#define GLB_IRQ_STATUS_CU1 BIT(17)
+#define GLB_IRQ_STATUS_DOU0 BIT(24)
+#define GLB_IRQ_STATUS_DOU1 BIT(25)
+
+#define GLB_IRQ_STATUS_PIPE0 (GLB_IRQ_STATUS_LPU0 |\
+ GLB_IRQ_STATUS_ATU0 |\
+ GLB_IRQ_STATUS_ATU1 |\
+ GLB_IRQ_STATUS_CU0 |\
+ GLB_IRQ_STATUS_DOU0)
+
+#define GLB_IRQ_STATUS_PIPE1 (GLB_IRQ_STATUS_LPU1 |\
+ GLB_IRQ_STATUS_ATU2 |\
+ GLB_IRQ_STATUS_ATU3 |\
+ GLB_IRQ_STATUS_CU1 |\
+ GLB_IRQ_STATUS_DOU1)
+
+#define GLB_IRQ_STATUS_ATU (GLB_IRQ_STATUS_ATU0 |\
+ GLB_IRQ_STATUS_ATU1 |\
+ GLB_IRQ_STATUS_ATU2 |\
+ GLB_IRQ_STATUS_ATU3)
+
+/* GCU_IRQ_BITS */
+#define GCU_IRQ_CVAL0 BIT(0)
+#define GCU_IRQ_CVAL1 BIT(1)
+#define GCU_IRQ_MODE BIT(4)
+#define GCU_IRQ_ERR BIT(11)
+
+/* GCU_STATUS_BITS */
+#define GCU_STATUS_MODE(x) ((x) & 0x7)
+#define GCU_STATUS_MERR BIT(4)
+#define GCU_STATUS_TCS0 BIT(8)
+#define GCU_STATUS_TCS1 BIT(9)
+#define GCU_STATUS_ACTIVE BIT(31)
+
+/* GCU_CONFIG_VALIDx BITS */
+#define GCU_CONFIG_CVAL BIT(0)
+
+/* PERIPHERAL registers */
+#define PERIPH_MAX_LINE_SIZE BIT(0)
+#define PERIPH_NUM_RICH_LAYERS BIT(4)
+#define PERIPH_SPLIT_EN BIT(8)
+#define PERIPH_TBU_EN BIT(12)
+#define PERIPH_AFBC_DMA_EN BIT(16)
+#define PERIPH_CONFIGURATION_ID 0x1D4
+
+/* LPU register */
+#define LPU_TBU_STATUS 0x0B4
+#define LPU_RAXI_CONTROL 0x0D0
+#define LPU_WAXI_CONTROL 0x0D4
+#define LPU_TBU_CONTROL 0x0D8
+
+/* LPU_xAXI_CONTROL_BITS */
+#define TO_RAXI_AOUTSTDCAPB(x) (x)
+#define TO_RAXI_BOUTSTDCAPB(x) ((x) << 8)
+#define TO_RAXI_BEN(x) ((x) << 15)
+#define TO_xAXI_BURSTLEN(x) ((x) << 16)
+#define TO_xAXI_AxQOS(x) ((x) << 24)
+#define TO_xAXI_ORD(x) ((x) << 31)
+#define TO_WAXI_OUTSTDCAPB(x) (x)
+
+#define RAXI_AOUTSTDCAPB_MASK 0x7F
+#define RAXI_BOUTSTDCAPB_MASK 0x7F00
+#define RAXI_BEN_MASK BIT(15)
+#define xAXI_BURSTLEN_MASK 0x3F0000
+#define xAXI_AxQOS_MASK 0xF000000
+#define xAXI_ORD_MASK BIT(31)
+#define WAXI_OUTSTDCAPB_MASK 0x3F
+
+/* LPU_TBU_CONTROL BITS */
+#define TO_TBU_DOUTSTDCAPB(x) (x)
+#define TBU_DOUTSTDCAPB_MASK 0x3F
+
+/* LPU_IRQ_BITS */
+#define LPU_IRQ_IBSY BIT(10)
+#define LPU_IRQ_ERR BIT(11)
+#define LPU_IRQ_EOW BIT(12)
+#define LPU_IRQ_PL0 BIT(13)
+
+/* LPU_STATUS_BITS */
+#define LPU_STATUS_AXIED(x) ((x) & 0xF)
+#define LPU_STATUS_AXIE BIT(4)
+#define LPU_STATUS_AXIRP BIT(5)
+#define LPU_STATUS_AXIWP BIT(6)
+#define LPU_STATUS_ACE0 BIT(16)
+#define LPU_STATUS_ACE1 BIT(17)
+#define LPU_STATUS_ACE2 BIT(18)
+#define LPU_STATUS_ACE3 BIT(19)
+#define LPU_STATUS_ACTIVE BIT(31)
+
+#define AXIEID_MASK 0xF
+#define AXIE_MASK LPU_STATUS_AXIE
+#define AXIRP_MASK LPU_STATUS_AXIRP
+#define AXIWP_MASK LPU_STATUS_AXIWP
+
+#define FROM_AXIEID(reg) ((reg) & AXIEID_MASK)
+#define TO_AXIE(x) ((x) << 4)
+#define FROM_AXIRP(reg) (((reg) & AXIRP_MASK) >> 5)
+#define FROM_AXIWP(reg) (((reg) & AXIWP_MASK) >> 6)
+
+/* LPU_TBU_STATUS_BITS */
+#define LPU_TBU_STATUS_TCF BIT(1)
+#define LPU_TBU_STATUS_TTNG BIT(2)
+#define LPU_TBU_STATUS_TITR BIT(8)
+#define LPU_TBU_STATUS_TEMR BIT(16)
+#define LPU_TBU_STATUS_TTF BIT(31)
+
+/* LPU_TBU_CONTROL BITS */
+#define LPU_TBU_CTRL_TLBPEN BIT(16)
+
+/* CROSSBAR CONTROL BITS */
+#define CBU_INPUT_CTRL_EN BIT(0)
+#define CBU_NUM_INPUT_IDS 5
+#define CBU_NUM_OUTPUT_IDS 5
+
+/* CU register */
+#define CU_BG_COLOR 0x0DC
+#define CU_INPUT0_SIZE 0x0E0
+#define CU_INPUT0_OFFSET 0x0E4
+#define CU_INPUT0_CONTROL 0x0E8
+#define CU_INPUT1_SIZE 0x0F0
+#define CU_INPUT1_OFFSET 0x0F4
+#define CU_INPUT1_CONTROL 0x0F8
+#define CU_INPUT2_SIZE 0x100
+#define CU_INPUT2_OFFSET 0x104
+#define CU_INPUT2_CONTROL 0x108
+#define CU_INPUT3_SIZE 0x110
+#define CU_INPUT3_OFFSET 0x114
+#define CU_INPUT3_CONTROL 0x118
+#define CU_INPUT4_SIZE 0x120
+#define CU_INPUT4_OFFSET 0x124
+#define CU_INPUT4_CONTROL 0x128
+
+#define CU_PER_INPUT_REGS 4
+
+#define CU_NUM_INPUT_IDS 5
+#define CU_NUM_OUTPUT_IDS 1
+
+/* CU control register bits */
+#define CU_CTRL_COPROC BIT(0)
+
+/* CU_IRQ_BITS */
+#define CU_IRQ_OVR BIT(9)
+#define CU_IRQ_ERR BIT(11)
+
+/* CU_STATUS_BITS */
+#define CU_STATUS_CPE BIT(0)
+#define CU_STATUS_ZME BIT(1)
+#define CU_STATUS_CFGE BIT(2)
+#define CU_STATUS_ACTIVE BIT(31)
+
+/* CU input control register bits */
+#define CU_INPUT_CTRL_EN BIT(0)
+#define CU_INPUT_CTRL_PAD BIT(1)
+#define CU_INPUT_CTRL_PMUL BIT(2)
+#define CU_INPUT_CTRL_ALPHA(x) (((x) & 0xFF) << 8)
+
+/* DOU register */
+
+/* DOU_IRQ_BITS */
+#define DOU_IRQ_UND BIT(8)
+#define DOU_IRQ_ERR BIT(11)
+#define DOU_IRQ_PL0 BIT(13)
+#define DOU_IRQ_PL1 BIT(14)
+
+/* DOU_STATUS_BITS */
+#define DOU_STATUS_DRIFTTO BIT(0)
+#define DOU_STATUS_FRAMETO BIT(1)
+#define DOU_STATUS_TETO BIT(2)
+#define DOU_STATUS_CSCE BIT(8)
+#define DOU_STATUS_ACTIVE BIT(31)
+
+/* Layer registers */
+#define LAYER_INFO 0x0C0
+#define LAYER_R_CONTROL 0x0D4
+#define LAYER_FMT 0x0D8
+#define LAYER_LT_COEFFTAB 0x0DC
+#define LAYER_PALPHA 0x0E4
+
+#define LAYER_YUV_RGB_COEFF0 0x130
+
+#define LAYER_AD_H_CROP 0x164
+#define LAYER_AD_V_CROP 0x168
+
+#define LAYER_RGB_RGB_COEFF0 0x170
+
+/* L_CONTROL_BITS */
+#define L_EN BIT(0)
+#define L_IT BIT(4)
+#define L_R2R BIT(5)
+#define L_FT BIT(6)
+#define L_ROT(x) (((x) & 3) << 8)
+#define L_HFLIP BIT(10)
+#define L_VFLIP BIT(11)
+#define L_TBU_EN BIT(16)
+#define L_A_RCACHE(x) (((x) & 0xF) << 28)
+#define L_ROT_R0 0
+#define L_ROT_R90 1
+#define L_ROT_R180 2
+#define L_ROT_R270 3
+
+/* LAYER_R_CONTROL BITS */
+#define LR_CHI422_BILINEAR 0
+#define LR_CHI422_REPLICATION 1
+#define LR_CHI420_JPEG (0 << 2)
+#define LR_CHI420_MPEG (1 << 2)
+
+#define L_ITSEL(x) ((x) & 0xFFF)
+#define L_FTSEL(x) (((x) & 0xFFF) << 16)
+
+#define LAYER_PER_PLANE_REGS 4
+
+/* Layer_WR registers */
+#define LAYER_WR_PROG_LINE 0x0D4
+#define LAYER_WR_FORMAT 0x0D8
+
+/* Layer_WR control bits */
+#define LW_OFM BIT(4)
+#define LW_LALPHA(x) (((x) & 0xFF) << 8)
+#define LW_A_WCACHE(x) (((x) & 0xF) << 28)
+#define LW_TBU_EN BIT(16)
+
+#define AxCACHE_MASK 0xF0000000
+
+/* Layer AXI R/W cache setting */
+#define AxCACHE_B BIT(0) /* Bufferable */
+#define AxCACHE_M BIT(1) /* Modifiable */
+#define AxCACHE_RA BIT(2) /* Read-Allocate */
+#define AxCACHE_WA BIT(3) /* Write-Allocate */
+
+/* Layer info bits */
+#define L_INFO_RF BIT(0)
+#define L_INFO_CM BIT(1)
+#define L_INFO_ABUF_SIZE(x) (((x) >> 4) & 0x7)
+
+/* Scaler registers */
+#define SC_COEFFTAB 0x0DC
+#define SC_OUT_SIZE 0x0E4
+#define SC_H_CROP 0x0E8
+#define SC_V_CROP 0x0EC
+#define SC_H_INIT_PH 0x0F0
+#define SC_H_DELTA_PH 0x0F4
+#define SC_V_INIT_PH 0x0F8
+#define SC_V_DELTA_PH 0x0FC
+#define SC_ENH_LIMITS 0x130
+#define SC_ENH_COEFF0 0x134
+
+#define SC_MAX_ENH_COEFF 9
+
+/* SC_CTRL_BITS */
+#define SC_CTRL_SCL BIT(0)
+#define SC_CTRL_LS BIT(1)
+#define SC_CTRL_AP BIT(4)
+#define SC_CTRL_IENH BIT(8)
+#define SC_CTRL_RGBSM BIT(16)
+#define SC_CTRL_ASM BIT(17)
+
+#define SC_VTSEL(vtal) ((vtal) << 16)
+
+#define SC_NUM_INPUTS_IDS 1
+#define SC_NUM_OUTPUTS_IDS 1
+
+#define MG_NUM_INPUTS_IDS 2
+#define MG_NUM_OUTPUTS_IDS 1
+
+/* Merger registers */
+#define MG_INPUT_ID0 BLK_INPUT_ID0
+#define MG_INPUT_ID1 (MG_INPUT_ID0 + 4)
+#define MG_SIZE BLK_SIZE
+
+/* Splitter registers */
+#define SP_OVERLAP_SIZE 0xD8
+
+/* Backend registers */
+#define BS_INFO 0x0C0
+#define BS_PROG_LINE 0x0D4
+#define BS_PREFETCH_LINE 0x0D8
+#define BS_BG_COLOR 0x0DC
+#define BS_ACTIVESIZE 0x0E0
+#define BS_HINTERVALS 0x0E4
+#define BS_VINTERVALS 0x0E8
+#define BS_SYNC 0x0EC
+#define BS_DRIFT_TO 0x100
+#define BS_FRAME_TO 0x104
+#define BS_TE_TO 0x108
+#define BS_T0_INTERVAL 0x110
+#define BS_T1_INTERVAL 0x114
+#define BS_T2_INTERVAL 0x118
+#define BS_CRC0_LOW 0x120
+#define BS_CRC0_HIGH 0x124
+#define BS_CRC1_LOW 0x128
+#define BS_CRC1_HIGH 0x12C
+#define BS_USER 0x130
+
+/* BS control register bits */
+#define BS_CTRL_EN BIT(0)
+#define BS_CTRL_VM BIT(1)
+#define BS_CTRL_BM BIT(2)
+#define BS_CTRL_HMASK BIT(4)
+#define BS_CTRL_VD BIT(5)
+#define BS_CTRL_TE BIT(8)
+#define BS_CTRL_TS BIT(9)
+#define BS_CTRL_TM BIT(12)
+#define BS_CTRL_DL BIT(16)
+#define BS_CTRL_SBS BIT(17)
+#define BS_CTRL_CRC BIT(18)
+#define BS_CTRL_PM BIT(20)
+
+/* BS active size/intervals */
+#define BS_H_INTVALS(hfp, hbp) (((hfp) & 0xFFF) + (((hbp) & 0x3FF) << 16))
+#define BS_V_INTVALS(vfp, vbp) (((vfp) & 0x3FFF) + (((vbp) & 0xFF) << 16))
+
+/* BS_SYNC bits */
+#define BS_SYNC_HSW(x) ((x) & 0x3FF)
+#define BS_SYNC_HSP BIT(12)
+#define BS_SYNC_VSW(x) (((x) & 0xFF) << 16)
+#define BS_SYNC_VSP BIT(28)
+
+#define BS_NUM_INPUT_IDS 0
+#define BS_NUM_OUTPUT_IDS 0
+
+/* Image process registers */
+#define IPS_DEPTH 0x0D8
+#define IPS_RGB_RGB_COEFF0 0x130
+#define IPS_RGB_YUV_COEFF0 0x170
+
+#define IPS_DEPTH_MARK 0xF
+
+/* IPS control register bits */
+#define IPS_CTRL_RGB BIT(0)
+#define IPS_CTRL_FT BIT(4)
+#define IPS_CTRL_YUV BIT(8)
+#define IPS_CTRL_CHD422 BIT(9)
+#define IPS_CTRL_CHD420 BIT(10)
+#define IPS_CTRL_LPF BIT(11)
+#define IPS_CTRL_DITH BIT(12)
+#define IPS_CTRL_CLAMP BIT(16)
+#define IPS_CTRL_SBS BIT(17)
+
+/* IPS info register bits */
+#define IPS_INFO_CHD420 BIT(10)
+
+#define IPS_NUM_INPUT_IDS 2
+#define IPS_NUM_OUTPUT_IDS 1
+
+/* FT_COEFF block registers */
+#define FT_COEFF0 0x80
+#define GLB_IT_COEFF 0x80
+
+/* GLB_SC_COEFF registers */
+#define GLB_SC_COEFF_ADDR 0x0080
+#define GLB_SC_COEFF_DATA 0x0084
+#define GLB_LT_COEFF_DATA 0x0080
+
+#define GLB_SC_COEFF_MAX_NUM 1024
+#define GLB_LT_COEFF_NUM 65
+/* GLB_SC_ADDR */
+#define SC_COEFF_R_ADDR BIT(18)
+#define SC_COEFF_G_ADDR BIT(17)
+#define SC_COEFF_B_ADDR BIT(16)
+
+#define SC_COEFF_DATA(x, y) (((y) & 0xFFFF) | (((x) & 0xFFFF) << 16))
+
+enum d71_blk_type {
+ D71_BLK_TYPE_GCU = 0x00,
+ D71_BLK_TYPE_LPU = 0x01,
+ D71_BLK_TYPE_CU = 0x02,
+ D71_BLK_TYPE_DOU = 0x03,
+ D71_BLK_TYPE_AEU = 0x04,
+ D71_BLK_TYPE_GLB_LT_COEFF = 0x05,
+ D71_BLK_TYPE_GLB_SCL_COEFF = 0x06, /* SH/SV scaler coeff */
+ D71_BLK_TYPE_GLB_SC_COEFF = 0x07,
+ D71_BLK_TYPE_PERIPH = 0x08,
+ D71_BLK_TYPE_LPU_TRUSTED = 0x09,
+ D71_BLK_TYPE_AEU_TRUSTED = 0x0A,
+ D71_BLK_TYPE_LPU_LAYER = 0x10,
+ D71_BLK_TYPE_LPU_WB_LAYER = 0x11,
+ D71_BLK_TYPE_CU_SPLITTER = 0x20,
+ D71_BLK_TYPE_CU_SCALER = 0x21,
+ D71_BLK_TYPE_CU_MERGER = 0x22,
+ D71_BLK_TYPE_DOU_IPS = 0x30,
+ D71_BLK_TYPE_DOU_BS = 0x31,
+ D71_BLK_TYPE_DOU_FT_COEFF = 0x32,
+ D71_BLK_TYPE_AEU_DS = 0x40,
+ D71_BLK_TYPE_AEU_AES = 0x41,
+ D71_BLK_TYPE_RESERVED = 0xFF
+};
+
+/* Constant of components */
+#define D71_MAX_PIPELINE 2
+#define D71_PIPELINE_MAX_SCALERS 2
+#define D71_PIPELINE_MAX_LAYERS 4
+
+#define D71_MAX_GLB_IT_COEFF 3
+#define D71_MAX_GLB_SCL_COEFF 4
+
+#define D71_MAX_LAYERS_PER_LPU 4
+#define D71_BLOCK_MAX_INPUT 9
+#define D71_BLOCK_MAX_OUTPUT 5
+#define D71_MAX_SC_PER_CU 2
+
+#define D71_BLOCK_OFFSET_PERIPH 0xFE00
+#define D71_BLOCK_SIZE 0x0200
+
+#define D71_DEFAULT_PREPRETCH_LINE 5
+#define D71_BUS_WIDTH_16_BYTES 16
+
+#define D71_MIN_LINE_SIZE 64
+#define D71_MIN_VERTICAL_SIZE 64
+#define D71_SC_MIN_LIN_SIZE 4
+#define D71_SC_MIN_VERTICAL_SIZE 4
+#define D71_SC_MAX_LIN_SIZE 2048
+#define D71_SC_MAX_VERTICAL_SIZE 4096
+
+#define D71_SC_MAX_UPSCALING 64
+#define D71_SC_MAX_DOWNSCALING 6
+#define D71_SC_SPLIT_OVERLAP 8
+#define D71_SC_ENH_SPLIT_OVERLAP 1
+
+#define D71_MG_MIN_MERGED_SIZE 4
+#define D71_MG_MAX_MERGED_HSIZE 4032
+#define D71_MG_MAX_MERGED_VSIZE 4096
+
+#define D71_PALPHA_DEF_MAP 0xFFAA5500
+#define D71_LAYER_CONTROL_DEFAULT 0x30000000
+#define D71_WB_LAYER_CONTROL_DEFAULT 0x3000FF00
+#define D71_BS_CONTROL_DEFAULT 0x00000002
+
+struct block_header {
+ u32 block_info;
+ u32 pipeline_info;
+ u32 input_ids[D71_BLOCK_MAX_INPUT];
+ u32 output_ids[D71_BLOCK_MAX_OUTPUT];
+};
+
+static inline u32 get_block_type(struct block_header *blk)
+{
+ return BLOCK_INFO_BLK_TYPE(blk->block_info);
+}
+
+#endif /* !_D71_REG_H_ */
#include "komeda_dev.h"
#include "komeda_kms.h"
-struct drm_crtc_helper_funcs komeda_crtc_helper_funcs = {
+/**
+ * komeda_crtc_atomic_check - build display output data flow
+ * @crtc: DRM crtc
+ * @state: the crtc state object
+ *
+ * crtc_atomic_check is the final check stage, so beside build a display data
+ * pipeline according to the crtc_state, but still needs to release or disable
+ * the unclaimed pipeline resources.
+ *
+ * RETURNS:
+ * Zero for success or -errno
+ */
+static int
+komeda_crtc_atomic_check(struct drm_crtc *crtc,
+ struct drm_crtc_state *state)
+{
+ struct komeda_crtc *kcrtc = to_kcrtc(crtc);
+ struct komeda_crtc_state *kcrtc_st = to_kcrtc_st(state);
+ int err;
+
+ if (state->active) {
+ err = komeda_build_display_data_flow(kcrtc, kcrtc_st);
+ if (err)
+ return err;
+ }
+
+ /* release unclaimed pipeline resources */
+ err = komeda_release_unclaimed_resources(kcrtc->master, kcrtc_st);
+ if (err)
+ return err;
+
+ return 0;
+}
+
+static u32 komeda_calc_mclk(struct komeda_crtc_state *kcrtc_st)
+{
+ unsigned long mclk = kcrtc_st->base.adjusted_mode.clock * 1000;
+
+ return mclk;
+}
+
+/* For active a crtc, mainly need two parts of preparation
+ * 1. adjust display operation mode.
+ * 2. enable needed clk
+ */
+static int
+komeda_crtc_prepare(struct komeda_crtc *kcrtc)
+{
+ struct komeda_dev *mdev = kcrtc->base.dev->dev_private;
+ struct komeda_pipeline *master = kcrtc->master;
+ struct komeda_crtc_state *kcrtc_st = to_kcrtc_st(kcrtc->base.state);
+ unsigned long pxlclk_rate = kcrtc_st->base.adjusted_mode.clock * 1000;
+ u32 new_mode;
+ int err;
+
+ mutex_lock(&mdev->lock);
+
+ new_mode = mdev->dpmode | BIT(master->id);
+ if (WARN_ON(new_mode == mdev->dpmode)) {
+ err = 0;
+ goto unlock;
+ }
+
+ err = mdev->funcs->change_opmode(mdev, new_mode);
+ if (err) {
+ DRM_ERROR("failed to change opmode: 0x%x -> 0x%x.\n,",
+ mdev->dpmode, new_mode);
+ goto unlock;
+ }
+
+ mdev->dpmode = new_mode;
+ /* Only need to enable mclk on single display mode, but no need to
+ * enable mclk it on dual display mode, since the dual mode always
+ * switch from single display mode, the mclk already enabled, no need
+ * to enable it again.
+ */
+ if (new_mode != KOMEDA_MODE_DUAL_DISP) {
+ err = clk_set_rate(mdev->mclk, komeda_calc_mclk(kcrtc_st));
+ if (err)
+ DRM_ERROR("failed to set mclk.\n");
+ err = clk_prepare_enable(mdev->mclk);
+ if (err)
+ DRM_ERROR("failed to enable mclk.\n");
+ }
+
+ err = clk_prepare_enable(master->aclk);
+ if (err)
+ DRM_ERROR("failed to enable axi clk for pipe%d.\n", master->id);
+ err = clk_set_rate(master->pxlclk, pxlclk_rate);
+ if (err)
+ DRM_ERROR("failed to set pxlclk for pipe%d\n", master->id);
+ err = clk_prepare_enable(master->pxlclk);
+ if (err)
+ DRM_ERROR("failed to enable pxl clk for pipe%d.\n", master->id);
+
+unlock:
+ mutex_unlock(&mdev->lock);
+
+ return err;
+}
+
+static int
+komeda_crtc_unprepare(struct komeda_crtc *kcrtc)
+{
+ struct komeda_dev *mdev = kcrtc->base.dev->dev_private;
+ struct komeda_pipeline *master = kcrtc->master;
+ u32 new_mode;
+ int err;
+
+ mutex_lock(&mdev->lock);
+
+ new_mode = mdev->dpmode & (~BIT(master->id));
+
+ if (WARN_ON(new_mode == mdev->dpmode)) {
+ err = 0;
+ goto unlock;
+ }
+
+ err = mdev->funcs->change_opmode(mdev, new_mode);
+ if (err) {
+ DRM_ERROR("failed to change opmode: 0x%x -> 0x%x.\n,",
+ mdev->dpmode, new_mode);
+ goto unlock;
+ }
+
+ mdev->dpmode = new_mode;
+
+ clk_disable_unprepare(master->pxlclk);
+ clk_disable_unprepare(master->aclk);
+ if (new_mode == KOMEDA_MODE_INACTIVE)
+ clk_disable_unprepare(mdev->mclk);
+
+unlock:
+ mutex_unlock(&mdev->lock);
+
+ return err;
+}
+
+void komeda_crtc_handle_event(struct komeda_crtc *kcrtc,
+ struct komeda_events *evts)
+{
+ struct drm_crtc *crtc = &kcrtc->base;
+ u32 events = evts->pipes[kcrtc->master->id];
+
+ if (events & KOMEDA_EVENT_VSYNC)
+ drm_crtc_handle_vblank(crtc);
+
+ /* will handle it together with the write back support */
+ if (events & KOMEDA_EVENT_EOW)
+ DRM_DEBUG("EOW.\n");
+
+ if (events & KOMEDA_EVENT_FLIP) {
+ unsigned long flags;
+ struct drm_pending_vblank_event *event;
+
+ spin_lock_irqsave(&crtc->dev->event_lock, flags);
+ if (kcrtc->disable_done) {
+ complete_all(kcrtc->disable_done);
+ kcrtc->disable_done = NULL;
+ } else if (crtc->state->event) {
+ event = crtc->state->event;
+ /*
+ * Consume event before notifying drm core that flip
+ * happened.
+ */
+ crtc->state->event = NULL;
+ drm_crtc_send_vblank_event(crtc, event);
+ } else {
+ DRM_WARN("CRTC[%d]: FLIP happen but no pending commit.\n",
+ drm_crtc_index(&kcrtc->base));
+ }
+ spin_unlock_irqrestore(&crtc->dev->event_lock, flags);
+ }
+}
+
+static void
+komeda_crtc_do_flush(struct drm_crtc *crtc,
+ struct drm_crtc_state *old)
+{
+ struct komeda_crtc *kcrtc = to_kcrtc(crtc);
+ struct komeda_crtc_state *kcrtc_st = to_kcrtc_st(crtc->state);
+ struct komeda_dev *mdev = kcrtc->base.dev->dev_private;
+ struct komeda_pipeline *master = kcrtc->master;
+
+ DRM_DEBUG_ATOMIC("CRTC%d_FLUSH: active_pipes: 0x%x, affected: 0x%x.\n",
+ drm_crtc_index(crtc),
+ kcrtc_st->active_pipes, kcrtc_st->affected_pipes);
+
+ /* step 1: update the pipeline/component state to HW */
+ if (has_bit(master->id, kcrtc_st->affected_pipes))
+ komeda_pipeline_update(master, old->state);
+
+ /* step 2: notify the HW to kickoff the update */
+ mdev->funcs->flush(mdev, master->id, kcrtc_st->active_pipes);
+}
+
+static void
+komeda_crtc_atomic_enable(struct drm_crtc *crtc,
+ struct drm_crtc_state *old)
+{
+ komeda_crtc_prepare(to_kcrtc(crtc));
+ drm_crtc_vblank_on(crtc);
+ komeda_crtc_do_flush(crtc, old);
+}
+
+static void
+komeda_crtc_atomic_disable(struct drm_crtc *crtc,
+ struct drm_crtc_state *old)
+{
+ struct komeda_crtc *kcrtc = to_kcrtc(crtc);
+ struct komeda_crtc_state *old_st = to_kcrtc_st(old);
+ struct komeda_dev *mdev = crtc->dev->dev_private;
+ struct komeda_pipeline *master = kcrtc->master;
+ struct completion *disable_done = &crtc->state->commit->flip_done;
+ struct completion temp;
+ int timeout;
+
+ DRM_DEBUG_ATOMIC("CRTC%d_DISABLE: active_pipes: 0x%x, affected: 0x%x.\n",
+ drm_crtc_index(crtc),
+ old_st->active_pipes, old_st->affected_pipes);
+
+ if (has_bit(master->id, old_st->active_pipes))
+ komeda_pipeline_disable(master, old->state);
+
+ /* crtc_disable has two scenarios according to the state->active switch.
+ * 1. active -> inactive
+ * this commit is a disable commit. and the commit will be finished
+ * or done after the disable operation. on this case we can directly
+ * use the crtc->state->event to tracking the HW disable operation.
+ * 2. active -> active
+ * the crtc->commit is not for disable, but a modeset operation when
+ * crtc is active, such commit actually has been completed by 3
+ * DRM operations:
+ * crtc_disable, update_planes(crtc_flush), crtc_enable
+ * so on this case the crtc->commit is for the whole process.
+ * we can not use it for tracing the disable, we need a temporary
+ * flip_done for tracing the disable. and crtc->state->event for
+ * the crtc_enable operation.
+ * That's also the reason why skip modeset commit in
+ * komeda_crtc_atomic_flush()
+ */
+ if (crtc->state->active) {
+ struct komeda_pipeline_state *pipe_st;
+ /* clear the old active_comps to zero */
+ pipe_st = komeda_pipeline_get_old_state(master, old->state);
+ pipe_st->active_comps = 0;
+
+ init_completion(&temp);
+ kcrtc->disable_done = &temp;
+ disable_done = &temp;
+ }
+
+ mdev->funcs->flush(mdev, master->id, 0);
+
+ /* wait the disable take affect.*/
+ timeout = wait_for_completion_timeout(disable_done, HZ);
+ if (timeout == 0) {
+ DRM_ERROR("disable pipeline%d timeout.\n", kcrtc->master->id);
+ if (crtc->state->active) {
+ unsigned long flags;
+
+ spin_lock_irqsave(&crtc->dev->event_lock, flags);
+ kcrtc->disable_done = NULL;
+ spin_unlock_irqrestore(&crtc->dev->event_lock, flags);
+ }
+ }
+
+ drm_crtc_vblank_off(crtc);
+ komeda_crtc_unprepare(kcrtc);
+}
+
+static void
+komeda_crtc_atomic_flush(struct drm_crtc *crtc,
+ struct drm_crtc_state *old)
+{
+ /* commit with modeset will be handled in enable/disable */
+ if (drm_atomic_crtc_needs_modeset(crtc->state))
+ return;
+
+ komeda_crtc_do_flush(crtc, old);
+}
+
+static enum drm_mode_status
+komeda_crtc_mode_valid(struct drm_crtc *crtc, const struct drm_display_mode *m)
+{
+ struct komeda_dev *mdev = crtc->dev->dev_private;
+ struct komeda_crtc *kcrtc = to_kcrtc(crtc);
+ struct komeda_pipeline *master = kcrtc->master;
+ long mode_clk, pxlclk;
+
+ if (m->flags & DRM_MODE_FLAG_INTERLACE)
+ return MODE_NO_INTERLACE;
+
+ /* main clock/AXI clk must be faster than pxlclk*/
+ mode_clk = m->clock * 1000;
+ pxlclk = clk_round_rate(master->pxlclk, mode_clk);
+ if (pxlclk != mode_clk) {
+ DRM_DEBUG_ATOMIC("pxlclk doesn't support %ld Hz\n", mode_clk);
+
+ return MODE_NOCLOCK;
+ }
+
+ if (clk_round_rate(mdev->mclk, mode_clk) < pxlclk) {
+ DRM_DEBUG_ATOMIC("mclk can't satisfy the requirement of %s-clk: %ld.\n",
+ m->name, pxlclk);
+
+ return MODE_CLOCK_HIGH;
+ }
+
+ if (clk_round_rate(master->aclk, mode_clk) < pxlclk) {
+ DRM_DEBUG_ATOMIC("aclk can't satisfy the requirement of %s-clk: %ld.\n",
+ m->name, pxlclk);
+
+ return MODE_CLOCK_HIGH;
+ }
+
+ return MODE_OK;
+}
+
+static bool komeda_crtc_mode_fixup(struct drm_crtc *crtc,
+ const struct drm_display_mode *m,
+ struct drm_display_mode *adjusted_mode)
+{
+ struct komeda_crtc *kcrtc = to_kcrtc(crtc);
+ struct komeda_pipeline *master = kcrtc->master;
+ long mode_clk = m->clock * 1000;
+
+ adjusted_mode->clock = clk_round_rate(master->pxlclk, mode_clk) / 1000;
+
+ return true;
+}
+
+static struct drm_crtc_helper_funcs komeda_crtc_helper_funcs = {
+ .atomic_check = komeda_crtc_atomic_check,
+ .atomic_flush = komeda_crtc_atomic_flush,
+ .atomic_enable = komeda_crtc_atomic_enable,
+ .atomic_disable = komeda_crtc_atomic_disable,
+ .mode_valid = komeda_crtc_mode_valid,
+ .mode_fixup = komeda_crtc_mode_fixup,
};
+static void komeda_crtc_reset(struct drm_crtc *crtc)
+{
+ struct komeda_crtc_state *state;
+
+ if (crtc->state)
+ __drm_atomic_helper_crtc_destroy_state(crtc->state);
+
+ kfree(to_kcrtc_st(crtc->state));
+ crtc->state = NULL;
+
+ state = kzalloc(sizeof(*state), GFP_KERNEL);
+ if (state) {
+ crtc->state = &state->base;
+ crtc->state->crtc = crtc;
+ }
+}
+
+static struct drm_crtc_state *
+komeda_crtc_atomic_duplicate_state(struct drm_crtc *crtc)
+{
+ struct komeda_crtc_state *old = to_kcrtc_st(crtc->state);
+ struct komeda_crtc_state *new;
+
+ new = kzalloc(sizeof(*new), GFP_KERNEL);
+ if (!new)
+ return NULL;
+
+ __drm_atomic_helper_crtc_duplicate_state(crtc, &new->base);
+
+ new->affected_pipes = old->active_pipes;
+
+ return &new->base;
+}
+
+static void komeda_crtc_atomic_destroy_state(struct drm_crtc *crtc,
+ struct drm_crtc_state *state)
+{
+ __drm_atomic_helper_crtc_destroy_state(state);
+ kfree(to_kcrtc_st(state));
+}
+
+static int komeda_crtc_vblank_enable(struct drm_crtc *crtc)
+{
+ struct komeda_dev *mdev = crtc->dev->dev_private;
+ struct komeda_crtc *kcrtc = to_kcrtc(crtc);
+
+ mdev->funcs->on_off_vblank(mdev, kcrtc->master->id, true);
+ return 0;
+}
+
+static void komeda_crtc_vblank_disable(struct drm_crtc *crtc)
+{
+ struct komeda_dev *mdev = crtc->dev->dev_private;
+ struct komeda_crtc *kcrtc = to_kcrtc(crtc);
+
+ mdev->funcs->on_off_vblank(mdev, kcrtc->master->id, false);
+}
+
static const struct drm_crtc_funcs komeda_crtc_funcs = {
+ .gamma_set = drm_atomic_helper_legacy_gamma_set,
+ .destroy = drm_crtc_cleanup,
+ .set_config = drm_atomic_helper_set_config,
+ .page_flip = drm_atomic_helper_page_flip,
+ .reset = komeda_crtc_reset,
+ .atomic_duplicate_state = komeda_crtc_atomic_duplicate_state,
+ .atomic_destroy_state = komeda_crtc_atomic_destroy_state,
+ .enable_vblank = komeda_crtc_vblank_enable,
+ .disable_vblank = komeda_crtc_vblank_disable,
};
int komeda_kms_setup_crtcs(struct komeda_kms_dev *kms,
#include <linux/of_device.h>
#include <linux/of_graph.h>
#include <linux/platform_device.h>
+#ifdef CONFIG_DEBUG_FS
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+#endif
#include <drm/drm_print.h>
#include "komeda_dev.h"
+static int komeda_register_show(struct seq_file *sf, void *x)
+{
+ struct komeda_dev *mdev = sf->private;
+ int i;
+
+ if (mdev->funcs->dump_register)
+ mdev->funcs->dump_register(mdev, sf);
+
+ for (i = 0; i < mdev->n_pipelines; i++)
+ komeda_pipeline_dump_register(mdev->pipelines[i], sf);
+
+ return 0;
+}
+
+static int komeda_register_open(struct inode *inode, struct file *filp)
+{
+ return single_open(filp, komeda_register_show, inode->i_private);
+}
+
+static const struct file_operations komeda_register_fops = {
+ .owner = THIS_MODULE,
+ .open = komeda_register_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+#ifdef CONFIG_DEBUG_FS
+static void komeda_debugfs_init(struct komeda_dev *mdev)
+{
+ if (!debugfs_initialized())
+ return;
+
+ mdev->debugfs_root = debugfs_create_dir("komeda", NULL);
+ if (IS_ERR_OR_NULL(mdev->debugfs_root))
+ return;
+
+ debugfs_create_file("register", 0444, mdev->debugfs_root,
+ mdev, &komeda_register_fops);
+}
+#endif
+
+static ssize_t
+core_id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct komeda_dev *mdev = dev_to_mdev(dev);
+
+ return snprintf(buf, PAGE_SIZE, "0x%08x\n", mdev->chip.core_id);
+}
+static DEVICE_ATTR_RO(core_id);
+
+static ssize_t
+config_id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct komeda_dev *mdev = dev_to_mdev(dev);
+ struct komeda_pipeline *pipe = mdev->pipelines[0];
+ union komeda_config_id config_id;
+ int i;
+
+ memset(&config_id, 0, sizeof(config_id));
+
+ config_id.max_line_sz = pipe->layers[0]->hsize_in.end;
+ config_id.n_pipelines = mdev->n_pipelines;
+ config_id.n_scalers = pipe->n_scalers;
+ config_id.n_layers = pipe->n_layers;
+ config_id.n_richs = 0;
+ for (i = 0; i < pipe->n_layers; i++) {
+ if (pipe->layers[i]->layer_type == KOMEDA_FMT_RICH_LAYER)
+ config_id.n_richs++;
+ }
+ return snprintf(buf, PAGE_SIZE, "0x%08x\n", config_id.value);
+}
+static DEVICE_ATTR_RO(config_id);
+
+static struct attribute *komeda_sysfs_entries[] = {
+ &dev_attr_core_id.attr,
+ &dev_attr_config_id.attr,
+ NULL,
+};
+
+static struct attribute_group komeda_sysfs_attr_group = {
+ .attrs = komeda_sysfs_entries,
+};
+
static int komeda_parse_pipe_dt(struct komeda_dev *mdev, struct device_node *np)
{
struct komeda_pipeline *pipe;
static int komeda_parse_dt(struct device *dev, struct komeda_dev *mdev)
{
+ struct platform_device *pdev = to_platform_device(dev);
struct device_node *child, *np = dev->of_node;
struct clk *clk;
int ret;
return PTR_ERR(clk);
mdev->mclk = clk;
+ mdev->irq = platform_get_irq(pdev, 0);
+ if (mdev->irq < 0) {
+ DRM_ERROR("could not get IRQ number.\n");
+ return mdev->irq;
+ }
for_each_available_child_of_node(np, child) {
if (of_node_cmp(child->name, "pipeline") == 0) {
if (!mdev)
return ERR_PTR(-ENOMEM);
+ mutex_init(&mdev->lock);
+
mdev->dev = dev;
mdev->reg_base = devm_ioremap_resource(dev, io_res);
if (IS_ERR(mdev->reg_base)) {
goto err_cleanup;
}
+ err = komeda_assemble_pipelines(mdev);
+ if (err) {
+ DRM_ERROR("assemble display pipelines failed.\n");
+ goto err_cleanup;
+ }
+
+ err = sysfs_create_group(&dev->kobj, &komeda_sysfs_attr_group);
+ if (err) {
+ DRM_ERROR("create sysfs group failed.\n");
+ goto err_cleanup;
+ }
+
+#ifdef CONFIG_DEBUG_FS
+ komeda_debugfs_init(mdev);
+#endif
+
return mdev;
err_cleanup:
struct komeda_dev_funcs *funcs = mdev->funcs;
int i;
+ sysfs_remove_group(&dev->kobj, &komeda_sysfs_attr_group);
+
+#ifdef CONFIG_DEBUG_FS
+ debugfs_remove_recursive(mdev->debugfs_root);
+#endif
+
for (i = 0; i < mdev->n_pipelines; i++) {
komeda_pipeline_destroy(mdev, mdev->pipelines[i]);
mdev->pipelines[i] = NULL;
#include "malidp_product.h"
#include "komeda_format_caps.h"
+#define KOMEDA_EVENT_VSYNC BIT_ULL(0)
+#define KOMEDA_EVENT_FLIP BIT_ULL(1)
+#define KOMEDA_EVENT_URUN BIT_ULL(2)
+#define KOMEDA_EVENT_IBSY BIT_ULL(3)
+#define KOMEDA_EVENT_OVR BIT_ULL(4)
+#define KOMEDA_EVENT_EOW BIT_ULL(5)
+#define KOMEDA_EVENT_MODE BIT_ULL(6)
+
+#define KOMEDA_ERR_TETO BIT_ULL(14)
+#define KOMEDA_ERR_TEMR BIT_ULL(15)
+#define KOMEDA_ERR_TITR BIT_ULL(16)
+#define KOMEDA_ERR_CPE BIT_ULL(17)
+#define KOMEDA_ERR_CFGE BIT_ULL(18)
+#define KOMEDA_ERR_AXIE BIT_ULL(19)
+#define KOMEDA_ERR_ACE0 BIT_ULL(20)
+#define KOMEDA_ERR_ACE1 BIT_ULL(21)
+#define KOMEDA_ERR_ACE2 BIT_ULL(22)
+#define KOMEDA_ERR_ACE3 BIT_ULL(23)
+#define KOMEDA_ERR_DRIFTTO BIT_ULL(24)
+#define KOMEDA_ERR_FRAMETO BIT_ULL(25)
+#define KOMEDA_ERR_CSCE BIT_ULL(26)
+#define KOMEDA_ERR_ZME BIT_ULL(27)
+#define KOMEDA_ERR_MERR BIT_ULL(28)
+#define KOMEDA_ERR_TCF BIT_ULL(29)
+#define KOMEDA_ERR_TTNG BIT_ULL(30)
+#define KOMEDA_ERR_TTF BIT_ULL(31)
+
/* malidp device id */
enum {
MALI_D71 = 0,
struct komeda_dev;
+struct komeda_events {
+ u64 global;
+ u64 pipes[KOMEDA_MAX_PIPELINES];
+};
+
/**
* struct komeda_dev_funcs
*
int (*enum_resources)(struct komeda_dev *mdev);
/** @cleanup: call to chip to cleanup komeda_dev->chip data */
void (*cleanup)(struct komeda_dev *mdev);
+ /**
+ * @irq_handler:
+ *
+ * for CORE to get the HW event from the CHIP when interrupt happened.
+ */
+ irqreturn_t (*irq_handler)(struct komeda_dev *mdev,
+ struct komeda_events *events);
+ /** @enable_irq: enable irq */
+ int (*enable_irq)(struct komeda_dev *mdev);
+ /** @disable_irq: disable irq */
+ int (*disable_irq)(struct komeda_dev *mdev);
+ /** @on_off_vblank: notify HW to on/off vblank */
+ void (*on_off_vblank)(struct komeda_dev *mdev,
+ int master_pipe, bool on);
+
+ /** @dump_register: Optional, dump registers to seq_file */
+ void (*dump_register)(struct komeda_dev *mdev, struct seq_file *seq);
+ /**
+ * @change_opmode:
+ *
+ * Notify HW to switch to a new display operation mode.
+ */
+ int (*change_opmode)(struct komeda_dev *mdev, int new_mode);
+ /** @flush: Notify the HW to flush or kickoff the update */
+ void (*flush)(struct komeda_dev *mdev,
+ int master_pipe, u32 active_pipes);
+};
+
+/*
+ * DISPLAY_MODE describes how many display been enabled, and which will be
+ * passed to CHIP by &komeda_dev_funcs->change_opmode(), then CHIP can do the
+ * pipeline resources assignment according to this usage hint.
+ * - KOMEDA_MODE_DISP0: Only one display enabled, pipeline-0 work as master.
+ * - KOMEDA_MODE_DISP1: Only one display enabled, pipeline-0 work as master.
+ * - KOMEDA_MODE_DUAL_DISP: Dual display mode, both display has been enabled.
+ * And D71 supports assign two pipelines to one single display on mode
+ * KOMEDA_MODE_DISP0/DISP1
+ */
+enum {
+ KOMEDA_MODE_INACTIVE = 0,
+ KOMEDA_MODE_DISP0 = BIT(0),
+ KOMEDA_MODE_DISP1 = BIT(1),
+ KOMEDA_MODE_DUAL_DISP = KOMEDA_MODE_DISP0 | KOMEDA_MODE_DISP1,
};
/**
* control-abilites of device.
*/
struct komeda_dev {
+ /** @dev: the base device structure */
struct device *dev;
+ /** @reg_base: the base address of komeda io space */
u32 __iomem *reg_base;
+ /** @chip: the basic chip information */
struct komeda_chip_info chip;
/** @fmt_tbl: initialized by &komeda_dev_funcs->init_format_table */
struct komeda_format_caps_table fmt_tbl;
/** @pclk: APB clock for register access */
struct clk *pclk;
- /** @mck: HW main engine clk */
+ /** @mclk: HW main engine clk */
struct clk *mclk;
+ /** @irq: irq number */
+ int irq;
+
+ /** @lock: used to protect dpmode */
+ struct mutex lock;
+ /** @dpmode: current display mode */
+ u32 dpmode;
+
+ /** @n_pipelines: the number of pipe in @pipelines */
int n_pipelines;
+ /** @pipelines: the komeda pipelines */
struct komeda_pipeline *pipelines[KOMEDA_MAX_PIPELINES];
/** @funcs: chip funcs to access to HW */
* destroyed by &komeda_dev_funcs.cleanup()
*/
void *chip_data;
+
+ /** @debugfs_root: root directory of komeda debugfs */
+ struct dentry *debugfs_root;
};
static inline bool
struct komeda_dev *komeda_dev_create(struct device *dev);
void komeda_dev_destroy(struct komeda_dev *mdev);
+struct komeda_dev *dev_to_mdev(struct device *dev);
+
#endif /*_KOMEDA_DEV_H_*/
struct komeda_kms_dev *kms;
};
+struct komeda_dev *dev_to_mdev(struct device *dev)
+{
+ struct komeda_drv *mdrv = dev_get_drvdata(dev);
+
+ return mdrv ? mdrv->mdev : NULL;
+}
+
static void komeda_unbind(struct device *dev)
{
struct komeda_drv *mdrv = dev_get_drvdata(dev);
},
};
-const struct of_device_id komeda_of_match[] = {
+static const struct of_device_id komeda_of_match[] = {
{ .compatible = "arm,mali-d71", .data = &komeda_products[MALI_D71], },
{},
};
#include <drm/drm_framebuffer.h>
#include "komeda_format_caps.h"
-/** struct komeda_fb - entend drm_framebuffer with komeda attribute */
+/**
+ * struct komeda_fb - Entending drm_framebuffer with komeda attribute
+ */
struct komeda_fb {
/** @base: &drm_framebuffer */
struct drm_framebuffer base;
- /* @format_caps: &komeda_format_caps */
+ /**
+ * @format_caps:
+ * extends drm_format_info for komeda specific information
+ */
const struct komeda_format_caps *format_caps;
/** @aligned_w: aligned frame buffer width */
u32 aligned_w;
#include <drm/drm_fb_helper.h>
#include <drm/drm_gem_cma_helper.h>
#include <drm/drm_gem_framebuffer_helper.h>
+#include <drm/drm_irq.h>
#include <drm/drm_vblank.h>
#include "komeda_dev.h"
struct drm_device *dev,
struct drm_mode_create_dumb *args)
{
- u32 alignment = 16; /* TODO get alignment from dev */
+ struct komeda_dev *mdev = dev->dev_private;
+ u32 pitch = DIV_ROUND_UP(args->width * args->bpp, 8);
- args->pitch = ALIGN(DIV_ROUND_UP(args->width * args->bpp, 8),
- alignment);
+ args->pitch = ALIGN(pitch, mdev->chip.bus_width);
return drm_gem_cma_dumb_create_internal(file, dev, args);
}
+static irqreturn_t komeda_kms_irq_handler(int irq, void *data)
+{
+ struct drm_device *drm = data;
+ struct komeda_dev *mdev = drm->dev_private;
+ struct komeda_kms_dev *kms = to_kdev(drm);
+ struct komeda_events evts;
+ irqreturn_t status;
+ u32 i;
+
+ /* Call into the CHIP to recognize events */
+ memset(&evts, 0, sizeof(evts));
+ status = mdev->funcs->irq_handler(mdev, &evts);
+
+ /* Notify the crtc to handle the events */
+ for (i = 0; i < kms->n_crtcs; i++)
+ komeda_crtc_handle_event(&kms->crtcs[i], &evts);
+
+ return status;
+}
+
static struct drm_driver komeda_kms_driver = {
.driver_features = DRIVER_GEM | DRIVER_MODESET | DRIVER_ATOMIC |
- DRIVER_PRIME,
+ DRIVER_PRIME | DRIVER_HAVE_IRQ,
.lastclose = drm_fb_helper_lastclose,
+ .irq_handler = komeda_kms_irq_handler,
.gem_free_object_unlocked = drm_gem_cma_free_object,
.gem_vm_ops = &drm_gem_cma_vm_ops,
.dumb_create = komeda_gem_cma_dumb_create,
.atomic_commit_tail = komeda_kms_commit_tail,
};
+static int komeda_kms_check(struct drm_device *dev,
+ struct drm_atomic_state *state)
+{
+ struct drm_crtc *crtc;
+ struct drm_crtc_state *old_crtc_st, *new_crtc_st;
+ int i, err;
+
+ err = drm_atomic_helper_check_modeset(dev, state);
+ if (err)
+ return err;
+
+ /* komeda need to re-calculate resource assumption in every commit
+ * so need to add all affected_planes (even unchanged) to
+ * drm_atomic_state.
+ */
+ for_each_oldnew_crtc_in_state(state, crtc, old_crtc_st, new_crtc_st, i) {
+ err = drm_atomic_add_affected_planes(state, crtc);
+ if (err)
+ return err;
+ }
+
+ err = drm_atomic_helper_check_planes(dev, state);
+ if (err)
+ return err;
+
+ return 0;
+}
+
static const struct drm_mode_config_funcs komeda_mode_config_funcs = {
.fb_create = komeda_fb_create,
- .atomic_check = drm_atomic_helper_check,
+ .atomic_check = komeda_kms_check,
.atomic_commit = drm_atomic_helper_commit,
};
drm_mode_config_reset(drm);
- err = drm_dev_register(drm, 0);
+ err = drm_irq_install(drm, mdev->irq);
if (err)
goto cleanup_mode_config;
+ err = mdev->funcs->enable_irq(mdev);
+ if (err)
+ goto uninstall_irq;
+
+ err = drm_dev_register(drm, 0);
+ if (err)
+ goto uninstall_irq;
+
return kms;
+uninstall_irq:
+ drm_irq_uninstall(drm);
cleanup_mode_config:
drm_mode_config_cleanup(drm);
+ komeda_kms_cleanup_private_objs(kms);
free_kms:
kfree(kms);
return ERR_PTR(err);
struct drm_device *drm = &kms->base;
struct komeda_dev *mdev = drm->dev_private;
+ mdev->funcs->disable_irq(mdev);
drm_dev_unregister(drm);
+ drm_irq_uninstall(drm);
component_unbind_all(mdev->dev, drm);
- komeda_kms_cleanup_private_objs(mdev);
+ komeda_kms_cleanup_private_objs(kms);
drm_mode_config_cleanup(drm);
drm->dev_private = NULL;
drm_dev_put(drm);
#include <drm/drm_crtc_helper.h>
#include <drm/drm_device.h>
#include <drm/drm_writeback.h>
+#include <video/videomode.h>
+#include <video/display_timing.h>
-/** struct komeda_plane - komeda instance of drm_plane */
+/**
+ * struct komeda_plane - komeda instance of drm_plane
+ */
struct komeda_plane {
/** @base: &drm_plane */
struct drm_plane base;
* merge into the master.
*/
struct komeda_pipeline *slave;
+
+ /** @disable_done: this flip_done is for tracing the disable */
+ struct completion *disable_done;
};
-/** struct komeda_crtc_state */
+/**
+ * struct komeda_crtc_state
+ */
struct komeda_crtc_state {
/** @base: &drm_crtc_state */
struct drm_crtc_state base;
/* private properties */
/* computed state which are used by validate/check */
+ /**
+ * @affected_pipes:
+ * the affected pipelines in once display instance
+ */
u32 affected_pipes;
+ /**
+ * @active_pipes:
+ * the active pipelines in once display instance
+ */
u32 active_pipes;
};
int komeda_kms_add_planes(struct komeda_kms_dev *kms, struct komeda_dev *mdev);
int komeda_kms_add_private_objs(struct komeda_kms_dev *kms,
struct komeda_dev *mdev);
-void komeda_kms_cleanup_private_objs(struct komeda_dev *mdev);
+void komeda_kms_cleanup_private_objs(struct komeda_kms_dev *kms);
+
+void komeda_crtc_handle_event(struct komeda_crtc *kcrtc,
+ struct komeda_events *evts);
struct komeda_kms_dev *komeda_kms_attach(struct komeda_dev *mdev);
void komeda_kms_detach(struct komeda_kms_dev *kms);
if (mdev->n_pipelines + 1 > KOMEDA_MAX_PIPELINES) {
DRM_ERROR("Exceed max support %d pipelines.\n",
KOMEDA_MAX_PIPELINES);
- return NULL;
+ return ERR_PTR(-ENOSPC);
}
if (size < sizeof(*pipe)) {
DRM_ERROR("Request pipeline size too small.\n");
- return NULL;
+ return ERR_PTR(-EINVAL);
}
pipe = devm_kzalloc(mdev->dev, size, GFP_KERNEL);
if (!pipe)
- return NULL;
+ return ERR_PTR(-ENOMEM);
pipe->mdev = mdev;
pipe->id = mdev->n_pipelines;
devm_kfree(mdev->dev, pipe);
}
-struct komeda_component **
+static struct komeda_component **
komeda_pipeline_get_component_pos(struct komeda_pipeline *pipe, int id)
{
struct komeda_dev *mdev = pipe->mdev;
if (max_active_inputs > KOMEDA_COMPONENT_N_INPUTS) {
WARN(1, "please large KOMEDA_COMPONENT_N_INPUTS to %d.\n",
max_active_inputs);
- return NULL;
+ return ERR_PTR(-ENOSPC);
}
pos = komeda_pipeline_get_component_pos(pipe, id);
if (!pos || (*pos))
- return NULL;
+ return ERR_PTR(-EINVAL);
if (has_bit(id, KOMEDA_PIPELINE_LAYERS)) {
idx = id - KOMEDA_COMPONENT_LAYER0;
num = &pipe->n_layers;
if (idx != pipe->n_layers) {
DRM_ERROR("please add Layer by id sequence.\n");
- return NULL;
+ return ERR_PTR(-EINVAL);
}
} else if (has_bit(id, KOMEDA_PIPELINE_SCALERS)) {
idx = id - KOMEDA_COMPONENT_SCALER0;
num = &pipe->n_scalers;
if (idx != pipe->n_scalers) {
DRM_ERROR("please add Scaler by id sequence.\n");
- return NULL;
+ return ERR_PTR(-EINVAL);
}
}
c = devm_kzalloc(pipe->mdev->dev, comp_sz, GFP_KERNEL);
if (!c)
- return NULL;
+ return ERR_PTR(-ENOMEM);
c->id = id;
c->hw_id = hw_id;
{
devm_kfree(mdev->dev, c);
}
+
+static void komeda_component_dump(struct komeda_component *c)
+{
+ if (!c)
+ return;
+
+ DRM_DEBUG(" %s: ID %d-0x%08lx.\n",
+ c->name, c->id, BIT(c->id));
+ DRM_DEBUG(" max_active_inputs:%d, supported_inputs: 0x%08x.\n",
+ c->max_active_inputs, c->supported_inputs);
+ DRM_DEBUG(" max_active_outputs:%d, supported_outputs: 0x%08x.\n",
+ c->max_active_outputs, c->supported_outputs);
+}
+
+static void komeda_pipeline_dump(struct komeda_pipeline *pipe)
+{
+ struct komeda_component *c;
+ int id;
+
+ DRM_INFO("Pipeline-%d: n_layers: %d, n_scalers: %d, output: %s\n",
+ pipe->id, pipe->n_layers, pipe->n_scalers,
+ pipe->of_output_dev ? pipe->of_output_dev->full_name : "none");
+
+ dp_for_each_set_bit(id, pipe->avail_comps) {
+ c = komeda_pipeline_get_component(pipe, id);
+
+ komeda_component_dump(c);
+ }
+}
+
+static void komeda_component_verify_inputs(struct komeda_component *c)
+{
+ struct komeda_pipeline *pipe = c->pipeline;
+ struct komeda_component *input;
+ int id;
+
+ dp_for_each_set_bit(id, c->supported_inputs) {
+ input = komeda_pipeline_get_component(pipe, id);
+ if (!input) {
+ c->supported_inputs &= ~(BIT(id));
+ DRM_WARN("Can not find input(ID-%d) for component: %s.\n",
+ id, c->name);
+ continue;
+ }
+
+ input->supported_outputs |= BIT(c->id);
+ }
+}
+
+static void komeda_pipeline_assemble(struct komeda_pipeline *pipe)
+{
+ struct komeda_component *c;
+ int id;
+
+ dp_for_each_set_bit(id, pipe->avail_comps) {
+ c = komeda_pipeline_get_component(pipe, id);
+
+ komeda_component_verify_inputs(c);
+ }
+}
+
+int komeda_assemble_pipelines(struct komeda_dev *mdev)
+{
+ struct komeda_pipeline *pipe;
+ int i;
+
+ for (i = 0; i < mdev->n_pipelines; i++) {
+ pipe = mdev->pipelines[i];
+
+ komeda_pipeline_assemble(pipe);
+ komeda_pipeline_dump(pipe);
+ }
+
+ return 0;
+}
+
+void komeda_pipeline_dump_register(struct komeda_pipeline *pipe,
+ struct seq_file *sf)
+{
+ struct komeda_component *c;
+ u32 id;
+
+ seq_printf(sf, "\n======== Pipeline-%d ==========\n", pipe->id);
+
+ if (pipe->funcs && pipe->funcs->dump_register)
+ pipe->funcs->dump_register(pipe, sf);
+
+ dp_for_each_set_bit(id, pipe->avail_comps) {
+ c = komeda_pipeline_get_component(pipe, id);
+
+ seq_printf(sf, "\n------%s------\n", c->name);
+ if (c->funcs->dump_register)
+ c->funcs->dump_register(c, sf);
+ }
+}
u32 __iomem *reg;
/** @id: component id */
u32 id;
- /** @hw_ic: component hw id,
- * which is initialized by chip and used by chip only
+ /**
+ * @hw_id: component hw id,
+ * which is initialized by chip and used by chip only
*/
u32 hw_id;
/**
* @max_active_inputs:
- * @max_active_outpus:
+ * @max_active_outputs:
*
- * maximum number of inputs/outputs that can be active in the same time
+ * maximum number of inputs/outputs that can be active at the same time
* Note:
* the number isn't the bit number of @supported_inputs or
* @supported_outputs, but may be less than it, since component may not
* support enabling all @supported_inputs/outputs at the same time.
*/
u8 max_active_inputs;
+ /** @max_active_outputs: maximum number of outputs */
u8 max_active_outputs;
/**
* @supported_inputs:
* @supported_outputs:
*
- * bitmask of BIT(component->id) for the supported inputs/outputs
+ * bitmask of BIT(component->id) for the supported inputs/outputs,
* describes the possibilities of how a component is linked into a
* pipeline.
*/
u32 supported_inputs;
+ /** @supported_outputs: bitmask of supported output componenet ids */
u32 supported_outputs;
/**
struct komeda_component_output {
/** @component: indicate which component the data comes from */
struct komeda_component *component;
- /** @output_port:
+ /**
+ * @output_port:
* the output port of the &komeda_component_output.component
*/
u8 output_port;
struct komeda_component_state {
/** @obj: tracking component_state by drm_atomic_state */
struct drm_private_state obj;
+ /** @component: backpointer to the component */
struct komeda_component *component;
/**
* @binding_user:
- * currently bound user, the user can be crtc/plane/wb_conn, which is
- * valid decided by @component and @inputs
+ * currently bound user, the user can be @crtc, @plane or @wb_conn,
+ * which is valid decided by @component and @inputs
*
* - Layer: its user always is plane.
* - compiz/improc/timing_ctrlr: the user is crtc.
* - scaler: plane when input is layer, wb_conn if input is compiz.
*/
union {
+ /** @crtc: backpointer for user crtc */
struct drm_crtc *crtc;
+ /** @plane: backpointer for user plane */
struct drm_plane *plane;
+ /** @wb_conn: backpointer for user wb_connector */
struct drm_connector *wb_conn;
void *binding_user;
};
+
/**
* @active_inputs:
*
* active_inputs is bitmask of @inputs index
*
- * - active_inputs = changed_active_inputs + unchanged_active_inputs
- * - affected_inputs = old->active_inputs + new->active_inputs;
+ * - active_inputs = changed_active_inputs | unchanged_active_inputs
+ * - affected_inputs = old->active_inputs | new->active_inputs;
* - disabling_inputs = affected_inputs ^ active_inputs;
- * - changed_inputs = disabling_inputs + changed_active_inputs;
+ * - changed_inputs = disabling_inputs | changed_active_inputs;
*
* NOTE:
* changed_inputs doesn't include all active_input but only
* level for dirty update.
*/
u16 active_inputs;
+ /** @changed_active_inputs: bitmask of the changed @active_inputs */
u16 changed_active_inputs;
+ /** @affected_inputs: bitmask for affected @inputs */
u16 affected_inputs;
/**
* @inputs:
return component_disabling_inputs(st) | st->changed_active_inputs;
}
+#define for_each_changed_input(st, i) \
+ for ((i) = 0; (i) < (st)->component->max_active_inputs; (i)++) \
+ if (has_bit((i), component_changed_inputs(st)))
+
#define to_comp(__c) (((__c) == NULL) ? NULL : &((__c)->base))
#define to_cpos(__c) ((struct komeda_component **)&(__c))
-/* these structures are going to be filled in in uture patches */
struct komeda_layer {
struct komeda_component base;
- /* layer specific features and caps */
- int layer_type; /* RICH, SIMPLE or WB */
+ /* accepted h/v input range before rotation */
+ struct malidp_range hsize_in, vsize_in;
+ u32 layer_type; /* RICH, SIMPLE or WB */
+ u32 supported_rots;
};
struct komeda_layer_state {
struct komeda_component_state base;
/* layer specific configuration state */
+ u16 hsize, vsize;
+ u32 rot;
+ dma_addr_t addr[3];
};
-struct komeda_compiz {
+struct komeda_scaler {
struct komeda_component base;
- /* compiz specific features and caps */
+ /* scaler features and caps */
};
-struct komeda_compiz_state {
+struct komeda_scaler_state {
struct komeda_component_state base;
- /* compiz specific configuration state */
};
-struct komeda_scaler {
+struct komeda_compiz {
struct komeda_component base;
- /* scaler features and caps */
+ struct malidp_range hsize, vsize;
};
-struct komeda_scaler_state {
+struct komeda_compiz_input_cfg {
+ u16 hsize, vsize;
+ u16 hoffset, voffset;
+ u8 pixel_blend_mode, layer_alpha;
+};
+
+struct komeda_compiz_state {
struct komeda_component_state base;
+ /* composition size */
+ u16 hsize, vsize;
+ struct komeda_compiz_input_cfg cins[KOMEDA_COMPONENT_N_INPUTS];
};
struct komeda_improc {
struct komeda_component base;
+ u32 supported_color_formats; /* DRM_RGB/YUV444/YUV420*/
+ u32 supported_color_depths; /* BIT(8) | BIT(10)*/
+ u8 supports_degamma : 1;
+ u8 supports_csc : 1;
+ u8 supports_gamma : 1;
};
struct komeda_improc_state {
struct komeda_component_state base;
+ u16 hsize, vsize;
};
/* display timing controller */
struct komeda_timing_ctrlr {
struct komeda_component base;
+ u8 supports_dual_link : 1;
};
struct komeda_timing_ctrlr_state {
struct komeda_component_state base;
};
+/* Why define A separated structure but not use plane_state directly ?
+ * 1. Komeda supports layer_split which means a plane_state can be split and
+ * handled by two layers, one layer only handle half of plane image.
+ * 2. Fix up the user properties according to HW's capabilities, like user
+ * set rotation to R180, but HW only supports REFLECT_X+Y. the rot here is
+ * after drm_rotation_simplify()
+ */
+struct komeda_data_flow_cfg {
+ struct komeda_component_output input;
+ u16 in_x, in_y, in_w, in_h;
+ u32 out_x, out_y, out_w, out_h;
+ u32 rot;
+ int blending_zorder;
+ u8 pixel_blend_mode, layer_alpha;
+};
+
/** struct komeda_pipeline_funcs */
struct komeda_pipeline_funcs {
/* dump_register: Optional, dump registers to seq_file */
int id;
/** @avail_comps: available components mask of pipeline */
u32 avail_comps;
+ /** @n_layers: the number of layer on @layers */
int n_layers;
+ /** @layers: the pipeline layers */
struct komeda_layer *layers[KOMEDA_PIPELINE_MAX_LAYERS];
+ /** @n_scalers: the number of scaler on @scalers */
int n_scalers;
+ /** @scalers: the pipeline scalers */
struct komeda_scaler *scalers[KOMEDA_PIPELINE_MAX_SCALERS];
+ /** @compiz: compositor */
struct komeda_compiz *compiz;
+ /** @wb_layer: writeback layer */
struct komeda_layer *wb_layer;
+ /** @improc: post image processor */
struct komeda_improc *improc;
+ /** @ctrlr: timing controller */
struct komeda_timing_ctrlr *ctrlr;
+ /** @funcs: chip pipeline functions */
struct komeda_pipeline_funcs *funcs; /* private pipeline functions */
/** @of_node: pipeline dt node */
struct komeda_pipeline_state {
/** @obj: tracking pipeline_state by drm_atomic_state */
struct drm_private_state obj;
+ /** @pipe: backpointer to the pipeline */
struct komeda_pipeline *pipe;
/** @crtc: currently bound crtc */
struct drm_crtc *crtc;
struct komeda_pipeline_funcs *funcs);
void komeda_pipeline_destroy(struct komeda_dev *mdev,
struct komeda_pipeline *pipe);
-
+int komeda_assemble_pipelines(struct komeda_dev *mdev);
struct komeda_component *
komeda_pipeline_get_component(struct komeda_pipeline *pipe, int id);
+void komeda_pipeline_dump_register(struct komeda_pipeline *pipe,
+ struct seq_file *sf);
+
/* component APIs */
struct komeda_component *
komeda_component_add(struct komeda_pipeline *pipe,
void komeda_component_destroy(struct komeda_dev *mdev,
struct komeda_component *c);
+struct komeda_plane_state;
+struct komeda_crtc_state;
+struct komeda_crtc;
+
+int komeda_build_layer_data_flow(struct komeda_layer *layer,
+ struct komeda_plane_state *kplane_st,
+ struct komeda_crtc_state *kcrtc_st,
+ struct komeda_data_flow_cfg *dflow);
+int komeda_build_display_data_flow(struct komeda_crtc *kcrtc,
+ struct komeda_crtc_state *kcrtc_st);
+
+int komeda_release_unclaimed_resources(struct komeda_pipeline *pipe,
+ struct komeda_crtc_state *kcrtc_st);
+
+struct komeda_pipeline_state *
+komeda_pipeline_get_old_state(struct komeda_pipeline *pipe,
+ struct drm_atomic_state *state);
+void komeda_pipeline_disable(struct komeda_pipeline *pipe,
+ struct drm_atomic_state *old_state);
+void komeda_pipeline_update(struct komeda_pipeline *pipe,
+ struct drm_atomic_state *old_state);
+
#endif /* _KOMEDA_PIPELINE_H_*/
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * (C) COPYRIGHT 2018 ARM Limited. All rights reserved.
+ * Author: James.Qian.Wang <james.qian.wang@arm.com>
+ *
+ */
+
+#include <drm/drm_print.h>
+#include <linux/clk.h>
+#include "komeda_dev.h"
+#include "komeda_kms.h"
+#include "komeda_pipeline.h"
+#include "komeda_framebuffer.h"
+
+static inline bool is_switching_user(void *old, void *new)
+{
+ if (!old || !new)
+ return false;
+
+ return old != new;
+}
+
+static struct komeda_pipeline_state *
+komeda_pipeline_get_state(struct komeda_pipeline *pipe,
+ struct drm_atomic_state *state)
+{
+ struct drm_private_state *priv_st;
+
+ priv_st = drm_atomic_get_private_obj_state(state, &pipe->obj);
+ if (IS_ERR(priv_st))
+ return ERR_CAST(priv_st);
+
+ return priv_to_pipe_st(priv_st);
+}
+
+struct komeda_pipeline_state *
+komeda_pipeline_get_old_state(struct komeda_pipeline *pipe,
+ struct drm_atomic_state *state)
+{
+ struct drm_private_state *priv_st;
+
+ priv_st = drm_atomic_get_old_private_obj_state(state, &pipe->obj);
+ if (priv_st)
+ return priv_to_pipe_st(priv_st);
+ return NULL;
+}
+
+static struct komeda_pipeline_state *
+komeda_pipeline_get_new_state(struct komeda_pipeline *pipe,
+ struct drm_atomic_state *state)
+{
+ struct drm_private_state *priv_st;
+
+ priv_st = drm_atomic_get_new_private_obj_state(state, &pipe->obj);
+ if (priv_st)
+ return priv_to_pipe_st(priv_st);
+ return NULL;
+}
+
+/* Assign pipeline for crtc */
+static struct komeda_pipeline_state *
+komeda_pipeline_get_state_and_set_crtc(struct komeda_pipeline *pipe,
+ struct drm_atomic_state *state,
+ struct drm_crtc *crtc)
+{
+ struct komeda_pipeline_state *st;
+
+ st = komeda_pipeline_get_state(pipe, state);
+ if (IS_ERR(st))
+ return st;
+
+ if (is_switching_user(crtc, st->crtc)) {
+ DRM_DEBUG_ATOMIC("CRTC%d required pipeline%d is busy.\n",
+ drm_crtc_index(crtc), pipe->id);
+ return ERR_PTR(-EBUSY);
+ }
+
+ /* pipeline only can be disabled when the it is free or unused */
+ if (!crtc && st->active_comps) {
+ DRM_DEBUG_ATOMIC("Disabling a busy pipeline:%d.\n", pipe->id);
+ return ERR_PTR(-EBUSY);
+ }
+
+ st->crtc = crtc;
+
+ if (crtc) {
+ struct komeda_crtc_state *kcrtc_st;
+
+ kcrtc_st = to_kcrtc_st(drm_atomic_get_new_crtc_state(state,
+ crtc));
+
+ kcrtc_st->active_pipes |= BIT(pipe->id);
+ kcrtc_st->affected_pipes |= BIT(pipe->id);
+ }
+ return st;
+}
+
+static struct komeda_component_state *
+komeda_component_get_state(struct komeda_component *c,
+ struct drm_atomic_state *state)
+{
+ struct drm_private_state *priv_st;
+
+ WARN_ON(!drm_modeset_is_locked(&c->pipeline->obj.lock));
+
+ priv_st = drm_atomic_get_private_obj_state(state, &c->obj);
+ if (IS_ERR(priv_st))
+ return ERR_CAST(priv_st);
+
+ return priv_to_comp_st(priv_st);
+}
+
+static struct komeda_component_state *
+komeda_component_get_old_state(struct komeda_component *c,
+ struct drm_atomic_state *state)
+{
+ struct drm_private_state *priv_st;
+
+ priv_st = drm_atomic_get_old_private_obj_state(state, &c->obj);
+ if (priv_st)
+ return priv_to_comp_st(priv_st);
+ return NULL;
+}
+
+/**
+ * komeda_component_get_state_and_set_user()
+ *
+ * @c: component to get state and set user
+ * @state: global atomic state
+ * @user: direct user, the binding user
+ * @crtc: the CRTC user, the big boss :)
+ *
+ * This function accepts two users:
+ * - The direct user: can be plane/crtc/wb_connector depends on component
+ * - The big boss (CRTC)
+ * CRTC is the big boss (the final user), because all component resources
+ * eventually will be assigned to CRTC, like the layer will be binding to
+ * kms_plane, but kms plane will be binding to a CRTC eventually.
+ *
+ * The big boss (CRTC) is for pipeline assignment, since &komeda_component isn't
+ * independent and can be assigned to CRTC freely, but belongs to a specific
+ * pipeline, only pipeline can be shared between crtc, and pipeline as a whole
+ * (include all the internal components) assigned to a specific CRTC.
+ *
+ * So when set a user to komeda_component, need first to check the status of
+ * component->pipeline to see if the pipeline is available on this specific
+ * CRTC. if the pipeline is busy (assigned to another CRTC), even the required
+ * component is free, the component still cannot be assigned to the direct user.
+ */
+static struct komeda_component_state *
+komeda_component_get_state_and_set_user(struct komeda_component *c,
+ struct drm_atomic_state *state,
+ void *user,
+ struct drm_crtc *crtc)
+{
+ struct komeda_pipeline_state *pipe_st;
+ struct komeda_component_state *st;
+
+ /* First check if the pipeline is available */
+ pipe_st = komeda_pipeline_get_state_and_set_crtc(c->pipeline,
+ state, crtc);
+ if (IS_ERR(pipe_st))
+ return ERR_CAST(pipe_st);
+
+ st = komeda_component_get_state(c, state);
+ if (IS_ERR(st))
+ return st;
+
+ /* check if the component has been occupied */
+ if (is_switching_user(user, st->binding_user)) {
+ DRM_DEBUG_ATOMIC("required %s is busy.\n", c->name);
+ return ERR_PTR(-EBUSY);
+ }
+
+ st->binding_user = user;
+ /* mark the component as active if user is valid */
+ if (st->binding_user)
+ pipe_st->active_comps |= BIT(c->id);
+
+ return st;
+}
+
+static void
+komeda_component_add_input(struct komeda_component_state *state,
+ struct komeda_component_output *input,
+ int idx)
+{
+ struct komeda_component *c = state->component;
+
+ WARN_ON((idx < 0 || idx >= c->max_active_inputs));
+
+ /* since the inputs[i] is only valid when it is active. So if a input[i]
+ * is a newly enabled input which switches from disable to enable, then
+ * the old inputs[i] is undefined (NOT zeroed), we can not rely on
+ * memcmp, but directly mark it changed
+ */
+ if (!has_bit(idx, state->affected_inputs) ||
+ memcmp(&state->inputs[idx], input, sizeof(*input))) {
+ memcpy(&state->inputs[idx], input, sizeof(*input));
+ state->changed_active_inputs |= BIT(idx);
+ }
+ state->active_inputs |= BIT(idx);
+ state->affected_inputs |= BIT(idx);
+}
+
+static int
+komeda_component_check_input(struct komeda_component_state *state,
+ struct komeda_component_output *input,
+ int idx)
+{
+ struct komeda_component *c = state->component;
+
+ if ((idx < 0) || (idx >= c->max_active_inputs)) {
+ DRM_DEBUG_ATOMIC("%s invalid input id: %d.\n", c->name, idx);
+ return -EINVAL;
+ }
+
+ if (has_bit(idx, state->active_inputs)) {
+ DRM_DEBUG_ATOMIC("%s required input_id: %d has been occupied already.\n",
+ c->name, idx);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void
+komeda_component_set_output(struct komeda_component_output *output,
+ struct komeda_component *comp,
+ u8 output_port)
+{
+ output->component = comp;
+ output->output_port = output_port;
+}
+
+static int
+komeda_component_validate_private(struct komeda_component *c,
+ struct komeda_component_state *st)
+{
+ int err;
+
+ if (!c->funcs->validate)
+ return 0;
+
+ err = c->funcs->validate(c, st);
+ if (err)
+ DRM_DEBUG_ATOMIC("%s validate private failed.\n", c->name);
+
+ return err;
+}
+
+static int
+komeda_layer_check_cfg(struct komeda_layer *layer,
+ struct komeda_plane_state *kplane_st,
+ struct komeda_data_flow_cfg *dflow)
+{
+ if (!in_range(&layer->hsize_in, dflow->in_w)) {
+ DRM_DEBUG_ATOMIC("src_w: %d is out of range.\n", dflow->in_w);
+ return -EINVAL;
+ }
+
+ if (!in_range(&layer->vsize_in, dflow->in_h)) {
+ DRM_DEBUG_ATOMIC("src_h: %d is out of range.\n", dflow->in_h);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int
+komeda_layer_validate(struct komeda_layer *layer,
+ struct komeda_plane_state *kplane_st,
+ struct komeda_data_flow_cfg *dflow)
+{
+ struct drm_plane_state *plane_st = &kplane_st->base;
+ struct drm_framebuffer *fb = plane_st->fb;
+ struct komeda_fb *kfb = to_kfb(fb);
+ struct komeda_component_state *c_st;
+ struct komeda_layer_state *st;
+ int i, err;
+
+ err = komeda_layer_check_cfg(layer, kplane_st, dflow);
+ if (err)
+ return err;
+
+ c_st = komeda_component_get_state_and_set_user(&layer->base,
+ plane_st->state, plane_st->plane, plane_st->crtc);
+ if (IS_ERR(c_st))
+ return PTR_ERR(c_st);
+
+ st = to_layer_st(c_st);
+
+ st->rot = dflow->rot;
+ st->hsize = kfb->aligned_w;
+ st->vsize = kfb->aligned_h;
+
+ for (i = 0; i < fb->format->num_planes; i++)
+ st->addr[i] = komeda_fb_get_pixel_addr(kfb, dflow->in_x,
+ dflow->in_y, i);
+
+ err = komeda_component_validate_private(&layer->base, c_st);
+ if (err)
+ return err;
+
+ /* update the data flow for the next stage */
+ komeda_component_set_output(&dflow->input, &layer->base, 0);
+
+ return 0;
+}
+
+static void pipeline_composition_size(struct komeda_crtc_state *kcrtc_st,
+ u16 *hsize, u16 *vsize)
+{
+ struct drm_display_mode *m = &kcrtc_st->base.adjusted_mode;
+
+ if (hsize)
+ *hsize = m->hdisplay;
+ if (vsize)
+ *vsize = m->vdisplay;
+}
+
+static int
+komeda_compiz_set_input(struct komeda_compiz *compiz,
+ struct komeda_crtc_state *kcrtc_st,
+ struct komeda_data_flow_cfg *dflow)
+{
+ struct drm_atomic_state *drm_st = kcrtc_st->base.state;
+ struct komeda_component_state *c_st, *old_st;
+ struct komeda_compiz_input_cfg *cin;
+ u16 compiz_w, compiz_h;
+ int idx = dflow->blending_zorder;
+
+ pipeline_composition_size(kcrtc_st, &compiz_w, &compiz_h);
+ /* check display rect */
+ if ((dflow->out_x + dflow->out_w > compiz_w) ||
+ (dflow->out_y + dflow->out_h > compiz_h) ||
+ dflow->out_w == 0 || dflow->out_h == 0) {
+ DRM_DEBUG_ATOMIC("invalid disp rect [x=%d, y=%d, w=%d, h=%d]\n",
+ dflow->out_x, dflow->out_y,
+ dflow->out_w, dflow->out_h);
+ return -EINVAL;
+ }
+
+ c_st = komeda_component_get_state_and_set_user(&compiz->base, drm_st,
+ kcrtc_st->base.crtc, kcrtc_st->base.crtc);
+ if (IS_ERR(c_st))
+ return PTR_ERR(c_st);
+
+ if (komeda_component_check_input(c_st, &dflow->input, idx))
+ return -EINVAL;
+
+ cin = &(to_compiz_st(c_st)->cins[idx]);
+
+ cin->hsize = dflow->out_w;
+ cin->vsize = dflow->out_h;
+ cin->hoffset = dflow->out_x;
+ cin->voffset = dflow->out_y;
+ cin->pixel_blend_mode = dflow->pixel_blend_mode;
+ cin->layer_alpha = dflow->layer_alpha;
+
+ old_st = komeda_component_get_old_state(&compiz->base, drm_st);
+ WARN_ON(!old_st);
+
+ /* compare with old to check if this input has been changed */
+ if (memcmp(&(to_compiz_st(old_st)->cins[idx]), cin, sizeof(*cin)))
+ c_st->changed_active_inputs |= BIT(idx);
+
+ komeda_component_add_input(c_st, &dflow->input, idx);
+
+ return 0;
+}
+
+static int
+komeda_compiz_validate(struct komeda_compiz *compiz,
+ struct komeda_crtc_state *state,
+ struct komeda_data_flow_cfg *dflow)
+{
+ struct komeda_component_state *c_st;
+ struct komeda_compiz_state *st;
+
+ c_st = komeda_component_get_state_and_set_user(&compiz->base,
+ state->base.state, state->base.crtc, state->base.crtc);
+ if (IS_ERR(c_st))
+ return PTR_ERR(c_st);
+
+ st = to_compiz_st(c_st);
+
+ pipeline_composition_size(state, &st->hsize, &st->vsize);
+
+ komeda_component_set_output(&dflow->input, &compiz->base, 0);
+
+ /* compiz output dflow will be fed to the next pipeline stage, prepare
+ * the data flow configuration for the next stage
+ */
+ if (dflow) {
+ dflow->in_w = st->hsize;
+ dflow->in_h = st->vsize;
+ dflow->out_w = dflow->in_w;
+ dflow->out_h = dflow->in_h;
+ /* the output data of compiz doesn't have alpha, it only can be
+ * used as bottom layer when blend it with master layers
+ */
+ dflow->pixel_blend_mode = DRM_MODE_BLEND_PIXEL_NONE;
+ dflow->layer_alpha = 0xFF;
+ dflow->blending_zorder = 0;
+ }
+
+ return 0;
+}
+
+static int
+komeda_improc_validate(struct komeda_improc *improc,
+ struct komeda_crtc_state *kcrtc_st,
+ struct komeda_data_flow_cfg *dflow)
+{
+ struct drm_crtc *crtc = kcrtc_st->base.crtc;
+ struct komeda_component_state *c_st;
+ struct komeda_improc_state *st;
+
+ c_st = komeda_component_get_state_and_set_user(&improc->base,
+ kcrtc_st->base.state, crtc, crtc);
+ if (IS_ERR(c_st))
+ return PTR_ERR(c_st);
+
+ st = to_improc_st(c_st);
+
+ st->hsize = dflow->in_w;
+ st->vsize = dflow->in_h;
+
+ komeda_component_add_input(&st->base, &dflow->input, 0);
+ komeda_component_set_output(&dflow->input, &improc->base, 0);
+
+ return 0;
+}
+
+static int
+komeda_timing_ctrlr_validate(struct komeda_timing_ctrlr *ctrlr,
+ struct komeda_crtc_state *kcrtc_st,
+ struct komeda_data_flow_cfg *dflow)
+{
+ struct drm_crtc *crtc = kcrtc_st->base.crtc;
+ struct komeda_timing_ctrlr_state *st;
+ struct komeda_component_state *c_st;
+
+ c_st = komeda_component_get_state_and_set_user(&ctrlr->base,
+ kcrtc_st->base.state, crtc, crtc);
+ if (IS_ERR(c_st))
+ return PTR_ERR(c_st);
+
+ st = to_ctrlr_st(c_st);
+
+ komeda_component_add_input(&st->base, &dflow->input, 0);
+ komeda_component_set_output(&dflow->input, &ctrlr->base, 0);
+
+ return 0;
+}
+
+int komeda_build_layer_data_flow(struct komeda_layer *layer,
+ struct komeda_plane_state *kplane_st,
+ struct komeda_crtc_state *kcrtc_st,
+ struct komeda_data_flow_cfg *dflow)
+{
+ struct drm_plane *plane = kplane_st->base.plane;
+ struct komeda_pipeline *pipe = layer->base.pipeline;
+ int err;
+
+ DRM_DEBUG_ATOMIC("%s handling [PLANE:%d:%s]: src[x/y:%d/%d, w/h:%d/%d] disp[x/y:%d/%d, w/h:%d/%d]",
+ layer->base.name, plane->base.id, plane->name,
+ dflow->in_x, dflow->in_y, dflow->in_w, dflow->in_h,
+ dflow->out_x, dflow->out_y, dflow->out_w, dflow->out_h);
+
+ err = komeda_layer_validate(layer, kplane_st, dflow);
+ if (err)
+ return err;
+
+ err = komeda_compiz_set_input(pipe->compiz, kcrtc_st, dflow);
+
+ return err;
+}
+
+/* build display output data flow, the data path is:
+ * compiz -> improc -> timing_ctrlr
+ */
+int komeda_build_display_data_flow(struct komeda_crtc *kcrtc,
+ struct komeda_crtc_state *kcrtc_st)
+{
+ struct komeda_pipeline *master = kcrtc->master;
+ struct komeda_data_flow_cfg m_dflow; /* master data flow */
+ int err;
+
+ memset(&m_dflow, 0, sizeof(m_dflow));
+
+ err = komeda_compiz_validate(master->compiz, kcrtc_st, &m_dflow);
+ if (err)
+ return err;
+
+ err = komeda_improc_validate(master->improc, kcrtc_st, &m_dflow);
+ if (err)
+ return err;
+
+ err = komeda_timing_ctrlr_validate(master->ctrlr, kcrtc_st, &m_dflow);
+ if (err)
+ return err;
+
+ return 0;
+}
+
+static void
+komeda_pipeline_unbound_components(struct komeda_pipeline *pipe,
+ struct komeda_pipeline_state *new)
+{
+ struct drm_atomic_state *drm_st = new->obj.state;
+ struct komeda_pipeline_state *old = priv_to_pipe_st(pipe->obj.state);
+ struct komeda_component_state *c_st;
+ struct komeda_component *c;
+ u32 disabling_comps, id;
+
+ WARN_ON(!old);
+
+ disabling_comps = (~new->active_comps) & old->active_comps;
+
+ /* unbound all disabling component */
+ dp_for_each_set_bit(id, disabling_comps) {
+ c = komeda_pipeline_get_component(pipe, id);
+ c_st = komeda_component_get_state_and_set_user(c,
+ drm_st, NULL, new->crtc);
+ WARN_ON(IS_ERR(c_st));
+ }
+}
+
+/* release unclaimed pipeline resource */
+int komeda_release_unclaimed_resources(struct komeda_pipeline *pipe,
+ struct komeda_crtc_state *kcrtc_st)
+{
+ struct drm_atomic_state *drm_st = kcrtc_st->base.state;
+ struct komeda_pipeline_state *st;
+
+ /* ignore the pipeline which is not affected */
+ if (!pipe || !has_bit(pipe->id, kcrtc_st->affected_pipes))
+ return 0;
+
+ if (has_bit(pipe->id, kcrtc_st->active_pipes))
+ st = komeda_pipeline_get_new_state(pipe, drm_st);
+ else
+ st = komeda_pipeline_get_state_and_set_crtc(pipe, drm_st, NULL);
+
+ if (WARN_ON(IS_ERR_OR_NULL(st)))
+ return -EINVAL;
+
+ komeda_pipeline_unbound_components(pipe, st);
+
+ return 0;
+}
+
+void komeda_pipeline_disable(struct komeda_pipeline *pipe,
+ struct drm_atomic_state *old_state)
+{
+ struct komeda_pipeline_state *old;
+ struct komeda_component *c;
+ struct komeda_component_state *c_st;
+ u32 id, disabling_comps = 0;
+
+ old = komeda_pipeline_get_old_state(pipe, old_state);
+
+ disabling_comps = old->active_comps;
+ DRM_DEBUG_ATOMIC("PIPE%d: disabling_comps: 0x%x.\n",
+ pipe->id, disabling_comps);
+
+ dp_for_each_set_bit(id, disabling_comps) {
+ c = komeda_pipeline_get_component(pipe, id);
+ c_st = priv_to_comp_st(c->obj.state);
+
+ /*
+ * If we disabled a component then all active_inputs should be
+ * put in the list of changed_active_inputs, so they get
+ * re-enabled.
+ * This usually happens during a modeset when the pipeline is
+ * first disabled and then the actual state gets committed
+ * again.
+ */
+ c_st->changed_active_inputs |= c_st->active_inputs;
+
+ c->funcs->disable(c);
+ }
+}
+
+void komeda_pipeline_update(struct komeda_pipeline *pipe,
+ struct drm_atomic_state *old_state)
+{
+ struct komeda_pipeline_state *new = priv_to_pipe_st(pipe->obj.state);
+ struct komeda_pipeline_state *old;
+ struct komeda_component *c;
+ u32 id, changed_comps = 0;
+
+ old = komeda_pipeline_get_old_state(pipe, old_state);
+
+ changed_comps = new->active_comps | old->active_comps;
+
+ DRM_DEBUG_ATOMIC("PIPE%d: active_comps: 0x%x, changed: 0x%x.\n",
+ pipe->id, new->active_comps, changed_comps);
+
+ dp_for_each_set_bit(id, changed_comps) {
+ c = komeda_pipeline_get_component(pipe, id);
+
+ if (new->active_comps & BIT(c->id))
+ c->funcs->update(c, priv_to_comp_st(c->obj.state));
+ else
+ c->funcs->disable(c);
+ }
+}
#include <drm/drm_atomic.h>
#include <drm/drm_atomic_helper.h>
#include <drm/drm_plane_helper.h>
+#include <drm/drm_print.h>
#include "komeda_dev.h"
#include "komeda_kms.h"
+static int
+komeda_plane_init_data_flow(struct drm_plane_state *st,
+ struct komeda_data_flow_cfg *dflow)
+{
+ struct drm_framebuffer *fb = st->fb;
+
+ memset(dflow, 0, sizeof(*dflow));
+
+ dflow->blending_zorder = st->zpos;
+
+ /* if format doesn't have alpha, fix blend mode to PIXEL_NONE */
+ dflow->pixel_blend_mode = fb->format->has_alpha ?
+ st->pixel_blend_mode : DRM_MODE_BLEND_PIXEL_NONE;
+ dflow->layer_alpha = st->alpha >> 8;
+
+ dflow->out_x = st->crtc_x;
+ dflow->out_y = st->crtc_y;
+ dflow->out_w = st->crtc_w;
+ dflow->out_h = st->crtc_h;
+
+ dflow->in_x = st->src_x >> 16;
+ dflow->in_y = st->src_y >> 16;
+ dflow->in_w = st->src_w >> 16;
+ dflow->in_h = st->src_h >> 16;
+
+ return 0;
+}
+
+/**
+ * komeda_plane_atomic_check - build input data flow
+ * @plane: DRM plane
+ * @state: the plane state object
+ *
+ * RETURNS:
+ * Zero for success or -errno
+ */
+static int
+komeda_plane_atomic_check(struct drm_plane *plane,
+ struct drm_plane_state *state)
+{
+ struct komeda_plane *kplane = to_kplane(plane);
+ struct komeda_plane_state *kplane_st = to_kplane_st(state);
+ struct komeda_layer *layer = kplane->layer;
+ struct drm_crtc_state *crtc_st;
+ struct komeda_crtc *kcrtc;
+ struct komeda_crtc_state *kcrtc_st;
+ struct komeda_data_flow_cfg dflow;
+ int err;
+
+ if (!state->crtc || !state->fb)
+ return 0;
+
+ crtc_st = drm_atomic_get_crtc_state(state->state, state->crtc);
+ if (!crtc_st->enable) {
+ DRM_DEBUG_ATOMIC("Cannot update plane on a disabled CRTC.\n");
+ return -EINVAL;
+ }
+
+ /* crtc is inactive, skip the resource assignment */
+ if (!crtc_st->active)
+ return 0;
+
+ kcrtc = to_kcrtc(state->crtc);
+ kcrtc_st = to_kcrtc_st(crtc_st);
+
+ err = komeda_plane_init_data_flow(state, &dflow);
+ if (err)
+ return err;
+
+ err = komeda_build_layer_data_flow(layer, kplane_st, kcrtc_st, &dflow);
+
+ return err;
+}
+
+/* plane doesn't represent a real HW, so there is no HW update for plane.
+ * komeda handles all the HW update in crtc->atomic_flush
+ */
+static void
+komeda_plane_atomic_update(struct drm_plane *plane,
+ struct drm_plane_state *old_state)
+{
+}
+
static const struct drm_plane_helper_funcs komeda_plane_helper_funcs = {
+ .atomic_check = komeda_plane_atomic_check,
+ .atomic_update = komeda_plane_atomic_update,
};
static void komeda_plane_destroy(struct drm_plane *plane)
kfree(to_kplane(plane));
}
+static void komeda_plane_reset(struct drm_plane *plane)
+{
+ struct komeda_plane_state *state;
+ struct komeda_plane *kplane = to_kplane(plane);
+
+ if (plane->state)
+ __drm_atomic_helper_plane_destroy_state(plane->state);
+
+ kfree(plane->state);
+ plane->state = NULL;
+
+ state = kzalloc(sizeof(*state), GFP_KERNEL);
+ if (state) {
+ state->base.rotation = DRM_MODE_ROTATE_0;
+ state->base.pixel_blend_mode = DRM_MODE_BLEND_PREMULTI;
+ state->base.alpha = DRM_BLEND_ALPHA_OPAQUE;
+ state->base.zpos = kplane->layer->base.id;
+ plane->state = &state->base;
+ plane->state->plane = plane;
+ }
+}
+
+static struct drm_plane_state *
+komeda_plane_atomic_duplicate_state(struct drm_plane *plane)
+{
+ struct komeda_plane_state *new;
+
+ if (WARN_ON(!plane->state))
+ return NULL;
+
+ new = kzalloc(sizeof(*new), GFP_KERNEL);
+ if (!new)
+ return NULL;
+
+ __drm_atomic_helper_plane_duplicate_state(plane, &new->base);
+
+ return &new->base;
+}
+
+static void
+komeda_plane_atomic_destroy_state(struct drm_plane *plane,
+ struct drm_plane_state *state)
+{
+ __drm_atomic_helper_plane_destroy_state(state);
+ kfree(to_kplane_st(state));
+}
+
static const struct drm_plane_funcs komeda_plane_funcs = {
+ .update_plane = drm_atomic_helper_update_plane,
+ .disable_plane = drm_atomic_helper_disable_plane,
+ .destroy = komeda_plane_destroy,
+ .reset = komeda_plane_reset,
+ .atomic_duplicate_state = komeda_plane_atomic_duplicate_state,
+ .atomic_destroy_state = komeda_plane_atomic_destroy_state,
};
/* for komeda, which is pipeline can be share between crtcs */
#include "komeda_dev.h"
#include "komeda_kms.h"
+static void
+komeda_component_state_reset(struct komeda_component_state *st)
+{
+ st->binding_user = NULL;
+ st->affected_inputs = st->active_inputs;
+ st->active_inputs = 0;
+ st->changed_active_inputs = 0;
+}
+
+static struct drm_private_state *
+komeda_layer_atomic_duplicate_state(struct drm_private_obj *obj)
+{
+ struct komeda_layer_state *st;
+
+ st = kmemdup(obj->state, sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return NULL;
+
+ komeda_component_state_reset(&st->base);
+ __drm_atomic_helper_private_obj_duplicate_state(obj, &st->base.obj);
+
+ return &st->base.obj;
+}
+
+static void
+komeda_layer_atomic_destroy_state(struct drm_private_obj *obj,
+ struct drm_private_state *state)
+{
+ struct komeda_layer_state *st = to_layer_st(priv_to_comp_st(state));
+
+ kfree(st);
+}
+
+static const struct drm_private_state_funcs komeda_layer_obj_funcs = {
+ .atomic_duplicate_state = komeda_layer_atomic_duplicate_state,
+ .atomic_destroy_state = komeda_layer_atomic_destroy_state,
+};
+
+static int komeda_layer_obj_add(struct komeda_kms_dev *kms,
+ struct komeda_layer *layer)
+{
+ struct komeda_layer_state *st;
+
+ st = kzalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return -ENOMEM;
+
+ st->base.component = &layer->base;
+ drm_atomic_private_obj_init(&kms->base, &layer->base.obj, &st->base.obj,
+ &komeda_layer_obj_funcs);
+ return 0;
+}
+
+static struct drm_private_state *
+komeda_compiz_atomic_duplicate_state(struct drm_private_obj *obj)
+{
+ struct komeda_compiz_state *st;
+
+ st = kmemdup(obj->state, sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return NULL;
+
+ komeda_component_state_reset(&st->base);
+ __drm_atomic_helper_private_obj_duplicate_state(obj, &st->base.obj);
+
+ return &st->base.obj;
+}
+
+static void
+komeda_compiz_atomic_destroy_state(struct drm_private_obj *obj,
+ struct drm_private_state *state)
+{
+ kfree(to_compiz_st(priv_to_comp_st(state)));
+}
+
+static const struct drm_private_state_funcs komeda_compiz_obj_funcs = {
+ .atomic_duplicate_state = komeda_compiz_atomic_duplicate_state,
+ .atomic_destroy_state = komeda_compiz_atomic_destroy_state,
+};
+
+static int komeda_compiz_obj_add(struct komeda_kms_dev *kms,
+ struct komeda_compiz *compiz)
+{
+ struct komeda_compiz_state *st;
+
+ st = kzalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return -ENOMEM;
+
+ st->base.component = &compiz->base;
+ drm_atomic_private_obj_init(&kms->base, &compiz->base.obj, &st->base.obj,
+ &komeda_compiz_obj_funcs);
+
+ return 0;
+}
+
+static struct drm_private_state *
+komeda_improc_atomic_duplicate_state(struct drm_private_obj *obj)
+{
+ struct komeda_improc_state *st;
+
+ st = kmemdup(obj->state, sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return NULL;
+
+ komeda_component_state_reset(&st->base);
+ __drm_atomic_helper_private_obj_duplicate_state(obj, &st->base.obj);
+
+ return &st->base.obj;
+}
+
+static void
+komeda_improc_atomic_destroy_state(struct drm_private_obj *obj,
+ struct drm_private_state *state)
+{
+ kfree(to_improc_st(priv_to_comp_st(state)));
+}
+
+static const struct drm_private_state_funcs komeda_improc_obj_funcs = {
+ .atomic_duplicate_state = komeda_improc_atomic_duplicate_state,
+ .atomic_destroy_state = komeda_improc_atomic_destroy_state,
+};
+
+static int komeda_improc_obj_add(struct komeda_kms_dev *kms,
+ struct komeda_improc *improc)
+{
+ struct komeda_improc_state *st;
+
+ st = kzalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return -ENOMEM;
+
+ st->base.component = &improc->base;
+ drm_atomic_private_obj_init(&kms->base, &improc->base.obj, &st->base.obj,
+ &komeda_improc_obj_funcs);
+
+ return 0;
+}
+
+static struct drm_private_state *
+komeda_timing_ctrlr_atomic_duplicate_state(struct drm_private_obj *obj)
+{
+ struct komeda_timing_ctrlr_state *st;
+
+ st = kmemdup(obj->state, sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return NULL;
+
+ komeda_component_state_reset(&st->base);
+ __drm_atomic_helper_private_obj_duplicate_state(obj, &st->base.obj);
+
+ return &st->base.obj;
+}
+
+static void
+komeda_timing_ctrlr_atomic_destroy_state(struct drm_private_obj *obj,
+ struct drm_private_state *state)
+{
+ kfree(to_ctrlr_st(priv_to_comp_st(state)));
+}
+
+static const struct drm_private_state_funcs komeda_timing_ctrlr_obj_funcs = {
+ .atomic_duplicate_state = komeda_timing_ctrlr_atomic_duplicate_state,
+ .atomic_destroy_state = komeda_timing_ctrlr_atomic_destroy_state,
+};
+
+static int komeda_timing_ctrlr_obj_add(struct komeda_kms_dev *kms,
+ struct komeda_timing_ctrlr *ctrlr)
+{
+ struct komeda_compiz_state *st;
+
+ st = kzalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return -ENOMEM;
+
+ st->base.component = &ctrlr->base;
+ drm_atomic_private_obj_init(&kms->base, &ctrlr->base.obj, &st->base.obj,
+ &komeda_timing_ctrlr_obj_funcs);
+
+ return 0;
+}
+
static struct drm_private_state *
komeda_pipeline_atomic_duplicate_state(struct drm_private_obj *obj)
{
struct komeda_dev *mdev)
{
struct komeda_pipeline *pipe;
- int i, err;
+ int i, j, err;
for (i = 0; i < mdev->n_pipelines; i++) {
pipe = mdev->pipelines[i];
if (err)
return err;
- /* Add component */
+ for (j = 0; j < pipe->n_layers; j++) {
+ err = komeda_layer_obj_add(kms, pipe->layers[j]);
+ if (err)
+ return err;
+ }
+
+ err = komeda_compiz_obj_add(kms, pipe->compiz);
+ if (err)
+ return err;
+
+ err = komeda_improc_obj_add(kms, pipe->improc);
+ if (err)
+ return err;
+
+ err = komeda_timing_ctrlr_obj_add(kms, pipe->ctrlr);
+ if (err)
+ return err;
}
return 0;
}
-void komeda_kms_cleanup_private_objs(struct komeda_dev *mdev)
+void komeda_kms_cleanup_private_objs(struct komeda_kms_dev *kms)
{
- struct komeda_pipeline *pipe;
- struct komeda_component *c;
- int i, id;
+ struct drm_mode_config *config = &kms->base.mode_config;
+ struct drm_private_obj *obj, *next;
- for (i = 0; i < mdev->n_pipelines; i++) {
- pipe = mdev->pipelines[i];
- dp_for_each_set_bit(id, pipe->avail_comps) {
- c = komeda_pipeline_get_component(pipe, id);
-
- drm_atomic_private_obj_fini(&c->obj);
- }
- drm_atomic_private_obj_fini(&pipe->obj);
- }
+ list_for_each_entry_safe(obj, next, &config->privobj_list, head)
+ drm_atomic_private_obj_fini(obj);
}
malidp_verify_afbc_framebuffer_caps(struct drm_device *dev,
const struct drm_mode_fb_cmd2 *mode_cmd)
{
- const struct drm_format_info *info;
-
- if ((mode_cmd->modifier[0] >> 56) != DRM_FORMAT_MOD_VENDOR_ARM) {
- DRM_DEBUG_KMS("Unknown modifier (not Arm)\n");
- return false;
- }
-
- if (mode_cmd->modifier[0] &
- ~DRM_FORMAT_MOD_ARM_AFBC(AFBC_MOD_VALID_BITS)) {
- DRM_DEBUG_KMS("Unsupported modifiers\n");
- return false;
- }
-
- info = drm_get_format_info(dev, mode_cmd);
- if (!info) {
- DRM_DEBUG_KMS("Unable to get the format information\n");
+ if (malidp_format_mod_supported(dev, mode_cmd->pixel_format,
+ mode_cmd->modifier[0]) == false)
return false;
- }
-
- if (info->num_planes != 1) {
- DRM_DEBUG_KMS("AFBC buffers expect one plane\n");
- return false;
- }
if (mode_cmd->offsets[0] != 0) {
DRM_DEBUG_KMS("AFBC buffers' plane offset should be 0\n");
return false;
}
- switch (mode_cmd->modifier[0] & AFBC_FORMAT_MOD_BLOCK_SIZE_MASK) {
- case AFBC_FORMAT_MOD_BLOCK_SIZE_16x16:
+ switch (mode_cmd->modifier[0] & AFBC_SIZE_MASK) {
+ case AFBC_SIZE_16X16:
if ((mode_cmd->width % 16) || (mode_cmd->height % 16)) {
DRM_DEBUG_KMS("AFBC buffers must be aligned to 16 pixels\n");
return false;
struct drm_gem_object *objs = NULL;
u32 afbc_superblock_size = 0, afbc_superblock_height = 0;
u32 afbc_superblock_width = 0, afbc_size = 0;
+ int bpp = 0;
- switch (mode_cmd->modifier[0] & AFBC_FORMAT_MOD_BLOCK_SIZE_MASK) {
- case AFBC_FORMAT_MOD_BLOCK_SIZE_16x16:
+ switch (mode_cmd->modifier[0] & AFBC_SIZE_MASK) {
+ case AFBC_SIZE_16X16:
afbc_superblock_height = 16;
afbc_superblock_width = 16;
break;
n_superblocks = (mode_cmd->width / afbc_superblock_width) *
(mode_cmd->height / afbc_superblock_height);
- afbc_superblock_size = info->cpp[0] * afbc_superblock_width *
- afbc_superblock_height;
+ bpp = malidp_format_get_bpp(info->format);
+
+ afbc_superblock_size = (bpp * afbc_superblock_width * afbc_superblock_height)
+ / BITS_PER_BYTE;
afbc_size = ALIGN(n_superblocks * AFBC_HEADER_SIZE, AFBC_SUPERBLK_ALIGNMENT);
afbc_size += n_superblocks * ALIGN(afbc_superblock_size, AFBC_SUPERBLK_ALIGNMENT);
- if (mode_cmd->width * info->cpp[0] != mode_cmd->pitches[0]) {
- DRM_DEBUG_KMS("Invalid value of pitch (=%u) should be same as width (=%u) * cpp (=%u)\n",
- mode_cmd->pitches[0], mode_cmd->width, info->cpp[0]);
+ if ((mode_cmd->width * bpp) != (mode_cmd->pitches[0] * BITS_PER_BYTE)) {
+ DRM_DEBUG_KMS("Invalid value of (pitch * BITS_PER_BYTE) (=%u) "
+ "should be same as width (=%u) * bpp (=%u)\n",
+ (mode_cmd->pitches[0] * BITS_PER_BYTE),
+ mode_cmd->width, bpp);
return false;
}
drm->mode_config.max_height = hwdev->max_line_size;
drm->mode_config.funcs = &malidp_mode_config_funcs;
drm->mode_config.helper_private = &malidp_mode_config_helpers;
+ drm->mode_config.allow_fb_modifiers = true;
ret = malidp_crtc_init(drm);
if (ret)
int malidp_de_planes_init(struct drm_device *drm);
int malidp_crtc_init(struct drm_device *drm);
+bool malidp_hw_format_is_linear_only(u32 format);
+bool malidp_hw_format_is_afbc_only(u32 format);
+
+bool malidp_format_mod_supported(struct drm_device *drm,
+ u32 format, u64 modifier);
+
#ifdef CONFIG_DEBUG_FS
void malidp_error(struct malidp_drm *malidp,
struct malidp_error_stats *error_stats, u32 status,
{ DRM_FORMAT_YUYV, DE_VIDEO1, 13 },
{ DRM_FORMAT_NV12, DE_VIDEO1 | SE_MEMWRITE, 14 },
{ DRM_FORMAT_YUV420, DE_VIDEO1, 15 },
+ { DRM_FORMAT_XYUV8888, DE_VIDEO1, 16 },
+ /* These are supported with AFBC only */
+ { DRM_FORMAT_YUV420_8BIT, DE_VIDEO1, 14 },
+ { DRM_FORMAT_VUY888, DE_VIDEO1, 16 },
+ { DRM_FORMAT_VUY101010, DE_VIDEO1, 17 },
+ { DRM_FORMAT_YUV420_10BIT, DE_VIDEO1, 18 }
};
#define MALIDP_ID(__group, __format) \
((((__group) & 0x7) << 3) | ((__format) & 0x7))
+#define AFBC_YUV_422_FORMAT_ID MALIDP_ID(5, 1)
+
#define MALIDP_COMMON_FORMATS \
/* fourcc, layers supporting the format, internal id */ \
{ DRM_FORMAT_ARGB2101010, DE_VIDEO1 | DE_GRAPHICS1 | DE_VIDEO2 | SE_MEMWRITE, MALIDP_ID(0, 0) }, \
{ DRM_FORMAT_ABGR1555, DE_VIDEO1 | DE_GRAPHICS1 | DE_VIDEO2, MALIDP_ID(4, 1) }, \
{ DRM_FORMAT_RGB565, DE_VIDEO1 | DE_GRAPHICS1 | DE_VIDEO2, MALIDP_ID(4, 2) }, \
{ DRM_FORMAT_BGR565, DE_VIDEO1 | DE_GRAPHICS1 | DE_VIDEO2, MALIDP_ID(4, 3) }, \
+ /* This is only supported with linear modifier */ \
+ { DRM_FORMAT_XYUV8888, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(5, 0) },\
+ /* This is only supported with AFBC modifier */ \
+ { DRM_FORMAT_VUY888, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(5, 0) }, \
{ DRM_FORMAT_YUYV, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(5, 2) }, \
+ /* This is only supported with linear modifier */ \
{ DRM_FORMAT_UYVY, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(5, 3) }, \
{ DRM_FORMAT_NV12, DE_VIDEO1 | DE_VIDEO2 | SE_MEMWRITE, MALIDP_ID(5, 6) }, \
+ /* This is only supported with AFBC modifier */ \
+ { DRM_FORMAT_YUV420_8BIT, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(5, 6) }, \
{ DRM_FORMAT_YUV420, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(5, 7) }, \
- { DRM_FORMAT_X0L2, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(6, 6)}
+ /* This is only supported with linear modifier */ \
+ { DRM_FORMAT_XVYU2101010, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(6, 0)}, \
+ /* This is only supported with AFBC modifier */ \
+ { DRM_FORMAT_VUY101010, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(6, 0)}, \
+ { DRM_FORMAT_X0L2, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(6, 6)}, \
+ /* This is only supported with AFBC modifier */ \
+ { DRM_FORMAT_YUV420_10BIT, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(6, 7)}, \
+ { DRM_FORMAT_P010, DE_VIDEO1 | DE_VIDEO2, MALIDP_ID(6, 7)}
static const struct malidp_format_id malidp550_de_formats[] = {
MALIDP_COMMON_FORMATS,
* yuv2rgb matrix offset, mmu control register offset, rotation_features
*/
{ DE_VIDEO1, MALIDP500_DE_LV_BASE, MALIDP500_DE_LV_PTR_BASE,
- MALIDP_DE_LV_STRIDE0, MALIDP500_LV_YUV2RGB, 0, ROTATE_ANY },
+ MALIDP_DE_LV_STRIDE0, MALIDP500_LV_YUV2RGB, 0, ROTATE_ANY,
+ MALIDP500_DE_LV_AD_CTRL },
{ DE_GRAPHICS1, MALIDP500_DE_LG1_BASE, MALIDP500_DE_LG1_PTR_BASE,
- MALIDP_DE_LG_STRIDE, 0, 0, ROTATE_ANY },
+ MALIDP_DE_LG_STRIDE, 0, 0, ROTATE_ANY,
+ MALIDP500_DE_LG1_AD_CTRL },
{ DE_GRAPHICS2, MALIDP500_DE_LG2_BASE, MALIDP500_DE_LG2_PTR_BASE,
- MALIDP_DE_LG_STRIDE, 0, 0, ROTATE_ANY },
+ MALIDP_DE_LG_STRIDE, 0, 0, ROTATE_ANY,
+ MALIDP500_DE_LG2_AD_CTRL },
};
static const struct malidp_layer malidp550_layers[] = {
* yuv2rgb matrix offset, mmu control register offset, rotation_features
*/
{ DE_VIDEO1, MALIDP550_DE_LV1_BASE, MALIDP550_DE_LV1_PTR_BASE,
- MALIDP_DE_LV_STRIDE0, MALIDP550_LV_YUV2RGB, 0, ROTATE_ANY },
+ MALIDP_DE_LV_STRIDE0, MALIDP550_LV_YUV2RGB, 0, ROTATE_ANY,
+ MALIDP550_DE_LV1_AD_CTRL },
{ DE_GRAPHICS1, MALIDP550_DE_LG_BASE, MALIDP550_DE_LG_PTR_BASE,
- MALIDP_DE_LG_STRIDE, 0, 0, ROTATE_ANY },
+ MALIDP_DE_LG_STRIDE, 0, 0, ROTATE_ANY,
+ MALIDP550_DE_LG_AD_CTRL },
{ DE_VIDEO2, MALIDP550_DE_LV2_BASE, MALIDP550_DE_LV2_PTR_BASE,
- MALIDP_DE_LV_STRIDE0, MALIDP550_LV_YUV2RGB, 0, ROTATE_ANY },
+ MALIDP_DE_LV_STRIDE0, MALIDP550_LV_YUV2RGB, 0, ROTATE_ANY,
+ MALIDP550_DE_LV2_AD_CTRL },
{ DE_SMART, MALIDP550_DE_LS_BASE, MALIDP550_DE_LS_PTR_BASE,
- MALIDP550_DE_LS_R1_STRIDE, 0, 0, ROTATE_NONE },
+ MALIDP550_DE_LS_R1_STRIDE, 0, 0, ROTATE_NONE, 0 },
};
static const struct malidp_layer malidp650_layers[] = {
*/
{ DE_VIDEO1, MALIDP550_DE_LV1_BASE, MALIDP550_DE_LV1_PTR_BASE,
MALIDP_DE_LV_STRIDE0, MALIDP550_LV_YUV2RGB,
- MALIDP650_DE_LV_MMU_CTRL, ROTATE_ANY },
+ MALIDP650_DE_LV_MMU_CTRL, ROTATE_ANY,
+ MALIDP550_DE_LV1_AD_CTRL },
{ DE_GRAPHICS1, MALIDP550_DE_LG_BASE, MALIDP550_DE_LG_PTR_BASE,
MALIDP_DE_LG_STRIDE, 0, MALIDP650_DE_LG_MMU_CTRL,
- ROTATE_COMPRESSED },
+ ROTATE_COMPRESSED, MALIDP550_DE_LG_AD_CTRL },
{ DE_VIDEO2, MALIDP550_DE_LV2_BASE, MALIDP550_DE_LV2_PTR_BASE,
MALIDP_DE_LV_STRIDE0, MALIDP550_LV_YUV2RGB,
- MALIDP650_DE_LV_MMU_CTRL, ROTATE_ANY },
+ MALIDP650_DE_LV_MMU_CTRL, ROTATE_ANY,
+ MALIDP550_DE_LV2_AD_CTRL },
{ DE_SMART, MALIDP550_DE_LS_BASE, MALIDP550_DE_LS_PTR_BASE,
MALIDP550_DE_LS_R1_STRIDE, 0, MALIDP650_DE_LS_MMU_CTRL,
- ROTATE_NONE },
+ ROTATE_NONE, 0 },
+};
+
+const u64 malidp_format_modifiers[] = {
+ /* All RGB formats (except XRGB, RGBX, XBGR, BGRX) */
+ DRM_FORMAT_MOD_ARM_AFBC(AFBC_SIZE_16X16 | AFBC_YTR | AFBC_SPARSE),
+ DRM_FORMAT_MOD_ARM_AFBC(AFBC_SIZE_16X16 | AFBC_YTR),
+
+ /* All RGB formats > 16bpp (except XRGB, RGBX, XBGR, BGRX) */
+ DRM_FORMAT_MOD_ARM_AFBC(AFBC_SIZE_16X16 | AFBC_YTR | AFBC_SPARSE | AFBC_SPLIT),
+
+ /* All 8 or 10 bit YUV 444 formats. */
+ /* In DP550, 10 bit YUV 420 format also supported */
+ DRM_FORMAT_MOD_ARM_AFBC(AFBC_SIZE_16X16 | AFBC_SPARSE | AFBC_SPLIT),
+
+ /* YUV 420, 422 P1 8 bit and YUV 444 8 bit/10 bit formats */
+ DRM_FORMAT_MOD_ARM_AFBC(AFBC_SIZE_16X16 | AFBC_SPARSE),
+ DRM_FORMAT_MOD_ARM_AFBC(AFBC_SIZE_16X16),
+
+ /* YUV 420, 422 P1 8, 10 bit formats */
+ DRM_FORMAT_MOD_ARM_AFBC(AFBC_SIZE_16X16 | AFBC_CBR | AFBC_SPARSE),
+ DRM_FORMAT_MOD_ARM_AFBC(AFBC_SIZE_16X16 | AFBC_CBR),
+
+ /* All formats */
+ DRM_FORMAT_MOD_LINEAR,
+
+ DRM_FORMAT_MOD_INVALID
};
#define SE_N_SCALING_COEFFS 96
malidp_hw_clearbits(hwdev, MALIDP_DISP_FUNC_ILACED, MALIDP_DE_DISPLAY_FUNC);
}
-static int malidp500_rotmem_required(struct malidp_hw_device *hwdev, u16 w, u16 h, u32 fmt)
+int malidp_format_get_bpp(u32 fmt)
+{
+ int bpp = drm_format_plane_cpp(fmt, 0) * 8;
+
+ if (bpp == 0) {
+ switch (fmt) {
+ case DRM_FORMAT_VUY101010:
+ bpp = 30;
+ case DRM_FORMAT_YUV420_10BIT:
+ bpp = 15;
+ break;
+ case DRM_FORMAT_YUV420_8BIT:
+ bpp = 12;
+ break;
+ default:
+ bpp = 0;
+ }
+ }
+
+ return bpp;
+}
+
+static int malidp500_rotmem_required(struct malidp_hw_device *hwdev, u16 w,
+ u16 h, u32 fmt, bool has_modifier)
{
/*
* Each layer needs enough rotation memory to fit 8 lines
* worth of pixel data. Required size is then:
* size = rotated_width * (bpp / 8) * 8;
*/
- return w * drm_format_plane_cpp(fmt, 0) * 8;
+ int bpp = malidp_format_get_bpp(fmt);
+
+ return w * bpp;
}
static void malidp500_se_write_pp_coefftab(struct malidp_hw_device *hwdev,
malidp_hw_clearbits(hwdev, MALIDP_DISP_FUNC_ILACED, MALIDP_DE_DISPLAY_FUNC);
}
-static int malidp550_rotmem_required(struct malidp_hw_device *hwdev, u16 w, u16 h, u32 fmt)
+static int malidpx50_get_bytes_per_column(u32 fmt)
{
- u32 bytes_per_col;
+ u32 bytes_per_column;
switch (fmt) {
/* 8 lines at 4 bytes per pixel */
case DRM_FORMAT_UYVY:
case DRM_FORMAT_YUYV:
case DRM_FORMAT_X0L0:
- case DRM_FORMAT_X0L2:
- bytes_per_col = 32;
+ bytes_per_column = 32;
break;
/* 16 lines at 1.5 bytes per pixel */
case DRM_FORMAT_NV12:
case DRM_FORMAT_YUV420:
- bytes_per_col = 24;
+ /* 8 lines at 3 bytes per pixel */
+ case DRM_FORMAT_VUY888:
+ /* 16 lines at 12 bits per pixel */
+ case DRM_FORMAT_YUV420_8BIT:
+ /* 8 lines at 3 bytes per pixel */
+ case DRM_FORMAT_P010:
+ bytes_per_column = 24;
+ break;
+ /* 8 lines at 30 bits per pixel */
+ case DRM_FORMAT_VUY101010:
+ /* 16 lines at 15 bits per pixel */
+ case DRM_FORMAT_YUV420_10BIT:
+ bytes_per_column = 30;
break;
default:
return -EINVAL;
}
- return w * bytes_per_col;
+ return bytes_per_column;
+}
+
+static int malidp550_rotmem_required(struct malidp_hw_device *hwdev, u16 w,
+ u16 h, u32 fmt, bool has_modifier)
+{
+ int bytes_per_column = 0;
+
+ switch (fmt) {
+ /* 8 lines at 15 bits per pixel */
+ case DRM_FORMAT_YUV420_10BIT:
+ bytes_per_column = 15;
+ break;
+ /* Uncompressed YUV 420 10 bit single plane cannot be rotated */
+ case DRM_FORMAT_X0L2:
+ if (has_modifier)
+ bytes_per_column = 8;
+ else
+ return -EINVAL;
+ break;
+ default:
+ bytes_per_column = malidpx50_get_bytes_per_column(fmt);
+ }
+
+ if (bytes_per_column == -EINVAL)
+ return bytes_per_column;
+
+ return w * bytes_per_column;
+}
+
+static int malidp650_rotmem_required(struct malidp_hw_device *hwdev, u16 w,
+ u16 h, u32 fmt, bool has_modifier)
+{
+ int bytes_per_column = 0;
+
+ switch (fmt) {
+ /* 16 lines at 2 bytes per pixel */
+ case DRM_FORMAT_X0L2:
+ bytes_per_column = 32;
+ break;
+ default:
+ bytes_per_column = malidpx50_get_bytes_per_column(fmt);
+ }
+
+ if (bytes_per_column == -EINVAL)
+ return bytes_per_column;
+
+ return w * bytes_per_column;
}
static int malidp550_se_set_scaling_coeffs(struct malidp_hw_device *hwdev,
.se_base = MALIDP550_SE_BASE,
.dc_base = MALIDP550_DC_BASE,
.out_depth_base = MALIDP550_DE_OUTPUT_DEPTH,
- .features = MALIDP_REGMAP_HAS_CLEARIRQ,
+ .features = MALIDP_REGMAP_HAS_CLEARIRQ |
+ MALIDP_DEVICE_AFBC_SUPPORT_SPLIT |
+ MALIDP_DEVICE_AFBC_YUV_420_10_SUPPORT_SPLIT |
+ MALIDP_DEVICE_AFBC_YUYV_USE_422_P2,
.n_layers = ARRAY_SIZE(malidp550_layers),
.layers = malidp550_layers,
.de_irq_map = {
.se_base = MALIDP550_SE_BASE,
.dc_base = MALIDP550_DC_BASE,
.out_depth_base = MALIDP550_DE_OUTPUT_DEPTH,
- .features = MALIDP_REGMAP_HAS_CLEARIRQ,
+ .features = MALIDP_REGMAP_HAS_CLEARIRQ |
+ MALIDP_DEVICE_AFBC_SUPPORT_SPLIT |
+ MALIDP_DEVICE_AFBC_YUYV_USE_422_P2,
.n_layers = ARRAY_SIZE(malidp650_layers),
.layers = malidp650_layers,
.de_irq_map = {
.in_config_mode = malidp550_in_config_mode,
.set_config_valid = malidp550_set_config_valid,
.modeset = malidp550_modeset,
- .rotmem_required = malidp550_rotmem_required,
+ .rotmem_required = malidp650_rotmem_required,
.se_set_scaling_coeffs = malidp550_se_set_scaling_coeffs,
.se_calc_mclk = malidp550_se_calc_mclk,
.enable_memwrite = malidp550_enable_memwrite,
};
u8 malidp_hw_get_format_id(const struct malidp_hw_regmap *map,
- u8 layer_id, u32 format)
+ u8 layer_id, u32 format, bool has_modifier)
{
unsigned int i;
for (i = 0; i < map->n_pixel_formats; i++) {
if (((map->pixel_formats[i].layer & layer_id) == layer_id) &&
- (map->pixel_formats[i].format == format))
- return map->pixel_formats[i].id;
+ (map->pixel_formats[i].format == format)) {
+ /*
+ * In some DP550 and DP650, DRM_FORMAT_YUYV + AFBC modifier
+ * is supported by a different h/w format id than
+ * DRM_FORMAT_YUYV (only).
+ */
+ if (format == DRM_FORMAT_YUYV &&
+ (has_modifier) &&
+ (map->features & MALIDP_DEVICE_AFBC_YUYV_USE_422_P2))
+ return AFBC_YUV_422_FORMAT_ID;
+ else
+ return map->pixel_formats[i].id;
+ }
}
return MALIDP_INVALID_FORMAT_ID;
}
+bool malidp_hw_format_is_linear_only(u32 format)
+{
+ switch (format) {
+ case DRM_FORMAT_ARGB2101010:
+ case DRM_FORMAT_RGBA1010102:
+ case DRM_FORMAT_BGRA1010102:
+ case DRM_FORMAT_ARGB8888:
+ case DRM_FORMAT_RGBA8888:
+ case DRM_FORMAT_BGRA8888:
+ case DRM_FORMAT_XBGR8888:
+ case DRM_FORMAT_XRGB8888:
+ case DRM_FORMAT_RGBX8888:
+ case DRM_FORMAT_BGRX8888:
+ case DRM_FORMAT_RGB888:
+ case DRM_FORMAT_RGB565:
+ case DRM_FORMAT_ARGB1555:
+ case DRM_FORMAT_RGBA5551:
+ case DRM_FORMAT_BGRA5551:
+ case DRM_FORMAT_UYVY:
+ case DRM_FORMAT_XYUV8888:
+ case DRM_FORMAT_XVYU2101010:
+ case DRM_FORMAT_X0L2:
+ case DRM_FORMAT_X0L0:
+ return true;
+ default:
+ return false;
+ }
+}
+
+bool malidp_hw_format_is_afbc_only(u32 format)
+{
+ switch (format) {
+ case DRM_FORMAT_VUY888:
+ case DRM_FORMAT_VUY101010:
+ case DRM_FORMAT_YUV420_8BIT:
+ case DRM_FORMAT_YUV420_10BIT:
+ return true;
+ default:
+ return false;
+ }
+}
+
static void malidp_hw_clear_irq(struct malidp_hw_device *hwdev, u8 block, u32 irq)
{
u32 base = malidp_get_block_base(hwdev, block);
s16 yuv2rgb_offset; /* offset to the YUV->RGB matrix entries */
u16 mmu_ctrl_offset; /* offset to the MMU control register */
enum rotation_features rot; /* type of rotation supported */
+ /* address offset for the AFBC decoder registers */
+ u16 afbc_decoder_offset;
};
enum malidp_scaling_coeff_set {
};
/* regmap features */
-#define MALIDP_REGMAP_HAS_CLEARIRQ (1 << 0)
+#define MALIDP_REGMAP_HAS_CLEARIRQ BIT(0)
+#define MALIDP_DEVICE_AFBC_SUPPORT_SPLIT BIT(1)
+#define MALIDP_DEVICE_AFBC_YUV_420_10_SUPPORT_SPLIT BIT(2)
+#define MALIDP_DEVICE_AFBC_YUYV_USE_422_P2 BIT(3)
struct malidp_hw_regmap {
/* address offset of the DE register bank */
* Calculate the required rotation memory given the active area
* and the buffer format.
*/
- int (*rotmem_required)(struct malidp_hw_device *hwdev, u16 w, u16 h, u32 fmt);
+ int (*rotmem_required)(struct malidp_hw_device *hwdev, u16 w, u16 h,
+ u32 fmt, bool has_modifier);
int (*se_set_scaling_coeffs)(struct malidp_hw_device *hwdev,
struct malidp_se_config *se_config,
void malidp_se_irq_fini(struct malidp_hw_device *hwdev);
u8 malidp_hw_get_format_id(const struct malidp_hw_regmap *map,
- u8 layer_id, u32 format);
+ u8 layer_id, u32 format, bool has_modifier);
+
+int malidp_format_get_bpp(u32 fmt);
static inline u8 malidp_hw_get_pitch_align(struct malidp_hw_device *hwdev, bool rotated)
{
#define MALIDP_GAMMA_LUT_SIZE 4096
-#define AFBC_MOD_VALID_BITS (AFBC_FORMAT_MOD_BLOCK_SIZE_MASK | \
- AFBC_FORMAT_MOD_YTR | AFBC_FORMAT_MOD_SPLIT | \
- AFBC_FORMAT_MOD_SPARSE | AFBC_FORMAT_MOD_CBR | \
- AFBC_FORMAT_MOD_TILED | AFBC_FORMAT_MOD_SC)
+#define AFBC_SIZE_MASK AFBC_FORMAT_MOD_BLOCK_SIZE_MASK
+#define AFBC_SIZE_16X16 AFBC_FORMAT_MOD_BLOCK_SIZE_16x16
+#define AFBC_YTR AFBC_FORMAT_MOD_YTR
+#define AFBC_SPARSE AFBC_FORMAT_MOD_SPARSE
+#define AFBC_CBR AFBC_FORMAT_MOD_CBR
+#define AFBC_SPLIT AFBC_FORMAT_MOD_SPLIT
+#define AFBC_TILED AFBC_FORMAT_MOD_TILED
+#define AFBC_SC AFBC_FORMAT_MOD_SC
+
+#define AFBC_MOD_VALID_BITS (AFBC_SIZE_MASK | AFBC_YTR | AFBC_SPLIT | \
+ AFBC_SPARSE | AFBC_CBR | AFBC_TILED | AFBC_SC)
+
+extern const u64 malidp_format_modifiers[];
#endif /* __MALIDP_HW_H__ */
return -EINVAL;
}
+ if (fb->modifier) {
+ DRM_DEBUG_KMS("Writeback framebuffer does not support modifiers\n");
+ return -EINVAL;
+ }
+
mw_state->format =
malidp_hw_get_format_id(&malidp->dev->hw->map, SE_MEMWRITE,
- fb->format->format);
+ fb->format->format, !!fb->modifier);
if (mw_state->format == MALIDP_INVALID_FORMAT_ID) {
struct drm_format_name_buf format_name;
&mw_state->addrs[0],
mw_state->format);
- drm_writeback_queue_job(mw_conn, conn_state->writeback_job);
- conn_state->writeback_job = NULL;
+ drm_writeback_queue_job(mw_conn, conn_state);
hwdev->hw->enable_memwrite(hwdev, mw_state->addrs,
mw_state->pitches, mw_state->n_planes,
fb->width, fb->height, mw_state->format,
#define MALIDP550_LS_ENABLE 0x01c
#define MALIDP550_LS_R1_IN_SIZE 0x020
+#define MODIFIERS_COUNT_MAX 15
+
/*
* This 4-entry look-up-table is used to determine the full 8-bit alpha value
* for formats with 1- or 2-bit alpha channels.
drm_printf(p, "\tmmu_prefetch_pgsize=%d\n", ms->mmu_prefetch_pgsize);
}
+bool malidp_format_mod_supported(struct drm_device *drm,
+ u32 format, u64 modifier)
+{
+ const struct drm_format_info *info;
+ const u64 *modifiers;
+ struct malidp_drm *malidp = drm->dev_private;
+ const struct malidp_hw_regmap *map = &malidp->dev->hw->map;
+
+ if (WARN_ON(modifier == DRM_FORMAT_MOD_INVALID))
+ return false;
+
+ /* Some pixel formats are supported without any modifier */
+ if (modifier == DRM_FORMAT_MOD_LINEAR) {
+ /*
+ * However these pixel formats need to be supported with
+ * modifiers only
+ */
+ return !malidp_hw_format_is_afbc_only(format);
+ }
+
+ if ((modifier >> 56) != DRM_FORMAT_MOD_VENDOR_ARM) {
+ DRM_ERROR("Unknown modifier (not Arm)\n");
+ return false;
+ }
+
+ if (modifier &
+ ~DRM_FORMAT_MOD_ARM_AFBC(AFBC_MOD_VALID_BITS)) {
+ DRM_DEBUG_KMS("Unsupported modifiers\n");
+ return false;
+ }
+
+ modifiers = malidp_format_modifiers;
+
+ /* SPLIT buffers must use SPARSE layout */
+ if (WARN_ON_ONCE((modifier & AFBC_SPLIT) && !(modifier & AFBC_SPARSE)))
+ return false;
+
+ /* CBR only applies to YUV formats, where YTR should be always 0 */
+ if (WARN_ON_ONCE((modifier & AFBC_CBR) && (modifier & AFBC_YTR)))
+ return false;
+
+ while (*modifiers != DRM_FORMAT_MOD_INVALID) {
+ if (*modifiers == modifier)
+ break;
+
+ modifiers++;
+ }
+
+ /* return false, if the modifier was not found */
+ if (*modifiers == DRM_FORMAT_MOD_INVALID) {
+ DRM_DEBUG_KMS("Unsupported modifier\n");
+ return false;
+ }
+
+ info = drm_format_info(format);
+
+ if (info->num_planes != 1) {
+ DRM_DEBUG_KMS("AFBC buffers expect one plane\n");
+ return false;
+ }
+
+ if (malidp_hw_format_is_linear_only(format) == true) {
+ DRM_DEBUG_KMS("Given format (0x%x) is supported is linear mode only\n",
+ format);
+ return false;
+ }
+
+ /*
+ * RGB formats need to provide YTR modifier and YUV formats should not
+ * provide YTR modifier.
+ */
+ if (!(info->is_yuv) != !!(modifier & AFBC_FORMAT_MOD_YTR)) {
+ DRM_DEBUG_KMS("AFBC_FORMAT_MOD_YTR is %s for %s formats\n",
+ info->is_yuv ? "disallowed" : "mandatory",
+ info->is_yuv ? "YUV" : "RGB");
+ return false;
+ }
+
+ if (modifier & AFBC_SPLIT) {
+ if (!info->is_yuv) {
+ if (drm_format_plane_cpp(format, 0) <= 2) {
+ DRM_DEBUG_KMS("RGB formats <= 16bpp are not supported with SPLIT\n");
+ return false;
+ }
+ }
+
+ if ((drm_format_horz_chroma_subsampling(format) != 1) ||
+ (drm_format_vert_chroma_subsampling(format) != 1)) {
+ if (!(format == DRM_FORMAT_YUV420_10BIT &&
+ (map->features & MALIDP_DEVICE_AFBC_YUV_420_10_SUPPORT_SPLIT))) {
+ DRM_DEBUG_KMS("Formats which are sub-sampled should never be split\n");
+ return false;
+ }
+ }
+ }
+
+ if (modifier & AFBC_CBR) {
+ if ((drm_format_horz_chroma_subsampling(format) == 1) ||
+ (drm_format_vert_chroma_subsampling(format) == 1)) {
+ DRM_DEBUG_KMS("Formats which are not sub-sampled should not have CBR set\n");
+ return false;
+ }
+ }
+
+ return true;
+}
+
+static bool malidp_format_mod_supported_per_plane(struct drm_plane *plane,
+ u32 format, u64 modifier)
+{
+ return malidp_format_mod_supported(plane->dev, format, modifier);
+}
+
static const struct drm_plane_funcs malidp_de_plane_funcs = {
.update_plane = drm_atomic_helper_update_plane,
.disable_plane = drm_atomic_helper_disable_plane,
.atomic_duplicate_state = malidp_duplicate_plane_state,
.atomic_destroy_state = malidp_destroy_plane_state,
.atomic_print_state = malidp_plane_atomic_print_state,
+ .format_mod_supported = malidp_format_mod_supported_per_plane,
};
static int malidp_se_check_scaling(struct malidp_plane *mp,
fb = state->fb;
ms->format = malidp_hw_get_format_id(&mp->hwdev->hw->map,
- mp->layer->id,
- fb->format->format);
+ mp->layer->id, fb->format->format,
+ !!fb->modifier);
if (ms->format == MALIDP_INVALID_FORMAT_ID)
return -EINVAL;
for (i = 0; i < ms->n_planes; i++) {
u8 alignment = malidp_hw_get_pitch_align(mp->hwdev, rotated);
- if ((fb->pitches[i] * drm_format_info_block_height(fb->format, i))
- & (alignment - 1)) {
+ if (((fb->pitches[i] * drm_format_info_block_height(fb->format, i))
+ & (alignment - 1)) && !(fb->modifier)) {
DRM_DEBUG_KMS("Invalid pitch %u for plane %d\n",
fb->pitches[i], i);
return -EINVAL;
return -EINVAL;
}
+ /* SMART layer does not support AFBC */
+ if (mp->layer->id == DE_SMART && fb->modifier) {
+ DRM_ERROR("AFBC framebuffer not supported in SMART layer");
+ return -EINVAL;
+ }
+
ms->rotmem_size = 0;
if (state->rotation & MALIDP_ROTATED_MASK) {
int val;
val = mp->hwdev->hw->rotmem_required(mp->hwdev, state->crtc_w,
state->crtc_h,
- fb->format->format);
+ fb->format->format,
+ !!(fb->modifier));
if (val < 0)
return val;
mp->layer->base + mp->layer->mmu_ctrl_offset);
}
+static void malidp_set_plane_base_addr(struct drm_framebuffer *fb,
+ struct malidp_plane *mp,
+ int plane_index)
+{
+ dma_addr_t paddr;
+ u16 ptr;
+ struct drm_plane *plane = &mp->base;
+ bool afbc = fb->modifier ? true : false;
+
+ ptr = mp->layer->ptr + (plane_index << 4);
+
+ /*
+ * drm_fb_cma_get_gem_addr() alters the physical base address of the
+ * framebuffer as per the plane's src_x, src_y co-ordinates (ie to
+ * take care of source cropping).
+ * For AFBC, this is not needed as the cropping is handled by _AD_CROP_H
+ * and _AD_CROP_V registers.
+ */
+ if (!afbc) {
+ paddr = drm_fb_cma_get_gem_addr(fb, plane->state,
+ plane_index);
+ } else {
+ struct drm_gem_cma_object *obj;
+
+ obj = drm_fb_cma_get_gem_obj(fb, plane_index);
+
+ if (WARN_ON(!obj))
+ return;
+ paddr = obj->paddr;
+ }
+
+ malidp_hw_write(mp->hwdev, lower_32_bits(paddr), ptr);
+ malidp_hw_write(mp->hwdev, upper_32_bits(paddr), ptr + 4);
+}
+
+static void malidp_de_set_plane_afbc(struct drm_plane *plane)
+{
+ struct malidp_plane *mp;
+ u32 src_w, src_h, val = 0, src_x, src_y;
+ struct drm_framebuffer *fb = plane->state->fb;
+
+ mp = to_malidp_plane(plane);
+
+ /* no afbc_decoder_offset means AFBC is not supported on this plane */
+ if (!mp->layer->afbc_decoder_offset)
+ return;
+
+ if (!fb->modifier) {
+ malidp_hw_write(mp->hwdev, 0, mp->layer->afbc_decoder_offset);
+ return;
+ }
+
+ /* convert src values from Q16 fixed point to integer */
+ src_w = plane->state->src_w >> 16;
+ src_h = plane->state->src_h >> 16;
+ src_x = plane->state->src_x >> 16;
+ src_y = plane->state->src_y >> 16;
+
+ val = ((fb->width - (src_x + src_w)) << MALIDP_AD_CROP_RIGHT_OFFSET) |
+ src_x;
+ malidp_hw_write(mp->hwdev, val,
+ mp->layer->afbc_decoder_offset + MALIDP_AD_CROP_H);
+
+ val = ((fb->height - (src_y + src_h)) << MALIDP_AD_CROP_BOTTOM_OFFSET) |
+ src_y;
+ malidp_hw_write(mp->hwdev, val,
+ mp->layer->afbc_decoder_offset + MALIDP_AD_CROP_V);
+
+ val = MALIDP_AD_EN;
+ if (fb->modifier & AFBC_FORMAT_MOD_SPLIT)
+ val |= MALIDP_AD_BS;
+ if (fb->modifier & AFBC_FORMAT_MOD_YTR)
+ val |= MALIDP_AD_YTR;
+
+ malidp_hw_write(mp->hwdev, val, mp->layer->afbc_decoder_offset);
+}
+
static void malidp_de_plane_update(struct drm_plane *plane,
struct drm_plane_state *old_state)
{
u8 plane_alpha = state->alpha >> 8;
u32 src_w, src_h, dest_w, dest_h, val;
int i;
+ struct drm_framebuffer *fb = plane->state->fb;
mp = to_malidp_plane(plane);
- /* convert src values from Q16 fixed point to integer */
- src_w = state->src_w >> 16;
- src_h = state->src_h >> 16;
+ /*
+ * For AFBC framebuffer, use the framebuffer width and height for
+ * configuring layer input size register.
+ */
+ if (fb->modifier) {
+ src_w = fb->width;
+ src_h = fb->height;
+ } else {
+ /* convert src values from Q16 fixed point to integer */
+ src_w = state->src_w >> 16;
+ src_h = state->src_h >> 16;
+ }
+
dest_w = state->crtc_w;
dest_h = state->crtc_h;
val = (val & ~LAYER_FORMAT_MASK) | ms->format;
malidp_hw_write(mp->hwdev, val, mp->layer->base);
- for (i = 0; i < ms->n_planes; i++) {
- /* calculate the offset for the layer's plane registers */
- u16 ptr = mp->layer->ptr + (i << 4);
- dma_addr_t fb_addr = drm_fb_cma_get_gem_addr(state->fb,
- state, i);
-
- malidp_hw_write(mp->hwdev, lower_32_bits(fb_addr), ptr);
- malidp_hw_write(mp->hwdev, upper_32_bits(fb_addr), ptr + 4);
- }
+ for (i = 0; i < ms->n_planes; i++)
+ malidp_set_plane_base_addr(fb, mp, i);
malidp_de_set_mmu_control(mp, ms);
mp->layer->base + MALIDP550_LS_R1_IN_SIZE);
}
+ malidp_de_set_plane_afbc(plane);
+
/* first clear the rotation bits */
val = malidp_hw_read(mp->hwdev, mp->layer->base + MALIDP_LAYER_CONTROL);
val &= ~LAYER_ROT_MASK;
BIT(DRM_MODE_BLEND_PREMULTI) |
BIT(DRM_MODE_BLEND_COVERAGE);
u32 *formats;
- int ret, i, j, n;
+ int ret, i = 0, j = 0, n;
+ u64 supported_modifiers[MODIFIERS_COUNT_MAX];
+ const u64 *modifiers;
+
+ modifiers = malidp_format_modifiers;
+
+ if (!(map->features & MALIDP_DEVICE_AFBC_SUPPORT_SPLIT)) {
+ /*
+ * Since our hardware does not support SPLIT, so build the list
+ * of supported modifiers excluding SPLIT ones.
+ */
+ while (*modifiers != DRM_FORMAT_MOD_INVALID) {
+ if (!(*modifiers & AFBC_SPLIT))
+ supported_modifiers[j++] = *modifiers;
+
+ modifiers++;
+ }
+ supported_modifiers[j++] = DRM_FORMAT_MOD_INVALID;
+ modifiers = supported_modifiers;
+ }
formats = kcalloc(map->n_pixel_formats, sizeof(*formats), GFP_KERNEL);
if (!formats) {
plane_type = (i == 0) ? DRM_PLANE_TYPE_PRIMARY :
DRM_PLANE_TYPE_OVERLAY;
+
+ /*
+ * All the layers except smart layer supports AFBC modifiers.
+ */
ret = drm_universal_plane_init(drm, &plane->base, crtcs,
- &malidp_de_plane_funcs, formats,
- n, NULL, plane_type, NULL);
+ &malidp_de_plane_funcs, formats, n,
+ (id == DE_SMART) ? NULL : modifiers, plane_type,
+ NULL);
+
if (ret < 0)
goto cleanup;
#define MALIDP500_LV_YUV2RGB ((s16)(-0xB8))
#define MALIDP500_DE_LV_BASE 0x00100
#define MALIDP500_DE_LV_PTR_BASE 0x00124
+#define MALIDP500_DE_LV_AD_CTRL 0x00400
#define MALIDP500_DE_LG1_BASE 0x00200
#define MALIDP500_DE_LG1_PTR_BASE 0x0021c
+#define MALIDP500_DE_LG1_AD_CTRL 0x0040c
#define MALIDP500_DE_LG2_BASE 0x00300
#define MALIDP500_DE_LG2_PTR_BASE 0x0031c
+#define MALIDP500_DE_LG2_AD_CTRL 0x00418
#define MALIDP500_SE_BASE 0x00c00
#define MALIDP500_SE_CONTROL 0x00c0c
#define MALIDP500_SE_MEMWRITE_OUT_SIZE 0x00c2c
#define MALIDP550_LV_YUV2RGB 0x00084
#define MALIDP550_DE_LV1_BASE 0x00100
#define MALIDP550_DE_LV1_PTR_BASE 0x00124
+#define MALIDP550_DE_LV1_AD_CTRL 0x001B8
#define MALIDP550_DE_LV2_BASE 0x00200
#define MALIDP550_DE_LV2_PTR_BASE 0x00224
+#define MALIDP550_DE_LV2_AD_CTRL 0x002B8
#define MALIDP550_DE_LG_BASE 0x00300
#define MALIDP550_DE_LG_PTR_BASE 0x0031c
+#define MALIDP550_DE_LG_AD_CTRL 0x00330
#define MALIDP550_DE_LS_BASE 0x00400
#define MALIDP550_DE_LS_PTR_BASE 0x0042c
#define MALIDP550_DE_PERF_BASE 0x00500
#define MALIDP_MMU_CTRL_PX_PS(x) (1 << (8 + (x)))
#define MALIDP_MMU_CTRL_PP_NUM_REQ(x) (((x) & 0x7f) << 12)
+/* AFBC register offsets relative to MALIDPXXX_DE_LX_AD_CTRL */
+/* The following register offsets are common for DP500, DP550 and DP650 */
+#define MALIDP_AD_CROP_H 0x4
+#define MALIDP_AD_CROP_V 0x8
+#define MALIDP_AD_END_PTR_LOW 0xc
+#define MALIDP_AD_END_PTR_HIGH 0x10
+
+/* AFBC decoder Registers */
+#define MALIDP_AD_EN BIT(0)
+#define MALIDP_AD_YTR BIT(4)
+#define MALIDP_AD_BS BIT(8)
+#define MALIDP_AD_CROP_RIGHT_OFFSET 16
+#define MALIDP_AD_CROP_BOTTOM_OFFSET 16
+
/*
* Starting with DP550 the register map blocks has been standardised to the
* following layout:
goto err_fballoc;
}
- strlcpy(info->fix.id, "armada-drmfb", sizeof(info->fix.id));
- info->par = fbh;
info->fbops = &armada_fb_ops;
info->fix.smem_start = obj->phys_addr;
info->fix.smem_len = obj->obj.size;
info->screen_base = ptr;
fbh->fb = &dfb->fb;
- drm_fb_helper_fill_fix(info, dfb->fb.pitches[0],
- dfb->fb.format->depth);
- drm_fb_helper_fill_var(info, fbh, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(info, fbh, sizes);
DRM_DEBUG_KMS("allocated %dx%d %dbpp fb: 0x%08llx\n",
dfb->fb.width, dfb->fb.height, dfb->fb.format->cpp[0] * 8,
--- /dev/null
+config DRM_ASPEED_GFX
+ tristate "ASPEED BMC Display Controller"
+ depends on DRM && OF
+ depends on (COMPILE_TEST || ARCH_ASPEED)
+ select DRM_KMS_HELPER
+ select DRM_KMS_CMA_HELPER
+ select DMA_CMA if HAVE_DMA_CONTIGUOUS
+ select CMA if HAVE_DMA_CONTIGUOUS
+ select MFD_SYSCON
+ help
+ Chose this option if you have an ASPEED AST2500 SOC Display
+ Controller (aka GFX).
+
+ If M is selected this module will be called aspeed_gfx.
--- /dev/null
+aspeed_gfx-y := aspeed_gfx_drv.o aspeed_gfx_crtc.o aspeed_gfx_out.o
+
+obj-$(CONFIG_DRM_ASPEED_GFX) += aspeed_gfx.o
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ */
+/* Copyright 2018 IBM Corporation */
+
+#include <drm/drm_device.h>
+#include <drm/drm_simple_kms_helper.h>
+
+struct aspeed_gfx {
+ void __iomem *base;
+ struct clk *clk;
+ struct reset_control *rst;
+ struct regmap *scu;
+
+ struct drm_simple_display_pipe pipe;
+ struct drm_connector connector;
+ struct drm_fbdev_cma *fbdev;
+};
+
+int aspeed_gfx_create_pipe(struct drm_device *drm);
+int aspeed_gfx_create_output(struct drm_device *drm);
+
+#define CRT_CTRL1 0x60 /* CRT Control I */
+#define CRT_CTRL2 0x64 /* CRT Control II */
+#define CRT_STATUS 0x68 /* CRT Status */
+#define CRT_MISC 0x6c /* CRT Misc Setting */
+#define CRT_HORIZ0 0x70 /* CRT Horizontal Total & Display Enable End */
+#define CRT_HORIZ1 0x74 /* CRT Horizontal Retrace Start & End */
+#define CRT_VERT0 0x78 /* CRT Vertical Total & Display Enable End */
+#define CRT_VERT1 0x7C /* CRT Vertical Retrace Start & End */
+#define CRT_ADDR 0x80 /* CRT Display Starting Address */
+#define CRT_OFFSET 0x84 /* CRT Display Offset & Terminal Count */
+#define CRT_THROD 0x88 /* CRT Threshold */
+#define CRT_XSCALE 0x8C /* CRT Scaling-Up Factor */
+#define CRT_CURSOR0 0x90 /* CRT Hardware Cursor X & Y Offset */
+#define CRT_CURSOR1 0x94 /* CRT Hardware Cursor X & Y Position */
+#define CRT_CURSOR2 0x98 /* CRT Hardware Cursor Pattern Address */
+#define CRT_9C 0x9C
+#define CRT_OSD_H 0xA0 /* CRT OSD Horizontal Start/End */
+#define CRT_OSD_V 0xA4 /* CRT OSD Vertical Start/End */
+#define CRT_OSD_ADDR 0xA8 /* CRT OSD Pattern Address */
+#define CRT_OSD_DISP 0xAC /* CRT OSD Offset */
+#define CRT_OSD_THRESH 0xB0 /* CRT OSD Threshold & Alpha */
+#define CRT_B4 0xB4
+#define CRT_STS_V 0xB8 /* CRT Status V */
+#define CRT_SCRATCH 0xBC /* Scratchpad */
+#define CRT_BB0_ADDR 0xD0 /* CRT Display BB0 Starting Address */
+#define CRT_BB1_ADDR 0xD4 /* CRT Display BB1 Starting Address */
+#define CRT_BB_COUNT 0xD8 /* CRT Display BB Terminal Count */
+#define OSD_COLOR1 0xE0 /* OSD Color Palette Index 1 & 0 */
+#define OSD_COLOR2 0xE4 /* OSD Color Palette Index 3 & 2 */
+#define OSD_COLOR3 0xE8 /* OSD Color Palette Index 5 & 4 */
+#define OSD_COLOR4 0xEC /* OSD Color Palette Index 7 & 6 */
+#define OSD_COLOR5 0xF0 /* OSD Color Palette Index 9 & 8 */
+#define OSD_COLOR6 0xF4 /* OSD Color Palette Index 11 & 10 */
+#define OSD_COLOR7 0xF8 /* OSD Color Palette Index 13 & 12 */
+#define OSD_COLOR8 0xFC /* OSD Color Palette Index 15 & 14 */
+
+/* CTRL1 */
+#define CRT_CTRL_EN BIT(0)
+#define CRT_CTRL_HW_CURSOR_EN BIT(1)
+#define CRT_CTRL_OSD_EN BIT(2)
+#define CRT_CTRL_INTERLACED BIT(3)
+#define CRT_CTRL_COLOR_RGB565 (0 << 7)
+#define CRT_CTRL_COLOR_YUV444 (1 << 7)
+#define CRT_CTRL_COLOR_XRGB8888 (2 << 7)
+#define CRT_CTRL_COLOR_RGB888 (3 << 7)
+#define CRT_CTRL_COLOR_YUV444_2RGB (5 << 7)
+#define CRT_CTRL_COLOR_YUV422 (7 << 7)
+#define CRT_CTRL_COLOR_MASK GENMASK(9, 7)
+#define CRT_CTRL_HSYNC_NEGATIVE BIT(16)
+#define CRT_CTRL_VSYNC_NEGATIVE BIT(17)
+#define CRT_CTRL_VERTICAL_INTR_EN BIT(30)
+#define CRT_CTRL_VERTICAL_INTR_STS BIT(31)
+
+/* CTRL2 */
+#define CRT_CTRL_DAC_EN BIT(0)
+#define CRT_CTRL_VBLANK_LINE(x) (((x) << 20) & CRT_CTRL_VBLANK_LINE_MASK)
+#define CRT_CTRL_VBLANK_LINE_MASK GENMASK(20, 31)
+
+/* CRT_HORIZ0 */
+#define CRT_H_TOTAL(x) (x)
+#define CRT_H_DE(x) ((x) << 16)
+
+/* CRT_HORIZ1 */
+#define CRT_H_RS_START(x) (x)
+#define CRT_H_RS_END(x) ((x) << 16)
+
+/* CRT_VIRT0 */
+#define CRT_V_TOTAL(x) (x)
+#define CRT_V_DE(x) ((x) << 16)
+
+/* CRT_VIRT1 */
+#define CRT_V_RS_START(x) (x)
+#define CRT_V_RS_END(x) ((x) << 16)
+
+/* CRT_OFFSET */
+#define CRT_DISP_OFFSET(x) (x)
+#define CRT_TERM_COUNT(x) ((x) << 16)
+
+/* CRT_THROD */
+#define CRT_THROD_LOW(x) (x)
+#define CRT_THROD_HIGH(x) ((x) << 8)
+
+/* Default Threshold Seting */
+#define G5_CRT_THROD_VAL (CRT_THROD_LOW(0x24) | CRT_THROD_HIGH(0x3C))
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+// Copyright 2018 IBM Corporation
+
+#include <linux/clk.h>
+#include <linux/reset.h>
+#include <linux/regmap.h>
+
+#include <drm/drm_crtc_helper.h>
+#include <drm/drm_device.h>
+#include <drm/drm_fb_cma_helper.h>
+#include <drm/drm_fourcc.h>
+#include <drm/drm_gem_cma_helper.h>
+#include <drm/drm_gem_framebuffer_helper.h>
+#include <drm/drm_panel.h>
+#include <drm/drm_simple_kms_helper.h>
+#include <drm/drm_vblank.h>
+
+#include "aspeed_gfx.h"
+
+static struct aspeed_gfx *
+drm_pipe_to_aspeed_gfx(struct drm_simple_display_pipe *pipe)
+{
+ return container_of(pipe, struct aspeed_gfx, pipe);
+}
+
+static int aspeed_gfx_set_pixel_fmt(struct aspeed_gfx *priv, u32 *bpp)
+{
+ struct drm_crtc *crtc = &priv->pipe.crtc;
+ struct drm_device *drm = crtc->dev;
+ const u32 format = crtc->primary->state->fb->format->format;
+ u32 ctrl1;
+
+ ctrl1 = readl(priv->base + CRT_CTRL1);
+ ctrl1 &= ~CRT_CTRL_COLOR_MASK;
+
+ switch (format) {
+ case DRM_FORMAT_RGB565:
+ dev_dbg(drm->dev, "Setting up RGB565 mode\n");
+ ctrl1 |= CRT_CTRL_COLOR_RGB565;
+ *bpp = 16;
+ break;
+ case DRM_FORMAT_XRGB8888:
+ dev_dbg(drm->dev, "Setting up XRGB8888 mode\n");
+ ctrl1 |= CRT_CTRL_COLOR_XRGB8888;
+ *bpp = 32;
+ break;
+ default:
+ dev_err(drm->dev, "Unhandled pixel format %08x\n", format);
+ return -EINVAL;
+ }
+
+ writel(ctrl1, priv->base + CRT_CTRL1);
+
+ return 0;
+}
+
+static void aspeed_gfx_enable_controller(struct aspeed_gfx *priv)
+{
+ u32 ctrl1 = readl(priv->base + CRT_CTRL1);
+ u32 ctrl2 = readl(priv->base + CRT_CTRL2);
+
+ /* SCU2C: set DAC source for display output to Graphics CRT (GFX) */
+ regmap_update_bits(priv->scu, 0x2c, BIT(16), BIT(16));
+
+ writel(ctrl1 | CRT_CTRL_EN, priv->base + CRT_CTRL1);
+ writel(ctrl2 | CRT_CTRL_DAC_EN, priv->base + CRT_CTRL2);
+}
+
+static void aspeed_gfx_disable_controller(struct aspeed_gfx *priv)
+{
+ u32 ctrl1 = readl(priv->base + CRT_CTRL1);
+ u32 ctrl2 = readl(priv->base + CRT_CTRL2);
+
+ writel(ctrl1 & ~CRT_CTRL_EN, priv->base + CRT_CTRL1);
+ writel(ctrl2 & ~CRT_CTRL_DAC_EN, priv->base + CRT_CTRL2);
+
+ regmap_update_bits(priv->scu, 0x2c, BIT(16), 0);
+}
+
+static void aspeed_gfx_crtc_mode_set_nofb(struct aspeed_gfx *priv)
+{
+ struct drm_display_mode *m = &priv->pipe.crtc.state->adjusted_mode;
+ u32 ctrl1, d_offset, t_count, bpp;
+ int err;
+
+ err = aspeed_gfx_set_pixel_fmt(priv, &bpp);
+ if (err)
+ return;
+
+#if 0
+ /* TODO: we have only been able to test with the 40MHz USB clock. The
+ * clock is fixed, so we cannot adjust it here. */
+ clk_set_rate(priv->pixel_clk, m->crtc_clock * 1000);
+#endif
+
+ ctrl1 = readl(priv->base + CRT_CTRL1);
+ ctrl1 &= ~(CRT_CTRL_INTERLACED |
+ CRT_CTRL_HSYNC_NEGATIVE |
+ CRT_CTRL_VSYNC_NEGATIVE);
+
+ if (m->flags & DRM_MODE_FLAG_INTERLACE)
+ ctrl1 |= CRT_CTRL_INTERLACED;
+
+ if (!(m->flags & DRM_MODE_FLAG_PHSYNC))
+ ctrl1 |= CRT_CTRL_HSYNC_NEGATIVE;
+
+ if (!(m->flags & DRM_MODE_FLAG_PVSYNC))
+ ctrl1 |= CRT_CTRL_VSYNC_NEGATIVE;
+
+ writel(ctrl1, priv->base + CRT_CTRL1);
+
+ /* Horizontal timing */
+ writel(CRT_H_TOTAL(m->htotal - 1) | CRT_H_DE(m->hdisplay - 1),
+ priv->base + CRT_HORIZ0);
+ writel(CRT_H_RS_START(m->hsync_start - 1) | CRT_H_RS_END(m->hsync_end),
+ priv->base + CRT_HORIZ1);
+
+
+ /* Vertical timing */
+ writel(CRT_V_TOTAL(m->vtotal - 1) | CRT_V_DE(m->vdisplay - 1),
+ priv->base + CRT_VERT0);
+ writel(CRT_V_RS_START(m->vsync_start) | CRT_V_RS_END(m->vsync_end),
+ priv->base + CRT_VERT1);
+
+ /*
+ * Display Offset: address difference between consecutive scan lines
+ * Terminal Count: memory size of one scan line
+ */
+ d_offset = m->hdisplay * bpp / 8;
+ t_count = (m->hdisplay * bpp + 127) / 128;
+ writel(CRT_DISP_OFFSET(d_offset) | CRT_TERM_COUNT(t_count),
+ priv->base + CRT_OFFSET);
+
+ /*
+ * Threshold: FIFO thresholds of refill and stop (16 byte chunks
+ * per line, rounded up)
+ */
+ writel(G5_CRT_THROD_VAL, priv->base + CRT_THROD);
+}
+
+static void aspeed_gfx_pipe_enable(struct drm_simple_display_pipe *pipe,
+ struct drm_crtc_state *crtc_state,
+ struct drm_plane_state *plane_state)
+{
+ struct aspeed_gfx *priv = drm_pipe_to_aspeed_gfx(pipe);
+ struct drm_crtc *crtc = &pipe->crtc;
+
+ aspeed_gfx_crtc_mode_set_nofb(priv);
+ aspeed_gfx_enable_controller(priv);
+ drm_crtc_vblank_on(crtc);
+}
+
+static void aspeed_gfx_pipe_disable(struct drm_simple_display_pipe *pipe)
+{
+ struct aspeed_gfx *priv = drm_pipe_to_aspeed_gfx(pipe);
+ struct drm_crtc *crtc = &pipe->crtc;
+
+ drm_crtc_vblank_off(crtc);
+ aspeed_gfx_disable_controller(priv);
+}
+
+static void aspeed_gfx_pipe_update(struct drm_simple_display_pipe *pipe,
+ struct drm_plane_state *plane_state)
+{
+ struct aspeed_gfx *priv = drm_pipe_to_aspeed_gfx(pipe);
+ struct drm_crtc *crtc = &pipe->crtc;
+ struct drm_framebuffer *fb = pipe->plane.state->fb;
+ struct drm_pending_vblank_event *event;
+ struct drm_gem_cma_object *gem;
+
+ spin_lock_irq(&crtc->dev->event_lock);
+ event = crtc->state->event;
+ if (event) {
+ crtc->state->event = NULL;
+
+ if (drm_crtc_vblank_get(crtc) == 0)
+ drm_crtc_arm_vblank_event(crtc, event);
+ else
+ drm_crtc_send_vblank_event(crtc, event);
+ }
+ spin_unlock_irq(&crtc->dev->event_lock);
+
+ if (!fb)
+ return;
+
+ gem = drm_fb_cma_get_gem_obj(fb, 0);
+ if (!gem)
+ return;
+ writel(gem->paddr, priv->base + CRT_ADDR);
+}
+
+static int aspeed_gfx_enable_vblank(struct drm_simple_display_pipe *pipe)
+{
+ struct aspeed_gfx *priv = drm_pipe_to_aspeed_gfx(pipe);
+ u32 reg = readl(priv->base + CRT_CTRL1);
+
+ /* Clear pending VBLANK IRQ */
+ writel(reg | CRT_CTRL_VERTICAL_INTR_STS, priv->base + CRT_CTRL1);
+
+ reg |= CRT_CTRL_VERTICAL_INTR_EN;
+ writel(reg, priv->base + CRT_CTRL1);
+
+ return 0;
+}
+
+static void aspeed_gfx_disable_vblank(struct drm_simple_display_pipe *pipe)
+{
+ struct aspeed_gfx *priv = drm_pipe_to_aspeed_gfx(pipe);
+ u32 reg = readl(priv->base + CRT_CTRL1);
+
+ reg &= ~CRT_CTRL_VERTICAL_INTR_EN;
+ writel(reg, priv->base + CRT_CTRL1);
+
+ /* Clear pending VBLANK IRQ */
+ writel(reg | CRT_CTRL_VERTICAL_INTR_STS, priv->base + CRT_CTRL1);
+}
+
+static struct drm_simple_display_pipe_funcs aspeed_gfx_funcs = {
+ .enable = aspeed_gfx_pipe_enable,
+ .disable = aspeed_gfx_pipe_disable,
+ .update = aspeed_gfx_pipe_update,
+ .prepare_fb = drm_gem_fb_simple_display_pipe_prepare_fb,
+ .enable_vblank = aspeed_gfx_enable_vblank,
+ .disable_vblank = aspeed_gfx_disable_vblank,
+};
+
+static const uint32_t aspeed_gfx_formats[] = {
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_RGB565,
+};
+
+int aspeed_gfx_create_pipe(struct drm_device *drm)
+{
+ struct aspeed_gfx *priv = drm->dev_private;
+
+ return drm_simple_display_pipe_init(drm, &priv->pipe, &aspeed_gfx_funcs,
+ aspeed_gfx_formats,
+ ARRAY_SIZE(aspeed_gfx_formats),
+ NULL,
+ &priv->connector);
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+// Copyright 2018 IBM Corporation
+
+#include <linux/clk.h>
+#include <linux/dma-mapping.h>
+#include <linux/irq.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <linux/reset.h>
+
+#include <drm/drm_atomic_helper.h>
+#include <drm/drm_crtc_helper.h>
+#include <drm/drm_device.h>
+#include <drm/drm_fb_cma_helper.h>
+#include <drm/drm_fb_helper.h>
+#include <drm/drm_gem_cma_helper.h>
+#include <drm/drm_gem_framebuffer_helper.h>
+#include <drm/drm_probe_helper.h>
+#include <drm/drm_simple_kms_helper.h>
+#include <drm/drm_vblank.h>
+#include <drm/drm_drv.h>
+
+#include "aspeed_gfx.h"
+
+/**
+ * DOC: ASPEED GFX Driver
+ *
+ * This driver is for the ASPEED BMC SoC's 'GFX' display hardware, also called
+ * the 'SOC Display Controller' in the datasheet. This driver runs on the ARM
+ * based BMC systems, unlike the ast driver which runs on a host CPU and is for
+ * a PCIe graphics device.
+ *
+ * The AST2500 supports a total of 3 output paths:
+ *
+ * 1. VGA output, the output target can choose either or both to the DAC
+ * or DVO interface.
+ *
+ * 2. Graphics CRT output, the output target can choose either or both to
+ * the DAC or DVO interface.
+ *
+ * 3. Video input from DVO, the video input can be used for video engine
+ * capture or DAC display output.
+ *
+ * Output options are selected in SCU2C.
+ *
+ * The "VGA mode" device is the PCI attached controller. The "Graphics CRT"
+ * is the ARM's internal display controller.
+ *
+ * The driver only supports a simple configuration consisting of a 40MHz
+ * pixel clock, fixed by hardware limitations, and the VGA output path.
+ *
+ * The driver was written with the 'AST2500 Software Programming Guide' v17,
+ * which is available under NDA from ASPEED.
+ */
+
+static const struct drm_mode_config_funcs aspeed_gfx_mode_config_funcs = {
+ .fb_create = drm_gem_fb_create,
+ .atomic_check = drm_atomic_helper_check,
+ .atomic_commit = drm_atomic_helper_commit,
+};
+
+static void aspeed_gfx_setup_mode_config(struct drm_device *drm)
+{
+ drm_mode_config_init(drm);
+
+ drm->mode_config.min_width = 0;
+ drm->mode_config.min_height = 0;
+ drm->mode_config.max_width = 800;
+ drm->mode_config.max_height = 600;
+ drm->mode_config.funcs = &aspeed_gfx_mode_config_funcs;
+}
+
+static irqreturn_t aspeed_gfx_irq_handler(int irq, void *data)
+{
+ struct drm_device *drm = data;
+ struct aspeed_gfx *priv = drm->dev_private;
+ u32 reg;
+
+ reg = readl(priv->base + CRT_CTRL1);
+
+ if (reg & CRT_CTRL_VERTICAL_INTR_STS) {
+ drm_crtc_handle_vblank(&priv->pipe.crtc);
+ writel(reg, priv->base + CRT_CTRL1);
+ return IRQ_HANDLED;
+ }
+
+ return IRQ_NONE;
+}
+
+
+
+static int aspeed_gfx_load(struct drm_device *drm)
+{
+ struct platform_device *pdev = to_platform_device(drm->dev);
+ struct aspeed_gfx *priv;
+ struct resource *res;
+ int ret;
+
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+ drm->dev_private = priv;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ priv->base = devm_ioremap_resource(drm->dev, res);
+ if (IS_ERR(priv->base))
+ return PTR_ERR(priv->base);
+
+ priv->scu = syscon_regmap_lookup_by_compatible("aspeed,ast2500-scu");
+ if (IS_ERR(priv->scu)) {
+ dev_err(&pdev->dev, "failed to find SCU regmap\n");
+ return PTR_ERR(priv->scu);
+ }
+
+ ret = of_reserved_mem_device_init(drm->dev);
+ if (ret) {
+ dev_err(&pdev->dev,
+ "failed to initialize reserved mem: %d\n", ret);
+ return ret;
+ }
+
+ ret = dma_set_mask_and_coherent(drm->dev, DMA_BIT_MASK(32));
+ if (ret) {
+ dev_err(&pdev->dev, "failed to set DMA mask: %d\n", ret);
+ return ret;
+ }
+
+ priv->rst = devm_reset_control_get_exclusive(&pdev->dev, NULL);
+ if (IS_ERR(priv->rst)) {
+ dev_err(&pdev->dev,
+ "missing or invalid reset controller device tree entry");
+ return PTR_ERR(priv->rst);
+ }
+ reset_control_deassert(priv->rst);
+
+ priv->clk = devm_clk_get(drm->dev, NULL);
+ if (IS_ERR(priv->clk)) {
+ dev_err(&pdev->dev,
+ "missing or invalid clk device tree entry");
+ return PTR_ERR(priv->clk);
+ }
+ clk_prepare_enable(priv->clk);
+
+ /* Sanitize control registers */
+ writel(0, priv->base + CRT_CTRL1);
+ writel(0, priv->base + CRT_CTRL2);
+
+ aspeed_gfx_setup_mode_config(drm);
+
+ ret = drm_vblank_init(drm, 1);
+ if (ret < 0) {
+ dev_err(drm->dev, "Failed to initialise vblank\n");
+ return ret;
+ }
+
+ ret = aspeed_gfx_create_output(drm);
+ if (ret < 0) {
+ dev_err(drm->dev, "Failed to create outputs\n");
+ return ret;
+ }
+
+ ret = aspeed_gfx_create_pipe(drm);
+ if (ret < 0) {
+ dev_err(drm->dev, "Cannot setup simple display pipe\n");
+ return ret;
+ }
+
+ ret = devm_request_irq(drm->dev, platform_get_irq(pdev, 0),
+ aspeed_gfx_irq_handler, 0, "aspeed gfx", drm);
+ if (ret < 0) {
+ dev_err(drm->dev, "Failed to install IRQ handler\n");
+ return ret;
+ }
+
+ drm_mode_config_reset(drm);
+
+ drm_fbdev_generic_setup(drm, 32);
+
+ return 0;
+}
+
+static void aspeed_gfx_unload(struct drm_device *drm)
+{
+ drm_kms_helper_poll_fini(drm);
+ drm_mode_config_cleanup(drm);
+
+ drm->dev_private = NULL;
+}
+
+DEFINE_DRM_GEM_CMA_FOPS(fops);
+
+static struct drm_driver aspeed_gfx_driver = {
+ .driver_features = DRIVER_GEM | DRIVER_MODESET |
+ DRIVER_PRIME | DRIVER_ATOMIC,
+ .gem_create_object = drm_cma_gem_create_object_default_funcs,
+ .dumb_create = drm_gem_cma_dumb_create,
+ .prime_handle_to_fd = drm_gem_prime_handle_to_fd,
+ .prime_fd_to_handle = drm_gem_prime_fd_to_handle,
+ .gem_prime_import_sg_table = drm_gem_cma_prime_import_sg_table,
+ .gem_prime_mmap = drm_gem_prime_mmap,
+ .fops = &fops,
+ .name = "aspeed-gfx-drm",
+ .desc = "ASPEED GFX DRM",
+ .date = "20180319",
+ .major = 1,
+ .minor = 0,
+};
+
+static const struct of_device_id aspeed_gfx_match[] = {
+ { .compatible = "aspeed,ast2500-gfx" },
+ { }
+};
+
+static int aspeed_gfx_probe(struct platform_device *pdev)
+{
+ struct drm_device *drm;
+ int ret;
+
+ drm = drm_dev_alloc(&aspeed_gfx_driver, &pdev->dev);
+ if (IS_ERR(drm))
+ return PTR_ERR(drm);
+
+ ret = aspeed_gfx_load(drm);
+ if (ret)
+ goto err_free;
+
+ ret = drm_dev_register(drm, 0);
+ if (ret)
+ goto err_unload;
+
+ return 0;
+
+err_unload:
+ aspeed_gfx_unload(drm);
+err_free:
+ drm_dev_put(drm);
+
+ return ret;
+}
+
+static int aspeed_gfx_remove(struct platform_device *pdev)
+{
+ struct drm_device *drm = platform_get_drvdata(pdev);
+
+ drm_dev_unregister(drm);
+ aspeed_gfx_unload(drm);
+ drm_dev_put(drm);
+
+ return 0;
+}
+
+static struct platform_driver aspeed_gfx_platform_driver = {
+ .probe = aspeed_gfx_probe,
+ .remove = aspeed_gfx_remove,
+ .driver = {
+ .name = "aspeed_gfx",
+ .of_match_table = aspeed_gfx_match,
+ },
+};
+
+module_platform_driver(aspeed_gfx_platform_driver);
+
+MODULE_AUTHOR("Joel Stanley <joel@jms.id.au>");
+MODULE_DESCRIPTION("ASPEED BMC DRM/KMS driver");
+MODULE_LICENSE("GPL");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+// Copyright 2018 IBM Corporation
+
+#include <drm/drm_atomic_helper.h>
+#include <drm/drm_connector.h>
+#include <drm/drm_crtc_helper.h>
+#include <drm/drm_probe_helper.h>
+
+#include "aspeed_gfx.h"
+
+static int aspeed_gfx_get_modes(struct drm_connector *connector)
+{
+ return drm_add_modes_noedid(connector, 800, 600);
+}
+
+static const struct
+drm_connector_helper_funcs aspeed_gfx_connector_helper_funcs = {
+ .get_modes = aspeed_gfx_get_modes,
+};
+
+static const struct drm_connector_funcs aspeed_gfx_connector_funcs = {
+ .fill_modes = drm_helper_probe_single_connector_modes,
+ .destroy = drm_connector_cleanup,
+ .reset = drm_atomic_helper_connector_reset,
+ .atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state,
+ .atomic_destroy_state = drm_atomic_helper_connector_destroy_state,
+};
+
+int aspeed_gfx_create_output(struct drm_device *drm)
+{
+ struct aspeed_gfx *priv = drm->dev_private;
+ int ret;
+
+ priv->connector.dpms = DRM_MODE_DPMS_OFF;
+ priv->connector.polled = 0;
+ drm_connector_helper_add(&priv->connector,
+ &aspeed_gfx_connector_helper_funcs);
+ ret = drm_connector_init(drm, &priv->connector,
+ &aspeed_gfx_connector_funcs,
+ DRM_MODE_CONNECTOR_Unknown);
+ return ret;
+}
};
struct ast_fbdev {
- struct drm_fb_helper helper;
+ struct drm_fb_helper helper; /* must be first */
struct ast_framebuffer afb;
void *sysram;
int size;
uint32_t handle,
uint64_t *offset);
-#define DRM_FILE_PAGE_OFFSET (0x100000000ULL >> PAGE_SHIFT)
-
int ast_mm_init(struct ast_private *ast);
void ast_mm_fini(struct ast_private *ast);
ret = PTR_ERR(info);
goto out;
}
- info->par = afbdev;
-
ret = ast_framebuffer_init(dev, &afbdev->afb, &mode_cmd, gobj);
if (ret)
goto out;
fb = &afbdev->afb.base;
afbdev->helper.fb = fb;
- strcpy(info->fix.id, "astdrmfb");
-
info->fbops = &astfb_ops;
info->apertures->ranges[0].base = pci_resource_start(dev->pdev, 0);
info->apertures->ranges[0].size = pci_resource_len(dev->pdev, 0);
- drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(info, &afbdev->helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(info, &afbdev->helper, sizes);
info->screen_base = sysram;
info->screen_size = size;
ret = ttm_bo_device_init(&ast->ttm.bdev,
&ast_bo_driver,
dev->anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET,
true);
if (ret) {
DRM_ERROR("Error initialising bo driver; %d\n", ret);
int ast_mmap(struct file *filp, struct vm_area_struct *vma)
{
- struct drm_file *file_priv;
- struct ast_private *ast;
+ struct drm_file *file_priv = filp->private_data;
+ struct ast_private *ast = file_priv->minor->dev->dev_private;
- if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET))
- return -EINVAL;
-
- file_priv = filp->private_data;
- ast = file_priv->minor->dev->dev_private;
return ttm_bo_mmap(filp, vma, &ast->ttm.bdev);
}
#include <drm/drm_crtc_helper.h>
#include <drm/drm_encoder.h>
#include <drm/drm_fb_helper.h>
+#include <drm/drm_simple_kms_helper.h>
#include <drm/drm_gem.h>
struct edid *edid;
/* drm */
- struct drm_device *dev;
- struct drm_crtc crtc;
- struct drm_encoder encoder;
+ struct drm_device *dev;
+ struct drm_simple_display_pipe pipe;
struct drm_connector connector;
- bool mode_config_initialized;
/* ttm */
struct {
return container_of(gem, struct bochs_bo, gem);
}
-#define DRM_FILE_PAGE_OFFSET (0x100000000ULL >> PAGE_SHIFT)
-
static inline u64 bochs_bo_mmap_offset(struct bochs_bo *bo)
{
return drm_vma_node_offset_addr(&bo->bo.vma_node);
/* ---------------------------------------------------------------------- */
-static void bochs_crtc_mode_set_nofb(struct drm_crtc *crtc)
+static const uint32_t bochs_formats[] = {
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_BGRX8888,
+};
+
+static void bochs_plane_update(struct bochs_device *bochs,
+ struct drm_plane_state *state)
{
- struct bochs_device *bochs =
- container_of(crtc, struct bochs_device, crtc);
+ struct bochs_bo *bo;
- bochs_hw_setmode(bochs, &crtc->mode);
+ if (!state->fb || !bochs->stride)
+ return;
+
+ bo = gem_to_bochs_bo(state->fb->obj[0]);
+ bochs_hw_setbase(bochs,
+ state->crtc_x,
+ state->crtc_y,
+ bo->bo.offset);
+ bochs_hw_setformat(bochs, state->fb->format);
}
-static void bochs_crtc_atomic_enable(struct drm_crtc *crtc,
- struct drm_crtc_state *old_crtc_state)
+static void bochs_pipe_enable(struct drm_simple_display_pipe *pipe,
+ struct drm_crtc_state *crtc_state,
+ struct drm_plane_state *plane_state)
{
+ struct bochs_device *bochs = pipe->crtc.dev->dev_private;
+
+ bochs_hw_setmode(bochs, &crtc_state->mode);
+ bochs_plane_update(bochs, plane_state);
}
-static void bochs_crtc_atomic_flush(struct drm_crtc *crtc,
- struct drm_crtc_state *old_crtc_state)
+static void bochs_pipe_update(struct drm_simple_display_pipe *pipe,
+ struct drm_plane_state *old_state)
{
- struct drm_device *dev = crtc->dev;
- struct drm_pending_vblank_event *event;
+ struct bochs_device *bochs = pipe->crtc.dev->dev_private;
+ struct drm_crtc *crtc = &pipe->crtc;
- if (crtc->state && crtc->state->event) {
- unsigned long irqflags;
+ bochs_plane_update(bochs, pipe->plane.state);
- spin_lock_irqsave(&dev->event_lock, irqflags);
- event = crtc->state->event;
+ if (crtc->state->event) {
+ spin_lock_irq(&crtc->dev->event_lock);
+ drm_crtc_send_vblank_event(crtc, crtc->state->event);
crtc->state->event = NULL;
- drm_crtc_send_vblank_event(crtc, event);
- spin_unlock_irqrestore(&dev->event_lock, irqflags);
+ spin_unlock_irq(&crtc->dev->event_lock);
}
}
-
-/* These provide the minimum set of functions required to handle a CRTC */
-static const struct drm_crtc_funcs bochs_crtc_funcs = {
- .set_config = drm_atomic_helper_set_config,
- .destroy = drm_crtc_cleanup,
- .page_flip = drm_atomic_helper_page_flip,
- .reset = drm_atomic_helper_crtc_reset,
- .atomic_duplicate_state = drm_atomic_helper_crtc_duplicate_state,
- .atomic_destroy_state = drm_atomic_helper_crtc_destroy_state,
-};
-
-static const struct drm_crtc_helper_funcs bochs_helper_funcs = {
- .mode_set_nofb = bochs_crtc_mode_set_nofb,
- .atomic_enable = bochs_crtc_atomic_enable,
- .atomic_flush = bochs_crtc_atomic_flush,
-};
-
-static const uint32_t bochs_formats[] = {
- DRM_FORMAT_XRGB8888,
- DRM_FORMAT_BGRX8888,
-};
-
-static void bochs_plane_atomic_update(struct drm_plane *plane,
- struct drm_plane_state *old_state)
-{
- struct bochs_device *bochs = plane->dev->dev_private;
- struct bochs_bo *bo;
-
- if (!plane->state->fb)
- return;
- bo = gem_to_bochs_bo(plane->state->fb->obj[0]);
- bochs_hw_setbase(bochs,
- plane->state->crtc_x,
- plane->state->crtc_y,
- bo->bo.offset);
- bochs_hw_setformat(bochs, plane->state->fb->format);
-}
-
-static int bochs_plane_prepare_fb(struct drm_plane *plane,
- struct drm_plane_state *new_state)
+static int bochs_pipe_prepare_fb(struct drm_simple_display_pipe *pipe,
+ struct drm_plane_state *new_state)
{
struct bochs_bo *bo;
return bochs_bo_pin(bo, TTM_PL_FLAG_VRAM);
}
-static void bochs_plane_cleanup_fb(struct drm_plane *plane,
- struct drm_plane_state *old_state)
+static void bochs_pipe_cleanup_fb(struct drm_simple_display_pipe *pipe,
+ struct drm_plane_state *old_state)
{
struct bochs_bo *bo;
bochs_bo_unpin(bo);
}
-static const struct drm_plane_helper_funcs bochs_plane_helper_funcs = {
- .atomic_update = bochs_plane_atomic_update,
- .prepare_fb = bochs_plane_prepare_fb,
- .cleanup_fb = bochs_plane_cleanup_fb,
-};
-
-static const struct drm_plane_funcs bochs_plane_funcs = {
- .update_plane = drm_atomic_helper_update_plane,
- .disable_plane = drm_atomic_helper_disable_plane,
- .destroy = drm_primary_helper_destroy,
- .reset = drm_atomic_helper_plane_reset,
- .atomic_duplicate_state = drm_atomic_helper_plane_duplicate_state,
- .atomic_destroy_state = drm_atomic_helper_plane_destroy_state,
-};
-
-static struct drm_plane *bochs_primary_plane(struct drm_device *dev)
-{
- struct drm_plane *primary;
- int ret;
-
- primary = kzalloc(sizeof(*primary), GFP_KERNEL);
- if (primary == NULL) {
- DRM_DEBUG_KMS("Failed to allocate primary plane\n");
- return NULL;
- }
-
- ret = drm_universal_plane_init(dev, primary, 0,
- &bochs_plane_funcs,
- bochs_formats,
- ARRAY_SIZE(bochs_formats),
- NULL,
- DRM_PLANE_TYPE_PRIMARY, NULL);
- if (ret) {
- kfree(primary);
- return NULL;
- }
-
- drm_plane_helper_add(primary, &bochs_plane_helper_funcs);
- return primary;
-}
-
-static void bochs_crtc_init(struct drm_device *dev)
-{
- struct bochs_device *bochs = dev->dev_private;
- struct drm_crtc *crtc = &bochs->crtc;
- struct drm_plane *primary = bochs_primary_plane(dev);
-
- drm_crtc_init_with_planes(dev, crtc, primary, NULL,
- &bochs_crtc_funcs, NULL);
- drm_crtc_helper_add(crtc, &bochs_helper_funcs);
-}
-
-static const struct drm_encoder_funcs bochs_encoder_encoder_funcs = {
- .destroy = drm_encoder_cleanup,
+static const struct drm_simple_display_pipe_funcs bochs_pipe_funcs = {
+ .enable = bochs_pipe_enable,
+ .update = bochs_pipe_update,
+ .prepare_fb = bochs_pipe_prepare_fb,
+ .cleanup_fb = bochs_pipe_cleanup_fb,
};
-static void bochs_encoder_init(struct drm_device *dev)
-{
- struct bochs_device *bochs = dev->dev_private;
- struct drm_encoder *encoder = &bochs->encoder;
-
- encoder->possible_crtcs = 0x1;
- drm_encoder_init(dev, encoder, &bochs_encoder_encoder_funcs,
- DRM_MODE_ENCODER_DAC, NULL);
-}
-
-
static int bochs_connector_get_modes(struct drm_connector *connector)
{
struct bochs_device *bochs =
return MODE_OK;
}
-static struct drm_encoder *
-bochs_connector_best_encoder(struct drm_connector *connector)
-{
- int enc_id = connector->encoder_ids[0];
- /* pick the encoder ids */
- if (enc_id)
- return drm_encoder_find(connector->dev, NULL, enc_id);
- return NULL;
-}
-
static const struct drm_connector_helper_funcs bochs_connector_connector_helper_funcs = {
.get_modes = bochs_connector_get_modes,
.mode_valid = bochs_connector_mode_valid,
- .best_encoder = bochs_connector_best_encoder,
};
static const struct drm_connector_funcs bochs_connector_connector_funcs = {
int bochs_kms_init(struct bochs_device *bochs)
{
drm_mode_config_init(bochs->dev);
- bochs->mode_config_initialized = true;
bochs->dev->mode_config.max_width = 8192;
bochs->dev->mode_config.max_height = 8192;
bochs->dev->mode_config.funcs = &bochs_mode_funcs;
- bochs_crtc_init(bochs->dev);
- bochs_encoder_init(bochs->dev);
bochs_connector_init(bochs->dev);
- drm_connector_attach_encoder(&bochs->connector,
- &bochs->encoder);
+ drm_simple_display_pipe_init(bochs->dev,
+ &bochs->pipe,
+ &bochs_pipe_funcs,
+ bochs_formats,
+ ARRAY_SIZE(bochs_formats),
+ NULL,
+ &bochs->connector);
drm_mode_config_reset(bochs->dev);
void bochs_kms_fini(struct bochs_device *bochs)
{
- if (bochs->mode_config_initialized) {
- drm_mode_config_cleanup(bochs->dev);
- bochs->mode_config_initialized = false;
- }
+ drm_atomic_helper_shutdown(bochs->dev);
+ drm_mode_config_cleanup(bochs->dev);
}
ret = ttm_bo_device_init(&bochs->ttm.bdev,
&bochs_bo_driver,
bochs->dev->anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET,
true);
if (ret) {
DRM_ERROR("Error initialising bo driver; %d\n", ret);
int bochs_mmap(struct file *filp, struct vm_area_struct *vma)
{
- struct drm_file *file_priv;
- struct bochs_device *bochs;
+ struct drm_file *file_priv = filp->private_data;
+ struct bochs_device *bochs = file_priv->minor->dev->dev_private;
- if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET))
- return -EINVAL;
-
- file_priv = filp->private_data;
- bochs = file_priv->minor->dev->dev_private;
return ttm_bo_mmap(filp, vma, &bochs->ttm.bdev);
}
*/
static const struct drm_bridge_timings default_dac_timings = {
/* Timing specifications, datasheet page 7 */
- .sampling_edge = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .input_bus_flags = DRM_BUS_FLAG_PIXDATA_SAMPLE_POSEDGE,
.setup_time_ps = 500,
.hold_time_ps = 1500,
};
*/
static const struct drm_bridge_timings ti_ths8134_dac_timings = {
/* From timing diagram, datasheet page 9 */
- .sampling_edge = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .input_bus_flags = DRM_BUS_FLAG_PIXDATA_SAMPLE_POSEDGE,
/* From datasheet, page 12 */
.setup_time_ps = 3000,
/* I guess this means latched input */
*/
static const struct drm_bridge_timings ti_ths8135_dac_timings = {
/* From timing diagram, datasheet page 14 */
- .sampling_edge = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .input_bus_flags = DRM_BUS_FLAG_PIXDATA_SAMPLE_POSEDGE,
/* From datasheet, page 16 */
.setup_time_ps = 2000,
.hold_time_ps = 500,
&bus_format, 1);
tc->connector.display_info.bus_flags =
DRM_BUS_FLAG_DE_HIGH |
- DRM_BUS_FLAG_PIXDATA_NEGEDGE |
- DRM_BUS_FLAG_SYNC_NEGEDGE;
+ DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE |
+ DRM_BUS_FLAG_SYNC_DRIVE_NEGEDGE;
drm_connector_attach_encoder(&tc->connector, tc->bridge.encoder);
return 0;
struct tfp410 {
struct drm_bridge bridge;
struct drm_connector connector;
+ unsigned int connector_type;
+ u32 bus_format;
struct i2c_adapter *ddc;
struct gpio_desc *hpd;
+ int hpd_irq;
struct delayed_work hpd_work;
+ struct gpio_desc *powerdown;
+
+ struct drm_bridge_timings timings;
struct device *dev;
};
return -ENODEV;
}
- if (dvi->hpd)
+ if (dvi->hpd_irq >= 0)
dvi->connector.polled = DRM_CONNECTOR_POLL_HPD;
+ else
+ dvi->connector.polled = DRM_CONNECTOR_POLL_CONNECT | DRM_CONNECTOR_POLL_DISCONNECT;
drm_connector_helper_add(&dvi->connector,
&tfp410_con_helper_funcs);
ret = drm_connector_init(bridge->dev, &dvi->connector,
- &tfp410_con_funcs, DRM_MODE_CONNECTOR_HDMIA);
+ &tfp410_con_funcs, dvi->connector_type);
if (ret) {
dev_err(dvi->dev, "drm_connector_init() failed: %d\n", ret);
return ret;
}
+ drm_display_info_set_bus_formats(&dvi->connector.display_info,
+ &dvi->bus_format, 1);
+
drm_connector_attach_encoder(&dvi->connector,
bridge->encoder);
return 0;
}
+static void tfp410_enable(struct drm_bridge *bridge)
+{
+ struct tfp410 *dvi = drm_bridge_to_tfp410(bridge);
+
+ gpiod_set_value_cansleep(dvi->powerdown, 0);
+}
+
+static void tfp410_disable(struct drm_bridge *bridge)
+{
+ struct tfp410 *dvi = drm_bridge_to_tfp410(bridge);
+
+ gpiod_set_value_cansleep(dvi->powerdown, 1);
+}
+
static const struct drm_bridge_funcs tfp410_bridge_funcs = {
.attach = tfp410_attach,
+ .enable = tfp410_enable,
+ .disable = tfp410_disable,
};
static void tfp410_hpd_work_func(struct work_struct *work)
return IRQ_HANDLED;
}
+static const struct drm_bridge_timings tfp410_default_timings = {
+ .input_bus_flags = DRM_BUS_FLAG_PIXDATA_SAMPLE_POSEDGE
+ | DRM_BUS_FLAG_DE_HIGH,
+ .setup_time_ps = 1200,
+ .hold_time_ps = 1300,
+};
+
+static int tfp410_parse_timings(struct tfp410 *dvi, bool i2c)
+{
+ struct drm_bridge_timings *timings = &dvi->timings;
+ struct device_node *ep;
+ u32 pclk_sample = 0;
+ u32 bus_width = 24;
+ s32 deskew = 0;
+
+ /* Start with defaults. */
+ *timings = tfp410_default_timings;
+
+ if (i2c)
+ /*
+ * In I2C mode timings are configured through the I2C interface.
+ * As the driver doesn't support I2C configuration yet, we just
+ * go with the defaults (BSEL=1, DSEL=1, DKEN=0, EDGE=1).
+ */
+ return 0;
+
+ /*
+ * In non-I2C mode, timings are configured through the BSEL, DSEL, DKEN
+ * and EDGE pins. They are specified in DT through endpoint properties
+ * and vendor-specific properties.
+ */
+ ep = of_graph_get_endpoint_by_regs(dvi->dev->of_node, 0, 0);
+ if (!ep)
+ return -EINVAL;
+
+ /* Get the sampling edge from the endpoint. */
+ of_property_read_u32(ep, "pclk-sample", &pclk_sample);
+ of_property_read_u32(ep, "bus-width", &bus_width);
+ of_node_put(ep);
+
+ timings->input_bus_flags = DRM_BUS_FLAG_DE_HIGH;
+
+ switch (pclk_sample) {
+ case 0:
+ timings->input_bus_flags |= DRM_BUS_FLAG_PIXDATA_SAMPLE_NEGEDGE
+ | DRM_BUS_FLAG_SYNC_SAMPLE_NEGEDGE;
+ break;
+ case 1:
+ timings->input_bus_flags |= DRM_BUS_FLAG_PIXDATA_SAMPLE_POSEDGE
+ | DRM_BUS_FLAG_SYNC_SAMPLE_POSEDGE;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ switch (bus_width) {
+ case 12:
+ dvi->bus_format = MEDIA_BUS_FMT_RGB888_2X12_LE;
+ break;
+ case 24:
+ dvi->bus_format = MEDIA_BUS_FMT_RGB888_1X24;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ /* Get the setup and hold time from vendor-specific properties. */
+ of_property_read_u32(dvi->dev->of_node, "ti,deskew", (u32 *)&deskew);
+ if (deskew < -4 || deskew > 3)
+ return -EINVAL;
+
+ timings->setup_time_ps = min(0, 1200 - 350 * deskew);
+ timings->hold_time_ps = min(0, 1300 + 350 * deskew);
+
+ return 0;
+}
+
static int tfp410_get_connector_properties(struct tfp410 *dvi)
{
struct device_node *connector_node, *ddc_phandle;
if (!connector_node)
return -ENODEV;
+ if (of_device_is_compatible(connector_node, "hdmi-connector"))
+ dvi->connector_type = DRM_MODE_CONNECTOR_HDMIA;
+ else
+ dvi->connector_type = DRM_MODE_CONNECTOR_DVID;
+
dvi->hpd = fwnode_get_named_gpiod(&connector_node->fwnode,
"hpd-gpios", 0, GPIOD_IN, "hpd");
if (IS_ERR(dvi->hpd)) {
return ret;
}
-static int tfp410_init(struct device *dev)
+static int tfp410_init(struct device *dev, bool i2c)
{
struct tfp410 *dvi;
int ret;
dvi->bridge.funcs = &tfp410_bridge_funcs;
dvi->bridge.of_node = dev->of_node;
+ dvi->bridge.timings = &dvi->timings;
dvi->dev = dev;
+ ret = tfp410_parse_timings(dvi, i2c);
+ if (ret)
+ goto fail;
+
ret = tfp410_get_connector_properties(dvi);
if (ret)
goto fail;
- if (dvi->hpd) {
+ dvi->powerdown = devm_gpiod_get_optional(dev, "powerdown",
+ GPIOD_OUT_HIGH);
+ if (IS_ERR(dvi->powerdown)) {
+ dev_err(dev, "failed to parse powerdown gpio\n");
+ return PTR_ERR(dvi->powerdown);
+ }
+
+ if (dvi->hpd)
+ dvi->hpd_irq = gpiod_to_irq(dvi->hpd);
+ else
+ dvi->hpd_irq = -ENXIO;
+
+ if (dvi->hpd_irq >= 0) {
INIT_DELAYED_WORK(&dvi->hpd_work, tfp410_hpd_work_func);
- ret = devm_request_threaded_irq(dev, gpiod_to_irq(dvi->hpd),
+ ret = devm_request_threaded_irq(dev, dvi->hpd_irq,
NULL, tfp410_hpd_irq_thread, IRQF_TRIGGER_RISING |
IRQF_TRIGGER_FALLING | IRQF_ONESHOT,
"hdmi-hpd", dvi);
static int tfp410_probe(struct platform_device *pdev)
{
- return tfp410_init(&pdev->dev);
+ return tfp410_init(&pdev->dev, false);
}
static int tfp410_remove(struct platform_device *pdev)
return -ENXIO;
}
- return tfp410_init(&client->dev);
+ return tfp410_init(&client->dev, true);
}
static int tfp410_i2c_remove(struct i2c_client *client)
tristate "Cirrus driver for QEMU emulated device"
depends on DRM && PCI && MMU
select DRM_KMS_HELPER
- select DRM_TTM
+ select DRM_GEM_SHMEM_HELPER
help
This is a KMS driver for emulated cirrus device in qemu.
It is *NOT* intended for real cirrus devices. This requires
-cirrus-y := cirrus_main.o cirrus_mode.o \
- cirrus_drv.o cirrus_fbdev.o cirrus_ttm.o
-
obj-$(CONFIG_DRM_CIRRUS_QEMU) += cirrus.o
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2012-2019 Red Hat
+ *
+ * This file is subject to the terms and conditions of the GNU General
+ * Public License version 2. See the file COPYING in the main
+ * directory of this archive for more details.
+ *
+ * Authors: Matthew Garrett
+ * Dave Airlie
+ * Gerd Hoffmann
+ *
+ * Portions of this code derived from cirrusfb.c:
+ * drivers/video/cirrusfb.c - driver for Cirrus Logic chipsets
+ *
+ * Copyright 1999-2001 Jeff Garzik <jgarzik@pobox.com>
+ */
+
+#include <linux/console.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+
+#include <video/cirrus.h>
+#include <video/vga.h>
+
+#include <drm/drm_atomic_helper.h>
+#include <drm/drm_atomic_state_helper.h>
+#include <drm/drm_connector.h>
+#include <drm/drm_damage_helper.h>
+#include <drm/drm_drv.h>
+#include <drm/drm_fb_helper.h>
+#include <drm/drm_file.h>
+#include <drm/drm_format_helper.h>
+#include <drm/drm_fourcc.h>
+#include <drm/drm_gem_shmem_helper.h>
+#include <drm/drm_gem_framebuffer_helper.h>
+#include <drm/drm_ioctl.h>
+#include <drm/drm_modeset_helper_vtables.h>
+#include <drm/drm_probe_helper.h>
+#include <drm/drm_simple_kms_helper.h>
+#include <drm/drm_vblank.h>
+
+#define DRIVER_NAME "cirrus"
+#define DRIVER_DESC "qemu cirrus vga"
+#define DRIVER_DATE "2019"
+#define DRIVER_MAJOR 2
+#define DRIVER_MINOR 0
+
+#define CIRRUS_MAX_PITCH (0x1FF << 3) /* (4096 - 1) & ~111b bytes */
+#define CIRRUS_VRAM_SIZE (4 * 1024 * 1024) /* 4 MB */
+
+struct cirrus_device {
+ struct drm_device dev;
+ struct drm_simple_display_pipe pipe;
+ struct drm_connector conn;
+ unsigned int cpp;
+ unsigned int pitch;
+ void __iomem *vram;
+ void __iomem *mmio;
+};
+
+/* ------------------------------------------------------------------ */
+/*
+ * The meat of this driver. The core passes us a mode and we have to program
+ * it. The modesetting here is the bare minimum required to satisfy the qemu
+ * emulation of this hardware, and running this against a real device is
+ * likely to result in an inadequately programmed mode. We've already had
+ * the opportunity to modify the mode, so whatever we receive here should
+ * be something that can be correctly programmed and displayed
+ */
+
+#define SEQ_INDEX 4
+#define SEQ_DATA 5
+
+static u8 rreg_seq(struct cirrus_device *cirrus, u8 reg)
+{
+ iowrite8(reg, cirrus->mmio + SEQ_INDEX);
+ return ioread8(cirrus->mmio + SEQ_DATA);
+}
+
+static void wreg_seq(struct cirrus_device *cirrus, u8 reg, u8 val)
+{
+ iowrite8(reg, cirrus->mmio + SEQ_INDEX);
+ iowrite8(val, cirrus->mmio + SEQ_DATA);
+}
+
+#define CRT_INDEX 0x14
+#define CRT_DATA 0x15
+
+static u8 rreg_crt(struct cirrus_device *cirrus, u8 reg)
+{
+ iowrite8(reg, cirrus->mmio + CRT_INDEX);
+ return ioread8(cirrus->mmio + CRT_DATA);
+}
+
+static void wreg_crt(struct cirrus_device *cirrus, u8 reg, u8 val)
+{
+ iowrite8(reg, cirrus->mmio + CRT_INDEX);
+ iowrite8(val, cirrus->mmio + CRT_DATA);
+}
+
+#define GFX_INDEX 0xe
+#define GFX_DATA 0xf
+
+static void wreg_gfx(struct cirrus_device *cirrus, u8 reg, u8 val)
+{
+ iowrite8(reg, cirrus->mmio + GFX_INDEX);
+ iowrite8(val, cirrus->mmio + GFX_DATA);
+}
+
+#define VGA_DAC_MASK 0x06
+
+static void wreg_hdr(struct cirrus_device *cirrus, u8 val)
+{
+ ioread8(cirrus->mmio + VGA_DAC_MASK);
+ ioread8(cirrus->mmio + VGA_DAC_MASK);
+ ioread8(cirrus->mmio + VGA_DAC_MASK);
+ ioread8(cirrus->mmio + VGA_DAC_MASK);
+ iowrite8(val, cirrus->mmio + VGA_DAC_MASK);
+}
+
+static int cirrus_convert_to(struct drm_framebuffer *fb)
+{
+ if (fb->format->cpp[0] == 4 && fb->pitches[0] > CIRRUS_MAX_PITCH) {
+ if (fb->width * 3 <= CIRRUS_MAX_PITCH)
+ /* convert from XR24 to RG24 */
+ return 3;
+ else
+ /* convert from XR24 to RG16 */
+ return 2;
+ }
+ return 0;
+}
+
+static int cirrus_cpp(struct drm_framebuffer *fb)
+{
+ int convert_cpp = cirrus_convert_to(fb);
+
+ if (convert_cpp)
+ return convert_cpp;
+ return fb->format->cpp[0];
+}
+
+static int cirrus_pitch(struct drm_framebuffer *fb)
+{
+ int convert_cpp = cirrus_convert_to(fb);
+
+ if (convert_cpp)
+ return convert_cpp * fb->width;
+ return fb->pitches[0];
+}
+
+static void cirrus_set_start_address(struct cirrus_device *cirrus, u32 offset)
+{
+ u32 addr;
+ u8 tmp;
+
+ addr = offset >> 2;
+ wreg_crt(cirrus, 0x0c, (u8)((addr >> 8) & 0xff));
+ wreg_crt(cirrus, 0x0d, (u8)(addr & 0xff));
+
+ tmp = rreg_crt(cirrus, 0x1b);
+ tmp &= 0xf2;
+ tmp |= (addr >> 16) & 0x01;
+ tmp |= (addr >> 15) & 0x0c;
+ wreg_crt(cirrus, 0x1b, tmp);
+
+ tmp = rreg_crt(cirrus, 0x1d);
+ tmp &= 0x7f;
+ tmp |= (addr >> 12) & 0x80;
+ wreg_crt(cirrus, 0x1d, tmp);
+}
+
+static int cirrus_mode_set(struct cirrus_device *cirrus,
+ struct drm_display_mode *mode,
+ struct drm_framebuffer *fb)
+{
+ int hsyncstart, hsyncend, htotal, hdispend;
+ int vtotal, vdispend;
+ int tmp;
+ int sr07 = 0, hdr = 0;
+
+ htotal = mode->htotal / 8;
+ hsyncend = mode->hsync_end / 8;
+ hsyncstart = mode->hsync_start / 8;
+ hdispend = mode->hdisplay / 8;
+
+ vtotal = mode->vtotal;
+ vdispend = mode->vdisplay;
+
+ vdispend -= 1;
+ vtotal -= 2;
+
+ htotal -= 5;
+ hdispend -= 1;
+ hsyncstart += 1;
+ hsyncend += 1;
+
+ wreg_crt(cirrus, VGA_CRTC_V_SYNC_END, 0x20);
+ wreg_crt(cirrus, VGA_CRTC_H_TOTAL, htotal);
+ wreg_crt(cirrus, VGA_CRTC_H_DISP, hdispend);
+ wreg_crt(cirrus, VGA_CRTC_H_SYNC_START, hsyncstart);
+ wreg_crt(cirrus, VGA_CRTC_H_SYNC_END, hsyncend);
+ wreg_crt(cirrus, VGA_CRTC_V_TOTAL, vtotal & 0xff);
+ wreg_crt(cirrus, VGA_CRTC_V_DISP_END, vdispend & 0xff);
+
+ tmp = 0x40;
+ if ((vdispend + 1) & 512)
+ tmp |= 0x20;
+ wreg_crt(cirrus, VGA_CRTC_MAX_SCAN, tmp);
+
+ /*
+ * Overflow bits for values that don't fit in the standard registers
+ */
+ tmp = 0x10;
+ if (vtotal & 0x100)
+ tmp |= 0x01;
+ if (vdispend & 0x100)
+ tmp |= 0x02;
+ if ((vdispend + 1) & 0x100)
+ tmp |= 0x08;
+ if (vtotal & 0x200)
+ tmp |= 0x20;
+ if (vdispend & 0x200)
+ tmp |= 0x40;
+ wreg_crt(cirrus, VGA_CRTC_OVERFLOW, tmp);
+
+ tmp = 0;
+
+ /* More overflow bits */
+
+ if ((htotal + 5) & 0x40)
+ tmp |= 0x10;
+ if ((htotal + 5) & 0x80)
+ tmp |= 0x20;
+ if (vtotal & 0x100)
+ tmp |= 0x40;
+ if (vtotal & 0x200)
+ tmp |= 0x80;
+
+ wreg_crt(cirrus, CL_CRT1A, tmp);
+
+ /* Disable Hercules/CGA compatibility */
+ wreg_crt(cirrus, VGA_CRTC_MODE, 0x03);
+
+ sr07 = rreg_seq(cirrus, 0x07);
+ sr07 &= 0xe0;
+ hdr = 0;
+
+ cirrus->cpp = cirrus_cpp(fb);
+ switch (cirrus->cpp * 8) {
+ case 8:
+ sr07 |= 0x11;
+ break;
+ case 16:
+ sr07 |= 0x17;
+ hdr = 0xc1;
+ break;
+ case 24:
+ sr07 |= 0x15;
+ hdr = 0xc5;
+ break;
+ case 32:
+ sr07 |= 0x19;
+ hdr = 0xc5;
+ break;
+ default:
+ return -1;
+ }
+
+ wreg_seq(cirrus, 0x7, sr07);
+
+ /* Program the pitch */
+ cirrus->pitch = cirrus_pitch(fb);
+ tmp = cirrus->pitch / 8;
+ wreg_crt(cirrus, VGA_CRTC_OFFSET, tmp);
+
+ /* Enable extended blanking and pitch bits, and enable full memory */
+ tmp = 0x22;
+ tmp |= (cirrus->pitch >> 7) & 0x10;
+ tmp |= (cirrus->pitch >> 6) & 0x40;
+ wreg_crt(cirrus, 0x1b, tmp);
+
+ /* Enable high-colour modes */
+ wreg_gfx(cirrus, VGA_GFX_MODE, 0x40);
+
+ /* And set graphics mode */
+ wreg_gfx(cirrus, VGA_GFX_MISC, 0x01);
+
+ wreg_hdr(cirrus, hdr);
+
+ cirrus_set_start_address(cirrus, 0);
+
+ /* Unblank (needed on S3 resume, vgabios doesn't do it then) */
+ outb(0x20, 0x3c0);
+ return 0;
+}
+
+static int cirrus_fb_blit_rect(struct drm_framebuffer *fb,
+ struct drm_rect *rect)
+{
+ struct cirrus_device *cirrus = fb->dev->dev_private;
+ void *vmap;
+
+ vmap = drm_gem_shmem_vmap(fb->obj[0]);
+ if (!vmap)
+ return -ENOMEM;
+
+ if (cirrus->cpp == fb->format->cpp[0])
+ drm_fb_memcpy_dstclip(cirrus->vram,
+ vmap, fb, rect);
+
+ else if (fb->format->cpp[0] == 4 && cirrus->cpp == 2)
+ drm_fb_xrgb8888_to_rgb565_dstclip(cirrus->vram,
+ cirrus->pitch,
+ vmap, fb, rect, false);
+
+ else if (fb->format->cpp[0] == 4 && cirrus->cpp == 3)
+ drm_fb_xrgb8888_to_rgb888_dstclip(cirrus->vram,
+ cirrus->pitch,
+ vmap, fb, rect);
+
+ else
+ WARN_ON_ONCE("cpp mismatch");
+
+ drm_gem_shmem_vunmap(fb->obj[0], vmap);
+ return 0;
+}
+
+static int cirrus_fb_blit_fullscreen(struct drm_framebuffer *fb)
+{
+ struct drm_rect fullscreen = {
+ .x1 = 0,
+ .x2 = fb->width,
+ .y1 = 0,
+ .y2 = fb->height,
+ };
+ return cirrus_fb_blit_rect(fb, &fullscreen);
+}
+
+static int cirrus_check_size(int width, int height,
+ struct drm_framebuffer *fb)
+{
+ int pitch = width * 2;
+
+ if (fb)
+ pitch = cirrus_pitch(fb);
+
+ if (pitch > CIRRUS_MAX_PITCH)
+ return -EINVAL;
+ if (pitch * height > CIRRUS_VRAM_SIZE)
+ return -EINVAL;
+ return 0;
+}
+
+/* ------------------------------------------------------------------ */
+/* cirrus connector */
+
+static int cirrus_conn_get_modes(struct drm_connector *conn)
+{
+ int count;
+
+ count = drm_add_modes_noedid(conn,
+ conn->dev->mode_config.max_width,
+ conn->dev->mode_config.max_height);
+ drm_set_preferred_mode(conn, 1024, 768);
+ return count;
+}
+
+static const struct drm_connector_helper_funcs cirrus_conn_helper_funcs = {
+ .get_modes = cirrus_conn_get_modes,
+};
+
+static const struct drm_connector_funcs cirrus_conn_funcs = {
+ .fill_modes = drm_helper_probe_single_connector_modes,
+ .destroy = drm_connector_cleanup,
+ .reset = drm_atomic_helper_connector_reset,
+ .atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state,
+ .atomic_destroy_state = drm_atomic_helper_connector_destroy_state,
+};
+
+static int cirrus_conn_init(struct cirrus_device *cirrus)
+{
+ drm_connector_helper_add(&cirrus->conn, &cirrus_conn_helper_funcs);
+ return drm_connector_init(&cirrus->dev, &cirrus->conn,
+ &cirrus_conn_funcs, DRM_MODE_CONNECTOR_VGA);
+
+}
+
+/* ------------------------------------------------------------------ */
+/* cirrus (simple) display pipe */
+
+static enum drm_mode_status cirrus_pipe_mode_valid(struct drm_crtc *crtc,
+ const struct drm_display_mode *mode)
+{
+ if (cirrus_check_size(mode->hdisplay, mode->vdisplay, NULL) < 0)
+ return MODE_BAD;
+ return MODE_OK;
+}
+
+static int cirrus_pipe_check(struct drm_simple_display_pipe *pipe,
+ struct drm_plane_state *plane_state,
+ struct drm_crtc_state *crtc_state)
+{
+ struct drm_framebuffer *fb = plane_state->fb;
+
+ if (!fb)
+ return 0;
+ return cirrus_check_size(fb->width, fb->height, fb);
+}
+
+static void cirrus_pipe_enable(struct drm_simple_display_pipe *pipe,
+ struct drm_crtc_state *crtc_state,
+ struct drm_plane_state *plane_state)
+{
+ struct cirrus_device *cirrus = pipe->crtc.dev->dev_private;
+
+ cirrus_mode_set(cirrus, &crtc_state->mode, plane_state->fb);
+ cirrus_fb_blit_fullscreen(plane_state->fb);
+}
+
+static void cirrus_pipe_update(struct drm_simple_display_pipe *pipe,
+ struct drm_plane_state *old_state)
+{
+ struct cirrus_device *cirrus = pipe->crtc.dev->dev_private;
+ struct drm_plane_state *state = pipe->plane.state;
+ struct drm_crtc *crtc = &pipe->crtc;
+ struct drm_rect rect;
+
+ if (pipe->plane.state->fb &&
+ cirrus->cpp != cirrus_cpp(pipe->plane.state->fb))
+ cirrus_mode_set(cirrus, &crtc->mode,
+ pipe->plane.state->fb);
+
+ if (drm_atomic_helper_damage_merged(old_state, state, &rect))
+ cirrus_fb_blit_rect(pipe->plane.state->fb, &rect);
+
+ if (crtc->state->event) {
+ spin_lock_irq(&crtc->dev->event_lock);
+ drm_crtc_send_vblank_event(crtc, crtc->state->event);
+ crtc->state->event = NULL;
+ spin_unlock_irq(&crtc->dev->event_lock);
+ }
+}
+
+static const struct drm_simple_display_pipe_funcs cirrus_pipe_funcs = {
+ .mode_valid = cirrus_pipe_mode_valid,
+ .check = cirrus_pipe_check,
+ .enable = cirrus_pipe_enable,
+ .update = cirrus_pipe_update,
+};
+
+static const uint32_t cirrus_formats[] = {
+ DRM_FORMAT_RGB565,
+ DRM_FORMAT_RGB888,
+ DRM_FORMAT_XRGB8888,
+};
+
+static const uint64_t cirrus_modifiers[] = {
+ DRM_FORMAT_MOD_LINEAR,
+ DRM_FORMAT_MOD_INVALID
+};
+
+static int cirrus_pipe_init(struct cirrus_device *cirrus)
+{
+ return drm_simple_display_pipe_init(&cirrus->dev,
+ &cirrus->pipe,
+ &cirrus_pipe_funcs,
+ cirrus_formats,
+ ARRAY_SIZE(cirrus_formats),
+ cirrus_modifiers,
+ &cirrus->conn);
+}
+
+/* ------------------------------------------------------------------ */
+/* cirrus framebuffers & mode config */
+
+static struct drm_framebuffer*
+cirrus_fb_create(struct drm_device *dev, struct drm_file *file_priv,
+ const struct drm_mode_fb_cmd2 *mode_cmd)
+{
+ if (mode_cmd->pixel_format != DRM_FORMAT_RGB565 &&
+ mode_cmd->pixel_format != DRM_FORMAT_RGB888 &&
+ mode_cmd->pixel_format != DRM_FORMAT_XRGB8888)
+ return ERR_PTR(-EINVAL);
+ if (cirrus_check_size(mode_cmd->width, mode_cmd->height, NULL) < 0)
+ return ERR_PTR(-EINVAL);
+ return drm_gem_fb_create_with_dirty(dev, file_priv, mode_cmd);
+}
+
+static const struct drm_mode_config_funcs cirrus_mode_config_funcs = {
+ .fb_create = cirrus_fb_create,
+ .atomic_check = drm_atomic_helper_check,
+ .atomic_commit = drm_atomic_helper_commit,
+};
+
+static void cirrus_mode_config_init(struct cirrus_device *cirrus)
+{
+ struct drm_device *dev = &cirrus->dev;
+
+ drm_mode_config_init(dev);
+ dev->mode_config.min_width = 0;
+ dev->mode_config.min_height = 0;
+ dev->mode_config.max_width = CIRRUS_MAX_PITCH / 2;
+ dev->mode_config.max_height = 1024;
+ dev->mode_config.preferred_depth = 16;
+ dev->mode_config.prefer_shadow = 0;
+ dev->mode_config.funcs = &cirrus_mode_config_funcs;
+}
+
+/* ------------------------------------------------------------------ */
+
+DEFINE_DRM_GEM_SHMEM_FOPS(cirrus_fops);
+
+static struct drm_driver cirrus_driver = {
+ .driver_features = DRIVER_MODESET | DRIVER_GEM | DRIVER_ATOMIC | DRIVER_PRIME,
+
+ .name = DRIVER_NAME,
+ .desc = DRIVER_DESC,
+ .date = DRIVER_DATE,
+ .major = DRIVER_MAJOR,
+ .minor = DRIVER_MINOR,
+
+ .fops = &cirrus_fops,
+ DRM_GEM_SHMEM_DRIVER_OPS,
+};
+
+static int cirrus_pci_probe(struct pci_dev *pdev,
+ const struct pci_device_id *ent)
+{
+ struct drm_device *dev;
+ struct cirrus_device *cirrus;
+ int ret;
+
+ ret = drm_fb_helper_remove_conflicting_pci_framebuffers(pdev, 0, "cirrusdrmfb");
+ if (ret)
+ return ret;
+
+ ret = pci_enable_device(pdev);
+ if (ret)
+ return ret;
+
+ ret = pci_request_regions(pdev, DRIVER_NAME);
+ if (ret)
+ return ret;
+
+ ret = -ENOMEM;
+ cirrus = kzalloc(sizeof(*cirrus), GFP_KERNEL);
+ if (cirrus == NULL)
+ goto err_pci_release;
+
+ dev = &cirrus->dev;
+ ret = drm_dev_init(dev, &cirrus_driver, &pdev->dev);
+ if (ret)
+ goto err_free_cirrus;
+ dev->dev_private = cirrus;
+
+ ret = -ENOMEM;
+ cirrus->vram = ioremap(pci_resource_start(pdev, 0),
+ pci_resource_len(pdev, 0));
+ if (cirrus->vram == NULL)
+ goto err_dev_put;
+
+ cirrus->mmio = ioremap(pci_resource_start(pdev, 1),
+ pci_resource_len(pdev, 1));
+ if (cirrus->mmio == NULL)
+ goto err_unmap_vram;
+
+ cirrus_mode_config_init(cirrus);
+
+ ret = cirrus_conn_init(cirrus);
+ if (ret < 0)
+ goto err_cleanup;
+
+ ret = cirrus_pipe_init(cirrus);
+ if (ret < 0)
+ goto err_cleanup;
+
+ drm_mode_config_reset(dev);
+
+ dev->pdev = pdev;
+ pci_set_drvdata(pdev, dev);
+ ret = drm_dev_register(dev, 0);
+ if (ret)
+ goto err_cleanup;
+
+ drm_fbdev_generic_setup(dev, dev->mode_config.preferred_depth);
+ return 0;
+
+err_cleanup:
+ drm_mode_config_cleanup(dev);
+ iounmap(cirrus->mmio);
+err_unmap_vram:
+ iounmap(cirrus->vram);
+err_dev_put:
+ drm_dev_put(dev);
+err_free_cirrus:
+ kfree(cirrus);
+err_pci_release:
+ pci_release_regions(pdev);
+ return ret;
+}
+
+static void cirrus_pci_remove(struct pci_dev *pdev)
+{
+ struct drm_device *dev = pci_get_drvdata(pdev);
+ struct cirrus_device *cirrus = dev->dev_private;
+
+ drm_dev_unregister(dev);
+ drm_mode_config_cleanup(dev);
+ iounmap(cirrus->mmio);
+ iounmap(cirrus->vram);
+ drm_dev_put(dev);
+ kfree(cirrus);
+ pci_release_regions(pdev);
+}
+
+static const struct pci_device_id pciidlist[] = {
+ {
+ .vendor = PCI_VENDOR_ID_CIRRUS,
+ .device = PCI_DEVICE_ID_CIRRUS_5446,
+ /* only bind to the cirrus chip in qemu */
+ .subvendor = PCI_SUBVENDOR_ID_REDHAT_QUMRANET,
+ .subdevice = PCI_SUBDEVICE_ID_QEMU,
+ }, {
+ .vendor = PCI_VENDOR_ID_CIRRUS,
+ .device = PCI_DEVICE_ID_CIRRUS_5446,
+ .subvendor = PCI_VENDOR_ID_XEN,
+ .subdevice = 0x0001,
+ },
+ { /* end if list */ }
+};
+
+static struct pci_driver cirrus_pci_driver = {
+ .name = DRIVER_NAME,
+ .id_table = pciidlist,
+ .probe = cirrus_pci_probe,
+ .remove = cirrus_pci_remove,
+};
+
+static int __init cirrus_init(void)
+{
+ if (vgacon_text_force())
+ return -EINVAL;
+ return pci_register_driver(&cirrus_pci_driver);
+}
+
+static void __exit cirrus_exit(void)
+{
+ pci_unregister_driver(&cirrus_pci_driver);
+}
+
+module_init(cirrus_init);
+module_exit(cirrus_exit);
+
+MODULE_DEVICE_TABLE(pci, pciidlist);
+MODULE_LICENSE("GPL");
+++ /dev/null
-/*
- * Copyright 2012 Red Hat <mjg@redhat.com>
- *
- * This file is subject to the terms and conditions of the GNU General
- * Public License version 2. See the file COPYING in the main
- * directory of this archive for more details.
- *
- * Authors: Matthew Garrett
- * Dave Airlie
- */
-#include <linux/module.h>
-#include <linux/console.h>
-#include <drm/drmP.h>
-#include <drm/drm_crtc_helper.h>
-#include <drm/drm_probe_helper.h>
-
-#include "cirrus_drv.h"
-
-int cirrus_modeset = -1;
-int cirrus_bpp = 16;
-
-MODULE_PARM_DESC(modeset, "Disable/Enable modesetting");
-module_param_named(modeset, cirrus_modeset, int, 0400);
-MODULE_PARM_DESC(bpp, "Max bits-per-pixel (default:16)");
-module_param_named(bpp, cirrus_bpp, int, 0400);
-
-/*
- * This is the generic driver code. This binds the driver to the drm core,
- * which then performs further device association and calls our graphics init
- * functions
- */
-
-static struct drm_driver driver;
-
-/* only bind to the cirrus chip in qemu */
-static const struct pci_device_id pciidlist[] = {
- { PCI_VENDOR_ID_CIRRUS, PCI_DEVICE_ID_CIRRUS_5446,
- PCI_SUBVENDOR_ID_REDHAT_QUMRANET, PCI_SUBDEVICE_ID_QEMU,
- 0, 0, 0 },
- { PCI_VENDOR_ID_CIRRUS, PCI_DEVICE_ID_CIRRUS_5446, PCI_VENDOR_ID_XEN,
- 0x0001, 0, 0, 0 },
- {0,}
-};
-
-
-static int cirrus_pci_probe(struct pci_dev *pdev,
- const struct pci_device_id *ent)
-{
- int ret;
-
- ret = drm_fb_helper_remove_conflicting_pci_framebuffers(pdev, 0, "cirrusdrmfb");
- if (ret)
- return ret;
-
- return drm_get_pci_dev(pdev, ent, &driver);
-}
-
-static void cirrus_pci_remove(struct pci_dev *pdev)
-{
- struct drm_device *dev = pci_get_drvdata(pdev);
-
- drm_put_dev(dev);
-}
-
-#ifdef CONFIG_PM_SLEEP
-static int cirrus_pm_suspend(struct device *dev)
-{
- struct pci_dev *pdev = to_pci_dev(dev);
- struct drm_device *drm_dev = pci_get_drvdata(pdev);
- struct cirrus_device *cdev = drm_dev->dev_private;
-
- drm_kms_helper_poll_disable(drm_dev);
-
- if (cdev->mode_info.gfbdev) {
- console_lock();
- drm_fb_helper_set_suspend(&cdev->mode_info.gfbdev->helper, 1);
- console_unlock();
- }
-
- return 0;
-}
-
-static int cirrus_pm_resume(struct device *dev)
-{
- struct pci_dev *pdev = to_pci_dev(dev);
- struct drm_device *drm_dev = pci_get_drvdata(pdev);
- struct cirrus_device *cdev = drm_dev->dev_private;
-
- drm_helper_resume_force_mode(drm_dev);
-
- if (cdev->mode_info.gfbdev) {
- console_lock();
- drm_fb_helper_set_suspend(&cdev->mode_info.gfbdev->helper, 0);
- console_unlock();
- }
-
- drm_kms_helper_poll_enable(drm_dev);
- return 0;
-}
-#endif
-
-static const struct file_operations cirrus_driver_fops = {
- .owner = THIS_MODULE,
- .open = drm_open,
- .release = drm_release,
- .unlocked_ioctl = drm_ioctl,
- .mmap = cirrus_mmap,
- .poll = drm_poll,
- .compat_ioctl = drm_compat_ioctl,
-};
-static struct drm_driver driver = {
- .driver_features = DRIVER_MODESET | DRIVER_GEM,
- .load = cirrus_driver_load,
- .unload = cirrus_driver_unload,
- .fops = &cirrus_driver_fops,
- .name = DRIVER_NAME,
- .desc = DRIVER_DESC,
- .date = DRIVER_DATE,
- .major = DRIVER_MAJOR,
- .minor = DRIVER_MINOR,
- .patchlevel = DRIVER_PATCHLEVEL,
- .gem_free_object_unlocked = cirrus_gem_free_object,
- .dumb_create = cirrus_dumb_create,
- .dumb_map_offset = cirrus_dumb_mmap_offset,
-};
-
-static const struct dev_pm_ops cirrus_pm_ops = {
- SET_SYSTEM_SLEEP_PM_OPS(cirrus_pm_suspend,
- cirrus_pm_resume)
-};
-
-static struct pci_driver cirrus_pci_driver = {
- .name = DRIVER_NAME,
- .id_table = pciidlist,
- .probe = cirrus_pci_probe,
- .remove = cirrus_pci_remove,
- .driver.pm = &cirrus_pm_ops,
-};
-
-static int __init cirrus_init(void)
-{
- if (vgacon_text_force() && cirrus_modeset == -1)
- return -EINVAL;
-
- if (cirrus_modeset == 0)
- return -EINVAL;
- return pci_register_driver(&cirrus_pci_driver);
-}
-
-static void __exit cirrus_exit(void)
-{
- pci_unregister_driver(&cirrus_pci_driver);
-}
-
-module_init(cirrus_init);
-module_exit(cirrus_exit);
-
-MODULE_DEVICE_TABLE(pci, pciidlist);
-MODULE_AUTHOR(DRIVER_AUTHOR);
-MODULE_DESCRIPTION(DRIVER_DESC);
-MODULE_LICENSE("GPL");
struct cirrus_fbdev;
struct cirrus_mode_info {
- bool mode_config_initialized;
struct cirrus_crtc *crtc;
/* pointer to fbdev info structure */
struct cirrus_fbdev *gfbdev;
struct cirrus_fbdev {
- struct drm_fb_helper helper;
+ struct drm_fb_helper helper; /* must be first */
struct drm_framebuffer *gfb;
void *sysram;
int size;
#define to_cirrus_obj(x) container_of(x, struct cirrus_gem_object, base)
-#define DRM_FILE_PAGE_OFFSET (0x100000000ULL >> PAGE_SHIFT)
/* cirrus_main.c */
int cirrus_device_init(struct cirrus_device *cdev,
+++ /dev/null
-/*
- * Copyright 2012 Red Hat
- *
- * This file is subject to the terms and conditions of the GNU General
- * Public License version 2. See the file COPYING in the main
- * directory of this archive for more details.
- *
- * Authors: Matthew Garrett
- * Dave Airlie
- */
-#include <linux/module.h>
-#include <drm/drmP.h>
-#include <drm/drm_util.h>
-#include <drm/drm_fb_helper.h>
-#include <drm/drm_crtc_helper.h>
-
-#include "cirrus_drv.h"
-
-static void cirrus_dirty_update(struct cirrus_fbdev *afbdev,
- int x, int y, int width, int height)
-{
- int i;
- struct drm_gem_object *obj;
- struct cirrus_bo *bo;
- int src_offset, dst_offset;
- int bpp = afbdev->gfb->format->cpp[0];
- int ret = -EBUSY;
- bool unmap = false;
- bool store_for_later = false;
- int x2, y2;
- unsigned long flags;
-
- obj = afbdev->gfb->obj[0];
- bo = gem_to_cirrus_bo(obj);
-
- /*
- * try and reserve the BO, if we fail with busy
- * then the BO is being moved and we should
- * store up the damage until later.
- */
- if (drm_can_sleep())
- ret = cirrus_bo_reserve(bo, true);
- if (ret) {
- if (ret != -EBUSY)
- return;
- store_for_later = true;
- }
-
- x2 = x + width - 1;
- y2 = y + height - 1;
- spin_lock_irqsave(&afbdev->dirty_lock, flags);
-
- if (afbdev->y1 < y)
- y = afbdev->y1;
- if (afbdev->y2 > y2)
- y2 = afbdev->y2;
- if (afbdev->x1 < x)
- x = afbdev->x1;
- if (afbdev->x2 > x2)
- x2 = afbdev->x2;
-
- if (store_for_later) {
- afbdev->x1 = x;
- afbdev->x2 = x2;
- afbdev->y1 = y;
- afbdev->y2 = y2;
- spin_unlock_irqrestore(&afbdev->dirty_lock, flags);
- return;
- }
-
- afbdev->x1 = afbdev->y1 = INT_MAX;
- afbdev->x2 = afbdev->y2 = 0;
- spin_unlock_irqrestore(&afbdev->dirty_lock, flags);
-
- if (!bo->kmap.virtual) {
- ret = ttm_bo_kmap(&bo->bo, 0, bo->bo.num_pages, &bo->kmap);
- if (ret) {
- DRM_ERROR("failed to kmap fb updates\n");
- cirrus_bo_unreserve(bo);
- return;
- }
- unmap = true;
- }
- for (i = y; i < y + height; i++) {
- /* assume equal stride for now */
- src_offset = dst_offset = i * afbdev->gfb->pitches[0] + (x * bpp);
- memcpy_toio(bo->kmap.virtual + src_offset, afbdev->sysram + src_offset, width * bpp);
-
- }
- if (unmap)
- ttm_bo_kunmap(&bo->kmap);
-
- cirrus_bo_unreserve(bo);
-}
-
-static void cirrus_fillrect(struct fb_info *info,
- const struct fb_fillrect *rect)
-{
- struct cirrus_fbdev *afbdev = info->par;
- drm_fb_helper_sys_fillrect(info, rect);
- cirrus_dirty_update(afbdev, rect->dx, rect->dy, rect->width,
- rect->height);
-}
-
-static void cirrus_copyarea(struct fb_info *info,
- const struct fb_copyarea *area)
-{
- struct cirrus_fbdev *afbdev = info->par;
- drm_fb_helper_sys_copyarea(info, area);
- cirrus_dirty_update(afbdev, area->dx, area->dy, area->width,
- area->height);
-}
-
-static void cirrus_imageblit(struct fb_info *info,
- const struct fb_image *image)
-{
- struct cirrus_fbdev *afbdev = info->par;
- drm_fb_helper_sys_imageblit(info, image);
- cirrus_dirty_update(afbdev, image->dx, image->dy, image->width,
- image->height);
-}
-
-
-static struct fb_ops cirrusfb_ops = {
- .owner = THIS_MODULE,
- .fb_check_var = drm_fb_helper_check_var,
- .fb_set_par = drm_fb_helper_set_par,
- .fb_fillrect = cirrus_fillrect,
- .fb_copyarea = cirrus_copyarea,
- .fb_imageblit = cirrus_imageblit,
- .fb_pan_display = drm_fb_helper_pan_display,
- .fb_blank = drm_fb_helper_blank,
- .fb_setcmap = drm_fb_helper_setcmap,
-};
-
-static int cirrusfb_create_object(struct cirrus_fbdev *afbdev,
- const struct drm_mode_fb_cmd2 *mode_cmd,
- struct drm_gem_object **gobj_p)
-{
- struct drm_device *dev = afbdev->helper.dev;
- struct cirrus_device *cdev = dev->dev_private;
- u32 bpp;
- u32 size;
- struct drm_gem_object *gobj;
- int ret = 0;
-
- bpp = drm_format_plane_cpp(mode_cmd->pixel_format, 0) * 8;
-
- if (!cirrus_check_framebuffer(cdev, mode_cmd->width, mode_cmd->height,
- bpp, mode_cmd->pitches[0]))
- return -EINVAL;
-
- size = mode_cmd->pitches[0] * mode_cmd->height;
- ret = cirrus_gem_create(dev, size, true, &gobj);
- if (ret)
- return ret;
-
- *gobj_p = gobj;
- return ret;
-}
-
-static int cirrusfb_create(struct drm_fb_helper *helper,
- struct drm_fb_helper_surface_size *sizes)
-{
- struct cirrus_fbdev *gfbdev =
- container_of(helper, struct cirrus_fbdev, helper);
- struct cirrus_device *cdev = gfbdev->helper.dev->dev_private;
- struct fb_info *info;
- struct drm_framebuffer *fb;
- struct drm_mode_fb_cmd2 mode_cmd;
- void *sysram;
- struct drm_gem_object *gobj = NULL;
- int size, ret;
-
- mode_cmd.width = sizes->surface_width;
- mode_cmd.height = sizes->surface_height;
- mode_cmd.pitches[0] = mode_cmd.width * ((sizes->surface_bpp + 7) / 8);
- mode_cmd.pixel_format = drm_mode_legacy_fb_format(sizes->surface_bpp,
- sizes->surface_depth);
- size = mode_cmd.pitches[0] * mode_cmd.height;
-
- ret = cirrusfb_create_object(gfbdev, &mode_cmd, &gobj);
- if (ret) {
- DRM_ERROR("failed to create fbcon backing object %d\n", ret);
- return ret;
- }
-
- sysram = vmalloc(size);
- if (!sysram)
- return -ENOMEM;
-
- info = drm_fb_helper_alloc_fbi(helper);
- if (IS_ERR(info)) {
- ret = PTR_ERR(info);
- goto err_vfree;
- }
-
- info->par = gfbdev;
-
- fb = kzalloc(sizeof(*fb), GFP_KERNEL);
- if (!fb) {
- ret = -ENOMEM;
- goto err_drm_gem_object_put_unlocked;
- }
-
- ret = cirrus_framebuffer_init(cdev->dev, fb, &mode_cmd, gobj);
- if (ret)
- goto err_kfree;
-
- gfbdev->sysram = sysram;
- gfbdev->size = size;
- gfbdev->gfb = fb;
-
- /* setup helper */
- gfbdev->helper.fb = fb;
-
- strcpy(info->fix.id, "cirrusdrmfb");
-
- info->fbops = &cirrusfb_ops;
-
- drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(info, &gfbdev->helper, sizes->fb_width,
- sizes->fb_height);
-
- /* setup aperture base/size for vesafb takeover */
- info->apertures->ranges[0].base = cdev->dev->mode_config.fb_base;
- info->apertures->ranges[0].size = cdev->mc.vram_size;
-
- info->fix.smem_start = cdev->dev->mode_config.fb_base;
- info->fix.smem_len = cdev->mc.vram_size;
-
- info->screen_base = sysram;
- info->screen_size = size;
-
- info->fix.mmio_start = 0;
- info->fix.mmio_len = 0;
-
- DRM_INFO("fb mappable at 0x%lX\n", info->fix.smem_start);
- DRM_INFO("vram aper at 0x%lX\n", (unsigned long)info->fix.smem_start);
- DRM_INFO("size %lu\n", (unsigned long)info->fix.smem_len);
- DRM_INFO("fb depth is %d\n", fb->format->depth);
- DRM_INFO(" pitch is %d\n", fb->pitches[0]);
-
- return 0;
-
-err_kfree:
- kfree(fb);
-err_drm_gem_object_put_unlocked:
- drm_gem_object_put_unlocked(gobj);
-err_vfree:
- vfree(sysram);
- return ret;
-}
-
-static int cirrus_fbdev_destroy(struct drm_device *dev,
- struct cirrus_fbdev *gfbdev)
-{
- struct drm_framebuffer *gfb = gfbdev->gfb;
-
- drm_helper_force_disable_all(dev);
-
- drm_fb_helper_unregister_fbi(&gfbdev->helper);
-
- vfree(gfbdev->sysram);
- drm_fb_helper_fini(&gfbdev->helper);
- if (gfb)
- drm_framebuffer_put(gfb);
-
- return 0;
-}
-
-static const struct drm_fb_helper_funcs cirrus_fb_helper_funcs = {
- .fb_probe = cirrusfb_create,
-};
-
-int cirrus_fbdev_init(struct cirrus_device *cdev)
-{
- struct cirrus_fbdev *gfbdev;
- int ret;
-
- /*bpp_sel = 8;*/
- gfbdev = kzalloc(sizeof(struct cirrus_fbdev), GFP_KERNEL);
- if (!gfbdev)
- return -ENOMEM;
-
- cdev->mode_info.gfbdev = gfbdev;
- spin_lock_init(&gfbdev->dirty_lock);
-
- drm_fb_helper_prepare(cdev->dev, &gfbdev->helper,
- &cirrus_fb_helper_funcs);
-
- ret = drm_fb_helper_init(cdev->dev, &gfbdev->helper,
- CIRRUSFB_CONN_LIMIT);
- if (ret)
- return ret;
-
- ret = drm_fb_helper_single_add_all_connectors(&gfbdev->helper);
- if (ret)
- return ret;
-
- /* disable all the possible outputs/crtcs before entering KMS mode */
- drm_helper_disable_unused_functions(cdev->dev);
-
- return drm_fb_helper_initial_config(&gfbdev->helper, cirrus_bpp);
-}
-
-void cirrus_fbdev_fini(struct cirrus_device *cdev)
-{
- if (!cdev->mode_info.gfbdev)
- return;
-
- cirrus_fbdev_destroy(cdev->dev, cdev->mode_info.gfbdev);
- kfree(cdev->mode_info.gfbdev);
- cdev->mode_info.gfbdev = NULL;
-}
+++ /dev/null
-/*
- * Copyright 2012 Red Hat
- *
- * This file is subject to the terms and conditions of the GNU General
- * Public License version 2. See the file COPYING in the main
- * directory of this archive for more details.
- *
- * Authors: Matthew Garrett
- * Dave Airlie
- */
-#include <drm/drmP.h>
-#include <drm/drm_crtc_helper.h>
-#include <drm/drm_gem_framebuffer_helper.h>
-
-#include "cirrus_drv.h"
-
-static const struct drm_framebuffer_funcs cirrus_fb_funcs = {
- .create_handle = drm_gem_fb_create_handle,
- .destroy = drm_gem_fb_destroy,
-};
-
-int cirrus_framebuffer_init(struct drm_device *dev,
- struct drm_framebuffer *gfb,
- const struct drm_mode_fb_cmd2 *mode_cmd,
- struct drm_gem_object *obj)
-{
- int ret;
-
- drm_helper_mode_fill_fb_struct(dev, gfb, mode_cmd);
- gfb->obj[0] = obj;
- ret = drm_framebuffer_init(dev, gfb, &cirrus_fb_funcs);
- if (ret) {
- DRM_ERROR("drm_framebuffer_init failed: %d\n", ret);
- return ret;
- }
- return 0;
-}
-
-static struct drm_framebuffer *
-cirrus_user_framebuffer_create(struct drm_device *dev,
- struct drm_file *filp,
- const struct drm_mode_fb_cmd2 *mode_cmd)
-{
- struct cirrus_device *cdev = dev->dev_private;
- struct drm_gem_object *obj;
- struct drm_framebuffer *fb;
- u32 bpp;
- int ret;
-
- bpp = drm_format_plane_cpp(mode_cmd->pixel_format, 0) * 8;
-
- if (!cirrus_check_framebuffer(cdev, mode_cmd->width, mode_cmd->height,
- bpp, mode_cmd->pitches[0]))
- return ERR_PTR(-EINVAL);
-
- obj = drm_gem_object_lookup(filp, mode_cmd->handles[0]);
- if (obj == NULL)
- return ERR_PTR(-ENOENT);
-
- fb = kzalloc(sizeof(*fb), GFP_KERNEL);
- if (!fb) {
- drm_gem_object_put_unlocked(obj);
- return ERR_PTR(-ENOMEM);
- }
-
- ret = cirrus_framebuffer_init(dev, fb, mode_cmd, obj);
- if (ret) {
- drm_gem_object_put_unlocked(obj);
- kfree(fb);
- return ERR_PTR(ret);
- }
- return fb;
-}
-
-static const struct drm_mode_config_funcs cirrus_mode_funcs = {
- .fb_create = cirrus_user_framebuffer_create,
-};
-
-/* Unmap the framebuffer from the core and release the memory */
-static void cirrus_vram_fini(struct cirrus_device *cdev)
-{
- iounmap(cdev->rmmio);
- cdev->rmmio = NULL;
- if (cdev->mc.vram_base)
- release_mem_region(cdev->mc.vram_base, cdev->mc.vram_size);
-}
-
-/* Map the framebuffer from the card and configure the core */
-static int cirrus_vram_init(struct cirrus_device *cdev)
-{
- /* BAR 0 is VRAM */
- cdev->mc.vram_base = pci_resource_start(cdev->dev->pdev, 0);
- cdev->mc.vram_size = pci_resource_len(cdev->dev->pdev, 0);
-
- if (!request_mem_region(cdev->mc.vram_base, cdev->mc.vram_size,
- "cirrusdrmfb_vram")) {
- DRM_ERROR("can't reserve VRAM\n");
- return -ENXIO;
- }
-
- return 0;
-}
-
-/*
- * Our emulated hardware has two sets of memory. One is video RAM and can
- * simply be used as a linear framebuffer - the other provides mmio access
- * to the display registers. The latter can also be accessed via IO port
- * access, but we map the range and use mmio to program them instead
- */
-
-int cirrus_device_init(struct cirrus_device *cdev,
- struct drm_device *ddev,
- struct pci_dev *pdev, uint32_t flags)
-{
- int ret;
-
- cdev->dev = ddev;
- cdev->flags = flags;
-
- /* Hardcode the number of CRTCs to 1 */
- cdev->num_crtc = 1;
-
- /* BAR 0 is the framebuffer, BAR 1 contains registers */
- cdev->rmmio_base = pci_resource_start(cdev->dev->pdev, 1);
- cdev->rmmio_size = pci_resource_len(cdev->dev->pdev, 1);
-
- if (!request_mem_region(cdev->rmmio_base, cdev->rmmio_size,
- "cirrusdrmfb_mmio")) {
- DRM_ERROR("can't reserve mmio registers\n");
- return -ENOMEM;
- }
-
- cdev->rmmio = ioremap(cdev->rmmio_base, cdev->rmmio_size);
-
- if (cdev->rmmio == NULL)
- return -ENOMEM;
-
- ret = cirrus_vram_init(cdev);
- if (ret) {
- release_mem_region(cdev->rmmio_base, cdev->rmmio_size);
- return ret;
- }
-
- return 0;
-}
-
-void cirrus_device_fini(struct cirrus_device *cdev)
-{
- release_mem_region(cdev->rmmio_base, cdev->rmmio_size);
- cirrus_vram_fini(cdev);
-}
-
-/*
- * Functions here will be called by the core once it's bound the driver to
- * a PCI device
- */
-
-int cirrus_driver_load(struct drm_device *dev, unsigned long flags)
-{
- struct cirrus_device *cdev;
- int r;
-
- cdev = kzalloc(sizeof(struct cirrus_device), GFP_KERNEL);
- if (cdev == NULL)
- return -ENOMEM;
- dev->dev_private = (void *)cdev;
-
- r = cirrus_device_init(cdev, dev, dev->pdev, flags);
- if (r) {
- dev_err(&dev->pdev->dev, "Fatal error during GPU init: %d\n", r);
- goto out;
- }
-
- r = cirrus_mm_init(cdev);
- if (r) {
- dev_err(&dev->pdev->dev, "fatal err on mm init\n");
- goto out;
- }
-
- /*
- * cirrus_modeset_init() is initializing/registering the emulated fbdev
- * and DRM internals can access/test some of the fields in
- * mode_config->funcs as part of the fbdev registration process.
- * Make sure dev->mode_config.funcs is properly set to avoid
- * dereferencing a NULL pointer.
- * FIXME: mode_config.funcs assignment should probably be done in
- * cirrus_modeset_init() (that's a common pattern seen in other DRM
- * drivers).
- */
- dev->mode_config.funcs = &cirrus_mode_funcs;
- r = cirrus_modeset_init(cdev);
- if (r) {
- dev_err(&dev->pdev->dev, "Fatal error during modeset init: %d\n", r);
- goto out;
- }
-
- return 0;
-out:
- cirrus_driver_unload(dev);
- return r;
-}
-
-void cirrus_driver_unload(struct drm_device *dev)
-{
- struct cirrus_device *cdev = dev->dev_private;
-
- if (cdev == NULL)
- return;
- cirrus_modeset_fini(cdev);
- cirrus_mm_fini(cdev);
- cirrus_device_fini(cdev);
- kfree(cdev);
- dev->dev_private = NULL;
-}
-
-int cirrus_gem_create(struct drm_device *dev,
- u32 size, bool iskernel,
- struct drm_gem_object **obj)
-{
- struct cirrus_bo *cirrusbo;
- int ret;
-
- *obj = NULL;
-
- size = roundup(size, PAGE_SIZE);
- if (size == 0)
- return -EINVAL;
-
- ret = cirrus_bo_create(dev, size, 0, 0, &cirrusbo);
- if (ret) {
- if (ret != -ERESTARTSYS)
- DRM_ERROR("failed to allocate GEM object\n");
- return ret;
- }
- *obj = &cirrusbo->gem;
- return 0;
-}
-
-int cirrus_dumb_create(struct drm_file *file,
- struct drm_device *dev,
- struct drm_mode_create_dumb *args)
-{
- int ret;
- struct drm_gem_object *gobj;
- u32 handle;
-
- args->pitch = args->width * ((args->bpp + 7) / 8);
- args->size = args->pitch * args->height;
-
- ret = cirrus_gem_create(dev, args->size, false,
- &gobj);
- if (ret)
- return ret;
-
- ret = drm_gem_handle_create(file, gobj, &handle);
- drm_gem_object_put_unlocked(gobj);
- if (ret)
- return ret;
-
- args->handle = handle;
- return 0;
-}
-
-static void cirrus_bo_unref(struct cirrus_bo **bo)
-{
- struct ttm_buffer_object *tbo;
-
- if ((*bo) == NULL)
- return;
-
- tbo = &((*bo)->bo);
- ttm_bo_put(tbo);
- *bo = NULL;
-}
-
-void cirrus_gem_free_object(struct drm_gem_object *obj)
-{
- struct cirrus_bo *cirrus_bo = gem_to_cirrus_bo(obj);
-
- cirrus_bo_unref(&cirrus_bo);
-}
-
-
-static inline u64 cirrus_bo_mmap_offset(struct cirrus_bo *bo)
-{
- return drm_vma_node_offset_addr(&bo->bo.vma_node);
-}
-
-int
-cirrus_dumb_mmap_offset(struct drm_file *file,
- struct drm_device *dev,
- uint32_t handle,
- uint64_t *offset)
-{
- struct drm_gem_object *obj;
- struct cirrus_bo *bo;
-
- obj = drm_gem_object_lookup(file, handle);
- if (obj == NULL)
- return -ENOENT;
-
- bo = gem_to_cirrus_bo(obj);
- *offset = cirrus_bo_mmap_offset(bo);
-
- drm_gem_object_put_unlocked(obj);
-
- return 0;
-}
-
-bool cirrus_check_framebuffer(struct cirrus_device *cdev, int width, int height,
- int bpp, int pitch)
-{
- const int max_pitch = 0x1FF << 3; /* (4096 - 1) & ~111b bytes */
- const int max_size = cdev->mc.vram_size;
-
- if (bpp > cirrus_bpp)
- return false;
- if (bpp > 32)
- return false;
-
- if (pitch > max_pitch)
- return false;
-
- if (pitch * height > max_size)
- return false;
-
- return true;
-}
+++ /dev/null
-
-/*
- * Copyright 2012 Red Hat
- *
- * This file is subject to the terms and conditions of the GNU General
- * Public License version 2. See the file COPYING in the main
- * directory of this archive for more details.
- *
- * Authors: Matthew Garrett
- * Dave Airlie
- *
- * Portions of this code derived from cirrusfb.c:
- * drivers/video/cirrusfb.c - driver for Cirrus Logic chipsets
- *
- * Copyright 1999-2001 Jeff Garzik <jgarzik@pobox.com>
- */
-#include <drm/drmP.h>
-#include <drm/drm_crtc_helper.h>
-#include <drm/drm_plane_helper.h>
-#include <drm/drm_probe_helper.h>
-
-#include <video/cirrus.h>
-
-#include "cirrus_drv.h"
-
-#define CIRRUS_LUT_SIZE 256
-
-#define PALETTE_INDEX 0x8
-#define PALETTE_DATA 0x9
-
-/*
- * This file contains setup code for the CRTC.
- */
-
-/*
- * The DRM core requires DPMS functions, but they make little sense in our
- * case and so are just stubs
- */
-
-static void cirrus_crtc_dpms(struct drm_crtc *crtc, int mode)
-{
- struct drm_device *dev = crtc->dev;
- struct cirrus_device *cdev = dev->dev_private;
- u8 sr01, gr0e;
-
- switch (mode) {
- case DRM_MODE_DPMS_ON:
- sr01 = 0x00;
- gr0e = 0x00;
- break;
- case DRM_MODE_DPMS_STANDBY:
- sr01 = 0x20;
- gr0e = 0x02;
- break;
- case DRM_MODE_DPMS_SUSPEND:
- sr01 = 0x20;
- gr0e = 0x04;
- break;
- case DRM_MODE_DPMS_OFF:
- sr01 = 0x20;
- gr0e = 0x06;
- break;
- default:
- return;
- }
-
- WREG8(SEQ_INDEX, 0x1);
- sr01 |= RREG8(SEQ_DATA) & ~0x20;
- WREG_SEQ(0x1, sr01);
-
- WREG8(GFX_INDEX, 0xe);
- gr0e |= RREG8(GFX_DATA) & ~0x06;
- WREG_GFX(0xe, gr0e);
-}
-
-static void cirrus_set_start_address(struct drm_crtc *crtc, unsigned offset)
-{
- struct cirrus_device *cdev = crtc->dev->dev_private;
- u32 addr;
- u8 tmp;
-
- addr = offset >> 2;
- WREG_CRT(0x0c, (u8)((addr >> 8) & 0xff));
- WREG_CRT(0x0d, (u8)(addr & 0xff));
-
- WREG8(CRT_INDEX, 0x1b);
- tmp = RREG8(CRT_DATA);
- tmp &= 0xf2;
- tmp |= (addr >> 16) & 0x01;
- tmp |= (addr >> 15) & 0x0c;
- WREG_CRT(0x1b, tmp);
- WREG8(CRT_INDEX, 0x1d);
- tmp = RREG8(CRT_DATA);
- tmp &= 0x7f;
- tmp |= (addr >> 12) & 0x80;
- WREG_CRT(0x1d, tmp);
-}
-
-/* cirrus is different - we will force move buffers out of VRAM */
-static int cirrus_crtc_do_set_base(struct drm_crtc *crtc,
- struct drm_framebuffer *fb,
- int x, int y, int atomic)
-{
- struct cirrus_device *cdev = crtc->dev->dev_private;
- struct cirrus_bo *bo;
- int ret;
- u64 gpu_addr;
-
- /* push the previous fb to system ram */
- if (!atomic && fb) {
- bo = gem_to_cirrus_bo(fb->obj[0]);
- ret = cirrus_bo_reserve(bo, false);
- if (ret)
- return ret;
- cirrus_bo_push_sysram(bo);
- cirrus_bo_unreserve(bo);
- }
-
- bo = gem_to_cirrus_bo(crtc->primary->fb->obj[0]);
-
- ret = cirrus_bo_reserve(bo, false);
- if (ret)
- return ret;
-
- ret = cirrus_bo_pin(bo, TTM_PL_FLAG_VRAM, &gpu_addr);
- if (ret) {
- cirrus_bo_unreserve(bo);
- return ret;
- }
-
- if (cdev->mode_info.gfbdev->gfb == crtc->primary->fb) {
- /* if pushing console in kmap it */
- ret = ttm_bo_kmap(&bo->bo, 0, bo->bo.num_pages, &bo->kmap);
- if (ret)
- DRM_ERROR("failed to kmap fbcon\n");
- }
- cirrus_bo_unreserve(bo);
-
- cirrus_set_start_address(crtc, (u32)gpu_addr);
- return 0;
-}
-
-static int cirrus_crtc_mode_set_base(struct drm_crtc *crtc, int x, int y,
- struct drm_framebuffer *old_fb)
-{
- return cirrus_crtc_do_set_base(crtc, old_fb, x, y, 0);
-}
-
-/*
- * The meat of this driver. The core passes us a mode and we have to program
- * it. The modesetting here is the bare minimum required to satisfy the qemu
- * emulation of this hardware, and running this against a real device is
- * likely to result in an inadequately programmed mode. We've already had
- * the opportunity to modify the mode, so whatever we receive here should
- * be something that can be correctly programmed and displayed
- */
-static int cirrus_crtc_mode_set(struct drm_crtc *crtc,
- struct drm_display_mode *mode,
- struct drm_display_mode *adjusted_mode,
- int x, int y, struct drm_framebuffer *old_fb)
-{
- struct drm_device *dev = crtc->dev;
- struct cirrus_device *cdev = dev->dev_private;
- const struct drm_framebuffer *fb = crtc->primary->fb;
- int hsyncstart, hsyncend, htotal, hdispend;
- int vtotal, vdispend;
- int tmp;
- int sr07 = 0, hdr = 0;
-
- htotal = mode->htotal / 8;
- hsyncend = mode->hsync_end / 8;
- hsyncstart = mode->hsync_start / 8;
- hdispend = mode->hdisplay / 8;
-
- vtotal = mode->vtotal;
- vdispend = mode->vdisplay;
-
- vdispend -= 1;
- vtotal -= 2;
-
- htotal -= 5;
- hdispend -= 1;
- hsyncstart += 1;
- hsyncend += 1;
-
- WREG_CRT(VGA_CRTC_V_SYNC_END, 0x20);
- WREG_CRT(VGA_CRTC_H_TOTAL, htotal);
- WREG_CRT(VGA_CRTC_H_DISP, hdispend);
- WREG_CRT(VGA_CRTC_H_SYNC_START, hsyncstart);
- WREG_CRT(VGA_CRTC_H_SYNC_END, hsyncend);
- WREG_CRT(VGA_CRTC_V_TOTAL, vtotal & 0xff);
- WREG_CRT(VGA_CRTC_V_DISP_END, vdispend & 0xff);
-
- tmp = 0x40;
- if ((vdispend + 1) & 512)
- tmp |= 0x20;
- WREG_CRT(VGA_CRTC_MAX_SCAN, tmp);
-
- /*
- * Overflow bits for values that don't fit in the standard registers
- */
- tmp = 16;
- if (vtotal & 256)
- tmp |= 1;
- if (vdispend & 256)
- tmp |= 2;
- if ((vdispend + 1) & 256)
- tmp |= 8;
- if (vtotal & 512)
- tmp |= 32;
- if (vdispend & 512)
- tmp |= 64;
- WREG_CRT(VGA_CRTC_OVERFLOW, tmp);
-
- tmp = 0;
-
- /* More overflow bits */
-
- if ((htotal + 5) & 64)
- tmp |= 16;
- if ((htotal + 5) & 128)
- tmp |= 32;
- if (vtotal & 256)
- tmp |= 64;
- if (vtotal & 512)
- tmp |= 128;
-
- WREG_CRT(CL_CRT1A, tmp);
-
- /* Disable Hercules/CGA compatibility */
- WREG_CRT(VGA_CRTC_MODE, 0x03);
-
- WREG8(SEQ_INDEX, 0x7);
- sr07 = RREG8(SEQ_DATA);
- sr07 &= 0xe0;
- hdr = 0;
- switch (fb->format->cpp[0] * 8) {
- case 8:
- sr07 |= 0x11;
- break;
- case 16:
- sr07 |= 0x17;
- hdr = 0xc1;
- break;
- case 24:
- sr07 |= 0x15;
- hdr = 0xc5;
- break;
- case 32:
- sr07 |= 0x19;
- hdr = 0xc5;
- break;
- default:
- return -1;
- }
-
- WREG_SEQ(0x7, sr07);
-
- /* Program the pitch */
- tmp = fb->pitches[0] / 8;
- WREG_CRT(VGA_CRTC_OFFSET, tmp);
-
- /* Enable extended blanking and pitch bits, and enable full memory */
- tmp = 0x22;
- tmp |= (fb->pitches[0] >> 7) & 0x10;
- tmp |= (fb->pitches[0] >> 6) & 0x40;
- WREG_CRT(0x1b, tmp);
-
- /* Enable high-colour modes */
- WREG_GFX(VGA_GFX_MODE, 0x40);
-
- /* And set graphics mode */
- WREG_GFX(VGA_GFX_MISC, 0x01);
-
- WREG_HDR(hdr);
- cirrus_crtc_do_set_base(crtc, old_fb, x, y, 0);
-
- /* Unblank (needed on S3 resume, vgabios doesn't do it then) */
- outb(0x20, 0x3c0);
- return 0;
-}
-
-/*
- * This is called before a mode is programmed. A typical use might be to
- * enable DPMS during the programming to avoid seeing intermediate stages,
- * but that's not relevant to us
- */
-static void cirrus_crtc_prepare(struct drm_crtc *crtc)
-{
-}
-
-static void cirrus_crtc_load_lut(struct drm_crtc *crtc)
-{
- struct drm_device *dev = crtc->dev;
- struct cirrus_device *cdev = dev->dev_private;
- u16 *r, *g, *b;
- int i;
-
- if (!crtc->enabled)
- return;
-
- r = crtc->gamma_store;
- g = r + crtc->gamma_size;
- b = g + crtc->gamma_size;
-
- for (i = 0; i < CIRRUS_LUT_SIZE; i++) {
- /* VGA registers */
- WREG8(PALETTE_INDEX, i);
- WREG8(PALETTE_DATA, *r++ >> 8);
- WREG8(PALETTE_DATA, *g++ >> 8);
- WREG8(PALETTE_DATA, *b++ >> 8);
- }
-}
-
-/*
- * This is called after a mode is programmed. It should reverse anything done
- * by the prepare function
- */
-static void cirrus_crtc_commit(struct drm_crtc *crtc)
-{
- cirrus_crtc_load_lut(crtc);
-}
-
-/*
- * The core can pass us a set of gamma values to program. We actually only
- * use this for 8-bit mode so can't perform smooth fades on deeper modes,
- * but it's a requirement that we provide the function
- */
-static int cirrus_crtc_gamma_set(struct drm_crtc *crtc, u16 *red, u16 *green,
- u16 *blue, uint32_t size,
- struct drm_modeset_acquire_ctx *ctx)
-{
- cirrus_crtc_load_lut(crtc);
-
- return 0;
-}
-
-/* Simple cleanup function */
-static void cirrus_crtc_destroy(struct drm_crtc *crtc)
-{
- struct cirrus_crtc *cirrus_crtc = to_cirrus_crtc(crtc);
-
- drm_crtc_cleanup(crtc);
- kfree(cirrus_crtc);
-}
-
-/* These provide the minimum set of functions required to handle a CRTC */
-static const struct drm_crtc_funcs cirrus_crtc_funcs = {
- .gamma_set = cirrus_crtc_gamma_set,
- .set_config = drm_crtc_helper_set_config,
- .destroy = cirrus_crtc_destroy,
-};
-
-static const struct drm_crtc_helper_funcs cirrus_helper_funcs = {
- .dpms = cirrus_crtc_dpms,
- .mode_set = cirrus_crtc_mode_set,
- .mode_set_base = cirrus_crtc_mode_set_base,
- .prepare = cirrus_crtc_prepare,
- .commit = cirrus_crtc_commit,
-};
-
-/* CRTC setup */
-static const uint32_t cirrus_formats_16[] = {
- DRM_FORMAT_RGB565,
-};
-
-static const uint32_t cirrus_formats_24[] = {
- DRM_FORMAT_RGB888,
- DRM_FORMAT_RGB565,
-};
-
-static const uint32_t cirrus_formats_32[] = {
- DRM_FORMAT_XRGB8888,
- DRM_FORMAT_ARGB8888,
- DRM_FORMAT_RGB888,
- DRM_FORMAT_RGB565,
-};
-
-static struct drm_plane *cirrus_primary_plane(struct drm_device *dev)
-{
- const uint32_t *formats;
- uint32_t nformats;
- struct drm_plane *primary;
- int ret;
-
- switch (cirrus_bpp) {
- case 16:
- formats = cirrus_formats_16;
- nformats = ARRAY_SIZE(cirrus_formats_16);
- break;
- case 24:
- formats = cirrus_formats_24;
- nformats = ARRAY_SIZE(cirrus_formats_24);
- break;
- case 32:
- formats = cirrus_formats_32;
- nformats = ARRAY_SIZE(cirrus_formats_32);
- break;
- default:
- return NULL;
- }
-
- primary = kzalloc(sizeof(*primary), GFP_KERNEL);
- if (primary == NULL) {
- DRM_DEBUG_KMS("Failed to allocate primary plane\n");
- return NULL;
- }
-
- ret = drm_universal_plane_init(dev, primary, 0,
- &drm_primary_helper_funcs,
- formats, nformats,
- NULL,
- DRM_PLANE_TYPE_PRIMARY, NULL);
- if (ret) {
- kfree(primary);
- primary = NULL;
- }
-
- return primary;
-}
-
-static void cirrus_crtc_init(struct drm_device *dev)
-{
- struct cirrus_device *cdev = dev->dev_private;
- struct cirrus_crtc *cirrus_crtc;
- struct drm_plane *primary;
-
- cirrus_crtc = kzalloc(sizeof(struct cirrus_crtc) +
- (CIRRUSFB_CONN_LIMIT * sizeof(struct drm_connector *)),
- GFP_KERNEL);
-
- if (cirrus_crtc == NULL)
- return;
-
- primary = cirrus_primary_plane(dev);
- if (primary == NULL) {
- kfree(cirrus_crtc);
- return;
- }
-
- drm_crtc_init_with_planes(dev, &cirrus_crtc->base,
- primary, NULL,
- &cirrus_crtc_funcs, NULL);
-
- drm_mode_crtc_set_gamma_size(&cirrus_crtc->base, CIRRUS_LUT_SIZE);
- cdev->mode_info.crtc = cirrus_crtc;
-
- drm_crtc_helper_add(&cirrus_crtc->base, &cirrus_helper_funcs);
-}
-
-static void cirrus_encoder_mode_set(struct drm_encoder *encoder,
- struct drm_display_mode *mode,
- struct drm_display_mode *adjusted_mode)
-{
-}
-
-static void cirrus_encoder_dpms(struct drm_encoder *encoder, int state)
-{
- return;
-}
-
-static void cirrus_encoder_prepare(struct drm_encoder *encoder)
-{
-}
-
-static void cirrus_encoder_commit(struct drm_encoder *encoder)
-{
-}
-
-static void cirrus_encoder_destroy(struct drm_encoder *encoder)
-{
- struct cirrus_encoder *cirrus_encoder = to_cirrus_encoder(encoder);
- drm_encoder_cleanup(encoder);
- kfree(cirrus_encoder);
-}
-
-static const struct drm_encoder_helper_funcs cirrus_encoder_helper_funcs = {
- .dpms = cirrus_encoder_dpms,
- .mode_set = cirrus_encoder_mode_set,
- .prepare = cirrus_encoder_prepare,
- .commit = cirrus_encoder_commit,
-};
-
-static const struct drm_encoder_funcs cirrus_encoder_encoder_funcs = {
- .destroy = cirrus_encoder_destroy,
-};
-
-static struct drm_encoder *cirrus_encoder_init(struct drm_device *dev)
-{
- struct drm_encoder *encoder;
- struct cirrus_encoder *cirrus_encoder;
-
- cirrus_encoder = kzalloc(sizeof(struct cirrus_encoder), GFP_KERNEL);
- if (!cirrus_encoder)
- return NULL;
-
- encoder = &cirrus_encoder->base;
- encoder->possible_crtcs = 0x1;
-
- drm_encoder_init(dev, encoder, &cirrus_encoder_encoder_funcs,
- DRM_MODE_ENCODER_DAC, NULL);
- drm_encoder_helper_add(encoder, &cirrus_encoder_helper_funcs);
-
- return encoder;
-}
-
-
-static int cirrus_vga_get_modes(struct drm_connector *connector)
-{
- int count;
-
- /* Just add a static list of modes */
- if (cirrus_bpp <= 24) {
- count = drm_add_modes_noedid(connector, 1280, 1024);
- drm_set_preferred_mode(connector, 1024, 768);
- } else {
- count = drm_add_modes_noedid(connector, 800, 600);
- drm_set_preferred_mode(connector, 800, 600);
- }
- return count;
-}
-
-static struct drm_encoder *cirrus_connector_best_encoder(struct drm_connector
- *connector)
-{
- int enc_id = connector->encoder_ids[0];
- /* pick the encoder ids */
- if (enc_id)
- return drm_encoder_find(connector->dev, NULL, enc_id);
- return NULL;
-}
-
-static void cirrus_connector_destroy(struct drm_connector *connector)
-{
- drm_connector_cleanup(connector);
- kfree(connector);
-}
-
-static const struct drm_connector_helper_funcs cirrus_vga_connector_helper_funcs = {
- .get_modes = cirrus_vga_get_modes,
- .best_encoder = cirrus_connector_best_encoder,
-};
-
-static const struct drm_connector_funcs cirrus_vga_connector_funcs = {
- .dpms = drm_helper_connector_dpms,
- .fill_modes = drm_helper_probe_single_connector_modes,
- .destroy = cirrus_connector_destroy,
-};
-
-static struct drm_connector *cirrus_vga_init(struct drm_device *dev)
-{
- struct drm_connector *connector;
- struct cirrus_connector *cirrus_connector;
-
- cirrus_connector = kzalloc(sizeof(struct cirrus_connector), GFP_KERNEL);
- if (!cirrus_connector)
- return NULL;
-
- connector = &cirrus_connector->base;
-
- drm_connector_init(dev, connector,
- &cirrus_vga_connector_funcs, DRM_MODE_CONNECTOR_VGA);
-
- drm_connector_helper_add(connector, &cirrus_vga_connector_helper_funcs);
-
- drm_connector_register(connector);
- return connector;
-}
-
-
-int cirrus_modeset_init(struct cirrus_device *cdev)
-{
- struct drm_encoder *encoder;
- struct drm_connector *connector;
- int ret;
-
- drm_mode_config_init(cdev->dev);
- cdev->mode_info.mode_config_initialized = true;
-
- cdev->dev->mode_config.max_width = CIRRUS_MAX_FB_WIDTH;
- cdev->dev->mode_config.max_height = CIRRUS_MAX_FB_HEIGHT;
-
- cdev->dev->mode_config.fb_base = cdev->mc.vram_base;
- cdev->dev->mode_config.preferred_depth = cirrus_bpp;
- /* don't prefer a shadow on virt GPU */
- cdev->dev->mode_config.prefer_shadow = 0;
-
- cirrus_crtc_init(cdev->dev);
-
- encoder = cirrus_encoder_init(cdev->dev);
- if (!encoder) {
- DRM_ERROR("cirrus_encoder_init failed\n");
- return -1;
- }
-
- connector = cirrus_vga_init(cdev->dev);
- if (!connector) {
- DRM_ERROR("cirrus_vga_init failed\n");
- return -1;
- }
-
- drm_connector_attach_encoder(connector, encoder);
-
- ret = cirrus_fbdev_init(cdev);
- if (ret) {
- DRM_ERROR("cirrus_fbdev_init failed\n");
- return ret;
- }
-
- return 0;
-}
-
-void cirrus_modeset_fini(struct cirrus_device *cdev)
-{
- cirrus_fbdev_fini(cdev);
-
- if (cdev->mode_info.mode_config_initialized) {
- drm_mode_config_cleanup(cdev->dev);
- cdev->mode_info.mode_config_initialized = false;
- }
-}
ret = ttm_bo_device_init(&cirrus->ttm.bdev,
&cirrus_bo_driver,
dev->anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET,
true);
if (ret) {
DRM_ERROR("Error initialising bo driver; %d\n", ret);
int cirrus_mmap(struct file *filp, struct vm_area_struct *vma)
{
- struct drm_file *file_priv;
- struct cirrus_device *cirrus;
+ struct drm_file *file_priv = filp->private_data;
+ struct cirrus_device *cirrus = file_priv->minor->dev->dev_private;
- if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET))
- return -EINVAL;
-
- file_priv = filp->private_data;
- cirrus = file_priv->minor->dev->dev_private;
return ttm_bo_mmap(filp, vma, &cirrus->ttm.bdev);
}
}
EXPORT_SYMBOL(drm_atomic_get_private_obj_state);
+/**
+ * drm_atomic_get_old_private_obj_state
+ * @state: global atomic state object
+ * @obj: private_obj to grab
+ *
+ * This function returns the old private object state for the given private_obj,
+ * or NULL if the private_obj is not part of the global atomic state.
+ */
+struct drm_private_state *
+drm_atomic_get_old_private_obj_state(struct drm_atomic_state *state,
+ struct drm_private_obj *obj)
+{
+ int i;
+
+ for (i = 0; i < state->num_private_objs; i++)
+ if (obj == state->private_objs[i].ptr)
+ return state->private_objs[i].old_state;
+
+ return NULL;
+}
+EXPORT_SYMBOL(drm_atomic_get_old_private_obj_state);
+
+/**
+ * drm_atomic_get_new_private_obj_state
+ * @state: global atomic state object
+ * @obj: private_obj to grab
+ *
+ * This function returns the new private object state for the given private_obj,
+ * or NULL if the private_obj is not part of the global atomic state.
+ */
+struct drm_private_state *
+drm_atomic_get_new_private_obj_state(struct drm_atomic_state *state,
+ struct drm_private_obj *obj)
+{
+ int i;
+
+ for (i = 0; i < state->num_private_objs; i++)
+ if (obj == state->private_objs[i].ptr)
+ return state->private_objs[i].new_state;
+
+ return NULL;
+}
+EXPORT_SYMBOL(drm_atomic_get_new_private_obj_state);
+
/**
* drm_atomic_get_connector_state - get connector state
* @state: global atomic state object
minor->debugfs_root, minor);
}
#endif
-
static enum drm_mode_status mode_valid_path(struct drm_connector *connector,
struct drm_encoder *encoder,
struct drm_crtc *crtc,
- struct drm_display_mode *mode)
+ const struct drm_display_mode *mode)
{
enum drm_mode_status ret;
struct drm_crtc *crtc = conn_state->crtc;
struct drm_crtc_state *crtc_state;
enum drm_mode_status mode_status;
- struct drm_display_mode *mode;
+ const struct drm_display_mode *mode;
if (!crtc || !encoder)
continue;
*
* NOTE: Commit work has multiple phases, first hardware commit, then
* cleanup. We want them to overlap, hence need system_unbound_wq to
- * make sure work items don't artifically stall on each another.
+ * make sure work items don't artificially stall on each another.
*/
drm_atomic_state_get(state);
*
* Asynchronous workers need to have sufficient parallelism to be able to run
* different atomic commits on different CRTCs in parallel. The simplest way to
- * achive this is by running them on the &system_unbound_wq work queue. Note
+ * achieve this is by running them on the &system_unbound_wq work queue. Note
* that drivers are not required to split up atomic commits and run an
* individual commit in parallel - userspace is supposed to do that if it cares.
* But it might be beneficial to do that for modesets, since those necessarily
int drm_atomic_helper_prepare_planes(struct drm_device *dev,
struct drm_atomic_state *state)
{
+ struct drm_connector *connector;
+ struct drm_connector_state *new_conn_state;
struct drm_plane *plane;
struct drm_plane_state *new_plane_state;
int ret, i, j;
+ for_each_new_connector_in_state(state, connector, new_conn_state, i) {
+ if (!new_conn_state->writeback_job)
+ continue;
+
+ ret = drm_writeback_prepare_job(new_conn_state->writeback_job);
+ if (ret < 0)
+ return ret;
+ }
+
for_each_new_plane_in_state(state, plane, new_plane_state, i) {
const struct drm_plane_helper_funcs *funcs;
#include <drm/drm_connector.h>
#include <drm/drm_atomic.h>
#include <drm/drm_device.h>
+#include <drm/drm_writeback.h>
#include <linux/slab.h>
#include <linux/dma-fence.h>
if (state->commit)
drm_crtc_commit_put(state->commit);
+
+ if (state->writeback_job)
+ drm_writeback_cleanup_job(state->writeback_job);
}
EXPORT_SYMBOL(__drm_atomic_helper_connector_destroy_state);
}
static int drm_atomic_plane_set_property(struct drm_plane *plane,
- struct drm_plane_state *state, struct drm_property *property,
- uint64_t val)
+ struct drm_plane_state *state, struct drm_file *file_priv,
+ struct drm_property *property, uint64_t val)
{
struct drm_device *dev = plane->dev;
struct drm_mode_config *config = &dev->mode_config;
int ret;
if (property == config->prop_fb_id) {
- struct drm_framebuffer *fb = drm_framebuffer_lookup(dev, NULL, val);
+ struct drm_framebuffer *fb;
+ fb = drm_framebuffer_lookup(dev, file_priv, val);
drm_atomic_set_fb_for_plane(state, fb);
if (fb)
drm_framebuffer_put(fb);
return -EINVAL;
} else if (property == config->prop_crtc_id) {
- struct drm_crtc *crtc = drm_crtc_find(dev, NULL, val);
+ struct drm_crtc *crtc = drm_crtc_find(dev, file_priv, val);
+ if (val && !crtc)
+ return -EACCES;
return drm_atomic_set_crtc_for_plane(state, crtc);
} else if (property == config->prop_crtc_x) {
state->crtc_x = U642I64(val);
return 0;
}
-static struct drm_writeback_job *
-drm_atomic_get_writeback_job(struct drm_connector_state *conn_state)
-{
- WARN_ON(conn_state->connector->connector_type != DRM_MODE_CONNECTOR_WRITEBACK);
-
- if (!conn_state->writeback_job)
- conn_state->writeback_job =
- kzalloc(sizeof(*conn_state->writeback_job), GFP_KERNEL);
-
- return conn_state->writeback_job;
-}
-
static int drm_atomic_set_writeback_fb_for_connector(
struct drm_connector_state *conn_state,
struct drm_framebuffer *fb)
{
- struct drm_writeback_job *job =
- drm_atomic_get_writeback_job(conn_state);
- if (!job)
- return -ENOMEM;
+ int ret;
- drm_framebuffer_assign(&job->fb, fb);
+ ret = drm_writeback_set_fb(conn_state, fb);
+ if (ret < 0)
+ return ret;
if (fb)
DRM_DEBUG_ATOMIC("Set [FB:%d] for connector state %p\n",
}
static int drm_atomic_connector_set_property(struct drm_connector *connector,
- struct drm_connector_state *state, struct drm_property *property,
- uint64_t val)
+ struct drm_connector_state *state, struct drm_file *file_priv,
+ struct drm_property *property, uint64_t val)
{
struct drm_device *dev = connector->dev;
struct drm_mode_config *config = &dev->mode_config;
if (property == config->prop_crtc_id) {
- struct drm_crtc *crtc = drm_crtc_find(dev, NULL, val);
+ struct drm_crtc *crtc = drm_crtc_find(dev, file_priv, val);
+ if (val && !crtc)
+ return -EACCES;
return drm_atomic_set_crtc_for_connector(state, crtc);
} else if (property == config->dpms_property) {
/* setting DPMS property requires special handling, which
return -EINVAL;
}
state->content_protection = val;
+ } else if (property == connector->colorspace_property) {
+ state->colorspace = val;
} else if (property == config->writeback_fb_id_property) {
- struct drm_framebuffer *fb = drm_framebuffer_lookup(dev, NULL, val);
- int ret = drm_atomic_set_writeback_fb_for_connector(state, fb);
+ struct drm_framebuffer *fb;
+ int ret;
+ fb = drm_framebuffer_lookup(dev, file_priv, val);
+ ret = drm_atomic_set_writeback_fb_for_connector(state, fb);
if (fb)
drm_framebuffer_put(fb);
return ret;
*val = state->picture_aspect_ratio;
} else if (property == config->content_type_property) {
*val = state->content_type;
+ } else if (property == connector->colorspace_property) {
+ *val = state->colorspace;
} else if (property == connector->scaling_mode_property) {
*val = state->scaling_mode;
} else if (property == connector->content_protection_property) {
}
int drm_atomic_set_property(struct drm_atomic_state *state,
+ struct drm_file *file_priv,
struct drm_mode_object *obj,
struct drm_property *prop,
uint64_t prop_value)
}
ret = drm_atomic_connector_set_property(connector,
- connector_state, prop, prop_value);
+ connector_state, file_priv,
+ prop, prop_value);
break;
}
case DRM_MODE_OBJECT_CRTC: {
}
ret = drm_atomic_plane_set_property(plane,
- plane_state, prop, prop_value);
+ plane_state, file_priv,
+ prop, prop_value);
break;
}
default:
for_each_new_connector_in_state(state, conn, conn_state, i) {
struct drm_writeback_connector *wb_conn;
- struct drm_writeback_job *job;
struct drm_out_fence_state *f;
struct dma_fence *fence;
s32 __user *fence_ptr;
+ if (!conn_state->writeback_job)
+ continue;
+
fence_ptr = get_out_fence_for_connector(state, conn);
if (!fence_ptr)
continue;
- job = drm_atomic_get_writeback_job(conn_state);
- if (!job)
- return -ENOMEM;
-
f = krealloc(*fence_state, sizeof(**fence_state) *
(*num_fences + 1), GFP_KERNEL);
if (!f)
return ret;
}
- job->out_fence = fence;
+ conn_state->writeback_job->out_fence = fence;
}
/*
goto out;
}
- ret = drm_atomic_set_property(state, obj, prop,
- prop_value);
+ ret = drm_atomic_set_property(state, file_priv,
+ obj, prop, prop_value);
if (ret) {
drm_mode_object_put(obj);
goto out;
return NULL;
kref_init(&master->refcount);
- spin_lock_init(&master->lock.spinlock);
- init_waitqueue_head(&master->lock.lock_queue);
+ drm_master_legacy_init(master);
idr_init(&master->magic_map);
master->dev = dev;
/* initialize the tree of output resource lessees */
- master->lessor = NULL;
- master->lessee_id = 0;
INIT_LIST_HEAD(&master->lessees);
INIT_LIST_HEAD(&master->lessee_list);
idr_init(&master->leases);
if (!drm_is_current_master(file_priv))
goto out;
- if (drm_core_check_feature(dev, DRIVER_LEGACY)) {
- /*
- * Since the master is disappearing, so is the
- * possibility to lock.
- */
- mutex_lock(&dev->struct_mutex);
- if (master->lock.hw_lock) {
- if (dev->sigdata.lock == master->lock.hw_lock)
- dev->sigdata.lock = NULL;
- master->lock.hw_lock = NULL;
- master->lock.file_priv = NULL;
- wake_up_interruptible_all(&master->lock.lock_queue);
- }
- mutex_unlock(&dev->struct_mutex);
- }
+ drm_legacy_lock_master_cleanup(dev, master);
if (dev->master == file_priv->master)
drm_drop_master(dev, file_priv);
mutex_unlock(&dev->struct_mutex);
}
+void drm_legacy_rmmaps(struct drm_device *dev)
+{
+ struct drm_map_list *r_list, *list_temp;
+
+ list_for_each_entry_safe(r_list, list_temp, &dev->maplist, head)
+ drm_legacy_rmmap(dev, r_list->map);
+}
+
/* The rmmap ioctl appears to be unnecessary. All mappings are torn down on
* the last close of the device, and this is necessary for cleanup when things
* exit uncleanly. Therefore, having userland manually remove mappings seems
* @name: Client name
* @funcs: DRM client functions (optional)
*
- * This initialises the client and opens a &drm_file. Use drm_client_add() to complete the process.
+ * This initialises the client and opens a &drm_file.
+ * Use drm_client_register() to complete the process.
* The caller needs to hold a reference on @dev before calling this function.
* The client is freed when the &drm_device is unregistered. See drm_client_release().
*
EXPORT_SYMBOL(drm_client_init);
/**
- * drm_client_add - Add client to the device list
+ * drm_client_register - Register client
* @client: DRM client
*
* Add the client to the &drm_device client list to activate its callbacks.
* @client must be initialized by a call to drm_client_init(). After
- * drm_client_add() it is no longer permissible to call drm_client_release()
+ * drm_client_register() it is no longer permissible to call drm_client_release()
* directly (outside the unregister callback), instead cleanup will happen
* automatically on driver unload.
*/
-void drm_client_add(struct drm_client_dev *client)
+void drm_client_register(struct drm_client_dev *client)
{
struct drm_device *dev = client->dev;
list_add(&client->list, &dev->clientlist);
mutex_unlock(&dev->clientlist_mutex);
}
-EXPORT_SYMBOL(drm_client_add);
+EXPORT_SYMBOL(drm_client_register);
/**
* drm_client_release - Release DRM client resources
INIT_LIST_HEAD(&connector->modes);
mutex_init(&connector->mutex);
connector->edid_blob_ptr = NULL;
+ connector->tile_blob_ptr = NULL;
connector->status = connector_status_unknown;
connector->display_info.panel_orientation =
DRM_MODE_PANEL_ORIENTATION_UNKNOWN;
drm_object_attach_property(&connector->base,
config->non_desktop_property,
0);
+ drm_object_attach_property(&connector->base,
+ config->tile_property,
+ 0);
if (drm_core_check_feature(dev, DRIVER_ATOMIC)) {
drm_object_attach_property(&connector->base, config->prop_crtc_id, 0);
};
DRM_ENUM_NAME_FN(drm_get_content_protection_name, drm_cp_enum_list)
+static const struct drm_prop_enum_list hdmi_colorspaces[] = {
+ /* For Default case, driver will set the colorspace */
+ { DRM_MODE_COLORIMETRY_DEFAULT, "Default" },
+ /* Standard Definition Colorimetry based on CEA 861 */
+ { DRM_MODE_COLORIMETRY_SMPTE_170M_YCC, "SMPTE_170M_YCC" },
+ { DRM_MODE_COLORIMETRY_BT709_YCC, "BT709_YCC" },
+ /* Standard Definition Colorimetry based on IEC 61966-2-4 */
+ { DRM_MODE_COLORIMETRY_XVYCC_601, "XVYCC_601" },
+ /* High Definition Colorimetry based on IEC 61966-2-4 */
+ { DRM_MODE_COLORIMETRY_XVYCC_709, "XVYCC_709" },
+ /* Colorimetry based on IEC 61966-2-1/Amendment 1 */
+ { DRM_MODE_COLORIMETRY_SYCC_601, "SYCC_601" },
+ /* Colorimetry based on IEC 61966-2-5 [33] */
+ { DRM_MODE_COLORIMETRY_OPYCC_601, "opYCC_601" },
+ /* Colorimetry based on IEC 61966-2-5 */
+ { DRM_MODE_COLORIMETRY_OPRGB, "opRGB" },
+ /* Colorimetry based on ITU-R BT.2020 */
+ { DRM_MODE_COLORIMETRY_BT2020_CYCC, "BT2020_CYCC" },
+ /* Colorimetry based on ITU-R BT.2020 */
+ { DRM_MODE_COLORIMETRY_BT2020_RGB, "BT2020_RGB" },
+ /* Colorimetry based on ITU-R BT.2020 */
+ { DRM_MODE_COLORIMETRY_BT2020_YCC, "BT2020_YCC" },
+ /* Added as part of Additional Colorimetry Extension in 861.G */
+ { DRM_MODE_COLORIMETRY_DCI_P3_RGB_D65, "DCI-P3_RGB_D65" },
+ { DRM_MODE_COLORIMETRY_DCI_P3_RGB_THEATER, "DCI-P3_RGB_Theater" },
+};
+
/**
* DOC: standard connector properties
*
*
* The driver may place further restrictions within these minimum
* and maximum bounds.
- *
- * The semantics for the vertical blank timestamp differ when
- * variable refresh rate is active. The vertical blank timestamp
- * is defined to be an estimate using the current mode's fixed
- * refresh rate timings. The semantics for the page-flip event
- * timestamp remain the same.
*/
/**
}
EXPORT_SYMBOL(drm_mode_create_aspect_ratio_property);
+/**
+ * DOC: standard connector properties
+ *
+ * Colorspace:
+ * drm_mode_create_colorspace_property - create colorspace property
+ * This property helps select a suitable colorspace based on the sink
+ * capability. Modern sink devices support wider gamut like BT2020.
+ * This helps switch to BT2020 mode if the BT2020 encoded video stream
+ * is being played by the user, same for any other colorspace. Thereby
+ * giving a good visual experience to users.
+ *
+ * The expectation from userspace is that it should parse the EDID
+ * and get supported colorspaces. Use this property and switch to the
+ * one supported. Sink supported colorspaces should be retrieved by
+ * userspace from EDID and driver will not explicitly expose them.
+ *
+ * Basically the expectation from userspace is:
+ * - Set up CRTC DEGAMMA/CTM/GAMMA to convert to some sink
+ * colorspace
+ * - Set this new property to let the sink know what it
+ * converted the CRTC output to.
+ * - This property is just to inform sink what colorspace
+ * source is trying to drive.
+ *
+ * Called by a driver the first time it's needed, must be attached to desired
+ * connectors.
+ */
+int drm_mode_create_colorspace_property(struct drm_connector *connector)
+{
+ struct drm_device *dev = connector->dev;
+ struct drm_property *prop;
+
+ if (connector->connector_type == DRM_MODE_CONNECTOR_HDMIA ||
+ connector->connector_type == DRM_MODE_CONNECTOR_HDMIB) {
+ prop = drm_property_create_enum(dev, DRM_MODE_PROP_ENUM,
+ "Colorspace",
+ hdmi_colorspaces,
+ ARRAY_SIZE(hdmi_colorspaces));
+ if (!prop)
+ return -ENOMEM;
+ } else {
+ DRM_DEBUG_KMS("Colorspace property not supported\n");
+ return 0;
+ }
+
+ connector->colorspace_property = prop;
+
+ return 0;
+}
+EXPORT_SYMBOL(drm_mode_create_colorspace_property);
+
/**
* drm_mode_create_content_type_property - create content type property
* @dev: DRM device
* This looks up the tile information for a connector, and creates a
* property for userspace to parse if it exists. The property is of
* the form of 8 integers using ':' as a separator.
+ * This is used for dual port tiled displays with DisplayPort SST
+ * or DisplayPort MST connectors.
*
* Returns:
* Zero on success, errno on failure.
*
* This function creates a new blob modeset object and assigns its id to the
* connector's edid property.
+ * Since we also parse tile information from EDID's displayID block, we also
+ * set the connector's tile property here. See drm_connector_set_tile_property()
+ * for more details.
*
* Returns:
* Zero on success, negative errno on failure.
edid,
&connector->base,
dev->mode_config.edid_property);
- return ret;
+ if (ret)
+ return ret;
+ return drm_connector_set_tile_property(connector);
}
EXPORT_SYMBOL(drm_connector_update_edid_property);
plane = crtc->primary;
+ /* allow disabling with the primary plane leased */
+ if (crtc_req->mode_valid && !drm_lease_held(file_priv, plane->base.id))
+ return -EACCES;
+
mutex_lock(&crtc->dev->mode_config.mutex);
DRM_MODESET_LOCK_ALL_BEGIN(dev, ctx,
DRM_MODESET_ACQUIRE_INTERRUPTIBLE, ret);
struct drm_connector *connector,
int mode);
int drm_atomic_set_property(struct drm_atomic_state *state,
+ struct drm_file *file_priv,
struct drm_mode_object *obj,
struct drm_property *prop,
uint64_t prop_value);
edid = drm_edid_duplicate(port->cached_edid);
else {
edid = drm_get_edid(connector, &port->aux.ddc);
- drm_connector_set_tile_property(connector);
}
port->has_audio = drm_detect_monitor_audio(edid);
drm_dp_mst_topology_put_port(port);
* Note that the lifetime rules for &drm_device instance has still a lot of
* historical baggage. Hence use the reference counting provided by
* drm_dev_get() and drm_dev_put() only carefully.
+ *
+ * Display driver example
+ * ~~~~~~~~~~~~~~~~~~~~~~
+ *
+ * The following example shows a typical structure of a DRM display driver.
+ * The example focus on the probe() function and the other functions that is
+ * almost always present and serves as a demonstration of devm_drm_dev_init()
+ * usage with its accompanying drm_driver->release callback.
+ *
+ * .. code-block:: c
+ *
+ * struct driver_device {
+ * struct drm_device drm;
+ * void *userspace_facing;
+ * struct clk *pclk;
+ * };
+ *
+ * static void driver_drm_release(struct drm_device *drm)
+ * {
+ * struct driver_device *priv = container_of(...);
+ *
+ * drm_mode_config_cleanup(drm);
+ * drm_dev_fini(drm);
+ * kfree(priv->userspace_facing);
+ * kfree(priv);
+ * }
+ *
+ * static struct drm_driver driver_drm_driver = {
+ * [...]
+ * .release = driver_drm_release,
+ * };
+ *
+ * static int driver_probe(struct platform_device *pdev)
+ * {
+ * struct driver_device *priv;
+ * struct drm_device *drm;
+ * int ret;
+ *
+ * [
+ * devm_kzalloc() can't be used here because the drm_device
+ * lifetime can exceed the device lifetime if driver unbind
+ * happens when userspace still has open file descriptors.
+ * ]
+ * priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+ * if (!priv)
+ * return -ENOMEM;
+ *
+ * drm = &priv->drm;
+ *
+ * ret = devm_drm_dev_init(&pdev->dev, drm, &driver_drm_driver);
+ * if (ret) {
+ * kfree(drm);
+ * return ret;
+ * }
+ *
+ * drm_mode_config_init(drm);
+ *
+ * priv->userspace_facing = kzalloc(..., GFP_KERNEL);
+ * if (!priv->userspace_facing)
+ * return -ENOMEM;
+ *
+ * priv->pclk = devm_clk_get(dev, "PCLK");
+ * if (IS_ERR(priv->pclk))
+ * return PTR_ERR(priv->pclk);
+ *
+ * [ Further setup, display pipeline etc ]
+ *
+ * platform_set_drvdata(pdev, drm);
+ *
+ * drm_mode_config_reset(drm);
+ *
+ * ret = drm_dev_register(drm);
+ * if (ret)
+ * return ret;
+ *
+ * drm_fbdev_generic_setup(drm, 32);
+ *
+ * return 0;
+ * }
+ *
+ * [ This function is called before the devm_ resources are released ]
+ * static int driver_remove(struct platform_device *pdev)
+ * {
+ * struct drm_device *drm = platform_get_drvdata(pdev);
+ *
+ * drm_dev_unregister(drm);
+ * drm_atomic_helper_shutdown(drm)
+ *
+ * return 0;
+ * }
+ *
+ * [ This function is called on kernel restart and shutdown ]
+ * static void driver_shutdown(struct platform_device *pdev)
+ * {
+ * drm_atomic_helper_shutdown(platform_get_drvdata(pdev));
+ * }
+ *
+ * static int __maybe_unused driver_pm_suspend(struct device *dev)
+ * {
+ * return drm_mode_config_helper_suspend(dev_get_drvdata(dev));
+ * }
+ *
+ * static int __maybe_unused driver_pm_resume(struct device *dev)
+ * {
+ * drm_mode_config_helper_resume(dev_get_drvdata(dev));
+ *
+ * return 0;
+ * }
+ *
+ * static const struct dev_pm_ops driver_pm_ops = {
+ * SET_SYSTEM_SLEEP_PM_OPS(driver_pm_suspend, driver_pm_resume)
+ * };
+ *
+ * static struct platform_driver driver_driver = {
+ * .driver = {
+ * [...]
+ * .pm = &driver_pm_ops,
+ * },
+ * .probe = driver_probe,
+ * .remove = driver_remove,
+ * .shutdown = driver_shutdown,
+ * };
+ * module_platform_driver(driver_driver);
+ *
+ * Drivers that want to support device unplugging (USB, DT overlay unload) should
+ * use drm_dev_unplug() instead of drm_dev_unregister(). The driver must protect
+ * regions that is accessing device resources to prevent use after they're
+ * released. This is done using drm_dev_enter() and drm_dev_exit(). There is one
+ * shortcoming however, drm_dev_unplug() marks the drm_device as unplugged before
+ * drm_atomic_helper_shutdown() is called. This means that if the disable code
+ * paths are protected, they will not run on regular driver module unload,
+ * possibily leaving the hardware enabled.
*/
/**
synchronize_srcu(&drm_unplug_srcu);
drm_dev_unregister(dev);
- drm_dev_put(dev);
}
EXPORT_SYMBOL(drm_dev_unplug);
}
}
+/**
+ * DOC: component helper usage recommendations
+ *
+ * DRM drivers that drive hardware where a logical device consists of a pile of
+ * independent hardware blocks are recommended to use the :ref:`component helper
+ * library<component>`. For consistency and better options for code reuse the
+ * following guidelines apply:
+ *
+ * - The entire device initialization procedure should be run from the
+ * &component_master_ops.master_bind callback, starting with drm_dev_init(),
+ * then binding all components with component_bind_all() and finishing with
+ * drm_dev_register().
+ *
+ * - The opaque pointer passed to all components through component_bind_all()
+ * should point at &struct drm_device of the device instance, not some driver
+ * specific private structure.
+ *
+ * - The component helper fills the niche where further standardization of
+ * interfaces is not practical. When there already is, or will be, a
+ * standardized interface like &drm_bridge or &drm_panel, providing its own
+ * functions to find such components at driver load time, like
+ * drm_of_find_panel_or_bridge(), then the component helper should not be
+ * used.
+ */
+
/**
* drm_dev_init - Initialise new DRM device
* @dev: DRM device
BUG_ON(!parent);
kref_init(&dev->ref);
- dev->dev = parent;
+ dev->dev = get_device(parent);
dev->driver = driver;
/* no per-device feature limits by default */
dev->driver_features = ~0u;
+ drm_legacy_init_members(dev);
INIT_LIST_HEAD(&dev->filelist);
INIT_LIST_HEAD(&dev->filelist_internal);
INIT_LIST_HEAD(&dev->clientlist);
- INIT_LIST_HEAD(&dev->ctxlist);
- INIT_LIST_HEAD(&dev->vmalist);
- INIT_LIST_HEAD(&dev->maplist);
INIT_LIST_HEAD(&dev->vblank_event_list);
- spin_lock_init(&dev->buf_lock);
spin_lock_init(&dev->event_lock);
mutex_init(&dev->struct_mutex);
mutex_init(&dev->filelist_mutex);
mutex_init(&dev->clientlist_mutex);
- mutex_init(&dev->ctxlist_mutex);
mutex_init(&dev->master_mutex);
dev->anon_inode = drm_fs_inode_new();
if (ret)
goto err_minors;
- ret = drm_ht_create(&dev->map_hash, 12);
+ ret = drm_legacy_create_map_hash(dev);
if (ret)
goto err_minors;
drm_gem_destroy(dev);
err_ctxbitmap:
drm_legacy_ctxbitmap_cleanup(dev);
- drm_ht_remove(&dev->map_hash);
+ drm_legacy_remove_map_hash(dev);
err_minors:
drm_minor_free(dev, DRM_MINOR_PRIMARY);
drm_minor_free(dev, DRM_MINOR_RENDER);
drm_fs_inode_free(dev->anon_inode);
err_free:
+ put_device(dev->dev);
mutex_destroy(&dev->master_mutex);
- mutex_destroy(&dev->ctxlist_mutex);
mutex_destroy(&dev->clientlist_mutex);
mutex_destroy(&dev->filelist_mutex);
mutex_destroy(&dev->struct_mutex);
+ drm_legacy_destroy_members(dev);
return ret;
}
EXPORT_SYMBOL(drm_dev_init);
+static void devm_drm_dev_init_release(void *data)
+{
+ drm_dev_put(data);
+}
+
+/**
+ * devm_drm_dev_init - Resource managed drm_dev_init()
+ * @parent: Parent device object
+ * @dev: DRM device
+ * @driver: DRM driver
+ *
+ * Managed drm_dev_init(). The DRM device initialized with this function is
+ * automatically put on driver detach using drm_dev_put(). You must supply a
+ * &drm_driver.release callback to control the finalization explicitly.
+ *
+ * RETURNS:
+ * 0 on success, or error code on failure.
+ */
+int devm_drm_dev_init(struct device *parent,
+ struct drm_device *dev,
+ struct drm_driver *driver)
+{
+ int ret;
+
+ if (WARN_ON(!parent || !driver->release))
+ return -EINVAL;
+
+ ret = drm_dev_init(dev, driver, parent);
+ if (ret)
+ return ret;
+
+ ret = devm_add_action(parent, devm_drm_dev_init_release, dev);
+ if (ret)
+ devm_drm_dev_init_release(dev);
+
+ return ret;
+}
+EXPORT_SYMBOL(devm_drm_dev_init);
+
/**
* drm_dev_fini - Finalize a dead DRM device
* @dev: DRM device
drm_gem_destroy(dev);
drm_legacy_ctxbitmap_cleanup(dev);
- drm_ht_remove(&dev->map_hash);
+ drm_legacy_remove_map_hash(dev);
drm_fs_inode_free(dev->anon_inode);
drm_minor_free(dev, DRM_MINOR_PRIMARY);
drm_minor_free(dev, DRM_MINOR_RENDER);
+ put_device(dev->dev);
+
mutex_destroy(&dev->master_mutex);
- mutex_destroy(&dev->ctxlist_mutex);
mutex_destroy(&dev->clientlist_mutex);
mutex_destroy(&dev->filelist_mutex);
mutex_destroy(&dev->struct_mutex);
+ drm_legacy_destroy_members(dev);
kfree(dev->unique);
}
EXPORT_SYMBOL(drm_dev_fini);
*/
void drm_dev_unregister(struct drm_device *dev)
{
- struct drm_map_list *r_list, *list_temp;
-
if (drm_core_check_feature(dev, DRIVER_LEGACY))
drm_lastclose(dev);
if (dev->agp)
drm_pci_agp_destroy(dev);
- list_for_each_entry_safe(r_list, list_temp, &dev->maplist, head)
- drm_legacy_rmmap(dev, r_list->map);
+ drm_legacy_rmmaps(dev);
remove_compat_control_link(dev);
drm_minor_unregister(dev, DRM_MINOR_PRIMARY);
#include <linux/init.h>
#include <linux/errno.h>
#include <linux/byteorder/generic.h>
+#include <drm/drm_print.h>
#include <drm/drm_dp_helper.h>
#include <drm/drm_dsc.h>
/**
* drm_dsc_dp_pps_header_init() - Initializes the PPS Header
* for DisplayPort as per the DP 1.4 spec.
- * @pps_sdp: Secondary data packet for DSC Picture Parameter Set
- * as defined in &struct drm_dsc_pps_infoframe
+ * @pps_header: Secondary data packet header for DSC Picture
+ * Parameter Set as defined in &struct dp_sdp_header
*
* DP 1.4 spec defines the secondary data packet for sending the
* picture parameter infoframes from the source to the sink.
- * This function populates the pps header defined in
- * &struct drm_dsc_pps_infoframe as per the header bytes defined
- * in &struct dp_sdp_header.
+ * This function populates the SDP header defined in
+ * &struct dp_sdp_header.
*/
-void drm_dsc_dp_pps_header_init(struct drm_dsc_pps_infoframe *pps_sdp)
+void drm_dsc_dp_pps_header_init(struct dp_sdp_header *pps_header)
{
- memset(&pps_sdp->pps_header, 0, sizeof(pps_sdp->pps_header));
+ memset(pps_header, 0, sizeof(*pps_header));
- pps_sdp->pps_header.HB1 = DP_SDP_PPS;
- pps_sdp->pps_header.HB2 = DP_SDP_PPS_HEADER_PAYLOAD_BYTES_MINUS_1;
+ pps_header->HB1 = DP_SDP_PPS;
+ pps_header->HB2 = DP_SDP_PPS_HEADER_PAYLOAD_BYTES_MINUS_1;
}
EXPORT_SYMBOL(drm_dsc_dp_pps_header_init);
/**
- * drm_dsc_pps_infoframe_pack() - Populates the DSC PPS infoframe
+ * drm_dsc_pps_payload_pack() - Populates the DSC PPS
*
- * @pps_sdp:
- * Secondary data packet for DSC Picture Parameter Set. This is defined
- * by &struct drm_dsc_pps_infoframe
+ * @pps_payload:
+ * Bitwise struct for DSC Picture Parameter Set. This is defined
+ * by &struct drm_dsc_picture_parameter_set
* @dsc_cfg:
* DSC Configuration data filled by driver as defined by
* &struct drm_dsc_config
*
- * DSC source device sends a secondary data packet filled with all the
- * picture parameter set (PPS) information required by the sink to decode
- * the compressed frame. Driver populates the dsC PPS infoframe using the DSC
- * configuration parameters in the order expected by the DSC Display Sink
- * device. For the DSC, the sink device expects the PPS payload in the big
- * endian format for the fields that span more than 1 byte.
+ * DSC source device sends a picture parameter set (PPS) containing the
+ * information required by the sink to decode the compressed frame. Driver
+ * populates the DSC PPS struct using the DSC configuration parameters in
+ * the order expected by the DSC Display Sink device. For the DSC, the sink
+ * device expects the PPS payload in big endian format for fields
+ * that span more than 1 byte.
*/
-void drm_dsc_pps_infoframe_pack(struct drm_dsc_pps_infoframe *pps_sdp,
+void drm_dsc_pps_payload_pack(struct drm_dsc_picture_parameter_set *pps_payload,
const struct drm_dsc_config *dsc_cfg)
{
int i;
/* Protect against someone accidently changing struct size */
- BUILD_BUG_ON(sizeof(pps_sdp->pps_payload) !=
+ BUILD_BUG_ON(sizeof(*pps_payload) !=
DP_SDP_PPS_HEADER_PAYLOAD_BYTES_MINUS_1 + 1);
- memset(&pps_sdp->pps_payload, 0, sizeof(pps_sdp->pps_payload));
+ memset(pps_payload, 0, sizeof(*pps_payload));
/* PPS 0 */
- pps_sdp->pps_payload.dsc_version =
+ pps_payload->dsc_version =
dsc_cfg->dsc_version_minor |
dsc_cfg->dsc_version_major << DSC_PPS_VERSION_MAJOR_SHIFT;
/* PPS 1, 2 is 0 */
/* PPS 3 */
- pps_sdp->pps_payload.pps_3 =
+ pps_payload->pps_3 =
dsc_cfg->line_buf_depth |
dsc_cfg->bits_per_component << DSC_PPS_BPC_SHIFT;
/* PPS 4 */
- pps_sdp->pps_payload.pps_4 =
+ pps_payload->pps_4 =
((dsc_cfg->bits_per_pixel & DSC_PPS_BPP_HIGH_MASK) >>
DSC_PPS_MSB_SHIFT) |
dsc_cfg->vbr_enable << DSC_PPS_VBR_EN_SHIFT |
- dsc_cfg->enable422 << DSC_PPS_SIMPLE422_SHIFT |
+ dsc_cfg->simple_422 << DSC_PPS_SIMPLE422_SHIFT |
dsc_cfg->convert_rgb << DSC_PPS_CONVERT_RGB_SHIFT |
dsc_cfg->block_pred_enable << DSC_PPS_BLOCK_PRED_EN_SHIFT;
/* PPS 5 */
- pps_sdp->pps_payload.bits_per_pixel_low =
+ pps_payload->bits_per_pixel_low =
(dsc_cfg->bits_per_pixel & DSC_PPS_LSB_MASK);
/*
*/
/* PPS 6, 7 */
- pps_sdp->pps_payload.pic_height = cpu_to_be16(dsc_cfg->pic_height);
+ pps_payload->pic_height = cpu_to_be16(dsc_cfg->pic_height);
/* PPS 8, 9 */
- pps_sdp->pps_payload.pic_width = cpu_to_be16(dsc_cfg->pic_width);
+ pps_payload->pic_width = cpu_to_be16(dsc_cfg->pic_width);
/* PPS 10, 11 */
- pps_sdp->pps_payload.slice_height = cpu_to_be16(dsc_cfg->slice_height);
+ pps_payload->slice_height = cpu_to_be16(dsc_cfg->slice_height);
/* PPS 12, 13 */
- pps_sdp->pps_payload.slice_width = cpu_to_be16(dsc_cfg->slice_width);
+ pps_payload->slice_width = cpu_to_be16(dsc_cfg->slice_width);
/* PPS 14, 15 */
- pps_sdp->pps_payload.chunk_size = cpu_to_be16(dsc_cfg->slice_chunk_size);
+ pps_payload->chunk_size = cpu_to_be16(dsc_cfg->slice_chunk_size);
/* PPS 16 */
- pps_sdp->pps_payload.initial_xmit_delay_high =
+ pps_payload->initial_xmit_delay_high =
((dsc_cfg->initial_xmit_delay &
DSC_PPS_INIT_XMIT_DELAY_HIGH_MASK) >>
DSC_PPS_MSB_SHIFT);
/* PPS 17 */
- pps_sdp->pps_payload.initial_xmit_delay_low =
+ pps_payload->initial_xmit_delay_low =
(dsc_cfg->initial_xmit_delay & DSC_PPS_LSB_MASK);
/* PPS 18, 19 */
- pps_sdp->pps_payload.initial_dec_delay =
+ pps_payload->initial_dec_delay =
cpu_to_be16(dsc_cfg->initial_dec_delay);
/* PPS 20 is 0 */
/* PPS 21 */
- pps_sdp->pps_payload.initial_scale_value =
+ pps_payload->initial_scale_value =
dsc_cfg->initial_scale_value;
/* PPS 22, 23 */
- pps_sdp->pps_payload.scale_increment_interval =
+ pps_payload->scale_increment_interval =
cpu_to_be16(dsc_cfg->scale_increment_interval);
/* PPS 24 */
- pps_sdp->pps_payload.scale_decrement_interval_high =
+ pps_payload->scale_decrement_interval_high =
((dsc_cfg->scale_decrement_interval &
DSC_PPS_SCALE_DEC_INT_HIGH_MASK) >>
DSC_PPS_MSB_SHIFT);
/* PPS 25 */
- pps_sdp->pps_payload.scale_decrement_interval_low =
+ pps_payload->scale_decrement_interval_low =
(dsc_cfg->scale_decrement_interval & DSC_PPS_LSB_MASK);
/* PPS 26[7:0], PPS 27[7:5] RESERVED */
/* PPS 27 */
- pps_sdp->pps_payload.first_line_bpg_offset =
+ pps_payload->first_line_bpg_offset =
dsc_cfg->first_line_bpg_offset;
/* PPS 28, 29 */
- pps_sdp->pps_payload.nfl_bpg_offset =
+ pps_payload->nfl_bpg_offset =
cpu_to_be16(dsc_cfg->nfl_bpg_offset);
/* PPS 30, 31 */
- pps_sdp->pps_payload.slice_bpg_offset =
+ pps_payload->slice_bpg_offset =
cpu_to_be16(dsc_cfg->slice_bpg_offset);
/* PPS 32, 33 */
- pps_sdp->pps_payload.initial_offset =
+ pps_payload->initial_offset =
cpu_to_be16(dsc_cfg->initial_offset);
/* PPS 34, 35 */
- pps_sdp->pps_payload.final_offset = cpu_to_be16(dsc_cfg->final_offset);
+ pps_payload->final_offset = cpu_to_be16(dsc_cfg->final_offset);
/* PPS 36 */
- pps_sdp->pps_payload.flatness_min_qp = dsc_cfg->flatness_min_qp;
+ pps_payload->flatness_min_qp = dsc_cfg->flatness_min_qp;
/* PPS 37 */
- pps_sdp->pps_payload.flatness_max_qp = dsc_cfg->flatness_max_qp;
+ pps_payload->flatness_max_qp = dsc_cfg->flatness_max_qp;
/* PPS 38, 39 */
- pps_sdp->pps_payload.rc_model_size =
+ pps_payload->rc_model_size =
cpu_to_be16(DSC_RC_MODEL_SIZE_CONST);
/* PPS 40 */
- pps_sdp->pps_payload.rc_edge_factor = DSC_RC_EDGE_FACTOR_CONST;
+ pps_payload->rc_edge_factor = DSC_RC_EDGE_FACTOR_CONST;
/* PPS 41 */
- pps_sdp->pps_payload.rc_quant_incr_limit0 =
+ pps_payload->rc_quant_incr_limit0 =
dsc_cfg->rc_quant_incr_limit0;
/* PPS 42 */
- pps_sdp->pps_payload.rc_quant_incr_limit1 =
+ pps_payload->rc_quant_incr_limit1 =
dsc_cfg->rc_quant_incr_limit1;
/* PPS 43 */
- pps_sdp->pps_payload.rc_tgt_offset = DSC_RC_TGT_OFFSET_LO_CONST |
+ pps_payload->rc_tgt_offset = DSC_RC_TGT_OFFSET_LO_CONST |
DSC_RC_TGT_OFFSET_HI_CONST << DSC_PPS_RC_TGT_OFFSET_HI_SHIFT;
/* PPS 44 - 57 */
for (i = 0; i < DSC_NUM_BUF_RANGES - 1; i++)
- pps_sdp->pps_payload.rc_buf_thresh[i] =
+ pps_payload->rc_buf_thresh[i] =
dsc_cfg->rc_buf_thresh[i];
/* PPS 58 - 87 */
* are as follows: Min_qp[15:11], max_qp[10:6], offset[5:0]
*/
for (i = 0; i < DSC_NUM_BUF_RANGES; i++) {
- pps_sdp->pps_payload.rc_range_parameters[i] =
+ pps_payload->rc_range_parameters[i] =
((dsc_cfg->rc_range_params[i].range_min_qp <<
DSC_PPS_RC_RANGE_MINQP_SHIFT) |
(dsc_cfg->rc_range_params[i].range_max_qp <<
DSC_PPS_RC_RANGE_MAXQP_SHIFT) |
(dsc_cfg->rc_range_params[i].range_bpg_offset));
- pps_sdp->pps_payload.rc_range_parameters[i] =
- cpu_to_be16(pps_sdp->pps_payload.rc_range_parameters[i]);
+ pps_payload->rc_range_parameters[i] =
+ cpu_to_be16(pps_payload->rc_range_parameters[i]);
}
/* PPS 88 */
- pps_sdp->pps_payload.native_422_420 = dsc_cfg->native_422 |
+ pps_payload->native_422_420 = dsc_cfg->native_422 |
dsc_cfg->native_420 << DSC_PPS_NATIVE_420_SHIFT;
/* PPS 89 */
- pps_sdp->pps_payload.second_line_bpg_offset =
+ pps_payload->second_line_bpg_offset =
dsc_cfg->second_line_bpg_offset;
/* PPS 90, 91 */
- pps_sdp->pps_payload.nsl_bpg_offset =
+ pps_payload->nsl_bpg_offset =
cpu_to_be16(dsc_cfg->nsl_bpg_offset);
/* PPS 92, 93 */
- pps_sdp->pps_payload.second_line_offset_adj =
+ pps_payload->second_line_offset_adj =
cpu_to_be16(dsc_cfg->second_line_offset_adj);
/* PPS 94 - 127 are O */
}
-EXPORT_SYMBOL(drm_dsc_pps_infoframe_pack);
+EXPORT_SYMBOL(drm_dsc_pps_payload_pack);
+
+/**
+ * drm_dsc_compute_rc_parameters() - Write rate control
+ * parameters to the dsc configuration defined in
+ * &struct drm_dsc_config in accordance with the DSC 1.2
+ * specification. Some configuration fields must be present
+ * beforehand.
+ *
+ * @vdsc_cfg:
+ * DSC Configuration data partially filled by driver
+ */
+int drm_dsc_compute_rc_parameters(struct drm_dsc_config *vdsc_cfg)
+{
+ unsigned long groups_per_line = 0;
+ unsigned long groups_total = 0;
+ unsigned long num_extra_mux_bits = 0;
+ unsigned long slice_bits = 0;
+ unsigned long hrd_delay = 0;
+ unsigned long final_scale = 0;
+ unsigned long rbs_min = 0;
+
+ if (vdsc_cfg->native_420 || vdsc_cfg->native_422) {
+ /* Number of groups used to code each line of a slice */
+ groups_per_line = DIV_ROUND_UP(vdsc_cfg->slice_width / 2,
+ DSC_RC_PIXELS_PER_GROUP);
+
+ /* chunksize in Bytes */
+ vdsc_cfg->slice_chunk_size = DIV_ROUND_UP(vdsc_cfg->slice_width / 2 *
+ vdsc_cfg->bits_per_pixel,
+ (8 * 16));
+ } else {
+ /* Number of groups used to code each line of a slice */
+ groups_per_line = DIV_ROUND_UP(vdsc_cfg->slice_width,
+ DSC_RC_PIXELS_PER_GROUP);
+
+ /* chunksize in Bytes */
+ vdsc_cfg->slice_chunk_size = DIV_ROUND_UP(vdsc_cfg->slice_width *
+ vdsc_cfg->bits_per_pixel,
+ (8 * 16));
+ }
+
+ if (vdsc_cfg->convert_rgb)
+ num_extra_mux_bits = 3 * (vdsc_cfg->mux_word_size +
+ (4 * vdsc_cfg->bits_per_component + 4)
+ - 2);
+ else if (vdsc_cfg->native_422)
+ num_extra_mux_bits = 4 * vdsc_cfg->mux_word_size +
+ (4 * vdsc_cfg->bits_per_component + 4) +
+ 3 * (4 * vdsc_cfg->bits_per_component) - 2;
+ else
+ num_extra_mux_bits = 3 * vdsc_cfg->mux_word_size +
+ (4 * vdsc_cfg->bits_per_component + 4) +
+ 2 * (4 * vdsc_cfg->bits_per_component) - 2;
+ /* Number of bits in one Slice */
+ slice_bits = 8 * vdsc_cfg->slice_chunk_size * vdsc_cfg->slice_height;
+
+ while ((num_extra_mux_bits > 0) &&
+ ((slice_bits - num_extra_mux_bits) % vdsc_cfg->mux_word_size))
+ num_extra_mux_bits--;
+
+ if (groups_per_line < vdsc_cfg->initial_scale_value - 8)
+ vdsc_cfg->initial_scale_value = groups_per_line + 8;
+
+ /* scale_decrement_interval calculation according to DSC spec 1.11 */
+ if (vdsc_cfg->initial_scale_value > 8)
+ vdsc_cfg->scale_decrement_interval = groups_per_line /
+ (vdsc_cfg->initial_scale_value - 8);
+ else
+ vdsc_cfg->scale_decrement_interval = DSC_SCALE_DECREMENT_INTERVAL_MAX;
+
+ vdsc_cfg->final_offset = vdsc_cfg->rc_model_size -
+ (vdsc_cfg->initial_xmit_delay *
+ vdsc_cfg->bits_per_pixel + 8) / 16 + num_extra_mux_bits;
+
+ if (vdsc_cfg->final_offset >= vdsc_cfg->rc_model_size) {
+ DRM_DEBUG_KMS("FinalOfs < RcModelSze for this InitialXmitDelay\n");
+ return -ERANGE;
+ }
+
+ final_scale = (vdsc_cfg->rc_model_size * 8) /
+ (vdsc_cfg->rc_model_size - vdsc_cfg->final_offset);
+ if (vdsc_cfg->slice_height > 1)
+ /*
+ * NflBpgOffset is 16 bit value with 11 fractional bits
+ * hence we multiply by 2^11 for preserving the
+ * fractional part
+ */
+ vdsc_cfg->nfl_bpg_offset = DIV_ROUND_UP((vdsc_cfg->first_line_bpg_offset << 11),
+ (vdsc_cfg->slice_height - 1));
+ else
+ vdsc_cfg->nfl_bpg_offset = 0;
+
+ /* 2^16 - 1 */
+ if (vdsc_cfg->nfl_bpg_offset > 65535) {
+ DRM_DEBUG_KMS("NflBpgOffset is too large for this slice height\n");
+ return -ERANGE;
+ }
+
+ /* Number of groups used to code the entire slice */
+ groups_total = groups_per_line * vdsc_cfg->slice_height;
+
+ /* slice_bpg_offset is 16 bit value with 11 fractional bits */
+ vdsc_cfg->slice_bpg_offset = DIV_ROUND_UP(((vdsc_cfg->rc_model_size -
+ vdsc_cfg->initial_offset +
+ num_extra_mux_bits) << 11),
+ groups_total);
+
+ if (final_scale > 9) {
+ /*
+ * ScaleIncrementInterval =
+ * finaloffset/((NflBpgOffset + SliceBpgOffset)*8(finalscale - 1.125))
+ * as (NflBpgOffset + SliceBpgOffset) has 11 bit fractional value,
+ * we need divide by 2^11 from pstDscCfg values
+ */
+ vdsc_cfg->scale_increment_interval =
+ (vdsc_cfg->final_offset * (1 << 11)) /
+ ((vdsc_cfg->nfl_bpg_offset +
+ vdsc_cfg->slice_bpg_offset) *
+ (final_scale - 9));
+ } else {
+ /*
+ * If finalScaleValue is less than or equal to 9, a value of 0 should
+ * be used to disable the scale increment at the end of the slice
+ */
+ vdsc_cfg->scale_increment_interval = 0;
+ }
+
+ if (vdsc_cfg->scale_increment_interval > 65535) {
+ DRM_DEBUG_KMS("ScaleIncrementInterval is large for slice height\n");
+ return -ERANGE;
+ }
+
+ /*
+ * DSC spec mentions that bits_per_pixel specifies the target
+ * bits/pixel (bpp) rate that is used by the encoder,
+ * in steps of 1/16 of a bit per pixel
+ */
+ rbs_min = vdsc_cfg->rc_model_size - vdsc_cfg->initial_offset +
+ DIV_ROUND_UP(vdsc_cfg->initial_xmit_delay *
+ vdsc_cfg->bits_per_pixel, 16) +
+ groups_per_line * vdsc_cfg->first_line_bpg_offset;
+
+ hrd_delay = DIV_ROUND_UP((rbs_min * 16), vdsc_cfg->bits_per_pixel);
+ vdsc_cfg->rc_bits = (hrd_delay * vdsc_cfg->bits_per_pixel) / 16;
+ vdsc_cfg->initial_dec_delay = hrd_delay - vdsc_cfg->initial_xmit_delay;
+
+ return 0;
+}
+EXPORT_SYMBOL(drm_dsc_compute_rc_parameters);
* maximum size and use that.
*/
#define EDID_QUIRK_DETAILED_USE_MAXIMUM_SIZE (1 << 4)
-/* Monitor forgot to set the first detailed is preferred bit. */
-#define EDID_QUIRK_FIRST_DETAILED_PREFERRED (1 << 5)
/* use +hsync +vsync for detailed mode */
#define EDID_QUIRK_DETAILED_SYNC_PP (1 << 6)
/* Force reduced-blanking timings for detailed modes */
{ "ACR", 44358, EDID_QUIRK_PREFER_LARGE_60 },
/* Acer F51 */
{ "API", 0x7602, EDID_QUIRK_PREFER_LARGE_60 },
- /* Unknown Acer */
- { "ACR", 2423, EDID_QUIRK_FIRST_DETAILED_PREFERRED },
/* AEO model 0 reports 8 bpc, but is a 6 bpc panel */
{ "AEO", 0, EDID_QUIRK_FORCE_6BPC },
{ "LPL", 0, EDID_QUIRK_DETAILED_USE_MAXIMUM_SIZE },
{ "LPL", 0x2a00, EDID_QUIRK_DETAILED_USE_MAXIMUM_SIZE },
- /* Philips 107p5 CRT */
- { "PHL", 57364, EDID_QUIRK_FIRST_DETAILED_PREFERRED },
-
- /* Proview AY765C */
- { "PTS", 765, EDID_QUIRK_FIRST_DETAILED_PREFERRED },
-
/* Samsung SyncMaster 205BW. Note: irony */
{ "SAM", 541, EDID_QUIRK_DETAILED_SYNC_PP },
/* Samsung SyncMaster 22[5-6]BW */
/* Rotel RSX-1058 forwards sink's EDID but only does HDMI 1.1*/
{ "ETR", 13896, EDID_QUIRK_FORCE_8BPC },
+ /* Valve Index Headset */
+ { "VLV", 0x91a8, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91b0, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91b1, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91b2, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91b3, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91b4, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91b5, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91b6, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91b7, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91b8, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91b9, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91ba, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91bb, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91bc, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91bd, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91be, EDID_QUIRK_NON_DESKTOP },
+ { "VLV", 0x91bf, EDID_QUIRK_NON_DESKTOP },
+
/* HTC Vive and Vive Pro VR Headsets */
{ "HVR", 0xaa01, EDID_QUIRK_NON_DESKTOP },
{ "HVR", 0xaa02, EDID_QUIRK_NON_DESKTOP },
/* Sony PlayStation VR Headset */
{ "SNY", 0x0704, EDID_QUIRK_NON_DESKTOP },
+
+ /* Sensics VR Headsets */
+ { "SEN", 0x1019, EDID_QUIRK_NON_DESKTOP },
+
+ /* OSVR HDK and HDK2 VR Headsets */
+ { "SVR", 0x1019, EDID_QUIRK_NON_DESKTOP },
};
/*
}
EXPORT_SYMBOL(drm_hdmi_avi_infoframe_from_display_mode);
+/* HDMI Colorspace Spec Definitions */
+#define FULL_COLORIMETRY_MASK 0x1FF
+#define NORMAL_COLORIMETRY_MASK 0x3
+#define EXTENDED_COLORIMETRY_MASK 0x7
+#define EXTENDED_ACE_COLORIMETRY_MASK 0xF
+
+#define C(x) ((x) << 0)
+#define EC(x) ((x) << 2)
+#define ACE(x) ((x) << 5)
+
+#define HDMI_COLORIMETRY_NO_DATA 0x0
+#define HDMI_COLORIMETRY_SMPTE_170M_YCC (C(1) | EC(0) | ACE(0))
+#define HDMI_COLORIMETRY_BT709_YCC (C(2) | EC(0) | ACE(0))
+#define HDMI_COLORIMETRY_XVYCC_601 (C(3) | EC(0) | ACE(0))
+#define HDMI_COLORIMETRY_XVYCC_709 (C(3) | EC(1) | ACE(0))
+#define HDMI_COLORIMETRY_SYCC_601 (C(3) | EC(2) | ACE(0))
+#define HDMI_COLORIMETRY_OPYCC_601 (C(3) | EC(3) | ACE(0))
+#define HDMI_COLORIMETRY_OPRGB (C(3) | EC(4) | ACE(0))
+#define HDMI_COLORIMETRY_BT2020_CYCC (C(3) | EC(5) | ACE(0))
+#define HDMI_COLORIMETRY_BT2020_RGB (C(3) | EC(6) | ACE(0))
+#define HDMI_COLORIMETRY_BT2020_YCC (C(3) | EC(6) | ACE(0))
+#define HDMI_COLORIMETRY_DCI_P3_RGB_D65 (C(3) | EC(7) | ACE(0))
+#define HDMI_COLORIMETRY_DCI_P3_RGB_THEATER (C(3) | EC(7) | ACE(1))
+
+static const u32 hdmi_colorimetry_val[] = {
+ [DRM_MODE_COLORIMETRY_NO_DATA] = HDMI_COLORIMETRY_NO_DATA,
+ [DRM_MODE_COLORIMETRY_SMPTE_170M_YCC] = HDMI_COLORIMETRY_SMPTE_170M_YCC,
+ [DRM_MODE_COLORIMETRY_BT709_YCC] = HDMI_COLORIMETRY_BT709_YCC,
+ [DRM_MODE_COLORIMETRY_XVYCC_601] = HDMI_COLORIMETRY_XVYCC_601,
+ [DRM_MODE_COLORIMETRY_XVYCC_709] = HDMI_COLORIMETRY_XVYCC_709,
+ [DRM_MODE_COLORIMETRY_SYCC_601] = HDMI_COLORIMETRY_SYCC_601,
+ [DRM_MODE_COLORIMETRY_OPYCC_601] = HDMI_COLORIMETRY_OPYCC_601,
+ [DRM_MODE_COLORIMETRY_OPRGB] = HDMI_COLORIMETRY_OPRGB,
+ [DRM_MODE_COLORIMETRY_BT2020_CYCC] = HDMI_COLORIMETRY_BT2020_CYCC,
+ [DRM_MODE_COLORIMETRY_BT2020_RGB] = HDMI_COLORIMETRY_BT2020_RGB,
+ [DRM_MODE_COLORIMETRY_BT2020_YCC] = HDMI_COLORIMETRY_BT2020_YCC,
+};
+
+#undef C
+#undef EC
+#undef ACE
+
+/**
+ * drm_hdmi_avi_infoframe_colorspace() - fill the HDMI AVI infoframe
+ * colorspace information
+ * @frame: HDMI AVI infoframe
+ * @conn_state: connector state
+ */
+void
+drm_hdmi_avi_infoframe_colorspace(struct hdmi_avi_infoframe *frame,
+ const struct drm_connector_state *conn_state)
+{
+ u32 colorimetry_val;
+ u32 colorimetry_index = conn_state->colorspace & FULL_COLORIMETRY_MASK;
+
+ if (colorimetry_index >= ARRAY_SIZE(hdmi_colorimetry_val))
+ colorimetry_val = HDMI_COLORIMETRY_NO_DATA;
+ else
+ colorimetry_val = hdmi_colorimetry_val[colorimetry_index];
+
+ frame->colorimetry = colorimetry_val & NORMAL_COLORIMETRY_MASK;
+ /*
+ * ToDo: Extend it for ACE formats as well. Modify the infoframe
+ * structure and extend it in drivers/video/hdmi
+ */
+ frame->extended_colorimetry = (colorimetry_val >> 2) &
+ EXTENDED_COLORIMETRY_MASK;
+}
+EXPORT_SYMBOL(drm_hdmi_avi_infoframe_colorspace);
+
/**
* drm_hdmi_avi_infoframe_quant_range() - fill the HDMI AVI infoframe
* quantization range information
static void dpms_legacy(struct drm_fb_helper *fb_helper, int dpms_mode)
{
struct drm_device *dev = fb_helper->dev;
- struct drm_crtc *crtc;
struct drm_connector *connector;
+ struct drm_mode_set *modeset;
int i, j;
drm_modeset_lock_all(dev);
for (i = 0; i < fb_helper->crtc_count; i++) {
- crtc = fb_helper->crtc_info[i].mode_set.crtc;
+ modeset = &fb_helper->crtc_info[i].mode_set;
- if (!crtc->enabled)
+ if (!modeset->crtc->enabled)
continue;
- /* Walk the connectors & encoders on this fb turning them on/off */
- drm_fb_helper_for_each_connector(fb_helper, j) {
- connector = fb_helper->connector_info[j]->connector;
+ for (j = 0; j < modeset->num_connectors; j++) {
+ connector = modeset->connectors[j];
connector->funcs->dpms(connector, dpms_mode);
drm_object_property_set_value(&connector->base,
dev->mode_config.dpms_property, dpms_mode);
}
fb_helper->fbdev = info;
+ info->skip_vt_switch = true;
return info;
int crtc_count = 0;
int i;
struct drm_fb_helper_surface_size sizes;
- int gamma_size = 0;
int best_depth = 0;
memset(&sizes, 0, sizeof(struct drm_fb_helper_surface_size));
if (preferred_bpp != sizes.surface_bpp)
sizes.surface_depth = sizes.surface_bpp = preferred_bpp;
- /* first up get a count of crtcs now in use and new min/maxes width/heights */
drm_fb_helper_for_each_connector(fb_helper, i) {
struct drm_fb_helper_connector *fb_helper_conn = fb_helper->connector_info[i];
struct drm_cmdline_mode *cmdline_mode;
sizes.surface_depth = best_depth;
}
+ /* first up get a count of crtcs now in use and new min/maxes width/heights */
crtc_count = 0;
for (i = 0; i < fb_helper->crtc_count; i++) {
struct drm_display_mode *desired_mode;
x = fb_helper->crtc_info[i].x;
y = fb_helper->crtc_info[i].y;
- if (gamma_size == 0)
- gamma_size = fb_helper->crtc_info[i].mode_set.crtc->gamma_size;
-
sizes.surface_width = max_t(u32, desired_mode->hdisplay + x, sizes.surface_width);
sizes.surface_height = max_t(u32, desired_mode->vdisplay + y, sizes.surface_height);
return 0;
}
-/**
- * drm_fb_helper_fill_fix - initializes fixed fbdev information
- * @info: fbdev registered by the helper
- * @pitch: desired pitch
- * @depth: desired depth
- *
- * Helper to fill in the fixed fbdev information useful for a non-accelerated
- * fbdev emulations. Drivers which support acceleration methods which impose
- * additional constraints need to set up their own limits.
- *
- * Drivers should call this (or their equivalent setup code) from their
- * &drm_fb_helper_funcs.fb_probe callback.
- */
-void drm_fb_helper_fill_fix(struct fb_info *info, uint32_t pitch,
- uint32_t depth)
+static void drm_fb_helper_fill_fix(struct fb_info *info, uint32_t pitch,
+ uint32_t depth)
{
info->fix.type = FB_TYPE_PACKED_PIXELS;
info->fix.visual = depth == 8 ? FB_VISUAL_PSEUDOCOLOR :
info->fix.line_length = pitch;
}
-EXPORT_SYMBOL(drm_fb_helper_fill_fix);
-/**
- * drm_fb_helper_fill_var - initalizes variable fbdev information
- * @info: fbdev instance to set up
- * @fb_helper: fb helper instance to use as template
- * @fb_width: desired fb width
- * @fb_height: desired fb height
- *
- * Sets up the variable fbdev metainformation from the given fb helper instance
- * and the drm framebuffer allocated in &drm_fb_helper.fb.
- *
- * Drivers should call this (or their equivalent setup code) from their
- * &drm_fb_helper_funcs.fb_probe callback after having allocated the fbdev
- * backing storage framebuffer.
- */
-void drm_fb_helper_fill_var(struct fb_info *info, struct drm_fb_helper *fb_helper,
- uint32_t fb_width, uint32_t fb_height)
+static void drm_fb_helper_fill_var(struct fb_info *info,
+ struct drm_fb_helper *fb_helper,
+ uint32_t fb_width, uint32_t fb_height)
{
struct drm_framebuffer *fb = fb_helper->fb;
info->var.xres = fb_width;
info->var.yres = fb_height;
}
-EXPORT_SYMBOL(drm_fb_helper_fill_var);
+
+/**
+ * drm_fb_helper_fill_info - initializes fbdev information
+ * @info: fbdev instance to set up
+ * @fb_helper: fb helper instance to use as template
+ * @sizes: describes fbdev size and scanout surface size
+ *
+ * Sets up the variable and fixed fbdev metainformation from the given fb helper
+ * instance and the drm framebuffer allocated in &drm_fb_helper.fb.
+ *
+ * Drivers should call this (or their equivalent setup code) from their
+ * &drm_fb_helper_funcs.fb_probe callback after having allocated the fbdev
+ * backing storage framebuffer.
+ */
+void drm_fb_helper_fill_info(struct fb_info *info,
+ struct drm_fb_helper *fb_helper,
+ struct drm_fb_helper_surface_size *sizes)
+{
+ struct drm_framebuffer *fb = fb_helper->fb;
+
+ drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
+ drm_fb_helper_fill_var(info, fb_helper,
+ sizes->fb_width, sizes->fb_height);
+
+ info->par = fb_helper;
+ snprintf(info->fix.id, sizeof(info->fix.id), "%sdrmfb",
+ fb_helper->dev->driver->name);
+
+}
+EXPORT_SYMBOL(drm_fb_helper_fill_info);
static int drm_fb_helper_probe_connector_modes(struct drm_fb_helper *fb_helper,
uint32_t maxX,
fb_helper->sw_rotations |= DRM_MODE_ROTATE_0;
}
+static struct drm_fb_helper_crtc *
+drm_fb_helper_crtc(struct drm_fb_helper *fb_helper, struct drm_crtc *crtc)
+{
+ int i;
+
+ for (i = 0; i < fb_helper->crtc_count; i++)
+ if (fb_helper->crtc_info[i].mode_set.crtc == crtc)
+ return &fb_helper->crtc_info[i];
+
+ return NULL;
+}
+
+/* Try to read the BIOS display configuration and use it for the initial config */
+static bool drm_fb_helper_firmware_config(struct drm_fb_helper *fb_helper,
+ struct drm_fb_helper_crtc **crtcs,
+ struct drm_display_mode **modes,
+ struct drm_fb_offset *offsets,
+ bool *enabled, int width, int height)
+{
+ struct drm_device *dev = fb_helper->dev;
+ unsigned int count = min(fb_helper->connector_count, BITS_PER_LONG);
+ unsigned long conn_configured, conn_seq, mask;
+ int i, j;
+ bool *save_enabled;
+ bool fallback = true, ret = true;
+ int num_connectors_enabled = 0;
+ int num_connectors_detected = 0;
+ struct drm_modeset_acquire_ctx ctx;
+
+ if (!drm_drv_uses_atomic_modeset(dev))
+ return false;
+
+ save_enabled = kcalloc(count, sizeof(bool), GFP_KERNEL);
+ if (!save_enabled)
+ return false;
+
+ drm_modeset_acquire_init(&ctx, 0);
+
+ while (drm_modeset_lock_all_ctx(dev, &ctx) != 0)
+ drm_modeset_backoff(&ctx);
+
+ memcpy(save_enabled, enabled, count);
+ mask = GENMASK(count - 1, 0);
+ conn_configured = 0;
+retry:
+ conn_seq = conn_configured;
+ for (i = 0; i < count; i++) {
+ struct drm_fb_helper_connector *fb_conn;
+ struct drm_connector *connector;
+ struct drm_encoder *encoder;
+ struct drm_fb_helper_crtc *new_crtc;
+
+ fb_conn = fb_helper->connector_info[i];
+ connector = fb_conn->connector;
+
+ if (conn_configured & BIT(i))
+ continue;
+
+ if (conn_seq == 0 && !connector->has_tile)
+ continue;
+
+ if (connector->status == connector_status_connected)
+ num_connectors_detected++;
+
+ if (!enabled[i]) {
+ DRM_DEBUG_KMS("connector %s not enabled, skipping\n",
+ connector->name);
+ conn_configured |= BIT(i);
+ continue;
+ }
+
+ if (connector->force == DRM_FORCE_OFF) {
+ DRM_DEBUG_KMS("connector %s is disabled by user, skipping\n",
+ connector->name);
+ enabled[i] = false;
+ continue;
+ }
+
+ encoder = connector->state->best_encoder;
+ if (!encoder || WARN_ON(!connector->state->crtc)) {
+ if (connector->force > DRM_FORCE_OFF)
+ goto bail;
+
+ DRM_DEBUG_KMS("connector %s has no encoder or crtc, skipping\n",
+ connector->name);
+ enabled[i] = false;
+ conn_configured |= BIT(i);
+ continue;
+ }
+
+ num_connectors_enabled++;
+
+ new_crtc = drm_fb_helper_crtc(fb_helper, connector->state->crtc);
+
+ /*
+ * Make sure we're not trying to drive multiple connectors
+ * with a single CRTC, since our cloning support may not
+ * match the BIOS.
+ */
+ for (j = 0; j < count; j++) {
+ if (crtcs[j] == new_crtc) {
+ DRM_DEBUG_KMS("fallback: cloned configuration\n");
+ goto bail;
+ }
+ }
+
+ DRM_DEBUG_KMS("looking for cmdline mode on connector %s\n",
+ connector->name);
+
+ /* go for command line mode first */
+ modes[i] = drm_pick_cmdline_mode(fb_conn);
+
+ /* try for preferred next */
+ if (!modes[i]) {
+ DRM_DEBUG_KMS("looking for preferred mode on connector %s %d\n",
+ connector->name, connector->has_tile);
+ modes[i] = drm_has_preferred_mode(fb_conn, width,
+ height);
+ }
+
+ /* No preferred mode marked by the EDID? Are there any modes? */
+ if (!modes[i] && !list_empty(&connector->modes)) {
+ DRM_DEBUG_KMS("using first mode listed on connector %s\n",
+ connector->name);
+ modes[i] = list_first_entry(&connector->modes,
+ struct drm_display_mode,
+ head);
+ }
+
+ /* last resort: use current mode */
+ if (!modes[i]) {
+ /*
+ * IMPORTANT: We want to use the adjusted mode (i.e.
+ * after the panel fitter upscaling) as the initial
+ * config, not the input mode, which is what crtc->mode
+ * usually contains. But since our current
+ * code puts a mode derived from the post-pfit timings
+ * into crtc->mode this works out correctly.
+ *
+ * This is crtc->mode and not crtc->state->mode for the
+ * fastboot check to work correctly.
+ */
+ DRM_DEBUG_KMS("looking for current mode on connector %s\n",
+ connector->name);
+ modes[i] = &connector->state->crtc->mode;
+ }
+ crtcs[i] = new_crtc;
+
+ DRM_DEBUG_KMS("connector %s on [CRTC:%d:%s]: %dx%d%s\n",
+ connector->name,
+ connector->state->crtc->base.id,
+ connector->state->crtc->name,
+ modes[i]->hdisplay, modes[i]->vdisplay,
+ modes[i]->flags & DRM_MODE_FLAG_INTERLACE ? "i" : "");
+
+ fallback = false;
+ conn_configured |= BIT(i);
+ }
+
+ if ((conn_configured & mask) != mask && conn_configured != conn_seq)
+ goto retry;
+
+ /*
+ * If the BIOS didn't enable everything it could, fall back to have the
+ * same user experiencing of lighting up as much as possible like the
+ * fbdev helper library.
+ */
+ if (num_connectors_enabled != num_connectors_detected &&
+ num_connectors_enabled < dev->mode_config.num_crtc) {
+ DRM_DEBUG_KMS("fallback: Not all outputs enabled\n");
+ DRM_DEBUG_KMS("Enabled: %i, detected: %i\n", num_connectors_enabled,
+ num_connectors_detected);
+ fallback = true;
+ }
+
+ if (fallback) {
+bail:
+ DRM_DEBUG_KMS("Not using firmware configuration\n");
+ memcpy(enabled, save_enabled, count);
+ ret = false;
+ }
+
+ drm_modeset_drop_locks(&ctx);
+ drm_modeset_acquire_fini(&ctx);
+
+ kfree(save_enabled);
+ return ret;
+}
+
static void drm_setup_crtcs(struct drm_fb_helper *fb_helper,
u32 width, u32 height)
{
DRM_DEBUG_KMS("No connectors reported connected with modes\n");
drm_enable_connectors(fb_helper, enabled);
- if (!(fb_helper->funcs->initial_config &&
- fb_helper->funcs->initial_config(fb_helper, crtcs, modes,
- offsets,
- enabled, width, height))) {
+ if (!drm_fb_helper_firmware_config(fb_helper, crtcs, modes, offsets,
+ enabled, width, height)) {
memset(modes, 0, fb_helper->connector_count*sizeof(modes[0]));
memset(crtcs, 0, fb_helper->connector_count*sizeof(crtcs[0]));
memset(offsets, 0, fb_helper->connector_count*sizeof(offsets[0]));
*
* This function will call down into the &drm_fb_helper_funcs.fb_probe callback
* to let the driver allocate and initialize the fbdev info structure and the
- * drm framebuffer used to back the fbdev. drm_fb_helper_fill_var() and
- * drm_fb_helper_fill_fix() are provided as helpers to setup simple default
- * values for the fbdev info structure.
+ * drm framebuffer used to back the fbdev. drm_fb_helper_fill_info() is provided
+ * as a helper to setup simple default values for the fbdev info structure.
*
* HANG DEBUGGING:
*
{
struct drm_fb_helper *fb_helper = info->par;
- if (!try_module_get(fb_helper->dev->driver->fops->owner))
+ /* No need to take a ref for fbcon because it unbinds on unregister */
+ if (user && !try_module_get(fb_helper->dev->driver->fops->owner))
return -ENODEV;
return 0;
{
struct drm_fb_helper *fb_helper = info->par;
- module_put(fb_helper->dev->driver->fops->owner);
+ if (user)
+ module_put(fb_helper->dev->driver->fops->owner);
return 0;
}
if (IS_ERR(fbi))
return PTR_ERR(fbi);
- fbi->par = fb_helper;
fbi->fbops = &drm_fbdev_fb_ops;
fbi->screen_size = fb->height * fb->pitches[0];
fbi->fix.smem_len = fbi->screen_size;
fbi->fix.smem_start =
page_to_phys(virt_to_page(fbi->screen_buffer));
#endif
- strcpy(fbi->fix.id, "DRM emulated");
-
- drm_fb_helper_fill_fix(fbi, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(fbi, fb_helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(fbi, fb_helper, sizes);
if (fb->funcs->dirty) {
struct fb_ops *fbops;
return ret;
}
- drm_client_add(&fb_helper->client);
-
if (!preferred_bpp)
preferred_bpp = dev->mode_config.preferred_depth;
if (!preferred_bpp)
if (ret)
DRM_DEV_DEBUG(dev->dev, "client hotplug ret=%d\n", ret);
+ drm_client_register(&fb_helper->client);
+
return 0;
}
EXPORT_SYMBOL(drm_fbdev_generic_setup);
/* for compatibility root is always authenticated */
file->authenticated = capable(CAP_SYS_ADMIN);
- file->lock_count = 0;
INIT_LIST_HEAD(&file->lhead);
INIT_LIST_HEAD(&file->fbs);
return 0;
}
-static void drm_legacy_dev_reinit(struct drm_device *dev)
-{
- if (dev->irq_enabled)
- drm_irq_uninstall(dev);
-
- mutex_lock(&dev->struct_mutex);
-
- drm_legacy_agp_clear(dev);
-
- drm_legacy_sg_cleanup(dev);
- drm_legacy_vma_flush(dev);
- drm_legacy_dma_takedown(dev);
-
- mutex_unlock(&dev->struct_mutex);
-
- dev->sigdata.lock = NULL;
-
- dev->context_flag = 0;
- dev->last_context = 0;
- dev->if_version = 0;
-
- DRM_DEBUG("lastclose completed\n");
-}
-
void drm_lastclose(struct drm_device * dev)
{
DRM_DEBUG("\n");
file_priv->event_space -= length;
list_add(&e->link, &file_priv->event_list);
spin_unlock_irq(&dev->event_lock);
+ wake_up_interruptible(&file_priv->event_wait);
break;
}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2016 Noralf Trønnes
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/io.h>
+
+#include <drm/drm_format_helper.h>
+#include <drm/drm_framebuffer.h>
+#include <drm/drm_fourcc.h>
+#include <drm/drm_rect.h>
+
+static unsigned int clip_offset(struct drm_rect *clip,
+ unsigned int pitch, unsigned int cpp)
+{
+ return clip->y1 * pitch + clip->x1 * cpp;
+}
+
+/**
+ * drm_fb_memcpy - Copy clip buffer
+ * @dst: Destination buffer
+ * @vaddr: Source buffer
+ * @fb: DRM framebuffer
+ * @clip: Clip rectangle area to copy
+ *
+ * This function does not apply clipping on dst, i.e. the destination
+ * is a small buffer containing the clip rect only.
+ */
+void drm_fb_memcpy(void *dst, void *vaddr, struct drm_framebuffer *fb,
+ struct drm_rect *clip)
+{
+ unsigned int cpp = drm_format_plane_cpp(fb->format->format, 0);
+ size_t len = (clip->x2 - clip->x1) * cpp;
+ unsigned int y, lines = clip->y2 - clip->y1;
+
+ vaddr += clip_offset(clip, fb->pitches[0], cpp);
+ for (y = 0; y < lines; y++) {
+ memcpy(dst, vaddr, len);
+ vaddr += fb->pitches[0];
+ dst += len;
+ }
+}
+EXPORT_SYMBOL(drm_fb_memcpy);
+
+/**
+ * drm_fb_memcpy_dstclip - Copy clip buffer
+ * @dst: Destination buffer (iomem)
+ * @vaddr: Source buffer
+ * @fb: DRM framebuffer
+ * @clip: Clip rectangle area to copy
+ *
+ * This function applies clipping on dst, i.e. the destination is a
+ * full (iomem) framebuffer but only the clip rect content is copied over.
+ */
+void drm_fb_memcpy_dstclip(void __iomem *dst, void *vaddr,
+ struct drm_framebuffer *fb,
+ struct drm_rect *clip)
+{
+ unsigned int cpp = drm_format_plane_cpp(fb->format->format, 0);
+ unsigned int offset = clip_offset(clip, fb->pitches[0], cpp);
+ size_t len = (clip->x2 - clip->x1) * cpp;
+ unsigned int y, lines = clip->y2 - clip->y1;
+
+ vaddr += offset;
+ dst += offset;
+ for (y = 0; y < lines; y++) {
+ memcpy_toio(dst, vaddr, len);
+ vaddr += fb->pitches[0];
+ dst += fb->pitches[0];
+ }
+}
+EXPORT_SYMBOL(drm_fb_memcpy_dstclip);
+
+/**
+ * drm_fb_swab16 - Swap bytes into clip buffer
+ * @dst: RGB565 destination buffer
+ * @vaddr: RGB565 source buffer
+ * @fb: DRM framebuffer
+ * @clip: Clip rectangle area to copy
+ */
+void drm_fb_swab16(u16 *dst, void *vaddr, struct drm_framebuffer *fb,
+ struct drm_rect *clip)
+{
+ size_t len = (clip->x2 - clip->x1) * sizeof(u16);
+ unsigned int x, y;
+ u16 *src, *buf;
+
+ /*
+ * The cma memory is write-combined so reads are uncached.
+ * Speed up by fetching one line at a time.
+ */
+ buf = kmalloc(len, GFP_KERNEL);
+ if (!buf)
+ return;
+
+ for (y = clip->y1; y < clip->y2; y++) {
+ src = vaddr + (y * fb->pitches[0]);
+ src += clip->x1;
+ memcpy(buf, src, len);
+ src = buf;
+ for (x = clip->x1; x < clip->x2; x++)
+ *dst++ = swab16(*src++);
+ }
+
+ kfree(buf);
+}
+EXPORT_SYMBOL(drm_fb_swab16);
+
+static void drm_fb_xrgb8888_to_rgb565_line(u16 *dbuf, u32 *sbuf,
+ unsigned int pixels,
+ bool swab)
+{
+ unsigned int x;
+ u16 val16;
+
+ for (x = 0; x < pixels; x++) {
+ val16 = ((sbuf[x] & 0x00F80000) >> 8) |
+ ((sbuf[x] & 0x0000FC00) >> 5) |
+ ((sbuf[x] & 0x000000F8) >> 3);
+ if (swab)
+ dbuf[x] = swab16(val16);
+ else
+ dbuf[x] = val16;
+ }
+}
+
+/**
+ * drm_fb_xrgb8888_to_rgb565 - Convert XRGB8888 to RGB565 clip buffer
+ * @dst: RGB565 destination buffer
+ * @vaddr: XRGB8888 source buffer
+ * @fb: DRM framebuffer
+ * @clip: Clip rectangle area to copy
+ * @swab: Swap bytes
+ *
+ * Drivers can use this function for RGB565 devices that don't natively
+ * support XRGB8888.
+ *
+ * This function does not apply clipping on dst, i.e. the destination
+ * is a small buffer containing the clip rect only.
+ */
+void drm_fb_xrgb8888_to_rgb565(void *dst, void *vaddr,
+ struct drm_framebuffer *fb,
+ struct drm_rect *clip, bool swab)
+{
+ size_t linepixels = clip->x2 - clip->x1;
+ size_t src_len = linepixels * sizeof(u32);
+ size_t dst_len = linepixels * sizeof(u16);
+ unsigned y, lines = clip->y2 - clip->y1;
+ void *sbuf;
+
+ /*
+ * The cma memory is write-combined so reads are uncached.
+ * Speed up by fetching one line at a time.
+ */
+ sbuf = kmalloc(src_len, GFP_KERNEL);
+ if (!sbuf)
+ return;
+
+ vaddr += clip_offset(clip, fb->pitches[0], sizeof(u32));
+ for (y = 0; y < lines; y++) {
+ memcpy(sbuf, vaddr, src_len);
+ drm_fb_xrgb8888_to_rgb565_line(dst, sbuf, linepixels, swab);
+ vaddr += fb->pitches[0];
+ dst += dst_len;
+ }
+
+ kfree(sbuf);
+}
+EXPORT_SYMBOL(drm_fb_xrgb8888_to_rgb565);
+
+/**
+ * drm_fb_xrgb8888_to_rgb565_dstclip - Convert XRGB8888 to RGB565 clip buffer
+ * @dst: RGB565 destination buffer (iomem)
+ * @dst_pitch: destination buffer pitch
+ * @vaddr: XRGB8888 source buffer
+ * @fb: DRM framebuffer
+ * @clip: Clip rectangle area to copy
+ * @swab: Swap bytes
+ *
+ * Drivers can use this function for RGB565 devices that don't natively
+ * support XRGB8888.
+ *
+ * This function applies clipping on dst, i.e. the destination is a
+ * full (iomem) framebuffer but only the clip rect content is copied over.
+ */
+void drm_fb_xrgb8888_to_rgb565_dstclip(void __iomem *dst, unsigned int dst_pitch,
+ void *vaddr, struct drm_framebuffer *fb,
+ struct drm_rect *clip, bool swab)
+{
+ size_t linepixels = clip->x2 - clip->x1;
+ size_t dst_len = linepixels * sizeof(u16);
+ unsigned y, lines = clip->y2 - clip->y1;
+ void *dbuf;
+
+ dbuf = kmalloc(dst_len, GFP_KERNEL);
+ if (!dbuf)
+ return;
+
+ vaddr += clip_offset(clip, fb->pitches[0], sizeof(u32));
+ dst += clip_offset(clip, dst_pitch, sizeof(u16));
+ for (y = 0; y < lines; y++) {
+ drm_fb_xrgb8888_to_rgb565_line(dbuf, vaddr, linepixels, swab);
+ memcpy_toio(dst, dbuf, dst_len);
+ vaddr += fb->pitches[0];
+ dst += dst_len;
+ }
+
+ kfree(dbuf);
+}
+EXPORT_SYMBOL(drm_fb_xrgb8888_to_rgb565_dstclip);
+
+static void drm_fb_xrgb8888_to_rgb888_line(u8 *dbuf, u32 *sbuf,
+ unsigned int pixels)
+{
+ unsigned int x;
+
+ for (x = 0; x < pixels; x++) {
+ *dbuf++ = (sbuf[x] & 0x000000FF) >> 0;
+ *dbuf++ = (sbuf[x] & 0x0000FF00) >> 8;
+ *dbuf++ = (sbuf[x] & 0x00FF0000) >> 16;
+ }
+}
+
+/**
+ * drm_fb_xrgb8888_to_rgb888_dstclip - Convert XRGB8888 to RGB888 clip buffer
+ * @dst: RGB565 destination buffer (iomem)
+ * @dst_pitch: destination buffer pitch
+ * @vaddr: XRGB8888 source buffer
+ * @fb: DRM framebuffer
+ * @clip: Clip rectangle area to copy
+ *
+ * Drivers can use this function for RGB888 devices that don't natively
+ * support XRGB8888.
+ *
+ * This function applies clipping on dst, i.e. the destination is a
+ * full (iomem) framebuffer but only the clip rect content is copied over.
+ */
+void drm_fb_xrgb8888_to_rgb888_dstclip(void __iomem *dst, unsigned int dst_pitch,
+ void *vaddr, struct drm_framebuffer *fb,
+ struct drm_rect *clip)
+{
+ size_t linepixels = clip->x2 - clip->x1;
+ size_t dst_len = linepixels * 3;
+ unsigned y, lines = clip->y2 - clip->y1;
+ void *dbuf;
+
+ dbuf = kmalloc(dst_len, GFP_KERNEL);
+ if (!dbuf)
+ return;
+
+ vaddr += clip_offset(clip, fb->pitches[0], sizeof(u32));
+ dst += clip_offset(clip, dst_pitch, sizeof(u16));
+ for (y = 0; y < lines; y++) {
+ drm_fb_xrgb8888_to_rgb888_line(dbuf, vaddr, linepixels);
+ memcpy_toio(dst, dbuf, dst_len);
+ vaddr += fb->pitches[0];
+ dst += dst_len;
+ }
+
+ kfree(dbuf);
+}
+EXPORT_SYMBOL(drm_fb_xrgb8888_to_rgb888_dstclip);
+
+/**
+ * drm_fb_xrgb8888_to_gray8 - Convert XRGB8888 to grayscale
+ * @dst: 8-bit grayscale destination buffer
+ * @vaddr: XRGB8888 source buffer
+ * @fb: DRM framebuffer
+ * @clip: Clip rectangle area to copy
+ *
+ * Drm doesn't have native monochrome or grayscale support.
+ * Such drivers can announce the commonly supported XR24 format to userspace
+ * and use this function to convert to the native format.
+ *
+ * Monochrome drivers will use the most significant bit,
+ * where 1 means foreground color and 0 background color.
+ *
+ * ITU BT.601 is used for the RGB -> luma (brightness) conversion.
+ */
+void drm_fb_xrgb8888_to_gray8(u8 *dst, void *vaddr, struct drm_framebuffer *fb,
+ struct drm_rect *clip)
+{
+ unsigned int len = (clip->x2 - clip->x1) * sizeof(u32);
+ unsigned int x, y;
+ void *buf;
+ u32 *src;
+
+ if (WARN_ON(fb->format->format != DRM_FORMAT_XRGB8888))
+ return;
+ /*
+ * The cma memory is write-combined so reads are uncached.
+ * Speed up by fetching one line at a time.
+ */
+ buf = kmalloc(len, GFP_KERNEL);
+ if (!buf)
+ return;
+
+ for (y = clip->y1; y < clip->y2; y++) {
+ src = vaddr + (y * fb->pitches[0]);
+ src += clip->x1;
+ memcpy(buf, src, len);
+ src = buf;
+ for (x = clip->x1; x < clip->x2; x++) {
+ u8 r = (*src & 0x00ff0000) >> 16;
+ u8 g = (*src & 0x0000ff00) >> 8;
+ u8 b = *src & 0x000000ff;
+
+ /* ITU BT.601: Y = 0.299 R + 0.587 G + 0.114 B */
+ *dst++ = (3 * r + 6 * g + b) / 10;
+ src++;
+ }
+ }
+
+ kfree(buf);
+}
+EXPORT_SYMBOL(drm_fb_xrgb8888_to_gray8);
+
{ .format = DRM_FORMAT_ABGR8888, .depth = 32, .num_planes = 1, .cpp = { 4, 0, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true },
{ .format = DRM_FORMAT_RGBA8888, .depth = 32, .num_planes = 1, .cpp = { 4, 0, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true },
{ .format = DRM_FORMAT_BGRA8888, .depth = 32, .num_planes = 1, .cpp = { 4, 0, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true },
+ { .format = DRM_FORMAT_XRGB16161616F, .depth = 0, .num_planes = 1, .cpp = { 8, 0, 0 }, .hsub = 1, .vsub = 1 },
+ { .format = DRM_FORMAT_XBGR16161616F, .depth = 0, .num_planes = 1, .cpp = { 8, 0, 0 }, .hsub = 1, .vsub = 1 },
+ { .format = DRM_FORMAT_ARGB16161616F, .depth = 0, .num_planes = 1, .cpp = { 8, 0, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true },
+ { .format = DRM_FORMAT_ABGR16161616F, .depth = 0, .num_planes = 1, .cpp = { 8, 0, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true },
{ .format = DRM_FORMAT_RGB888_A8, .depth = 32, .num_planes = 2, .cpp = { 3, 1, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true },
{ .format = DRM_FORMAT_BGR888_A8, .depth = 32, .num_planes = 2, .cpp = { 3, 1, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true },
{ .format = DRM_FORMAT_XRGB8888_A8, .depth = 32, .num_planes = 2, .cpp = { 4, 1, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true },
{ .format = DRM_FORMAT_UYVY, .depth = 0, .num_planes = 1, .cpp = { 2, 0, 0 }, .hsub = 2, .vsub = 1, .is_yuv = true },
{ .format = DRM_FORMAT_VYUY, .depth = 0, .num_planes = 1, .cpp = { 2, 0, 0 }, .hsub = 2, .vsub = 1, .is_yuv = true },
{ .format = DRM_FORMAT_XYUV8888, .depth = 0, .num_planes = 1, .cpp = { 4, 0, 0 }, .hsub = 1, .vsub = 1, .is_yuv = true },
+ { .format = DRM_FORMAT_VUY888, .depth = 0, .num_planes = 1, .cpp = { 3, 0, 0 }, .hsub = 1, .vsub = 1, .is_yuv = true },
{ .format = DRM_FORMAT_AYUV, .depth = 0, .num_planes = 1, .cpp = { 4, 0, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true, .is_yuv = true },
+ { .format = DRM_FORMAT_Y210, .depth = 0, .num_planes = 1, .cpp = { 4, 0, 0 }, .hsub = 2, .vsub = 1, .is_yuv = true },
+ { .format = DRM_FORMAT_Y212, .depth = 0, .num_planes = 1, .cpp = { 4, 0, 0 }, .hsub = 2, .vsub = 1, .is_yuv = true },
+ { .format = DRM_FORMAT_Y216, .depth = 0, .num_planes = 1, .cpp = { 4, 0, 0 }, .hsub = 2, .vsub = 1, .is_yuv = true },
+ { .format = DRM_FORMAT_Y410, .depth = 0, .num_planes = 1, .cpp = { 4, 0, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true, .is_yuv = true },
+ { .format = DRM_FORMAT_Y412, .depth = 0, .num_planes = 1, .cpp = { 8, 0, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true, .is_yuv = true },
+ { .format = DRM_FORMAT_Y416, .depth = 0, .num_planes = 1, .cpp = { 8, 0, 0 }, .hsub = 1, .vsub = 1, .has_alpha = true, .is_yuv = true },
+ { .format = DRM_FORMAT_XVYU2101010, .depth = 0, .num_planes = 1, .cpp = { 4, 0, 0 }, .hsub = 1, .vsub = 1, .is_yuv = true },
+ { .format = DRM_FORMAT_XVYU12_16161616, .depth = 0, .num_planes = 1, .cpp = { 8, 0, 0 }, .hsub = 1, .vsub = 1, .is_yuv = true },
+ { .format = DRM_FORMAT_XVYU16161616, .depth = 0, .num_planes = 1, .cpp = { 8, 0, 0 }, .hsub = 1, .vsub = 1, .is_yuv = true },
{ .format = DRM_FORMAT_Y0L0, .depth = 0, .num_planes = 1,
.char_per_block = { 8, 0, 0 }, .block_w = { 2, 0, 0 }, .block_h = { 2, 0, 0 },
.hsub = 2, .vsub = 2, .has_alpha = true, .is_yuv = true },
{ .format = DRM_FORMAT_P016, .depth = 0, .num_planes = 2,
.char_per_block = { 2, 4, 0 }, .block_w = { 1, 0, 0 }, .block_h = { 1, 0, 0 },
.hsub = 2, .vsub = 2, .is_yuv = true},
+ { .format = DRM_FORMAT_P210, .depth = 0,
+ .num_planes = 2, .char_per_block = { 2, 4, 0 },
+ .block_w = { 1, 0, 0 }, .block_h = { 1, 0, 0 }, .hsub = 2,
+ .vsub = 1, .is_yuv = true },
+ { .format = DRM_FORMAT_VUY101010, .depth = 0,
+ .num_planes = 1, .cpp = { 0, 0, 0 }, .hsub = 1, .vsub = 1,
+ .is_yuv = true },
+ { .format = DRM_FORMAT_YUV420_8BIT, .depth = 0,
+ .num_planes = 1, .cpp = { 0, 0, 0 }, .hsub = 2, .vsub = 2,
+ .is_yuv = true },
+ { .format = DRM_FORMAT_YUV420_10BIT, .depth = 0,
+ .num_planes = 1, .cpp = { 0, 0, 0 }, .hsub = 2, .vsub = 2,
+ .is_yuv = true },
};
unsigned int i;
* up at a later date, and as our interface with shmfs for memory allocation.
*/
-/*
- * We make up offsets for buffer objects so we can recognize them at
- * mmap time.
- */
-
-/* pgoff in mmap is an unsigned long, so we need to make sure that
- * the faked up offset will fit
- */
-
-#if BITS_PER_LONG == 64
-#define DRM_FILE_PAGE_OFFSET_START ((0xFFFFFFFFUL >> PAGE_SHIFT) + 1)
-#define DRM_FILE_PAGE_OFFSET_SIZE ((0xFFFFFFFFUL >> PAGE_SHIFT) * 16)
-#else
-#define DRM_FILE_PAGE_OFFSET_START ((0xFFFFFFFUL >> PAGE_SHIFT) + 1)
-#define DRM_FILE_PAGE_OFFSET_SIZE ((0xFFFFFFFUL >> PAGE_SHIFT) * 16)
-#endif
-
/**
* drm_gem_init - Initialize the GEM device fields
* @dev: drm_devic structure to initialize
kref_init(&obj->refcount);
obj->handle_count = 0;
obj->size = size;
+ reservation_object_init(&obj->_resv);
+ if (!obj->resv)
+ obj->resv = &obj->_resv;
+
drm_vma_node_reset(&obj->vma_node);
}
EXPORT_SYMBOL(drm_gem_private_object_init);
}
EXPORT_SYMBOL(drm_gem_put_pages);
+static int objects_lookup(struct drm_file *filp, u32 *handle, int count,
+ struct drm_gem_object **objs)
+{
+ int i, ret = 0;
+ struct drm_gem_object *obj;
+
+ spin_lock(&filp->table_lock);
+
+ for (i = 0; i < count; i++) {
+ /* Check if we currently have a reference on the object */
+ obj = idr_find(&filp->object_idr, handle[i]);
+ if (!obj) {
+ ret = -ENOENT;
+ break;
+ }
+ drm_gem_object_get(obj);
+ objs[i] = obj;
+ }
+ spin_unlock(&filp->table_lock);
+
+ return ret;
+}
+
+/**
+ * drm_gem_objects_lookup - look up GEM objects from an array of handles
+ * @filp: DRM file private date
+ * @bo_handles: user pointer to array of userspace handle
+ * @count: size of handle array
+ * @objs_out: returned pointer to array of drm_gem_object pointers
+ *
+ * Takes an array of userspace handles and returns a newly allocated array of
+ * GEM objects.
+ *
+ * For a single handle lookup, use drm_gem_object_lookup().
+ *
+ * Returns:
+ *
+ * @objs filled in with GEM object pointers. Returned GEM objects need to be
+ * released with drm_gem_object_put(). -ENOENT is returned on a lookup
+ * failure. 0 is returned on success.
+ *
+ */
+int drm_gem_objects_lookup(struct drm_file *filp, void __user *bo_handles,
+ int count, struct drm_gem_object ***objs_out)
+{
+ int ret;
+ u32 *handles;
+ struct drm_gem_object **objs;
+
+ if (!count)
+ return 0;
+
+ objs = kvmalloc_array(count, sizeof(struct drm_gem_object *),
+ GFP_KERNEL | __GFP_ZERO);
+ if (!objs)
+ return -ENOMEM;
+
+ handles = kvmalloc_array(count, sizeof(u32), GFP_KERNEL);
+ if (!handles) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ if (copy_from_user(handles, bo_handles, count * sizeof(u32))) {
+ ret = -EFAULT;
+ DRM_DEBUG("Failed to copy in GEM handles\n");
+ goto out;
+ }
+
+ ret = objects_lookup(filp, handles, count, objs);
+ *objs_out = objs;
+
+out:
+ kvfree(handles);
+ return ret;
+
+}
+EXPORT_SYMBOL(drm_gem_objects_lookup);
+
/**
* drm_gem_object_lookup - look up a GEM object from its handle
* @filp: DRM file private date
*
* A reference to the object named by the handle if such exists on @filp, NULL
* otherwise.
+ *
+ * If looking up an array of handles, use drm_gem_objects_lookup().
*/
struct drm_gem_object *
drm_gem_object_lookup(struct drm_file *filp, u32 handle)
{
+ struct drm_gem_object *obj = NULL;
+
+ objects_lookup(filp, &handle, 1, &obj);
+ return obj;
+}
+EXPORT_SYMBOL(drm_gem_object_lookup);
+
+/**
+ * drm_gem_reservation_object_wait - Wait on GEM object's reservation's objects
+ * shared and/or exclusive fences.
+ * @filep: DRM file private date
+ * @handle: userspace handle
+ * @wait_all: if true, wait on all fences, else wait on just exclusive fence
+ * @timeout: timeout value in jiffies or zero to return immediately
+ *
+ * Returns:
+ *
+ * Returns -ERESTARTSYS if interrupted, 0 if the wait timed out, or
+ * greater than 0 on success.
+ */
+long drm_gem_reservation_object_wait(struct drm_file *filep, u32 handle,
+ bool wait_all, unsigned long timeout)
+{
+ long ret;
struct drm_gem_object *obj;
- spin_lock(&filp->table_lock);
+ obj = drm_gem_object_lookup(filep, handle);
+ if (!obj) {
+ DRM_DEBUG("Failed to look up GEM BO %d\n", handle);
+ return -EINVAL;
+ }
- /* Check if we currently have a reference on the object */
- obj = idr_find(&filp->object_idr, handle);
- if (obj)
- drm_gem_object_get(obj);
+ ret = reservation_object_wait_timeout_rcu(obj->resv, wait_all,
+ true, timeout);
+ if (ret == 0)
+ ret = -ETIME;
+ else if (ret > 0)
+ ret = 0;
- spin_unlock(&filp->table_lock);
+ drm_gem_object_put_unlocked(obj);
- return obj;
+ return ret;
}
-EXPORT_SYMBOL(drm_gem_object_lookup);
+EXPORT_SYMBOL(drm_gem_reservation_object_wait);
/**
* drm_gem_close_ioctl - implementation of the GEM_CLOSE ioctl
if (obj->filp)
fput(obj->filp);
+ reservation_object_fini(&obj->_resv);
drm_gem_free_mmap_offset(obj);
}
EXPORT_SYMBOL(drm_gem_object_release);
obj->dev->driver->gem_prime_vunmap(obj, vaddr);
}
EXPORT_SYMBOL(drm_gem_vunmap);
+
+/**
+ * drm_gem_lock_reservations - Sets up the ww context and acquires
+ * the lock on an array of GEM objects.
+ *
+ * Once you've locked your reservations, you'll want to set up space
+ * for your shared fences (if applicable), submit your job, then
+ * drm_gem_unlock_reservations().
+ *
+ * @objs: drm_gem_objects to lock
+ * @count: Number of objects in @objs
+ * @acquire_ctx: struct ww_acquire_ctx that will be initialized as
+ * part of tracking this set of locked reservations.
+ */
+int
+drm_gem_lock_reservations(struct drm_gem_object **objs, int count,
+ struct ww_acquire_ctx *acquire_ctx)
+{
+ int contended = -1;
+ int i, ret;
+
+ ww_acquire_init(acquire_ctx, &reservation_ww_class);
+
+retry:
+ if (contended != -1) {
+ struct drm_gem_object *obj = objs[contended];
+
+ ret = ww_mutex_lock_slow_interruptible(&obj->resv->lock,
+ acquire_ctx);
+ if (ret) {
+ ww_acquire_done(acquire_ctx);
+ return ret;
+ }
+ }
+
+ for (i = 0; i < count; i++) {
+ if (i == contended)
+ continue;
+
+ ret = ww_mutex_lock_interruptible(&objs[i]->resv->lock,
+ acquire_ctx);
+ if (ret) {
+ int j;
+
+ for (j = 0; j < i; j++)
+ ww_mutex_unlock(&objs[j]->resv->lock);
+
+ if (contended != -1 && contended >= i)
+ ww_mutex_unlock(&objs[contended]->resv->lock);
+
+ if (ret == -EDEADLK) {
+ contended = i;
+ goto retry;
+ }
+
+ ww_acquire_done(acquire_ctx);
+ return ret;
+ }
+ }
+
+ ww_acquire_done(acquire_ctx);
+
+ return 0;
+}
+EXPORT_SYMBOL(drm_gem_lock_reservations);
+
+void
+drm_gem_unlock_reservations(struct drm_gem_object **objs, int count,
+ struct ww_acquire_ctx *acquire_ctx)
+{
+ int i;
+
+ for (i = 0; i < count; i++)
+ ww_mutex_unlock(&objs[i]->resv->lock);
+
+ ww_acquire_fini(acquire_ctx);
+}
+EXPORT_SYMBOL(drm_gem_unlock_reservations);
+
+/**
+ * drm_gem_fence_array_add - Adds the fence to an array of fences to be
+ * waited on, deduplicating fences from the same context.
+ *
+ * @fence_array: array of dma_fence * for the job to block on.
+ * @fence: the dma_fence to add to the list of dependencies.
+ *
+ * Returns:
+ * 0 on success, or an error on failing to expand the array.
+ */
+int drm_gem_fence_array_add(struct xarray *fence_array,
+ struct dma_fence *fence)
+{
+ struct dma_fence *entry;
+ unsigned long index;
+ u32 id = 0;
+ int ret;
+
+ if (!fence)
+ return 0;
+
+ /* Deduplicate if we already depend on a fence from the same context.
+ * This lets the size of the array of deps scale with the number of
+ * engines involved, rather than the number of BOs.
+ */
+ xa_for_each(fence_array, index, entry) {
+ if (entry->context != fence->context)
+ continue;
+
+ if (dma_fence_is_later(fence, entry)) {
+ dma_fence_put(entry);
+ xa_store(fence_array, index, fence, GFP_KERNEL);
+ } else {
+ dma_fence_put(fence);
+ }
+ return 0;
+ }
+
+ ret = xa_alloc(fence_array, &id, fence, xa_limit_32b, GFP_KERNEL);
+ if (ret != 0)
+ dma_fence_put(fence);
+
+ return ret;
+}
+EXPORT_SYMBOL(drm_gem_fence_array_add);
+
+/**
+ * drm_gem_fence_array_add_implicit - Adds the implicit dependencies tracked
+ * in the GEM object's reservation object to an array of dma_fences for use in
+ * scheduling a rendering job.
+ *
+ * This should be called after drm_gem_lock_reservations() on your array of
+ * GEM objects used in the job but before updating the reservations with your
+ * own fences.
+ *
+ * @fence_array: array of dma_fence * for the job to block on.
+ * @obj: the gem object to add new dependencies from.
+ * @write: whether the job might write the object (so we need to depend on
+ * shared fences in the reservation object).
+ */
+int drm_gem_fence_array_add_implicit(struct xarray *fence_array,
+ struct drm_gem_object *obj,
+ bool write)
+{
+ int ret;
+ struct dma_fence **fences;
+ unsigned int i, fence_count;
+
+ if (!write) {
+ struct dma_fence *fence =
+ reservation_object_get_excl_rcu(obj->resv);
+
+ return drm_gem_fence_array_add(fence_array, fence);
+ }
+
+ ret = reservation_object_get_fences_rcu(obj->resv, NULL,
+ &fence_count, &fences);
+ if (ret || !fence_count)
+ return ret;
+
+ for (i = 0; i < fence_count; i++) {
+ ret = drm_gem_fence_array_add(fence_array, fences[i]);
+ if (ret)
+ break;
+ }
+
+ for (; i < fence_count; i++)
+ dma_fence_put(fences[i]);
+ kfree(fences);
+ return ret;
+}
+EXPORT_SYMBOL(drm_gem_fence_array_add_implicit);
cma_obj = to_drm_gem_cma_obj(gem_obj);
- if (cma_obj->vaddr) {
- dma_free_wc(gem_obj->dev->dev, cma_obj->base.size,
- cma_obj->vaddr, cma_obj->paddr);
- } else if (gem_obj->import_attach) {
+ if (gem_obj->import_attach) {
if (cma_obj->vaddr)
dma_buf_vunmap(gem_obj->import_attach->dmabuf, cma_obj->vaddr);
drm_prime_gem_destroy(gem_obj, cma_obj->sgt);
+ } else if (cma_obj->vaddr) {
+ dma_free_wc(gem_obj->dev->dev, cma_obj->base.size,
+ cma_obj->vaddr, cma_obj->paddr);
}
drm_gem_object_release(gem_obj);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2018 Noralf Trønnes
+ */
+
+#include <linux/dma-buf.h>
+#include <linux/export.h>
+#include <linux/mutex.h>
+#include <linux/shmem_fs.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+
+#include <drm/drm_device.h>
+#include <drm/drm_drv.h>
+#include <drm/drm_gem_shmem_helper.h>
+#include <drm/drm_prime.h>
+#include <drm/drm_print.h>
+
+/**
+ * DOC: overview
+ *
+ * This library provides helpers for GEM objects backed by shmem buffers
+ * allocated using anonymous pageable memory.
+ */
+
+static const struct drm_gem_object_funcs drm_gem_shmem_funcs = {
+ .free = drm_gem_shmem_free_object,
+ .print_info = drm_gem_shmem_print_info,
+ .pin = drm_gem_shmem_pin,
+ .unpin = drm_gem_shmem_unpin,
+ .get_sg_table = drm_gem_shmem_get_sg_table,
+ .vmap = drm_gem_shmem_vmap,
+ .vunmap = drm_gem_shmem_vunmap,
+ .vm_ops = &drm_gem_shmem_vm_ops,
+};
+
+/**
+ * drm_gem_shmem_create - Allocate an object with the given size
+ * @dev: DRM device
+ * @size: Size of the object to allocate
+ *
+ * This function creates a shmem GEM object.
+ *
+ * Returns:
+ * A struct drm_gem_shmem_object * on success or an ERR_PTR()-encoded negative
+ * error code on failure.
+ */
+struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t size)
+{
+ struct drm_gem_shmem_object *shmem;
+ struct drm_gem_object *obj;
+ int ret;
+
+ size = PAGE_ALIGN(size);
+
+ if (dev->driver->gem_create_object)
+ obj = dev->driver->gem_create_object(dev, size);
+ else
+ obj = kzalloc(sizeof(*shmem), GFP_KERNEL);
+ if (!obj)
+ return ERR_PTR(-ENOMEM);
+
+ if (!obj->funcs)
+ obj->funcs = &drm_gem_shmem_funcs;
+
+ ret = drm_gem_object_init(dev, obj, size);
+ if (ret)
+ goto err_free;
+
+ ret = drm_gem_create_mmap_offset(obj);
+ if (ret)
+ goto err_release;
+
+ shmem = to_drm_gem_shmem_obj(obj);
+ mutex_init(&shmem->pages_lock);
+ mutex_init(&shmem->vmap_lock);
+
+ /*
+ * Our buffers are kept pinned, so allocating them
+ * from the MOVABLE zone is a really bad idea, and
+ * conflicts with CMA. See comments above new_inode()
+ * why this is required _and_ expected if you're
+ * going to pin these pages.
+ */
+ mapping_set_gfp_mask(obj->filp->f_mapping, GFP_HIGHUSER |
+ __GFP_RETRY_MAYFAIL | __GFP_NOWARN);
+
+ return shmem;
+
+err_release:
+ drm_gem_object_release(obj);
+err_free:
+ kfree(obj);
+
+ return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_create);
+
+/**
+ * drm_gem_shmem_free_object - Free resources associated with a shmem GEM object
+ * @obj: GEM object to free
+ *
+ * This function cleans up the GEM object state and frees the memory used to
+ * store the object itself.
+ */
+void drm_gem_shmem_free_object(struct drm_gem_object *obj)
+{
+ struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+
+ WARN_ON(shmem->vmap_use_count);
+
+ if (obj->import_attach) {
+ shmem->pages_use_count--;
+ drm_prime_gem_destroy(obj, shmem->sgt);
+ kvfree(shmem->pages);
+ } else {
+ if (shmem->sgt) {
+ dma_unmap_sg(obj->dev->dev, shmem->sgt->sgl,
+ shmem->sgt->nents, DMA_BIDIRECTIONAL);
+
+ drm_gem_shmem_put_pages(shmem);
+ sg_free_table(shmem->sgt);
+ kfree(shmem->sgt);
+ }
+ }
+
+ WARN_ON(shmem->pages_use_count);
+
+ drm_gem_object_release(obj);
+ mutex_destroy(&shmem->pages_lock);
+ mutex_destroy(&shmem->vmap_lock);
+ kfree(shmem);
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_free_object);
+
+static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+ struct drm_gem_object *obj = &shmem->base;
+ struct page **pages;
+
+ if (shmem->pages_use_count++ > 0)
+ return 0;
+
+ pages = drm_gem_get_pages(obj);
+ if (IS_ERR(pages)) {
+ DRM_DEBUG_KMS("Failed to get pages (%ld)\n", PTR_ERR(pages));
+ shmem->pages_use_count = 0;
+ return PTR_ERR(pages);
+ }
+
+ shmem->pages = pages;
+
+ return 0;
+}
+
+/*
+ * drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
+ * @shmem: shmem GEM object
+ *
+ * This function makes sure that backing pages exists for the shmem GEM object
+ * and increases the use count.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+{
+ int ret;
+
+ ret = mutex_lock_interruptible(&shmem->pages_lock);
+ if (ret)
+ return ret;
+ ret = drm_gem_shmem_get_pages_locked(shmem);
+ mutex_unlock(&shmem->pages_lock);
+
+ return ret;
+}
+EXPORT_SYMBOL(drm_gem_shmem_get_pages);
+
+static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+ struct drm_gem_object *obj = &shmem->base;
+
+ if (WARN_ON_ONCE(!shmem->pages_use_count))
+ return;
+
+ if (--shmem->pages_use_count > 0)
+ return;
+
+ drm_gem_put_pages(obj, shmem->pages,
+ shmem->pages_mark_dirty_on_put,
+ shmem->pages_mark_accessed_on_put);
+ shmem->pages = NULL;
+}
+
+/*
+ * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object
+ * @shmem: shmem GEM object
+ *
+ * This function decreases the use count and puts the backing pages when use drops to zero.
+ */
+void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
+{
+ mutex_lock(&shmem->pages_lock);
+ drm_gem_shmem_put_pages_locked(shmem);
+ mutex_unlock(&shmem->pages_lock);
+}
+EXPORT_SYMBOL(drm_gem_shmem_put_pages);
+
+/**
+ * drm_gem_shmem_pin - Pin backing pages for a shmem GEM object
+ * @obj: GEM object
+ *
+ * This function makes sure the backing pages are pinned in memory while the
+ * buffer is exported.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_pin(struct drm_gem_object *obj)
+{
+ struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+
+ return drm_gem_shmem_get_pages(shmem);
+}
+EXPORT_SYMBOL(drm_gem_shmem_pin);
+
+/**
+ * drm_gem_shmem_unpin - Unpin backing pages for a shmem GEM object
+ * @obj: GEM object
+ *
+ * This function removes the requirement that the backing pages are pinned in
+ * memory.
+ */
+void drm_gem_shmem_unpin(struct drm_gem_object *obj)
+{
+ struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+
+ drm_gem_shmem_put_pages(shmem);
+}
+EXPORT_SYMBOL(drm_gem_shmem_unpin);
+
+static void *drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem)
+{
+ struct drm_gem_object *obj = &shmem->base;
+ int ret;
+
+ if (shmem->vmap_use_count++ > 0)
+ return shmem->vaddr;
+
+ ret = drm_gem_shmem_get_pages(shmem);
+ if (ret)
+ goto err_zero_use;
+
+ if (obj->import_attach)
+ shmem->vaddr = dma_buf_vmap(obj->import_attach->dmabuf);
+ else
+ shmem->vaddr = vmap(shmem->pages, obj->size >> PAGE_SHIFT, VM_MAP, PAGE_KERNEL);
+
+ if (!shmem->vaddr) {
+ DRM_DEBUG_KMS("Failed to vmap pages\n");
+ ret = -ENOMEM;
+ goto err_put_pages;
+ }
+
+ return shmem->vaddr;
+
+err_put_pages:
+ drm_gem_shmem_put_pages(shmem);
+err_zero_use:
+ shmem->vmap_use_count = 0;
+
+ return ERR_PTR(ret);
+}
+
+/*
+ * drm_gem_shmem_vmap - Create a virtual mapping for a shmem GEM object
+ * @shmem: shmem GEM object
+ *
+ * This function makes sure that a virtual address exists for the buffer backing
+ * the shmem GEM object.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+void *drm_gem_shmem_vmap(struct drm_gem_object *obj)
+{
+ struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+ void *vaddr;
+ int ret;
+
+ ret = mutex_lock_interruptible(&shmem->vmap_lock);
+ if (ret)
+ return ERR_PTR(ret);
+ vaddr = drm_gem_shmem_vmap_locked(shmem);
+ mutex_unlock(&shmem->vmap_lock);
+
+ return vaddr;
+}
+EXPORT_SYMBOL(drm_gem_shmem_vmap);
+
+static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem)
+{
+ struct drm_gem_object *obj = &shmem->base;
+
+ if (WARN_ON_ONCE(!shmem->vmap_use_count))
+ return;
+
+ if (--shmem->vmap_use_count > 0)
+ return;
+
+ if (obj->import_attach)
+ dma_buf_vunmap(obj->import_attach->dmabuf, shmem->vaddr);
+ else
+ vunmap(shmem->vaddr);
+
+ shmem->vaddr = NULL;
+ drm_gem_shmem_put_pages(shmem);
+}
+
+/*
+ * drm_gem_shmem_vunmap - Unmap a virtual mapping fo a shmem GEM object
+ * @shmem: shmem GEM object
+ *
+ * This function removes the virtual address when use count drops to zero.
+ */
+void drm_gem_shmem_vunmap(struct drm_gem_object *obj, void *vaddr)
+{
+ struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+
+ mutex_lock(&shmem->vmap_lock);
+ drm_gem_shmem_vunmap_locked(shmem);
+ mutex_unlock(&shmem->vmap_lock);
+}
+EXPORT_SYMBOL(drm_gem_shmem_vunmap);
+
+struct drm_gem_shmem_object *
+drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
+ struct drm_device *dev, size_t size,
+ uint32_t *handle)
+{
+ struct drm_gem_shmem_object *shmem;
+ int ret;
+
+ shmem = drm_gem_shmem_create(dev, size);
+ if (IS_ERR(shmem))
+ return shmem;
+
+ /*
+ * Allocate an id of idr table where the obj is registered
+ * and handle has the id what user can see.
+ */
+ ret = drm_gem_handle_create(file_priv, &shmem->base, handle);
+ /* drop reference from allocate - handle holds it now. */
+ drm_gem_object_put_unlocked(&shmem->base);
+ if (ret)
+ return ERR_PTR(ret);
+
+ return shmem;
+}
+EXPORT_SYMBOL(drm_gem_shmem_create_with_handle);
+
+/**
+ * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object
+ * @file: DRM file structure to create the dumb buffer for
+ * @dev: DRM device
+ * @args: IOCTL data
+ *
+ * This function computes the pitch of the dumb buffer and rounds it up to an
+ * integer number of bytes per pixel. Drivers for hardware that doesn't have
+ * any additional restrictions on the pitch can directly use this function as
+ * their &drm_driver.dumb_create callback.
+ *
+ * For hardware with additional restrictions, drivers can adjust the fields
+ * set up by userspace before calling into this function.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_dumb_create(struct drm_file *file, struct drm_device *dev,
+ struct drm_mode_create_dumb *args)
+{
+ u32 min_pitch = DIV_ROUND_UP(args->width * args->bpp, 8);
+ struct drm_gem_shmem_object *shmem;
+
+ if (!args->pitch || !args->size) {
+ args->pitch = min_pitch;
+ args->size = args->pitch * args->height;
+ } else {
+ /* ensure sane minimum values */
+ if (args->pitch < min_pitch)
+ args->pitch = min_pitch;
+ if (args->size < args->pitch * args->height)
+ args->size = args->pitch * args->height;
+ }
+
+ shmem = drm_gem_shmem_create_with_handle(file, dev, args->size, &args->handle);
+
+ return PTR_ERR_OR_ZERO(shmem);
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_dumb_create);
+
+static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+ struct drm_gem_object *obj = vma->vm_private_data;
+ struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+ loff_t num_pages = obj->size >> PAGE_SHIFT;
+ struct page *page;
+
+ if (vmf->pgoff >= num_pages || WARN_ON_ONCE(!shmem->pages))
+ return VM_FAULT_SIGBUS;
+
+ page = shmem->pages[vmf->pgoff];
+
+ return vmf_insert_page(vma, vmf->address, page);
+}
+
+static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)
+{
+ struct drm_gem_object *obj = vma->vm_private_data;
+ struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+ int ret;
+
+ ret = drm_gem_shmem_get_pages(shmem);
+ WARN_ON_ONCE(ret != 0);
+
+ drm_gem_vm_open(vma);
+}
+
+static void drm_gem_shmem_vm_close(struct vm_area_struct *vma)
+{
+ struct drm_gem_object *obj = vma->vm_private_data;
+ struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+
+ drm_gem_shmem_put_pages(shmem);
+ drm_gem_vm_close(vma);
+}
+
+const struct vm_operations_struct drm_gem_shmem_vm_ops = {
+ .fault = drm_gem_shmem_fault,
+ .open = drm_gem_shmem_vm_open,
+ .close = drm_gem_shmem_vm_close,
+};
+EXPORT_SYMBOL_GPL(drm_gem_shmem_vm_ops);
+
+/**
+ * drm_gem_shmem_mmap - Memory-map a shmem GEM object
+ * @filp: File object
+ * @vma: VMA for the area to be mapped
+ *
+ * This function implements an augmented version of the GEM DRM file mmap
+ * operation for shmem objects. Drivers which employ the shmem helpers should
+ * use this function as their &file_operations.mmap handler in the DRM device file's
+ * file_operations structure.
+ *
+ * Instead of directly referencing this function, drivers should use the
+ * DEFINE_DRM_GEM_SHMEM_FOPS() macro.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+ struct drm_gem_shmem_object *shmem;
+ int ret;
+
+ ret = drm_gem_mmap(filp, vma);
+ if (ret)
+ return ret;
+
+ shmem = to_drm_gem_shmem_obj(vma->vm_private_data);
+
+ ret = drm_gem_shmem_get_pages(shmem);
+ if (ret) {
+ drm_gem_vm_close(vma);
+ return ret;
+ }
+
+ /* VM_PFNMAP was set by drm_gem_mmap() */
+ vma->vm_flags &= ~VM_PFNMAP;
+ vma->vm_flags |= VM_MIXEDMAP;
+
+ /* Remove the fake offset */
+ vma->vm_pgoff -= drm_vma_node_start(&shmem->base.vma_node);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);
+
+/**
+ * drm_gem_shmem_print_info() - Print &drm_gem_shmem_object info for debugfs
+ * @p: DRM printer
+ * @indent: Tab indentation level
+ * @obj: GEM object
+ */
+void drm_gem_shmem_print_info(struct drm_printer *p, unsigned int indent,
+ const struct drm_gem_object *obj)
+{
+ const struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+
+ drm_printf_indent(p, indent, "pages_use_count=%u\n", shmem->pages_use_count);
+ drm_printf_indent(p, indent, "vmap_use_count=%u\n", shmem->vmap_use_count);
+ drm_printf_indent(p, indent, "vaddr=%p\n", shmem->vaddr);
+}
+EXPORT_SYMBOL(drm_gem_shmem_print_info);
+
+/**
+ * drm_gem_shmem_get_sg_table - Provide a scatter/gather table of pinned
+ * pages for a shmem GEM object
+ * @obj: GEM object
+ *
+ * This function exports a scatter/gather table suitable for PRIME usage by
+ * calling the standard DMA mapping API.
+ *
+ * Returns:
+ * A pointer to the scatter/gather table of pinned pages or NULL on failure.
+ */
+struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_object *obj)
+{
+ struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+
+ return drm_prime_pages_to_sg(shmem->pages, obj->size >> PAGE_SHIFT);
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_get_sg_table);
+
+/**
+ * drm_gem_shmem_get_pages_sgt - Pin pages, dma map them, and return a
+ * scatter/gather table for a shmem GEM object.
+ * @obj: GEM object
+ *
+ * This function returns a scatter/gather table suitable for driver usage. If
+ * the sg table doesn't exist, the pages are pinned, dma-mapped, and a sg
+ * table created.
+ *
+ * Returns:
+ * A pointer to the scatter/gather table of pinned pages or errno on failure.
+ */
+struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_object *obj)
+{
+ int ret;
+ struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+ struct sg_table *sgt;
+
+ if (shmem->sgt)
+ return shmem->sgt;
+
+ WARN_ON(obj->import_attach);
+
+ ret = drm_gem_shmem_get_pages(shmem);
+ if (ret)
+ return ERR_PTR(ret);
+
+ sgt = drm_gem_shmem_get_sg_table(&shmem->base);
+ if (IS_ERR(sgt)) {
+ ret = PTR_ERR(sgt);
+ goto err_put_pages;
+ }
+ /* Map the pages for use by the h/w. */
+ dma_map_sg(obj->dev->dev, sgt->sgl, sgt->nents, DMA_BIDIRECTIONAL);
+
+ shmem->sgt = sgt;
+
+ return sgt;
+
+err_put_pages:
+ drm_gem_shmem_put_pages(shmem);
+ return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_get_pages_sgt);
+
+/**
+ * drm_gem_shmem_prime_import_sg_table - Produce a shmem GEM object from
+ * another driver's scatter/gather table of pinned pages
+ * @dev: Device to import into
+ * @attach: DMA-BUF attachment
+ * @sgt: Scatter/gather table of pinned pages
+ *
+ * This function imports a scatter/gather table exported via DMA-BUF by
+ * another driver. Drivers that use the shmem helpers should set this as their
+ * &drm_driver.gem_prime_import_sg_table callback.
+ *
+ * Returns:
+ * A pointer to a newly created GEM object or an ERR_PTR-encoded negative
+ * error code on failure.
+ */
+struct drm_gem_object *
+drm_gem_shmem_prime_import_sg_table(struct drm_device *dev,
+ struct dma_buf_attachment *attach,
+ struct sg_table *sgt)
+{
+ size_t size = PAGE_ALIGN(attach->dmabuf->size);
+ size_t npages = size >> PAGE_SHIFT;
+ struct drm_gem_shmem_object *shmem;
+ int ret;
+
+ shmem = drm_gem_shmem_create(dev, size);
+ if (IS_ERR(shmem))
+ return ERR_CAST(shmem);
+
+ shmem->pages = kvmalloc_array(npages, sizeof(struct page *), GFP_KERNEL);
+ if (!shmem->pages) {
+ ret = -ENOMEM;
+ goto err_free_gem;
+ }
+
+ ret = drm_prime_sg_to_page_addr_arrays(sgt, shmem->pages, NULL, npages);
+ if (ret < 0)
+ goto err_free_array;
+
+ shmem->sgt = sgt;
+ shmem->pages_use_count = 1; /* Permanently pinned from our point of view */
+
+ DRM_DEBUG_PRIME("size = %zu\n", size);
+
+ return &shmem->base;
+
+err_free_array:
+ kvfree(shmem->pages);
+err_free_gem:
+ drm_gem_object_put_unlocked(&shmem->base);
+
+ return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_prime_import_sg_table);
/* drm_irq.c */
/* IOCTLS */
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
int drm_legacy_irq_control(struct drm_device *dev, void *data,
struct drm_file *file_priv);
+#endif
int drm_crtc_get_sequence_ioctl(struct drm_device *dev, void *data,
struct drm_file *filp);
struct drm_file *file_private);
int drm_syncobj_fd_to_handle_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_private);
+int drm_syncobj_transfer_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private);
int drm_syncobj_wait_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_private);
+int drm_syncobj_timeline_wait_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private);
int drm_syncobj_reset_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_private);
int drm_syncobj_signal_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_private);
+int drm_syncobj_timeline_signal_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private);
+int drm_syncobj_query_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private);
/* drm_framebuffer.c */
void drm_framebuffer_print_info(struct drm_printer *p, unsigned int indent,
return -EINVAL;
}
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
typedef struct drm_map32 {
u32 offset; /* Requested physical address (0 for SAREA) */
u32 size; /* Requested physical size (bytes) */
map.handle = compat_ptr(handle);
return drm_ioctl_kernel(file, drm_legacy_rmmap_ioctl, &map, DRM_AUTH);
}
+#endif
typedef struct drm_client32 {
int idx; /* Which client desired? */
return 0;
}
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
typedef struct drm_buf_desc32 {
int count; /* Number of buffers of this size */
int size; /* Size in bytes */
return 0;
}
+#endif
#if IS_ENABLED(CONFIG_AGP)
typedef struct drm_agp_mode32 {
}
#endif /* CONFIG_AGP */
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
typedef struct drm_scatter_gather32 {
u32 size; /**< In bytes -- will round to page boundary */
u32 handle; /**< Used for mapping / unmapping */
return drm_ioctl_kernel(file, drm_legacy_sg_free, &request,
DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY);
}
-
+#endif
#if defined(CONFIG_X86)
typedef struct drm_update_draw32 {
drm_drawable_t handle;
#define DRM_IOCTL32_DEF(n, f) [DRM_IOCTL_NR(n##32)] = {.fn = f, .name = #n}
DRM_IOCTL32_DEF(DRM_IOCTL_VERSION, compat_drm_version),
DRM_IOCTL32_DEF(DRM_IOCTL_GET_UNIQUE, compat_drm_getunique),
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
DRM_IOCTL32_DEF(DRM_IOCTL_GET_MAP, compat_drm_getmap),
+#endif
DRM_IOCTL32_DEF(DRM_IOCTL_GET_CLIENT, compat_drm_getclient),
DRM_IOCTL32_DEF(DRM_IOCTL_GET_STATS, compat_drm_getstats),
DRM_IOCTL32_DEF(DRM_IOCTL_SET_UNIQUE, compat_drm_setunique),
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
DRM_IOCTL32_DEF(DRM_IOCTL_ADD_MAP, compat_drm_addmap),
DRM_IOCTL32_DEF(DRM_IOCTL_ADD_BUFS, compat_drm_addbufs),
DRM_IOCTL32_DEF(DRM_IOCTL_MARK_BUFS, compat_drm_markbufs),
DRM_IOCTL32_DEF(DRM_IOCTL_GET_SAREA_CTX, compat_drm_getsareactx),
DRM_IOCTL32_DEF(DRM_IOCTL_RES_CTX, compat_drm_resctx),
DRM_IOCTL32_DEF(DRM_IOCTL_DMA, compat_drm_dma),
+#endif
#if IS_ENABLED(CONFIG_AGP)
DRM_IOCTL32_DEF(DRM_IOCTL_AGP_ENABLE, compat_drm_agp_enable),
DRM_IOCTL32_DEF(DRM_IOCTL_AGP_INFO, compat_drm_agp_info),
DRM_IOCTL32_DEF(DRM_IOCTL_AGP_BIND, compat_drm_agp_bind),
DRM_IOCTL32_DEF(DRM_IOCTL_AGP_UNBIND, compat_drm_agp_unbind),
#endif
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
DRM_IOCTL32_DEF(DRM_IOCTL_SG_ALLOC, compat_drm_sg_alloc),
DRM_IOCTL32_DEF(DRM_IOCTL_SG_FREE, compat_drm_sg_free),
+#endif
#if defined(CONFIG_X86) || defined(CONFIG_IA64)
DRM_IOCTL32_DEF(DRM_IOCTL_UPDATE_DRAW, compat_drm_update_draw),
#endif
case DRM_CAP_SYNCOBJ:
req->value = drm_core_check_feature(dev, DRIVER_SYNCOBJ);
return 0;
+ case DRM_CAP_SYNCOBJ_TIMELINE:
+ req->value = drm_core_check_feature(dev, DRIVER_SYNCOBJ_TIMELINE);
+ return 0;
}
/* Other caps only work with KMS drivers */
return err;
}
-static inline bool
-drm_render_driver_and_ioctl(const struct drm_device *dev, u32 flags)
-{
- return drm_core_check_feature(dev, DRIVER_RENDER) &&
- (flags & DRM_RENDER_ALLOW);
-}
-
/**
* drm_ioctl_permit - Check ioctl permissions against caller
*
*/
int drm_ioctl_permit(u32 flags, struct drm_file *file_priv)
{
- const struct drm_device *dev = file_priv->minor->dev;
-
/* ROOT_ONLY is only for CAP_SYS_ADMIN */
if (unlikely((flags & DRM_ROOT_ONLY) && !capable(CAP_SYS_ADMIN)))
return -EACCES;
- /* AUTH is only for master ... */
- if (unlikely((flags & DRM_AUTH) && drm_is_primary_client(file_priv))) {
- /* authenticated ones, or render capable on DRM_RENDER_ALLOW. */
- if (!file_priv->authenticated &&
- !drm_render_driver_and_ioctl(dev, flags))
- return -EACCES;
- }
+ /* AUTH is only for authenticated or render client */
+ if (unlikely((flags & DRM_AUTH) && !drm_is_render_client(file_priv) &&
+ !file_priv->authenticated))
+ return -EACCES;
/* MASTER is only for master or control clients */
if (unlikely((flags & DRM_MASTER) &&
.name = #ioctl \
}
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
+#define DRM_LEGACY_IOCTL_DEF(ioctl, _func, _flags) DRM_IOCTL_DEF(ioctl, _func, _flags)
+#else
+#define DRM_LEGACY_IOCTL_DEF(ioctl, _func, _flags) DRM_IOCTL_DEF(ioctl, drm_invalid_op, _flags)
+#endif
+
/* Ioctl table */
static const struct drm_ioctl_desc drm_ioctls[] = {
DRM_IOCTL_DEF(DRM_IOCTL_VERSION, drm_version,
DRM_IOCTL_DEF(DRM_IOCTL_GET_UNIQUE, drm_getunique, DRM_UNLOCKED),
DRM_IOCTL_DEF(DRM_IOCTL_GET_MAGIC, drm_getmagic, DRM_UNLOCKED),
DRM_IOCTL_DEF(DRM_IOCTL_IRQ_BUSID, drm_irq_by_busid, DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_GET_MAP, drm_legacy_getmap_ioctl, DRM_UNLOCKED),
+
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_GET_MAP, drm_legacy_getmap_ioctl, DRM_UNLOCKED),
+
DRM_IOCTL_DEF(DRM_IOCTL_GET_CLIENT, drm_getclient, DRM_UNLOCKED),
DRM_IOCTL_DEF(DRM_IOCTL_GET_STATS, drm_getstats, DRM_UNLOCKED),
DRM_IOCTL_DEF(DRM_IOCTL_GET_CAP, drm_getcap, DRM_UNLOCKED|DRM_RENDER_ALLOW),
DRM_IOCTL_DEF(DRM_IOCTL_UNBLOCK, drm_noop, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
DRM_IOCTL_DEF(DRM_IOCTL_AUTH_MAGIC, drm_authmagic, DRM_UNLOCKED|DRM_MASTER),
- DRM_IOCTL_DEF(DRM_IOCTL_ADD_MAP, drm_legacy_addmap_ioctl, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_RM_MAP, drm_legacy_rmmap_ioctl, DRM_AUTH),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_ADD_MAP, drm_legacy_addmap_ioctl, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_RM_MAP, drm_legacy_rmmap_ioctl, DRM_AUTH),
- DRM_IOCTL_DEF(DRM_IOCTL_SET_SAREA_CTX, drm_legacy_setsareactx, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_GET_SAREA_CTX, drm_legacy_getsareactx, DRM_AUTH),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_SET_SAREA_CTX, drm_legacy_setsareactx, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_GET_SAREA_CTX, drm_legacy_getsareactx, DRM_AUTH),
DRM_IOCTL_DEF(DRM_IOCTL_SET_MASTER, drm_setmaster_ioctl, DRM_UNLOCKED|DRM_ROOT_ONLY),
DRM_IOCTL_DEF(DRM_IOCTL_DROP_MASTER, drm_dropmaster_ioctl, DRM_UNLOCKED|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_ADD_CTX, drm_legacy_addctx, DRM_AUTH|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_RM_CTX, drm_legacy_rmctx, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_MOD_CTX, drm_noop, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_GET_CTX, drm_legacy_getctx, DRM_AUTH),
- DRM_IOCTL_DEF(DRM_IOCTL_SWITCH_CTX, drm_legacy_switchctx, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_NEW_CTX, drm_legacy_newctx, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_RES_CTX, drm_legacy_resctx, DRM_AUTH),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_ADD_CTX, drm_legacy_addctx, DRM_AUTH|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_RM_CTX, drm_legacy_rmctx, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_MOD_CTX, drm_noop, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_GET_CTX, drm_legacy_getctx, DRM_AUTH),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_SWITCH_CTX, drm_legacy_switchctx, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_NEW_CTX, drm_legacy_newctx, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_RES_CTX, drm_legacy_resctx, DRM_AUTH),
DRM_IOCTL_DEF(DRM_IOCTL_ADD_DRAW, drm_noop, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
DRM_IOCTL_DEF(DRM_IOCTL_RM_DRAW, drm_noop, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_LOCK, drm_legacy_lock, DRM_AUTH),
- DRM_IOCTL_DEF(DRM_IOCTL_UNLOCK, drm_legacy_unlock, DRM_AUTH),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_LOCK, drm_legacy_lock, DRM_AUTH),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_UNLOCK, drm_legacy_unlock, DRM_AUTH),
DRM_IOCTL_DEF(DRM_IOCTL_FINISH, drm_noop, DRM_AUTH),
- DRM_IOCTL_DEF(DRM_IOCTL_ADD_BUFS, drm_legacy_addbufs, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_MARK_BUFS, drm_legacy_markbufs, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_INFO_BUFS, drm_legacy_infobufs, DRM_AUTH),
- DRM_IOCTL_DEF(DRM_IOCTL_MAP_BUFS, drm_legacy_mapbufs, DRM_AUTH),
- DRM_IOCTL_DEF(DRM_IOCTL_FREE_BUFS, drm_legacy_freebufs, DRM_AUTH),
- DRM_IOCTL_DEF(DRM_IOCTL_DMA, drm_legacy_dma_ioctl, DRM_AUTH),
-
- DRM_IOCTL_DEF(DRM_IOCTL_CONTROL, drm_legacy_irq_control, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_ADD_BUFS, drm_legacy_addbufs, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_MARK_BUFS, drm_legacy_markbufs, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_INFO_BUFS, drm_legacy_infobufs, DRM_AUTH),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_MAP_BUFS, drm_legacy_mapbufs, DRM_AUTH),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_FREE_BUFS, drm_legacy_freebufs, DRM_AUTH),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_DMA, drm_legacy_dma_ioctl, DRM_AUTH),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_CONTROL, drm_legacy_irq_control, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
#if IS_ENABLED(CONFIG_AGP)
DRM_IOCTL_DEF(DRM_IOCTL_AGP_ACQUIRE, drm_agp_acquire_ioctl, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
DRM_IOCTL_DEF(DRM_IOCTL_AGP_UNBIND, drm_agp_unbind_ioctl, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
#endif
- DRM_IOCTL_DEF(DRM_IOCTL_SG_ALLOC, drm_legacy_sg_alloc, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
- DRM_IOCTL_DEF(DRM_IOCTL_SG_FREE, drm_legacy_sg_free, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_SG_ALLOC, drm_legacy_sg_alloc, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
+ DRM_LEGACY_IOCTL_DEF(DRM_IOCTL_SG_FREE, drm_legacy_sg_free, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
DRM_IOCTL_DEF(DRM_IOCTL_WAIT_VBLANK, drm_wait_vblank_ioctl, DRM_UNLOCKED),
DRM_UNLOCKED|DRM_RENDER_ALLOW),
DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_FD_TO_HANDLE, drm_syncobj_fd_to_handle_ioctl,
DRM_UNLOCKED|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_TRANSFER, drm_syncobj_transfer_ioctl,
+ DRM_UNLOCKED|DRM_RENDER_ALLOW),
DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_WAIT, drm_syncobj_wait_ioctl,
DRM_UNLOCKED|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_TIMELINE_WAIT, drm_syncobj_timeline_wait_ioctl,
+ DRM_UNLOCKED|DRM_RENDER_ALLOW),
DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_RESET, drm_syncobj_reset_ioctl,
DRM_UNLOCKED|DRM_RENDER_ALLOW),
DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_SIGNAL, drm_syncobj_signal_ioctl,
DRM_UNLOCKED|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_TIMELINE_SIGNAL, drm_syncobj_timeline_signal_ioctl,
+ DRM_UNLOCKED|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_QUERY, drm_syncobj_query_ioctl,
+ DRM_UNLOCKED|DRM_RENDER_ALLOW),
DRM_IOCTL_DEF(DRM_IOCTL_CRTC_GET_SEQUENCE, drm_crtc_get_sequence_ioctl, DRM_UNLOCKED),
DRM_IOCTL_DEF(DRM_IOCTL_CRTC_QUEUE_SEQUENCE, drm_crtc_queue_sequence_ioctl, DRM_UNLOCKED),
DRM_IOCTL_DEF(DRM_IOCTL_MODE_CREATE_LEASE, drm_mode_create_lease_ioctl, DRM_MASTER|DRM_UNLOCKED),
}
EXPORT_SYMBOL(drm_irq_uninstall);
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
int drm_legacy_irq_control(struct drm_device *dev, void *data,
struct drm_file *file_priv)
{
return -EINVAL;
}
}
+#endif
/* Backward compatibility for drm_kms_helper.edid_firmware */
static int edid_firmware_set(const char *val, const struct kernel_param *kp)
{
- DRM_NOTE("drm_kms_firmware.edid_firmware is deprecated, please use drm.edid_firmware intead.\n");
+ DRM_NOTE("drm_kms_firmware.edid_firmware is deprecated, please use drm.edid_firmware instead.\n");
return __drm_set_edid_firmware_path(val);
}
*/
bool _drm_lease_held(struct drm_file *file_priv, int id)
{
- if (file_priv == NULL || file_priv->master == NULL)
+ if (!file_priv || !file_priv->master)
return true;
return _drm_lease_held_master(file_priv->master, id);
struct drm_master *master;
bool ret;
- if (file_priv == NULL || file_priv->master == NULL)
+ if (!file_priv || !file_priv->master || !file_priv->master->lessor)
return true;
master = file_priv->master;
int count_in, count_out;
uint32_t crtcs_out = 0;
- if (file_priv == NULL || file_priv->master == NULL)
+ if (!file_priv || !file_priv->master || !file_priv->master->lessor)
return crtcs_in;
master = file_priv->master;
error = 0;
if (!idr_find(&dev->mode_config.object_idr, object))
error = -ENOENT;
- else if (!_drm_lease_held_master(lessor, object))
- error = -EACCES;
else if (_drm_has_leased(lessor, object))
error = -EBUSY;
/* step one - get references to all the mode objects
and check for validity. */
for (o = 0; o < object_count; o++) {
- if ((int) object_ids[o] < 0) {
- ret = -EINVAL;
- goto out_free_objects;
- }
-
objects[o] = drm_mode_object_find(dev, lessor_priv,
object_ids[o],
DRM_MODE_OBJECT_ANY);
#define DRM_KERNEL_CONTEXT 0
#define DRM_RESERVED_CONTEXTS 1
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
void drm_legacy_ctxbitmap_init(struct drm_device *dev);
void drm_legacy_ctxbitmap_cleanup(struct drm_device *dev);
-void drm_legacy_ctxbitmap_free(struct drm_device *dev, int ctx_handle);
void drm_legacy_ctxbitmap_flush(struct drm_device *dev, struct drm_file *file);
+#else
+static inline void drm_legacy_ctxbitmap_init(struct drm_device *dev) {}
+static inline void drm_legacy_ctxbitmap_cleanup(struct drm_device *dev) {}
+static inline void drm_legacy_ctxbitmap_flush(struct drm_device *dev, struct drm_file *file) {}
+#endif
+void drm_legacy_ctxbitmap_free(struct drm_device *dev, int ctx_handle);
+
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
int drm_legacy_resctx(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_addctx(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_getctx(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_setsareactx(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_getsareactx(struct drm_device *d, void *v, struct drm_file *f);
+#endif
/*
* Generic Buffer Management
#define DRM_MAP_HASH_OFFSET 0x10000000
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
+static inline int drm_legacy_create_map_hash(struct drm_device *dev)
+{
+ return drm_ht_create(&dev->map_hash, 12);
+}
+
+static inline void drm_legacy_remove_map_hash(struct drm_device *dev)
+{
+ drm_ht_remove(&dev->map_hash);
+}
+#else
+static inline int drm_legacy_create_map_hash(struct drm_device *dev)
+{
+ return 0;
+}
+
+static inline void drm_legacy_remove_map_hash(struct drm_device *dev) {}
+#endif
+
+
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
int drm_legacy_getmap_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_priv);
int drm_legacy_addmap_ioctl(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_rmmap_ioctl(struct drm_device *d, void *v, struct drm_file *f);
+
int drm_legacy_addbufs(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_infobufs(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_markbufs(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_freebufs(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_mapbufs(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_dma_ioctl(struct drm_device *d, void *v, struct drm_file *f);
+#endif
int __drm_legacy_infobufs(struct drm_device *, void *, int *,
int (*)(void *, int, struct drm_buf_entry *));
int (*)(void *, int, unsigned long, struct drm_buf *),
struct drm_file *);
-#ifdef CONFIG_DRM_VM
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
+void drm_legacy_master_rmmaps(struct drm_device *dev,
+ struct drm_master *master);
+void drm_legacy_rmmaps(struct drm_device *dev);
+#else
+static inline void drm_legacy_master_rmmaps(struct drm_device *dev,
+ struct drm_master *master) {}
+static inline void drm_legacy_rmmaps(struct drm_device *dev) {}
+#endif
+
+#if IS_ENABLED(CONFIG_DRM_VM) && IS_ENABLED(CONFIG_DRM_LEGACY)
void drm_legacy_vma_flush(struct drm_device *d);
#else
static inline void drm_legacy_vma_flush(struct drm_device *d)
};
/* drm_lock.c */
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
int drm_legacy_lock(struct drm_device *d, void *v, struct drm_file *f);
int drm_legacy_unlock(struct drm_device *d, void *v, struct drm_file *f);
void drm_legacy_lock_release(struct drm_device *dev, struct file *filp);
+#else
+static inline void drm_legacy_lock_release(struct drm_device *dev, struct file *filp) {}
+#endif
/* DMA support */
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
int drm_legacy_dma_setup(struct drm_device *dev);
void drm_legacy_dma_takedown(struct drm_device *dev);
+#else
+static inline int drm_legacy_dma_setup(struct drm_device *dev)
+{
+ return 0;
+}
+#endif
+
void drm_legacy_free_buffer(struct drm_device *dev,
struct drm_buf * buf);
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
void drm_legacy_reclaim_buffers(struct drm_device *dev,
struct drm_file *filp);
+#else
+static inline void drm_legacy_reclaim_buffers(struct drm_device *dev,
+ struct drm_file *filp) {}
+#endif
/* Scatter Gather Support */
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
void drm_legacy_sg_cleanup(struct drm_device *dev);
int drm_legacy_sg_alloc(struct drm_device *dev, void *data,
struct drm_file *file_priv);
int drm_legacy_sg_free(struct drm_device *dev, void *data,
struct drm_file *file_priv);
+#endif
+
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
+void drm_legacy_init_members(struct drm_device *dev);
+void drm_legacy_destroy_members(struct drm_device *dev);
+void drm_legacy_dev_reinit(struct drm_device *dev);
+#else
+static inline void drm_legacy_init_members(struct drm_device *dev) {}
+static inline void drm_legacy_destroy_members(struct drm_device *dev) {}
+static inline void drm_legacy_dev_reinit(struct drm_device *dev) {}
+#endif
+
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
+void drm_legacy_lock_master_cleanup(struct drm_device *dev, struct drm_master *master);
+#else
+static inline void drm_legacy_lock_master_cleanup(struct drm_device *dev, struct drm_master *master) {}
+#endif
+
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
+void drm_master_legacy_init(struct drm_master *master);
+#else
+static inline void drm_master_legacy_init(struct drm_master *master) {}
+#endif
#endif /* __DRM_LEGACY_H__ */
--- /dev/null
+/**
+ * \file drm_legacy_misc.c
+ * Misc legacy support functions.
+ *
+ * \author Rickard E. (Rik) Faith <faith@valinux.com>
+ * \author Gareth Hughes <gareth@valinux.com>
+ */
+
+/*
+ * Created: Tue Feb 2 08:37:54 1999 by faith@valinux.com
+ *
+ * Copyright 1999 Precision Insight, Inc., Cedar Park, Texas.
+ * Copyright 2000 VA Linux Systems, Inc., Sunnyvale, California.
+ * All Rights Reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * VA LINUX SYSTEMS AND/OR ITS SUPPLIERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ */
+
+#include <drm/drmP.h>
+#include "drm_internal.h"
+#include "drm_legacy.h"
+
+void drm_legacy_init_members(struct drm_device *dev)
+{
+ INIT_LIST_HEAD(&dev->ctxlist);
+ INIT_LIST_HEAD(&dev->vmalist);
+ INIT_LIST_HEAD(&dev->maplist);
+ spin_lock_init(&dev->buf_lock);
+ mutex_init(&dev->ctxlist_mutex);
+}
+
+void drm_legacy_destroy_members(struct drm_device *dev)
+{
+ mutex_destroy(&dev->ctxlist_mutex);
+}
+
+void drm_legacy_dev_reinit(struct drm_device *dev)
+{
+ if (dev->irq_enabled)
+ drm_irq_uninstall(dev);
+
+ mutex_lock(&dev->struct_mutex);
+
+ drm_legacy_agp_clear(dev);
+
+ drm_legacy_sg_cleanup(dev);
+ drm_legacy_vma_flush(dev);
+ drm_legacy_dma_takedown(dev);
+
+ mutex_unlock(&dev->struct_mutex);
+
+ dev->sigdata.lock = NULL;
+
+ dev->context_flag = 0;
+ dev->last_context = 0;
+ dev->if_version = 0;
+
+ DRM_DEBUG("lastclose completed\n");
+}
+
+void drm_master_legacy_init(struct drm_master *master)
+{
+ spin_lock_init(&master->lock.spinlock);
+ init_waitqueue_head(&master->lock.lock_queue);
+}
_DRM_LOCKING_CONTEXT(file_priv->master->lock.hw_lock->lock));
}
}
+
+void drm_legacy_lock_master_cleanup(struct drm_device *dev, struct drm_master *master)
+{
+ if (!drm_core_check_feature(dev, DRIVER_LEGACY))
+ return;
+
+ /*
+ * Since the master is disappearing, so is the
+ * possibility to lock.
+ */ mutex_lock(&dev->struct_mutex);
+ if (master->lock.hw_lock) {
+ if (dev->sigdata.lock == master->lock.hw_lock)
+ dev->sigdata.lock = NULL;
+ master->lock.hw_lock = NULL;
+ master->lock.file_priv = NULL;
+ wake_up_interruptible_all(&master->lock.lock_queue);
+ }
+ mutex_unlock(&dev->struct_mutex);
+}
#include <linux/highmem.h>
#include <linux/export.h>
+#include <xen/xen.h>
#include <drm/drmP.h>
#include "drm_legacy.h"
}
EXPORT_SYMBOL(drm_legacy_ioremapfree);
-u64 drm_get_max_iomem(void)
+bool drm_need_swiotlb(int dma_bits)
{
struct resource *tmp;
resource_size_t max_iomem = 0;
+ /*
+ * Xen paravirtual hosts require swiotlb regardless of requested dma
+ * transfer size.
+ *
+ * NOTE: Really, what it requires is use of the dma_alloc_coherent
+ * allocator used in ttm_dma_populate() instead of
+ * ttm_populate_and_map_pages(), which bounce buffers so much in
+ * Xen it leads to swiotlb buffer exhaustion.
+ */
+ if (xen_pv_domain())
+ return true;
+
+ /*
+ * Enforce dma_alloc_coherent when memory encryption is active as well
+ * for the same reasons as for Xen paravirtual hosts.
+ */
+ if (mem_encrypt_active())
+ return true;
+
for (tmp = iomem_resource.child; tmp; tmp = tmp->sibling) {
max_iomem = max(max_iomem, tmp->end);
}
- return max_iomem;
+ return max_iomem > ((u64)1 << dma_bits);
}
-EXPORT_SYMBOL(drm_get_max_iomem);
+EXPORT_SYMBOL(drm_need_swiotlb);
return -ENOMEM;
dev->mode_config.prop_crtc_id = prop;
- prop = drm_property_create(dev, DRM_MODE_PROP_BLOB, "FB_DAMAGE_CLIPS",
- 0);
+ prop = drm_property_create(dev,
+ DRM_MODE_PROP_ATOMIC | DRM_MODE_PROP_BLOB,
+ "FB_DAMAGE_CLIPS", 0);
if (!prop)
return -ENOMEM;
dev->mode_config.prop_fb_damage_clips = prop;
}
static int set_property_atomic(struct drm_mode_object *obj,
+ struct drm_file *file_priv,
struct drm_property *prop,
uint64_t prop_value)
{
obj_to_connector(obj),
prop_value);
} else {
- ret = drm_atomic_set_property(state, obj, prop, prop_value);
+ ret = drm_atomic_set_property(state, file_priv, obj, prop, prop_value);
if (ret)
goto out;
ret = drm_atomic_commit(state);
goto out_unref;
if (drm_drv_uses_atomic_modeset(property->dev))
- ret = set_property_atomic(arg_obj, property, arg->value);
+ ret = set_property_atomic(arg_obj, file_priv, property, arg->value);
else
ret = set_property_legacy(arg_obj, property, arg->value);
* @bus_flags: information about pixelclk, sync and DE polarity will be stored
* here
*
- * Sets DRM_BUS_FLAG_DE_(LOW|HIGH), DRM_BUS_FLAG_PIXDATA_(POS|NEG)EDGE and
- * DISPLAY_FLAGS_SYNC_(POS|NEG)EDGE in @bus_flags according to DISPLAY_FLAGS
+ * Sets DRM_BUS_FLAG_DE_(LOW|HIGH), DRM_BUS_FLAG_PIXDATA_DRIVE_(POS|NEG)EDGE
+ * and DISPLAY_FLAGS_SYNC_(POS|NEG)EDGE in @bus_flags according to DISPLAY_FLAGS
* found in @vm
*/
void drm_bus_flags_from_videomode(const struct videomode *vm, u32 *bus_flags)
{
*bus_flags = 0;
if (vm->flags & DISPLAY_FLAGS_PIXDATA_POSEDGE)
- *bus_flags |= DRM_BUS_FLAG_PIXDATA_POSEDGE;
+ *bus_flags |= DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE;
if (vm->flags & DISPLAY_FLAGS_PIXDATA_NEGEDGE)
- *bus_flags |= DRM_BUS_FLAG_PIXDATA_NEGEDGE;
+ *bus_flags |= DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE;
if (vm->flags & DISPLAY_FLAGS_SYNC_POSEDGE)
- *bus_flags |= DRM_BUS_FLAG_SYNC_POSEDGE;
+ *bus_flags |= DRM_BUS_FLAG_SYNC_DRIVE_POSEDGE;
if (vm->flags & DISPLAY_FLAGS_SYNC_NEGEDGE)
- *bus_flags |= DRM_BUS_FLAG_SYNC_NEGEDGE;
+ *bus_flags |= DRM_BUS_FLAG_SYNC_DRIVE_NEGEDGE;
if (vm->flags & DISPLAY_FLAGS_DE_LOW)
*bus_flags |= DRM_BUS_FLAG_DE_LOW;
.orientation = DRM_MODE_PANEL_ORIENTATION_RIGHT_UP,
};
+static const struct drm_dmi_panel_orientation_data lcd1200x1920_rightside_up = {
+ .width = 1200,
+ .height = 1920,
+ .orientation = DRM_MODE_PANEL_ORIENTATION_RIGHT_UP,
+};
+
static const struct dmi_system_id orientation_data[] = {
{ /* Acer One 10 (S1003) */
.matches = {
DMI_EXACT_MATCH(DMI_PRODUCT_VERSION, "Lenovo MIIX 320-10ICR"),
},
.driver_data = (void *)&lcd800x1280_rightside_up,
+ }, { /* Lenovo Ideapad D330 */
+ .matches = {
+ DMI_EXACT_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+ DMI_EXACT_MATCH(DMI_PRODUCT_NAME, "81H3"),
+ DMI_EXACT_MATCH(DMI_PRODUCT_VERSION, "Lenovo ideapad D330-10IGM"),
+ },
+ .driver_data = (void *)&lcd1200x1920_rightside_up,
}, { /* VIOS LTH17 */
.matches = {
DMI_EXACT_MATCH(DMI_SYS_VENDOR, "VIOS"),
if (ret)
goto out;
+ if (!drm_lease_held(file_priv, crtc->cursor->base.id)) {
+ ret = -EACCES;
+ goto out;
+ }
+
ret = drm_mode_cursor_universal(crtc, req, file_priv, &ctx);
goto out;
}
plane = crtc->primary;
+ if (!drm_lease_held(file_priv, plane->base.id))
+ return -EACCES;
+
if (crtc->funcs->page_flip_target) {
u32 current_vblank;
int r;
.size = obj->size,
.flags = flags,
.priv = obj,
+ .resv = obj->resv,
};
if (dev->driver->gem_prime_res_obj)
va_end(args);
}
EXPORT_SYMBOL(drm_err);
+
+/**
+ * drm_print_regset32 - print the contents of registers to a
+ * &drm_printer stream.
+ *
+ * @p: the &drm printer
+ * @regset: the list of registers to print.
+ *
+ * Often in driver debug, it's useful to be able to either capture the
+ * contents of registers in the steady state using debugfs or at
+ * specific points during operation. This lets the driver have a
+ * single list of registers for both.
+ */
+void drm_print_regset32(struct drm_printer *p, struct debugfs_regset32 *regset)
+{
+ int namelen = 0;
+ int i;
+
+ for (i = 0; i < regset->nregs; i++)
+ namelen = max(namelen, (int)strlen(regset->regs[i].name));
+
+ for (i = 0; i < regset->nregs; i++) {
+ drm_printf(p, "%*s = 0x%08x\n",
+ namelen, regset->regs[i].name,
+ readl(regset->base + regset->regs[i].offset));
+ }
+}
+EXPORT_SYMBOL(drm_print_regset32);
struct task_struct *task;
struct dma_fence *fence;
struct dma_fence_cb fence_cb;
+ u64 point;
};
static void syncobj_wait_syncobj_func(struct drm_syncobj *syncobj,
static void drm_syncobj_fence_add_wait(struct drm_syncobj *syncobj,
struct syncobj_wait_entry *wait)
{
+ struct dma_fence *fence;
+
if (wait->fence)
return;
* have the lock, try one more time just to be sure we don't add a
* callback when a fence has already been set.
*/
- if (syncobj->fence)
- wait->fence = dma_fence_get(
- rcu_dereference_protected(syncobj->fence, 1));
- else
+ fence = dma_fence_get(rcu_dereference_protected(syncobj->fence, 1));
+ if (!fence || dma_fence_chain_find_seqno(&fence, wait->point)) {
+ dma_fence_put(fence);
list_add_tail(&wait->node, &syncobj->cb_list);
+ } else if (!fence) {
+ wait->fence = dma_fence_get_stub();
+ } else {
+ wait->fence = fence;
+ }
spin_unlock(&syncobj->lock);
}
spin_unlock(&syncobj->lock);
}
+/**
+ * drm_syncobj_add_point - add new timeline point to the syncobj
+ * @syncobj: sync object to add timeline point do
+ * @chain: chain node to use to add the point
+ * @fence: fence to encapsulate in the chain node
+ * @point: sequence number to use for the point
+ *
+ * Add the chain node as new timeline point to the syncobj.
+ */
+void drm_syncobj_add_point(struct drm_syncobj *syncobj,
+ struct dma_fence_chain *chain,
+ struct dma_fence *fence,
+ uint64_t point)
+{
+ struct syncobj_wait_entry *cur, *tmp;
+ struct dma_fence *prev;
+
+ dma_fence_get(fence);
+
+ spin_lock(&syncobj->lock);
+
+ prev = drm_syncobj_fence_get(syncobj);
+ /* You are adding an unorder point to timeline, which could cause payload returned from query_ioctl is 0! */
+ if (prev && prev->seqno >= point)
+ DRM_ERROR("You are adding an unorder point to timeline!\n");
+ dma_fence_chain_init(chain, prev, fence, point);
+ rcu_assign_pointer(syncobj->fence, &chain->base);
+
+ list_for_each_entry_safe(cur, tmp, &syncobj->cb_list, node)
+ syncobj_wait_syncobj_func(syncobj, cur);
+ spin_unlock(&syncobj->lock);
+
+ /* Walk the chain once to trigger garbage collection */
+ dma_fence_chain_for_each(fence, prev);
+ dma_fence_put(prev);
+}
+EXPORT_SYMBOL(drm_syncobj_add_point);
+
/**
* drm_syncobj_replace_fence - replace fence in a sync object.
* @syncobj: Sync object to replace fence in
rcu_assign_pointer(syncobj->fence, fence);
if (fence != old_fence) {
- list_for_each_entry_safe(cur, tmp, &syncobj->cb_list, node) {
- list_del_init(&cur->node);
+ list_for_each_entry_safe(cur, tmp, &syncobj->cb_list, node)
syncobj_wait_syncobj_func(syncobj, cur);
- }
}
spin_unlock(&syncobj->lock);
dma_fence_put(fence);
}
+/* 5s default for wait submission */
+#define DRM_SYNCOBJ_WAIT_FOR_SUBMIT_TIMEOUT 5000000000ULL
/**
* drm_syncobj_find_fence - lookup and reference the fence in a sync object
* @file_private: drm file private pointer
struct dma_fence **fence)
{
struct drm_syncobj *syncobj = drm_syncobj_find(file_private, handle);
- int ret = 0;
+ struct syncobj_wait_entry wait;
+ u64 timeout = nsecs_to_jiffies64(DRM_SYNCOBJ_WAIT_FOR_SUBMIT_TIMEOUT);
+ int ret;
if (!syncobj)
return -ENOENT;
*fence = drm_syncobj_fence_get(syncobj);
- if (!*fence) {
+ drm_syncobj_put(syncobj);
+
+ if (*fence) {
+ ret = dma_fence_chain_find_seqno(fence, point);
+ if (!ret)
+ return 0;
+ dma_fence_put(*fence);
+ } else {
ret = -EINVAL;
}
- drm_syncobj_put(syncobj);
+
+ if (!(flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT))
+ return ret;
+
+ memset(&wait, 0, sizeof(wait));
+ wait.task = current;
+ wait.point = point;
+ drm_syncobj_fence_add_wait(syncobj, &wait);
+
+ do {
+ set_current_state(TASK_INTERRUPTIBLE);
+ if (wait.fence) {
+ ret = 0;
+ break;
+ }
+ if (timeout == 0) {
+ ret = -ETIME;
+ break;
+ }
+
+ if (signal_pending(current)) {
+ ret = -ERESTARTSYS;
+ break;
+ }
+
+ timeout = schedule_timeout(timeout);
+ } while (1);
+
+ __set_current_state(TASK_RUNNING);
+ *fence = wait.fence;
+
+ if (wait.node.next)
+ drm_syncobj_remove_wait(syncobj, &wait);
+
return ret;
}
EXPORT_SYMBOL(drm_syncobj_find_fence);
&args->handle);
}
+static int drm_syncobj_transfer_to_timeline(struct drm_file *file_private,
+ struct drm_syncobj_transfer *args)
+{
+ struct drm_syncobj *timeline_syncobj = NULL;
+ struct dma_fence *fence;
+ struct dma_fence_chain *chain;
+ int ret;
+
+ timeline_syncobj = drm_syncobj_find(file_private, args->dst_handle);
+ if (!timeline_syncobj) {
+ return -ENOENT;
+ }
+ ret = drm_syncobj_find_fence(file_private, args->src_handle,
+ args->src_point, args->flags,
+ &fence);
+ if (ret)
+ goto err;
+ chain = kzalloc(sizeof(struct dma_fence_chain), GFP_KERNEL);
+ if (!chain) {
+ ret = -ENOMEM;
+ goto err1;
+ }
+ drm_syncobj_add_point(timeline_syncobj, chain, fence, args->dst_point);
+err1:
+ dma_fence_put(fence);
+err:
+ drm_syncobj_put(timeline_syncobj);
+
+ return ret;
+}
+
+static int
+drm_syncobj_transfer_to_binary(struct drm_file *file_private,
+ struct drm_syncobj_transfer *args)
+{
+ struct drm_syncobj *binary_syncobj = NULL;
+ struct dma_fence *fence;
+ int ret;
+
+ binary_syncobj = drm_syncobj_find(file_private, args->dst_handle);
+ if (!binary_syncobj)
+ return -ENOENT;
+ ret = drm_syncobj_find_fence(file_private, args->src_handle,
+ args->src_point, args->flags, &fence);
+ if (ret)
+ goto err;
+ drm_syncobj_replace_fence(binary_syncobj, fence);
+ dma_fence_put(fence);
+err:
+ drm_syncobj_put(binary_syncobj);
+
+ return ret;
+}
+int
+drm_syncobj_transfer_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private)
+{
+ struct drm_syncobj_transfer *args = data;
+ int ret;
+
+ if (!drm_core_check_feature(dev, DRIVER_SYNCOBJ_TIMELINE))
+ return -EOPNOTSUPP;
+
+ if (args->pad)
+ return -EINVAL;
+
+ if (args->dst_point)
+ ret = drm_syncobj_transfer_to_timeline(file_private, args);
+ else
+ ret = drm_syncobj_transfer_to_binary(file_private, args);
+
+ return ret;
+}
+
static void syncobj_wait_fence_func(struct dma_fence *fence,
struct dma_fence_cb *cb)
{
static void syncobj_wait_syncobj_func(struct drm_syncobj *syncobj,
struct syncobj_wait_entry *wait)
{
+ struct dma_fence *fence;
+
/* This happens inside the syncobj lock */
- wait->fence = dma_fence_get(rcu_dereference_protected(syncobj->fence,
- lockdep_is_held(&syncobj->lock)));
+ fence = rcu_dereference_protected(syncobj->fence,
+ lockdep_is_held(&syncobj->lock));
+ dma_fence_get(fence);
+ if (!fence || dma_fence_chain_find_seqno(&fence, wait->point)) {
+ dma_fence_put(fence);
+ return;
+ } else if (!fence) {
+ wait->fence = dma_fence_get_stub();
+ } else {
+ wait->fence = fence;
+ }
+
wake_up_process(wait->task);
+ list_del_init(&wait->node);
}
static signed long drm_syncobj_array_wait_timeout(struct drm_syncobj **syncobjs,
+ void __user *user_points,
uint32_t count,
uint32_t flags,
signed long timeout,
{
struct syncobj_wait_entry *entries;
struct dma_fence *fence;
+ uint64_t *points;
uint32_t signaled_count, i;
- entries = kcalloc(count, sizeof(*entries), GFP_KERNEL);
- if (!entries)
+ points = kmalloc_array(count, sizeof(*points), GFP_KERNEL);
+ if (points == NULL)
return -ENOMEM;
+ if (!user_points) {
+ memset(points, 0, count * sizeof(uint64_t));
+
+ } else if (copy_from_user(points, user_points,
+ sizeof(uint64_t) * count)) {
+ timeout = -EFAULT;
+ goto err_free_points;
+ }
+
+ entries = kcalloc(count, sizeof(*entries), GFP_KERNEL);
+ if (!entries) {
+ timeout = -ENOMEM;
+ goto err_free_points;
+ }
/* Walk the list of sync objects and initialize entries. We do
* this up-front so that we can properly return -EINVAL if there is
* a syncobj with a missing fence and then never have the chance of
*/
signaled_count = 0;
for (i = 0; i < count; ++i) {
+ struct dma_fence *fence;
+
entries[i].task = current;
- entries[i].fence = drm_syncobj_fence_get(syncobjs[i]);
- if (!entries[i].fence) {
+ entries[i].point = points[i];
+ fence = drm_syncobj_fence_get(syncobjs[i]);
+ if (!fence || dma_fence_chain_find_seqno(&fence, points[i])) {
+ dma_fence_put(fence);
if (flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT) {
continue;
} else {
}
}
- if (dma_fence_is_signaled(entries[i].fence)) {
+ if (fence)
+ entries[i].fence = fence;
+ else
+ entries[i].fence = dma_fence_get_stub();
+
+ if ((flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_AVAILABLE) ||
+ dma_fence_is_signaled(entries[i].fence)) {
if (signaled_count == 0 && idx)
*idx = i;
signaled_count++;
if (!fence)
continue;
- if (dma_fence_is_signaled(fence) ||
+ if ((flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_AVAILABLE) ||
+ dma_fence_is_signaled(fence) ||
(!entries[i].fence_cb.func &&
dma_fence_add_callback(fence,
&entries[i].fence_cb,
}
kfree(entries);
+err_free_points:
+ kfree(points);
+
return timeout;
}
*
* Calculate the timeout in jiffies from an absolute time in sec/nsec.
*/
-static signed long drm_timeout_abs_to_jiffies(int64_t timeout_nsec)
+signed long drm_timeout_abs_to_jiffies(int64_t timeout_nsec)
{
ktime_t abs_timeout, now;
u64 timeout_ns, timeout_jiffies64;
return timeout_jiffies64 + 1;
}
+EXPORT_SYMBOL(drm_timeout_abs_to_jiffies);
static int drm_syncobj_array_wait(struct drm_device *dev,
struct drm_file *file_private,
struct drm_syncobj_wait *wait,
- struct drm_syncobj **syncobjs)
+ struct drm_syncobj_timeline_wait *timeline_wait,
+ struct drm_syncobj **syncobjs, bool timeline)
{
- signed long timeout = drm_timeout_abs_to_jiffies(wait->timeout_nsec);
+ signed long timeout = 0;
uint32_t first = ~0;
- timeout = drm_syncobj_array_wait_timeout(syncobjs,
- wait->count_handles,
- wait->flags,
- timeout, &first);
- if (timeout < 0)
- return timeout;
-
- wait->first_signaled = first;
+ if (!timeline) {
+ timeout = drm_timeout_abs_to_jiffies(wait->timeout_nsec);
+ timeout = drm_syncobj_array_wait_timeout(syncobjs,
+ NULL,
+ wait->count_handles,
+ wait->flags,
+ timeout, &first);
+ if (timeout < 0)
+ return timeout;
+ wait->first_signaled = first;
+ } else {
+ timeout = drm_timeout_abs_to_jiffies(timeline_wait->timeout_nsec);
+ timeout = drm_syncobj_array_wait_timeout(syncobjs,
+ u64_to_user_ptr(timeline_wait->points),
+ timeline_wait->count_handles,
+ timeline_wait->flags,
+ timeout, &first);
+ if (timeout < 0)
+ return timeout;
+ timeline_wait->first_signaled = first;
+ }
return 0;
}
return ret;
ret = drm_syncobj_array_wait(dev, file_private,
- args, syncobjs);
+ args, NULL, syncobjs, false);
+
+ drm_syncobj_array_free(syncobjs, args->count_handles);
+
+ return ret;
+}
+
+int
+drm_syncobj_timeline_wait_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private)
+{
+ struct drm_syncobj_timeline_wait *args = data;
+ struct drm_syncobj **syncobjs;
+ int ret = 0;
+
+ if (!drm_core_check_feature(dev, DRIVER_SYNCOBJ_TIMELINE))
+ return -EOPNOTSUPP;
+
+ if (args->flags & ~(DRM_SYNCOBJ_WAIT_FLAGS_WAIT_ALL |
+ DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT |
+ DRM_SYNCOBJ_WAIT_FLAGS_WAIT_AVAILABLE))
+ return -EINVAL;
+
+ if (args->count_handles == 0)
+ return -EINVAL;
+
+ ret = drm_syncobj_array_find(file_private,
+ u64_to_user_ptr(args->handles),
+ args->count_handles,
+ &syncobjs);
+ if (ret < 0)
+ return ret;
+
+ ret = drm_syncobj_array_wait(dev, file_private,
+ NULL, args, syncobjs, true);
drm_syncobj_array_free(syncobjs, args->count_handles);
return ret;
}
+
int
drm_syncobj_reset_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_private)
return ret;
}
+
+int
+drm_syncobj_timeline_signal_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private)
+{
+ struct drm_syncobj_timeline_array *args = data;
+ struct drm_syncobj **syncobjs;
+ struct dma_fence_chain **chains;
+ uint64_t *points;
+ uint32_t i, j;
+ int ret;
+
+ if (!drm_core_check_feature(dev, DRIVER_SYNCOBJ_TIMELINE))
+ return -EOPNOTSUPP;
+
+ if (args->pad != 0)
+ return -EINVAL;
+
+ if (args->count_handles == 0)
+ return -EINVAL;
+
+ ret = drm_syncobj_array_find(file_private,
+ u64_to_user_ptr(args->handles),
+ args->count_handles,
+ &syncobjs);
+ if (ret < 0)
+ return ret;
+
+ points = kmalloc_array(args->count_handles, sizeof(*points),
+ GFP_KERNEL);
+ if (!points) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ if (!u64_to_user_ptr(args->points)) {
+ memset(points, 0, args->count_handles * sizeof(uint64_t));
+ } else if (copy_from_user(points, u64_to_user_ptr(args->points),
+ sizeof(uint64_t) * args->count_handles)) {
+ ret = -EFAULT;
+ goto err_points;
+ }
+
+ chains = kmalloc_array(args->count_handles, sizeof(void *), GFP_KERNEL);
+ if (!chains) {
+ ret = -ENOMEM;
+ goto err_points;
+ }
+ for (i = 0; i < args->count_handles; i++) {
+ chains[i] = kzalloc(sizeof(struct dma_fence_chain), GFP_KERNEL);
+ if (!chains[i]) {
+ for (j = 0; j < i; j++)
+ kfree(chains[j]);
+ ret = -ENOMEM;
+ goto err_chains;
+ }
+ }
+
+ for (i = 0; i < args->count_handles; i++) {
+ struct dma_fence *fence = dma_fence_get_stub();
+
+ drm_syncobj_add_point(syncobjs[i], chains[i],
+ fence, points[i]);
+ dma_fence_put(fence);
+ }
+err_chains:
+ kfree(chains);
+err_points:
+ kfree(points);
+out:
+ drm_syncobj_array_free(syncobjs, args->count_handles);
+
+ return ret;
+}
+
+int drm_syncobj_query_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private)
+{
+ struct drm_syncobj_timeline_array *args = data;
+ struct drm_syncobj **syncobjs;
+ uint64_t __user *points = u64_to_user_ptr(args->points);
+ uint32_t i;
+ int ret;
+
+ if (!drm_core_check_feature(dev, DRIVER_SYNCOBJ_TIMELINE))
+ return -EOPNOTSUPP;
+
+ if (args->pad != 0)
+ return -EINVAL;
+
+ if (args->count_handles == 0)
+ return -EINVAL;
+
+ ret = drm_syncobj_array_find(file_private,
+ u64_to_user_ptr(args->handles),
+ args->count_handles,
+ &syncobjs);
+ if (ret < 0)
+ return ret;
+
+ for (i = 0; i < args->count_handles; i++) {
+ struct dma_fence_chain *chain;
+ struct dma_fence *fence;
+ uint64_t point;
+
+ fence = drm_syncobj_fence_get(syncobjs[i]);
+ chain = to_dma_fence_chain(fence);
+ if (chain) {
+ struct dma_fence *iter, *last_signaled = NULL;
+
+ dma_fence_chain_for_each(iter, fence) {
+ if (!iter)
+ break;
+ dma_fence_put(last_signaled);
+ last_signaled = dma_fence_get(iter);
+ if (!to_dma_fence_chain(last_signaled)->prev_seqno)
+ /* It is most likely that timeline has
+ * unorder points. */
+ break;
+ }
+ point = dma_fence_is_signaled(last_signaled) ?
+ last_signaled->seqno :
+ to_dma_fence_chain(last_signaled)->prev_seqno;
+ dma_fence_put(last_signaled);
+ } else {
+ point = 0;
+ }
+ ret = copy_to_user(&points[i], &point, sizeof(uint64_t));
+ ret = ret ? -EFAULT : 0;
+ if (ret)
+ break;
+ }
+ drm_syncobj_array_free(syncobjs, args->count_handles);
+
+ return ret;
+}
vma->vm_ops = &drm_vm_ops;
break;
}
- /* fall through to _DRM_FRAME_BUFFER... */
#endif
+ /* fall through - to _DRM_FRAME_BUFFER... */
case _DRM_FRAME_BUFFER:
case _DRM_REGISTERS:
offset = drm_core_get_reg_ofs(dev);
vma->vm_end - vma->vm_start, vma->vm_page_prot))
return -EAGAIN;
vma->vm_page_prot = drm_dma_prot(map->type, vma);
- /* fall through to _DRM_SHM */
+ /* fall through - to _DRM_SHM */
case _DRM_SHM:
vma->vm_ops = &drm_vm_shm_ops;
vma->vm_private_data = (void *)map;
}
EXPORT_SYMBOL(drm_legacy_mmap);
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
void drm_legacy_vma_flush(struct drm_device *dev)
{
struct drm_vma_entry *vma, *vma_temp;
kfree(vma);
}
}
+#endif
}
EXPORT_SYMBOL(drm_writeback_connector_init);
+int drm_writeback_set_fb(struct drm_connector_state *conn_state,
+ struct drm_framebuffer *fb)
+{
+ WARN_ON(conn_state->connector->connector_type != DRM_MODE_CONNECTOR_WRITEBACK);
+
+ if (!conn_state->writeback_job) {
+ conn_state->writeback_job =
+ kzalloc(sizeof(*conn_state->writeback_job), GFP_KERNEL);
+ if (!conn_state->writeback_job)
+ return -ENOMEM;
+
+ conn_state->writeback_job->connector =
+ drm_connector_to_writeback(conn_state->connector);
+ }
+
+ drm_framebuffer_assign(&conn_state->writeback_job->fb, fb);
+ return 0;
+}
+
+int drm_writeback_prepare_job(struct drm_writeback_job *job)
+{
+ struct drm_writeback_connector *connector = job->connector;
+ const struct drm_connector_helper_funcs *funcs =
+ connector->base.helper_private;
+ int ret;
+
+ if (funcs->prepare_writeback_job) {
+ ret = funcs->prepare_writeback_job(connector, job);
+ if (ret < 0)
+ return ret;
+ }
+
+ job->prepared = true;
+ return 0;
+}
+EXPORT_SYMBOL(drm_writeback_prepare_job);
+
/**
* drm_writeback_queue_job - Queue a writeback job for later signalling
* @wb_connector: The writeback connector to queue a job on
- * @job: The job to queue
+ * @conn_state: The connector state containing the job to queue
*
- * This function adds a job to the job_queue for a writeback connector. It
- * should be considered to take ownership of the writeback job, and so any other
- * references to the job must be cleared after calling this function.
+ * This function adds the job contained in @conn_state to the job_queue for a
+ * writeback connector. It takes ownership of the writeback job and sets the
+ * @conn_state->writeback_job to NULL, and so no access to the job may be
+ * performed by the caller after this function returns.
*
* Drivers must ensure that for a given writeback connector, jobs are queued in
* exactly the same order as they will be completed by the hardware (and
* See also: drm_writeback_signal_completion()
*/
void drm_writeback_queue_job(struct drm_writeback_connector *wb_connector,
- struct drm_writeback_job *job)
+ struct drm_connector_state *conn_state)
{
+ struct drm_writeback_job *job;
unsigned long flags;
+ job = conn_state->writeback_job;
+ conn_state->writeback_job = NULL;
+
spin_lock_irqsave(&wb_connector->job_lock, flags);
list_add_tail(&job->list_entry, &wb_connector->job_queue);
spin_unlock_irqrestore(&wb_connector->job_lock, flags);
}
EXPORT_SYMBOL(drm_writeback_queue_job);
+void drm_writeback_cleanup_job(struct drm_writeback_job *job)
+{
+ struct drm_writeback_connector *connector = job->connector;
+ const struct drm_connector_helper_funcs *funcs =
+ connector->base.helper_private;
+
+ if (job->prepared && funcs->cleanup_writeback_job)
+ funcs->cleanup_writeback_job(connector, job);
+
+ if (job->fb)
+ drm_framebuffer_put(job->fb);
+
+ kfree(job);
+}
+EXPORT_SYMBOL(drm_writeback_cleanup_job);
+
/*
* @cleanup_work: deferred cleanup of a writeback job
*
struct drm_writeback_job *job = container_of(work,
struct drm_writeback_job,
cleanup_work);
- drm_framebuffer_put(job->fb);
- kfree(job);
-}
+ drm_writeback_cleanup_job(job);
+}
/**
* drm_writeback_signal_completion - Signal the completion of a writeback job
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
.gem_prime_export = drm_gem_prime_export,
.gem_prime_import = drm_gem_prime_import,
- .gem_prime_res_obj = etnaviv_gem_prime_res_obj,
.gem_prime_pin = etnaviv_gem_prime_pin,
.gem_prime_unpin = etnaviv_gem_prime_unpin,
.gem_prime_get_sg_table = etnaviv_gem_prime_get_sg_table,
}
drm->dev_private = priv;
+ dev->dma_parms = &priv->dma_parms;
+ dma_set_max_seg_size(dev, SZ_2G);
+
mutex_init(&priv->gem_lock);
INIT_LIST_HEAD(&priv->gem_list);
priv->num_gpus = 0;
component_unbind_all(dev, drm);
+ dev->dma_parms = NULL;
+
drm->dev_private = NULL;
kfree(priv);
struct etnaviv_drm_private {
int num_gpus;
+ struct device_dma_parameters dma_parms;
struct etnaviv_gpu *gpu[ETNA_MAX_PIPES];
/* list of GEM objects: */
void etnaviv_gem_prime_vunmap(struct drm_gem_object *obj, void *vaddr);
int etnaviv_gem_prime_mmap(struct drm_gem_object *obj,
struct vm_area_struct *vma);
-struct reservation_object *etnaviv_gem_prime_res_obj(struct drm_gem_object *obj);
struct drm_gem_object *etnaviv_gem_prime_import_sg_table(struct drm_device *dev,
struct dma_buf_attachment *attach, struct sg_table *sg);
int etnaviv_gem_prime_pin(struct drm_gem_object *obj);
}
if (op & ETNA_PREP_NOSYNC) {
- if (!reservation_object_test_signaled_rcu(etnaviv_obj->resv,
+ if (!reservation_object_test_signaled_rcu(obj->resv,
write))
return -EBUSY;
} else {
unsigned long remain = etnaviv_timeout_to_jiffies(timeout);
- ret = reservation_object_wait_timeout_rcu(etnaviv_obj->resv,
+ ret = reservation_object_wait_timeout_rcu(obj->resv,
write, true, remain);
if (ret <= 0)
return ret == 0 ? -ETIMEDOUT : ret;
static void etnaviv_gem_describe(struct drm_gem_object *obj, struct seq_file *m)
{
struct etnaviv_gem_object *etnaviv_obj = to_etnaviv_bo(obj);
- struct reservation_object *robj = etnaviv_obj->resv;
+ struct reservation_object *robj = obj->resv;
struct reservation_object_list *fobj;
struct dma_fence *fence;
unsigned long off = drm_vma_node_start(&obj->vma_node);
drm_gem_free_mmap_offset(obj);
etnaviv_obj->ops->release(etnaviv_obj);
- if (etnaviv_obj->resv == &etnaviv_obj->_resv)
- reservation_object_fini(&etnaviv_obj->_resv);
drm_gem_object_release(obj);
kfree(etnaviv_obj);
etnaviv_obj->flags = flags;
etnaviv_obj->ops = ops;
- if (robj) {
- etnaviv_obj->resv = robj;
- } else {
- etnaviv_obj->resv = &etnaviv_obj->_resv;
- reservation_object_init(&etnaviv_obj->_resv);
- }
+ if (robj)
+ etnaviv_obj->base.resv = robj;
mutex_init(&etnaviv_obj->lock);
INIT_LIST_HEAD(&etnaviv_obj->vram_list);
lockdep_set_class(&to_etnaviv_bo(obj)->lock, &etnaviv_shm_lock_class);
ret = drm_gem_object_init(dev, obj, size);
- if (ret == 0) {
- struct address_space *mapping;
-
- /*
- * Our buffers are kept pinned, so allocating them
- * from the MOVABLE zone is a really bad idea, and
- * conflicts with CMA. See comments above new_inode()
- * why this is required _and_ expected if you're
- * going to pin these pages.
- */
- mapping = obj->filp->f_mapping;
- mapping_set_gfp_mask(mapping, GFP_HIGHUSER |
- __GFP_RETRY_MAYFAIL | __GFP_NOWARN);
- }
-
if (ret)
goto fail;
+ /*
+ * Our buffers are kept pinned, so allocating them from the MOVABLE
+ * zone is a really bad idea, and conflicts with CMA. See comments
+ * above new_inode() why this is required _and_ expected if you're
+ * going to pin these pages.
+ */
+ mapping_set_gfp_mask(obj->filp->f_mapping, GFP_HIGHUSER |
+ __GFP_RETRY_MAYFAIL | __GFP_NOWARN);
+
etnaviv_gem_obj_add(dev, obj);
ret = drm_gem_handle_create(file, obj, handle);
struct sg_table *sgt;
void *vaddr;
- /* normally (resv == &_resv) except for imported bo's */
- struct reservation_object *resv;
- struct reservation_object _resv;
-
struct list_head vram_list;
/* cache maintenance */
return ERR_PTR(ret);
}
-
-struct reservation_object *etnaviv_gem_prime_res_obj(struct drm_gem_object *obj)
-{
- struct etnaviv_gem_object *etnaviv_obj = to_etnaviv_bo(obj);
-
- return etnaviv_obj->resv;
-}
static void submit_unlock_object(struct etnaviv_gem_submit *submit, int i)
{
if (submit->bos[i].flags & BO_LOCKED) {
- struct etnaviv_gem_object *etnaviv_obj = submit->bos[i].obj;
+ struct drm_gem_object *obj = &submit->bos[i].obj->base;
- ww_mutex_unlock(&etnaviv_obj->resv->lock);
+ ww_mutex_unlock(&obj->resv->lock);
submit->bos[i].flags &= ~BO_LOCKED;
}
}
retry:
for (i = 0; i < submit->nr_bos; i++) {
- struct etnaviv_gem_object *etnaviv_obj = submit->bos[i].obj;
+ struct drm_gem_object *obj = &submit->bos[i].obj->base;
if (slow_locked == i)
slow_locked = -1;
contended = i;
if (!(submit->bos[i].flags & BO_LOCKED)) {
- ret = ww_mutex_lock_interruptible(&etnaviv_obj->resv->lock,
+ ret = ww_mutex_lock_interruptible(&obj->resv->lock,
ticket);
if (ret == -EALREADY)
DRM_ERROR("BO at index %u already on submit list\n",
submit_unlock_object(submit, slow_locked);
if (ret == -EDEADLK) {
- struct etnaviv_gem_object *etnaviv_obj;
+ struct drm_gem_object *obj;
- etnaviv_obj = submit->bos[contended].obj;
+ obj = &submit->bos[contended].obj->base;
/* we lost out in a seqno race, lock and retry.. */
- ret = ww_mutex_lock_slow_interruptible(&etnaviv_obj->resv->lock,
+ ret = ww_mutex_lock_slow_interruptible(&obj->resv->lock,
ticket);
if (!ret) {
submit->bos[contended].flags |= BO_LOCKED;
for (i = 0; i < submit->nr_bos; i++) {
struct etnaviv_gem_submit_bo *bo = &submit->bos[i];
- struct reservation_object *robj = bo->obj->resv;
+ struct reservation_object *robj = bo->obj->base.resv;
if (!(bo->flags & ETNA_SUBMIT_BO_WRITE)) {
ret = reservation_object_reserve_shared(robj, 1);
int i;
for (i = 0; i < submit->nr_bos; i++) {
- struct etnaviv_gem_object *etnaviv_obj = submit->bos[i].obj;
+ struct drm_gem_object *obj = &submit->bos[i].obj->base;
if (submit->bos[i].flags & ETNA_SUBMIT_BO_WRITE)
- reservation_object_add_excl_fence(etnaviv_obj->resv,
+ reservation_object_add_excl_fence(obj->resv,
submit->out_fence);
else
- reservation_object_add_shared_fence(etnaviv_obj->resv,
+ reservation_object_add_shared_fence(obj->resv,
submit->out_fence);
submit_unlock_object(submit, i);
dev_info(gpu->dev, "model: GC%x, revision: %x\n",
gpu->identity.model, gpu->identity.revision);
+ gpu->idle_mask = ~VIVS_HI_IDLE_STATE_AXI_LP;
/*
* If there is a match in the HWDB, we aren't interested in the
* remaining register values, as they might be wrong.
}
/* GC600 idle register reports zero bits where modules aren't present */
- if (gpu->identity.model == chipModel_GC600) {
+ if (gpu->identity.model == chipModel_GC600)
gpu->idle_mask = VIVS_HI_IDLE_STATE_TX |
VIVS_HI_IDLE_STATE_RA |
VIVS_HI_IDLE_STATE_SE |
VIVS_HI_IDLE_STATE_PE |
VIVS_HI_IDLE_STATE_DE |
VIVS_HI_IDLE_STATE_FE;
- } else {
- gpu->idle_mask = ~VIVS_HI_IDLE_STATE_AXI_LP;
- }
etnaviv_hw_specs(gpu);
}
if (regmap_update_bits(ctx->sysreg, DSD_CFG_MUX,
DSD_CFG_MUX_TE_UNMASK_GLOBAL, ~0))
- DRM_ERROR("Cannot update sysreg.\n");
+ DRM_DEV_ERROR(ctx->dev, "Cannot update sysreg.\n");
}
static void decon_commit(struct exynos_drm_crtc *crtc)
break;
}
- DRM_DEBUG_KMS("cpp = %u\n", fb->format->cpp[0]);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "cpp = %u\n", fb->format->cpp[0]);
/*
* In case of exynos, setting dma-burst to 16Word causes permanent
struct decon_context *ctx = crtc->ctx;
int win, i, ret;
- DRM_DEBUG_KMS("%s\n", __FILE__);
-
for (i = 0; i < ARRAY_SIZE(decon_clks_name); i++) {
ret = clk_prepare_enable(ctx->clks[i]);
if (ret < 0)
if (!wait_event_timeout(ctx->wait_vsync_queue,
!atomic_read(&ctx->wait_vsync_event),
HZ/20))
- DRM_DEBUG_KMS("vblank wait timed out.\n");
+ DRM_DEV_DEBUG_KMS(ctx->dev, "vblank wait timed out.\n");
}
static void decon_clear_channels(struct exynos_drm_crtc *crtc)
struct decon_context *ctx = crtc->ctx;
unsigned int win, ch_enabled = 0;
- DRM_DEBUG_KMS("%s\n", __FILE__);
-
/* Check if any channel is enabled. */
for (win = 0; win < WINDOWS_NR; win++) {
u32 val = readl(ctx->regs + WINCON(win));
break;
}
- DRM_DEBUG_KMS("cpp = %d\n", fb->format->cpp[0]);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "cpp = %d\n", fb->format->cpp[0]);
/*
* In case of exynos, setting dma-burst to 16Word causes permanent
writel(state->src.x, ctx->regs + VIDW_OFFSET_X(win));
writel(state->src.y, ctx->regs + VIDW_OFFSET_Y(win));
- DRM_DEBUG_KMS("start addr = 0x%lx\n",
+ DRM_DEV_DEBUG_KMS(ctx->dev, "start addr = 0x%lx\n",
(unsigned long)val);
- DRM_DEBUG_KMS("ovl_width = %d, ovl_height = %d\n",
+ DRM_DEV_DEBUG_KMS(ctx->dev, "ovl_width = %d, ovl_height = %d\n",
state->crtc.w, state->crtc.h);
val = VIDOSDxA_TOPLEFT_X(state->crtc.x) |
writel(val, ctx->regs + VIDOSD_B(win));
- DRM_DEBUG_KMS("osd pos: tx = %d, ty = %d, bx = %d, by = %d\n",
+ DRM_DEV_DEBUG_KMS(ctx->dev, "osd pos: tx = %d, ty = %d, bx = %d, by = %d\n",
state->crtc.x, state->crtc.y, last_x, last_y);
/* OSD alpha */
ret = decon_ctx_initialize(ctx, drm_dev);
if (ret) {
- DRM_ERROR("decon_ctx_initialize failed.\n");
+ DRM_DEV_ERROR(dev, "decon_ctx_initialize failed.\n");
return ret;
}
ret = clk_prepare_enable(ctx->pclk);
if (ret < 0) {
- DRM_ERROR("Failed to prepare_enable the pclk [%d]\n", ret);
+ DRM_DEV_ERROR(dev, "Failed to prepare_enable the pclk [%d]\n",
+ ret);
return ret;
}
ret = clk_prepare_enable(ctx->aclk);
if (ret < 0) {
- DRM_ERROR("Failed to prepare_enable the aclk [%d]\n", ret);
+ DRM_DEV_ERROR(dev, "Failed to prepare_enable the aclk [%d]\n",
+ ret);
return ret;
}
ret = clk_prepare_enable(ctx->eclk);
if (ret < 0) {
- DRM_ERROR("Failed to prepare_enable the eclk [%d]\n", ret);
+ DRM_DEV_ERROR(dev, "Failed to prepare_enable the eclk [%d]\n",
+ ret);
return ret;
}
ret = clk_prepare_enable(ctx->vclk);
if (ret < 0) {
- DRM_ERROR("Failed to prepare_enable the vclk [%d]\n", ret);
+ DRM_DEV_ERROR(dev, "Failed to prepare_enable the vclk [%d]\n",
+ ret);
return ret;
}
mode = drm_mode_create(connector->dev);
if (!mode) {
- DRM_ERROR("failed to create a new display mode.\n");
+ DRM_DEV_ERROR(dp->dev,
+ "failed to create a new display mode.\n");
return num_modes;
}
if (dp->ptn_bridge) {
ret = drm_bridge_attach(&dp->encoder, dp->ptn_bridge, bridge);
if (ret) {
- DRM_ERROR("Failed to attach bridge to drm\n");
+ DRM_DEV_ERROR(dp->dev,
+ "Failed to attach bridge to drm\n");
bridge->next = NULL;
return ret;
}
ret = of_get_videomode(dp->dev->of_node, &dp->vm, OF_USE_NATIVE_MODE);
if (ret) {
- DRM_ERROR("failed: of_get_videomode() : %d\n", ret);
+ DRM_DEV_ERROR(dp->dev,
+ "failed: of_get_videomode() : %d\n", ret);
return ret;
}
return 0;
int ret;
if (get_dma_ops(priv->dma_dev) != get_dma_ops(subdrv_dev)) {
- DRM_ERROR("Device %s lacks support for IOMMU\n",
+ DRM_DEV_ERROR(subdrv_dev, "Device %s lacks support for IOMMU\n",
dev_name(subdrv_dev));
return -EINVAL;
}
mode = drm_mode_create(connector->dev);
if (!mode) {
- DRM_ERROR("failed to create a new display mode\n");
+ DRM_DEV_ERROR(ctx->dev,
+ "failed to create a new display mode\n");
return 0;
}
drm_display_mode_from_videomode(ctx->vm, mode);
&exynos_dpi_connector_funcs,
DRM_MODE_CONNECTOR_VGA);
if (ret) {
- DRM_ERROR("failed to initialize connector with drm\n");
+ DRM_DEV_ERROR(ctx->dev,
+ "failed to initialize connector with drm\n");
return ret;
}
ret = exynos_dpi_create_connector(encoder);
if (ret) {
- DRM_ERROR("failed to create connector ret = %d\n", ret);
+ DRM_DEV_ERROR(encoder_to_dpi(encoder)->dev,
+ "failed to create connector ret = %d\n", ret);
drm_encoder_cleanup(encoder);
return ret;
}
ret = drm_connector_init(drm, connector, &exynos_dsi_connector_funcs,
DRM_MODE_CONNECTOR_DSI);
if (ret) {
- DRM_ERROR("Failed to initialize connector with drm\n");
+ DRM_DEV_ERROR(dsi->dev,
+ "Failed to initialize connector with drm\n");
return ret;
}
int ret = exynos_dsi_create_connector(encoder);
if (ret) {
- DRM_ERROR("failed to create connector ret = %d\n", ret);
+ DRM_DEV_ERROR(dsi->dev,
+ "failed to create connector ret = %d\n",
+ ret);
drm_encoder_cleanup(encoder);
return ret;
}
* supported without IOMMU.
*/
if (IS_NONCONTIG_BUFFER(flags)) {
- DRM_ERROR("Non-contiguous GEM memory is not supported.\n");
+ DRM_DEV_ERROR(drm_dev->dev,
+ "Non-contiguous GEM memory is not supported.\n");
return -EINVAL;
}
ret = drm_framebuffer_init(dev, fb, &exynos_drm_fb_funcs);
if (ret < 0) {
- DRM_ERROR("failed to initialize framebuffer\n");
+ DRM_DEV_ERROR(dev->dev,
+ "failed to initialize framebuffer\n");
goto err;
}
exynos_gem[i] = exynos_drm_gem_get(file_priv,
mode_cmd->handles[i]);
if (!exynos_gem[i]) {
- DRM_ERROR("failed to lookup gem object\n");
+ DRM_DEV_ERROR(dev->dev,
+ "failed to lookup gem object\n");
ret = -ENOENT;
goto err;
}
exynos_gem->dma_addr, exynos_gem->size,
exynos_gem->dma_attrs);
if (ret < 0) {
- DRM_ERROR("failed to mmap.\n");
+ DRM_DEV_ERROR(to_dma_dev(helper->dev), "failed to mmap.\n");
return ret;
}
fbi = drm_fb_helper_alloc_fbi(helper);
if (IS_ERR(fbi)) {
- DRM_ERROR("failed to allocate fb info.\n");
+ DRM_DEV_ERROR(to_dma_dev(helper->dev),
+ "failed to allocate fb info.\n");
return PTR_ERR(fbi);
}
- fbi->par = helper;
fbi->fbops = &exynos_drm_fb_ops;
- drm_fb_helper_fill_fix(fbi, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(fbi, helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(fbi, helper, sizes);
nr_pages = exynos_gem->size >> PAGE_SHIFT;
exynos_gem->kvaddr = (void __iomem *) vmap(exynos_gem->pages, nr_pages,
VM_MAP, pgprot_writecombine(PAGE_KERNEL));
if (!exynos_gem->kvaddr) {
- DRM_ERROR("failed to map pages to kernel space.\n");
+ DRM_DEV_ERROR(to_dma_dev(helper->dev),
+ "failed to map pages to kernel space.\n");
return -EIO;
}
unsigned long size;
int ret;
- DRM_DEBUG_KMS("surface width(%d), height(%d) and bpp(%d\n",
- sizes->surface_width, sizes->surface_height,
- sizes->surface_bpp);
+ DRM_DEV_DEBUG_KMS(dev->dev,
+ "surface width(%d), height(%d) and bpp(%d\n",
+ sizes->surface_width, sizes->surface_height,
+ sizes->surface_bpp);
mode_cmd.width = sizes->surface_width;
mode_cmd.height = sizes->surface_height;
helper->fb =
exynos_drm_framebuffer_init(dev, &mode_cmd, &exynos_gem, 1);
if (IS_ERR(helper->fb)) {
- DRM_ERROR("failed to create drm framebuffer.\n");
+ DRM_DEV_ERROR(dev->dev, "failed to create drm framebuffer.\n");
ret = PTR_ERR(helper->fb);
goto err_destroy_gem;
}
ret = drm_fb_helper_init(dev, helper, MAX_CONNECTOR);
if (ret < 0) {
- DRM_ERROR("failed to initialize drm fb helper.\n");
+ DRM_DEV_ERROR(dev->dev,
+ "failed to initialize drm fb helper.\n");
goto err_init;
}
ret = drm_fb_helper_single_add_all_connectors(helper);
if (ret < 0) {
- DRM_ERROR("failed to register drm_fb_helper_connector.\n");
+ DRM_DEV_ERROR(dev->dev,
+ "failed to register drm_fb_helper_connector.\n");
goto err_setup;
}
ret = drm_fb_helper_initial_config(helper, PREFERRED_BPP);
if (ret < 0) {
- DRM_ERROR("failed to set up hw configuration.\n");
+ DRM_DEV_ERROR(dev->dev,
+ "failed to set up hw configuration.\n");
goto err_setup;
}
{
u32 cfg;
- DRM_DEBUG_KMS("enable[%d]\n", enable);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "enable[%d]\n", enable);
cfg = fimc_read(ctx, EXYNOS_CIGCTRL);
if (enable)
{
u32 cfg;
- DRM_DEBUG_KMS("enable[%d]\n", enable);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "enable[%d]\n", enable);
cfg = fimc_read(ctx, EXYNOS_CIGCTRL);
if (enable) {
flag = EXYNOS_CISTATUS_OVFIY | EXYNOS_CISTATUS_OVFICB |
EXYNOS_CISTATUS_OVFICR;
- DRM_DEBUG_KMS("flag[0x%x]\n", flag);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "flag[0x%x]\n", flag);
if (status & flag) {
fimc_set_bits(ctx, EXYNOS_CIWDOFST,
EXYNOS_CIWDOFST_CLROVFIY | EXYNOS_CIWDOFST_CLROVFICB |
EXYNOS_CIWDOFST_CLROVFICR);
- dev_err(ctx->dev, "occurred overflow at %d, status 0x%x.\n",
- ctx->id, status);
+ DRM_DEV_ERROR(ctx->dev,
+ "occurred overflow at %d, status 0x%x.\n",
+ ctx->id, status);
return true;
}
cfg = fimc_read(ctx, EXYNOS_CISTATUS);
- DRM_DEBUG_KMS("cfg[0x%x]\n", cfg);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "cfg[0x%x]\n", cfg);
if (!(cfg & EXYNOS_CISTATUS_FRAMEEND))
return false;
if (frame_cnt == 0)
frame_cnt = EXYNOS_CISTATUS2_GET_FRAMECOUNT_PRESENT(cfg);
- DRM_DEBUG_KMS("present[%d]before[%d]\n",
- EXYNOS_CISTATUS2_GET_FRAMECOUNT_PRESENT(cfg),
- EXYNOS_CISTATUS2_GET_FRAMECOUNT_BEFORE(cfg));
+ DRM_DEV_DEBUG_KMS(ctx->dev, "present[%d]before[%d]\n",
+ EXYNOS_CISTATUS2_GET_FRAMECOUNT_PRESENT(cfg),
+ EXYNOS_CISTATUS2_GET_FRAMECOUNT_BEFORE(cfg));
if (frame_cnt == 0) {
- DRM_ERROR("failed to get frame count.\n");
+ DRM_DEV_ERROR(ctx->dev, "failed to get frame count.\n");
return -EIO;
}
buf_id = frame_cnt - 1;
- DRM_DEBUG_KMS("buf_id[%d]\n", buf_id);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "buf_id[%d]\n", buf_id);
return buf_id;
}
{
u32 cfg;
- DRM_DEBUG_KMS("enable[%d]\n", enable);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "enable[%d]\n", enable);
cfg = fimc_read(ctx, EXYNOS_CIOCTRL);
if (enable)
{
u32 cfg;
- DRM_DEBUG_KMS("fmt[0x%x]\n", fmt);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "fmt[0x%x]\n", fmt);
/* RGB */
cfg = fimc_read(ctx, EXYNOS_CISCCTRL);
{
u32 cfg;
- DRM_DEBUG_KMS("fmt[0x%x]\n", fmt);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "fmt[0x%x]\n", fmt);
cfg = fimc_read(ctx, EXYNOS_MSCTRL);
cfg &= ~EXYNOS_MSCTRL_INFORMAT_RGB;
unsigned int degree = rotation & DRM_MODE_ROTATE_MASK;
u32 cfg1, cfg2;
- DRM_DEBUG_KMS("rotation[%x]\n", rotation);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "rotation[%x]\n", rotation);
cfg1 = fimc_read(ctx, EXYNOS_MSCTRL);
cfg1 &= ~(EXYNOS_MSCTRL_FLIP_X_MIRROR |
v1 = buf->rect.y;
v2 = buf->buf.height - buf->rect.h - buf->rect.y;
- DRM_DEBUG_KMS("x[%d]y[%d]w[%d]h[%d]hsize[%d]vsize[%d]\n",
- buf->rect.x, buf->rect.y, buf->rect.w, buf->rect.h,
- real_width, buf->buf.height);
- DRM_DEBUG_KMS("h1[%d]h2[%d]v1[%d]v2[%d]\n", h1, h2, v1, v2);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "x[%d]y[%d]w[%d]h[%d]hsize[%d]vsize[%d]\n",
+ buf->rect.x, buf->rect.y, buf->rect.w, buf->rect.h,
+ real_width, buf->buf.height);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "h1[%d]h2[%d]v1[%d]v2[%d]\n", h1, h2, v1,
+ v2);
/*
* set window offset 1, 2 size
unsigned int real_width = buf->buf.pitch[0] / buf->format->cpp[0];
u32 cfg;
- DRM_DEBUG_KMS("hsize[%d]vsize[%d]\n", real_width, buf->buf.height);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "hsize[%d]vsize[%d]\n", real_width,
+ buf->buf.height);
/* original size */
cfg = (EXYNOS_ORGISIZE_HORIZONTAL(real_width) |
fimc_write(ctx, cfg, EXYNOS_ORGISIZE);
- DRM_DEBUG_KMS("x[%d]y[%d]w[%d]h[%d]\n", buf->rect.x, buf->rect.y,
- buf->rect.w, buf->rect.h);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "x[%d]y[%d]w[%d]h[%d]\n", buf->rect.x,
+ buf->rect.y, buf->rect.w, buf->rect.h);
/* set input DMA image size */
cfg = fimc_read(ctx, EXYNOS_CIREAL_ISIZE);
{
u32 cfg;
- DRM_DEBUG_KMS("fmt[0x%x]\n", fmt);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "fmt[0x%x]\n", fmt);
/* RGB */
cfg = fimc_read(ctx, EXYNOS_CISCCTRL);
{
u32 cfg;
- DRM_DEBUG_KMS("fmt[0x%x]\n", fmt);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "fmt[0x%x]\n", fmt);
cfg = fimc_read(ctx, EXYNOS_CIEXTEN);
unsigned int degree = rotation & DRM_MODE_ROTATE_MASK;
u32 cfg;
- DRM_DEBUG_KMS("rotation[0x%x]\n", rotation);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "rotation[0x%x]\n", rotation);
cfg = fimc_read(ctx, EXYNOS_CITRGFMT);
cfg &= ~EXYNOS_CITRGFMT_FLIP_MASK;
pre_dst_width = src_w >> hfactor;
pre_dst_height = src_h >> vfactor;
- DRM_DEBUG_KMS("pre_dst_width[%d]pre_dst_height[%d]\n",
- pre_dst_width, pre_dst_height);
- DRM_DEBUG_KMS("hfactor[%d]vfactor[%d]\n", hfactor, vfactor);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "pre_dst_width[%d]pre_dst_height[%d]\n",
+ pre_dst_width, pre_dst_height);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "hfactor[%d]vfactor[%d]\n", hfactor,
+ vfactor);
sc->hratio = (src_w << 14) / (dst_w << hfactor);
sc->vratio = (src_h << 14) / (dst_h << vfactor);
sc->up_h = (dst_w >= src_w) ? true : false;
sc->up_v = (dst_h >= src_h) ? true : false;
- DRM_DEBUG_KMS("hratio[%d]vratio[%d]up_h[%d]up_v[%d]\n",
- sc->hratio, sc->vratio, sc->up_h, sc->up_v);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "hratio[%d]vratio[%d]up_h[%d]up_v[%d]\n",
+ sc->hratio, sc->vratio, sc->up_h, sc->up_v);
shfactor = FIMC_SHFACTOR - (hfactor + vfactor);
- DRM_DEBUG_KMS("shfactor[%d]\n", shfactor);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "shfactor[%d]\n", shfactor);
cfg = (EXYNOS_CISCPRERATIO_SHFACTOR(shfactor) |
EXYNOS_CISCPRERATIO_PREHORRATIO(1 << hfactor) |
{
u32 cfg, cfg_ext;
- DRM_DEBUG_KMS("range[%d]bypass[%d]up_h[%d]up_v[%d]\n",
- sc->range, sc->bypass, sc->up_h, sc->up_v);
- DRM_DEBUG_KMS("hratio[%d]vratio[%d]\n",
- sc->hratio, sc->vratio);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "range[%d]bypass[%d]up_h[%d]up_v[%d]\n",
+ sc->range, sc->bypass, sc->up_h, sc->up_v);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "hratio[%d]vratio[%d]\n",
+ sc->hratio, sc->vratio);
cfg = fimc_read(ctx, EXYNOS_CISCCTRL);
cfg &= ~(EXYNOS_CISCCTRL_SCALERBYPASS |
unsigned int real_width = buf->buf.pitch[0] / buf->format->cpp[0];
u32 cfg, cfg_ext;
- DRM_DEBUG_KMS("hsize[%d]vsize[%d]\n", real_width, buf->buf.height);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "hsize[%d]vsize[%d]\n", real_width,
+ buf->buf.height);
/* original size */
cfg = (EXYNOS_ORGOSIZE_HORIZONTAL(real_width) |
fimc_write(ctx, cfg, EXYNOS_ORGOSIZE);
- DRM_DEBUG_KMS("x[%d]y[%d]w[%d]h[%d]\n", buf->rect.x, buf->rect.y,
- buf->rect.w, buf->rect.h);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "x[%d]y[%d]w[%d]h[%d]\n", buf->rect.x,
+ buf->rect.y,
+ buf->rect.w, buf->rect.h);
/* CSC ITU */
cfg = fimc_read(ctx, EXYNOS_CIGCTRL);
u32 buf_num;
u32 cfg;
- DRM_DEBUG_KMS("buf_id[%d]enqueu[%d]\n", buf_id, enqueue);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "buf_id[%d]enqueu[%d]\n", buf_id, enqueue);
spin_lock_irqsave(&ctx->lock, flags);
struct fimc_context *ctx = dev_id;
int buf_id;
- DRM_DEBUG_KMS("fimc id[%d]\n", ctx->id);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "fimc id[%d]\n", ctx->id);
fimc_clear_irq(ctx);
if (fimc_check_ovf(ctx))
if (buf_id < 0)
return IRQ_HANDLED;
- DRM_DEBUG_KMS("buf_id[%d]\n", buf_id);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "buf_id[%d]\n", buf_id);
if (ctx->task) {
struct exynos_drm_ipp_task *task = ctx->task;
struct exynos_drm_ipp *ipp = &ctx->ipp;
ctx->drm_dev = drm_dev;
+ ipp->drm_dev = drm_dev;
exynos_drm_register_dma(drm_dev, dev);
- exynos_drm_ipp_register(drm_dev, ipp, &ipp_funcs,
+ exynos_drm_ipp_register(dev, ipp, &ipp_funcs,
DRM_EXYNOS_IPP_CAP_CROP | DRM_EXYNOS_IPP_CAP_ROTATE |
DRM_EXYNOS_IPP_CAP_SCALE | DRM_EXYNOS_IPP_CAP_CONVERT,
ctx->formats, ctx->num_formats, "fimc");
struct drm_device *drm_dev = data;
struct exynos_drm_ipp *ipp = &ctx->ipp;
- exynos_drm_ipp_unregister(drm_dev, ipp);
+ exynos_drm_ipp_unregister(dev, ipp);
exynos_drm_unregister_dma(drm_dev, dev);
}
{
struct fimc_context *ctx = get_fimc_context(dev);
- DRM_DEBUG_KMS("id[%d]\n", ctx->id);
+ DRM_DEV_DEBUG_KMS(dev, "id[%d]\n", ctx->id);
clk_disable_unprepare(ctx->clocks[FIMC_CLK_GATE]);
return 0;
}
{
struct fimc_context *ctx = get_fimc_context(dev);
- DRM_DEBUG_KMS("id[%d]\n", ctx->id);
+ DRM_DEV_DEBUG_KMS(dev, "id[%d]\n", ctx->id);
return clk_prepare_enable(ctx->clocks[FIMC_CLK_GATE]);
}
#endif
if (!wait_event_timeout(ctx->wait_vsync_queue,
!atomic_read(&ctx->wait_vsync_event),
HZ/20))
- DRM_DEBUG_KMS("vblank wait timed out.\n");
+ DRM_DEV_DEBUG_KMS(ctx->dev, "vblank wait timed out.\n");
}
static void fimd_enable_video_output(struct fimd_context *ctx, unsigned int win,
struct fimd_context *ctx = crtc->ctx;
unsigned int win, ch_enabled = 0;
- DRM_DEBUG_KMS("%s\n", __FILE__);
-
/* Hardware is in unknown state, so ensure it gets enabled properly */
pm_runtime_get_sync(ctx->dev);
u32 clkdiv;
if (mode->clock == 0) {
- DRM_INFO("Mode has zero clock value.\n");
+ DRM_DEV_ERROR(ctx->dev, "Mode has zero clock value.\n");
return -EINVAL;
}
lcd_rate = clk_get_rate(ctx->lcd_clk);
if (2 * lcd_rate < ideal_clk) {
- DRM_INFO("sclk_fimd clock too low(%lu) for requested pixel clock(%lu)\n",
- lcd_rate, ideal_clk);
+ DRM_DEV_ERROR(ctx->dev,
+ "sclk_fimd clock too low(%lu) for requested pixel clock(%lu)\n",
+ lcd_rate, ideal_clk);
return -EINVAL;
}
/* Find the clock divider value that gets us closest to ideal_clk */
clkdiv = DIV_ROUND_CLOSEST(lcd_rate, ideal_clk);
if (clkdiv >= 0x200) {
- DRM_INFO("requested pixel clock(%lu) too low\n", ideal_clk);
+ DRM_DEV_ERROR(ctx->dev, "requested pixel clock(%lu) too low\n",
+ ideal_clk);
return -EINVAL;
}
driver_data->lcdblk_offset,
0x3 << driver_data->lcdblk_vt_shift,
0x1 << driver_data->lcdblk_vt_shift)) {
- DRM_ERROR("Failed to update sysreg for I80 i/f.\n");
+ DRM_DEV_ERROR(ctx->dev,
+ "Failed to update sysreg for I80 i/f.\n");
return;
}
} else {
driver_data->lcdblk_offset,
0x1 << driver_data->lcdblk_bypass_shift,
0x1 << driver_data->lcdblk_bypass_shift)) {
- DRM_ERROR("Failed to update sysreg for bypass setting.\n");
+ DRM_DEV_ERROR(ctx->dev,
+ "Failed to update sysreg for bypass setting.\n");
return;
}
driver_data->lcdblk_offset,
0x1 << driver_data->lcdblk_mic_bypass_shift,
0x1 << driver_data->lcdblk_mic_bypass_shift)) {
- DRM_ERROR("Failed to update sysreg for bypass mic.\n");
+ DRM_DEV_ERROR(ctx->dev,
+ "Failed to update sysreg for bypass mic.\n");
return;
}
val = (unsigned long)(dma_addr + size);
writel(val, ctx->regs + VIDWx_BUF_END(win, 0));
- DRM_DEBUG_KMS("start addr = 0x%lx, end addr = 0x%lx, size = 0x%lx\n",
- (unsigned long)dma_addr, val, size);
- DRM_DEBUG_KMS("ovl_width = %d, ovl_height = %d\n",
- state->crtc.w, state->crtc.h);
+ DRM_DEV_DEBUG_KMS(ctx->dev,
+ "start addr = 0x%lx, end addr = 0x%lx, size = 0x%lx\n",
+ (unsigned long)dma_addr, val, size);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "ovl_width = %d, ovl_height = %d\n",
+ state->crtc.w, state->crtc.h);
/* buffer size */
buf_offsize = pitch - (state->crtc.w * cpp);
writel(val, ctx->regs + VIDOSD_B(win));
- DRM_DEBUG_KMS("osd pos: tx = %d, ty = %d, bx = %d, by = %d\n",
- state->crtc.x, state->crtc.y, last_x, last_y);
+ DRM_DEV_DEBUG_KMS(ctx->dev,
+ "osd pos: tx = %d, ty = %d, bx = %d, by = %d\n",
+ state->crtc.x, state->crtc.y, last_x, last_y);
/* OSD size */
if (win != 3 && win != 4) {
val = state->crtc.w * state->crtc.h;
writel(val, ctx->regs + offset);
- DRM_DEBUG_KMS("osd size = 0x%x\n", (unsigned int)val);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "osd size = 0x%x\n",
+ (unsigned int)val);
}
fimd_win_set_pixfmt(ctx, win, fb, state->src.w);
ret = clk_prepare_enable(ctx->bus_clk);
if (ret < 0) {
- DRM_ERROR("Failed to prepare_enable the bus clk [%d]\n", ret);
+ DRM_DEV_ERROR(dev,
+ "Failed to prepare_enable the bus clk [%d]\n",
+ ret);
return ret;
}
ret = clk_prepare_enable(ctx->lcd_clk);
if (ret < 0) {
- DRM_ERROR("Failed to prepare_enable the lcd clk [%d]\n", ret);
+ DRM_DEV_ERROR(dev,
+ "Failed to prepare_enable the lcd clk [%d]\n",
+ ret);
return ret;
}
int ret;
if (!size) {
- DRM_ERROR("invalid userptr size.\n");
+ DRM_DEV_ERROR(g2d->dev, "invalid userptr size.\n");
return ERR_PTR(-EINVAL);
}
ret = get_vaddr_frames(start, npages, FOLL_FORCE | FOLL_WRITE,
g2d_userptr->vec);
if (ret != npages) {
- DRM_ERROR("failed to get user pages from userptr.\n");
+ DRM_DEV_ERROR(g2d->dev,
+ "failed to get user pages from userptr.\n");
if (ret < 0)
goto err_destroy_framevec;
ret = -EFAULT;
frame_vector_pages(g2d_userptr->vec),
npages, offset, size, GFP_KERNEL);
if (ret < 0) {
- DRM_ERROR("failed to get sgt from pages.\n");
+ DRM_DEV_ERROR(g2d->dev, "failed to get sgt from pages.\n");
goto err_free_sgt;
}
if (!dma_map_sg(to_dma_dev(g2d->drm_dev), sgt->sgl, sgt->nents,
DMA_BIDIRECTIONAL)) {
- DRM_ERROR("failed to map sgt with dma region.\n");
+ DRM_DEV_ERROR(g2d->dev, "failed to map sgt with dma region.\n");
ret = -ENOMEM;
goto err_sg_free_table;
}
g2d->current_pool = 0;
}
-static enum g2d_reg_type g2d_get_reg_type(int reg_offset)
+static enum g2d_reg_type g2d_get_reg_type(struct g2d_data *g2d, int reg_offset)
{
enum g2d_reg_type reg_type;
break;
default:
reg_type = REG_TYPE_NONE;
- DRM_ERROR("Unknown register offset![%d]\n", reg_offset);
+ DRM_DEV_ERROR(g2d->dev, "Unknown register offset![%d]\n",
+ reg_offset);
break;
}
return bpp;
}
-static bool g2d_check_buf_desc_is_valid(struct g2d_buf_desc *buf_desc,
- enum g2d_reg_type reg_type,
- unsigned long size)
+static bool g2d_check_buf_desc_is_valid(struct g2d_data *g2d,
+ struct g2d_buf_desc *buf_desc,
+ enum g2d_reg_type reg_type,
+ unsigned long size)
{
int width, height;
unsigned long bpp, last_pos;
/* This check also makes sure that right_x > left_x. */
width = (int)buf_desc->right_x - (int)buf_desc->left_x;
if (width < G2D_LEN_MIN || width > G2D_LEN_MAX) {
- DRM_ERROR("width[%d] is out of range!\n", width);
+ DRM_DEV_ERROR(g2d->dev, "width[%d] is out of range!\n", width);
return false;
}
/* This check also makes sure that bottom_y > top_y. */
height = (int)buf_desc->bottom_y - (int)buf_desc->top_y;
if (height < G2D_LEN_MIN || height > G2D_LEN_MAX) {
- DRM_ERROR("height[%d] is out of range!\n", height);
+ DRM_DEV_ERROR(g2d->dev,
+ "height[%d] is out of range!\n", height);
return false;
}
*/
if (last_pos >= size) {
- DRM_ERROR("last engine access position [%lu] "
- "is out of range [%lu]!\n", last_pos, size);
+ DRM_DEV_ERROR(g2d->dev, "last engine access position [%lu] "
+ "is out of range [%lu]!\n", last_pos, size);
return false;
}
offset = cmdlist->data[reg_pos];
handle = cmdlist->data[reg_pos + 1];
- reg_type = g2d_get_reg_type(offset);
+ reg_type = g2d_get_reg_type(g2d, offset);
if (reg_type == REG_TYPE_NONE) {
ret = -EFAULT;
goto err;
goto err;
}
- if (!g2d_check_buf_desc_is_valid(buf_desc,
+ if (!g2d_check_buf_desc_is_valid(g2d, buf_desc,
reg_type, exynos_gem->size)) {
exynos_drm_gem_put(exynos_gem);
ret = -EFAULT;
goto err;
}
- if (!g2d_check_buf_desc_is_valid(buf_desc, reg_type,
- g2d_userptr.size)) {
+ if (!g2d_check_buf_desc_is_valid(g2d, buf_desc,
+ reg_type,
+ g2d_userptr.size)) {
ret = -EFAULT;
goto err;
}
*
* Has to be called under runqueue lock.
*/
-static void g2d_remove_runqueue_nodes(struct g2d_data *g2d, struct drm_file* file)
+static void g2d_remove_runqueue_nodes(struct g2d_data *g2d, struct drm_file *file)
{
struct g2d_runqueue_node *node, *n;
if (!for_addr)
goto err;
- reg_type = g2d_get_reg_type(reg_offset);
+ reg_type = g2d_get_reg_type(g2d, reg_offset);
/* check userptr buffer type. */
if ((cmdlist->data[index] & ~0x7fffffff) >> 31) {
if (for_addr)
goto err;
- reg_type = g2d_get_reg_type(reg_offset);
+ reg_type = g2d_get_reg_type(g2d, reg_offset);
buf_desc = &buf_info->descs[reg_type];
buf_desc->stride = cmdlist->data[index + 1];
if (for_addr)
goto err;
- reg_type = g2d_get_reg_type(reg_offset);
+ reg_type = g2d_get_reg_type(g2d, reg_offset);
buf_desc = &buf_info->descs[reg_type];
value = cmdlist->data[index + 1];
if (for_addr)
goto err;
- reg_type = g2d_get_reg_type(reg_offset);
+ reg_type = g2d_get_reg_type(g2d, reg_offset);
buf_desc = &buf_info->descs[reg_type];
value = cmdlist->data[index + 1];
if (for_addr)
goto err;
- reg_type = g2d_get_reg_type(reg_offset);
+ reg_type = g2d_get_reg_type(g2d, reg_offset);
buf_desc = &buf_info->descs[reg_type];
value = cmdlist->data[index + 1];
int ret = -ENOMEM;
if (exynos_gem->dma_addr) {
- DRM_DEBUG_KMS("already allocated.\n");
+ DRM_DEV_DEBUG_KMS(to_dma_dev(dev), "already allocated.\n");
return 0;
}
exynos_gem->pages = kvmalloc_array(nr_pages, sizeof(struct page *),
GFP_KERNEL | __GFP_ZERO);
if (!exynos_gem->pages) {
- DRM_ERROR("failed to allocate pages.\n");
+ DRM_DEV_ERROR(to_dma_dev(dev), "failed to allocate pages.\n");
return -ENOMEM;
}
&exynos_gem->dma_addr, GFP_KERNEL,
exynos_gem->dma_attrs);
if (!exynos_gem->cookie) {
- DRM_ERROR("failed to allocate buffer.\n");
+ DRM_DEV_ERROR(to_dma_dev(dev), "failed to allocate buffer.\n");
goto err_free;
}
exynos_gem->dma_addr, exynos_gem->size,
exynos_gem->dma_attrs);
if (ret < 0) {
- DRM_ERROR("failed to get sgtable.\n");
+ DRM_DEV_ERROR(to_dma_dev(dev), "failed to get sgtable.\n");
goto err_dma_free;
}
if (drm_prime_sg_to_page_addr_arrays(&sgt, exynos_gem->pages, NULL,
nr_pages)) {
- DRM_ERROR("invalid sgtable.\n");
+ DRM_DEV_ERROR(to_dma_dev(dev), "invalid sgtable.\n");
ret = -EINVAL;
goto err_sgt_free;
}
sg_free_table(&sgt);
- DRM_DEBUG_KMS("dma_addr(0x%lx), size(0x%lx)\n",
+ DRM_DEV_DEBUG_KMS(to_dma_dev(dev), "dma_addr(0x%lx), size(0x%lx)\n",
(unsigned long)exynos_gem->dma_addr, exynos_gem->size);
return 0;
struct drm_device *dev = exynos_gem->base.dev;
if (!exynos_gem->dma_addr) {
- DRM_DEBUG_KMS("dma_addr is invalid.\n");
+ DRM_DEV_DEBUG_KMS(dev->dev, "dma_addr is invalid.\n");
return;
}
- DRM_DEBUG_KMS("dma_addr(0x%lx), size(0x%lx)\n",
+ DRM_DEV_DEBUG_KMS(dev->dev, "dma_addr(0x%lx), size(0x%lx)\n",
(unsigned long)exynos_gem->dma_addr, exynos_gem->size);
dma_free_attrs(to_dma_dev(dev), exynos_gem->size, exynos_gem->cookie,
if (ret)
return ret;
- DRM_DEBUG_KMS("gem handle = 0x%x\n", *handle);
+ DRM_DEV_DEBUG_KMS(to_dma_dev(obj->dev), "gem handle = 0x%x\n", *handle);
/* drop reference from allocate - handle holds it now. */
drm_gem_object_put_unlocked(obj);
{
struct drm_gem_object *obj = &exynos_gem->base;
- DRM_DEBUG_KMS("handle count = %d\n", obj->handle_count);
+ DRM_DEV_DEBUG_KMS(to_dma_dev(obj->dev), "handle count = %d\n",
+ obj->handle_count);
/*
* do not release memory region from exporter.
ret = drm_gem_object_init(dev, obj, size);
if (ret < 0) {
- DRM_ERROR("failed to initialize gem object\n");
+ DRM_DEV_ERROR(dev->dev, "failed to initialize gem object\n");
kfree(exynos_gem);
return ERR_PTR(ret);
}
return ERR_PTR(ret);
}
- DRM_DEBUG_KMS("created file object = %pK\n", obj->filp);
+ DRM_DEV_DEBUG_KMS(dev->dev, "created file object = %pK\n", obj->filp);
return exynos_gem;
}
int ret;
if (flags & ~(EXYNOS_BO_MASK)) {
- DRM_ERROR("invalid GEM buffer flags: %u\n", flags);
+ DRM_DEV_ERROR(dev->dev,
+ "invalid GEM buffer flags: %u\n", flags);
return ERR_PTR(-EINVAL);
}
if (!size) {
- DRM_ERROR("invalid GEM buffer size: %lu\n", size);
+ DRM_DEV_ERROR(dev->dev, "invalid GEM buffer size: %lu\n", size);
return ERR_PTR(-EINVAL);
}
obj = drm_gem_object_lookup(file_priv, args->handle);
if (!obj) {
- DRM_ERROR("failed to lookup gem object.\n");
+ DRM_DEV_ERROR(dev->dev, "failed to lookup gem object.\n");
return -EINVAL;
}
struct exynos_drm_gem *exynos_gem = to_exynos_gem(obj);
int ret;
- DRM_DEBUG_KMS("flags = 0x%x\n", exynos_gem->flags);
+ DRM_DEV_DEBUG_KMS(to_dma_dev(obj->dev), "flags = 0x%x\n",
+ exynos_gem->flags);
/* non-cachable as default. */
if (exynos_gem->flags & EXYNOS_BO_CACHABLE)
}
if (cfg) {
- DRM_ERROR("failed to reset gsc h/w.\n");
+ DRM_DEV_ERROR(ctx->dev, "failed to reset gsc h/w.\n");
return -EBUSY;
}
{
u32 cfg;
- DRM_DEBUG_KMS("enable[%d]overflow[%d]level[%d]\n",
- enable, overflow, done);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "enable[%d]overflow[%d]level[%d]\n",
+ enable, overflow, done);
cfg = gsc_read(GSC_IRQ);
cfg |= (GSC_IRQ_OR_MASK | GSC_IRQ_FRMDONE_MASK);
{
u32 cfg;
- DRM_DEBUG_KMS("fmt[0x%x]\n", fmt);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "fmt[0x%x]\n", fmt);
cfg = gsc_read(GSC_IN_CON);
cfg &= ~(GSC_IN_RGB_TYPE_MASK | GSC_IN_YUV422_1P_ORDER_MASK |
{
u32 cfg;
- DRM_DEBUG_KMS("fmt[0x%x]\n", fmt);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "fmt[0x%x]\n", fmt);
cfg = gsc_read(GSC_OUT_CON);
cfg &= ~(GSC_OUT_RGB_TYPE_MASK | GSC_OUT_YUV422_1P_ORDER_MASK |
gsc_write(cfg, GSC_OUT_CON);
}
-static int gsc_get_ratio_shift(u32 src, u32 dst, u32 *ratio)
+static int gsc_get_ratio_shift(struct gsc_context *ctx, u32 src, u32 dst,
+ u32 *ratio)
{
- DRM_DEBUG_KMS("src[%d]dst[%d]\n", src, dst);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "src[%d]dst[%d]\n", src, dst);
if (src >= dst * 8) {
- DRM_ERROR("failed to make ratio and shift.\n");
+ DRM_DEV_ERROR(ctx->dev, "failed to make ratio and shift.\n");
return -EINVAL;
} else if (src >= dst * 4)
*ratio = 4;
dst_h = dst->h;
}
- ret = gsc_get_ratio_shift(src_w, dst_w, &sc->pre_hratio);
+ ret = gsc_get_ratio_shift(ctx, src_w, dst_w, &sc->pre_hratio);
if (ret) {
- dev_err(ctx->dev, "failed to get ratio horizontal.\n");
+ DRM_DEV_ERROR(ctx->dev, "failed to get ratio horizontal.\n");
return ret;
}
- ret = gsc_get_ratio_shift(src_h, dst_h, &sc->pre_vratio);
+ ret = gsc_get_ratio_shift(ctx, src_h, dst_h, &sc->pre_vratio);
if (ret) {
- dev_err(ctx->dev, "failed to get ratio vertical.\n");
+ DRM_DEV_ERROR(ctx->dev, "failed to get ratio vertical.\n");
return ret;
}
- DRM_DEBUG_KMS("pre_hratio[%d]pre_vratio[%d]\n",
- sc->pre_hratio, sc->pre_vratio);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "pre_hratio[%d]pre_vratio[%d]\n",
+ sc->pre_hratio, sc->pre_vratio);
sc->main_hratio = (src_w << 16) / dst_w;
sc->main_vratio = (src_h << 16) / dst_h;
- DRM_DEBUG_KMS("main_hratio[%ld]main_vratio[%ld]\n",
- sc->main_hratio, sc->main_vratio);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "main_hratio[%ld]main_vratio[%ld]\n",
+ sc->main_hratio, sc->main_vratio);
gsc_get_prescaler_shfactor(sc->pre_hratio, sc->pre_vratio,
&sc->pre_shfactor);
- DRM_DEBUG_KMS("pre_shfactor[%d]\n", sc->pre_shfactor);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "pre_shfactor[%d]\n", sc->pre_shfactor);
cfg = (GSC_PRESC_SHFACTOR(sc->pre_shfactor) |
GSC_PRESC_H_RATIO(sc->pre_hratio) |
{
u32 cfg;
- DRM_DEBUG_KMS("main_hratio[%ld]main_vratio[%ld]\n",
- sc->main_hratio, sc->main_vratio);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "main_hratio[%ld]main_vratio[%ld]\n",
+ sc->main_hratio, sc->main_vratio);
gsc_set_h_coef(ctx, sc->main_hratio);
cfg = GSC_MAIN_H_RATIO_VALUE(sc->main_hratio);
if (cfg & (mask << i))
buf_num--;
- DRM_DEBUG_KMS("buf_num[%d]\n", buf_num);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "buf_num[%d]\n", buf_num);
return buf_num;
}
u32 cfg, curr_index, i;
u32 buf_id = GSC_MAX_SRC;
- DRM_DEBUG_KMS("gsc id[%d]\n", ctx->id);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "gsc id[%d]\n", ctx->id);
cfg = gsc_read(GSC_IN_BASE_ADDR_Y_MASK);
curr_index = GSC_IN_CURR_GET_INDEX(cfg);
}
}
- DRM_DEBUG_KMS("cfg[0x%x]curr_index[%d]buf_id[%d]\n", cfg,
- curr_index, buf_id);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "cfg[0x%x]curr_index[%d]buf_id[%d]\n", cfg,
+ curr_index, buf_id);
if (buf_id == GSC_MAX_SRC) {
- DRM_ERROR("failed to get in buffer index.\n");
+ DRM_DEV_ERROR(ctx->dev, "failed to get in buffer index.\n");
return -EINVAL;
}
u32 cfg, curr_index, i;
u32 buf_id = GSC_MAX_DST;
- DRM_DEBUG_KMS("gsc id[%d]\n", ctx->id);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "gsc id[%d]\n", ctx->id);
cfg = gsc_read(GSC_OUT_BASE_ADDR_Y_MASK);
curr_index = GSC_OUT_CURR_GET_INDEX(cfg);
}
if (buf_id == GSC_MAX_DST) {
- DRM_ERROR("failed to get out buffer index.\n");
+ DRM_DEV_ERROR(ctx->dev, "failed to get out buffer index.\n");
return -EINVAL;
}
gsc_dst_set_buf_seq(ctx, buf_id, false);
- DRM_DEBUG_KMS("cfg[0x%x]curr_index[%d]buf_id[%d]\n", cfg,
- curr_index, buf_id);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "cfg[0x%x]curr_index[%d]buf_id[%d]\n", cfg,
+ curr_index, buf_id);
return buf_id;
}
u32 status;
int err = 0;
- DRM_DEBUG_KMS("gsc id[%d]\n", ctx->id);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "gsc id[%d]\n", ctx->id);
status = gsc_read(GSC_IRQ);
if (status & GSC_IRQ_STATUS_OR_IRQ) {
src_buf_id = gsc_get_src_buf_index(ctx);
dst_buf_id = gsc_get_dst_buf_index(ctx);
- DRM_DEBUG_KMS("buf_id_src[%d]buf_id_dst[%d]\n", src_buf_id,
- dst_buf_id);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "buf_id_src[%d]buf_id_dst[%d]\n",
+ src_buf_id, dst_buf_id);
if (src_buf_id < 0 || dst_buf_id < 0)
err = -EINVAL;
struct drm_device *drm_dev = data;
struct exynos_drm_ipp *ipp = &ctx->ipp;
+ ctx->drm_dev = drm_dev;
ctx->drm_dev = drm_dev;
exynos_drm_register_dma(drm_dev, dev);
- exynos_drm_ipp_register(drm_dev, ipp, &ipp_funcs,
+ exynos_drm_ipp_register(dev, ipp, &ipp_funcs,
DRM_EXYNOS_IPP_CAP_CROP | DRM_EXYNOS_IPP_CAP_ROTATE |
DRM_EXYNOS_IPP_CAP_SCALE | DRM_EXYNOS_IPP_CAP_CONVERT,
ctx->formats, ctx->num_formats, "gsc");
struct drm_device *drm_dev = data;
struct exynos_drm_ipp *ipp = &ctx->ipp;
- exynos_drm_ipp_unregister(drm_dev, ipp);
+ exynos_drm_ipp_unregister(dev, ipp);
exynos_drm_unregister_dma(drm_dev, dev);
}
struct gsc_context *ctx = get_gsc_context(dev);
int i;
- DRM_DEBUG_KMS("id[%d]\n", ctx->id);
+ DRM_DEV_DEBUG_KMS(dev, "id[%d]\n", ctx->id);
for (i = ctx->num_clocks - 1; i >= 0; i--)
clk_disable_unprepare(ctx->clocks[i]);
struct gsc_context *ctx = get_gsc_context(dev);
int i, ret;
- DRM_DEBUG_KMS("id[%d]\n", ctx->id);
+ DRM_DEV_DEBUG_KMS(dev, "id[%d]\n", ctx->id);
for (i = 0; i < ctx->num_clocks; i++) {
ret = clk_prepare_enable(ctx->clocks[i]);
* Returns:
* Zero on success, error code on failure.
*/
-int exynos_drm_ipp_register(struct drm_device *dev, struct exynos_drm_ipp *ipp,
+int exynos_drm_ipp_register(struct device *dev, struct exynos_drm_ipp *ipp,
const struct exynos_drm_ipp_funcs *funcs, unsigned int caps,
const struct exynos_drm_ipp_formats *formats,
unsigned int num_formats, const char *name)
list_add_tail(&ipp->head, &ipp_list);
ipp->id = num_ipp++;
- DRM_DEBUG_DRIVER("Registered ipp %d\n", ipp->id);
+ DRM_DEV_DEBUG_DRIVER(dev, "Registered ipp %d\n", ipp->id);
return 0;
}
* @dev: DRM device
* @ipp: ipp module
*/
-void exynos_drm_ipp_unregister(struct drm_device *dev,
+void exynos_drm_ipp_unregister(struct device *dev,
struct exynos_drm_ipp *ipp)
{
WARN_ON(ipp->task);
task->src.rect.h = task->dst.rect.h = UINT_MAX;
task->transform.rotation = DRM_MODE_ROTATE_0;
- DRM_DEBUG_DRIVER("Allocated task %pK\n", task);
+ DRM_DEV_DEBUG_DRIVER(task->dev, "Allocated task %pK\n", task);
return task;
}
size -= map[i].size;
}
- DRM_DEBUG_DRIVER("Got task %pK configuration from userspace\n", task);
+ DRM_DEV_DEBUG_DRIVER(task->dev,
+ "Got task %pK configuration from userspace\n",
+ task);
return 0;
}
static void exynos_drm_ipp_task_free(struct exynos_drm_ipp *ipp,
struct exynos_drm_ipp_task *task)
{
- DRM_DEBUG_DRIVER("Freeing task %pK\n", task);
+ DRM_DEV_DEBUG_DRIVER(task->dev, "Freeing task %pK\n", task);
exynos_drm_ipp_task_release_buf(&task->src);
exynos_drm_ipp_task_release_buf(&task->dst);
if (task->event)
- drm_event_cancel_free(ipp->dev, &task->event->base);
+ drm_event_cancel_free(ipp->drm_dev, &task->event->base);
kfree(task);
}
buf == src ? DRM_EXYNOS_IPP_FORMAT_SOURCE :
DRM_EXYNOS_IPP_FORMAT_DESTINATION);
if (!fmt) {
- DRM_DEBUG_DRIVER("Task %pK: %s format not supported\n", task,
- buf == src ? "src" : "dst");
+ DRM_DEV_DEBUG_DRIVER(task->dev,
+ "Task %pK: %s format not supported\n",
+ task, buf == src ? "src" : "dst");
return -EINVAL;
}
bool rotate = (rotation != DRM_MODE_ROTATE_0);
bool scale = false;
- DRM_DEBUG_DRIVER("Checking task %pK\n", task);
+ DRM_DEV_DEBUG_DRIVER(task->dev, "Checking task %pK\n", task);
if (src->rect.w == UINT_MAX)
src->rect.w = src->buf.width;
src->rect.y + src->rect.h > (src->buf.height) ||
dst->rect.x + dst->rect.w > (dst->buf.width) ||
dst->rect.y + dst->rect.h > (dst->buf.height)) {
- DRM_DEBUG_DRIVER("Task %pK: defined area is outside provided buffers\n",
- task);
+ DRM_DEV_DEBUG_DRIVER(task->dev,
+ "Task %pK: defined area is outside provided buffers\n",
+ task);
return -EINVAL;
}
(!(ipp->capabilities & DRM_EXYNOS_IPP_CAP_SCALE) && scale) ||
(!(ipp->capabilities & DRM_EXYNOS_IPP_CAP_CONVERT) &&
src->buf.fourcc != dst->buf.fourcc)) {
- DRM_DEBUG_DRIVER("Task %pK: hw capabilities exceeded\n", task);
+ DRM_DEV_DEBUG_DRIVER(task->dev, "Task %pK: hw capabilities exceeded\n",
+ task);
return -EINVAL;
}
if (ret)
return ret;
- DRM_DEBUG_DRIVER("Task %pK: all checks done.\n", task);
+ DRM_DEV_DEBUG_DRIVER(ipp->dev, "Task %pK: all checks done.\n",
+ task);
return ret;
}
struct exynos_drm_ipp_buffer *src = &task->src, *dst = &task->dst;
int ret = 0;
- DRM_DEBUG_DRIVER("Setting buffer for task %pK\n", task);
+ DRM_DEV_DEBUG_DRIVER(task->dev, "Setting buffer for task %pK\n",
+ task);
ret = exynos_drm_ipp_task_setup_buffer(src, filp);
if (ret) {
- DRM_DEBUG_DRIVER("Task %pK: src buffer setup failed\n", task);
+ DRM_DEV_DEBUG_DRIVER(task->dev,
+ "Task %pK: src buffer setup failed\n",
+ task);
return ret;
}
ret = exynos_drm_ipp_task_setup_buffer(dst, filp);
if (ret) {
- DRM_DEBUG_DRIVER("Task %pK: dst buffer setup failed\n", task);
+ DRM_DEV_DEBUG_DRIVER(task->dev,
+ "Task %pK: dst buffer setup failed\n",
+ task);
return ret;
}
- DRM_DEBUG_DRIVER("Task %pK: buffers prepared.\n", task);
+ DRM_DEV_DEBUG_DRIVER(task->dev, "Task %pK: buffers prepared.\n",
+ task);
return ret;
}
e->event.base.length = sizeof(e->event);
e->event.user_data = user_data;
- ret = drm_event_reserve_init(task->dev, file_priv, &e->base,
+ ret = drm_event_reserve_init(task->ipp->drm_dev, file_priv, &e->base,
&e->event.base);
if (ret)
goto free;
task->event->event.tv_usec = now.tv_nsec / NSEC_PER_USEC;
task->event->event.sequence = atomic_inc_return(&task->ipp->sequence);
- drm_send_event(task->dev, &task->event->base);
+ drm_send_event(task->ipp->drm_dev, &task->event->base);
}
static int exynos_drm_ipp_task_cleanup(struct exynos_drm_ipp_task *task)
struct exynos_drm_ipp *ipp = task->ipp;
unsigned long flags;
- DRM_DEBUG_DRIVER("ipp: %d, task %pK done: %d\n", ipp->id, task, ret);
+ DRM_DEV_DEBUG_DRIVER(task->dev, "ipp: %d, task %pK done: %d\n",
+ ipp->id, task, ret);
spin_lock_irqsave(&ipp->lock, flags);
if (ipp->task == task)
unsigned long flags;
int ret;
- DRM_DEBUG_DRIVER("ipp: %d, try to run new task\n", ipp->id);
+ DRM_DEV_DEBUG_DRIVER(ipp->dev, "ipp: %d, try to run new task\n",
+ ipp->id);
spin_lock_irqsave(&ipp->lock, flags);
spin_unlock_irqrestore(&ipp->lock, flags);
- DRM_DEBUG_DRIVER("ipp: %d, selected task %pK to run\n", ipp->id, task);
+ DRM_DEV_DEBUG_DRIVER(ipp->dev,
+ "ipp: %d, selected task %pK to run\n", ipp->id,
+ task);
ret = ipp->funcs->commit(ipp, task);
if (ret)
* then freed after exynos_drm_ipp_task_done()
*/
if (arg->flags & DRM_EXYNOS_IPP_FLAG_NONBLOCK) {
- DRM_DEBUG_DRIVER("ipp: %d, nonblocking processing task %pK\n",
- ipp->id, task);
+ DRM_DEV_DEBUG_DRIVER(ipp->dev,
+ "ipp: %d, nonblocking processing task %pK\n",
+ ipp->id, task);
task->flags |= DRM_EXYNOS_IPP_TASK_ASYNC;
exynos_drm_ipp_schedule_task(task->ipp, task);
ret = 0;
} else {
- DRM_DEBUG_DRIVER("ipp: %d, processing task %pK\n", ipp->id,
- task);
+ DRM_DEV_DEBUG_DRIVER(ipp->dev, "ipp: %d, processing task %pK\n",
+ ipp->id, task);
exynos_drm_ipp_schedule_task(ipp, task);
ret = wait_event_interruptible(ipp->done_wq,
task->flags & DRM_EXYNOS_IPP_TASK_DONE);
* struct exynos_drm_ipp - central picture processor module structure
*/
struct exynos_drm_ipp {
- struct drm_device *dev;
+ struct drm_device *drm_dev;
+ struct device *dev;
struct list_head head;
unsigned int id;
* has to be performed by the picture processor hardware module
*/
struct exynos_drm_ipp_task {
- struct drm_device *dev;
+ struct device *dev;
struct exynos_drm_ipp *ipp;
struct list_head head;
#define IPP_SCALE_LIMIT(val...) \
.type = (DRM_EXYNOS_IPP_LIMIT_TYPE_SCALE), val
-int exynos_drm_ipp_register(struct drm_device *dev, struct exynos_drm_ipp *ipp,
+int exynos_drm_ipp_register(struct device *dev, struct exynos_drm_ipp *ipp,
const struct exynos_drm_ipp_funcs *funcs, unsigned int caps,
const struct exynos_drm_ipp_formats *formats,
unsigned int num_formats, const char *name);
-void exynos_drm_ipp_unregister(struct drm_device *dev,
+void exynos_drm_ipp_unregister(struct device *dev,
struct exynos_drm_ipp *ipp);
void exynos_drm_ipp_task_done(struct exynos_drm_ipp_task *task, int ret);
ret = regmap_read(mic->sysreg, DSD_CFG_MUX, &val);
if (ret) {
- DRM_ERROR("mic: Failed to read system register\n");
+ DRM_DEV_ERROR(mic->dev,
+ "mic: Failed to read system register\n");
return;
}
ret = regmap_write(mic->sysreg, DSD_CFG_MUX, val);
if (ret)
- DRM_ERROR("mic: Failed to read system register\n");
+ DRM_DEV_ERROR(mic->dev,
+ "mic: Failed to read system register\n");
}
static int mic_sw_reset(struct exynos_mic *mic)
struct videomode vm = mic->vm;
u32 reg, bs_size_2d;
- DRM_DEBUG("w: %u, h: %u\n", vm.hactive, vm.vactive);
+ DRM_DEV_DEBUG(mic->dev, "w: %u, h: %u\n", vm.hactive, vm.vactive);
bs_size_2d = ((vm.hactive >> 2) << 1) + (vm.vactive % 4);
reg = MIC_BS_SIZE_2D(bs_size_2d);
writel(reg, mic->reg + MIC_2D_OUTPUT_TIMING_2);
ret = mic_sw_reset(mic);
if (ret) {
- DRM_ERROR("Failed to reset\n");
+ DRM_DEV_ERROR(mic->dev, "Failed to reset\n");
goto turn_off;
}
for (i = 0; i < NUM_CLKS; i++) {
ret = clk_prepare_enable(mic->clks[i]);
if (ret < 0) {
- DRM_ERROR("Failed to enable clock (%s)\n",
- clk_names[i]);
+ DRM_DEV_ERROR(dev, "Failed to enable clock (%s)\n",
+ clk_names[i]);
while (--i > -1)
clk_disable_unprepare(mic->clks[i]);
return ret;
mic = devm_kzalloc(dev, sizeof(*mic), GFP_KERNEL);
if (!mic) {
- DRM_ERROR("mic: Failed to allocate memory for MIC object\n");
+ DRM_DEV_ERROR(dev,
+ "mic: Failed to allocate memory for MIC object\n");
ret = -ENOMEM;
goto err;
}
ret = of_address_to_resource(dev->of_node, 0, &res);
if (ret) {
- DRM_ERROR("mic: Failed to get mem region for MIC\n");
+ DRM_DEV_ERROR(dev, "mic: Failed to get mem region for MIC\n");
goto err;
}
mic->reg = devm_ioremap(dev, res.start, resource_size(&res));
if (!mic->reg) {
- DRM_ERROR("mic: Failed to remap for MIC\n");
+ DRM_DEV_ERROR(dev, "mic: Failed to remap for MIC\n");
ret = -ENOMEM;
goto err;
}
mic->sysreg = syscon_regmap_lookup_by_phandle(dev->of_node,
"samsung,disp-syscon");
if (IS_ERR(mic->sysreg)) {
- DRM_ERROR("mic: Failed to get system register.\n");
+ DRM_DEV_ERROR(dev, "mic: Failed to get system register.\n");
ret = PTR_ERR(mic->sysreg);
goto err;
}
for (i = 0; i < NUM_CLKS; i++) {
mic->clks[i] = devm_clk_get(dev, clk_names[i]);
if (IS_ERR(mic->clks[i])) {
- DRM_ERROR("mic: Failed to get clock (%s)\n",
- clk_names[i]);
+ DRM_DEV_ERROR(dev, "mic: Failed to get clock (%s)\n",
+ clk_names[i]);
ret = PTR_ERR(mic->clks[i]);
goto err;
}
if (ret)
goto err_pm;
- DRM_DEBUG_KMS("MIC has been probed\n");
+ DRM_DEV_DEBUG_KMS(dev, "MIC has been probed\n");
return 0;
exynos_state->crtc.w = actual_w;
exynos_state->crtc.h = actual_h;
- DRM_DEBUG_KMS("plane : offset_x/y(%d,%d), width/height(%d,%d)",
- exynos_state->crtc.x, exynos_state->crtc.y,
- exynos_state->crtc.w, exynos_state->crtc.h);
+ DRM_DEV_DEBUG_KMS(crtc->dev->dev,
+ "plane : offset_x/y(%d,%d), width/height(%d,%d)",
+ exynos_state->crtc.x, exynos_state->crtc.y,
+ exynos_state->crtc.w, exynos_state->crtc.h);
}
static void exynos_drm_plane_reset(struct drm_plane *plane)
struct exynos_drm_plane_state *state)
{
struct drm_framebuffer *fb = state->base.fb;
+ struct drm_device *dev = fb->dev;
switch (fb->modifier) {
case DRM_FORMAT_MOD_SAMSUNG_64_32_TILE:
break;
default:
- DRM_ERROR("unsupported pixel format modifier");
+ DRM_DEV_ERROR(dev->dev, "unsupported pixel format modifier");
return -ENOTSUPP;
}
exynos_drm_plane_check_size(const struct exynos_drm_plane_config *config,
struct exynos_drm_plane_state *state)
{
+ struct drm_crtc *crtc = state->base.crtc;
bool width_ok = false, height_ok = false;
if (config->capabilities & EXYNOS_DRM_PLANE_CAP_SCALE)
if (width_ok && height_ok)
return 0;
- DRM_DEBUG_KMS("scaling mode is not supported");
+ DRM_DEV_DEBUG_KMS(crtc->dev->dev, "scaling mode is not supported");
return -ENOTSUPP;
}
config->num_pixel_formats,
NULL, config->type, NULL);
if (err) {
- DRM_ERROR("failed to initialize plane\n");
+ DRM_DEV_ERROR(dev->dev, "failed to initialize plane\n");
return err;
}
struct exynos_drm_ipp *ipp = &rot->ipp;
rot->drm_dev = drm_dev;
+ ipp->drm_dev = drm_dev;
exynos_drm_register_dma(drm_dev, dev);
- exynos_drm_ipp_register(drm_dev, ipp, &ipp_funcs,
+ exynos_drm_ipp_register(dev, ipp, &ipp_funcs,
DRM_EXYNOS_IPP_CAP_CROP | DRM_EXYNOS_IPP_CAP_ROTATE,
rot->formats, rot->num_formats, "rotator");
void *data)
{
struct rot_context *rot = dev_get_drvdata(dev);
- struct drm_device *drm_dev = data;
struct exynos_drm_ipp *ipp = &rot->ipp;
- exynos_drm_ipp_unregister(drm_dev, ipp);
+ exynos_drm_ipp_unregister(dev, ipp);
exynos_drm_unregister_dma(rot->drm_dev, rot->dev);
}
struct exynos_drm_ipp *ipp = &scaler->ipp;
scaler->drm_dev = drm_dev;
+ ipp->drm_dev = drm_dev;
exynos_drm_register_dma(drm_dev, dev);
- exynos_drm_ipp_register(drm_dev, ipp, &ipp_funcs,
+ exynos_drm_ipp_register(dev, ipp, &ipp_funcs,
DRM_EXYNOS_IPP_CAP_CROP | DRM_EXYNOS_IPP_CAP_ROTATE |
DRM_EXYNOS_IPP_CAP_SCALE | DRM_EXYNOS_IPP_CAP_CONVERT,
scaler->scaler_data->formats,
void *data)
{
struct scaler_context *scaler = dev_get_drvdata(dev);
- struct drm_device *drm_dev = data;
struct exynos_drm_ipp *ipp = &scaler->ipp;
- exynos_drm_ipp_unregister(drm_dev, ipp);
+ exynos_drm_ipp_unregister(dev, ipp);
exynos_drm_unregister_dma(scaler->drm_dev, scaler->dev);
}
struct vidi_context {
struct drm_encoder encoder;
- struct platform_device *pdev;
struct drm_device *drm_dev;
+ struct device *dev;
struct exynos_drm_crtc *crtc;
struct drm_connector connector;
struct exynos_drm_plane planes[WINDOWS_NR];
return;
addr = exynos_drm_fb_dma_addr(state->fb, 0);
- DRM_DEBUG_KMS("dma_addr = %pad\n", &addr);
+ DRM_DEV_DEBUG_KMS(ctx->dev, "dma_addr = %pad\n", &addr);
}
static void vidi_enable(struct exynos_drm_crtc *crtc)
/* if raw_edid isn't same as fake data then it can't be tested. */
if (ctx->raw_edid != (struct edid *)fake_edid_info) {
- DRM_DEBUG_KMS("edid data is not fake data.\n");
+ DRM_DEV_DEBUG_KMS(dev, "edid data is not fake data.\n");
return -EINVAL;
}
- DRM_DEBUG_KMS("requested connection.\n");
+ DRM_DEV_DEBUG_KMS(dev, "requested connection.\n");
drm_helper_hpd_irq_event(ctx->drm_dev);
struct drm_exynos_vidi_connection *vidi = data;
if (!vidi) {
- DRM_DEBUG_KMS("user data for vidi is null.\n");
+ DRM_DEV_DEBUG_KMS(ctx->dev,
+ "user data for vidi is null.\n");
return -EINVAL;
}
if (vidi->connection > 1) {
- DRM_DEBUG_KMS("connection should be 0 or 1.\n");
+ DRM_DEV_DEBUG_KMS(ctx->dev,
+ "connection should be 0 or 1.\n");
return -EINVAL;
}
if (ctx->connected == vidi->connection) {
- DRM_DEBUG_KMS("same connection request.\n");
+ DRM_DEV_DEBUG_KMS(ctx->dev,
+ "same connection request.\n");
return -EINVAL;
}
raw_edid = (struct edid *)(unsigned long)vidi->edid;
if (!drm_edid_is_valid(raw_edid)) {
- DRM_DEBUG_KMS("edid data is invalid.\n");
+ DRM_DEV_DEBUG_KMS(ctx->dev,
+ "edid data is invalid.\n");
return -EINVAL;
}
ctx->raw_edid = drm_edid_duplicate(raw_edid);
if (!ctx->raw_edid) {
- DRM_DEBUG_KMS("failed to allocate raw_edid.\n");
+ DRM_DEV_DEBUG_KMS(ctx->dev,
+ "failed to allocate raw_edid.\n");
return -ENOMEM;
}
} else {
* to ctx->raw_edid through specific ioctl.
*/
if (!ctx->raw_edid) {
- DRM_DEBUG_KMS("raw_edid is null.\n");
+ DRM_DEV_DEBUG_KMS(ctx->dev, "raw_edid is null.\n");
return -EFAULT;
}
edid_len = (1 + ctx->raw_edid->extensions) * EDID_LENGTH;
edid = kmemdup(ctx->raw_edid, edid_len, GFP_KERNEL);
if (!edid) {
- DRM_DEBUG_KMS("failed to allocate edid\n");
+ DRM_DEV_DEBUG_KMS(ctx->dev, "failed to allocate edid\n");
return -ENOMEM;
}
ret = drm_connector_init(ctx->drm_dev, connector,
&vidi_connector_funcs, DRM_MODE_CONNECTOR_VIRTUAL);
if (ret) {
- DRM_ERROR("Failed to initialize connector with drm\n");
+ DRM_DEV_ERROR(ctx->dev,
+ "Failed to initialize connector with drm\n");
return ret;
}
ctx->crtc = exynos_drm_crtc_create(drm_dev, &exynos_plane->base,
EXYNOS_DISPLAY_TYPE_VIDI, &vidi_crtc_ops, ctx);
if (IS_ERR(ctx->crtc)) {
- DRM_ERROR("failed to create crtc.\n");
+ DRM_DEV_ERROR(dev, "failed to create crtc.\n");
return PTR_ERR(ctx->crtc);
}
ret = vidi_create_connector(encoder);
if (ret) {
- DRM_ERROR("failed to create connector ret = %d\n", ret);
+ DRM_DEV_ERROR(dev, "failed to create connector ret = %d\n",
+ ret);
drm_encoder_cleanup(encoder);
return ret;
}
static int vidi_probe(struct platform_device *pdev)
{
struct vidi_context *ctx;
+ struct device *dev = &pdev->dev;
int ret;
- ctx = devm_kzalloc(&pdev->dev, sizeof(*ctx), GFP_KERNEL);
+ ctx = devm_kzalloc(dev, sizeof(*ctx), GFP_KERNEL);
if (!ctx)
return -ENOMEM;
- ctx->pdev = pdev;
+ ctx->dev = dev;
timer_setup(&ctx->timer, vidi_fake_vblank_timer, 0);
platform_set_drvdata(pdev, ctx);
- ret = device_create_file(&pdev->dev, &dev_attr_connection);
+ ret = device_create_file(dev, &dev_attr_connection);
if (ret < 0) {
- DRM_ERROR("failed to create connection sysfs.\n");
+ DRM_DEV_ERROR(dev,
+ "failed to create connection sysfs.\n");
return ret;
}
- ret = component_add(&pdev->dev, &vidi_component_ops);
+ ret = component_add(dev, &vidi_component_ops);
if (ret)
goto err_remove_file;
return ret;
err_remove_file:
- device_remove_file(&pdev->dev, &dev_attr_connection);
+ device_remove_file(dev, &dev_attr_connection);
return ret;
}
return -ENODEV;
hdata->dvi_mode = !drm_detect_hdmi_monitor(edid);
- DRM_DEBUG_KMS("%s : width[%d] x height[%d]\n",
- (hdata->dvi_mode ? "dvi monitor" : "hdmi monitor"),
- edid->width_cm, edid->height_cm);
+ DRM_DEV_DEBUG_KMS(hdata->dev, "%s : width[%d] x height[%d]\n",
+ (hdata->dvi_mode ? "dvi monitor" : "hdmi monitor"),
+ edid->width_cm, edid->height_cm);
drm_connector_update_edid_property(connector, edid);
cec_notifier_set_phys_addr_from_edid(hdata->notifier, edid);
if (confs->data[i].pixel_clock == pixel_clock)
return i;
- DRM_DEBUG_KMS("Could not find phy config for %d\n", pixel_clock);
+ DRM_DEV_DEBUG_KMS(hdata->dev, "Could not find phy config for %d\n",
+ pixel_clock);
return -EINVAL;
}
struct hdmi_context *hdata = connector_to_hdmi(connector);
int ret;
- DRM_DEBUG_KMS("xres=%d, yres=%d, refresh=%d, intl=%d clock=%d\n",
- mode->hdisplay, mode->vdisplay, mode->vrefresh,
- (mode->flags & DRM_MODE_FLAG_INTERLACE) ? true :
- false, mode->clock * 1000);
+ DRM_DEV_DEBUG_KMS(hdata->dev,
+ "xres=%d, yres=%d, refresh=%d, intl=%d clock=%d\n",
+ mode->hdisplay, mode->vdisplay, mode->vrefresh,
+ (mode->flags & DRM_MODE_FLAG_INTERLACE) ? true :
+ false, mode->clock * 1000);
ret = hdmi_find_phy_conf(hdata, mode->clock * 1000);
if (ret < 0)
ret = drm_connector_init(hdata->drm_dev, connector,
&hdmi_connector_funcs, DRM_MODE_CONNECTOR_HDMIA);
if (ret) {
- DRM_ERROR("Failed to initialize connector with drm\n");
+ DRM_DEV_ERROR(hdata->dev,
+ "Failed to initialize connector with drm\n");
return ret;
}
if (hdata->bridge) {
ret = drm_bridge_attach(encoder, hdata->bridge, NULL);
if (ret)
- DRM_ERROR("Failed to attach bridge\n");
+ DRM_DEV_ERROR(hdata->dev, "Failed to attach bridge\n");
}
return ret;
DRM_INFO("desired mode doesn't exist so\n");
DRM_INFO("use the most suitable mode among modes.\n");
- DRM_DEBUG_KMS("Adjusted Mode: [%d]x[%d] [%d]Hz\n",
- m->hdisplay, m->vdisplay, m->vrefresh);
+ DRM_DEV_DEBUG_KMS(dev->dev,
+ "Adjusted Mode: [%d]x[%d] [%d]Hz\n",
+ m->hdisplay, m->vdisplay,
+ m->vrefresh);
drm_mode_copy(adjusted_mode, m);
break;
u32 val = hdmi_reg_read(hdata, HDMI_PHY_STATUS);
if (val & HDMI_PHY_STATUS_READY) {
- DRM_DEBUG_KMS("PLL stabilized after %d tries\n", tries);
+ DRM_DEV_DEBUG_KMS(hdata->dev,
+ "PLL stabilized after %d tries\n",
+ tries);
return;
}
usleep_range(10, 20);
}
- DRM_ERROR("PLL could not reach steady state\n");
+ DRM_DEV_ERROR(hdata->dev, "PLL could not reach steady state\n");
}
static void hdmi_v13_mode_apply(struct hdmi_context *hdata)
ret = hdmi_find_phy_conf(hdata, m->clock * 1000);
if (ret < 0) {
- DRM_ERROR("failed to find hdmiphy conf\n");
+ DRM_DEV_ERROR(hdata->dev, "failed to find hdmiphy conf\n");
return;
}
phy_conf = hdata->drv_data->phy_confs.data[ret].conf;
hdmiphy_enable_mode_set(hdata, true);
ret = hdmiphy_reg_write_buf(hdata, 0, phy_conf, 32);
if (ret) {
- DRM_ERROR("failed to configure hdmiphy\n");
+ DRM_DEV_ERROR(hdata->dev, "failed to configure hdmiphy\n");
return;
}
hdmiphy_enable_mode_set(hdata, false);
pm_runtime_get_sync(hdata->dev);
if (regulator_bulk_enable(ARRAY_SIZE(supply), hdata->regul_bulk))
- DRM_DEBUG_KMS("failed to enable regulator bulk\n");
+ DRM_DEV_DEBUG_KMS(hdata->dev,
+ "failed to enable regulator bulk\n");
regmap_update_bits(hdata->pmureg, PMU_HDMI_PHY_CONTROL,
PMU_HDMI_PHY_ENABLE_BIT, 1);
np = of_graph_get_remote_port_parent(ep);
of_node_put(ep);
if (!np) {
- DRM_ERROR("failed to get remote port parent");
+ DRM_DEV_ERROR(dev, "failed to get remote port parent");
return -EINVAL;
}
struct device *dev = hdata->dev;
int i, ret;
- DRM_DEBUG_KMS("HDMI resource init\n");
+ DRM_DEV_DEBUG_KMS(dev, "HDMI resource init\n");
hdata->hpd_gpio = devm_gpiod_get(dev, "hpd", GPIOD_IN);
if (IS_ERR(hdata->hpd_gpio)) {
- DRM_ERROR("cannot get hpd gpio property\n");
+ DRM_DEV_ERROR(dev, "cannot get hpd gpio property\n");
return PTR_ERR(hdata->hpd_gpio);
}
hdata->irq = gpiod_to_irq(hdata->hpd_gpio);
if (hdata->irq < 0) {
- DRM_ERROR("failed to get GPIO irq\n");
+ DRM_DEV_ERROR(dev, "failed to get GPIO irq\n");
return hdata->irq;
}
ret = devm_regulator_bulk_get(dev, ARRAY_SIZE(supply), hdata->regul_bulk);
if (ret) {
if (ret != -EPROBE_DEFER)
- DRM_ERROR("failed to get regulators\n");
+ DRM_DEV_ERROR(dev, "failed to get regulators\n");
return ret;
}
ret = regulator_enable(hdata->reg_hdmi_en);
if (ret) {
- DRM_ERROR("failed to enable hdmi-en regulator\n");
+ DRM_DEV_ERROR(dev,
+ "failed to enable hdmi-en regulator\n");
return ret;
}
}
ret = hdmi_create_connector(encoder);
if (ret) {
- DRM_ERROR("failed to create connector ret = %d\n", ret);
+ DRM_DEV_ERROR(dev, "failed to create connector ret = %d\n",
+ ret);
drm_encoder_cleanup(encoder);
return ret;
}
np = of_parse_phandle(hdata->dev->of_node, "ddc", 0);
if (!np) {
- DRM_ERROR("Failed to find ddc node in device tree\n");
+ DRM_DEV_ERROR(hdata->dev,
+ "Failed to find ddc node in device tree\n");
return -ENODEV;
}
if (!np) {
np = of_parse_phandle(hdata->dev->of_node, "phy", 0);
if (!np) {
- DRM_ERROR("Failed to find hdmiphy node in device tree\n");
+ DRM_DEV_ERROR(hdata->dev,
+ "Failed to find hdmiphy node in device tree\n");
return -ENODEV;
}
}
if (hdata->drv_data->is_apb_phy) {
hdata->regs_hdmiphy = of_iomap(np, 0);
if (!hdata->regs_hdmiphy) {
- DRM_ERROR("failed to ioremap hdmi phy\n");
+ DRM_DEV_ERROR(hdata->dev,
+ "failed to ioremap hdmi phy\n");
ret = -ENOMEM;
goto out;
}
ret = hdmi_resources_init(hdata);
if (ret) {
if (ret != -EPROBE_DEFER)
- DRM_ERROR("hdmi_resources_init failed\n");
+ DRM_DEV_ERROR(dev, "hdmi_resources_init failed\n");
return ret;
}
IRQF_TRIGGER_FALLING | IRQF_ONESHOT,
"hdmi", hdata);
if (ret) {
- DRM_ERROR("failed to register hdmi interrupt\n");
+ DRM_DEV_ERROR(dev, "failed to register hdmi interrupt\n");
goto err_hdmiphy;
}
hdata->pmureg = syscon_regmap_lookup_by_phandle(dev->of_node,
"samsung,syscon-phandle");
if (IS_ERR(hdata->pmureg)) {
- DRM_ERROR("syscon regmap lookup failed.\n");
+ DRM_DEV_ERROR(dev, "syscon regmap lookup failed.\n");
ret = -EPROBE_DEFER;
goto err_hdmiphy;
}
hdata->sysreg = syscon_regmap_lookup_by_phandle(dev->of_node,
"samsung,sysreg-phandle");
if (IS_ERR(hdata->sysreg)) {
- DRM_ERROR("sysreg regmap lookup failed.\n");
+ DRM_DEV_ERROR(dev, "sysreg regmap lookup failed.\n");
ret = -EPROBE_DEFER;
goto err_hdmiphy;
}
{
#define DUMPREG(reg_id) \
do { \
- DRM_DEBUG_KMS(#reg_id " = %08x\n", \
- (u32)readl(ctx->mixer_regs + reg_id)); \
+ DRM_DEV_DEBUG_KMS(ctx->dev, #reg_id " = %08x\n", \
+ (u32)readl(ctx->mixer_regs + reg_id)); \
} while (0)
DUMPREG(MXR_STATUS);
{
#define DUMPREG(reg_id) \
do { \
- DRM_DEBUG_KMS(#reg_id " = %08x\n", \
- (u32) readl(ctx->vp_regs + reg_id)); \
+ DRM_DEV_DEBUG_KMS(ctx->dev, #reg_id " = %08x\n", \
+ (u32) readl(ctx->vp_regs + reg_id)); \
} while (0)
DUMPREG(VP_ENABLE);
/* acquire resources: regs, irqs, clocks */
ret = mixer_resources_init(mixer_ctx);
if (ret) {
- DRM_ERROR("mixer_resources_init failed ret=%d\n", ret);
+ DRM_DEV_ERROR(mixer_ctx->dev,
+ "mixer_resources_init failed ret=%d\n", ret);
return ret;
}
/* acquire vp resources: regs, irqs, clocks */
ret = vp_resources_init(mixer_ctx);
if (ret) {
- DRM_ERROR("vp_resources_init failed ret=%d\n", ret);
+ DRM_DEV_ERROR(mixer_ctx->dev,
+ "vp_resources_init failed ret=%d\n", ret);
return ret;
}
}
{
struct mixer_context *mixer_ctx = crtc->ctx;
- DRM_DEBUG_KMS("win: %d\n", plane->index);
+ DRM_DEV_DEBUG_KMS(mixer_ctx->dev, "win: %d\n", plane->index);
if (!test_bit(MXR_BIT_POWERED, &mixer_ctx->flags))
return;
struct mixer_context *mixer_ctx = crtc->ctx;
unsigned long flags;
- DRM_DEBUG_KMS("win: %d\n", plane->index);
+ DRM_DEV_DEBUG_KMS(mixer_ctx->dev, "win: %d\n", plane->index);
if (!test_bit(MXR_BIT_POWERED, &mixer_ctx->flags))
return;
struct mixer_context *ctx = crtc->ctx;
u32 w = mode->hdisplay, h = mode->vdisplay;
- DRM_DEBUG_KMS("xres=%d, yres=%d, refresh=%d, intl=%d\n", w, h,
- mode->vrefresh, !!(mode->flags & DRM_MODE_FLAG_INTERLACE));
+ DRM_DEV_DEBUG_KMS(ctx->dev, "xres=%d, yres=%d, refresh=%d, intl=%d\n",
+ w, h, mode->vrefresh,
+ !!(mode->flags & DRM_MODE_FLAG_INTERLACE));
if (ctx->mxr_ver == MXR_VER_128_0_0_184)
return MODE_OK;
ctx = devm_kzalloc(&pdev->dev, sizeof(*ctx), GFP_KERNEL);
if (!ctx) {
- DRM_ERROR("failed to alloc mixer context.\n");
+ DRM_DEV_ERROR(dev, "failed to alloc mixer context.\n");
return -ENOMEM;
}
ret = clk_prepare_enable(ctx->mixer);
if (ret < 0) {
- DRM_ERROR("Failed to prepare_enable the mixer clk [%d]\n", ret);
+ DRM_DEV_ERROR(ctx->dev,
+ "Failed to prepare_enable the mixer clk [%d]\n",
+ ret);
return ret;
}
ret = clk_prepare_enable(ctx->hdmi);
if (ret < 0) {
- DRM_ERROR("Failed to prepare_enable the hdmi clk [%d]\n", ret);
+ DRM_DEV_ERROR(dev,
+ "Failed to prepare_enable the hdmi clk [%d]\n",
+ ret);
return ret;
}
if (test_bit(MXR_BIT_VP_ENABLED, &ctx->flags)) {
ret = clk_prepare_enable(ctx->vp);
if (ret < 0) {
- DRM_ERROR("Failed to prepare_enable the vp clk [%d]\n",
- ret);
+ DRM_DEV_ERROR(dev,
+ "Failed to prepare_enable the vp clk [%d]\n",
+ ret);
return ret;
}
if (test_bit(MXR_BIT_HAS_SCLK, &ctx->flags)) {
ret = clk_prepare_enable(ctx->sclk_mixer);
if (ret < 0) {
- DRM_ERROR("Failed to prepare_enable the " \
+ DRM_DEV_ERROR(dev,
+ "Failed to prepare_enable the " \
"sclk_mixer clk [%d]\n",
- ret);
+ ret);
return ret;
}
}
drm_display_mode_to_videomode(mode, &vm);
/* INV_PXCK as default (most display sample data on rising edge) */
- if (!(con->display_info.bus_flags & DRM_BUS_FLAG_PIXDATA_POSEDGE))
+ if (!(con->display_info.bus_flags & DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE))
pol |= DCU_SYN_POL_INV_PXCK;
if (vm.flags & DISPLAY_FLAGS_HSYNC_LOW)
ret = PTR_ERR(info);
goto out;
}
- info->par = fbdev;
mode_cmd.pixel_format = drm_mode_legacy_fb_format(bpp, depth);
fbdev->psb_fb_helper.fb = fb;
- drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
- strcpy(info->fix.id, "psbdrmfb");
-
if (dev_priv->ops->accel_2d && pitch_lines > 8) /* 2D engine */
info->fbops = &psbfb_ops;
else if (gtt_roll) { /* GTT rolling seems best */
info->apertures->ranges[0].size = dev_priv->gtt.stolen_size;
}
- drm_fb_helper_fill_var(info, &fbdev->psb_fb_helper,
- sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(info, &fbdev->psb_fb_helper, sizes);
info->fix.mmio_start = pci_resource_start(dev->pdev, 0);
info->fix.mmio_len = pci_resource_len(dev->pdev, 0);
};
struct psb_fbdev {
- struct drm_fb_helper psb_fb_helper;
+ struct drm_fb_helper psb_fb_helper; /* must be first */
struct psb_framebuffer pfb;
};
};
struct hibmc_fbdev {
- struct drm_fb_helper helper;
+ struct drm_fb_helper helper; /* must be first */
struct hibmc_framebuffer *fb;
int size;
};
goto out_release_fbi;
}
- info->par = hi_fbdev;
-
hi_fbdev->fb = hibmc_framebuffer_init(priv->dev, &mode_cmd, gobj);
if (IS_ERR(hi_fbdev->fb)) {
ret = PTR_ERR(hi_fbdev->fb);
priv->fbdev->size = size;
hi_fbdev->helper.fb = &hi_fbdev->fb->fb;
- strcpy(info->fix.id, "hibmcdrmfb");
-
info->fbops = &hibmc_drm_fb_ops;
- drm_fb_helper_fill_fix(info, hi_fbdev->fb->fb.pitches[0],
- hi_fbdev->fb->fb.format->depth);
- drm_fb_helper_fill_var(info, &priv->fbdev->helper, sizes->fb_width,
- sizes->fb_height);
+ drm_fb_helper_fill_info(info, &priv->fbdev->helper, sizes);
info->screen_base = bo->kmap.virtual;
info->screen_size = size;
return MODE_OK;
}
-static struct drm_encoder *
-hibmc_connector_best_encoder(struct drm_connector *connector)
-{
- return drm_encoder_find(connector->dev, NULL, connector->encoder_ids[0]);
-}
-
static const struct drm_connector_helper_funcs
hibmc_connector_helper_funcs = {
.get_modes = hibmc_connector_get_modes,
.mode_valid = hibmc_connector_mode_valid,
- .best_encoder = hibmc_connector_best_encoder,
};
static const struct drm_connector_funcs hibmc_connector_funcs = {
#include "hibmc_drm_drv.h"
-#define DRM_FILE_PAGE_OFFSET (0x100000000ULL >> PAGE_SHIFT)
-
static inline struct hibmc_drm_private *
hibmc_bdev(struct ttm_bo_device *bd)
{
ret = ttm_bo_device_init(&hibmc->bdev,
&hibmc_bo_driver,
dev->anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET,
true);
if (ret) {
DRM_ERROR("error initializing bo driver: %d\n", ret);
int hibmc_mmap(struct file *filp, struct vm_area_struct *vma)
{
- struct drm_file *file_priv;
- struct hibmc_drm_private *hibmc;
-
- if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET))
- return -EINVAL;
+ struct drm_file *file_priv = filp->private_data;
+ struct hibmc_drm_private *hibmc = file_priv->minor->dev->dev_private;
- file_priv = filp->private_data;
- hibmc = file_priv->minor->dev->dev_private;
return ttm_bo_mmap(filp, vma, &hibmc->bdev);
}
--- /dev/null
+header_test_*.c
subdir-ccflags-y += \
$(call as-instr,movntdqa (%eax)$(comma)%xmm0,-DCONFIG_AS_MOVNTDQA)
+# Extra header tests
+include $(src)/Makefile.header-test
+
# Please keep these build lists sorted!
# core driver code
-i915-y := i915_drv.o \
+i915-y += i915_drv.o \
i915_irq.o \
i915_memcpy.o \
i915_mm.o \
i915_sw_fence.o \
i915_syncmap.o \
i915_sysfs.o \
+ i915_user_extensions.o \
intel_csr.o \
intel_device_info.o \
intel_pm.o \
i915_gem_tiling.o \
i915_gem_userptr.o \
i915_gemfs.o \
+ i915_globals.o \
i915_query.o \
i915_request.o \
i915_scheduler.o \
i915_trace_points.o \
i915_vma.o \
intel_breadcrumbs.o \
+ intel_context.o \
intel_engine_cs.o \
intel_hangcheck.o \
intel_lrc.o \
--- /dev/null
+# SPDX-License-Identifier: MIT
+# Copyright © 2019 Intel Corporation
+
+# Test the headers are compilable as standalone units
+header_test := \
+ i915_active_types.h \
+ i915_gem_context_types.h \
+ i915_priolist_types.h \
+ i915_scheduler_types.h \
+ i915_timeline_types.h \
+ intel_atomic_plane.h \
+ intel_audio.h \
+ intel_cdclk.h \
+ intel_color.h \
+ intel_connector.h \
+ intel_context_types.h \
+ intel_crt.h \
+ intel_csr.h \
+ intel_ddi.h \
+ intel_dp.h \
+ intel_dvo.h \
+ intel_engine_types.h \
+ intel_fbc.h \
+ intel_fbdev.h \
+ intel_frontbuffer.h \
+ intel_hdcp.h \
+ intel_hdmi.h \
+ intel_lspcon.h \
+ intel_lvds.h \
+ intel_panel.h \
+ intel_pipe_crc.h \
+ intel_pm.h \
+ intel_psr.h \
+ intel_sdvo.h \
+ intel_sprite.h \
+ intel_tv.h \
+ intel_workarounds_types.h
+
+quiet_cmd_header_test = HDRTEST $@
+ cmd_header_test = echo "\#include \"$(<F)\"" > $@
+
+header_test_%.c: %.h
+ $(call cmd,header_test)
+
+i915-$(CONFIG_DRM_I915_WERROR) += $(foreach h,$(header_test),$(patsubst %.h,header_test_%.o,$(h)))
+
+clean-files += $(foreach h,$(header_test),$(patsubst %.h,header_test_%.c,$(h)))
execlist.o scheduler.o sched_policy.o mmio_context.o cmd_parser.o debugfs.o \
fb_decoder.o dmabuf.o page_track.o
-ccflags-y += -I$(src) -I$(src)/$(GVT_DIR)
+ccflags-y += -I $(srctree)/$(src) -I $(srctree)/$(src)/$(GVT_DIR)/
i915-y += $(addprefix $(GVT_DIR)/, $(GVT_SOURCE))
#define F_POST_HANDLE (1<<2)
u32 flag;
-#define R_RCS (1 << RCS)
-#define R_VCS1 (1 << VCS)
-#define R_VCS2 (1 << VCS2)
+#define R_RCS BIT(RCS0)
+#define R_VCS1 BIT(VCS0)
+#define R_VCS2 BIT(VCS1)
#define R_VCS (R_VCS1 | R_VCS2)
-#define R_BCS (1 << BCS)
-#define R_VECS (1 << VECS)
+#define R_BCS BIT(BCS0)
+#define R_VECS BIT(VECS0)
#define R_ALL (R_RCS | R_VCS | R_BCS | R_VECS)
/* rings that support this cmd: BLT/RCS/VCS/VECS */
u16 rings;
};
static const struct decode_info *ring_decode_info[I915_NUM_ENGINES][8] = {
- [RCS] = {
+ [RCS0] = {
&decode_info_mi,
NULL,
NULL,
NULL,
},
- [VCS] = {
+ [VCS0] = {
&decode_info_mi,
NULL,
NULL,
NULL,
},
- [BCS] = {
+ [BCS0] = {
&decode_info_mi,
NULL,
&decode_info_2d,
NULL,
},
- [VECS] = {
+ [VECS0] = {
&decode_info_mi,
NULL,
NULL,
NULL,
},
- [VCS2] = {
+ [VCS1] = {
&decode_info_mi,
NULL,
NULL,
struct cmd_entry *e;
hash_for_each_possible(gvt->cmd_table, e, hlist, opcode) {
- if ((opcode == e->info->opcode) &&
- (e->info->rings & (1 << ring_id)))
+ if (opcode == e->info->opcode && e->info->rings & BIT(ring_id))
return e->info;
}
return NULL;
struct intel_gvt *gvt = s->vgpu->gvt;
for (i = 1; i < cmd_len; i += 2) {
- if (IS_BROADWELL(gvt->dev_priv) &&
- (s->ring_id != RCS)) {
- if (s->ring_id == BCS &&
- cmd_reg(s, i) ==
- i915_mmio_reg_offset(DERRMR))
+ if (IS_BROADWELL(gvt->dev_priv) && s->ring_id != RCS0) {
+ if (s->ring_id == BCS0 &&
+ cmd_reg(s, i) == i915_mmio_reg_offset(DERRMR))
ret |= 0;
else
- ret |= (cmd_reg_inhibit(s, i)) ?
- -EBADRQC : 0;
+ ret |= cmd_reg_inhibit(s, i) ? -EBADRQC : 0;
}
if (ret)
break;
};
static struct cmd_interrupt_event cmd_interrupt_events[] = {
- [RCS] = {
+ [RCS0] = {
.pipe_control_notify = RCS_PIPE_CONTROL,
.mi_flush_dw = INTEL_GVT_EVENT_RESERVED,
.mi_user_interrupt = RCS_MI_USER_INTERRUPT,
},
- [BCS] = {
+ [BCS0] = {
.pipe_control_notify = INTEL_GVT_EVENT_RESERVED,
.mi_flush_dw = BCS_MI_FLUSH_DW,
.mi_user_interrupt = BCS_MI_USER_INTERRUPT,
},
- [VCS] = {
+ [VCS0] = {
.pipe_control_notify = INTEL_GVT_EVENT_RESERVED,
.mi_flush_dw = VCS_MI_FLUSH_DW,
.mi_user_interrupt = VCS_MI_USER_INTERRUPT,
},
- [VCS2] = {
+ [VCS1] = {
.pipe_control_notify = INTEL_GVT_EVENT_RESERVED,
.mi_flush_dw = VCS2_MI_FLUSH_DW,
.mi_user_interrupt = VCS2_MI_USER_INTERRUPT,
},
- [VECS] = {
+ [VECS0] = {
.pipe_control_notify = INTEL_GVT_EVENT_RESERVED,
.mi_flush_dw = VECS_MI_FLUSH_DW,
.mi_user_interrupt = VECS_MI_USER_INTERRUPT,
bool index_mode = false;
unsigned int post_sync;
int ret = 0;
+ u32 hws_pga, val;
post_sync = (cmd_val(s, 1) & PIPE_CONTROL_POST_SYNC_OP_MASK) >> 14;
index_mode = true;
ret |= cmd_address_audit(s, gma, sizeof(u64),
index_mode);
+ if (ret)
+ return ret;
+ if (index_mode) {
+ hws_pga = s->vgpu->hws_pga[s->ring_id];
+ gma = hws_pga + gma;
+ patch_value(s, cmd_ptr(s, 2), gma);
+ val = cmd_val(s, 1) & (~(1 << 21));
+ patch_value(s, cmd_ptr(s, 1), val);
+ }
}
}
}
info->tile_val << 10);
}
- vgpu_vreg_t(vgpu, PIPE_FRMCOUNT_G4X(info->pipe))++;
- intel_vgpu_trigger_virtual_event(vgpu, info->event);
+ if (info->plane == PLANE_PRIMARY)
+ vgpu_vreg_t(vgpu, PIPE_FLIPCOUNT_G4X(info->pipe))++;
+
+ if (info->async_flip)
+ intel_vgpu_trigger_virtual_event(vgpu, info->event);
+ else
+ set_bit(info->event, vgpu->irq.flip_done_event[info->pipe]);
+
return 0;
}
unsigned long gma;
bool index_mode = false;
int ret = 0;
+ u32 hws_pga, val;
/* Check post-sync and ppgtt bit */
if (((cmd_val(s, 0) >> 14) & 0x3) && (cmd_val(s, 1) & (1 << 2))) {
if (cmd_val(s, 0) & (1 << 21))
index_mode = true;
ret = cmd_address_audit(s, gma, sizeof(u64), index_mode);
+ if (ret)
+ return ret;
+ if (index_mode) {
+ hws_pga = s->vgpu->hws_pga[s->ring_id];
+ gma = hws_pga + gma;
+ patch_value(s, cmd_ptr(s, 1), gma);
+ val = cmd_val(s, 0) & (~(1 << 21));
+ patch_value(s, cmd_ptr(s, 0), val);
+ }
}
/* Check notify bit */
if ((cmd_val(s, 0) & (1 << 8)))
if (!pipe_is_enabled(vgpu, pipe))
continue;
- vgpu_vreg_t(vgpu, PIPE_FLIPCOUNT_G4X(pipe))++;
intel_vgpu_trigger_virtual_event(vgpu, event);
}
struct drm_i915_private *dev_priv = to_i915(dev);
struct drm_i915_gem_object *obj;
- obj = i915_gem_object_alloc(dev_priv);
+ obj = i915_gem_object_alloc();
if (obj == NULL)
return NULL;
((a)->lrca == (b)->lrca))
static int context_switch_events[] = {
- [RCS] = RCS_AS_CONTEXT_SWITCH,
- [BCS] = BCS_AS_CONTEXT_SWITCH,
- [VCS] = VCS_AS_CONTEXT_SWITCH,
- [VCS2] = VCS2_AS_CONTEXT_SWITCH,
- [VECS] = VECS_AS_CONTEXT_SWITCH,
+ [RCS0] = RCS_AS_CONTEXT_SWITCH,
+ [BCS0] = BCS_AS_CONTEXT_SWITCH,
+ [VCS0] = VCS_AS_CONTEXT_SWITCH,
+ [VCS1] = VCS2_AS_CONTEXT_SWITCH,
+ [VECS0] = VECS_AS_CONTEXT_SWITCH,
};
-static int ring_id_to_context_switch_event(int ring_id)
+static int ring_id_to_context_switch_event(unsigned int ring_id)
{
- if (WARN_ON(ring_id < RCS ||
- ring_id >= ARRAY_SIZE(context_switch_events)))
+ if (WARN_ON(ring_id >= ARRAY_SIZE(context_switch_events)))
return -EINVAL;
return context_switch_events[ring_id];
gvt_dbg_el("complete workload %p status %d\n", workload,
workload->status);
- if (workload->status || (vgpu->resetting_eng & ENGINE_MASK(ring_id)))
+ if (workload->status || (vgpu->resetting_eng & BIT(ring_id)))
goto out;
if (!list_empty(workload_q_head(vgpu, ring_id))) {
vgpu_vreg(vgpu, ctx_status_ptr_reg) = ctx_status_ptr.dw;
}
-static void clean_execlist(struct intel_vgpu *vgpu, unsigned long engine_mask)
+static void clean_execlist(struct intel_vgpu *vgpu,
+ intel_engine_mask_t engine_mask)
{
- unsigned int tmp;
struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
struct intel_engine_cs *engine;
struct intel_vgpu_submission *s = &vgpu->submission;
+ intel_engine_mask_t tmp;
for_each_engine_masked(engine, dev_priv, engine_mask, tmp) {
kfree(s->ring_scan_buffer[engine->id]);
}
static void reset_execlist(struct intel_vgpu *vgpu,
- unsigned long engine_mask)
+ intel_engine_mask_t engine_mask)
{
struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
struct intel_engine_cs *engine;
- unsigned int tmp;
+ intel_engine_mask_t tmp;
for_each_engine_masked(engine, dev_priv, engine_mask, tmp)
init_vgpu_execlist(vgpu, engine->id);
}
static int init_execlist(struct intel_vgpu *vgpu,
- unsigned long engine_mask)
+ intel_engine_mask_t engine_mask)
{
reset_execlist(vgpu, engine_mask);
return 0;
int intel_vgpu_submit_execlist(struct intel_vgpu *vgpu, int ring_id);
void intel_vgpu_reset_execlist(struct intel_vgpu *vgpu,
- unsigned long engine_mask);
+ intel_engine_mask_t engine_mask);
#endif /*_GVT_EXECLIST_H_*/
list_for_each_safe(pos, n, >t->oos_page_free_list_head) {
oos_page = container_of(pos, struct intel_vgpu_oos_page, list);
list_del(&oos_page->list);
+ free_page((unsigned long)oos_page->mem);
kfree(oos_page);
}
}
ret = -ENOMEM;
goto fail;
}
+ oos_page->mem = (void *)__get_free_pages(GFP_KERNEL, 0);
+ if (!oos_page->mem) {
+ ret = -ENOMEM;
+ kfree(oos_page);
+ goto fail;
+ }
INIT_LIST_HEAD(&oos_page->list);
INIT_LIST_HEAD(&oos_page->vm_list);
struct list_head list;
struct list_head vm_list;
int id;
- unsigned char mem[I915_GTT_PAGE_SIZE];
+ void *mem;
};
#define GTT_ENTRY_NUM_IN_ONE_PAGE 512
struct intel_vgpu_mmio {
void *vreg;
- void *sreg;
};
#define INTEL_GVT_MAX_BAR_NUM 4
#define vgpu_cfg_space(vgpu) ((vgpu)->cfg_space.virtual_cfg_space)
-#define INTEL_GVT_MAX_PIPE 4
-
struct intel_vgpu_irq {
bool irq_warn_once[INTEL_GVT_EVENT_MAX];
- DECLARE_BITMAP(flip_done_event[INTEL_GVT_MAX_PIPE],
+ DECLARE_BITMAP(flip_done_event[I915_MAX_PIPES],
INTEL_GVT_EVENT_MAX);
};
struct intel_vgpu_submission_ops {
const char *name;
- int (*init)(struct intel_vgpu *vgpu, unsigned long engine_mask);
- void (*clean)(struct intel_vgpu *vgpu, unsigned long engine_mask);
- void (*reset)(struct intel_vgpu *vgpu, unsigned long engine_mask);
+ int (*init)(struct intel_vgpu *vgpu, intel_engine_mask_t engine_mask);
+ void (*clean)(struct intel_vgpu *vgpu, intel_engine_mask_t engine_mask);
+ void (*reset)(struct intel_vgpu *vgpu, intel_engine_mask_t engine_mask);
};
struct intel_vgpu_submission {
(*(u64 *)(vgpu->mmio.vreg + i915_mmio_reg_offset(reg)))
#define vgpu_vreg64(vgpu, offset) \
(*(u64 *)(vgpu->mmio.vreg + (offset)))
-#define vgpu_sreg_t(vgpu, reg) \
- (*(u32 *)(vgpu->mmio.sreg + i915_mmio_reg_offset(reg)))
-#define vgpu_sreg(vgpu, offset) \
- (*(u32 *)(vgpu->mmio.sreg + (offset)))
#define for_each_active_vgpu(gvt, vgpu, id) \
idr_for_each_entry((&(gvt)->vgpu_idr), (vgpu), (id)) \
void intel_gvt_destroy_vgpu(struct intel_vgpu *vgpu);
void intel_gvt_release_vgpu(struct intel_vgpu *vgpu);
void intel_gvt_reset_vgpu_locked(struct intel_vgpu *vgpu, bool dmlr,
- unsigned int engine_mask);
+ intel_engine_mask_t engine_mask);
void intel_gvt_reset_vgpu(struct intel_vgpu *vgpu);
void intel_gvt_activate_vgpu(struct intel_vgpu *vgpu);
void intel_gvt_deactivate_vgpu(struct intel_vgpu *vgpu);
static int gdrst_mmio_write(struct intel_vgpu *vgpu, unsigned int offset,
void *p_data, unsigned int bytes)
{
- unsigned int engine_mask = 0;
+ intel_engine_mask_t engine_mask = 0;
u32 data;
write_vreg(vgpu, offset, p_data, bytes);
} else {
if (data & GEN6_GRDOM_RENDER) {
gvt_dbg_mmio("vgpu%d: request RCS reset\n", vgpu->id);
- engine_mask |= (1 << RCS);
+ engine_mask |= BIT(RCS0);
}
if (data & GEN6_GRDOM_MEDIA) {
gvt_dbg_mmio("vgpu%d: request VCS reset\n", vgpu->id);
- engine_mask |= (1 << VCS);
+ engine_mask |= BIT(VCS0);
}
if (data & GEN6_GRDOM_BLT) {
gvt_dbg_mmio("vgpu%d: request BCS Reset\n", vgpu->id);
- engine_mask |= (1 << BCS);
+ engine_mask |= BIT(BCS0);
}
if (data & GEN6_GRDOM_VECS) {
gvt_dbg_mmio("vgpu%d: request VECS Reset\n", vgpu->id);
- engine_mask |= (1 << VECS);
+ engine_mask |= BIT(VECS0);
}
if (data & GEN8_GRDOM_MEDIA2) {
gvt_dbg_mmio("vgpu%d: request VCS2 Reset\n", vgpu->id);
- if (HAS_BSD2(vgpu->gvt->dev_priv))
- engine_mask |= (1 << VCS2);
+ engine_mask |= BIT(VCS1);
}
+ engine_mask &= INTEL_INFO(vgpu->gvt->dev_priv)->engine_mask;
}
/* vgpu_lock already hold by emulate mmio r/w */
void *p_data, unsigned int bytes)
{
struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
- unsigned int index = DSPSURF_TO_PIPE(offset);
- i915_reg_t surflive_reg = DSPSURFLIVE(index);
- int flip_event[] = {
- [PIPE_A] = PRIMARY_A_FLIP_DONE,
- [PIPE_B] = PRIMARY_B_FLIP_DONE,
- [PIPE_C] = PRIMARY_C_FLIP_DONE,
- };
+ u32 pipe = DSPSURF_TO_PIPE(offset);
+ int event = SKL_FLIP_EVENT(pipe, PLANE_PRIMARY);
write_vreg(vgpu, offset, p_data, bytes);
- vgpu_vreg_t(vgpu, surflive_reg) = vgpu_vreg(vgpu, offset);
+ vgpu_vreg_t(vgpu, DSPSURFLIVE(pipe)) = vgpu_vreg(vgpu, offset);
+
+ vgpu_vreg_t(vgpu, PIPE_FLIPCOUNT_G4X(pipe))++;
+
+ if (vgpu_vreg_t(vgpu, DSPCNTR(pipe)) & PLANE_CTL_ASYNC_FLIP)
+ intel_vgpu_trigger_virtual_event(vgpu, event);
+ else
+ set_bit(event, vgpu->irq.flip_done_event[pipe]);
- set_bit(flip_event[index], vgpu->irq.flip_done_event[index]);
return 0;
}
static int spr_surf_mmio_write(struct intel_vgpu *vgpu, unsigned int offset,
void *p_data, unsigned int bytes)
{
- unsigned int index = SPRSURF_TO_PIPE(offset);
- i915_reg_t surflive_reg = SPRSURFLIVE(index);
- int flip_event[] = {
- [PIPE_A] = SPRITE_A_FLIP_DONE,
- [PIPE_B] = SPRITE_B_FLIP_DONE,
- [PIPE_C] = SPRITE_C_FLIP_DONE,
- };
+ u32 pipe = SPRSURF_TO_PIPE(offset);
+ int event = SKL_FLIP_EVENT(pipe, PLANE_SPRITE0);
+
+ write_vreg(vgpu, offset, p_data, bytes);
+ vgpu_vreg_t(vgpu, SPRSURFLIVE(pipe)) = vgpu_vreg(vgpu, offset);
+
+ if (vgpu_vreg_t(vgpu, SPRCTL(pipe)) & PLANE_CTL_ASYNC_FLIP)
+ intel_vgpu_trigger_virtual_event(vgpu, event);
+ else
+ set_bit(event, vgpu->irq.flip_done_event[pipe]);
+
+ return 0;
+}
+
+static int reg50080_mmio_write(struct intel_vgpu *vgpu,
+ unsigned int offset, void *p_data,
+ unsigned int bytes)
+{
+ struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
+ enum pipe pipe = REG_50080_TO_PIPE(offset);
+ enum plane_id plane = REG_50080_TO_PLANE(offset);
+ int event = SKL_FLIP_EVENT(pipe, plane);
write_vreg(vgpu, offset, p_data, bytes);
- vgpu_vreg_t(vgpu, surflive_reg) = vgpu_vreg(vgpu, offset);
+ if (plane == PLANE_PRIMARY) {
+ vgpu_vreg_t(vgpu, DSPSURFLIVE(pipe)) = vgpu_vreg(vgpu, offset);
+ vgpu_vreg_t(vgpu, PIPE_FLIPCOUNT_G4X(pipe))++;
+ } else {
+ vgpu_vreg_t(vgpu, SPRSURFLIVE(pipe)) = vgpu_vreg(vgpu, offset);
+ }
+
+ if ((vgpu_vreg(vgpu, offset) & REG50080_FLIP_TYPE_MASK) == REG50080_FLIP_TYPE_ASYNC)
+ intel_vgpu_trigger_virtual_event(vgpu, event);
+ else
+ set_bit(event, vgpu->irq.flip_done_event[pipe]);
- set_bit(flip_event[index], vgpu->irq.flip_done_event[index]);
return 0;
}
return 0;
ret = intel_vgpu_select_submission_ops(vgpu,
- ENGINE_MASK(ring_id),
+ BIT(ring_id),
INTEL_VGPU_EXECLIST_SUBMISSION);
if (ret)
return ret;
switch (offset) {
case 0x4260:
- id = RCS;
+ id = RCS0;
break;
case 0x4264:
- id = VCS;
+ id = VCS0;
break;
case 0x4268:
- id = VCS2;
+ id = VCS1;
break;
case 0x426c:
- id = BCS;
+ id = BCS0;
break;
case 0x4270:
- id = VECS;
+ id = VECS0;
break;
default:
return -EINVAL;
MMIO_F(prefix(BLT_RING_BASE), s, f, am, rm, d, r, w); \
MMIO_F(prefix(GEN6_BSD_RING_BASE), s, f, am, rm, d, r, w); \
MMIO_F(prefix(VEBOX_RING_BASE), s, f, am, rm, d, r, w); \
- if (HAS_BSD2(dev_priv)) \
+ if (HAS_ENGINE(dev_priv, VCS1)) \
MMIO_F(prefix(GEN8_BSD2_RING_BASE), s, f, am, rm, d, r, w); \
} while (0)
MMIO_DH(GEN7_SC_INSTDONE, D_BDW_PLUS, mmio_read_from_hw, NULL);
MMIO_GM_RDR(_MMIO(0x2148), D_ALL, NULL, NULL);
- MMIO_GM_RDR(CCID, D_ALL, NULL, NULL);
+ MMIO_GM_RDR(CCID(RENDER_RING_BASE), D_ALL, NULL, NULL);
MMIO_GM_RDR(_MMIO(0x12198), D_ALL, NULL, NULL);
MMIO_D(GEN7_CXT_SIZE, D_ALL);
MMIO_DH(DSPSURF(PIPE_A), D_ALL, NULL, pri_surf_mmio_write);
MMIO_D(DSPOFFSET(PIPE_A), D_ALL);
MMIO_D(DSPSURFLIVE(PIPE_A), D_ALL);
+ MMIO_DH(REG_50080(PIPE_A, PLANE_PRIMARY), D_ALL, NULL,
+ reg50080_mmio_write);
MMIO_D(DSPCNTR(PIPE_B), D_ALL);
MMIO_D(DSPADDR(PIPE_B), D_ALL);
MMIO_DH(DSPSURF(PIPE_B), D_ALL, NULL, pri_surf_mmio_write);
MMIO_D(DSPOFFSET(PIPE_B), D_ALL);
MMIO_D(DSPSURFLIVE(PIPE_B), D_ALL);
+ MMIO_DH(REG_50080(PIPE_B, PLANE_PRIMARY), D_ALL, NULL,
+ reg50080_mmio_write);
MMIO_D(DSPCNTR(PIPE_C), D_ALL);
MMIO_D(DSPADDR(PIPE_C), D_ALL);
MMIO_DH(DSPSURF(PIPE_C), D_ALL, NULL, pri_surf_mmio_write);
MMIO_D(DSPOFFSET(PIPE_C), D_ALL);
MMIO_D(DSPSURFLIVE(PIPE_C), D_ALL);
+ MMIO_DH(REG_50080(PIPE_C, PLANE_PRIMARY), D_ALL, NULL,
+ reg50080_mmio_write);
MMIO_D(SPRCTL(PIPE_A), D_ALL);
MMIO_D(SPRLINOFF(PIPE_A), D_ALL);
MMIO_D(SPROFFSET(PIPE_A), D_ALL);
MMIO_D(SPRSCALE(PIPE_A), D_ALL);
MMIO_D(SPRSURFLIVE(PIPE_A), D_ALL);
+ MMIO_DH(REG_50080(PIPE_A, PLANE_SPRITE0), D_ALL, NULL,
+ reg50080_mmio_write);
MMIO_D(SPRCTL(PIPE_B), D_ALL);
MMIO_D(SPRLINOFF(PIPE_B), D_ALL);
MMIO_D(SPROFFSET(PIPE_B), D_ALL);
MMIO_D(SPRSCALE(PIPE_B), D_ALL);
MMIO_D(SPRSURFLIVE(PIPE_B), D_ALL);
+ MMIO_DH(REG_50080(PIPE_B, PLANE_SPRITE0), D_ALL, NULL,
+ reg50080_mmio_write);
MMIO_D(SPRCTL(PIPE_C), D_ALL);
MMIO_D(SPRLINOFF(PIPE_C), D_ALL);
MMIO_D(SPROFFSET(PIPE_C), D_ALL);
MMIO_D(SPRSCALE(PIPE_C), D_ALL);
MMIO_D(SPRSURFLIVE(PIPE_C), D_ALL);
+ MMIO_DH(REG_50080(PIPE_C, PLANE_SPRITE0), D_ALL, NULL,
+ reg50080_mmio_write);
MMIO_D(HTOTAL(TRANSCODER_A), D_ALL);
MMIO_D(HBLANK(TRANSCODER_A), D_ALL);
MMIO_DH(DBUF_CTL, D_SKL_PLUS, NULL, gen9_dbuf_ctl_mmio_write);
- MMIO_D(_MMIO(0xa210), D_SKL_PLUS);
+ MMIO_D(GEN9_PG_ENABLE, D_SKL_PLUS);
MMIO_D(GEN9_MEDIA_PG_IDLE_HYSTERESIS, D_SKL_PLUS);
MMIO_D(GEN9_RENDER_PG_IDLE_HYSTERESIS, D_SKL_PLUS);
MMIO_DFH(GEN9_GAMT_ECO_REG_RW_IA, D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
- MMIO_DH(_MMIO(0x4ddc), D_SKL_PLUS, NULL, NULL);
- MMIO_DH(_MMIO(0x42080), D_SKL_PLUS, NULL, NULL);
- MMIO_D(_MMIO(0x45504), D_SKL_PLUS);
- MMIO_D(_MMIO(0x45520), D_SKL_PLUS);
- MMIO_D(_MMIO(0x46000), D_SKL_PLUS);
- MMIO_DH(_MMIO(0x46010), D_SKL_PLUS, NULL, skl_lcpll_write);
- MMIO_DH(_MMIO(0x46014), D_SKL_PLUS, NULL, skl_lcpll_write);
- MMIO_D(_MMIO(0x6C040), D_SKL_PLUS);
- MMIO_D(_MMIO(0x6C048), D_SKL_PLUS);
- MMIO_D(_MMIO(0x6C050), D_SKL_PLUS);
- MMIO_D(_MMIO(0x6C044), D_SKL_PLUS);
- MMIO_D(_MMIO(0x6C04C), D_SKL_PLUS);
- MMIO_D(_MMIO(0x6C054), D_SKL_PLUS);
- MMIO_D(_MMIO(0x6c058), D_SKL_PLUS);
- MMIO_D(_MMIO(0x6c05c), D_SKL_PLUS);
- MMIO_DH(_MMIO(0x6c060), D_SKL_PLUS, dpll_status_read, NULL);
+ MMIO_DH(MMCD_MISC_CTRL, D_SKL_PLUS, NULL, NULL);
+ MMIO_DH(CHICKEN_PAR1_1, D_SKL_PLUS, NULL, NULL);
+ MMIO_D(DC_STATE_EN, D_SKL_PLUS);
+ MMIO_D(DC_STATE_DEBUG, D_SKL_PLUS);
+ MMIO_D(CDCLK_CTL, D_SKL_PLUS);
+ MMIO_DH(LCPLL1_CTL, D_SKL_PLUS, NULL, skl_lcpll_write);
+ MMIO_DH(LCPLL2_CTL, D_SKL_PLUS, NULL, skl_lcpll_write);
+ MMIO_D(_MMIO(_DPLL1_CFGCR1), D_SKL_PLUS);
+ MMIO_D(_MMIO(_DPLL2_CFGCR1), D_SKL_PLUS);
+ MMIO_D(_MMIO(_DPLL3_CFGCR1), D_SKL_PLUS);
+ MMIO_D(_MMIO(_DPLL1_CFGCR2), D_SKL_PLUS);
+ MMIO_D(_MMIO(_DPLL2_CFGCR2), D_SKL_PLUS);
+ MMIO_D(_MMIO(_DPLL3_CFGCR2), D_SKL_PLUS);
+ MMIO_D(DPLL_CTRL1, D_SKL_PLUS);
+ MMIO_D(DPLL_CTRL2, D_SKL_PLUS);
+ MMIO_DH(DPLL_STATUS, D_SKL_PLUS, dpll_status_read, NULL);
MMIO_DH(SKL_PS_WIN_POS(PIPE_A, 0), D_SKL_PLUS, NULL, pf_write);
MMIO_DH(SKL_PS_WIN_POS(PIPE_A, 1), D_SKL_PLUS, NULL, pf_write);
MMIO_DH(_MMIO(_REG_701C4(PIPE_C, 3)), D_SKL_PLUS, NULL, NULL);
MMIO_DH(_MMIO(_REG_701C4(PIPE_C, 4)), D_SKL_PLUS, NULL, NULL);
- MMIO_D(_MMIO(0x70380), D_SKL_PLUS);
- MMIO_D(_MMIO(0x71380), D_SKL_PLUS);
+ MMIO_D(_MMIO(_PLANE_CTL_3_A), D_SKL_PLUS);
+ MMIO_D(_MMIO(_PLANE_CTL_3_B), D_SKL_PLUS);
MMIO_D(_MMIO(0x72380), D_SKL_PLUS);
MMIO_D(_MMIO(0x7239c), D_SKL_PLUS);
- MMIO_D(_MMIO(0x7039c), D_SKL_PLUS);
+ MMIO_D(_MMIO(_PLANE_SURF_3_A), D_SKL_PLUS);
- MMIO_D(_MMIO(0x8f074), D_SKL_PLUS);
- MMIO_D(_MMIO(0x8f004), D_SKL_PLUS);
- MMIO_D(_MMIO(0x8f034), D_SKL_PLUS);
+ MMIO_D(CSR_SSP_BASE, D_SKL_PLUS);
+ MMIO_D(CSR_HTP_SKL, D_SKL_PLUS);
+ MMIO_D(CSR_LAST_WRITE, D_SKL_PLUS);
- MMIO_D(_MMIO(0xb11c), D_SKL_PLUS);
+ MMIO_D(BDW_SCRATCH1, D_SKL_PLUS);
- MMIO_D(_MMIO(0x51000), D_SKL_PLUS);
- MMIO_D(_MMIO(0x6c00c), D_SKL_PLUS);
+ MMIO_D(SKL_DFSM, D_SKL_PLUS);
+ MMIO_D(DISPIO_CR_TX_BMU_CR0, D_SKL_PLUS);
- MMIO_F(_MMIO(0xc800), 0x7f8, F_CMD_ACCESS, 0, 0, D_SKL_PLUS,
+ MMIO_F(GEN9_GFX_MOCS(0), 0x7f8, F_CMD_ACCESS, 0, 0, D_SKL_PLUS,
NULL, NULL);
- MMIO_F(_MMIO(0xb020), 0x80, F_CMD_ACCESS, 0, 0, D_SKL_PLUS,
+ MMIO_F(GEN7_L3CNTLREG2, 0x80, F_CMD_ACCESS, 0, 0, D_SKL_PLUS,
NULL, NULL);
MMIO_D(RPM_CONFIG0, D_SKL_PLUS);
MMIO_D(_MMIO(0xd08), D_SKL_PLUS);
MMIO_D(RC6_LOCATION, D_SKL_PLUS);
- MMIO_DFH(_MMIO(0x20e0), D_SKL_PLUS, F_MODE_MASK, NULL, NULL);
- MMIO_DFH(_MMIO(0x20ec), D_SKL_PLUS, F_MODE_MASK | F_CMD_ACCESS,
+ MMIO_DFH(GEN7_FF_SLICE_CS_CHICKEN1, D_SKL_PLUS, F_MODE_MASK,
+ NULL, NULL);
+ MMIO_DFH(GEN9_CS_DEBUG_MODE1, D_SKL_PLUS, F_MODE_MASK | F_CMD_ACCESS,
NULL, NULL);
/* TRTT */
- MMIO_DFH(_MMIO(0x4de0), D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
- MMIO_DFH(_MMIO(0x4de4), D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
- MMIO_DFH(_MMIO(0x4de8), D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
- MMIO_DFH(_MMIO(0x4dec), D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
- MMIO_DFH(_MMIO(0x4df0), D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
- MMIO_DFH(_MMIO(0x4df4), D_SKL_PLUS, F_CMD_ACCESS,
+ MMIO_DFH(TRVATTL3PTRDW(0), D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
+ MMIO_DFH(TRVATTL3PTRDW(1), D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
+ MMIO_DFH(TRVATTL3PTRDW(2), D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
+ MMIO_DFH(TRVATTL3PTRDW(3), D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
+ MMIO_DFH(TRVADR, D_SKL_PLUS, F_CMD_ACCESS, NULL, NULL);
+ MMIO_DFH(TRTTE, D_SKL_PLUS, F_CMD_ACCESS,
NULL, gen9_trtte_write);
MMIO_DH(_MMIO(0x4dfc), D_SKL_PLUS, NULL, gen9_trtt_chicken_write);
MMIO_DH(DMA_CTRL, D_SKL_PLUS, NULL, dma_ctrl_write);
MMIO_D(_MMIO(0x65900), D_SKL_PLUS);
- MMIO_D(_MMIO(0x1082c0), D_SKL_PLUS);
+ MMIO_D(GEN6_STOLEN_RESERVED, D_SKL_PLUS);
MMIO_D(_MMIO(0x4068), D_SKL_PLUS);
MMIO_D(_MMIO(0x67054), D_SKL_PLUS);
MMIO_D(_MMIO(0x6e560), D_SKL_PLUS);
MMIO_DFH(GEN9_WM_CHICKEN3, D_SKL_PLUS, F_MODE_MASK | F_CMD_ACCESS,
NULL, NULL);
- MMIO_D(_MMIO(0x4ab8), D_KBL | D_CFL);
- MMIO_D(_MMIO(0x2248), D_SKL_PLUS);
+ MMIO_D(GAMT_CHKN_BIT_REG, D_KBL);
+ MMIO_D(GEN9_CTX_PREEMPT_REG, D_KBL | D_SKL);
return 0;
}
/* Special MMIO blocks. */
static struct gvt_mmio_block mmio_blocks[] = {
{D_SKL_PLUS, _MMIO(CSR_MMIO_START_RANGE), 0x3000, NULL, NULL},
- {D_ALL, _MMIO(MCHBAR_MIRROR_BASE_SNB), 0x40000, NULL, NULL},
+ {D_ALL, MCHBAR_MIRROR_REG_BASE, 0x4000, NULL, NULL},
{D_ALL, _MMIO(VGT_PVINFO_PAGE), VGT_PVINFO_SIZE,
pvinfo_mmio_read, pvinfo_mmio_write},
{D_ALL, LGC_PALETTE(PIPE_A, 0), 1024, NULL, NULL},
return mmio_info->read(vgpu, offset, pdata, bytes);
else {
u64 ro_mask = mmio_info->ro_mask;
- u32 old_vreg = 0, old_sreg = 0;
+ u32 old_vreg = 0;
u64 data = 0;
if (intel_gvt_mmio_has_mode_mask(gvt, mmio_info->offset)) {
old_vreg = vgpu_vreg(vgpu, offset);
- old_sreg = vgpu_sreg(vgpu, offset);
}
if (likely(!ro_mask))
vgpu_vreg(vgpu, offset) = (old_vreg & ~mask)
| (vgpu_vreg(vgpu, offset) & mask);
- vgpu_sreg(vgpu, offset) = (old_sreg & ~mask)
- | (vgpu_sreg(vgpu, offset) & mask);
}
}
SET_BIT_INFO(irq, 4, VCS_MI_FLUSH_DW, INTEL_GVT_IRQ_INFO_GT1);
SET_BIT_INFO(irq, 8, VCS_AS_CONTEXT_SWITCH, INTEL_GVT_IRQ_INFO_GT1);
- if (HAS_BSD2(gvt->dev_priv)) {
+ if (HAS_ENGINE(gvt->dev_priv, VCS1)) {
SET_BIT_INFO(irq, 16, VCS2_MI_USER_INTERRUPT,
INTEL_GVT_IRQ_INFO_GT1);
SET_BIT_INFO(irq, 20, VCS2_MI_FLUSH_DW,
if (dmlr) {
memcpy(vgpu->mmio.vreg, mmio, info->mmio_size);
- memcpy(vgpu->mmio.sreg, mmio, info->mmio_size);
vgpu_vreg_t(vgpu, GEN6_GT_THREAD_STATUS_REG) = 0;
* touched
*/
memcpy(vgpu->mmio.vreg, mmio, GVT_GEN8_MMIO_RESET_OFFSET);
- memcpy(vgpu->mmio.sreg, mmio, GVT_GEN8_MMIO_RESET_OFFSET);
}
}
{
const struct intel_gvt_device_info *info = &vgpu->gvt->device_info;
- vgpu->mmio.vreg = vzalloc(array_size(info->mmio_size, 2));
+ vgpu->mmio.vreg = vzalloc(info->mmio_size);
if (!vgpu->mmio.vreg)
return -ENOMEM;
- vgpu->mmio.sreg = vgpu->mmio.vreg + info->mmio_size;
-
intel_vgpu_reset_mmio(vgpu, true);
return 0;
void intel_vgpu_clean_mmio(struct intel_vgpu *vgpu)
{
vfree(vgpu->mmio.vreg);
- vgpu->mmio.vreg = vgpu->mmio.sreg = NULL;
+ vgpu->mmio.vreg = NULL;
}
/* Raw offset is appened to each line for convenience. */
static struct engine_mmio gen8_engine_mmio_list[] __cacheline_aligned = {
- {RCS, GFX_MODE_GEN7, 0xffff, false}, /* 0x229c */
- {RCS, GEN9_CTX_PREEMPT_REG, 0x0, false}, /* 0x2248 */
- {RCS, HWSTAM, 0x0, false}, /* 0x2098 */
- {RCS, INSTPM, 0xffff, true}, /* 0x20c0 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 0), 0, false}, /* 0x24d0 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 1), 0, false}, /* 0x24d4 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 2), 0, false}, /* 0x24d8 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 3), 0, false}, /* 0x24dc */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 4), 0, false}, /* 0x24e0 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 5), 0, false}, /* 0x24e4 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 6), 0, false}, /* 0x24e8 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 7), 0, false}, /* 0x24ec */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 8), 0, false}, /* 0x24f0 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 9), 0, false}, /* 0x24f4 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 10), 0, false}, /* 0x24f8 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 11), 0, false}, /* 0x24fc */
- {RCS, CACHE_MODE_1, 0xffff, true}, /* 0x7004 */
- {RCS, GEN7_GT_MODE, 0xffff, true}, /* 0x7008 */
- {RCS, CACHE_MODE_0_GEN7, 0xffff, true}, /* 0x7000 */
- {RCS, GEN7_COMMON_SLICE_CHICKEN1, 0xffff, true}, /* 0x7010 */
- {RCS, HDC_CHICKEN0, 0xffff, true}, /* 0x7300 */
- {RCS, VF_GUARDBAND, 0xffff, true}, /* 0x83a4 */
-
- {BCS, RING_GFX_MODE(BLT_RING_BASE), 0xffff, false}, /* 0x2229c */
- {BCS, RING_MI_MODE(BLT_RING_BASE), 0xffff, false}, /* 0x2209c */
- {BCS, RING_INSTPM(BLT_RING_BASE), 0xffff, false}, /* 0x220c0 */
- {BCS, RING_HWSTAM(BLT_RING_BASE), 0x0, false}, /* 0x22098 */
- {BCS, RING_EXCC(BLT_RING_BASE), 0x0, false}, /* 0x22028 */
- {RCS, INVALID_MMIO_REG, 0, false } /* Terminated */
+ {RCS0, GFX_MODE_GEN7, 0xffff, false}, /* 0x229c */
+ {RCS0, GEN9_CTX_PREEMPT_REG, 0x0, false}, /* 0x2248 */
+ {RCS0, HWSTAM, 0x0, false}, /* 0x2098 */
+ {RCS0, INSTPM, 0xffff, true}, /* 0x20c0 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 0), 0, false}, /* 0x24d0 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 1), 0, false}, /* 0x24d4 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 2), 0, false}, /* 0x24d8 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 3), 0, false}, /* 0x24dc */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 4), 0, false}, /* 0x24e0 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 5), 0, false}, /* 0x24e4 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 6), 0, false}, /* 0x24e8 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 7), 0, false}, /* 0x24ec */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 8), 0, false}, /* 0x24f0 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 9), 0, false}, /* 0x24f4 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 10), 0, false}, /* 0x24f8 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 11), 0, false}, /* 0x24fc */
+ {RCS0, CACHE_MODE_1, 0xffff, true}, /* 0x7004 */
+ {RCS0, GEN7_GT_MODE, 0xffff, true}, /* 0x7008 */
+ {RCS0, CACHE_MODE_0_GEN7, 0xffff, true}, /* 0x7000 */
+ {RCS0, GEN7_COMMON_SLICE_CHICKEN1, 0xffff, true}, /* 0x7010 */
+ {RCS0, HDC_CHICKEN0, 0xffff, true}, /* 0x7300 */
+ {RCS0, VF_GUARDBAND, 0xffff, true}, /* 0x83a4 */
+
+ {BCS0, RING_GFX_MODE(BLT_RING_BASE), 0xffff, false}, /* 0x2229c */
+ {BCS0, RING_MI_MODE(BLT_RING_BASE), 0xffff, false}, /* 0x2209c */
+ {BCS0, RING_INSTPM(BLT_RING_BASE), 0xffff, false}, /* 0x220c0 */
+ {BCS0, RING_HWSTAM(BLT_RING_BASE), 0x0, false}, /* 0x22098 */
+ {BCS0, RING_EXCC(BLT_RING_BASE), 0xffff, false}, /* 0x22028 */
+ {RCS0, INVALID_MMIO_REG, 0, false } /* Terminated */
};
static struct engine_mmio gen9_engine_mmio_list[] __cacheline_aligned = {
- {RCS, GFX_MODE_GEN7, 0xffff, false}, /* 0x229c */
- {RCS, GEN9_CTX_PREEMPT_REG, 0x0, false}, /* 0x2248 */
- {RCS, HWSTAM, 0x0, false}, /* 0x2098 */
- {RCS, INSTPM, 0xffff, true}, /* 0x20c0 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 0), 0, false}, /* 0x24d0 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 1), 0, false}, /* 0x24d4 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 2), 0, false}, /* 0x24d8 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 3), 0, false}, /* 0x24dc */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 4), 0, false}, /* 0x24e0 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 5), 0, false}, /* 0x24e4 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 6), 0, false}, /* 0x24e8 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 7), 0, false}, /* 0x24ec */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 8), 0, false}, /* 0x24f0 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 9), 0, false}, /* 0x24f4 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 10), 0, false}, /* 0x24f8 */
- {RCS, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 11), 0, false}, /* 0x24fc */
- {RCS, CACHE_MODE_1, 0xffff, true}, /* 0x7004 */
- {RCS, GEN7_GT_MODE, 0xffff, true}, /* 0x7008 */
- {RCS, CACHE_MODE_0_GEN7, 0xffff, true}, /* 0x7000 */
- {RCS, GEN7_COMMON_SLICE_CHICKEN1, 0xffff, true}, /* 0x7010 */
- {RCS, HDC_CHICKEN0, 0xffff, true}, /* 0x7300 */
- {RCS, VF_GUARDBAND, 0xffff, true}, /* 0x83a4 */
-
- {RCS, GEN8_PRIVATE_PAT_LO, 0, false}, /* 0x40e0 */
- {RCS, GEN8_PRIVATE_PAT_HI, 0, false}, /* 0x40e4 */
- {RCS, GEN8_CS_CHICKEN1, 0xffff, true}, /* 0x2580 */
- {RCS, COMMON_SLICE_CHICKEN2, 0xffff, true}, /* 0x7014 */
- {RCS, GEN9_CS_DEBUG_MODE1, 0xffff, false}, /* 0x20ec */
- {RCS, GEN8_L3SQCREG4, 0, false}, /* 0xb118 */
- {RCS, GEN7_HALF_SLICE_CHICKEN1, 0xffff, true}, /* 0xe100 */
- {RCS, HALF_SLICE_CHICKEN2, 0xffff, true}, /* 0xe180 */
- {RCS, HALF_SLICE_CHICKEN3, 0xffff, true}, /* 0xe184 */
- {RCS, GEN9_HALF_SLICE_CHICKEN5, 0xffff, true}, /* 0xe188 */
- {RCS, GEN9_HALF_SLICE_CHICKEN7, 0xffff, true}, /* 0xe194 */
- {RCS, GEN8_ROW_CHICKEN, 0xffff, true}, /* 0xe4f0 */
- {RCS, TRVATTL3PTRDW(0), 0, false}, /* 0x4de0 */
- {RCS, TRVATTL3PTRDW(1), 0, false}, /* 0x4de4 */
- {RCS, TRNULLDETCT, 0, false}, /* 0x4de8 */
- {RCS, TRINVTILEDETCT, 0, false}, /* 0x4dec */
- {RCS, TRVADR, 0, false}, /* 0x4df0 */
- {RCS, TRTTE, 0, false}, /* 0x4df4 */
-
- {BCS, RING_GFX_MODE(BLT_RING_BASE), 0xffff, false}, /* 0x2229c */
- {BCS, RING_MI_MODE(BLT_RING_BASE), 0xffff, false}, /* 0x2209c */
- {BCS, RING_INSTPM(BLT_RING_BASE), 0xffff, false}, /* 0x220c0 */
- {BCS, RING_HWSTAM(BLT_RING_BASE), 0x0, false}, /* 0x22098 */
- {BCS, RING_EXCC(BLT_RING_BASE), 0x0, false}, /* 0x22028 */
-
- {VCS2, RING_EXCC(GEN8_BSD2_RING_BASE), 0xffff, false}, /* 0x1c028 */
-
- {VECS, RING_EXCC(VEBOX_RING_BASE), 0xffff, false}, /* 0x1a028 */
-
- {RCS, GEN8_HDC_CHICKEN1, 0xffff, true}, /* 0x7304 */
- {RCS, GEN9_CTX_PREEMPT_REG, 0x0, false}, /* 0x2248 */
- {RCS, GEN7_UCGCTL4, 0x0, false}, /* 0x940c */
- {RCS, GAMT_CHKN_BIT_REG, 0x0, false}, /* 0x4ab8 */
-
- {RCS, GEN9_GAMT_ECO_REG_RW_IA, 0x0, false}, /* 0x4ab0 */
- {RCS, GEN9_CSFE_CHICKEN1_RCS, 0xffff, false}, /* 0x20d4 */
- {RCS, _MMIO(0x20D8), 0xffff, true}, /* 0x20d8 */
-
- {RCS, GEN8_GARBCNTL, 0x0, false}, /* 0xb004 */
- {RCS, GEN7_FF_THREAD_MODE, 0x0, false}, /* 0x20a0 */
- {RCS, FF_SLICE_CS_CHICKEN2, 0xffff, false}, /* 0x20e4 */
- {RCS, INVALID_MMIO_REG, 0, false } /* Terminated */
+ {RCS0, GFX_MODE_GEN7, 0xffff, false}, /* 0x229c */
+ {RCS0, GEN9_CTX_PREEMPT_REG, 0x0, false}, /* 0x2248 */
+ {RCS0, HWSTAM, 0x0, false}, /* 0x2098 */
+ {RCS0, INSTPM, 0xffff, true}, /* 0x20c0 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 0), 0, false}, /* 0x24d0 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 1), 0, false}, /* 0x24d4 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 2), 0, false}, /* 0x24d8 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 3), 0, false}, /* 0x24dc */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 4), 0, false}, /* 0x24e0 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 5), 0, false}, /* 0x24e4 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 6), 0, false}, /* 0x24e8 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 7), 0, false}, /* 0x24ec */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 8), 0, false}, /* 0x24f0 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 9), 0, false}, /* 0x24f4 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 10), 0, false}, /* 0x24f8 */
+ {RCS0, RING_FORCE_TO_NONPRIV(RENDER_RING_BASE, 11), 0, false}, /* 0x24fc */
+ {RCS0, CACHE_MODE_1, 0xffff, true}, /* 0x7004 */
+ {RCS0, GEN7_GT_MODE, 0xffff, true}, /* 0x7008 */
+ {RCS0, CACHE_MODE_0_GEN7, 0xffff, true}, /* 0x7000 */
+ {RCS0, GEN7_COMMON_SLICE_CHICKEN1, 0xffff, true}, /* 0x7010 */
+ {RCS0, HDC_CHICKEN0, 0xffff, true}, /* 0x7300 */
+ {RCS0, VF_GUARDBAND, 0xffff, true}, /* 0x83a4 */
+
+ {RCS0, GEN8_PRIVATE_PAT_LO, 0, false}, /* 0x40e0 */
+ {RCS0, GEN8_PRIVATE_PAT_HI, 0, false}, /* 0x40e4 */
+ {RCS0, GEN8_CS_CHICKEN1, 0xffff, true}, /* 0x2580 */
+ {RCS0, COMMON_SLICE_CHICKEN2, 0xffff, true}, /* 0x7014 */
+ {RCS0, GEN9_CS_DEBUG_MODE1, 0xffff, false}, /* 0x20ec */
+ {RCS0, GEN8_L3SQCREG4, 0, false}, /* 0xb118 */
+ {RCS0, GEN7_HALF_SLICE_CHICKEN1, 0xffff, true}, /* 0xe100 */
+ {RCS0, HALF_SLICE_CHICKEN2, 0xffff, true}, /* 0xe180 */
+ {RCS0, HALF_SLICE_CHICKEN3, 0xffff, true}, /* 0xe184 */
+ {RCS0, GEN9_HALF_SLICE_CHICKEN5, 0xffff, true}, /* 0xe188 */
+ {RCS0, GEN9_HALF_SLICE_CHICKEN7, 0xffff, true}, /* 0xe194 */
+ {RCS0, GEN8_ROW_CHICKEN, 0xffff, true}, /* 0xe4f0 */
+ {RCS0, TRVATTL3PTRDW(0), 0, false}, /* 0x4de0 */
+ {RCS0, TRVATTL3PTRDW(1), 0, false}, /* 0x4de4 */
+ {RCS0, TRNULLDETCT, 0, false}, /* 0x4de8 */
+ {RCS0, TRINVTILEDETCT, 0, false}, /* 0x4dec */
+ {RCS0, TRVADR, 0, false}, /* 0x4df0 */
+ {RCS0, TRTTE, 0, false}, /* 0x4df4 */
+
+ {BCS0, RING_GFX_MODE(BLT_RING_BASE), 0xffff, false}, /* 0x2229c */
+ {BCS0, RING_MI_MODE(BLT_RING_BASE), 0xffff, false}, /* 0x2209c */
+ {BCS0, RING_INSTPM(BLT_RING_BASE), 0xffff, false}, /* 0x220c0 */
+ {BCS0, RING_HWSTAM(BLT_RING_BASE), 0x0, false}, /* 0x22098 */
+ {BCS0, RING_EXCC(BLT_RING_BASE), 0xffff, false}, /* 0x22028 */
+
+ {VCS1, RING_EXCC(GEN8_BSD2_RING_BASE), 0xffff, false}, /* 0x1c028 */
+
+ {VECS0, RING_EXCC(VEBOX_RING_BASE), 0xffff, false}, /* 0x1a028 */
+
+ {RCS0, GEN8_HDC_CHICKEN1, 0xffff, true}, /* 0x7304 */
+ {RCS0, GEN9_CTX_PREEMPT_REG, 0x0, false}, /* 0x2248 */
+ {RCS0, GEN7_UCGCTL4, 0x0, false}, /* 0x940c */
+ {RCS0, GAMT_CHKN_BIT_REG, 0x0, false}, /* 0x4ab8 */
+
+ {RCS0, GEN9_GAMT_ECO_REG_RW_IA, 0x0, false}, /* 0x4ab0 */
+ {RCS0, GEN9_CSFE_CHICKEN1_RCS, 0xffff, false}, /* 0x20d4 */
+
+ {RCS0, GEN8_GARBCNTL, 0x0, false}, /* 0xb004 */
+ {RCS0, GEN7_FF_THREAD_MODE, 0x0, false}, /* 0x20a0 */
+ {RCS0, FF_SLICE_CS_CHICKEN2, 0xffff, false}, /* 0x20e4 */
+ {RCS0, INVALID_MMIO_REG, 0, false } /* Terminated */
};
static struct {
{
i915_reg_t offset;
u32 regs[] = {
- [RCS] = 0xc800,
- [VCS] = 0xc900,
- [VCS2] = 0xca00,
- [BCS] = 0xcc00,
- [VECS] = 0xcb00,
+ [RCS0] = 0xc800,
+ [VCS0] = 0xc900,
+ [VCS1] = 0xca00,
+ [BCS0] = 0xcc00,
+ [VECS0] = 0xcb00,
};
int ring_id, i;
goto out;
/* no MOCS register in context except render engine */
- if (req->engine->id != RCS)
+ if (req->engine->id != RCS0)
goto out;
ret = restore_render_mocs_control_for_inhibit(vgpu, req);
static void handle_tlb_pending_event(struct intel_vgpu *vgpu, int ring_id)
{
struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
+ struct intel_uncore *uncore = &dev_priv->uncore;
struct intel_vgpu_submission *s = &vgpu->submission;
enum forcewake_domains fw;
i915_reg_t reg;
u32 regs[] = {
- [RCS] = 0x4260,
- [VCS] = 0x4264,
- [VCS2] = 0x4268,
- [BCS] = 0x426c,
- [VECS] = 0x4270,
+ [RCS0] = 0x4260,
+ [VCS0] = 0x4264,
+ [VCS1] = 0x4268,
+ [BCS0] = 0x426c,
+ [VECS0] = 0x4270,
};
if (WARN_ON(ring_id >= ARRAY_SIZE(regs)))
* otherwise device can go to RC6 state and interrupt invalidation
* process
*/
- fw = intel_uncore_forcewake_for_reg(dev_priv, reg,
+ fw = intel_uncore_forcewake_for_reg(uncore, reg,
FW_REG_READ | FW_REG_WRITE);
- if (ring_id == RCS && (INTEL_GEN(dev_priv) >= 9))
+ if (ring_id == RCS0 && INTEL_GEN(dev_priv) >= 9)
fw |= FORCEWAKE_RENDER;
- intel_uncore_forcewake_get(dev_priv, fw);
+ intel_uncore_forcewake_get(uncore, fw);
- I915_WRITE_FW(reg, 0x1);
+ intel_uncore_write_fw(uncore, reg, 0x1);
- if (wait_for_atomic((I915_READ_FW(reg) == 0), 50))
+ if (wait_for_atomic((intel_uncore_read_fw(uncore, reg) == 0), 50))
gvt_vgpu_err("timeout in invalidate ring (%d) tlb\n", ring_id);
else
vgpu_vreg_t(vgpu, reg) = 0;
- intel_uncore_forcewake_put(dev_priv, fw);
+ intel_uncore_forcewake_put(uncore, fw);
gvt_dbg_core("invalidate TLB for ring %d\n", ring_id);
}
u32 old_v, new_v;
u32 regs[] = {
- [RCS] = 0xc800,
- [VCS] = 0xc900,
- [VCS2] = 0xca00,
- [BCS] = 0xcc00,
- [VECS] = 0xcb00,
+ [RCS0] = 0xc800,
+ [VCS0] = 0xc900,
+ [VCS1] = 0xca00,
+ [BCS0] = 0xcc00,
+ [VECS0] = 0xcb00,
};
int i;
if (WARN_ON(ring_id >= ARRAY_SIZE(regs)))
return;
- if ((IS_KABYLAKE(dev_priv) || IS_BROXTON(dev_priv)
- || IS_COFFEELAKE(dev_priv)) && ring_id == RCS)
+ if (ring_id == RCS0 &&
+ (IS_KABYLAKE(dev_priv) ||
+ IS_BROXTON(dev_priv) ||
+ IS_COFFEELAKE(dev_priv)))
return;
if (!pre && !gen9_render_mocs.initialized)
offset.reg += 4;
}
- if (ring_id == RCS) {
+ if (ring_id == RCS0) {
l3_offset.reg = 0xb020;
for (i = 0; i < GEN9_MOCS_SIZE / 2; i++) {
if (pre)
* itself.
*/
if (mmio->in_context &&
- !is_inhibit_context(&s->shadow_ctx->__engine[ring_id]))
+ !is_inhibit_context(intel_context_lookup(s->shadow_ctx,
+ dev_priv->engine[ring_id])))
continue;
if (mmio->mask)
* performace for batch mmio read/write, so we need
* handle forcewake mannually.
*/
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
switch_mmio(pre, next, ring_id);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
/**
#define _REG_701C0(pipe, plane) (0x701c0 + pipe * 0x1000 + (plane - 1) * 0x100)
#define _REG_701C4(pipe, plane) (0x701c4 + pipe * 0x1000 + (plane - 1) * 0x100)
+#define SKL_FLIP_EVENT(pipe, plane) (PRIMARY_A_FLIP_DONE + (plane) * 3 + (pipe))
+
+#define PLANE_CTL_ASYNC_FLIP (1 << 9)
+#define REG50080_FLIP_TYPE_MASK 0x3
+#define REG50080_FLIP_TYPE_ASYNC 0x1
+
+#define REG_50080(_pipe, _plane) ({ \
+ typeof(_pipe) (p) = (_pipe); \
+ typeof(_plane) (q) = (_plane); \
+ (((p) == PIPE_A) ? (((q) == PLANE_PRIMARY) ? (_MMIO(0x50080)) : \
+ (_MMIO(0x50090))) : \
+ (((p) == PIPE_B) ? (((q) == PLANE_PRIMARY) ? (_MMIO(0x50088)) : \
+ (_MMIO(0x50098))) : \
+ (((p) == PIPE_C) ? (((q) == PLANE_PRIMARY) ? (_MMIO(0x5008C)) : \
+ (_MMIO(0x5009C))) : \
+ (_MMIO(0x50080))))); })
+
+#define REG_50080_TO_PIPE(_reg) ({ \
+ typeof(_reg) (reg) = (_reg); \
+ (((reg) == 0x50080 || (reg) == 0x50090) ? (PIPE_A) : \
+ (((reg) == 0x50088 || (reg) == 0x50098) ? (PIPE_B) : \
+ (((reg) == 0x5008C || (reg) == 0x5009C) ? (PIPE_C) : \
+ (INVALID_PIPE)))); })
+
+#define REG_50080_TO_PLANE(_reg) ({ \
+ typeof(_reg) (reg) = (_reg); \
+ (((reg) == 0x50080 || (reg) == 0x50088 || (reg) == 0x5008C) ? \
+ (PLANE_PRIMARY) : \
+ (((reg) == 0x50090 || (reg) == 0x50098 || (reg) == 0x5009C) ? \
+ (PLANE_SPRITE0) : (I915_MAX_PLANES))); })
+
#define GFX_MODE_BIT_SET_IN_MASK(val, bit) \
((((bit) & 0xffff0000) == 0) && !!((val) & (((bit) << 16))))
#define RING_GFX_MODE(base) _MMIO((base) + 0x29c)
#define VF_GUARDBAND _MMIO(0x83a4)
+/* define the effective range of MCHBAR register on Sandybridge+ */
+#define MCHBAR_MIRROR_REG_BASE _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x4000)
+
#endif
i915_mmio_reg_offset(EU_PERF_CNTL6),
};
- if (workload->ring_id != RCS)
+ if (workload->ring_id != RCS0)
return;
if (save) {
COPY_REG_MASKED(ctx_ctrl);
COPY_REG(ctx_timestamp);
- if (ring_id == RCS) {
+ if (ring_id == RCS0) {
COPY_REG(bb_per_ctx_ptr);
COPY_REG(rcs_indirect_ctx);
COPY_REG(rcs_indirect_ctx_offset);
context_page_num = context_page_num >> PAGE_SHIFT;
- if (IS_BROADWELL(gvt->dev_priv) && ring_id == RCS)
+ if (IS_BROADWELL(gvt->dev_priv) && ring_id == RCS0)
context_page_num = 19;
i = 2;
if (ret)
goto err_unpin;
- if ((workload->ring_id == RCS) &&
- (workload->wa_ctx.indirect_ctx.size != 0)) {
+ if (workload->ring_id == RCS0 && workload->wa_ctx.indirect_ctx.size) {
ret = intel_gvt_scan_and_shadow_wa_ctx(&workload->wa_ctx);
if (ret)
goto err_shadow;
context_page_num = rq->engine->context_size;
context_page_num = context_page_num >> PAGE_SHIFT;
- if (IS_BROADWELL(gvt->dev_priv) && rq->engine->id == RCS)
+ if (IS_BROADWELL(gvt->dev_priv) && rq->engine->id == RCS0)
context_page_num = 19;
i = 2;
}
void intel_vgpu_clean_workloads(struct intel_vgpu *vgpu,
- unsigned long engine_mask)
+ intel_engine_mask_t engine_mask)
{
struct intel_vgpu_submission *s = &vgpu->submission;
struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
struct intel_engine_cs *engine;
struct intel_vgpu_workload *pos, *n;
- unsigned int tmp;
+ intel_engine_mask_t tmp;
/* free the unsubmited workloads in the queues. */
for_each_engine_masked(engine, dev_priv, engine_mask, tmp) {
workload->status = 0;
}
- if (!workload->status && !(vgpu->resetting_eng &
- ENGINE_MASK(ring_id))) {
+ if (!workload->status &&
+ !(vgpu->resetting_eng & BIT(ring_id))) {
update_guest_context(workload);
for_each_set_bit(event, workload->pending_events,
list_del_init(&workload->list);
- if (workload->status || (vgpu->resetting_eng & ENGINE_MASK(ring_id))) {
+ if (workload->status || vgpu->resetting_eng & BIT(ring_id)) {
/* if workload->status is not successful means HW GPU
* has occurred GPU hang or something wrong with i915/GVT,
* and GVT won't inject context switch interrupt to guest.
* cleaned up during the resetting process later, so doing
* the workload clean up here doesn't have any impact.
**/
- intel_vgpu_clean_workloads(vgpu, ENGINE_MASK(ring_id));
+ intel_vgpu_clean_workloads(vgpu, BIT(ring_id));
}
workload->complete(workload);
workload->ring_id, workload);
if (need_force_wake)
- intel_uncore_forcewake_get(gvt->dev_priv,
+ intel_uncore_forcewake_get(&gvt->dev_priv->uncore,
FORCEWAKE_ALL);
ret = dispatch_workload(workload);
complete_current_workload(gvt, ring_id);
if (need_force_wake)
- intel_uncore_forcewake_put(gvt->dev_priv,
+ intel_uncore_forcewake_put(&gvt->dev_priv->uncore,
FORCEWAKE_ALL);
intel_runtime_pm_put_unchecked(gvt->dev_priv);
struct i915_hw_ppgtt *i915_ppgtt = s->shadow_ctx->ppgtt;
int i;
- if (i915_vm_is_48bit(&i915_ppgtt->vm))
+ if (i915_vm_is_4lvl(&i915_ppgtt->vm)) {
px_dma(&i915_ppgtt->pml4) = s->i915_context_pml4;
- else {
+ } else {
for (i = 0; i < GEN8_3LVL_PDPES; i++)
px_dma(i915_ppgtt->pdp.page_directory[i]) =
s->i915_context_pdps[i];
*
*/
void intel_vgpu_reset_submission(struct intel_vgpu *vgpu,
- unsigned long engine_mask)
+ intel_engine_mask_t engine_mask)
{
struct intel_vgpu_submission *s = &vgpu->submission;
struct i915_hw_ppgtt *i915_ppgtt = s->shadow_ctx->ppgtt;
int i;
- if (i915_vm_is_48bit(&i915_ppgtt->vm))
+ if (i915_vm_is_4lvl(&i915_ppgtt->vm))
s->i915_context_pml4 = px_dma(&i915_ppgtt->pml4);
else {
for (i = 0; i < GEN8_3LVL_PDPES; i++)
*
*/
int intel_vgpu_select_submission_ops(struct intel_vgpu *vgpu,
- unsigned long engine_mask,
+ intel_engine_mask_t engine_mask,
unsigned int interface)
{
struct intel_vgpu_submission *s = &vgpu->submission;
workload->rb_start = start;
workload->rb_ctl = ctl;
- if (ring_id == RCS) {
+ if (ring_id == RCS0) {
intel_gvt_hypervisor_read_gpa(vgpu, ring_context_gpa +
RING_CTX_OFF(bb_per_ctx_ptr.val), &per_ctx, 4);
intel_gvt_hypervisor_read_gpa(vgpu, ring_context_gpa +
int intel_vgpu_setup_submission(struct intel_vgpu *vgpu);
void intel_vgpu_reset_submission(struct intel_vgpu *vgpu,
- unsigned long engine_mask);
+ intel_engine_mask_t engine_mask);
void intel_vgpu_clean_submission(struct intel_vgpu *vgpu);
int intel_vgpu_select_submission_ops(struct intel_vgpu *vgpu,
- unsigned long engine_mask,
+ intel_engine_mask_t engine_mask,
unsigned int interface);
extern const struct intel_vgpu_submission_ops
void intel_vgpu_destroy_workload(struct intel_vgpu_workload *workload);
void intel_vgpu_clean_workloads(struct intel_vgpu *vgpu,
- unsigned long engine_mask);
+ intel_engine_mask_t engine_mask);
#endif
vgpu_vreg_t(vgpu, vgtif_reg(display_ready)) = 0;
vgpu_vreg_t(vgpu, vgtif_reg(vgt_id)) = vgpu->id;
- vgpu_vreg_t(vgpu, vgtif_reg(vgt_caps)) = VGT_CAPS_FULL_48BIT_PPGTT;
+ vgpu_vreg_t(vgpu, vgtif_reg(vgt_caps)) = VGT_CAPS_FULL_PPGTT;
vgpu_vreg_t(vgpu, vgtif_reg(vgt_caps)) |= VGT_CAPS_HWSP_EMULATION;
vgpu_vreg_t(vgpu, vgtif_reg(vgt_caps)) |= VGT_CAPS_HUGE_GTT;
* GPU engines. For FLR, engine_mask is ignored.
*/
void intel_gvt_reset_vgpu_locked(struct intel_vgpu *vgpu, bool dmlr,
- unsigned int engine_mask)
+ intel_engine_mask_t engine_mask)
{
struct intel_gvt *gvt = vgpu->gvt;
struct intel_gvt_workload_scheduler *scheduler = &gvt->scheduler;
- unsigned int resetting_eng = dmlr ? ALL_ENGINES : engine_mask;
+ intel_engine_mask_t resetting_eng = dmlr ? ALL_ENGINES : engine_mask;
gvt_dbg_core("------------------------------------------\n");
gvt_dbg_core("resseting vgpu%d, dmlr %d, engine_mask %08x\n",
#include "i915_drv.h"
#include "i915_active.h"
+#include "i915_globals.h"
#define BKL(ref) (&(ref)->i915->drm.struct_mutex)
* nodes from a local slab cache to hopefully reduce the fragmentation.
*/
static struct i915_global_active {
+ struct i915_global base;
struct kmem_cache *slab_cache;
} global;
#include "selftests/i915_active.c"
#endif
+static void i915_global_active_shrink(void)
+{
+ kmem_cache_shrink(global.slab_cache);
+}
+
+static void i915_global_active_exit(void)
+{
+ kmem_cache_destroy(global.slab_cache);
+}
+
+static struct i915_global_active global = { {
+ .shrink = i915_global_active_shrink,
+ .exit = i915_global_active_exit,
+} };
+
int __init i915_global_active_init(void)
{
global.slab_cache = KMEM_CACHE(active_node, SLAB_HWCACHE_ALIGN);
if (!global.slab_cache)
return -ENOMEM;
+ i915_global_register(&global.base);
return 0;
}
-
-void __exit i915_global_active_exit(void)
-{
- kmem_cache_destroy(global.slab_cache);
-}
active->retire = fn ?: i915_active_retire_noop;
}
-static inline struct i915_request *
-__i915_active_request_peek(const struct i915_active_request *active)
-{
- /*
- * Inside the error capture (running with the driver in an unknown
- * state), we want to bend the rules slightly (a lot).
- *
- * Work is in progress to make it safer, in the meantime this keeps
- * the known issue from spamming the logs.
- */
- return rcu_dereference_protected(active->request, 1);
-}
-
/**
* i915_active_request_raw - return the active request
* @active - the active tracker
static inline void i915_active_fini(struct i915_active *ref) { }
#endif
-int i915_global_active_init(void);
-void i915_global_active_exit(void);
-
#endif /* _I915_ACTIVE_H_ */
if (!IS_GEN(engine->i915, 7))
return;
- switch (engine->id) {
- case RCS:
+ switch (engine->class) {
+ case RENDER_CLASS:
if (IS_HASWELL(engine->i915)) {
cmd_tables = hsw_render_ring_cmds;
cmd_table_count =
engine->get_cmd_length_mask = gen7_render_get_cmd_length_mask;
break;
- case VCS:
+ case VIDEO_DECODE_CLASS:
cmd_tables = gen7_video_cmds;
cmd_table_count = ARRAY_SIZE(gen7_video_cmds);
engine->get_cmd_length_mask = gen7_bsd_get_cmd_length_mask;
break;
- case BCS:
+ case COPY_ENGINE_CLASS:
if (IS_HASWELL(engine->i915)) {
cmd_tables = hsw_blt_ring_cmds;
cmd_table_count = ARRAY_SIZE(hsw_blt_ring_cmds);
engine->get_cmd_length_mask = gen7_blt_get_cmd_length_mask;
break;
- case VECS:
+ case VIDEO_ENHANCEMENT_CLASS:
cmd_tables = hsw_vebox_cmds;
cmd_table_count = ARRAY_SIZE(hsw_vebox_cmds);
/* VECS can use the same length_mask function as VCS */
engine->get_cmd_length_mask = gen7_bsd_get_cmd_length_mask;
break;
default:
- MISSING_CASE(engine->id);
+ MISSING_CASE(engine->class);
return;
}
*
*/
-#include <linux/sort.h>
#include <linux/sched/mm.h>
+#include <linux/sort.h>
+
#include <drm/drm_debugfs.h>
#include <drm/drm_fourcc.h>
-#include "intel_drv.h"
-#include "intel_guc_submission.h"
#include "i915_reset.h"
+#include "intel_dp.h"
+#include "intel_drv.h"
+#include "intel_fbc.h"
+#include "intel_guc_submission.h"
+#include "intel_hdcp.h"
+#include "intel_hdmi.h"
+#include "intel_pm.h"
+#include "intel_psr.h"
static inline struct drm_i915_private *node_to_i915(struct drm_info_node *node)
{
struct i915_gem_context *ctx;
list_for_each_entry(ctx, &i915->contexts.list, link) {
- struct intel_engine_cs *engine;
- enum intel_engine_id id;
-
- for_each_engine(engine, i915, id) {
- struct intel_context *ce = to_intel_context(ctx, engine);
+ struct intel_context *ce;
+ list_for_each_entry(ce, &ctx->active_engines, active_link) {
if (ce->state)
per_file_stats(0, ce->state->obj, &kstats);
if (ce->ring)
rcu_read_lock();
task = pid_task(ctx->pid ?: file->pid, PIDTYPE_PID);
- snprintf(name, sizeof(name), "%s/%d",
- task ? task->comm : "<unknown>",
- ctx->user_handle);
+ snprintf(name, sizeof(name), "%s",
+ task ? task->comm : "<unknown>");
rcu_read_unlock();
print_file_stats(m, name, stats);
} else if (!HAS_PCH_SPLIT(dev_priv)) {
seq_printf(m, "Interrupt enable: %08x\n",
- I915_READ(IER));
+ I915_READ(GEN2_IER));
seq_printf(m, "Interrupt identity: %08x\n",
- I915_READ(IIR));
+ I915_READ(GEN2_IIR));
seq_printf(m, "Interrupt mask: %08x\n",
- I915_READ(IMR));
+ I915_READ(GEN2_IMR));
for_each_pipe(dev_priv, pipe)
seq_printf(m, "Pipe %c stat: %08x\n",
pipe_name(pipe),
for_each_engine(engine, dev_priv, id) {
seq_printf(m,
"Graphics Interrupt mask (%s): %08x\n",
- engine->name, I915_READ_IMR(engine));
+ engine->name, ENGINE_READ(engine, RING_IMR));
}
}
}
/* RPSTAT1 is in the GT power well */
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
reqf = I915_READ(GEN6_RPNSWREQ);
if (INTEL_GEN(dev_priv) >= 9)
cagf = intel_gpu_freq(dev_priv,
intel_get_cagf(dev_priv, rpstat));
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
if (INTEL_GEN(dev_priv) >= 11) {
pm_ier = I915_READ(GEN11_GPM_WGBOXPERF_INTR_ENABLE);
intel_wakeref_t wakeref;
enum intel_engine_id id;
+ seq_printf(m, "Reset flags: %lx\n", dev_priv->gpu_error.flags);
if (test_bit(I915_WEDGED, &dev_priv->gpu_error.flags))
- seq_puts(m, "Wedged\n");
+ seq_puts(m, "\tWedged\n");
if (test_bit(I915_RESET_BACKOFF, &dev_priv->gpu_error.flags))
- seq_puts(m, "Reset in progress: struct_mutex backoff\n");
- if (waitqueue_active(&dev_priv->gpu_error.wait_queue))
- seq_puts(m, "Waiter holding struct mutex\n");
- if (waitqueue_active(&dev_priv->gpu_error.reset_queue))
- seq_puts(m, "struct_mutex blocked for reset\n");
+ seq_puts(m, "\tDevice (global) reset in progress\n");
if (!i915_modparams.enable_hangcheck) {
seq_puts(m, "Hangcheck disabled\n");
with_intel_runtime_pm(dev_priv, wakeref) {
for_each_engine(engine, dev_priv, id) {
acthd[id] = intel_engine_get_active_head(engine);
- seqno[id] = intel_engine_get_seqno(engine);
+ seqno[id] = intel_engine_get_hangcheck_seqno(engine);
}
- intel_engine_get_instdone(dev_priv->engine[RCS], &instdone);
+ intel_engine_get_instdone(dev_priv->engine[RCS0], &instdone);
}
if (timer_pending(&dev_priv->gpu_error.hangcheck_work.timer))
for_each_engine(engine, dev_priv, id) {
seq_printf(m, "%s:\n", engine->name);
seq_printf(m, "\tseqno = %x [current %x, last %x], %dms ago\n",
- engine->hangcheck.seqno, seqno[id],
- intel_engine_last_submit(engine),
+ engine->hangcheck.last_seqno,
+ seqno[id],
+ engine->hangcheck.next_seqno,
jiffies_to_msecs(jiffies -
engine->hangcheck.action_timestamp));
(long long)engine->hangcheck.acthd,
(long long)acthd[id]);
- if (engine->id == RCS) {
+ if (engine->id == RCS0) {
seq_puts(m, "\tinstdone read =\n");
i915_instdone_info(dev_priv, m, &instdone);
static int i915_forcewake_domains(struct seq_file *m, void *data)
{
struct drm_i915_private *i915 = node_to_i915(m->private);
+ struct intel_uncore *uncore = &i915->uncore;
struct intel_uncore_forcewake_domain *fw_domain;
unsigned int tmp;
seq_printf(m, "user.bypass_count = %u\n",
- i915->uncore.user_forcewake.count);
+ uncore->user_forcewake.count);
- for_each_fw_domain(fw_domain, i915, tmp)
+ for_each_fw_domain(fw_domain, uncore, tmp)
seq_printf(m, "%s.wake_count = %u\n",
intel_uncore_forcewake_domain_to_str(fw_domain->id),
READ_ONCE(fw_domain->wake_count));
{
struct drm_i915_private *dev_priv = node_to_i915(m->private);
struct drm_device *dev = &dev_priv->drm;
- struct intel_engine_cs *engine;
struct i915_gem_context *ctx;
- enum intel_engine_id id;
int ret;
ret = mutex_lock_interruptible(&dev->struct_mutex);
return ret;
list_for_each_entry(ctx, &dev_priv->contexts.list, link) {
+ struct intel_context *ce;
+
seq_puts(m, "HW context ");
if (!list_empty(&ctx->hw_id_link))
seq_printf(m, "%x [pin %u]", ctx->hw_id,
seq_putc(m, ctx->remap_slice ? 'R' : 'r');
seq_putc(m, '\n');
- for_each_engine(engine, dev_priv, id) {
- struct intel_context *ce =
- to_intel_context(ctx, engine);
-
- seq_printf(m, "%s: ", engine->name);
+ list_for_each_entry(ce, &ctx->active_engines, active_link) {
+ seq_printf(m, "%s: ", ce->engine->name);
if (ce->state)
describe_obj(m, ce->state->obj);
if (ce->ring)
static int i915_rps_boost_info(struct seq_file *m, void *data)
{
struct drm_i915_private *dev_priv = node_to_i915(m->private);
- struct drm_device *dev = &dev_priv->drm;
struct intel_rps *rps = &dev_priv->gt_pm.rps;
u32 act_freq = rps->cur_freq;
intel_wakeref_t wakeref;
- struct drm_file *file;
with_intel_runtime_pm_if_in_use(dev_priv, wakeref) {
if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) {
intel_gpu_freq(dev_priv, rps->efficient_freq),
intel_gpu_freq(dev_priv, rps->boost_freq));
- mutex_lock(&dev->filelist_mutex);
- list_for_each_entry_reverse(file, &dev->filelist, lhead) {
- struct drm_i915_file_private *file_priv = file->driver_priv;
- struct task_struct *task;
-
- rcu_read_lock();
- task = pid_task(file->pid, PIDTYPE_PID);
- seq_printf(m, "%s [%d]: %d boosts\n",
- task ? task->comm : "<unknown>",
- task ? task->pid : -1,
- atomic_read(&file_priv->rps_client.boosts));
- rcu_read_unlock();
- }
- seq_printf(m, "Kernel (anonymous) boosts: %d\n",
- atomic_read(&rps->boosts));
- mutex_unlock(&dev->filelist_mutex);
+ seq_printf(m, "Wait boosts: %d\n", atomic_read(&rps->boosts));
if (INTEL_GEN(dev_priv) >= 6 &&
rps->enabled &&
u32 rpup, rpupei;
u32 rpdown, rpdownei;
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
rpup = I915_READ_FW(GEN6_RP_CUR_UP) & GEN6_RP_EI_MASK;
rpupei = I915_READ_FW(GEN6_RP_CUR_UP_EI) & GEN6_RP_EI_MASK;
rpdown = I915_READ_FW(GEN6_RP_CUR_DOWN) & GEN6_RP_EI_MASK;
rpdownei = I915_READ_FW(GEN6_RP_CUR_DOWN_EI) & GEN6_RP_EI_MASK;
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
seq_printf(m, "\nRPS Autotuning (current \"%s\" window):\n",
rps_power_to_str(rps->power.mode));
const bool edram = INTEL_GEN(dev_priv) > 8;
seq_printf(m, "LLC: %s\n", yesno(HAS_LLC(dev_priv)));
- seq_printf(m, "%s: %lluMB\n", edram ? "eDRAM" : "eLLC",
- intel_uncore_edram_size(dev_priv)/1024/1024);
+ seq_printf(m, "%s: %uMB\n", edram ? "eDRAM" : "eLLC",
+ dev_priv->edram_size_mb);
return 0;
}
const struct intel_guc *guc = &dev_priv->guc;
struct guc_stage_desc *desc = guc->stage_desc_pool_vaddr;
struct intel_guc_client *client = guc->execbuf_client;
- unsigned int tmp;
+ intel_engine_mask_t tmp;
int index;
if (!USES_GUC_SUBMISSION(dev_priv))
i915_edp_psr_debug_set(void *data, u64 val)
{
struct drm_i915_private *dev_priv = data;
- struct drm_modeset_acquire_ctx ctx;
intel_wakeref_t wakeref;
int ret;
wakeref = intel_runtime_pm_get(dev_priv);
- drm_modeset_acquire_init(&ctx, DRM_MODESET_ACQUIRE_INTERRUPTIBLE);
-
-retry:
- ret = intel_psr_set_debugfs_mode(dev_priv, &ctx, val);
- if (ret == -EDEADLK) {
- ret = drm_modeset_backoff(&ctx);
- if (!ret)
- goto retry;
- }
-
- drm_modeset_drop_locks(&ctx);
- drm_modeset_acquire_fini(&ctx);
+ ret = intel_psr_debug_set(dev_priv, val);
intel_runtime_pm_put(dev_priv, wakeref);
seq_printf(m, "Runtime power status: %s\n",
enableddisabled(!dev_priv->power_domains.wakeref));
- seq_printf(m, "GPU idle: %s (epoch %u)\n",
- yesno(!dev_priv->gt.awake), dev_priv->gt.epoch);
+ seq_printf(m, "GPU idle: %s\n", yesno(!dev_priv->gt.awake));
seq_printf(m, "IRQs disabled: %s\n",
yesno(!intel_irqs_enabled(dev_priv)));
#ifdef CONFIG_PM
if (connector->status == connector_status_disconnected)
return;
- seq_printf(m, "\tname: %s\n", connector->display_info.name);
seq_printf(m, "\tphysical dimensions: %dx%dmm\n",
connector->display_info.width_mm,
connector->display_info.height_mm);
wakeref = intel_runtime_pm_get(dev_priv);
- seq_printf(m, "GT awake? %s (epoch %u)\n",
- yesno(dev_priv->gt.awake), dev_priv->gt.epoch);
+ seq_printf(m, "GT awake? %s\n", yesno(dev_priv->gt.awake));
seq_printf(m, "Global active requests: %d\n",
dev_priv->gt.active_requests);
seq_printf(m, "CS timestamp frequency: %u kHz\n",
static int i915_wa_registers(struct seq_file *m, void *unused)
{
struct drm_i915_private *i915 = node_to_i915(m->private);
- const struct i915_wa_list *wal = &i915->engine[RCS]->ctx_wa_list;
+ const struct i915_wa_list *wal = &i915->engine[RCS0]->ctx_wa_list;
struct i915_wa *wa;
unsigned int i;
static int
i915_wedged_get(void *data, u64 *val)
{
- struct drm_i915_private *dev_priv = data;
-
- *val = i915_terminally_wedged(&dev_priv->gpu_error);
+ int ret = i915_terminally_wedged(data);
- return 0;
+ switch (ret) {
+ case -EIO:
+ *val = 1;
+ return 0;
+ case 0:
+ *val = 0;
+ return 0;
+ default:
+ return ret;
+ }
}
static int
{
struct drm_i915_private *i915 = data;
- /*
- * There is no safeguard against this debugfs entry colliding
- * with the hangcheck calling same i915_handle_error() in
- * parallel, causing an explosion. For now we assume that the
- * test harness is responsible enough not to inject gpu hangs
- * while it is writing to 'i915_wedged'
- */
-
- if (i915_reset_backoff(&i915->gpu_error))
- return -EAGAIN;
+ /* Flush any previous reset before applying for a new one */
+ wait_event(i915->gpu_error.reset_queue,
+ !test_bit(I915_RESET_BACKOFF, &i915->gpu_error.flags));
i915_handle_error(i915, val, I915_ERROR_CAPTURE,
"Manually set wedged engine mask = %llx", val);
i915_drop_caches_set(void *data, u64 val)
{
struct drm_i915_private *i915 = data;
- intel_wakeref_t wakeref;
- int ret = 0;
DRM_DEBUG("Dropping caches: 0x%08llx [0x%08llx]\n",
val, val & DROP_ALL);
- wakeref = intel_runtime_pm_get(i915);
if (val & DROP_RESET_ACTIVE &&
wait_for(intel_engines_are_idle(i915), I915_IDLE_ENGINES_TIMEOUT))
/* No need to check and wait for gpu resets, only libdrm auto-restarts
* on ioctls on -EAGAIN. */
if (val & (DROP_ACTIVE | DROP_RETIRE | DROP_RESET_SEQNO)) {
+ int ret;
+
ret = mutex_lock_interruptible(&i915->drm.struct_mutex);
if (ret)
- goto out;
+ return ret;
if (val & DROP_ACTIVE)
ret = i915_gem_wait_for_idle(i915,
mutex_unlock(&i915->drm.struct_mutex);
}
- if (val & DROP_RESET_ACTIVE && i915_terminally_wedged(&i915->gpu_error))
+ if (val & DROP_RESET_ACTIVE && i915_terminally_wedged(i915))
i915_handle_error(i915, ALL_ENGINES, 0, NULL);
fs_reclaim_acquire(GFP_KERNEL);
if (val & DROP_FREED)
i915_gem_drain_freed_objects(i915);
-out:
- intel_runtime_pm_put(i915, wakeref);
-
- return ret;
+ return 0;
}
DEFINE_SIMPLE_ATTRIBUTE(i915_drop_caches_fops,
return 0;
file->private_data = (void *)(uintptr_t)intel_runtime_pm_get(i915);
- intel_uncore_forcewake_user_get(i915);
+ intel_uncore_forcewake_user_get(&i915->uncore);
return 0;
}
if (INTEL_GEN(i915) < 6)
return 0;
- intel_uncore_forcewake_user_put(i915);
+ intel_uncore_forcewake_user_put(&i915->uncore);
intel_runtime_pm_put(i915,
(intel_wakeref_t)(uintptr_t)file->private_data);
yesno(crtc_state->dsc_params.compression_enable));
seq_printf(m, "DSC_Sink_Support: %s\n",
yesno(drm_dp_sink_supports_dsc(intel_dp->dsc_dpcd)));
+ seq_printf(m, "Force_DSC_Enable: %s\n",
+ yesno(intel_dp->force_dsc_en));
if (!intel_dp_is_edp(intel_dp))
seq_printf(m, "FEC_Sink_Support: %s\n",
yesno(drm_dp_sink_supports_fec(intel_dp->fec_capable)));
#include <drm/i915_drm.h>
#include "i915_drv.h"
-#include "i915_trace.h"
#include "i915_pmu.h"
-#include "i915_reset.h"
#include "i915_query.h"
+#include "i915_reset.h"
+#include "i915_trace.h"
#include "i915_vgpu.h"
+#include "intel_audio.h"
+#include "intel_cdclk.h"
+#include "intel_csr.h"
+#include "intel_dp.h"
#include "intel_drv.h"
+#include "intel_fbdev.h"
+#include "intel_pm.h"
+#include "intel_sprite.h"
#include "intel_uc.h"
#include "intel_workarounds.h"
DRM_DEBUG_KMS("Found Cannon Lake LP PCH (CNP-LP)\n");
WARN_ON(!IS_CANNONLAKE(dev_priv) && !IS_COFFEELAKE(dev_priv));
return PCH_CNP;
+ case INTEL_PCH_CMP_DEVICE_ID_TYPE:
+ DRM_DEBUG_KMS("Found Comet Lake PCH (CMP)\n");
+ WARN_ON(!IS_COFFEELAKE(dev_priv));
+ /* CometPoint is CNP Compatible */
+ return PCH_CNP;
case INTEL_PCH_ICP_DEVICE_ID_TYPE:
DRM_DEBUG_KMS("Found Ice Lake PCH\n");
WARN_ON(!IS_ICELAKE(dev_priv));
* make an educated guess as to which PCH is really there.
*/
- if (IS_GEN(dev_priv, 5))
- id = INTEL_PCH_IBX_DEVICE_ID_TYPE;
- else if (IS_GEN(dev_priv, 6) || IS_IVYBRIDGE(dev_priv))
- id = INTEL_PCH_CPT_DEVICE_ID_TYPE;
+ if (IS_ICELAKE(dev_priv))
+ id = INTEL_PCH_ICP_DEVICE_ID_TYPE;
+ else if (IS_CANNONLAKE(dev_priv) || IS_COFFEELAKE(dev_priv))
+ id = INTEL_PCH_CNP_DEVICE_ID_TYPE;
+ else if (IS_KABYLAKE(dev_priv) || IS_SKYLAKE(dev_priv))
+ id = INTEL_PCH_SPT_DEVICE_ID_TYPE;
else if (IS_HSW_ULT(dev_priv) || IS_BDW_ULT(dev_priv))
id = INTEL_PCH_LPT_LP_DEVICE_ID_TYPE;
else if (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv))
id = INTEL_PCH_LPT_DEVICE_ID_TYPE;
- else if (IS_SKYLAKE(dev_priv) || IS_KABYLAKE(dev_priv))
- id = INTEL_PCH_SPT_DEVICE_ID_TYPE;
- else if (IS_COFFEELAKE(dev_priv) || IS_CANNONLAKE(dev_priv))
- id = INTEL_PCH_CNP_DEVICE_ID_TYPE;
- else if (IS_ICELAKE(dev_priv))
- id = INTEL_PCH_ICP_DEVICE_ID_TYPE;
+ else if (IS_GEN(dev_priv, 6) || IS_IVYBRIDGE(dev_priv))
+ id = INTEL_PCH_CPT_DEVICE_ID_TYPE;
+ else if (IS_GEN(dev_priv, 5))
+ id = INTEL_PCH_IBX_DEVICE_ID_TYPE;
if (id)
DRM_DEBUG_KMS("Assuming PCH ID %04x\n", id);
value = dev_priv->overlay ? 1 : 0;
break;
case I915_PARAM_HAS_BSD:
- value = !!dev_priv->engine[VCS];
+ value = !!dev_priv->engine[VCS0];
break;
case I915_PARAM_HAS_BLT:
- value = !!dev_priv->engine[BCS];
+ value = !!dev_priv->engine[BCS0];
break;
case I915_PARAM_HAS_VEBOX:
- value = !!dev_priv->engine[VECS];
+ value = !!dev_priv->engine[VECS0];
break;
case I915_PARAM_HAS_BSD2:
- value = !!dev_priv->engine[VCS2];
+ value = !!dev_priv->engine[VCS1];
break;
case I915_PARAM_HAS_LLC:
value = HAS_LLC(dev_priv);
value = HAS_WT(dev_priv);
break;
case I915_PARAM_HAS_ALIASING_PPGTT:
- value = min_t(int, INTEL_PPGTT(dev_priv), I915_GEM_PPGTT_FULL);
+ value = INTEL_PPGTT(dev_priv);
break;
case I915_PARAM_HAS_SEMAPHORES:
- value = 0;
+ value = !!(dev_priv->caps.scheduler & I915_SCHEDULER_CAP_SEMAPHORES);
break;
case I915_PARAM_HAS_SECURE_BATCHES:
value = capable(CAP_SYS_ADMIN);
return 0;
cleanup_gem:
- if (i915_gem_suspend(dev_priv))
- DRM_ERROR("failed to idle hardware; continuing to unload!\n");
+ i915_gem_suspend(dev_priv);
i915_gem_fini(dev_priv);
cleanup_modeset:
intel_modeset_cleanup(dev);
if (i915_inject_load_failure())
return -ENODEV;
+ intel_device_info_subplatform_init(dev_priv);
+
+ intel_uncore_init_early(&dev_priv->uncore);
+
spin_lock_init(&dev_priv->irq_lock);
spin_lock_init(&dev_priv->gpu_error.lock);
mutex_init(&dev_priv->backlight_lock);
- spin_lock_init(&dev_priv->uncore.lock);
mutex_init(&dev_priv->sb_lock);
mutex_init(&dev_priv->av_mutex);
mutex_init(&dev_priv->wm.wm_mutex);
mutex_init(&dev_priv->pps_mutex);
+ mutex_init(&dev_priv->hdcp_comp_mutex);
i915_memcpy_init_early(dev_priv);
intel_runtime_pm_init_early(dev_priv);
i915_engines_cleanup(dev_priv);
}
-static int i915_mmio_setup(struct drm_i915_private *dev_priv)
-{
- struct pci_dev *pdev = dev_priv->drm.pdev;
- int mmio_bar;
- int mmio_size;
-
- mmio_bar = IS_GEN(dev_priv, 2) ? 1 : 0;
- /*
- * Before gen4, the registers and the GTT are behind different BARs.
- * However, from gen4 onwards, the registers and the GTT are shared
- * in the same BAR, so we want to restrict this ioremap from
- * clobbering the GTT which we want ioremap_wc instead. Fortunately,
- * the register BAR remains the same size for all the earlier
- * generations up to Ironlake.
- */
- if (INTEL_GEN(dev_priv) < 5)
- mmio_size = 512 * 1024;
- else
- mmio_size = 2 * 1024 * 1024;
- dev_priv->regs = pci_iomap(pdev, mmio_bar, mmio_size);
- if (dev_priv->regs == NULL) {
- DRM_ERROR("failed to map registers\n");
-
- return -EIO;
- }
-
- /* Try to make sure MCHBAR is enabled before poking at it */
- intel_setup_mchbar(dev_priv);
-
- return 0;
-}
-
-static void i915_mmio_cleanup(struct drm_i915_private *dev_priv)
-{
- struct pci_dev *pdev = dev_priv->drm.pdev;
-
- intel_teardown_mchbar(dev_priv);
- pci_iounmap(pdev, dev_priv->regs);
-}
-
/**
* i915_driver_init_mmio - setup device MMIO
* @dev_priv: device private
if (i915_get_bridge_dev(dev_priv))
return -EIO;
- ret = i915_mmio_setup(dev_priv);
+ ret = intel_uncore_init_mmio(&dev_priv->uncore);
if (ret < 0)
goto err_bridge;
- intel_uncore_init(dev_priv);
+ /* Try to make sure MCHBAR is enabled before poking at it */
+ intel_setup_mchbar(dev_priv);
intel_device_info_init_mmio(dev_priv);
- intel_uncore_prune(dev_priv);
+ intel_uncore_prune_mmio_domains(&dev_priv->uncore);
intel_uc_init_mmio(dev_priv);
return 0;
err_uncore:
- intel_uncore_fini(dev_priv);
- i915_mmio_cleanup(dev_priv);
+ intel_teardown_mchbar(dev_priv);
+ intel_uncore_fini_mmio(&dev_priv->uncore);
err_bridge:
pci_dev_put(dev_priv->bridge_dev);
*/
static void i915_driver_cleanup_mmio(struct drm_i915_private *dev_priv)
{
- intel_uncore_fini(dev_priv);
- i915_mmio_cleanup(dev_priv);
+ intel_teardown_mchbar(dev_priv);
+ intel_uncore_fini_mmio(&dev_priv->uncore);
pci_dev_put(dev_priv->bridge_dev);
}
intel_gvt_sanitize_options(dev_priv);
}
-static enum dram_rank skl_get_dimm_rank(u8 size, u32 rank)
+#define DRAM_TYPE_STR(type) [INTEL_DRAM_ ## type] = #type
+
+static const char *intel_dram_type_str(enum intel_dram_type type)
{
- if (size == 0)
- return I915_DRAM_RANK_INVALID;
- if (rank == SKL_DRAM_RANK_SINGLE)
- return I915_DRAM_RANK_SINGLE;
- else if (rank == SKL_DRAM_RANK_DUAL)
- return I915_DRAM_RANK_DUAL;
+ static const char * const str[] = {
+ DRAM_TYPE_STR(UNKNOWN),
+ DRAM_TYPE_STR(DDR3),
+ DRAM_TYPE_STR(DDR4),
+ DRAM_TYPE_STR(LPDDR3),
+ DRAM_TYPE_STR(LPDDR4),
+ };
+
+ if (type >= ARRAY_SIZE(str))
+ type = INTEL_DRAM_UNKNOWN;
- return I915_DRAM_RANK_INVALID;
+ return str[type];
}
-static bool
-skl_is_16gb_dimm(enum dram_rank rank, u8 size, u8 width)
+#undef DRAM_TYPE_STR
+
+static int intel_dimm_num_devices(const struct dram_dimm_info *dimm)
{
- if (rank == I915_DRAM_RANK_SINGLE && width == 8 && size == 16)
- return true;
- else if (rank == I915_DRAM_RANK_DUAL && width == 8 && size == 32)
- return true;
- else if (rank == SKL_DRAM_RANK_SINGLE && width == 16 && size == 8)
- return true;
- else if (rank == SKL_DRAM_RANK_DUAL && width == 16 && size == 16)
- return true;
+ return dimm->ranks * 64 / (dimm->width ?: 1);
+}
- return false;
+/* Returns total GB for the whole DIMM */
+static int skl_get_dimm_size(u16 val)
+{
+ return val & SKL_DRAM_SIZE_MASK;
}
-static int
-skl_dram_get_channel_info(struct dram_channel_info *ch, u32 val)
+static int skl_get_dimm_width(u16 val)
{
- u32 tmp_l, tmp_s;
- u32 s_val = val >> SKL_DRAM_S_SHIFT;
+ if (skl_get_dimm_size(val) == 0)
+ return 0;
- if (!val)
- return -EINVAL;
+ switch (val & SKL_DRAM_WIDTH_MASK) {
+ case SKL_DRAM_WIDTH_X8:
+ case SKL_DRAM_WIDTH_X16:
+ case SKL_DRAM_WIDTH_X32:
+ val = (val & SKL_DRAM_WIDTH_MASK) >> SKL_DRAM_WIDTH_SHIFT;
+ return 8 << val;
+ default:
+ MISSING_CASE(val);
+ return 0;
+ }
+}
+
+static int skl_get_dimm_ranks(u16 val)
+{
+ if (skl_get_dimm_size(val) == 0)
+ return 0;
+
+ val = (val & SKL_DRAM_RANK_MASK) >> SKL_DRAM_RANK_SHIFT;
+
+ return val + 1;
+}
+
+/* Returns total GB for the whole DIMM */
+static int cnl_get_dimm_size(u16 val)
+{
+ return (val & CNL_DRAM_SIZE_MASK) / 2;
+}
+
+static int cnl_get_dimm_width(u16 val)
+{
+ if (cnl_get_dimm_size(val) == 0)
+ return 0;
+
+ switch (val & CNL_DRAM_WIDTH_MASK) {
+ case CNL_DRAM_WIDTH_X8:
+ case CNL_DRAM_WIDTH_X16:
+ case CNL_DRAM_WIDTH_X32:
+ val = (val & CNL_DRAM_WIDTH_MASK) >> CNL_DRAM_WIDTH_SHIFT;
+ return 8 << val;
+ default:
+ MISSING_CASE(val);
+ return 0;
+ }
+}
+
+static int cnl_get_dimm_ranks(u16 val)
+{
+ if (cnl_get_dimm_size(val) == 0)
+ return 0;
+
+ val = (val & CNL_DRAM_RANK_MASK) >> CNL_DRAM_RANK_SHIFT;
+
+ return val + 1;
+}
- tmp_l = val & SKL_DRAM_SIZE_MASK;
- tmp_s = s_val & SKL_DRAM_SIZE_MASK;
+static bool
+skl_is_16gb_dimm(const struct dram_dimm_info *dimm)
+{
+ /* Convert total GB to Gb per DRAM device */
+ return 8 * dimm->size / (intel_dimm_num_devices(dimm) ?: 1) == 16;
+}
+
+static void
+skl_dram_get_dimm_info(struct drm_i915_private *dev_priv,
+ struct dram_dimm_info *dimm,
+ int channel, char dimm_name, u16 val)
+{
+ if (INTEL_GEN(dev_priv) >= 10) {
+ dimm->size = cnl_get_dimm_size(val);
+ dimm->width = cnl_get_dimm_width(val);
+ dimm->ranks = cnl_get_dimm_ranks(val);
+ } else {
+ dimm->size = skl_get_dimm_size(val);
+ dimm->width = skl_get_dimm_width(val);
+ dimm->ranks = skl_get_dimm_ranks(val);
+ }
+
+ DRM_DEBUG_KMS("CH%u DIMM %c size: %u GB, width: X%u, ranks: %u, 16Gb DIMMs: %s\n",
+ channel, dimm_name, dimm->size, dimm->width, dimm->ranks,
+ yesno(skl_is_16gb_dimm(dimm)));
+}
+
+static int
+skl_dram_get_channel_info(struct drm_i915_private *dev_priv,
+ struct dram_channel_info *ch,
+ int channel, u32 val)
+{
+ skl_dram_get_dimm_info(dev_priv, &ch->dimm_l,
+ channel, 'L', val & 0xffff);
+ skl_dram_get_dimm_info(dev_priv, &ch->dimm_s,
+ channel, 'S', val >> 16);
- if (tmp_l == 0 && tmp_s == 0)
+ if (ch->dimm_l.size == 0 && ch->dimm_s.size == 0) {
+ DRM_DEBUG_KMS("CH%u not populated\n", channel);
return -EINVAL;
+ }
- ch->l_info.size = tmp_l;
- ch->s_info.size = tmp_s;
-
- tmp_l = (val & SKL_DRAM_WIDTH_MASK) >> SKL_DRAM_WIDTH_SHIFT;
- tmp_s = (s_val & SKL_DRAM_WIDTH_MASK) >> SKL_DRAM_WIDTH_SHIFT;
- ch->l_info.width = (1 << tmp_l) * 8;
- ch->s_info.width = (1 << tmp_s) * 8;
-
- tmp_l = val & SKL_DRAM_RANK_MASK;
- tmp_s = s_val & SKL_DRAM_RANK_MASK;
- ch->l_info.rank = skl_get_dimm_rank(ch->l_info.size, tmp_l);
- ch->s_info.rank = skl_get_dimm_rank(ch->s_info.size, tmp_s);
-
- if (ch->l_info.rank == I915_DRAM_RANK_DUAL ||
- ch->s_info.rank == I915_DRAM_RANK_DUAL)
- ch->rank = I915_DRAM_RANK_DUAL;
- else if (ch->l_info.rank == I915_DRAM_RANK_SINGLE &&
- ch->s_info.rank == I915_DRAM_RANK_SINGLE)
- ch->rank = I915_DRAM_RANK_DUAL;
+ if (ch->dimm_l.ranks == 2 || ch->dimm_s.ranks == 2)
+ ch->ranks = 2;
+ else if (ch->dimm_l.ranks == 1 && ch->dimm_s.ranks == 1)
+ ch->ranks = 2;
else
- ch->rank = I915_DRAM_RANK_SINGLE;
+ ch->ranks = 1;
- ch->is_16gb_dimm = skl_is_16gb_dimm(ch->l_info.rank, ch->l_info.size,
- ch->l_info.width) ||
- skl_is_16gb_dimm(ch->s_info.rank, ch->s_info.size,
- ch->s_info.width);
+ ch->is_16gb_dimm =
+ skl_is_16gb_dimm(&ch->dimm_l) ||
+ skl_is_16gb_dimm(&ch->dimm_s);
- DRM_DEBUG_KMS("(size:width:rank) L(%dGB:X%d:%s) S(%dGB:X%d:%s)\n",
- ch->l_info.size, ch->l_info.width,
- ch->l_info.rank ? "dual" : "single",
- ch->s_info.size, ch->s_info.width,
- ch->s_info.rank ? "dual" : "single");
+ DRM_DEBUG_KMS("CH%u ranks: %u, 16Gb DIMMs: %s\n",
+ channel, ch->ranks, yesno(ch->is_16gb_dimm));
return 0;
}
static bool
-intel_is_dram_symmetric(u32 val_ch0, u32 val_ch1,
- struct dram_channel_info *ch0)
+intel_is_dram_symmetric(const struct dram_channel_info *ch0,
+ const struct dram_channel_info *ch1)
{
- return (val_ch0 == val_ch1 &&
- (ch0->s_info.size == 0 ||
- (ch0->l_info.size == ch0->s_info.size &&
- ch0->l_info.width == ch0->s_info.width &&
- ch0->l_info.rank == ch0->s_info.rank)));
+ return !memcmp(ch0, ch1, sizeof(*ch0)) &&
+ (ch0->dimm_s.size == 0 ||
+ !memcmp(&ch0->dimm_l, &ch0->dimm_s, sizeof(ch0->dimm_l)));
}
static int
skl_dram_get_channels_info(struct drm_i915_private *dev_priv)
{
struct dram_info *dram_info = &dev_priv->dram_info;
- struct dram_channel_info ch0, ch1;
- u32 val_ch0, val_ch1;
+ struct dram_channel_info ch0 = {}, ch1 = {};
+ u32 val;
int ret;
- val_ch0 = I915_READ(SKL_MAD_DIMM_CH0_0_0_0_MCHBAR_MCMAIN);
- ret = skl_dram_get_channel_info(&ch0, val_ch0);
+ val = I915_READ(SKL_MAD_DIMM_CH0_0_0_0_MCHBAR_MCMAIN);
+ ret = skl_dram_get_channel_info(dev_priv, &ch0, 0, val);
if (ret == 0)
dram_info->num_channels++;
- val_ch1 = I915_READ(SKL_MAD_DIMM_CH1_0_0_0_MCHBAR_MCMAIN);
- ret = skl_dram_get_channel_info(&ch1, val_ch1);
+ val = I915_READ(SKL_MAD_DIMM_CH1_0_0_0_MCHBAR_MCMAIN);
+ ret = skl_dram_get_channel_info(dev_priv, &ch1, 1, val);
if (ret == 0)
dram_info->num_channels++;
* will be same as if single rank memory, so consider single rank
* memory.
*/
- if (ch0.rank == I915_DRAM_RANK_SINGLE ||
- ch1.rank == I915_DRAM_RANK_SINGLE)
- dram_info->rank = I915_DRAM_RANK_SINGLE;
+ if (ch0.ranks == 1 || ch1.ranks == 1)
+ dram_info->ranks = 1;
else
- dram_info->rank = max(ch0.rank, ch1.rank);
+ dram_info->ranks = max(ch0.ranks, ch1.ranks);
- if (dram_info->rank == I915_DRAM_RANK_INVALID) {
+ if (dram_info->ranks == 0) {
DRM_INFO("couldn't get memory rank information\n");
return -EINVAL;
}
dram_info->is_16gb_dimm = ch0.is_16gb_dimm || ch1.is_16gb_dimm;
- dev_priv->dram_info.symmetric_memory = intel_is_dram_symmetric(val_ch0,
- val_ch1,
- &ch0);
+ dram_info->symmetric_memory = intel_is_dram_symmetric(&ch0, &ch1);
- DRM_DEBUG_KMS("memory configuration is %sSymmetric memory\n",
- dev_priv->dram_info.symmetric_memory ? "" : "not ");
+ DRM_DEBUG_KMS("Memory configuration is symmetric? %s\n",
+ yesno(dram_info->symmetric_memory));
return 0;
}
+static enum intel_dram_type
+skl_get_dram_type(struct drm_i915_private *dev_priv)
+{
+ u32 val;
+
+ val = I915_READ(SKL_MAD_INTER_CHANNEL_0_0_0_MCHBAR_MCMAIN);
+
+ switch (val & SKL_DRAM_DDR_TYPE_MASK) {
+ case SKL_DRAM_DDR_TYPE_DDR3:
+ return INTEL_DRAM_DDR3;
+ case SKL_DRAM_DDR_TYPE_DDR4:
+ return INTEL_DRAM_DDR4;
+ case SKL_DRAM_DDR_TYPE_LPDDR3:
+ return INTEL_DRAM_LPDDR3;
+ case SKL_DRAM_DDR_TYPE_LPDDR4:
+ return INTEL_DRAM_LPDDR4;
+ default:
+ MISSING_CASE(val);
+ return INTEL_DRAM_UNKNOWN;
+ }
+}
+
static int
skl_get_dram_info(struct drm_i915_private *dev_priv)
{
u32 mem_freq_khz, val;
int ret;
+ dram_info->type = skl_get_dram_type(dev_priv);
+ DRM_DEBUG_KMS("DRAM type: %s\n", intel_dram_type_str(dram_info->type));
+
ret = skl_dram_get_channels_info(dev_priv);
if (ret)
return ret;
return 0;
}
+/* Returns Gb per DRAM device */
+static int bxt_get_dimm_size(u32 val)
+{
+ switch (val & BXT_DRAM_SIZE_MASK) {
+ case BXT_DRAM_SIZE_4GBIT:
+ return 4;
+ case BXT_DRAM_SIZE_6GBIT:
+ return 6;
+ case BXT_DRAM_SIZE_8GBIT:
+ return 8;
+ case BXT_DRAM_SIZE_12GBIT:
+ return 12;
+ case BXT_DRAM_SIZE_16GBIT:
+ return 16;
+ default:
+ MISSING_CASE(val);
+ return 0;
+ }
+}
+
+static int bxt_get_dimm_width(u32 val)
+{
+ if (!bxt_get_dimm_size(val))
+ return 0;
+
+ val = (val & BXT_DRAM_WIDTH_MASK) >> BXT_DRAM_WIDTH_SHIFT;
+
+ return 8 << val;
+}
+
+static int bxt_get_dimm_ranks(u32 val)
+{
+ if (!bxt_get_dimm_size(val))
+ return 0;
+
+ switch (val & BXT_DRAM_RANK_MASK) {
+ case BXT_DRAM_RANK_SINGLE:
+ return 1;
+ case BXT_DRAM_RANK_DUAL:
+ return 2;
+ default:
+ MISSING_CASE(val);
+ return 0;
+ }
+}
+
+static enum intel_dram_type bxt_get_dimm_type(u32 val)
+{
+ if (!bxt_get_dimm_size(val))
+ return INTEL_DRAM_UNKNOWN;
+
+ switch (val & BXT_DRAM_TYPE_MASK) {
+ case BXT_DRAM_TYPE_DDR3:
+ return INTEL_DRAM_DDR3;
+ case BXT_DRAM_TYPE_LPDDR3:
+ return INTEL_DRAM_LPDDR3;
+ case BXT_DRAM_TYPE_DDR4:
+ return INTEL_DRAM_DDR4;
+ case BXT_DRAM_TYPE_LPDDR4:
+ return INTEL_DRAM_LPDDR4;
+ default:
+ MISSING_CASE(val);
+ return INTEL_DRAM_UNKNOWN;
+ }
+}
+
+static void bxt_get_dimm_info(struct dram_dimm_info *dimm,
+ u32 val)
+{
+ dimm->width = bxt_get_dimm_width(val);
+ dimm->ranks = bxt_get_dimm_ranks(val);
+
+ /*
+ * Size in register is Gb per DRAM device. Convert to total
+ * GB to match the way we report this for non-LP platforms.
+ */
+ dimm->size = bxt_get_dimm_size(val) * intel_dimm_num_devices(dimm) / 8;
+}
+
static int
bxt_get_dram_info(struct drm_i915_private *dev_priv)
{
* Now read each DUNIT8/9/10/11 to check the rank of each dimms.
*/
for (i = BXT_D_CR_DRP0_DUNIT_START; i <= BXT_D_CR_DRP0_DUNIT_END; i++) {
- u8 size, width;
- enum dram_rank rank;
- u32 tmp;
+ struct dram_dimm_info dimm;
+ enum intel_dram_type type;
val = I915_READ(BXT_D_CR_DRP0_DUNIT(i));
if (val == 0xFFFFFFFF)
continue;
dram_info->num_channels++;
- tmp = val & BXT_DRAM_RANK_MASK;
-
- if (tmp == BXT_DRAM_RANK_SINGLE)
- rank = I915_DRAM_RANK_SINGLE;
- else if (tmp == BXT_DRAM_RANK_DUAL)
- rank = I915_DRAM_RANK_DUAL;
- else
- rank = I915_DRAM_RANK_INVALID;
-
- tmp = val & BXT_DRAM_SIZE_MASK;
- if (tmp == BXT_DRAM_SIZE_4GB)
- size = 4;
- else if (tmp == BXT_DRAM_SIZE_6GB)
- size = 6;
- else if (tmp == BXT_DRAM_SIZE_8GB)
- size = 8;
- else if (tmp == BXT_DRAM_SIZE_12GB)
- size = 12;
- else if (tmp == BXT_DRAM_SIZE_16GB)
- size = 16;
- else
- size = 0;
-
- tmp = (val & BXT_DRAM_WIDTH_MASK) >> BXT_DRAM_WIDTH_SHIFT;
- width = (1 << tmp) * 8;
- DRM_DEBUG_KMS("dram size:%dGB width:X%d rank:%s\n", size,
- width, rank == I915_DRAM_RANK_SINGLE ? "single" :
- rank == I915_DRAM_RANK_DUAL ? "dual" : "unknown");
+
+ bxt_get_dimm_info(&dimm, val);
+ type = bxt_get_dimm_type(val);
+
+ WARN_ON(type != INTEL_DRAM_UNKNOWN &&
+ dram_info->type != INTEL_DRAM_UNKNOWN &&
+ dram_info->type != type);
+
+ DRM_DEBUG_KMS("CH%u DIMM size: %u GB, width: X%u, ranks: %u, type: %s\n",
+ i - BXT_D_CR_DRP0_DUNIT_START,
+ dimm.size, dimm.width, dimm.ranks,
+ intel_dram_type_str(type));
/*
* If any of the channel is single rank channel,
* worst case output will be same as if single rank
* memory, so consider single rank memory.
*/
- if (dram_info->rank == I915_DRAM_RANK_INVALID)
- dram_info->rank = rank;
- else if (rank == I915_DRAM_RANK_SINGLE)
- dram_info->rank = I915_DRAM_RANK_SINGLE;
+ if (dram_info->ranks == 0)
+ dram_info->ranks = dimm.ranks;
+ else if (dimm.ranks == 1)
+ dram_info->ranks = 1;
+
+ if (type != INTEL_DRAM_UNKNOWN)
+ dram_info->type = type;
}
- if (dram_info->rank == I915_DRAM_RANK_INVALID) {
- DRM_INFO("couldn't get memory rank information\n");
+ if (dram_info->type == INTEL_DRAM_UNKNOWN ||
+ dram_info->ranks == 0) {
+ DRM_INFO("couldn't get memory information\n");
return -EINVAL;
}
intel_get_dram_info(struct drm_i915_private *dev_priv)
{
struct dram_info *dram_info = &dev_priv->dram_info;
- char bandwidth_str[32];
int ret;
- dram_info->valid = false;
- dram_info->rank = I915_DRAM_RANK_INVALID;
- dram_info->bandwidth_kbps = 0;
- dram_info->num_channels = 0;
-
/*
* Assume 16Gb DIMMs are present until proven otherwise.
* This is only used for the level 0 watermark latency
*/
dram_info->is_16gb_dimm = !IS_GEN9_LP(dev_priv);
- if (INTEL_GEN(dev_priv) < 9 || IS_GEMINILAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) < 9)
return;
- /* Need to calculate bandwidth only for Gen9 */
- if (IS_BROXTON(dev_priv))
+ if (IS_GEN9_LP(dev_priv))
ret = bxt_get_dram_info(dev_priv);
- else if (IS_GEN(dev_priv, 9))
- ret = skl_get_dram_info(dev_priv);
else
- ret = skl_dram_get_channels_info(dev_priv);
+ ret = skl_get_dram_info(dev_priv);
if (ret)
return;
- if (dram_info->bandwidth_kbps)
- sprintf(bandwidth_str, "%d KBps", dram_info->bandwidth_kbps);
+ DRM_DEBUG_KMS("DRAM bandwidth: %u kBps, channels: %u\n",
+ dram_info->bandwidth_kbps,
+ dram_info->num_channels);
+
+ DRM_DEBUG_KMS("DRAM ranks: %u, 16Gb DIMMs: %s\n",
+ dram_info->ranks, yesno(dram_info->is_16gb_dimm));
+}
+
+static u32 gen9_edram_size_mb(struct drm_i915_private *dev_priv, u32 cap)
+{
+ const unsigned int ways[8] = { 4, 8, 12, 16, 16, 16, 16, 16 };
+ const unsigned int sets[4] = { 1, 1, 2, 2 };
+
+ return EDRAM_NUM_BANKS(cap) *
+ ways[EDRAM_WAYS_IDX(cap)] *
+ sets[EDRAM_SETS_IDX(cap)];
+}
+
+static void edram_detect(struct drm_i915_private *dev_priv)
+{
+ u32 edram_cap = 0;
+
+ if (!(IS_HASWELL(dev_priv) ||
+ IS_BROADWELL(dev_priv) ||
+ INTEL_GEN(dev_priv) >= 9))
+ return;
+
+ edram_cap = __raw_uncore_read32(&dev_priv->uncore, HSW_EDRAM_CAP);
+
+ /* NB: We can't write IDICR yet because we don't have gt funcs set up */
+
+ if (!(edram_cap & EDRAM_ENABLED))
+ return;
+
+ /*
+ * The needed capability bits for size calculation are not there with
+ * pre gen9 so return 128MB always.
+ */
+ if (INTEL_GEN(dev_priv) < 9)
+ dev_priv->edram_size_mb = 128;
else
- sprintf(bandwidth_str, "unknown");
- DRM_DEBUG_KMS("DRAM bandwidth:%s, total-channels: %u\n",
- bandwidth_str, dram_info->num_channels);
- DRM_DEBUG_KMS("DRAM rank: %s rank 16GB-dimm:%s\n",
- (dram_info->rank == I915_DRAM_RANK_DUAL) ?
- "dual" : "single", yesno(dram_info->is_16gb_dimm));
+ dev_priv->edram_size_mb =
+ gen9_edram_size_mb(dev_priv, edram_cap);
+
+ DRM_INFO("Found %uMB of eDRAM\n", dev_priv->edram_size_mb);
}
/**
if (HAS_PPGTT(dev_priv)) {
if (intel_vgpu_active(dev_priv) &&
- !intel_vgpu_has_full_48bit_ppgtt(dev_priv)) {
+ !intel_vgpu_has_full_ppgtt(dev_priv)) {
i915_report_error(dev_priv,
"incompatible vGPU found, support for isolated ppGTT required\n");
return -ENXIO;
intel_sanitize_options(dev_priv);
+ /* needs to be done before ggtt probe */
+ edram_detect(dev_priv);
+
i915_perf_init(dev_priv);
ret = i915_ggtt_probe_hw(dev_priv);
if (drm_debug & DRM_UT_DRIVER) {
struct drm_printer p = drm_debug_printer("i915 device info:");
- drm_printf(&p, "pciid=0x%04x rev=0x%02x platform=%s gen=%i\n",
+ drm_printf(&p, "pciid=0x%04x rev=0x%02x platform=%s (subplatform=0x%x) gen=%i\n",
INTEL_DEVID(dev_priv),
INTEL_REVID(dev_priv),
intel_platform_name(INTEL_INFO(dev_priv)->platform),
+ intel_subplatform(RUNTIME_INFO(dev_priv),
+ INTEL_INFO(dev_priv)->platform),
INTEL_GEN(dev_priv));
intel_device_info_dump_flags(INTEL_INFO(dev_priv), &p);
memcpy(device_info, match_info, sizeof(*device_info));
RUNTIME_INFO(i915)->device_id = pdev->device;
- BUILD_BUG_ON(INTEL_MAX_PLATFORMS >
- BITS_PER_TYPE(device_info->platform_mask));
BUG_ON(device_info->gen > BITS_PER_TYPE(device_info->gen_mask));
return i915;
i915_driver_unregister(dev_priv);
+ /*
+ * After unregistering the device to prevent any new users, cancel
+ * all in-flight requests so that we can quickly unbind the active
+ * resources.
+ */
+ i915_gem_set_wedged(dev_priv);
+
/* Flush any external code that still may be under the RCU lock */
synchronize_rcu();
- if (i915_gem_suspend(dev_priv))
- DRM_ERROR("failed to idle hardware; continuing to unload!\n");
+ i915_gem_suspend(dev_priv);
drm_atomic_helper_shutdown(dev);
static int i915_drm_prepare(struct drm_device *dev)
{
struct drm_i915_private *i915 = to_i915(dev);
- int err;
/*
* NB intel_display_suspend() may issue new requests after we've
* split out that work and pull it forward so that after point,
* the GPU is not woken again.
*/
- err = i915_gem_suspend(i915);
- if (err)
- dev_err(&i915->drm.pdev->dev,
- "GEM idle failed, suspend/resume might fail\n");
+ i915_gem_suspend(i915);
- return err;
+ return 0;
}
static int i915_drm_suspend(struct drm_device *dev)
i915_gem_suspend_late(dev_priv);
- intel_uncore_suspend(dev_priv);
+ intel_uncore_suspend(&dev_priv->uncore);
intel_power_domains_suspend(dev_priv,
get_suspend_mode(dev_priv, hibernation));
DRM_ERROR("Resume prepare failed: %d, continuing anyway\n",
ret);
- intel_uncore_resume_early(dev_priv);
+ intel_uncore_resume_early(&dev_priv->uncore);
+
+ i915_check_and_clear_faults(dev_priv);
if (INTEL_GEN(dev_priv) >= 11 || IS_GEN9_LP(dev_priv)) {
gen9_sanitize_dc_state(dev_priv);
if (!force_on)
return 0;
- err = intel_wait_for_register(dev_priv,
+ err = intel_wait_for_register(&dev_priv->uncore,
VLV_GTLC_SURVIVABILITY_REG,
VLV_GFX_CLK_STATUS_BIT,
VLV_GFX_CLK_STATUS_BIT,
intel_runtime_pm_disable_interrupts(dev_priv);
- intel_uncore_suspend(dev_priv);
+ intel_uncore_suspend(&dev_priv->uncore);
ret = 0;
if (INTEL_GEN(dev_priv) >= 11) {
if (ret) {
DRM_ERROR("Runtime suspend failed, disabling it (%d)\n", ret);
- intel_uncore_runtime_resume(dev_priv);
+ intel_uncore_runtime_resume(&dev_priv->uncore);
intel_runtime_pm_enable_interrupts(dev_priv);
enable_rpm_wakeref_asserts(dev_priv);
intel_runtime_pm_cleanup(dev_priv);
- if (intel_uncore_arm_unclaimed_mmio_detection(dev_priv))
+ if (intel_uncore_arm_unclaimed_mmio_detection(&dev_priv->uncore))
DRM_ERROR("Unclaimed access detected prior to suspending\n");
dev_priv->runtime_pm.suspended = true;
intel_opregion_notify_adapter(dev_priv, PCI_D1);
}
- assert_forcewakes_inactive(dev_priv);
+ assert_forcewakes_inactive(&dev_priv->uncore);
if (!IS_VALLEYVIEW(dev_priv) && !IS_CHERRYVIEW(dev_priv))
intel_hpd_poll_init(dev_priv);
intel_opregion_notify_adapter(dev_priv, PCI_D0);
dev_priv->runtime_pm.suspended = false;
- if (intel_uncore_unclaimed_mmio(dev_priv))
+ if (intel_uncore_unclaimed_mmio(&dev_priv->uncore))
DRM_DEBUG_DRIVER("Unclaimed access during suspend, bios?\n");
if (INTEL_GEN(dev_priv) >= 11) {
ret = vlv_resume_prepare(dev_priv, true);
}
- intel_uncore_runtime_resume(dev_priv);
+ intel_uncore_runtime_resume(&dev_priv->uncore);
intel_runtime_pm_enable_interrupts(dev_priv);
DRM_IOCTL_DEF_DRV(I915_SET_SPRITE_COLORKEY, intel_sprite_set_colorkey_ioctl, DRM_MASTER),
DRM_IOCTL_DEF_DRV(I915_GET_SPRITE_COLORKEY, drm_noop, DRM_MASTER),
DRM_IOCTL_DEF_DRV(I915_GEM_WAIT, i915_gem_wait_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
- DRM_IOCTL_DEF_DRV(I915_GEM_CONTEXT_CREATE, i915_gem_context_create_ioctl, DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(I915_GEM_CONTEXT_CREATE_EXT, i915_gem_context_create_ioctl, DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(I915_GEM_CONTEXT_DESTROY, i915_gem_context_destroy_ioctl, DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(I915_REG_READ, i915_reg_read_ioctl, DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(I915_GET_RESET_STATS, i915_gem_context_reset_stats_ioctl, DRM_RENDER_ALLOW),
#include <drm/drm_util.h>
#include <drm/drm_dsc.h>
#include <drm/drm_connector.h>
+#include <drm/i915_mei_hdcp_interface.h>
#include "i915_fixed.h"
#include "i915_params.h"
#include "intel_device_info.h"
#include "intel_display.h"
#include "intel_dpll_mgr.h"
+#include "intel_frontbuffer.h"
#include "intel_lrc.h"
#include "intel_opregion.h"
#include "intel_ringbuffer.h"
+#include "intel_uc.h"
#include "intel_uncore.h"
#include "intel_wopcm.h"
#include "intel_workarounds.h"
-#include "intel_uc.h"
#include "i915_gem.h"
#include "i915_gem_context.h"
#define DRIVER_NAME "i915"
#define DRIVER_DESC "Intel Graphics"
-#define DRIVER_DATE "20190207"
-#define DRIVER_TIMESTAMP 1549572331
+#define DRIVER_DATE "20190417"
+#define DRIVER_TIMESTAMP 1555492067
/* Use I915_STATE_WARN(x) and I915_STATE_WARN_ON() (rather than WARN() and
* WARN_ON()) for hw state sanity checks to check for unexpected conditions
*/
#define DRM_I915_THROTTLE_JIFFIES msecs_to_jiffies(20)
} mm;
+
struct idr context_idr;
+ struct mutex context_idr_lock; /* guards context_idr */
- struct intel_rps_client {
- atomic_t boosts;
- } rps_client;
+ struct idr vm_idr;
+ struct mutex vm_idr_lock; /* guards vm_idr */
unsigned int bsd_engine;
void (*get_cdclk)(struct drm_i915_private *dev_priv,
struct intel_cdclk_state *cdclk_state);
void (*set_cdclk)(struct drm_i915_private *dev_priv,
- const struct intel_cdclk_state *cdclk_state);
+ const struct intel_cdclk_state *cdclk_state,
+ enum pipe pipe);
int (*get_fifo_size)(struct drm_i915_private *dev_priv,
enum i9xx_plane_id i9xx_plane);
int (*compute_pipe_wm)(struct intel_crtc_state *cstate);
/* display clock increase/decrease */
/* pll clock increase/decrease */
+ int (*color_check)(struct intel_crtc_state *crtc_state);
/*
* Program double buffered color management registers during
* vblank evasion. The registers should then latch during the
#define I915_COLOR_UNEVICTABLE (-1) /* a non-vma sharing the address space */
-enum fb_op_origin {
- ORIGIN_GTT,
- ORIGIN_CPU,
- ORIGIN_CS,
- ORIGIN_FLIP,
- ORIGIN_DIRTYFB,
-};
-
struct intel_fbc {
/* This is always the inner lock when overlapping with struct_mutex and
* it's the outer lock when overlapping with stolen_lock. */
u32 debug;
bool sink_support;
- bool prepared, enabled;
+ bool enabled;
struct intel_dp *dp;
enum pipe pipe;
bool active;
u16 su_x_granularity;
};
+/*
+ * Sorted by south display engine compatibility.
+ * If the new PCH comes with a south display engine that is not
+ * inherited from the latest item, please do not add it to the
+ * end. Instead, add it right after its "parent" PCH.
+ */
enum intel_pch {
+ PCH_NOP = -1, /* PCH without south display */
PCH_NONE = 0, /* No PCH present */
PCH_IBX, /* Ibexpeak PCH */
PCH_CPT, /* Cougarpoint/Pantherpoint PCH */
PCH_LPT, /* Lynxpoint/Wildcatpoint PCH */
PCH_SPT, /* Sunrisepoint PCH */
PCH_KBP, /* Kaby Lake PCH */
- PCH_CNP, /* Cannon Lake PCH */
+ PCH_CNP, /* Cannon/Comet Lake PCH */
PCH_ICP, /* Ice Lake PCH */
- PCH_NOP, /* PCH without south display */
};
enum intel_sbi_destination {
#define HDMI_LEVEL_SHIFT_UNKNOWN 0xff
u8 hdmi_level_shift;
+ u8 present:1;
u8 supports_dvi:1;
u8 supports_hdmi:1;
u8 supports_dp:1;
enum psr_lines_to_wait lines_to_wait;
int tp1_wakeup_time_us;
int tp2_tp3_wakeup_time_us;
+ int psr2_tp2_tp3_wakeup_time_us;
} psr;
struct {
u16 plane_res_b;
u8 plane_res_l;
bool plane_en;
+ bool ignore_lines;
};
/* Stores plane specific WM parameters */
INTEL_PIPE_CRC_SOURCE_NONE,
INTEL_PIPE_CRC_SOURCE_PLANE1,
INTEL_PIPE_CRC_SOURCE_PLANE2,
- INTEL_PIPE_CRC_SOURCE_PF,
+ INTEL_PIPE_CRC_SOURCE_PLANE3,
+ INTEL_PIPE_CRC_SOURCE_PLANE4,
+ INTEL_PIPE_CRC_SOURCE_PLANE5,
+ INTEL_PIPE_CRC_SOURCE_PLANE6,
+ INTEL_PIPE_CRC_SOURCE_PLANE7,
INTEL_PIPE_CRC_SOURCE_PIPE,
/* TV/DP on pre-gen5/vlv can't use the pipe source. */
INTEL_PIPE_CRC_SOURCE_TV,
struct drm_i915_private {
struct drm_device drm;
- struct kmem_cache *objects;
- struct kmem_cache *vmas;
- struct kmem_cache *luts;
- struct kmem_cache *requests;
- struct kmem_cache *dependencies;
- struct kmem_cache *priorities;
-
const struct intel_device_info __info; /* Use INTEL_INFO() to access. */
struct intel_runtime_info __runtime; /* Use RUNTIME_INFO() to access. */
struct intel_driver_caps caps;
*/
resource_size_t stolen_usable_size; /* Total size minus reserved ranges */
- void __iomem *regs;
-
struct intel_uncore uncore;
struct i915_virtual_gpu vgpu;
struct intel_cdclk_state actual;
/* The current hardware cdclk state */
struct intel_cdclk_state hw;
+
+ int force_min_cdclk;
} cdclk;
/**
struct intel_l3_parity l3_parity;
- /* Cannot be determined by PCIID. You must always read a register. */
- u32 edram_cap;
+ /*
+ * edram size in MB.
+ * Cannot be determined by PCIID. You must always read a register.
+ */
+ u32 edram_size_mb;
/*
* Protects RPS/RC6 register access and PCU communication.
*
*/
struct mutex av_mutex;
+ int audio_power_refcount;
struct {
struct mutex mutex;
bool valid;
bool is_16gb_dimm;
u8 num_channels;
- enum dram_rank {
- I915_DRAM_RANK_INVALID = 0,
- I915_DRAM_RANK_SINGLE,
- I915_DRAM_RANK_DUAL
- } rank;
+ u8 ranks;
u32 bandwidth_kbps;
bool symmetric_memory;
+ enum intel_dram_type {
+ INTEL_DRAM_UNKNOWN,
+ INTEL_DRAM_DDR3,
+ INTEL_DRAM_DDR4,
+ INTEL_DRAM_LPDDR3,
+ INTEL_DRAM_LPDDR4
+ } type;
} dram_info;
struct i915_runtime_pm runtime_pm;
/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */
struct {
- void (*resume)(struct drm_i915_private *);
void (*cleanup_engine)(struct intel_engine_cs *engine);
struct i915_gt_timelines {
struct list_head hwsp_free_list;
} timelines;
+ intel_engine_mask_t active_engines;
struct list_head active_rings;
struct list_head closed_vma;
u32 active_requests;
*/
intel_wakeref_t awake;
- /**
- * The number of times we have woken up.
- */
- unsigned int epoch;
-#define I915_EPOCH_INVALID 0
-
/**
* We leave the user IRQ off as much as possible,
* but this means that requests will finish and never
struct i915_vma *scratch;
} gt;
+ /* For i945gm vblank irq vs. C3 workaround */
+ struct {
+ struct work_struct work;
+ struct pm_qos_request pm_qos;
+ u8 c3_disable_latency;
+ u8 enabled;
+ } i945gm_vblank;
+
/* perform PHY state sanity checks? */
bool chv_phy_assert[2];
struct i915_pmu pmu;
+ struct i915_hdcp_comp_master *hdcp_master;
+ bool hdcp_comp_added;
+
+ /* Mutex to protect the above hdcp component related values. */
+ struct mutex hdcp_comp_mutex;
+
/*
* NOTE: This is the dri1/ums dungeon, don't add stuff here. Your patch
* will be rejected. Instead look for a better place.
*/
};
+struct dram_dimm_info {
+ u8 size, width, ranks;
+};
+
struct dram_channel_info {
- struct info {
- u8 size, width;
- enum dram_rank rank;
- } l_info, s_info;
- enum dram_rank rank;
+ struct dram_dimm_info dimm_l, dimm_s;
+ u8 ranks;
bool is_16gb_dimm;
};
return container_of(huc, struct drm_i915_private, huc);
}
+static inline struct drm_i915_private *uncore_to_i915(struct intel_uncore *uncore)
+{
+ return container_of(uncore, struct drm_i915_private, uncore);
+}
+
/* Simple iterator over all initialised engines */
#define for_each_engine(engine__, dev_priv__, id__) \
for ((id__) = 0; \
/* Iterator over subset of engines selected by mask */
#define for_each_engine_masked(engine__, dev_priv__, mask__, tmp__) \
- for ((tmp__) = (mask__) & INTEL_INFO(dev_priv__)->ring_mask; \
+ for ((tmp__) = (mask__) & INTEL_INFO(dev_priv__)->engine_mask; \
(tmp__) ? \
((engine__) = (dev_priv__)->engine[__mask_next_bit(tmp__)]), 1 : \
0;)
#define IS_REVID(p, since, until) \
(INTEL_REVID(p) >= (since) && INTEL_REVID(p) <= (until))
-#define IS_PLATFORM(dev_priv, p) (INTEL_INFO(dev_priv)->platform_mask & BIT(p))
+static __always_inline unsigned int
+__platform_mask_index(const struct intel_runtime_info *info,
+ enum intel_platform p)
+{
+ const unsigned int pbits =
+ BITS_PER_TYPE(info->platform_mask[0]) - INTEL_SUBPLATFORM_BITS;
+
+ /* Expand the platform_mask array if this fails. */
+ BUILD_BUG_ON(INTEL_MAX_PLATFORMS >
+ pbits * ARRAY_SIZE(info->platform_mask));
+
+ return p / pbits;
+}
+
+static __always_inline unsigned int
+__platform_mask_bit(const struct intel_runtime_info *info,
+ enum intel_platform p)
+{
+ const unsigned int pbits =
+ BITS_PER_TYPE(info->platform_mask[0]) - INTEL_SUBPLATFORM_BITS;
+
+ return p % pbits + INTEL_SUBPLATFORM_BITS;
+}
+
+static inline u32
+intel_subplatform(const struct intel_runtime_info *info, enum intel_platform p)
+{
+ const unsigned int pi = __platform_mask_index(info, p);
+
+ return info->platform_mask[pi] & INTEL_SUBPLATFORM_BITS;
+}
+
+static __always_inline bool
+IS_PLATFORM(const struct drm_i915_private *i915, enum intel_platform p)
+{
+ const struct intel_runtime_info *info = RUNTIME_INFO(i915);
+ const unsigned int pi = __platform_mask_index(info, p);
+ const unsigned int pb = __platform_mask_bit(info, p);
+
+ BUILD_BUG_ON(!__builtin_constant_p(p));
+
+ return info->platform_mask[pi] & BIT(pb);
+}
+
+static __always_inline bool
+IS_SUBPLATFORM(const struct drm_i915_private *i915,
+ enum intel_platform p, unsigned int s)
+{
+ const struct intel_runtime_info *info = RUNTIME_INFO(i915);
+ const unsigned int pi = __platform_mask_index(info, p);
+ const unsigned int pb = __platform_mask_bit(info, p);
+ const unsigned int msb = BITS_PER_TYPE(info->platform_mask[0]) - 1;
+ const u32 mask = info->platform_mask[pi];
+
+ BUILD_BUG_ON(!__builtin_constant_p(p));
+ BUILD_BUG_ON(!__builtin_constant_p(s));
+ BUILD_BUG_ON((s) >= INTEL_SUBPLATFORM_BITS);
+
+ /* Shift and test on the MSB position so sign flag can be used. */
+ return ((mask << (msb - pb)) & (mask << (msb - s))) & BIT(msb);
+}
+
+#define IS_MOBILE(dev_priv) (INTEL_INFO(dev_priv)->is_mobile)
#define IS_I830(dev_priv) IS_PLATFORM(dev_priv, INTEL_I830)
#define IS_I845G(dev_priv) IS_PLATFORM(dev_priv, INTEL_I845G)
#define IS_G45(dev_priv) IS_PLATFORM(dev_priv, INTEL_G45)
#define IS_GM45(dev_priv) IS_PLATFORM(dev_priv, INTEL_GM45)
#define IS_G4X(dev_priv) (IS_G45(dev_priv) || IS_GM45(dev_priv))
-#define IS_PINEVIEW_G(dev_priv) (INTEL_DEVID(dev_priv) == 0xa001)
-#define IS_PINEVIEW_M(dev_priv) (INTEL_DEVID(dev_priv) == 0xa011)
#define IS_PINEVIEW(dev_priv) IS_PLATFORM(dev_priv, INTEL_PINEVIEW)
#define IS_G33(dev_priv) IS_PLATFORM(dev_priv, INTEL_G33)
-#define IS_IRONLAKE_M(dev_priv) (INTEL_DEVID(dev_priv) == 0x0046)
+#define IS_IRONLAKE(dev_priv) IS_PLATFORM(dev_priv, INTEL_IRONLAKE)
+#define IS_IRONLAKE_M(dev_priv) \
+ (IS_PLATFORM(dev_priv, INTEL_IRONLAKE) && IS_MOBILE(dev_priv))
#define IS_IVYBRIDGE(dev_priv) IS_PLATFORM(dev_priv, INTEL_IVYBRIDGE)
#define IS_IVB_GT1(dev_priv) (IS_IVYBRIDGE(dev_priv) && \
INTEL_INFO(dev_priv)->gt == 1)
#define IS_COFFEELAKE(dev_priv) IS_PLATFORM(dev_priv, INTEL_COFFEELAKE)
#define IS_CANNONLAKE(dev_priv) IS_PLATFORM(dev_priv, INTEL_CANNONLAKE)
#define IS_ICELAKE(dev_priv) IS_PLATFORM(dev_priv, INTEL_ICELAKE)
-#define IS_MOBILE(dev_priv) (INTEL_INFO(dev_priv)->is_mobile)
+#define IS_ELKHARTLAKE(dev_priv) IS_PLATFORM(dev_priv, INTEL_ELKHARTLAKE)
#define IS_HSW_EARLY_SDV(dev_priv) (IS_HASWELL(dev_priv) && \
(INTEL_DEVID(dev_priv) & 0xFF00) == 0x0C00)
-#define IS_BDW_ULT(dev_priv) (IS_BROADWELL(dev_priv) && \
- ((INTEL_DEVID(dev_priv) & 0xf) == 0x6 || \
- (INTEL_DEVID(dev_priv) & 0xf) == 0xb || \
- (INTEL_DEVID(dev_priv) & 0xf) == 0xe))
-/* ULX machines are also considered ULT. */
-#define IS_BDW_ULX(dev_priv) (IS_BROADWELL(dev_priv) && \
- (INTEL_DEVID(dev_priv) & 0xf) == 0xe)
+#define IS_BDW_ULT(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_BROADWELL, INTEL_SUBPLATFORM_ULT)
+#define IS_BDW_ULX(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_BROADWELL, INTEL_SUBPLATFORM_ULX)
#define IS_BDW_GT3(dev_priv) (IS_BROADWELL(dev_priv) && \
INTEL_INFO(dev_priv)->gt == 3)
-#define IS_HSW_ULT(dev_priv) (IS_HASWELL(dev_priv) && \
- (INTEL_DEVID(dev_priv) & 0xFF00) == 0x0A00)
+#define IS_HSW_ULT(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_HASWELL, INTEL_SUBPLATFORM_ULT)
#define IS_HSW_GT3(dev_priv) (IS_HASWELL(dev_priv) && \
INTEL_INFO(dev_priv)->gt == 3)
#define IS_HSW_GT1(dev_priv) (IS_HASWELL(dev_priv) && \
INTEL_INFO(dev_priv)->gt == 1)
/* ULX machines are also considered ULT. */
-#define IS_HSW_ULX(dev_priv) (INTEL_DEVID(dev_priv) == 0x0A0E || \
- INTEL_DEVID(dev_priv) == 0x0A1E)
-#define IS_SKL_ULT(dev_priv) (INTEL_DEVID(dev_priv) == 0x1906 || \
- INTEL_DEVID(dev_priv) == 0x1913 || \
- INTEL_DEVID(dev_priv) == 0x1916 || \
- INTEL_DEVID(dev_priv) == 0x1921 || \
- INTEL_DEVID(dev_priv) == 0x1926)
-#define IS_SKL_ULX(dev_priv) (INTEL_DEVID(dev_priv) == 0x190E || \
- INTEL_DEVID(dev_priv) == 0x1915 || \
- INTEL_DEVID(dev_priv) == 0x191E)
-#define IS_KBL_ULT(dev_priv) (INTEL_DEVID(dev_priv) == 0x5906 || \
- INTEL_DEVID(dev_priv) == 0x5913 || \
- INTEL_DEVID(dev_priv) == 0x5916 || \
- INTEL_DEVID(dev_priv) == 0x5921 || \
- INTEL_DEVID(dev_priv) == 0x5926)
-#define IS_KBL_ULX(dev_priv) (INTEL_DEVID(dev_priv) == 0x590E || \
- INTEL_DEVID(dev_priv) == 0x5915 || \
- INTEL_DEVID(dev_priv) == 0x591E)
-#define IS_AML_ULX(dev_priv) (INTEL_DEVID(dev_priv) == 0x591C || \
- INTEL_DEVID(dev_priv) == 0x87C0 || \
- INTEL_DEVID(dev_priv) == 0x87CA)
+#define IS_HSW_ULX(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_HASWELL, INTEL_SUBPLATFORM_ULX)
+#define IS_SKL_ULT(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_SKYLAKE, INTEL_SUBPLATFORM_ULT)
+#define IS_SKL_ULX(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_SKYLAKE, INTEL_SUBPLATFORM_ULX)
+#define IS_KBL_ULT(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_KABYLAKE, INTEL_SUBPLATFORM_ULT)
+#define IS_KBL_ULX(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_KABYLAKE, INTEL_SUBPLATFORM_ULX)
+#define IS_AML_ULX(dev_priv) \
+ (IS_SUBPLATFORM(dev_priv, INTEL_KABYLAKE, INTEL_SUBPLATFORM_AML) || \
+ IS_SUBPLATFORM(dev_priv, INTEL_COFFEELAKE, INTEL_SUBPLATFORM_AML))
#define IS_SKL_GT2(dev_priv) (IS_SKYLAKE(dev_priv) && \
INTEL_INFO(dev_priv)->gt == 2)
#define IS_SKL_GT3(dev_priv) (IS_SKYLAKE(dev_priv) && \
INTEL_INFO(dev_priv)->gt == 2)
#define IS_KBL_GT3(dev_priv) (IS_KABYLAKE(dev_priv) && \
INTEL_INFO(dev_priv)->gt == 3)
-#define IS_CFL_ULT(dev_priv) (IS_COFFEELAKE(dev_priv) && \
- (INTEL_DEVID(dev_priv) & 0x00F0) == 0x00A0)
+#define IS_CFL_ULT(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_COFFEELAKE, INTEL_SUBPLATFORM_ULT)
#define IS_CFL_GT2(dev_priv) (IS_COFFEELAKE(dev_priv) && \
INTEL_INFO(dev_priv)->gt == 2)
#define IS_CFL_GT3(dev_priv) (IS_COFFEELAKE(dev_priv) && \
INTEL_INFO(dev_priv)->gt == 3)
-#define IS_CNL_WITH_PORT_F(dev_priv) (IS_CANNONLAKE(dev_priv) && \
- (INTEL_DEVID(dev_priv) & 0x0004) == 0x0004)
-#define IS_ICL_WITH_PORT_F(dev_priv) (IS_ICELAKE(dev_priv) && \
- INTEL_DEVID(dev_priv) != 0x8A51)
+#define IS_CNL_WITH_PORT_F(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_CANNONLAKE, INTEL_SUBPLATFORM_PORTF)
+#define IS_ICL_WITH_PORT_F(dev_priv) \
+ IS_SUBPLATFORM(dev_priv, INTEL_ICELAKE, INTEL_SUBPLATFORM_PORTF)
#define IS_ALPHA_SUPPORT(intel_info) ((intel_info)->is_alpha_support)
#define IS_GEN9_LP(dev_priv) (IS_GEN(dev_priv, 9) && IS_LP(dev_priv))
#define IS_GEN9_BC(dev_priv) (IS_GEN(dev_priv, 9) && !IS_LP(dev_priv))
-#define ENGINE_MASK(id) BIT(id)
-#define RENDER_RING ENGINE_MASK(RCS)
-#define BSD_RING ENGINE_MASK(VCS)
-#define BLT_RING ENGINE_MASK(BCS)
-#define VEBOX_RING ENGINE_MASK(VECS)
-#define BSD2_RING ENGINE_MASK(VCS2)
-#define BSD3_RING ENGINE_MASK(VCS3)
-#define BSD4_RING ENGINE_MASK(VCS4)
-#define VEBOX2_RING ENGINE_MASK(VECS2)
-#define ALL_ENGINES (~0)
-
-#define HAS_ENGINE(dev_priv, id) \
- (!!(INTEL_INFO(dev_priv)->ring_mask & ENGINE_MASK(id)))
-
-#define HAS_BSD(dev_priv) HAS_ENGINE(dev_priv, VCS)
-#define HAS_BSD2(dev_priv) HAS_ENGINE(dev_priv, VCS2)
-#define HAS_BLT(dev_priv) HAS_ENGINE(dev_priv, BCS)
-#define HAS_VEBOX(dev_priv) HAS_ENGINE(dev_priv, VECS)
+#define HAS_ENGINE(dev_priv, id) (INTEL_INFO(dev_priv)->engine_mask & BIT(id))
+
+#define ENGINE_INSTANCES_MASK(dev_priv, first, count) ({ \
+ unsigned int first__ = (first); \
+ unsigned int count__ = (count); \
+ (INTEL_INFO(dev_priv)->engine_mask & \
+ GENMASK(first__ + count__ - 1, first__)) >> first__; \
+})
+#define VDBOX_MASK(dev_priv) \
+ ENGINE_INSTANCES_MASK(dev_priv, VCS0, I915_MAX_VCS)
+#define VEBOX_MASK(dev_priv) \
+ ENGINE_INSTANCES_MASK(dev_priv, VECS0, I915_MAX_VECS)
#define HAS_LLC(dev_priv) (INTEL_INFO(dev_priv)->has_llc)
#define HAS_SNOOP(dev_priv) (INTEL_INFO(dev_priv)->has_snoop)
-#define HAS_EDRAM(dev_priv) (!!((dev_priv)->edram_cap & EDRAM_ENABLED))
+#define HAS_EDRAM(dev_priv) ((dev_priv)->edram_size_mb)
#define HAS_WT(dev_priv) ((IS_HASWELL(dev_priv) || \
IS_BROADWELL(dev_priv)) && HAS_EDRAM(dev_priv))
#define HAS_EXECLISTS(dev_priv) HAS_LOGICAL_RING_CONTEXTS(dev_priv)
-#define INTEL_PPGTT(dev_priv) (INTEL_INFO(dev_priv)->ppgtt)
+#define INTEL_PPGTT(dev_priv) (INTEL_INFO(dev_priv)->ppgtt_type)
#define HAS_PPGTT(dev_priv) \
(INTEL_PPGTT(dev_priv) != INTEL_PPGTT_NONE)
#define HAS_FULL_PPGTT(dev_priv) \
(INTEL_PPGTT(dev_priv) >= INTEL_PPGTT_FULL)
-#define HAS_FULL_48BIT_PPGTT(dev_priv) \
- (INTEL_PPGTT(dev_priv) >= INTEL_PPGTT_FULL_4LVL)
#define HAS_PAGE_SIZES(dev_priv, sizes) ({ \
GEM_BUG_ON((sizes) == 0); \
#define HAS_DDI(dev_priv) (INTEL_INFO(dev_priv)->display.has_ddi)
#define HAS_FPGA_DBG_UNCLAIMED(dev_priv) (INTEL_INFO(dev_priv)->has_fpga_dbg)
#define HAS_PSR(dev_priv) (INTEL_INFO(dev_priv)->display.has_psr)
+#define HAS_TRANSCODER_EDP(dev_priv) (INTEL_INFO(dev_priv)->trans_offsets[TRANSCODER_EDP] != 0)
#define HAS_RC6(dev_priv) (INTEL_INFO(dev_priv)->has_rc6)
#define HAS_RC6p(dev_priv) (INTEL_INFO(dev_priv)->has_rc6p)
#define INTEL_PCH_KBP_DEVICE_ID_TYPE 0xA280
#define INTEL_PCH_CNP_DEVICE_ID_TYPE 0xA300
#define INTEL_PCH_CNP_LP_DEVICE_ID_TYPE 0x9D80
+#define INTEL_PCH_CMP_DEVICE_ID_TYPE 0x0280
#define INTEL_PCH_ICP_DEVICE_ID_TYPE 0x3480
#define INTEL_PCH_P2X_DEVICE_ID_TYPE 0x7100
#define INTEL_PCH_P3X_DEVICE_ID_TYPE 0x7000
#define INTEL_PCH_ID(dev_priv) ((dev_priv)->pch_id)
#define HAS_PCH_ICP(dev_priv) (INTEL_PCH_TYPE(dev_priv) == PCH_ICP)
#define HAS_PCH_CNP(dev_priv) (INTEL_PCH_TYPE(dev_priv) == PCH_CNP)
-#define HAS_PCH_CNP_LP(dev_priv) \
- (INTEL_PCH_ID(dev_priv) == INTEL_PCH_CNP_LP_DEVICE_ID_TYPE)
#define HAS_PCH_KBP(dev_priv) (INTEL_PCH_TYPE(dev_priv) == PCH_KBP)
#define HAS_PCH_SPT(dev_priv) (INTEL_PCH_TYPE(dev_priv) == PCH_SPT)
#define HAS_PCH_LPT(dev_priv) (INTEL_PCH_TYPE(dev_priv) == PCH_LPT)
int i915_gem_freeze(struct drm_i915_private *dev_priv);
int i915_gem_freeze_late(struct drm_i915_private *dev_priv);
-void *i915_gem_object_alloc(struct drm_i915_private *dev_priv);
-void i915_gem_object_free(struct drm_i915_gem_object *obj);
void i915_gem_object_init(struct drm_i915_gem_object *obj,
const struct drm_i915_gem_object_ops *ops);
struct drm_i915_gem_object *
int pass = 2;
do {
rcu_barrier();
+ i915_gem_drain_freed_objects(i915);
drain_workqueue(i915->wq);
} while (--pass);
}
void *__must_check i915_gem_object_pin_map(struct drm_i915_gem_object *obj,
enum i915_map_type type);
+void __i915_gem_object_flush_map(struct drm_i915_gem_object *obj,
+ unsigned long offset,
+ unsigned long size);
+static inline void i915_gem_object_flush_map(struct drm_i915_gem_object *obj)
+{
+ __i915_gem_object_flush_map(obj, 0, obj->base.size);
+}
+
/**
* i915_gem_object_unpin_map - releases an earlier mapping
* @obj: the object to unmap
i915_gem_object_unpin_pages(obj);
}
-int __must_check i915_mutex_lock_interruptible(struct drm_device *dev);
+static inline int __must_check
+i915_mutex_lock_interruptible(struct drm_device *dev)
+{
+ return mutex_lock_interruptible(&dev->struct_mutex);
+}
+
int i915_gem_dumb_create(struct drm_file *file_priv,
struct drm_device *dev,
struct drm_mode_create_dumb *args);
int __must_check i915_gem_set_global_seqno(struct drm_device *dev, u32 seqno);
-struct i915_request *
-i915_gem_find_active_request(struct intel_engine_cs *engine);
-
-static inline bool i915_reset_backoff(struct i915_gpu_error *error)
-{
- return unlikely(test_bit(I915_RESET_BACKOFF, &error->flags));
-}
-
-static inline bool i915_terminally_wedged(struct i915_gpu_error *error)
+static inline bool __i915_wedged(struct i915_gpu_error *error)
{
return unlikely(test_bit(I915_WEDGED, &error->flags));
}
-static inline bool i915_reset_backoff_or_wedged(struct i915_gpu_error *error)
+static inline bool i915_reset_failed(struct drm_i915_private *i915)
{
- return i915_reset_backoff(error) | i915_terminally_wedged(error);
+ return __i915_wedged(&i915->gpu_error);
}
static inline u32 i915_reset_count(struct i915_gpu_error *error)
void i915_gem_cleanup_engines(struct drm_i915_private *dev_priv);
int i915_gem_wait_for_idle(struct drm_i915_private *dev_priv,
unsigned int flags, long timeout);
-int __must_check i915_gem_suspend(struct drm_i915_private *dev_priv);
+void i915_gem_suspend(struct drm_i915_private *dev_priv);
void i915_gem_suspend_late(struct drm_i915_private *dev_priv);
void i915_gem_resume(struct drm_i915_private *dev_priv);
vm_fault_t i915_gem_fault(struct vm_fault *vmf);
int i915_gem_object_wait(struct drm_i915_gem_object *obj,
unsigned int flags,
- long timeout,
- struct intel_rps_client *rps);
+ long timeout);
int i915_gem_object_wait_priority(struct drm_i915_gem_object *obj,
unsigned int flags,
const struct i915_sched_attr *attr);
i915_reserve_fence(struct drm_i915_private *dev_priv);
void i915_unreserve_fence(struct drm_i915_fence_reg *fence);
-void i915_gem_revoke_fences(struct drm_i915_private *dev_priv);
void i915_gem_restore_fences(struct drm_i915_private *dev_priv);
void i915_gem_detect_bit_6_swizzle(struct drm_i915_private *dev_priv);
int i915_perf_remove_config_ioctl(struct drm_device *dev, void *data,
struct drm_file *file);
void i915_oa_init_reg_state(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx,
+ struct intel_context *ce,
u32 *reg_state);
/* i915_gem_evict.c */
return DIV_ROUND_UP_ULL(intel_rc6_residency_ns(dev_priv, reg), 1000);
}
-#define I915_READ8(reg) dev_priv->uncore.funcs.mmio_readb(dev_priv, (reg), true)
-#define I915_WRITE8(reg, val) dev_priv->uncore.funcs.mmio_writeb(dev_priv, (reg), (val), true)
+#define __I915_REG_OP(op__, dev_priv__, ...) \
+ intel_uncore_##op__(&(dev_priv__)->uncore, __VA_ARGS__)
+
+#define I915_READ8(reg__) __I915_REG_OP(read8, dev_priv, (reg__))
+#define I915_WRITE8(reg__, val__) __I915_REG_OP(write8, dev_priv, (reg__), (val__))
-#define I915_READ16(reg) dev_priv->uncore.funcs.mmio_readw(dev_priv, (reg), true)
-#define I915_WRITE16(reg, val) dev_priv->uncore.funcs.mmio_writew(dev_priv, (reg), (val), true)
-#define I915_READ16_NOTRACE(reg) dev_priv->uncore.funcs.mmio_readw(dev_priv, (reg), false)
-#define I915_WRITE16_NOTRACE(reg, val) dev_priv->uncore.funcs.mmio_writew(dev_priv, (reg), (val), false)
+#define I915_READ16(reg__) __I915_REG_OP(read16, dev_priv, (reg__))
+#define I915_WRITE16(reg__, val__) __I915_REG_OP(write16, dev_priv, (reg__), (val__))
+#define I915_READ16_NOTRACE(reg__) __I915_REG_OP(read16_notrace, dev_priv, (reg__))
+#define I915_WRITE16_NOTRACE(reg__, val__) __I915_REG_OP(write16_notrace, dev_priv, (reg__), (val__))
-#define I915_READ(reg) dev_priv->uncore.funcs.mmio_readl(dev_priv, (reg), true)
-#define I915_WRITE(reg, val) dev_priv->uncore.funcs.mmio_writel(dev_priv, (reg), (val), true)
-#define I915_READ_NOTRACE(reg) dev_priv->uncore.funcs.mmio_readl(dev_priv, (reg), false)
-#define I915_WRITE_NOTRACE(reg, val) dev_priv->uncore.funcs.mmio_writel(dev_priv, (reg), (val), false)
+#define I915_READ(reg__) __I915_REG_OP(read, dev_priv, (reg__))
+#define I915_WRITE(reg__, val__) __I915_REG_OP(write, dev_priv, (reg__), (val__))
+#define I915_READ_NOTRACE(reg__) __I915_REG_OP(read_notrace, dev_priv, (reg__))
+#define I915_WRITE_NOTRACE(reg__, val__) __I915_REG_OP(write_notrace, dev_priv, (reg__), (val__))
/* Be very careful with read/write 64-bit values. On 32-bit machines, they
* will be implemented using 2 32-bit writes in an arbitrary order with
*
* You have been warned.
*/
-#define I915_READ64(reg) dev_priv->uncore.funcs.mmio_readq(dev_priv, (reg), true)
-
-#define I915_READ64_2x32(lower_reg, upper_reg) ({ \
- u32 upper, lower, old_upper, loop = 0; \
- upper = I915_READ(upper_reg); \
- do { \
- old_upper = upper; \
- lower = I915_READ(lower_reg); \
- upper = I915_READ(upper_reg); \
- } while (upper != old_upper && loop++ < 2); \
- (u64)upper << 32 | lower; })
-
-#define POSTING_READ(reg) (void)I915_READ_NOTRACE(reg)
-#define POSTING_READ16(reg) (void)I915_READ16_NOTRACE(reg)
-
-#define __raw_read(x, s) \
-static inline uint##x##_t __raw_i915_read##x(const struct drm_i915_private *dev_priv, \
- i915_reg_t reg) \
-{ \
- return read##s(dev_priv->regs + i915_mmio_reg_offset(reg)); \
-}
-
-#define __raw_write(x, s) \
-static inline void __raw_i915_write##x(const struct drm_i915_private *dev_priv, \
- i915_reg_t reg, uint##x##_t val) \
-{ \
- write##s(val, dev_priv->regs + i915_mmio_reg_offset(reg)); \
-}
-__raw_read(8, b)
-__raw_read(16, w)
-__raw_read(32, l)
-__raw_read(64, q)
-
-__raw_write(8, b)
-__raw_write(16, w)
-__raw_write(32, l)
-__raw_write(64, q)
+#define I915_READ64(reg__) __I915_REG_OP(read64, dev_priv, (reg__))
+#define I915_READ64_2x32(lower_reg__, upper_reg__) \
+ __I915_REG_OP(read64_2x32, dev_priv, (lower_reg__), (upper_reg__))
-#undef __raw_read
-#undef __raw_write
+#define POSTING_READ(reg__) __I915_REG_OP(posting_read, dev_priv, (reg__))
+#define POSTING_READ16(reg__) __I915_REG_OP(posting_read16, dev_priv, (reg__))
/* These are untraced mmio-accessors that are only valid to be used inside
* critical sections, such as inside IRQ handlers, where forcewake is explicitly
* therefore generally be serialised, by either the dev_priv->uncore.lock or
* a more localised lock guarding all access to that bank of registers.
*/
-#define I915_READ_FW(reg__) __raw_i915_read32(dev_priv, (reg__))
-#define I915_WRITE_FW(reg__, val__) __raw_i915_write32(dev_priv, (reg__), (val__))
-#define I915_WRITE64_FW(reg__, val__) __raw_i915_write64(dev_priv, (reg__), (val__))
-#define POSTING_READ_FW(reg__) (void)I915_READ_FW(reg__)
+#define I915_READ_FW(reg__) __I915_REG_OP(read_fw, dev_priv, (reg__))
+#define I915_WRITE_FW(reg__, val__) __I915_REG_OP(write_fw, dev_priv, (reg__), (val__))
+#define I915_WRITE64_FW(reg__, val__) __I915_REG_OP(write64_fw, dev_priv, (reg__), (val__))
+#define POSTING_READ_FW(reg__) __I915_REG_OP(posting_read_fw, dev_priv, (reg__))
/* "Broadcast RGB" property */
#define INTEL_BROADCAST_RGB_AUTO 0
#include "i915_drv.h"
#include "i915_gem_clflush.h"
#include "i915_gemfs.h"
+#include "i915_globals.h"
#include "i915_reset.h"
#include "i915_trace.h"
#include "i915_vgpu.h"
#include "intel_drv.h"
#include "intel_frontbuffer.h"
#include "intel_mocs.h"
+#include "intel_pm.h"
#include "intel_workarounds.h"
static void i915_gem_flush_free_objects(struct drm_i915_private *i915);
spin_unlock(&dev_priv->mm.object_stat_lock);
}
-static int
-i915_gem_wait_for_error(struct i915_gpu_error *error)
-{
- int ret;
-
- might_sleep();
-
- /*
- * Only wait 10 seconds for the gpu reset to complete to avoid hanging
- * userspace. If it takes that long something really bad is going on and
- * we should simply try to bail out and fail as gracefully as possible.
- */
- ret = wait_event_interruptible_timeout(error->reset_queue,
- !i915_reset_backoff(error),
- I915_RESET_TIMEOUT);
- if (ret == 0) {
- DRM_ERROR("Timed out waiting for the gpu reset to complete\n");
- return -EIO;
- } else if (ret < 0) {
- return ret;
- } else {
- return 0;
- }
-}
-
-int i915_mutex_lock_interruptible(struct drm_device *dev)
-{
- struct drm_i915_private *dev_priv = to_i915(dev);
- int ret;
-
- ret = i915_gem_wait_for_error(&dev_priv->gpu_error);
- if (ret)
- return ret;
-
- ret = mutex_lock_interruptible(&dev->struct_mutex);
- if (ret)
- return ret;
-
- return 0;
-}
-
-static u32 __i915_gem_park(struct drm_i915_private *i915)
+static void __i915_gem_park(struct drm_i915_private *i915)
{
intel_wakeref_t wakeref;
GEM_BUG_ON(!list_empty(&i915->gt.active_rings));
if (!i915->gt.awake)
- return I915_EPOCH_INVALID;
-
- GEM_BUG_ON(i915->gt.epoch == I915_EPOCH_INVALID);
+ return;
/*
* Be paranoid and flush a concurrent interrupt to make sure
intel_display_power_put(i915, POWER_DOMAIN_GT_IRQ, wakeref);
- return i915->gt.epoch;
+ i915_globals_park();
}
void i915_gem_park(struct drm_i915_private *i915)
i915->gt.awake = intel_display_power_get(i915, POWER_DOMAIN_GT_IRQ);
GEM_BUG_ON(!i915->gt.awake);
- if (unlikely(++i915->gt.epoch == 0)) /* keep 0 as invalid */
- i915->gt.epoch = 1;
+ i915_globals_unpark();
intel_enable_gt_powersave(i915);
i915_update_gfx_val(i915);
obj->cache_dirty = true;
}
-static void
+void
__i915_gem_object_release_shmem(struct drm_i915_gem_object *obj,
struct sg_table *pages,
bool needs_clflush)
static long
i915_gem_object_wait_fence(struct dma_fence *fence,
unsigned int flags,
- long timeout,
- struct intel_rps_client *rps_client)
+ long timeout)
{
struct i915_request *rq;
if (i915_request_completed(rq))
goto out;
- /*
- * This client is about to stall waiting for the GPU. In many cases
- * this is undesirable and limits the throughput of the system, as
- * many clients cannot continue processing user input/output whilst
- * blocked. RPS autotuning may take tens of milliseconds to respond
- * to the GPU load and thus incurs additional latency for the client.
- * We can circumvent that by promoting the GPU frequency to maximum
- * before we wait. This makes the GPU throttle up much more quickly
- * (good for benchmarks and user experience, e.g. window animations),
- * but at a cost of spending more power processing the workload
- * (bad for battery). Not all clients even want their results
- * immediately and for them we should just let the GPU select its own
- * frequency to maximise efficiency. To prevent a single client from
- * forcing the clocks too high for the whole system, we only allow
- * each client to waitboost once in a busy period.
- */
- if (rps_client && !i915_request_started(rq)) {
- if (INTEL_GEN(rq->i915) >= 6)
- gen6_rps_boost(rq, rps_client);
- }
-
timeout = i915_request_wait(rq, flags, timeout);
out:
static long
i915_gem_object_wait_reservation(struct reservation_object *resv,
unsigned int flags,
- long timeout,
- struct intel_rps_client *rps_client)
+ long timeout)
{
unsigned int seq = __read_seqcount_begin(&resv->seq);
struct dma_fence *excl;
for (i = 0; i < count; i++) {
timeout = i915_gem_object_wait_fence(shared[i],
- flags, timeout,
- rps_client);
+ flags, timeout);
if (timeout < 0)
break;
}
if (excl && timeout >= 0)
- timeout = i915_gem_object_wait_fence(excl, flags, timeout,
- rps_client);
+ timeout = i915_gem_object_wait_fence(excl, flags, timeout);
dma_fence_put(excl);
* @obj: i915 gem object
* @flags: how to wait (under a lock, for all rendering or just for writes etc)
* @timeout: how long to wait
- * @rps_client: client (user process) to charge for any waitboosting
*/
int
i915_gem_object_wait(struct drm_i915_gem_object *obj,
unsigned int flags,
- long timeout,
- struct intel_rps_client *rps_client)
+ long timeout)
{
might_sleep();
GEM_BUG_ON(timeout < 0);
- timeout = i915_gem_object_wait_reservation(obj->resv,
- flags, timeout,
- rps_client);
+ timeout = i915_gem_object_wait_reservation(obj->resv, flags, timeout);
return timeout < 0 ? timeout : 0;
}
-static struct intel_rps_client *to_rps_client(struct drm_file *file)
-{
- struct drm_i915_file_private *fpriv = file->driver_priv;
-
- return &fpriv->rps_client;
-}
-
static int
i915_gem_phys_pwrite(struct drm_i915_gem_object *obj,
struct drm_i915_gem_pwrite *args,
return 0;
}
-void *i915_gem_object_alloc(struct drm_i915_private *dev_priv)
-{
- return kmem_cache_zalloc(dev_priv->objects, GFP_KERNEL);
-}
-
-void i915_gem_object_free(struct drm_i915_gem_object *obj)
-{
- struct drm_i915_private *dev_priv = to_i915(obj->base.dev);
- kmem_cache_free(dev_priv->objects, obj);
-}
-
static int
i915_gem_create(struct drm_file *file,
struct drm_i915_private *dev_priv,
- u64 size,
+ u64 *size_p,
u32 *handle_p)
{
struct drm_i915_gem_object *obj;
- int ret;
u32 handle;
+ u64 size;
+ int ret;
- size = roundup(size, PAGE_SIZE);
+ size = round_up(*size_p, PAGE_SIZE);
if (size == 0)
return -EINVAL;
return ret;
*handle_p = handle;
+ *size_p = size;
return 0;
}
args->pitch = ALIGN(args->width * DIV_ROUND_UP(args->bpp, 8), 64);
args->size = args->pitch * args->height;
return i915_gem_create(file, to_i915(dev),
- args->size, &args->handle);
+ &args->size, &args->handle);
}
static bool gpu_write_needs_clflush(struct drm_i915_gem_object *obj)
i915_gem_flush_free_objects(dev_priv);
return i915_gem_create(file, dev_priv,
- args->size, &args->handle);
+ &args->size, &args->handle);
}
static inline enum fb_op_origin
ret = i915_gem_object_wait(obj,
I915_WAIT_INTERRUPTIBLE |
I915_WAIT_LOCKED,
- MAX_SCHEDULE_TIMEOUT,
- NULL);
+ MAX_SCHEDULE_TIMEOUT);
if (ret)
return ret;
I915_WAIT_INTERRUPTIBLE |
I915_WAIT_LOCKED |
I915_WAIT_ALL,
- MAX_SCHEDULE_TIMEOUT,
- NULL);
+ MAX_SCHEDULE_TIMEOUT);
if (ret)
return ret;
ret = i915_gem_object_wait(obj,
I915_WAIT_INTERRUPTIBLE,
- MAX_SCHEDULE_TIMEOUT,
- to_rps_client(file));
+ MAX_SCHEDULE_TIMEOUT);
if (ret)
goto out;
ret = i915_gem_object_wait(obj,
I915_WAIT_INTERRUPTIBLE |
I915_WAIT_ALL,
- MAX_SCHEDULE_TIMEOUT,
- to_rps_client(file));
+ MAX_SCHEDULE_TIMEOUT);
if (ret)
goto err;
if ((write_domain | read_domains) & I915_GEM_GPU_DOMAINS)
return -EINVAL;
- /* Having something in the write domain implies it's in the read
+ /*
+ * Having something in the write domain implies it's in the read
* domain, and only that read domain. Enforce that in the request.
*/
- if (write_domain != 0 && read_domains != write_domain)
+ if (write_domain && read_domains != write_domain)
return -EINVAL;
+ if (!read_domains)
+ return 0;
+
obj = i915_gem_object_lookup(file, args->handle);
if (!obj)
return -ENOENT;
- /* Try to flush the object off the GPU without holding the lock.
+ /*
+ * Already in the desired write domain? Nothing for us to do!
+ *
+ * We apply a little bit of cunning here to catch a broader set of
+ * no-ops. If obj->write_domain is set, we must be in the same
+ * obj->read_domains, and only that domain. Therefore, if that
+ * obj->write_domain matches the request read_domains, we are
+ * already in the same read/write domain and can skip the operation,
+ * without having to further check the requested write_domain.
+ */
+ if (READ_ONCE(obj->write_domain) == read_domains) {
+ err = 0;
+ goto out;
+ }
+
+ /*
+ * Try to flush the object off the GPU without holding the lock.
* We will repeat the flush holding the lock in the normal manner
* to catch cases where we are gazumped.
*/
I915_WAIT_INTERRUPTIBLE |
I915_WAIT_PRIORITY |
(write_domain ? I915_WAIT_ALL : 0),
- MAX_SCHEDULE_TIMEOUT,
- to_rps_client(file));
+ MAX_SCHEDULE_TIMEOUT);
if (err)
goto out;
* 2 - Recognise WC as a separate cache domain so that we can flush the
* delayed writes via GTT before performing direct access via WC.
*
+ * 3 - Remove implicit set-domain(GTT) and synchronisation on initial
+ * pagefault; swapin remains transparent.
+ *
* Restrictions:
*
* * snoopable objects cannot be accessed via the GTT. It can cause machine
*/
int i915_gem_mmap_gtt_version(void)
{
- return 2;
+ return 3;
}
static inline struct i915_ggtt_view
intel_wakeref_t wakeref;
struct i915_vma *vma;
pgoff_t page_offset;
+ int srcu;
int ret;
/* Sanity check that we allow writing into this object */
trace_i915_gem_object_fault(obj, page_offset, true, write);
- /* Try to flush the object off the GPU first without holding the lock.
- * Upon acquiring the lock, we will perform our sanity checks and then
- * repeat the flush holding the lock in the normal manner to catch cases
- * where we are gazumped.
- */
- ret = i915_gem_object_wait(obj,
- I915_WAIT_INTERRUPTIBLE,
- MAX_SCHEDULE_TIMEOUT,
- NULL);
- if (ret)
- goto err;
-
ret = i915_gem_object_pin_pages(obj);
if (ret)
goto err;
wakeref = intel_runtime_pm_get(dev_priv);
+ srcu = i915_reset_trylock(dev_priv);
+ if (srcu < 0) {
+ ret = srcu;
+ goto err_rpm;
+ }
+
ret = i915_mutex_lock_interruptible(dev);
if (ret)
- goto err_rpm;
+ goto err_reset;
/* Access to snoopable pages through the GTT is incoherent. */
if (obj->cache_level != I915_CACHE_NONE && !HAS_LLC(dev_priv)) {
goto err_unlock;
}
-
/* Now pin it into the GTT as needed */
vma = i915_gem_object_ggtt_pin(obj, NULL, 0, 0,
PIN_MAPPABLE |
goto err_unlock;
}
- ret = i915_gem_object_set_to_gtt_domain(obj, write);
- if (ret)
- goto err_unpin;
-
ret = i915_vma_pin_fence(vma);
if (ret)
goto err_unpin;
__i915_vma_unpin(vma);
err_unlock:
mutex_unlock(&dev->struct_mutex);
+err_reset:
+ i915_reset_unlock(dev_priv, srcu);
err_rpm:
intel_runtime_pm_put(dev_priv, wakeref);
i915_gem_object_unpin_pages(obj);
* fail). But any other -EIO isn't ours (e.g. swap in failure)
* and so needs to be reported.
*/
- if (!i915_terminally_wedged(&dev_priv->gpu_error))
+ if (!i915_terminally_wedged(dev_priv))
return VM_FAULT_SIGBUS;
/* else: fall through */
case -EAGAIN:
struct page *page;
__i915_gem_object_release_shmem(obj, pages, true);
-
i915_gem_gtt_finish_pages(obj, pages);
if (i915_gem_object_needs_bit17_swizzle(obj))
do {
cond_resched();
page = shmem_read_mapping_page_gfp(mapping, i, gfp);
- if (likely(!IS_ERR(page)))
+ if (!IS_ERR(page))
break;
if (!*s) {
lockdep_assert_held(&obj->mm.lock);
+ /* Make the pages coherent with the GPU (flushing any swapin). */
+ if (obj->cache_dirty) {
+ obj->write_domain = 0;
+ if (i915_gem_object_has_struct_page(obj))
+ drm_clflush_sg(pages);
+ obj->cache_dirty = false;
+ }
+
obj->mm.get_page.sg_pos = pages->sgl;
obj->mm.get_page.sg_idx = 0;
goto out_unlock;
}
+void __i915_gem_object_flush_map(struct drm_i915_gem_object *obj,
+ unsigned long offset,
+ unsigned long size)
+{
+ enum i915_map_type has_type;
+ void *ptr;
+
+ GEM_BUG_ON(!i915_gem_object_has_pinned_pages(obj));
+ GEM_BUG_ON(range_overflows_t(typeof(obj->base.size),
+ offset, size, obj->base.size));
+
+ obj->mm.dirty = true;
+
+ if (obj->cache_coherent & I915_BO_CACHE_COHERENT_FOR_WRITE)
+ return;
+
+ ptr = page_unpack_bits(obj->mm.mapping, &has_type);
+ if (has_type == I915_MAP_WC)
+ return;
+
+ drm_clflush_virt_range(ptr + offset, size);
+ if (size == obj->base.size) {
+ obj->write_domain &= ~I915_GEM_DOMAIN_CPU;
+ obj->cache_dirty = false;
+ }
+}
+
static int
i915_gem_object_pwrite_gtt(struct drm_i915_gem_object *obj,
const struct drm_i915_gem_pwrite *arg)
u64 remain, offset;
unsigned int pg;
- /* Before we instantiate/pin the backing store for our use, we
+ /* Caller already validated user args */
+ GEM_BUG_ON(!access_ok(user_data, arg->size));
+
+ /*
+ * Before we instantiate/pin the backing store for our use, we
* can prepopulate the shmemfs filp efficiently using a write into
* the pagecache. We avoid the penalty of instantiating all the
* pages, important if the user is just writing to a few and never
if (obj->mm.madv != I915_MADV_WILLNEED)
return -EFAULT;
- /* Before the pages are instantiated the object is treated as being
+ /*
+ * Before the pages are instantiated the object is treated as being
* in the CPU domain. The pages will be clflushed as required before
* use, and we can freely write into the pages directly. If userspace
* races pwrite with any other operation; corruption will ensue -
struct page *page;
void *data, *vaddr;
int err;
+ char c;
len = PAGE_SIZE - pg;
if (len > remain)
len = remain;
+ /* Prefault the user page to reduce potential recursion */
+ err = __get_user(c, user_data);
+ if (err)
+ return err;
+
+ err = __get_user(c, user_data + len - 1);
+ if (err)
+ return err;
+
err = pagecache_write_begin(obj->base.filp, mapping,
offset, len, 0,
&page, &data);
if (err < 0)
return err;
- vaddr = kmap(page);
- unwritten = copy_from_user(vaddr + pg, user_data, len);
- kunmap(page);
+ vaddr = kmap_atomic(page);
+ unwritten = __copy_from_user_inatomic(vaddr + pg,
+ user_data,
+ len);
+ kunmap_atomic(vaddr);
err = pagecache_write_end(obj->base.filp, mapping,
offset, len, len - unwritten,
if (err < 0)
return err;
+ /* We don't handle -EFAULT, leave it to the caller to check */
if (unwritten)
- return -EFAULT;
+ return -ENODEV;
remain -= len;
user_data += len;
return 0;
}
-static bool match_ring(struct i915_request *rq)
-{
- struct drm_i915_private *dev_priv = rq->i915;
- u32 ring = I915_READ(RING_START(rq->engine->mmio_base));
-
- return ring == i915_ggtt_offset(rq->ring->vma);
-}
-
-struct i915_request *
-i915_gem_find_active_request(struct intel_engine_cs *engine)
-{
- struct i915_request *request, *active = NULL;
- unsigned long flags;
-
- /*
- * We are called by the error capture, reset and to dump engine
- * state at random points in time. In particular, note that neither is
- * crucially ordered with an interrupt. After a hang, the GPU is dead
- * and we assume that no more writes can happen (we waited long enough
- * for all writes that were in transaction to be flushed) - adding an
- * extra delay for a recent interrupt is pointless. Hence, we do
- * not need an engine->irq_seqno_barrier() before the seqno reads.
- * At all other times, we must assume the GPU is still running, but
- * we only care about the snapshot of this moment.
- */
- spin_lock_irqsave(&engine->timeline.lock, flags);
- list_for_each_entry(request, &engine->timeline.requests, link) {
- if (i915_request_completed(request))
- continue;
-
- if (!i915_request_started(request))
- break;
-
- /* More than one preemptible request may match! */
- if (!match_ring(request))
- break;
-
- active = request;
- break;
- }
- spin_unlock_irqrestore(&engine->timeline.lock, flags);
-
- return active;
-}
-
static void
i915_gem_retire_work_handler(struct work_struct *work)
{
round_jiffies_up_relative(HZ));
}
-static void shrink_caches(struct drm_i915_private *i915)
+static bool switch_to_kernel_context_sync(struct drm_i915_private *i915,
+ unsigned long mask)
{
- /*
- * kmem_cache_shrink() discards empty slabs and reorders partially
- * filled slabs to prioritise allocating from the mostly full slabs,
- * with the aim of reducing fragmentation.
- */
- kmem_cache_shrink(i915->priorities);
- kmem_cache_shrink(i915->dependencies);
- kmem_cache_shrink(i915->requests);
- kmem_cache_shrink(i915->luts);
- kmem_cache_shrink(i915->vmas);
- kmem_cache_shrink(i915->objects);
-}
-
-struct sleep_rcu_work {
- union {
- struct rcu_head rcu;
- struct work_struct work;
- };
- struct drm_i915_private *i915;
- unsigned int epoch;
-};
+ bool result = true;
-static inline bool
-same_epoch(struct drm_i915_private *i915, unsigned int epoch)
-{
/*
- * There is a small chance that the epoch wrapped since we started
- * sleeping. If we assume that epoch is at least a u32, then it will
- * take at least 2^32 * 100ms for it to wrap, or about 326 years.
+ * Even if we fail to switch, give whatever is running a small chance
+ * to save itself before we report the failure. Yes, this may be a
+ * false positive due to e.g. ENOMEM, caveat emptor!
*/
- return epoch == READ_ONCE(i915->gt.epoch);
-}
-
-static void __sleep_work(struct work_struct *work)
-{
- struct sleep_rcu_work *s = container_of(work, typeof(*s), work);
- struct drm_i915_private *i915 = s->i915;
- unsigned int epoch = s->epoch;
-
- kfree(s);
- if (same_epoch(i915, epoch))
- shrink_caches(i915);
-}
+ if (i915_gem_switch_to_kernel_context(i915, mask))
+ result = false;
-static void __sleep_rcu(struct rcu_head *rcu)
-{
- struct sleep_rcu_work *s = container_of(rcu, typeof(*s), rcu);
- struct drm_i915_private *i915 = s->i915;
-
- destroy_rcu_head(&s->rcu);
+ if (i915_gem_wait_for_idle(i915,
+ I915_WAIT_LOCKED |
+ I915_WAIT_FOR_IDLE_BOOST,
+ I915_GEM_IDLE_TIMEOUT))
+ result = false;
+
+ if (!result) {
+ if (i915_modparams.reset) { /* XXX hide warning from gem_eio */
+ dev_err(i915->drm.dev,
+ "Failed to idle engines, declaring wedged!\n");
+ GEM_TRACE_DUMP();
+ }
- if (same_epoch(i915, s->epoch)) {
- INIT_WORK(&s->work, __sleep_work);
- queue_work(i915->wq, &s->work);
- } else {
- kfree(s);
+ /* Forcibly cancel outstanding work and leave the gpu quiet. */
+ i915_gem_set_wedged(i915);
}
-}
-static inline bool
-new_requests_since_last_retire(const struct drm_i915_private *i915)
-{
- return (READ_ONCE(i915->gt.active_requests) ||
- work_pending(&i915->gt.idle_work.work));
+ i915_retire_requests(i915); /* ensure we flush after wedging */
+ return result;
}
-static void assert_kernel_context_is_current(struct drm_i915_private *i915)
+static bool load_power_context(struct drm_i915_private *i915)
{
- struct intel_engine_cs *engine;
- enum intel_engine_id id;
+ /* Force loading the kernel context on all engines */
+ if (!switch_to_kernel_context_sync(i915, ALL_ENGINES))
+ return false;
- if (i915_terminally_wedged(&i915->gpu_error))
- return;
+ /*
+ * Immediately park the GPU so that we enable powersaving and
+ * treat it as idle. The next time we issue a request, we will
+ * unpark and start using the engine->pinned_default_state, otherwise
+ * it is in limbo and an early reset may fail.
+ */
+ __i915_gem_park(i915);
- GEM_BUG_ON(i915->gt.active_requests);
- for_each_engine(engine, i915, id) {
- GEM_BUG_ON(__i915_active_request_peek(&engine->timeline.last_request));
- GEM_BUG_ON(engine->last_retired_context !=
- to_intel_context(i915->kernel_context, engine));
- }
+ return true;
}
static void
i915_gem_idle_work_handler(struct work_struct *work)
{
- struct drm_i915_private *dev_priv =
- container_of(work, typeof(*dev_priv), gt.idle_work.work);
- unsigned int epoch = I915_EPOCH_INVALID;
+ struct drm_i915_private *i915 =
+ container_of(work, typeof(*i915), gt.idle_work.work);
bool rearm_hangcheck;
- if (!READ_ONCE(dev_priv->gt.awake))
+ if (!READ_ONCE(i915->gt.awake))
return;
- if (READ_ONCE(dev_priv->gt.active_requests))
+ if (READ_ONCE(i915->gt.active_requests))
return;
- /*
- * Flush out the last user context, leaving only the pinned
- * kernel context resident. When we are idling on the kernel_context,
- * no more new requests (with a context switch) are emitted and we
- * can finally rest. A consequence is that the idle work handler is
- * always called at least twice before idling (and if the system is
- * idle that implies a round trip through the retire worker).
- */
- mutex_lock(&dev_priv->drm.struct_mutex);
- i915_gem_switch_to_kernel_context(dev_priv);
- mutex_unlock(&dev_priv->drm.struct_mutex);
-
- GEM_TRACE("active_requests=%d (after switch-to-kernel-context)\n",
- READ_ONCE(dev_priv->gt.active_requests));
-
- /*
- * Wait for last execlists context complete, but bail out in case a
- * new request is submitted. As we don't trust the hardware, we
- * continue on if the wait times out. This is necessary to allow
- * the machine to suspend even if the hardware dies, and we will
- * try to recover in resume (after depriving the hardware of power,
- * it may be in a better mmod).
- */
- __wait_for(if (new_requests_since_last_retire(dev_priv)) return,
- intel_engines_are_idle(dev_priv),
- I915_IDLE_ENGINES_TIMEOUT * 1000,
- 10, 500);
-
rearm_hangcheck =
- cancel_delayed_work_sync(&dev_priv->gpu_error.hangcheck_work);
+ cancel_delayed_work_sync(&i915->gpu_error.hangcheck_work);
- if (!mutex_trylock(&dev_priv->drm.struct_mutex)) {
+ if (!mutex_trylock(&i915->drm.struct_mutex)) {
/* Currently busy, come back later */
- mod_delayed_work(dev_priv->wq,
- &dev_priv->gt.idle_work,
+ mod_delayed_work(i915->wq,
+ &i915->gt.idle_work,
msecs_to_jiffies(50));
goto out_rearm;
}
/*
- * New request retired after this work handler started, extend active
- * period until next instance of the work.
+ * Flush out the last user context, leaving only the pinned
+ * kernel context resident. Should anything unfortunate happen
+ * while we are idle (such as the GPU being power cycled), no users
+ * will be harmed.
*/
- if (new_requests_since_last_retire(dev_priv))
- goto out_unlock;
+ if (!work_pending(&i915->gt.idle_work.work) &&
+ !i915->gt.active_requests) {
+ ++i915->gt.active_requests; /* don't requeue idle */
- epoch = __i915_gem_park(dev_priv);
+ switch_to_kernel_context_sync(i915, i915->gt.active_engines);
- assert_kernel_context_is_current(dev_priv);
+ if (!--i915->gt.active_requests) {
+ __i915_gem_park(i915);
+ rearm_hangcheck = false;
+ }
+ }
- rearm_hangcheck = false;
-out_unlock:
- mutex_unlock(&dev_priv->drm.struct_mutex);
+ mutex_unlock(&i915->drm.struct_mutex);
out_rearm:
if (rearm_hangcheck) {
- GEM_BUG_ON(!dev_priv->gt.awake);
- i915_queue_hangcheck(dev_priv);
- }
-
- /*
- * When we are idle, it is an opportune time to reap our caches.
- * However, we have many objects that utilise RCU and the ordered
- * i915->wq that this work is executing on. To try and flush any
- * pending frees now we are idle, we first wait for an RCU grace
- * period, and then queue a task (that will run last on the wq) to
- * shrink and re-optimize the caches.
- */
- if (same_epoch(dev_priv, epoch)) {
- struct sleep_rcu_work *s = kmalloc(sizeof(*s), GFP_KERNEL);
- if (s) {
- init_rcu_head(&s->rcu);
- s->i915 = dev_priv;
- s->epoch = epoch;
- call_rcu(&s->rcu, __sleep_rcu);
- }
+ GEM_BUG_ON(!i915->gt.awake);
+ i915_queue_hangcheck(i915);
}
}
list_del(&lut->obj_link);
list_del(&lut->ctx_link);
- kmem_cache_free(i915->luts, lut);
+ i915_lut_handle_free(lut);
__i915_gem_object_release_unless_active(obj);
}
I915_WAIT_INTERRUPTIBLE |
I915_WAIT_PRIORITY |
I915_WAIT_ALL,
- to_wait_timeout(args->timeout_ns),
- to_rps_client(file));
+ to_wait_timeout(args->timeout_ns));
if (args->timeout_ns > 0) {
args->timeout_ns -= ktime_to_ns(ktime_sub(ktime_get(), start));
* stalls, so allow the gpu to boost to maximum clocks.
*/
if (flags & I915_WAIT_FOR_IDLE_BOOST)
- gen6_rps_boost(rq, NULL);
+ gen6_rps_boost(rq);
timeout = i915_request_wait(rq, flags, timeout);
i915_request_put(rq);
lockdep_assert_held(&i915->drm.struct_mutex);
- if (GEM_SHOW_DEBUG() && !timeout) {
- /* Presume that timeout was non-zero to begin with! */
- dev_warn(&i915->drm.pdev->dev,
- "Missed idle-completion interrupt!\n");
- GEM_TRACE_DUMP();
- }
-
err = wait_for_engines(i915);
if (err)
return err;
i915_retire_requests(i915);
- GEM_BUG_ON(i915->gt.active_requests);
}
return 0;
I915_WAIT_INTERRUPTIBLE |
I915_WAIT_LOCKED |
(write ? I915_WAIT_ALL : 0),
- MAX_SCHEDULE_TIMEOUT,
- NULL);
+ MAX_SCHEDULE_TIMEOUT);
if (ret)
return ret;
I915_WAIT_INTERRUPTIBLE |
I915_WAIT_LOCKED |
(write ? I915_WAIT_ALL : 0),
- MAX_SCHEDULE_TIMEOUT,
- NULL);
+ MAX_SCHEDULE_TIMEOUT);
if (ret)
return ret;
I915_WAIT_INTERRUPTIBLE |
I915_WAIT_LOCKED |
I915_WAIT_ALL,
- MAX_SCHEDULE_TIMEOUT,
- NULL);
+ MAX_SCHEDULE_TIMEOUT);
if (ret)
return ret;
ret = i915_gem_object_wait(obj,
I915_WAIT_INTERRUPTIBLE,
- MAX_SCHEDULE_TIMEOUT,
- to_rps_client(file));
+ MAX_SCHEDULE_TIMEOUT);
if (ret)
goto out;
I915_WAIT_INTERRUPTIBLE |
I915_WAIT_LOCKED |
(write ? I915_WAIT_ALL : 0),
- MAX_SCHEDULE_TIMEOUT,
- NULL);
+ MAX_SCHEDULE_TIMEOUT);
if (ret)
return ret;
long ret;
/* ABI: return -EIO if already wedged */
- if (i915_terminally_wedged(&dev_priv->gpu_error))
- return -EIO;
+ ret = i915_terminally_wedged(dev_priv);
+ if (ret)
+ return ret;
spin_lock(&file_priv->mm.lock);
list_for_each_entry(request, &file_priv->mm.request_list, client_link) {
}
vma = i915_vma_instance(obj, vm, view);
- if (unlikely(IS_ERR(vma)))
+ if (IS_ERR(vma))
return vma;
if (i915_vma_misplaced(vma, size, alignment, flags)) {
return vma;
}
-static __always_inline unsigned int __busy_read_flag(unsigned int id)
+static __always_inline u32 __busy_read_flag(u8 id)
{
- /* Note that we could alias engines in the execbuf API, but
- * that would be very unwise as it prevents userspace from
- * fine control over engine selection. Ahem.
- *
- * This should be something like EXEC_MAX_ENGINE instead of
- * I915_NUM_ENGINES.
- */
- BUILD_BUG_ON(I915_NUM_ENGINES > 16);
- return 0x10000 << id;
+ if (id == (u8)I915_ENGINE_CLASS_INVALID)
+ return 0xffff0000u;
+
+ GEM_BUG_ON(id >= 16);
+ return 0x10000u << id;
}
-static __always_inline unsigned int __busy_write_id(unsigned int id)
+static __always_inline u32 __busy_write_id(u8 id)
{
- /* The uABI guarantees an active writer is also amongst the read
+ /*
+ * The uABI guarantees an active writer is also amongst the read
* engines. This would be true if we accessed the activity tracking
* under the lock, but as we perform the lookup of the object and
* its activity locklessly we can not guarantee that the last_write
* last_read - hence we always set both read and write busy for
* last_write.
*/
- return id | __busy_read_flag(id);
+ if (id == (u8)I915_ENGINE_CLASS_INVALID)
+ return 0xffffffffu;
+
+ return (id + 1) | __busy_read_flag(id);
}
static __always_inline unsigned int
-__busy_set_if_active(const struct dma_fence *fence,
- unsigned int (*flag)(unsigned int id))
+__busy_set_if_active(const struct dma_fence *fence, u32 (*flag)(u8 id))
{
- struct i915_request *rq;
+ const struct i915_request *rq;
- /* We have to check the current hw status of the fence as the uABI
+ /*
+ * We have to check the current hw status of the fence as the uABI
* guarantees forward progress. We could rely on the idle worker
* to eventually flush us, but to minimise latency just ask the
* hardware.
return 0;
/* opencode to_request() in order to avoid const warnings */
- rq = container_of(fence, struct i915_request, fence);
+ rq = container_of(fence, const struct i915_request, fence);
if (i915_request_completed(rq))
return 0;
- return flag(rq->engine->uabi_id);
+ /* Beware type-expansion follies! */
+ BUILD_BUG_ON(!typecheck(u8, rq->engine->uabi_class));
+ return flag(rq->engine->uabi_class);
}
static __always_inline unsigned int
if (!obj)
goto out;
- /* A discrepancy here is that we do not report the status of
+ /*
+ * A discrepancy here is that we do not report the status of
* non-i915 fences, i.e. even though we may report the object as idle,
* a call to set-domain may still stall waiting for foreign rendering.
* This also means that wait-ioctl may report an object as busy,
if (overflows_type(size, obj->base.size))
return ERR_PTR(-E2BIG);
- obj = i915_gem_object_alloc(dev_priv);
+ obj = i915_gem_object_alloc();
if (obj == NULL)
return ERR_PTR(-ENOMEM);
drm_gem_object_release(&obj->base);
i915_gem_info_remove_obj(i915, obj->base.size);
- kfree(obj->bit_17);
+ bitmap_free(obj->bit_17);
i915_gem_object_free(obj);
GEM_BUG_ON(!atomic_read(&i915->mm.free_count));
GEM_TRACE("\n");
wakeref = intel_runtime_pm_get(i915);
- intel_uncore_forcewake_get(i915, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&i915->uncore, FORCEWAKE_ALL);
/*
* As we have just resumed the machine and woken the device up from
* back to defaults, recovering from whatever wedged state we left it
* in and so worth trying to use the device once more.
*/
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_terminally_wedged(i915))
i915_gem_unset_wedged(i915);
/*
*/
intel_engines_sanitize(i915, false);
- intel_uncore_forcewake_put(i915, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&i915->uncore, FORCEWAKE_ALL);
intel_runtime_pm_put(i915, wakeref);
mutex_lock(&i915->drm.struct_mutex);
mutex_unlock(&i915->drm.struct_mutex);
}
-int i915_gem_suspend(struct drm_i915_private *i915)
+void i915_gem_suspend(struct drm_i915_private *i915)
{
intel_wakeref_t wakeref;
- int ret;
GEM_TRACE("\n");
wakeref = intel_runtime_pm_get(i915);
- intel_suspend_gt_powersave(i915);
flush_workqueue(i915->wq);
* state. Fortunately, the kernel_context is disposable and we do
* not rely on its state.
*/
- if (!i915_terminally_wedged(&i915->gpu_error)) {
- ret = i915_gem_switch_to_kernel_context(i915);
- if (ret)
- goto err_unlock;
-
- ret = i915_gem_wait_for_idle(i915,
- I915_WAIT_INTERRUPTIBLE |
- I915_WAIT_LOCKED |
- I915_WAIT_FOR_IDLE_BOOST,
- MAX_SCHEDULE_TIMEOUT);
- if (ret && ret != -EIO)
- goto err_unlock;
-
- assert_kernel_context_is_current(i915);
- }
- i915_retire_requests(i915); /* ensure we flush after wedging */
+ switch_to_kernel_context_sync(i915, i915->gt.active_engines);
mutex_unlock(&i915->drm.struct_mutex);
i915_reset_flush(i915);
*/
drain_delayed_work(&i915->gt.idle_work);
- intel_uc_suspend(i915);
-
/*
* Assert that we successfully flushed all the work and
* reset the GPU back to its idle, low power state.
*/
- WARN_ON(i915->gt.awake);
- if (WARN_ON(!intel_engines_are_idle(i915)))
- i915_gem_set_wedged(i915); /* no hope, discard everything */
+ GEM_BUG_ON(i915->gt.awake);
- intel_runtime_pm_put(i915, wakeref);
- return 0;
+ intel_uc_suspend(i915);
-err_unlock:
- mutex_unlock(&i915->drm.struct_mutex);
intel_runtime_pm_put(i915, wakeref);
- return ret;
}
void i915_gem_suspend_late(struct drm_i915_private *i915)
WARN_ON(i915->gt.awake);
mutex_lock(&i915->drm.struct_mutex);
- intel_uncore_forcewake_get(i915, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&i915->uncore, FORCEWAKE_ALL);
i915_gem_restore_gtt_mappings(i915);
i915_gem_restore_fences(i915);
* guarantee that the context image is complete. So let's just reset
* it and start again.
*/
- i915->gt.resume(i915);
+ intel_gt_resume(i915);
if (i915_gem_init_hw(i915))
goto err_wedged;
intel_uc_resume(i915);
/* Always reload a context for powersaving. */
- if (i915_gem_switch_to_kernel_context(i915))
+ if (!load_power_context(i915))
goto err_wedged;
out_unlock:
- intel_uncore_forcewake_put(i915, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&i915->uncore, FORCEWAKE_ALL);
mutex_unlock(&i915->drm.struct_mutex);
return;
err_wedged:
- if (!i915_terminally_wedged(&i915->gpu_error)) {
- DRM_ERROR("failed to re-initialize GPU, declaring wedged!\n");
+ if (!i915_reset_failed(i915)) {
+ dev_err(i915->drm.dev,
+ "Failed to re-initialize GPU, declaring it wedged!\n");
i915_gem_set_wedged(i915);
}
goto out_unlock;
}
}
+ intel_engines_set_scheduler_caps(i915);
+
return 0;
}
dev_priv->gt.last_init_time = ktime_get();
/* Double layer security blanket, see i915_gem_init() */
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
if (HAS_EDRAM(dev_priv) && INTEL_GEN(dev_priv) < 9)
I915_WRITE(HSW_IDICR, I915_READ(HSW_IDICR) | IDIHASHMSK(0xf));
init_unused_rings(dev_priv);
BUG_ON(!dev_priv->kernel_context);
- if (i915_terminally_wedged(&dev_priv->gpu_error)) {
- ret = -EIO;
+ ret = i915_terminally_wedged(dev_priv);
+ if (ret)
goto out;
- }
ret = i915_ppgtt_init_hw(dev_priv);
if (ret) {
if (ret)
goto cleanup_uc;
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
return 0;
cleanup_uc:
intel_uc_fini_hw(dev_priv);
out:
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
return ret;
}
struct i915_gem_context *ctx;
struct intel_engine_cs *engine;
enum intel_engine_id id;
- int err;
+ int err = 0;
/*
* As we reset the gpu during very early sanitisation, the current
goto err_active;
}
- err = i915_gem_switch_to_kernel_context(i915);
- if (err)
- goto err_active;
-
- if (i915_gem_wait_for_idle(i915, I915_WAIT_LOCKED, HZ / 5)) {
- i915_gem_set_wedged(i915);
- err = -EIO; /* Caller will declare us wedged */
+ /* Flush the default context image to memory, and enable powersaving. */
+ if (!load_power_context(i915)) {
+ err = -EIO;
goto err_active;
}
- assert_kernel_context_is_current(i915);
-
- /*
- * Immediately park the GPU so that we enable powersaving and
- * treat it as idle. The next time we issue a request, we will
- * unpark and start using the engine->pinned_default_state, otherwise
- * it is in limbo and an early reset may fail.
- */
- __i915_gem_park(i915);
-
for_each_engine(engine, i915, id) {
+ struct intel_context *ce;
struct i915_vma *state;
void *vaddr;
- GEM_BUG_ON(to_intel_context(ctx, engine)->pin_count);
+ ce = intel_context_lookup(ctx, engine);
+ if (!ce)
+ continue;
- state = to_intel_context(ctx, engine)->state;
+ state = ce->state;
if (!state)
continue;
+ GEM_BUG_ON(intel_context_is_pinned(ce));
+
/*
* As we will hold a reference to the logical state, it will
* not be torn down with the context, and importantly the
goto err_active;
engine->default_state = i915_gem_object_get(state->obj);
+ i915_gem_object_set_cache_coherency(engine->default_state,
+ I915_CACHE_LLC);
/* Check we can acquire the image of the context state */
vaddr = i915_gem_object_pin_map(engine->default_state,
err_active:
/*
* If we have to abandon now, we expect the engines to be idle
- * and ready to be torn-down. First try to flush any remaining
- * request, ensure we are pointing at the kernel context and
- * then remove it.
+ * and ready to be torn-down. The quickest way we can accomplish
+ * this is by declaring ourselves wedged.
*/
- if (WARN_ON(i915_gem_switch_to_kernel_context(i915)))
- goto out_ctx;
-
- if (WARN_ON(i915_gem_wait_for_idle(i915,
- I915_WAIT_LOCKED,
- MAX_SCHEDULE_TIMEOUT)))
- goto out_ctx;
-
- i915_gem_contexts_lost(i915);
+ i915_gem_set_wedged(i915);
goto out_ctx;
}
dev_priv->mm.unordered_timeline = dma_fence_context_alloc(1);
- if (HAS_LOGICAL_RING_CONTEXTS(dev_priv)) {
- dev_priv->gt.resume = intel_lr_context_resume;
+ if (HAS_LOGICAL_RING_CONTEXTS(dev_priv))
dev_priv->gt.cleanup_engine = intel_logical_ring_cleanup;
- } else {
- dev_priv->gt.resume = intel_legacy_submission_resume;
+ else
dev_priv->gt.cleanup_engine = intel_engine_cleanup;
- }
i915_timelines_init(dev_priv);
* just magically go away.
*/
mutex_lock(&dev_priv->drm.struct_mutex);
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
ret = i915_gem_init_ggtt(dev_priv);
if (ret) {
goto err_init_hw;
}
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
mutex_unlock(&dev_priv->drm.struct_mutex);
return 0;
err_init_hw:
mutex_unlock(&dev_priv->drm.struct_mutex);
- WARN_ON(i915_gem_suspend(dev_priv));
+ i915_gem_suspend(dev_priv);
i915_gem_suspend_late(dev_priv);
i915_gem_drain_workqueue(dev_priv);
i915_gem_fini_scratch(dev_priv);
err_ggtt:
err_unlock:
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
mutex_unlock(&dev_priv->drm.struct_mutex);
err_uc_misc:
* wedged. But we only want to do this where the GPU is angry,
* for all other failure, such as an allocation failure, bail.
*/
- if (!i915_terminally_wedged(&dev_priv->gpu_error)) {
+ if (!i915_reset_failed(dev_priv)) {
i915_load_error(dev_priv,
"Failed to initialize GPU, declaring it wedged!\n");
i915_gem_set_wedged(dev_priv);
int i915_gem_init_early(struct drm_i915_private *dev_priv)
{
- int err = -ENOMEM;
-
- dev_priv->objects = KMEM_CACHE(drm_i915_gem_object, SLAB_HWCACHE_ALIGN);
- if (!dev_priv->objects)
- goto err_out;
-
- dev_priv->vmas = KMEM_CACHE(i915_vma, SLAB_HWCACHE_ALIGN);
- if (!dev_priv->vmas)
- goto err_objects;
-
- dev_priv->luts = KMEM_CACHE(i915_lut_handle, 0);
- if (!dev_priv->luts)
- goto err_vmas;
-
- dev_priv->requests = KMEM_CACHE(i915_request,
- SLAB_HWCACHE_ALIGN |
- SLAB_RECLAIM_ACCOUNT |
- SLAB_TYPESAFE_BY_RCU);
- if (!dev_priv->requests)
- goto err_luts;
-
- dev_priv->dependencies = KMEM_CACHE(i915_dependency,
- SLAB_HWCACHE_ALIGN |
- SLAB_RECLAIM_ACCOUNT);
- if (!dev_priv->dependencies)
- goto err_requests;
-
- dev_priv->priorities = KMEM_CACHE(i915_priolist, SLAB_HWCACHE_ALIGN);
- if (!dev_priv->priorities)
- goto err_dependencies;
+ int err;
INIT_LIST_HEAD(&dev_priv->gt.active_rings);
INIT_LIST_HEAD(&dev_priv->gt.closed_vma);
init_waitqueue_head(&dev_priv->gpu_error.wait_queue);
init_waitqueue_head(&dev_priv->gpu_error.reset_queue);
mutex_init(&dev_priv->gpu_error.wedge_mutex);
+ init_srcu_struct(&dev_priv->gpu_error.reset_backoff_srcu);
atomic_set(&dev_priv->mm.bsd_engine_dispatch_index, 0);
DRM_NOTE("Unable to create a private tmpfs mount, hugepage support will be disabled(%d).\n", err);
return 0;
-
-err_dependencies:
- kmem_cache_destroy(dev_priv->dependencies);
-err_requests:
- kmem_cache_destroy(dev_priv->requests);
-err_luts:
- kmem_cache_destroy(dev_priv->luts);
-err_vmas:
- kmem_cache_destroy(dev_priv->vmas);
-err_objects:
- kmem_cache_destroy(dev_priv->objects);
-err_out:
- return err;
}
void i915_gem_cleanup_early(struct drm_i915_private *dev_priv)
GEM_BUG_ON(atomic_read(&dev_priv->mm.free_count));
WARN_ON(dev_priv->mm.object_count);
- kmem_cache_destroy(dev_priv->priorities);
- kmem_cache_destroy(dev_priv->dependencies);
- kmem_cache_destroy(dev_priv->requests);
- kmem_cache_destroy(dev_priv->luts);
- kmem_cache_destroy(dev_priv->vmas);
- kmem_cache_destroy(dev_priv->objects);
-
- /* And ensure that our DESTROY_BY_RCU slabs are truly destroyed */
- rcu_barrier();
+ cleanup_srcu_struct(&dev_priv->gpu_error.reset_backoff_srcu);
i915_gemfs_fini(dev_priv);
}
#define GEM_TRACE_DUMP_ON(expr) BUILD_BUG_ON_INVALID(expr)
#endif
-#define I915_NUM_ENGINES 8
+#define I915_GEM_IDLE_TIMEOUT (HZ / 5)
void i915_gem_park(struct drm_i915_private *i915);
void i915_gem_unpark(struct drm_i915_private *i915);
static inline void __tasklet_disable_sync_once(struct tasklet_struct *t)
{
- if (atomic_inc_return(&t->count) == 1)
+ if (!atomic_fetch_inc(&t->count))
tasklet_unlock_wait(t);
}
return !atomic_read(&t->count);
}
+static inline bool __tasklet_enable(struct tasklet_struct *t)
+{
+ return atomic_dec_and_test(&t->count);
+}
+
#endif /* __I915_GEM_H__ */
#include <linux/log2.h>
#include <drm/i915_drm.h>
#include "i915_drv.h"
+#include "i915_globals.h"
#include "i915_trace.h"
+#include "i915_user_extensions.h"
#include "intel_lrc_reg.h"
#include "intel_workarounds.h"
+#define I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE (1 << 1)
+#define I915_CONTEXT_PARAM_VM 0x9
+
#define ALL_L3_SLICES(dev) (1 << NUM_L3_SLICES(dev)) - 1
+static struct i915_global_gem_context {
+ struct i915_global base;
+ struct kmem_cache *slab_luts;
+} global;
+
+struct i915_lut_handle *i915_lut_handle_alloc(void)
+{
+ return kmem_cache_alloc(global.slab_luts, GFP_KERNEL);
+}
+
+void i915_lut_handle_free(struct i915_lut_handle *lut)
+{
+ return kmem_cache_free(global.slab_luts, lut);
+}
+
static void lut_close(struct i915_gem_context *ctx)
{
struct i915_lut_handle *lut, *ln;
list_for_each_entry_safe(lut, ln, &ctx->handles_list, ctx_link) {
list_del(&lut->obj_link);
- kmem_cache_free(ctx->i915->luts, lut);
+ i915_lut_handle_free(lut);
}
+ INIT_LIST_HEAD(&ctx->handles_list);
rcu_read_lock();
radix_tree_for_each_slot(slot, &ctx->handles_vma, &iter, 0) {
struct i915_vma *vma = rcu_dereference_raw(*slot);
radix_tree_iter_delete(&ctx->handles_vma, &iter, slot);
+
+ vma->open_count--;
__i915_gem_object_release_unless_active(vma->obj);
}
rcu_read_unlock();
static void i915_gem_context_free(struct i915_gem_context *ctx)
{
- unsigned int n;
+ struct intel_context *it, *n;
lockdep_assert_held(&ctx->i915->drm.struct_mutex);
GEM_BUG_ON(!i915_gem_context_is_closed(ctx));
+ GEM_BUG_ON(!list_empty(&ctx->active_engines));
release_hw_id(ctx);
i915_ppgtt_put(ctx->ppgtt);
- for (n = 0; n < ARRAY_SIZE(ctx->__engine); n++) {
- struct intel_context *ce = &ctx->__engine[n];
+ rbtree_postorder_for_each_entry_safe(it, n, &ctx->hw_contexts, node)
+ intel_context_put(it);
- if (ce->ops)
- ce->ops->destroy(ce);
- }
+ if (ctx->timeline)
+ i915_timeline_put(ctx->timeline);
kfree(ctx->name);
put_pid(ctx->pid);
list_del(&ctx->link);
+ mutex_destroy(&ctx->mutex);
kfree_rcu(ctx, rcu);
}
* the ppgtt).
*/
lut_close(ctx);
- if (ctx->ppgtt)
- i915_ppgtt_close(&ctx->ppgtt->vm);
ctx->file_priv = ERR_PTR(-EBADF);
i915_gem_context_put(ctx);
desc = GEN8_CTX_VALID | GEN8_CTX_PRIVILEGE;
address_mode = INTEL_LEGACY_32B_CONTEXT;
- if (ppgtt && i915_vm_is_48bit(&ppgtt->vm))
+ if (ppgtt && i915_vm_is_4lvl(&ppgtt->vm))
address_mode = INTEL_LEGACY_64B_CONTEXT;
desc |= address_mode << GEN8_CTX_ADDRESSING_MODE_SHIFT;
return desc;
}
-static void intel_context_retire(struct i915_active_request *active,
- struct i915_request *rq)
-{
- struct intel_context *ce =
- container_of(active, typeof(*ce), active_tracker);
-
- intel_context_unpin(ce);
-}
-
-void
-intel_context_init(struct intel_context *ce,
- struct i915_gem_context *ctx,
- struct intel_engine_cs *engine)
-{
- ce->gem_context = ctx;
-
- INIT_LIST_HEAD(&ce->signal_link);
- INIT_LIST_HEAD(&ce->signals);
-
- /* Use the whole device by default */
- ce->sseu = intel_device_default_sseu(ctx->i915);
-
- i915_active_request_init(&ce->active_tracker,
- NULL, intel_context_retire);
-}
-
static struct i915_gem_context *
-__create_hw_context(struct drm_i915_private *dev_priv,
- struct drm_i915_file_private *file_priv)
+__create_context(struct drm_i915_private *dev_priv)
{
struct i915_gem_context *ctx;
- unsigned int n;
- int ret;
+ int i;
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
- if (ctx == NULL)
+ if (!ctx)
return ERR_PTR(-ENOMEM);
kref_init(&ctx->ref);
list_add_tail(&ctx->link, &dev_priv->contexts.list);
ctx->i915 = dev_priv;
ctx->sched.priority = I915_USER_PRIORITY(I915_PRIORITY_NORMAL);
+ INIT_LIST_HEAD(&ctx->active_engines);
+ mutex_init(&ctx->mutex);
- for (n = 0; n < ARRAY_SIZE(ctx->__engine); n++)
- intel_context_init(&ctx->__engine[n], ctx, dev_priv->engine[n]);
+ ctx->hw_contexts = RB_ROOT;
+ spin_lock_init(&ctx->hw_contexts_lock);
INIT_RADIX_TREE(&ctx->handles_vma, GFP_KERNEL);
INIT_LIST_HEAD(&ctx->handles_list);
INIT_LIST_HEAD(&ctx->hw_id_link);
- /* Default context will never have a file_priv */
- ret = DEFAULT_CONTEXT_HANDLE;
- if (file_priv) {
- ret = idr_alloc(&file_priv->context_idr, ctx,
- DEFAULT_CONTEXT_HANDLE, 0, GFP_KERNEL);
- if (ret < 0)
- goto err_lut;
- }
- ctx->user_handle = ret;
-
- ctx->file_priv = file_priv;
- if (file_priv) {
- ctx->pid = get_task_pid(current, PIDTYPE_PID);
- ctx->name = kasprintf(GFP_KERNEL, "%s[%d]/%x",
- current->comm,
- pid_nr(ctx->pid),
- ctx->user_handle);
- if (!ctx->name) {
- ret = -ENOMEM;
- goto err_pid;
- }
- }
-
/* NB: Mark all slices as needing a remap so that when the context first
* loads it will restore whatever remap state already exists. If there
* is no remap info, it will be a NOP. */
ctx->remap_slice = ALL_L3_SLICES(dev_priv);
i915_gem_context_set_bannable(ctx);
+ i915_gem_context_set_recoverable(ctx);
+
ctx->ring_size = 4 * PAGE_SIZE;
ctx->desc_template =
default_desc_template(dev_priv, dev_priv->mm.aliasing_ppgtt);
+ for (i = 0; i < ARRAY_SIZE(ctx->hang_timestamp); i++)
+ ctx->hang_timestamp[i] = jiffies - CONTEXT_FAST_HANG_JIFFIES;
+
return ctx;
+}
-err_pid:
- put_pid(ctx->pid);
- idr_remove(&file_priv->context_idr, ctx->user_handle);
-err_lut:
- context_close(ctx);
- return ERR_PTR(ret);
+static struct i915_hw_ppgtt *
+__set_ppgtt(struct i915_gem_context *ctx, struct i915_hw_ppgtt *ppgtt)
+{
+ struct i915_hw_ppgtt *old = ctx->ppgtt;
+
+ ctx->ppgtt = i915_ppgtt_get(ppgtt);
+ ctx->desc_template = default_desc_template(ctx->i915, ppgtt);
+
+ return old;
}
-static void __destroy_hw_context(struct i915_gem_context *ctx,
- struct drm_i915_file_private *file_priv)
+static void __assign_ppgtt(struct i915_gem_context *ctx,
+ struct i915_hw_ppgtt *ppgtt)
{
- idr_remove(&file_priv->context_idr, ctx->user_handle);
- context_close(ctx);
+ if (ppgtt == ctx->ppgtt)
+ return;
+
+ ppgtt = __set_ppgtt(ctx, ppgtt);
+ if (ppgtt)
+ i915_ppgtt_put(ppgtt);
}
static struct i915_gem_context *
-i915_gem_create_context(struct drm_i915_private *dev_priv,
- struct drm_i915_file_private *file_priv)
+i915_gem_create_context(struct drm_i915_private *dev_priv, unsigned int flags)
{
struct i915_gem_context *ctx;
lockdep_assert_held(&dev_priv->drm.struct_mutex);
+ BUILD_BUG_ON(I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE &
+ ~I915_CONTEXT_CREATE_FLAGS_UNKNOWN);
+ if (flags & I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE &&
+ !HAS_EXECLISTS(dev_priv))
+ return ERR_PTR(-EINVAL);
+
/* Reap the most stale context */
contexts_free_first(dev_priv);
- ctx = __create_hw_context(dev_priv, file_priv);
+ ctx = __create_context(dev_priv);
if (IS_ERR(ctx))
return ctx;
if (HAS_FULL_PPGTT(dev_priv)) {
struct i915_hw_ppgtt *ppgtt;
- ppgtt = i915_ppgtt_create(dev_priv, file_priv);
+ ppgtt = i915_ppgtt_create(dev_priv);
if (IS_ERR(ppgtt)) {
DRM_DEBUG_DRIVER("PPGTT setup failed (%ld)\n",
PTR_ERR(ppgtt));
- __destroy_hw_context(ctx, file_priv);
+ context_close(ctx);
return ERR_CAST(ppgtt);
}
- ctx->ppgtt = ppgtt;
- ctx->desc_template = default_desc_template(dev_priv, ppgtt);
+ __assign_ppgtt(ctx, ppgtt);
+ i915_ppgtt_put(ppgtt);
+ }
+
+ if (flags & I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE) {
+ struct i915_timeline *timeline;
+
+ timeline = i915_timeline_create(dev_priv, NULL);
+ if (IS_ERR(timeline)) {
+ context_close(ctx);
+ return ERR_CAST(timeline);
+ }
+
+ ctx->timeline = timeline;
}
trace_i915_context_create(ctx);
if (ret)
return ERR_PTR(ret);
- ctx = i915_gem_create_context(to_i915(dev), NULL);
+ ctx = i915_gem_create_context(to_i915(dev), 0);
if (IS_ERR(ctx))
goto out;
+ ret = i915_gem_context_pin_hw_id(ctx);
+ if (ret) {
+ context_close(ctx);
+ ctx = ERR_PTR(ret);
+ goto out;
+ }
+
ctx->file_priv = ERR_PTR(-EBADF);
i915_gem_context_set_closed(ctx); /* not user accessible */
i915_gem_context_clear_bannable(ctx);
struct i915_gem_context *ctx;
int err;
- ctx = i915_gem_create_context(i915, NULL);
+ ctx = i915_gem_create_context(i915, 0);
if (IS_ERR(ctx))
return ctx;
static bool needs_preempt_context(struct drm_i915_private *i915)
{
- return HAS_LOGICAL_RING_PREEMPTION(i915);
+ return HAS_EXECLISTS(i915);
}
int i915_gem_contexts_init(struct drm_i915_private *dev_priv)
GEM_BUG_ON(dev_priv->kernel_context);
GEM_BUG_ON(dev_priv->preempt_context);
- intel_engine_init_ctx_wa(dev_priv->engine[RCS]);
+ intel_engine_init_ctx_wa(dev_priv->engine[RCS0]);
init_contexts(dev_priv);
/* lowest priority; idle task */
static int context_idr_cleanup(int id, void *p, void *data)
{
- struct i915_gem_context *ctx = p;
+ context_close(p);
+ return 0;
+}
- context_close(ctx);
+static int vm_idr_cleanup(int id, void *p, void *data)
+{
+ i915_ppgtt_put(p);
return 0;
}
+static int gem_context_register(struct i915_gem_context *ctx,
+ struct drm_i915_file_private *fpriv)
+{
+ int ret;
+
+ ctx->file_priv = fpriv;
+ if (ctx->ppgtt)
+ ctx->ppgtt->vm.file = fpriv;
+
+ ctx->pid = get_task_pid(current, PIDTYPE_PID);
+ ctx->name = kasprintf(GFP_KERNEL, "%s[%d]",
+ current->comm, pid_nr(ctx->pid));
+ if (!ctx->name) {
+ ret = -ENOMEM;
+ goto err_pid;
+ }
+
+ /* And finally expose ourselves to userspace via the idr */
+ mutex_lock(&fpriv->context_idr_lock);
+ ret = idr_alloc(&fpriv->context_idr, ctx, 0, 0, GFP_KERNEL);
+ mutex_unlock(&fpriv->context_idr_lock);
+ if (ret >= 0)
+ goto out;
+
+ kfree(fetch_and_zero(&ctx->name));
+err_pid:
+ put_pid(fetch_and_zero(&ctx->pid));
+out:
+ return ret;
+}
+
int i915_gem_context_open(struct drm_i915_private *i915,
struct drm_file *file)
{
struct drm_i915_file_private *file_priv = file->driver_priv;
struct i915_gem_context *ctx;
+ int err;
+
+ mutex_init(&file_priv->context_idr_lock);
+ mutex_init(&file_priv->vm_idr_lock);
idr_init(&file_priv->context_idr);
+ idr_init_base(&file_priv->vm_idr, 1);
mutex_lock(&i915->drm.struct_mutex);
- ctx = i915_gem_create_context(i915, file_priv);
+ ctx = i915_gem_create_context(i915, 0);
mutex_unlock(&i915->drm.struct_mutex);
if (IS_ERR(ctx)) {
- idr_destroy(&file_priv->context_idr);
- return PTR_ERR(ctx);
+ err = PTR_ERR(ctx);
+ goto err;
}
+ err = gem_context_register(ctx, file_priv);
+ if (err < 0)
+ goto err_ctx;
+
GEM_BUG_ON(i915_gem_context_is_kernel(ctx));
+ GEM_BUG_ON(err > 0);
return 0;
+
+err_ctx:
+ mutex_lock(&i915->drm.struct_mutex);
+ context_close(ctx);
+ mutex_unlock(&i915->drm.struct_mutex);
+err:
+ idr_destroy(&file_priv->vm_idr);
+ idr_destroy(&file_priv->context_idr);
+ mutex_destroy(&file_priv->vm_idr_lock);
+ mutex_destroy(&file_priv->context_idr_lock);
+ return err;
}
void i915_gem_context_close(struct drm_file *file)
idr_for_each(&file_priv->context_idr, context_idr_cleanup, NULL);
idr_destroy(&file_priv->context_idr);
+ mutex_destroy(&file_priv->context_idr_lock);
+
+ idr_for_each(&file_priv->vm_idr, vm_idr_cleanup, NULL);
+ idr_destroy(&file_priv->vm_idr);
+ mutex_destroy(&file_priv->vm_idr_lock);
+}
+
+int i915_gem_vm_create_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file)
+{
+ struct drm_i915_private *i915 = to_i915(dev);
+ struct drm_i915_gem_vm_control *args = data;
+ struct drm_i915_file_private *file_priv = file->driver_priv;
+ struct i915_hw_ppgtt *ppgtt;
+ int err;
+
+ if (!HAS_FULL_PPGTT(i915))
+ return -ENODEV;
+
+ if (args->flags)
+ return -EINVAL;
+
+ ppgtt = i915_ppgtt_create(i915);
+ if (IS_ERR(ppgtt))
+ return PTR_ERR(ppgtt);
+
+ ppgtt->vm.file = file_priv;
+
+ if (args->extensions) {
+ err = i915_user_extensions(u64_to_user_ptr(args->extensions),
+ NULL, 0,
+ ppgtt);
+ if (err)
+ goto err_put;
+ }
+
+ err = mutex_lock_interruptible(&file_priv->vm_idr_lock);
+ if (err)
+ goto err_put;
+
+ err = idr_alloc(&file_priv->vm_idr, ppgtt, 0, 0, GFP_KERNEL);
+ if (err < 0)
+ goto err_unlock;
+
+ GEM_BUG_ON(err == 0); /* reserved for default/unassigned ppgtt */
+ ppgtt->user_handle = err;
+
+ mutex_unlock(&file_priv->vm_idr_lock);
+
+ args->vm_id = err;
+ return 0;
+
+err_unlock:
+ mutex_unlock(&file_priv->vm_idr_lock);
+err_put:
+ i915_ppgtt_put(ppgtt);
+ return err;
+}
+
+int i915_gem_vm_destroy_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file)
+{
+ struct drm_i915_file_private *file_priv = file->driver_priv;
+ struct drm_i915_gem_vm_control *args = data;
+ struct i915_hw_ppgtt *ppgtt;
+ int err;
+ u32 id;
+
+ if (args->flags)
+ return -EINVAL;
+
+ if (args->extensions)
+ return -EINVAL;
+
+ id = args->vm_id;
+ if (!id)
+ return -ENOENT;
+
+ err = mutex_lock_interruptible(&file_priv->vm_idr_lock);
+ if (err)
+ return err;
+
+ ppgtt = idr_remove(&file_priv->vm_idr, id);
+ if (ppgtt) {
+ GEM_BUG_ON(ppgtt->user_handle != id);
+ ppgtt->user_handle = 0;
+ }
+
+ mutex_unlock(&file_priv->vm_idr_lock);
+ if (!ppgtt)
+ return -ENOENT;
+
+ i915_ppgtt_put(ppgtt);
+ return 0;
}
static struct i915_request *
rq = i915_active_request_raw(&timeline->last_request,
&engine->i915->drm.struct_mutex);
- if (rq && rq->engine == engine) {
- GEM_TRACE("last request for %s on engine %s: %llx:%llu\n",
- timeline->name, engine->name,
- rq->fence.context, rq->fence.seqno);
+ if (rq && rq->engine->mask & engine->mask) {
+ GEM_TRACE("last request on engine %s: %llx:%llu\n",
+ engine->name, rq->fence.context, rq->fence.seqno);
GEM_BUG_ON(rq->timeline != timeline);
return rq;
}
return NULL;
}
-static bool engine_has_kernel_context_barrier(struct intel_engine_cs *engine)
+struct context_barrier_task {
+ struct i915_active base;
+ void (*task)(void *data);
+ void *data;
+};
+
+static void cb_retire(struct i915_active *base)
+{
+ struct context_barrier_task *cb = container_of(base, typeof(*cb), base);
+
+ if (cb->task)
+ cb->task(cb->data);
+
+ i915_active_fini(&cb->base);
+ kfree(cb);
+}
+
+I915_SELFTEST_DECLARE(static intel_engine_mask_t context_barrier_inject_fault);
+static int context_barrier_task(struct i915_gem_context *ctx,
+ intel_engine_mask_t engines,
+ int (*emit)(struct i915_request *rq, void *data),
+ void (*task)(void *data),
+ void *data)
{
- struct drm_i915_private *i915 = engine->i915;
- const struct intel_context * const ce =
- to_intel_context(i915->kernel_context, engine);
- struct i915_timeline *barrier = ce->ring->timeline;
- struct intel_ring *ring;
- bool any_active = false;
+ struct drm_i915_private *i915 = ctx->i915;
+ struct context_barrier_task *cb;
+ struct intel_context *ce, *next;
+ intel_wakeref_t wakeref;
+ int err = 0;
lockdep_assert_held(&i915->drm.struct_mutex);
- list_for_each_entry(ring, &i915->gt.active_rings, active_link) {
- struct i915_request *rq;
+ GEM_BUG_ON(!task);
- rq = last_request_on_engine(ring->timeline, engine);
- if (!rq)
- continue;
+ cb = kmalloc(sizeof(*cb), GFP_KERNEL);
+ if (!cb)
+ return -ENOMEM;
+
+ i915_active_init(i915, &cb->base, cb_retire);
+ i915_active_acquire(&cb->base);
- any_active = true;
+ wakeref = intel_runtime_pm_get(i915);
+ rbtree_postorder_for_each_entry_safe(ce, next, &ctx->hw_contexts, node) {
+ struct intel_engine_cs *engine = ce->engine;
+ struct i915_request *rq;
- if (rq->hw_context == ce)
+ if (!(engine->mask & engines))
continue;
- /*
- * Was this request submitted after the previous
- * switch-to-kernel-context?
- */
- if (!i915_timeline_sync_is_later(barrier, &rq->fence)) {
- GEM_TRACE("%s needs barrier for %llx:%lld\n",
- ring->timeline->name,
- rq->fence.context,
- rq->fence.seqno);
- return false;
+ if (I915_SELFTEST_ONLY(context_barrier_inject_fault &
+ engine->mask)) {
+ err = -ENXIO;
+ break;
+ }
+
+ rq = i915_request_alloc(engine, ctx);
+ if (IS_ERR(rq)) {
+ err = PTR_ERR(rq);
+ break;
}
- GEM_TRACE("%s has barrier after %llx:%lld\n",
- ring->timeline->name,
- rq->fence.context,
- rq->fence.seqno);
+ err = 0;
+ if (emit)
+ err = emit(rq, data);
+ if (err == 0)
+ err = i915_active_ref(&cb->base, rq->fence.context, rq);
+
+ i915_request_add(rq);
+ if (err)
+ break;
}
+ intel_runtime_pm_put(i915, wakeref);
- /*
- * If any other timeline was still active and behind the last barrier,
- * then our last switch-to-kernel-context must still be queued and
- * will run last (leaving the engine in the kernel context when it
- * eventually idles).
- */
- if (any_active)
- return true;
+ cb->task = err ? NULL : task; /* caller needs to unwind instead */
+ cb->data = data;
- /* The engine is idle; check that it is idling in the kernel context. */
- return engine->last_retired_context == ce;
+ i915_active_release(&cb->base);
+
+ return err;
}
-int i915_gem_switch_to_kernel_context(struct drm_i915_private *i915)
+int i915_gem_switch_to_kernel_context(struct drm_i915_private *i915,
+ intel_engine_mask_t mask)
{
struct intel_engine_cs *engine;
- enum intel_engine_id id;
GEM_TRACE("awake?=%s\n", yesno(i915->gt.awake));
lockdep_assert_held(&i915->drm.struct_mutex);
GEM_BUG_ON(!i915->kernel_context);
- i915_retire_requests(i915);
+ /* Inoperable, so presume the GPU is safely pointing into the void! */
+ if (i915_terminally_wedged(i915))
+ return 0;
- for_each_engine(engine, i915, id) {
+ for_each_engine_masked(engine, i915, mask, mask) {
struct intel_ring *ring;
struct i915_request *rq;
- GEM_BUG_ON(!to_intel_context(i915->kernel_context, engine));
- if (engine_has_kernel_context_barrier(engine))
- continue;
-
- GEM_TRACE("emit barrier on %s\n", engine->name);
-
rq = i915_request_alloc(engine, i915->kernel_context);
if (IS_ERR(rq))
return PTR_ERR(rq);
i915_sw_fence_await_sw_fence_gfp(&rq->submit,
&prev->submit,
I915_FENCE_GFP);
- i915_timeline_sync_set(rq->timeline, &prev->fence);
}
i915_request_add(rq);
return 0;
}
-static bool client_is_banned(struct drm_i915_file_private *file_priv)
-{
- return atomic_read(&file_priv->ban_score) >= I915_CLIENT_SCORE_BANNED;
-}
-
-int i915_gem_context_create_ioctl(struct drm_device *dev, void *data,
- struct drm_file *file)
+static int get_ppgtt(struct drm_i915_file_private *file_priv,
+ struct i915_gem_context *ctx,
+ struct drm_i915_gem_context_param *args)
{
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct drm_i915_gem_context_create *args = data;
- struct drm_i915_file_private *file_priv = file->driver_priv;
- struct i915_gem_context *ctx;
+ struct i915_hw_ppgtt *ppgtt;
int ret;
- if (!DRIVER_CAPS(dev_priv)->has_logical_contexts)
+ return -EINVAL; /* nothing to see here; please move along */
+
+ if (!ctx->ppgtt)
return -ENODEV;
- if (args->pad != 0)
- return -EINVAL;
+ /* XXX rcu acquire? */
+ ret = mutex_lock_interruptible(&ctx->i915->drm.struct_mutex);
+ if (ret)
+ return ret;
- if (client_is_banned(file_priv)) {
- DRM_DEBUG("client %s[%d] banned from creating ctx\n",
- current->comm,
- pid_nr(get_task_pid(current, PIDTYPE_PID)));
+ ppgtt = i915_ppgtt_get(ctx->ppgtt);
+ mutex_unlock(&ctx->i915->drm.struct_mutex);
- return -EIO;
+ ret = mutex_lock_interruptible(&file_priv->vm_idr_lock);
+ if (ret)
+ goto err_put;
+
+ if (!ppgtt->user_handle) {
+ ret = idr_alloc(&file_priv->vm_idr, ppgtt, 0, 0, GFP_KERNEL);
+ GEM_BUG_ON(!ret);
+ if (ret < 0)
+ goto err_unlock;
+
+ ppgtt->user_handle = ret;
+ i915_ppgtt_get(ppgtt);
}
- ret = i915_mutex_lock_interruptible(dev);
- if (ret)
- return ret;
+ args->size = 0;
+ args->value = ppgtt->user_handle;
- ctx = i915_gem_create_context(dev_priv, file_priv);
- mutex_unlock(&dev->struct_mutex);
- if (IS_ERR(ctx))
- return PTR_ERR(ctx);
+ ret = 0;
+err_unlock:
+ mutex_unlock(&file_priv->vm_idr_lock);
+err_put:
+ i915_ppgtt_put(ppgtt);
+ return ret;
+}
- GEM_BUG_ON(i915_gem_context_is_kernel(ctx));
+static void set_ppgtt_barrier(void *data)
+{
+ struct i915_hw_ppgtt *old = data;
- args->ctx_id = ctx->user_handle;
- DRM_DEBUG("HW context %d created\n", args->ctx_id);
+ if (INTEL_GEN(old->vm.i915) < 8)
+ gen6_ppgtt_unpin_all(old);
- return 0;
+ i915_ppgtt_put(old);
}
-int i915_gem_context_destroy_ioctl(struct drm_device *dev, void *data,
- struct drm_file *file)
+static int emit_ppgtt_update(struct i915_request *rq, void *data)
{
- struct drm_i915_gem_context_destroy *args = data;
- struct drm_i915_file_private *file_priv = file->driver_priv;
- struct i915_gem_context *ctx;
- int ret;
+ struct i915_hw_ppgtt *ppgtt = rq->gem_context->ppgtt;
+ struct intel_engine_cs *engine = rq->engine;
+ u32 base = engine->mmio_base;
+ u32 *cs;
+ int i;
- if (args->pad != 0)
- return -EINVAL;
+ if (i915_vm_is_4lvl(&ppgtt->vm)) {
+ const dma_addr_t pd_daddr = px_dma(&ppgtt->pml4);
- if (args->ctx_id == DEFAULT_CONTEXT_HANDLE)
- return -ENOENT;
+ cs = intel_ring_begin(rq, 6);
+ if (IS_ERR(cs))
+ return PTR_ERR(cs);
- ctx = i915_gem_context_lookup(file_priv, args->ctx_id);
- if (!ctx)
- return -ENOENT;
+ *cs++ = MI_LOAD_REGISTER_IMM(2);
- ret = mutex_lock_interruptible(&dev->struct_mutex);
- if (ret)
- goto out;
+ *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_UDW(base, 0));
+ *cs++ = upper_32_bits(pd_daddr);
+ *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_LDW(base, 0));
+ *cs++ = lower_32_bits(pd_daddr);
- __destroy_hw_context(ctx, file_priv);
- mutex_unlock(&dev->struct_mutex);
+ *cs++ = MI_NOOP;
+ intel_ring_advance(rq, cs);
+ } else if (HAS_LOGICAL_RING_CONTEXTS(engine->i915)) {
+ cs = intel_ring_begin(rq, 4 * GEN8_3LVL_PDPES + 2);
+ if (IS_ERR(cs))
+ return PTR_ERR(cs);
+
+ *cs++ = MI_LOAD_REGISTER_IMM(2 * GEN8_3LVL_PDPES);
+ for (i = GEN8_3LVL_PDPES; i--; ) {
+ const dma_addr_t pd_daddr = i915_page_dir_dma_addr(ppgtt, i);
+
+ *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_UDW(base, i));
+ *cs++ = upper_32_bits(pd_daddr);
+ *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_LDW(base, i));
+ *cs++ = lower_32_bits(pd_daddr);
+ }
+ *cs++ = MI_NOOP;
+ intel_ring_advance(rq, cs);
+ } else {
+ /* ppGTT is not part of the legacy context image */
+ gen6_ppgtt_pin(ppgtt);
+ }
-out:
- i915_gem_context_put(ctx);
return 0;
}
-static int get_sseu(struct i915_gem_context *ctx,
- struct drm_i915_gem_context_param *args)
+static int set_ppgtt(struct drm_i915_file_private *file_priv,
+ struct i915_gem_context *ctx,
+ struct drm_i915_gem_context_param *args)
{
- struct drm_i915_gem_context_param_sseu user_sseu;
- struct intel_engine_cs *engine;
- struct intel_context *ce;
- int ret;
+ struct i915_hw_ppgtt *ppgtt, *old;
+ int err;
- if (args->size == 0)
- goto out;
- else if (args->size < sizeof(user_sseu))
+ return -EINVAL; /* nothing to see here; please move along */
+
+ if (args->size)
return -EINVAL;
- if (copy_from_user(&user_sseu, u64_to_user_ptr(args->value),
- sizeof(user_sseu)))
- return -EFAULT;
+ if (!ctx->ppgtt)
+ return -ENODEV;
- if (user_sseu.flags || user_sseu.rsvd)
- return -EINVAL;
+ if (upper_32_bits(args->value))
+ return -ENOENT;
- engine = intel_engine_lookup_user(ctx->i915,
- user_sseu.engine_class,
- user_sseu.engine_instance);
- if (!engine)
- return -EINVAL;
+ err = mutex_lock_interruptible(&file_priv->vm_idr_lock);
+ if (err)
+ return err;
- /* Only use for mutex here is to serialize get_param and set_param. */
- ret = mutex_lock_interruptible(&ctx->i915->drm.struct_mutex);
- if (ret)
- return ret;
+ ppgtt = idr_find(&file_priv->vm_idr, args->value);
+ if (ppgtt) {
+ GEM_BUG_ON(ppgtt->user_handle != args->value);
+ i915_ppgtt_get(ppgtt);
+ }
+ mutex_unlock(&file_priv->vm_idr_lock);
+ if (!ppgtt)
+ return -ENOENT;
- ce = to_intel_context(ctx, engine);
+ err = mutex_lock_interruptible(&ctx->i915->drm.struct_mutex);
+ if (err)
+ goto out;
- user_sseu.slice_mask = ce->sseu.slice_mask;
- user_sseu.subslice_mask = ce->sseu.subslice_mask;
- user_sseu.min_eus_per_subslice = ce->sseu.min_eus_per_subslice;
- user_sseu.max_eus_per_subslice = ce->sseu.max_eus_per_subslice;
+ if (ppgtt == ctx->ppgtt)
+ goto unlock;
- mutex_unlock(&ctx->i915->drm.struct_mutex);
+ /* Teardown the existing obj:vma cache, it will have to be rebuilt. */
+ lut_close(ctx);
- if (copy_to_user(u64_to_user_ptr(args->value), &user_sseu,
- sizeof(user_sseu)))
- return -EFAULT;
+ old = __set_ppgtt(ctx, ppgtt);
-out:
- args->size = sizeof(user_sseu);
+ /*
+ * We need to flush any requests using the current ppgtt before
+ * we release it as the requests do not hold a reference themselves,
+ * only indirectly through the context.
+ */
+ err = context_barrier_task(ctx, ALL_ENGINES,
+ emit_ppgtt_update,
+ set_ppgtt_barrier,
+ old);
+ if (err) {
+ ctx->ppgtt = old;
+ ctx->desc_template = default_desc_template(ctx->i915, old);
+ i915_ppgtt_put(ppgtt);
+ }
- return 0;
-}
-
-int i915_gem_context_getparam_ioctl(struct drm_device *dev, void *data,
- struct drm_file *file)
-{
- struct drm_i915_file_private *file_priv = file->driver_priv;
- struct drm_i915_gem_context_param *args = data;
- struct i915_gem_context *ctx;
- int ret = 0;
-
- ctx = i915_gem_context_lookup(file_priv, args->ctx_id);
- if (!ctx)
- return -ENOENT;
-
- switch (args->param) {
- case I915_CONTEXT_PARAM_BAN_PERIOD:
- ret = -EINVAL;
- break;
- case I915_CONTEXT_PARAM_NO_ZEROMAP:
- args->size = 0;
- args->value = test_bit(UCONTEXT_NO_ZEROMAP, &ctx->user_flags);
- break;
- case I915_CONTEXT_PARAM_GTT_SIZE:
- args->size = 0;
-
- if (ctx->ppgtt)
- args->value = ctx->ppgtt->vm.total;
- else if (to_i915(dev)->mm.aliasing_ppgtt)
- args->value = to_i915(dev)->mm.aliasing_ppgtt->vm.total;
- else
- args->value = to_i915(dev)->ggtt.vm.total;
- break;
- case I915_CONTEXT_PARAM_NO_ERROR_CAPTURE:
- args->size = 0;
- args->value = i915_gem_context_no_error_capture(ctx);
- break;
- case I915_CONTEXT_PARAM_BANNABLE:
- args->size = 0;
- args->value = i915_gem_context_is_bannable(ctx);
- break;
- case I915_CONTEXT_PARAM_PRIORITY:
- args->size = 0;
- args->value = ctx->sched.priority >> I915_USER_PRIORITY_SHIFT;
- break;
- case I915_CONTEXT_PARAM_SSEU:
- ret = get_sseu(ctx, args);
- break;
- default:
- ret = -EINVAL;
- break;
- }
-
- i915_gem_context_put(ctx);
- return ret;
+unlock:
+ mutex_unlock(&ctx->i915->drm.struct_mutex);
+
+out:
+ i915_ppgtt_put(ppgtt);
+ return err;
}
static int gen8_emit_rpcs_config(struct i915_request *rq,
}
static int
-gen8_modify_rpcs_gpu(struct intel_context *ce,
- struct intel_engine_cs *engine,
- struct intel_sseu sseu)
+gen8_modify_rpcs(struct intel_context *ce, struct intel_sseu sseu)
{
- struct drm_i915_private *i915 = engine->i915;
- struct i915_request *rq, *prev;
+ struct drm_i915_private *i915 = ce->engine->i915;
+ struct i915_request *rq;
intel_wakeref_t wakeref;
int ret;
- GEM_BUG_ON(!ce->pin_count);
+ lockdep_assert_held(&ce->pin_mutex);
- lockdep_assert_held(&i915->drm.struct_mutex);
+ /*
+ * If the context is not idle, we have to submit an ordered request to
+ * modify its context image via the kernel context (writing to our own
+ * image, or into the registers directory, does not stick). Pristine
+ * and idle contexts will be configured on pinning.
+ */
+ if (!intel_context_is_pinned(ce))
+ return 0;
/* Submitting requests etc needs the hw awake. */
wakeref = intel_runtime_pm_get(i915);
- rq = i915_request_alloc(engine, i915->kernel_context);
+ rq = i915_request_alloc(ce->engine, i915->kernel_context);
if (IS_ERR(rq)) {
ret = PTR_ERR(rq);
goto out_put;
}
/* Queue this switch after all other activity by this context. */
- prev = i915_active_request_raw(&ce->ring->timeline->last_request,
- &i915->drm.struct_mutex);
- if (prev && !i915_request_completed(prev)) {
- ret = i915_request_await_dma_fence(rq, &prev->fence);
- if (ret < 0)
- goto out_add;
- }
-
- /* Order all following requests to be after. */
- ret = i915_timeline_set_barrier(ce->ring->timeline, rq);
+ ret = i915_active_request_set(&ce->ring->timeline->last_request, rq);
if (ret)
goto out_add;
struct intel_engine_cs *engine,
struct intel_sseu sseu)
{
- struct intel_context *ce = to_intel_context(ctx, engine);
+ struct intel_context *ce;
int ret = 0;
GEM_BUG_ON(INTEL_GEN(ctx->i915) < 8);
- GEM_BUG_ON(engine->id != RCS);
+ GEM_BUG_ON(engine->id != RCS0);
+
+ ce = intel_context_pin_lock(ctx, engine);
+ if (IS_ERR(ce))
+ return PTR_ERR(ce);
/* Nothing to do if unmodified. */
if (!memcmp(&ce->sseu, &sseu, sizeof(sseu)))
- return 0;
-
- /*
- * If context is not idle we have to submit an ordered request to modify
- * its context image via the kernel context. Pristine and idle contexts
- * will be configured on pinning.
- */
- if (ce->pin_count)
- ret = gen8_modify_rpcs_gpu(ce, engine, sseu);
+ goto unlock;
+ ret = gen8_modify_rpcs(ce, sseu);
if (!ret)
ce->sseu = sseu;
+unlock:
+ intel_context_pin_unlock(ce);
return ret;
}
return -EINVAL;
engine = intel_engine_lookup_user(i915,
- user_sseu.engine_class,
- user_sseu.engine_instance);
+ user_sseu.engine.engine_class,
+ user_sseu.engine.engine_instance);
if (!engine)
return -EINVAL;
return 0;
}
-int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,
- struct drm_file *file)
+static int ctx_setparam(struct drm_i915_file_private *fpriv,
+ struct i915_gem_context *ctx,
+ struct drm_i915_gem_context_param *args)
{
- struct drm_i915_file_private *file_priv = file->driver_priv;
- struct drm_i915_gem_context_param *args = data;
- struct i915_gem_context *ctx;
int ret = 0;
- ctx = i915_gem_context_lookup(file_priv, args->ctx_id);
- if (!ctx)
- return -ENOENT;
-
switch (args->param) {
- case I915_CONTEXT_PARAM_BAN_PERIOD:
- ret = -EINVAL;
- break;
case I915_CONTEXT_PARAM_NO_ZEROMAP:
if (args->size)
ret = -EINVAL;
else
clear_bit(UCONTEXT_NO_ZEROMAP, &ctx->user_flags);
break;
+
case I915_CONTEXT_PARAM_NO_ERROR_CAPTURE:
if (args->size)
ret = -EINVAL;
else
i915_gem_context_clear_no_error_capture(ctx);
break;
+
case I915_CONTEXT_PARAM_BANNABLE:
if (args->size)
ret = -EINVAL;
i915_gem_context_clear_bannable(ctx);
break;
+ case I915_CONTEXT_PARAM_RECOVERABLE:
+ if (args->size)
+ ret = -EINVAL;
+ else if (args->value)
+ i915_gem_context_set_recoverable(ctx);
+ else
+ i915_gem_context_clear_recoverable(ctx);
+ break;
+
case I915_CONTEXT_PARAM_PRIORITY:
{
s64 priority = args->value;
if (args->size)
ret = -EINVAL;
- else if (!(to_i915(dev)->caps.scheduler & I915_SCHEDULER_CAP_PRIORITY))
+ else if (!(ctx->i915->caps.scheduler & I915_SCHEDULER_CAP_PRIORITY))
ret = -ENODEV;
else if (priority > I915_CONTEXT_MAX_USER_PRIORITY ||
priority < I915_CONTEXT_MIN_USER_PRIORITY)
I915_USER_PRIORITY(priority);
}
break;
+
case I915_CONTEXT_PARAM_SSEU:
ret = set_sseu(ctx, args);
break;
+
+ case I915_CONTEXT_PARAM_VM:
+ ret = set_ppgtt(fpriv, ctx, args);
+ break;
+
+ case I915_CONTEXT_PARAM_BAN_PERIOD:
default:
ret = -EINVAL;
break;
}
+ return ret;
+}
+
+struct create_ext {
+ struct i915_gem_context *ctx;
+ struct drm_i915_file_private *fpriv;
+};
+
+static int create_setparam(struct i915_user_extension __user *ext, void *data)
+{
+ struct drm_i915_gem_context_create_ext_setparam local;
+ const struct create_ext *arg = data;
+
+ if (copy_from_user(&local, ext, sizeof(local)))
+ return -EFAULT;
+
+ if (local.param.ctx_id)
+ return -EINVAL;
+
+ return ctx_setparam(arg->fpriv, arg->ctx, &local.param);
+}
+
+static const i915_user_extension_fn create_extensions[] = {
+ [I915_CONTEXT_CREATE_EXT_SETPARAM] = create_setparam,
+};
+
+static bool client_is_banned(struct drm_i915_file_private *file_priv)
+{
+ return atomic_read(&file_priv->ban_score) >= I915_CLIENT_SCORE_BANNED;
+}
+
+int i915_gem_context_create_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file)
+{
+ struct drm_i915_private *i915 = to_i915(dev);
+ struct drm_i915_gem_context_create_ext *args = data;
+ struct create_ext ext_data;
+ int ret;
+
+ if (!DRIVER_CAPS(i915)->has_logical_contexts)
+ return -ENODEV;
+
+ if (args->flags & I915_CONTEXT_CREATE_FLAGS_UNKNOWN)
+ return -EINVAL;
+
+ ret = i915_terminally_wedged(i915);
+ if (ret)
+ return ret;
+
+ ext_data.fpriv = file->driver_priv;
+ if (client_is_banned(ext_data.fpriv)) {
+ DRM_DEBUG("client %s[%d] banned from creating ctx\n",
+ current->comm,
+ pid_nr(get_task_pid(current, PIDTYPE_PID)));
+ return -EIO;
+ }
+
+ ret = i915_mutex_lock_interruptible(dev);
+ if (ret)
+ return ret;
+
+ ext_data.ctx = i915_gem_create_context(i915, args->flags);
+ mutex_unlock(&dev->struct_mutex);
+ if (IS_ERR(ext_data.ctx))
+ return PTR_ERR(ext_data.ctx);
+
+ if (args->flags & I915_CONTEXT_CREATE_FLAGS_USE_EXTENSIONS) {
+ ret = i915_user_extensions(u64_to_user_ptr(args->extensions),
+ create_extensions,
+ ARRAY_SIZE(create_extensions),
+ &ext_data);
+ if (ret)
+ goto err_ctx;
+ }
+
+ ret = gem_context_register(ext_data.ctx, ext_data.fpriv);
+ if (ret < 0)
+ goto err_ctx;
+
+ args->ctx_id = ret;
+ DRM_DEBUG("HW context %d created\n", args->ctx_id);
+
+ return 0;
+
+err_ctx:
+ mutex_lock(&dev->struct_mutex);
+ context_close(ext_data.ctx);
+ mutex_unlock(&dev->struct_mutex);
+ return ret;
+}
+
+int i915_gem_context_destroy_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file)
+{
+ struct drm_i915_gem_context_destroy *args = data;
+ struct drm_i915_file_private *file_priv = file->driver_priv;
+ struct i915_gem_context *ctx;
+
+ if (args->pad != 0)
+ return -EINVAL;
+
+ if (!args->ctx_id)
+ return -ENOENT;
+
+ if (mutex_lock_interruptible(&file_priv->context_idr_lock))
+ return -EINTR;
+
+ ctx = idr_remove(&file_priv->context_idr, args->ctx_id);
+ mutex_unlock(&file_priv->context_idr_lock);
+ if (!ctx)
+ return -ENOENT;
+
+ mutex_lock(&dev->struct_mutex);
+ context_close(ctx);
+ mutex_unlock(&dev->struct_mutex);
+
+ return 0;
+}
+
+static int get_sseu(struct i915_gem_context *ctx,
+ struct drm_i915_gem_context_param *args)
+{
+ struct drm_i915_gem_context_param_sseu user_sseu;
+ struct intel_engine_cs *engine;
+ struct intel_context *ce;
+
+ if (args->size == 0)
+ goto out;
+ else if (args->size < sizeof(user_sseu))
+ return -EINVAL;
+
+ if (copy_from_user(&user_sseu, u64_to_user_ptr(args->value),
+ sizeof(user_sseu)))
+ return -EFAULT;
+
+ if (user_sseu.flags || user_sseu.rsvd)
+ return -EINVAL;
+
+ engine = intel_engine_lookup_user(ctx->i915,
+ user_sseu.engine.engine_class,
+ user_sseu.engine.engine_instance);
+ if (!engine)
+ return -EINVAL;
+
+ ce = intel_context_pin_lock(ctx, engine); /* serialises with set_sseu */
+ if (IS_ERR(ce))
+ return PTR_ERR(ce);
+
+ user_sseu.slice_mask = ce->sseu.slice_mask;
+ user_sseu.subslice_mask = ce->sseu.subslice_mask;
+ user_sseu.min_eus_per_subslice = ce->sseu.min_eus_per_subslice;
+ user_sseu.max_eus_per_subslice = ce->sseu.max_eus_per_subslice;
+
+ intel_context_pin_unlock(ce);
+
+ if (copy_to_user(u64_to_user_ptr(args->value), &user_sseu,
+ sizeof(user_sseu)))
+ return -EFAULT;
+
+out:
+ args->size = sizeof(user_sseu);
+
+ return 0;
+}
+
+int i915_gem_context_getparam_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file)
+{
+ struct drm_i915_file_private *file_priv = file->driver_priv;
+ struct drm_i915_gem_context_param *args = data;
+ struct i915_gem_context *ctx;
+ int ret = 0;
+
+ ctx = i915_gem_context_lookup(file_priv, args->ctx_id);
+ if (!ctx)
+ return -ENOENT;
+
+ switch (args->param) {
+ case I915_CONTEXT_PARAM_NO_ZEROMAP:
+ args->size = 0;
+ args->value = test_bit(UCONTEXT_NO_ZEROMAP, &ctx->user_flags);
+ break;
+
+ case I915_CONTEXT_PARAM_GTT_SIZE:
+ args->size = 0;
+ if (ctx->ppgtt)
+ args->value = ctx->ppgtt->vm.total;
+ else if (to_i915(dev)->mm.aliasing_ppgtt)
+ args->value = to_i915(dev)->mm.aliasing_ppgtt->vm.total;
+ else
+ args->value = to_i915(dev)->ggtt.vm.total;
+ break;
+
+ case I915_CONTEXT_PARAM_NO_ERROR_CAPTURE:
+ args->size = 0;
+ args->value = i915_gem_context_no_error_capture(ctx);
+ break;
+
+ case I915_CONTEXT_PARAM_BANNABLE:
+ args->size = 0;
+ args->value = i915_gem_context_is_bannable(ctx);
+ break;
+
+ case I915_CONTEXT_PARAM_RECOVERABLE:
+ args->size = 0;
+ args->value = i915_gem_context_is_recoverable(ctx);
+ break;
+
+ case I915_CONTEXT_PARAM_PRIORITY:
+ args->size = 0;
+ args->value = ctx->sched.priority >> I915_USER_PRIORITY_SHIFT;
+ break;
+
+ case I915_CONTEXT_PARAM_SSEU:
+ ret = get_sseu(ctx, args);
+ break;
+
+ case I915_CONTEXT_PARAM_VM:
+ ret = get_ppgtt(file_priv, ctx, args);
+ break;
+
+ case I915_CONTEXT_PARAM_BAN_PERIOD:
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ i915_gem_context_put(ctx);
+ return ret;
+}
+
+int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file)
+{
+ struct drm_i915_file_private *file_priv = file->driver_priv;
+ struct drm_i915_gem_context_param *args = data;
+ struct i915_gem_context *ctx;
+ int ret;
+
+ ctx = i915_gem_context_lookup(file_priv, args->ctx_id);
+ if (!ctx)
+ return -ENOENT;
+
+ ret = ctx_setparam(file_priv, ctx, args);
+
i915_gem_context_put(ctx);
return ret;
}
#include "selftests/mock_context.c"
#include "selftests/i915_gem_context.c"
#endif
+
+static void i915_global_gem_context_shrink(void)
+{
+ kmem_cache_shrink(global.slab_luts);
+}
+
+static void i915_global_gem_context_exit(void)
+{
+ kmem_cache_destroy(global.slab_luts);
+}
+
+static struct i915_global_gem_context global = { {
+ .shrink = i915_global_gem_context_shrink,
+ .exit = i915_global_gem_context_exit,
+} };
+
+int __init i915_global_gem_context_init(void)
+{
+ global.slab_luts = KMEM_CACHE(i915_lut_handle, 0);
+ if (!global.slab_luts)
+ return -ENOMEM;
+
+ i915_global_register(&global.base);
+ return 0;
+}
#ifndef __I915_GEM_CONTEXT_H__
#define __I915_GEM_CONTEXT_H__
-#include <linux/bitops.h>
-#include <linux/list.h>
-#include <linux/radix-tree.h>
+#include "i915_gem_context_types.h"
#include "i915_gem.h"
#include "i915_scheduler.h"
+#include "intel_context.h"
#include "intel_device_info.h"
-struct pid;
-
struct drm_device;
struct drm_file;
-struct drm_i915_private;
-struct drm_i915_file_private;
-struct i915_hw_ppgtt;
-struct i915_request;
-struct i915_vma;
-struct intel_ring;
-
-#define DEFAULT_CONTEXT_HANDLE 0
-
-struct intel_context;
-
-struct intel_context_ops {
- void (*unpin)(struct intel_context *ce);
- void (*destroy)(struct intel_context *ce);
-};
-
-/*
- * Powergating configuration for a particular (context,engine).
- */
-struct intel_sseu {
- u8 slice_mask;
- u8 subslice_mask;
- u8 min_eus_per_subslice;
- u8 max_eus_per_subslice;
-};
-
-/**
- * struct i915_gem_context - client state
- *
- * The struct i915_gem_context represents the combined view of the driver and
- * logical hardware state for a particular client.
- */
-struct i915_gem_context {
- /** i915: i915 device backpointer */
- struct drm_i915_private *i915;
-
- /** file_priv: owning file descriptor */
- struct drm_i915_file_private *file_priv;
-
- /**
- * @ppgtt: unique address space (GTT)
- *
- * In full-ppgtt mode, each context has its own address space ensuring
- * complete seperation of one client from all others.
- *
- * In other modes, this is a NULL pointer with the expectation that
- * the caller uses the shared global GTT.
- */
- struct i915_hw_ppgtt *ppgtt;
-
- /**
- * @pid: process id of creator
- *
- * Note that who created the context may not be the principle user,
- * as the context may be shared across a local socket. However,
- * that should only affect the default context, all contexts created
- * explicitly by the client are expected to be isolated.
- */
- struct pid *pid;
-
- /**
- * @name: arbitrary name
- *
- * A name is constructed for the context from the creator's process
- * name, pid and user handle in order to uniquely identify the
- * context in messages.
- */
- const char *name;
-
- /** link: place with &drm_i915_private.context_list */
- struct list_head link;
- struct llist_node free_link;
-
- /**
- * @ref: reference count
- *
- * A reference to a context is held by both the client who created it
- * and on each request submitted to the hardware using the request
- * (to ensure the hardware has access to the state until it has
- * finished all pending writes). See i915_gem_context_get() and
- * i915_gem_context_put() for access.
- */
- struct kref ref;
-
- /**
- * @rcu: rcu_head for deferred freeing.
- */
- struct rcu_head rcu;
-
- /**
- * @user_flags: small set of booleans controlled by the user
- */
- unsigned long user_flags;
-#define UCONTEXT_NO_ZEROMAP 0
-#define UCONTEXT_NO_ERROR_CAPTURE 1
-#define UCONTEXT_BANNABLE 2
-
- /**
- * @flags: small set of booleans
- */
- unsigned long flags;
-#define CONTEXT_BANNED 0
-#define CONTEXT_CLOSED 1
-#define CONTEXT_FORCE_SINGLE_SUBMISSION 2
-
- /**
- * @hw_id: - unique identifier for the context
- *
- * The hardware needs to uniquely identify the context for a few
- * functions like fault reporting, PASID, scheduling. The
- * &drm_i915_private.context_hw_ida is used to assign a unqiue
- * id for the lifetime of the context.
- *
- * @hw_id_pin_count: - number of times this context had been pinned
- * for use (should be, at most, once per engine).
- *
- * @hw_id_link: - all contexts with an assigned id are tracked
- * for possible repossession.
- */
- unsigned int hw_id;
- atomic_t hw_id_pin_count;
- struct list_head hw_id_link;
-
- /**
- * @user_handle: userspace identifier
- *
- * A unique per-file identifier is generated from
- * &drm_i915_file_private.contexts.
- */
- u32 user_handle;
-
- struct i915_sched_attr sched;
-
- /** engine: per-engine logical HW state */
- struct intel_context {
- struct i915_gem_context *gem_context;
- struct intel_engine_cs *active;
- struct list_head signal_link;
- struct list_head signals;
- struct i915_vma *state;
- struct intel_ring *ring;
- u32 *lrc_reg_state;
- u64 lrc_desc;
- int pin_count;
-
- /**
- * active_tracker: Active tracker for the external rq activity
- * on this intel_context object.
- */
- struct i915_active_request active_tracker;
-
- const struct intel_context_ops *ops;
-
- /** sseu: Control eu/slice partitioning */
- struct intel_sseu sseu;
- } __engine[I915_NUM_ENGINES];
-
- /** ring_size: size for allocating the per-engine ring buffer */
- u32 ring_size;
- /** desc_template: invariant fields for the HW context descriptor */
- u32 desc_template;
-
- /** guilty_count: How many times this context has caused a GPU hang. */
- atomic_t guilty_count;
- /**
- * @active_count: How many times this context was active during a GPU
- * hang, but did not cause it.
- */
- atomic_t active_count;
-
-#define CONTEXT_SCORE_GUILTY 10
-#define CONTEXT_SCORE_BAN_THRESHOLD 40
- /** ban_score: Accumulated score of all hangs caused by this context. */
- atomic_t ban_score;
-
- /** remap_slice: Bitmask of cache lines that need remapping */
- u8 remap_slice;
-
- /** handles_vma: rbtree to look up our context specific obj/vma for
- * the user handle. (user handles are per fd, but the binding is
- * per vm, which may be one per context or shared with the global GTT)
- */
- struct radix_tree_root handles_vma;
-
- /** handles_list: reverse list of all the rbtree entries in use for
- * this context, which allows us to free all the allocations on
- * context close.
- */
- struct list_head handles_list;
-};
-
static inline bool i915_gem_context_is_closed(const struct i915_gem_context *ctx)
{
return test_bit(CONTEXT_CLOSED, &ctx->flags);
clear_bit(UCONTEXT_BANNABLE, &ctx->user_flags);
}
+static inline bool i915_gem_context_is_recoverable(const struct i915_gem_context *ctx)
+{
+ return test_bit(UCONTEXT_RECOVERABLE, &ctx->user_flags);
+}
+
+static inline void i915_gem_context_set_recoverable(struct i915_gem_context *ctx)
+{
+ set_bit(UCONTEXT_RECOVERABLE, &ctx->user_flags);
+}
+
+static inline void i915_gem_context_clear_recoverable(struct i915_gem_context *ctx)
+{
+ clear_bit(UCONTEXT_RECOVERABLE, &ctx->user_flags);
+}
+
static inline bool i915_gem_context_is_banned(const struct i915_gem_context *ctx)
{
return test_bit(CONTEXT_BANNED, &ctx->flags);
atomic_dec(&ctx->hw_id_pin_count);
}
-static inline bool i915_gem_context_is_default(const struct i915_gem_context *c)
-{
- return c->user_handle == DEFAULT_CONTEXT_HANDLE;
-}
-
static inline bool i915_gem_context_is_kernel(struct i915_gem_context *ctx)
{
return !ctx->file_priv;
}
-static inline struct intel_context *
-to_intel_context(struct i915_gem_context *ctx,
- const struct intel_engine_cs *engine)
-{
- return &ctx->__engine[engine->id];
-}
-
-static inline struct intel_context *
-intel_context_pin(struct i915_gem_context *ctx, struct intel_engine_cs *engine)
-{
- return engine->context_pin(engine, ctx);
-}
-
-static inline void __intel_context_pin(struct intel_context *ce)
-{
- GEM_BUG_ON(!ce->pin_count);
- ce->pin_count++;
-}
-
-static inline void intel_context_unpin(struct intel_context *ce)
-{
- GEM_BUG_ON(!ce->pin_count);
- if (--ce->pin_count)
- return;
-
- GEM_BUG_ON(!ce->ops);
- ce->ops->unpin(ce);
-}
-
/* i915_gem_context.c */
int __must_check i915_gem_contexts_init(struct drm_i915_private *dev_priv);
void i915_gem_contexts_lost(struct drm_i915_private *dev_priv);
void i915_gem_context_close(struct drm_file *file);
int i915_switch_context(struct i915_request *rq);
-int i915_gem_switch_to_kernel_context(struct drm_i915_private *dev_priv);
+int i915_gem_switch_to_kernel_context(struct drm_i915_private *i915,
+ intel_engine_mask_t engine_mask);
void i915_gem_context_release(struct kref *ctx_ref);
struct i915_gem_context *
i915_gem_context_create_gvt(struct drm_device *dev);
+int i915_gem_vm_create_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file);
+int i915_gem_vm_destroy_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file);
+
int i915_gem_context_create_ioctl(struct drm_device *dev, void *data,
struct drm_file *file);
int i915_gem_context_destroy_ioctl(struct drm_device *dev, void *data,
kref_put(&ctx->ref, i915_gem_context_release);
}
-void intel_context_init(struct intel_context *ce,
- struct i915_gem_context *ctx,
- struct intel_engine_cs *engine);
+struct i915_lut_handle *i915_lut_handle_alloc(void);
+void i915_lut_handle_free(struct i915_lut_handle *lut);
#endif /* !__I915_GEM_CONTEXT_H__ */
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __I915_GEM_CONTEXT_TYPES_H__
+#define __I915_GEM_CONTEXT_TYPES_H__
+
+#include <linux/atomic.h>
+#include <linux/list.h>
+#include <linux/llist.h>
+#include <linux/kref.h>
+#include <linux/mutex.h>
+#include <linux/radix-tree.h>
+#include <linux/rbtree.h>
+#include <linux/rcupdate.h>
+#include <linux/types.h>
+
+#include "i915_scheduler.h"
+#include "intel_context_types.h"
+
+struct pid;
+
+struct drm_i915_private;
+struct drm_i915_file_private;
+struct i915_hw_ppgtt;
+struct i915_timeline;
+struct intel_ring;
+
+/**
+ * struct i915_gem_context - client state
+ *
+ * The struct i915_gem_context represents the combined view of the driver and
+ * logical hardware state for a particular client.
+ */
+struct i915_gem_context {
+ /** i915: i915 device backpointer */
+ struct drm_i915_private *i915;
+
+ /** file_priv: owning file descriptor */
+ struct drm_i915_file_private *file_priv;
+
+ struct i915_timeline *timeline;
+
+ /**
+ * @ppgtt: unique address space (GTT)
+ *
+ * In full-ppgtt mode, each context has its own address space ensuring
+ * complete seperation of one client from all others.
+ *
+ * In other modes, this is a NULL pointer with the expectation that
+ * the caller uses the shared global GTT.
+ */
+ struct i915_hw_ppgtt *ppgtt;
+
+ /**
+ * @pid: process id of creator
+ *
+ * Note that who created the context may not be the principle user,
+ * as the context may be shared across a local socket. However,
+ * that should only affect the default context, all contexts created
+ * explicitly by the client are expected to be isolated.
+ */
+ struct pid *pid;
+
+ /**
+ * @name: arbitrary name
+ *
+ * A name is constructed for the context from the creator's process
+ * name, pid and user handle in order to uniquely identify the
+ * context in messages.
+ */
+ const char *name;
+
+ /** link: place with &drm_i915_private.context_list */
+ struct list_head link;
+ struct llist_node free_link;
+
+ /**
+ * @ref: reference count
+ *
+ * A reference to a context is held by both the client who created it
+ * and on each request submitted to the hardware using the request
+ * (to ensure the hardware has access to the state until it has
+ * finished all pending writes). See i915_gem_context_get() and
+ * i915_gem_context_put() for access.
+ */
+ struct kref ref;
+
+ /**
+ * @rcu: rcu_head for deferred freeing.
+ */
+ struct rcu_head rcu;
+
+ /**
+ * @user_flags: small set of booleans controlled by the user
+ */
+ unsigned long user_flags;
+#define UCONTEXT_NO_ZEROMAP 0
+#define UCONTEXT_NO_ERROR_CAPTURE 1
+#define UCONTEXT_BANNABLE 2
+#define UCONTEXT_RECOVERABLE 3
+
+ /**
+ * @flags: small set of booleans
+ */
+ unsigned long flags;
+#define CONTEXT_BANNED 0
+#define CONTEXT_CLOSED 1
+#define CONTEXT_FORCE_SINGLE_SUBMISSION 2
+
+ /**
+ * @hw_id: - unique identifier for the context
+ *
+ * The hardware needs to uniquely identify the context for a few
+ * functions like fault reporting, PASID, scheduling. The
+ * &drm_i915_private.context_hw_ida is used to assign a unqiue
+ * id for the lifetime of the context.
+ *
+ * @hw_id_pin_count: - number of times this context had been pinned
+ * for use (should be, at most, once per engine).
+ *
+ * @hw_id_link: - all contexts with an assigned id are tracked
+ * for possible repossession.
+ */
+ unsigned int hw_id;
+ atomic_t hw_id_pin_count;
+ struct list_head hw_id_link;
+
+ struct list_head active_engines;
+ struct mutex mutex;
+
+ struct i915_sched_attr sched;
+
+ /** hw_contexts: per-engine logical HW state */
+ struct rb_root hw_contexts;
+ spinlock_t hw_contexts_lock;
+
+ /** ring_size: size for allocating the per-engine ring buffer */
+ u32 ring_size;
+ /** desc_template: invariant fields for the HW context descriptor */
+ u32 desc_template;
+
+ /** guilty_count: How many times this context has caused a GPU hang. */
+ atomic_t guilty_count;
+ /**
+ * @active_count: How many times this context was active during a GPU
+ * hang, but did not cause it.
+ */
+ atomic_t active_count;
+
+ /**
+ * @hang_timestamp: The last time(s) this context caused a GPU hang
+ */
+ unsigned long hang_timestamp[2];
+#define CONTEXT_FAST_HANG_JIFFIES (120 * HZ) /* 3 hangs within 120s? Banned! */
+
+ /** remap_slice: Bitmask of cache lines that need remapping */
+ u8 remap_slice;
+
+ /** handles_vma: rbtree to look up our context specific obj/vma for
+ * the user handle. (user handles are per fd, but the binding is
+ * per vm, which may be one per context or shared with the global GTT)
+ */
+ struct radix_tree_root handles_vma;
+
+ /** handles_list: reverse list of all the rbtree entries in use for
+ * this context, which allows us to free all the allocations on
+ * context close.
+ */
+ struct list_head handles_list;
+};
+
+#endif /* __I915_GEM_CONTEXT_TYPES_H__ */
{
struct drm_i915_gem_object *obj = dma_buf_to_obj(dma_buf);
+ i915_gem_object_flush_map(obj);
i915_gem_object_unpin_map(obj);
}
get_dma_buf(dma_buf);
- obj = i915_gem_object_alloc(to_i915(dev));
+ obj = i915_gem_object_alloc();
if (obj == NULL) {
ret = -ENOMEM;
goto fail_detach;
static bool ggtt_is_idle(struct drm_i915_private *i915)
{
- struct intel_engine_cs *engine;
- enum intel_engine_id id;
-
- if (i915->gt.active_requests)
- return false;
-
- for_each_engine(engine, i915, id) {
- if (!intel_engine_has_kernel_context(engine))
- return false;
- }
-
- return true;
+ return !i915->gt.active_requests;
}
static int ggtt_flush(struct drm_i915_private *i915)
{
int err;
- /* Not everything in the GGTT is tracked via vma (otherwise we
+ /*
+ * Not everything in the GGTT is tracked via vma (otherwise we
* could evict as required with minimal stalling) so we are forced
* to idle the GPU and explicitly retire outstanding requests in
* the hopes that we can then remove contexts and the like only
* bound by their active reference.
*/
- err = i915_gem_switch_to_kernel_context(i915);
+ err = i915_gem_switch_to_kernel_context(i915, i915->gt.active_engines);
if (err)
return err;
* keeping all of their resources pinned.
*/
- ce = to_intel_context(eb->ctx, eb->engine);
- if (!ce->ring) /* first use, assume empty! */
+ ce = intel_context_lookup(eb->ctx, eb->engine);
+ if (!ce || !ce->ring) /* first use, assume empty! */
return 0;
rq = __eb_wait_for_ring(ce->ring);
}
vma = i915_vma_instance(obj, eb->vm, NULL);
- if (unlikely(IS_ERR(vma))) {
+ if (IS_ERR(vma)) {
err = PTR_ERR(vma);
goto err_obj;
}
- lut = kmem_cache_alloc(eb->i915->luts, GFP_KERNEL);
+ lut = i915_lut_handle_alloc();
if (unlikely(!lut)) {
err = -ENOMEM;
goto err_obj;
err = radix_tree_insert(handles_vma, handle, vma);
if (unlikely(err)) {
- kmem_cache_free(eb->i915->luts, lut);
+ i915_lut_handle_free(lut);
goto err_obj;
}
{
GEM_BUG_ON(cache->rq_size >= cache->rq->batch->obj->base.size / sizeof(u32));
cache->rq_cmd[cache->rq_size] = MI_BATCH_BUFFER_END;
+
+ __i915_gem_object_flush_map(cache->rq->batch->obj, 0, cache->rq_size);
i915_gem_object_unpin_map(cache->rq->batch->obj);
+
i915_gem_chipset_flush(cache->rq->i915);
i915_request_add(cache->rq);
if (IS_ERR(cmd))
return PTR_ERR(cmd);
- err = i915_gem_object_set_to_wc_domain(obj, false);
- if (err)
- goto err_unmap;
-
batch = i915_vma_instance(obj, vma->vm, NULL);
if (IS_ERR(batch)) {
err = PTR_ERR(batch);
u32 *cs;
int i;
- if (!IS_GEN(rq->i915, 7) || rq->engine->id != RCS) {
+ if (!IS_GEN(rq->i915, 7) || rq->engine->id != RCS0) {
DRM_DEBUG("sol reset is gen7/rcs only\n");
return -EINVAL;
}
#define I915_USER_RINGS (4)
static const enum intel_engine_id user_ring_map[I915_USER_RINGS + 1] = {
- [I915_EXEC_DEFAULT] = RCS,
- [I915_EXEC_RENDER] = RCS,
- [I915_EXEC_BLT] = BCS,
- [I915_EXEC_BSD] = VCS,
- [I915_EXEC_VEBOX] = VECS
+ [I915_EXEC_DEFAULT] = RCS0,
+ [I915_EXEC_RENDER] = RCS0,
+ [I915_EXEC_BLT] = BCS0,
+ [I915_EXEC_BSD] = VCS0,
+ [I915_EXEC_VEBOX] = VECS0
};
static struct intel_engine_cs *
return NULL;
}
- if (user_ring_id == I915_EXEC_BSD && HAS_BSD2(dev_priv)) {
+ if (user_ring_id == I915_EXEC_BSD && HAS_ENGINE(dev_priv, VCS1)) {
unsigned int bsd_idx = args->flags & I915_EXEC_BSD_MASK;
if (bsd_idx == I915_EXEC_BSD_DEFAULT) {
if (args->flags & I915_EXEC_IS_PINNED)
eb.batch_flags |= I915_DISPATCH_PINNED;
- eb.engine = eb_select_engine(eb.i915, file, args);
- if (!eb.engine)
- return -EINVAL;
-
if (args->flags & I915_EXEC_FENCE_IN) {
in_fence = sync_file_get_fence(lower_32_bits(args->rsvd2));
if (!in_fence)
if (unlikely(err))
goto err_destroy;
+ eb.engine = eb_select_engine(eb.i915, file, args);
+ if (!eb.engine) {
+ err = -EINVAL;
+ goto err_engine;
+ }
+
/*
* Take a local wakeref for preparing to dispatch the execbuf as
* we expect to access the hardware fairly frequently in the
mutex_unlock(&dev->struct_mutex);
err_rpm:
intel_runtime_pm_put(eb.i915, wakeref);
+err_engine:
i915_gem_context_put(eb.ctx);
err_destroy:
eb_destroy(&eb);
struct i915_vma *vma)
{
intel_wakeref_t wakeref;
+ struct i915_vma *old;
int ret;
if (vma) {
return ret;
}
- if (fence->vma) {
- struct i915_vma *old = fence->vma;
-
+ old = xchg(&fence->vma, NULL);
+ if (old) {
ret = i915_active_request_retire(&old->last_fence,
&old->obj->base.dev->struct_mutex);
- if (ret)
+ if (ret) {
+ fence->vma = old;
return ret;
+ }
i915_vma_flush_writes(old);
- }
- if (fence->vma && fence->vma != vma) {
- /* Ensure that all userspace CPU access is completed before
+ /*
+ * Ensure that all userspace CPU access is completed before
* stealing the fence.
*/
- GEM_BUG_ON(fence->vma->fence != fence);
- i915_vma_revoke_mmap(fence->vma);
-
- fence->vma->fence = NULL;
- fence->vma = NULL;
+ if (old != vma) {
+ GEM_BUG_ON(old->fence != fence);
+ i915_vma_revoke_mmap(old);
+ old->fence = NULL;
+ }
list_move(&fence->link, &fence->i915->mm.fence_list);
}
- /* We only need to update the register itself if the device is awake.
+ /*
+ * We only need to update the register itself if the device is awake.
* If the device is currently powered down, we will defer the write
* to the runtime resume, see i915_gem_restore_fences().
+ *
+ * This only works for removing the fence register, on acquisition
+ * the caller must hold the rpm wakeref. The fence register must
+ * be cleared before we can use any other fences to ensure that
+ * the new fences do not overlap the elided clears, confusing HW.
*/
wakeref = intel_runtime_pm_get_if_in_use(fence->i915);
- if (wakeref) {
- fence_write(fence, vma);
- intel_runtime_pm_put(fence->i915, wakeref);
+ if (!wakeref) {
+ GEM_BUG_ON(vma);
+ return 0;
}
- if (vma) {
- if (fence->vma != vma) {
- vma->fence = fence;
- fence->vma = vma;
- }
+ WRITE_ONCE(fence->vma, vma);
+ fence_write(fence, vma);
+ if (vma) {
+ vma->fence = fence;
list_move_tail(&fence->link, &fence->i915->mm.fence_list);
}
+ intel_runtime_pm_put(fence->i915, wakeref);
return 0;
}
list_add(&fence->link, &fence->i915->mm.fence_list);
}
-/**
- * i915_gem_revoke_fences - revoke fence state
- * @dev_priv: i915 device private
- *
- * Removes all GTT mmappings via the fence registers. This forces any user
- * of the fence to reacquire that fence before continuing with their access.
- * One use is during GPU reset where the fence register is lost and we need to
- * revoke concurrent userspace access via GTT mmaps until the hardware has been
- * reset and the fence registers have been restored.
- */
-void i915_gem_revoke_fences(struct drm_i915_private *dev_priv)
-{
- int i;
-
- lockdep_assert_held(&dev_priv->drm.struct_mutex);
-
- for (i = 0; i < dev_priv->num_fence_regs; i++) {
- struct drm_i915_fence_reg *fence = &dev_priv->fence_regs[i];
-
- GEM_BUG_ON(fence->vma && fence->vma->fence != fence);
-
- if (fence->vma)
- i915_vma_revoke_mmap(fence->vma);
- }
-}
-
/**
* i915_gem_restore_fences - restore fence state
* @dev_priv: i915 device private
{
int i;
+ rcu_read_lock(); /* keep obj alive as we dereference */
for (i = 0; i < dev_priv->num_fence_regs; i++) {
struct drm_i915_fence_reg *reg = &dev_priv->fence_regs[i];
- struct i915_vma *vma = reg->vma;
+ struct i915_vma *vma = READ_ONCE(reg->vma);
GEM_BUG_ON(vma && vma->fence != reg);
* Commit delayed tiling changes if we have an object still
* attached to the fence, otherwise just clear the fence.
*/
- if (vma && !i915_gem_object_is_tiled(vma->obj)) {
- GEM_BUG_ON(!reg->dirty);
- GEM_BUG_ON(i915_vma_has_userfault(vma));
-
- list_move(®->link, &dev_priv->mm.fence_list);
- vma->fence = NULL;
+ if (vma && !i915_gem_object_is_tiled(vma->obj))
vma = NULL;
- }
fence_write(reg, vma);
- reg->vma = vma;
}
+ rcu_read_unlock();
}
/**
*/
swizzle_x = I915_BIT_6_SWIZZLE_NONE;
swizzle_y = I915_BIT_6_SWIZZLE_NONE;
- } else if (IS_MOBILE(dev_priv) ||
- IS_I915G(dev_priv) || IS_I945G(dev_priv)) {
+ } else if (IS_G45(dev_priv) || IS_I965G(dev_priv) || IS_G33(dev_priv)) {
+ /* The 965, G33, and newer, have a very flexible memory
+ * configuration. It will enable dual-channel mode
+ * (interleaving) on as much memory as it can, and the GPU
+ * will additionally sometimes enable different bit 6
+ * swizzling for tiled objects from the CPU.
+ *
+ * Here's what I found on the G965:
+ * slot fill memory size swizzling
+ * 0A 0B 1A 1B 1-ch 2-ch
+ * 512 0 0 0 512 0 O
+ * 512 0 512 0 16 1008 X
+ * 512 0 0 512 16 1008 X
+ * 0 512 0 512 16 1008 X
+ * 1024 1024 1024 0 2048 1024 O
+ *
+ * We could probably detect this based on either the DRB
+ * matching, which was the case for the swizzling required in
+ * the table above, or from the 1-ch value being less than
+ * the minimum size of a rank.
+ *
+ * Reports indicate that the swizzling actually
+ * varies depending upon page placement inside the
+ * channels, i.e. we see swizzled pages where the
+ * banks of memory are paired and unswizzled on the
+ * uneven portion, so leave that as unknown.
+ */
+ if (I915_READ16(C0DRB3) == I915_READ16(C1DRB3)) {
+ swizzle_x = I915_BIT_6_SWIZZLE_9_10;
+ swizzle_y = I915_BIT_6_SWIZZLE_9;
+ }
+ } else {
u32 dcc;
/* On 9xx chipsets, channel interleave by the CPU is
swizzle_x = I915_BIT_6_SWIZZLE_UNKNOWN;
swizzle_y = I915_BIT_6_SWIZZLE_UNKNOWN;
}
- } else {
- /* The 965, G33, and newer, have a very flexible memory
- * configuration. It will enable dual-channel mode
- * (interleaving) on as much memory as it can, and the GPU
- * will additionally sometimes enable different bit 6
- * swizzling for tiled objects from the CPU.
- *
- * Here's what I found on the G965:
- * slot fill memory size swizzling
- * 0A 0B 1A 1B 1-ch 2-ch
- * 512 0 0 0 512 0 O
- * 512 0 512 0 16 1008 X
- * 512 0 0 512 16 1008 X
- * 0 512 0 512 16 1008 X
- * 1024 1024 1024 0 2048 1024 O
- *
- * We could probably detect this based on either the DRB
- * matching, which was the case for the swizzling required in
- * the table above, or from the 1-ch value being less than
- * the minimum size of a rank.
- *
- * Reports indicate that the swizzling actually
- * varies depending upon page placement inside the
- * channels, i.e. we see swizzled pages where the
- * banks of memory are paired and unswizzled on the
- * uneven portion, so leave that as unknown.
- */
- if (I915_READ16(C0DRB3) == I915_READ16(C1DRB3)) {
- swizzle_x = I915_BIT_6_SWIZZLE_9_10;
- swizzle_y = I915_BIT_6_SWIZZLE_9;
- }
}
if (swizzle_x == I915_BIT_6_SWIZZLE_UNKNOWN ||
int i;
if (obj->bit_17 == NULL) {
- obj->bit_17 = kcalloc(BITS_TO_LONGS(page_count),
- sizeof(long), GFP_KERNEL);
+ obj->bit_17 = bitmap_zalloc(page_count, GFP_KERNEL);
if (obj->bit_17 == NULL) {
DRM_ERROR("Failed to allocate memory for bit 17 "
"record\n");
* for all.
*/
size = I915_GTT_PAGE_SIZE_4K;
- if (i915_vm_is_48bit(vm) &&
+ if (i915_vm_is_4lvl(vm) &&
HAS_PAGE_SIZES(vm->i915, I915_GTT_PAGE_SIZE_64K)) {
size = I915_GTT_PAGE_SIZE_64K;
gfp |= __GFP_NOWARN;
vm->scratch_page.page = page;
vm->scratch_page.daddr = addr;
- vm->scratch_page.order = order;
+ vm->scratch_order = order;
return 0;
unmap_page:
static void cleanup_scratch_page(struct i915_address_space *vm)
{
struct i915_page_dma *p = &vm->scratch_page;
+ int order = vm->scratch_order;
- dma_unmap_page(vm->dma, p->daddr, BIT(p->order) << PAGE_SHIFT,
+ dma_unmap_page(vm->dma, p->daddr, BIT(order) << PAGE_SHIFT,
PCI_DMA_BIDIRECTIONAL);
- __free_pages(p->page, p->order);
+ __free_pages(p->page, order);
}
static struct i915_page_table *alloc_pt(struct i915_address_space *vm)
pdp->page_directory = NULL;
}
-static inline bool use_4lvl(const struct i915_address_space *vm)
-{
- return i915_vm_is_48bit(vm);
-}
-
static struct i915_page_directory_pointer *
alloc_pdp(struct i915_address_space *vm)
{
struct i915_page_directory_pointer *pdp;
int ret = -ENOMEM;
- GEM_BUG_ON(!use_4lvl(vm));
+ GEM_BUG_ON(!i915_vm_is_4lvl(vm));
pdp = kzalloc(sizeof(*pdp), GFP_KERNEL);
if (!pdp)
{
__pdp_fini(pdp);
- if (!use_4lvl(vm))
+ if (!i915_vm_is_4lvl(vm))
return;
cleanup_px(vm, pdp);
memset_p((void **)pml4->pdps, vm->scratch_pdp, GEN8_PML4ES_PER_PML4);
}
-/* PDE TLBs are a pain to invalidate on GEN8+. When we modify
+/*
+ * PDE TLBs are a pain to invalidate on GEN8+. When we modify
* the page table structures, we mark them dirty so that
* context switching/execlist queuing code takes extra steps
* to ensure that tlbs are flushed.
*/
static void mark_tlbs_dirty(struct i915_hw_ppgtt *ppgtt)
{
- ppgtt->pd_dirty_rings = INTEL_INFO(ppgtt->vm.i915)->ring_mask;
+ ppgtt->pd_dirty_engines = ALL_ENGINES;
}
/* Removes entries from a single page table, releasing it if it's empty.
u64 start, u64 length)
{
unsigned int num_entries = gen8_pte_count(start, length);
- unsigned int pte = gen8_pte_index(start);
- unsigned int pte_end = pte + num_entries;
gen8_pte_t *vaddr;
GEM_BUG_ON(num_entries > pt->used_ptes);
return true;
vaddr = kmap_atomic_px(pt);
- while (pte < pte_end)
- vaddr[pte++] = vm->scratch_pte;
+ memset64(vaddr + gen8_pte_index(start), vm->scratch_pte, num_entries);
kunmap_atomic(vaddr);
return false;
gen8_ppgtt_pdpe_t *vaddr;
pdp->page_directory[pdpe] = pd;
- if (!use_4lvl(vm))
+ if (!i915_vm_is_4lvl(vm))
return;
vaddr = kmap_atomic_px(pdp);
struct i915_page_directory_pointer *pdp;
unsigned int pml4e;
- GEM_BUG_ON(!use_4lvl(vm));
+ GEM_BUG_ON(!i915_vm_is_4lvl(vm));
gen8_for_each_pml4e(pdp, pml4, start, length, pml4e) {
GEM_BUG_ON(pdp == vm->scratch_pdp);
GEM_BUG_ON(!clone->has_read_only);
- vm->scratch_page.order = clone->scratch_page.order;
+ vm->scratch_order = clone->scratch_order;
vm->scratch_pte = clone->scratch_pte;
vm->scratch_pt = clone->scratch_pt;
vm->scratch_pd = clone->scratch_pd;
vm->scratch_pte =
gen8_pte_encode(vm->scratch_page.daddr,
I915_CACHE_LLC,
- PTE_READ_ONLY);
+ vm->has_read_only);
vm->scratch_pt = alloc_pt(vm);
if (IS_ERR(vm->scratch_pt)) {
goto free_pt;
}
- if (use_4lvl(vm)) {
+ if (i915_vm_is_4lvl(vm)) {
vm->scratch_pdp = alloc_pdp(vm);
if (IS_ERR(vm->scratch_pdp)) {
ret = PTR_ERR(vm->scratch_pdp);
gen8_initialize_pt(vm, vm->scratch_pt);
gen8_initialize_pd(vm, vm->scratch_pd);
- if (use_4lvl(vm))
+ if (i915_vm_is_4lvl(vm))
gen8_initialize_pdp(vm, vm->scratch_pdp);
return 0;
enum vgt_g2v_type msg;
int i;
- if (use_4lvl(vm)) {
+ if (i915_vm_is_4lvl(vm)) {
const u64 daddr = px_dma(&ppgtt->pml4);
I915_WRITE(vgtif_reg(pdp[0].lo), lower_32_bits(daddr));
if (!vm->scratch_page.daddr)
return;
- if (use_4lvl(vm))
+ if (i915_vm_is_4lvl(vm))
free_pdp(vm, vm->scratch_pdp);
free_pd(vm, vm->scratch_pd);
free_pt(vm, vm->scratch_pt);
if (intel_vgpu_active(dev_priv))
gen8_ppgtt_notify_vgt(ppgtt, false);
- if (use_4lvl(vm))
+ if (i915_vm_is_4lvl(vm))
gen8_ppgtt_cleanup_4lvl(ppgtt);
else
gen8_ppgtt_cleanup_3lvl(&ppgtt->vm, &ppgtt->pdp);
return -ENOMEM;
}
+static void ppgtt_init(struct drm_i915_private *i915,
+ struct i915_hw_ppgtt *ppgtt)
+{
+ kref_init(&ppgtt->ref);
+
+ ppgtt->vm.i915 = i915;
+ ppgtt->vm.dma = &i915->drm.pdev->dev;
+ ppgtt->vm.total = BIT_ULL(INTEL_INFO(i915)->ppgtt_size);
+
+ i915_address_space_init(&ppgtt->vm, VM_CLASS_PPGTT);
+
+ ppgtt->vm.vma_ops.bind_vma = ppgtt_bind_vma;
+ ppgtt->vm.vma_ops.unbind_vma = ppgtt_unbind_vma;
+ ppgtt->vm.vma_ops.set_pages = ppgtt_set_pages;
+ ppgtt->vm.vma_ops.clear_pages = clear_pages;
+}
+
/*
* GEN8 legacy ppgtt programming is accomplished through a max 4 PDP registers
* with a net effect resembling a 2-level page table in normal x86 terms. Each
if (!ppgtt)
return ERR_PTR(-ENOMEM);
- kref_init(&ppgtt->ref);
+ ppgtt_init(i915, ppgtt);
- ppgtt->vm.i915 = i915;
- ppgtt->vm.dma = &i915->drm.pdev->dev;
-
- ppgtt->vm.total = HAS_FULL_48BIT_PPGTT(i915) ?
- 1ULL << 48 :
- 1ULL << 32;
-
- /* From bdw, there is support for read-only pages in the PPGTT. */
- ppgtt->vm.has_read_only = true;
-
- i915_address_space_init(&ppgtt->vm, VM_CLASS_PPGTT);
+ /*
+ * From bdw, there is hw support for read-only pages in the PPGTT.
+ *
+ * Gen11 has HSDES#:1807136187 unresolved. Disable ro support
+ * for now.
+ */
+ ppgtt->vm.has_read_only = INTEL_GEN(i915) != 11;
/* There are only few exceptions for gen >=6. chv and bxt.
* And we are not sure about the latter so play safe for now.
if (err)
goto err_free;
- if (use_4lvl(&ppgtt->vm)) {
+ if (i915_vm_is_4lvl(&ppgtt->vm)) {
err = setup_px(&ppgtt->vm, &ppgtt->pml4);
if (err)
goto err_scratch;
ppgtt->vm.cleanup = gen8_ppgtt_cleanup;
- ppgtt->vm.vma_ops.bind_vma = ppgtt_bind_vma;
- ppgtt->vm.vma_ops.unbind_vma = ppgtt_unbind_vma;
- ppgtt->vm.vma_ops.set_pages = ppgtt_set_pages;
- ppgtt->vm.vma_ops.clear_pages = clear_pages;
-
return ppgtt;
err_scratch:
while (num_entries) {
struct i915_page_table *pt = ppgtt->base.pd.page_table[pde++];
- const unsigned int end = min(pte + num_entries, GEN6_PTES);
- const unsigned int count = end - pte;
+ const unsigned int count = min(num_entries, GEN6_PTES - pte);
gen6_pte_t *vaddr;
GEM_BUG_ON(pt == vm->scratch_pt);
*/
vaddr = kmap_atomic_px(pt);
- do {
- vaddr[pte++] = scratch_pte;
- } while (pte < end);
+ memset32(vaddr + pte, scratch_pte, count);
kunmap_atomic(vaddr);
pte = 0;
GEM_BUG_ON(!IS_ALIGNED(size, I915_GTT_PAGE_SIZE));
GEM_BUG_ON(size > ggtt->vm.total);
- vma = kmem_cache_zalloc(i915->vmas, GFP_KERNEL);
+ vma = i915_vma_alloc();
if (!vma)
return ERR_PTR(-ENOMEM);
struct gen6_hw_ppgtt *ppgtt = to_gen6_ppgtt(base);
int err;
+ GEM_BUG_ON(ppgtt->base.vm.closed);
+
/*
* Workaround the limited maximum vma->pin_count and the aliasing_ppgtt
* which will be pinned into every active context.
i915_vma_unpin(ppgtt->vma);
}
+void gen6_ppgtt_unpin_all(struct i915_hw_ppgtt *base)
+{
+ struct gen6_hw_ppgtt *ppgtt = to_gen6_ppgtt(base);
+
+ if (!ppgtt->pin_count)
+ return;
+
+ ppgtt->pin_count = 0;
+ i915_vma_unpin(ppgtt->vma);
+}
+
static struct i915_hw_ppgtt *gen6_ppgtt_create(struct drm_i915_private *i915)
{
struct i915_ggtt * const ggtt = &i915->ggtt;
if (!ppgtt)
return ERR_PTR(-ENOMEM);
- kref_init(&ppgtt->base.ref);
-
- ppgtt->base.vm.i915 = i915;
- ppgtt->base.vm.dma = &i915->drm.pdev->dev;
-
- ppgtt->base.vm.total = I915_PDES * GEN6_PTES * I915_GTT_PAGE_SIZE;
-
- i915_address_space_init(&ppgtt->base.vm, VM_CLASS_PPGTT);
+ ppgtt_init(i915, &ppgtt->base);
ppgtt->base.vm.allocate_va_range = gen6_alloc_va_range;
ppgtt->base.vm.clear_range = gen6_ppgtt_clear_range;
ppgtt->base.vm.insert_entries = gen6_ppgtt_insert_entries;
ppgtt->base.vm.cleanup = gen6_ppgtt_cleanup;
- ppgtt->base.vm.vma_ops.bind_vma = ppgtt_bind_vma;
- ppgtt->base.vm.vma_ops.unbind_vma = ppgtt_unbind_vma;
- ppgtt->base.vm.vma_ops.set_pages = ppgtt_set_pages;
- ppgtt->base.vm.vma_ops.clear_pages = clear_pages;
-
ppgtt->base.vm.pte_encode = ggtt->vm.pte_encode;
err = gen6_ppgtt_init_scratch(ppgtt);
}
struct i915_hw_ppgtt *
-i915_ppgtt_create(struct drm_i915_private *i915,
- struct drm_i915_file_private *fpriv)
+i915_ppgtt_create(struct drm_i915_private *i915)
{
struct i915_hw_ppgtt *ppgtt;
if (IS_ERR(ppgtt))
return ppgtt;
- ppgtt->vm.file = fpriv;
-
trace_i915_ppgtt_create(&ppgtt->vm);
return ppgtt;
}
-void i915_ppgtt_close(struct i915_address_space *vm)
-{
- GEM_BUG_ON(vm->closed);
- vm->closed = true;
-}
-
static void ppgtt_destroy_vma(struct i915_address_space *vm)
{
struct list_head *phases[] = {
struct i915_hw_ppgtt *ppgtt;
int err;
- ppgtt = i915_ppgtt_create(i915, ERR_PTR(-EPERM));
+ ppgtt = i915_ppgtt_create(i915);
if (IS_ERR(ppgtt))
return PTR_ERR(ppgtt);
}
ret = 0;
- if (unlikely(IS_ERR(vma->pages))) {
+ if (IS_ERR(vma->pages)) {
ret = PTR_ERR(vma->pages);
vma->pages = NULL;
DRM_ERROR("Failed to get pages for VMA view type %u (%d)!\n",
struct i915_page_dma {
struct page *page;
- int order;
union {
dma_addr_t daddr;
#define VM_CLASS_PPGTT 1
u64 scratch_pte;
+ int scratch_order;
struct i915_page_dma scratch_page;
struct i915_page_table *scratch_pt;
struct i915_page_directory *scratch_pd;
#define i915_is_ggtt(vm) ((vm)->is_ggtt)
static inline bool
-i915_vm_is_48bit(const struct i915_address_space *vm)
+i915_vm_is_4lvl(const struct i915_address_space *vm)
{
return (vm->total - 1) >> 32;
}
static inline bool
i915_vm_has_scratch_64K(struct i915_address_space *vm)
{
- return vm->scratch_page.order == get_order(I915_GTT_PAGE_SIZE_64K);
+ return vm->scratch_order == get_order(I915_GTT_PAGE_SIZE_64K);
}
/* The Graphics Translation Table is the way in which GEN hardware translates a
struct i915_address_space vm;
struct kref ref;
- unsigned long pd_dirty_rings;
+ intel_engine_mask_t pd_dirty_engines;
union {
struct i915_pml4 pml4; /* GEN8+ & 48b PPGTT */
struct i915_page_directory_pointer pdp; /* GEN8+ */
struct i915_page_directory pd; /* GEN6-7 */
};
+
+ u32 user_handle;
};
struct gen6_hw_ppgtt {
static inline unsigned int
i915_pdpes_per_pdp(const struct i915_address_space *vm)
{
- if (i915_vm_is_48bit(vm))
+ if (i915_vm_is_4lvl(vm))
return GEN8_PML4ES_PER_PML4;
return GEN8_3LVL_PDPES;
void i915_ggtt_cleanup_hw(struct drm_i915_private *dev_priv);
int i915_ppgtt_init_hw(struct drm_i915_private *dev_priv);
+
+struct i915_hw_ppgtt *i915_ppgtt_create(struct drm_i915_private *dev_priv);
void i915_ppgtt_release(struct kref *kref);
-struct i915_hw_ppgtt *i915_ppgtt_create(struct drm_i915_private *dev_priv,
- struct drm_i915_file_private *fpriv);
-void i915_ppgtt_close(struct i915_address_space *vm);
-static inline void i915_ppgtt_get(struct i915_hw_ppgtt *ppgtt)
+
+static inline struct i915_hw_ppgtt *i915_ppgtt_get(struct i915_hw_ppgtt *ppgtt)
{
- if (ppgtt)
- kref_get(&ppgtt->ref);
+ kref_get(&ppgtt->ref);
+ return ppgtt;
}
+
static inline void i915_ppgtt_put(struct i915_hw_ppgtt *ppgtt)
{
if (ppgtt)
int gen6_ppgtt_pin(struct i915_hw_ppgtt *base);
void gen6_ppgtt_unpin(struct i915_hw_ppgtt *base);
+void gen6_ppgtt_unpin_all(struct i915_hw_ppgtt *base);
void i915_check_and_clear_faults(struct drm_i915_private *dev_priv);
void i915_gem_suspend_gtt_mappings(struct drm_i915_private *dev_priv);
if (overflows_type(size, obj->base.size))
return ERR_PTR(-E2BIG);
- obj = i915_gem_object_alloc(i915);
+ obj = i915_gem_object_alloc();
if (!obj)
return ERR_PTR(-ENOMEM);
#include "i915_drv.h"
#include "i915_gem_object.h"
+#include "i915_globals.h"
+
+static struct i915_global_object {
+ struct i915_global base;
+ struct kmem_cache *slab_objects;
+} global;
+
+struct drm_i915_gem_object *i915_gem_object_alloc(void)
+{
+ return kmem_cache_zalloc(global.slab_objects, GFP_KERNEL);
+}
+
+void i915_gem_object_free(struct drm_i915_gem_object *obj)
+{
+ return kmem_cache_free(global.slab_objects, obj);
+}
/**
* Mark up the object's coherency levels for a given cache_level
obj->cache_dirty =
!(obj->cache_coherent & I915_BO_CACHE_COHERENT_FOR_WRITE);
}
+
+static void i915_global_objects_shrink(void)
+{
+ kmem_cache_shrink(global.slab_objects);
+}
+
+static void i915_global_objects_exit(void)
+{
+ kmem_cache_destroy(global.slab_objects);
+}
+
+static struct i915_global_object global = { {
+ .shrink = i915_global_objects_shrink,
+ .exit = i915_global_objects_exit,
+} };
+
+int __init i915_global_objects_init(void)
+{
+ global.slab_objects =
+ KMEM_CACHE(drm_i915_gem_object, SLAB_HWCACHE_ALIGN);
+ if (!global.slab_objects)
+ return -ENOMEM;
+
+ i915_global_register(&global.base);
+ return 0;
+}
return container_of(gem, struct drm_i915_gem_object, base);
}
+struct drm_i915_gem_object *i915_gem_object_alloc(void);
+void i915_gem_object_free(struct drm_i915_gem_object *obj);
+
/**
* i915_gem_object_lookup_rcu - look up a temporary GEM object from its handle
* @filp: DRM file private date
unsigned int cache_level);
void i915_gem_object_flush_if_display(struct drm_i915_gem_object *obj);
-#endif
+void __i915_gem_object_release_shmem(struct drm_i915_gem_object *obj,
+ struct sg_table *pages,
+ bool needs_clflush);
+#endif
static const struct intel_renderstate_rodata *
render_state_get_rodata(const struct intel_engine_cs *engine)
{
- if (engine->id != RCS)
+ if (engine->id != RCS0)
return NULL;
switch (INTEL_GEN(engine->i915)) {
drm_clflush_virt_range(d, i * sizeof(u32));
kunmap_atomic(d);
- ret = i915_gem_object_set_to_gtt_domain(so->obj, false);
+ ret = 0;
out:
i915_gem_obj_finish_shmem_access(so->obj);
return ret;
struct drm_i915_gem_object *obj;
unsigned int cache_level;
- obj = i915_gem_object_alloc(dev_priv);
+ obj = i915_gem_object_alloc();
if (obj == NULL)
return NULL;
/* Try to preallocate memory required to save swizzling on put-pages */
if (i915_gem_object_needs_bit17_swizzle(obj)) {
if (!obj->bit_17) {
- obj->bit_17 = kcalloc(BITS_TO_LONGS(obj->base.size >> PAGE_SHIFT),
- sizeof(long), GFP_KERNEL);
+ obj->bit_17 = bitmap_zalloc(obj->base.size >> PAGE_SHIFT,
+ GFP_KERNEL);
}
} else {
- kfree(obj->bit_17);
+ bitmap_free(obj->bit_17);
obj->bit_17 = NULL;
}
if (!pages)
return;
- if (obj->mm.madv != I915_MADV_WILLNEED)
- obj->mm.dirty = false;
-
+ __i915_gem_object_release_shmem(obj, pages, true);
i915_gem_gtt_finish_pages(obj, pages);
for_each_sgt_page(page, sgt_iter, pages) {
return -ENODEV;
}
- obj = i915_gem_object_alloc(dev_priv);
+ obj = i915_gem_object_alloc();
if (obj == NULL)
return -ENOMEM;
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2019 Intel Corporation
+ */
+
+#include <linux/slab.h>
+#include <linux/workqueue.h>
+
+#include "i915_active.h"
+#include "i915_gem_context.h"
+#include "i915_gem_object.h"
+#include "i915_globals.h"
+#include "i915_request.h"
+#include "i915_scheduler.h"
+#include "i915_vma.h"
+
+static LIST_HEAD(globals);
+
+static atomic_t active;
+static atomic_t epoch;
+static struct park_work {
+ struct rcu_work work;
+ int epoch;
+} park;
+
+static void i915_globals_shrink(void)
+{
+ struct i915_global *global;
+
+ /*
+ * kmem_cache_shrink() discards empty slabs and reorders partially
+ * filled slabs to prioritise allocating from the mostly full slabs,
+ * with the aim of reducing fragmentation.
+ */
+ list_for_each_entry(global, &globals, link)
+ global->shrink();
+}
+
+static void __i915_globals_park(struct work_struct *work)
+{
+ /* Confirm nothing woke up in the last grace period */
+ if (park.epoch == atomic_read(&epoch))
+ i915_globals_shrink();
+}
+
+void __init i915_global_register(struct i915_global *global)
+{
+ GEM_BUG_ON(!global->shrink);
+ GEM_BUG_ON(!global->exit);
+
+ list_add_tail(&global->link, &globals);
+}
+
+static void __i915_globals_cleanup(void)
+{
+ struct i915_global *global, *next;
+
+ list_for_each_entry_safe_reverse(global, next, &globals, link)
+ global->exit();
+}
+
+static __initconst int (* const initfn[])(void) = {
+ i915_global_active_init,
+ i915_global_context_init,
+ i915_global_gem_context_init,
+ i915_global_objects_init,
+ i915_global_request_init,
+ i915_global_scheduler_init,
+ i915_global_vma_init,
+};
+
+int __init i915_globals_init(void)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(initfn); i++) {
+ int err;
+
+ err = initfn[i]();
+ if (err) {
+ __i915_globals_cleanup();
+ return err;
+ }
+ }
+
+ INIT_RCU_WORK(&park.work, __i915_globals_park);
+ return 0;
+}
+
+void i915_globals_park(void)
+{
+ /*
+ * Defer shrinking the global slab caches (and other work) until
+ * after a RCU grace period has completed with no activity. This
+ * is to try and reduce the latency impact on the consumers caused
+ * by us shrinking the caches the same time as they are trying to
+ * allocate, with the assumption being that if we idle long enough
+ * for an RCU grace period to elapse since the last use, it is likely
+ * to be longer until we need the caches again.
+ */
+ if (!atomic_dec_and_test(&active))
+ return;
+
+ park.epoch = atomic_inc_return(&epoch);
+ queue_rcu_work(system_wq, &park.work);
+}
+
+void i915_globals_unpark(void)
+{
+ atomic_inc(&epoch);
+ atomic_inc(&active);
+}
+
+void __exit i915_globals_exit(void)
+{
+ /* Flush any residual park_work */
+ atomic_inc(&epoch);
+ flush_rcu_work(&park.work);
+
+ __i915_globals_cleanup();
+
+ /* And ensure that our DESTROY_BY_RCU slabs are truly destroyed */
+ rcu_barrier();
+}
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef _I915_GLOBALS_H_
+#define _I915_GLOBALS_H_
+
+typedef void (*i915_global_func_t)(void);
+
+struct i915_global {
+ struct list_head link;
+
+ i915_global_func_t shrink;
+ i915_global_func_t exit;
+};
+
+void i915_global_register(struct i915_global *global);
+
+int i915_globals_init(void);
+void i915_globals_park(void);
+void i915_globals_unpark(void);
+void i915_globals_exit(void);
+
+/* constructors */
+int i915_global_active_init(void);
+int i915_global_context_init(void);
+int i915_global_gem_context_init(void);
+int i915_global_objects_init(void);
+int i915_global_request_init(void);
+int i915_global_scheduler_init(void);
+int i915_global_vma_init(void);
+
+#endif /* _I915_GLOBALS_H_ */
err_printf(m, "%s [%d]:\n", name, count);
while (count--) {
- err_printf(m, " %08x_%08x %8u %02x %02x %02x",
+ err_printf(m, " %08x_%08x %8u %02x %02x",
upper_32_bits(err->gtt_offset),
lower_32_bits(err->gtt_offset),
err->size,
err->read_domains,
- err->write_domain,
- err->wseqno);
+ err->write_domain);
err_puts(m, tiling_flag(err->tiling));
err_puts(m, dirty_flag(err->dirty));
err_puts(m, purgeable_flag(err->purgeable));
err_puts(m, err->userptr ? " userptr" : "");
- err_puts(m, err->engine != -1 ? " " : "");
- err_puts(m, engine_name(m->i915, err->engine));
err_puts(m, i915_cache_level_str(m->i915, err->cache_level));
if (err->name)
err_printf(m, " INSTDONE: 0x%08x\n",
ee->instdone.instdone);
- if (ee->engine_id != RCS || INTEL_GEN(m->i915) <= 3)
+ if (ee->engine_id != RCS0 || INTEL_GEN(m->i915) <= 3)
return;
err_printf(m, " SC_INSTDONE: 0x%08x\n",
ee->instdone.row[slice][subslice]);
}
-static const char *bannable(const struct drm_i915_error_context *ctx)
-{
- return ctx->bannable ? "" : " (unbannable)";
-}
-
static void error_print_request(struct drm_i915_error_state_buf *m,
const char *prefix,
const struct drm_i915_error_request *erq,
if (!erq->seqno)
return;
- err_printf(m, "%s pid %d, ban score %d, seqno %8x:%08x%s%s, prio %d, emitted %dms, start %08x, head %08x, tail %08x\n",
- prefix, erq->pid, erq->ban_score,
- erq->context, erq->seqno,
+ err_printf(m, "%s pid %d, seqno %8x:%08x%s%s, prio %d, emitted %dms, start %08x, head %08x, tail %08x\n",
+ prefix, erq->pid, erq->context, erq->seqno,
test_bit(DMA_FENCE_FLAG_SIGNALED_BIT,
&erq->flags) ? "!" : "",
test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT,
const char *header,
const struct drm_i915_error_context *ctx)
{
- err_printf(m, "%s%s[%d] user_handle %d hw_id %d, prio %d, ban score %d%s guilty %d active %d\n",
- header, ctx->comm, ctx->pid, ctx->handle, ctx->hw_id,
- ctx->sched_attr.priority, ctx->ban_score, bannable(ctx),
- ctx->guilty, ctx->active);
+ err_printf(m, "%s%s[%d] hw_id %d, prio %d, guilty %d active %d\n",
+ header, ctx->comm, ctx->pid, ctx->hw_id,
+ ctx->sched_attr.priority, ctx->guilty, ctx->active);
}
static void error_print_engine(struct drm_i915_error_state_buf *m,
if (INTEL_GEN(m->i915) >= 6) {
err_printf(m, " RC PSMI: 0x%08x\n", ee->rc_psmi);
err_printf(m, " FAULT_REG: 0x%08x\n", ee->fault_reg);
- err_printf(m, " SYNC_0: 0x%08x\n",
- ee->semaphore_mboxes[0]);
- err_printf(m, " SYNC_1: 0x%08x\n",
- ee->semaphore_mboxes[1]);
- if (HAS_VEBOX(m->i915))
- err_printf(m, " SYNC_2: 0x%08x\n",
- ee->semaphore_mboxes[2]);
}
if (HAS_PPGTT(m->i915)) {
err_printf(m, " GFX_MODE: 0x%08x\n", ee->vm_info.gfx_mode);
ee->vm_info.pp_dir_base);
}
}
- err_printf(m, " seqno: 0x%08x\n", ee->seqno);
- err_printf(m, " last_seqno: 0x%08x\n", ee->last_seqno);
err_printf(m, " ring->head: 0x%08x\n", ee->cpu_ring_head);
err_printf(m, " ring->tail: 0x%08x\n", ee->cpu_ring_tail);
err_printf(m, " hangcheck timestamp: %dms (%lu%s)\n",
if (!error->engine[i].context.pid)
continue;
- err_printf(m, "Active process (on ring %s): %s [%d], score %d%s\n",
+ err_printf(m, "Active process (on ring %s): %s [%d]\n",
engine_name(m->i915, i),
error->engine[i].context.comm,
- error->engine[i].context.pid,
- error->engine[i].context.ban_score,
- bannable(&error->engine[i].context));
+ error->engine[i].context.pid);
}
err_printf(m, "Reset count: %u\n", error->reset_count);
err_printf(m, "Suspend count: %u\n", error->suspend_count);
err_printf(m, "Platform: %s\n", intel_platform_name(error->device_info.platform));
+ err_printf(m, "Subplatform: 0x%x\n",
+ intel_subplatform(&error->runtime_info,
+ error->device_info.platform));
err_print_pciid(m, m->i915);
err_printf(m, "IOMMU enabled?: %d\n", error->iommu);
if (obj) {
err_puts(m, m->i915->engine[i]->name);
if (ee->context.pid)
- err_printf(m, " (submitted by %s [%d], ctx %d [%d], score %d%s)",
+ err_printf(m, " (submitted by %s [%d])",
ee->context.comm,
- ee->context.pid,
- ee->context.handle,
- ee->context.hw_id,
- ee->context.ban_score,
- bannable(&ee->context));
+ ee->context.pid);
err_printf(m, " --- gtt_offset = 0x%08x %08x\n",
upper_32_bits(obj->gtt_offset),
lower_32_bits(obj->gtt_offset));
return dst;
}
-/* The error capture is special as tries to run underneath the normal
- * locking rules - so we use the raw version of the i915_active_request lookup.
- */
-static inline u32
-__active_get_seqno(struct i915_active_request *active)
-{
- struct i915_request *request;
-
- request = __i915_active_request_peek(active);
- return request ? request->global_seqno : 0;
-}
-
-static inline int
-__active_get_engine_id(struct i915_active_request *active)
-{
- struct i915_request *request;
-
- request = __i915_active_request_peek(active);
- return request ? request->engine->id : -1;
-}
-
static void capture_bo(struct drm_i915_error_buffer *err,
struct i915_vma *vma)
{
err->size = obj->base.size;
err->name = obj->base.name;
- err->wseqno = __active_get_seqno(&obj->frontbuffer_write);
- err->engine = __active_get_engine_id(&obj->frontbuffer_write);
-
err->gtt_offset = vma->node.start;
err->read_domains = obj->read_domains;
err->write_domain = obj->write_domain;
* It's only a small step better than a random number in its current form.
*/
static u32 i915_error_generate_code(struct i915_gpu_state *error,
- unsigned long engine_mask)
+ intel_engine_mask_t engine_mask)
{
/*
* IPEHR would be an ideal way to detect errors, as it's the gross
error->nfence = i;
}
-static void gen6_record_semaphore_state(struct intel_engine_cs *engine,
- struct drm_i915_error_engine *ee)
-{
- struct drm_i915_private *dev_priv = engine->i915;
-
- ee->semaphore_mboxes[0] = I915_READ(RING_SYNC_0(engine->mmio_base));
- ee->semaphore_mboxes[1] = I915_READ(RING_SYNC_1(engine->mmio_base));
- if (HAS_VEBOX(dev_priv))
- ee->semaphore_mboxes[2] =
- I915_READ(RING_SYNC_2(engine->mmio_base));
-}
-
static void error_record_engine_registers(struct i915_gpu_state *error,
struct intel_engine_cs *engine,
struct drm_i915_error_engine *ee)
struct drm_i915_private *dev_priv = engine->i915;
if (INTEL_GEN(dev_priv) >= 6) {
- ee->rc_psmi = I915_READ(RING_PSMI_CTL(engine->mmio_base));
- if (INTEL_GEN(dev_priv) >= 8) {
+ ee->rc_psmi = ENGINE_READ(engine, RING_PSMI_CTL);
+ if (INTEL_GEN(dev_priv) >= 8)
ee->fault_reg = I915_READ(GEN8_RING_FAULT_REG);
- } else {
- gen6_record_semaphore_state(engine, ee);
+ else
ee->fault_reg = I915_READ(RING_FAULT_REG(engine));
- }
}
if (INTEL_GEN(dev_priv) >= 4) {
- ee->faddr = I915_READ(RING_DMA_FADD(engine->mmio_base));
- ee->ipeir = I915_READ(RING_IPEIR(engine->mmio_base));
- ee->ipehr = I915_READ(RING_IPEHR(engine->mmio_base));
- ee->instps = I915_READ(RING_INSTPS(engine->mmio_base));
- ee->bbaddr = I915_READ(RING_BBADDR(engine->mmio_base));
+ ee->faddr = ENGINE_READ(engine, RING_DMA_FADD);
+ ee->ipeir = ENGINE_READ(engine, RING_IPEIR);
+ ee->ipehr = ENGINE_READ(engine, RING_IPEHR);
+ ee->instps = ENGINE_READ(engine, RING_INSTPS);
+ ee->bbaddr = ENGINE_READ(engine, RING_BBADDR);
if (INTEL_GEN(dev_priv) >= 8) {
- ee->faddr |= (u64) I915_READ(RING_DMA_FADD_UDW(engine->mmio_base)) << 32;
- ee->bbaddr |= (u64) I915_READ(RING_BBADDR_UDW(engine->mmio_base)) << 32;
+ ee->faddr |= (u64)ENGINE_READ(engine, RING_DMA_FADD_UDW) << 32;
+ ee->bbaddr |= (u64)ENGINE_READ(engine, RING_BBADDR_UDW) << 32;
}
- ee->bbstate = I915_READ(RING_BBSTATE(engine->mmio_base));
+ ee->bbstate = ENGINE_READ(engine, RING_BBSTATE);
} else {
- ee->faddr = I915_READ(DMA_FADD_I8XX);
- ee->ipeir = I915_READ(IPEIR);
- ee->ipehr = I915_READ(IPEHR);
+ ee->faddr = ENGINE_READ(engine, DMA_FADD_I8XX);
+ ee->ipeir = ENGINE_READ(engine, IPEIR);
+ ee->ipehr = ENGINE_READ(engine, IPEHR);
}
intel_engine_get_instdone(engine, &ee->instdone);
- ee->instpm = I915_READ(RING_INSTPM(engine->mmio_base));
+ ee->instpm = ENGINE_READ(engine, RING_INSTPM);
ee->acthd = intel_engine_get_active_head(engine);
- ee->seqno = intel_engine_get_seqno(engine);
- ee->last_seqno = intel_engine_last_submit(engine);
- ee->start = I915_READ_START(engine);
- ee->head = I915_READ_HEAD(engine);
- ee->tail = I915_READ_TAIL(engine);
- ee->ctl = I915_READ_CTL(engine);
+ ee->start = ENGINE_READ(engine, RING_START);
+ ee->head = ENGINE_READ(engine, RING_HEAD);
+ ee->tail = ENGINE_READ(engine, RING_TAIL);
+ ee->ctl = ENGINE_READ(engine, RING_CTL);
if (INTEL_GEN(dev_priv) > 2)
- ee->mode = I915_READ_MODE(engine);
+ ee->mode = ENGINE_READ(engine, RING_MI_MODE);
if (!HWS_NEEDS_PHYSICAL(dev_priv)) {
i915_reg_t mmio;
if (IS_GEN(dev_priv, 7)) {
switch (engine->id) {
default:
- case RCS:
+ MISSING_CASE(engine->id);
+ case RCS0:
mmio = RENDER_HWS_PGA_GEN7;
break;
- case BCS:
+ case BCS0:
mmio = BLT_HWS_PGA_GEN7;
break;
- case VCS:
+ case VCS0:
mmio = BSD_HWS_PGA_GEN7;
break;
- case VECS:
+ case VECS0:
mmio = VEBOX_HWS_PGA_GEN7;
break;
}
ee->vm_info.gfx_mode = I915_READ(RING_MODE_GEN7(engine));
- if (IS_GEN(dev_priv, 6))
+ if (IS_GEN(dev_priv, 6)) {
ee->vm_info.pp_dir_base =
- I915_READ(RING_PP_DIR_BASE_READ(engine));
- else if (IS_GEN(dev_priv, 7))
+ ENGINE_READ(engine, RING_PP_DIR_BASE_READ);
+ } else if (IS_GEN(dev_priv, 7)) {
ee->vm_info.pp_dir_base =
- I915_READ(RING_PP_DIR_BASE(engine));
- else if (INTEL_GEN(dev_priv) >= 8)
+ ENGINE_READ(engine, RING_PP_DIR_BASE);
+ } else if (INTEL_GEN(dev_priv) >= 8) {
+ u32 base = engine->mmio_base;
+
for (i = 0; i < 4; i++) {
ee->vm_info.pdp[i] =
- I915_READ(GEN8_RING_PDP_UDW(engine, i));
+ I915_READ(GEN8_RING_PDP_UDW(base, i));
ee->vm_info.pdp[i] <<= 32;
ee->vm_info.pdp[i] |=
- I915_READ(GEN8_RING_PDP_LDW(engine, i));
+ I915_READ(GEN8_RING_PDP_LDW(base, i));
}
+ }
}
}
struct i915_gem_context *ctx = request->gem_context;
erq->flags = request->fence.flags;
- erq->context = ctx->hw_id;
+ erq->context = request->fence.context;
+ erq->seqno = request->fence.seqno;
erq->sched_attr = request->sched.attr;
- erq->ban_score = atomic_read(&ctx->ban_score);
- erq->seqno = request->global_seqno;
erq->jiffies = request->emitted_jiffies;
erq->start = i915_ggtt_offset(request->ring->vma);
erq->head = request->head;
rcu_read_unlock();
}
- e->handle = ctx->user_handle;
e->hw_id = ctx->hw_id;
e->sched_attr = ctx->sched;
- e->ban_score = atomic_read(&ctx->ban_score);
- e->bannable = i915_gem_context_is_bannable(ctx);
e->guilty = atomic_read(&ctx->guilty_count);
e->active = atomic_read(&ctx->active_count);
}
error_record_engine_registers(error, engine, ee);
error_record_engine_execlists(engine, ee);
- request = i915_gem_find_active_request(engine);
+ request = intel_engine_find_active_request(engine);
if (request) {
struct i915_gem_context *ctx = request->gem_context;
struct intel_ring *ring;
}
if (INTEL_GEN(dev_priv) >= 5)
- error->ccid = I915_READ(CCID);
+ error->ccid = I915_READ(CCID(RENDER_RING_BASE));
/* 3: Feature specific registers */
if (IS_GEN_RANGE(dev_priv, 6, 7)) {
error->gtier[0] = I915_READ(GTIER);
error->ngtier = 1;
} else if (IS_GEN(dev_priv, 2)) {
- error->ier = I915_READ16(IER);
+ error->ier = I915_READ16(GEN2_IER);
} else if (!IS_VALLEYVIEW(dev_priv)) {
- error->ier = I915_READ(IER);
+ error->ier = I915_READ(GEN2_IER);
}
error->eir = I915_READ(EIR);
error->pgtbl_er = I915_READ(PGTBL_ER);
}
static const char *
-error_msg(struct i915_gpu_state *error, unsigned long engines, const char *msg)
+error_msg(struct i915_gpu_state *error,
+ intel_engine_mask_t engines, const char *msg)
{
int len;
int i;
engines &= ~BIT(i);
len = scnprintf(error->error_msg, sizeof(error->error_msg),
- "GPU HANG: ecode %d:%lx:0x%08x",
+ "GPU HANG: ecode %d:%x:0x%08x",
INTEL_GEN(error->i915), engines,
i915_error_generate_code(error, engines));
if (engines) {
* to pick up.
*/
void i915_capture_error_state(struct drm_i915_private *i915,
- unsigned long engine_mask,
+ intel_engine_mask_t engine_mask,
const char *msg)
{
static bool warned;
u32 cpu_ring_head;
u32 cpu_ring_tail;
- u32 last_seqno;
-
/* Register state */
u32 start;
u32 tail;
u32 bbstate;
u32 instpm;
u32 instps;
- u32 seqno;
u64 bbaddr;
u64 acthd;
u32 fault_reg;
u64 faddr;
u32 rc_psmi; /* sleep state */
- u32 semaphore_mboxes[I915_NUM_ENGINES - 1];
struct intel_instdone instdone;
struct drm_i915_error_context {
char comm[TASK_COMM_LEN];
pid_t pid;
- u32 handle;
u32 hw_id;
- int ban_score;
int active;
int guilty;
- bool bannable;
struct i915_sched_attr sched_attr;
} context;
long jiffies;
pid_t pid;
u32 context;
- int ban_score;
u32 seqno;
u32 start;
u32 head;
struct drm_i915_error_buffer {
u32 size;
u32 name;
- u32 wseqno;
u64 gtt_offset;
u32 read_domains;
u32 write_domain;
u32 dirty:1;
u32 purgeable:1;
u32 userptr:1;
- s32 engine:4;
u32 cache_level:3;
} *active_bo[I915_NUM_ENGINES], *pinned_bo;
u32 active_bo_count[I915_NUM_ENGINES], pinned_bo_count;
atomic_t pending_fb_pin;
- /**
- * State variable controlling the reset flow and count
- *
- * This is a counter which gets incremented when reset is triggered,
- *
- * Before the reset commences, the I915_RESET_BACKOFF bit is set
- * meaning that any waiters holding onto the struct_mutex should
- * relinquish the lock immediately in order for the reset to start.
- *
- * If reset is not completed successfully, the I915_WEDGE bit is
- * set meaning that hardware is terminally sour and there is no
- * recovery. All waiters on the reset_queue will be woken when
- * that happens.
- *
- * This counter is used by the wait_seqno code to notice that reset
- * event happened and it needs to restart the entire ioctl (since most
- * likely the seqno it waited for won't ever signal anytime soon).
- *
- * This is important for lock-free wait paths, where no contended lock
- * naturally enforces the correct ordering between the bail-out of the
- * waiter and the gpu reset work code.
- */
- unsigned long reset_count;
-
/**
* flags: Control various stages of the GPU reset
*
- * #I915_RESET_BACKOFF - When we start a reset, we want to stop any
- * other users acquiring the struct_mutex. To do this we set the
- * #I915_RESET_BACKOFF bit in the error flags when we detect a reset
- * and then check for that bit before acquiring the struct_mutex (in
- * i915_mutex_lock_interruptible()?). I915_RESET_BACKOFF serves a
- * secondary role in preventing two concurrent global reset attempts.
+ * #I915_RESET_BACKOFF - When we start a global reset, we need to
+ * serialise with any other users attempting to do the same, and
+ * any global resources that may be clobber by the reset (such as
+ * FENCE registers).
*
* #I915_RESET_ENGINE[num_engines] - Since the driver doesn't need to
* acquire the struct_mutex to reset an engine, we need an explicit
#define I915_RESET_ENGINE 2
#define I915_WEDGED (BITS_PER_LONG - 1)
+ /** Number of times the device has been reset (global) */
+ u32 reset_count;
+
/** Number of times an engine has been reset */
u32 reset_engine_count[I915_NUM_ENGINES];
*/
wait_queue_head_t reset_queue;
+ struct srcu_struct reset_backoff_srcu;
+
struct i915_gpu_restart *restart;
};
struct i915_gpu_state *i915_capture_gpu_state(struct drm_i915_private *i915);
void i915_capture_error_state(struct drm_i915_private *dev_priv,
- unsigned long engine_mask,
+ intel_engine_mask_t engine_mask,
const char *error_msg);
static inline struct i915_gpu_state *
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/sysrq.h>
-#include <linux/slab.h>
#include <linux/circ_buf.h>
-#include <drm/drm_irq.h>
+#include <linux/cpuidle.h>
+#include <linux/slab.h>
+#include <linux/sysrq.h>
+
#include <drm/drm_drv.h>
+#include <drm/drm_irq.h>
#include <drm/i915_drm.h>
+
#include "i915_drv.h"
#include "i915_trace.h"
#include "intel_drv.h"
+#include "intel_psr.h"
/**
* DOC: interrupt handling
[HPD_PORT_F] = SDE_TC4_HOTPLUG_ICP
};
-/* IIR can theoretically queue up two events. Be paranoid. */
-#define GEN8_IRQ_RESET_NDX(type, which) do { \
- I915_WRITE(GEN8_##type##_IMR(which), 0xffffffff); \
- POSTING_READ(GEN8_##type##_IMR(which)); \
- I915_WRITE(GEN8_##type##_IER(which), 0); \
- I915_WRITE(GEN8_##type##_IIR(which), 0xffffffff); \
- POSTING_READ(GEN8_##type##_IIR(which)); \
- I915_WRITE(GEN8_##type##_IIR(which), 0xffffffff); \
- POSTING_READ(GEN8_##type##_IIR(which)); \
-} while (0)
-
-#define GEN3_IRQ_RESET(type) do { \
- I915_WRITE(type##IMR, 0xffffffff); \
- POSTING_READ(type##IMR); \
- I915_WRITE(type##IER, 0); \
- I915_WRITE(type##IIR, 0xffffffff); \
- POSTING_READ(type##IIR); \
- I915_WRITE(type##IIR, 0xffffffff); \
- POSTING_READ(type##IIR); \
-} while (0)
-
-#define GEN2_IRQ_RESET(type) do { \
- I915_WRITE16(type##IMR, 0xffff); \
- POSTING_READ16(type##IMR); \
- I915_WRITE16(type##IER, 0); \
- I915_WRITE16(type##IIR, 0xffff); \
- POSTING_READ16(type##IIR); \
- I915_WRITE16(type##IIR, 0xffff); \
- POSTING_READ16(type##IIR); \
-} while (0)
+static void gen3_irq_reset(struct intel_uncore *uncore, i915_reg_t imr,
+ i915_reg_t iir, i915_reg_t ier)
+{
+ intel_uncore_write(uncore, imr, 0xffffffff);
+ intel_uncore_posting_read(uncore, imr);
+
+ intel_uncore_write(uncore, ier, 0);
+
+ /* IIR can theoretically queue up two events. Be paranoid. */
+ intel_uncore_write(uncore, iir, 0xffffffff);
+ intel_uncore_posting_read(uncore, iir);
+ intel_uncore_write(uncore, iir, 0xffffffff);
+ intel_uncore_posting_read(uncore, iir);
+}
+
+static void gen2_irq_reset(struct intel_uncore *uncore)
+{
+ intel_uncore_write16(uncore, GEN2_IMR, 0xffff);
+ intel_uncore_posting_read16(uncore, GEN2_IMR);
+
+ intel_uncore_write16(uncore, GEN2_IER, 0);
+
+ /* IIR can theoretically queue up two events. Be paranoid. */
+ intel_uncore_write16(uncore, GEN2_IIR, 0xffff);
+ intel_uncore_posting_read16(uncore, GEN2_IIR);
+ intel_uncore_write16(uncore, GEN2_IIR, 0xffff);
+ intel_uncore_posting_read16(uncore, GEN2_IIR);
+}
+
+#define GEN8_IRQ_RESET_NDX(uncore, type, which) \
+({ \
+ unsigned int which_ = which; \
+ gen3_irq_reset((uncore), GEN8_##type##_IMR(which_), \
+ GEN8_##type##_IIR(which_), GEN8_##type##_IER(which_)); \
+})
+
+#define GEN3_IRQ_RESET(uncore, type) \
+ gen3_irq_reset((uncore), type##IMR, type##IIR, type##IER)
+
+#define GEN2_IRQ_RESET(uncore) \
+ gen2_irq_reset(uncore)
/*
* We should clear IMR at preinstall/uninstall, and just check at postinstall.
*/
-static void gen3_assert_iir_is_zero(struct drm_i915_private *dev_priv,
- i915_reg_t reg)
+static void gen3_assert_iir_is_zero(struct intel_uncore *uncore, i915_reg_t reg)
{
- u32 val = I915_READ(reg);
+ u32 val = intel_uncore_read(uncore, reg);
if (val == 0)
return;
WARN(1, "Interrupt register 0x%x is not zero: 0x%08x\n",
i915_mmio_reg_offset(reg), val);
- I915_WRITE(reg, 0xffffffff);
- POSTING_READ(reg);
- I915_WRITE(reg, 0xffffffff);
- POSTING_READ(reg);
+ intel_uncore_write(uncore, reg, 0xffffffff);
+ intel_uncore_posting_read(uncore, reg);
+ intel_uncore_write(uncore, reg, 0xffffffff);
+ intel_uncore_posting_read(uncore, reg);
}
-static void gen2_assert_iir_is_zero(struct drm_i915_private *dev_priv,
- i915_reg_t reg)
+static void gen2_assert_iir_is_zero(struct intel_uncore *uncore)
{
- u16 val = I915_READ16(reg);
+ u16 val = intel_uncore_read16(uncore, GEN2_IIR);
if (val == 0)
return;
WARN(1, "Interrupt register 0x%x is not zero: 0x%08x\n",
- i915_mmio_reg_offset(reg), val);
- I915_WRITE16(reg, 0xffff);
- POSTING_READ16(reg);
- I915_WRITE16(reg, 0xffff);
- POSTING_READ16(reg);
-}
-
-#define GEN8_IRQ_INIT_NDX(type, which, imr_val, ier_val) do { \
- gen3_assert_iir_is_zero(dev_priv, GEN8_##type##_IIR(which)); \
- I915_WRITE(GEN8_##type##_IER(which), (ier_val)); \
- I915_WRITE(GEN8_##type##_IMR(which), (imr_val)); \
- POSTING_READ(GEN8_##type##_IMR(which)); \
-} while (0)
-
-#define GEN3_IRQ_INIT(type, imr_val, ier_val) do { \
- gen3_assert_iir_is_zero(dev_priv, type##IIR); \
- I915_WRITE(type##IER, (ier_val)); \
- I915_WRITE(type##IMR, (imr_val)); \
- POSTING_READ(type##IMR); \
-} while (0)
-
-#define GEN2_IRQ_INIT(type, imr_val, ier_val) do { \
- gen2_assert_iir_is_zero(dev_priv, type##IIR); \
- I915_WRITE16(type##IER, (ier_val)); \
- I915_WRITE16(type##IMR, (imr_val)); \
- POSTING_READ16(type##IMR); \
-} while (0)
+ i915_mmio_reg_offset(GEN2_IIR), val);
+ intel_uncore_write16(uncore, GEN2_IIR, 0xffff);
+ intel_uncore_posting_read16(uncore, GEN2_IIR);
+ intel_uncore_write16(uncore, GEN2_IIR, 0xffff);
+ intel_uncore_posting_read16(uncore, GEN2_IIR);
+}
+
+static void gen3_irq_init(struct intel_uncore *uncore,
+ i915_reg_t imr, u32 imr_val,
+ i915_reg_t ier, u32 ier_val,
+ i915_reg_t iir)
+{
+ gen3_assert_iir_is_zero(uncore, iir);
+
+ intel_uncore_write(uncore, ier, ier_val);
+ intel_uncore_write(uncore, imr, imr_val);
+ intel_uncore_posting_read(uncore, imr);
+}
+
+static void gen2_irq_init(struct intel_uncore *uncore,
+ u32 imr_val, u32 ier_val)
+{
+ gen2_assert_iir_is_zero(uncore);
+
+ intel_uncore_write16(uncore, GEN2_IER, ier_val);
+ intel_uncore_write16(uncore, GEN2_IMR, imr_val);
+ intel_uncore_posting_read16(uncore, GEN2_IMR);
+}
+
+#define GEN8_IRQ_INIT_NDX(uncore, type, which, imr_val, ier_val) \
+({ \
+ unsigned int which_ = which; \
+ gen3_irq_init((uncore), \
+ GEN8_##type##_IMR(which_), imr_val, \
+ GEN8_##type##_IER(which_), ier_val, \
+ GEN8_##type##_IIR(which_)); \
+})
+
+#define GEN3_IRQ_INIT(uncore, type, imr_val, ier_val) \
+ gen3_irq_init((uncore), \
+ type##IMR, imr_val, \
+ type##IER, ier_val, \
+ type##IIR)
+
+#define GEN2_IRQ_INIT(uncore, imr_val, ier_val) \
+ gen2_irq_init((uncore), imr_val, ier_val)
static void gen6_rps_irq_handler(struct drm_i915_private *dev_priv, u32 pm_iir);
static void gen9_guc_irq_handler(struct drm_i915_private *dev_priv, u32 pm_iir);
const unsigned int bank,
const unsigned int bit)
{
- void __iomem * const regs = i915->regs;
+ void __iomem * const regs = i915->uncore.regs;
u32 dw;
lockdep_assert_held(&i915->irq_lock);
return INTEL_GEN(dev_priv) >= 8 ? GEN8_GT_IIR(2) : GEN6_PMIIR;
}
-static i915_reg_t gen6_pm_imr(struct drm_i915_private *dev_priv)
+static void write_pm_imr(struct drm_i915_private *dev_priv)
{
- if (INTEL_GEN(dev_priv) >= 11)
- return GEN11_GPM_WGBOXPERF_INTR_MASK;
- else if (INTEL_GEN(dev_priv) >= 8)
- return GEN8_GT_IMR(2);
- else
- return GEN6_PMIMR;
+ i915_reg_t reg;
+ u32 mask = dev_priv->pm_imr;
+
+ if (INTEL_GEN(dev_priv) >= 11) {
+ reg = GEN11_GPM_WGBOXPERF_INTR_MASK;
+ /* pm is in upper half */
+ mask = mask << 16;
+ } else if (INTEL_GEN(dev_priv) >= 8) {
+ reg = GEN8_GT_IMR(2);
+ } else {
+ reg = GEN6_PMIMR;
+ }
+
+ I915_WRITE(reg, mask);
+ POSTING_READ(reg);
}
-static i915_reg_t gen6_pm_ier(struct drm_i915_private *dev_priv)
+static void write_pm_ier(struct drm_i915_private *dev_priv)
{
- if (INTEL_GEN(dev_priv) >= 11)
- return GEN11_GPM_WGBOXPERF_INTR_ENABLE;
- else if (INTEL_GEN(dev_priv) >= 8)
- return GEN8_GT_IER(2);
- else
- return GEN6_PMIER;
+ i915_reg_t reg;
+ u32 mask = dev_priv->pm_ier;
+
+ if (INTEL_GEN(dev_priv) >= 11) {
+ reg = GEN11_GPM_WGBOXPERF_INTR_ENABLE;
+ /* pm is in upper half */
+ mask = mask << 16;
+ } else if (INTEL_GEN(dev_priv) >= 8) {
+ reg = GEN8_GT_IER(2);
+ } else {
+ reg = GEN6_PMIER;
+ }
+
+ I915_WRITE(reg, mask);
}
/**
if (new_val != dev_priv->pm_imr) {
dev_priv->pm_imr = new_val;
- I915_WRITE(gen6_pm_imr(dev_priv), dev_priv->pm_imr);
- POSTING_READ(gen6_pm_imr(dev_priv));
+ write_pm_imr(dev_priv);
}
}
lockdep_assert_held(&dev_priv->irq_lock);
dev_priv->pm_ier |= enable_mask;
- I915_WRITE(gen6_pm_ier(dev_priv), dev_priv->pm_ier);
+ write_pm_ier(dev_priv);
gen6_unmask_pm_irq(dev_priv, enable_mask);
/* unmask_pm_irq provides an implicit barrier (POSTING_READ) */
}
dev_priv->pm_ier &= ~disable_mask;
__gen6_mask_pm_irq(dev_priv, disable_mask);
- I915_WRITE(gen6_pm_ier(dev_priv), dev_priv->pm_ier);
+ write_pm_ier(dev_priv);
/* though a barrier is missing here, but don't really need a one */
}
POSTING_READ(reg);
}
+static bool i915_has_asle(struct drm_i915_private *dev_priv)
+{
+ if (!dev_priv->opregion.asle)
+ return false;
+
+ return IS_PINEVIEW(dev_priv) || IS_MOBILE(dev_priv);
+}
+
/**
* i915_enable_asle_pipestat - enable ASLE pipestat for OpRegion
* @dev_priv: i915 device private
*/
static void i915_enable_asle_pipestat(struct drm_i915_private *dev_priv)
{
- if (!dev_priv->opregion.asle || !IS_MOBILE(dev_priv))
+ if (!i915_has_asle(dev_priv))
return;
spin_lock_irq(&dev_priv->irq_lock);
rps->last_adj = adj;
+ /*
+ * Limit deboosting and boosting to keep ourselves at the extremes
+ * when in the respective power modes (i.e. slowly decrease frequencies
+ * while in the HIGH_POWER zone and slowly increase frequencies while
+ * in the LOW_POWER zone). On idle, we will hit the timeout and drop
+ * to the next level quickly, and conversely if busy we expect to
+ * hit a waitboost and rapidly switch into max power.
+ */
+ if ((adj < 0 && rps->power.mode == HIGH_POWER) ||
+ (adj > 0 && rps->power.mode == LOW_POWER))
+ rps->last_adj = 0;
+
/* sysfs frequency interfaces may have snuck in while servicing the
* interrupt
*/
u32 gt_iir)
{
if (gt_iir & GT_RENDER_USER_INTERRUPT)
- intel_engine_breadcrumbs_irq(dev_priv->engine[RCS]);
+ intel_engine_breadcrumbs_irq(dev_priv->engine[RCS0]);
if (gt_iir & ILK_BSD_USER_INTERRUPT)
- intel_engine_breadcrumbs_irq(dev_priv->engine[VCS]);
+ intel_engine_breadcrumbs_irq(dev_priv->engine[VCS0]);
}
static void snb_gt_irq_handler(struct drm_i915_private *dev_priv,
u32 gt_iir)
{
if (gt_iir & GT_RENDER_USER_INTERRUPT)
- intel_engine_breadcrumbs_irq(dev_priv->engine[RCS]);
+ intel_engine_breadcrumbs_irq(dev_priv->engine[RCS0]);
if (gt_iir & GT_BSD_USER_INTERRUPT)
- intel_engine_breadcrumbs_irq(dev_priv->engine[VCS]);
+ intel_engine_breadcrumbs_irq(dev_priv->engine[VCS0]);
if (gt_iir & GT_BLT_USER_INTERRUPT)
- intel_engine_breadcrumbs_irq(dev_priv->engine[BCS]);
+ intel_engine_breadcrumbs_irq(dev_priv->engine[BCS0]);
if (gt_iir & (GT_BLT_CS_ERROR_INTERRUPT |
GT_BSD_CS_ERROR_INTERRUPT |
if (iir & GT_RENDER_USER_INTERRUPT) {
intel_engine_breadcrumbs_irq(engine);
- tasklet |= USES_GUC_SUBMISSION(engine->i915);
+ tasklet |= intel_engine_needs_breadcrumb_tasklet(engine);
}
if (tasklet)
static void gen8_gt_irq_ack(struct drm_i915_private *i915,
u32 master_ctl, u32 gt_iir[4])
{
- void __iomem * const regs = i915->regs;
+ void __iomem * const regs = i915->uncore.regs;
#define GEN8_GT_IRQS (GEN8_GT_RCS_IRQ | \
GEN8_GT_BCS_IRQ | \
+ GEN8_GT_VCS0_IRQ | \
GEN8_GT_VCS1_IRQ | \
- GEN8_GT_VCS2_IRQ | \
GEN8_GT_VECS_IRQ | \
GEN8_GT_PM_IRQ | \
GEN8_GT_GUC_IRQ)
raw_reg_write(regs, GEN8_GT_IIR(0), gt_iir[0]);
}
- if (master_ctl & (GEN8_GT_VCS1_IRQ | GEN8_GT_VCS2_IRQ)) {
+ if (master_ctl & (GEN8_GT_VCS0_IRQ | GEN8_GT_VCS1_IRQ)) {
gt_iir[1] = raw_reg_read(regs, GEN8_GT_IIR(1));
if (likely(gt_iir[1]))
raw_reg_write(regs, GEN8_GT_IIR(1), gt_iir[1]);
u32 master_ctl, u32 gt_iir[4])
{
if (master_ctl & (GEN8_GT_RCS_IRQ | GEN8_GT_BCS_IRQ)) {
- gen8_cs_irq_handler(i915->engine[RCS],
+ gen8_cs_irq_handler(i915->engine[RCS0],
gt_iir[0] >> GEN8_RCS_IRQ_SHIFT);
- gen8_cs_irq_handler(i915->engine[BCS],
+ gen8_cs_irq_handler(i915->engine[BCS0],
gt_iir[0] >> GEN8_BCS_IRQ_SHIFT);
}
- if (master_ctl & (GEN8_GT_VCS1_IRQ | GEN8_GT_VCS2_IRQ)) {
- gen8_cs_irq_handler(i915->engine[VCS],
+ if (master_ctl & (GEN8_GT_VCS0_IRQ | GEN8_GT_VCS1_IRQ)) {
+ gen8_cs_irq_handler(i915->engine[VCS0],
+ gt_iir[1] >> GEN8_VCS0_IRQ_SHIFT);
+ gen8_cs_irq_handler(i915->engine[VCS1],
gt_iir[1] >> GEN8_VCS1_IRQ_SHIFT);
- gen8_cs_irq_handler(i915->engine[VCS2],
- gt_iir[1] >> GEN8_VCS2_IRQ_SHIFT);
}
if (master_ctl & GEN8_GT_VECS_IRQ) {
- gen8_cs_irq_handler(i915->engine[VECS],
+ gen8_cs_irq_handler(i915->engine[VECS0],
gt_iir[3] >> GEN8_VECS_IRQ_SHIFT);
}
{
struct intel_pipe_crc *pipe_crc = &dev_priv->pipe_crc[pipe];
struct intel_crtc *crtc = intel_get_crtc_for_pipe(dev_priv, pipe);
- u32 crcs[5];
+ u32 crcs[5] = { crc0, crc1, crc2, crc3, crc4 };
+
+ trace_intel_pipe_crc(crtc, crcs);
spin_lock(&pipe_crc->lock);
/*
}
spin_unlock(&pipe_crc->lock);
- crcs[0] = crc0;
- crcs[1] = crc1;
- crcs[2] = crc2;
- crcs[3] = crc3;
- crcs[4] = crc4;
drm_crtc_add_crc_entry(&crtc->base, true,
drm_crtc_accurate_vblank_count(&crtc->base),
crcs);
/* The RPS events need forcewake, so we add them to a work queue and mask their
* IMR bits until the work is done. Other interrupts can be processed without
* the work queue. */
+static void gen11_rps_irq_handler(struct drm_i915_private *i915, u32 pm_iir)
+{
+ struct intel_rps *rps = &i915->gt_pm.rps;
+ const u32 events = i915->pm_rps_events & pm_iir;
+
+ lockdep_assert_held(&i915->irq_lock);
+
+ if (unlikely(!events))
+ return;
+
+ gen6_mask_pm_irq(i915, events);
+
+ if (!rps->interrupts_enabled)
+ return;
+
+ rps->pm_iir |= events;
+ schedule_work(&rps->work);
+}
+
static void gen6_rps_irq_handler(struct drm_i915_private *dev_priv, u32 pm_iir)
{
struct intel_rps *rps = &dev_priv->gt_pm.rps;
if (INTEL_GEN(dev_priv) >= 8)
return;
- if (HAS_VEBOX(dev_priv)) {
- if (pm_iir & PM_VEBOX_USER_INTERRUPT)
- intel_engine_breadcrumbs_irq(dev_priv->engine[VECS]);
+ if (pm_iir & PM_VEBOX_USER_INTERRUPT)
+ intel_engine_breadcrumbs_irq(dev_priv->engine[VECS0]);
- if (pm_iir & PM_VEBOX_CS_ERROR_INTERRUPT)
- DRM_DEBUG("Command parser error, pm_iir 0x%08x\n", pm_iir);
- }
+ if (pm_iir & PM_VEBOX_CS_ERROR_INTERRUPT)
+ DRM_DEBUG("Command parser error, pm_iir 0x%08x\n", pm_iir);
}
static void gen9_guc_irq_handler(struct drm_i915_private *dev_priv, u32 gt_iir)
DRM_ERROR("Unexpected DE HPD interrupt 0x%08x\n", iir);
}
+static u32 gen8_de_port_aux_mask(struct drm_i915_private *dev_priv)
+{
+ u32 mask = GEN8_AUX_CHANNEL_A;
+
+ if (INTEL_GEN(dev_priv) >= 9)
+ mask |= GEN9_AUX_CHANNEL_B |
+ GEN9_AUX_CHANNEL_C |
+ GEN9_AUX_CHANNEL_D;
+
+ if (IS_CNL_WITH_PORT_F(dev_priv))
+ mask |= CNL_AUX_CHANNEL_F;
+
+ if (INTEL_GEN(dev_priv) >= 11)
+ mask |= ICL_AUX_CHANNEL_E |
+ CNL_AUX_CHANNEL_F;
+
+ return mask;
+}
+
static irqreturn_t
gen8_de_irq_handler(struct drm_i915_private *dev_priv, u32 master_ctl)
{
I915_WRITE(GEN8_DE_PORT_IIR, iir);
ret = IRQ_HANDLED;
- tmp_mask = GEN8_AUX_CHANNEL_A;
- if (INTEL_GEN(dev_priv) >= 9)
- tmp_mask |= GEN9_AUX_CHANNEL_B |
- GEN9_AUX_CHANNEL_C |
- GEN9_AUX_CHANNEL_D;
-
- if (INTEL_GEN(dev_priv) >= 11)
- tmp_mask |= ICL_AUX_CHANNEL_E;
-
- if (IS_CNL_WITH_PORT_F(dev_priv) ||
- INTEL_GEN(dev_priv) >= 11)
- tmp_mask |= CNL_AUX_CHANNEL_F;
-
- if (iir & tmp_mask) {
+ if (iir & gen8_de_port_aux_mask(dev_priv)) {
dp_aux_irq_handler(dev_priv);
found = true;
}
I915_WRITE(SDEIIR, iir);
ret = IRQ_HANDLED;
- if (HAS_PCH_ICP(dev_priv))
+ if (INTEL_PCH_TYPE(dev_priv) >= PCH_ICP)
icp_irq_handler(dev_priv, iir);
- else if (HAS_PCH_SPT(dev_priv) ||
- HAS_PCH_KBP(dev_priv) ||
- HAS_PCH_CNP(dev_priv))
+ else if (INTEL_PCH_TYPE(dev_priv) >= PCH_SPT)
spt_irq_handler(dev_priv, iir);
else
cpt_irq_handler(dev_priv, iir);
static irqreturn_t gen8_irq_handler(int irq, void *arg)
{
struct drm_i915_private *dev_priv = to_i915(arg);
- void __iomem * const regs = dev_priv->regs;
+ void __iomem * const regs = dev_priv->uncore.regs;
u32 master_ctl;
u32 gt_iir[4];
gen11_gt_engine_identity(struct drm_i915_private * const i915,
const unsigned int bank, const unsigned int bit)
{
- void __iomem * const regs = i915->regs;
+ void __iomem * const regs = i915->uncore.regs;
u32 timeout_ts;
u32 ident;
const u8 instance, const u16 iir)
{
if (instance == OTHER_GTPM_INSTANCE)
- return gen6_rps_irq_handler(i915, iir);
+ return gen11_rps_irq_handler(i915, iir);
WARN_ONCE(1, "unhandled other interrupt instance=0x%x, iir=0x%x\n",
instance, iir);
gen11_gt_bank_handler(struct drm_i915_private * const i915,
const unsigned int bank)
{
- void __iomem * const regs = i915->regs;
+ void __iomem * const regs = i915->uncore.regs;
unsigned long intr_dw;
unsigned int bit;
intr_dw = raw_reg_read(regs, GEN11_GT_INTR_DW(bank));
- if (unlikely(!intr_dw)) {
- DRM_ERROR("GT_INTR_DW%u blank!\n", bank);
- return;
- }
-
for_each_set_bit(bit, &intr_dw, 32) {
- const u32 ident = gen11_gt_engine_identity(i915,
- bank, bit);
+ const u32 ident = gen11_gt_engine_identity(i915, bank, bit);
gen11_gt_identity_handler(i915, ident);
}
static u32
gen11_gu_misc_irq_ack(struct drm_i915_private *dev_priv, const u32 master_ctl)
{
- void __iomem * const regs = dev_priv->regs;
+ void __iomem * const regs = dev_priv->uncore.regs;
u32 iir;
if (!(master_ctl & GEN11_GU_MISC_IRQ))
static irqreturn_t gen11_irq_handler(int irq, void *arg)
{
struct drm_i915_private * const i915 = to_i915(arg);
- void __iomem * const regs = i915->regs;
+ void __iomem * const regs = i915->uncore.regs;
u32 master_ctl;
u32 gu_misc_iir;
return 0;
}
+static int i945gm_enable_vblank(struct drm_device *dev, unsigned int pipe)
+{
+ struct drm_i915_private *dev_priv = to_i915(dev);
+
+ if (dev_priv->i945gm_vblank.enabled++ == 0)
+ schedule_work(&dev_priv->i945gm_vblank.work);
+
+ return i8xx_enable_vblank(dev, pipe);
+}
+
static int i965_enable_vblank(struct drm_device *dev, unsigned int pipe)
{
struct drm_i915_private *dev_priv = to_i915(dev);
spin_unlock_irqrestore(&dev_priv->irq_lock, irqflags);
}
+static void i945gm_disable_vblank(struct drm_device *dev, unsigned int pipe)
+{
+ struct drm_i915_private *dev_priv = to_i915(dev);
+
+ i8xx_disable_vblank(dev, pipe);
+
+ if (--dev_priv->i945gm_vblank.enabled == 0)
+ schedule_work(&dev_priv->i945gm_vblank.work);
+}
+
static void i965_disable_vblank(struct drm_device *dev, unsigned int pipe)
{
struct drm_i915_private *dev_priv = to_i915(dev);
spin_unlock_irqrestore(&dev_priv->irq_lock, irqflags);
}
+static void i945gm_vblank_work_func(struct work_struct *work)
+{
+ struct drm_i915_private *dev_priv =
+ container_of(work, struct drm_i915_private, i945gm_vblank.work);
+
+ /*
+ * Vblank interrupts fail to wake up the device from C3,
+ * hence we want to prevent C3 usage while vblank interrupts
+ * are enabled.
+ */
+ pm_qos_update_request(&dev_priv->i945gm_vblank.pm_qos,
+ READ_ONCE(dev_priv->i945gm_vblank.enabled) ?
+ dev_priv->i945gm_vblank.c3_disable_latency :
+ PM_QOS_DEFAULT_VALUE);
+}
+
+static int cstate_disable_latency(const char *name)
+{
+ const struct cpuidle_driver *drv;
+ int i;
+
+ drv = cpuidle_get_driver();
+ if (!drv)
+ return 0;
+
+ for (i = 0; i < drv->state_count; i++) {
+ const struct cpuidle_state *state = &drv->states[i];
+
+ if (!strcmp(state->name, name))
+ return state->exit_latency ?
+ state->exit_latency - 1 : 0;
+ }
+
+ return 0;
+}
+
+static void i945gm_vblank_work_init(struct drm_i915_private *dev_priv)
+{
+ INIT_WORK(&dev_priv->i945gm_vblank.work,
+ i945gm_vblank_work_func);
+
+ dev_priv->i945gm_vblank.c3_disable_latency =
+ cstate_disable_latency("C3");
+ pm_qos_add_request(&dev_priv->i945gm_vblank.pm_qos,
+ PM_QOS_CPU_DMA_LATENCY,
+ PM_QOS_DEFAULT_VALUE);
+}
+
+static void i945gm_vblank_work_fini(struct drm_i915_private *dev_priv)
+{
+ cancel_work_sync(&dev_priv->i945gm_vblank.work);
+ pm_qos_remove_request(&dev_priv->i945gm_vblank.pm_qos);
+}
+
static void ibx_irq_reset(struct drm_i915_private *dev_priv)
{
+ struct intel_uncore *uncore = &dev_priv->uncore;
+
if (HAS_PCH_NOP(dev_priv))
return;
- GEN3_IRQ_RESET(SDE);
+ GEN3_IRQ_RESET(uncore, SDE);
if (HAS_PCH_CPT(dev_priv) || HAS_PCH_LPT(dev_priv))
I915_WRITE(SERR_INT, 0xffffffff);
static void gen5_gt_irq_reset(struct drm_i915_private *dev_priv)
{
- GEN3_IRQ_RESET(GT);
+ struct intel_uncore *uncore = &dev_priv->uncore;
+
+ GEN3_IRQ_RESET(uncore, GT);
if (INTEL_GEN(dev_priv) >= 6)
- GEN3_IRQ_RESET(GEN6_PM);
+ GEN3_IRQ_RESET(uncore, GEN6_PM);
}
static void vlv_display_irq_reset(struct drm_i915_private *dev_priv)
{
+ struct intel_uncore *uncore = &dev_priv->uncore;
+
if (IS_CHERRYVIEW(dev_priv))
I915_WRITE(DPINVGTT, DPINVGTT_STATUS_MASK_CHV);
else
i9xx_pipestat_irq_reset(dev_priv);
- GEN3_IRQ_RESET(VLV_);
+ GEN3_IRQ_RESET(uncore, VLV_);
dev_priv->irq_mask = ~0u;
}
static void vlv_display_irq_postinstall(struct drm_i915_private *dev_priv)
{
+ struct intel_uncore *uncore = &dev_priv->uncore;
+
u32 pipestat_mask;
u32 enable_mask;
enum pipe pipe;
dev_priv->irq_mask = ~enable_mask;
- GEN3_IRQ_INIT(VLV_, dev_priv->irq_mask, enable_mask);
+ GEN3_IRQ_INIT(uncore, VLV_, dev_priv->irq_mask, enable_mask);
}
/* drm_dma.h hooks
static void ironlake_irq_reset(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
- GEN3_IRQ_RESET(DE);
+ GEN3_IRQ_RESET(uncore, DE);
if (IS_GEN(dev_priv, 7))
I915_WRITE(GEN7_ERR_INT, 0xffffffff);
static void gen8_gt_irq_reset(struct drm_i915_private *dev_priv)
{
- GEN8_IRQ_RESET_NDX(GT, 0);
- GEN8_IRQ_RESET_NDX(GT, 1);
- GEN8_IRQ_RESET_NDX(GT, 2);
- GEN8_IRQ_RESET_NDX(GT, 3);
+ struct intel_uncore *uncore = &dev_priv->uncore;
+
+ GEN8_IRQ_RESET_NDX(uncore, GT, 0);
+ GEN8_IRQ_RESET_NDX(uncore, GT, 1);
+ GEN8_IRQ_RESET_NDX(uncore, GT, 2);
+ GEN8_IRQ_RESET_NDX(uncore, GT, 3);
}
static void gen8_irq_reset(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
int pipe;
- gen8_master_intr_disable(dev_priv->regs);
+ gen8_master_intr_disable(dev_priv->uncore.regs);
gen8_gt_irq_reset(dev_priv);
for_each_pipe(dev_priv, pipe)
if (intel_display_power_is_enabled(dev_priv,
POWER_DOMAIN_PIPE(pipe)))
- GEN8_IRQ_RESET_NDX(DE_PIPE, pipe);
+ GEN8_IRQ_RESET_NDX(uncore, DE_PIPE, pipe);
- GEN3_IRQ_RESET(GEN8_DE_PORT_);
- GEN3_IRQ_RESET(GEN8_DE_MISC_);
- GEN3_IRQ_RESET(GEN8_PCU_);
+ GEN3_IRQ_RESET(uncore, GEN8_DE_PORT_);
+ GEN3_IRQ_RESET(uncore, GEN8_DE_MISC_);
+ GEN3_IRQ_RESET(uncore, GEN8_PCU_);
if (HAS_PCH_SPLIT(dev_priv))
ibx_irq_reset(dev_priv);
static void gen11_irq_reset(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = dev->dev_private;
+ struct intel_uncore *uncore = &dev_priv->uncore;
int pipe;
- gen11_master_intr_disable(dev_priv->regs);
+ gen11_master_intr_disable(dev_priv->uncore.regs);
gen11_gt_irq_reset(dev_priv);
for_each_pipe(dev_priv, pipe)
if (intel_display_power_is_enabled(dev_priv,
POWER_DOMAIN_PIPE(pipe)))
- GEN8_IRQ_RESET_NDX(DE_PIPE, pipe);
+ GEN8_IRQ_RESET_NDX(uncore, DE_PIPE, pipe);
- GEN3_IRQ_RESET(GEN8_DE_PORT_);
- GEN3_IRQ_RESET(GEN8_DE_MISC_);
- GEN3_IRQ_RESET(GEN11_DE_HPD_);
- GEN3_IRQ_RESET(GEN11_GU_MISC_);
- GEN3_IRQ_RESET(GEN8_PCU_);
+ GEN3_IRQ_RESET(uncore, GEN8_DE_PORT_);
+ GEN3_IRQ_RESET(uncore, GEN8_DE_MISC_);
+ GEN3_IRQ_RESET(uncore, GEN11_DE_HPD_);
+ GEN3_IRQ_RESET(uncore, GEN11_GU_MISC_);
+ GEN3_IRQ_RESET(uncore, GEN8_PCU_);
- if (HAS_PCH_ICP(dev_priv))
- GEN3_IRQ_RESET(SDE);
+ if (INTEL_PCH_TYPE(dev_priv) >= PCH_ICP)
+ GEN3_IRQ_RESET(uncore, SDE);
}
void gen8_irq_power_well_post_enable(struct drm_i915_private *dev_priv,
u8 pipe_mask)
{
+ struct intel_uncore *uncore = &dev_priv->uncore;
+
u32 extra_ier = GEN8_PIPE_VBLANK | GEN8_PIPE_FIFO_UNDERRUN;
enum pipe pipe;
}
for_each_pipe_masked(dev_priv, pipe, pipe_mask)
- GEN8_IRQ_INIT_NDX(DE_PIPE, pipe,
+ GEN8_IRQ_INIT_NDX(uncore, DE_PIPE, pipe,
dev_priv->de_irq_mask[pipe],
~dev_priv->de_irq_mask[pipe] | extra_ier);
void gen8_irq_power_well_pre_disable(struct drm_i915_private *dev_priv,
u8 pipe_mask)
{
+ struct intel_uncore *uncore = &dev_priv->uncore;
enum pipe pipe;
spin_lock_irq(&dev_priv->irq_lock);
}
for_each_pipe_masked(dev_priv, pipe, pipe_mask)
- GEN8_IRQ_RESET_NDX(DE_PIPE, pipe);
+ GEN8_IRQ_RESET_NDX(uncore, DE_PIPE, pipe);
spin_unlock_irq(&dev_priv->irq_lock);
static void cherryview_irq_reset(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
I915_WRITE(GEN8_MASTER_IRQ, 0);
POSTING_READ(GEN8_MASTER_IRQ);
gen8_gt_irq_reset(dev_priv);
- GEN3_IRQ_RESET(GEN8_PCU_);
+ GEN3_IRQ_RESET(uncore, GEN8_PCU_);
spin_lock_irq(&dev_priv->irq_lock);
if (dev_priv->display_irqs_enabled)
gen11_hpd_detection_setup(dev_priv);
- if (HAS_PCH_ICP(dev_priv))
+ if (INTEL_PCH_TYPE(dev_priv) >= PCH_ICP)
icp_hpd_irq_setup(dev_priv);
}
else
mask = SDE_GMBUS_CPT;
- gen3_assert_iir_is_zero(dev_priv, SDEIIR);
+ gen3_assert_iir_is_zero(&dev_priv->uncore, SDEIIR);
I915_WRITE(SDEIMR, ~mask);
if (HAS_PCH_IBX(dev_priv) || HAS_PCH_CPT(dev_priv) ||
static void gen5_gt_irq_postinstall(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
u32 pm_irqs, gt_irqs;
pm_irqs = gt_irqs = 0;
gt_irqs |= GT_BLT_USER_INTERRUPT | GT_BSD_USER_INTERRUPT;
}
- GEN3_IRQ_INIT(GT, dev_priv->gt_irq_mask, gt_irqs);
+ GEN3_IRQ_INIT(uncore, GT, dev_priv->gt_irq_mask, gt_irqs);
if (INTEL_GEN(dev_priv) >= 6) {
/*
* RPS interrupts will get enabled/disabled on demand when RPS
* itself is enabled/disabled.
*/
- if (HAS_VEBOX(dev_priv)) {
+ if (HAS_ENGINE(dev_priv, VECS0)) {
pm_irqs |= PM_VEBOX_USER_INTERRUPT;
dev_priv->pm_ier |= PM_VEBOX_USER_INTERRUPT;
}
dev_priv->pm_imr = 0xffffffff;
- GEN3_IRQ_INIT(GEN6_PM, dev_priv->pm_imr, pm_irqs);
+ GEN3_IRQ_INIT(uncore, GEN6_PM, dev_priv->pm_imr, pm_irqs);
}
}
static int ironlake_irq_postinstall(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
u32 display_mask, extra_mask;
if (INTEL_GEN(dev_priv) >= 7) {
}
if (IS_HASWELL(dev_priv)) {
- gen3_assert_iir_is_zero(dev_priv, EDP_PSR_IIR);
+ gen3_assert_iir_is_zero(uncore, EDP_PSR_IIR);
intel_psr_irq_control(dev_priv, dev_priv->psr.debug);
display_mask |= DE_EDP_PSR_INT_HSW;
}
ibx_irq_pre_postinstall(dev);
- GEN3_IRQ_INIT(DE, dev_priv->irq_mask, display_mask | extra_mask);
+ GEN3_IRQ_INIT(uncore, DE, dev_priv->irq_mask,
+ display_mask | extra_mask);
gen5_gt_irq_postinstall(dev);
static void gen8_gt_irq_postinstall(struct drm_i915_private *dev_priv)
{
+ struct intel_uncore *uncore = &dev_priv->uncore;
+
/* These are interrupts we'll toggle with the ring mask register */
u32 gt_interrupts[] = {
- GT_RENDER_USER_INTERRUPT << GEN8_RCS_IRQ_SHIFT |
- GT_CONTEXT_SWITCH_INTERRUPT << GEN8_RCS_IRQ_SHIFT |
- GT_RENDER_USER_INTERRUPT << GEN8_BCS_IRQ_SHIFT |
- GT_CONTEXT_SWITCH_INTERRUPT << GEN8_BCS_IRQ_SHIFT,
- GT_RENDER_USER_INTERRUPT << GEN8_VCS1_IRQ_SHIFT |
- GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VCS1_IRQ_SHIFT |
- GT_RENDER_USER_INTERRUPT << GEN8_VCS2_IRQ_SHIFT |
- GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VCS2_IRQ_SHIFT,
+ (GT_RENDER_USER_INTERRUPT << GEN8_RCS_IRQ_SHIFT |
+ GT_CONTEXT_SWITCH_INTERRUPT << GEN8_RCS_IRQ_SHIFT |
+ GT_RENDER_USER_INTERRUPT << GEN8_BCS_IRQ_SHIFT |
+ GT_CONTEXT_SWITCH_INTERRUPT << GEN8_BCS_IRQ_SHIFT),
+
+ (GT_RENDER_USER_INTERRUPT << GEN8_VCS0_IRQ_SHIFT |
+ GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VCS0_IRQ_SHIFT |
+ GT_RENDER_USER_INTERRUPT << GEN8_VCS1_IRQ_SHIFT |
+ GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VCS1_IRQ_SHIFT),
+
0,
- GT_RENDER_USER_INTERRUPT << GEN8_VECS_IRQ_SHIFT |
- GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VECS_IRQ_SHIFT
- };
+
+ (GT_RENDER_USER_INTERRUPT << GEN8_VECS_IRQ_SHIFT |
+ GT_CONTEXT_SWITCH_INTERRUPT << GEN8_VECS_IRQ_SHIFT)
+ };
dev_priv->pm_ier = 0x0;
dev_priv->pm_imr = ~dev_priv->pm_ier;
- GEN8_IRQ_INIT_NDX(GT, 0, ~gt_interrupts[0], gt_interrupts[0]);
- GEN8_IRQ_INIT_NDX(GT, 1, ~gt_interrupts[1], gt_interrupts[1]);
+ GEN8_IRQ_INIT_NDX(uncore, GT, 0, ~gt_interrupts[0], gt_interrupts[0]);
+ GEN8_IRQ_INIT_NDX(uncore, GT, 1, ~gt_interrupts[1], gt_interrupts[1]);
/*
* RPS interrupts will get enabled/disabled on demand when RPS itself
* is enabled/disabled. Same wil be the case for GuC interrupts.
*/
- GEN8_IRQ_INIT_NDX(GT, 2, dev_priv->pm_imr, dev_priv->pm_ier);
- GEN8_IRQ_INIT_NDX(GT, 3, ~gt_interrupts[3], gt_interrupts[3]);
+ GEN8_IRQ_INIT_NDX(uncore, GT, 2, dev_priv->pm_imr, dev_priv->pm_ier);
+ GEN8_IRQ_INIT_NDX(uncore, GT, 3, ~gt_interrupts[3], gt_interrupts[3]);
}
static void gen8_de_irq_postinstall(struct drm_i915_private *dev_priv)
{
+ struct intel_uncore *uncore = &dev_priv->uncore;
+
u32 de_pipe_masked = GEN8_PIPE_CDCLK_CRC_DONE;
u32 de_pipe_enables;
u32 de_port_masked = GEN8_AUX_CHANNEL_A;
else if (IS_BROADWELL(dev_priv))
de_port_enables |= GEN8_PORT_DP_A_HOTPLUG;
- gen3_assert_iir_is_zero(dev_priv, EDP_PSR_IIR);
+ gen3_assert_iir_is_zero(uncore, EDP_PSR_IIR);
intel_psr_irq_control(dev_priv, dev_priv->psr.debug);
for_each_pipe(dev_priv, pipe) {
if (intel_display_power_is_enabled(dev_priv,
POWER_DOMAIN_PIPE(pipe)))
- GEN8_IRQ_INIT_NDX(DE_PIPE, pipe,
+ GEN8_IRQ_INIT_NDX(uncore, DE_PIPE, pipe,
dev_priv->de_irq_mask[pipe],
de_pipe_enables);
}
- GEN3_IRQ_INIT(GEN8_DE_PORT_, ~de_port_masked, de_port_enables);
- GEN3_IRQ_INIT(GEN8_DE_MISC_, ~de_misc_masked, de_misc_masked);
+ GEN3_IRQ_INIT(uncore, GEN8_DE_PORT_, ~de_port_masked, de_port_enables);
+ GEN3_IRQ_INIT(uncore, GEN8_DE_MISC_, ~de_misc_masked, de_misc_masked);
if (INTEL_GEN(dev_priv) >= 11) {
u32 de_hpd_masked = 0;
u32 de_hpd_enables = GEN11_DE_TC_HOTPLUG_MASK |
GEN11_DE_TBT_HOTPLUG_MASK;
- GEN3_IRQ_INIT(GEN11_DE_HPD_, ~de_hpd_masked, de_hpd_enables);
+ GEN3_IRQ_INIT(uncore, GEN11_DE_HPD_, ~de_hpd_masked,
+ de_hpd_enables);
gen11_hpd_detection_setup(dev_priv);
} else if (IS_GEN9_LP(dev_priv)) {
bxt_hpd_detection_setup(dev_priv);
if (HAS_PCH_SPLIT(dev_priv))
ibx_irq_postinstall(dev);
- gen8_master_intr_enable(dev_priv->regs);
+ gen8_master_intr_enable(dev_priv->uncore.regs);
return 0;
}
I915_WRITE(SDEIER, 0xffffffff);
POSTING_READ(SDEIER);
- gen3_assert_iir_is_zero(dev_priv, SDEIIR);
+ gen3_assert_iir_is_zero(&dev_priv->uncore, SDEIIR);
I915_WRITE(SDEIMR, ~mask);
icp_hpd_detection_setup(dev_priv);
static int gen11_irq_postinstall(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = dev->dev_private;
+ struct intel_uncore *uncore = &dev_priv->uncore;
u32 gu_misc_masked = GEN11_GU_MISC_GSE;
- if (HAS_PCH_ICP(dev_priv))
+ if (INTEL_PCH_TYPE(dev_priv) >= PCH_ICP)
icp_irq_postinstall(dev);
gen11_gt_irq_postinstall(dev_priv);
gen8_de_irq_postinstall(dev_priv);
- GEN3_IRQ_INIT(GEN11_GU_MISC_, ~gu_misc_masked, gu_misc_masked);
+ GEN3_IRQ_INIT(uncore, GEN11_GU_MISC_, ~gu_misc_masked, gu_misc_masked);
I915_WRITE(GEN11_DISPLAY_INT_CTL, GEN11_DISPLAY_IRQ_ENABLE);
- gen11_master_intr_enable(dev_priv->regs);
+ gen11_master_intr_enable(dev_priv->uncore.regs);
POSTING_READ(GEN11_GFX_MSTR_IRQ);
return 0;
static void i8xx_irq_reset(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
i9xx_pipestat_irq_reset(dev_priv);
- GEN2_IRQ_RESET();
+ GEN2_IRQ_RESET(uncore);
}
static int i8xx_irq_postinstall(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
u16 enable_mask;
I915_WRITE16(EMR, ~(I915_ERROR_PAGE_TABLE |
I915_MASTER_ERROR_INTERRUPT |
I915_USER_INTERRUPT;
- GEN2_IRQ_INIT(, dev_priv->irq_mask, enable_mask);
+ GEN2_IRQ_INIT(uncore, dev_priv->irq_mask, enable_mask);
/* Interrupt setup is already guaranteed to be single-threaded, this is
* just to make the assert_spin_locked check happy. */
u16 eir = 0, eir_stuck = 0;
u16 iir;
- iir = I915_READ16(IIR);
+ iir = I915_READ16(GEN2_IIR);
if (iir == 0)
break;
if (iir & I915_MASTER_ERROR_INTERRUPT)
i8xx_error_irq_ack(dev_priv, &eir, &eir_stuck);
- I915_WRITE16(IIR, iir);
+ I915_WRITE16(GEN2_IIR, iir);
if (iir & I915_USER_INTERRUPT)
- intel_engine_breadcrumbs_irq(dev_priv->engine[RCS]);
+ intel_engine_breadcrumbs_irq(dev_priv->engine[RCS0]);
if (iir & I915_MASTER_ERROR_INTERRUPT)
i8xx_error_irq_handler(dev_priv, eir, eir_stuck);
static void i915_irq_reset(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
if (I915_HAS_HOTPLUG(dev_priv)) {
i915_hotplug_interrupt_update(dev_priv, 0xffffffff, 0);
i9xx_pipestat_irq_reset(dev_priv);
- GEN3_IRQ_RESET();
+ GEN3_IRQ_RESET(uncore, GEN2_);
}
static int i915_irq_postinstall(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
u32 enable_mask;
I915_WRITE(EMR, ~(I915_ERROR_PAGE_TABLE |
dev_priv->irq_mask &= ~I915_DISPLAY_PORT_INTERRUPT;
}
- GEN3_IRQ_INIT(, dev_priv->irq_mask, enable_mask);
+ GEN3_IRQ_INIT(uncore, GEN2_, dev_priv->irq_mask, enable_mask);
/* Interrupt setup is already guaranteed to be single-threaded, this is
* just to make the assert_spin_locked check happy. */
u32 hotplug_status = 0;
u32 iir;
- iir = I915_READ(IIR);
+ iir = I915_READ(GEN2_IIR);
if (iir == 0)
break;
if (iir & I915_MASTER_ERROR_INTERRUPT)
i9xx_error_irq_ack(dev_priv, &eir, &eir_stuck);
- I915_WRITE(IIR, iir);
+ I915_WRITE(GEN2_IIR, iir);
if (iir & I915_USER_INTERRUPT)
- intel_engine_breadcrumbs_irq(dev_priv->engine[RCS]);
+ intel_engine_breadcrumbs_irq(dev_priv->engine[RCS0]);
if (iir & I915_MASTER_ERROR_INTERRUPT)
i9xx_error_irq_handler(dev_priv, eir, eir_stuck);
static void i965_irq_reset(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
i915_hotplug_interrupt_update(dev_priv, 0xffffffff, 0);
I915_WRITE(PORT_HOTPLUG_STAT, I915_READ(PORT_HOTPLUG_STAT));
i9xx_pipestat_irq_reset(dev_priv);
- GEN3_IRQ_RESET();
+ GEN3_IRQ_RESET(uncore, GEN2_);
}
static int i965_irq_postinstall(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_uncore *uncore = &dev_priv->uncore;
u32 enable_mask;
u32 error_mask;
if (IS_G4X(dev_priv))
enable_mask |= I915_BSD_USER_INTERRUPT;
- GEN3_IRQ_INIT(, dev_priv->irq_mask, enable_mask);
+ GEN3_IRQ_INIT(uncore, GEN2_, dev_priv->irq_mask, enable_mask);
/* Interrupt setup is already guaranteed to be single-threaded, this is
* just to make the assert_spin_locked check happy. */
u32 hotplug_status = 0;
u32 iir;
- iir = I915_READ(IIR);
+ iir = I915_READ(GEN2_IIR);
if (iir == 0)
break;
if (iir & I915_MASTER_ERROR_INTERRUPT)
i9xx_error_irq_ack(dev_priv, &eir, &eir_stuck);
- I915_WRITE(IIR, iir);
+ I915_WRITE(GEN2_IIR, iir);
if (iir & I915_USER_INTERRUPT)
- intel_engine_breadcrumbs_irq(dev_priv->engine[RCS]);
+ intel_engine_breadcrumbs_irq(dev_priv->engine[RCS0]);
if (iir & I915_BSD_USER_INTERRUPT)
- intel_engine_breadcrumbs_irq(dev_priv->engine[VCS]);
+ intel_engine_breadcrumbs_irq(dev_priv->engine[VCS0]);
if (iir & I915_MASTER_ERROR_INTERRUPT)
i9xx_error_irq_handler(dev_priv, eir, eir_stuck);
struct intel_rps *rps = &dev_priv->gt_pm.rps;
int i;
+ if (IS_I945GM(dev_priv))
+ i945gm_vblank_work_init(dev_priv);
+
intel_hpd_init_work(dev_priv);
INIT_WORK(&rps->work, gen6_pm_rps_work);
GEN6_PM_RP_DOWN_THRESHOLD |
GEN6_PM_RP_DOWN_TIMEOUT);
+ /* We share the register with other engine */
+ if (INTEL_GEN(dev_priv) > 9)
+ GEM_WARN_ON(dev_priv->pm_rps_events & 0xffff0000);
+
rps->pm_intrmsk_mbz = 0;
/*
else if (INTEL_GEN(dev_priv) >= 3)
dev->driver->get_vblank_counter = i915_get_vblank_counter;
- /*
- * Opt out of the vblank disable timer on everything except gen2.
- * Gen2 doesn't have a hardware frame counter and so depends on
- * vblank interrupts to produce sane vblank seuquence numbers.
- */
- if (!IS_GEN(dev_priv, 2))
- dev->vblank_disable_immediate = true;
+ dev->vblank_disable_immediate = true;
/* Most platforms treat the display irq block as an always-on
* power domain. vlv/chv can disable it at runtime and need
dev->driver->disable_vblank = gen8_disable_vblank;
if (IS_GEN9_LP(dev_priv))
dev_priv->display.hpd_irq_setup = bxt_hpd_irq_setup;
- else if (HAS_PCH_SPT(dev_priv) || HAS_PCH_KBP(dev_priv) ||
- HAS_PCH_CNP(dev_priv))
+ else if (INTEL_PCH_TYPE(dev_priv) >= PCH_SPT)
dev_priv->display.hpd_irq_setup = spt_hpd_irq_setup;
else
dev_priv->display.hpd_irq_setup = ilk_hpd_irq_setup;
dev->driver->irq_uninstall = i8xx_irq_reset;
dev->driver->enable_vblank = i8xx_enable_vblank;
dev->driver->disable_vblank = i8xx_disable_vblank;
+ } else if (IS_I945GM(dev_priv)) {
+ dev->driver->irq_preinstall = i915_irq_reset;
+ dev->driver->irq_postinstall = i915_irq_postinstall;
+ dev->driver->irq_uninstall = i915_irq_reset;
+ dev->driver->irq_handler = i915_irq_handler;
+ dev->driver->enable_vblank = i945gm_enable_vblank;
+ dev->driver->disable_vblank = i945gm_disable_vblank;
} else if (IS_GEN(dev_priv, 3)) {
dev->driver->irq_preinstall = i915_irq_reset;
dev->driver->irq_postinstall = i915_irq_postinstall;
{
int i;
+ if (IS_I945GM(i915))
+ i945gm_vblank_work_fini(i915);
+
for (i = 0; i < MAX_L3_SLICES; ++i)
kfree(i915->l3_parity.remap_info[i]);
}
#include <drm/drm_drv.h>
-#include "i915_active.h"
#include "i915_drv.h"
+#include "i915_globals.h"
#include "i915_selftest.h"
+#include "intel_fbdev.h"
-#define PLATFORM(x) .platform = (x), .platform_mask = BIT(x)
+#define PLATFORM(x) .platform = (x)
#define GEN(x) .gen = (x), .gen_mask = BIT((x) - 1)
-#define GEN_DEFAULT_PIPEOFFSETS \
+#define I845_PIPE_OFFSETS \
+ .pipe_offsets = { \
+ [TRANSCODER_A] = PIPE_A_OFFSET, \
+ }, \
+ .trans_offsets = { \
+ [TRANSCODER_A] = TRANSCODER_A_OFFSET, \
+ }
+
+#define I9XX_PIPE_OFFSETS \
+ .pipe_offsets = { \
+ [TRANSCODER_A] = PIPE_A_OFFSET, \
+ [TRANSCODER_B] = PIPE_B_OFFSET, \
+ }, \
+ .trans_offsets = { \
+ [TRANSCODER_A] = TRANSCODER_A_OFFSET, \
+ [TRANSCODER_B] = TRANSCODER_B_OFFSET, \
+ }
+
+#define IVB_PIPE_OFFSETS \
+ .pipe_offsets = { \
+ [TRANSCODER_A] = PIPE_A_OFFSET, \
+ [TRANSCODER_B] = PIPE_B_OFFSET, \
+ [TRANSCODER_C] = PIPE_C_OFFSET, \
+ }, \
+ .trans_offsets = { \
+ [TRANSCODER_A] = TRANSCODER_A_OFFSET, \
+ [TRANSCODER_B] = TRANSCODER_B_OFFSET, \
+ [TRANSCODER_C] = TRANSCODER_C_OFFSET, \
+ }
+
+#define HSW_PIPE_OFFSETS \
.pipe_offsets = { \
[TRANSCODER_A] = PIPE_A_OFFSET, \
[TRANSCODER_B] = PIPE_B_OFFSET, \
[TRANSCODER_EDP] = TRANSCODER_EDP_OFFSET, \
}
-#define GEN_CHV_PIPEOFFSETS \
+#define CHV_PIPE_OFFSETS \
.pipe_offsets = { \
[TRANSCODER_A] = PIPE_A_OFFSET, \
[TRANSCODER_B] = PIPE_B_OFFSET, \
[TRANSCODER_C] = CHV_TRANSCODER_C_OFFSET, \
}
-#define CURSOR_OFFSETS \
- .cursor_offsets = { CURSOR_A_OFFSET, CURSOR_B_OFFSET, CHV_CURSOR_C_OFFSET }
+#define I845_CURSOR_OFFSETS \
+ .cursor_offsets = { \
+ [PIPE_A] = CURSOR_A_OFFSET, \
+ }
+
+#define I9XX_CURSOR_OFFSETS \
+ .cursor_offsets = { \
+ [PIPE_A] = CURSOR_A_OFFSET, \
+ [PIPE_B] = CURSOR_B_OFFSET, \
+ }
+
+#define CHV_CURSOR_OFFSETS \
+ .cursor_offsets = { \
+ [PIPE_A] = CURSOR_A_OFFSET, \
+ [PIPE_B] = CURSOR_B_OFFSET, \
+ [PIPE_C] = CHV_CURSOR_C_OFFSET, \
+ }
#define IVB_CURSOR_OFFSETS \
- .cursor_offsets = { CURSOR_A_OFFSET, IVB_CURSOR_B_OFFSET, IVB_CURSOR_C_OFFSET }
+ .cursor_offsets = { \
+ [PIPE_A] = CURSOR_A_OFFSET, \
+ [PIPE_B] = IVB_CURSOR_B_OFFSET, \
+ [PIPE_C] = IVB_CURSOR_C_OFFSET, \
+ }
-#define BDW_COLORS \
- .color = { .degamma_lut_size = 512, .gamma_lut_size = 512 }
+#define I9XX_COLORS \
+ .color = { .gamma_lut_size = 256 }
+#define I965_COLORS \
+ .color = { .gamma_lut_size = 129, \
+ .gamma_lut_tests = DRM_COLOR_LUT_NON_DECREASING, \
+ }
+#define ILK_COLORS \
+ .color = { .gamma_lut_size = 1024 }
+#define IVB_COLORS \
+ .color = { .degamma_lut_size = 1024, .gamma_lut_size = 1024 }
#define CHV_COLORS \
.color = { .degamma_lut_size = 65, .gamma_lut_size = 257, \
.degamma_lut_tests = DRM_COLOR_LUT_NON_DECREASING, \
.gamma_lut_tests = DRM_COLOR_LUT_NON_DECREASING, \
}
#define GLK_COLORS \
- .color = { .degamma_lut_size = 0, .gamma_lut_size = 1024, \
+ .color = { .degamma_lut_size = 33, .gamma_lut_size = 1024, \
.degamma_lut_tests = DRM_COLOR_LUT_NON_DECREASING | \
DRM_COLOR_LUT_EQUAL_CHANNELS, \
}
#define GEN_DEFAULT_PAGE_SIZES \
.page_sizes = I915_GTT_PAGE_SIZE_4K
-#define GEN2_FEATURES \
+#define I830_FEATURES \
+ GEN(2), \
+ .is_mobile = 1, \
+ .num_pipes = 2, \
+ .display.has_overlay = 1, \
+ .display.cursor_needs_physical = 1, \
+ .display.overlay_needs_physical = 1, \
+ .display.has_gmch = 1, \
+ .gpu_reset_clobbers_display = true, \
+ .hws_needs_physical = 1, \
+ .unfenced_needs_alignment = 1, \
+ .engine_mask = BIT(RCS0), \
+ .has_snoop = true, \
+ .has_coherent_ggtt = false, \
+ I9XX_PIPE_OFFSETS, \
+ I9XX_CURSOR_OFFSETS, \
+ I9XX_COLORS, \
+ GEN_DEFAULT_PAGE_SIZES
+
+#define I845_FEATURES \
GEN(2), \
.num_pipes = 1, \
.display.has_overlay = 1, \
.gpu_reset_clobbers_display = true, \
.hws_needs_physical = 1, \
.unfenced_needs_alignment = 1, \
- .ring_mask = RENDER_RING, \
+ .engine_mask = BIT(RCS0), \
.has_snoop = true, \
.has_coherent_ggtt = false, \
- GEN_DEFAULT_PIPEOFFSETS, \
- GEN_DEFAULT_PAGE_SIZES, \
- CURSOR_OFFSETS
+ I845_PIPE_OFFSETS, \
+ I845_CURSOR_OFFSETS, \
+ I9XX_COLORS, \
+ GEN_DEFAULT_PAGE_SIZES
static const struct intel_device_info intel_i830_info = {
- GEN2_FEATURES,
+ I830_FEATURES,
PLATFORM(INTEL_I830),
- .is_mobile = 1,
- .display.cursor_needs_physical = 1,
- .num_pipes = 2, /* legal, last one wins */
};
static const struct intel_device_info intel_i845g_info = {
- GEN2_FEATURES,
+ I845_FEATURES,
PLATFORM(INTEL_I845G),
};
static const struct intel_device_info intel_i85x_info = {
- GEN2_FEATURES,
+ I830_FEATURES,
PLATFORM(INTEL_I85X),
- .is_mobile = 1,
- .num_pipes = 2, /* legal, last one wins */
- .display.cursor_needs_physical = 1,
.display.has_fbc = 1,
};
static const struct intel_device_info intel_i865g_info = {
- GEN2_FEATURES,
+ I845_FEATURES,
PLATFORM(INTEL_I865G),
};
.num_pipes = 2, \
.display.has_gmch = 1, \
.gpu_reset_clobbers_display = true, \
- .ring_mask = RENDER_RING, \
+ .engine_mask = BIT(RCS0), \
.has_snoop = true, \
.has_coherent_ggtt = true, \
- GEN_DEFAULT_PIPEOFFSETS, \
- GEN_DEFAULT_PAGE_SIZES, \
- CURSOR_OFFSETS
+ I9XX_PIPE_OFFSETS, \
+ I9XX_CURSOR_OFFSETS, \
+ I9XX_COLORS, \
+ GEN_DEFAULT_PAGE_SIZES
static const struct intel_device_info intel_i915g_info = {
GEN3_FEATURES,
.display.has_overlay = 1,
};
-static const struct intel_device_info intel_pineview_info = {
+static const struct intel_device_info intel_pineview_g_info = {
+ GEN3_FEATURES,
+ PLATFORM(INTEL_PINEVIEW),
+ .display.has_hotplug = 1,
+ .display.has_overlay = 1,
+};
+
+static const struct intel_device_info intel_pineview_m_info = {
GEN3_FEATURES,
PLATFORM(INTEL_PINEVIEW),
.is_mobile = 1,
.display.has_hotplug = 1, \
.display.has_gmch = 1, \
.gpu_reset_clobbers_display = true, \
- .ring_mask = RENDER_RING, \
+ .engine_mask = BIT(RCS0), \
.has_snoop = true, \
.has_coherent_ggtt = true, \
- GEN_DEFAULT_PIPEOFFSETS, \
- GEN_DEFAULT_PAGE_SIZES, \
- CURSOR_OFFSETS
+ I9XX_PIPE_OFFSETS, \
+ I9XX_CURSOR_OFFSETS, \
+ I965_COLORS, \
+ GEN_DEFAULT_PAGE_SIZES
static const struct intel_device_info intel_i965g_info = {
GEN4_FEATURES,
static const struct intel_device_info intel_g45_info = {
GEN4_FEATURES,
PLATFORM(INTEL_G45),
- .ring_mask = RENDER_RING | BSD_RING,
+ .engine_mask = BIT(RCS0) | BIT(VCS0),
.gpu_reset_clobbers_display = false,
};
.is_mobile = 1,
.display.has_fbc = 1,
.display.supports_tv = 1,
- .ring_mask = RENDER_RING | BSD_RING,
+ .engine_mask = BIT(RCS0) | BIT(VCS0),
.gpu_reset_clobbers_display = false,
};
GEN(5), \
.num_pipes = 2, \
.display.has_hotplug = 1, \
- .ring_mask = RENDER_RING | BSD_RING, \
+ .engine_mask = BIT(RCS0) | BIT(VCS0), \
.has_snoop = true, \
.has_coherent_ggtt = true, \
/* ilk does support rc6, but we do not implement [power] contexts */ \
.has_rc6 = 0, \
- GEN_DEFAULT_PIPEOFFSETS, \
- GEN_DEFAULT_PAGE_SIZES, \
- CURSOR_OFFSETS
+ I9XX_PIPE_OFFSETS, \
+ I9XX_CURSOR_OFFSETS, \
+ ILK_COLORS, \
+ GEN_DEFAULT_PAGE_SIZES
static const struct intel_device_info intel_ironlake_d_info = {
GEN5_FEATURES,
.num_pipes = 2, \
.display.has_hotplug = 1, \
.display.has_fbc = 1, \
- .ring_mask = RENDER_RING | BSD_RING | BLT_RING, \
+ .engine_mask = BIT(RCS0) | BIT(VCS0) | BIT(BCS0), \
.has_coherent_ggtt = true, \
.has_llc = 1, \
.has_rc6 = 1, \
.has_rc6p = 1, \
- .ppgtt = INTEL_PPGTT_ALIASING, \
- GEN_DEFAULT_PIPEOFFSETS, \
- GEN_DEFAULT_PAGE_SIZES, \
- CURSOR_OFFSETS
+ .ppgtt_type = INTEL_PPGTT_ALIASING, \
+ .ppgtt_size = 31, \
+ I9XX_PIPE_OFFSETS, \
+ I9XX_CURSOR_OFFSETS, \
+ ILK_COLORS, \
+ GEN_DEFAULT_PAGE_SIZES
#define SNB_D_PLATFORM \
GEN6_FEATURES, \
.num_pipes = 3, \
.display.has_hotplug = 1, \
.display.has_fbc = 1, \
- .ring_mask = RENDER_RING | BSD_RING | BLT_RING, \
+ .engine_mask = BIT(RCS0) | BIT(VCS0) | BIT(BCS0), \
.has_coherent_ggtt = true, \
.has_llc = 1, \
.has_rc6 = 1, \
.has_rc6p = 1, \
- .ppgtt = INTEL_PPGTT_FULL, \
- GEN_DEFAULT_PIPEOFFSETS, \
- GEN_DEFAULT_PAGE_SIZES, \
- IVB_CURSOR_OFFSETS
+ .ppgtt_type = INTEL_PPGTT_FULL, \
+ .ppgtt_size = 31, \
+ IVB_PIPE_OFFSETS, \
+ IVB_CURSOR_OFFSETS, \
+ IVB_COLORS, \
+ GEN_DEFAULT_PAGE_SIZES
#define IVB_D_PLATFORM \
GEN7_FEATURES, \
.has_rc6 = 1,
.display.has_gmch = 1,
.display.has_hotplug = 1,
- .ppgtt = INTEL_PPGTT_FULL,
+ .ppgtt_type = INTEL_PPGTT_FULL,
+ .ppgtt_size = 31,
.has_snoop = true,
.has_coherent_ggtt = false,
- .ring_mask = RENDER_RING | BSD_RING | BLT_RING,
+ .engine_mask = BIT(RCS0) | BIT(VCS0) | BIT(BCS0),
.display_mmio_offset = VLV_DISPLAY_BASE,
+ I9XX_PIPE_OFFSETS,
+ I9XX_CURSOR_OFFSETS,
+ I965_COLORS,
GEN_DEFAULT_PAGE_SIZES,
- GEN_DEFAULT_PIPEOFFSETS,
- CURSOR_OFFSETS
};
#define G75_FEATURES \
GEN7_FEATURES, \
- .ring_mask = RENDER_RING | BSD_RING | BLT_RING | VEBOX_RING, \
+ .engine_mask = BIT(RCS0) | BIT(VCS0) | BIT(BCS0) | BIT(VECS0), \
.display.has_ddi = 1, \
.has_fpga_dbg = 1, \
.display.has_psr = 1, \
.display.has_dp_mst = 1, \
.has_rc6p = 0 /* RC6p removed-by HSW */, \
+ HSW_PIPE_OFFSETS, \
.has_runtime_pm = 1
#define HSW_PLATFORM \
#define GEN8_FEATURES \
G75_FEATURES, \
GEN(8), \
- BDW_COLORS, \
.page_sizes = I915_GTT_PAGE_SIZE_4K | \
I915_GTT_PAGE_SIZE_2M, \
.has_logical_ring_contexts = 1, \
- .ppgtt = INTEL_PPGTT_FULL_4LVL, \
+ .ppgtt_type = INTEL_PPGTT_FULL, \
+ .ppgtt_size = 48, \
.has_64bit_reloc = 1, \
.has_reset_engine = 1
static const struct intel_device_info intel_broadwell_gt3_info = {
BDW_PLATFORM,
.gt = 3,
- .ring_mask = RENDER_RING | BSD_RING | BLT_RING | VEBOX_RING | BSD2_RING,
+ .engine_mask =
+ BIT(RCS0) | BIT(VCS0) | BIT(BCS0) | BIT(VECS0) | BIT(VCS1),
};
static const struct intel_device_info intel_cherryview_info = {
.num_pipes = 3,
.display.has_hotplug = 1,
.is_lp = 1,
- .ring_mask = RENDER_RING | BSD_RING | BLT_RING | VEBOX_RING,
+ .engine_mask = BIT(RCS0) | BIT(VCS0) | BIT(BCS0) | BIT(VECS0),
.has_64bit_reloc = 1,
.has_runtime_pm = 1,
.has_rc6 = 1,
.has_logical_ring_contexts = 1,
.display.has_gmch = 1,
- .ppgtt = INTEL_PPGTT_FULL,
+ .ppgtt_type = INTEL_PPGTT_FULL,
+ .ppgtt_size = 32,
.has_reset_engine = 1,
.has_snoop = true,
.has_coherent_ggtt = false,
.display_mmio_offset = VLV_DISPLAY_BASE,
- GEN_DEFAULT_PAGE_SIZES,
- GEN_CHV_PIPEOFFSETS,
- CURSOR_OFFSETS,
+ CHV_PIPE_OFFSETS,
+ CHV_CURSOR_OFFSETS,
CHV_COLORS,
+ GEN_DEFAULT_PAGE_SIZES,
};
#define GEN9_DEFAULT_PAGE_SIZES \
#define SKL_GT3_PLUS_PLATFORM \
SKL_PLATFORM, \
- .ring_mask = RENDER_RING | BSD_RING | BLT_RING | VEBOX_RING | BSD2_RING
+ .engine_mask = \
+ BIT(RCS0) | BIT(VCS0) | BIT(BCS0) | BIT(VECS0) | BIT(VCS1)
static const struct intel_device_info intel_skylake_gt3_info = {
GEN(9), \
.is_lp = 1, \
.display.has_hotplug = 1, \
- .ring_mask = RENDER_RING | BSD_RING | BLT_RING | VEBOX_RING, \
+ .engine_mask = BIT(RCS0) | BIT(VCS0) | BIT(BCS0) | BIT(VECS0), \
.num_pipes = 3, \
.has_64bit_reloc = 1, \
.display.has_ddi = 1, \
.has_logical_ring_contexts = 1, \
.has_logical_ring_preemption = 1, \
.has_guc = 1, \
- .ppgtt = INTEL_PPGTT_FULL_4LVL, \
+ .ppgtt_type = INTEL_PPGTT_FULL, \
+ .ppgtt_size = 48, \
.has_reset_engine = 1, \
.has_snoop = true, \
.has_coherent_ggtt = false, \
.display.has_ipc = 1, \
- GEN9_DEFAULT_PAGE_SIZES, \
- GEN_DEFAULT_PIPEOFFSETS, \
+ HSW_PIPE_OFFSETS, \
IVB_CURSOR_OFFSETS, \
- BDW_COLORS
+ IVB_COLORS, \
+ GEN9_DEFAULT_PAGE_SIZES
static const struct intel_device_info intel_broxton_info = {
GEN9_LP_FEATURES,
static const struct intel_device_info intel_kabylake_gt3_info = {
KBL_PLATFORM,
.gt = 3,
- .ring_mask = RENDER_RING | BSD_RING | BLT_RING | VEBOX_RING | BSD2_RING,
+ .engine_mask =
+ BIT(RCS0) | BIT(VCS0) | BIT(BCS0) | BIT(VECS0) | BIT(VCS1),
};
#define CFL_PLATFORM \
static const struct intel_device_info intel_coffeelake_gt3_info = {
CFL_PLATFORM,
.gt = 3,
- .ring_mask = RENDER_RING | BSD_RING | BLT_RING | VEBOX_RING | BSD2_RING,
+ .engine_mask =
+ BIT(RCS0) | BIT(VCS0) | BIT(BCS0) | BIT(VECS0) | BIT(VCS1),
};
#define GEN10_FEATURES \
}, \
GEN(11), \
.ddb_size = 2048, \
- .has_logical_ring_elsq = 1
+ .has_logical_ring_elsq = 1, \
+ .color = { .degamma_lut_size = 33, .gamma_lut_size = 1024 }
static const struct intel_device_info intel_icelake_11_info = {
GEN11_FEATURES,
PLATFORM(INTEL_ICELAKE),
+ .engine_mask =
+ BIT(RCS0) | BIT(BCS0) | BIT(VECS0) | BIT(VCS0) | BIT(VCS2),
+};
+
+static const struct intel_device_info intel_elkhartlake_info = {
+ GEN11_FEATURES,
+ PLATFORM(INTEL_ELKHARTLAKE),
.is_alpha_support = 1,
- .ring_mask = RENDER_RING | BLT_RING | VEBOX_RING | BSD_RING | BSD3_RING,
+ .engine_mask = BIT(RCS0) | BIT(BCS0) | BIT(VCS0),
+ .ppgtt_size = 36,
};
#undef GEN
INTEL_I965GM_IDS(&intel_i965gm_info),
INTEL_GM45_IDS(&intel_gm45_info),
INTEL_G45_IDS(&intel_g45_info),
- INTEL_PINEVIEW_IDS(&intel_pineview_info),
+ INTEL_PINEVIEW_G_IDS(&intel_pineview_g_info),
+ INTEL_PINEVIEW_M_IDS(&intel_pineview_m_info),
INTEL_IRONLAKE_D_IDS(&intel_ironlake_d_info),
INTEL_IRONLAKE_M_IDS(&intel_ironlake_m_info),
INTEL_SNB_D_GT1_IDS(&intel_sandybridge_d_gt1_info),
INTEL_WHL_U_GT2_IDS(&intel_coffeelake_gt2_info),
INTEL_AML_CFL_GT2_IDS(&intel_coffeelake_gt2_info),
INTEL_WHL_U_GT3_IDS(&intel_coffeelake_gt3_info),
+ INTEL_CML_GT1_IDS(&intel_coffeelake_gt1_info),
+ INTEL_CML_GT2_IDS(&intel_coffeelake_gt2_info),
INTEL_CNL_IDS(&intel_cannonlake_info),
INTEL_ICL_11_IDS(&intel_icelake_11_info),
+ INTEL_EHL_IDS(&intel_elkhartlake_info),
{0, 0, 0}
};
MODULE_DEVICE_TABLE(pci, pciidlist);
bool use_kms = true;
int err;
- i915_global_active_init();
+ err = i915_globals_init();
+ if (err)
+ return err;
err = i915_mock_selftests();
if (err)
return;
pci_unregister_driver(&i915_pci_driver);
- i915_global_active_exit();
+ i915_globals_exit();
}
module_init(i915_init);
static struct intel_context *oa_pin_context(struct drm_i915_private *i915,
struct i915_gem_context *ctx)
{
- struct intel_engine_cs *engine = i915->engine[RCS];
+ struct intel_engine_cs *engine = i915->engine[RCS0];
struct intel_context *ce;
int ret;
free_oa_buffer(dev_priv);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
intel_runtime_pm_put(dev_priv, stream->wakeref);
if (stream->ctx)
goto unlock;
}
- ret = i915_gem_object_set_cache_level(bo, I915_CACHE_LLC);
- if (ret)
- goto err_unref;
+ i915_gem_object_set_cache_coherency(bo, I915_CACHE_LLC);
/* PreHSW required 512K alignment, HSW requires 16M */
vma = i915_gem_object_ggtt_pin(bo, NULL, 0, SZ_16M, 0);
* It's fine to put out-of-date values into these per-context registers
* in the case that the OA unit has been disabled.
*/
-static void gen8_update_reg_state_unlocked(struct i915_gem_context *ctx,
- u32 *reg_state,
- const struct i915_oa_config *oa_config)
+static void
+gen8_update_reg_state_unlocked(struct intel_context *ce,
+ u32 *reg_state,
+ const struct i915_oa_config *oa_config)
{
- struct drm_i915_private *dev_priv = ctx->i915;
- u32 ctx_oactxctrl = dev_priv->perf.oa.ctx_oactxctrl_offset;
- u32 ctx_flexeu0 = dev_priv->perf.oa.ctx_flexeu0_offset;
+ struct drm_i915_private *i915 = ce->gem_context->i915;
+ u32 ctx_oactxctrl = i915->perf.oa.ctx_oactxctrl_offset;
+ u32 ctx_flexeu0 = i915->perf.oa.ctx_flexeu0_offset;
/* The MMIO offsets for Flex EU registers aren't contiguous */
i915_reg_t flex_regs[] = {
EU_PERF_CNTL0,
int i;
CTX_REG(reg_state, ctx_oactxctrl, GEN8_OACTXCONTROL,
- (dev_priv->perf.oa.period_exponent << GEN8_OA_TIMER_PERIOD_SHIFT) |
- (dev_priv->perf.oa.periodic ? GEN8_OA_TIMER_ENABLE : 0) |
+ (i915->perf.oa.period_exponent << GEN8_OA_TIMER_PERIOD_SHIFT) |
+ (i915->perf.oa.periodic ? GEN8_OA_TIMER_ENABLE : 0) |
GEN8_OA_COUNTER_RESUME);
for (i = 0; i < ARRAY_SIZE(flex_regs); i++) {
CTX_REG(reg_state, state_offset, flex_regs[i], value);
}
- CTX_REG(reg_state, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
- gen8_make_rpcs(dev_priv,
- &to_intel_context(ctx,
- dev_priv->engine[RCS])->sseu));
+ CTX_REG(reg_state,
+ CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE,
+ gen8_make_rpcs(i915, &ce->sseu));
}
/*
static int gen8_configure_all_contexts(struct drm_i915_private *dev_priv,
const struct i915_oa_config *oa_config)
{
- struct intel_engine_cs *engine = dev_priv->engine[RCS];
+ struct intel_engine_cs *engine = dev_priv->engine[RCS0];
unsigned int map_type = i915_coherent_map_type(dev_priv);
struct i915_gem_context *ctx;
struct i915_request *rq;
/* Update all contexts now that we've stalled the submission. */
list_for_each_entry(ctx, &dev_priv->contexts.list, link) {
- struct intel_context *ce = to_intel_context(ctx, engine);
+ struct intel_context *ce = intel_context_lookup(ctx, engine);
u32 *regs;
/* OA settings will be set upon first use */
- if (!ce->state)
+ if (!ce || !ce->state)
continue;
regs = i915_gem_object_pin_map(ce->state->obj, map_type);
ce->state->obj->mm.dirty = true;
regs += LRC_STATE_PN * PAGE_SIZE / sizeof(*regs);
- gen8_update_reg_state_unlocked(ctx, regs, oa_config);
+ gen8_update_reg_state_unlocked(ce, regs, oa_config);
i915_gem_object_unpin_map(ce->state->obj);
}
static void gen7_oa_disable(struct i915_perf_stream *stream)
{
- struct drm_i915_private *dev_priv = stream->dev_priv;
+ struct intel_uncore *uncore = &stream->dev_priv->uncore;
- I915_WRITE(GEN7_OACONTROL, 0);
- if (intel_wait_for_register(dev_priv,
+ intel_uncore_write(uncore, GEN7_OACONTROL, 0);
+ if (intel_wait_for_register(uncore,
GEN7_OACONTROL, GEN7_OACONTROL_ENABLE, 0,
50))
DRM_ERROR("wait for OA to be disabled timed out\n");
static void gen8_oa_disable(struct i915_perf_stream *stream)
{
- struct drm_i915_private *dev_priv = stream->dev_priv;
+ struct intel_uncore *uncore = &stream->dev_priv->uncore;
- I915_WRITE(GEN8_OACONTROL, 0);
- if (intel_wait_for_register(dev_priv,
+ intel_uncore_write(uncore, GEN8_OACONTROL, 0);
+ if (intel_wait_for_register(uncore,
GEN8_OACONTROL, GEN8_OA_COUNTER_ENABLE, 0,
50))
DRM_ERROR("wait for OA to be disabled timed out\n");
* references will effectively disable RC6.
*/
stream->wakeref = intel_runtime_pm_get(dev_priv);
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
ret = alloc_oa_buffer(dev_priv);
if (ret)
err_oa_buf_alloc:
put_oa_config(dev_priv, stream->oa_config);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
intel_runtime_pm_put(dev_priv, stream->wakeref);
err_config:
}
void i915_oa_init_reg_state(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx,
- u32 *reg_state)
+ struct intel_context *ce,
+ u32 *regs)
{
struct i915_perf_stream *stream;
- if (engine->id != RCS)
+ if (engine->class != RENDER_CLASS)
return;
stream = engine->i915->perf.oa.exclusive_stream;
if (stream)
- gen8_update_reg_state_unlocked(ctx, reg_state, stream->oa_config);
+ gen8_update_reg_state_unlocked(ce, regs, stream->oa_config);
}
/**
sysfs_attr_init(&dev_priv->perf.oa.test_config.sysfs_metric_id.attr);
- if (IS_HASWELL(dev_priv)) {
- i915_perf_load_test_config_hsw(dev_priv);
- } else if (IS_BROADWELL(dev_priv)) {
- i915_perf_load_test_config_bdw(dev_priv);
- } else if (IS_CHERRYVIEW(dev_priv)) {
- i915_perf_load_test_config_chv(dev_priv);
+ if (INTEL_GEN(dev_priv) >= 11) {
+ i915_perf_load_test_config_icl(dev_priv);
+ } else if (IS_CANNONLAKE(dev_priv)) {
+ i915_perf_load_test_config_cnl(dev_priv);
+ } else if (IS_COFFEELAKE(dev_priv)) {
+ if (IS_CFL_GT2(dev_priv))
+ i915_perf_load_test_config_cflgt2(dev_priv);
+ if (IS_CFL_GT3(dev_priv))
+ i915_perf_load_test_config_cflgt3(dev_priv);
+ } else if (IS_GEMINILAKE(dev_priv)) {
+ i915_perf_load_test_config_glk(dev_priv);
+ } else if (IS_KABYLAKE(dev_priv)) {
+ if (IS_KBL_GT2(dev_priv))
+ i915_perf_load_test_config_kblgt2(dev_priv);
+ else if (IS_KBL_GT3(dev_priv))
+ i915_perf_load_test_config_kblgt3(dev_priv);
+ } else if (IS_BROXTON(dev_priv)) {
+ i915_perf_load_test_config_bxt(dev_priv);
} else if (IS_SKYLAKE(dev_priv)) {
if (IS_SKL_GT2(dev_priv))
i915_perf_load_test_config_sklgt2(dev_priv);
i915_perf_load_test_config_sklgt3(dev_priv);
else if (IS_SKL_GT4(dev_priv))
i915_perf_load_test_config_sklgt4(dev_priv);
- } else if (IS_BROXTON(dev_priv)) {
- i915_perf_load_test_config_bxt(dev_priv);
- } else if (IS_KABYLAKE(dev_priv)) {
- if (IS_KBL_GT2(dev_priv))
- i915_perf_load_test_config_kblgt2(dev_priv);
- else if (IS_KBL_GT3(dev_priv))
- i915_perf_load_test_config_kblgt3(dev_priv);
- } else if (IS_GEMINILAKE(dev_priv)) {
- i915_perf_load_test_config_glk(dev_priv);
- } else if (IS_COFFEELAKE(dev_priv)) {
- if (IS_CFL_GT2(dev_priv))
- i915_perf_load_test_config_cflgt2(dev_priv);
- if (IS_CFL_GT3(dev_priv))
- i915_perf_load_test_config_cflgt3(dev_priv);
- } else if (IS_CANNONLAKE(dev_priv)) {
- i915_perf_load_test_config_cnl(dev_priv);
- } else if (IS_ICELAKE(dev_priv)) {
- i915_perf_load_test_config_icl(dev_priv);
- }
+ } else if (IS_CHERRYVIEW(dev_priv)) {
+ i915_perf_load_test_config_chv(dev_priv);
+ } else if (IS_BROADWELL(dev_priv)) {
+ i915_perf_load_test_config_bdw(dev_priv);
+ } else if (IS_HASWELL(dev_priv)) {
+ i915_perf_load_test_config_hsw(dev_priv);
+}
if (dev_priv->perf.oa.test_config.id == 0)
goto sysfs_error;
*
* Use RCS as proxy for all engines.
*/
- else if (intel_engine_supports_stats(i915->engine[RCS]))
+ else if (intel_engine_supports_stats(i915->engine[RCS0]))
enable &= ~BIT(I915_SAMPLE_BUSY);
/*
spin_unlock_irq(&i915->pmu.lock);
}
-static bool grab_forcewake(struct drm_i915_private *i915, bool fw)
-{
- if (!fw)
- intel_uncore_forcewake_get(i915, FORCEWAKE_ALL);
-
- return true;
-}
-
static void
add_sample(struct i915_pmu_sample *sample, u32 val)
{
struct intel_engine_cs *engine;
enum intel_engine_id id;
intel_wakeref_t wakeref;
- bool fw = false;
+ unsigned long flags;
if ((dev_priv->pmu.enable & ENGINE_SAMPLE_MASK) == 0)
return;
- if (!dev_priv->gt.awake)
- return;
-
- wakeref = intel_runtime_pm_get_if_in_use(dev_priv);
+ wakeref = 0;
+ if (READ_ONCE(dev_priv->gt.awake))
+ wakeref = intel_runtime_pm_get_if_in_use(dev_priv);
if (!wakeref)
return;
+ spin_lock_irqsave(&dev_priv->uncore.lock, flags);
for_each_engine(engine, dev_priv, id) {
- u32 current_seqno = intel_engine_get_seqno(engine);
- u32 last_seqno = intel_engine_last_submit(engine);
+ struct intel_engine_pmu *pmu = &engine->pmu;
+ bool busy;
u32 val;
- val = !i915_seqno_passed(current_seqno, last_seqno);
-
- if (val)
- add_sample(&engine->pmu.sample[I915_SAMPLE_BUSY],
- period_ns);
-
- if (val && (engine->pmu.enable &
- (BIT(I915_SAMPLE_WAIT) | BIT(I915_SAMPLE_SEMA)))) {
- fw = grab_forcewake(dev_priv, fw);
-
- val = I915_READ_FW(RING_CTL(engine->mmio_base));
- } else {
- val = 0;
- }
+ val = I915_READ_FW(RING_CTL(engine->mmio_base));
+ if (val == 0) /* powerwell off => engine idle */
+ continue;
if (val & RING_WAIT)
- add_sample(&engine->pmu.sample[I915_SAMPLE_WAIT],
- period_ns);
-
+ add_sample(&pmu->sample[I915_SAMPLE_WAIT], period_ns);
if (val & RING_WAIT_SEMAPHORE)
- add_sample(&engine->pmu.sample[I915_SAMPLE_SEMA],
- period_ns);
- }
+ add_sample(&pmu->sample[I915_SAMPLE_SEMA], period_ns);
- if (fw)
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ /*
+ * While waiting on a semaphore or event, MI_MODE reports the
+ * ring as idle. However, previously using the seqno, and with
+ * execlists sampling, we account for the ring waiting as the
+ * engine being busy. Therefore, we record the sample as being
+ * busy if either waiting or !idle.
+ */
+ busy = val & (RING_WAIT_SEMAPHORE | RING_WAIT);
+ if (!busy) {
+ val = I915_READ_FW(RING_MI_MODE(engine->mmio_base));
+ busy = !(val & MODE_IDLE);
+ }
+ if (busy)
+ add_sample(&pmu->sample[I915_SAMPLE_BUSY], period_ns);
+ }
+ spin_unlock_irqrestore(&dev_priv->uncore.lock, flags);
intel_runtime_pm_put(dev_priv, wakeref);
}
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2018 Intel Corporation
+ */
+
+#ifndef _I915_PRIOLIST_TYPES_H_
+#define _I915_PRIOLIST_TYPES_H_
+
+#include <linux/list.h>
+#include <linux/rbtree.h>
+
+#include <uapi/drm/i915_drm.h>
+
+enum {
+ I915_PRIORITY_MIN = I915_CONTEXT_MIN_USER_PRIORITY - 1,
+ I915_PRIORITY_NORMAL = I915_CONTEXT_DEFAULT_PRIORITY,
+ I915_PRIORITY_MAX = I915_CONTEXT_MAX_USER_PRIORITY + 1,
+
+ I915_PRIORITY_INVALID = INT_MIN
+};
+
+#define I915_USER_PRIORITY_SHIFT 3
+#define I915_USER_PRIORITY(x) ((x) << I915_USER_PRIORITY_SHIFT)
+
+#define I915_PRIORITY_COUNT BIT(I915_USER_PRIORITY_SHIFT)
+#define I915_PRIORITY_MASK (I915_PRIORITY_COUNT - 1)
+
+#define I915_PRIORITY_WAIT ((u8)BIT(0))
+#define I915_PRIORITY_NEWCLIENT ((u8)BIT(1))
+#define I915_PRIORITY_NOSEMAPHORE ((u8)BIT(2))
+
+#define __NO_PREEMPTION (I915_PRIORITY_WAIT)
+
+struct i915_priolist {
+ struct list_head requests[I915_PRIORITY_COUNT];
+ struct rb_node node;
+ unsigned long used;
+ int priority;
+};
+
+#endif /* _I915_PRIOLIST_TYPES_H_ */
/*
* VGT capabilities type
*/
-#define VGT_CAPS_FULL_48BIT_PPGTT BIT(2)
+#define VGT_CAPS_FULL_PPGTT BIT(2)
#define VGT_CAPS_HWSP_EMULATION BIT(3)
#define VGT_CAPS_HUGE_GTT BIT(4)
#include "i915_query.h"
#include <uapi/drm/i915_drm.h>
+static int copy_query_item(void *query_hdr, size_t query_sz,
+ u32 total_length,
+ struct drm_i915_query_item *query_item)
+{
+ if (query_item->length == 0)
+ return total_length;
+
+ if (query_item->length < total_length)
+ return -EINVAL;
+
+ if (copy_from_user(query_hdr, u64_to_user_ptr(query_item->data_ptr),
+ query_sz))
+ return -EFAULT;
+
+ if (!access_ok(u64_to_user_ptr(query_item->data_ptr),
+ total_length))
+ return -EFAULT;
+
+ return 0;
+}
+
static int query_topology_info(struct drm_i915_private *dev_priv,
struct drm_i915_query_item *query_item)
{
const struct sseu_dev_info *sseu = &RUNTIME_INFO(dev_priv)->sseu;
struct drm_i915_query_topology_info topo;
u32 slice_length, subslice_length, eu_length, total_length;
+ int ret;
if (query_item->flags != 0)
return -EINVAL;
total_length = sizeof(topo) + slice_length + subslice_length + eu_length;
- if (query_item->length == 0)
- return total_length;
-
- if (query_item->length < total_length)
- return -EINVAL;
-
- if (copy_from_user(&topo, u64_to_user_ptr(query_item->data_ptr),
- sizeof(topo)))
- return -EFAULT;
+ ret = copy_query_item(&topo, sizeof(topo), total_length,
+ query_item);
+ if (ret != 0)
+ return ret;
if (topo.flags != 0)
return -EINVAL;
- if (!access_ok(u64_to_user_ptr(query_item->data_ptr),
- total_length))
- return -EFAULT;
-
memset(&topo, 0, sizeof(topo));
topo.max_slices = sseu->max_slices;
topo.max_subslices = sseu->max_subslices;
#ifndef _I915_REG_H_
#define _I915_REG_H_
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+
/**
* DOC: The i915 register macro definition style guide
*
* significant to least significant bit. Indent the register content macros
* using two extra spaces between ``#define`` and the macro name.
*
- * For bit fields, define a ``_MASK`` and a ``_SHIFT`` macro. Define bit field
- * contents so that they are already shifted in place, and can be directly
- * OR'd. For convenience, function-like macros may be used to define bit fields,
- * but do note that the macros may be needed to read as well as write the
- * register contents.
+ * Define bit fields using ``REG_GENMASK(h, l)``. Define bit field contents
+ * using ``REG_FIELD_PREP(mask, value)``. This will define the values already
+ * shifted in place, so they can be directly OR'd together. For convenience,
+ * function-like macros may be used to define bit fields, but do note that the
+ * macros may be needed to read as well as write the register contents.
*
- * Define bits using ``(1 << N)`` instead of ``BIT(N)``. We may change this in
- * the future, but this is the prevailing style. Do **not** add ``_BIT`` suffix
- * to the name.
+ * Define bits using ``REG_BIT(N)``. Do **not** add ``_BIT`` suffix to the name.
*
* Group the register and its contents together without blank lines, separate
* from other registers and their contents with one blank line.
* #define _FOO_A 0xf000
* #define _FOO_B 0xf001
* #define FOO(pipe) _MMIO_PIPE(pipe, _FOO_A, _FOO_B)
- * #define FOO_ENABLE (1 << 31)
- * #define FOO_MODE_MASK (0xf << 16)
- * #define FOO_MODE_SHIFT 16
- * #define FOO_MODE_BAR (0 << 16)
- * #define FOO_MODE_BAZ (1 << 16)
- * #define FOO_MODE_QUX_SNB (2 << 16)
+ * #define FOO_ENABLE REG_BIT(31)
+ * #define FOO_MODE_MASK REG_GENMASK(19, 16)
+ * #define FOO_MODE_BAR REG_FIELD_PREP(FOO_MODE_MASK, 0)
+ * #define FOO_MODE_BAZ REG_FIELD_PREP(FOO_MODE_MASK, 1)
+ * #define FOO_MODE_QUX_SNB REG_FIELD_PREP(FOO_MODE_MASK, 2)
*
* #define BAR _MMIO(0xb000)
* #define GEN8_BAR _MMIO(0xb888)
*/
+/**
+ * REG_BIT() - Prepare a u32 bit value
+ * @__n: 0-based bit number
+ *
+ * Local wrapper for BIT() to force u32, with compile time checks.
+ *
+ * @return: Value with bit @__n set.
+ */
+#define REG_BIT(__n) \
+ ((u32)(BIT(__n) + \
+ BUILD_BUG_ON_ZERO(__builtin_constant_p(__n) && \
+ ((__n) < 0 || (__n) > 31))))
+
+/**
+ * REG_GENMASK() - Prepare a continuous u32 bitmask
+ * @__high: 0-based high bit
+ * @__low: 0-based low bit
+ *
+ * Local wrapper for GENMASK() to force u32, with compile time checks.
+ *
+ * @return: Continuous bitmask from @__high to @__low, inclusive.
+ */
+#define REG_GENMASK(__high, __low) \
+ ((u32)(GENMASK(__high, __low) + \
+ BUILD_BUG_ON_ZERO(__builtin_constant_p(__high) && \
+ __builtin_constant_p(__low) && \
+ ((__low) < 0 || (__high) > 31 || (__low) > (__high)))))
+
+/*
+ * Local integer constant expression version of is_power_of_2().
+ */
+#define IS_POWER_OF_2(__x) ((__x) && (((__x) & ((__x) - 1)) == 0))
+
+/**
+ * REG_FIELD_PREP() - Prepare a u32 bitfield value
+ * @__mask: shifted mask defining the field's length and position
+ * @__val: value to put in the field
+
+ * Local copy of FIELD_PREP() to generate an integer constant expression, force
+ * u32 and for consistency with REG_FIELD_GET(), REG_BIT() and REG_GENMASK().
+ *
+ * @return: @__val masked and shifted into the field defined by @__mask.
+ */
+#define REG_FIELD_PREP(__mask, __val) \
+ ((u32)((((typeof(__mask))(__val) << __bf_shf(__mask)) & (__mask)) + \
+ BUILD_BUG_ON_ZERO(!__is_constexpr(__mask)) + \
+ BUILD_BUG_ON_ZERO((__mask) == 0 || (__mask) > U32_MAX) + \
+ BUILD_BUG_ON_ZERO(!IS_POWER_OF_2((__mask) + (1ULL << __bf_shf(__mask)))) + \
+ BUILD_BUG_ON_ZERO(__builtin_choose_expr(__is_constexpr(__val), (~((__mask) >> __bf_shf(__mask)) & (__val)), 0))))
+
+/**
+ * REG_FIELD_GET() - Extract a u32 bitfield value
+ * @__mask: shifted mask defining the field's length and position
+ * @__val: value to extract the bitfield value from
+ *
+ * Local wrapper for FIELD_GET() to force u32 and for consistency with
+ * REG_FIELD_PREP(), REG_BIT() and REG_GENMASK().
+ *
+ * @return: Masked and shifted value of the field defined by @__mask in @__val.
+ */
+#define REG_FIELD_GET(__mask, __val) ((u32)FIELD_GET(__mask, __val))
+
typedef struct {
u32 reg;
} i915_reg_t;
/* Engine ID */
-#define RCS_HW 0
-#define VCS_HW 1
-#define BCS_HW 2
-#define VECS_HW 3
-#define VCS2_HW 4
-#define VCS3_HW 6
-#define VCS4_HW 7
-#define VECS2_HW 12
+#define RCS0_HW 0
+#define VCS0_HW 1
+#define BCS0_HW 2
+#define VECS0_HW 3
+#define VCS1_HW 4
+#define VCS2_HW 6
+#define VCS3_HW 7
+#define VECS1_HW 12
/* Engine class */
#define GEN11_VECS_SFC_USAGE(engine) _MMIO((engine)->mmio_base + 0x2014)
#define GEN11_VECS_SFC_USAGE_BIT (1 << 0)
-#define RING_PP_DIR_BASE(engine) _MMIO((engine)->mmio_base + 0x228)
-#define RING_PP_DIR_BASE_READ(engine) _MMIO((engine)->mmio_base + 0x518)
-#define RING_PP_DIR_DCLV(engine) _MMIO((engine)->mmio_base + 0x220)
+#define RING_PP_DIR_BASE(base) _MMIO((base) + 0x228)
+#define RING_PP_DIR_BASE_READ(base) _MMIO((base) + 0x518)
+#define RING_PP_DIR_DCLV(base) _MMIO((base) + 0x220)
#define PP_DIR_DCLV_2G 0xffffffff
-#define GEN8_RING_PDP_UDW(engine, n) _MMIO((engine)->mmio_base + 0x270 + (n) * 8 + 4)
-#define GEN8_RING_PDP_LDW(engine, n) _MMIO((engine)->mmio_base + 0x270 + (n) * 8)
+#define GEN8_RING_PDP_UDW(base, n) _MMIO((base) + 0x270 + (n) * 8 + 4)
+#define GEN8_RING_PDP_LDW(base, n) _MMIO((base) + 0x270 + (n) * 8)
#define GEN8_R_PWR_CLK_STATE _MMIO(0x20C8)
#define GEN8_RPCS_ENABLE (1 << 31)
/* See configdb bunit SB addr map */
#define BUNIT_REG_BISOC 0x11
-#define PUNIT_REG_DSPFREQ 0x36
+/* PUNIT_REG_*SSPM0 */
+#define _SSPM0_SSC(val) ((val) << 0)
+#define SSPM0_SSC_MASK _SSPM0_SSC(0x3)
+#define SSPM0_SSC_PWR_ON _SSPM0_SSC(0x0)
+#define SSPM0_SSC_CLK_GATE _SSPM0_SSC(0x1)
+#define SSPM0_SSC_RESET _SSPM0_SSC(0x2)
+#define SSPM0_SSC_PWR_GATE _SSPM0_SSC(0x3)
+#define _SSPM0_SSS(val) ((val) << 24)
+#define SSPM0_SSS_MASK _SSPM0_SSS(0x3)
+#define SSPM0_SSS_PWR_ON _SSPM0_SSS(0x0)
+#define SSPM0_SSS_CLK_GATE _SSPM0_SSS(0x1)
+#define SSPM0_SSS_RESET _SSPM0_SSS(0x2)
+#define SSPM0_SSS_PWR_GATE _SSPM0_SSS(0x3)
+
+/* PUNIT_REG_*SSPM1 */
+#define SSPM1_FREQSTAT_SHIFT 24
+#define SSPM1_FREQSTAT_MASK (0x1f << SSPM1_FREQSTAT_SHIFT)
+#define SSPM1_FREQGUAR_SHIFT 8
+#define SSPM1_FREQGUAR_MASK (0x1f << SSPM1_FREQGUAR_SHIFT)
+#define SSPM1_FREQ_SHIFT 0
+#define SSPM1_FREQ_MASK (0x1f << SSPM1_FREQ_SHIFT)
+
+#define PUNIT_REG_VEDSSPM0 0x32
+#define PUNIT_REG_VEDSSPM1 0x33
+
+#define PUNIT_REG_DSPSSPM 0x36
#define DSPFREQSTAT_SHIFT_CHV 24
#define DSPFREQSTAT_MASK_CHV (0x1f << DSPFREQSTAT_SHIFT_CHV)
#define DSPFREQGUAR_SHIFT_CHV 8
#define DP_SSS_RESET(pipe) _DP_SSS(0x2, (pipe))
#define DP_SSS_PWR_GATE(pipe) _DP_SSS(0x3, (pipe))
+#define PUNIT_REG_ISPSSPM0 0x39
+#define PUNIT_REG_ISPSSPM1 0x3a
+
/*
* i915_power_well_id:
*
#define _CNL_PORT_TX_DW4_LN1_AE 0x1624D0
#define CNL_PORT_TX_DW4_GRP(port) _MMIO(_CNL_PORT_TX_DW_GRP(4, (port)))
#define CNL_PORT_TX_DW4_LN0(port) _MMIO(_CNL_PORT_TX_DW_LN0(4, (port)))
-#define CNL_PORT_TX_DW4_LN(port, ln) _MMIO(_CNL_PORT_TX_DW_LN0(4, (port)) + \
+#define CNL_PORT_TX_DW4_LN(ln, port) _MMIO(_CNL_PORT_TX_DW_LN0(4, (port)) + \
((ln) * (_CNL_PORT_TX_DW4_LN1_AE - \
_CNL_PORT_TX_DW4_LN0_AE)))
#define ICL_PORT_TX_DW4_AUX(port) _MMIO(_ICL_PORT_TX_DW_AUX(4, port))
#define ICL_PORT_TX_DW4_GRP(port) _MMIO(_ICL_PORT_TX_DW_GRP(4, port))
#define ICL_PORT_TX_DW4_LN0(port) _MMIO(_ICL_PORT_TX_DW_LN(4, 0, port))
-#define ICL_PORT_TX_DW4_LN(port, ln) _MMIO(_ICL_PORT_TX_DW_LN(4, ln, port))
+#define ICL_PORT_TX_DW4_LN(ln, port) _MMIO(_ICL_PORT_TX_DW_LN(4, ln, port))
#define LOADGEN_SELECT (1 << 31)
#define POST_CURSOR_1(x) ((x) << 12)
#define POST_CURSOR_1_MASK (0x3F << 12)
#define ICL_PORT_TX_DW7_AUX(port) _MMIO(_ICL_PORT_TX_DW_AUX(7, port))
#define ICL_PORT_TX_DW7_GRP(port) _MMIO(_ICL_PORT_TX_DW_GRP(7, port))
#define ICL_PORT_TX_DW7_LN0(port) _MMIO(_ICL_PORT_TX_DW_LN(7, 0, port))
-#define ICL_PORT_TX_DW7_LN(port, ln) _MMIO(_ICL_PORT_TX_DW_LN(7, ln, port))
+#define ICL_PORT_TX_DW7_LN(ln, port) _MMIO(_ICL_PORT_TX_DW_LN(7, ln, port))
#define N_SCALAR(x) ((x) << 24)
#define N_SCALAR_MASK (0x7F << 24)
-#define MG_PHY_PORT_LN(port, ln, ln0p1, ln0p2, ln1p1) \
+#define MG_PHY_PORT_LN(ln, port, ln0p1, ln0p2, ln1p1) \
_MMIO(_PORT((port) - PORT_C, ln0p1, ln0p2) + (ln) * ((ln1p1) - (ln0p1)))
#define MG_TX_LINK_PARAMS_TX1LN0_PORT1 0x16812C
#define MG_TX_LINK_PARAMS_TX1LN1_PORT3 0x16A52C
#define MG_TX_LINK_PARAMS_TX1LN0_PORT4 0x16B12C
#define MG_TX_LINK_PARAMS_TX1LN1_PORT4 0x16B52C
-#define MG_TX1_LINK_PARAMS(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_TX_LINK_PARAMS_TX1LN0_PORT1, \
+#define MG_TX1_LINK_PARAMS(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_TX_LINK_PARAMS_TX1LN0_PORT1, \
MG_TX_LINK_PARAMS_TX1LN0_PORT2, \
MG_TX_LINK_PARAMS_TX1LN1_PORT1)
#define MG_TX_LINK_PARAMS_TX2LN1_PORT3 0x16A4AC
#define MG_TX_LINK_PARAMS_TX2LN0_PORT4 0x16B0AC
#define MG_TX_LINK_PARAMS_TX2LN1_PORT4 0x16B4AC
-#define MG_TX2_LINK_PARAMS(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_TX_LINK_PARAMS_TX2LN0_PORT1, \
+#define MG_TX2_LINK_PARAMS(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_TX_LINK_PARAMS_TX2LN0_PORT1, \
MG_TX_LINK_PARAMS_TX2LN0_PORT2, \
MG_TX_LINK_PARAMS_TX2LN1_PORT1)
#define CRI_USE_FS32 (1 << 5)
#define MG_TX_PISO_READLOAD_TX1LN1_PORT3 0x16A54C
#define MG_TX_PISO_READLOAD_TX1LN0_PORT4 0x16B14C
#define MG_TX_PISO_READLOAD_TX1LN1_PORT4 0x16B54C
-#define MG_TX1_PISO_READLOAD(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_TX_PISO_READLOAD_TX1LN0_PORT1, \
+#define MG_TX1_PISO_READLOAD(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_TX_PISO_READLOAD_TX1LN0_PORT1, \
MG_TX_PISO_READLOAD_TX1LN0_PORT2, \
MG_TX_PISO_READLOAD_TX1LN1_PORT1)
#define MG_TX_PISO_READLOAD_TX2LN1_PORT3 0x16A4CC
#define MG_TX_PISO_READLOAD_TX2LN0_PORT4 0x16B0CC
#define MG_TX_PISO_READLOAD_TX2LN1_PORT4 0x16B4CC
-#define MG_TX2_PISO_READLOAD(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_TX_PISO_READLOAD_TX2LN0_PORT1, \
+#define MG_TX2_PISO_READLOAD(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_TX_PISO_READLOAD_TX2LN0_PORT1, \
MG_TX_PISO_READLOAD_TX2LN0_PORT2, \
MG_TX_PISO_READLOAD_TX2LN1_PORT1)
#define CRI_CALCINIT (1 << 1)
#define MG_TX_SWINGCTRL_TX1LN1_PORT3 0x16A548
#define MG_TX_SWINGCTRL_TX1LN0_PORT4 0x16B148
#define MG_TX_SWINGCTRL_TX1LN1_PORT4 0x16B548
-#define MG_TX1_SWINGCTRL(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_TX_SWINGCTRL_TX1LN0_PORT1, \
+#define MG_TX1_SWINGCTRL(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_TX_SWINGCTRL_TX1LN0_PORT1, \
MG_TX_SWINGCTRL_TX1LN0_PORT2, \
MG_TX_SWINGCTRL_TX1LN1_PORT1)
#define MG_TX_SWINGCTRL_TX2LN1_PORT3 0x16A4C8
#define MG_TX_SWINGCTRL_TX2LN0_PORT4 0x16B0C8
#define MG_TX_SWINGCTRL_TX2LN1_PORT4 0x16B4C8
-#define MG_TX2_SWINGCTRL(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_TX_SWINGCTRL_TX2LN0_PORT1, \
+#define MG_TX2_SWINGCTRL(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_TX_SWINGCTRL_TX2LN0_PORT1, \
MG_TX_SWINGCTRL_TX2LN0_PORT2, \
MG_TX_SWINGCTRL_TX2LN1_PORT1)
#define CRI_TXDEEMPH_OVERRIDE_17_12(x) ((x) << 0)
#define MG_TX_DRVCTRL_TX1LN1_TXPORT3 0x16A544
#define MG_TX_DRVCTRL_TX1LN0_TXPORT4 0x16B144
#define MG_TX_DRVCTRL_TX1LN1_TXPORT4 0x16B544
-#define MG_TX1_DRVCTRL(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_TX_DRVCTRL_TX1LN0_TXPORT1, \
+#define MG_TX1_DRVCTRL(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_TX_DRVCTRL_TX1LN0_TXPORT1, \
MG_TX_DRVCTRL_TX1LN0_TXPORT2, \
MG_TX_DRVCTRL_TX1LN1_TXPORT1)
#define MG_TX_DRVCTRL_TX2LN1_PORT3 0x16A4C4
#define MG_TX_DRVCTRL_TX2LN0_PORT4 0x16B0C4
#define MG_TX_DRVCTRL_TX2LN1_PORT4 0x16B4C4
-#define MG_TX2_DRVCTRL(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_TX_DRVCTRL_TX2LN0_PORT1, \
+#define MG_TX2_DRVCTRL(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_TX_DRVCTRL_TX2LN0_PORT1, \
MG_TX_DRVCTRL_TX2LN0_PORT2, \
MG_TX_DRVCTRL_TX2LN1_PORT1)
#define CRI_TXDEEMPH_OVERRIDE_11_6(x) ((x) << 24)
#define MG_CLKHUB_LN1_PORT3 0x16A79C
#define MG_CLKHUB_LN0_PORT4 0x16B39C
#define MG_CLKHUB_LN1_PORT4 0x16B79C
-#define MG_CLKHUB(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_CLKHUB_LN0_PORT1, \
+#define MG_CLKHUB(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_CLKHUB_LN0_PORT1, \
MG_CLKHUB_LN0_PORT2, \
MG_CLKHUB_LN1_PORT1)
#define CFG_LOW_RATE_LKREN_EN (1 << 11)
#define MG_TX_DCC_TX1LN1_PORT3 0x16A510
#define MG_TX_DCC_TX1LN0_PORT4 0x16B110
#define MG_TX_DCC_TX1LN1_PORT4 0x16B510
-#define MG_TX1_DCC(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_TX_DCC_TX1LN0_PORT1, \
+#define MG_TX1_DCC(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_TX_DCC_TX1LN0_PORT1, \
MG_TX_DCC_TX1LN0_PORT2, \
MG_TX_DCC_TX1LN1_PORT1)
#define MG_TX_DCC_TX2LN0_PORT1 0x168090
#define MG_TX_DCC_TX2LN1_PORT3 0x16A490
#define MG_TX_DCC_TX2LN0_PORT4 0x16B090
#define MG_TX_DCC_TX2LN1_PORT4 0x16B490
-#define MG_TX2_DCC(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_TX_DCC_TX2LN0_PORT1, \
+#define MG_TX2_DCC(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_TX_DCC_TX2LN0_PORT1, \
MG_TX_DCC_TX2LN0_PORT2, \
MG_TX_DCC_TX2LN1_PORT1)
#define CFG_AMI_CK_DIV_OVERRIDE_VAL(x) ((x) << 25)
#define MG_DP_MODE_LN1_ACU_PORT3 0x16A7A0
#define MG_DP_MODE_LN0_ACU_PORT4 0x16B3A0
#define MG_DP_MODE_LN1_ACU_PORT4 0x16B7A0
-#define MG_DP_MODE(port, ln) \
- MG_PHY_PORT_LN(port, ln, MG_DP_MODE_LN0_ACU_PORT1, \
+#define MG_DP_MODE(ln, port) \
+ MG_PHY_PORT_LN(ln, port, MG_DP_MODE_LN0_ACU_PORT1, \
MG_DP_MODE_LN0_ACU_PORT2, \
MG_DP_MODE_LN1_ACU_PORT1)
#define MG_DP_MODE_CFG_DP_X2_MODE (1 << 7)
#define RING_HWS_PGA(base) _MMIO((base) + 0x80)
#define RING_HWS_PGA_GEN6(base) _MMIO((base) + 0x2080)
#define RING_RESET_CTL(base) _MMIO((base) + 0xd0)
-#define RESET_CTL_REQUEST_RESET (1 << 0)
-#define RESET_CTL_READY_TO_RESET (1 << 1)
+#define RESET_CTL_CAT_ERROR REG_BIT(2)
+#define RESET_CTL_READY_TO_RESET REG_BIT(1)
+#define RESET_CTL_REQUEST_RESET REG_BIT(0)
+
#define RING_SEMA_WAIT_POLL(base) _MMIO((base) + 0x24c)
#define HSW_GTT_CACHE_EN _MMIO(0x4024)
#define HWS_START_ADDRESS_SHIFT 4
#define PWRCTXA _MMIO(0x2088) /* 965GM+ only */
#define PWRCTX_EN (1 << 0)
-#define IPEIR _MMIO(0x2088)
-#define IPEHR _MMIO(0x208c)
+#define IPEIR(base) _MMIO((base) + 0x88)
+#define IPEHR(base) _MMIO((base) + 0x8c)
#define GEN2_INSTDONE _MMIO(0x2090)
#define NOPID _MMIO(0x2094)
#define HWSTAM _MMIO(0x2098)
-#define DMA_FADD_I8XX _MMIO(0x20d0)
+#define DMA_FADD_I8XX(base) _MMIO((base) + 0xd0)
#define RING_BBSTATE(base) _MMIO((base) + 0x110)
#define RING_BB_PPGTT (1 << 5)
#define RING_SBBADDR(base) _MMIO((base) + 0x114) /* hsw+ */
#define VLV_GU_CTL0 _MMIO(VLV_DISPLAY_BASE + 0x2030)
#define VLV_GU_CTL1 _MMIO(VLV_DISPLAY_BASE + 0x2034)
#define SCPD0 _MMIO(0x209c) /* 915+ only */
-#define IER _MMIO(0x20a0)
-#define IIR _MMIO(0x20a4)
-#define IMR _MMIO(0x20a8)
-#define ISR _MMIO(0x20ac)
+#define GEN2_IER _MMIO(0x20a0)
+#define GEN2_IIR _MMIO(0x20a4)
+#define GEN2_IMR _MMIO(0x20a8)
+#define GEN2_ISR _MMIO(0x20ac)
#define VLV_GUNIT_CLOCK_GATE _MMIO(VLV_DISPLAY_BASE + 0x2060)
#define GINT_DIS (1 << 22)
#define GCFG_DIS (1 << 8)
#define INSTPM_FORCE_ORDERING (1 << 7) /* GEN6+ */
#define INSTPM_TLB_INVALIDATE (1 << 9)
#define INSTPM_SYNC_FLUSH (1 << 5)
-#define ACTHD _MMIO(0x20c8)
+#define ACTHD(base) _MMIO((base) + 0xc8)
#define MEM_MODE _MMIO(0x20cc)
#define MEM_DISPLAY_B_TRICKLE_FEED_DISABLE (1 << 3) /* 830 only */
#define MEM_DISPLAY_A_TRICKLE_FEED_DISABLE (1 << 2) /* 830/845 only */
/*
* Logical Context regs
*/
-#define CCID _MMIO(0x2180)
+#define CCID(base) _MMIO((base) + 0x180)
#define CCID_EN BIT(0)
#define CCID_EXTENDED_STATE_RESTORE BIT(2)
#define CCID_EXTENDED_STATE_SAVE BIT(3)
/* Pipe A CRC regs */
#define _PIPE_CRC_CTL_A 0x60050
#define PIPE_CRC_ENABLE (1 << 31)
+/* skl+ source selection */
+#define PIPE_CRC_SOURCE_PLANE_1_SKL (0 << 28)
+#define PIPE_CRC_SOURCE_PLANE_2_SKL (2 << 28)
+#define PIPE_CRC_SOURCE_DMUX_SKL (4 << 28)
+#define PIPE_CRC_SOURCE_PLANE_3_SKL (6 << 28)
+#define PIPE_CRC_SOURCE_PLANE_4_SKL (7 << 28)
+#define PIPE_CRC_SOURCE_PLANE_5_SKL (5 << 28)
+#define PIPE_CRC_SOURCE_PLANE_6_SKL (3 << 28)
+#define PIPE_CRC_SOURCE_PLANE_7_SKL (1 << 28)
/* ivb+ source selection */
#define PIPE_CRC_SOURCE_PRIMARY_IVB (0 << 29)
#define PIPE_CRC_SOURCE_SPRITE_IVB (1 << 29)
#define PIPESRC(trans) _MMIO_TRANS2(trans, _PIPEASRC)
#define PIPE_MULT(trans) _MMIO_TRANS2(trans, _PIPE_MULT_A)
-/* VLV eDP PSR registers */
-#define _PSRCTLA (VLV_DISPLAY_BASE + 0x60090)
-#define _PSRCTLB (VLV_DISPLAY_BASE + 0x61090)
-#define VLV_EDP_PSR_ENABLE (1 << 0)
-#define VLV_EDP_PSR_RESET (1 << 1)
-#define VLV_EDP_PSR_MODE_MASK (7 << 2)
-#define VLV_EDP_PSR_MODE_HW_TIMER (1 << 3)
-#define VLV_EDP_PSR_MODE_SW_TIMER (1 << 2)
-#define VLV_EDP_PSR_SINGLE_FRAME_UPDATE (1 << 7)
-#define VLV_EDP_PSR_ACTIVE_ENTRY (1 << 8)
-#define VLV_EDP_PSR_SRC_TRANSMITTER_STATE (1 << 9)
-#define VLV_EDP_PSR_DBL_FRAME (1 << 10)
-#define VLV_EDP_PSR_FRAME_COUNT_MASK (0xff << 16)
-#define VLV_EDP_PSR_IDLE_FRAME_SHIFT 16
-#define VLV_PSRCTL(pipe) _MMIO_PIPE(pipe, _PSRCTLA, _PSRCTLB)
-
-#define _VSCSDPA (VLV_DISPLAY_BASE + 0x600a0)
-#define _VSCSDPB (VLV_DISPLAY_BASE + 0x610a0)
-#define VLV_EDP_PSR_SDP_FREQ_MASK (3 << 30)
-#define VLV_EDP_PSR_SDP_FREQ_ONCE (1 << 31)
-#define VLV_EDP_PSR_SDP_FREQ_EVFRAME (1 << 30)
-#define VLV_VSCSDP(pipe) _MMIO_PIPE(pipe, _VSCSDPA, _VSCSDPB)
-
-#define _PSRSTATA (VLV_DISPLAY_BASE + 0x60094)
-#define _PSRSTATB (VLV_DISPLAY_BASE + 0x61094)
-#define VLV_EDP_PSR_LAST_STATE_MASK (7 << 3)
-#define VLV_EDP_PSR_CURR_STATE_MASK 7
-#define VLV_EDP_PSR_DISABLED (0 << 0)
-#define VLV_EDP_PSR_INACTIVE (1 << 0)
-#define VLV_EDP_PSR_IN_TRANS_TO_ACTIVE (2 << 0)
-#define VLV_EDP_PSR_ACTIVE_NORFB_UP (3 << 0)
-#define VLV_EDP_PSR_ACTIVE_SF_UPDATE (4 << 0)
-#define VLV_EDP_PSR_EXIT (5 << 0)
-#define VLV_EDP_PSR_IN_TRANS (1 << 7)
-#define VLV_PSRSTAT(pipe) _MMIO_PIPE(pipe, _PSRSTATA, _PSRSTATB)
-
/* HSW+ eDP PSR registers */
#define HSW_EDP_PSR_BASE 0x64800
#define BDW_EDP_PSR_BASE 0x6f800
#define EDP_PSR_TP2_TP3_TIME_100us (1 << 8)
#define EDP_PSR_TP2_TP3_TIME_2500us (2 << 8)
#define EDP_PSR_TP2_TP3_TIME_0us (3 << 8)
+#define EDP_PSR_TP4_TIME_0US (3 << 6) /* ICL+ */
#define EDP_PSR_TP1_TIME_500us (0 << 4)
#define EDP_PSR_TP1_TIME_100us (1 << 4)
#define EDP_PSR_TP1_TIME_2500us (2 << 4)
#define VIDEO_DIP_ENABLE (1 << 31)
#define VIDEO_DIP_PORT(port) ((port) << 29)
#define VIDEO_DIP_PORT_MASK (3 << 29)
-#define VIDEO_DIP_ENABLE_GCP (1 << 25)
+#define VIDEO_DIP_ENABLE_GCP (1 << 25) /* ilk+ */
#define VIDEO_DIP_ENABLE_AVI (1 << 21)
#define VIDEO_DIP_ENABLE_VENDOR (2 << 21)
-#define VIDEO_DIP_ENABLE_GAMUT (4 << 21)
+#define VIDEO_DIP_ENABLE_GAMUT (4 << 21) /* ilk+ */
#define VIDEO_DIP_ENABLE_SPD (8 << 21)
#define VIDEO_DIP_SELECT_AVI (0 << 19)
#define VIDEO_DIP_SELECT_VENDOR (1 << 19)
+#define VIDEO_DIP_SELECT_GAMUT (2 << 19)
#define VIDEO_DIP_SELECT_SPD (3 << 19)
#define VIDEO_DIP_SELECT_MASK (3 << 19)
#define VIDEO_DIP_FREQ_ONCE (0 << 16)
#define _PP_STATUS 0x61200
#define PP_STATUS(pps_idx) _MMIO_PPS(pps_idx, _PP_STATUS)
-#define PP_ON (1 << 31)
+#define PP_ON REG_BIT(31)
#define _PP_CONTROL_1 0xc7204
#define _PP_CONTROL_2 0xc7304
#define ICP_PP_CONTROL(x) _MMIO(((x) == 1) ? _PP_CONTROL_1 : \
_PP_CONTROL_2)
-#define POWER_CYCLE_DELAY_MASK (0x1f << 4)
-#define POWER_CYCLE_DELAY_SHIFT 4
-#define VDD_OVERRIDE_FORCE (1 << 3)
-#define BACKLIGHT_ENABLE (1 << 2)
-#define PWR_DOWN_ON_RESET (1 << 1)
-#define PWR_STATE_TARGET (1 << 0)
+#define POWER_CYCLE_DELAY_MASK REG_GENMASK(8, 4)
+#define VDD_OVERRIDE_FORCE REG_BIT(3)
+#define BACKLIGHT_ENABLE REG_BIT(2)
+#define PWR_DOWN_ON_RESET REG_BIT(1)
+#define PWR_STATE_TARGET REG_BIT(0)
/*
* Indicates that all dependencies of the panel are on:
*
* - pipe enabled
* - LVDS/DVOB/DVOC on
*/
-#define PP_READY (1 << 30)
-#define PP_SEQUENCE_NONE (0 << 28)
-#define PP_SEQUENCE_POWER_UP (1 << 28)
-#define PP_SEQUENCE_POWER_DOWN (2 << 28)
-#define PP_SEQUENCE_MASK (3 << 28)
-#define PP_SEQUENCE_SHIFT 28
-#define PP_CYCLE_DELAY_ACTIVE (1 << 27)
-#define PP_SEQUENCE_STATE_MASK 0x0000000f
-#define PP_SEQUENCE_STATE_OFF_IDLE (0x0 << 0)
-#define PP_SEQUENCE_STATE_OFF_S0_1 (0x1 << 0)
-#define PP_SEQUENCE_STATE_OFF_S0_2 (0x2 << 0)
-#define PP_SEQUENCE_STATE_OFF_S0_3 (0x3 << 0)
-#define PP_SEQUENCE_STATE_ON_IDLE (0x8 << 0)
-#define PP_SEQUENCE_STATE_ON_S1_0 (0x9 << 0)
-#define PP_SEQUENCE_STATE_ON_S1_2 (0xa << 0)
-#define PP_SEQUENCE_STATE_ON_S1_3 (0xb << 0)
-#define PP_SEQUENCE_STATE_RESET (0xf << 0)
+#define PP_READY REG_BIT(30)
+#define PP_SEQUENCE_MASK REG_GENMASK(29, 28)
+#define PP_SEQUENCE_NONE REG_FIELD_PREP(PP_SEQUENCE_MASK, 0)
+#define PP_SEQUENCE_POWER_UP REG_FIELD_PREP(PP_SEQUENCE_MASK, 1)
+#define PP_SEQUENCE_POWER_DOWN REG_FIELD_PREP(PP_SEQUENCE_MASK, 2)
+#define PP_CYCLE_DELAY_ACTIVE REG_BIT(27)
+#define PP_SEQUENCE_STATE_MASK REG_GENMASK(3, 0)
+#define PP_SEQUENCE_STATE_OFF_IDLE REG_FIELD_PREP(PP_SEQUENCE_STATE_MASK, 0x0)
+#define PP_SEQUENCE_STATE_OFF_S0_1 REG_FIELD_PREP(PP_SEQUENCE_STATE_MASK, 0x1)
+#define PP_SEQUENCE_STATE_OFF_S0_2 REG_FIELD_PREP(PP_SEQUENCE_STATE_MASK, 0x2)
+#define PP_SEQUENCE_STATE_OFF_S0_3 REG_FIELD_PREP(PP_SEQUENCE_STATE_MASK, 0x3)
+#define PP_SEQUENCE_STATE_ON_IDLE REG_FIELD_PREP(PP_SEQUENCE_STATE_MASK, 0x8)
+#define PP_SEQUENCE_STATE_ON_S1_1 REG_FIELD_PREP(PP_SEQUENCE_STATE_MASK, 0x9)
+#define PP_SEQUENCE_STATE_ON_S1_2 REG_FIELD_PREP(PP_SEQUENCE_STATE_MASK, 0xa)
+#define PP_SEQUENCE_STATE_ON_S1_3 REG_FIELD_PREP(PP_SEQUENCE_STATE_MASK, 0xb)
+#define PP_SEQUENCE_STATE_RESET REG_FIELD_PREP(PP_SEQUENCE_STATE_MASK, 0xf)
#define _PP_CONTROL 0x61204
#define PP_CONTROL(pps_idx) _MMIO_PPS(pps_idx, _PP_CONTROL)
-#define PANEL_UNLOCK_REGS (0xabcd << 16)
-#define PANEL_UNLOCK_MASK (0xffff << 16)
-#define BXT_POWER_CYCLE_DELAY_MASK 0x1f0
-#define BXT_POWER_CYCLE_DELAY_SHIFT 4
-#define EDP_FORCE_VDD (1 << 3)
-#define EDP_BLC_ENABLE (1 << 2)
-#define PANEL_POWER_RESET (1 << 1)
-#define PANEL_POWER_ON (1 << 0)
+#define PANEL_UNLOCK_MASK REG_GENMASK(31, 16)
+#define PANEL_UNLOCK_REGS REG_FIELD_PREP(PANEL_UNLOCK_MASK, 0xabcd)
+#define BXT_POWER_CYCLE_DELAY_MASK REG_GENMASK(8, 4)
+#define EDP_FORCE_VDD REG_BIT(3)
+#define EDP_BLC_ENABLE REG_BIT(2)
+#define PANEL_POWER_RESET REG_BIT(1)
+#define PANEL_POWER_ON REG_BIT(0)
#define _PP_ON_DELAYS 0x61208
#define PP_ON_DELAYS(pps_idx) _MMIO_PPS(pps_idx, _PP_ON_DELAYS)
-#define PANEL_PORT_SELECT_SHIFT 30
-#define PANEL_PORT_SELECT_MASK (3 << 30)
-#define PANEL_PORT_SELECT_LVDS (0 << 30)
-#define PANEL_PORT_SELECT_DPA (1 << 30)
-#define PANEL_PORT_SELECT_DPC (2 << 30)
-#define PANEL_PORT_SELECT_DPD (3 << 30)
-#define PANEL_PORT_SELECT_VLV(port) ((port) << 30)
-#define PANEL_POWER_UP_DELAY_MASK 0x1fff0000
-#define PANEL_POWER_UP_DELAY_SHIFT 16
-#define PANEL_LIGHT_ON_DELAY_MASK 0x1fff
-#define PANEL_LIGHT_ON_DELAY_SHIFT 0
+#define PANEL_PORT_SELECT_MASK REG_GENMASK(31, 30)
+#define PANEL_PORT_SELECT_LVDS REG_FIELD_PREP(PANEL_PORT_SELECT_MASK, 0)
+#define PANEL_PORT_SELECT_DPA REG_FIELD_PREP(PANEL_PORT_SELECT_MASK, 1)
+#define PANEL_PORT_SELECT_DPC REG_FIELD_PREP(PANEL_PORT_SELECT_MASK, 2)
+#define PANEL_PORT_SELECT_DPD REG_FIELD_PREP(PANEL_PORT_SELECT_MASK, 3)
+#define PANEL_PORT_SELECT_VLV(port) REG_FIELD_PREP(PANEL_PORT_SELECT_MASK, port)
+#define PANEL_POWER_UP_DELAY_MASK REG_GENMASK(28, 16)
+#define PANEL_LIGHT_ON_DELAY_MASK REG_GENMASK(12, 0)
#define _PP_OFF_DELAYS 0x6120C
#define PP_OFF_DELAYS(pps_idx) _MMIO_PPS(pps_idx, _PP_OFF_DELAYS)
-#define PANEL_POWER_DOWN_DELAY_MASK 0x1fff0000
-#define PANEL_POWER_DOWN_DELAY_SHIFT 16
-#define PANEL_LIGHT_OFF_DELAY_MASK 0x1fff
-#define PANEL_LIGHT_OFF_DELAY_SHIFT 0
+#define PANEL_POWER_DOWN_DELAY_MASK REG_GENMASK(28, 16)
+#define PANEL_LIGHT_OFF_DELAY_MASK REG_GENMASK(12, 0)
#define _PP_DIVISOR 0x61210
#define PP_DIVISOR(pps_idx) _MMIO_PPS(pps_idx, _PP_DIVISOR)
-#define PP_REFERENCE_DIVIDER_MASK 0xffffff00
-#define PP_REFERENCE_DIVIDER_SHIFT 8
-#define PANEL_POWER_CYCLE_DELAY_MASK 0x1f
-#define PANEL_POWER_CYCLE_DELAY_SHIFT 0
+#define PP_REFERENCE_DIVIDER_MASK REG_GENMASK(31, 8)
+#define PANEL_POWER_CYCLE_DELAY_MASK REG_GENMASK(4, 0)
/* Panel fitting */
#define PFIT_CONTROL _MMIO(DISPLAY_MMIO_BASE(dev_priv) + 0x61230)
#define PIPECONF_SINGLE_WIDE 0
#define PIPECONF_PIPE_UNLOCKED 0
#define PIPECONF_PIPE_LOCKED (1 << 25)
-#define PIPECONF_PALETTE 0
-#define PIPECONF_GAMMA (1 << 24)
#define PIPECONF_FORCE_BORDER (1 << 25)
+#define PIPECONF_GAMMA_MODE_MASK_I9XX (1 << 24) /* gmch */
+#define PIPECONF_GAMMA_MODE_MASK_ILK (3 << 24) /* ilk-ivb */
+#define PIPECONF_GAMMA_MODE_8BIT (0 << 24) /* gmch,ilk-ivb */
+#define PIPECONF_GAMMA_MODE_10BIT (1 << 24) /* gmch,ilk-ivb */
+#define PIPECONF_GAMMA_MODE_12BIT (2 << 24) /* ilk-ivb */
+#define PIPECONF_GAMMA_MODE_SPLIT (3 << 24) /* ivb */
+#define PIPECONF_GAMMA_MODE(x) ((x) << 24) /* pass in GAMMA_MODE_MODE_* */
+#define PIPECONF_GAMMA_MODE_SHIFT 24
#define PIPECONF_INTERLACE_MASK (7 << 21)
#define PIPECONF_INTERLACE_MASK_HSW (3 << 21)
/* Note that pre-gen3 does not support interlaced display directly. Panel
#define PIPEFRAMEPIXEL(pipe) _MMIO_PIPE2(pipe, _PIPEAFRAMEPIXEL)
#define PIPESTAT(pipe) _MMIO_PIPE2(pipe, _PIPEASTAT)
+#define _PIPEAGCMAX 0x70010
+#define _PIPEBGCMAX 0x71010
+#define PIPEGCMAX(pipe, i) _MMIO_PIPE2(pipe, _PIPEAGCMAX + (i) * 4)
+
#define _PIPE_MISC_A 0x70030
#define _PIPE_MISC_B 0x71030
#define PIPEMISC_YUV420_ENABLE (1 << 27)
#define _CUR_WM_TRANS_A_0 0x70168
#define _CUR_WM_TRANS_B_0 0x71168
#define PLANE_WM_EN (1 << 31)
+#define PLANE_WM_IGNORE_LINES (1 << 30)
#define PLANE_WM_LINES_SHIFT 14
#define PLANE_WM_LINES_MASK 0x1f
#define PLANE_WM_BLOCKS_MASK 0x7ff /* skl+: 10 bits, icl+ 11 bits */
#define MCURSOR_PIPE_SELECT_SHIFT 28
#define MCURSOR_PIPE_SELECT(pipe) ((pipe) << 28)
#define MCURSOR_GAMMA_ENABLE (1 << 26)
-#define MCURSOR_PIPE_CSC_ENABLE (1 << 24)
+#define MCURSOR_PIPE_CSC_ENABLE (1 << 24) /* ilk+ */
#define MCURSOR_ROTATE_180 (1 << 15)
#define MCURSOR_TRICKLE_FEED_DISABLE (1 << 14)
#define _CURABASE 0x70084
#define DISPPLANE_RGBA888 (0xf << 26)
#define DISPPLANE_STEREO_ENABLE (1 << 25)
#define DISPPLANE_STEREO_DISABLE 0
-#define DISPPLANE_PIPE_CSC_ENABLE (1 << 24)
+#define DISPPLANE_PIPE_CSC_ENABLE (1 << 24) /* ilk+ */
#define DISPPLANE_SEL_PIPE_SHIFT 24
#define DISPPLANE_SEL_PIPE_MASK (3 << DISPPLANE_SEL_PIPE_SHIFT)
#define DISPPLANE_SEL_PIPE(pipe) ((pipe) << DISPPLANE_SEL_PIPE_SHIFT)
#define PLANE_CTL_FORMAT_YUV422 (0 << 24)
#define PLANE_CTL_FORMAT_NV12 (1 << 24)
#define PLANE_CTL_FORMAT_XRGB_2101010 (2 << 24)
+#define PLANE_CTL_FORMAT_P010 (3 << 24)
#define PLANE_CTL_FORMAT_XRGB_8888 (4 << 24)
+#define PLANE_CTL_FORMAT_P012 (5 << 24)
#define PLANE_CTL_FORMAT_XRGB_16161616F (6 << 24)
+#define PLANE_CTL_FORMAT_P016 (7 << 24)
#define PLANE_CTL_FORMAT_AYUV (8 << 24)
#define PLANE_CTL_FORMAT_INDEXED (12 << 24)
#define PLANE_CTL_FORMAT_RGB_565 (14 << 24)
#define ICL_PLANE_CTL_FORMAT_MASK (0x1f << 23)
#define PLANE_CTL_PIPE_CSC_ENABLE (1 << 23) /* Pre-GLK */
+#define PLANE_CTL_FORMAT_Y210 (1 << 23)
+#define PLANE_CTL_FORMAT_Y212 (3 << 23)
+#define PLANE_CTL_FORMAT_Y216 (5 << 23)
+#define PLANE_CTL_FORMAT_Y410 (7 << 23)
+#define PLANE_CTL_FORMAT_Y412 (9 << 23)
+#define PLANE_CTL_FORMAT_Y416 (0xb << 23)
#define PLANE_CTL_KEY_ENABLE_MASK (0x3 << 21)
#define PLANE_CTL_KEY_ENABLE_SOURCE (1 << 21)
#define PLANE_CTL_KEY_ENABLE_DESTINATION (2 << 21)
#define _LGC_PALETTE_B 0x4a800
#define LGC_PALETTE(pipe, i) _MMIO(_PIPE(pipe, _LGC_PALETTE_A, _LGC_PALETTE_B) + (i) * 4)
+/* ilk/snb precision palette */
+#define _PREC_PALETTE_A 0x4b000
+#define _PREC_PALETTE_B 0x4c000
+#define PREC_PALETTE(pipe, i) _MMIO(_PIPE(pipe, _PREC_PALETTE_A, _PREC_PALETTE_B) + (i) * 4)
+
+#define _PREC_PIPEAGCMAX 0x4d000
+#define _PREC_PIPEBGCMAX 0x4d010
+#define PREC_PIPEGCMAX(pipe, i) _MMIO(_PIPE(pipe, _PIPEAGCMAX, _PIPEBGCMAX) + (i) * 4)
+
#define _GAMMA_MODE_A 0x4a480
#define _GAMMA_MODE_B 0x4ac80
#define GAMMA_MODE(pipe) _MMIO_PIPE(pipe, _GAMMA_MODE_A, _GAMMA_MODE_B)
-#define GAMMA_MODE_MODE_MASK (3 << 0)
-#define GAMMA_MODE_MODE_8BIT (0 << 0)
-#define GAMMA_MODE_MODE_10BIT (1 << 0)
-#define GAMMA_MODE_MODE_12BIT (2 << 0)
-#define GAMMA_MODE_MODE_SPLIT (3 << 0)
+#define PRE_CSC_GAMMA_ENABLE (1 << 31)
+#define POST_CSC_GAMMA_ENABLE (1 << 30)
+#define GAMMA_MODE_MODE_MASK (3 << 0)
+#define GAMMA_MODE_MODE_8BIT (0 << 0)
+#define GAMMA_MODE_MODE_10BIT (1 << 0)
+#define GAMMA_MODE_MODE_12BIT (2 << 0)
+#define GAMMA_MODE_MODE_SPLIT (3 << 0)
/* DMC/CSR */
#define CSR_PROGRAM(i) _MMIO(0x80000 + (i) * 4)
#define GEN8_GT_VECS_IRQ (1 << 6)
#define GEN8_GT_GUC_IRQ (1 << 5)
#define GEN8_GT_PM_IRQ (1 << 4)
-#define GEN8_GT_VCS2_IRQ (1 << 3)
-#define GEN8_GT_VCS1_IRQ (1 << 2)
+#define GEN8_GT_VCS1_IRQ (1 << 3) /* NB: VCS2 in bspec! */
+#define GEN8_GT_VCS0_IRQ (1 << 2) /* NB: VCS1 in bpsec! */
#define GEN8_GT_BCS_IRQ (1 << 1)
#define GEN8_GT_RCS_IRQ (1 << 0)
#define GEN8_RCS_IRQ_SHIFT 0
#define GEN8_BCS_IRQ_SHIFT 16
-#define GEN8_VCS1_IRQ_SHIFT 0
-#define GEN8_VCS2_IRQ_SHIFT 16
+#define GEN8_VCS0_IRQ_SHIFT 0 /* NB: VCS1 in bspec! */
+#define GEN8_VCS1_IRQ_SHIFT 16 /* NB: VCS2 in bpsec! */
#define GEN8_VECS_IRQ_SHIFT 0
#define GEN8_WD_IRQ_SHIFT 16
#define GEN9_LBS_SLA_RETRY_TIMER_DECREMENT_ENABLE (1 << 2)
/*GEN11 chicken */
-#define _PIPEA_CHICKEN 0x70038
-#define _PIPEB_CHICKEN 0x71038
-#define _PIPEC_CHICKEN 0x72038
-#define PER_PIXEL_ALPHA_BYPASS_EN (1 << 7)
-#define PM_FILL_MAINTAIN_DBUF_FULLNESS (1 << 0)
-#define PIPE_CHICKEN(pipe) _MMIO_PIPE(pipe, _PIPEA_CHICKEN,\
- _PIPEB_CHICKEN)
+#define _PIPEA_CHICKEN 0x70038
+#define _PIPEB_CHICKEN 0x71038
+#define _PIPEC_CHICKEN 0x72038
+#define PIPE_CHICKEN(pipe) _MMIO_PIPE(pipe, _PIPEA_CHICKEN,\
+ _PIPEB_CHICKEN)
+#define PIXEL_ROUNDING_TRUNC_FB_PASSTHRU (1 << 15)
+#define PER_PIXEL_ALPHA_BYPASS_EN (1 << 7)
/* PCH */
#define _ICL_VIDEO_DIP_PPS_ECC_B 0x613D4
#define HSW_TVIDEO_DIP_CTL(trans) _MMIO_TRANS2(trans, _HSW_VIDEO_DIP_CTL_A)
+#define HSW_TVIDEO_DIP_GCP(trans) _MMIO_TRANS2(trans, _HSW_VIDEO_DIP_GCP_A)
#define HSW_TVIDEO_DIP_AVI_DATA(trans, i) _MMIO_TRANS2(trans, _HSW_VIDEO_DIP_AVI_DATA_A + (i) * 4)
#define HSW_TVIDEO_DIP_VS_DATA(trans, i) _MMIO_TRANS2(trans, _HSW_VIDEO_DIP_VS_DATA_A + (i) * 4)
#define HSW_TVIDEO_DIP_SPD_DATA(trans, i) _MMIO_TRANS2(trans, _HSW_VIDEO_DIP_SPD_DATA_A + (i) * 4)
-#define HSW_TVIDEO_DIP_GCP(trans) _MMIO_TRANS2(trans, _HSW_VIDEO_DIP_GCP_A)
+#define HSW_TVIDEO_DIP_GMP_DATA(trans, i) _MMIO_TRANS2(trans, _HSW_VIDEO_DIP_GMP_DATA_A + (i) * 4)
#define HSW_TVIDEO_DIP_VSC_DATA(trans, i) _MMIO_TRANS2(trans, _HSW_VIDEO_DIP_VSC_DATA_A + (i) * 4)
#define ICL_VIDEO_DIP_PPS_DATA(trans, i) _MMIO_TRANS2(trans, _ICL_VIDEO_DIP_PPS_DATA_A + (i) * 4)
#define ICL_VIDEO_DIP_PPS_ECC(trans, i) _MMIO_TRANS2(trans, _ICL_VIDEO_DIP_PPS_ECC_A + (i) * 4)
#define GEN9_MEDIA_PG_IDLE_HYSTERESIS _MMIO(0xA0C4)
#define GEN9_RENDER_PG_IDLE_HYSTERESIS _MMIO(0xA0C8)
#define GEN9_PG_ENABLE _MMIO(0xA210)
-#define GEN9_RENDER_PG_ENABLE (1 << 0)
-#define GEN9_MEDIA_PG_ENABLE (1 << 1)
+#define GEN9_RENDER_PG_ENABLE REG_BIT(0)
+#define GEN9_MEDIA_PG_ENABLE REG_BIT(1)
+#define GEN11_MEDIA_SAMPLER_PG_ENABLE REG_BIT(2)
#define GEN8_PUSHBUS_CONTROL _MMIO(0xA248)
#define GEN8_PUSHBUS_ENABLE _MMIO(0xA250)
#define GEN8_PUSHBUS_SHIFT _MMIO(0xA25C)
#define GEN6_PMIER _MMIO(0x4402C)
#define GEN6_PM_MBOX_EVENT (1 << 25)
#define GEN6_PM_THERMAL_EVENT (1 << 24)
+
+/*
+ * For Gen11 these are in the upper word of the GPM_WGBOXPERF
+ * registers. Shifting is handled on accessing the imr and ier.
+ */
#define GEN6_PM_RP_DOWN_TIMEOUT (1 << 6)
#define GEN6_PM_RP_UP_THRESHOLD (1 << 5)
#define GEN6_PM_RP_DOWN_THRESHOLD (1 << 4)
#define DPLL_CFGCR1_KDIV(x) ((x) << 6)
#define DPLL_CFGCR1_KDIV_1 (1 << 6)
#define DPLL_CFGCR1_KDIV_2 (2 << 6)
-#define DPLL_CFGCR1_KDIV_4 (4 << 6)
+#define DPLL_CFGCR1_KDIV_3 (4 << 6)
#define DPLL_CFGCR1_PDIV_MASK (0xf << 2)
#define DPLL_CFGCR1_PDIV_SHIFT (2)
#define DPLL_CFGCR1_PDIV(x) ((x) << 2)
#define BXT_DRAM_WIDTH_X64 (0x3 << 4)
#define BXT_DRAM_SIZE_MASK (0x7 << 6)
#define BXT_DRAM_SIZE_SHIFT 6
-#define BXT_DRAM_SIZE_4GB (0x0 << 6)
-#define BXT_DRAM_SIZE_6GB (0x1 << 6)
-#define BXT_DRAM_SIZE_8GB (0x2 << 6)
-#define BXT_DRAM_SIZE_12GB (0x3 << 6)
-#define BXT_DRAM_SIZE_16GB (0x4 << 6)
+#define BXT_DRAM_SIZE_4GBIT (0x0 << 6)
+#define BXT_DRAM_SIZE_6GBIT (0x1 << 6)
+#define BXT_DRAM_SIZE_8GBIT (0x2 << 6)
+#define BXT_DRAM_SIZE_12GBIT (0x3 << 6)
+#define BXT_DRAM_SIZE_16GBIT (0x4 << 6)
+#define BXT_DRAM_TYPE_MASK (0x7 << 22)
+#define BXT_DRAM_TYPE_SHIFT 22
+#define BXT_DRAM_TYPE_DDR3 (0x0 << 22)
+#define BXT_DRAM_TYPE_LPDDR3 (0x1 << 22)
+#define BXT_DRAM_TYPE_LPDDR4 (0x2 << 22)
+#define BXT_DRAM_TYPE_DDR4 (0x4 << 22)
#define SKL_MEMORY_FREQ_MULTIPLIER_HZ 266666666
#define SKL_MC_BIOS_DATA_0_0_0_MCHBAR_PCU _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5E04)
#define SKL_REQ_DATA_MASK (0xF << 0)
+#define SKL_MAD_INTER_CHANNEL_0_0_0_MCHBAR_MCMAIN _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5000)
+#define SKL_DRAM_DDR_TYPE_MASK (0x3 << 0)
+#define SKL_DRAM_DDR_TYPE_DDR4 (0 << 0)
+#define SKL_DRAM_DDR_TYPE_DDR3 (1 << 0)
+#define SKL_DRAM_DDR_TYPE_LPDDR3 (2 << 0)
+#define SKL_DRAM_DDR_TYPE_LPDDR4 (3 << 0)
+
#define SKL_MAD_DIMM_CH0_0_0_0_MCHBAR_MCMAIN _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x500C)
#define SKL_MAD_DIMM_CH1_0_0_0_MCHBAR_MCMAIN _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5010)
#define SKL_DRAM_S_SHIFT 16
#define SKL_DRAM_WIDTH_X32 (0x2 << 8)
#define SKL_DRAM_RANK_MASK (0x1 << 10)
#define SKL_DRAM_RANK_SHIFT 10
-#define SKL_DRAM_RANK_SINGLE (0x0 << 10)
-#define SKL_DRAM_RANK_DUAL (0x1 << 10)
+#define SKL_DRAM_RANK_1 (0x0 << 10)
+#define SKL_DRAM_RANK_2 (0x1 << 10)
+#define SKL_DRAM_RANK_MASK (0x1 << 10)
+#define CNL_DRAM_SIZE_MASK 0x7F
+#define CNL_DRAM_WIDTH_MASK (0x3 << 7)
+#define CNL_DRAM_WIDTH_SHIFT 7
+#define CNL_DRAM_WIDTH_X8 (0x0 << 7)
+#define CNL_DRAM_WIDTH_X16 (0x1 << 7)
+#define CNL_DRAM_WIDTH_X32 (0x2 << 7)
+#define CNL_DRAM_RANK_MASK (0x3 << 9)
+#define CNL_DRAM_RANK_SHIFT 9
+#define CNL_DRAM_RANK_1 (0x0 << 9)
+#define CNL_DRAM_RANK_2 (0x1 << 9)
+#define CNL_DRAM_RANK_3 (0x2 << 9)
+#define CNL_DRAM_RANK_4 (0x3 << 9)
/* Please see hsw_read_dcomp() and hsw_write_dcomp() before using this register,
* since on HSW we can't write to it using I915_WRITE. */
#define _PIPE_A_CSC_COEFF_BU 0x4901c
#define _PIPE_A_CSC_COEFF_RV_GV 0x49020
#define _PIPE_A_CSC_COEFF_BV 0x49024
+
#define _PIPE_A_CSC_MODE 0x49028
-#define CSC_BLACK_SCREEN_OFFSET (1 << 2)
-#define CSC_POSITION_BEFORE_GAMMA (1 << 1)
-#define CSC_MODE_YUV_TO_RGB (1 << 0)
+#define ICL_CSC_ENABLE (1 << 31)
+#define ICL_OUTPUT_CSC_ENABLE (1 << 30)
+#define CSC_BLACK_SCREEN_OFFSET (1 << 2)
+#define CSC_POSITION_BEFORE_GAMMA (1 << 1)
+#define CSC_MODE_YUV_TO_RGB (1 << 0)
+
#define _PIPE_A_CSC_PREOFF_HI 0x49030
#define _PIPE_A_CSC_PREOFF_ME 0x49034
#define _PIPE_A_CSC_PREOFF_LO 0x49038
#define PIPE_CSC_POSTOFF_ME(pipe) _MMIO_PIPE(pipe, _PIPE_A_CSC_POSTOFF_ME, _PIPE_B_CSC_POSTOFF_ME)
#define PIPE_CSC_POSTOFF_LO(pipe) _MMIO_PIPE(pipe, _PIPE_A_CSC_POSTOFF_LO, _PIPE_B_CSC_POSTOFF_LO)
+/* Pipe Output CSC */
+#define _PIPE_A_OUTPUT_CSC_COEFF_RY_GY 0x49050
+#define _PIPE_A_OUTPUT_CSC_COEFF_BY 0x49054
+#define _PIPE_A_OUTPUT_CSC_COEFF_RU_GU 0x49058
+#define _PIPE_A_OUTPUT_CSC_COEFF_BU 0x4905c
+#define _PIPE_A_OUTPUT_CSC_COEFF_RV_GV 0x49060
+#define _PIPE_A_OUTPUT_CSC_COEFF_BV 0x49064
+#define _PIPE_A_OUTPUT_CSC_PREOFF_HI 0x49068
+#define _PIPE_A_OUTPUT_CSC_PREOFF_ME 0x4906c
+#define _PIPE_A_OUTPUT_CSC_PREOFF_LO 0x49070
+#define _PIPE_A_OUTPUT_CSC_POSTOFF_HI 0x49074
+#define _PIPE_A_OUTPUT_CSC_POSTOFF_ME 0x49078
+#define _PIPE_A_OUTPUT_CSC_POSTOFF_LO 0x4907c
+
+#define _PIPE_B_OUTPUT_CSC_COEFF_RY_GY 0x49150
+#define _PIPE_B_OUTPUT_CSC_COEFF_BY 0x49154
+#define _PIPE_B_OUTPUT_CSC_COEFF_RU_GU 0x49158
+#define _PIPE_B_OUTPUT_CSC_COEFF_BU 0x4915c
+#define _PIPE_B_OUTPUT_CSC_COEFF_RV_GV 0x49160
+#define _PIPE_B_OUTPUT_CSC_COEFF_BV 0x49164
+#define _PIPE_B_OUTPUT_CSC_PREOFF_HI 0x49168
+#define _PIPE_B_OUTPUT_CSC_PREOFF_ME 0x4916c
+#define _PIPE_B_OUTPUT_CSC_PREOFF_LO 0x49170
+#define _PIPE_B_OUTPUT_CSC_POSTOFF_HI 0x49174
+#define _PIPE_B_OUTPUT_CSC_POSTOFF_ME 0x49178
+#define _PIPE_B_OUTPUT_CSC_POSTOFF_LO 0x4917c
+
+#define PIPE_CSC_OUTPUT_COEFF_RY_GY(pipe) _MMIO_PIPE(pipe,\
+ _PIPE_A_OUTPUT_CSC_COEFF_RY_GY,\
+ _PIPE_B_OUTPUT_CSC_COEFF_RY_GY)
+#define PIPE_CSC_OUTPUT_COEFF_BY(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_COEFF_BY, \
+ _PIPE_B_OUTPUT_CSC_COEFF_BY)
+#define PIPE_CSC_OUTPUT_COEFF_RU_GU(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_COEFF_RU_GU, \
+ _PIPE_B_OUTPUT_CSC_COEFF_RU_GU)
+#define PIPE_CSC_OUTPUT_COEFF_BU(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_COEFF_BU, \
+ _PIPE_B_OUTPUT_CSC_COEFF_BU)
+#define PIPE_CSC_OUTPUT_COEFF_RV_GV(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_COEFF_RV_GV, \
+ _PIPE_B_OUTPUT_CSC_COEFF_RV_GV)
+#define PIPE_CSC_OUTPUT_COEFF_BV(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_COEFF_BV, \
+ _PIPE_B_OUTPUT_CSC_COEFF_BV)
+#define PIPE_CSC_OUTPUT_PREOFF_HI(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_PREOFF_HI, \
+ _PIPE_B_OUTPUT_CSC_PREOFF_HI)
+#define PIPE_CSC_OUTPUT_PREOFF_ME(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_PREOFF_ME, \
+ _PIPE_B_OUTPUT_CSC_PREOFF_ME)
+#define PIPE_CSC_OUTPUT_PREOFF_LO(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_PREOFF_LO, \
+ _PIPE_B_OUTPUT_CSC_PREOFF_LO)
+#define PIPE_CSC_OUTPUT_POSTOFF_HI(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_POSTOFF_HI, \
+ _PIPE_B_OUTPUT_CSC_POSTOFF_HI)
+#define PIPE_CSC_OUTPUT_POSTOFF_ME(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_POSTOFF_ME, \
+ _PIPE_B_OUTPUT_CSC_POSTOFF_ME)
+#define PIPE_CSC_OUTPUT_POSTOFF_LO(pipe) _MMIO_PIPE(pipe, \
+ _PIPE_A_OUTPUT_CSC_POSTOFF_LO, \
+ _PIPE_B_OUTPUT_CSC_POSTOFF_LO)
+
/* pipe degamma/gamma LUTs on IVB+ */
#define _PAL_PREC_INDEX_A 0x4A400
#define _PAL_PREC_INDEX_B 0x4AC00
#define PAL_PREC_SPLIT_MODE (1 << 31)
#define PAL_PREC_AUTO_INCREMENT (1 << 15)
#define PAL_PREC_INDEX_VALUE_MASK (0x3ff << 0)
+#define PAL_PREC_INDEX_VALUE(x) ((x) << 0)
#define _PAL_PREC_DATA_A 0x4A404
#define _PAL_PREC_DATA_B 0x4AC04
#define _PAL_PREC_DATA_C 0x4B404
#define PREC_PAL_DATA(pipe) _MMIO_PIPE(pipe, _PAL_PREC_DATA_A, _PAL_PREC_DATA_B)
#define PREC_PAL_GC_MAX(pipe, i) _MMIO(_PIPE(pipe, _PAL_PREC_GC_MAX_A, _PAL_PREC_GC_MAX_B) + (i) * 4)
#define PREC_PAL_EXT_GC_MAX(pipe, i) _MMIO(_PIPE(pipe, _PAL_PREC_EXT_GC_MAX_A, _PAL_PREC_EXT_GC_MAX_B) + (i) * 4)
+#define PREC_PAL_EXT2_GC_MAX(pipe, i) _MMIO(_PIPE(pipe, _PAL_PREC_EXT2_GC_MAX_A, _PAL_PREC_EXT2_GC_MAX_B) + (i) * 4)
#define _PRE_CSC_GAMC_INDEX_A 0x4A484
#define _PRE_CSC_GAMC_INDEX_B 0x4AC84
*
*/
-#include <linux/prefetch.h>
#include <linux/dma-fence-array.h>
+#include <linux/irq_work.h>
+#include <linux/prefetch.h>
#include <linux/sched.h>
#include <linux/sched/clock.h>
#include <linux/sched/signal.h>
-#include "i915_drv.h"
#include "i915_active.h"
+#include "i915_drv.h"
+#include "i915_globals.h"
#include "i915_reset.h"
+#include "intel_pm.h"
+
+struct execute_cb {
+ struct list_head link;
+ struct irq_work work;
+ struct i915_sw_fence *fence;
+};
+
+static struct i915_global_request {
+ struct i915_global base;
+ struct kmem_cache *slab_requests;
+ struct kmem_cache *slab_dependencies;
+ struct kmem_cache *slab_execute_cbs;
+} global;
static const char *i915_fence_get_driver_name(struct dma_fence *fence)
{
if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
return "signaled";
- return to_request(fence)->timeline->name;
+ return to_request(fence)->gem_context->name ?: "[i915]";
}
static bool i915_fence_signaled(struct dma_fence *fence)
bool interruptible,
signed long timeout)
{
- return i915_request_wait(to_request(fence), interruptible, timeout);
+ return i915_request_wait(to_request(fence),
+ interruptible | I915_WAIT_PRIORITY,
+ timeout);
}
static void i915_fence_release(struct dma_fence *fence)
* caught trying to reuse dead objects.
*/
i915_sw_fence_fini(&rq->submit);
+ i915_sw_fence_fini(&rq->semaphore);
- kmem_cache_free(rq->i915->requests, rq);
+ kmem_cache_free(global.slab_requests, rq);
}
const struct dma_fence_ops i915_fence_ops = {
* is just about to be. Either works, if we miss the last two
* noops - they are safe to be replayed on a reset.
*/
- GEM_TRACE("marking %s as inactive\n", ring->timeline->name);
tail = READ_ONCE(request->tail);
list_del(&ring->active_link);
} else {
static void __retire_engine_request(struct intel_engine_cs *engine,
struct i915_request *rq)
{
- GEM_TRACE("%s(%s) fence %llx:%lld, global=%d, current %d:%d\n",
+ GEM_TRACE("%s(%s) fence %llx:%lld, current %d\n",
__func__, engine->name,
rq->fence.context, rq->fence.seqno,
- rq->global_seqno,
- hwsp_seqno(rq),
- intel_engine_get_seqno(engine));
+ hwsp_seqno(rq));
GEM_BUG_ON(!i915_request_completed(rq));
{
struct i915_active_request *active, *next;
- GEM_TRACE("%s fence %llx:%lld, global=%d, current %d:%d\n",
+ GEM_TRACE("%s fence %llx:%lld, current %d\n",
request->engine->name,
request->fence.context, request->fence.seqno,
- request->global_seqno,
- hwsp_seqno(request),
- intel_engine_get_seqno(request->engine));
+ hwsp_seqno(request));
lockdep_assert_held(&request->i915->drm.struct_mutex);
GEM_BUG_ON(!i915_sw_fence_signaled(&request->submit));
i915_request_remove_from_client(request);
- /* Retirement decays the ban score as it is a sign of ctx progress */
- atomic_dec_if_positive(&request->gem_context->ban_score);
intel_context_unpin(request->hw_context);
__retire_engine_upto(request->engine, request);
unreserve_gt(request->i915);
- i915_sched_node_fini(request->i915, &request->sched);
+ i915_sched_node_fini(&request->sched);
i915_request_put(request);
}
struct intel_ring *ring = rq->ring;
struct i915_request *tmp;
- GEM_TRACE("%s fence %llx:%lld, global=%d, current %d:%d\n",
+ GEM_TRACE("%s fence %llx:%lld, current %d\n",
rq->engine->name,
rq->fence.context, rq->fence.seqno,
- rq->global_seqno,
- hwsp_seqno(rq),
- intel_engine_get_seqno(rq->engine));
+ hwsp_seqno(rq));
lockdep_assert_held(&rq->i915->drm.struct_mutex);
GEM_BUG_ON(!i915_request_completed(rq));
} while (tmp != rq);
}
-static u32 timeline_get_seqno(struct i915_timeline *tl)
+static void irq_execute_cb(struct irq_work *wrk)
+{
+ struct execute_cb *cb = container_of(wrk, typeof(*cb), work);
+
+ i915_sw_fence_complete(cb->fence);
+ kmem_cache_free(global.slab_execute_cbs, cb);
+}
+
+static void __notify_execute_cb(struct i915_request *rq)
+{
+ struct execute_cb *cb;
+
+ lockdep_assert_held(&rq->lock);
+
+ if (list_empty(&rq->execute_cb))
+ return;
+
+ list_for_each_entry(cb, &rq->execute_cb, link)
+ irq_work_queue(&cb->work);
+
+ /*
+ * XXX Rollback on __i915_request_unsubmit()
+ *
+ * In the future, perhaps when we have an active time-slicing scheduler,
+ * it will be interesting to unsubmit parallel execution and remove
+ * busywaits from the GPU until their master is restarted. This is
+ * quite hairy, we have to carefully rollback the fence and do a
+ * preempt-to-idle cycle on the target engine, all the while the
+ * master execute_cb may refire.
+ */
+ INIT_LIST_HEAD(&rq->execute_cb);
+}
+
+static int
+i915_request_await_execution(struct i915_request *rq,
+ struct i915_request *signal,
+ gfp_t gfp)
{
- return tl->seqno += 1 + tl->has_initial_breadcrumb;
+ struct execute_cb *cb;
+
+ if (i915_request_is_active(signal))
+ return 0;
+
+ cb = kmem_cache_alloc(global.slab_execute_cbs, gfp);
+ if (!cb)
+ return -ENOMEM;
+
+ cb->fence = &rq->submit;
+ i915_sw_fence_await(cb->fence);
+ init_irq_work(&cb->work, irq_execute_cb);
+
+ spin_lock_irq(&signal->lock);
+ if (i915_request_is_active(signal)) {
+ i915_sw_fence_complete(cb->fence);
+ kmem_cache_free(global.slab_execute_cbs, cb);
+ } else {
+ list_add_tail(&cb->link, &signal->execute_cb);
+ }
+ spin_unlock_irq(&signal->lock);
+
+ return 0;
}
static void move_to_timeline(struct i915_request *request,
spin_unlock(&request->timeline->lock);
}
-static u32 next_global_seqno(struct i915_timeline *tl)
-{
- if (!++tl->seqno)
- ++tl->seqno;
- return tl->seqno;
-}
-
void __i915_request_submit(struct i915_request *request)
{
struct intel_engine_cs *engine = request->engine;
- u32 seqno;
- GEM_TRACE("%s fence %llx:%lld -> global=%d, current %d:%d\n",
+ GEM_TRACE("%s fence %llx:%lld -> current %d\n",
engine->name,
request->fence.context, request->fence.seqno,
- engine->timeline.seqno + 1,
- hwsp_seqno(request),
- intel_engine_get_seqno(engine));
+ hwsp_seqno(request));
GEM_BUG_ON(!irqs_disabled());
lockdep_assert_held(&engine->timeline.lock);
- GEM_BUG_ON(request->global_seqno);
-
- seqno = next_global_seqno(&engine->timeline);
- GEM_BUG_ON(!seqno);
- GEM_BUG_ON(intel_engine_signaled(engine, seqno));
+ if (i915_gem_context_is_banned(request->gem_context))
+ i915_request_skip(request, -EIO);
/* We may be recursing from the signal callback of another i915 fence */
spin_lock_nested(&request->lock, SINGLE_DEPTH_NESTING);
+
GEM_BUG_ON(test_bit(I915_FENCE_FLAG_ACTIVE, &request->fence.flags));
set_bit(I915_FENCE_FLAG_ACTIVE, &request->fence.flags);
- request->global_seqno = seqno;
+
if (test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &request->fence.flags) &&
!i915_request_enable_breadcrumb(request))
intel_engine_queue_breadcrumbs(engine);
+
+ __notify_execute_cb(request);
+
spin_unlock(&request->lock);
engine->emit_fini_breadcrumb(request,
{
struct intel_engine_cs *engine = request->engine;
- GEM_TRACE("%s fence %llx:%lld <- global=%d, current %d:%d\n",
+ GEM_TRACE("%s fence %llx:%lld, current %d\n",
engine->name,
request->fence.context, request->fence.seqno,
- request->global_seqno,
- hwsp_seqno(request),
- intel_engine_get_seqno(engine));
+ hwsp_seqno(request));
GEM_BUG_ON(!irqs_disabled());
lockdep_assert_held(&engine->timeline.lock);
* Only unwind in reverse order, required so that the per-context list
* is kept in seqno/ring order.
*/
- GEM_BUG_ON(!request->global_seqno);
- GEM_BUG_ON(request->global_seqno != engine->timeline.seqno);
- GEM_BUG_ON(intel_engine_has_completed(engine, request->global_seqno));
- engine->timeline.seqno--;
/* We may be recursing from the signal callback of another i915 fence */
spin_lock_nested(&request->lock, SINGLE_DEPTH_NESTING);
- request->global_seqno = 0;
+
+ /*
+ * As we do not allow WAIT to preempt inflight requests,
+ * once we have executed a request, along with triggering
+ * any execution callbacks, we must preserve its ordering
+ * within the non-preemptible FIFO.
+ */
+ BUILD_BUG_ON(__NO_PREEMPTION & ~I915_PRIORITY_MASK); /* only internal */
+ request->sched.attr.priority |= __NO_PREEMPTION;
+
if (test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &request->fence.flags))
i915_request_cancel_breadcrumb(request);
+
GEM_BUG_ON(!test_bit(I915_FENCE_FLAG_ACTIVE, &request->fence.flags));
clear_bit(I915_FENCE_FLAG_ACTIVE, &request->fence.flags);
+
spin_unlock(&request->lock);
/* Transfer back from the global per-engine timeline to per-context */
return NOTIFY_DONE;
}
+static int __i915_sw_fence_call
+semaphore_notify(struct i915_sw_fence *fence, enum i915_sw_fence_notify state)
+{
+ struct i915_request *request =
+ container_of(fence, typeof(*request), semaphore);
+
+ switch (state) {
+ case FENCE_COMPLETE:
+ /*
+ * We only check a small portion of our dependencies
+ * and so cannot guarantee that there remains no
+ * semaphore chain across all. Instead of opting
+ * for the full NOSEMAPHORE boost, we go for the
+ * smaller (but still preempting) boost of
+ * NEWCLIENT. This will be enough to boost over
+ * a busywaiting request (as that cannot be
+ * NEWCLIENT) without accidentally boosting
+ * a busywait over real work elsewhere.
+ */
+ i915_schedule_bump_priority(request, I915_PRIORITY_NEWCLIENT);
+ break;
+
+ case FENCE_FREE:
+ i915_request_put(request);
+ break;
+ }
+
+ return NOTIFY_DONE;
+}
+
static void ring_retire_requests(struct intel_ring *ring)
{
struct i915_request *rq, *rn;
ring_retire_requests(ring);
out:
- return kmem_cache_alloc(ce->gem_context->i915->requests, GFP_KERNEL);
-}
-
-static int add_timeline_barrier(struct i915_request *rq)
-{
- return i915_request_await_active_request(rq, &rq->timeline->barrier);
+ return kmem_cache_alloc(global.slab_requests, GFP_KERNEL);
}
/**
i915_request_alloc(struct intel_engine_cs *engine, struct i915_gem_context *ctx)
{
struct drm_i915_private *i915 = engine->i915;
- struct i915_request *rq;
struct intel_context *ce;
+ struct i915_timeline *tl;
+ struct i915_request *rq;
+ u32 seqno;
int ret;
lockdep_assert_held(&i915->drm.struct_mutex);
* ABI: Before userspace accesses the GPU (e.g. execbuffer), report
* EIO if the GPU is already wedged.
*/
- if (i915_terminally_wedged(&i915->gpu_error))
- return ERR_PTR(-EIO);
+ ret = i915_terminally_wedged(i915);
+ if (ret)
+ return ERR_PTR(ret);
/*
* Pinning the contexts may generate requests in order to acquire
return ERR_CAST(ce);
reserve_gt(i915);
+ mutex_lock(&ce->ring->timeline->mutex);
/* Move our oldest request to the slab-cache (if not in use!) */
rq = list_first_entry(&ce->ring->request_list, typeof(*rq), ring_link);
*
* Do not use kmem_cache_zalloc() here!
*/
- rq = kmem_cache_alloc(i915->requests,
+ rq = kmem_cache_alloc(global.slab_requests,
GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN);
if (unlikely(!rq)) {
rq = i915_request_alloc_slow(ce);
}
}
- rq->rcustate = get_state_synchronize_rcu();
-
INIT_LIST_HEAD(&rq->active_list);
+ INIT_LIST_HEAD(&rq->execute_cb);
+
+ tl = ce->ring->timeline;
+ ret = i915_timeline_get_seqno(tl, rq, &seqno);
+ if (ret)
+ goto err_free;
+
rq->i915 = i915;
rq->engine = engine;
rq->gem_context = ctx;
rq->hw_context = ce;
rq->ring = ce->ring;
- rq->timeline = ce->ring->timeline;
+ rq->timeline = tl;
GEM_BUG_ON(rq->timeline == &engine->timeline);
- rq->hwsp_seqno = rq->timeline->hwsp_seqno;
+ rq->hwsp_seqno = tl->hwsp_seqno;
+ rq->hwsp_cacheline = tl->hwsp_cacheline;
+ rq->rcustate = get_state_synchronize_rcu(); /* acts as smp_mb() */
spin_lock_init(&rq->lock);
- dma_fence_init(&rq->fence,
- &i915_fence_ops,
- &rq->lock,
- rq->timeline->fence_context,
- timeline_get_seqno(rq->timeline));
+ dma_fence_init(&rq->fence, &i915_fence_ops, &rq->lock,
+ tl->fence_context, seqno);
/* We bump the ref for the fence chain */
i915_sw_fence_init(&i915_request_get(rq)->submit, submit_notify);
+ i915_sw_fence_init(&i915_request_get(rq)->semaphore, semaphore_notify);
i915_sched_node_init(&rq->sched);
/* No zalloc, must clear what we need by hand */
- rq->global_seqno = 0;
rq->file_priv = NULL;
rq->batch = NULL;
rq->capture_list = NULL;
*/
rq->head = rq->ring->emit;
- ret = add_timeline_barrier(rq);
- if (ret)
- goto err_unwind;
-
ret = engine->request_alloc(rq);
if (ret)
goto err_unwind;
rq->infix = rq->ring->emit; /* end of header; start of user payload */
/* Check that we didn't interrupt ourselves with a new request */
+ lockdep_assert_held(&rq->timeline->mutex);
GEM_BUG_ON(rq->timeline->seqno != rq->fence.seqno);
+ rq->cookie = lockdep_pin_lock(&rq->timeline->mutex);
+
return rq;
err_unwind:
GEM_BUG_ON(!list_empty(&rq->sched.signalers_list));
GEM_BUG_ON(!list_empty(&rq->sched.waiters_list));
- kmem_cache_free(i915->requests, rq);
+err_free:
+ kmem_cache_free(global.slab_requests, rq);
err_unreserve:
+ mutex_unlock(&ce->ring->timeline->mutex);
unreserve_gt(i915);
intel_context_unpin(ce);
return ERR_PTR(ret);
}
+static int
+emit_semaphore_wait(struct i915_request *to,
+ struct i915_request *from,
+ gfp_t gfp)
+{
+ u32 hwsp_offset;
+ u32 *cs;
+ int err;
+
+ GEM_BUG_ON(!from->timeline->has_initial_breadcrumb);
+ GEM_BUG_ON(INTEL_GEN(to->i915) < 8);
+
+ /* Just emit the first semaphore we see as request space is limited. */
+ if (to->sched.semaphores & from->engine->mask)
+ return i915_sw_fence_await_dma_fence(&to->submit,
+ &from->fence, 0,
+ I915_FENCE_GFP);
+
+ err = i915_sw_fence_await_dma_fence(&to->semaphore,
+ &from->fence, 0,
+ I915_FENCE_GFP);
+ if (err < 0)
+ return err;
+
+ /* We need to pin the signaler's HWSP until we are finished reading. */
+ err = i915_timeline_read_hwsp(from, to, &hwsp_offset);
+ if (err)
+ return err;
+
+ /* Only submit our spinner after the signaler is running! */
+ err = i915_request_await_execution(to, from, gfp);
+ if (err)
+ return err;
+
+ cs = intel_ring_begin(to, 4);
+ if (IS_ERR(cs))
+ return PTR_ERR(cs);
+
+ /*
+ * Using greater-than-or-equal here means we have to worry
+ * about seqno wraparound. To side step that issue, we swap
+ * the timeline HWSP upon wrapping, so that everyone listening
+ * for the old (pre-wrap) values do not see the much smaller
+ * (post-wrap) values than they were expecting (and so wait
+ * forever).
+ */
+ *cs++ = MI_SEMAPHORE_WAIT |
+ MI_SEMAPHORE_GLOBAL_GTT |
+ MI_SEMAPHORE_POLL |
+ MI_SEMAPHORE_SAD_GTE_SDD;
+ *cs++ = from->fence.seqno;
+ *cs++ = hwsp_offset;
+ *cs++ = 0;
+
+ intel_ring_advance(to, cs);
+ to->sched.semaphores |= from->engine->mask;
+ to->sched.flags |= I915_SCHED_HAS_SEMAPHORE_CHAIN;
+ return 0;
+}
+
static int
i915_request_await_request(struct i915_request *to, struct i915_request *from)
{
return 0;
if (to->engine->schedule) {
- ret = i915_sched_node_add_dependency(to->i915,
- &to->sched,
- &from->sched);
+ ret = i915_sched_node_add_dependency(&to->sched, &from->sched);
if (ret < 0)
return ret;
}
ret = i915_sw_fence_await_sw_fence_gfp(&to->submit,
&from->submit,
I915_FENCE_GFP);
+ } else if (intel_engine_has_semaphores(to->engine) &&
+ to->gem_context->sched.priority >= I915_PRIORITY_NORMAL) {
+ ret = emit_semaphore_wait(to, from, I915_FENCE_GFP);
} else {
ret = i915_sw_fence_await_dma_fence(&to->submit,
&from->fence, 0,
memset(vaddr + head, 0, rq->postfix - head);
}
+static struct i915_request *
+__i915_request_add_to_timeline(struct i915_request *rq)
+{
+ struct i915_timeline *timeline = rq->timeline;
+ struct i915_request *prev;
+
+ /*
+ * Dependency tracking and request ordering along the timeline
+ * is special cased so that we can eliminate redundant ordering
+ * operations while building the request (we know that the timeline
+ * itself is ordered, and here we guarantee it).
+ *
+ * As we know we will need to emit tracking along the timeline,
+ * we embed the hooks into our request struct -- at the cost of
+ * having to have specialised no-allocation interfaces (which will
+ * be beneficial elsewhere).
+ *
+ * A second benefit to open-coding i915_request_await_request is
+ * that we can apply a slight variant of the rules specialised
+ * for timelines that jump between engines (such as virtual engines).
+ * If we consider the case of virtual engine, we must emit a dma-fence
+ * to prevent scheduling of the second request until the first is
+ * complete (to maximise our greedy late load balancing) and this
+ * precludes optimising to use semaphores serialisation of a single
+ * timeline across engines.
+ */
+ prev = i915_active_request_raw(&timeline->last_request,
+ &rq->i915->drm.struct_mutex);
+ if (prev && !i915_request_completed(prev)) {
+ if (is_power_of_2(prev->engine->mask | rq->engine->mask))
+ i915_sw_fence_await_sw_fence(&rq->submit,
+ &prev->submit,
+ &rq->submitq);
+ else
+ __i915_sw_fence_await_dma_fence(&rq->submit,
+ &prev->fence,
+ &rq->dmaq);
+ if (rq->engine->schedule)
+ __i915_sched_node_add_dependency(&rq->sched,
+ &prev->sched,
+ &rq->dep,
+ 0);
+ }
+
+ spin_lock_irq(&timeline->lock);
+ list_add_tail(&rq->link, &timeline->requests);
+ spin_unlock_irq(&timeline->lock);
+
+ GEM_BUG_ON(timeline->seqno != rq->fence.seqno);
+ __i915_active_request_set(&timeline->last_request, rq);
+
+ return prev;
+}
+
/*
* NB: This function is not allowed to fail. Doing so would mean the the
* request is not being tracked for completion but the work itself is
GEM_TRACE("%s fence %llx:%lld\n",
engine->name, request->fence.context, request->fence.seqno);
- lockdep_assert_held(&request->i915->drm.struct_mutex);
+ lockdep_assert_held(&request->timeline->mutex);
+ lockdep_unpin_lock(&request->timeline->mutex, request->cookie);
+
trace_i915_request_add(request);
/*
GEM_BUG_ON(IS_ERR(cs));
request->postfix = intel_ring_offset(request, cs);
- /*
- * Seal the request and mark it as pending execution. Note that
- * we may inspect this state, without holding any locks, during
- * hangcheck. Hence we apply the barrier to ensure that we do not
- * see a more recent value in the hws than we are tracking.
- */
-
- prev = i915_active_request_raw(&timeline->last_request,
- &request->i915->drm.struct_mutex);
- if (prev && !i915_request_completed(prev)) {
- i915_sw_fence_await_sw_fence(&request->submit, &prev->submit,
- &request->submitq);
- if (engine->schedule)
- __i915_sched_node_add_dependency(&request->sched,
- &prev->sched,
- &request->dep,
- 0);
- }
-
- spin_lock_irq(&timeline->lock);
- list_add_tail(&request->link, &timeline->requests);
- spin_unlock_irq(&timeline->lock);
-
- GEM_BUG_ON(timeline->seqno != request->fence.seqno);
- __i915_active_request_set(&timeline->last_request, request);
+ prev = __i915_request_add_to_timeline(request);
list_add_tail(&request->ring_link, &ring->request_list);
- if (list_is_first(&request->ring_link, &ring->request_list)) {
- GEM_TRACE("marking %s as active\n", ring->timeline->name);
+ if (list_is_first(&request->ring_link, &ring->request_list))
list_add(&ring->active_link, &request->i915->gt.active_rings);
- }
+ request->i915->gt.active_engines |= request->engine->mask;
request->emitted_jiffies = jiffies;
/*
* run at the earliest possible convenience.
*/
local_bh_disable();
+ i915_sw_fence_commit(&request->semaphore);
rcu_read_lock(); /* RCU serialisation for set-wedged protection */
if (engine->schedule) {
struct i915_sched_attr attr = request->gem_context->sched;
+ /*
+ * Boost actual workloads past semaphores!
+ *
+ * With semaphores we spin on one engine waiting for another,
+ * simply to reduce the latency of starting our work when
+ * the signaler completes. However, if there is any other
+ * work that we could be doing on this engine instead, that
+ * is better utilisation and will reduce the overall duration
+ * of the current work. To avoid PI boosting a semaphore
+ * far in the distance past over useful work, we keep a history
+ * of any semaphore use along our dependency chain.
+ */
+ if (!(request->sched.flags & I915_SCHED_HAS_SEMAPHORE_CHAIN))
+ attr.priority |= I915_PRIORITY_NOSEMAPHORE;
+
/*
* Boost priorities to new clients (new request flows).
*
*/
if (prev && i915_request_completed(prev))
i915_request_retire_upto(prev);
+
+ mutex_unlock(&request->timeline->mutex);
}
static unsigned long local_clock_us(unsigned int *cpu)
if (__i915_spin_request(rq, state, 5))
goto out;
- if (flags & I915_WAIT_PRIORITY)
+ /*
+ * This client is about to stall waiting for the GPU. In many cases
+ * this is undesirable and limits the throughput of the system, as
+ * many clients cannot continue processing user input/output whilst
+ * blocked. RPS autotuning may take tens of milliseconds to respond
+ * to the GPU load and thus incurs additional latency for the client.
+ * We can circumvent that by promoting the GPU frequency to maximum
+ * before we sleep. This makes the GPU throttle up much more quickly
+ * (good for benchmarks and user experience, e.g. window animations),
+ * but at a cost of spending more power processing the workload
+ * (bad for battery).
+ */
+ if (flags & I915_WAIT_PRIORITY) {
+ if (!i915_request_started(rq) && INTEL_GEN(rq->i915) >= 6)
+ gen6_rps_boost(rq);
+ local_bh_disable(); /* suspend tasklets for reprioritisation */
i915_schedule_bump_priority(rq, I915_PRIORITY_WAIT);
+ local_bh_enable(); /* kick tasklets en masse */
+ }
wait.tsk = current;
if (dma_fence_add_callback(&rq->fence, &wait.cb, request_wait_wake))
if (!i915->gt.active_requests)
return;
- list_for_each_entry_safe(ring, tmp, &i915->gt.active_rings, active_link)
+ list_for_each_entry_safe(ring, tmp,
+ &i915->gt.active_rings, active_link) {
+ intel_ring_get(ring); /* last rq holds reference! */
ring_retire_requests(ring);
+ intel_ring_put(ring);
+ }
}
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftests/mock_request.c"
#include "selftests/i915_request.c"
#endif
+
+static void i915_global_request_shrink(void)
+{
+ kmem_cache_shrink(global.slab_dependencies);
+ kmem_cache_shrink(global.slab_execute_cbs);
+ kmem_cache_shrink(global.slab_requests);
+}
+
+static void i915_global_request_exit(void)
+{
+ kmem_cache_destroy(global.slab_dependencies);
+ kmem_cache_destroy(global.slab_execute_cbs);
+ kmem_cache_destroy(global.slab_requests);
+}
+
+static struct i915_global_request global = { {
+ .shrink = i915_global_request_shrink,
+ .exit = i915_global_request_exit,
+} };
+
+int __init i915_global_request_init(void)
+{
+ global.slab_requests = KMEM_CACHE(i915_request,
+ SLAB_HWCACHE_ALIGN |
+ SLAB_RECLAIM_ACCOUNT |
+ SLAB_TYPESAFE_BY_RCU);
+ if (!global.slab_requests)
+ return -ENOMEM;
+
+ global.slab_execute_cbs = KMEM_CACHE(execute_cb,
+ SLAB_HWCACHE_ALIGN |
+ SLAB_RECLAIM_ACCOUNT |
+ SLAB_TYPESAFE_BY_RCU);
+ if (!global.slab_execute_cbs)
+ goto err_requests;
+
+ global.slab_dependencies = KMEM_CACHE(i915_dependency,
+ SLAB_HWCACHE_ALIGN |
+ SLAB_RECLAIM_ACCOUNT);
+ if (!global.slab_dependencies)
+ goto err_execute_cbs;
+
+ i915_global_register(&global.base);
+ return 0;
+
+err_execute_cbs:
+ kmem_cache_destroy(global.slab_execute_cbs);
+err_requests:
+ kmem_cache_destroy(global.slab_requests);
+ return -ENOMEM;
+}
#define I915_REQUEST_H
#include <linux/dma-fence.h>
+#include <linux/lockdep.h>
#include "i915_gem.h"
#include "i915_scheduler.h"
+#include "i915_selftest.h"
#include "i915_sw_fence.h"
#include <uapi/drm/i915_drm.h>
struct drm_i915_gem_object;
struct i915_request;
struct i915_timeline;
+struct i915_timeline_cacheline;
struct i915_capture_list {
struct i915_capture_list *next;
*/
unsigned long rcustate;
+ /*
+ * We pin the timeline->mutex while constructing the request to
+ * ensure that no caller accidentally drops it during construction.
+ * The timeline->mutex must be held to ensure that only this caller
+ * can use the ring and manipulate the associated timeline during
+ * construction.
+ */
+ struct pin_cookie cookie;
+
/*
* Fences for the various phases in the request's lifetime.
*
* It is used by the driver to then queue the request for execution.
*/
struct i915_sw_fence submit;
- wait_queue_entry_t submitq;
+ union {
+ wait_queue_entry_t submitq;
+ struct i915_sw_dma_fence_cb dmaq;
+ };
+ struct list_head execute_cb;
+ struct i915_sw_fence semaphore;
/*
* A list of everyone we wait upon, and everyone who waits upon us.
*/
const u32 *hwsp_seqno;
- /**
- * GEM sequence number associated with this request on the
- * global execution timeline. It is zero when the request is not
- * on the HW queue (i.e. not on the engine timeline list).
- * Its value is guarded by the timeline spinlock.
+ /*
+ * If we need to access the timeline's seqno for this request in
+ * another request, we need to keep a read reference to this associated
+ * cacheline, so that we do not free and recycle it before the foreign
+ * observers have completed. Hence, we keep a pointer to the cacheline
+ * inside the timeline's HWSP vma, but it is only valid while this
+ * request has not completed and guarded by the timeline mutex.
*/
- u32 global_seqno;
+ struct i915_timeline_cacheline *hwsp_cacheline;
/** Position in the ring of the start of the request */
u32 head;
struct drm_i915_file_private *file_priv;
/** file_priv list entry for this request */
struct list_head client_link;
+
+ I915_SELFTEST_DECLARE(struct {
+ struct list_head link;
+ unsigned long delay;
+ } mock;)
};
#define I915_FENCE_GFP (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN)
dma_fence_put(&rq->fence);
}
-/**
- * i915_request_global_seqno - report the current global seqno
- * @request - the request
- *
- * A request is assigned a global seqno only when it is on the hardware
- * execution queue. The global seqno can be used to maintain a list of
- * requests on the same engine in retirement order, for example for
- * constructing a priority queue for waiting. Prior to its execution, or
- * if it is subsequently removed in the event of preemption, its global
- * seqno is zero. As both insertion and removal from the execution queue
- * may operate in IRQ context, it is not guarded by the usual struct_mutex
- * BKL. Instead those relying on the global seqno must be prepared for its
- * value to change between reads. Only when the request is complete can
- * the global seqno be stable (due to the memory barriers on submitting
- * the commands to the hardware to write the breadcrumb, if the HWS shows
- * that it has passed the global seqno and the global seqno is unchanged
- * after the read, it is indeed complete).
- */
-static inline u32
-i915_request_global_seqno(const struct i915_request *request)
-{
- return READ_ONCE(request->global_seqno);
-}
-
int i915_request_await_object(struct i915_request *to,
struct drm_i915_gem_object *obj,
bool write);
* i915_request_started - check if the request has begun being executed
* @rq: the request
*
- * Returns true if the request has been submitted to hardware, and the hardware
- * has advanced passed the end of the previous request and so should be either
- * currently processing the request (though it may be preempted and so
- * not necessarily the next request to complete) or have completed the request.
+ * If the timeline is not using initial breadcrumbs, a request is
+ * considered started if the previous request on its timeline (i.e.
+ * context) has been signaled.
+ *
+ * If the timeline is using semaphores, it will also be emitting an
+ * "initial breadcrumb" after the semaphores are complete and just before
+ * it began executing the user payload. A request can therefore be active
+ * on the HW and not yet started as it is still busywaiting on its
+ * dependencies (via HW semaphores).
+ *
+ * If the request has started, its dependencies will have been signaled
+ * (either by fences or by semaphores) and it will have begun processing
+ * the user payload.
+ *
+ * However, even if a request has started, it may have been preempted and
+ * so no longer active, or it may have already completed.
+ *
+ * See also i915_request_is_active().
+ *
+ * Returns true if the request has begun executing the user payload, or
+ * has completed:
*/
static inline bool i915_request_started(const struct i915_request *rq)
{
/* XXX How to handle concurrent GGTT updates using tiling registers? */
#define RESET_UNDER_STOP_MACHINE 0
+static void rmw_set(struct intel_uncore *uncore, i915_reg_t reg, u32 set)
+{
+ intel_uncore_rmw(uncore, reg, 0, set);
+}
+
+static void rmw_clear(struct intel_uncore *uncore, i915_reg_t reg, u32 clr)
+{
+ intel_uncore_rmw(uncore, reg, clr, 0);
+}
+
+static void rmw_set_fw(struct intel_uncore *uncore, i915_reg_t reg, u32 set)
+{
+ intel_uncore_rmw_fw(uncore, reg, 0, set);
+}
+
+static void rmw_clear_fw(struct intel_uncore *uncore, i915_reg_t reg, u32 clr)
+{
+ intel_uncore_rmw_fw(uncore, reg, clr, 0);
+}
+
static void engine_skip_context(struct i915_request *rq)
{
struct intel_engine_cs *engine = rq->engine;
struct i915_gem_context *hung_ctx = rq->gem_context;
- struct i915_timeline *timeline = rq->timeline;
lockdep_assert_held(&engine->timeline.lock);
- GEM_BUG_ON(timeline == &engine->timeline);
-
- spin_lock(&timeline->lock);
-
- if (i915_request_is_active(rq)) {
- list_for_each_entry_continue(rq,
- &engine->timeline.requests, link)
- if (rq->gem_context == hung_ctx)
- i915_request_skip(rq, -EIO);
- }
- list_for_each_entry(rq, &timeline->requests, link)
- i915_request_skip(rq, -EIO);
+ if (!i915_request_is_active(rq))
+ return;
- spin_unlock(&timeline->lock);
+ list_for_each_entry_continue(rq, &engine->timeline.requests, link)
+ if (rq->gem_context == hung_ctx)
+ i915_request_skip(rq, -EIO);
}
static void client_mark_guilty(struct drm_i915_file_private *file_priv,
static bool context_mark_guilty(struct i915_gem_context *ctx)
{
- unsigned int score;
- bool banned, bannable;
+ unsigned long prev_hang;
+ bool banned;
+ int i;
atomic_inc(&ctx->guilty_count);
- bannable = i915_gem_context_is_bannable(ctx);
- score = atomic_add_return(CONTEXT_SCORE_GUILTY, &ctx->ban_score);
- banned = score >= CONTEXT_SCORE_BAN_THRESHOLD;
-
- /* Cool contexts don't accumulate client ban score */
- if (!bannable)
+ /* Cool contexts are too cool to be banned! (Used for reset testing.) */
+ if (!i915_gem_context_is_bannable(ctx))
return false;
+ /* Record the timestamp for the last N hangs */
+ prev_hang = ctx->hang_timestamp[0];
+ for (i = 0; i < ARRAY_SIZE(ctx->hang_timestamp) - 1; i++)
+ ctx->hang_timestamp[i] = ctx->hang_timestamp[i + 1];
+ ctx->hang_timestamp[i] = jiffies;
+
+ /* If we have hung N+1 times in rapid succession, we ban the context! */
+ banned = !i915_gem_context_is_recoverable(ctx);
+ if (time_before(jiffies, prev_hang + CONTEXT_FAST_HANG_JIFFIES))
+ banned = true;
if (banned) {
- DRM_DEBUG_DRIVER("context %s: guilty %d, score %u, banned\n",
- ctx->name, atomic_read(&ctx->guilty_count),
- score);
+ DRM_DEBUG_DRIVER("context %s: guilty %d, banned\n",
+ ctx->name, atomic_read(&ctx->guilty_count));
i915_gem_context_set_banned(ctx);
}
void i915_reset_request(struct i915_request *rq, bool guilty)
{
+ GEM_TRACE("%s rq=%llx:%lld, guilty? %s\n",
+ rq->engine->name,
+ rq->fence.context,
+ rq->fence.seqno,
+ yesno(guilty));
+
lockdep_assert_held(&rq->engine->timeline.lock);
GEM_BUG_ON(i915_request_completed(rq));
static void gen3_stop_engine(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
+ struct intel_uncore *uncore = engine->uncore;
const u32 base = engine->mmio_base;
+ GEM_TRACE("%s\n", engine->name);
+
if (intel_engine_stop_cs(engine))
- DRM_DEBUG_DRIVER("%s: timed out on STOP_RING\n", engine->name);
+ GEM_TRACE("%s: timed out on STOP_RING\n", engine->name);
- I915_WRITE_FW(RING_HEAD(base), I915_READ_FW(RING_TAIL(base)));
- POSTING_READ_FW(RING_HEAD(base)); /* paranoia */
+ intel_uncore_write_fw(uncore,
+ RING_HEAD(base),
+ intel_uncore_read_fw(uncore, RING_TAIL(base)));
+ intel_uncore_posting_read_fw(uncore, RING_HEAD(base)); /* paranoia */
- I915_WRITE_FW(RING_HEAD(base), 0);
- I915_WRITE_FW(RING_TAIL(base), 0);
- POSTING_READ_FW(RING_TAIL(base));
+ intel_uncore_write_fw(uncore, RING_HEAD(base), 0);
+ intel_uncore_write_fw(uncore, RING_TAIL(base), 0);
+ intel_uncore_posting_read_fw(uncore, RING_TAIL(base));
/* The ring must be empty before it is disabled */
- I915_WRITE_FW(RING_CTL(base), 0);
+ intel_uncore_write_fw(uncore, RING_CTL(base), 0);
/* Check acts as a post */
- if (I915_READ_FW(RING_HEAD(base)) != 0)
- DRM_DEBUG_DRIVER("%s: ring head not parked\n",
- engine->name);
+ if (intel_uncore_read_fw(uncore, RING_HEAD(base)))
+ GEM_TRACE("%s: ring head [%x] not parked\n",
+ engine->name,
+ intel_uncore_read_fw(uncore, RING_HEAD(base)));
}
static void i915_stop_engines(struct drm_i915_private *i915,
- unsigned int engine_mask)
+ intel_engine_mask_t engine_mask)
{
struct intel_engine_cs *engine;
- enum intel_engine_id id;
+ intel_engine_mask_t tmp;
if (INTEL_GEN(i915) < 3)
return;
- for_each_engine_masked(engine, i915, engine_mask, id)
+ for_each_engine_masked(engine, i915, engine_mask, tmp)
gen3_stop_engine(engine);
}
}
static int i915_do_reset(struct drm_i915_private *i915,
- unsigned int engine_mask,
+ intel_engine_mask_t engine_mask,
unsigned int retry)
{
struct pci_dev *pdev = i915->drm.pdev;
}
static int g33_do_reset(struct drm_i915_private *i915,
- unsigned int engine_mask,
+ intel_engine_mask_t engine_mask,
unsigned int retry)
{
struct pci_dev *pdev = i915->drm.pdev;
return wait_for_atomic(g4x_reset_complete(pdev), 50);
}
-static int g4x_do_reset(struct drm_i915_private *dev_priv,
- unsigned int engine_mask,
+static int g4x_do_reset(struct drm_i915_private *i915,
+ intel_engine_mask_t engine_mask,
unsigned int retry)
{
- struct pci_dev *pdev = dev_priv->drm.pdev;
+ struct pci_dev *pdev = i915->drm.pdev;
+ struct intel_uncore *uncore = &i915->uncore;
int ret;
/* WaVcpClkGateDisableForMediaReset:ctg,elk */
- I915_WRITE_FW(VDECCLK_GATE_D,
- I915_READ(VDECCLK_GATE_D) | VCP_UNIT_CLOCK_GATE_DISABLE);
- POSTING_READ_FW(VDECCLK_GATE_D);
+ rmw_set_fw(uncore, VDECCLK_GATE_D, VCP_UNIT_CLOCK_GATE_DISABLE);
+ intel_uncore_posting_read_fw(uncore, VDECCLK_GATE_D);
pci_write_config_byte(pdev, I915_GDRST,
GRDOM_MEDIA | GRDOM_RESET_ENABLE);
out:
pci_write_config_byte(pdev, I915_GDRST, 0);
- I915_WRITE_FW(VDECCLK_GATE_D,
- I915_READ(VDECCLK_GATE_D) & ~VCP_UNIT_CLOCK_GATE_DISABLE);
- POSTING_READ_FW(VDECCLK_GATE_D);
+ rmw_clear_fw(uncore, VDECCLK_GATE_D, VCP_UNIT_CLOCK_GATE_DISABLE);
+ intel_uncore_posting_read_fw(uncore, VDECCLK_GATE_D);
return ret;
}
-static int ironlake_do_reset(struct drm_i915_private *dev_priv,
- unsigned int engine_mask,
+static int ironlake_do_reset(struct drm_i915_private *i915,
+ intel_engine_mask_t engine_mask,
unsigned int retry)
{
+ struct intel_uncore *uncore = &i915->uncore;
int ret;
- I915_WRITE_FW(ILK_GDSR, ILK_GRDOM_RENDER | ILK_GRDOM_RESET_ENABLE);
- ret = __intel_wait_for_register_fw(dev_priv, ILK_GDSR,
+ intel_uncore_write_fw(uncore, ILK_GDSR,
+ ILK_GRDOM_RENDER | ILK_GRDOM_RESET_ENABLE);
+ ret = __intel_wait_for_register_fw(uncore, ILK_GDSR,
ILK_GRDOM_RESET_ENABLE, 0,
5000, 0,
NULL);
goto out;
}
- I915_WRITE_FW(ILK_GDSR, ILK_GRDOM_MEDIA | ILK_GRDOM_RESET_ENABLE);
- ret = __intel_wait_for_register_fw(dev_priv, ILK_GDSR,
+ intel_uncore_write_fw(uncore, ILK_GDSR,
+ ILK_GRDOM_MEDIA | ILK_GRDOM_RESET_ENABLE);
+ ret = __intel_wait_for_register_fw(uncore, ILK_GDSR,
ILK_GRDOM_RESET_ENABLE, 0,
5000, 0,
NULL);
}
out:
- I915_WRITE_FW(ILK_GDSR, 0);
- POSTING_READ_FW(ILK_GDSR);
+ intel_uncore_write_fw(uncore, ILK_GDSR, 0);
+ intel_uncore_posting_read_fw(uncore, ILK_GDSR);
return ret;
}
/* Reset the hardware domains (GENX_GRDOM_*) specified by mask */
-static int gen6_hw_domain_reset(struct drm_i915_private *dev_priv,
+static int gen6_hw_domain_reset(struct drm_i915_private *i915,
u32 hw_domain_mask)
{
+ struct intel_uncore *uncore = &i915->uncore;
int err;
/*
* for fifo space for the write or forcewake the chip for
* the read
*/
- I915_WRITE_FW(GEN6_GDRST, hw_domain_mask);
+ intel_uncore_write_fw(uncore, GEN6_GDRST, hw_domain_mask);
/* Wait for the device to ack the reset requests */
- err = __intel_wait_for_register_fw(dev_priv,
+ err = __intel_wait_for_register_fw(uncore,
GEN6_GDRST, hw_domain_mask, 0,
500, 0,
NULL);
}
static int gen6_reset_engines(struct drm_i915_private *i915,
- unsigned int engine_mask,
+ intel_engine_mask_t engine_mask,
unsigned int retry)
{
struct intel_engine_cs *engine;
- const u32 hw_engine_mask[I915_NUM_ENGINES] = {
- [RCS] = GEN6_GRDOM_RENDER,
- [BCS] = GEN6_GRDOM_BLT,
- [VCS] = GEN6_GRDOM_MEDIA,
- [VCS2] = GEN8_GRDOM_MEDIA2,
- [VECS] = GEN6_GRDOM_VECS,
+ const u32 hw_engine_mask[] = {
+ [RCS0] = GEN6_GRDOM_RENDER,
+ [BCS0] = GEN6_GRDOM_BLT,
+ [VCS0] = GEN6_GRDOM_MEDIA,
+ [VCS1] = GEN8_GRDOM_MEDIA2,
+ [VECS0] = GEN6_GRDOM_VECS,
};
u32 hw_mask;
if (engine_mask == ALL_ENGINES) {
hw_mask = GEN6_GRDOM_FULL;
} else {
- unsigned int tmp;
+ intel_engine_mask_t tmp;
hw_mask = 0;
- for_each_engine_masked(engine, i915, engine_mask, tmp)
+ for_each_engine_masked(engine, i915, engine_mask, tmp) {
+ GEM_BUG_ON(engine->id >= ARRAY_SIZE(hw_engine_mask));
hw_mask |= hw_engine_mask[engine->id];
+ }
}
return gen6_hw_domain_reset(i915, hw_mask);
}
-static u32 gen11_lock_sfc(struct drm_i915_private *dev_priv,
- struct intel_engine_cs *engine)
+static u32 gen11_lock_sfc(struct intel_engine_cs *engine)
{
- u8 vdbox_sfc_access = RUNTIME_INFO(dev_priv)->vdbox_sfc_access;
+ struct intel_uncore *uncore = engine->uncore;
+ u8 vdbox_sfc_access = RUNTIME_INFO(engine->i915)->vdbox_sfc_access;
i915_reg_t sfc_forced_lock, sfc_forced_lock_ack;
u32 sfc_forced_lock_bit, sfc_forced_lock_ack_bit;
i915_reg_t sfc_usage;
* ends up being locked to the engine we want to reset, we have to reset
* it as well (we will unlock it once the reset sequence is completed).
*/
- I915_WRITE_FW(sfc_forced_lock,
- I915_READ_FW(sfc_forced_lock) | sfc_forced_lock_bit);
+ rmw_set_fw(uncore, sfc_forced_lock, sfc_forced_lock_bit);
- if (__intel_wait_for_register_fw(dev_priv,
+ if (__intel_wait_for_register_fw(uncore,
sfc_forced_lock_ack,
sfc_forced_lock_ack_bit,
sfc_forced_lock_ack_bit,
return 0;
}
- if (I915_READ_FW(sfc_usage) & sfc_usage_bit)
+ if (intel_uncore_read_fw(uncore, sfc_usage) & sfc_usage_bit)
return sfc_reset_bit;
return 0;
}
-static void gen11_unlock_sfc(struct drm_i915_private *dev_priv,
- struct intel_engine_cs *engine)
+static void gen11_unlock_sfc(struct intel_engine_cs *engine)
{
- u8 vdbox_sfc_access = RUNTIME_INFO(dev_priv)->vdbox_sfc_access;
+ struct intel_uncore *uncore = engine->uncore;
+ u8 vdbox_sfc_access = RUNTIME_INFO(engine->i915)->vdbox_sfc_access;
i915_reg_t sfc_forced_lock;
u32 sfc_forced_lock_bit;
return;
}
- I915_WRITE_FW(sfc_forced_lock,
- I915_READ_FW(sfc_forced_lock) & ~sfc_forced_lock_bit);
+ rmw_clear_fw(uncore, sfc_forced_lock, sfc_forced_lock_bit);
}
static int gen11_reset_engines(struct drm_i915_private *i915,
- unsigned int engine_mask,
+ intel_engine_mask_t engine_mask,
unsigned int retry)
{
- const u32 hw_engine_mask[I915_NUM_ENGINES] = {
- [RCS] = GEN11_GRDOM_RENDER,
- [BCS] = GEN11_GRDOM_BLT,
- [VCS] = GEN11_GRDOM_MEDIA,
- [VCS2] = GEN11_GRDOM_MEDIA2,
- [VCS3] = GEN11_GRDOM_MEDIA3,
- [VCS4] = GEN11_GRDOM_MEDIA4,
- [VECS] = GEN11_GRDOM_VECS,
- [VECS2] = GEN11_GRDOM_VECS2,
+ const u32 hw_engine_mask[] = {
+ [RCS0] = GEN11_GRDOM_RENDER,
+ [BCS0] = GEN11_GRDOM_BLT,
+ [VCS0] = GEN11_GRDOM_MEDIA,
+ [VCS1] = GEN11_GRDOM_MEDIA2,
+ [VCS2] = GEN11_GRDOM_MEDIA3,
+ [VCS3] = GEN11_GRDOM_MEDIA4,
+ [VECS0] = GEN11_GRDOM_VECS,
+ [VECS1] = GEN11_GRDOM_VECS2,
};
struct intel_engine_cs *engine;
- unsigned int tmp;
+ intel_engine_mask_t tmp;
u32 hw_mask;
int ret;
- BUILD_BUG_ON(VECS2 + 1 != I915_NUM_ENGINES);
-
if (engine_mask == ALL_ENGINES) {
hw_mask = GEN11_GRDOM_FULL;
} else {
hw_mask = 0;
for_each_engine_masked(engine, i915, engine_mask, tmp) {
+ GEM_BUG_ON(engine->id >= ARRAY_SIZE(hw_engine_mask));
hw_mask |= hw_engine_mask[engine->id];
- hw_mask |= gen11_lock_sfc(i915, engine);
+ hw_mask |= gen11_lock_sfc(engine);
}
}
if (engine_mask != ALL_ENGINES)
for_each_engine_masked(engine, i915, engine_mask, tmp)
- gen11_unlock_sfc(i915, engine);
+ gen11_unlock_sfc(engine);
return ret;
}
static int gen8_engine_reset_prepare(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
+ struct intel_uncore *uncore = engine->uncore;
+ const i915_reg_t reg = RING_RESET_CTL(engine->mmio_base);
+ u32 request, mask, ack;
int ret;
- I915_WRITE_FW(RING_RESET_CTL(engine->mmio_base),
- _MASKED_BIT_ENABLE(RESET_CTL_REQUEST_RESET));
+ ack = intel_uncore_read_fw(uncore, reg);
+ if (ack & RESET_CTL_CAT_ERROR) {
+ /*
+ * For catastrophic errors, ready-for-reset sequence
+ * needs to be bypassed: HAS#396813
+ */
+ request = RESET_CTL_CAT_ERROR;
+ mask = RESET_CTL_CAT_ERROR;
+
+ /* Catastrophic errors need to be cleared by HW */
+ ack = 0;
+ } else if (!(ack & RESET_CTL_READY_TO_RESET)) {
+ request = RESET_CTL_REQUEST_RESET;
+ mask = RESET_CTL_READY_TO_RESET;
+ ack = RESET_CTL_READY_TO_RESET;
+ } else {
+ return 0;
+ }
- ret = __intel_wait_for_register_fw(dev_priv,
- RING_RESET_CTL(engine->mmio_base),
- RESET_CTL_READY_TO_RESET,
- RESET_CTL_READY_TO_RESET,
- 700, 0,
- NULL);
+ intel_uncore_write_fw(uncore, reg, _MASKED_BIT_ENABLE(request));
+ ret = __intel_wait_for_register_fw(uncore, reg, mask, ack,
+ 700, 0, NULL);
if (ret)
- DRM_ERROR("%s: reset request timeout\n", engine->name);
+ DRM_ERROR("%s reset request timed out: {request: %08x, RESET_CTL: %08x}\n",
+ engine->name, request,
+ intel_uncore_read_fw(uncore, reg));
return ret;
}
static void gen8_engine_reset_cancel(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
-
- I915_WRITE_FW(RING_RESET_CTL(engine->mmio_base),
- _MASKED_BIT_DISABLE(RESET_CTL_REQUEST_RESET));
+ intel_uncore_write_fw(engine->uncore,
+ RING_RESET_CTL(engine->mmio_base),
+ _MASKED_BIT_DISABLE(RESET_CTL_REQUEST_RESET));
}
static int gen8_reset_engines(struct drm_i915_private *i915,
- unsigned int engine_mask,
+ intel_engine_mask_t engine_mask,
unsigned int retry)
{
struct intel_engine_cs *engine;
const bool reset_non_ready = retry >= 1;
- unsigned int tmp;
+ intel_engine_mask_t tmp;
int ret;
for_each_engine_masked(engine, i915, engine_mask, tmp) {
}
typedef int (*reset_func)(struct drm_i915_private *,
- unsigned int engine_mask,
+ intel_engine_mask_t engine_mask,
unsigned int retry);
static reset_func intel_get_gpu_reset(struct drm_i915_private *i915)
{
- if (!i915_modparams.reset)
- return NULL;
-
if (INTEL_GEN(i915) >= 8)
return gen8_reset_engines;
else if (INTEL_GEN(i915) >= 6)
return NULL;
}
-int intel_gpu_reset(struct drm_i915_private *i915, unsigned int engine_mask)
+int intel_gpu_reset(struct drm_i915_private *i915,
+ intel_engine_mask_t engine_mask)
{
const int retries = engine_mask == ALL_ENGINES ? RESET_MAX_RETRIES : 1;
reset_func reset;
* If the power well sleeps during the reset, the reset
* request may be dropped and never completes (causing -EIO).
*/
- intel_uncore_forcewake_get(i915, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&i915->uncore, FORCEWAKE_ALL);
for (retry = 0; ret == -ETIMEDOUT && retry < retries; retry++) {
/*
* We stop engines, otherwise we might get failed reset and a
*
* FIXME: Wa for more modern gens needs to be validated
*/
- i915_stop_engines(i915, engine_mask);
+ if (retry)
+ i915_stop_engines(i915, engine_mask);
GEM_TRACE("engine_mask=%x\n", engine_mask);
preempt_disable();
ret = reset(i915, engine_mask, retry);
preempt_enable();
}
- intel_uncore_forcewake_put(i915, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&i915->uncore, FORCEWAKE_ALL);
return ret;
}
if (USES_GUC(i915))
return false;
+ if (!i915_modparams.reset)
+ return NULL;
+
return intel_get_gpu_reset(i915);
}
GEM_BUG_ON(!HAS_GUC(i915));
- intel_uncore_forcewake_get(i915, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&i915->uncore, FORCEWAKE_ALL);
ret = gen6_hw_domain_reset(i915, guc_domain);
- intel_uncore_forcewake_put(i915, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&i915->uncore, FORCEWAKE_ALL);
return ret;
}
* written to the powercontext is undefined and so we may lose
* GPU state upon resume, i.e. fail to restart after a reset.
*/
- intel_uncore_forcewake_get(engine->i915, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(engine->uncore, FORCEWAKE_ALL);
engine->reset.prepare(engine);
}
+static void revoke_mmaps(struct drm_i915_private *i915)
+{
+ int i;
+
+ for (i = 0; i < i915->num_fence_regs; i++) {
+ struct drm_vma_offset_node *node;
+ struct i915_vma *vma;
+ u64 vma_offset;
+
+ vma = READ_ONCE(i915->fence_regs[i].vma);
+ if (!vma)
+ continue;
+
+ if (!i915_vma_has_userfault(vma))
+ continue;
+
+ GEM_BUG_ON(vma->fence != &i915->fence_regs[i]);
+ node = &vma->obj->base.vma_node;
+ vma_offset = vma->ggtt_view.partial.offset << PAGE_SHIFT;
+ unmap_mapping_range(i915->drm.anon_inode->i_mapping,
+ drm_vma_node_offset_addr(node) + vma_offset,
+ vma->size,
+ 1);
+ }
+}
+
static void reset_prepare(struct drm_i915_private *i915)
{
struct intel_engine_cs *engine;
for_each_engine(engine, i915, id)
reset_prepare_engine(engine);
- intel_uc_sanitize(i915);
+ intel_uc_reset_prepare(i915);
+}
+
+static void gt_revoke(struct drm_i915_private *i915)
+{
+ revoke_mmaps(i915);
}
-static int gt_reset(struct drm_i915_private *i915, unsigned int stalled_mask)
+static int gt_reset(struct drm_i915_private *i915,
+ intel_engine_mask_t stalled_mask)
{
struct intel_engine_cs *engine;
enum intel_engine_id id;
return err;
for_each_engine(engine, i915, id)
- intel_engine_reset(engine, stalled_mask & ENGINE_MASK(id));
+ intel_engine_reset(engine, stalled_mask & engine->mask);
i915_gem_restore_fences(i915);
static void reset_finish_engine(struct intel_engine_cs *engine)
{
engine->reset.finish(engine);
- intel_uncore_forcewake_put(engine->i915, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(engine->uncore, FORCEWAKE_ALL);
}
struct i915_gpu_restart {
struct intel_engine_cs *engine;
enum intel_engine_id id;
- for_each_engine(engine, i915, id)
+ for_each_engine(engine, i915, id) {
reset_finish_engine(engine);
+ intel_engine_signal_breadcrumbs(engine);
+ }
}
static void reset_restart(struct drm_i915_private *i915)
spin_lock_irqsave(&engine->timeline.lock, flags);
__i915_request_submit(request);
i915_request_mark_complete(request);
- intel_engine_write_global_seqno(engine, request->global_seqno);
spin_unlock_irqrestore(&engine->timeline.lock, flags);
intel_engine_queue_breadcrumbs(engine);
}
-void i915_gem_set_wedged(struct drm_i915_private *i915)
+static void __i915_gem_set_wedged(struct drm_i915_private *i915)
{
struct i915_gpu_error *error = &i915->gpu_error;
struct intel_engine_cs *engine;
enum intel_engine_id id;
- mutex_lock(&error->wedge_mutex);
- if (test_bit(I915_WEDGED, &error->flags)) {
- mutex_unlock(&error->wedge_mutex);
+ if (test_bit(I915_WEDGED, &error->flags))
return;
- }
if (GEM_SHOW_DEBUG() && !intel_engines_are_idle(i915)) {
struct drm_printer p = drm_debug_printer(__func__);
* rolling the global seqno forward (since this would complete requests
* for which we haven't set the fence error to EIO yet).
*/
- for_each_engine(engine, i915, id)
- reset_prepare_engine(engine);
+ reset_prepare(i915);
/* Even if the GPU reset fails, it should still stop the engines */
- if (INTEL_GEN(i915) >= 5)
+ if (!INTEL_INFO(i915)->gpu_reset_clobbers_display)
intel_gpu_reset(i915, ALL_ENGINES);
for_each_engine(engine, i915, id) {
* either this call here to intel_engine_write_global_seqno, or the one
* in nop_submit_request.
*/
- synchronize_rcu();
+ synchronize_rcu_expedited();
/* Mark all executing requests as skipped */
for_each_engine(engine, i915, id)
engine->cancel_requests(engine);
- for_each_engine(engine, i915, id) {
- reset_finish_engine(engine);
- intel_engine_signal_breadcrumbs(engine);
- }
+ reset_finish(i915);
smp_mb__before_atomic();
set_bit(I915_WEDGED, &error->flags);
GEM_TRACE("end\n");
- mutex_unlock(&error->wedge_mutex);
+}
- wake_up_all(&error->reset_queue);
+void i915_gem_set_wedged(struct drm_i915_private *i915)
+{
+ struct i915_gpu_error *error = &i915->gpu_error;
+ intel_wakeref_t wakeref;
+
+ mutex_lock(&error->wedge_mutex);
+ with_intel_runtime_pm(i915, wakeref)
+ __i915_gem_set_wedged(i915);
+ mutex_unlock(&error->wedge_mutex);
}
-bool i915_gem_unset_wedged(struct drm_i915_private *i915)
+static bool __i915_gem_unset_wedged(struct drm_i915_private *i915)
{
struct i915_gpu_error *error = &i915->gpu_error;
struct i915_timeline *tl;
- bool ret = false;
if (!test_bit(I915_WEDGED, &error->flags))
return true;
if (!i915->gt.scratch) /* Never full initialised, recovery impossible */
return false;
- mutex_lock(&error->wedge_mutex);
-
GEM_TRACE("start\n");
/*
mutex_lock(&i915->gt.timelines.mutex);
list_for_each_entry(tl, &i915->gt.timelines.active_list, link) {
struct i915_request *rq;
- long timeout;
rq = i915_active_request_get_unlocked(&tl->last_request);
if (!rq)
continue;
/*
- * We can't use our normal waiter as we want to
- * avoid recursively trying to handle the current
- * reset. The basic dma_fence_default_wait() installs
- * a callback for dma_fence_signal(), which is
- * triggered by our nop handler (indirectly, the
- * callback enables the signaler thread which is
- * woken by the nop_submit_request() advancing the seqno
- * and when the seqno passes the fence, the signaler
- * then signals the fence waking us up).
+ * All internal dependencies (i915_requests) will have
+ * been flushed by the set-wedge, but we may be stuck waiting
+ * for external fences. These should all be capped to 10s
+ * (I915_FENCE_TIMEOUT) so this wait should not be unbounded
+ * in the worst case.
*/
- timeout = dma_fence_default_wait(&rq->fence, true,
- MAX_SCHEDULE_TIMEOUT);
+ dma_fence_default_wait(&rq->fence, false, MAX_SCHEDULE_TIMEOUT);
i915_request_put(rq);
- if (timeout < 0) {
- mutex_unlock(&i915->gt.timelines.mutex);
- goto unlock;
- }
}
mutex_unlock(&i915->gt.timelines.mutex);
smp_mb__before_atomic(); /* complete takeover before enabling execbuf */
clear_bit(I915_WEDGED, &i915->gpu_error.flags);
- ret = true;
-unlock:
- mutex_unlock(&i915->gpu_error.wedge_mutex);
- return ret;
+ return true;
}
-struct __i915_reset {
- struct drm_i915_private *i915;
- unsigned int stalled_mask;
-};
-
-static int __i915_reset__BKL(void *data)
+bool i915_gem_unset_wedged(struct drm_i915_private *i915)
{
- struct __i915_reset *arg = data;
- int err;
+ struct i915_gpu_error *error = &i915->gpu_error;
+ bool result;
- err = intel_gpu_reset(arg->i915, ALL_ENGINES);
- if (err)
- return err;
+ mutex_lock(&error->wedge_mutex);
+ result = __i915_gem_unset_wedged(i915);
+ mutex_unlock(&error->wedge_mutex);
- return gt_reset(arg->i915, arg->stalled_mask);
+ return result;
}
-#if RESET_UNDER_STOP_MACHINE
-/*
- * XXX An alternative to using stop_machine would be to park only the
- * processes that have a GGTT mmap. By remote parking the threads (SIGSTOP)
- * we should be able to prevent their memmory accesses via the lost fence
- * registers over the course of the reset without the potential recursive
- * of mutexes between the pagefault handler and reset.
- *
- * See igt/gem_mmap_gtt/hang
- */
-#define __do_reset(fn, arg) stop_machine(fn, arg, NULL)
-#else
-#define __do_reset(fn, arg) fn(arg)
-#endif
-
-static int do_reset(struct drm_i915_private *i915, unsigned int stalled_mask)
+static int do_reset(struct drm_i915_private *i915,
+ intel_engine_mask_t stalled_mask)
{
- struct __i915_reset arg = { i915, stalled_mask };
int err, i;
- err = __do_reset(__i915_reset__BKL, &arg);
+ gt_revoke(i915);
+
+ err = intel_gpu_reset(i915, ALL_ENGINES);
for (i = 0; err && i < RESET_MAX_RETRIES; i++) {
- msleep(100);
- err = __do_reset(__i915_reset__BKL, &arg);
+ msleep(10 * (i + 1));
+ err = intel_gpu_reset(i915, ALL_ENGINES);
}
+ if (err)
+ return err;
- return err;
+ return gt_reset(i915, stalled_mask);
}
/**
* Reset the chip. Useful if a hang is detected. Marks the device as wedged
* on failure.
*
- * Caller must hold the struct_mutex.
- *
* Procedure is fairly simple:
* - reset the chip using the reset reg
* - re-init context state
* - re-init display
*/
void i915_reset(struct drm_i915_private *i915,
- unsigned int stalled_mask,
+ intel_engine_mask_t stalled_mask,
const char *reason)
{
struct i915_gpu_error *error = &i915->gpu_error;
GEM_BUG_ON(!test_bit(I915_RESET_BACKOFF, &error->flags));
/* Clear any previous failed attempts at recovery. Time to try again. */
- if (!i915_gem_unset_wedged(i915))
+ if (!__i915_gem_unset_wedged(i915))
return;
if (reason)
goto error;
}
+ if (INTEL_INFO(i915)->gpu_reset_clobbers_display)
+ intel_runtime_pm_disable_interrupts(i915);
+
if (do_reset(i915, stalled_mask)) {
dev_err(i915->drm.dev, "Failed to reset chip\n");
goto taint;
}
+ if (INTEL_INFO(i915)->gpu_reset_clobbers_display)
+ intel_runtime_pm_enable_interrupts(i915);
+
intel_overlay_reset(i915);
/*
finish:
reset_finish(i915);
- if (!i915_terminally_wedged(error))
+ if (!__i915_wedged(error))
reset_restart(i915);
return;
*/
add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
error:
- i915_gem_set_wedged(i915);
+ __i915_gem_set_wedged(i915);
goto finish;
}
static inline int intel_gt_reset_engine(struct drm_i915_private *i915,
struct intel_engine_cs *engine)
{
- return intel_gpu_reset(i915, intel_engine_flag(engine));
+ return intel_gpu_reset(i915, engine->mask);
}
/**
i915_wedge_on_timeout(&w, i915, 5 * HZ) {
intel_prepare_reset(i915);
+ /* Flush everyone using a resource about to be clobbered */
+ synchronize_srcu_expedited(&error->reset_backoff_srcu);
+
+ mutex_lock(&error->wedge_mutex);
i915_reset(i915, engine_mask, reason);
+ mutex_unlock(&error->wedge_mutex);
intel_finish_reset(i915);
}
kobject_uevent_env(kobj, KOBJ_CHANGE, reset_done_event);
}
-void i915_clear_error_registers(struct drm_i915_private *dev_priv)
+static void clear_register(struct intel_uncore *uncore, i915_reg_t reg)
{
+ intel_uncore_rmw(uncore, reg, 0, 0);
+}
+
+void i915_clear_error_registers(struct drm_i915_private *i915)
+{
+ struct intel_uncore *uncore = &i915->uncore;
u32 eir;
- if (!IS_GEN(dev_priv, 2))
- I915_WRITE(PGTBL_ER, I915_READ(PGTBL_ER));
+ if (!IS_GEN(i915, 2))
+ clear_register(uncore, PGTBL_ER);
- if (INTEL_GEN(dev_priv) < 4)
- I915_WRITE(IPEIR, I915_READ(IPEIR));
+ if (INTEL_GEN(i915) < 4)
+ clear_register(uncore, IPEIR(RENDER_RING_BASE));
else
- I915_WRITE(IPEIR_I965, I915_READ(IPEIR_I965));
+ clear_register(uncore, IPEIR_I965);
- I915_WRITE(EIR, I915_READ(EIR));
- eir = I915_READ(EIR);
+ clear_register(uncore, EIR);
+ eir = intel_uncore_read(uncore, EIR);
if (eir) {
/*
* some errors might have become stuck,
* mask them.
*/
DRM_DEBUG_DRIVER("EIR stuck: 0x%08x, masking\n", eir);
- I915_WRITE(EMR, I915_READ(EMR) | eir);
- I915_WRITE(IIR, I915_MASTER_ERROR_INTERRUPT);
+ rmw_set(uncore, EMR, eir);
+ intel_uncore_write(uncore, GEN2_IIR,
+ I915_MASTER_ERROR_INTERRUPT);
}
- if (INTEL_GEN(dev_priv) >= 8) {
- I915_WRITE(GEN8_RING_FAULT_REG,
- I915_READ(GEN8_RING_FAULT_REG) & ~RING_FAULT_VALID);
- POSTING_READ(GEN8_RING_FAULT_REG);
- } else if (INTEL_GEN(dev_priv) >= 6) {
+ if (INTEL_GEN(i915) >= 8) {
+ rmw_clear(uncore, GEN8_RING_FAULT_REG, RING_FAULT_VALID);
+ intel_uncore_posting_read(uncore, GEN8_RING_FAULT_REG);
+ } else if (INTEL_GEN(i915) >= 6) {
struct intel_engine_cs *engine;
enum intel_engine_id id;
- for_each_engine(engine, dev_priv, id) {
- I915_WRITE(RING_FAULT_REG(engine),
- I915_READ(RING_FAULT_REG(engine)) &
- ~RING_FAULT_VALID);
+ for_each_engine(engine, i915, id) {
+ rmw_clear(uncore,
+ RING_FAULT_REG(engine), RING_FAULT_VALID);
+ intel_uncore_posting_read(uncore,
+ RING_FAULT_REG(engine));
}
- POSTING_READ(RING_FAULT_REG(dev_priv->engine[RCS]));
}
}
* of a ring dump etc.).
*/
void i915_handle_error(struct drm_i915_private *i915,
- u32 engine_mask,
+ intel_engine_mask_t engine_mask,
unsigned long flags,
const char *fmt, ...)
{
+ struct i915_gpu_error *error = &i915->gpu_error;
struct intel_engine_cs *engine;
intel_wakeref_t wakeref;
- unsigned int tmp;
+ intel_engine_mask_t tmp;
char error_msg[80];
char *msg = NULL;
*/
wakeref = intel_runtime_pm_get(i915);
- engine_mask &= INTEL_INFO(i915)->ring_mask;
+ engine_mask &= INTEL_INFO(i915)->engine_mask;
if (flags & I915_ERROR_CAPTURE) {
i915_capture_error_state(i915, engine_mask, msg);
* Try engine reset when available. We fall back to full reset if
* single reset fails.
*/
- if (intel_has_reset_engine(i915) &&
- !i915_terminally_wedged(&i915->gpu_error)) {
+ if (intel_has_reset_engine(i915) && !__i915_wedged(error)) {
for_each_engine_masked(engine, i915, engine_mask, tmp) {
BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
- &i915->gpu_error.flags))
+ &error->flags))
continue;
if (i915_reset_engine(engine, msg) == 0)
- engine_mask &= ~intel_engine_flag(engine);
+ engine_mask &= ~engine->mask;
clear_bit(I915_RESET_ENGINE + engine->id,
- &i915->gpu_error.flags);
- wake_up_bit(&i915->gpu_error.flags,
+ &error->flags);
+ wake_up_bit(&error->flags,
I915_RESET_ENGINE + engine->id);
}
}
goto out;
/* Full reset needs the mutex, stop any other user trying to do so. */
- if (test_and_set_bit(I915_RESET_BACKOFF, &i915->gpu_error.flags)) {
- wait_event(i915->gpu_error.reset_queue,
- !test_bit(I915_RESET_BACKOFF,
- &i915->gpu_error.flags));
- goto out;
+ if (test_and_set_bit(I915_RESET_BACKOFF, &error->flags)) {
+ wait_event(error->reset_queue,
+ !test_bit(I915_RESET_BACKOFF, &error->flags));
+ goto out; /* piggy-back on the other reset */
}
+ /* Make sure i915_reset_trylock() sees the I915_RESET_BACKOFF */
+ synchronize_rcu_expedited();
+
/* Prevent any other reset-engine attempt. */
for_each_engine(engine, i915, tmp) {
while (test_and_set_bit(I915_RESET_ENGINE + engine->id,
- &i915->gpu_error.flags))
- wait_on_bit(&i915->gpu_error.flags,
+ &error->flags))
+ wait_on_bit(&error->flags,
I915_RESET_ENGINE + engine->id,
TASK_UNINTERRUPTIBLE);
}
for_each_engine(engine, i915, tmp) {
clear_bit(I915_RESET_ENGINE + engine->id,
- &i915->gpu_error.flags);
+ &error->flags);
}
- clear_bit(I915_RESET_BACKOFF, &i915->gpu_error.flags);
- wake_up_all(&i915->gpu_error.reset_queue);
+ clear_bit(I915_RESET_BACKOFF, &error->flags);
+ wake_up_all(&error->reset_queue);
out:
intel_runtime_pm_put(i915, wakeref);
}
+int i915_reset_trylock(struct drm_i915_private *i915)
+{
+ struct i915_gpu_error *error = &i915->gpu_error;
+ int srcu;
+
+ might_lock(&error->reset_backoff_srcu);
+ might_sleep();
+
+ rcu_read_lock();
+ while (test_bit(I915_RESET_BACKOFF, &error->flags)) {
+ rcu_read_unlock();
+
+ if (wait_event_interruptible(error->reset_queue,
+ !test_bit(I915_RESET_BACKOFF,
+ &error->flags)))
+ return -EINTR;
+
+ rcu_read_lock();
+ }
+ srcu = srcu_read_lock(&error->reset_backoff_srcu);
+ rcu_read_unlock();
+
+ return srcu;
+}
+
+void i915_reset_unlock(struct drm_i915_private *i915, int tag)
+__releases(&i915->gpu_error.reset_backoff_srcu)
+{
+ struct i915_gpu_error *error = &i915->gpu_error;
+
+ srcu_read_unlock(&error->reset_backoff_srcu, tag);
+}
+
+int i915_terminally_wedged(struct drm_i915_private *i915)
+{
+ struct i915_gpu_error *error = &i915->gpu_error;
+
+ might_sleep();
+
+ if (!__i915_wedged(error))
+ return 0;
+
+ /* Reset still in progress? Maybe we will recover? */
+ if (!test_bit(I915_RESET_BACKOFF, &error->flags))
+ return -EIO;
+
+ /* XXX intel_reset_finish() still takes struct_mutex!!! */
+ if (mutex_is_locked(&i915->drm.struct_mutex))
+ return -EAGAIN;
+
+ if (wait_event_interruptible(error->reset_queue,
+ !test_bit(I915_RESET_BACKOFF,
+ &error->flags)))
+ return -EINTR;
+
+ return __i915_wedged(error) ? -EIO : 0;
+}
+
bool i915_reset_flush(struct drm_i915_private *i915)
{
int err;
#include <linux/compiler.h>
#include <linux/types.h>
+#include <linux/srcu.h>
+
+#include "intel_engine_types.h"
struct drm_i915_private;
+struct i915_request;
struct intel_engine_cs;
struct intel_guc;
__printf(4, 5)
void i915_handle_error(struct drm_i915_private *i915,
- u32 engine_mask,
+ intel_engine_mask_t engine_mask,
unsigned long flags,
const char *fmt, ...);
#define I915_ERROR_CAPTURE BIT(0)
void i915_clear_error_registers(struct drm_i915_private *i915);
void i915_reset(struct drm_i915_private *i915,
- unsigned int stalled_mask,
+ intel_engine_mask_t stalled_mask,
const char *reason);
int i915_reset_engine(struct intel_engine_cs *engine,
const char *reason);
void i915_reset_request(struct i915_request *rq, bool guilty);
bool i915_reset_flush(struct drm_i915_private *i915);
+int __must_check i915_reset_trylock(struct drm_i915_private *i915);
+void i915_reset_unlock(struct drm_i915_private *i915, int tag);
+
+int i915_terminally_wedged(struct drm_i915_private *i915);
+
bool intel_has_gpu_reset(struct drm_i915_private *i915);
bool intel_has_reset_engine(struct drm_i915_private *i915);
-int intel_gpu_reset(struct drm_i915_private *i915, u32 engine_mask);
+int intel_gpu_reset(struct drm_i915_private *i915,
+ intel_engine_mask_t engine_mask);
int intel_reset_guc(struct drm_i915_private *i915);
#include <linux/mutex.h>
#include "i915_drv.h"
+#include "i915_globals.h"
#include "i915_request.h"
#include "i915_scheduler.h"
+static struct i915_global_scheduler {
+ struct i915_global base;
+ struct kmem_cache *slab_dependencies;
+ struct kmem_cache *slab_priorities;
+} global;
+
static DEFINE_SPINLOCK(schedule_lock);
static const struct i915_request *
return container_of(node, const struct i915_request, sched);
}
+static inline bool node_started(const struct i915_sched_node *node)
+{
+ return i915_request_started(node_to_request(node));
+}
+
static inline bool node_signaled(const struct i915_sched_node *node)
{
return i915_request_completed(node_to_request(node));
INIT_LIST_HEAD(&node->waiters_list);
INIT_LIST_HEAD(&node->link);
node->attr.priority = I915_PRIORITY_INVALID;
+ node->semaphores = 0;
+ node->flags = 0;
}
static struct i915_dependency *
-i915_dependency_alloc(struct drm_i915_private *i915)
+i915_dependency_alloc(void)
{
- return kmem_cache_alloc(i915->dependencies, GFP_KERNEL);
+ return kmem_cache_alloc(global.slab_dependencies, GFP_KERNEL);
}
static void
-i915_dependency_free(struct drm_i915_private *i915,
- struct i915_dependency *dep)
+i915_dependency_free(struct i915_dependency *dep)
{
- kmem_cache_free(i915->dependencies, dep);
+ kmem_cache_free(global.slab_dependencies, dep);
}
bool __i915_sched_node_add_dependency(struct i915_sched_node *node,
{
bool ret = false;
- spin_lock(&schedule_lock);
+ spin_lock_irq(&schedule_lock);
if (!node_signaled(signal)) {
INIT_LIST_HEAD(&dep->dfs_link);
dep->signaler = signal;
dep->flags = flags;
+ /* Keep track of whether anyone on this chain has a semaphore */
+ if (signal->flags & I915_SCHED_HAS_SEMAPHORE_CHAIN &&
+ !node_started(signal))
+ node->flags |= I915_SCHED_HAS_SEMAPHORE_CHAIN;
+
ret = true;
}
- spin_unlock(&schedule_lock);
+ spin_unlock_irq(&schedule_lock);
return ret;
}
-int i915_sched_node_add_dependency(struct drm_i915_private *i915,
- struct i915_sched_node *node,
+int i915_sched_node_add_dependency(struct i915_sched_node *node,
struct i915_sched_node *signal)
{
struct i915_dependency *dep;
- dep = i915_dependency_alloc(i915);
+ dep = i915_dependency_alloc();
if (!dep)
return -ENOMEM;
if (!__i915_sched_node_add_dependency(node, signal, dep,
I915_DEPENDENCY_ALLOC))
- i915_dependency_free(i915, dep);
+ i915_dependency_free(dep);
return 0;
}
-void i915_sched_node_fini(struct drm_i915_private *i915,
- struct i915_sched_node *node)
+void i915_sched_node_fini(struct i915_sched_node *node)
{
struct i915_dependency *dep, *tmp;
GEM_BUG_ON(!list_empty(&node->link));
- spin_lock(&schedule_lock);
+ spin_lock_irq(&schedule_lock);
/*
* Everyone we depended upon (the fences we wait to be signaled)
list_del(&dep->wait_link);
if (dep->flags & I915_DEPENDENCY_ALLOC)
- i915_dependency_free(i915, dep);
+ i915_dependency_free(dep);
}
/* Remove ourselves from everyone who depends upon us */
list_del(&dep->signal_link);
if (dep->flags & I915_DEPENDENCY_ALLOC)
- i915_dependency_free(i915, dep);
+ i915_dependency_free(dep);
}
- spin_unlock(&schedule_lock);
+ spin_unlock_irq(&schedule_lock);
}
static inline struct i915_priolist *to_priolist(struct rb_node *rb)
if (prio == I915_PRIORITY_NORMAL) {
p = &execlists->default_priolist;
} else {
- p = kmem_cache_alloc(engine->i915->priorities, GFP_ATOMIC);
+ p = kmem_cache_alloc(global.slab_priorities, GFP_ATOMIC);
/* Convert an allocation failure to a priority bump */
if (unlikely(!p)) {
prio = I915_PRIORITY_NORMAL; /* recurses just once */
list_for_each_entry(dep, &dfs, dfs_link) {
struct i915_sched_node *node = dep->signaler;
+ /* If we are already flying, we know we have no signalers */
+ if (node_started(node))
+ continue;
+
/*
* Within an engine, there can be no cycle, but we may
* refer to the same dependency chain multiple times
if (node_signaled(p->signaler))
continue;
- GEM_BUG_ON(p->signaler->attr.priority < node->attr.priority);
if (prio > READ_ONCE(p->signaler->attr.priority))
list_move_tail(&p->dfs_link, &dfs);
}
memset(&cache, 0, sizeof(cache));
engine = rq->engine;
- spin_lock_irq(&engine->timeline.lock);
+ spin_lock(&engine->timeline.lock);
/* Fifo and depth-first replacement ensure our deps execute before us */
list_for_each_entry_safe_reverse(dep, p, &dfs, dfs_link) {
tasklet_hi_schedule(&engine->execlists.tasklet);
}
- spin_unlock_irq(&engine->timeline.lock);
+ spin_unlock(&engine->timeline.lock);
}
void i915_schedule(struct i915_request *rq, const struct i915_sched_attr *attr)
{
- spin_lock(&schedule_lock);
+ spin_lock_irq(&schedule_lock);
__i915_schedule(rq, attr);
- spin_unlock(&schedule_lock);
+ spin_unlock_irq(&schedule_lock);
}
void i915_schedule_bump_priority(struct i915_request *rq, unsigned int bump)
{
struct i915_sched_attr attr;
+ unsigned long flags;
GEM_BUG_ON(bump & ~I915_PRIORITY_MASK);
if (READ_ONCE(rq->sched.attr.priority) == I915_PRIORITY_INVALID)
return;
- spin_lock_bh(&schedule_lock);
+ spin_lock_irqsave(&schedule_lock, flags);
attr = rq->sched.attr;
attr.priority |= bump;
__i915_schedule(rq, &attr);
- spin_unlock_bh(&schedule_lock);
+ spin_unlock_irqrestore(&schedule_lock, flags);
+}
+
+void __i915_priolist_free(struct i915_priolist *p)
+{
+ kmem_cache_free(global.slab_priorities, p);
+}
+
+static void i915_global_scheduler_shrink(void)
+{
+ kmem_cache_shrink(global.slab_dependencies);
+ kmem_cache_shrink(global.slab_priorities);
+}
+
+static void i915_global_scheduler_exit(void)
+{
+ kmem_cache_destroy(global.slab_dependencies);
+ kmem_cache_destroy(global.slab_priorities);
+}
+
+static struct i915_global_scheduler global = { {
+ .shrink = i915_global_scheduler_shrink,
+ .exit = i915_global_scheduler_exit,
+} };
+
+int __init i915_global_scheduler_init(void)
+{
+ global.slab_dependencies = KMEM_CACHE(i915_dependency,
+ SLAB_HWCACHE_ALIGN);
+ if (!global.slab_dependencies)
+ return -ENOMEM;
+
+ global.slab_priorities = KMEM_CACHE(i915_priolist,
+ SLAB_HWCACHE_ALIGN);
+ if (!global.slab_priorities)
+ goto err_priorities;
+
+ i915_global_register(&global.base);
+ return 0;
+
+err_priorities:
+ kmem_cache_destroy(global.slab_priorities);
+ return -ENOMEM;
}
#define _I915_SCHEDULER_H_
#include <linux/bitops.h>
+#include <linux/list.h>
#include <linux/kernel.h>
-#include <uapi/drm/i915_drm.h>
+#include "i915_scheduler_types.h"
-struct drm_i915_private;
-struct i915_request;
-struct intel_engine_cs;
+#define priolist_for_each_request(it, plist, idx) \
+ for (idx = 0; idx < ARRAY_SIZE((plist)->requests); idx++) \
+ list_for_each_entry(it, &(plist)->requests[idx], sched.link)
-enum {
- I915_PRIORITY_MIN = I915_CONTEXT_MIN_USER_PRIORITY - 1,
- I915_PRIORITY_NORMAL = I915_CONTEXT_DEFAULT_PRIORITY,
- I915_PRIORITY_MAX = I915_CONTEXT_MAX_USER_PRIORITY + 1,
-
- I915_PRIORITY_INVALID = INT_MIN
-};
-
-#define I915_USER_PRIORITY_SHIFT 2
-#define I915_USER_PRIORITY(x) ((x) << I915_USER_PRIORITY_SHIFT)
-
-#define I915_PRIORITY_COUNT BIT(I915_USER_PRIORITY_SHIFT)
-#define I915_PRIORITY_MASK (I915_PRIORITY_COUNT - 1)
-
-#define I915_PRIORITY_WAIT ((u8)BIT(0))
-#define I915_PRIORITY_NEWCLIENT ((u8)BIT(1))
-
-struct i915_sched_attr {
- /**
- * @priority: execution and service priority
- *
- * All clients are equal, but some are more equal than others!
- *
- * Requests from a context with a greater (more positive) value of
- * @priority will be executed before those with a lower @priority
- * value, forming a simple QoS.
- *
- * The &drm_i915_private.kernel_context is assigned the lowest priority.
- */
- int priority;
-};
-
-/*
- * "People assume that time is a strict progression of cause to effect, but
- * actually, from a nonlinear, non-subjective viewpoint, it's more like a big
- * ball of wibbly-wobbly, timey-wimey ... stuff." -The Doctor, 2015
- *
- * Requests exist in a complex web of interdependencies. Each request
- * has to wait for some other request to complete before it is ready to be run
- * (e.g. we have to wait until the pixels have been rendering into a texture
- * before we can copy from it). We track the readiness of a request in terms
- * of fences, but we also need to keep the dependency tree for the lifetime
- * of the request (beyond the life of an individual fence). We use the tree
- * at various points to reorder the requests whilst keeping the requests
- * in order with respect to their various dependencies.
- *
- * There is no active component to the "scheduler". As we know the dependency
- * DAG of each request, we are able to insert it into a sorted queue when it
- * is ready, and are able to reorder its portion of the graph to accommodate
- * dynamic priority changes.
- */
-struct i915_sched_node {
- struct list_head signalers_list; /* those before us, we depend upon */
- struct list_head waiters_list; /* those after us, they depend upon us */
- struct list_head link;
- struct i915_sched_attr attr;
-};
-
-struct i915_dependency {
- struct i915_sched_node *signaler;
- struct list_head signal_link;
- struct list_head wait_link;
- struct list_head dfs_link;
- unsigned long flags;
-#define I915_DEPENDENCY_ALLOC BIT(0)
-};
+#define priolist_for_each_request_consume(it, n, plist, idx) \
+ for (; \
+ (plist)->used ? (idx = __ffs((plist)->used)), 1 : 0; \
+ (plist)->used &= ~BIT(idx)) \
+ list_for_each_entry_safe(it, n, \
+ &(plist)->requests[idx], \
+ sched.link)
void i915_sched_node_init(struct i915_sched_node *node);
struct i915_dependency *dep,
unsigned long flags);
-int i915_sched_node_add_dependency(struct drm_i915_private *i915,
- struct i915_sched_node *node,
+int i915_sched_node_add_dependency(struct i915_sched_node *node,
struct i915_sched_node *signal);
-void i915_sched_node_fini(struct drm_i915_private *i915,
- struct i915_sched_node *node);
+void i915_sched_node_fini(struct i915_sched_node *node);
void i915_schedule(struct i915_request *request,
const struct i915_sched_attr *attr);
struct list_head *
i915_sched_lookup_priolist(struct intel_engine_cs *engine, int prio);
+void __i915_priolist_free(struct i915_priolist *p);
+static inline void i915_priolist_free(struct i915_priolist *p)
+{
+ if (p->priority != I915_PRIORITY_NORMAL)
+ __i915_priolist_free(p);
+}
+
#endif /* _I915_SCHEDULER_H_ */
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2018 Intel Corporation
+ */
+
+#ifndef _I915_SCHEDULER_TYPES_H_
+#define _I915_SCHEDULER_TYPES_H_
+
+#include <linux/list.h>
+
+#include "i915_priolist_types.h"
+#include "intel_engine_types.h"
+
+struct drm_i915_private;
+struct i915_request;
+struct intel_engine_cs;
+
+struct i915_sched_attr {
+ /**
+ * @priority: execution and service priority
+ *
+ * All clients are equal, but some are more equal than others!
+ *
+ * Requests from a context with a greater (more positive) value of
+ * @priority will be executed before those with a lower @priority
+ * value, forming a simple QoS.
+ *
+ * The &drm_i915_private.kernel_context is assigned the lowest priority.
+ */
+ int priority;
+};
+
+/*
+ * "People assume that time is a strict progression of cause to effect, but
+ * actually, from a nonlinear, non-subjective viewpoint, it's more like a big
+ * ball of wibbly-wobbly, timey-wimey ... stuff." -The Doctor, 2015
+ *
+ * Requests exist in a complex web of interdependencies. Each request
+ * has to wait for some other request to complete before it is ready to be run
+ * (e.g. we have to wait until the pixels have been rendering into a texture
+ * before we can copy from it). We track the readiness of a request in terms
+ * of fences, but we also need to keep the dependency tree for the lifetime
+ * of the request (beyond the life of an individual fence). We use the tree
+ * at various points to reorder the requests whilst keeping the requests
+ * in order with respect to their various dependencies.
+ *
+ * There is no active component to the "scheduler". As we know the dependency
+ * DAG of each request, we are able to insert it into a sorted queue when it
+ * is ready, and are able to reorder its portion of the graph to accommodate
+ * dynamic priority changes.
+ */
+struct i915_sched_node {
+ struct list_head signalers_list; /* those before us, we depend upon */
+ struct list_head waiters_list; /* those after us, they depend upon us */
+ struct list_head link;
+ struct i915_sched_attr attr;
+ unsigned int flags;
+#define I915_SCHED_HAS_SEMAPHORE_CHAIN BIT(0)
+ intel_engine_mask_t semaphores;
+};
+
+struct i915_dependency {
+ struct i915_sched_node *signaler;
+ struct list_head signal_link;
+ struct list_head wait_link;
+ struct list_head dfs_link;
+ unsigned long flags;
+#define I915_DEPENDENCY_ALLOC BIT(0)
+};
+
+#endif /* _I915_SCHEDULER_TYPES_H_ */
*/
#include <drm/i915_drm.h>
-#include "intel_drv.h"
+
#include "i915_reg.h"
+#include "intel_drv.h"
+#include "intel_fbc.h"
static void i915_save_display(struct drm_i915_private *dev_priv)
{
__i915_sw_fence_notify(fence, FENCE_FREE);
}
-static void i915_sw_fence_complete(struct i915_sw_fence *fence)
+void i915_sw_fence_complete(struct i915_sw_fence *fence)
{
debug_fence_assert(fence);
__i915_sw_fence_complete(fence, NULL);
}
-static void i915_sw_fence_await(struct i915_sw_fence *fence)
+void i915_sw_fence_await(struct i915_sw_fence *fence)
{
debug_fence_assert(fence);
WARN_ON(atomic_inc_return(&fence->pending) <= 1);
return __i915_sw_fence_await_sw_fence(fence, signaler, NULL, gfp);
}
-struct i915_sw_dma_fence_cb {
- struct dma_fence_cb base;
- struct i915_sw_fence *fence;
-};
-
struct i915_sw_dma_fence_cb_timer {
struct i915_sw_dma_fence_cb base;
struct dma_fence *dma;
return ret;
}
+static void __dma_i915_sw_fence_wake(struct dma_fence *dma,
+ struct dma_fence_cb *data)
+{
+ struct i915_sw_dma_fence_cb *cb = container_of(data, typeof(*cb), base);
+
+ i915_sw_fence_complete(cb->fence);
+}
+
+int __i915_sw_fence_await_dma_fence(struct i915_sw_fence *fence,
+ struct dma_fence *dma,
+ struct i915_sw_dma_fence_cb *cb)
+{
+ int ret;
+
+ debug_fence_assert(fence);
+
+ if (dma_fence_is_signaled(dma))
+ return 0;
+
+ cb->fence = fence;
+ i915_sw_fence_await(fence);
+
+ ret = dma_fence_add_callback(dma, &cb->base, __dma_i915_sw_fence_wake);
+ if (ret == 0) {
+ ret = 1;
+ } else {
+ i915_sw_fence_complete(fence);
+ if (ret == -ENOENT) /* fence already signaled */
+ ret = 0;
+ }
+
+ return ret;
+}
+
int i915_sw_fence_await_reservation(struct i915_sw_fence *fence,
struct reservation_object *resv,
const struct dma_fence_ops *exclude,
#ifndef _I915_SW_FENCE_H_
#define _I915_SW_FENCE_H_
+#include <linux/dma-fence.h>
#include <linux/gfp.h>
#include <linux/kref.h>
#include <linux/notifier.h> /* for NOTIFY_DONE */
#include <linux/wait.h>
struct completion;
-struct dma_fence;
-struct dma_fence_ops;
struct reservation_object;
struct i915_sw_fence {
int i915_sw_fence_await_sw_fence_gfp(struct i915_sw_fence *fence,
struct i915_sw_fence *after,
gfp_t gfp);
+
+struct i915_sw_dma_fence_cb {
+ struct dma_fence_cb base;
+ struct i915_sw_fence *fence;
+};
+
+int __i915_sw_fence_await_dma_fence(struct i915_sw_fence *fence,
+ struct dma_fence *dma,
+ struct i915_sw_dma_fence_cb *cb);
int i915_sw_fence_await_dma_fence(struct i915_sw_fence *fence,
struct dma_fence *dma,
unsigned long timeout,
gfp_t gfp);
+
int i915_sw_fence_await_reservation(struct i915_sw_fence *fence,
struct reservation_object *resv,
const struct dma_fence_ops *exclude,
unsigned long timeout,
gfp_t gfp);
+void i915_sw_fence_await(struct i915_sw_fence *fence);
+void i915_sw_fence_complete(struct i915_sw_fence *fence);
+
static inline bool i915_sw_fence_signaled(const struct i915_sw_fence *fence)
{
return atomic_read(&fence->pending) <= 0;
#include "i915_drv.h"
-#include "i915_timeline.h"
+#include "i915_active.h"
#include "i915_syncmap.h"
+#include "i915_timeline.h"
+
+#define ptr_set_bit(ptr, bit) ((typeof(ptr))((unsigned long)(ptr) | BIT(bit)))
+#define ptr_test_bit(ptr, bit) ((unsigned long)(ptr) & BIT(bit))
struct i915_timeline_hwsp {
- struct i915_vma *vma;
+ struct i915_gt_timelines *gt;
struct list_head free_link;
+ struct i915_vma *vma;
u64 free_bitmap;
};
-static inline struct i915_timeline_hwsp *
-i915_timeline_hwsp(const struct i915_timeline *tl)
+struct i915_timeline_cacheline {
+ struct i915_active active;
+ struct i915_timeline_hwsp *hwsp;
+ void *vaddr;
+#define CACHELINE_BITS 6
+#define CACHELINE_FREE CACHELINE_BITS
+};
+
+static inline struct drm_i915_private *
+hwsp_to_i915(struct i915_timeline_hwsp *hwsp)
{
- return tl->hwsp_ggtt->private;
+ return container_of(hwsp->gt, struct drm_i915_private, gt.timelines);
}
static struct i915_vma *__hwsp_alloc(struct drm_i915_private *i915)
vma->private = hwsp;
hwsp->vma = vma;
hwsp->free_bitmap = ~0ull;
+ hwsp->gt = gt;
spin_lock(>->hwsp_lock);
list_add(&hwsp->free_link, >->hwsp_free_list);
return hwsp->vma;
}
-static void hwsp_free(struct i915_timeline *timeline)
+static void __idle_hwsp_free(struct i915_timeline_hwsp *hwsp, int cacheline)
{
- struct i915_gt_timelines *gt = &timeline->i915->gt.timelines;
- struct i915_timeline_hwsp *hwsp;
-
- hwsp = i915_timeline_hwsp(timeline);
- if (!hwsp) /* leave global HWSP alone! */
- return;
+ struct i915_gt_timelines *gt = hwsp->gt;
spin_lock(>->hwsp_lock);
if (!hwsp->free_bitmap)
list_add_tail(&hwsp->free_link, >->hwsp_free_list);
- hwsp->free_bitmap |= BIT_ULL(timeline->hwsp_offset / CACHELINE_BYTES);
+ GEM_BUG_ON(cacheline >= BITS_PER_TYPE(hwsp->free_bitmap));
+ hwsp->free_bitmap |= BIT_ULL(cacheline);
/* And if no one is left using it, give the page back to the system */
if (hwsp->free_bitmap == ~0ull) {
spin_unlock(>->hwsp_lock);
}
+static void __idle_cacheline_free(struct i915_timeline_cacheline *cl)
+{
+ GEM_BUG_ON(!i915_active_is_idle(&cl->active));
+
+ i915_gem_object_unpin_map(cl->hwsp->vma->obj);
+ i915_vma_put(cl->hwsp->vma);
+ __idle_hwsp_free(cl->hwsp, ptr_unmask_bits(cl->vaddr, CACHELINE_BITS));
+
+ i915_active_fini(&cl->active);
+ kfree(cl);
+}
+
+static void __cacheline_retire(struct i915_active *active)
+{
+ struct i915_timeline_cacheline *cl =
+ container_of(active, typeof(*cl), active);
+
+ i915_vma_unpin(cl->hwsp->vma);
+ if (ptr_test_bit(cl->vaddr, CACHELINE_FREE))
+ __idle_cacheline_free(cl);
+}
+
+static struct i915_timeline_cacheline *
+cacheline_alloc(struct i915_timeline_hwsp *hwsp, unsigned int cacheline)
+{
+ struct i915_timeline_cacheline *cl;
+ void *vaddr;
+
+ GEM_BUG_ON(cacheline >= BIT(CACHELINE_BITS));
+
+ cl = kmalloc(sizeof(*cl), GFP_KERNEL);
+ if (!cl)
+ return ERR_PTR(-ENOMEM);
+
+ vaddr = i915_gem_object_pin_map(hwsp->vma->obj, I915_MAP_WB);
+ if (IS_ERR(vaddr)) {
+ kfree(cl);
+ return ERR_CAST(vaddr);
+ }
+
+ i915_vma_get(hwsp->vma);
+ cl->hwsp = hwsp;
+ cl->vaddr = page_pack_bits(vaddr, cacheline);
+
+ i915_active_init(hwsp_to_i915(hwsp), &cl->active, __cacheline_retire);
+
+ return cl;
+}
+
+static void cacheline_acquire(struct i915_timeline_cacheline *cl)
+{
+ if (cl && i915_active_acquire(&cl->active))
+ __i915_vma_pin(cl->hwsp->vma);
+}
+
+static void cacheline_release(struct i915_timeline_cacheline *cl)
+{
+ if (cl)
+ i915_active_release(&cl->active);
+}
+
+static void cacheline_free(struct i915_timeline_cacheline *cl)
+{
+ GEM_BUG_ON(ptr_test_bit(cl->vaddr, CACHELINE_FREE));
+ cl->vaddr = ptr_set_bit(cl->vaddr, CACHELINE_FREE);
+
+ if (i915_active_is_idle(&cl->active))
+ __idle_cacheline_free(cl);
+}
+
int i915_timeline_init(struct drm_i915_private *i915,
struct i915_timeline *timeline,
- const char *name,
struct i915_vma *hwsp)
{
void *vaddr;
BUILD_BUG_ON(KSYNCMAP < I915_NUM_ENGINES);
timeline->i915 = i915;
- timeline->name = name;
timeline->pin_count = 0;
timeline->has_initial_breadcrumb = !hwsp;
+ timeline->hwsp_cacheline = NULL;
- timeline->hwsp_offset = I915_GEM_HWS_SEQNO_ADDR;
if (!hwsp) {
+ struct i915_timeline_cacheline *cl;
unsigned int cacheline;
hwsp = hwsp_alloc(timeline, &cacheline);
if (IS_ERR(hwsp))
return PTR_ERR(hwsp);
+ cl = cacheline_alloc(hwsp->private, cacheline);
+ if (IS_ERR(cl)) {
+ __idle_hwsp_free(hwsp->private, cacheline);
+ return PTR_ERR(cl);
+ }
+
+ timeline->hwsp_cacheline = cl;
timeline->hwsp_offset = cacheline * CACHELINE_BYTES;
- }
- timeline->hwsp_ggtt = i915_vma_get(hwsp);
- vaddr = i915_gem_object_pin_map(hwsp->obj, I915_MAP_WB);
- if (IS_ERR(vaddr)) {
- hwsp_free(timeline);
- i915_vma_put(hwsp);
- return PTR_ERR(vaddr);
+ vaddr = page_mask_bits(cl->vaddr);
+ } else {
+ timeline->hwsp_offset = I915_GEM_HWS_SEQNO_ADDR;
+
+ vaddr = i915_gem_object_pin_map(hwsp->obj, I915_MAP_WB);
+ if (IS_ERR(vaddr))
+ return PTR_ERR(vaddr);
}
timeline->hwsp_seqno =
memset(vaddr + timeline->hwsp_offset, 0, CACHELINE_BYTES);
+ timeline->hwsp_ggtt = i915_vma_get(hwsp);
+ GEM_BUG_ON(timeline->hwsp_offset >= hwsp->size);
+
timeline->fence_context = dma_fence_context_alloc(1);
spin_lock_init(&timeline->lock);
+ mutex_init(&timeline->mutex);
- INIT_ACTIVE_REQUEST(&timeline->barrier);
INIT_ACTIVE_REQUEST(&timeline->last_request);
INIT_LIST_HEAD(&timeline->requests);
{
GEM_BUG_ON(timeline->pin_count);
GEM_BUG_ON(!list_empty(&timeline->requests));
- GEM_BUG_ON(i915_active_request_isset(&timeline->barrier));
i915_syncmap_free(&timeline->sync);
- hwsp_free(timeline);
- i915_gem_object_unpin_map(timeline->hwsp_ggtt->obj);
+ if (timeline->hwsp_cacheline)
+ cacheline_free(timeline->hwsp_cacheline);
+ else
+ i915_gem_object_unpin_map(timeline->hwsp_ggtt->obj);
+
i915_vma_put(timeline->hwsp_ggtt);
}
struct i915_timeline *
i915_timeline_create(struct drm_i915_private *i915,
- const char *name,
struct i915_vma *global_hwsp)
{
struct i915_timeline *timeline;
if (!timeline)
return ERR_PTR(-ENOMEM);
- err = i915_timeline_init(i915, timeline, name, global_hwsp);
+ err = i915_timeline_init(i915, timeline, global_hwsp);
if (err) {
kfree(timeline);
return ERR_PTR(err);
i915_ggtt_offset(tl->hwsp_ggtt) +
offset_in_page(tl->hwsp_offset);
+ cacheline_acquire(tl->hwsp_cacheline);
timeline_add_to_active(tl);
return 0;
return err;
}
+static u32 timeline_advance(struct i915_timeline *tl)
+{
+ GEM_BUG_ON(!tl->pin_count);
+ GEM_BUG_ON(tl->seqno & tl->has_initial_breadcrumb);
+
+ return tl->seqno += 1 + tl->has_initial_breadcrumb;
+}
+
+static void timeline_rollback(struct i915_timeline *tl)
+{
+ tl->seqno -= 1 + tl->has_initial_breadcrumb;
+}
+
+static noinline int
+__i915_timeline_get_seqno(struct i915_timeline *tl,
+ struct i915_request *rq,
+ u32 *seqno)
+{
+ struct i915_timeline_cacheline *cl;
+ unsigned int cacheline;
+ struct i915_vma *vma;
+ void *vaddr;
+ int err;
+
+ /*
+ * If there is an outstanding GPU reference to this cacheline,
+ * such as it being sampled by a HW semaphore on another timeline,
+ * we cannot wraparound our seqno value (the HW semaphore does
+ * a strict greater-than-or-equals compare, not i915_seqno_passed).
+ * So if the cacheline is still busy, we must detach ourselves
+ * from it and leave it inflight alongside its users.
+ *
+ * However, if nobody is watching and we can guarantee that nobody
+ * will, we could simply reuse the same cacheline.
+ *
+ * if (i915_active_request_is_signaled(&tl->last_request) &&
+ * i915_active_is_signaled(&tl->hwsp_cacheline->active))
+ * return 0;
+ *
+ * That seems unlikely for a busy timeline that needed to wrap in
+ * the first place, so just replace the cacheline.
+ */
+
+ vma = hwsp_alloc(tl, &cacheline);
+ if (IS_ERR(vma)) {
+ err = PTR_ERR(vma);
+ goto err_rollback;
+ }
+
+ err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL | PIN_HIGH);
+ if (err) {
+ __idle_hwsp_free(vma->private, cacheline);
+ goto err_rollback;
+ }
+
+ cl = cacheline_alloc(vma->private, cacheline);
+ if (IS_ERR(cl)) {
+ err = PTR_ERR(cl);
+ __idle_hwsp_free(vma->private, cacheline);
+ goto err_unpin;
+ }
+ GEM_BUG_ON(cl->hwsp->vma != vma);
+
+ /*
+ * Attach the old cacheline to the current request, so that we only
+ * free it after the current request is retired, which ensures that
+ * all writes into the cacheline from previous requests are complete.
+ */
+ err = i915_active_ref(&tl->hwsp_cacheline->active,
+ tl->fence_context, rq);
+ if (err)
+ goto err_cacheline;
+
+ cacheline_release(tl->hwsp_cacheline); /* ownership now xfered to rq */
+ cacheline_free(tl->hwsp_cacheline);
+
+ i915_vma_unpin(tl->hwsp_ggtt); /* binding kept alive by old cacheline */
+ i915_vma_put(tl->hwsp_ggtt);
+
+ tl->hwsp_ggtt = i915_vma_get(vma);
+
+ vaddr = page_mask_bits(cl->vaddr);
+ tl->hwsp_offset = cacheline * CACHELINE_BYTES;
+ tl->hwsp_seqno =
+ memset(vaddr + tl->hwsp_offset, 0, CACHELINE_BYTES);
+
+ tl->hwsp_offset += i915_ggtt_offset(vma);
+
+ cacheline_acquire(cl);
+ tl->hwsp_cacheline = cl;
+
+ *seqno = timeline_advance(tl);
+ GEM_BUG_ON(i915_seqno_passed(*tl->hwsp_seqno, *seqno));
+ return 0;
+
+err_cacheline:
+ cacheline_free(cl);
+err_unpin:
+ i915_vma_unpin(vma);
+err_rollback:
+ timeline_rollback(tl);
+ return err;
+}
+
+int i915_timeline_get_seqno(struct i915_timeline *tl,
+ struct i915_request *rq,
+ u32 *seqno)
+{
+ *seqno = timeline_advance(tl);
+
+ /* Replace the HWSP on wraparound for HW semaphores */
+ if (unlikely(!*seqno && tl->hwsp_cacheline))
+ return __i915_timeline_get_seqno(tl, rq, seqno);
+
+ return 0;
+}
+
+static int cacheline_ref(struct i915_timeline_cacheline *cl,
+ struct i915_request *rq)
+{
+ return i915_active_ref(&cl->active, rq->fence.context, rq);
+}
+
+int i915_timeline_read_hwsp(struct i915_request *from,
+ struct i915_request *to,
+ u32 *hwsp)
+{
+ struct i915_timeline_cacheline *cl = from->hwsp_cacheline;
+ struct i915_timeline *tl = from->timeline;
+ int err;
+
+ GEM_BUG_ON(to->timeline == tl);
+
+ mutex_lock_nested(&tl->mutex, SINGLE_DEPTH_NESTING);
+ err = i915_request_completed(from);
+ if (!err)
+ err = cacheline_ref(cl, to);
+ if (!err) {
+ if (likely(cl == tl->hwsp_cacheline)) {
+ *hwsp = tl->hwsp_offset;
+ } else { /* across a seqno wrap, recover the original offset */
+ *hwsp = i915_ggtt_offset(cl->hwsp->vma) +
+ ptr_unmask_bits(cl->vaddr, CACHELINE_BITS) *
+ CACHELINE_BYTES;
+ }
+ }
+ mutex_unlock(&tl->mutex);
+
+ return err;
+}
+
void i915_timeline_unpin(struct i915_timeline *tl)
{
GEM_BUG_ON(!tl->pin_count);
return;
timeline_remove_from_active(tl);
+ cacheline_release(tl->hwsp_cacheline);
/*
* Since this timeline is idle, all bariers upon which we were waiting
#ifndef I915_TIMELINE_H
#define I915_TIMELINE_H
-#include <linux/list.h>
-#include <linux/kref.h>
+#include <linux/lockdep.h>
#include "i915_active.h"
-#include "i915_request.h"
#include "i915_syncmap.h"
-#include "i915_utils.h"
-
-struct i915_vma;
-struct i915_timeline_hwsp;
-
-struct i915_timeline {
- u64 fence_context;
- u32 seqno;
-
- spinlock_t lock;
-#define TIMELINE_CLIENT 0 /* default subclass */
-#define TIMELINE_ENGINE 1
-
- unsigned int pin_count;
- const u32 *hwsp_seqno;
- struct i915_vma *hwsp_ggtt;
- u32 hwsp_offset;
-
- bool has_initial_breadcrumb;
-
- /**
- * List of breadcrumbs associated with GPU requests currently
- * outstanding.
- */
- struct list_head requests;
-
- /* Contains an RCU guarded pointer to the last request. No reference is
- * held to the request, users must carefully acquire a reference to
- * the request using i915_active_request_get_request_rcu(), or hold the
- * struct_mutex.
- */
- struct i915_active_request last_request;
-
- /**
- * We track the most recent seqno that we wait on in every context so
- * that we only have to emit a new await and dependency on a more
- * recent sync point. As the contexts may be executed out-of-order, we
- * have to track each individually and can not rely on an absolute
- * global_seqno. When we know that all tracked fences are completed
- * (i.e. when the driver is idle), we know that the syncmap is
- * redundant and we can discard it without loss of generality.
- */
- struct i915_syncmap *sync;
-
- /**
- * Barrier provides the ability to serialize ordering between different
- * timelines.
- *
- * Users can call i915_timeline_set_barrier which will make all
- * subsequent submissions to this timeline be executed only after the
- * barrier has been completed.
- */
- struct i915_active_request barrier;
-
- struct list_head link;
- const char *name;
- struct drm_i915_private *i915;
-
- struct kref kref;
-};
+#include "i915_timeline_types.h"
int i915_timeline_init(struct drm_i915_private *i915,
struct i915_timeline *tl,
- const char *name,
struct i915_vma *hwsp);
void i915_timeline_fini(struct i915_timeline *tl);
struct i915_timeline *
i915_timeline_create(struct drm_i915_private *i915,
- const char *name,
struct i915_vma *global_hwsp);
static inline struct i915_timeline *
}
int i915_timeline_pin(struct i915_timeline *tl);
+int i915_timeline_get_seqno(struct i915_timeline *tl,
+ struct i915_request *rq,
+ u32 *seqno);
void i915_timeline_unpin(struct i915_timeline *tl);
+int i915_timeline_read_hwsp(struct i915_request *from,
+ struct i915_request *until,
+ u32 *hwsp_offset);
+
void i915_timelines_init(struct drm_i915_private *i915);
void i915_timelines_park(struct drm_i915_private *i915);
void i915_timelines_fini(struct drm_i915_private *i915);
-/**
- * i915_timeline_set_barrier - orders submission between different timelines
- * @timeline: timeline to set the barrier on
- * @rq: request after which new submissions can proceed
- *
- * Sets the passed in request as the serialization point for all subsequent
- * submissions on @timeline. Subsequent requests will not be submitted to GPU
- * until the barrier has been completed.
- */
-static inline int
-i915_timeline_set_barrier(struct i915_timeline *tl, struct i915_request *rq)
-{
- return i915_active_request_set(&tl->barrier, rq);
-}
-
#endif
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2016 Intel Corporation
+ */
+
+#ifndef __I915_TIMELINE_TYPES_H__
+#define __I915_TIMELINE_TYPES_H__
+
+#include <linux/list.h>
+#include <linux/kref.h>
+#include <linux/mutex.h>
+#include <linux/types.h>
+
+#include "i915_active_types.h"
+
+struct drm_i915_private;
+struct i915_vma;
+struct i915_timeline_cacheline;
+struct i915_syncmap;
+
+struct i915_timeline {
+ u64 fence_context;
+ u32 seqno;
+
+ spinlock_t lock;
+#define TIMELINE_CLIENT 0 /* default subclass */
+#define TIMELINE_ENGINE 1
+ struct mutex mutex; /* protects the flow of requests */
+
+ unsigned int pin_count;
+ const u32 *hwsp_seqno;
+ struct i915_vma *hwsp_ggtt;
+ u32 hwsp_offset;
+
+ struct i915_timeline_cacheline *hwsp_cacheline;
+
+ bool has_initial_breadcrumb;
+
+ /**
+ * List of breadcrumbs associated with GPU requests currently
+ * outstanding.
+ */
+ struct list_head requests;
+
+ /* Contains an RCU guarded pointer to the last request. No reference is
+ * held to the request, users must carefully acquire a reference to
+ * the request using i915_active_request_get_request_rcu(), or hold the
+ * struct_mutex.
+ */
+ struct i915_active_request last_request;
+
+ /**
+ * We track the most recent seqno that we wait on in every context so
+ * that we only have to emit a new await and dependency on a more
+ * recent sync point. As the contexts may be executed out-of-order, we
+ * have to track each individually and can not rely on an absolute
+ * global_seqno. When we know that all tracked fences are completed
+ * (i.e. when the driver is idle), we know that the syncmap is
+ * redundant and we can discard it without loss of generality.
+ */
+ struct i915_syncmap *sync;
+
+ struct list_head link;
+ struct drm_i915_private *i915;
+
+ struct kref kref;
+};
+
+#endif /* __I915_TIMELINE_TYPES_H__ */
/* watermark/fifo updates */
+TRACE_EVENT(intel_pipe_enable,
+ TP_PROTO(struct drm_i915_private *dev_priv, enum pipe pipe),
+ TP_ARGS(dev_priv, pipe),
+
+ TP_STRUCT__entry(
+ __array(u32, frame, 3)
+ __array(u32, scanline, 3)
+ __field(enum pipe, pipe)
+ ),
+
+ TP_fast_assign(
+ enum pipe _pipe;
+ for_each_pipe(dev_priv, _pipe) {
+ __entry->frame[_pipe] =
+ dev_priv->drm.driver->get_vblank_counter(&dev_priv->drm, _pipe);
+ __entry->scanline[_pipe] =
+ intel_get_crtc_scanline(intel_get_crtc_for_pipe(dev_priv, _pipe));
+ }
+ __entry->pipe = pipe;
+ ),
+
+ TP_printk("pipe %c enable, pipe A: frame=%u, scanline=%u, pipe B: frame=%u, scanline=%u, pipe C: frame=%u, scanline=%u",
+ pipe_name(__entry->pipe),
+ __entry->frame[PIPE_A], __entry->scanline[PIPE_A],
+ __entry->frame[PIPE_B], __entry->scanline[PIPE_B],
+ __entry->frame[PIPE_C], __entry->scanline[PIPE_C])
+);
+
+TRACE_EVENT(intel_pipe_disable,
+ TP_PROTO(struct drm_i915_private *dev_priv, enum pipe pipe),
+ TP_ARGS(dev_priv, pipe),
+
+ TP_STRUCT__entry(
+ __array(u32, frame, 3)
+ __array(u32, scanline, 3)
+ __field(enum pipe, pipe)
+ ),
+
+ TP_fast_assign(
+ enum pipe _pipe;
+ for_each_pipe(dev_priv, _pipe) {
+ __entry->frame[_pipe] =
+ dev_priv->drm.driver->get_vblank_counter(&dev_priv->drm, _pipe);
+ __entry->scanline[_pipe] =
+ intel_get_crtc_scanline(intel_get_crtc_for_pipe(dev_priv, _pipe));
+ }
+ __entry->pipe = pipe;
+ ),
+
+ TP_printk("pipe %c disable, pipe A: frame=%u, scanline=%u, pipe B: frame=%u, scanline=%u, pipe C: frame=%u, scanline=%u",
+ pipe_name(__entry->pipe),
+ __entry->frame[PIPE_A], __entry->scanline[PIPE_A],
+ __entry->frame[PIPE_B], __entry->scanline[PIPE_B],
+ __entry->frame[PIPE_C], __entry->scanline[PIPE_C])
+);
+
+TRACE_EVENT(intel_pipe_crc,
+ TP_PROTO(struct intel_crtc *crtc, const u32 *crcs),
+ TP_ARGS(crtc, crcs),
+
+ TP_STRUCT__entry(
+ __field(enum pipe, pipe)
+ __field(u32, frame)
+ __field(u32, scanline)
+ __array(u32, crcs, 5)
+ ),
+
+ TP_fast_assign(
+ __entry->pipe = crtc->pipe;
+ __entry->frame = crtc->base.dev->driver->get_vblank_counter(crtc->base.dev,
+ crtc->pipe);
+ __entry->scanline = intel_get_crtc_scanline(crtc);
+ memcpy(__entry->crcs, crcs, sizeof(__entry->crcs));
+ ),
+
+ TP_printk("pipe %c, frame=%u, scanline=%u crc=%08x %08x %08x %08x %08x",
+ pipe_name(__entry->pipe), __entry->frame, __entry->scanline,
+ __entry->crcs[0], __entry->crcs[1], __entry->crcs[2],
+ __entry->crcs[3], __entry->crcs[4])
+);
+
TRACE_EVENT(intel_cpu_fifo_underrun,
TP_PROTO(struct drm_i915_private *dev_priv, enum pipe pipe),
TP_ARGS(dev_priv, pipe),
__field(u16, class)
__field(u16, instance)
__field(u32, seqno)
- __field(u32, global)
),
TP_fast_assign(
__entry->instance = rq->engine->instance;
__entry->ctx = rq->fence.context;
__entry->seqno = rq->fence.seqno;
- __entry->global = rq->global_seqno;
),
- TP_printk("dev=%u, engine=%u:%u, hw_id=%u, ctx=%llu, seqno=%u, global=%u",
+ TP_printk("dev=%u, engine=%u:%u, hw_id=%u, ctx=%llu, seqno=%u",
__entry->dev, __entry->class, __entry->instance,
- __entry->hw_id, __entry->ctx, __entry->seqno,
- __entry->global)
+ __entry->hw_id, __entry->ctx, __entry->seqno)
);
DEFINE_EVENT(i915_request, i915_request_add,
__field(u16, class)
__field(u16, instance)
__field(u32, seqno)
- __field(u32, global_seqno)
__field(u32, port)
__field(u32, prio)
),
__entry->instance = rq->engine->instance;
__entry->ctx = rq->fence.context;
__entry->seqno = rq->fence.seqno;
- __entry->global_seqno = rq->global_seqno;
__entry->prio = rq->sched.attr.priority;
__entry->port = port;
),
- TP_printk("dev=%u, engine=%u:%u, hw_id=%u, ctx=%llu, seqno=%u, prio=%u, global=%u, port=%u",
+ TP_printk("dev=%u, engine=%u:%u, hw_id=%u, ctx=%llu, seqno=%u, prio=%u, port=%u",
__entry->dev, __entry->class, __entry->instance,
__entry->hw_id, __entry->ctx, __entry->seqno,
- __entry->prio, __entry->global_seqno, __entry->port)
+ __entry->prio, __entry->port)
);
TRACE_EVENT(i915_request_out,
__field(u16, class)
__field(u16, instance)
__field(u32, seqno)
- __field(u32, global_seqno)
__field(u32, completed)
),
__entry->instance = rq->engine->instance;
__entry->ctx = rq->fence.context;
__entry->seqno = rq->fence.seqno;
- __entry->global_seqno = rq->global_seqno;
__entry->completed = i915_request_completed(rq);
),
- TP_printk("dev=%u, engine=%u:%u, hw_id=%u, ctx=%llu, seqno=%u, global=%u, completed?=%u",
+ TP_printk("dev=%u, engine=%u:%u, hw_id=%u, ctx=%llu, seqno=%u, completed?=%u",
__entry->dev, __entry->class, __entry->instance,
__entry->hw_id, __entry->ctx, __entry->seqno,
- __entry->global_seqno, __entry->completed)
+ __entry->completed)
);
#else
__field(u16, class)
__field(u16, instance)
__field(u32, seqno)
- __field(u32, global)
__field(unsigned int, flags)
),
__entry->instance = rq->engine->instance;
__entry->ctx = rq->fence.context;
__entry->seqno = rq->fence.seqno;
- __entry->global = rq->global_seqno;
__entry->flags = flags;
),
- TP_printk("dev=%u, engine=%u:%u, hw_id=%u, ctx=%llu, seqno=%u, global=%u, blocking=%u, flags=0x%x",
+ TP_printk("dev=%u, engine=%u:%u, hw_id=%u, ctx=%llu, seqno=%u, blocking=%u, flags=0x%x",
__entry->dev, __entry->class, __entry->instance,
__entry->hw_id, __entry->ctx, __entry->seqno,
- __entry->global, !!(__entry->flags & I915_WAIT_LOCKED),
+ !!(__entry->flags & I915_WAIT_LOCKED),
__entry->flags)
);
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2018 Intel Corporation
+ */
+
+#include <linux/nospec.h>
+#include <linux/sched/signal.h>
+#include <linux/uaccess.h>
+
+#include <uapi/drm/i915_drm.h>
+
+#include "i915_user_extensions.h"
+#include "i915_utils.h"
+
+int i915_user_extensions(struct i915_user_extension __user *ext,
+ const i915_user_extension_fn *tbl,
+ unsigned int count,
+ void *data)
+{
+ unsigned int stackdepth = 512;
+
+ while (ext) {
+ int i, err;
+ u32 name;
+ u64 next;
+
+ if (!stackdepth--) /* recursion vs useful flexibility */
+ return -E2BIG;
+
+ err = check_user_mbz(&ext->flags);
+ if (err)
+ return err;
+
+ for (i = 0; i < ARRAY_SIZE(ext->rsvd); i++) {
+ err = check_user_mbz(&ext->rsvd[i]);
+ if (err)
+ return err;
+ }
+
+ if (get_user(name, &ext->name))
+ return -EFAULT;
+
+ err = -EINVAL;
+ if (name < count) {
+ name = array_index_nospec(name, count);
+ if (tbl[name])
+ err = tbl[name](ext, data);
+ }
+ if (err)
+ return err;
+
+ if (get_user(next, &ext->next_extension) ||
+ overflows_type(next, ext))
+ return -EFAULT;
+
+ ext = u64_to_user_ptr(next);
+ }
+
+ return 0;
+}
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2018 Intel Corporation
+ */
+
+#ifndef I915_USER_EXTENSIONS_H
+#define I915_USER_EXTENSIONS_H
+
+struct i915_user_extension;
+
+typedef int (*i915_user_extension_fn)(struct i915_user_extension __user *ext,
+ void *data);
+
+int i915_user_extensions(struct i915_user_extension __user *ext,
+ const i915_user_extension_fn *tbl,
+ unsigned int count,
+ void *data);
+
+#endif /* I915_USER_EXTENSIONS_H */
__T; \
})
+/*
+ * container_of_user: Extract the superclass from a pointer to a member.
+ *
+ * Exactly like container_of() with the exception that it plays nicely
+ * with sparse for __user @ptr.
+ */
+#define container_of_user(ptr, type, member) ({ \
+ void __user *__mptr = (void __user *)(ptr); \
+ BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
+ !__same_type(*(ptr), void), \
+ "pointer type mismatch in container_of()"); \
+ ((type __user *)(__mptr - offsetof(type, member))); })
+
+/*
+ * check_user_mbz: Check that a user value exists and is zero
+ *
+ * Frequently in our uABI we reserve space for future extensions, and
+ * two ensure that userspace is prepared we enforce that space must
+ * be zero. (Then any future extension can safely assume a default value
+ * of 0.)
+ *
+ * check_user_mbz() combines checking that the user pointer is accessible
+ * and that the contained value is zero.
+ *
+ * Returns: -EFAULT if not accessible, -EINVAL if !zero, or 0 on success.
+ */
+#define check_user_mbz(U) ({ \
+ typeof(*(U)) mbz__; \
+ get_user(mbz__, (U)) ? -EFAULT : mbz__ ? -EINVAL : 0; \
+})
+
static inline u64 ptr_to_u64(const void *ptr)
{
return (uintptr_t)ptr;
*/
void i915_check_vgpu(struct drm_i915_private *dev_priv)
{
+ struct intel_uncore *uncore = &dev_priv->uncore;
u64 magic;
u16 version_major;
BUILD_BUG_ON(sizeof(struct vgt_if) != VGT_PVINFO_SIZE);
- magic = __raw_i915_read64(dev_priv, vgtif_reg(magic));
+ magic = __raw_uncore_read64(uncore, vgtif_reg(magic));
if (magic != VGT_MAGIC)
return;
- version_major = __raw_i915_read16(dev_priv, vgtif_reg(version_major));
+ version_major = __raw_uncore_read16(uncore, vgtif_reg(version_major));
if (version_major < VGT_VERSION_MAJOR) {
DRM_INFO("VGT interface version mismatch!\n");
return;
}
- dev_priv->vgpu.caps = __raw_i915_read32(dev_priv, vgtif_reg(vgt_caps));
+ dev_priv->vgpu.caps = __raw_uncore_read32(uncore, vgtif_reg(vgt_caps));
dev_priv->vgpu.active = true;
DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");
}
-bool intel_vgpu_has_full_48bit_ppgtt(struct drm_i915_private *dev_priv)
+bool intel_vgpu_has_full_ppgtt(struct drm_i915_private *dev_priv)
{
- return dev_priv->vgpu.caps & VGT_CAPS_FULL_48BIT_PPGTT;
+ return dev_priv->vgpu.caps & VGT_CAPS_FULL_PPGTT;
}
struct _balloon_info_ {
void i915_check_vgpu(struct drm_i915_private *dev_priv);
-bool intel_vgpu_has_full_48bit_ppgtt(struct drm_i915_private *dev_priv);
+bool intel_vgpu_has_full_ppgtt(struct drm_i915_private *dev_priv);
static inline bool
intel_vgpu_has_hwsp_emulation(struct drm_i915_private *dev_priv)
#include "i915_vma.h"
#include "i915_drv.h"
+#include "i915_globals.h"
#include "intel_ringbuffer.h"
#include "intel_frontbuffer.h"
#include <drm/drm_gem.h>
+static struct i915_global_vma {
+ struct i915_global base;
+ struct kmem_cache *slab_vmas;
+} global;
+
+struct i915_vma *i915_vma_alloc(void)
+{
+ return kmem_cache_zalloc(global.slab_vmas, GFP_KERNEL);
+}
+
+void i915_vma_free(struct i915_vma *vma)
+{
+ return kmem_cache_free(global.slab_vmas, vma);
+}
+
#if IS_ENABLED(CONFIG_DRM_I915_ERRLOG_GEM) && IS_ENABLED(CONFIG_DRM_DEBUG_MM)
#include <linux/stackdepot.h>
/* The aliasing_ppgtt should never be used directly! */
GEM_BUG_ON(vm == &vm->i915->mm.aliasing_ppgtt->vm);
- vma = kmem_cache_zalloc(vm->i915->vmas, GFP_KERNEL);
+ vma = i915_vma_alloc();
if (vma == NULL)
return ERR_PTR(-ENOMEM);
cmp = i915_vma_compare(pos, vm, view);
if (cmp == 0) {
spin_unlock(&obj->vma.lock);
- kmem_cache_free(vm->i915->vmas, vma);
+ i915_vma_free(vma);
return pos;
}
return vma;
err_vma:
- kmem_cache_free(vm->i915->vmas, vma);
+ i915_vma_free(vma);
return ERR_PTR(-E2BIG);
}
static void __i915_vma_destroy(struct i915_vma *vma)
{
- struct drm_i915_private *i915 = vma->vm->i915;
-
GEM_BUG_ON(vma->node.allocated);
GEM_BUG_ON(vma->fence);
i915_active_fini(&vma->active);
- kmem_cache_free(i915->vmas, vma);
+ i915_vma_free(vma);
}
void i915_vma_destroy(struct i915_vma *vma)
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftests/i915_vma.c"
#endif
+
+static void i915_global_vma_shrink(void)
+{
+ kmem_cache_shrink(global.slab_vmas);
+}
+
+static void i915_global_vma_exit(void)
+{
+ kmem_cache_destroy(global.slab_vmas);
+}
+
+static struct i915_global_vma global = { {
+ .shrink = i915_global_vma_shrink,
+ .exit = i915_global_vma_exit,
+} };
+
+int __init i915_global_vma_init(void)
+{
+ global.slab_vmas = KMEM_CACHE(i915_vma, SLAB_HWCACHE_ALIGN);
+ if (!global.slab_vmas)
+ return -ENOMEM;
+
+ i915_global_register(&global.base);
+ return 0;
+}
list_for_each_entry(V, &(OBJ)->vma.list, obj_link) \
for_each_until(!i915_vma_is_ggtt(V))
+struct i915_vma *i915_vma_alloc(void);
+void i915_vma_free(struct i915_vma *vma);
+
#endif
* Jani Nikula <jani.nikula@intel.com>
*/
-#include <drm/drm_mipi_dsi.h>
#include <drm/drm_atomic_helper.h>
+#include <drm/drm_mipi_dsi.h>
+
+#include "intel_connector.h"
+#include "intel_ddi.h"
#include "intel_dsi.h"
+#include "intel_panel.h"
static inline int header_credits_available(struct drm_i915_private *dev_priv,
enum transcoder dsi_trans)
for (lane = 0; lane <= 3; lane++) {
/* Bspec: must not use GRP register for write */
- tmp = I915_READ(ICL_PORT_TX_DW4_LN(port, lane));
+ tmp = I915_READ(ICL_PORT_TX_DW4_LN(lane, port));
tmp &= ~(POST_CURSOR_1_MASK | POST_CURSOR_2_MASK |
CURSOR_COEFF_MASK);
tmp |= POST_CURSOR_1(0x0);
tmp |= POST_CURSOR_2(0x0);
tmp |= CURSOR_COEFF(0x3f);
- I915_WRITE(ICL_PORT_TX_DW4_LN(port, lane), tmp);
+ I915_WRITE(ICL_PORT_TX_DW4_LN(lane, port), tmp);
}
}
}
tmp &= ~LOADGEN_SELECT;
I915_WRITE(ICL_PORT_TX_DW4_AUX(port), tmp);
for (lane = 0; lane <= 3; lane++) {
- tmp = I915_READ(ICL_PORT_TX_DW4_LN(port, lane));
+ tmp = I915_READ(ICL_PORT_TX_DW4_LN(lane, port));
tmp &= ~LOADGEN_SELECT;
if (lane != 2)
tmp |= LOADGEN_SELECT;
- I915_WRITE(ICL_PORT_TX_DW4_LN(port, lane), tmp);
+ I915_WRITE(ICL_PORT_TX_DW4_LN(lane, port), tmp);
}
}
I915_WRITE(PIPECONF(dsi_trans), tmp);
/* wait for transcoder to be enabled */
- if (intel_wait_for_register(dev_priv, PIPECONF(dsi_trans),
+ if (intel_wait_for_register(&dev_priv->uncore,
+ PIPECONF(dsi_trans),
I965_PIPECONF_ACTIVE,
I965_PIPECONF_ACTIVE, 10))
DRM_ERROR("DSI transcoder not enabled\n");
I915_WRITE(PIPECONF(dsi_trans), tmp);
/* wait for transcoder to be disabled */
- if (intel_wait_for_register(dev_priv, PIPECONF(dsi_trans),
+ if (intel_wait_for_register(&dev_priv->uncore,
+ PIPECONF(dsi_trans),
I965_PIPECONF_ACTIVE, 0, 50))
DRM_ERROR("DSI trancoder not disabled\n");
}
intel_wakeref_t wakeref;
wakeref = fetch_and_zero(&intel_dsi->io_wakeref[port]);
- if (wakeref) {
- intel_display_power_put(dev_priv,
- port == PORT_A ?
- POWER_DOMAIN_PORT_DDI_A_IO :
- POWER_DOMAIN_PORT_DDI_B_IO,
- wakeref);
- }
+ intel_display_power_put(dev_priv,
+ port == PORT_A ?
+ POWER_DOMAIN_PORT_DDI_A_IO :
+ POWER_DOMAIN_PORT_DDI_B_IO,
+ wakeref);
}
/* set mode to DDI */
{
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
struct intel_dsi *intel_dsi = enc_to_intel_dsi(&encoder->base);
- u32 pll_id;
/* FIXME: adapt icl_ddi_clock_get() for DSI and use that? */
- pll_id = intel_get_shared_dpll_id(dev_priv, pipe_config->shared_dpll);
- pipe_config->port_clock = cnl_calc_wrpll_link(dev_priv, pll_id);
+ pipe_config->port_clock =
+ cnl_calc_wrpll_link(dev_priv, &pipe_config->dpll_hw_state);
pipe_config->base.adjusted_mode.crtc_clock = intel_dsi->pclk;
pipe_config->output_types |= BIT(INTEL_OUTPUT_DSI);
}
struct intel_encoder *encoder;
struct intel_connector *intel_connector;
struct drm_connector *connector;
- struct drm_display_mode *scan, *fixed_mode = NULL;
+ struct drm_display_mode *fixed_mode;
enum port port;
if (!intel_bios_is_dsi_present(dev_priv, &port))
/* attach connector to encoder */
intel_connector_attach_encoder(intel_connector, encoder);
- /* fill mode info from VBT */
mutex_lock(&dev->mode_config.mutex);
- intel_dsi_vbt_get_modes(intel_dsi);
- list_for_each_entry(scan, &connector->probed_modes, head) {
- if (scan->type & DRM_MODE_TYPE_PREFERRED) {
- fixed_mode = drm_mode_duplicate(dev, scan);
- break;
- }
- }
+ fixed_mode = intel_panel_vbt_fixed_mode(intel_connector);
mutex_unlock(&dev->mode_config.mutex);
if (!fixed_mode) {
goto err;
}
- connector->display_info.width_mm = fixed_mode->width_mm;
- connector->display_info.height_mm = fixed_mode->height_mm;
intel_panel_init(&intel_connector->panel, fixed_mode, NULL);
intel_panel_setup_backlight(connector, INVALID_PIPE);
-
if (dev_priv->vbt.dsi.config->dual_link)
intel_dsi->ports = BIT(PORT_A) | BIT(PORT_B);
else
#include <drm/drm_plane_helper.h>
#include "intel_drv.h"
+#include "intel_hdcp.h"
+#include "intel_sprite.h"
/**
* intel_digital_connector_atomic_get_property - hook for connector->atomic_get_property.
*/
if (new_conn_state->force_audio != old_conn_state->force_audio ||
new_conn_state->broadcast_rgb != old_conn_state->broadcast_rgb ||
+ new_conn_state->base.colorspace != old_conn_state->base.colorspace ||
new_conn_state->base.picture_aspect_ratio != old_conn_state->base.picture_aspect_ratio ||
new_conn_state->base.content_type != old_conn_state->base.content_type ||
new_conn_state->base.scaling_mode != old_conn_state->base.scaling_mode)
if (plane_state && plane_state->base.fb &&
plane_state->base.fb->format->is_yuv &&
plane_state->base.fb->format->num_planes > 1) {
+ struct intel_plane *plane = to_intel_plane(plane_state->base.plane);
if (IS_GEN(dev_priv, 9) &&
!IS_GEMINILAKE(dev_priv)) {
mode = SKL_PS_SCALER_MODE_NV12;
- } else if (icl_is_hdr_plane(to_intel_plane(plane_state->base.plane))) {
+ } else if (icl_is_hdr_plane(dev_priv, plane->id)) {
/*
* On gen11+'s HDR planes we only use the scaler for
* scaling. They have a dedicated chroma upsampler, so
#include <drm/drm_fourcc.h>
#include <drm/drm_plane_helper.h>
+#include "intel_atomic_plane.h"
#include "intel_drv.h"
+#include "intel_pm.h"
+#include "intel_sprite.h"
struct intel_plane *intel_plane_alloc(void)
{
new_crtc_state->active_planes &= ~BIT(plane->id);
new_crtc_state->nv12_planes &= ~BIT(plane->id);
+ new_crtc_state->c8_planes &= ~BIT(plane->id);
new_plane_state->base.visible = false;
if (!new_plane_state->base.crtc && !old_plane_state->base.crtc)
new_crtc_state->active_planes |= BIT(plane->id);
if (new_plane_state->base.visible &&
- new_plane_state->base.fb->format->format == DRM_FORMAT_NV12)
+ is_planar_yuv_format(new_plane_state->base.fb->format->format))
new_crtc_state->nv12_planes |= BIT(plane->id);
+ if (new_plane_state->base.visible &&
+ new_plane_state->base.fb->format->format == DRM_FORMAT_C8)
+ new_crtc_state->c8_planes |= BIT(plane->id);
+
if (new_plane_state->base.visible || old_plane_state->base.visible)
new_crtc_state->update_planes |= BIT(plane->id);
return NULL;
}
+void intel_update_plane(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state,
+ const struct intel_plane_state *plane_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+
+ trace_intel_update_plane(&plane->base, crtc);
+ plane->update_plane(plane, crtc_state, plane_state);
+}
+
+void intel_update_slave(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state,
+ const struct intel_plane_state *plane_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+
+ trace_intel_update_plane(&plane->base, crtc);
+ plane->update_slave(plane, crtc_state, plane_state);
+}
+
+void intel_disable_plane(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+
+ trace_intel_disable_plane(&plane->base, crtc);
+ plane->disable_plane(plane, crtc_state);
+}
+
void skl_update_planes_on_crtc(struct intel_atomic_state *state,
struct intel_crtc *crtc)
{
intel_atomic_get_new_plane_state(state, plane);
if (new_plane_state->base.visible) {
- trace_intel_update_plane(&plane->base, crtc);
- plane->update_plane(plane, new_crtc_state, new_plane_state);
+ intel_update_plane(plane, new_crtc_state, new_plane_state);
} else if (new_plane_state->slave) {
struct intel_plane *master =
new_plane_state->linked_plane;
new_plane_state =
intel_atomic_get_new_plane_state(state, master);
- trace_intel_update_plane(&plane->base, crtc);
- plane->update_slave(plane, new_crtc_state, new_plane_state);
+ intel_update_slave(plane, new_crtc_state, new_plane_state);
} else {
- trace_intel_disable_plane(&plane->base, crtc);
- plane->disable_plane(plane, new_crtc_state);
+ intel_disable_plane(plane, new_crtc_state);
}
}
}
!(update_mask & BIT(plane->id)))
continue;
- if (new_plane_state->base.visible) {
- trace_intel_update_plane(&plane->base, crtc);
- plane->update_plane(plane, new_crtc_state, new_plane_state);
- } else {
- trace_intel_disable_plane(&plane->base, crtc);
- plane->disable_plane(plane, new_crtc_state);
- }
+ if (new_plane_state->base.visible)
+ intel_update_plane(plane, new_crtc_state, new_plane_state);
+ else
+ intel_disable_plane(plane, new_crtc_state);
}
}
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_ATOMIC_PLANE_H__
+#define __INTEL_ATOMIC_PLANE_H__
+
+struct drm_plane;
+struct intel_atomic_state;
+struct intel_crtc;
+struct intel_crtc_state;
+struct intel_plane;
+struct intel_plane_state;
+
+extern const struct drm_plane_helper_funcs intel_plane_helper_funcs;
+
+void intel_update_plane(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state,
+ const struct intel_plane_state *plane_state);
+void intel_update_slave(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state,
+ const struct intel_plane_state *plane_state);
+void intel_disable_plane(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state);
+struct intel_plane *intel_plane_alloc(void);
+void intel_plane_free(struct intel_plane *plane);
+struct drm_plane_state *intel_plane_duplicate_state(struct drm_plane *plane);
+void intel_plane_destroy_state(struct drm_plane *plane,
+ struct drm_plane_state *state);
+void skl_update_planes_on_crtc(struct intel_atomic_state *state,
+ struct intel_crtc *crtc);
+void i9xx_update_planes_on_crtc(struct intel_atomic_state *state,
+ struct intel_crtc *crtc);
+int intel_plane_atomic_check_with_state(const struct intel_crtc_state *old_crtc_state,
+ struct intel_crtc_state *crtc_state,
+ const struct intel_plane_state *old_plane_state,
+ struct intel_plane_state *intel_state);
+
+#endif /* __INTEL_ATOMIC_PLANE_H__ */
* DEALINGS IN THE SOFTWARE.
*/
-#include <linux/kernel.h>
#include <linux/component.h>
+#include <linux/kernel.h>
+
+#include <drm/drm_edid.h>
#include <drm/i915_component.h>
#include <drm/intel_lpe_audio.h>
-#include "intel_drv.h"
-#include <drm/drm_edid.h>
#include "i915_drv.h"
+#include "intel_audio.h"
+#include "intel_drv.h"
/**
* DOC: High Definition Audio over HDMI and Display Port
}
}
-static void i915_audio_component_get_power(struct device *kdev)
+static void glk_force_audio_cdclk(struct drm_i915_private *dev_priv,
+ bool enable)
{
- intel_display_power_get(kdev_to_i915(kdev), POWER_DOMAIN_AUDIO);
+ struct drm_modeset_acquire_ctx ctx;
+ struct drm_atomic_state *state;
+ int ret;
+
+ drm_modeset_acquire_init(&ctx, 0);
+ state = drm_atomic_state_alloc(&dev_priv->drm);
+ if (WARN_ON(!state))
+ return;
+
+ state->acquire_ctx = &ctx;
+
+retry:
+ to_intel_atomic_state(state)->cdclk.force_min_cdclk_changed = true;
+ to_intel_atomic_state(state)->cdclk.force_min_cdclk =
+ enable ? 2 * 96000 : 0;
+
+ /*
+ * Protects dev_priv->cdclk.force_min_cdclk
+ * Need to lock this here in case we have no active pipes
+ * and thus wouldn't lock it during the commit otherwise.
+ */
+ ret = drm_modeset_lock(&dev_priv->drm.mode_config.connection_mutex,
+ &ctx);
+ if (!ret)
+ ret = drm_atomic_commit(state);
+
+ if (ret == -EDEADLK) {
+ drm_atomic_state_clear(state);
+ drm_modeset_backoff(&ctx);
+ goto retry;
+ }
+
+ WARN_ON(ret);
+
+ drm_atomic_state_put(state);
+
+ drm_modeset_drop_locks(&ctx);
+ drm_modeset_acquire_fini(&ctx);
}
-static void i915_audio_component_put_power(struct device *kdev)
+static unsigned long i915_audio_component_get_power(struct device *kdev)
{
- intel_display_power_put_unchecked(kdev_to_i915(kdev),
- POWER_DOMAIN_AUDIO);
+ struct drm_i915_private *dev_priv = kdev_to_i915(kdev);
+ intel_wakeref_t ret;
+
+ /* Catch potential impedance mismatches before they occur! */
+ BUILD_BUG_ON(sizeof(intel_wakeref_t) > sizeof(unsigned long));
+
+ ret = intel_display_power_get(dev_priv, POWER_DOMAIN_AUDIO);
+
+ /* Force CDCLK to 2*BCLK as long as we need audio to be powered. */
+ if (dev_priv->audio_power_refcount++ == 0)
+ if (IS_CANNONLAKE(dev_priv) || IS_GEMINILAKE(dev_priv))
+ glk_force_audio_cdclk(dev_priv, true);
+
+ return ret;
+}
+
+static void i915_audio_component_put_power(struct device *kdev,
+ unsigned long cookie)
+{
+ struct drm_i915_private *dev_priv = kdev_to_i915(kdev);
+
+ /* Stop forcing CDCLK to 2*BCLK if no need for audio to be powered. */
+ if (--dev_priv->audio_power_refcount == 0)
+ if (IS_CANNONLAKE(dev_priv) || IS_GEMINILAKE(dev_priv))
+ glk_force_audio_cdclk(dev_priv, false);
+
+ intel_display_power_put(dev_priv, POWER_DOMAIN_AUDIO, cookie);
}
static void i915_audio_component_codec_wake_override(struct device *kdev,
bool enable)
{
struct drm_i915_private *dev_priv = kdev_to_i915(kdev);
+ unsigned long cookie;
u32 tmp;
if (!IS_GEN(dev_priv, 9))
return;
- i915_audio_component_get_power(kdev);
+ cookie = i915_audio_component_get_power(kdev);
/*
* Enable/disable generating the codec wake signal, overriding the
usleep_range(1000, 1500);
}
- i915_audio_component_put_power(kdev);
+ i915_audio_component_put_power(kdev, cookie);
}
/* Get CDCLK in kHz */
struct i915_audio_component *acomp = dev_priv->audio_component;
struct intel_encoder *encoder;
struct intel_crtc *crtc;
+ unsigned long cookie;
int err = 0;
if (!HAS_DDI(dev_priv))
return 0;
- i915_audio_component_get_power(kdev);
+ cookie = i915_audio_component_get_power(kdev);
mutex_lock(&dev_priv->av_mutex);
/* 1. get the pipe */
unlock:
mutex_unlock(&dev_priv->av_mutex);
- i915_audio_component_put_power(kdev);
+ i915_audio_component_put_power(kdev, cookie);
return err;
}
* We ignore any error during registration and continue with reduced
* functionality (i.e. without HDMI audio).
*/
-void i915_audio_component_init(struct drm_i915_private *dev_priv)
+static void i915_audio_component_init(struct drm_i915_private *dev_priv)
{
int ret;
* Deregisters the audio component, breaking any existing binding to the
* corresponding snd_hda_intel driver's master component.
*/
-void i915_audio_component_cleanup(struct drm_i915_private *dev_priv)
+static void i915_audio_component_cleanup(struct drm_i915_private *dev_priv)
{
if (!dev_priv->audio_component_registered)
return;
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_AUDIO_H__
+#define __INTEL_AUDIO_H__
+
+struct drm_connector_state;
+struct drm_i915_private;
+struct intel_crtc_state;
+struct intel_encoder;
+
+void intel_init_audio_hooks(struct drm_i915_private *dev_priv);
+void intel_audio_codec_enable(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ const struct drm_connector_state *conn_state);
+void intel_audio_codec_disable(struct intel_encoder *encoder,
+ const struct intel_crtc_state *old_crtc_state,
+ const struct drm_connector_state *old_conn_state);
+void intel_audio_init(struct drm_i915_private *dev_priv);
+void intel_audio_deinit(struct drm_i915_private *dev_priv);
+
+#endif /* __INTEL_AUDIO_H__ */
dev_priv->vbt.psr.tp1_wakeup_time_us = psr_table->tp1_wakeup_time * 100;
dev_priv->vbt.psr.tp2_tp3_wakeup_time_us = psr_table->tp2_tp3_wakeup_time * 100;
}
+
+ if (bdb->version >= 226) {
+ u32 wakeup_time = psr_table->psr2_tp2_tp3_wakeup_time;
+
+ wakeup_time = (wakeup_time >> (2 * panel_type)) & 0x3;
+ switch (wakeup_time) {
+ case 0:
+ wakeup_time = 500;
+ break;
+ case 1:
+ wakeup_time = 100;
+ break;
+ case 3:
+ wakeup_time = 50;
+ break;
+ default:
+ case 2:
+ wakeup_time = 2500;
+ break;
+ }
+ dev_priv->vbt.psr.psr2_tp2_tp3_wakeup_time_us = wakeup_time;
+ } else {
+ /* Reusing PSR1 wakeup time for PSR2 in older VBTs */
+ dev_priv->vbt.psr.psr2_tp2_tp3_wakeup_time_us = dev_priv->vbt.psr.tp2_tp3_wakeup_time_us;
+ }
}
static void parse_dsi_backlight_ports(struct drm_i915_private *dev_priv,
if (!info->alternate_ddc_pin)
return;
- for_each_port_masked(p, (1 << port) - 1) {
+ for (p = PORT_A; p < I915_MAX_PORTS; p++) {
struct ddi_vbt_port_info *i = &dev_priv->vbt.ddi_port_info[p];
- if (info->alternate_ddc_pin != i->alternate_ddc_pin)
+ if (p == port || !i->present ||
+ info->alternate_ddc_pin != i->alternate_ddc_pin)
continue;
DRM_DEBUG_KMS("port %c trying to use the same DDC pin (0x%x) as port %c, "
* port. Otherwise they share the same ddc bin and
* system couldn't communicate with them separately.
*
- * Due to parsing the ports in alphabetical order,
- * a higher port will always clobber a lower one.
+ * Due to parsing the ports in child device order,
+ * a later device will always clobber an earlier one.
*/
i->supports_dvi = false;
i->supports_hdmi = false;
if (!info->alternate_aux_channel)
return;
- for_each_port_masked(p, (1 << port) - 1) {
+ for (p = PORT_A; p < I915_MAX_PORTS; p++) {
struct ddi_vbt_port_info *i = &dev_priv->vbt.ddi_port_info[p];
- if (info->alternate_aux_channel != i->alternate_aux_channel)
+ if (p == port || !i->present ||
+ info->alternate_aux_channel != i->alternate_aux_channel)
continue;
DRM_DEBUG_KMS("port %c trying to use the same AUX CH (0x%x) as port %c, "
* port. Otherwise they share the same aux channel
* and system couldn't communicate with them separately.
*
- * Due to parsing the ports in alphabetical order,
- * a higher port will always clobber a lower one.
+ * Due to parsing the ports in child device order,
+ * a later device will always clobber an earlier one.
*/
i->supports_dp = false;
i->alternate_aux_channel = 0;
return 0;
}
-static void parse_ddi_port(struct drm_i915_private *dev_priv, enum port port,
- u8 bdb_version)
+static enum port dvo_port_to_port(u8 dvo_port)
{
- struct child_device_config *it, *child = NULL;
- struct ddi_vbt_port_info *info = &dev_priv->vbt.ddi_port_info[port];
- int i, j;
- bool is_dvi, is_hdmi, is_dp, is_edp, is_crt;
- /* Each DDI port can have more than one value on the "DVO Port" field,
+ /*
+ * Each DDI port can have more than one value on the "DVO Port" field,
* so look for all the possible values for each port.
*/
- int dvo_ports[][3] = {
- {DVO_PORT_HDMIA, DVO_PORT_DPA, -1},
- {DVO_PORT_HDMIB, DVO_PORT_DPB, -1},
- {DVO_PORT_HDMIC, DVO_PORT_DPC, -1},
- {DVO_PORT_HDMID, DVO_PORT_DPD, -1},
- {DVO_PORT_CRT, DVO_PORT_HDMIE, DVO_PORT_DPE},
- {DVO_PORT_HDMIF, DVO_PORT_DPF, -1},
+ static const int dvo_ports[][3] = {
+ [PORT_A] = { DVO_PORT_HDMIA, DVO_PORT_DPA, -1},
+ [PORT_B] = { DVO_PORT_HDMIB, DVO_PORT_DPB, -1},
+ [PORT_C] = { DVO_PORT_HDMIC, DVO_PORT_DPC, -1},
+ [PORT_D] = { DVO_PORT_HDMID, DVO_PORT_DPD, -1},
+ [PORT_E] = { DVO_PORT_CRT, DVO_PORT_HDMIE, DVO_PORT_DPE},
+ [PORT_F] = { DVO_PORT_HDMIF, DVO_PORT_DPF, -1},
};
+ enum port port;
+ int i;
- /*
- * Find the first child device to reference the port, report if more
- * than one found.
- */
- for (i = 0; i < dev_priv->vbt.child_dev_num; i++) {
- it = dev_priv->vbt.child_dev + i;
-
- for (j = 0; j < 3; j++) {
- if (dvo_ports[port][j] == -1)
+ for (port = PORT_A; port < ARRAY_SIZE(dvo_ports); port++) {
+ for (i = 0; i < ARRAY_SIZE(dvo_ports[port]); i++) {
+ if (dvo_ports[port][i] == -1)
break;
- if (it->dvo_port == dvo_ports[port][j]) {
- if (child) {
- DRM_DEBUG_KMS("More than one child device for port %c in VBT, using the first.\n",
- port_name(port));
- } else {
- child = it;
- }
- }
+ if (dvo_port == dvo_ports[port][i])
+ return port;
}
}
- if (!child)
+
+ return PORT_NONE;
+}
+
+static void parse_ddi_port(struct drm_i915_private *dev_priv,
+ const struct child_device_config *child,
+ u8 bdb_version)
+{
+ struct ddi_vbt_port_info *info;
+ bool is_dvi, is_hdmi, is_dp, is_edp, is_crt;
+ enum port port;
+
+ port = dvo_port_to_port(child->dvo_port);
+ if (port == PORT_NONE)
+ return;
+
+ info = &dev_priv->vbt.ddi_port_info[port];
+
+ if (info->present) {
+ DRM_DEBUG_KMS("More than one child device for port %c in VBT, using the first.\n",
+ port_name(port));
return;
+ }
+
+ info->present = true;
is_dvi = child->device_type & DEVICE_TYPE_TMDS_DVI_SIGNALING;
is_dp = child->device_type & DEVICE_TYPE_DISPLAYPORT_OUTPUT;
static void parse_ddi_ports(struct drm_i915_private *dev_priv, u8 bdb_version)
{
- enum port port;
+ const struct child_device_config *child;
+ int i;
if (!HAS_DDI(dev_priv) && !IS_CHERRYVIEW(dev_priv))
return;
- if (!dev_priv->vbt.child_dev_num)
- return;
-
if (bdb_version < 155)
return;
- for (port = PORT_A; port < I915_MAX_PORTS; port++)
- parse_ddi_port(dev_priv, port, bdb_version);
+ for (i = 0; i < dev_priv->vbt.child_dev_num; i++) {
+ child = dev_priv->vbt.child_dev + i;
+
+ parse_ddi_port(dev_priv, child, bdb_version);
+ }
}
static void
dvo_port = child->dvo_port;
if (dvo_port == DVO_PORT_MIPIA ||
- (dvo_port == DVO_PORT_MIPIB && IS_ICELAKE(dev_priv)) ||
- (dvo_port == DVO_PORT_MIPIC && !IS_ICELAKE(dev_priv))) {
+ (dvo_port == DVO_PORT_MIPIB && INTEL_GEN(dev_priv) >= 11) ||
+ (dvo_port == DVO_PORT_MIPIC && INTEL_GEN(dev_priv) < 11)) {
if (port)
*port = dvo_port - DVO_PORT_MIPIA;
return true;
#include "i915_drv.h"
-#define task_asleep(tsk) ((tsk)->state & TASK_NORMAL && !(tsk)->on_rq)
-
static void irq_enable(struct intel_engine_cs *engine)
{
if (!engine->irq_enable)
return i915_seqno_passed(__hwsp_seqno(rq), rq->fence.seqno);
}
-bool intel_engine_breadcrumbs_irq(struct intel_engine_cs *engine)
+void intel_engine_breadcrumbs_irq(struct intel_engine_cs *engine)
{
struct intel_breadcrumbs *b = &engine->breadcrumbs;
struct intel_context *ce, *cn;
dma_fence_signal(&rq->fence);
i915_request_put(rq);
}
-
- return !list_empty(&signal);
}
-bool intel_engine_signal_breadcrumbs(struct intel_engine_cs *engine)
+void intel_engine_signal_breadcrumbs(struct intel_engine_cs *engine)
{
- bool result;
-
local_irq_disable();
- result = intel_engine_breadcrumbs_irq(engine);
+ intel_engine_breadcrumbs_irq(engine);
local_irq_enable();
-
- return result;
}
static void signal_irq_work(struct irq_work *work)
* DEALINGS IN THE SOFTWARE.
*/
+#include "intel_cdclk.h"
#include "intel_drv.h"
/**
else
return 0;
- tmp = I915_READ(IS_MOBILE(dev_priv) ? HPLLVCO_MOBILE : HPLLVCO);
+ tmp = I915_READ(IS_PINEVIEW(dev_priv) || IS_MOBILE(dev_priv) ?
+ HPLLVCO_MOBILE : HPLLVCO);
vco = vco_table[tmp & 0x7];
if (vco == 0)
cdclk_state->vco);
mutex_lock(&dev_priv->pcu_lock);
- val = vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ);
+ val = vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM);
mutex_unlock(&dev_priv->pcu_lock);
if (IS_VALLEYVIEW(dev_priv))
}
static void vlv_set_cdclk(struct drm_i915_private *dev_priv,
- const struct intel_cdclk_state *cdclk_state)
+ const struct intel_cdclk_state *cdclk_state,
+ enum pipe pipe)
{
int cdclk = cdclk_state->cdclk;
u32 val, cmd = cdclk_state->voltage_level;
wakeref = intel_display_power_get(dev_priv, POWER_DOMAIN_PIPE_A);
mutex_lock(&dev_priv->pcu_lock);
- val = vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ);
+ val = vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM);
val &= ~DSPFREQGUAR_MASK;
val |= (cmd << DSPFREQGUAR_SHIFT);
- vlv_punit_write(dev_priv, PUNIT_REG_DSPFREQ, val);
- if (wait_for((vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ) &
+ vlv_punit_write(dev_priv, PUNIT_REG_DSPSSPM, val);
+ if (wait_for((vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM) &
DSPFREQSTAT_MASK) == (cmd << DSPFREQSTAT_SHIFT),
50)) {
DRM_ERROR("timed out waiting for CDclk change\n");
}
static void chv_set_cdclk(struct drm_i915_private *dev_priv,
- const struct intel_cdclk_state *cdclk_state)
+ const struct intel_cdclk_state *cdclk_state,
+ enum pipe pipe)
{
int cdclk = cdclk_state->cdclk;
u32 val, cmd = cdclk_state->voltage_level;
wakeref = intel_display_power_get(dev_priv, POWER_DOMAIN_PIPE_A);
mutex_lock(&dev_priv->pcu_lock);
- val = vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ);
+ val = vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM);
val &= ~DSPFREQGUAR_MASK_CHV;
val |= (cmd << DSPFREQGUAR_SHIFT_CHV);
- vlv_punit_write(dev_priv, PUNIT_REG_DSPFREQ, val);
- if (wait_for((vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ) &
+ vlv_punit_write(dev_priv, PUNIT_REG_DSPSSPM, val);
+ if (wait_for((vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM) &
DSPFREQSTAT_MASK_CHV) == (cmd << DSPFREQSTAT_SHIFT_CHV),
50)) {
DRM_ERROR("timed out waiting for CDclk change\n");
}
static void bdw_set_cdclk(struct drm_i915_private *dev_priv,
- const struct intel_cdclk_state *cdclk_state)
+ const struct intel_cdclk_state *cdclk_state,
+ enum pipe pipe)
{
int cdclk = cdclk_state->cdclk;
u32 val;
I915_WRITE(LCPLL1_CTL, I915_READ(LCPLL1_CTL) | LCPLL_PLL_ENABLE);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
LCPLL1_CTL, LCPLL_PLL_LOCK, LCPLL_PLL_LOCK,
5))
DRM_ERROR("DPLL0 not locked\n");
static void skl_dpll0_disable(struct drm_i915_private *dev_priv)
{
I915_WRITE(LCPLL1_CTL, I915_READ(LCPLL1_CTL) & ~LCPLL_PLL_ENABLE);
- if (intel_wait_for_register(dev_priv,
- LCPLL1_CTL, LCPLL_PLL_LOCK, 0,
- 1))
+ if (intel_wait_for_register(&dev_priv->uncore,
+ LCPLL1_CTL, LCPLL_PLL_LOCK, 0,
+ 1))
DRM_ERROR("Couldn't disable DPLL0\n");
dev_priv->cdclk.hw.vco = 0;
}
static void skl_set_cdclk(struct drm_i915_private *dev_priv,
- const struct intel_cdclk_state *cdclk_state)
+ const struct intel_cdclk_state *cdclk_state,
+ enum pipe pipe)
{
int cdclk = cdclk_state->cdclk;
int vco = cdclk_state->vco;
dev_priv->cdclk.hw.vco = -1;
}
-/**
- * skl_init_cdclk - Initialize CDCLK on SKL
- * @dev_priv: i915 device
- *
- * Initialize CDCLK for SKL and derivatives. This is generally
- * done only during the display core initialization sequence,
- * after which the DMC will take care of turning CDCLK off/on
- * as needed.
- */
-void skl_init_cdclk(struct drm_i915_private *dev_priv)
+static void skl_init_cdclk(struct drm_i915_private *dev_priv)
{
struct intel_cdclk_state cdclk_state;
cdclk_state.cdclk = skl_calc_cdclk(0, cdclk_state.vco);
cdclk_state.voltage_level = skl_calc_voltage_level(cdclk_state.cdclk);
- skl_set_cdclk(dev_priv, &cdclk_state);
+ skl_set_cdclk(dev_priv, &cdclk_state, INVALID_PIPE);
}
-/**
- * skl_uninit_cdclk - Uninitialize CDCLK on SKL
- * @dev_priv: i915 device
- *
- * Uninitialize CDCLK for SKL and derivatives. This is done only
- * during the display core uninitialization sequence.
- */
-void skl_uninit_cdclk(struct drm_i915_private *dev_priv)
+static void skl_uninit_cdclk(struct drm_i915_private *dev_priv)
{
struct intel_cdclk_state cdclk_state = dev_priv->cdclk.hw;
cdclk_state.vco = 0;
cdclk_state.voltage_level = skl_calc_voltage_level(cdclk_state.cdclk);
- skl_set_cdclk(dev_priv, &cdclk_state);
+ skl_set_cdclk(dev_priv, &cdclk_state, INVALID_PIPE);
}
static int bxt_calc_cdclk(int min_cdclk)
I915_WRITE(BXT_DE_PLL_ENABLE, 0);
/* Timeout 200us */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
BXT_DE_PLL_ENABLE, BXT_DE_PLL_LOCK, 0,
1))
DRM_ERROR("timeout waiting for DE PLL unlock\n");
I915_WRITE(BXT_DE_PLL_ENABLE, BXT_DE_PLL_PLL_ENABLE);
/* Timeout 200us */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
BXT_DE_PLL_ENABLE,
BXT_DE_PLL_LOCK,
BXT_DE_PLL_LOCK,
}
static void bxt_set_cdclk(struct drm_i915_private *dev_priv,
- const struct intel_cdclk_state *cdclk_state)
+ const struct intel_cdclk_state *cdclk_state,
+ enum pipe pipe)
{
int cdclk = cdclk_state->cdclk;
int vco = cdclk_state->vco;
bxt_de_pll_enable(dev_priv, vco);
val = divider | skl_cdclk_decimal(cdclk);
- /*
- * FIXME if only the cd2x divider needs changing, it could be done
- * without shutting off the pipe (if only one pipe is active).
- */
- val |= BXT_CDCLK_CD2X_PIPE_NONE;
+ if (pipe == INVALID_PIPE)
+ val |= BXT_CDCLK_CD2X_PIPE_NONE;
+ else
+ val |= BXT_CDCLK_CD2X_PIPE(pipe);
/*
* Disable SSA Precharge when CD clock frequency < 500 MHz,
* enable otherwise.
val |= BXT_CDCLK_SSA_PRECHARGE_ENABLE;
I915_WRITE(CDCLK_CTL, val);
+ if (pipe != INVALID_PIPE)
+ intel_wait_for_vblank(dev_priv, pipe);
+
mutex_lock(&dev_priv->pcu_lock);
/*
* The timeout isn't specified, the 2ms used here is based on
dev_priv->cdclk.hw.vco = -1;
}
-/**
- * bxt_init_cdclk - Initialize CDCLK on BXT
- * @dev_priv: i915 device
- *
- * Initialize CDCLK for BXT and derivatives. This is generally
- * done only during the display core initialization sequence,
- * after which the DMC will take care of turning CDCLK off/on
- * as needed.
- */
-void bxt_init_cdclk(struct drm_i915_private *dev_priv)
+static void bxt_init_cdclk(struct drm_i915_private *dev_priv)
{
struct intel_cdclk_state cdclk_state;
}
cdclk_state.voltage_level = bxt_calc_voltage_level(cdclk_state.cdclk);
- bxt_set_cdclk(dev_priv, &cdclk_state);
+ bxt_set_cdclk(dev_priv, &cdclk_state, INVALID_PIPE);
}
-/**
- * bxt_uninit_cdclk - Uninitialize CDCLK on BXT
- * @dev_priv: i915 device
- *
- * Uninitialize CDCLK for BXT and derivatives. This is done only
- * during the display core uninitialization sequence.
- */
-void bxt_uninit_cdclk(struct drm_i915_private *dev_priv)
+static void bxt_uninit_cdclk(struct drm_i915_private *dev_priv)
{
struct intel_cdclk_state cdclk_state = dev_priv->cdclk.hw;
cdclk_state.vco = 0;
cdclk_state.voltage_level = bxt_calc_voltage_level(cdclk_state.cdclk);
- bxt_set_cdclk(dev_priv, &cdclk_state);
+ bxt_set_cdclk(dev_priv, &cdclk_state, INVALID_PIPE);
}
static int cnl_calc_cdclk(int min_cdclk)
}
static void cnl_set_cdclk(struct drm_i915_private *dev_priv,
- const struct intel_cdclk_state *cdclk_state)
+ const struct intel_cdclk_state *cdclk_state,
+ enum pipe pipe)
{
int cdclk = cdclk_state->cdclk;
int vco = cdclk_state->vco;
cnl_cdclk_pll_enable(dev_priv, vco);
val = divider | skl_cdclk_decimal(cdclk);
- /*
- * FIXME if only the cd2x divider needs changing, it could be done
- * without shutting off the pipe (if only one pipe is active).
- */
- val |= BXT_CDCLK_CD2X_PIPE_NONE;
+ if (pipe == INVALID_PIPE)
+ val |= BXT_CDCLK_CD2X_PIPE_NONE;
+ else
+ val |= BXT_CDCLK_CD2X_PIPE(pipe);
I915_WRITE(CDCLK_CTL, val);
+ if (pipe != INVALID_PIPE)
+ intel_wait_for_vblank(dev_priv, pipe);
+
/* inform PCU of the change */
mutex_lock(&dev_priv->pcu_lock);
sandybridge_pcode_write(dev_priv, SKL_PCODE_CDCLK_CONTROL,
}
static void icl_set_cdclk(struct drm_i915_private *dev_priv,
- const struct intel_cdclk_state *cdclk_state)
+ const struct intel_cdclk_state *cdclk_state,
+ enum pipe pipe)
{
unsigned int cdclk = cdclk_state->cdclk;
unsigned int vco = cdclk_state->vco;
if (dev_priv->cdclk.hw.vco != vco)
cnl_cdclk_pll_enable(dev_priv, vco);
+ /*
+ * On ICL CD2X_DIV can only be 1, so we'll never end up changing the
+ * divider here synchronized to a pipe while CDCLK is on, nor will we
+ * need the corresponding vblank wait.
+ */
I915_WRITE(CDCLK_CTL, ICL_CDCLK_CD2X_PIPE_NONE |
skl_cdclk_decimal(cdclk));
icl_calc_voltage_level(cdclk_state->cdclk);
}
-/**
- * icl_init_cdclk - Initialize CDCLK on ICL
- * @dev_priv: i915 device
- *
- * Initialize CDCLK for ICL. This consists mainly of initializing
- * dev_priv->cdclk.hw and sanitizing the state of the hardware if needed. This
- * is generally done only during the display core initialization sequence, after
- * which the DMC will take care of turning CDCLK off/on as needed.
- */
-void icl_init_cdclk(struct drm_i915_private *dev_priv)
+static void icl_init_cdclk(struct drm_i915_private *dev_priv)
{
struct intel_cdclk_state sanitized_state;
u32 val;
sanitized_state.voltage_level =
icl_calc_voltage_level(sanitized_state.cdclk);
- icl_set_cdclk(dev_priv, &sanitized_state);
+ icl_set_cdclk(dev_priv, &sanitized_state, INVALID_PIPE);
}
-/**
- * icl_uninit_cdclk - Uninitialize CDCLK on ICL
- * @dev_priv: i915 device
- *
- * Uninitialize CDCLK for ICL. This is done only during the display core
- * uninitialization sequence.
- */
-void icl_uninit_cdclk(struct drm_i915_private *dev_priv)
+static void icl_uninit_cdclk(struct drm_i915_private *dev_priv)
{
struct intel_cdclk_state cdclk_state = dev_priv->cdclk.hw;
cdclk_state.vco = 0;
cdclk_state.voltage_level = icl_calc_voltage_level(cdclk_state.cdclk);
- icl_set_cdclk(dev_priv, &cdclk_state);
+ icl_set_cdclk(dev_priv, &cdclk_state, INVALID_PIPE);
}
-/**
- * cnl_init_cdclk - Initialize CDCLK on CNL
- * @dev_priv: i915 device
- *
- * Initialize CDCLK for CNL. This is generally
- * done only during the display core initialization sequence,
- * after which the DMC will take care of turning CDCLK off/on
- * as needed.
- */
-void cnl_init_cdclk(struct drm_i915_private *dev_priv)
+static void cnl_init_cdclk(struct drm_i915_private *dev_priv)
{
struct intel_cdclk_state cdclk_state;
cdclk_state.vco = cnl_cdclk_pll_vco(dev_priv, cdclk_state.cdclk);
cdclk_state.voltage_level = cnl_calc_voltage_level(cdclk_state.cdclk);
- cnl_set_cdclk(dev_priv, &cdclk_state);
+ cnl_set_cdclk(dev_priv, &cdclk_state, INVALID_PIPE);
}
-/**
- * cnl_uninit_cdclk - Uninitialize CDCLK on CNL
- * @dev_priv: i915 device
- *
- * Uninitialize CDCLK for CNL. This is done only
- * during the display core uninitialization sequence.
- */
-void cnl_uninit_cdclk(struct drm_i915_private *dev_priv)
+static void cnl_uninit_cdclk(struct drm_i915_private *dev_priv)
{
struct intel_cdclk_state cdclk_state = dev_priv->cdclk.hw;
cdclk_state.vco = 0;
cdclk_state.voltage_level = cnl_calc_voltage_level(cdclk_state.cdclk);
- cnl_set_cdclk(dev_priv, &cdclk_state);
+ cnl_set_cdclk(dev_priv, &cdclk_state, INVALID_PIPE);
+}
+
+/**
+ * intel_cdclk_init - Initialize CDCLK
+ * @i915: i915 device
+ *
+ * Initialize CDCLK. This consists mainly of initializing dev_priv->cdclk.hw and
+ * sanitizing the state of the hardware if needed. This is generally done only
+ * during the display core initialization sequence, after which the DMC will
+ * take care of turning CDCLK off/on as needed.
+ */
+void intel_cdclk_init(struct drm_i915_private *i915)
+{
+ if (INTEL_GEN(i915) >= 11)
+ icl_init_cdclk(i915);
+ else if (IS_CANNONLAKE(i915))
+ cnl_init_cdclk(i915);
+ else if (IS_GEN9_BC(i915))
+ skl_init_cdclk(i915);
+ else if (IS_GEN9_LP(i915))
+ bxt_init_cdclk(i915);
+}
+
+/**
+ * intel_cdclk_uninit - Uninitialize CDCLK
+ * @i915: i915 device
+ *
+ * Uninitialize CDCLK. This is done only during the display core
+ * uninitialization sequence.
+ */
+void intel_cdclk_uninit(struct drm_i915_private *i915)
+{
+ if (INTEL_GEN(i915) >= 11)
+ icl_uninit_cdclk(i915);
+ else if (IS_CANNONLAKE(i915))
+ cnl_uninit_cdclk(i915);
+ else if (IS_GEN9_BC(i915))
+ skl_uninit_cdclk(i915);
+ else if (IS_GEN9_LP(i915))
+ bxt_uninit_cdclk(i915);
}
/**
a->ref != b->ref;
}
+/**
+ * intel_cdclk_needs_cd2x_update - Determine if two CDCLK states require a cd2x divider update
+ * @dev_priv: Not a CDCLK state, it's the drm_i915_private!
+ * @a: first CDCLK state
+ * @b: second CDCLK state
+ *
+ * Returns:
+ * True if the CDCLK states require just a cd2x divider update, false if not.
+ */
+bool intel_cdclk_needs_cd2x_update(struct drm_i915_private *dev_priv,
+ const struct intel_cdclk_state *a,
+ const struct intel_cdclk_state *b)
+{
+ /* Older hw doesn't have the capability */
+ if (INTEL_GEN(dev_priv) < 10 && !IS_GEN9_LP(dev_priv))
+ return false;
+
+ return a->cdclk != b->cdclk &&
+ a->vco == b->vco &&
+ a->ref == b->ref;
+}
+
/**
* intel_cdclk_changed - Determine if two CDCLK states are different
* @a: first CDCLK state
a->voltage_level != b->voltage_level;
}
+/**
+ * intel_cdclk_swap_state - make atomic CDCLK configuration effective
+ * @state: atomic state
+ *
+ * This is the CDCLK version of drm_atomic_helper_swap_state() since the
+ * helper does not handle driver-specific global state.
+ *
+ * Similarly to the atomic helpers this function does a complete swap,
+ * i.e. it also puts the old state into @state. This is used by the commit
+ * code to determine how CDCLK has changed (for instance did it increase or
+ * decrease).
+ */
+void intel_cdclk_swap_state(struct intel_atomic_state *state)
+{
+ struct drm_i915_private *dev_priv = to_i915(state->base.dev);
+
+ swap(state->cdclk.logical, dev_priv->cdclk.logical);
+ swap(state->cdclk.actual, dev_priv->cdclk.actual);
+}
+
void intel_dump_cdclk_state(const struct intel_cdclk_state *cdclk_state,
const char *context)
{
* intel_set_cdclk - Push the CDCLK state to the hardware
* @dev_priv: i915 device
* @cdclk_state: new CDCLK state
+ * @pipe: pipe with which to synchronize the update
*
* Program the hardware based on the passed in CDCLK state,
* if necessary.
*/
-void intel_set_cdclk(struct drm_i915_private *dev_priv,
- const struct intel_cdclk_state *cdclk_state)
+static void intel_set_cdclk(struct drm_i915_private *dev_priv,
+ const struct intel_cdclk_state *cdclk_state,
+ enum pipe pipe)
{
if (!intel_cdclk_changed(&dev_priv->cdclk.hw, cdclk_state))
return;
intel_dump_cdclk_state(cdclk_state, "Changing CDCLK to");
- dev_priv->display.set_cdclk(dev_priv, cdclk_state);
+ dev_priv->display.set_cdclk(dev_priv, cdclk_state, pipe);
if (WARN(intel_cdclk_changed(&dev_priv->cdclk.hw, cdclk_state),
"cdclk state doesn't match!\n")) {
}
}
+/**
+ * intel_set_cdclk_pre_plane_update - Push the CDCLK state to the hardware
+ * @dev_priv: i915 device
+ * @old_state: old CDCLK state
+ * @new_state: new CDCLK state
+ * @pipe: pipe with which to synchronize the update
+ *
+ * Program the hardware before updating the HW plane state based on the passed
+ * in CDCLK state, if necessary.
+ */
+void
+intel_set_cdclk_pre_plane_update(struct drm_i915_private *dev_priv,
+ const struct intel_cdclk_state *old_state,
+ const struct intel_cdclk_state *new_state,
+ enum pipe pipe)
+{
+ if (pipe == INVALID_PIPE || old_state->cdclk <= new_state->cdclk)
+ intel_set_cdclk(dev_priv, new_state, pipe);
+}
+
+/**
+ * intel_set_cdclk_post_plane_update - Push the CDCLK state to the hardware
+ * @dev_priv: i915 device
+ * @old_state: old CDCLK state
+ * @new_state: new CDCLK state
+ * @pipe: pipe with which to synchronize the update
+ *
+ * Program the hardware after updating the HW plane state based on the passed
+ * in CDCLK state, if necessary.
+ */
+void
+intel_set_cdclk_post_plane_update(struct drm_i915_private *dev_priv,
+ const struct intel_cdclk_state *old_state,
+ const struct intel_cdclk_state *new_state,
+ enum pipe pipe)
+{
+ if (pipe != INVALID_PIPE && old_state->cdclk > new_state->cdclk)
+ intel_set_cdclk(dev_priv, new_state, pipe);
+}
+
static int intel_pixel_rate_to_cdclk(struct drm_i915_private *dev_priv,
int pixel_rate)
{
/*
* According to BSpec, "The CD clock frequency must be at least twice
* the frequency of the Azalia BCLK." and BCLK is 96 MHz by default.
- *
- * FIXME: Check the actual, not default, BCLK being used.
- *
- * FIXME: This does not depend on ->has_audio because the higher CDCLK
- * is required for audio probe, also when there are no audio capable
- * displays connected at probe time. This leads to unnecessarily high
- * CDCLK when audio is not required.
- *
- * FIXME: This limit is only applied when there are displays connected
- * at probe time. If we probe without displays, we'll still end up using
- * the platform minimum CDCLK, failing audio probe.
*/
- if (INTEL_GEN(dev_priv) >= 9)
+ if (crtc_state->has_audio && INTEL_GEN(dev_priv) >= 9)
min_cdclk = max(2 * 96000, min_cdclk);
/*
intel_state->min_cdclk[i] = min_cdclk;
}
- min_cdclk = 0;
+ min_cdclk = intel_state->cdclk.force_min_cdclk;
for_each_pipe(dev_priv, pipe)
min_cdclk = max(intel_state->min_cdclk[pipe], min_cdclk);
vlv_calc_voltage_level(dev_priv, cdclk);
if (!intel_state->active_crtcs) {
- cdclk = vlv_calc_cdclk(dev_priv, 0);
+ cdclk = vlv_calc_cdclk(dev_priv,
+ intel_state->cdclk.force_min_cdclk);
intel_state->cdclk.actual.cdclk = cdclk;
intel_state->cdclk.actual.voltage_level =
bdw_calc_voltage_level(cdclk);
if (!intel_state->active_crtcs) {
- cdclk = bdw_calc_cdclk(0);
+ cdclk = bdw_calc_cdclk(intel_state->cdclk.force_min_cdclk);
intel_state->cdclk.actual.cdclk = cdclk;
intel_state->cdclk.actual.voltage_level =
skl_calc_voltage_level(cdclk);
if (!intel_state->active_crtcs) {
- cdclk = skl_calc_cdclk(0, vco);
+ cdclk = skl_calc_cdclk(intel_state->cdclk.force_min_cdclk, vco);
intel_state->cdclk.actual.vco = vco;
intel_state->cdclk.actual.cdclk = cdclk;
if (!intel_state->active_crtcs) {
if (IS_GEMINILAKE(dev_priv)) {
- cdclk = glk_calc_cdclk(0);
+ cdclk = glk_calc_cdclk(intel_state->cdclk.force_min_cdclk);
vco = glk_de_pll_vco(dev_priv, cdclk);
} else {
- cdclk = bxt_calc_cdclk(0);
+ cdclk = bxt_calc_cdclk(intel_state->cdclk.force_min_cdclk);
vco = bxt_de_pll_vco(dev_priv, cdclk);
}
cnl_compute_min_voltage_level(intel_state));
if (!intel_state->active_crtcs) {
- cdclk = cnl_calc_cdclk(0);
+ cdclk = cnl_calc_cdclk(intel_state->cdclk.force_min_cdclk);
vco = cnl_cdclk_pll_vco(dev_priv, cdclk);
intel_state->cdclk.actual.vco = vco;
cnl_compute_min_voltage_level(intel_state));
if (!intel_state->active_crtcs) {
- cdclk = icl_calc_cdclk(0, ref);
+ cdclk = icl_calc_cdclk(intel_state->cdclk.force_min_cdclk, ref);
vco = icl_calc_cdclk_pll_vco(dev_priv, cdclk);
intel_state->cdclk.actual.vco = vco;
*/
void intel_update_max_cdclk(struct drm_i915_private *dev_priv)
{
- if (IS_ICELAKE(dev_priv)) {
+ if (INTEL_GEN(dev_priv) >= 11) {
if (dev_priv->cdclk.hw.ref == 24000)
dev_priv->max_cdclk_freq = 648000;
else
rawclk |= CNP_RAWCLK_DEN(DIV_ROUND_CLOSEST(numerator * 1000,
fraction) - 1);
- if (HAS_PCH_ICP(dev_priv))
+ if (INTEL_PCH_TYPE(dev_priv) >= PCH_ICP)
rawclk |= ICP_RAWCLK_NUM(numerator);
}
*/
void intel_update_rawclk(struct drm_i915_private *dev_priv)
{
- if (HAS_PCH_CNP(dev_priv) || HAS_PCH_ICP(dev_priv))
+ if (INTEL_PCH_TYPE(dev_priv) >= PCH_CNP)
dev_priv->rawclk_freq = cnp_rawclk(dev_priv);
else if (HAS_PCH_SPLIT(dev_priv))
dev_priv->rawclk_freq = pch_rawclk(dev_priv);
*/
void intel_init_cdclk_hooks(struct drm_i915_private *dev_priv)
{
- if (IS_CHERRYVIEW(dev_priv)) {
- dev_priv->display.set_cdclk = chv_set_cdclk;
- dev_priv->display.modeset_calc_cdclk =
- vlv_modeset_calc_cdclk;
- } else if (IS_VALLEYVIEW(dev_priv)) {
- dev_priv->display.set_cdclk = vlv_set_cdclk;
- dev_priv->display.modeset_calc_cdclk =
- vlv_modeset_calc_cdclk;
- } else if (IS_BROADWELL(dev_priv)) {
- dev_priv->display.set_cdclk = bdw_set_cdclk;
+ if (INTEL_GEN(dev_priv) >= 11) {
+ dev_priv->display.set_cdclk = icl_set_cdclk;
+ dev_priv->display.modeset_calc_cdclk = icl_modeset_calc_cdclk;
+ } else if (IS_CANNONLAKE(dev_priv)) {
+ dev_priv->display.set_cdclk = cnl_set_cdclk;
dev_priv->display.modeset_calc_cdclk =
- bdw_modeset_calc_cdclk;
+ cnl_modeset_calc_cdclk;
} else if (IS_GEN9_LP(dev_priv)) {
dev_priv->display.set_cdclk = bxt_set_cdclk;
dev_priv->display.modeset_calc_cdclk =
dev_priv->display.set_cdclk = skl_set_cdclk;
dev_priv->display.modeset_calc_cdclk =
skl_modeset_calc_cdclk;
- } else if (IS_CANNONLAKE(dev_priv)) {
- dev_priv->display.set_cdclk = cnl_set_cdclk;
+ } else if (IS_BROADWELL(dev_priv)) {
+ dev_priv->display.set_cdclk = bdw_set_cdclk;
dev_priv->display.modeset_calc_cdclk =
- cnl_modeset_calc_cdclk;
- } else if (IS_ICELAKE(dev_priv)) {
- dev_priv->display.set_cdclk = icl_set_cdclk;
- dev_priv->display.modeset_calc_cdclk = icl_modeset_calc_cdclk;
+ bdw_modeset_calc_cdclk;
+ } else if (IS_CHERRYVIEW(dev_priv)) {
+ dev_priv->display.set_cdclk = chv_set_cdclk;
+ dev_priv->display.modeset_calc_cdclk =
+ vlv_modeset_calc_cdclk;
+ } else if (IS_VALLEYVIEW(dev_priv)) {
+ dev_priv->display.set_cdclk = vlv_set_cdclk;
+ dev_priv->display.modeset_calc_cdclk =
+ vlv_modeset_calc_cdclk;
}
- if (IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 11)
dev_priv->display.get_cdclk = icl_get_cdclk;
else if (IS_CANNONLAKE(dev_priv))
dev_priv->display.get_cdclk = cnl_get_cdclk;
- else if (IS_GEN9_BC(dev_priv))
- dev_priv->display.get_cdclk = skl_get_cdclk;
else if (IS_GEN9_LP(dev_priv))
dev_priv->display.get_cdclk = bxt_get_cdclk;
+ else if (IS_GEN9_BC(dev_priv))
+ dev_priv->display.get_cdclk = skl_get_cdclk;
else if (IS_BROADWELL(dev_priv))
dev_priv->display.get_cdclk = bdw_get_cdclk;
else if (IS_HASWELL(dev_priv))
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_CDCLK_H__
+#define __INTEL_CDCLK_H__
+
+#include <linux/types.h>
+
+#include "intel_display.h"
+
+struct drm_i915_private;
+struct intel_atomic_state;
+struct intel_cdclk_state;
+struct intel_crtc_state;
+
+int intel_crtc_compute_min_cdclk(const struct intel_crtc_state *crtc_state);
+void intel_cdclk_init(struct drm_i915_private *i915);
+void intel_cdclk_uninit(struct drm_i915_private *i915);
+void intel_init_cdclk_hooks(struct drm_i915_private *dev_priv);
+void intel_update_max_cdclk(struct drm_i915_private *dev_priv);
+void intel_update_cdclk(struct drm_i915_private *dev_priv);
+void intel_update_rawclk(struct drm_i915_private *dev_priv);
+bool intel_cdclk_needs_cd2x_update(struct drm_i915_private *dev_priv,
+ const struct intel_cdclk_state *a,
+ const struct intel_cdclk_state *b);
+bool intel_cdclk_needs_modeset(const struct intel_cdclk_state *a,
+ const struct intel_cdclk_state *b);
+bool intel_cdclk_changed(const struct intel_cdclk_state *a,
+ const struct intel_cdclk_state *b);
+void intel_cdclk_swap_state(struct intel_atomic_state *state);
+void
+intel_set_cdclk_pre_plane_update(struct drm_i915_private *dev_priv,
+ const struct intel_cdclk_state *old_state,
+ const struct intel_cdclk_state *new_state,
+ enum pipe pipe);
+void
+intel_set_cdclk_post_plane_update(struct drm_i915_private *dev_priv,
+ const struct intel_cdclk_state *old_state,
+ const struct intel_cdclk_state *new_state,
+ enum pipe pipe);
+void intel_dump_cdclk_state(const struct intel_cdclk_state *cdclk_state,
+ const char *context);
+
+#endif /* __INTEL_CDCLK_H__ */
*
*/
+#include "intel_color.h"
#include "intel_drv.h"
#define CTM_COEFF_SIGN (1ULL << 63)
#define CTM_COEFF_ABS(coeff) ((coeff) & (CTM_COEFF_SIGN - 1))
#define LEGACY_LUT_LENGTH 256
-
-/* Post offset values for RGB->YCBCR conversion */
-#define POSTOFF_RGB_TO_YUV_HI 0x800
-#define POSTOFF_RGB_TO_YUV_ME 0x100
-#define POSTOFF_RGB_TO_YUV_LO 0x800
-
-/*
- * These values are direct register values specified in the Bspec,
- * for RGB->YUV conversion matrix (colorspace BT709)
- */
-#define CSC_RGB_TO_YUV_RU_GU 0x2ba809d8
-#define CSC_RGB_TO_YUV_BU 0x37e80000
-#define CSC_RGB_TO_YUV_RY_GY 0x1e089cc0
-#define CSC_RGB_TO_YUV_BY 0xb5280000
-#define CSC_RGB_TO_YUV_RV_GV 0xbce89ad8
-#define CSC_RGB_TO_YUV_BV 0x1e080000
-
/*
* Extract the CSC coefficient from a CTM coefficient (in U32.32 fixed point
* format). This macro takes the coefficient we want transformed and the
#define ILK_CSC_COEFF_FP(coeff, fbits) \
(clamp_val(((coeff) >> (32 - (fbits) - 3)) + 4, 0, 0xfff) & 0xff8)
-#define ILK_CSC_COEFF_LIMITED_RANGE \
- ILK_CSC_COEFF_FP(CTM_COEFF_LIMITED_RANGE, 9)
-#define ILK_CSC_COEFF_1_0 \
- ((7 << 12) | ILK_CSC_COEFF_FP(CTM_COEFF_1_0, 8))
+#define ILK_CSC_COEFF_LIMITED_RANGE 0x0dc0
+#define ILK_CSC_COEFF_1_0 0x7800
+
+#define ILK_CSC_POSTOFF_LIMITED_RANGE (16 * (1 << 12) / 255)
+
+static const u16 ilk_csc_off_zero[3] = {};
+
+static const u16 ilk_csc_coeff_identity[9] = {
+ ILK_CSC_COEFF_1_0, 0, 0,
+ 0, ILK_CSC_COEFF_1_0, 0,
+ 0, 0, ILK_CSC_COEFF_1_0,
+};
+
+static const u16 ilk_csc_postoff_limited_range[3] = {
+ ILK_CSC_POSTOFF_LIMITED_RANGE,
+ ILK_CSC_POSTOFF_LIMITED_RANGE,
+ ILK_CSC_POSTOFF_LIMITED_RANGE,
+};
+
+static const u16 ilk_csc_coeff_limited_range[9] = {
+ ILK_CSC_COEFF_LIMITED_RANGE, 0, 0,
+ 0, ILK_CSC_COEFF_LIMITED_RANGE, 0,
+ 0, 0, ILK_CSC_COEFF_LIMITED_RANGE,
+};
+
+/*
+ * These values are direct register values specified in the Bspec,
+ * for RGB->YUV conversion matrix (colorspace BT709)
+ */
+static const u16 ilk_csc_coeff_rgb_to_ycbcr[9] = {
+ 0x1e08, 0x9cc0, 0xb528,
+ 0x2ba8, 0x09d8, 0x37e8,
+ 0xbce8, 0x9ad8, 0x1e08,
+};
+
+/* Post offset values for RGB->YCBCR conversion */
+static const u16 ilk_csc_postoff_rgb_to_ycbcr[3] = {
+ 0x0800, 0x0100, 0x0800,
+};
static bool lut_is_legacy(const struct drm_property_blob *lut)
{
return result;
}
-static void ilk_load_ycbcr_conversion_matrix(struct intel_crtc *crtc)
+static void ilk_update_pipe_csc(struct intel_crtc *crtc,
+ const u16 preoff[3],
+ const u16 coeff[9],
+ const u16 postoff[3])
{
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
enum pipe pipe = crtc->pipe;
- I915_WRITE(PIPE_CSC_PREOFF_HI(pipe), 0);
- I915_WRITE(PIPE_CSC_PREOFF_ME(pipe), 0);
- I915_WRITE(PIPE_CSC_PREOFF_LO(pipe), 0);
+ I915_WRITE(PIPE_CSC_PREOFF_HI(pipe), preoff[0]);
+ I915_WRITE(PIPE_CSC_PREOFF_ME(pipe), preoff[1]);
+ I915_WRITE(PIPE_CSC_PREOFF_LO(pipe), preoff[2]);
- I915_WRITE(PIPE_CSC_COEFF_RU_GU(pipe), CSC_RGB_TO_YUV_RU_GU);
- I915_WRITE(PIPE_CSC_COEFF_BU(pipe), CSC_RGB_TO_YUV_BU);
+ I915_WRITE(PIPE_CSC_COEFF_RY_GY(pipe), coeff[0] << 16 | coeff[1]);
+ I915_WRITE(PIPE_CSC_COEFF_BY(pipe), coeff[2] << 16);
- I915_WRITE(PIPE_CSC_COEFF_RY_GY(pipe), CSC_RGB_TO_YUV_RY_GY);
- I915_WRITE(PIPE_CSC_COEFF_BY(pipe), CSC_RGB_TO_YUV_BY);
+ I915_WRITE(PIPE_CSC_COEFF_RU_GU(pipe), coeff[3] << 16 | coeff[4]);
+ I915_WRITE(PIPE_CSC_COEFF_BU(pipe), coeff[5] << 16);
- I915_WRITE(PIPE_CSC_COEFF_RV_GV(pipe), CSC_RGB_TO_YUV_RV_GV);
- I915_WRITE(PIPE_CSC_COEFF_BV(pipe), CSC_RGB_TO_YUV_BV);
+ I915_WRITE(PIPE_CSC_COEFF_RV_GV(pipe), coeff[6] << 16 | coeff[7]);
+ I915_WRITE(PIPE_CSC_COEFF_BV(pipe), coeff[8] << 16);
- I915_WRITE(PIPE_CSC_POSTOFF_HI(pipe), POSTOFF_RGB_TO_YUV_HI);
- I915_WRITE(PIPE_CSC_POSTOFF_ME(pipe), POSTOFF_RGB_TO_YUV_ME);
- I915_WRITE(PIPE_CSC_POSTOFF_LO(pipe), POSTOFF_RGB_TO_YUV_LO);
- I915_WRITE(PIPE_CSC_MODE(pipe), 0);
+ if (INTEL_GEN(dev_priv) >= 7) {
+ I915_WRITE(PIPE_CSC_POSTOFF_HI(pipe), postoff[0]);
+ I915_WRITE(PIPE_CSC_POSTOFF_ME(pipe), postoff[1]);
+ I915_WRITE(PIPE_CSC_POSTOFF_LO(pipe), postoff[2]);
+ }
}
-static void ilk_load_csc_matrix(const struct intel_crtc_state *crtc_state)
+static void icl_update_output_csc(struct intel_crtc *crtc,
+ const u16 preoff[3],
+ const u16 coeff[9],
+ const u16 postoff[3])
{
- struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
- bool limited_color_range = false;
enum pipe pipe = crtc->pipe;
- u16 coeffs[9] = {};
- int i;
+
+ I915_WRITE(PIPE_CSC_OUTPUT_PREOFF_HI(pipe), preoff[0]);
+ I915_WRITE(PIPE_CSC_OUTPUT_PREOFF_ME(pipe), preoff[1]);
+ I915_WRITE(PIPE_CSC_OUTPUT_PREOFF_LO(pipe), preoff[2]);
+
+ I915_WRITE(PIPE_CSC_OUTPUT_COEFF_RY_GY(pipe), coeff[0] << 16 | coeff[1]);
+ I915_WRITE(PIPE_CSC_OUTPUT_COEFF_BY(pipe), coeff[2] << 16);
+
+ I915_WRITE(PIPE_CSC_OUTPUT_COEFF_RU_GU(pipe), coeff[3] << 16 | coeff[4]);
+ I915_WRITE(PIPE_CSC_OUTPUT_COEFF_BU(pipe), coeff[5] << 16);
+
+ I915_WRITE(PIPE_CSC_OUTPUT_COEFF_RV_GV(pipe), coeff[6] << 16 | coeff[7]);
+ I915_WRITE(PIPE_CSC_OUTPUT_COEFF_BV(pipe), coeff[8] << 16);
+
+ I915_WRITE(PIPE_CSC_OUTPUT_POSTOFF_HI(pipe), postoff[0]);
+ I915_WRITE(PIPE_CSC_OUTPUT_POSTOFF_ME(pipe), postoff[1]);
+ I915_WRITE(PIPE_CSC_OUTPUT_POSTOFF_LO(pipe), postoff[2]);
+}
+
+static bool ilk_csc_limited_range(const struct intel_crtc_state *crtc_state)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
/*
* FIXME if there's a gamma LUT after the CSC, we should
* do the range compression using the gamma LUT instead.
*/
- if (INTEL_GEN(dev_priv) >= 8 || IS_HASWELL(dev_priv))
- limited_color_range = crtc_state->limited_color_range;
+ return crtc_state->limited_color_range &&
+ (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv) ||
+ IS_GEN_RANGE(dev_priv, 9, 10));
+}
- if (crtc_state->output_format == INTEL_OUTPUT_FORMAT_YCBCR420 ||
- crtc_state->output_format == INTEL_OUTPUT_FORMAT_YCBCR444) {
- ilk_load_ycbcr_conversion_matrix(crtc);
- return;
- } else if (crtc_state->base.ctm) {
- struct drm_color_ctm *ctm = crtc_state->base.ctm->data;
- const u64 *input;
- u64 temp[9];
+static void ilk_csc_convert_ctm(const struct intel_crtc_state *crtc_state,
+ u16 coeffs[9])
+{
+ const struct drm_color_ctm *ctm = crtc_state->base.ctm->data;
+ const u64 *input;
+ u64 temp[9];
+ int i;
- if (limited_color_range)
- input = ctm_mult_by_limited(temp, ctm->matrix);
- else
- input = ctm->matrix;
+ if (ilk_csc_limited_range(crtc_state))
+ input = ctm_mult_by_limited(temp, ctm->matrix);
+ else
+ input = ctm->matrix;
+
+ /*
+ * Convert fixed point S31.32 input to format supported by the
+ * hardware.
+ */
+ for (i = 0; i < 9; i++) {
+ u64 abs_coeff = ((1ULL << 63) - 1) & input[i];
/*
- * Convert fixed point S31.32 input to format supported by the
+ * Clamp input value to min/max supported by
* hardware.
*/
- for (i = 0; i < ARRAY_SIZE(coeffs); i++) {
- u64 abs_coeff = ((1ULL << 63) - 1) & input[i];
-
- /*
- * Clamp input value to min/max supported by
- * hardware.
- */
- abs_coeff = clamp_val(abs_coeff, 0, CTM_COEFF_4_0 - 1);
-
- /* sign bit */
- if (CTM_COEFF_NEGATIVE(input[i]))
- coeffs[i] |= 1 << 15;
-
- if (abs_coeff < CTM_COEFF_0_125)
- coeffs[i] |= (3 << 12) |
- ILK_CSC_COEFF_FP(abs_coeff, 12);
- else if (abs_coeff < CTM_COEFF_0_25)
- coeffs[i] |= (2 << 12) |
- ILK_CSC_COEFF_FP(abs_coeff, 11);
- else if (abs_coeff < CTM_COEFF_0_5)
- coeffs[i] |= (1 << 12) |
- ILK_CSC_COEFF_FP(abs_coeff, 10);
- else if (abs_coeff < CTM_COEFF_1_0)
- coeffs[i] |= ILK_CSC_COEFF_FP(abs_coeff, 9);
- else if (abs_coeff < CTM_COEFF_2_0)
- coeffs[i] |= (7 << 12) |
- ILK_CSC_COEFF_FP(abs_coeff, 8);
- else
- coeffs[i] |= (6 << 12) |
- ILK_CSC_COEFF_FP(abs_coeff, 7);
- }
- } else {
- /*
- * Load an identity matrix if no coefficients are provided.
- *
- * TODO: Check what kind of values actually come out of the
- * pipe with these coeff/postoff values and adjust to get the
- * best accuracy. Perhaps we even need to take the bpc value
- * into consideration.
- */
- for (i = 0; i < 3; i++) {
- if (limited_color_range)
- coeffs[i * 3 + i] =
- ILK_CSC_COEFF_LIMITED_RANGE;
- else
- coeffs[i * 3 + i] = ILK_CSC_COEFF_1_0;
- }
+ abs_coeff = clamp_val(abs_coeff, 0, CTM_COEFF_4_0 - 1);
+
+ coeffs[i] = 0;
+
+ /* sign bit */
+ if (CTM_COEFF_NEGATIVE(input[i]))
+ coeffs[i] |= 1 << 15;
+
+ if (abs_coeff < CTM_COEFF_0_125)
+ coeffs[i] |= (3 << 12) |
+ ILK_CSC_COEFF_FP(abs_coeff, 12);
+ else if (abs_coeff < CTM_COEFF_0_25)
+ coeffs[i] |= (2 << 12) |
+ ILK_CSC_COEFF_FP(abs_coeff, 11);
+ else if (abs_coeff < CTM_COEFF_0_5)
+ coeffs[i] |= (1 << 12) |
+ ILK_CSC_COEFF_FP(abs_coeff, 10);
+ else if (abs_coeff < CTM_COEFF_1_0)
+ coeffs[i] |= ILK_CSC_COEFF_FP(abs_coeff, 9);
+ else if (abs_coeff < CTM_COEFF_2_0)
+ coeffs[i] |= (7 << 12) |
+ ILK_CSC_COEFF_FP(abs_coeff, 8);
+ else
+ coeffs[i] |= (6 << 12) |
+ ILK_CSC_COEFF_FP(abs_coeff, 7);
}
+}
- I915_WRITE(PIPE_CSC_COEFF_RY_GY(pipe), coeffs[0] << 16 | coeffs[1]);
- I915_WRITE(PIPE_CSC_COEFF_BY(pipe), coeffs[2] << 16);
-
- I915_WRITE(PIPE_CSC_COEFF_RU_GU(pipe), coeffs[3] << 16 | coeffs[4]);
- I915_WRITE(PIPE_CSC_COEFF_BU(pipe), coeffs[5] << 16);
-
- I915_WRITE(PIPE_CSC_COEFF_RV_GV(pipe), coeffs[6] << 16 | coeffs[7]);
- I915_WRITE(PIPE_CSC_COEFF_BV(pipe), coeffs[8] << 16);
+static void ilk_load_csc_matrix(const struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ bool limited_color_range = ilk_csc_limited_range(crtc_state);
- I915_WRITE(PIPE_CSC_PREOFF_HI(pipe), 0);
- I915_WRITE(PIPE_CSC_PREOFF_ME(pipe), 0);
- I915_WRITE(PIPE_CSC_PREOFF_LO(pipe), 0);
+ if (crtc_state->base.ctm) {
+ u16 coeff[9];
+
+ ilk_csc_convert_ctm(crtc_state, coeff);
+ ilk_update_pipe_csc(crtc, ilk_csc_off_zero, coeff,
+ limited_color_range ?
+ ilk_csc_postoff_limited_range :
+ ilk_csc_off_zero);
+ } else if (crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB) {
+ ilk_update_pipe_csc(crtc, ilk_csc_off_zero,
+ ilk_csc_coeff_rgb_to_ycbcr,
+ ilk_csc_postoff_rgb_to_ycbcr);
+ } else if (limited_color_range) {
+ ilk_update_pipe_csc(crtc, ilk_csc_off_zero,
+ ilk_csc_coeff_limited_range,
+ ilk_csc_postoff_limited_range);
+ } else if (crtc_state->csc_enable) {
+ /*
+ * On GLK+ both pipe CSC and degamma LUT are controlled
+ * by csc_enable. Hence for the cases where the degama
+ * LUT is needed but CSC is not we need to load an
+ * identity matrix.
+ */
+ WARN_ON(!IS_CANNONLAKE(dev_priv) && !IS_GEMINILAKE(dev_priv));
- if (INTEL_GEN(dev_priv) > 6) {
- u16 postoff = 0;
+ ilk_update_pipe_csc(crtc, ilk_csc_off_zero,
+ ilk_csc_coeff_identity,
+ ilk_csc_off_zero);
+ }
- if (limited_color_range)
- postoff = (16 * (1 << 12) / 255) & 0x1fff;
+ I915_WRITE(PIPE_CSC_MODE(crtc->pipe), crtc_state->csc_mode);
+}
- I915_WRITE(PIPE_CSC_POSTOFF_HI(pipe), postoff);
- I915_WRITE(PIPE_CSC_POSTOFF_ME(pipe), postoff);
- I915_WRITE(PIPE_CSC_POSTOFF_LO(pipe), postoff);
+static void icl_load_csc_matrix(const struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
- I915_WRITE(PIPE_CSC_MODE(pipe), 0);
- } else {
- u32 mode = CSC_MODE_YUV_TO_RGB;
+ if (crtc_state->base.ctm) {
+ u16 coeff[9];
- if (limited_color_range)
- mode |= CSC_BLACK_SCREEN_OFFSET;
+ ilk_csc_convert_ctm(crtc_state, coeff);
+ ilk_update_pipe_csc(crtc, ilk_csc_off_zero,
+ coeff, ilk_csc_off_zero);
+ }
- I915_WRITE(PIPE_CSC_MODE(pipe), mode);
+ if (crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB) {
+ icl_update_output_csc(crtc, ilk_csc_off_zero,
+ ilk_csc_coeff_rgb_to_ycbcr,
+ ilk_csc_postoff_rgb_to_ycbcr);
+ } else if (crtc_state->limited_color_range) {
+ icl_update_output_csc(crtc, ilk_csc_off_zero,
+ ilk_csc_coeff_limited_range,
+ ilk_csc_postoff_limited_range);
}
+
+ I915_WRITE(PIPE_CSC_MODE(crtc->pipe), crtc_state->csc_mode);
}
/*
struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
enum pipe pipe = crtc->pipe;
- u32 mode;
if (crtc_state->base.ctm) {
const struct drm_color_ctm *ctm = crtc_state->base.ctm->data;
I915_WRITE(CGM_PIPE_CSC_COEFF8(pipe), coeffs[8]);
}
- mode = (crtc_state->base.ctm ? CGM_PIPE_MODE_CSC : 0);
- if (!crtc_state_is_legacy_gamma(crtc_state)) {
- mode |= (crtc_state->base.degamma_lut ? CGM_PIPE_MODE_DEGAMMA : 0) |
- (crtc_state->base.gamma_lut ? CGM_PIPE_MODE_GAMMA : 0);
- }
- I915_WRITE(CGM_PIPE_MODE(pipe), mode);
+ I915_WRITE(CGM_PIPE_MODE(pipe), crtc_state->cgm_mode);
+}
+
+/* i965+ "10.6" bit interpolated format "even DW" (low 8 bits) */
+static u32 i965_lut_10p6_ldw(const struct drm_color_lut *color)
+{
+ return (color->red & 0xff) << 16 |
+ (color->green & 0xff) << 8 |
+ (color->blue & 0xff);
+}
+
+/* i965+ "10.6" interpolated format "odd DW" (high 8 bits) */
+static u32 i965_lut_10p6_udw(const struct drm_color_lut *color)
+{
+ return (color->red >> 8) << 16 |
+ (color->green >> 8) << 8 |
+ (color->blue >> 8);
+}
+
+static u32 ilk_lut_10(const struct drm_color_lut *color)
+{
+ return drm_color_lut_extract(color->red, 10) << 20 |
+ drm_color_lut_extract(color->green, 10) << 10 |
+ drm_color_lut_extract(color->blue, 10);
}
/* Loads the legacy palette/gamma unit for the CRTC. */
(drm_color_lut_extract(lut[i].green, 8) << 8) |
drm_color_lut_extract(lut[i].blue, 8);
- if (HAS_GMCH(dev_priv))
- I915_WRITE(PALETTE(pipe, i), word);
- else
- I915_WRITE(LGC_PALETTE(pipe, i), word);
- }
- } else {
- for (i = 0; i < 256; i++) {
- u32 word = (i << 16) | (i << 8) | i;
-
if (HAS_GMCH(dev_priv))
I915_WRITE(PALETTE(pipe, i), word);
else
i9xx_load_luts_internal(crtc_state, crtc_state->base.gamma_lut);
}
+static void i9xx_color_commit(const struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ enum pipe pipe = crtc->pipe;
+ u32 val;
+
+ val = I915_READ(PIPECONF(pipe));
+ val &= ~PIPECONF_GAMMA_MODE_MASK_I9XX;
+ val |= PIPECONF_GAMMA_MODE(crtc_state->gamma_mode);
+ I915_WRITE(PIPECONF(pipe), val);
+}
+
+static void ilk_color_commit(const struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ enum pipe pipe = crtc->pipe;
+ u32 val;
+
+ val = I915_READ(PIPECONF(pipe));
+ val &= ~PIPECONF_GAMMA_MODE_MASK_ILK;
+ val |= PIPECONF_GAMMA_MODE(crtc_state->gamma_mode);
+ I915_WRITE(PIPECONF(pipe), val);
+
+ ilk_load_csc_matrix(crtc_state);
+}
+
static void hsw_color_commit(const struct intel_crtc_state *crtc_state)
{
struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
ilk_load_csc_matrix(crtc_state);
}
-static void bdw_load_degamma_lut(const struct intel_crtc_state *crtc_state)
+static void skl_color_commit(const struct intel_crtc_state *crtc_state)
{
struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
- const struct drm_property_blob *degamma_lut = crtc_state->base.degamma_lut;
- u32 i, lut_size = INTEL_INFO(dev_priv)->color.degamma_lut_size;
enum pipe pipe = crtc->pipe;
+ u32 val = 0;
- I915_WRITE(PREC_PAL_INDEX(pipe),
- PAL_PREC_SPLIT_MODE | PAL_PREC_AUTO_INCREMENT);
+ /*
+ * We don't (yet) allow userspace to control the pipe background color,
+ * so force it to black, but apply pipe gamma and CSC appropriately
+ * so that its handling will match how we program our planes.
+ */
+ if (crtc_state->gamma_enable)
+ val |= SKL_BOTTOM_COLOR_GAMMA_ENABLE;
+ if (crtc_state->csc_enable)
+ val |= SKL_BOTTOM_COLOR_CSC_ENABLE;
+ I915_WRITE(SKL_BOTTOM_COLOR(pipe), val);
- if (degamma_lut) {
- const struct drm_color_lut *lut = degamma_lut->data;
+ I915_WRITE(GAMMA_MODE(crtc->pipe), crtc_state->gamma_mode);
- for (i = 0; i < lut_size; i++) {
- u32 word =
- drm_color_lut_extract(lut[i].red, 10) << 20 |
- drm_color_lut_extract(lut[i].green, 10) << 10 |
- drm_color_lut_extract(lut[i].blue, 10);
+ if (INTEL_GEN(dev_priv) >= 11)
+ icl_load_csc_matrix(crtc_state);
+ else
+ ilk_load_csc_matrix(crtc_state);
+}
- I915_WRITE(PREC_PAL_DATA(pipe), word);
- }
- } else {
- for (i = 0; i < lut_size; i++) {
- u32 v = (i * ((1 << 10) - 1)) / (lut_size - 1);
+static void i965_load_lut_10p6(struct intel_crtc *crtc,
+ const struct drm_property_blob *blob)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ const struct drm_color_lut *lut = blob->data;
+ int i, lut_size = drm_color_lut_size(blob);
+ enum pipe pipe = crtc->pipe;
- I915_WRITE(PREC_PAL_DATA(pipe),
- (v << 20) | (v << 10) | v);
- }
+ for (i = 0; i < lut_size - 1; i++) {
+ I915_WRITE(PALETTE(pipe, 2 * i + 0),
+ i965_lut_10p6_ldw(&lut[i]));
+ I915_WRITE(PALETTE(pipe, 2 * i + 1),
+ i965_lut_10p6_udw(&lut[i]));
}
+
+ I915_WRITE(PIPEGCMAX(pipe, 0), lut[i].red);
+ I915_WRITE(PIPEGCMAX(pipe, 1), lut[i].green);
+ I915_WRITE(PIPEGCMAX(pipe, 2), lut[i].blue);
}
-static void bdw_load_gamma_lut(const struct intel_crtc_state *crtc_state, u32 offset)
+static void i965_load_luts(const struct intel_crtc_state *crtc_state)
{
struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
- struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
const struct drm_property_blob *gamma_lut = crtc_state->base.gamma_lut;
- u32 i, lut_size = INTEL_INFO(dev_priv)->color.gamma_lut_size;
+
+ if (crtc_state->gamma_mode == GAMMA_MODE_MODE_8BIT)
+ i9xx_load_luts(crtc_state);
+ else
+ i965_load_lut_10p6(crtc, gamma_lut);
+}
+
+static void ilk_load_lut_10(struct intel_crtc *crtc,
+ const struct drm_property_blob *blob)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ const struct drm_color_lut *lut = blob->data;
+ int i, lut_size = drm_color_lut_size(blob);
enum pipe pipe = crtc->pipe;
- WARN_ON(offset & ~PAL_PREC_INDEX_VALUE_MASK);
+ for (i = 0; i < lut_size; i++)
+ I915_WRITE(PREC_PALETTE(pipe, i), ilk_lut_10(&lut[i]));
+}
- I915_WRITE(PREC_PAL_INDEX(pipe),
- (offset ? PAL_PREC_SPLIT_MODE : 0) |
- PAL_PREC_AUTO_INCREMENT |
- offset);
+static void ilk_load_luts(const struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ const struct drm_property_blob *gamma_lut = crtc_state->base.gamma_lut;
- if (gamma_lut) {
- const struct drm_color_lut *lut = gamma_lut->data;
+ if (crtc_state->gamma_mode == GAMMA_MODE_MODE_8BIT)
+ i9xx_load_luts(crtc_state);
+ else
+ ilk_load_lut_10(crtc, gamma_lut);
+}
- for (i = 0; i < lut_size; i++) {
- u32 word =
- (drm_color_lut_extract(lut[i].red, 10) << 20) |
- (drm_color_lut_extract(lut[i].green, 10) << 10) |
- drm_color_lut_extract(lut[i].blue, 10);
+static int ivb_lut_10_size(u32 prec_index)
+{
+ if (prec_index & PAL_PREC_SPLIT_MODE)
+ return 512;
+ else
+ return 1024;
+}
- I915_WRITE(PREC_PAL_DATA(pipe), word);
- }
+/*
+ * IVB/HSW Bspec / PAL_PREC_INDEX:
+ * "Restriction : Index auto increment mode is not
+ * supported and must not be enabled."
+ */
+static void ivb_load_lut_10(struct intel_crtc *crtc,
+ const struct drm_property_blob *blob,
+ u32 prec_index)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ int hw_lut_size = ivb_lut_10_size(prec_index);
+ const struct drm_color_lut *lut = blob->data;
+ int i, lut_size = drm_color_lut_size(blob);
+ enum pipe pipe = crtc->pipe;
- /* Program the max register to clamp values > 1.0. */
- i = lut_size - 1;
- I915_WRITE(PREC_PAL_GC_MAX(pipe, 0),
- drm_color_lut_extract(lut[i].red, 16));
- I915_WRITE(PREC_PAL_GC_MAX(pipe, 1),
- drm_color_lut_extract(lut[i].green, 16));
- I915_WRITE(PREC_PAL_GC_MAX(pipe, 2),
- drm_color_lut_extract(lut[i].blue, 16));
- } else {
- for (i = 0; i < lut_size; i++) {
- u32 v = (i * ((1 << 10) - 1)) / (lut_size - 1);
+ for (i = 0; i < hw_lut_size; i++) {
+ /* We discard half the user entries in split gamma mode */
+ const struct drm_color_lut *entry =
+ &lut[i * (lut_size - 1) / (hw_lut_size - 1)];
- I915_WRITE(PREC_PAL_DATA(pipe),
- (v << 20) | (v << 10) | v);
- }
+ I915_WRITE(PREC_PAL_INDEX(pipe), prec_index++);
+ I915_WRITE(PREC_PAL_DATA(pipe), ilk_lut_10(entry));
+ }
+
+ /*
+ * Reset the index, otherwise it prevents the legacy palette to be
+ * written properly.
+ */
+ I915_WRITE(PREC_PAL_INDEX(pipe), 0);
+}
- I915_WRITE(PREC_PAL_GC_MAX(pipe, 0), (1 << 16) - 1);
- I915_WRITE(PREC_PAL_GC_MAX(pipe, 1), (1 << 16) - 1);
- I915_WRITE(PREC_PAL_GC_MAX(pipe, 2), (1 << 16) - 1);
+/* On BDW+ the index auto increment mode actually works */
+static void bdw_load_lut_10(struct intel_crtc *crtc,
+ const struct drm_property_blob *blob,
+ u32 prec_index)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ int hw_lut_size = ivb_lut_10_size(prec_index);
+ const struct drm_color_lut *lut = blob->data;
+ int i, lut_size = drm_color_lut_size(blob);
+ enum pipe pipe = crtc->pipe;
+
+ I915_WRITE(PREC_PAL_INDEX(pipe), prec_index |
+ PAL_PREC_AUTO_INCREMENT);
+
+ for (i = 0; i < hw_lut_size; i++) {
+ /* We discard half the user entries in split gamma mode */
+ const struct drm_color_lut *entry =
+ &lut[i * (lut_size - 1) / (hw_lut_size - 1)];
+
+ I915_WRITE(PREC_PAL_DATA(pipe), ilk_lut_10(entry));
}
+
+ /*
+ * Reset the index, otherwise it prevents the legacy palette to be
+ * written properly.
+ */
+ I915_WRITE(PREC_PAL_INDEX(pipe), 0);
}
-/* Loads the palette/gamma unit for the CRTC on Broadwell+. */
-static void broadwell_load_luts(const struct intel_crtc_state *crtc_state)
+static void ivb_load_lut_10_max(struct intel_crtc *crtc)
{
- struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
enum pipe pipe = crtc->pipe;
- if (crtc_state_is_legacy_gamma(crtc_state)) {
+ /* Program the max register to clamp values > 1.0. */
+ I915_WRITE(PREC_PAL_EXT_GC_MAX(pipe, 0), 1 << 16);
+ I915_WRITE(PREC_PAL_EXT_GC_MAX(pipe, 1), 1 << 16);
+ I915_WRITE(PREC_PAL_EXT_GC_MAX(pipe, 2), 1 << 16);
+
+ /*
+ * Program the gc max 2 register to clamp values > 1.0.
+ * ToDo: Extend the ABI to be able to program values
+ * from 3.0 to 7.0
+ */
+ if (INTEL_GEN(dev_priv) >= 10 || IS_GEMINILAKE(dev_priv)) {
+ I915_WRITE(PREC_PAL_EXT2_GC_MAX(pipe, 0), 1 << 16);
+ I915_WRITE(PREC_PAL_EXT2_GC_MAX(pipe, 1), 1 << 16);
+ I915_WRITE(PREC_PAL_EXT2_GC_MAX(pipe, 2), 1 << 16);
+ }
+}
+
+static void ivb_load_luts(const struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ const struct drm_property_blob *gamma_lut = crtc_state->base.gamma_lut;
+ const struct drm_property_blob *degamma_lut = crtc_state->base.degamma_lut;
+
+ if (crtc_state->gamma_mode == GAMMA_MODE_MODE_8BIT) {
i9xx_load_luts(crtc_state);
+ } else if (crtc_state->gamma_mode == GAMMA_MODE_MODE_SPLIT) {
+ ivb_load_lut_10(crtc, degamma_lut, PAL_PREC_SPLIT_MODE |
+ PAL_PREC_INDEX_VALUE(0));
+ ivb_load_lut_10_max(crtc);
+ ivb_load_lut_10(crtc, gamma_lut, PAL_PREC_SPLIT_MODE |
+ PAL_PREC_INDEX_VALUE(512));
} else {
- bdw_load_degamma_lut(crtc_state);
- bdw_load_gamma_lut(crtc_state,
- INTEL_INFO(dev_priv)->color.degamma_lut_size);
+ const struct drm_property_blob *blob = gamma_lut ?: degamma_lut;
- /*
- * Reset the index, otherwise it prevents the legacy palette to be
- * written properly.
- */
- I915_WRITE(PREC_PAL_INDEX(pipe), 0);
+ ivb_load_lut_10(crtc, blob,
+ PAL_PREC_INDEX_VALUE(0));
+ ivb_load_lut_10_max(crtc);
+ }
+}
+
+static void bdw_load_luts(const struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ const struct drm_property_blob *gamma_lut = crtc_state->base.gamma_lut;
+ const struct drm_property_blob *degamma_lut = crtc_state->base.degamma_lut;
+
+ if (crtc_state->gamma_mode == GAMMA_MODE_MODE_8BIT) {
+ i9xx_load_luts(crtc_state);
+ } else if (crtc_state->gamma_mode == GAMMA_MODE_MODE_SPLIT) {
+ bdw_load_lut_10(crtc, degamma_lut, PAL_PREC_SPLIT_MODE |
+ PAL_PREC_INDEX_VALUE(0));
+ ivb_load_lut_10_max(crtc);
+ bdw_load_lut_10(crtc, gamma_lut, PAL_PREC_SPLIT_MODE |
+ PAL_PREC_INDEX_VALUE(512));
+ } else {
+ const struct drm_property_blob *blob = gamma_lut ?: degamma_lut;
+
+ bdw_load_lut_10(crtc, blob,
+ PAL_PREC_INDEX_VALUE(0));
+ ivb_load_lut_10_max(crtc);
}
}
struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
enum pipe pipe = crtc->pipe;
- const u32 lut_size = 33;
+ const u32 lut_size = INTEL_INFO(dev_priv)->color.degamma_lut_size;
+ const struct drm_color_lut *lut = crtc_state->base.degamma_lut->data;
u32 i;
/*
I915_WRITE(PRE_CSC_GAMC_INDEX(pipe), 0);
I915_WRITE(PRE_CSC_GAMC_INDEX(pipe), PRE_CSC_GAMC_AUTO_INCREMENT);
+ for (i = 0; i < lut_size; i++) {
+ /*
+ * First 33 entries represent range from 0 to 1.0
+ * 34th and 35th entry will represent extended range
+ * inputs 3.0 and 7.0 respectively, currently clamped
+ * at 1.0. Since the precision is 16bit, the user
+ * value can be directly filled to register.
+ * The pipe degamma table in GLK+ onwards doesn't
+ * support different values per channel, so this just
+ * programs green value which will be equal to Red and
+ * Blue into the lut registers.
+ * ToDo: Extend to max 7.0. Enable 32 bit input value
+ * as compared to just 16 to achieve this.
+ */
+ I915_WRITE(PRE_CSC_GAMC_DATA(pipe), lut[i].green);
+ }
+
+ /* Clamp values > 1.0. */
+ while (i++ < 35)
+ I915_WRITE(PRE_CSC_GAMC_DATA(pipe), 1 << 16);
+}
+
+static void glk_load_degamma_lut_linear(const struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ enum pipe pipe = crtc->pipe;
+ const u32 lut_size = INTEL_INFO(dev_priv)->color.degamma_lut_size;
+ u32 i;
+
/*
- * FIXME: The pipe degamma table in geminilake doesn't support
- * different values per channel, so this just loads a linear table.
+ * When setting the auto-increment bit, the hardware seems to
+ * ignore the index bits, so we need to reset it to index 0
+ * separately.
*/
+ I915_WRITE(PRE_CSC_GAMC_INDEX(pipe), 0);
+ I915_WRITE(PRE_CSC_GAMC_INDEX(pipe), PRE_CSC_GAMC_AUTO_INCREMENT);
+
for (i = 0; i < lut_size; i++) {
- u32 v = (i * (1 << 16)) / (lut_size - 1);
+ u32 v = (i << 16) / (lut_size - 1);
I915_WRITE(PRE_CSC_GAMC_DATA(pipe), v);
}
/* Clamp values > 1.0. */
while (i++ < 35)
- I915_WRITE(PRE_CSC_GAMC_DATA(pipe), (1 << 16));
+ I915_WRITE(PRE_CSC_GAMC_DATA(pipe), 1 << 16);
}
static void glk_load_luts(const struct intel_crtc_state *crtc_state)
{
+ const struct drm_property_blob *gamma_lut = crtc_state->base.gamma_lut;
struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
- struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
- enum pipe pipe = crtc->pipe;
- glk_load_degamma_lut(crtc_state);
+ /*
+ * On GLK+ both pipe CSC and degamma LUT are controlled
+ * by csc_enable. Hence for the cases where the CSC is
+ * needed but degamma LUT is not we need to load a
+ * linear degamma LUT. In fact we'll just always load
+ * the degama LUT so that we don't have to reload
+ * it every time the pipe CSC is being enabled.
+ */
+ if (crtc_state->base.degamma_lut)
+ glk_load_degamma_lut(crtc_state);
+ else
+ glk_load_degamma_lut_linear(crtc_state);
- if (crtc_state_is_legacy_gamma(crtc_state)) {
+ if (crtc_state->gamma_mode == GAMMA_MODE_MODE_8BIT) {
i9xx_load_luts(crtc_state);
} else {
- bdw_load_gamma_lut(crtc_state, 0);
+ bdw_load_lut_10(crtc, gamma_lut, PAL_PREC_INDEX_VALUE(0));
+ ivb_load_lut_10_max(crtc);
+ }
+}
- /*
- * Reset the index, otherwise it prevents the legacy palette to be
- * written properly.
- */
- I915_WRITE(PREC_PAL_INDEX(pipe), 0);
+static void icl_load_luts(const struct intel_crtc_state *crtc_state)
+{
+ const struct drm_property_blob *gamma_lut = crtc_state->base.gamma_lut;
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+
+ if (crtc_state->base.degamma_lut)
+ glk_load_degamma_lut(crtc_state);
+
+ if ((crtc_state->gamma_mode & GAMMA_MODE_MODE_MASK) ==
+ GAMMA_MODE_MODE_8BIT) {
+ i9xx_load_luts(crtc_state);
+ } else {
+ bdw_load_lut_10(crtc, gamma_lut, PAL_PREC_INDEX_VALUE(0));
+ ivb_load_lut_10_max(crtc);
}
}
cherryview_load_csc_matrix(crtc_state);
if (crtc_state_is_legacy_gamma(crtc_state)) {
- i9xx_load_luts_internal(crtc_state, gamma_lut);
+ i9xx_load_luts(crtc_state);
return;
}
I915_WRITE(CGM_PIPE_GAMMA(pipe, i, 1), word1);
}
}
-
- /*
- * Also program a linear LUT in the legacy block (behind the
- * CGM block).
- */
- i9xx_load_luts_internal(crtc_state, NULL);
}
void intel_color_load_luts(const struct intel_crtc_state *crtc_state)
{
struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
- if (dev_priv->display.color_commit)
- dev_priv->display.color_commit(crtc_state);
+ dev_priv->display.color_commit(crtc_state);
+}
+
+int intel_color_check(struct intel_crtc_state *crtc_state)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+
+ return dev_priv->display.color_check(crtc_state);
+}
+
+static bool need_plane_update(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state)
+{
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+
+ /*
+ * On pre-SKL the pipe gamma enable and pipe csc enable for
+ * the pipe bottom color are configured via the primary plane.
+ * We have to reconfigure that even if the plane is inactive.
+ */
+ return crtc_state->active_planes & BIT(plane->id) ||
+ (INTEL_GEN(dev_priv) < 9 &&
+ plane->id == PLANE_PRIMARY);
+}
+
+static int
+intel_color_add_affected_planes(struct intel_crtc_state *new_crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(new_crtc_state->base.crtc);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ struct intel_atomic_state *state =
+ to_intel_atomic_state(new_crtc_state->base.state);
+ const struct intel_crtc_state *old_crtc_state =
+ intel_atomic_get_old_crtc_state(state, crtc);
+ struct intel_plane *plane;
+
+ if (!new_crtc_state->base.active ||
+ drm_atomic_crtc_needs_modeset(&new_crtc_state->base))
+ return 0;
+
+ if (new_crtc_state->gamma_enable == old_crtc_state->gamma_enable &&
+ new_crtc_state->csc_enable == old_crtc_state->csc_enable)
+ return 0;
+
+ for_each_intel_plane_on_crtc(&dev_priv->drm, crtc, plane) {
+ struct intel_plane_state *plane_state;
+
+ if (!need_plane_update(plane, new_crtc_state))
+ continue;
+
+ plane_state = intel_atomic_get_plane_state(state, plane);
+ if (IS_ERR(plane_state))
+ return PTR_ERR(plane_state);
+
+ new_crtc_state->update_planes |= BIT(plane->id);
+ }
+
+ return 0;
}
static int check_lut_size(const struct drm_property_blob *lut, int expected)
return 0;
}
-int intel_color_check(struct intel_crtc_state *crtc_state)
+static int check_luts(const struct intel_crtc_state *crtc_state)
{
struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
const struct drm_property_blob *gamma_lut = crtc_state->base.gamma_lut;
int gamma_length, degamma_length;
u32 gamma_tests, degamma_tests;
+ /* Always allow legacy gamma LUT with no further checking. */
+ if (crtc_state_is_legacy_gamma(crtc_state))
+ return 0;
+
+ /* C8 relies on its palette being stored in the legacy LUT */
+ if (crtc_state->c8_planes)
+ return -EINVAL;
+
degamma_length = INTEL_INFO(dev_priv)->color.degamma_lut_size;
gamma_length = INTEL_INFO(dev_priv)->color.gamma_lut_size;
degamma_tests = INTEL_INFO(dev_priv)->color.degamma_lut_tests;
gamma_tests = INTEL_INFO(dev_priv)->color.gamma_lut_tests;
- /* Always allow legacy gamma LUT with no further checking. */
- if (crtc_state_is_legacy_gamma(crtc_state)) {
- crtc_state->gamma_mode = GAMMA_MODE_MODE_8BIT;
- return 0;
- }
-
if (check_lut_size(degamma_lut, degamma_length) ||
check_lut_size(gamma_lut, gamma_length))
return -EINVAL;
drm_color_lut_check(gamma_lut, gamma_tests))
return -EINVAL;
- if (INTEL_GEN(dev_priv) >= 10 || IS_GEMINILAKE(dev_priv))
- crtc_state->gamma_mode = GAMMA_MODE_MODE_10BIT;
- else if (INTEL_GEN(dev_priv) >= 9 || IS_BROADWELL(dev_priv))
- crtc_state->gamma_mode = GAMMA_MODE_MODE_SPLIT;
+ return 0;
+}
+
+static u32 i9xx_gamma_mode(struct intel_crtc_state *crtc_state)
+{
+ if (!crtc_state->gamma_enable ||
+ crtc_state_is_legacy_gamma(crtc_state))
+ return GAMMA_MODE_MODE_8BIT;
else
- crtc_state->gamma_mode = GAMMA_MODE_MODE_8BIT;
+ return GAMMA_MODE_MODE_10BIT; /* i965+ only */
+}
+
+static int i9xx_color_check(struct intel_crtc_state *crtc_state)
+{
+ int ret;
+
+ ret = check_luts(crtc_state);
+ if (ret)
+ return ret;
+
+ crtc_state->gamma_enable =
+ crtc_state->base.gamma_lut &&
+ !crtc_state->c8_planes;
+
+ crtc_state->gamma_mode = i9xx_gamma_mode(crtc_state);
+
+ ret = intel_color_add_affected_planes(crtc_state);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static u32 chv_cgm_mode(const struct intel_crtc_state *crtc_state)
+{
+ u32 cgm_mode = 0;
+
+ if (crtc_state_is_legacy_gamma(crtc_state))
+ return 0;
+
+ if (crtc_state->base.degamma_lut)
+ cgm_mode |= CGM_PIPE_MODE_DEGAMMA;
+ if (crtc_state->base.ctm)
+ cgm_mode |= CGM_PIPE_MODE_CSC;
+ if (crtc_state->base.gamma_lut)
+ cgm_mode |= CGM_PIPE_MODE_GAMMA;
+
+ return cgm_mode;
+}
+
+/*
+ * CHV color pipeline:
+ * u0.10 -> CGM degamma -> u0.14 -> CGM csc -> u0.14 -> CGM gamma ->
+ * u0.10 -> WGC csc -> u0.10 -> pipe gamma -> u0.10
+ *
+ * We always bypass the WGC csc and use the CGM csc
+ * instead since it has degamma and better precision.
+ */
+static int chv_color_check(struct intel_crtc_state *crtc_state)
+{
+ int ret;
+
+ ret = check_luts(crtc_state);
+ if (ret)
+ return ret;
+
+ /*
+ * Pipe gamma will be used only for the legacy LUT.
+ * Otherwise we bypass it and use the CGM gamma instead.
+ */
+ crtc_state->gamma_enable =
+ crtc_state_is_legacy_gamma(crtc_state) &&
+ !crtc_state->c8_planes;
+
+ crtc_state->gamma_mode = GAMMA_MODE_MODE_8BIT;
+
+ crtc_state->cgm_mode = chv_cgm_mode(crtc_state);
+
+ ret = intel_color_add_affected_planes(crtc_state);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static u32 ilk_gamma_mode(const struct intel_crtc_state *crtc_state)
+{
+ if (!crtc_state->gamma_enable ||
+ crtc_state_is_legacy_gamma(crtc_state))
+ return GAMMA_MODE_MODE_8BIT;
+ else
+ return GAMMA_MODE_MODE_10BIT;
+}
+
+static int ilk_color_check(struct intel_crtc_state *crtc_state)
+{
+ int ret;
+
+ ret = check_luts(crtc_state);
+ if (ret)
+ return ret;
+
+ crtc_state->gamma_enable =
+ crtc_state->base.gamma_lut &&
+ !crtc_state->c8_planes;
+
+ /*
+ * We don't expose the ctm on ilk/snb currently,
+ * nor do we enable YCbCr output. Also RGB limited
+ * range output is handled by the hw automagically.
+ */
+ crtc_state->csc_enable = false;
+
+ crtc_state->gamma_mode = ilk_gamma_mode(crtc_state);
+
+ crtc_state->csc_mode = 0;
+
+ ret = intel_color_add_affected_planes(crtc_state);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static u32 ivb_gamma_mode(const struct intel_crtc_state *crtc_state)
+{
+ if (!crtc_state->gamma_enable ||
+ crtc_state_is_legacy_gamma(crtc_state))
+ return GAMMA_MODE_MODE_8BIT;
+ else if (crtc_state->base.gamma_lut &&
+ crtc_state->base.degamma_lut)
+ return GAMMA_MODE_MODE_SPLIT;
+ else
+ return GAMMA_MODE_MODE_10BIT;
+}
+
+static u32 ivb_csc_mode(const struct intel_crtc_state *crtc_state)
+{
+ bool limited_color_range = ilk_csc_limited_range(crtc_state);
+
+ /*
+ * CSC comes after the LUT in degamma, RGB->YCbCr,
+ * and RGB full->limited range mode.
+ */
+ if (crtc_state->base.degamma_lut ||
+ crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB ||
+ limited_color_range)
+ return 0;
+
+ return CSC_POSITION_BEFORE_GAMMA;
+}
+
+static int ivb_color_check(struct intel_crtc_state *crtc_state)
+{
+ bool limited_color_range = ilk_csc_limited_range(crtc_state);
+ int ret;
+
+ ret = check_luts(crtc_state);
+ if (ret)
+ return ret;
+
+ crtc_state->gamma_enable =
+ (crtc_state->base.gamma_lut ||
+ crtc_state->base.degamma_lut) &&
+ !crtc_state->c8_planes;
+
+ crtc_state->csc_enable =
+ crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB ||
+ crtc_state->base.ctm || limited_color_range;
+
+ crtc_state->gamma_mode = ivb_gamma_mode(crtc_state);
+
+ crtc_state->csc_mode = ivb_csc_mode(crtc_state);
+
+ ret = intel_color_add_affected_planes(crtc_state);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static u32 glk_gamma_mode(const struct intel_crtc_state *crtc_state)
+{
+ if (!crtc_state->gamma_enable ||
+ crtc_state_is_legacy_gamma(crtc_state))
+ return GAMMA_MODE_MODE_8BIT;
+ else
+ return GAMMA_MODE_MODE_10BIT;
+}
+
+static int glk_color_check(struct intel_crtc_state *crtc_state)
+{
+ int ret;
+
+ ret = check_luts(crtc_state);
+ if (ret)
+ return ret;
+
+ crtc_state->gamma_enable =
+ crtc_state->base.gamma_lut &&
+ !crtc_state->c8_planes;
+
+ /* On GLK+ degamma LUT is controlled by csc_enable */
+ crtc_state->csc_enable =
+ crtc_state->base.degamma_lut ||
+ crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB ||
+ crtc_state->base.ctm || crtc_state->limited_color_range;
+
+ crtc_state->gamma_mode = glk_gamma_mode(crtc_state);
+
+ crtc_state->csc_mode = 0;
+
+ ret = intel_color_add_affected_planes(crtc_state);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static u32 icl_gamma_mode(const struct intel_crtc_state *crtc_state)
+{
+ u32 gamma_mode = 0;
+
+ if (crtc_state->base.degamma_lut)
+ gamma_mode |= PRE_CSC_GAMMA_ENABLE;
+
+ if (crtc_state->base.gamma_lut &&
+ !crtc_state->c8_planes)
+ gamma_mode |= POST_CSC_GAMMA_ENABLE;
+
+ if (!crtc_state->base.gamma_lut ||
+ crtc_state_is_legacy_gamma(crtc_state))
+ gamma_mode |= GAMMA_MODE_MODE_8BIT;
+ else
+ gamma_mode |= GAMMA_MODE_MODE_10BIT;
+
+ return gamma_mode;
+}
+
+static u32 icl_csc_mode(const struct intel_crtc_state *crtc_state)
+{
+ u32 csc_mode = 0;
+
+ if (crtc_state->base.ctm)
+ csc_mode |= ICL_CSC_ENABLE;
+
+ if (crtc_state->output_format != INTEL_OUTPUT_FORMAT_RGB ||
+ crtc_state->limited_color_range)
+ csc_mode |= ICL_OUTPUT_CSC_ENABLE;
+
+ return csc_mode;
+}
+
+static int icl_color_check(struct intel_crtc_state *crtc_state)
+{
+ int ret;
+
+ ret = check_luts(crtc_state);
+ if (ret)
+ return ret;
+
+ crtc_state->gamma_mode = icl_gamma_mode(crtc_state);
+
+ crtc_state->csc_mode = icl_csc_mode(crtc_state);
return 0;
}
void intel_color_init(struct intel_crtc *crtc)
{
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ bool has_ctm = INTEL_INFO(dev_priv)->color.degamma_lut_size != 0;
drm_mode_crtc_set_gamma_size(&crtc->base, 256);
- if (IS_CHERRYVIEW(dev_priv)) {
- dev_priv->display.load_luts = cherryview_load_luts;
- } else if (IS_HASWELL(dev_priv)) {
- dev_priv->display.load_luts = i9xx_load_luts;
- dev_priv->display.color_commit = hsw_color_commit;
- } else if (IS_BROADWELL(dev_priv) || IS_GEN9_BC(dev_priv) ||
- IS_BROXTON(dev_priv)) {
- dev_priv->display.load_luts = broadwell_load_luts;
- dev_priv->display.color_commit = hsw_color_commit;
- } else if (IS_GEMINILAKE(dev_priv) || IS_CANNONLAKE(dev_priv)) {
- dev_priv->display.load_luts = glk_load_luts;
- dev_priv->display.color_commit = hsw_color_commit;
+ if (HAS_GMCH(dev_priv)) {
+ if (IS_CHERRYVIEW(dev_priv)) {
+ dev_priv->display.color_check = chv_color_check;
+ dev_priv->display.color_commit = i9xx_color_commit;
+ dev_priv->display.load_luts = cherryview_load_luts;
+ } else if (INTEL_GEN(dev_priv) >= 4) {
+ dev_priv->display.color_check = i9xx_color_check;
+ dev_priv->display.color_commit = i9xx_color_commit;
+ dev_priv->display.load_luts = i965_load_luts;
+ } else {
+ dev_priv->display.color_check = i9xx_color_check;
+ dev_priv->display.color_commit = i9xx_color_commit;
+ dev_priv->display.load_luts = i9xx_load_luts;
+ }
} else {
- dev_priv->display.load_luts = i9xx_load_luts;
+ if (INTEL_GEN(dev_priv) >= 11)
+ dev_priv->display.color_check = icl_color_check;
+ else if (INTEL_GEN(dev_priv) >= 10 || IS_GEMINILAKE(dev_priv))
+ dev_priv->display.color_check = glk_color_check;
+ else if (INTEL_GEN(dev_priv) >= 7)
+ dev_priv->display.color_check = ivb_color_check;
+ else
+ dev_priv->display.color_check = ilk_color_check;
+
+ if (INTEL_GEN(dev_priv) >= 9)
+ dev_priv->display.color_commit = skl_color_commit;
+ else if (IS_BROADWELL(dev_priv) || IS_HASWELL(dev_priv))
+ dev_priv->display.color_commit = hsw_color_commit;
+ else
+ dev_priv->display.color_commit = ilk_color_commit;
+
+ if (INTEL_GEN(dev_priv) >= 11)
+ dev_priv->display.load_luts = icl_load_luts;
+ else if (IS_CANNONLAKE(dev_priv) || IS_GEMINILAKE(dev_priv))
+ dev_priv->display.load_luts = glk_load_luts;
+ else if (INTEL_GEN(dev_priv) >= 8)
+ dev_priv->display.load_luts = bdw_load_luts;
+ else if (INTEL_GEN(dev_priv) >= 7)
+ dev_priv->display.load_luts = ivb_load_luts;
+ else
+ dev_priv->display.load_luts = ilk_load_luts;
}
- /* Enable color management support when we have degamma & gamma LUTs. */
- if (INTEL_INFO(dev_priv)->color.degamma_lut_size != 0 &&
- INTEL_INFO(dev_priv)->color.gamma_lut_size != 0)
- drm_crtc_enable_color_mgmt(&crtc->base,
- INTEL_INFO(dev_priv)->color.degamma_lut_size,
- true,
- INTEL_INFO(dev_priv)->color.gamma_lut_size);
+ drm_crtc_enable_color_mgmt(&crtc->base,
+ INTEL_INFO(dev_priv)->color.degamma_lut_size,
+ has_ctm,
+ INTEL_INFO(dev_priv)->color.gamma_lut_size);
}
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_COLOR_H__
+#define __INTEL_COLOR_H__
+
+struct intel_crtc_state;
+struct intel_crtc;
+
+void intel_color_init(struct intel_crtc *crtc);
+int intel_color_check(struct intel_crtc_state *crtc_state);
+void intel_color_commit(const struct intel_crtc_state *crtc_state);
+void intel_color_load_luts(const struct intel_crtc_state *crtc_state);
+
+#endif /* __INTEL_COLOR_H__ */
for_each_combo_port_reverse(dev_priv, port) {
u32 val;
- if (!icl_combo_phy_verify_state(dev_priv, port))
+ if (port == PORT_A &&
+ !icl_combo_phy_verify_state(dev_priv, port))
DRM_WARN("Port %c combo PHY HW state changed unexpectedly\n",
port_name(port));
* DEALINGS IN THE SOFTWARE.
*/
-#include <linux/slab.h>
#include <linux/i2c.h>
+#include <linux/slab.h>
+
#include <drm/drm_atomic_helper.h>
#include <drm/drm_edid.h>
-#include "intel_drv.h"
+
#include "i915_drv.h"
+#include "intel_connector.h"
+#include "intel_drv.h"
+#include "intel_hdcp.h"
+#include "intel_panel.h"
int intel_connector_init(struct intel_connector *connector)
{
kfree(intel_connector->detect_edid);
+ intel_hdcp_cleanup(intel_connector);
+
if (!IS_ERR_OR_NULL(intel_connector->edid))
kfree(intel_connector->edid);
connector->dev->mode_config.aspect_ratio_property,
DRM_MODE_PICTURE_ASPECT_NONE);
}
+
+void
+intel_attach_colorspace_property(struct drm_connector *connector)
+{
+ if (!drm_mode_create_colorspace_property(connector))
+ drm_object_attach_property(&connector->base,
+ connector->colorspace_property, 0);
+}
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_CONNECTOR_H__
+#define __INTEL_CONNECTOR_H__
+
+#include "intel_display.h"
+
+struct drm_connector;
+struct edid;
+struct i2c_adapter;
+struct intel_connector;
+struct intel_encoder;
+
+int intel_connector_init(struct intel_connector *connector);
+struct intel_connector *intel_connector_alloc(void);
+void intel_connector_free(struct intel_connector *connector);
+void intel_connector_destroy(struct drm_connector *connector);
+int intel_connector_register(struct drm_connector *connector);
+void intel_connector_unregister(struct drm_connector *connector);
+void intel_connector_attach_encoder(struct intel_connector *connector,
+ struct intel_encoder *encoder);
+bool intel_connector_get_hw_state(struct intel_connector *connector);
+enum pipe intel_connector_get_pipe(struct intel_connector *connector);
+int intel_connector_update_modes(struct drm_connector *connector,
+ struct edid *edid);
+int intel_ddc_get_modes(struct drm_connector *c, struct i2c_adapter *adapter);
+void intel_attach_force_audio_property(struct drm_connector *connector);
+void intel_attach_broadcast_rgb_property(struct drm_connector *connector);
+void intel_attach_aspect_ratio_property(struct drm_connector *connector);
+void intel_attach_colorspace_property(struct drm_connector *connector);
+
+#endif /* __INTEL_CONNECTOR_H__ */
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2019 Intel Corporation
+ */
+
+#include "i915_drv.h"
+#include "i915_gem_context.h"
+#include "i915_globals.h"
+#include "intel_context.h"
+#include "intel_ringbuffer.h"
+
+static struct i915_global_context {
+ struct i915_global base;
+ struct kmem_cache *slab_ce;
+} global;
+
+struct intel_context *intel_context_alloc(void)
+{
+ return kmem_cache_zalloc(global.slab_ce, GFP_KERNEL);
+}
+
+void intel_context_free(struct intel_context *ce)
+{
+ kmem_cache_free(global.slab_ce, ce);
+}
+
+struct intel_context *
+intel_context_lookup(struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine)
+{
+ struct intel_context *ce = NULL;
+ struct rb_node *p;
+
+ spin_lock(&ctx->hw_contexts_lock);
+ p = ctx->hw_contexts.rb_node;
+ while (p) {
+ struct intel_context *this =
+ rb_entry(p, struct intel_context, node);
+
+ if (this->engine == engine) {
+ GEM_BUG_ON(this->gem_context != ctx);
+ ce = this;
+ break;
+ }
+
+ if (this->engine < engine)
+ p = p->rb_right;
+ else
+ p = p->rb_left;
+ }
+ spin_unlock(&ctx->hw_contexts_lock);
+
+ return ce;
+}
+
+struct intel_context *
+__intel_context_insert(struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine,
+ struct intel_context *ce)
+{
+ struct rb_node **p, *parent;
+ int err = 0;
+
+ spin_lock(&ctx->hw_contexts_lock);
+
+ parent = NULL;
+ p = &ctx->hw_contexts.rb_node;
+ while (*p) {
+ struct intel_context *this;
+
+ parent = *p;
+ this = rb_entry(parent, struct intel_context, node);
+
+ if (this->engine == engine) {
+ err = -EEXIST;
+ ce = this;
+ break;
+ }
+
+ if (this->engine < engine)
+ p = &parent->rb_right;
+ else
+ p = &parent->rb_left;
+ }
+ if (!err) {
+ rb_link_node(&ce->node, parent, p);
+ rb_insert_color(&ce->node, &ctx->hw_contexts);
+ }
+
+ spin_unlock(&ctx->hw_contexts_lock);
+
+ return ce;
+}
+
+void __intel_context_remove(struct intel_context *ce)
+{
+ struct i915_gem_context *ctx = ce->gem_context;
+
+ spin_lock(&ctx->hw_contexts_lock);
+ rb_erase(&ce->node, &ctx->hw_contexts);
+ spin_unlock(&ctx->hw_contexts_lock);
+}
+
+static struct intel_context *
+intel_context_instance(struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine)
+{
+ struct intel_context *ce, *pos;
+
+ ce = intel_context_lookup(ctx, engine);
+ if (likely(ce))
+ return ce;
+
+ ce = intel_context_alloc();
+ if (!ce)
+ return ERR_PTR(-ENOMEM);
+
+ intel_context_init(ce, ctx, engine);
+
+ pos = __intel_context_insert(ctx, engine, ce);
+ if (unlikely(pos != ce)) /* Beaten! Use their HW context instead */
+ intel_context_free(ce);
+
+ GEM_BUG_ON(intel_context_lookup(ctx, engine) != pos);
+ return pos;
+}
+
+struct intel_context *
+intel_context_pin_lock(struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine)
+ __acquires(ce->pin_mutex)
+{
+ struct intel_context *ce;
+
+ ce = intel_context_instance(ctx, engine);
+ if (IS_ERR(ce))
+ return ce;
+
+ if (mutex_lock_interruptible(&ce->pin_mutex))
+ return ERR_PTR(-EINTR);
+
+ return ce;
+}
+
+struct intel_context *
+intel_context_pin(struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine)
+{
+ struct intel_context *ce;
+ int err;
+
+ ce = intel_context_instance(ctx, engine);
+ if (IS_ERR(ce))
+ return ce;
+
+ if (likely(atomic_inc_not_zero(&ce->pin_count)))
+ return ce;
+
+ if (mutex_lock_interruptible(&ce->pin_mutex))
+ return ERR_PTR(-EINTR);
+
+ if (likely(!atomic_read(&ce->pin_count))) {
+ err = ce->ops->pin(ce);
+ if (err)
+ goto err;
+
+ i915_gem_context_get(ctx);
+ GEM_BUG_ON(ce->gem_context != ctx);
+
+ mutex_lock(&ctx->mutex);
+ list_add(&ce->active_link, &ctx->active_engines);
+ mutex_unlock(&ctx->mutex);
+
+ intel_context_get(ce);
+ smp_mb__before_atomic(); /* flush pin before it is visible */
+ }
+
+ atomic_inc(&ce->pin_count);
+ GEM_BUG_ON(!intel_context_is_pinned(ce)); /* no overflow! */
+
+ mutex_unlock(&ce->pin_mutex);
+ return ce;
+
+err:
+ mutex_unlock(&ce->pin_mutex);
+ return ERR_PTR(err);
+}
+
+void intel_context_unpin(struct intel_context *ce)
+{
+ if (likely(atomic_add_unless(&ce->pin_count, -1, 1)))
+ return;
+
+ /* We may be called from inside intel_context_pin() to evict another */
+ intel_context_get(ce);
+ mutex_lock_nested(&ce->pin_mutex, SINGLE_DEPTH_NESTING);
+
+ if (likely(atomic_dec_and_test(&ce->pin_count))) {
+ ce->ops->unpin(ce);
+
+ mutex_lock(&ce->gem_context->mutex);
+ list_del(&ce->active_link);
+ mutex_unlock(&ce->gem_context->mutex);
+
+ i915_gem_context_put(ce->gem_context);
+ intel_context_put(ce);
+ }
+
+ mutex_unlock(&ce->pin_mutex);
+ intel_context_put(ce);
+}
+
+static void intel_context_retire(struct i915_active_request *active,
+ struct i915_request *rq)
+{
+ struct intel_context *ce =
+ container_of(active, typeof(*ce), active_tracker);
+
+ intel_context_unpin(ce);
+}
+
+void
+intel_context_init(struct intel_context *ce,
+ struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine)
+{
+ kref_init(&ce->ref);
+
+ ce->gem_context = ctx;
+ ce->engine = engine;
+ ce->ops = engine->cops;
+
+ INIT_LIST_HEAD(&ce->signal_link);
+ INIT_LIST_HEAD(&ce->signals);
+
+ mutex_init(&ce->pin_mutex);
+
+ /* Use the whole device by default */
+ ce->sseu = intel_device_default_sseu(ctx->i915);
+
+ i915_active_request_init(&ce->active_tracker,
+ NULL, intel_context_retire);
+}
+
+static void i915_global_context_shrink(void)
+{
+ kmem_cache_shrink(global.slab_ce);
+}
+
+static void i915_global_context_exit(void)
+{
+ kmem_cache_destroy(global.slab_ce);
+}
+
+static struct i915_global_context global = { {
+ .shrink = i915_global_context_shrink,
+ .exit = i915_global_context_exit,
+} };
+
+int __init i915_global_context_init(void)
+{
+ global.slab_ce = KMEM_CACHE(intel_context, SLAB_HWCACHE_ALIGN);
+ if (!global.slab_ce)
+ return -ENOMEM;
+
+ i915_global_register(&global.base);
+ return 0;
+}
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_CONTEXT_H__
+#define __INTEL_CONTEXT_H__
+
+#include <linux/lockdep.h>
+
+#include "intel_context_types.h"
+#include "intel_engine_types.h"
+
+struct intel_context *intel_context_alloc(void);
+void intel_context_free(struct intel_context *ce);
+
+void intel_context_init(struct intel_context *ce,
+ struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine);
+
+/**
+ * intel_context_lookup - Find the matching HW context for this (ctx, engine)
+ * @ctx - the parent GEM context
+ * @engine - the target HW engine
+ *
+ * May return NULL if the HW context hasn't been instantiated (i.e. unused).
+ */
+struct intel_context *
+intel_context_lookup(struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine);
+
+/**
+ * intel_context_pin_lock - Stablises the 'pinned' status of the HW context
+ * @ctx - the parent GEM context
+ * @engine - the target HW engine
+ *
+ * Acquire a lock on the pinned status of the HW context, such that the context
+ * can neither be bound to the GPU or unbound whilst the lock is held, i.e.
+ * intel_context_is_pinned() remains stable.
+ */
+struct intel_context *
+intel_context_pin_lock(struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine);
+
+static inline bool
+intel_context_is_pinned(struct intel_context *ce)
+{
+ return atomic_read(&ce->pin_count);
+}
+
+static inline void intel_context_pin_unlock(struct intel_context *ce)
+__releases(ce->pin_mutex)
+{
+ mutex_unlock(&ce->pin_mutex);
+}
+
+struct intel_context *
+__intel_context_insert(struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine,
+ struct intel_context *ce);
+void
+__intel_context_remove(struct intel_context *ce);
+
+struct intel_context *
+intel_context_pin(struct i915_gem_context *ctx, struct intel_engine_cs *engine);
+
+static inline void __intel_context_pin(struct intel_context *ce)
+{
+ GEM_BUG_ON(!intel_context_is_pinned(ce));
+ atomic_inc(&ce->pin_count);
+}
+
+void intel_context_unpin(struct intel_context *ce);
+
+static inline struct intel_context *intel_context_get(struct intel_context *ce)
+{
+ kref_get(&ce->ref);
+ return ce;
+}
+
+static inline void intel_context_put(struct intel_context *ce)
+{
+ kref_put(&ce->ref, ce->ops->destroy);
+}
+
+#endif /* __INTEL_CONTEXT_H__ */
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_CONTEXT_TYPES__
+#define __INTEL_CONTEXT_TYPES__
+
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/rbtree.h>
+#include <linux/types.h>
+
+#include "i915_active_types.h"
+
+struct i915_gem_context;
+struct i915_vma;
+struct intel_context;
+struct intel_ring;
+
+struct intel_context_ops {
+ int (*pin)(struct intel_context *ce);
+ void (*unpin)(struct intel_context *ce);
+
+ void (*reset)(struct intel_context *ce);
+ void (*destroy)(struct kref *kref);
+};
+
+/*
+ * Powergating configuration for a particular (context,engine).
+ */
+struct intel_sseu {
+ u8 slice_mask;
+ u8 subslice_mask;
+ u8 min_eus_per_subslice;
+ u8 max_eus_per_subslice;
+};
+
+struct intel_context {
+ struct kref ref;
+
+ struct i915_gem_context *gem_context;
+ struct intel_engine_cs *engine;
+ struct intel_engine_cs *active;
+
+ struct list_head active_link;
+ struct list_head signal_link;
+ struct list_head signals;
+
+ struct i915_vma *state;
+ struct intel_ring *ring;
+
+ u32 *lrc_reg_state;
+ u64 lrc_desc;
+
+ atomic_t pin_count;
+ struct mutex pin_mutex; /* guards pinning and associated on-gpuing */
+
+ /**
+ * active_tracker: Active tracker for the external rq activity
+ * on this intel_context object.
+ */
+ struct i915_active_request active_tracker;
+
+ const struct intel_context_ops *ops;
+ struct rb_node node;
+
+ /** sseu: Control eu/slice partitioning */
+ struct intel_sseu sseu;
+};
+
+#endif /* __INTEL_CONTEXT_TYPES__ */
#include <linux/dmi.h>
#include <linux/i2c.h>
#include <linux/slab.h>
+
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
#include <drm/drm_edid.h>
#include <drm/drm_probe_helper.h>
-#include "intel_drv.h"
#include <drm/i915_drm.h>
+
#include "i915_drv.h"
+#include "intel_connector.h"
+#include "intel_crt.h"
+#include "intel_ddi.h"
+#include "intel_drv.h"
/* Here's the desired hotplug mode */
#define ADPA_HOTPLUG_BITS (ADPA_CRT_HOTPLUG_PERIOD_128 | \
I915_WRITE(crt->adpa_reg, adpa);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
crt->adpa_reg,
ADPA_CRT_HOTPLUG_FORCE_TRIGGER, 0,
1000))
I915_WRITE(crt->adpa_reg, adpa);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
crt->adpa_reg,
ADPA_CRT_HOTPLUG_FORCE_TRIGGER, 0,
1000)) {
CRT_HOTPLUG_FORCE_DETECT,
CRT_HOTPLUG_FORCE_DETECT);
/* wait for FORCE_DETECT to go off */
- if (intel_wait_for_register(dev_priv, PORT_HOTPLUG_EN,
+ if (intel_wait_for_register(&dev_priv->uncore, PORT_HOTPLUG_EN,
CRT_HOTPLUG_FORCE_DETECT, 0,
1000))
DRM_DEBUG_KMS("timed out waiting for FORCE_DETECT to go off");
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_CRT_H__
+#define __INTEL_CRT_H__
+
+#include "i915_reg.h"
+
+enum pipe;
+struct drm_encoder;
+struct drm_i915_private;
+struct drm_i915_private;
+
+bool intel_crt_port_enabled(struct drm_i915_private *dev_priv,
+ i915_reg_t adpa_reg, enum pipe *pipe);
+void intel_crt_init(struct drm_i915_private *dev_priv);
+void intel_crt_reset(struct drm_encoder *encoder);
+
+#endif /* __INTEL_CRT_H__ */
* IN THE SOFTWARE.
*
*/
+
#include <linux/firmware.h>
+
#include "i915_drv.h"
#include "i915_reg.h"
+#include "intel_csr.h"
/**
* DOC: csr support for dmc
if (INTEL_GEN(dev_priv) >= 12) {
/* Allow to load fw via parameter using the last known size */
csr->max_fw_size = GEN12_CSR_MAX_FW_SIZE;
- } else if (IS_ICELAKE(dev_priv)) {
+ } else if (IS_GEN(dev_priv, 11)) {
csr->fw_path = ICL_CSR_PATH;
csr->required_version = ICL_CSR_VERSION_REQUIRED;
csr->max_fw_size = ICL_CSR_MAX_FW_SIZE;
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_CSR_H__
+#define __INTEL_CSR_H__
+
+struct drm_i915_private;
+
+void intel_csr_ucode_init(struct drm_i915_private *i915);
+void intel_csr_load_program(struct drm_i915_private *i915);
+void intel_csr_ucode_fini(struct drm_i915_private *i915);
+void intel_csr_ucode_suspend(struct drm_i915_private *i915);
+void intel_csr_ucode_resume(struct drm_i915_private *i915);
+
+#endif /* __INTEL_CSR_H__ */
*/
#include <drm/drm_scdc_helper.h>
+
#include "i915_drv.h"
+#include "intel_audio.h"
+#include "intel_connector.h"
+#include "intel_ddi.h"
+#include "intel_dp.h"
#include "intel_drv.h"
#include "intel_dsi.h"
+#include "intel_hdcp.h"
+#include "intel_hdmi.h"
+#include "intel_lspcon.h"
+#include "intel_panel.h"
+#include "intel_psr.h"
struct ddi_buf_trans {
u32 trans1; /* balance leg enable, de-emph level */
level = dev_priv->vbt.ddi_port_info[port].hdmi_level_shift;
- if (IS_ICELAKE(dev_priv)) {
+ if (INTEL_GEN(dev_priv) >= 11) {
if (intel_port_is_combophy(dev_priv, port))
icl_get_combo_buf_trans(dev_priv, port, INTEL_OUTPUT_HDMI,
0, &n_entries);
return (refclk * n * 100) / (p * r);
}
-static int skl_calc_wrpll_link(struct drm_i915_private *dev_priv,
- enum intel_dpll_id pll_id)
+static int skl_calc_wrpll_link(const struct intel_dpll_hw_state *pll_state)
{
- i915_reg_t cfgcr1_reg, cfgcr2_reg;
- u32 cfgcr1_val, cfgcr2_val;
u32 p0, p1, p2, dco_freq;
- cfgcr1_reg = DPLL_CFGCR1(pll_id);
- cfgcr2_reg = DPLL_CFGCR2(pll_id);
-
- cfgcr1_val = I915_READ(cfgcr1_reg);
- cfgcr2_val = I915_READ(cfgcr2_reg);
-
- p0 = cfgcr2_val & DPLL_CFGCR2_PDIV_MASK;
- p2 = cfgcr2_val & DPLL_CFGCR2_KDIV_MASK;
+ p0 = pll_state->cfgcr2 & DPLL_CFGCR2_PDIV_MASK;
+ p2 = pll_state->cfgcr2 & DPLL_CFGCR2_KDIV_MASK;
- if (cfgcr2_val & DPLL_CFGCR2_QDIV_MODE(1))
- p1 = (cfgcr2_val & DPLL_CFGCR2_QDIV_RATIO_MASK) >> 8;
+ if (pll_state->cfgcr2 & DPLL_CFGCR2_QDIV_MODE(1))
+ p1 = (pll_state->cfgcr2 & DPLL_CFGCR2_QDIV_RATIO_MASK) >> 8;
else
p1 = 1;
break;
}
- dco_freq = (cfgcr1_val & DPLL_CFGCR1_DCO_INTEGER_MASK) * 24 * 1000;
+ dco_freq = (pll_state->cfgcr1 & DPLL_CFGCR1_DCO_INTEGER_MASK)
+ * 24 * 1000;
- dco_freq += (((cfgcr1_val & DPLL_CFGCR1_DCO_FRACTION_MASK) >> 9) * 24 *
- 1000) / 0x8000;
+ dco_freq += (((pll_state->cfgcr1 & DPLL_CFGCR1_DCO_FRACTION_MASK) >> 9)
+ * 24 * 1000) / 0x8000;
if (WARN_ON(p0 == 0 || p1 == 0 || p2 == 0))
return 0;
}
int cnl_calc_wrpll_link(struct drm_i915_private *dev_priv,
- enum intel_dpll_id pll_id)
+ struct intel_dpll_hw_state *pll_state)
{
- u32 cfgcr0, cfgcr1;
u32 p0, p1, p2, dco_freq, ref_clock;
- if (INTEL_GEN(dev_priv) >= 11) {
- cfgcr0 = I915_READ(ICL_DPLL_CFGCR0(pll_id));
- cfgcr1 = I915_READ(ICL_DPLL_CFGCR1(pll_id));
- } else {
- cfgcr0 = I915_READ(CNL_DPLL_CFGCR0(pll_id));
- cfgcr1 = I915_READ(CNL_DPLL_CFGCR1(pll_id));
- }
-
- p0 = cfgcr1 & DPLL_CFGCR1_PDIV_MASK;
- p2 = cfgcr1 & DPLL_CFGCR1_KDIV_MASK;
+ p0 = pll_state->cfgcr1 & DPLL_CFGCR1_PDIV_MASK;
+ p2 = pll_state->cfgcr1 & DPLL_CFGCR1_KDIV_MASK;
- if (cfgcr1 & DPLL_CFGCR1_QDIV_MODE(1))
- p1 = (cfgcr1 & DPLL_CFGCR1_QDIV_RATIO_MASK) >>
+ if (pll_state->cfgcr1 & DPLL_CFGCR1_QDIV_MODE(1))
+ p1 = (pll_state->cfgcr1 & DPLL_CFGCR1_QDIV_RATIO_MASK) >>
DPLL_CFGCR1_QDIV_RATIO_SHIFT;
else
p1 = 1;
case DPLL_CFGCR1_KDIV_2:
p2 = 2;
break;
- case DPLL_CFGCR1_KDIV_4:
- p2 = 4;
+ case DPLL_CFGCR1_KDIV_3:
+ p2 = 3;
break;
}
ref_clock = cnl_hdmi_pll_ref_clock(dev_priv);
- dco_freq = (cfgcr0 & DPLL_CFGCR0_DCO_INTEGER_MASK) * ref_clock;
+ dco_freq = (pll_state->cfgcr0 & DPLL_CFGCR0_DCO_INTEGER_MASK)
+ * ref_clock;
- dco_freq += (((cfgcr0 & DPLL_CFGCR0_DCO_FRACTION_MASK) >>
+ dco_freq += (((pll_state->cfgcr0 & DPLL_CFGCR0_DCO_FRACTION_MASK) >>
DPLL_CFGCR0_DCO_FRACTION_SHIFT) * ref_clock) / 0x8000;
if (WARN_ON(p0 == 0 || p1 == 0 || p2 == 0))
}
static int icl_calc_mg_pll_link(struct drm_i915_private *dev_priv,
- enum port port)
+ const struct intel_dpll_hw_state *pll_state)
{
- enum tc_port tc_port = intel_port_to_tc(dev_priv, port);
- u32 mg_pll_div0, mg_clktop_hsclkctl;
- u32 m1, m2_int, m2_frac, div1, div2, refclk;
+ u32 m1, m2_int, m2_frac, div1, div2, ref_clock;
u64 tmp;
- refclk = dev_priv->cdclk.hw.ref;
-
- mg_pll_div0 = I915_READ(MG_PLL_DIV0(tc_port));
- mg_clktop_hsclkctl = I915_READ(MG_CLKTOP2_HSCLKCTL(tc_port));
+ ref_clock = dev_priv->cdclk.hw.ref;
- m1 = I915_READ(MG_PLL_DIV1(tc_port)) & MG_PLL_DIV1_FBPREDIV_MASK;
- m2_int = mg_pll_div0 & MG_PLL_DIV0_FBDIV_INT_MASK;
- m2_frac = (mg_pll_div0 & MG_PLL_DIV0_FRACNEN_H) ?
- (mg_pll_div0 & MG_PLL_DIV0_FBDIV_FRAC_MASK) >>
- MG_PLL_DIV0_FBDIV_FRAC_SHIFT : 0;
+ m1 = pll_state->mg_pll_div1 & MG_PLL_DIV1_FBPREDIV_MASK;
+ m2_int = pll_state->mg_pll_div0 & MG_PLL_DIV0_FBDIV_INT_MASK;
+ m2_frac = (pll_state->mg_pll_div0 & MG_PLL_DIV0_FRACNEN_H) ?
+ (pll_state->mg_pll_div0 & MG_PLL_DIV0_FBDIV_FRAC_MASK) >>
+ MG_PLL_DIV0_FBDIV_FRAC_SHIFT : 0;
- switch (mg_clktop_hsclkctl & MG_CLKTOP2_HSCLKCTL_HSDIV_RATIO_MASK) {
+ switch (pll_state->mg_clktop2_hsclkctl &
+ MG_CLKTOP2_HSCLKCTL_HSDIV_RATIO_MASK) {
case MG_CLKTOP2_HSCLKCTL_HSDIV_RATIO_2:
div1 = 2;
break;
div1 = 7;
break;
default:
- MISSING_CASE(mg_clktop_hsclkctl);
+ MISSING_CASE(pll_state->mg_clktop2_hsclkctl);
return 0;
}
- div2 = (mg_clktop_hsclkctl & MG_CLKTOP2_HSCLKCTL_DSDIV_RATIO_MASK) >>
+ div2 = (pll_state->mg_clktop2_hsclkctl &
+ MG_CLKTOP2_HSCLKCTL_DSDIV_RATIO_MASK) >>
MG_CLKTOP2_HSCLKCTL_DSDIV_RATIO_SHIFT;
+
/* div2 value of 0 is same as 1 means no div */
if (div2 == 0)
div2 = 1;
* Adjust the original formula to delay the division by 2^22 in order to
* minimize possible rounding errors.
*/
- tmp = (u64)m1 * m2_int * refclk +
- (((u64)m1 * m2_frac * refclk) >> 22);
+ tmp = (u64)m1 * m2_int * ref_clock +
+ (((u64)m1 * m2_frac * ref_clock) >> 22);
tmp = div_u64(tmp, 5 * div1 * div2);
return tmp;
struct intel_crtc_state *pipe_config)
{
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct intel_dpll_hw_state *pll_state = &pipe_config->dpll_hw_state;
enum port port = encoder->port;
- int link_clock = 0;
- u32 pll_id;
+ int link_clock;
- pll_id = intel_get_shared_dpll_id(dev_priv, pipe_config->shared_dpll);
if (intel_port_is_combophy(dev_priv, port)) {
- if (intel_crtc_has_type(pipe_config, INTEL_OUTPUT_HDMI))
- link_clock = cnl_calc_wrpll_link(dev_priv, pll_id);
- else
- link_clock = icl_calc_dp_combo_pll_link(dev_priv,
- pll_id);
+ link_clock = cnl_calc_wrpll_link(dev_priv, pll_state);
} else {
+ enum intel_dpll_id pll_id = intel_get_shared_dpll_id(dev_priv,
+ pipe_config->shared_dpll);
+
if (pll_id == DPLL_ID_ICL_TBTPLL)
link_clock = icl_calc_tbt_pll_link(dev_priv, port);
else
- link_clock = icl_calc_mg_pll_link(dev_priv, port);
+ link_clock = icl_calc_mg_pll_link(dev_priv, pll_state);
}
pipe_config->port_clock = link_clock;
+
ddi_dotclock_get(pipe_config);
}
struct intel_crtc_state *pipe_config)
{
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
- int link_clock = 0;
- u32 cfgcr0;
- enum intel_dpll_id pll_id;
-
- pll_id = intel_get_shared_dpll_id(dev_priv, pipe_config->shared_dpll);
+ struct intel_dpll_hw_state *pll_state = &pipe_config->dpll_hw_state;
+ int link_clock;
- cfgcr0 = I915_READ(CNL_DPLL_CFGCR0(pll_id));
-
- if (cfgcr0 & DPLL_CFGCR0_HDMI_MODE) {
- link_clock = cnl_calc_wrpll_link(dev_priv, pll_id);
+ if (pll_state->cfgcr0 & DPLL_CFGCR0_HDMI_MODE) {
+ link_clock = cnl_calc_wrpll_link(dev_priv, pll_state);
} else {
- link_clock = cfgcr0 & DPLL_CFGCR0_LINK_RATE_MASK;
+ link_clock = pll_state->cfgcr0 & DPLL_CFGCR0_LINK_RATE_MASK;
switch (link_clock) {
case DPLL_CFGCR0_LINK_RATE_810:
}
static void skl_ddi_clock_get(struct intel_encoder *encoder,
- struct intel_crtc_state *pipe_config)
+ struct intel_crtc_state *pipe_config)
{
- struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
- int link_clock = 0;
- u32 dpll_ctl1;
- enum intel_dpll_id pll_id;
-
- pll_id = intel_get_shared_dpll_id(dev_priv, pipe_config->shared_dpll);
+ struct intel_dpll_hw_state *pll_state = &pipe_config->dpll_hw_state;
+ int link_clock;
- dpll_ctl1 = I915_READ(DPLL_CTRL1);
-
- if (dpll_ctl1 & DPLL_CTRL1_HDMI_MODE(pll_id)) {
- link_clock = skl_calc_wrpll_link(dev_priv, pll_id);
+ /*
+ * ctrl1 register is already shifted for each pll, just use 0 to get
+ * the internal shift for each field
+ */
+ if (pll_state->ctrl1 & DPLL_CTRL1_HDMI_MODE(0)) {
+ link_clock = skl_calc_wrpll_link(pll_state);
} else {
- link_clock = dpll_ctl1 & DPLL_CTRL1_LINK_RATE_MASK(pll_id);
- link_clock >>= DPLL_CTRL1_LINK_RATE_SHIFT(pll_id);
+ link_clock = pll_state->ctrl1 & DPLL_CTRL1_LINK_RATE_MASK(0);
+ link_clock >>= DPLL_CTRL1_LINK_RATE_SHIFT(0);
switch (link_clock) {
case DPLL_CTRL1_LINK_RATE_810:
ddi_dotclock_get(pipe_config);
}
-static int bxt_calc_pll_link(struct intel_crtc_state *crtc_state)
+static int bxt_calc_pll_link(const struct intel_dpll_hw_state *pll_state)
{
- struct intel_dpll_hw_state *state;
struct dpll clock;
- /* For DDI ports we always use a shared PLL. */
- if (WARN_ON(!crtc_state->shared_dpll))
- return 0;
-
- state = &crtc_state->dpll_hw_state;
-
clock.m1 = 2;
- clock.m2 = (state->pll0 & PORT_PLL_M2_MASK) << 22;
- if (state->pll3 & PORT_PLL_M2_FRAC_ENABLE)
- clock.m2 |= state->pll2 & PORT_PLL_M2_FRAC_MASK;
- clock.n = (state->pll1 & PORT_PLL_N_MASK) >> PORT_PLL_N_SHIFT;
- clock.p1 = (state->ebb0 & PORT_PLL_P1_MASK) >> PORT_PLL_P1_SHIFT;
- clock.p2 = (state->ebb0 & PORT_PLL_P2_MASK) >> PORT_PLL_P2_SHIFT;
+ clock.m2 = (pll_state->pll0 & PORT_PLL_M2_MASK) << 22;
+ if (pll_state->pll3 & PORT_PLL_M2_FRAC_ENABLE)
+ clock.m2 |= pll_state->pll2 & PORT_PLL_M2_FRAC_MASK;
+ clock.n = (pll_state->pll1 & PORT_PLL_N_MASK) >> PORT_PLL_N_SHIFT;
+ clock.p1 = (pll_state->ebb0 & PORT_PLL_P1_MASK) >> PORT_PLL_P1_SHIFT;
+ clock.p2 = (pll_state->ebb0 & PORT_PLL_P2_MASK) >> PORT_PLL_P2_SHIFT;
return chv_calc_dpll_params(100000, &clock);
}
static void bxt_ddi_clock_get(struct intel_encoder *encoder,
struct intel_crtc_state *pipe_config)
{
- pipe_config->port_clock = bxt_calc_pll_link(pipe_config);
+ pipe_config->port_clock =
+ bxt_calc_pll_link(&pipe_config->dpll_hw_state);
ddi_dotclock_get(pipe_config);
}
{
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
- if (IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 11)
icl_ddi_clock_get(encoder, pipe_config);
else if (IS_CANNONLAKE(dev_priv))
cnl_ddi_clock_get(encoder, pipe_config);
goto out;
}
- if (port == PORT_A)
+ if (HAS_TRANSCODER_EDP(dev_priv) && port == PORT_A)
cpu_transcoder = TRANSCODER_EDP;
else
cpu_transcoder = (enum transcoder) pipe;
if (!(tmp & DDI_BUF_CTL_ENABLE))
goto out;
- if (port == PORT_A) {
+ if (HAS_TRANSCODER_EDP(dev_priv) && port == PORT_A) {
tmp = I915_READ(TRANS_DDI_FUNC_CTL(TRANSCODER_EDP));
switch (tmp & TRANS_DDI_EDP_INPUT_MASK) {
enum port port = encoder->port;
int n_entries;
- if (IS_ICELAKE(dev_priv)) {
+ if (INTEL_GEN(dev_priv) >= 11) {
if (intel_port_is_combophy(dev_priv, port))
icl_get_combo_buf_trans(dev_priv, port, encoder->type,
intel_dp->link_rate, &n_entries);
/* Program PORT_TX_DW4 */
/* We cannot write to GRP. It would overrite individual loadgen */
for (ln = 0; ln < 4; ln++) {
- val = I915_READ(CNL_PORT_TX_DW4_LN(port, ln));
+ val = I915_READ(CNL_PORT_TX_DW4_LN(ln, port));
val &= ~(POST_CURSOR_1_MASK | POST_CURSOR_2_MASK |
CURSOR_COEFF_MASK);
val |= POST_CURSOR_1(ddi_translations[level].dw4_post_cursor_1);
val |= POST_CURSOR_2(ddi_translations[level].dw4_post_cursor_2);
val |= CURSOR_COEFF(ddi_translations[level].dw4_cursor_coeff);
- I915_WRITE(CNL_PORT_TX_DW4_LN(port, ln), val);
+ I915_WRITE(CNL_PORT_TX_DW4_LN(ln, port), val);
}
/* Program PORT_TX_DW5 */
* > 6 GHz (LN0=0, LN1=0, LN2=0, LN3=0)
*/
for (ln = 0; ln <= 3; ln++) {
- val = I915_READ(CNL_PORT_TX_DW4_LN(port, ln));
+ val = I915_READ(CNL_PORT_TX_DW4_LN(ln, port));
val &= ~LOADGEN_SELECT;
if ((rate <= 600000 && width == 4 && ln >= 1) ||
(rate <= 600000 && width < 4 && (ln == 1 || ln == 2))) {
val |= LOADGEN_SELECT;
}
- I915_WRITE(CNL_PORT_TX_DW4_LN(port, ln), val);
+ I915_WRITE(CNL_PORT_TX_DW4_LN(ln, port), val);
}
/* 3. Set PORT_CL_DW5 SUS Clock Config to 11b */
/* Program PORT_TX_DW4 */
/* We cannot write to GRP. It would overwrite individual loadgen. */
for (ln = 0; ln <= 3; ln++) {
- val = I915_READ(ICL_PORT_TX_DW4_LN(port, ln));
+ val = I915_READ(ICL_PORT_TX_DW4_LN(ln, port));
val &= ~(POST_CURSOR_1_MASK | POST_CURSOR_2_MASK |
CURSOR_COEFF_MASK);
val |= POST_CURSOR_1(ddi_translations[level].dw4_post_cursor_1);
val |= POST_CURSOR_2(ddi_translations[level].dw4_post_cursor_2);
val |= CURSOR_COEFF(ddi_translations[level].dw4_cursor_coeff);
- I915_WRITE(ICL_PORT_TX_DW4_LN(port, ln), val);
+ I915_WRITE(ICL_PORT_TX_DW4_LN(ln, port), val);
}
/* Program PORT_TX_DW7 */
* > 6 GHz (LN0=0, LN1=0, LN2=0, LN3=0)
*/
for (ln = 0; ln <= 3; ln++) {
- val = I915_READ(ICL_PORT_TX_DW4_LN(port, ln));
+ val = I915_READ(ICL_PORT_TX_DW4_LN(ln, port));
val &= ~LOADGEN_SELECT;
if ((rate <= 600000 && width == 4 && ln >= 1) ||
(rate <= 600000 && width < 4 && (ln == 1 || ln == 2))) {
val |= LOADGEN_SELECT;
}
- I915_WRITE(ICL_PORT_TX_DW4_LN(port, ln), val);
+ I915_WRITE(ICL_PORT_TX_DW4_LN(ln, port), val);
}
/* 3. Set PORT_CL_DW5 SUS Clock Config to 11b */
/* Set MG_TX_LINK_PARAMS cri_use_fs32 to 0. */
for (ln = 0; ln < 2; ln++) {
- val = I915_READ(MG_TX1_LINK_PARAMS(port, ln));
+ val = I915_READ(MG_TX1_LINK_PARAMS(ln, port));
val &= ~CRI_USE_FS32;
- I915_WRITE(MG_TX1_LINK_PARAMS(port, ln), val);
+ I915_WRITE(MG_TX1_LINK_PARAMS(ln, port), val);
- val = I915_READ(MG_TX2_LINK_PARAMS(port, ln));
+ val = I915_READ(MG_TX2_LINK_PARAMS(ln, port));
val &= ~CRI_USE_FS32;
- I915_WRITE(MG_TX2_LINK_PARAMS(port, ln), val);
+ I915_WRITE(MG_TX2_LINK_PARAMS(ln, port), val);
}
/* Program MG_TX_SWINGCTRL with values from vswing table */
for (ln = 0; ln < 2; ln++) {
- val = I915_READ(MG_TX1_SWINGCTRL(port, ln));
+ val = I915_READ(MG_TX1_SWINGCTRL(ln, port));
val &= ~CRI_TXDEEMPH_OVERRIDE_17_12_MASK;
val |= CRI_TXDEEMPH_OVERRIDE_17_12(
ddi_translations[level].cri_txdeemph_override_17_12);
- I915_WRITE(MG_TX1_SWINGCTRL(port, ln), val);
+ I915_WRITE(MG_TX1_SWINGCTRL(ln, port), val);
- val = I915_READ(MG_TX2_SWINGCTRL(port, ln));
+ val = I915_READ(MG_TX2_SWINGCTRL(ln, port));
val &= ~CRI_TXDEEMPH_OVERRIDE_17_12_MASK;
val |= CRI_TXDEEMPH_OVERRIDE_17_12(
ddi_translations[level].cri_txdeemph_override_17_12);
- I915_WRITE(MG_TX2_SWINGCTRL(port, ln), val);
+ I915_WRITE(MG_TX2_SWINGCTRL(ln, port), val);
}
/* Program MG_TX_DRVCTRL with values from vswing table */
for (ln = 0; ln < 2; ln++) {
- val = I915_READ(MG_TX1_DRVCTRL(port, ln));
+ val = I915_READ(MG_TX1_DRVCTRL(ln, port));
val &= ~(CRI_TXDEEMPH_OVERRIDE_11_6_MASK |
CRI_TXDEEMPH_OVERRIDE_5_0_MASK);
val |= CRI_TXDEEMPH_OVERRIDE_5_0(
CRI_TXDEEMPH_OVERRIDE_11_6(
ddi_translations[level].cri_txdeemph_override_11_6) |
CRI_TXDEEMPH_OVERRIDE_EN;
- I915_WRITE(MG_TX1_DRVCTRL(port, ln), val);
+ I915_WRITE(MG_TX1_DRVCTRL(ln, port), val);
- val = I915_READ(MG_TX2_DRVCTRL(port, ln));
+ val = I915_READ(MG_TX2_DRVCTRL(ln, port));
val &= ~(CRI_TXDEEMPH_OVERRIDE_11_6_MASK |
CRI_TXDEEMPH_OVERRIDE_5_0_MASK);
val |= CRI_TXDEEMPH_OVERRIDE_5_0(
CRI_TXDEEMPH_OVERRIDE_11_6(
ddi_translations[level].cri_txdeemph_override_11_6) |
CRI_TXDEEMPH_OVERRIDE_EN;
- I915_WRITE(MG_TX2_DRVCTRL(port, ln), val);
+ I915_WRITE(MG_TX2_DRVCTRL(ln, port), val);
/* FIXME: Program CRI_LOADGEN_SEL after the spec is updated */
}
* values from table for which TX1 and TX2 enabled.
*/
for (ln = 0; ln < 2; ln++) {
- val = I915_READ(MG_CLKHUB(port, ln));
+ val = I915_READ(MG_CLKHUB(ln, port));
if (link_clock < 300000)
val |= CFG_LOW_RATE_LKREN_EN;
else
val &= ~CFG_LOW_RATE_LKREN_EN;
- I915_WRITE(MG_CLKHUB(port, ln), val);
+ I915_WRITE(MG_CLKHUB(ln, port), val);
}
/* Program the MG_TX_DCC<LN, port being used> based on the link frequency */
for (ln = 0; ln < 2; ln++) {
- val = I915_READ(MG_TX1_DCC(port, ln));
+ val = I915_READ(MG_TX1_DCC(ln, port));
val &= ~CFG_AMI_CK_DIV_OVERRIDE_VAL_MASK;
if (link_clock <= 500000) {
val &= ~CFG_AMI_CK_DIV_OVERRIDE_EN;
val |= CFG_AMI_CK_DIV_OVERRIDE_EN |
CFG_AMI_CK_DIV_OVERRIDE_VAL(1);
}
- I915_WRITE(MG_TX1_DCC(port, ln), val);
+ I915_WRITE(MG_TX1_DCC(ln, port), val);
- val = I915_READ(MG_TX2_DCC(port, ln));
+ val = I915_READ(MG_TX2_DCC(ln, port));
val &= ~CFG_AMI_CK_DIV_OVERRIDE_VAL_MASK;
if (link_clock <= 500000) {
val &= ~CFG_AMI_CK_DIV_OVERRIDE_EN;
val |= CFG_AMI_CK_DIV_OVERRIDE_EN |
CFG_AMI_CK_DIV_OVERRIDE_VAL(1);
}
- I915_WRITE(MG_TX2_DCC(port, ln), val);
+ I915_WRITE(MG_TX2_DCC(ln, port), val);
}
/* Program MG_TX_PISO_READLOAD with values from vswing table */
for (ln = 0; ln < 2; ln++) {
- val = I915_READ(MG_TX1_PISO_READLOAD(port, ln));
+ val = I915_READ(MG_TX1_PISO_READLOAD(ln, port));
val |= CRI_CALCINIT;
- I915_WRITE(MG_TX1_PISO_READLOAD(port, ln), val);
+ I915_WRITE(MG_TX1_PISO_READLOAD(ln, port), val);
- val = I915_READ(MG_TX2_PISO_READLOAD(port, ln));
+ val = I915_READ(MG_TX2_PISO_READLOAD(ln, port));
val |= CRI_CALCINIT;
- I915_WRITE(MG_TX2_PISO_READLOAD(port, ln), val);
+ I915_WRITE(MG_TX2_PISO_READLOAD(ln, port), val);
}
}
struct intel_encoder *encoder = &dport->base;
int level = intel_ddi_dp_level(intel_dp);
- if (IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 11)
icl_ddi_vswing_sequence(encoder, intel_dp->link_rate,
level, encoder->type);
else if (IS_CANNONLAKE(dev_priv))
mutex_lock(&dev_priv->dpll_lock);
- if (IS_ICELAKE(dev_priv)) {
+ if (INTEL_GEN(dev_priv) >= 11) {
if (!intel_port_is_combophy(dev_priv, port))
I915_WRITE(DDI_CLK_SEL(port),
icl_pll_to_ddi_clk_sel(encoder, crtc_state));
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
enum port port = encoder->port;
- if (IS_ICELAKE(dev_priv)) {
+ if (INTEL_GEN(dev_priv) >= 11) {
if (!intel_port_is_combophy(dev_priv, port))
I915_WRITE(DDI_CLK_SEL(port), DDI_CLK_SEL_NONE);
} else if (IS_CANNONLAKE(dev_priv)) {
struct drm_i915_private *dev_priv = to_i915(dig_port->base.base.dev);
enum port port = dig_port->base.port;
enum tc_port tc_port = intel_port_to_tc(dev_priv, port);
- i915_reg_t mg_regs[2] = { MG_DP_MODE(port, 0), MG_DP_MODE(port, 1) };
u32 val;
- int i;
+ int ln;
if (tc_port == PORT_TC_NONE)
return;
- for (i = 0; i < ARRAY_SIZE(mg_regs); i++) {
- val = I915_READ(mg_regs[i]);
+ for (ln = 0; ln < 2; ln++) {
+ val = I915_READ(MG_DP_MODE(ln, port));
val |= MG_DP_MODE_CFG_TR2PWR_GATING |
MG_DP_MODE_CFG_TRPWR_GATING |
MG_DP_MODE_CFG_CLNPWR_GATING |
MG_DP_MODE_CFG_DIGPWR_GATING |
MG_DP_MODE_CFG_GAONPWR_GATING;
- I915_WRITE(mg_regs[i], val);
+ I915_WRITE(MG_DP_MODE(ln, port), val);
}
val = I915_READ(MG_MISC_SUS0(tc_port));
struct drm_i915_private *dev_priv = to_i915(dig_port->base.base.dev);
enum port port = dig_port->base.port;
enum tc_port tc_port = intel_port_to_tc(dev_priv, port);
- i915_reg_t mg_regs[2] = { MG_DP_MODE(port, 0), MG_DP_MODE(port, 1) };
u32 val;
- int i;
+ int ln;
if (tc_port == PORT_TC_NONE)
return;
- for (i = 0; i < ARRAY_SIZE(mg_regs); i++) {
- val = I915_READ(mg_regs[i]);
+ for (ln = 0; ln < 2; ln++) {
+ val = I915_READ(MG_DP_MODE(ln, port));
val &= ~(MG_DP_MODE_CFG_TR2PWR_GATING |
MG_DP_MODE_CFG_TRPWR_GATING |
MG_DP_MODE_CFG_CLNPWR_GATING |
MG_DP_MODE_CFG_DIGPWR_GATING |
MG_DP_MODE_CFG_GAONPWR_GATING);
- I915_WRITE(mg_regs[i], val);
+ I915_WRITE(MG_DP_MODE(ln, port), val);
}
val = I915_READ(MG_MISC_SUS0(tc_port));
if (tc_port == PORT_TC_NONE || intel_dig_port->tc_type == TC_PORT_TBT)
return;
- ln0 = I915_READ(MG_DP_MODE(port, 0));
- ln1 = I915_READ(MG_DP_MODE(port, 1));
+ ln0 = I915_READ(MG_DP_MODE(0, port));
+ ln1 = I915_READ(MG_DP_MODE(1, port));
switch (intel_dig_port->tc_type) {
case TC_PORT_TYPEC:
return;
}
- I915_WRITE(MG_DP_MODE(port, 0), ln0);
- I915_WRITE(MG_DP_MODE(port, 1), ln1);
+ I915_WRITE(MG_DP_MODE(0, port), ln0);
+ I915_WRITE(MG_DP_MODE(1, port), ln1);
}
static void intel_dp_sink_set_fec_ready(struct intel_dp *intel_dp,
val |= DP_TP_CTL_FEC_ENABLE;
I915_WRITE(DP_TP_CTL(port), val);
- if (intel_wait_for_register(dev_priv, DP_TP_STATUS(port),
+ if (intel_wait_for_register(&dev_priv->uncore, DP_TP_STATUS(port),
DP_TP_STATUS_FEC_ENABLE_LIVE,
DP_TP_STATUS_FEC_ENABLE_LIVE,
1))
icl_program_mg_dp_mode(dig_port);
icl_disable_phy_clock_gating(dig_port);
- if (IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 11)
icl_ddi_vswing_sequence(encoder, crtc_state->port_clock,
level, encoder->type);
else if (IS_CANNONLAKE(dev_priv))
icl_program_mg_dp_mode(dig_port);
icl_disable_phy_clock_gating(dig_port);
- if (IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 11)
icl_ddi_vswing_sequence(encoder, crtc_state->port_clock,
level, INTEL_OUTPUT_HDMI);
else if (IS_CANNONLAKE(dev_priv))
{
struct intel_dp *intel_dp = enc_to_intel_dp(&encoder->base);
- intel_psr_enable(intel_dp, crtc_state);
+ intel_ddi_set_pipe_settings(crtc_state);
+
+ intel_psr_update(intel_dp, crtc_state);
intel_edp_drrs_enable(intel_dp, crtc_state);
intel_panel_update_backlight(encoder, crtc_state, conn_state);
void intel_ddi_compute_min_voltage_level(struct drm_i915_private *dev_priv,
struct intel_crtc_state *crtc_state)
{
- if (IS_ICELAKE(dev_priv) && crtc_state->port_clock > 594000)
+ if (INTEL_GEN(dev_priv) >= 11 && crtc_state->port_clock > 594000)
crtc_state->min_voltage_level = 1;
else if (IS_CANNONLAKE(dev_priv) && crtc_state->port_clock > 594000)
crtc_state->min_voltage_level = 2;
pipe_config->has_hdmi_sink = true;
intel_dig_port = enc_to_dig_port(&encoder->base);
- if (intel_dig_port->infoframe_enabled(encoder, pipe_config))
+ pipe_config->infoframes.enable |=
+ intel_hdmi_infoframes_enabled(encoder, pipe_config);
+
+ if (pipe_config->infoframes.enable)
pipe_config->has_infoframe = true;
if (temp & TRANS_DDI_HDMI_SCRAMBLING)
bxt_ddi_phy_get_lane_lat_optim_mask(encoder);
intel_ddi_compute_min_voltage_level(dev_priv, pipe_config);
+
+ intel_hdmi_read_gcp_infoframe(encoder, pipe_config);
+
+ intel_read_infoframe(encoder, pipe_config,
+ HDMI_INFOFRAME_TYPE_AVI,
+ &pipe_config->infoframes.avi);
+ intel_read_infoframe(encoder, pipe_config,
+ HDMI_INFOFRAME_TYPE_SPD,
+ &pipe_config->infoframes.spd);
+ intel_read_infoframe(encoder, pipe_config,
+ HDMI_INFOFRAME_TYPE_VENDOR,
+ &pipe_config->infoframes.hdmi);
}
static enum intel_output_type
enum port port = encoder->port;
int ret;
- if (port == PORT_A)
+ if (HAS_TRANSCODER_EDP(dev_priv) && port == PORT_A)
pipe_config->cpu_transcoder = TRANSCODER_EDP;
if (intel_crtc_has_type(pipe_config, INTEL_OUTPUT_HDMI))
goto out;
}
- crtc_state->mode_changed = true;
-
- ret = drm_atomic_add_affected_connectors(state, crtc);
- if (ret)
- goto out;
-
- ret = drm_atomic_add_affected_planes(state, crtc);
- if (ret)
- goto out;
+ crtc_state->connectors_changed = true;
ret = drm_atomic_commit(state);
out:
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_DDI_H__
+#define __INTEL_DDI_H__
+
+#include <drm/i915_drm.h>
+
+#include "intel_display.h"
+
+struct drm_connector_state;
+struct drm_i915_private;
+struct intel_connector;
+struct intel_crtc;
+struct intel_crtc_state;
+struct intel_dp;
+struct intel_dpll_hw_state;
+struct intel_encoder;
+
+void intel_ddi_fdi_post_disable(struct intel_encoder *intel_encoder,
+ const struct intel_crtc_state *old_crtc_state,
+ const struct drm_connector_state *old_conn_state);
+void hsw_fdi_link_train(struct intel_crtc *crtc,
+ const struct intel_crtc_state *crtc_state);
+void intel_ddi_init(struct drm_i915_private *dev_priv, enum port port);
+bool intel_ddi_get_hw_state(struct intel_encoder *encoder, enum pipe *pipe);
+void intel_ddi_enable_transcoder_func(const struct intel_crtc_state *crtc_state);
+void intel_ddi_disable_transcoder_func(const struct intel_crtc_state *crtc_state);
+void intel_ddi_enable_pipe_clock(const struct intel_crtc_state *crtc_state);
+void intel_ddi_disable_pipe_clock(const struct intel_crtc_state *crtc_state);
+void intel_ddi_set_pipe_settings(const struct intel_crtc_state *crtc_state);
+void intel_ddi_prepare_link_retrain(struct intel_dp *intel_dp);
+bool intel_ddi_connector_get_hw_state(struct intel_connector *intel_connector);
+void intel_ddi_get_config(struct intel_encoder *encoder,
+ struct intel_crtc_state *pipe_config);
+void intel_ddi_set_vc_payload_alloc(const struct intel_crtc_state *crtc_state,
+ bool state);
+void intel_ddi_compute_min_voltage_level(struct drm_i915_private *dev_priv,
+ struct intel_crtc_state *crtc_state);
+u32 bxt_signal_levels(struct intel_dp *intel_dp);
+u32 ddi_signal_levels(struct intel_dp *intel_dp);
+u8 intel_ddi_dp_voltage_max(struct intel_encoder *encoder);
+u8 intel_ddi_dp_pre_emphasis_max(struct intel_encoder *encoder,
+ u8 voltage_swing);
+int intel_ddi_toggle_hdcp_signalling(struct intel_encoder *intel_encoder,
+ bool enable);
+void icl_sanitize_encoder_pll_mapping(struct intel_encoder *encoder);
+int cnl_calc_wrpll_link(struct drm_i915_private *dev_priv,
+ struct intel_dpll_hw_state *state);
+
+#endif /* __INTEL_DDI_H__ */
PLATFORM_NAME(COFFEELAKE),
PLATFORM_NAME(CANNONLAKE),
PLATFORM_NAME(ICELAKE),
+ PLATFORM_NAME(ELKHARTLAKE),
};
#undef PLATFORM_NAME
u8 eu_en;
int s;
- sseu->max_slices = 1;
- sseu->max_subslices = 8;
- sseu->max_eus_per_subslice = 8;
+ if (IS_ELKHARTLAKE(dev_priv)) {
+ sseu->max_slices = 1;
+ sseu->max_subslices = 4;
+ sseu->max_eus_per_subslice = 8;
+ } else {
+ sseu->max_slices = 1;
+ sseu->max_subslices = 8;
+ sseu->max_eus_per_subslice = 8;
+ }
s_en = I915_READ(GEN11_GT_SLICE_ENABLE) & GEN11_GT_S_ENA_MASK;
ss_en = ~I915_READ(GEN11_GT_SUBSLICE_DISABLE);
return 0;
}
+#undef INTEL_VGA_DEVICE
+#define INTEL_VGA_DEVICE(id, info) (id)
+
+static const u16 subplatform_ult_ids[] = {
+ INTEL_HSW_ULT_GT1_IDS(0),
+ INTEL_HSW_ULT_GT2_IDS(0),
+ INTEL_HSW_ULT_GT3_IDS(0),
+ INTEL_BDW_ULT_GT1_IDS(0),
+ INTEL_BDW_ULT_GT2_IDS(0),
+ INTEL_BDW_ULT_GT3_IDS(0),
+ INTEL_BDW_ULT_RSVD_IDS(0),
+ INTEL_SKL_ULT_GT1_IDS(0),
+ INTEL_SKL_ULT_GT2_IDS(0),
+ INTEL_SKL_ULT_GT3_IDS(0),
+ INTEL_KBL_ULT_GT1_IDS(0),
+ INTEL_KBL_ULT_GT2_IDS(0),
+ INTEL_KBL_ULT_GT3_IDS(0),
+ INTEL_CFL_U_GT2_IDS(0),
+ INTEL_CFL_U_GT3_IDS(0),
+ INTEL_WHL_U_GT1_IDS(0),
+ INTEL_WHL_U_GT2_IDS(0),
+ INTEL_WHL_U_GT3_IDS(0)
+};
+
+static const u16 subplatform_ulx_ids[] = {
+ INTEL_HSW_ULX_GT1_IDS(0),
+ INTEL_HSW_ULX_GT2_IDS(0),
+ INTEL_BDW_ULX_GT1_IDS(0),
+ INTEL_BDW_ULX_GT2_IDS(0),
+ INTEL_BDW_ULX_GT3_IDS(0),
+ INTEL_BDW_ULX_RSVD_IDS(0),
+ INTEL_SKL_ULX_GT1_IDS(0),
+ INTEL_SKL_ULX_GT2_IDS(0),
+ INTEL_KBL_ULX_GT1_IDS(0),
+ INTEL_KBL_ULX_GT2_IDS(0)
+};
+
+static const u16 subplatform_aml_ids[] = {
+ INTEL_AML_KBL_GT2_IDS(0),
+ INTEL_AML_CFL_GT2_IDS(0)
+};
+
+static const u16 subplatform_portf_ids[] = {
+ INTEL_CNL_PORT_F_IDS(0),
+ INTEL_ICL_PORT_F_IDS(0)
+};
+
+static bool find_devid(u16 id, const u16 *p, unsigned int num)
+{
+ for (; num; num--, p++) {
+ if (*p == id)
+ return true;
+ }
+
+ return false;
+}
+
+void intel_device_info_subplatform_init(struct drm_i915_private *i915)
+{
+ const struct intel_device_info *info = INTEL_INFO(i915);
+ const struct intel_runtime_info *rinfo = RUNTIME_INFO(i915);
+ const unsigned int pi = __platform_mask_index(rinfo, info->platform);
+ const unsigned int pb = __platform_mask_bit(rinfo, info->platform);
+ u16 devid = INTEL_DEVID(i915);
+ u32 mask = 0;
+
+ /* Make sure IS_<platform> checks are working. */
+ RUNTIME_INFO(i915)->platform_mask[pi] = BIT(pb);
+
+ /* Find and mark subplatform bits based on the PCI device id. */
+ if (find_devid(devid, subplatform_ult_ids,
+ ARRAY_SIZE(subplatform_ult_ids))) {
+ mask = BIT(INTEL_SUBPLATFORM_ULT);
+ } else if (find_devid(devid, subplatform_ulx_ids,
+ ARRAY_SIZE(subplatform_ulx_ids))) {
+ mask = BIT(INTEL_SUBPLATFORM_ULX);
+ if (IS_HASWELL(i915) || IS_BROADWELL(i915)) {
+ /* ULX machines are also considered ULT. */
+ mask |= BIT(INTEL_SUBPLATFORM_ULT);
+ }
+ } else if (find_devid(devid, subplatform_aml_ids,
+ ARRAY_SIZE(subplatform_aml_ids))) {
+ mask = BIT(INTEL_SUBPLATFORM_AML);
+ } else if (find_devid(devid, subplatform_portf_ids,
+ ARRAY_SIZE(subplatform_portf_ids))) {
+ mask = BIT(INTEL_SUBPLATFORM_PORTF);
+ }
+
+ GEM_BUG_ON(mask & ~INTEL_SUBPLATFORM_BITS);
+
+ RUNTIME_INFO(i915)->platform_mask[pi] |= mask;
+}
+
/**
* intel_device_info_runtime_init - initialize runtime info
* @dev_priv: the i915 device
runtime->num_scalers[PIPE_C] = 1;
}
- BUILD_BUG_ON(I915_NUM_ENGINES > BITS_PER_TYPE(intel_ring_mask_t));
+ BUILD_BUG_ON(BITS_PER_TYPE(intel_engine_mask_t) < I915_NUM_ENGINES);
- if (IS_GEN(dev_priv, 11))
+ if (INTEL_GEN(dev_priv) >= 11)
for_each_pipe(dev_priv, pipe)
runtime->num_sprites[pipe] = 6;
else if (IS_GEN(dev_priv, 10) || IS_GEMINILAKE(dev_priv))
if (IS_GEN(dev_priv, 6) && intel_vtd_active()) {
DRM_INFO("Disabling ppGTT for VT-d support\n");
- info->ppgtt = INTEL_PPGTT_NONE;
+ info->ppgtt_type = INTEL_PPGTT_NONE;
}
/* Initialize command stream timestamp frequency */
unsigned int logical_vdbox = 0;
unsigned int i;
u32 media_fuse;
+ u16 vdbox_mask;
+ u16 vebox_mask;
if (INTEL_GEN(dev_priv) < 11)
return;
media_fuse = ~I915_READ(GEN11_GT_VEBOX_VDBOX_DISABLE);
- RUNTIME_INFO(dev_priv)->vdbox_enable = media_fuse & GEN11_GT_VDBOX_DISABLE_MASK;
- RUNTIME_INFO(dev_priv)->vebox_enable = (media_fuse & GEN11_GT_VEBOX_DISABLE_MASK) >>
- GEN11_GT_VEBOX_DISABLE_SHIFT;
+ vdbox_mask = media_fuse & GEN11_GT_VDBOX_DISABLE_MASK;
+ vebox_mask = (media_fuse & GEN11_GT_VEBOX_DISABLE_MASK) >>
+ GEN11_GT_VEBOX_DISABLE_SHIFT;
- DRM_DEBUG_DRIVER("vdbox enable: %04x\n", RUNTIME_INFO(dev_priv)->vdbox_enable);
for (i = 0; i < I915_MAX_VCS; i++) {
if (!HAS_ENGINE(dev_priv, _VCS(i)))
continue;
- if (!(BIT(i) & RUNTIME_INFO(dev_priv)->vdbox_enable)) {
- info->ring_mask &= ~ENGINE_MASK(_VCS(i));
+ if (!(BIT(i) & vdbox_mask)) {
+ info->engine_mask &= ~BIT(_VCS(i));
DRM_DEBUG_DRIVER("vcs%u fused off\n", i);
continue;
}
if (logical_vdbox++ % 2 == 0)
RUNTIME_INFO(dev_priv)->vdbox_sfc_access |= BIT(i);
}
+ DRM_DEBUG_DRIVER("vdbox enable: %04x, instances: %04lx\n",
+ vdbox_mask, VDBOX_MASK(dev_priv));
+ GEM_BUG_ON(vdbox_mask != VDBOX_MASK(dev_priv));
- DRM_DEBUG_DRIVER("vebox enable: %04x\n", RUNTIME_INFO(dev_priv)->vebox_enable);
for (i = 0; i < I915_MAX_VECS; i++) {
if (!HAS_ENGINE(dev_priv, _VECS(i)))
continue;
- if (!(BIT(i) & RUNTIME_INFO(dev_priv)->vebox_enable)) {
- info->ring_mask &= ~ENGINE_MASK(_VECS(i));
+ if (!(BIT(i) & vebox_mask)) {
+ info->engine_mask &= ~BIT(_VECS(i));
DRM_DEBUG_DRIVER("vecs%u fused off\n", i);
}
}
+ DRM_DEBUG_DRIVER("vebox enable: %04x, instances: %04lx\n",
+ vebox_mask, VEBOX_MASK(dev_priv));
+ GEM_BUG_ON(vebox_mask != VEBOX_MASK(dev_priv));
}
#include <uapi/drm/i915_drm.h>
+#include "intel_engine_types.h"
#include "intel_display.h"
struct drm_printer;
INTEL_CANNONLAKE,
/* gen11 */
INTEL_ICELAKE,
+ INTEL_ELKHARTLAKE,
INTEL_MAX_PLATFORMS
};
-enum intel_ppgtt {
+/*
+ * Subplatform bits share the same namespace per parent platform. In other words
+ * it is fine for the same bit to be used on multiple parent platforms.
+ */
+
+#define INTEL_SUBPLATFORM_BITS (3)
+
+/* HSW/BDW/SKL/KBL/CFL */
+#define INTEL_SUBPLATFORM_ULT (0)
+#define INTEL_SUBPLATFORM_ULX (1)
+#define INTEL_SUBPLATFORM_AML (2)
+
+/* CNL/ICL */
+#define INTEL_SUBPLATFORM_PORTF (0)
+
+enum intel_ppgtt_type {
INTEL_PPGTT_NONE = I915_GEM_PPGTT_NONE,
INTEL_PPGTT_ALIASING = I915_GEM_PPGTT_ALIASING,
INTEL_PPGTT_FULL = I915_GEM_PPGTT_FULL,
- INTEL_PPGTT_FULL_4LVL,
};
#define DEV_INFO_FOR_EACH_FLAG(func) \
u8 eu_mask[GEN_MAX_SLICES * GEN_MAX_SUBSLICES];
};
-typedef u8 intel_ring_mask_t;
-
struct intel_device_info {
u16 gen_mask;
u8 gen;
u8 gt; /* GT number, 0 if undefined */
- intel_ring_mask_t ring_mask; /* Rings supported by the HW */
+ intel_engine_mask_t engine_mask; /* Engines supported by the HW */
enum intel_platform platform;
- u32 platform_mask;
- enum intel_ppgtt ppgtt;
+ enum intel_ppgtt_type ppgtt_type;
+ unsigned int ppgtt_size; /* log2, e.g. 31/32/48 bits */
+
unsigned int page_sizes; /* page sizes supported by the HW */
u32 display_mmio_offset;
};
struct intel_runtime_info {
+ /*
+ * Platform mask is used for optimizing or-ed IS_PLATFORM calls into
+ * into single runtime conditionals, and also to provide groundwork
+ * for future per platform, or per SKU build optimizations.
+ *
+ * Array can be extended when necessary if the corresponding
+ * BUILD_BUG_ON is hit.
+ */
+ u32 platform_mask[2];
+
u16 device_id;
u8 num_sprites[I915_MAX_PIPES];
u8 num_scalers[I915_MAX_PIPES];
- u8 num_rings;
+ u8 num_engines;
/* Slice/subslice/EU info */
struct sseu_dev_info sseu;
u32 cs_timestamp_frequency_khz;
- /* Enabled (not fused off) media engine bitmasks. */
- u8 vdbox_enable;
- u8 vebox_enable;
-
/* Media engine access to SFC per instance */
u8 vdbox_sfc_access;
};
const char *intel_platform_name(enum intel_platform platform);
+void intel_device_info_subplatform_init(struct drm_i915_private *dev_priv);
void intel_device_info_runtime_init(struct drm_i915_private *dev_priv);
void intel_device_info_dump_flags(const struct intel_device_info *info,
struct drm_printer *p);
#include "i915_drv.h"
#include "i915_gem_clflush.h"
+#include "i915_reset.h"
#include "i915_trace.h"
+#include "intel_atomic_plane.h"
+#include "intel_color.h"
+#include "intel_cdclk.h"
+#include "intel_crt.h"
+#include "intel_ddi.h"
+#include "intel_dp.h"
#include "intel_drv.h"
#include "intel_dsi.h"
+#include "intel_dvo.h"
+#include "intel_fbc.h"
+#include "intel_fbdev.h"
#include "intel_frontbuffer.h"
-
-#include "intel_drv.h"
-#include "intel_dsi.h"
-#include "intel_frontbuffer.h"
-
-#include "i915_drv.h"
-#include "i915_gem_clflush.h"
-#include "i915_reset.h"
-#include "i915_trace.h"
+#include "intel_hdcp.h"
+#include "intel_hdmi.h"
+#include "intel_lvds.h"
+#include "intel_pipe_crc.h"
+#include "intel_pm.h"
+#include "intel_psr.h"
+#include "intel_sdvo.h"
+#include "intel_sprite.h"
+#include "intel_tv.h"
/* Primary plane formats for gen <= 3 */
static const u32 i8xx_primary_formats[] = {
const struct intel_crtc_state *pipe_config);
static void chv_prepare_pll(struct intel_crtc *crtc,
const struct intel_crtc_state *pipe_config);
-static void intel_begin_crtc_commit(struct drm_crtc *, struct drm_crtc_state *);
-static void intel_finish_crtc_commit(struct drm_crtc *, struct drm_crtc_state *);
+static void intel_begin_crtc_commit(struct intel_atomic_state *, struct intel_crtc *);
+static void intel_finish_crtc_commit(struct intel_atomic_state *, struct intel_crtc *);
static void intel_crtc_init_scalers(struct intel_crtc *crtc,
struct intel_crtc_state *crtc_state);
static void skylake_pfit_enable(const struct intel_crtc_state *crtc_state);
};
static void
-skl_wa_clkgate(struct drm_i915_private *dev_priv, int pipe, bool enable)
+skl_wa_827(struct drm_i915_private *dev_priv, int pipe, bool enable)
{
if (enable)
I915_WRITE(CLKGATE_DIS_PSL(pipe),
+ I915_READ(CLKGATE_DIS_PSL(pipe)) |
DUPS1_GATING_DIS | DUPS2_GATING_DIS);
else
I915_WRITE(CLKGATE_DIS_PSL(pipe),
const struct intel_crtc_state *crtc_state,
int target)
{
- struct drm_device *dev = crtc_state->base.crtc->dev;
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_LVDS)) {
/*
* We haven't figured out how to reliably set up different
* single/dual channel state, if we even can.
*/
- if (intel_is_dual_link_lvds(dev))
+ if (intel_is_dual_link_lvds(dev_priv))
return limit->p2.p2_fast;
else
return limit->p2.p2_slow;
return found;
}
-bool bxt_find_best_dpll(struct intel_crtc_state *crtc_state, int target_clock,
+bool bxt_find_best_dpll(struct intel_crtc_state *crtc_state,
struct dpll *best_clock)
{
int refclk = 100000;
const struct intel_limit *limit = &intel_limits_bxt;
return chv_find_best_dpll(limit, crtc_state,
- target_clock, refclk, NULL, best_clock);
+ crtc_state->port_clock, refclk,
+ NULL, best_clock);
}
bool intel_crtc_active(struct intel_crtc *crtc)
i915_reg_t reg = PIPECONF(cpu_transcoder);
/* Wait for the Pipe State to go off */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
reg, I965_PIPECONF_ACTIVE, 0,
100))
WARN(1, "pipe_off wait timed out\n");
POSTING_READ(DPLL(pipe));
udelay(150);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
DPLL(pipe),
DPLL_LOCK_VLV,
DPLL_LOCK_VLV,
I915_WRITE(DPLL(pipe), pipe_config->dpll_hw_state.dpll);
/* Check PLL is locked */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
DPLL(pipe), DPLL_LOCK_VLV, DPLL_LOCK_VLV,
1))
DRM_ERROR("PLL %d failed to lock\n", pipe);
}
}
-static int intel_num_dvo_pipes(struct drm_i915_private *dev_priv)
+static bool i9xx_has_pps(struct drm_i915_private *dev_priv)
{
- struct intel_crtc *crtc;
- int count = 0;
-
- for_each_intel_crtc(&dev_priv->drm, crtc) {
- count += crtc->base.state->active &&
- intel_crtc_has_type(crtc->config, INTEL_OUTPUT_DVO);
- }
+ if (IS_I830(dev_priv))
+ return false;
- return count;
+ return IS_PINEVIEW(dev_priv) || IS_MOBILE(dev_priv);
}
static void i9xx_enable_pll(struct intel_crtc *crtc,
assert_pipe_disabled(dev_priv, crtc->pipe);
/* PLL is protected by panel, make sure we can write it */
- if (IS_MOBILE(dev_priv) && !IS_I830(dev_priv))
+ if (i9xx_has_pps(dev_priv))
assert_panel_unlocked(dev_priv, crtc->pipe);
- /* Enable DVO 2x clock on both PLLs if necessary */
- if (IS_I830(dev_priv) && intel_num_dvo_pipes(dev_priv) > 0) {
- /*
- * It appears to be important that we don't enable this
- * for the current pipe before otherwise configuring the
- * PLL. No idea how this should be handled if multiple
- * DVO outputs are enabled simultaneosly.
- */
- dpll |= DPLL_DVO_2X_MODE;
- I915_WRITE(DPLL(!crtc->pipe),
- I915_READ(DPLL(!crtc->pipe)) | DPLL_DVO_2X_MODE);
- }
-
/*
* Apparently we need to have VGA mode enabled prior to changing
* the P1/P2 dividers. Otherwise the DPLL will keep using the old
* dividers, even though the register value does change.
*/
- I915_WRITE(reg, 0);
-
+ I915_WRITE(reg, dpll & ~DPLL_VGA_MODE_DIS);
I915_WRITE(reg, dpll);
/* Wait for the clocks to stabilize. */
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
enum pipe pipe = crtc->pipe;
- /* Disable DVO 2x clock on both PLLs if necessary */
- if (IS_I830(dev_priv) &&
- intel_crtc_has_type(crtc_state, INTEL_OUTPUT_DVO) &&
- !intel_num_dvo_pipes(dev_priv)) {
- I915_WRITE(DPLL(PIPE_B),
- I915_READ(DPLL(PIPE_B)) & ~DPLL_DVO_2X_MODE);
- I915_WRITE(DPLL(PIPE_A),
- I915_READ(DPLL(PIPE_A)) & ~DPLL_DVO_2X_MODE);
- }
-
/* Don't disable pipe or pipe PLLs if needed */
if (IS_I830(dev_priv))
return;
BUG();
}
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
dpll_reg, port_mask, expected_mask,
1000))
WARN(1, "timed out waiting for port %c ready: got 0x%x, expected 0x%x\n",
}
val &= ~TRANS_INTERLACE_MASK;
- if ((pipeconf_val & PIPECONF_INTERLACE_MASK) == PIPECONF_INTERLACED_ILK)
+ if ((pipeconf_val & PIPECONF_INTERLACE_MASK) == PIPECONF_INTERLACED_ILK) {
if (HAS_PCH_IBX(dev_priv) &&
intel_crtc_has_type(crtc_state, INTEL_OUTPUT_SDVO))
val |= TRANS_LEGACY_INTERLACED_ILK;
else
val |= TRANS_INTERLACED;
- else
+ } else {
val |= TRANS_PROGRESSIVE;
+ }
I915_WRITE(reg, val | TRANS_ENABLE);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
reg, TRANS_STATE_ENABLE, TRANS_STATE_ENABLE,
100))
DRM_ERROR("failed to enable transcoder %c\n", pipe_name(pipe));
val |= TRANS_PROGRESSIVE;
I915_WRITE(LPT_TRANSCONF, val);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
LPT_TRANSCONF,
TRANS_STATE_ENABLE,
TRANS_STATE_ENABLE,
val &= ~TRANS_ENABLE;
I915_WRITE(reg, val);
/* wait for PCH transcoder off, transcoder state */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
reg, TRANS_STATE_ENABLE, 0,
50))
DRM_ERROR("failed to disable transcoder %c\n", pipe_name(pipe));
val &= ~TRANS_ENABLE;
I915_WRITE(LPT_TRANSCONF, val);
/* wait for PCH transcoder off, transcoder state */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
LPT_TRANSCONF, TRANS_STATE_ENABLE, 0,
50))
DRM_ERROR("Failed to disable PCH transcoder\n");
/* FIXME: assert CPU port conditions for SNB+ */
}
+ trace_intel_pipe_enable(dev_priv, pipe);
+
reg = PIPECONF(cpu_transcoder);
val = I915_READ(reg);
if (val & PIPECONF_ENABLE) {
*/
assert_planes_disabled(crtc);
+ trace_intel_pipe_disable(dev_priv, pipe);
+
reg = PIPECONF(cpu_transcoder);
val = I915_READ(reg);
if ((val & PIPECONF_ENABLE) == 0)
return DRM_FORMAT_RGB565;
case PLANE_CTL_FORMAT_NV12:
return DRM_FORMAT_NV12;
+ case PLANE_CTL_FORMAT_P010:
+ return DRM_FORMAT_P010;
+ case PLANE_CTL_FORMAT_P012:
+ return DRM_FORMAT_P012;
+ case PLANE_CTL_FORMAT_P016:
+ return DRM_FORMAT_P016;
+ case PLANE_CTL_FORMAT_Y210:
+ return DRM_FORMAT_Y210;
+ case PLANE_CTL_FORMAT_Y212:
+ return DRM_FORMAT_Y212;
+ case PLANE_CTL_FORMAT_Y216:
+ return DRM_FORMAT_Y216;
+ case PLANE_CTL_FORMAT_Y410:
+ return DRM_FORMAT_XVYU2101010;
+ case PLANE_CTL_FORMAT_Y412:
+ return DRM_FORMAT_XVYU12_16161616;
+ case PLANE_CTL_FORMAT_Y416:
+ return DRM_FORMAT_XVYU16161616;
default:
case PLANE_CTL_FORMAT_XRGB_8888:
if (rgb_order) {
return DRM_FORMAT_XBGR2101010;
else
return DRM_FORMAT_XRGB2101010;
+ case PLANE_CTL_FORMAT_XRGB_16161616F:
+ if (rgb_order) {
+ if (alpha)
+ return DRM_FORMAT_ABGR16161616F;
+ else
+ return DRM_FORMAT_XBGR16161616F;
+ } else {
+ if (alpha)
+ return DRM_FORMAT_ARGB16161616F;
+ else
+ return DRM_FORMAT_XRGB16161616F;
+ }
}
}
if (plane->id == PLANE_PRIMARY)
intel_pre_disable_primary_noatomic(&crtc->base);
- trace_intel_disable_plane(&plane->base, crtc);
- plane->disable_plane(plane, crtc_state);
+ intel_disable_plane(plane, crtc_state);
}
static void
* Handle the AUX surface first since
* the main surface setup depends on it.
*/
- if (fb->format->format == DRM_FORMAT_NV12) {
+ if (is_planar_yuv_format(fb->format->format)) {
ret = skl_check_nv12_aux_surface(plane_state);
if (ret)
return ret;
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
u32 dspcntr = 0;
- dspcntr |= DISPPLANE_GAMMA_ENABLE;
+ if (crtc_state->gamma_enable)
+ dspcntr |= DISPPLANE_GAMMA_ENABLE;
- if (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv))
+ if (crtc_state->csc_enable)
dspcntr |= DISPPLANE_PIPE_CSC_ENABLE;
if (INTEL_GEN(dev_priv) < 5)
*
* On pre-g4x there is no way to gamma correct the
* pipe bottom color but we'll keep on doing this
- * anyway.
+ * anyway so that the crtc state readout works correctly.
*/
dspcntr = i9xx_plane_ctl_crtc(crtc_state);
return PLANE_CTL_FORMAT_XRGB_2101010;
case DRM_FORMAT_XBGR2101010:
return PLANE_CTL_ORDER_RGBX | PLANE_CTL_FORMAT_XRGB_2101010;
+ case DRM_FORMAT_XBGR16161616F:
+ case DRM_FORMAT_ABGR16161616F:
+ return PLANE_CTL_FORMAT_XRGB_16161616F | PLANE_CTL_ORDER_RGBX;
+ case DRM_FORMAT_XRGB16161616F:
+ case DRM_FORMAT_ARGB16161616F:
+ return PLANE_CTL_FORMAT_XRGB_16161616F;
case DRM_FORMAT_YUYV:
return PLANE_CTL_FORMAT_YUV422 | PLANE_CTL_YUV422_YUYV;
case DRM_FORMAT_YVYU:
return PLANE_CTL_FORMAT_YUV422 | PLANE_CTL_YUV422_VYUY;
case DRM_FORMAT_NV12:
return PLANE_CTL_FORMAT_NV12;
+ case DRM_FORMAT_P010:
+ return PLANE_CTL_FORMAT_P010;
+ case DRM_FORMAT_P012:
+ return PLANE_CTL_FORMAT_P012;
+ case DRM_FORMAT_P016:
+ return PLANE_CTL_FORMAT_P016;
+ case DRM_FORMAT_Y210:
+ return PLANE_CTL_FORMAT_Y210;
+ case DRM_FORMAT_Y212:
+ return PLANE_CTL_FORMAT_Y212;
+ case DRM_FORMAT_Y216:
+ return PLANE_CTL_FORMAT_Y216;
+ case DRM_FORMAT_XVYU2101010:
+ return PLANE_CTL_FORMAT_Y410;
+ case DRM_FORMAT_XVYU12_16161616:
+ return PLANE_CTL_FORMAT_Y412;
+ case DRM_FORMAT_XVYU16161616:
+ return PLANE_CTL_FORMAT_Y416;
default:
MISSING_CASE(pixel_format);
}
if (INTEL_GEN(dev_priv) >= 10 || IS_GEMINILAKE(dev_priv))
return plane_ctl;
- plane_ctl |= PLANE_CTL_PIPE_GAMMA_ENABLE;
- plane_ctl |= PLANE_CTL_PIPE_CSC_ENABLE;
+ if (crtc_state->gamma_enable)
+ plane_ctl |= PLANE_CTL_PIPE_GAMMA_ENABLE;
+
+ if (crtc_state->csc_enable)
+ plane_ctl |= PLANE_CTL_PIPE_CSC_ENABLE;
return plane_ctl;
}
if (INTEL_GEN(dev_priv) >= 11)
return plane_color_ctl;
- plane_color_ctl |= PLANE_COLOR_PIPE_GAMMA_ENABLE;
- plane_color_ctl |= PLANE_COLOR_PIPE_CSC_ENABLE;
+ if (crtc_state->gamma_enable)
+ plane_color_ctl |= PLANE_COLOR_PIPE_GAMMA_ENABLE;
+
+ if (crtc_state->csc_enable)
+ plane_color_ctl |= PLANE_COLOR_PIPE_CSC_ENABLE;
return plane_color_ctl;
}
u32 glk_plane_color_ctl(const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
+ struct drm_i915_private *dev_priv =
+ to_i915(plane_state->base.plane->dev);
const struct drm_framebuffer *fb = plane_state->base.fb;
struct intel_plane *plane = to_intel_plane(plane_state->base.plane);
u32 plane_color_ctl = 0;
plane_color_ctl |= PLANE_COLOR_PLANE_GAMMA_DISABLE;
plane_color_ctl |= glk_plane_color_ctl_alpha(plane_state);
- if (fb->format->is_yuv && !icl_is_hdr_plane(plane)) {
+ if (fb->format->is_yuv && !icl_is_hdr_plane(dev_priv, plane->id)) {
if (plane_state->base.color_encoding == DRM_COLOR_YCBCR_BT709)
plane_color_ctl |= PLANE_COLOR_CSC_MODE_YUV709_TO_RGB709;
else
* The display has been reset as well,
* so need a full re-initialization.
*/
- intel_runtime_pm_disable_interrupts(dev_priv);
- intel_runtime_pm_enable_interrupts(dev_priv);
-
intel_pps_unlock_regs_wa(dev_priv);
intel_modeset_init_hw(dev);
intel_init_clock_gating(dev_priv);
* and rounding for per-pixel values 00 and 0xff
*/
tmp |= PER_PIXEL_ALPHA_BYPASS_EN;
-
/*
- * W/A for underruns with linear/X-tiled with
- * WM1+ disabled.
+ * Display WA # 1605353570: icl
+ * Set the pixel rounding bit to 1 for allowing
+ * passthrough of Frame buffer pixels unmodified
+ * across pipe
*/
- tmp |= PM_FILL_MAINTAIN_DBUF_FULLNESS;
-
+ tmp |= PIXEL_ROUNDING_TRUNC_FB_PASSTHRU;
I915_WRITE(PIPE_CHICKEN(pipe), tmp);
}
ironlake_pfit_disable(old_crtc_state);
}
- /*
- * We don't (yet) allow userspace to control the pipe background color,
- * so force it to black, but apply pipe gamma and CSC so that its
- * handling will match how we program our planes.
- */
- if (INTEL_GEN(dev_priv) >= 9)
- I915_WRITE(SKL_BOTTOM_COLOR(crtc->pipe),
- SKL_BOTTOM_COLOR_GAMMA_ENABLE |
- SKL_BOTTOM_COLOR_CSC_ENABLE);
-
if (INTEL_GEN(dev_priv) >= 11)
icl_set_pipe_chicken(crtc);
}
return 0;
}
- if (format && format->format == DRM_FORMAT_NV12 &&
+ if (format && is_planar_yuv_format(format->format) &&
(src_h < SKL_MIN_YUV_420_SRC_H || src_w < SKL_MIN_YUV_420_SRC_W)) {
- DRM_DEBUG_KMS("NV12: src dimensions not met\n");
+ DRM_DEBUG_KMS("Planar YUV: src dimensions not met\n");
return -EINVAL;
}
/* range checks */
if (src_w < SKL_MIN_SRC_W || src_h < SKL_MIN_SRC_H ||
dst_w < SKL_MIN_DST_W || dst_h < SKL_MIN_DST_H ||
- (IS_GEN(dev_priv, 11) &&
+ (INTEL_GEN(dev_priv) >= 11 &&
(src_w > ICL_MAX_SRC_W || src_h > ICL_MAX_SRC_H ||
dst_w > ICL_MAX_DST_W || dst_h > ICL_MAX_DST_H)) ||
- (!IS_GEN(dev_priv, 11) &&
+ (INTEL_GEN(dev_priv) < 11 &&
(src_w > SKL_MAX_SRC_W || src_h > SKL_MAX_SRC_H ||
dst_w > SKL_MAX_DST_W || dst_h > SKL_MAX_DST_H))) {
DRM_DEBUG_KMS("scaler_user index %u.%u: src %ux%u dst %ux%u "
{
struct intel_plane *intel_plane =
to_intel_plane(plane_state->base.plane);
+ struct drm_i915_private *dev_priv = to_i915(intel_plane->base.dev);
struct drm_framebuffer *fb = plane_state->base.fb;
int ret;
bool force_detach = !fb || !plane_state->base.visible;
bool need_scaler = false;
/* Pre-gen11 and SDR planes always need a scaler for planar formats. */
- if (!icl_is_hdr_plane(intel_plane) &&
- fb && fb->format->format == DRM_FORMAT_NV12)
+ if (!icl_is_hdr_plane(dev_priv, intel_plane->id) &&
+ fb && is_planar_yuv_format(fb->format->format))
need_scaler = true;
ret = skl_update_scaler(crtc_state, force_detach,
case DRM_FORMAT_ARGB8888:
case DRM_FORMAT_XRGB2101010:
case DRM_FORMAT_XBGR2101010:
+ case DRM_FORMAT_XBGR16161616F:
+ case DRM_FORMAT_ABGR16161616F:
+ case DRM_FORMAT_XRGB16161616F:
+ case DRM_FORMAT_ARGB16161616F:
case DRM_FORMAT_YUYV:
case DRM_FORMAT_YVYU:
case DRM_FORMAT_UYVY:
case DRM_FORMAT_VYUY:
case DRM_FORMAT_NV12:
+ case DRM_FORMAT_P010:
+ case DRM_FORMAT_P012:
+ case DRM_FORMAT_P016:
+ case DRM_FORMAT_Y210:
+ case DRM_FORMAT_Y212:
+ case DRM_FORMAT_Y216:
+ case DRM_FORMAT_XVYU2101010:
+ case DRM_FORMAT_XVYU12_16161616:
+ case DRM_FORMAT_XVYU16161616:
break;
default:
DRM_DEBUG_KMS("[PLANE:%d:%s] FB:%d unsupported scaling format 0x%x\n",
* and don't wait for vblanks until the end of crtc_enable, then
* the HW state readout code will complain that the expected
* IPS_CTL value is not the one we read. */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
IPS_CTL, IPS_ENABLE, IPS_ENABLE,
50))
DRM_ERROR("Timed out waiting for IPS enable\n");
* 42ms timeout value leads to occasional timeouts so use 100ms
* instead.
*/
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
IPS_CTL, IPS_ENABLE, 0,
100))
DRM_ERROR("Timed out waiting for IPS disable\n");
/* Display WA 827 */
if (needs_nv12_wa(dev_priv, old_crtc_state) &&
!needs_nv12_wa(dev_priv, pipe_config)) {
- skl_wa_clkgate(dev_priv, crtc->pipe, false);
+ skl_wa_827(dev_priv, crtc->pipe, false);
}
}
/* Display WA 827 */
if (!needs_nv12_wa(dev_priv, old_crtc_state) &&
needs_nv12_wa(dev_priv, pipe_config)) {
- skl_wa_clkgate(dev_priv, crtc->pipe, true);
+ skl_wa_827(dev_priv, crtc->pipe, true);
}
/*
!(update_mask & BIT(plane->id)))
continue;
- plane->disable_plane(plane, new_crtc_state);
+ intel_disable_plane(plane, new_crtc_state);
if (old_plane_state->base.visible)
fb_bits |= plane->frontbuffer_bit;
}
}
+static void intel_disable_primary_plane(const struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ struct intel_plane *plane = to_intel_plane(crtc->base.primary);
+
+ plane->disable_plane(plane, crtc_state);
+}
+
static void ironlake_crtc_enable(struct intel_crtc_state *pipe_config,
struct drm_atomic_state *old_state)
{
*/
intel_color_load_luts(pipe_config);
intel_color_commit(pipe_config);
+ /* update DSPCNTR to configure gamma for pipe bottom color */
+ intel_disable_primary_plane(pipe_config);
if (dev_priv->display.initial_watermarks != NULL)
dev_priv->display.initial_watermarks(old_intel_state, pipe_config);
*/
intel_color_load_luts(pipe_config);
intel_color_commit(pipe_config);
+ /* update DSPCNTR to configure gamma/csc for pipe bottom color */
+ if (INTEL_GEN(dev_priv) < 9)
+ intel_disable_primary_plane(pipe_config);
if (INTEL_GEN(dev_priv) >= 11)
icl_set_pipe_chicken(intel_crtc);
if (port == PORT_NONE)
return false;
- if (IS_ICELAKE(dev_priv))
+ if (IS_ELKHARTLAKE(dev_priv))
+ return port <= PORT_C;
+
+ if (INTEL_GEN(dev_priv) >= 11)
return port <= PORT_B;
return false;
bool intel_port_is_tc(struct drm_i915_private *dev_priv, enum port port)
{
- if (IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 11 && !IS_ELKHARTLAKE(dev_priv))
return port >= PORT_C && port <= PORT_F;
return false;
intel_color_load_luts(pipe_config);
intel_color_commit(pipe_config);
+ /* update DSPCNTR to configure gamma for pipe bottom color */
+ intel_disable_primary_plane(pipe_config);
dev_priv->display.initial_watermarks(old_intel_state,
pipe_config);
intel_color_load_luts(pipe_config);
intel_color_commit(pipe_config);
+ /* update DSPCNTR to configure gamma for pipe bottom color */
+ intel_disable_primary_plane(pipe_config);
if (dev_priv->display.initial_watermarks != NULL)
dev_priv->display.initial_watermarks(old_intel_state,
if (!hsw_crtc_state_ips_capable(crtc_state))
return false;
- if (crtc_state->ips_force_disable)
+ /*
+ * When IPS gets enabled, the pipe CRC changes. Since IPS gets
+ * enabled and disabled dynamically based on package C states,
+ * user space can't make reliable use of the CRCs, so let's just
+ * completely disable it.
+ */
+ if (crtc_state->crc_enabled)
return false;
/* IPS should be fine as long as at least one plane is enabled. */
static int intel_crtc_compute_config(struct intel_crtc *crtc,
struct intel_crtc_state *pipe_config)
{
- struct drm_device *dev = crtc->base.dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
const struct drm_display_mode *adjusted_mode = &pipe_config->base.adjusted_mode;
int clock_limit = dev_priv->max_dotclk_freq;
}
if (intel_crtc_has_type(pipe_config, INTEL_OUTPUT_LVDS) &&
- intel_is_dual_link_lvds(dev)) {
+ intel_is_dual_link_lvds(dev_priv)) {
DRM_DEBUG_KMS("Odd pipe source width not supported with dual link LVDS\n");
return -EINVAL;
}
dpll |= PLL_P2_DIVIDE_BY_4;
}
- if (!IS_I830(dev_priv) &&
+ /*
+ * Bspec:
+ * "[Almador Errata}: For the correct operation of the muxed DVO pins
+ * (GDEVSELB/I2Cdata, GIRDBY/I2CClk) and (GFRAMEB/DVI_Data,
+ * GTRDYB/DVI_Clk): Bit 31 (DPLL VCO Enable) and Bit 30 (2X Clock
+ * Enable) must be set to “1” in both the DPLL A Control Register
+ * (06014h-06017h) and DPLL B Control Register (06018h-0601Bh)."
+ *
+ * For simplicity We simply keep both bits always enabled in
+ * both DPLLS. The spec says we should disable the DVO 2X clock
+ * when not needed, but this seems to work fine in practice.
+ */
+ if (IS_I830(dev_priv) ||
intel_crtc_has_type(crtc_state, INTEL_OUTPUT_DVO))
dpll |= DPLL_DVO_2X_MODE;
pipeconf |= PIPECONF_INTERLACE_W_FIELD_INDICATION;
else
pipeconf |= PIPECONF_INTERLACE_W_SYNC_SHIFT;
- } else
+ } else {
pipeconf |= PIPECONF_PROGRESSIVE;
+ }
if ((IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) &&
crtc_state->limited_color_range)
pipeconf |= PIPECONF_COLOR_RANGE_SELECT;
+ pipeconf |= PIPECONF_GAMMA_MODE(crtc_state->gamma_mode);
+
I915_WRITE(PIPECONF(crtc->pipe), pipeconf);
POSTING_READ(PIPECONF(crtc->pipe));
}
static int g4x_crtc_compute_clock(struct intel_crtc *crtc,
struct intel_crtc_state *crtc_state)
{
- struct drm_device *dev = crtc->base.dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
const struct intel_limit *limit;
int refclk = 96000;
DRM_DEBUG_KMS("using SSC reference clock of %d kHz\n", refclk);
}
- if (intel_is_dual_link_lvds(dev))
+ if (intel_is_dual_link_lvds(dev_priv))
limit = &intel_limits_g4x_dual_channel_lvds;
else
limit = &intel_limits_g4x_single_channel_lvds;
return 0;
}
+static bool i9xx_has_pfit(struct drm_i915_private *dev_priv)
+{
+ if (IS_I830(dev_priv))
+ return false;
+
+ return INTEL_GEN(dev_priv) >= 4 ||
+ IS_PINEVIEW(dev_priv) || IS_MOBILE(dev_priv);
+}
+
static void i9xx_get_pfit_config(struct intel_crtc *crtc,
struct intel_crtc_state *pipe_config)
{
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
u32 tmp;
- if (INTEL_GEN(dev_priv) <= 3 &&
- (IS_I830(dev_priv) || !IS_MOBILE(dev_priv)))
+ if (!i9xx_has_pfit(dev_priv))
return;
tmp = I915_READ(PFIT_CONTROL);
pipe_config->output_format = output;
}
+static void i9xx_get_pipe_color_config(struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ struct intel_plane *plane = to_intel_plane(crtc->base.primary);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ enum i9xx_plane_id i9xx_plane = plane->i9xx_plane;
+ u32 tmp;
+
+ tmp = I915_READ(DSPCNTR(i9xx_plane));
+
+ if (tmp & DISPPLANE_GAMMA_ENABLE)
+ crtc_state->gamma_enable = true;
+
+ if (!HAS_GMCH(dev_priv) &&
+ tmp & DISPPLANE_PIPE_CSC_ENABLE)
+ crtc_state->csc_enable = true;
+}
+
static bool i9xx_get_pipe_config(struct intel_crtc *crtc,
struct intel_crtc_state *pipe_config)
{
(tmp & PIPECONF_COLOR_RANGE_SELECT))
pipe_config->limited_color_range = true;
+ pipe_config->gamma_mode = (tmp & PIPECONF_GAMMA_MODE_MASK_I9XX) >>
+ PIPECONF_GAMMA_MODE_SHIFT;
+
+ if (IS_CHERRYVIEW(dev_priv))
+ pipe_config->cgm_mode = I915_READ(CGM_PIPE_MODE(crtc->pipe));
+
+ i9xx_get_pipe_color_config(pipe_config);
+
if (INTEL_GEN(dev_priv) < 4)
pipe_config->double_wide = tmp & PIPECONF_DOUBLE_WIDE;
}
pipe_config->dpll_hw_state.dpll = I915_READ(DPLL(crtc->pipe));
if (!IS_VALLEYVIEW(dev_priv) && !IS_CHERRYVIEW(dev_priv)) {
- /*
- * DPLL_DVO_2X_MODE must be enabled for both DPLLs
- * on 830. Filter it out here so that we don't
- * report errors due to that.
- */
- if (IS_I830(dev_priv))
- pipe_config->dpll_hw_state.dpll &= ~DPLL_DVO_2X_MODE;
-
pipe_config->dpll_hw_state.fp0 = I915_READ(FP0(crtc->pipe));
pipe_config->dpll_hw_state.fp1 = I915_READ(FP1(crtc->pipe));
} else {
if (crtc_state->limited_color_range)
val |= PIPECONF_COLOR_RANGE_SELECT;
+ val |= PIPECONF_GAMMA_MODE(crtc_state->gamma_mode);
+
I915_WRITE(PIPECONF(pipe), val);
POSTING_READ(PIPECONF(pipe));
}
return i9xx_dpll_compute_m(dpll) < factor * dpll->n;
}
-static void ironlake_compute_dpll(struct intel_crtc *intel_crtc,
+static void ironlake_compute_dpll(struct intel_crtc *crtc,
struct intel_crtc_state *crtc_state,
struct dpll *reduced_clock)
{
- struct drm_crtc *crtc = &intel_crtc->base;
- struct drm_device *dev = crtc->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
u32 dpll, fp, fp2;
int factor;
if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_LVDS)) {
if ((intel_panel_use_ssc(dev_priv) &&
dev_priv->vbt.lvds_ssc_freq == 100000) ||
- (HAS_PCH_IBX(dev_priv) && intel_is_dual_link_lvds(dev)))
+ (HAS_PCH_IBX(dev_priv) &&
+ intel_is_dual_link_lvds(dev_priv)))
factor = 25;
- } else if (crtc_state->sdvo_tv_clock)
+ } else if (crtc_state->sdvo_tv_clock) {
factor = 20;
+ }
fp = i9xx_dpll_compute_fp(&crtc_state->dpll);
static int ironlake_crtc_compute_clock(struct intel_crtc *crtc,
struct intel_crtc_state *crtc_state)
{
- struct drm_device *dev = crtc->base.dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
const struct intel_limit *limit;
int refclk = 120000;
refclk = dev_priv->vbt.lvds_ssc_freq;
}
- if (intel_is_dual_link_lvds(dev)) {
+ if (intel_is_dual_link_lvds(dev_priv)) {
if (refclk == 100000)
limit = &intel_limits_ironlake_dual_lvds_100m;
else
ironlake_compute_dpll(crtc, crtc_state, NULL);
- if (!intel_get_shared_dpll(crtc, crtc_state, NULL)) {
+ if (!intel_get_shared_dpll(crtc_state, NULL)) {
DRM_DEBUG_KMS("failed to find PLL for pipe %c\n",
pipe_name(crtc->pipe));
return -EINVAL;
if (tmp & PIPECONF_COLOR_RANGE_SELECT)
pipe_config->limited_color_range = true;
+ pipe_config->gamma_mode = (tmp & PIPECONF_GAMMA_MODE_MASK_ILK) >>
+ PIPECONF_GAMMA_MODE_SHIFT;
+
+ pipe_config->csc_mode = I915_READ(PIPE_CSC_MODE(crtc->pipe));
+
+ i9xx_get_pipe_color_config(pipe_config);
+
if (I915_READ(PCH_TRANSCONF(crtc->pipe)) & TRANS_ENABLE) {
struct intel_shared_dpll *pll;
enum intel_dpll_id pll_id;
I915_WRITE(LCPLL_CTL, val);
POSTING_READ(LCPLL_CTL);
- if (intel_wait_for_register(dev_priv, LCPLL_CTL, LCPLL_PLL_LOCK, 0, 1))
+ if (intel_wait_for_register(&dev_priv->uncore,
+ LCPLL_CTL, LCPLL_PLL_LOCK, 0, 1))
DRM_ERROR("LCPLL still locked\n");
val = hsw_read_dcomp(dev_priv);
* Make sure we're not on PC8 state before disabling PC8, otherwise
* we'll hang the machine. To prevent PC8 state, just enable force_wake.
*/
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
if (val & LCPLL_POWER_DOWN_ALLOW) {
val &= ~LCPLL_POWER_DOWN_ALLOW;
val &= ~LCPLL_PLL_DISABLE;
I915_WRITE(LCPLL_CTL, val);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
LCPLL_CTL, LCPLL_PLL_LOCK, LCPLL_PLL_LOCK,
5))
DRM_ERROR("LCPLL not locked yet\n");
DRM_ERROR("Switching back to LCPLL failed\n");
}
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
intel_update_cdclk(dev_priv);
intel_dump_cdclk_state(&dev_priv->cdclk.hw, "Current CDCLK");
to_intel_atomic_state(crtc_state->base.state);
if (!intel_crtc_has_type(crtc_state, INTEL_OUTPUT_DSI) ||
- IS_ICELAKE(dev_priv)) {
+ INTEL_GEN(dev_priv) >= 11) {
struct intel_encoder *encoder =
intel_get_crtc_new_encoder(state, crtc_state);
- if (!intel_get_shared_dpll(crtc, crtc_state, encoder)) {
+ if (!intel_get_shared_dpll(crtc_state, encoder)) {
DRM_DEBUG_KMS("failed to find PLL for pipe %c\n",
pipe_name(crtc->pipe));
return -EINVAL;
temp = I915_READ(DPCLKA_CFGCR0_ICL) &
DPCLKA_CFGCR0_DDI_CLK_SEL_MASK(port);
id = temp >> DPCLKA_CFGCR0_DDI_CLK_SEL_SHIFT(port);
-
- if (WARN_ON(!intel_dpll_is_combophy(id)))
- return;
} else if (intel_port_is_tc(dev_priv, port)) {
id = icl_tc_port_to_pll_id(intel_port_to_tc(dev_priv, port));
} else {
static bool hsw_get_transcoder_state(struct intel_crtc *crtc,
struct intel_crtc_state *pipe_config,
- u64 *power_domain_mask)
+ u64 *power_domain_mask,
+ intel_wakeref_t *wakerefs)
{
struct drm_device *dev = crtc->base.dev;
struct drm_i915_private *dev_priv = to_i915(dev);
enum intel_display_power_domain power_domain;
- unsigned long panel_transcoder_mask = BIT(TRANSCODER_EDP);
+ unsigned long panel_transcoder_mask = 0;
unsigned long enabled_panel_transcoders = 0;
enum transcoder panel_transcoder;
+ intel_wakeref_t wf;
u32 tmp;
- if (IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 11)
panel_transcoder_mask |=
BIT(TRANSCODER_DSI_0) | BIT(TRANSCODER_DSI_1);
+ if (HAS_TRANSCODER_EDP(dev_priv))
+ panel_transcoder_mask |= BIT(TRANSCODER_EDP);
+
/*
* The pipe->transcoder mapping is fixed with the exception of the eDP
* and DSI transcoders handled below.
enabled_panel_transcoders != BIT(TRANSCODER_EDP));
power_domain = POWER_DOMAIN_TRANSCODER(pipe_config->cpu_transcoder);
- if (!intel_display_power_get_if_enabled(dev_priv, power_domain))
+ WARN_ON(*power_domain_mask & BIT_ULL(power_domain));
+
+ wf = intel_display_power_get_if_enabled(dev_priv, power_domain);
+ if (!wf)
return false;
- WARN_ON(*power_domain_mask & BIT_ULL(power_domain));
+ wakerefs[power_domain] = wf;
*power_domain_mask |= BIT_ULL(power_domain);
tmp = I915_READ(PIPECONF(pipe_config->cpu_transcoder));
static bool bxt_get_dsi_transcoder_state(struct intel_crtc *crtc,
struct intel_crtc_state *pipe_config,
- u64 *power_domain_mask)
+ u64 *power_domain_mask,
+ intel_wakeref_t *wakerefs)
{
struct drm_device *dev = crtc->base.dev;
struct drm_i915_private *dev_priv = to_i915(dev);
enum intel_display_power_domain power_domain;
- enum port port;
enum transcoder cpu_transcoder;
+ intel_wakeref_t wf;
+ enum port port;
u32 tmp;
for_each_port_masked(port, BIT(PORT_A) | BIT(PORT_C)) {
cpu_transcoder = TRANSCODER_DSI_C;
power_domain = POWER_DOMAIN_TRANSCODER(cpu_transcoder);
- if (!intel_display_power_get_if_enabled(dev_priv, power_domain))
+ WARN_ON(*power_domain_mask & BIT_ULL(power_domain));
+
+ wf = intel_display_power_get_if_enabled(dev_priv, power_domain);
+ if (!wf)
continue;
- WARN_ON(*power_domain_mask & BIT_ULL(power_domain));
+ wakerefs[power_domain] = wf;
*power_domain_mask |= BIT_ULL(power_domain);
/*
port = (tmp & TRANS_DDI_PORT_MASK) >> TRANS_DDI_PORT_SHIFT;
- if (IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 11)
icelake_get_ddi_pll(dev_priv, port, pipe_config);
else if (IS_CANNONLAKE(dev_priv))
cannonlake_get_ddi_pll(dev_priv, port, pipe_config);
struct intel_crtc_state *pipe_config)
{
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ intel_wakeref_t wakerefs[POWER_DOMAIN_NUM], wf;
enum intel_display_power_domain power_domain;
u64 power_domain_mask;
bool active;
intel_crtc_init_scalers(crtc, pipe_config);
power_domain = POWER_DOMAIN_PIPE(crtc->pipe);
- if (!intel_display_power_get_if_enabled(dev_priv, power_domain))
+ wf = intel_display_power_get_if_enabled(dev_priv, power_domain);
+ if (!wf)
return false;
+
+ wakerefs[power_domain] = wf;
power_domain_mask = BIT_ULL(power_domain);
pipe_config->shared_dpll = NULL;
- active = hsw_get_transcoder_state(crtc, pipe_config, &power_domain_mask);
+ active = hsw_get_transcoder_state(crtc, pipe_config,
+ &power_domain_mask, wakerefs);
if (IS_GEN9_LP(dev_priv) &&
- bxt_get_dsi_transcoder_state(crtc, pipe_config, &power_domain_mask)) {
+ bxt_get_dsi_transcoder_state(crtc, pipe_config,
+ &power_domain_mask, wakerefs)) {
WARN_ON(active);
active = true;
}
goto out;
if (!transcoder_is_dsi(pipe_config->cpu_transcoder) ||
- IS_ICELAKE(dev_priv)) {
+ INTEL_GEN(dev_priv) >= 11) {
haswell_get_ddi_port_state(crtc, pipe_config);
intel_get_pipe_timings(crtc, pipe_config);
}
intel_get_pipe_src_size(crtc, pipe_config);
intel_get_crtc_ycbcr_config(crtc, pipe_config);
- pipe_config->gamma_mode =
- I915_READ(GAMMA_MODE(crtc->pipe)) & GAMMA_MODE_MODE_MASK;
+ pipe_config->gamma_mode = I915_READ(GAMMA_MODE(crtc->pipe));
+
+ pipe_config->csc_mode = I915_READ(PIPE_CSC_MODE(crtc->pipe));
+
+ if (INTEL_GEN(dev_priv) >= 9) {
+ u32 tmp = I915_READ(SKL_BOTTOM_COLOR(crtc->pipe));
+
+ if (tmp & SKL_BOTTOM_COLOR_GAMMA_ENABLE)
+ pipe_config->gamma_enable = true;
+
+ if (tmp & SKL_BOTTOM_COLOR_CSC_ENABLE)
+ pipe_config->csc_enable = true;
+ } else {
+ i9xx_get_pipe_color_config(pipe_config);
+ }
power_domain = POWER_DOMAIN_PIPE_PANEL_FITTER(crtc->pipe);
- if (intel_display_power_get_if_enabled(dev_priv, power_domain)) {
- WARN_ON(power_domain_mask & BIT_ULL(power_domain));
+ WARN_ON(power_domain_mask & BIT_ULL(power_domain));
+
+ wf = intel_display_power_get_if_enabled(dev_priv, power_domain);
+ if (wf) {
+ wakerefs[power_domain] = wf;
power_domain_mask |= BIT_ULL(power_domain);
if (INTEL_GEN(dev_priv) >= 9)
out:
for_each_power_domain(power_domain, power_domain_mask)
- intel_display_power_put_unchecked(dev_priv, power_domain);
+ intel_display_power_put(dev_priv,
+ power_domain, wakerefs[power_domain]);
return active;
}
static u32 i845_cursor_ctl_crtc(const struct intel_crtc_state *crtc_state)
{
- return CURSOR_GAMMA_ENABLE;
+ u32 cntl = 0;
+
+ if (crtc_state->gamma_enable)
+ cntl |= CURSOR_GAMMA_ENABLE;
+
+ return cntl;
}
static u32 i845_cursor_ctl(const struct intel_crtc_state *crtc_state,
if (INTEL_GEN(dev_priv) >= 11)
return cntl;
- cntl |= MCURSOR_GAMMA_ENABLE;
+ if (crtc_state->gamma_enable)
+ cntl = MCURSOR_GAMMA_ENABLE;
- if (HAS_DDI(dev_priv))
+ if (crtc_state->csc_enable)
cntl |= MCURSOR_PIPE_CSC_ENABLE;
if (INTEL_GEN(dev_priv) < 5 && !IS_G4X(dev_priv))
}
if (!linked_state) {
- DRM_DEBUG_KMS("Need %d free Y planes for NV12\n",
+ DRM_DEBUG_KMS("Need %d free Y planes for planar YUV\n",
hweight8(crtc_state->nv12_planes));
return -EINVAL;
return ret;
}
- if (mode_changed || crtc_state->color_mgmt_changed) {
+ if (mode_changed || pipe_config->update_pipe ||
+ crtc_state->color_mgmt_changed) {
ret = intel_color_check(pipe_config);
if (ret)
return ret;
-
- /*
- * Changing color management on Intel hardware is
- * handled as part of planes update.
- */
- crtc_state->planes_changed = true;
}
ret = 0;
m_n->link_m, m_n->link_n, m_n->tu);
}
+static void
+intel_dump_infoframe(struct drm_i915_private *dev_priv,
+ const union hdmi_infoframe *frame)
+{
+ if ((drm_debug & DRM_UT_KMS) == 0)
+ return;
+
+ hdmi_infoframe_log(KERN_DEBUG, dev_priv->drm.dev, frame);
+}
+
#define OUTPUT_TYPE(x) [INTEL_OUTPUT_ ## x] = #x
static const char * const output_type_str[] = {
DRM_DEBUG_KMS("audio: %i, infoframes: %i\n",
pipe_config->has_audio, pipe_config->has_infoframe);
+ DRM_DEBUG_KMS("infoframes enabled: 0x%x\n",
+ pipe_config->infoframes.enable);
+
+ if (pipe_config->infoframes.enable &
+ intel_hdmi_infoframe_enable(HDMI_PACKET_TYPE_GENERAL_CONTROL))
+ DRM_DEBUG_KMS("GCP: 0x%x\n", pipe_config->infoframes.gcp);
+ if (pipe_config->infoframes.enable &
+ intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_AVI))
+ intel_dump_infoframe(dev_priv, &pipe_config->infoframes.avi);
+ if (pipe_config->infoframes.enable &
+ intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_SPD))
+ intel_dump_infoframe(dev_priv, &pipe_config->infoframes.spd);
+ if (pipe_config->infoframes.enable &
+ intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_VENDOR))
+ intel_dump_infoframe(dev_priv, &pipe_config->infoframes.hdmi);
+
DRM_DEBUG_KMS("requested mode:\n");
drm_mode_debug_printmodeline(&pipe_config->base.mode);
DRM_DEBUG_KMS("adjusted mode:\n");
saved_state->shared_dpll = crtc_state->shared_dpll;
saved_state->dpll_hw_state = crtc_state->dpll_hw_state;
saved_state->pch_pfit.force_thru = crtc_state->pch_pfit.force_thru;
- saved_state->ips_force_disable = crtc_state->ips_force_disable;
+ saved_state->crc_enabled = crtc_state->crc_enabled;
if (IS_G4X(dev_priv) ||
IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
saved_state->wm = crtc_state->wm;
return false;
}
+static bool
+intel_compare_infoframe(const union hdmi_infoframe *a,
+ const union hdmi_infoframe *b)
+{
+ return memcmp(a, b, sizeof(*a)) == 0;
+}
+
+static void
+pipe_config_infoframe_err(struct drm_i915_private *dev_priv,
+ bool adjust, const char *name,
+ const union hdmi_infoframe *a,
+ const union hdmi_infoframe *b)
+{
+ if (adjust) {
+ if ((drm_debug & DRM_UT_KMS) == 0)
+ return;
+
+ drm_dbg(DRM_UT_KMS, "mismatch in %s infoframe", name);
+ drm_dbg(DRM_UT_KMS, "expected:");
+ hdmi_infoframe_log(KERN_DEBUG, dev_priv->drm.dev, a);
+ drm_dbg(DRM_UT_KMS, "found");
+ hdmi_infoframe_log(KERN_DEBUG, dev_priv->drm.dev, b);
+ } else {
+ drm_err("mismatch in %s infoframe", name);
+ drm_err("expected:");
+ hdmi_infoframe_log(KERN_ERR, dev_priv->drm.dev, a);
+ drm_err("found");
+ hdmi_infoframe_log(KERN_ERR, dev_priv->drm.dev, b);
+ }
+}
+
static void __printf(3, 4)
pipe_config_err(bool adjust, const char *name, const char *format, ...)
{
} \
} while (0)
-#define PIPE_CONF_QUIRK(quirk) \
+#define PIPE_CONF_CHECK_INFOFRAME(name) do { \
+ if (!intel_compare_infoframe(¤t_config->infoframes.name, \
+ &pipe_config->infoframes.name)) { \
+ pipe_config_infoframe_err(dev_priv, adjust, __stringify(name), \
+ ¤t_config->infoframes.name, \
+ &pipe_config->infoframes.name); \
+ ret = false; \
+ } \
+} while (0)
+
+#define PIPE_CONF_QUIRK(quirk) \
((current_config->quirks | pipe_config->quirks) & (quirk))
PIPE_CONF_CHECK_I(cpu_transcoder);
PIPE_CONF_CHECK_I(scaler_state.scaler_id);
PIPE_CONF_CHECK_CLOCK_FUZZY(pixel_rate);
+
+ PIPE_CONF_CHECK_X(gamma_mode);
+ if (IS_CHERRYVIEW(dev_priv))
+ PIPE_CONF_CHECK_X(cgm_mode);
+ else
+ PIPE_CONF_CHECK_X(csc_mode);
+ PIPE_CONF_CHECK_BOOL(gamma_enable);
+ PIPE_CONF_CHECK_BOOL(csc_enable);
}
PIPE_CONF_CHECK_BOOL(double_wide);
PIPE_CONF_CHECK_I(min_voltage_level);
+ PIPE_CONF_CHECK_X(infoframes.enable);
+ PIPE_CONF_CHECK_X(infoframes.gcp);
+ PIPE_CONF_CHECK_INFOFRAME(avi);
+ PIPE_CONF_CHECK_INFOFRAME(spd);
+ PIPE_CONF_CHECK_INFOFRAME(hdmi);
+
#undef PIPE_CONF_CHECK_X
#undef PIPE_CONF_CHECK_I
#undef PIPE_CONF_CHECK_BOOL
struct drm_crtc_state *new_state)
{
struct drm_i915_private *dev_priv = to_i915(crtc->dev);
- struct skl_ddb_allocation hw_ddb, *sw_ddb;
- struct skl_pipe_wm hw_wm, *sw_wm;
- struct skl_plane_wm *hw_plane_wm, *sw_plane_wm;
+ struct skl_hw_state {
+ struct skl_ddb_entry ddb_y[I915_MAX_PLANES];
+ struct skl_ddb_entry ddb_uv[I915_MAX_PLANES];
+ struct skl_ddb_allocation ddb;
+ struct skl_pipe_wm wm;
+ } *hw;
+ struct skl_ddb_allocation *sw_ddb;
+ struct skl_pipe_wm *sw_wm;
struct skl_ddb_entry *hw_ddb_entry, *sw_ddb_entry;
- struct skl_ddb_entry hw_ddb_y[I915_MAX_PLANES];
- struct skl_ddb_entry hw_ddb_uv[I915_MAX_PLANES];
struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
const enum pipe pipe = intel_crtc->pipe;
int plane, level, max_level = ilk_wm_max_level(dev_priv);
if (INTEL_GEN(dev_priv) < 9 || !new_state->active)
return;
- skl_pipe_wm_get_hw_state(intel_crtc, &hw_wm);
+ hw = kzalloc(sizeof(*hw), GFP_KERNEL);
+ if (!hw)
+ return;
+
+ skl_pipe_wm_get_hw_state(intel_crtc, &hw->wm);
sw_wm = &to_intel_crtc_state(new_state)->wm.skl.optimal;
- skl_pipe_ddb_get_hw_state(intel_crtc, hw_ddb_y, hw_ddb_uv);
+ skl_pipe_ddb_get_hw_state(intel_crtc, hw->ddb_y, hw->ddb_uv);
- skl_ddb_get_hw_state(dev_priv, &hw_ddb);
+ skl_ddb_get_hw_state(dev_priv, &hw->ddb);
sw_ddb = &dev_priv->wm.skl_hw.ddb;
- if (INTEL_GEN(dev_priv) >= 11)
- if (hw_ddb.enabled_slices != sw_ddb->enabled_slices)
- DRM_ERROR("mismatch in DBUF Slices (expected %u, got %u)\n",
- sw_ddb->enabled_slices,
- hw_ddb.enabled_slices);
+ if (INTEL_GEN(dev_priv) >= 11 &&
+ hw->ddb.enabled_slices != sw_ddb->enabled_slices)
+ DRM_ERROR("mismatch in DBUF Slices (expected %u, got %u)\n",
+ sw_ddb->enabled_slices,
+ hw->ddb.enabled_slices);
+
/* planes */
for_each_universal_plane(dev_priv, pipe, plane) {
- hw_plane_wm = &hw_wm.planes[plane];
+ struct skl_plane_wm *hw_plane_wm, *sw_plane_wm;
+
+ hw_plane_wm = &hw->wm.planes[plane];
sw_plane_wm = &sw_wm->planes[plane];
/* Watermarks */
}
/* DDB */
- hw_ddb_entry = &hw_ddb_y[plane];
+ hw_ddb_entry = &hw->ddb_y[plane];
sw_ddb_entry = &to_intel_crtc_state(new_state)->wm.skl.plane_ddb_y[plane];
if (!skl_ddb_entry_equal(hw_ddb_entry, sw_ddb_entry)) {
* once the plane becomes visible, we can skip this check
*/
if (1) {
- hw_plane_wm = &hw_wm.planes[PLANE_CURSOR];
+ struct skl_plane_wm *hw_plane_wm, *sw_plane_wm;
+
+ hw_plane_wm = &hw->wm.planes[PLANE_CURSOR];
sw_plane_wm = &sw_wm->planes[PLANE_CURSOR];
/* Watermarks */
}
/* DDB */
- hw_ddb_entry = &hw_ddb_y[PLANE_CURSOR];
+ hw_ddb_entry = &hw->ddb_y[PLANE_CURSOR];
sw_ddb_entry = &to_intel_crtc_state(new_state)->wm.skl.plane_ddb_y[PLANE_CURSOR];
if (!skl_ddb_entry_equal(hw_ddb_entry, sw_ddb_entry)) {
hw_ddb_entry->start, hw_ddb_entry->end);
}
}
+
+ kfree(hw);
}
static void
for_each_new_intel_plane_in_state(state, plane,
plane_state, i)
- assert_plane(plane, plane_state->base.visible);
+ assert_plane(plane, plane_state->slave ||
+ plane_state->base.visible);
}
static void
return -EINVAL;
}
+ /* keep the current setting */
+ if (!intel_state->cdclk.force_min_cdclk_changed)
+ intel_state->cdclk.force_min_cdclk =
+ dev_priv->cdclk.force_min_cdclk;
+
intel_state->modeset = true;
intel_state->active_crtcs = dev_priv->active_crtcs;
intel_state->cdclk.logical = dev_priv->cdclk.logical;
intel_state->cdclk.actual = dev_priv->cdclk.actual;
+ intel_state->cdclk.pipe = INVALID_PIPE;
for_each_oldnew_crtc_in_state(state, crtc, old_crtc_state, new_crtc_state, i) {
if (new_crtc_state->active)
* adjusted_mode bits in the crtc directly.
*/
if (dev_priv->display.modeset_calc_cdclk) {
+ enum pipe pipe;
+
ret = dev_priv->display.modeset_calc_cdclk(state);
if (ret < 0)
return ret;
return ret;
}
+ if (is_power_of_2(intel_state->active_crtcs)) {
+ struct drm_crtc *crtc;
+ struct drm_crtc_state *crtc_state;
+
+ pipe = ilog2(intel_state->active_crtcs);
+ crtc = &intel_get_crtc_for_pipe(dev_priv, pipe)->base;
+ crtc_state = drm_atomic_get_new_crtc_state(state, crtc);
+ if (crtc_state && needs_modeset(crtc_state))
+ pipe = INVALID_PIPE;
+ } else {
+ pipe = INVALID_PIPE;
+ }
+
/* All pipes must be switched off while we change the cdclk. */
- if (intel_cdclk_needs_modeset(&dev_priv->cdclk.actual,
- &intel_state->cdclk.actual)) {
+ if (pipe != INVALID_PIPE &&
+ intel_cdclk_needs_cd2x_update(dev_priv,
+ &dev_priv->cdclk.actual,
+ &intel_state->cdclk.actual)) {
+ ret = intel_lock_all_pipes(state);
+ if (ret < 0)
+ return ret;
+
+ intel_state->cdclk.pipe = pipe;
+ } else if (intel_cdclk_needs_modeset(&dev_priv->cdclk.actual,
+ &intel_state->cdclk.actual)) {
ret = intel_modeset_all_pipes(state);
if (ret < 0)
return ret;
+
+ intel_state->cdclk.pipe = INVALID_PIPE;
}
DRM_DEBUG_KMS("New cdclk calculated to be logical %u kHz, actual %u kHz\n",
DRM_DEBUG_KMS("New voltage level calculated to be logical %u, actual %u\n",
intel_state->cdclk.logical.voltage_level,
intel_state->cdclk.actual.voltage_level);
- } else {
- to_intel_atomic_state(state)->cdclk.logical = dev_priv->cdclk.logical;
}
intel_modeset_clear_plls(state);
struct drm_crtc *crtc;
struct drm_crtc_state *old_crtc_state, *crtc_state;
int ret, i;
- bool any_ms = false;
+ bool any_ms = intel_state->cdclk.force_min_cdclk_changed;
/* Catch I915_MODE_FLAG_INHERITED */
for_each_oldnew_crtc_in_state(state, crtc, old_crtc_state,
else if (new_plane_state)
intel_fbc_enable(intel_crtc, pipe_config, new_plane_state);
- intel_begin_crtc_commit(crtc, old_crtc_state);
+ intel_begin_crtc_commit(to_intel_atomic_state(state), intel_crtc);
if (INTEL_GEN(dev_priv) >= 9)
skl_update_planes_on_crtc(to_intel_atomic_state(state), intel_crtc);
else
i9xx_update_planes_on_crtc(to_intel_atomic_state(state), intel_crtc);
- intel_finish_crtc_commit(crtc, old_crtc_state);
+ intel_finish_crtc_commit(to_intel_atomic_state(state), intel_crtc);
}
static void intel_update_crtcs(struct drm_atomic_state *state)
if (intel_state->modeset) {
drm_atomic_helper_update_legacy_modeset_state(state->dev, state);
- intel_set_cdclk(dev_priv, &dev_priv->cdclk.actual);
+ intel_set_cdclk_pre_plane_update(dev_priv,
+ &intel_state->cdclk.actual,
+ &dev_priv->cdclk.actual,
+ intel_state->cdclk.pipe);
/*
* SKL workaround: bspec recommends we disable the SAGV when we
/* Now enable the clocks, plane, pipe, and connectors that we set up. */
dev_priv->display.update_crtcs(state);
+ if (intel_state->modeset)
+ intel_set_cdclk_post_plane_update(dev_priv,
+ &intel_state->cdclk.actual,
+ &dev_priv->cdclk.actual,
+ intel_state->cdclk.pipe);
+
/* FIXME: We should call drm_atomic_helper_commit_hw_done() here
* already, but still need the state for the delayed optimization. To
* fix this:
* so enable debugging for the next modeset - and hope we catch
* the culprit.
*/
- intel_uncore_arm_unclaimed_mmio_detection(dev_priv);
+ intel_uncore_arm_unclaimed_mmio_detection(&dev_priv->uncore);
intel_display_power_put(dev_priv, POWER_DOMAIN_MODESET, wakeref);
}
intel_state->min_voltage_level,
sizeof(intel_state->min_voltage_level));
dev_priv->active_crtcs = intel_state->active_crtcs;
- dev_priv->cdclk.logical = intel_state->cdclk.logical;
- dev_priv->cdclk.actual = intel_state->cdclk.actual;
+ dev_priv->cdclk.force_min_cdclk =
+ intel_state->cdclk.force_min_cdclk;
+
+ intel_cdclk_swap_state(intel_state);
}
drm_atomic_state_get(state);
* vblank without our intervention, so leave RPS alone.
*/
if (!i915_request_started(rq))
- gen6_rps_boost(rq, NULL);
+ gen6_rps_boost(rq);
i915_request_put(rq);
drm_crtc_vblank_put(wait->crtc);
* or
* cdclk/crtc_clock
*/
- mult = pixel_format == DRM_FORMAT_NV12 ? 2 : 3;
+ mult = is_planar_yuv_format(pixel_format) ? 2 : 3;
tmpclk1 = (1 << 16) * mult - 1;
tmpclk2 = (1 << 8) * ((max_dotclk << 8) / crtc_clock);
max_scale = min(tmpclk1, tmpclk2);
return max_scale;
}
-static void intel_begin_crtc_commit(struct drm_crtc *crtc,
- struct drm_crtc_state *old_crtc_state)
+static void intel_begin_crtc_commit(struct intel_atomic_state *state,
+ struct intel_crtc *crtc)
{
- struct drm_device *dev = crtc->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
- struct intel_crtc_state *old_intel_cstate =
- to_intel_crtc_state(old_crtc_state);
- struct intel_atomic_state *old_intel_state =
- to_intel_atomic_state(old_crtc_state->state);
- struct intel_crtc_state *intel_cstate =
- intel_atomic_get_new_crtc_state(old_intel_state, intel_crtc);
- bool modeset = needs_modeset(&intel_cstate->base);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ struct intel_crtc_state *old_crtc_state =
+ intel_atomic_get_old_crtc_state(state, crtc);
+ struct intel_crtc_state *new_crtc_state =
+ intel_atomic_get_new_crtc_state(state, crtc);
+ bool modeset = needs_modeset(&new_crtc_state->base);
/* Perform vblank evasion around commit operation */
- intel_pipe_update_start(intel_cstate);
+ intel_pipe_update_start(new_crtc_state);
if (modeset)
goto out;
- if (intel_cstate->base.color_mgmt_changed ||
- intel_cstate->update_pipe)
- intel_color_commit(intel_cstate);
+ if (new_crtc_state->base.color_mgmt_changed ||
+ new_crtc_state->update_pipe)
+ intel_color_commit(new_crtc_state);
- if (intel_cstate->update_pipe)
- intel_update_pipe_config(old_intel_cstate, intel_cstate);
+ if (new_crtc_state->update_pipe)
+ intel_update_pipe_config(old_crtc_state, new_crtc_state);
else if (INTEL_GEN(dev_priv) >= 9)
- skl_detach_scalers(intel_cstate);
+ skl_detach_scalers(new_crtc_state);
out:
if (dev_priv->display.atomic_update_watermarks)
- dev_priv->display.atomic_update_watermarks(old_intel_state,
- intel_cstate);
+ dev_priv->display.atomic_update_watermarks(state,
+ new_crtc_state);
}
void intel_crtc_arm_fifo_underrun(struct intel_crtc *crtc,
}
}
-static void intel_finish_crtc_commit(struct drm_crtc *crtc,
- struct drm_crtc_state *old_crtc_state)
+static void intel_finish_crtc_commit(struct intel_atomic_state *state,
+ struct intel_crtc *crtc)
{
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
- struct intel_atomic_state *old_intel_state =
- to_intel_atomic_state(old_crtc_state->state);
+ struct intel_crtc_state *old_crtc_state =
+ intel_atomic_get_old_crtc_state(state, crtc);
struct intel_crtc_state *new_crtc_state =
- intel_atomic_get_new_crtc_state(old_intel_state, intel_crtc);
+ intel_atomic_get_new_crtc_state(state, crtc);
intel_pipe_update_end(new_crtc_state);
if (new_crtc_state->update_pipe &&
!needs_modeset(&new_crtc_state->base) &&
- old_crtc_state->mode.private_flags & I915_MODE_FLAG_INHERITED)
- intel_crtc_arm_fifo_underrun(intel_crtc, new_crtc_state);
+ old_crtc_state->base.mode.private_flags & I915_MODE_FLAG_INHERITED)
+ intel_crtc_arm_fifo_underrun(crtc, new_crtc_state);
}
/**
*/
crtc_state->active_planes = new_crtc_state->active_planes;
- if (plane->state->visible) {
- trace_intel_update_plane(plane, to_intel_crtc(crtc));
- intel_plane->update_plane(intel_plane, crtc_state,
- to_intel_plane_state(plane->state));
- } else {
- trace_intel_disable_plane(plane, to_intel_crtc(crtc));
- intel_plane->disable_plane(intel_plane, crtc_state);
- }
+ if (plane->state->visible)
+ intel_update_plane(intel_plane, crtc_state,
+ to_intel_plane_state(plane->state));
+ else
+ intel_disable_plane(intel_plane, crtc_state);
intel_plane_unpin_fb(to_intel_plane_state(old_plane_state));
if (!HAS_DISPLAY(dev_priv))
return;
- if (IS_ICELAKE(dev_priv)) {
+ if (IS_ELKHARTLAKE(dev_priv)) {
+ intel_ddi_init(dev_priv, PORT_A);
+ intel_ddi_init(dev_priv, PORT_B);
+ intel_ddi_init(dev_priv, PORT_C);
+ icl_dsi_init(dev_priv);
+ } else if (INTEL_GEN(dev_priv) >= 11) {
intel_ddi_init(dev_priv, PORT_A);
intel_ddi_init(dev_priv, PORT_B);
intel_ddi_init(dev_priv, PORT_C);
intel_update_czclk(dev_priv);
intel_modeset_init_hw(dev);
+ intel_hdcp_component_init(dev_priv);
+
if (dev_priv->max_cdclk_freq == 0)
intel_update_max_cdclk(dev_priv);
pipe_name(pipe), clock.vco, clock.dot);
fp = i9xx_dpll_compute_fp(&clock);
- dpll = (I915_READ(DPLL(pipe)) & DPLL_DVO_2X_MODE) |
+ dpll = DPLL_DVO_2X_MODE |
DPLL_VGA_MODE_DIS |
((clock.p1 - 2) << DPLL_FPA01_P1_POST_DIV_SHIFT) |
PLL_P2_DIVIDE_BY_4 |
/* flush any delayed tasks or pending work */
flush_scheduled_work();
+ intel_hdcp_component_fini(dev_priv);
+
drm_mode_config_cleanup(dev);
intel_overlay_cleanup(dev_priv);
u32 power_well_driver;
- int num_transcoders;
-
struct intel_cursor_error_state {
u32 control;
u32 position;
} plane[I915_MAX_PIPES];
struct intel_transcoder_error_state {
+ bool available;
bool power_domain_on;
enum transcoder cpu_transcoder;
};
int i;
+ BUILD_BUG_ON(ARRAY_SIZE(transcoders) != ARRAY_SIZE(error->transcoder));
+
if (!HAS_DISPLAY(dev_priv))
return NULL;
error->pipe[i].stat = I915_READ(PIPESTAT(i));
}
- /* Note: this does not include DSI transcoders. */
- error->num_transcoders = INTEL_INFO(dev_priv)->num_pipes;
- if (HAS_DDI(dev_priv))
- error->num_transcoders++; /* Account for eDP. */
-
- for (i = 0; i < error->num_transcoders; i++) {
+ for (i = 0; i < ARRAY_SIZE(error->transcoder); i++) {
enum transcoder cpu_transcoder = transcoders[i];
+ if (!INTEL_INFO(dev_priv)->trans_offsets[cpu_transcoder])
+ continue;
+
+ error->transcoder[i].available = true;
error->transcoder[i].power_domain_on =
__intel_display_power_is_enabled(dev_priv,
POWER_DOMAIN_TRANSCODER(cpu_transcoder));
err_printf(m, " BASE: %08x\n", error->cursor[i].base);
}
- for (i = 0; i < error->num_transcoders; i++) {
+ for (i = 0; i < ARRAY_SIZE(error->transcoder); i++) {
+ if (!error->transcoder[i].available)
+ continue;
+
err_printf(m, "CPU transcoder: %s\n",
transcoder_name(error->transcoder[i].cpu_transcoder));
err_printf(m, " Power: %s\n",
*
*/
-#include <linux/i2c.h>
-#include <linux/slab.h>
#include <linux/export.h>
-#include <linux/types.h>
+#include <linux/i2c.h>
#include <linux/notifier.h>
#include <linux/reboot.h>
+#include <linux/slab.h>
+#include <linux/types.h>
#include <asm/byteorder.h>
+
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
#include <drm/drm_dp_helper.h>
#include <drm/drm_edid.h>
#include <drm/drm_hdcp.h>
#include <drm/drm_probe_helper.h>
-#include "intel_drv.h"
#include <drm/i915_drm.h>
+
#include "i915_drv.h"
+#include "intel_audio.h"
+#include "intel_connector.h"
+#include "intel_ddi.h"
+#include "intel_dp.h"
+#include "intel_drv.h"
+#include "intel_hdcp.h"
+#include "intel_hdmi.h"
+#include "intel_lspcon.h"
+#include "intel_lvds.h"
+#include "intel_panel.h"
+#include "intel_psr.h"
#define DP_DPRX_ESI_LEN 14
regs->pp_stat = PP_STATUS(pps_idx);
regs->pp_on = PP_ON_DELAYS(pps_idx);
regs->pp_off = PP_OFF_DELAYS(pps_idx);
- if (!IS_GEN9_LP(dev_priv) && !HAS_PCH_CNP(dev_priv) &&
- !HAS_PCH_ICP(dev_priv))
+
+ /* Cycle delay moved from PP_DIVISOR to PP_CONTROL */
+ if (IS_GEN9_LP(dev_priv) || INTEL_PCH_TYPE(dev_priv) >= PCH_CNP)
+ regs->pp_div = INVALID_MMIO_REG;
+ else
regs->pp_div = PP_DIVISOR(pps_idx);
}
}
}
-struct link_config_limits {
- int min_clock, max_clock;
- int min_lane_count, max_lane_count;
- int min_bpp, max_bpp;
-};
-
static bool intel_dp_source_supports_fec(struct intel_dp *intel_dp,
const struct intel_crtc_state *pipe_config)
{
}
/* Adjust link config limits based on compliance test requests. */
-static void
+void
intel_dp_adjust_compliance_config(struct intel_dp *intel_dp,
struct intel_crtc_state *pipe_config,
struct link_config_limits *limits)
return 0;
}
+int intel_dp_min_bpp(const struct intel_crtc_state *crtc_state)
+{
+ if (crtc_state->output_format == INTEL_OUTPUT_FORMAT_RGB)
+ return 6 * 3;
+ else
+ return 8 * 3;
+}
+
static int
intel_dp_compute_link_config(struct intel_encoder *encoder,
struct intel_crtc_state *pipe_config,
limits.min_lane_count = 1;
limits.max_lane_count = intel_dp_max_lane_count(intel_dp);
- limits.min_bpp = 6 * 3;
+ limits.min_bpp = intel_dp_min_bpp(pipe_config);
limits.max_bpp = intel_dp_compute_bpp(intel_dp, pipe_config);
if (intel_dp_is_edp(intel_dp)) {
return 0;
}
+bool intel_dp_limited_color_range(const struct intel_crtc_state *crtc_state,
+ const struct drm_connector_state *conn_state)
+{
+ const struct intel_digital_connector_state *intel_conn_state =
+ to_intel_digital_connector_state(conn_state);
+ const struct drm_display_mode *adjusted_mode =
+ &crtc_state->base.adjusted_mode;
+
+ if (intel_conn_state->broadcast_rgb == INTEL_BROADCAST_RGB_AUTO) {
+ /*
+ * See:
+ * CEA-861-E - 5.1 Default Encoding Parameters
+ * VESA DisplayPort Ver.1.2a - 5.1.1.1 Video Colorimetry
+ */
+ return crtc_state->pipe_bpp != 18 &&
+ drm_default_rgb_quant_range(adjusted_mode) ==
+ HDMI_QUANTIZATION_RANGE_LIMITED;
+ } else {
+ return intel_conn_state->broadcast_rgb ==
+ INTEL_BROADCAST_RGB_LIMITED;
+ }
+}
+
int
intel_dp_compute_config(struct intel_encoder *encoder,
struct intel_crtc_state *pipe_config,
to_intel_digital_connector_state(conn_state);
bool constant_n = drm_dp_has_quirk(&intel_dp->desc,
DP_DPCD_QUIRK_CONSTANT_N);
- int ret;
+ int ret, output_bpp;
if (HAS_PCH_SPLIT(dev_priv) && !HAS_DDI(dev_priv) && port != PORT_A)
pipe_config->has_pch_encoder = true;
if (ret < 0)
return ret;
- if (intel_conn_state->broadcast_rgb == INTEL_BROADCAST_RGB_AUTO) {
- /*
- * See:
- * CEA-861-E - 5.1 Default Encoding Parameters
- * VESA DisplayPort Ver.1.2a - 5.1.1.1 Video Colorimetry
- */
- pipe_config->limited_color_range =
- pipe_config->pipe_bpp != 18 &&
- drm_default_rgb_quant_range(adjusted_mode) ==
- HDMI_QUANTIZATION_RANGE_LIMITED;
- } else {
- pipe_config->limited_color_range =
- intel_conn_state->broadcast_rgb == INTEL_BROADCAST_RGB_LIMITED;
- }
+ pipe_config->limited_color_range =
+ intel_dp_limited_color_range(pipe_config, conn_state);
- if (!pipe_config->dsc_params.compression_enable)
- intel_link_compute_m_n(pipe_config->pipe_bpp,
- pipe_config->lane_count,
- adjusted_mode->crtc_clock,
- pipe_config->port_clock,
- &pipe_config->dp_m_n,
- constant_n);
+ if (pipe_config->dsc_params.compression_enable)
+ output_bpp = pipe_config->dsc_params.compressed_bpp;
else
- intel_link_compute_m_n(pipe_config->dsc_params.compressed_bpp,
- pipe_config->lane_count,
- adjusted_mode->crtc_clock,
- pipe_config->port_clock,
- &pipe_config->dp_m_n,
- constant_n);
+ output_bpp = pipe_config->pipe_bpp;
+
+ intel_link_compute_m_n(output_bpp,
+ pipe_config->lane_count,
+ adjusted_mode->crtc_clock,
+ pipe_config->port_clock,
+ &pipe_config->dp_m_n,
+ constant_n);
if (intel_connector->panel.downclock_mode != NULL &&
dev_priv->drrs.type == SEAMLESS_DRRS_SUPPORT) {
pipe_config->has_drrs = true;
- intel_link_compute_m_n(pipe_config->pipe_bpp,
+ intel_link_compute_m_n(output_bpp,
pipe_config->lane_count,
intel_connector->panel.downclock_mode->clock,
pipe_config->port_clock,
I915_READ(pp_stat_reg),
I915_READ(pp_ctrl_reg));
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
pp_stat_reg, mask, value,
5000))
DRM_ERROR("Panel status timeout: status %08x control %08x\n",
if (port == PORT_A)
return;
- if (intel_wait_for_register(dev_priv,DP_TP_STATUS(port),
+ if (intel_wait_for_register(&dev_priv->uncore, DP_TP_STATUS(port),
DP_TP_STATUS_IDLE_DONE,
DP_TP_STATUS_IDLE_DONE,
1))
intel_dp_handle_test_request(intel_dp);
if (val & DP_CP_IRQ)
- intel_hdcp_check_link(intel_dp->attached_connector);
+ intel_hdcp_handle_cp_irq(intel_dp->attached_connector);
if (val & DP_SINK_SPECIFIC_IRQ)
DRM_DEBUG_DRIVER("Sink specific irq unhandled\n");
edp_panel_vdd_off_sync(intel_dp);
}
+static void intel_dp_hdcp_wait_for_cp_irq(struct intel_hdcp *hdcp, int timeout)
+{
+ long ret;
+
+#define C (hdcp->cp_irq_count_cached != atomic_read(&hdcp->cp_irq_count))
+ ret = wait_event_interruptible_timeout(hdcp->cp_irq_queue, C,
+ msecs_to_jiffies(timeout));
+
+ if (!ret)
+ DRM_DEBUG_KMS("Timedout at waiting for CP_IRQ\n");
+}
+
static
int intel_dp_hdcp_write_an_aksv(struct intel_digital_port *intel_dig_port,
u8 *an)
return 0;
}
+struct hdcp2_dp_errata_stream_type {
+ u8 msg_id;
+ u8 stream_type;
+} __packed;
+
+static struct hdcp2_dp_msg_data {
+ u8 msg_id;
+ u32 offset;
+ bool msg_detectable;
+ u32 timeout;
+ u32 timeout2; /* Added for non_paired situation */
+ } hdcp2_msg_data[] = {
+ {HDCP_2_2_AKE_INIT, DP_HDCP_2_2_AKE_INIT_OFFSET, false, 0, 0},
+ {HDCP_2_2_AKE_SEND_CERT, DP_HDCP_2_2_AKE_SEND_CERT_OFFSET,
+ false, HDCP_2_2_CERT_TIMEOUT_MS, 0},
+ {HDCP_2_2_AKE_NO_STORED_KM, DP_HDCP_2_2_AKE_NO_STORED_KM_OFFSET,
+ false, 0, 0},
+ {HDCP_2_2_AKE_STORED_KM, DP_HDCP_2_2_AKE_STORED_KM_OFFSET,
+ false, 0, 0},
+ {HDCP_2_2_AKE_SEND_HPRIME, DP_HDCP_2_2_AKE_SEND_HPRIME_OFFSET,
+ true, HDCP_2_2_HPRIME_PAIRED_TIMEOUT_MS,
+ HDCP_2_2_HPRIME_NO_PAIRED_TIMEOUT_MS},
+ {HDCP_2_2_AKE_SEND_PAIRING_INFO,
+ DP_HDCP_2_2_AKE_SEND_PAIRING_INFO_OFFSET, true,
+ HDCP_2_2_PAIRING_TIMEOUT_MS, 0},
+ {HDCP_2_2_LC_INIT, DP_HDCP_2_2_LC_INIT_OFFSET, false, 0, 0},
+ {HDCP_2_2_LC_SEND_LPRIME, DP_HDCP_2_2_LC_SEND_LPRIME_OFFSET,
+ false, HDCP_2_2_DP_LPRIME_TIMEOUT_MS, 0},
+ {HDCP_2_2_SKE_SEND_EKS, DP_HDCP_2_2_SKE_SEND_EKS_OFFSET, false,
+ 0, 0},
+ {HDCP_2_2_REP_SEND_RECVID_LIST,
+ DP_HDCP_2_2_REP_SEND_RECVID_LIST_OFFSET, true,
+ HDCP_2_2_RECVID_LIST_TIMEOUT_MS, 0},
+ {HDCP_2_2_REP_SEND_ACK, DP_HDCP_2_2_REP_SEND_ACK_OFFSET, false,
+ 0, 0},
+ {HDCP_2_2_REP_STREAM_MANAGE,
+ DP_HDCP_2_2_REP_STREAM_MANAGE_OFFSET, false,
+ 0, 0},
+ {HDCP_2_2_REP_STREAM_READY, DP_HDCP_2_2_REP_STREAM_READY_OFFSET,
+ false, HDCP_2_2_STREAM_READY_TIMEOUT_MS, 0},
+/* local define to shovel this through the write_2_2 interface */
+#define HDCP_2_2_ERRATA_DP_STREAM_TYPE 50
+ {HDCP_2_2_ERRATA_DP_STREAM_TYPE,
+ DP_HDCP_2_2_REG_STREAM_TYPE_OFFSET, false,
+ 0, 0},
+ };
+
+static inline
+int intel_dp_hdcp2_read_rx_status(struct intel_digital_port *intel_dig_port,
+ u8 *rx_status)
+{
+ ssize_t ret;
+
+ ret = drm_dp_dpcd_read(&intel_dig_port->dp.aux,
+ DP_HDCP_2_2_REG_RXSTATUS_OFFSET, rx_status,
+ HDCP_2_2_DP_RXSTATUS_LEN);
+ if (ret != HDCP_2_2_DP_RXSTATUS_LEN) {
+ DRM_DEBUG_KMS("Read bstatus from DP/AUX failed (%zd)\n", ret);
+ return ret >= 0 ? -EIO : ret;
+ }
+
+ return 0;
+}
+
+static
+int hdcp2_detect_msg_availability(struct intel_digital_port *intel_dig_port,
+ u8 msg_id, bool *msg_ready)
+{
+ u8 rx_status;
+ int ret;
+
+ *msg_ready = false;
+ ret = intel_dp_hdcp2_read_rx_status(intel_dig_port, &rx_status);
+ if (ret < 0)
+ return ret;
+
+ switch (msg_id) {
+ case HDCP_2_2_AKE_SEND_HPRIME:
+ if (HDCP_2_2_DP_RXSTATUS_H_PRIME(rx_status))
+ *msg_ready = true;
+ break;
+ case HDCP_2_2_AKE_SEND_PAIRING_INFO:
+ if (HDCP_2_2_DP_RXSTATUS_PAIRING(rx_status))
+ *msg_ready = true;
+ break;
+ case HDCP_2_2_REP_SEND_RECVID_LIST:
+ if (HDCP_2_2_DP_RXSTATUS_READY(rx_status))
+ *msg_ready = true;
+ break;
+ default:
+ DRM_ERROR("Unidentified msg_id: %d\n", msg_id);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static ssize_t
+intel_dp_hdcp2_wait_for_msg(struct intel_digital_port *intel_dig_port,
+ struct hdcp2_dp_msg_data *hdcp2_msg_data)
+{
+ struct intel_dp *dp = &intel_dig_port->dp;
+ struct intel_hdcp *hdcp = &dp->attached_connector->hdcp;
+ u8 msg_id = hdcp2_msg_data->msg_id;
+ int ret, timeout;
+ bool msg_ready = false;
+
+ if (msg_id == HDCP_2_2_AKE_SEND_HPRIME && !hdcp->is_paired)
+ timeout = hdcp2_msg_data->timeout2;
+ else
+ timeout = hdcp2_msg_data->timeout;
+
+ /*
+ * There is no way to detect the CERT, LPRIME and STREAM_READY
+ * availability. So Wait for timeout and read the msg.
+ */
+ if (!hdcp2_msg_data->msg_detectable) {
+ mdelay(timeout);
+ ret = 0;
+ } else {
+ /*
+ * As we want to check the msg availability at timeout, Ignoring
+ * the timeout at wait for CP_IRQ.
+ */
+ intel_dp_hdcp_wait_for_cp_irq(hdcp, timeout);
+ ret = hdcp2_detect_msg_availability(intel_dig_port,
+ msg_id, &msg_ready);
+ if (!msg_ready)
+ ret = -ETIMEDOUT;
+ }
+
+ if (ret)
+ DRM_DEBUG_KMS("msg_id %d, ret %d, timeout(mSec): %d\n",
+ hdcp2_msg_data->msg_id, ret, timeout);
+
+ return ret;
+}
+
+static struct hdcp2_dp_msg_data *get_hdcp2_dp_msg_data(u8 msg_id)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(hdcp2_msg_data); i++)
+ if (hdcp2_msg_data[i].msg_id == msg_id)
+ return &hdcp2_msg_data[i];
+
+ return NULL;
+}
+
+static
+int intel_dp_hdcp2_write_msg(struct intel_digital_port *intel_dig_port,
+ void *buf, size_t size)
+{
+ struct intel_dp *dp = &intel_dig_port->dp;
+ struct intel_hdcp *hdcp = &dp->attached_connector->hdcp;
+ unsigned int offset;
+ u8 *byte = buf;
+ ssize_t ret, bytes_to_write, len;
+ struct hdcp2_dp_msg_data *hdcp2_msg_data;
+
+ hdcp2_msg_data = get_hdcp2_dp_msg_data(*byte);
+ if (!hdcp2_msg_data)
+ return -EINVAL;
+
+ offset = hdcp2_msg_data->offset;
+
+ /* No msg_id in DP HDCP2.2 msgs */
+ bytes_to_write = size - 1;
+ byte++;
+
+ hdcp->cp_irq_count_cached = atomic_read(&hdcp->cp_irq_count);
+
+ while (bytes_to_write) {
+ len = bytes_to_write > DP_AUX_MAX_PAYLOAD_BYTES ?
+ DP_AUX_MAX_PAYLOAD_BYTES : bytes_to_write;
+
+ ret = drm_dp_dpcd_write(&intel_dig_port->dp.aux,
+ offset, (void *)byte, len);
+ if (ret < 0)
+ return ret;
+
+ bytes_to_write -= ret;
+ byte += ret;
+ offset += ret;
+ }
+
+ return size;
+}
+
+static
+ssize_t get_receiver_id_list_size(struct intel_digital_port *intel_dig_port)
+{
+ u8 rx_info[HDCP_2_2_RXINFO_LEN];
+ u32 dev_cnt;
+ ssize_t ret;
+
+ ret = drm_dp_dpcd_read(&intel_dig_port->dp.aux,
+ DP_HDCP_2_2_REG_RXINFO_OFFSET,
+ (void *)rx_info, HDCP_2_2_RXINFO_LEN);
+ if (ret != HDCP_2_2_RXINFO_LEN)
+ return ret >= 0 ? -EIO : ret;
+
+ dev_cnt = (HDCP_2_2_DEV_COUNT_HI(rx_info[0]) << 4 |
+ HDCP_2_2_DEV_COUNT_LO(rx_info[1]));
+
+ if (dev_cnt > HDCP_2_2_MAX_DEVICE_COUNT)
+ dev_cnt = HDCP_2_2_MAX_DEVICE_COUNT;
+
+ ret = sizeof(struct hdcp2_rep_send_receiverid_list) -
+ HDCP_2_2_RECEIVER_IDS_MAX_LEN +
+ (dev_cnt * HDCP_2_2_RECEIVER_ID_LEN);
+
+ return ret;
+}
+
+static
+int intel_dp_hdcp2_read_msg(struct intel_digital_port *intel_dig_port,
+ u8 msg_id, void *buf, size_t size)
+{
+ unsigned int offset;
+ u8 *byte = buf;
+ ssize_t ret, bytes_to_recv, len;
+ struct hdcp2_dp_msg_data *hdcp2_msg_data;
+
+ hdcp2_msg_data = get_hdcp2_dp_msg_data(msg_id);
+ if (!hdcp2_msg_data)
+ return -EINVAL;
+ offset = hdcp2_msg_data->offset;
+
+ ret = intel_dp_hdcp2_wait_for_msg(intel_dig_port, hdcp2_msg_data);
+ if (ret < 0)
+ return ret;
+
+ if (msg_id == HDCP_2_2_REP_SEND_RECVID_LIST) {
+ ret = get_receiver_id_list_size(intel_dig_port);
+ if (ret < 0)
+ return ret;
+
+ size = ret;
+ }
+ bytes_to_recv = size - 1;
+
+ /* DP adaptation msgs has no msg_id */
+ byte++;
+
+ while (bytes_to_recv) {
+ len = bytes_to_recv > DP_AUX_MAX_PAYLOAD_BYTES ?
+ DP_AUX_MAX_PAYLOAD_BYTES : bytes_to_recv;
+
+ ret = drm_dp_dpcd_read(&intel_dig_port->dp.aux, offset,
+ (void *)byte, len);
+ if (ret < 0) {
+ DRM_DEBUG_KMS("msg_id %d, ret %zd\n", msg_id, ret);
+ return ret;
+ }
+
+ bytes_to_recv -= ret;
+ byte += ret;
+ offset += ret;
+ }
+ byte = buf;
+ *byte = msg_id;
+
+ return size;
+}
+
+static
+int intel_dp_hdcp2_config_stream_type(struct intel_digital_port *intel_dig_port,
+ bool is_repeater, u8 content_type)
+{
+ struct hdcp2_dp_errata_stream_type stream_type_msg;
+
+ if (is_repeater)
+ return 0;
+
+ /*
+ * Errata for DP: As Stream type is used for encryption, Receiver
+ * should be communicated with stream type for the decryption of the
+ * content.
+ * Repeater will be communicated with stream type as a part of it's
+ * auth later in time.
+ */
+ stream_type_msg.msg_id = HDCP_2_2_ERRATA_DP_STREAM_TYPE;
+ stream_type_msg.stream_type = content_type;
+
+ return intel_dp_hdcp2_write_msg(intel_dig_port, &stream_type_msg,
+ sizeof(stream_type_msg));
+}
+
+static
+int intel_dp_hdcp2_check_link(struct intel_digital_port *intel_dig_port)
+{
+ u8 rx_status;
+ int ret;
+
+ ret = intel_dp_hdcp2_read_rx_status(intel_dig_port, &rx_status);
+ if (ret)
+ return ret;
+
+ if (HDCP_2_2_DP_RXSTATUS_REAUTH_REQ(rx_status))
+ ret = HDCP_REAUTH_REQUEST;
+ else if (HDCP_2_2_DP_RXSTATUS_LINK_FAILED(rx_status))
+ ret = HDCP_LINK_INTEGRITY_FAILURE;
+ else if (HDCP_2_2_DP_RXSTATUS_READY(rx_status))
+ ret = HDCP_TOPOLOGY_CHANGE;
+
+ return ret;
+}
+
+static
+int intel_dp_hdcp2_capable(struct intel_digital_port *intel_dig_port,
+ bool *capable)
+{
+ u8 rx_caps[3];
+ int ret;
+
+ *capable = false;
+ ret = drm_dp_dpcd_read(&intel_dig_port->dp.aux,
+ DP_HDCP_2_2_REG_RX_CAPS_OFFSET,
+ rx_caps, HDCP_2_2_RXCAPS_LEN);
+ if (ret != HDCP_2_2_RXCAPS_LEN)
+ return ret >= 0 ? -EIO : ret;
+
+ if (rx_caps[0] == HDCP_2_2_RX_CAPS_VERSION_VAL &&
+ HDCP_2_2_DP_HDCP_CAPABLE(rx_caps[2]))
+ *capable = true;
+
+ return 0;
+}
+
static const struct intel_hdcp_shim intel_dp_hdcp_shim = {
.write_an_aksv = intel_dp_hdcp_write_an_aksv,
.read_bksv = intel_dp_hdcp_read_bksv,
.toggle_signalling = intel_dp_hdcp_toggle_signalling,
.check_link = intel_dp_hdcp_check_link,
.hdcp_capable = intel_dp_hdcp_capable,
+ .write_2_2_msg = intel_dp_hdcp2_write_msg,
+ .read_2_2_msg = intel_dp_hdcp2_read_msg,
+ .config_stream_type = intel_dp_hdcp2_config_stream_type,
+ .check_2_2_link = intel_dp_hdcp2_check_link,
+ .hdcp_2_2_capable = intel_dp_hdcp2_capable,
+ .protocol = HDCP_PROTOCOL_DP,
};
static void intel_edp_panel_vdd_sanitize(struct intel_dp *intel_dp)
intel_pps_readout_hw_state(struct intel_dp *intel_dp, struct edp_power_seq *seq)
{
struct drm_i915_private *dev_priv = dp_to_i915(intel_dp);
- u32 pp_on, pp_off, pp_div = 0, pp_ctl = 0;
+ u32 pp_on, pp_off, pp_ctl;
struct pps_registers regs;
intel_pps_get_registers(intel_dp, ®s);
- /* Workaround: Need to write PP_CONTROL with the unlock key as
- * the very first thing. */
pp_ctl = ironlake_get_pp_control(intel_dp);
+ /* Ensure PPS is unlocked */
+ if (!HAS_DDI(dev_priv))
+ I915_WRITE(regs.pp_ctrl, pp_ctl);
+
pp_on = I915_READ(regs.pp_on);
pp_off = I915_READ(regs.pp_off);
- if (!IS_GEN9_LP(dev_priv) && !HAS_PCH_CNP(dev_priv) &&
- !HAS_PCH_ICP(dev_priv)) {
- I915_WRITE(regs.pp_ctrl, pp_ctl);
- pp_div = I915_READ(regs.pp_div);
- }
/* Pull timing values out of registers */
- seq->t1_t3 = (pp_on & PANEL_POWER_UP_DELAY_MASK) >>
- PANEL_POWER_UP_DELAY_SHIFT;
+ seq->t1_t3 = REG_FIELD_GET(PANEL_POWER_UP_DELAY_MASK, pp_on);
+ seq->t8 = REG_FIELD_GET(PANEL_LIGHT_ON_DELAY_MASK, pp_on);
+ seq->t9 = REG_FIELD_GET(PANEL_LIGHT_OFF_DELAY_MASK, pp_off);
+ seq->t10 = REG_FIELD_GET(PANEL_POWER_DOWN_DELAY_MASK, pp_off);
- seq->t8 = (pp_on & PANEL_LIGHT_ON_DELAY_MASK) >>
- PANEL_LIGHT_ON_DELAY_SHIFT;
+ if (i915_mmio_reg_valid(regs.pp_div)) {
+ u32 pp_div;
- seq->t9 = (pp_off & PANEL_LIGHT_OFF_DELAY_MASK) >>
- PANEL_LIGHT_OFF_DELAY_SHIFT;
-
- seq->t10 = (pp_off & PANEL_POWER_DOWN_DELAY_MASK) >>
- PANEL_POWER_DOWN_DELAY_SHIFT;
+ pp_div = I915_READ(regs.pp_div);
- if (IS_GEN9_LP(dev_priv) || HAS_PCH_CNP(dev_priv) ||
- HAS_PCH_ICP(dev_priv)) {
- seq->t11_t12 = ((pp_ctl & BXT_POWER_CYCLE_DELAY_MASK) >>
- BXT_POWER_CYCLE_DELAY_SHIFT) * 1000;
+ seq->t11_t12 = REG_FIELD_GET(PANEL_POWER_CYCLE_DELAY_MASK, pp_div) * 1000;
} else {
- seq->t11_t12 = ((pp_div & PANEL_POWER_CYCLE_DELAY_MASK) >>
- PANEL_POWER_CYCLE_DELAY_SHIFT) * 1000;
+ seq->t11_t12 = REG_FIELD_GET(BXT_POWER_CYCLE_DELAY_MASK, pp_ctl) * 1000;
}
}
bool force_disable_vdd)
{
struct drm_i915_private *dev_priv = dp_to_i915(intel_dp);
- u32 pp_on, pp_off, pp_div, port_sel = 0;
+ u32 pp_on, pp_off, port_sel = 0;
int div = dev_priv->rawclk_freq / 1000;
struct pps_registers regs;
enum port port = dp_to_dig_port(intel_dp)->base.port;
I915_WRITE(regs.pp_ctrl, pp);
}
- pp_on = (seq->t1_t3 << PANEL_POWER_UP_DELAY_SHIFT) |
- (seq->t8 << PANEL_LIGHT_ON_DELAY_SHIFT);
- pp_off = (seq->t9 << PANEL_LIGHT_OFF_DELAY_SHIFT) |
- (seq->t10 << PANEL_POWER_DOWN_DELAY_SHIFT);
- /* Compute the divisor for the pp clock, simply match the Bspec
- * formula. */
- if (IS_GEN9_LP(dev_priv) || HAS_PCH_CNP(dev_priv) ||
- HAS_PCH_ICP(dev_priv)) {
- pp_div = I915_READ(regs.pp_ctrl);
- pp_div &= ~BXT_POWER_CYCLE_DELAY_MASK;
- pp_div |= (DIV_ROUND_UP(seq->t11_t12, 1000)
- << BXT_POWER_CYCLE_DELAY_SHIFT);
- } else {
- pp_div = ((100 * div)/2 - 1) << PP_REFERENCE_DIVIDER_SHIFT;
- pp_div |= (DIV_ROUND_UP(seq->t11_t12, 1000)
- << PANEL_POWER_CYCLE_DELAY_SHIFT);
- }
+ pp_on = REG_FIELD_PREP(PANEL_POWER_UP_DELAY_MASK, seq->t1_t3) |
+ REG_FIELD_PREP(PANEL_LIGHT_ON_DELAY_MASK, seq->t8);
+ pp_off = REG_FIELD_PREP(PANEL_LIGHT_OFF_DELAY_MASK, seq->t9) |
+ REG_FIELD_PREP(PANEL_POWER_DOWN_DELAY_MASK, seq->t10);
/* Haswell doesn't have any port selection bits for the panel
* power sequencer any more. */
I915_WRITE(regs.pp_on, pp_on);
I915_WRITE(regs.pp_off, pp_off);
- if (IS_GEN9_LP(dev_priv) || HAS_PCH_CNP(dev_priv) ||
- HAS_PCH_ICP(dev_priv))
- I915_WRITE(regs.pp_ctrl, pp_div);
- else
- I915_WRITE(regs.pp_div, pp_div);
+
+ /*
+ * Compute the divisor for the pp clock, simply match the Bspec formula.
+ */
+ if (i915_mmio_reg_valid(regs.pp_div)) {
+ I915_WRITE(regs.pp_div,
+ REG_FIELD_PREP(PP_REFERENCE_DIVIDER_MASK, (100 * div) / 2 - 1) |
+ REG_FIELD_PREP(PANEL_POWER_CYCLE_DELAY_MASK, DIV_ROUND_UP(seq->t11_t12, 1000)));
+ } else {
+ u32 pp_ctl;
+
+ pp_ctl = I915_READ(regs.pp_ctrl);
+ pp_ctl &= ~BXT_POWER_CYCLE_DELAY_MASK;
+ pp_ctl |= REG_FIELD_PREP(BXT_POWER_CYCLE_DELAY_MASK, DIV_ROUND_UP(seq->t11_t12, 1000));
+ I915_WRITE(regs.pp_ctrl, pp_ctl);
+ }
DRM_DEBUG_KMS("panel power sequencer register settings: PP_ON %#x, PP_OFF %#x, PP_DIV %#x\n",
I915_READ(regs.pp_on),
I915_READ(regs.pp_off),
- (IS_GEN9_LP(dev_priv) || HAS_PCH_CNP(dev_priv) ||
- HAS_PCH_ICP(dev_priv)) ?
- (I915_READ(regs.pp_ctrl) & BXT_POWER_CYCLE_DELAY_MASK) :
- I915_READ(regs.pp_div));
+ i915_mmio_reg_valid(regs.pp_div) ?
+ I915_READ(regs.pp_div) :
+ (I915_READ(regs.pp_ctrl) & BXT_POWER_CYCLE_DELAY_MASK));
}
static void intel_dp_pps_init(struct intel_dp *intel_dp)
return NULL;
}
- downclock_mode = intel_find_panel_downclock(dev_priv, fixed_mode,
- &connector->base);
-
+ downclock_mode = intel_panel_edid_downclock_mode(connector, fixed_mode);
if (!downclock_mode) {
DRM_DEBUG_KMS("Downclock mode is not found. DRRS not supported\n");
return NULL;
struct drm_display_mode *fixed_mode = NULL;
struct drm_display_mode *downclock_mode = NULL;
bool has_dpcd;
- struct drm_display_mode *scan;
enum pipe pipe = INVALID_PIPE;
intel_wakeref_t wakeref;
struct edid *edid;
* eDP and LVDS bail out early in this case to prevent interfering
* with an already powered-on LVDS power sequencer.
*/
- if (intel_get_lvds_encoder(&dev_priv->drm)) {
+ if (intel_get_lvds_encoder(dev_priv)) {
WARN_ON(!(HAS_PCH_IBX(dev_priv) || HAS_PCH_CPT(dev_priv)));
DRM_INFO("LVDS was detected, not registering eDP\n");
}
intel_connector->edid = edid;
- /* prefer fixed mode from EDID if available */
- list_for_each_entry(scan, &connector->probed_modes, head) {
- if ((scan->type & DRM_MODE_TYPE_PREFERRED)) {
- fixed_mode = drm_mode_duplicate(dev, scan);
- downclock_mode = intel_dp_drrs_init(
- intel_connector, fixed_mode);
- break;
- }
- }
+ fixed_mode = intel_panel_edid_fixed_mode(intel_connector);
+ if (fixed_mode)
+ downclock_mode = intel_dp_drrs_init(intel_connector, fixed_mode);
/* fallback to VBT if available for eDP */
- if (!fixed_mode && dev_priv->vbt.lfp_lvds_vbt_mode) {
- fixed_mode = drm_mode_duplicate(dev,
- dev_priv->vbt.lfp_lvds_vbt_mode);
- if (fixed_mode) {
- fixed_mode->type |= DRM_MODE_TYPE_PREFERRED;
- connector->display_info.width_mm = fixed_mode->width_mm;
- connector->display_info.height_mm = fixed_mode->height_mm;
- }
- }
+ if (!fixed_mode)
+ fixed_mode = intel_panel_vbt_fixed_mode(intel_connector);
mutex_unlock(&dev->mode_config.mutex);
if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) {
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_DP_H__
+#define __INTEL_DP_H__
+
+#include <linux/types.h>
+
+#include <drm/i915_drm.h>
+
+#include "i915_reg.h"
+
+enum pipe;
+struct drm_connector_state;
+struct drm_encoder;
+struct drm_i915_private;
+struct drm_modeset_acquire_ctx;
+struct intel_connector;
+struct intel_crtc_state;
+struct intel_digital_port;
+struct intel_dp;
+struct intel_encoder;
+
+struct link_config_limits {
+ int min_clock, max_clock;
+ int min_lane_count, max_lane_count;
+ int min_bpp, max_bpp;
+};
+
+void intel_dp_adjust_compliance_config(struct intel_dp *intel_dp,
+ struct intel_crtc_state *pipe_config,
+ struct link_config_limits *limits);
+bool intel_dp_limited_color_range(const struct intel_crtc_state *crtc_state,
+ const struct drm_connector_state *conn_state);
+int intel_dp_min_bpp(const struct intel_crtc_state *crtc_state);
+bool intel_dp_port_enabled(struct drm_i915_private *dev_priv,
+ i915_reg_t dp_reg, enum port port,
+ enum pipe *pipe);
+bool intel_dp_init(struct drm_i915_private *dev_priv, i915_reg_t output_reg,
+ enum port port);
+bool intel_dp_init_connector(struct intel_digital_port *intel_dig_port,
+ struct intel_connector *intel_connector);
+void intel_dp_set_link_params(struct intel_dp *intel_dp,
+ int link_rate, u8 lane_count,
+ bool link_mst);
+int intel_dp_get_link_train_fallback_values(struct intel_dp *intel_dp,
+ int link_rate, u8 lane_count);
+int intel_dp_retrain_link(struct intel_encoder *encoder,
+ struct drm_modeset_acquire_ctx *ctx);
+void intel_dp_sink_dpms(struct intel_dp *intel_dp, int mode);
+void intel_dp_sink_set_decompression_state(struct intel_dp *intel_dp,
+ const struct intel_crtc_state *crtc_state,
+ bool enable);
+void intel_dp_encoder_reset(struct drm_encoder *encoder);
+void intel_dp_encoder_suspend(struct intel_encoder *intel_encoder);
+void intel_dp_encoder_flush_work(struct drm_encoder *encoder);
+int intel_dp_compute_config(struct intel_encoder *encoder,
+ struct intel_crtc_state *pipe_config,
+ struct drm_connector_state *conn_state);
+bool intel_dp_is_edp(struct intel_dp *intel_dp);
+bool intel_dp_is_port_edp(struct drm_i915_private *dev_priv, enum port port);
+enum irqreturn intel_dp_hpd_pulse(struct intel_digital_port *intel_dig_port,
+ bool long_hpd);
+void intel_edp_backlight_on(const struct intel_crtc_state *crtc_state,
+ const struct drm_connector_state *conn_state);
+void intel_edp_backlight_off(const struct drm_connector_state *conn_state);
+void intel_edp_panel_vdd_on(struct intel_dp *intel_dp);
+void intel_edp_panel_on(struct intel_dp *intel_dp);
+void intel_edp_panel_off(struct intel_dp *intel_dp);
+void intel_dp_mst_suspend(struct drm_i915_private *dev_priv);
+void intel_dp_mst_resume(struct drm_i915_private *dev_priv);
+int intel_dp_max_link_rate(struct intel_dp *intel_dp);
+int intel_dp_max_lane_count(struct intel_dp *intel_dp);
+int intel_dp_rate_select(struct intel_dp *intel_dp, int rate);
+void intel_power_sequencer_reset(struct drm_i915_private *dev_priv);
+u32 intel_dp_pack_aux(const u8 *src, int src_bytes);
+
+void intel_edp_drrs_enable(struct intel_dp *intel_dp,
+ const struct intel_crtc_state *crtc_state);
+void intel_edp_drrs_disable(struct intel_dp *intel_dp,
+ const struct intel_crtc_state *crtc_state);
+void intel_edp_drrs_invalidate(struct drm_i915_private *dev_priv,
+ unsigned int frontbuffer_bits);
+void intel_edp_drrs_flush(struct drm_i915_private *dev_priv,
+ unsigned int frontbuffer_bits);
+
+void
+intel_dp_program_link_training_pattern(struct intel_dp *intel_dp,
+ u8 dp_train_pat);
+void
+intel_dp_set_signal_levels(struct intel_dp *intel_dp);
+void intel_dp_set_idle_link_train(struct intel_dp *intel_dp);
+u8
+intel_dp_voltage_max(struct intel_dp *intel_dp);
+u8
+intel_dp_pre_emphasis_max(struct intel_dp *intel_dp, u8 voltage_swing);
+void intel_dp_compute_rate(struct intel_dp *intel_dp, int port_clock,
+ u8 *link_bw, u8 *rate_select);
+bool intel_dp_source_supports_hbr2(struct intel_dp *intel_dp);
+bool intel_dp_source_supports_hbr3(struct intel_dp *intel_dp);
+bool
+intel_dp_get_link_status(struct intel_dp *intel_dp, u8 *link_status);
+u16 intel_dp_dsc_get_output_bpp(int link_clock, u8 lane_count,
+ int mode_clock, int mode_hdisplay);
+u8 intel_dp_dsc_get_slice_count(struct intel_dp *intel_dp, int mode_clock,
+ int mode_hdisplay);
+
+bool intel_dp_read_dpcd(struct intel_dp *intel_dp);
+int intel_dp_link_required(int pixel_clock, int bpp);
+int intel_dp_max_data_rate(int max_link_clock, int max_lanes);
+bool intel_digital_port_connected(struct intel_encoder *encoder);
+void icl_tc_phy_disconnect(struct drm_i915_private *dev_priv,
+ struct intel_digital_port *dig_port);
+
+static inline unsigned int intel_dp_unused_lane_mask(int lane_count)
+{
+ return ~((1 << lane_count) - 1) & 0xf;
+}
+
+#endif /* __INTEL_DP_H__ */
* IN THE SOFTWARE.
*/
+#include "intel_dp.h"
#include "intel_drv.h"
static void
*
*/
-#include "i915_drv.h"
-#include "intel_drv.h"
#include <drm/drm_atomic_helper.h>
#include <drm/drm_edid.h>
#include <drm/drm_probe_helper.h>
+#include "i915_drv.h"
+#include "intel_audio.h"
+#include "intel_connector.h"
+#include "intel_ddi.h"
+#include "intel_dp.h"
+#include "intel_drv.h"
+
+static int intel_dp_mst_compute_link_config(struct intel_encoder *encoder,
+ struct intel_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state,
+ struct link_config_limits *limits)
+{
+ struct drm_atomic_state *state = crtc_state->base.state;
+ struct intel_dp_mst_encoder *intel_mst = enc_to_mst(&encoder->base);
+ struct intel_dp *intel_dp = &intel_mst->primary->dp;
+ struct intel_connector *connector =
+ to_intel_connector(conn_state->connector);
+ const struct drm_display_mode *adjusted_mode =
+ &crtc_state->base.adjusted_mode;
+ void *port = connector->port;
+ bool constant_n = drm_dp_has_quirk(&intel_dp->desc,
+ DP_DPCD_QUIRK_CONSTANT_N);
+ int bpp, slots = -EINVAL;
+
+ crtc_state->lane_count = limits->max_lane_count;
+ crtc_state->port_clock = limits->max_clock;
+
+ for (bpp = limits->max_bpp; bpp >= limits->min_bpp; bpp -= 2 * 3) {
+ crtc_state->pipe_bpp = bpp;
+
+ crtc_state->pbn = drm_dp_calc_pbn_mode(adjusted_mode->crtc_clock,
+ crtc_state->pipe_bpp);
+
+ slots = drm_dp_atomic_find_vcpi_slots(state, &intel_dp->mst_mgr,
+ port, crtc_state->pbn);
+ if (slots == -EDEADLK)
+ return slots;
+ if (slots >= 0)
+ break;
+ }
+
+ if (slots < 0) {
+ DRM_DEBUG_KMS("failed finding vcpi slots:%d\n", slots);
+ return slots;
+ }
+
+ intel_link_compute_m_n(crtc_state->pipe_bpp,
+ crtc_state->lane_count,
+ adjusted_mode->crtc_clock,
+ crtc_state->port_clock,
+ &crtc_state->dp_m_n,
+ constant_n);
+ crtc_state->dp_m_n.tu = slots;
+
+ return 0;
+}
+
static int intel_dp_mst_compute_config(struct intel_encoder *encoder,
struct intel_crtc_state *pipe_config,
struct drm_connector_state *conn_state)
{
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
struct intel_dp_mst_encoder *intel_mst = enc_to_mst(&encoder->base);
- struct intel_digital_port *intel_dig_port = intel_mst->primary;
- struct intel_dp *intel_dp = &intel_dig_port->dp;
- struct drm_connector *connector = conn_state->connector;
- void *port = to_intel_connector(connector)->port;
- struct drm_atomic_state *state = pipe_config->base.state;
- struct drm_crtc *crtc = pipe_config->base.crtc;
- struct drm_crtc_state *old_crtc_state =
- drm_atomic_get_old_crtc_state(state, crtc);
- int bpp;
- int lane_count, slots =
- to_intel_crtc_state(old_crtc_state)->dp_m_n.tu;
- const struct drm_display_mode *adjusted_mode = &pipe_config->base.adjusted_mode;
- int mst_pbn;
- bool constant_n = drm_dp_has_quirk(&intel_dp->desc,
- DP_DPCD_QUIRK_CONSTANT_N);
+ struct intel_dp *intel_dp = &intel_mst->primary->dp;
+ struct intel_connector *connector =
+ to_intel_connector(conn_state->connector);
+ struct intel_digital_connector_state *intel_conn_state =
+ to_intel_digital_connector_state(conn_state);
+ const struct drm_display_mode *adjusted_mode =
+ &pipe_config->base.adjusted_mode;
+ void *port = connector->port;
+ struct link_config_limits limits;
+ int ret;
if (adjusted_mode->flags & DRM_MODE_FLAG_DBLSCAN)
return -EINVAL;
pipe_config->output_format = INTEL_OUTPUT_FORMAT_RGB;
pipe_config->has_pch_encoder = false;
- bpp = 24;
- if (intel_dp->compliance.test_data.bpc) {
- bpp = intel_dp->compliance.test_data.bpc * 3;
- DRM_DEBUG_KMS("Setting pipe bpp to %d\n",
- bpp);
- }
+
+ if (intel_conn_state->force_audio == HDMI_AUDIO_AUTO)
+ pipe_config->has_audio =
+ drm_dp_mst_port_has_audio(&intel_dp->mst_mgr, port);
+ else
+ pipe_config->has_audio =
+ intel_conn_state->force_audio == HDMI_AUDIO_ON;
+
/*
* for MST we always configure max link bw - the spec doesn't
* seem to suggest we should do otherwise.
*/
- lane_count = intel_dp_max_lane_count(intel_dp);
-
- pipe_config->lane_count = lane_count;
-
- pipe_config->pipe_bpp = bpp;
+ limits.min_clock =
+ limits.max_clock = intel_dp_max_link_rate(intel_dp);
- pipe_config->port_clock = intel_dp_max_link_rate(intel_dp);
+ limits.min_lane_count =
+ limits.max_lane_count = intel_dp_max_lane_count(intel_dp);
- if (drm_dp_mst_port_has_audio(&intel_dp->mst_mgr, port))
- pipe_config->has_audio = true;
+ limits.min_bpp = intel_dp_min_bpp(pipe_config);
+ limits.max_bpp = pipe_config->pipe_bpp;
- mst_pbn = drm_dp_calc_pbn_mode(adjusted_mode->crtc_clock, bpp);
- pipe_config->pbn = mst_pbn;
+ intel_dp_adjust_compliance_config(intel_dp, pipe_config, &limits);
- slots = drm_dp_atomic_find_vcpi_slots(state, &intel_dp->mst_mgr, port,
- mst_pbn);
- if (slots < 0) {
- DRM_DEBUG_KMS("failed finding vcpi slots:%d\n",
- slots);
- return slots;
- }
-
- intel_link_compute_m_n(bpp, lane_count,
- adjusted_mode->crtc_clock,
- pipe_config->port_clock,
- &pipe_config->dp_m_n,
- constant_n);
+ ret = intel_dp_mst_compute_link_config(encoder, pipe_config,
+ conn_state, &limits);
+ if (ret)
+ return ret;
- pipe_config->dp_m_n.tu = slots;
+ pipe_config->limited_color_range =
+ intel_dp_limited_color_range(pipe_config, conn_state);
if (IS_GEN9_LP(dev_priv))
pipe_config->lane_lat_optim_mask =
struct drm_crtc *new_crtc = new_conn_state->crtc;
struct drm_crtc_state *crtc_state;
struct drm_dp_mst_topology_mgr *mgr;
- int ret = 0;
+ int ret;
+
+ ret = intel_digital_connector_atomic_check(connector, new_conn_state);
+ if (ret)
+ return ret;
if (!old_conn_state->crtc)
return 0;
DRM_DEBUG_KMS("active links %d\n", intel_dp->active_mst_links);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
DP_TP_STATUS(port),
DP_TP_STATUS_ACT_SENT,
DP_TP_STATUS_ACT_SENT,
static const struct drm_connector_funcs intel_dp_mst_connector_funcs = {
.detect = intel_dp_mst_detect,
.fill_modes = drm_helper_probe_single_connector_modes,
+ .atomic_get_property = intel_digital_connector_atomic_get_property,
+ .atomic_set_property = intel_digital_connector_atomic_set_property,
.late_register = intel_connector_register,
.early_unregister = intel_connector_unregister,
.destroy = intel_connector_destroy,
.atomic_destroy_state = drm_atomic_helper_connector_destroy_state,
- .atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state,
+ .atomic_duplicate_state = intel_digital_connector_duplicate_state,
};
static int intel_dp_mst_get_modes(struct drm_connector *connector)
struct intel_connector *intel_connector = to_intel_connector(connector);
struct intel_dp *intel_dp = intel_connector->mst_port;
int max_dotclk = to_i915(connector->dev)->max_dotclk_freq;
- int bpp = 24; /* MST uses fixed bpp */
int max_rate, mode_rate, max_lanes, max_link_clock;
if (drm_connector_is_unregistered(connector))
max_lanes = intel_dp_max_lane_count(intel_dp);
max_rate = intel_dp_max_data_rate(max_link_clock, max_lanes);
- mode_rate = intel_dp_link_required(mode->clock, bpp);
+ mode_rate = intel_dp_link_required(mode->clock, 18);
/* TODO - validate mode against available PBN for link */
if (mode->clock < 10000)
if (ret)
goto err;
+ intel_attach_force_audio_property(connector);
+ intel_attach_broadcast_rgb_property(connector);
+ drm_connector_attach_max_bpc_property(connector, 6, 12);
+
return connector;
err:
* DEALINGS IN THE SOFTWARE.
*/
+#include "intel_dp.h"
#include "intel_drv.h"
/**
static void bxt_phy_wait_grc_done(struct drm_i915_private *dev_priv,
enum dpio_phy phy)
{
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
BXT_PORT_REF_DW3(phy),
GRC_DONE, GRC_DONE,
10))
* The flag should get set in 100us according to the HW team, but
* use 1ms due to occasional timeouts observed with that.
*/
- if (intel_wait_for_register_fw(dev_priv, BXT_PORT_CL1CM_DW0(phy),
+ if (intel_wait_for_register_fw(&dev_priv->uncore,
+ BXT_PORT_CL1CM_DW0(phy),
PHY_RESERVED | PHY_POWER_GOOD,
PHY_POWER_GOOD,
1))
}
static struct intel_shared_dpll *
-intel_find_shared_dpll(struct intel_crtc *crtc,
- struct intel_crtc_state *crtc_state,
+intel_find_shared_dpll(struct intel_crtc_state *crtc_state,
enum intel_dpll_id range_min,
enum intel_dpll_id range_max)
{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
struct intel_shared_dpll *pll, *unused_pll = NULL;
struct intel_shared_dpll_state *shared_dpll;
}
static struct intel_shared_dpll *
-ibx_get_dpll(struct intel_crtc *crtc, struct intel_crtc_state *crtc_state,
+ibx_get_dpll(struct intel_crtc_state *crtc_state,
struct intel_encoder *encoder)
{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
struct intel_shared_dpll *pll;
enum intel_dpll_id i;
crtc->base.base.id, crtc->base.name,
pll->info->name);
} else {
- pll = intel_find_shared_dpll(crtc, crtc_state,
+ pll = intel_find_shared_dpll(crtc_state,
DPLL_ID_PCH_PLL_A,
DPLL_ID_PCH_PLL_B);
}
*r2_out = best.r2;
}
-static struct intel_shared_dpll *hsw_ddi_hdmi_get_dpll(int clock,
- struct intel_crtc *crtc,
- struct intel_crtc_state *crtc_state)
+static struct intel_shared_dpll *hsw_ddi_hdmi_get_dpll(struct intel_crtc_state *crtc_state)
{
struct intel_shared_dpll *pll;
u32 val;
unsigned int p, n2, r2;
- hsw_ddi_calculate_wrpll(clock * 1000, &r2, &n2, &p);
+ hsw_ddi_calculate_wrpll(crtc_state->port_clock * 1000, &r2, &n2, &p);
val = WRPLL_PLL_ENABLE | WRPLL_PLL_LCPLL |
WRPLL_DIVIDER_REFERENCE(r2) | WRPLL_DIVIDER_FEEDBACK(n2) |
crtc_state->dpll_hw_state.wrpll = val;
- pll = intel_find_shared_dpll(crtc, crtc_state,
+ pll = intel_find_shared_dpll(crtc_state,
DPLL_ID_WRPLL1, DPLL_ID_WRPLL2);
if (!pll)
}
static struct intel_shared_dpll *
-hsw_ddi_dp_get_dpll(struct intel_encoder *encoder, int clock)
+hsw_ddi_dp_get_dpll(struct intel_crtc_state *crtc_state)
{
- struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
struct intel_shared_dpll *pll;
enum intel_dpll_id pll_id;
+ int clock = crtc_state->port_clock;
switch (clock / 2) {
case 81000:
}
static struct intel_shared_dpll *
-hsw_get_dpll(struct intel_crtc *crtc, struct intel_crtc_state *crtc_state,
+hsw_get_dpll(struct intel_crtc_state *crtc_state,
struct intel_encoder *encoder)
{
struct intel_shared_dpll *pll;
- int clock = crtc_state->port_clock;
memset(&crtc_state->dpll_hw_state, 0,
sizeof(crtc_state->dpll_hw_state));
if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_HDMI)) {
- pll = hsw_ddi_hdmi_get_dpll(clock, crtc, crtc_state);
+ pll = hsw_ddi_hdmi_get_dpll(crtc_state);
} else if (intel_crtc_has_dp_encoder(crtc_state)) {
- pll = hsw_ddi_dp_get_dpll(encoder, clock);
+ pll = hsw_ddi_dp_get_dpll(crtc_state);
} else if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_ANALOG)) {
if (WARN_ON(crtc_state->port_clock / 2 != 135000))
return NULL;
crtc_state->dpll_hw_state.spll =
SPLL_PLL_ENABLE | SPLL_PLL_FREQ_1350MHz | SPLL_PLL_SSC;
- pll = intel_find_shared_dpll(crtc, crtc_state,
+ pll = intel_find_shared_dpll(crtc_state,
DPLL_ID_SPLL, DPLL_ID_SPLL);
} else {
return NULL;
I915_WRITE(regs[id].ctl,
I915_READ(regs[id].ctl) | LCPLL_PLL_ENABLE);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
DPLL_STATUS,
DPLL_LOCK(id),
DPLL_LOCK(id),
return true;
}
-static bool skl_ddi_hdmi_pll_dividers(struct intel_crtc *crtc,
- struct intel_crtc_state *crtc_state,
- int clock)
+static bool skl_ddi_hdmi_pll_dividers(struct intel_crtc_state *crtc_state)
{
u32 ctrl1, cfgcr1, cfgcr2;
struct skl_wrpll_params wrpll_params = { 0, };
ctrl1 |= DPLL_CTRL1_HDMI_MODE(0);
- if (!skl_ddi_calculate_wrpll(clock * 1000, &wrpll_params))
+ if (!skl_ddi_calculate_wrpll(crtc_state->port_clock * 1000,
+ &wrpll_params))
return false;
cfgcr1 = DPLL_CFGCR1_FREQ_ENABLE |
}
static bool
-skl_ddi_dp_set_dpll_hw_state(int clock,
- struct intel_dpll_hw_state *dpll_hw_state)
+skl_ddi_dp_set_dpll_hw_state(struct intel_crtc_state *crtc_state)
{
u32 ctrl1;
* as the DPLL id in this function.
*/
ctrl1 = DPLL_CTRL1_OVERRIDE(0);
- switch (clock / 2) {
+ switch (crtc_state->port_clock / 2) {
case 81000:
ctrl1 |= DPLL_CTRL1_LINK_RATE(DPLL_CTRL1_LINK_RATE_810, 0);
break;
break;
}
- dpll_hw_state->ctrl1 = ctrl1;
+ memset(&crtc_state->dpll_hw_state, 0,
+ sizeof(crtc_state->dpll_hw_state));
+
+ crtc_state->dpll_hw_state.ctrl1 = ctrl1;
+
return true;
}
static struct intel_shared_dpll *
-skl_get_dpll(struct intel_crtc *crtc, struct intel_crtc_state *crtc_state,
+skl_get_dpll(struct intel_crtc_state *crtc_state,
struct intel_encoder *encoder)
{
struct intel_shared_dpll *pll;
- int clock = crtc_state->port_clock;
bool bret;
- struct intel_dpll_hw_state dpll_hw_state;
-
- memset(&dpll_hw_state, 0, sizeof(dpll_hw_state));
if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_HDMI)) {
- bret = skl_ddi_hdmi_pll_dividers(crtc, crtc_state, clock);
+ bret = skl_ddi_hdmi_pll_dividers(crtc_state);
if (!bret) {
DRM_DEBUG_KMS("Could not get HDMI pll dividers.\n");
return NULL;
}
} else if (intel_crtc_has_dp_encoder(crtc_state)) {
- bret = skl_ddi_dp_set_dpll_hw_state(clock, &dpll_hw_state);
+ bret = skl_ddi_dp_set_dpll_hw_state(crtc_state);
if (!bret) {
DRM_DEBUG_KMS("Could not set DP dpll HW state.\n");
return NULL;
}
- crtc_state->dpll_hw_state = dpll_hw_state;
} else {
return NULL;
}
if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_EDP))
- pll = intel_find_shared_dpll(crtc, crtc_state,
+ pll = intel_find_shared_dpll(crtc_state,
DPLL_ID_SKL_DPLL0,
DPLL_ID_SKL_DPLL0);
else
- pll = intel_find_shared_dpll(crtc, crtc_state,
+ pll = intel_find_shared_dpll(crtc_state,
DPLL_ID_SKL_DPLL1,
DPLL_ID_SKL_DPLL3);
if (!pll)
};
static bool
-bxt_ddi_hdmi_pll_dividers(struct intel_crtc *intel_crtc,
- struct intel_crtc_state *crtc_state, int clock,
+bxt_ddi_hdmi_pll_dividers(struct intel_crtc_state *crtc_state,
struct bxt_clk_div *clk_div)
{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
struct dpll best_clock;
/* Calculate HDMI div */
* FIXME: tie the following calculation into
* i9xx_crtc_compute_clock
*/
- if (!bxt_find_best_dpll(crtc_state, clock, &best_clock)) {
+ if (!bxt_find_best_dpll(crtc_state, &best_clock)) {
DRM_DEBUG_DRIVER("no PLL dividers found for clock %d pipe %c\n",
- clock, pipe_name(intel_crtc->pipe));
+ crtc_state->port_clock,
+ pipe_name(crtc->pipe));
return false;
}
return true;
}
-static void bxt_ddi_dp_pll_dividers(int clock, struct bxt_clk_div *clk_div)
+static void bxt_ddi_dp_pll_dividers(struct intel_crtc_state *crtc_state,
+ struct bxt_clk_div *clk_div)
{
+ int clock = crtc_state->port_clock;
int i;
*clk_div = bxt_dp_clk_val[0];
clk_div->vco = clock * 10 / 2 * clk_div->p1 * clk_div->p2;
}
-static bool bxt_ddi_set_dpll_hw_state(int clock,
- struct bxt_clk_div *clk_div,
- struct intel_dpll_hw_state *dpll_hw_state)
+static bool bxt_ddi_set_dpll_hw_state(struct intel_crtc_state *crtc_state,
+ const struct bxt_clk_div *clk_div)
{
+ struct intel_dpll_hw_state *dpll_hw_state = &crtc_state->dpll_hw_state;
+ int clock = crtc_state->port_clock;
int vco = clk_div->vco;
u32 prop_coef, int_coef, gain_ctl, targ_cnt;
u32 lanestagger;
+ memset(dpll_hw_state, 0, sizeof(*dpll_hw_state));
+
if (vco >= 6200000 && vco <= 6700000) {
prop_coef = 4;
int_coef = 9;
}
static bool
-bxt_ddi_dp_set_dpll_hw_state(int clock,
- struct intel_dpll_hw_state *dpll_hw_state)
+bxt_ddi_dp_set_dpll_hw_state(struct intel_crtc_state *crtc_state)
{
- struct bxt_clk_div clk_div = {0};
+ struct bxt_clk_div clk_div = {};
- bxt_ddi_dp_pll_dividers(clock, &clk_div);
+ bxt_ddi_dp_pll_dividers(crtc_state, &clk_div);
- return bxt_ddi_set_dpll_hw_state(clock, &clk_div, dpll_hw_state);
+ return bxt_ddi_set_dpll_hw_state(crtc_state, &clk_div);
}
static bool
-bxt_ddi_hdmi_set_dpll_hw_state(struct intel_crtc *intel_crtc,
- struct intel_crtc_state *crtc_state, int clock,
- struct intel_dpll_hw_state *dpll_hw_state)
+bxt_ddi_hdmi_set_dpll_hw_state(struct intel_crtc_state *crtc_state)
{
- struct bxt_clk_div clk_div = { };
+ struct bxt_clk_div clk_div = {};
- bxt_ddi_hdmi_pll_dividers(intel_crtc, crtc_state, clock, &clk_div);
+ bxt_ddi_hdmi_pll_dividers(crtc_state, &clk_div);
- return bxt_ddi_set_dpll_hw_state(clock, &clk_div, dpll_hw_state);
+ return bxt_ddi_set_dpll_hw_state(crtc_state, &clk_div);
}
static struct intel_shared_dpll *
-bxt_get_dpll(struct intel_crtc *crtc,
- struct intel_crtc_state *crtc_state,
- struct intel_encoder *encoder)
+bxt_get_dpll(struct intel_crtc_state *crtc_state,
+ struct intel_encoder *encoder)
{
- struct intel_dpll_hw_state dpll_hw_state = { };
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
struct intel_shared_dpll *pll;
- int i, clock = crtc_state->port_clock;
+ enum intel_dpll_id id;
if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_HDMI) &&
- !bxt_ddi_hdmi_set_dpll_hw_state(crtc, crtc_state, clock,
- &dpll_hw_state))
+ !bxt_ddi_hdmi_set_dpll_hw_state(crtc_state))
return NULL;
if (intel_crtc_has_dp_encoder(crtc_state) &&
- !bxt_ddi_dp_set_dpll_hw_state(clock, &dpll_hw_state))
+ !bxt_ddi_dp_set_dpll_hw_state(crtc_state))
return NULL;
- memset(&crtc_state->dpll_hw_state, 0,
- sizeof(crtc_state->dpll_hw_state));
-
- crtc_state->dpll_hw_state = dpll_hw_state;
-
/* 1:1 mapping between ports and PLLs */
- i = (enum intel_dpll_id) encoder->port;
- pll = intel_get_shared_dpll_by_id(dev_priv, i);
+ id = (enum intel_dpll_id) encoder->port;
+ pll = intel_get_shared_dpll_by_id(dev_priv, id);
DRM_DEBUG_KMS("[CRTC:%d:%s] using pre-allocated %s\n",
crtc->base.base.id, crtc->base.name, pll->info->name);
struct intel_dpll_mgr {
const struct dpll_info *dpll_info;
- struct intel_shared_dpll *(*get_dpll)(struct intel_crtc *crtc,
- struct intel_crtc_state *crtc_state,
+ struct intel_shared_dpll *(*get_dpll)(struct intel_crtc_state *crtc_state,
struct intel_encoder *encoder);
void (*dump_hw_state)(struct drm_i915_private *dev_priv,
I915_WRITE(CNL_DPLL_ENABLE(id), val);
/* 2. Wait for DPLL power state enabled in DPLL_ENABLE. */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
CNL_DPLL_ENABLE(id),
PLL_POWER_STATE,
PLL_POWER_STATE,
I915_WRITE(CNL_DPLL_ENABLE(id), val);
/* 7. Wait for PLL lock status in DPLL_ENABLE. */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
CNL_DPLL_ENABLE(id),
PLL_LOCK,
PLL_LOCK,
I915_WRITE(CNL_DPLL_ENABLE(id), val);
/* 4. Wait for PLL not locked status in DPLL_ENABLE. */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
CNL_DPLL_ENABLE(id),
PLL_LOCK,
0,
I915_WRITE(CNL_DPLL_ENABLE(id), val);
/* 7. Wait for DPLL power state disabled in DPLL_ENABLE. */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
CNL_DPLL_ENABLE(id),
PLL_POWER_STATE,
0,
}
static bool
-cnl_ddi_calculate_wrpll(int clock,
- struct drm_i915_private *dev_priv,
+cnl_ddi_calculate_wrpll(struct intel_crtc_state *crtc_state,
struct skl_wrpll_params *wrpll_params)
{
- u32 afe_clock = clock * 5;
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+ u32 afe_clock = crtc_state->port_clock * 5;
u32 ref_clock;
u32 dco_min = 7998000;
u32 dco_max = 10000000;
ref_clock = cnl_hdmi_pll_ref_clock(dev_priv);
- cnl_wrpll_params_populate(wrpll_params, best_dco, ref_clock, pdiv, qdiv,
- kdiv);
+ cnl_wrpll_params_populate(wrpll_params, best_dco, ref_clock,
+ pdiv, qdiv, kdiv);
return true;
}
-static bool cnl_ddi_hdmi_pll_dividers(struct intel_crtc *crtc,
- struct intel_crtc_state *crtc_state,
- int clock)
+static bool cnl_ddi_hdmi_pll_dividers(struct intel_crtc_state *crtc_state)
{
- struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
u32 cfgcr0, cfgcr1;
struct skl_wrpll_params wrpll_params = { 0, };
cfgcr0 = DPLL_CFGCR0_HDMI_MODE;
- if (!cnl_ddi_calculate_wrpll(clock, dev_priv, &wrpll_params))
+ if (!cnl_ddi_calculate_wrpll(crtc_state, &wrpll_params))
return false;
cfgcr0 |= DPLL_CFGCR0_DCO_FRACTION(wrpll_params.dco_fraction) |
}
static bool
-cnl_ddi_dp_set_dpll_hw_state(int clock,
- struct intel_dpll_hw_state *dpll_hw_state)
+cnl_ddi_dp_set_dpll_hw_state(struct intel_crtc_state *crtc_state)
{
u32 cfgcr0;
cfgcr0 = DPLL_CFGCR0_SSC_ENABLE;
- switch (clock / 2) {
+ switch (crtc_state->port_clock / 2) {
case 81000:
cfgcr0 |= DPLL_CFGCR0_LINK_RATE_810;
break;
break;
}
- dpll_hw_state->cfgcr0 = cfgcr0;
+ memset(&crtc_state->dpll_hw_state, 0,
+ sizeof(crtc_state->dpll_hw_state));
+
+ crtc_state->dpll_hw_state.cfgcr0 = cfgcr0;
+
return true;
}
static struct intel_shared_dpll *
-cnl_get_dpll(struct intel_crtc *crtc, struct intel_crtc_state *crtc_state,
+cnl_get_dpll(struct intel_crtc_state *crtc_state,
struct intel_encoder *encoder)
{
struct intel_shared_dpll *pll;
- int clock = crtc_state->port_clock;
bool bret;
- struct intel_dpll_hw_state dpll_hw_state;
-
- memset(&dpll_hw_state, 0, sizeof(dpll_hw_state));
if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_HDMI)) {
- bret = cnl_ddi_hdmi_pll_dividers(crtc, crtc_state, clock);
+ bret = cnl_ddi_hdmi_pll_dividers(crtc_state);
if (!bret) {
DRM_DEBUG_KMS("Could not get HDMI pll dividers.\n");
return NULL;
}
} else if (intel_crtc_has_dp_encoder(crtc_state)) {
- bret = cnl_ddi_dp_set_dpll_hw_state(clock, &dpll_hw_state);
+ bret = cnl_ddi_dp_set_dpll_hw_state(crtc_state);
if (!bret) {
DRM_DEBUG_KMS("Could not set DP dpll HW state.\n");
return NULL;
}
- crtc_state->dpll_hw_state = dpll_hw_state;
} else {
DRM_DEBUG_KMS("Skip DPLL setup for output_types 0x%x\n",
crtc_state->output_types);
return NULL;
}
- pll = intel_find_shared_dpll(crtc, crtc_state,
+ pll = intel_find_shared_dpll(crtc_state,
DPLL_ID_SKL_DPLL0,
DPLL_ID_SKL_DPLL2);
if (!pll) {
.dump_hw_state = cnl_dump_hw_state,
};
+struct icl_combo_pll_params {
+ int clock;
+ struct skl_wrpll_params wrpll;
+};
+
/*
* These values alrea already adjusted: they're the bits we write to the
* registers, not the logical values.
*/
-static const struct skl_wrpll_params icl_dp_combo_pll_24MHz_values[] = {
- { .dco_integer = 0x151, .dco_fraction = 0x4000, /* [0]: 5.4 */
- .pdiv = 0x2 /* 3 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x151, .dco_fraction = 0x4000, /* [1]: 2.7 */
- .pdiv = 0x2 /* 3 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x151, .dco_fraction = 0x4000, /* [2]: 1.62 */
- .pdiv = 0x4 /* 5 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x151, .dco_fraction = 0x4000, /* [3]: 3.24 */
- .pdiv = 0x4 /* 5 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x168, .dco_fraction = 0x0000, /* [4]: 2.16 */
- .pdiv = 0x1 /* 2 */, .kdiv = 2, .qdiv_mode = 1, .qdiv_ratio = 2},
- { .dco_integer = 0x168, .dco_fraction = 0x0000, /* [5]: 4.32 */
- .pdiv = 0x1 /* 2 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x195, .dco_fraction = 0x0000, /* [6]: 6.48 */
- .pdiv = 0x2 /* 3 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x151, .dco_fraction = 0x4000, /* [7]: 8.1 */
- .pdiv = 0x1 /* 2 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0},
+static const struct icl_combo_pll_params icl_dp_combo_pll_24MHz_values[] = {
+ { 540000,
+ { .dco_integer = 0x151, .dco_fraction = 0x4000, /* [0]: 5.4 */
+ .pdiv = 0x2 /* 3 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 270000,
+ { .dco_integer = 0x151, .dco_fraction = 0x4000, /* [1]: 2.7 */
+ .pdiv = 0x2 /* 3 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 162000,
+ { .dco_integer = 0x151, .dco_fraction = 0x4000, /* [2]: 1.62 */
+ .pdiv = 0x4 /* 5 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 324000,
+ { .dco_integer = 0x151, .dco_fraction = 0x4000, /* [3]: 3.24 */
+ .pdiv = 0x4 /* 5 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 216000,
+ { .dco_integer = 0x168, .dco_fraction = 0x0000, /* [4]: 2.16 */
+ .pdiv = 0x1 /* 2 */, .kdiv = 2, .qdiv_mode = 1, .qdiv_ratio = 2, }, },
+ { 432000,
+ { .dco_integer = 0x168, .dco_fraction = 0x0000, /* [5]: 4.32 */
+ .pdiv = 0x1 /* 2 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 648000,
+ { .dco_integer = 0x195, .dco_fraction = 0x0000, /* [6]: 6.48 */
+ .pdiv = 0x2 /* 3 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 810000,
+ { .dco_integer = 0x151, .dco_fraction = 0x4000, /* [7]: 8.1 */
+ .pdiv = 0x1 /* 2 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
};
+
/* Also used for 38.4 MHz values. */
-static const struct skl_wrpll_params icl_dp_combo_pll_19_2MHz_values[] = {
- { .dco_integer = 0x1A5, .dco_fraction = 0x7000, /* [0]: 5.4 */
- .pdiv = 0x2 /* 3 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x1A5, .dco_fraction = 0x7000, /* [1]: 2.7 */
- .pdiv = 0x2 /* 3 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x1A5, .dco_fraction = 0x7000, /* [2]: 1.62 */
- .pdiv = 0x4 /* 5 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x1A5, .dco_fraction = 0x7000, /* [3]: 3.24 */
- .pdiv = 0x4 /* 5 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x1C2, .dco_fraction = 0x0000, /* [4]: 2.16 */
- .pdiv = 0x1 /* 2 */, .kdiv = 2, .qdiv_mode = 1, .qdiv_ratio = 2},
- { .dco_integer = 0x1C2, .dco_fraction = 0x0000, /* [5]: 4.32 */
- .pdiv = 0x1 /* 2 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x1FA, .dco_fraction = 0x2000, /* [6]: 6.48 */
- .pdiv = 0x2 /* 3 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0},
- { .dco_integer = 0x1A5, .dco_fraction = 0x7000, /* [7]: 8.1 */
- .pdiv = 0x1 /* 2 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0},
+static const struct icl_combo_pll_params icl_dp_combo_pll_19_2MHz_values[] = {
+ { 540000,
+ { .dco_integer = 0x1A5, .dco_fraction = 0x7000, /* [0]: 5.4 */
+ .pdiv = 0x2 /* 3 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 270000,
+ { .dco_integer = 0x1A5, .dco_fraction = 0x7000, /* [1]: 2.7 */
+ .pdiv = 0x2 /* 3 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 162000,
+ { .dco_integer = 0x1A5, .dco_fraction = 0x7000, /* [2]: 1.62 */
+ .pdiv = 0x4 /* 5 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 324000,
+ { .dco_integer = 0x1A5, .dco_fraction = 0x7000, /* [3]: 3.24 */
+ .pdiv = 0x4 /* 5 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 216000,
+ { .dco_integer = 0x1C2, .dco_fraction = 0x0000, /* [4]: 2.16 */
+ .pdiv = 0x1 /* 2 */, .kdiv = 2, .qdiv_mode = 1, .qdiv_ratio = 2, }, },
+ { 432000,
+ { .dco_integer = 0x1C2, .dco_fraction = 0x0000, /* [5]: 4.32 */
+ .pdiv = 0x1 /* 2 */, .kdiv = 2, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 648000,
+ { .dco_integer = 0x1FA, .dco_fraction = 0x2000, /* [6]: 6.48 */
+ .pdiv = 0x2 /* 3 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
+ { 810000,
+ { .dco_integer = 0x1A5, .dco_fraction = 0x7000, /* [7]: 8.1 */
+ .pdiv = 0x1 /* 2 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0, }, },
};
static const struct skl_wrpll_params icl_tbt_pll_24MHz_values = {
.pdiv = 0x4 /* 5 */, .kdiv = 1, .qdiv_mode = 0, .qdiv_ratio = 0,
};
-static bool icl_calc_dp_combo_pll(struct drm_i915_private *dev_priv, int clock,
+static bool icl_calc_dp_combo_pll(struct intel_crtc_state *crtc_state,
struct skl_wrpll_params *pll_params)
{
- const struct skl_wrpll_params *params;
-
- params = dev_priv->cdclk.hw.ref == 24000 ?
- icl_dp_combo_pll_24MHz_values :
- icl_dp_combo_pll_19_2MHz_values;
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+ const struct icl_combo_pll_params *params =
+ dev_priv->cdclk.hw.ref == 24000 ?
+ icl_dp_combo_pll_24MHz_values :
+ icl_dp_combo_pll_19_2MHz_values;
+ int clock = crtc_state->port_clock;
+ int i;
- switch (clock) {
- case 540000:
- *pll_params = params[0];
- break;
- case 270000:
- *pll_params = params[1];
- break;
- case 162000:
- *pll_params = params[2];
- break;
- case 324000:
- *pll_params = params[3];
- break;
- case 216000:
- *pll_params = params[4];
- break;
- case 432000:
- *pll_params = params[5];
- break;
- case 648000:
- *pll_params = params[6];
- break;
- case 810000:
- *pll_params = params[7];
- break;
- default:
- MISSING_CASE(clock);
- return false;
+ for (i = 0; i < ARRAY_SIZE(icl_dp_combo_pll_24MHz_values); i++) {
+ if (clock == params[i].clock) {
+ *pll_params = params[i].wrpll;
+ return true;
+ }
}
- return true;
+ MISSING_CASE(clock);
+ return false;
}
-static bool icl_calc_tbt_pll(struct drm_i915_private *dev_priv, int clock,
+static bool icl_calc_tbt_pll(struct intel_crtc_state *crtc_state,
struct skl_wrpll_params *pll_params)
{
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+
*pll_params = dev_priv->cdclk.hw.ref == 24000 ?
icl_tbt_pll_24MHz_values : icl_tbt_pll_19_2MHz_values;
return true;
}
static bool icl_calc_dpll_state(struct intel_crtc_state *crtc_state,
- struct intel_encoder *encoder, int clock,
- struct intel_dpll_hw_state *pll_state)
+ struct intel_encoder *encoder)
{
- struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
u32 cfgcr0, cfgcr1;
struct skl_wrpll_params pll_params = { 0 };
bool ret;
if (intel_port_is_tc(dev_priv, encoder->port))
- ret = icl_calc_tbt_pll(dev_priv, clock, &pll_params);
+ ret = icl_calc_tbt_pll(crtc_state, &pll_params);
else if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_HDMI) ||
intel_crtc_has_type(crtc_state, INTEL_OUTPUT_DSI))
- ret = cnl_ddi_calculate_wrpll(clock, dev_priv, &pll_params);
+ ret = cnl_ddi_calculate_wrpll(crtc_state, &pll_params);
else
- ret = icl_calc_dp_combo_pll(dev_priv, clock, &pll_params);
+ ret = icl_calc_dp_combo_pll(crtc_state, &pll_params);
if (!ret)
return false;
DPLL_CFGCR1_PDIV(pll_params.pdiv) |
DPLL_CFGCR1_CENTRAL_FREQ_8400;
- pll_state->cfgcr0 = cfgcr0;
- pll_state->cfgcr1 = cfgcr1;
- return true;
-}
-
-int icl_calc_dp_combo_pll_link(struct drm_i915_private *dev_priv,
- u32 pll_id)
-{
- u32 cfgcr0, cfgcr1;
- u32 pdiv, kdiv, qdiv_mode, qdiv_ratio, dco_integer, dco_fraction;
- const struct skl_wrpll_params *params;
- int index, n_entries, link_clock;
-
- /* Read back values from DPLL CFGCR registers */
- cfgcr0 = I915_READ(ICL_DPLL_CFGCR0(pll_id));
- cfgcr1 = I915_READ(ICL_DPLL_CFGCR1(pll_id));
-
- dco_integer = cfgcr0 & DPLL_CFGCR0_DCO_INTEGER_MASK;
- dco_fraction = (cfgcr0 & DPLL_CFGCR0_DCO_FRACTION_MASK) >>
- DPLL_CFGCR0_DCO_FRACTION_SHIFT;
- pdiv = (cfgcr1 & DPLL_CFGCR1_PDIV_MASK) >> DPLL_CFGCR1_PDIV_SHIFT;
- kdiv = (cfgcr1 & DPLL_CFGCR1_KDIV_MASK) >> DPLL_CFGCR1_KDIV_SHIFT;
- qdiv_mode = (cfgcr1 & DPLL_CFGCR1_QDIV_MODE(1)) >>
- DPLL_CFGCR1_QDIV_MODE_SHIFT;
- qdiv_ratio = (cfgcr1 & DPLL_CFGCR1_QDIV_RATIO_MASK) >>
- DPLL_CFGCR1_QDIV_RATIO_SHIFT;
-
- params = dev_priv->cdclk.hw.ref == 24000 ?
- icl_dp_combo_pll_24MHz_values :
- icl_dp_combo_pll_19_2MHz_values;
- n_entries = ARRAY_SIZE(icl_dp_combo_pll_24MHz_values);
-
- for (index = 0; index < n_entries; index++) {
- if (dco_integer == params[index].dco_integer &&
- dco_fraction == params[index].dco_fraction &&
- pdiv == params[index].pdiv &&
- kdiv == params[index].kdiv &&
- qdiv_mode == params[index].qdiv_mode &&
- qdiv_ratio == params[index].qdiv_ratio)
- break;
- }
+ memset(&crtc_state->dpll_hw_state, 0,
+ sizeof(crtc_state->dpll_hw_state));
- /* Map PLL Index to Link Clock */
- switch (index) {
- default:
- MISSING_CASE(index);
- /* fall through */
- case 0:
- link_clock = 540000;
- break;
- case 1:
- link_clock = 270000;
- break;
- case 2:
- link_clock = 162000;
- break;
- case 3:
- link_clock = 324000;
- break;
- case 4:
- link_clock = 216000;
- break;
- case 5:
- link_clock = 432000;
- break;
- case 6:
- link_clock = 648000;
- break;
- case 7:
- link_clock = 810000;
- break;
- }
+ crtc_state->dpll_hw_state.cfgcr0 = cfgcr0;
+ crtc_state->dpll_hw_state.cfgcr1 = cfgcr1;
- return link_clock;
+ return true;
}
+
static enum tc_port icl_pll_id_to_tc_port(enum intel_dpll_id id)
{
return id - DPLL_ID_ICL_MGPLL1;
return tc_port + DPLL_ID_ICL_MGPLL1;
}
-bool intel_dpll_is_combophy(enum intel_dpll_id id)
-{
- return id == DPLL_ID_ICL_DPLL0 || id == DPLL_ID_ICL_DPLL1;
-}
-
static bool icl_mg_pll_find_divisors(int clock_khz, bool is_dp, bool use_ssc,
u32 *target_dco_khz,
struct intel_dpll_hw_state *state)
* The specification for this function uses real numbers, so the math had to be
* adapted to integer-only calculation, that's why it looks so different.
*/
-static bool icl_calc_mg_pll_state(struct intel_crtc_state *crtc_state,
- struct intel_encoder *encoder, int clock,
- struct intel_dpll_hw_state *pll_state)
+static bool icl_calc_mg_pll_state(struct intel_crtc_state *crtc_state)
{
- struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+ struct intel_dpll_hw_state *pll_state = &crtc_state->dpll_hw_state;
int refclk_khz = dev_priv->cdclk.hw.ref;
+ int clock = crtc_state->port_clock;
u32 dco_khz, m1div, m2div_int, m2div_rem, m2div_frac;
u32 iref_ndiv, iref_trim, iref_pulse_w;
u32 prop_coeff, int_coeff;
bool use_ssc = false;
bool is_dp = !intel_crtc_has_type(crtc_state, INTEL_OUTPUT_HDMI);
+ memset(pll_state, 0, sizeof(*pll_state));
+
if (!icl_mg_pll_find_divisors(clock, is_dp, use_ssc, &dco_khz,
pll_state)) {
DRM_DEBUG_KMS("Failed to find divisors for clock %d\n", clock);
}
static struct intel_shared_dpll *
-icl_get_dpll(struct intel_crtc *crtc, struct intel_crtc_state *crtc_state,
+icl_get_dpll(struct intel_crtc_state *crtc_state,
struct intel_encoder *encoder)
{
- struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
struct intel_digital_port *intel_dig_port;
struct intel_shared_dpll *pll;
- struct intel_dpll_hw_state pll_state = {};
enum port port = encoder->port;
enum intel_dpll_id min, max;
- int clock = crtc_state->port_clock;
bool ret;
if (intel_port_is_combophy(dev_priv, port)) {
min = DPLL_ID_ICL_DPLL0;
max = DPLL_ID_ICL_DPLL1;
- ret = icl_calc_dpll_state(crtc_state, encoder, clock,
- &pll_state);
+ ret = icl_calc_dpll_state(crtc_state, encoder);
} else if (intel_port_is_tc(dev_priv, port)) {
if (encoder->type == INTEL_OUTPUT_DP_MST) {
struct intel_dp_mst_encoder *mst_encoder;
if (intel_dig_port->tc_type == TC_PORT_TBT) {
min = DPLL_ID_ICL_TBTPLL;
max = min;
- ret = icl_calc_dpll_state(crtc_state, encoder, clock,
- &pll_state);
+ ret = icl_calc_dpll_state(crtc_state, encoder);
} else {
enum tc_port tc_port;
tc_port = intel_port_to_tc(dev_priv, port);
min = icl_tc_port_to_pll_id(tc_port);
max = min;
- ret = icl_calc_mg_pll_state(crtc_state, encoder, clock,
- &pll_state);
+ ret = icl_calc_mg_pll_state(crtc_state);
}
} else {
MISSING_CASE(port);
return NULL;
}
- crtc_state->dpll_hw_state = pll_state;
- pll = intel_find_shared_dpll(crtc, crtc_state, min, max);
+ pll = intel_find_shared_dpll(crtc_state, min, max);
if (!pll) {
DRM_DEBUG_KMS("No PLL selected\n");
return NULL;
return pll;
}
-static i915_reg_t icl_pll_id_to_enable_reg(enum intel_dpll_id id)
+static bool mg_pll_get_hw_state(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll,
+ struct intel_dpll_hw_state *hw_state)
{
- if (intel_dpll_is_combophy(id))
- return CNL_DPLL_ENABLE(id);
- else if (id == DPLL_ID_ICL_TBTPLL)
- return TBT_PLL_ENABLE;
+ const enum intel_dpll_id id = pll->info->id;
+ enum tc_port tc_port = icl_pll_id_to_tc_port(id);
+ intel_wakeref_t wakeref;
+ bool ret = false;
+ u32 val;
+
+ wakeref = intel_display_power_get_if_enabled(dev_priv,
+ POWER_DOMAIN_PLLS);
+ if (!wakeref)
+ return false;
- return MG_PLL_ENABLE(icl_pll_id_to_tc_port(id));
+ val = I915_READ(MG_PLL_ENABLE(tc_port));
+ if (!(val & PLL_ENABLE))
+ goto out;
+
+ hw_state->mg_refclkin_ctl = I915_READ(MG_REFCLKIN_CTL(tc_port));
+ hw_state->mg_refclkin_ctl &= MG_REFCLKIN_CTL_OD_2_MUX_MASK;
+
+ hw_state->mg_clktop2_coreclkctl1 =
+ I915_READ(MG_CLKTOP2_CORECLKCTL1(tc_port));
+ hw_state->mg_clktop2_coreclkctl1 &=
+ MG_CLKTOP2_CORECLKCTL1_A_DIVRATIO_MASK;
+
+ hw_state->mg_clktop2_hsclkctl =
+ I915_READ(MG_CLKTOP2_HSCLKCTL(tc_port));
+ hw_state->mg_clktop2_hsclkctl &=
+ MG_CLKTOP2_HSCLKCTL_TLINEDRV_CLKSEL_MASK |
+ MG_CLKTOP2_HSCLKCTL_CORE_INPUTSEL_MASK |
+ MG_CLKTOP2_HSCLKCTL_HSDIV_RATIO_MASK |
+ MG_CLKTOP2_HSCLKCTL_DSDIV_RATIO_MASK;
+
+ hw_state->mg_pll_div0 = I915_READ(MG_PLL_DIV0(tc_port));
+ hw_state->mg_pll_div1 = I915_READ(MG_PLL_DIV1(tc_port));
+ hw_state->mg_pll_lf = I915_READ(MG_PLL_LF(tc_port));
+ hw_state->mg_pll_frac_lock = I915_READ(MG_PLL_FRAC_LOCK(tc_port));
+ hw_state->mg_pll_ssc = I915_READ(MG_PLL_SSC(tc_port));
+
+ hw_state->mg_pll_bias = I915_READ(MG_PLL_BIAS(tc_port));
+ hw_state->mg_pll_tdc_coldst_bias =
+ I915_READ(MG_PLL_TDC_COLDST_BIAS(tc_port));
+
+ if (dev_priv->cdclk.hw.ref == 38400) {
+ hw_state->mg_pll_tdc_coldst_bias_mask = MG_PLL_TDC_COLDST_COLDSTART;
+ hw_state->mg_pll_bias_mask = 0;
+ } else {
+ hw_state->mg_pll_tdc_coldst_bias_mask = -1U;
+ hw_state->mg_pll_bias_mask = -1U;
+ }
+
+ hw_state->mg_pll_tdc_coldst_bias &= hw_state->mg_pll_tdc_coldst_bias_mask;
+ hw_state->mg_pll_bias &= hw_state->mg_pll_bias_mask;
+
+ ret = true;
+out:
+ intel_display_power_put(dev_priv, POWER_DOMAIN_PLLS, wakeref);
+ return ret;
}
static bool icl_pll_get_hw_state(struct drm_i915_private *dev_priv,
struct intel_shared_dpll *pll,
- struct intel_dpll_hw_state *hw_state)
+ struct intel_dpll_hw_state *hw_state,
+ i915_reg_t enable_reg)
{
const enum intel_dpll_id id = pll->info->id;
intel_wakeref_t wakeref;
if (!wakeref)
return false;
- val = I915_READ(icl_pll_id_to_enable_reg(id));
+ val = I915_READ(enable_reg);
if (!(val & PLL_ENABLE))
goto out;
- if (intel_dpll_is_combophy(id) ||
- id == DPLL_ID_ICL_TBTPLL) {
- hw_state->cfgcr0 = I915_READ(ICL_DPLL_CFGCR0(id));
- hw_state->cfgcr1 = I915_READ(ICL_DPLL_CFGCR1(id));
- } else {
- enum tc_port tc_port = icl_pll_id_to_tc_port(id);
-
- hw_state->mg_refclkin_ctl = I915_READ(MG_REFCLKIN_CTL(tc_port));
- hw_state->mg_refclkin_ctl &= MG_REFCLKIN_CTL_OD_2_MUX_MASK;
-
- hw_state->mg_clktop2_coreclkctl1 =
- I915_READ(MG_CLKTOP2_CORECLKCTL1(tc_port));
- hw_state->mg_clktop2_coreclkctl1 &=
- MG_CLKTOP2_CORECLKCTL1_A_DIVRATIO_MASK;
-
- hw_state->mg_clktop2_hsclkctl =
- I915_READ(MG_CLKTOP2_HSCLKCTL(tc_port));
- hw_state->mg_clktop2_hsclkctl &=
- MG_CLKTOP2_HSCLKCTL_TLINEDRV_CLKSEL_MASK |
- MG_CLKTOP2_HSCLKCTL_CORE_INPUTSEL_MASK |
- MG_CLKTOP2_HSCLKCTL_HSDIV_RATIO_MASK |
- MG_CLKTOP2_HSCLKCTL_DSDIV_RATIO_MASK;
-
- hw_state->mg_pll_div0 = I915_READ(MG_PLL_DIV0(tc_port));
- hw_state->mg_pll_div1 = I915_READ(MG_PLL_DIV1(tc_port));
- hw_state->mg_pll_lf = I915_READ(MG_PLL_LF(tc_port));
- hw_state->mg_pll_frac_lock = I915_READ(MG_PLL_FRAC_LOCK(tc_port));
- hw_state->mg_pll_ssc = I915_READ(MG_PLL_SSC(tc_port));
-
- hw_state->mg_pll_bias = I915_READ(MG_PLL_BIAS(tc_port));
- hw_state->mg_pll_tdc_coldst_bias =
- I915_READ(MG_PLL_TDC_COLDST_BIAS(tc_port));
-
- if (dev_priv->cdclk.hw.ref == 38400) {
- hw_state->mg_pll_tdc_coldst_bias_mask = MG_PLL_TDC_COLDST_COLDSTART;
- hw_state->mg_pll_bias_mask = 0;
- } else {
- hw_state->mg_pll_tdc_coldst_bias_mask = -1U;
- hw_state->mg_pll_bias_mask = -1U;
- }
-
- hw_state->mg_pll_tdc_coldst_bias &= hw_state->mg_pll_tdc_coldst_bias_mask;
- hw_state->mg_pll_bias &= hw_state->mg_pll_bias_mask;
- }
+ hw_state->cfgcr0 = I915_READ(ICL_DPLL_CFGCR0(id));
+ hw_state->cfgcr1 = I915_READ(ICL_DPLL_CFGCR1(id));
ret = true;
out:
return ret;
}
+static bool combo_pll_get_hw_state(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll,
+ struct intel_dpll_hw_state *hw_state)
+{
+ return icl_pll_get_hw_state(dev_priv, pll, hw_state,
+ CNL_DPLL_ENABLE(pll->info->id));
+}
+
+static bool tbt_pll_get_hw_state(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll,
+ struct intel_dpll_hw_state *hw_state)
+{
+ return icl_pll_get_hw_state(dev_priv, pll, hw_state, TBT_PLL_ENABLE);
+}
+
static void icl_dpll_write(struct drm_i915_private *dev_priv,
struct intel_shared_dpll *pll)
{
POSTING_READ(MG_PLL_TDC_COLDST_BIAS(tc_port));
}
-static void icl_pll_enable(struct drm_i915_private *dev_priv,
- struct intel_shared_dpll *pll)
+static void icl_pll_power_enable(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll,
+ i915_reg_t enable_reg)
{
- const enum intel_dpll_id id = pll->info->id;
- i915_reg_t enable_reg = icl_pll_id_to_enable_reg(id);
u32 val;
val = I915_READ(enable_reg);
* The spec says we need to "wait" but it also says it should be
* immediate.
*/
- if (intel_wait_for_register(dev_priv, enable_reg, PLL_POWER_STATE,
- PLL_POWER_STATE, 1))
- DRM_ERROR("PLL %d Power not enabled\n", id);
+ if (intel_wait_for_register(&dev_priv->uncore, enable_reg,
+ PLL_POWER_STATE, PLL_POWER_STATE, 1))
+ DRM_ERROR("PLL %d Power not enabled\n", pll->info->id);
+}
- if (intel_dpll_is_combophy(id) || id == DPLL_ID_ICL_TBTPLL)
- icl_dpll_write(dev_priv, pll);
- else
- icl_mg_pll_write(dev_priv, pll);
+static void icl_pll_enable(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll,
+ i915_reg_t enable_reg)
+{
+ u32 val;
+
+ val = I915_READ(enable_reg);
+ val |= PLL_ENABLE;
+ I915_WRITE(enable_reg, val);
+
+ /* Timeout is actually 600us. */
+ if (intel_wait_for_register(&dev_priv->uncore, enable_reg,
+ PLL_LOCK, PLL_LOCK, 1))
+ DRM_ERROR("PLL %d not locked\n", pll->info->id);
+}
+
+static void combo_pll_enable(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll)
+{
+ i915_reg_t enable_reg = CNL_DPLL_ENABLE(pll->info->id);
+
+ icl_pll_power_enable(dev_priv, pll, enable_reg);
+
+ icl_dpll_write(dev_priv, pll);
/*
* DVFS pre sequence would be here, but in our driver the cdclk code
* paths should already be setting the appropriate voltage, hence we do
- * nothign here.
+ * nothing here.
*/
- val = I915_READ(enable_reg);
- val |= PLL_ENABLE;
- I915_WRITE(enable_reg, val);
+ icl_pll_enable(dev_priv, pll, enable_reg);
- if (intel_wait_for_register(dev_priv, enable_reg, PLL_LOCK, PLL_LOCK,
- 1)) /* 600us actually. */
- DRM_ERROR("PLL %d not locked\n", id);
+ /* DVFS post sequence would be here. See the comment above. */
+}
+
+static void tbt_pll_enable(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll)
+{
+ icl_pll_power_enable(dev_priv, pll, TBT_PLL_ENABLE);
+
+ icl_dpll_write(dev_priv, pll);
+
+ /*
+ * DVFS pre sequence would be here, but in our driver the cdclk code
+ * paths should already be setting the appropriate voltage, hence we do
+ * nothing here.
+ */
+
+ icl_pll_enable(dev_priv, pll, TBT_PLL_ENABLE);
+
+ /* DVFS post sequence would be here. See the comment above. */
+}
+
+static void mg_pll_enable(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll)
+{
+ i915_reg_t enable_reg =
+ MG_PLL_ENABLE(icl_pll_id_to_tc_port(pll->info->id));
+
+ icl_pll_power_enable(dev_priv, pll, enable_reg);
+
+ icl_mg_pll_write(dev_priv, pll);
+
+ /*
+ * DVFS pre sequence would be here, but in our driver the cdclk code
+ * paths should already be setting the appropriate voltage, hence we do
+ * nothing here.
+ */
+
+ icl_pll_enable(dev_priv, pll, enable_reg);
/* DVFS post sequence would be here. See the comment above. */
}
static void icl_pll_disable(struct drm_i915_private *dev_priv,
- struct intel_shared_dpll *pll)
+ struct intel_shared_dpll *pll,
+ i915_reg_t enable_reg)
{
- const enum intel_dpll_id id = pll->info->id;
- i915_reg_t enable_reg = icl_pll_id_to_enable_reg(id);
u32 val;
/* The first steps are done by intel_ddi_post_disable(). */
I915_WRITE(enable_reg, val);
/* Timeout is actually 1us. */
- if (intel_wait_for_register(dev_priv, enable_reg, PLL_LOCK, 0, 1))
- DRM_ERROR("PLL %d locked\n", id);
+ if (intel_wait_for_register(&dev_priv->uncore,
+ enable_reg, PLL_LOCK, 0, 1))
+ DRM_ERROR("PLL %d locked\n", pll->info->id);
/* DVFS post sequence would be here. See the comment above. */
* The spec says we need to "wait" but it also says it should be
* immediate.
*/
- if (intel_wait_for_register(dev_priv, enable_reg, PLL_POWER_STATE, 0,
- 1))
- DRM_ERROR("PLL %d Power not disabled\n", id);
+ if (intel_wait_for_register(&dev_priv->uncore,
+ enable_reg, PLL_POWER_STATE, 0, 1))
+ DRM_ERROR("PLL %d Power not disabled\n", pll->info->id);
+}
+
+static void combo_pll_disable(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll)
+{
+ icl_pll_disable(dev_priv, pll, CNL_DPLL_ENABLE(pll->info->id));
+}
+
+static void tbt_pll_disable(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll)
+{
+ icl_pll_disable(dev_priv, pll, TBT_PLL_ENABLE);
+}
+
+static void mg_pll_disable(struct drm_i915_private *dev_priv,
+ struct intel_shared_dpll *pll)
+{
+ i915_reg_t enable_reg =
+ MG_PLL_ENABLE(icl_pll_id_to_tc_port(pll->info->id));
+
+ icl_pll_disable(dev_priv, pll, enable_reg);
}
static void icl_dump_hw_state(struct drm_i915_private *dev_priv,
hw_state->mg_pll_tdc_coldst_bias);
}
-static const struct intel_shared_dpll_funcs icl_pll_funcs = {
- .enable = icl_pll_enable,
- .disable = icl_pll_disable,
- .get_hw_state = icl_pll_get_hw_state,
+static const struct intel_shared_dpll_funcs combo_pll_funcs = {
+ .enable = combo_pll_enable,
+ .disable = combo_pll_disable,
+ .get_hw_state = combo_pll_get_hw_state,
+};
+
+static const struct intel_shared_dpll_funcs tbt_pll_funcs = {
+ .enable = tbt_pll_enable,
+ .disable = tbt_pll_disable,
+ .get_hw_state = tbt_pll_get_hw_state,
+};
+
+static const struct intel_shared_dpll_funcs mg_pll_funcs = {
+ .enable = mg_pll_enable,
+ .disable = mg_pll_disable,
+ .get_hw_state = mg_pll_get_hw_state,
};
static const struct dpll_info icl_plls[] = {
- { "DPLL 0", &icl_pll_funcs, DPLL_ID_ICL_DPLL0, 0 },
- { "DPLL 1", &icl_pll_funcs, DPLL_ID_ICL_DPLL1, 0 },
- { "TBT PLL", &icl_pll_funcs, DPLL_ID_ICL_TBTPLL, 0 },
- { "MG PLL 1", &icl_pll_funcs, DPLL_ID_ICL_MGPLL1, 0 },
- { "MG PLL 2", &icl_pll_funcs, DPLL_ID_ICL_MGPLL2, 0 },
- { "MG PLL 3", &icl_pll_funcs, DPLL_ID_ICL_MGPLL3, 0 },
- { "MG PLL 4", &icl_pll_funcs, DPLL_ID_ICL_MGPLL4, 0 },
+ { "DPLL 0", &combo_pll_funcs, DPLL_ID_ICL_DPLL0, 0 },
+ { "DPLL 1", &combo_pll_funcs, DPLL_ID_ICL_DPLL1, 0 },
+ { "TBT PLL", &tbt_pll_funcs, DPLL_ID_ICL_TBTPLL, 0 },
+ { "MG PLL 1", &mg_pll_funcs, DPLL_ID_ICL_MGPLL1, 0 },
+ { "MG PLL 2", &mg_pll_funcs, DPLL_ID_ICL_MGPLL2, 0 },
+ { "MG PLL 3", &mg_pll_funcs, DPLL_ID_ICL_MGPLL3, 0 },
+ { "MG PLL 4", &mg_pll_funcs, DPLL_ID_ICL_MGPLL4, 0 },
{ },
};
.dump_hw_state = icl_dump_hw_state,
};
+static const struct dpll_info ehl_plls[] = {
+ { "DPLL 0", &combo_pll_funcs, DPLL_ID_ICL_DPLL0, 0 },
+ { "DPLL 1", &combo_pll_funcs, DPLL_ID_ICL_DPLL1, 0 },
+ { },
+};
+
+static const struct intel_dpll_mgr ehl_pll_mgr = {
+ .dpll_info = ehl_plls,
+ .get_dpll = icl_get_dpll,
+ .dump_hw_state = icl_dump_hw_state,
+};
+
/**
* intel_shared_dpll_init - Initialize shared DPLLs
* @dev: drm device
const struct dpll_info *dpll_info;
int i;
- if (IS_ICELAKE(dev_priv))
+ if (IS_ELKHARTLAKE(dev_priv))
+ dpll_mgr = &ehl_pll_mgr;
+ else if (INTEL_GEN(dev_priv) >= 11)
dpll_mgr = &icl_pll_mgr;
else if (IS_CANNONLAKE(dev_priv))
dpll_mgr = &cnl_pll_mgr;
/**
* intel_get_shared_dpll - get a shared DPLL for CRTC and encoder combination
- * @crtc: CRTC
- * @crtc_state: atomic state for @crtc
+ * @crtc_state: atomic state for the crtc
* @encoder: encoder
*
* Find an appropriate DPLL for the given CRTC and encoder combination. A
- * reference from the @crtc to the returned pll is registered in the atomic
- * state. That configuration is made effective by calling
+ * reference from the @crtc_state to the returned pll is registered in the
+ * atomic state. That configuration is made effective by calling
* intel_shared_dpll_swap_state(). The reference should be released by calling
* intel_release_shared_dpll().
*
* Returns:
- * A shared DPLL to be used by @crtc and @encoder with the given @crtc_state.
+ * A shared DPLL to be used by @crtc_state and @encoder.
*/
struct intel_shared_dpll *
-intel_get_shared_dpll(struct intel_crtc *crtc,
- struct intel_crtc_state *crtc_state,
+intel_get_shared_dpll(struct intel_crtc_state *crtc_state,
struct intel_encoder *encoder)
{
- struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
const struct intel_dpll_mgr *dpll_mgr = dev_priv->dpll_mgr;
if (WARN_ON(!dpll_mgr))
return NULL;
- return dpll_mgr->get_dpll(crtc, crtc_state, encoder);
+ return dpll_mgr->get_dpll(crtc_state, encoder);
}
/**
bool state);
#define assert_shared_dpll_enabled(d, p) assert_shared_dpll(d, p, true)
#define assert_shared_dpll_disabled(d, p) assert_shared_dpll(d, p, false)
-struct intel_shared_dpll *intel_get_shared_dpll(struct intel_crtc *crtc,
- struct intel_crtc_state *state,
+struct intel_shared_dpll *intel_get_shared_dpll(struct intel_crtc_state *state,
struct intel_encoder *encoder);
void intel_release_shared_dpll(struct intel_shared_dpll *dpll,
struct intel_crtc *crtc,
void intel_dpll_dump_hw_state(struct drm_i915_private *dev_priv,
struct intel_dpll_hw_state *hw_state);
-int icl_calc_dp_combo_pll_link(struct drm_i915_private *dev_priv,
- u32 pll_id);
int cnl_hdmi_pll_ref_clock(struct drm_i915_private *dev_priv);
enum intel_dpll_id icl_tc_port_to_pll_id(enum tc_port tc_port);
bool intel_dpll_is_combophy(enum intel_dpll_id id);
#include <linux/async.h>
#include <linux/i2c.h>
-#include <linux/hdmi.h>
#include <linux/sched/clock.h>
#include <linux/stackdepot.h>
-#include <drm/i915_drm.h>
-#include "i915_drv.h"
+
+#include <drm/drm_atomic.h>
#include <drm/drm_crtc.h>
-#include <drm/drm_encoder.h>
-#include <drm/drm_fb_helper.h>
#include <drm/drm_dp_dual_mode_helper.h>
#include <drm/drm_dp_mst_helper.h>
+#include <drm/drm_encoder.h>
+#include <drm/drm_fb_helper.h>
#include <drm/drm_probe_helper.h>
#include <drm/drm_rect.h>
#include <drm/drm_vblank.h>
-#include <drm/drm_atomic.h>
+#include <drm/i915_drm.h>
+#include <drm/i915_mei_hdcp_interface.h>
#include <media/cec-notifier.h>
+#include "i915_drv.h"
+
struct drm_printer;
/**
struct intel_digital_port;
+enum check_link_response {
+ HDCP_LINK_PROTECTED = 0,
+ HDCP_TOPOLOGY_CHANGE,
+ HDCP_LINK_INTEGRITY_FAILURE,
+ HDCP_REAUTH_REQUEST
+};
+
/*
* This structure serves as a translation layer between the generic HDCP code
* and the bus-specific code. What that means is that HDCP over HDMI differs
/* Detects panel's hdcp capability. This is optional for HDMI. */
int (*hdcp_capable)(struct intel_digital_port *intel_dig_port,
bool *hdcp_capable);
+
+ /* HDCP adaptation(DP/HDMI) required on the port */
+ enum hdcp_wired_protocol protocol;
+
+ /* Detects whether sink is HDCP2.2 capable */
+ int (*hdcp_2_2_capable)(struct intel_digital_port *intel_dig_port,
+ bool *capable);
+
+ /* Write HDCP2.2 messages */
+ int (*write_2_2_msg)(struct intel_digital_port *intel_dig_port,
+ void *buf, size_t size);
+
+ /* Read HDCP2.2 messages */
+ int (*read_2_2_msg)(struct intel_digital_port *intel_dig_port,
+ u8 msg_id, void *buf, size_t size);
+
+ /*
+ * Implementation of DP HDCP2.2 Errata for the communication of stream
+ * type to Receivers. In DP HDCP2.2 Stream type is one of the input to
+ * the HDCP2.2 Cipher for En/De-Cryption. Not applicable for HDMI.
+ */
+ int (*config_stream_type)(struct intel_digital_port *intel_dig_port,
+ bool is_repeater, u8 type);
+
+ /* HDCP2.2 Link Integrity Check */
+ int (*check_2_2_link)(struct intel_digital_port *intel_dig_port);
};
struct intel_hdcp {
u64 value;
struct delayed_work check_work;
struct work_struct prop_work;
+
+ /* HDCP1.4 Encryption status */
+ bool hdcp_encrypted;
+
+ /* HDCP2.2 related definitions */
+ /* Flag indicates whether this connector supports HDCP2.2 or not. */
+ bool hdcp2_supported;
+
+ /* HDCP2.2 Encryption status */
+ bool hdcp2_encrypted;
+
+ /*
+ * Content Stream Type defined by content owner. TYPE0(0x0) content can
+ * flow in the link protected by HDCP2.2 or HDCP1.4, where as TYPE1(0x1)
+ * content can flow only through a link protected by HDCP2.2.
+ */
+ u8 content_type;
+ struct hdcp_port_data port_data;
+
+ bool is_paired;
+ bool is_repeater;
+
+ /*
+ * Count of ReceiverID_List received. Initialized to 0 at AKE_INIT.
+ * Incremented after processing the RepeaterAuth_Send_ReceiverID_List.
+ * When it rolls over re-auth has to be triggered.
+ */
+ u32 seq_num_v;
+
+ /*
+ * Count of RepeaterAuth_Stream_Manage msg propagated.
+ * Initialized to 0 on AKE_INIT. Incremented after every successful
+ * transmission of RepeaterAuth_Stream_Manage message. When it rolls
+ * over re-Auth has to be triggered.
+ */
+ u32 seq_num_m;
+
+ /*
+ * Work queue to signal the CP_IRQ. Used for the waiters to read the
+ * available information from HDCP DP sink.
+ */
+ wait_queue_head_t cp_irq_queue;
+ atomic_t cp_irq_count;
+ int cp_irq_count_cached;
};
struct intel_connector {
* state only when all crtc's are DPMS off.
*/
struct intel_cdclk_state actual;
+
+ int force_min_cdclk;
+ bool force_min_cdclk_changed;
+ /* pipe to which cd2x update is synchronized */
+ enum pipe pipe;
} cdclk;
bool dpll_set, modeset;
struct intel_link_m_n fdi_m_n;
bool ips_enabled;
- bool ips_force_disable;
+
+ bool crc_enabled;
bool enable_fbc;
/* Gamma mode programmed on the pipe */
u32 gamma_mode;
+ union {
+ /* CSC mode programmed on the pipe */
+ u32 csc_mode;
+
+ /* CHV CGM mode */
+ u32 cgm_mode;
+ };
+
/* bitmask of visible planes (enum plane_id) */
u8 active_planes;
u8 nv12_planes;
+ u8 c8_planes;
/* bitmask of planes that will be updated during the commit */
u8 update_planes;
+ struct {
+ u32 enable;
+ u32 gcp;
+ union hdmi_infoframe avi;
+ union hdmi_infoframe spd;
+ union hdmi_infoframe hdmi;
+ } infoframes;
+
/* HDMI scrambling status */
bool hdmi_scrambling;
/* Output down scaling is done in LSPCON device */
bool lspcon_downsampling;
+ /* enable pipe gamma? */
+ bool gamma_enable;
+
+ /* enable pipe csc? */
+ bool csc_enable;
+
/* Display Stream compression state */
struct {
bool compression_enable;
struct intel_crtc_state *config;
- /* global reset count when the last flip was submitted */
- unsigned int reset_count;
-
/* Access to these should be protected by dev_priv->irq_lock. */
bool cpu_fifo_underrun_disabled;
bool pch_fifo_underrun_disabled;
const struct intel_crtc_state *crtc_state,
unsigned int type,
const void *frame, ssize_t len);
+ void (*read_infoframe)(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ unsigned int type,
+ void *frame, ssize_t len);
void (*set_infoframes)(struct intel_encoder *encoder,
bool enable,
const struct intel_crtc_state *crtc_state,
const struct drm_connector_state *conn_state);
- bool (*infoframe_enabled)(struct intel_encoder *encoder,
+ u32 (*infoframes_enabled)(struct intel_encoder *encoder,
const struct intel_crtc_state *pipe_config);
};
void gen6_reset_rps_interrupts(struct drm_i915_private *dev_priv);
void gen6_enable_rps_interrupts(struct drm_i915_private *dev_priv);
void gen6_disable_rps_interrupts(struct drm_i915_private *dev_priv);
+void gen6_rps_reset_ei(struct drm_i915_private *dev_priv);
static inline u32 gen6_sanitize_rps_pm_mask(const struct drm_i915_private *i915,
u32 mask)
void gen9_enable_guc_interrupts(struct drm_i915_private *dev_priv);
void gen9_disable_guc_interrupts(struct drm_i915_private *dev_priv);
-/* intel_crt.c */
-bool intel_crt_port_enabled(struct drm_i915_private *dev_priv,
- i915_reg_t adpa_reg, enum pipe *pipe);
-void intel_crt_init(struct drm_i915_private *dev_priv);
-void intel_crt_reset(struct drm_encoder *encoder);
-
-/* intel_ddi.c */
-void intel_ddi_fdi_post_disable(struct intel_encoder *intel_encoder,
- const struct intel_crtc_state *old_crtc_state,
- const struct drm_connector_state *old_conn_state);
-void hsw_fdi_link_train(struct intel_crtc *crtc,
- const struct intel_crtc_state *crtc_state);
-void intel_ddi_init(struct drm_i915_private *dev_priv, enum port port);
-bool intel_ddi_get_hw_state(struct intel_encoder *encoder, enum pipe *pipe);
-void intel_ddi_enable_transcoder_func(const struct intel_crtc_state *crtc_state);
-void intel_ddi_disable_transcoder_func(const struct intel_crtc_state *crtc_state);
-void intel_ddi_enable_pipe_clock(const struct intel_crtc_state *crtc_state);
-void intel_ddi_disable_pipe_clock(const struct intel_crtc_state *crtc_state);
-void intel_ddi_set_pipe_settings(const struct intel_crtc_state *crtc_state);
-void intel_ddi_prepare_link_retrain(struct intel_dp *intel_dp);
-bool intel_ddi_connector_get_hw_state(struct intel_connector *intel_connector);
-void intel_ddi_get_config(struct intel_encoder *encoder,
- struct intel_crtc_state *pipe_config);
-
-void intel_ddi_set_vc_payload_alloc(const struct intel_crtc_state *crtc_state,
- bool state);
-void intel_ddi_compute_min_voltage_level(struct drm_i915_private *dev_priv,
- struct intel_crtc_state *crtc_state);
-u32 bxt_signal_levels(struct intel_dp *intel_dp);
-u32 ddi_signal_levels(struct intel_dp *intel_dp);
-u8 intel_ddi_dp_voltage_max(struct intel_encoder *encoder);
-u8 intel_ddi_dp_pre_emphasis_max(struct intel_encoder *encoder,
- u8 voltage_swing);
-int intel_ddi_toggle_hdcp_signalling(struct intel_encoder *intel_encoder,
- bool enable);
-void icl_sanitize_encoder_pll_mapping(struct intel_encoder *encoder);
-int cnl_calc_wrpll_link(struct drm_i915_private *dev_priv,
- enum intel_dpll_id pll_id);
-
-unsigned int intel_fb_align_height(const struct drm_framebuffer *fb,
- int color_plane, unsigned int height);
-
-/* intel_audio.c */
-void intel_init_audio_hooks(struct drm_i915_private *dev_priv);
-void intel_audio_codec_enable(struct intel_encoder *encoder,
- const struct intel_crtc_state *crtc_state,
- const struct drm_connector_state *conn_state);
-void intel_audio_codec_disable(struct intel_encoder *encoder,
- const struct intel_crtc_state *old_crtc_state,
- const struct drm_connector_state *old_conn_state);
-void i915_audio_component_init(struct drm_i915_private *dev_priv);
-void i915_audio_component_cleanup(struct drm_i915_private *dev_priv);
-void intel_audio_init(struct drm_i915_private *dev_priv);
-void intel_audio_deinit(struct drm_i915_private *dev_priv);
-
-/* intel_cdclk.c */
-int intel_crtc_compute_min_cdclk(const struct intel_crtc_state *crtc_state);
-void skl_init_cdclk(struct drm_i915_private *dev_priv);
-void skl_uninit_cdclk(struct drm_i915_private *dev_priv);
-void cnl_init_cdclk(struct drm_i915_private *dev_priv);
-void cnl_uninit_cdclk(struct drm_i915_private *dev_priv);
-void bxt_init_cdclk(struct drm_i915_private *dev_priv);
-void bxt_uninit_cdclk(struct drm_i915_private *dev_priv);
-void icl_init_cdclk(struct drm_i915_private *dev_priv);
-void icl_uninit_cdclk(struct drm_i915_private *dev_priv);
-void intel_init_cdclk_hooks(struct drm_i915_private *dev_priv);
-void intel_update_max_cdclk(struct drm_i915_private *dev_priv);
-void intel_update_cdclk(struct drm_i915_private *dev_priv);
-void intel_update_rawclk(struct drm_i915_private *dev_priv);
-bool intel_cdclk_needs_modeset(const struct intel_cdclk_state *a,
- const struct intel_cdclk_state *b);
-bool intel_cdclk_changed(const struct intel_cdclk_state *a,
- const struct intel_cdclk_state *b);
-void intel_set_cdclk(struct drm_i915_private *dev_priv,
- const struct intel_cdclk_state *cdclk_state);
-void intel_dump_cdclk_state(const struct intel_cdclk_state *cdclk_state,
- const char *context);
-
/* intel_display.c */
+void intel_plane_destroy(struct drm_plane *plane);
void i830_enable_pipe(struct drm_i915_private *dev_priv, enum pipe pipe);
void i830_disable_pipe(struct drm_i915_private *dev_priv, enum pipe pipe);
enum pipe intel_crtc_pch_transcoder(struct intel_crtc *crtc);
unsigned int intel_fb_xy_to_linear(int x, int y,
const struct intel_plane_state *state,
int plane);
+unsigned int intel_fb_align_height(const struct drm_framebuffer *fb,
+ int color_plane, unsigned int height);
void intel_add_fb_offsets(int *x, int *y,
const struct intel_plane_state *state, int plane);
unsigned int intel_rotation_info_size(const struct intel_rotation_info *rot_info);
void intel_dp_set_m_n(const struct intel_crtc_state *crtc_state,
enum link_m_n_set m_n);
int intel_dotclock_calculate(int link_freq, const struct intel_link_m_n *m_n);
-bool bxt_find_best_dpll(struct intel_crtc_state *crtc_state, int target_clock,
+bool bxt_find_best_dpll(struct intel_crtc_state *crtc_state,
struct dpll *best_clock);
int chv_calc_dpll_params(int refclk, struct dpll *pll_clock);
u32 pixel_format, u64 modifier,
unsigned int rotation);
-/* intel_connector.c */
-int intel_connector_init(struct intel_connector *connector);
-struct intel_connector *intel_connector_alloc(void);
-void intel_connector_free(struct intel_connector *connector);
-void intel_connector_destroy(struct drm_connector *connector);
-int intel_connector_register(struct drm_connector *connector);
-void intel_connector_unregister(struct drm_connector *connector);
-void intel_connector_attach_encoder(struct intel_connector *connector,
- struct intel_encoder *encoder);
-bool intel_connector_get_hw_state(struct intel_connector *connector);
-enum pipe intel_connector_get_pipe(struct intel_connector *connector);
-int intel_connector_update_modes(struct drm_connector *connector,
- struct edid *edid);
-int intel_ddc_get_modes(struct drm_connector *c, struct i2c_adapter *adapter);
-void intel_attach_force_audio_property(struct drm_connector *connector);
-void intel_attach_broadcast_rgb_property(struct drm_connector *connector);
-void intel_attach_aspect_ratio_property(struct drm_connector *connector);
-
-/* intel_csr.c */
-void intel_csr_ucode_init(struct drm_i915_private *);
-void intel_csr_load_program(struct drm_i915_private *);
-void intel_csr_ucode_fini(struct drm_i915_private *);
-void intel_csr_ucode_suspend(struct drm_i915_private *);
-void intel_csr_ucode_resume(struct drm_i915_private *);
-
-/* intel_dp.c */
-bool intel_dp_port_enabled(struct drm_i915_private *dev_priv,
- i915_reg_t dp_reg, enum port port,
- enum pipe *pipe);
-bool intel_dp_init(struct drm_i915_private *dev_priv, i915_reg_t output_reg,
- enum port port);
-bool intel_dp_init_connector(struct intel_digital_port *intel_dig_port,
- struct intel_connector *intel_connector);
-void intel_dp_set_link_params(struct intel_dp *intel_dp,
- int link_rate, u8 lane_count,
- bool link_mst);
-int intel_dp_get_link_train_fallback_values(struct intel_dp *intel_dp,
- int link_rate, u8 lane_count);
+/* intel_dp_link_training.c */
void intel_dp_start_link_train(struct intel_dp *intel_dp);
void intel_dp_stop_link_train(struct intel_dp *intel_dp);
-int intel_dp_retrain_link(struct intel_encoder *encoder,
- struct drm_modeset_acquire_ctx *ctx);
-void intel_dp_sink_dpms(struct intel_dp *intel_dp, int mode);
-void intel_dp_sink_set_decompression_state(struct intel_dp *intel_dp,
- const struct intel_crtc_state *crtc_state,
- bool enable);
-void intel_dp_encoder_reset(struct drm_encoder *encoder);
-void intel_dp_encoder_suspend(struct intel_encoder *intel_encoder);
-void intel_dp_encoder_flush_work(struct drm_encoder *encoder);
-int intel_dp_compute_config(struct intel_encoder *encoder,
- struct intel_crtc_state *pipe_config,
- struct drm_connector_state *conn_state);
-bool intel_dp_is_edp(struct intel_dp *intel_dp);
-bool intel_dp_is_port_edp(struct drm_i915_private *dev_priv, enum port port);
-enum irqreturn intel_dp_hpd_pulse(struct intel_digital_port *intel_dig_port,
- bool long_hpd);
-void intel_edp_backlight_on(const struct intel_crtc_state *crtc_state,
- const struct drm_connector_state *conn_state);
-void intel_edp_backlight_off(const struct drm_connector_state *conn_state);
-void intel_edp_panel_vdd_on(struct intel_dp *intel_dp);
-void intel_edp_panel_on(struct intel_dp *intel_dp);
-void intel_edp_panel_off(struct intel_dp *intel_dp);
-void intel_dp_mst_suspend(struct drm_i915_private *dev_priv);
-void intel_dp_mst_resume(struct drm_i915_private *dev_priv);
-int intel_dp_max_link_rate(struct intel_dp *intel_dp);
-int intel_dp_max_lane_count(struct intel_dp *intel_dp);
-int intel_dp_rate_select(struct intel_dp *intel_dp, int rate);
-void intel_dp_hot_plug(struct intel_encoder *intel_encoder);
-void intel_power_sequencer_reset(struct drm_i915_private *dev_priv);
-u32 intel_dp_pack_aux(const u8 *src, int src_bytes);
-void intel_plane_destroy(struct drm_plane *plane);
-void intel_edp_drrs_enable(struct intel_dp *intel_dp,
- const struct intel_crtc_state *crtc_state);
-void intel_edp_drrs_disable(struct intel_dp *intel_dp,
- const struct intel_crtc_state *crtc_state);
-void intel_edp_drrs_invalidate(struct drm_i915_private *dev_priv,
- unsigned int frontbuffer_bits);
-void intel_edp_drrs_flush(struct drm_i915_private *dev_priv,
- unsigned int frontbuffer_bits);
-
-void
-intel_dp_program_link_training_pattern(struct intel_dp *intel_dp,
- u8 dp_train_pat);
-void
-intel_dp_set_signal_levels(struct intel_dp *intel_dp);
-void intel_dp_set_idle_link_train(struct intel_dp *intel_dp);
-u8
-intel_dp_voltage_max(struct intel_dp *intel_dp);
-u8
-intel_dp_pre_emphasis_max(struct intel_dp *intel_dp, u8 voltage_swing);
-void intel_dp_compute_rate(struct intel_dp *intel_dp, int port_clock,
- u8 *link_bw, u8 *rate_select);
-bool intel_dp_source_supports_hbr2(struct intel_dp *intel_dp);
-bool intel_dp_source_supports_hbr3(struct intel_dp *intel_dp);
-bool
-intel_dp_get_link_status(struct intel_dp *intel_dp, u8 link_status[DP_LINK_STATUS_SIZE]);
-u16 intel_dp_dsc_get_output_bpp(int link_clock, u8 lane_count,
- int mode_clock, int mode_hdisplay);
-u8 intel_dp_dsc_get_slice_count(struct intel_dp *intel_dp, int mode_clock,
- int mode_hdisplay);
/* intel_vdsc.c */
int intel_dp_compute_dsc_params(struct intel_dp *intel_dp,
enum intel_display_power_domain
intel_dsc_power_domain(const struct intel_crtc_state *crtc_state);
-static inline unsigned int intel_dp_unused_lane_mask(int lane_count)
-{
- return ~((1 << lane_count) - 1) & 0xf;
-}
-
-bool intel_dp_read_dpcd(struct intel_dp *intel_dp);
-int intel_dp_link_required(int pixel_clock, int bpp);
-int intel_dp_max_data_rate(int max_link_clock, int max_lanes);
-bool intel_digital_port_connected(struct intel_encoder *encoder);
-void icl_tc_phy_disconnect(struct drm_i915_private *dev_priv,
- struct intel_digital_port *dig_port);
-
/* intel_dp_aux_backlight.c */
int intel_dp_aux_init_backlight_funcs(struct intel_connector *intel_connector);
/* intel_dsi_dcs_backlight.c */
int intel_dsi_dcs_init_backlight_funcs(struct intel_connector *intel_connector);
-/* intel_dvo.c */
-void intel_dvo_init(struct drm_i915_private *dev_priv);
/* intel_hotplug.c */
void intel_hpd_poll_init(struct drm_i915_private *dev_priv);
bool intel_encoder_hotplug(struct intel_encoder *encoder,
struct intel_connector *connector);
-/* legacy fbdev emulation in intel_fbdev.c */
-#ifdef CONFIG_DRM_FBDEV_EMULATION
-extern int intel_fbdev_init(struct drm_device *dev);
-extern void intel_fbdev_initial_config_async(struct drm_device *dev);
-extern void intel_fbdev_unregister(struct drm_i915_private *dev_priv);
-extern void intel_fbdev_fini(struct drm_i915_private *dev_priv);
-extern void intel_fbdev_set_suspend(struct drm_device *dev, int state, bool synchronous);
-extern void intel_fbdev_output_poll_changed(struct drm_device *dev);
-extern void intel_fbdev_restore_mode(struct drm_device *dev);
-#else
-static inline int intel_fbdev_init(struct drm_device *dev)
-{
- return 0;
-}
-
-static inline void intel_fbdev_initial_config_async(struct drm_device *dev)
-{
-}
-
-static inline void intel_fbdev_unregister(struct drm_i915_private *dev_priv)
-{
-}
-
-static inline void intel_fbdev_fini(struct drm_i915_private *dev_priv)
-{
-}
-
-static inline void intel_fbdev_set_suspend(struct drm_device *dev, int state, bool synchronous)
-{
-}
-
-static inline void intel_fbdev_output_poll_changed(struct drm_device *dev)
-{
-}
-
-static inline void intel_fbdev_restore_mode(struct drm_device *dev)
-{
-}
-#endif
-
-/* intel_fbc.c */
-void intel_fbc_choose_crtc(struct drm_i915_private *dev_priv,
- struct intel_atomic_state *state);
-bool intel_fbc_is_active(struct drm_i915_private *dev_priv);
-void intel_fbc_pre_update(struct intel_crtc *crtc,
- struct intel_crtc_state *crtc_state,
- struct intel_plane_state *plane_state);
-void intel_fbc_post_update(struct intel_crtc *crtc);
-void intel_fbc_init(struct drm_i915_private *dev_priv);
-void intel_fbc_init_pipe_state(struct drm_i915_private *dev_priv);
-void intel_fbc_enable(struct intel_crtc *crtc,
- struct intel_crtc_state *crtc_state,
- struct intel_plane_state *plane_state);
-void intel_fbc_disable(struct intel_crtc *crtc);
-void intel_fbc_global_disable(struct drm_i915_private *dev_priv);
-void intel_fbc_invalidate(struct drm_i915_private *dev_priv,
- unsigned int frontbuffer_bits,
- enum fb_op_origin origin);
-void intel_fbc_flush(struct drm_i915_private *dev_priv,
- unsigned int frontbuffer_bits, enum fb_op_origin origin);
-void intel_fbc_cleanup_cfb(struct drm_i915_private *dev_priv);
-void intel_fbc_handle_fifo_underrun_irq(struct drm_i915_private *dev_priv);
-int intel_fbc_reset_underrun(struct drm_i915_private *dev_priv);
-
-/* intel_hdmi.c */
-void intel_hdmi_init(struct drm_i915_private *dev_priv, i915_reg_t hdmi_reg,
- enum port port);
-void intel_hdmi_init_connector(struct intel_digital_port *intel_dig_port,
- struct intel_connector *intel_connector);
-struct intel_hdmi *enc_to_intel_hdmi(struct drm_encoder *encoder);
-int intel_hdmi_compute_config(struct intel_encoder *encoder,
- struct intel_crtc_state *pipe_config,
- struct drm_connector_state *conn_state);
-bool intel_hdmi_handle_sink_scrambling(struct intel_encoder *encoder,
- struct drm_connector *connector,
- bool high_tmds_clock_ratio,
- bool scrambling);
-void intel_dp_dual_mode_set_tmds_output(struct intel_hdmi *hdmi, bool enable);
-void intel_infoframe_init(struct intel_digital_port *intel_dig_port);
-
-/* intel_lvds.c */
-bool intel_lvds_port_enabled(struct drm_i915_private *dev_priv,
- i915_reg_t lvds_reg, enum pipe *pipe);
-void intel_lvds_init(struct drm_i915_private *dev_priv);
-struct intel_encoder *intel_get_lvds_encoder(struct drm_device *dev);
-bool intel_is_dual_link_lvds(struct drm_device *dev);
-
/* intel_overlay.c */
void intel_overlay_setup(struct drm_i915_private *dev_priv);
void intel_overlay_cleanup(struct drm_i915_private *dev_priv);
struct drm_file *file_priv);
void intel_overlay_reset(struct drm_i915_private *dev_priv);
-
-/* intel_panel.c */
-int intel_panel_init(struct intel_panel *panel,
- struct drm_display_mode *fixed_mode,
- struct drm_display_mode *downclock_mode);
-void intel_panel_fini(struct intel_panel *panel);
-void intel_fixed_panel_mode(const struct drm_display_mode *fixed_mode,
- struct drm_display_mode *adjusted_mode);
-void intel_pch_panel_fitting(struct intel_crtc *crtc,
- struct intel_crtc_state *pipe_config,
- int fitting_mode);
-void intel_gmch_panel_fitting(struct intel_crtc *crtc,
- struct intel_crtc_state *pipe_config,
- int fitting_mode);
-void intel_panel_set_backlight_acpi(const struct drm_connector_state *conn_state,
- u32 level, u32 max);
-int intel_panel_setup_backlight(struct drm_connector *connector,
- enum pipe pipe);
-void intel_panel_enable_backlight(const struct intel_crtc_state *crtc_state,
- const struct drm_connector_state *conn_state);
-void intel_panel_update_backlight(struct intel_encoder *encoder,
- const struct intel_crtc_state *crtc_state,
- const struct drm_connector_state *conn_state);
-void intel_panel_disable_backlight(const struct drm_connector_state *old_conn_state);
-extern struct drm_display_mode *intel_find_panel_downclock(
- struct drm_i915_private *dev_priv,
- struct drm_display_mode *fixed_mode,
- struct drm_connector *connector);
-
-#if IS_ENABLED(CONFIG_BACKLIGHT_CLASS_DEVICE)
-int intel_backlight_device_register(struct intel_connector *connector);
-void intel_backlight_device_unregister(struct intel_connector *connector);
-#else /* CONFIG_BACKLIGHT_CLASS_DEVICE */
-static inline int intel_backlight_device_register(struct intel_connector *connector)
-{
- return 0;
-}
-static inline void intel_backlight_device_unregister(struct intel_connector *connector)
-{
-}
-#endif /* CONFIG_BACKLIGHT_CLASS_DEVICE */
-
-/* intel_hdcp.c */
-void intel_hdcp_atomic_check(struct drm_connector *connector,
- struct drm_connector_state *old_state,
- struct drm_connector_state *new_state);
-int intel_hdcp_init(struct intel_connector *connector,
- const struct intel_hdcp_shim *hdcp_shim);
-int intel_hdcp_enable(struct intel_connector *connector);
-int intel_hdcp_disable(struct intel_connector *connector);
-int intel_hdcp_check_link(struct intel_connector *connector);
-bool is_hdcp_supported(struct drm_i915_private *dev_priv, enum port port);
-bool intel_hdcp_capable(struct intel_connector *connector);
-
-/* intel_psr.c */
-#define CAN_PSR(dev_priv) (HAS_PSR(dev_priv) && dev_priv->psr.sink_support)
-void intel_psr_init_dpcd(struct intel_dp *intel_dp);
-void intel_psr_enable(struct intel_dp *intel_dp,
- const struct intel_crtc_state *crtc_state);
-void intel_psr_disable(struct intel_dp *intel_dp,
- const struct intel_crtc_state *old_crtc_state);
-int intel_psr_set_debugfs_mode(struct drm_i915_private *dev_priv,
- struct drm_modeset_acquire_ctx *ctx,
- u64 value);
-void intel_psr_invalidate(struct drm_i915_private *dev_priv,
- unsigned frontbuffer_bits,
- enum fb_op_origin origin);
-void intel_psr_flush(struct drm_i915_private *dev_priv,
- unsigned frontbuffer_bits,
- enum fb_op_origin origin);
-void intel_psr_init(struct drm_i915_private *dev_priv);
-void intel_psr_compute_config(struct intel_dp *intel_dp,
- struct intel_crtc_state *crtc_state);
-void intel_psr_irq_control(struct drm_i915_private *dev_priv, u32 debug);
-void intel_psr_irq_handler(struct drm_i915_private *dev_priv, u32 psr_iir);
-void intel_psr_short_pulse(struct intel_dp *intel_dp);
-int intel_psr_wait_for_idle(const struct intel_crtc_state *new_crtc_state,
- u32 *out_value);
-bool intel_psr_enabled(struct intel_dp *intel_dp);
-
/* intel_quirks.c */
void intel_init_quirks(struct drm_i915_private *dev_priv);
u8 req_slices);
static inline void
-assert_rpm_device_not_suspended(struct drm_i915_private *i915)
+assert_rpm_device_not_suspended(struct i915_runtime_pm *rpm)
{
- WARN_ONCE(i915->runtime_pm.suspended,
+ WARN_ONCE(rpm->suspended,
"Device suspended during HW access\n");
}
static inline void
-assert_rpm_wakelock_held(struct drm_i915_private *i915)
+__assert_rpm_wakelock_held(struct i915_runtime_pm *rpm)
{
- assert_rpm_device_not_suspended(i915);
- WARN_ONCE(!atomic_read(&i915->runtime_pm.wakeref_count),
+ assert_rpm_device_not_suspended(rpm);
+ WARN_ONCE(!atomic_read(&rpm->wakeref_count),
"RPM wakelock ref not held during HW access");
}
+static inline void
+assert_rpm_wakelock_held(struct drm_i915_private *i915)
+{
+ __assert_rpm_wakelock_held(&i915->runtime_pm);
+}
+
/**
* disable_rpm_wakeref_asserts - disable the RPM assert checks
* @i915: i915 device instance
bool chv_phy_powergate_ch(struct drm_i915_private *dev_priv, enum dpio_phy phy,
enum dpio_channel ch, bool override);
-
-/* intel_pm.c */
-void intel_init_clock_gating(struct drm_i915_private *dev_priv);
-void intel_suspend_hw(struct drm_i915_private *dev_priv);
-int ilk_wm_max_level(const struct drm_i915_private *dev_priv);
-void intel_update_watermarks(struct intel_crtc *crtc);
-void intel_init_pm(struct drm_i915_private *dev_priv);
-void intel_init_clock_gating_hooks(struct drm_i915_private *dev_priv);
-void intel_pm_setup(struct drm_i915_private *dev_priv);
-void intel_gpu_ips_init(struct drm_i915_private *dev_priv);
-void intel_gpu_ips_teardown(void);
-void intel_init_gt_powersave(struct drm_i915_private *dev_priv);
-void intel_cleanup_gt_powersave(struct drm_i915_private *dev_priv);
-void intel_sanitize_gt_powersave(struct drm_i915_private *dev_priv);
-void intel_enable_gt_powersave(struct drm_i915_private *dev_priv);
-void intel_disable_gt_powersave(struct drm_i915_private *dev_priv);
-void intel_suspend_gt_powersave(struct drm_i915_private *dev_priv);
-void gen6_rps_busy(struct drm_i915_private *dev_priv);
-void gen6_rps_reset_ei(struct drm_i915_private *dev_priv);
-void gen6_rps_idle(struct drm_i915_private *dev_priv);
-void gen6_rps_boost(struct i915_request *rq, struct intel_rps_client *rps);
-void g4x_wm_get_hw_state(struct drm_i915_private *dev_priv);
-void vlv_wm_get_hw_state(struct drm_i915_private *dev_priv);
-void ilk_wm_get_hw_state(struct drm_i915_private *dev_priv);
-void skl_wm_get_hw_state(struct drm_i915_private *dev_priv);
-void skl_pipe_ddb_get_hw_state(struct intel_crtc *crtc,
- struct skl_ddb_entry *ddb_y,
- struct skl_ddb_entry *ddb_uv);
-void skl_ddb_get_hw_state(struct drm_i915_private *dev_priv,
- struct skl_ddb_allocation *ddb /* out */);
-void skl_pipe_wm_get_hw_state(struct intel_crtc *crtc,
- struct skl_pipe_wm *out);
-void g4x_wm_sanitize(struct drm_i915_private *dev_priv);
-void vlv_wm_sanitize(struct drm_i915_private *dev_priv);
-bool intel_can_enable_sagv(struct drm_atomic_state *state);
-int intel_enable_sagv(struct drm_i915_private *dev_priv);
-int intel_disable_sagv(struct drm_i915_private *dev_priv);
-bool skl_wm_level_equals(const struct skl_wm_level *l1,
- const struct skl_wm_level *l2);
-bool skl_ddb_allocation_overlaps(const struct skl_ddb_entry *ddb,
- const struct skl_ddb_entry entries[],
- int num_entries, int ignore_idx);
-void skl_write_plane_wm(struct intel_plane *plane,
- const struct intel_crtc_state *crtc_state);
-void skl_write_cursor_wm(struct intel_plane *plane,
- const struct intel_crtc_state *crtc_state);
-bool ilk_disable_lp_wm(struct drm_device *dev);
-int skl_check_pipe_max_pixel_rate(struct intel_crtc *intel_crtc,
- struct intel_crtc_state *cstate);
-void intel_init_ipc(struct drm_i915_private *dev_priv);
-void intel_enable_ipc(struct drm_i915_private *dev_priv);
-
-/* intel_sdvo.c */
-bool intel_sdvo_port_enabled(struct drm_i915_private *dev_priv,
- i915_reg_t sdvo_reg, enum pipe *pipe);
-bool intel_sdvo_init(struct drm_i915_private *dev_priv,
- i915_reg_t reg, enum port port);
-
-
-/* intel_sprite.c */
-int intel_usecs_to_scanlines(const struct drm_display_mode *adjusted_mode,
- int usecs);
-struct intel_plane *intel_sprite_plane_create(struct drm_i915_private *dev_priv,
- enum pipe pipe, int plane);
-int intel_sprite_set_colorkey_ioctl(struct drm_device *dev, void *data,
- struct drm_file *file_priv);
-void intel_pipe_update_start(const struct intel_crtc_state *new_crtc_state);
-void intel_pipe_update_end(struct intel_crtc_state *new_crtc_state);
-int intel_plane_check_stride(const struct intel_plane_state *plane_state);
-int intel_plane_check_src_coordinates(struct intel_plane_state *plane_state);
-int chv_plane_check_rotation(const struct intel_plane_state *plane_state);
-struct intel_plane *
-skl_universal_plane_create(struct drm_i915_private *dev_priv,
- enum pipe pipe, enum plane_id plane_id);
-
-static inline bool icl_is_nv12_y_plane(enum plane_id id)
-{
- /* Don't need to do a gen check, these planes are only available on gen11 */
- if (id == PLANE_SPRITE4 || id == PLANE_SPRITE5)
- return true;
-
- return false;
-}
-
-static inline bool icl_is_hdr_plane(struct intel_plane *plane)
-{
- if (INTEL_GEN(to_i915(plane->base.dev)) < 11)
- return false;
-
- return plane->id < PLANE_SPRITE2;
-}
-
-/* intel_tv.c */
-void intel_tv_init(struct drm_i915_private *dev_priv);
-
/* intel_atomic.c */
int intel_digital_connector_atomic_get_property(struct drm_connector *connector,
const struct drm_connector_state *state,
struct intel_crtc *intel_crtc,
struct intel_crtc_state *crtc_state);
-/* intel_atomic_plane.c */
-struct intel_plane *intel_plane_alloc(void);
-void intel_plane_free(struct intel_plane *plane);
-struct drm_plane_state *intel_plane_duplicate_state(struct drm_plane *plane);
-void intel_plane_destroy_state(struct drm_plane *plane,
- struct drm_plane_state *state);
-extern const struct drm_plane_helper_funcs intel_plane_helper_funcs;
-void skl_update_planes_on_crtc(struct intel_atomic_state *state,
- struct intel_crtc *crtc);
-void i9xx_update_planes_on_crtc(struct intel_atomic_state *state,
- struct intel_crtc *crtc);
-int intel_plane_atomic_check_with_state(const struct intel_crtc_state *old_crtc_state,
- struct intel_crtc_state *crtc_state,
- const struct intel_plane_state *old_plane_state,
- struct intel_plane_state *intel_state);
-
-/* intel_color.c */
-void intel_color_init(struct intel_crtc *crtc);
-int intel_color_check(struct intel_crtc_state *crtc_state);
-void intel_color_commit(const struct intel_crtc_state *crtc_state);
-void intel_color_load_luts(const struct intel_crtc_state *crtc_state);
-
-/* intel_lspcon.c */
-bool lspcon_init(struct intel_digital_port *intel_dig_port);
-void lspcon_resume(struct intel_lspcon *lspcon);
-void lspcon_wait_pcon_mode(struct intel_lspcon *lspcon);
-void lspcon_write_infoframe(struct intel_encoder *encoder,
- const struct intel_crtc_state *crtc_state,
- unsigned int type,
- const void *buf, ssize_t len);
-void lspcon_set_infoframes(struct intel_encoder *encoder,
- bool enable,
- const struct intel_crtc_state *crtc_state,
- const struct drm_connector_state *conn_state);
-bool lspcon_infoframe_enabled(struct intel_encoder *encoder,
- const struct intel_crtc_state *pipe_config);
-void lspcon_ycbcr420_config(struct drm_connector *connector,
- struct intel_crtc_state *crtc_state);
-
-/* intel_pipe_crc.c */
-#ifdef CONFIG_DEBUG_FS
-int intel_crtc_set_crc_source(struct drm_crtc *crtc, const char *source_name);
-int intel_crtc_verify_crc_source(struct drm_crtc *crtc,
- const char *source_name, size_t *values_cnt);
-const char *const *intel_crtc_get_crc_sources(struct drm_crtc *crtc,
- size_t *count);
-void intel_crtc_disable_pipe_crc(struct intel_crtc *crtc);
-void intel_crtc_enable_pipe_crc(struct intel_crtc *crtc);
-#else
-#define intel_crtc_set_crc_source NULL
-#define intel_crtc_verify_crc_source NULL
-#define intel_crtc_get_crc_sources NULL
-static inline void intel_crtc_disable_pipe_crc(struct intel_crtc *crtc)
-{
-}
-
-static inline void intel_crtc_enable_pipe_crc(struct intel_crtc *crtc)
-{
-}
-#endif
#endif /* __INTEL_DRV_H__ */
/* intel_dsi_vbt.c */
bool intel_dsi_vbt_init(struct intel_dsi *intel_dsi, u16 panel_id);
-int intel_dsi_vbt_get_modes(struct intel_dsi *intel_dsi);
void intel_dsi_vbt_exec_sequence(struct intel_dsi *intel_dsi,
enum mipi_seq seq_id);
void intel_dsi_msleep(struct intel_dsi *intel_dsi, int msec);
break;
}
- if (!IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) < 11)
vlv_dsi_wait_for_fifo_empty(intel_dsi, port);
out:
/* pull up/down */
value = *data++ & 1;
- if (IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 11)
icl_exec_gpio(dev_priv, gpio_source, gpio_index, value);
else if (IS_VALLEYVIEW(dev_priv))
vlv_exec_gpio(dev_priv, gpio_source, gpio_number, value);
msleep(msec);
}
-int intel_dsi_vbt_get_modes(struct intel_dsi *intel_dsi)
-{
- struct intel_connector *connector = intel_dsi->attached_connector;
- struct drm_device *dev = intel_dsi->base.base.dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct drm_display_mode *mode;
-
- mode = drm_mode_duplicate(dev, dev_priv->vbt.lfp_lvds_vbt_mode);
- if (!mode)
- return 0;
-
- mode->type |= DRM_MODE_TYPE_PREFERRED;
-
- drm_mode_probed_add(&connector->base, mode);
-
- return 1;
-}
-
#define ICL_PREPARE_CNT_MAX 0x7
#define ICL_CLK_ZERO_CNT_MAX 0xf
#define ICL_TRAIL_CNT_MAX 0x7
intel_dsi->burst_mode_ratio = burst_mode_ratio;
- if (IS_ICELAKE(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 11)
icl_dphy_param_init(intel_dsi);
else
vlv_dphy_param_init(intel_dsi);
* Authors:
* Eric Anholt <eric@anholt.net>
*/
+
#include <linux/i2c.h>
#include <linux/slab.h>
+
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
-#include "intel_drv.h"
#include <drm/i915_drm.h>
-#include "i915_drv.h"
+
#include "dvo.h"
+#include "i915_drv.h"
+#include "intel_connector.h"
+#include "intel_drv.h"
+#include "intel_dvo.h"
+#include "intel_panel.h"
#define SIL164_ADDR 0x38
#define CH7xxx_ADDR 0x76
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_DVO_H__
+#define __INTEL_DVO_H__
+
+struct drm_i915_private;
+
+void intel_dvo_init(struct drm_i915_private *dev_priv);
+
+#endif /* __INTEL_DVO_H__ */
#define MAX_MMIO_BASES 3
struct engine_info {
unsigned int hw_id;
- unsigned int uabi_id;
u8 class;
u8 instance;
/* mmio bases table *must* be sorted in reverse gen order */
};
static const struct engine_info intel_engines[] = {
- [RCS] = {
- .hw_id = RCS_HW,
- .uabi_id = I915_EXEC_RENDER,
+ [RCS0] = {
+ .hw_id = RCS0_HW,
.class = RENDER_CLASS,
.instance = 0,
.mmio_bases = {
{ .gen = 1, .base = RENDER_RING_BASE }
},
},
- [BCS] = {
- .hw_id = BCS_HW,
- .uabi_id = I915_EXEC_BLT,
+ [BCS0] = {
+ .hw_id = BCS0_HW,
.class = COPY_ENGINE_CLASS,
.instance = 0,
.mmio_bases = {
{ .gen = 6, .base = BLT_RING_BASE }
},
},
- [VCS] = {
- .hw_id = VCS_HW,
- .uabi_id = I915_EXEC_BSD,
+ [VCS0] = {
+ .hw_id = VCS0_HW,
.class = VIDEO_DECODE_CLASS,
.instance = 0,
.mmio_bases = {
{ .gen = 4, .base = BSD_RING_BASE }
},
},
- [VCS2] = {
- .hw_id = VCS2_HW,
- .uabi_id = I915_EXEC_BSD,
+ [VCS1] = {
+ .hw_id = VCS1_HW,
.class = VIDEO_DECODE_CLASS,
.instance = 1,
.mmio_bases = {
{ .gen = 8, .base = GEN8_BSD2_RING_BASE }
},
},
- [VCS3] = {
- .hw_id = VCS3_HW,
- .uabi_id = I915_EXEC_BSD,
+ [VCS2] = {
+ .hw_id = VCS2_HW,
.class = VIDEO_DECODE_CLASS,
.instance = 2,
.mmio_bases = {
{ .gen = 11, .base = GEN11_BSD3_RING_BASE }
},
},
- [VCS4] = {
- .hw_id = VCS4_HW,
- .uabi_id = I915_EXEC_BSD,
+ [VCS3] = {
+ .hw_id = VCS3_HW,
.class = VIDEO_DECODE_CLASS,
.instance = 3,
.mmio_bases = {
{ .gen = 11, .base = GEN11_BSD4_RING_BASE }
},
},
- [VECS] = {
- .hw_id = VECS_HW,
- .uabi_id = I915_EXEC_VEBOX,
+ [VECS0] = {
+ .hw_id = VECS0_HW,
.class = VIDEO_ENHANCEMENT_CLASS,
.instance = 0,
.mmio_bases = {
{ .gen = 7, .base = VEBOX_RING_BASE }
},
},
- [VECS2] = {
- .hw_id = VECS2_HW,
- .uabi_id = I915_EXEC_VEBOX,
+ [VECS1] = {
+ .hw_id = VECS1_HW,
.class = VIDEO_ENHANCEMENT_CLASS,
.instance = 1,
.mmio_bases = {
void intel_engine_set_hwsp_writemask(struct intel_engine_cs *engine, u32 mask)
{
- struct drm_i915_private *dev_priv = engine->i915;
- i915_reg_t hwstam;
-
/*
* Though they added more rings on g4x/ilk, they did not add
* per-engine HWSTAM until gen6.
*/
- if (INTEL_GEN(dev_priv) < 6 && engine->class != RENDER_CLASS)
+ if (INTEL_GEN(engine->i915) < 6 && engine->class != RENDER_CLASS)
return;
- hwstam = RING_HWSTAM(engine->mmio_base);
- if (INTEL_GEN(dev_priv) >= 3)
- I915_WRITE(hwstam, mask);
+ if (INTEL_GEN(engine->i915) >= 3)
+ ENGINE_WRITE(engine, RING_HWSTAM, mask);
else
- I915_WRITE16(hwstam, mask);
+ ENGINE_WRITE16(engine, RING_HWSTAM, mask);
}
static void intel_engine_sanitize_mmio(struct intel_engine_cs *engine)
if (!engine)
return -ENOMEM;
+ BUILD_BUG_ON(BITS_PER_TYPE(engine->mask) < I915_NUM_ENGINES);
+
engine->id = id;
+ engine->mask = BIT(id);
engine->i915 = dev_priv;
+ engine->uncore = &dev_priv->uncore;
__sprint_engine_name(engine->name, info);
engine->hw_id = engine->guc_id = info->hw_id;
engine->mmio_base = __engine_mmio_base(dev_priv, info->mmio_bases);
engine->class = info->class;
engine->instance = info->instance;
- engine->uabi_id = info->uabi_id;
engine->uabi_class = intel_engine_classes[info->class].uabi_class;
engine->context_size = __intel_engine_context_size(dev_priv,
int intel_engines_init_mmio(struct drm_i915_private *dev_priv)
{
struct intel_device_info *device_info = mkwrite_device_info(dev_priv);
- const unsigned int ring_mask = INTEL_INFO(dev_priv)->ring_mask;
+ const unsigned int engine_mask = INTEL_INFO(dev_priv)->engine_mask;
struct intel_engine_cs *engine;
enum intel_engine_id id;
unsigned int mask = 0;
unsigned int i;
int err;
- WARN_ON(ring_mask == 0);
- WARN_ON(ring_mask &
+ WARN_ON(engine_mask == 0);
+ WARN_ON(engine_mask &
GENMASK(BITS_PER_TYPE(mask) - 1, I915_NUM_ENGINES));
if (i915_inject_load_failure())
if (err)
goto cleanup;
- mask |= ENGINE_MASK(i);
+ mask |= BIT(i);
}
/*
* are added to the driver by a warning and disabling the forgotten
* engines.
*/
- if (WARN_ON(mask != ring_mask))
- device_info->ring_mask = mask;
+ if (WARN_ON(mask != engine_mask))
+ device_info->engine_mask = mask;
/* We always presume we have at least RCS available for later probing */
- if (WARN_ON(!HAS_ENGINE(dev_priv, RCS))) {
+ if (WARN_ON(!HAS_ENGINE(dev_priv, RCS0))) {
err = -ENODEV;
goto cleanup;
}
- RUNTIME_INFO(dev_priv)->num_rings = hweight32(mask);
+ RUNTIME_INFO(dev_priv)->num_engines = hweight32(mask);
i915_check_and_clear_faults(dev_priv);
return err;
}
-void intel_engine_write_global_seqno(struct intel_engine_cs *engine, u32 seqno)
-{
- intel_write_status_page(engine, I915_GEM_HWS_INDEX, seqno);
- GEM_BUG_ON(intel_engine_get_seqno(engine) != seqno);
-}
-
static void intel_engine_init_batch_pool(struct intel_engine_cs *engine)
{
i915_gem_batch_pool_init(&engine->batch_pool, engine);
return PTR_ERR(obj);
}
- ret = i915_gem_object_set_cache_level(obj, I915_CACHE_LLC);
- if (ret)
- goto err;
+ i915_gem_object_set_cache_coherency(obj, I915_CACHE_LLC);
vma = i915_vma_instance(obj, &engine->i915->ggtt.vm, NULL);
if (IS_ERR(vma)) {
err = i915_timeline_init(engine->i915,
&engine->timeline,
- engine->name,
engine->status_page.vma);
if (err)
goto err_hwsp;
return err;
}
-static void __intel_context_unpin(struct i915_gem_context *ctx,
- struct intel_engine_cs *engine)
+void intel_engines_set_scheduler_caps(struct drm_i915_private *i915)
{
- intel_context_unpin(to_intel_context(ctx, engine));
+ static const struct {
+ u8 engine;
+ u8 sched;
+ } map[] = {
+#define MAP(x, y) { ilog2(I915_ENGINE_HAS_##x), ilog2(I915_SCHEDULER_CAP_##y) }
+ MAP(PREEMPTION, PREEMPTION),
+ MAP(SEMAPHORES, SEMAPHORES),
+#undef MAP
+ };
+ struct intel_engine_cs *engine;
+ enum intel_engine_id id;
+ u32 enabled, disabled;
+
+ enabled = 0;
+ disabled = 0;
+ for_each_engine(engine, i915, id) { /* all engines must agree! */
+ int i;
+
+ if (engine->schedule)
+ enabled |= (I915_SCHEDULER_CAP_ENABLED |
+ I915_SCHEDULER_CAP_PRIORITY);
+ else
+ disabled |= (I915_SCHEDULER_CAP_ENABLED |
+ I915_SCHEDULER_CAP_PRIORITY);
+
+ for (i = 0; i < ARRAY_SIZE(map); i++) {
+ if (engine->flags & BIT(map[i].engine))
+ enabled |= BIT(map[i].sched);
+ else
+ disabled |= BIT(map[i].sched);
+ }
+ }
+
+ i915->caps.scheduler = enabled & ~disabled;
+ if (!(i915->caps.scheduler & I915_SCHEDULER_CAP_ENABLED))
+ i915->caps.scheduler = 0;
}
struct measure_breadcrumb {
return -ENOMEM;
if (i915_timeline_init(engine->i915,
- &frame->timeline, "measure",
+ &frame->timeline,
engine->status_page.vma))
goto out_frame;
return dw;
}
+static int pin_context(struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine,
+ struct intel_context **out)
+{
+ struct intel_context *ce;
+
+ ce = intel_context_pin(ctx, engine);
+ if (IS_ERR(ce))
+ return PTR_ERR(ce);
+
+ *out = ce;
+ return 0;
+}
+
/**
* intel_engines_init_common - initialize cengine state which might require hw access
* @engine: Engine to initialize.
int intel_engine_init_common(struct intel_engine_cs *engine)
{
struct drm_i915_private *i915 = engine->i915;
- struct intel_context *ce;
int ret;
- engine->set_default_submission(engine);
-
/* We may need to do things with the shrinker which
* require us to immediately switch back to the default
* context. This can cause a problem as pinning the
* be available. To avoid this we always pin the default
* context.
*/
- ce = intel_context_pin(i915->kernel_context, engine);
- if (IS_ERR(ce))
- return PTR_ERR(ce);
+ ret = pin_context(i915->kernel_context, engine,
+ &engine->kernel_context);
+ if (ret)
+ return ret;
/*
* Similarly the preempt context must always be available so that
- * we can interrupt the engine at any time.
+ * we can interrupt the engine at any time. However, as preemption
+ * is optional, we allow it to fail.
*/
- if (i915->preempt_context) {
- ce = intel_context_pin(i915->preempt_context, engine);
- if (IS_ERR(ce)) {
- ret = PTR_ERR(ce);
- goto err_unpin_kernel;
- }
- }
+ if (i915->preempt_context)
+ pin_context(i915->preempt_context, engine,
+ &engine->preempt_context);
ret = measure_breadcrumb_dw(engine);
if (ret < 0)
- goto err_unpin_preempt;
+ goto err_unpin;
engine->emit_fini_breadcrumb_dw = ret;
- return 0;
+ engine->set_default_submission(engine);
-err_unpin_preempt:
- if (i915->preempt_context)
- __intel_context_unpin(i915->preempt_context, engine);
+ return 0;
-err_unpin_kernel:
- __intel_context_unpin(i915->kernel_context, engine);
+err_unpin:
+ if (engine->preempt_context)
+ intel_context_unpin(engine->preempt_context);
+ intel_context_unpin(engine->kernel_context);
return ret;
}
+void intel_gt_resume(struct drm_i915_private *i915)
+{
+ struct intel_engine_cs *engine;
+ enum intel_engine_id id;
+
+ /*
+ * After resume, we may need to poke into the pinned kernel
+ * contexts to paper over any damage caused by the sudden suspend.
+ * Only the kernel contexts should remain pinned over suspend,
+ * allowing us to fixup the user contexts on their first pin.
+ */
+ for_each_engine(engine, i915, id) {
+ struct intel_context *ce;
+
+ ce = engine->kernel_context;
+ if (ce)
+ ce->ops->reset(ce);
+
+ ce = engine->preempt_context;
+ if (ce)
+ ce->ops->reset(ce);
+ }
+}
+
/**
* intel_engines_cleanup_common - cleans up the engine state created by
* the common initiailizers.
*/
void intel_engine_cleanup_common(struct intel_engine_cs *engine)
{
- struct drm_i915_private *i915 = engine->i915;
-
cleanup_status_page(engine);
intel_engine_fini_breadcrumbs(engine);
if (engine->default_state)
i915_gem_object_put(engine->default_state);
- if (i915->preempt_context)
- __intel_context_unpin(i915->preempt_context, engine);
- __intel_context_unpin(i915->kernel_context, engine);
+ if (engine->preempt_context)
+ intel_context_unpin(engine->preempt_context);
+ intel_context_unpin(engine->kernel_context);
i915_timeline_fini(&engine->timeline);
u64 intel_engine_get_active_head(const struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
+ struct drm_i915_private *i915 = engine->i915;
+
u64 acthd;
- if (INTEL_GEN(dev_priv) >= 8)
- acthd = I915_READ64_2x32(RING_ACTHD(engine->mmio_base),
- RING_ACTHD_UDW(engine->mmio_base));
- else if (INTEL_GEN(dev_priv) >= 4)
- acthd = I915_READ(RING_ACTHD(engine->mmio_base));
+ if (INTEL_GEN(i915) >= 8)
+ acthd = ENGINE_READ64(engine, RING_ACTHD, RING_ACTHD_UDW);
+ else if (INTEL_GEN(i915) >= 4)
+ acthd = ENGINE_READ(engine, RING_ACTHD);
else
- acthd = I915_READ(ACTHD);
+ acthd = ENGINE_READ(engine, ACTHD);
return acthd;
}
u64 intel_engine_get_last_batch_head(const struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
u64 bbaddr;
- if (INTEL_GEN(dev_priv) >= 8)
- bbaddr = I915_READ64_2x32(RING_BBADDR(engine->mmio_base),
- RING_BBADDR_UDW(engine->mmio_base));
+ if (INTEL_GEN(engine->i915) >= 8)
+ bbaddr = ENGINE_READ64(engine, RING_BBADDR, RING_BBADDR_UDW);
else
- bbaddr = I915_READ(RING_BBADDR(engine->mmio_base));
+ bbaddr = ENGINE_READ(engine, RING_BBADDR);
return bbaddr;
}
int intel_engine_stop_cs(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
+ struct intel_uncore *uncore = engine->uncore;
const u32 base = engine->mmio_base;
const i915_reg_t mode = RING_MI_MODE(base);
int err;
- if (INTEL_GEN(dev_priv) < 3)
+ if (INTEL_GEN(engine->i915) < 3)
return -ENODEV;
GEM_TRACE("%s\n", engine->name);
- I915_WRITE_FW(mode, _MASKED_BIT_ENABLE(STOP_RING));
+ intel_uncore_write_fw(uncore, mode, _MASKED_BIT_ENABLE(STOP_RING));
err = 0;
- if (__intel_wait_for_register_fw(dev_priv,
+ if (__intel_wait_for_register_fw(uncore,
mode, MODE_IDLE, MODE_IDLE,
1000, 0,
NULL)) {
}
/* A final mmio read to let GPU writes be hopefully flushed to memory */
- POSTING_READ_FW(mode);
+ intel_uncore_posting_read_fw(uncore, mode);
return err;
}
void intel_engine_cancel_stop_cs(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
-
GEM_TRACE("%s\n", engine->name);
- I915_WRITE_FW(RING_MI_MODE(engine->mmio_base),
- _MASKED_BIT_DISABLE(STOP_RING));
+ ENGINE_WRITE_FW(engine, RING_MI_MODE, _MASKED_BIT_DISABLE(STOP_RING));
}
const char *i915_cache_level_str(struct drm_i915_private *i915, int type)
read_subslice_reg(struct drm_i915_private *dev_priv, int slice,
int subslice, i915_reg_t reg)
{
+ struct intel_uncore *uncore = &dev_priv->uncore;
u32 mcr_slice_subslice_mask;
u32 mcr_slice_subslice_select;
u32 default_mcr_s_ss_select;
default_mcr_s_ss_select = intel_calculate_mcr_s_ss_select(dev_priv);
- fw_domains = intel_uncore_forcewake_for_reg(dev_priv, reg,
+ fw_domains = intel_uncore_forcewake_for_reg(uncore, reg,
FW_REG_READ);
- fw_domains |= intel_uncore_forcewake_for_reg(dev_priv,
+ fw_domains |= intel_uncore_forcewake_for_reg(uncore,
GEN8_MCR_SELECTOR,
FW_REG_READ | FW_REG_WRITE);
- spin_lock_irq(&dev_priv->uncore.lock);
- intel_uncore_forcewake_get__locked(dev_priv, fw_domains);
+ spin_lock_irq(&uncore->lock);
+ intel_uncore_forcewake_get__locked(uncore, fw_domains);
- mcr = I915_READ_FW(GEN8_MCR_SELECTOR);
+ mcr = intel_uncore_read_fw(uncore, GEN8_MCR_SELECTOR);
WARN_ON_ONCE((mcr & mcr_slice_subslice_mask) !=
default_mcr_s_ss_select);
mcr &= ~mcr_slice_subslice_mask;
mcr |= mcr_slice_subslice_select;
- I915_WRITE_FW(GEN8_MCR_SELECTOR, mcr);
+ intel_uncore_write_fw(uncore, GEN8_MCR_SELECTOR, mcr);
- ret = I915_READ_FW(reg);
+ ret = intel_uncore_read_fw(uncore, reg);
mcr &= ~mcr_slice_subslice_mask;
mcr |= default_mcr_s_ss_select;
- I915_WRITE_FW(GEN8_MCR_SELECTOR, mcr);
+ intel_uncore_write_fw(uncore, GEN8_MCR_SELECTOR, mcr);
- intel_uncore_forcewake_put__locked(dev_priv, fw_domains);
- spin_unlock_irq(&dev_priv->uncore.lock);
+ intel_uncore_forcewake_put__locked(uncore, fw_domains);
+ spin_unlock_irq(&uncore->lock);
return ret;
}
struct intel_instdone *instdone)
{
struct drm_i915_private *dev_priv = engine->i915;
+ struct intel_uncore *uncore = engine->uncore;
u32 mmio_base = engine->mmio_base;
int slice;
int subslice;
switch (INTEL_GEN(dev_priv)) {
default:
- instdone->instdone = I915_READ(RING_INSTDONE(mmio_base));
+ instdone->instdone =
+ intel_uncore_read(uncore, RING_INSTDONE(mmio_base));
- if (engine->id != RCS)
+ if (engine->id != RCS0)
break;
- instdone->slice_common = I915_READ(GEN7_SC_INSTDONE);
+ instdone->slice_common =
+ intel_uncore_read(uncore, GEN7_SC_INSTDONE);
for_each_instdone_slice_subslice(dev_priv, slice, subslice) {
instdone->sampler[slice][subslice] =
read_subslice_reg(dev_priv, slice, subslice,
}
break;
case 7:
- instdone->instdone = I915_READ(RING_INSTDONE(mmio_base));
+ instdone->instdone =
+ intel_uncore_read(uncore, RING_INSTDONE(mmio_base));
- if (engine->id != RCS)
+ if (engine->id != RCS0)
break;
- instdone->slice_common = I915_READ(GEN7_SC_INSTDONE);
- instdone->sampler[0][0] = I915_READ(GEN7_SAMPLER_INSTDONE);
- instdone->row[0][0] = I915_READ(GEN7_ROW_INSTDONE);
+ instdone->slice_common =
+ intel_uncore_read(uncore, GEN7_SC_INSTDONE);
+ instdone->sampler[0][0] =
+ intel_uncore_read(uncore, GEN7_SAMPLER_INSTDONE);
+ instdone->row[0][0] =
+ intel_uncore_read(uncore, GEN7_ROW_INSTDONE);
break;
case 6:
case 5:
case 4:
- instdone->instdone = I915_READ(RING_INSTDONE(mmio_base));
-
- if (engine->id == RCS)
+ instdone->instdone =
+ intel_uncore_read(uncore, RING_INSTDONE(mmio_base));
+ if (engine->id == RCS0)
/* HACK: Using the wrong struct member */
- instdone->slice_common = I915_READ(GEN4_INSTDONE1);
+ instdone->slice_common =
+ intel_uncore_read(uncore, GEN4_INSTDONE1);
break;
case 3:
case 2:
- instdone->instdone = I915_READ(GEN2_INSTDONE);
+ instdone->instdone = intel_uncore_read(uncore, GEN2_INSTDONE);
break;
}
}
return true;
/* First check that no commands are left in the ring */
- if ((I915_READ_HEAD(engine) & HEAD_ADDR) !=
- (I915_READ_TAIL(engine) & TAIL_ADDR))
+ if ((ENGINE_READ(engine, RING_HEAD) & HEAD_ADDR) !=
+ (ENGINE_READ(engine, RING_TAIL) & TAIL_ADDR))
idle = false;
/* No bit for gen2, so assume the CS parser is idle */
- if (INTEL_GEN(dev_priv) > 2 && !(I915_READ_MODE(engine) & MODE_IDLE))
+ if (INTEL_GEN(dev_priv) > 2 &&
+ !(ENGINE_READ(engine, RING_MI_MODE) & MODE_IDLE))
idle = false;
intel_runtime_pm_put(dev_priv, wakeref);
*/
bool intel_engine_is_idle(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
-
/* More white lies, if wedged, hw state is inconsistent */
- if (i915_terminally_wedged(&dev_priv->gpu_error))
+ if (i915_reset_failed(engine->i915))
return true;
- /* Any inflight/incomplete requests? */
- if (!intel_engine_signaled(engine, intel_engine_last_submit(engine)))
- return false;
-
/* Waiting to drain ELSP? */
if (READ_ONCE(engine->execlists.active)) {
struct tasklet_struct *t = &engine->execlists.tasklet;
return ring_is_idle(engine);
}
-bool intel_engines_are_idle(struct drm_i915_private *dev_priv)
+bool intel_engines_are_idle(struct drm_i915_private *i915)
{
struct intel_engine_cs *engine;
enum intel_engine_id id;
* If the driver is wedged, HW state may be very inconsistent and
* report that it is still busy, even though we have stopped using it.
*/
- if (i915_terminally_wedged(&dev_priv->gpu_error))
+ if (i915_reset_failed(i915))
return true;
- for_each_engine(engine, dev_priv, id) {
+ /* Already parked (and passed an idleness test); must still be idle */
+ if (!READ_ONCE(i915->gt.awake))
+ return true;
+
+ for_each_engine(engine, i915, id) {
if (!intel_engine_is_idle(engine))
return false;
}
return true;
}
-/**
- * intel_engine_has_kernel_context:
- * @engine: the engine
- *
- * Returns true if the last context to be executed on this engine, or has been
- * executed if the engine is already idle, is the kernel context
- * (#i915.kernel_context).
- */
-bool intel_engine_has_kernel_context(const struct intel_engine_cs *engine)
-{
- const struct intel_context *kernel_context =
- to_intel_context(engine->i915->kernel_context, engine);
- struct i915_request *rq;
-
- lockdep_assert_held(&engine->i915->drm.struct_mutex);
-
- /*
- * Check the last context seen by the engine. If active, it will be
- * the last request that remains in the timeline. When idle, it is
- * the last executed context as tracked by retirement.
- */
- rq = __i915_active_request_peek(&engine->timeline.last_request);
- if (rq)
- return rq->hw_context == kernel_context;
- else
- return engine->last_retired_context == kernel_context;
-}
-
void intel_engines_reset_default_submission(struct drm_i915_private *i915)
{
struct intel_engine_cs *engine;
i915_gem_batch_pool_fini(&engine->batch_pool);
engine->execlists.no_priolist = false;
}
+
+ i915->gt.active_engines = 0;
}
/**
x = print_sched_attr(rq->i915, &rq->sched.attr, buf, x, sizeof(buf));
- drm_printf(m, "%s%x%s%s [%llx:%llx]%s @ %dms: %s\n",
+ drm_printf(m, "%s %llx:%llx%s%s %s @ %dms: %s\n",
prefix,
- rq->global_seqno,
+ rq->fence.context, rq->fence.seqno,
i915_request_completed(rq) ? "!" :
i915_request_started(rq) ? "*" :
"",
test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT,
&rq->fence.flags) ? "+" : "",
- rq->fence.context, rq->fence.seqno,
buf,
jiffies_to_msecs(jiffies - rq->emitted_jiffies),
name);
&engine->execlists;
u64 addr;
- if (engine->id == RCS && IS_GEN_RANGE(dev_priv, 4, 7))
- drm_printf(m, "\tCCID: 0x%08x\n", I915_READ(CCID));
+ if (engine->id == RCS0 && IS_GEN_RANGE(dev_priv, 4, 7))
+ drm_printf(m, "\tCCID: 0x%08x\n", ENGINE_READ(engine, CCID));
drm_printf(m, "\tRING_START: 0x%08x\n",
- I915_READ(RING_START(engine->mmio_base)));
+ ENGINE_READ(engine, RING_START));
drm_printf(m, "\tRING_HEAD: 0x%08x\n",
- I915_READ(RING_HEAD(engine->mmio_base)) & HEAD_ADDR);
+ ENGINE_READ(engine, RING_HEAD) & HEAD_ADDR);
drm_printf(m, "\tRING_TAIL: 0x%08x\n",
- I915_READ(RING_TAIL(engine->mmio_base)) & TAIL_ADDR);
+ ENGINE_READ(engine, RING_TAIL) & TAIL_ADDR);
drm_printf(m, "\tRING_CTL: 0x%08x%s\n",
- I915_READ(RING_CTL(engine->mmio_base)),
- I915_READ(RING_CTL(engine->mmio_base)) & (RING_WAIT | RING_WAIT_SEMAPHORE) ? " [waiting]" : "");
+ ENGINE_READ(engine, RING_CTL),
+ ENGINE_READ(engine, RING_CTL) & (RING_WAIT | RING_WAIT_SEMAPHORE) ? " [waiting]" : "");
if (INTEL_GEN(engine->i915) > 2) {
drm_printf(m, "\tRING_MODE: 0x%08x%s\n",
- I915_READ(RING_MI_MODE(engine->mmio_base)),
- I915_READ(RING_MI_MODE(engine->mmio_base)) & (MODE_IDLE) ? " [idle]" : "");
+ ENGINE_READ(engine, RING_MI_MODE),
+ ENGINE_READ(engine, RING_MI_MODE) & (MODE_IDLE) ? " [idle]" : "");
}
if (INTEL_GEN(dev_priv) >= 6) {
- drm_printf(m, "\tRING_IMR: %08x\n", I915_READ_IMR(engine));
+ drm_printf(m, "\tRING_IMR: %08x\n",
+ ENGINE_READ(engine, RING_IMR));
}
addr = intel_engine_get_active_head(engine);
drm_printf(m, "\tBBADDR: 0x%08x_%08x\n",
upper_32_bits(addr), lower_32_bits(addr));
if (INTEL_GEN(dev_priv) >= 8)
- addr = I915_READ64_2x32(RING_DMA_FADD(engine->mmio_base),
- RING_DMA_FADD_UDW(engine->mmio_base));
+ addr = ENGINE_READ64(engine, RING_DMA_FADD, RING_DMA_FADD_UDW);
else if (INTEL_GEN(dev_priv) >= 4)
- addr = I915_READ(RING_DMA_FADD(engine->mmio_base));
+ addr = ENGINE_READ(engine, RING_DMA_FADD);
else
- addr = I915_READ(DMA_FADD_I8XX);
+ addr = ENGINE_READ(engine, DMA_FADD_I8XX);
drm_printf(m, "\tDMA_FADDR: 0x%08x_%08x\n",
upper_32_bits(addr), lower_32_bits(addr));
if (INTEL_GEN(dev_priv) >= 4) {
drm_printf(m, "\tIPEIR: 0x%08x\n",
- I915_READ(RING_IPEIR(engine->mmio_base)));
+ ENGINE_READ(engine, RING_IPEIR));
drm_printf(m, "\tIPEHR: 0x%08x\n",
- I915_READ(RING_IPEHR(engine->mmio_base)));
+ ENGINE_READ(engine, RING_IPEHR));
} else {
- drm_printf(m, "\tIPEIR: 0x%08x\n", I915_READ(IPEIR));
- drm_printf(m, "\tIPEHR: 0x%08x\n", I915_READ(IPEHR));
+ drm_printf(m, "\tIPEIR: 0x%08x\n", ENGINE_READ(engine, IPEIR));
+ drm_printf(m, "\tIPEHR: 0x%08x\n", ENGINE_READ(engine, IPEHR));
}
if (HAS_EXECLISTS(dev_priv)) {
const u32 *hws =
&engine->status_page.addr[I915_HWS_CSB_BUF0_INDEX];
+ const u8 num_entries = execlists->csb_size;
unsigned int idx;
u8 read, write;
- drm_printf(m, "\tExeclist status: 0x%08x %08x\n",
- I915_READ(RING_EXECLIST_STATUS_LO(engine)),
- I915_READ(RING_EXECLIST_STATUS_HI(engine)));
+ drm_printf(m, "\tExeclist status: 0x%08x %08x, entries %u\n",
+ ENGINE_READ(engine, RING_EXECLIST_STATUS_LO),
+ ENGINE_READ(engine, RING_EXECLIST_STATUS_HI),
+ num_entries);
read = execlists->csb_head;
write = READ_ONCE(*execlists->csb_write);
- drm_printf(m, "\tExeclist CSB read %d, write %d [mmio:%d], tasklet queued? %s (%s)\n",
+ drm_printf(m, "\tExeclist CSB read %d, write %d, tasklet queued? %s (%s)\n",
read, write,
- GEN8_CSB_WRITE_PTR(I915_READ(RING_CONTEXT_STATUS_PTR(engine))),
yesno(test_bit(TASKLET_STATE_SCHED,
&engine->execlists.tasklet.state)),
enableddisabled(!atomic_read(&engine->execlists.tasklet.count)));
- if (read >= GEN8_CSB_ENTRIES)
+ if (read >= num_entries)
read = 0;
- if (write >= GEN8_CSB_ENTRIES)
+ if (write >= num_entries)
write = 0;
if (read > write)
- write += GEN8_CSB_ENTRIES;
+ write += num_entries;
while (read < write) {
- idx = ++read % GEN8_CSB_ENTRIES;
- drm_printf(m, "\tExeclist CSB[%d]: 0x%08x [mmio:0x%08x], context: %d [mmio:%d]\n",
- idx,
- hws[idx * 2],
- I915_READ(RING_CONTEXT_STATUS_BUF_LO(engine, idx)),
- hws[idx * 2 + 1],
- I915_READ(RING_CONTEXT_STATUS_BUF_HI(engine, idx)));
+ idx = ++read % num_entries;
+ drm_printf(m, "\tExeclist CSB[%d]: 0x%08x, context: %d\n",
+ idx, hws[idx * 2], hws[idx * 2 + 1]);
}
rcu_read_lock();
char hdr[80];
snprintf(hdr, sizeof(hdr),
- "\t\tELSP[%d] count=%d, ring:{start:%08x, hwsp:%08x}, rq: ",
+ "\t\tELSP[%d] count=%d, ring:{start:%08x, hwsp:%08x, seqno:%08x}, rq: ",
idx, count,
i915_ggtt_offset(rq->ring->vma),
- rq->timeline->hwsp_offset);
+ rq->timeline->hwsp_offset,
+ hwsp_seqno(rq));
print_request(m, rq, hdr);
} else {
drm_printf(m, "\t\tELSP[%d] idle\n", idx);
rcu_read_unlock();
} else if (INTEL_GEN(dev_priv) > 6) {
drm_printf(m, "\tPP_DIR_BASE: 0x%08x\n",
- I915_READ(RING_PP_DIR_BASE(engine)));
+ ENGINE_READ(engine, RING_PP_DIR_BASE));
drm_printf(m, "\tPP_DIR_BASE_READ: 0x%08x\n",
- I915_READ(RING_PP_DIR_BASE_READ(engine)));
+ ENGINE_READ(engine, RING_PP_DIR_BASE_READ));
drm_printf(m, "\tPP_DIR_DCLV: 0x%08x\n",
- I915_READ(RING_PP_DIR_DCLV(engine)));
+ ENGINE_READ(engine, RING_PP_DIR_DCLV));
}
}
va_end(ap);
}
- if (i915_terminally_wedged(&engine->i915->gpu_error))
+ if (i915_reset_failed(engine->i915))
drm_printf(m, "*** WEDGED ***\n");
- drm_printf(m, "\tcurrent seqno %x, last %x, hangcheck %x [%d ms]\n",
- intel_engine_get_seqno(engine),
- intel_engine_last_submit(engine),
- engine->hangcheck.seqno,
+ drm_printf(m, "\tHangcheck %x:%x [%d ms]\n",
+ engine->hangcheck.last_seqno,
+ engine->hangcheck.next_seqno,
jiffies_to_msecs(jiffies - engine->hangcheck.action_timestamp));
drm_printf(m, "\tReset count: %d (global %d)\n",
i915_reset_engine_count(error, engine),
if (&rq->link != &engine->timeline.requests)
print_request(m, rq, "\t\tlast ");
- rq = i915_gem_find_active_request(engine);
+ rq = intel_engine_find_active_request(engine);
if (rq) {
print_request(m, rq, "\t\tactive ");
write_sequnlock_irqrestore(&engine->stats.lock, flags);
}
+static bool match_ring(struct i915_request *rq)
+{
+ u32 ring = ENGINE_READ(rq->engine, RING_START);
+
+ return ring == i915_ggtt_offset(rq->ring->vma);
+}
+
+struct i915_request *
+intel_engine_find_active_request(struct intel_engine_cs *engine)
+{
+ struct i915_request *request, *active = NULL;
+ unsigned long flags;
+
+ /*
+ * We are called by the error capture, reset and to dump engine
+ * state at random points in time. In particular, note that neither is
+ * crucially ordered with an interrupt. After a hang, the GPU is dead
+ * and we assume that no more writes can happen (we waited long enough
+ * for all writes that were in transaction to be flushed) - adding an
+ * extra delay for a recent interrupt is pointless. Hence, we do
+ * not need an engine->irq_seqno_barrier() before the seqno reads.
+ * At all other times, we must assume the GPU is still running, but
+ * we only care about the snapshot of this moment.
+ */
+ spin_lock_irqsave(&engine->timeline.lock, flags);
+ list_for_each_entry(request, &engine->timeline.requests, link) {
+ if (i915_request_completed(request))
+ continue;
+
+ if (!i915_request_started(request))
+ break;
+
+ /* More than one preemptible request may match! */
+ if (!match_ring(request))
+ break;
+
+ active = request;
+ break;
+ }
+ spin_unlock_irqrestore(&engine->timeline.lock, flags);
+
+ return active;
+}
+
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftests/mock_engine.c"
#include "selftests/intel_engine_cs.c"
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_ENGINE_TYPES__
+#define __INTEL_ENGINE_TYPES__
+
+#include <linux/hashtable.h>
+#include <linux/irq_work.h>
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/types.h>
+
+#include "i915_gem.h"
+#include "i915_priolist_types.h"
+#include "i915_selftest.h"
+#include "i915_timeline_types.h"
+#include "intel_workarounds_types.h"
+
+#include "i915_gem_batch_pool.h"
+#include "i915_pmu.h"
+
+#define I915_MAX_SLICES 3
+#define I915_MAX_SUBSLICES 8
+
+#define I915_CMD_HASH_ORDER 9
+
+struct dma_fence;
+struct drm_i915_reg_table;
+struct i915_gem_context;
+struct i915_request;
+struct i915_sched_attr;
+struct intel_uncore;
+
+typedef u8 intel_engine_mask_t;
+#define ALL_ENGINES ((intel_engine_mask_t)~0ul)
+
+struct intel_hw_status_page {
+ struct i915_vma *vma;
+ u32 *addr;
+};
+
+struct intel_instdone {
+ u32 instdone;
+ /* The following exist only in the RCS engine */
+ u32 slice_common;
+ u32 sampler[I915_MAX_SLICES][I915_MAX_SUBSLICES];
+ u32 row[I915_MAX_SLICES][I915_MAX_SUBSLICES];
+};
+
+struct intel_engine_hangcheck {
+ u64 acthd;
+ u32 last_seqno;
+ u32 next_seqno;
+ unsigned long action_timestamp;
+ struct intel_instdone instdone;
+};
+
+struct intel_ring {
+ struct kref ref;
+ struct i915_vma *vma;
+ void *vaddr;
+
+ struct i915_timeline *timeline;
+ struct list_head request_list;
+ struct list_head active_link;
+
+ u32 head;
+ u32 tail;
+ u32 emit;
+
+ u32 space;
+ u32 size;
+ u32 effective_size;
+};
+
+/*
+ * we use a single page to load ctx workarounds so all of these
+ * values are referred in terms of dwords
+ *
+ * struct i915_wa_ctx_bb:
+ * offset: specifies batch starting position, also helpful in case
+ * if we want to have multiple batches at different offsets based on
+ * some criteria. It is not a requirement at the moment but provides
+ * an option for future use.
+ * size: size of the batch in DWORDS
+ */
+struct i915_ctx_workarounds {
+ struct i915_wa_ctx_bb {
+ u32 offset;
+ u32 size;
+ } indirect_ctx, per_ctx;
+ struct i915_vma *vma;
+};
+
+#define I915_MAX_VCS 4
+#define I915_MAX_VECS 2
+
+/*
+ * Engine IDs definitions.
+ * Keep instances of the same type engine together.
+ */
+enum intel_engine_id {
+ RCS0 = 0,
+ BCS0,
+ VCS0,
+ VCS1,
+ VCS2,
+ VCS3,
+#define _VCS(n) (VCS0 + (n))
+ VECS0,
+ VECS1,
+#define _VECS(n) (VECS0 + (n))
+ I915_NUM_ENGINES
+};
+
+struct st_preempt_hang {
+ struct completion completion;
+ unsigned int count;
+ bool inject_hang;
+};
+
+/**
+ * struct intel_engine_execlists - execlist submission queue and port state
+ *
+ * The struct intel_engine_execlists represents the combined logical state of
+ * driver and the hardware state for execlist mode of submission.
+ */
+struct intel_engine_execlists {
+ /**
+ * @tasklet: softirq tasklet for bottom handler
+ */
+ struct tasklet_struct tasklet;
+
+ /**
+ * @default_priolist: priority list for I915_PRIORITY_NORMAL
+ */
+ struct i915_priolist default_priolist;
+
+ /**
+ * @no_priolist: priority lists disabled
+ */
+ bool no_priolist;
+
+ /**
+ * @submit_reg: gen-specific execlist submission register
+ * set to the ExecList Submission Port (elsp) register pre-Gen11 and to
+ * the ExecList Submission Queue Contents register array for Gen11+
+ */
+ u32 __iomem *submit_reg;
+
+ /**
+ * @ctrl_reg: the enhanced execlists control register, used to load the
+ * submit queue on the HW and to request preemptions to idle
+ */
+ u32 __iomem *ctrl_reg;
+
+ /**
+ * @port: execlist port states
+ *
+ * For each hardware ELSP (ExecList Submission Port) we keep
+ * track of the last request and the number of times we submitted
+ * that port to hw. We then count the number of times the hw reports
+ * a context completion or preemption. As only one context can
+ * be active on hw, we limit resubmission of context to port[0]. This
+ * is called Lite Restore, of the context.
+ */
+ struct execlist_port {
+ /**
+ * @request_count: combined request and submission count
+ */
+ struct i915_request *request_count;
+#define EXECLIST_COUNT_BITS 2
+#define port_request(p) ptr_mask_bits((p)->request_count, EXECLIST_COUNT_BITS)
+#define port_count(p) ptr_unmask_bits((p)->request_count, EXECLIST_COUNT_BITS)
+#define port_pack(rq, count) ptr_pack_bits(rq, count, EXECLIST_COUNT_BITS)
+#define port_unpack(p, count) ptr_unpack_bits((p)->request_count, count, EXECLIST_COUNT_BITS)
+#define port_set(p, packed) ((p)->request_count = (packed))
+#define port_isset(p) ((p)->request_count)
+#define port_index(p, execlists) ((p) - (execlists)->port)
+
+ /**
+ * @context_id: context ID for port
+ */
+ GEM_DEBUG_DECL(u32 context_id);
+
+#define EXECLIST_MAX_PORTS 2
+ } port[EXECLIST_MAX_PORTS];
+
+ /**
+ * @active: is the HW active? We consider the HW as active after
+ * submitting any context for execution and until we have seen the
+ * last context completion event. After that, we do not expect any
+ * more events until we submit, and so can park the HW.
+ *
+ * As we have a small number of different sources from which we feed
+ * the HW, we track the state of each inside a single bitfield.
+ */
+ unsigned int active;
+#define EXECLISTS_ACTIVE_USER 0
+#define EXECLISTS_ACTIVE_PREEMPT 1
+#define EXECLISTS_ACTIVE_HWACK 2
+
+ /**
+ * @port_mask: number of execlist ports - 1
+ */
+ unsigned int port_mask;
+
+ /**
+ * @queue_priority_hint: Highest pending priority.
+ *
+ * When we add requests into the queue, or adjust the priority of
+ * executing requests, we compute the maximum priority of those
+ * pending requests. We can then use this value to determine if
+ * we need to preempt the executing requests to service the queue.
+ * However, since the we may have recorded the priority of an inflight
+ * request we wanted to preempt but since completed, at the time of
+ * dequeuing the priority hint may no longer may match the highest
+ * available request priority.
+ */
+ int queue_priority_hint;
+
+ /**
+ * @queue: queue of requests, in priority lists
+ */
+ struct rb_root_cached queue;
+
+ /**
+ * @csb_write: control register for Context Switch buffer
+ *
+ * Note this register may be either mmio or HWSP shadow.
+ */
+ u32 *csb_write;
+
+ /**
+ * @csb_status: status array for Context Switch buffer
+ *
+ * Note these register may be either mmio or HWSP shadow.
+ */
+ u32 *csb_status;
+
+ /**
+ * @preempt_complete_status: expected CSB upon completing preemption
+ */
+ u32 preempt_complete_status;
+
+ /**
+ * @csb_size: context status buffer FIFO size
+ */
+ u8 csb_size;
+
+ /**
+ * @csb_head: context status buffer head
+ */
+ u8 csb_head;
+
+ I915_SELFTEST_DECLARE(struct st_preempt_hang preempt_hang;)
+};
+
+#define INTEL_ENGINE_CS_MAX_NAME 8
+
+struct intel_engine_cs {
+ struct drm_i915_private *i915;
+ struct intel_uncore *uncore;
+ char name[INTEL_ENGINE_CS_MAX_NAME];
+
+ enum intel_engine_id id;
+ unsigned int hw_id;
+ unsigned int guc_id;
+ intel_engine_mask_t mask;
+
+ u8 uabi_class;
+
+ u8 class;
+ u8 instance;
+ u32 context_size;
+ u32 mmio_base;
+
+ struct intel_ring *buffer;
+
+ struct i915_timeline timeline;
+
+ struct intel_context *kernel_context; /* pinned */
+ struct intel_context *preempt_context; /* pinned; optional */
+
+ struct drm_i915_gem_object *default_state;
+ void *pinned_default_state;
+
+ /* Rather than have every client wait upon all user interrupts,
+ * with the herd waking after every interrupt and each doing the
+ * heavyweight seqno dance, we delegate the task (of being the
+ * bottom-half of the user interrupt) to the first client. After
+ * every interrupt, we wake up one client, who does the heavyweight
+ * coherent seqno read and either goes back to sleep (if incomplete),
+ * or wakes up all the completed clients in parallel, before then
+ * transferring the bottom-half status to the next client in the queue.
+ *
+ * Compared to walking the entire list of waiters in a single dedicated
+ * bottom-half, we reduce the latency of the first waiter by avoiding
+ * a context switch, but incur additional coherent seqno reads when
+ * following the chain of request breadcrumbs. Since it is most likely
+ * that we have a single client waiting on each seqno, then reducing
+ * the overhead of waking that client is much preferred.
+ */
+ struct intel_breadcrumbs {
+ spinlock_t irq_lock;
+ struct list_head signalers;
+
+ struct irq_work irq_work; /* for use from inside irq_lock */
+
+ unsigned int irq_enabled;
+
+ bool irq_armed;
+ } breadcrumbs;
+
+ struct intel_engine_pmu {
+ /**
+ * @enable: Bitmask of enable sample events on this engine.
+ *
+ * Bits correspond to sample event types, for instance
+ * I915_SAMPLE_QUEUED is bit 0 etc.
+ */
+ u32 enable;
+ /**
+ * @enable_count: Reference count for the enabled samplers.
+ *
+ * Index number corresponds to @enum drm_i915_pmu_engine_sample.
+ */
+ unsigned int enable_count[I915_ENGINE_SAMPLE_COUNT];
+ /**
+ * @sample: Counter values for sampling events.
+ *
+ * Our internal timer stores the current counters in this field.
+ *
+ * Index number corresponds to @enum drm_i915_pmu_engine_sample.
+ */
+ struct i915_pmu_sample sample[I915_ENGINE_SAMPLE_COUNT];
+ } pmu;
+
+ /*
+ * A pool of objects to use as shadow copies of client batch buffers
+ * when the command parser is enabled. Prevents the client from
+ * modifying the batch contents after software parsing.
+ */
+ struct i915_gem_batch_pool batch_pool;
+
+ struct intel_hw_status_page status_page;
+ struct i915_ctx_workarounds wa_ctx;
+ struct i915_wa_list ctx_wa_list;
+ struct i915_wa_list wa_list;
+ struct i915_wa_list whitelist;
+
+ u32 irq_keep_mask; /* always keep these interrupts */
+ u32 irq_enable_mask; /* bitmask to enable ring interrupt */
+ void (*irq_enable)(struct intel_engine_cs *engine);
+ void (*irq_disable)(struct intel_engine_cs *engine);
+
+ int (*init_hw)(struct intel_engine_cs *engine);
+
+ struct {
+ void (*prepare)(struct intel_engine_cs *engine);
+ void (*reset)(struct intel_engine_cs *engine, bool stalled);
+ void (*finish)(struct intel_engine_cs *engine);
+ } reset;
+
+ void (*park)(struct intel_engine_cs *engine);
+ void (*unpark)(struct intel_engine_cs *engine);
+
+ void (*set_default_submission)(struct intel_engine_cs *engine);
+
+ const struct intel_context_ops *cops;
+
+ int (*request_alloc)(struct i915_request *rq);
+ int (*init_context)(struct i915_request *rq);
+
+ int (*emit_flush)(struct i915_request *request, u32 mode);
+#define EMIT_INVALIDATE BIT(0)
+#define EMIT_FLUSH BIT(1)
+#define EMIT_BARRIER (EMIT_INVALIDATE | EMIT_FLUSH)
+ int (*emit_bb_start)(struct i915_request *rq,
+ u64 offset, u32 length,
+ unsigned int dispatch_flags);
+#define I915_DISPATCH_SECURE BIT(0)
+#define I915_DISPATCH_PINNED BIT(1)
+ int (*emit_init_breadcrumb)(struct i915_request *rq);
+ u32 *(*emit_fini_breadcrumb)(struct i915_request *rq,
+ u32 *cs);
+ unsigned int emit_fini_breadcrumb_dw;
+
+ /* Pass the request to the hardware queue (e.g. directly into
+ * the legacy ringbuffer or to the end of an execlist).
+ *
+ * This is called from an atomic context with irqs disabled; must
+ * be irq safe.
+ */
+ void (*submit_request)(struct i915_request *rq);
+
+ /*
+ * Call when the priority on a request has changed and it and its
+ * dependencies may need rescheduling. Note the request itself may
+ * not be ready to run!
+ */
+ void (*schedule)(struct i915_request *request,
+ const struct i915_sched_attr *attr);
+
+ /*
+ * Cancel all requests on the hardware, or queued for execution.
+ * This should only cancel the ready requests that have been
+ * submitted to the engine (via the engine->submit_request callback).
+ * This is called when marking the device as wedged.
+ */
+ void (*cancel_requests)(struct intel_engine_cs *engine);
+
+ void (*cleanup)(struct intel_engine_cs *engine);
+
+ struct intel_engine_execlists execlists;
+
+ /* Contexts are pinned whilst they are active on the GPU. The last
+ * context executed remains active whilst the GPU is idle - the
+ * switch away and write to the context object only occurs on the
+ * next execution. Contexts are only unpinned on retirement of the
+ * following request ensuring that we can always write to the object
+ * on the context switch even after idling. Across suspend, we switch
+ * to the kernel context and trash it as the save may not happen
+ * before the hardware is powered down.
+ */
+ struct intel_context *last_retired_context;
+
+ /* status_notifier: list of callbacks for context-switch changes */
+ struct atomic_notifier_head context_status_notifier;
+
+ struct intel_engine_hangcheck hangcheck;
+
+#define I915_ENGINE_NEEDS_CMD_PARSER BIT(0)
+#define I915_ENGINE_SUPPORTS_STATS BIT(1)
+#define I915_ENGINE_HAS_PREEMPTION BIT(2)
+#define I915_ENGINE_HAS_SEMAPHORES BIT(3)
+#define I915_ENGINE_NEEDS_BREADCRUMB_TASKLET BIT(4)
+ unsigned int flags;
+
+ /*
+ * Table of commands the command parser needs to know about
+ * for this engine.
+ */
+ DECLARE_HASHTABLE(cmd_hash, I915_CMD_HASH_ORDER);
+
+ /*
+ * Table of registers allowed in commands that read/write registers.
+ */
+ const struct drm_i915_reg_table *reg_tables;
+ int reg_table_count;
+
+ /*
+ * Returns the bitmask for the length field of the specified command.
+ * Return 0 for an unrecognized/invalid command.
+ *
+ * If the command parser finds an entry for a command in the engine's
+ * cmd_tables, it gets the command's length based on the table entry.
+ * If not, it calls this function to determine the per-engine length
+ * field encoding for the command (i.e. different opcode ranges use
+ * certain bits to encode the command length in the header).
+ */
+ u32 (*get_cmd_length_mask)(u32 cmd_header);
+
+ struct {
+ /**
+ * @lock: Lock protecting the below fields.
+ */
+ seqlock_t lock;
+ /**
+ * @enabled: Reference count indicating number of listeners.
+ */
+ unsigned int enabled;
+ /**
+ * @active: Number of contexts currently scheduled in.
+ */
+ unsigned int active;
+ /**
+ * @enabled_at: Timestamp when busy stats were enabled.
+ */
+ ktime_t enabled_at;
+ /**
+ * @start: Timestamp of the last idle to active transition.
+ *
+ * Idle is defined as active == 0, active is active > 0.
+ */
+ ktime_t start;
+ /**
+ * @total: Total time this engine was busy.
+ *
+ * Accumulated time not counting the most recent block in cases
+ * where engine is currently busy (active > 0).
+ */
+ ktime_t total;
+ } stats;
+};
+
+static inline bool
+intel_engine_needs_cmd_parser(const struct intel_engine_cs *engine)
+{
+ return engine->flags & I915_ENGINE_NEEDS_CMD_PARSER;
+}
+
+static inline bool
+intel_engine_supports_stats(const struct intel_engine_cs *engine)
+{
+ return engine->flags & I915_ENGINE_SUPPORTS_STATS;
+}
+
+static inline bool
+intel_engine_has_preemption(const struct intel_engine_cs *engine)
+{
+ return engine->flags & I915_ENGINE_HAS_PREEMPTION;
+}
+
+static inline bool
+intel_engine_has_semaphores(const struct intel_engine_cs *engine)
+{
+ return engine->flags & I915_ENGINE_HAS_SEMAPHORES;
+}
+
+static inline bool
+intel_engine_needs_breadcrumb_tasklet(const struct intel_engine_cs *engine)
+{
+ return engine->flags & I915_ENGINE_NEEDS_BREADCRUMB_TASKLET;
+}
+
+#define instdone_slice_mask(dev_priv__) \
+ (IS_GEN(dev_priv__, 7) ? \
+ 1 : RUNTIME_INFO(dev_priv__)->sseu.slice_mask)
+
+#define instdone_subslice_mask(dev_priv__) \
+ (IS_GEN(dev_priv__, 7) ? \
+ 1 : RUNTIME_INFO(dev_priv__)->sseu.subslice_mask[0])
+
+#define for_each_instdone_slice_subslice(dev_priv__, slice__, subslice__) \
+ for ((slice__) = 0, (subslice__) = 0; \
+ (slice__) < I915_MAX_SLICES; \
+ (subslice__) = ((subslice__) + 1) < I915_MAX_SUBSLICES ? (subslice__) + 1 : 0, \
+ (slice__) += ((subslice__) == 0)) \
+ for_each_if((BIT(slice__) & instdone_slice_mask(dev_priv__)) && \
+ (BIT(subslice__) & instdone_subslice_mask(dev_priv__)))
+
+#endif /* __INTEL_ENGINE_TYPES_H__ */
#include <drm/drm_fourcc.h>
-#include "intel_drv.h"
#include "i915_drv.h"
+#include "intel_drv.h"
+#include "intel_fbc.h"
+#include "intel_frontbuffer.h"
static inline bool fbc_supported(struct drm_i915_private *dev_priv)
{
I915_WRITE(FBC_CONTROL, fbc_ctl);
/* Wait for compressing bit to clear */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
FBC_STATUS, FBC_STAT_COMPRESSING, 0,
10)) {
DRM_DEBUG_KMS("FBC idle timed out\n");
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_FBC_H__
+#define __INTEL_FBC_H__
+
+#include <linux/types.h>
+
+#include "intel_frontbuffer.h"
+
+struct drm_i915_private;
+struct intel_atomic_state;
+struct intel_crtc;
+struct intel_crtc_state;
+struct intel_plane_state;
+
+void intel_fbc_choose_crtc(struct drm_i915_private *dev_priv,
+ struct intel_atomic_state *state);
+bool intel_fbc_is_active(struct drm_i915_private *dev_priv);
+void intel_fbc_pre_update(struct intel_crtc *crtc,
+ struct intel_crtc_state *crtc_state,
+ struct intel_plane_state *plane_state);
+void intel_fbc_post_update(struct intel_crtc *crtc);
+void intel_fbc_init(struct drm_i915_private *dev_priv);
+void intel_fbc_init_pipe_state(struct drm_i915_private *dev_priv);
+void intel_fbc_enable(struct intel_crtc *crtc,
+ struct intel_crtc_state *crtc_state,
+ struct intel_plane_state *plane_state);
+void intel_fbc_disable(struct intel_crtc *crtc);
+void intel_fbc_global_disable(struct drm_i915_private *dev_priv);
+void intel_fbc_invalidate(struct drm_i915_private *dev_priv,
+ unsigned int frontbuffer_bits,
+ enum fb_op_origin origin);
+void intel_fbc_flush(struct drm_i915_private *dev_priv,
+ unsigned int frontbuffer_bits, enum fb_op_origin origin);
+void intel_fbc_cleanup_cfb(struct drm_i915_private *dev_priv);
+void intel_fbc_handle_fifo_underrun_irq(struct drm_i915_private *dev_priv);
+int intel_fbc_reset_underrun(struct drm_i915_private *dev_priv);
+
+#endif /* __INTEL_FBC_H__ */
*/
#include <linux/async.h>
-#include <linux/module.h>
-#include <linux/kernel.h>
#include <linux/console.h>
+#include <linux/delay.h>
#include <linux/errno.h>
-#include <linux/string.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
#include <linux/mm.h>
-#include <linux/tty.h>
+#include <linux/module.h>
+#include <linux/string.h>
#include <linux/sysrq.h>
-#include <linux/delay.h>
-#include <linux/init.h>
+#include <linux/tty.h>
#include <linux/vga_switcheroo.h>
#include <drm/drm_crtc.h>
#include <drm/drm_fb_helper.h>
#include <drm/drm_fourcc.h>
+#include <drm/i915_drm.h>
+#include "i915_drv.h"
#include "intel_drv.h"
+#include "intel_fbdev.h"
#include "intel_frontbuffer.h"
-#include <drm/i915_drm.h>
-#include "i915_drv.h"
static void intel_fbdev_invalidate(struct intel_fbdev *ifbdev)
{
goto out_unpin;
}
- info->par = helper;
-
ifbdev->helper.fb = fb;
- strcpy(info->fix.id, "inteldrmfb");
-
info->fbops = &intelfb_ops;
/* setup aperture base/size for vesafb takeover */
info->screen_base = vaddr;
info->screen_size = vma->node.size;
- /* This driver doesn't need a VT switch to restore the mode on resume */
- info->skip_vt_switch = true;
-
- drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(info, &ifbdev->helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(info, &ifbdev->helper, sizes);
/* If the object is shmemfs backed, it will have given us zeroed pages.
* If the object is stolen however, it will be full of whatever
return ret;
}
-static struct drm_fb_helper_crtc *
-intel_fb_helper_crtc(struct drm_fb_helper *fb_helper, struct drm_crtc *crtc)
-{
- int i;
-
- for (i = 0; i < fb_helper->crtc_count; i++)
- if (fb_helper->crtc_info[i].mode_set.crtc == crtc)
- return &fb_helper->crtc_info[i];
-
- return NULL;
-}
-
-/*
- * Try to read the BIOS display configuration and use it for the initial
- * fb configuration.
- *
- * The BIOS or boot loader will generally create an initial display
- * configuration for us that includes some set of active pipes and displays.
- * This routine tries to figure out which pipes and connectors are active
- * and stuffs them into the crtcs and modes array given to us by the
- * drm_fb_helper code.
- *
- * The overall sequence is:
- * intel_fbdev_init - from driver load
- * intel_fbdev_init_bios - initialize the intel_fbdev using BIOS data
- * drm_fb_helper_init - build fb helper structs
- * drm_fb_helper_single_add_all_connectors - more fb helper structs
- * intel_fbdev_initial_config - apply the config
- * drm_fb_helper_initial_config - call ->probe then register_framebuffer()
- * drm_setup_crtcs - build crtc config for fbdev
- * intel_fb_initial_config - find active connectors etc
- * drm_fb_helper_single_fb_probe - set up fbdev
- * intelfb_create - re-use or alloc fb, build out fbdev structs
- *
- * Note that we don't make special consideration whether we could actually
- * switch to the selected modes without a full modeset. E.g. when the display
- * is in VGA mode we need to recalculate watermarks and set a new high-res
- * framebuffer anyway.
- */
-static bool intel_fb_initial_config(struct drm_fb_helper *fb_helper,
- struct drm_fb_helper_crtc **crtcs,
- struct drm_display_mode **modes,
- struct drm_fb_offset *offsets,
- bool *enabled, int width, int height)
-{
- struct drm_i915_private *dev_priv = to_i915(fb_helper->dev);
- unsigned long conn_configured, conn_seq, mask;
- unsigned int count = min(fb_helper->connector_count, BITS_PER_LONG);
- int i, j;
- bool *save_enabled;
- bool fallback = true, ret = true;
- int num_connectors_enabled = 0;
- int num_connectors_detected = 0;
- struct drm_modeset_acquire_ctx ctx;
-
- save_enabled = kcalloc(count, sizeof(bool), GFP_KERNEL);
- if (!save_enabled)
- return false;
-
- drm_modeset_acquire_init(&ctx, 0);
-
- while (drm_modeset_lock_all_ctx(fb_helper->dev, &ctx) != 0)
- drm_modeset_backoff(&ctx);
-
- memcpy(save_enabled, enabled, count);
- mask = GENMASK(count - 1, 0);
- conn_configured = 0;
-retry:
- conn_seq = conn_configured;
- for (i = 0; i < count; i++) {
- struct drm_fb_helper_connector *fb_conn;
- struct drm_connector *connector;
- struct drm_encoder *encoder;
- struct drm_fb_helper_crtc *new_crtc;
-
- fb_conn = fb_helper->connector_info[i];
- connector = fb_conn->connector;
-
- if (conn_configured & BIT(i))
- continue;
-
- if (conn_seq == 0 && !connector->has_tile)
- continue;
-
- if (connector->status == connector_status_connected)
- num_connectors_detected++;
-
- if (!enabled[i]) {
- DRM_DEBUG_KMS("connector %s not enabled, skipping\n",
- connector->name);
- conn_configured |= BIT(i);
- continue;
- }
-
- if (connector->force == DRM_FORCE_OFF) {
- DRM_DEBUG_KMS("connector %s is disabled by user, skipping\n",
- connector->name);
- enabled[i] = false;
- continue;
- }
-
- encoder = connector->state->best_encoder;
- if (!encoder || WARN_ON(!connector->state->crtc)) {
- if (connector->force > DRM_FORCE_OFF)
- goto bail;
-
- DRM_DEBUG_KMS("connector %s has no encoder or crtc, skipping\n",
- connector->name);
- enabled[i] = false;
- conn_configured |= BIT(i);
- continue;
- }
-
- num_connectors_enabled++;
-
- new_crtc = intel_fb_helper_crtc(fb_helper,
- connector->state->crtc);
-
- /*
- * Make sure we're not trying to drive multiple connectors
- * with a single CRTC, since our cloning support may not
- * match the BIOS.
- */
- for (j = 0; j < count; j++) {
- if (crtcs[j] == new_crtc) {
- DRM_DEBUG_KMS("fallback: cloned configuration\n");
- goto bail;
- }
- }
-
- DRM_DEBUG_KMS("looking for cmdline mode on connector %s\n",
- connector->name);
-
- /* go for command line mode first */
- modes[i] = drm_pick_cmdline_mode(fb_conn);
-
- /* try for preferred next */
- if (!modes[i]) {
- DRM_DEBUG_KMS("looking for preferred mode on connector %s %d\n",
- connector->name, connector->has_tile);
- modes[i] = drm_has_preferred_mode(fb_conn, width,
- height);
- }
-
- /* No preferred mode marked by the EDID? Are there any modes? */
- if (!modes[i] && !list_empty(&connector->modes)) {
- DRM_DEBUG_KMS("using first mode listed on connector %s\n",
- connector->name);
- modes[i] = list_first_entry(&connector->modes,
- struct drm_display_mode,
- head);
- }
-
- /* last resort: use current mode */
- if (!modes[i]) {
- /*
- * IMPORTANT: We want to use the adjusted mode (i.e.
- * after the panel fitter upscaling) as the initial
- * config, not the input mode, which is what crtc->mode
- * usually contains. But since our current
- * code puts a mode derived from the post-pfit timings
- * into crtc->mode this works out correctly.
- *
- * This is crtc->mode and not crtc->state->mode for the
- * fastboot check to work correctly. crtc_state->mode has
- * I915_MODE_FLAG_INHERITED, which we clear to force check
- * state.
- */
- DRM_DEBUG_KMS("looking for current mode on connector %s\n",
- connector->name);
- modes[i] = &connector->state->crtc->mode;
- }
- crtcs[i] = new_crtc;
-
- DRM_DEBUG_KMS("connector %s on [CRTC:%d:%s]: %dx%d%s\n",
- connector->name,
- connector->state->crtc->base.id,
- connector->state->crtc->name,
- modes[i]->hdisplay, modes[i]->vdisplay,
- modes[i]->flags & DRM_MODE_FLAG_INTERLACE ? "i" :"");
-
- fallback = false;
- conn_configured |= BIT(i);
- }
-
- if ((conn_configured & mask) != mask && conn_configured != conn_seq)
- goto retry;
-
- /*
- * If the BIOS didn't enable everything it could, fall back to have the
- * same user experiencing of lighting up as much as possible like the
- * fbdev helper library.
- */
- if (num_connectors_enabled != num_connectors_detected &&
- num_connectors_enabled < INTEL_INFO(dev_priv)->num_pipes) {
- DRM_DEBUG_KMS("fallback: Not all outputs enabled\n");
- DRM_DEBUG_KMS("Enabled: %i, detected: %i\n", num_connectors_enabled,
- num_connectors_detected);
- fallback = true;
- }
-
- if (fallback) {
-bail:
- DRM_DEBUG_KMS("Not using firmware configuration\n");
- memcpy(enabled, save_enabled, count);
- ret = false;
- }
-
- drm_modeset_drop_locks(&ctx);
- drm_modeset_acquire_fini(&ctx);
-
- kfree(save_enabled);
- return ret;
-}
-
static const struct drm_fb_helper_funcs intel_fb_helper_funcs = {
- .initial_config = intel_fb_initial_config,
.fb_probe = intelfb_create,
};
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_FBDEV_H__
+#define __INTEL_FBDEV_H__
+
+#include <linux/types.h>
+
+struct drm_device;
+struct drm_i915_private;
+
+#ifdef CONFIG_DRM_FBDEV_EMULATION
+int intel_fbdev_init(struct drm_device *dev);
+void intel_fbdev_initial_config_async(struct drm_device *dev);
+void intel_fbdev_unregister(struct drm_i915_private *dev_priv);
+void intel_fbdev_fini(struct drm_i915_private *dev_priv);
+void intel_fbdev_set_suspend(struct drm_device *dev, int state, bool synchronous);
+void intel_fbdev_output_poll_changed(struct drm_device *dev);
+void intel_fbdev_restore_mode(struct drm_device *dev);
+#else
+static inline int intel_fbdev_init(struct drm_device *dev)
+{
+ return 0;
+}
+
+static inline void intel_fbdev_initial_config_async(struct drm_device *dev)
+{
+}
+
+static inline void intel_fbdev_unregister(struct drm_i915_private *dev_priv)
+{
+}
+
+static inline void intel_fbdev_fini(struct drm_i915_private *dev_priv)
+{
+}
+
+static inline void intel_fbdev_set_suspend(struct drm_device *dev, int state, bool synchronous)
+{
+}
+
+static inline void intel_fbdev_output_poll_changed(struct drm_device *dev)
+{
+}
+
+static inline void intel_fbdev_restore_mode(struct drm_device *dev)
+{
+}
+#endif
+
+#endif /* __INTEL_FBDEV_H__ */
#include "i915_drv.h"
#include "intel_drv.h"
+#include "intel_fbc.h"
/**
* DOC: fifo underrun handling
*/
+#include "i915_drv.h"
+#include "intel_dp.h"
#include "intel_drv.h"
+#include "intel_fbc.h"
#include "intel_frontbuffer.h"
-#include "i915_drv.h"
+#include "intel_psr.h"
void __intel_fb_obj_invalidate(struct drm_i915_gem_object *obj,
enum fb_op_origin origin,
#ifndef __INTEL_FRONTBUFFER_H__
#define __INTEL_FRONTBUFFER_H__
+#include "i915_gem_object.h"
+
struct drm_i915_private;
struct drm_i915_gem_object;
+enum fb_op_origin {
+ ORIGIN_GTT,
+ ORIGIN_CPU,
+ ORIGIN_CS,
+ ORIGIN_FLIP,
+ ORIGIN_DIRTYFB,
+};
+
void intel_frontbuffer_flip_prepare(struct drm_i915_private *dev_priv,
unsigned frontbuffer_bits);
void intel_frontbuffer_flip_complete(struct drm_i915_private *dev_priv,
#define MI_SEMAPHORE_SIGNAL MI_INSTR(0x1b, 0) /* GEN8+ */
#define MI_SEMAPHORE_TARGET(engine) ((engine)<<15)
#define MI_SEMAPHORE_WAIT MI_INSTR(0x1c, 2) /* GEN8+ */
-#define MI_SEMAPHORE_POLL (1<<15)
-#define MI_SEMAPHORE_SAD_GTE_SDD (1<<12)
+#define MI_SEMAPHORE_POLL (1 << 15)
+#define MI_SEMAPHORE_SAD_GT_SDD (0 << 12)
+#define MI_SEMAPHORE_SAD_GTE_SDD (1 << 12)
+#define MI_SEMAPHORE_SAD_LT_SDD (2 << 12)
+#define MI_SEMAPHORE_SAD_LTE_SDD (3 << 12)
+#define MI_SEMAPHORE_SAD_EQ_SDD (4 << 12)
+#define MI_SEMAPHORE_SAD_NEQ_SDD (5 << 12)
#define MI_STORE_DWORD_IMM MI_INSTR(0x20, 1)
#define MI_STORE_DWORD_IMM_GEN4 MI_INSTR(0x20, 2)
#define MI_MEM_VIRTUAL (1 << 22) /* 945,g33,965 */
BUILD_BUG_ON(GUC_MAX_MMIO_MSG_LEN > SOFT_SCRATCH_COUNT);
for (i = 0; i < guc->send_regs.count; i++) {
- fw_domains |= intel_uncore_forcewake_for_reg(dev_priv,
+ fw_domains |= intel_uncore_forcewake_for_reg(&dev_priv->uncore,
guc_send_reg(guc, i),
FW_REG_READ | FW_REG_WRITE);
}
goto err_log;
GEM_BUG_ON(!guc->ads_vma);
+ if (HAS_GUC_CT(dev_priv)) {
+ ret = intel_guc_ct_init(&guc->ct);
+ if (ret)
+ goto err_ads;
+ }
+
/* We need to notify the guc whenever we change the GGTT */
i915_ggtt_enable_guc(dev_priv);
return 0;
+err_ads:
+ intel_guc_ads_destroy(guc);
err_log:
intel_guc_log_destroy(&guc->log);
err_shared:
struct drm_i915_private *dev_priv = guc_to_i915(guc);
i915_ggtt_disable_guc(dev_priv);
+
+ if (HAS_GUC_CT(dev_priv))
+ intel_guc_ct_fini(&guc->ct);
+
intel_guc_ads_destroy(guc);
intel_guc_log_destroy(&guc->log);
guc_shared_data_destroy(guc);
* they are power context saved so it's ok to release forcewake
* when we are done here and take it again at xfer time.
*/
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_BLITTER);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_BLITTER);
I915_WRITE(SOFT_SCRATCH(0), 0);
for (i = 0; i < GUC_CTL_MAX_DWORDS; i++)
I915_WRITE(SOFT_SCRATCH(1 + i), params[i]);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_BLITTER);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_BLITTER);
}
int intel_guc_send_nop(struct intel_guc *guc, const u32 *action, u32 len,
u32 *response_buf, u32 response_buf_size)
{
struct drm_i915_private *dev_priv = guc_to_i915(guc);
+ struct intel_uncore *uncore = &dev_priv->uncore;
u32 status;
int i;
int ret;
*action != INTEL_GUC_ACTION_DEREGISTER_COMMAND_TRANSPORT_BUFFER);
mutex_lock(&guc->send_mutex);
- intel_uncore_forcewake_get(dev_priv, guc->send_regs.fw_domains);
+ intel_uncore_forcewake_get(uncore, guc->send_regs.fw_domains);
for (i = 0; i < len; i++)
- I915_WRITE(guc_send_reg(guc, i), action[i]);
+ intel_uncore_write(uncore, guc_send_reg(guc, i), action[i]);
- POSTING_READ(guc_send_reg(guc, i - 1));
+ intel_uncore_posting_read(uncore, guc_send_reg(guc, i - 1));
intel_guc_notify(guc);
* No GuC command should ever take longer than 10ms.
* Fast commands should still complete in 10us.
*/
- ret = __intel_wait_for_register_fw(dev_priv,
+ ret = __intel_wait_for_register_fw(uncore,
guc_send_reg(guc, 0),
INTEL_GUC_MSG_TYPE_MASK,
INTEL_GUC_MSG_TYPE_RESPONSE <<
ret = INTEL_GUC_MSG_TO_DATA(status);
out:
- intel_uncore_forcewake_put(dev_priv, guc->send_regs.fw_domains);
+ intel_uncore_forcewake_put(uncore, guc->send_regs.fw_domains);
mutex_unlock(&guc->send_mutex);
return ret;
spin_unlock(&guc->irq_lock);
enable_rpm_wakeref_asserts(dev_priv);
- intel_guc_to_host_process_recv_msg(guc, msg);
+ intel_guc_to_host_process_recv_msg(guc, &msg, 1);
}
-void intel_guc_to_host_process_recv_msg(struct intel_guc *guc, u32 msg)
+int intel_guc_to_host_process_recv_msg(struct intel_guc *guc,
+ const u32 *payload, u32 len)
{
+ u32 msg;
+
+ if (unlikely(!len))
+ return -EPROTO;
+
/* Make sure to handle only enabled messages */
- msg &= guc->msg_enabled_mask;
+ msg = payload[0] & guc->msg_enabled_mask;
if (msg & (INTEL_GUC_RECV_MSG_FLUSH_LOG_BUFFER |
INTEL_GUC_RECV_MSG_CRASH_DUMP_POSTED))
intel_guc_log_handle_flush_event(&guc->log);
+
+ return 0;
}
int intel_guc_sample_forcewake(struct intel_guc *guc)
if (ret)
return ret;
- ret = __intel_wait_for_register(dev_priv, SOFT_SCRATCH(14),
+ ret = __intel_wait_for_register(&dev_priv->uncore, SOFT_SCRATCH(14),
INTEL_GUC_SLEEP_STATE_INVALID_MASK,
0, 0, 10, &status);
if (ret)
#include "intel_guc_log.h"
#include "intel_guc_reg.h"
#include "intel_uc_fw.h"
+#include "i915_utils.h"
#include "i915_vma.h"
struct guc_preempt_work {
void intel_guc_to_host_event_handler(struct intel_guc *guc);
void intel_guc_to_host_event_handler_nop(struct intel_guc *guc);
void intel_guc_to_host_event_handler_mmio(struct intel_guc *guc);
-void intel_guc_to_host_process_recv_msg(struct intel_guc *guc, u32 msg);
+int intel_guc_to_host_process_recv_msg(struct intel_guc *guc,
+ const u32 *payload, u32 len);
int intel_guc_sample_forcewake(struct intel_guc *guc);
int intel_guc_auth_huc(struct intel_guc *guc, u32 rsa_offset);
int intel_guc_suspend(struct intel_guc *guc);
* to find it. Note that we have to skip our header (1 page),
* because our GuC shared data is there.
*/
- kernel_ctx_vma = to_intel_context(dev_priv->kernel_context,
- dev_priv->engine[RCS])->state;
+ kernel_ctx_vma = dev_priv->engine[RCS0]->kernel_context->state;
blob->ads.golden_context_lrca =
intel_guc_ggtt_offset(guc, kernel_ctx_vma) + skipped_offset;
return err;
}
-static bool ctch_is_open(struct intel_guc_ct_channel *ctch)
-{
- return ctch->vma != NULL;
-}
-
static int ctch_init(struct intel_guc *guc,
struct intel_guc_ct_channel *ctch)
{
static void ctch_fini(struct intel_guc *guc,
struct intel_guc_ct_channel *ctch)
{
+ GEM_BUG_ON(ctch->enabled);
+
i915_vma_unpin_and_release(&ctch->vma, I915_VMA_RELEASE_MAP);
}
-static int ctch_open(struct intel_guc *guc,
- struct intel_guc_ct_channel *ctch)
+static int ctch_enable(struct intel_guc *guc,
+ struct intel_guc_ct_channel *ctch)
{
u32 base;
int err;
int i;
- CT_DEBUG_DRIVER("CT: channel %d reopen=%s\n",
- ctch->owner, yesno(ctch_is_open(ctch)));
+ GEM_BUG_ON(!ctch->vma);
- if (!ctch->vma) {
- err = ctch_init(guc, ctch);
- if (unlikely(err))
- goto err_out;
- GEM_BUG_ON(!ctch->vma);
- }
+ GEM_BUG_ON(ctch->enabled);
/* vma should be already allocated and map'ed */
base = intel_guc_ggtt_offset(guc, ctch->vma);
base + PAGE_SIZE/4 * CTB_RECV,
INTEL_GUC_CT_BUFFER_TYPE_RECV);
if (unlikely(err))
- goto err_fini;
+ goto err_out;
err = guc_action_register_ct_buffer(guc,
base + PAGE_SIZE/4 * CTB_SEND,
if (unlikely(err))
goto err_deregister;
+ ctch->enabled = true;
+
return 0;
err_deregister:
guc_action_deregister_ct_buffer(guc,
ctch->owner,
INTEL_GUC_CT_BUFFER_TYPE_RECV);
-err_fini:
- ctch_fini(guc, ctch);
err_out:
DRM_ERROR("CT: can't open channel %d; err=%d\n", ctch->owner, err);
return err;
}
-static void ctch_close(struct intel_guc *guc,
- struct intel_guc_ct_channel *ctch)
+static void ctch_disable(struct intel_guc *guc,
+ struct intel_guc_ct_channel *ctch)
{
- GEM_BUG_ON(!ctch_is_open(ctch));
+ GEM_BUG_ON(!ctch->enabled);
+
+ ctch->enabled = false;
guc_action_deregister_ct_buffer(guc,
ctch->owner,
guc_action_deregister_ct_buffer(guc,
ctch->owner,
INTEL_GUC_CT_BUFFER_TYPE_RECV);
- ctch_fini(guc, ctch);
}
static u32 ctch_get_next_fence(struct intel_guc_ct_channel *ctch)
u32 fence;
int err;
- GEM_BUG_ON(!ctch_is_open(ctch));
+ GEM_BUG_ON(!ctch->enabled);
GEM_BUG_ON(!len);
GEM_BUG_ON(len & ~GUC_CT_MSG_LEN_MASK);
GEM_BUG_ON(!response_buf && response_buf_size);
u32 action, u32 len, const u32 *payload)
{
struct intel_guc *guc = ct_to_guc(ct);
+ int ret;
CT_DEBUG_DRIVER("CT: request %x %*ph\n", action, 4 * len, payload);
switch (action) {
case INTEL_GUC_ACTION_DEFAULT:
- if (unlikely(len < 1))
+ ret = intel_guc_to_host_process_recv_msg(guc, payload, len);
+ if (unlikely(ret))
goto fail_unexpected;
- intel_guc_to_host_process_recv_msg(guc, *payload);
break;
default:
u32 msg[GUC_CT_MSG_LEN_MASK + 1]; /* one extra dw for the header */
int err = 0;
- if (!ctch_is_open(ctch))
+ if (!ctch->enabled)
return;
do {
ct_process_host_channel(ct);
}
+/**
+ * intel_guc_ct_init - Init CT communication
+ * @ct: pointer to CT struct
+ *
+ * Allocate memory required for communication via
+ * the CT channel.
+ *
+ * Shall only be called for platforms with HAS_GUC_CT.
+ *
+ * Return: 0 on success, a negative errno code on failure.
+ */
+int intel_guc_ct_init(struct intel_guc_ct *ct)
+{
+ struct intel_guc *guc = ct_to_guc(ct);
+ struct intel_guc_ct_channel *ctch = &ct->host_channel;
+ int err;
+
+ err = ctch_init(guc, ctch);
+ if (unlikely(err)) {
+ DRM_ERROR("CT: can't open channel %d; err=%d\n",
+ ctch->owner, err);
+ return err;
+ }
+
+ GEM_BUG_ON(!ctch->vma);
+ return 0;
+}
+
+/**
+ * intel_guc_ct_fini - Fini CT communication
+ * @ct: pointer to CT struct
+ *
+ * Deallocate memory required for communication via
+ * the CT channel.
+ *
+ * Shall only be called for platforms with HAS_GUC_CT.
+ */
+void intel_guc_ct_fini(struct intel_guc_ct *ct)
+{
+ struct intel_guc *guc = ct_to_guc(ct);
+ struct intel_guc_ct_channel *ctch = &ct->host_channel;
+
+ ctch_fini(guc, ctch);
+}
+
/**
* intel_guc_ct_enable - Enable buffer based command transport.
* @ct: pointer to CT struct
GEM_BUG_ON(!HAS_GUC_CT(i915));
- err = ctch_open(guc, ctch);
+ if (ctch->enabled)
+ return 0;
+
+ err = ctch_enable(guc, ctch);
if (unlikely(err))
return err;
GEM_BUG_ON(!HAS_GUC_CT(i915));
- if (!ctch_is_open(ctch))
+ if (!ctch->enabled)
return;
- ctch_close(guc, ctch);
+ ctch_disable(guc, ctch);
/* Disable send */
guc->send = intel_guc_send_nop;
struct intel_guc_ct_buffer ctbs[2];
u32 owner;
u32 next_fence;
+ bool enabled;
};
/** Holds all command transport channels.
};
void intel_guc_ct_init_early(struct intel_guc_ct *ct);
+int intel_guc_ct_init(struct intel_guc_ct *ct);
+void intel_guc_ct_fini(struct intel_guc_ct *ct);
int intel_guc_ct_enable(struct intel_guc_ct *ct);
void intel_guc_ct_disable(struct intel_guc_ct *ct);
GEM_BUG_ON(guc_fw->type != INTEL_UC_FW_TYPE_GUC);
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
guc_prepare_xfer(guc);
ret = guc_xfer_ucode(guc, vma);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
return ret;
}
void intel_guc_log_relay_close(struct intel_guc_log *log)
{
+ struct intel_guc *guc = log_to_guc(log);
+ struct drm_i915_private *i915 = guc_to_i915(guc);
+
guc_log_disable_flush_events(log);
+ synchronize_irq(i915->drm.irq);
+
flush_work(&log->relay.flush_work);
mutex_lock(&log->relay.lock);
desc->db_id = client->doorbell_id;
for_each_engine_masked(engine, dev_priv, client->engines, tmp) {
- struct intel_context *ce = to_intel_context(ctx, engine);
+ struct intel_context *ce = intel_context_lookup(ctx, engine);
u32 guc_engine_id = engine->guc_id;
struct guc_execlist_context *lrc = &desc->lrc[guc_engine_id];
* for now who owns a GuC client. But for future owner of GuC
* client, need to make sure lrc is pinned prior to enter here.
*/
- if (!ce->state)
+ if (!ce || !ce->state)
break; /* XXX: continue? */
/*
spin_lock(&client->wq_lock);
guc_wq_item_append(client, engine->guc_id, ctx_desc,
- ring_tail, rq->global_seqno);
+ ring_tail, rq->fence.seqno);
guc_ring_doorbell(client);
client->submissions[engine->id] += 1;
preempt_work[engine->id]);
struct intel_guc_client *client = guc->preempt_client;
struct guc_stage_desc *stage_desc = __get_stage_desc(client);
- struct intel_context *ce = to_intel_context(client->owner, engine);
+ struct intel_context *ce = engine->preempt_context;
u32 data[7];
if (!ce->ring->emit) { /* recreate upon load/resume */
u32 *cs;
cs = ce->ring->vaddr;
- if (engine->id == RCS) {
+ if (engine->class == RENDER_CLASS) {
cs = gen8_emit_ggtt_write_rcs(cs,
GUC_PREEMPT_FINISHED,
addr,
} else {
cs = gen8_emit_ggtt_write(cs,
GUC_PREEMPT_FINISHED,
- addr);
+ addr,
+ 0);
*cs++ = MI_NOOP;
*cs++ = MI_NOOP;
}
struct guc_ctx_report *report =
&data->preempt_ctx_report[engine->guc_id];
- WARN_ON(wait_for_atomic(report->report_return_status ==
- INTEL_GUC_REPORT_STATUS_COMPLETE,
- GUC_PREEMPT_POSTPROCESS_DELAY_MS));
+ if (wait_for_atomic(report->report_return_status ==
+ INTEL_GUC_REPORT_STATUS_COMPLETE,
+ GUC_PREEMPT_POSTPROCESS_DELAY_MS))
+ DRM_ERROR("Timed out waiting for GuC preemption report\n");
/*
* GuC is expecting that we're also going to clear the affected context
* counter, let's also reset the return status to not depend on GuC
static inline int port_prio(const struct execlist_port *port)
{
- return rq_prio(port_request(port));
+ return rq_prio(port_request(port)) | __NO_PREEMPTION;
}
static bool __guc_dequeue(struct intel_engine_cs *engine)
}
rb_erase_cached(&p->node, &execlists->queue);
- if (p->priority != I915_PRIORITY_NORMAL)
- kmem_cache_free(engine->i915->priorities, p);
+ i915_priolist_free(p);
}
done:
execlists->queue_priority_hint =
flush_workqueue(engine->i915->guc.preempt_wq);
}
+static void guc_reset(struct intel_engine_cs *engine, bool stalled)
+{
+ struct intel_engine_execlists * const execlists = &engine->execlists;
+ struct i915_request *rq;
+ unsigned long flags;
+
+ spin_lock_irqsave(&engine->timeline.lock, flags);
+
+ execlists_cancel_port_requests(execlists);
+
+ /* Push back any incomplete requests for replay after the reset. */
+ rq = execlists_unwind_incomplete_requests(execlists);
+ if (!rq)
+ goto out_unlock;
+
+ if (!i915_request_started(rq))
+ stalled = false;
+
+ i915_reset_request(rq, stalled);
+ intel_lr_context_reset(engine, rq->hw_context, rq->head, stalled);
+
+out_unlock:
+ spin_unlock_irqrestore(&engine->timeline.lock, flags);
+}
+
+static void guc_cancel_requests(struct intel_engine_cs *engine)
+{
+ struct intel_engine_execlists * const execlists = &engine->execlists;
+ struct i915_request *rq, *rn;
+ struct rb_node *rb;
+ unsigned long flags;
+
+ GEM_TRACE("%s\n", engine->name);
+
+ /*
+ * Before we call engine->cancel_requests(), we should have exclusive
+ * access to the submission state. This is arranged for us by the
+ * caller disabling the interrupt generation, the tasklet and other
+ * threads that may then access the same state, giving us a free hand
+ * to reset state. However, we still need to let lockdep be aware that
+ * we know this state may be accessed in hardirq context, so we
+ * disable the irq around this manipulation and we want to keep
+ * the spinlock focused on its duties and not accidentally conflate
+ * coverage to the submission's irq state. (Similarly, although we
+ * shouldn't need to disable irq around the manipulation of the
+ * submission's irq state, we also wish to remind ourselves that
+ * it is irq state.)
+ */
+ spin_lock_irqsave(&engine->timeline.lock, flags);
+
+ /* Cancel the requests on the HW and clear the ELSP tracker. */
+ execlists_cancel_port_requests(execlists);
+
+ /* Mark all executing requests as skipped. */
+ list_for_each_entry(rq, &engine->timeline.requests, link) {
+ if (!i915_request_signaled(rq))
+ dma_fence_set_error(&rq->fence, -EIO);
+
+ i915_request_mark_complete(rq);
+ }
+
+ /* Flush the queued requests to the timeline list (for retiring). */
+ while ((rb = rb_first_cached(&execlists->queue))) {
+ struct i915_priolist *p = to_priolist(rb);
+ int i;
+
+ priolist_for_each_request_consume(rq, rn, p, i) {
+ list_del_init(&rq->sched.link);
+ __i915_request_submit(rq);
+ dma_fence_set_error(&rq->fence, -EIO);
+ i915_request_mark_complete(rq);
+ }
+
+ rb_erase_cached(&p->node, &execlists->queue);
+ i915_priolist_free(p);
+ }
+
+ /* Remaining _unready_ requests will be nop'ed when submitted */
+
+ execlists->queue_priority_hint = INT_MIN;
+ execlists->queue = RB_ROOT_CACHED;
+ GEM_BUG_ON(port_isset(execlists->port));
+
+ spin_unlock_irqrestore(&engine->timeline.lock, flags);
+}
+
+static void guc_reset_finish(struct intel_engine_cs *engine)
+{
+ struct intel_engine_execlists * const execlists = &engine->execlists;
+
+ if (__tasklet_enable(&execlists->tasklet))
+ /* And kick in case we missed a new request submission. */
+ tasklet_hi_schedule(&execlists->tasklet);
+
+ GEM_TRACE("%s: depth->%d\n", engine->name,
+ atomic_read(&execlists->tasklet.count));
+}
+
/*
* Everything below here is concerned with setup & teardown, and is
* therefore not part of the somewhat time-critical batch-submission
GEM_BUG_ON(guc->preempt_client);
client = guc_client_alloc(dev_priv,
- INTEL_INFO(dev_priv)->ring_mask,
+ INTEL_INFO(dev_priv)->engine_mask,
GUC_CLIENT_PRIORITY_KMD_NORMAL,
dev_priv->kernel_context);
if (IS_ERR(client)) {
if (dev_priv->preempt_context) {
client = guc_client_alloc(dev_priv,
- INTEL_INFO(dev_priv)->ring_mask,
+ INTEL_INFO(dev_priv)->engine_mask,
GUC_CLIENT_PRIORITY_KMD_HIGH,
dev_priv->preempt_context);
if (IS_ERR(client)) {
static void guc_submission_park(struct intel_engine_cs *engine)
{
intel_engine_unpin_breadcrumbs_irq(engine);
+ engine->flags &= ~I915_ENGINE_NEEDS_BREADCRUMB_TASKLET;
}
static void guc_submission_unpark(struct intel_engine_cs *engine)
{
+ engine->flags |= I915_ENGINE_NEEDS_BREADCRUMB_TASKLET;
intel_engine_pin_breadcrumbs_irq(engine);
}
engine->unpark = guc_submission_unpark;
engine->reset.prepare = guc_reset_prepare;
+ engine->reset.reset = guc_reset;
+ engine->reset.finish = guc_reset_finish;
+
+ engine->cancel_requests = guc_cancel_requests;
engine->flags &= ~I915_ENGINE_SUPPORTS_STATS;
}
#include "i915_gem.h"
#include "i915_selftest.h"
+#include "intel_engine_types.h"
struct drm_i915_private;
int slice;
int subslice;
- if (engine->id != RCS)
+ if (engine->id != RCS0)
return true;
intel_engine_get_instdone(engine, &instdone);
* and break the hang. This should work on
* all but the second generation chipsets.
*/
- tmp = I915_READ_CTL(engine);
+ tmp = ENGINE_READ(engine, RING_CTL);
if (tmp & RING_WAIT) {
- i915_handle_error(dev_priv, BIT(engine->id), 0,
+ i915_handle_error(dev_priv, engine->mask, 0,
"stuck wait on %s", engine->name);
- I915_WRITE_CTL(engine, tmp);
+ ENGINE_WRITE(engine, RING_CTL, tmp);
return ENGINE_WAIT_KICK;
}
struct hangcheck *hc)
{
hc->acthd = intel_engine_get_active_head(engine);
- hc->seqno = intel_engine_get_seqno(engine);
+ hc->seqno = intel_engine_get_hangcheck_seqno(engine);
}
static void hangcheck_store_sample(struct intel_engine_cs *engine,
const struct hangcheck *hc)
{
engine->hangcheck.acthd = hc->acthd;
- engine->hangcheck.seqno = hc->seqno;
+ engine->hangcheck.last_seqno = hc->seqno;
}
static enum intel_engine_hangcheck_action
hangcheck_get_action(struct intel_engine_cs *engine,
const struct hangcheck *hc)
{
- if (engine->hangcheck.seqno != hc->seqno)
+ if (engine->hangcheck.last_seqno != hc->seqno)
return ENGINE_ACTIVE_SEQNO;
if (intel_engine_is_idle(engine))
unsigned int stuck)
{
struct intel_engine_cs *engine;
+ intel_engine_mask_t tmp;
char msg[80];
- unsigned int tmp;
int len;
/* If some rings hung but others were still busy, only
if (!READ_ONCE(dev_priv->gt.awake))
return;
- if (i915_terminally_wedged(&dev_priv->gpu_error))
+ if (i915_terminally_wedged(dev_priv))
return;
/* As enabling the GPU requires fairly extensive mmio access,
* periodically arm the mmio checker to see if we are triggering
* any invalid access.
*/
- intel_uncore_arm_unclaimed_mmio_detection(dev_priv);
+ intel_uncore_arm_unclaimed_mmio_detection(&dev_priv->uncore);
for_each_engine(engine, dev_priv, id) {
struct hangcheck hc;
hangcheck_store_sample(engine, &hc);
if (hc.stalled) {
- hung |= intel_engine_flag(engine);
+ hung |= engine->mask;
if (hc.action != ENGINE_DEAD)
- stuck |= intel_engine_flag(engine);
+ stuck |= engine->mask;
}
if (hc.wedged)
- wedged |= intel_engine_flag(engine);
+ wedged |= engine->mask;
}
if (GEM_SHOW_DEBUG() && (hung | stuck)) {
* Sean Paul <seanpaul@chromium.org>
*/
-#include <drm/drm_hdcp.h>
+#include <linux/component.h>
#include <linux/i2c.h>
#include <linux/random.h>
-#include "intel_drv.h"
+#include <drm/drm_hdcp.h>
+#include <drm/i915_component.h>
+
#include "i915_reg.h"
+#include "intel_drv.h"
+#include "intel_hdcp.h"
#define KEY_LOAD_TRIES 5
#define ENCRYPT_STATUS_CHANGE_TIMEOUT_MS 50
+#define HDCP2_LC_RETRY_CNT 3
static
bool intel_hdcp_is_ksv_valid(u8 *ksv)
return capable;
}
+/* Is HDCP2.2 capable on Platform and Sink */
+static bool intel_hdcp2_capable(struct intel_connector *connector)
+{
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ bool capable = false;
+
+ /* I915 support for HDCP2.2 */
+ if (!hdcp->hdcp2_supported)
+ return false;
+
+ /* MEI interface is solid */
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ if (!dev_priv->hdcp_comp_added || !dev_priv->hdcp_master) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return false;
+ }
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+
+ /* Sink's capability for HDCP2.2 */
+ hdcp->shim->hdcp_2_2_capable(intel_dig_port, &capable);
+
+ return capable;
+}
+
+static inline bool intel_hdcp_in_use(struct intel_connector *connector)
+{
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ enum port port = connector->encoder->port;
+ u32 reg;
+
+ reg = I915_READ(PORT_HDCP_STATUS(port));
+ return reg & HDCP_STATUS_ENC;
+}
+
+static inline bool intel_hdcp2_in_use(struct intel_connector *connector)
+{
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ enum port port = connector->encoder->port;
+ u32 reg;
+
+ reg = I915_READ(HDCP2_STATUS_DDI(port));
+ return reg & LINK_ENCRYPTION_STATUS;
+}
+
static int intel_hdcp_poll_ksv_fifo(struct intel_digital_port *intel_dig_port,
const struct intel_hdcp_shim *shim)
{
}
/* Wait for the keys to load (500us) */
- ret = __intel_wait_for_register(dev_priv, HDCP_KEY_STATUS,
+ ret = __intel_wait_for_register(&dev_priv->uncore, HDCP_KEY_STATUS,
HDCP_KEY_LOAD_DONE, HDCP_KEY_LOAD_DONE,
10, 1, &val);
if (ret)
static int intel_write_sha_text(struct drm_i915_private *dev_priv, u32 sha_text)
{
I915_WRITE(HDCP_SHA_TEXT, sha_text);
- if (intel_wait_for_register(dev_priv, HDCP_REP_CTL,
+ if (intel_wait_for_register(&dev_priv->uncore, HDCP_REP_CTL,
HDCP_SHA1_READY, HDCP_SHA1_READY, 1)) {
DRM_ERROR("Timed out waiting for SHA1 ready\n");
return -ETIMEDOUT;
/* Tell the HW we're done with the hash and wait for it to ACK */
I915_WRITE(HDCP_REP_CTL, rep_ctl | HDCP_SHA1_COMPLETE_HASH);
- if (intel_wait_for_register(dev_priv, HDCP_REP_CTL,
+ if (intel_wait_for_register(&dev_priv->uncore, HDCP_REP_CTL,
HDCP_SHA1_COMPLETE,
HDCP_SHA1_COMPLETE, 1)) {
DRM_ERROR("Timed out waiting for SHA1 complete\n");
I915_WRITE(PORT_HDCP_CONF(port), HDCP_CONF_CAPTURE_AN);
/* Wait for An to be acquired */
- if (intel_wait_for_register(dev_priv, PORT_HDCP_STATUS(port),
+ if (intel_wait_for_register(&dev_priv->uncore, PORT_HDCP_STATUS(port),
HDCP_STATUS_AN_READY,
HDCP_STATUS_AN_READY, 1)) {
DRM_ERROR("Timed out waiting for An\n");
}
/* Wait for encryption confirmation */
- if (intel_wait_for_register(dev_priv, PORT_HDCP_STATUS(port),
+ if (intel_wait_for_register(&dev_priv->uncore, PORT_HDCP_STATUS(port),
HDCP_STATUS_ENC, HDCP_STATUS_ENC,
ENCRYPT_STATUS_CHANGE_TIMEOUT_MS)) {
DRM_ERROR("Timed out waiting for encryption\n");
DRM_DEBUG_KMS("[%s:%d] HDCP is being disabled...\n",
connector->base.name, connector->base.base.id);
+ hdcp->hdcp_encrypted = false;
I915_WRITE(PORT_HDCP_CONF(port), 0);
- if (intel_wait_for_register(dev_priv, PORT_HDCP_STATUS(port), ~0, 0,
+ if (intel_wait_for_register(&dev_priv->uncore,
+ PORT_HDCP_STATUS(port), ~0, 0,
ENCRYPT_STATUS_CHANGE_TIMEOUT_MS)) {
DRM_ERROR("Failed to disable HDCP, timeout clearing status\n");
return -ETIMEDOUT;
/* Incase of authentication failures, HDCP spec expects reauth. */
for (i = 0; i < tries; i++) {
ret = intel_hdcp_auth(conn_to_dig_port(connector), hdcp->shim);
- if (!ret)
+ if (!ret) {
+ hdcp->hdcp_encrypted = true;
return 0;
+ }
DRM_DEBUG_KMS("HDCP Auth failure (%d)\n", ret);
return container_of(hdcp, struct intel_connector, hdcp);
}
-static void intel_hdcp_check_work(struct work_struct *work)
+/* Implements Part 3 of the HDCP authorization procedure */
+static int intel_hdcp_check_link(struct intel_connector *connector)
{
- struct intel_hdcp *hdcp = container_of(to_delayed_work(work),
- struct intel_hdcp,
- check_work);
- struct intel_connector *connector = intel_hdcp_to_connector(hdcp);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ struct drm_i915_private *dev_priv = connector->base.dev->dev_private;
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ enum port port = intel_dig_port->base.port;
+ int ret = 0;
- if (!intel_hdcp_check_link(connector))
- schedule_delayed_work(&hdcp->check_work,
- DRM_HDCP_CHECK_PERIOD_MS);
+ mutex_lock(&hdcp->mutex);
+
+ /* Check_link valid only when HDCP1.4 is enabled */
+ if (hdcp->value != DRM_MODE_CONTENT_PROTECTION_ENABLED ||
+ !hdcp->hdcp_encrypted) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (WARN_ON(!intel_hdcp_in_use(connector))) {
+ DRM_ERROR("%s:%d HDCP link stopped encryption,%x\n",
+ connector->base.name, connector->base.base.id,
+ I915_READ(PORT_HDCP_STATUS(port)));
+ ret = -ENXIO;
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_DESIRED;
+ schedule_work(&hdcp->prop_work);
+ goto out;
+ }
+
+ if (hdcp->shim->check_link(intel_dig_port)) {
+ if (hdcp->value != DRM_MODE_CONTENT_PROTECTION_UNDESIRED) {
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_ENABLED;
+ schedule_work(&hdcp->prop_work);
+ }
+ goto out;
+ }
+
+ DRM_DEBUG_KMS("[%s:%d] HDCP link failed, retrying authentication\n",
+ connector->base.name, connector->base.base.id);
+
+ ret = _intel_hdcp_disable(connector);
+ if (ret) {
+ DRM_ERROR("Failed to disable hdcp (%d)\n", ret);
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_DESIRED;
+ schedule_work(&hdcp->prop_work);
+ goto out;
+ }
+
+ ret = _intel_hdcp_enable(connector);
+ if (ret) {
+ DRM_ERROR("Failed to enable hdcp (%d)\n", ret);
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_DESIRED;
+ schedule_work(&hdcp->prop_work);
+ goto out;
+ }
+
+out:
+ mutex_unlock(&hdcp->mutex);
+ return ret;
}
static void intel_hdcp_prop_work(struct work_struct *work)
return INTEL_GEN(dev_priv) >= 9 && port < PORT_E;
}
-int intel_hdcp_init(struct intel_connector *connector,
- const struct intel_hdcp_shim *shim)
+static int
+hdcp2_prepare_ake_init(struct intel_connector *connector,
+ struct hdcp2_ake_init *ake_data)
{
- struct intel_hdcp *hdcp = &connector->hdcp;
+ struct hdcp_port_data *data = &connector->hdcp.port_data;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
int ret;
- ret = drm_connector_attach_content_protection_property(
- &connector->base);
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
+
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
+ }
+
+ ret = comp->ops->initiate_hdcp2_session(comp->mei_dev, data, ake_data);
if (ret)
- return ret;
+ DRM_DEBUG_KMS("Prepare_ake_init failed. %d\n", ret);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
- hdcp->shim = shim;
- mutex_init(&hdcp->mutex);
- INIT_DELAYED_WORK(&hdcp->check_work, intel_hdcp_check_work);
- INIT_WORK(&hdcp->prop_work, intel_hdcp_prop_work);
- return 0;
+ return ret;
}
-int intel_hdcp_enable(struct intel_connector *connector)
+static int
+hdcp2_verify_rx_cert_prepare_km(struct intel_connector *connector,
+ struct hdcp2_ake_send_cert *rx_cert,
+ bool *paired,
+ struct hdcp2_ake_no_stored_km *ek_pub_km,
+ size_t *msg_sz)
{
- struct intel_hdcp *hdcp = &connector->hdcp;
+ struct hdcp_port_data *data = &connector->hdcp.port_data;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
int ret;
- if (!hdcp->shim)
- return -ENOENT;
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
- mutex_lock(&hdcp->mutex);
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
+ }
- ret = _intel_hdcp_enable(connector);
- if (ret)
- goto out;
+ ret = comp->ops->verify_receiver_cert_prepare_km(comp->mei_dev, data,
+ rx_cert, paired,
+ ek_pub_km, msg_sz);
+ if (ret < 0)
+ DRM_DEBUG_KMS("Verify rx_cert failed. %d\n", ret);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
- hdcp->value = DRM_MODE_CONTENT_PROTECTION_ENABLED;
- schedule_work(&hdcp->prop_work);
- schedule_delayed_work(&hdcp->check_work,
- DRM_HDCP_CHECK_PERIOD_MS);
-out:
- mutex_unlock(&hdcp->mutex);
return ret;
}
-int intel_hdcp_disable(struct intel_connector *connector)
+static int hdcp2_verify_hprime(struct intel_connector *connector,
+ struct hdcp2_ake_send_hprime *rx_hprime)
{
- struct intel_hdcp *hdcp = &connector->hdcp;
- int ret = 0;
+ struct hdcp_port_data *data = &connector->hdcp.port_data;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
+ int ret;
- if (!hdcp->shim)
- return -ENOENT;
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
- mutex_lock(&hdcp->mutex);
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
+ }
- if (hdcp->value != DRM_MODE_CONTENT_PROTECTION_UNDESIRED) {
- hdcp->value = DRM_MODE_CONTENT_PROTECTION_UNDESIRED;
- ret = _intel_hdcp_disable(connector);
+ ret = comp->ops->verify_hprime(comp->mei_dev, data, rx_hprime);
+ if (ret < 0)
+ DRM_DEBUG_KMS("Verify hprime failed. %d\n", ret);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+
+ return ret;
+}
+
+static int
+hdcp2_store_pairing_info(struct intel_connector *connector,
+ struct hdcp2_ake_send_pairing_info *pairing_info)
+{
+ struct hdcp_port_data *data = &connector->hdcp.port_data;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
+ int ret;
+
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
+
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
}
- mutex_unlock(&hdcp->mutex);
- cancel_delayed_work_sync(&hdcp->check_work);
+ ret = comp->ops->store_pairing_info(comp->mei_dev, data, pairing_info);
+ if (ret < 0)
+ DRM_DEBUG_KMS("Store pairing info failed. %d\n", ret);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+
return ret;
}
-void intel_hdcp_atomic_check(struct drm_connector *connector,
- struct drm_connector_state *old_state,
- struct drm_connector_state *new_state)
+static int
+hdcp2_prepare_lc_init(struct intel_connector *connector,
+ struct hdcp2_lc_init *lc_init)
{
- u64 old_cp = old_state->content_protection;
- u64 new_cp = new_state->content_protection;
- struct drm_crtc_state *crtc_state;
+ struct hdcp_port_data *data = &connector->hdcp.port_data;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
+ int ret;
- if (!new_state->crtc) {
- /*
- * If the connector is being disabled with CP enabled, mark it
- * desired so it's re-enabled when the connector is brought back
- */
- if (old_cp == DRM_MODE_CONTENT_PROTECTION_ENABLED)
- new_state->content_protection =
- DRM_MODE_CONTENT_PROTECTION_DESIRED;
- return;
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
+
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
}
- /*
- * Nothing to do if the state didn't change, or HDCP was activated since
- * the last commit
- */
- if (old_cp == new_cp ||
- (old_cp == DRM_MODE_CONTENT_PROTECTION_DESIRED &&
- new_cp == DRM_MODE_CONTENT_PROTECTION_ENABLED))
- return;
+ ret = comp->ops->initiate_locality_check(comp->mei_dev, data, lc_init);
+ if (ret < 0)
+ DRM_DEBUG_KMS("Prepare lc_init failed. %d\n", ret);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
- crtc_state = drm_atomic_get_new_crtc_state(new_state->state,
- new_state->crtc);
- crtc_state->mode_changed = true;
+ return ret;
}
-/* Implements Part 3 of the HDCP authorization procedure */
-int intel_hdcp_check_link(struct intel_connector *connector)
+static int
+hdcp2_verify_lprime(struct intel_connector *connector,
+ struct hdcp2_lc_send_lprime *rx_lprime)
{
- struct intel_hdcp *hdcp = &connector->hdcp;
- struct drm_i915_private *dev_priv = connector->base.dev->dev_private;
- struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
- enum port port = intel_dig_port->base.port;
- int ret = 0;
+ struct hdcp_port_data *data = &connector->hdcp.port_data;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
+ int ret;
- if (!hdcp->shim)
- return -ENOENT;
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
- mutex_lock(&hdcp->mutex);
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
+ }
- if (hdcp->value == DRM_MODE_CONTENT_PROTECTION_UNDESIRED)
- goto out;
+ ret = comp->ops->verify_lprime(comp->mei_dev, data, rx_lprime);
+ if (ret < 0)
+ DRM_DEBUG_KMS("Verify L_Prime failed. %d\n", ret);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
- if (!(I915_READ(PORT_HDCP_STATUS(port)) & HDCP_STATUS_ENC)) {
- DRM_ERROR("%s:%d HDCP check failed: link is not encrypted,%x\n",
- connector->base.name, connector->base.base.id,
- I915_READ(PORT_HDCP_STATUS(port)));
- ret = -ENXIO;
- hdcp->value = DRM_MODE_CONTENT_PROTECTION_DESIRED;
- schedule_work(&hdcp->prop_work);
- goto out;
+ return ret;
+}
+
+static int hdcp2_prepare_skey(struct intel_connector *connector,
+ struct hdcp2_ske_send_eks *ske_data)
+{
+ struct hdcp_port_data *data = &connector->hdcp.port_data;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
+ int ret;
+
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
+
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
}
- if (hdcp->shim->check_link(intel_dig_port)) {
- if (hdcp->value != DRM_MODE_CONTENT_PROTECTION_UNDESIRED) {
- hdcp->value = DRM_MODE_CONTENT_PROTECTION_ENABLED;
- schedule_work(&hdcp->prop_work);
- }
- goto out;
+ ret = comp->ops->get_session_key(comp->mei_dev, data, ske_data);
+ if (ret < 0)
+ DRM_DEBUG_KMS("Get session key failed. %d\n", ret);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+
+ return ret;
+}
+
+static int
+hdcp2_verify_rep_topology_prepare_ack(struct intel_connector *connector,
+ struct hdcp2_rep_send_receiverid_list
+ *rep_topology,
+ struct hdcp2_rep_send_ack *rep_send_ack)
+{
+ struct hdcp_port_data *data = &connector->hdcp.port_data;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
+ int ret;
+
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
+
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
}
- DRM_DEBUG_KMS("[%s:%d] HDCP link failed, retrying authentication\n",
- connector->base.name, connector->base.base.id);
+ ret = comp->ops->repeater_check_flow_prepare_ack(comp->mei_dev, data,
+ rep_topology,
+ rep_send_ack);
+ if (ret < 0)
+ DRM_DEBUG_KMS("Verify rep topology failed. %d\n", ret);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
- ret = _intel_hdcp_disable(connector);
- if (ret) {
- DRM_ERROR("Failed to disable hdcp (%d)\n", ret);
- hdcp->value = DRM_MODE_CONTENT_PROTECTION_DESIRED;
- schedule_work(&hdcp->prop_work);
- goto out;
+ return ret;
+}
+
+static int
+hdcp2_verify_mprime(struct intel_connector *connector,
+ struct hdcp2_rep_stream_ready *stream_ready)
+{
+ struct hdcp_port_data *data = &connector->hdcp.port_data;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
+ int ret;
+
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
+
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
}
- ret = _intel_hdcp_enable(connector);
- if (ret) {
- DRM_DEBUG_KMS("Failed to enable hdcp (%d)\n", ret);
- hdcp->value = DRM_MODE_CONTENT_PROTECTION_DESIRED;
- schedule_work(&hdcp->prop_work);
- goto out;
+ ret = comp->ops->verify_mprime(comp->mei_dev, data, stream_ready);
+ if (ret < 0)
+ DRM_DEBUG_KMS("Verify mprime failed. %d\n", ret);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+
+ return ret;
+}
+
+static int hdcp2_authenticate_port(struct intel_connector *connector)
+{
+ struct hdcp_port_data *data = &connector->hdcp.port_data;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
+ int ret;
+
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
+
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
}
-out:
- mutex_unlock(&hdcp->mutex);
+ ret = comp->ops->enable_hdcp_authentication(comp->mei_dev, data);
+ if (ret < 0)
+ DRM_DEBUG_KMS("Enable hdcp auth failed. %d\n", ret);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+
return ret;
}
+
+static int hdcp2_close_mei_session(struct intel_connector *connector)
+{
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct i915_hdcp_comp_master *comp;
+ int ret;
+
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ comp = dev_priv->hdcp_master;
+
+ if (!comp || !comp->ops) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return -EINVAL;
+ }
+
+ ret = comp->ops->close_hdcp_session(comp->mei_dev,
+ &connector->hdcp.port_data);
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+
+ return ret;
+}
+
+static int hdcp2_deauthenticate_port(struct intel_connector *connector)
+{
+ return hdcp2_close_mei_session(connector);
+}
+
+/* Authentication flow starts from here */
+static int hdcp2_authentication_key_exchange(struct intel_connector *connector)
+{
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ union {
+ struct hdcp2_ake_init ake_init;
+ struct hdcp2_ake_send_cert send_cert;
+ struct hdcp2_ake_no_stored_km no_stored_km;
+ struct hdcp2_ake_send_hprime send_hprime;
+ struct hdcp2_ake_send_pairing_info pairing_info;
+ } msgs;
+ const struct intel_hdcp_shim *shim = hdcp->shim;
+ size_t size;
+ int ret;
+
+ /* Init for seq_num */
+ hdcp->seq_num_v = 0;
+ hdcp->seq_num_m = 0;
+
+ ret = hdcp2_prepare_ake_init(connector, &msgs.ake_init);
+ if (ret < 0)
+ return ret;
+
+ ret = shim->write_2_2_msg(intel_dig_port, &msgs.ake_init,
+ sizeof(msgs.ake_init));
+ if (ret < 0)
+ return ret;
+
+ ret = shim->read_2_2_msg(intel_dig_port, HDCP_2_2_AKE_SEND_CERT,
+ &msgs.send_cert, sizeof(msgs.send_cert));
+ if (ret < 0)
+ return ret;
+
+ if (msgs.send_cert.rx_caps[0] != HDCP_2_2_RX_CAPS_VERSION_VAL)
+ return -EINVAL;
+
+ hdcp->is_repeater = HDCP_2_2_RX_REPEATER(msgs.send_cert.rx_caps[2]);
+
+ /*
+ * Here msgs.no_stored_km will hold msgs corresponding to the km
+ * stored also.
+ */
+ ret = hdcp2_verify_rx_cert_prepare_km(connector, &msgs.send_cert,
+ &hdcp->is_paired,
+ &msgs.no_stored_km, &size);
+ if (ret < 0)
+ return ret;
+
+ ret = shim->write_2_2_msg(intel_dig_port, &msgs.no_stored_km, size);
+ if (ret < 0)
+ return ret;
+
+ ret = shim->read_2_2_msg(intel_dig_port, HDCP_2_2_AKE_SEND_HPRIME,
+ &msgs.send_hprime, sizeof(msgs.send_hprime));
+ if (ret < 0)
+ return ret;
+
+ ret = hdcp2_verify_hprime(connector, &msgs.send_hprime);
+ if (ret < 0)
+ return ret;
+
+ if (!hdcp->is_paired) {
+ /* Pairing is required */
+ ret = shim->read_2_2_msg(intel_dig_port,
+ HDCP_2_2_AKE_SEND_PAIRING_INFO,
+ &msgs.pairing_info,
+ sizeof(msgs.pairing_info));
+ if (ret < 0)
+ return ret;
+
+ ret = hdcp2_store_pairing_info(connector, &msgs.pairing_info);
+ if (ret < 0)
+ return ret;
+ hdcp->is_paired = true;
+ }
+
+ return 0;
+}
+
+static int hdcp2_locality_check(struct intel_connector *connector)
+{
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ union {
+ struct hdcp2_lc_init lc_init;
+ struct hdcp2_lc_send_lprime send_lprime;
+ } msgs;
+ const struct intel_hdcp_shim *shim = hdcp->shim;
+ int tries = HDCP2_LC_RETRY_CNT, ret, i;
+
+ for (i = 0; i < tries; i++) {
+ ret = hdcp2_prepare_lc_init(connector, &msgs.lc_init);
+ if (ret < 0)
+ continue;
+
+ ret = shim->write_2_2_msg(intel_dig_port, &msgs.lc_init,
+ sizeof(msgs.lc_init));
+ if (ret < 0)
+ continue;
+
+ ret = shim->read_2_2_msg(intel_dig_port,
+ HDCP_2_2_LC_SEND_LPRIME,
+ &msgs.send_lprime,
+ sizeof(msgs.send_lprime));
+ if (ret < 0)
+ continue;
+
+ ret = hdcp2_verify_lprime(connector, &msgs.send_lprime);
+ if (!ret)
+ break;
+ }
+
+ return ret;
+}
+
+static int hdcp2_session_key_exchange(struct intel_connector *connector)
+{
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ struct hdcp2_ske_send_eks send_eks;
+ int ret;
+
+ ret = hdcp2_prepare_skey(connector, &send_eks);
+ if (ret < 0)
+ return ret;
+
+ ret = hdcp->shim->write_2_2_msg(intel_dig_port, &send_eks,
+ sizeof(send_eks));
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static
+int hdcp2_propagate_stream_management_info(struct intel_connector *connector)
+{
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ union {
+ struct hdcp2_rep_stream_manage stream_manage;
+ struct hdcp2_rep_stream_ready stream_ready;
+ } msgs;
+ const struct intel_hdcp_shim *shim = hdcp->shim;
+ int ret;
+
+ /* Prepare RepeaterAuth_Stream_Manage msg */
+ msgs.stream_manage.msg_id = HDCP_2_2_REP_STREAM_MANAGE;
+ drm_hdcp2_u32_to_seq_num(msgs.stream_manage.seq_num_m, hdcp->seq_num_m);
+
+ /* K no of streams is fixed as 1. Stored as big-endian. */
+ msgs.stream_manage.k = cpu_to_be16(1);
+
+ /* For HDMI this is forced to be 0x0. For DP SST also this is 0x0. */
+ msgs.stream_manage.streams[0].stream_id = 0;
+ msgs.stream_manage.streams[0].stream_type = hdcp->content_type;
+
+ /* Send it to Repeater */
+ ret = shim->write_2_2_msg(intel_dig_port, &msgs.stream_manage,
+ sizeof(msgs.stream_manage));
+ if (ret < 0)
+ return ret;
+
+ ret = shim->read_2_2_msg(intel_dig_port, HDCP_2_2_REP_STREAM_READY,
+ &msgs.stream_ready, sizeof(msgs.stream_ready));
+ if (ret < 0)
+ return ret;
+
+ hdcp->port_data.seq_num_m = hdcp->seq_num_m;
+ hdcp->port_data.streams[0].stream_type = hdcp->content_type;
+
+ ret = hdcp2_verify_mprime(connector, &msgs.stream_ready);
+ if (ret < 0)
+ return ret;
+
+ hdcp->seq_num_m++;
+
+ if (hdcp->seq_num_m > HDCP_2_2_SEQ_NUM_MAX) {
+ DRM_DEBUG_KMS("seq_num_m roll over.\n");
+ return -1;
+ }
+
+ return 0;
+}
+
+static
+int hdcp2_authenticate_repeater_topology(struct intel_connector *connector)
+{
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ union {
+ struct hdcp2_rep_send_receiverid_list recvid_list;
+ struct hdcp2_rep_send_ack rep_ack;
+ } msgs;
+ const struct intel_hdcp_shim *shim = hdcp->shim;
+ u8 *rx_info;
+ u32 seq_num_v;
+ int ret;
+
+ ret = shim->read_2_2_msg(intel_dig_port, HDCP_2_2_REP_SEND_RECVID_LIST,
+ &msgs.recvid_list, sizeof(msgs.recvid_list));
+ if (ret < 0)
+ return ret;
+
+ rx_info = msgs.recvid_list.rx_info;
+
+ if (HDCP_2_2_MAX_CASCADE_EXCEEDED(rx_info[1]) ||
+ HDCP_2_2_MAX_DEVS_EXCEEDED(rx_info[1])) {
+ DRM_DEBUG_KMS("Topology Max Size Exceeded\n");
+ return -EINVAL;
+ }
+
+ /* Converting and Storing the seq_num_v to local variable as DWORD */
+ seq_num_v = drm_hdcp2_seq_num_to_u32(msgs.recvid_list.seq_num_v);
+
+ if (seq_num_v < hdcp->seq_num_v) {
+ /* Roll over of the seq_num_v from repeater. Reauthenticate. */
+ DRM_DEBUG_KMS("Seq_num_v roll over.\n");
+ return -EINVAL;
+ }
+
+ ret = hdcp2_verify_rep_topology_prepare_ack(connector,
+ &msgs.recvid_list,
+ &msgs.rep_ack);
+ if (ret < 0)
+ return ret;
+
+ hdcp->seq_num_v = seq_num_v;
+ ret = shim->write_2_2_msg(intel_dig_port, &msgs.rep_ack,
+ sizeof(msgs.rep_ack));
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static int hdcp2_authenticate_repeater(struct intel_connector *connector)
+{
+ int ret;
+
+ ret = hdcp2_authenticate_repeater_topology(connector);
+ if (ret < 0)
+ return ret;
+
+ return hdcp2_propagate_stream_management_info(connector);
+}
+
+static int hdcp2_authenticate_sink(struct intel_connector *connector)
+{
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ const struct intel_hdcp_shim *shim = hdcp->shim;
+ int ret;
+
+ ret = hdcp2_authentication_key_exchange(connector);
+ if (ret < 0) {
+ DRM_DEBUG_KMS("AKE Failed. Err : %d\n", ret);
+ return ret;
+ }
+
+ ret = hdcp2_locality_check(connector);
+ if (ret < 0) {
+ DRM_DEBUG_KMS("Locality Check failed. Err : %d\n", ret);
+ return ret;
+ }
+
+ ret = hdcp2_session_key_exchange(connector);
+ if (ret < 0) {
+ DRM_DEBUG_KMS("SKE Failed. Err : %d\n", ret);
+ return ret;
+ }
+
+ if (shim->config_stream_type) {
+ ret = shim->config_stream_type(intel_dig_port,
+ hdcp->is_repeater,
+ hdcp->content_type);
+ if (ret < 0)
+ return ret;
+ }
+
+ if (hdcp->is_repeater) {
+ ret = hdcp2_authenticate_repeater(connector);
+ if (ret < 0) {
+ DRM_DEBUG_KMS("Repeater Auth Failed. Err: %d\n", ret);
+ return ret;
+ }
+ }
+
+ hdcp->port_data.streams[0].stream_type = hdcp->content_type;
+ ret = hdcp2_authenticate_port(connector);
+ if (ret < 0)
+ return ret;
+
+ return ret;
+}
+
+static int hdcp2_enable_encryption(struct intel_connector *connector)
+{
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ enum port port = connector->encoder->port;
+ int ret;
+
+ WARN_ON(I915_READ(HDCP2_STATUS_DDI(port)) & LINK_ENCRYPTION_STATUS);
+
+ if (hdcp->shim->toggle_signalling) {
+ ret = hdcp->shim->toggle_signalling(intel_dig_port, true);
+ if (ret) {
+ DRM_ERROR("Failed to enable HDCP signalling. %d\n",
+ ret);
+ return ret;
+ }
+ }
+
+ if (I915_READ(HDCP2_STATUS_DDI(port)) & LINK_AUTH_STATUS) {
+ /* Link is Authenticated. Now set for Encryption */
+ I915_WRITE(HDCP2_CTL_DDI(port),
+ I915_READ(HDCP2_CTL_DDI(port)) |
+ CTL_LINK_ENCRYPTION_REQ);
+ }
+
+ ret = intel_wait_for_register(&dev_priv->uncore, HDCP2_STATUS_DDI(port),
+ LINK_ENCRYPTION_STATUS,
+ LINK_ENCRYPTION_STATUS,
+ ENCRYPT_STATUS_CHANGE_TIMEOUT_MS);
+
+ return ret;
+}
+
+static int hdcp2_disable_encryption(struct intel_connector *connector)
+{
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ enum port port = connector->encoder->port;
+ int ret;
+
+ WARN_ON(!(I915_READ(HDCP2_STATUS_DDI(port)) & LINK_ENCRYPTION_STATUS));
+
+ I915_WRITE(HDCP2_CTL_DDI(port),
+ I915_READ(HDCP2_CTL_DDI(port)) & ~CTL_LINK_ENCRYPTION_REQ);
+
+ ret = intel_wait_for_register(&dev_priv->uncore, HDCP2_STATUS_DDI(port),
+ LINK_ENCRYPTION_STATUS, 0x0,
+ ENCRYPT_STATUS_CHANGE_TIMEOUT_MS);
+ if (ret == -ETIMEDOUT)
+ DRM_DEBUG_KMS("Disable Encryption Timedout");
+
+ if (hdcp->shim->toggle_signalling) {
+ ret = hdcp->shim->toggle_signalling(intel_dig_port, false);
+ if (ret) {
+ DRM_ERROR("Failed to disable HDCP signalling. %d\n",
+ ret);
+ return ret;
+ }
+ }
+
+ return ret;
+}
+
+static int hdcp2_authenticate_and_encrypt(struct intel_connector *connector)
+{
+ int ret, i, tries = 3;
+
+ for (i = 0; i < tries; i++) {
+ ret = hdcp2_authenticate_sink(connector);
+ if (!ret)
+ break;
+
+ /* Clearing the mei hdcp session */
+ DRM_DEBUG_KMS("HDCP2.2 Auth %d of %d Failed.(%d)\n",
+ i + 1, tries, ret);
+ if (hdcp2_deauthenticate_port(connector) < 0)
+ DRM_DEBUG_KMS("Port deauth failed.\n");
+ }
+
+ if (i != tries) {
+ /*
+ * Ensuring the required 200mSec min time interval between
+ * Session Key Exchange and encryption.
+ */
+ msleep(HDCP_2_2_DELAY_BEFORE_ENCRYPTION_EN);
+ ret = hdcp2_enable_encryption(connector);
+ if (ret < 0) {
+ DRM_DEBUG_KMS("Encryption Enable Failed.(%d)\n", ret);
+ if (hdcp2_deauthenticate_port(connector) < 0)
+ DRM_DEBUG_KMS("Port deauth failed.\n");
+ }
+ }
+
+ return ret;
+}
+
+static int _intel_hdcp2_enable(struct intel_connector *connector)
+{
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ int ret;
+
+ DRM_DEBUG_KMS("[%s:%d] HDCP2.2 is being enabled. Type: %d\n",
+ connector->base.name, connector->base.base.id,
+ hdcp->content_type);
+
+ ret = hdcp2_authenticate_and_encrypt(connector);
+ if (ret) {
+ DRM_DEBUG_KMS("HDCP2 Type%d Enabling Failed. (%d)\n",
+ hdcp->content_type, ret);
+ return ret;
+ }
+
+ DRM_DEBUG_KMS("[%s:%d] HDCP2.2 is enabled. Type %d\n",
+ connector->base.name, connector->base.base.id,
+ hdcp->content_type);
+
+ hdcp->hdcp2_encrypted = true;
+ return 0;
+}
+
+static int _intel_hdcp2_disable(struct intel_connector *connector)
+{
+ int ret;
+
+ DRM_DEBUG_KMS("[%s:%d] HDCP2.2 is being Disabled\n",
+ connector->base.name, connector->base.base.id);
+
+ ret = hdcp2_disable_encryption(connector);
+
+ if (hdcp2_deauthenticate_port(connector) < 0)
+ DRM_DEBUG_KMS("Port deauth failed.\n");
+
+ connector->hdcp.hdcp2_encrypted = false;
+
+ return ret;
+}
+
+/* Implements the Link Integrity Check for HDCP2.2 */
+static int intel_hdcp2_check_link(struct intel_connector *connector)
+{
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ enum port port = connector->encoder->port;
+ int ret = 0;
+
+ mutex_lock(&hdcp->mutex);
+
+ /* hdcp2_check_link is expected only when HDCP2.2 is Enabled */
+ if (hdcp->value != DRM_MODE_CONTENT_PROTECTION_ENABLED ||
+ !hdcp->hdcp2_encrypted) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (WARN_ON(!intel_hdcp2_in_use(connector))) {
+ DRM_ERROR("HDCP2.2 link stopped the encryption, %x\n",
+ I915_READ(HDCP2_STATUS_DDI(port)));
+ ret = -ENXIO;
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_DESIRED;
+ schedule_work(&hdcp->prop_work);
+ goto out;
+ }
+
+ ret = hdcp->shim->check_2_2_link(intel_dig_port);
+ if (ret == HDCP_LINK_PROTECTED) {
+ if (hdcp->value != DRM_MODE_CONTENT_PROTECTION_UNDESIRED) {
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_ENABLED;
+ schedule_work(&hdcp->prop_work);
+ }
+ goto out;
+ }
+
+ if (ret == HDCP_TOPOLOGY_CHANGE) {
+ if (hdcp->value == DRM_MODE_CONTENT_PROTECTION_UNDESIRED)
+ goto out;
+
+ DRM_DEBUG_KMS("HDCP2.2 Downstream topology change\n");
+ ret = hdcp2_authenticate_repeater_topology(connector);
+ if (!ret) {
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_ENABLED;
+ schedule_work(&hdcp->prop_work);
+ goto out;
+ }
+ DRM_DEBUG_KMS("[%s:%d] Repeater topology auth failed.(%d)\n",
+ connector->base.name, connector->base.base.id,
+ ret);
+ } else {
+ DRM_DEBUG_KMS("[%s:%d] HDCP2.2 link failed, retrying auth\n",
+ connector->base.name, connector->base.base.id);
+ }
+
+ ret = _intel_hdcp2_disable(connector);
+ if (ret) {
+ DRM_ERROR("[%s:%d] Failed to disable hdcp2.2 (%d)\n",
+ connector->base.name, connector->base.base.id, ret);
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_DESIRED;
+ schedule_work(&hdcp->prop_work);
+ goto out;
+ }
+
+ ret = _intel_hdcp2_enable(connector);
+ if (ret) {
+ DRM_DEBUG_KMS("[%s:%d] Failed to enable hdcp2.2 (%d)\n",
+ connector->base.name, connector->base.base.id,
+ ret);
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_DESIRED;
+ schedule_work(&hdcp->prop_work);
+ goto out;
+ }
+
+out:
+ mutex_unlock(&hdcp->mutex);
+ return ret;
+}
+
+static void intel_hdcp_check_work(struct work_struct *work)
+{
+ struct intel_hdcp *hdcp = container_of(to_delayed_work(work),
+ struct intel_hdcp,
+ check_work);
+ struct intel_connector *connector = intel_hdcp_to_connector(hdcp);
+
+ if (!intel_hdcp2_check_link(connector))
+ schedule_delayed_work(&hdcp->check_work,
+ DRM_HDCP2_CHECK_PERIOD_MS);
+ else if (!intel_hdcp_check_link(connector))
+ schedule_delayed_work(&hdcp->check_work,
+ DRM_HDCP_CHECK_PERIOD_MS);
+}
+
+static int i915_hdcp_component_bind(struct device *i915_kdev,
+ struct device *mei_kdev, void *data)
+{
+ struct drm_i915_private *dev_priv = kdev_to_i915(i915_kdev);
+
+ DRM_DEBUG("I915 HDCP comp bind\n");
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ dev_priv->hdcp_master = (struct i915_hdcp_comp_master *)data;
+ dev_priv->hdcp_master->mei_dev = mei_kdev;
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+
+ return 0;
+}
+
+static void i915_hdcp_component_unbind(struct device *i915_kdev,
+ struct device *mei_kdev, void *data)
+{
+ struct drm_i915_private *dev_priv = kdev_to_i915(i915_kdev);
+
+ DRM_DEBUG("I915 HDCP comp unbind\n");
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ dev_priv->hdcp_master = NULL;
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+}
+
+static const struct component_ops i915_hdcp_component_ops = {
+ .bind = i915_hdcp_component_bind,
+ .unbind = i915_hdcp_component_unbind,
+};
+
+static inline int initialize_hdcp_port_data(struct intel_connector *connector)
+{
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ struct hdcp_port_data *data = &hdcp->port_data;
+
+ data->port = connector->encoder->port;
+ data->port_type = (u8)HDCP_PORT_TYPE_INTEGRATED;
+ data->protocol = (u8)hdcp->shim->protocol;
+
+ data->k = 1;
+ if (!data->streams)
+ data->streams = kcalloc(data->k,
+ sizeof(struct hdcp2_streamid_type),
+ GFP_KERNEL);
+ if (!data->streams) {
+ DRM_ERROR("Out of Memory\n");
+ return -ENOMEM;
+ }
+
+ data->streams[0].stream_id = 0;
+ data->streams[0].stream_type = hdcp->content_type;
+
+ return 0;
+}
+
+static bool is_hdcp2_supported(struct drm_i915_private *dev_priv)
+{
+ if (!IS_ENABLED(CONFIG_INTEL_MEI_HDCP))
+ return false;
+
+ return (INTEL_GEN(dev_priv) >= 10 || IS_GEMINILAKE(dev_priv) ||
+ IS_KABYLAKE(dev_priv));
+}
+
+void intel_hdcp_component_init(struct drm_i915_private *dev_priv)
+{
+ int ret;
+
+ if (!is_hdcp2_supported(dev_priv))
+ return;
+
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ WARN_ON(dev_priv->hdcp_comp_added);
+
+ dev_priv->hdcp_comp_added = true;
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ ret = component_add_typed(dev_priv->drm.dev, &i915_hdcp_component_ops,
+ I915_COMPONENT_HDCP);
+ if (ret < 0) {
+ DRM_DEBUG_KMS("Failed at component add(%d)\n", ret);
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ dev_priv->hdcp_comp_added = false;
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return;
+ }
+}
+
+static void intel_hdcp2_init(struct intel_connector *connector)
+{
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ int ret;
+
+ ret = initialize_hdcp_port_data(connector);
+ if (ret) {
+ DRM_DEBUG_KMS("Mei hdcp data init failed\n");
+ return;
+ }
+
+ hdcp->hdcp2_supported = true;
+}
+
+int intel_hdcp_init(struct intel_connector *connector,
+ const struct intel_hdcp_shim *shim)
+{
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ int ret;
+
+ if (!shim)
+ return -EINVAL;
+
+ ret = drm_connector_attach_content_protection_property(&connector->base);
+ if (ret)
+ return ret;
+
+ hdcp->shim = shim;
+ mutex_init(&hdcp->mutex);
+ INIT_DELAYED_WORK(&hdcp->check_work, intel_hdcp_check_work);
+ INIT_WORK(&hdcp->prop_work, intel_hdcp_prop_work);
+
+ if (is_hdcp2_supported(dev_priv))
+ intel_hdcp2_init(connector);
+ init_waitqueue_head(&hdcp->cp_irq_queue);
+
+ return 0;
+}
+
+int intel_hdcp_enable(struct intel_connector *connector)
+{
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ unsigned long check_link_interval = DRM_HDCP_CHECK_PERIOD_MS;
+ int ret = -EINVAL;
+
+ if (!hdcp->shim)
+ return -ENOENT;
+
+ mutex_lock(&hdcp->mutex);
+ WARN_ON(hdcp->value == DRM_MODE_CONTENT_PROTECTION_ENABLED);
+
+ /*
+ * Considering that HDCP2.2 is more secure than HDCP1.4, If the setup
+ * is capable of HDCP2.2, it is preferred to use HDCP2.2.
+ */
+ if (intel_hdcp2_capable(connector)) {
+ ret = _intel_hdcp2_enable(connector);
+ if (!ret)
+ check_link_interval = DRM_HDCP2_CHECK_PERIOD_MS;
+ }
+
+ /* When HDCP2.2 fails, HDCP1.4 will be attempted */
+ if (ret && intel_hdcp_capable(connector)) {
+ ret = _intel_hdcp_enable(connector);
+ }
+
+ if (!ret) {
+ schedule_delayed_work(&hdcp->check_work, check_link_interval);
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_ENABLED;
+ schedule_work(&hdcp->prop_work);
+ }
+
+ mutex_unlock(&hdcp->mutex);
+ return ret;
+}
+
+int intel_hdcp_disable(struct intel_connector *connector)
+{
+ struct intel_hdcp *hdcp = &connector->hdcp;
+ int ret = 0;
+
+ if (!hdcp->shim)
+ return -ENOENT;
+
+ mutex_lock(&hdcp->mutex);
+
+ if (hdcp->value != DRM_MODE_CONTENT_PROTECTION_UNDESIRED) {
+ hdcp->value = DRM_MODE_CONTENT_PROTECTION_UNDESIRED;
+ if (hdcp->hdcp2_encrypted)
+ ret = _intel_hdcp2_disable(connector);
+ else if (hdcp->hdcp_encrypted)
+ ret = _intel_hdcp_disable(connector);
+ }
+
+ mutex_unlock(&hdcp->mutex);
+ cancel_delayed_work_sync(&hdcp->check_work);
+ return ret;
+}
+
+void intel_hdcp_component_fini(struct drm_i915_private *dev_priv)
+{
+ mutex_lock(&dev_priv->hdcp_comp_mutex);
+ if (!dev_priv->hdcp_comp_added) {
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+ return;
+ }
+
+ dev_priv->hdcp_comp_added = false;
+ mutex_unlock(&dev_priv->hdcp_comp_mutex);
+
+ component_del(dev_priv->drm.dev, &i915_hdcp_component_ops);
+}
+
+void intel_hdcp_cleanup(struct intel_connector *connector)
+{
+ if (!connector->hdcp.shim)
+ return;
+
+ mutex_lock(&connector->hdcp.mutex);
+ kfree(connector->hdcp.port_data.streams);
+ mutex_unlock(&connector->hdcp.mutex);
+}
+
+void intel_hdcp_atomic_check(struct drm_connector *connector,
+ struct drm_connector_state *old_state,
+ struct drm_connector_state *new_state)
+{
+ u64 old_cp = old_state->content_protection;
+ u64 new_cp = new_state->content_protection;
+ struct drm_crtc_state *crtc_state;
+
+ if (!new_state->crtc) {
+ /*
+ * If the connector is being disabled with CP enabled, mark it
+ * desired so it's re-enabled when the connector is brought back
+ */
+ if (old_cp == DRM_MODE_CONTENT_PROTECTION_ENABLED)
+ new_state->content_protection =
+ DRM_MODE_CONTENT_PROTECTION_DESIRED;
+ return;
+ }
+
+ /*
+ * Nothing to do if the state didn't change, or HDCP was activated since
+ * the last commit
+ */
+ if (old_cp == new_cp ||
+ (old_cp == DRM_MODE_CONTENT_PROTECTION_DESIRED &&
+ new_cp == DRM_MODE_CONTENT_PROTECTION_ENABLED))
+ return;
+
+ crtc_state = drm_atomic_get_new_crtc_state(new_state->state,
+ new_state->crtc);
+ crtc_state->mode_changed = true;
+}
+
+/* Handles the CP_IRQ raised from the DP HDCP sink */
+void intel_hdcp_handle_cp_irq(struct intel_connector *connector)
+{
+ struct intel_hdcp *hdcp = &connector->hdcp;
+
+ if (!hdcp->shim)
+ return;
+
+ atomic_inc(&connector->hdcp.cp_irq_count);
+ wake_up_all(&connector->hdcp.cp_irq_queue);
+
+ schedule_delayed_work(&hdcp->check_work, 0);
+}
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_HDCP_H__
+#define __INTEL_HDCP_H__
+
+#include <linux/types.h>
+
+#include <drm/i915_drm.h>
+
+struct drm_connector;
+struct drm_connector_state;
+struct drm_i915_private;
+struct intel_connector;
+struct intel_hdcp_shim;
+
+void intel_hdcp_atomic_check(struct drm_connector *connector,
+ struct drm_connector_state *old_state,
+ struct drm_connector_state *new_state);
+int intel_hdcp_init(struct intel_connector *connector,
+ const struct intel_hdcp_shim *hdcp_shim);
+int intel_hdcp_enable(struct intel_connector *connector);
+int intel_hdcp_disable(struct intel_connector *connector);
+bool is_hdcp_supported(struct drm_i915_private *dev_priv, enum port port);
+bool intel_hdcp_capable(struct intel_connector *connector);
+void intel_hdcp_component_init(struct drm_i915_private *dev_priv);
+void intel_hdcp_component_fini(struct drm_i915_private *dev_priv);
+void intel_hdcp_cleanup(struct intel_connector *connector);
+void intel_hdcp_handle_cp_irq(struct intel_connector *connector);
+
+#endif /* __INTEL_HDCP_H__ */
* Jesse Barnes <jesse.barnes@intel.com>
*/
-#include <linux/i2c.h>
-#include <linux/slab.h>
#include <linux/delay.h>
#include <linux/hdmi.h>
+#include <linux/i2c.h>
+#include <linux/slab.h>
+
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
#include <drm/drm_edid.h>
#include <drm/drm_hdcp.h>
#include <drm/drm_scdc_helper.h>
-#include "intel_drv.h"
#include <drm/i915_drm.h>
#include <drm/intel_lpe_audio.h>
+
#include "i915_drv.h"
+#include "intel_audio.h"
+#include "intel_connector.h"
+#include "intel_ddi.h"
+#include "intel_dp.h"
+#include "intel_drv.h"
+#include "intel_hdcp.h"
+#include "intel_hdmi.h"
+#include "intel_lspcon.h"
+#include "intel_sdvo.h"
+#include "intel_panel.h"
static struct drm_device *intel_hdmi_to_dev(struct intel_hdmi *intel_hdmi)
{
static u32 g4x_infoframe_index(unsigned int type)
{
switch (type) {
+ case HDMI_PACKET_TYPE_GAMUT_METADATA:
+ return VIDEO_DIP_SELECT_GAMUT;
case HDMI_INFOFRAME_TYPE_AVI:
return VIDEO_DIP_SELECT_AVI;
case HDMI_INFOFRAME_TYPE_SPD:
static u32 g4x_infoframe_enable(unsigned int type)
{
switch (type) {
+ case HDMI_PACKET_TYPE_GENERAL_CONTROL:
+ return VIDEO_DIP_ENABLE_GCP;
+ case HDMI_PACKET_TYPE_GAMUT_METADATA:
+ return VIDEO_DIP_ENABLE_GAMUT;
+ case DP_SDP_VSC:
+ return 0;
case HDMI_INFOFRAME_TYPE_AVI:
return VIDEO_DIP_ENABLE_AVI;
case HDMI_INFOFRAME_TYPE_SPD:
static u32 hsw_infoframe_enable(unsigned int type)
{
switch (type) {
+ case HDMI_PACKET_TYPE_GENERAL_CONTROL:
+ return VIDEO_DIP_ENABLE_GCP_HSW;
+ case HDMI_PACKET_TYPE_GAMUT_METADATA:
+ return VIDEO_DIP_ENABLE_GMP_HSW;
case DP_SDP_VSC:
return VIDEO_DIP_ENABLE_VSC_HSW;
case DP_SDP_PPS:
int i)
{
switch (type) {
+ case HDMI_PACKET_TYPE_GAMUT_METADATA:
+ return HSW_TVIDEO_DIP_GMP_DATA(cpu_transcoder, i);
case DP_SDP_VSC:
return HSW_TVIDEO_DIP_VSC_DATA(cpu_transcoder, i);
case DP_SDP_PPS:
POSTING_READ(VIDEO_DIP_CTL);
}
-static bool g4x_infoframe_enabled(struct intel_encoder *encoder,
+static void g4x_read_infoframe(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ unsigned int type,
+ void *frame, ssize_t len)
+{
+ struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ u32 val, *data = frame;
+ int i;
+
+ val = I915_READ(VIDEO_DIP_CTL);
+
+ val &= ~(VIDEO_DIP_SELECT_MASK | 0xf); /* clear DIP data offset */
+ val |= g4x_infoframe_index(type);
+
+ I915_WRITE(VIDEO_DIP_CTL, val);
+
+ for (i = 0; i < len; i += 4)
+ *data++ = I915_READ(VIDEO_DIP_DATA);
+}
+
+static u32 g4x_infoframes_enabled(struct intel_encoder *encoder,
const struct intel_crtc_state *pipe_config)
{
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
u32 val = I915_READ(VIDEO_DIP_CTL);
if ((val & VIDEO_DIP_ENABLE) == 0)
- return false;
+ return 0;
if ((val & VIDEO_DIP_PORT_MASK) != VIDEO_DIP_PORT(encoder->port))
- return false;
+ return 0;
return val & (VIDEO_DIP_ENABLE_AVI |
VIDEO_DIP_ENABLE_VENDOR | VIDEO_DIP_ENABLE_SPD);
POSTING_READ(reg);
}
-static bool ibx_infoframe_enabled(struct intel_encoder *encoder,
+static void ibx_read_infoframe(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ unsigned int type,
+ void *frame, ssize_t len)
+{
+ struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ u32 val, *data = frame;
+ int i;
+
+ val = I915_READ(TVIDEO_DIP_CTL(crtc->pipe));
+
+ val &= ~(VIDEO_DIP_SELECT_MASK | 0xf); /* clear DIP data offset */
+ val |= g4x_infoframe_index(type);
+
+ I915_WRITE(TVIDEO_DIP_CTL(crtc->pipe), val);
+
+ for (i = 0; i < len; i += 4)
+ *data++ = I915_READ(TVIDEO_DIP_DATA(crtc->pipe));
+}
+
+static u32 ibx_infoframes_enabled(struct intel_encoder *encoder,
const struct intel_crtc_state *pipe_config)
{
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
u32 val = I915_READ(reg);
if ((val & VIDEO_DIP_ENABLE) == 0)
- return false;
+ return 0;
if ((val & VIDEO_DIP_PORT_MASK) != VIDEO_DIP_PORT(encoder->port))
- return false;
+ return 0;
return val & (VIDEO_DIP_ENABLE_AVI |
VIDEO_DIP_ENABLE_VENDOR | VIDEO_DIP_ENABLE_GAMUT |
POSTING_READ(reg);
}
-static bool cpt_infoframe_enabled(struct intel_encoder *encoder,
+static void cpt_read_infoframe(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ unsigned int type,
+ void *frame, ssize_t len)
+{
+ struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ u32 val, *data = frame;
+ int i;
+
+ val = I915_READ(TVIDEO_DIP_CTL(crtc->pipe));
+
+ val &= ~(VIDEO_DIP_SELECT_MASK | 0xf); /* clear DIP data offset */
+ val |= g4x_infoframe_index(type);
+
+ I915_WRITE(TVIDEO_DIP_CTL(crtc->pipe), val);
+
+ for (i = 0; i < len; i += 4)
+ *data++ = I915_READ(TVIDEO_DIP_DATA(crtc->pipe));
+}
+
+static u32 cpt_infoframes_enabled(struct intel_encoder *encoder,
const struct intel_crtc_state *pipe_config)
{
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
u32 val = I915_READ(TVIDEO_DIP_CTL(pipe));
if ((val & VIDEO_DIP_ENABLE) == 0)
- return false;
+ return 0;
return val & (VIDEO_DIP_ENABLE_AVI |
VIDEO_DIP_ENABLE_VENDOR | VIDEO_DIP_ENABLE_GAMUT |
POSTING_READ(reg);
}
-static bool vlv_infoframe_enabled(struct intel_encoder *encoder,
+static void vlv_read_infoframe(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ unsigned int type,
+ void *frame, ssize_t len)
+{
+ struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ u32 val, *data = frame;
+ int i;
+
+ val = I915_READ(VLV_TVIDEO_DIP_CTL(crtc->pipe));
+
+ val &= ~(VIDEO_DIP_SELECT_MASK | 0xf); /* clear DIP data offset */
+ val |= g4x_infoframe_index(type);
+
+ I915_WRITE(VLV_TVIDEO_DIP_CTL(crtc->pipe), val);
+
+ for (i = 0; i < len; i += 4)
+ *data++ = I915_READ(VLV_TVIDEO_DIP_DATA(crtc->pipe));
+}
+
+static u32 vlv_infoframes_enabled(struct intel_encoder *encoder,
const struct intel_crtc_state *pipe_config)
{
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
u32 val = I915_READ(VLV_TVIDEO_DIP_CTL(pipe));
if ((val & VIDEO_DIP_ENABLE) == 0)
- return false;
+ return 0;
if ((val & VIDEO_DIP_PORT_MASK) != VIDEO_DIP_PORT(encoder->port))
- return false;
+ return 0;
return val & (VIDEO_DIP_ENABLE_AVI |
VIDEO_DIP_ENABLE_VENDOR | VIDEO_DIP_ENABLE_GAMUT |
POSTING_READ(ctl_reg);
}
-static bool hsw_infoframe_enabled(struct intel_encoder *encoder,
+static void hsw_read_infoframe(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ unsigned int type,
+ void *frame, ssize_t len)
+{
+ struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ enum transcoder cpu_transcoder = crtc_state->cpu_transcoder;
+ u32 val, *data = frame;
+ int i;
+
+ val = I915_READ(HSW_TVIDEO_DIP_CTL(cpu_transcoder));
+
+ for (i = 0; i < len; i += 4)
+ *data++ = I915_READ(hsw_dip_data_reg(dev_priv, cpu_transcoder,
+ type, i >> 2));
+}
+
+static u32 hsw_infoframes_enabled(struct intel_encoder *encoder,
const struct intel_crtc_state *pipe_config)
{
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
VIDEO_DIP_ENABLE_GMP_HSW | VIDEO_DIP_ENABLE_SPD_HSW);
}
+static const u8 infoframe_type_to_idx[] = {
+ HDMI_PACKET_TYPE_GENERAL_CONTROL,
+ HDMI_PACKET_TYPE_GAMUT_METADATA,
+ DP_SDP_VSC,
+ HDMI_INFOFRAME_TYPE_AVI,
+ HDMI_INFOFRAME_TYPE_SPD,
+ HDMI_INFOFRAME_TYPE_VENDOR,
+};
+
+u32 intel_hdmi_infoframe_enable(unsigned int type)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(infoframe_type_to_idx); i++) {
+ if (infoframe_type_to_idx[i] == type)
+ return BIT(i);
+ }
+
+ return 0;
+}
+
+u32 intel_hdmi_infoframes_enabled(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state)
+{
+ struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct intel_digital_port *dig_port = enc_to_dig_port(&encoder->base);
+ u32 val, ret = 0;
+ int i;
+
+ val = dig_port->infoframes_enabled(encoder, crtc_state);
+
+ /* map from hardware bits to dip idx */
+ for (i = 0; i < ARRAY_SIZE(infoframe_type_to_idx); i++) {
+ unsigned int type = infoframe_type_to_idx[i];
+
+ if (HAS_DDI(dev_priv)) {
+ if (val & hsw_infoframe_enable(type))
+ ret |= BIT(i);
+ } else {
+ if (val & g4x_infoframe_enable(type))
+ ret |= BIT(i);
+ }
+ }
+
+ return ret;
+}
+
/*
* The data we write to the DIP data buffer registers is 1 byte bigger than the
* HDMI infoframe size because of an ECC/reserved byte at position 3 (starting
*/
static void intel_write_infoframe(struct intel_encoder *encoder,
const struct intel_crtc_state *crtc_state,
- union hdmi_infoframe *frame)
+ enum hdmi_infoframe_type type,
+ const union hdmi_infoframe *frame)
{
struct intel_digital_port *intel_dig_port = enc_to_dig_port(&encoder->base);
u8 buffer[VIDEO_DIP_DATA_SIZE];
ssize_t len;
+ if ((crtc_state->infoframes.enable &
+ intel_hdmi_infoframe_enable(type)) == 0)
+ return;
+
+ if (WARN_ON(frame->any.type != type))
+ return;
+
/* see comment above for the reason for this offset */
- len = hdmi_infoframe_pack(frame, buffer + 1, sizeof(buffer) - 1);
- if (len < 0)
+ len = hdmi_infoframe_pack_only(frame, buffer + 1, sizeof(buffer) - 1);
+ if (WARN_ON(len < 0))
return;
/* Insert the 'hole' (see big comment above) at position 3 */
buffer[3] = 0;
len++;
- intel_dig_port->write_infoframe(encoder,
- crtc_state,
- frame->any.type, buffer, len);
+ intel_dig_port->write_infoframe(encoder, crtc_state, type, buffer, len);
}
-static void intel_hdmi_set_avi_infoframe(struct intel_encoder *encoder,
- const struct intel_crtc_state *crtc_state,
- const struct drm_connector_state *conn_state)
+void intel_read_infoframe(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ enum hdmi_infoframe_type type,
+ union hdmi_infoframe *frame)
+{
+ struct intel_digital_port *intel_dig_port = enc_to_dig_port(&encoder->base);
+ u8 buffer[VIDEO_DIP_DATA_SIZE];
+ int ret;
+
+ if ((crtc_state->infoframes.enable &
+ intel_hdmi_infoframe_enable(type)) == 0)
+ return;
+
+ intel_dig_port->read_infoframe(encoder, crtc_state,
+ type, buffer, sizeof(buffer));
+
+ /* Fill the 'hole' (see big comment above) at position 3 */
+ memmove(&buffer[1], &buffer[0], 3);
+
+ /* see comment above for the reason for this offset */
+ ret = hdmi_infoframe_unpack(frame, buffer + 1, sizeof(buffer) - 1);
+ if (ret) {
+ DRM_DEBUG_KMS("Failed to unpack infoframe type 0x%02x\n", type);
+ return;
+ }
+
+ if (frame->any.type != type)
+ DRM_DEBUG_KMS("Found the wrong infoframe type 0x%x (expected 0x%02x)\n",
+ frame->any.type, type);
+}
+
+static bool
+intel_hdmi_compute_avi_infoframe(struct intel_encoder *encoder,
+ struct intel_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state)
{
+ struct hdmi_avi_infoframe *frame = &crtc_state->infoframes.avi.avi;
const struct drm_display_mode *adjusted_mode =
&crtc_state->base.adjusted_mode;
- union hdmi_infoframe frame;
+ struct drm_connector *connector = conn_state->connector;
int ret;
- ret = drm_hdmi_avi_infoframe_from_display_mode(&frame.avi,
- conn_state->connector,
+ if (!crtc_state->has_infoframe)
+ return true;
+
+ crtc_state->infoframes.enable |=
+ intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_AVI);
+
+ ret = drm_hdmi_avi_infoframe_from_display_mode(frame, connector,
adjusted_mode);
- if (ret < 0) {
- DRM_ERROR("couldn't fill AVI infoframe\n");
- return;
- }
+ if (ret)
+ return false;
if (crtc_state->output_format == INTEL_OUTPUT_FORMAT_YCBCR420)
- frame.avi.colorspace = HDMI_COLORSPACE_YUV420;
+ frame->colorspace = HDMI_COLORSPACE_YUV420;
else if (crtc_state->output_format == INTEL_OUTPUT_FORMAT_YCBCR444)
- frame.avi.colorspace = HDMI_COLORSPACE_YUV444;
+ frame->colorspace = HDMI_COLORSPACE_YUV444;
else
- frame.avi.colorspace = HDMI_COLORSPACE_RGB;
+ frame->colorspace = HDMI_COLORSPACE_RGB;
+
+ drm_hdmi_avi_infoframe_colorspace(frame, conn_state);
- drm_hdmi_avi_infoframe_quant_range(&frame.avi,
- conn_state->connector,
+ drm_hdmi_avi_infoframe_quant_range(frame, connector,
adjusted_mode,
crtc_state->limited_color_range ?
HDMI_QUANTIZATION_RANGE_LIMITED :
HDMI_QUANTIZATION_RANGE_FULL);
- drm_hdmi_avi_infoframe_content_type(&frame.avi,
- conn_state);
+ drm_hdmi_avi_infoframe_content_type(frame, conn_state);
/* TODO: handle pixel repetition for YCBCR420 outputs */
- intel_write_infoframe(encoder, crtc_state,
- &frame);
+
+ ret = hdmi_avi_infoframe_check(frame);
+ if (WARN_ON(ret))
+ return false;
+
+ return true;
}
-static void intel_hdmi_set_spd_infoframe(struct intel_encoder *encoder,
- const struct intel_crtc_state *crtc_state)
+static bool
+intel_hdmi_compute_spd_infoframe(struct intel_encoder *encoder,
+ struct intel_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state)
{
- union hdmi_infoframe frame;
+ struct hdmi_spd_infoframe *frame = &crtc_state->infoframes.spd.spd;
int ret;
- ret = hdmi_spd_infoframe_init(&frame.spd, "Intel", "Integrated gfx");
- if (ret < 0) {
- DRM_ERROR("couldn't fill SPD infoframe\n");
- return;
- }
+ if (!crtc_state->has_infoframe)
+ return true;
- frame.spd.sdi = HDMI_SPD_SDI_PC;
+ crtc_state->infoframes.enable |=
+ intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_SPD);
- intel_write_infoframe(encoder, crtc_state,
- &frame);
+ ret = hdmi_spd_infoframe_init(frame, "Intel", "Integrated gfx");
+ if (WARN_ON(ret))
+ return false;
+
+ frame->sdi = HDMI_SPD_SDI_PC;
+
+ ret = hdmi_spd_infoframe_check(frame);
+ if (WARN_ON(ret))
+ return false;
+
+ return true;
}
-static void
-intel_hdmi_set_hdmi_infoframe(struct intel_encoder *encoder,
- const struct intel_crtc_state *crtc_state,
- const struct drm_connector_state *conn_state)
-{
- union hdmi_infoframe frame;
+static bool
+intel_hdmi_compute_hdmi_infoframe(struct intel_encoder *encoder,
+ struct intel_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state)
+{
+ struct hdmi_vendor_infoframe *frame =
+ &crtc_state->infoframes.hdmi.vendor.hdmi;
+ const struct drm_display_info *info =
+ &conn_state->connector->display_info;
int ret;
- ret = drm_hdmi_vendor_infoframe_from_display_mode(&frame.vendor.hdmi,
+ if (!crtc_state->has_infoframe || !info->has_hdmi_infoframe)
+ return true;
+
+ crtc_state->infoframes.enable |=
+ intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_VENDOR);
+
+ ret = drm_hdmi_vendor_infoframe_from_display_mode(frame,
conn_state->connector,
&crtc_state->base.adjusted_mode);
- if (ret < 0)
- return;
+ if (WARN_ON(ret))
+ return false;
- intel_write_infoframe(encoder, crtc_state,
- &frame);
+ ret = hdmi_vendor_infoframe_check(frame);
+ if (WARN_ON(ret))
+ return false;
+
+ return true;
}
static void g4x_set_infoframes(struct intel_encoder *encoder,
I915_WRITE(reg, val);
POSTING_READ(reg);
- intel_hdmi_set_avi_infoframe(encoder, crtc_state, conn_state);
- intel_hdmi_set_spd_infoframe(encoder, crtc_state);
- intel_hdmi_set_hdmi_infoframe(encoder, crtc_state, conn_state);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_AVI,
+ &crtc_state->infoframes.avi);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_SPD,
+ &crtc_state->infoframes.spd);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_VENDOR,
+ &crtc_state->infoframes.hdmi);
}
static bool hdmi_sink_is_deep_color(const struct drm_connector_state *conn_state)
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
i915_reg_t reg;
- u32 val = 0;
+
+ if ((crtc_state->infoframes.enable &
+ intel_hdmi_infoframe_enable(HDMI_PACKET_TYPE_GENERAL_CONTROL)) == 0)
+ return false;
if (HAS_DDI(dev_priv))
reg = HSW_TVIDEO_DIP_GCP(crtc_state->cpu_transcoder);
else
return false;
+ I915_WRITE(reg, crtc_state->infoframes.gcp);
+
+ return true;
+}
+
+void intel_hdmi_read_gcp_infoframe(struct intel_encoder *encoder,
+ struct intel_crtc_state *crtc_state)
+{
+ struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ i915_reg_t reg;
+
+ if ((crtc_state->infoframes.enable &
+ intel_hdmi_infoframe_enable(HDMI_PACKET_TYPE_GENERAL_CONTROL)) == 0)
+ return;
+
+ if (HAS_DDI(dev_priv))
+ reg = HSW_TVIDEO_DIP_GCP(crtc_state->cpu_transcoder);
+ else if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
+ reg = VLV_TVIDEO_DIP_GCP(crtc->pipe);
+ else if (HAS_PCH_SPLIT(dev_priv))
+ reg = TVIDEO_DIP_GCP(crtc->pipe);
+ else
+ return;
+
+ crtc_state->infoframes.gcp = I915_READ(reg);
+}
+
+static void intel_hdmi_compute_gcp_infoframe(struct intel_encoder *encoder,
+ struct intel_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state)
+{
+ struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+
+ if (IS_G4X(dev_priv) || !crtc_state->has_infoframe)
+ return;
+
+ crtc_state->infoframes.enable |=
+ intel_hdmi_infoframe_enable(HDMI_PACKET_TYPE_GENERAL_CONTROL);
+
/* Indicate color depth whenever the sink supports deep color */
if (hdmi_sink_is_deep_color(conn_state))
- val |= GCP_COLOR_INDICATION;
+ crtc_state->infoframes.gcp |= GCP_COLOR_INDICATION;
/* Enable default_phase whenever the display mode is suitably aligned */
if (gcp_default_phase_possible(crtc_state->pipe_bpp,
&crtc_state->base.adjusted_mode))
- val |= GCP_DEFAULT_PHASE_ENABLE;
-
- I915_WRITE(reg, val);
-
- return val != 0;
+ crtc_state->infoframes.gcp |= GCP_DEFAULT_PHASE_ENABLE;
}
static void ibx_set_infoframes(struct intel_encoder *encoder,
I915_WRITE(reg, val);
POSTING_READ(reg);
- intel_hdmi_set_avi_infoframe(encoder, crtc_state, conn_state);
- intel_hdmi_set_spd_infoframe(encoder, crtc_state);
- intel_hdmi_set_hdmi_infoframe(encoder, crtc_state, conn_state);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_AVI,
+ &crtc_state->infoframes.avi);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_SPD,
+ &crtc_state->infoframes.spd);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_VENDOR,
+ &crtc_state->infoframes.hdmi);
}
static void cpt_set_infoframes(struct intel_encoder *encoder,
I915_WRITE(reg, val);
POSTING_READ(reg);
- intel_hdmi_set_avi_infoframe(encoder, crtc_state, conn_state);
- intel_hdmi_set_spd_infoframe(encoder, crtc_state);
- intel_hdmi_set_hdmi_infoframe(encoder, crtc_state, conn_state);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_AVI,
+ &crtc_state->infoframes.avi);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_SPD,
+ &crtc_state->infoframes.spd);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_VENDOR,
+ &crtc_state->infoframes.hdmi);
}
static void vlv_set_infoframes(struct intel_encoder *encoder,
I915_WRITE(reg, val);
POSTING_READ(reg);
- intel_hdmi_set_avi_infoframe(encoder, crtc_state, conn_state);
- intel_hdmi_set_spd_infoframe(encoder, crtc_state);
- intel_hdmi_set_hdmi_infoframe(encoder, crtc_state, conn_state);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_AVI,
+ &crtc_state->infoframes.avi);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_SPD,
+ &crtc_state->infoframes.spd);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_VENDOR,
+ &crtc_state->infoframes.hdmi);
}
static void hsw_set_infoframes(struct intel_encoder *encoder,
I915_WRITE(reg, val);
POSTING_READ(reg);
- intel_hdmi_set_avi_infoframe(encoder, crtc_state, conn_state);
- intel_hdmi_set_spd_infoframe(encoder, crtc_state);
- intel_hdmi_set_hdmi_infoframe(encoder, crtc_state, conn_state);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_AVI,
+ &crtc_state->infoframes.avi);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_SPD,
+ &crtc_state->infoframes.spd);
+ intel_write_infoframe(encoder, crtc_state,
+ HDMI_INFOFRAME_TYPE_VENDOR,
+ &crtc_state->infoframes.hdmi);
}
void intel_dp_dual_mode_set_tmds_output(struct intel_hdmi *hdmi, bool enable)
return ret;
}
+static int kbl_repositioning_enc_en_signal(struct intel_connector *connector)
+{
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct intel_digital_port *intel_dig_port = conn_to_dig_port(connector);
+ struct drm_crtc *crtc = connector->base.state->crtc;
+ struct intel_crtc *intel_crtc = container_of(crtc,
+ struct intel_crtc, base);
+ u32 scanline;
+ int ret;
+
+ for (;;) {
+ scanline = I915_READ(PIPEDSL(intel_crtc->pipe));
+ if (scanline > 100 && scanline < 200)
+ break;
+ usleep_range(25, 50);
+ }
+
+ ret = intel_ddi_toggle_hdcp_signalling(&intel_dig_port->base, false);
+ if (ret) {
+ DRM_ERROR("Disable HDCP signalling failed (%d)\n", ret);
+ return ret;
+ }
+ ret = intel_ddi_toggle_hdcp_signalling(&intel_dig_port->base, true);
+ if (ret) {
+ DRM_ERROR("Enable HDCP signalling failed (%d)\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
static
int intel_hdmi_hdcp_toggle_signalling(struct intel_digital_port *intel_dig_port,
bool enable)
{
+ struct intel_hdmi *hdmi = &intel_dig_port->hdmi;
+ struct intel_connector *connector = hdmi->attached_connector;
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
int ret;
if (!enable)
enable ? "Enable" : "Disable", ret);
return ret;
}
+
+ /*
+ * WA: To fix incorrect positioning of the window of
+ * opportunity and enc_en signalling in KABYLAKE.
+ */
+ if (IS_KABYLAKE(dev_priv) && enable)
+ return kbl_repositioning_enc_en_signal(connector);
+
return 0;
}
return true;
}
+static struct hdcp2_hdmi_msg_data {
+ u8 msg_id;
+ u32 timeout;
+ u32 timeout2;
+ } hdcp2_msg_data[] = {
+ {HDCP_2_2_AKE_INIT, 0, 0},
+ {HDCP_2_2_AKE_SEND_CERT, HDCP_2_2_CERT_TIMEOUT_MS, 0},
+ {HDCP_2_2_AKE_NO_STORED_KM, 0, 0},
+ {HDCP_2_2_AKE_STORED_KM, 0, 0},
+ {HDCP_2_2_AKE_SEND_HPRIME, HDCP_2_2_HPRIME_PAIRED_TIMEOUT_MS,
+ HDCP_2_2_HPRIME_NO_PAIRED_TIMEOUT_MS},
+ {HDCP_2_2_AKE_SEND_PAIRING_INFO, HDCP_2_2_PAIRING_TIMEOUT_MS,
+ 0},
+ {HDCP_2_2_LC_INIT, 0, 0},
+ {HDCP_2_2_LC_SEND_LPRIME, HDCP_2_2_HDMI_LPRIME_TIMEOUT_MS, 0},
+ {HDCP_2_2_SKE_SEND_EKS, 0, 0},
+ {HDCP_2_2_REP_SEND_RECVID_LIST,
+ HDCP_2_2_RECVID_LIST_TIMEOUT_MS, 0},
+ {HDCP_2_2_REP_SEND_ACK, 0, 0},
+ {HDCP_2_2_REP_STREAM_MANAGE, 0, 0},
+ {HDCP_2_2_REP_STREAM_READY, HDCP_2_2_STREAM_READY_TIMEOUT_MS,
+ 0},
+ };
+
+static
+int intel_hdmi_hdcp2_read_rx_status(struct intel_digital_port *intel_dig_port,
+ u8 *rx_status)
+{
+ return intel_hdmi_hdcp_read(intel_dig_port,
+ HDCP_2_2_HDMI_REG_RXSTATUS_OFFSET,
+ rx_status,
+ HDCP_2_2_HDMI_RXSTATUS_LEN);
+}
+
+static int get_hdcp2_msg_timeout(u8 msg_id, bool is_paired)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(hdcp2_msg_data); i++)
+ if (hdcp2_msg_data[i].msg_id == msg_id &&
+ (msg_id != HDCP_2_2_AKE_SEND_HPRIME || is_paired))
+ return hdcp2_msg_data[i].timeout;
+ else if (hdcp2_msg_data[i].msg_id == msg_id)
+ return hdcp2_msg_data[i].timeout2;
+
+ return -EINVAL;
+}
+
+static inline
+int hdcp2_detect_msg_availability(struct intel_digital_port *intel_digital_port,
+ u8 msg_id, bool *msg_ready,
+ ssize_t *msg_sz)
+{
+ u8 rx_status[HDCP_2_2_HDMI_RXSTATUS_LEN];
+ int ret;
+
+ ret = intel_hdmi_hdcp2_read_rx_status(intel_digital_port, rx_status);
+ if (ret < 0) {
+ DRM_DEBUG_KMS("rx_status read failed. Err %d\n", ret);
+ return ret;
+ }
+
+ *msg_sz = ((HDCP_2_2_HDMI_RXSTATUS_MSG_SZ_HI(rx_status[1]) << 8) |
+ rx_status[0]);
+
+ if (msg_id == HDCP_2_2_REP_SEND_RECVID_LIST)
+ *msg_ready = (HDCP_2_2_HDMI_RXSTATUS_READY(rx_status[1]) &&
+ *msg_sz);
+ else
+ *msg_ready = *msg_sz;
+
+ return 0;
+}
+
+static ssize_t
+intel_hdmi_hdcp2_wait_for_msg(struct intel_digital_port *intel_dig_port,
+ u8 msg_id, bool paired)
+{
+ bool msg_ready = false;
+ int timeout, ret;
+ ssize_t msg_sz = 0;
+
+ timeout = get_hdcp2_msg_timeout(msg_id, paired);
+ if (timeout < 0)
+ return timeout;
+
+ ret = __wait_for(ret = hdcp2_detect_msg_availability(intel_dig_port,
+ msg_id, &msg_ready,
+ &msg_sz),
+ !ret && msg_ready && msg_sz, timeout * 1000,
+ 1000, 5 * 1000);
+ if (ret)
+ DRM_DEBUG_KMS("msg_id: %d, ret: %d, timeout: %d\n",
+ msg_id, ret, timeout);
+
+ return ret ? ret : msg_sz;
+}
+
+static
+int intel_hdmi_hdcp2_write_msg(struct intel_digital_port *intel_dig_port,
+ void *buf, size_t size)
+{
+ unsigned int offset;
+
+ offset = HDCP_2_2_HDMI_REG_WR_MSG_OFFSET;
+ return intel_hdmi_hdcp_write(intel_dig_port, offset, buf, size);
+}
+
+static
+int intel_hdmi_hdcp2_read_msg(struct intel_digital_port *intel_dig_port,
+ u8 msg_id, void *buf, size_t size)
+{
+ struct intel_hdmi *hdmi = &intel_dig_port->hdmi;
+ struct intel_hdcp *hdcp = &hdmi->attached_connector->hdcp;
+ unsigned int offset;
+ ssize_t ret;
+
+ ret = intel_hdmi_hdcp2_wait_for_msg(intel_dig_port, msg_id,
+ hdcp->is_paired);
+ if (ret < 0)
+ return ret;
+
+ /*
+ * Available msg size should be equal to or lesser than the
+ * available buffer.
+ */
+ if (ret > size) {
+ DRM_DEBUG_KMS("msg_sz(%zd) is more than exp size(%zu)\n",
+ ret, size);
+ return -1;
+ }
+
+ offset = HDCP_2_2_HDMI_REG_RD_MSG_OFFSET;
+ ret = intel_hdmi_hdcp_read(intel_dig_port, offset, buf, ret);
+ if (ret)
+ DRM_DEBUG_KMS("Failed to read msg_id: %d(%zd)\n", msg_id, ret);
+
+ return ret;
+}
+
+static
+int intel_hdmi_hdcp2_check_link(struct intel_digital_port *intel_dig_port)
+{
+ u8 rx_status[HDCP_2_2_HDMI_RXSTATUS_LEN];
+ int ret;
+
+ ret = intel_hdmi_hdcp2_read_rx_status(intel_dig_port, rx_status);
+ if (ret)
+ return ret;
+
+ /*
+ * Re-auth request and Link Integrity Failures are represented by
+ * same bit. i.e reauth_req.
+ */
+ if (HDCP_2_2_HDMI_RXSTATUS_REAUTH_REQ(rx_status[1]))
+ ret = HDCP_REAUTH_REQUEST;
+ else if (HDCP_2_2_HDMI_RXSTATUS_READY(rx_status[1]))
+ ret = HDCP_TOPOLOGY_CHANGE;
+
+ return ret;
+}
+
+static
+int intel_hdmi_hdcp2_capable(struct intel_digital_port *intel_dig_port,
+ bool *capable)
+{
+ u8 hdcp2_version;
+ int ret;
+
+ *capable = false;
+ ret = intel_hdmi_hdcp_read(intel_dig_port, HDCP_2_2_HDMI_REG_VER_OFFSET,
+ &hdcp2_version, sizeof(hdcp2_version));
+ if (!ret && hdcp2_version & HDCP_2_2_HDMI_SUPPORT_MASK)
+ *capable = true;
+
+ return ret;
+}
+
+static inline
+enum hdcp_wired_protocol intel_hdmi_hdcp2_protocol(void)
+{
+ return HDCP_PROTOCOL_HDMI;
+}
+
static const struct intel_hdcp_shim intel_hdmi_hdcp_shim = {
.write_an_aksv = intel_hdmi_hdcp_write_an_aksv,
.read_bksv = intel_hdmi_hdcp_read_bksv,
.read_v_prime_part = intel_hdmi_hdcp_read_v_prime_part,
.toggle_signalling = intel_hdmi_hdcp_toggle_signalling,
.check_link = intel_hdmi_hdcp_check_link,
+ .write_2_2_msg = intel_hdmi_hdcp2_write_msg,
+ .read_2_2_msg = intel_hdmi_hdcp2_read_msg,
+ .check_2_2_link = intel_hdmi_hdcp2_check_link,
+ .hdcp_2_2_capable = intel_hdmi_hdcp2_capable,
+ .protocol = HDCP_PROTOCOL_HDMI,
};
static void intel_hdmi_prepare(struct intel_encoder *encoder,
struct intel_crtc_state *pipe_config)
{
struct intel_hdmi *intel_hdmi = enc_to_intel_hdmi(&encoder->base);
- struct intel_digital_port *intel_dig_port = hdmi_to_dig_port(intel_hdmi);
struct drm_device *dev = encoder->base.dev;
struct drm_i915_private *dev_priv = to_i915(dev);
u32 tmp, flags = 0;
if (tmp & HDMI_MODE_SELECT_HDMI)
pipe_config->has_hdmi_sink = true;
- if (intel_dig_port->infoframe_enabled(encoder, pipe_config))
+ pipe_config->infoframes.enable |=
+ intel_hdmi_infoframes_enabled(encoder, pipe_config);
+
+ if (pipe_config->infoframes.enable)
pipe_config->has_infoframe = true;
if (tmp & SDVO_AUDIO_ENABLE)
pipe_config->base.adjusted_mode.crtc_clock = dotclock;
pipe_config->lane_count = 4;
+
+ intel_hdmi_read_gcp_infoframe(encoder, pipe_config);
+
+ intel_read_infoframe(encoder, pipe_config,
+ HDMI_INFOFRAME_TYPE_AVI,
+ &pipe_config->infoframes.avi);
+ intel_read_infoframe(encoder, pipe_config,
+ HDMI_INFOFRAME_TYPE_SPD,
+ &pipe_config->infoframes.spd);
+ intel_read_infoframe(encoder, pipe_config,
+ HDMI_INFOFRAME_TYPE_VENDOR,
+ &pipe_config->infoframes.hdmi);
}
static void intel_enable_hdmi_audio(struct intel_encoder *encoder,
/* Display Wa_1405510057:icl */
if (crtc_state->output_format == INTEL_OUTPUT_FORMAT_YCBCR420 &&
- bpc == 10 && IS_ICELAKE(dev_priv) &&
+ bpc == 10 && INTEL_GEN(dev_priv) >= 11 &&
(adjusted_mode->crtc_hblank_end -
adjusted_mode->crtc_hblank_start) % 8 == 2)
return false;
}
}
+ intel_hdmi_compute_gcp_infoframe(encoder, pipe_config, conn_state);
+
+ if (!intel_hdmi_compute_avi_infoframe(encoder, pipe_config, conn_state)) {
+ DRM_DEBUG_KMS("bad AVI infoframe\n");
+ return -EINVAL;
+ }
+
+ if (!intel_hdmi_compute_spd_infoframe(encoder, pipe_config, conn_state)) {
+ DRM_DEBUG_KMS("bad SPD infoframe\n");
+ return -EINVAL;
+ }
+
+ if (!intel_hdmi_compute_hdmi_infoframe(encoder, pipe_config, conn_state)) {
+ DRM_DEBUG_KMS("bad HDMI infoframe\n");
+ return -EINVAL;
+ }
+
return 0;
}
wakeref = intel_display_power_get(dev_priv, POWER_DOMAIN_GMBUS);
- if (IS_ICELAKE(dev_priv) &&
+ if (INTEL_GEN(dev_priv) >= 11 &&
!intel_digital_port_connected(encoder))
goto out;
intel_hdmi_add_properties(struct intel_hdmi *intel_hdmi, struct drm_connector *connector)
{
struct drm_i915_private *dev_priv = to_i915(connector->dev);
+ struct intel_digital_port *intel_dig_port =
+ hdmi_to_dig_port(intel_hdmi);
intel_attach_force_audio_property(connector);
intel_attach_broadcast_rgb_property(connector);
intel_attach_aspect_ratio_property(connector);
+
+ /*
+ * Attach Colorspace property for Non LSPCON based device
+ * ToDo: This needs to be extended for LSPCON implementation
+ * as well. Will be implemented separately.
+ */
+ if (!intel_dig_port->lspcon.active)
+ intel_attach_colorspace_property(connector);
+
drm_connector_attach_content_type_property(connector);
connector->state->picture_aspect_ratio = HDMI_PICTURE_ASPECT_NONE;
return info->alternate_ddc_pin;
}
- if (IS_CHERRYVIEW(dev_priv))
- ddc_pin = chv_port_to_ddc_pin(dev_priv, port);
- else if (IS_GEN9_LP(dev_priv))
- ddc_pin = bxt_port_to_ddc_pin(dev_priv, port);
+ if (HAS_PCH_ICP(dev_priv))
+ ddc_pin = icl_port_to_ddc_pin(dev_priv, port);
else if (HAS_PCH_CNP(dev_priv))
ddc_pin = cnp_port_to_ddc_pin(dev_priv, port);
- else if (HAS_PCH_ICP(dev_priv))
- ddc_pin = icl_port_to_ddc_pin(dev_priv, port);
+ else if (IS_GEN9_LP(dev_priv))
+ ddc_pin = bxt_port_to_ddc_pin(dev_priv, port);
+ else if (IS_CHERRYVIEW(dev_priv))
+ ddc_pin = chv_port_to_ddc_pin(dev_priv, port);
else
ddc_pin = g4x_port_to_ddc_pin(dev_priv, port);
if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) {
intel_dig_port->write_infoframe = vlv_write_infoframe;
+ intel_dig_port->read_infoframe = vlv_read_infoframe;
intel_dig_port->set_infoframes = vlv_set_infoframes;
- intel_dig_port->infoframe_enabled = vlv_infoframe_enabled;
+ intel_dig_port->infoframes_enabled = vlv_infoframes_enabled;
} else if (IS_G4X(dev_priv)) {
intel_dig_port->write_infoframe = g4x_write_infoframe;
+ intel_dig_port->read_infoframe = g4x_read_infoframe;
intel_dig_port->set_infoframes = g4x_set_infoframes;
- intel_dig_port->infoframe_enabled = g4x_infoframe_enabled;
+ intel_dig_port->infoframes_enabled = g4x_infoframes_enabled;
} else if (HAS_DDI(dev_priv)) {
if (intel_dig_port->lspcon.active) {
- intel_dig_port->write_infoframe =
- lspcon_write_infoframe;
+ intel_dig_port->write_infoframe = lspcon_write_infoframe;
+ intel_dig_port->read_infoframe = lspcon_read_infoframe;
intel_dig_port->set_infoframes = lspcon_set_infoframes;
- intel_dig_port->infoframe_enabled =
- lspcon_infoframe_enabled;
+ intel_dig_port->infoframes_enabled = lspcon_infoframes_enabled;
} else {
- intel_dig_port->set_infoframes = hsw_set_infoframes;
- intel_dig_port->infoframe_enabled =
- hsw_infoframe_enabled;
intel_dig_port->write_infoframe = hsw_write_infoframe;
+ intel_dig_port->read_infoframe = hsw_read_infoframe;
+ intel_dig_port->set_infoframes = hsw_set_infoframes;
+ intel_dig_port->infoframes_enabled = hsw_infoframes_enabled;
}
} else if (HAS_PCH_IBX(dev_priv)) {
intel_dig_port->write_infoframe = ibx_write_infoframe;
+ intel_dig_port->read_infoframe = ibx_read_infoframe;
intel_dig_port->set_infoframes = ibx_set_infoframes;
- intel_dig_port->infoframe_enabled = ibx_infoframe_enabled;
+ intel_dig_port->infoframes_enabled = ibx_infoframes_enabled;
} else {
intel_dig_port->write_infoframe = cpt_write_infoframe;
+ intel_dig_port->read_infoframe = cpt_read_infoframe;
intel_dig_port->set_infoframes = cpt_set_infoframes;
- intel_dig_port->infoframe_enabled = cpt_infoframe_enabled;
+ intel_dig_port->infoframes_enabled = cpt_infoframes_enabled;
}
}
intel_hdmi_add_properties(intel_hdmi, connector);
+ intel_connector_attach_encoder(intel_connector, intel_encoder);
+ intel_hdmi->attached_connector = intel_connector;
+
if (is_hdcp_supported(dev_priv, port)) {
int ret = intel_hdcp_init(intel_connector,
&intel_hdmi_hdcp_shim);
DRM_DEBUG_KMS("HDCP init failed, skipping.\n");
}
- intel_connector_attach_encoder(intel_connector, intel_encoder);
- intel_hdmi->attached_connector = intel_connector;
-
/* For G4X desktop chip, PEG_BAND_GAP_DATA 3:0 must first be written
* 0xd. Failure to do so will result in spurious interrupts being
* generated on the port when a cable is not attached.
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_HDMI_H__
+#define __INTEL_HDMI_H__
+
+#include <linux/hdmi.h>
+#include <linux/types.h>
+
+#include <drm/i915_drm.h>
+
+#include "i915_reg.h"
+
+struct drm_connector;
+struct drm_encoder;
+struct drm_i915_private;
+struct intel_connector;
+struct intel_digital_port;
+struct intel_encoder;
+struct intel_crtc_state;
+struct intel_hdmi;
+struct drm_connector_state;
+union hdmi_infoframe;
+
+void intel_hdmi_init(struct drm_i915_private *dev_priv, i915_reg_t hdmi_reg,
+ enum port port);
+void intel_hdmi_init_connector(struct intel_digital_port *intel_dig_port,
+ struct intel_connector *intel_connector);
+struct intel_hdmi *enc_to_intel_hdmi(struct drm_encoder *encoder);
+int intel_hdmi_compute_config(struct intel_encoder *encoder,
+ struct intel_crtc_state *pipe_config,
+ struct drm_connector_state *conn_state);
+bool intel_hdmi_handle_sink_scrambling(struct intel_encoder *encoder,
+ struct drm_connector *connector,
+ bool high_tmds_clock_ratio,
+ bool scrambling);
+void intel_dp_dual_mode_set_tmds_output(struct intel_hdmi *hdmi, bool enable);
+void intel_infoframe_init(struct intel_digital_port *intel_dig_port);
+u32 intel_hdmi_infoframes_enabled(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state);
+u32 intel_hdmi_infoframe_enable(unsigned int type);
+void intel_hdmi_read_gcp_infoframe(struct intel_encoder *encoder,
+ struct intel_crtc_state *crtc_state);
+void intel_read_infoframe(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ enum hdmi_infoframe_type type,
+ union hdmi_infoframe *frame);
+
+#endif /* __INTEL_HDMI_H__ */
}
/* Check authentication status, it should be done by now */
- ret = __intel_wait_for_register(i915,
+ ret = __intel_wait_for_register(&i915->uncore,
HUC_STATUS2,
HUC_FW_VERIFIED,
HUC_FW_VERIFIED,
{
struct intel_huc *huc = container_of(huc_fw, struct intel_huc, fw);
struct drm_i915_private *dev_priv = huc_to_i915(huc);
+ struct intel_uncore *uncore = &dev_priv->uncore;
unsigned long offset = 0;
u32 size;
int ret;
GEM_BUG_ON(huc_fw->type != INTEL_UC_FW_TYPE_HUC);
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
/* Set the source address for the uCode */
offset = intel_guc_ggtt_offset(&dev_priv->guc, vma) +
huc_fw->header_offset;
- I915_WRITE(DMA_ADDR_0_LOW, lower_32_bits(offset));
- I915_WRITE(DMA_ADDR_0_HIGH, upper_32_bits(offset) & 0xFFFF);
+ intel_uncore_write(uncore, DMA_ADDR_0_LOW,
+ lower_32_bits(offset));
+ intel_uncore_write(uncore, DMA_ADDR_0_HIGH,
+ upper_32_bits(offset) & 0xFFFF);
- /* Hardware doesn't look at destination address for HuC. Set it to 0,
+ /*
+ * Hardware doesn't look at destination address for HuC. Set it to 0,
* but still program the correct address space.
*/
- I915_WRITE(DMA_ADDR_1_LOW, 0);
- I915_WRITE(DMA_ADDR_1_HIGH, DMA_ADDRESS_SPACE_WOPCM);
+ intel_uncore_write(uncore, DMA_ADDR_1_LOW, 0);
+ intel_uncore_write(uncore, DMA_ADDR_1_HIGH, DMA_ADDRESS_SPACE_WOPCM);
size = huc_fw->header_size + huc_fw->ucode_size;
- I915_WRITE(DMA_COPY_SIZE, size);
+ intel_uncore_write(uncore, DMA_COPY_SIZE, size);
/* Start the DMA */
- I915_WRITE(DMA_CTRL, _MASKED_BIT_ENABLE(HUC_UKERNEL | START_DMA));
+ intel_uncore_write(uncore, DMA_CTRL,
+ _MASKED_BIT_ENABLE(HUC_UKERNEL | START_DMA));
/* Wait for DMA to finish */
- ret = intel_wait_for_register_fw(dev_priv, DMA_CTRL, START_DMA, 0, 100);
+ ret = intel_wait_for_register_fw(uncore, DMA_CTRL, START_DMA, 0, 100);
DRM_DEBUG_DRIVER("HuC DMA transfer wait over with ret %d\n", ret);
/* Disable the bits once DMA is over */
- I915_WRITE(DMA_CTRL, _MASKED_BIT_DISABLE(HUC_UKERNEL));
+ intel_uncore_write(uncore, DMA_CTRL, _MASKED_BIT_DISABLE(HUC_UKERNEL));
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(uncore, FORCEWAKE_ALL);
return ret;
}
add_wait_queue(&dev_priv->gmbus_wait_queue, &wait);
I915_WRITE_FW(GMBUS4, irq_enable);
- ret = intel_wait_for_register_fw(dev_priv,
+ ret = intel_wait_for_register_fw(&dev_priv->uncore,
GMBUS2, GMBUS_ACTIVE, 0,
10);
#define WA_TAIL_DWORDS 2
#define WA_TAIL_BYTES (sizeof(u32) * WA_TAIL_DWORDS)
-static int execlists_context_deferred_alloc(struct i915_gem_context *ctx,
- struct intel_engine_cs *engine,
- struct intel_context *ce);
+#define ACTIVE_PRIORITY (I915_PRIORITY_NEWCLIENT | I915_PRIORITY_NOSEMAPHORE)
+
+static int execlists_context_deferred_alloc(struct intel_context *ce,
+ struct intel_engine_cs *engine);
static void execlists_init_reg_state(u32 *reg_state,
- struct i915_gem_context *ctx,
+ struct intel_context *ce,
struct intel_engine_cs *engine,
struct intel_ring *ring);
-static inline u32 intel_hws_seqno_address(struct intel_engine_cs *engine)
-{
- return (i915_ggtt_offset(engine->status_page.vma) +
- I915_GEM_HWS_INDEX_ADDR);
-}
-
static inline struct i915_priolist *to_priolist(struct rb_node *rb)
{
return rb_entry(rb, struct i915_priolist, node);
return rq->sched.attr.priority;
}
+static int effective_prio(const struct i915_request *rq)
+{
+ int prio = rq_prio(rq);
+
+ /*
+ * On unwinding the active request, we give it a priority bump
+ * equivalent to a freshly submitted request. This protects it from
+ * being gazumped again, but it would be preferable if we didn't
+ * let it be gazumped in the first place!
+ *
+ * See __unwind_incomplete_requests()
+ */
+ if (~prio & ACTIVE_PRIORITY && __i915_request_has_started(rq)) {
+ /*
+ * After preemption, we insert the active request at the
+ * end of the new priority level. This means that we will be
+ * _lower_ priority than the preemptee all things equal (and
+ * so the preemption is valid), so adjust our comparison
+ * accordingly.
+ */
+ prio |= ACTIVE_PRIORITY;
+ prio--;
+ }
+
+ /* Restrict mere WAIT boosts from triggering preemption */
+ return prio | __NO_PREEMPTION;
+}
+
static int queue_prio(const struct intel_engine_execlists *execlists)
{
struct i915_priolist *p;
static inline bool need_preempt(const struct intel_engine_cs *engine,
const struct i915_request *rq)
{
- const int last_prio = rq_prio(rq);
+ int last_prio;
- if (!intel_engine_has_preemption(engine))
+ if (!engine->preempt_context)
return false;
if (i915_request_completed(rq))
* preempt. If that hint is stale or we may be trying to preempt
* ourselves, ignore the request.
*/
+ last_prio = effective_prio(rq);
if (!__execlists_need_preempt(engine->execlists.queue_priority_hint,
last_prio))
return false;
}
__maybe_unused static inline bool
-assert_priority_queue(const struct intel_engine_execlists *execlists,
- const struct i915_request *prev,
+assert_priority_queue(const struct i915_request *prev,
const struct i915_request *next)
{
- if (!prev)
- return true;
+ const struct intel_engine_execlists *execlists =
+ &prev->engine->execlists;
/*
* Without preemption, the prev may refer to the still active element
* engine info, SW context ID and SW counter need to form a unique number
* (Context ID) per lrc.
*/
-static void
-intel_lr_context_descriptor_update(struct i915_gem_context *ctx,
- struct intel_engine_cs *engine,
- struct intel_context *ce)
+static u64
+lrc_descriptor(struct intel_context *ce, struct intel_engine_cs *engine)
{
+ struct i915_gem_context *ctx = ce->gem_context;
u64 desc;
BUILD_BUG_ON(MAX_CONTEXT_HW_ID > (BIT(GEN8_CTX_ID_WIDTH)));
* Consider updating oa_get_render_ctx_id in i915_perf.c when changing
* anything below.
*/
- if (INTEL_GEN(ctx->i915) >= 11) {
+ if (INTEL_GEN(engine->i915) >= 11) {
GEM_BUG_ON(ctx->hw_id >= BIT(GEN11_SW_CTX_ID_WIDTH));
desc |= (u64)ctx->hw_id << GEN11_SW_CTX_ID_SHIFT;
/* bits 37-47 */
desc |= (u64)ctx->hw_id << GEN8_CTX_ID_SHIFT; /* bits 32-52 */
}
- ce->lrc_desc = desc;
+ return desc;
}
static void unwind_wa_tail(struct i915_request *rq)
{
struct i915_request *rq, *rn, *active = NULL;
struct list_head *uninitialized_var(pl);
- int prio = I915_PRIORITY_INVALID | I915_PRIORITY_NEWCLIENT;
+ int prio = I915_PRIORITY_INVALID | ACTIVE_PRIORITY;
lockdep_assert_held(&engine->timeline.lock);
* The active request is now effectively the start of a new client
* stream, so give it the equivalent small priority bump to prevent
* it being gazumped a second time by another peer.
+ *
+ * Note we have to be careful not to apply a priority boost to a request
+ * still spinning on its semaphores. If the request hasn't started, that
+ * means it is still waiting for its dependencies to be signaled, and
+ * if we apply a priority boost to this request, we will boost it past
+ * its signalers and so break PI.
+ *
+ * One consequence of this preemption boost is that we may jump
+ * over lesser priorities (such as I915_PRIORITY_WAIT), effectively
+ * making those priorities non-preemptible. They will be moved forward
+ * in the priority queue, but they will not gain immediate access to
+ * the GPU.
*/
- if (!(prio & I915_PRIORITY_NEWCLIENT)) {
- prio |= I915_PRIORITY_NEWCLIENT;
+ if (~prio & ACTIVE_PRIORITY && __i915_request_has_started(active)) {
+ prio |= ACTIVE_PRIORITY;
active->sched.attr.priority = prio;
list_move_tail(&active->sched.link,
i915_sched_lookup_priolist(engine, prio));
return active;
}
-void
+struct i915_request *
execlists_unwind_incomplete_requests(struct intel_engine_execlists *execlists)
{
struct intel_engine_cs *engine =
container_of(execlists, typeof(*engine), execlists);
- __unwind_incomplete_requests(engine);
+ return __unwind_incomplete_requests(engine);
}
static inline void
desc = execlists_update_context(rq);
GEM_DEBUG_EXEC(port[n].context_id = upper_32_bits(desc));
- GEM_TRACE("%s in[%d]: ctx=%d.%d, global=%d (fence %llx:%lld) (current %d:%d), prio=%d\n",
+ GEM_TRACE("%s in[%d]: ctx=%d.%d, fence %llx:%lld (current %d), prio=%d\n",
engine->name, n,
port[n].context_id, count,
- rq->global_seqno,
rq->fence.context, rq->fence.seqno,
hwsp_seqno(rq),
- intel_engine_get_seqno(engine),
rq_prio(rq));
} else {
GEM_BUG_ON(!n);
return true;
}
+static bool can_merge_rq(const struct i915_request *prev,
+ const struct i915_request *next)
+{
+ GEM_BUG_ON(!assert_priority_queue(prev, next));
+
+ if (!can_merge_ctx(prev->hw_context, next->hw_context))
+ return false;
+
+ return true;
+}
+
static void port_assign(struct execlist_port *port, struct i915_request *rq)
{
GEM_BUG_ON(rq == port_request(port));
static void inject_preempt_context(struct intel_engine_cs *engine)
{
struct intel_engine_execlists *execlists = &engine->execlists;
- struct intel_context *ce =
- to_intel_context(engine->i915->preempt_context, engine);
+ struct intel_context *ce = engine->preempt_context;
unsigned int n;
GEM_BUG_ON(execlists->preempt_complete_status !=
int i;
priolist_for_each_request_consume(rq, rn, p, i) {
- GEM_BUG_ON(!assert_priority_queue(execlists, last, rq));
-
/*
* Can we combine this request with the current port?
* It has to be the same context/ringbuffer and not
* second request, and so we never need to tell the
* hardware about the first.
*/
- if (last &&
- !can_merge_ctx(rq->hw_context, last->hw_context)) {
+ if (last && !can_merge_rq(last, rq)) {
/*
* If we are on the second port and cannot
* combine this request with the last, then we
if (port == last_port)
goto done;
+ /*
+ * We must not populate both ELSP[] with the
+ * same LRCA, i.e. we must submit 2 different
+ * contexts if we submit 2 ELSP.
+ */
+ if (last->hw_context == rq->hw_context)
+ goto done;
+
/*
* If GVT overrides us we only ever submit
* port[0], leaving port[1] empty. Note that we
ctx_single_port_submission(rq->hw_context))
goto done;
- GEM_BUG_ON(last->hw_context == rq->hw_context);
if (submit)
port_assign(port, last);
}
rb_erase_cached(&p->node, &execlists->queue);
- if (p->priority != I915_PRIORITY_NORMAL)
- kmem_cache_free(engine->i915->priorities, p);
+ i915_priolist_free(p);
}
done:
* request triggering preemption on the next dequeue (or subsequent
* interrupt for secondary ports).
*/
- execlists->queue_priority_hint =
- port != execlists->port ? rq_prio(last) : INT_MIN;
+ execlists->queue_priority_hint = queue_prio(execlists);
if (submit) {
port_assign(port, last);
while (num_ports-- && port_isset(port)) {
struct i915_request *rq = port_request(port);
- GEM_TRACE("%s:port%u global=%d (fence %llx:%lld), (current %d:%d)\n",
+ GEM_TRACE("%s:port%u fence %llx:%lld, (current %d)\n",
rq->engine->name,
(unsigned int)(port - execlists->port),
- rq->global_seqno,
rq->fence.context, rq->fence.seqno,
- hwsp_seqno(rq),
- intel_engine_get_seqno(rq->engine));
+ hwsp_seqno(rq));
GEM_BUG_ON(!execlists->active);
execlists_context_schedule_out(rq,
clflush((void *)last);
}
-static void reset_csb_pointers(struct intel_engine_execlists *execlists)
-{
- const unsigned int reset_value = GEN8_CSB_ENTRIES - 1;
-
- /*
- * After a reset, the HW starts writing into CSB entry [0]. We
- * therefore have to set our HEAD pointer back one entry so that
- * the *first* entry we check is entry 0. To complicate this further,
- * as we don't wait for the first interrupt after reset, we have to
- * fake the HW write to point back to the last entry so that our
- * inline comparison of our cached head position against the last HW
- * write works even before the first interrupt.
- */
- execlists->csb_head = reset_value;
- WRITE_ONCE(*execlists->csb_write, reset_value);
-
- invalidate_csb_entries(&execlists->csb_status[0],
- &execlists->csb_status[GEN8_CSB_ENTRIES - 1]);
-}
-
-static void nop_submission_tasklet(unsigned long data)
-{
- /* The driver is wedged; don't process any more events. */
-}
-
-static void execlists_cancel_requests(struct intel_engine_cs *engine)
-{
- struct intel_engine_execlists * const execlists = &engine->execlists;
- struct i915_request *rq, *rn;
- struct rb_node *rb;
- unsigned long flags;
-
- GEM_TRACE("%s current %d\n",
- engine->name, intel_engine_get_seqno(engine));
-
- /*
- * Before we call engine->cancel_requests(), we should have exclusive
- * access to the submission state. This is arranged for us by the
- * caller disabling the interrupt generation, the tasklet and other
- * threads that may then access the same state, giving us a free hand
- * to reset state. However, we still need to let lockdep be aware that
- * we know this state may be accessed in hardirq context, so we
- * disable the irq around this manipulation and we want to keep
- * the spinlock focused on its duties and not accidentally conflate
- * coverage to the submission's irq state. (Similarly, although we
- * shouldn't need to disable irq around the manipulation of the
- * submission's irq state, we also wish to remind ourselves that
- * it is irq state.)
- */
- spin_lock_irqsave(&engine->timeline.lock, flags);
-
- /* Cancel the requests on the HW and clear the ELSP tracker. */
- execlists_cancel_port_requests(execlists);
- execlists_user_end(execlists);
-
- /* Mark all executing requests as skipped. */
- list_for_each_entry(rq, &engine->timeline.requests, link) {
- GEM_BUG_ON(!rq->global_seqno);
-
- if (!i915_request_signaled(rq))
- dma_fence_set_error(&rq->fence, -EIO);
-
- i915_request_mark_complete(rq);
- }
-
- /* Flush the queued requests to the timeline list (for retiring). */
- while ((rb = rb_first_cached(&execlists->queue))) {
- struct i915_priolist *p = to_priolist(rb);
- int i;
-
- priolist_for_each_request_consume(rq, rn, p, i) {
- list_del_init(&rq->sched.link);
- __i915_request_submit(rq);
- dma_fence_set_error(&rq->fence, -EIO);
- i915_request_mark_complete(rq);
- }
-
- rb_erase_cached(&p->node, &execlists->queue);
- if (p->priority != I915_PRIORITY_NORMAL)
- kmem_cache_free(engine->i915->priorities, p);
- }
-
- intel_write_status_page(engine,
- I915_GEM_HWS_INDEX,
- intel_engine_last_submit(engine));
-
- /* Remaining _unready_ requests will be nop'ed when submitted */
-
- execlists->queue_priority_hint = INT_MIN;
- execlists->queue = RB_ROOT_CACHED;
- GEM_BUG_ON(port_isset(execlists->port));
-
- GEM_BUG_ON(__tasklet_is_enabled(&execlists->tasklet));
- execlists->tasklet.func = nop_submission_tasklet;
-
- spin_unlock_irqrestore(&engine->timeline.lock, flags);
-}
-
static inline bool
reset_in_progress(const struct intel_engine_execlists *execlists)
{
struct intel_engine_execlists * const execlists = &engine->execlists;
struct execlist_port *port = execlists->port;
const u32 * const buf = execlists->csb_status;
+ const u8 num_entries = execlists->csb_size;
u8 head, tail;
lockdep_assert_held(&engine->timeline.lock);
unsigned int status;
unsigned int count;
- if (++head == GEN8_CSB_ENTRIES)
+ if (++head == num_entries)
head = 0;
/*
EXECLISTS_ACTIVE_USER));
rq = port_unpack(port, &count);
- GEM_TRACE("%s out[0]: ctx=%d.%d, global=%d (fence %llx:%lld) (current %d:%d), prio=%d\n",
+ GEM_TRACE("%s out[0]: ctx=%d.%d, fence %llx:%lld (current %d), prio=%d\n",
engine->name,
port->context_id, count,
- rq ? rq->global_seqno : 0,
rq ? rq->fence.context : 0,
rq ? rq->fence.seqno : 0,
rq ? hwsp_seqno(rq) : 0,
- intel_engine_get_seqno(engine),
rq ? rq_prio(rq) : 0);
/* Check the context/desc id for this event matches */
* the wash as hardware, working or not, will need to do the
* invalidation before.
*/
- invalidate_csb_entries(&buf[0], &buf[GEN8_CSB_ENTRIES - 1]);
+ invalidate_csb_entries(&buf[0], &buf[num_entries - 1]);
}
static void __execlists_submission_tasklet(struct intel_engine_cs *const engine)
spin_unlock_irqrestore(&engine->timeline.lock, flags);
}
-static void execlists_context_destroy(struct intel_context *ce)
+static void __execlists_context_fini(struct intel_context *ce)
{
- GEM_BUG_ON(ce->pin_count);
-
- if (!ce->state)
- return;
-
- intel_ring_free(ce->ring);
+ intel_ring_put(ce->ring);
GEM_BUG_ON(i915_gem_object_is_active(ce->state->obj));
i915_gem_object_put(ce->state->obj);
}
+static void execlists_context_destroy(struct kref *kref)
+{
+ struct intel_context *ce = container_of(kref, typeof(*ce), ref);
+
+ GEM_BUG_ON(intel_context_is_pinned(ce));
+
+ if (ce->state)
+ __execlists_context_fini(ce);
+
+ intel_context_free(ce);
+}
+
+static int __context_pin(struct i915_vma *vma)
+{
+ unsigned int flags;
+ int err;
+
+ flags = PIN_GLOBAL | PIN_HIGH;
+ flags |= PIN_OFFSET_BIAS | i915_ggtt_pin_bias(vma);
+
+ err = i915_vma_pin(vma, 0, 0, flags);
+ if (err)
+ return err;
+
+ vma->obj->pin_global++;
+ vma->obj->mm.dirty = true;
+
+ return 0;
+}
+
+static void __context_unpin(struct i915_vma *vma)
+{
+ vma->obj->pin_global--;
+ __i915_vma_unpin(vma);
+}
+
static void execlists_context_unpin(struct intel_context *ce)
{
struct intel_engine_cs *engine;
intel_ring_unpin(ce->ring);
- ce->state->obj->pin_global--;
i915_gem_object_unpin_map(ce->state->obj);
- i915_vma_unpin(ce->state);
-
- i915_gem_context_put(ce->gem_context);
-}
-
-static int __context_pin(struct i915_gem_context *ctx, struct i915_vma *vma)
-{
- unsigned int flags;
- int err;
-
- /*
- * Clear this page out of any CPU caches for coherent swap-in/out.
- * We only want to do this on the first bind so that we do not stall
- * on an active context (which by nature is already on the GPU).
- */
- if (!(vma->flags & I915_VMA_GLOBAL_BIND)) {
- err = i915_gem_object_set_to_wc_domain(vma->obj, true);
- if (err)
- return err;
- }
-
- flags = PIN_GLOBAL | PIN_HIGH;
- flags |= PIN_OFFSET_BIAS | i915_ggtt_pin_bias(vma);
-
- return i915_vma_pin(vma, 0, 0, flags);
+ __context_unpin(ce->state);
}
static void
-__execlists_update_reg_state(struct intel_engine_cs *engine,
- struct intel_context *ce)
+__execlists_update_reg_state(struct intel_context *ce,
+ struct intel_engine_cs *engine)
{
- u32 *regs = ce->lrc_reg_state;
struct intel_ring *ring = ce->ring;
+ u32 *regs = ce->lrc_reg_state;
+
+ GEM_BUG_ON(!intel_ring_offset_valid(ring, ring->head));
+ GEM_BUG_ON(!intel_ring_offset_valid(ring, ring->tail));
regs[CTX_RING_BUFFER_START + 1] = i915_ggtt_offset(ring->vma);
regs[CTX_RING_HEAD + 1] = ring->head;
/* RPCS */
if (engine->class == RENDER_CLASS)
- regs[CTX_R_PWR_CLK_STATE + 1] = gen8_make_rpcs(engine->i915,
- &ce->sseu);
+ regs[CTX_R_PWR_CLK_STATE + 1] =
+ gen8_make_rpcs(engine->i915, &ce->sseu);
}
-static struct intel_context *
-__execlists_context_pin(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx,
- struct intel_context *ce)
+static int
+__execlists_context_pin(struct intel_context *ce,
+ struct intel_engine_cs *engine)
{
void *vaddr;
int ret;
- ret = execlists_context_deferred_alloc(ctx, engine, ce);
+ GEM_BUG_ON(!ce->gem_context->ppgtt);
+
+ ret = execlists_context_deferred_alloc(ce, engine);
if (ret)
goto err;
GEM_BUG_ON(!ce->state);
- ret = __context_pin(ctx, ce->state);
+ ret = __context_pin(ce->state);
if (ret)
goto err;
vaddr = i915_gem_object_pin_map(ce->state->obj,
- i915_coherent_map_type(ctx->i915) |
+ i915_coherent_map_type(engine->i915) |
I915_MAP_OVERRIDE);
if (IS_ERR(vaddr)) {
ret = PTR_ERR(vaddr);
if (ret)
goto unpin_map;
- ret = i915_gem_context_pin_hw_id(ctx);
+ ret = i915_gem_context_pin_hw_id(ce->gem_context);
if (ret)
goto unpin_ring;
- intel_lr_context_descriptor_update(ctx, engine, ce);
-
- GEM_BUG_ON(!intel_ring_offset_valid(ce->ring, ce->ring->head));
-
+ ce->lrc_desc = lrc_descriptor(ce, engine);
ce->lrc_reg_state = vaddr + LRC_STATE_PN * PAGE_SIZE;
+ __execlists_update_reg_state(ce, engine);
- __execlists_update_reg_state(engine, ce);
-
- ce->state->obj->pin_global++;
- i915_gem_context_get(ctx);
- return ce;
+ return 0;
unpin_ring:
intel_ring_unpin(ce->ring);
unpin_map:
i915_gem_object_unpin_map(ce->state->obj);
unpin_vma:
- __i915_vma_unpin(ce->state);
+ __context_unpin(ce->state);
err:
- ce->pin_count = 0;
- return ERR_PTR(ret);
+ return ret;
}
-static const struct intel_context_ops execlists_context_ops = {
- .unpin = execlists_context_unpin,
- .destroy = execlists_context_destroy,
-};
-
-static struct intel_context *
-execlists_context_pin(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx)
+static int execlists_context_pin(struct intel_context *ce)
{
- struct intel_context *ce = to_intel_context(ctx, engine);
-
- lockdep_assert_held(&ctx->i915->drm.struct_mutex);
- GEM_BUG_ON(!ctx->ppgtt);
+ return __execlists_context_pin(ce, ce->engine);
+}
- if (likely(ce->pin_count++))
- return ce;
- GEM_BUG_ON(!ce->pin_count); /* no overflow please! */
+static void execlists_context_reset(struct intel_context *ce)
+{
+ /*
+ * Because we emit WA_TAIL_DWORDS there may be a disparity
+ * between our bookkeeping in ce->ring->head and ce->ring->tail and
+ * that stored in context. As we only write new commands from
+ * ce->ring->tail onwards, everything before that is junk. If the GPU
+ * starts reading from its RING_HEAD from the context, it may try to
+ * execute that junk and die.
+ *
+ * The contexts that are stilled pinned on resume belong to the
+ * kernel, and are local to each engine. All other contexts will
+ * have their head/tail sanitized upon pinning before use, so they
+ * will never see garbage,
+ *
+ * So to avoid that we reset the context images upon resume. For
+ * simplicity, we just zero everything out.
+ */
+ intel_ring_reset(ce->ring, 0);
+ __execlists_update_reg_state(ce, ce->engine);
+}
- ce->ops = &execlists_context_ops;
+static const struct intel_context_ops execlists_context_ops = {
+ .pin = execlists_context_pin,
+ .unpin = execlists_context_unpin,
- return __execlists_context_pin(engine, ctx, ce);
-}
+ .reset = execlists_context_reset,
+ .destroy = execlists_context_destroy,
+};
static int gen8_emit_init_breadcrumb(struct i915_request *rq)
{
*cs++ = rq->fence.seqno - 1;
intel_ring_advance(rq, cs);
+
+ /* Record the updated position of the request's payload */
+ rq->infix = intel_ring_offset(rq, cs);
+
return 0;
}
*cs++ = MI_LOAD_REGISTER_IMM(2 * GEN8_3LVL_PDPES) | MI_LRI_FORCE_POSTED;
for (i = GEN8_3LVL_PDPES; i--; ) {
const dma_addr_t pd_daddr = i915_page_dir_dma_addr(ppgtt, i);
+ u32 base = engine->mmio_base;
- *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_UDW(engine, i));
+ *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_UDW(base, i));
*cs++ = upper_32_bits(pd_daddr);
- *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_LDW(engine, i));
+ *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_LDW(base, i));
*cs++ = lower_32_bits(pd_daddr);
}
*cs++ = MI_NOOP;
{
int ret;
- GEM_BUG_ON(!request->hw_context->pin_count);
+ GEM_BUG_ON(!intel_context_is_pinned(request->hw_context));
/*
* Flush enough space to reduce the likelihood of waiting after
*/
/* Unconditionally invalidate GPU caches and TLBs. */
- if (i915_vm_is_48bit(&request->gem_context->ppgtt->vm))
+ if (i915_vm_is_4lvl(&request->gem_context->ppgtt->vm))
ret = request->engine->emit_flush(request, EMIT_INVALIDATE);
else
ret = emit_pdps(request);
unsigned int i;
int ret;
- if (GEM_DEBUG_WARN_ON(engine->id != RCS))
+ if (GEM_DEBUG_WARN_ON(engine->id != RCS0))
return -EINVAL;
switch (INTEL_GEN(engine->i915)) {
intel_engine_set_hwsp_writemask(engine, ~0u); /* HWSTAM */
- /*
- * Make sure we're not enabling the new 12-deep CSB
- * FIFO as that requires a slightly updated handling
- * in the ctx switch irq. Since we're currently only
- * using only 2 elements of the enhanced execlists the
- * deeper FIFO it's not needed and it's not worth adding
- * more statements to the irq handler to support it.
- */
if (INTEL_GEN(dev_priv) >= 11)
I915_WRITE(RING_MODE_GEN7(engine),
- _MASKED_BIT_DISABLE(GEN11_GFX_DISABLE_LEGACY_MODE));
+ _MASKED_BIT_ENABLE(GEN11_GFX_DISABLE_LEGACY_MODE));
else
I915_WRITE(RING_MODE_GEN7(engine),
_MASKED_BIT_ENABLE(GFX_RUN_LIST_ENABLE));
__tasklet_disable_sync_once(&execlists->tasklet);
GEM_BUG_ON(!reset_in_progress(execlists));
+ intel_engine_stop_cs(engine);
+
/* And flush any current direct submission. */
spin_lock_irqsave(&engine->timeline.lock, flags);
- process_csb(engine); /* drain preemption events */
spin_unlock_irqrestore(&engine->timeline.lock, flags);
}
-static void execlists_reset(struct intel_engine_cs *engine, bool stalled)
+static bool lrc_regs_ok(const struct i915_request *rq)
+{
+ const struct intel_ring *ring = rq->ring;
+ const u32 *regs = rq->hw_context->lrc_reg_state;
+
+ /* Quick spot check for the common signs of context corruption */
+
+ if (regs[CTX_RING_BUFFER_CONTROL + 1] !=
+ (RING_CTL_SIZE(ring->size) | RING_VALID))
+ return false;
+
+ if (regs[CTX_RING_BUFFER_START + 1] != i915_ggtt_offset(ring->vma))
+ return false;
+
+ return true;
+}
+
+static void reset_csb_pointers(struct intel_engine_execlists *execlists)
+{
+ const unsigned int reset_value = execlists->csb_size - 1;
+
+ /*
+ * After a reset, the HW starts writing into CSB entry [0]. We
+ * therefore have to set our HEAD pointer back one entry so that
+ * the *first* entry we check is entry 0. To complicate this further,
+ * as we don't wait for the first interrupt after reset, we have to
+ * fake the HW write to point back to the last entry so that our
+ * inline comparison of our cached head position against the last HW
+ * write works even before the first interrupt.
+ */
+ execlists->csb_head = reset_value;
+ WRITE_ONCE(*execlists->csb_write, reset_value);
+ wmb(); /* Make sure this is visible to HW (paranoia?) */
+
+ invalidate_csb_entries(&execlists->csb_status[0],
+ &execlists->csb_status[reset_value]);
+}
+
+static void __execlists_reset(struct intel_engine_cs *engine, bool stalled)
{
struct intel_engine_execlists * const execlists = &engine->execlists;
+ struct intel_context *ce;
struct i915_request *rq;
- unsigned long flags;
u32 *regs;
- spin_lock_irqsave(&engine->timeline.lock, flags);
+ process_csb(engine); /* drain preemption events */
+
+ /* Following the reset, we need to reload the CSB read/write pointers */
+ reset_csb_pointers(&engine->execlists);
+
+ /*
+ * Save the currently executing context, even if we completed
+ * its request, it was still running at the time of the
+ * reset and will have been clobbered.
+ */
+ if (!port_isset(execlists->port))
+ goto out_clear;
+
+ ce = port_request(execlists->port)->hw_context;
/*
* Catch up with any missed context-switch interrupts.
/* Push back any incomplete requests for replay after the reset. */
rq = __unwind_incomplete_requests(engine);
+ if (!rq)
+ goto out_replay;
- /* Following the reset, we need to reload the CSB read/write pointers */
- reset_csb_pointers(&engine->execlists);
+ if (rq->hw_context != ce) { /* caught just before a CS event */
+ rq = NULL;
+ goto out_replay;
+ }
- GEM_TRACE("%s seqno=%d, current=%d, stalled? %s\n",
- engine->name,
- rq ? rq->global_seqno : 0,
- intel_engine_get_seqno(engine),
- yesno(stalled));
- if (!rq)
- goto out_unlock;
+ /*
+ * If this request hasn't started yet, e.g. it is waiting on a
+ * semaphore, we need to avoid skipping the request or else we
+ * break the signaling chain. However, if the context is corrupt
+ * the request will not restart and we will be stuck with a wedged
+ * device. It is quite often the case that if we issue a reset
+ * while the GPU is loading the context image, that the context
+ * image becomes corrupt.
+ *
+ * Otherwise, if we have not started yet, the request should replay
+ * perfectly and we do not need to flag the result as being erroneous.
+ */
+ if (!i915_request_started(rq) && lrc_regs_ok(rq))
+ goto out_replay;
/*
* If the request was innocent, we leave the request in the ELSP
* image back to the expected values to skip over the guilty request.
*/
i915_reset_request(rq, stalled);
- if (!stalled)
- goto out_unlock;
+ if (!stalled && lrc_regs_ok(rq))
+ goto out_replay;
/*
* We want a simple context + ring to execute the breadcrumb update.
* future request will be after userspace has had the opportunity
* to recreate its own state.
*/
- regs = rq->hw_context->lrc_reg_state;
+ regs = ce->lrc_reg_state;
if (engine->pinned_default_state) {
memcpy(regs, /* skip restoring the vanilla PPHWSP */
engine->pinned_default_state + LRC_STATE_PN * PAGE_SIZE,
engine->context_size - PAGE_SIZE);
}
+ execlists_init_reg_state(regs, ce, engine, ce->ring);
- /* Move the RING_HEAD onto the breadcrumb, past the hanging batch */
- rq->ring->head = intel_ring_wrap(rq->ring, rq->postfix);
- intel_ring_update_space(rq->ring);
+ /* Rerun the request; its payload has been neutered (if guilty). */
+out_replay:
+ ce->ring->head =
+ rq ? intel_ring_wrap(ce->ring, rq->head) : ce->ring->tail;
+ intel_ring_update_space(ce->ring);
+ __execlists_update_reg_state(ce, engine);
+
+out_clear:
+ execlists_clear_all_active(execlists);
+}
+
+static void execlists_reset(struct intel_engine_cs *engine, bool stalled)
+{
+ unsigned long flags;
+
+ GEM_TRACE("%s\n", engine->name);
+
+ spin_lock_irqsave(&engine->timeline.lock, flags);
- execlists_init_reg_state(regs, rq->gem_context, engine, rq->ring);
- __execlists_update_reg_state(engine, rq->hw_context);
+ __execlists_reset(engine, stalled);
+
+ spin_unlock_irqrestore(&engine->timeline.lock, flags);
+}
+
+static void nop_submission_tasklet(unsigned long data)
+{
+ /* The driver is wedged; don't process any more events. */
+}
+
+static void execlists_cancel_requests(struct intel_engine_cs *engine)
+{
+ struct intel_engine_execlists * const execlists = &engine->execlists;
+ struct i915_request *rq, *rn;
+ struct rb_node *rb;
+ unsigned long flags;
+
+ GEM_TRACE("%s\n", engine->name);
+
+ /*
+ * Before we call engine->cancel_requests(), we should have exclusive
+ * access to the submission state. This is arranged for us by the
+ * caller disabling the interrupt generation, the tasklet and other
+ * threads that may then access the same state, giving us a free hand
+ * to reset state. However, we still need to let lockdep be aware that
+ * we know this state may be accessed in hardirq context, so we
+ * disable the irq around this manipulation and we want to keep
+ * the spinlock focused on its duties and not accidentally conflate
+ * coverage to the submission's irq state. (Similarly, although we
+ * shouldn't need to disable irq around the manipulation of the
+ * submission's irq state, we also wish to remind ourselves that
+ * it is irq state.)
+ */
+ spin_lock_irqsave(&engine->timeline.lock, flags);
+
+ __execlists_reset(engine, true);
+
+ /* Mark all executing requests as skipped. */
+ list_for_each_entry(rq, &engine->timeline.requests, link) {
+ if (!i915_request_signaled(rq))
+ dma_fence_set_error(&rq->fence, -EIO);
+
+ i915_request_mark_complete(rq);
+ }
+
+ /* Flush the queued requests to the timeline list (for retiring). */
+ while ((rb = rb_first_cached(&execlists->queue))) {
+ struct i915_priolist *p = to_priolist(rb);
+ int i;
+
+ priolist_for_each_request_consume(rq, rn, p, i) {
+ list_del_init(&rq->sched.link);
+ __i915_request_submit(rq);
+ dma_fence_set_error(&rq->fence, -EIO);
+ i915_request_mark_complete(rq);
+ }
+
+ rb_erase_cached(&p->node, &execlists->queue);
+ i915_priolist_free(p);
+ }
+
+ /* Remaining _unready_ requests will be nop'ed when submitted */
+
+ execlists->queue_priority_hint = INT_MIN;
+ execlists->queue = RB_ROOT_CACHED;
+ GEM_BUG_ON(port_isset(execlists->port));
+
+ GEM_BUG_ON(__tasklet_is_enabled(&execlists->tasklet));
+ execlists->tasklet.func = nop_submission_tasklet;
-out_unlock:
spin_unlock_irqrestore(&engine->timeline.lock, flags);
}
* After a GPU reset, we may have requests to replay. Do so now while
* we still have the forcewake to be sure that the GPU is not allowed
* to sleep before we restart and reload a context.
- *
*/
GEM_BUG_ON(!reset_in_progress(execlists));
if (!RB_EMPTY_ROOT(&execlists->queue.rb_root))
execlists->tasklet.func(execlists->tasklet.data);
- tasklet_enable(&execlists->tasklet);
+ if (__tasklet_enable(&execlists->tasklet))
+ /* And kick in case we missed a new request submission. */
+ tasklet_hi_schedule(&execlists->tasklet);
GEM_TRACE("%s: depth->%d\n", engine->name,
atomic_read(&execlists->tasklet.count));
}
{
u32 *cs;
- cs = intel_ring_begin(rq, 6);
+ cs = intel_ring_begin(rq, 4);
if (IS_ERR(cs))
return PTR_ERR(cs);
* particular all the gen that do not need the w/a at all!), if we
* took care to make sure that on every switch into this context
* (both ordinary and for preemption) that arbitrartion was enabled
- * we would be fine. However, there doesn't seem to be a downside to
- * being paranoid and making sure it is set before each batch and
- * every context-switch.
- *
- * Note that if we fail to enable arbitration before the request
- * is complete, then we do not see the context-switch interrupt and
- * the engine hangs (with RING_HEAD == RING_TAIL).
- *
- * That satisfies both the GPGPU w/a and our heavy-handed paranoia.
+ * we would be fine. However, for gen8 there is another w/a that
+ * requires us to not preempt inside GPGPU execution, so we keep
+ * arbitration disabled for gen8 batches. Arbitration will be
+ * re-enabled before we close the request
+ * (engine->emit_fini_breadcrumb).
*/
+ *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
+
+ /* FIXME(BDW+): Address space and security selectors. */
+ *cs++ = MI_BATCH_BUFFER_START_GEN8 |
+ (flags & I915_DISPATCH_SECURE ? 0 : BIT(8));
+ *cs++ = lower_32_bits(offset);
+ *cs++ = upper_32_bits(offset);
+
+ intel_ring_advance(rq, cs);
+
+ return 0;
+}
+
+static int gen9_emit_bb_start(struct i915_request *rq,
+ u64 offset, u32 len,
+ const unsigned int flags)
+{
+ u32 *cs;
+
+ cs = intel_ring_begin(rq, 6);
+ if (IS_ERR(cs))
+ return PTR_ERR(cs);
+
*cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
- /* FIXME(BDW): Address space and security selectors. */
*cs++ = MI_BATCH_BUFFER_START_GEN8 |
(flags & I915_DISPATCH_SECURE ? 0 : BIT(8));
*cs++ = lower_32_bits(offset);
static void gen8_logical_ring_enable_irq(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
- I915_WRITE_IMR(engine,
- ~(engine->irq_enable_mask | engine->irq_keep_mask));
- POSTING_READ_FW(RING_IMR(engine->mmio_base));
+ ENGINE_WRITE(engine, RING_IMR,
+ ~(engine->irq_enable_mask | engine->irq_keep_mask));
+ ENGINE_POSTING_READ(engine, RING_IMR);
}
static void gen8_logical_ring_disable_irq(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
- I915_WRITE_IMR(engine, ~engine->irq_keep_mask);
+ ENGINE_WRITE(engine, RING_IMR, ~engine->irq_keep_mask);
}
static int gen8_emit_flush(struct i915_request *request, u32 mode)
static u32 *gen8_emit_fini_breadcrumb(struct i915_request *request, u32 *cs)
{
- /* w/a: bit 5 needs to be zero for MI_FLUSH_DW address. */
- BUILD_BUG_ON(I915_GEM_HWS_INDEX_ADDR & (1 << 5));
-
cs = gen8_emit_ggtt_write(cs,
request->fence.seqno,
- request->timeline->hwsp_offset);
+ request->timeline->hwsp_offset,
+ 0);
cs = gen8_emit_ggtt_write(cs,
- request->global_seqno,
- intel_hws_seqno_address(request->engine));
+ intel_engine_next_hangcheck_seqno(request->engine),
+ I915_GEM_HWS_HANGCHECK_ADDR,
+ MI_FLUSH_DW_STORE_INDEX);
+
*cs++ = MI_USER_INTERRUPT;
*cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
PIPE_CONTROL_CS_STALL);
cs = gen8_emit_ggtt_write_rcs(cs,
- request->global_seqno,
- intel_hws_seqno_address(request->engine),
- PIPE_CONTROL_CS_STALL);
+ intel_engine_next_hangcheck_seqno(request->engine),
+ I915_GEM_HWS_HANGCHECK_ADDR,
+ PIPE_CONTROL_STORE_DATA_INDEX);
*cs++ = MI_USER_INTERRUPT;
*cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
dev_priv = engine->i915;
if (engine->buffer) {
- WARN_ON((I915_READ_MODE(engine) & MODE_IDLE) == 0);
+ WARN_ON((ENGINE_READ(engine, RING_MI_MODE) & MODE_IDLE) == 0);
}
if (engine->cleanup)
engine->execlists.tasklet.func = execlists_submission_tasklet;
engine->reset.prepare = execlists_reset_prepare;
+ engine->reset.reset = execlists_reset;
+ engine->reset.finish = execlists_reset_finish;
engine->park = NULL;
engine->unpark = NULL;
engine->flags |= I915_ENGINE_SUPPORTS_STATS;
- if (engine->i915->preempt_context)
+ if (!intel_vgpu_active(engine->i915))
+ engine->flags |= I915_ENGINE_HAS_SEMAPHORES;
+ if (engine->preempt_context &&
+ HAS_LOGICAL_RING_PREEMPTION(engine->i915))
engine->flags |= I915_ENGINE_HAS_PREEMPTION;
-
- engine->i915->caps.scheduler =
- I915_SCHEDULER_CAP_ENABLED |
- I915_SCHEDULER_CAP_PRIORITY;
- if (intel_engine_has_preemption(engine))
- engine->i915->caps.scheduler |= I915_SCHEDULER_CAP_PREEMPTION;
}
static void
engine->reset.reset = execlists_reset;
engine->reset.finish = execlists_reset_finish;
- engine->context_pin = execlists_context_pin;
+ engine->cops = &execlists_context_ops;
engine->request_alloc = execlists_request_alloc;
engine->emit_flush = gen8_emit_flush;
* until a more refined solution exists.
*/
}
- engine->emit_bb_start = gen8_emit_bb_start;
+ if (IS_GEN(engine->i915, 8))
+ engine->emit_bb_start = gen8_emit_bb_start;
+ else
+ engine->emit_bb_start = gen9_emit_bb_start;
}
static inline void
if (INTEL_GEN(engine->i915) < 11) {
const u8 irq_shifts[] = {
- [RCS] = GEN8_RCS_IRQ_SHIFT,
- [BCS] = GEN8_BCS_IRQ_SHIFT,
- [VCS] = GEN8_VCS1_IRQ_SHIFT,
- [VCS2] = GEN8_VCS2_IRQ_SHIFT,
- [VECS] = GEN8_VECS_IRQ_SHIFT,
+ [RCS0] = GEN8_RCS_IRQ_SHIFT,
+ [BCS0] = GEN8_BCS_IRQ_SHIFT,
+ [VCS0] = GEN8_VCS0_IRQ_SHIFT,
+ [VCS1] = GEN8_VCS1_IRQ_SHIFT,
+ [VECS0] = GEN8_VECS_IRQ_SHIFT,
};
shift = irq_shifts[engine->id];
{
struct drm_i915_private *i915 = engine->i915;
struct intel_engine_execlists * const execlists = &engine->execlists;
+ u32 base = engine->mmio_base;
int ret;
ret = intel_engine_init_common(engine);
intel_engine_init_workarounds(engine);
if (HAS_LOGICAL_RING_ELSQ(i915)) {
- execlists->submit_reg = i915->regs +
- i915_mmio_reg_offset(RING_EXECLIST_SQ_CONTENTS(engine));
- execlists->ctrl_reg = i915->regs +
- i915_mmio_reg_offset(RING_EXECLIST_CONTROL(engine));
+ execlists->submit_reg = i915->uncore.regs +
+ i915_mmio_reg_offset(RING_EXECLIST_SQ_CONTENTS(base));
+ execlists->ctrl_reg = i915->uncore.regs +
+ i915_mmio_reg_offset(RING_EXECLIST_CONTROL(base));
} else {
- execlists->submit_reg = i915->regs +
- i915_mmio_reg_offset(RING_ELSP(engine));
+ execlists->submit_reg = i915->uncore.regs +
+ i915_mmio_reg_offset(RING_ELSP(base));
}
execlists->preempt_complete_status = ~0u;
- if (i915->preempt_context) {
- struct intel_context *ce =
- to_intel_context(i915->preempt_context, engine);
-
+ if (engine->preempt_context)
execlists->preempt_complete_status =
- upper_32_bits(ce->lrc_desc);
- }
+ upper_32_bits(engine->preempt_context->lrc_desc);
execlists->csb_status =
&engine->status_page.addr[I915_HWS_CSB_BUF0_INDEX];
execlists->csb_write =
&engine->status_page.addr[intel_hws_csb_write_index(i915)];
+ if (INTEL_GEN(engine->i915) < 11)
+ execlists->csb_size = GEN8_CSB_ENTRIES;
+ else
+ execlists->csb_size = GEN11_CSB_ENTRIES;
+
reset_csb_pointers(execlists);
return 0;
}
static void execlists_init_reg_state(u32 *regs,
- struct i915_gem_context *ctx,
+ struct intel_context *ce,
struct intel_engine_cs *engine,
struct intel_ring *ring)
{
- struct drm_i915_private *dev_priv = engine->i915;
- u32 base = engine->mmio_base;
+ struct i915_hw_ppgtt *ppgtt = ce->gem_context->ppgtt;
bool rcs = engine->class == RENDER_CLASS;
+ u32 base = engine->mmio_base;
/* A context is actually a big batch buffer with several
* MI_LOAD_REGISTER_IMM commands followed by (reg, value) pairs. The
regs[CTX_LRI_HEADER_0] = MI_LOAD_REGISTER_IMM(rcs ? 14 : 11) |
MI_LRI_FORCE_POSTED;
- CTX_REG(regs, CTX_CONTEXT_CONTROL, RING_CONTEXT_CONTROL(engine),
+ CTX_REG(regs, CTX_CONTEXT_CONTROL, RING_CONTEXT_CONTROL(base),
_MASKED_BIT_DISABLE(CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT) |
_MASKED_BIT_ENABLE(CTX_CTRL_INHIBIT_SYN_CTX_SWITCH));
- if (INTEL_GEN(dev_priv) < 11) {
+ if (INTEL_GEN(engine->i915) < 11) {
regs[CTX_CONTEXT_CONTROL + 1] |=
_MASKED_BIT_DISABLE(CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT |
CTX_CTRL_RS_CTX_ENABLE);
CTX_REG(regs, CTX_CTX_TIMESTAMP, RING_CTX_TIMESTAMP(base), 0);
/* PDP values well be assigned later if needed */
- CTX_REG(regs, CTX_PDP3_UDW, GEN8_RING_PDP_UDW(engine, 3), 0);
- CTX_REG(regs, CTX_PDP3_LDW, GEN8_RING_PDP_LDW(engine, 3), 0);
- CTX_REG(regs, CTX_PDP2_UDW, GEN8_RING_PDP_UDW(engine, 2), 0);
- CTX_REG(regs, CTX_PDP2_LDW, GEN8_RING_PDP_LDW(engine, 2), 0);
- CTX_REG(regs, CTX_PDP1_UDW, GEN8_RING_PDP_UDW(engine, 1), 0);
- CTX_REG(regs, CTX_PDP1_LDW, GEN8_RING_PDP_LDW(engine, 1), 0);
- CTX_REG(regs, CTX_PDP0_UDW, GEN8_RING_PDP_UDW(engine, 0), 0);
- CTX_REG(regs, CTX_PDP0_LDW, GEN8_RING_PDP_LDW(engine, 0), 0);
-
- if (i915_vm_is_48bit(&ctx->ppgtt->vm)) {
+ CTX_REG(regs, CTX_PDP3_UDW, GEN8_RING_PDP_UDW(base, 3), 0);
+ CTX_REG(regs, CTX_PDP3_LDW, GEN8_RING_PDP_LDW(base, 3), 0);
+ CTX_REG(regs, CTX_PDP2_UDW, GEN8_RING_PDP_UDW(base, 2), 0);
+ CTX_REG(regs, CTX_PDP2_LDW, GEN8_RING_PDP_LDW(base, 2), 0);
+ CTX_REG(regs, CTX_PDP1_UDW, GEN8_RING_PDP_UDW(base, 1), 0);
+ CTX_REG(regs, CTX_PDP1_LDW, GEN8_RING_PDP_LDW(base, 1), 0);
+ CTX_REG(regs, CTX_PDP0_UDW, GEN8_RING_PDP_UDW(base, 0), 0);
+ CTX_REG(regs, CTX_PDP0_LDW, GEN8_RING_PDP_LDW(base, 0), 0);
+
+ if (i915_vm_is_4lvl(&ppgtt->vm)) {
/* 64b PPGTT (48bit canonical)
* PDP0_DESCRIPTOR contains the base address to PML4 and
* other PDP Descriptors are ignored.
*/
- ASSIGN_CTX_PML4(ctx->ppgtt, regs);
+ ASSIGN_CTX_PML4(ppgtt, regs);
} else {
- ASSIGN_CTX_PDP(ctx->ppgtt, regs, 3);
- ASSIGN_CTX_PDP(ctx->ppgtt, regs, 2);
- ASSIGN_CTX_PDP(ctx->ppgtt, regs, 1);
- ASSIGN_CTX_PDP(ctx->ppgtt, regs, 0);
+ ASSIGN_CTX_PDP(ppgtt, regs, 3);
+ ASSIGN_CTX_PDP(ppgtt, regs, 2);
+ ASSIGN_CTX_PDP(ppgtt, regs, 1);
+ ASSIGN_CTX_PDP(ppgtt, regs, 0);
}
if (rcs) {
regs[CTX_LRI_HEADER_2] = MI_LOAD_REGISTER_IMM(1);
CTX_REG(regs, CTX_R_PWR_CLK_STATE, GEN8_R_PWR_CLK_STATE, 0);
- i915_oa_init_reg_state(engine, ctx, regs);
+ i915_oa_init_reg_state(engine, ce, regs);
}
regs[CTX_END] = MI_BATCH_BUFFER_END;
- if (INTEL_GEN(dev_priv) >= 10)
+ if (INTEL_GEN(engine->i915) >= 10)
regs[CTX_END] |= BIT(0);
}
static int
-populate_lr_context(struct i915_gem_context *ctx,
+populate_lr_context(struct intel_context *ce,
struct drm_i915_gem_object *ctx_obj,
struct intel_engine_cs *engine,
struct intel_ring *ring)
u32 *regs;
int ret;
- ret = i915_gem_object_set_to_cpu_domain(ctx_obj, true);
- if (ret) {
- DRM_DEBUG_DRIVER("Could not set to CPU domain\n");
- return ret;
- }
-
vaddr = i915_gem_object_pin_map(ctx_obj, I915_MAP_WB);
if (IS_ERR(vaddr)) {
ret = PTR_ERR(vaddr);
DRM_DEBUG_DRIVER("Could not map object pages! (%d)\n", ret);
return ret;
}
- ctx_obj->mm.dirty = true;
if (engine->default_state) {
/*
/* The second page of the context object contains some fields which must
* be set up prior to the first execution. */
regs = vaddr + LRC_STATE_PN * PAGE_SIZE;
- execlists_init_reg_state(regs, ctx, engine, ring);
+ execlists_init_reg_state(regs, ce, engine, ring);
if (!engine->default_state)
regs[CTX_CONTEXT_CONTROL + 1] |=
_MASKED_BIT_ENABLE(CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT);
- if (ctx == ctx->i915->preempt_context && INTEL_GEN(engine->i915) < 11)
+ if (ce->gem_context == engine->i915->preempt_context &&
+ INTEL_GEN(engine->i915) < 11)
regs[CTX_CONTEXT_CONTROL + 1] |=
_MASKED_BIT_ENABLE(CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT |
CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT);
+ ret = 0;
err_unpin_ctx:
+ __i915_gem_object_flush_map(ctx_obj,
+ LRC_HEADER_PAGES * PAGE_SIZE,
+ engine->context_size);
i915_gem_object_unpin_map(ctx_obj);
return ret;
}
-static int execlists_context_deferred_alloc(struct i915_gem_context *ctx,
- struct intel_engine_cs *engine,
- struct intel_context *ce)
+static struct i915_timeline *get_timeline(struct i915_gem_context *ctx)
+{
+ if (ctx->timeline)
+ return i915_timeline_get(ctx->timeline);
+ else
+ return i915_timeline_create(ctx->i915, NULL);
+}
+
+static int execlists_context_deferred_alloc(struct intel_context *ce,
+ struct intel_engine_cs *engine)
{
struct drm_i915_gem_object *ctx_obj;
struct i915_vma *vma;
*/
context_size += LRC_HEADER_PAGES * PAGE_SIZE;
- ctx_obj = i915_gem_object_create(ctx->i915, context_size);
+ ctx_obj = i915_gem_object_create(engine->i915, context_size);
if (IS_ERR(ctx_obj))
return PTR_ERR(ctx_obj);
- vma = i915_vma_instance(ctx_obj, &ctx->i915->ggtt.vm, NULL);
+ vma = i915_vma_instance(ctx_obj, &engine->i915->ggtt.vm, NULL);
if (IS_ERR(vma)) {
ret = PTR_ERR(vma);
goto error_deref_obj;
}
- timeline = i915_timeline_create(ctx->i915, ctx->name, NULL);
+ timeline = get_timeline(ce->gem_context);
if (IS_ERR(timeline)) {
ret = PTR_ERR(timeline);
goto error_deref_obj;
}
- ring = intel_engine_create_ring(engine, timeline, ctx->ring_size);
+ ring = intel_engine_create_ring(engine,
+ timeline,
+ ce->gem_context->ring_size);
i915_timeline_put(timeline);
if (IS_ERR(ring)) {
ret = PTR_ERR(ring);
goto error_deref_obj;
}
- ret = populate_lr_context(ctx, ctx_obj, engine, ring);
+ ret = populate_lr_context(ce, ctx_obj, engine, ring);
if (ret) {
DRM_DEBUG_DRIVER("Failed to populate LRC: %d\n", ret);
goto error_ring_free;
return 0;
error_ring_free:
- intel_ring_free(ring);
+ intel_ring_put(ring);
error_deref_obj:
i915_gem_object_put(ctx_obj);
return ret;
}
-void intel_lr_context_resume(struct drm_i915_private *i915)
-{
- struct intel_engine_cs *engine;
- struct i915_gem_context *ctx;
- enum intel_engine_id id;
-
- /*
- * Because we emit WA_TAIL_DWORDS there may be a disparity
- * between our bookkeeping in ce->ring->head and ce->ring->tail and
- * that stored in context. As we only write new commands from
- * ce->ring->tail onwards, everything before that is junk. If the GPU
- * starts reading from its RING_HEAD from the context, it may try to
- * execute that junk and die.
- *
- * So to avoid that we reset the context images upon resume. For
- * simplicity, we just zero everything out.
- */
- list_for_each_entry(ctx, &i915->contexts.list, link) {
- for_each_engine(engine, i915, id) {
- struct intel_context *ce =
- to_intel_context(ctx, engine);
-
- if (!ce->state)
- continue;
-
- intel_ring_reset(ce->ring, 0);
-
- if (ce->pin_count) /* otherwise done in context_pin */
- __execlists_update_reg_state(engine, ce);
- }
- }
-}
-
void intel_execlists_show_requests(struct intel_engine_cs *engine,
struct drm_printer *m,
void (*show_request)(struct drm_printer *m,
spin_unlock_irqrestore(&engine->timeline.lock, flags);
}
+void intel_lr_context_reset(struct intel_engine_cs *engine,
+ struct intel_context *ce,
+ u32 head,
+ bool scrub)
+{
+ /*
+ * We want a simple context + ring to execute the breadcrumb update.
+ * We cannot rely on the context being intact across the GPU hang,
+ * so clear it and rebuild just what we need for the breadcrumb.
+ * All pending requests for this context will be zapped, and any
+ * future request will be after userspace has had the opportunity
+ * to recreate its own state.
+ */
+ if (scrub) {
+ u32 *regs = ce->lrc_reg_state;
+
+ if (engine->pinned_default_state) {
+ memcpy(regs, /* skip restoring the vanilla PPHWSP */
+ engine->pinned_default_state + LRC_STATE_PN * PAGE_SIZE,
+ engine->context_size - PAGE_SIZE);
+ }
+ execlists_init_reg_state(regs, ce, engine, ce->ring);
+ }
+
+ /* Rerun the request; its payload has been neutered (if guilty). */
+ ce->ring->head = head;
+ intel_ring_update_space(ce->ring);
+
+ __execlists_update_reg_state(ce, engine);
+}
+
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftests/intel_lrc.c"
#endif
#include "i915_gem_context.h"
/* Execlists regs */
-#define RING_ELSP(engine) _MMIO((engine)->mmio_base + 0x230)
-#define RING_EXECLIST_STATUS_LO(engine) _MMIO((engine)->mmio_base + 0x234)
-#define RING_EXECLIST_STATUS_HI(engine) _MMIO((engine)->mmio_base + 0x234 + 4)
-#define RING_CONTEXT_CONTROL(engine) _MMIO((engine)->mmio_base + 0x244)
+#define RING_ELSP(base) _MMIO((base) + 0x230)
+#define RING_EXECLIST_STATUS_LO(base) _MMIO((base) + 0x234)
+#define RING_EXECLIST_STATUS_HI(base) _MMIO((base) + 0x234 + 4)
+#define RING_CONTEXT_CONTROL(base) _MMIO((base) + 0x244)
#define CTX_CTRL_INHIBIT_SYN_CTX_SWITCH (1 << 3)
#define CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT (1 << 0)
-#define CTX_CTRL_RS_CTX_ENABLE (1 << 1)
+#define CTX_CTRL_RS_CTX_ENABLE (1 << 1)
#define CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT (1 << 2)
-#define RING_CONTEXT_STATUS_BUF_BASE(engine) _MMIO((engine)->mmio_base + 0x370)
-#define RING_CONTEXT_STATUS_BUF_LO(engine, i) _MMIO((engine)->mmio_base + 0x370 + (i) * 8)
-#define RING_CONTEXT_STATUS_BUF_HI(engine, i) _MMIO((engine)->mmio_base + 0x370 + (i) * 8 + 4)
-#define RING_CONTEXT_STATUS_PTR(engine) _MMIO((engine)->mmio_base + 0x3a0)
-#define RING_EXECLIST_SQ_CONTENTS(engine) _MMIO((engine)->mmio_base + 0x510)
-#define RING_EXECLIST_CONTROL(engine) _MMIO((engine)->mmio_base + 0x550)
+#define RING_CONTEXT_STATUS_PTR(base) _MMIO((base) + 0x3a0)
+#define RING_EXECLIST_SQ_CONTENTS(base) _MMIO((base) + 0x510)
+#define RING_EXECLIST_CONTROL(base) _MMIO((base) + 0x550)
+
#define EL_CTRL_LOAD (1 << 0)
/* The docs specify that the write pointer wraps around after 5h, "After status
#define GEN8_CSB_PTR_MASK 0x7
#define GEN8_CSB_READ_PTR_MASK (GEN8_CSB_PTR_MASK << 8)
#define GEN8_CSB_WRITE_PTR_MASK (GEN8_CSB_PTR_MASK << 0)
-#define GEN8_CSB_WRITE_PTR(csb_status) \
- (((csb_status) & GEN8_CSB_WRITE_PTR_MASK) >> 0)
-#define GEN8_CSB_READ_PTR(csb_status) \
- (((csb_status) & GEN8_CSB_READ_PTR_MASK) >> 8)
+
+#define GEN11_CSB_ENTRIES 12
+#define GEN11_CSB_PTR_MASK 0xf
+#define GEN11_CSB_READ_PTR_MASK (GEN11_CSB_PTR_MASK << 8)
+#define GEN11_CSB_WRITE_PTR_MASK (GEN11_CSB_PTR_MASK << 0)
enum {
INTEL_CONTEXT_SCHEDULE_IN = 0,
struct drm_i915_private;
struct i915_gem_context;
-void intel_lr_context_resume(struct drm_i915_private *dev_priv);
void intel_execlists_set_default_submission(struct intel_engine_cs *engine);
+void intel_lr_context_reset(struct intel_engine_cs *engine,
+ struct intel_context *ce,
+ u32 head,
+ bool scrub);
+
void intel_execlists_show_requests(struct intel_engine_cs *engine,
struct drm_printer *m,
void (*show_request)(struct drm_printer *m,
*
*
*/
-#include <drm/drm_edid.h>
+
#include <drm/drm_atomic_helper.h>
#include <drm/drm_dp_dual_mode_helper.h>
+#include <drm/drm_edid.h>
+
+#include "intel_dp.h"
#include "intel_drv.h"
+#include "intel_lspcon.h"
/* LSPCON OUI Vendor ID(signatures) */
#define LSPCON_VENDOR_PARADE_OUI 0x001CF8
DRM_DEBUG_DRIVER("AVI infoframes updated successfully\n");
}
+void lspcon_read_infoframe(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ unsigned int type,
+ void *frame, ssize_t len)
+{
+ /* FIXME implement this */
+}
+
void lspcon_set_infoframes(struct intel_encoder *encoder,
bool enable,
const struct intel_crtc_state *crtc_state,
return;
}
+ /* FIXME precompute infoframes */
+
ret = drm_hdmi_avi_infoframe_from_display_mode(&frame.avi,
conn_state->connector,
adjusted_mode);
buf, ret);
}
-bool lspcon_infoframe_enabled(struct intel_encoder *encoder,
+u32 lspcon_infoframes_enabled(struct intel_encoder *encoder,
const struct intel_crtc_state *pipe_config)
{
+ /* FIXME actually read this from the hw */
return enc_to_intel_lspcon(&encoder->base)->active;
}
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_LSPCON_H__
+#define __INTEL_LSPCON_H__
+
+#include <linux/types.h>
+
+struct drm_connector;
+struct drm_connector_state;
+struct intel_crtc_state;
+struct intel_digital_port;
+struct intel_encoder;
+struct intel_lspcon;
+
+bool lspcon_init(struct intel_digital_port *intel_dig_port);
+void lspcon_resume(struct intel_lspcon *lspcon);
+void lspcon_wait_pcon_mode(struct intel_lspcon *lspcon);
+void lspcon_write_infoframe(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ unsigned int type,
+ const void *buf, ssize_t len);
+void lspcon_read_infoframe(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ unsigned int type,
+ void *frame, ssize_t len);
+void lspcon_set_infoframes(struct intel_encoder *encoder,
+ bool enable,
+ const struct intel_crtc_state *crtc_state,
+ const struct drm_connector_state *conn_state);
+u32 lspcon_infoframes_enabled(struct intel_encoder *encoder,
+ const struct intel_crtc_state *pipe_config);
+void lspcon_ycbcr420_config(struct drm_connector *connector,
+ struct intel_crtc_state *crtc_state);
+
+#endif /* __INTEL_LSPCON_H__ */
*/
#include <acpi/button.h>
+#include <linux/acpi.h>
#include <linux/dmi.h>
#include <linux/i2c.h>
#include <linux/slab.h>
#include <linux/vga_switcheroo.h>
+
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
#include <drm/drm_edid.h>
-#include "intel_drv.h"
#include <drm/i915_drm.h>
+
#include "i915_drv.h"
-#include <linux/acpi.h>
+#include "intel_connector.h"
+#include "intel_drv.h"
+#include "intel_lvds.h"
+#include "intel_panel.h"
/* Private structure for the integrated LVDS support */
struct intel_lvds_pps {
pps->powerdown_on_reset = I915_READ(PP_CONTROL(0)) & PANEL_POWER_RESET;
val = I915_READ(PP_ON_DELAYS(0));
- pps->port = (val & PANEL_PORT_SELECT_MASK) >>
- PANEL_PORT_SELECT_SHIFT;
- pps->t1_t2 = (val & PANEL_POWER_UP_DELAY_MASK) >>
- PANEL_POWER_UP_DELAY_SHIFT;
- pps->t5 = (val & PANEL_LIGHT_ON_DELAY_MASK) >>
- PANEL_LIGHT_ON_DELAY_SHIFT;
+ pps->port = REG_FIELD_GET(PANEL_PORT_SELECT_MASK, val);
+ pps->t1_t2 = REG_FIELD_GET(PANEL_POWER_UP_DELAY_MASK, val);
+ pps->t5 = REG_FIELD_GET(PANEL_LIGHT_ON_DELAY_MASK, val);
val = I915_READ(PP_OFF_DELAYS(0));
- pps->t3 = (val & PANEL_POWER_DOWN_DELAY_MASK) >>
- PANEL_POWER_DOWN_DELAY_SHIFT;
- pps->tx = (val & PANEL_LIGHT_OFF_DELAY_MASK) >>
- PANEL_LIGHT_OFF_DELAY_SHIFT;
+ pps->t3 = REG_FIELD_GET(PANEL_POWER_DOWN_DELAY_MASK, val);
+ pps->tx = REG_FIELD_GET(PANEL_LIGHT_OFF_DELAY_MASK, val);
val = I915_READ(PP_DIVISOR(0));
- pps->divider = (val & PP_REFERENCE_DIVIDER_MASK) >>
- PP_REFERENCE_DIVIDER_SHIFT;
- val = (val & PANEL_POWER_CYCLE_DELAY_MASK) >>
- PANEL_POWER_CYCLE_DELAY_SHIFT;
+ pps->divider = REG_FIELD_GET(PP_REFERENCE_DIVIDER_MASK, val);
+ val = REG_FIELD_GET(PANEL_POWER_CYCLE_DELAY_MASK, val);
/*
* Remove the BSpec specified +1 (100ms) offset that accounts for a
* too short power-cycle delay due to the asynchronous programming of
val |= PANEL_POWER_RESET;
I915_WRITE(PP_CONTROL(0), val);
- I915_WRITE(PP_ON_DELAYS(0), (pps->port << PANEL_PORT_SELECT_SHIFT) |
- (pps->t1_t2 << PANEL_POWER_UP_DELAY_SHIFT) |
- (pps->t5 << PANEL_LIGHT_ON_DELAY_SHIFT));
- I915_WRITE(PP_OFF_DELAYS(0), (pps->t3 << PANEL_POWER_DOWN_DELAY_SHIFT) |
- (pps->tx << PANEL_LIGHT_OFF_DELAY_SHIFT));
+ I915_WRITE(PP_ON_DELAYS(0),
+ REG_FIELD_PREP(PANEL_PORT_SELECT_MASK, pps->port) |
+ REG_FIELD_PREP(PANEL_POWER_UP_DELAY_MASK, pps->t1_t2) |
+ REG_FIELD_PREP(PANEL_LIGHT_ON_DELAY_MASK, pps->t5));
+
+ I915_WRITE(PP_OFF_DELAYS(0),
+ REG_FIELD_PREP(PANEL_POWER_DOWN_DELAY_MASK, pps->t3) |
+ REG_FIELD_PREP(PANEL_LIGHT_OFF_DELAY_MASK, pps->tx));
- val = pps->divider << PP_REFERENCE_DIVIDER_SHIFT;
- val |= (DIV_ROUND_UP(pps->t4, 1000) + 1) <<
- PANEL_POWER_CYCLE_DELAY_SHIFT;
- I915_WRITE(PP_DIVISOR(0), val);
+ I915_WRITE(PP_DIVISOR(0),
+ REG_FIELD_PREP(PP_REFERENCE_DIVIDER_MASK, pps->divider) |
+ REG_FIELD_PREP(PANEL_POWER_CYCLE_DELAY_MASK,
+ DIV_ROUND_UP(pps->t4, 1000) + 1));
}
static void intel_pre_enable_lvds(struct intel_encoder *encoder,
I915_WRITE(PP_CONTROL(0), I915_READ(PP_CONTROL(0)) | PANEL_POWER_ON);
POSTING_READ(lvds_encoder->reg);
- if (intel_wait_for_register(dev_priv, PP_STATUS(0), PP_ON, PP_ON, 5000))
+ if (intel_wait_for_register(&dev_priv->uncore,
+ PP_STATUS(0), PP_ON, PP_ON, 5000))
DRM_ERROR("timed out waiting for panel to power on\n");
intel_panel_enable_backlight(pipe_config, conn_state);
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
I915_WRITE(PP_CONTROL(0), I915_READ(PP_CONTROL(0)) & ~PANEL_POWER_ON);
- if (intel_wait_for_register(dev_priv, PP_STATUS(0), PP_ON, 0, 1000))
+ if (intel_wait_for_register(&dev_priv->uncore,
+ PP_STATUS(0), PP_ON, 0, 1000))
DRM_ERROR("timed out waiting for panel to power off\n");
I915_WRITE(lvds_encoder->reg, I915_READ(lvds_encoder->reg) & ~LVDS_PORT_EN);
{ } /* terminating entry */
};
-struct intel_encoder *intel_get_lvds_encoder(struct drm_device *dev)
+struct intel_encoder *intel_get_lvds_encoder(struct drm_i915_private *dev_priv)
{
- struct intel_encoder *intel_encoder;
+ struct intel_encoder *encoder;
- for_each_intel_encoder(dev, intel_encoder)
- if (intel_encoder->type == INTEL_OUTPUT_LVDS)
- return intel_encoder;
+ for_each_intel_encoder(&dev_priv->drm, encoder) {
+ if (encoder->type == INTEL_OUTPUT_LVDS)
+ return encoder;
+ }
return NULL;
}
-bool intel_is_dual_link_lvds(struct drm_device *dev)
+bool intel_is_dual_link_lvds(struct drm_i915_private *dev_priv)
{
- struct intel_encoder *encoder = intel_get_lvds_encoder(dev);
+ struct intel_encoder *encoder = intel_get_lvds_encoder(dev_priv);
return encoder && to_lvds_encoder(&encoder->base)->is_dual_link;
}
struct intel_connector *intel_connector;
struct drm_connector *connector;
struct drm_encoder *encoder;
- struct drm_display_mode *scan; /* *modes, *bios_mode; */
struct drm_display_mode *fixed_mode = NULL;
struct drm_display_mode *downclock_mode = NULL;
struct edid *edid;
}
intel_connector->edid = edid;
- list_for_each_entry(scan, &connector->probed_modes, head) {
- if (scan->type & DRM_MODE_TYPE_PREFERRED) {
- DRM_DEBUG_KMS("using preferred mode from EDID: ");
- drm_mode_debug_printmodeline(scan);
-
- fixed_mode = drm_mode_duplicate(dev, scan);
- if (fixed_mode)
- goto out;
- }
- }
+ fixed_mode = intel_panel_edid_fixed_mode(intel_connector);
+ if (fixed_mode)
+ goto out;
/* Failed to get EDID, what about VBT? */
- if (dev_priv->vbt.lfp_lvds_vbt_mode) {
- DRM_DEBUG_KMS("using mode from VBT: ");
- drm_mode_debug_printmodeline(dev_priv->vbt.lfp_lvds_vbt_mode);
-
- fixed_mode = drm_mode_duplicate(dev, dev_priv->vbt.lfp_lvds_vbt_mode);
- if (fixed_mode) {
- fixed_mode->type |= DRM_MODE_TYPE_PREFERRED;
- connector->display_info.width_mm = fixed_mode->width_mm;
- connector->display_info.height_mm = fixed_mode->height_mm;
- goto out;
- }
- }
+ fixed_mode = intel_panel_vbt_fixed_mode(intel_connector);
+ if (fixed_mode)
+ goto out;
/*
* If we didn't get EDID, try checking if the panel is already turned
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_LVDS_H__
+#define __INTEL_LVDS_H__
+
+#include <linux/types.h>
+
+#include "i915_reg.h"
+
+enum pipe;
+struct drm_i915_private;
+
+bool intel_lvds_port_enabled(struct drm_i915_private *dev_priv,
+ i915_reg_t lvds_reg, enum pipe *pipe);
+void intel_lvds_init(struct drm_i915_private *dev_priv);
+struct intel_encoder *intel_get_lvds_encoder(struct drm_i915_private *dev_priv);
+bool intel_is_dual_link_lvds(struct drm_i915_private *dev_priv);
+
+#endif /* __INTEL_LVDS_H__ */
{
bool result = false;
- if (IS_ICELAKE(dev_priv)) {
+ if (INTEL_GEN(dev_priv) >= 11) {
table->size = ARRAY_SIZE(icelake_mocs_table);
table->table = icelake_mocs_table;
table->n_entries = GEN11_NUM_MOCS_ENTRIES;
static i915_reg_t mocs_register(enum intel_engine_id engine_id, int index)
{
switch (engine_id) {
- case RCS:
+ case RCS0:
return GEN9_GFX_MOCS(index);
- case VCS:
+ case VCS0:
return GEN9_MFX0_MOCS(index);
- case BCS:
+ case BCS0:
return GEN9_BLT_MOCS(index);
- case VECS:
+ case VECS0:
return GEN9_VEBOX_MOCS(index);
- case VCS2:
+ case VCS1:
return GEN9_MFX1_MOCS(index);
- case VCS3:
+ case VCS2:
return GEN11_MFX2_MOCS(index);
default:
MISSING_CASE(engine_id);
#include <drm/i915_drm.h>
-#include "intel_opregion.h"
#include "i915_drv.h"
#include "intel_drv.h"
+#include "intel_opregion.h"
+#include "intel_panel.h"
#define OPREGION_HEADER_OFFSET 0
#define OPREGION_ACPI_OFFSET 0x100
static struct i915_request *alloc_request(struct intel_overlay *overlay)
{
struct drm_i915_private *dev_priv = overlay->i915;
- struct intel_engine_cs *engine = dev_priv->engine[RCS];
+ struct intel_engine_cs *engine = dev_priv->engine[RCS0];
return i915_request_alloc(engine, dev_priv->kernel_context);
}
if (!overlay->old_vma)
return 0;
- if (I915_READ(ISR) & I915_OVERLAY_PLANE_FLIP_PENDING_INTERRUPT) {
+ if (I915_READ(GEN2_ISR) & I915_OVERLAY_PLANE_FLIP_PENDING_INTERRUPT) {
/* synchronous slowpath */
struct i915_request *rq;
return NULL;
error->dovsta = I915_READ(DOVSTA);
- error->isr = I915_READ(ISR);
+ error->isr = I915_READ(GEN2_ISR);
error->base = overlay->flip_addr;
memcpy_fromio(&error->regs, overlay->regs, sizeof(error->regs));
#include <linux/kernel.h>
#include <linux/moduleparam.h>
#include <linux/pwm.h>
+
+#include "intel_connector.h"
#include "intel_drv.h"
+#include "intel_panel.h"
#define CRC_PMIC_PWM_PERIOD_NS 21333
drm_mode_set_crtcinfo(adjusted_mode, 0);
}
-/**
- * intel_find_panel_downclock - find the reduced downclock for LVDS in EDID
- * @dev_priv: i915 device instance
- * @fixed_mode : panel native mode
- * @connector: LVDS/eDP connector
- *
- * Return downclock_avail
- * Find the reduced downclock for LVDS/eDP in EDID.
- */
-struct drm_display_mode *
-intel_find_panel_downclock(struct drm_i915_private *dev_priv,
- struct drm_display_mode *fixed_mode,
- struct drm_connector *connector)
+static bool is_downclock_mode(const struct drm_display_mode *downclock_mode,
+ const struct drm_display_mode *fixed_mode)
{
- struct drm_display_mode *scan, *tmp_mode;
- int temp_downclock;
+ return drm_mode_match(downclock_mode, fixed_mode,
+ DRM_MODE_MATCH_TIMINGS |
+ DRM_MODE_MATCH_FLAGS |
+ DRM_MODE_MATCH_3D_FLAGS) &&
+ downclock_mode->clock < fixed_mode->clock;
+}
- temp_downclock = fixed_mode->clock;
- tmp_mode = NULL;
+struct drm_display_mode *
+intel_panel_edid_downclock_mode(struct intel_connector *connector,
+ const struct drm_display_mode *fixed_mode)
+{
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ const struct drm_display_mode *scan, *best_mode = NULL;
+ struct drm_display_mode *downclock_mode;
+ int best_clock = fixed_mode->clock;
- list_for_each_entry(scan, &connector->probed_modes, head) {
+ list_for_each_entry(scan, &connector->base.probed_modes, head) {
/*
* If one mode has the same resolution with the fixed_panel
* mode while they have the different refresh rate, it means
* case we can set the different FPx0/1 to dynamically select
* between low and high frequency.
*/
- if (scan->hdisplay == fixed_mode->hdisplay &&
- scan->hsync_start == fixed_mode->hsync_start &&
- scan->hsync_end == fixed_mode->hsync_end &&
- scan->htotal == fixed_mode->htotal &&
- scan->vdisplay == fixed_mode->vdisplay &&
- scan->vsync_start == fixed_mode->vsync_start &&
- scan->vsync_end == fixed_mode->vsync_end &&
- scan->vtotal == fixed_mode->vtotal) {
- if (scan->clock < temp_downclock) {
- /*
- * The downclock is already found. But we
- * expect to find the lower downclock.
- */
- temp_downclock = scan->clock;
- tmp_mode = scan;
- }
+ if (is_downclock_mode(scan, fixed_mode) &&
+ scan->clock < best_clock) {
+ /*
+ * The downclock is already found. But we
+ * expect to find the lower downclock.
+ */
+ best_clock = scan->clock;
+ best_mode = scan;
}
}
- if (temp_downclock < fixed_mode->clock)
- return drm_mode_duplicate(&dev_priv->drm, tmp_mode);
- else
+ if (!best_mode)
+ return NULL;
+
+ downclock_mode = drm_mode_duplicate(&dev_priv->drm, best_mode);
+ if (!downclock_mode)
+ return NULL;
+
+ DRM_DEBUG_KMS("[CONNECTOR:%d:%s] using downclock mode from EDID: ",
+ connector->base.base.id, connector->base.name);
+ drm_mode_debug_printmodeline(downclock_mode);
+
+ return downclock_mode;
+}
+
+struct drm_display_mode *
+intel_panel_edid_fixed_mode(struct intel_connector *connector)
+{
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ const struct drm_display_mode *scan;
+ struct drm_display_mode *fixed_mode;
+
+ if (list_empty(&connector->base.probed_modes))
+ return NULL;
+
+ /* prefer fixed mode from EDID if available */
+ list_for_each_entry(scan, &connector->base.probed_modes, head) {
+ if ((scan->type & DRM_MODE_TYPE_PREFERRED) == 0)
+ continue;
+
+ fixed_mode = drm_mode_duplicate(&dev_priv->drm, scan);
+ if (!fixed_mode)
+ return NULL;
+
+ DRM_DEBUG_KMS("[CONNECTOR:%d:%s] using preferred mode from EDID: ",
+ connector->base.base.id, connector->base.name);
+ drm_mode_debug_printmodeline(fixed_mode);
+
+ return fixed_mode;
+ }
+
+ scan = list_first_entry(&connector->base.probed_modes,
+ typeof(*scan), head);
+
+ fixed_mode = drm_mode_duplicate(&dev_priv->drm, scan);
+ if (!fixed_mode)
return NULL;
+
+ fixed_mode->type |= DRM_MODE_TYPE_PREFERRED;
+
+ DRM_DEBUG_KMS("[CONNECTOR:%d:%s] using first mode from EDID: ",
+ connector->base.base.id, connector->base.name);
+ drm_mode_debug_printmodeline(fixed_mode);
+
+ return fixed_mode;
+}
+
+struct drm_display_mode *
+intel_panel_vbt_fixed_mode(struct intel_connector *connector)
+{
+ struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
+ struct drm_display_info *info = &connector->base.display_info;
+ struct drm_display_mode *fixed_mode;
+
+ if (!dev_priv->vbt.lfp_lvds_vbt_mode)
+ return NULL;
+
+ fixed_mode = drm_mode_duplicate(&dev_priv->drm,
+ dev_priv->vbt.lfp_lvds_vbt_mode);
+ if (!fixed_mode)
+ return NULL;
+
+ fixed_mode->type |= DRM_MODE_TYPE_PREFERRED;
+
+ DRM_DEBUG_KMS("[CONNECTOR:%d:%s] using mode from VBT: ",
+ connector->base.base.id, connector->base.name);
+ drm_mode_debug_printmodeline(fixed_mode);
+
+ info->width_mm = fixed_mode->width_mm;
+ info->height_mm = fixed_mode->height_mm;
+
+ return fixed_mode;
}
/* adjusted_mode has been preset to be the panel's fixed mode */
panel->backlight.set = bxt_set_backlight;
panel->backlight.get = bxt_get_backlight;
panel->backlight.hz_to_pwm = bxt_hz_to_pwm;
- } else if (HAS_PCH_CNP(dev_priv) || HAS_PCH_ICP(dev_priv)) {
+ } else if (INTEL_PCH_TYPE(dev_priv) >= PCH_CNP) {
panel->backlight.setup = cnp_setup_backlight;
panel->backlight.enable = cnp_enable_backlight;
panel->backlight.disable = cnp_disable_backlight;
panel->backlight.set = bxt_set_backlight;
panel->backlight.get = bxt_get_backlight;
panel->backlight.hz_to_pwm = cnp_hz_to_pwm;
- } else if (HAS_PCH_LPT(dev_priv) || HAS_PCH_SPT(dev_priv) ||
- HAS_PCH_KBP(dev_priv)) {
+ } else if (INTEL_PCH_TYPE(dev_priv) >= PCH_LPT) {
panel->backlight.setup = lpt_setup_backlight;
panel->backlight.enable = lpt_enable_backlight;
panel->backlight.disable = lpt_disable_backlight;
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_PANEL_H__
+#define __INTEL_PANEL_H__
+
+#include <linux/types.h>
+
+#include "intel_display.h"
+
+struct drm_connector;
+struct drm_connector_state;
+struct drm_display_mode;
+struct intel_connector;
+struct intel_crtc;
+struct intel_crtc_state;
+struct intel_encoder;
+struct intel_panel;
+
+int intel_panel_init(struct intel_panel *panel,
+ struct drm_display_mode *fixed_mode,
+ struct drm_display_mode *downclock_mode);
+void intel_panel_fini(struct intel_panel *panel);
+void intel_fixed_panel_mode(const struct drm_display_mode *fixed_mode,
+ struct drm_display_mode *adjusted_mode);
+void intel_pch_panel_fitting(struct intel_crtc *crtc,
+ struct intel_crtc_state *pipe_config,
+ int fitting_mode);
+void intel_gmch_panel_fitting(struct intel_crtc *crtc,
+ struct intel_crtc_state *pipe_config,
+ int fitting_mode);
+void intel_panel_set_backlight_acpi(const struct drm_connector_state *conn_state,
+ u32 level, u32 max);
+int intel_panel_setup_backlight(struct drm_connector *connector,
+ enum pipe pipe);
+void intel_panel_enable_backlight(const struct intel_crtc_state *crtc_state,
+ const struct drm_connector_state *conn_state);
+void intel_panel_update_backlight(struct intel_encoder *encoder,
+ const struct intel_crtc_state *crtc_state,
+ const struct drm_connector_state *conn_state);
+void intel_panel_disable_backlight(const struct drm_connector_state *old_conn_state);
+struct drm_display_mode *
+intel_panel_edid_downclock_mode(struct intel_connector *connector,
+ const struct drm_display_mode *fixed_mode);
+struct drm_display_mode *
+intel_panel_edid_fixed_mode(struct intel_connector *connector);
+struct drm_display_mode *
+intel_panel_vbt_fixed_mode(struct intel_connector *connector);
+
+#if IS_ENABLED(CONFIG_BACKLIGHT_CLASS_DEVICE)
+int intel_backlight_device_register(struct intel_connector *connector);
+void intel_backlight_device_unregister(struct intel_connector *connector);
+#else /* CONFIG_BACKLIGHT_CLASS_DEVICE */
+static inline int intel_backlight_device_register(struct intel_connector *connector)
+{
+ return 0;
+}
+static inline void intel_backlight_device_unregister(struct intel_connector *connector)
+{
+}
+#endif /* CONFIG_BACKLIGHT_CLASS_DEVICE */
+
+#endif /* __INTEL_PANEL_H__ */
*
*/
-#include <linux/seq_file.h>
#include <linux/circ_buf.h>
#include <linux/ctype.h>
#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+
#include "intel_drv.h"
+#include "intel_pipe_crc.h"
static const char * const pipe_crc_sources[] = {
- "none",
- "plane1",
- "plane2",
- "pf",
- "pipe",
- "TV",
- "DP-B",
- "DP-C",
- "DP-D",
- "auto",
+ [INTEL_PIPE_CRC_SOURCE_NONE] = "none",
+ [INTEL_PIPE_CRC_SOURCE_PLANE1] = "plane1",
+ [INTEL_PIPE_CRC_SOURCE_PLANE2] = "plane2",
+ [INTEL_PIPE_CRC_SOURCE_PLANE3] = "plane3",
+ [INTEL_PIPE_CRC_SOURCE_PLANE4] = "plane4",
+ [INTEL_PIPE_CRC_SOURCE_PLANE5] = "plane5",
+ [INTEL_PIPE_CRC_SOURCE_PLANE6] = "plane6",
+ [INTEL_PIPE_CRC_SOURCE_PLANE7] = "plane7",
+ [INTEL_PIPE_CRC_SOURCE_PIPE] = "pipe",
+ [INTEL_PIPE_CRC_SOURCE_TV] = "TV",
+ [INTEL_PIPE_CRC_SOURCE_DP_B] = "DP-B",
+ [INTEL_PIPE_CRC_SOURCE_DP_C] = "DP-C",
+ [INTEL_PIPE_CRC_SOURCE_DP_D] = "DP-D",
+ [INTEL_PIPE_CRC_SOURCE_AUTO] = "auto",
};
static int i8xx_pipe_crc_ctl_reg(enum intel_pipe_crc_source *source,
enum intel_pipe_crc_source *source,
u32 *val)
{
- bool need_stable_symbols = false;
-
if (*source == INTEL_PIPE_CRC_SOURCE_AUTO) {
int ret = i9xx_pipe_crc_auto_source(dev_priv, pipe, source);
if (ret)
return -EINVAL;
*val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_TV_PRE;
break;
- case INTEL_PIPE_CRC_SOURCE_DP_B:
- if (!IS_G4X(dev_priv))
- return -EINVAL;
- *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_DP_B_G4X;
- need_stable_symbols = true;
- break;
- case INTEL_PIPE_CRC_SOURCE_DP_C:
- if (!IS_G4X(dev_priv))
- return -EINVAL;
- *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_DP_C_G4X;
- need_stable_symbols = true;
- break;
- case INTEL_PIPE_CRC_SOURCE_DP_D:
- if (!IS_G4X(dev_priv))
- return -EINVAL;
- *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_DP_D_G4X;
- need_stable_symbols = true;
- break;
case INTEL_PIPE_CRC_SOURCE_NONE:
*val = 0;
break;
default:
+ /*
+ * The DP CRC source doesn't work on g4x.
+ * It can be made to work to some degree by selecting
+ * the correct CRC source before the port is enabled,
+ * and not touching the CRC source bits again until
+ * the port is disabled. But even then the bits
+ * eventually get stuck and a reboot is needed to get
+ * working CRCs on the pipe again. Let's simply
+ * refuse to use DP CRCs on g4x.
+ */
return -EINVAL;
}
- /*
- * When the pipe CRC tap point is after the transcoders we need
- * to tweak symbol-level features to produce a deterministic series of
- * symbols for a given frame. We need to reset those features only once
- * a frame (instead of every nth symbol):
- * - DC-balance: used to ensure a better clock recovery from the data
- * link (SDVO)
- * - DisplayPort scrambling: used for EMI reduction
- */
- if (need_stable_symbols) {
- u32 tmp = I915_READ(PORT_DFT2_G4X);
-
- WARN_ON(!IS_G4X(dev_priv));
-
- I915_WRITE(PORT_DFT_I9XX,
- I915_READ(PORT_DFT_I9XX) | DC_BALANCE_RESET);
-
- if (pipe == PIPE_A)
- tmp |= PIPE_A_SCRAMBLE_RESET;
- else
- tmp |= PIPE_B_SCRAMBLE_RESET;
-
- I915_WRITE(PORT_DFT2_G4X, tmp);
- }
-
return 0;
}
if (!(tmp & PIPE_SCRAMBLE_RESET_MASK))
tmp &= ~DC_BALANCE_RESET_VLV;
I915_WRITE(PORT_DFT2_G4X, tmp);
-
-}
-
-static void g4x_undo_pipe_scramble_reset(struct drm_i915_private *dev_priv,
- enum pipe pipe)
-{
- u32 tmp = I915_READ(PORT_DFT2_G4X);
-
- if (pipe == PIPE_A)
- tmp &= ~PIPE_A_SCRAMBLE_RESET;
- else
- tmp &= ~PIPE_B_SCRAMBLE_RESET;
- I915_WRITE(PORT_DFT2_G4X, tmp);
-
- if (!(tmp & PIPE_SCRAMBLE_RESET_MASK)) {
- I915_WRITE(PORT_DFT_I9XX,
- I915_READ(PORT_DFT_I9XX) & ~DC_BALANCE_RESET);
- }
}
static int ilk_pipe_crc_ctl_reg(enum intel_pipe_crc_source *source,
return 0;
}
-static void hsw_pipe_A_crc_wa(struct drm_i915_private *dev_priv,
- bool enable)
+static void
+intel_crtc_crc_setup_workarounds(struct intel_crtc *crtc, bool enable)
{
- struct drm_device *dev = &dev_priv->drm;
- struct intel_crtc *crtc = intel_get_crtc_for_pipe(dev_priv, PIPE_A);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
struct intel_crtc_state *pipe_config;
struct drm_atomic_state *state;
struct drm_modeset_acquire_ctx ctx;
- int ret = 0;
+ int ret;
drm_modeset_acquire_init(&ctx, 0);
- state = drm_atomic_state_alloc(dev);
+ state = drm_atomic_state_alloc(&dev_priv->drm);
if (!state) {
ret = -ENOMEM;
goto unlock;
goto put_state;
}
- if (HAS_IPS(dev_priv)) {
- /*
- * When IPS gets enabled, the pipe CRC changes. Since IPS gets
- * enabled and disabled dynamically based on package C states,
- * user space can't make reliable use of the CRCs, so let's just
- * completely disable it.
- */
- pipe_config->ips_force_disable = enable;
- }
+ pipe_config->base.mode_changed = pipe_config->has_psr;
+ pipe_config->crc_enabled = enable;
- if (IS_HASWELL(dev_priv)) {
+ if (IS_HASWELL(dev_priv) && crtc->pipe == PIPE_A) {
pipe_config->pch_pfit.force_thru = enable;
if (pipe_config->cpu_transcoder == TRANSCODER_EDP &&
pipe_config->pch_pfit.enabled != enable)
static int ivb_pipe_crc_ctl_reg(struct drm_i915_private *dev_priv,
enum pipe pipe,
enum intel_pipe_crc_source *source,
- u32 *val,
- bool set_wa)
+ u32 *val)
{
if (*source == INTEL_PIPE_CRC_SOURCE_AUTO)
- *source = INTEL_PIPE_CRC_SOURCE_PF;
+ *source = INTEL_PIPE_CRC_SOURCE_PIPE;
switch (*source) {
case INTEL_PIPE_CRC_SOURCE_PLANE1:
case INTEL_PIPE_CRC_SOURCE_PLANE2:
*val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_SPRITE_IVB;
break;
- case INTEL_PIPE_CRC_SOURCE_PF:
- if (set_wa && (IS_HASWELL(dev_priv) ||
- IS_BROADWELL(dev_priv)) && pipe == PIPE_A)
- hsw_pipe_A_crc_wa(dev_priv, true);
-
+ case INTEL_PIPE_CRC_SOURCE_PIPE:
*val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_PF_IVB;
break;
case INTEL_PIPE_CRC_SOURCE_NONE:
return 0;
}
+static int skl_pipe_crc_ctl_reg(struct drm_i915_private *dev_priv,
+ enum pipe pipe,
+ enum intel_pipe_crc_source *source,
+ u32 *val)
+{
+ if (*source == INTEL_PIPE_CRC_SOURCE_AUTO)
+ *source = INTEL_PIPE_CRC_SOURCE_PIPE;
+
+ switch (*source) {
+ case INTEL_PIPE_CRC_SOURCE_PLANE1:
+ *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_PLANE_1_SKL;
+ break;
+ case INTEL_PIPE_CRC_SOURCE_PLANE2:
+ *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_PLANE_2_SKL;
+ break;
+ case INTEL_PIPE_CRC_SOURCE_PLANE3:
+ *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_PLANE_3_SKL;
+ break;
+ case INTEL_PIPE_CRC_SOURCE_PLANE4:
+ *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_PLANE_4_SKL;
+ break;
+ case INTEL_PIPE_CRC_SOURCE_PLANE5:
+ *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_PLANE_5_SKL;
+ break;
+ case INTEL_PIPE_CRC_SOURCE_PLANE6:
+ *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_PLANE_6_SKL;
+ break;
+ case INTEL_PIPE_CRC_SOURCE_PLANE7:
+ *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_PLANE_7_SKL;
+ break;
+ case INTEL_PIPE_CRC_SOURCE_PIPE:
+ *val = PIPE_CRC_ENABLE | PIPE_CRC_SOURCE_DMUX_SKL;
+ break;
+ case INTEL_PIPE_CRC_SOURCE_NONE:
+ *val = 0;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static int get_new_crc_ctl_reg(struct drm_i915_private *dev_priv,
enum pipe pipe,
- enum intel_pipe_crc_source *source, u32 *val,
- bool set_wa)
+ enum intel_pipe_crc_source *source, u32 *val)
{
if (IS_GEN(dev_priv, 2))
return i8xx_pipe_crc_ctl_reg(source, val);
return vlv_pipe_crc_ctl_reg(dev_priv, pipe, source, val);
else if (IS_GEN_RANGE(dev_priv, 5, 6))
return ilk_pipe_crc_ctl_reg(source, val);
+ else if (INTEL_GEN(dev_priv) < 9)
+ return ivb_pipe_crc_ctl_reg(dev_priv, pipe, source, val);
else
- return ivb_pipe_crc_ctl_reg(dev_priv, pipe, source, val, set_wa);
+ return skl_pipe_crc_ctl_reg(dev_priv, pipe, source, val);
}
static int
switch (source) {
case INTEL_PIPE_CRC_SOURCE_PIPE:
case INTEL_PIPE_CRC_SOURCE_TV:
- case INTEL_PIPE_CRC_SOURCE_DP_B:
- case INTEL_PIPE_CRC_SOURCE_DP_C:
- case INTEL_PIPE_CRC_SOURCE_DP_D:
case INTEL_PIPE_CRC_SOURCE_NONE:
return 0;
default:
case INTEL_PIPE_CRC_SOURCE_PIPE:
case INTEL_PIPE_CRC_SOURCE_PLANE1:
case INTEL_PIPE_CRC_SOURCE_PLANE2:
- case INTEL_PIPE_CRC_SOURCE_PF:
+ case INTEL_PIPE_CRC_SOURCE_NONE:
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int skl_crc_source_valid(struct drm_i915_private *dev_priv,
+ const enum intel_pipe_crc_source source)
+{
+ switch (source) {
+ case INTEL_PIPE_CRC_SOURCE_PIPE:
+ case INTEL_PIPE_CRC_SOURCE_PLANE1:
+ case INTEL_PIPE_CRC_SOURCE_PLANE2:
+ case INTEL_PIPE_CRC_SOURCE_PLANE3:
+ case INTEL_PIPE_CRC_SOURCE_PLANE4:
+ case INTEL_PIPE_CRC_SOURCE_PLANE5:
+ case INTEL_PIPE_CRC_SOURCE_PLANE6:
+ case INTEL_PIPE_CRC_SOURCE_PLANE7:
case INTEL_PIPE_CRC_SOURCE_NONE:
return 0;
default:
return vlv_crc_source_valid(dev_priv, source);
else if (IS_GEN_RANGE(dev_priv, 5, 6))
return ilk_crc_source_valid(dev_priv, source);
- else
+ else if (INTEL_GEN(dev_priv) < 9)
return ivb_crc_source_valid(dev_priv, source);
+ else
+ return skl_crc_source_valid(dev_priv, source);
}
const char *const *intel_crtc_get_crc_sources(struct drm_crtc *crtc,
intel_wakeref_t wakeref;
u32 val = 0; /* shut up gcc */
int ret = 0;
+ bool enable;
if (display_crc_ctl_parse_source(source_name, &source) < 0) {
DRM_DEBUG_DRIVER("unknown source %s\n", source_name);
return -EIO;
}
- ret = get_new_crc_ctl_reg(dev_priv, crtc->index, &source, &val, true);
+ enable = source != INTEL_PIPE_CRC_SOURCE_NONE;
+ if (enable)
+ intel_crtc_crc_setup_workarounds(to_intel_crtc(crtc), true);
+
+ ret = get_new_crc_ctl_reg(dev_priv, crtc->index, &source, &val);
if (ret != 0)
goto out;
POSTING_READ(PIPE_CRC_CTL(crtc->index));
if (!source) {
- if (IS_G4X(dev_priv))
- g4x_undo_pipe_scramble_reset(dev_priv, crtc->index);
- else if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
+ if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
vlv_undo_pipe_scramble_reset(dev_priv, crtc->index);
- else if ((IS_HASWELL(dev_priv) ||
- IS_BROADWELL(dev_priv)) && crtc->index == PIPE_A)
- hsw_pipe_A_crc_wa(dev_priv, false);
}
pipe_crc->skipped = 0;
out:
+ if (!enable)
+ intel_crtc_crc_setup_workarounds(to_intel_crtc(crtc), false);
+
intel_display_power_put(dev_priv, power_domain, wakeref);
return ret;
if (!crtc->crc.opened)
return;
- if (get_new_crc_ctl_reg(dev_priv, crtc->index, &pipe_crc->source, &val, false) < 0)
+ if (get_new_crc_ctl_reg(dev_priv, crtc->index, &pipe_crc->source, &val) < 0)
return;
/* Don't need pipe_crc->lock here, IRQs are not generated. */
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_PIPE_CRC_H__
+#define __INTEL_PIPE_CRC_H__
+
+#include <linux/types.h>
+
+struct drm_crtc;
+struct intel_crtc;
+
+#ifdef CONFIG_DEBUG_FS
+int intel_crtc_set_crc_source(struct drm_crtc *crtc, const char *source_name);
+int intel_crtc_verify_crc_source(struct drm_crtc *crtc,
+ const char *source_name, size_t *values_cnt);
+const char *const *intel_crtc_get_crc_sources(struct drm_crtc *crtc,
+ size_t *count);
+void intel_crtc_disable_pipe_crc(struct intel_crtc *crtc);
+void intel_crtc_enable_pipe_crc(struct intel_crtc *crtc);
+#else
+#define intel_crtc_set_crc_source NULL
+#define intel_crtc_verify_crc_source NULL
+#define intel_crtc_get_crc_sources NULL
+static inline void intel_crtc_disable_pipe_crc(struct intel_crtc *crtc)
+{
+}
+
+static inline void intel_crtc_enable_pipe_crc(struct intel_crtc *crtc)
+{
+}
+#endif
+
+#endif /* __INTEL_PIPE_CRC_H__ */
#include "i915_drv.h"
#include "intel_drv.h"
+#include "intel_fbc.h"
+#include "intel_pm.h"
+#include "intel_sprite.h"
#include "../../../platform/x86/intel_ips.h"
/**
mutex_lock(&dev_priv->pcu_lock);
- val = vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ);
+ val = vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM);
if (enable)
val |= DSP_MAXFIFO_PM5_ENABLE;
else
val &= ~DSP_MAXFIFO_PM5_ENABLE;
- vlv_punit_write(dev_priv, PUNIT_REG_DSPFREQ, val);
+ vlv_punit_write(dev_priv, PUNIT_REG_DSPSSPM, val);
mutex_unlock(&dev_priv->pcu_lock);
}
u32 reg;
unsigned int wm;
- latency = intel_get_cxsr_latency(IS_PINEVIEW_G(dev_priv),
+ latency = intel_get_cxsr_latency(!IS_MOBILE(dev_priv),
dev_priv->is_ddr3,
dev_priv->fsb_freq,
dev_priv->mem_freq);
if (INTEL_GEN(dev_priv) < 11)
return enabled_slices;
- if (I915_READ(DBUF_CTL_S2) & DBUF_POWER_STATE)
+ /*
+ * FIXME: for now we'll only ever use 1 slice; pretend that we have
+ * only that 1 slice enabled until we have a proper way for on-demand
+ * toggling of the second slice.
+ */
+ if (0 && I915_READ(DBUF_CTL_S2) & DBUF_POWER_STATE)
enabled_slices++;
return enabled_slices;
alloc->end = ddb_size * (width_before_pipe + pipe_width) / total_width;
}
-static unsigned int skl_cursor_allocation(int num_active)
+static int skl_compute_wm_params(const struct intel_crtc_state *crtc_state,
+ int width, const struct drm_format_info *format,
+ u64 modifier, unsigned int rotation,
+ u32 plane_pixel_rate, struct skl_wm_params *wp,
+ int color_plane);
+static void skl_compute_plane_wm(const struct intel_crtc_state *cstate,
+ int level,
+ const struct skl_wm_params *wp,
+ const struct skl_wm_level *result_prev,
+ struct skl_wm_level *result /* out */);
+
+static unsigned int
+skl_cursor_allocation(const struct intel_crtc_state *crtc_state,
+ int num_active)
{
- if (num_active == 1)
- return 32;
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+ int level, max_level = ilk_wm_max_level(dev_priv);
+ struct skl_wm_level wm = {};
+ int ret, min_ddb_alloc = 0;
+ struct skl_wm_params wp;
+
+ ret = skl_compute_wm_params(crtc_state, 256,
+ drm_format_info(DRM_FORMAT_ARGB8888),
+ DRM_FORMAT_MOD_LINEAR,
+ DRM_MODE_ROTATE_0,
+ crtc_state->pixel_rate, &wp, 0);
+ WARN_ON(ret);
- return 8;
+ for (level = 0; level <= max_level; level++) {
+ skl_compute_plane_wm(crtc_state, level, &wp, &wm, &wm);
+ if (wm.min_ddb_alloc == U16_MAX)
+ break;
+
+ min_ddb_alloc = wm.min_ddb_alloc;
+ }
+
+ return max(num_active == 1 ? 32 : 8, min_ddb_alloc);
}
static void skl_ddb_entry_init_from_hw(struct drm_i915_private *dev_priv,
val = I915_READ(PLANE_BUF_CFG(pipe, plane_id));
val2 = I915_READ(PLANE_NV12_BUF_CFG(pipe, plane_id));
- if (fourcc == DRM_FORMAT_NV12)
+ if (is_planar_yuv_format(fourcc))
swap(val, val2);
skl_ddb_entry_init_from_hw(dev_priv, ddb_y, val);
if (intel_plane->id == PLANE_CURSOR)
return 0;
- if (plane == 1 && format != DRM_FORMAT_NV12)
+ if (plane == 1 && !is_planar_yuv_format(format))
return 0;
/*
height = drm_rect_height(&intel_pstate->base.src) >> 16;
/* UV plane does 1/2 pixel sub-sampling */
- if (plane == 1 && format == DRM_FORMAT_NV12) {
+ if (plane == 1 && is_planar_yuv_format(format)) {
width /= 2;
height /= 2;
}
struct drm_i915_private *dev_priv = to_i915(crtc->dev);
struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
struct skl_ddb_entry *alloc = &cstate->wm.skl.ddb;
- struct skl_plane_wm *wm;
u16 alloc_size, start = 0;
u16 total[I915_MAX_PLANES] = {};
u16 uv_total[I915_MAX_PLANES] = {};
return 0;
/* Allocate fixed number of blocks for cursor. */
- total[PLANE_CURSOR] = skl_cursor_allocation(num_active);
+ total[PLANE_CURSOR] = skl_cursor_allocation(cstate, num_active);
alloc_size -= total[PLANE_CURSOR];
cstate->wm.skl.plane_ddb_y[PLANE_CURSOR].start =
alloc->end - total[PLANE_CURSOR];
for (level = ilk_wm_max_level(dev_priv); level >= 0; level--) {
blocks = 0;
for_each_plane_id_on_crtc(intel_crtc, plane_id) {
- if (plane_id == PLANE_CURSOR)
+ const struct skl_plane_wm *wm =
+ &cstate->wm.skl.optimal.planes[plane_id];
+
+ if (plane_id == PLANE_CURSOR) {
+ if (WARN_ON(wm->wm[level].min_ddb_alloc >
+ total[PLANE_CURSOR])) {
+ blocks = U32_MAX;
+ break;
+ }
continue;
+ }
- wm = &cstate->wm.skl.optimal.planes[plane_id];
blocks += wm->wm[level].min_ddb_alloc;
blocks += wm->uv_wm[level].min_ddb_alloc;
}
- if (blocks < alloc_size) {
+ if (blocks <= alloc_size) {
alloc_size -= blocks;
break;
}
* proportional to its relative data rate.
*/
for_each_plane_id_on_crtc(intel_crtc, plane_id) {
+ const struct skl_plane_wm *wm =
+ &cstate->wm.skl.optimal.planes[plane_id];
u64 rate;
u16 extra;
if (total_data_rate == 0)
break;
- wm = &cstate->wm.skl.optimal.planes[plane_id];
-
rate = plane_data_rate[plane_id];
extra = min_t(u16, alloc_size,
DIV64_U64_ROUND_UP(alloc_size * rate,
/* Set the actual DDB start/end points for each plane */
start = alloc->start;
for_each_plane_id_on_crtc(intel_crtc, plane_id) {
- struct skl_ddb_entry *plane_alloc, *uv_plane_alloc;
+ struct skl_ddb_entry *plane_alloc =
+ &cstate->wm.skl.plane_ddb_y[plane_id];
+ struct skl_ddb_entry *uv_plane_alloc =
+ &cstate->wm.skl.plane_ddb_uv[plane_id];
if (plane_id == PLANE_CURSOR)
continue;
- plane_alloc = &cstate->wm.skl.plane_ddb_y[plane_id];
- uv_plane_alloc = &cstate->wm.skl.plane_ddb_uv[plane_id];
-
/* Gen11+ uses a separate plane for UV watermarks */
WARN_ON(INTEL_GEN(dev_priv) >= 11 && uv_total[plane_id]);
*/
for (level++; level <= ilk_wm_max_level(dev_priv); level++) {
for_each_plane_id_on_crtc(intel_crtc, plane_id) {
- wm = &cstate->wm.skl.optimal.planes[plane_id];
- memset(&wm->wm[level], 0, sizeof(wm->wm[level]));
+ struct skl_plane_wm *wm =
+ &cstate->wm.skl.optimal.planes[plane_id];
+
+ /*
+ * We only disable the watermarks for each plane if
+ * they exceed the ddb allocation of said plane. This
+ * is done so that we don't end up touching cursor
+ * watermarks needlessly when some other plane reduces
+ * our max possible watermark level.
+ *
+ * Bspec has this to say about the PLANE_WM enable bit:
+ * "All the watermarks at this level for all enabled
+ * planes must be enabled before the level will be used."
+ * So this is actually safe to do.
+ */
+ if (wm->wm[level].min_ddb_alloc > total[plane_id] ||
+ wm->uv_wm[level].min_ddb_alloc > uv_total[plane_id])
+ memset(&wm->wm[level], 0, sizeof(wm->wm[level]));
+
+ /*
+ * Wa_1408961008:icl, ehl
+ * Underruns with WM1+ disabled
+ */
+ if (IS_GEN(dev_priv, 11) &&
+ level == 1 && wm->wm[0].plane_en) {
+ wm->wm[level].plane_res_b = wm->wm[0].plane_res_b;
+ wm->wm[level].plane_res_l = wm->wm[0].plane_res_l;
+ wm->wm[level].ignore_lines = wm->wm[0].ignore_lines;
+ }
}
}
* don't have enough DDB blocks for it.
*/
for_each_plane_id_on_crtc(intel_crtc, plane_id) {
- wm = &cstate->wm.skl.optimal.planes[plane_id];
+ struct skl_plane_wm *wm =
+ &cstate->wm.skl.optimal.planes[plane_id];
+
if (wm->trans_wm.plane_res_b >= total[plane_id])
memset(&wm->trans_wm, 0, sizeof(wm->trans_wm));
}
}
static int
-skl_compute_plane_wm_params(const struct intel_crtc_state *cstate,
- const struct intel_plane_state *intel_pstate,
- struct skl_wm_params *wp, int color_plane)
+skl_compute_wm_params(const struct intel_crtc_state *crtc_state,
+ int width, const struct drm_format_info *format,
+ u64 modifier, unsigned int rotation,
+ u32 plane_pixel_rate, struct skl_wm_params *wp,
+ int color_plane)
{
- struct intel_plane *plane = to_intel_plane(intel_pstate->base.plane);
- struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
- const struct drm_plane_state *pstate = &intel_pstate->base;
- const struct drm_framebuffer *fb = pstate->fb;
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
u32 interm_pbpl;
- /* only NV12 format has two planes */
- if (color_plane == 1 && fb->format->format != DRM_FORMAT_NV12) {
- DRM_DEBUG_KMS("Non NV12 format have single plane\n");
+ /* only planar format has two planes */
+ if (color_plane == 1 && !is_planar_yuv_format(format->format)) {
+ DRM_DEBUG_KMS("Non planar format have single plane\n");
return -EINVAL;
}
- wp->y_tiled = fb->modifier == I915_FORMAT_MOD_Y_TILED ||
- fb->modifier == I915_FORMAT_MOD_Yf_TILED ||
- fb->modifier == I915_FORMAT_MOD_Y_TILED_CCS ||
- fb->modifier == I915_FORMAT_MOD_Yf_TILED_CCS;
- wp->x_tiled = fb->modifier == I915_FORMAT_MOD_X_TILED;
- wp->rc_surface = fb->modifier == I915_FORMAT_MOD_Y_TILED_CCS ||
- fb->modifier == I915_FORMAT_MOD_Yf_TILED_CCS;
- wp->is_planar = fb->format->format == DRM_FORMAT_NV12;
-
- if (plane->id == PLANE_CURSOR) {
- wp->width = intel_pstate->base.crtc_w;
- } else {
- /*
- * Src coordinates are already rotated by 270 degrees for
- * the 90/270 degree plane rotation cases (to match the
- * GTT mapping), hence no need to account for rotation here.
- */
- wp->width = drm_rect_width(&intel_pstate->base.src) >> 16;
- }
+ wp->y_tiled = modifier == I915_FORMAT_MOD_Y_TILED ||
+ modifier == I915_FORMAT_MOD_Yf_TILED ||
+ modifier == I915_FORMAT_MOD_Y_TILED_CCS ||
+ modifier == I915_FORMAT_MOD_Yf_TILED_CCS;
+ wp->x_tiled = modifier == I915_FORMAT_MOD_X_TILED;
+ wp->rc_surface = modifier == I915_FORMAT_MOD_Y_TILED_CCS ||
+ modifier == I915_FORMAT_MOD_Yf_TILED_CCS;
+ wp->is_planar = is_planar_yuv_format(format->format);
+ wp->width = width;
if (color_plane == 1 && wp->is_planar)
wp->width /= 2;
- wp->cpp = fb->format->cpp[color_plane];
- wp->plane_pixel_rate = skl_adjusted_plane_pixel_rate(cstate,
- intel_pstate);
+ wp->cpp = format->cpp[color_plane];
+ wp->plane_pixel_rate = plane_pixel_rate;
if (INTEL_GEN(dev_priv) >= 11 &&
- fb->modifier == I915_FORMAT_MOD_Yf_TILED && wp->cpp == 1)
+ modifier == I915_FORMAT_MOD_Yf_TILED && wp->cpp == 1)
wp->dbuf_block_size = 256;
else
wp->dbuf_block_size = 512;
- if (drm_rotation_90_or_270(pstate->rotation)) {
-
+ if (drm_rotation_90_or_270(rotation)) {
switch (wp->cpp) {
case 1:
wp->y_min_scanlines = 16;
wp->y_tile_minimum = mul_u32_fixed16(wp->y_min_scanlines,
wp->plane_blocks_per_line);
+
wp->linetime_us = fixed16_to_u32_round_up(
- intel_get_linetime_us(cstate));
+ intel_get_linetime_us(crtc_state));
return 0;
}
+static int
+skl_compute_plane_wm_params(const struct intel_crtc_state *crtc_state,
+ const struct intel_plane_state *plane_state,
+ struct skl_wm_params *wp, int color_plane)
+{
+ struct intel_plane *plane = to_intel_plane(plane_state->base.plane);
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ int width;
+
+ if (plane->id == PLANE_CURSOR) {
+ width = plane_state->base.crtc_w;
+ } else {
+ /*
+ * Src coordinates are already rotated by 270 degrees for
+ * the 90/270 degree plane rotation cases (to match the
+ * GTT mapping), hence no need to account for rotation here.
+ */
+ width = drm_rect_width(&plane_state->base.src) >> 16;
+ }
+
+ return skl_compute_wm_params(crtc_state, width,
+ fb->format, fb->modifier,
+ plane_state->base.rotation,
+ skl_adjusted_plane_pixel_rate(crtc_state, plane_state),
+ wp, color_plane);
+}
+
static bool skl_wm_has_lines(struct drm_i915_private *dev_priv, int level)
{
if (INTEL_GEN(dev_priv) >= 10 || IS_GEMINILAKE(dev_priv))
}
static void skl_compute_plane_wm(const struct intel_crtc_state *cstate,
- const struct intel_plane_state *intel_pstate,
int level,
const struct skl_wm_params *wp,
const struct skl_wm_level *result_prev,
struct skl_wm_level *result /* out */)
{
- struct drm_i915_private *dev_priv =
- to_i915(intel_pstate->base.plane->dev);
+ struct drm_i915_private *dev_priv = to_i915(cstate->base.crtc->dev);
u32 latency = dev_priv->wm.skl_latency[level];
uint_fixed_16_16_t method1, method2;
uint_fixed_16_16_t selected_result;
static void
skl_compute_wm_levels(const struct intel_crtc_state *cstate,
- const struct intel_plane_state *intel_pstate,
const struct skl_wm_params *wm_params,
struct skl_wm_level *levels)
{
- struct drm_i915_private *dev_priv =
- to_i915(intel_pstate->base.plane->dev);
+ struct drm_i915_private *dev_priv = to_i915(cstate->base.crtc->dev);
int level, max_level = ilk_wm_max_level(dev_priv);
struct skl_wm_level *result_prev = &levels[0];
for (level = 0; level <= max_level; level++) {
struct skl_wm_level *result = &levels[level];
- skl_compute_plane_wm(cstate, intel_pstate, level, wm_params,
+ skl_compute_plane_wm(cstate, level, wm_params,
result_prev, result);
result_prev = result;
if (ret)
return ret;
- skl_compute_wm_levels(crtc_state, plane_state, &wm_params, wm->wm);
+ skl_compute_wm_levels(crtc_state, &wm_params, wm->wm);
skl_compute_transition_wm(crtc_state, &wm_params, wm);
return 0;
if (ret)
return ret;
- skl_compute_wm_levels(crtc_state, plane_state, &wm_params, wm->uv_wm);
+ skl_compute_wm_levels(crtc_state, &wm_params, wm->uv_wm);
return 0;
}
-static int skl_build_plane_wm(struct skl_pipe_wm *pipe_wm,
- struct intel_crtc_state *crtc_state,
+static int skl_build_plane_wm(struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
struct intel_plane *plane = to_intel_plane(plane_state->base.plane);
return 0;
}
-static int icl_build_plane_wm(struct skl_pipe_wm *pipe_wm,
- struct intel_crtc_state *crtc_state,
+static int icl_build_plane_wm(struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
enum plane_id plane_id = to_intel_plane(plane_state->base.plane)->id;
return 0;
}
-static int skl_build_pipe_wm(struct intel_crtc_state *cstate,
- struct skl_pipe_wm *pipe_wm)
+static int skl_build_pipe_wm(struct intel_crtc_state *cstate)
{
struct drm_i915_private *dev_priv = to_i915(cstate->base.crtc->dev);
+ struct skl_pipe_wm *pipe_wm = &cstate->wm.skl.optimal;
struct drm_crtc_state *crtc_state = &cstate->base;
struct drm_plane *plane;
const struct drm_plane_state *pstate;
to_intel_plane_state(pstate);
if (INTEL_GEN(dev_priv) >= 11)
- ret = icl_build_plane_wm(pipe_wm,
- cstate, intel_pstate);
+ ret = icl_build_plane_wm(cstate, intel_pstate);
else
- ret = skl_build_plane_wm(pipe_wm,
- cstate, intel_pstate);
+ ret = skl_build_plane_wm(cstate, intel_pstate);
if (ret)
return ret;
}
{
u32 val = 0;
- if (level->plane_en) {
+ if (level->plane_en)
val |= PLANE_WM_EN;
- val |= level->plane_res_b;
- val |= level->plane_res_l << PLANE_WM_LINES_SHIFT;
- }
+ if (level->ignore_lines)
+ val |= PLANE_WM_IGNORE_LINES;
+ val |= level->plane_res_b;
+ val |= level->plane_res_l << PLANE_WM_LINES_SHIFT;
I915_WRITE_FW(reg, val);
}
const struct skl_wm_level *l2)
{
return l1->plane_en == l2->plane_en &&
+ l1->ignore_lines == l2->ignore_lines &&
l1->plane_res_l == l2->plane_res_l &&
l1->plane_res_b == l2->plane_res_b;
}
}
bool skl_ddb_allocation_overlaps(const struct skl_ddb_entry *ddb,
- const struct skl_ddb_entry entries[],
+ const struct skl_ddb_entry *entries,
int num_entries, int ignore_idx)
{
int i;
return false;
}
-static int skl_update_pipe_wm(struct intel_crtc_state *cstate,
- const struct skl_pipe_wm *old_pipe_wm,
- struct skl_pipe_wm *pipe_wm, /* out */
- bool *changed /* out */)
-{
- struct intel_crtc *crtc = to_intel_crtc(cstate->base.crtc);
- int ret;
-
- ret = skl_build_pipe_wm(cstate, pipe_wm);
- if (ret)
- return ret;
-
- *changed = !skl_pipe_wm_equals(crtc, old_pipe_wm, pipe_wm);
-
- return 0;
-}
-
static u32
pipes_modified(struct intel_atomic_state *state)
{
return 0;
}
+static char enast(bool enable)
+{
+ return enable ? '*' : ' ';
+}
+
static void
skl_print_wm_changes(struct intel_atomic_state *state)
{
struct intel_crtc *crtc;
int i;
+ if ((drm_debug & DRM_UT_KMS) == 0)
+ return;
+
for_each_oldnew_intel_crtc_in_state(state, crtc, old_crtc_state,
new_crtc_state, i) {
+ const struct skl_pipe_wm *old_pipe_wm, *new_pipe_wm;
+
+ old_pipe_wm = &old_crtc_state->wm.skl.optimal;
+ new_pipe_wm = &new_crtc_state->wm.skl.optimal;
+
for_each_intel_plane_on_crtc(&dev_priv->drm, crtc, plane) {
enum plane_id plane_id = plane->id;
const struct skl_ddb_entry *old, *new;
if (skl_ddb_entry_equal(old, new))
continue;
- DRM_DEBUG_KMS("[PLANE:%d:%s] ddb (%d - %d) -> (%d - %d)\n",
+ DRM_DEBUG_KMS("[PLANE:%d:%s] ddb (%4d - %4d) -> (%4d - %4d), size %4d -> %4d\n",
+ plane->base.base.id, plane->base.name,
+ old->start, old->end, new->start, new->end,
+ skl_ddb_entry_size(old), skl_ddb_entry_size(new));
+ }
+
+ for_each_intel_plane_on_crtc(&dev_priv->drm, crtc, plane) {
+ enum plane_id plane_id = plane->id;
+ const struct skl_plane_wm *old_wm, *new_wm;
+
+ old_wm = &old_pipe_wm->planes[plane_id];
+ new_wm = &new_pipe_wm->planes[plane_id];
+
+ if (skl_plane_wm_equals(dev_priv, old_wm, new_wm))
+ continue;
+
+ DRM_DEBUG_KMS("[PLANE:%d:%s] level %cwm0,%cwm1,%cwm2,%cwm3,%cwm4,%cwm5,%cwm6,%cwm7,%ctwm"
+ " -> %cwm0,%cwm1,%cwm2,%cwm3,%cwm4,%cwm5,%cwm6,%cwm7,%ctwm\n",
+ plane->base.base.id, plane->base.name,
+ enast(old_wm->wm[0].plane_en), enast(old_wm->wm[1].plane_en),
+ enast(old_wm->wm[2].plane_en), enast(old_wm->wm[3].plane_en),
+ enast(old_wm->wm[4].plane_en), enast(old_wm->wm[5].plane_en),
+ enast(old_wm->wm[6].plane_en), enast(old_wm->wm[7].plane_en),
+ enast(old_wm->trans_wm.plane_en),
+ enast(new_wm->wm[0].plane_en), enast(new_wm->wm[1].plane_en),
+ enast(new_wm->wm[2].plane_en), enast(new_wm->wm[3].plane_en),
+ enast(new_wm->wm[4].plane_en), enast(new_wm->wm[5].plane_en),
+ enast(new_wm->wm[6].plane_en), enast(new_wm->wm[7].plane_en),
+ enast(new_wm->trans_wm.plane_en));
+
+ DRM_DEBUG_KMS("[PLANE:%d:%s] lines %c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d"
+ " -> %c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d,%c%3d\n",
plane->base.base.id, plane->base.name,
- old->start, old->end,
- new->start, new->end);
+ enast(old_wm->wm[0].ignore_lines), old_wm->wm[0].plane_res_l,
+ enast(old_wm->wm[1].ignore_lines), old_wm->wm[1].plane_res_l,
+ enast(old_wm->wm[2].ignore_lines), old_wm->wm[2].plane_res_l,
+ enast(old_wm->wm[3].ignore_lines), old_wm->wm[3].plane_res_l,
+ enast(old_wm->wm[4].ignore_lines), old_wm->wm[4].plane_res_l,
+ enast(old_wm->wm[5].ignore_lines), old_wm->wm[5].plane_res_l,
+ enast(old_wm->wm[6].ignore_lines), old_wm->wm[6].plane_res_l,
+ enast(old_wm->wm[7].ignore_lines), old_wm->wm[7].plane_res_l,
+ enast(old_wm->trans_wm.ignore_lines), old_wm->trans_wm.plane_res_l,
+
+ enast(new_wm->wm[0].ignore_lines), new_wm->wm[0].plane_res_l,
+ enast(new_wm->wm[1].ignore_lines), new_wm->wm[1].plane_res_l,
+ enast(new_wm->wm[2].ignore_lines), new_wm->wm[2].plane_res_l,
+ enast(new_wm->wm[3].ignore_lines), new_wm->wm[3].plane_res_l,
+ enast(new_wm->wm[4].ignore_lines), new_wm->wm[4].plane_res_l,
+ enast(new_wm->wm[5].ignore_lines), new_wm->wm[5].plane_res_l,
+ enast(new_wm->wm[6].ignore_lines), new_wm->wm[6].plane_res_l,
+ enast(new_wm->wm[7].ignore_lines), new_wm->wm[7].plane_res_l,
+ enast(new_wm->trans_wm.ignore_lines), new_wm->trans_wm.plane_res_l);
+
+ DRM_DEBUG_KMS("[PLANE:%d:%s] blocks %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d"
+ " -> %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d\n",
+ plane->base.base.id, plane->base.name,
+ old_wm->wm[0].plane_res_b, old_wm->wm[1].plane_res_b,
+ old_wm->wm[2].plane_res_b, old_wm->wm[3].plane_res_b,
+ old_wm->wm[4].plane_res_b, old_wm->wm[5].plane_res_b,
+ old_wm->wm[6].plane_res_b, old_wm->wm[7].plane_res_b,
+ old_wm->trans_wm.plane_res_b,
+ new_wm->wm[0].plane_res_b, new_wm->wm[1].plane_res_b,
+ new_wm->wm[2].plane_res_b, new_wm->wm[3].plane_res_b,
+ new_wm->wm[4].plane_res_b, new_wm->wm[5].plane_res_b,
+ new_wm->wm[6].plane_res_b, new_wm->wm[7].plane_res_b,
+ new_wm->trans_wm.plane_res_b);
+
+ DRM_DEBUG_KMS("[PLANE:%d:%s] min_ddb %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d"
+ " -> %4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d,%4d\n",
+ plane->base.base.id, plane->base.name,
+ old_wm->wm[0].min_ddb_alloc, old_wm->wm[1].min_ddb_alloc,
+ old_wm->wm[2].min_ddb_alloc, old_wm->wm[3].min_ddb_alloc,
+ old_wm->wm[4].min_ddb_alloc, old_wm->wm[5].min_ddb_alloc,
+ old_wm->wm[6].min_ddb_alloc, old_wm->wm[7].min_ddb_alloc,
+ old_wm->trans_wm.min_ddb_alloc,
+ new_wm->wm[0].min_ddb_alloc, new_wm->wm[1].min_ddb_alloc,
+ new_wm->wm[2].min_ddb_alloc, new_wm->wm[3].min_ddb_alloc,
+ new_wm->wm[4].min_ddb_alloc, new_wm->wm[5].min_ddb_alloc,
+ new_wm->wm[6].min_ddb_alloc, new_wm->wm[7].min_ddb_alloc,
+ new_wm->trans_wm.min_ddb_alloc);
}
}
}
skl_compute_wm(struct intel_atomic_state *state)
{
struct intel_crtc *crtc;
- struct intel_crtc_state *cstate;
+ struct intel_crtc_state *new_crtc_state;
struct intel_crtc_state *old_crtc_state;
struct skl_ddb_values *results = &state->wm_results;
- struct skl_pipe_wm *pipe_wm;
bool changed = false;
int ret, i;
* pipe allocations had to change.
*/
for_each_oldnew_intel_crtc_in_state(state, crtc, old_crtc_state,
- cstate, i) {
- const struct skl_pipe_wm *old_pipe_wm =
- &old_crtc_state->wm.skl.optimal;
-
- pipe_wm = &cstate->wm.skl.optimal;
- ret = skl_update_pipe_wm(cstate, old_pipe_wm, pipe_wm, &changed);
+ new_crtc_state, i) {
+ ret = skl_build_pipe_wm(new_crtc_state);
if (ret)
return ret;
if (ret)
return ret;
- if (changed)
+ if (!skl_pipe_wm_equals(crtc,
+ &old_crtc_state->wm.skl.optimal,
+ &new_crtc_state->wm.skl.optimal))
results->dirty_pipes |= drm_crtc_mask(&crtc->base);
}
struct skl_wm_level *level)
{
level->plane_en = val & PLANE_WM_EN;
+ level->ignore_lines = val & PLANE_WM_IGNORE_LINES;
level->plane_res_b = val & PLANE_WM_BLOCKS_MASK;
level->plane_res_l = (val >> PLANE_WM_LINES_SHIFT) &
PLANE_WM_LINES_MASK;
if (IS_CHERRYVIEW(dev_priv)) {
mutex_lock(&dev_priv->pcu_lock);
- val = vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ);
+ val = vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM);
if (val & DSP_MAXFIFO_PM5_ENABLE)
wm->level = VLV_WM_LEVEL_PM5;
ei_down * threshold_down / 100));
I915_WRITE(GEN6_RP_CONTROL,
- GEN6_RP_MEDIA_TURBO |
+ (INTEL_GEN(dev_priv) > 9 ? 0 : GEN6_RP_MEDIA_TURBO) |
GEN6_RP_MEDIA_HW_NORMAL_MODE |
GEN6_RP_MEDIA_IS_GFX |
GEN6_RP_ENABLE |
* punit into committing the voltage change) as that takes a lot less
* power than the render powerwell.
*/
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_MEDIA);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_MEDIA);
err = valleyview_set_rps(dev_priv, val);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_MEDIA);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_MEDIA);
if (err)
DRM_ERROR("Failed to set RPS for idle\n");
mutex_unlock(&dev_priv->pcu_lock);
}
-void gen6_rps_boost(struct i915_request *rq,
- struct intel_rps_client *rps_client)
+void gen6_rps_boost(struct i915_request *rq)
{
struct intel_rps *rps = &rq->i915->gt_pm.rps;
unsigned long flags;
if (READ_ONCE(rps->cur_freq) < rps->boost_freq)
schedule_work(&rps->work);
- atomic_inc(rps_client ? &rps_client->boosts : &rps->boosts);
+ atomic_inc(&rps->boosts);
}
int intel_set_rps(struct drm_i915_private *dev_priv, u8 val)
{
/* We're doing forcewake before Disabling RC6,
* This what the BIOS expects when going into suspend */
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
I915_WRITE(GEN6_RC_CONTROL, 0);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static void valleyview_disable_rps(struct drm_i915_private *dev_priv)
/* See the Gen9_GT_PM_Programming_Guide doc for the below */
static void gen9_enable_rps(struct drm_i915_private *dev_priv)
{
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
/* Program defaults and thresholds for RPS */
if (IS_GEN(dev_priv, 9))
* RP_INTERRUPT_LIMITS & RPNSWREQ registers */
reset_rps(dev_priv, gen6_set_rps);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
+}
+
+static void gen11_enable_rc6(struct drm_i915_private *dev_priv)
+{
+ struct intel_engine_cs *engine;
+ enum intel_engine_id id;
+
+ /* 1a: Software RC state - RC0 */
+ I915_WRITE(GEN6_RC_STATE, 0);
+
+ /*
+ * 1b: Get forcewake during program sequence. Although the driver
+ * hasn't enabled a state yet where we need forcewake, BIOS may have.
+ */
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
+
+ /* 2a: Disable RC states. */
+ I915_WRITE(GEN6_RC_CONTROL, 0);
+
+ /* 2b: Program RC6 thresholds.*/
+ I915_WRITE(GEN6_RC6_WAKE_RATE_LIMIT, 54 << 16 | 85);
+ I915_WRITE(GEN10_MEDIA_WAKE_RATE_LIMIT, 150);
+
+ I915_WRITE(GEN6_RC_EVALUATION_INTERVAL, 125000); /* 12500 * 1280ns */
+ I915_WRITE(GEN6_RC_IDLE_HYSTERSIS, 25); /* 25 * 1280ns */
+ for_each_engine(engine, dev_priv, id)
+ I915_WRITE(RING_MAX_IDLE(engine->mmio_base), 10);
+
+ if (HAS_GUC(dev_priv))
+ I915_WRITE(GUC_MAX_IDLE_COUNT, 0xA);
+
+ I915_WRITE(GEN6_RC_SLEEP, 0);
+
+ I915_WRITE(GEN6_RC6_THRESHOLD, 50000); /* 50/125ms per EI */
+
+ /*
+ * 2c: Program Coarse Power Gating Policies.
+ *
+ * Bspec's guidance is to use 25us (really 25 * 1280ns) here. What we
+ * use instead is a more conservative estimate for the maximum time
+ * it takes us to service a CS interrupt and submit a new ELSP - that
+ * is the time which the GPU is idle waiting for the CPU to select the
+ * next request to execute. If the idle hysteresis is less than that
+ * interrupt service latency, the hardware will automatically gate
+ * the power well and we will then incur the wake up cost on top of
+ * the service latency. A similar guide from intel_pstate is that we
+ * do not want the enable hysteresis to less than the wakeup latency.
+ *
+ * igt/gem_exec_nop/sequential provides a rough estimate for the
+ * service latency, and puts it around 10us for Broadwell (and other
+ * big core) and around 40us for Broxton (and other low power cores).
+ * [Note that for legacy ringbuffer submission, this is less than 1us!]
+ * However, the wakeup latency on Broxton is closer to 100us. To be
+ * conservative, we have to factor in a context switch on top (due
+ * to ksoftirqd).
+ */
+ I915_WRITE(GEN9_MEDIA_PG_IDLE_HYSTERESIS, 250);
+ I915_WRITE(GEN9_RENDER_PG_IDLE_HYSTERESIS, 250);
+
+ /* 3a: Enable RC6 */
+ I915_WRITE(GEN6_RC_CONTROL,
+ GEN6_RC_CTL_HW_ENABLE |
+ GEN6_RC_CTL_RC6_ENABLE |
+ GEN6_RC_CTL_EI_MODE(1));
+
+ /* 3b: Enable Coarse Power Gating only when RC6 is enabled. */
+ I915_WRITE(GEN9_PG_ENABLE,
+ GEN9_RENDER_PG_ENABLE |
+ GEN9_MEDIA_PG_ENABLE |
+ GEN11_MEDIA_SAMPLER_PG_ENABLE);
+
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static void gen9_enable_rc6(struct drm_i915_private *dev_priv)
/* 1b: Get forcewake during program sequence. Although the driver
* hasn't enabled a state yet where we need forcewake, BIOS may have.*/
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
/* 2a: Disable RC states. */
I915_WRITE(GEN6_RC_CONTROL, 0);
I915_WRITE(GEN9_PG_ENABLE,
GEN9_RENDER_PG_ENABLE | GEN9_MEDIA_PG_ENABLE);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static void gen8_enable_rc6(struct drm_i915_private *dev_priv)
/* 1b: Get forcewake during program sequence. Although the driver
* hasn't enabled a state yet where we need forcewake, BIOS may have.*/
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
/* 2a: Disable RC states. */
I915_WRITE(GEN6_RC_CONTROL, 0);
GEN7_RC_CTL_TO_MODE |
GEN6_RC_CTL_RC6_ENABLE);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static void gen8_enable_rps(struct drm_i915_private *dev_priv)
{
struct intel_rps *rps = &dev_priv->gt_pm.rps;
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
/* 1 Program defaults and thresholds for RPS*/
I915_WRITE(GEN6_RPNSWREQ,
reset_rps(dev_priv, gen6_set_rps);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static void gen6_enable_rc6(struct drm_i915_private *dev_priv)
I915_WRITE(GTFIFODBG, gtfifodbg);
}
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
/* disable the counters and set deterministic thresholds */
I915_WRITE(GEN6_RC_CONTROL, 0);
DRM_ERROR("Couldn't fix incorrect rc6 voltage\n");
}
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static void gen6_enable_rps(struct drm_i915_private *dev_priv)
* Perhaps there might be some value in exposing these to
* userspace...
*/
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
/* Power down if completely idle for over 50ms */
I915_WRITE(GEN6_RP_DOWN_TIMEOUT, 50000);
reset_rps(dev_priv, gen6_set_rps);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static void gen6_update_ring_freq(struct drm_i915_private *dev_priv)
/* 1a & 1b: Get forcewake during program sequence. Although the driver
* hasn't enabled a state yet where we need forcewake, BIOS may have.*/
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
/* Disable RC states. */
I915_WRITE(GEN6_RC_CONTROL, 0);
rc6_mode = GEN7_RC_CTL_TO_MODE;
I915_WRITE(GEN6_RC_CONTROL, rc6_mode);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static void cherryview_enable_rps(struct drm_i915_private *dev_priv)
{
u32 val;
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
/* 1: Program defaults and thresholds for RPS*/
I915_WRITE(GEN6_RP_DOWN_TIMEOUT, 1000000);
reset_rps(dev_priv, valleyview_set_rps);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static void valleyview_enable_rc6(struct drm_i915_private *dev_priv)
I915_WRITE(GTFIFODBG, gtfifodbg);
}
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
/* Disable RC states. */
I915_WRITE(GEN6_RC_CONTROL, 0);
I915_WRITE(GEN6_RC_CONTROL,
GEN7_RC_CTL_TO_MODE | VLV_RC_CTL_CTX_RST_PARALLEL);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static void valleyview_enable_rps(struct drm_i915_private *dev_priv)
{
u32 val;
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
I915_WRITE(GEN6_RP_DOWN_TIMEOUT, 1000000);
I915_WRITE(GEN6_RP_UP_THRESHOLD, 59400);
reset_rps(dev_priv, valleyview_set_rps);
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
}
static unsigned long intel_pxfreq(u32 vidfreq)
return val;
}
-static struct drm_i915_private *i915_mch_dev;
+static struct drm_i915_private __rcu *i915_mch_dev;
static struct drm_i915_private *mchdev_get(void)
{
struct drm_i915_private *i915;
rcu_read_lock();
- i915 = i915_mch_dev;
+ i915 = rcu_dereference(i915_mch_dev);
if (!kref_get_unless_zero(&i915->drm.ref))
i915 = NULL;
rcu_read_unlock();
pm_runtime_put(&dev_priv->drm.pdev->dev);
}
-/**
- * intel_suspend_gt_powersave - suspend PM work and helper threads
- * @dev_priv: i915 device
- *
- * We don't want to disable RC6 or other features here, we just want
- * to make sure any work we've queued has finished and won't bother
- * us while we're suspended.
- */
-void intel_suspend_gt_powersave(struct drm_i915_private *dev_priv)
-{
- if (INTEL_GEN(dev_priv) < 6)
- return;
-
- /* gen6_rps_idle() will be called later to disable interrupts */
-}
-
void intel_sanitize_gt_powersave(struct drm_i915_private *dev_priv)
{
dev_priv->gt_pm.rps.enabled = true; /* force RPS disabling */
cherryview_enable_rc6(dev_priv);
else if (IS_VALLEYVIEW(dev_priv))
valleyview_enable_rc6(dev_priv);
+ else if (INTEL_GEN(dev_priv) >= 11)
+ gen11_enable_rc6(dev_priv);
else if (INTEL_GEN(dev_priv) >= 9)
gen9_enable_rc6(dev_priv);
else if (IS_BROADWELL(dev_priv))
*/
void intel_init_clock_gating_hooks(struct drm_i915_private *dev_priv)
{
- if (IS_ICELAKE(dev_priv))
+ if (IS_GEN(dev_priv, 11))
dev_priv->display.init_clock_gating = icl_init_clock_gating;
else if (IS_CANNONLAKE(dev_priv))
dev_priv->display.init_clock_gating = cnl_init_clock_gating;
dev_priv->display.initial_watermarks = g4x_initial_watermarks;
dev_priv->display.optimize_watermarks = g4x_optimize_watermarks;
} else if (IS_PINEVIEW(dev_priv)) {
- if (!intel_get_cxsr_latency(IS_PINEVIEW_G(dev_priv),
+ if (!intel_get_cxsr_latency(!IS_MOBILE(dev_priv),
dev_priv->is_ddr3,
dev_priv->fsb_freq,
dev_priv->mem_freq)) {
I915_WRITE_FW(GEN6_PCODE_DATA1, 0);
I915_WRITE_FW(GEN6_PCODE_MAILBOX, GEN6_PCODE_READY | mbox);
- if (__intel_wait_for_register_fw(dev_priv,
+ if (__intel_wait_for_register_fw(&dev_priv->uncore,
GEN6_PCODE_MAILBOX, GEN6_PCODE_READY, 0,
500, 0, NULL)) {
DRM_ERROR("timeout waiting for pcode read (from mbox %x) to finish for %ps\n",
I915_WRITE_FW(GEN6_PCODE_DATA1, 0);
I915_WRITE_FW(GEN6_PCODE_MAILBOX, GEN6_PCODE_READY | mbox);
- if (__intel_wait_for_register_fw(dev_priv,
+ if (__intel_wait_for_register_fw(&dev_priv->uncore,
GEN6_PCODE_MAILBOX, GEN6_PCODE_READY, 0,
fast_timeout_us, slow_timeout_ms,
NULL)) {
u64 intel_rc6_residency_ns(struct drm_i915_private *dev_priv,
const i915_reg_t reg)
{
+ struct intel_uncore *uncore = &dev_priv->uncore;
u64 time_hw, prev_hw, overflow_hw;
unsigned int fw_domains;
unsigned long flags;
if (WARN_ON_ONCE(i >= ARRAY_SIZE(dev_priv->gt_pm.rc6.cur_residency)))
return 0;
- fw_domains = intel_uncore_forcewake_for_reg(dev_priv, reg, FW_REG_READ);
+ fw_domains = intel_uncore_forcewake_for_reg(uncore, reg, FW_REG_READ);
- spin_lock_irqsave(&dev_priv->uncore.lock, flags);
- intel_uncore_forcewake_get__locked(dev_priv, fw_domains);
+ spin_lock_irqsave(&uncore->lock, flags);
+ intel_uncore_forcewake_get__locked(uncore, fw_domains);
/* On VLV and CHV, residency time is in CZ units rather than 1.28us */
if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) {
}
overflow_hw = BIT_ULL(32);
- time_hw = I915_READ_FW(reg);
+ time_hw = intel_uncore_read_fw(uncore, reg);
}
/*
time_hw += dev_priv->gt_pm.rc6.cur_residency[i];
dev_priv->gt_pm.rc6.cur_residency[i] = time_hw;
- intel_uncore_forcewake_put__locked(dev_priv, fw_domains);
- spin_unlock_irqrestore(&dev_priv->uncore.lock, flags);
+ intel_uncore_forcewake_put__locked(uncore, fw_domains);
+ spin_unlock_irqrestore(&uncore->lock, flags);
return mul_u64_u32_div(time_hw, mul, div);
}
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_PM_H__
+#define __INTEL_PM_H__
+
+#include <linux/types.h>
+
+struct drm_atomic_state;
+struct drm_device;
+struct drm_i915_private;
+struct i915_request;
+struct intel_crtc;
+struct intel_crtc_state;
+struct intel_plane;
+struct skl_ddb_allocation;
+struct skl_ddb_entry;
+struct skl_pipe_wm;
+struct skl_wm_level;
+
+void intel_init_clock_gating(struct drm_i915_private *dev_priv);
+void intel_suspend_hw(struct drm_i915_private *dev_priv);
+int ilk_wm_max_level(const struct drm_i915_private *dev_priv);
+void intel_update_watermarks(struct intel_crtc *crtc);
+void intel_init_pm(struct drm_i915_private *dev_priv);
+void intel_init_clock_gating_hooks(struct drm_i915_private *dev_priv);
+void intel_pm_setup(struct drm_i915_private *dev_priv);
+void intel_gpu_ips_init(struct drm_i915_private *dev_priv);
+void intel_gpu_ips_teardown(void);
+void intel_init_gt_powersave(struct drm_i915_private *dev_priv);
+void intel_cleanup_gt_powersave(struct drm_i915_private *dev_priv);
+void intel_sanitize_gt_powersave(struct drm_i915_private *dev_priv);
+void intel_enable_gt_powersave(struct drm_i915_private *dev_priv);
+void intel_disable_gt_powersave(struct drm_i915_private *dev_priv);
+void gen6_rps_busy(struct drm_i915_private *dev_priv);
+void gen6_rps_idle(struct drm_i915_private *dev_priv);
+void gen6_rps_boost(struct i915_request *rq);
+void g4x_wm_get_hw_state(struct drm_i915_private *dev_priv);
+void vlv_wm_get_hw_state(struct drm_i915_private *dev_priv);
+void ilk_wm_get_hw_state(struct drm_i915_private *dev_priv);
+void skl_wm_get_hw_state(struct drm_i915_private *dev_priv);
+void skl_pipe_ddb_get_hw_state(struct intel_crtc *crtc,
+ struct skl_ddb_entry *ddb_y,
+ struct skl_ddb_entry *ddb_uv);
+void skl_ddb_get_hw_state(struct drm_i915_private *dev_priv,
+ struct skl_ddb_allocation *ddb /* out */);
+void skl_pipe_wm_get_hw_state(struct intel_crtc *crtc,
+ struct skl_pipe_wm *out);
+void g4x_wm_sanitize(struct drm_i915_private *dev_priv);
+void vlv_wm_sanitize(struct drm_i915_private *dev_priv);
+bool intel_can_enable_sagv(struct drm_atomic_state *state);
+int intel_enable_sagv(struct drm_i915_private *dev_priv);
+int intel_disable_sagv(struct drm_i915_private *dev_priv);
+bool skl_wm_level_equals(const struct skl_wm_level *l1,
+ const struct skl_wm_level *l2);
+bool skl_ddb_allocation_overlaps(const struct skl_ddb_entry *ddb,
+ const struct skl_ddb_entry *entries,
+ int num_entries, int ignore_idx);
+void skl_write_plane_wm(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state);
+void skl_write_cursor_wm(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state);
+bool ilk_disable_lp_wm(struct drm_device *dev);
+int skl_check_pipe_max_pixel_rate(struct intel_crtc *intel_crtc,
+ struct intel_crtc_state *cstate);
+void intel_init_ipc(struct drm_i915_private *dev_priv);
+void intel_enable_ipc(struct drm_i915_private *dev_priv);
+
+#endif /* __INTEL_PM_H__ */
* DEALINGS IN THE SOFTWARE.
*/
+#include <drm/drm_atomic_helper.h>
+
+#include "i915_drv.h"
+#include "intel_dp.h"
+#include "intel_drv.h"
+#include "intel_psr.h"
+#include "intel_sprite.h"
+
/**
* DOC: Panel Self Refresh (PSR/SRD)
*
* must be correctly synchronized/cancelled when shutting down the pipe."
*/
-
-#include "intel_drv.h"
-#include "i915_drv.h"
-
static bool psr_global_enabled(u32 debug)
{
switch (debug & I915_PSR_DEBUG_MODE_MASK) {
case I915_PSR_DEBUG_DISABLE:
case I915_PSR_DEBUG_FORCE_PSR1:
return false;
- case I915_PSR_DEBUG_DEFAULT:
- if (i915_modparams.enable_psr <= 0)
- return false;
default:
return crtc_state->has_psr2;
}
drm_dp_dpcd_writeb(&intel_dp->aux, DP_SET_POWER, DP_SET_POWER_D0);
}
-static void hsw_activate_psr1(struct intel_dp *intel_dp)
+static u32 intel_psr1_get_tp_time(struct intel_dp *intel_dp)
{
struct drm_i915_private *dev_priv = dp_to_i915(intel_dp);
- u32 max_sleep_time = 0x1f;
- u32 val = EDP_PSR_ENABLE;
+ u32 val = 0;
- /* Let's use 6 as the minimum to cover all known cases including the
- * off-by-one issue that HW has in some cases.
- */
- int idle_frames = max(6, dev_priv->vbt.psr.idle_frames);
-
- /* sink_sync_latency of 8 means source has to wait for more than 8
- * frames, we'll go with 9 frames for now
- */
- idle_frames = max(idle_frames, dev_priv->psr.sink_sync_latency + 1);
- val |= idle_frames << EDP_PSR_IDLE_FRAME_SHIFT;
-
- val |= max_sleep_time << EDP_PSR_MAX_SLEEP_TIME_SHIFT;
- if (IS_HASWELL(dev_priv))
- val |= EDP_PSR_MIN_LINK_ENTRY_TIME_8_LINES;
-
- if (dev_priv->psr.link_standby)
- val |= EDP_PSR_LINK_STANDBY;
+ if (INTEL_GEN(dev_priv) >= 11)
+ val |= EDP_PSR_TP4_TIME_0US;
if (dev_priv->vbt.psr.tp1_wakeup_time_us == 0)
- val |= EDP_PSR_TP1_TIME_0us;
+ val |= EDP_PSR_TP1_TIME_0us;
else if (dev_priv->vbt.psr.tp1_wakeup_time_us <= 100)
val |= EDP_PSR_TP1_TIME_100us;
else if (dev_priv->vbt.psr.tp1_wakeup_time_us <= 500)
val |= EDP_PSR_TP1_TIME_2500us;
if (dev_priv->vbt.psr.tp2_tp3_wakeup_time_us == 0)
- val |= EDP_PSR_TP2_TP3_TIME_0us;
+ val |= EDP_PSR_TP2_TP3_TIME_0us;
else if (dev_priv->vbt.psr.tp2_tp3_wakeup_time_us <= 100)
val |= EDP_PSR_TP2_TP3_TIME_100us;
else if (dev_priv->vbt.psr.tp2_tp3_wakeup_time_us <= 500)
else
val |= EDP_PSR_TP1_TP2_SEL;
+ return val;
+}
+
+static void hsw_activate_psr1(struct intel_dp *intel_dp)
+{
+ struct drm_i915_private *dev_priv = dp_to_i915(intel_dp);
+ u32 max_sleep_time = 0x1f;
+ u32 val = EDP_PSR_ENABLE;
+
+ /* Let's use 6 as the minimum to cover all known cases including the
+ * off-by-one issue that HW has in some cases.
+ */
+ int idle_frames = max(6, dev_priv->vbt.psr.idle_frames);
+
+ /* sink_sync_latency of 8 means source has to wait for more than 8
+ * frames, we'll go with 9 frames for now
+ */
+ idle_frames = max(idle_frames, dev_priv->psr.sink_sync_latency + 1);
+ val |= idle_frames << EDP_PSR_IDLE_FRAME_SHIFT;
+
+ val |= max_sleep_time << EDP_PSR_MAX_SLEEP_TIME_SHIFT;
+ if (IS_HASWELL(dev_priv))
+ val |= EDP_PSR_MIN_LINK_ENTRY_TIME_8_LINES;
+
+ if (dev_priv->psr.link_standby)
+ val |= EDP_PSR_LINK_STANDBY;
+
+ val |= intel_psr1_get_tp_time(intel_dp);
+
if (INTEL_GEN(dev_priv) >= 8)
val |= EDP_PSR_CRC_ENABLE;
val |= EDP_PSR2_FRAME_BEFORE_SU(dev_priv->psr.sink_sync_latency + 1);
- if (dev_priv->vbt.psr.tp2_tp3_wakeup_time_us >= 0 &&
- dev_priv->vbt.psr.tp2_tp3_wakeup_time_us <= 50)
+ if (dev_priv->vbt.psr.psr2_tp2_tp3_wakeup_time_us >= 0 &&
+ dev_priv->vbt.psr.psr2_tp2_tp3_wakeup_time_us <= 50)
val |= EDP_PSR2_TP2_TIME_50us;
- else if (dev_priv->vbt.psr.tp2_tp3_wakeup_time_us <= 100)
+ else if (dev_priv->vbt.psr.psr2_tp2_tp3_wakeup_time_us <= 100)
val |= EDP_PSR2_TP2_TIME_100us;
- else if (dev_priv->vbt.psr.tp2_tp3_wakeup_time_us <= 500)
+ else if (dev_priv->vbt.psr.psr2_tp2_tp3_wakeup_time_us <= 500)
val |= EDP_PSR2_TP2_TIME_500us;
else
val |= EDP_PSR2_TP2_TIME_2500us;
+ /*
+ * PSR2 HW is incorrectly using EDP_PSR_TP1_TP3_SEL and BSpec is
+ * recommending keep this bit unset while PSR2 is enabled.
+ */
+ I915_WRITE(EDP_PSR_CTL, 0);
+
I915_WRITE(EDP_PSR2_CTL, val);
}
int crtc_vdisplay = crtc_state->base.adjusted_mode.crtc_vdisplay;
int psr_max_h = 0, psr_max_v = 0;
- /*
- * FIXME psr2_support is messed up. It's both computed
- * dynamically during PSR enable, and extracted from sink
- * caps during eDP detection.
- */
if (!dev_priv->psr.sink_psr2_support)
return false;
return false;
}
+ if (crtc_state->crc_enabled) {
+ DRM_DEBUG_KMS("PSR2 not enabled because it would inhibit pipe CRC calculation\n");
+ return false;
+ }
+
return true;
}
return;
}
- if (IS_HASWELL(dev_priv) &&
- adjusted_mode->flags & DRM_MODE_FLAG_INTERLACE) {
- DRM_DEBUG_KMS("PSR condition failed: Interlaced is Enabled\n");
+ if (adjusted_mode->flags & DRM_MODE_FLAG_INTERLACE) {
+ DRM_DEBUG_KMS("PSR condition failed: Interlaced mode enabled\n");
return;
}
{
struct intel_dp *intel_dp = dev_priv->psr.dp;
- if (dev_priv->psr.enabled)
- return;
+ WARN_ON(dev_priv->psr.enabled);
+
+ dev_priv->psr.psr2_enabled = intel_psr2_enabled(dev_priv, crtc_state);
+ dev_priv->psr.busy_frontbuffer_bits = 0;
+ dev_priv->psr.pipe = to_intel_crtc(crtc_state->base.crtc)->pipe;
DRM_DEBUG_KMS("Enabling PSR%s\n",
dev_priv->psr.psr2_enabled ? "2" : "1");
WARN_ON(dev_priv->drrs.dp);
mutex_lock(&dev_priv->psr.lock);
- if (dev_priv->psr.prepared) {
- DRM_DEBUG_KMS("PSR already in use\n");
+
+ if (!psr_global_enabled(dev_priv->psr.debug)) {
+ DRM_DEBUG_KMS("PSR disabled by flag\n");
goto unlock;
}
- dev_priv->psr.psr2_enabled = intel_psr2_enabled(dev_priv, crtc_state);
- dev_priv->psr.busy_frontbuffer_bits = 0;
- dev_priv->psr.prepared = true;
- dev_priv->psr.pipe = to_intel_crtc(crtc_state->base.crtc)->pipe;
-
- if (psr_global_enabled(dev_priv->psr.debug))
- intel_psr_enable_locked(dev_priv, crtc_state);
- else
- DRM_DEBUG_KMS("PSR disabled by flag\n");
+ intel_psr_enable_locked(dev_priv, crtc_state);
unlock:
mutex_unlock(&dev_priv->psr.lock);
}
/* Wait till PSR is idle */
- if (intel_wait_for_register(dev_priv, psr_status, psr_status_mask, 0,
- 2000))
+ if (intel_wait_for_register(&dev_priv->uncore,
+ psr_status, psr_status_mask, 0, 2000))
DRM_ERROR("Timed out waiting PSR idle state\n");
/* Disable PSR on Sink */
return;
mutex_lock(&dev_priv->psr.lock);
- if (!dev_priv->psr.prepared) {
- mutex_unlock(&dev_priv->psr.lock);
- return;
- }
intel_psr_disable_locked(intel_dp);
- dev_priv->psr.prepared = false;
mutex_unlock(&dev_priv->psr.lock);
cancel_work_sync(&dev_priv->psr.work);
}
+static void psr_force_hw_tracking_exit(struct drm_i915_private *dev_priv)
+{
+ /*
+ * Display WA #0884: all
+ * This documented WA for bxt can be safely applied
+ * broadly so we can force HW tracking to exit PSR
+ * instead of disabling and re-enabling.
+ * Workaround tells us to write 0 to CUR_SURFLIVE_A,
+ * but it makes more sense write to the current active
+ * pipe.
+ */
+ I915_WRITE(CURSURFLIVE(dev_priv->psr.pipe), 0);
+}
+
+/**
+ * intel_psr_update - Update PSR state
+ * @intel_dp: Intel DP
+ * @crtc_state: new CRTC state
+ *
+ * This functions will update PSR states, disabling, enabling or switching PSR
+ * version when executing fastsets. For full modeset, intel_psr_disable() and
+ * intel_psr_enable() should be called instead.
+ */
+void intel_psr_update(struct intel_dp *intel_dp,
+ const struct intel_crtc_state *crtc_state)
+{
+ struct drm_i915_private *dev_priv = dp_to_i915(intel_dp);
+ struct i915_psr *psr = &dev_priv->psr;
+ bool enable, psr2_enable;
+
+ if (!CAN_PSR(dev_priv) || READ_ONCE(psr->dp) != intel_dp)
+ return;
+
+ mutex_lock(&dev_priv->psr.lock);
+
+ enable = crtc_state->has_psr && psr_global_enabled(psr->debug);
+ psr2_enable = intel_psr2_enabled(dev_priv, crtc_state);
+
+ if (enable == psr->enabled && psr2_enable == psr->psr2_enabled) {
+ /* Force a PSR exit when enabling CRC to avoid CRC timeouts */
+ if (crtc_state->crc_enabled && psr->enabled)
+ psr_force_hw_tracking_exit(dev_priv);
+
+ goto unlock;
+ }
+
+ if (psr->enabled)
+ intel_psr_disable_locked(intel_dp);
+
+ if (enable)
+ intel_psr_enable_locked(dev_priv, crtc_state);
+
+unlock:
+ mutex_unlock(&dev_priv->psr.lock);
+}
+
/**
* intel_psr_wait_for_idle - wait for PSR1 to idle
* @new_crtc_state: new CRTC state
* defensive enough to cover everything.
*/
- return __intel_wait_for_register(dev_priv, EDP_PSR_STATUS,
+ return __intel_wait_for_register(&dev_priv->uncore, EDP_PSR_STATUS,
EDP_PSR_STATUS_STATE_MASK,
EDP_PSR_STATUS_STATE_IDLE, 2, 50,
out_value);
mutex_unlock(&dev_priv->psr.lock);
- err = intel_wait_for_register(dev_priv, reg, mask, 0, 50);
+ err = intel_wait_for_register(&dev_priv->uncore, reg, mask, 0, 50);
if (err)
DRM_ERROR("Timed out waiting for PSR Idle for re-enable\n");
return err == 0 && dev_priv->psr.enabled;
}
-static bool switching_psr(struct drm_i915_private *dev_priv,
- struct intel_crtc_state *crtc_state,
- u32 mode)
+static int intel_psr_fastset_force(struct drm_i915_private *dev_priv)
{
- /* Can't switch psr state anyway if PSR2 is not supported. */
- if (!crtc_state || !crtc_state->has_psr2)
- return false;
+ struct drm_device *dev = &dev_priv->drm;
+ struct drm_modeset_acquire_ctx ctx;
+ struct drm_atomic_state *state;
+ struct drm_crtc *crtc;
+ int err;
- if (dev_priv->psr.psr2_enabled && mode == I915_PSR_DEBUG_FORCE_PSR1)
- return true;
+ state = drm_atomic_state_alloc(dev);
+ if (!state)
+ return -ENOMEM;
- if (!dev_priv->psr.psr2_enabled && mode != I915_PSR_DEBUG_FORCE_PSR1)
- return true;
+ drm_modeset_acquire_init(&ctx, DRM_MODESET_ACQUIRE_INTERRUPTIBLE);
+ state->acquire_ctx = &ctx;
+
+retry:
+ drm_for_each_crtc(crtc, dev) {
+ struct drm_crtc_state *crtc_state;
+ struct intel_crtc_state *intel_crtc_state;
+
+ crtc_state = drm_atomic_get_crtc_state(state, crtc);
+ if (IS_ERR(crtc_state)) {
+ err = PTR_ERR(crtc_state);
+ goto error;
+ }
+
+ intel_crtc_state = to_intel_crtc_state(crtc_state);
- return false;
+ if (crtc_state->active && intel_crtc_state->has_psr) {
+ /* Mark mode as changed to trigger a pipe->update() */
+ crtc_state->mode_changed = true;
+ break;
+ }
+ }
+
+ err = drm_atomic_commit(state);
+
+error:
+ if (err == -EDEADLK) {
+ drm_atomic_state_clear(state);
+ err = drm_modeset_backoff(&ctx);
+ if (!err)
+ goto retry;
+ }
+
+ drm_modeset_drop_locks(&ctx);
+ drm_modeset_acquire_fini(&ctx);
+ drm_atomic_state_put(state);
+
+ return err;
}
-int intel_psr_set_debugfs_mode(struct drm_i915_private *dev_priv,
- struct drm_modeset_acquire_ctx *ctx,
- u64 val)
+int intel_psr_debug_set(struct drm_i915_private *dev_priv, u64 val)
{
- struct drm_device *dev = &dev_priv->drm;
- struct drm_connector_state *conn_state;
- struct intel_crtc_state *crtc_state = NULL;
- struct drm_crtc_commit *commit;
- struct drm_crtc *crtc;
- struct intel_dp *dp;
+ const u32 mode = val & I915_PSR_DEBUG_MODE_MASK;
+ u32 old_mode;
int ret;
- bool enable;
- u32 mode = val & I915_PSR_DEBUG_MODE_MASK;
if (val & ~(I915_PSR_DEBUG_IRQ | I915_PSR_DEBUG_MODE_MASK) ||
mode > I915_PSR_DEBUG_FORCE_PSR1) {
return -EINVAL;
}
- ret = drm_modeset_lock(&dev->mode_config.connection_mutex, ctx);
- if (ret)
- return ret;
-
- /* dev_priv->psr.dp should be set once and then never touched again. */
- dp = READ_ONCE(dev_priv->psr.dp);
- conn_state = dp->attached_connector->base.state;
- crtc = conn_state->crtc;
- if (crtc) {
- ret = drm_modeset_lock(&crtc->mutex, ctx);
- if (ret)
- return ret;
-
- crtc_state = to_intel_crtc_state(crtc->state);
- commit = crtc_state->base.commit;
- } else {
- commit = conn_state->commit;
- }
- if (commit) {
- ret = wait_for_completion_interruptible(&commit->hw_done);
- if (ret)
- return ret;
- }
-
ret = mutex_lock_interruptible(&dev_priv->psr.lock);
if (ret)
return ret;
- enable = psr_global_enabled(val);
-
- if (!enable || switching_psr(dev_priv, crtc_state, mode))
- intel_psr_disable_locked(dev_priv->psr.dp);
-
+ old_mode = dev_priv->psr.debug & I915_PSR_DEBUG_MODE_MASK;
dev_priv->psr.debug = val;
- if (crtc)
- dev_priv->psr.psr2_enabled = intel_psr2_enabled(dev_priv, crtc_state);
-
intel_psr_irq_control(dev_priv, dev_priv->psr.debug);
- if (dev_priv->psr.prepared && enable)
- intel_psr_enable_locked(dev_priv, crtc_state);
-
mutex_unlock(&dev_priv->psr.lock);
+
+ if (old_mode != mode)
+ ret = intel_psr_fastset_force(dev_priv);
+
return ret;
}
dev_priv->psr.busy_frontbuffer_bits &= ~frontbuffer_bits;
/* By definition flush = invalidate + flush */
- if (frontbuffer_bits) {
- /*
- * Display WA #0884: all
- * This documented WA for bxt can be safely applied
- * broadly so we can force HW tracking to exit PSR
- * instead of disabling and re-enabling.
- * Workaround tells us to write 0 to CUR_SURFLIVE_A,
- * but it makes more sense write to the current active
- * pipe.
- */
- I915_WRITE(CURSURFLIVE(dev_priv->psr.pipe), 0);
- }
+ if (frontbuffer_bits)
+ psr_force_hw_tracking_exit(dev_priv);
if (!dev_priv->psr.active && !dev_priv->psr.busy_frontbuffer_bits)
schedule_work(&dev_priv->psr.work);
if (val) {
DRM_DEBUG_KMS("PSR interruption error set\n");
dev_priv->psr.sink_not_reliable = true;
- return;
}
/* Set link_standby x link_off defaults */
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_PSR_H__
+#define __INTEL_PSR_H__
+
+#include "intel_frontbuffer.h"
+
+struct drm_i915_private;
+struct intel_crtc_state;
+struct intel_dp;
+
+#define CAN_PSR(dev_priv) (HAS_PSR(dev_priv) && dev_priv->psr.sink_support)
+void intel_psr_init_dpcd(struct intel_dp *intel_dp);
+void intel_psr_enable(struct intel_dp *intel_dp,
+ const struct intel_crtc_state *crtc_state);
+void intel_psr_disable(struct intel_dp *intel_dp,
+ const struct intel_crtc_state *old_crtc_state);
+void intel_psr_update(struct intel_dp *intel_dp,
+ const struct intel_crtc_state *crtc_state);
+int intel_psr_debug_set(struct drm_i915_private *dev_priv, u64 value);
+void intel_psr_invalidate(struct drm_i915_private *dev_priv,
+ unsigned frontbuffer_bits,
+ enum fb_op_origin origin);
+void intel_psr_flush(struct drm_i915_private *dev_priv,
+ unsigned frontbuffer_bits,
+ enum fb_op_origin origin);
+void intel_psr_init(struct drm_i915_private *dev_priv);
+void intel_psr_compute_config(struct intel_dp *intel_dp,
+ struct intel_crtc_state *crtc_state);
+void intel_psr_irq_control(struct drm_i915_private *dev_priv, u32 debug);
+void intel_psr_irq_handler(struct drm_i915_private *dev_priv, u32 psr_iir);
+void intel_psr_short_pulse(struct intel_dp *intel_dp);
+int intel_psr_wait_for_idle(const struct intel_crtc_state *new_crtc_state,
+ u32 *out_value);
+bool intel_psr_enabled(struct intel_dp *intel_dp);
+
+#endif /* __INTEL_PSR_H__ */
*/
#define LEGACY_REQUEST_SIZE 200
-static inline u32 intel_hws_seqno_address(struct intel_engine_cs *engine)
-{
- return (i915_ggtt_offset(engine->status_page.vma) +
- I915_GEM_HWS_INDEX_ADDR);
-}
-
unsigned int intel_ring_update_space(struct intel_ring *ring)
{
unsigned int space;
*cs++ = rq->fence.seqno;
*cs++ = GFX_OP_PIPE_CONTROL(4);
- *cs++ = PIPE_CONTROL_QW_WRITE | PIPE_CONTROL_CS_STALL;
- *cs++ = intel_hws_seqno_address(rq->engine) | PIPE_CONTROL_GLOBAL_GTT;
- *cs++ = rq->global_seqno;
+ *cs++ = PIPE_CONTROL_QW_WRITE | PIPE_CONTROL_STORE_DATA_INDEX;
+ *cs++ = I915_GEM_HWS_HANGCHECK_ADDR | PIPE_CONTROL_GLOBAL_GTT;
+ *cs++ = intel_engine_next_hangcheck_seqno(rq->engine);
*cs++ = MI_USER_INTERRUPT;
*cs++ = MI_NOOP;
*cs++ = GFX_OP_PIPE_CONTROL(4);
*cs++ = (PIPE_CONTROL_QW_WRITE |
- PIPE_CONTROL_GLOBAL_GTT_IVB |
- PIPE_CONTROL_CS_STALL);
- *cs++ = intel_hws_seqno_address(rq->engine);
- *cs++ = rq->global_seqno;
+ PIPE_CONTROL_STORE_DATA_INDEX |
+ PIPE_CONTROL_GLOBAL_GTT_IVB);
+ *cs++ = I915_GEM_HWS_HANGCHECK_ADDR;
+ *cs++ = intel_engine_next_hangcheck_seqno(rq->engine);
*cs++ = MI_USER_INTERRUPT;
*cs++ = MI_NOOP;
*cs++ = rq->fence.seqno;
*cs++ = MI_FLUSH_DW | MI_FLUSH_DW_OP_STOREDW | MI_FLUSH_DW_STORE_INDEX;
- *cs++ = I915_GEM_HWS_INDEX_ADDR | MI_FLUSH_DW_USE_GTT;
- *cs++ = rq->global_seqno;
+ *cs++ = I915_GEM_HWS_HANGCHECK_ADDR | MI_FLUSH_DW_USE_GTT;
+ *cs++ = intel_engine_next_hangcheck_seqno(rq->engine);
*cs++ = MI_USER_INTERRUPT;
*cs++ = MI_NOOP;
*cs++ = rq->fence.seqno;
*cs++ = MI_FLUSH_DW | MI_FLUSH_DW_OP_STOREDW | MI_FLUSH_DW_STORE_INDEX;
- *cs++ = I915_GEM_HWS_INDEX_ADDR | MI_FLUSH_DW_USE_GTT;
- *cs++ = rq->global_seqno;
+ *cs++ = I915_GEM_HWS_HANGCHECK_ADDR | MI_FLUSH_DW_USE_GTT;
+ *cs++ = intel_engine_next_hangcheck_seqno(rq->engine);
for (i = 0; i < GEN7_XCS_WA; i++) {
*cs++ = MI_STORE_DWORD_INDEX;
*/
default:
GEM_BUG_ON(engine->id);
- case RCS:
+ /* fallthrough */
+ case RCS0:
hwsp = RENDER_HWS_PGA_GEN7;
break;
- case BCS:
+ case BCS0:
hwsp = BLT_HWS_PGA_GEN7;
break;
- case VCS:
+ case VCS0:
hwsp = BSD_HWS_PGA_GEN7;
break;
- case VECS:
+ case VECS0:
hwsp = VEBOX_HWS_PGA_GEN7;
break;
}
static void flush_cs_tlb(struct intel_engine_cs *engine)
{
struct drm_i915_private *dev_priv = engine->i915;
- i915_reg_t instpm = RING_INSTPM(engine->mmio_base);
if (!IS_GEN_RANGE(dev_priv, 6, 7))
return;
/* ring should be idle before issuing a sync flush*/
- WARN_ON((I915_READ_MODE(engine) & MODE_IDLE) == 0);
-
- I915_WRITE(instpm,
- _MASKED_BIT_ENABLE(INSTPM_TLB_INVALIDATE |
- INSTPM_SYNC_FLUSH));
- if (intel_wait_for_register(dev_priv,
- instpm, INSTPM_SYNC_FLUSH, 0,
+ WARN_ON((ENGINE_READ(engine, RING_MI_MODE) & MODE_IDLE) == 0);
+
+ ENGINE_WRITE(engine, RING_INSTPM,
+ _MASKED_BIT_ENABLE(INSTPM_TLB_INVALIDATE |
+ INSTPM_SYNC_FLUSH));
+ if (intel_wait_for_register(engine->uncore,
+ RING_INSTPM(engine->mmio_base),
+ INSTPM_SYNC_FLUSH, 0,
1000))
DRM_ERROR("%s: wait for SyncFlush to complete for TLB invalidation timed out\n",
engine->name);
struct drm_i915_private *dev_priv = engine->i915;
if (INTEL_GEN(dev_priv) > 2) {
- I915_WRITE_MODE(engine, _MASKED_BIT_ENABLE(STOP_RING));
- if (intel_wait_for_register(dev_priv,
+ ENGINE_WRITE(engine,
+ RING_MI_MODE, _MASKED_BIT_ENABLE(STOP_RING));
+ if (intel_wait_for_register(engine->uncore,
RING_MI_MODE(engine->mmio_base),
MODE_IDLE,
MODE_IDLE,
1000)) {
DRM_ERROR("%s : timed out trying to stop ring\n",
engine->name);
- /* Sometimes we observe that the idle flag is not
+
+ /*
+ * Sometimes we observe that the idle flag is not
* set even though the ring is empty. So double
* check before giving up.
*/
- if (I915_READ_HEAD(engine) != I915_READ_TAIL(engine))
+ if (ENGINE_READ(engine, RING_HEAD) !=
+ ENGINE_READ(engine, RING_TAIL))
return false;
}
}
- I915_WRITE_HEAD(engine, I915_READ_TAIL(engine));
+ ENGINE_WRITE(engine, RING_HEAD, ENGINE_READ(engine, RING_TAIL));
- I915_WRITE_HEAD(engine, 0);
- I915_WRITE_TAIL(engine, 0);
+ ENGINE_WRITE(engine, RING_HEAD, 0);
+ ENGINE_WRITE(engine, RING_TAIL, 0);
/* The ring must be empty before it is disabled */
- I915_WRITE_CTL(engine, 0);
+ ENGINE_WRITE(engine, RING_CTL, 0);
- return (I915_READ_HEAD(engine) & HEAD_ADDR) == 0;
+ return (ENGINE_READ(engine, RING_HEAD) & HEAD_ADDR) == 0;
}
static int init_ring_common(struct intel_engine_cs *engine)
struct intel_ring *ring = engine->buffer;
int ret = 0;
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(engine->uncore, FORCEWAKE_ALL);
if (!stop_ring(engine)) {
/* G45 ring initialization often fails to reset head to zero */
DRM_DEBUG_DRIVER("%s head not reset to zero "
"ctl %08x head %08x tail %08x start %08x\n",
engine->name,
- I915_READ_CTL(engine),
- I915_READ_HEAD(engine),
- I915_READ_TAIL(engine),
- I915_READ_START(engine));
+ ENGINE_READ(engine, RING_CTL),
+ ENGINE_READ(engine, RING_HEAD),
+ ENGINE_READ(engine, RING_TAIL),
+ ENGINE_READ(engine, RING_START));
if (!stop_ring(engine)) {
DRM_ERROR("failed to set %s head to zero "
"ctl %08x head %08x tail %08x start %08x\n",
engine->name,
- I915_READ_CTL(engine),
- I915_READ_HEAD(engine),
- I915_READ_TAIL(engine),
- I915_READ_START(engine));
+ ENGINE_READ(engine, RING_CTL),
+ ENGINE_READ(engine, RING_HEAD),
+ ENGINE_READ(engine, RING_TAIL),
+ ENGINE_READ(engine, RING_START));
ret = -EIO;
goto out;
}
intel_engine_reset_breadcrumbs(engine);
/* Enforce ordering by reading HEAD register back */
- I915_READ_HEAD(engine);
+ ENGINE_READ(engine, RING_HEAD);
/* Initialize the ring. This must happen _after_ we've cleared the ring
* registers with the above sequence (the readback of the HEAD registers
* also enforces ordering), otherwise the hw might lose the new ring
* register values. */
- I915_WRITE_START(engine, i915_ggtt_offset(ring->vma));
+ ENGINE_WRITE(engine, RING_START, i915_ggtt_offset(ring->vma));
/* WaClearRingBufHeadRegAtInit:ctg,elk */
- if (I915_READ_HEAD(engine))
+ if (ENGINE_READ(engine, RING_HEAD))
DRM_DEBUG_DRIVER("%s initialization failed [head=%08x], fudging\n",
- engine->name, I915_READ_HEAD(engine));
+ engine->name, ENGINE_READ(engine, RING_HEAD));
/* Check that the ring offsets point within the ring! */
GEM_BUG_ON(!intel_ring_offset_valid(ring, ring->head));
intel_ring_update_space(ring);
/* First wake the ring up to an empty/idle ring */
- I915_WRITE_HEAD(engine, ring->head);
- I915_WRITE_TAIL(engine, ring->head);
- (void)I915_READ_TAIL(engine);
+ ENGINE_WRITE(engine, RING_HEAD, ring->head);
+ ENGINE_WRITE(engine, RING_TAIL, ring->head);
+ ENGINE_POSTING_READ(engine, RING_TAIL);
- I915_WRITE_CTL(engine, RING_CTL_SIZE(ring->size) | RING_VALID);
+ ENGINE_WRITE(engine, RING_CTL, RING_CTL_SIZE(ring->size) | RING_VALID);
/* If the head is still not zero, the ring is dead */
- if (intel_wait_for_register(dev_priv, RING_CTL(engine->mmio_base),
+ if (intel_wait_for_register(engine->uncore,
+ RING_CTL(engine->mmio_base),
RING_VALID, RING_VALID,
50)) {
DRM_ERROR("%s initialization failed "
"ctl %08x (valid? %d) head %08x [%08x] tail %08x [%08x] start %08x [expected %08x]\n",
engine->name,
- I915_READ_CTL(engine),
- I915_READ_CTL(engine) & RING_VALID,
- I915_READ_HEAD(engine), ring->head,
- I915_READ_TAIL(engine), ring->tail,
- I915_READ_START(engine),
+ ENGINE_READ(engine, RING_CTL),
+ ENGINE_READ(engine, RING_CTL) & RING_VALID,
+ ENGINE_READ(engine, RING_HEAD), ring->head,
+ ENGINE_READ(engine, RING_TAIL), ring->tail,
+ ENGINE_READ(engine, RING_START),
i915_ggtt_offset(ring->vma));
ret = -EIO;
goto out;
}
if (INTEL_GEN(dev_priv) > 2)
- I915_WRITE_MODE(engine, _MASKED_BIT_DISABLE(STOP_RING));
+ ENGINE_WRITE(engine,
+ RING_MI_MODE, _MASKED_BIT_DISABLE(STOP_RING));
/* Now awake, let it get started */
if (ring->tail != ring->head) {
- I915_WRITE_TAIL(engine, ring->tail);
- (void)I915_READ_TAIL(engine);
+ ENGINE_WRITE(engine, RING_TAIL, ring->tail);
+ ENGINE_POSTING_READ(engine, RING_TAIL);
}
/* Papering over lost _interrupts_ immediately following the restart */
intel_engine_queue_breadcrumbs(engine);
out:
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(engine->uncore, FORCEWAKE_ALL);
return ret;
}
}
}
- GEM_TRACE("%s seqno=%d, current=%d, stalled? %s\n",
- engine->name,
- rq ? rq->global_seqno : 0,
- intel_engine_get_seqno(engine),
- yesno(stalled));
/*
* The guilty request will get skipped on a hung engine.
*
I915_WRITE(INSTPM, _MASKED_BIT_ENABLE(INSTPM_FORCE_ORDERING));
if (INTEL_GEN(dev_priv) >= 6)
- I915_WRITE_IMR(engine, ~engine->irq_keep_mask);
+ ENGINE_WRITE(engine, RING_IMR, ~engine->irq_keep_mask);
return 0;
}
/* Mark all submitted requests as skipped. */
list_for_each_entry(request, &engine->timeline.requests, link) {
- GEM_BUG_ON(!request->global_seqno);
-
if (!i915_request_signaled(request))
dma_fence_set_error(&request->fence, -EIO);
i915_request_mark_complete(request);
}
- intel_write_status_page(engine,
- I915_GEM_HWS_INDEX,
- intel_engine_last_submit(engine));
-
/* Remaining _unready_ requests will be nop'ed when submitted */
spin_unlock_irqrestore(&engine->timeline.lock, flags);
static void i9xx_submit_request(struct i915_request *request)
{
- struct drm_i915_private *dev_priv = request->i915;
-
i915_request_submit(request);
- I915_WRITE_TAIL(request->engine,
- intel_ring_set_tail(request->ring, request->tail));
+ ENGINE_WRITE(request->engine, RING_TAIL,
+ intel_ring_set_tail(request->ring, request->tail));
}
static u32 *i9xx_emit_breadcrumb(struct i915_request *rq, u32 *cs)
*cs++ = rq->fence.seqno;
*cs++ = MI_STORE_DWORD_INDEX;
- *cs++ = I915_GEM_HWS_INDEX_ADDR;
- *cs++ = rq->global_seqno;
+ *cs++ = I915_GEM_HWS_HANGCHECK_ADDR;
+ *cs++ = intel_engine_next_hangcheck_seqno(rq->engine);
*cs++ = MI_USER_INTERRUPT;
*cs++ = MI_FLUSH;
*cs++ = MI_STORE_DWORD_INDEX;
- *cs++ = I915_GEM_HWS_SEQNO_ADDR;
- *cs++ = rq->fence.seqno;
+ *cs++ = I915_GEM_HWS_HANGCHECK_ADDR;
+ *cs++ = intel_engine_next_hangcheck_seqno(rq->engine);
BUILD_BUG_ON(GEN5_WA_STORES < 1);
for (i = 0; i < GEN5_WA_STORES; i++) {
*cs++ = MI_STORE_DWORD_INDEX;
- *cs++ = I915_GEM_HWS_INDEX_ADDR;
- *cs++ = rq->global_seqno;
+ *cs++ = I915_GEM_HWS_SEQNO_ADDR;
+ *cs++ = rq->fence.seqno;
}
*cs++ = MI_USER_INTERRUPT;
static void
i9xx_irq_enable(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
-
- dev_priv->irq_mask &= ~engine->irq_enable_mask;
- I915_WRITE(IMR, dev_priv->irq_mask);
- POSTING_READ_FW(RING_IMR(engine->mmio_base));
+ engine->i915->irq_mask &= ~engine->irq_enable_mask;
+ intel_uncore_write(engine->uncore, GEN2_IMR, engine->i915->irq_mask);
+ intel_uncore_posting_read_fw(engine->uncore, GEN2_IMR);
}
static void
i9xx_irq_disable(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
-
- dev_priv->irq_mask |= engine->irq_enable_mask;
- I915_WRITE(IMR, dev_priv->irq_mask);
+ engine->i915->irq_mask |= engine->irq_enable_mask;
+ intel_uncore_write(engine->uncore, GEN2_IMR, engine->i915->irq_mask);
}
static void
struct drm_i915_private *dev_priv = engine->i915;
dev_priv->irq_mask &= ~engine->irq_enable_mask;
- I915_WRITE16(IMR, dev_priv->irq_mask);
+ I915_WRITE16(GEN2_IMR, dev_priv->irq_mask);
POSTING_READ16(RING_IMR(engine->mmio_base));
}
struct drm_i915_private *dev_priv = engine->i915;
dev_priv->irq_mask |= engine->irq_enable_mask;
- I915_WRITE16(IMR, dev_priv->irq_mask);
+ I915_WRITE16(GEN2_IMR, dev_priv->irq_mask);
}
static int
static void
gen6_irq_enable(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
-
- I915_WRITE_IMR(engine,
- ~(engine->irq_enable_mask |
- engine->irq_keep_mask));
+ ENGINE_WRITE(engine, RING_IMR,
+ ~(engine->irq_enable_mask | engine->irq_keep_mask));
/* Flush/delay to ensure the RING_IMR is active before the GT IMR */
- POSTING_READ_FW(RING_IMR(engine->mmio_base));
+ ENGINE_POSTING_READ(engine, RING_IMR);
- gen5_enable_gt_irq(dev_priv, engine->irq_enable_mask);
+ gen5_enable_gt_irq(engine->i915, engine->irq_enable_mask);
}
static void
gen6_irq_disable(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
-
- I915_WRITE_IMR(engine, ~engine->irq_keep_mask);
- gen5_disable_gt_irq(dev_priv, engine->irq_enable_mask);
+ ENGINE_WRITE(engine, RING_IMR, ~engine->irq_keep_mask);
+ gen5_disable_gt_irq(engine->i915, engine->irq_enable_mask);
}
static void
hsw_vebox_irq_enable(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
-
- I915_WRITE_IMR(engine, ~engine->irq_enable_mask);
+ ENGINE_WRITE(engine, RING_IMR, ~engine->irq_enable_mask);
/* Flush/delay to ensure the RING_IMR is active before the GT IMR */
- POSTING_READ_FW(RING_IMR(engine->mmio_base));
+ ENGINE_POSTING_READ(engine, RING_IMR);
- gen6_unmask_pm_irq(dev_priv, engine->irq_enable_mask);
+ gen6_unmask_pm_irq(engine->i915, engine->irq_enable_mask);
}
static void
hsw_vebox_irq_disable(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
-
- I915_WRITE_IMR(engine, ~0);
- gen6_mask_pm_irq(dev_priv, engine->irq_enable_mask);
+ ENGINE_WRITE(engine, RING_IMR, ~0);
+ gen6_mask_pm_irq(engine->i915, engine->irq_enable_mask);
}
static int
else
flags |= PIN_HIGH;
- if (!(vma->flags & I915_VMA_GLOBAL_BIND)) {
- if (flags & PIN_MAPPABLE || map == I915_MAP_WC)
- ret = i915_gem_object_set_to_gtt_domain(vma->obj, true);
- else
- ret = i915_gem_object_set_to_cpu_domain(vma->obj, true);
- if (unlikely(ret))
- goto unpin_timeline;
- }
-
ret = i915_vma_pin(vma, 0, 0, flags);
if (unlikely(ret))
goto unpin_timeline;
if (!ring)
return ERR_PTR(-ENOMEM);
+ kref_init(&ring->ref);
INIT_LIST_HEAD(&ring->request_list);
ring->timeline = i915_timeline_get(timeline);
return ring;
}
-void
-intel_ring_free(struct intel_ring *ring)
+void intel_ring_free(struct kref *ref)
{
+ struct intel_ring *ring = container_of(ref, typeof(*ring), ref);
struct drm_i915_gem_object *obj = ring->vma->obj;
i915_vma_close(ring->vma);
kfree(ring);
}
-static void intel_ring_context_destroy(struct intel_context *ce)
+static void __ring_context_fini(struct intel_context *ce)
{
- GEM_BUG_ON(ce->pin_count);
-
- if (!ce->state)
- return;
-
GEM_BUG_ON(i915_gem_object_is_active(ce->state->obj));
i915_gem_object_put(ce->state->obj);
}
+static void ring_context_destroy(struct kref *ref)
+{
+ struct intel_context *ce = container_of(ref, typeof(*ce), ref);
+
+ GEM_BUG_ON(intel_context_is_pinned(ce));
+
+ if (ce->state)
+ __ring_context_fini(ce);
+
+ intel_context_free(ce);
+}
+
static int __context_pin_ppgtt(struct i915_gem_context *ctx)
{
struct i915_hw_ppgtt *ppgtt;
if (!vma)
return 0;
- /*
- * Clear this page out of any CPU caches for coherent swap-in/out.
- * We only want to do this on the first bind so that we do not stall
- * on an active context (which by nature is already on the GPU).
- */
- if (!(vma->flags & I915_VMA_GLOBAL_BIND)) {
- err = i915_gem_object_set_to_gtt_domain(vma->obj, true);
- if (err)
- return err;
- }
-
err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL | PIN_HIGH);
if (err)
return err;
* it cannot reclaim the object until we release it.
*/
vma->obj->pin_global++;
+ vma->obj->mm.dirty = true;
return 0;
}
i915_vma_unpin(vma);
}
-static void intel_ring_context_unpin(struct intel_context *ce)
+static void ring_context_unpin(struct intel_context *ce)
{
__context_unpin_ppgtt(ce->gem_context);
__context_unpin(ce);
-
- i915_gem_context_put(ce->gem_context);
}
static struct i915_vma *
if (IS_ERR(obj))
return ERR_CAST(obj);
+ /*
+ * Try to make the context utilize L3 as well as LLC.
+ *
+ * On VLV we don't have L3 controls in the PTEs so we
+ * shouldn't touch the cache level, especially as that
+ * would make the object snooped which might have a
+ * negative performance impact.
+ *
+ * Snooping is required on non-llc platforms in execlist
+ * mode, but since all GGTT accesses use PAT entry 0 we
+ * get snooping anyway regardless of cache_level.
+ *
+ * This is only applicable for Ivy Bridge devices since
+ * later platforms don't have L3 control bits in the PTE.
+ */
+ if (IS_IVYBRIDGE(i915))
+ i915_gem_object_set_cache_coherency(obj, I915_CACHE_L3_LLC);
+
if (engine->default_state) {
void *defaults, *vaddr;
}
memcpy(vaddr, defaults, engine->context_size);
-
i915_gem_object_unpin_map(engine->default_state);
- i915_gem_object_unpin_map(obj);
- }
- /*
- * Try to make the context utilize L3 as well as LLC.
- *
- * On VLV we don't have L3 controls in the PTEs so we
- * shouldn't touch the cache level, especially as that
- * would make the object snooped which might have a
- * negative performance impact.
- *
- * Snooping is required on non-llc platforms in execlist
- * mode, but since all GGTT accesses use PAT entry 0 we
- * get snooping anyway regardless of cache_level.
- *
- * This is only applicable for Ivy Bridge devices since
- * later platforms don't have L3 control bits in the PTE.
- */
- if (IS_IVYBRIDGE(i915)) {
- /* Ignore any error, regard it as a simple optimisation */
- i915_gem_object_set_cache_level(obj, I915_CACHE_L3_LLC);
+ i915_gem_object_flush_map(obj);
+ i915_gem_object_unpin_map(obj);
}
vma = i915_vma_instance(obj, &i915->ggtt.vm, NULL);
return ERR_PTR(err);
}
-static struct intel_context *
-__ring_context_pin(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx,
- struct intel_context *ce)
+static int ring_context_pin(struct intel_context *ce)
{
+ struct intel_engine_cs *engine = ce->engine;
int err;
+ /* One ringbuffer to rule them all */
+ GEM_BUG_ON(!engine->buffer);
+ ce->ring = engine->buffer;
+
if (!ce->state && engine->context_size) {
struct i915_vma *vma;
vma = alloc_context_vma(engine);
- if (IS_ERR(vma)) {
- err = PTR_ERR(vma);
- goto err;
- }
+ if (IS_ERR(vma))
+ return PTR_ERR(vma);
ce->state = vma;
}
err = __context_pin(ce);
if (err)
- goto err;
+ return err;
err = __context_pin_ppgtt(ce->gem_context);
if (err)
goto err_unpin;
- i915_gem_context_get(ctx);
-
- /* One ringbuffer to rule them all */
- GEM_BUG_ON(!engine->buffer);
- ce->ring = engine->buffer;
-
- return ce;
+ return 0;
err_unpin:
__context_unpin(ce);
-err:
- ce->pin_count = 0;
- return ERR_PTR(err);
+ return err;
}
-static const struct intel_context_ops ring_context_ops = {
- .unpin = intel_ring_context_unpin,
- .destroy = intel_ring_context_destroy,
-};
-
-static struct intel_context *
-intel_ring_context_pin(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx)
+static void ring_context_reset(struct intel_context *ce)
{
- struct intel_context *ce = to_intel_context(ctx, engine);
-
- lockdep_assert_held(&ctx->i915->drm.struct_mutex);
-
- if (likely(ce->pin_count++))
- return ce;
- GEM_BUG_ON(!ce->pin_count); /* no overflow please! */
+ intel_ring_reset(ce->ring, 0);
+}
- ce->ops = &ring_context_ops;
+static const struct intel_context_ops ring_context_ops = {
+ .pin = ring_context_pin,
+ .unpin = ring_context_unpin,
- return __ring_context_pin(engine, ctx, ce);
-}
+ .reset = ring_context_reset,
+ .destroy = ring_context_destroy,
+};
static int intel_init_ring_buffer(struct intel_engine_cs *engine)
{
if (err)
return err;
- timeline = i915_timeline_create(engine->i915,
- engine->name,
- engine->status_page.vma);
+ timeline = i915_timeline_create(engine->i915, engine->status_page.vma);
if (IS_ERR(timeline)) {
err = PTR_ERR(timeline);
goto err;
err_unpin:
intel_ring_unpin(ring);
err_ring:
- intel_ring_free(ring);
+ intel_ring_put(ring);
err:
intel_engine_cleanup_common(engine);
return err;
struct drm_i915_private *dev_priv = engine->i915;
WARN_ON(INTEL_GEN(dev_priv) > 2 &&
- (I915_READ_MODE(engine) & MODE_IDLE) == 0);
+ (ENGINE_READ(engine, RING_MI_MODE) & MODE_IDLE) == 0);
intel_ring_unpin(engine->buffer);
- intel_ring_free(engine->buffer);
+ intel_ring_put(engine->buffer);
if (engine->cleanup)
engine->cleanup(engine);
kfree(engine);
}
-void intel_legacy_submission_resume(struct drm_i915_private *dev_priv)
-{
- struct intel_engine_cs *engine;
- enum intel_engine_id id;
-
- /* Restart from the beginning of the rings for convenience */
- for_each_engine(engine, dev_priv, id)
- intel_ring_reset(engine->buffer, 0);
-}
-
static int load_pd_dir(struct i915_request *rq,
const struct i915_hw_ppgtt *ppgtt)
{
return PTR_ERR(cs);
*cs++ = MI_LOAD_REGISTER_IMM(1);
- *cs++ = i915_mmio_reg_offset(RING_PP_DIR_DCLV(engine));
+ *cs++ = i915_mmio_reg_offset(RING_PP_DIR_DCLV(engine->mmio_base));
*cs++ = PP_DIR_DCLV_2G;
*cs++ = MI_LOAD_REGISTER_IMM(1);
- *cs++ = i915_mmio_reg_offset(RING_PP_DIR_BASE(engine));
+ *cs++ = i915_mmio_reg_offset(RING_PP_DIR_BASE(engine->mmio_base));
*cs++ = ppgtt->pd.base.ggtt_offset << 10;
intel_ring_advance(rq, cs);
/* Stall until the page table load is complete */
*cs++ = MI_STORE_REGISTER_MEM | MI_SRM_LRM_GLOBAL_GTT;
- *cs++ = i915_mmio_reg_offset(RING_PP_DIR_BASE(engine));
+ *cs++ = i915_mmio_reg_offset(RING_PP_DIR_BASE(engine->mmio_base));
*cs++ = i915_scratch_offset(rq->i915);
*cs++ = MI_NOOP;
struct drm_i915_private *i915 = rq->i915;
struct intel_engine_cs *engine = rq->engine;
enum intel_engine_id id;
- const int num_rings =
- IS_HSW_GT1(i915) ? RUNTIME_INFO(i915)->num_rings - 1 : 0;
+ const int num_engines =
+ IS_HSW_GT1(i915) ? RUNTIME_INFO(i915)->num_engines - 1 : 0;
bool force_restore = false;
int len;
u32 *cs;
len = 4;
if (IS_GEN(i915, 7))
- len += 2 + (num_rings ? 4*num_rings + 6 : 0);
+ len += 2 + (num_engines ? 4 * num_engines + 6 : 0);
if (flags & MI_FORCE_RESTORE) {
GEM_BUG_ON(flags & MI_RESTORE_INHIBIT);
flags &= ~MI_FORCE_RESTORE;
/* WaProgramMiArbOnOffAroundMiSetContext:ivb,vlv,hsw,bdw,chv */
if (IS_GEN(i915, 7)) {
*cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
- if (num_rings) {
+ if (num_engines) {
struct intel_engine_cs *signaller;
- *cs++ = MI_LOAD_REGISTER_IMM(num_rings);
+ *cs++ = MI_LOAD_REGISTER_IMM(num_engines);
for_each_engine(signaller, i915, id) {
if (signaller == engine)
continue;
* placeholder we use to flush other contexts.
*/
*cs++ = MI_SET_CONTEXT;
- *cs++ = i915_ggtt_offset(to_intel_context(i915->kernel_context,
- engine)->state) |
+ *cs++ = i915_ggtt_offset(engine->kernel_context->state) |
MI_MM_SPACE_GTT |
MI_RESTORE_INHIBIT;
}
*cs++ = MI_NOOP;
if (IS_GEN(i915, 7)) {
- if (num_rings) {
+ if (num_engines) {
struct intel_engine_cs *signaller;
i915_reg_t last_reg = {}; /* keep gcc quiet */
- *cs++ = MI_LOAD_REGISTER_IMM(num_rings);
+ *cs++ = MI_LOAD_REGISTER_IMM(num_engines);
for_each_engine(signaller, i915, id) {
if (signaller == engine)
continue;
* explanation.
*/
loops = 1;
- if (engine->id == BCS && IS_VALLEYVIEW(engine->i915))
+ if (engine->id == BCS0 && IS_VALLEYVIEW(engine->i915))
loops = 32;
do {
goto err;
} while (--loops);
- if (intel_engine_flag(engine) & ppgtt->pd_dirty_rings) {
- unwind_mm = intel_engine_flag(engine);
- ppgtt->pd_dirty_rings &= ~unwind_mm;
+ if (ppgtt->pd_dirty_engines & engine->mask) {
+ unwind_mm = engine->mask;
+ ppgtt->pd_dirty_engines &= ~unwind_mm;
hw_flags = MI_FORCE_RESTORE;
}
}
if (rq->hw_context->state) {
- GEM_BUG_ON(engine->id != RCS);
+ GEM_BUG_ON(engine->id != RCS0);
/*
* The kernel context(s) is treated as pure scratch and is not
err_mm:
if (unwind_mm)
- ppgtt->pd_dirty_rings |= unwind_mm;
+ ppgtt->pd_dirty_engines |= unwind_mm;
err:
return ret;
}
{
int ret;
- GEM_BUG_ON(!request->hw_context->pin_count);
+ GEM_BUG_ON(!intel_context_is_pinned(request->hw_context));
GEM_BUG_ON(request->timeline->has_initial_breadcrumb);
/*
static void gen6_bsd_submit_request(struct i915_request *request)
{
- struct drm_i915_private *dev_priv = request->i915;
+ struct intel_uncore *uncore = request->engine->uncore;
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
/* Every tail move must follow the sequence below */
/* Disable notification that the ring is IDLE. The GT
* will then assume that it is busy and bring it out of rc6.
*/
- I915_WRITE_FW(GEN6_BSD_SLEEP_PSMI_CONTROL,
- _MASKED_BIT_ENABLE(GEN6_BSD_SLEEP_MSG_DISABLE));
+ intel_uncore_write_fw(uncore, GEN6_BSD_SLEEP_PSMI_CONTROL,
+ _MASKED_BIT_ENABLE(GEN6_BSD_SLEEP_MSG_DISABLE));
/* Clear the context id. Here be magic! */
- I915_WRITE64_FW(GEN6_BSD_RNCID, 0x0);
+ intel_uncore_write64_fw(uncore, GEN6_BSD_RNCID, 0x0);
/* Wait for the ring not to be idle, i.e. for it to wake up. */
- if (__intel_wait_for_register_fw(dev_priv,
+ if (__intel_wait_for_register_fw(uncore,
GEN6_BSD_SLEEP_PSMI_CONTROL,
GEN6_BSD_SLEEP_INDICATOR,
0,
/* Let the ring send IDLE messages to the GT again,
* and so let it sleep to conserve power when idle.
*/
- I915_WRITE_FW(GEN6_BSD_SLEEP_PSMI_CONTROL,
- _MASKED_BIT_DISABLE(GEN6_BSD_SLEEP_MSG_DISABLE));
+ intel_uncore_write_fw(uncore, GEN6_BSD_SLEEP_PSMI_CONTROL,
+ _MASKED_BIT_DISABLE(GEN6_BSD_SLEEP_MSG_DISABLE));
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(uncore, FORCEWAKE_ALL);
}
static int mi_flush_dw(struct i915_request *rq, u32 flags)
engine->reset.reset = reset_ring;
engine->reset.finish = reset_finish;
- engine->context_pin = intel_ring_context_pin;
+ engine->cops = &ring_context_ops;
engine->request_alloc = ring_request_alloc;
/*
#include <linux/hashtable.h>
#include <linux/irq_work.h>
+#include <linux/random.h>
#include <linux/seqlock.h>
#include "i915_gem_batch_pool.h"
-
-#include "i915_reg.h"
#include "i915_pmu.h"
+#include "i915_reg.h"
#include "i915_request.h"
#include "i915_selftest.h"
#include "i915_timeline.h"
+#include "intel_engine_types.h"
#include "intel_gpu_commands.h"
#include "intel_workarounds.h"
struct drm_printer;
-struct i915_sched_attr;
-
-#define I915_CMD_HASH_ORDER 9
/* Early gen2 devices have a cacheline of just 32 bytes, using 64 is overkill,
* but keeps the logic simple. Indeed, the whole purpose of this macro is just
#define CACHELINE_BYTES 64
#define CACHELINE_DWORDS (CACHELINE_BYTES / sizeof(u32))
-struct intel_hw_status_page {
- struct i915_vma *vma;
- u32 *addr;
-};
+/*
+ * The register defines to be used with the following macros need to accept a
+ * base param, e.g:
+ *
+ * REG_FOO(base) _MMIO((base) + <relative offset>)
+ * ENGINE_READ(engine, REG_FOO);
+ *
+ * register arrays are to be defined and accessed as follows:
+ *
+ * REG_BAR(base, i) _MMIO((base) + <relative offset> + (i) * <shift>)
+ * ENGINE_READ_IDX(engine, REG_BAR, i)
+ */
+
+#define __ENGINE_REG_OP(op__, engine__, ...) \
+ intel_uncore_##op__((engine__)->uncore, __VA_ARGS__)
-#define I915_READ_TAIL(engine) I915_READ(RING_TAIL((engine)->mmio_base))
-#define I915_WRITE_TAIL(engine, val) I915_WRITE(RING_TAIL((engine)->mmio_base), val)
+#define __ENGINE_READ_OP(op__, engine__, reg__) \
+ __ENGINE_REG_OP(op__, (engine__), reg__((engine__)->mmio_base))
-#define I915_READ_START(engine) I915_READ(RING_START((engine)->mmio_base))
-#define I915_WRITE_START(engine, val) I915_WRITE(RING_START((engine)->mmio_base), val)
+#define ENGINE_READ16(...) __ENGINE_READ_OP(read16, __VA_ARGS__)
+#define ENGINE_READ(...) __ENGINE_READ_OP(read, __VA_ARGS__)
+#define ENGINE_READ_FW(...) __ENGINE_READ_OP(read_fw, __VA_ARGS__)
+#define ENGINE_POSTING_READ(...) __ENGINE_READ_OP(posting_read, __VA_ARGS__)
-#define I915_READ_HEAD(engine) I915_READ(RING_HEAD((engine)->mmio_base))
-#define I915_WRITE_HEAD(engine, val) I915_WRITE(RING_HEAD((engine)->mmio_base), val)
+#define ENGINE_READ64(engine__, lower_reg__, upper_reg__) \
+ __ENGINE_REG_OP(read64_2x32, (engine__), \
+ lower_reg__((engine__)->mmio_base), \
+ upper_reg__((engine__)->mmio_base))
-#define I915_READ_CTL(engine) I915_READ(RING_CTL((engine)->mmio_base))
-#define I915_WRITE_CTL(engine, val) I915_WRITE(RING_CTL((engine)->mmio_base), val)
+#define ENGINE_READ_IDX(engine__, reg__, idx__) \
+ __ENGINE_REG_OP(read, (engine__), reg__((engine__)->mmio_base, (idx__)))
-#define I915_READ_IMR(engine) I915_READ(RING_IMR((engine)->mmio_base))
-#define I915_WRITE_IMR(engine, val) I915_WRITE(RING_IMR((engine)->mmio_base), val)
+#define __ENGINE_WRITE_OP(op__, engine__, reg__, val__) \
+ __ENGINE_REG_OP(op__, (engine__), reg__((engine__)->mmio_base), (val__))
-#define I915_READ_MODE(engine) I915_READ(RING_MI_MODE((engine)->mmio_base))
-#define I915_WRITE_MODE(engine, val) I915_WRITE(RING_MI_MODE((engine)->mmio_base), val)
+#define ENGINE_WRITE16(...) __ENGINE_WRITE_OP(write16, __VA_ARGS__)
+#define ENGINE_WRITE(...) __ENGINE_WRITE_OP(write, __VA_ARGS__)
+#define ENGINE_WRITE_FW(...) __ENGINE_WRITE_OP(write_fw, __VA_ARGS__)
/* seqno size is actually only a uint32, but since we plan to use MI_FLUSH_DW to
* do the writes, and that must have qw aligned offsets, simply pretend it's 8b.
return "unknown";
}
-#define I915_MAX_SLICES 3
-#define I915_MAX_SUBSLICES 8
-
-#define instdone_slice_mask(dev_priv__) \
- (IS_GEN(dev_priv__, 7) ? \
- 1 : RUNTIME_INFO(dev_priv__)->sseu.slice_mask)
-
-#define instdone_subslice_mask(dev_priv__) \
- (IS_GEN(dev_priv__, 7) ? \
- 1 : RUNTIME_INFO(dev_priv__)->sseu.subslice_mask[0])
-
-#define for_each_instdone_slice_subslice(dev_priv__, slice__, subslice__) \
- for ((slice__) = 0, (subslice__) = 0; \
- (slice__) < I915_MAX_SLICES; \
- (subslice__) = ((subslice__) + 1) < I915_MAX_SUBSLICES ? (subslice__) + 1 : 0, \
- (slice__) += ((subslice__) == 0)) \
- for_each_if((BIT(slice__) & instdone_slice_mask(dev_priv__)) && \
- (BIT(subslice__) & instdone_subslice_mask(dev_priv__)))
-
-struct intel_instdone {
- u32 instdone;
- /* The following exist only in the RCS engine */
- u32 slice_common;
- u32 sampler[I915_MAX_SLICES][I915_MAX_SUBSLICES];
- u32 row[I915_MAX_SLICES][I915_MAX_SUBSLICES];
-};
-
-struct intel_engine_hangcheck {
- u64 acthd;
- u32 seqno;
- unsigned long action_timestamp;
- struct intel_instdone instdone;
-};
-
-struct intel_ring {
- struct i915_vma *vma;
- void *vaddr;
-
- struct i915_timeline *timeline;
- struct list_head request_list;
- struct list_head active_link;
-
- u32 head;
- u32 tail;
- u32 emit;
-
- u32 space;
- u32 size;
- u32 effective_size;
-};
-
-struct i915_gem_context;
-struct drm_i915_reg_table;
-
-/*
- * we use a single page to load ctx workarounds so all of these
- * values are referred in terms of dwords
- *
- * struct i915_wa_ctx_bb:
- * offset: specifies batch starting position, also helpful in case
- * if we want to have multiple batches at different offsets based on
- * some criteria. It is not a requirement at the moment but provides
- * an option for future use.
- * size: size of the batch in DWORDS
- */
-struct i915_ctx_workarounds {
- struct i915_wa_ctx_bb {
- u32 offset;
- u32 size;
- } indirect_ctx, per_ctx;
- struct i915_vma *vma;
-};
-
-struct i915_request;
-
-#define I915_MAX_VCS 4
-#define I915_MAX_VECS 2
-
-/*
- * Engine IDs definitions.
- * Keep instances of the same type engine together.
- */
-enum intel_engine_id {
- RCS = 0,
- BCS,
- VCS,
- VCS2,
- VCS3,
- VCS4,
-#define _VCS(n) (VCS + (n))
- VECS,
- VECS2
-#define _VECS(n) (VECS + (n))
-};
-
-struct i915_priolist {
- struct list_head requests[I915_PRIORITY_COUNT];
- struct rb_node node;
- unsigned long used;
- int priority;
-};
-
-#define priolist_for_each_request(it, plist, idx) \
- for (idx = 0; idx < ARRAY_SIZE((plist)->requests); idx++) \
- list_for_each_entry(it, &(plist)->requests[idx], sched.link)
-
-#define priolist_for_each_request_consume(it, n, plist, idx) \
- for (; (idx = ffs((plist)->used)); (plist)->used &= ~BIT(idx - 1)) \
- list_for_each_entry_safe(it, n, \
- &(plist)->requests[idx - 1], \
- sched.link)
-
-struct st_preempt_hang {
- struct completion completion;
- unsigned int count;
- bool inject_hang;
-};
-
-/**
- * struct intel_engine_execlists - execlist submission queue and port state
- *
- * The struct intel_engine_execlists represents the combined logical state of
- * driver and the hardware state for execlist mode of submission.
- */
-struct intel_engine_execlists {
- /**
- * @tasklet: softirq tasklet for bottom handler
- */
- struct tasklet_struct tasklet;
-
- /**
- * @default_priolist: priority list for I915_PRIORITY_NORMAL
- */
- struct i915_priolist default_priolist;
-
- /**
- * @no_priolist: priority lists disabled
- */
- bool no_priolist;
-
- /**
- * @submit_reg: gen-specific execlist submission register
- * set to the ExecList Submission Port (elsp) register pre-Gen11 and to
- * the ExecList Submission Queue Contents register array for Gen11+
- */
- u32 __iomem *submit_reg;
-
- /**
- * @ctrl_reg: the enhanced execlists control register, used to load the
- * submit queue on the HW and to request preemptions to idle
- */
- u32 __iomem *ctrl_reg;
-
- /**
- * @port: execlist port states
- *
- * For each hardware ELSP (ExecList Submission Port) we keep
- * track of the last request and the number of times we submitted
- * that port to hw. We then count the number of times the hw reports
- * a context completion or preemption. As only one context can
- * be active on hw, we limit resubmission of context to port[0]. This
- * is called Lite Restore, of the context.
- */
- struct execlist_port {
- /**
- * @request_count: combined request and submission count
- */
- struct i915_request *request_count;
-#define EXECLIST_COUNT_BITS 2
-#define port_request(p) ptr_mask_bits((p)->request_count, EXECLIST_COUNT_BITS)
-#define port_count(p) ptr_unmask_bits((p)->request_count, EXECLIST_COUNT_BITS)
-#define port_pack(rq, count) ptr_pack_bits(rq, count, EXECLIST_COUNT_BITS)
-#define port_unpack(p, count) ptr_unpack_bits((p)->request_count, count, EXECLIST_COUNT_BITS)
-#define port_set(p, packed) ((p)->request_count = (packed))
-#define port_isset(p) ((p)->request_count)
-#define port_index(p, execlists) ((p) - (execlists)->port)
-
- /**
- * @context_id: context ID for port
- */
- GEM_DEBUG_DECL(u32 context_id);
-
-#define EXECLIST_MAX_PORTS 2
- } port[EXECLIST_MAX_PORTS];
-
- /**
- * @active: is the HW active? We consider the HW as active after
- * submitting any context for execution and until we have seen the
- * last context completion event. After that, we do not expect any
- * more events until we submit, and so can park the HW.
- *
- * As we have a small number of different sources from which we feed
- * the HW, we track the state of each inside a single bitfield.
- */
- unsigned int active;
-#define EXECLISTS_ACTIVE_USER 0
-#define EXECLISTS_ACTIVE_PREEMPT 1
-#define EXECLISTS_ACTIVE_HWACK 2
-
- /**
- * @port_mask: number of execlist ports - 1
- */
- unsigned int port_mask;
-
- /**
- * @queue_priority_hint: Highest pending priority.
- *
- * When we add requests into the queue, or adjust the priority of
- * executing requests, we compute the maximum priority of those
- * pending requests. We can then use this value to determine if
- * we need to preempt the executing requests to service the queue.
- * However, since the we may have recorded the priority of an inflight
- * request we wanted to preempt but since completed, at the time of
- * dequeuing the priority hint may no longer may match the highest
- * available request priority.
- */
- int queue_priority_hint;
-
- /**
- * @queue: queue of requests, in priority lists
- */
- struct rb_root_cached queue;
-
- /**
- * @csb_write: control register for Context Switch buffer
- *
- * Note this register may be either mmio or HWSP shadow.
- */
- u32 *csb_write;
-
- /**
- * @csb_status: status array for Context Switch buffer
- *
- * Note these register may be either mmio or HWSP shadow.
- */
- u32 *csb_status;
-
- /**
- * @preempt_complete_status: expected CSB upon completing preemption
- */
- u32 preempt_complete_status;
-
- /**
- * @csb_head: context status buffer head
- */
- u8 csb_head;
-
- I915_SELFTEST_DECLARE(struct st_preempt_hang preempt_hang;)
-};
-
-#define INTEL_ENGINE_CS_MAX_NAME 8
-
-struct intel_engine_cs {
- struct drm_i915_private *i915;
- char name[INTEL_ENGINE_CS_MAX_NAME];
-
- enum intel_engine_id id;
- unsigned int hw_id;
- unsigned int guc_id;
-
- u8 uabi_id;
- u8 uabi_class;
-
- u8 class;
- u8 instance;
- u32 context_size;
- u32 mmio_base;
-
- struct intel_ring *buffer;
-
- struct i915_timeline timeline;
-
- struct drm_i915_gem_object *default_state;
- void *pinned_default_state;
-
- /* Rather than have every client wait upon all user interrupts,
- * with the herd waking after every interrupt and each doing the
- * heavyweight seqno dance, we delegate the task (of being the
- * bottom-half of the user interrupt) to the first client. After
- * every interrupt, we wake up one client, who does the heavyweight
- * coherent seqno read and either goes back to sleep (if incomplete),
- * or wakes up all the completed clients in parallel, before then
- * transferring the bottom-half status to the next client in the queue.
- *
- * Compared to walking the entire list of waiters in a single dedicated
- * bottom-half, we reduce the latency of the first waiter by avoiding
- * a context switch, but incur additional coherent seqno reads when
- * following the chain of request breadcrumbs. Since it is most likely
- * that we have a single client waiting on each seqno, then reducing
- * the overhead of waking that client is much preferred.
- */
- struct intel_breadcrumbs {
- spinlock_t irq_lock;
- struct list_head signalers;
-
- struct irq_work irq_work; /* for use from inside irq_lock */
-
- unsigned int irq_enabled;
-
- bool irq_armed;
- } breadcrumbs;
-
- struct {
- /**
- * @enable: Bitmask of enable sample events on this engine.
- *
- * Bits correspond to sample event types, for instance
- * I915_SAMPLE_QUEUED is bit 0 etc.
- */
- u32 enable;
- /**
- * @enable_count: Reference count for the enabled samplers.
- *
- * Index number corresponds to @enum drm_i915_pmu_engine_sample.
- */
- unsigned int enable_count[I915_ENGINE_SAMPLE_COUNT];
- /**
- * @sample: Counter values for sampling events.
- *
- * Our internal timer stores the current counters in this field.
- *
- * Index number corresponds to @enum drm_i915_pmu_engine_sample.
- */
- struct i915_pmu_sample sample[I915_ENGINE_SAMPLE_COUNT];
- } pmu;
-
- /*
- * A pool of objects to use as shadow copies of client batch buffers
- * when the command parser is enabled. Prevents the client from
- * modifying the batch contents after software parsing.
- */
- struct i915_gem_batch_pool batch_pool;
-
- struct intel_hw_status_page status_page;
- struct i915_ctx_workarounds wa_ctx;
- struct i915_wa_list ctx_wa_list;
- struct i915_wa_list wa_list;
- struct i915_wa_list whitelist;
-
- u32 irq_keep_mask; /* always keep these interrupts */
- u32 irq_enable_mask; /* bitmask to enable ring interrupt */
- void (*irq_enable)(struct intel_engine_cs *engine);
- void (*irq_disable)(struct intel_engine_cs *engine);
-
- int (*init_hw)(struct intel_engine_cs *engine);
-
- struct {
- void (*prepare)(struct intel_engine_cs *engine);
- void (*reset)(struct intel_engine_cs *engine, bool stalled);
- void (*finish)(struct intel_engine_cs *engine);
- } reset;
-
- void (*park)(struct intel_engine_cs *engine);
- void (*unpark)(struct intel_engine_cs *engine);
-
- void (*set_default_submission)(struct intel_engine_cs *engine);
-
- struct intel_context *(*context_pin)(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx);
-
- int (*request_alloc)(struct i915_request *rq);
- int (*init_context)(struct i915_request *rq);
-
- int (*emit_flush)(struct i915_request *request, u32 mode);
-#define EMIT_INVALIDATE BIT(0)
-#define EMIT_FLUSH BIT(1)
-#define EMIT_BARRIER (EMIT_INVALIDATE | EMIT_FLUSH)
- int (*emit_bb_start)(struct i915_request *rq,
- u64 offset, u32 length,
- unsigned int dispatch_flags);
-#define I915_DISPATCH_SECURE BIT(0)
-#define I915_DISPATCH_PINNED BIT(1)
- int (*emit_init_breadcrumb)(struct i915_request *rq);
- u32 *(*emit_fini_breadcrumb)(struct i915_request *rq,
- u32 *cs);
- unsigned int emit_fini_breadcrumb_dw;
-
- /* Pass the request to the hardware queue (e.g. directly into
- * the legacy ringbuffer or to the end of an execlist).
- *
- * This is called from an atomic context with irqs disabled; must
- * be irq safe.
- */
- void (*submit_request)(struct i915_request *rq);
-
- /*
- * Call when the priority on a request has changed and it and its
- * dependencies may need rescheduling. Note the request itself may
- * not be ready to run!
- */
- void (*schedule)(struct i915_request *request,
- const struct i915_sched_attr *attr);
-
- /*
- * Cancel all requests on the hardware, or queued for execution.
- * This should only cancel the ready requests that have been
- * submitted to the engine (via the engine->submit_request callback).
- * This is called when marking the device as wedged.
- */
- void (*cancel_requests)(struct intel_engine_cs *engine);
-
- void (*cleanup)(struct intel_engine_cs *engine);
-
- struct intel_engine_execlists execlists;
-
- /* Contexts are pinned whilst they are active on the GPU. The last
- * context executed remains active whilst the GPU is idle - the
- * switch away and write to the context object only occurs on the
- * next execution. Contexts are only unpinned on retirement of the
- * following request ensuring that we can always write to the object
- * on the context switch even after idling. Across suspend, we switch
- * to the kernel context and trash it as the save may not happen
- * before the hardware is powered down.
- */
- struct intel_context *last_retired_context;
-
- /* status_notifier: list of callbacks for context-switch changes */
- struct atomic_notifier_head context_status_notifier;
-
- struct intel_engine_hangcheck hangcheck;
-
-#define I915_ENGINE_NEEDS_CMD_PARSER BIT(0)
-#define I915_ENGINE_SUPPORTS_STATS BIT(1)
-#define I915_ENGINE_HAS_PREEMPTION BIT(2)
- unsigned int flags;
-
- /*
- * Table of commands the command parser needs to know about
- * for this engine.
- */
- DECLARE_HASHTABLE(cmd_hash, I915_CMD_HASH_ORDER);
-
- /*
- * Table of registers allowed in commands that read/write registers.
- */
- const struct drm_i915_reg_table *reg_tables;
- int reg_table_count;
-
- /*
- * Returns the bitmask for the length field of the specified command.
- * Return 0 for an unrecognized/invalid command.
- *
- * If the command parser finds an entry for a command in the engine's
- * cmd_tables, it gets the command's length based on the table entry.
- * If not, it calls this function to determine the per-engine length
- * field encoding for the command (i.e. different opcode ranges use
- * certain bits to encode the command length in the header).
- */
- u32 (*get_cmd_length_mask)(u32 cmd_header);
-
- struct {
- /**
- * @lock: Lock protecting the below fields.
- */
- seqlock_t lock;
- /**
- * @enabled: Reference count indicating number of listeners.
- */
- unsigned int enabled;
- /**
- * @active: Number of contexts currently scheduled in.
- */
- unsigned int active;
- /**
- * @enabled_at: Timestamp when busy stats were enabled.
- */
- ktime_t enabled_at;
- /**
- * @start: Timestamp of the last idle to active transition.
- *
- * Idle is defined as active == 0, active is active > 0.
- */
- ktime_t start;
- /**
- * @total: Total time this engine was busy.
- *
- * Accumulated time not counting the most recent block in cases
- * where engine is currently busy (active > 0).
- */
- ktime_t total;
- } stats;
-};
-
-static inline bool
-intel_engine_needs_cmd_parser(const struct intel_engine_cs *engine)
-{
- return engine->flags & I915_ENGINE_NEEDS_CMD_PARSER;
-}
-
-static inline bool
-intel_engine_supports_stats(const struct intel_engine_cs *engine)
-{
- return engine->flags & I915_ENGINE_SUPPORTS_STATS;
-}
-
-static inline bool
-intel_engine_has_preemption(const struct intel_engine_cs *engine)
-{
- return engine->flags & I915_ENGINE_HAS_PREEMPTION;
-}
+void intel_engines_set_scheduler_caps(struct drm_i915_private *i915);
static inline bool __execlists_need_preempt(int prio, int last)
{
void
execlists_cancel_port_requests(struct intel_engine_execlists * const execlists);
-void
+struct i915_request *
execlists_unwind_incomplete_requests(struct intel_engine_execlists *execlists);
static inline unsigned int
return port;
}
-static inline unsigned int
-intel_engine_flag(const struct intel_engine_cs *engine)
-{
- return BIT(engine->id);
-}
-
static inline u32
intel_read_status_page(const struct intel_engine_cs *engine, int reg)
{
*
* The area from dword 0x30 to 0x3ff is available for driver usage.
*/
-#define I915_GEM_HWS_INDEX 0x30
-#define I915_GEM_HWS_INDEX_ADDR (I915_GEM_HWS_INDEX * sizeof(u32))
#define I915_GEM_HWS_PREEMPT 0x32
#define I915_GEM_HWS_PREEMPT_ADDR (I915_GEM_HWS_PREEMPT * sizeof(u32))
+#define I915_GEM_HWS_HANGCHECK 0x34
+#define I915_GEM_HWS_HANGCHECK_ADDR (I915_GEM_HWS_HANGCHECK * sizeof(u32))
#define I915_GEM_HWS_SEQNO 0x40
#define I915_GEM_HWS_SEQNO_ADDR (I915_GEM_HWS_SEQNO * sizeof(u32))
#define I915_GEM_HWS_SCRATCH 0x80
void intel_ring_reset(struct intel_ring *ring, u32 tail);
unsigned int intel_ring_update_space(struct intel_ring *ring);
void intel_ring_unpin(struct intel_ring *ring);
-void intel_ring_free(struct intel_ring *ring);
+void intel_ring_free(struct kref *ref);
+
+static inline struct intel_ring *intel_ring_get(struct intel_ring *ring)
+{
+ kref_get(&ring->ref);
+ return ring;
+}
+
+static inline void intel_ring_put(struct intel_ring *ring)
+{
+ kref_put(&ring->ref, intel_ring_free);
+}
void intel_engine_stop(struct intel_engine_cs *engine);
void intel_engine_cleanup(struct intel_engine_cs *engine);
-void intel_legacy_submission_resume(struct drm_i915_private *dev_priv);
-
int __must_check intel_ring_cacheline_align(struct i915_request *rq);
u32 __must_check *intel_ring_begin(struct i915_request *rq, unsigned int n);
return (head - tail - CACHELINE_BYTES) & (size - 1);
}
-void intel_engine_write_global_seqno(struct intel_engine_cs *engine, u32 seqno);
-
int intel_engine_setup_common(struct intel_engine_cs *engine);
int intel_engine_init_common(struct intel_engine_cs *engine);
void intel_engine_cleanup_common(struct intel_engine_cs *engine);
u64 intel_engine_get_active_head(const struct intel_engine_cs *engine);
u64 intel_engine_get_last_batch_head(const struct intel_engine_cs *engine);
-static inline u32 intel_engine_last_submit(struct intel_engine_cs *engine)
-{
- /*
- * We are only peeking at the tail of the submit queue (and not the
- * queue itself) in order to gain a hint as to the current active
- * state of the engine. Callers are not expected to be taking
- * engine->timeline->lock, nor are they expected to be concerned
- * wtih serialising this hint with anything, so document it as
- * a hint and nothing more.
- */
- return READ_ONCE(engine->timeline.seqno);
-}
-
-static inline u32 intel_engine_get_seqno(struct intel_engine_cs *engine)
-{
- return intel_read_status_page(engine, I915_GEM_HWS_INDEX);
-}
-
-static inline bool intel_engine_signaled(struct intel_engine_cs *engine,
- u32 seqno)
-{
- return i915_seqno_passed(intel_engine_get_seqno(engine), seqno);
-}
-
-static inline bool intel_engine_has_completed(struct intel_engine_cs *engine,
- u32 seqno)
-{
- GEM_BUG_ON(!seqno);
- return intel_engine_signaled(engine, seqno);
-}
-
-static inline bool intel_engine_has_started(struct intel_engine_cs *engine,
- u32 seqno)
-{
- GEM_BUG_ON(!seqno);
- return intel_engine_signaled(engine, seqno - 1);
-}
-
void intel_engine_get_instdone(struct intel_engine_cs *engine,
struct intel_instdone *instdone);
void intel_engine_pin_breadcrumbs_irq(struct intel_engine_cs *engine);
void intel_engine_unpin_breadcrumbs_irq(struct intel_engine_cs *engine);
-bool intel_engine_signal_breadcrumbs(struct intel_engine_cs *engine);
+void intel_engine_signal_breadcrumbs(struct intel_engine_cs *engine);
void intel_engine_disarm_breadcrumbs(struct intel_engine_cs *engine);
static inline void
irq_work_queue(&engine->breadcrumbs.irq_work);
}
-bool intel_engine_breadcrumbs_irq(struct intel_engine_cs *engine);
+void intel_engine_breadcrumbs_irq(struct intel_engine_cs *engine);
void intel_engine_reset_breadcrumbs(struct intel_engine_cs *engine);
void intel_engine_fini_breadcrumbs(struct intel_engine_cs *engine);
}
static inline u32 *
-gen8_emit_ggtt_write(u32 *cs, u32 value, u32 gtt_offset)
+gen8_emit_ggtt_write(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
{
/* w/a: bit 5 needs to be zero for MI_FLUSH_DW address. */
GEM_BUG_ON(gtt_offset & (1 << 5));
/* Offset should be aligned to 8 bytes for both (QW/DW) write types */
GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
- *cs++ = (MI_FLUSH_DW + 1) | MI_FLUSH_DW_OP_STOREDW;
+ *cs++ = (MI_FLUSH_DW + 1) | MI_FLUSH_DW_OP_STOREDW | flags;
*cs++ = gtt_offset | MI_FLUSH_DW_USE_GTT;
*cs++ = 0;
*cs++ = value;
}
void intel_engines_sanitize(struct drm_i915_private *i915, bool force);
+void intel_gt_resume(struct drm_i915_private *i915);
bool intel_engine_is_idle(struct intel_engine_cs *engine);
bool intel_engines_are_idle(struct drm_i915_private *dev_priv);
-bool intel_engine_has_kernel_context(const struct intel_engine_cs *engine);
void intel_engine_lost_context(struct intel_engine_cs *engine);
void intel_engines_park(struct drm_i915_private *i915);
ktime_t intel_engine_get_busy_time(struct intel_engine_cs *engine);
+struct i915_request *
+intel_engine_find_active_request(struct intel_engine_cs *engine);
+
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
static inline bool inject_preempt_hang(struct intel_engine_execlists *execlists)
#endif
+static inline u32
+intel_engine_next_hangcheck_seqno(struct intel_engine_cs *engine)
+{
+ return engine->hangcheck.next_seqno =
+ next_pseudo_random32(engine->hangcheck.next_seqno);
+}
+
+static inline u32
+intel_engine_get_hangcheck_seqno(struct intel_engine_cs *engine)
+{
+ return intel_read_status_page(engine, I915_GEM_HWS_HANGCHECK);
+}
+
#endif /* _INTEL_RINGBUFFER_H_ */
#include <drm/drm_print.h>
#include "i915_drv.h"
+#include "intel_cdclk.h"
+#include "intel_crt.h"
+#include "intel_csr.h"
+#include "intel_dp.h"
#include "intel_drv.h"
/**
rpm->debug.count, atomic_read(&rpm->wakeref_count))) {
char *buf;
- buf = kmalloc(PAGE_SIZE, GFP_KERNEL);
+ buf = kmalloc(PAGE_SIZE, GFP_NOWAIT | __GFP_NOWARN);
if (!buf)
return;
unsigned long i;
char *buf;
- buf = kmalloc(PAGE_SIZE, GFP_KERNEL);
+ buf = kmalloc(PAGE_SIZE, GFP_NOWAIT | __GFP_NOWARN);
if (!buf)
return;
if (dbg.count <= alloc)
break;
- s = krealloc(dbg.owners, dbg.count * sizeof(*s), GFP_KERNEL);
+ s = krealloc(dbg.owners,
+ dbg.count * sizeof(*s),
+ GFP_NOWAIT | __GFP_NOWARN);
if (!s)
goto out;
int pw_idx = power_well->desc->hsw.idx;
/* Timeout for PW1:10 us, AUX:not specified, other PWs:20 us. */
- WARN_ON(intel_wait_for_register(dev_priv,
+ WARN_ON(intel_wait_for_register(&dev_priv->uncore,
regs->driver,
HSW_PWR_WELL_CTL_STATE(pw_idx),
HSW_PWR_WELL_CTL_STATE(pw_idx),
enum skl_power_gate pg)
{
/* Timeout 5us for PG#0, for other PGs 1us */
- WARN_ON(intel_wait_for_register(dev_priv, SKL_FUSE_STATUS,
+ WARN_ON(intel_wait_for_register(&dev_priv->uncore, SKL_FUSE_STATUS,
SKL_FUSE_PG_DIST_STATUS(pg),
SKL_FUSE_PG_DIST_STATUS(pg), 1));
}
* The PHY may be busy with some initial calibration and whatnot,
* so the power state can take a while to actually change.
*/
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
DISPLAY_PHY_STATUS,
phy_status_mask,
phy_status,
vlv_set_power_well(dev_priv, power_well, true);
/* Poll for phypwrgood signal */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
DISPLAY_PHY_STATUS,
PHY_POWERGOOD(phy),
PHY_POWERGOOD(phy),
mutex_lock(&dev_priv->pcu_lock);
- state = vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ) & DP_SSS_MASK(pipe);
+ state = vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM) & DP_SSS_MASK(pipe);
/*
* We only ever set the power-on and power-gate states, anything
* else is unexpected.
* A transient state at this point would mean some unexpected party
* is poking at the power controls too.
*/
- ctrl = vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ) & DP_SSC_MASK(pipe);
+ ctrl = vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM) & DP_SSC_MASK(pipe);
WARN_ON(ctrl << 16 != state);
mutex_unlock(&dev_priv->pcu_lock);
mutex_lock(&dev_priv->pcu_lock);
#define COND \
- ((vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ) & DP_SSS_MASK(pipe)) == state)
+ ((vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM) & DP_SSS_MASK(pipe)) == state)
if (COND)
goto out;
- ctrl = vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ);
+ ctrl = vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM);
ctrl &= ~DP_SSC_MASK(pipe);
ctrl |= enable ? DP_SSC_PWR_ON(pipe) : DP_SSC_PWR_GATE(pipe);
- vlv_punit_write(dev_priv, PUNIT_REG_DSPFREQ, ctrl);
+ vlv_punit_write(dev_priv, PUNIT_REG_DSPSSPM, ctrl);
if (wait_for(COND, 100))
DRM_ERROR("timeout setting power well state %08x (%08x)\n",
state,
- vlv_punit_read(dev_priv, PUNIT_REG_DSPFREQ));
+ vlv_punit_read(dev_priv, PUNIT_REG_DSPSSPM));
#undef COND
* The enabling order will be from lower to higher indexed wells,
* the disabling order is reversed.
*/
- if (IS_ICELAKE(dev_priv)) {
+ if (IS_GEN(dev_priv, 11)) {
err = set_power_wells(power_domains, icl_power_wells);
} else if (IS_CANNONLAKE(dev_priv)) {
err = set_power_wells(power_domains, cnl_power_wells);
!(I915_READ(DBUF_CTL_S2) & DBUF_POWER_STATE))
DRM_ERROR("DBuf power enable timeout\n");
else
- dev_priv->wm.skl_hw.ddb.enabled_slices = 2;
+ /*
+ * FIXME: for now pretend that we only have 1 slice, see
+ * intel_enabled_dbuf_slices_num().
+ */
+ dev_priv->wm.skl_hw.ddb.enabled_slices = 1;
}
static void icl_dbuf_disable(struct drm_i915_private *dev_priv)
(I915_READ(DBUF_CTL_S2) & DBUF_POWER_STATE))
DRM_ERROR("DBuf power disable timeout!\n");
else
- dev_priv->wm.skl_hw.ddb.enabled_slices = 0;
+ /*
+ * FIXME: for now pretend that the first slice is always
+ * enabled, see intel_enabled_dbuf_slices_num().
+ */
+ dev_priv->wm.skl_hw.ddb.enabled_slices = 1;
}
static void icl_mbus_init(struct drm_i915_private *dev_priv)
mutex_unlock(&power_domains->lock);
- skl_init_cdclk(dev_priv);
+ intel_cdclk_init(dev_priv);
gen9_dbuf_enable(dev_priv);
gen9_dbuf_disable(dev_priv);
- skl_uninit_cdclk(dev_priv);
+ intel_cdclk_uninit(dev_priv);
/* The spec doesn't call for removing the reset handshake flag */
/* disable PG1 and Misc I/O */
mutex_unlock(&power_domains->lock);
- bxt_init_cdclk(dev_priv);
+ intel_cdclk_init(dev_priv);
gen9_dbuf_enable(dev_priv);
gen9_dbuf_disable(dev_priv);
- bxt_uninit_cdclk(dev_priv);
+ intel_cdclk_uninit(dev_priv);
/* The spec doesn't call for removing the reset handshake flag */
mutex_unlock(&power_domains->lock);
/* 5. Enable CD clock */
- cnl_init_cdclk(dev_priv);
+ intel_cdclk_init(dev_priv);
/* 6. Enable DBUF */
gen9_dbuf_enable(dev_priv);
gen9_dbuf_disable(dev_priv);
/* 3. Disable CD clock */
- cnl_uninit_cdclk(dev_priv);
+ intel_cdclk_uninit(dev_priv);
/*
* 4. Disable Power Well 1 (PG1).
mutex_unlock(&power_domains->lock);
/* 5. Enable CDCLK. */
- icl_init_cdclk(dev_priv);
+ intel_cdclk_init(dev_priv);
/* 6. Enable DBUF. */
icl_dbuf_enable(dev_priv);
icl_dbuf_disable(dev_priv);
/* 3. Disable CD clock */
- icl_uninit_cdclk(dev_priv);
+ intel_cdclk_uninit(dev_priv);
/*
* 4. Disable Power Well 1 (PG1).
cmn->desc->ops->disable(dev_priv, cmn);
}
+static bool vlv_punit_is_power_gated(struct drm_i915_private *dev_priv, u32 reg0)
+{
+ bool ret;
+
+ mutex_lock(&dev_priv->pcu_lock);
+ ret = (vlv_punit_read(dev_priv, reg0) & SSPM0_SSC_MASK) == SSPM0_SSC_PWR_GATE;
+ mutex_unlock(&dev_priv->pcu_lock);
+
+ return ret;
+}
+
+static void assert_ved_power_gated(struct drm_i915_private *dev_priv)
+{
+ WARN(!vlv_punit_is_power_gated(dev_priv, PUNIT_REG_VEDSSPM0),
+ "VED not power gated\n");
+}
+
+static void assert_isp_power_gated(struct drm_i915_private *dev_priv)
+{
+ static const struct pci_device_id isp_ids[] = {
+ {PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x0f38)},
+ {PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x22b8)},
+ {}
+ };
+
+ WARN(!pci_dev_present(isp_ids) &&
+ !vlv_punit_is_power_gated(dev_priv, PUNIT_REG_ISPSSPM0),
+ "ISP not power gated\n");
+}
+
static void intel_power_domains_verify_state(struct drm_i915_private *dev_priv);
/**
power_domains->initializing = true;
- if (IS_ICELAKE(i915)) {
+ if (INTEL_GEN(i915) >= 11) {
icl_display_core_init(i915, resume);
} else if (IS_CANNONLAKE(i915)) {
cnl_display_core_init(i915, resume);
mutex_lock(&power_domains->lock);
chv_phy_control_init(i915);
mutex_unlock(&power_domains->lock);
+ assert_isp_power_gated(i915);
} else if (IS_VALLEYVIEW(i915)) {
mutex_lock(&power_domains->lock);
vlv_cmnlane_wa(i915);
mutex_unlock(&power_domains->lock);
+ assert_ved_power_gated(i915);
+ assert_isp_power_gated(i915);
} else if (IS_IVYBRIDGE(i915) || INTEL_GEN(i915) >= 7) {
intel_pch_reset_handshake(i915, !HAS_PCH_NOP(i915));
}
intel_power_domains_verify_state(i915);
}
- if (IS_ICELAKE(i915))
+ if (INTEL_GEN(i915) >= 11)
icl_display_core_uninit(i915);
else if (IS_CANNONLAKE(i915))
cnl_display_core_uninit(i915);
* Authors:
* Eric Anholt <eric@anholt.net>
*/
-#include <linux/i2c.h>
-#include <linux/slab.h>
+
#include <linux/delay.h>
#include <linux/export.h>
+#include <linux/i2c.h>
+#include <linux/slab.h>
+
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
#include <drm/drm_edid.h>
-#include "intel_drv.h"
#include <drm/i915_drm.h>
+
#include "i915_drv.h"
+#include "intel_connector.h"
+#include "intel_drv.h"
+#include "intel_hdmi.h"
+#include "intel_panel.h"
+#include "intel_sdvo.h"
#include "intel_sdvo_regs.h"
#define SDVO_TMDS_MASK (SDVO_OUTPUT_TMDS0 | SDVO_OUTPUT_TMDS1)
&tx_rate, 1);
}
-static bool intel_sdvo_set_avi_infoframe(struct intel_sdvo *intel_sdvo,
- const struct intel_crtc_state *pipe_config,
- const struct drm_connector_state *conn_state)
+static ssize_t intel_sdvo_read_infoframe(struct intel_sdvo *intel_sdvo,
+ unsigned int if_index,
+ u8 *data, unsigned int length)
{
+ u8 set_buf_index[2] = { if_index, 0 };
+ u8 hbuf_size, tx_rate, av_split;
+ int i;
+
+ if (!intel_sdvo_get_value(intel_sdvo,
+ SDVO_CMD_GET_HBUF_AV_SPLIT,
+ &av_split, 1))
+ return -ENXIO;
+
+ if (av_split < if_index)
+ return 0;
+
+ if (!intel_sdvo_get_value(intel_sdvo,
+ SDVO_CMD_GET_HBUF_TXRATE,
+ &tx_rate, 1))
+ return -ENXIO;
+
+ if (tx_rate == SDVO_HBUF_TX_DISABLED)
+ return 0;
+
+ if (!intel_sdvo_set_value(intel_sdvo,
+ SDVO_CMD_SET_HBUF_INDEX,
+ set_buf_index, 2))
+ return -ENXIO;
+
+ if (!intel_sdvo_get_value(intel_sdvo, SDVO_CMD_GET_HBUF_INFO,
+ &hbuf_size, 1))
+ return -ENXIO;
+
+ /* Buffer size is 0 based, hooray! */
+ hbuf_size++;
+
+ DRM_DEBUG_KMS("reading sdvo hbuf: %i, hbuf_size %i, hbuf_size: %i\n",
+ if_index, length, hbuf_size);
+
+ hbuf_size = min_t(unsigned int, length, hbuf_size);
+
+ for (i = 0; i < hbuf_size; i += 8) {
+ if (!intel_sdvo_write_cmd(intel_sdvo, SDVO_CMD_GET_HBUF_DATA, NULL, 0))
+ return -ENXIO;
+ if (!intel_sdvo_read_response(intel_sdvo, &data[i],
+ min_t(unsigned int, 8, hbuf_size - i)))
+ return -ENXIO;
+ }
+
+ return hbuf_size;
+}
+
+static bool intel_sdvo_compute_avi_infoframe(struct intel_sdvo *intel_sdvo,
+ struct intel_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state)
+{
+ struct hdmi_avi_infoframe *frame = &crtc_state->infoframes.avi.avi;
const struct drm_display_mode *adjusted_mode =
- &pipe_config->base.adjusted_mode;
- u8 sdvo_data[HDMI_INFOFRAME_SIZE(AVI)];
- union hdmi_infoframe frame;
+ &crtc_state->base.adjusted_mode;
int ret;
- ssize_t len;
- ret = drm_hdmi_avi_infoframe_from_display_mode(&frame.avi,
+ if (!crtc_state->has_hdmi_sink)
+ return true;
+
+ crtc_state->infoframes.enable |=
+ intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_AVI);
+
+ ret = drm_hdmi_avi_infoframe_from_display_mode(frame,
conn_state->connector,
adjusted_mode);
- if (ret < 0) {
- DRM_ERROR("couldn't fill AVI infoframe\n");
+ if (ret)
return false;
- }
- drm_hdmi_avi_infoframe_quant_range(&frame.avi,
+ drm_hdmi_avi_infoframe_quant_range(frame,
conn_state->connector,
adjusted_mode,
- pipe_config->limited_color_range ?
+ crtc_state->limited_color_range ?
HDMI_QUANTIZATION_RANGE_LIMITED :
HDMI_QUANTIZATION_RANGE_FULL);
- len = hdmi_infoframe_pack(&frame, sdvo_data, sizeof(sdvo_data));
- if (len < 0)
+ ret = hdmi_avi_infoframe_check(frame);
+ if (WARN_ON(ret))
+ return false;
+
+ return true;
+}
+
+static bool intel_sdvo_set_avi_infoframe(struct intel_sdvo *intel_sdvo,
+ const struct intel_crtc_state *crtc_state)
+{
+ u8 sdvo_data[HDMI_INFOFRAME_SIZE(AVI)];
+ const union hdmi_infoframe *frame = &crtc_state->infoframes.avi;
+ ssize_t len;
+
+ if ((crtc_state->infoframes.enable &
+ intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_AVI)) == 0)
+ return true;
+
+ if (WARN_ON(frame->any.type != HDMI_INFOFRAME_TYPE_AVI))
+ return false;
+
+ len = hdmi_infoframe_pack_only(frame, sdvo_data, sizeof(sdvo_data));
+ if (WARN_ON(len < 0))
return false;
return intel_sdvo_write_infoframe(intel_sdvo, SDVO_HBUF_INDEX_AVI_IF,
sdvo_data, sizeof(sdvo_data));
}
+static void intel_sdvo_get_avi_infoframe(struct intel_sdvo *intel_sdvo,
+ struct intel_crtc_state *crtc_state)
+{
+ u8 sdvo_data[HDMI_INFOFRAME_SIZE(AVI)];
+ union hdmi_infoframe *frame = &crtc_state->infoframes.avi;
+ ssize_t len;
+ int ret;
+
+ if (!crtc_state->has_hdmi_sink)
+ return;
+
+ len = intel_sdvo_read_infoframe(intel_sdvo, SDVO_HBUF_INDEX_AVI_IF,
+ sdvo_data, sizeof(sdvo_data));
+ if (len < 0) {
+ DRM_DEBUG_KMS("failed to read AVI infoframe\n");
+ return;
+ } else if (len == 0) {
+ return;
+ }
+
+ crtc_state->infoframes.enable |=
+ intel_hdmi_infoframe_enable(HDMI_INFOFRAME_TYPE_AVI);
+
+ ret = hdmi_infoframe_unpack(frame, sdvo_data, sizeof(sdvo_data));
+ if (ret) {
+ DRM_DEBUG_KMS("Failed to unpack AVI infoframe\n");
+ return;
+ }
+
+ if (frame->any.type != HDMI_INFOFRAME_TYPE_AVI)
+ DRM_DEBUG_KMS("Found the wrong infoframe type 0x%x (expected 0x%02x)\n",
+ frame->any.type, HDMI_INFOFRAME_TYPE_AVI);
+}
+
static bool intel_sdvo_set_tv_format(struct intel_sdvo *intel_sdvo,
const struct drm_connector_state *conn_state)
{
if (intel_sdvo_connector->is_hdmi)
adjusted_mode->picture_aspect_ratio = conn_state->picture_aspect_ratio;
+ if (!intel_sdvo_compute_avi_infoframe(intel_sdvo,
+ pipe_config, conn_state)) {
+ DRM_DEBUG_KMS("bad AVI infoframe\n");
+ return -EINVAL;
+ }
+
return 0;
}
intel_sdvo_set_encode(intel_sdvo, SDVO_ENCODE_HDMI);
intel_sdvo_set_colorimetry(intel_sdvo,
SDVO_COLORIMETRY_RGB256);
- intel_sdvo_set_avi_infoframe(intel_sdvo,
- crtc_state, conn_state);
+ intel_sdvo_set_avi_infoframe(intel_sdvo, crtc_state);
} else
intel_sdvo_set_encode(intel_sdvo, SDVO_ENCODE_DVI);
}
}
+ WARN(encoder_pixel_multiplier != pipe_config->pixel_multiplier,
+ "SDVO pixel multiplier mismatch, port: %i, encoder: %i\n",
+ pipe_config->pixel_multiplier, encoder_pixel_multiplier);
+
if (sdvox & HDMI_COLOR_RANGE_16_235)
pipe_config->limited_color_range = true;
pipe_config->has_hdmi_sink = true;
}
- WARN(encoder_pixel_multiplier != pipe_config->pixel_multiplier,
- "SDVO pixel multiplier mismatch, port: %i, encoder: %i\n",
- pipe_config->pixel_multiplier, encoder_pixel_multiplier);
+ intel_sdvo_get_avi_infoframe(intel_sdvo, pipe_config);
}
static void intel_disable_sdvo(struct intel_encoder *encoder,
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_SDVO_H__
+#define __INTEL_SDVO_H__
+
+#include <linux/types.h>
+
+#include <drm/i915_drm.h>
+
+#include "i915_reg.h"
+
+struct drm_i915_private;
+enum pipe;
+
+bool intel_sdvo_port_enabled(struct drm_i915_private *dev_priv,
+ i915_reg_t sdvo_reg, enum pipe *pipe);
+bool intel_sdvo_init(struct drm_i915_private *dev_priv,
+ i915_reg_t reg, enum port port);
+
+#endif /* __INTEL_SDVO_H__ */
WARN_ON(!mutex_is_locked(&dev_priv->sb_lock));
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
VLV_IOSF_DOORBELL_REQ, IOSF_SB_BUSY, 0,
5)) {
DRM_DEBUG_DRIVER("IOSF sideband idle wait (%s) timed out\n",
I915_WRITE(VLV_IOSF_DATA, is_read ? 0 : *val);
I915_WRITE(VLV_IOSF_DOORBELL_REQ, cmd);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
VLV_IOSF_DOORBELL_REQ, IOSF_SB_BUSY, 0,
5)) {
DRM_DEBUG_DRIVER("IOSF sideband finish wait (%s) timed out\n",
u32 value = 0;
WARN_ON(!mutex_is_locked(&dev_priv->sb_lock));
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
SBI_CTL_STAT, SBI_BUSY, 0,
100)) {
DRM_ERROR("timeout waiting for SBI to become ready\n");
value = SBI_CTL_DEST_MPHY | SBI_CTL_OP_IORD;
I915_WRITE(SBI_CTL_STAT, value | SBI_BUSY);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
SBI_CTL_STAT,
SBI_BUSY,
0,
WARN_ON(!mutex_is_locked(&dev_priv->sb_lock));
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
SBI_CTL_STAT, SBI_BUSY, 0,
100)) {
DRM_ERROR("timeout waiting for SBI to become ready\n");
tmp = SBI_CTL_DEST_MPHY | SBI_CTL_OP_IOWR;
I915_WRITE(SBI_CTL_STAT, SBI_BUSY | tmp);
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
SBI_CTL_STAT,
SBI_BUSY,
0,
* registers; newer ones are much simpler and we can use the new DRM plane
* support.
*/
+
+#include <drm/drm_atomic.h>
#include <drm/drm_atomic_helper.h>
+#include <drm/drm_color_mgmt.h>
#include <drm/drm_crtc.h>
#include <drm/drm_fourcc.h>
-#include <drm/drm_rect.h>
-#include <drm/drm_atomic.h>
#include <drm/drm_plane_helper.h>
-#include "intel_drv.h"
-#include "intel_frontbuffer.h"
+#include <drm/drm_rect.h>
#include <drm/i915_drm.h>
+
#include "i915_drv.h"
-#include <drm/drm_color_mgmt.h>
+#include "intel_atomic_plane.h"
+#include "intel_drv.h"
+#include "intel_frontbuffer.h"
+#include "intel_pm.h"
+#include "intel_psr.h"
+#include "intel_sprite.h"
+
+bool is_planar_yuv_format(u32 pixelformat)
+{
+ switch (pixelformat) {
+ case DRM_FORMAT_NV12:
+ case DRM_FORMAT_P010:
+ case DRM_FORMAT_P012:
+ case DRM_FORMAT_P016:
+ return true;
+ default:
+ return false;
+ }
+}
int intel_usecs_to_scanlines(const struct drm_display_mode *adjusted_mode,
int usecs)
{
const struct drm_framebuffer *fb = plane_state->base.fb;
struct drm_rect *src = &plane_state->base.src;
- u32 src_x, src_y, src_w, src_h;
+ u32 src_x, src_y, src_w, src_h, hsub, vsub;
+ bool rotated = drm_rotation_90_or_270(plane_state->base.rotation);
/*
* Hardware doesn't handle subpixel coordinates.
src->y1 = src_y << 16;
src->y2 = (src_y + src_h) << 16;
- if (fb->format->is_yuv &&
- (src_x & 1 || src_w & 1)) {
- DRM_DEBUG_KMS("src x/w (%u, %u) must be a multiple of 2 for YUV planes\n",
- src_x, src_w);
+ if (!fb->format->is_yuv)
+ return 0;
+
+ /* YUV specific checks */
+ if (!rotated) {
+ hsub = fb->format->hsub;
+ vsub = fb->format->vsub;
+ } else {
+ hsub = vsub = max(fb->format->hsub, fb->format->vsub);
+ }
+
+ if (src_x % hsub || src_w % hsub) {
+ DRM_DEBUG_KMS("src x/w (%u, %u) must be a multiple of %u for %sYUV planes\n",
+ src_x, src_w, hsub, rotated ? "rotated " : "");
return -EINVAL;
}
- if (fb->format->is_yuv &&
- fb->format->num_planes > 1 &&
- (src_y & 1 || src_h & 1)) {
- DRM_DEBUG_KMS("src y/h (%u, %u) must be a multiple of 2 for planar YUV planes\n",
- src_y, src_h);
+ if (src_y % vsub || src_h % vsub) {
+ DRM_DEBUG_KMS("src y/h (%u, %u) must be a multiple of %u for %sYUV planes\n",
+ src_y, src_h, vsub, rotated ? "rotated " : "");
return -EINVAL;
}
0, INT_MAX);
/* TODO: handle sub-pixel coordinates */
- if (plane_state->base.fb->format->format == DRM_FORMAT_NV12 &&
- !icl_is_hdr_plane(plane)) {
+ if (is_planar_yuv_format(plane_state->base.fb->format->format) &&
+ !icl_is_hdr_plane(dev_priv, plane->id)) {
y_hphase = skl_scaler_calc_phase(1, hscale, false);
y_vphase = skl_scaler_calc_phase(1, vscale, false);
I915_WRITE_FW(PLANE_AUX_DIST(pipe, plane_id),
(plane_state->color_plane[1].offset - surf_addr) | aux_stride);
- if (icl_is_hdr_plane(plane)) {
+ if (icl_is_hdr_plane(dev_priv, plane_id)) {
u32 cus_ctl = 0;
if (linked) {
if (INTEL_GEN(dev_priv) >= 10 || IS_GEMINILAKE(dev_priv))
I915_WRITE_FW(PLANE_COLOR_CTL(pipe, plane_id), plane_color_ctl);
- if (fb->format->is_yuv && icl_is_hdr_plane(plane))
+ if (fb->format->is_yuv && icl_is_hdr_plane(dev_priv, plane_id))
icl_program_input_csc(plane, crtc_state, plane_state);
skl_write_plane_wm(plane, crtc_state);
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
+ if (icl_is_hdr_plane(dev_priv, plane_id))
+ I915_WRITE_FW(PLANE_CUS_CTL(pipe, plane_id), 0);
+
skl_write_plane_wm(plane, crtc_state);
I915_WRITE_FW(PLANE_CTL(pipe, plane_id), 0);
static u32 vlv_sprite_ctl_crtc(const struct intel_crtc_state *crtc_state)
{
- return SP_GAMMA_ENABLE;
+ u32 sprctl = 0;
+
+ if (crtc_state->gamma_enable)
+ sprctl |= SP_GAMMA_ENABLE;
+
+ return sprctl;
}
static u32 vlv_sprite_ctl(const struct intel_crtc_state *crtc_state,
static u32 ivb_sprite_ctl_crtc(const struct intel_crtc_state *crtc_state)
{
- struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
u32 sprctl = 0;
- sprctl |= SPRITE_GAMMA_ENABLE;
+ if (crtc_state->gamma_enable)
+ sprctl |= SPRITE_GAMMA_ENABLE;
- if (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv))
+ if (crtc_state->csc_enable)
sprctl |= SPRITE_PIPE_CSC_ENABLE;
return sprctl;
static u32 g4x_sprite_ctl_crtc(const struct intel_crtc_state *crtc_state)
{
- return DVS_GAMMA_ENABLE;
+ u32 dvscntr = 0;
+
+ if (crtc_state->gamma_enable)
+ dvscntr |= DVS_GAMMA_ENABLE;
+
+ if (crtc_state->csc_enable)
+ dvscntr |= DVS_PIPE_CSC_ENABLE;
+
+ return dvscntr;
}
static u32 g4x_sprite_ctl(const struct intel_crtc_state *crtc_state,
/*
* 90/270 is not allowed with RGB64 16:16:16:16 and
* Indexed 8-bit. RGB 16-bit 5:6:5 is allowed gen11 onwards.
- * TBD: Add RGB64 case once its added in supported format
- * list.
*/
switch (fb->format->format) {
case DRM_FORMAT_RGB565:
break;
/* fall through */
case DRM_FORMAT_C8:
+ case DRM_FORMAT_XRGB16161616F:
+ case DRM_FORMAT_XBGR16161616F:
+ case DRM_FORMAT_ARGB16161616F:
+ case DRM_FORMAT_ABGR16161616F:
+ case DRM_FORMAT_Y210:
+ case DRM_FORMAT_Y212:
+ case DRM_FORMAT_Y216:
+ case DRM_FORMAT_XVYU12_16161616:
+ case DRM_FORMAT_XVYU16161616:
DRM_DEBUG_KMS("Unsupported pixel format %s for 90/270!\n",
drm_get_format_name(fb->format->format,
&format_name));
int src_w = drm_rect_width(&plane_state->base.src) >> 16;
/* Display WA #1106 */
- if (fb->format->format == DRM_FORMAT_NV12 && src_w & 3 &&
+ if (is_planar_yuv_format(fb->format->format) && src_w & 3 &&
(rotation == DRM_MODE_ROTATE_270 ||
rotation == (DRM_MODE_REFLECT_X | DRM_MODE_ROTATE_90))) {
- DRM_DEBUG_KMS("src width must be multiple of 4 for rotated NV12\n");
+ DRM_DEBUG_KMS("src width must be multiple of 4 for rotated planar YUV\n");
return -EINVAL;
}
DRM_FORMAT_VYUY,
};
+static const u32 icl_plane_formats[] = {
+ DRM_FORMAT_C8,
+ DRM_FORMAT_RGB565,
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_XBGR8888,
+ DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_ABGR8888,
+ DRM_FORMAT_XRGB2101010,
+ DRM_FORMAT_XBGR2101010,
+ DRM_FORMAT_YUYV,
+ DRM_FORMAT_YVYU,
+ DRM_FORMAT_UYVY,
+ DRM_FORMAT_VYUY,
+ DRM_FORMAT_Y210,
+ DRM_FORMAT_Y212,
+ DRM_FORMAT_Y216,
+ DRM_FORMAT_XVYU2101010,
+ DRM_FORMAT_XVYU12_16161616,
+ DRM_FORMAT_XVYU16161616,
+};
+
+static const u32 icl_hdr_plane_formats[] = {
+ DRM_FORMAT_C8,
+ DRM_FORMAT_RGB565,
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_XBGR8888,
+ DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_ABGR8888,
+ DRM_FORMAT_XRGB2101010,
+ DRM_FORMAT_XBGR2101010,
+ DRM_FORMAT_XRGB16161616F,
+ DRM_FORMAT_XBGR16161616F,
+ DRM_FORMAT_ARGB16161616F,
+ DRM_FORMAT_ABGR16161616F,
+ DRM_FORMAT_YUYV,
+ DRM_FORMAT_YVYU,
+ DRM_FORMAT_UYVY,
+ DRM_FORMAT_VYUY,
+ DRM_FORMAT_Y210,
+ DRM_FORMAT_Y212,
+ DRM_FORMAT_Y216,
+ DRM_FORMAT_XVYU2101010,
+ DRM_FORMAT_XVYU12_16161616,
+ DRM_FORMAT_XVYU16161616,
+};
+
static const u32 skl_planar_formats[] = {
DRM_FORMAT_C8,
DRM_FORMAT_RGB565,
DRM_FORMAT_NV12,
};
+static const u32 glk_planar_formats[] = {
+ DRM_FORMAT_C8,
+ DRM_FORMAT_RGB565,
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_XBGR8888,
+ DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_ABGR8888,
+ DRM_FORMAT_XRGB2101010,
+ DRM_FORMAT_XBGR2101010,
+ DRM_FORMAT_YUYV,
+ DRM_FORMAT_YVYU,
+ DRM_FORMAT_UYVY,
+ DRM_FORMAT_VYUY,
+ DRM_FORMAT_NV12,
+ DRM_FORMAT_P010,
+ DRM_FORMAT_P012,
+ DRM_FORMAT_P016,
+};
+
+static const u32 icl_planar_formats[] = {
+ DRM_FORMAT_C8,
+ DRM_FORMAT_RGB565,
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_XBGR8888,
+ DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_ABGR8888,
+ DRM_FORMAT_XRGB2101010,
+ DRM_FORMAT_XBGR2101010,
+ DRM_FORMAT_YUYV,
+ DRM_FORMAT_YVYU,
+ DRM_FORMAT_UYVY,
+ DRM_FORMAT_VYUY,
+ DRM_FORMAT_NV12,
+ DRM_FORMAT_P010,
+ DRM_FORMAT_P012,
+ DRM_FORMAT_P016,
+ DRM_FORMAT_Y210,
+ DRM_FORMAT_Y212,
+ DRM_FORMAT_Y216,
+ DRM_FORMAT_XVYU2101010,
+ DRM_FORMAT_XVYU12_16161616,
+ DRM_FORMAT_XVYU16161616,
+};
+
+static const u32 icl_hdr_planar_formats[] = {
+ DRM_FORMAT_C8,
+ DRM_FORMAT_RGB565,
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_XBGR8888,
+ DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_ABGR8888,
+ DRM_FORMAT_XRGB2101010,
+ DRM_FORMAT_XBGR2101010,
+ DRM_FORMAT_XRGB16161616F,
+ DRM_FORMAT_XBGR16161616F,
+ DRM_FORMAT_ARGB16161616F,
+ DRM_FORMAT_ABGR16161616F,
+ DRM_FORMAT_YUYV,
+ DRM_FORMAT_YVYU,
+ DRM_FORMAT_UYVY,
+ DRM_FORMAT_VYUY,
+ DRM_FORMAT_NV12,
+ DRM_FORMAT_P010,
+ DRM_FORMAT_P012,
+ DRM_FORMAT_P016,
+ DRM_FORMAT_Y210,
+ DRM_FORMAT_Y212,
+ DRM_FORMAT_Y216,
+ DRM_FORMAT_XVYU2101010,
+ DRM_FORMAT_XVYU12_16161616,
+ DRM_FORMAT_XVYU16161616,
+};
+
static const u64 skl_plane_format_modifiers_noccs[] = {
I915_FORMAT_MOD_Yf_TILED,
I915_FORMAT_MOD_Y_TILED,
case DRM_FORMAT_UYVY:
case DRM_FORMAT_VYUY:
case DRM_FORMAT_NV12:
+ case DRM_FORMAT_P010:
+ case DRM_FORMAT_P012:
+ case DRM_FORMAT_P016:
+ case DRM_FORMAT_XVYU2101010:
if (modifier == I915_FORMAT_MOD_Yf_TILED)
return true;
/* fall through */
case DRM_FORMAT_C8:
+ case DRM_FORMAT_XBGR16161616F:
+ case DRM_FORMAT_ABGR16161616F:
+ case DRM_FORMAT_XRGB16161616F:
+ case DRM_FORMAT_ARGB16161616F:
+ case DRM_FORMAT_Y210:
+ case DRM_FORMAT_Y212:
+ case DRM_FORMAT_Y216:
+ case DRM_FORMAT_XVYU12_16161616:
+ case DRM_FORMAT_XVYU16161616:
if (modifier == DRM_FORMAT_MOD_LINEAR ||
modifier == I915_FORMAT_MOD_X_TILED ||
modifier == I915_FORMAT_MOD_Y_TILED)
plane->update_slave = icl_update_slave;
if (skl_plane_has_planar(dev_priv, pipe, plane_id)) {
- formats = skl_planar_formats;
- num_formats = ARRAY_SIZE(skl_planar_formats);
+ if (icl_is_hdr_plane(dev_priv, plane_id)) {
+ formats = icl_hdr_planar_formats;
+ num_formats = ARRAY_SIZE(icl_hdr_planar_formats);
+ } else if (INTEL_GEN(dev_priv) >= 11) {
+ formats = icl_planar_formats;
+ num_formats = ARRAY_SIZE(icl_planar_formats);
+ } else if (INTEL_GEN(dev_priv) == 10 || IS_GEMINILAKE(dev_priv)) {
+ formats = glk_planar_formats;
+ num_formats = ARRAY_SIZE(glk_planar_formats);
+ } else {
+ formats = skl_planar_formats;
+ num_formats = ARRAY_SIZE(skl_planar_formats);
+ }
+ } else if (icl_is_hdr_plane(dev_priv, plane_id)) {
+ formats = icl_hdr_plane_formats;
+ num_formats = ARRAY_SIZE(icl_hdr_plane_formats);
+ } else if (INTEL_GEN(dev_priv) >= 11) {
+ formats = icl_plane_formats;
+ num_formats = ARRAY_SIZE(icl_plane_formats);
} else {
formats = skl_plane_formats;
num_formats = ARRAY_SIZE(skl_plane_formats);
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_SPRITE_H__
+#define __INTEL_SPRITE_H__
+
+#include <linux/types.h>
+
+#include "i915_drv.h"
+#include "intel_display.h"
+
+struct drm_device;
+struct drm_display_mode;
+struct drm_file;
+struct drm_i915_private;
+struct intel_crtc_state;
+struct intel_plane_state;
+
+bool is_planar_yuv_format(u32 pixelformat);
+int intel_usecs_to_scanlines(const struct drm_display_mode *adjusted_mode,
+ int usecs);
+struct intel_plane *intel_sprite_plane_create(struct drm_i915_private *dev_priv,
+ enum pipe pipe, int plane);
+int intel_sprite_set_colorkey_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_priv);
+void intel_pipe_update_start(const struct intel_crtc_state *new_crtc_state);
+void intel_pipe_update_end(struct intel_crtc_state *new_crtc_state);
+int intel_plane_check_stride(const struct intel_plane_state *plane_state);
+int intel_plane_check_src_coordinates(struct intel_plane_state *plane_state);
+int chv_plane_check_rotation(const struct intel_plane_state *plane_state);
+struct intel_plane *
+skl_universal_plane_create(struct drm_i915_private *dev_priv,
+ enum pipe pipe, enum plane_id plane_id);
+
+static inline bool icl_is_nv12_y_plane(enum plane_id id)
+{
+ /* Don't need to do a gen check, these planes are only available on gen11 */
+ if (id == PLANE_SPRITE4 || id == PLANE_SPRITE5)
+ return true;
+
+ return false;
+}
+
+static inline bool icl_is_hdr_plane(struct drm_i915_private *dev_priv,
+ enum plane_id plane_id)
+{
+ if (INTEL_GEN(dev_priv) < 11)
+ return false;
+
+ return plane_id < PLANE_SPRITE2;
+}
+
+#endif /* __INTEL_SPRITE_H__ */
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
#include <drm/drm_edid.h>
-#include "intel_drv.h"
#include <drm/i915_drm.h>
+
#include "i915_drv.h"
+#include "intel_connector.h"
+#include "intel_drv.h"
+#include "intel_tv.h"
enum tv_margin {
TV_MARGIN_LEFT, TV_MARGIN_TOP,
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __INTEL_TV_H__
+#define __INTEL_TV_H__
+
+struct drm_i915_private;
+
+void intel_tv_init(struct drm_i915_private *dev_priv);
+
+#endif /* __INTEL_TV_H__ */
GEM_BUG_ON(!HAS_GUC(i915));
- guc_disable_communication(guc);
-
intel_huc_sanitize(huc);
intel_guc_sanitize(guc);
intel_guc_init_params(guc);
ret = intel_guc_fw_upload(guc);
- if (ret == 0 || ret != -ETIMEDOUT)
+ if (ret == 0)
break;
DRM_DEBUG_DRIVER("GuC fw load failed: %d; will reset and "
guc_disable_communication(guc);
}
+/**
+ * intel_uc_reset_prepare - Prepare for reset
+ * @i915: device private
+ *
+ * Preparing for full gpu reset.
+ */
+void intel_uc_reset_prepare(struct drm_i915_private *i915)
+{
+ struct intel_guc *guc = &i915->guc;
+
+ if (!USES_GUC(i915))
+ return;
+
+ guc_disable_communication(guc);
+ intel_uc_sanitize(i915);
+}
+
int intel_uc_suspend(struct drm_i915_private *i915)
{
struct intel_guc *guc = &i915->guc;
return err;
}
- gen9_disable_guc_interrupts(i915);
+ guc_disable_communication(guc);
return 0;
}
if (guc->fw.load_status != INTEL_UC_FIRMWARE_SUCCESS)
return 0;
- gen9_enable_guc_interrupts(i915);
+ guc_enable_communication(guc);
err = intel_guc_resume(guc);
if (err) {
void intel_uc_fini_hw(struct drm_i915_private *dev_priv);
int intel_uc_init(struct drm_i915_private *dev_priv);
void intel_uc_fini(struct drm_i915_private *dev_priv);
+void intel_uc_reset_prepare(struct drm_i915_private *i915);
int intel_uc_suspend(struct drm_i915_private *dev_priv);
int intel_uc_resume(struct drm_i915_private *dev_priv);
* IN THE SOFTWARE.
*/
+#include <linux/pm_runtime.h>
+#include <asm/iosf_mbi.h>
+
#include "i915_drv.h"
-#include "intel_drv.h"
#include "i915_vgpu.h"
-
-#include <asm/iosf_mbi.h>
-#include <linux/pm_runtime.h>
+#include "intel_drv.h"
+#include "intel_pm.h"
#define FORCEWAKE_ACK_TIMEOUT_MS 50
#define GT_FIFO_TIMEOUT_MS 10
-#define __raw_posting_read(dev_priv__, reg__) (void)__raw_i915_read32((dev_priv__), (reg__))
+#define __raw_posting_read(...) ((void)__raw_uncore_read32(__VA_ARGS__))
static const char * const forcewake_domain_names[] = {
"render",
return "unknown";
}
+#define fw_ack(d) readl((d)->reg_ack)
+#define fw_set(d, val) writel(_MASKED_BIT_ENABLE((val)), (d)->reg_set)
+#define fw_clear(d, val) writel(_MASKED_BIT_DISABLE((val)), (d)->reg_set)
+
static inline void
-fw_domain_reset(struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d)
+fw_domain_reset(const struct intel_uncore_forcewake_domain *d)
{
/*
* We don't really know if the powerwell for the forcewake domain we are
* trying to reset here does exist at this point (engines could be fused
* off in ICL+), so no waiting for acks
*/
- __raw_i915_write32(i915, d->reg_set, i915->uncore.fw_reset);
+ /* WaRsClearFWBitsAtReset:bdw,skl */
+ fw_clear(d, 0xffff);
}
static inline void
}
static inline int
-__wait_for_ack(const struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d,
+__wait_for_ack(const struct intel_uncore_forcewake_domain *d,
const u32 ack,
const u32 value)
{
- return wait_for_atomic((__raw_i915_read32(i915, d->reg_ack) & ack) == value,
+ return wait_for_atomic((fw_ack(d) & ack) == value,
FORCEWAKE_ACK_TIMEOUT_MS);
}
static inline int
-wait_ack_clear(const struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d,
+wait_ack_clear(const struct intel_uncore_forcewake_domain *d,
const u32 ack)
{
- return __wait_for_ack(i915, d, ack, 0);
+ return __wait_for_ack(d, ack, 0);
}
static inline int
-wait_ack_set(const struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d,
+wait_ack_set(const struct intel_uncore_forcewake_domain *d,
const u32 ack)
{
- return __wait_for_ack(i915, d, ack, ack);
+ return __wait_for_ack(d, ack, ack);
}
static inline void
-fw_domain_wait_ack_clear(const struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d)
+fw_domain_wait_ack_clear(const struct intel_uncore_forcewake_domain *d)
{
- if (wait_ack_clear(i915, d, FORCEWAKE_KERNEL))
+ if (wait_ack_clear(d, FORCEWAKE_KERNEL))
DRM_ERROR("%s: timed out waiting for forcewake ack to clear.\n",
intel_uncore_forcewake_domain_to_str(d->id));
}
};
static int
-fw_domain_wait_ack_with_fallback(const struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d,
+fw_domain_wait_ack_with_fallback(const struct intel_uncore_forcewake_domain *d,
const enum ack_type type)
{
const u32 ack_bit = FORCEWAKE_KERNEL;
pass = 1;
do {
- wait_ack_clear(i915, d, FORCEWAKE_KERNEL_FALLBACK);
+ wait_ack_clear(d, FORCEWAKE_KERNEL_FALLBACK);
- __raw_i915_write32(i915, d->reg_set,
- _MASKED_BIT_ENABLE(FORCEWAKE_KERNEL_FALLBACK));
+ fw_set(d, FORCEWAKE_KERNEL_FALLBACK);
/* Give gt some time to relax before the polling frenzy */
udelay(10 * pass);
- wait_ack_set(i915, d, FORCEWAKE_KERNEL_FALLBACK);
+ wait_ack_set(d, FORCEWAKE_KERNEL_FALLBACK);
- ack_detected = (__raw_i915_read32(i915, d->reg_ack) & ack_bit) == value;
+ ack_detected = (fw_ack(d) & ack_bit) == value;
- __raw_i915_write32(i915, d->reg_set,
- _MASKED_BIT_DISABLE(FORCEWAKE_KERNEL_FALLBACK));
+ fw_clear(d, FORCEWAKE_KERNEL_FALLBACK);
} while (!ack_detected && pass++ < 10);
DRM_DEBUG_DRIVER("%s had to use fallback to %s ack, 0x%x (passes %u)\n",
intel_uncore_forcewake_domain_to_str(d->id),
type == ACK_SET ? "set" : "clear",
- __raw_i915_read32(i915, d->reg_ack),
+ fw_ack(d),
pass);
return ack_detected ? 0 : -ETIMEDOUT;
}
static inline void
-fw_domain_wait_ack_clear_fallback(const struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d)
+fw_domain_wait_ack_clear_fallback(const struct intel_uncore_forcewake_domain *d)
{
- if (likely(!wait_ack_clear(i915, d, FORCEWAKE_KERNEL)))
+ if (likely(!wait_ack_clear(d, FORCEWAKE_KERNEL)))
return;
- if (fw_domain_wait_ack_with_fallback(i915, d, ACK_CLEAR))
- fw_domain_wait_ack_clear(i915, d);
+ if (fw_domain_wait_ack_with_fallback(d, ACK_CLEAR))
+ fw_domain_wait_ack_clear(d);
}
static inline void
-fw_domain_get(struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d)
+fw_domain_get(const struct intel_uncore_forcewake_domain *d)
{
- __raw_i915_write32(i915, d->reg_set, i915->uncore.fw_set);
+ fw_set(d, FORCEWAKE_KERNEL);
}
static inline void
-fw_domain_wait_ack_set(const struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d)
+fw_domain_wait_ack_set(const struct intel_uncore_forcewake_domain *d)
{
- if (wait_ack_set(i915, d, FORCEWAKE_KERNEL))
+ if (wait_ack_set(d, FORCEWAKE_KERNEL))
DRM_ERROR("%s: timed out waiting for forcewake ack request.\n",
intel_uncore_forcewake_domain_to_str(d->id));
}
static inline void
-fw_domain_wait_ack_set_fallback(const struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d)
+fw_domain_wait_ack_set_fallback(const struct intel_uncore_forcewake_domain *d)
{
- if (likely(!wait_ack_set(i915, d, FORCEWAKE_KERNEL)))
+ if (likely(!wait_ack_set(d, FORCEWAKE_KERNEL)))
return;
- if (fw_domain_wait_ack_with_fallback(i915, d, ACK_SET))
- fw_domain_wait_ack_set(i915, d);
+ if (fw_domain_wait_ack_with_fallback(d, ACK_SET))
+ fw_domain_wait_ack_set(d);
}
static inline void
-fw_domain_put(const struct drm_i915_private *i915,
- const struct intel_uncore_forcewake_domain *d)
+fw_domain_put(const struct intel_uncore_forcewake_domain *d)
{
- __raw_i915_write32(i915, d->reg_set, i915->uncore.fw_clear);
+ fw_clear(d, FORCEWAKE_KERNEL);
}
static void
-fw_domains_get(struct drm_i915_private *i915, enum forcewake_domains fw_domains)
+fw_domains_get(struct intel_uncore *uncore, enum forcewake_domains fw_domains)
{
struct intel_uncore_forcewake_domain *d;
unsigned int tmp;
- GEM_BUG_ON(fw_domains & ~i915->uncore.fw_domains);
+ GEM_BUG_ON(fw_domains & ~uncore->fw_domains);
- for_each_fw_domain_masked(d, fw_domains, i915, tmp) {
- fw_domain_wait_ack_clear(i915, d);
- fw_domain_get(i915, d);
+ for_each_fw_domain_masked(d, fw_domains, uncore, tmp) {
+ fw_domain_wait_ack_clear(d);
+ fw_domain_get(d);
}
- for_each_fw_domain_masked(d, fw_domains, i915, tmp)
- fw_domain_wait_ack_set(i915, d);
+ for_each_fw_domain_masked(d, fw_domains, uncore, tmp)
+ fw_domain_wait_ack_set(d);
- i915->uncore.fw_domains_active |= fw_domains;
+ uncore->fw_domains_active |= fw_domains;
}
static void
-fw_domains_get_with_fallback(struct drm_i915_private *i915,
+fw_domains_get_with_fallback(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
struct intel_uncore_forcewake_domain *d;
unsigned int tmp;
- GEM_BUG_ON(fw_domains & ~i915->uncore.fw_domains);
+ GEM_BUG_ON(fw_domains & ~uncore->fw_domains);
- for_each_fw_domain_masked(d, fw_domains, i915, tmp) {
- fw_domain_wait_ack_clear_fallback(i915, d);
- fw_domain_get(i915, d);
+ for_each_fw_domain_masked(d, fw_domains, uncore, tmp) {
+ fw_domain_wait_ack_clear_fallback(d);
+ fw_domain_get(d);
}
- for_each_fw_domain_masked(d, fw_domains, i915, tmp)
- fw_domain_wait_ack_set_fallback(i915, d);
+ for_each_fw_domain_masked(d, fw_domains, uncore, tmp)
+ fw_domain_wait_ack_set_fallback(d);
- i915->uncore.fw_domains_active |= fw_domains;
+ uncore->fw_domains_active |= fw_domains;
}
static void
-fw_domains_put(struct drm_i915_private *i915, enum forcewake_domains fw_domains)
+fw_domains_put(struct intel_uncore *uncore, enum forcewake_domains fw_domains)
{
struct intel_uncore_forcewake_domain *d;
unsigned int tmp;
- GEM_BUG_ON(fw_domains & ~i915->uncore.fw_domains);
+ GEM_BUG_ON(fw_domains & ~uncore->fw_domains);
- for_each_fw_domain_masked(d, fw_domains, i915, tmp)
- fw_domain_put(i915, d);
+ for_each_fw_domain_masked(d, fw_domains, uncore, tmp)
+ fw_domain_put(d);
- i915->uncore.fw_domains_active &= ~fw_domains;
+ uncore->fw_domains_active &= ~fw_domains;
}
static void
-fw_domains_reset(struct drm_i915_private *i915,
+fw_domains_reset(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
struct intel_uncore_forcewake_domain *d;
if (!fw_domains)
return;
- GEM_BUG_ON(fw_domains & ~i915->uncore.fw_domains);
+ GEM_BUG_ON(fw_domains & ~uncore->fw_domains);
- for_each_fw_domain_masked(d, fw_domains, i915, tmp)
- fw_domain_reset(i915, d);
+ for_each_fw_domain_masked(d, fw_domains, uncore, tmp)
+ fw_domain_reset(d);
}
-static inline u32 gt_thread_status(struct drm_i915_private *dev_priv)
+static inline u32 gt_thread_status(struct intel_uncore *uncore)
{
u32 val;
- val = __raw_i915_read32(dev_priv, GEN6_GT_THREAD_STATUS_REG);
+ val = __raw_uncore_read32(uncore, GEN6_GT_THREAD_STATUS_REG);
val &= GEN6_GT_THREAD_STATUS_CORE_MASK;
return val;
}
-static void __gen6_gt_wait_for_thread_c0(struct drm_i915_private *dev_priv)
+static void __gen6_gt_wait_for_thread_c0(struct intel_uncore *uncore)
{
/*
* w/a for a sporadic read returning 0 by waiting for the GT
* thread to wake up.
*/
- WARN_ONCE(wait_for_atomic_us(gt_thread_status(dev_priv) == 0, 5000),
+ WARN_ONCE(wait_for_atomic_us(gt_thread_status(uncore) == 0, 5000),
"GT thread status wait timed out\n");
}
-static void fw_domains_get_with_thread_status(struct drm_i915_private *dev_priv,
+static void fw_domains_get_with_thread_status(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
- fw_domains_get(dev_priv, fw_domains);
+ fw_domains_get(uncore, fw_domains);
/* WaRsForcewakeWaitTC0:snb,ivb,hsw,bdw,vlv */
- __gen6_gt_wait_for_thread_c0(dev_priv);
+ __gen6_gt_wait_for_thread_c0(uncore);
}
-static inline u32 fifo_free_entries(struct drm_i915_private *dev_priv)
+static inline u32 fifo_free_entries(struct intel_uncore *uncore)
{
- u32 count = __raw_i915_read32(dev_priv, GTFIFOCTL);
+ u32 count = __raw_uncore_read32(uncore, GTFIFOCTL);
return count & GT_FIFO_FREE_ENTRIES_MASK;
}
-static void __gen6_gt_wait_for_fifo(struct drm_i915_private *dev_priv)
+static void __gen6_gt_wait_for_fifo(struct intel_uncore *uncore)
{
u32 n;
/* On VLV, FIFO will be shared by both SW and HW.
* So, we need to read the FREE_ENTRIES everytime */
- if (IS_VALLEYVIEW(dev_priv))
- n = fifo_free_entries(dev_priv);
+ if (IS_VALLEYVIEW(uncore_to_i915(uncore)))
+ n = fifo_free_entries(uncore);
else
- n = dev_priv->uncore.fifo_count;
+ n = uncore->fifo_count;
if (n <= GT_FIFO_NUM_RESERVED_ENTRIES) {
- if (wait_for_atomic((n = fifo_free_entries(dev_priv)) >
+ if (wait_for_atomic((n = fifo_free_entries(uncore)) >
GT_FIFO_NUM_RESERVED_ENTRIES,
GT_FIFO_TIMEOUT_MS)) {
DRM_DEBUG("GT_FIFO timeout, entries: %u\n", n);
}
}
- dev_priv->uncore.fifo_count = n - 1;
+ uncore->fifo_count = n - 1;
}
static enum hrtimer_restart
{
struct intel_uncore_forcewake_domain *domain =
container_of(timer, struct intel_uncore_forcewake_domain, timer);
- struct drm_i915_private *dev_priv =
- container_of(domain, struct drm_i915_private, uncore.fw_domain[domain->id]);
+ struct intel_uncore *uncore = forcewake_domain_to_uncore(domain);
unsigned long irqflags;
- assert_rpm_device_not_suspended(dev_priv);
+ assert_rpm_device_not_suspended(uncore->rpm);
if (xchg(&domain->active, false))
return HRTIMER_RESTART;
- spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
+ spin_lock_irqsave(&uncore->lock, irqflags);
if (WARN_ON(domain->wake_count == 0))
domain->wake_count++;
if (--domain->wake_count == 0)
- dev_priv->uncore.funcs.force_wake_put(dev_priv, domain->mask);
+ uncore->funcs.force_wake_put(uncore, domain->mask);
- spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
+ spin_unlock_irqrestore(&uncore->lock, irqflags);
return HRTIMER_NORESTART;
}
/* Note callers must have acquired the PUNIT->PMIC bus, before calling this. */
static unsigned int
-intel_uncore_forcewake_reset(struct drm_i915_private *dev_priv)
+intel_uncore_forcewake_reset(struct intel_uncore *uncore)
{
unsigned long irqflags;
struct intel_uncore_forcewake_domain *domain;
active_domains = 0;
- for_each_fw_domain(domain, dev_priv, tmp) {
+ for_each_fw_domain(domain, uncore, tmp) {
smp_store_mb(domain->active, false);
if (hrtimer_cancel(&domain->timer) == 0)
continue;
intel_uncore_fw_release_timer(&domain->timer);
}
- spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
+ spin_lock_irqsave(&uncore->lock, irqflags);
- for_each_fw_domain(domain, dev_priv, tmp) {
+ for_each_fw_domain(domain, uncore, tmp) {
if (hrtimer_active(&domain->timer))
active_domains |= domain->mask;
}
break;
}
- spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
+ spin_unlock_irqrestore(&uncore->lock, irqflags);
cond_resched();
}
WARN_ON(active_domains);
- fw = dev_priv->uncore.fw_domains_active;
+ fw = uncore->fw_domains_active;
if (fw)
- dev_priv->uncore.funcs.force_wake_put(dev_priv, fw);
+ uncore->funcs.force_wake_put(uncore, fw);
- fw_domains_reset(dev_priv, dev_priv->uncore.fw_domains);
- assert_forcewakes_inactive(dev_priv);
+ fw_domains_reset(uncore, uncore->fw_domains);
+ assert_forcewakes_inactive(uncore);
- spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
+ spin_unlock_irqrestore(&uncore->lock, irqflags);
return fw; /* track the lost user forcewake domains */
}
-static u64 gen9_edram_size(struct drm_i915_private *dev_priv)
-{
- const unsigned int ways[8] = { 4, 8, 12, 16, 16, 16, 16, 16 };
- const unsigned int sets[4] = { 1, 1, 2, 2 };
- const u32 cap = dev_priv->edram_cap;
-
- return EDRAM_NUM_BANKS(cap) *
- ways[EDRAM_WAYS_IDX(cap)] *
- sets[EDRAM_SETS_IDX(cap)] *
- 1024 * 1024;
-}
-
-u64 intel_uncore_edram_size(struct drm_i915_private *dev_priv)
-{
- if (!HAS_EDRAM(dev_priv))
- return 0;
-
- /* The needed capability bits for size calculation
- * are not there with pre gen9 so return 128MB always.
- */
- if (INTEL_GEN(dev_priv) < 9)
- return 128 * 1024 * 1024;
-
- return gen9_edram_size(dev_priv);
-}
-
-static void intel_uncore_edram_detect(struct drm_i915_private *dev_priv)
-{
- if (IS_HASWELL(dev_priv) ||
- IS_BROADWELL(dev_priv) ||
- INTEL_GEN(dev_priv) >= 9) {
- dev_priv->edram_cap = __raw_i915_read32(dev_priv,
- HSW_EDRAM_CAP);
-
- /* NB: We can't write IDICR yet because we do not have gt funcs
- * set up */
- } else {
- dev_priv->edram_cap = 0;
- }
-
- if (HAS_EDRAM(dev_priv))
- DRM_INFO("Found %lluMB of eDRAM\n",
- intel_uncore_edram_size(dev_priv) / (1024 * 1024));
-}
-
static bool
-fpga_check_for_unclaimed_mmio(struct drm_i915_private *dev_priv)
+fpga_check_for_unclaimed_mmio(struct intel_uncore *uncore)
{
u32 dbg;
- dbg = __raw_i915_read32(dev_priv, FPGA_DBG);
+ dbg = __raw_uncore_read32(uncore, FPGA_DBG);
if (likely(!(dbg & FPGA_DBG_RM_NOCLAIM)))
return false;
- __raw_i915_write32(dev_priv, FPGA_DBG, FPGA_DBG_RM_NOCLAIM);
+ __raw_uncore_write32(uncore, FPGA_DBG, FPGA_DBG_RM_NOCLAIM);
return true;
}
static bool
-vlv_check_for_unclaimed_mmio(struct drm_i915_private *dev_priv)
+vlv_check_for_unclaimed_mmio(struct intel_uncore *uncore)
{
u32 cer;
- cer = __raw_i915_read32(dev_priv, CLAIM_ER);
+ cer = __raw_uncore_read32(uncore, CLAIM_ER);
if (likely(!(cer & (CLAIM_ER_OVERFLOW | CLAIM_ER_CTR_MASK))))
return false;
- __raw_i915_write32(dev_priv, CLAIM_ER, CLAIM_ER_CLR);
+ __raw_uncore_write32(uncore, CLAIM_ER, CLAIM_ER_CLR);
return true;
}
static bool
-gen6_check_for_fifo_debug(struct drm_i915_private *dev_priv)
+gen6_check_for_fifo_debug(struct intel_uncore *uncore)
{
u32 fifodbg;
- fifodbg = __raw_i915_read32(dev_priv, GTFIFODBG);
+ fifodbg = __raw_uncore_read32(uncore, GTFIFODBG);
if (unlikely(fifodbg)) {
DRM_DEBUG_DRIVER("GTFIFODBG = 0x08%x\n", fifodbg);
- __raw_i915_write32(dev_priv, GTFIFODBG, fifodbg);
+ __raw_uncore_write32(uncore, GTFIFODBG, fifodbg);
}
return fifodbg;
}
static bool
-check_for_unclaimed_mmio(struct drm_i915_private *dev_priv)
+check_for_unclaimed_mmio(struct intel_uncore *uncore)
{
bool ret = false;
- if (HAS_FPGA_DBG_UNCLAIMED(dev_priv))
- ret |= fpga_check_for_unclaimed_mmio(dev_priv);
+ if (intel_uncore_has_fpga_dbg_unclaimed(uncore))
+ ret |= fpga_check_for_unclaimed_mmio(uncore);
- if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
- ret |= vlv_check_for_unclaimed_mmio(dev_priv);
+ if (intel_uncore_has_dbg_unclaimed(uncore))
+ ret |= vlv_check_for_unclaimed_mmio(uncore);
- if (IS_GEN_RANGE(dev_priv, 6, 7))
- ret |= gen6_check_for_fifo_debug(dev_priv);
+ if (intel_uncore_has_fifo(uncore))
+ ret |= gen6_check_for_fifo_debug(uncore);
return ret;
}
-static void __intel_uncore_early_sanitize(struct drm_i915_private *dev_priv,
+static void __intel_uncore_early_sanitize(struct intel_uncore *uncore,
unsigned int restore_forcewake)
{
/* clear out unclaimed reg detection bit */
- if (check_for_unclaimed_mmio(dev_priv))
+ if (check_for_unclaimed_mmio(uncore))
DRM_DEBUG("unclaimed mmio detected on uncore init, clearing\n");
/* WaDisableShadowRegForCpd:chv */
- if (IS_CHERRYVIEW(dev_priv)) {
- __raw_i915_write32(dev_priv, GTFIFOCTL,
- __raw_i915_read32(dev_priv, GTFIFOCTL) |
- GT_FIFO_CTL_BLOCK_ALL_POLICY_STALL |
- GT_FIFO_CTL_RC6_POLICY_STALL);
+ if (IS_CHERRYVIEW(uncore_to_i915(uncore))) {
+ __raw_uncore_write32(uncore, GTFIFOCTL,
+ __raw_uncore_read32(uncore, GTFIFOCTL) |
+ GT_FIFO_CTL_BLOCK_ALL_POLICY_STALL |
+ GT_FIFO_CTL_RC6_POLICY_STALL);
}
iosf_mbi_punit_acquire();
- intel_uncore_forcewake_reset(dev_priv);
+ intel_uncore_forcewake_reset(uncore);
if (restore_forcewake) {
- spin_lock_irq(&dev_priv->uncore.lock);
- dev_priv->uncore.funcs.force_wake_get(dev_priv,
- restore_forcewake);
-
- if (IS_GEN_RANGE(dev_priv, 6, 7))
- dev_priv->uncore.fifo_count =
- fifo_free_entries(dev_priv);
- spin_unlock_irq(&dev_priv->uncore.lock);
+ spin_lock_irq(&uncore->lock);
+ uncore->funcs.force_wake_get(uncore, restore_forcewake);
+
+ if (intel_uncore_has_fifo(uncore))
+ uncore->fifo_count = fifo_free_entries(uncore);
+ spin_unlock_irq(&uncore->lock);
}
iosf_mbi_punit_release();
}
-void intel_uncore_suspend(struct drm_i915_private *dev_priv)
+void intel_uncore_suspend(struct intel_uncore *uncore)
{
iosf_mbi_punit_acquire();
iosf_mbi_unregister_pmic_bus_access_notifier_unlocked(
- &dev_priv->uncore.pmic_bus_access_nb);
- dev_priv->uncore.fw_domains_saved =
- intel_uncore_forcewake_reset(dev_priv);
+ &uncore->pmic_bus_access_nb);
+ uncore->fw_domains_saved = intel_uncore_forcewake_reset(uncore);
iosf_mbi_punit_release();
}
-void intel_uncore_resume_early(struct drm_i915_private *dev_priv)
+void intel_uncore_resume_early(struct intel_uncore *uncore)
{
unsigned int restore_forcewake;
- restore_forcewake = fetch_and_zero(&dev_priv->uncore.fw_domains_saved);
- __intel_uncore_early_sanitize(dev_priv, restore_forcewake);
+ restore_forcewake = fetch_and_zero(&uncore->fw_domains_saved);
+ __intel_uncore_early_sanitize(uncore, restore_forcewake);
- iosf_mbi_register_pmic_bus_access_notifier(
- &dev_priv->uncore.pmic_bus_access_nb);
- i915_check_and_clear_faults(dev_priv);
+ iosf_mbi_register_pmic_bus_access_notifier(&uncore->pmic_bus_access_nb);
}
-void intel_uncore_runtime_resume(struct drm_i915_private *dev_priv)
+void intel_uncore_runtime_resume(struct intel_uncore *uncore)
{
- iosf_mbi_register_pmic_bus_access_notifier(
- &dev_priv->uncore.pmic_bus_access_nb);
+ iosf_mbi_register_pmic_bus_access_notifier(&uncore->pmic_bus_access_nb);
}
void intel_uncore_sanitize(struct drm_i915_private *dev_priv)
intel_sanitize_gt_powersave(dev_priv);
}
-static void __intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
+static void __intel_uncore_forcewake_get(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
struct intel_uncore_forcewake_domain *domain;
unsigned int tmp;
- fw_domains &= dev_priv->uncore.fw_domains;
+ fw_domains &= uncore->fw_domains;
- for_each_fw_domain_masked(domain, fw_domains, dev_priv, tmp) {
+ for_each_fw_domain_masked(domain, fw_domains, uncore, tmp) {
if (domain->wake_count++) {
fw_domains &= ~domain->mask;
domain->active = true;
}
if (fw_domains)
- dev_priv->uncore.funcs.force_wake_get(dev_priv, fw_domains);
+ uncore->funcs.force_wake_get(uncore, fw_domains);
}
/**
* intel_uncore_forcewake_get - grab forcewake domain references
- * @dev_priv: i915 device instance
+ * @uncore: the intel_uncore structure
* @fw_domains: forcewake domains to get reference on
*
* This function can be used get GT's forcewake domain references.
* call to intel_unforce_forcewake_put(). Usually caller wants all the domains
* to be kept awake so the @fw_domains would be then FORCEWAKE_ALL.
*/
-void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
+void intel_uncore_forcewake_get(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
unsigned long irqflags;
- if (!dev_priv->uncore.funcs.force_wake_get)
+ if (!uncore->funcs.force_wake_get)
return;
- assert_rpm_wakelock_held(dev_priv);
+ __assert_rpm_wakelock_held(uncore->rpm);
- spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
- __intel_uncore_forcewake_get(dev_priv, fw_domains);
- spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
+ spin_lock_irqsave(&uncore->lock, irqflags);
+ __intel_uncore_forcewake_get(uncore, fw_domains);
+ spin_unlock_irqrestore(&uncore->lock, irqflags);
}
/**
* intel_uncore_forcewake_user_get - claim forcewake on behalf of userspace
- * @dev_priv: i915 device instance
+ * @uncore: the intel_uncore structure
*
* This function is a wrapper around intel_uncore_forcewake_get() to acquire
* the GT powerwell and in the process disable our debugging for the
* duration of userspace's bypass.
*/
-void intel_uncore_forcewake_user_get(struct drm_i915_private *dev_priv)
+void intel_uncore_forcewake_user_get(struct intel_uncore *uncore)
{
- spin_lock_irq(&dev_priv->uncore.lock);
- if (!dev_priv->uncore.user_forcewake.count++) {
- intel_uncore_forcewake_get__locked(dev_priv, FORCEWAKE_ALL);
+ spin_lock_irq(&uncore->lock);
+ if (!uncore->user_forcewake.count++) {
+ intel_uncore_forcewake_get__locked(uncore, FORCEWAKE_ALL);
/* Save and disable mmio debugging for the user bypass */
- dev_priv->uncore.user_forcewake.saved_mmio_check =
- dev_priv->uncore.unclaimed_mmio_check;
- dev_priv->uncore.user_forcewake.saved_mmio_debug =
+ uncore->user_forcewake.saved_mmio_check =
+ uncore->unclaimed_mmio_check;
+ uncore->user_forcewake.saved_mmio_debug =
i915_modparams.mmio_debug;
- dev_priv->uncore.unclaimed_mmio_check = 0;
+ uncore->unclaimed_mmio_check = 0;
i915_modparams.mmio_debug = 0;
}
- spin_unlock_irq(&dev_priv->uncore.lock);
+ spin_unlock_irq(&uncore->lock);
}
/**
* intel_uncore_forcewake_user_put - release forcewake on behalf of userspace
- * @dev_priv: i915 device instance
+ * @uncore: the intel_uncore structure
*
* This function complements intel_uncore_forcewake_user_get() and releases
* the GT powerwell taken on behalf of the userspace bypass.
*/
-void intel_uncore_forcewake_user_put(struct drm_i915_private *dev_priv)
+void intel_uncore_forcewake_user_put(struct intel_uncore *uncore)
{
- spin_lock_irq(&dev_priv->uncore.lock);
- if (!--dev_priv->uncore.user_forcewake.count) {
- if (intel_uncore_unclaimed_mmio(dev_priv))
- dev_info(dev_priv->drm.dev,
+ spin_lock_irq(&uncore->lock);
+ if (!--uncore->user_forcewake.count) {
+ if (intel_uncore_unclaimed_mmio(uncore))
+ dev_info(uncore_to_i915(uncore)->drm.dev,
"Invalid mmio detected during user access\n");
- dev_priv->uncore.unclaimed_mmio_check =
- dev_priv->uncore.user_forcewake.saved_mmio_check;
+ uncore->unclaimed_mmio_check =
+ uncore->user_forcewake.saved_mmio_check;
i915_modparams.mmio_debug =
- dev_priv->uncore.user_forcewake.saved_mmio_debug;
+ uncore->user_forcewake.saved_mmio_debug;
- intel_uncore_forcewake_put__locked(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put__locked(uncore, FORCEWAKE_ALL);
}
- spin_unlock_irq(&dev_priv->uncore.lock);
+ spin_unlock_irq(&uncore->lock);
}
/**
* intel_uncore_forcewake_get__locked - grab forcewake domain references
- * @dev_priv: i915 device instance
+ * @uncore: the intel_uncore structure
* @fw_domains: forcewake domains to get reference on
*
* See intel_uncore_forcewake_get(). This variant places the onus
* on the caller to explicitly handle the dev_priv->uncore.lock spinlock.
*/
-void intel_uncore_forcewake_get__locked(struct drm_i915_private *dev_priv,
+void intel_uncore_forcewake_get__locked(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
- lockdep_assert_held(&dev_priv->uncore.lock);
+ lockdep_assert_held(&uncore->lock);
- if (!dev_priv->uncore.funcs.force_wake_get)
+ if (!uncore->funcs.force_wake_get)
return;
- __intel_uncore_forcewake_get(dev_priv, fw_domains);
+ __intel_uncore_forcewake_get(uncore, fw_domains);
}
-static void __intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
+static void __intel_uncore_forcewake_put(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
struct intel_uncore_forcewake_domain *domain;
unsigned int tmp;
- fw_domains &= dev_priv->uncore.fw_domains;
+ fw_domains &= uncore->fw_domains;
- for_each_fw_domain_masked(domain, fw_domains, dev_priv, tmp) {
+ for_each_fw_domain_masked(domain, fw_domains, uncore, tmp) {
if (WARN_ON(domain->wake_count == 0))
continue;
/**
* intel_uncore_forcewake_put - release a forcewake domain reference
- * @dev_priv: i915 device instance
+ * @uncore: the intel_uncore structure
* @fw_domains: forcewake domains to put references
*
* This function drops the device-level forcewakes for specified
* domains obtained by intel_uncore_forcewake_get().
*/
-void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
+void intel_uncore_forcewake_put(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
unsigned long irqflags;
- if (!dev_priv->uncore.funcs.force_wake_put)
+ if (!uncore->funcs.force_wake_put)
return;
- spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
- __intel_uncore_forcewake_put(dev_priv, fw_domains);
- spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
+ spin_lock_irqsave(&uncore->lock, irqflags);
+ __intel_uncore_forcewake_put(uncore, fw_domains);
+ spin_unlock_irqrestore(&uncore->lock, irqflags);
}
/**
* intel_uncore_forcewake_put__locked - grab forcewake domain references
- * @dev_priv: i915 device instance
+ * @uncore: the intel_uncore structure
* @fw_domains: forcewake domains to get reference on
*
* See intel_uncore_forcewake_put(). This variant places the onus
* on the caller to explicitly handle the dev_priv->uncore.lock spinlock.
*/
-void intel_uncore_forcewake_put__locked(struct drm_i915_private *dev_priv,
+void intel_uncore_forcewake_put__locked(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
- lockdep_assert_held(&dev_priv->uncore.lock);
+ lockdep_assert_held(&uncore->lock);
- if (!dev_priv->uncore.funcs.force_wake_put)
+ if (!uncore->funcs.force_wake_put)
return;
- __intel_uncore_forcewake_put(dev_priv, fw_domains);
+ __intel_uncore_forcewake_put(uncore, fw_domains);
}
-void assert_forcewakes_inactive(struct drm_i915_private *dev_priv)
+void assert_forcewakes_inactive(struct intel_uncore *uncore)
{
- if (!dev_priv->uncore.funcs.force_wake_get)
+ if (!uncore->funcs.force_wake_get)
return;
- WARN(dev_priv->uncore.fw_domains_active,
+ WARN(uncore->fw_domains_active,
"Expected all fw_domains to be inactive, but %08x are still on\n",
- dev_priv->uncore.fw_domains_active);
+ uncore->fw_domains_active);
}
-void assert_forcewakes_active(struct drm_i915_private *dev_priv,
+void assert_forcewakes_active(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
- if (!dev_priv->uncore.funcs.force_wake_get)
+ if (!uncore->funcs.force_wake_get)
return;
- assert_rpm_wakelock_held(dev_priv);
+ __assert_rpm_wakelock_held(uncore->rpm);
- fw_domains &= dev_priv->uncore.fw_domains;
- WARN(fw_domains & ~dev_priv->uncore.fw_domains_active,
+ fw_domains &= uncore->fw_domains;
+ WARN(fw_domains & ~uncore->fw_domains_active,
"Expected %08x fw_domains to be active, but %08x are off\n",
- fw_domains, fw_domains & ~dev_priv->uncore.fw_domains_active);
+ fw_domains, fw_domains & ~uncore->fw_domains_active);
}
/* We give fast paths for the really cool registers */
#define GEN11_NEEDS_FORCE_WAKE(reg) \
((reg) < 0x40000 || ((reg) >= 0x1c0000 && (reg) < 0x1dc000))
-#define __gen6_reg_read_fw_domains(offset) \
+#define __gen6_reg_read_fw_domains(uncore, offset) \
({ \
enum forcewake_domains __fwd; \
if (NEEDS_FORCE_WAKE(offset)) \
})
static enum forcewake_domains
-find_fw_domain(struct drm_i915_private *dev_priv, u32 offset)
+find_fw_domain(struct intel_uncore *uncore, u32 offset)
{
const struct intel_forcewake_range *entry;
entry = BSEARCH(offset,
- dev_priv->uncore.fw_domains_table,
- dev_priv->uncore.fw_domains_table_entries,
+ uncore->fw_domains_table,
+ uncore->fw_domains_table_entries,
fw_range_cmp);
if (!entry)
* translate it here to the list of available domains.
*/
if (entry->domains == FORCEWAKE_ALL)
- return dev_priv->uncore.fw_domains;
+ return uncore->fw_domains;
- WARN(entry->domains & ~dev_priv->uncore.fw_domains,
+ WARN(entry->domains & ~uncore->fw_domains,
"Uninitialized forcewake domain(s) 0x%x accessed at 0x%x\n",
- entry->domains & ~dev_priv->uncore.fw_domains, offset);
+ entry->domains & ~uncore->fw_domains, offset);
return entry->domains;
}
GEN_FW_RANGE(0x30000, 0x3ffff, FORCEWAKE_MEDIA),
};
-#define __fwtable_reg_read_fw_domains(offset) \
+#define __fwtable_reg_read_fw_domains(uncore, offset) \
({ \
enum forcewake_domains __fwd = 0; \
if (NEEDS_FORCE_WAKE((offset))) \
- __fwd = find_fw_domain(dev_priv, offset); \
+ __fwd = find_fw_domain(uncore, offset); \
__fwd; \
})
-#define __gen11_fwtable_reg_read_fw_domains(offset) \
+#define __gen11_fwtable_reg_read_fw_domains(uncore, offset) \
({ \
enum forcewake_domains __fwd = 0; \
if (GEN11_NEEDS_FORCE_WAKE((offset))) \
- __fwd = find_fw_domain(dev_priv, offset); \
+ __fwd = find_fw_domain(uncore, offset); \
__fwd; \
})
__is_genX_shadowed(8)
__is_genX_shadowed(11)
-#define __gen8_reg_write_fw_domains(offset) \
+#define __gen8_reg_write_fw_domains(uncore, offset) \
({ \
enum forcewake_domains __fwd; \
if (NEEDS_FORCE_WAKE(offset) && !is_gen8_shadowed(offset)) \
GEN_FW_RANGE(0x30000, 0x37fff, FORCEWAKE_MEDIA),
};
-#define __fwtable_reg_write_fw_domains(offset) \
+#define __fwtable_reg_write_fw_domains(uncore, offset) \
({ \
enum forcewake_domains __fwd = 0; \
if (NEEDS_FORCE_WAKE((offset)) && !is_gen8_shadowed(offset)) \
- __fwd = find_fw_domain(dev_priv, offset); \
+ __fwd = find_fw_domain(uncore, offset); \
__fwd; \
})
-#define __gen11_fwtable_reg_write_fw_domains(offset) \
+#define __gen11_fwtable_reg_write_fw_domains(uncore, offset) \
({ \
enum forcewake_domains __fwd = 0; \
if (GEN11_NEEDS_FORCE_WAKE((offset)) && !is_gen11_shadowed(offset)) \
- __fwd = find_fw_domain(dev_priv, offset); \
+ __fwd = find_fw_domain(uncore, offset); \
__fwd; \
})
};
static void
-ilk_dummy_write(struct drm_i915_private *dev_priv)
+ilk_dummy_write(struct intel_uncore *uncore)
{
/* WaIssueDummyWriteToWakeupFromRC6:ilk Issue a dummy write to wake up
* the chip from rc6 before touching it for real. MI_MODE is masked,
* hence harmless to write 0 into. */
- __raw_i915_write32(dev_priv, MI_MODE, 0);
+ __raw_uncore_write32(uncore, MI_MODE, 0);
}
static void
-__unclaimed_reg_debug(struct drm_i915_private *dev_priv,
+__unclaimed_reg_debug(struct intel_uncore *uncore,
const i915_reg_t reg,
const bool read,
const bool before)
{
- if (WARN(check_for_unclaimed_mmio(dev_priv) && !before,
+ if (WARN(check_for_unclaimed_mmio(uncore) && !before,
"Unclaimed %s register 0x%x\n",
read ? "read from" : "write to",
i915_mmio_reg_offset(reg)))
}
static inline void
-unclaimed_reg_debug(struct drm_i915_private *dev_priv,
+unclaimed_reg_debug(struct intel_uncore *uncore,
const i915_reg_t reg,
const bool read,
const bool before)
if (likely(!i915_modparams.mmio_debug))
return;
- __unclaimed_reg_debug(dev_priv, reg, read, before);
+ __unclaimed_reg_debug(uncore, reg, read, before);
}
#define GEN2_READ_HEADER(x) \
u##x val = 0; \
- assert_rpm_wakelock_held(dev_priv);
+ __assert_rpm_wakelock_held(uncore->rpm);
#define GEN2_READ_FOOTER \
trace_i915_reg_rw(false, reg, val, sizeof(val), trace); \
#define __gen2_read(x) \
static u##x \
-gen2_read##x(struct drm_i915_private *dev_priv, i915_reg_t reg, bool trace) { \
+gen2_read##x(struct intel_uncore *uncore, i915_reg_t reg, bool trace) { \
GEN2_READ_HEADER(x); \
- val = __raw_i915_read##x(dev_priv, reg); \
+ val = __raw_uncore_read##x(uncore, reg); \
GEN2_READ_FOOTER; \
}
#define __gen5_read(x) \
static u##x \
-gen5_read##x(struct drm_i915_private *dev_priv, i915_reg_t reg, bool trace) { \
+gen5_read##x(struct intel_uncore *uncore, i915_reg_t reg, bool trace) { \
GEN2_READ_HEADER(x); \
- ilk_dummy_write(dev_priv); \
- val = __raw_i915_read##x(dev_priv, reg); \
+ ilk_dummy_write(uncore); \
+ val = __raw_uncore_read##x(uncore, reg); \
GEN2_READ_FOOTER; \
}
u32 offset = i915_mmio_reg_offset(reg); \
unsigned long irqflags; \
u##x val = 0; \
- assert_rpm_wakelock_held(dev_priv); \
- spin_lock_irqsave(&dev_priv->uncore.lock, irqflags); \
- unclaimed_reg_debug(dev_priv, reg, true, true)
+ __assert_rpm_wakelock_held(uncore->rpm); \
+ spin_lock_irqsave(&uncore->lock, irqflags); \
+ unclaimed_reg_debug(uncore, reg, true, true)
#define GEN6_READ_FOOTER \
- unclaimed_reg_debug(dev_priv, reg, true, false); \
- spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags); \
+ unclaimed_reg_debug(uncore, reg, true, false); \
+ spin_unlock_irqrestore(&uncore->lock, irqflags); \
trace_i915_reg_rw(false, reg, val, sizeof(val), trace); \
return val
-static noinline void ___force_wake_auto(struct drm_i915_private *dev_priv,
+static noinline void ___force_wake_auto(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
struct intel_uncore_forcewake_domain *domain;
unsigned int tmp;
- GEM_BUG_ON(fw_domains & ~dev_priv->uncore.fw_domains);
+ GEM_BUG_ON(fw_domains & ~uncore->fw_domains);
- for_each_fw_domain_masked(domain, fw_domains, dev_priv, tmp)
+ for_each_fw_domain_masked(domain, fw_domains, uncore, tmp)
fw_domain_arm_timer(domain);
- dev_priv->uncore.funcs.force_wake_get(dev_priv, fw_domains);
+ uncore->funcs.force_wake_get(uncore, fw_domains);
}
-static inline void __force_wake_auto(struct drm_i915_private *dev_priv,
+static inline void __force_wake_auto(struct intel_uncore *uncore,
enum forcewake_domains fw_domains)
{
if (WARN_ON(!fw_domains))
return;
/* Turn on all requested but inactive supported forcewake domains. */
- fw_domains &= dev_priv->uncore.fw_domains;
- fw_domains &= ~dev_priv->uncore.fw_domains_active;
+ fw_domains &= uncore->fw_domains;
+ fw_domains &= ~uncore->fw_domains_active;
if (fw_domains)
- ___force_wake_auto(dev_priv, fw_domains);
+ ___force_wake_auto(uncore, fw_domains);
}
#define __gen_read(func, x) \
static u##x \
-func##_read##x(struct drm_i915_private *dev_priv, i915_reg_t reg, bool trace) { \
+func##_read##x(struct intel_uncore *uncore, i915_reg_t reg, bool trace) { \
enum forcewake_domains fw_engine; \
GEN6_READ_HEADER(x); \
- fw_engine = __##func##_reg_read_fw_domains(offset); \
+ fw_engine = __##func##_reg_read_fw_domains(uncore, offset); \
if (fw_engine) \
- __force_wake_auto(dev_priv, fw_engine); \
- val = __raw_i915_read##x(dev_priv, reg); \
+ __force_wake_auto(uncore, fw_engine); \
+ val = __raw_uncore_read##x(uncore, reg); \
GEN6_READ_FOOTER; \
}
#define __gen6_read(x) __gen_read(gen6, x)
#define GEN2_WRITE_HEADER \
trace_i915_reg_rw(true, reg, val, sizeof(val), trace); \
- assert_rpm_wakelock_held(dev_priv); \
+ __assert_rpm_wakelock_held(uncore->rpm); \
#define GEN2_WRITE_FOOTER
#define __gen2_write(x) \
static void \
-gen2_write##x(struct drm_i915_private *dev_priv, i915_reg_t reg, u##x val, bool trace) { \
+gen2_write##x(struct intel_uncore *uncore, i915_reg_t reg, u##x val, bool trace) { \
GEN2_WRITE_HEADER; \
- __raw_i915_write##x(dev_priv, reg, val); \
+ __raw_uncore_write##x(uncore, reg, val); \
GEN2_WRITE_FOOTER; \
}
#define __gen5_write(x) \
static void \
-gen5_write##x(struct drm_i915_private *dev_priv, i915_reg_t reg, u##x val, bool trace) { \
+gen5_write##x(struct intel_uncore *uncore, i915_reg_t reg, u##x val, bool trace) { \
GEN2_WRITE_HEADER; \
- ilk_dummy_write(dev_priv); \
- __raw_i915_write##x(dev_priv, reg, val); \
+ ilk_dummy_write(uncore); \
+ __raw_uncore_write##x(uncore, reg, val); \
GEN2_WRITE_FOOTER; \
}
u32 offset = i915_mmio_reg_offset(reg); \
unsigned long irqflags; \
trace_i915_reg_rw(true, reg, val, sizeof(val), trace); \
- assert_rpm_wakelock_held(dev_priv); \
- spin_lock_irqsave(&dev_priv->uncore.lock, irqflags); \
- unclaimed_reg_debug(dev_priv, reg, false, true)
+ __assert_rpm_wakelock_held(uncore->rpm); \
+ spin_lock_irqsave(&uncore->lock, irqflags); \
+ unclaimed_reg_debug(uncore, reg, false, true)
#define GEN6_WRITE_FOOTER \
- unclaimed_reg_debug(dev_priv, reg, false, false); \
- spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags)
+ unclaimed_reg_debug(uncore, reg, false, false); \
+ spin_unlock_irqrestore(&uncore->lock, irqflags)
#define __gen6_write(x) \
static void \
-gen6_write##x(struct drm_i915_private *dev_priv, i915_reg_t reg, u##x val, bool trace) { \
+gen6_write##x(struct intel_uncore *uncore, i915_reg_t reg, u##x val, bool trace) { \
GEN6_WRITE_HEADER; \
if (NEEDS_FORCE_WAKE(offset)) \
- __gen6_gt_wait_for_fifo(dev_priv); \
- __raw_i915_write##x(dev_priv, reg, val); \
+ __gen6_gt_wait_for_fifo(uncore); \
+ __raw_uncore_write##x(uncore, reg, val); \
GEN6_WRITE_FOOTER; \
}
#define __gen_write(func, x) \
static void \
-func##_write##x(struct drm_i915_private *dev_priv, i915_reg_t reg, u##x val, bool trace) { \
+func##_write##x(struct intel_uncore *uncore, i915_reg_t reg, u##x val, bool trace) { \
enum forcewake_domains fw_engine; \
GEN6_WRITE_HEADER; \
- fw_engine = __##func##_reg_write_fw_domains(offset); \
+ fw_engine = __##func##_reg_write_fw_domains(uncore, offset); \
if (fw_engine) \
- __force_wake_auto(dev_priv, fw_engine); \
- __raw_i915_write##x(dev_priv, reg, val); \
+ __force_wake_auto(uncore, fw_engine); \
+ __raw_uncore_write##x(uncore, reg, val); \
GEN6_WRITE_FOOTER; \
}
#define __gen8_write(x) __gen_write(gen8, x)
#undef GEN6_WRITE_FOOTER
#undef GEN6_WRITE_HEADER
-#define ASSIGN_WRITE_MMIO_VFUNCS(i915, x) \
+#define ASSIGN_WRITE_MMIO_VFUNCS(uncore, x) \
do { \
- (i915)->uncore.funcs.mmio_writeb = x##_write8; \
- (i915)->uncore.funcs.mmio_writew = x##_write16; \
- (i915)->uncore.funcs.mmio_writel = x##_write32; \
+ (uncore)->funcs.mmio_writeb = x##_write8; \
+ (uncore)->funcs.mmio_writew = x##_write16; \
+ (uncore)->funcs.mmio_writel = x##_write32; \
} while (0)
-#define ASSIGN_READ_MMIO_VFUNCS(i915, x) \
+#define ASSIGN_READ_MMIO_VFUNCS(uncore, x) \
do { \
- (i915)->uncore.funcs.mmio_readb = x##_read8; \
- (i915)->uncore.funcs.mmio_readw = x##_read16; \
- (i915)->uncore.funcs.mmio_readl = x##_read32; \
- (i915)->uncore.funcs.mmio_readq = x##_read64; \
+ (uncore)->funcs.mmio_readb = x##_read8; \
+ (uncore)->funcs.mmio_readw = x##_read16; \
+ (uncore)->funcs.mmio_readl = x##_read32; \
+ (uncore)->funcs.mmio_readq = x##_read64; \
} while (0)
-static void fw_domain_init(struct drm_i915_private *dev_priv,
+static void fw_domain_init(struct intel_uncore *uncore,
enum forcewake_domain_id domain_id,
i915_reg_t reg_set,
i915_reg_t reg_ack)
if (WARN_ON(domain_id >= FW_DOMAIN_ID_COUNT))
return;
- d = &dev_priv->uncore.fw_domain[domain_id];
+ d = &uncore->fw_domain[domain_id];
WARN_ON(d->wake_count);
WARN_ON(!i915_mmio_reg_valid(reg_ack));
d->wake_count = 0;
- d->reg_set = reg_set;
- d->reg_ack = reg_ack;
+ d->reg_set = uncore->regs + i915_mmio_reg_offset(reg_set);
+ d->reg_ack = uncore->regs + i915_mmio_reg_offset(reg_ack);
d->id = domain_id;
hrtimer_init(&d->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
d->timer.function = intel_uncore_fw_release_timer;
- dev_priv->uncore.fw_domains |= BIT(domain_id);
+ uncore->fw_domains |= BIT(domain_id);
- fw_domain_reset(dev_priv, d);
+ fw_domain_reset(d);
}
-static void fw_domain_fini(struct drm_i915_private *dev_priv,
+static void fw_domain_fini(struct intel_uncore *uncore,
enum forcewake_domain_id domain_id)
{
struct intel_uncore_forcewake_domain *d;
if (WARN_ON(domain_id >= FW_DOMAIN_ID_COUNT))
return;
- d = &dev_priv->uncore.fw_domain[domain_id];
+ d = &uncore->fw_domain[domain_id];
WARN_ON(d->wake_count);
WARN_ON(hrtimer_cancel(&d->timer));
memset(d, 0, sizeof(*d));
- dev_priv->uncore.fw_domains &= ~BIT(domain_id);
+ uncore->fw_domains &= ~BIT(domain_id);
}
-static void intel_uncore_fw_domains_init(struct drm_i915_private *dev_priv)
+static void intel_uncore_fw_domains_init(struct intel_uncore *uncore)
{
- if (INTEL_GEN(dev_priv) <= 5 || intel_vgpu_active(dev_priv))
- return;
+ struct drm_i915_private *i915 = uncore_to_i915(uncore);
- if (IS_GEN(dev_priv, 6)) {
- dev_priv->uncore.fw_reset = 0;
- dev_priv->uncore.fw_set = FORCEWAKE_KERNEL;
- dev_priv->uncore.fw_clear = 0;
- } else {
- /* WaRsClearFWBitsAtReset:bdw,skl */
- dev_priv->uncore.fw_reset = _MASKED_BIT_DISABLE(0xffff);
- dev_priv->uncore.fw_set = _MASKED_BIT_ENABLE(FORCEWAKE_KERNEL);
- dev_priv->uncore.fw_clear = _MASKED_BIT_DISABLE(FORCEWAKE_KERNEL);
- }
+ if (!intel_uncore_has_forcewake(uncore))
+ return;
- if (INTEL_GEN(dev_priv) >= 11) {
+ if (INTEL_GEN(i915) >= 11) {
int i;
- dev_priv->uncore.funcs.force_wake_get =
+ uncore->funcs.force_wake_get =
fw_domains_get_with_fallback;
- dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
- fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+ uncore->funcs.force_wake_put = fw_domains_put;
+ fw_domain_init(uncore, FW_DOMAIN_ID_RENDER,
FORCEWAKE_RENDER_GEN9,
FORCEWAKE_ACK_RENDER_GEN9);
- fw_domain_init(dev_priv, FW_DOMAIN_ID_BLITTER,
+ fw_domain_init(uncore, FW_DOMAIN_ID_BLITTER,
FORCEWAKE_BLITTER_GEN9,
FORCEWAKE_ACK_BLITTER_GEN9);
for (i = 0; i < I915_MAX_VCS; i++) {
- if (!HAS_ENGINE(dev_priv, _VCS(i)))
+ if (!HAS_ENGINE(i915, _VCS(i)))
continue;
- fw_domain_init(dev_priv, FW_DOMAIN_ID_MEDIA_VDBOX0 + i,
+ fw_domain_init(uncore, FW_DOMAIN_ID_MEDIA_VDBOX0 + i,
FORCEWAKE_MEDIA_VDBOX_GEN11(i),
FORCEWAKE_ACK_MEDIA_VDBOX_GEN11(i));
}
for (i = 0; i < I915_MAX_VECS; i++) {
- if (!HAS_ENGINE(dev_priv, _VECS(i)))
+ if (!HAS_ENGINE(i915, _VECS(i)))
continue;
- fw_domain_init(dev_priv, FW_DOMAIN_ID_MEDIA_VEBOX0 + i,
+ fw_domain_init(uncore, FW_DOMAIN_ID_MEDIA_VEBOX0 + i,
FORCEWAKE_MEDIA_VEBOX_GEN11(i),
FORCEWAKE_ACK_MEDIA_VEBOX_GEN11(i));
}
- } else if (IS_GEN_RANGE(dev_priv, 9, 10)) {
- dev_priv->uncore.funcs.force_wake_get =
+ } else if (IS_GEN_RANGE(i915, 9, 10)) {
+ uncore->funcs.force_wake_get =
fw_domains_get_with_fallback;
- dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
- fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+ uncore->funcs.force_wake_put = fw_domains_put;
+ fw_domain_init(uncore, FW_DOMAIN_ID_RENDER,
FORCEWAKE_RENDER_GEN9,
FORCEWAKE_ACK_RENDER_GEN9);
- fw_domain_init(dev_priv, FW_DOMAIN_ID_BLITTER,
+ fw_domain_init(uncore, FW_DOMAIN_ID_BLITTER,
FORCEWAKE_BLITTER_GEN9,
FORCEWAKE_ACK_BLITTER_GEN9);
- fw_domain_init(dev_priv, FW_DOMAIN_ID_MEDIA,
+ fw_domain_init(uncore, FW_DOMAIN_ID_MEDIA,
FORCEWAKE_MEDIA_GEN9, FORCEWAKE_ACK_MEDIA_GEN9);
- } else if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) {
- dev_priv->uncore.funcs.force_wake_get = fw_domains_get;
- dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
- fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+ } else if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) {
+ uncore->funcs.force_wake_get = fw_domains_get;
+ uncore->funcs.force_wake_put = fw_domains_put;
+ fw_domain_init(uncore, FW_DOMAIN_ID_RENDER,
FORCEWAKE_VLV, FORCEWAKE_ACK_VLV);
- fw_domain_init(dev_priv, FW_DOMAIN_ID_MEDIA,
+ fw_domain_init(uncore, FW_DOMAIN_ID_MEDIA,
FORCEWAKE_MEDIA_VLV, FORCEWAKE_ACK_MEDIA_VLV);
- } else if (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv)) {
- dev_priv->uncore.funcs.force_wake_get =
+ } else if (IS_HASWELL(i915) || IS_BROADWELL(i915)) {
+ uncore->funcs.force_wake_get =
fw_domains_get_with_thread_status;
- dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
- fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+ uncore->funcs.force_wake_put = fw_domains_put;
+ fw_domain_init(uncore, FW_DOMAIN_ID_RENDER,
FORCEWAKE_MT, FORCEWAKE_ACK_HSW);
- } else if (IS_IVYBRIDGE(dev_priv)) {
+ } else if (IS_IVYBRIDGE(i915)) {
u32 ecobus;
/* IVB configs may use multi-threaded forcewake */
* (correctly) interpreted by the test below as MT
* forcewake being disabled.
*/
- dev_priv->uncore.funcs.force_wake_get =
+ uncore->funcs.force_wake_get =
fw_domains_get_with_thread_status;
- dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
+ uncore->funcs.force_wake_put = fw_domains_put;
/* We need to init first for ECOBUS access and then
* determine later if we want to reinit, in case of MT access is
* before the ecobus check.
*/
- __raw_i915_write32(dev_priv, FORCEWAKE, 0);
- __raw_posting_read(dev_priv, ECOBUS);
+ __raw_uncore_write32(uncore, FORCEWAKE, 0);
+ __raw_posting_read(uncore, ECOBUS);
- fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+ fw_domain_init(uncore, FW_DOMAIN_ID_RENDER,
FORCEWAKE_MT, FORCEWAKE_MT_ACK);
- spin_lock_irq(&dev_priv->uncore.lock);
- fw_domains_get_with_thread_status(dev_priv, FORCEWAKE_RENDER);
- ecobus = __raw_i915_read32(dev_priv, ECOBUS);
- fw_domains_put(dev_priv, FORCEWAKE_RENDER);
- spin_unlock_irq(&dev_priv->uncore.lock);
+ spin_lock_irq(&uncore->lock);
+ fw_domains_get_with_thread_status(uncore, FORCEWAKE_RENDER);
+ ecobus = __raw_uncore_read32(uncore, ECOBUS);
+ fw_domains_put(uncore, FORCEWAKE_RENDER);
+ spin_unlock_irq(&uncore->lock);
if (!(ecobus & FORCEWAKE_MT_ENABLE)) {
DRM_INFO("No MT forcewake available on Ivybridge, this can result in issues\n");
DRM_INFO("when using vblank-synced partial screen updates.\n");
- fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+ fw_domain_init(uncore, FW_DOMAIN_ID_RENDER,
FORCEWAKE, FORCEWAKE_ACK);
}
- } else if (IS_GEN(dev_priv, 6)) {
- dev_priv->uncore.funcs.force_wake_get =
+ } else if (IS_GEN(i915, 6)) {
+ uncore->funcs.force_wake_get =
fw_domains_get_with_thread_status;
- dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
- fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
+ uncore->funcs.force_wake_put = fw_domains_put;
+ fw_domain_init(uncore, FW_DOMAIN_ID_RENDER,
FORCEWAKE, FORCEWAKE_ACK);
}
/* All future platforms are expected to require complex power gating */
- WARN_ON(dev_priv->uncore.fw_domains == 0);
+ WARN_ON(uncore->fw_domains == 0);
}
-#define ASSIGN_FW_DOMAINS_TABLE(d) \
+#define ASSIGN_FW_DOMAINS_TABLE(uncore, d) \
{ \
- dev_priv->uncore.fw_domains_table = \
+ (uncore)->fw_domains_table = \
(struct intel_forcewake_range *)(d); \
- dev_priv->uncore.fw_domains_table_entries = ARRAY_SIZE((d)); \
+ (uncore)->fw_domains_table_entries = ARRAY_SIZE((d)); \
}
static int i915_pmic_bus_access_notifier(struct notifier_block *nb,
* the access.
*/
disable_rpm_wakeref_asserts(dev_priv);
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(&dev_priv->uncore, FORCEWAKE_ALL);
enable_rpm_wakeref_asserts(dev_priv);
break;
case MBI_PMIC_BUS_ACCESS_END:
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(&dev_priv->uncore, FORCEWAKE_ALL);
break;
}
return NOTIFY_OK;
}
-void intel_uncore_init(struct drm_i915_private *dev_priv)
+static int uncore_mmio_setup(struct intel_uncore *uncore)
{
- i915_check_vgpu(dev_priv);
+ struct drm_i915_private *i915 = uncore_to_i915(uncore);
+ struct pci_dev *pdev = i915->drm.pdev;
+ int mmio_bar;
+ int mmio_size;
- intel_uncore_edram_detect(dev_priv);
- intel_uncore_fw_domains_init(dev_priv);
- __intel_uncore_early_sanitize(dev_priv, 0);
+ mmio_bar = IS_GEN(i915, 2) ? 1 : 0;
+ /*
+ * Before gen4, the registers and the GTT are behind different BARs.
+ * However, from gen4 onwards, the registers and the GTT are shared
+ * in the same BAR, so we want to restrict this ioremap from
+ * clobbering the GTT which we want ioremap_wc instead. Fortunately,
+ * the register BAR remains the same size for all the earlier
+ * generations up to Ironlake.
+ */
+ if (INTEL_GEN(i915) < 5)
+ mmio_size = 512 * 1024;
+ else
+ mmio_size = 2 * 1024 * 1024;
+ uncore->regs = pci_iomap(pdev, mmio_bar, mmio_size);
+ if (uncore->regs == NULL) {
+ DRM_ERROR("failed to map registers\n");
+
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static void uncore_mmio_cleanup(struct intel_uncore *uncore)
+{
+ struct drm_i915_private *i915 = uncore_to_i915(uncore);
+ struct pci_dev *pdev = i915->drm.pdev;
+
+ pci_iounmap(pdev, uncore->regs);
+}
+
+void intel_uncore_init_early(struct intel_uncore *uncore)
+{
+ spin_lock_init(&uncore->lock);
+}
+
+int intel_uncore_init_mmio(struct intel_uncore *uncore)
+{
+ struct drm_i915_private *i915 = uncore_to_i915(uncore);
+ int ret;
+
+ ret = uncore_mmio_setup(uncore);
+ if (ret)
+ return ret;
+
+ i915_check_vgpu(i915);
+
+ if (INTEL_GEN(i915) > 5 && !intel_vgpu_active(i915))
+ uncore->flags |= UNCORE_HAS_FORCEWAKE;
- dev_priv->uncore.unclaimed_mmio_check = 1;
- dev_priv->uncore.pmic_bus_access_nb.notifier_call =
+ intel_uncore_fw_domains_init(uncore);
+ __intel_uncore_early_sanitize(uncore, 0);
+
+ uncore->unclaimed_mmio_check = 1;
+ uncore->pmic_bus_access_nb.notifier_call =
i915_pmic_bus_access_notifier;
- if (IS_GEN_RANGE(dev_priv, 2, 4) || intel_vgpu_active(dev_priv)) {
- ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, gen2);
- ASSIGN_READ_MMIO_VFUNCS(dev_priv, gen2);
- } else if (IS_GEN(dev_priv, 5)) {
- ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, gen5);
- ASSIGN_READ_MMIO_VFUNCS(dev_priv, gen5);
- } else if (IS_GEN_RANGE(dev_priv, 6, 7)) {
- ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, gen6);
-
- if (IS_VALLEYVIEW(dev_priv)) {
- ASSIGN_FW_DOMAINS_TABLE(__vlv_fw_ranges);
- ASSIGN_READ_MMIO_VFUNCS(dev_priv, fwtable);
+ uncore->rpm = &i915->runtime_pm;
+
+ if (!intel_uncore_has_forcewake(uncore)) {
+ if (IS_GEN(i915, 5)) {
+ ASSIGN_WRITE_MMIO_VFUNCS(uncore, gen5);
+ ASSIGN_READ_MMIO_VFUNCS(uncore, gen5);
+ } else {
+ ASSIGN_WRITE_MMIO_VFUNCS(uncore, gen2);
+ ASSIGN_READ_MMIO_VFUNCS(uncore, gen2);
+ }
+ } else if (IS_GEN_RANGE(i915, 6, 7)) {
+ ASSIGN_WRITE_MMIO_VFUNCS(uncore, gen6);
+
+ if (IS_VALLEYVIEW(i915)) {
+ ASSIGN_FW_DOMAINS_TABLE(uncore, __vlv_fw_ranges);
+ ASSIGN_READ_MMIO_VFUNCS(uncore, fwtable);
} else {
- ASSIGN_READ_MMIO_VFUNCS(dev_priv, gen6);
+ ASSIGN_READ_MMIO_VFUNCS(uncore, gen6);
}
- } else if (IS_GEN(dev_priv, 8)) {
- if (IS_CHERRYVIEW(dev_priv)) {
- ASSIGN_FW_DOMAINS_TABLE(__chv_fw_ranges);
- ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, fwtable);
- ASSIGN_READ_MMIO_VFUNCS(dev_priv, fwtable);
+ } else if (IS_GEN(i915, 8)) {
+ if (IS_CHERRYVIEW(i915)) {
+ ASSIGN_FW_DOMAINS_TABLE(uncore, __chv_fw_ranges);
+ ASSIGN_WRITE_MMIO_VFUNCS(uncore, fwtable);
+ ASSIGN_READ_MMIO_VFUNCS(uncore, fwtable);
} else {
- ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, gen8);
- ASSIGN_READ_MMIO_VFUNCS(dev_priv, gen6);
+ ASSIGN_WRITE_MMIO_VFUNCS(uncore, gen8);
+ ASSIGN_READ_MMIO_VFUNCS(uncore, gen6);
}
- } else if (IS_GEN_RANGE(dev_priv, 9, 10)) {
- ASSIGN_FW_DOMAINS_TABLE(__gen9_fw_ranges);
- ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, fwtable);
- ASSIGN_READ_MMIO_VFUNCS(dev_priv, fwtable);
+ } else if (IS_GEN_RANGE(i915, 9, 10)) {
+ ASSIGN_FW_DOMAINS_TABLE(uncore, __gen9_fw_ranges);
+ ASSIGN_WRITE_MMIO_VFUNCS(uncore, fwtable);
+ ASSIGN_READ_MMIO_VFUNCS(uncore, fwtable);
} else {
- ASSIGN_FW_DOMAINS_TABLE(__gen11_fw_ranges);
- ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, gen11_fwtable);
- ASSIGN_READ_MMIO_VFUNCS(dev_priv, gen11_fwtable);
+ ASSIGN_FW_DOMAINS_TABLE(uncore, __gen11_fw_ranges);
+ ASSIGN_WRITE_MMIO_VFUNCS(uncore, gen11_fwtable);
+ ASSIGN_READ_MMIO_VFUNCS(uncore, gen11_fwtable);
}
- iosf_mbi_register_pmic_bus_access_notifier(
- &dev_priv->uncore.pmic_bus_access_nb);
+ if (HAS_FPGA_DBG_UNCLAIMED(i915))
+ uncore->flags |= UNCORE_HAS_FPGA_DBG_UNCLAIMED;
+
+ if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915))
+ uncore->flags |= UNCORE_HAS_DBG_UNCLAIMED;
+
+ if (IS_GEN_RANGE(i915, 6, 7))
+ uncore->flags |= UNCORE_HAS_FIFO;
+
+ iosf_mbi_register_pmic_bus_access_notifier(&uncore->pmic_bus_access_nb);
+
+ return 0;
}
/*
* the forcewake domains. Prune them, to make sure they only reference existing
* engines.
*/
-void intel_uncore_prune(struct drm_i915_private *dev_priv)
+void intel_uncore_prune_mmio_domains(struct intel_uncore *uncore)
{
- if (INTEL_GEN(dev_priv) >= 11) {
- enum forcewake_domains fw_domains = dev_priv->uncore.fw_domains;
+ struct drm_i915_private *i915 = uncore_to_i915(uncore);
+
+ if (INTEL_GEN(i915) >= 11) {
+ enum forcewake_domains fw_domains = uncore->fw_domains;
enum forcewake_domain_id domain_id;
int i;
for (i = 0; i < I915_MAX_VCS; i++) {
domain_id = FW_DOMAIN_ID_MEDIA_VDBOX0 + i;
- if (HAS_ENGINE(dev_priv, _VCS(i)))
+ if (HAS_ENGINE(i915, _VCS(i)))
continue;
if (fw_domains & BIT(domain_id))
- fw_domain_fini(dev_priv, domain_id);
+ fw_domain_fini(uncore, domain_id);
}
for (i = 0; i < I915_MAX_VECS; i++) {
domain_id = FW_DOMAIN_ID_MEDIA_VEBOX0 + i;
- if (HAS_ENGINE(dev_priv, _VECS(i)))
+ if (HAS_ENGINE(i915, _VECS(i)))
continue;
if (fw_domains & BIT(domain_id))
- fw_domain_fini(dev_priv, domain_id);
+ fw_domain_fini(uncore, domain_id);
}
}
}
-void intel_uncore_fini(struct drm_i915_private *dev_priv)
+void intel_uncore_fini_mmio(struct intel_uncore *uncore)
{
/* Paranoia: make sure we have disabled everything before we exit. */
- intel_uncore_sanitize(dev_priv);
+ intel_uncore_sanitize(uncore_to_i915(uncore));
iosf_mbi_punit_acquire();
iosf_mbi_unregister_pmic_bus_access_notifier_unlocked(
- &dev_priv->uncore.pmic_bus_access_nb);
- intel_uncore_forcewake_reset(dev_priv);
+ &uncore->pmic_bus_access_nb);
+ intel_uncore_forcewake_reset(uncore);
iosf_mbi_punit_release();
+ uncore_mmio_cleanup(uncore);
}
static const struct reg_whitelist {
/**
* __intel_wait_for_register_fw - wait until register matches expected state
- * @dev_priv: the i915 device
+ * @uncore: the struct intel_uncore
* @reg: the register to read
* @mask: mask to apply to register value
* @value: expected value
*
* Returns 0 if the register matches the desired condition, or -ETIMEOUT.
*/
-int __intel_wait_for_register_fw(struct drm_i915_private *dev_priv,
+int __intel_wait_for_register_fw(struct intel_uncore *uncore,
i915_reg_t reg,
u32 mask,
u32 value,
u32 *out_value)
{
u32 uninitialized_var(reg_value);
-#define done (((reg_value = I915_READ_FW(reg)) & mask) == value)
+#define done (((reg_value = intel_uncore_read_fw(uncore, reg)) & mask) == value)
int ret;
/* Catch any overuse of this function */
/**
* __intel_wait_for_register - wait until register matches expected state
- * @dev_priv: the i915 device
+ * @uncore: the struct intel_uncore
* @reg: the register to read
* @mask: mask to apply to register value
* @value: expected value
*
* Returns 0 if the register matches the desired condition, or -ETIMEOUT.
*/
-int __intel_wait_for_register(struct drm_i915_private *dev_priv,
- i915_reg_t reg,
- u32 mask,
- u32 value,
- unsigned int fast_timeout_us,
- unsigned int slow_timeout_ms,
- u32 *out_value)
+int __intel_wait_for_register(struct intel_uncore *uncore,
+ i915_reg_t reg,
+ u32 mask,
+ u32 value,
+ unsigned int fast_timeout_us,
+ unsigned int slow_timeout_ms,
+ u32 *out_value)
{
unsigned fw =
- intel_uncore_forcewake_for_reg(dev_priv, reg, FW_REG_READ);
+ intel_uncore_forcewake_for_reg(uncore, reg, FW_REG_READ);
u32 reg_value;
int ret;
might_sleep_if(slow_timeout_ms);
- spin_lock_irq(&dev_priv->uncore.lock);
- intel_uncore_forcewake_get__locked(dev_priv, fw);
+ spin_lock_irq(&uncore->lock);
+ intel_uncore_forcewake_get__locked(uncore, fw);
- ret = __intel_wait_for_register_fw(dev_priv,
+ ret = __intel_wait_for_register_fw(uncore,
reg, mask, value,
fast_timeout_us, 0, ®_value);
- intel_uncore_forcewake_put__locked(dev_priv, fw);
- spin_unlock_irq(&dev_priv->uncore.lock);
+ intel_uncore_forcewake_put__locked(uncore, fw);
+ spin_unlock_irq(&uncore->lock);
if (ret && slow_timeout_ms)
- ret = __wait_for(reg_value = I915_READ_NOTRACE(reg),
+ ret = __wait_for(reg_value = intel_uncore_read_notrace(uncore,
+ reg),
(reg_value & mask) == value,
slow_timeout_ms * 1000, 10, 1000);
return ret;
}
-bool intel_uncore_unclaimed_mmio(struct drm_i915_private *dev_priv)
+bool intel_uncore_unclaimed_mmio(struct intel_uncore *uncore)
{
- return check_for_unclaimed_mmio(dev_priv);
+ return check_for_unclaimed_mmio(uncore);
}
bool
-intel_uncore_arm_unclaimed_mmio_detection(struct drm_i915_private *dev_priv)
+intel_uncore_arm_unclaimed_mmio_detection(struct intel_uncore *uncore)
{
bool ret = false;
- spin_lock_irq(&dev_priv->uncore.lock);
+ spin_lock_irq(&uncore->lock);
- if (unlikely(dev_priv->uncore.unclaimed_mmio_check <= 0))
+ if (unlikely(uncore->unclaimed_mmio_check <= 0))
goto out;
- if (unlikely(intel_uncore_unclaimed_mmio(dev_priv))) {
+ if (unlikely(intel_uncore_unclaimed_mmio(uncore))) {
if (!i915_modparams.mmio_debug) {
DRM_DEBUG("Unclaimed register detected, "
"enabling oneshot unclaimed register reporting. "
"Please use i915.mmio_debug=N for more information.\n");
i915_modparams.mmio_debug++;
}
- dev_priv->uncore.unclaimed_mmio_check--;
+ uncore->unclaimed_mmio_check--;
ret = true;
}
out:
- spin_unlock_irq(&dev_priv->uncore.lock);
+ spin_unlock_irq(&uncore->lock);
return ret;
}
static enum forcewake_domains
-intel_uncore_forcewake_for_read(struct drm_i915_private *dev_priv,
+intel_uncore_forcewake_for_read(struct intel_uncore *uncore,
i915_reg_t reg)
{
+ struct drm_i915_private *i915 = uncore_to_i915(uncore);
u32 offset = i915_mmio_reg_offset(reg);
enum forcewake_domains fw_domains;
- if (INTEL_GEN(dev_priv) >= 11) {
- fw_domains = __gen11_fwtable_reg_read_fw_domains(offset);
- } else if (HAS_FWTABLE(dev_priv)) {
- fw_domains = __fwtable_reg_read_fw_domains(offset);
- } else if (INTEL_GEN(dev_priv) >= 6) {
- fw_domains = __gen6_reg_read_fw_domains(offset);
+ if (INTEL_GEN(i915) >= 11) {
+ fw_domains = __gen11_fwtable_reg_read_fw_domains(uncore, offset);
+ } else if (HAS_FWTABLE(i915)) {
+ fw_domains = __fwtable_reg_read_fw_domains(uncore, offset);
+ } else if (INTEL_GEN(i915) >= 6) {
+ fw_domains = __gen6_reg_read_fw_domains(uncore, offset);
} else {
- WARN_ON(!IS_GEN_RANGE(dev_priv, 2, 5));
+ /* on devices with FW we expect to hit one of the above cases */
+ if (intel_uncore_has_forcewake(uncore))
+ MISSING_CASE(INTEL_GEN(i915));
+
fw_domains = 0;
}
- WARN_ON(fw_domains & ~dev_priv->uncore.fw_domains);
+ WARN_ON(fw_domains & ~uncore->fw_domains);
return fw_domains;
}
static enum forcewake_domains
-intel_uncore_forcewake_for_write(struct drm_i915_private *dev_priv,
+intel_uncore_forcewake_for_write(struct intel_uncore *uncore,
i915_reg_t reg)
{
+ struct drm_i915_private *i915 = uncore_to_i915(uncore);
u32 offset = i915_mmio_reg_offset(reg);
enum forcewake_domains fw_domains;
- if (INTEL_GEN(dev_priv) >= 11) {
- fw_domains = __gen11_fwtable_reg_write_fw_domains(offset);
- } else if (HAS_FWTABLE(dev_priv) && !IS_VALLEYVIEW(dev_priv)) {
- fw_domains = __fwtable_reg_write_fw_domains(offset);
- } else if (IS_GEN(dev_priv, 8)) {
- fw_domains = __gen8_reg_write_fw_domains(offset);
- } else if (IS_GEN_RANGE(dev_priv, 6, 7)) {
+ if (INTEL_GEN(i915) >= 11) {
+ fw_domains = __gen11_fwtable_reg_write_fw_domains(uncore, offset);
+ } else if (HAS_FWTABLE(i915) && !IS_VALLEYVIEW(i915)) {
+ fw_domains = __fwtable_reg_write_fw_domains(uncore, offset);
+ } else if (IS_GEN(i915, 8)) {
+ fw_domains = __gen8_reg_write_fw_domains(uncore, offset);
+ } else if (IS_GEN_RANGE(i915, 6, 7)) {
fw_domains = FORCEWAKE_RENDER;
} else {
- WARN_ON(!IS_GEN_RANGE(dev_priv, 2, 5));
+ /* on devices with FW we expect to hit one of the above cases */
+ if (intel_uncore_has_forcewake(uncore))
+ MISSING_CASE(INTEL_GEN(i915));
+
fw_domains = 0;
}
- WARN_ON(fw_domains & ~dev_priv->uncore.fw_domains);
+ WARN_ON(fw_domains & ~uncore->fw_domains);
return fw_domains;
}
/**
* intel_uncore_forcewake_for_reg - which forcewake domains are needed to access
* a register
- * @dev_priv: pointer to struct drm_i915_private
+ * @uncore: pointer to struct intel_uncore
* @reg: register in question
* @op: operation bitmask of FW_REG_READ and/or FW_REG_WRITE
*
* callers to do FIFO management on their own or risk losing writes.
*/
enum forcewake_domains
-intel_uncore_forcewake_for_reg(struct drm_i915_private *dev_priv,
+intel_uncore_forcewake_for_reg(struct intel_uncore *uncore,
i915_reg_t reg, unsigned int op)
{
enum forcewake_domains fw_domains = 0;
WARN_ON(!op);
- if (intel_vgpu_active(dev_priv))
+ if (!intel_uncore_has_forcewake(uncore))
return 0;
if (op & FW_REG_READ)
- fw_domains = intel_uncore_forcewake_for_read(dev_priv, reg);
+ fw_domains = intel_uncore_forcewake_for_read(uncore, reg);
if (op & FW_REG_WRITE)
- fw_domains |= intel_uncore_forcewake_for_write(dev_priv, reg);
+ fw_domains |= intel_uncore_forcewake_for_write(uncore, reg);
return fw_domains;
}
#include <linux/spinlock.h>
#include <linux/notifier.h>
#include <linux/hrtimer.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
#include "i915_reg.h"
struct drm_i915_private;
+struct i915_runtime_pm;
+struct intel_uncore;
enum forcewake_domain_id {
FW_DOMAIN_ID_RENDER = 0,
};
struct intel_uncore_funcs {
- void (*force_wake_get)(struct drm_i915_private *dev_priv,
+ void (*force_wake_get)(struct intel_uncore *uncore,
enum forcewake_domains domains);
- void (*force_wake_put)(struct drm_i915_private *dev_priv,
+ void (*force_wake_put)(struct intel_uncore *uncore,
enum forcewake_domains domains);
- u8 (*mmio_readb)(struct drm_i915_private *dev_priv,
+ u8 (*mmio_readb)(struct intel_uncore *uncore,
i915_reg_t r, bool trace);
- u16 (*mmio_readw)(struct drm_i915_private *dev_priv,
+ u16 (*mmio_readw)(struct intel_uncore *uncore,
i915_reg_t r, bool trace);
- u32 (*mmio_readl)(struct drm_i915_private *dev_priv,
+ u32 (*mmio_readl)(struct intel_uncore *uncore,
i915_reg_t r, bool trace);
- u64 (*mmio_readq)(struct drm_i915_private *dev_priv,
+ u64 (*mmio_readq)(struct intel_uncore *uncore,
i915_reg_t r, bool trace);
- void (*mmio_writeb)(struct drm_i915_private *dev_priv,
+ void (*mmio_writeb)(struct intel_uncore *uncore,
i915_reg_t r, u8 val, bool trace);
- void (*mmio_writew)(struct drm_i915_private *dev_priv,
+ void (*mmio_writew)(struct intel_uncore *uncore,
i915_reg_t r, u16 val, bool trace);
- void (*mmio_writel)(struct drm_i915_private *dev_priv,
+ void (*mmio_writel)(struct intel_uncore *uncore,
i915_reg_t r, u32 val, bool trace);
};
};
struct intel_uncore {
+ void __iomem *regs;
+
+ struct i915_runtime_pm *rpm;
+
spinlock_t lock; /** lock is also taken in irq contexts. */
+ unsigned int flags;
+#define UNCORE_HAS_FORCEWAKE BIT(0)
+#define UNCORE_HAS_FPGA_DBG_UNCLAIMED BIT(1)
+#define UNCORE_HAS_DBG_UNCLAIMED BIT(2)
+#define UNCORE_HAS_FIFO BIT(3)
+
const struct intel_forcewake_range *fw_domains_table;
unsigned int fw_domains_table_entries;
enum forcewake_domains fw_domains_active;
enum forcewake_domains fw_domains_saved; /* user domains saved for S3 */
- u32 fw_set;
- u32 fw_clear;
- u32 fw_reset;
-
struct intel_uncore_forcewake_domain {
enum forcewake_domain_id id;
enum forcewake_domains mask;
unsigned int wake_count;
bool active;
struct hrtimer timer;
- i915_reg_t reg_set;
- i915_reg_t reg_ack;
+ u32 __iomem *reg_set;
+ u32 __iomem *reg_ack;
} fw_domain[FW_DOMAIN_ID_COUNT];
struct {
};
/* Iterate over initialised fw domains */
-#define for_each_fw_domain_masked(domain__, mask__, dev_priv__, tmp__) \
+#define for_each_fw_domain_masked(domain__, mask__, uncore__, tmp__) \
for (tmp__ = (mask__); \
- tmp__ ? (domain__ = &(dev_priv__)->uncore.fw_domain[__mask_next_bit(tmp__)]), 1 : 0;)
+ tmp__ ? (domain__ = &(uncore__)->fw_domain[__mask_next_bit(tmp__)]), 1 : 0;)
+
+#define for_each_fw_domain(domain__, uncore__, tmp__) \
+ for_each_fw_domain_masked(domain__, (uncore__)->fw_domains, uncore__, tmp__)
+
+static inline struct intel_uncore *
+forcewake_domain_to_uncore(const struct intel_uncore_forcewake_domain *d)
+{
+ return container_of(d, struct intel_uncore, fw_domain[d->id]);
+}
-#define for_each_fw_domain(domain__, dev_priv__, tmp__) \
- for_each_fw_domain_masked(domain__, (dev_priv__)->uncore.fw_domains, dev_priv__, tmp__)
+static inline bool
+intel_uncore_has_forcewake(const struct intel_uncore *uncore)
+{
+ return uncore->flags & UNCORE_HAS_FORCEWAKE;
+}
+static inline bool
+intel_uncore_has_fpga_dbg_unclaimed(const struct intel_uncore *uncore)
+{
+ return uncore->flags & UNCORE_HAS_FPGA_DBG_UNCLAIMED;
+}
+
+static inline bool
+intel_uncore_has_dbg_unclaimed(const struct intel_uncore *uncore)
+{
+ return uncore->flags & UNCORE_HAS_DBG_UNCLAIMED;
+}
+
+static inline bool
+intel_uncore_has_fifo(const struct intel_uncore *uncore)
+{
+ return uncore->flags & UNCORE_HAS_FIFO;
+}
void intel_uncore_sanitize(struct drm_i915_private *dev_priv);
-void intel_uncore_init(struct drm_i915_private *dev_priv);
-void intel_uncore_prune(struct drm_i915_private *dev_priv);
-bool intel_uncore_unclaimed_mmio(struct drm_i915_private *dev_priv);
-bool intel_uncore_arm_unclaimed_mmio_detection(struct drm_i915_private *dev_priv);
-void intel_uncore_fini(struct drm_i915_private *dev_priv);
-void intel_uncore_suspend(struct drm_i915_private *dev_priv);
-void intel_uncore_resume_early(struct drm_i915_private *dev_priv);
-void intel_uncore_runtime_resume(struct drm_i915_private *dev_priv);
-
-u64 intel_uncore_edram_size(struct drm_i915_private *dev_priv);
-void assert_forcewakes_inactive(struct drm_i915_private *dev_priv);
-void assert_forcewakes_active(struct drm_i915_private *dev_priv,
+void intel_uncore_init_early(struct intel_uncore *uncore);
+int intel_uncore_init_mmio(struct intel_uncore *uncore);
+void intel_uncore_prune_mmio_domains(struct intel_uncore *uncore);
+bool intel_uncore_unclaimed_mmio(struct intel_uncore *uncore);
+bool intel_uncore_arm_unclaimed_mmio_detection(struct intel_uncore *uncore);
+void intel_uncore_fini_mmio(struct intel_uncore *uncore);
+void intel_uncore_suspend(struct intel_uncore *uncore);
+void intel_uncore_resume_early(struct intel_uncore *uncore);
+void intel_uncore_runtime_resume(struct intel_uncore *uncore);
+
+void assert_forcewakes_inactive(struct intel_uncore *uncore);
+void assert_forcewakes_active(struct intel_uncore *uncore,
enum forcewake_domains fw_domains);
const char *intel_uncore_forcewake_domain_to_str(const enum forcewake_domain_id id);
enum forcewake_domains
-intel_uncore_forcewake_for_reg(struct drm_i915_private *dev_priv,
+intel_uncore_forcewake_for_reg(struct intel_uncore *uncore,
i915_reg_t reg, unsigned int op);
#define FW_REG_READ (1)
#define FW_REG_WRITE (2)
-void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
+void intel_uncore_forcewake_get(struct intel_uncore *uncore,
enum forcewake_domains domains);
-void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
+void intel_uncore_forcewake_put(struct intel_uncore *uncore,
enum forcewake_domains domains);
/* Like above but the caller must manage the uncore.lock itself.
* Must be used with I915_READ_FW and friends.
*/
-void intel_uncore_forcewake_get__locked(struct drm_i915_private *dev_priv,
+void intel_uncore_forcewake_get__locked(struct intel_uncore *uncore,
enum forcewake_domains domains);
-void intel_uncore_forcewake_put__locked(struct drm_i915_private *dev_priv,
+void intel_uncore_forcewake_put__locked(struct intel_uncore *uncore,
enum forcewake_domains domains);
-void intel_uncore_forcewake_user_get(struct drm_i915_private *dev_priv);
-void intel_uncore_forcewake_user_put(struct drm_i915_private *dev_priv);
+void intel_uncore_forcewake_user_get(struct intel_uncore *uncore);
+void intel_uncore_forcewake_user_put(struct intel_uncore *uncore);
-int __intel_wait_for_register(struct drm_i915_private *dev_priv,
+int __intel_wait_for_register(struct intel_uncore *uncore,
i915_reg_t reg,
u32 mask,
u32 value,
unsigned int fast_timeout_us,
unsigned int slow_timeout_ms,
u32 *out_value);
-static inline
-int intel_wait_for_register(struct drm_i915_private *dev_priv,
- i915_reg_t reg,
- u32 mask,
- u32 value,
- unsigned int timeout_ms)
+static inline int
+intel_wait_for_register(struct intel_uncore *uncore,
+ i915_reg_t reg,
+ u32 mask,
+ u32 value,
+ unsigned int timeout_ms)
{
- return __intel_wait_for_register(dev_priv, reg, mask, value, 2,
+ return __intel_wait_for_register(uncore, reg, mask, value, 2,
timeout_ms, NULL);
}
-int __intel_wait_for_register_fw(struct drm_i915_private *dev_priv,
+
+int __intel_wait_for_register_fw(struct intel_uncore *uncore,
i915_reg_t reg,
u32 mask,
u32 value,
unsigned int fast_timeout_us,
unsigned int slow_timeout_ms,
u32 *out_value);
-static inline
-int intel_wait_for_register_fw(struct drm_i915_private *dev_priv,
- i915_reg_t reg,
- u32 mask,
- u32 value,
+static inline int
+intel_wait_for_register_fw(struct intel_uncore *uncore,
+ i915_reg_t reg,
+ u32 mask,
+ u32 value,
unsigned int timeout_ms)
{
- return __intel_wait_for_register_fw(dev_priv, reg, mask, value,
+ return __intel_wait_for_register_fw(uncore, reg, mask, value,
2, timeout_ms, NULL);
}
+/* register access functions */
+#define __raw_read(x__, s__) \
+static inline u##x__ __raw_uncore_read##x__(const struct intel_uncore *uncore, \
+ i915_reg_t reg) \
+{ \
+ return read##s__(uncore->regs + i915_mmio_reg_offset(reg)); \
+}
+
+#define __raw_write(x__, s__) \
+static inline void __raw_uncore_write##x__(const struct intel_uncore *uncore, \
+ i915_reg_t reg, u##x__ val) \
+{ \
+ write##s__(val, uncore->regs + i915_mmio_reg_offset(reg)); \
+}
+__raw_read(8, b)
+__raw_read(16, w)
+__raw_read(32, l)
+__raw_read(64, q)
+
+__raw_write(8, b)
+__raw_write(16, w)
+__raw_write(32, l)
+__raw_write(64, q)
+
+#undef __raw_read
+#undef __raw_write
+
+#define __uncore_read(name__, x__, s__, trace__) \
+static inline u##x__ intel_uncore_##name__(struct intel_uncore *uncore, \
+ i915_reg_t reg) \
+{ \
+ return uncore->funcs.mmio_read##s__(uncore, reg, (trace__)); \
+}
+
+#define __uncore_write(name__, x__, s__, trace__) \
+static inline void intel_uncore_##name__(struct intel_uncore *uncore, \
+ i915_reg_t reg, u##x__ val) \
+{ \
+ uncore->funcs.mmio_write##s__(uncore, reg, val, (trace__)); \
+}
+
+__uncore_read(read8, 8, b, true)
+__uncore_read(read16, 16, w, true)
+__uncore_read(read, 32, l, true)
+__uncore_read(read16_notrace, 16, w, false)
+__uncore_read(read_notrace, 32, l, false)
+
+__uncore_write(write8, 8, b, true)
+__uncore_write(write16, 16, w, true)
+__uncore_write(write, 32, l, true)
+__uncore_write(write_notrace, 32, l, false)
+
+/* Be very careful with read/write 64-bit values. On 32-bit machines, they
+ * will be implemented using 2 32-bit writes in an arbitrary order with
+ * an arbitrary delay between them. This can cause the hardware to
+ * act upon the intermediate value, possibly leading to corruption and
+ * machine death. For this reason we do not support I915_WRITE64, or
+ * uncore->funcs.mmio_writeq.
+ *
+ * When reading a 64-bit value as two 32-bit values, the delay may cause
+ * the two reads to mismatch, e.g. a timestamp overflowing. Also note that
+ * occasionally a 64-bit register does not actually support a full readq
+ * and must be read using two 32-bit reads.
+ *
+ * You have been warned.
+ */
+__uncore_read(read64, 64, q, true)
+
+static inline u64
+intel_uncore_read64_2x32(struct intel_uncore *uncore,
+ i915_reg_t lower_reg, i915_reg_t upper_reg)
+{
+ u32 upper, lower, old_upper, loop = 0;
+ upper = intel_uncore_read(uncore, upper_reg);
+ do {
+ old_upper = upper;
+ lower = intel_uncore_read(uncore, lower_reg);
+ upper = intel_uncore_read(uncore, upper_reg);
+ } while (upper != old_upper && loop++ < 2);
+ return (u64)upper << 32 | lower;
+}
+
+#define intel_uncore_posting_read(...) ((void)intel_uncore_read_notrace(__VA_ARGS__))
+#define intel_uncore_posting_read16(...) ((void)intel_uncore_read16_notrace(__VA_ARGS__))
+
+#undef __uncore_read
+#undef __uncore_write
+
+/* These are untraced mmio-accessors that are only valid to be used inside
+ * critical sections, such as inside IRQ handlers, where forcewake is explicitly
+ * controlled.
+ *
+ * Think twice, and think again, before using these.
+ *
+ * As an example, these accessors can possibly be used between:
+ *
+ * spin_lock_irq(&uncore->lock);
+ * intel_uncore_forcewake_get__locked();
+ *
+ * and
+ *
+ * intel_uncore_forcewake_put__locked();
+ * spin_unlock_irq(&uncore->lock);
+ *
+ *
+ * Note: some registers may not need forcewake held, so
+ * intel_uncore_forcewake_{get,put} can be omitted, see
+ * intel_uncore_forcewake_for_reg().
+ *
+ * Certain architectures will die if the same cacheline is concurrently accessed
+ * by different clients (e.g. on Ivybridge). Access to registers should
+ * therefore generally be serialised, by either the dev_priv->uncore.lock or
+ * a more localised lock guarding all access to that bank of registers.
+ */
+#define intel_uncore_read_fw(...) __raw_uncore_read32(__VA_ARGS__)
+#define intel_uncore_write_fw(...) __raw_uncore_write32(__VA_ARGS__)
+#define intel_uncore_write64_fw(...) __raw_uncore_write64(__VA_ARGS__)
+#define intel_uncore_posting_read_fw(...) ((void)intel_uncore_read_fw(__VA_ARGS__))
+
+static inline void intel_uncore_rmw(struct intel_uncore *uncore,
+ i915_reg_t reg, u32 clear, u32 set)
+{
+ u32 val;
+
+ val = intel_uncore_read(uncore, reg);
+ val &= ~clear;
+ val |= set;
+ intel_uncore_write(uncore, reg, val);
+}
+
+static inline void intel_uncore_rmw_fw(struct intel_uncore *uncore,
+ i915_reg_t reg, u32 clear, u32 set)
+{
+ u32 val;
+
+ val = intel_uncore_read_fw(uncore, reg);
+ val &= ~clear;
+ val |= set;
+ intel_uncore_write_fw(uncore, reg, val);
+}
+
#define raw_reg_read(base, reg) \
readl(base + i915_mmio_reg_offset(reg))
#define raw_reg_write(base, reg, value) \
/* TP wake up time in multiple of 100 */
u16 tp1_wakeup_time;
u16 tp2_tp3_wakeup_time;
+
+ /* PSR2 TP2/TP3 wakeup time for 16 panels */
+ u32 psr2_tp2_tp3_wakeup_time;
} __packed;
struct bdb_psr {
}
}
-static int intel_compute_rc_parameters(struct drm_dsc_config *vdsc_cfg)
-{
- unsigned long groups_per_line = 0;
- unsigned long groups_total = 0;
- unsigned long num_extra_mux_bits = 0;
- unsigned long slice_bits = 0;
- unsigned long hrd_delay = 0;
- unsigned long final_scale = 0;
- unsigned long rbs_min = 0;
-
- /* Number of groups used to code each line of a slice */
- groups_per_line = DIV_ROUND_UP(vdsc_cfg->slice_width,
- DSC_RC_PIXELS_PER_GROUP);
-
- /* chunksize in Bytes */
- vdsc_cfg->slice_chunk_size = DIV_ROUND_UP(vdsc_cfg->slice_width *
- vdsc_cfg->bits_per_pixel,
- (8 * 16));
-
- if (vdsc_cfg->convert_rgb)
- num_extra_mux_bits = 3 * (vdsc_cfg->mux_word_size +
- (4 * vdsc_cfg->bits_per_component + 4)
- - 2);
- else
- num_extra_mux_bits = 3 * vdsc_cfg->mux_word_size +
- (4 * vdsc_cfg->bits_per_component + 4) +
- 2 * (4 * vdsc_cfg->bits_per_component) - 2;
- /* Number of bits in one Slice */
- slice_bits = 8 * vdsc_cfg->slice_chunk_size * vdsc_cfg->slice_height;
-
- while ((num_extra_mux_bits > 0) &&
- ((slice_bits - num_extra_mux_bits) % vdsc_cfg->mux_word_size))
- num_extra_mux_bits--;
-
- if (groups_per_line < vdsc_cfg->initial_scale_value - 8)
- vdsc_cfg->initial_scale_value = groups_per_line + 8;
-
- /* scale_decrement_interval calculation according to DSC spec 1.11 */
- if (vdsc_cfg->initial_scale_value > 8)
- vdsc_cfg->scale_decrement_interval = groups_per_line /
- (vdsc_cfg->initial_scale_value - 8);
- else
- vdsc_cfg->scale_decrement_interval = DSC_SCALE_DECREMENT_INTERVAL_MAX;
-
- vdsc_cfg->final_offset = vdsc_cfg->rc_model_size -
- (vdsc_cfg->initial_xmit_delay *
- vdsc_cfg->bits_per_pixel + 8) / 16 + num_extra_mux_bits;
-
- if (vdsc_cfg->final_offset >= vdsc_cfg->rc_model_size) {
- DRM_DEBUG_KMS("FinalOfs < RcModelSze for this InitialXmitDelay\n");
- return -ERANGE;
- }
-
- final_scale = (vdsc_cfg->rc_model_size * 8) /
- (vdsc_cfg->rc_model_size - vdsc_cfg->final_offset);
- if (vdsc_cfg->slice_height > 1)
- /*
- * NflBpgOffset is 16 bit value with 11 fractional bits
- * hence we multiply by 2^11 for preserving the
- * fractional part
- */
- vdsc_cfg->nfl_bpg_offset = DIV_ROUND_UP((vdsc_cfg->first_line_bpg_offset << 11),
- (vdsc_cfg->slice_height - 1));
- else
- vdsc_cfg->nfl_bpg_offset = 0;
-
- /* 2^16 - 1 */
- if (vdsc_cfg->nfl_bpg_offset > 65535) {
- DRM_DEBUG_KMS("NflBpgOffset is too large for this slice height\n");
- return -ERANGE;
- }
-
- /* Number of groups used to code the entire slice */
- groups_total = groups_per_line * vdsc_cfg->slice_height;
-
- /* slice_bpg_offset is 16 bit value with 11 fractional bits */
- vdsc_cfg->slice_bpg_offset = DIV_ROUND_UP(((vdsc_cfg->rc_model_size -
- vdsc_cfg->initial_offset +
- num_extra_mux_bits) << 11),
- groups_total);
-
- if (final_scale > 9) {
- /*
- * ScaleIncrementInterval =
- * finaloffset/((NflBpgOffset + SliceBpgOffset)*8(finalscale - 1.125))
- * as (NflBpgOffset + SliceBpgOffset) has 11 bit fractional value,
- * we need divide by 2^11 from pstDscCfg values
- */
- vdsc_cfg->scale_increment_interval =
- (vdsc_cfg->final_offset * (1 << 11)) /
- ((vdsc_cfg->nfl_bpg_offset +
- vdsc_cfg->slice_bpg_offset) *
- (final_scale - 9));
- } else {
- /*
- * If finalScaleValue is less than or equal to 9, a value of 0 should
- * be used to disable the scale increment at the end of the slice
- */
- vdsc_cfg->scale_increment_interval = 0;
- }
-
- if (vdsc_cfg->scale_increment_interval > 65535) {
- DRM_DEBUG_KMS("ScaleIncrementInterval is large for slice height\n");
- return -ERANGE;
- }
-
- /*
- * DSC spec mentions that bits_per_pixel specifies the target
- * bits/pixel (bpp) rate that is used by the encoder,
- * in steps of 1/16 of a bit per pixel
- */
- rbs_min = vdsc_cfg->rc_model_size - vdsc_cfg->initial_offset +
- DIV_ROUND_UP(vdsc_cfg->initial_xmit_delay *
- vdsc_cfg->bits_per_pixel, 16) +
- groups_per_line * vdsc_cfg->first_line_bpg_offset;
-
- hrd_delay = DIV_ROUND_UP((rbs_min * 16), vdsc_cfg->bits_per_pixel);
- vdsc_cfg->rc_bits = (hrd_delay * vdsc_cfg->bits_per_pixel) / 16;
- vdsc_cfg->initial_dec_delay = hrd_delay - vdsc_cfg->initial_xmit_delay;
-
- return 0;
-}
-
int intel_dp_compute_dsc_params(struct intel_dp *intel_dp,
struct intel_crtc_state *pipe_config)
{
DSC_1_1_MAX_LINEBUF_DEPTH_BITS : line_buf_depth;
/* Gen 11 does not support YCbCr */
- vdsc_cfg->enable422 = false;
+ vdsc_cfg->simple_422 = false;
/* Gen 11 does not support VBR */
vdsc_cfg->vbr_enable = false;
vdsc_cfg->block_pred_enable =
vdsc_cfg->initial_scale_value = (vdsc_cfg->rc_model_size << 3) /
(vdsc_cfg->rc_model_size - vdsc_cfg->initial_offset);
- return intel_compute_rc_parameters(vdsc_cfg);
+ return drm_dsc_compute_rc_parameters(vdsc_cfg);
}
enum intel_display_power_domain
pps_val |= DSC_BLOCK_PREDICTION;
if (vdsc_cfg->convert_rgb)
pps_val |= DSC_COLOR_SPACE_CONVERSION;
- if (vdsc_cfg->enable422)
+ if (vdsc_cfg->simple_422)
pps_val |= DSC_422_ENABLE;
if (vdsc_cfg->vbr_enable)
pps_val |= DSC_VBR_ENABLE;
struct drm_dsc_pps_infoframe dp_dsc_pps_sdp;
/* Prepare DP SDP PPS header as per DP 1.4 spec, Table 2-123 */
- drm_dsc_dp_pps_header_init(&dp_dsc_pps_sdp);
+ drm_dsc_dp_pps_header_init(&dp_dsc_pps_sdp.pps_header);
/* Fill the PPS payload bytes as per DSC spec 1.2 Table 4-1 */
- drm_dsc_pps_infoframe_pack(&dp_dsc_pps_sdp, vdsc_cfg);
+ drm_dsc_pps_payload_pack(&dp_dsc_pps_sdp.pps_payload, vdsc_cfg);
intel_dig_port->write_infoframe(encoder, crtc_state,
DP_SDP_PPS, &dp_dsc_pps_sdp,
WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2,
GEN11_TDL_CLOCK_GATING_FIX_DISABLE);
- /* WaEnableStateCacheRedirectToCS:icl */
- WA_SET_BIT_MASKED(GEN9_SLICE_COMMON_ECO_CHICKEN1,
- GEN11_STATE_CACHE_REDIRECT_TO_CS);
-
/* Wa_2006665173:icl (pre-prod) */
if (IS_ICL_REVID(i915, ICL_REVID_A0, ICL_REVID_A0))
WA_SET_BIT_MASKED(GEN11_COMMON_SLICE_CHICKEN3,
GEN10_CACHE_MODE_SS,
0, /* write-only, so skip validation */
_MASKED_BIT_ENABLE(FLOAT_BLEND_OPTIMIZATION_ENABLE));
+
+ /* WaDisableGPGPUMidThreadPreemption:icl */
+ WA_SET_FIELD_MASKED(GEN8_CS_CHICKEN1,
+ GEN9_PREEMPT_GPGPU_LEVEL_MASK,
+ GEN9_PREEMPT_GPGPU_THREAD_GROUP_LEVEL);
}
void intel_engine_init_ctx_wa(struct intel_engine_cs *engine)
wa_init_start(wal, "context");
- if (INTEL_GEN(i915) < 8)
- return;
- else if (IS_BROADWELL(i915))
- bdw_ctx_workarounds_init(engine);
- else if (IS_CHERRYVIEW(i915))
- chv_ctx_workarounds_init(engine);
- else if (IS_SKYLAKE(i915))
- skl_ctx_workarounds_init(engine);
- else if (IS_BROXTON(i915))
- bxt_ctx_workarounds_init(engine);
- else if (IS_KABYLAKE(i915))
- kbl_ctx_workarounds_init(engine);
- else if (IS_GEMINILAKE(i915))
- glk_ctx_workarounds_init(engine);
- else if (IS_COFFEELAKE(i915))
- cfl_ctx_workarounds_init(engine);
+ if (IS_GEN(i915, 11))
+ icl_ctx_workarounds_init(engine);
else if (IS_CANNONLAKE(i915))
cnl_ctx_workarounds_init(engine);
- else if (IS_ICELAKE(i915))
- icl_ctx_workarounds_init(engine);
+ else if (IS_COFFEELAKE(i915))
+ cfl_ctx_workarounds_init(engine);
+ else if (IS_GEMINILAKE(i915))
+ glk_ctx_workarounds_init(engine);
+ else if (IS_KABYLAKE(i915))
+ kbl_ctx_workarounds_init(engine);
+ else if (IS_BROXTON(i915))
+ bxt_ctx_workarounds_init(engine);
+ else if (IS_SKYLAKE(i915))
+ skl_ctx_workarounds_init(engine);
+ else if (IS_CHERRYVIEW(i915))
+ chv_ctx_workarounds_init(engine);
+ else if (IS_BROADWELL(i915))
+ bdw_ctx_workarounds_init(engine);
+ else if (INTEL_GEN(i915) < 8)
+ return;
else
MISSING_CASE(INTEL_GEN(i915));
}
static void
-wa_init_mcr(struct drm_i915_private *dev_priv, struct i915_wa_list *wal)
+wa_init_mcr(struct drm_i915_private *i915, struct i915_wa_list *wal)
{
- const struct sseu_dev_info *sseu = &RUNTIME_INFO(dev_priv)->sseu;
+ const struct sseu_dev_info *sseu = &RUNTIME_INFO(i915)->sseu;
u32 mcr_slice_subslice_mask;
/*
* something more complex that requires checking the range of every
* MMIO read).
*/
- if (INTEL_GEN(dev_priv) >= 10 &&
+ if (INTEL_GEN(i915) >= 10 &&
is_power_of_2(sseu->slice_mask)) {
/*
* read FUSE3 for enabled L3 Bank IDs, if L3 Bank matches
* enabled subslice, no need to redirect MCR packet
*/
u32 slice = fls(sseu->slice_mask);
- u32 fuse3 = I915_READ(GEN10_MIRROR_FUSE3);
+ u32 fuse3 =
+ intel_uncore_read(&i915->uncore, GEN10_MIRROR_FUSE3);
u8 ss_mask = sseu->subslice_mask[slice];
u8 enabled_mask = (ss_mask | ss_mask >>
WARN_ON((enabled_mask & disabled_mask) != enabled_mask);
}
- if (INTEL_GEN(dev_priv) >= 11)
+ if (INTEL_GEN(i915) >= 11)
mcr_slice_subslice_mask = GEN11_MCR_SLICE_MASK |
GEN11_MCR_SUBSLICE_MASK;
else
wa_write_masked_or(wal,
GEN8_MCR_SELECTOR,
mcr_slice_subslice_mask,
- intel_calculate_mcr_s_ss_select(dev_priv));
+ intel_calculate_mcr_s_ss_select(i915));
}
static void
static void
gt_init_workarounds(struct drm_i915_private *i915, struct i915_wa_list *wal)
{
- if (INTEL_GEN(i915) < 8)
- return;
- else if (IS_BROADWELL(i915))
- return;
- else if (IS_CHERRYVIEW(i915))
- return;
- else if (IS_SKYLAKE(i915))
- skl_gt_workarounds_init(i915, wal);
- else if (IS_BROXTON(i915))
- bxt_gt_workarounds_init(i915, wal);
- else if (IS_KABYLAKE(i915))
- kbl_gt_workarounds_init(i915, wal);
- else if (IS_GEMINILAKE(i915))
- glk_gt_workarounds_init(i915, wal);
- else if (IS_COFFEELAKE(i915))
- cfl_gt_workarounds_init(i915, wal);
+ if (IS_GEN(i915, 11))
+ icl_gt_workarounds_init(i915, wal);
else if (IS_CANNONLAKE(i915))
cnl_gt_workarounds_init(i915, wal);
- else if (IS_ICELAKE(i915))
- icl_gt_workarounds_init(i915, wal);
+ else if (IS_COFFEELAKE(i915))
+ cfl_gt_workarounds_init(i915, wal);
+ else if (IS_GEMINILAKE(i915))
+ glk_gt_workarounds_init(i915, wal);
+ else if (IS_KABYLAKE(i915))
+ kbl_gt_workarounds_init(i915, wal);
+ else if (IS_BROXTON(i915))
+ bxt_gt_workarounds_init(i915, wal);
+ else if (IS_SKYLAKE(i915))
+ skl_gt_workarounds_init(i915, wal);
+ else if (INTEL_GEN(i915) <= 8)
+ return;
else
MISSING_CASE(INTEL_GEN(i915));
}
}
static enum forcewake_domains
-wal_get_fw_for_rmw(struct drm_i915_private *dev_priv,
- const struct i915_wa_list *wal)
+wal_get_fw_for_rmw(struct intel_uncore *uncore, const struct i915_wa_list *wal)
{
enum forcewake_domains fw = 0;
struct i915_wa *wa;
unsigned int i;
for (i = 0, wa = wal->list; i < wal->count; i++, wa++)
- fw |= intel_uncore_forcewake_for_reg(dev_priv,
+ fw |= intel_uncore_forcewake_for_reg(uncore,
wa->reg,
FW_REG_READ |
FW_REG_WRITE);
}
static void
-wa_list_apply(struct drm_i915_private *dev_priv, const struct i915_wa_list *wal)
+wa_list_apply(struct intel_uncore *uncore, const struct i915_wa_list *wal)
{
enum forcewake_domains fw;
unsigned long flags;
if (!wal->count)
return;
- fw = wal_get_fw_for_rmw(dev_priv, wal);
+ fw = wal_get_fw_for_rmw(uncore, wal);
- spin_lock_irqsave(&dev_priv->uncore.lock, flags);
- intel_uncore_forcewake_get__locked(dev_priv, fw);
+ spin_lock_irqsave(&uncore->lock, flags);
+ intel_uncore_forcewake_get__locked(uncore, fw);
for (i = 0, wa = wal->list; i < wal->count; i++, wa++) {
- u32 val = I915_READ_FW(wa->reg);
-
- val &= ~wa->mask;
- val |= wa->val;
-
- I915_WRITE_FW(wa->reg, val);
+ intel_uncore_rmw_fw(uncore, wa->reg, wa->mask, wa->val);
}
- intel_uncore_forcewake_put__locked(dev_priv, fw);
- spin_unlock_irqrestore(&dev_priv->uncore.lock, flags);
+ intel_uncore_forcewake_put__locked(uncore, fw);
+ spin_unlock_irqrestore(&uncore->lock, flags);
}
-void intel_gt_apply_workarounds(struct drm_i915_private *dev_priv)
+void intel_gt_apply_workarounds(struct drm_i915_private *i915)
{
- wa_list_apply(dev_priv, &dev_priv->gt_wa_list);
+ wa_list_apply(&i915->uncore, &i915->gt_wa_list);
}
static bool
return true;
}
-static bool wa_list_verify(struct drm_i915_private *dev_priv,
+static bool wa_list_verify(struct intel_uncore *uncore,
const struct i915_wa_list *wal,
const char *from)
{
bool ok = true;
for (i = 0, wa = wal->list; i < wal->count; i++, wa++)
- ok &= wa_verify(wa, I915_READ(wa->reg), wal->name, from);
+ ok &= wa_verify(wa,
+ intel_uncore_read(uncore, wa->reg),
+ wal->name, from);
return ok;
}
-bool intel_gt_verify_workarounds(struct drm_i915_private *dev_priv,
+bool intel_gt_verify_workarounds(struct drm_i915_private *i915,
const char *from)
{
- return wa_list_verify(dev_priv, &dev_priv->gt_wa_list, from);
+ return wa_list_verify(&i915->uncore, &i915->gt_wa_list, from);
}
static void
/* WaAllowUMDToModifySamplerMode:icl */
whitelist_reg(w, GEN10_SAMPLER_MODE);
+
+ /* WaEnableStateCacheRedirectToCS:icl */
+ whitelist_reg(w, GEN9_SLICE_COMMON_ECO_CHICKEN1);
}
void intel_engine_init_whitelist(struct intel_engine_cs *engine)
struct drm_i915_private *i915 = engine->i915;
struct i915_wa_list *w = &engine->whitelist;
- GEM_BUG_ON(engine->id != RCS);
+ GEM_BUG_ON(engine->id != RCS0);
wa_init_start(w, "whitelist");
- if (INTEL_GEN(i915) < 8)
- return;
- else if (IS_BROADWELL(i915))
- return;
- else if (IS_CHERRYVIEW(i915))
- return;
- else if (IS_SKYLAKE(i915))
- skl_whitelist_build(w);
- else if (IS_BROXTON(i915))
- bxt_whitelist_build(w);
- else if (IS_KABYLAKE(i915))
- kbl_whitelist_build(w);
- else if (IS_GEMINILAKE(i915))
- glk_whitelist_build(w);
- else if (IS_COFFEELAKE(i915))
- cfl_whitelist_build(w);
+ if (IS_GEN(i915, 11))
+ icl_whitelist_build(w);
else if (IS_CANNONLAKE(i915))
cnl_whitelist_build(w);
- else if (IS_ICELAKE(i915))
- icl_whitelist_build(w);
+ else if (IS_COFFEELAKE(i915))
+ cfl_whitelist_build(w);
+ else if (IS_GEMINILAKE(i915))
+ glk_whitelist_build(w);
+ else if (IS_KABYLAKE(i915))
+ kbl_whitelist_build(w);
+ else if (IS_BROXTON(i915))
+ bxt_whitelist_build(w);
+ else if (IS_SKYLAKE(i915))
+ skl_whitelist_build(w);
+ else if (INTEL_GEN(i915) <= 8)
+ return;
else
MISSING_CASE(INTEL_GEN(i915));
void intel_engine_apply_whitelist(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
const struct i915_wa_list *wal = &engine->whitelist;
+ struct intel_uncore *uncore = engine->uncore;
const u32 base = engine->mmio_base;
struct i915_wa *wa;
unsigned int i;
return;
for (i = 0, wa = wal->list; i < wal->count; i++, wa++)
- I915_WRITE(RING_FORCE_TO_NONPRIV(base, i),
- i915_mmio_reg_offset(wa->reg));
+ intel_uncore_write(uncore,
+ RING_FORCE_TO_NONPRIV(base, i),
+ i915_mmio_reg_offset(wa->reg));
/* And clear the rest just in case of garbage */
for (; i < RING_MAX_NONPRIV_SLOTS; i++)
- I915_WRITE(RING_FORCE_TO_NONPRIV(base, i),
- i915_mmio_reg_offset(RING_NOPID(base)));
+ intel_uncore_write(uncore,
+ RING_FORCE_TO_NONPRIV(base, i),
+ i915_mmio_reg_offset(RING_NOPID(base)));
}
static void
{
struct drm_i915_private *i915 = engine->i915;
- if (IS_ICELAKE(i915)) {
+ if (IS_GEN(i915, 11)) {
/* This is not an Wa. Enable for better image quality */
wa_masked_en(wal,
_3D_CHICKEN3,
GEN7_DISABLE_SAMPLER_PREFETCH);
}
- if (IS_GEN(i915, 9) || IS_CANNONLAKE(i915)) {
- /* WaEnablePreemptionGranularityControlByUMD:skl,bxt,kbl,cfl,cnl */
+ if (IS_GEN_RANGE(i915, 9, 11)) {
+ /* FtrPerCtxtPreemptionGranularityControl:skl,bxt,kbl,cfl,cnl,icl */
wa_masked_en(wal,
GEN7_FF_SLICE_CS_CHICKEN1,
GEN9_FFSC_PERCTX_PREEMPT_CTRL);
if (I915_SELFTEST_ONLY(INTEL_GEN(engine->i915) < 8))
return;
- if (engine->id == RCS)
+ if (engine->id == RCS0)
rcs_engine_wa_init(engine, wal);
else
xcs_engine_wa_init(engine, wal);
void intel_engine_apply_workarounds(struct intel_engine_cs *engine)
{
- wa_list_apply(engine->i915, &engine->wa_list);
+ wa_list_apply(engine->uncore, &engine->wa_list);
}
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include <linux/slab.h>
-struct i915_wa {
- i915_reg_t reg;
- u32 mask;
- u32 val;
-};
-
-struct i915_wa_list {
- const char *name;
- struct i915_wa *list;
- unsigned int count;
- unsigned int wa_count;
-};
+#include "intel_workarounds_types.h"
static inline void intel_wa_list_free(struct i915_wa_list *wal)
{
void intel_engine_init_ctx_wa(struct intel_engine_cs *engine);
int intel_engine_emit_ctx_wa(struct i915_request *rq);
-void intel_gt_init_workarounds(struct drm_i915_private *dev_priv);
-void intel_gt_apply_workarounds(struct drm_i915_private *dev_priv);
-bool intel_gt_verify_workarounds(struct drm_i915_private *dev_priv,
+void intel_gt_init_workarounds(struct drm_i915_private *i915);
+void intel_gt_apply_workarounds(struct drm_i915_private *i915);
+bool intel_gt_verify_workarounds(struct drm_i915_private *i915,
const char *from);
void intel_engine_init_whitelist(struct intel_engine_cs *engine);
--- /dev/null
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright © 2014-2018 Intel Corporation
+ */
+
+#ifndef __INTEL_WORKAROUNDS_TYPES_H__
+#define __INTEL_WORKAROUNDS_TYPES_H__
+
+#include <linux/types.h>
+
+#include "i915_reg.h"
+
+struct i915_wa {
+ i915_reg_t reg;
+ u32 mask;
+ u32 val;
+};
+
+struct i915_wa_list {
+ const char *name;
+ struct i915_wa *list;
+ unsigned int count;
+ unsigned int wa_count;
+};
+
+#endif /* __INTEL_WORKAROUNDS_TYPES_H__ */
if (overflows_type(dma_size, obj->base.size))
return ERR_PTR(-E2BIG);
- obj = i915_gem_object_alloc(i915);
+ obj = i915_gem_object_alloc();
if (!obj)
return ERR_PTR(-ENOMEM);
if (overflows_type(size, obj->base.size))
return ERR_PTR(-E2BIG);
- obj = i915_gem_object_alloc(i915);
+ obj = i915_gem_object_alloc();
if (!obj)
return ERR_PTR(-ENOMEM);
if (overflows_type(size, obj->base.size))
return ERR_PTR(-E2BIG);
- obj = i915_gem_object_alloc(i915);
+ obj = i915_gem_object_alloc();
if (!obj)
return ERR_PTR(-ENOMEM);
if (IS_ERR(obj))
return ERR_CAST(obj);
- err = i915_gem_object_set_to_wc_domain(obj, true);
- if (err)
- goto err;
-
cmd = i915_gem_object_pin_map(obj, I915_MAP_WC);
if (IS_ERR(cmd)) {
err = PTR_ERR(cmd);
* huge-gtt-pages.
*/
- if (!ppgtt || !i915_vm_is_48bit(&ppgtt->vm)) {
+ if (!ppgtt || !i915_vm_is_4lvl(&ppgtt->vm)) {
pr_info("48b PPGTT not supported, skipping\n");
return 0;
}
* land in the now stale 2M page.
*/
- err = gpu_write(vma, ctx, dev_priv->engine[RCS], 0, 0xdeadbeaf);
+ err = gpu_write(vma, ctx, dev_priv->engine[RCS0], 0, 0xdeadbeaf);
if (err)
goto out_unpin;
}
*vaddr = 0xdeadbeaf;
+ __i915_gem_object_flush_map(obj, 0, 64);
i915_gem_object_unpin_map(obj);
vma = i915_vma_instance(obj, vm, NULL);
if (err)
goto out_unpin;
- err = gpu_write(vma, ctx, i915->engine[RCS], 0, 0xdeadbeaf);
+ err = gpu_write(vma, ctx, i915->engine[RCS0], 0, 0xdeadbeaf);
if (err)
goto out_unpin;
return -ENOMEM;
/* Pretend to be a device which supports the 48b PPGTT */
- mkwrite_device_info(dev_priv)->ppgtt = INTEL_PPGTT_FULL_4LVL;
+ mkwrite_device_info(dev_priv)->ppgtt_type = INTEL_PPGTT_FULL;
+ mkwrite_device_info(dev_priv)->ppgtt_size = 48;
mutex_lock(&dev_priv->drm.struct_mutex);
- ppgtt = i915_ppgtt_create(dev_priv, ERR_PTR(-ENODEV));
+ ppgtt = i915_ppgtt_create(dev_priv);
if (IS_ERR(ppgtt)) {
err = PTR_ERR(ppgtt);
goto out_unlock;
}
- if (!i915_vm_is_48bit(&ppgtt->vm)) {
+ if (!i915_vm_is_4lvl(&ppgtt->vm)) {
pr_err("failed to create 48b PPGTT\n");
err = -EINVAL;
goto out_close;
err = i915_subtests(tests, ppgtt);
out_close:
- i915_ppgtt_close(&ppgtt->vm);
i915_ppgtt_put(ppgtt);
out_unlock:
return 0;
}
- if (i915_terminally_wedged(&dev_priv->gpu_error))
+ if (i915_terminally_wedged(dev_priv))
return 0;
file = mock_file(dev_priv);
SUBTEST(live_active_retire),
};
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_terminally_wedged(i915))
return 0;
return i915_subtests(tests, i915);
static int pm_prepare(struct drm_i915_private *i915)
{
- int err = 0;
-
- if (i915_gem_suspend(i915)) {
- pr_err("i915_gem_suspend failed\n");
- err = -EINVAL;
- }
+ i915_gem_suspend(i915);
- return err;
+ return 0;
}
static void pm_suspend(struct drm_i915_private *i915)
SUBTEST(igt_gem_hibernate),
};
+ if (i915_terminally_wedged(i915))
+ return 0;
+
return i915_subtests(tests, i915);
}
if (IS_ERR(vma))
return PTR_ERR(vma);
- rq = i915_request_alloc(i915->engine[RCS], i915->kernel_context);
+ rq = i915_request_alloc(i915->engine[RCS0], i915->kernel_context);
if (IS_ERR(rq)) {
i915_vma_unpin(vma);
return PTR_ERR(rq);
static bool needs_fence_registers(struct drm_i915_private *i915)
{
- return !i915_terminally_wedged(&i915->gpu_error);
+ return !i915_terminally_wedged(i915);
}
static bool needs_mi_store_dword(struct drm_i915_private *i915)
{
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_terminally_wedged(i915))
return false;
- return intel_engine_can_store_dword(i915->engine[RCS]);
+ return intel_engine_can_store_dword(i915->engine[RCS0]);
}
static const struct igt_coherency_mode {
}
for (n = 0; n < nctx; n++) {
- ctx[n] = i915_gem_create_context(i915, file->driver_priv);
+ ctx[n] = live_context(i915, file);
if (IS_ERR(ctx[n])) {
err = PTR_ERR(ctx[n]);
goto out_unlock;
offset += PAGE_SIZE;
}
*cmd = MI_BATCH_BUFFER_END;
+ i915_gem_object_flush_map(obj);
i915_gem_object_unpin_map(obj);
err = i915_gem_object_set_to_gtt_domain(obj, false);
return 0;
}
-static int cpu_check(struct drm_i915_gem_object *obj, unsigned int max)
+static noinline int cpu_check(struct drm_i915_gem_object *obj,
+ unsigned int idx, unsigned int max)
{
unsigned int n, m, needs_flush;
int err;
for (m = 0; m < max; m++) {
if (map[m] != m) {
- pr_err("Invalid value at page %d, offset %d: found %x expected %x\n",
- n, m, map[m], m);
+ pr_err("%pS: Invalid value at object %d page %d/%ld, offset %d/%d: found %x expected %x\n",
+ __builtin_return_address(0), idx,
+ n, real_page_count(obj), m, max,
+ map[m], m);
err = -EINVAL;
goto out_unmap;
}
for (; m < DW_PER_PAGE; m++) {
if (map[m] != STACK_MAGIC) {
- pr_err("Invalid value at page %d, offset %d: found %x expected %x\n",
- n, m, map[m], STACK_MAGIC);
+ pr_err("%pS: Invalid value at object %d page %d, offset %d: found %x expected %x (uninitialised)\n",
+ __builtin_return_address(0), idx, n, m,
+ map[m], STACK_MAGIC);
err = -EINVAL;
goto out_unmap;
}
static int igt_ctx_exec(void *arg)
{
struct drm_i915_private *i915 = arg;
- struct drm_i915_gem_object *obj = NULL;
- unsigned long ncontexts, ndwords, dw;
- struct igt_live_test t;
- struct drm_file *file;
- IGT_TIMEOUT(end_time);
- LIST_HEAD(objects);
+ struct intel_engine_cs *engine;
+ enum intel_engine_id id;
int err = -ENODEV;
/*
if (!DRIVER_CAPS(i915)->has_logical_contexts)
return 0;
+ for_each_engine(engine, i915, id) {
+ struct drm_i915_gem_object *obj = NULL;
+ unsigned long ncontexts, ndwords, dw;
+ struct igt_live_test t;
+ struct drm_file *file;
+ IGT_TIMEOUT(end_time);
+ LIST_HEAD(objects);
+
+ if (!intel_engine_can_store_dword(engine))
+ continue;
+
+ if (!engine->context_size)
+ continue; /* No logical context support in HW */
+
+ file = mock_file(i915);
+ if (IS_ERR(file))
+ return PTR_ERR(file);
+
+ mutex_lock(&i915->drm.struct_mutex);
+
+ err = igt_live_test_begin(&t, i915, __func__, engine->name);
+ if (err)
+ goto out_unlock;
+
+ ncontexts = 0;
+ ndwords = 0;
+ dw = 0;
+ while (!time_after(jiffies, end_time)) {
+ struct i915_gem_context *ctx;
+ intel_wakeref_t wakeref;
+
+ ctx = live_context(i915, file);
+ if (IS_ERR(ctx)) {
+ err = PTR_ERR(ctx);
+ goto out_unlock;
+ }
+
+ if (!obj) {
+ obj = create_test_object(ctx, file, &objects);
+ if (IS_ERR(obj)) {
+ err = PTR_ERR(obj);
+ goto out_unlock;
+ }
+ }
+
+ with_intel_runtime_pm(i915, wakeref)
+ err = gpu_fill(obj, ctx, engine, dw);
+ if (err) {
+ pr_err("Failed to fill dword %lu [%lu/%lu] with gpu (%s) in ctx %u [full-ppgtt? %s], err=%d\n",
+ ndwords, dw, max_dwords(obj),
+ engine->name, ctx->hw_id,
+ yesno(!!ctx->ppgtt), err);
+ goto out_unlock;
+ }
+
+ if (++dw == max_dwords(obj)) {
+ obj = NULL;
+ dw = 0;
+ }
+
+ ndwords++;
+ ncontexts++;
+ }
+
+ pr_info("Submitted %lu contexts to %s, filling %lu dwords\n",
+ ncontexts, engine->name, ndwords);
+
+ ncontexts = dw = 0;
+ list_for_each_entry(obj, &objects, st_link) {
+ unsigned int rem =
+ min_t(unsigned int, ndwords - dw, max_dwords(obj));
+
+ err = cpu_check(obj, ncontexts++, rem);
+ if (err)
+ break;
+
+ dw += rem;
+ }
+
+out_unlock:
+ if (igt_live_test_end(&t))
+ err = -EIO;
+ mutex_unlock(&i915->drm.struct_mutex);
+
+ mock_file_free(i915, file);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+static int igt_shared_ctx_exec(void *arg)
+{
+ struct drm_i915_private *i915 = arg;
+ struct i915_gem_context *parent;
+ struct intel_engine_cs *engine;
+ enum intel_engine_id id;
+ struct igt_live_test t;
+ struct drm_file *file;
+ int err = 0;
+
+ /*
+ * Create a few different contexts with the same mm and write
+ * through each ctx using the GPU making sure those writes end
+ * up in the expected pages of our obj.
+ */
+ if (!DRIVER_CAPS(i915)->has_logical_contexts)
+ return 0;
+
file = mock_file(i915);
if (IS_ERR(file))
return PTR_ERR(file);
mutex_lock(&i915->drm.struct_mutex);
+ parent = live_context(i915, file);
+ if (IS_ERR(parent)) {
+ err = PTR_ERR(parent);
+ goto out_unlock;
+ }
+
+ if (!parent->ppgtt) { /* not full-ppgtt; nothing to share */
+ err = 0;
+ goto out_unlock;
+ }
+
err = igt_live_test_begin(&t, i915, __func__, "");
if (err)
goto out_unlock;
- ncontexts = 0;
- ndwords = 0;
- dw = 0;
- while (!time_after(jiffies, end_time)) {
- struct intel_engine_cs *engine;
- struct i915_gem_context *ctx;
- unsigned int id;
+ for_each_engine(engine, i915, id) {
+ unsigned long ncontexts, ndwords, dw;
+ struct drm_i915_gem_object *obj = NULL;
+ IGT_TIMEOUT(end_time);
+ LIST_HEAD(objects);
- ctx = i915_gem_create_context(i915, file->driver_priv);
- if (IS_ERR(ctx)) {
- err = PTR_ERR(ctx);
- goto out_unlock;
- }
+ if (!intel_engine_can_store_dword(engine))
+ continue;
- for_each_engine(engine, i915, id) {
+ dw = 0;
+ ndwords = 0;
+ ncontexts = 0;
+ while (!time_after(jiffies, end_time)) {
+ struct i915_gem_context *ctx;
intel_wakeref_t wakeref;
- if (!engine->context_size)
- continue; /* No logical context support in HW */
+ ctx = kernel_context(i915);
+ if (IS_ERR(ctx)) {
+ err = PTR_ERR(ctx);
+ goto out_test;
+ }
- if (!intel_engine_can_store_dword(engine))
- continue;
+ __assign_ppgtt(ctx, parent->ppgtt);
if (!obj) {
- obj = create_test_object(ctx, file, &objects);
+ obj = create_test_object(parent, file, &objects);
if (IS_ERR(obj)) {
err = PTR_ERR(obj);
- goto out_unlock;
+ kernel_context_close(ctx);
+ goto out_test;
}
}
ndwords, dw, max_dwords(obj),
engine->name, ctx->hw_id,
yesno(!!ctx->ppgtt), err);
- goto out_unlock;
+ kernel_context_close(ctx);
+ goto out_test;
}
if (++dw == max_dwords(obj)) {
obj = NULL;
dw = 0;
}
+
ndwords++;
+ ncontexts++;
+
+ kernel_context_close(ctx);
}
- ncontexts++;
- }
- pr_info("Submitted %lu contexts (across %u engines), filling %lu dwords\n",
- ncontexts, RUNTIME_INFO(i915)->num_rings, ndwords);
+ pr_info("Submitted %lu contexts to %s, filling %lu dwords\n",
+ ncontexts, engine->name, ndwords);
- dw = 0;
- list_for_each_entry(obj, &objects, st_link) {
- unsigned int rem =
- min_t(unsigned int, ndwords - dw, max_dwords(obj));
+ ncontexts = dw = 0;
+ list_for_each_entry(obj, &objects, st_link) {
+ unsigned int rem =
+ min_t(unsigned int, ndwords - dw, max_dwords(obj));
- err = cpu_check(obj, rem);
- if (err)
- break;
+ err = cpu_check(obj, ncontexts++, rem);
+ if (err)
+ goto out_test;
- dw += rem;
+ dw += rem;
+ }
}
-
-out_unlock:
+out_test:
if (igt_live_test_end(&t))
err = -EIO;
+out_unlock:
mutex_unlock(&i915->drm.struct_mutex);
mock_file_free(i915, file);
*cmd++ = upper_32_bits(vma->node.start);
*cmd = MI_BATCH_BUFFER_END;
+ __i915_gem_object_flush_map(obj, 0, 64);
i915_gem_object_unpin_map(obj);
- err = i915_gem_object_set_to_gtt_domain(obj, false);
- if (err)
- goto err;
-
vma = i915_vma_instance(obj, vma->vm, NULL);
if (IS_ERR(vma)) {
err = PTR_ERR(vma);
unsigned int flags)
{
struct intel_sseu default_sseu = intel_device_default_sseu(i915);
- struct intel_engine_cs *engine = i915->engine[RCS];
+ struct intel_engine_cs *engine = i915->engine[RCS0];
struct drm_i915_gem_object *obj;
struct i915_gem_context *ctx;
struct intel_sseu pg_sseu;
mutex_lock(&i915->drm.struct_mutex);
- ctx = i915_gem_create_context(i915, file->driver_priv);
+ ctx = live_context(i915, file);
if (IS_ERR(ctx)) {
ret = PTR_ERR(ctx);
goto out_unlock;
}
+ i915_gem_context_clear_bannable(ctx); /* to reset and beyond! */
obj = i915_gem_object_create_internal(i915, PAGE_SIZE);
if (IS_ERR(obj)) {
struct drm_i915_gem_object *obj = NULL;
struct i915_gem_context *ctx;
struct i915_hw_ppgtt *ppgtt;
- unsigned long ndwords, dw;
+ unsigned long idx, ndwords, dw;
struct igt_live_test t;
struct drm_file *file;
I915_RND_STATE(prng);
if (err)
goto out_unlock;
- ctx = i915_gem_create_context(i915, file->driver_priv);
+ ctx = live_context(i915, file);
if (IS_ERR(ctx)) {
err = PTR_ERR(ctx);
goto out_unlock;
}
}
pr_info("Submitted %lu dwords (across %u engines)\n",
- ndwords, RUNTIME_INFO(i915)->num_rings);
+ ndwords, RUNTIME_INFO(i915)->num_engines);
dw = 0;
+ idx = 0;
list_for_each_entry(obj, &objects, st_link) {
unsigned int rem =
min_t(unsigned int, ndwords - dw, max_dwords(obj));
if (i915_gem_object_is_readonly(obj))
num_writes = 0;
- err = cpu_check(obj, num_writes);
+ err = cpu_check(obj, idx++, num_writes);
if (err)
break;
}
*cmd++ = value;
*cmd = MI_BATCH_BUFFER_END;
+ __i915_gem_object_flush_map(obj, 0, 64);
i915_gem_object_unpin_map(obj);
- err = i915_gem_object_set_to_gtt_domain(obj, false);
- if (err)
- goto err;
-
vma = i915_vma_instance(obj, &ctx->ppgtt->vm, NULL);
if (IS_ERR(vma)) {
err = PTR_ERR(vma);
*cmd++ = result;
}
*cmd = MI_BATCH_BUFFER_END;
- i915_gem_object_unpin_map(obj);
- err = i915_gem_object_set_to_gtt_domain(obj, false);
- if (err)
- goto err;
+ i915_gem_object_flush_map(obj);
+ i915_gem_object_unpin_map(obj);
vma = i915_vma_instance(obj, &ctx->ppgtt->vm, NULL);
if (IS_ERR(vma)) {
if (err)
goto out_unlock;
- ctx_a = i915_gem_create_context(i915, file->driver_priv);
+ ctx_a = live_context(i915, file);
if (IS_ERR(ctx_a)) {
err = PTR_ERR(ctx_a);
goto out_unlock;
}
- ctx_b = i915_gem_create_context(i915, file->driver_priv);
+ ctx_b = live_context(i915, file);
if (IS_ERR(ctx_b)) {
err = PTR_ERR(ctx_b);
goto out_unlock;
div64_u64_rem(i915_prandom_u64_state(&prng),
vm_total, &offset);
- offset &= ~sizeof(u32);
+ offset &= -sizeof(u32);
offset += I915_GTT_PAGE_SIZE;
err = write_to_scratch(ctx_a, engine,
count += this;
}
pr_info("Checked %lu scratch offsets across %d engines\n",
- count, RUNTIME_INFO(i915)->num_rings);
+ count, RUNTIME_INFO(i915)->num_engines);
out_rpm:
intel_runtime_pm_put(i915, wakeref);
}
static __maybe_unused const char *
-__engine_name(struct drm_i915_private *i915, unsigned int engines)
+__engine_name(struct drm_i915_private *i915, intel_engine_mask_t engines)
{
struct intel_engine_cs *engine;
- unsigned int tmp;
+ intel_engine_mask_t tmp;
if (engines == ALL_ENGINES)
return "all";
static int __igt_switch_to_kernel_context(struct drm_i915_private *i915,
struct i915_gem_context *ctx,
- unsigned int engines)
+ intel_engine_mask_t engines)
{
struct intel_engine_cs *engine;
- unsigned int tmp;
- int err;
+ intel_engine_mask_t tmp;
+ int pass;
GEM_TRACE("Testing %s\n", __engine_name(i915, engines));
- for_each_engine_masked(engine, i915, engines, tmp) {
- struct i915_request *rq;
+ for (pass = 0; pass < 4; pass++) { /* Once busy; once idle; repeat */
+ bool from_idle = pass & 1;
+ int err;
- rq = i915_request_alloc(engine, ctx);
- if (IS_ERR(rq))
- return PTR_ERR(rq);
+ if (!from_idle) {
+ for_each_engine_masked(engine, i915, engines, tmp) {
+ struct i915_request *rq;
- i915_request_add(rq);
- }
+ rq = i915_request_alloc(engine, ctx);
+ if (IS_ERR(rq))
+ return PTR_ERR(rq);
- err = i915_gem_switch_to_kernel_context(i915);
- if (err)
- return err;
-
- for_each_engine_masked(engine, i915, engines, tmp) {
- if (!engine_has_kernel_context_barrier(engine)) {
- pr_err("kernel context not last on engine %s!\n",
- engine->name);
- return -EINVAL;
+ i915_request_add(rq);
+ }
}
- }
- err = i915_gem_wait_for_idle(i915,
- I915_WAIT_LOCKED,
- MAX_SCHEDULE_TIMEOUT);
- if (err)
- return err;
+ err = i915_gem_switch_to_kernel_context(i915,
+ i915->gt.active_engines);
+ if (err)
+ return err;
- GEM_BUG_ON(i915->gt.active_requests);
- for_each_engine_masked(engine, i915, engines, tmp) {
- if (engine->last_retired_context->gem_context != i915->kernel_context) {
- pr_err("engine %s not idling in kernel context!\n",
- engine->name);
+ if (!from_idle) {
+ err = i915_gem_wait_for_idle(i915,
+ I915_WAIT_LOCKED,
+ MAX_SCHEDULE_TIMEOUT);
+ if (err)
+ return err;
+ }
+
+ if (i915->gt.active_requests) {
+ pr_err("%d active requests remain after switching to kernel context, pass %d (%s) on %s engine%s\n",
+ i915->gt.active_requests,
+ pass, from_idle ? "idle" : "busy",
+ __engine_name(i915, engines),
+ is_power_of_2(engines) ? "" : "s");
return -EINVAL;
}
- }
- err = i915_gem_switch_to_kernel_context(i915);
- if (err)
- return err;
+ /* XXX Bonus points for proving we are the kernel context! */
- if (i915->gt.active_requests) {
- pr_err("switch-to-kernel-context emitted %d requests even though it should already be idling in the kernel context\n",
- i915->gt.active_requests);
- return -EINVAL;
+ mutex_unlock(&i915->drm.struct_mutex);
+ drain_delayed_work(&i915->gt.idle_work);
+ mutex_lock(&i915->drm.struct_mutex);
}
- for_each_engine_masked(engine, i915, engines, tmp) {
- if (!intel_engine_has_kernel_context(engine)) {
- pr_err("kernel context not last on engine %s!\n",
- engine->name);
- return -EINVAL;
- }
- }
+ if (igt_flush_test(i915, I915_WAIT_LOCKED))
+ return -EIO;
return 0;
}
out_unlock:
GEM_TRACE_DUMP_ON(err);
- if (igt_flush_test(i915, I915_WAIT_LOCKED))
- err = -EIO;
intel_runtime_pm_put(i915, wakeref);
mutex_unlock(&i915->drm.struct_mutex);
return err;
}
+static void mock_barrier_task(void *data)
+{
+ unsigned int *counter = data;
+
+ ++*counter;
+}
+
+static int mock_context_barrier(void *arg)
+{
+#undef pr_fmt
+#define pr_fmt(x) "context_barrier_task():" # x
+ struct drm_i915_private *i915 = arg;
+ struct i915_gem_context *ctx;
+ struct i915_request *rq;
+ intel_wakeref_t wakeref;
+ unsigned int counter;
+ int err;
+
+ /*
+ * The context barrier provides us with a callback after it emits
+ * a request; useful for retiring old state after loading new.
+ */
+
+ mutex_lock(&i915->drm.struct_mutex);
+
+ ctx = mock_context(i915, "mock");
+ if (!ctx) {
+ err = -ENOMEM;
+ goto unlock;
+ }
+
+ counter = 0;
+ err = context_barrier_task(ctx, 0,
+ NULL, mock_barrier_task, &counter);
+ if (err) {
+ pr_err("Failed at line %d, err=%d\n", __LINE__, err);
+ goto out;
+ }
+ if (counter == 0) {
+ pr_err("Did not retire immediately with 0 engines\n");
+ err = -EINVAL;
+ goto out;
+ }
+
+ counter = 0;
+ err = context_barrier_task(ctx, ALL_ENGINES,
+ NULL, mock_barrier_task, &counter);
+ if (err) {
+ pr_err("Failed at line %d, err=%d\n", __LINE__, err);
+ goto out;
+ }
+ if (counter == 0) {
+ pr_err("Did not retire immediately for all inactive engines\n");
+ err = -EINVAL;
+ goto out;
+ }
+
+ rq = ERR_PTR(-ENODEV);
+ with_intel_runtime_pm(i915, wakeref)
+ rq = i915_request_alloc(i915->engine[RCS0], ctx);
+ if (IS_ERR(rq)) {
+ pr_err("Request allocation failed!\n");
+ goto out;
+ }
+ i915_request_add(rq);
+ GEM_BUG_ON(list_empty(&ctx->active_engines));
+
+ counter = 0;
+ context_barrier_inject_fault = BIT(RCS0);
+ err = context_barrier_task(ctx, ALL_ENGINES,
+ NULL, mock_barrier_task, &counter);
+ context_barrier_inject_fault = 0;
+ if (err == -ENXIO)
+ err = 0;
+ else
+ pr_err("Did not hit fault injection!\n");
+ if (counter != 0) {
+ pr_err("Invoked callback on error!\n");
+ err = -EIO;
+ }
+ if (err)
+ goto out;
+
+ counter = 0;
+ err = context_barrier_task(ctx, ALL_ENGINES,
+ NULL, mock_barrier_task, &counter);
+ if (err) {
+ pr_err("Failed at line %d, err=%d\n", __LINE__, err);
+ goto out;
+ }
+ mock_device_flush(i915);
+ if (counter == 0) {
+ pr_err("Did not retire on each active engines\n");
+ err = -EINVAL;
+ goto out;
+ }
+
+out:
+ mock_context_close(ctx);
+unlock:
+ mutex_unlock(&i915->drm.struct_mutex);
+ return err;
+#undef pr_fmt
+#define pr_fmt(x) x
+}
+
int i915_gem_context_mock_selftests(void)
{
static const struct i915_subtest tests[] = {
SUBTEST(igt_switch_to_kernel_context),
+ SUBTEST(mock_context_barrier),
};
struct drm_i915_private *i915;
int err;
SUBTEST(igt_ctx_exec),
SUBTEST(igt_ctx_readonly),
SUBTEST(igt_ctx_sseu),
+ SUBTEST(igt_shared_ctx_exec),
SUBTEST(igt_vm_isolation),
};
- if (i915_terminally_wedged(&dev_priv->gpu_error))
+ if (i915_terminally_wedged(dev_priv))
return 0;
return i915_subtests(tests, dev_priv);
goto err;
}
memset(ptr + PAGE_SIZE, 0xaa, PAGE_SIZE);
+ i915_gem_object_flush_map(obj);
i915_gem_object_unpin_map(obj);
ptr = dma_buf_kmap(dmabuf, 1);
err = PTR_ERR(obj);
goto cleanup;
}
- i915_gem_object_set_cache_level(obj, I915_CACHE_LLC);
+ i915_gem_object_set_cache_coherency(obj, I915_CACHE_LLC);
quirk_add(obj, &objects);
vma = i915_gem_object_ggtt_pin(obj, NULL, 0, 0,
err = PTR_ERR(obj);
goto cleanup;
}
- i915_gem_object_set_cache_level(obj, I915_CACHE_LLC);
+ i915_gem_object_set_cache_coherency(obj, I915_CACHE_LLC);
quirk_add(obj, &objects);
/* Neighbouring; same colour - should fit */
SUBTEST(igt_evict_contexts),
};
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_terminally_wedged(i915))
return 0;
return i915_subtests(tests, i915);
if (overflows_type(size, obj->base.size))
return ERR_PTR(-E2BIG);
- obj = i915_gem_object_alloc(i915);
+ obj = i915_gem_object_alloc();
if (!obj)
goto err;
return PTR_ERR(file);
mutex_lock(&dev_priv->drm.struct_mutex);
- ppgtt = i915_ppgtt_create(dev_priv, file->driver_priv);
+ ppgtt = i915_ppgtt_create(dev_priv);
if (IS_ERR(ppgtt)) {
err = PTR_ERR(ppgtt);
goto out_unlock;
err = func(dev_priv, &ppgtt->vm, 0, ppgtt->vm.total, end_time);
- i915_ppgtt_close(&ppgtt->vm);
i915_ppgtt_put(ppgtt);
out_unlock:
mutex_unlock(&dev_priv->drm.struct_mutex);
SUBTEST(igt_gtt_insert),
};
struct drm_i915_private *i915;
- struct i915_ggtt ggtt;
+ struct i915_ggtt *ggtt;
int err;
i915 = mock_gem_device();
if (!i915)
return -ENOMEM;
- mock_init_ggtt(i915, &ggtt);
+ ggtt = kmalloc(sizeof(*ggtt), GFP_KERNEL);
+ if (!ggtt) {
+ err = -ENOMEM;
+ goto out_put;
+ }
+ mock_init_ggtt(i915, ggtt);
mutex_lock(&i915->drm.struct_mutex);
- err = i915_subtests(tests, &ggtt);
+ err = i915_subtests(tests, ggtt);
mock_device_flush(i915);
mutex_unlock(&i915->drm.struct_mutex);
i915_gem_drain_freed_objects(i915);
- mock_fini_ggtt(&ggtt);
+ mock_fini_ggtt(ggtt);
+ kfree(ggtt);
+out_put:
drm_dev_put(&i915->drm);
-
return err;
}
if (err)
return err;
- rq = i915_request_alloc(i915->engine[RCS], i915->kernel_context);
+ rq = i915_request_alloc(i915->engine[RCS0], i915->kernel_context);
if (IS_ERR(rq)) {
i915_vma_unpin(vma);
return PTR_ERR(rq);
for (loop = 0; loop < 3; loop++) {
intel_wakeref_t wakeref;
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_terminally_wedged(i915))
break;
obj = i915_gem_object_create_internal(i915, PAGE_SIZE);
/* Basic preliminary test to create a request and let it loose! */
mutex_lock(&i915->drm.struct_mutex);
- request = mock_request(i915->engine[RCS],
+ request = mock_request(i915->engine[RCS0],
i915->kernel_context,
HZ / 10);
if (!request)
/* Submit a request, then wait upon it */
mutex_lock(&i915->drm.struct_mutex);
- request = mock_request(i915->engine[RCS], i915->kernel_context, T);
+ request = mock_request(i915->engine[RCS0], i915->kernel_context, T);
if (!request) {
err = -ENOMEM;
goto out_unlock;
/* Submit a request, treat it as a fence and wait upon it */
mutex_lock(&i915->drm.struct_mutex);
- request = mock_request(i915->engine[RCS], i915->kernel_context, T);
+ request = mock_request(i915->engine[RCS0], i915->kernel_context, T);
if (!request) {
err = -ENOMEM;
goto out_locked;
}
- mutex_unlock(&i915->drm.struct_mutex); /* safe as we are single user */
if (dma_fence_wait_timeout(&request->fence, false, T) != -ETIME) {
pr_err("fence wait success before submit (expected timeout)!\n");
- goto out_device;
+ goto out_locked;
}
- mutex_lock(&i915->drm.struct_mutex);
i915_request_add(request);
mutex_unlock(&i915->drm.struct_mutex);
mutex_lock(&i915->drm.struct_mutex);
ctx[0] = mock_context(i915, "A");
- request = mock_request(i915->engine[RCS], ctx[0], 2 * HZ);
+ request = mock_request(i915->engine[RCS0], ctx[0], 2 * HZ);
if (!request) {
err = -ENOMEM;
goto err_context_0;
i915_request_add(request);
ctx[1] = mock_context(i915, "B");
- vip = mock_request(i915->engine[RCS], ctx[1], 0);
+ vip = mock_request(i915->engine[RCS0], ctx[1], 0);
if (!vip) {
err = -ENOMEM;
goto err_context_1;
mutex_unlock(&i915->drm.struct_mutex);
if (i915_request_wait(vip, 0, HZ) == -ETIME) {
- pr_err("timed out waiting for high priority request, vip.seqno=%d, current seqno=%d\n",
- vip->global_seqno, intel_engine_get_seqno(i915->engine[RCS]));
+ pr_err("timed out waiting for high priority request\n");
goto err;
}
{
struct drm_i915_private *i915 = arg;
struct smoketest t = {
- .engine = i915->engine[RCS],
+ .engine = i915->engine[RCS0],
.ncontexts = 1024,
.max_batch = 1024,
.request_alloc = __mock_request_alloc
}
*cmd = MI_BATCH_BUFFER_END;
- i915_gem_chipset_flush(i915);
+ __i915_gem_object_flush_map(obj, 0, 64);
i915_gem_object_unpin_map(obj);
- err = i915_gem_object_set_to_gtt_domain(obj, false);
- if (err)
- goto err;
+ i915_gem_chipset_flush(i915);
vma = i915_vma_instance(obj, &i915->ggtt.vm, NULL);
if (IS_ERR(vma)) {
if (err)
goto err;
- err = i915_gem_object_set_to_wc_domain(obj, true);
- if (err)
- goto err;
-
cmd = i915_gem_object_pin_map(obj, I915_MAP_WC);
if (IS_ERR(cmd)) {
err = PTR_ERR(cmd);
*cmd++ = lower_32_bits(vma->node.start);
}
*cmd++ = MI_BATCH_BUFFER_END; /* terminate early in case of error */
- i915_gem_chipset_flush(i915);
+ __i915_gem_object_flush_map(obj, 0, 64);
i915_gem_object_unpin_map(obj);
+ i915_gem_chipset_flush(i915);
+
return vma;
err:
num_fences += atomic_long_read(&t[id].num_fences);
}
pr_info("Completed %lu waits for %lu fences across %d engines and %d cpus\n",
- num_waits, num_fences, RUNTIME_INFO(i915)->num_rings, ncpus);
+ num_waits, num_fences, RUNTIME_INFO(i915)->num_engines, ncpus);
mutex_lock(&i915->drm.struct_mutex);
ret = igt_live_test_end(&live) ?: ret;
SUBTEST(live_breadcrumbs_smoketest),
};
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_terminally_wedged(i915))
return 0;
return i915_subtests(tests, i915);
if (signal_pending(current))
return -EINTR;
- pr_debug(DRIVER_NAME ": Running %s\n", st->name);
+ pr_info(DRIVER_NAME ": Running %s\n", st->name);
if (data)
err = st->live(data);
else
if (!apply_subtest_filter(caller, st->name))
continue;
- pr_debug(DRIVER_NAME ": Running %s/%s\n", caller, st->name);
+ pr_info(DRIVER_NAME ": Running %s/%s\n", caller, st->name);
GEM_TRACE("Running %s/%s\n", caller, st->name);
err = st->func(data);
unsigned long target, delay;
struct timed_fence tf;
+ preempt_disable();
timed_fence_init(&tf, target = jiffies);
if (!i915_sw_fence_done(&tf.fence)) {
pr_err("Fence with immediate expiration not signaled\n");
goto err;
}
+ preempt_enable();
timed_fence_fini(&tf);
for_each_prime_number(delay, i915_selftest.timeout_jiffies/2) {
+ preempt_disable();
timed_fence_init(&tf, target = jiffies + delay);
if (i915_sw_fence_done(&tf.fence)) {
pr_err("Fence with future expiration (%lu jiffies) already signaled\n", delay);
goto err;
}
+ preempt_enable();
i915_sw_fence_wait(&tf.fence);
+
+ preempt_disable();
if (!i915_sw_fence_done(&tf.fence)) {
pr_err("Fence not signaled after wait\n");
goto err;
target, jiffies);
goto err;
}
-
+ preempt_enable();
timed_fence_fini(&tf);
}
return 0;
err:
+ preempt_enable();
timed_fence_fini(&tf);
return -EINVAL;
}
unsigned long cacheline;
int err;
- tl = i915_timeline_create(state->i915, "mock", NULL);
+ tl = i915_timeline_create(state->i915, NULL);
if (IS_ERR(tl))
return PTR_ERR(tl);
{
struct i915_timeline *tl;
- tl = i915_timeline_create(i915, "live", NULL);
+ tl = i915_timeline_create(i915, NULL);
if (IS_ERR(tl))
return tl;
#undef NUM_TIMELINES
}
+static int live_hwsp_wrap(void *arg)
+{
+ struct drm_i915_private *i915 = arg;
+ struct intel_engine_cs *engine;
+ struct i915_timeline *tl;
+ enum intel_engine_id id;
+ intel_wakeref_t wakeref;
+ int err = 0;
+
+ /*
+ * Across a seqno wrap, we need to keep the old cacheline alive for
+ * foreign GPU references.
+ */
+
+ mutex_lock(&i915->drm.struct_mutex);
+ wakeref = intel_runtime_pm_get(i915);
+
+ tl = i915_timeline_create(i915, NULL);
+ if (IS_ERR(tl)) {
+ err = PTR_ERR(tl);
+ goto out_rpm;
+ }
+ if (!tl->has_initial_breadcrumb || !tl->hwsp_cacheline)
+ goto out_free;
+
+ err = i915_timeline_pin(tl);
+ if (err)
+ goto out_free;
+
+ for_each_engine(engine, i915, id) {
+ const u32 *hwsp_seqno[2];
+ struct i915_request *rq;
+ u32 seqno[2];
+
+ if (!intel_engine_can_store_dword(engine))
+ continue;
+
+ rq = i915_request_alloc(engine, i915->kernel_context);
+ if (IS_ERR(rq)) {
+ err = PTR_ERR(rq);
+ goto out;
+ }
+
+ tl->seqno = -4u;
+
+ err = i915_timeline_get_seqno(tl, rq, &seqno[0]);
+ if (err) {
+ i915_request_add(rq);
+ goto out;
+ }
+ pr_debug("seqno[0]:%08x, hwsp_offset:%08x\n",
+ seqno[0], tl->hwsp_offset);
+
+ err = emit_ggtt_store_dw(rq, tl->hwsp_offset, seqno[0]);
+ if (err) {
+ i915_request_add(rq);
+ goto out;
+ }
+ hwsp_seqno[0] = tl->hwsp_seqno;
+
+ err = i915_timeline_get_seqno(tl, rq, &seqno[1]);
+ if (err) {
+ i915_request_add(rq);
+ goto out;
+ }
+ pr_debug("seqno[1]:%08x, hwsp_offset:%08x\n",
+ seqno[1], tl->hwsp_offset);
+
+ err = emit_ggtt_store_dw(rq, tl->hwsp_offset, seqno[1]);
+ if (err) {
+ i915_request_add(rq);
+ goto out;
+ }
+ hwsp_seqno[1] = tl->hwsp_seqno;
+
+ /* With wrap should come a new hwsp */
+ GEM_BUG_ON(seqno[1] >= seqno[0]);
+ GEM_BUG_ON(hwsp_seqno[0] == hwsp_seqno[1]);
+
+ i915_request_add(rq);
+
+ if (i915_request_wait(rq, I915_WAIT_LOCKED, HZ / 5) < 0) {
+ pr_err("Wait for timeline writes timed out!\n");
+ err = -EIO;
+ goto out;
+ }
+
+ if (*hwsp_seqno[0] != seqno[0] || *hwsp_seqno[1] != seqno[1]) {
+ pr_err("Bad timeline values: found (%x, %x), expected (%x, %x)\n",
+ *hwsp_seqno[0], *hwsp_seqno[1],
+ seqno[0], seqno[1]);
+ err = -EINVAL;
+ goto out;
+ }
+
+ i915_retire_requests(i915); /* recycle HWSP */
+ }
+
+out:
+ if (igt_flush_test(i915, I915_WAIT_LOCKED))
+ err = -EIO;
+
+ i915_timeline_unpin(tl);
+out_free:
+ i915_timeline_put(tl);
+out_rpm:
+ intel_runtime_pm_put(i915, wakeref);
+ mutex_unlock(&i915->drm.struct_mutex);
+
+ return err;
+}
+
static int live_hwsp_recycle(void *arg)
{
struct drm_i915_private *i915 = arg;
SUBTEST(live_hwsp_recycle),
SUBTEST(live_hwsp_engine),
SUBTEST(live_hwsp_alternate),
+ SUBTEST(live_hwsp_wrap),
};
+ if (i915_terminally_wedged(i915))
+ return 0;
+
return i915_subtests(tests, i915);
}
SUBTEST(igt_vma_partial),
};
struct drm_i915_private *i915;
- struct i915_ggtt ggtt;
+ struct i915_ggtt *ggtt;
int err;
i915 = mock_gem_device();
if (!i915)
return -ENOMEM;
- mock_init_ggtt(i915, &ggtt);
+ ggtt = kmalloc(sizeof(*ggtt), GFP_KERNEL);
+ if (!ggtt) {
+ err = -ENOMEM;
+ goto out_put;
+ }
+ mock_init_ggtt(i915, ggtt);
mutex_lock(&i915->drm.struct_mutex);
- err = i915_subtests(tests, &ggtt);
+ err = i915_subtests(tests, ggtt);
mock_device_flush(i915);
mutex_unlock(&i915->drm.struct_mutex);
i915_gem_drain_freed_objects(i915);
- mock_fini_ggtt(&ggtt);
+ mock_fini_ggtt(ggtt);
+ kfree(ggtt);
+out_put:
drm_dev_put(&i915->drm);
-
return err;
}
cond_resched();
if (flags & I915_WAIT_LOCKED &&
- i915_gem_switch_to_kernel_context(i915)) {
+ i915_gem_switch_to_kernel_context(i915, i915->gt.active_engines)) {
pr_err("Failed to switch back to kernel context; declaring wedged\n");
i915_gem_set_wedged(i915);
}
i915_gem_set_wedged(i915);
}
- return i915_terminally_wedged(&i915->gpu_error) ? -EIO : 0;
+ return i915_terminally_wedged(i915);
}
goto err_hws;
}
- i915_gem_object_set_cache_level(spin->hws, I915_CACHE_LLC);
+ i915_gem_object_set_cache_coherency(spin->hws, I915_CACHE_LLC);
vaddr = i915_gem_object_pin_map(spin->hws, I915_MAP_WB);
if (IS_ERR(vaddr)) {
err = PTR_ERR(vaddr);
i915_gem_chipset_flush(spin->i915);
+ if (engine->emit_init_breadcrumb &&
+ rq->timeline->has_initial_breadcrumb) {
+ err = engine->emit_init_breadcrumb(rq);
+ if (err)
+ goto cancel_rq;
+ }
+
err = engine->emit_bb_start(rq, vma->node.start, PAGE_SIZE, 0);
cancel_rq:
dev_priv->preempt_context : dev_priv->kernel_context;
if (client->owner != ctx_owner ||
- client->engines != INTEL_INFO(dev_priv)->ring_mask ||
+ client->engines != INTEL_INFO(dev_priv)->engine_mask ||
client->priority != client_priority ||
client->doorbell_id == GUC_DOORBELL_INVALID)
return -EINVAL;
for (i = 0; i < ATTEMPTS; i++) {
clients[i] = guc_client_alloc(dev_priv,
- INTEL_INFO(dev_priv)->ring_mask,
+ INTEL_INFO(dev_priv)->engine_mask,
i % GUC_CLIENT_PRIORITY_NUM,
dev_priv->kernel_context);
if (IS_ERR(h->ctx))
return PTR_ERR(h->ctx);
+ GEM_BUG_ON(i915_gem_context_is_bannable(h->ctx));
+
h->hws = i915_gem_object_create_internal(i915, PAGE_SIZE);
if (IS_ERR(h->hws)) {
err = PTR_ERR(h->hws);
goto err_hws;
}
- i915_gem_object_set_cache_level(h->hws, I915_CACHE_LLC);
+ i915_gem_object_set_cache_coherency(h->hws, I915_CACHE_LLC);
vaddr = i915_gem_object_pin_map(h->hws, I915_MAP_WB);
if (IS_ERR(vaddr)) {
err = PTR_ERR(vaddr);
*batch++ = MI_BATCH_BUFFER_END; /* not reached */
i915_gem_chipset_flush(h->i915);
+ if (rq->engine->emit_init_breadcrumb) {
+ err = rq->engine->emit_init_breadcrumb(rq);
+ if (err)
+ goto cancel_rq;
+ }
+
flags = 0;
if (INTEL_GEN(vm->i915) <= 5)
flags |= I915_DISPATCH_SECURE;
timeout = i915_request_wait(rq,
I915_WAIT_LOCKED,
MAX_SCHEDULE_TIMEOUT);
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_reset_failed(i915))
timeout = -EIO;
i915_request_put(rq);
igt_global_reset_unlock(i915);
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_reset_failed(i915))
err = -EIO;
return err;
i915_gem_set_wedged(i915);
- mutex_lock(&i915->drm.struct_mutex);
- GEM_BUG_ON(!i915_terminally_wedged(&i915->gpu_error));
+ GEM_BUG_ON(!i915_reset_failed(i915));
i915_reset(i915, ALL_ENGINES, NULL);
- mutex_unlock(&i915->drm.struct_mutex);
intel_runtime_pm_put(i915, wakeref);
igt_global_reset_unlock(i915);
- return i915_terminally_wedged(&i915->gpu_error) ? -EIO : 0;
+ return i915_reset_failed(i915) ? -EIO : 0;
}
static bool wait_for_idle(struct intel_engine_cs *engine)
return wait_for(intel_engine_is_idle(engine), IGT_IDLE_TIMEOUT) == 0;
}
+static int igt_reset_nop(void *arg)
+{
+ struct drm_i915_private *i915 = arg;
+ struct intel_engine_cs *engine;
+ struct i915_gem_context *ctx;
+ unsigned int reset_count, count;
+ enum intel_engine_id id;
+ intel_wakeref_t wakeref;
+ struct drm_file *file;
+ IGT_TIMEOUT(end_time);
+ int err = 0;
+
+ /* Check that we can reset during non-user portions of requests */
+
+ file = mock_file(i915);
+ if (IS_ERR(file))
+ return PTR_ERR(file);
+
+ mutex_lock(&i915->drm.struct_mutex);
+ ctx = live_context(i915, file);
+ mutex_unlock(&i915->drm.struct_mutex);
+ if (IS_ERR(ctx)) {
+ err = PTR_ERR(ctx);
+ goto out;
+ }
+
+ i915_gem_context_clear_bannable(ctx);
+ wakeref = intel_runtime_pm_get(i915);
+ reset_count = i915_reset_count(&i915->gpu_error);
+ count = 0;
+ do {
+ mutex_lock(&i915->drm.struct_mutex);
+ for_each_engine(engine, i915, id) {
+ int i;
+
+ for (i = 0; i < 16; i++) {
+ struct i915_request *rq;
+
+ rq = i915_request_alloc(engine, ctx);
+ if (IS_ERR(rq)) {
+ err = PTR_ERR(rq);
+ break;
+ }
+
+ i915_request_add(rq);
+ }
+ }
+ mutex_unlock(&i915->drm.struct_mutex);
+
+ igt_global_reset_lock(i915);
+ i915_reset(i915, ALL_ENGINES, NULL);
+ igt_global_reset_unlock(i915);
+ if (i915_reset_failed(i915)) {
+ err = -EIO;
+ break;
+ }
+
+ if (i915_reset_count(&i915->gpu_error) !=
+ reset_count + ++count) {
+ pr_err("Full GPU reset not recorded!\n");
+ err = -EINVAL;
+ break;
+ }
+
+ if (!i915_reset_flush(i915)) {
+ struct drm_printer p =
+ drm_info_printer(i915->drm.dev);
+
+ pr_err("%s failed to idle after reset\n",
+ engine->name);
+ intel_engine_dump(engine, &p,
+ "%s\n", engine->name);
+
+ err = -EIO;
+ break;
+ }
+
+ err = igt_flush_test(i915, 0);
+ if (err)
+ break;
+ } while (time_before(jiffies, end_time));
+ pr_info("%s: %d resets\n", __func__, count);
+
+ mutex_lock(&i915->drm.struct_mutex);
+ err = igt_flush_test(i915, I915_WAIT_LOCKED);
+ mutex_unlock(&i915->drm.struct_mutex);
+
+ intel_runtime_pm_put(i915, wakeref);
+
+out:
+ mock_file_free(i915, file);
+ if (i915_reset_failed(i915))
+ err = -EIO;
+ return err;
+}
+
+static int igt_reset_nop_engine(void *arg)
+{
+ struct drm_i915_private *i915 = arg;
+ struct intel_engine_cs *engine;
+ struct i915_gem_context *ctx;
+ enum intel_engine_id id;
+ intel_wakeref_t wakeref;
+ struct drm_file *file;
+ int err = 0;
+
+ /* Check that we can engine-reset during non-user portions */
+
+ if (!intel_has_reset_engine(i915))
+ return 0;
+
+ file = mock_file(i915);
+ if (IS_ERR(file))
+ return PTR_ERR(file);
+
+ mutex_lock(&i915->drm.struct_mutex);
+ ctx = live_context(i915, file);
+ mutex_unlock(&i915->drm.struct_mutex);
+ if (IS_ERR(ctx)) {
+ err = PTR_ERR(ctx);
+ goto out;
+ }
+
+ i915_gem_context_clear_bannable(ctx);
+ wakeref = intel_runtime_pm_get(i915);
+ for_each_engine(engine, i915, id) {
+ unsigned int reset_count, reset_engine_count;
+ unsigned int count;
+ IGT_TIMEOUT(end_time);
+
+ reset_count = i915_reset_count(&i915->gpu_error);
+ reset_engine_count = i915_reset_engine_count(&i915->gpu_error,
+ engine);
+ count = 0;
+
+ set_bit(I915_RESET_ENGINE + id, &i915->gpu_error.flags);
+ do {
+ int i;
+
+ if (!wait_for_idle(engine)) {
+ pr_err("%s failed to idle before reset\n",
+ engine->name);
+ err = -EIO;
+ break;
+ }
+
+ mutex_lock(&i915->drm.struct_mutex);
+ for (i = 0; i < 16; i++) {
+ struct i915_request *rq;
+
+ rq = i915_request_alloc(engine, ctx);
+ if (IS_ERR(rq)) {
+ err = PTR_ERR(rq);
+ break;
+ }
+
+ i915_request_add(rq);
+ }
+ mutex_unlock(&i915->drm.struct_mutex);
+
+ err = i915_reset_engine(engine, NULL);
+ if (err) {
+ pr_err("i915_reset_engine failed\n");
+ break;
+ }
+
+ if (i915_reset_count(&i915->gpu_error) != reset_count) {
+ pr_err("Full GPU reset recorded! (engine reset expected)\n");
+ err = -EINVAL;
+ break;
+ }
+
+ if (i915_reset_engine_count(&i915->gpu_error, engine) !=
+ reset_engine_count + ++count) {
+ pr_err("%s engine reset not recorded!\n",
+ engine->name);
+ err = -EINVAL;
+ break;
+ }
+
+ if (!i915_reset_flush(i915)) {
+ struct drm_printer p =
+ drm_info_printer(i915->drm.dev);
+
+ pr_err("%s failed to idle after reset\n",
+ engine->name);
+ intel_engine_dump(engine, &p,
+ "%s\n", engine->name);
+
+ err = -EIO;
+ break;
+ }
+ } while (time_before(jiffies, end_time));
+ clear_bit(I915_RESET_ENGINE + id, &i915->gpu_error.flags);
+ pr_info("%s(%s): %d resets\n", __func__, engine->name, count);
+
+ if (err)
+ break;
+
+ err = igt_flush_test(i915, 0);
+ if (err)
+ break;
+ }
+
+ mutex_lock(&i915->drm.struct_mutex);
+ err = igt_flush_test(i915, I915_WAIT_LOCKED);
+ mutex_unlock(&i915->drm.struct_mutex);
+
+ intel_runtime_pm_put(i915, wakeref);
+out:
+ mock_file_free(i915, file);
+ if (i915_reset_failed(i915))
+ err = -EIO;
+ return err;
+}
+
static int __igt_reset_engine(struct drm_i915_private *i915, bool active)
{
struct intel_engine_cs *engine;
break;
}
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_reset_failed(i915))
err = -EIO;
if (active) {
return 0;
if (i915_request_wait(rq, 0, 5 * HZ) < 0) {
- GEM_TRACE("%s timed out waiting for completion of fence %llx:%lld, seqno %d.\n",
+ GEM_TRACE("%s timed out waiting for completion of fence %llx:%lld\n",
rq->engine->name,
rq->fence.context,
- rq->fence.seqno,
- i915_request_global_seqno(rq));
+ rq->fence.seqno);
GEM_TRACE_DUMP();
i915_gem_set_wedged(rq->i915);
count++;
if (rq) {
- i915_request_wait(rq, 0, MAX_SCHEDULE_TIMEOUT);
+ if (i915_request_wait(rq, 0, HZ / 5) < 0) {
+ struct drm_printer p =
+ drm_info_printer(i915->drm.dev);
+
+ pr_err("i915_reset_engine(%s:%s):"
+ " failed to complete request after reset\n",
+ engine->name, test_name);
+ intel_engine_dump(engine, &p,
+ "%s\n", engine->name);
+ i915_request_put(rq);
+
+ GEM_TRACE_DUMP();
+ i915_gem_set_wedged(i915);
+ err = -EIO;
+ break;
+ }
+
i915_request_put(rq);
}
break;
}
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_reset_failed(i915))
err = -EIO;
if (flags & TEST_ACTIVE) {
return 0;
}
-static u32 fake_hangcheck(struct drm_i915_private *i915, u32 mask)
+static u32 fake_hangcheck(struct drm_i915_private *i915,
+ intel_engine_mask_t mask)
{
u32 count = i915_reset_count(&i915->gpu_error);
long timeout;
int err;
- if (!intel_engine_can_store_dword(i915->engine[RCS]))
+ if (!intel_engine_can_store_dword(i915->engine[RCS0]))
return 0;
/* Check that we detect a stuck waiter and issue a reset */
if (err)
goto unlock;
- rq = hang_create_request(&h, i915->engine[RCS]);
+ rq = hang_create_request(&h, i915->engine[RCS0]);
if (IS_ERR(rq)) {
err = PTR_ERR(rq);
goto fini;
mutex_unlock(&i915->drm.struct_mutex);
igt_global_reset_unlock(i915);
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_reset_failed(i915))
return -EIO;
return err;
struct hang h;
int err;
- if (!intel_engine_can_store_dword(i915->engine[RCS]))
+ if (!intel_engine_can_store_dword(i915->engine[RCS0]))
return 0;
/* Check that we can recover an unbind stuck on a hanging request */
- igt_global_reset_lock(i915);
-
mutex_lock(&i915->drm.struct_mutex);
err = hang_init(&h, i915);
if (err)
goto out_obj;
}
- rq = hang_create_request(&h, i915->engine[RCS]);
+ rq = hang_create_request(&h, i915->engine[RCS0]);
if (IS_ERR(rq)) {
err = PTR_ERR(rq);
goto out_obj;
}
out_reset:
- fake_hangcheck(rq->i915, intel_engine_flag(rq->engine));
+ igt_global_reset_lock(i915);
+ fake_hangcheck(rq->i915, rq->engine->mask);
+ igt_global_reset_unlock(i915);
if (tsk) {
struct igt_wedge_me w;
hang_fini(&h);
unlock:
mutex_unlock(&i915->drm.struct_mutex);
- igt_global_reset_unlock(i915);
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_reset_failed(i915))
return -EIO;
return err;
goto fini;
}
- reset_count = fake_hangcheck(i915, ENGINE_MASK(id));
+ reset_count = fake_hangcheck(i915, BIT(id));
if (prev->fence.error != -EIO) {
pr_err("GPU reset not recorded on hanging request [fence.error=%d]!\n",
mutex_unlock(&i915->drm.struct_mutex);
igt_global_reset_unlock(i915);
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_reset_failed(i915))
return -EIO;
return err;
static int igt_handle_error(void *arg)
{
struct drm_i915_private *i915 = arg;
- struct intel_engine_cs *engine = i915->engine[RCS];
+ struct intel_engine_cs *engine = i915->engine[RCS0];
struct hang h;
struct i915_request *rq;
struct i915_gpu_state *error;
/* Temporarily disable error capture */
error = xchg(&i915->gpu_error.first_error, (void *)-1);
- i915_handle_error(i915, ENGINE_MASK(engine->id), 0, NULL);
+ i915_handle_error(i915, engine->mask, 0, NULL);
xchg(&i915->gpu_error.first_error, error);
i915_request_wait(rq,
I915_WAIT_LOCKED,
MAX_SCHEDULE_TIMEOUT);
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_reset_failed(i915))
err = -EIO;
}
/* Flush any requests before we get started and check basics */
force_reset(i915);
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_reset_failed(i915))
goto unlock;
if (intel_has_gpu_reset(i915)) {
SUBTEST(igt_global_reset), /* attempt to recover GPU first */
SUBTEST(igt_wedged_reset),
SUBTEST(igt_hang_sanitycheck),
+ SUBTEST(igt_reset_nop),
+ SUBTEST(igt_reset_nop_engine),
SUBTEST(igt_reset_idle_engine),
SUBTEST(igt_reset_active_engine),
SUBTEST(igt_reset_engines),
if (!intel_has_gpu_reset(i915))
return 0;
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_terminally_wedged(i915))
return -EIO; /* we're long past hope of a successful reset */
wakeref = intel_runtime_pm_get(i915);
#include "../i915_selftest.h"
#include "igt_flush_test.h"
+#include "igt_live_test.h"
#include "igt_spinner.h"
#include "i915_random.h"
return err;
}
+static int live_busywait_preempt(void *arg)
+{
+ struct drm_i915_private *i915 = arg;
+ struct i915_gem_context *ctx_hi, *ctx_lo;
+ struct intel_engine_cs *engine;
+ struct drm_i915_gem_object *obj;
+ struct i915_vma *vma;
+ enum intel_engine_id id;
+ intel_wakeref_t wakeref;
+ int err = -ENOMEM;
+ u32 *map;
+
+ /*
+ * Verify that even without HAS_LOGICAL_RING_PREEMPTION, we can
+ * preempt the busywaits used to synchronise between rings.
+ */
+
+ mutex_lock(&i915->drm.struct_mutex);
+ wakeref = intel_runtime_pm_get(i915);
+
+ ctx_hi = kernel_context(i915);
+ if (!ctx_hi)
+ goto err_unlock;
+ ctx_hi->sched.priority = INT_MAX;
+
+ ctx_lo = kernel_context(i915);
+ if (!ctx_lo)
+ goto err_ctx_hi;
+ ctx_lo->sched.priority = INT_MIN;
+
+ obj = i915_gem_object_create_internal(i915, PAGE_SIZE);
+ if (IS_ERR(obj)) {
+ err = PTR_ERR(obj);
+ goto err_ctx_lo;
+ }
+
+ map = i915_gem_object_pin_map(obj, I915_MAP_WC);
+ if (IS_ERR(map)) {
+ err = PTR_ERR(map);
+ goto err_obj;
+ }
+
+ vma = i915_vma_instance(obj, &i915->ggtt.vm, NULL);
+ if (IS_ERR(vma)) {
+ err = PTR_ERR(vma);
+ goto err_map;
+ }
+
+ err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL);
+ if (err)
+ goto err_map;
+
+ for_each_engine(engine, i915, id) {
+ struct i915_request *lo, *hi;
+ struct igt_live_test t;
+ u32 *cs;
+
+ if (!intel_engine_can_store_dword(engine))
+ continue;
+
+ if (igt_live_test_begin(&t, i915, __func__, engine->name)) {
+ err = -EIO;
+ goto err_vma;
+ }
+
+ /*
+ * We create two requests. The low priority request
+ * busywaits on a semaphore (inside the ringbuffer where
+ * is should be preemptible) and the high priority requests
+ * uses a MI_STORE_DWORD_IMM to update the semaphore value
+ * allowing the first request to complete. If preemption
+ * fails, we hang instead.
+ */
+
+ lo = i915_request_alloc(engine, ctx_lo);
+ if (IS_ERR(lo)) {
+ err = PTR_ERR(lo);
+ goto err_vma;
+ }
+
+ cs = intel_ring_begin(lo, 8);
+ if (IS_ERR(cs)) {
+ err = PTR_ERR(cs);
+ i915_request_add(lo);
+ goto err_vma;
+ }
+
+ *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
+ *cs++ = i915_ggtt_offset(vma);
+ *cs++ = 0;
+ *cs++ = 1;
+
+ /* XXX Do we need a flush + invalidate here? */
+
+ *cs++ = MI_SEMAPHORE_WAIT |
+ MI_SEMAPHORE_GLOBAL_GTT |
+ MI_SEMAPHORE_POLL |
+ MI_SEMAPHORE_SAD_EQ_SDD;
+ *cs++ = 0;
+ *cs++ = i915_ggtt_offset(vma);
+ *cs++ = 0;
+
+ intel_ring_advance(lo, cs);
+ i915_request_add(lo);
+
+ if (wait_for(READ_ONCE(*map), 10)) {
+ err = -ETIMEDOUT;
+ goto err_vma;
+ }
+
+ /* Low priority request should be busywaiting now */
+ if (i915_request_wait(lo, I915_WAIT_LOCKED, 1) != -ETIME) {
+ pr_err("%s: Busywaiting request did not!\n",
+ engine->name);
+ err = -EIO;
+ goto err_vma;
+ }
+
+ hi = i915_request_alloc(engine, ctx_hi);
+ if (IS_ERR(hi)) {
+ err = PTR_ERR(hi);
+ goto err_vma;
+ }
+
+ cs = intel_ring_begin(hi, 4);
+ if (IS_ERR(cs)) {
+ err = PTR_ERR(cs);
+ i915_request_add(hi);
+ goto err_vma;
+ }
+
+ *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
+ *cs++ = i915_ggtt_offset(vma);
+ *cs++ = 0;
+ *cs++ = 0;
+
+ intel_ring_advance(hi, cs);
+ i915_request_add(hi);
+
+ if (i915_request_wait(lo, I915_WAIT_LOCKED, HZ / 5) < 0) {
+ struct drm_printer p = drm_info_printer(i915->drm.dev);
+
+ pr_err("%s: Failed to preempt semaphore busywait!\n",
+ engine->name);
+
+ intel_engine_dump(engine, &p, "%s\n", engine->name);
+ GEM_TRACE_DUMP();
+
+ i915_gem_set_wedged(i915);
+ err = -EIO;
+ goto err_vma;
+ }
+ GEM_BUG_ON(READ_ONCE(*map));
+
+ if (igt_live_test_end(&t)) {
+ err = -EIO;
+ goto err_vma;
+ }
+ }
+
+ err = 0;
+err_vma:
+ i915_vma_unpin(vma);
+err_map:
+ i915_gem_object_unpin_map(obj);
+err_obj:
+ i915_gem_object_put(obj);
+err_ctx_lo:
+ kernel_context_close(ctx_lo);
+err_ctx_hi:
+ kernel_context_close(ctx_hi);
+err_unlock:
+ if (igt_flush_test(i915, I915_WAIT_LOCKED))
+ err = -EIO;
+ intel_runtime_pm_put(i915, wakeref);
+ mutex_unlock(&i915->drm.struct_mutex);
+ return err;
+}
+
static int live_preempt(void *arg)
{
struct drm_i915_private *i915 = arg;
if (!HAS_LOGICAL_RING_PREEMPTION(i915))
return 0;
+ if (!(i915->caps.scheduler & I915_SCHEDULER_CAP_PREEMPTION))
+ pr_err("Logical preemption supported, but not exposed\n");
+
mutex_lock(&i915->drm.struct_mutex);
wakeref = intel_runtime_pm_get(i915);
I915_USER_PRIORITY(I915_CONTEXT_MIN_USER_PRIORITY);
for_each_engine(engine, i915, id) {
+ struct igt_live_test t;
struct i915_request *rq;
+ if (!intel_engine_has_preemption(engine))
+ continue;
+
+ if (igt_live_test_begin(&t, i915, __func__, engine->name)) {
+ err = -EIO;
+ goto err_ctx_lo;
+ }
+
rq = igt_spinner_create_request(&spin_lo, ctx_lo, engine,
MI_ARB_CHECK);
if (IS_ERR(rq)) {
igt_spinner_end(&spin_hi);
igt_spinner_end(&spin_lo);
- if (igt_flush_test(i915, I915_WAIT_LOCKED)) {
+
+ if (igt_live_test_end(&t)) {
err = -EIO;
goto err_ctx_lo;
}
goto err_ctx_hi;
for_each_engine(engine, i915, id) {
+ struct igt_live_test t;
struct i915_request *rq;
+ if (!intel_engine_has_preemption(engine))
+ continue;
+
+ if (igt_live_test_begin(&t, i915, __func__, engine->name)) {
+ err = -EIO;
+ goto err_ctx_lo;
+ }
+
rq = igt_spinner_create_request(&spin_lo, ctx_lo, engine,
MI_ARB_CHECK);
if (IS_ERR(rq)) {
igt_spinner_end(&spin_hi);
igt_spinner_end(&spin_lo);
- if (igt_flush_test(i915, I915_WAIT_LOCKED)) {
+
+ if (igt_live_test_end(&t)) {
err = -EIO;
goto err_ctx_lo;
}
struct i915_request *rq_a, *rq_b;
int depth;
+ if (!intel_engine_has_preemption(engine))
+ continue;
+
engine->execlists.preempt_hang.count = 0;
rq_a = igt_spinner_create_request(&a.spin,
goto err_client_b;
}
+static int __i915_sw_fence_call
+dummy_notify(struct i915_sw_fence *fence, enum i915_sw_fence_notify state)
+{
+ return NOTIFY_DONE;
+}
+
+static struct i915_request *dummy_request(struct intel_engine_cs *engine)
+{
+ struct i915_request *rq;
+
+ rq = kzalloc(sizeof(*rq), GFP_KERNEL);
+ if (!rq)
+ return NULL;
+
+ INIT_LIST_HEAD(&rq->active_list);
+ rq->engine = engine;
+
+ i915_sched_node_init(&rq->sched);
+
+ /* mark this request as permanently incomplete */
+ rq->fence.seqno = 1;
+ BUILD_BUG_ON(sizeof(rq->fence.seqno) != 8); /* upper 32b == 0 */
+ rq->hwsp_seqno = (u32 *)&rq->fence.seqno + 1;
+ GEM_BUG_ON(i915_request_completed(rq));
+
+ i915_sw_fence_init(&rq->submit, dummy_notify);
+ i915_sw_fence_commit(&rq->submit);
+
+ return rq;
+}
+
+static void dummy_request_free(struct i915_request *dummy)
+{
+ i915_request_mark_complete(dummy);
+ i915_sched_node_fini(&dummy->sched);
+ i915_sw_fence_fini(&dummy->submit);
+
+ dma_fence_free(&dummy->fence);
+}
+
+static int live_suppress_wait_preempt(void *arg)
+{
+ struct drm_i915_private *i915 = arg;
+ struct preempt_client client[4];
+ struct intel_engine_cs *engine;
+ enum intel_engine_id id;
+ intel_wakeref_t wakeref;
+ int err = -ENOMEM;
+ int i;
+
+ /*
+ * Waiters are given a little priority nudge, but not enough
+ * to actually cause any preemption. Double check that we do
+ * not needlessly generate preempt-to-idle cycles.
+ */
+
+ if (!HAS_LOGICAL_RING_PREEMPTION(i915))
+ return 0;
+
+ mutex_lock(&i915->drm.struct_mutex);
+ wakeref = intel_runtime_pm_get(i915);
+
+ if (preempt_client_init(i915, &client[0])) /* ELSP[0] */
+ goto err_unlock;
+ if (preempt_client_init(i915, &client[1])) /* ELSP[1] */
+ goto err_client_0;
+ if (preempt_client_init(i915, &client[2])) /* head of queue */
+ goto err_client_1;
+ if (preempt_client_init(i915, &client[3])) /* bystander */
+ goto err_client_2;
+
+ for_each_engine(engine, i915, id) {
+ int depth;
+
+ if (!intel_engine_has_preemption(engine))
+ continue;
+
+ if (!engine->emit_init_breadcrumb)
+ continue;
+
+ for (depth = 0; depth < ARRAY_SIZE(client); depth++) {
+ struct i915_request *rq[ARRAY_SIZE(client)];
+ struct i915_request *dummy;
+
+ engine->execlists.preempt_hang.count = 0;
+
+ dummy = dummy_request(engine);
+ if (!dummy)
+ goto err_client_3;
+
+ for (i = 0; i < ARRAY_SIZE(client); i++) {
+ rq[i] = igt_spinner_create_request(&client[i].spin,
+ client[i].ctx, engine,
+ MI_NOOP);
+ if (IS_ERR(rq[i])) {
+ err = PTR_ERR(rq[i]);
+ goto err_wedged;
+ }
+
+ /* Disable NEWCLIENT promotion */
+ __i915_active_request_set(&rq[i]->timeline->last_request,
+ dummy);
+ i915_request_add(rq[i]);
+ }
+
+ dummy_request_free(dummy);
+
+ GEM_BUG_ON(i915_request_completed(rq[0]));
+ if (!igt_wait_for_spinner(&client[0].spin, rq[0])) {
+ pr_err("%s: First client failed to start\n",
+ engine->name);
+ goto err_wedged;
+ }
+ GEM_BUG_ON(!i915_request_started(rq[0]));
+
+ if (i915_request_wait(rq[depth],
+ I915_WAIT_LOCKED |
+ I915_WAIT_PRIORITY,
+ 1) != -ETIME) {
+ pr_err("%s: Waiter depth:%d completed!\n",
+ engine->name, depth);
+ goto err_wedged;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(client); i++)
+ igt_spinner_end(&client[i].spin);
+
+ if (igt_flush_test(i915, I915_WAIT_LOCKED))
+ goto err_wedged;
+
+ if (engine->execlists.preempt_hang.count) {
+ pr_err("%s: Preemption recorded x%d, depth %d; should have been suppressed!\n",
+ engine->name,
+ engine->execlists.preempt_hang.count,
+ depth);
+ err = -EINVAL;
+ goto err_client_3;
+ }
+ }
+ }
+
+ err = 0;
+err_client_3:
+ preempt_client_fini(&client[3]);
+err_client_2:
+ preempt_client_fini(&client[2]);
+err_client_1:
+ preempt_client_fini(&client[1]);
+err_client_0:
+ preempt_client_fini(&client[0]);
+err_unlock:
+ if (igt_flush_test(i915, I915_WAIT_LOCKED))
+ err = -EIO;
+ intel_runtime_pm_put(i915, wakeref);
+ mutex_unlock(&i915->drm.struct_mutex);
+ return err;
+
+err_wedged:
+ for (i = 0; i < ARRAY_SIZE(client); i++)
+ igt_spinner_end(&client[i].spin);
+ i915_gem_set_wedged(i915);
+ err = -EIO;
+ goto err_client_3;
+}
+
static int live_chain_preempt(void *arg)
{
struct drm_i915_private *i915 = arg;
struct i915_sched_attr attr = {
.priority = I915_USER_PRIORITY(I915_PRIORITY_MAX),
};
- int count, i;
+ struct igt_live_test t;
+ struct i915_request *rq;
+ int ring_size, count, i;
- for_each_prime_number_from(count, 1, 32) { /* must fit ring! */
- struct i915_request *rq;
+ if (!intel_engine_has_preemption(engine))
+ continue;
+ rq = igt_spinner_create_request(&lo.spin,
+ lo.ctx, engine,
+ MI_ARB_CHECK);
+ if (IS_ERR(rq))
+ goto err_wedged;
+ i915_request_add(rq);
+
+ ring_size = rq->wa_tail - rq->head;
+ if (ring_size < 0)
+ ring_size += rq->ring->size;
+ ring_size = rq->ring->size / ring_size;
+ pr_debug("%s(%s): Using maximum of %d requests\n",
+ __func__, engine->name, ring_size);
+
+ igt_spinner_end(&lo.spin);
+ if (i915_request_wait(rq, I915_WAIT_LOCKED, HZ / 2) < 0) {
+ pr_err("Timed out waiting to flush %s\n", engine->name);
+ goto err_wedged;
+ }
+
+ if (igt_live_test_begin(&t, i915, __func__, engine->name)) {
+ err = -EIO;
+ goto err_wedged;
+ }
+
+ for_each_prime_number_from(count, 1, ring_size) {
rq = igt_spinner_create_request(&hi.spin,
hi.ctx, engine,
MI_ARB_CHECK);
goto err_wedged;
}
igt_spinner_end(&lo.spin);
+
+ rq = i915_request_alloc(engine, lo.ctx);
+ if (IS_ERR(rq))
+ goto err_wedged;
+ i915_request_add(rq);
+ if (i915_request_wait(rq, I915_WAIT_LOCKED, HZ / 5) < 0) {
+ struct drm_printer p =
+ drm_info_printer(i915->drm.dev);
+
+ pr_err("Failed to flush low priority chain of %d requests\n",
+ count);
+ intel_engine_dump(engine, &p,
+ "%s\n", engine->name);
+ goto err_wedged;
+ }
+ }
+
+ if (igt_live_test_end(&t)) {
+ err = -EIO;
+ goto err_wedged;
}
}
pr_info("Submitted %lu crescendo:%x requests across %d engines and %d contexts\n",
count, flags,
- RUNTIME_INFO(smoke->i915)->num_rings, smoke->ncontext);
+ RUNTIME_INFO(smoke->i915)->num_engines, smoke->ncontext);
return 0;
}
pr_info("Submitted %lu random:%x requests across %d engines and %d contexts\n",
count, flags,
- RUNTIME_INFO(smoke->i915)->num_rings, smoke->ncontext);
+ RUNTIME_INFO(smoke->i915)->num_engines, smoke->ncontext);
return 0;
}
};
const unsigned int phase[] = { 0, BATCH };
intel_wakeref_t wakeref;
+ struct igt_live_test t;
int err = -ENOMEM;
u32 *cs;
int n;
for (n = 0; n < PAGE_SIZE / sizeof(*cs) - 1; n++)
cs[n] = MI_ARB_CHECK;
cs[n] = MI_BATCH_BUFFER_END;
+ i915_gem_object_flush_map(smoke.batch);
i915_gem_object_unpin_map(smoke.batch);
- err = i915_gem_object_set_to_gtt_domain(smoke.batch, false);
- if (err)
+ if (igt_live_test_begin(&t, smoke.i915, __func__, "all")) {
+ err = -EIO;
goto err_batch;
+ }
for (n = 0; n < smoke.ncontext; n++) {
smoke.contexts[n] = kernel_context(smoke.i915);
}
err_ctx:
- if (igt_flush_test(smoke.i915, I915_WAIT_LOCKED))
+ if (igt_live_test_end(&t))
err = -EIO;
for (n = 0; n < smoke.ncontext; n++) {
{
static const struct i915_subtest tests[] = {
SUBTEST(live_sanitycheck),
+ SUBTEST(live_busywait_preempt),
SUBTEST(live_preempt),
SUBTEST(live_late_preempt),
SUBTEST(live_suppress_self_preempt),
+ SUBTEST(live_suppress_wait_preempt),
SUBTEST(live_chain_preempt),
SUBTEST(live_preempt_hang),
SUBTEST(live_preempt_smoke),
if (!HAS_EXECLISTS(i915))
return 0;
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_terminally_wedged(i915))
return 0;
return i915_subtests(tests, i915);
return 0;
}
-static int intel_uncore_check_forcewake_domains(struct drm_i915_private *dev_priv)
+static int live_forcewake_ops(void *arg)
+{
+ static const struct reg {
+ const char *name;
+ unsigned long platforms;
+ unsigned int offset;
+ } registers[] = {
+ {
+ "RING_START",
+ INTEL_GEN_MASK(6, 7),
+ 0x38,
+ },
+ {
+ "RING_MI_MODE",
+ INTEL_GEN_MASK(8, BITS_PER_LONG),
+ 0x9c,
+ }
+ };
+ const struct reg *r;
+ struct drm_i915_private *i915 = arg;
+ struct intel_uncore_forcewake_domain *domain;
+ struct intel_uncore *uncore = &i915->uncore;
+ struct intel_engine_cs *engine;
+ enum intel_engine_id id;
+ intel_wakeref_t wakeref;
+ unsigned int tmp;
+ int err = 0;
+
+ GEM_BUG_ON(i915->gt.awake);
+
+ /* vlv/chv with their pcu behave differently wrt reads */
+ if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915)) {
+ pr_debug("PCU fakes forcewake badly; skipping\n");
+ return 0;
+ }
+
+ /*
+ * Not quite as reliable across the gen as one would hope.
+ *
+ * Either our theory of operation is incorrect, or there remain
+ * external parties interfering with the powerwells.
+ *
+ * https://bugs.freedesktop.org/show_bug.cgi?id=110210
+ */
+ if (!IS_ENABLED(CONFIG_DRM_I915_SELFTEST_BROKEN))
+ return 0;
+
+ /* We have to pick carefully to get the exact behaviour we need */
+ for (r = registers; r->name; r++)
+ if (r->platforms & INTEL_INFO(i915)->gen_mask)
+ break;
+ if (!r->name) {
+ pr_debug("Forcewaked register not known for %s; skipping\n",
+ intel_platform_name(INTEL_INFO(i915)->platform));
+ return 0;
+ }
+
+ wakeref = intel_runtime_pm_get(i915);
+
+ for_each_fw_domain(domain, uncore, tmp) {
+ smp_store_mb(domain->active, false);
+ if (!hrtimer_cancel(&domain->timer))
+ continue;
+
+ intel_uncore_fw_release_timer(&domain->timer);
+ }
+
+ for_each_engine(engine, i915, id) {
+ i915_reg_t mmio = _MMIO(engine->mmio_base + r->offset);
+ u32 __iomem *reg = uncore->regs + engine->mmio_base + r->offset;
+ enum forcewake_domains fw_domains;
+ u32 val;
+
+ if (!engine->default_state)
+ continue;
+
+ fw_domains = intel_uncore_forcewake_for_reg(uncore, mmio,
+ FW_REG_READ);
+ if (!fw_domains)
+ continue;
+
+ for_each_fw_domain_masked(domain, fw_domains, uncore, tmp) {
+ if (!domain->wake_count)
+ continue;
+
+ pr_err("fw_domain %s still active, aborting test!\n",
+ intel_uncore_forcewake_domain_to_str(domain->id));
+ err = -EINVAL;
+ goto out_rpm;
+ }
+
+ intel_uncore_forcewake_get(uncore, fw_domains);
+ val = readl(reg);
+ intel_uncore_forcewake_put(uncore, fw_domains);
+
+ /* Flush the forcewake release (delayed onto a timer) */
+ for_each_fw_domain_masked(domain, fw_domains, uncore, tmp) {
+ smp_store_mb(domain->active, false);
+ if (hrtimer_cancel(&domain->timer))
+ intel_uncore_fw_release_timer(&domain->timer);
+
+ preempt_disable();
+ err = wait_ack_clear(domain, FORCEWAKE_KERNEL);
+ preempt_enable();
+ if (err) {
+ pr_err("Failed to clear fw_domain %s\n",
+ intel_uncore_forcewake_domain_to_str(domain->id));
+ goto out_rpm;
+ }
+ }
+
+ if (!val) {
+ pr_err("%s:%s was zero while fw was held!\n",
+ engine->name, r->name);
+ err = -EINVAL;
+ goto out_rpm;
+ }
+
+ /* We then expect the read to return 0 outside of the fw */
+ if (wait_for(readl(reg) == 0, 100)) {
+ pr_err("%s:%s=%0x, fw_domains 0x%x still up after 100ms!\n",
+ engine->name, r->name, readl(reg), fw_domains);
+ err = -ETIMEDOUT;
+ goto out_rpm;
+ }
+ }
+
+out_rpm:
+ intel_runtime_pm_put(i915, wakeref);
+ return err;
+}
+
+static int live_forcewake_domains(void *arg)
{
#define FW_RANGE 0x40000
+ struct drm_i915_private *dev_priv = arg;
+ struct intel_uncore *uncore = &dev_priv->uncore;
unsigned long *valid;
u32 offset;
int err;
if (!IS_ENABLED(CONFIG_DRM_I915_SELFTEST_BROKEN))
return 0;
- valid = kcalloc(BITS_TO_LONGS(FW_RANGE), sizeof(*valid),
- GFP_KERNEL);
+ valid = bitmap_zalloc(FW_RANGE, GFP_KERNEL);
if (!valid)
return -ENOMEM;
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
- check_for_unclaimed_mmio(dev_priv);
+ check_for_unclaimed_mmio(uncore);
for (offset = 0; offset < FW_RANGE; offset += 4) {
i915_reg_t reg = { offset };
(void)I915_READ_FW(reg);
- if (!check_for_unclaimed_mmio(dev_priv))
+ if (!check_for_unclaimed_mmio(uncore))
set_bit(offset, valid);
}
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
+ intel_uncore_forcewake_put(uncore, FORCEWAKE_ALL);
err = 0;
for_each_set_bit(offset, valid, FW_RANGE) {
i915_reg_t reg = { offset };
iosf_mbi_punit_acquire();
- intel_uncore_forcewake_reset(dev_priv);
+ intel_uncore_forcewake_reset(uncore);
iosf_mbi_punit_release();
- check_for_unclaimed_mmio(dev_priv);
+ check_for_unclaimed_mmio(uncore);
(void)I915_READ(reg);
- if (check_for_unclaimed_mmio(dev_priv)) {
+ if (check_for_unclaimed_mmio(uncore)) {
pr_err("Unclaimed mmio read to register 0x%04x\n",
offset);
err = -EINVAL;
}
}
- kfree(valid);
+ bitmap_free(valid);
return err;
}
int intel_uncore_live_selftests(struct drm_i915_private *i915)
{
+ static const struct i915_subtest tests[] = {
+ SUBTEST(live_forcewake_ops),
+ SUBTEST(live_forcewake_domains),
+ };
+
int err;
/* Confirm the table we load is still valid */
if (err)
return err;
- err = intel_uncore_check_forcewake_domains(i915);
- if (err)
- return err;
-
- return 0;
+ return i915_subtests(tests, i915);
}
#include "igt_spinner.h"
#include "igt_wedge_me.h"
#include "mock_context.h"
+#include "mock_drm.h"
+
+static const struct wo_register {
+ enum intel_platform platform;
+ u32 reg;
+} wo_registers[] = {
+ { INTEL_GEMINILAKE, 0x731c }
+};
#define REF_NAME_MAX (INTEL_ENGINE_CS_MAX_NAME + 4)
struct wa_lists {
if (IS_ERR(result))
return result;
- i915_gem_object_set_cache_level(result, I915_CACHE_LLC);
+ i915_gem_object_set_cache_coherency(result, I915_CACHE_LLC);
cs = i915_gem_object_pin_map(result, I915_MAP_WB);
if (IS_ERR(cs)) {
goto err_obj;
}
memset(cs, 0xc5, PAGE_SIZE);
+ i915_gem_object_flush_map(result);
i915_gem_object_unpin_map(result);
vma = i915_vma_instance(result, &engine->i915->ggtt.vm, NULL);
err = 0;
igt_wedge_on_timeout(&wedge, ctx->i915, HZ / 5) /* a safety net! */
err = i915_gem_object_set_to_cpu_domain(results, false);
- if (i915_terminally_wedged(&ctx->i915->gpu_error))
+ if (i915_terminally_wedged(ctx->i915))
err = -EIO;
if (err)
goto out_put;
static int do_device_reset(struct intel_engine_cs *engine)
{
- i915_reset(engine->i915, ENGINE_MASK(engine->id), "live_workarounds");
+ i915_reset(engine->i915, engine->mask, "live_workarounds");
return 0;
}
if (IS_ERR(ctx))
return PTR_ERR(ctx);
+ GEM_BUG_ON(i915_gem_context_is_bannable(ctx));
+
rq = ERR_PTR(-ENODEV);
- with_intel_runtime_pm(engine->i915, wakeref) {
- if (spin)
- rq = igt_spinner_create_request(spin,
- ctx, engine,
- MI_NOOP);
- else
- rq = i915_request_alloc(engine, ctx);
- }
+ with_intel_runtime_pm(engine->i915, wakeref)
+ rq = igt_spinner_create_request(spin, ctx, engine, MI_NOOP);
kernel_context_close(ctx);
const char *name)
{
struct drm_i915_private *i915 = engine->i915;
- bool want_spin = reset == do_engine_reset;
struct i915_gem_context *ctx;
struct igt_spinner spin;
intel_wakeref_t wakeref;
pr_info("Checking %d whitelisted registers (RING_NONPRIV) [%s]\n",
engine->whitelist.count, name);
- if (want_spin) {
- err = igt_spinner_init(&spin, i915);
- if (err)
- return err;
- }
+ err = igt_spinner_init(&spin, i915);
+ if (err)
+ return err;
ctx = kernel_context(i915);
if (IS_ERR(ctx))
goto out;
}
- err = switch_to_scratch_context(engine, want_spin ? &spin : NULL);
+ err = switch_to_scratch_context(engine, &spin);
if (err)
goto out;
with_intel_runtime_pm(i915, wakeref)
err = reset(engine);
- if (want_spin) {
- igt_spinner_end(&spin);
- igt_spinner_fini(&spin);
- }
+ igt_spinner_end(&spin);
+ igt_spinner_fini(&spin);
if (err) {
pr_err("%s reset failed\n", name);
return err;
}
+static struct i915_vma *create_scratch(struct i915_gem_context *ctx)
+{
+ struct drm_i915_gem_object *obj;
+ struct i915_vma *vma;
+ void *ptr;
+ int err;
+
+ obj = i915_gem_object_create_internal(ctx->i915, PAGE_SIZE);
+ if (IS_ERR(obj))
+ return ERR_CAST(obj);
+
+ i915_gem_object_set_cache_coherency(obj, I915_CACHE_LLC);
+
+ ptr = i915_gem_object_pin_map(obj, I915_MAP_WB);
+ if (IS_ERR(ptr)) {
+ err = PTR_ERR(ptr);
+ goto err_obj;
+ }
+ memset(ptr, 0xc5, PAGE_SIZE);
+ i915_gem_object_flush_map(obj);
+ i915_gem_object_unpin_map(obj);
+
+ vma = i915_vma_instance(obj, &ctx->ppgtt->vm, NULL);
+ if (IS_ERR(vma)) {
+ err = PTR_ERR(vma);
+ goto err_obj;
+ }
+
+ err = i915_vma_pin(vma, 0, 0, PIN_USER);
+ if (err)
+ goto err_obj;
+
+ err = i915_gem_object_set_to_cpu_domain(obj, false);
+ if (err)
+ goto err_obj;
+
+ return vma;
+
+err_obj:
+ i915_gem_object_put(obj);
+ return ERR_PTR(err);
+}
+
+static struct i915_vma *create_batch(struct i915_gem_context *ctx)
+{
+ struct drm_i915_gem_object *obj;
+ struct i915_vma *vma;
+ int err;
+
+ obj = i915_gem_object_create_internal(ctx->i915, 16 * PAGE_SIZE);
+ if (IS_ERR(obj))
+ return ERR_CAST(obj);
+
+ vma = i915_vma_instance(obj, &ctx->ppgtt->vm, NULL);
+ if (IS_ERR(vma)) {
+ err = PTR_ERR(vma);
+ goto err_obj;
+ }
+
+ err = i915_vma_pin(vma, 0, 0, PIN_USER);
+ if (err)
+ goto err_obj;
+
+ err = i915_gem_object_set_to_wc_domain(obj, true);
+ if (err)
+ goto err_obj;
+
+ return vma;
+
+err_obj:
+ i915_gem_object_put(obj);
+ return ERR_PTR(err);
+}
+
+static u32 reg_write(u32 old, u32 new, u32 rsvd)
+{
+ if (rsvd == 0x0000ffff) {
+ old &= ~(new >> 16);
+ old |= new & (new >> 16);
+ } else {
+ old &= ~rsvd;
+ old |= new & rsvd;
+ }
+
+ return old;
+}
+
+static bool wo_register(struct intel_engine_cs *engine, u32 reg)
+{
+ enum intel_platform platform = INTEL_INFO(engine->i915)->platform;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(wo_registers); i++) {
+ if (wo_registers[i].platform == platform &&
+ wo_registers[i].reg == reg)
+ return true;
+ }
+
+ return false;
+}
+
+static int check_dirty_whitelist(struct i915_gem_context *ctx,
+ struct intel_engine_cs *engine)
+{
+ const u32 values[] = {
+ 0x00000000,
+ 0x01010101,
+ 0x10100101,
+ 0x03030303,
+ 0x30300303,
+ 0x05050505,
+ 0x50500505,
+ 0x0f0f0f0f,
+ 0xf00ff00f,
+ 0x10101010,
+ 0xf0f01010,
+ 0x30303030,
+ 0xa0a03030,
+ 0x50505050,
+ 0xc0c05050,
+ 0xf0f0f0f0,
+ 0x11111111,
+ 0x33333333,
+ 0x55555555,
+ 0x0000ffff,
+ 0x00ff00ff,
+ 0xff0000ff,
+ 0xffff00ff,
+ 0xffffffff,
+ };
+ struct i915_vma *scratch;
+ struct i915_vma *batch;
+ int err = 0, i, v;
+ u32 *cs, *results;
+
+ scratch = create_scratch(ctx);
+ if (IS_ERR(scratch))
+ return PTR_ERR(scratch);
+
+ batch = create_batch(ctx);
+ if (IS_ERR(batch)) {
+ err = PTR_ERR(batch);
+ goto out_scratch;
+ }
+
+ for (i = 0; i < engine->whitelist.count; i++) {
+ u32 reg = i915_mmio_reg_offset(engine->whitelist.list[i].reg);
+ u64 addr = scratch->node.start;
+ struct i915_request *rq;
+ u32 srm, lrm, rsvd;
+ u32 expect;
+ int idx;
+
+ if (wo_register(engine, reg))
+ continue;
+
+ srm = MI_STORE_REGISTER_MEM;
+ lrm = MI_LOAD_REGISTER_MEM;
+ if (INTEL_GEN(ctx->i915) >= 8)
+ lrm++, srm++;
+
+ pr_debug("%s: Writing garbage to %x\n",
+ engine->name, reg);
+
+ cs = i915_gem_object_pin_map(batch->obj, I915_MAP_WC);
+ if (IS_ERR(cs)) {
+ err = PTR_ERR(cs);
+ goto out_batch;
+ }
+
+ /* SRM original */
+ *cs++ = srm;
+ *cs++ = reg;
+ *cs++ = lower_32_bits(addr);
+ *cs++ = upper_32_bits(addr);
+
+ idx = 1;
+ for (v = 0; v < ARRAY_SIZE(values); v++) {
+ /* LRI garbage */
+ *cs++ = MI_LOAD_REGISTER_IMM(1);
+ *cs++ = reg;
+ *cs++ = values[v];
+
+ /* SRM result */
+ *cs++ = srm;
+ *cs++ = reg;
+ *cs++ = lower_32_bits(addr + sizeof(u32) * idx);
+ *cs++ = upper_32_bits(addr + sizeof(u32) * idx);
+ idx++;
+ }
+ for (v = 0; v < ARRAY_SIZE(values); v++) {
+ /* LRI garbage */
+ *cs++ = MI_LOAD_REGISTER_IMM(1);
+ *cs++ = reg;
+ *cs++ = ~values[v];
+
+ /* SRM result */
+ *cs++ = srm;
+ *cs++ = reg;
+ *cs++ = lower_32_bits(addr + sizeof(u32) * idx);
+ *cs++ = upper_32_bits(addr + sizeof(u32) * idx);
+ idx++;
+ }
+ GEM_BUG_ON(idx * sizeof(u32) > scratch->size);
+
+ /* LRM original -- don't leave garbage in the context! */
+ *cs++ = lrm;
+ *cs++ = reg;
+ *cs++ = lower_32_bits(addr);
+ *cs++ = upper_32_bits(addr);
+
+ *cs++ = MI_BATCH_BUFFER_END;
+
+ i915_gem_object_flush_map(batch->obj);
+ i915_gem_object_unpin_map(batch->obj);
+ i915_gem_chipset_flush(ctx->i915);
+
+ rq = i915_request_alloc(engine, ctx);
+ if (IS_ERR(rq)) {
+ err = PTR_ERR(rq);
+ goto out_batch;
+ }
+
+ if (engine->emit_init_breadcrumb) { /* Be nice if we hang */
+ err = engine->emit_init_breadcrumb(rq);
+ if (err)
+ goto err_request;
+ }
+
+ err = engine->emit_bb_start(rq,
+ batch->node.start, PAGE_SIZE,
+ 0);
+ if (err)
+ goto err_request;
+
+err_request:
+ i915_request_add(rq);
+ if (err)
+ goto out_batch;
+
+ if (i915_request_wait(rq, I915_WAIT_LOCKED, HZ / 5) < 0) {
+ pr_err("%s: Futzing %x timedout; cancelling test\n",
+ engine->name, reg);
+ i915_gem_set_wedged(ctx->i915);
+ err = -EIO;
+ goto out_batch;
+ }
+
+ results = i915_gem_object_pin_map(scratch->obj, I915_MAP_WB);
+ if (IS_ERR(results)) {
+ err = PTR_ERR(results);
+ goto out_batch;
+ }
+
+ GEM_BUG_ON(values[ARRAY_SIZE(values) - 1] != 0xffffffff);
+ rsvd = results[ARRAY_SIZE(values)]; /* detect write masking */
+ if (!rsvd) {
+ pr_err("%s: Unable to write to whitelisted register %x\n",
+ engine->name, reg);
+ err = -EINVAL;
+ goto out_unpin;
+ }
+
+ expect = results[0];
+ idx = 1;
+ for (v = 0; v < ARRAY_SIZE(values); v++) {
+ expect = reg_write(expect, values[v], rsvd);
+ if (results[idx] != expect)
+ err++;
+ idx++;
+ }
+ for (v = 0; v < ARRAY_SIZE(values); v++) {
+ expect = reg_write(expect, ~values[v], rsvd);
+ if (results[idx] != expect)
+ err++;
+ idx++;
+ }
+ if (err) {
+ pr_err("%s: %d mismatch between values written to whitelisted register [%x], and values read back!\n",
+ engine->name, err, reg);
+
+ pr_info("%s: Whitelisted register: %x, original value %08x, rsvd %08x\n",
+ engine->name, reg, results[0], rsvd);
+
+ expect = results[0];
+ idx = 1;
+ for (v = 0; v < ARRAY_SIZE(values); v++) {
+ u32 w = values[v];
+
+ expect = reg_write(expect, w, rsvd);
+ pr_info("Wrote %08x, read %08x, expect %08x\n",
+ w, results[idx], expect);
+ idx++;
+ }
+ for (v = 0; v < ARRAY_SIZE(values); v++) {
+ u32 w = ~values[v];
+
+ expect = reg_write(expect, w, rsvd);
+ pr_info("Wrote %08x, read %08x, expect %08x\n",
+ w, results[idx], expect);
+ idx++;
+ }
+
+ err = -EINVAL;
+ }
+out_unpin:
+ i915_gem_object_unpin_map(scratch->obj);
+ if (err)
+ break;
+ }
+
+ if (igt_flush_test(ctx->i915, I915_WAIT_LOCKED))
+ err = -EIO;
+out_batch:
+ i915_vma_unpin_and_release(&batch, 0);
+out_scratch:
+ i915_vma_unpin_and_release(&scratch, 0);
+ return err;
+}
+
+static int live_dirty_whitelist(void *arg)
+{
+ struct drm_i915_private *i915 = arg;
+ struct intel_engine_cs *engine;
+ struct i915_gem_context *ctx;
+ enum intel_engine_id id;
+ intel_wakeref_t wakeref;
+ struct drm_file *file;
+ int err = 0;
+
+ /* Can the user write to the whitelisted registers? */
+
+ if (INTEL_GEN(i915) < 7) /* minimum requirement for LRI, SRM, LRM */
+ return 0;
+
+ wakeref = intel_runtime_pm_get(i915);
+
+ mutex_unlock(&i915->drm.struct_mutex);
+ file = mock_file(i915);
+ mutex_lock(&i915->drm.struct_mutex);
+ if (IS_ERR(file)) {
+ err = PTR_ERR(file);
+ goto out_rpm;
+ }
+
+ ctx = live_context(i915, file);
+ if (IS_ERR(ctx)) {
+ err = PTR_ERR(ctx);
+ goto out_file;
+ }
+
+ for_each_engine(engine, i915, id) {
+ if (engine->whitelist.count == 0)
+ continue;
+
+ err = check_dirty_whitelist(ctx, engine);
+ if (err)
+ goto out_file;
+ }
+
+out_file:
+ mutex_unlock(&i915->drm.struct_mutex);
+ mock_file_free(i915, file);
+ mutex_lock(&i915->drm.struct_mutex);
+out_rpm:
+ intel_runtime_pm_put(i915, wakeref);
+ return err;
+}
+
static int live_reset_whitelist(void *arg)
{
struct drm_i915_private *i915 = arg;
- struct intel_engine_cs *engine = i915->engine[RCS];
+ struct intel_engine_cs *engine = i915->engine[RCS0];
int err = 0;
/* If we reset the gpu, we should not lose the RING_NONPRIV */
enum intel_engine_id id;
bool ok = true;
- ok &= wa_list_verify(i915, &lists->gt_wa_list, str);
+ ok &= wa_list_verify(&i915->uncore, &lists->gt_wa_list, str);
for_each_engine(engine, i915, id)
- ok &= wa_list_verify(i915, &lists->engine[id].wa_list, str);
+ ok &= wa_list_verify(engine->uncore,
+ &lists->engine[id].wa_list, str);
return ok;
}
int intel_workarounds_live_selftests(struct drm_i915_private *i915)
{
static const struct i915_subtest tests[] = {
+ SUBTEST(live_dirty_whitelist),
SUBTEST(live_reset_whitelist),
SUBTEST(live_gpu_reset_gt_engine_workarounds),
SUBTEST(live_engine_reset_gt_engine_workarounds),
};
int err;
- if (i915_terminally_wedged(&i915->gpu_error))
+ if (i915_terminally_wedged(i915))
return 0;
mutex_lock(&i915->drm.struct_mutex);
const char *name)
{
struct i915_gem_context *ctx;
- unsigned int n;
int ret;
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
INIT_LIST_HEAD(&ctx->link);
ctx->i915 = i915;
+ ctx->hw_contexts = RB_ROOT;
+ spin_lock_init(&ctx->hw_contexts_lock);
+
INIT_RADIX_TREE(&ctx->handles_vma, GFP_KERNEL);
INIT_LIST_HEAD(&ctx->handles_list);
INIT_LIST_HEAD(&ctx->hw_id_link);
-
- for (n = 0; n < ARRAY_SIZE(ctx->__engine); n++)
- intel_context_init(&ctx->__engine[n], ctx, i915->engine[n]);
+ INIT_LIST_HEAD(&ctx->active_engines);
+ mutex_init(&ctx->mutex);
ret = i915_gem_context_pin_hw_id(ctx);
if (ret < 0)
goto err_handles;
if (name) {
+ struct i915_hw_ppgtt *ppgtt;
+
ctx->name = kstrdup(name, GFP_KERNEL);
if (!ctx->name)
goto err_put;
- ctx->ppgtt = mock_ppgtt(i915, name);
- if (!ctx->ppgtt)
+ ppgtt = mock_ppgtt(i915, name);
+ if (!ppgtt)
goto err_put;
+
+ __set_ppgtt(ctx, ppgtt);
}
return ctx;
struct i915_gem_context *
live_context(struct drm_i915_private *i915, struct drm_file *file)
{
+ struct i915_gem_context *ctx;
+ int err;
+
lockdep_assert_held(&i915->drm.struct_mutex);
- return i915_gem_create_context(i915, file->driver_priv);
+ ctx = i915_gem_create_context(i915, 0);
+ if (IS_ERR(ctx))
+ return ctx;
+
+ err = gem_context_register(ctx, file->driver_priv);
+ if (err < 0)
+ goto err_ctx;
+
+ return ctx;
+
+err_ctx:
+ context_close(ctx);
+ return ERR_PTR(err);
}
struct i915_gem_context *
if (!ring)
return NULL;
- if (i915_timeline_init(engine->i915,
- &ring->timeline, engine->name,
- NULL)) {
+ if (i915_timeline_init(engine->i915, &ring->timeline, NULL)) {
kfree(ring);
return NULL;
}
+ kref_init(&ring->base.ref);
ring->base.size = sz;
ring->base.effective_size = sz;
ring->base.vaddr = (void *)(ring + 1);
kfree(ring);
}
-static struct mock_request *first_request(struct mock_engine *engine)
+static struct i915_request *first_request(struct mock_engine *engine)
{
return list_first_entry_or_null(&engine->hw_queue,
- struct mock_request,
- link);
+ struct i915_request,
+ mock.link);
}
-static void advance(struct mock_request *request)
+static void advance(struct i915_request *request)
{
- list_del_init(&request->link);
- intel_engine_write_global_seqno(request->base.engine,
- request->base.global_seqno);
- i915_request_mark_complete(&request->base);
- GEM_BUG_ON(!i915_request_completed(&request->base));
+ list_del_init(&request->mock.link);
+ i915_request_mark_complete(request);
+ GEM_BUG_ON(!i915_request_completed(request));
- intel_engine_queue_breadcrumbs(request->base.engine);
+ intel_engine_queue_breadcrumbs(request->engine);
}
static void hw_delay_complete(struct timer_list *t)
{
struct mock_engine *engine = from_timer(engine, t, hw_delay);
- struct mock_request *request;
+ struct i915_request *request;
unsigned long flags;
spin_lock_irqsave(&engine->hw_lock, flags);
* requeue the timer for the next delayed request.
*/
while ((request = first_request(engine))) {
- if (request->delay) {
- mod_timer(&engine->hw_delay, jiffies + request->delay);
+ if (request->mock.delay) {
+ mod_timer(&engine->hw_delay,
+ jiffies + request->mock.delay);
break;
}
static void mock_context_unpin(struct intel_context *ce)
{
mock_timeline_unpin(ce->ring->timeline);
- i915_gem_context_put(ce->gem_context);
}
-static void mock_context_destroy(struct intel_context *ce)
+static void mock_context_destroy(struct kref *ref)
{
- GEM_BUG_ON(ce->pin_count);
+ struct intel_context *ce = container_of(ref, typeof(*ce), ref);
+
+ GEM_BUG_ON(intel_context_is_pinned(ce));
if (ce->ring)
mock_ring_free(ce->ring);
-}
-static const struct intel_context_ops mock_context_ops = {
- .unpin = mock_context_unpin,
- .destroy = mock_context_destroy,
-};
+ intel_context_free(ce);
+}
-static struct intel_context *
-mock_context_pin(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx)
+static int mock_context_pin(struct intel_context *ce)
{
- struct intel_context *ce = to_intel_context(ctx, engine);
- int err = -ENOMEM;
-
- if (ce->pin_count++)
- return ce;
-
if (!ce->ring) {
- ce->ring = mock_ring(engine);
+ ce->ring = mock_ring(ce->engine);
if (!ce->ring)
- goto err;
+ return -ENOMEM;
}
mock_timeline_pin(ce->ring->timeline);
+ return 0;
+}
- ce->ops = &mock_context_ops;
- i915_gem_context_get(ctx);
- return ce;
+static const struct intel_context_ops mock_context_ops = {
+ .pin = mock_context_pin,
+ .unpin = mock_context_unpin,
-err:
- ce->pin_count = 0;
- return ERR_PTR(err);
-}
+ .destroy = mock_context_destroy,
+};
static int mock_request_alloc(struct i915_request *request)
{
- struct mock_request *mock = container_of(request, typeof(*mock), base);
-
- INIT_LIST_HEAD(&mock->link);
- mock->delay = 0;
+ INIT_LIST_HEAD(&request->mock.link);
+ request->mock.delay = 0;
return 0;
}
static void mock_submit_request(struct i915_request *request)
{
- struct mock_request *mock = container_of(request, typeof(*mock), base);
struct mock_engine *engine =
container_of(request->engine, typeof(*engine), base);
unsigned long flags;
i915_request_submit(request);
- GEM_BUG_ON(!request->global_seqno);
spin_lock_irqsave(&engine->hw_lock, flags);
- list_add_tail(&mock->link, &engine->hw_queue);
- if (mock->link.prev == &engine->hw_queue) {
- if (mock->delay)
- mod_timer(&engine->hw_delay, jiffies + mock->delay);
+ list_add_tail(&request->mock.link, &engine->hw_queue);
+ if (list_is_first(&request->mock.link, &engine->hw_queue)) {
+ if (request->mock.delay)
+ mod_timer(&engine->hw_delay,
+ jiffies + request->mock.delay);
else
- advance(mock);
+ advance(request);
}
spin_unlock_irqrestore(&engine->hw_lock, flags);
}
+static void mock_reset_prepare(struct intel_engine_cs *engine)
+{
+}
+
+static void mock_reset(struct intel_engine_cs *engine, bool stalled)
+{
+ GEM_BUG_ON(stalled);
+}
+
+static void mock_reset_finish(struct intel_engine_cs *engine)
+{
+}
+
+static void mock_cancel_requests(struct intel_engine_cs *engine)
+{
+ struct i915_request *request;
+ unsigned long flags;
+
+ spin_lock_irqsave(&engine->timeline.lock, flags);
+
+ /* Mark all submitted requests as skipped. */
+ list_for_each_entry(request, &engine->timeline.requests, sched.link) {
+ if (!i915_request_signaled(request))
+ dma_fence_set_error(&request->fence, -EIO);
+
+ i915_request_mark_complete(request);
+ }
+
+ spin_unlock_irqrestore(&engine->timeline.lock, flags);
+}
+
struct intel_engine_cs *mock_engine(struct drm_i915_private *i915,
const char *name,
int id)
engine->base.i915 = i915;
snprintf(engine->base.name, sizeof(engine->base.name), "%s", name);
engine->base.id = id;
+ engine->base.mask = BIT(id);
engine->base.status_page.addr = (void *)(engine + 1);
- engine->base.context_pin = mock_context_pin;
+ engine->base.cops = &mock_context_ops;
engine->base.request_alloc = mock_request_alloc;
engine->base.emit_flush = mock_emit_flush;
engine->base.emit_fini_breadcrumb = mock_emit_breadcrumb;
engine->base.submit_request = mock_submit_request;
- if (i915_timeline_init(i915,
- &engine->base.timeline,
- engine->base.name,
- NULL))
+ engine->base.reset.prepare = mock_reset_prepare;
+ engine->base.reset.reset = mock_reset;
+ engine->base.reset.finish = mock_reset_finish;
+ engine->base.cancel_requests = mock_cancel_requests;
+
+ if (i915_timeline_init(i915, &engine->base.timeline, NULL))
goto err_free;
i915_timeline_set_subclass(&engine->base.timeline, TIMELINE_ENGINE);
timer_setup(&engine->hw_delay, hw_delay_complete, 0);
INIT_LIST_HEAD(&engine->hw_queue);
- if (IS_ERR(intel_context_pin(i915->kernel_context, &engine->base)))
+ if (pin_context(i915->kernel_context, &engine->base,
+ &engine->base.kernel_context))
goto err_breadcrumbs;
return &engine->base;
{
struct mock_engine *mock =
container_of(engine, typeof(*mock), base);
- struct mock_request *request, *rn;
+ struct i915_request *request, *rn;
del_timer_sync(&mock->hw_delay);
spin_lock_irq(&mock->hw_lock);
- list_for_each_entry_safe(request, rn, &mock->hw_queue, link)
+ list_for_each_entry_safe(request, rn, &mock->hw_queue, mock.link)
advance(request);
spin_unlock_irq(&mock->hw_lock);
}
void mock_engine_reset(struct intel_engine_cs *engine)
{
- intel_engine_write_global_seqno(engine, 0);
}
void mock_engine_free(struct intel_engine_cs *engine)
if (ce)
intel_context_unpin(ce);
- __intel_context_unpin(engine->i915->kernel_context, engine);
+ intel_context_unpin(engine->kernel_context);
intel_engine_fini_breadcrumbs(engine);
i915_timeline_fini(&engine->timeline);
destroy_workqueue(i915->wq);
- kmem_cache_destroy(i915->priorities);
- kmem_cache_destroy(i915->dependencies);
- kmem_cache_destroy(i915->requests);
- kmem_cache_destroy(i915->vmas);
- kmem_cache_destroy(i915->objects);
-
i915_gemfs_fini(i915);
drm_mode_config_cleanup(&i915->drm);
static void mock_idle_work_handler(struct work_struct *work)
{
+ struct drm_i915_private *i915 =
+ container_of(work, typeof(*i915), gt.idle_work.work);
+
+ i915->gt.active_engines = 0;
}
static int pm_domain_resume(struct device *dev)
I915_GTT_PAGE_SIZE_64K |
I915_GTT_PAGE_SIZE_2M;
- mock_uncore_init(i915);
+ mock_uncore_init(&i915->uncore);
i915_gem_init__mm(i915);
init_waitqueue_head(&i915->gpu_error.wait_queue);
init_waitqueue_head(&i915->gpu_error.reset_queue);
+ init_srcu_struct(&i915->gpu_error.reset_backoff_srcu);
mutex_init(&i915->gpu_error.wedge_mutex);
i915->wq = alloc_ordered_workqueue("mock", 0);
i915->gt.awake = true;
- i915->objects = KMEM_CACHE(mock_object, SLAB_HWCACHE_ALIGN);
- if (!i915->objects)
- goto err_wq;
-
- i915->vmas = KMEM_CACHE(i915_vma, SLAB_HWCACHE_ALIGN);
- if (!i915->vmas)
- goto err_objects;
-
- i915->requests = KMEM_CACHE(mock_request,
- SLAB_HWCACHE_ALIGN |
- SLAB_RECLAIM_ACCOUNT |
- SLAB_TYPESAFE_BY_RCU);
- if (!i915->requests)
- goto err_vmas;
-
- i915->dependencies = KMEM_CACHE(i915_dependency,
- SLAB_HWCACHE_ALIGN |
- SLAB_RECLAIM_ACCOUNT);
- if (!i915->dependencies)
- goto err_requests;
-
- i915->priorities = KMEM_CACHE(i915_priolist, SLAB_HWCACHE_ALIGN);
- if (!i915->priorities)
- goto err_dependencies;
-
i915_timelines_init(i915);
INIT_LIST_HEAD(&i915->gt.active_rings);
mock_init_ggtt(i915, &i915->ggtt);
- mkwrite_device_info(i915)->ring_mask = BIT(0);
+ mkwrite_device_info(i915)->engine_mask = BIT(0);
i915->kernel_context = mock_context(i915, NULL);
if (!i915->kernel_context)
goto err_unlock;
- i915->engine[RCS] = mock_engine(i915, "mock", RCS);
- if (!i915->engine[RCS])
+ i915->engine[RCS0] = mock_engine(i915, "mock", RCS0);
+ if (!i915->engine[RCS0])
goto err_context;
mutex_unlock(&i915->drm.struct_mutex);
err_unlock:
mutex_unlock(&i915->drm.struct_mutex);
i915_timelines_fini(i915);
- kmem_cache_destroy(i915->priorities);
-err_dependencies:
- kmem_cache_destroy(i915->dependencies);
-err_requests:
- kmem_cache_destroy(i915->requests);
-err_vmas:
- kmem_cache_destroy(i915->vmas);
-err_objects:
- kmem_cache_destroy(i915->objects);
-err_wq:
destroy_workqueue(i915->wq);
err_drv:
drm_mode_config_cleanup(&i915->drm);
unsigned long delay)
{
struct i915_request *request;
- struct mock_request *mock;
/* NB the i915->requests slab cache is enlarged to fit mock_request */
request = i915_request_alloc(engine, context);
if (IS_ERR(request))
return NULL;
- mock = container_of(request, typeof(*mock), base);
- mock->delay = delay;
-
- return &mock->base;
+ request->mock.delay = delay;
+ return request;
}
bool mock_cancel_request(struct i915_request *request)
{
- struct mock_request *mock = container_of(request, typeof(*mock), base);
struct mock_engine *engine =
container_of(request->engine, typeof(*engine), base);
bool was_queued;
spin_lock_irq(&engine->hw_lock);
- was_queued = !list_empty(&mock->link);
- list_del_init(&mock->link);
+ was_queued = !list_empty(&request->mock.link);
+ list_del_init(&request->mock.link);
spin_unlock_irq(&engine->hw_lock);
if (was_queued)
#include "../i915_request.h"
-struct mock_request {
- struct i915_request base;
-
- struct list_head link;
- unsigned long delay;
-};
-
struct i915_request *
mock_request(struct intel_engine_cs *engine,
struct i915_gem_context *context,
timeline->fence_context = context;
spin_lock_init(&timeline->lock);
+ mutex_init(&timeline->mutex);
- INIT_ACTIVE_REQUEST(&timeline->barrier);
INIT_ACTIVE_REQUEST(&timeline->last_request);
INIT_LIST_HEAD(&timeline->requests);
#define __nop_write(x) \
static void \
-nop_write##x(struct drm_i915_private *dev_priv, i915_reg_t reg, u##x val, bool trace) { }
+nop_write##x(struct intel_uncore *uncore, i915_reg_t reg, u##x val, bool trace) { }
__nop_write(8)
__nop_write(16)
__nop_write(32)
#define __nop_read(x) \
static u##x \
-nop_read##x(struct drm_i915_private *dev_priv, i915_reg_t reg, bool trace) { return 0; }
+nop_read##x(struct intel_uncore *uncore, i915_reg_t reg, bool trace) { return 0; }
__nop_read(8)
__nop_read(16)
__nop_read(32)
__nop_read(64)
-void mock_uncore_init(struct drm_i915_private *i915)
+void mock_uncore_init(struct intel_uncore *uncore)
{
- ASSIGN_WRITE_MMIO_VFUNCS(i915, nop);
- ASSIGN_READ_MMIO_VFUNCS(i915, nop);
+ ASSIGN_WRITE_MMIO_VFUNCS(uncore, nop);
+ ASSIGN_READ_MMIO_VFUNCS(uncore, nop);
}
#ifndef __MOCK_UNCORE_H
#define __MOCK_UNCORE_H
-void mock_uncore_init(struct drm_i915_private *i915);
+void mock_uncore_init(struct intel_uncore *uncore);
#endif /* !__MOCK_UNCORE_H */
* Author: Jani Nikula <jani.nikula@intel.com>
*/
+#include <linux/gpio/consumer.h>
+#include <linux/slab.h>
+
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
#include <drm/drm_edid.h>
-#include <drm/i915_drm.h>
#include <drm/drm_mipi_dsi.h>
-#include <linux/slab.h>
-#include <linux/gpio/consumer.h>
+#include <drm/i915_drm.h>
+
#include "i915_drv.h"
+#include "intel_connector.h"
#include "intel_drv.h"
#include "intel_dsi.h"
+#include "intel_panel.h"
/* return pixels in terms of txbyteclkhs */
static u16 txbyteclkhs(u16 pixels, int bpp, int lane_count,
mask = LP_CTRL_FIFO_EMPTY | HS_CTRL_FIFO_EMPTY |
LP_DATA_FIFO_EMPTY | HS_DATA_FIFO_EMPTY;
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
MIPI_GEN_FIFO_STAT(port), mask, mask,
100))
DRM_ERROR("DPI FIFOs are not empty\n");
/* note: this is never true for reads */
if (packet.payload_length) {
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
MIPI_GEN_FIFO_STAT(port),
data_mask, 0,
50))
I915_WRITE(MIPI_INTR_STAT(port), GEN_READ_DATA_AVAIL);
}
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
MIPI_GEN_FIFO_STAT(port),
ctrl_mask, 0,
50)) {
/* ->rx_len is set only for reads */
if (msg->rx_len) {
data_mask = GEN_READ_DATA_AVAIL;
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
MIPI_INTR_STAT(port),
data_mask, data_mask,
50))
I915_WRITE(MIPI_DPI_CONTROL(port), cmd);
mask = SPL_PKT_SENT_INTERRUPT;
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
MIPI_INTR_STAT(port), mask, mask,
100))
DRM_ERROR("Video mode command 0x%08x send failed.\n", cmd);
/* Wait for Pwr ACK */
for_each_dsi_port(port, intel_dsi->ports) {
- if (intel_wait_for_register(dev_priv,
- MIPI_CTRL(port), GLK_MIPIIO_PORT_POWERED,
- GLK_MIPIIO_PORT_POWERED, 20))
+ if (intel_wait_for_register(&dev_priv->uncore,
+ MIPI_CTRL(port),
+ GLK_MIPIIO_PORT_POWERED,
+ GLK_MIPIIO_PORT_POWERED,
+ 20))
DRM_ERROR("MIPIO port is powergated\n");
}
/* Check for cold boot scenario */
for_each_dsi_port(port, intel_dsi->ports) {
- cold_boot |= !(I915_READ(MIPI_DEVICE_READY(port)) &
- DEVICE_READY);
+ cold_boot |=
+ !(I915_READ(MIPI_DEVICE_READY(port)) & DEVICE_READY);
}
return cold_boot;
/* Wait for MIPI PHY status bit to set */
for_each_dsi_port(port, intel_dsi->ports) {
- if (intel_wait_for_register(dev_priv,
- MIPI_CTRL(port), GLK_PHY_STATUS_PORT_READY,
- GLK_PHY_STATUS_PORT_READY, 20))
+ if (intel_wait_for_register(&dev_priv->uncore,
+ MIPI_CTRL(port),
+ GLK_PHY_STATUS_PORT_READY,
+ GLK_PHY_STATUS_PORT_READY,
+ 20))
DRM_ERROR("PHY is not ON\n");
}
I915_WRITE(MIPI_DEVICE_READY(port), val);
/* Wait for ULPS active */
- if (intel_wait_for_register(dev_priv,
- MIPI_CTRL(port), GLK_ULPS_NOT_ACTIVE, 0, 20))
+ if (intel_wait_for_register(&dev_priv->uncore,
+ MIPI_CTRL(port),
+ GLK_ULPS_NOT_ACTIVE,
+ 0,
+ 20))
DRM_ERROR("ULPS not active\n");
/* Exit ULPS */
/* Wait for Stop state */
for_each_dsi_port(port, intel_dsi->ports) {
- if (intel_wait_for_register(dev_priv,
- MIPI_CTRL(port), GLK_DATA_LANE_STOP_STATE,
- GLK_DATA_LANE_STOP_STATE, 20))
+ if (intel_wait_for_register(&dev_priv->uncore,
+ MIPI_CTRL(port),
+ GLK_DATA_LANE_STOP_STATE,
+ GLK_DATA_LANE_STOP_STATE,
+ 20))
DRM_ERROR("Date lane not in STOP state\n");
}
/* Wait for AFE LATCH */
for_each_dsi_port(port, intel_dsi->ports) {
- if (intel_wait_for_register(dev_priv,
- BXT_MIPI_PORT_CTRL(port), AFE_LATCHOUT,
- AFE_LATCHOUT, 20))
+ if (intel_wait_for_register(&dev_priv->uncore,
+ BXT_MIPI_PORT_CTRL(port),
+ AFE_LATCHOUT,
+ AFE_LATCHOUT,
+ 20))
DRM_ERROR("D-PHY not entering LP-11 state\n");
}
}
/* Wait for MIPI PHY status bit to unset */
for_each_dsi_port(port, intel_dsi->ports) {
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
MIPI_CTRL(port),
GLK_PHY_STATUS_PORT_READY, 0, 20))
DRM_ERROR("PHY is not turning OFF\n");
/* Wait for Pwr ACK bit to unset */
for_each_dsi_port(port, intel_dsi->ports) {
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
MIPI_CTRL(port),
GLK_MIPIIO_PORT_POWERED, 0, 20))
DRM_ERROR("MIPI IO Port is not powergated\n");
/* Wait for MIPI PHY status bit to unset */
for_each_dsi_port(port, intel_dsi->ports) {
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
MIPI_CTRL(port),
GLK_PHY_STATUS_PORT_READY, 0, 20))
DRM_ERROR("PHY is not turning OFF\n");
* Port A only. MIPI Port C has no similar bit for checking.
*/
if ((IS_GEN9_LP(dev_priv) || port == PORT_A) &&
- intel_wait_for_register(dev_priv,
+ intel_wait_for_register(&dev_priv->uncore,
port_ctrl, AFE_LATCHOUT, 0,
30))
DRM_ERROR("DSI LP not going Low\n");
struct drm_encoder *encoder;
struct intel_connector *intel_connector;
struct drm_connector *connector;
- struct drm_display_mode *scan, *fixed_mode = NULL;
+ struct drm_display_mode *fixed_mode;
enum port port;
DRM_DEBUG_KMS("\n");
intel_connector_attach_encoder(intel_connector, intel_encoder);
mutex_lock(&dev->mode_config.mutex);
- intel_dsi_vbt_get_modes(intel_dsi);
- list_for_each_entry(scan, &connector->probed_modes, head) {
- if ((scan->type & DRM_MODE_TYPE_PREFERRED)) {
- fixed_mode = drm_mode_duplicate(dev, scan);
- break;
- }
- }
+ fixed_mode = intel_panel_vbt_fixed_mode(intel_connector);
mutex_unlock(&dev->mode_config.mutex);
if (!fixed_mode) {
goto err;
}
- connector->display_info.width_mm = fixed_mode->width_mm;
- connector->display_info.height_mm = fixed_mode->height_mm;
-
intel_panel_init(&intel_connector->panel, fixed_mode, NULL);
intel_panel_setup_backlight(connector, INVALID_PIPE);
* PLL lock should deassert within 200us.
* Wait up to 1ms before timing out.
*/
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
BXT_DSI_PLL_ENABLE,
BXT_DSI_PLL_LOCKED,
0,
I915_WRITE(BXT_DSI_PLL_ENABLE, val);
/* Timeout and fail if PLL not locked */
- if (intel_wait_for_register(dev_priv,
+ if (intel_wait_for_register(&dev_priv->uncore,
BXT_DSI_PLL_ENABLE,
BXT_DSI_PLL_LOCKED,
BXT_DSI_PLL_LOCKED,
}
/* Special case for LDB, one device for two channels */
- if (of_node_cmp(np->name, "lvds-channel") == 0) {
+ if (of_node_name_eq(np, "lvds-channel")) {
np = of_get_parent(np);
of_node_put(np);
}
sig_cfg.enable_pol = !(imx_crtc_state->bus_flags & DRM_BUS_FLAG_DE_LOW);
/* Default to driving pixel data on negative clock edges */
sig_cfg.clk_pol = !!(imx_crtc_state->bus_flags &
- DRM_BUS_FLAG_PIXDATA_POSEDGE);
+ DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE);
sig_cfg.bus_format = imx_crtc_state->bus_format;
sig_cfg.v_to_h_sync = 0;
sig_cfg.hsync_pin = imx_crtc_state->di_hsync_pin;
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0 OR MIT
+# Copyright 2017-2019 Qiang Yu <yuq825@gmail.com>
+
+config DRM_LIMA
+ tristate "LIMA (DRM support for ARM Mali 400/450 GPU)"
+ depends on DRM
+ depends on ARM || ARM64 || COMPILE_TEST
+ depends on MMU
+ depends on COMMON_CLK
+ depends on OF
+ select DRM_SCHED
+ help
+ DRM driver for ARM Mali 400/450 GPUs.
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0 OR MIT
+# Copyright 2017-2019 Qiang Yu <yuq825@gmail.com>
+
+lima-y := \
+ lima_drv.o \
+ lima_device.o \
+ lima_pmu.o \
+ lima_l2_cache.o \
+ lima_mmu.o \
+ lima_gp.o \
+ lima_pp.o \
+ lima_gem.o \
+ lima_vm.o \
+ lima_sched.o \
+ lima_ctx.o \
+ lima_gem_prime.o \
+ lima_dlbu.o \
+ lima_bcast.o \
+ lima_object.o
+
+obj-$(CONFIG_DRM_LIMA) += lima.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/io.h>
+#include <linux/device.h>
+
+#include "lima_device.h"
+#include "lima_bcast.h"
+#include "lima_regs.h"
+
+#define bcast_write(reg, data) writel(data, ip->iomem + reg)
+#define bcast_read(reg) readl(ip->iomem + reg)
+
+void lima_bcast_enable(struct lima_device *dev, int num_pp)
+{
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_pp;
+ struct lima_ip *ip = dev->ip + lima_ip_bcast;
+ int i, mask = bcast_read(LIMA_BCAST_BROADCAST_MASK) & 0xffff0000;
+
+ for (i = 0; i < num_pp; i++) {
+ struct lima_ip *pp = pipe->processor[i];
+
+ mask |= 1 << (pp->id - lima_ip_pp0);
+ }
+
+ bcast_write(LIMA_BCAST_BROADCAST_MASK, mask);
+}
+
+int lima_bcast_init(struct lima_ip *ip)
+{
+ int i, mask = 0;
+
+ for (i = lima_ip_pp0; i <= lima_ip_pp7; i++) {
+ if (ip->dev->ip[i].present)
+ mask |= 1 << (i - lima_ip_pp0);
+ }
+
+ bcast_write(LIMA_BCAST_BROADCAST_MASK, mask << 16);
+ bcast_write(LIMA_BCAST_INTERRUPT_MASK, mask);
+ return 0;
+}
+
+void lima_bcast_fini(struct lima_ip *ip)
+{
+
+}
+
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_BCAST_H__
+#define __LIMA_BCAST_H__
+
+struct lima_ip;
+
+int lima_bcast_init(struct lima_ip *ip);
+void lima_bcast_fini(struct lima_ip *ip);
+
+void lima_bcast_enable(struct lima_device *dev, int num_pp);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/slab.h>
+
+#include "lima_device.h"
+#include "lima_ctx.h"
+
+int lima_ctx_create(struct lima_device *dev, struct lima_ctx_mgr *mgr, u32 *id)
+{
+ struct lima_ctx *ctx;
+ int i, err;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return -ENOMEM;
+ ctx->dev = dev;
+ kref_init(&ctx->refcnt);
+
+ for (i = 0; i < lima_pipe_num; i++) {
+ err = lima_sched_context_init(dev->pipe + i, ctx->context + i, &ctx->guilty);
+ if (err)
+ goto err_out0;
+ }
+
+ err = xa_alloc(&mgr->handles, id, ctx, xa_limit_32b, GFP_KERNEL);
+ if (err < 0)
+ goto err_out0;
+
+ return 0;
+
+err_out0:
+ for (i--; i >= 0; i--)
+ lima_sched_context_fini(dev->pipe + i, ctx->context + i);
+ kfree(ctx);
+ return err;
+}
+
+static void lima_ctx_do_release(struct kref *ref)
+{
+ struct lima_ctx *ctx = container_of(ref, struct lima_ctx, refcnt);
+ int i;
+
+ for (i = 0; i < lima_pipe_num; i++)
+ lima_sched_context_fini(ctx->dev->pipe + i, ctx->context + i);
+ kfree(ctx);
+}
+
+int lima_ctx_free(struct lima_ctx_mgr *mgr, u32 id)
+{
+ struct lima_ctx *ctx;
+ int ret = 0;
+
+ mutex_lock(&mgr->lock);
+ ctx = xa_erase(&mgr->handles, id);
+ if (ctx)
+ kref_put(&ctx->refcnt, lima_ctx_do_release);
+ else
+ ret = -EINVAL;
+ mutex_unlock(&mgr->lock);
+ return ret;
+}
+
+struct lima_ctx *lima_ctx_get(struct lima_ctx_mgr *mgr, u32 id)
+{
+ struct lima_ctx *ctx;
+
+ mutex_lock(&mgr->lock);
+ ctx = xa_load(&mgr->handles, id);
+ if (ctx)
+ kref_get(&ctx->refcnt);
+ mutex_unlock(&mgr->lock);
+ return ctx;
+}
+
+void lima_ctx_put(struct lima_ctx *ctx)
+{
+ kref_put(&ctx->refcnt, lima_ctx_do_release);
+}
+
+void lima_ctx_mgr_init(struct lima_ctx_mgr *mgr)
+{
+ mutex_init(&mgr->lock);
+ xa_init_flags(&mgr->handles, XA_FLAGS_ALLOC);
+}
+
+void lima_ctx_mgr_fini(struct lima_ctx_mgr *mgr)
+{
+ struct lima_ctx *ctx;
+ unsigned long id;
+
+ xa_for_each(&mgr->handles, id, ctx) {
+ kref_put(&ctx->refcnt, lima_ctx_do_release);
+ }
+
+ xa_destroy(&mgr->handles);
+ mutex_destroy(&mgr->lock);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_CTX_H__
+#define __LIMA_CTX_H__
+
+#include <linux/xarray.h>
+
+#include "lima_device.h"
+
+struct lima_ctx {
+ struct kref refcnt;
+ struct lima_device *dev;
+ struct lima_sched_context context[lima_pipe_num];
+ atomic_t guilty;
+};
+
+struct lima_ctx_mgr {
+ struct mutex lock;
+ struct xarray handles;
+};
+
+int lima_ctx_create(struct lima_device *dev, struct lima_ctx_mgr *mgr, u32 *id);
+int lima_ctx_free(struct lima_ctx_mgr *mgr, u32 id);
+struct lima_ctx *lima_ctx_get(struct lima_ctx_mgr *mgr, u32 id);
+void lima_ctx_put(struct lima_ctx *ctx);
+void lima_ctx_mgr_init(struct lima_ctx_mgr *mgr);
+void lima_ctx_mgr_fini(struct lima_ctx_mgr *mgr);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/regulator/consumer.h>
+#include <linux/reset.h>
+#include <linux/clk.h>
+#include <linux/dma-mapping.h>
+#include <linux/platform_device.h>
+
+#include "lima_device.h"
+#include "lima_gp.h"
+#include "lima_pp.h"
+#include "lima_mmu.h"
+#include "lima_pmu.h"
+#include "lima_l2_cache.h"
+#include "lima_dlbu.h"
+#include "lima_bcast.h"
+#include "lima_vm.h"
+
+struct lima_ip_desc {
+ char *name;
+ char *irq_name;
+ bool must_have[lima_gpu_num];
+ int offset[lima_gpu_num];
+
+ int (*init)(struct lima_ip *ip);
+ void (*fini)(struct lima_ip *ip);
+};
+
+#define LIMA_IP_DESC(ipname, mst0, mst1, off0, off1, func, irq) \
+ [lima_ip_##ipname] = { \
+ .name = #ipname, \
+ .irq_name = irq, \
+ .must_have = { \
+ [lima_gpu_mali400] = mst0, \
+ [lima_gpu_mali450] = mst1, \
+ }, \
+ .offset = { \
+ [lima_gpu_mali400] = off0, \
+ [lima_gpu_mali450] = off1, \
+ }, \
+ .init = lima_##func##_init, \
+ .fini = lima_##func##_fini, \
+ }
+
+static struct lima_ip_desc lima_ip_desc[lima_ip_num] = {
+ LIMA_IP_DESC(pmu, false, false, 0x02000, 0x02000, pmu, "pmu"),
+ LIMA_IP_DESC(l2_cache0, true, true, 0x01000, 0x10000, l2_cache, NULL),
+ LIMA_IP_DESC(l2_cache1, false, true, -1, 0x01000, l2_cache, NULL),
+ LIMA_IP_DESC(l2_cache2, false, false, -1, 0x11000, l2_cache, NULL),
+ LIMA_IP_DESC(gp, true, true, 0x00000, 0x00000, gp, "gp"),
+ LIMA_IP_DESC(pp0, true, true, 0x08000, 0x08000, pp, "pp0"),
+ LIMA_IP_DESC(pp1, false, false, 0x0A000, 0x0A000, pp, "pp1"),
+ LIMA_IP_DESC(pp2, false, false, 0x0C000, 0x0C000, pp, "pp2"),
+ LIMA_IP_DESC(pp3, false, false, 0x0E000, 0x0E000, pp, "pp3"),
+ LIMA_IP_DESC(pp4, false, false, -1, 0x28000, pp, "pp4"),
+ LIMA_IP_DESC(pp5, false, false, -1, 0x2A000, pp, "pp5"),
+ LIMA_IP_DESC(pp6, false, false, -1, 0x2C000, pp, "pp6"),
+ LIMA_IP_DESC(pp7, false, false, -1, 0x2E000, pp, "pp7"),
+ LIMA_IP_DESC(gpmmu, true, true, 0x03000, 0x03000, mmu, "gpmmu"),
+ LIMA_IP_DESC(ppmmu0, true, true, 0x04000, 0x04000, mmu, "ppmmu0"),
+ LIMA_IP_DESC(ppmmu1, false, false, 0x05000, 0x05000, mmu, "ppmmu1"),
+ LIMA_IP_DESC(ppmmu2, false, false, 0x06000, 0x06000, mmu, "ppmmu2"),
+ LIMA_IP_DESC(ppmmu3, false, false, 0x07000, 0x07000, mmu, "ppmmu3"),
+ LIMA_IP_DESC(ppmmu4, false, false, -1, 0x1C000, mmu, "ppmmu4"),
+ LIMA_IP_DESC(ppmmu5, false, false, -1, 0x1D000, mmu, "ppmmu5"),
+ LIMA_IP_DESC(ppmmu6, false, false, -1, 0x1E000, mmu, "ppmmu6"),
+ LIMA_IP_DESC(ppmmu7, false, false, -1, 0x1F000, mmu, "ppmmu7"),
+ LIMA_IP_DESC(dlbu, false, true, -1, 0x14000, dlbu, NULL),
+ LIMA_IP_DESC(bcast, false, true, -1, 0x13000, bcast, NULL),
+ LIMA_IP_DESC(pp_bcast, false, true, -1, 0x16000, pp_bcast, "pp"),
+ LIMA_IP_DESC(ppmmu_bcast, false, true, -1, 0x15000, mmu, NULL),
+};
+
+const char *lima_ip_name(struct lima_ip *ip)
+{
+ return lima_ip_desc[ip->id].name;
+}
+
+static int lima_clk_init(struct lima_device *dev)
+{
+ int err;
+ unsigned long bus_rate, gpu_rate;
+
+ dev->clk_bus = devm_clk_get(dev->dev, "bus");
+ if (IS_ERR(dev->clk_bus)) {
+ dev_err(dev->dev, "get bus clk failed %ld\n", PTR_ERR(dev->clk_bus));
+ return PTR_ERR(dev->clk_bus);
+ }
+
+ dev->clk_gpu = devm_clk_get(dev->dev, "core");
+ if (IS_ERR(dev->clk_gpu)) {
+ dev_err(dev->dev, "get core clk failed %ld\n", PTR_ERR(dev->clk_gpu));
+ return PTR_ERR(dev->clk_gpu);
+ }
+
+ bus_rate = clk_get_rate(dev->clk_bus);
+ dev_info(dev->dev, "bus rate = %lu\n", bus_rate);
+
+ gpu_rate = clk_get_rate(dev->clk_gpu);
+ dev_info(dev->dev, "mod rate = %lu", gpu_rate);
+
+ err = clk_prepare_enable(dev->clk_bus);
+ if (err)
+ return err;
+
+ err = clk_prepare_enable(dev->clk_gpu);
+ if (err)
+ goto error_out0;
+
+ dev->reset = devm_reset_control_get_optional(dev->dev, NULL);
+ if (IS_ERR(dev->reset)) {
+ err = PTR_ERR(dev->reset);
+ goto error_out1;
+ } else if (dev->reset != NULL) {
+ err = reset_control_deassert(dev->reset);
+ if (err)
+ goto error_out1;
+ }
+
+ return 0;
+
+error_out1:
+ clk_disable_unprepare(dev->clk_gpu);
+error_out0:
+ clk_disable_unprepare(dev->clk_bus);
+ return err;
+}
+
+static void lima_clk_fini(struct lima_device *dev)
+{
+ if (dev->reset != NULL)
+ reset_control_assert(dev->reset);
+ clk_disable_unprepare(dev->clk_gpu);
+ clk_disable_unprepare(dev->clk_bus);
+}
+
+static int lima_regulator_init(struct lima_device *dev)
+{
+ int ret;
+
+ dev->regulator = devm_regulator_get_optional(dev->dev, "mali");
+ if (IS_ERR(dev->regulator)) {
+ ret = PTR_ERR(dev->regulator);
+ dev->regulator = NULL;
+ if (ret == -ENODEV)
+ return 0;
+ dev_err(dev->dev, "failed to get regulator: %d\n", ret);
+ return ret;
+ }
+
+ ret = regulator_enable(dev->regulator);
+ if (ret < 0) {
+ dev_err(dev->dev, "failed to enable regulator: %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void lima_regulator_fini(struct lima_device *dev)
+{
+ if (dev->regulator)
+ regulator_disable(dev->regulator);
+}
+
+static int lima_init_ip(struct lima_device *dev, int index)
+{
+ struct lima_ip_desc *desc = lima_ip_desc + index;
+ struct lima_ip *ip = dev->ip + index;
+ int offset = desc->offset[dev->id];
+ bool must = desc->must_have[dev->id];
+ int err;
+
+ if (offset < 0)
+ return 0;
+
+ ip->dev = dev;
+ ip->id = index;
+ ip->iomem = dev->iomem + offset;
+ if (desc->irq_name) {
+ err = platform_get_irq_byname(dev->pdev, desc->irq_name);
+ if (err < 0)
+ goto out;
+ ip->irq = err;
+ }
+
+ err = desc->init(ip);
+ if (!err) {
+ ip->present = true;
+ return 0;
+ }
+
+out:
+ return must ? err : 0;
+}
+
+static void lima_fini_ip(struct lima_device *ldev, int index)
+{
+ struct lima_ip_desc *desc = lima_ip_desc + index;
+ struct lima_ip *ip = ldev->ip + index;
+
+ if (ip->present)
+ desc->fini(ip);
+}
+
+static int lima_init_gp_pipe(struct lima_device *dev)
+{
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_gp;
+ int err;
+
+ err = lima_sched_pipe_init(pipe, "gp");
+ if (err)
+ return err;
+
+ pipe->l2_cache[pipe->num_l2_cache++] = dev->ip + lima_ip_l2_cache0;
+ pipe->mmu[pipe->num_mmu++] = dev->ip + lima_ip_gpmmu;
+ pipe->processor[pipe->num_processor++] = dev->ip + lima_ip_gp;
+
+ err = lima_gp_pipe_init(dev);
+ if (err) {
+ lima_sched_pipe_fini(pipe);
+ return err;
+ }
+
+ return 0;
+}
+
+static void lima_fini_gp_pipe(struct lima_device *dev)
+{
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_gp;
+
+ lima_gp_pipe_fini(dev);
+ lima_sched_pipe_fini(pipe);
+}
+
+static int lima_init_pp_pipe(struct lima_device *dev)
+{
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_pp;
+ int err, i;
+
+ err = lima_sched_pipe_init(pipe, "pp");
+ if (err)
+ return err;
+
+ for (i = 0; i < LIMA_SCHED_PIPE_MAX_PROCESSOR; i++) {
+ struct lima_ip *pp = dev->ip + lima_ip_pp0 + i;
+ struct lima_ip *ppmmu = dev->ip + lima_ip_ppmmu0 + i;
+ struct lima_ip *l2_cache;
+
+ if (dev->id == lima_gpu_mali400)
+ l2_cache = dev->ip + lima_ip_l2_cache0;
+ else
+ l2_cache = dev->ip + lima_ip_l2_cache1 + (i >> 2);
+
+ if (pp->present && ppmmu->present && l2_cache->present) {
+ pipe->mmu[pipe->num_mmu++] = ppmmu;
+ pipe->processor[pipe->num_processor++] = pp;
+ if (!pipe->l2_cache[i >> 2])
+ pipe->l2_cache[pipe->num_l2_cache++] = l2_cache;
+ }
+ }
+
+ if (dev->ip[lima_ip_bcast].present) {
+ pipe->bcast_processor = dev->ip + lima_ip_pp_bcast;
+ pipe->bcast_mmu = dev->ip + lima_ip_ppmmu_bcast;
+ }
+
+ err = lima_pp_pipe_init(dev);
+ if (err) {
+ lima_sched_pipe_fini(pipe);
+ return err;
+ }
+
+ return 0;
+}
+
+static void lima_fini_pp_pipe(struct lima_device *dev)
+{
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_pp;
+
+ lima_pp_pipe_fini(dev);
+ lima_sched_pipe_fini(pipe);
+}
+
+int lima_device_init(struct lima_device *ldev)
+{
+ int err, i;
+ struct resource *res;
+
+ dma_set_coherent_mask(ldev->dev, DMA_BIT_MASK(32));
+
+ err = lima_clk_init(ldev);
+ if (err) {
+ dev_err(ldev->dev, "clk init fail %d\n", err);
+ return err;
+ }
+
+ err = lima_regulator_init(ldev);
+ if (err) {
+ dev_err(ldev->dev, "regulator init fail %d\n", err);
+ goto err_out0;
+ }
+
+ ldev->empty_vm = lima_vm_create(ldev);
+ if (!ldev->empty_vm) {
+ err = -ENOMEM;
+ goto err_out1;
+ }
+
+ ldev->va_start = 0;
+ if (ldev->id == lima_gpu_mali450) {
+ ldev->va_end = LIMA_VA_RESERVE_START;
+ ldev->dlbu_cpu = dma_alloc_wc(
+ ldev->dev, LIMA_PAGE_SIZE,
+ &ldev->dlbu_dma, GFP_KERNEL);
+ if (!ldev->dlbu_cpu) {
+ err = -ENOMEM;
+ goto err_out2;
+ }
+ } else
+ ldev->va_end = LIMA_VA_RESERVE_END;
+
+ res = platform_get_resource(ldev->pdev, IORESOURCE_MEM, 0);
+ ldev->iomem = devm_ioremap_resource(ldev->dev, res);
+ if (IS_ERR(ldev->iomem)) {
+ dev_err(ldev->dev, "fail to ioremap iomem\n");
+ err = PTR_ERR(ldev->iomem);
+ goto err_out3;
+ }
+
+ for (i = 0; i < lima_ip_num; i++) {
+ err = lima_init_ip(ldev, i);
+ if (err)
+ goto err_out4;
+ }
+
+ err = lima_init_gp_pipe(ldev);
+ if (err)
+ goto err_out4;
+
+ err = lima_init_pp_pipe(ldev);
+ if (err)
+ goto err_out5;
+
+ return 0;
+
+err_out5:
+ lima_fini_gp_pipe(ldev);
+err_out4:
+ while (--i >= 0)
+ lima_fini_ip(ldev, i);
+err_out3:
+ if (ldev->dlbu_cpu)
+ dma_free_wc(ldev->dev, LIMA_PAGE_SIZE,
+ ldev->dlbu_cpu, ldev->dlbu_dma);
+err_out2:
+ lima_vm_put(ldev->empty_vm);
+err_out1:
+ lima_regulator_fini(ldev);
+err_out0:
+ lima_clk_fini(ldev);
+ return err;
+}
+
+void lima_device_fini(struct lima_device *ldev)
+{
+ int i;
+
+ lima_fini_pp_pipe(ldev);
+ lima_fini_gp_pipe(ldev);
+
+ for (i = lima_ip_num - 1; i >= 0; i--)
+ lima_fini_ip(ldev, i);
+
+ if (ldev->dlbu_cpu)
+ dma_free_wc(ldev->dev, LIMA_PAGE_SIZE,
+ ldev->dlbu_cpu, ldev->dlbu_dma);
+
+ lima_vm_put(ldev->empty_vm);
+
+ lima_regulator_fini(ldev);
+
+ lima_clk_fini(ldev);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_DEVICE_H__
+#define __LIMA_DEVICE_H__
+
+#include <drm/drm_device.h>
+#include <linux/delay.h>
+
+#include "lima_sched.h"
+
+enum lima_gpu_id {
+ lima_gpu_mali400 = 0,
+ lima_gpu_mali450,
+ lima_gpu_num,
+};
+
+enum lima_ip_id {
+ lima_ip_pmu,
+ lima_ip_gpmmu,
+ lima_ip_ppmmu0,
+ lima_ip_ppmmu1,
+ lima_ip_ppmmu2,
+ lima_ip_ppmmu3,
+ lima_ip_ppmmu4,
+ lima_ip_ppmmu5,
+ lima_ip_ppmmu6,
+ lima_ip_ppmmu7,
+ lima_ip_gp,
+ lima_ip_pp0,
+ lima_ip_pp1,
+ lima_ip_pp2,
+ lima_ip_pp3,
+ lima_ip_pp4,
+ lima_ip_pp5,
+ lima_ip_pp6,
+ lima_ip_pp7,
+ lima_ip_l2_cache0,
+ lima_ip_l2_cache1,
+ lima_ip_l2_cache2,
+ lima_ip_dlbu,
+ lima_ip_bcast,
+ lima_ip_pp_bcast,
+ lima_ip_ppmmu_bcast,
+ lima_ip_num,
+};
+
+struct lima_device;
+
+struct lima_ip {
+ struct lima_device *dev;
+ enum lima_ip_id id;
+ bool present;
+
+ void __iomem *iomem;
+ int irq;
+
+ union {
+ /* gp/pp */
+ bool async_reset;
+ /* l2 cache */
+ spinlock_t lock;
+ } data;
+};
+
+enum lima_pipe_id {
+ lima_pipe_gp,
+ lima_pipe_pp,
+ lima_pipe_num,
+};
+
+struct lima_device {
+ struct device *dev;
+ struct drm_device *ddev;
+ struct platform_device *pdev;
+
+ enum lima_gpu_id id;
+ u32 gp_version;
+ u32 pp_version;
+ int num_pp;
+
+ void __iomem *iomem;
+ struct clk *clk_bus;
+ struct clk *clk_gpu;
+ struct reset_control *reset;
+ struct regulator *regulator;
+
+ struct lima_ip ip[lima_ip_num];
+ struct lima_sched_pipe pipe[lima_pipe_num];
+
+ struct lima_vm *empty_vm;
+ uint64_t va_start;
+ uint64_t va_end;
+
+ u32 *dlbu_cpu;
+ dma_addr_t dlbu_dma;
+};
+
+static inline struct lima_device *
+to_lima_dev(struct drm_device *dev)
+{
+ return dev->dev_private;
+}
+
+int lima_device_init(struct lima_device *ldev);
+void lima_device_fini(struct lima_device *ldev);
+
+const char *lima_ip_name(struct lima_ip *ip);
+
+typedef int (*lima_poll_func_t)(struct lima_ip *);
+
+static inline int lima_poll_timeout(struct lima_ip *ip, lima_poll_func_t func,
+ int sleep_us, int timeout_us)
+{
+ ktime_t timeout = ktime_add_us(ktime_get(), timeout_us);
+
+ might_sleep_if(sleep_us);
+ while (1) {
+ if (func(ip))
+ return 0;
+
+ if (timeout_us && ktime_compare(ktime_get(), timeout) > 0)
+ return -ETIMEDOUT;
+
+ if (sleep_us)
+ usleep_range((sleep_us >> 2) + 1, sleep_us);
+ }
+ return 0;
+}
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/io.h>
+#include <linux/device.h>
+
+#include "lima_device.h"
+#include "lima_dlbu.h"
+#include "lima_vm.h"
+#include "lima_regs.h"
+
+#define dlbu_write(reg, data) writel(data, ip->iomem + reg)
+#define dlbu_read(reg) readl(ip->iomem + reg)
+
+void lima_dlbu_enable(struct lima_device *dev, int num_pp)
+{
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_pp;
+ struct lima_ip *ip = dev->ip + lima_ip_dlbu;
+ int i, mask = 0;
+
+ for (i = 0; i < num_pp; i++) {
+ struct lima_ip *pp = pipe->processor[i];
+
+ mask |= 1 << (pp->id - lima_ip_pp0);
+ }
+
+ dlbu_write(LIMA_DLBU_PP_ENABLE_MASK, mask);
+}
+
+void lima_dlbu_disable(struct lima_device *dev)
+{
+ struct lima_ip *ip = dev->ip + lima_ip_dlbu;
+
+ dlbu_write(LIMA_DLBU_PP_ENABLE_MASK, 0);
+}
+
+void lima_dlbu_set_reg(struct lima_ip *ip, u32 *reg)
+{
+ dlbu_write(LIMA_DLBU_TLLIST_VBASEADDR, reg[0]);
+ dlbu_write(LIMA_DLBU_FB_DIM, reg[1]);
+ dlbu_write(LIMA_DLBU_TLLIST_CONF, reg[2]);
+ dlbu_write(LIMA_DLBU_START_TILE_POS, reg[3]);
+}
+
+int lima_dlbu_init(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+
+ dlbu_write(LIMA_DLBU_MASTER_TLLIST_PHYS_ADDR, dev->dlbu_dma | 1);
+ dlbu_write(LIMA_DLBU_MASTER_TLLIST_VADDR, LIMA_VA_RESERVE_DLBU);
+
+ return 0;
+}
+
+void lima_dlbu_fini(struct lima_ip *ip)
+{
+
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_DLBU_H__
+#define __LIMA_DLBU_H__
+
+struct lima_ip;
+struct lima_device;
+
+void lima_dlbu_enable(struct lima_device *dev, int num_pp);
+void lima_dlbu_disable(struct lima_device *dev);
+
+void lima_dlbu_set_reg(struct lima_ip *ip, u32 *reg);
+
+int lima_dlbu_init(struct lima_ip *ip);
+void lima_dlbu_fini(struct lima_ip *ip);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/uaccess.h>
+#include <linux/slab.h>
+#include <drm/drm_ioctl.h>
+#include <drm/drm_drv.h>
+#include <drm/drm_prime.h>
+#include <drm/lima_drm.h>
+
+#include "lima_drv.h"
+#include "lima_gem.h"
+#include "lima_gem_prime.h"
+#include "lima_vm.h"
+
+int lima_sched_timeout_ms;
+
+MODULE_PARM_DESC(sched_timeout_ms, "task run timeout in ms (0 = no timeout (default))");
+module_param_named(sched_timeout_ms, lima_sched_timeout_ms, int, 0444);
+
+static int lima_ioctl_get_param(struct drm_device *dev, void *data, struct drm_file *file)
+{
+ struct drm_lima_get_param *args = data;
+ struct lima_device *ldev = to_lima_dev(dev);
+
+ if (args->pad)
+ return -EINVAL;
+
+ switch (args->param) {
+ case DRM_LIMA_PARAM_GPU_ID:
+ switch (ldev->id) {
+ case lima_gpu_mali400:
+ args->value = DRM_LIMA_PARAM_GPU_ID_MALI400;
+ break;
+ case lima_gpu_mali450:
+ args->value = DRM_LIMA_PARAM_GPU_ID_MALI450;
+ break;
+ default:
+ args->value = DRM_LIMA_PARAM_GPU_ID_UNKNOWN;
+ break;
+ }
+ break;
+
+ case DRM_LIMA_PARAM_NUM_PP:
+ args->value = ldev->pipe[lima_pipe_pp].num_processor;
+ break;
+
+ case DRM_LIMA_PARAM_GP_VERSION:
+ args->value = ldev->gp_version;
+ break;
+
+ case DRM_LIMA_PARAM_PP_VERSION:
+ args->value = ldev->pp_version;
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int lima_ioctl_gem_create(struct drm_device *dev, void *data, struct drm_file *file)
+{
+ struct drm_lima_gem_create *args = data;
+
+ if (args->pad)
+ return -EINVAL;
+
+ if (args->flags)
+ return -EINVAL;
+
+ if (args->size == 0)
+ return -EINVAL;
+
+ return lima_gem_create_handle(dev, file, args->size, args->flags, &args->handle);
+}
+
+static int lima_ioctl_gem_info(struct drm_device *dev, void *data, struct drm_file *file)
+{
+ struct drm_lima_gem_info *args = data;
+
+ return lima_gem_get_info(file, args->handle, &args->va, &args->offset);
+}
+
+static int lima_ioctl_gem_submit(struct drm_device *dev, void *data, struct drm_file *file)
+{
+ struct drm_lima_gem_submit *args = data;
+ struct lima_device *ldev = to_lima_dev(dev);
+ struct lima_drm_priv *priv = file->driver_priv;
+ struct drm_lima_gem_submit_bo *bos;
+ struct lima_sched_pipe *pipe;
+ struct lima_sched_task *task;
+ struct lima_ctx *ctx;
+ struct lima_submit submit = {0};
+ size_t size;
+ int err = 0;
+
+ if (args->pipe >= lima_pipe_num || args->nr_bos == 0)
+ return -EINVAL;
+
+ if (args->flags & ~(LIMA_SUBMIT_FLAG_EXPLICIT_FENCE))
+ return -EINVAL;
+
+ pipe = ldev->pipe + args->pipe;
+ if (args->frame_size != pipe->frame_size)
+ return -EINVAL;
+
+ bos = kvcalloc(args->nr_bos, sizeof(*submit.bos) + sizeof(*submit.lbos), GFP_KERNEL);
+ if (!bos)
+ return -ENOMEM;
+
+ size = args->nr_bos * sizeof(*submit.bos);
+ if (copy_from_user(bos, u64_to_user_ptr(args->bos), size)) {
+ err = -EFAULT;
+ goto out0;
+ }
+
+ task = kmem_cache_zalloc(pipe->task_slab, GFP_KERNEL);
+ if (!task) {
+ err = -ENOMEM;
+ goto out0;
+ }
+
+ task->frame = task + 1;
+ if (copy_from_user(task->frame, u64_to_user_ptr(args->frame), args->frame_size)) {
+ err = -EFAULT;
+ goto out1;
+ }
+
+ err = pipe->task_validate(pipe, task);
+ if (err)
+ goto out1;
+
+ ctx = lima_ctx_get(&priv->ctx_mgr, args->ctx);
+ if (!ctx) {
+ err = -ENOENT;
+ goto out1;
+ }
+
+ submit.pipe = args->pipe;
+ submit.bos = bos;
+ submit.lbos = (void *)bos + size;
+ submit.nr_bos = args->nr_bos;
+ submit.task = task;
+ submit.ctx = ctx;
+ submit.flags = args->flags;
+ submit.in_sync[0] = args->in_sync[0];
+ submit.in_sync[1] = args->in_sync[1];
+ submit.out_sync = args->out_sync;
+
+ err = lima_gem_submit(file, &submit);
+
+ lima_ctx_put(ctx);
+out1:
+ if (err)
+ kmem_cache_free(pipe->task_slab, task);
+out0:
+ kvfree(bos);
+ return err;
+}
+
+static int lima_ioctl_gem_wait(struct drm_device *dev, void *data, struct drm_file *file)
+{
+ struct drm_lima_gem_wait *args = data;
+
+ if (args->op & ~(LIMA_GEM_WAIT_READ|LIMA_GEM_WAIT_WRITE))
+ return -EINVAL;
+
+ return lima_gem_wait(file, args->handle, args->op, args->timeout_ns);
+}
+
+static int lima_ioctl_ctx_create(struct drm_device *dev, void *data, struct drm_file *file)
+{
+ struct drm_lima_ctx_create *args = data;
+ struct lima_drm_priv *priv = file->driver_priv;
+ struct lima_device *ldev = to_lima_dev(dev);
+
+ if (args->_pad)
+ return -EINVAL;
+
+ return lima_ctx_create(ldev, &priv->ctx_mgr, &args->id);
+}
+
+static int lima_ioctl_ctx_free(struct drm_device *dev, void *data, struct drm_file *file)
+{
+ struct drm_lima_ctx_create *args = data;
+ struct lima_drm_priv *priv = file->driver_priv;
+
+ if (args->_pad)
+ return -EINVAL;
+
+ return lima_ctx_free(&priv->ctx_mgr, args->id);
+}
+
+static int lima_drm_driver_open(struct drm_device *dev, struct drm_file *file)
+{
+ int err;
+ struct lima_drm_priv *priv;
+ struct lima_device *ldev = to_lima_dev(dev);
+
+ priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->vm = lima_vm_create(ldev);
+ if (!priv->vm) {
+ err = -ENOMEM;
+ goto err_out0;
+ }
+
+ lima_ctx_mgr_init(&priv->ctx_mgr);
+
+ file->driver_priv = priv;
+ return 0;
+
+err_out0:
+ kfree(priv);
+ return err;
+}
+
+static void lima_drm_driver_postclose(struct drm_device *dev, struct drm_file *file)
+{
+ struct lima_drm_priv *priv = file->driver_priv;
+
+ lima_ctx_mgr_fini(&priv->ctx_mgr);
+ lima_vm_put(priv->vm);
+ kfree(priv);
+}
+
+static const struct drm_ioctl_desc lima_drm_driver_ioctls[] = {
+ DRM_IOCTL_DEF_DRV(LIMA_GET_PARAM, lima_ioctl_get_param, DRM_AUTH|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(LIMA_GEM_CREATE, lima_ioctl_gem_create, DRM_AUTH|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(LIMA_GEM_INFO, lima_ioctl_gem_info, DRM_AUTH|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(LIMA_GEM_SUBMIT, lima_ioctl_gem_submit, DRM_AUTH|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(LIMA_GEM_WAIT, lima_ioctl_gem_wait, DRM_AUTH|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(LIMA_CTX_CREATE, lima_ioctl_ctx_create, DRM_AUTH|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(LIMA_CTX_FREE, lima_ioctl_ctx_free, DRM_AUTH|DRM_RENDER_ALLOW),
+};
+
+static const struct file_operations lima_drm_driver_fops = {
+ .owner = THIS_MODULE,
+ .open = drm_open,
+ .release = drm_release,
+ .unlocked_ioctl = drm_ioctl,
+#ifdef CONFIG_COMPAT
+ .compat_ioctl = drm_compat_ioctl,
+#endif
+ .mmap = lima_gem_mmap,
+};
+
+static struct drm_driver lima_drm_driver = {
+ .driver_features = DRIVER_RENDER | DRIVER_GEM | DRIVER_PRIME | DRIVER_SYNCOBJ,
+ .open = lima_drm_driver_open,
+ .postclose = lima_drm_driver_postclose,
+ .ioctls = lima_drm_driver_ioctls,
+ .num_ioctls = ARRAY_SIZE(lima_drm_driver_ioctls),
+ .fops = &lima_drm_driver_fops,
+ .gem_free_object_unlocked = lima_gem_free_object,
+ .gem_open_object = lima_gem_object_open,
+ .gem_close_object = lima_gem_object_close,
+ .gem_vm_ops = &lima_gem_vm_ops,
+ .name = "lima",
+ .desc = "lima DRM",
+ .date = "20190217",
+ .major = 1,
+ .minor = 0,
+ .patchlevel = 0,
+
+ .prime_fd_to_handle = drm_gem_prime_fd_to_handle,
+ .gem_prime_import_sg_table = lima_gem_prime_import_sg_table,
+ .prime_handle_to_fd = drm_gem_prime_handle_to_fd,
+ .gem_prime_get_sg_table = lima_gem_prime_get_sg_table,
+ .gem_prime_mmap = lima_gem_prime_mmap,
+};
+
+static int lima_pdev_probe(struct platform_device *pdev)
+{
+ struct lima_device *ldev;
+ struct drm_device *ddev;
+ int err;
+
+ err = lima_sched_slab_init();
+ if (err)
+ return err;
+
+ ldev = devm_kzalloc(&pdev->dev, sizeof(*ldev), GFP_KERNEL);
+ if (!ldev) {
+ err = -ENOMEM;
+ goto err_out0;
+ }
+
+ ldev->pdev = pdev;
+ ldev->dev = &pdev->dev;
+ ldev->id = (enum lima_gpu_id)of_device_get_match_data(&pdev->dev);
+
+ platform_set_drvdata(pdev, ldev);
+
+ /* Allocate and initialize the DRM device. */
+ ddev = drm_dev_alloc(&lima_drm_driver, &pdev->dev);
+ if (IS_ERR(ddev))
+ return PTR_ERR(ddev);
+
+ ddev->dev_private = ldev;
+ ldev->ddev = ddev;
+
+ err = lima_device_init(ldev);
+ if (err) {
+ dev_err(&pdev->dev, "Fatal error during GPU init\n");
+ goto err_out1;
+ }
+
+ /*
+ * Register the DRM device with the core and the connectors with
+ * sysfs.
+ */
+ err = drm_dev_register(ddev, 0);
+ if (err < 0)
+ goto err_out2;
+
+ return 0;
+
+err_out2:
+ lima_device_fini(ldev);
+err_out1:
+ drm_dev_put(ddev);
+err_out0:
+ lima_sched_slab_fini();
+ return err;
+}
+
+static int lima_pdev_remove(struct platform_device *pdev)
+{
+ struct lima_device *ldev = platform_get_drvdata(pdev);
+ struct drm_device *ddev = ldev->ddev;
+
+ drm_dev_unregister(ddev);
+ lima_device_fini(ldev);
+ drm_dev_put(ddev);
+ lima_sched_slab_fini();
+ return 0;
+}
+
+static const struct of_device_id dt_match[] = {
+ { .compatible = "arm,mali-400", .data = (void *)lima_gpu_mali400 },
+ { .compatible = "arm,mali-450", .data = (void *)lima_gpu_mali450 },
+ {}
+};
+MODULE_DEVICE_TABLE(of, dt_match);
+
+static struct platform_driver lima_platform_driver = {
+ .probe = lima_pdev_probe,
+ .remove = lima_pdev_remove,
+ .driver = {
+ .name = "lima",
+ .of_match_table = dt_match,
+ },
+};
+
+static int __init lima_init(void)
+{
+ return platform_driver_register(&lima_platform_driver);
+}
+module_init(lima_init);
+
+static void __exit lima_exit(void)
+{
+ platform_driver_unregister(&lima_platform_driver);
+}
+module_exit(lima_exit);
+
+MODULE_AUTHOR("Lima Project Developers");
+MODULE_DESCRIPTION("Lima DRM Driver");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_DRV_H__
+#define __LIMA_DRV_H__
+
+#include <drm/drm_file.h>
+
+#include "lima_ctx.h"
+
+extern int lima_sched_timeout_ms;
+
+struct lima_vm;
+struct lima_bo;
+struct lima_sched_task;
+
+struct drm_lima_gem_submit_bo;
+
+struct lima_drm_priv {
+ struct lima_vm *vm;
+ struct lima_ctx_mgr ctx_mgr;
+};
+
+struct lima_submit {
+ struct lima_ctx *ctx;
+ int pipe;
+ u32 flags;
+
+ struct drm_lima_gem_submit_bo *bos;
+ struct lima_bo **lbos;
+ u32 nr_bos;
+
+ u32 in_sync[2];
+ u32 out_sync;
+
+ struct lima_sched_task *task;
+};
+
+static inline struct lima_drm_priv *
+to_lima_drm_priv(struct drm_file *file)
+{
+ return file->driver_priv;
+}
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/mm.h>
+#include <linux/sync_file.h>
+#include <linux/pfn_t.h>
+
+#include <drm/drm_file.h>
+#include <drm/drm_syncobj.h>
+#include <drm/drm_utils.h>
+
+#include <drm/lima_drm.h>
+
+#include "lima_drv.h"
+#include "lima_gem.h"
+#include "lima_gem_prime.h"
+#include "lima_vm.h"
+#include "lima_object.h"
+
+int lima_gem_create_handle(struct drm_device *dev, struct drm_file *file,
+ u32 size, u32 flags, u32 *handle)
+{
+ int err;
+ struct lima_bo *bo;
+ struct lima_device *ldev = to_lima_dev(dev);
+
+ bo = lima_bo_create(ldev, size, flags, NULL, NULL);
+ if (IS_ERR(bo))
+ return PTR_ERR(bo);
+
+ err = drm_gem_handle_create(file, &bo->gem, handle);
+
+ /* drop reference from allocate - handle holds it now */
+ drm_gem_object_put_unlocked(&bo->gem);
+
+ return err;
+}
+
+void lima_gem_free_object(struct drm_gem_object *obj)
+{
+ struct lima_bo *bo = to_lima_bo(obj);
+
+ if (!list_empty(&bo->va))
+ dev_err(obj->dev->dev, "lima gem free bo still has va\n");
+
+ lima_bo_destroy(bo);
+}
+
+int lima_gem_object_open(struct drm_gem_object *obj, struct drm_file *file)
+{
+ struct lima_bo *bo = to_lima_bo(obj);
+ struct lima_drm_priv *priv = to_lima_drm_priv(file);
+ struct lima_vm *vm = priv->vm;
+
+ return lima_vm_bo_add(vm, bo, true);
+}
+
+void lima_gem_object_close(struct drm_gem_object *obj, struct drm_file *file)
+{
+ struct lima_bo *bo = to_lima_bo(obj);
+ struct lima_drm_priv *priv = to_lima_drm_priv(file);
+ struct lima_vm *vm = priv->vm;
+
+ lima_vm_bo_del(vm, bo);
+}
+
+int lima_gem_get_info(struct drm_file *file, u32 handle, u32 *va, u64 *offset)
+{
+ struct drm_gem_object *obj;
+ struct lima_bo *bo;
+ struct lima_drm_priv *priv = to_lima_drm_priv(file);
+ struct lima_vm *vm = priv->vm;
+ int err;
+
+ obj = drm_gem_object_lookup(file, handle);
+ if (!obj)
+ return -ENOENT;
+
+ bo = to_lima_bo(obj);
+
+ *va = lima_vm_get_va(vm, bo);
+
+ err = drm_gem_create_mmap_offset(obj);
+ if (!err)
+ *offset = drm_vma_node_offset_addr(&obj->vma_node);
+
+ drm_gem_object_put_unlocked(obj);
+ return err;
+}
+
+static vm_fault_t lima_gem_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+ struct drm_gem_object *obj = vma->vm_private_data;
+ struct lima_bo *bo = to_lima_bo(obj);
+ pfn_t pfn;
+ pgoff_t pgoff;
+
+ /* We don't use vmf->pgoff since that has the fake offset: */
+ pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
+ pfn = __pfn_to_pfn_t(page_to_pfn(bo->pages[pgoff]), PFN_DEV);
+
+ return vmf_insert_mixed(vma, vmf->address, pfn);
+}
+
+const struct vm_operations_struct lima_gem_vm_ops = {
+ .fault = lima_gem_fault,
+ .open = drm_gem_vm_open,
+ .close = drm_gem_vm_close,
+};
+
+void lima_set_vma_flags(struct vm_area_struct *vma)
+{
+ pgprot_t prot = vm_get_page_prot(vma->vm_flags);
+
+ vma->vm_flags |= VM_MIXEDMAP;
+ vma->vm_flags &= ~VM_PFNMAP;
+ vma->vm_page_prot = pgprot_writecombine(prot);
+}
+
+int lima_gem_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+ int ret;
+
+ ret = drm_gem_mmap(filp, vma);
+ if (ret)
+ return ret;
+
+ lima_set_vma_flags(vma);
+ return 0;
+}
+
+static int lima_gem_sync_bo(struct lima_sched_task *task, struct lima_bo *bo,
+ bool write, bool explicit)
+{
+ int err = 0;
+
+ if (!write) {
+ err = reservation_object_reserve_shared(bo->gem.resv, 1);
+ if (err)
+ return err;
+ }
+
+ /* explicit sync use user passed dep fence */
+ if (explicit)
+ return 0;
+
+ return drm_gem_fence_array_add_implicit(&task->deps, &bo->gem, write);
+}
+
+static int lima_gem_lock_bos(struct lima_bo **bos, u32 nr_bos,
+ struct ww_acquire_ctx *ctx)
+{
+ int i, ret = 0, contended, slow_locked = -1;
+
+ ww_acquire_init(ctx, &reservation_ww_class);
+
+retry:
+ for (i = 0; i < nr_bos; i++) {
+ if (i == slow_locked) {
+ slow_locked = -1;
+ continue;
+ }
+
+ ret = ww_mutex_lock_interruptible(&bos[i]->gem.resv->lock, ctx);
+ if (ret < 0) {
+ contended = i;
+ goto err;
+ }
+ }
+
+ ww_acquire_done(ctx);
+ return 0;
+
+err:
+ for (i--; i >= 0; i--)
+ ww_mutex_unlock(&bos[i]->gem.resv->lock);
+
+ if (slow_locked >= 0)
+ ww_mutex_unlock(&bos[slow_locked]->gem.resv->lock);
+
+ if (ret == -EDEADLK) {
+ /* we lost out in a seqno race, lock and retry.. */
+ ret = ww_mutex_lock_slow_interruptible(
+ &bos[contended]->gem.resv->lock, ctx);
+ if (!ret) {
+ slow_locked = contended;
+ goto retry;
+ }
+ }
+ ww_acquire_fini(ctx);
+
+ return ret;
+}
+
+static void lima_gem_unlock_bos(struct lima_bo **bos, u32 nr_bos,
+ struct ww_acquire_ctx *ctx)
+{
+ int i;
+
+ for (i = 0; i < nr_bos; i++)
+ ww_mutex_unlock(&bos[i]->gem.resv->lock);
+ ww_acquire_fini(ctx);
+}
+
+static int lima_gem_add_deps(struct drm_file *file, struct lima_submit *submit)
+{
+ int i, err;
+
+ for (i = 0; i < ARRAY_SIZE(submit->in_sync); i++) {
+ struct dma_fence *fence = NULL;
+
+ if (!submit->in_sync[i])
+ continue;
+
+ err = drm_syncobj_find_fence(file, submit->in_sync[i],
+ 0, 0, &fence);
+ if (err)
+ return err;
+
+ err = drm_gem_fence_array_add(&submit->task->deps, fence);
+ if (err) {
+ dma_fence_put(fence);
+ return err;
+ }
+ }
+
+ return 0;
+}
+
+int lima_gem_submit(struct drm_file *file, struct lima_submit *submit)
+{
+ int i, err = 0;
+ struct ww_acquire_ctx ctx;
+ struct lima_drm_priv *priv = to_lima_drm_priv(file);
+ struct lima_vm *vm = priv->vm;
+ struct drm_syncobj *out_sync = NULL;
+ struct dma_fence *fence;
+ struct lima_bo **bos = submit->lbos;
+
+ if (submit->out_sync) {
+ out_sync = drm_syncobj_find(file, submit->out_sync);
+ if (!out_sync)
+ return -ENOENT;
+ }
+
+ for (i = 0; i < submit->nr_bos; i++) {
+ struct drm_gem_object *obj;
+ struct lima_bo *bo;
+
+ obj = drm_gem_object_lookup(file, submit->bos[i].handle);
+ if (!obj) {
+ err = -ENOENT;
+ goto err_out0;
+ }
+
+ bo = to_lima_bo(obj);
+
+ /* increase refcnt of gpu va map to prevent unmapped when executing,
+ * will be decreased when task done
+ */
+ err = lima_vm_bo_add(vm, bo, false);
+ if (err) {
+ drm_gem_object_put_unlocked(obj);
+ goto err_out0;
+ }
+
+ bos[i] = bo;
+ }
+
+ err = lima_gem_lock_bos(bos, submit->nr_bos, &ctx);
+ if (err)
+ goto err_out0;
+
+ err = lima_sched_task_init(
+ submit->task, submit->ctx->context + submit->pipe,
+ bos, submit->nr_bos, vm);
+ if (err)
+ goto err_out1;
+
+ err = lima_gem_add_deps(file, submit);
+ if (err)
+ goto err_out2;
+
+ for (i = 0; i < submit->nr_bos; i++) {
+ err = lima_gem_sync_bo(
+ submit->task, bos[i],
+ submit->bos[i].flags & LIMA_SUBMIT_BO_WRITE,
+ submit->flags & LIMA_SUBMIT_FLAG_EXPLICIT_FENCE);
+ if (err)
+ goto err_out2;
+ }
+
+ fence = lima_sched_context_queue_task(
+ submit->ctx->context + submit->pipe, submit->task);
+
+ for (i = 0; i < submit->nr_bos; i++) {
+ if (submit->bos[i].flags & LIMA_SUBMIT_BO_WRITE)
+ reservation_object_add_excl_fence(bos[i]->gem.resv, fence);
+ else
+ reservation_object_add_shared_fence(bos[i]->gem.resv, fence);
+ }
+
+ lima_gem_unlock_bos(bos, submit->nr_bos, &ctx);
+
+ for (i = 0; i < submit->nr_bos; i++)
+ drm_gem_object_put_unlocked(&bos[i]->gem);
+
+ if (out_sync) {
+ drm_syncobj_replace_fence(out_sync, fence);
+ drm_syncobj_put(out_sync);
+ }
+
+ dma_fence_put(fence);
+
+ return 0;
+
+err_out2:
+ lima_sched_task_fini(submit->task);
+err_out1:
+ lima_gem_unlock_bos(bos, submit->nr_bos, &ctx);
+err_out0:
+ for (i = 0; i < submit->nr_bos; i++) {
+ if (!bos[i])
+ break;
+ lima_vm_bo_del(vm, bos[i]);
+ drm_gem_object_put_unlocked(&bos[i]->gem);
+ }
+ if (out_sync)
+ drm_syncobj_put(out_sync);
+ return err;
+}
+
+int lima_gem_wait(struct drm_file *file, u32 handle, u32 op, s64 timeout_ns)
+{
+ bool write = op & LIMA_GEM_WAIT_WRITE;
+ long ret, timeout;
+
+ if (!op)
+ return 0;
+
+ timeout = drm_timeout_abs_to_jiffies(timeout_ns);
+
+ ret = drm_gem_reservation_object_wait(file, handle, write, timeout);
+ if (ret == 0)
+ ret = timeout ? -ETIMEDOUT : -EBUSY;
+
+ return ret;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_GEM_H__
+#define __LIMA_GEM_H__
+
+struct lima_bo;
+struct lima_submit;
+
+extern const struct vm_operations_struct lima_gem_vm_ops;
+
+struct lima_bo *lima_gem_create_bo(struct drm_device *dev, u32 size, u32 flags);
+int lima_gem_create_handle(struct drm_device *dev, struct drm_file *file,
+ u32 size, u32 flags, u32 *handle);
+void lima_gem_free_object(struct drm_gem_object *obj);
+int lima_gem_object_open(struct drm_gem_object *obj, struct drm_file *file);
+void lima_gem_object_close(struct drm_gem_object *obj, struct drm_file *file);
+int lima_gem_get_info(struct drm_file *file, u32 handle, u32 *va, u64 *offset);
+int lima_gem_mmap(struct file *filp, struct vm_area_struct *vma);
+int lima_gem_submit(struct drm_file *file, struct lima_submit *submit);
+int lima_gem_wait(struct drm_file *file, u32 handle, u32 op, s64 timeout_ns);
+
+void lima_set_vma_flags(struct vm_area_struct *vma);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/dma-buf.h>
+#include <drm/drm_prime.h>
+#include <drm/drm_drv.h>
+#include <drm/drm_file.h>
+
+#include "lima_device.h"
+#include "lima_object.h"
+#include "lima_gem.h"
+#include "lima_gem_prime.h"
+
+struct drm_gem_object *lima_gem_prime_import_sg_table(
+ struct drm_device *dev, struct dma_buf_attachment *attach,
+ struct sg_table *sgt)
+{
+ struct lima_device *ldev = to_lima_dev(dev);
+ struct lima_bo *bo;
+
+ bo = lima_bo_create(ldev, attach->dmabuf->size, 0, sgt,
+ attach->dmabuf->resv);
+ if (IS_ERR(bo))
+ return ERR_CAST(bo);
+
+ return &bo->gem;
+}
+
+struct sg_table *lima_gem_prime_get_sg_table(struct drm_gem_object *obj)
+{
+ struct lima_bo *bo = to_lima_bo(obj);
+ int npages = obj->size >> PAGE_SHIFT;
+
+ return drm_prime_pages_to_sg(bo->pages, npages);
+}
+
+int lima_gem_prime_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma)
+{
+ int ret;
+
+ ret = drm_gem_mmap_obj(obj, obj->size, vma);
+ if (ret)
+ return ret;
+
+ lima_set_vma_flags(vma);
+ return 0;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_GEM_PRIME_H__
+#define __LIMA_GEM_PRIME_H__
+
+struct drm_gem_object *lima_gem_prime_import_sg_table(
+ struct drm_device *dev, struct dma_buf_attachment *attach,
+ struct sg_table *sgt);
+struct sg_table *lima_gem_prime_get_sg_table(struct drm_gem_object *obj);
+int lima_gem_prime_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/device.h>
+#include <linux/slab.h>
+
+#include <drm/lima_drm.h>
+
+#include "lima_device.h"
+#include "lima_gp.h"
+#include "lima_regs.h"
+
+#define gp_write(reg, data) writel(data, ip->iomem + reg)
+#define gp_read(reg) readl(ip->iomem + reg)
+
+static irqreturn_t lima_gp_irq_handler(int irq, void *data)
+{
+ struct lima_ip *ip = data;
+ struct lima_device *dev = ip->dev;
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_gp;
+ u32 state = gp_read(LIMA_GP_INT_STAT);
+ u32 status = gp_read(LIMA_GP_STATUS);
+ bool done = false;
+
+ /* for shared irq case */
+ if (!state)
+ return IRQ_NONE;
+
+ if (state & LIMA_GP_IRQ_MASK_ERROR) {
+ dev_err(dev->dev, "gp error irq state=%x status=%x\n",
+ state, status);
+
+ /* mask all interrupts before hard reset */
+ gp_write(LIMA_GP_INT_MASK, 0);
+
+ pipe->error = true;
+ done = true;
+ } else {
+ bool valid = state & (LIMA_GP_IRQ_VS_END_CMD_LST |
+ LIMA_GP_IRQ_PLBU_END_CMD_LST);
+ bool active = status & (LIMA_GP_STATUS_VS_ACTIVE |
+ LIMA_GP_STATUS_PLBU_ACTIVE);
+ done = valid && !active;
+ }
+
+ gp_write(LIMA_GP_INT_CLEAR, state);
+
+ if (done)
+ lima_sched_pipe_task_done(pipe);
+
+ return IRQ_HANDLED;
+}
+
+static void lima_gp_soft_reset_async(struct lima_ip *ip)
+{
+ if (ip->data.async_reset)
+ return;
+
+ gp_write(LIMA_GP_INT_MASK, 0);
+ gp_write(LIMA_GP_INT_CLEAR, LIMA_GP_IRQ_RESET_COMPLETED);
+ gp_write(LIMA_GP_CMD, LIMA_GP_CMD_SOFT_RESET);
+ ip->data.async_reset = true;
+}
+
+static int lima_gp_soft_reset_async_wait(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ int err;
+ u32 v;
+
+ if (!ip->data.async_reset)
+ return 0;
+
+ err = readl_poll_timeout(ip->iomem + LIMA_GP_INT_RAWSTAT, v,
+ v & LIMA_GP_IRQ_RESET_COMPLETED,
+ 0, 100);
+ if (err) {
+ dev_err(dev->dev, "gp soft reset time out\n");
+ return err;
+ }
+
+ gp_write(LIMA_GP_INT_CLEAR, LIMA_GP_IRQ_MASK_ALL);
+ gp_write(LIMA_GP_INT_MASK, LIMA_GP_IRQ_MASK_USED);
+
+ ip->data.async_reset = false;
+ return 0;
+}
+
+static int lima_gp_task_validate(struct lima_sched_pipe *pipe,
+ struct lima_sched_task *task)
+{
+ struct drm_lima_gp_frame *frame = task->frame;
+ u32 *f = frame->frame;
+ (void)pipe;
+
+ if (f[LIMA_GP_VSCL_START_ADDR >> 2] >
+ f[LIMA_GP_VSCL_END_ADDR >> 2] ||
+ f[LIMA_GP_PLBUCL_START_ADDR >> 2] >
+ f[LIMA_GP_PLBUCL_END_ADDR >> 2] ||
+ f[LIMA_GP_PLBU_ALLOC_START_ADDR >> 2] >
+ f[LIMA_GP_PLBU_ALLOC_END_ADDR >> 2])
+ return -EINVAL;
+
+ if (f[LIMA_GP_VSCL_START_ADDR >> 2] ==
+ f[LIMA_GP_VSCL_END_ADDR >> 2] &&
+ f[LIMA_GP_PLBUCL_START_ADDR >> 2] ==
+ f[LIMA_GP_PLBUCL_END_ADDR >> 2])
+ return -EINVAL;
+
+ return 0;
+}
+
+static void lima_gp_task_run(struct lima_sched_pipe *pipe,
+ struct lima_sched_task *task)
+{
+ struct lima_ip *ip = pipe->processor[0];
+ struct drm_lima_gp_frame *frame = task->frame;
+ u32 *f = frame->frame;
+ u32 cmd = 0;
+ int i;
+
+ if (f[LIMA_GP_VSCL_START_ADDR >> 2] !=
+ f[LIMA_GP_VSCL_END_ADDR >> 2])
+ cmd |= LIMA_GP_CMD_START_VS;
+ if (f[LIMA_GP_PLBUCL_START_ADDR >> 2] !=
+ f[LIMA_GP_PLBUCL_END_ADDR >> 2])
+ cmd |= LIMA_GP_CMD_START_PLBU;
+
+ /* before any hw ops, wait last success task async soft reset */
+ lima_gp_soft_reset_async_wait(ip);
+
+ for (i = 0; i < LIMA_GP_FRAME_REG_NUM; i++)
+ writel(f[i], ip->iomem + LIMA_GP_VSCL_START_ADDR + i * 4);
+
+ gp_write(LIMA_GP_CMD, LIMA_GP_CMD_UPDATE_PLBU_ALLOC);
+ gp_write(LIMA_GP_CMD, cmd);
+}
+
+static int lima_gp_hard_reset_poll(struct lima_ip *ip)
+{
+ gp_write(LIMA_GP_PERF_CNT_0_LIMIT, 0xC01A0000);
+ return gp_read(LIMA_GP_PERF_CNT_0_LIMIT) == 0xC01A0000;
+}
+
+static int lima_gp_hard_reset(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ int ret;
+
+ gp_write(LIMA_GP_PERF_CNT_0_LIMIT, 0xC0FFE000);
+ gp_write(LIMA_GP_INT_MASK, 0);
+ gp_write(LIMA_GP_CMD, LIMA_GP_CMD_RESET);
+ ret = lima_poll_timeout(ip, lima_gp_hard_reset_poll, 10, 100);
+ if (ret) {
+ dev_err(dev->dev, "gp hard reset timeout\n");
+ return ret;
+ }
+
+ gp_write(LIMA_GP_PERF_CNT_0_LIMIT, 0);
+ gp_write(LIMA_GP_INT_CLEAR, LIMA_GP_IRQ_MASK_ALL);
+ gp_write(LIMA_GP_INT_MASK, LIMA_GP_IRQ_MASK_USED);
+ return 0;
+}
+
+static void lima_gp_task_fini(struct lima_sched_pipe *pipe)
+{
+ lima_gp_soft_reset_async(pipe->processor[0]);
+}
+
+static void lima_gp_task_error(struct lima_sched_pipe *pipe)
+{
+ struct lima_ip *ip = pipe->processor[0];
+
+ dev_err(ip->dev->dev, "gp task error int_state=%x status=%x\n",
+ gp_read(LIMA_GP_INT_STAT), gp_read(LIMA_GP_STATUS));
+
+ lima_gp_hard_reset(ip);
+}
+
+static void lima_gp_task_mmu_error(struct lima_sched_pipe *pipe)
+{
+ lima_sched_pipe_task_done(pipe);
+}
+
+static void lima_gp_print_version(struct lima_ip *ip)
+{
+ u32 version, major, minor;
+ char *name;
+
+ version = gp_read(LIMA_GP_VERSION);
+ major = (version >> 8) & 0xFF;
+ minor = version & 0xFF;
+ switch (version >> 16) {
+ case 0xA07:
+ name = "mali200";
+ break;
+ case 0xC07:
+ name = "mali300";
+ break;
+ case 0xB07:
+ name = "mali400";
+ break;
+ case 0xD07:
+ name = "mali450";
+ break;
+ default:
+ name = "unknown";
+ break;
+ }
+ dev_info(ip->dev->dev, "%s - %s version major %d minor %d\n",
+ lima_ip_name(ip), name, major, minor);
+}
+
+static struct kmem_cache *lima_gp_task_slab;
+static int lima_gp_task_slab_refcnt;
+
+int lima_gp_init(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ int err;
+
+ lima_gp_print_version(ip);
+
+ ip->data.async_reset = false;
+ lima_gp_soft_reset_async(ip);
+ err = lima_gp_soft_reset_async_wait(ip);
+ if (err)
+ return err;
+
+ err = devm_request_irq(dev->dev, ip->irq, lima_gp_irq_handler,
+ IRQF_SHARED, lima_ip_name(ip), ip);
+ if (err) {
+ dev_err(dev->dev, "gp %s fail to request irq\n",
+ lima_ip_name(ip));
+ return err;
+ }
+
+ dev->gp_version = gp_read(LIMA_GP_VERSION);
+
+ return 0;
+}
+
+void lima_gp_fini(struct lima_ip *ip)
+{
+
+}
+
+int lima_gp_pipe_init(struct lima_device *dev)
+{
+ int frame_size = sizeof(struct drm_lima_gp_frame);
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_gp;
+
+ if (!lima_gp_task_slab) {
+ lima_gp_task_slab = kmem_cache_create_usercopy(
+ "lima_gp_task", sizeof(struct lima_sched_task) + frame_size,
+ 0, SLAB_HWCACHE_ALIGN, sizeof(struct lima_sched_task),
+ frame_size, NULL);
+ if (!lima_gp_task_slab)
+ return -ENOMEM;
+ }
+ lima_gp_task_slab_refcnt++;
+
+ pipe->frame_size = frame_size;
+ pipe->task_slab = lima_gp_task_slab;
+
+ pipe->task_validate = lima_gp_task_validate;
+ pipe->task_run = lima_gp_task_run;
+ pipe->task_fini = lima_gp_task_fini;
+ pipe->task_error = lima_gp_task_error;
+ pipe->task_mmu_error = lima_gp_task_mmu_error;
+
+ return 0;
+}
+
+void lima_gp_pipe_fini(struct lima_device *dev)
+{
+ if (!--lima_gp_task_slab_refcnt) {
+ kmem_cache_destroy(lima_gp_task_slab);
+ lima_gp_task_slab = NULL;
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_GP_H__
+#define __LIMA_GP_H__
+
+struct lima_ip;
+struct lima_device;
+
+int lima_gp_init(struct lima_ip *ip);
+void lima_gp_fini(struct lima_ip *ip);
+
+int lima_gp_pipe_init(struct lima_device *dev);
+void lima_gp_pipe_fini(struct lima_device *dev);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/iopoll.h>
+#include <linux/device.h>
+
+#include "lima_device.h"
+#include "lima_l2_cache.h"
+#include "lima_regs.h"
+
+#define l2_cache_write(reg, data) writel(data, ip->iomem + reg)
+#define l2_cache_read(reg) readl(ip->iomem + reg)
+
+static int lima_l2_cache_wait_idle(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ int err;
+ u32 v;
+
+ err = readl_poll_timeout(ip->iomem + LIMA_L2_CACHE_STATUS, v,
+ !(v & LIMA_L2_CACHE_STATUS_COMMAND_BUSY),
+ 0, 1000);
+ if (err) {
+ dev_err(dev->dev, "l2 cache wait command timeout\n");
+ return err;
+ }
+ return 0;
+}
+
+int lima_l2_cache_flush(struct lima_ip *ip)
+{
+ int ret;
+
+ spin_lock(&ip->data.lock);
+ l2_cache_write(LIMA_L2_CACHE_COMMAND, LIMA_L2_CACHE_COMMAND_CLEAR_ALL);
+ ret = lima_l2_cache_wait_idle(ip);
+ spin_unlock(&ip->data.lock);
+ return ret;
+}
+
+int lima_l2_cache_init(struct lima_ip *ip)
+{
+ int i, err;
+ u32 size;
+ struct lima_device *dev = ip->dev;
+
+ /* l2_cache2 only exists when one of PP4-7 present */
+ if (ip->id == lima_ip_l2_cache2) {
+ for (i = lima_ip_pp4; i <= lima_ip_pp7; i++) {
+ if (dev->ip[i].present)
+ break;
+ }
+ if (i > lima_ip_pp7)
+ return -ENODEV;
+ }
+
+ spin_lock_init(&ip->data.lock);
+
+ size = l2_cache_read(LIMA_L2_CACHE_SIZE);
+ dev_info(dev->dev, "l2 cache %uK, %u-way, %ubyte cache line, %ubit external bus\n",
+ 1 << (((size >> 16) & 0xff) - 10),
+ 1 << ((size >> 8) & 0xff),
+ 1 << (size & 0xff),
+ 1 << ((size >> 24) & 0xff));
+
+ err = lima_l2_cache_flush(ip);
+ if (err)
+ return err;
+
+ l2_cache_write(LIMA_L2_CACHE_ENABLE,
+ LIMA_L2_CACHE_ENABLE_ACCESS|LIMA_L2_CACHE_ENABLE_READ_ALLOCATE);
+ l2_cache_write(LIMA_L2_CACHE_MAX_READS, 0x1c);
+
+ return 0;
+}
+
+void lima_l2_cache_fini(struct lima_ip *ip)
+{
+
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_L2_CACHE_H__
+#define __LIMA_L2_CACHE_H__
+
+struct lima_ip;
+
+int lima_l2_cache_init(struct lima_ip *ip);
+void lima_l2_cache_fini(struct lima_ip *ip);
+
+int lima_l2_cache_flush(struct lima_ip *ip);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/device.h>
+
+#include "lima_device.h"
+#include "lima_mmu.h"
+#include "lima_vm.h"
+#include "lima_object.h"
+#include "lima_regs.h"
+
+#define mmu_write(reg, data) writel(data, ip->iomem + reg)
+#define mmu_read(reg) readl(ip->iomem + reg)
+
+#define lima_mmu_send_command(cmd, addr, val, cond) \
+({ \
+ int __ret; \
+ \
+ mmu_write(LIMA_MMU_COMMAND, cmd); \
+ __ret = readl_poll_timeout(ip->iomem + (addr), val, \
+ cond, 0, 100); \
+ if (__ret) \
+ dev_err(dev->dev, \
+ "mmu command %x timeout\n", cmd); \
+ __ret; \
+})
+
+static irqreturn_t lima_mmu_irq_handler(int irq, void *data)
+{
+ struct lima_ip *ip = data;
+ struct lima_device *dev = ip->dev;
+ u32 status = mmu_read(LIMA_MMU_INT_STATUS);
+ struct lima_sched_pipe *pipe;
+
+ /* for shared irq case */
+ if (!status)
+ return IRQ_NONE;
+
+ if (status & LIMA_MMU_INT_PAGE_FAULT) {
+ u32 fault = mmu_read(LIMA_MMU_PAGE_FAULT_ADDR);
+
+ dev_err(dev->dev, "mmu page fault at 0x%x from bus id %d of type %s on %s\n",
+ fault, LIMA_MMU_STATUS_BUS_ID(status),
+ status & LIMA_MMU_STATUS_PAGE_FAULT_IS_WRITE ? "write" : "read",
+ lima_ip_name(ip));
+ }
+
+ if (status & LIMA_MMU_INT_READ_BUS_ERROR)
+ dev_err(dev->dev, "mmu %s irq bus error\n", lima_ip_name(ip));
+
+ /* mask all interrupts before resume */
+ mmu_write(LIMA_MMU_INT_MASK, 0);
+ mmu_write(LIMA_MMU_INT_CLEAR, status);
+
+ pipe = dev->pipe + (ip->id == lima_ip_gpmmu ? lima_pipe_gp : lima_pipe_pp);
+ lima_sched_pipe_mmu_error(pipe);
+
+ return IRQ_HANDLED;
+}
+
+int lima_mmu_init(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ int err;
+ u32 v;
+
+ if (ip->id == lima_ip_ppmmu_bcast)
+ return 0;
+
+ mmu_write(LIMA_MMU_DTE_ADDR, 0xCAFEBABE);
+ if (mmu_read(LIMA_MMU_DTE_ADDR) != 0xCAFEB000) {
+ dev_err(dev->dev, "mmu %s dte write test fail\n", lima_ip_name(ip));
+ return -EIO;
+ }
+
+ mmu_write(LIMA_MMU_COMMAND, LIMA_MMU_COMMAND_HARD_RESET);
+ err = lima_mmu_send_command(LIMA_MMU_COMMAND_HARD_RESET,
+ LIMA_MMU_DTE_ADDR, v, v == 0);
+ if (err)
+ return err;
+
+ err = devm_request_irq(dev->dev, ip->irq, lima_mmu_irq_handler,
+ IRQF_SHARED, lima_ip_name(ip), ip);
+ if (err) {
+ dev_err(dev->dev, "mmu %s fail to request irq\n", lima_ip_name(ip));
+ return err;
+ }
+
+ mmu_write(LIMA_MMU_INT_MASK, LIMA_MMU_INT_PAGE_FAULT | LIMA_MMU_INT_READ_BUS_ERROR);
+ mmu_write(LIMA_MMU_DTE_ADDR, dev->empty_vm->pd.dma);
+ return lima_mmu_send_command(LIMA_MMU_COMMAND_ENABLE_PAGING,
+ LIMA_MMU_STATUS, v,
+ v & LIMA_MMU_STATUS_PAGING_ENABLED);
+}
+
+void lima_mmu_fini(struct lima_ip *ip)
+{
+
+}
+
+void lima_mmu_switch_vm(struct lima_ip *ip, struct lima_vm *vm)
+{
+ struct lima_device *dev = ip->dev;
+ u32 v;
+
+ lima_mmu_send_command(LIMA_MMU_COMMAND_ENABLE_STALL,
+ LIMA_MMU_STATUS, v,
+ v & LIMA_MMU_STATUS_STALL_ACTIVE);
+
+ if (vm)
+ mmu_write(LIMA_MMU_DTE_ADDR, vm->pd.dma);
+
+ /* flush the TLB */
+ mmu_write(LIMA_MMU_COMMAND, LIMA_MMU_COMMAND_ZAP_CACHE);
+
+ lima_mmu_send_command(LIMA_MMU_COMMAND_DISABLE_STALL,
+ LIMA_MMU_STATUS, v,
+ !(v & LIMA_MMU_STATUS_STALL_ACTIVE));
+}
+
+void lima_mmu_page_fault_resume(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ u32 status = mmu_read(LIMA_MMU_STATUS);
+ u32 v;
+
+ if (status & LIMA_MMU_STATUS_PAGE_FAULT_ACTIVE) {
+ dev_info(dev->dev, "mmu resume\n");
+
+ mmu_write(LIMA_MMU_INT_MASK, 0);
+ mmu_write(LIMA_MMU_DTE_ADDR, 0xCAFEBABE);
+ lima_mmu_send_command(LIMA_MMU_COMMAND_HARD_RESET,
+ LIMA_MMU_DTE_ADDR, v, v == 0);
+ mmu_write(LIMA_MMU_INT_MASK, LIMA_MMU_INT_PAGE_FAULT | LIMA_MMU_INT_READ_BUS_ERROR);
+ mmu_write(LIMA_MMU_DTE_ADDR, dev->empty_vm->pd.dma);
+ lima_mmu_send_command(LIMA_MMU_COMMAND_ENABLE_PAGING,
+ LIMA_MMU_STATUS, v,
+ v & LIMA_MMU_STATUS_PAGING_ENABLED);
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_MMU_H__
+#define __LIMA_MMU_H__
+
+struct lima_ip;
+struct lima_vm;
+
+int lima_mmu_init(struct lima_ip *ip);
+void lima_mmu_fini(struct lima_ip *ip);
+
+void lima_mmu_switch_vm(struct lima_ip *ip, struct lima_vm *vm);
+void lima_mmu_page_fault_resume(struct lima_ip *ip);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <drm/drm_prime.h>
+#include <linux/pagemap.h>
+#include <linux/dma-mapping.h>
+
+#include "lima_object.h"
+
+void lima_bo_destroy(struct lima_bo *bo)
+{
+ if (bo->sgt) {
+ kfree(bo->pages);
+ drm_prime_gem_destroy(&bo->gem, bo->sgt);
+ } else {
+ if (bo->pages_dma_addr) {
+ int i, npages = bo->gem.size >> PAGE_SHIFT;
+
+ for (i = 0; i < npages; i++) {
+ if (bo->pages_dma_addr[i])
+ dma_unmap_page(bo->gem.dev->dev,
+ bo->pages_dma_addr[i],
+ PAGE_SIZE, DMA_BIDIRECTIONAL);
+ }
+ }
+
+ if (bo->pages)
+ drm_gem_put_pages(&bo->gem, bo->pages, true, true);
+ }
+
+ kfree(bo->pages_dma_addr);
+ drm_gem_object_release(&bo->gem);
+ kfree(bo);
+}
+
+static struct lima_bo *lima_bo_create_struct(struct lima_device *dev, u32 size, u32 flags,
+ struct reservation_object *resv)
+{
+ struct lima_bo *bo;
+ int err;
+
+ size = PAGE_ALIGN(size);
+
+ bo = kzalloc(sizeof(*bo), GFP_KERNEL);
+ if (!bo)
+ return ERR_PTR(-ENOMEM);
+
+ mutex_init(&bo->lock);
+ INIT_LIST_HEAD(&bo->va);
+ bo->gem.resv = resv;
+
+ err = drm_gem_object_init(dev->ddev, &bo->gem, size);
+ if (err) {
+ kfree(bo);
+ return ERR_PTR(err);
+ }
+
+ return bo;
+}
+
+struct lima_bo *lima_bo_create(struct lima_device *dev, u32 size,
+ u32 flags, struct sg_table *sgt,
+ struct reservation_object *resv)
+{
+ int i, err;
+ size_t npages;
+ struct lima_bo *bo, *ret;
+
+ bo = lima_bo_create_struct(dev, size, flags, resv);
+ if (IS_ERR(bo))
+ return bo;
+
+ npages = bo->gem.size >> PAGE_SHIFT;
+
+ bo->pages_dma_addr = kcalloc(npages, sizeof(dma_addr_t), GFP_KERNEL);
+ if (!bo->pages_dma_addr) {
+ ret = ERR_PTR(-ENOMEM);
+ goto err_out;
+ }
+
+ if (sgt) {
+ bo->sgt = sgt;
+
+ bo->pages = kcalloc(npages, sizeof(*bo->pages), GFP_KERNEL);
+ if (!bo->pages) {
+ ret = ERR_PTR(-ENOMEM);
+ goto err_out;
+ }
+
+ err = drm_prime_sg_to_page_addr_arrays(
+ sgt, bo->pages, bo->pages_dma_addr, npages);
+ if (err) {
+ ret = ERR_PTR(err);
+ goto err_out;
+ }
+ } else {
+ mapping_set_gfp_mask(bo->gem.filp->f_mapping, GFP_DMA32);
+ bo->pages = drm_gem_get_pages(&bo->gem);
+ if (IS_ERR(bo->pages)) {
+ ret = ERR_CAST(bo->pages);
+ bo->pages = NULL;
+ goto err_out;
+ }
+
+ for (i = 0; i < npages; i++) {
+ dma_addr_t addr = dma_map_page(dev->dev, bo->pages[i], 0,
+ PAGE_SIZE, DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(dev->dev, addr)) {
+ ret = ERR_PTR(-EFAULT);
+ goto err_out;
+ }
+ bo->pages_dma_addr[i] = addr;
+ }
+
+ }
+
+ return bo;
+
+err_out:
+ lima_bo_destroy(bo);
+ return ret;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2018-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_OBJECT_H__
+#define __LIMA_OBJECT_H__
+
+#include <drm/drm_gem.h>
+
+#include "lima_device.h"
+
+struct lima_bo {
+ struct drm_gem_object gem;
+
+ struct page **pages;
+ dma_addr_t *pages_dma_addr;
+ struct sg_table *sgt;
+ void *vaddr;
+
+ struct mutex lock;
+ struct list_head va;
+};
+
+static inline struct lima_bo *
+to_lima_bo(struct drm_gem_object *obj)
+{
+ return container_of(obj, struct lima_bo, gem);
+}
+
+struct lima_bo *lima_bo_create(struct lima_device *dev, u32 size,
+ u32 flags, struct sg_table *sgt,
+ struct reservation_object *resv);
+void lima_bo_destroy(struct lima_bo *bo);
+void *lima_bo_vmap(struct lima_bo *bo);
+void lima_bo_vunmap(struct lima_bo *bo);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/iopoll.h>
+#include <linux/device.h>
+
+#include "lima_device.h"
+#include "lima_pmu.h"
+#include "lima_regs.h"
+
+#define pmu_write(reg, data) writel(data, ip->iomem + reg)
+#define pmu_read(reg) readl(ip->iomem + reg)
+
+static int lima_pmu_wait_cmd(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ int err;
+ u32 v;
+
+ err = readl_poll_timeout(ip->iomem + LIMA_PMU_INT_RAWSTAT,
+ v, v & LIMA_PMU_INT_CMD_MASK,
+ 100, 100000);
+ if (err) {
+ dev_err(dev->dev, "timeout wait pmd cmd\n");
+ return err;
+ }
+
+ pmu_write(LIMA_PMU_INT_CLEAR, LIMA_PMU_INT_CMD_MASK);
+ return 0;
+}
+
+int lima_pmu_init(struct lima_ip *ip)
+{
+ int err;
+ u32 stat;
+
+ pmu_write(LIMA_PMU_INT_MASK, 0);
+
+ /* If this value is too low, when in high GPU clk freq,
+ * GPU will be in unstable state.
+ */
+ pmu_write(LIMA_PMU_SW_DELAY, 0xffff);
+
+ /* status reg 1=off 0=on */
+ stat = pmu_read(LIMA_PMU_STATUS);
+
+ /* power up all ip */
+ if (stat) {
+ pmu_write(LIMA_PMU_POWER_UP, stat);
+ err = lima_pmu_wait_cmd(ip);
+ if (err)
+ return err;
+ }
+ return 0;
+}
+
+void lima_pmu_fini(struct lima_ip *ip)
+{
+
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_PMU_H__
+#define __LIMA_PMU_H__
+
+struct lima_ip;
+
+int lima_pmu_init(struct lima_ip *ip);
+void lima_pmu_fini(struct lima_ip *ip);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/device.h>
+#include <linux/slab.h>
+
+#include <drm/lima_drm.h>
+
+#include "lima_device.h"
+#include "lima_pp.h"
+#include "lima_dlbu.h"
+#include "lima_bcast.h"
+#include "lima_vm.h"
+#include "lima_regs.h"
+
+#define pp_write(reg, data) writel(data, ip->iomem + reg)
+#define pp_read(reg) readl(ip->iomem + reg)
+
+static void lima_pp_handle_irq(struct lima_ip *ip, u32 state)
+{
+ struct lima_device *dev = ip->dev;
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_pp;
+
+ if (state & LIMA_PP_IRQ_MASK_ERROR) {
+ u32 status = pp_read(LIMA_PP_STATUS);
+
+ dev_err(dev->dev, "pp error irq state=%x status=%x\n",
+ state, status);
+
+ pipe->error = true;
+
+ /* mask all interrupts before hard reset */
+ pp_write(LIMA_PP_INT_MASK, 0);
+ }
+
+ pp_write(LIMA_PP_INT_CLEAR, state);
+}
+
+static irqreturn_t lima_pp_irq_handler(int irq, void *data)
+{
+ struct lima_ip *ip = data;
+ struct lima_device *dev = ip->dev;
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_pp;
+ u32 state = pp_read(LIMA_PP_INT_STATUS);
+
+ /* for shared irq case */
+ if (!state)
+ return IRQ_NONE;
+
+ lima_pp_handle_irq(ip, state);
+
+ if (atomic_dec_and_test(&pipe->task))
+ lima_sched_pipe_task_done(pipe);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t lima_pp_bcast_irq_handler(int irq, void *data)
+{
+ int i;
+ irqreturn_t ret = IRQ_NONE;
+ struct lima_ip *pp_bcast = data;
+ struct lima_device *dev = pp_bcast->dev;
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_pp;
+ struct drm_lima_m450_pp_frame *frame = pipe->current_task->frame;
+
+ for (i = 0; i < frame->num_pp; i++) {
+ struct lima_ip *ip = pipe->processor[i];
+ u32 status, state;
+
+ if (pipe->done & (1 << i))
+ continue;
+
+ /* status read first in case int state change in the middle
+ * which may miss the interrupt handling
+ */
+ status = pp_read(LIMA_PP_STATUS);
+ state = pp_read(LIMA_PP_INT_STATUS);
+
+ if (state) {
+ lima_pp_handle_irq(ip, state);
+ ret = IRQ_HANDLED;
+ } else {
+ if (status & LIMA_PP_STATUS_RENDERING_ACTIVE)
+ continue;
+ }
+
+ pipe->done |= (1 << i);
+ if (atomic_dec_and_test(&pipe->task))
+ lima_sched_pipe_task_done(pipe);
+ }
+
+ return ret;
+}
+
+static void lima_pp_soft_reset_async(struct lima_ip *ip)
+{
+ if (ip->data.async_reset)
+ return;
+
+ pp_write(LIMA_PP_INT_MASK, 0);
+ pp_write(LIMA_PP_INT_RAWSTAT, LIMA_PP_IRQ_MASK_ALL);
+ pp_write(LIMA_PP_CTRL, LIMA_PP_CTRL_SOFT_RESET);
+ ip->data.async_reset = true;
+}
+
+static int lima_pp_soft_reset_poll(struct lima_ip *ip)
+{
+ return !(pp_read(LIMA_PP_STATUS) & LIMA_PP_STATUS_RENDERING_ACTIVE) &&
+ pp_read(LIMA_PP_INT_RAWSTAT) == LIMA_PP_IRQ_RESET_COMPLETED;
+}
+
+static int lima_pp_soft_reset_async_wait_one(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ int ret;
+
+ ret = lima_poll_timeout(ip, lima_pp_soft_reset_poll, 0, 100);
+ if (ret) {
+ dev_err(dev->dev, "pp %s reset time out\n", lima_ip_name(ip));
+ return ret;
+ }
+
+ pp_write(LIMA_PP_INT_CLEAR, LIMA_PP_IRQ_MASK_ALL);
+ pp_write(LIMA_PP_INT_MASK, LIMA_PP_IRQ_MASK_USED);
+ return 0;
+}
+
+static int lima_pp_soft_reset_async_wait(struct lima_ip *ip)
+{
+ int i, err = 0;
+
+ if (!ip->data.async_reset)
+ return 0;
+
+ if (ip->id == lima_ip_pp_bcast) {
+ struct lima_device *dev = ip->dev;
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_pp;
+ struct drm_lima_m450_pp_frame *frame = pipe->current_task->frame;
+
+ for (i = 0; i < frame->num_pp; i++)
+ err |= lima_pp_soft_reset_async_wait_one(pipe->processor[i]);
+ } else
+ err = lima_pp_soft_reset_async_wait_one(ip);
+
+ ip->data.async_reset = false;
+ return err;
+}
+
+static void lima_pp_write_frame(struct lima_ip *ip, u32 *frame, u32 *wb)
+{
+ int i, j, n = 0;
+
+ for (i = 0; i < LIMA_PP_FRAME_REG_NUM; i++)
+ writel(frame[i], ip->iomem + LIMA_PP_FRAME + i * 4);
+
+ for (i = 0; i < 3; i++) {
+ for (j = 0; j < LIMA_PP_WB_REG_NUM; j++)
+ writel(wb[n++], ip->iomem + LIMA_PP_WB(i) + j * 4);
+ }
+}
+
+static int lima_pp_hard_reset_poll(struct lima_ip *ip)
+{
+ pp_write(LIMA_PP_PERF_CNT_0_LIMIT, 0xC01A0000);
+ return pp_read(LIMA_PP_PERF_CNT_0_LIMIT) == 0xC01A0000;
+}
+
+static int lima_pp_hard_reset(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ int ret;
+
+ pp_write(LIMA_PP_PERF_CNT_0_LIMIT, 0xC0FFE000);
+ pp_write(LIMA_PP_INT_MASK, 0);
+ pp_write(LIMA_PP_CTRL, LIMA_PP_CTRL_FORCE_RESET);
+ ret = lima_poll_timeout(ip, lima_pp_hard_reset_poll, 10, 100);
+ if (ret) {
+ dev_err(dev->dev, "pp hard reset timeout\n");
+ return ret;
+ }
+
+ pp_write(LIMA_PP_PERF_CNT_0_LIMIT, 0);
+ pp_write(LIMA_PP_INT_CLEAR, LIMA_PP_IRQ_MASK_ALL);
+ pp_write(LIMA_PP_INT_MASK, LIMA_PP_IRQ_MASK_USED);
+ return 0;
+}
+
+static void lima_pp_print_version(struct lima_ip *ip)
+{
+ u32 version, major, minor;
+ char *name;
+
+ version = pp_read(LIMA_PP_VERSION);
+ major = (version >> 8) & 0xFF;
+ minor = version & 0xFF;
+ switch (version >> 16) {
+ case 0xC807:
+ name = "mali200";
+ break;
+ case 0xCE07:
+ name = "mali300";
+ break;
+ case 0xCD07:
+ name = "mali400";
+ break;
+ case 0xCF07:
+ name = "mali450";
+ break;
+ default:
+ name = "unknown";
+ break;
+ }
+ dev_info(ip->dev->dev, "%s - %s version major %d minor %d\n",
+ lima_ip_name(ip), name, major, minor);
+}
+
+int lima_pp_init(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ int err;
+
+ lima_pp_print_version(ip);
+
+ ip->data.async_reset = false;
+ lima_pp_soft_reset_async(ip);
+ err = lima_pp_soft_reset_async_wait(ip);
+ if (err)
+ return err;
+
+ err = devm_request_irq(dev->dev, ip->irq, lima_pp_irq_handler,
+ IRQF_SHARED, lima_ip_name(ip), ip);
+ if (err) {
+ dev_err(dev->dev, "pp %s fail to request irq\n",
+ lima_ip_name(ip));
+ return err;
+ }
+
+ dev->pp_version = pp_read(LIMA_PP_VERSION);
+
+ return 0;
+}
+
+void lima_pp_fini(struct lima_ip *ip)
+{
+
+}
+
+int lima_pp_bcast_init(struct lima_ip *ip)
+{
+ struct lima_device *dev = ip->dev;
+ int err;
+
+ err = devm_request_irq(dev->dev, ip->irq, lima_pp_bcast_irq_handler,
+ IRQF_SHARED, lima_ip_name(ip), ip);
+ if (err) {
+ dev_err(dev->dev, "pp %s fail to request irq\n",
+ lima_ip_name(ip));
+ return err;
+ }
+
+ return 0;
+}
+
+void lima_pp_bcast_fini(struct lima_ip *ip)
+{
+
+}
+
+static int lima_pp_task_validate(struct lima_sched_pipe *pipe,
+ struct lima_sched_task *task)
+{
+ u32 num_pp;
+
+ if (pipe->bcast_processor) {
+ struct drm_lima_m450_pp_frame *f = task->frame;
+
+ num_pp = f->num_pp;
+
+ if (f->_pad)
+ return -EINVAL;
+ } else {
+ struct drm_lima_m400_pp_frame *f = task->frame;
+
+ num_pp = f->num_pp;
+ }
+
+ if (num_pp == 0 || num_pp > pipe->num_processor)
+ return -EINVAL;
+
+ return 0;
+}
+
+static void lima_pp_task_run(struct lima_sched_pipe *pipe,
+ struct lima_sched_task *task)
+{
+ if (pipe->bcast_processor) {
+ struct drm_lima_m450_pp_frame *frame = task->frame;
+ struct lima_device *dev = pipe->bcast_processor->dev;
+ struct lima_ip *ip = pipe->bcast_processor;
+ int i;
+
+ pipe->done = 0;
+ atomic_set(&pipe->task, frame->num_pp);
+
+ if (frame->use_dlbu) {
+ lima_dlbu_enable(dev, frame->num_pp);
+
+ frame->frame[LIMA_PP_FRAME >> 2] = LIMA_VA_RESERVE_DLBU;
+ lima_dlbu_set_reg(dev->ip + lima_ip_dlbu, frame->dlbu_regs);
+ } else
+ lima_dlbu_disable(dev);
+
+ lima_bcast_enable(dev, frame->num_pp);
+
+ lima_pp_soft_reset_async_wait(ip);
+
+ lima_pp_write_frame(ip, frame->frame, frame->wb);
+
+ for (i = 0; i < frame->num_pp; i++) {
+ struct lima_ip *ip = pipe->processor[i];
+
+ pp_write(LIMA_PP_STACK, frame->fragment_stack_address[i]);
+ if (!frame->use_dlbu)
+ pp_write(LIMA_PP_FRAME, frame->plbu_array_address[i]);
+ }
+
+ pp_write(LIMA_PP_CTRL, LIMA_PP_CTRL_START_RENDERING);
+ } else {
+ struct drm_lima_m400_pp_frame *frame = task->frame;
+ int i;
+
+ atomic_set(&pipe->task, frame->num_pp);
+
+ for (i = 0; i < frame->num_pp; i++) {
+ struct lima_ip *ip = pipe->processor[i];
+
+ frame->frame[LIMA_PP_FRAME >> 2] =
+ frame->plbu_array_address[i];
+ frame->frame[LIMA_PP_STACK >> 2] =
+ frame->fragment_stack_address[i];
+
+ lima_pp_soft_reset_async_wait(ip);
+
+ lima_pp_write_frame(ip, frame->frame, frame->wb);
+
+ pp_write(LIMA_PP_CTRL, LIMA_PP_CTRL_START_RENDERING);
+ }
+ }
+}
+
+static void lima_pp_task_fini(struct lima_sched_pipe *pipe)
+{
+ if (pipe->bcast_processor)
+ lima_pp_soft_reset_async(pipe->bcast_processor);
+ else {
+ int i;
+
+ for (i = 0; i < pipe->num_processor; i++)
+ lima_pp_soft_reset_async(pipe->processor[i]);
+ }
+}
+
+static void lima_pp_task_error(struct lima_sched_pipe *pipe)
+{
+ int i;
+
+ for (i = 0; i < pipe->num_processor; i++) {
+ struct lima_ip *ip = pipe->processor[i];
+
+ dev_err(ip->dev->dev, "pp task error %d int_state=%x status=%x\n",
+ i, pp_read(LIMA_PP_INT_STATUS), pp_read(LIMA_PP_STATUS));
+
+ lima_pp_hard_reset(ip);
+ }
+}
+
+static void lima_pp_task_mmu_error(struct lima_sched_pipe *pipe)
+{
+ if (atomic_dec_and_test(&pipe->task))
+ lima_sched_pipe_task_done(pipe);
+}
+
+static struct kmem_cache *lima_pp_task_slab;
+static int lima_pp_task_slab_refcnt;
+
+int lima_pp_pipe_init(struct lima_device *dev)
+{
+ int frame_size;
+ struct lima_sched_pipe *pipe = dev->pipe + lima_pipe_pp;
+
+ if (dev->id == lima_gpu_mali400)
+ frame_size = sizeof(struct drm_lima_m400_pp_frame);
+ else
+ frame_size = sizeof(struct drm_lima_m450_pp_frame);
+
+ if (!lima_pp_task_slab) {
+ lima_pp_task_slab = kmem_cache_create_usercopy(
+ "lima_pp_task", sizeof(struct lima_sched_task) + frame_size,
+ 0, SLAB_HWCACHE_ALIGN, sizeof(struct lima_sched_task),
+ frame_size, NULL);
+ if (!lima_pp_task_slab)
+ return -ENOMEM;
+ }
+ lima_pp_task_slab_refcnt++;
+
+ pipe->frame_size = frame_size;
+ pipe->task_slab = lima_pp_task_slab;
+
+ pipe->task_validate = lima_pp_task_validate;
+ pipe->task_run = lima_pp_task_run;
+ pipe->task_fini = lima_pp_task_fini;
+ pipe->task_error = lima_pp_task_error;
+ pipe->task_mmu_error = lima_pp_task_mmu_error;
+
+ return 0;
+}
+
+void lima_pp_pipe_fini(struct lima_device *dev)
+{
+ if (!--lima_pp_task_slab_refcnt) {
+ kmem_cache_destroy(lima_pp_task_slab);
+ lima_pp_task_slab = NULL;
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_PP_H__
+#define __LIMA_PP_H__
+
+struct lima_ip;
+struct lima_device;
+
+int lima_pp_init(struct lima_ip *ip);
+void lima_pp_fini(struct lima_ip *ip);
+
+int lima_pp_bcast_init(struct lima_ip *ip);
+void lima_pp_bcast_fini(struct lima_ip *ip);
+
+int lima_pp_pipe_init(struct lima_device *dev);
+void lima_pp_pipe_fini(struct lima_device *dev);
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2010-2017 ARM Limited. All rights reserved.
+ * Copyright 2017-2019 Qiang Yu <yuq825@gmail.com>
+ */
+
+#ifndef __LIMA_REGS_H__
+#define __LIMA_REGS_H__
+
+/* This file's register definition is collected from the
+ * official ARM Mali Utgard GPU kernel driver source code
+ */
+
+/* PMU regs */
+#define LIMA_PMU_POWER_UP 0x00
+#define LIMA_PMU_POWER_DOWN 0x04
+#define LIMA_PMU_POWER_GP0_MASK BIT(0)
+#define LIMA_PMU_POWER_L2_MASK BIT(1)
+#define LIMA_PMU_POWER_PP_MASK(i) BIT(2 + i)
+
+/*
+ * On Mali450 each block automatically starts up its corresponding L2
+ * and the PPs are not fully independent controllable.
+ * Instead PP0, PP1-3 and PP4-7 can be turned on or off.
+ */
+#define LIMA450_PMU_POWER_PP0_MASK BIT(1)
+#define LIMA450_PMU_POWER_PP13_MASK BIT(2)
+#define LIMA450_PMU_POWER_PP47_MASK BIT(3)
+
+#define LIMA_PMU_STATUS 0x08
+#define LIMA_PMU_INT_MASK 0x0C
+#define LIMA_PMU_INT_RAWSTAT 0x10
+#define LIMA_PMU_INT_CLEAR 0x18
+#define LIMA_PMU_INT_CMD_MASK BIT(0)
+#define LIMA_PMU_SW_DELAY 0x1C
+
+/* L2 cache regs */
+#define LIMA_L2_CACHE_SIZE 0x0004
+#define LIMA_L2_CACHE_STATUS 0x0008
+#define LIMA_L2_CACHE_STATUS_COMMAND_BUSY BIT(0)
+#define LIMA_L2_CACHE_STATUS_DATA_BUSY BIT(1)
+#define LIMA_L2_CACHE_COMMAND 0x0010
+#define LIMA_L2_CACHE_COMMAND_CLEAR_ALL BIT(0)
+#define LIMA_L2_CACHE_CLEAR_PAGE 0x0014
+#define LIMA_L2_CACHE_MAX_READS 0x0018
+#define LIMA_L2_CACHE_ENABLE 0x001C
+#define LIMA_L2_CACHE_ENABLE_ACCESS BIT(0)
+#define LIMA_L2_CACHE_ENABLE_READ_ALLOCATE BIT(1)
+#define LIMA_L2_CACHE_PERFCNT_SRC0 0x0020
+#define LIMA_L2_CACHE_PERFCNT_VAL0 0x0024
+#define LIMA_L2_CACHE_PERFCNT_SRC1 0x0028
+#define LIMA_L2_CACHE_ERFCNT_VAL1 0x002C
+
+/* GP regs */
+#define LIMA_GP_VSCL_START_ADDR 0x00
+#define LIMA_GP_VSCL_END_ADDR 0x04
+#define LIMA_GP_PLBUCL_START_ADDR 0x08
+#define LIMA_GP_PLBUCL_END_ADDR 0x0c
+#define LIMA_GP_PLBU_ALLOC_START_ADDR 0x10
+#define LIMA_GP_PLBU_ALLOC_END_ADDR 0x14
+#define LIMA_GP_CMD 0x20
+#define LIMA_GP_CMD_START_VS BIT(0)
+#define LIMA_GP_CMD_START_PLBU BIT(1)
+#define LIMA_GP_CMD_UPDATE_PLBU_ALLOC BIT(4)
+#define LIMA_GP_CMD_RESET BIT(5)
+#define LIMA_GP_CMD_FORCE_HANG BIT(6)
+#define LIMA_GP_CMD_STOP_BUS BIT(9)
+#define LIMA_GP_CMD_SOFT_RESET BIT(10)
+#define LIMA_GP_INT_RAWSTAT 0x24
+#define LIMA_GP_INT_CLEAR 0x28
+#define LIMA_GP_INT_MASK 0x2C
+#define LIMA_GP_INT_STAT 0x30
+#define LIMA_GP_IRQ_VS_END_CMD_LST BIT(0)
+#define LIMA_GP_IRQ_PLBU_END_CMD_LST BIT(1)
+#define LIMA_GP_IRQ_PLBU_OUT_OF_MEM BIT(2)
+#define LIMA_GP_IRQ_VS_SEM_IRQ BIT(3)
+#define LIMA_GP_IRQ_PLBU_SEM_IRQ BIT(4)
+#define LIMA_GP_IRQ_HANG BIT(5)
+#define LIMA_GP_IRQ_FORCE_HANG BIT(6)
+#define LIMA_GP_IRQ_PERF_CNT_0_LIMIT BIT(7)
+#define LIMA_GP_IRQ_PERF_CNT_1_LIMIT BIT(8)
+#define LIMA_GP_IRQ_WRITE_BOUND_ERR BIT(9)
+#define LIMA_GP_IRQ_SYNC_ERROR BIT(10)
+#define LIMA_GP_IRQ_AXI_BUS_ERROR BIT(11)
+#define LIMA_GP_IRQ_AXI_BUS_STOPPED BIT(12)
+#define LIMA_GP_IRQ_VS_INVALID_CMD BIT(13)
+#define LIMA_GP_IRQ_PLB_INVALID_CMD BIT(14)
+#define LIMA_GP_IRQ_RESET_COMPLETED BIT(19)
+#define LIMA_GP_IRQ_SEMAPHORE_UNDERFLOW BIT(20)
+#define LIMA_GP_IRQ_SEMAPHORE_OVERFLOW BIT(21)
+#define LIMA_GP_IRQ_PTR_ARRAY_OUT_OF_BOUNDS BIT(22)
+#define LIMA_GP_WRITE_BOUND_LOW 0x34
+#define LIMA_GP_PERF_CNT_0_ENABLE 0x3C
+#define LIMA_GP_PERF_CNT_1_ENABLE 0x40
+#define LIMA_GP_PERF_CNT_0_SRC 0x44
+#define LIMA_GP_PERF_CNT_1_SRC 0x48
+#define LIMA_GP_PERF_CNT_0_VALUE 0x4C
+#define LIMA_GP_PERF_CNT_1_VALUE 0x50
+#define LIMA_GP_PERF_CNT_0_LIMIT 0x54
+#define LIMA_GP_STATUS 0x68
+#define LIMA_GP_STATUS_VS_ACTIVE BIT(1)
+#define LIMA_GP_STATUS_BUS_STOPPED BIT(2)
+#define LIMA_GP_STATUS_PLBU_ACTIVE BIT(3)
+#define LIMA_GP_STATUS_BUS_ERROR BIT(6)
+#define LIMA_GP_STATUS_WRITE_BOUND_ERR BIT(8)
+#define LIMA_GP_VERSION 0x6C
+#define LIMA_GP_VSCL_START_ADDR_READ 0x80
+#define LIMA_GP_PLBCL_START_ADDR_READ 0x84
+#define LIMA_GP_CONTR_AXI_BUS_ERROR_STAT 0x94
+
+#define LIMA_GP_IRQ_MASK_ALL \
+ ( \
+ LIMA_GP_IRQ_VS_END_CMD_LST | \
+ LIMA_GP_IRQ_PLBU_END_CMD_LST | \
+ LIMA_GP_IRQ_PLBU_OUT_OF_MEM | \
+ LIMA_GP_IRQ_VS_SEM_IRQ | \
+ LIMA_GP_IRQ_PLBU_SEM_IRQ | \
+ LIMA_GP_IRQ_HANG | \
+ LIMA_GP_IRQ_FORCE_HANG | \
+ LIMA_GP_IRQ_PERF_CNT_0_LIMIT | \
+ LIMA_GP_IRQ_PERF_CNT_1_LIMIT | \
+ LIMA_GP_IRQ_WRITE_BOUND_ERR | \
+ LIMA_GP_IRQ_SYNC_ERROR | \
+ LIMA_GP_IRQ_AXI_BUS_ERROR | \
+ LIMA_GP_IRQ_AXI_BUS_STOPPED | \
+ LIMA_GP_IRQ_VS_INVALID_CMD | \
+ LIMA_GP_IRQ_PLB_INVALID_CMD | \
+ LIMA_GP_IRQ_RESET_COMPLETED | \
+ LIMA_GP_IRQ_SEMAPHORE_UNDERFLOW | \
+ LIMA_GP_IRQ_SEMAPHORE_OVERFLOW | \
+ LIMA_GP_IRQ_PTR_ARRAY_OUT_OF_BOUNDS)
+
+#define LIMA_GP_IRQ_MASK_ERROR \
+ ( \
+ LIMA_GP_IRQ_PLBU_OUT_OF_MEM | \
+ LIMA_GP_IRQ_FORCE_HANG | \
+ LIMA_GP_IRQ_WRITE_BOUND_ERR | \
+ LIMA_GP_IRQ_SYNC_ERROR | \
+ LIMA_GP_IRQ_AXI_BUS_ERROR | \
+ LIMA_GP_IRQ_VS_INVALID_CMD | \
+ LIMA_GP_IRQ_PLB_INVALID_CMD | \
+ LIMA_GP_IRQ_SEMAPHORE_UNDERFLOW | \
+ LIMA_GP_IRQ_SEMAPHORE_OVERFLOW | \
+ LIMA_GP_IRQ_PTR_ARRAY_OUT_OF_BOUNDS)
+
+#define LIMA_GP_IRQ_MASK_USED \
+ ( \
+ LIMA_GP_IRQ_VS_END_CMD_LST | \
+ LIMA_GP_IRQ_PLBU_END_CMD_LST | \
+ LIMA_GP_IRQ_MASK_ERROR)
+
+/* PP regs */
+#define LIMA_PP_FRAME 0x0000
+#define LIMA_PP_RSW 0x0004
+#define LIMA_PP_STACK 0x0030
+#define LIMA_PP_STACK_SIZE 0x0034
+#define LIMA_PP_ORIGIN_OFFSET_X 0x0040
+#define LIMA_PP_WB(i) (0x0100 * (i + 1))
+#define LIMA_PP_WB_SOURCE_SELECT 0x0000
+#define LIMA_PP_WB_SOURCE_ADDR 0x0004
+
+#define LIMA_PP_VERSION 0x1000
+#define LIMA_PP_CURRENT_REND_LIST_ADDR 0x1004
+#define LIMA_PP_STATUS 0x1008
+#define LIMA_PP_STATUS_RENDERING_ACTIVE BIT(0)
+#define LIMA_PP_STATUS_BUS_STOPPED BIT(4)
+#define LIMA_PP_CTRL 0x100c
+#define LIMA_PP_CTRL_STOP_BUS BIT(0)
+#define LIMA_PP_CTRL_FLUSH_CACHES BIT(3)
+#define LIMA_PP_CTRL_FORCE_RESET BIT(5)
+#define LIMA_PP_CTRL_START_RENDERING BIT(6)
+#define LIMA_PP_CTRL_SOFT_RESET BIT(7)
+#define LIMA_PP_INT_RAWSTAT 0x1020
+#define LIMA_PP_INT_CLEAR 0x1024
+#define LIMA_PP_INT_MASK 0x1028
+#define LIMA_PP_INT_STATUS 0x102c
+#define LIMA_PP_IRQ_END_OF_FRAME BIT(0)
+#define LIMA_PP_IRQ_END_OF_TILE BIT(1)
+#define LIMA_PP_IRQ_HANG BIT(2)
+#define LIMA_PP_IRQ_FORCE_HANG BIT(3)
+#define LIMA_PP_IRQ_BUS_ERROR BIT(4)
+#define LIMA_PP_IRQ_BUS_STOP BIT(5)
+#define LIMA_PP_IRQ_CNT_0_LIMIT BIT(6)
+#define LIMA_PP_IRQ_CNT_1_LIMIT BIT(7)
+#define LIMA_PP_IRQ_WRITE_BOUNDARY_ERROR BIT(8)
+#define LIMA_PP_IRQ_INVALID_PLIST_COMMAND BIT(9)
+#define LIMA_PP_IRQ_CALL_STACK_UNDERFLOW BIT(10)
+#define LIMA_PP_IRQ_CALL_STACK_OVERFLOW BIT(11)
+#define LIMA_PP_IRQ_RESET_COMPLETED BIT(12)
+#define LIMA_PP_WRITE_BOUNDARY_LOW 0x1044
+#define LIMA_PP_BUS_ERROR_STATUS 0x1050
+#define LIMA_PP_PERF_CNT_0_ENABLE 0x1080
+#define LIMA_PP_PERF_CNT_0_SRC 0x1084
+#define LIMA_PP_PERF_CNT_0_LIMIT 0x1088
+#define LIMA_PP_PERF_CNT_0_VALUE 0x108c
+#define LIMA_PP_PERF_CNT_1_ENABLE 0x10a0
+#define LIMA_PP_PERF_CNT_1_SRC 0x10a4
+#define LIMA_PP_PERF_CNT_1_LIMIT 0x10a8
+#define LIMA_PP_PERF_CNT_1_VALUE 0x10ac
+#define LIMA_PP_PERFMON_CONTR 0x10b0
+#define LIMA_PP_PERFMON_BASE 0x10b4
+
+#define LIMA_PP_IRQ_MASK_ALL \
+ ( \
+ LIMA_PP_IRQ_END_OF_FRAME | \
+ LIMA_PP_IRQ_END_OF_TILE | \
+ LIMA_PP_IRQ_HANG | \
+ LIMA_PP_IRQ_FORCE_HANG | \
+ LIMA_PP_IRQ_BUS_ERROR | \
+ LIMA_PP_IRQ_BUS_STOP | \
+ LIMA_PP_IRQ_CNT_0_LIMIT | \
+ LIMA_PP_IRQ_CNT_1_LIMIT | \
+ LIMA_PP_IRQ_WRITE_BOUNDARY_ERROR | \
+ LIMA_PP_IRQ_INVALID_PLIST_COMMAND | \
+ LIMA_PP_IRQ_CALL_STACK_UNDERFLOW | \
+ LIMA_PP_IRQ_CALL_STACK_OVERFLOW | \
+ LIMA_PP_IRQ_RESET_COMPLETED)
+
+#define LIMA_PP_IRQ_MASK_ERROR \
+ ( \
+ LIMA_PP_IRQ_FORCE_HANG | \
+ LIMA_PP_IRQ_BUS_ERROR | \
+ LIMA_PP_IRQ_WRITE_BOUNDARY_ERROR | \
+ LIMA_PP_IRQ_INVALID_PLIST_COMMAND | \
+ LIMA_PP_IRQ_CALL_STACK_UNDERFLOW | \
+ LIMA_PP_IRQ_CALL_STACK_OVERFLOW)
+
+#define LIMA_PP_IRQ_MASK_USED \
+ ( \
+ LIMA_PP_IRQ_END_OF_FRAME | \
+ LIMA_PP_IRQ_MASK_ERROR)
+
+/* MMU regs */
+#define LIMA_MMU_DTE_ADDR 0x0000
+#define LIMA_MMU_STATUS 0x0004
+#define LIMA_MMU_STATUS_PAGING_ENABLED BIT(0)
+#define LIMA_MMU_STATUS_PAGE_FAULT_ACTIVE BIT(1)
+#define LIMA_MMU_STATUS_STALL_ACTIVE BIT(2)
+#define LIMA_MMU_STATUS_IDLE BIT(3)
+#define LIMA_MMU_STATUS_REPLAY_BUFFER_EMPTY BIT(4)
+#define LIMA_MMU_STATUS_PAGE_FAULT_IS_WRITE BIT(5)
+#define LIMA_MMU_STATUS_BUS_ID(x) ((x >> 6) & 0x1F)
+#define LIMA_MMU_COMMAND 0x0008
+#define LIMA_MMU_COMMAND_ENABLE_PAGING 0x00
+#define LIMA_MMU_COMMAND_DISABLE_PAGING 0x01
+#define LIMA_MMU_COMMAND_ENABLE_STALL 0x02
+#define LIMA_MMU_COMMAND_DISABLE_STALL 0x03
+#define LIMA_MMU_COMMAND_ZAP_CACHE 0x04
+#define LIMA_MMU_COMMAND_PAGE_FAULT_DONE 0x05
+#define LIMA_MMU_COMMAND_HARD_RESET 0x06
+#define LIMA_MMU_PAGE_FAULT_ADDR 0x000C
+#define LIMA_MMU_ZAP_ONE_LINE 0x0010
+#define LIMA_MMU_INT_RAWSTAT 0x0014
+#define LIMA_MMU_INT_CLEAR 0x0018
+#define LIMA_MMU_INT_MASK 0x001C
+#define LIMA_MMU_INT_PAGE_FAULT BIT(0)
+#define LIMA_MMU_INT_READ_BUS_ERROR BIT(1)
+#define LIMA_MMU_INT_STATUS 0x0020
+
+#define LIMA_VM_FLAG_PRESENT BIT(0)
+#define LIMA_VM_FLAG_READ_PERMISSION BIT(1)
+#define LIMA_VM_FLAG_WRITE_PERMISSION BIT(2)
+#define LIMA_VM_FLAG_OVERRIDE_CACHE BIT(3)
+#define LIMA_VM_FLAG_WRITE_CACHEABLE BIT(4)
+#define LIMA_VM_FLAG_WRITE_ALLOCATE BIT(5)
+#define LIMA_VM_FLAG_WRITE_BUFFERABLE BIT(6)
+#define LIMA_VM_FLAG_READ_CACHEABLE BIT(7)
+#define LIMA_VM_FLAG_READ_ALLOCATE BIT(8)
+#define LIMA_VM_FLAG_MASK 0x1FF
+
+#define LIMA_VM_FLAGS_CACHE ( \
+ LIMA_VM_FLAG_PRESENT | \
+ LIMA_VM_FLAG_READ_PERMISSION | \
+ LIMA_VM_FLAG_WRITE_PERMISSION | \
+ LIMA_VM_FLAG_OVERRIDE_CACHE | \
+ LIMA_VM_FLAG_WRITE_CACHEABLE | \
+ LIMA_VM_FLAG_WRITE_BUFFERABLE | \
+ LIMA_VM_FLAG_READ_CACHEABLE | \
+ LIMA_VM_FLAG_READ_ALLOCATE)
+
+#define LIMA_VM_FLAGS_UNCACHE ( \
+ LIMA_VM_FLAG_PRESENT | \
+ LIMA_VM_FLAG_READ_PERMISSION | \
+ LIMA_VM_FLAG_WRITE_PERMISSION)
+
+/* DLBU regs */
+#define LIMA_DLBU_MASTER_TLLIST_PHYS_ADDR 0x0000
+#define LIMA_DLBU_MASTER_TLLIST_VADDR 0x0004
+#define LIMA_DLBU_TLLIST_VBASEADDR 0x0008
+#define LIMA_DLBU_FB_DIM 0x000C
+#define LIMA_DLBU_TLLIST_CONF 0x0010
+#define LIMA_DLBU_START_TILE_POS 0x0014
+#define LIMA_DLBU_PP_ENABLE_MASK 0x0018
+
+/* BCAST regs */
+#define LIMA_BCAST_BROADCAST_MASK 0x0
+#define LIMA_BCAST_INTERRUPT_MASK 0x4
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/kthread.h>
+#include <linux/slab.h>
+#include <linux/xarray.h>
+
+#include "lima_drv.h"
+#include "lima_sched.h"
+#include "lima_vm.h"
+#include "lima_mmu.h"
+#include "lima_l2_cache.h"
+#include "lima_object.h"
+
+struct lima_fence {
+ struct dma_fence base;
+ struct lima_sched_pipe *pipe;
+};
+
+static struct kmem_cache *lima_fence_slab;
+static int lima_fence_slab_refcnt;
+
+int lima_sched_slab_init(void)
+{
+ if (!lima_fence_slab) {
+ lima_fence_slab = kmem_cache_create(
+ "lima_fence", sizeof(struct lima_fence), 0,
+ SLAB_HWCACHE_ALIGN, NULL);
+ if (!lima_fence_slab)
+ return -ENOMEM;
+ }
+
+ lima_fence_slab_refcnt++;
+ return 0;
+}
+
+void lima_sched_slab_fini(void)
+{
+ if (!--lima_fence_slab_refcnt) {
+ kmem_cache_destroy(lima_fence_slab);
+ lima_fence_slab = NULL;
+ }
+}
+
+static inline struct lima_fence *to_lima_fence(struct dma_fence *fence)
+{
+ return container_of(fence, struct lima_fence, base);
+}
+
+static const char *lima_fence_get_driver_name(struct dma_fence *fence)
+{
+ return "lima";
+}
+
+static const char *lima_fence_get_timeline_name(struct dma_fence *fence)
+{
+ struct lima_fence *f = to_lima_fence(fence);
+
+ return f->pipe->base.name;
+}
+
+static void lima_fence_release_rcu(struct rcu_head *rcu)
+{
+ struct dma_fence *f = container_of(rcu, struct dma_fence, rcu);
+ struct lima_fence *fence = to_lima_fence(f);
+
+ kmem_cache_free(lima_fence_slab, fence);
+}
+
+static void lima_fence_release(struct dma_fence *fence)
+{
+ struct lima_fence *f = to_lima_fence(fence);
+
+ call_rcu(&f->base.rcu, lima_fence_release_rcu);
+}
+
+static const struct dma_fence_ops lima_fence_ops = {
+ .get_driver_name = lima_fence_get_driver_name,
+ .get_timeline_name = lima_fence_get_timeline_name,
+ .release = lima_fence_release,
+};
+
+static struct lima_fence *lima_fence_create(struct lima_sched_pipe *pipe)
+{
+ struct lima_fence *fence;
+
+ fence = kmem_cache_zalloc(lima_fence_slab, GFP_KERNEL);
+ if (!fence)
+ return NULL;
+
+ fence->pipe = pipe;
+ dma_fence_init(&fence->base, &lima_fence_ops, &pipe->fence_lock,
+ pipe->fence_context, ++pipe->fence_seqno);
+
+ return fence;
+}
+
+static inline struct lima_sched_task *to_lima_task(struct drm_sched_job *job)
+{
+ return container_of(job, struct lima_sched_task, base);
+}
+
+static inline struct lima_sched_pipe *to_lima_pipe(struct drm_gpu_scheduler *sched)
+{
+ return container_of(sched, struct lima_sched_pipe, base);
+}
+
+int lima_sched_task_init(struct lima_sched_task *task,
+ struct lima_sched_context *context,
+ struct lima_bo **bos, int num_bos,
+ struct lima_vm *vm)
+{
+ int err, i;
+
+ task->bos = kmemdup(bos, sizeof(*bos) * num_bos, GFP_KERNEL);
+ if (!task->bos)
+ return -ENOMEM;
+
+ for (i = 0; i < num_bos; i++)
+ drm_gem_object_get(&bos[i]->gem);
+
+ err = drm_sched_job_init(&task->base, &context->base, vm);
+ if (err) {
+ kfree(task->bos);
+ return err;
+ }
+
+ task->num_bos = num_bos;
+ task->vm = lima_vm_get(vm);
+
+ xa_init_flags(&task->deps, XA_FLAGS_ALLOC);
+
+ return 0;
+}
+
+void lima_sched_task_fini(struct lima_sched_task *task)
+{
+ struct dma_fence *fence;
+ unsigned long index;
+ int i;
+
+ drm_sched_job_cleanup(&task->base);
+
+ xa_for_each(&task->deps, index, fence) {
+ dma_fence_put(fence);
+ }
+ xa_destroy(&task->deps);
+
+ if (task->bos) {
+ for (i = 0; i < task->num_bos; i++)
+ drm_gem_object_put_unlocked(&task->bos[i]->gem);
+ kfree(task->bos);
+ }
+
+ lima_vm_put(task->vm);
+}
+
+int lima_sched_context_init(struct lima_sched_pipe *pipe,
+ struct lima_sched_context *context,
+ atomic_t *guilty)
+{
+ struct drm_sched_rq *rq = pipe->base.sched_rq + DRM_SCHED_PRIORITY_NORMAL;
+
+ return drm_sched_entity_init(&context->base, &rq, 1, guilty);
+}
+
+void lima_sched_context_fini(struct lima_sched_pipe *pipe,
+ struct lima_sched_context *context)
+{
+ drm_sched_entity_fini(&context->base);
+}
+
+struct dma_fence *lima_sched_context_queue_task(struct lima_sched_context *context,
+ struct lima_sched_task *task)
+{
+ struct dma_fence *fence = dma_fence_get(&task->base.s_fence->finished);
+
+ drm_sched_entity_push_job(&task->base, &context->base);
+ return fence;
+}
+
+static struct dma_fence *lima_sched_dependency(struct drm_sched_job *job,
+ struct drm_sched_entity *entity)
+{
+ struct lima_sched_task *task = to_lima_task(job);
+
+ if (!xa_empty(&task->deps))
+ return xa_erase(&task->deps, task->last_dep++);
+
+ return NULL;
+}
+
+static struct dma_fence *lima_sched_run_job(struct drm_sched_job *job)
+{
+ struct lima_sched_task *task = to_lima_task(job);
+ struct lima_sched_pipe *pipe = to_lima_pipe(job->sched);
+ struct lima_fence *fence;
+ struct dma_fence *ret;
+ struct lima_vm *vm = NULL, *last_vm = NULL;
+ int i;
+
+ /* after GPU reset */
+ if (job->s_fence->finished.error < 0)
+ return NULL;
+
+ fence = lima_fence_create(pipe);
+ if (!fence)
+ return NULL;
+ task->fence = &fence->base;
+
+ /* for caller usage of the fence, otherwise irq handler
+ * may consume the fence before caller use it
+ */
+ ret = dma_fence_get(task->fence);
+
+ pipe->current_task = task;
+
+ /* this is needed for MMU to work correctly, otherwise GP/PP
+ * will hang or page fault for unknown reason after running for
+ * a while.
+ *
+ * Need to investigate:
+ * 1. is it related to TLB
+ * 2. how much performance will be affected by L2 cache flush
+ * 3. can we reduce the calling of this function because all
+ * GP/PP use the same L2 cache on mali400
+ *
+ * TODO:
+ * 1. move this to task fini to save some wait time?
+ * 2. when GP/PP use different l2 cache, need PP wait GP l2
+ * cache flush?
+ */
+ for (i = 0; i < pipe->num_l2_cache; i++)
+ lima_l2_cache_flush(pipe->l2_cache[i]);
+
+ if (task->vm != pipe->current_vm) {
+ vm = lima_vm_get(task->vm);
+ last_vm = pipe->current_vm;
+ pipe->current_vm = task->vm;
+ }
+
+ if (pipe->bcast_mmu)
+ lima_mmu_switch_vm(pipe->bcast_mmu, vm);
+ else {
+ for (i = 0; i < pipe->num_mmu; i++)
+ lima_mmu_switch_vm(pipe->mmu[i], vm);
+ }
+
+ if (last_vm)
+ lima_vm_put(last_vm);
+
+ pipe->error = false;
+ pipe->task_run(pipe, task);
+
+ return task->fence;
+}
+
+static void lima_sched_handle_error_task(struct lima_sched_pipe *pipe,
+ struct lima_sched_task *task)
+{
+ drm_sched_stop(&pipe->base);
+
+ if (task)
+ drm_sched_increase_karma(&task->base);
+
+ pipe->task_error(pipe);
+
+ if (pipe->bcast_mmu)
+ lima_mmu_page_fault_resume(pipe->bcast_mmu);
+ else {
+ int i;
+
+ for (i = 0; i < pipe->num_mmu; i++)
+ lima_mmu_page_fault_resume(pipe->mmu[i]);
+ }
+
+ if (pipe->current_vm)
+ lima_vm_put(pipe->current_vm);
+
+ pipe->current_vm = NULL;
+ pipe->current_task = NULL;
+
+ drm_sched_resubmit_jobs(&pipe->base);
+ drm_sched_start(&pipe->base, true);
+}
+
+static void lima_sched_timedout_job(struct drm_sched_job *job)
+{
+ struct lima_sched_pipe *pipe = to_lima_pipe(job->sched);
+ struct lima_sched_task *task = to_lima_task(job);
+
+ DRM_ERROR("lima job timeout\n");
+
+ lima_sched_handle_error_task(pipe, task);
+}
+
+static void lima_sched_free_job(struct drm_sched_job *job)
+{
+ struct lima_sched_task *task = to_lima_task(job);
+ struct lima_sched_pipe *pipe = to_lima_pipe(job->sched);
+ struct lima_vm *vm = task->vm;
+ struct lima_bo **bos = task->bos;
+ int i;
+
+ dma_fence_put(task->fence);
+
+ for (i = 0; i < task->num_bos; i++)
+ lima_vm_bo_del(vm, bos[i]);
+
+ lima_sched_task_fini(task);
+ kmem_cache_free(pipe->task_slab, task);
+}
+
+static const struct drm_sched_backend_ops lima_sched_ops = {
+ .dependency = lima_sched_dependency,
+ .run_job = lima_sched_run_job,
+ .timedout_job = lima_sched_timedout_job,
+ .free_job = lima_sched_free_job,
+};
+
+static void lima_sched_error_work(struct work_struct *work)
+{
+ struct lima_sched_pipe *pipe =
+ container_of(work, struct lima_sched_pipe, error_work);
+ struct lima_sched_task *task = pipe->current_task;
+
+ lima_sched_handle_error_task(pipe, task);
+}
+
+int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name)
+{
+ long timeout;
+
+ if (lima_sched_timeout_ms <= 0)
+ timeout = MAX_SCHEDULE_TIMEOUT;
+ else
+ timeout = msecs_to_jiffies(lima_sched_timeout_ms);
+
+ pipe->fence_context = dma_fence_context_alloc(1);
+ spin_lock_init(&pipe->fence_lock);
+
+ INIT_WORK(&pipe->error_work, lima_sched_error_work);
+
+ return drm_sched_init(&pipe->base, &lima_sched_ops, 1, 0, timeout, name);
+}
+
+void lima_sched_pipe_fini(struct lima_sched_pipe *pipe)
+{
+ drm_sched_fini(&pipe->base);
+}
+
+void lima_sched_pipe_task_done(struct lima_sched_pipe *pipe)
+{
+ if (pipe->error)
+ schedule_work(&pipe->error_work);
+ else {
+ struct lima_sched_task *task = pipe->current_task;
+
+ pipe->task_fini(pipe);
+ dma_fence_signal(task->fence);
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_SCHED_H__
+#define __LIMA_SCHED_H__
+
+#include <drm/gpu_scheduler.h>
+
+struct lima_vm;
+
+struct lima_sched_task {
+ struct drm_sched_job base;
+
+ struct lima_vm *vm;
+ void *frame;
+
+ struct xarray deps;
+ unsigned long last_dep;
+
+ struct lima_bo **bos;
+ int num_bos;
+
+ /* pipe fence */
+ struct dma_fence *fence;
+};
+
+struct lima_sched_context {
+ struct drm_sched_entity base;
+};
+
+#define LIMA_SCHED_PIPE_MAX_MMU 8
+#define LIMA_SCHED_PIPE_MAX_L2_CACHE 2
+#define LIMA_SCHED_PIPE_MAX_PROCESSOR 8
+
+struct lima_ip;
+
+struct lima_sched_pipe {
+ struct drm_gpu_scheduler base;
+
+ u64 fence_context;
+ u32 fence_seqno;
+ spinlock_t fence_lock;
+
+ struct lima_sched_task *current_task;
+ struct lima_vm *current_vm;
+
+ struct lima_ip *mmu[LIMA_SCHED_PIPE_MAX_MMU];
+ int num_mmu;
+
+ struct lima_ip *l2_cache[LIMA_SCHED_PIPE_MAX_L2_CACHE];
+ int num_l2_cache;
+
+ struct lima_ip *processor[LIMA_SCHED_PIPE_MAX_PROCESSOR];
+ int num_processor;
+
+ struct lima_ip *bcast_processor;
+ struct lima_ip *bcast_mmu;
+
+ u32 done;
+ bool error;
+ atomic_t task;
+
+ int frame_size;
+ struct kmem_cache *task_slab;
+
+ int (*task_validate)(struct lima_sched_pipe *pipe, struct lima_sched_task *task);
+ void (*task_run)(struct lima_sched_pipe *pipe, struct lima_sched_task *task);
+ void (*task_fini)(struct lima_sched_pipe *pipe);
+ void (*task_error)(struct lima_sched_pipe *pipe);
+ void (*task_mmu_error)(struct lima_sched_pipe *pipe);
+
+ struct work_struct error_work;
+};
+
+int lima_sched_task_init(struct lima_sched_task *task,
+ struct lima_sched_context *context,
+ struct lima_bo **bos, int num_bos,
+ struct lima_vm *vm);
+void lima_sched_task_fini(struct lima_sched_task *task);
+
+int lima_sched_context_init(struct lima_sched_pipe *pipe,
+ struct lima_sched_context *context,
+ atomic_t *guilty);
+void lima_sched_context_fini(struct lima_sched_pipe *pipe,
+ struct lima_sched_context *context);
+struct dma_fence *lima_sched_context_queue_task(struct lima_sched_context *context,
+ struct lima_sched_task *task);
+
+int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name);
+void lima_sched_pipe_fini(struct lima_sched_pipe *pipe);
+void lima_sched_pipe_task_done(struct lima_sched_pipe *pipe);
+
+static inline void lima_sched_pipe_mmu_error(struct lima_sched_pipe *pipe)
+{
+ pipe->error = true;
+ pipe->task_mmu_error(pipe);
+}
+
+int lima_sched_slab_init(void);
+void lima_sched_slab_fini(void);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR MIT
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#include <linux/slab.h>
+#include <linux/dma-mapping.h>
+
+#include "lima_device.h"
+#include "lima_vm.h"
+#include "lima_object.h"
+#include "lima_regs.h"
+
+struct lima_bo_va {
+ struct list_head list;
+ unsigned int ref_count;
+
+ struct drm_mm_node node;
+
+ struct lima_vm *vm;
+};
+
+#define LIMA_VM_PD_SHIFT 22
+#define LIMA_VM_PT_SHIFT 12
+#define LIMA_VM_PB_SHIFT (LIMA_VM_PD_SHIFT + LIMA_VM_NUM_PT_PER_BT_SHIFT)
+#define LIMA_VM_BT_SHIFT LIMA_VM_PT_SHIFT
+
+#define LIMA_VM_PT_MASK ((1 << LIMA_VM_PD_SHIFT) - 1)
+#define LIMA_VM_BT_MASK ((1 << LIMA_VM_PB_SHIFT) - 1)
+
+#define LIMA_PDE(va) (va >> LIMA_VM_PD_SHIFT)
+#define LIMA_PTE(va) ((va & LIMA_VM_PT_MASK) >> LIMA_VM_PT_SHIFT)
+#define LIMA_PBE(va) (va >> LIMA_VM_PB_SHIFT)
+#define LIMA_BTE(va) ((va & LIMA_VM_BT_MASK) >> LIMA_VM_BT_SHIFT)
+
+
+static void lima_vm_unmap_page_table(struct lima_vm *vm, u32 start, u32 end)
+{
+ u32 addr;
+
+ for (addr = start; addr <= end; addr += LIMA_PAGE_SIZE) {
+ u32 pbe = LIMA_PBE(addr);
+ u32 bte = LIMA_BTE(addr);
+
+ vm->bts[pbe].cpu[bte] = 0;
+ }
+}
+
+static int lima_vm_map_page_table(struct lima_vm *vm, dma_addr_t *dma,
+ u32 start, u32 end)
+{
+ u64 addr;
+ int i = 0;
+
+ for (addr = start; addr <= end; addr += LIMA_PAGE_SIZE) {
+ u32 pbe = LIMA_PBE(addr);
+ u32 bte = LIMA_BTE(addr);
+
+ if (!vm->bts[pbe].cpu) {
+ dma_addr_t pts;
+ u32 *pd;
+ int j;
+
+ vm->bts[pbe].cpu = dma_alloc_wc(
+ vm->dev->dev, LIMA_PAGE_SIZE << LIMA_VM_NUM_PT_PER_BT_SHIFT,
+ &vm->bts[pbe].dma, GFP_KERNEL | __GFP_ZERO);
+ if (!vm->bts[pbe].cpu) {
+ if (addr != start)
+ lima_vm_unmap_page_table(vm, start, addr - 1);
+ return -ENOMEM;
+ }
+
+ pts = vm->bts[pbe].dma;
+ pd = vm->pd.cpu + (pbe << LIMA_VM_NUM_PT_PER_BT_SHIFT);
+ for (j = 0; j < LIMA_VM_NUM_PT_PER_BT; j++) {
+ pd[j] = pts | LIMA_VM_FLAG_PRESENT;
+ pts += LIMA_PAGE_SIZE;
+ }
+ }
+
+ vm->bts[pbe].cpu[bte] = dma[i++] | LIMA_VM_FLAGS_CACHE;
+ }
+
+ return 0;
+}
+
+static struct lima_bo_va *
+lima_vm_bo_find(struct lima_vm *vm, struct lima_bo *bo)
+{
+ struct lima_bo_va *bo_va, *ret = NULL;
+
+ list_for_each_entry(bo_va, &bo->va, list) {
+ if (bo_va->vm == vm) {
+ ret = bo_va;
+ break;
+ }
+ }
+
+ return ret;
+}
+
+int lima_vm_bo_add(struct lima_vm *vm, struct lima_bo *bo, bool create)
+{
+ struct lima_bo_va *bo_va;
+ int err;
+
+ mutex_lock(&bo->lock);
+
+ bo_va = lima_vm_bo_find(vm, bo);
+ if (bo_va) {
+ bo_va->ref_count++;
+ mutex_unlock(&bo->lock);
+ return 0;
+ }
+
+ /* should not create new bo_va if not asked by caller */
+ if (!create) {
+ mutex_unlock(&bo->lock);
+ return -ENOENT;
+ }
+
+ bo_va = kzalloc(sizeof(*bo_va), GFP_KERNEL);
+ if (!bo_va) {
+ err = -ENOMEM;
+ goto err_out0;
+ }
+
+ bo_va->vm = vm;
+ bo_va->ref_count = 1;
+
+ mutex_lock(&vm->lock);
+
+ err = drm_mm_insert_node(&vm->mm, &bo_va->node, bo->gem.size);
+ if (err)
+ goto err_out1;
+
+ err = lima_vm_map_page_table(vm, bo->pages_dma_addr, bo_va->node.start,
+ bo_va->node.start + bo_va->node.size - 1);
+ if (err)
+ goto err_out2;
+
+ mutex_unlock(&vm->lock);
+
+ list_add_tail(&bo_va->list, &bo->va);
+
+ mutex_unlock(&bo->lock);
+ return 0;
+
+err_out2:
+ drm_mm_remove_node(&bo_va->node);
+err_out1:
+ mutex_unlock(&vm->lock);
+ kfree(bo_va);
+err_out0:
+ mutex_unlock(&bo->lock);
+ return err;
+}
+
+void lima_vm_bo_del(struct lima_vm *vm, struct lima_bo *bo)
+{
+ struct lima_bo_va *bo_va;
+
+ mutex_lock(&bo->lock);
+
+ bo_va = lima_vm_bo_find(vm, bo);
+ if (--bo_va->ref_count > 0) {
+ mutex_unlock(&bo->lock);
+ return;
+ }
+
+ mutex_lock(&vm->lock);
+
+ lima_vm_unmap_page_table(vm, bo_va->node.start,
+ bo_va->node.start + bo_va->node.size - 1);
+
+ drm_mm_remove_node(&bo_va->node);
+
+ mutex_unlock(&vm->lock);
+
+ list_del(&bo_va->list);
+
+ mutex_unlock(&bo->lock);
+
+ kfree(bo_va);
+}
+
+u32 lima_vm_get_va(struct lima_vm *vm, struct lima_bo *bo)
+{
+ struct lima_bo_va *bo_va;
+ u32 ret;
+
+ mutex_lock(&bo->lock);
+
+ bo_va = lima_vm_bo_find(vm, bo);
+ ret = bo_va->node.start;
+
+ mutex_unlock(&bo->lock);
+
+ return ret;
+}
+
+struct lima_vm *lima_vm_create(struct lima_device *dev)
+{
+ struct lima_vm *vm;
+
+ vm = kzalloc(sizeof(*vm), GFP_KERNEL);
+ if (!vm)
+ return NULL;
+
+ vm->dev = dev;
+ mutex_init(&vm->lock);
+ kref_init(&vm->refcount);
+
+ vm->pd.cpu = dma_alloc_wc(dev->dev, LIMA_PAGE_SIZE, &vm->pd.dma,
+ GFP_KERNEL | __GFP_ZERO);
+ if (!vm->pd.cpu)
+ goto err_out0;
+
+ if (dev->dlbu_cpu) {
+ int err = lima_vm_map_page_table(
+ vm, &dev->dlbu_dma, LIMA_VA_RESERVE_DLBU,
+ LIMA_VA_RESERVE_DLBU + LIMA_PAGE_SIZE - 1);
+ if (err)
+ goto err_out1;
+ }
+
+ drm_mm_init(&vm->mm, dev->va_start, dev->va_end - dev->va_start);
+
+ return vm;
+
+err_out1:
+ dma_free_wc(dev->dev, LIMA_PAGE_SIZE, vm->pd.cpu, vm->pd.dma);
+err_out0:
+ kfree(vm);
+ return NULL;
+}
+
+void lima_vm_release(struct kref *kref)
+{
+ struct lima_vm *vm = container_of(kref, struct lima_vm, refcount);
+ int i;
+
+ drm_mm_takedown(&vm->mm);
+
+ for (i = 0; i < LIMA_VM_NUM_BT; i++) {
+ if (vm->bts[i].cpu)
+ dma_free_wc(vm->dev->dev, LIMA_PAGE_SIZE << LIMA_VM_NUM_PT_PER_BT_SHIFT,
+ vm->bts[i].cpu, vm->bts[i].dma);
+ }
+
+ if (vm->pd.cpu)
+ dma_free_wc(vm->dev->dev, LIMA_PAGE_SIZE, vm->pd.cpu, vm->pd.dma);
+
+ kfree(vm);
+}
+
+void lima_vm_print(struct lima_vm *vm)
+{
+ int i, j, k;
+ u32 *pd, *pt;
+
+ if (!vm->pd.cpu)
+ return;
+
+ pd = vm->pd.cpu;
+ for (i = 0; i < LIMA_VM_NUM_BT; i++) {
+ if (!vm->bts[i].cpu)
+ continue;
+
+ pt = vm->bts[i].cpu;
+ for (j = 0; j < LIMA_VM_NUM_PT_PER_BT; j++) {
+ int idx = (i << LIMA_VM_NUM_PT_PER_BT_SHIFT) + j;
+
+ printk(KERN_INFO "lima vm pd %03x:%08x\n", idx, pd[idx]);
+
+ for (k = 0; k < LIMA_PAGE_ENT_NUM; k++) {
+ u32 pte = *pt++;
+
+ if (pte)
+ printk(KERN_INFO " pt %03x:%08x\n", k, pte);
+ }
+ }
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+/* Copyright 2017-2019 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_VM_H__
+#define __LIMA_VM_H__
+
+#include <drm/drm_mm.h>
+#include <linux/kref.h>
+
+#define LIMA_PAGE_SIZE 4096
+#define LIMA_PAGE_MASK (LIMA_PAGE_SIZE - 1)
+#define LIMA_PAGE_ENT_NUM (LIMA_PAGE_SIZE / sizeof(u32))
+
+#define LIMA_VM_NUM_PT_PER_BT_SHIFT 3
+#define LIMA_VM_NUM_PT_PER_BT (1 << LIMA_VM_NUM_PT_PER_BT_SHIFT)
+#define LIMA_VM_NUM_BT (LIMA_PAGE_ENT_NUM >> LIMA_VM_NUM_PT_PER_BT_SHIFT)
+
+#define LIMA_VA_RESERVE_START 0xFFF00000
+#define LIMA_VA_RESERVE_DLBU LIMA_VA_RESERVE_START
+#define LIMA_VA_RESERVE_END 0x100000000
+
+struct lima_device;
+
+struct lima_vm_page {
+ u32 *cpu;
+ dma_addr_t dma;
+};
+
+struct lima_vm {
+ struct mutex lock;
+ struct kref refcount;
+
+ struct drm_mm mm;
+
+ struct lima_device *dev;
+
+ struct lima_vm_page pd;
+ struct lima_vm_page bts[LIMA_VM_NUM_BT];
+};
+
+int lima_vm_bo_add(struct lima_vm *vm, struct lima_bo *bo, bool create);
+void lima_vm_bo_del(struct lima_vm *vm, struct lima_bo *bo);
+
+u32 lima_vm_get_va(struct lima_vm *vm, struct lima_bo *bo);
+
+struct lima_vm *lima_vm_create(struct lima_device *dev);
+void lima_vm_release(struct kref *kref);
+
+static inline struct lima_vm *lima_vm_get(struct lima_vm *vm)
+{
+ kref_get(&vm->refcount);
+ return vm;
+}
+
+static inline void lima_vm_put(struct lima_vm *vm)
+{
+ kref_put(&vm->refcount, lima_vm_release);
+}
+
+void lima_vm_print(struct lima_vm *vm);
+
+#endif
meson-drm-y := meson_drv.o meson_plane.o meson_crtc.o meson_venc_cvbs.o
-meson-drm-y += meson_viu.o meson_vpp.o meson_venc.o meson_vclk.o meson_canvas.o meson_overlay.o
+meson-drm-y += meson_viu.o meson_vpp.o meson_venc.o meson_vclk.o meson_overlay.o
obj-$(CONFIG_DRM_MESON) += meson-drm.o
obj-$(CONFIG_DRM_MESON_DW_HDMI) += meson_dw_hdmi.o
+++ /dev/null
-/*
- * Copyright (C) 2016 BayLibre, SAS
- * Author: Neil Armstrong <narmstrong@baylibre.com>
- * Copyright (C) 2015 Amlogic, Inc. All rights reserved.
- * Copyright (C) 2014 Endless Mobile
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation; either version 2 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include "meson_drv.h"
-#include "meson_canvas.h"
-#include "meson_registers.h"
-
-/**
- * DOC: Canvas
- *
- * CANVAS is a memory zone where physical memory frames information
- * are stored for the VIU to scanout.
- */
-
-/* DMC Registers */
-#define DMC_CAV_LUT_DATAL 0x48 /* 0x12 offset in data sheet */
-#define CANVAS_WIDTH_LBIT 29
-#define CANVAS_WIDTH_LWID 3
-#define DMC_CAV_LUT_DATAH 0x4c /* 0x13 offset in data sheet */
-#define CANVAS_WIDTH_HBIT 0
-#define CANVAS_HEIGHT_BIT 9
-#define CANVAS_BLKMODE_BIT 24
-#define CANVAS_ENDIAN_BIT 26
-#define DMC_CAV_LUT_ADDR 0x50 /* 0x14 offset in data sheet */
-#define CANVAS_LUT_WR_EN (0x2 << 8)
-#define CANVAS_LUT_RD_EN (0x1 << 8)
-
-void meson_canvas_setup(struct meson_drm *priv,
- uint32_t canvas_index, uint32_t addr,
- uint32_t stride, uint32_t height,
- unsigned int wrap,
- unsigned int blkmode,
- unsigned int endian)
-{
- unsigned int val;
-
- regmap_write(priv->dmc, DMC_CAV_LUT_DATAL,
- (((addr + 7) >> 3)) |
- (((stride + 7) >> 3) << CANVAS_WIDTH_LBIT));
-
- regmap_write(priv->dmc, DMC_CAV_LUT_DATAH,
- ((((stride + 7) >> 3) >> CANVAS_WIDTH_LWID) <<
- CANVAS_WIDTH_HBIT) |
- (height << CANVAS_HEIGHT_BIT) |
- (wrap << 22) |
- (blkmode << CANVAS_BLKMODE_BIT) |
- (endian << CANVAS_ENDIAN_BIT));
-
- regmap_write(priv->dmc, DMC_CAV_LUT_ADDR,
- CANVAS_LUT_WR_EN | canvas_index);
-
- /* Force a read-back to make sure everything is flushed. */
- regmap_read(priv->dmc, DMC_CAV_LUT_DATAH, &val);
-}
+++ /dev/null
-/*
- * Copyright (C) 2016 BayLibre, SAS
- * Author: Neil Armstrong <narmstrong@baylibre.com>
- * Copyright (C) 2014 Endless Mobile
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation; either version 2 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-/* Canvas LUT Memory */
-
-#ifndef __MESON_CANVAS_H
-#define __MESON_CANVAS_H
-
-#define MESON_CANVAS_ID_OSD1 0x4e
-#define MESON_CANVAS_ID_VD1_0 0x60
-#define MESON_CANVAS_ID_VD1_1 0x61
-#define MESON_CANVAS_ID_VD1_2 0x62
-
-/* Canvas configuration. */
-#define MESON_CANVAS_WRAP_NONE 0x00
-#define MESON_CANVAS_WRAP_X 0x01
-#define MESON_CANVAS_WRAP_Y 0x02
-
-#define MESON_CANVAS_BLKMODE_LINEAR 0x00
-#define MESON_CANVAS_BLKMODE_32x32 0x01
-#define MESON_CANVAS_BLKMODE_64x64 0x02
-
-#define MESON_CANVAS_ENDIAN_SWAP16 0x1
-#define MESON_CANVAS_ENDIAN_SWAP32 0x3
-#define MESON_CANVAS_ENDIAN_SWAP64 0x7
-#define MESON_CANVAS_ENDIAN_SWAP128 0xf
-
-void meson_canvas_setup(struct meson_drm *priv,
- uint32_t canvas_index, uint32_t addr,
- uint32_t stride, uint32_t height,
- unsigned int wrap,
- unsigned int blkmode,
- unsigned int endian);
-
-#endif /* __MESON_CANVAS_H */
#include "meson_venc.h"
#include "meson_vpp.h"
#include "meson_viu.h"
-#include "meson_canvas.h"
#include "meson_registers.h"
+#define MESON_G12A_VIU_OFFSET 0x5ec0
+
/* CRTC definition */
struct meson_crtc {
struct drm_crtc base;
struct drm_pending_vblank_event *event;
struct meson_drm *priv;
+ void (*enable_osd1)(struct meson_drm *priv);
+ void (*enable_vd1)(struct meson_drm *priv);
+ unsigned int viu_offset;
};
#define to_meson_crtc(x) container_of(x, struct meson_crtc, base)
};
+static void meson_g12a_crtc_atomic_enable(struct drm_crtc *crtc,
+ struct drm_crtc_state *old_state)
+{
+ struct meson_crtc *meson_crtc = to_meson_crtc(crtc);
+ struct drm_crtc_state *crtc_state = crtc->state;
+ struct meson_drm *priv = meson_crtc->priv;
+
+ DRM_DEBUG_DRIVER("\n");
+
+ if (!crtc_state) {
+ DRM_ERROR("Invalid crtc_state\n");
+ return;
+ }
+
+ /* VD1 Preblend vertical start/end */
+ writel(FIELD_PREP(GENMASK(11, 0), 2303),
+ priv->io_base + _REG(VPP_PREBLEND_VD1_V_START_END));
+
+ /* Setup Blender */
+ writel(crtc_state->mode.hdisplay |
+ crtc_state->mode.vdisplay << 16,
+ priv->io_base + _REG(VPP_POSTBLEND_H_SIZE));
+
+ writel_relaxed(0 << 16 |
+ (crtc_state->mode.hdisplay - 1),
+ priv->io_base + _REG(VPP_OSD1_BLD_H_SCOPE));
+ writel_relaxed(0 << 16 |
+ (crtc_state->mode.vdisplay - 1),
+ priv->io_base + _REG(VPP_OSD1_BLD_V_SCOPE));
+ writel_relaxed(crtc_state->mode.hdisplay << 16 |
+ crtc_state->mode.vdisplay,
+ priv->io_base + _REG(VPP_OUT_H_V_SIZE));
+
+ drm_crtc_vblank_on(crtc);
+
+ priv->viu.osd1_enabled = true;
+}
+
static void meson_crtc_atomic_enable(struct drm_crtc *crtc,
struct drm_crtc_state *old_state)
{
priv->viu.osd1_enabled = true;
}
+static void meson_g12a_crtc_atomic_disable(struct drm_crtc *crtc,
+ struct drm_crtc_state *old_state)
+{
+ struct meson_crtc *meson_crtc = to_meson_crtc(crtc);
+ struct meson_drm *priv = meson_crtc->priv;
+
+ DRM_DEBUG_DRIVER("\n");
+
+ drm_crtc_vblank_off(crtc);
+
+ priv->viu.osd1_enabled = false;
+ priv->viu.osd1_commit = false;
+
+ priv->viu.vd1_enabled = false;
+ priv->viu.vd1_commit = false;
+
+ if (crtc->state->event && !crtc->state->active) {
+ spin_lock_irq(&crtc->dev->event_lock);
+ drm_crtc_send_vblank_event(crtc, crtc->state->event);
+ spin_unlock_irq(&crtc->dev->event_lock);
+
+ crtc->state->event = NULL;
+ }
+}
+
static void meson_crtc_atomic_disable(struct drm_crtc *crtc,
struct drm_crtc_state *old_state)
{
.atomic_disable = meson_crtc_atomic_disable,
};
+static const struct drm_crtc_helper_funcs meson_g12a_crtc_helper_funcs = {
+ .atomic_begin = meson_crtc_atomic_begin,
+ .atomic_flush = meson_crtc_atomic_flush,
+ .atomic_enable = meson_g12a_crtc_atomic_enable,
+ .atomic_disable = meson_g12a_crtc_atomic_disable,
+};
+
+static void meson_crtc_enable_osd1(struct meson_drm *priv)
+{
+ writel_bits_relaxed(VPP_OSD1_POSTBLEND, VPP_OSD1_POSTBLEND,
+ priv->io_base + _REG(VPP_MISC));
+}
+
+static void meson_g12a_crtc_enable_osd1(struct meson_drm *priv)
+{
+ writel_relaxed(priv->viu.osd_blend_din0_scope_h,
+ priv->io_base +
+ _REG(VIU_OSD_BLEND_DIN0_SCOPE_H));
+ writel_relaxed(priv->viu.osd_blend_din0_scope_v,
+ priv->io_base +
+ _REG(VIU_OSD_BLEND_DIN0_SCOPE_V));
+ writel_relaxed(priv->viu.osb_blend0_size,
+ priv->io_base +
+ _REG(VIU_OSD_BLEND_BLEND0_SIZE));
+ writel_relaxed(priv->viu.osb_blend1_size,
+ priv->io_base +
+ _REG(VIU_OSD_BLEND_BLEND1_SIZE));
+}
+
+static void meson_crtc_enable_vd1(struct meson_drm *priv)
+{
+ writel_bits_relaxed(VPP_VD1_PREBLEND | VPP_VD1_POSTBLEND |
+ VPP_COLOR_MNG_ENABLE,
+ VPP_VD1_PREBLEND | VPP_VD1_POSTBLEND |
+ VPP_COLOR_MNG_ENABLE,
+ priv->io_base + _REG(VPP_MISC));
+}
+
+static void meson_g12a_crtc_enable_vd1(struct meson_drm *priv)
+{
+ writel_relaxed(((1 << 16) | /* post bld premult*/
+ (1 << 8) | /* post src */
+ (1 << 4) | /* pre bld premult*/
+ (1 << 0)),
+ priv->io_base + _REG(VD1_BLEND_SRC_CTRL));
+}
+
void meson_crtc_irq(struct meson_drm *priv)
{
struct meson_crtc *meson_crtc = to_meson_crtc(priv->crtc);
writel_relaxed(priv->viu.osd_sc_v_ctrl0,
priv->io_base + _REG(VPP_OSD_VSC_CTRL0));
- if (priv->canvas)
- meson_canvas_config(priv->canvas, priv->canvas_id_osd1,
- priv->viu.osd1_addr, priv->viu.osd1_stride,
- priv->viu.osd1_height, MESON_CANVAS_WRAP_NONE,
- MESON_CANVAS_BLKMODE_LINEAR, 0);
- else
- meson_canvas_setup(priv, MESON_CANVAS_ID_OSD1,
+ meson_canvas_config(priv->canvas, priv->canvas_id_osd1,
priv->viu.osd1_addr, priv->viu.osd1_stride,
priv->viu.osd1_height, MESON_CANVAS_WRAP_NONE,
MESON_CANVAS_BLKMODE_LINEAR, 0);
/* Enable OSD1 */
- writel_bits_relaxed(VPP_OSD1_POSTBLEND, VPP_OSD1_POSTBLEND,
- priv->io_base + _REG(VPP_MISC));
+ if (meson_crtc->enable_osd1)
+ meson_crtc->enable_osd1(priv);
priv->viu.osd1_commit = false;
}
switch (priv->viu.vd1_planes) {
case 3:
- if (priv->canvas)
- meson_canvas_config(priv->canvas,
- priv->canvas_id_vd1_2,
- priv->viu.vd1_addr2,
- priv->viu.vd1_stride2,
- priv->viu.vd1_height2,
- MESON_CANVAS_WRAP_NONE,
- MESON_CANVAS_BLKMODE_LINEAR,
- MESON_CANVAS_ENDIAN_SWAP64);
- else
- meson_canvas_setup(priv, MESON_CANVAS_ID_VD1_2,
- priv->viu.vd1_addr2,
- priv->viu.vd1_stride2,
- priv->viu.vd1_height2,
- MESON_CANVAS_WRAP_NONE,
- MESON_CANVAS_BLKMODE_LINEAR,
- MESON_CANVAS_ENDIAN_SWAP64);
+ meson_canvas_config(priv->canvas,
+ priv->canvas_id_vd1_2,
+ priv->viu.vd1_addr2,
+ priv->viu.vd1_stride2,
+ priv->viu.vd1_height2,
+ MESON_CANVAS_WRAP_NONE,
+ MESON_CANVAS_BLKMODE_LINEAR,
+ MESON_CANVAS_ENDIAN_SWAP64);
/* fallthrough */
case 2:
- if (priv->canvas)
- meson_canvas_config(priv->canvas,
- priv->canvas_id_vd1_1,
- priv->viu.vd1_addr1,
- priv->viu.vd1_stride1,
- priv->viu.vd1_height1,
- MESON_CANVAS_WRAP_NONE,
- MESON_CANVAS_BLKMODE_LINEAR,
- MESON_CANVAS_ENDIAN_SWAP64);
- else
- meson_canvas_setup(priv, MESON_CANVAS_ID_VD1_1,
- priv->viu.vd1_addr2,
- priv->viu.vd1_stride2,
- priv->viu.vd1_height2,
- MESON_CANVAS_WRAP_NONE,
- MESON_CANVAS_BLKMODE_LINEAR,
- MESON_CANVAS_ENDIAN_SWAP64);
+ meson_canvas_config(priv->canvas,
+ priv->canvas_id_vd1_1,
+ priv->viu.vd1_addr1,
+ priv->viu.vd1_stride1,
+ priv->viu.vd1_height1,
+ MESON_CANVAS_WRAP_NONE,
+ MESON_CANVAS_BLKMODE_LINEAR,
+ MESON_CANVAS_ENDIAN_SWAP64);
/* fallthrough */
case 1:
- if (priv->canvas)
- meson_canvas_config(priv->canvas,
- priv->canvas_id_vd1_0,
- priv->viu.vd1_addr0,
- priv->viu.vd1_stride0,
- priv->viu.vd1_height0,
- MESON_CANVAS_WRAP_NONE,
- MESON_CANVAS_BLKMODE_LINEAR,
- MESON_CANVAS_ENDIAN_SWAP64);
- else
- meson_canvas_setup(priv, MESON_CANVAS_ID_VD1_0,
- priv->viu.vd1_addr2,
- priv->viu.vd1_stride2,
- priv->viu.vd1_height2,
- MESON_CANVAS_WRAP_NONE,
- MESON_CANVAS_BLKMODE_LINEAR,
- MESON_CANVAS_ENDIAN_SWAP64);
+ meson_canvas_config(priv->canvas,
+ priv->canvas_id_vd1_0,
+ priv->viu.vd1_addr0,
+ priv->viu.vd1_stride0,
+ priv->viu.vd1_height0,
+ MESON_CANVAS_WRAP_NONE,
+ MESON_CANVAS_BLKMODE_LINEAR,
+ MESON_CANVAS_ENDIAN_SWAP64);
};
writel_relaxed(priv->viu.vd1_if0_gen_reg,
- priv->io_base + _REG(VD1_IF0_GEN_REG));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_GEN_REG));
writel_relaxed(priv->viu.vd1_if0_gen_reg,
- priv->io_base + _REG(VD2_IF0_GEN_REG));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_GEN_REG));
writel_relaxed(priv->viu.vd1_if0_gen_reg2,
- priv->io_base + _REG(VD1_IF0_GEN_REG2));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_GEN_REG2));
writel_relaxed(priv->viu.viu_vd1_fmt_ctrl,
- priv->io_base + _REG(VIU_VD1_FMT_CTRL));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VIU_VD1_FMT_CTRL));
writel_relaxed(priv->viu.viu_vd1_fmt_ctrl,
- priv->io_base + _REG(VIU_VD2_FMT_CTRL));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VIU_VD2_FMT_CTRL));
writel_relaxed(priv->viu.viu_vd1_fmt_w,
- priv->io_base + _REG(VIU_VD1_FMT_W));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VIU_VD1_FMT_W));
writel_relaxed(priv->viu.viu_vd1_fmt_w,
- priv->io_base + _REG(VIU_VD2_FMT_W));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VIU_VD2_FMT_W));
writel_relaxed(priv->viu.vd1_if0_canvas0,
- priv->io_base + _REG(VD1_IF0_CANVAS0));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_CANVAS0));
writel_relaxed(priv->viu.vd1_if0_canvas0,
- priv->io_base + _REG(VD1_IF0_CANVAS1));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_CANVAS1));
writel_relaxed(priv->viu.vd1_if0_canvas0,
- priv->io_base + _REG(VD2_IF0_CANVAS0));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_CANVAS0));
writel_relaxed(priv->viu.vd1_if0_canvas0,
- priv->io_base + _REG(VD2_IF0_CANVAS1));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_CANVAS1));
writel_relaxed(priv->viu.vd1_if0_luma_x0,
- priv->io_base + _REG(VD1_IF0_LUMA_X0));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_LUMA_X0));
writel_relaxed(priv->viu.vd1_if0_luma_x0,
- priv->io_base + _REG(VD1_IF0_LUMA_X1));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_LUMA_X1));
writel_relaxed(priv->viu.vd1_if0_luma_x0,
- priv->io_base + _REG(VD2_IF0_LUMA_X0));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_LUMA_X0));
writel_relaxed(priv->viu.vd1_if0_luma_x0,
- priv->io_base + _REG(VD2_IF0_LUMA_X1));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_LUMA_X1));
writel_relaxed(priv->viu.vd1_if0_luma_y0,
- priv->io_base + _REG(VD1_IF0_LUMA_Y0));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_LUMA_Y0));
writel_relaxed(priv->viu.vd1_if0_luma_y0,
- priv->io_base + _REG(VD1_IF0_LUMA_Y1));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_LUMA_Y1));
writel_relaxed(priv->viu.vd1_if0_luma_y0,
- priv->io_base + _REG(VD2_IF0_LUMA_Y0));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_LUMA_Y0));
writel_relaxed(priv->viu.vd1_if0_luma_y0,
- priv->io_base + _REG(VD2_IF0_LUMA_Y1));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_LUMA_Y1));
writel_relaxed(priv->viu.vd1_if0_chroma_x0,
- priv->io_base + _REG(VD1_IF0_CHROMA_X0));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_CHROMA_X0));
writel_relaxed(priv->viu.vd1_if0_chroma_x0,
- priv->io_base + _REG(VD1_IF0_CHROMA_X1));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_CHROMA_X1));
writel_relaxed(priv->viu.vd1_if0_chroma_x0,
- priv->io_base + _REG(VD2_IF0_CHROMA_X0));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_CHROMA_X0));
writel_relaxed(priv->viu.vd1_if0_chroma_x0,
- priv->io_base + _REG(VD2_IF0_CHROMA_X1));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_CHROMA_X1));
writel_relaxed(priv->viu.vd1_if0_chroma_y0,
- priv->io_base + _REG(VD1_IF0_CHROMA_Y0));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_CHROMA_Y0));
writel_relaxed(priv->viu.vd1_if0_chroma_y0,
- priv->io_base + _REG(VD1_IF0_CHROMA_Y1));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_CHROMA_Y1));
writel_relaxed(priv->viu.vd1_if0_chroma_y0,
- priv->io_base + _REG(VD2_IF0_CHROMA_Y0));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_CHROMA_Y0));
writel_relaxed(priv->viu.vd1_if0_chroma_y0,
- priv->io_base + _REG(VD2_IF0_CHROMA_Y1));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_CHROMA_Y1));
writel_relaxed(priv->viu.vd1_if0_repeat_loop,
- priv->io_base + _REG(VD1_IF0_RPT_LOOP));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_RPT_LOOP));
writel_relaxed(priv->viu.vd1_if0_repeat_loop,
- priv->io_base + _REG(VD2_IF0_RPT_LOOP));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_RPT_LOOP));
writel_relaxed(priv->viu.vd1_if0_luma0_rpt_pat,
- priv->io_base + _REG(VD1_IF0_LUMA0_RPT_PAT));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_LUMA0_RPT_PAT));
writel_relaxed(priv->viu.vd1_if0_luma0_rpt_pat,
- priv->io_base + _REG(VD2_IF0_LUMA0_RPT_PAT));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_LUMA0_RPT_PAT));
writel_relaxed(priv->viu.vd1_if0_luma0_rpt_pat,
- priv->io_base + _REG(VD1_IF0_LUMA1_RPT_PAT));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_LUMA1_RPT_PAT));
writel_relaxed(priv->viu.vd1_if0_luma0_rpt_pat,
- priv->io_base + _REG(VD2_IF0_LUMA1_RPT_PAT));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_LUMA1_RPT_PAT));
writel_relaxed(priv->viu.vd1_if0_chroma0_rpt_pat,
- priv->io_base + _REG(VD1_IF0_CHROMA0_RPT_PAT));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_CHROMA0_RPT_PAT));
writel_relaxed(priv->viu.vd1_if0_chroma0_rpt_pat,
- priv->io_base + _REG(VD2_IF0_CHROMA0_RPT_PAT));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_CHROMA0_RPT_PAT));
writel_relaxed(priv->viu.vd1_if0_chroma0_rpt_pat,
- priv->io_base + _REG(VD1_IF0_CHROMA1_RPT_PAT));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_CHROMA1_RPT_PAT));
writel_relaxed(priv->viu.vd1_if0_chroma0_rpt_pat,
- priv->io_base + _REG(VD2_IF0_CHROMA1_RPT_PAT));
- writel_relaxed(0, priv->io_base + _REG(VD1_IF0_LUMA_PSEL));
- writel_relaxed(0, priv->io_base + _REG(VD1_IF0_CHROMA_PSEL));
- writel_relaxed(0, priv->io_base + _REG(VD2_IF0_LUMA_PSEL));
- writel_relaxed(0, priv->io_base + _REG(VD2_IF0_CHROMA_PSEL));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_CHROMA1_RPT_PAT));
+ writel_relaxed(0, priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_LUMA_PSEL));
+ writel_relaxed(0, priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_CHROMA_PSEL));
+ writel_relaxed(0, priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_LUMA_PSEL));
+ writel_relaxed(0, priv->io_base + meson_crtc->viu_offset +
+ _REG(VD2_IF0_CHROMA_PSEL));
writel_relaxed(priv->viu.vd1_range_map_y,
- priv->io_base + _REG(VD1_IF0_RANGE_MAP_Y));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_RANGE_MAP_Y));
writel_relaxed(priv->viu.vd1_range_map_cb,
- priv->io_base + _REG(VD1_IF0_RANGE_MAP_CB));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_RANGE_MAP_CB));
writel_relaxed(priv->viu.vd1_range_map_cr,
- priv->io_base + _REG(VD1_IF0_RANGE_MAP_CR));
+ priv->io_base + meson_crtc->viu_offset +
+ _REG(VD1_IF0_RANGE_MAP_CR));
writel_relaxed(0x78404,
priv->io_base + _REG(VPP_SC_MISC));
writel_relaxed(priv->viu.vpp_pic_in_height,
writel_relaxed(0x42, priv->io_base + _REG(VPP_SCALE_COEF_IDX));
/* Enable VD1 */
- writel_bits_relaxed(VPP_VD1_PREBLEND | VPP_VD1_POSTBLEND |
- VPP_COLOR_MNG_ENABLE,
- VPP_VD1_PREBLEND | VPP_VD1_POSTBLEND |
- VPP_COLOR_MNG_ENABLE,
- priv->io_base + _REG(VPP_MISC));
+ if (meson_crtc->enable_vd1)
+ meson_crtc->enable_vd1(priv);
priv->viu.vd1_commit = false;
}
return ret;
}
- drm_crtc_helper_add(crtc, &meson_crtc_helper_funcs);
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ meson_crtc->enable_osd1 = meson_g12a_crtc_enable_osd1;
+ meson_crtc->enable_vd1 = meson_g12a_crtc_enable_vd1;
+ meson_crtc->viu_offset = MESON_G12A_VIU_OFFSET;
+ drm_crtc_helper_add(crtc, &meson_g12a_crtc_helper_funcs);
+ } else {
+ meson_crtc->enable_osd1 = meson_crtc_enable_osd1;
+ meson_crtc->enable_vd1 = meson_crtc_enable_vd1;
+ drm_crtc_helper_add(crtc, &meson_crtc_helper_funcs);
+ }
priv->crtc = crtc;
#include "meson_vpp.h"
#include "meson_viu.h"
#include "meson_venc.h"
-#include "meson_canvas.h"
#include "meson_registers.h"
#define DRIVER_NAME "meson"
return IRQ_HANDLED;
}
+static int meson_dumb_create(struct drm_file *file, struct drm_device *dev,
+ struct drm_mode_create_dumb *args)
+{
+ /*
+ * We need 64bytes aligned stride, and PAGE aligned size
+ */
+ args->pitch = ALIGN(DIV_ROUND_UP(args->width * args->bpp, 8), SZ_64);
+ args->size = PAGE_ALIGN(args->pitch * args->height);
+
+ return drm_gem_cma_dumb_create_internal(file, dev, args);
+}
+
DEFINE_DRM_GEM_CMA_FOPS(fops);
static struct drm_driver meson_driver = {
.gem_prime_mmap = drm_gem_cma_prime_mmap,
/* GEM Ops */
- .dumb_create = drm_gem_cma_dumb_create,
+ .dumb_create = meson_dumb_create,
.gem_free_object_unlocked = drm_gem_cma_free_object,
.gem_vm_ops = &drm_gem_cma_vm_ops,
}
priv->canvas = meson_canvas_get(dev);
- if (!IS_ERR(priv->canvas)) {
- ret = meson_canvas_alloc(priv->canvas, &priv->canvas_id_osd1);
- if (ret)
- goto free_drm;
- ret = meson_canvas_alloc(priv->canvas, &priv->canvas_id_vd1_0);
- if (ret) {
- meson_canvas_free(priv->canvas, priv->canvas_id_osd1);
- goto free_drm;
- }
- ret = meson_canvas_alloc(priv->canvas, &priv->canvas_id_vd1_1);
- if (ret) {
- meson_canvas_free(priv->canvas, priv->canvas_id_osd1);
- meson_canvas_free(priv->canvas, priv->canvas_id_vd1_0);
- goto free_drm;
- }
- ret = meson_canvas_alloc(priv->canvas, &priv->canvas_id_vd1_2);
- if (ret) {
- meson_canvas_free(priv->canvas, priv->canvas_id_osd1);
- meson_canvas_free(priv->canvas, priv->canvas_id_vd1_0);
- meson_canvas_free(priv->canvas, priv->canvas_id_vd1_1);
- goto free_drm;
- }
- } else {
- priv->canvas = NULL;
-
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "dmc");
- if (!res) {
- ret = -EINVAL;
- goto free_drm;
- }
- /* Simply ioremap since it may be a shared register zone */
- regs = devm_ioremap(dev, res->start, resource_size(res));
- if (!regs) {
- ret = -EADDRNOTAVAIL;
- goto free_drm;
- }
+ if (IS_ERR(priv->canvas)) {
+ ret = PTR_ERR(priv->canvas);
+ goto free_drm;
+ }
- priv->dmc = devm_regmap_init_mmio(dev, regs,
- &meson_regmap_config);
- if (IS_ERR(priv->dmc)) {
- dev_err(&pdev->dev, "Couldn't create the DMC regmap\n");
- ret = PTR_ERR(priv->dmc);
- goto free_drm;
- }
+ ret = meson_canvas_alloc(priv->canvas, &priv->canvas_id_osd1);
+ if (ret)
+ goto free_drm;
+ ret = meson_canvas_alloc(priv->canvas, &priv->canvas_id_vd1_0);
+ if (ret) {
+ meson_canvas_free(priv->canvas, priv->canvas_id_osd1);
+ goto free_drm;
+ }
+ ret = meson_canvas_alloc(priv->canvas, &priv->canvas_id_vd1_1);
+ if (ret) {
+ meson_canvas_free(priv->canvas, priv->canvas_id_osd1);
+ meson_canvas_free(priv->canvas, priv->canvas_id_vd1_0);
+ goto free_drm;
+ }
+ ret = meson_canvas_alloc(priv->canvas, &priv->canvas_id_vd1_2);
+ if (ret) {
+ meson_canvas_free(priv->canvas, priv->canvas_id_osd1);
+ meson_canvas_free(priv->canvas, priv->canvas_id_vd1_0);
+ meson_canvas_free(priv->canvas, priv->canvas_id_vd1_1);
+ goto free_drm;
}
priv->vsync_irq = platform_get_irq(pdev, 0);
{ .compatible = "amlogic,meson-gxbb-vpu" },
{ .compatible = "amlogic,meson-gxl-vpu" },
{ .compatible = "amlogic,meson-gxm-vpu" },
+ { .compatible = "amlogic,meson-g12a-vpu" },
{}
};
MODULE_DEVICE_TABLE(of, dt_match);
struct device *dev;
void __iomem *io_base;
struct regmap *hhi;
- struct regmap *dmc;
int vsync_irq;
struct meson_canvas *canvas;
uint32_t osd_sc_h_phase_step;
uint32_t osd_sc_h_ctrl0;
uint32_t osd_sc_v_ctrl0;
+ uint32_t osd_blend_din0_scope_h;
+ uint32_t osd_blend_din0_scope_v;
+ uint32_t osb_blend0_size;
+ uint32_t osb_blend1_size;
bool vd1_enabled;
bool vd1_commit;
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/component.h>
+#include <linux/of_device.h>
#include <linux/of_graph.h>
#include <linux/reset.h>
#include <linux/clk.h>
#define HDMITX_TOP_ADDR_REG 0x0
#define HDMITX_TOP_DATA_REG 0x4
#define HDMITX_TOP_CTRL_REG 0x8
+#define HDMITX_TOP_G12A_OFFSET 0x8000
/* Controller Communication Channel */
#define HDMITX_DWC_ADDR_REG 0x10
#define HHI_HDMI_PHY_CNTL1 0x3a4 /* 0xe9 */
#define HHI_HDMI_PHY_CNTL2 0x3a8 /* 0xea */
#define HHI_HDMI_PHY_CNTL3 0x3ac /* 0xeb */
+#define HHI_HDMI_PHY_CNTL4 0x3b0 /* 0xec */
+#define HHI_HDMI_PHY_CNTL5 0x3b4 /* 0xed */
static DEFINE_SPINLOCK(reg_lock);
MESON_VENC_SOURCE_ENCP = 2,
};
+struct meson_dw_hdmi;
+
+struct meson_dw_hdmi_data {
+ unsigned int (*top_read)(struct meson_dw_hdmi *dw_hdmi,
+ unsigned int addr);
+ void (*top_write)(struct meson_dw_hdmi *dw_hdmi,
+ unsigned int addr, unsigned int data);
+ unsigned int (*dwc_read)(struct meson_dw_hdmi *dw_hdmi,
+ unsigned int addr);
+ void (*dwc_write)(struct meson_dw_hdmi *dw_hdmi,
+ unsigned int addr, unsigned int data);
+};
+
struct meson_dw_hdmi {
struct drm_encoder encoder;
struct dw_hdmi_plat_data dw_plat_data;
struct meson_drm *priv;
struct device *dev;
void __iomem *hdmitx;
+ const struct meson_dw_hdmi_data *data;
struct reset_control *hdmitx_apb;
struct reset_control *hdmitx_ctrl;
struct reset_control *hdmitx_phy;
return data;
}
+static unsigned int dw_hdmi_g12a_top_read(struct meson_dw_hdmi *dw_hdmi,
+ unsigned int addr)
+{
+ return readl(dw_hdmi->hdmitx + HDMITX_TOP_G12A_OFFSET + (addr << 2));
+}
+
static inline void dw_hdmi_top_write(struct meson_dw_hdmi *dw_hdmi,
unsigned int addr, unsigned int data)
{
spin_unlock_irqrestore(®_lock, flags);
}
+static inline void dw_hdmi_g12a_top_write(struct meson_dw_hdmi *dw_hdmi,
+ unsigned int addr, unsigned int data)
+{
+ writel(data, dw_hdmi->hdmitx + HDMITX_TOP_G12A_OFFSET + (addr << 2));
+}
+
/* Helper to change specific bits in PHY registers */
static inline void dw_hdmi_top_write_bits(struct meson_dw_hdmi *dw_hdmi,
unsigned int addr,
unsigned int mask,
unsigned int val)
{
- unsigned int data = dw_hdmi_top_read(dw_hdmi, addr);
+ unsigned int data = dw_hdmi->data->top_read(dw_hdmi, addr);
data &= ~mask;
data |= val;
- dw_hdmi_top_write(dw_hdmi, addr, data);
+ dw_hdmi->data->top_write(dw_hdmi, addr, data);
}
static unsigned int dw_hdmi_dwc_read(struct meson_dw_hdmi *dw_hdmi,
return data;
}
+static unsigned int dw_hdmi_g12a_dwc_read(struct meson_dw_hdmi *dw_hdmi,
+ unsigned int addr)
+{
+ return readb(dw_hdmi->hdmitx + addr);
+}
+
static inline void dw_hdmi_dwc_write(struct meson_dw_hdmi *dw_hdmi,
unsigned int addr, unsigned int data)
{
spin_unlock_irqrestore(®_lock, flags);
}
+static inline void dw_hdmi_g12a_dwc_write(struct meson_dw_hdmi *dw_hdmi,
+ unsigned int addr, unsigned int data)
+{
+ writeb(data, dw_hdmi->hdmitx + addr);
+}
+
/* Helper to change specific bits in controller registers */
static inline void dw_hdmi_dwc_write_bits(struct meson_dw_hdmi *dw_hdmi,
unsigned int addr,
unsigned int mask,
unsigned int val)
{
- unsigned int data = dw_hdmi_dwc_read(dw_hdmi, addr);
+ unsigned int data = dw_hdmi->data->dwc_read(dw_hdmi, addr);
data &= ~mask;
data |= val;
- dw_hdmi_dwc_write(dw_hdmi, addr, data);
+ dw_hdmi->data->dwc_write(dw_hdmi, addr, data);
}
/* Bridge */
regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL0, 0x33632122);
regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL3, 0x2000115b);
}
+ } else if (dw_hdmi_is_compatible(dw_hdmi,
+ "amlogic,meson-g12a-dw-hdmi")) {
+ if (pixel_clock >= 371250) {
+ /* 5.94Gbps, 3.7125Gbps */
+ regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL0, 0x37eb65c4);
+ regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL3, 0x2ab0ff3b);
+ regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL5, 0x0000080b);
+ } else if (pixel_clock >= 297000) {
+ /* 2.97Gbps */
+ regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL0, 0x33eb6262);
+ regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL3, 0x2ab0ff3b);
+ regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL5, 0x00000003);
+ } else {
+ /* 1.485Gbps, and below */
+ regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL0, 0x33eb4242);
+ regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL3, 0x2ab0ff3b);
+ regmap_write(priv->hhi, HHI_HDMI_PHY_CNTL5, 0x00000003);
+ }
}
}
regmap_update_bits(priv->hhi, HHI_MEM_PD_REG0, 0xff << 8, 0);
/* Bring out of reset */
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_SW_RESET, 0);
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_SW_RESET, 0);
/* Enable internal pixclk, tmds_clk, spdif_clk, i2s_clk, cecclk */
dw_hdmi_top_write_bits(dw_hdmi, HDMITX_TOP_CLK_CNTL,
0x3 << 4, 0x3 << 4);
/* Enable normal output to PHY */
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_BIST_CNTL, BIT(12));
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_BIST_CNTL, BIT(12));
/* TMDS pattern setup (TOFIX Handle the YUV420 case) */
if (mode->clock > 340000) {
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_01, 0);
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_23,
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_01,
+ 0);
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_23,
0x03ff03ff);
} else {
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_01,
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_01,
0x001f001f);
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_23,
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_23,
0x001f001f);
}
/* Load TMDS pattern */
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_CNTL, 0x1);
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_CNTL, 0x1);
msleep(20);
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_CNTL, 0x2);
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_TMDS_CLK_PTTN_CNTL, 0x2);
/* Setup PHY parameters */
meson_hdmi_phy_setup_mode(dw_hdmi, mode);
/* BIT_INVERT */
if (dw_hdmi_is_compatible(dw_hdmi, "amlogic,meson-gxl-dw-hdmi") ||
- dw_hdmi_is_compatible(dw_hdmi, "amlogic,meson-gxm-dw-hdmi"))
+ dw_hdmi_is_compatible(dw_hdmi, "amlogic,meson-gxm-dw-hdmi") ||
+ dw_hdmi_is_compatible(dw_hdmi, "amlogic,meson-g12a-dw-hdmi"))
regmap_update_bits(priv->hhi, HHI_HDMI_PHY_CNTL1,
BIT(17), 0);
else
{
struct meson_dw_hdmi *dw_hdmi = (struct meson_dw_hdmi *)data;
- return !!dw_hdmi_top_read(dw_hdmi, HDMITX_TOP_STAT0) ?
+ return !!dw_hdmi->data->top_read(dw_hdmi, HDMITX_TOP_STAT0) ?
connector_status_connected : connector_status_disconnected;
}
struct meson_dw_hdmi *dw_hdmi = (struct meson_dw_hdmi *)data;
/* Setup HPD Filter */
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_HPD_FILTER,
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_HPD_FILTER,
(0xa << 12) | 0xa0);
/* Clear interrupts */
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_INTR_STAT_CLR,
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_INTR_STAT_CLR,
HDMITX_TOP_INTR_HPD_RISE | HDMITX_TOP_INTR_HPD_FALL);
/* Unmask interrupts */
struct meson_dw_hdmi *dw_hdmi = dev_id;
u32 stat;
- stat = dw_hdmi_top_read(dw_hdmi, HDMITX_TOP_INTR_STAT);
- dw_hdmi_top_write(dw_hdmi, HDMITX_TOP_INTR_STAT_CLR, stat);
+ stat = dw_hdmi->data->top_read(dw_hdmi, HDMITX_TOP_INTR_STAT);
+ dw_hdmi->data->top_write(dw_hdmi, HDMITX_TOP_INTR_STAT_CLR, stat);
/* HPD Events, handle in the threaded interrupt handler */
if (stat & (HDMITX_TOP_INTR_HPD_RISE | HDMITX_TOP_INTR_HPD_FALL)) {
static int meson_dw_hdmi_reg_read(void *context, unsigned int reg,
unsigned int *result)
{
- *result = dw_hdmi_dwc_read(context, reg);
+ struct meson_dw_hdmi *dw_hdmi = context;
+
+ *result = dw_hdmi->data->dwc_read(dw_hdmi, reg);
return 0;
static int meson_dw_hdmi_reg_write(void *context, unsigned int reg,
unsigned int val)
{
- dw_hdmi_dwc_write(context, reg, val);
+ struct meson_dw_hdmi *dw_hdmi = context;
+
+ dw_hdmi->data->dwc_write(dw_hdmi, reg, val);
return 0;
}
.fast_io = true,
};
+static const struct meson_dw_hdmi_data meson_dw_hdmi_gx_data = {
+ .top_read = dw_hdmi_top_read,
+ .top_write = dw_hdmi_top_write,
+ .dwc_read = dw_hdmi_dwc_read,
+ .dwc_write = dw_hdmi_dwc_write,
+};
+
+static const struct meson_dw_hdmi_data meson_dw_hdmi_g12a_data = {
+ .top_read = dw_hdmi_g12a_top_read,
+ .top_write = dw_hdmi_g12a_top_write,
+ .dwc_read = dw_hdmi_g12a_dwc_read,
+ .dwc_write = dw_hdmi_g12a_dwc_write,
+};
+
static bool meson_hdmi_connector_is_available(struct device *dev)
{
struct device_node *ep, *remote;
void *data)
{
struct platform_device *pdev = to_platform_device(dev);
+ const struct meson_dw_hdmi_data *match;
struct meson_dw_hdmi *meson_dw_hdmi;
struct drm_device *drm = data;
struct meson_drm *priv = drm->dev_private;
return -ENODEV;
}
+ match = of_device_get_match_data(&pdev->dev);
+ if (!match) {
+ dev_err(&pdev->dev, "failed to get match data\n");
+ return -ENODEV;
+ }
+
meson_dw_hdmi = devm_kzalloc(dev, sizeof(*meson_dw_hdmi),
GFP_KERNEL);
if (!meson_dw_hdmi)
meson_dw_hdmi->priv = priv;
meson_dw_hdmi->dev = dev;
+ meson_dw_hdmi->data = match;
dw_plat_data = &meson_dw_hdmi->dw_plat_data;
encoder = &meson_dw_hdmi->encoder;
reset_control_reset(meson_dw_hdmi->hdmitx_phy);
/* Enable APB3 fail on error */
- writel_bits_relaxed(BIT(15), BIT(15),
- meson_dw_hdmi->hdmitx + HDMITX_TOP_CTRL_REG);
- writel_bits_relaxed(BIT(15), BIT(15),
- meson_dw_hdmi->hdmitx + HDMITX_DWC_CTRL_REG);
+ if (!meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ writel_bits_relaxed(BIT(15), BIT(15),
+ meson_dw_hdmi->hdmitx + HDMITX_TOP_CTRL_REG);
+ writel_bits_relaxed(BIT(15), BIT(15),
+ meson_dw_hdmi->hdmitx + HDMITX_DWC_CTRL_REG);
+ }
/* Bring out of reset */
- dw_hdmi_top_write(meson_dw_hdmi, HDMITX_TOP_SW_RESET, 0);
+ meson_dw_hdmi->data->top_write(meson_dw_hdmi,
+ HDMITX_TOP_SW_RESET, 0);
msleep(20);
- dw_hdmi_top_write(meson_dw_hdmi, HDMITX_TOP_CLK_CNTL, 0xff);
+ meson_dw_hdmi->data->top_write(meson_dw_hdmi,
+ HDMITX_TOP_CLK_CNTL, 0xff);
/* Enable HDMI-TX Interrupt */
- dw_hdmi_top_write(meson_dw_hdmi, HDMITX_TOP_INTR_STAT_CLR,
- HDMITX_TOP_INTR_CORE);
+ meson_dw_hdmi->data->top_write(meson_dw_hdmi, HDMITX_TOP_INTR_STAT_CLR,
+ HDMITX_TOP_INTR_CORE);
- dw_hdmi_top_write(meson_dw_hdmi, HDMITX_TOP_INTR_MASKN,
- HDMITX_TOP_INTR_CORE);
+ meson_dw_hdmi->data->top_write(meson_dw_hdmi, HDMITX_TOP_INTR_MASKN,
+ HDMITX_TOP_INTR_CORE);
/* Bridge / Connector */
}
static const struct of_device_id meson_dw_hdmi_of_table[] = {
- { .compatible = "amlogic,meson-gxbb-dw-hdmi" },
- { .compatible = "amlogic,meson-gxl-dw-hdmi" },
- { .compatible = "amlogic,meson-gxm-dw-hdmi" },
+ { .compatible = "amlogic,meson-gxbb-dw-hdmi",
+ .data = &meson_dw_hdmi_gx_data },
+ { .compatible = "amlogic,meson-gxl-dw-hdmi",
+ .data = &meson_dw_hdmi_gx_data },
+ { .compatible = "amlogic,meson-gxm-dw-hdmi",
+ .data = &meson_dw_hdmi_gx_data },
+ { .compatible = "amlogic,meson-g12a-dw-hdmi",
+ .data = &meson_dw_hdmi_g12a_data },
{ }
};
MODULE_DEVICE_TABLE(of, meson_dw_hdmi_of_table);
#define __MESON_DW_HDMI_H
/*
- * Bit 7 RW Reserved. Default 1.
- * Bit 6 RW Reserved. Default 1.
- * Bit 5 RW Reserved. Default 1.
+ * Bit 15-10: RW Reserved. Default 1 starting from G12A
+ * Bit 9 RW sw_reset_i2c starting from G12A
+ * Bit 8 RW sw_reset_axiarb starting from G12A
+ * Bit 7 RW Reserved. Default 1, sw_reset_emp starting from G12A
+ * Bit 6 RW Reserved. Default 1, sw_reset_flt starting from G12A
+ * Bit 5 RW Reserved. Default 1, sw_reset_hdcp22 starting from G12A
* Bit 4 RW sw_reset_phyif: PHY interface. 1=Apply reset; 0=Release from reset.
* Default 1.
* Bit 3 RW sw_reset_intr: interrupt module. 1=Apply reset;
#define HDMITX_TOP_SW_RESET (0x000)
/*
+ * Bit 31 RW free_clk_en: 0=Enable clock gating for power saving; 1= Disable
* Bit 12 RW i2s_ws_inv:1=Invert i2s_ws; 0=No invert. Default 0.
* Bit 11 RW i2s_clk_inv: 1=Invert i2s_clk; 0=No invert. Default 0.
* Bit 10 RW spdif_clk_inv: 1=Invert spdif_clk; 0=No invert. Default 0.
* Bit 9 RW tmds_clk_inv: 1=Invert tmds_clk; 0=No invert. Default 0.
* Bit 8 RW pixel_clk_inv: 1=Invert pixel_clk; 0=No invert. Default 0.
- * Bit 4 RW cec_clk_en: 1=enable cec_clk; 0=disable. Default 0.
+ * Bit 7 RW hdcp22_skpclk_en: starting from G12A, 1=enable; 0=disable
+ * Bit 6 RW hdcp22_esmclk_en: starting from G12A, 1=enable; 0=disable
+ * Bit 5 RW hdcp22_tmdsclk_en: starting from G12A, 1=enable; 0=disable
+ * Bit 4 RW cec_clk_en: 1=enable cec_clk; 0=disable. Default 0. Reserved for G12A
* Bit 3 RW i2s_clk_en: 1=enable i2s_clk; 0=disable. Default 0.
* Bit 2 RW spdif_clk_en: 1=enable spdif_clk; 0=disable. Default 0.
* Bit 1 RW tmds_clk_en: 1=enable tmds_clk; 0=disable. Default 0.
#define HDMITX_TOP_CLK_CNTL (0x001)
/*
+ * Bit 31:28 RW rxsense_glitch_width: starting from G12A
+ * Bit 27:16 RW rxsense_valid_width: starting from G12A
* Bit 11: 0 RW hpd_valid_width: filter out width <= M*1024. Default 0.
* Bit 15:12 RW hpd_glitch_width: filter out glitch <= N. Default 0.
*/
/*
* intr_maskn: MASK_N, one bit per interrupt source.
* 1=Enable interrupt source; 0=Disable interrupt source. Default 0.
+ * [ 7] rxsense_fall starting from G12A
+ * [ 6] rxsense_rise starting from G12A
+ * [ 5] err_i2c_timeout starting from G12A
* [ 4] hdcp22_rndnum_err
* [ 3] nonce_rfrsh_rise
* [ 2] hpd_fall_intr
* Bit 30: 0 RW intr_stat: For each bit, write 1 to manually set the interrupt
* bit, read back the interrupt status.
* Bit 31 R IP interrupt status
+ * Bit 7 RW rxsense_fall starting from G12A
+ * Bit 6 RW rxsense_rise starting from G12A
+ * Bit 5 RW err_i2c_timeout starting from G12A
* Bit 2 RW hpd_fall
* Bit 1 RW hpd_rise
* Bit 0 RW IP interrupt
#define HDMITX_TOP_INTR_STAT (0x004)
/*
+ * [7] rxsense_fall starting from G12A
+ * [6] rxsense_rise starting from G12A
+ * [5] err_i2c_timeout starting from G12A
* [4] hdcp22_rndnum_err
* [3] nonce_rfrsh_rise
* [2] hpd_fall
#define HDMITX_TOP_INTR_CORE BIT(0)
#define HDMITX_TOP_INTR_HPD_RISE BIT(1)
#define HDMITX_TOP_INTR_HPD_FALL BIT(2)
+#define HDMITX_TOP_INTR_RXSENSE_RISE BIT(6)
+#define HDMITX_TOP_INTR_RXSENSE_FALL BIT(7)
/* Bit 14:12 RW tmds_sel: 3'b000=Output zero; 3'b001=Output normal TMDS data;
* 3'b010=Output PRBS data; 3'b100=Output shift pattern. Default 0.
*/
#define HDMITX_TOP_REVOCMEM_STAT (0x00D)
-/* Bit 0 R filtered HPD status. */
+/* Bit 1 R filtered RxSense status
+ * Bit 0 R filtered HPD status.
+ */
#define HDMITX_TOP_STAT0 (0x00E)
#endif /* __MESON_DW_HDMI_H */
#include "meson_overlay.h"
#include "meson_vpp.h"
#include "meson_viu.h"
-#include "meson_canvas.h"
#include "meson_registers.h"
/* VD1_IF0_GEN_REG */
DRM_DEBUG_DRIVER("\n");
- /* Fallback is canvas provider is not available */
- if (!priv->canvas) {
- priv->canvas_id_vd1_0 = MESON_CANVAS_ID_VD1_0;
- priv->canvas_id_vd1_1 = MESON_CANVAS_ID_VD1_1;
- priv->canvas_id_vd1_2 = MESON_CANVAS_ID_VD1_2;
- }
-
interlace_mode = state->crtc->mode.flags & DRM_MODE_FLAG_INTERLACE;
spin_lock_irqsave(&priv->drm->event_lock, flags);
priv->viu.vd1_enabled = false;
/* Disable VD1 */
- writel_bits_relaxed(VPP_VD1_POSTBLEND | VPP_VD1_PREBLEND, 0,
- priv->io_base + _REG(VPP_MISC));
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ writel_relaxed(0, priv->io_base + _REG(VD1_BLEND_SRC_CTRL));
+ writel_relaxed(0, priv->io_base + _REG(VD2_BLEND_SRC_CTRL));
+ writel_relaxed(0, priv->io_base + _REG(VD1_IF0_GEN_REG + 0x17b0));
+ writel_relaxed(0, priv->io_base + _REG(VD2_IF0_GEN_REG + 0x17b0));
+ } else
+ writel_bits_relaxed(VPP_VD1_POSTBLEND | VPP_VD1_PREBLEND, 0,
+ priv->io_base + _REG(VPP_MISC));
}
#include "meson_plane.h"
#include "meson_vpp.h"
#include "meson_viu.h"
-#include "meson_canvas.h"
#include "meson_registers.h"
/* OSD_SCI_WH_M1 */
(0xFF << OSD_GLOBAL_ALPHA_SHIFT) |
OSD_BLK0_ENABLE;
- if (priv->canvas)
- canvas_id_osd1 = priv->canvas_id_osd1;
- else
- canvas_id_osd1 = MESON_CANVAS_ID_OSD1;
+ canvas_id_osd1 = priv->canvas_id_osd1;
/* Set up BLK0 to point to the right canvas */
priv->viu.osd1_blk0_cfg[0] = ((canvas_id_osd1 << OSD_CANVAS_SEL) |
priv->viu.osd1_blk0_cfg[3] = ((dest.x2 - 1) << 16) | dest.x1;
priv->viu.osd1_blk0_cfg[4] = ((dest.y2 - 1) << 16) | dest.y1;
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ priv->viu.osd_blend_din0_scope_h = ((dest.x2 - 1) << 16) | dest.x1;
+ priv->viu.osd_blend_din0_scope_v = ((dest.y2 - 1) << 16) | dest.y1;
+ priv->viu.osb_blend0_size = dst_h << 16 | dst_w;
+ priv->viu.osb_blend1_size = dst_h << 16 | dst_w;
+ }
+
/* Update Canvas with buffer address */
gem = drm_fb_cma_get_gem_obj(fb, 0);
struct meson_drm *priv = meson_plane->priv;
/* Disable OSD1 */
- writel_bits_relaxed(VPP_OSD1_POSTBLEND, 0,
- priv->io_base + _REG(VPP_MISC));
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu"))
+ writel_bits_relaxed(BIT(0) | BIT(21), 0,
+ priv->io_base + _REG(VIU_OSD1_CTRL_STAT));
+ else
+ writel_bits_relaxed(VPP_OSD1_POSTBLEND, 0,
+ priv->io_base + _REG(VPP_MISC));
meson_plane->enabled = false;
#define VIU_OSD2_FIFO_CTRL_STAT 0x1a4b
#define VIU_OSD2_TEST_RDDATA 0x1a4c
#define VIU_OSD2_PROT_CTRL 0x1a4e
+#define VIU_OSD2_MALI_UNPACK_CTRL 0x1abd
+#define VIU_OSD2_DIMM_CTRL 0x1acf
+
+#define VIU_OSD3_CTRL_STAT 0x3d80
+#define VIU_OSD3_CTRL_STAT2 0x3d81
+#define VIU_OSD3_COLOR_ADDR 0x3d82
+#define VIU_OSD3_COLOR 0x3d83
+#define VIU_OSD3_TCOLOR_AG0 0x3d84
+#define VIU_OSD3_TCOLOR_AG1 0x3d85
+#define VIU_OSD3_TCOLOR_AG2 0x3d86
+#define VIU_OSD3_TCOLOR_AG3 0x3d87
+#define VIU_OSD3_BLK0_CFG_W0 0x3d88
+#define VIU_OSD3_BLK0_CFG_W1 0x3d8c
+#define VIU_OSD3_BLK0_CFG_W2 0x3d90
+#define VIU_OSD3_BLK0_CFG_W3 0x3d94
+#define VIU_OSD3_BLK0_CFG_W4 0x3d98
+#define VIU_OSD3_BLK1_CFG_W4 0x3d99
+#define VIU_OSD3_BLK2_CFG_W4 0x3d9a
+#define VIU_OSD3_FIFO_CTRL_STAT 0x3d9c
+#define VIU_OSD3_TEST_RDDATA 0x3d9d
+#define VIU_OSD3_PROT_CTRL 0x3d9e
+#define VIU_OSD3_MALI_UNPACK_CTRL 0x3d9f
+#define VIU_OSD3_DIMM_CTRL 0x3da0
#define VD1_IF0_GEN_REG 0x1a50
#define VD1_IF0_CANVAS0 0x1a51
#define VIU_OSD1_MATRIX_COEF31_32 0x1a9e
#define VIU_OSD1_MATRIX_COEF40_41 0x1a9f
#define VD1_IF0_GEN_REG3 0x1aa7
+
+#define VIU_OSD_BLENDO_H_START_END 0x1aa9
+#define VIU_OSD_BLENDO_V_START_END 0x1aaa
+#define VIU_OSD_BLEND_GEN_CTRL0 0x1aab
+#define VIU_OSD_BLEND_GEN_CTRL1 0x1aac
+#define VIU_OSD_BLEND_DUMMY_DATA 0x1aad
+#define VIU_OSD_BLEND_CURRENT_XY 0x1aae
+
+#define VIU_OSD2_MATRIX_CTRL 0x1ab0
+#define VIU_OSD2_MATRIX_COEF00_01 0x1ab1
+#define VIU_OSD2_MATRIX_COEF02_10 0x1ab2
+#define VIU_OSD2_MATRIX_COEF11_12 0x1ab3
+#define VIU_OSD2_MATRIX_COEF20_21 0x1ab4
+#define VIU_OSD2_MATRIX_COEF22 0x1ab5
+#define VIU_OSD2_MATRIX_OFFSET0_1 0x1ab6
+#define VIU_OSD2_MATRIX_OFFSET2 0x1ab7
+#define VIU_OSD2_MATRIX_PRE_OFFSET0_1 0x1ab8
+#define VIU_OSD2_MATRIX_PRE_OFFSET2 0x1ab9
+#define VIU_OSD2_MATRIX_PROBE_COLOR 0x1aba
+#define VIU_OSD2_MATRIX_HL_COLOR 0x1abb
+#define VIU_OSD2_MATRIX_PROBE_POS 0x1abc
#define VIU_OSD1_EOTF_CTL 0x1ad4
#define VIU_OSD1_EOTF_COEF00_01 0x1ad5
#define VIU_OSD1_EOTF_COEF02_10 0x1ad6
#define VPP_OSD_SCALE_COEF 0x1dcd
#define VPP_INT_LINE_NUM 0x1dce
+#define VPP_WRAP_OSD1_MATRIX_COEF00_01 0x3d60
+#define VPP_WRAP_OSD1_MATRIX_COEF02_10 0x3d61
+#define VPP_WRAP_OSD1_MATRIX_COEF11_12 0x3d62
+#define VPP_WRAP_OSD1_MATRIX_COEF20_21 0x3d63
+#define VPP_WRAP_OSD1_MATRIX_COEF22 0x3d64
+#define VPP_WRAP_OSD1_MATRIX_COEF13_14 0x3d65
+#define VPP_WRAP_OSD1_MATRIX_COEF23_24 0x3d66
+#define VPP_WRAP_OSD1_MATRIX_COEF15_25 0x3d67
+#define VPP_WRAP_OSD1_MATRIX_CLIP 0x3d68
+#define VPP_WRAP_OSD1_MATRIX_OFFSET0_1 0x3d69
+#define VPP_WRAP_OSD1_MATRIX_OFFSET2 0x3d6a
+#define VPP_WRAP_OSD1_MATRIX_PRE_OFFSET0_1 0x3d6b
+#define VPP_WRAP_OSD1_MATRIX_PRE_OFFSET2 0x3d6c
+#define VPP_WRAP_OSD1_MATRIX_EN_CTRL 0x3d6d
+
+#define VPP_WRAP_OSD2_MATRIX_COEF00_01 0x3d70
+#define VPP_WRAP_OSD2_MATRIX_COEF02_10 0x3d71
+#define VPP_WRAP_OSD2_MATRIX_COEF11_12 0x3d72
+#define VPP_WRAP_OSD2_MATRIX_COEF20_21 0x3d73
+#define VPP_WRAP_OSD2_MATRIX_COEF22 0x3d74
+#define VPP_WRAP_OSD2_MATRIX_COEF13_14 0x3d75
+#define VPP_WRAP_OSD2_MATRIX_COEF23_24 0x3d76
+#define VPP_WRAP_OSD2_MATRIX_COEF15_25 0x3d77
+#define VPP_WRAP_OSD2_MATRIX_CLIP 0x3d78
+#define VPP_WRAP_OSD2_MATRIX_OFFSET0_1 0x3d79
+#define VPP_WRAP_OSD2_MATRIX_OFFSET2 0x3d7a
+#define VPP_WRAP_OSD2_MATRIX_PRE_OFFSET0_1 0x3d7b
+#define VPP_WRAP_OSD2_MATRIX_PRE_OFFSET2 0x3d7c
+#define VPP_WRAP_OSD2_MATRIX_EN_CTRL 0x3d7d
+
+#define VPP_WRAP_OSD3_MATRIX_COEF00_01 0x3db0
+#define VPP_WRAP_OSD3_MATRIX_COEF02_10 0x3db1
+#define VPP_WRAP_OSD3_MATRIX_COEF11_12 0x3db2
+#define VPP_WRAP_OSD3_MATRIX_COEF20_21 0x3db3
+#define VPP_WRAP_OSD3_MATRIX_COEF22 0x3db4
+#define VPP_WRAP_OSD3_MATRIX_COEF13_14 0x3db5
+#define VPP_WRAP_OSD3_MATRIX_COEF23_24 0x3db6
+#define VPP_WRAP_OSD3_MATRIX_COEF15_25 0x3db7
+#define VPP_WRAP_OSD3_MATRIX_CLIP 0x3db8
+#define VPP_WRAP_OSD3_MATRIX_OFFSET0_1 0x3db9
+#define VPP_WRAP_OSD3_MATRIX_OFFSET2 0x3dba
+#define VPP_WRAP_OSD3_MATRIX_PRE_OFFSET0_1 0x3dbb
+#define VPP_WRAP_OSD3_MATRIX_PRE_OFFSET2 0x3dbc
+#define VPP_WRAP_OSD3_MATRIX_EN_CTRL 0x3dbd
+
+/* osd2 scaler */
+#define OSD2_VSC_PHASE_STEP 0x3d00
+#define OSD2_VSC_INI_PHASE 0x3d01
+#define OSD2_VSC_CTRL0 0x3d02
+#define OSD2_HSC_PHASE_STEP 0x3d03
+#define OSD2_HSC_INI_PHASE 0x3d04
+#define OSD2_HSC_CTRL0 0x3d05
+#define OSD2_HSC_INI_PAT_CTRL 0x3d06
+#define OSD2_SC_DUMMY_DATA 0x3d07
+#define OSD2_SC_CTRL0 0x3d08
+#define OSD2_SCI_WH_M1 0x3d09
+#define OSD2_SCO_H_START_END 0x3d0a
+#define OSD2_SCO_V_START_END 0x3d0b
+#define OSD2_SCALE_COEF_IDX 0x3d18
+#define OSD2_SCALE_COEF 0x3d19
+
+/* osd34 scaler */
+#define OSD34_SCALE_COEF_IDX 0x3d1e
+#define OSD34_SCALE_COEF 0x3d1f
+#define OSD34_VSC_PHASE_STEP 0x3d20
+#define OSD34_VSC_INI_PHASE 0x3d21
+#define OSD34_VSC_CTRL0 0x3d22
+#define OSD34_HSC_PHASE_STEP 0x3d23
+#define OSD34_HSC_INI_PHASE 0x3d24
+#define OSD34_HSC_CTRL0 0x3d25
+#define OSD34_HSC_INI_PAT_CTRL 0x3d26
+#define OSD34_SC_DUMMY_DATA 0x3d27
+#define OSD34_SC_CTRL0 0x3d28
+#define OSD34_SCI_WH_M1 0x3d29
+#define OSD34_SCO_H_START_END 0x3d2a
+#define OSD34_SCO_V_START_END 0x3d2b
/* viu2 */
#define VIU2_ADDR_START 0x1e00
#define VIU2_ADDR_END 0x1eff
#define OSDSR_YBIC_VCOEF0 0x3149
#define OSDSR_CBIC_VCOEF0 0x314a
+/* osd afbcd on gxtvbb */
+#define OSD1_AFBCD_ENABLE 0x31a0
+#define OSD1_AFBCD_MODE 0x31a1
+#define OSD1_AFBCD_SIZE_IN 0x31a2
+#define OSD1_AFBCD_HDR_PTR 0x31a3
+#define OSD1_AFBCD_FRAME_PTR 0x31a4
+#define OSD1_AFBCD_CHROMA_PTR 0x31a5
+#define OSD1_AFBCD_CONV_CTRL 0x31a6
+#define OSD1_AFBCD_STATUS 0x31a8
+#define OSD1_AFBCD_PIXEL_HSCOPE 0x31a9
+#define OSD1_AFBCD_PIXEL_VSCOPE 0x31aa
+#define VIU_MISC_CTRL1 0x1a07
+
+/* add for gxm and 962e dv core2 */
+#define DOLBY_CORE2A_SWAP_CTRL1 0x3434
+#define DOLBY_CORE2A_SWAP_CTRL2 0x3435
+
+/* osd afbc on g12a */
+#define VPU_MAFBC_BLOCK_ID 0x3a00
+#define VPU_MAFBC_IRQ_RAW_STATUS 0x3a01
+#define VPU_MAFBC_IRQ_CLEAR 0x3a02
+#define VPU_MAFBC_IRQ_MASK 0x3a03
+#define VPU_MAFBC_IRQ_STATUS 0x3a04
+#define VPU_MAFBC_COMMAND 0x3a05
+#define VPU_MAFBC_STATUS 0x3a06
+#define VPU_MAFBC_SURFACE_CFG 0x3a07
+
+/* osd afbc on g12a */
+#define VPU_MAFBC_HEADER_BUF_ADDR_LOW_S0 0x3a10
+#define VPU_MAFBC_HEADER_BUF_ADDR_HIGH_S0 0x3a11
+#define VPU_MAFBC_FORMAT_SPECIFIER_S0 0x3a12
+#define VPU_MAFBC_BUFFER_WIDTH_S0 0x3a13
+#define VPU_MAFBC_BUFFER_HEIGHT_S0 0x3a14
+#define VPU_MAFBC_BOUNDING_BOX_X_START_S0 0x3a15
+#define VPU_MAFBC_BOUNDING_BOX_X_END_S0 0x3a16
+#define VPU_MAFBC_BOUNDING_BOX_Y_START_S0 0x3a17
+#define VPU_MAFBC_BOUNDING_BOX_Y_END_S0 0x3a18
+#define VPU_MAFBC_OUTPUT_BUF_ADDR_LOW_S0 0x3a19
+#define VPU_MAFBC_OUTPUT_BUF_ADDR_HIGH_S0 0x3a1a
+#define VPU_MAFBC_OUTPUT_BUF_STRIDE_S0 0x3a1b
+#define VPU_MAFBC_PREFETCH_CFG_S0 0x3a1c
+
+#define VPU_MAFBC_HEADER_BUF_ADDR_LOW_S1 0x3a30
+#define VPU_MAFBC_HEADER_BUF_ADDR_HIGH_S1 0x3a31
+#define VPU_MAFBC_FORMAT_SPECIFIER_S1 0x3a32
+#define VPU_MAFBC_BUFFER_WIDTH_S1 0x3a33
+#define VPU_MAFBC_BUFFER_HEIGHT_S1 0x3a34
+#define VPU_MAFBC_BOUNDING_BOX_X_START_S1 0x3a35
+#define VPU_MAFBC_BOUNDING_BOX_X_END_S1 0x3a36
+#define VPU_MAFBC_BOUNDING_BOX_Y_START_S1 0x3a37
+#define VPU_MAFBC_BOUNDING_BOX_Y_END_S1 0x3a38
+#define VPU_MAFBC_OUTPUT_BUF_ADDR_LOW_S1 0x3a39
+#define VPU_MAFBC_OUTPUT_BUF_ADDR_HIGH_S1 0x3a3a
+#define VPU_MAFBC_OUTPUT_BUF_STRIDE_S1 0x3a3b
+#define VPU_MAFBC_PREFETCH_CFG_S1 0x3a3c
+
+#define VPU_MAFBC_HEADER_BUF_ADDR_LOW_S2 0x3a50
+#define VPU_MAFBC_HEADER_BUF_ADDR_HIGH_S2 0x3a51
+#define VPU_MAFBC_FORMAT_SPECIFIER_S2 0x3a52
+#define VPU_MAFBC_BUFFER_WIDTH_S2 0x3a53
+#define VPU_MAFBC_BUFFER_HEIGHT_S2 0x3a54
+#define VPU_MAFBC_BOUNDING_BOX_X_START_S2 0x3a55
+#define VPU_MAFBC_BOUNDING_BOX_X_END_S2 0x3a56
+#define VPU_MAFBC_BOUNDING_BOX_Y_START_S2 0x3a57
+#define VPU_MAFBC_BOUNDING_BOX_Y_END_S2 0x3a58
+#define VPU_MAFBC_OUTPUT_BUF_ADDR_LOW_S2 0x3a59
+#define VPU_MAFBC_OUTPUT_BUF_ADDR_HIGH_S2 0x3a5a
+#define VPU_MAFBC_OUTPUT_BUF_STRIDE_S2 0x3a5b
+#define VPU_MAFBC_PREFETCH_CFG_S2 0x3a5c
+
+#define VPU_MAFBC_HEADER_BUF_ADDR_LOW_S3 0x3a70
+#define VPU_MAFBC_HEADER_BUF_ADDR_HIGH_S3 0x3a71
+#define VPU_MAFBC_FORMAT_SPECIFIER_S3 0x3a72
+#define VPU_MAFBC_BUFFER_WIDTH_S3 0x3a73
+#define VPU_MAFBC_BUFFER_HEIGHT_S3 0x3a74
+#define VPU_MAFBC_BOUNDING_BOX_X_START_S3 0x3a75
+#define VPU_MAFBC_BOUNDING_BOX_X_END_S3 0x3a76
+#define VPU_MAFBC_BOUNDING_BOX_Y_START_S3 0x3a77
+#define VPU_MAFBC_BOUNDING_BOX_Y_END_S3 0x3a78
+#define VPU_MAFBC_OUTPUT_BUF_ADDR_LOW_S3 0x3a79
+#define VPU_MAFBC_OUTPUT_BUF_ADDR_HIGH_S3 0x3a7a
+#define VPU_MAFBC_OUTPUT_BUF_STRIDE_S3 0x3a7b
+#define VPU_MAFBC_PREFETCH_CFG_S3 0x3a7c
+
+#define DOLBY_PATH_CTRL 0x1a0c
+#define OSD_PATH_MISC_CTRL 0x1a0e
+#define MALI_AFBCD_TOP_CTRL 0x1a0f
+
+#define VIU_OSD_BLEND_CTRL 0x39b0
+#define VIU_OSD_BLEND_CTRL1 0x39c0
+#define VIU_OSD_BLEND_DIN0_SCOPE_H 0x39b1
+#define VIU_OSD_BLEND_DIN0_SCOPE_V 0x39b2
+#define VIU_OSD_BLEND_DIN1_SCOPE_H 0x39b3
+#define VIU_OSD_BLEND_DIN1_SCOPE_V 0x39b4
+#define VIU_OSD_BLEND_DIN2_SCOPE_H 0x39b5
+#define VIU_OSD_BLEND_DIN2_SCOPE_V 0x39b6
+#define VIU_OSD_BLEND_DIN3_SCOPE_H 0x39b7
+#define VIU_OSD_BLEND_DIN3_SCOPE_V 0x39b8
+#define VIU_OSD_BLEND_DUMMY_DATA0 0x39b9
+#define VIU_OSD_BLEND_DUMMY_ALPHA 0x39ba
+#define VIU_OSD_BLEND_BLEND0_SIZE 0x39bb
+#define VIU_OSD_BLEND_BLEND1_SIZE 0x39bc
+#define VIU_OSD_BLEND_RO_CURRENT_XY 0x39bf
+
+#define VPP_OUT_H_V_SIZE 0x1da5
+
+#define VPP_VD2_HDR_IN_SIZE 0x1df0
+#define VPP_OSD1_IN_SIZE 0x1df1
+#define VPP_GCLK_CTRL2 0x1df2
+#define VD2_PPS_DUMMY_DATA 0x1df4
+#define VPP_OSD1_BLD_H_SCOPE 0x1df5
+#define VPP_OSD1_BLD_V_SCOPE 0x1df6
+#define VPP_OSD2_BLD_H_SCOPE 0x1df7
+#define VPP_OSD2_BLD_V_SCOPE 0x1df8
+#define VPP_WRBAK_CTRL 0x1df9
+#define VPP_SLEEP_CTRL 0x1dfa
+#define VD1_BLEND_SRC_CTRL 0x1dfb
+#define VD2_BLEND_SRC_CTRL 0x1dfc
+#define OSD1_BLEND_SRC_CTRL 0x1dfd
+#define OSD2_BLEND_SRC_CTRL 0x1dfe
+
+#define VPP_POST_BLEND_BLEND_DUMMY_DATA 0x3968
+#define VPP_POST_BLEND_DUMMY_ALPHA 0x3969
+#define VPP_RDARB_MODE 0x3978
+#define VPP_RDARB_REQEN_SLV 0x3979
+#define VPU_RDARB_MODE_L2C1 0x279d
+
#endif /* __MESON_REGISTERS_H */
#define HHI_HDMI_PLL_CNTL4 0x32C /* 0xcb offset in data sheet */
#define HHI_HDMI_PLL_CNTL5 0x330 /* 0xcc offset in data sheet */
#define HHI_HDMI_PLL_CNTL6 0x334 /* 0xcd offset in data sheet */
+#define HHI_HDMI_PLL_CNTL7 0x338 /* 0xce offset in data sheet */
#define HDMI_PLL_RESET BIT(28)
+#define HDMI_PLL_RESET_G12A BIT(29)
#define HDMI_PLL_LOCK BIT(31)
+#define HDMI_PLL_LOCK_G12A (3 << 30)
#define FREQ_1000_1001(_freq) DIV_ROUND_CLOSEST(_freq * 1000, 1001)
regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL5, 0x71486980);
regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL6, 0x00000e55);
regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL, 0x4800023d);
+
+ /* Poll for lock bit */
+ regmap_read_poll_timeout(priv->hhi, HHI_HDMI_PLL_CNTL, val,
+ (val & HDMI_PLL_LOCK), 10, 0);
} else if (meson_vpu_is_compatible(priv, "amlogic,meson-gxm-vpu") ||
meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu")) {
regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL, 0x4000027b);
HDMI_PLL_RESET, HDMI_PLL_RESET);
regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL,
HDMI_PLL_RESET, 0);
- }
- /* Poll for lock bit */
- regmap_read_poll_timeout(priv->hhi, HHI_HDMI_PLL_CNTL, val,
- (val & HDMI_PLL_LOCK), 10, 0);
+ /* Poll for lock bit */
+ regmap_read_poll_timeout(priv->hhi, HHI_HDMI_PLL_CNTL, val,
+ (val & HDMI_PLL_LOCK), 10, 0);
+ } else if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL, 0x1a0504f7);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL2, 0x00010000);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL3, 0x00000000);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL4, 0x6a28dc00);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL5, 0x65771290);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL6, 0x39272000);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL7, 0x56540000);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL, 0x3a0504f7);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL, 0x1a0504f7);
+
+ /* Poll for lock bit */
+ regmap_read_poll_timeout(priv->hhi, HHI_HDMI_PLL_CNTL, val,
+ ((val & HDMI_PLL_LOCK_G12A) == HDMI_PLL_LOCK_G12A),
+ 10, 0);
+ }
/* Disable VCLK2 */
regmap_update_bits(priv->hhi, HHI_VIID_CLK_CNTL, VCLK2_EN, 0);
VCLK2_DIV_MASK, (55 - 1));
/* select vid_pll for vclk2 */
- regmap_update_bits(priv->hhi, HHI_VIID_CLK_CNTL,
- VCLK2_SEL_MASK, (4 << VCLK2_SEL_SHIFT));
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu"))
+ regmap_update_bits(priv->hhi, HHI_VIID_CLK_CNTL,
+ VCLK2_SEL_MASK, (0 << VCLK2_SEL_SHIFT));
+ else
+ regmap_update_bits(priv->hhi, HHI_VIID_CLK_CNTL,
+ VCLK2_SEL_MASK, (4 << VCLK2_SEL_SHIFT));
+
/* enable vclk2 gate */
regmap_update_bits(priv->hhi, HHI_VIID_CLK_CNTL, VCLK2_EN, VCLK2_EN);
},
[MESON_VCLK_HDMI_297000] = {
.pixel_freq = 297000,
- .pll_base_freq = 2970000,
- .pll_od1 = 1,
+ .pll_base_freq = 5940000,
+ .pll_od1 = 2,
.pll_od2 = 1,
.pll_od3 = 1,
.vid_pll_div = VID_PLL_DIV_5,
/* Poll for lock bit */
regmap_read_poll_timeout(priv->hhi, HHI_HDMI_PLL_CNTL, val,
(val & HDMI_PLL_LOCK), 10, 0);
+ } else if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL, 0x0b3a0400 | m);
+
+ /* Enable and reset */
+ regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL,
+ 0x3 << 28, 0x3 << 28);
+
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL2, frac);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL3, 0x00000000);
+
+ /* G12A HDMI PLL Needs specific parameters for 5.4GHz */
+ if (m >= 0xf7) {
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL4, 0xea68dc00);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL5, 0x65771290);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL6, 0x39272000);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL7, 0x55540000);
+ } else {
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL4, 0x0a691c00);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL5, 0x33771290);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL6, 0x39270000);
+ regmap_write(priv->hhi, HHI_HDMI_PLL_CNTL7, 0x50540000);
+ }
+
+ do {
+ /* Reset PLL */
+ regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL,
+ HDMI_PLL_RESET_G12A, HDMI_PLL_RESET_G12A);
+
+ /* UN-Reset PLL */
+ regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL,
+ HDMI_PLL_RESET_G12A, 0);
+
+ /* Poll for lock bits */
+ if (!regmap_read_poll_timeout(priv->hhi,
+ HHI_HDMI_PLL_CNTL, val,
+ ((val & HDMI_PLL_LOCK_G12A)
+ == HDMI_PLL_LOCK_G12A),
+ 10, 100))
+ break;
+ } while(1);
}
if (meson_vpu_is_compatible(priv, "amlogic,meson-gxbb-vpu"))
regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL2,
3 << 16, pll_od_to_reg(od1) << 16);
else if (meson_vpu_is_compatible(priv, "amlogic,meson-gxm-vpu") ||
- meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu"))
+ meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu"))
regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL3,
3 << 21, pll_od_to_reg(od1) << 21);
+ else if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu"))
+ regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL,
+ 3 << 16, pll_od_to_reg(od1) << 16);
if (meson_vpu_is_compatible(priv, "amlogic,meson-gxbb-vpu"))
regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL2,
3 << 22, pll_od_to_reg(od2) << 22);
else if (meson_vpu_is_compatible(priv, "amlogic,meson-gxm-vpu") ||
- meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu"))
+ meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu"))
regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL3,
3 << 23, pll_od_to_reg(od2) << 23);
+ else if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu"))
+ regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL,
+ 3 << 18, pll_od_to_reg(od2) << 18);
if (meson_vpu_is_compatible(priv, "amlogic,meson-gxbb-vpu"))
regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL2,
3 << 18, pll_od_to_reg(od3) << 18);
else if (meson_vpu_is_compatible(priv, "amlogic,meson-gxm-vpu") ||
- meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu"))
+ meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu"))
regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL3,
3 << 19, pll_od_to_reg(od3) << 19);
-
+ else if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu"))
+ regmap_update_bits(priv->hhi, HHI_HDMI_PLL_CNTL,
+ 3 << 20, pll_od_to_reg(od3) << 20);
}
#define XTAL_FREQ 24000
#define HDMI_FRAC_MAX_GXBB 4096
#define HDMI_FRAC_MAX_GXL 1024
+#define HDMI_FRAC_MAX_G12A 131072
static unsigned int meson_hdmi_pll_get_frac(struct meson_drm *priv,
unsigned int m,
parent_freq *= 2;
}
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu"))
+ frac_max = HDMI_FRAC_MAX_G12A;
+
/* We can have a perfect match !*/
if (pll_freq / m == parent_freq &&
pll_freq % m == 0)
if (frac >= HDMI_FRAC_MAX_GXBB)
return false;
} else if (meson_vpu_is_compatible(priv, "amlogic,meson-gxm-vpu") ||
- meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu")) {
+ meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu") ||
+ meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
/* Empiric supported min/max dividers */
if (m < 106 || m > 247)
return false;
break;
}
+ meson_hdmi_pll_set_params(priv, m, frac, od1, od2, od3);
+ } else if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ switch (pll_base_freq) {
+ case 2970000:
+ m = 0x7b;
+ frac = vic_alternate_clock ? 0x140b4 : 0x18000;
+ break;
+ case 4320000:
+ m = vic_alternate_clock ? 0xb3 : 0xb4;
+ frac = vic_alternate_clock ? 0x1a3ee : 0;
+ break;
+ case 5940000:
+ m = 0xf7;
+ frac = vic_alternate_clock ? 0x8148 : 0x10000;
+ break;
+ }
+
meson_hdmi_pll_set_params(priv, m, frac, od1, od2, od3);
}
/* HHI Registers */
#define HHI_GCLK_MPEG2 0x148 /* 0x52 offset in data sheet */
#define HHI_VDAC_CNTL0 0x2F4 /* 0xbd offset in data sheet */
+#define HHI_VDAC_CNTL0_G12A 0x2EC /* 0xbd offset in data sheet */
#define HHI_VDAC_CNTL1 0x2F8 /* 0xbe offset in data sheet */
+#define HHI_VDAC_CNTL1_G12A 0x2F0 /* 0xbe offset in data sheet */
#define HHI_HDMI_PHY_CNTL0 0x3a0 /* 0xe8 offset in data sheet */
struct meson_cvbs_enci_mode meson_cvbs_enci_pal = {
void meson_venc_init(struct meson_drm *priv)
{
/* Disable CVBS VDAC */
- regmap_write(priv->hhi, HHI_VDAC_CNTL0, 0);
- regmap_write(priv->hhi, HHI_VDAC_CNTL1, 8);
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ regmap_write(priv->hhi, HHI_VDAC_CNTL0_G12A, 0);
+ regmap_write(priv->hhi, HHI_VDAC_CNTL1_G12A, 8);
+ } else {
+ regmap_write(priv->hhi, HHI_VDAC_CNTL0, 0);
+ regmap_write(priv->hhi, HHI_VDAC_CNTL1, 8);
+ }
/* Power Down Dacs */
writel_relaxed(0xff, priv->io_base + _REG(VENC_VDAC_SETTING));
/* HHI VDAC Registers */
#define HHI_VDAC_CNTL0 0x2F4 /* 0xbd offset in data sheet */
+#define HHI_VDAC_CNTL0_G12A 0x2EC /* 0xbd offset in data sheet */
#define HHI_VDAC_CNTL1 0x2F8 /* 0xbe offset in data sheet */
+#define HHI_VDAC_CNTL1_G12A 0x2F0 /* 0xbe offset in data sheet */
struct meson_venc_cvbs {
struct drm_encoder encoder;
struct meson_drm *priv = meson_venc_cvbs->priv;
/* Disable CVBS VDAC */
- regmap_write(priv->hhi, HHI_VDAC_CNTL0, 0);
- regmap_write(priv->hhi, HHI_VDAC_CNTL1, 8);
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ regmap_write(priv->hhi, HHI_VDAC_CNTL0_G12A, 0);
+ regmap_write(priv->hhi, HHI_VDAC_CNTL1_G12A, 0);
+ } else {
+ regmap_write(priv->hhi, HHI_VDAC_CNTL0, 0);
+ regmap_write(priv->hhi, HHI_VDAC_CNTL1, 8);
+ }
}
static void meson_venc_cvbs_encoder_enable(struct drm_encoder *encoder)
/* VDAC0 source is not from ATV */
writel_bits_relaxed(BIT(5), 0, priv->io_base + _REG(VENC_VDAC_DACSEL0));
- if (meson_vpu_is_compatible(priv, "amlogic,meson-gxbb-vpu"))
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-gxbb-vpu")) {
regmap_write(priv->hhi, HHI_VDAC_CNTL0, 1);
- else if (meson_vpu_is_compatible(priv, "amlogic,meson-gxm-vpu") ||
- meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu"))
+ regmap_write(priv->hhi, HHI_VDAC_CNTL1, 0);
+ } else if (meson_vpu_is_compatible(priv, "amlogic,meson-gxm-vpu") ||
+ meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu")) {
regmap_write(priv->hhi, HHI_VDAC_CNTL0, 0xf0001);
-
- regmap_write(priv->hhi, HHI_VDAC_CNTL1, 0);
+ regmap_write(priv->hhi, HHI_VDAC_CNTL1, 0);
+ } else if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ regmap_write(priv->hhi, HHI_VDAC_CNTL0_G12A, 0x906001);
+ regmap_write(priv->hhi, HHI_VDAC_CNTL1_G12A, 0);
+ }
}
static void meson_venc_cvbs_encoder_mode_set(struct drm_encoder *encoder,
#include "meson_viu.h"
#include "meson_vpp.h"
#include "meson_venc.h"
-#include "meson_canvas.h"
#include "meson_registers.h"
/**
EOTF_COEFF_RIGHTSHIFT /* right shift */
};
-void meson_viu_set_osd_matrix(struct meson_drm *priv,
- enum viu_matrix_sel_e m_select,
+static void meson_viu_set_g12a_osd1_matrix(struct meson_drm *priv,
+ int *m, bool csc_on)
+{
+ /* VPP WRAP OSD1 matrix */
+ writel(((m[0] & 0xfff) << 16) | (m[1] & 0xfff),
+ priv->io_base + _REG(VPP_WRAP_OSD1_MATRIX_PRE_OFFSET0_1));
+ writel(m[2] & 0xfff,
+ priv->io_base + _REG(VPP_WRAP_OSD1_MATRIX_PRE_OFFSET2));
+ writel(((m[3] & 0x1fff) << 16) | (m[4] & 0x1fff),
+ priv->io_base + _REG(VPP_WRAP_OSD1_MATRIX_COEF00_01));
+ writel(((m[5] & 0x1fff) << 16) | (m[6] & 0x1fff),
+ priv->io_base + _REG(VPP_WRAP_OSD1_MATRIX_COEF02_10));
+ writel(((m[7] & 0x1fff) << 16) | (m[8] & 0x1fff),
+ priv->io_base + _REG(VPP_WRAP_OSD1_MATRIX_COEF11_12));
+ writel(((m[9] & 0x1fff) << 16) | (m[10] & 0x1fff),
+ priv->io_base + _REG(VPP_WRAP_OSD1_MATRIX_COEF20_21));
+ writel((m[11] & 0x1fff) << 16,
+ priv->io_base + _REG(VPP_WRAP_OSD1_MATRIX_COEF22));
+
+ writel(((m[18] & 0xfff) << 16) | (m[19] & 0xfff),
+ priv->io_base + _REG(VPP_WRAP_OSD1_MATRIX_OFFSET0_1));
+ writel(m[20] & 0xfff,
+ priv->io_base + _REG(VPP_WRAP_OSD1_MATRIX_OFFSET2));
+
+ writel_bits_relaxed(BIT(0), csc_on ? BIT(0) : 0,
+ priv->io_base + _REG(VPP_WRAP_OSD1_MATRIX_EN_CTRL));
+}
+
+static void meson_viu_set_osd_matrix(struct meson_drm *priv,
+ enum viu_matrix_sel_e m_select,
int *m, bool csc_on)
{
if (m_select == VIU_MATRIX_OSD) {
#define OSD_EOTF_LUT_SIZE 33
#define OSD_OETF_LUT_SIZE 41
-void meson_viu_set_osd_lut(struct meson_drm *priv, enum viu_lut_sel_e lut_sel,
- unsigned int *r_map, unsigned int *g_map,
- unsigned int *b_map,
- bool csc_on)
+static void
+meson_viu_set_osd_lut(struct meson_drm *priv, enum viu_lut_sel_e lut_sel,
+ unsigned int *r_map, unsigned int *g_map,
+ unsigned int *b_map, bool csc_on)
{
unsigned int addr_port;
unsigned int data_port;
if (meson_vpu_is_compatible(priv, "amlogic,meson-gxm-vpu") ||
meson_vpu_is_compatible(priv, "amlogic,meson-gxl-vpu"))
meson_viu_load_matrix(priv);
+ else if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu"))
+ meson_viu_set_g12a_osd1_matrix(priv, RGB709_to_YUV709l_coeff,
+ true);
/* Initialize OSD1 fifo control register */
reg = BIT(0) | /* Urgent DDR request priority */
- (4 << 5) | /* hold_fifo_lines */
- (3 << 10) | /* burst length 64 */
- (32 << 12) | /* fifo_depth_val: 32*8=256 */
- (2 << 22) | /* 4 words in 1 burst */
- (2 << 24);
+ (4 << 5); /* hold_fifo_lines */
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu"))
+ reg |= (1 << 10) | /* burst length 32 */
+ (32 << 12) | /* fifo_depth_val: 32*8=256 */
+ (2 << 22) | /* 4 words in 1 burst */
+ (2 << 24) |
+ (1 << 31);
+ else
+ reg |= (3 << 10) | /* burst length 64 */
+ (32 << 12) | /* fifo_depth_val: 32*8=256 */
+ (2 << 22) | /* 4 words in 1 burst */
+ (2 << 24);
writel_relaxed(reg, priv->io_base + _REG(VIU_OSD1_FIFO_CTRL_STAT));
writel_relaxed(reg, priv->io_base + _REG(VIU_OSD2_FIFO_CTRL_STAT));
writel_relaxed(0x00FF00C0,
priv->io_base + _REG(VD2_IF0_LUMA_FIFO_SIZE));
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ writel_relaxed(4 << 29 |
+ 1 << 27 |
+ 1 << 26 | /* blend_din0 input to blend0 */
+ 1 << 25 | /* blend1_dout to blend2 */
+ 1 << 24 | /* blend1_din3 input to blend1 */
+ 1 << 20 |
+ 0 << 16 |
+ 1,
+ priv->io_base + _REG(VIU_OSD_BLEND_CTRL));
+ writel_relaxed(3 << 8 |
+ 1 << 20,
+ priv->io_base + _REG(OSD1_BLEND_SRC_CTRL));
+ writel_relaxed(1 << 20,
+ priv->io_base + _REG(OSD2_BLEND_SRC_CTRL));
+ writel_relaxed(0, priv->io_base + _REG(VD1_BLEND_SRC_CTRL));
+ writel_relaxed(0, priv->io_base + _REG(VD2_BLEND_SRC_CTRL));
+ writel_relaxed(0,
+ priv->io_base + _REG(VIU_OSD_BLEND_DUMMY_DATA0));
+ writel_relaxed(0,
+ priv->io_base + _REG(VIU_OSD_BLEND_DUMMY_ALPHA));
+ writel_bits_relaxed(0x3 << 2, 0x3 << 2,
+ priv->io_base + _REG(DOLBY_PATH_CTRL));
+ }
priv->viu.osd1_enabled = false;
priv->viu.osd1_commit = false;
writel_relaxed(0x20000, priv->io_base + _REG(VPP_DOLBY_CTRL));
writel_relaxed(0x1020080,
priv->io_base + _REG(VPP_DUMMY_DATA1));
- }
+ } else if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu"))
+ writel_relaxed(0xf, priv->io_base + _REG(DOLBY_PATH_CTRL));
/* Initialize vpu fifo control registers */
- writel_relaxed(readl_relaxed(priv->io_base + _REG(VPP_OFIFO_SIZE)) |
- 0x77f, priv->io_base + _REG(VPP_OFIFO_SIZE));
+ if (meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu"))
+ writel_relaxed(0xfff << 20 | 0x1000,
+ priv->io_base + _REG(VPP_OFIFO_SIZE));
+ else
+ writel_relaxed(readl_relaxed(priv->io_base + _REG(VPP_OFIFO_SIZE)) |
+ 0x77f, priv->io_base + _REG(VPP_OFIFO_SIZE));
writel_relaxed(0x08080808, priv->io_base + _REG(VPP_HOLD_LINES));
- /* Turn off preblend */
- writel_bits_relaxed(VPP_PREBLEND_ENABLE, 0,
- priv->io_base + _REG(VPP_MISC));
-
- /* Turn off POSTBLEND */
- writel_bits_relaxed(VPP_POSTBLEND_ENABLE, 0,
- priv->io_base + _REG(VPP_MISC));
-
- /* Force all planes off */
- writel_bits_relaxed(VPP_OSD1_POSTBLEND | VPP_OSD2_POSTBLEND |
- VPP_VD1_POSTBLEND | VPP_VD2_POSTBLEND |
- VPP_VD1_PREBLEND | VPP_VD2_PREBLEND, 0,
- priv->io_base + _REG(VPP_MISC));
-
- /* Setup default VD settings */
- writel_relaxed(4096,
- priv->io_base + _REG(VPP_PREBLEND_VD1_H_START_END));
- writel_relaxed(4096,
- priv->io_base + _REG(VPP_BLEND_VD2_H_START_END));
+ if (!meson_vpu_is_compatible(priv, "amlogic,meson-g12a-vpu")) {
+ /* Turn off preblend */
+ writel_bits_relaxed(VPP_PREBLEND_ENABLE, 0,
+ priv->io_base + _REG(VPP_MISC));
+
+ /* Turn off POSTBLEND */
+ writel_bits_relaxed(VPP_POSTBLEND_ENABLE, 0,
+ priv->io_base + _REG(VPP_MISC));
+
+ /* Force all planes off */
+ writel_bits_relaxed(VPP_OSD1_POSTBLEND | VPP_OSD2_POSTBLEND |
+ VPP_VD1_POSTBLEND | VPP_VD2_POSTBLEND |
+ VPP_VD1_PREBLEND | VPP_VD2_PREBLEND, 0,
+ priv->io_base + _REG(VPP_MISC));
+
+ /* Setup default VD settings */
+ writel_relaxed(4096,
+ priv->io_base + _REG(VPP_PREBLEND_VD1_H_START_END));
+ writel_relaxed(4096,
+ priv->io_base + _REG(VPP_BLEND_VD2_H_START_END));
+ }
/* Disable Scalers */
writel_relaxed(0, priv->io_base + _REG(VPP_OSD_SC_CTRL0));
};
struct mga_fbdev {
- struct drm_fb_helper helper;
+ struct drm_fb_helper helper; /* must be first */
struct mga_framebuffer mfb;
void *sysram;
int size;
struct mga_i2c_chan *mgag200_i2c_create(struct drm_device *dev);
void mgag200_i2c_destroy(struct mga_i2c_chan *i2c);
-#define DRM_FILE_PAGE_OFFSET (0x100000000ULL >> PAGE_SHIFT)
void mgag200_ttm_placement(struct mgag200_bo *bo, int domain);
static inline int mgag200_bo_reserve(struct mgag200_bo *bo, bool no_wait)
goto err_alloc_fbi;
}
- info->par = mfbdev;
-
ret = mgag200_framebuffer_init(dev, &mfbdev->mfb, &mode_cmd, gobj);
if (ret)
goto err_alloc_fbi;
/* setup helper */
mfbdev->helper.fb = fb;
- strcpy(info->fix.id, "mgadrmfb");
-
info->fbops = &mgag200fb_ops;
/* setup aperture base/size for vesafb takeover */
info->apertures->ranges[0].base = mdev->dev->mode_config.fb_base;
info->apertures->ranges[0].size = mdev->mc.vram_size;
- drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(info, &mfbdev->helper, sizes->fb_width,
- sizes->fb_height);
+ drm_fb_helper_fill_info(info, &mfbdev->helper, sizes);
info->screen_base = sysram;
info->screen_size = size;
ret = ttm_bo_device_init(&mdev->ttm.bdev,
&mgag200_bo_driver,
dev->anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET,
true);
if (ret) {
DRM_ERROR("Error initialising bo driver; %d\n", ret);
int mgag200_mmap(struct file *filp, struct vm_area_struct *vma)
{
- struct drm_file *file_priv;
- struct mga_device *mdev;
+ struct drm_file *file_priv = filp->private_data;
+ struct mga_device *mdev = file_priv->minor->dev->dev_private;
- if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET))
- return -EINVAL;
-
- file_priv = filp->private_data;
- mdev = file_priv->minor->dev->dev_private;
return ttm_bo_mmap(filp, vma, &mdev->ttm.bdev);
}
help
DRM/KMS driver for MSM/snapdragon.
+config DRM_MSM_GPU_STATE
+ bool
+ depends on DRM_MSM && (DEBUG_FS || DEV_COREDUMP)
+ default y
+
config DRM_MSM_REGISTER_LOGGING
bool "MSM DRM register logging"
depends on DRM_MSM
# SPDX-License-Identifier: GPL-2.0
-ccflags-y := -Idrivers/gpu/drm/msm
-ccflags-y += -Idrivers/gpu/drm/msm/disp/dpu1
-ccflags-$(CONFIG_DRM_MSM_DSI) += -Idrivers/gpu/drm/msm/dsi
+ccflags-y := -I $(srctree)/$(src)
+ccflags-y += -I $(srctree)/$(src)/disp/dpu1
+ccflags-$(CONFIG_DRM_MSM_DSI) += -I $(srctree)/$(src)/dsi
msm-y := \
adreno/adreno_device.o \
adreno/a6xx_gpu.o \
adreno/a6xx_gmu.o \
adreno/a6xx_hfi.o \
- adreno/a6xx_gpu_state.o \
hdmi/hdmi.o \
hdmi/hdmi_audio.o \
hdmi/hdmi_bridge.o \
msm-$(CONFIG_DEBUG_FS) += adreno/a5xx_debugfs.o
+msm-$(CONFIG_DRM_MSM_GPU_STATE) += adreno/a6xx_gpu_state.o
+
msm-$(CONFIG_DRM_FBDEV_EMULATION) += msm_fbdev.o
msm-$(CONFIG_COMMON_CLK) += disp/mdp4/mdp4_lvds_pll.o
msm-$(CONFIG_COMMON_CLK) += hdmi/hdmi_pll_8960.o
#include <linux/types.h>
#include <linux/cpumask.h>
#include <linux/qcom_scm.h>
-#include <linux/dma-mapping.h>
-#include <linux/of_address.h>
-#include <linux/soc/qcom/mdt_loader.h>
#include <linux/pm_opp.h>
#include <linux/nvmem-consumer.h>
#include <linux/slab.h>
#define GPU_PAS_ID 13
-static int zap_shader_load_mdt(struct msm_gpu *gpu, const char *fwname)
-{
- struct device *dev = &gpu->pdev->dev;
- const struct firmware *fw;
- struct device_node *np;
- struct resource r;
- phys_addr_t mem_phys;
- ssize_t mem_size;
- void *mem_region = NULL;
- int ret;
-
- if (!IS_ENABLED(CONFIG_ARCH_QCOM))
- return -EINVAL;
-
- np = of_get_child_by_name(dev->of_node, "zap-shader");
- if (!np)
- return -ENODEV;
-
- np = of_parse_phandle(np, "memory-region", 0);
- if (!np)
- return -EINVAL;
-
- ret = of_address_to_resource(np, 0, &r);
- if (ret)
- return ret;
-
- mem_phys = r.start;
- mem_size = resource_size(&r);
-
- /* Request the MDT file for the firmware */
- fw = adreno_request_fw(to_adreno_gpu(gpu), fwname);
- if (IS_ERR(fw)) {
- DRM_DEV_ERROR(dev, "Unable to load %s\n", fwname);
- return PTR_ERR(fw);
- }
-
- /* Figure out how much memory we need */
- mem_size = qcom_mdt_get_size(fw);
- if (mem_size < 0) {
- ret = mem_size;
- goto out;
- }
-
- /* Allocate memory for the firmware image */
- mem_region = memremap(mem_phys, mem_size, MEMREMAP_WC);
- if (!mem_region) {
- ret = -ENOMEM;
- goto out;
- }
-
- /*
- * Load the rest of the MDT
- *
- * Note that we could be dealing with two different paths, since
- * with upstream linux-firmware it would be in a qcom/ subdir..
- * adreno_request_fw() handles this, but qcom_mdt_load() does
- * not. But since we've already gotten thru adreno_request_fw()
- * we know which of the two cases it is:
- */
- if (to_adreno_gpu(gpu)->fwloc == FW_LOCATION_LEGACY) {
- ret = qcom_mdt_load(dev, fw, fwname, GPU_PAS_ID,
- mem_region, mem_phys, mem_size, NULL);
- } else {
- char *newname;
-
- newname = kasprintf(GFP_KERNEL, "qcom/%s", fwname);
-
- ret = qcom_mdt_load(dev, fw, newname, GPU_PAS_ID,
- mem_region, mem_phys, mem_size, NULL);
- kfree(newname);
- }
- if (ret)
- goto out;
-
- /* Send the image to the secure world */
- ret = qcom_scm_pas_auth_and_reset(GPU_PAS_ID);
- if (ret)
- DRM_DEV_ERROR(dev, "Unable to authorize the image\n");
-
-out:
- if (mem_region)
- memunmap(mem_region);
-
- release_firmware(fw);
-
- return ret;
-}
-
static void a5xx_flush(struct msm_gpu *gpu, struct msm_ringbuffer *ring)
{
struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu);
static int a5xx_zap_shader_init(struct msm_gpu *gpu)
{
static bool loaded;
- struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu);
- struct platform_device *pdev = gpu->pdev;
int ret;
/*
if (loaded)
return a5xx_zap_shader_resume(gpu);
- /* We need SCM to be able to load the firmware */
- if (!qcom_scm_is_available()) {
- DRM_DEV_ERROR(&pdev->dev, "SCM is not available\n");
- return -EPROBE_DEFER;
- }
-
- /* Each GPU has a target specific zap shader firmware name to use */
- if (!adreno_gpu->info->zapfw) {
- DRM_DEV_ERROR(&pdev->dev,
- "Zap shader firmware file not specified for this target\n");
- return -ENODEV;
- }
-
- ret = zap_shader_load_mdt(gpu, adreno_gpu->info->zapfw);
+ ret = adreno_zap_shader_load(gpu, GPU_PAS_ID);
loaded = !ret;
-
return ret;
}
#include <linux/clk.h>
#include <linux/interconnect.h>
+#include <linux/pm_domain.h>
#include <linux/pm_opp.h>
#include <soc/qcom/cmd-db.h>
#include "a6xx_gpu.h"
#include "a6xx_gmu.xml.h"
+static void a6xx_gmu_fault(struct a6xx_gmu *gmu)
+{
+ struct a6xx_gpu *a6xx_gpu = container_of(gmu, struct a6xx_gpu, gmu);
+ struct adreno_gpu *adreno_gpu = &a6xx_gpu->base;
+ struct msm_gpu *gpu = &adreno_gpu->base;
+ struct drm_device *dev = gpu->dev;
+ struct msm_drm_private *priv = dev->dev_private;
+
+ /* FIXME: add a banner here */
+ gmu->hung = true;
+
+ /* Turn off the hangcheck timer while we are resetting */
+ del_timer(&gpu->hangcheck_timer);
+
+ /* Queue the GPU handler because we need to treat this as a recovery */
+ queue_work(priv->wq, &gpu->recover_work);
+}
+
static irqreturn_t a6xx_gmu_irq(int irq, void *data)
{
struct a6xx_gmu *gmu = data;
if (status & A6XX_GMU_AO_HOST_INTERRUPT_STATUS_WDOG_BITE) {
dev_err_ratelimited(gmu->dev, "GMU watchdog expired\n");
- /* Temporary until we can recover safely */
- BUG();
+ a6xx_gmu_fault(gmu);
}
if (status & A6XX_GMU_AO_HOST_INTERRUPT_STATUS_HOST_AHB_BUS_ERROR)
if (status & A6XX_GMU_GMU2HOST_INTR_INFO_CM3_FAULT) {
dev_err_ratelimited(gmu->dev, "GMU firmware fault\n");
- /* Temporary until we can recover safely */
- BUG();
+ a6xx_gmu_fault(gmu);
}
return IRQ_HANDLED;
}
/* Wait for the GMU to get to its most idle state */
-int a6xx_gmu_wait_for_idle(struct a6xx_gpu *a6xx_gpu)
+int a6xx_gmu_wait_for_idle(struct a6xx_gmu *gmu)
{
- struct a6xx_gmu *gmu = &a6xx_gpu->gmu;
-
return spin_until(a6xx_gmu_check_idle_level(gmu));
}
if (!rpmh_init) {
a6xx_gmu_rpmh_init(gmu);
rpmh_init = true;
- } else if (state != GMU_RESET) {
+ } else {
ret = a6xx_rpmh_start(gmu);
if (ret)
return ret;
A6XX_GMU_AO_HOST_INTERRUPT_STATUS_HOST_AHB_BUS_ERROR | \
A6XX_GMU_AO_HOST_INTERRUPT_STATUS_FENCE_ERR)
-static void a6xx_gmu_irq_enable(struct a6xx_gmu *gmu)
-{
- gmu_write(gmu, REG_A6XX_GMU_AO_HOST_INTERRUPT_CLR, ~0);
- gmu_write(gmu, REG_A6XX_GMU_GMU2HOST_INTR_CLR, ~0);
-
- gmu_write(gmu, REG_A6XX_GMU_AO_HOST_INTERRUPT_MASK,
- ~A6XX_GMU_IRQ_MASK);
- gmu_write(gmu, REG_A6XX_GMU_GMU2HOST_INTR_MASK,
- ~A6XX_HFI_IRQ_MASK);
-
- enable_irq(gmu->gmu_irq);
- enable_irq(gmu->hfi_irq);
-}
-
static void a6xx_gmu_irq_disable(struct a6xx_gmu *gmu)
{
disable_irq(gmu->gmu_irq);
gmu_write(gmu, REG_A6XX_GMU_GMU2HOST_INTR_MASK, ~0);
}
-int a6xx_gmu_reset(struct a6xx_gpu *a6xx_gpu)
+static void a6xx_gmu_rpmh_off(struct a6xx_gmu *gmu)
{
- struct a6xx_gmu *gmu = &a6xx_gpu->gmu;
- int ret;
u32 val;
- /* Flush all the queues */
- a6xx_hfi_stop(gmu);
-
- /* Stop the interrupts */
- a6xx_gmu_irq_disable(gmu);
-
- /* Force off SPTP in case the GMU is managing it */
- a6xx_sptprac_disable(gmu);
-
/* Make sure there are no outstanding RPMh votes */
gmu_poll_timeout(gmu, REG_A6XX_RSCC_TCS0_DRV0_STATUS, val,
(val & 1), 100, 10000);
(val & 1), 100, 10000);
gmu_poll_timeout(gmu, REG_A6XX_RSCC_TCS3_DRV0_STATUS, val,
(val & 1), 100, 1000);
+}
- /* Force off the GX GSDC */
- regulator_force_disable(gmu->gx);
-
- /* Disable the resources */
- clk_bulk_disable_unprepare(gmu->nr_clocks, gmu->clocks);
- pm_runtime_put_sync(gmu->dev);
-
- /* Re-enable the resources */
- pm_runtime_get_sync(gmu->dev);
-
- /* Use a known rate to bring up the GMU */
- clk_set_rate(gmu->core_clk, 200000000);
- ret = clk_bulk_prepare_enable(gmu->nr_clocks, gmu->clocks);
- if (ret)
- goto out;
-
- a6xx_gmu_irq_enable(gmu);
-
- ret = a6xx_gmu_fw_start(gmu, GMU_RESET);
- if (!ret)
- ret = a6xx_hfi_start(gmu, GMU_COLD_BOOT);
+/* Force the GMU off in case it isn't responsive */
+static void a6xx_gmu_force_off(struct a6xx_gmu *gmu)
+{
+ /* Flush all the queues */
+ a6xx_hfi_stop(gmu);
- /* Set the GPU back to the highest power frequency */
- __a6xx_gmu_set_freq(gmu, gmu->nr_gpu_freqs - 1);
+ /* Stop the interrupts */
+ a6xx_gmu_irq_disable(gmu);
-out:
- if (ret)
- a6xx_gmu_clear_oob(gmu, GMU_OOB_BOOT_SLUMBER);
+ /* Force off SPTP in case the GMU is managing it */
+ a6xx_sptprac_disable(gmu);
- return ret;
+ /* Make sure there are no outstanding RPMh votes */
+ a6xx_gmu_rpmh_off(gmu);
}
int a6xx_gmu_resume(struct a6xx_gpu *a6xx_gpu)
if (WARN(!gmu->mmio, "The GMU is not set up yet\n"))
return 0;
+ gmu->hung = false;
+
/* Turn on the resources */
pm_runtime_get_sync(gmu->dev);
/* Use a known rate to bring up the GMU */
clk_set_rate(gmu->core_clk, 200000000);
ret = clk_bulk_prepare_enable(gmu->nr_clocks, gmu->clocks);
- if (ret)
- goto out;
+ if (ret) {
+ pm_runtime_put(gmu->dev);
+ return ret;
+ }
/* Set the bus quota to a reasonable value for boot */
icc_set_bw(gpu->icc_path, 0, MBps_to_icc(3072));
- a6xx_gmu_irq_enable(gmu);
+ /* Enable the GMU interrupt */
+ gmu_write(gmu, REG_A6XX_GMU_AO_HOST_INTERRUPT_CLR, ~0);
+ gmu_write(gmu, REG_A6XX_GMU_AO_HOST_INTERRUPT_MASK, ~A6XX_GMU_IRQ_MASK);
+ enable_irq(gmu->gmu_irq);
/* Check to see if we are doing a cold or warm boot */
status = gmu_read(gmu, REG_A6XX_GMU_GENERAL_7) == 1 ?
goto out;
ret = a6xx_hfi_start(gmu, status);
+ if (ret)
+ goto out;
+
+ /*
+ * Turn on the GMU firmware fault interrupt after we know the boot
+ * sequence is successful
+ */
+ gmu_write(gmu, REG_A6XX_GMU_GMU2HOST_INTR_CLR, ~0);
+ gmu_write(gmu, REG_A6XX_GMU_GMU2HOST_INTR_MASK, ~A6XX_HFI_IRQ_MASK);
+ enable_irq(gmu->hfi_irq);
/* Set the GPU to the highest power frequency */
__a6xx_gmu_set_freq(gmu, gmu->nr_gpu_freqs - 1);
+ /*
+ * "enable" the GX power domain which won't actually do anything but it
+ * will make sure that the refcounting is correct in case we need to
+ * bring down the GX after a GMU failure
+ */
+ if (!IS_ERR(gmu->gxpd))
+ pm_runtime_get(gmu->gxpd);
+
out:
- /* Make sure to turn off the boot OOB request on error */
- if (ret)
- a6xx_gmu_clear_oob(gmu, GMU_OOB_BOOT_SLUMBER);
+ /* On failure, shut down the GMU to leave it in a good state */
+ if (ret) {
+ disable_irq(gmu->gmu_irq);
+ a6xx_rpmh_stop(gmu);
+ pm_runtime_put(gmu->dev);
+ }
return ret;
}
return true;
}
-int a6xx_gmu_stop(struct a6xx_gpu *a6xx_gpu)
+/* Gracefully try to shut down the GMU and by extension the GPU */
+static void a6xx_gmu_shutdown(struct a6xx_gmu *gmu)
{
+ struct a6xx_gpu *a6xx_gpu = container_of(gmu, struct a6xx_gpu, gmu);
struct adreno_gpu *adreno_gpu = &a6xx_gpu->base;
struct msm_gpu *gpu = &adreno_gpu->base;
- struct a6xx_gmu *gmu = &a6xx_gpu->gmu;
u32 val;
/*
val = gmu_read(gmu, REG_A6XX_GPU_GMU_CX_GMU_RPMH_POWER_STATE);
if (val != 0xf) {
- int ret = a6xx_gmu_wait_for_idle(a6xx_gpu);
+ int ret = a6xx_gmu_wait_for_idle(gmu);
- /* Temporary until we can recover safely */
- BUG_ON(ret);
+ /* If the GMU isn't responding assume it is hung */
+ if (ret) {
+ a6xx_gmu_force_off(gmu);
+ return;
+ }
+
+ /* Clear the VBIF pipe before shutting down */
+ gpu_write(gpu, REG_A6XX_VBIF_XIN_HALT_CTRL0, 0xf);
+ spin_until((gpu_read(gpu, REG_A6XX_VBIF_XIN_HALT_CTRL1) & 0xf)
+ == 0xf);
+ gpu_write(gpu, REG_A6XX_VBIF_XIN_HALT_CTRL0, 0);
/* tell the GMU we want to slumber */
a6xx_gmu_notify_slumber(gmu);
/* Tell RPMh to power off the GPU */
a6xx_rpmh_stop(gmu);
+}
+
+
+int a6xx_gmu_stop(struct a6xx_gpu *a6xx_gpu)
+{
+ struct a6xx_gmu *gmu = &a6xx_gpu->gmu;
+ struct msm_gpu *gpu = &a6xx_gpu->base.base;
+
+ if (!pm_runtime_active(gmu->dev))
+ return 0;
+
+ /*
+ * Force the GMU off if we detected a hang, otherwise try to shut it
+ * down gracefully
+ */
+ if (gmu->hung)
+ a6xx_gmu_force_off(gmu);
+ else
+ a6xx_gmu_shutdown(gmu);
/* Remove the bus vote */
icc_set_bw(gpu->icc_path, 0, 0);
+ /*
+ * Make sure the GX domain is off before turning off the GMU (CX)
+ * domain. Usually the GMU does this but only if the shutdown sequence
+ * was successful
+ */
+ if (!IS_ERR(gmu->gxpd))
+ pm_runtime_put_sync(gmu->gxpd);
+
clk_bulk_disable_unprepare(gmu->nr_clocks, gmu->clocks);
pm_runtime_put_sync(gmu->dev);
}
/* Return the 'arc-level' for the given frequency */
-static u32 a6xx_gmu_get_arc_level(struct device *dev, unsigned long freq)
+static unsigned int a6xx_gmu_get_arc_level(struct device *dev,
+ unsigned long freq)
{
struct dev_pm_opp *opp;
- struct device_node *np;
- u32 val = 0;
+ unsigned int val;
if (!freq)
return 0;
- opp = dev_pm_opp_find_freq_exact(dev, freq, true);
+ opp = dev_pm_opp_find_freq_exact(dev, freq, true);
if (IS_ERR(opp))
return 0;
- np = dev_pm_opp_get_of_node(opp);
-
- if (np) {
- of_property_read_u32(np, "opp-level", &val);
- of_node_put(np);
- }
+ val = dev_pm_opp_get_level(opp);
dev_pm_opp_put(opp);
/* Construct a vote for each frequency */
for (i = 0; i < freqs_count; i++) {
u8 pindex = 0, sindex = 0;
- u32 level = a6xx_gmu_get_arc_level(dev, freqs[i]);
+ unsigned int level = a6xx_gmu_get_arc_level(dev, freqs[i]);
/* Get the primary index that matches the arc level */
for (j = 0; j < pri_count; j++) {
if (IS_ERR_OR_NULL(gmu->mmio))
return;
- pm_runtime_disable(gmu->dev);
a6xx_gmu_stop(a6xx_gpu);
+ pm_runtime_disable(gmu->dev);
+
+ if (!IS_ERR(gmu->gxpd)) {
+ pm_runtime_disable(gmu->gxpd);
+ dev_pm_domain_detach(gmu->gxpd, false);
+ }
+
a6xx_gmu_irq_disable(gmu);
a6xx_gmu_memory_free(gmu, gmu->hfi);
gmu->idle_level = GMU_IDLE_STATE_ACTIVE;
pm_runtime_enable(gmu->dev);
- gmu->gx = devm_regulator_get(gmu->dev, "vdd");
/* Get the list of clocks */
ret = a6xx_gmu_clocks_probe(gmu);
if (gmu->hfi_irq < 0 || gmu->gmu_irq < 0)
goto err;
+ /*
+ * Get a link to the GX power domain to reset the GPU in case of GMU
+ * crash
+ */
+ gmu->gxpd = dev_pm_domain_attach_by_name(gmu->dev, "gx");
+
/* Get the power levels for the GMU and GPU */
a6xx_gmu_pwrlevels_probe(gmu);
/* the GMU is coming up for the first time or back from a power collapse */
#define GMU_COLD_BOOT 1
-/* The GMU is being soft reset after a fault */
-#define GMU_RESET 2
-
/*
* These define the level of control that the GMU has - the higher the number
* the more things that the GMU hardware controls on its own.
int hfi_irq;
int gmu_irq;
- struct regulator *gx;
-
struct iommu_domain *domain;
u64 uncached_iova_base;
+ struct device *gxpd;
+
int idle_level;
struct a6xx_gmu_bo *hfi;
struct a6xx_hfi_queue queues[2];
- struct tasklet_struct hfi_tasklet;
+ bool hung;
};
static inline u32 gmu_read(struct a6xx_gmu *gmu, u32 offset)
#include <linux/devfreq.h>
+#define GPU_PAS_ID 13
+
static inline bool _a6xx_check_idle(struct msm_gpu *gpu)
{
struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu);
return 0;
}
+static int a6xx_zap_shader_init(struct msm_gpu *gpu)
+{
+ static bool loaded;
+ int ret;
+
+ if (loaded)
+ return 0;
+
+ ret = adreno_zap_shader_load(gpu, GPU_PAS_ID);
+
+ loaded = !ret;
+ return ret;
+}
+
#define A6XX_INT_MASK (A6XX_RBBM_INT_0_MASK_CP_AHB_ERROR | \
A6XX_RBBM_INT_0_MASK_RBBM_ATB_ASYNCFIFO_OVERFLOW | \
A6XX_RBBM_INT_0_MASK_CP_HW_ERROR | \
if (ret)
goto out;
- gpu_write(gpu, REG_A6XX_RBBM_SECVID_TRUST_CNTL, 0x0);
+ /*
+ * Try to load a zap shader into the secure world. If successful
+ * we can use the CP to switch out of secure mode. If not then we
+ * have no resource but to try to switch ourselves out manually. If we
+ * guessed wrong then access to the RBBM_SECVID_TRUST_CNTL register will
+ * be blocked and a permissions violation will soon follow.
+ */
+ ret = a6xx_zap_shader_init(gpu);
+ if (!ret) {
+ OUT_PKT7(gpu->rb[0], CP_SET_SECURE_MODE, 1);
+ OUT_RING(gpu->rb[0], 0x00000000);
+
+ a6xx_flush(gpu, gpu->rb[0]);
+ if (!a6xx_idle(gpu, gpu->rb[0]))
+ return -EINVAL;
+ } else {
+ /* Print a warning so if we die, we know why */
+ dev_warn_once(gpu->dev->dev,
+ "Zap shader not enabled - using SECVID_TRUST_CNTL instead\n");
+ gpu_write(gpu, REG_A6XX_RBBM_SECVID_TRUST_CNTL, 0x0);
+ ret = 0;
+ }
out:
/*
struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu);
int ret;
- ret = a6xx_gmu_resume(a6xx_gpu);
-
gpu->needs_hw_init = true;
+ ret = a6xx_gmu_resume(a6xx_gpu);
+ if (ret)
+ return ret;
+
msm_gpu_resume_devfreq(gpu);
- return ret;
+ return 0;
}
static int a6xx_pm_suspend(struct msm_gpu *gpu)
devfreq_suspend_device(gpu->devfreq.devfreq);
- /*
- * Make sure the GMU is idle before continuing (because some transitions
- * may use VBIF
- */
- a6xx_gmu_wait_for_idle(a6xx_gpu);
-
- /* Clear the VBIF pipe before shutting down */
- /* FIXME: This accesses the GPU - do we need to make sure it is on? */
- gpu_write(gpu, REG_A6XX_VBIF_XIN_HALT_CTRL0, 0xf);
- spin_until((gpu_read(gpu, REG_A6XX_VBIF_XIN_HALT_CTRL1) & 0xf) == 0xf);
- gpu_write(gpu, REG_A6XX_VBIF_XIN_HALT_CTRL0, 0);
-
return a6xx_gmu_stop(a6xx_gpu);
}
.active_ring = a6xx_active_ring,
.irq = a6xx_irq,
.destroy = a6xx_destroy,
-#if defined(CONFIG_DEBUG_FS) || defined(CONFIG_DEV_COREDUMP)
+#if defined(CONFIG_DRM_MSM_GPU_STATE)
.show = a6xx_show,
#endif
.gpu_busy = a6xx_gpu_busy,
.gpu_get_freq = a6xx_gmu_get_freq,
.gpu_set_freq = a6xx_gmu_set_freq,
+#if defined(CONFIG_DRM_MSM_GPU_STATE)
.gpu_state_get = a6xx_gpu_state_get,
.gpu_state_put = a6xx_gpu_state_put,
+#endif
},
.get_timestamp = a6xx_get_timestamp,
};
int a6xx_gmu_resume(struct a6xx_gpu *gpu);
int a6xx_gmu_stop(struct a6xx_gpu *gpu);
-int a6xx_gmu_wait_for_idle(struct a6xx_gpu *gpu);
+int a6xx_gmu_wait_for_idle(struct a6xx_gmu *gmu);
-int a6xx_gmu_reset(struct a6xx_gpu *a6xx_gpu);
bool a6xx_gmu_isidle(struct a6xx_gmu *gmu);
int a6xx_gmu_set_oob(struct a6xx_gmu *gmu, enum a6xx_gmu_oob_state state);
.gmem = SZ_1M,
.inactive_period = DRM_MSM_INACTIVE_PERIOD,
.init = a6xx_gpu_init,
+ .zapfw = "a630_zap.mdt",
},
};
ret = pm_runtime_get_sync(&pdev->dev);
if (ret < 0) {
+ pm_runtime_put_sync(&pdev->dev);
DRM_DEV_ERROR(dev->dev, "Couldn't power up the GPU: %d\n", ret);
return NULL;
}
#include <linux/ascii85.h>
#include <linux/interconnect.h>
+#include <linux/qcom_scm.h>
#include <linux/kernel.h>
+#include <linux/of_address.h>
#include <linux/pm_opp.h>
#include <linux/slab.h>
+#include <linux/soc/qcom/mdt_loader.h>
#include "adreno_gpu.h"
#include "msm_gem.h"
#include "msm_mmu.h"
+static bool zap_available = true;
+
+static int zap_shader_load_mdt(struct msm_gpu *gpu, const char *fwname,
+ u32 pasid)
+{
+ struct device *dev = &gpu->pdev->dev;
+ const struct firmware *fw;
+ struct device_node *np, *mem_np;
+ struct resource r;
+ phys_addr_t mem_phys;
+ ssize_t mem_size;
+ void *mem_region = NULL;
+ int ret;
+
+ if (!IS_ENABLED(CONFIG_ARCH_QCOM)) {
+ zap_available = false;
+ return -EINVAL;
+ }
+
+ np = of_get_child_by_name(dev->of_node, "zap-shader");
+ if (!np) {
+ zap_available = false;
+ return -ENODEV;
+ }
+
+ mem_np = of_parse_phandle(np, "memory-region", 0);
+ of_node_put(np);
+ if (!mem_np) {
+ zap_available = false;
+ return -EINVAL;
+ }
+
+ ret = of_address_to_resource(mem_np, 0, &r);
+ of_node_put(mem_np);
+ if (ret)
+ return ret;
+
+ mem_phys = r.start;
+ mem_size = resource_size(&r);
+
+ /* Request the MDT file for the firmware */
+ fw = adreno_request_fw(to_adreno_gpu(gpu), fwname);
+ if (IS_ERR(fw)) {
+ DRM_DEV_ERROR(dev, "Unable to load %s\n", fwname);
+ return PTR_ERR(fw);
+ }
+
+ /* Figure out how much memory we need */
+ mem_size = qcom_mdt_get_size(fw);
+ if (mem_size < 0) {
+ ret = mem_size;
+ goto out;
+ }
+
+ /* Allocate memory for the firmware image */
+ mem_region = memremap(mem_phys, mem_size, MEMREMAP_WC);
+ if (!mem_region) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ /*
+ * Load the rest of the MDT
+ *
+ * Note that we could be dealing with two different paths, since
+ * with upstream linux-firmware it would be in a qcom/ subdir..
+ * adreno_request_fw() handles this, but qcom_mdt_load() does
+ * not. But since we've already gotten through adreno_request_fw()
+ * we know which of the two cases it is:
+ */
+ if (to_adreno_gpu(gpu)->fwloc == FW_LOCATION_LEGACY) {
+ ret = qcom_mdt_load(dev, fw, fwname, pasid,
+ mem_region, mem_phys, mem_size, NULL);
+ } else {
+ char *newname;
+
+ newname = kasprintf(GFP_KERNEL, "qcom/%s", fwname);
+
+ ret = qcom_mdt_load(dev, fw, newname, pasid,
+ mem_region, mem_phys, mem_size, NULL);
+ kfree(newname);
+ }
+ if (ret)
+ goto out;
+
+ /* Send the image to the secure world */
+ ret = qcom_scm_pas_auth_and_reset(pasid);
+
+ /*
+ * If the scm call returns -EOPNOTSUPP we assume that this target
+ * doesn't need/support the zap shader so quietly fail
+ */
+ if (ret == -EOPNOTSUPP)
+ zap_available = false;
+ else if (ret)
+ DRM_DEV_ERROR(dev, "Unable to authorize the image\n");
+
+out:
+ if (mem_region)
+ memunmap(mem_region);
+
+ release_firmware(fw);
+
+ return ret;
+}
+
+int adreno_zap_shader_load(struct msm_gpu *gpu, u32 pasid)
+{
+ struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu);
+ struct platform_device *pdev = gpu->pdev;
+
+ /* Short cut if we determine the zap shader isn't available/needed */
+ if (!zap_available)
+ return -ENODEV;
+
+ /* We need SCM to be able to load the firmware */
+ if (!qcom_scm_is_available()) {
+ DRM_DEV_ERROR(&pdev->dev, "SCM is not available\n");
+ return -EPROBE_DEFER;
+ }
+
+ /* Each GPU has a target specific zap shader firmware name to use */
+ if (!adreno_gpu->info->zapfw) {
+ zap_available = false;
+ DRM_DEV_ERROR(&pdev->dev,
+ "Zap shader firmware file not specified for this target\n");
+ return -ENODEV;
+ }
+
+ return zap_shader_load_mdt(gpu, adreno_gpu->info->zapfw, pasid);
+}
+
int adreno_get_param(struct msm_gpu *gpu, uint32_t param, uint64_t *value)
{
struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu);
case MSM_PARAM_NR_RINGS:
*value = gpu->nr_rings;
return 0;
+ case MSM_PARAM_PP_PGTABLE:
+ *value = 0;
+ return 0;
+ case MSM_PARAM_FAULTS:
+ *value = gpu->global_faults;
+ return 0;
default:
DBG("%s: invalid param: %u", gpu->name, param);
return -EINVAL;
int adreno_gpu_state_get(struct msm_gpu *gpu, struct msm_gpu_state *state);
int adreno_gpu_state_put(struct msm_gpu_state *state);
+/*
+ * For a5xx and a6xx targets load the zap shader that is used to pull the GPU
+ * out of secure mode
+ */
+int adreno_zap_shader_load(struct msm_gpu *gpu, u32 pasid);
+
/* ringbuffer helpers (the parts that are adreno specific) */
static inline void
#define LEFT_MIXER 0
#define RIGHT_MIXER 1
+/* timeout in ms waiting for frame done */
+#define DPU_CRTC_FRAME_DONE_TIMEOUT_MS 60
+
static struct dpu_kms *_dpu_crtc_get_kms(struct drm_crtc *crtc)
{
struct msm_drm_private *priv = crtc->dev->dev_private;
trace_dpu_crtc_complete_commit(DRMID(crtc));
}
-static void _dpu_crtc_setup_mixer_for_encoder(
- struct drm_crtc *crtc,
- struct drm_encoder *enc)
-{
- struct dpu_crtc_state *cstate = to_dpu_crtc_state(crtc->state);
- struct dpu_kms *dpu_kms = _dpu_crtc_get_kms(crtc);
- struct dpu_rm *rm = &dpu_kms->rm;
- struct dpu_crtc_mixer *mixer;
- struct dpu_hw_ctl *last_valid_ctl = NULL;
- int i;
- struct dpu_rm_hw_iter lm_iter, ctl_iter;
-
- dpu_rm_init_hw_iter(&lm_iter, enc->base.id, DPU_HW_BLK_LM);
- dpu_rm_init_hw_iter(&ctl_iter, enc->base.id, DPU_HW_BLK_CTL);
-
- /* Set up all the mixers and ctls reserved by this encoder */
- for (i = cstate->num_mixers; i < ARRAY_SIZE(cstate->mixers); i++) {
- mixer = &cstate->mixers[i];
-
- if (!dpu_rm_get_hw(rm, &lm_iter))
- break;
- mixer->hw_lm = (struct dpu_hw_mixer *)lm_iter.hw;
-
- /* CTL may be <= LMs, if <, multiple LMs controlled by 1 CTL */
- if (!dpu_rm_get_hw(rm, &ctl_iter)) {
- DPU_DEBUG("no ctl assigned to lm %d, using previous\n",
- mixer->hw_lm->idx - LM_0);
- mixer->lm_ctl = last_valid_ctl;
- } else {
- mixer->lm_ctl = (struct dpu_hw_ctl *)ctl_iter.hw;
- last_valid_ctl = mixer->lm_ctl;
- }
-
- /* Shouldn't happen, mixers are always >= ctls */
- if (!mixer->lm_ctl) {
- DPU_ERROR("no valid ctls found for lm %d\n",
- mixer->hw_lm->idx - LM_0);
- return;
- }
-
- cstate->num_mixers++;
- DPU_DEBUG("setup mixer %d: lm %d\n",
- i, mixer->hw_lm->idx - LM_0);
- DPU_DEBUG("setup mixer %d: ctl %d\n",
- i, mixer->lm_ctl->idx - CTL_0);
- }
-}
-
-static void _dpu_crtc_setup_mixers(struct drm_crtc *crtc)
-{
- struct drm_encoder *enc;
-
- WARN_ON(!drm_modeset_is_locked(&crtc->mutex));
-
- /* Check for mixers on all encoders attached to this crtc */
- drm_for_each_encoder_mask(enc, crtc->dev, crtc->state->encoder_mask)
- _dpu_crtc_setup_mixer_for_encoder(crtc, enc);
-}
-
static void _dpu_crtc_setup_lm_bounds(struct drm_crtc *crtc,
struct drm_crtc_state *state)
{
dev = crtc->dev;
smmu_state = &dpu_crtc->smmu_state;
- if (!cstate->num_mixers) {
- _dpu_crtc_setup_mixers(crtc);
- _dpu_crtc_setup_lm_bounds(crtc, crtc->state);
- }
+ _dpu_crtc_setup_lm_bounds(crtc, crtc->state);
if (dpu_crtc->event) {
WARN_ON(dpu_crtc->event);
DPU_ATRACE_BEGIN("frame done completion wait");
ret = wait_for_completion_timeout(&dpu_crtc->frame_done_comp,
- msecs_to_jiffies(DPU_FRAME_DONE_TIMEOUT));
+ msecs_to_jiffies(DPU_CRTC_FRAME_DONE_TIMEOUT_MS));
if (!ret) {
DRM_ERROR("frame done wait timed out, ret:%d\n", ret);
rc = -ETIMEDOUT;
#define MAX_VDISPLAY_SPLIT 1080
+/* timeout in frames waiting for frame done */
+#define DPU_ENCODER_FRAME_DONE_TIMEOUT_FRAMES 5
+
/**
* enum dpu_enc_rc_events - events for resource control state machine
* @DPU_ENC_RC_EVENT_KICKOFF:
* Bit0 = phys_encs[0] etc.
* @crtc_frame_event_cb: callback handler for frame event
* @crtc_frame_event_cb_data: callback handler private data
- * @frame_done_timeout: frame done timeout in Hz
+ * @frame_done_timeout_ms: frame done timeout in ms
* @frame_done_timer: watchdog timer for frame done event
* @vsync_event_timer: vsync timer
* @disp_info: local copy of msm_display_info struct
void (*crtc_frame_event_cb)(void *, u32 event);
void *crtc_frame_event_cb_data;
- atomic_t frame_done_timeout;
+ atomic_t frame_done_timeout_ms;
struct timer_list frame_done_timer;
struct timer_list vsync_event_timer;
list_for_each_entry(cur_mode, &connector->modes, head) {
if (cur_mode->vdisplay == adj_mode->vdisplay &&
- cur_mode->hdisplay == adj_mode->hdisplay &&
- cur_mode->vrefresh == adj_mode->vrefresh) {
+ cur_mode->hdisplay == adj_mode->hdisplay &&
+ drm_mode_vrefresh(cur_mode) == drm_mode_vrefresh(adj_mode)) {
adj_mode->private = cur_mode->private;
adj_mode->private_flags |= cur_mode->private_flags;
}
struct dpu_kms *dpu_kms;
struct list_head *connector_list;
struct drm_connector *conn = NULL, *conn_iter;
- struct dpu_rm_hw_iter pp_iter, ctl_iter;
+ struct drm_crtc *drm_crtc;
+ struct dpu_crtc_state *cstate;
+ struct dpu_rm_hw_iter hw_iter;
struct msm_display_topology topology;
struct dpu_hw_ctl *hw_ctl[MAX_CHANNELS_PER_ENC] = { NULL };
- int i = 0, ret;
+ struct dpu_hw_mixer *hw_lm[MAX_CHANNELS_PER_ENC] = { NULL };
+ int num_lm = 0, num_ctl = 0;
+ int i, j, ret;
if (!drm_enc) {
DPU_ERROR("invalid encoder\n");
return;
}
+ drm_for_each_crtc(drm_crtc, drm_enc->dev)
+ if (drm_crtc->state->encoder_mask & drm_encoder_mask(drm_enc))
+ break;
+
topology = dpu_encoder_get_topology(dpu_enc, dpu_kms, adj_mode);
/* Reserve dynamic resources now. Indicating non-AtomicTest phase */
- ret = dpu_rm_reserve(&dpu_kms->rm, drm_enc, drm_enc->crtc->state,
+ ret = dpu_rm_reserve(&dpu_kms->rm, drm_enc, drm_crtc->state,
topology, false);
if (ret) {
DPU_ERROR_ENC(dpu_enc,
return;
}
- dpu_rm_init_hw_iter(&pp_iter, drm_enc->base.id, DPU_HW_BLK_PINGPONG);
+ dpu_rm_init_hw_iter(&hw_iter, drm_enc->base.id, DPU_HW_BLK_PINGPONG);
for (i = 0; i < MAX_CHANNELS_PER_ENC; i++) {
dpu_enc->hw_pp[i] = NULL;
- if (!dpu_rm_get_hw(&dpu_kms->rm, &pp_iter))
+ if (!dpu_rm_get_hw(&dpu_kms->rm, &hw_iter))
+ break;
+ dpu_enc->hw_pp[i] = (struct dpu_hw_pingpong *) hw_iter.hw;
+ }
+
+ dpu_rm_init_hw_iter(&hw_iter, drm_enc->base.id, DPU_HW_BLK_CTL);
+ for (i = 0; i < MAX_CHANNELS_PER_ENC; i++) {
+ if (!dpu_rm_get_hw(&dpu_kms->rm, &hw_iter))
break;
- dpu_enc->hw_pp[i] = (struct dpu_hw_pingpong *) pp_iter.hw;
+ hw_ctl[i] = (struct dpu_hw_ctl *)hw_iter.hw;
+ num_ctl++;
}
- dpu_rm_init_hw_iter(&ctl_iter, drm_enc->base.id, DPU_HW_BLK_CTL);
+ dpu_rm_init_hw_iter(&hw_iter, drm_enc->base.id, DPU_HW_BLK_LM);
for (i = 0; i < MAX_CHANNELS_PER_ENC; i++) {
- if (!dpu_rm_get_hw(&dpu_kms->rm, &ctl_iter))
+ if (!dpu_rm_get_hw(&dpu_kms->rm, &hw_iter))
break;
- hw_ctl[i] = (struct dpu_hw_ctl *)ctl_iter.hw;
+ hw_lm[i] = (struct dpu_hw_mixer *)hw_iter.hw;
+ num_lm++;
}
+ cstate = to_dpu_crtc_state(drm_crtc->state);
+
+ for (i = 0; i < num_lm; i++) {
+ int ctl_idx = (i < num_ctl) ? i : (num_ctl-1);
+
+ cstate->mixers[i].hw_lm = hw_lm[i];
+ cstate->mixers[i].lm_ctl = hw_ctl[ctl_idx];
+ }
+
+ cstate->num_mixers = num_lm;
+
for (i = 0; i < dpu_enc->num_phys_encs; i++) {
struct dpu_encoder_phys *phys = dpu_enc->phys_encs[i];
if (!dpu_enc->hw_pp[i]) {
DPU_ERROR_ENC(dpu_enc, "no pp block assigned"
"at idx: %d\n", i);
- return;
+ goto error;
}
if (!hw_ctl[i]) {
DPU_ERROR_ENC(dpu_enc, "no ctl block assigned"
"at idx: %d\n", i);
- return;
+ goto error;
}
phys->hw_pp = dpu_enc->hw_pp[i];
phys->hw_ctl = hw_ctl[i];
+ dpu_rm_init_hw_iter(&hw_iter, drm_enc->base.id,
+ DPU_HW_BLK_INTF);
+ for (j = 0; j < MAX_CHANNELS_PER_ENC; j++) {
+ struct dpu_hw_intf *hw_intf;
+
+ if (!dpu_rm_get_hw(&dpu_kms->rm, &hw_iter))
+ break;
+
+ hw_intf = (struct dpu_hw_intf *)hw_iter.hw;
+ if (hw_intf->idx == phys->intf_idx)
+ phys->hw_intf = hw_intf;
+ }
+
+ if (!phys->hw_intf) {
+ DPU_ERROR_ENC(dpu_enc,
+ "no intf block assigned at idx: %d\n",
+ i);
+ goto error;
+ }
+
phys->connector = conn->state->connector;
if (phys->ops.mode_set)
phys->ops.mode_set(phys, mode, adj_mode);
}
dpu_enc->mode_set_complete = true;
+
+error:
+ dpu_rm_release(&dpu_kms->rm, drm_enc);
}
static void _dpu_encoder_virt_enable_helper(struct drm_encoder *drm_enc)
}
/* after phys waits for frame-done, should be no more frames pending */
- if (atomic_xchg(&dpu_enc->frame_done_timeout, 0)) {
+ if (atomic_xchg(&dpu_enc->frame_done_timeout_ms, 0)) {
DPU_ERROR("enc%d timeout pending\n", drm_enc->base.id);
del_timer_sync(&dpu_enc->frame_done_timer);
}
}
if (!dpu_enc->frame_busy_mask[0]) {
- atomic_set(&dpu_enc->frame_done_timeout, 0);
+ atomic_set(&dpu_enc->frame_done_timeout_ms, 0);
del_timer(&dpu_enc->frame_done_timer);
dpu_encoder_resource_control(drm_enc,
if (!ctl)
continue;
- if (phys->split_role != ENC_ROLE_SLAVE)
+ /*
+ * This is cleared in frame_done worker, which isn't invoked
+ * for async commits. So don't set this for async, since it'll
+ * roll over to the next commit.
+ */
+ if (!async && phys->split_role != ENC_ROLE_SLAVE)
set_bit(i, dpu_enc->frame_busy_mask);
+
if (!phys->ops.needs_single_flush ||
!phys->ops.needs_single_flush(phys))
_dpu_encoder_trigger_flush(&dpu_enc->base, phys, 0x0,
trace_dpu_enc_kickoff(DRMID(drm_enc));
- atomic_set(&dpu_enc->frame_done_timeout,
- DPU_FRAME_DONE_TIMEOUT * 1000 /
- drm_enc->crtc->state->adjusted_mode.vrefresh);
- mod_timer(&dpu_enc->frame_done_timer, jiffies +
- ((atomic_read(&dpu_enc->frame_done_timeout) * HZ) / 1000));
+ /*
+ * Asynchronous frames don't handle FRAME_DONE events. As such, they
+ * shouldn't enable the frame_done watchdog since it will always time
+ * out.
+ */
+ if (!async) {
+ unsigned long timeout_ms;
+ timeout_ms = DPU_ENCODER_FRAME_DONE_TIMEOUT_FRAMES * 1000 /
+ drm_mode_vrefresh(&drm_enc->crtc->state->adjusted_mode);
+
+ atomic_set(&dpu_enc->frame_done_timeout_ms, timeout_ms);
+ mod_timer(&dpu_enc->frame_done_timer,
+ jiffies + msecs_to_jiffies(timeout_ms));
+ }
/* All phys encs are ready to go, trigger the kickoff */
_dpu_encoder_kickoff_phys(dpu_enc, async);
DRM_DEBUG_KMS("id:%u invalid timeout frame_busy_mask=%lu\n",
DRMID(drm_enc), dpu_enc->frame_busy_mask[0]);
return;
- } else if (!atomic_xchg(&dpu_enc->frame_done_timeout, 0)) {
+ } else if (!atomic_xchg(&dpu_enc->frame_done_timeout_ms, 0)) {
DRM_DEBUG_KMS("id:%u invalid timeout\n", DRMID(drm_enc));
return;
}
spin_lock_init(&dpu_enc->enc_spinlock);
- atomic_set(&dpu_enc->frame_done_timeout, 0);
+ atomic_set(&dpu_enc->frame_done_timeout_ms, 0);
timer_setup(&dpu_enc->frame_done_timer,
dpu_encoder_frame_done_timeout, 0);
* @hw_mdptop: Hardware interface to the top registers
* @hw_ctl: Hardware interface to the ctl registers
* @hw_pp: Hardware interface to the ping pong registers
+ * @hw_intf: Hardware interface to the intf registers
* @dpu_kms: Pointer to the dpu_kms top level
* @cached_mode: DRM mode cached at mode_set time, acted on in enable
* @enabled: Whether the encoder has enabled and running a mode
struct dpu_hw_mdp *hw_mdptop;
struct dpu_hw_ctl *hw_ctl;
struct dpu_hw_pingpong *hw_pp;
+ struct dpu_hw_intf *hw_intf;
struct dpu_kms *dpu_kms;
struct drm_display_mode cached_mode;
enum dpu_enc_split_role split_role;
return atomic_inc_return(&phys->pending_kickoff_cnt);
}
-/**
- * struct dpu_encoder_phys_vid - sub-class of dpu_encoder_phys to handle video
- * mode specific operations
- * @base: Baseclass physical encoder structure
- * @hw_intf: Hardware interface to the intf registers
- * @timing_params: Current timing parameter
- */
-struct dpu_encoder_phys_vid {
- struct dpu_encoder_phys base;
- struct dpu_hw_intf *hw_intf;
- struct intf_timing_params timing_params;
-};
-
/**
* struct dpu_encoder_phys_cmd - sub-class of dpu_encoder_phys to handle command
* mode specific operations
return;
}
- tc_cfg.vsync_count = vsync_hz / (mode->vtotal * mode->vrefresh);
+ tc_cfg.vsync_count = vsync_hz /
+ (mode->vtotal * drm_mode_vrefresh(mode));
/* enable external TE after kickoff to avoid premature autorefresh */
tc_cfg.hw_vsync_mode = 0;
DPU_DEBUG_CMDENC(cmd_enc,
"tc %d vsync_clk_speed_hz %u vtotal %u vrefresh %u\n",
phys_enc->hw_pp->idx - PINGPONG_0, vsync_hz,
- mode->vtotal, mode->vrefresh);
+ mode->vtotal, drm_mode_vrefresh(mode));
DPU_DEBUG_CMDENC(cmd_enc,
"tc %d enable %u start_pos %u rd_ptr_irq %u\n",
phys_enc->hw_pp->idx - PINGPONG_0, tc_enable, tc_cfg.start_pos,
#include "dpu_trace.h"
#define DPU_DEBUG_VIDENC(e, fmt, ...) DPU_DEBUG("enc%d intf%d " fmt, \
- (e) && (e)->base.parent ? \
- (e)->base.parent->base.id : -1, \
+ (e) && (e)->parent ? \
+ (e)->parent->base.id : -1, \
(e) && (e)->hw_intf ? \
(e)->hw_intf->idx - INTF_0 : -1, ##__VA_ARGS__)
#define DPU_ERROR_VIDENC(e, fmt, ...) DPU_ERROR("enc%d intf%d " fmt, \
- (e) && (e)->base.parent ? \
- (e)->base.parent->base.id : -1, \
+ (e) && (e)->parent ? \
+ (e)->parent->base.id : -1, \
(e) && (e)->hw_intf ? \
(e)->hw_intf->idx - INTF_0 : -1, ##__VA_ARGS__)
}
static void drm_mode_to_intf_timing_params(
- const struct dpu_encoder_phys_vid *vid_enc,
+ const struct dpu_encoder_phys *phys_enc,
const struct drm_display_mode *mode,
struct intf_timing_params *timing)
{
timing->hsync_skew = mode->hskew;
/* DSI controller cannot handle active-low sync signals. */
- if (vid_enc->hw_intf->cap->type == INTF_DSI) {
+ if (phys_enc->hw_intf->cap->type == INTF_DSI) {
timing->hsync_polarity = 0;
timing->vsync_polarity = 0;
}
* lines based on the chip worst case latencies.
*/
static u32 programmable_fetch_get_num_lines(
- struct dpu_encoder_phys_vid *vid_enc,
+ struct dpu_encoder_phys *phys_enc,
const struct intf_timing_params *timing)
{
u32 worst_case_needed_lines =
- vid_enc->hw_intf->cap->prog_fetch_lines_worst_case;
+ phys_enc->hw_intf->cap->prog_fetch_lines_worst_case;
u32 start_of_frame_lines =
timing->v_back_porch + timing->vsync_pulse_width;
u32 needed_vfp_lines = worst_case_needed_lines - start_of_frame_lines;
/* Fetch must be outside active lines, otherwise undefined. */
if (start_of_frame_lines >= worst_case_needed_lines) {
- DPU_DEBUG_VIDENC(vid_enc,
+ DPU_DEBUG_VIDENC(phys_enc,
"prog fetch is not needed, large vbp+vsw\n");
actual_vfp_lines = 0;
} else if (timing->v_front_porch < needed_vfp_lines) {
/* Warn fetch needed, but not enough porch in panel config */
pr_warn_once
("low vbp+vfp may lead to perf issues in some cases\n");
- DPU_DEBUG_VIDENC(vid_enc,
+ DPU_DEBUG_VIDENC(phys_enc,
"less vfp than fetch req, using entire vfp\n");
actual_vfp_lines = timing->v_front_porch;
} else {
- DPU_DEBUG_VIDENC(vid_enc, "room in vfp for needed prefetch\n");
+ DPU_DEBUG_VIDENC(phys_enc, "room in vfp for needed prefetch\n");
actual_vfp_lines = needed_vfp_lines;
}
- DPU_DEBUG_VIDENC(vid_enc,
+ DPU_DEBUG_VIDENC(phys_enc,
"v_front_porch %u v_back_porch %u vsync_pulse_width %u\n",
timing->v_front_porch, timing->v_back_porch,
timing->vsync_pulse_width);
- DPU_DEBUG_VIDENC(vid_enc,
+ DPU_DEBUG_VIDENC(phys_enc,
"wc_lines %u needed_vfp_lines %u actual_vfp_lines %u\n",
worst_case_needed_lines, needed_vfp_lines, actual_vfp_lines);
static void programmable_fetch_config(struct dpu_encoder_phys *phys_enc,
const struct intf_timing_params *timing)
{
- struct dpu_encoder_phys_vid *vid_enc =
- to_dpu_encoder_phys_vid(phys_enc);
struct intf_prog_fetch f = { 0 };
u32 vfp_fetch_lines = 0;
u32 horiz_total = 0;
u32 vfp_fetch_start_vsync_counter = 0;
unsigned long lock_flags;
- if (WARN_ON_ONCE(!vid_enc->hw_intf->ops.setup_prg_fetch))
+ if (WARN_ON_ONCE(!phys_enc->hw_intf->ops.setup_prg_fetch))
return;
- vfp_fetch_lines = programmable_fetch_get_num_lines(vid_enc, timing);
+ vfp_fetch_lines = programmable_fetch_get_num_lines(phys_enc, timing);
if (vfp_fetch_lines) {
vert_total = get_vertical_total(timing);
horiz_total = get_horizontal_total(timing);
f.fetch_start = vfp_fetch_start_vsync_counter;
}
- DPU_DEBUG_VIDENC(vid_enc,
+ DPU_DEBUG_VIDENC(phys_enc,
"vfp_fetch_lines %u vfp_fetch_start_vsync_counter %u\n",
vfp_fetch_lines, vfp_fetch_start_vsync_counter);
spin_lock_irqsave(phys_enc->enc_spinlock, lock_flags);
- vid_enc->hw_intf->ops.setup_prg_fetch(vid_enc->hw_intf, &f);
+ phys_enc->hw_intf->ops.setup_prg_fetch(phys_enc->hw_intf, &f);
spin_unlock_irqrestore(phys_enc->enc_spinlock, lock_flags);
}
struct drm_display_mode *adj_mode)
{
if (phys_enc)
- DPU_DEBUG_VIDENC(to_dpu_encoder_phys_vid(phys_enc), "\n");
+ DPU_DEBUG_VIDENC(phys_enc, "\n");
/*
* Modifying mode has consequences when the mode comes back to us
static void dpu_encoder_phys_vid_setup_timing_engine(
struct dpu_encoder_phys *phys_enc)
{
- struct dpu_encoder_phys_vid *vid_enc;
struct drm_display_mode mode;
struct intf_timing_params timing_params = { 0 };
const struct dpu_format *fmt = NULL;
}
mode = phys_enc->cached_mode;
- vid_enc = to_dpu_encoder_phys_vid(phys_enc);
- if (!vid_enc->hw_intf->ops.setup_timing_gen) {
+ if (!phys_enc->hw_intf->ops.setup_timing_gen) {
DPU_ERROR("timing engine setup is not supported\n");
return;
}
- DPU_DEBUG_VIDENC(vid_enc, "enabling mode:\n");
+ DPU_DEBUG_VIDENC(phys_enc, "enabling mode:\n");
drm_mode_debug_printmodeline(&mode);
if (phys_enc->split_role != ENC_ROLE_SOLO) {
mode.hsync_start >>= 1;
mode.hsync_end >>= 1;
- DPU_DEBUG_VIDENC(vid_enc,
+ DPU_DEBUG_VIDENC(phys_enc,
"split_role %d, halve horizontal %d %d %d %d\n",
phys_enc->split_role,
mode.hdisplay, mode.htotal,
mode.hsync_start, mode.hsync_end);
}
- drm_mode_to_intf_timing_params(vid_enc, &mode, &timing_params);
+ drm_mode_to_intf_timing_params(phys_enc, &mode, &timing_params);
fmt = dpu_get_dpu_format(fmt_fourcc);
- DPU_DEBUG_VIDENC(vid_enc, "fmt_fourcc 0x%X\n", fmt_fourcc);
+ DPU_DEBUG_VIDENC(phys_enc, "fmt_fourcc 0x%X\n", fmt_fourcc);
- intf_cfg.intf = vid_enc->hw_intf->idx;
+ intf_cfg.intf = phys_enc->hw_intf->idx;
intf_cfg.intf_mode_sel = DPU_CTL_MODE_SEL_VID;
intf_cfg.stream_sel = 0; /* Don't care value for video mode */
intf_cfg.mode_3d = dpu_encoder_helper_get_3d_blend_mode(phys_enc);
spin_lock_irqsave(phys_enc->enc_spinlock, lock_flags);
- vid_enc->hw_intf->ops.setup_timing_gen(vid_enc->hw_intf,
+ phys_enc->hw_intf->ops.setup_timing_gen(phys_enc->hw_intf,
&timing_params, fmt);
phys_enc->hw_ctl->ops.setup_intf_cfg(phys_enc->hw_ctl, &intf_cfg);
spin_unlock_irqrestore(phys_enc->enc_spinlock, lock_flags);
programmable_fetch_config(phys_enc, &timing_params);
-
- vid_enc->timing_params = timing_params;
}
static void dpu_encoder_phys_vid_vblank_irq(void *arg, int irq_idx)
phys_enc);
}
-static bool _dpu_encoder_phys_is_dual_ctl(struct dpu_encoder_phys *phys_enc)
-{
- struct dpu_crtc_state *dpu_cstate;
-
- if (!phys_enc)
- return false;
-
- dpu_cstate = to_dpu_crtc_state(phys_enc->parent->crtc->state);
-
- return dpu_cstate->num_ctls > 1;
-}
-
static bool dpu_encoder_phys_vid_needs_single_flush(
struct dpu_encoder_phys *phys_enc)
{
- return (phys_enc && _dpu_encoder_phys_is_dual_ctl(phys_enc));
+ return phys_enc->split_role != ENC_ROLE_SOLO;
}
static void _dpu_encoder_phys_vid_setup_irq_hw_idx(
struct drm_display_mode *mode,
struct drm_display_mode *adj_mode)
{
- struct dpu_encoder_phys_vid *vid_enc;
-
if (!phys_enc || !phys_enc->dpu_kms) {
DPU_ERROR("invalid encoder/kms\n");
return;
}
- vid_enc = to_dpu_encoder_phys_vid(phys_enc);
-
if (adj_mode) {
phys_enc->cached_mode = *adj_mode;
drm_mode_debug_printmodeline(adj_mode);
- DPU_DEBUG_VIDENC(vid_enc, "caching mode:\n");
+ DPU_DEBUG_VIDENC(phys_enc, "caching mode:\n");
}
_dpu_encoder_phys_vid_setup_irq_hw_idx(phys_enc);
bool enable)
{
int ret = 0;
- struct dpu_encoder_phys_vid *vid_enc;
int refcount;
if (!phys_enc) {
}
refcount = atomic_read(&phys_enc->vblank_refcount);
- vid_enc = to_dpu_encoder_phys_vid(phys_enc);
/* Slave encoders don't report vblank */
if (!dpu_encoder_phys_vid_is_master(phys_enc))
if (ret) {
DRM_ERROR("failed: id:%u intf:%d ret:%d enable:%d refcnt:%d\n",
DRMID(phys_enc->parent),
- vid_enc->hw_intf->idx - INTF_0, ret, enable,
+ phys_enc->hw_intf->idx - INTF_0, ret, enable,
refcount);
}
return ret;
static void dpu_encoder_phys_vid_enable(struct dpu_encoder_phys *phys_enc)
{
- struct msm_drm_private *priv;
- struct dpu_encoder_phys_vid *vid_enc;
- struct dpu_rm_hw_iter iter;
struct dpu_hw_ctl *ctl;
u32 flush_mask = 0;
- if (!phys_enc || !phys_enc->parent || !phys_enc->parent->dev ||
- !phys_enc->parent->dev->dev_private) {
- DPU_ERROR("invalid encoder/device\n");
- return;
- }
- priv = phys_enc->parent->dev->dev_private;
-
- vid_enc = to_dpu_encoder_phys_vid(phys_enc);
ctl = phys_enc->hw_ctl;
- dpu_rm_init_hw_iter(&iter, phys_enc->parent->base.id, DPU_HW_BLK_INTF);
- while (dpu_rm_get_hw(&phys_enc->dpu_kms->rm, &iter)) {
- struct dpu_hw_intf *hw_intf = (struct dpu_hw_intf *)iter.hw;
-
- if (hw_intf->idx == phys_enc->intf_idx) {
- vid_enc->hw_intf = hw_intf;
- break;
- }
- }
-
- if (!vid_enc->hw_intf) {
- DPU_ERROR("hw_intf not assigned\n");
- return;
- }
-
- DPU_DEBUG_VIDENC(vid_enc, "\n");
+ DPU_DEBUG_VIDENC(phys_enc, "\n");
- if (WARN_ON(!vid_enc->hw_intf->ops.enable_timing))
+ if (WARN_ON(!phys_enc->hw_intf->ops.enable_timing))
return;
- dpu_encoder_helper_split_config(phys_enc, vid_enc->hw_intf->idx);
+ dpu_encoder_helper_split_config(phys_enc, phys_enc->hw_intf->idx);
dpu_encoder_phys_vid_setup_timing_engine(phys_enc);
!dpu_encoder_phys_vid_is_master(phys_enc))
goto skip_flush;
- ctl->ops.get_bitmask_intf(ctl, &flush_mask, vid_enc->hw_intf->idx);
+ ctl->ops.get_bitmask_intf(ctl, &flush_mask, phys_enc->hw_intf->idx);
ctl->ops.update_pending_flush(ctl, flush_mask);
skip_flush:
- DPU_DEBUG_VIDENC(vid_enc, "update pending flush ctl %d flush_mask %x\n",
- ctl->idx - CTL_0, flush_mask);
+ DPU_DEBUG_VIDENC(phys_enc,
+ "update pending flush ctl %d flush_mask %x\n",
+ ctl->idx - CTL_0, flush_mask);
/* ctl_flush & timing engine enable will be triggered by framework */
if (phys_enc->enable_state == DPU_ENC_DISABLED)
static void dpu_encoder_phys_vid_destroy(struct dpu_encoder_phys *phys_enc)
{
- struct dpu_encoder_phys_vid *vid_enc;
-
if (!phys_enc) {
DPU_ERROR("invalid encoder\n");
return;
}
- vid_enc = to_dpu_encoder_phys_vid(phys_enc);
- DPU_DEBUG_VIDENC(vid_enc, "\n");
- kfree(vid_enc);
+ DPU_DEBUG_VIDENC(phys_enc, "\n");
+ kfree(phys_enc);
}
static void dpu_encoder_phys_vid_get_hw_resources(
static void dpu_encoder_phys_vid_prepare_for_kickoff(
struct dpu_encoder_phys *phys_enc)
{
- struct dpu_encoder_phys_vid *vid_enc;
struct dpu_hw_ctl *ctl;
int rc;
DPU_ERROR("invalid encoder/parameters\n");
return;
}
- vid_enc = to_dpu_encoder_phys_vid(phys_enc);
ctl = phys_enc->hw_ctl;
if (!ctl || !ctl->ops.wait_reset_status)
*/
rc = ctl->ops.wait_reset_status(ctl);
if (rc) {
- DPU_ERROR_VIDENC(vid_enc, "ctl %d reset failure: %d\n",
+ DPU_ERROR_VIDENC(phys_enc, "ctl %d reset failure: %d\n",
ctl->idx, rc);
dpu_encoder_helper_unregister_irq(phys_enc, INTR_IDX_VSYNC);
}
static void dpu_encoder_phys_vid_disable(struct dpu_encoder_phys *phys_enc)
{
struct msm_drm_private *priv;
- struct dpu_encoder_phys_vid *vid_enc;
unsigned long lock_flags;
int ret;
}
priv = phys_enc->parent->dev->dev_private;
- vid_enc = to_dpu_encoder_phys_vid(phys_enc);
- if (!vid_enc->hw_intf || !phys_enc->hw_ctl) {
+ if (!phys_enc->hw_intf || !phys_enc->hw_ctl) {
DPU_ERROR("invalid hw_intf %d hw_ctl %d\n",
- vid_enc->hw_intf != 0, phys_enc->hw_ctl != 0);
+ phys_enc->hw_intf != 0, phys_enc->hw_ctl != 0);
return;
}
- DPU_DEBUG_VIDENC(vid_enc, "\n");
-
- if (WARN_ON(!vid_enc->hw_intf->ops.enable_timing))
+ if (WARN_ON(!phys_enc->hw_intf->ops.enable_timing))
return;
if (phys_enc->enable_state == DPU_ENC_DISABLED) {
}
spin_lock_irqsave(phys_enc->enc_spinlock, lock_flags);
- vid_enc->hw_intf->ops.enable_timing(vid_enc->hw_intf, 0);
+ phys_enc->hw_intf->ops.enable_timing(phys_enc->hw_intf, 0);
if (dpu_encoder_phys_vid_is_master(phys_enc))
dpu_encoder_phys_inc_pending(phys_enc);
spin_unlock_irqrestore(phys_enc->enc_spinlock, lock_flags);
atomic_set(&phys_enc->pending_kickoff_cnt, 0);
DRM_ERROR("wait disable failed: id:%u intf:%d ret:%d\n",
DRMID(phys_enc->parent),
- vid_enc->hw_intf->idx - INTF_0, ret);
+ phys_enc->hw_intf->idx - INTF_0, ret);
}
}
struct dpu_encoder_phys *phys_enc)
{
unsigned long lock_flags;
- struct dpu_encoder_phys_vid *vid_enc;
if (!phys_enc) {
DPU_ERROR("invalid encoder\n");
return;
}
- vid_enc = to_dpu_encoder_phys_vid(phys_enc);
- DPU_DEBUG_VIDENC(vid_enc, "enable_state %d\n", phys_enc->enable_state);
-
/*
* Video mode must flush CTL before enabling timing engine
* Video encoders need to turn on their interfaces now
*/
if (phys_enc->enable_state == DPU_ENC_ENABLING) {
trace_dpu_enc_phys_vid_post_kickoff(DRMID(phys_enc->parent),
- vid_enc->hw_intf->idx - INTF_0);
+ phys_enc->hw_intf->idx - INTF_0);
spin_lock_irqsave(phys_enc->enc_spinlock, lock_flags);
- vid_enc->hw_intf->ops.enable_timing(vid_enc->hw_intf, 1);
+ phys_enc->hw_intf->ops.enable_timing(phys_enc->hw_intf, 1);
spin_unlock_irqrestore(phys_enc->enc_spinlock, lock_flags);
phys_enc->enable_state = DPU_ENC_ENABLED;
}
static void dpu_encoder_phys_vid_irq_control(struct dpu_encoder_phys *phys_enc,
bool enable)
{
- struct dpu_encoder_phys_vid *vid_enc;
int ret;
if (!phys_enc)
return;
- vid_enc = to_dpu_encoder_phys_vid(phys_enc);
-
trace_dpu_enc_phys_vid_irq_ctrl(DRMID(phys_enc->parent),
- vid_enc->hw_intf->idx - INTF_0,
+ phys_enc->hw_intf->idx - INTF_0,
enable,
atomic_read(&phys_enc->vblank_refcount));
static int dpu_encoder_phys_vid_get_line_count(
struct dpu_encoder_phys *phys_enc)
{
- struct dpu_encoder_phys_vid *vid_enc;
-
if (!phys_enc)
return -EINVAL;
if (!dpu_encoder_phys_vid_is_master(phys_enc))
return -EINVAL;
- vid_enc = to_dpu_encoder_phys_vid(phys_enc);
- if (!vid_enc->hw_intf || !vid_enc->hw_intf->ops.get_line_count)
+ if (!phys_enc->hw_intf || !phys_enc->hw_intf->ops.get_line_count)
return -EINVAL;
- return vid_enc->hw_intf->ops.get_line_count(vid_enc->hw_intf);
+ return phys_enc->hw_intf->ops.get_line_count(phys_enc->hw_intf);
}
static void dpu_encoder_phys_vid_init_ops(struct dpu_encoder_phys_ops *ops)
struct dpu_enc_phys_init_params *p)
{
struct dpu_encoder_phys *phys_enc = NULL;
- struct dpu_encoder_phys_vid *vid_enc = NULL;
struct dpu_encoder_irq *irq;
int i, ret = 0;
goto fail;
}
- vid_enc = kzalloc(sizeof(*vid_enc), GFP_KERNEL);
- if (!vid_enc) {
+ phys_enc = kzalloc(sizeof(*phys_enc), GFP_KERNEL);
+ if (!phys_enc) {
ret = -ENOMEM;
goto fail;
}
- phys_enc = &vid_enc->base;
-
phys_enc->hw_mdptop = p->dpu_kms->hw_mdp;
phys_enc->intf_idx = p->intf_idx;
- DPU_DEBUG_VIDENC(vid_enc, "\n");
+ DPU_DEBUG_VIDENC(phys_enc, "\n");
dpu_encoder_phys_vid_init_ops(&phys_enc->ops);
phys_enc->parent = p->parent;
init_waitqueue_head(&phys_enc->pending_kickoff_wq);
phys_enc->enable_state = DPU_ENC_DISABLED;
- DPU_DEBUG_VIDENC(vid_enc, "created intf idx:%d\n", p->intf_idx);
+ DPU_DEBUG_VIDENC(phys_enc, "created intf idx:%d\n", p->intf_idx);
return phys_enc;
fail:
DPU_ERROR("failed to create encoder\n");
- if (vid_enc)
+ if (phys_enc)
dpu_encoder_phys_vid_destroy(phys_enc);
return ERR_PTR(ret);
#define DPU_NAME_SIZE 12
-/* timeout in frames waiting for frame done */
-#define DPU_FRAME_DONE_TIMEOUT 60
-
/*
* struct dpu_irq_callback - IRQ callback handlers
* @list: list to callback
ot_params.width = drm_rect_width(&pdpu->pipe_cfg.src_rect);
ot_params.height = drm_rect_height(&pdpu->pipe_cfg.src_rect);
ot_params.is_wfd = !pdpu->is_rt_pipe;
- ot_params.frame_rate = crtc->mode.vrefresh;
+ ot_params.frame_rate = drm_mode_vrefresh(&crtc->mode);
ot_params.vbif_idx = VBIF_RT;
ot_params.clk_ctrl = pdpu->pipe_hw->cap->clk_ctrl;
ot_params.rd = true;
return -EINVAL;
}
- total_lines_x100 = mode->vtotal * mode->vrefresh;
+ total_lines_x100 = mode->vtotal * drm_mode_vrefresh(mode);
if (!total_lines_x100) {
DRM_DEV_ERROR(dev, "%s: vtotal(%d) or vrefresh(%d) is 0\n",
- __func__, mode->vtotal, mode->vrefresh);
+ __func__, mode->vtotal, drm_mode_vrefresh(mode));
return -EINVAL;
}
struct msm_gpu_show_priv *show_priv;
int ret;
- if (!gpu)
+ if (!gpu || !gpu->funcs->gpu_state_get)
return -ENODEV;
show_priv = kmalloc(sizeof(*show_priv), GFP_KERNEL);
* MSM_GEM_INFO ioctl.
* - 1.4.0 - softpin, MSM_RELOC_BO_DUMP, and GEM_INFO support to set/get
* GEM object's debug name
+ * - 1.5.0 - Add SUBMITQUERY_QUERY ioctl
*/
#define MSM_VERSION_MAJOR 1
-#define MSM_VERSION_MINOR 4
+#define MSM_VERSION_MINOR 5
#define MSM_VERSION_PATCHLEVEL 0
static const struct drm_mode_config_funcs mode_config_funcs = {
priv->wq = alloc_ordered_workqueue("msm", 0);
+ INIT_WORK(&priv->free_work, msm_gem_free_work);
+ init_llist_head(&priv->free_list);
+
INIT_LIST_HEAD(&priv->inactive_list);
drm_mode_config_init(ddev);
args->flags, &args->id);
}
+static int msm_ioctl_submitqueue_query(struct drm_device *dev, void *data,
+ struct drm_file *file)
+{
+ return msm_submitqueue_query(dev, file->driver_priv, data);
+}
static int msm_ioctl_submitqueue_close(struct drm_device *dev, void *data,
struct drm_file *file)
DRM_IOCTL_DEF_DRV(MSM_GEM_MADVISE, msm_ioctl_gem_madvise, DRM_AUTH|DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(MSM_SUBMITQUEUE_NEW, msm_ioctl_submitqueue_new, DRM_AUTH|DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(MSM_SUBMITQUEUE_CLOSE, msm_ioctl_submitqueue_close, DRM_AUTH|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(MSM_SUBMITQUEUE_QUERY, msm_ioctl_submitqueue_query, DRM_AUTH|DRM_RENDER_ALLOW),
};
static const struct vm_operations_struct vm_ops = {
.irq_uninstall = msm_irq_uninstall,
.enable_vblank = msm_enable_vblank,
.disable_vblank = msm_disable_vblank,
- .gem_free_object = msm_gem_free_object,
+ .gem_free_object_unlocked = msm_gem_free_object,
.gem_vm_ops = &vm_ops,
.dumb_create = msm_gem_dumb_create,
.dumb_map_offset = msm_gem_dumb_map_offset,
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
.gem_prime_export = drm_gem_prime_export,
.gem_prime_import = drm_gem_prime_import,
- .gem_prime_res_obj = msm_gem_prime_res_obj,
.gem_prime_pin = msm_gem_prime_pin,
.gem_prime_unpin = msm_gem_prime_unpin,
.gem_prime_get_sg_table = msm_gem_prime_get_sg_table,
/* list of GEM objects: */
struct list_head inactive_list;
+ /* worker for delayed free of objects: */
+ struct work_struct free_work;
+ struct llist_head free_list;
+
struct workqueue_struct *wq;
unsigned int num_planes;
void *msm_gem_prime_vmap(struct drm_gem_object *obj);
void msm_gem_prime_vunmap(struct drm_gem_object *obj, void *vaddr);
int msm_gem_prime_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma);
-struct reservation_object *msm_gem_prime_res_obj(struct drm_gem_object *obj);
struct drm_gem_object *msm_gem_prime_import_sg_table(struct drm_device *dev,
struct dma_buf_attachment *attach, struct sg_table *sg);
int msm_gem_prime_pin(struct drm_gem_object *obj);
struct msm_gem_address_space *aspace, bool locked);
struct drm_gem_object *msm_gem_import(struct drm_device *dev,
struct dma_buf *dmabuf, struct sg_table *sgt);
+void msm_gem_free_work(struct work_struct *work);
__printf(2, 3)
void msm_gem_object_set_name(struct drm_gem_object *bo, const char *fmt, ...);
u32 id);
int msm_submitqueue_create(struct drm_device *drm, struct msm_file_private *ctx,
u32 prio, u32 flags, u32 *id);
+int msm_submitqueue_query(struct drm_device *drm, struct msm_file_private *ctx,
+ struct drm_msm_submitqueue_query *args);
int msm_submitqueue_remove(struct msm_file_private *ctx, u32 id);
void msm_submitqueue_close(struct msm_file_private *ctx);
fbdev->fb = fb;
helper->fb = fb;
- fbi->par = helper;
fbi->fbops = &msm_fb_ops;
- strcpy(fbi->fix.id, "msm");
-
- drm_fb_helper_fill_fix(fbi, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(fbi, helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(fbi, helper, sizes);
dev->mode_config.fb_base = paddr;
int msm_gem_sync_object(struct drm_gem_object *obj,
struct msm_fence_context *fctx, bool exclusive)
{
- struct msm_gem_object *msm_obj = to_msm_bo(obj);
struct reservation_object_list *fobj;
struct dma_fence *fence;
int i, ret;
- fobj = reservation_object_get_list(msm_obj->resv);
+ fobj = reservation_object_get_list(obj->resv);
if (!fobj || (fobj->shared_count == 0)) {
- fence = reservation_object_get_excl(msm_obj->resv);
+ fence = reservation_object_get_excl(obj->resv);
/* don't need to wait on our own fences, since ring is fifo */
if (fence && (fence->context != fctx->context)) {
ret = dma_fence_wait(fence, true);
for (i = 0; i < fobj->shared_count; i++) {
fence = rcu_dereference_protected(fobj->shared[i],
- reservation_object_held(msm_obj->resv));
+ reservation_object_held(obj->resv));
if (fence->context != fctx->context) {
ret = dma_fence_wait(fence, true);
if (ret)
WARN_ON(msm_obj->madv != MSM_MADV_WILLNEED);
msm_obj->gpu = gpu;
if (exclusive)
- reservation_object_add_excl_fence(msm_obj->resv, fence);
+ reservation_object_add_excl_fence(obj->resv, fence);
else
- reservation_object_add_shared_fence(msm_obj->resv, fence);
+ reservation_object_add_shared_fence(obj->resv, fence);
list_del_init(&msm_obj->mm_list);
list_add_tail(&msm_obj->mm_list, &gpu->active_list);
}
int msm_gem_cpu_prep(struct drm_gem_object *obj, uint32_t op, ktime_t *timeout)
{
- struct msm_gem_object *msm_obj = to_msm_bo(obj);
bool write = !!(op & MSM_PREP_WRITE);
unsigned long remain =
op & MSM_PREP_NOSYNC ? 0 : timeout_to_jiffies(timeout);
long ret;
- ret = reservation_object_wait_timeout_rcu(msm_obj->resv, write,
+ ret = reservation_object_wait_timeout_rcu(obj->resv, write,
true, remain);
if (ret == 0)
return remain == 0 ? -EBUSY : -ETIMEDOUT;
void msm_gem_describe(struct drm_gem_object *obj, struct seq_file *m)
{
struct msm_gem_object *msm_obj = to_msm_bo(obj);
- struct reservation_object *robj = msm_obj->resv;
+ struct reservation_object *robj = obj->resv;
struct reservation_object_list *fobj;
struct dma_fence *fence;
struct msm_gem_vma *vma;
/* don't call directly! Use drm_gem_object_put() and friends */
void msm_gem_free_object(struct drm_gem_object *obj)
{
- struct drm_device *dev = obj->dev;
struct msm_gem_object *msm_obj = to_msm_bo(obj);
+ struct drm_device *dev = obj->dev;
+ struct msm_drm_private *priv = dev->dev_private;
+
+ if (llist_add(&msm_obj->freed, &priv->free_list))
+ queue_work(priv->wq, &priv->free_work);
+}
+
+static void free_object(struct msm_gem_object *msm_obj)
+{
+ struct drm_gem_object *obj = &msm_obj->base;
+ struct drm_device *dev = obj->dev;
WARN_ON(!mutex_is_locked(&dev->struct_mutex));
put_pages(obj);
}
- if (msm_obj->resv == &msm_obj->_resv)
- reservation_object_fini(msm_obj->resv);
-
drm_gem_object_release(obj);
mutex_unlock(&msm_obj->lock);
kfree(msm_obj);
}
+void msm_gem_free_work(struct work_struct *work)
+{
+ struct msm_drm_private *priv =
+ container_of(work, struct msm_drm_private, free_work);
+ struct drm_device *dev = priv->dev;
+ struct llist_node *freed;
+ struct msm_gem_object *msm_obj, *next;
+
+ while ((freed = llist_del_all(&priv->free_list))) {
+
+ mutex_lock(&dev->struct_mutex);
+
+ llist_for_each_entry_safe(msm_obj, next,
+ freed, freed)
+ free_object(msm_obj);
+
+ mutex_unlock(&dev->struct_mutex);
+
+ if (need_resched())
+ break;
+ }
+}
+
/* convenience method to construct a GEM buffer object, and userspace handle */
int msm_gem_new_handle(struct drm_device *dev, struct drm_file *file,
uint32_t size, uint32_t flags, uint32_t *handle,
msm_obj->flags = flags;
msm_obj->madv = MSM_MADV_WILLNEED;
- if (resv) {
- msm_obj->resv = resv;
- } else {
- msm_obj->resv = &msm_obj->_resv;
- reservation_object_init(msm_obj->resv);
- }
+ if (resv)
+ msm_obj->base.resv = resv;
INIT_LIST_HEAD(&msm_obj->submit_entry);
INIT_LIST_HEAD(&msm_obj->vmas);
ret = drm_gem_object_init(dev, obj, size);
if (ret)
goto fail;
+ /*
+ * Our buffers are kept pinned, so allocating them from the
+ * MOVABLE zone is a really bad idea, and conflicts with CMA.
+ * See comments above new_inode() why this is required _and_
+ * expected if you're going to pin these pages.
+ */
+ mapping_set_gfp_mask(obj->filp->f_mapping, GFP_HIGHUSER);
}
return obj;
struct list_head vmas; /* list of msm_gem_vma */
+ struct llist_node freed;
+
/* normally (resv == &_resv) except for imported bo's */
struct reservation_object *resv;
struct reservation_object _resv;
void msm_gem_purge(struct drm_gem_object *obj, enum msm_gem_lock subclass);
void msm_gem_vunmap(struct drm_gem_object *obj, enum msm_gem_lock subclass);
+void msm_gem_free_work(struct work_struct *work);
/* Created per submit-ioctl, to track bo's and cmdstream bufs, etc,
* associated with the cmdstream submission for synchronization (and
} *cmd; /* array of size nr_cmds */
struct {
uint32_t flags;
- struct msm_gem_object *obj;
+ union {
+ struct msm_gem_object *obj;
+ uint32_t handle;
+ };
uint64_t iova;
} bos[0];
};
if (!obj->import_attach)
msm_gem_put_pages(obj);
}
-
-struct reservation_object *msm_gem_prime_res_obj(struct drm_gem_object *obj)
-{
- struct msm_gem_object *msm_obj = to_msm_bo(obj);
-
- return msm_obj->resv;
-}
kfree(submit);
}
-static inline unsigned long __must_check
-copy_from_user_inatomic(void *to, const void __user *from, unsigned long n)
-{
- if (access_ok(from, n))
- return __copy_from_user_inatomic(to, from, n);
- return -EFAULT;
-}
-
static int submit_lookup_objects(struct msm_gem_submit *submit,
struct drm_msm_gem_submit *args, struct drm_file *file)
{
unsigned i;
int ret = 0;
- spin_lock(&file->table_lock);
- pagefault_disable();
-
for (i = 0; i < args->nr_bos; i++) {
struct drm_msm_gem_submit_bo submit_bo;
- struct drm_gem_object *obj;
- struct msm_gem_object *msm_obj;
void __user *userptr =
u64_to_user_ptr(args->bos + (i * sizeof(submit_bo)));
*/
submit->bos[i].flags = 0;
- if (copy_from_user_inatomic(&submit_bo, userptr, sizeof(submit_bo))) {
- pagefault_enable();
- spin_unlock(&file->table_lock);
- if (copy_from_user(&submit_bo, userptr, sizeof(submit_bo))) {
- ret = -EFAULT;
- goto out;
- }
- spin_lock(&file->table_lock);
- pagefault_disable();
+ if (copy_from_user(&submit_bo, userptr, sizeof(submit_bo))) {
+ ret = -EFAULT;
+ i = 0;
+ goto out;
}
/* at least one of READ and/or WRITE flags should be set: */
!(submit_bo.flags & MANDATORY_FLAGS)) {
DRM_ERROR("invalid flags: %x\n", submit_bo.flags);
ret = -EINVAL;
- goto out_unlock;
+ i = 0;
+ goto out;
}
+ submit->bos[i].handle = submit_bo.handle;
submit->bos[i].flags = submit_bo.flags;
/* in validate_objects() we figure out if this is true: */
submit->bos[i].iova = submit_bo.presumed;
+ }
+
+ spin_lock(&file->table_lock);
+
+ for (i = 0; i < args->nr_bos; i++) {
+ struct drm_gem_object *obj;
+ struct msm_gem_object *msm_obj;
/* normally use drm_gem_object_lookup(), but for bulk lookup
* all under single table_lock just hit object_idr directly:
*/
- obj = idr_find(&file->object_idr, submit_bo.handle);
+ obj = idr_find(&file->object_idr, submit->bos[i].handle);
if (!obj) {
- DRM_ERROR("invalid handle %u at index %u\n", submit_bo.handle, i);
+ DRM_ERROR("invalid handle %u at index %u\n", submit->bos[i].handle, i);
ret = -EINVAL;
goto out_unlock;
}
if (!list_empty(&msm_obj->submit_entry)) {
DRM_ERROR("handle %u at index %u already on submit list\n",
- submit_bo.handle, i);
+ submit->bos[i].handle, i);
ret = -EINVAL;
goto out_unlock;
}
}
out_unlock:
- pagefault_enable();
spin_unlock(&file->table_lock);
out:
msm_gem_unpin_iova(&msm_obj->base, submit->gpu->aspace);
if (submit->bos[i].flags & BO_LOCKED)
- ww_mutex_unlock(&msm_obj->resv->lock);
+ ww_mutex_unlock(&msm_obj->base.resv->lock);
if (backoff && !(submit->bos[i].flags & BO_VALID))
submit->bos[i].iova = 0;
contended = i;
if (!(submit->bos[i].flags & BO_LOCKED)) {
- ret = ww_mutex_lock_interruptible(&msm_obj->resv->lock,
+ ret = ww_mutex_lock_interruptible(&msm_obj->base.resv->lock,
&submit->ticket);
if (ret)
goto fail;
if (ret == -EDEADLK) {
struct msm_gem_object *msm_obj = submit->bos[contended].obj;
/* we lost out in a seqno race, lock and retry.. */
- ret = ww_mutex_lock_slow_interruptible(&msm_obj->resv->lock,
+ ret = ww_mutex_lock_slow_interruptible(&msm_obj->base.resv->lock,
&submit->ticket);
if (!ret) {
submit->bos[contended].flags |= BO_LOCKED;
* strange place to call it. OTOH this is a
* convenient can-fail point to hook it in.
*/
- ret = reservation_object_reserve_shared(msm_obj->resv,
+ ret = reservation_object_reserve_shared(msm_obj->base.resv,
1);
if (ret)
return ret;
vma->mapped = true;
- if (aspace->mmu)
+ if (aspace && aspace->mmu)
ret = aspace->mmu->funcs->map(aspace->mmu, vma->iova, sgt,
size, prot);
if (submit) {
struct task_struct *task;
+ /* Increment the fault counts */
+ gpu->global_faults++;
+ submit->queue->faults++;
+
task = get_pid_task(submit->pid, PIDTYPE_PID);
if (task) {
comm = kstrdup(task->comm, GFP_KERNEL);
-
- /*
- * So slightly annoying, in other paths like
- * mmap'ing gem buffers, mmap_sem is acquired
- * before struct_mutex, which means we can't
- * hold struct_mutex across the call to
- * get_cmdline(). But submits are retired
- * from the same in-order workqueue, so we can
- * safely drop the lock here without worrying
- * about the submit going away.
- */
- mutex_unlock(&dev->struct_mutex);
cmd = kstrdup_quotable_cmdline(task, GFP_KERNEL);
put_task_struct(task);
- mutex_lock(&dev->struct_mutex);
}
if (comm && cmd) {
/* does gpu need hw_init? */
bool needs_hw_init;
+ /* number of GPU hangs (for all contexts) */
+ int global_faults;
+
/* worker for handling active-list retiring: */
struct work_struct retire_work;
int cnt)
{
struct msm_iommu *iommu = to_msm_iommu(mmu);
- int ret;
- pm_runtime_get_sync(mmu->dev);
- ret = iommu_attach_device(iommu->domain, mmu->dev);
- pm_runtime_put_sync(mmu->dev);
-
- return ret;
+ return iommu_attach_device(iommu->domain, mmu->dev);
}
static void msm_iommu_detach(struct msm_mmu *mmu, const char * const *names,
{
struct msm_iommu *iommu = to_msm_iommu(mmu);
- pm_runtime_get_sync(mmu->dev);
iommu_detach_device(iommu->domain, mmu->dev);
- pm_runtime_put_sync(mmu->dev);
}
static int msm_iommu_map(struct msm_mmu *mmu, uint64_t iova,
struct msm_iommu *iommu = to_msm_iommu(mmu);
size_t ret;
-// pm_runtime_get_sync(mmu->dev);
ret = iommu_map_sg(iommu->domain, iova, sgt->sgl, sgt->nents, prot);
-// pm_runtime_put_sync(mmu->dev);
WARN_ON(!ret);
return (ret == len) ? 0 : -EINVAL;
{
struct msm_iommu *iommu = to_msm_iommu(mmu);
- pm_runtime_get_sync(mmu->dev);
iommu_unmap(iommu->domain, iova, len);
- pm_runtime_put_sync(mmu->dev);
return 0;
}
return msm_submitqueue_create(drm, ctx, default_prio, 0, NULL);
}
+static int msm_submitqueue_query_faults(struct msm_gpu_submitqueue *queue,
+ struct drm_msm_submitqueue_query *args)
+{
+ size_t size = min_t(size_t, args->len, sizeof(queue->faults));
+ int ret;
+
+ /* If a zero length was passed in, return the data size we expect */
+ if (!args->len) {
+ args->len = sizeof(queue->faults);
+ return 0;
+ }
+
+ /* Set the length to the actual size of the data */
+ args->len = size;
+
+ ret = copy_to_user(u64_to_user_ptr(args->data), &queue->faults, size);
+
+ return ret ? -EFAULT : 0;
+}
+
+int msm_submitqueue_query(struct drm_device *drm, struct msm_file_private *ctx,
+ struct drm_msm_submitqueue_query *args)
+{
+ struct msm_gpu_submitqueue *queue;
+ int ret = -EINVAL;
+
+ if (args->pad)
+ return -EINVAL;
+
+ queue = msm_submitqueue_get(ctx, args->id);
+ if (!queue)
+ return -ENOENT;
+
+ if (args->param == MSM_SUBMITQUEUE_PARAM_FAULTS)
+ ret = msm_submitqueue_query_faults(queue, args);
+
+ msm_submitqueue_put(queue);
+
+ return ret;
+}
+
int msm_submitqueue_remove(struct msm_file_private *ctx, u32 id)
{
struct msm_gpu_submitqueue *entry;
if (!(bus_flags & DRM_BUS_FLAG_DE_LOW))
vdctrl0 |= VDCTRL0_ENABLE_ACT_HIGH;
/*
- * DRM_BUS_FLAG_PIXDATA_ defines are controller centric,
+ * DRM_BUS_FLAG_PIXDATA_DRIVE_ defines are controller centric,
* controllers VDCTRL0_DOTCLK is display centric.
* Drive on positive edge -> display samples on falling edge
- * DRM_BUS_FLAG_PIXDATA_POSEDGE -> VDCTRL0_DOTCLK_ACT_FALLING
+ * DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE -> VDCTRL0_DOTCLK_ACT_FALLING
*/
- if (bus_flags & DRM_BUS_FLAG_PIXDATA_POSEDGE)
+ if (bus_flags & DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE)
vdctrl0 |= VDCTRL0_DOTCLK_ACT_FALLING;
writel(vdctrl0, mxsfb->base + LCDC_VDCTRL0);
-ccflags-y += -I$(src)/include
-ccflags-y += -I$(src)/include/nvkm
-ccflags-y += -I$(src)/nvkm
-ccflags-y += -I$(src)
+ccflags-y += -I $(srctree)/$(src)/include
+ccflags-y += -I $(srctree)/$(src)/include/nvkm
+ccflags-y += -I $(srctree)/$(src)/nvkm
+ccflags-y += -I $(srctree)/$(src)
# NVKM - HW resource manager
#- code also used by various userspace tools/tests
select INPUT if ACPI && X86
select THERMAL if ACPI && X86
select ACPI_VIDEO if ACPI && X86
- select DRM_VM
help
Choose this option for open-source NVIDIA support.
+config NOUVEAU_LEGACY_CTX_SUPPORT
+ bool "Nouveau legacy context support"
+ depends on DRM_NOUVEAU
+ select DRM_LEGACY
+ default y
+ help
+ There was a version of the nouveau DDX that relied on legacy
+ ctx ioctls not erroring out. But that was back in time a long
+ ways, so offer a way to disable it now. For uapi compat with
+ old nouveau ddx this should be on by default, but modern distros
+ should consider turning it off.
+
config NOUVEAU_PLATFORM_DRIVER
bool "Nouveau (NVIDIA) SoC GPUs"
depends on DRM_NOUVEAU && ARCH_TEGRA
struct mutex mutex;
struct list_head head;
struct i2c_adapter i2c;
+ u8 enabled;
};
int nvkm_i2c_bus_acquire(struct nvkm_i2c_bus *);
struct mutex mutex;
struct list_head head;
struct i2c_adapter i2c;
+ u8 enabled;
u32 intr;
};
#ifdef CONFIG_ACPI
-/*
- * Hans de Goede: This define belongs in acpi/video.h, I've submitted a patch
- * to the acpi subsys to move it there from drivers/acpi/acpi_video.c .
- * This should be dropped once that is merged.
- */
-#ifndef ACPI_VIDEO_NOTIFY_PROBE
-#define ACPI_VIDEO_NOTIFY_PROBE 0x81
-#endif
-
static int
nouveau_display_acpi_ntfy(struct notifier_block *nb, unsigned long val,
void *data)
static int
nouveau_do_resume(struct drm_device *dev, bool runtime)
{
+ int ret = 0;
struct nouveau_drm *drm = nouveau_drm(dev);
NV_DEBUG(drm, "resuming object tree...\n");
- nvif_client_resume(&drm->master.base);
+ ret = nvif_client_resume(&drm->master.base);
+ if (ret) {
+ NV_ERROR(drm, "Client resume failed with error: %d\n", ret);
+ return ret;
+ }
NV_DEBUG(drm, "resuming fence...\n");
if (drm->fence && nouveau_fence(drm)->resume)
{
struct pci_dev *pdev = to_pci_dev(dev);
struct drm_device *drm_dev = pci_get_drvdata(pdev);
+ struct nouveau_drm *drm = nouveau_drm(drm_dev);
struct nvif_device *device = &nouveau_drm(drm_dev)->client.device;
int ret;
pci_set_master(pdev);
ret = nouveau_do_resume(drm_dev, true);
+ if (ret) {
+ NV_ERROR(drm, "resume failed with: %d\n", ret);
+ return ret;
+ }
/* do magic */
nvif_mask(&device->object, 0x088488, (1 << 25), (1 << 25));
static struct drm_driver
driver_stub = {
.driver_features =
- DRIVER_GEM | DRIVER_MODESET | DRIVER_PRIME | DRIVER_RENDER |
- DRIVER_KMS_LEGACY_CONTEXT,
+ DRIVER_GEM | DRIVER_MODESET | DRIVER_PRIME | DRIVER_RENDER
+#if defined(CONFIG_NOUVEAU_LEGACY_CTX_SUPPORT)
+ | DRIVER_KMS_LEGACY_CONTEXT
+#endif
+ ,
.open = nouveau_drm_open,
.postclose = nouveau_drm_postclose,
struct nouveau_channel;
struct platform_device;
-#define DRM_FILE_PAGE_OFFSET (0x100000000ULL >> PAGE_SHIFT)
-
#include "nouveau_fence.h"
#include "nouveau_bios.h"
#include "nouveau_vmm.h"
ret = PTR_ERR(info);
goto out_unlock;
}
- info->skip_vt_switch = 1;
-
- info->par = fbcon;
/* setup helper */
fbcon->helper.fb = &fb->base;
- strcpy(info->fix.id, "nouveaufb");
if (!chan)
info->flags = FBINFO_HWACCEL_DISABLED;
else
info->screen_base = nvbo_kmap_obj_iovirtual(fb->nvbo);
info->screen_size = fb->nvbo->bo.mem.num_pages << PAGE_SHIFT;
- drm_fb_helper_fill_fix(info, fb->base.pitches[0],
- fb->base.format->depth);
- drm_fb_helper_fill_var(info, &fbcon->helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(info, &fbcon->helper, sizes);
/* Use default scratch pixmap (info->pixmap.flags = FB_PIXMAP_SYSTEM) */
#include "nouveau_display.h"
struct nouveau_fbdev {
- struct drm_fb_helper helper;
+ struct drm_fb_helper helper; /* must be first */
unsigned int saved_flags;
struct nvif_object surf2d;
struct nvif_object clip;
struct drm_file *file_priv = filp->private_data;
struct nouveau_drm *drm = nouveau_drm(file_priv->minor->dev);
- if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET))
- return drm_legacy_mmap(filp, vma);
-
return ttm_bo_mmap(filp, vma, &drm->ttm.bdev);
}
ret = ttm_bo_device_init(&drm->ttm.bdev,
&nouveau_bo_driver,
dev->anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET,
drm->client.mmu.dmabits <= 32 ? true : false);
if (ret) {
NV_ERROR(drm, "error initialising bo driver, %d\n", ret);
return ret;
bar_len = device->func->resource_size(device, bar_nr);
+ if (!bar_len)
+ return -ENOMEM;
if (bar_nr == 3 && bar->bar2_halve)
bar_len >>= 1;
struct nvkm_device *device = bar->base.subdev.device;
static struct lock_class_key bar1_lock;
static struct lock_class_key bar2_lock;
- u64 start, limit;
+ u64 start, limit, size;
int ret;
ret = nvkm_gpuobj_new(device, 0x20000, 0, false, NULL, &bar->mem);
/* BAR2 */
start = 0x0100000000ULL;
- limit = start + device->func->resource_size(device, 3);
+ size = device->func->resource_size(device, 3);
+ if (!size)
+ return -ENOMEM;
+ limit = start + size;
ret = nvkm_vmm_new(device, start, limit-- - start, NULL, 0,
&bar2_lock, "bar2", &bar->bar2_vmm);
/* BAR1 */
start = 0x0000000000ULL;
- limit = start + device->func->resource_size(device, 1);
+ size = device->func->resource_size(device, 1);
+ if (!size)
+ return -ENOMEM;
+ limit = start + size;
ret = nvkm_vmm_new(device, start, limit-- - start, NULL, 0,
&bar1_lock, "bar1", &bar->bar1_vmm);
+ if (ret)
+ return ret;
atomic_inc(&bar->bar1_vmm->engref[NVKM_SUBDEV_BAR]);
bar->bar1_vmm->debug = bar->base.subdev.debug;
nvkm_error(subdev, "unable to calc plls\n");
return -EINVAL;
}
- nvkm_debug(subdev, "sucessfully calced PLLs for clock %i kHz"
+ nvkm_debug(subdev, "successfully calced PLLs for clock %i kHz"
" (refclock: %i kHz)\n", next->freq, ret);
} else {
/* calculate refpll coefficients */
{
struct nvkm_i2c_pad *pad = aux->pad;
int ret;
+
AUX_TRACE(aux, "acquire");
mutex_lock(&aux->mutex);
- ret = nvkm_i2c_pad_acquire(pad, NVKM_I2C_PAD_AUX);
+
+ if (aux->enabled)
+ ret = nvkm_i2c_pad_acquire(pad, NVKM_I2C_PAD_AUX);
+ else
+ ret = -EIO;
+
if (ret)
mutex_unlock(&aux->mutex);
return ret;
}
}
+void
+nvkm_i2c_aux_init(struct nvkm_i2c_aux *aux)
+{
+ AUX_TRACE(aux, "init");
+ mutex_lock(&aux->mutex);
+ aux->enabled = true;
+ mutex_unlock(&aux->mutex);
+}
+
+void
+nvkm_i2c_aux_fini(struct nvkm_i2c_aux *aux)
+{
+ AUX_TRACE(aux, "fini");
+ mutex_lock(&aux->mutex);
+ aux->enabled = false;
+ mutex_unlock(&aux->mutex);
+}
+
int
nvkm_i2c_aux_ctor(const struct nvkm_i2c_aux_func *func,
struct nvkm_i2c_pad *pad, int id,
int nvkm_i2c_aux_new_(const struct nvkm_i2c_aux_func *, struct nvkm_i2c_pad *,
int id, struct nvkm_i2c_aux **);
void nvkm_i2c_aux_del(struct nvkm_i2c_aux **);
+void nvkm_i2c_aux_init(struct nvkm_i2c_aux *);
+void nvkm_i2c_aux_fini(struct nvkm_i2c_aux *);
int nvkm_i2c_aux_xfer(struct nvkm_i2c_aux *, bool retry, u8 type,
u32 addr, u8 *data, u8 *size);
{
struct nvkm_i2c *i2c = nvkm_i2c(subdev);
struct nvkm_i2c_pad *pad;
+ struct nvkm_i2c_bus *bus;
+ struct nvkm_i2c_aux *aux;
u32 mask;
+ list_for_each_entry(aux, &i2c->aux, head) {
+ nvkm_i2c_aux_fini(aux);
+ }
+
+ list_for_each_entry(bus, &i2c->bus, head) {
+ nvkm_i2c_bus_fini(bus);
+ }
+
if ((mask = (1 << i2c->func->aux) - 1), i2c->func->aux_stat) {
i2c->func->aux_mask(i2c, NVKM_I2C_ANY, mask, 0);
i2c->func->aux_stat(i2c, &mask, &mask, &mask, &mask);
struct nvkm_i2c *i2c = nvkm_i2c(subdev);
struct nvkm_i2c_bus *bus;
struct nvkm_i2c_pad *pad;
+ struct nvkm_i2c_aux *aux;
list_for_each_entry(pad, &i2c->pad, head) {
nvkm_i2c_pad_init(pad);
nvkm_i2c_bus_init(bus);
}
+ list_for_each_entry(aux, &i2c->aux, head) {
+ nvkm_i2c_aux_init(aux);
+ }
+
return 0;
}
BUS_TRACE(bus, "init");
if (bus->func->init)
bus->func->init(bus);
+
+ mutex_lock(&bus->mutex);
+ bus->enabled = true;
+ mutex_unlock(&bus->mutex);
+}
+
+void
+nvkm_i2c_bus_fini(struct nvkm_i2c_bus *bus)
+{
+ BUS_TRACE(bus, "fini");
+ mutex_lock(&bus->mutex);
+ bus->enabled = false;
+ mutex_unlock(&bus->mutex);
}
void
{
struct nvkm_i2c_pad *pad = bus->pad;
int ret;
+
BUS_TRACE(bus, "acquire");
mutex_lock(&bus->mutex);
- ret = nvkm_i2c_pad_acquire(pad, NVKM_I2C_PAD_I2C);
+
+ if (bus->enabled)
+ ret = nvkm_i2c_pad_acquire(pad, NVKM_I2C_PAD_I2C);
+ else
+ ret = -EIO;
+
if (ret)
mutex_unlock(&bus->mutex);
return ret;
int id, struct nvkm_i2c_bus **);
void nvkm_i2c_bus_del(struct nvkm_i2c_bus **);
void nvkm_i2c_bus_init(struct nvkm_i2c_bus *);
+void nvkm_i2c_bus_fini(struct nvkm_i2c_bus *);
int nvkm_i2c_bit_xfer(struct nvkm_i2c_bus *, struct i2c_msg *, int);
void
nvkm_vmm_part(struct nvkm_vmm *vmm, struct nvkm_memory *inst)
{
- if (inst && vmm->func->part) {
+ if (inst && vmm && vmm->func->part) {
mutex_lock(&vmm->mutex);
vmm->func->part(vmm, inst);
mutex_unlock(&vmm->mutex);
Driver for OPA362 external analog TV amplifier controlled
through a GPIO.
-config DRM_OMAP_ENCODER_TFP410
- tristate "TFP410 DPI to DVI Encoder"
- help
- Driver for TFP410 DPI to DVI encoder.
-
config DRM_OMAP_ENCODER_TPD12S015
tristate "TPD12S015 HDMI ESD protection and level shifter"
help
Driver for TPD12S015, which offers HDMI ESD protection and level
shifting.
-config DRM_OMAP_CONNECTOR_DVI
- tristate "DVI Connector"
- depends on I2C
- help
- Driver for a generic DVI connector.
-
config DRM_OMAP_CONNECTOR_HDMI
tristate "HDMI Connector"
help
help
Driver for a generic analog TV connector.
-config DRM_OMAP_PANEL_DPI
- tristate "Generic DPI panel"
- depends on BACKLIGHT_CLASS_DEVICE
- help
- Driver for generic DPI panels.
-
config DRM_OMAP_PANEL_DSI_CM
tristate "Generic DSI Command Mode Panel"
depends on BACKLIGHT_CLASS_DEVICE
# SPDX-License-Identifier: GPL-2.0
obj-$(CONFIG_DRM_OMAP_ENCODER_OPA362) += encoder-opa362.o
-obj-$(CONFIG_DRM_OMAP_ENCODER_TFP410) += encoder-tfp410.o
obj-$(CONFIG_DRM_OMAP_ENCODER_TPD12S015) += encoder-tpd12s015.o
-obj-$(CONFIG_DRM_OMAP_CONNECTOR_DVI) += connector-dvi.o
obj-$(CONFIG_DRM_OMAP_CONNECTOR_HDMI) += connector-hdmi.o
obj-$(CONFIG_DRM_OMAP_CONNECTOR_ANALOG_TV) += connector-analog-tv.o
-obj-$(CONFIG_DRM_OMAP_PANEL_DPI) += panel-dpi.o
obj-$(CONFIG_DRM_OMAP_PANEL_DSI_CM) += panel-dsi-cm.o
obj-$(CONFIG_DRM_OMAP_PANEL_SONY_ACX565AKM) += panel-sony-acx565akm.o
obj-$(CONFIG_DRM_OMAP_PANEL_LGPHILIPS_LB035Q02) += panel-lgphilips-lb035q02.o
{
}
-static int tvc_enable(struct omap_dss_device *dssdev)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
- int r;
-
- dev_dbg(ddata->dev, "enable\n");
-
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return r;
-}
-
-static void tvc_disable(struct omap_dss_device *dssdev)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
-
- dev_dbg(ddata->dev, "disable\n");
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
-
- src->ops->disable(src);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
-}
-
static const struct omap_dss_device_ops tvc_ops = {
.connect = tvc_connect,
.disconnect = tvc_disconnect,
-
- .enable = tvc_enable,
- .disable = tvc_disable,
};
static int tvc_probe(struct platform_device *pdev)
dssdev->ops = &tvc_ops;
dssdev->dev = &pdev->dev;
dssdev->type = OMAP_DISPLAY_TYPE_VENC;
+ dssdev->display = true;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(0);
static int __exit tvc_remove(struct platform_device *pdev)
{
struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct omap_dss_device *dssdev = &ddata->dssdev;
omapdss_device_unregister(&ddata->dssdev);
- tvc_disable(dssdev);
-
return 0;
}
+++ /dev/null
-/*
- * Generic DVI Connector driver
- *
- * Copyright (C) 2013 Texas Instruments Incorporated - http://www.ti.com/
- * Author: Tomi Valkeinen <tomi.valkeinen@ti.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 as published by
- * the Free Software Foundation.
- */
-
-#include <linux/gpio/consumer.h>
-#include <linux/i2c.h>
-#include <linux/module.h>
-#include <linux/platform_device.h>
-#include <linux/slab.h>
-
-#include <drm/drm_edid.h>
-
-#include "../dss/omapdss.h"
-
-struct panel_drv_data {
- struct omap_dss_device dssdev;
-
- struct i2c_adapter *i2c_adapter;
-
- struct gpio_desc *hpd_gpio;
-
- void (*hpd_cb)(void *cb_data, enum drm_connector_status status);
- void *hpd_cb_data;
- bool hpd_enabled;
- /* mutex for hpd fields above */
- struct mutex hpd_lock;
-};
-
-#define to_panel_data(x) container_of(x, struct panel_drv_data, dssdev)
-
-static int dvic_connect(struct omap_dss_device *src,
- struct omap_dss_device *dst)
-{
- return 0;
-}
-
-static void dvic_disconnect(struct omap_dss_device *src,
- struct omap_dss_device *dst)
-{
-}
-
-static int dvic_enable(struct omap_dss_device *dssdev)
-{
- struct omap_dss_device *src = dssdev->src;
- int r;
-
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return 0;
-}
-
-static void dvic_disable(struct omap_dss_device *dssdev)
-{
- struct omap_dss_device *src = dssdev->src;
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
-
- src->ops->disable(src);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
-}
-
-static int dvic_ddc_read(struct i2c_adapter *adapter,
- unsigned char *buf, u16 count, u8 offset)
-{
- int r, retries;
-
- for (retries = 3; retries > 0; retries--) {
- struct i2c_msg msgs[] = {
- {
- .addr = DDC_ADDR,
- .flags = 0,
- .len = 1,
- .buf = &offset,
- }, {
- .addr = DDC_ADDR,
- .flags = I2C_M_RD,
- .len = count,
- .buf = buf,
- }
- };
-
- r = i2c_transfer(adapter, msgs, 2);
- if (r == 2)
- return 0;
-
- if (r != -EAGAIN)
- break;
- }
-
- return r < 0 ? r : -EIO;
-}
-
-static int dvic_read_edid(struct omap_dss_device *dssdev,
- u8 *edid, int len)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
- int r, l, bytes_read;
-
- l = min(EDID_LENGTH, len);
- r = dvic_ddc_read(ddata->i2c_adapter, edid, l, 0);
- if (r)
- return r;
-
- bytes_read = l;
-
- /* if there are extensions, read second block */
- if (len > EDID_LENGTH && edid[0x7e] > 0) {
- l = min(EDID_LENGTH, len - EDID_LENGTH);
-
- r = dvic_ddc_read(ddata->i2c_adapter, edid + EDID_LENGTH,
- l, EDID_LENGTH);
- if (r)
- return r;
-
- bytes_read += l;
- }
-
- return bytes_read;
-}
-
-static bool dvic_detect(struct omap_dss_device *dssdev)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
- unsigned char out;
- int r;
-
- if (ddata->hpd_gpio)
- return gpiod_get_value_cansleep(ddata->hpd_gpio);
-
- if (!ddata->i2c_adapter)
- return true;
-
- r = dvic_ddc_read(ddata->i2c_adapter, &out, 1, 0);
-
- return r == 0;
-}
-
-static void dvic_register_hpd_cb(struct omap_dss_device *dssdev,
- void (*cb)(void *cb_data,
- enum drm_connector_status status),
- void *cb_data)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
-
- mutex_lock(&ddata->hpd_lock);
- ddata->hpd_cb = cb;
- ddata->hpd_cb_data = cb_data;
- mutex_unlock(&ddata->hpd_lock);
-}
-
-static void dvic_unregister_hpd_cb(struct omap_dss_device *dssdev)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
-
- mutex_lock(&ddata->hpd_lock);
- ddata->hpd_cb = NULL;
- ddata->hpd_cb_data = NULL;
- mutex_unlock(&ddata->hpd_lock);
-}
-
-static const struct omap_dss_device_ops dvic_ops = {
- .connect = dvic_connect,
- .disconnect = dvic_disconnect,
-
- .enable = dvic_enable,
- .disable = dvic_disable,
-
- .read_edid = dvic_read_edid,
- .detect = dvic_detect,
-
- .register_hpd_cb = dvic_register_hpd_cb,
- .unregister_hpd_cb = dvic_unregister_hpd_cb,
-};
-
-static irqreturn_t dvic_hpd_isr(int irq, void *data)
-{
- struct panel_drv_data *ddata = data;
-
- mutex_lock(&ddata->hpd_lock);
- if (ddata->hpd_enabled && ddata->hpd_cb) {
- enum drm_connector_status status;
-
- if (dvic_detect(&ddata->dssdev))
- status = connector_status_connected;
- else
- status = connector_status_disconnected;
-
- ddata->hpd_cb(ddata->hpd_cb_data, status);
- }
- mutex_unlock(&ddata->hpd_lock);
-
- return IRQ_HANDLED;
-}
-
-static int dvic_probe_of(struct platform_device *pdev)
-{
- struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct device_node *node = pdev->dev.of_node;
- struct device_node *adapter_node;
- struct i2c_adapter *adapter;
- struct gpio_desc *gpio;
- int r;
-
- gpio = devm_gpiod_get_optional(&pdev->dev, "hpd", GPIOD_IN);
- if (IS_ERR(gpio)) {
- dev_err(&pdev->dev, "failed to parse HPD gpio\n");
- return PTR_ERR(gpio);
- }
-
- ddata->hpd_gpio = gpio;
-
- mutex_init(&ddata->hpd_lock);
-
- if (ddata->hpd_gpio) {
- r = devm_request_threaded_irq(&pdev->dev,
- gpiod_to_irq(ddata->hpd_gpio), NULL, dvic_hpd_isr,
- IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING | IRQF_ONESHOT,
- "DVI HPD", ddata);
- if (r)
- return r;
- }
-
- adapter_node = of_parse_phandle(node, "ddc-i2c-bus", 0);
- if (adapter_node) {
- adapter = of_get_i2c_adapter_by_node(adapter_node);
- of_node_put(adapter_node);
- if (adapter == NULL) {
- dev_err(&pdev->dev, "failed to parse ddc-i2c-bus\n");
- return -EPROBE_DEFER;
- }
-
- ddata->i2c_adapter = adapter;
- }
-
- return 0;
-}
-
-static int dvic_probe(struct platform_device *pdev)
-{
- struct panel_drv_data *ddata;
- struct omap_dss_device *dssdev;
- int r;
-
- ddata = devm_kzalloc(&pdev->dev, sizeof(*ddata), GFP_KERNEL);
- if (!ddata)
- return -ENOMEM;
-
- platform_set_drvdata(pdev, ddata);
-
- r = dvic_probe_of(pdev);
- if (r)
- return r;
-
- dssdev = &ddata->dssdev;
- dssdev->ops = &dvic_ops;
- dssdev->dev = &pdev->dev;
- dssdev->type = OMAP_DISPLAY_TYPE_DVI;
- dssdev->owner = THIS_MODULE;
- dssdev->of_ports = BIT(0);
-
- if (ddata->hpd_gpio)
- dssdev->ops_flags |= OMAP_DSS_DEVICE_OP_DETECT
- | OMAP_DSS_DEVICE_OP_HPD;
- if (ddata->i2c_adapter)
- dssdev->ops_flags |= OMAP_DSS_DEVICE_OP_DETECT
- | OMAP_DSS_DEVICE_OP_EDID;
-
- omapdss_display_init(dssdev);
- omapdss_device_register(dssdev);
-
- return 0;
-}
-
-static int __exit dvic_remove(struct platform_device *pdev)
-{
- struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct omap_dss_device *dssdev = &ddata->dssdev;
-
- omapdss_device_unregister(&ddata->dssdev);
-
- dvic_disable(dssdev);
-
- i2c_put_adapter(ddata->i2c_adapter);
-
- mutex_destroy(&ddata->hpd_lock);
-
- return 0;
-}
-
-static const struct of_device_id dvic_of_match[] = {
- { .compatible = "omapdss,dvi-connector", },
- {},
-};
-
-MODULE_DEVICE_TABLE(of, dvic_of_match);
-
-static struct platform_driver dvi_connector_driver = {
- .probe = dvic_probe,
- .remove = __exit_p(dvic_remove),
- .driver = {
- .name = "connector-dvi",
- .of_match_table = dvic_of_match,
- .suppress_bind_attrs = true,
- },
-};
-
-module_platform_driver(dvi_connector_driver);
-
-MODULE_AUTHOR("Tomi Valkeinen <tomi.valkeinen@ti.com>");
-MODULE_DESCRIPTION("Generic DVI Connector driver");
-MODULE_LICENSE("GPL");
{
}
-static int hdmic_enable(struct omap_dss_device *dssdev)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
- int r;
-
- dev_dbg(ddata->dev, "enable\n");
-
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return r;
-}
-
-static void hdmic_disable(struct omap_dss_device *dssdev)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
-
- dev_dbg(ddata->dev, "disable\n");
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
-
- src->ops->disable(src);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
-}
-
static bool hdmic_detect(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
.connect = hdmic_connect,
.disconnect = hdmic_disconnect,
- .enable = hdmic_enable,
- .disable = hdmic_disable,
-
.detect = hdmic_detect,
.register_hpd_cb = hdmic_register_hpd_cb,
.unregister_hpd_cb = hdmic_unregister_hpd_cb,
dssdev->ops = &hdmic_ops;
dssdev->dev = &pdev->dev;
dssdev->type = OMAP_DISPLAY_TYPE_HDMI;
+ dssdev->display = true;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(0);
dssdev->ops_flags = ddata->hpd_gpio
static int __exit hdmic_remove(struct platform_device *pdev)
{
struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct omap_dss_device *dssdev = &ddata->dssdev;
omapdss_device_unregister(&ddata->dssdev);
- hdmic_disable(dssdev);
-
return 0;
}
omapdss_device_disconnect(dst, dst->next);
}
-static int opa362_enable(struct omap_dss_device *dssdev)
+static void opa362_enable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
- int r;
-
- dev_dbg(dssdev->dev, "enable\n");
-
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
if (ddata->enable_gpio)
gpiod_set_value_cansleep(ddata->enable_gpio, 1);
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return 0;
}
static void opa362_disable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
-
- dev_dbg(dssdev->dev, "disable\n");
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
if (ddata->enable_gpio)
gpiod_set_value_cansleep(ddata->enable_gpio, 0);
-
- src->ops->disable(src);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
}
static const struct omap_dss_device_ops opa362_ops = {
dssdev->ops = &opa362_ops;
dssdev->dev = &pdev->dev;
dssdev->type = OMAP_DISPLAY_TYPE_VENC;
- dssdev->output_type = OMAP_DISPLAY_TYPE_VENC;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(1) | BIT(0);
omapdss_device_put(dssdev->next);
omapdss_device_unregister(&ddata->dssdev);
- WARN_ON(omapdss_device_is_enabled(dssdev));
- if (omapdss_device_is_enabled(dssdev))
- opa362_disable(dssdev);
-
- WARN_ON(omapdss_device_is_connected(dssdev));
- if (omapdss_device_is_connected(dssdev))
- omapdss_device_disconnect(NULL, dssdev);
+ opa362_disable(dssdev);
return 0;
}
+++ /dev/null
-/*
- * TFP410 DPI-to-DVI encoder driver
- *
- * Copyright (C) 2013 Texas Instruments Incorporated - http://www.ti.com/
- * Author: Tomi Valkeinen <tomi.valkeinen@ti.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 as published by
- * the Free Software Foundation.
- */
-
-#include <linux/gpio/consumer.h>
-#include <linux/module.h>
-#include <linux/platform_device.h>
-#include <linux/slab.h>
-
-#include "../dss/omapdss.h"
-
-struct panel_drv_data {
- struct omap_dss_device dssdev;
-
- struct gpio_desc *pd_gpio;
-};
-
-#define to_panel_data(x) container_of(x, struct panel_drv_data, dssdev)
-
-static int tfp410_connect(struct omap_dss_device *src,
- struct omap_dss_device *dst)
-{
- return omapdss_device_connect(dst->dss, dst, dst->next);
-}
-
-static void tfp410_disconnect(struct omap_dss_device *src,
- struct omap_dss_device *dst)
-{
- omapdss_device_disconnect(dst, dst->next);
-}
-
-static int tfp410_enable(struct omap_dss_device *dssdev)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
- int r;
-
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
-
- if (ddata->pd_gpio)
- gpiod_set_value_cansleep(ddata->pd_gpio, 0);
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return 0;
-}
-
-static void tfp410_disable(struct omap_dss_device *dssdev)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
-
- if (ddata->pd_gpio)
- gpiod_set_value_cansleep(ddata->pd_gpio, 0);
-
- src->ops->disable(src);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
-}
-
-static const struct omap_dss_device_ops tfp410_ops = {
- .connect = tfp410_connect,
- .disconnect = tfp410_disconnect,
- .enable = tfp410_enable,
- .disable = tfp410_disable,
-};
-
-static int tfp410_probe(struct platform_device *pdev)
-{
- struct panel_drv_data *ddata;
- struct omap_dss_device *dssdev;
- struct gpio_desc *gpio;
-
- ddata = devm_kzalloc(&pdev->dev, sizeof(*ddata), GFP_KERNEL);
- if (!ddata)
- return -ENOMEM;
-
- platform_set_drvdata(pdev, ddata);
-
- /* Powerdown GPIO */
- gpio = devm_gpiod_get_optional(&pdev->dev, "powerdown", GPIOD_OUT_HIGH);
- if (IS_ERR(gpio)) {
- dev_err(&pdev->dev, "failed to parse powerdown gpio\n");
- return PTR_ERR(gpio);
- }
-
- ddata->pd_gpio = gpio;
-
- dssdev = &ddata->dssdev;
- dssdev->ops = &tfp410_ops;
- dssdev->dev = &pdev->dev;
- dssdev->type = OMAP_DISPLAY_TYPE_DPI;
- dssdev->output_type = OMAP_DISPLAY_TYPE_DVI;
- dssdev->owner = THIS_MODULE;
- dssdev->of_ports = BIT(1) | BIT(0);
- dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_SYNC_POSEDGE
- | DRM_BUS_FLAG_PIXDATA_POSEDGE;
-
- dssdev->next = omapdss_of_find_connected_device(pdev->dev.of_node, 1);
- if (IS_ERR(dssdev->next)) {
- if (PTR_ERR(dssdev->next) != -EPROBE_DEFER)
- dev_err(&pdev->dev, "failed to find video sink\n");
- return PTR_ERR(dssdev->next);
- }
-
- omapdss_device_register(dssdev);
-
- return 0;
-}
-
-static int __exit tfp410_remove(struct platform_device *pdev)
-{
- struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct omap_dss_device *dssdev = &ddata->dssdev;
-
- if (dssdev->next)
- omapdss_device_put(dssdev->next);
- omapdss_device_unregister(&ddata->dssdev);
-
- WARN_ON(omapdss_device_is_enabled(dssdev));
- if (omapdss_device_is_enabled(dssdev))
- tfp410_disable(dssdev);
-
- WARN_ON(omapdss_device_is_connected(dssdev));
- if (omapdss_device_is_connected(dssdev))
- omapdss_device_disconnect(NULL, dssdev);
-
- return 0;
-}
-
-static const struct of_device_id tfp410_of_match[] = {
- { .compatible = "omapdss,ti,tfp410", },
- {},
-};
-
-MODULE_DEVICE_TABLE(of, tfp410_of_match);
-
-static struct platform_driver tfp410_driver = {
- .probe = tfp410_probe,
- .remove = __exit_p(tfp410_remove),
- .driver = {
- .name = "tfp410",
- .of_match_table = tfp410_of_match,
- .suppress_bind_attrs = true,
- },
-};
-
-module_platform_driver(tfp410_driver);
-
-MODULE_AUTHOR("Tomi Valkeinen <tomi.valkeinen@ti.com>");
-MODULE_DESCRIPTION("TFP410 DPI to DVI encoder driver");
-MODULE_LICENSE("GPL");
omapdss_device_disconnect(dst, dst->next);
}
-static int tpd_enable(struct omap_dss_device *dssdev)
-{
- struct omap_dss_device *src = dssdev->src;
- int r;
-
- if (dssdev->state == OMAP_DSS_DISPLAY_ACTIVE)
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return r;
-}
-
-static void tpd_disable(struct omap_dss_device *dssdev)
-{
- struct omap_dss_device *src = dssdev->src;
-
- if (dssdev->state != OMAP_DSS_DISPLAY_ACTIVE)
- return;
-
- src->ops->disable(src);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
-}
-
static bool tpd_detect(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
static const struct omap_dss_device_ops tpd_ops = {
.connect = tpd_connect,
.disconnect = tpd_disconnect,
- .enable = tpd_enable,
- .disable = tpd_disable,
.detect = tpd_detect,
.register_hpd_cb = tpd_register_hpd_cb,
.unregister_hpd_cb = tpd_unregister_hpd_cb,
dssdev->ops = &tpd_ops;
dssdev->dev = &pdev->dev;
dssdev->type = OMAP_DISPLAY_TYPE_HDMI;
- dssdev->output_type = OMAP_DISPLAY_TYPE_HDMI;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(1) | BIT(0);
dssdev->ops_flags = OMAP_DSS_DEVICE_OP_DETECT
omapdss_device_put(dssdev->next);
omapdss_device_unregister(&ddata->dssdev);
- WARN_ON(omapdss_device_is_enabled(dssdev));
- if (omapdss_device_is_enabled(dssdev))
- tpd_disable(dssdev);
-
- WARN_ON(omapdss_device_is_connected(dssdev));
- if (omapdss_device_is_connected(dssdev))
- omapdss_device_disconnect(NULL, dssdev);
-
return 0;
}
+++ /dev/null
-/*
- * Generic MIPI DPI Panel Driver
- *
- * Copyright (C) 2013 Texas Instruments Incorporated - http://www.ti.com/
- * Author: Tomi Valkeinen <tomi.valkeinen@ti.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 as published by
- * the Free Software Foundation.
- */
-
-#include <linux/gpio/consumer.h>
-#include <linux/module.h>
-#include <linux/platform_device.h>
-#include <linux/slab.h>
-#include <linux/of.h>
-#include <linux/regulator/consumer.h>
-#include <linux/backlight.h>
-
-#include <video/of_display_timing.h>
-
-#include "../dss/omapdss.h"
-
-struct panel_drv_data {
- struct omap_dss_device dssdev;
-
- struct videomode vm;
-
- struct backlight_device *backlight;
-
- struct gpio_desc *enable_gpio;
- struct regulator *vcc_supply;
-};
-
-#define to_panel_data(p) container_of(p, struct panel_drv_data, dssdev)
-
-static int panel_dpi_connect(struct omap_dss_device *src,
- struct omap_dss_device *dst)
-{
- return 0;
-}
-
-static void panel_dpi_disconnect(struct omap_dss_device *src,
- struct omap_dss_device *dst)
-{
-}
-
-static int panel_dpi_enable(struct omap_dss_device *dssdev)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
- int r;
-
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
-
- r = regulator_enable(ddata->vcc_supply);
- if (r) {
- src->ops->disable(src);
- return r;
- }
-
- gpiod_set_value_cansleep(ddata->enable_gpio, 1);
- backlight_enable(ddata->backlight);
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return 0;
-}
-
-static void panel_dpi_disable(struct omap_dss_device *dssdev)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
-
- backlight_disable(ddata->backlight);
-
- gpiod_set_value_cansleep(ddata->enable_gpio, 0);
- regulator_disable(ddata->vcc_supply);
-
- src->ops->disable(src);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
-}
-
-static void panel_dpi_get_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
-
- *vm = ddata->vm;
-}
-
-static const struct omap_dss_device_ops panel_dpi_ops = {
- .connect = panel_dpi_connect,
- .disconnect = panel_dpi_disconnect,
-
- .enable = panel_dpi_enable,
- .disable = panel_dpi_disable,
-
- .get_timings = panel_dpi_get_timings,
-};
-
-static int panel_dpi_probe_of(struct platform_device *pdev)
-{
- struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct device_node *node = pdev->dev.of_node;
- int r;
- struct display_timing timing;
- struct gpio_desc *gpio;
-
- gpio = devm_gpiod_get_optional(&pdev->dev, "enable", GPIOD_OUT_LOW);
- if (IS_ERR(gpio))
- return PTR_ERR(gpio);
-
- ddata->enable_gpio = gpio;
-
- /*
- * Many different panels are supported by this driver and there are
- * probably very different needs for their reset pins in regards to
- * timing and order relative to the enable gpio. So for now it's just
- * ensured that the reset line isn't active.
- */
- gpio = devm_gpiod_get_optional(&pdev->dev, "reset", GPIOD_OUT_LOW);
- if (IS_ERR(gpio))
- return PTR_ERR(gpio);
-
- ddata->vcc_supply = devm_regulator_get(&pdev->dev, "vcc");
- if (IS_ERR(ddata->vcc_supply))
- return PTR_ERR(ddata->vcc_supply);
-
- ddata->backlight = devm_of_find_backlight(&pdev->dev);
-
- if (IS_ERR(ddata->backlight))
- return PTR_ERR(ddata->backlight);
-
- r = of_get_display_timing(node, "panel-timing", &timing);
- if (r) {
- dev_err(&pdev->dev, "failed to get video timing\n");
- return r;
- }
-
- videomode_from_timing(&timing, &ddata->vm);
-
- return 0;
-}
-
-static int panel_dpi_probe(struct platform_device *pdev)
-{
- struct panel_drv_data *ddata;
- struct omap_dss_device *dssdev;
- int r;
-
- ddata = devm_kzalloc(&pdev->dev, sizeof(*ddata), GFP_KERNEL);
- if (ddata == NULL)
- return -ENOMEM;
-
- platform_set_drvdata(pdev, ddata);
-
- r = panel_dpi_probe_of(pdev);
- if (r)
- return r;
-
- dssdev = &ddata->dssdev;
- dssdev->dev = &pdev->dev;
- dssdev->ops = &panel_dpi_ops;
- dssdev->type = OMAP_DISPLAY_TYPE_DPI;
- dssdev->owner = THIS_MODULE;
- dssdev->of_ports = BIT(0);
- drm_bus_flags_from_videomode(&ddata->vm, &dssdev->bus_flags);
-
- omapdss_display_init(dssdev);
- omapdss_device_register(dssdev);
-
- return 0;
-}
-
-static int __exit panel_dpi_remove(struct platform_device *pdev)
-{
- struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct omap_dss_device *dssdev = &ddata->dssdev;
-
- omapdss_device_unregister(dssdev);
-
- panel_dpi_disable(dssdev);
-
- return 0;
-}
-
-static const struct of_device_id panel_dpi_of_match[] = {
- { .compatible = "omapdss,panel-dpi", },
- {},
-};
-
-MODULE_DEVICE_TABLE(of, panel_dpi_of_match);
-
-static struct platform_driver panel_dpi_driver = {
- .probe = panel_dpi_probe,
- .remove = __exit_p(panel_dpi_remove),
- .driver = {
- .name = "panel-dpi",
- .of_match_table = panel_dpi_of_match,
- .suppress_bind_attrs = true,
- },
-};
-
-module_platform_driver(panel_dpi_driver);
-
-MODULE_AUTHOR("Tomi Valkeinen <tomi.valkeinen@ti.com>");
-MODULE_DESCRIPTION("Generic MIPI DPI Panel Driver");
-MODULE_LICENSE("GPL");
#include <linux/of_device.h>
#include <linux/regulator/consumer.h>
+#include <drm/drm_connector.h>
+
#include <video/mipi_display.h>
#include <video/of_display_timing.h>
struct panel_drv_data {
struct omap_dss_device dssdev;
+ struct omap_dss_device *src;
struct videomode vm;
static int dsicm_dcs_read_1(struct panel_drv_data *ddata, u8 dcs_cmd, u8 *data)
{
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
int r;
u8 buf[1];
static int dsicm_dcs_write_0(struct panel_drv_data *ddata, u8 dcs_cmd)
{
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
return src->ops->dsi.dcs_write(src, ddata->channel, &dcs_cmd, 1);
}
static int dsicm_dcs_write_1(struct panel_drv_data *ddata, u8 dcs_cmd, u8 param)
{
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
u8 buf[2] = { dcs_cmd, param };
return src->ops->dsi.dcs_write(src, ddata->channel, buf, 2);
static int dsicm_sleep_in(struct panel_drv_data *ddata)
{
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
u8 cmd;
int r;
static int dsicm_set_update_window(struct panel_drv_data *ddata,
u16 x, u16 y, u16 w, u16 h)
{
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
int r;
u16 x1 = x;
u16 x2 = x + w - 1;
static int dsicm_enter_ulps(struct panel_drv_data *ddata)
{
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
int r;
if (ddata->ulps_enabled)
static int dsicm_exit_ulps(struct panel_drv_data *ddata)
{
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
int r;
if (!ddata->ulps_enabled)
return 0;
- r = src->ops->enable(src);
- if (r) {
- dev_err(&ddata->pdev->dev, "failed to enable DSI\n");
- goto err1;
- }
-
+ src->ops->enable(src);
src->ops->dsi.enable_hs(src, ddata->channel, true);
r = _dsicm_enable_te(ddata, true);
enable_irq(gpiod_to_irq(ddata->ext_te_gpio));
ddata->ulps_enabled = false;
}
-err1:
+
dsicm_queue_ulps_work(ddata);
return r;
static int dsicm_bl_update_status(struct backlight_device *dev)
{
struct panel_drv_data *ddata = dev_get_drvdata(&dev->dev);
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
int r = 0;
int level;
{
struct platform_device *pdev = to_platform_device(dev);
struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
u8 errors = 0;
int r;
{
struct platform_device *pdev = to_platform_device(dev);
struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
u8 id1, id2, id3;
int r;
{
struct platform_device *pdev = to_platform_device(dev);
struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
unsigned long t;
int r;
{
struct platform_device *pdev = to_platform_device(dev);
struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
unsigned long t;
int r;
static int dsicm_power_on(struct panel_drv_data *ddata)
{
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
u8 id1, id2, id3;
int r;
struct omap_dss_dsi_config dsi_config = {
goto err_vddi;
}
- r = src->ops->enable(src);
- if (r) {
- dev_err(&ddata->pdev->dev, "failed to enable DSI\n");
- goto err_vddi;
- }
+ src->ops->enable(src);
dsicm_hw_reset(ddata);
static void dsicm_power_off(struct panel_drv_data *ddata)
{
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
int r;
src->ops->dsi.disable_video_output(src, ddata->channel);
return r;
}
+ ddata->src = src;
return 0;
}
struct panel_drv_data *ddata = to_panel_data(dst);
src->ops->dsi.release_vc(src, ddata->channel);
+ ddata->src = NULL;
}
-static int dsicm_enable(struct omap_dss_device *dssdev)
+static void dsicm_enable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
+ struct omap_dss_device *src = ddata->src;
int r;
- dev_dbg(&ddata->pdev->dev, "enable\n");
-
mutex_lock(&ddata->lock);
- if (!omapdss_device_is_connected(dssdev)) {
- r = -ENODEV;
- goto err;
- }
-
- if (omapdss_device_is_enabled(dssdev)) {
- r = 0;
- goto err;
- }
-
src->ops->dsi.bus_lock(src);
r = dsicm_power_on(ddata);
if (r)
goto err;
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
mutex_unlock(&ddata->lock);
dsicm_bl_power(ddata, true);
- return 0;
+ return;
err:
- dev_dbg(&ddata->pdev->dev, "enable failed\n");
+ dev_dbg(&ddata->pdev->dev, "enable failed (%d)\n", r);
mutex_unlock(&ddata->lock);
- return r;
}
static void dsicm_disable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
+ struct omap_dss_device *src = ddata->src;
int r;
- dev_dbg(&ddata->pdev->dev, "disable\n");
-
dsicm_bl_power(ddata, false);
mutex_lock(&ddata->lock);
src->ops->dsi.bus_lock(src);
- if (omapdss_device_is_enabled(dssdev)) {
- r = dsicm_wake_up(ddata);
- if (!r)
- dsicm_power_off(ddata);
- }
+ r = dsicm_wake_up(ddata);
+ if (!r)
+ dsicm_power_off(ddata);
src->ops->dsi.bus_unlock(src);
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
-
mutex_unlock(&ddata->lock);
}
static void dsicm_framedone_cb(int err, void *data)
{
struct panel_drv_data *ddata = data;
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
dev_dbg(&ddata->pdev->dev, "framedone, err %d\n", err);
src->ops->dsi.bus_unlock(src);
static irqreturn_t dsicm_te_isr(int irq, void *data)
{
struct panel_drv_data *ddata = data;
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
int old;
int r;
{
struct panel_drv_data *ddata = container_of(work, struct panel_drv_data,
te_timeout_work.work);
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
dev_err(&ddata->pdev->dev, "TE not received for 250ms!\n");
u16 x, u16 y, u16 w, u16 h)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
+ struct omap_dss_device *src = ddata->src;
int r;
dev_dbg(&ddata->pdev->dev, "update %d, %d, %d x %d\n", x, y, w, h);
static int dsicm_sync(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
+ struct omap_dss_device *src = ddata->src;
dev_dbg(&ddata->pdev->dev, "sync\n");
static int _dsicm_enable_te(struct panel_drv_data *ddata, bool enable)
{
- struct omap_dss_device *src = ddata->dssdev.src;
+ struct omap_dss_device *src = ddata->src;
int r;
if (enable)
static int dsicm_enable_te(struct omap_dss_device *dssdev, bool enable)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
+ struct omap_dss_device *src = ddata->src;
int r;
mutex_lock(&ddata->lock);
u16 x, u16 y, u16 w, u16 h)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
+ struct omap_dss_device *src = ddata->src;
int r;
int first = 1;
int plen;
struct panel_drv_data *ddata = container_of(work, struct panel_drv_data,
ulps_work.work);
struct omap_dss_device *dssdev = &ddata->dssdev;
- struct omap_dss_device *src = dssdev->src;
+ struct omap_dss_device *src = ddata->src;
mutex_lock(&ddata->lock);
mutex_unlock(&ddata->lock);
}
-static void dsicm_get_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+static int dsicm_get_modes(struct omap_dss_device *dssdev,
+ struct drm_connector *connector)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- *vm = ddata->vm;
+ connector->display_info.width_mm = ddata->width_mm;
+ connector->display_info.height_mm = ddata->height_mm;
+
+ return omapdss_display_get_modes(connector, &ddata->vm);
}
static int dsicm_check_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+ struct drm_display_mode *mode)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
int ret = 0;
- if (vm->hactive != ddata->vm.hactive)
+ if (mode->hdisplay != ddata->vm.hactive)
ret = -EINVAL;
- if (vm->vactive != ddata->vm.vactive)
+ if (mode->vdisplay != ddata->vm.vactive)
ret = -EINVAL;
if (ret) {
dev_warn(dssdev->dev, "wrong resolution: %d x %d",
- vm->hactive, vm->vactive);
+ mode->hdisplay, mode->vdisplay);
dev_warn(dssdev->dev, "panel resolution: %d x %d",
ddata->vm.hactive, ddata->vm.vactive);
}
return ret;
}
-static void dsicm_get_size(struct omap_dss_device *dssdev,
- unsigned int *width, unsigned int *height)
-{
- struct panel_drv_data *ddata = to_panel_data(dssdev);
-
- *width = ddata->width_mm;
- *height = ddata->height_mm;
-}
-
static const struct omap_dss_device_ops dsicm_ops = {
.connect = dsicm_connect,
.disconnect = dsicm_disconnect,
.enable = dsicm_enable,
.disable = dsicm_disable,
- .get_timings = dsicm_get_timings,
+ .get_modes = dsicm_get_modes,
.check_timings = dsicm_check_timings,
};
.update = dsicm_update,
.sync = dsicm_sync,
- .get_size = dsicm_get_size,
-
.enable_te = dsicm_enable_te,
.get_te = dsicm_get_te,
dssdev->ops = &dsicm_ops;
dssdev->driver = &dsicm_dss_driver;
dssdev->type = OMAP_DISPLAY_TYPE_DSI;
+ dssdev->display = true;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(0);
+ dssdev->ops_flags = OMAP_DSS_DEVICE_OP_MODES;
dssdev->caps = OMAP_DSS_DISPLAY_CAP_MANUAL_UPDATE |
OMAP_DSS_DISPLAY_CAP_TEAR_ELIM;
omapdss_device_unregister(dssdev);
- dsicm_disable(dssdev);
- omapdss_device_disconnect(dssdev->src, dssdev);
+ if (omapdss_device_is_enabled(dssdev))
+ dsicm_disable(dssdev);
+ omapdss_device_disconnect(ddata->src, dssdev);
sysfs_remove_group(&pdev->dev.kobj, &dsicm_attr_group);
{
}
-static int lb035q02_enable(struct omap_dss_device *dssdev)
+static void lb035q02_enable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
- int r;
-
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
if (ddata->enable_gpio)
gpiod_set_value_cansleep(ddata->enable_gpio, 1);
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return 0;
}
static void lb035q02_disable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
if (ddata->enable_gpio)
gpiod_set_value_cansleep(ddata->enable_gpio, 0);
-
- src->ops->disable(src);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
}
-static void lb035q02_get_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+static int lb035q02_get_modes(struct omap_dss_device *dssdev,
+ struct drm_connector *connector)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- *vm = ddata->vm;
+ return omapdss_display_get_modes(connector, &ddata->vm);
}
static const struct omap_dss_device_ops lb035q02_ops = {
.enable = lb035q02_enable,
.disable = lb035q02_disable,
- .get_timings = lb035q02_get_timings,
+ .get_modes = lb035q02_get_modes,
};
static int lb035q02_probe_of(struct spi_device *spi)
dssdev->dev = &spi->dev;
dssdev->ops = &lb035q02_ops;
dssdev->type = OMAP_DISPLAY_TYPE_DPI;
+ dssdev->display = true;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(0);
+ dssdev->ops_flags = OMAP_DSS_DEVICE_OP_MODES;
/*
* Note: According to the panel documentation:
* DE is active LOW
* DATA needs to be driven on the FALLING edge
*/
- dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_SYNC_NEGEDGE
- | DRM_BUS_FLAG_PIXDATA_POSEDGE;
+ dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH
+ | DRM_BUS_FLAG_SYNC_DRIVE_NEGEDGE
+ | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE;
omapdss_display_init(dssdev);
omapdss_device_register(dssdev);
{
}
-static int nec_8048_enable(struct omap_dss_device *dssdev)
+static void nec_8048_enable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
- int r;
-
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
gpiod_set_value_cansleep(ddata->res_gpio, 1);
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return 0;
}
static void nec_8048_disable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
gpiod_set_value_cansleep(ddata->res_gpio, 0);
-
- src->ops->disable(src);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
}
-static void nec_8048_get_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+static int nec_8048_get_modes(struct omap_dss_device *dssdev,
+ struct drm_connector *connector)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- *vm = ddata->vm;
+ return omapdss_display_get_modes(connector, &ddata->vm);
}
static const struct omap_dss_device_ops nec_8048_ops = {
.enable = nec_8048_enable,
.disable = nec_8048_disable,
- .get_timings = nec_8048_get_timings,
+ .get_modes = nec_8048_get_modes,
};
static int nec_8048_probe(struct spi_device *spi)
dssdev->dev = &spi->dev;
dssdev->ops = &nec_8048_ops;
dssdev->type = OMAP_DISPLAY_TYPE_DPI;
+ dssdev->display = true;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(0);
- dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_SYNC_POSEDGE
- | DRM_BUS_FLAG_PIXDATA_POSEDGE;
+ dssdev->ops_flags = OMAP_DSS_DEVICE_OP_MODES;
+ dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH
+ | DRM_BUS_FLAG_SYNC_DRIVE_POSEDGE
+ | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE;
omapdss_display_init(dssdev);
omapdss_device_register(dssdev);
{
}
-static int sharp_ls_enable(struct omap_dss_device *dssdev)
+static void sharp_ls_pre_enable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
int r;
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
if (ddata->vcc) {
r = regulator_enable(ddata->vcc);
- if (r != 0)
- return r;
+ if (r)
+ dev_err(dssdev->dev, "%s: failed to enable regulator\n",
+ __func__);
}
+}
- r = src->ops->enable(src);
- if (r) {
- regulator_disable(ddata->vcc);
- return r;
- }
+static void sharp_ls_enable(struct omap_dss_device *dssdev)
+{
+ struct panel_drv_data *ddata = to_panel_data(dssdev);
/* wait couple of vsyncs until enabling the LCD */
msleep(50);
if (ddata->ini_gpio)
gpiod_set_value_cansleep(ddata->ini_gpio, 1);
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return 0;
}
static void sharp_ls_disable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
if (ddata->ini_gpio)
gpiod_set_value_cansleep(ddata->ini_gpio, 0);
gpiod_set_value_cansleep(ddata->resb_gpio, 0);
/* wait at least 5 vsyncs after disabling the LCD */
-
msleep(100);
+}
- src->ops->disable(src);
+static void sharp_ls_post_disable(struct omap_dss_device *dssdev)
+{
+ struct panel_drv_data *ddata = to_panel_data(dssdev);
if (ddata->vcc)
regulator_disable(ddata->vcc);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
}
-static void sharp_ls_get_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+static int sharp_ls_get_modes(struct omap_dss_device *dssdev,
+ struct drm_connector *connector)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- *vm = ddata->vm;
+ return omapdss_display_get_modes(connector, &ddata->vm);
}
static const struct omap_dss_device_ops sharp_ls_ops = {
.connect = sharp_ls_connect,
.disconnect = sharp_ls_disconnect,
+ .pre_enable = sharp_ls_pre_enable,
.enable = sharp_ls_enable,
.disable = sharp_ls_disable,
+ .post_disable = sharp_ls_post_disable,
- .get_timings = sharp_ls_get_timings,
+ .get_modes = sharp_ls_get_modes,
};
static int sharp_ls_get_gpio_of(struct device *dev, int index, int val,
dssdev->dev = &pdev->dev;
dssdev->ops = &sharp_ls_ops;
dssdev->type = OMAP_DISPLAY_TYPE_DPI;
+ dssdev->display = true;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(0);
+ dssdev->ops_flags = OMAP_DSS_DEVICE_OP_MODES;
/*
* Note: According to the panel documentation:
* DATA needs to be driven on the FALLING edge
*/
- dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_SYNC_NEGEDGE
- | DRM_BUS_FLAG_PIXDATA_POSEDGE;
+ dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH
+ | DRM_BUS_FLAG_SYNC_DRIVE_NEGEDGE
+ | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE;
omapdss_display_init(dssdev);
omapdss_device_register(dssdev);
omapdss_device_unregister(dssdev);
- sharp_ls_disable(dssdev);
+ if (omapdss_device_is_enabled(dssdev)) {
+ sharp_ls_disable(dssdev);
+ sharp_ls_post_disable(dssdev);
+ }
return 0;
}
static int acx565akm_panel_power_on(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
- int r;
dev_dbg(&ddata->spi->dev, "%s\n", __func__);
- r = src->ops->enable(src);
- if (r) {
- pr_err("%s sdi enable failed\n", __func__);
- return r;
- }
-
/*FIXME tweak me */
msleep(50);
static void acx565akm_panel_power_off(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
dev_dbg(dssdev->dev, "%s\n", __func__);
/* FIXME need to tweak this delay */
msleep(100);
-
- src->ops->disable(src);
}
-static int acx565akm_enable(struct omap_dss_device *dssdev)
+static void acx565akm_enable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- int r;
-
- dev_dbg(dssdev->dev, "%s\n", __func__);
-
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
mutex_lock(&ddata->mutex);
- r = acx565akm_panel_power_on(dssdev);
+ acx565akm_panel_power_on(dssdev);
mutex_unlock(&ddata->mutex);
- if (r)
- return r;
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return 0;
}
static void acx565akm_disable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- dev_dbg(dssdev->dev, "%s\n", __func__);
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
-
mutex_lock(&ddata->mutex);
acx565akm_panel_power_off(dssdev);
mutex_unlock(&ddata->mutex);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
}
-static void acx565akm_get_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+static int acx565akm_get_modes(struct omap_dss_device *dssdev,
+ struct drm_connector *connector)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- *vm = ddata->vm;
+ return omapdss_display_get_modes(connector, &ddata->vm);
}
static const struct omap_dss_device_ops acx565akm_ops = {
.enable = acx565akm_enable,
.disable = acx565akm_disable,
- .get_timings = acx565akm_get_timings,
+ .get_modes = acx565akm_get_modes,
};
static int acx565akm_probe(struct spi_device *spi)
dssdev->dev = &spi->dev;
dssdev->ops = &acx565akm_ops;
dssdev->type = OMAP_DISPLAY_TYPE_SDI;
+ dssdev->display = true;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(0);
- dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_SYNC_NEGEDGE
- | DRM_BUS_FLAG_PIXDATA_POSEDGE;
+ dssdev->ops_flags = OMAP_DSS_DEVICE_OP_MODES;
+ dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH
+ | DRM_BUS_FLAG_SYNC_DRIVE_NEGEDGE
+ | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE;
omapdss_display_init(dssdev);
omapdss_device_register(dssdev);
omapdss_device_unregister(dssdev);
- acx565akm_disable(dssdev);
+ if (omapdss_device_is_enabled(dssdev))
+ acx565akm_disable(dssdev);
return 0;
}
struct videomode vm;
+ struct backlight_device *backlight;
+
struct spi_device *spi_dev;
};
{
}
-static int td028ttec1_panel_enable(struct omap_dss_device *dssdev)
+static void td028ttec1_panel_enable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
- int r;
-
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
+ int r = 0;
- dev_dbg(dssdev->dev, "td028ttec1_panel_enable() - state %d\n",
- dssdev->state);
+ dev_dbg(dssdev->dev, "%s: state %d\n", __func__, dssdev->state);
/* three times command zero */
r |= jbt_ret_write_0(ddata, 0x00);
usleep_range(1000, 2000);
if (r) {
- dev_warn(dssdev->dev, "transfer error\n");
- goto transfer_err;
+ dev_warn(dssdev->dev, "%s: transfer error\n", __func__);
+ return;
}
/* deep standby out */
r |= jbt_ret_write_0(ddata, JBT_REG_DISPLAY_ON);
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
-transfer_err:
+ if (r)
+ dev_err(dssdev->dev, "%s: write error\n", __func__);
- return r ? -EIO : 0;
+ backlight_enable(ddata->backlight);
}
static void td028ttec1_panel_disable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
- if (!omapdss_device_is_enabled(dssdev))
- return;
+ backlight_disable(ddata->backlight);
dev_dbg(dssdev->dev, "td028ttec1_panel_disable()\n");
jbt_reg_write_2(ddata, JBT_REG_OUTPUT_CONTROL, 0x8002);
jbt_ret_write_0(ddata, JBT_REG_SLEEP_IN);
jbt_reg_write_1(ddata, JBT_REG_POWER_ON_OFF, 0x00);
-
- src->ops->disable(src);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
}
-static void td028ttec1_panel_get_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+static int td028ttec1_panel_get_modes(struct omap_dss_device *dssdev,
+ struct drm_connector *connector)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- *vm = ddata->vm;
+ return omapdss_display_get_modes(connector, &ddata->vm);
}
static const struct omap_dss_device_ops td028ttec1_ops = {
.enable = td028ttec1_panel_enable,
.disable = td028ttec1_panel_disable,
- .get_timings = td028ttec1_panel_get_timings,
+ .get_modes = td028ttec1_panel_get_modes,
};
static int td028ttec1_panel_probe(struct spi_device *spi)
if (ddata == NULL)
return -ENOMEM;
+ ddata->backlight = devm_of_find_backlight(&spi->dev);
+ if (IS_ERR(ddata->backlight))
+ return PTR_ERR(ddata->backlight);
+
dev_set_drvdata(&spi->dev, ddata);
ddata->spi_dev = spi;
dssdev->dev = &spi->dev;
dssdev->ops = &td028ttec1_ops;
dssdev->type = OMAP_DISPLAY_TYPE_DPI;
+ dssdev->display = true;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(0);
+ dssdev->ops_flags = OMAP_DSS_DEVICE_OP_MODES;
/*
* Note: According to the panel documentation:
* SYNC needs to be driven on the FALLING edge
*/
- dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_SYNC_POSEDGE
- | DRM_BUS_FLAG_PIXDATA_NEGEDGE;
+ dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH
+ | DRM_BUS_FLAG_SYNC_DRIVE_POSEDGE
+ | DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE;
omapdss_display_init(dssdev);
omapdss_device_register(dssdev);
{
}
-static int tpo_td043_enable(struct omap_dss_device *dssdev)
+static void tpo_td043_enable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
int r;
- if (!omapdss_device_is_connected(dssdev))
- return -ENODEV;
-
- if (omapdss_device_is_enabled(dssdev))
- return 0;
-
- r = src->ops->enable(src);
- if (r)
- return r;
-
/*
* If we are resuming from system suspend, SPI clocks might not be
* enabled yet, so we'll program the LCD from SPI PM resume callback.
if (!ddata->spi_suspended) {
r = tpo_td043_power_on(ddata);
if (r) {
- src->ops->disable(src);
- return r;
+ dev_err(&ddata->spi->dev, "%s: power on failed (%d)\n",
+ __func__, r);
+ return;
}
}
-
- dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
-
- return 0;
}
static void tpo_td043_disable(struct omap_dss_device *dssdev)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- struct omap_dss_device *src = dssdev->src;
-
- if (!omapdss_device_is_enabled(dssdev))
- return;
-
- src->ops->disable(src);
if (!ddata->spi_suspended)
tpo_td043_power_off(ddata);
-
- dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
}
-static void tpo_td043_get_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+static int tpo_td043_get_modes(struct omap_dss_device *dssdev,
+ struct drm_connector *connector)
{
struct panel_drv_data *ddata = to_panel_data(dssdev);
- *vm = ddata->vm;
+ return omapdss_display_get_modes(connector, &ddata->vm);
}
static const struct omap_dss_device_ops tpo_td043_ops = {
.enable = tpo_td043_enable,
.disable = tpo_td043_disable,
- .get_timings = tpo_td043_get_timings,
+ .get_modes = tpo_td043_get_modes,
};
static int tpo_td043_probe(struct spi_device *spi)
dssdev->dev = &spi->dev;
dssdev->ops = &tpo_td043_ops;
dssdev->type = OMAP_DISPLAY_TYPE_DPI;
+ dssdev->display = true;
dssdev->owner = THIS_MODULE;
dssdev->of_ports = BIT(0);
+ dssdev->ops_flags = OMAP_DSS_DEVICE_OP_MODES;
/*
* Note: According to the panel documentation:
* SYNC needs to be driven on the FALLING edge
*/
- dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_SYNC_POSEDGE
- | DRM_BUS_FLAG_PIXDATA_NEGEDGE;
+ dssdev->bus_flags = DRM_BUS_FLAG_DE_HIGH
+ | DRM_BUS_FLAG_SYNC_DRIVE_POSEDGE
+ | DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE;
omapdss_display_init(dssdev);
omapdss_device_register(dssdev);
omapdss_device_unregister(dssdev);
- tpo_td043_disable(dssdev);
+ if (omapdss_device_is_enabled(dssdev))
+ tpo_td043_disable(dssdev);
sysfs_remove_group(&spi->dev.kobj, &tpo_td043_attr_group);
#include <linux/mutex.h>
#include <linux/of.h>
#include <linux/of_graph.h>
+#include <linux/platform_device.h>
#include "dss.h"
#include "omapdss.h"
}
EXPORT_SYMBOL(omapdss_device_put);
-struct omap_dss_device *omapdss_find_device_by_port(struct device_node *src,
- unsigned int port)
+struct omap_dss_device *omapdss_find_device_by_node(struct device_node *node)
{
struct omap_dss_device *dssdev;
list_for_each_entry(dssdev, &omapdss_devices_list, list) {
- if (dssdev->dev->of_node == src && dssdev->of_ports & BIT(port))
+ if (dssdev->dev->of_node == node)
return omapdss_device_get(dssdev);
}
}
/*
- * Search for the next device starting at @from. The type argument specfies
- * which device types to consider when searching. Searching for multiple types
- * is supported by and'ing their type flags. Release the reference to the @from
- * device, and acquire a reference to the returned device if found.
+ * Search for the next output device starting at @from. Release the reference to
+ * the @from device, and acquire a reference to the returned device if found.
*/
-struct omap_dss_device *omapdss_device_get_next(struct omap_dss_device *from,
- enum omap_dss_device_type type)
+struct omap_dss_device *omapdss_device_next_output(struct omap_dss_device *from)
{
struct omap_dss_device *dssdev;
struct list_head *list;
goto done;
}
- /*
- * Accept display entities if the display type is requested,
- * and output entities if the output type is requested.
- */
- if ((type & OMAP_DSS_DEVICE_TYPE_DISPLAY) &&
- !dssdev->output_type)
- goto done;
- if ((type & OMAP_DSS_DEVICE_TYPE_OUTPUT) && dssdev->id &&
- dssdev->next)
+ if (dssdev->id &&
+ (dssdev->next || dssdev->bridge || dssdev->panel))
goto done;
}
mutex_unlock(&omapdss_devices_lock);
return dssdev;
}
-EXPORT_SYMBOL(omapdss_device_get_next);
+EXPORT_SYMBOL(omapdss_device_next_output);
+
+static bool omapdss_device_is_connected(struct omap_dss_device *dssdev)
+{
+ return dssdev->dss;
+}
int omapdss_device_connect(struct dss_device *dss,
struct omap_dss_device *src,
{
int ret;
- dev_dbg(dst->dev, "connect\n");
+ dev_dbg(&dss->pdev->dev, "connect(%s, %s)\n",
+ src ? dev_name(src->dev) : "NULL",
+ dst ? dev_name(dst->dev) : "NULL");
+
+ if (!dst) {
+ /*
+ * The destination is NULL when the source is connected to a
+ * bridge or panel instead of a DSS device. Stop here, we will
+ * attach the bridge or panel later when we will have a DRM
+ * encoder.
+ */
+ return src && (src->bridge || src->panel) ? 0 : -EINVAL;
+ }
if (omapdss_device_is_connected(dst))
return -EBUSY;
return ret;
}
- if (src) {
- WARN_ON(src->dst);
- dst->src = src;
- src->dst = dst;
- }
-
return 0;
}
EXPORT_SYMBOL_GPL(omapdss_device_connect);
void omapdss_device_disconnect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- dev_dbg(dst->dev, "disconnect\n");
+ struct dss_device *dss = src ? src->dss : dst->dss;
- if (!dst->id && !omapdss_device_is_connected(dst)) {
- WARN_ON(dst->output_type);
+ dev_dbg(&dss->pdev->dev, "disconnect(%s, %s)\n",
+ src ? dev_name(src->dev) : "NULL",
+ dst ? dev_name(dst->dev) : "NULL");
+
+ if (!dst) {
+ WARN_ON(!src->bridge && !src->panel);
return;
}
- if (src) {
- if (WARN_ON(dst != src->dst))
- return;
-
- dst->src = NULL;
- src->dst = NULL;
+ if (!dst->id && !omapdss_device_is_connected(dst)) {
+ WARN_ON(!dst->display);
+ return;
}
WARN_ON(dst->state != OMAP_DSS_DISPLAY_DISABLED);
}
EXPORT_SYMBOL_GPL(omapdss_device_disconnect);
+void omapdss_device_pre_enable(struct omap_dss_device *dssdev)
+{
+ if (!dssdev)
+ return;
+
+ omapdss_device_pre_enable(dssdev->next);
+
+ if (dssdev->ops->pre_enable)
+ dssdev->ops->pre_enable(dssdev);
+}
+EXPORT_SYMBOL_GPL(omapdss_device_pre_enable);
+
+void omapdss_device_enable(struct omap_dss_device *dssdev)
+{
+ if (!dssdev)
+ return;
+
+ if (dssdev->ops->enable)
+ dssdev->ops->enable(dssdev);
+
+ omapdss_device_enable(dssdev->next);
+
+ dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
+}
+EXPORT_SYMBOL_GPL(omapdss_device_enable);
+
+void omapdss_device_disable(struct omap_dss_device *dssdev)
+{
+ if (!dssdev)
+ return;
+
+ omapdss_device_disable(dssdev->next);
+
+ if (dssdev->ops->disable)
+ dssdev->ops->disable(dssdev);
+}
+EXPORT_SYMBOL_GPL(omapdss_device_disable);
+
+void omapdss_device_post_disable(struct omap_dss_device *dssdev)
+{
+ if (!dssdev)
+ return;
+
+ if (dssdev->ops->post_disable)
+ dssdev->ops->post_disable(dssdev);
+
+ omapdss_device_post_disable(dssdev->next);
+
+ dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
+}
+EXPORT_SYMBOL_GPL(omapdss_device_post_disable);
+
/* -----------------------------------------------------------------------------
* Components Handling
*/
struct list_head list;
struct device_node *node;
bool dss_core_component;
+ const char *compat;
};
static bool omapdss_list_contains(const struct device_node *node)
static void omapdss_walk_device(struct device *dev, struct device_node *node,
bool dss_core)
{
+ struct omapdss_comp_node *comp;
struct device_node *n;
- struct omapdss_comp_node *comp = devm_kzalloc(dev, sizeof(*comp),
- GFP_KERNEL);
+ const char *compat;
+ int ret;
+ ret = of_property_read_string(node, "compatible", &compat);
+ if (ret < 0)
+ return;
+
+ comp = devm_kzalloc(dev, sizeof(*comp), GFP_KERNEL);
if (comp) {
comp->node = node;
comp->dss_core_component = dss_core;
+ comp->compat = compat;
list_add(&comp->list, &omapdss_comp_list);
}
omapdss_walk_device(dev, dev->of_node, true);
- for_each_available_child_of_node(dev->of_node, child) {
- if (!of_find_property(child, "compatible", NULL))
- continue;
-
+ for_each_available_child_of_node(dev->of_node, child)
omapdss_walk_device(dev, child, true);
- }
}
EXPORT_SYMBOL(omapdss_gather_components);
{
if (comp->dss_core_component)
return true;
+ if (!strstarts(comp->compat, "omapdss,"))
+ return true;
if (omapdss_device_is_registered(comp->node))
return true;
#include <linux/kernel.h>
#include <linux/of.h>
+#include <drm/drm_connector.h>
+#include <drm/drm_modes.h>
+
#include "omapdss.h"
static int disp_num_counter;
if (id < 0)
id = disp_num_counter++;
- dssdev->alias_id = id;
-
/* Use 'label' property for name, if it exists */
of_property_read_string(dssdev->dev->of_node, "label", &dssdev->name);
return omapdss_device_get(output);
}
EXPORT_SYMBOL_GPL(omapdss_display_get);
+
+int omapdss_display_get_modes(struct drm_connector *connector,
+ const struct videomode *vm)
+{
+ struct drm_display_mode *mode;
+
+ mode = drm_mode_create(connector->dev);
+ if (!mode)
+ return 0;
+
+ drm_display_mode_from_videomode(vm, mode);
+
+ mode->type = DRM_MODE_TYPE_DRIVER | DRM_MODE_TYPE_PREFERRED;
+ drm_mode_set_name(mode);
+ drm_mode_probed_add(connector, mode);
+
+ return 1;
+}
+EXPORT_SYMBOL_GPL(omapdss_display_get_modes);
struct mutex lock;
- struct videomode vm;
struct dss_lcd_mgr_config mgr_config;
+ unsigned long pixelclock;
int data_lines;
struct omap_dss_device output;
static int dpi_set_mode(struct dpi_data *dpi)
{
- const struct videomode *vm = &dpi->vm;
int lck_div = 0, pck_div = 0;
unsigned long fck = 0;
int r = 0;
if (dpi->pll)
r = dpi_set_pll_clk(dpi, dpi->output.dispc_channel,
- vm->pixelclock, &fck, &lck_div, &pck_div);
+ dpi->pixelclock, &fck, &lck_div, &pck_div);
else
- r = dpi_set_dispc_clk(dpi, vm->pixelclock, &fck,
+ r = dpi_set_dispc_clk(dpi, dpi->pixelclock, &fck,
&lck_div, &pck_div);
if (r)
return r;
dss_mgr_set_lcd_config(&dpi->output, &dpi->mgr_config);
}
-static int dpi_display_enable(struct omap_dss_device *dssdev)
+static void dpi_display_enable(struct omap_dss_device *dssdev)
{
struct dpi_data *dpi = dpi_get_data_from_dssdev(dssdev);
struct omap_dss_device *out = &dpi->output;
mutex_lock(&dpi->lock);
- if (!out->dispc_channel_connected) {
- DSSERR("failed to enable display: no output/manager\n");
- r = -ENODEV;
- goto err_no_out_mgr;
- }
-
if (dpi->vdds_dsi_reg) {
r = regulator_enable(dpi->vdds_dsi_reg);
if (r)
mutex_unlock(&dpi->lock);
- return 0;
+ return;
err_mgr_enable:
err_set_mode:
if (dpi->vdds_dsi_reg)
regulator_disable(dpi->vdds_dsi_reg);
err_reg_enable:
-err_no_out_mgr:
mutex_unlock(&dpi->lock);
- return r;
}
static void dpi_display_disable(struct omap_dss_device *dssdev)
}
static void dpi_set_timings(struct omap_dss_device *dssdev,
- const struct videomode *vm)
+ const struct drm_display_mode *mode)
{
struct dpi_data *dpi = dpi_get_data_from_dssdev(dssdev);
mutex_lock(&dpi->lock);
- dpi->vm = *vm;
+ dpi->pixelclock = mode->clock * 1000;
mutex_unlock(&dpi->lock);
}
static int dpi_check_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+ struct drm_display_mode *mode)
{
struct dpi_data *dpi = dpi_get_data_from_dssdev(dssdev);
int lck_div, pck_div;
struct dpi_clk_calc_ctx ctx;
bool ok;
- if (vm->hactive % 8 != 0)
+ if (mode->hdisplay % 8 != 0)
return -EINVAL;
- if (vm->pixelclock == 0)
+ if (mode->clock == 0)
return -EINVAL;
if (dpi->pll) {
- ok = dpi_pll_clk_calc(dpi, vm->pixelclock, &ctx);
+ ok = dpi_pll_clk_calc(dpi, mode->clock * 1000, &ctx);
if (!ok)
return -EINVAL;
fck = ctx.pll_cinfo.clkout[ctx.clkout_idx];
} else {
- ok = dpi_dss_clk_calc(dpi, vm->pixelclock, &ctx);
+ ok = dpi_dss_clk_calc(dpi, mode->clock * 1000, &ctx);
if (!ok)
return -EINVAL;
pck = fck / lck_div / pck_div;
- vm->pixelclock = pck;
+ mode->clock = pck / 1000;
return 0;
}
struct omap_dss_device *dst)
{
struct dpi_data *dpi = dpi_get_data_from_dssdev(dst);
- int r;
dpi_init_pll(dpi);
- r = omapdss_device_connect(dst->dss, dst, dst->next);
- if (r)
- return r;
-
- dst->dispc_channel_connected = true;
- return 0;
+ return omapdss_device_connect(dst->dss, dst, dst->next);
}
static void dpi_disconnect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- dst->dispc_channel_connected = false;
-
omapdss_device_disconnect(dst, dst->next);
}
out->dev = &dpi->pdev->dev;
out->id = OMAP_DSS_OUTPUT_DPI;
- out->output_type = OMAP_DISPLAY_TYPE_DPI;
+ out->type = OMAP_DISPLAY_TYPE_DPI;
out->dispc_channel = dpi_get_channel(dpi);
out->of_ports = BIT(port_num);
out->ops = &dpi_ops;
out->owner = THIS_MODULE;
- out->next = omapdss_of_find_connected_device(out->dev->of_node, 0);
- if (IS_ERR(out->next)) {
- if (PTR_ERR(out->next) != -EPROBE_DEFER)
- dev_err(out->dev, "failed to find video sink\n");
- return PTR_ERR(out->next);
- }
-
- r = omapdss_output_validate(out);
- if (r) {
- omapdss_device_put(out->next);
- out->next = NULL;
+ r = omapdss_device_init_output(out);
+ if (r < 0)
return r;
- }
omapdss_device_register(out);
struct dpi_data *dpi = port->data;
struct omap_dss_device *out = &dpi->output;
- if (out->next)
- omapdss_device_put(out->next);
omapdss_device_unregister(out);
+ omapdss_device_cleanup_output(out);
}
static const struct soc_device_attribute dpi_soc_devices[] = {
*/
dsi_enable_scp_clk(dsi);
- if (!dsi->vdds_dsi_enabled) {
- r = regulator_enable(dsi->vdds_dsi_reg);
- if (r)
- goto err0;
- dsi->vdds_dsi_enabled = true;
- }
+ r = regulator_enable(dsi->vdds_dsi_reg);
+ if (r)
+ goto err0;
/* XXX PLL does not come out of reset without this... */
dispc_pck_free_enable(dsi->dss->dispc, 1);
return 0;
err1:
- if (dsi->vdds_dsi_enabled) {
- regulator_disable(dsi->vdds_dsi_reg);
- dsi->vdds_dsi_enabled = false;
- }
+ regulator_disable(dsi->vdds_dsi_reg);
err0:
dsi_disable_scp_clk(dsi);
dsi_runtime_put(dsi);
return r;
}
-static void dsi_pll_uninit(struct dsi_data *dsi, bool disconnect_lanes)
+static void dsi_pll_disable(struct dss_pll *pll)
{
+ struct dsi_data *dsi = container_of(pll, struct dsi_data, pll);
+
dsi_pll_power(dsi, DSI_PLL_POWER_OFF);
- if (disconnect_lanes) {
- WARN_ON(!dsi->vdds_dsi_enabled);
- regulator_disable(dsi->vdds_dsi_reg);
- dsi->vdds_dsi_enabled = false;
- }
+
+ regulator_disable(dsi->vdds_dsi_reg);
dsi_disable_scp_clk(dsi);
dsi_runtime_put(dsi);
- DSSDBG("PLL uninit done\n");
-}
-
-static void dsi_pll_disable(struct dss_pll *pll)
-{
- struct dsi_data *dsi = container_of(pll, struct dsi_data, pll);
-
- dsi_pll_uninit(dsi, true);
+ DSSDBG("PLL disable done\n");
}
static int dsi_dump_dsi_clocks(struct seq_file *s, void *p)
{
struct dsi_data *dsi = to_dsi_data(dssdev);
int bpp = dsi_get_pixel_size(dsi->pix_fmt);
- struct omap_dss_device *out = &dsi->output;
u8 data_type;
u16 word_count;
int r;
- if (!out->dispc_channel_connected) {
- DSSERR("failed to enable display: no output/manager\n");
- return -ENODEV;
- }
-
r = dsi_display_init_dispc(dsi);
if (r)
- goto err_init_dispc;
+ return r;
if (dsi->mode == OMAP_DSS_DSI_VIDEO_MODE) {
switch (dsi->pix_fmt) {
}
err_pix_fmt:
dsi_display_uninit_dispc(dsi);
-err_init_dispc:
return r;
}
r = dss_pll_enable(&dsi->pll);
if (r)
- goto err0;
+ return r;
r = dsi_configure_dsi_clocks(dsi);
if (r)
- goto err1;
+ goto err0;
dss_select_dsi_clk_source(dsi->dss, dsi->module_id,
dsi->module_id == 0 ?
DSSDBG("PLL OK\n");
+ if (!dsi->vdds_dsi_enabled) {
+ r = regulator_enable(dsi->vdds_dsi_reg);
+ if (r)
+ goto err1;
+
+ dsi->vdds_dsi_enabled = true;
+ }
+
r = dsi_cio_init(dsi);
if (r)
goto err2;
err3:
dsi_cio_uninit(dsi);
err2:
- dss_select_dsi_clk_source(dsi->dss, dsi->module_id, DSS_CLK_SRC_FCK);
+ regulator_disable(dsi->vdds_dsi_reg);
+ dsi->vdds_dsi_enabled = false;
err1:
- dss_pll_disable(&dsi->pll);
+ dss_select_dsi_clk_source(dsi->dss, dsi->module_id, DSS_CLK_SRC_FCK);
err0:
+ dss_pll_disable(&dsi->pll);
+
return r;
}
dss_select_dsi_clk_source(dsi->dss, dsi->module_id, DSS_CLK_SRC_FCK);
dsi_cio_uninit(dsi);
- dsi_pll_uninit(dsi, disconnect_lanes);
+ dss_pll_disable(&dsi->pll);
+
+ if (disconnect_lanes) {
+ regulator_disable(dsi->vdds_dsi_reg);
+ dsi->vdds_dsi_enabled = false;
+ }
}
-static int dsi_display_enable(struct omap_dss_device *dssdev)
+static void dsi_display_enable(struct omap_dss_device *dssdev)
{
struct dsi_data *dsi = to_dsi_data(dssdev);
- int r = 0;
+ int r;
DSSDBG("dsi_display_enable\n");
mutex_unlock(&dsi->lock);
- return 0;
+ return;
err_init_dsi:
dsi_runtime_put(dsi);
err_get_dsi:
mutex_unlock(&dsi->lock);
DSSDBG("dsi_display_enable FAILED\n");
- return r;
}
static void dsi_display_disable(struct omap_dss_device *dssdev,
static int dsi_connect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- int r;
-
- r = omapdss_device_connect(dst->dss, dst, dst->next);
- if (r)
- return r;
-
- dst->dispc_channel_connected = true;
- return 0;
+ return omapdss_device_connect(dst->dss, dst, dst->next);
}
static void dsi_disconnect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- dst->dispc_channel_connected = false;
-
omapdss_device_disconnect(dst, dst->next);
}
out->id = dsi->module_id == 0 ?
OMAP_DSS_OUTPUT_DSI1 : OMAP_DSS_OUTPUT_DSI2;
- out->output_type = OMAP_DISPLAY_TYPE_DSI;
+ out->type = OMAP_DISPLAY_TYPE_DSI;
out->name = dsi->module_id == 0 ? "dsi.0" : "dsi.1";
out->dispc_channel = dsi_get_channel(dsi);
out->ops = &dsi_ops;
out->owner = THIS_MODULE;
out->of_ports = BIT(0);
- out->bus_flags = DRM_BUS_FLAG_PIXDATA_POSEDGE
+ out->bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE
| DRM_BUS_FLAG_DE_HIGH
- | DRM_BUS_FLAG_SYNC_NEGEDGE;
-
- out->next = omapdss_of_find_connected_device(out->dev->of_node, 0);
- if (IS_ERR(out->next)) {
- if (PTR_ERR(out->next) != -EPROBE_DEFER)
- dev_err(out->dev, "failed to find video sink\n");
- return PTR_ERR(out->next);
- }
+ | DRM_BUS_FLAG_SYNC_DRIVE_NEGEDGE;
- r = omapdss_output_validate(out);
- if (r) {
- omapdss_device_put(out->next);
- out->next = NULL;
+ r = omapdss_device_init_output(out);
+ if (r < 0)
return r;
- }
omapdss_device_register(out);
{
struct omap_dss_device *out = &dsi->output;
- if (out->next)
- omapdss_device_put(out->next);
omapdss_device_unregister(out);
+ omapdss_device_cleanup_output(out);
}
static int dsi_probe_of(struct dsi_data *dsi)
* more details.
*/
-#include <linux/device.h>
#include <linux/err.h>
-#include <linux/module.h>
#include <linux/of.h>
#include <linux/of_graph.h>
-#include <linux/seq_file.h>
#include "omapdss.h"
-static struct device_node *
-dss_of_port_get_parent_device(struct device_node *port)
-{
- struct device_node *np;
- int i;
-
- if (!port)
- return NULL;
-
- np = of_get_parent(port);
-
- for (i = 0; i < 2 && np; ++i) {
- struct property *prop;
-
- prop = of_find_property(np, "compatible", NULL);
-
- if (prop)
- return np;
-
- np = of_get_next_parent(np);
- }
-
- return NULL;
-}
-
struct omap_dss_device *
omapdss_of_find_connected_device(struct device_node *node, unsigned int port)
{
- struct device_node *src_node;
- struct device_node *src_port;
- struct device_node *ep;
- struct omap_dss_device *src;
- u32 port_number = 0;
+ struct device_node *remote_node;
+ struct omap_dss_device *dssdev;
- /* Get the endpoint... */
- ep = of_graph_get_endpoint_by_regs(node, port, 0);
- if (!ep)
+ remote_node = of_graph_get_remote_node(node, port, 0);
+ if (!remote_node)
return NULL;
- /* ... and its remote port... */
- src_port = of_graph_get_remote_port(ep);
- of_node_put(ep);
- if (!src_port)
- return NULL;
-
- /* ... and the remote port's number and parent... */
- of_property_read_u32(src_port, "reg", &port_number);
- src_node = dss_of_port_get_parent_device(src_port);
- of_node_put(src_port);
- if (!src_node)
- return ERR_PTR(-EINVAL);
-
- /* ... and finally the connected device. */
- src = omapdss_find_device_by_port(src_node, port_number);
- of_node_put(src_node);
+ dssdev = omapdss_find_device_by_node(remote_node);
+ of_node_put(remote_node);
- return src ? src : ERR_PTR(-EPROBE_DEFER);
+ return dssdev ? dssdev : ERR_PTR(-EPROBE_DEFER);
}
EXPORT_SYMBOL_GPL(omapdss_of_find_connected_device);
DSSDBG("shutdown\n");
- for_each_dss_display(dssdev) {
+ for_each_dss_output(dssdev) {
if (dssdev->state == OMAP_DSS_DISPLAY_ACTIVE)
dssdev->ops->disable(dssdev);
}
}
static void hdmi_display_set_timings(struct omap_dss_device *dssdev,
- const struct videomode *vm)
+ const struct drm_display_mode *mode)
{
struct omap_hdmi *hdmi = dssdev_to_hdmi(dssdev);
mutex_lock(&hdmi->lock);
- hdmi->cfg.vm = *vm;
+ drm_display_mode_to_videomode(mode, &hdmi->cfg.vm);
- dispc_set_tv_pclk(hdmi->dss->dispc, vm->pixelclock);
+ dispc_set_tv_pclk(hdmi->dss->dispc, mode->clock * 1000);
mutex_unlock(&hdmi->lock);
}
hdmi_wp_audio_enable(&hd->wp, false);
}
-static int hdmi_display_enable(struct omap_dss_device *dssdev)
+static void hdmi_display_enable(struct omap_dss_device *dssdev)
{
struct omap_hdmi *hdmi = dssdev_to_hdmi(dssdev);
unsigned long flags;
- int r = 0;
+ int r;
DSSDBG("ENTER hdmi_display_enable\n");
mutex_lock(&hdmi->lock);
- if (!dssdev->dispc_channel_connected) {
- DSSERR("failed to enable display: no output/manager\n");
- r = -ENODEV;
- goto err0;
- }
-
r = hdmi_power_on_full(hdmi);
if (r) {
DSSERR("failed to power on device\n");
- goto err0;
+ goto done;
}
if (hdmi->audio_configured) {
hdmi->display_enabled = true;
spin_unlock_irqrestore(&hdmi->audio_playing_lock, flags);
+done:
mutex_unlock(&hdmi->lock);
- return 0;
-
-err0:
- mutex_unlock(&hdmi->lock);
- return r;
}
static void hdmi_display_disable(struct omap_dss_device *dssdev)
static int hdmi_connect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- int r;
-
- r = omapdss_device_connect(dst->dss, dst, dst->next);
- if (r)
- return r;
-
- dst->dispc_channel_connected = true;
- return 0;
+ return omapdss_device_connect(dst->dss, dst, dst->next);
}
static void hdmi_disconnect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- dst->dispc_channel_connected = false;
-
omapdss_device_disconnect(dst, dst->next);
}
out->dev = &hdmi->pdev->dev;
out->id = OMAP_DSS_OUTPUT_HDMI;
- out->output_type = OMAP_DISPLAY_TYPE_HDMI;
+ out->type = OMAP_DISPLAY_TYPE_HDMI;
out->name = "hdmi.0";
out->dispc_channel = OMAP_DSS_CHANNEL_DIGIT;
out->ops = &hdmi_ops;
out->of_ports = BIT(0);
out->ops_flags = OMAP_DSS_DEVICE_OP_EDID;
- out->next = omapdss_of_find_connected_device(out->dev->of_node, 0);
- if (IS_ERR(out->next)) {
- if (PTR_ERR(out->next) != -EPROBE_DEFER)
- dev_err(out->dev, "failed to find video sink\n");
- return PTR_ERR(out->next);
- }
-
- r = omapdss_output_validate(out);
- if (r) {
- omapdss_device_put(out->next);
- out->next = NULL;
+ r = omapdss_device_init_output(out);
+ if (r < 0)
return r;
- }
omapdss_device_register(out);
{
struct omap_dss_device *out = &hdmi->output;
- if (out->next)
- omapdss_device_put(out->next);
omapdss_device_unregister(out);
+ omapdss_device_cleanup_output(out);
}
static int hdmi4_probe_of(struct omap_hdmi *hdmi)
}
static void hdmi_display_set_timings(struct omap_dss_device *dssdev,
- const struct videomode *vm)
+ const struct drm_display_mode *mode)
{
struct omap_hdmi *hdmi = dssdev_to_hdmi(dssdev);
mutex_lock(&hdmi->lock);
- hdmi->cfg.vm = *vm;
+ drm_display_mode_to_videomode(mode, &hdmi->cfg.vm);
- dispc_set_tv_pclk(hdmi->dss->dispc, vm->pixelclock);
+ dispc_set_tv_pclk(hdmi->dss->dispc, mode->clock * 1000);
mutex_unlock(&hdmi->lock);
}
REG_FLD_MOD(hd->wp.base, HDMI_WP_SYSCONFIG, hd->wp_idlemode, 3, 2);
}
-static int hdmi_display_enable(struct omap_dss_device *dssdev)
+static void hdmi_display_enable(struct omap_dss_device *dssdev)
{
struct omap_hdmi *hdmi = dssdev_to_hdmi(dssdev);
unsigned long flags;
- int r = 0;
+ int r;
DSSDBG("ENTER hdmi_display_enable\n");
mutex_lock(&hdmi->lock);
- if (!dssdev->dispc_channel_connected) {
- DSSERR("failed to enable display: no output/manager\n");
- r = -ENODEV;
- goto err0;
- }
-
r = hdmi_power_on_full(hdmi);
if (r) {
DSSERR("failed to power on device\n");
- goto err0;
+ goto done;
}
if (hdmi->audio_configured) {
hdmi->display_enabled = true;
spin_unlock_irqrestore(&hdmi->audio_playing_lock, flags);
+done:
mutex_unlock(&hdmi->lock);
- return 0;
-
-err0:
- mutex_unlock(&hdmi->lock);
- return r;
}
static void hdmi_display_disable(struct omap_dss_device *dssdev)
static int hdmi_connect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- int r;
-
- r = omapdss_device_connect(dst->dss, dst, dst->next);
- if (r)
- return r;
-
- dst->dispc_channel_connected = true;
- return 0;
+ return omapdss_device_connect(dst->dss, dst, dst->next);
}
static void hdmi_disconnect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- dst->dispc_channel_connected = false;
-
omapdss_device_disconnect(dst, dst->next);
}
out->dev = &hdmi->pdev->dev;
out->id = OMAP_DSS_OUTPUT_HDMI;
- out->output_type = OMAP_DISPLAY_TYPE_HDMI;
+ out->type = OMAP_DISPLAY_TYPE_HDMI;
out->name = "hdmi.0";
out->dispc_channel = OMAP_DSS_CHANNEL_DIGIT;
out->ops = &hdmi_ops;
out->of_ports = BIT(0);
out->ops_flags = OMAP_DSS_DEVICE_OP_EDID;
- out->next = omapdss_of_find_connected_device(out->dev->of_node, 0);
- if (IS_ERR(out->next)) {
- if (PTR_ERR(out->next) != -EPROBE_DEFER)
- dev_err(out->dev, "failed to find video sink\n");
- return PTR_ERR(out->next);
- }
-
- r = omapdss_output_validate(out);
- if (r) {
- omapdss_device_put(out->next);
- out->next = NULL;
+ r = omapdss_device_init_output(out);
+ if (r < 0)
return r;
- }
omapdss_device_register(out);
{
struct omap_dss_device *out = &hdmi->output;
- if (out->next)
- omapdss_device_put(out->next);
omapdss_device_unregister(out);
+ omapdss_device_cleanup_output(out);
}
static int hdmi5_probe_of(struct omap_hdmi *hdmi)
{},
};
+static const struct of_device_id omapdss_of_fixups_whitelist[] __initconst = {
+ { .compatible = "composite-video-connector" },
+ { .compatible = "hdmi-connector" },
+ { .compatible = "lgphilips,lb035q02" },
+ { .compatible = "nec,nl8048hl11" },
+ { .compatible = "panel-dsi-cm" },
+ { .compatible = "sharp,ls037v7dw01" },
+ { .compatible = "sony,acx565akm" },
+ { .compatible = "svideo-connector" },
+ { .compatible = "ti,opa362" },
+ { .compatible = "ti,tpd12s015" },
+ { .compatible = "toppoly,td028ttec1" },
+ { .compatible = "tpo,td028ttec1" },
+ { .compatible = "tpo,td043mtea1" },
+};
+
static int __init omapdss_boot_init(void)
{
struct device_node *dss, *child;
n = list_first_entry(&dss_conv_list, struct dss_conv_node,
list);
- if (!n->root)
+ if (of_match_node(omapdss_of_fixups_whitelist, n->node))
omapdss_omapify_node(n->node);
list_del(&n->list);
#define __OMAP_DRM_DSS_H
#include <linux/list.h>
-#include <linux/kobject.h>
#include <linux/device.h>
#include <linux/interrupt.h>
#include <video/videomode.h>
struct snd_aes_iec958;
struct snd_cea_861_aud_if;
struct hdmi_avi_infoframe;
+struct drm_connector;
enum omap_display_type {
OMAP_DISPLAY_TYPE_NONE = 0,
void (*disconnect)(struct omap_dss_device *dssdev,
struct omap_dss_device *dst);
- int (*enable)(struct omap_dss_device *dssdev);
+ void (*pre_enable)(struct omap_dss_device *dssdev);
+ void (*enable)(struct omap_dss_device *dssdev);
void (*disable)(struct omap_dss_device *dssdev);
+ void (*post_disable)(struct omap_dss_device *dssdev);
int (*check_timings)(struct omap_dss_device *dssdev,
- struct videomode *vm);
- void (*get_timings)(struct omap_dss_device *dssdev,
- struct videomode *vm);
+ struct drm_display_mode *mode);
void (*set_timings)(struct omap_dss_device *dssdev,
- const struct videomode *vm);
+ const struct drm_display_mode *mode);
bool (*detect)(struct omap_dss_device *dssdev);
int (*read_edid)(struct omap_dss_device *dssdev, u8 *buf, int len);
+ int (*get_modes)(struct omap_dss_device *dssdev,
+ struct drm_connector *connector);
+
union {
const struct omapdss_hdmi_ops hdmi;
const struct omapdss_dsi_ops dsi;
* enum omap_dss_device_ops_flag - Indicates which device ops are supported
* @OMAP_DSS_DEVICE_OP_DETECT: The device supports output connection detection
* @OMAP_DSS_DEVICE_OP_HPD: The device supports all hot-plug-related operations
- * @OMAP_DSS_DEVICE_OP_EDID: The device supports readind EDID
+ * @OMAP_DSS_DEVICE_OP_EDID: The device supports reading EDID
+ * @OMAP_DSS_DEVICE_OP_MODES: The device supports reading modes
*/
enum omap_dss_device_ops_flag {
OMAP_DSS_DEVICE_OP_DETECT = BIT(0),
OMAP_DSS_DEVICE_OP_HPD = BIT(1),
OMAP_DSS_DEVICE_OP_EDID = BIT(2),
-};
-
-enum omap_dss_device_type {
- OMAP_DSS_DEVICE_TYPE_OUTPUT = (1 << 0),
- OMAP_DSS_DEVICE_TYPE_DISPLAY = (1 << 1),
+ OMAP_DSS_DEVICE_OP_MODES = BIT(3),
};
struct omap_dss_device {
- struct kobject kobj;
struct device *dev;
struct module *owner;
struct dss_device *dss;
- struct omap_dss_device *src;
- struct omap_dss_device *dst;
struct omap_dss_device *next;
+ struct drm_bridge *bridge;
+ struct drm_panel *panel;
struct list_head list;
- unsigned int alias_id;
-
+ /*
+ * DSS type that this device generates (for DSS internal devices) or
+ * requires (for external encoders, connectors and panels). Must be a
+ * non-zero (different than OMAP_DISPLAY_TYPE_NONE) value.
+ */
enum omap_display_type type;
+
/*
- * DSS output type that this device generates (for DSS internal devices)
- * or requires (for external encoders). Must be OMAP_DISPLAY_TYPE_NONE
- * for display devices (connectors and panels) and to non-zero value for
- * all other devices.
+ * True if the device is a display (panel or connector) at the end of
+ * the pipeline, false otherwise.
*/
- enum omap_display_type output_type;
+ bool display;
const char *name;
unsigned long ops_flags;
u32 bus_flags;
- /* helper variable for driver suspend/resume */
- bool activate_after_resume;
-
enum omap_display_caps caps;
enum omap_dss_display_state state;
/* DISPC channel for this output */
enum omap_channel dispc_channel;
- bool dispc_channel_connected;
/* output instance */
enum omap_dss_output_id id;
int (*memory_read)(struct omap_dss_device *dssdev,
void *buf, size_t size,
u16 x, u16 y, u16 w, u16 h);
-
- void (*get_size)(struct omap_dss_device *dssdev,
- unsigned int *width, unsigned int *height);
};
struct dss_device *omapdss_get_dss(void);
return !!omapdss_get_dss();
}
-#define for_each_dss_display(d) \
- while ((d = omapdss_device_get_next(d, OMAP_DSS_DEVICE_TYPE_DISPLAY)) != NULL)
void omapdss_display_init(struct omap_dss_device *dssdev);
struct omap_dss_device *omapdss_display_get(struct omap_dss_device *output);
+int omapdss_display_get_modes(struct drm_connector *connector,
+ const struct videomode *vm);
void omapdss_device_register(struct omap_dss_device *dssdev);
void omapdss_device_unregister(struct omap_dss_device *dssdev);
struct omap_dss_device *omapdss_device_get(struct omap_dss_device *dssdev);
void omapdss_device_put(struct omap_dss_device *dssdev);
-struct omap_dss_device *omapdss_find_device_by_port(struct device_node *src,
- unsigned int port);
-struct omap_dss_device *omapdss_device_get_next(struct omap_dss_device *from,
- enum omap_dss_device_type type);
+struct omap_dss_device *omapdss_find_device_by_node(struct device_node *node);
int omapdss_device_connect(struct dss_device *dss,
struct omap_dss_device *src,
struct omap_dss_device *dst);
void omapdss_device_disconnect(struct omap_dss_device *src,
struct omap_dss_device *dst);
+void omapdss_device_pre_enable(struct omap_dss_device *dssdev);
+void omapdss_device_enable(struct omap_dss_device *dssdev);
+void omapdss_device_disable(struct omap_dss_device *dssdev);
+void omapdss_device_post_disable(struct omap_dss_device *dssdev);
int omap_dss_get_num_overlay_managers(void);
int omap_dss_get_num_overlays(void);
#define for_each_dss_output(d) \
- while ((d = omapdss_device_get_next(d, OMAP_DSS_DEVICE_TYPE_OUTPUT)) != NULL)
-int omapdss_output_validate(struct omap_dss_device *out);
+ while ((d = omapdss_device_next_output(d)) != NULL)
+struct omap_dss_device *omapdss_device_next_output(struct omap_dss_device *from);
+int omapdss_device_init_output(struct omap_dss_device *out);
+void omapdss_device_cleanup_output(struct omap_dss_device *out);
typedef void (*omap_dispc_isr_t) (void *arg, u32 mask);
int omap_dispc_register_isr(omap_dispc_isr_t isr, void *arg, u32 mask);
int omapdss_compat_init(void);
void omapdss_compat_uninit(void);
-static inline bool omapdss_device_is_connected(struct omap_dss_device *dssdev)
-{
- return dssdev->src;
-}
-
static inline bool omapdss_device_is_enabled(struct omap_dss_device *dssdev)
{
return dssdev->state == OMAP_DSS_DISPLAY_ACTIVE;
#include <linux/platform_device.h>
#include <linux/slab.h>
#include <linux/of.h>
+#include <linux/of_graph.h>
+
+#include <drm/drm_panel.h>
#include "dss.h"
#include "omapdss.h"
-int omapdss_output_validate(struct omap_dss_device *out)
+int omapdss_device_init_output(struct omap_dss_device *out)
{
- if (out->next && out->output_type != out->next->type) {
+ struct device_node *remote_node;
+
+ remote_node = of_graph_get_remote_node(out->dev->of_node, 0, 0);
+ if (!remote_node) {
+ dev_dbg(out->dev, "failed to find video sink\n");
+ return 0;
+ }
+
+ out->next = omapdss_find_device_by_node(remote_node);
+ out->bridge = of_drm_find_bridge(remote_node);
+ out->panel = of_drm_find_panel(remote_node);
+ if (IS_ERR(out->panel))
+ out->panel = NULL;
+
+ of_node_put(remote_node);
+
+ if (out->next && out->type != out->next->type) {
dev_err(out->dev, "output type and display type don't match\n");
+ omapdss_device_put(out->next);
+ out->next = NULL;
return -EINVAL;
}
- return 0;
+ return out->next || out->bridge || out->panel ? 0 : -EPROBE_DEFER;
+}
+EXPORT_SYMBOL(omapdss_device_init_output);
+
+void omapdss_device_cleanup_output(struct omap_dss_device *out)
+{
+ if (out->next)
+ omapdss_device_put(out->next);
}
-EXPORT_SYMBOL(omapdss_output_validate);
+EXPORT_SYMBOL(omapdss_device_cleanup_output);
int dss_install_mgr_ops(struct dss_device *dss,
const struct dss_mgr_ops *mgr_ops,
struct regulator *vdds_sdi_reg;
struct dss_lcd_mgr_config mgr_config;
- struct videomode vm;
+ unsigned long pixelclock;
int datapairs;
struct omap_dss_device output;
dss_mgr_set_lcd_config(&sdi->output, &sdi->mgr_config);
}
-static int sdi_display_enable(struct omap_dss_device *dssdev)
+static void sdi_display_enable(struct omap_dss_device *dssdev)
{
struct sdi_device *sdi = dssdev_to_sdi(dssdev);
struct dispc_clock_info dispc_cinfo;
unsigned long fck;
int r;
- if (!sdi->output.dispc_channel_connected) {
- DSSERR("failed to enable display: no output/manager\n");
- return -ENODEV;
- }
-
r = regulator_enable(sdi->vdds_sdi_reg);
if (r)
- goto err_reg_enable;
+ return;
r = dispc_runtime_get(sdi->dss->dispc);
if (r)
goto err_get_dispc;
- r = sdi_calc_clock_div(sdi, sdi->vm.pixelclock, &fck, &dispc_cinfo);
+ r = sdi_calc_clock_div(sdi, sdi->pixelclock, &fck, &dispc_cinfo);
if (r)
goto err_calc_clock_div;
if (r)
goto err_mgr_enable;
- return 0;
+ return;
err_mgr_enable:
dss_sdi_disable(sdi->dss);
dispc_runtime_put(sdi->dss->dispc);
err_get_dispc:
regulator_disable(sdi->vdds_sdi_reg);
-err_reg_enable:
- return r;
}
static void sdi_display_disable(struct omap_dss_device *dssdev)
}
static void sdi_set_timings(struct omap_dss_device *dssdev,
- const struct videomode *vm)
+ const struct drm_display_mode *mode)
{
struct sdi_device *sdi = dssdev_to_sdi(dssdev);
- sdi->vm = *vm;
+ sdi->pixelclock = mode->clock * 1000;
}
static int sdi_check_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+ struct drm_display_mode *mode)
{
struct sdi_device *sdi = dssdev_to_sdi(dssdev);
struct dispc_clock_info dispc_cinfo;
+ unsigned long pixelclock = mode->clock * 1000;
unsigned long fck;
unsigned long pck;
int r;
- if (vm->pixelclock == 0)
+ if (pixelclock == 0)
return -EINVAL;
- r = sdi_calc_clock_div(sdi, vm->pixelclock, &fck, &dispc_cinfo);
+ r = sdi_calc_clock_div(sdi, pixelclock, &fck, &dispc_cinfo);
if (r)
return r;
pck = fck / dispc_cinfo.lck_div / dispc_cinfo.pck_div;
- if (pck != vm->pixelclock) {
+ if (pck != pixelclock) {
DSSWARN("Pixel clock adjusted from %lu Hz to %lu Hz\n",
- vm->pixelclock, pck);
+ pixelclock, pck);
- vm->pixelclock = pck;
+ mode->clock = pck / 1000;
}
return 0;
static int sdi_connect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- int r;
-
- r = omapdss_device_connect(dst->dss, dst, dst->next);
- if (r)
- return r;
-
- dst->dispc_channel_connected = true;
- return 0;
+ return omapdss_device_connect(dst->dss, dst, dst->next);
}
static void sdi_disconnect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- dst->dispc_channel_connected = false;
-
omapdss_device_disconnect(dst, dst->next);
}
out->dev = &sdi->pdev->dev;
out->id = OMAP_DSS_OUTPUT_SDI;
- out->output_type = OMAP_DISPLAY_TYPE_SDI;
+ out->type = OMAP_DISPLAY_TYPE_SDI;
out->name = "sdi.0";
out->dispc_channel = OMAP_DSS_CHANNEL_LCD;
/* We have SDI only on OMAP3, where it's on port 1 */
out->of_ports = BIT(1);
out->ops = &sdi_ops;
out->owner = THIS_MODULE;
- out->bus_flags = DRM_BUS_FLAG_PIXDATA_POSEDGE /* 15.5.9.1.2 */
- | DRM_BUS_FLAG_SYNC_POSEDGE;
-
- out->next = omapdss_of_find_connected_device(out->dev->of_node, 1);
- if (IS_ERR(out->next)) {
- if (PTR_ERR(out->next) != -EPROBE_DEFER)
- dev_err(out->dev, "failed to find video sink\n");
- return PTR_ERR(out->next);
- }
+ out->bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE /* 15.5.9.1.2 */
+ | DRM_BUS_FLAG_SYNC_DRIVE_POSEDGE;
- r = omapdss_output_validate(out);
- if (r) {
- omapdss_device_put(out->next);
- out->next = NULL;
+ r = omapdss_device_init_output(out);
+ if (r < 0)
return r;
- }
omapdss_device_register(out);
static void sdi_uninit_output(struct sdi_device *sdi)
{
- if (sdi->output.next)
- omapdss_device_put(sdi->output.next);
omapdss_device_unregister(&sdi->output);
+ omapdss_device_cleanup_output(&sdi->output);
}
int sdi_init_port(struct dss_device *dss, struct platform_device *pdev,
VENC_MODE_NTSC,
};
-static const struct videomode omap_dss_pal_vm = {
- .hactive = 720,
- .vactive = 574,
- .pixelclock = 13500000,
- .hsync_len = 64,
- .hfront_porch = 12,
- .hback_porch = 68,
- .vsync_len = 5,
- .vfront_porch = 5,
- .vback_porch = 41,
-
- .flags = DISPLAY_FLAGS_INTERLACED | DISPLAY_FLAGS_HSYNC_LOW |
- DISPLAY_FLAGS_VSYNC_LOW | DISPLAY_FLAGS_DE_HIGH |
- DISPLAY_FLAGS_PIXDATA_POSEDGE |
- DISPLAY_FLAGS_SYNC_NEGEDGE,
+static const struct drm_display_mode omap_dss_pal_mode = {
+ .hdisplay = 720,
+ .hsync_start = 732,
+ .hsync_end = 796,
+ .htotal = 864,
+ .vdisplay = 574,
+ .vsync_start = 579,
+ .vsync_end = 584,
+ .vtotal = 625,
+ .clock = 13500,
+
+ .flags = DRM_MODE_FLAG_INTERLACE | DRM_MODE_FLAG_NHSYNC |
+ DRM_MODE_FLAG_NVSYNC,
};
-static const struct videomode omap_dss_ntsc_vm = {
- .hactive = 720,
- .vactive = 482,
- .pixelclock = 13500000,
- .hsync_len = 64,
- .hfront_porch = 16,
- .hback_porch = 58,
- .vsync_len = 6,
- .vfront_porch = 6,
- .vback_porch = 31,
-
- .flags = DISPLAY_FLAGS_INTERLACED | DISPLAY_FLAGS_HSYNC_LOW |
- DISPLAY_FLAGS_VSYNC_LOW | DISPLAY_FLAGS_DE_HIGH |
- DISPLAY_FLAGS_PIXDATA_POSEDGE |
- DISPLAY_FLAGS_SYNC_NEGEDGE,
+static const struct drm_display_mode omap_dss_ntsc_mode = {
+ .hdisplay = 720,
+ .hsync_start = 736,
+ .hsync_end = 800,
+ .htotal = 858,
+ .vdisplay = 482,
+ .vsync_start = 488,
+ .vsync_end = 494,
+ .vtotal = 525,
+ .clock = 13500,
+
+ .flags = DRM_MODE_FLAG_INTERLACE | DRM_MODE_FLAG_NHSYNC |
+ DRM_MODE_FLAG_NVSYNC,
};
-static enum venc_videomode venc_get_videomode(const struct videomode *vm)
-{
- if (!(vm->flags & DISPLAY_FLAGS_INTERLACED))
- return VENC_MODE_UNKNOWN;
-
- if (vm->pixelclock == omap_dss_pal_vm.pixelclock &&
- vm->hactive == omap_dss_pal_vm.hactive &&
- vm->vactive == omap_dss_pal_vm.vactive)
- return VENC_MODE_PAL;
-
- if (vm->pixelclock == omap_dss_ntsc_vm.pixelclock &&
- vm->hactive == omap_dss_ntsc_vm.hactive &&
- vm->vactive == omap_dss_ntsc_vm.vactive)
- return VENC_MODE_NTSC;
-
- return VENC_MODE_UNKNOWN;
-}
-
struct venc_device {
struct platform_device *pdev;
void __iomem *base;
struct mutex venc_lock;
- u32 wss_data;
struct regulator *vdda_dac_reg;
struct dss_device *dss;
struct clk *tv_dac_clk;
- struct videomode vm;
+ const struct venc_config *config;
enum omap_dss_venc_type type;
bool invert_polarity;
bool requires_tv_dac_clk;
venc_write_reg(venc, VENC_BLACK_LEVEL, config->black_level);
venc_write_reg(venc, VENC_BLANK_LEVEL, config->blank_level);
venc_write_reg(venc, VENC_M_CONTROL, config->m_control);
- venc_write_reg(venc, VENC_BSTAMP_WSS_DATA, config->bstamp_wss_data |
- venc->wss_data);
+ venc_write_reg(venc, VENC_BSTAMP_WSS_DATA, config->bstamp_wss_data);
venc_write_reg(venc, VENC_S_CARR, config->s_carr);
venc_write_reg(venc, VENC_L21__WC_CTL, config->l21__wc_ctl);
venc_write_reg(venc, VENC_SAVID__EAVID, config->savid__eavid);
WARN_ON(r < 0 && r != -ENOSYS);
}
-static const struct venc_config *venc_timings_to_config(const struct videomode *vm)
-{
- switch (venc_get_videomode(vm)) {
- default:
- WARN_ON_ONCE(1);
- case VENC_MODE_PAL:
- return &venc_config_pal_trm;
- case VENC_MODE_NTSC:
- return &venc_config_ntsc_trm;
- }
-}
-
static int venc_power_on(struct venc_device *venc)
{
u32 l;
goto err0;
venc_reset(venc);
- venc_write_config(venc, venc_timings_to_config(&venc->vm));
+ venc_write_config(venc, venc->config);
dss_set_venc_output(venc->dss, venc->type);
dss_set_dac_pwrdn_bgz(venc->dss, 1);
venc_runtime_put(venc);
}
-static int venc_display_enable(struct omap_dss_device *dssdev)
+static void venc_display_enable(struct omap_dss_device *dssdev)
{
struct venc_device *venc = dssdev_to_venc(dssdev);
- int r;
DSSDBG("venc_display_enable\n");
mutex_lock(&venc->venc_lock);
- if (!dssdev->dispc_channel_connected) {
- DSSERR("Failed to enable display: no output/manager\n");
- r = -ENODEV;
- goto err0;
- }
-
- r = venc_power_on(venc);
- if (r)
- goto err0;
-
- venc->wss_data = 0;
+ venc_power_on(venc);
mutex_unlock(&venc->venc_lock);
-
- return 0;
-err0:
- mutex_unlock(&venc->venc_lock);
- return r;
}
static void venc_display_disable(struct omap_dss_device *dssdev)
mutex_unlock(&venc->venc_lock);
}
-static void venc_get_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+static int venc_get_modes(struct omap_dss_device *dssdev,
+ struct drm_connector *connector)
{
- struct venc_device *venc = dssdev_to_venc(dssdev);
+ static const struct drm_display_mode *modes[] = {
+ &omap_dss_pal_mode,
+ &omap_dss_ntsc_mode,
+ };
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(modes); ++i) {
+ struct drm_display_mode *mode;
+
+ mode = drm_mode_duplicate(connector->dev, modes[i]);
+ if (!mode)
+ return i;
+
+ mode->type = DRM_MODE_TYPE_DRIVER | DRM_MODE_TYPE_PREFERRED;
+ drm_mode_set_name(mode);
+ drm_mode_probed_add(connector, mode);
+ }
- mutex_lock(&venc->venc_lock);
- *vm = venc->vm;
- mutex_unlock(&venc->venc_lock);
+ return ARRAY_SIZE(modes);
+}
+
+static enum venc_videomode venc_get_videomode(const struct drm_display_mode *mode)
+{
+ if (!(mode->flags & DRM_MODE_FLAG_INTERLACE))
+ return VENC_MODE_UNKNOWN;
+
+ if (mode->clock == omap_dss_pal_mode.clock &&
+ mode->hdisplay == omap_dss_pal_mode.hdisplay &&
+ mode->vdisplay == omap_dss_pal_mode.vdisplay)
+ return VENC_MODE_PAL;
+
+ if (mode->clock == omap_dss_ntsc_mode.clock &&
+ mode->hdisplay == omap_dss_ntsc_mode.hdisplay &&
+ mode->vdisplay == omap_dss_ntsc_mode.vdisplay)
+ return VENC_MODE_NTSC;
+
+ return VENC_MODE_UNKNOWN;
}
static void venc_set_timings(struct omap_dss_device *dssdev,
- const struct videomode *vm)
+ const struct drm_display_mode *mode)
{
struct venc_device *venc = dssdev_to_venc(dssdev);
+ enum venc_videomode venc_mode = venc_get_videomode(mode);
DSSDBG("venc_set_timings\n");
mutex_lock(&venc->venc_lock);
- /* Reset WSS data when the TV standard changes. */
- if (memcmp(&venc->vm, vm, sizeof(*vm)))
- venc->wss_data = 0;
+ switch (venc_mode) {
+ default:
+ WARN_ON_ONCE(1);
+ /* Fall-through */
+ case VENC_MODE_PAL:
+ venc->config = &venc_config_pal_trm;
+ break;
- venc->vm = *vm;
+ case VENC_MODE_NTSC:
+ venc->config = &venc_config_ntsc_trm;
+ break;
+ }
dispc_set_tv_pclk(venc->dss->dispc, 13500000);
}
static int venc_check_timings(struct omap_dss_device *dssdev,
- struct videomode *vm)
+ struct drm_display_mode *mode)
{
DSSDBG("venc_check_timings\n");
- switch (venc_get_videomode(vm)) {
+ switch (venc_get_videomode(mode)) {
case VENC_MODE_PAL:
- *vm = omap_dss_pal_vm;
- return 0;
+ drm_mode_copy(mode, &omap_dss_pal_mode);
+ break;
case VENC_MODE_NTSC:
- *vm = omap_dss_ntsc_vm;
- return 0;
+ drm_mode_copy(mode, &omap_dss_ntsc_mode);
+ break;
default:
return -EINVAL;
}
+
+ drm_mode_set_crtcinfo(mode, CRTC_INTERLACE_HALVE_V);
+ drm_mode_set_name(mode);
+ return 0;
}
static int venc_dump_regs(struct seq_file *s, void *p)
static int venc_connect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- int r;
-
- r = omapdss_device_connect(dst->dss, dst, dst->next);
- if (r)
- return r;
-
- dst->dispc_channel_connected = true;
- return 0;
+ return omapdss_device_connect(dst->dss, dst, dst->next);
}
static void venc_disconnect(struct omap_dss_device *src,
struct omap_dss_device *dst)
{
- dst->dispc_channel_connected = false;
-
omapdss_device_disconnect(dst, dst->next);
}
.disable = venc_display_disable,
.check_timings = venc_check_timings,
- .get_timings = venc_get_timings,
.set_timings = venc_set_timings,
+
+ .get_modes = venc_get_modes,
};
/* -----------------------------------------------------------------------------
out->dev = &venc->pdev->dev;
out->id = OMAP_DSS_OUTPUT_VENC;
- out->output_type = OMAP_DISPLAY_TYPE_VENC;
+ out->type = OMAP_DISPLAY_TYPE_VENC;
out->name = "venc.0";
out->dispc_channel = OMAP_DSS_CHANNEL_DIGIT;
out->ops = &venc_ops;
out->owner = THIS_MODULE;
out->of_ports = BIT(0);
+ out->ops_flags = OMAP_DSS_DEVICE_OP_MODES;
- out->next = omapdss_of_find_connected_device(out->dev->of_node, 0);
- if (IS_ERR(out->next)) {
- if (PTR_ERR(out->next) != -EPROBE_DEFER)
- dev_err(out->dev, "failed to find video sink\n");
- return PTR_ERR(out->next);
- }
-
- r = omapdss_output_validate(out);
- if (r) {
- omapdss_device_put(out->next);
- out->next = NULL;
+ r = omapdss_device_init_output(out);
+ if (r < 0)
return r;
- }
omapdss_device_register(out);
static void venc_uninit_output(struct venc_device *venc)
{
- if (venc->output.next)
- omapdss_device_put(venc->output.next);
omapdss_device_unregister(&venc->output);
+ omapdss_device_cleanup_output(&venc->output);
}
static int venc_probe_of(struct venc_device *venc)
mutex_init(&venc->venc_lock);
- venc->wss_data = 0;
- venc->vm = omap_dss_pal_vm;
+ venc->config = &venc_config_pal_trm;
venc_mem = platform_get_resource(venc->pdev, IORESOURCE_MEM, 0);
venc->base = devm_ioremap_resource(&pdev->dev, venc_mem);
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
+#include <drm/drm_panel.h>
#include <drm/drm_probe_helper.h>
#include "omap_drv.h"
struct omap_connector {
struct drm_connector base;
struct omap_dss_device *output;
- struct omap_dss_device *display;
struct omap_dss_device *hpd;
bool hdmi_mode;
};
static void omap_connector_hpd_notify(struct drm_connector *connector,
- struct omap_dss_device *src,
enum drm_connector_status status)
{
- if (status == connector_status_disconnected) {
- /*
- * If the source is an HDMI encoder, notify it of disconnection.
- * This is required to let the HDMI encoder reset any internal
- * state related to connection status, such as the CEC address.
- */
- if (src && src->type == OMAP_DISPLAY_TYPE_HDMI &&
- src->ops->hdmi.lost_hotplug)
- src->ops->hdmi.lost_hotplug(src);
+ struct omap_connector *omap_connector = to_omap_connector(connector);
+ struct omap_dss_device *dssdev;
+
+ if (status != connector_status_disconnected)
+ return;
+
+ /*
+ * Notify all devics in the pipeline of disconnection. This is required
+ * to let the HDMI encoders reset their internal state related to
+ * connection status, such as the CEC address.
+ */
+ for (dssdev = omap_connector->output; dssdev; dssdev = dssdev->next) {
+ if (dssdev->ops && dssdev->ops->hdmi.lost_hotplug)
+ dssdev->ops->hdmi.lost_hotplug(dssdev);
}
}
if (old_status == status)
return;
- omap_connector_hpd_notify(connector, omap_connector->hpd, status);
+ omap_connector_hpd_notify(connector, status);
drm_kms_helper_hotplug_event(dev);
}
enum omap_dss_device_ops_flag op)
{
struct omap_connector *omap_connector = to_omap_connector(connector);
- struct omap_dss_device *dssdev;
+ struct omap_dss_device *dssdev = NULL;
+ struct omap_dss_device *d;
- for (dssdev = omap_connector->display; dssdev; dssdev = dssdev->src) {
- if (dssdev->ops_flags & op)
- return dssdev;
+ for (d = omap_connector->output; d; d = d->next) {
+ if (d->ops_flags & op)
+ dssdev = d;
}
- return NULL;
+ return dssdev;
}
static enum drm_connector_status omap_connector_detect(
struct drm_connector *connector, bool force)
{
- struct omap_connector *omap_connector = to_omap_connector(connector);
struct omap_dss_device *dssdev;
enum drm_connector_status status;
? connector_status_connected
: connector_status_disconnected;
- omap_connector_hpd_notify(connector, dssdev->src, status);
+ omap_connector_hpd_notify(connector, status);
} else {
- switch (omap_connector->display->type) {
- case OMAP_DISPLAY_TYPE_DPI:
- case OMAP_DISPLAY_TYPE_DBI:
- case OMAP_DISPLAY_TYPE_SDI:
- case OMAP_DISPLAY_TYPE_DSI:
+ switch (connector->connector_type) {
+ case DRM_MODE_CONNECTOR_DPI:
+ case DRM_MODE_CONNECTOR_LVDS:
+ case DRM_MODE_CONNECTOR_DSI:
status = connector_status_connected;
break;
default:
}
}
- VERB("%s: %d (force=%d)", omap_connector->display->name, status, force);
+ VERB("%s: %d (force=%d)", connector->name, status, force);
return status;
}
{
struct omap_connector *omap_connector = to_omap_connector(connector);
- DBG("%s", omap_connector->display->name);
+ DBG("%s", connector->name);
if (omap_connector->hpd) {
struct omap_dss_device *hpd = omap_connector->hpd;
drm_connector_cleanup(connector);
omapdss_device_put(omap_connector->output);
- omapdss_device_put(omap_connector->display);
kfree(omap_connector);
}
{
struct omap_connector *omap_connector = to_omap_connector(connector);
struct omap_dss_device *dssdev;
- struct drm_display_mode *mode;
- struct videomode vm = {0};
- DBG("%s", omap_connector->display->name);
+ DBG("%s", connector->name);
/*
* If display exposes EDID, then we parse that in the normal way to
return omap_connector_get_modes_edid(connector, dssdev);
/*
- * Otherwise we have either a fixed resolution panel or an output that
- * doesn't support modes discovery (e.g. DVI or VGA with the DDC bus
- * unconnected, or analog TV). Start by querying the size.
+ * Otherwise if the display pipeline reports modes (e.g. with a fixed
+ * resolution panel or an analog TV output), query it.
*/
- dssdev = omap_connector->display;
- if (dssdev->driver && dssdev->driver->get_size)
- dssdev->driver->get_size(dssdev,
- &connector->display_info.width_mm,
- &connector->display_info.height_mm);
+ dssdev = omap_connector_find_device(connector,
+ OMAP_DSS_DEVICE_OP_MODES);
+ if (dssdev)
+ return dssdev->ops->get_modes(dssdev, connector);
/*
- * Iterate over the pipeline to find the first device that can provide
- * timing information. If we can't find any, we just let the KMS core
- * add the default modes.
+ * Otherwise if the display pipeline uses a drm_panel, we delegate the
+ * operation to the panel API.
*/
- for (dssdev = omap_connector->display; dssdev; dssdev = dssdev->src) {
- if (dssdev->ops->get_timings)
- break;
- }
- if (!dssdev)
- return 0;
+ if (omap_connector->output->panel)
+ return drm_panel_get_modes(omap_connector->output->panel);
- /* Add a single mode corresponding to the fixed panel timings. */
- mode = drm_mode_create(connector->dev);
- if (!mode)
- return 0;
+ /*
+ * We can't retrieve modes, which can happen for instance for a DVI or
+ * VGA output with the DDC bus unconnected. The KMS core will add the
+ * default modes.
+ */
+ return 0;
+}
- dssdev->ops->get_timings(dssdev, &vm);
+enum drm_mode_status omap_connector_mode_fixup(struct omap_dss_device *dssdev,
+ const struct drm_display_mode *mode,
+ struct drm_display_mode *adjusted_mode)
+{
+ int ret;
- drm_display_mode_from_videomode(&vm, mode);
+ drm_mode_copy(adjusted_mode, mode);
- mode->type = DRM_MODE_TYPE_DRIVER | DRM_MODE_TYPE_PREFERRED;
- drm_mode_set_name(mode);
- drm_mode_probed_add(connector, mode);
+ for (; dssdev; dssdev = dssdev->next) {
+ if (!dssdev->ops->check_timings)
+ continue;
+
+ ret = dssdev->ops->check_timings(dssdev, adjusted_mode);
+ if (ret)
+ return MODE_BAD;
+ }
- return 1;
+ return MODE_OK;
}
-static int omap_connector_mode_valid(struct drm_connector *connector,
+static enum drm_mode_status omap_connector_mode_valid(struct drm_connector *connector,
struct drm_display_mode *mode)
{
struct omap_connector *omap_connector = to_omap_connector(connector);
- enum omap_channel channel = omap_connector->output->dispc_channel;
- struct omap_drm_private *priv = connector->dev->dev_private;
- struct omap_dss_device *dssdev;
- struct videomode vm = {0};
- struct drm_device *dev = connector->dev;
- struct drm_display_mode *new_mode;
- int r, ret = MODE_BAD;
-
- drm_display_mode_to_videomode(mode, &vm);
- mode->vrefresh = drm_mode_vrefresh(mode);
+ struct drm_display_mode new_mode = { { 0 } };
+ enum drm_mode_status status;
- r = priv->dispc_ops->mgr_check_timings(priv->dispc, channel, &vm);
- if (r)
+ status = omap_connector_mode_fixup(omap_connector->output, mode,
+ &new_mode);
+ if (status != MODE_OK)
goto done;
- for (dssdev = omap_connector->output; dssdev; dssdev = dssdev->next) {
- if (!dssdev->ops->check_timings)
- continue;
-
- r = dssdev->ops->check_timings(dssdev, &vm);
- if (r)
- goto done;
- }
-
- /* check if vrefresh is still valid */
- new_mode = drm_mode_duplicate(dev, mode);
- if (!new_mode)
- return MODE_BAD;
-
- new_mode->clock = vm.pixelclock / 1000;
- new_mode->vrefresh = 0;
- if (mode->vrefresh == drm_mode_vrefresh(new_mode))
- ret = MODE_OK;
- drm_mode_destroy(dev, new_mode);
+ /* Check if vrefresh is still valid. */
+ if (drm_mode_vrefresh(mode) != drm_mode_vrefresh(&new_mode))
+ status = MODE_NOCLOCK;
done:
DBG("connector: mode %s: " DRM_MODE_FMT,
- (ret == MODE_OK) ? "valid" : "invalid",
+ (status == MODE_OK) ? "valid" : "invalid",
DRM_MODE_ARG(mode));
- return ret;
+ return status;
}
static const struct drm_connector_funcs omap_connector_funcs = {
.mode_valid = omap_connector_mode_valid,
};
-static int omap_connector_get_type(struct omap_dss_device *display)
+static int omap_connector_get_type(struct omap_dss_device *output)
{
- switch (display->type) {
+ struct omap_dss_device *display;
+ enum omap_display_type type;
+
+ display = omapdss_display_get(output);
+ type = display->type;
+ omapdss_device_put(display);
+
+ switch (type) {
case OMAP_DISPLAY_TYPE_HDMI:
return DRM_MODE_CONNECTOR_HDMIA;
case OMAP_DISPLAY_TYPE_DVI:
/* initialize connector */
struct drm_connector *omap_connector_init(struct drm_device *dev,
struct omap_dss_device *output,
- struct omap_dss_device *display,
struct drm_encoder *encoder)
{
struct drm_connector *connector = NULL;
struct omap_connector *omap_connector;
struct omap_dss_device *dssdev;
- DBG("%s", display->name);
+ DBG("%s", output->name);
omap_connector = kzalloc(sizeof(*omap_connector), GFP_KERNEL);
if (!omap_connector)
goto fail;
omap_connector->output = omapdss_device_get(output);
- omap_connector->display = omapdss_device_get(display);
connector = &omap_connector->base;
connector->interlace_allowed = 1;
connector->doublescan_allowed = 0;
drm_connector_init(dev, connector, &omap_connector_funcs,
- omap_connector_get_type(display));
+ omap_connector_get_type(output));
drm_connector_helper_add(connector, &omap_connector_helper_funcs);
/*
#include <linux/types.h>
+enum drm_mode_status;
+
struct drm_connector;
struct drm_device;
struct drm_encoder;
struct drm_connector *omap_connector_init(struct drm_device *dev,
struct omap_dss_device *output,
- struct omap_dss_device *display,
struct drm_encoder *encoder);
-struct drm_encoder *omap_connector_attached_encoder(
- struct drm_connector *connector);
bool omap_connector_get_hdmi_mode(struct drm_connector *connector);
void omap_connector_enable_hpd(struct drm_connector *connector);
void omap_connector_disable_hpd(struct drm_connector *connector);
+enum drm_mode_status omap_connector_mode_fixup(struct omap_dss_device *dssdev,
+ const struct drm_display_mode *mode,
+ struct drm_display_mode *adjusted_mode);
#endif /* __OMAPDRM_CONNECTOR_H__ */
if (WARN_ON(omap_crtc->enabled == enable))
return;
- if (omap_crtc->pipe->output->output_type == OMAP_DISPLAY_TYPE_HDMI) {
+ if (omap_crtc->pipe->output->type == OMAP_DISPLAY_TYPE_HDMI) {
priv->dispc_ops->mgr_enable(priv->dispc, channel, enable);
omap_crtc->enabled = enable;
return;
const struct drm_display_mode *mode)
{
struct omap_drm_private *priv = crtc->dev->dev_private;
+ struct omap_crtc *omap_crtc = to_omap_crtc(crtc);
+ struct videomode vm = {0};
+ int r;
+
+ drm_display_mode_to_videomode(mode, &vm);
+ r = priv->dispc_ops->mgr_check_timings(priv->dispc, omap_crtc->channel,
+ &vm);
+ if (r)
+ return r;
/* Check for bandwidth limit */
if (priv->max_bandwidth) {
&omap_crtc_funcs, NULL);
if (ret < 0) {
dev_err(dev->dev, "%s(): could not init crtc for: %s\n",
- __func__, pipe->display->name);
+ __func__, pipe->output->name);
kfree(omap_crtc);
return ERR_PTR(ret);
}
#include <drm/drm_atomic_helper.h>
#include <drm/drm_fb_helper.h>
#include <drm/drm_probe_helper.h>
+#include <drm/drm_panel.h>
#include "omap_dmm_tiler.h"
#include "omap_drv.h"
for (i = 0; i < priv->num_pipes; i++) {
struct omap_drm_pipeline *pipe = &priv->pipes[i];
+ if (pipe->output->panel)
+ drm_panel_detach(pipe->output->panel);
+
omapdss_device_disconnect(NULL, pipe->output);
omapdss_device_put(pipe->output);
- omapdss_device_put(pipe->display);
pipe->output = NULL;
- pipe->display = NULL;
}
memset(&priv->channels, 0, sizeof(priv->channels));
priv->num_pipes = 0;
}
-static int omap_compare_pipes(const void *a, const void *b)
-{
- const struct omap_drm_pipeline *pipe1 = a;
- const struct omap_drm_pipeline *pipe2 = b;
-
- if (pipe1->display->alias_id > pipe2->display->alias_id)
- return 1;
- else if (pipe1->display->alias_id < pipe2->display->alias_id)
- return -1;
- return 0;
-}
-
static int omap_connect_pipelines(struct drm_device *ddev)
{
struct omap_drm_private *priv = ddev->dev_private;
struct omap_dss_device *output = NULL;
- unsigned int i;
int r;
- if (!omapdss_stack_is_ready())
- return -EPROBE_DEFER;
-
for_each_dss_output(output) {
r = omapdss_device_connect(priv->dss, NULL, output);
if (r == -EPROBE_DEFER) {
omapdss_device_put(output);
- goto cleanup;
+ return r;
} else if (r) {
dev_warn(output->dev, "could not connect output %s\n",
output->name);
pipe = &priv->pipes[priv->num_pipes++];
pipe->output = omapdss_device_get(output);
- pipe->display = omapdss_display_get(output);
if (priv->num_pipes == ARRAY_SIZE(priv->pipes)) {
/* To balance the 'for_each_dss_output' loop */
}
}
- /* Sort the list by DT aliases */
- sort(priv->pipes, priv->num_pipes, sizeof(priv->pipes[0]),
- omap_compare_pipes, NULL);
-
- /*
- * Populate the pipeline lookup table by DISPC channel. Only one display
- * is allowed per channel.
- */
- for (i = 0; i < priv->num_pipes; ++i) {
- struct omap_drm_pipeline *pipe = &priv->pipes[i];
- enum omap_channel channel = pipe->output->dispc_channel;
-
- if (WARN_ON(priv->channels[channel] != NULL)) {
- r = -EINVAL;
- goto cleanup;
- }
-
- priv->channels[channel] = pipe;
- }
-
return 0;
+}
-cleanup:
- /*
- * if we are deferring probe, we disconnect the devices we previously
- * connected
- */
- omap_disconnect_pipelines(ddev);
+static int omap_compare_pipelines(const void *a, const void *b)
+{
+ const struct omap_drm_pipeline *pipe1 = a;
+ const struct omap_drm_pipeline *pipe2 = b;
- return r;
+ if (pipe1->alias_id > pipe2->alias_id)
+ return 1;
+ else if (pipe1->alias_id < pipe2->alias_id)
+ return -1;
+ return 0;
}
static int omap_modeset_init_properties(struct drm_device *dev)
return 0;
}
+static int omap_display_id(struct omap_dss_device *output)
+{
+ struct device_node *node = NULL;
+
+ if (output->next) {
+ struct omap_dss_device *display;
+
+ display = omapdss_display_get(output);
+ node = display->dev->of_node;
+ omapdss_device_put(display);
+ } else if (output->bridge) {
+ struct drm_bridge *bridge = output->bridge;
+
+ while (bridge->next)
+ bridge = bridge->next;
+
+ node = bridge->of_node;
+ } else if (output->panel) {
+ node = output->panel->dev->of_node;
+ }
+
+ return node ? of_alias_get_id(node, "display") : -ENODEV;
+}
+
static int omap_modeset_init(struct drm_device *dev)
{
struct omap_drm_private *priv = dev->dev_private;
int ret;
u32 plane_crtc_mask;
+ if (!omapdss_stack_is_ready())
+ return -EPROBE_DEFER;
+
drm_mode_config_init(dev);
ret = omap_modeset_init_properties(dev);
* configuration does not match the expectations or exceeds
* the available resources, the configuration is rejected.
*/
+ ret = omap_connect_pipelines(dev);
+ if (ret < 0)
+ return ret;
+
if (priv->num_pipes > num_mgrs || priv->num_pipes > num_ovls) {
dev_err(dev->dev, "%s(): Too many connected displays\n",
__func__);
priv->planes[priv->num_planes++] = plane;
}
- /* Create the CRTCs, encoders and connectors. */
+ /*
+ * Create the encoders, attach the bridges and get the pipeline alias
+ * IDs.
+ */
for (i = 0; i < priv->num_pipes; i++) {
struct omap_drm_pipeline *pipe = &priv->pipes[i];
- struct omap_dss_device *display = pipe->display;
- struct drm_connector *connector;
- struct drm_encoder *encoder;
- struct drm_crtc *crtc;
+ int id;
- encoder = omap_encoder_init(dev, pipe->output, display);
- if (!encoder)
+ pipe->encoder = omap_encoder_init(dev, pipe->output);
+ if (!pipe->encoder)
return -ENOMEM;
- connector = omap_connector_init(dev, pipe->output, display,
- encoder);
- if (!connector)
- return -ENOMEM;
+ if (pipe->output->bridge) {
+ ret = drm_bridge_attach(pipe->encoder,
+ pipe->output->bridge, NULL);
+ if (ret < 0)
+ return ret;
+ }
+
+ id = omap_display_id(pipe->output);
+ pipe->alias_id = id >= 0 ? id : i;
+ }
+
+ /* Sort the pipelines by DT aliases. */
+ sort(priv->pipes, priv->num_pipes, sizeof(priv->pipes[0]),
+ omap_compare_pipelines, NULL);
+
+ /*
+ * Populate the pipeline lookup table by DISPC channel. Only one display
+ * is allowed per channel.
+ */
+ for (i = 0; i < priv->num_pipes; ++i) {
+ struct omap_drm_pipeline *pipe = &priv->pipes[i];
+ enum omap_channel channel = pipe->output->dispc_channel;
+
+ if (WARN_ON(priv->channels[channel] != NULL))
+ return -EINVAL;
+
+ priv->channels[channel] = pipe;
+ }
+
+ /* Create the connectors and CRTCs. */
+ for (i = 0; i < priv->num_pipes; i++) {
+ struct omap_drm_pipeline *pipe = &priv->pipes[i];
+ struct drm_encoder *encoder = pipe->encoder;
+ struct drm_crtc *crtc;
+
+ if (!pipe->output->bridge) {
+ pipe->connector = omap_connector_init(dev, pipe->output,
+ encoder);
+ if (!pipe->connector)
+ return -ENOMEM;
+
+ drm_connector_attach_encoder(pipe->connector, encoder);
+
+ if (pipe->output->panel) {
+ ret = drm_panel_attach(pipe->output->panel,
+ pipe->connector);
+ if (ret < 0)
+ return ret;
+ }
+ }
crtc = omap_crtc_init(dev, pipe, priv->planes[i]);
if (IS_ERR(crtc))
return PTR_ERR(crtc);
- drm_connector_attach_encoder(connector, encoder);
encoder->possible_crtcs = 1 << i;
-
pipe->crtc = crtc;
- pipe->encoder = encoder;
- pipe->connector = connector;
}
DBG("registered %u planes, %u crtcs/encoders/connectors\n",
static void omap_modeset_enable_external_hpd(struct drm_device *ddev)
{
struct omap_drm_private *priv = ddev->dev_private;
- int i;
+ unsigned int i;
- for (i = 0; i < priv->num_pipes; i++)
- omap_connector_enable_hpd(priv->pipes[i].connector);
+ for (i = 0; i < priv->num_pipes; i++) {
+ if (priv->pipes[i].connector)
+ omap_connector_enable_hpd(priv->pipes[i].connector);
+ }
}
/*
static void omap_modeset_disable_external_hpd(struct drm_device *ddev)
{
struct omap_drm_private *priv = ddev->dev_private;
- int i;
+ unsigned int i;
- for (i = 0; i < priv->num_pipes; i++)
- omap_connector_disable_hpd(priv->pipes[i].connector);
+ for (i = 0; i < priv->num_pipes; i++) {
+ if (priv->pipes[i].connector)
+ omap_connector_disable_hpd(priv->pipes[i].connector);
+ }
}
/*
omap_crtc_pre_init(priv);
- ret = omap_connect_pipelines(ddev);
- if (ret)
- goto err_crtc_uninit;
-
soc = soc_device_match(omapdrm_soc_devices);
priv->omaprev = soc ? (unsigned int)soc->data : 0;
priv->wq = alloc_ordered_workqueue("omapdrm", 0);
omap_gem_deinit(ddev);
destroy_workqueue(priv->wq);
omap_disconnect_pipelines(ddev);
-err_crtc_uninit:
omap_crtc_pre_uninit(priv);
drm_dev_put(ddev);
return ret;
}
#ifdef CONFIG_PM_SLEEP
-static int omap_drm_suspend_all_displays(struct drm_device *ddev)
-{
- struct omap_drm_private *priv = ddev->dev_private;
- int i;
-
- for (i = 0; i < priv->num_pipes; i++) {
- struct omap_dss_device *display = priv->pipes[i].display;
-
- if (display->state == OMAP_DSS_DISPLAY_ACTIVE) {
- display->ops->disable(display);
- display->activate_after_resume = true;
- } else {
- display->activate_after_resume = false;
- }
- }
-
- return 0;
-}
-
-static int omap_drm_resume_all_displays(struct drm_device *ddev)
-{
- struct omap_drm_private *priv = ddev->dev_private;
- int i;
-
- for (i = 0; i < priv->num_pipes; i++) {
- struct omap_dss_device *display = priv->pipes[i].display;
-
- if (display->activate_after_resume) {
- display->ops->enable(display);
- display->activate_after_resume = false;
- }
- }
-
- return 0;
-}
-
static int omap_drm_suspend(struct device *dev)
{
struct omap_drm_private *priv = dev_get_drvdata(dev);
struct drm_device *drm_dev = priv->ddev;
- drm_kms_helper_poll_disable(drm_dev);
-
- drm_modeset_lock_all(drm_dev);
- omap_drm_suspend_all_displays(drm_dev);
- drm_modeset_unlock_all(drm_dev);
-
- return 0;
+ return drm_mode_config_helper_suspend(drm_dev);
}
static int omap_drm_resume(struct device *dev)
struct omap_drm_private *priv = dev_get_drvdata(dev);
struct drm_device *drm_dev = priv->ddev;
- drm_modeset_lock_all(drm_dev);
- omap_drm_resume_all_displays(drm_dev);
- drm_modeset_unlock_all(drm_dev);
-
- drm_kms_helper_poll_enable(drm_dev);
+ drm_mode_config_helper_resume(drm_dev);
return omap_gem_resume(drm_dev);
}
struct drm_encoder *encoder;
struct drm_connector *connector;
struct omap_dss_device *output;
- struct omap_dss_device *display;
+ unsigned int alias_id;
};
struct omap_drm_private {
#include <drm/drm_crtc.h>
#include <drm/drm_modeset_helper_vtables.h>
#include <drm/drm_edid.h>
+#include <drm/drm_panel.h>
#include "omap_drv.h"
struct omap_encoder {
struct drm_encoder base;
struct omap_dss_device *output;
- struct omap_dss_device *display;
};
static void omap_encoder_destroy(struct drm_encoder *encoder)
.destroy = omap_encoder_destroy,
};
-static void omap_encoder_hdmi_mode_set(struct drm_encoder *encoder,
+static void omap_encoder_update_videomode_flags(struct videomode *vm,
+ u32 bus_flags)
+{
+ if (!(vm->flags & (DISPLAY_FLAGS_DE_LOW |
+ DISPLAY_FLAGS_DE_HIGH))) {
+ if (bus_flags & DRM_BUS_FLAG_DE_LOW)
+ vm->flags |= DISPLAY_FLAGS_DE_LOW;
+ else if (bus_flags & DRM_BUS_FLAG_DE_HIGH)
+ vm->flags |= DISPLAY_FLAGS_DE_HIGH;
+ }
+
+ if (!(vm->flags & (DISPLAY_FLAGS_PIXDATA_POSEDGE |
+ DISPLAY_FLAGS_PIXDATA_NEGEDGE))) {
+ if (bus_flags & DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE)
+ vm->flags |= DISPLAY_FLAGS_PIXDATA_POSEDGE;
+ else if (bus_flags & DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE)
+ vm->flags |= DISPLAY_FLAGS_PIXDATA_NEGEDGE;
+ }
+
+ if (!(vm->flags & (DISPLAY_FLAGS_SYNC_POSEDGE |
+ DISPLAY_FLAGS_SYNC_NEGEDGE))) {
+ if (bus_flags & DRM_BUS_FLAG_SYNC_DRIVE_POSEDGE)
+ vm->flags |= DISPLAY_FLAGS_SYNC_POSEDGE;
+ else if (bus_flags & DRM_BUS_FLAG_SYNC_DRIVE_NEGEDGE)
+ vm->flags |= DISPLAY_FLAGS_SYNC_NEGEDGE;
+ }
+}
+
+static void omap_encoder_hdmi_mode_set(struct drm_connector *connector,
+ struct drm_encoder *encoder,
struct drm_display_mode *adjusted_mode)
{
- struct drm_device *dev = encoder->dev;
struct omap_encoder *omap_encoder = to_omap_encoder(encoder);
struct omap_dss_device *dssdev = omap_encoder->output;
- struct drm_connector *connector;
bool hdmi_mode;
- hdmi_mode = false;
- list_for_each_entry(connector, &dev->mode_config.connector_list, head) {
- if (connector->encoder == encoder) {
- hdmi_mode = omap_connector_get_hdmi_mode(connector);
- break;
- }
- }
+ hdmi_mode = omap_connector_get_hdmi_mode(connector);
if (dssdev->ops->hdmi.set_hdmi_mode)
dssdev->ops->hdmi.set_hdmi_mode(dssdev, hdmi_mode);
struct drm_display_mode *adjusted_mode)
{
struct omap_encoder *omap_encoder = to_omap_encoder(encoder);
+ struct omap_dss_device *output = omap_encoder->output;
struct omap_dss_device *dssdev;
+ struct drm_device *dev = encoder->dev;
+ struct drm_connector *connector;
+ struct drm_bridge *bridge;
struct videomode vm = { 0 };
+ u32 bus_flags;
+
+ list_for_each_entry(connector, &dev->mode_config.connector_list, head) {
+ if (connector->encoder == encoder)
+ break;
+ }
drm_display_mode_to_videomode(adjusted_mode, &vm);
*
* A better solution is to use DRM's bus-flags through the whole driver.
*/
- for (dssdev = omap_encoder->output; dssdev; dssdev = dssdev->next) {
- unsigned long bus_flags = dssdev->bus_flags;
-
- if (!(vm.flags & (DISPLAY_FLAGS_DE_LOW |
- DISPLAY_FLAGS_DE_HIGH))) {
- if (bus_flags & DRM_BUS_FLAG_DE_LOW)
- vm.flags |= DISPLAY_FLAGS_DE_LOW;
- else if (bus_flags & DRM_BUS_FLAG_DE_HIGH)
- vm.flags |= DISPLAY_FLAGS_DE_HIGH;
- }
-
- if (!(vm.flags & (DISPLAY_FLAGS_PIXDATA_POSEDGE |
- DISPLAY_FLAGS_PIXDATA_NEGEDGE))) {
- if (bus_flags & DRM_BUS_FLAG_PIXDATA_POSEDGE)
- vm.flags |= DISPLAY_FLAGS_PIXDATA_POSEDGE;
- else if (bus_flags & DRM_BUS_FLAG_PIXDATA_NEGEDGE)
- vm.flags |= DISPLAY_FLAGS_PIXDATA_NEGEDGE;
- }
-
- if (!(vm.flags & (DISPLAY_FLAGS_SYNC_POSEDGE |
- DISPLAY_FLAGS_SYNC_NEGEDGE))) {
- if (bus_flags & DRM_BUS_FLAG_SYNC_POSEDGE)
- vm.flags |= DISPLAY_FLAGS_SYNC_POSEDGE;
- else if (bus_flags & DRM_BUS_FLAG_SYNC_NEGEDGE)
- vm.flags |= DISPLAY_FLAGS_SYNC_NEGEDGE;
- }
+ for (dssdev = output; dssdev; dssdev = dssdev->next)
+ omap_encoder_update_videomode_flags(&vm, dssdev->bus_flags);
+
+ for (bridge = output->bridge; bridge; bridge = bridge->next) {
+ if (!bridge->timings)
+ continue;
+
+ bus_flags = bridge->timings->input_bus_flags;
+ omap_encoder_update_videomode_flags(&vm, bus_flags);
}
+ bus_flags = connector->display_info.bus_flags;
+ omap_encoder_update_videomode_flags(&vm, bus_flags);
+
/* Set timings for all devices in the display pipeline. */
- dss_mgr_set_timings(omap_encoder->output, &vm);
+ dss_mgr_set_timings(output, &vm);
- for (dssdev = omap_encoder->output; dssdev; dssdev = dssdev->next) {
+ for (dssdev = output; dssdev; dssdev = dssdev->next) {
if (dssdev->ops->set_timings)
- dssdev->ops->set_timings(dssdev, &vm);
+ dssdev->ops->set_timings(dssdev, adjusted_mode);
}
/* Set the HDMI mode and HDMI infoframe if applicable. */
- if (omap_encoder->output->output_type == OMAP_DISPLAY_TYPE_HDMI)
- omap_encoder_hdmi_mode_set(encoder, adjusted_mode);
+ if (output->type == OMAP_DISPLAY_TYPE_HDMI)
+ omap_encoder_hdmi_mode_set(connector, encoder, adjusted_mode);
}
static void omap_encoder_disable(struct drm_encoder *encoder)
{
struct omap_encoder *omap_encoder = to_omap_encoder(encoder);
- struct omap_dss_device *dssdev = omap_encoder->display;
+ struct omap_dss_device *dssdev = omap_encoder->output;
+ struct drm_device *dev = encoder->dev;
+
+ dev_dbg(dev->dev, "disable(%s)\n", dssdev->name);
+
+ /* Disable the panel if present. */
+ if (dssdev->panel) {
+ drm_panel_disable(dssdev->panel);
+ drm_panel_unprepare(dssdev->panel);
+ }
+
+ /*
+ * Disable the chain of external devices, starting at the one at the
+ * internal encoder's output.
+ */
+ omapdss_device_disable(dssdev->next);
+
+ /*
+ * Disable the internal encoder. This will disable the DSS output. The
+ * DSI is treated as an exception as DSI pipelines still use the legacy
+ * flow where the pipeline output controls the encoder.
+ */
+ if (dssdev->type != OMAP_DISPLAY_TYPE_DSI) {
+ dssdev->ops->disable(dssdev);
+ dssdev->state = OMAP_DSS_DISPLAY_DISABLED;
+ }
- dssdev->ops->disable(dssdev);
+ /*
+ * Perform the post-disable operations on the chain of external devices
+ * to complete the display pipeline disable.
+ */
+ omapdss_device_post_disable(dssdev->next);
}
static void omap_encoder_enable(struct drm_encoder *encoder)
{
struct omap_encoder *omap_encoder = to_omap_encoder(encoder);
- struct omap_dss_device *dssdev = omap_encoder->display;
- int r;
-
- r = dssdev->ops->enable(dssdev);
- if (r)
- dev_err(encoder->dev->dev,
- "Failed to enable display '%s': %d\n",
- dssdev->name, r);
+ struct omap_dss_device *dssdev = omap_encoder->output;
+ struct drm_device *dev = encoder->dev;
+
+ dev_dbg(dev->dev, "enable(%s)\n", dssdev->name);
+
+ /* Prepare the chain of external devices for pipeline enable. */
+ omapdss_device_pre_enable(dssdev->next);
+
+ /*
+ * Enable the internal encoder. This will enable the DSS output. The
+ * DSI is treated as an exception as DSI pipelines still use the legacy
+ * flow where the pipeline output controls the encoder.
+ */
+ if (dssdev->type != OMAP_DISPLAY_TYPE_DSI) {
+ dssdev->ops->enable(dssdev);
+ dssdev->state = OMAP_DSS_DISPLAY_ACTIVE;
+ }
+
+ /*
+ * Enable the chain of external devices, starting at the one at the
+ * internal encoder's output.
+ */
+ omapdss_device_enable(dssdev->next);
+
+ /* Enable the panel if present. */
+ if (dssdev->panel) {
+ drm_panel_prepare(dssdev->panel);
+ drm_panel_enable(dssdev->panel);
+ }
}
static int omap_encoder_atomic_check(struct drm_encoder *encoder,
struct drm_connector_state *conn_state)
{
struct omap_encoder *omap_encoder = to_omap_encoder(encoder);
- enum omap_channel channel = omap_encoder->output->dispc_channel;
- struct drm_device *dev = encoder->dev;
- struct omap_drm_private *priv = dev->dev_private;
- struct omap_dss_device *dssdev;
- struct videomode vm = { 0 };
- int ret;
-
- drm_display_mode_to_videomode(&crtc_state->mode, &vm);
-
- ret = priv->dispc_ops->mgr_check_timings(priv->dispc, channel, &vm);
- if (ret)
- goto done;
-
- for (dssdev = omap_encoder->output; dssdev; dssdev = dssdev->next) {
- if (!dssdev->ops->check_timings)
- continue;
-
- ret = dssdev->ops->check_timings(dssdev, &vm);
- if (ret)
- goto done;
+ enum drm_mode_status status;
+
+ status = omap_connector_mode_fixup(omap_encoder->output,
+ &crtc_state->mode,
+ &crtc_state->adjusted_mode);
+ if (status != MODE_OK) {
+ dev_err(encoder->dev->dev, "invalid timings: %d\n", status);
+ return -EINVAL;
}
- drm_display_mode_from_videomode(&vm, &crtc_state->adjusted_mode);
-
-done:
- if (ret)
- dev_err(dev->dev, "invalid timings: %d\n", ret);
-
- return ret;
+ return 0;
}
static const struct drm_encoder_helper_funcs omap_encoder_helper_funcs = {
/* initialize encoder */
struct drm_encoder *omap_encoder_init(struct drm_device *dev,
- struct omap_dss_device *output,
- struct omap_dss_device *display)
+ struct omap_dss_device *output)
{
struct drm_encoder *encoder = NULL;
struct omap_encoder *omap_encoder;
goto fail;
omap_encoder->output = output;
- omap_encoder->display = display;
encoder = &omap_encoder->base;
struct omap_dss_device;
struct drm_encoder *omap_encoder_init(struct drm_device *dev,
- struct omap_dss_device *output,
- struct omap_dss_device *display);
+ struct omap_dss_device *output);
#endif /* __OMAPDRM_ENCODER_H__ */
fbdev->fb = fb;
helper->fb = fb;
- fbi->par = helper;
fbi->fbops = &omap_fb_ops;
- strcpy(fbi->fix.id, MODULE_NAME);
-
- drm_fb_helper_fill_fix(fbi, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(fbi, helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(fbi, helper, sizes);
dev->mode_config.fb_base = dma_addr;
that it can be automatically turned off when the panel goes into a
low power state.
+config DRM_PANEL_FEIYANG_FY07024DI26A30D
+ tristate "Feiyang FY07024DI26A30-D MIPI-DSI LCD panel"
+ depends on OF
+ depends on DRM_MIPI_DSI
+ depends on BACKLIGHT_CLASS_DEVICE
+ help
+ Say Y if you want to enable support for panels based on the
+ Feiyang FY07024DI26A30-D MIPI-DSI interface.
+
config DRM_PANEL_ILITEK_IL9322
tristate "Ilitek ILI9322 320x240 QVGA panels"
depends on OF && SPI
Say Y here if you want to enable support for Raydium RM68200
720x1280 DSI video mode panel.
+config DRM_PANEL_ROCKTECH_JH057N00900
+ tristate "Rocktech JH057N00900 MIPI touchscreen panel"
+ depends on OF
+ depends on DRM_MIPI_DSI
+ depends on BACKLIGHT_CLASS_DEVICE
+ help
+ Say Y here if you want to enable support for Rocktech JH057N00900
+ MIPI DSI panel as e.g. used in the Librem 5 devkit. It has a
+ resolution of 720x1440 pixels, a built in backlight and touch
+ controller.
+ Touch input support is provided by the goodix driver and needs to be
+ selected separately.
+
+config DRM_PANEL_RONBO_RB070D30
+ tristate "Ronbo Electronics RB070D30 panel"
+ depends on OF
+ depends on DRM_MIPI_DSI
+ depends on BACKLIGHT_CLASS_DEVICE
+ help
+ Say Y here if you want to enable support for Ronbo Electronics
+ RB070D30 1024x600 DSI panel.
+
config DRM_PANEL_SAMSUNG_S6D16D0
tristate "Samsung S6D16D0 DSI video mode panel"
depends on OF
obj-$(CONFIG_DRM_PANEL_ARM_VERSATILE) += panel-arm-versatile.o
obj-$(CONFIG_DRM_PANEL_LVDS) += panel-lvds.o
obj-$(CONFIG_DRM_PANEL_SIMPLE) += panel-simple.o
+obj-$(CONFIG_DRM_PANEL_FEIYANG_FY07024DI26A30D) += panel-feiyang-fy07024di26a30d.o
obj-$(CONFIG_DRM_PANEL_ILITEK_IL9322) += panel-ilitek-ili9322.o
obj-$(CONFIG_DRM_PANEL_ILITEK_ILI9881C) += panel-ilitek-ili9881c.o
obj-$(CONFIG_DRM_PANEL_INNOLUX_P079ZCA) += panel-innolux-p079zca.o
obj-$(CONFIG_DRM_PANEL_PANASONIC_VVX10F034N00) += panel-panasonic-vvx10f034n00.o
obj-$(CONFIG_DRM_PANEL_RASPBERRYPI_TOUCHSCREEN) += panel-raspberrypi-touchscreen.o
obj-$(CONFIG_DRM_PANEL_RAYDIUM_RM68200) += panel-raydium-rm68200.o
+obj-$(CONFIG_DRM_PANEL_ROCKTECH_JH057N00900) += panel-rocktech-jh057n00900.o
+obj-$(CONFIG_DRM_PANEL_RONBO_RB070D30) += panel-ronbo-rb070d30.o
obj-$(CONFIG_DRM_PANEL_SAMSUNG_LD9040) += panel-samsung-ld9040.o
obj-$(CONFIG_DRM_PANEL_SAMSUNG_S6D16D0) += panel-samsung-s6d16d0.o
obj-$(CONFIG_DRM_PANEL_SAMSUNG_S6E3HA2) += panel-samsung-s6e3ha2.o
.vrefresh = 390,
.flags = DRM_MODE_FLAG_PHSYNC | DRM_MODE_FLAG_PVSYNC,
},
- .bus_flags = DRM_BUS_FLAG_PIXDATA_NEGEDGE,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE,
},
/*
* Sanyo ALR252RGT 240x320 portrait display found on the
.vrefresh = 116,
.flags = DRM_MODE_FLAG_NHSYNC | DRM_MODE_FLAG_NVSYNC,
},
- .bus_flags = DRM_BUS_FLAG_PIXDATA_NEGEDGE,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE,
.ib2 = true,
},
};
struct versatile_panel *vpanel = to_versatile_panel(panel);
struct drm_display_mode *mode;
- strncpy(connector->display_info.name, vpanel->panel_type->name,
- DRM_DISPLAY_INFO_LEN);
connector->display_info.width_mm = vpanel->panel_type->width_mm;
connector->display_info.height_mm = vpanel->panel_type->height_mm;
connector->display_info.bus_flags = vpanel->panel_type->bus_flags;
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2018 Amarula Solutions
+ * Author: Jagan Teki <jagan@amarulasolutions.com>
+ */
+
+#include <drm/drm_mipi_dsi.h>
+#include <drm/drm_modes.h>
+#include <drm/drm_panel.h>
+#include <drm/drm_print.h>
+
+#include <linux/backlight.h>
+#include <linux/gpio/consumer.h>
+#include <linux/delay.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/regulator/consumer.h>
+
+#define FEIYANG_INIT_CMD_LEN 2
+
+struct feiyang {
+ struct drm_panel panel;
+ struct mipi_dsi_device *dsi;
+
+ struct backlight_device *backlight;
+ struct regulator *dvdd;
+ struct regulator *avdd;
+ struct gpio_desc *reset;
+};
+
+static inline struct feiyang *panel_to_feiyang(struct drm_panel *panel)
+{
+ return container_of(panel, struct feiyang, panel);
+}
+
+struct feiyang_init_cmd {
+ u8 data[FEIYANG_INIT_CMD_LEN];
+};
+
+static const struct feiyang_init_cmd feiyang_init_cmds[] = {
+ { .data = { 0x80, 0x58 } },
+ { .data = { 0x81, 0x47 } },
+ { .data = { 0x82, 0xD4 } },
+ { .data = { 0x83, 0x88 } },
+ { .data = { 0x84, 0xA9 } },
+ { .data = { 0x85, 0xC3 } },
+ { .data = { 0x86, 0x82 } },
+};
+
+static int feiyang_prepare(struct drm_panel *panel)
+{
+ struct feiyang *ctx = panel_to_feiyang(panel);
+ struct mipi_dsi_device *dsi = ctx->dsi;
+ unsigned int i;
+ int ret;
+
+ ret = regulator_enable(ctx->dvdd);
+ if (ret)
+ return ret;
+
+ /* T1 (dvdd start + dvdd rise) 0 < T1 <= 10ms */
+ msleep(10);
+
+ ret = regulator_enable(ctx->avdd);
+ if (ret)
+ return ret;
+
+ /* T3 (dvdd rise + avdd start + avdd rise) T3 >= 20ms */
+ msleep(20);
+
+ gpiod_set_value(ctx->reset, 0);
+
+ /*
+ * T5 + T6 (avdd rise + video & logic signal rise)
+ * T5 >= 10ms, 0 < T6 <= 10ms
+ */
+ msleep(20);
+
+ gpiod_set_value(ctx->reset, 1);
+
+ /* T12 (video & logic signal rise + backlight rise) T12 >= 200ms */
+ msleep(200);
+
+ for (i = 0; i < ARRAY_SIZE(feiyang_init_cmds); i++) {
+ const struct feiyang_init_cmd *cmd =
+ &feiyang_init_cmds[i];
+
+ ret = mipi_dsi_dcs_write_buffer(dsi, cmd->data,
+ FEIYANG_INIT_CMD_LEN);
+ if (ret < 0)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int feiyang_enable(struct drm_panel *panel)
+{
+ struct feiyang *ctx = panel_to_feiyang(panel);
+
+ /* T12 (video & logic signal rise + backlight rise) T12 >= 200ms */
+ msleep(200);
+
+ mipi_dsi_dcs_set_display_on(ctx->dsi);
+ backlight_enable(ctx->backlight);
+
+ return 0;
+}
+
+static int feiyang_disable(struct drm_panel *panel)
+{
+ struct feiyang *ctx = panel_to_feiyang(panel);
+
+ backlight_disable(ctx->backlight);
+ return mipi_dsi_dcs_set_display_off(ctx->dsi);
+}
+
+static int feiyang_unprepare(struct drm_panel *panel)
+{
+ struct feiyang *ctx = panel_to_feiyang(panel);
+ int ret;
+
+ ret = mipi_dsi_dcs_set_display_off(ctx->dsi);
+ if (ret < 0)
+ DRM_DEV_ERROR(panel->dev, "failed to set display off: %d\n",
+ ret);
+
+ ret = mipi_dsi_dcs_enter_sleep_mode(ctx->dsi);
+ if (ret < 0)
+ DRM_DEV_ERROR(panel->dev, "failed to enter sleep mode: %d\n",
+ ret);
+
+ /* T13 (backlight fall + video & logic signal fall) T13 >= 200ms */
+ msleep(200);
+
+ gpiod_set_value(ctx->reset, 0);
+
+ regulator_disable(ctx->avdd);
+
+ /* T11 (dvdd rise to fall) 0 < T11 <= 10ms */
+ msleep(10);
+
+ regulator_disable(ctx->dvdd);
+
+ return 0;
+}
+
+static const struct drm_display_mode feiyang_default_mode = {
+ .clock = 55000,
+
+ .hdisplay = 1024,
+ .hsync_start = 1024 + 310,
+ .hsync_end = 1024 + 310 + 20,
+ .htotal = 1024 + 310 + 20 + 90,
+
+ .vdisplay = 600,
+ .vsync_start = 600 + 12,
+ .vsync_end = 600 + 12 + 2,
+ .vtotal = 600 + 12 + 2 + 21,
+ .vrefresh = 60,
+
+ .type = DRM_MODE_TYPE_DRIVER | DRM_MODE_TYPE_PREFERRED,
+};
+
+static int feiyang_get_modes(struct drm_panel *panel)
+{
+ struct drm_connector *connector = panel->connector;
+ struct feiyang *ctx = panel_to_feiyang(panel);
+ struct drm_display_mode *mode;
+
+ mode = drm_mode_duplicate(panel->drm, &feiyang_default_mode);
+ if (!mode) {
+ DRM_DEV_ERROR(&ctx->dsi->dev, "failed to add mode %ux%ux@%u\n",
+ feiyang_default_mode.hdisplay,
+ feiyang_default_mode.vdisplay,
+ feiyang_default_mode.vrefresh);
+ return -ENOMEM;
+ }
+
+ drm_mode_set_name(mode);
+
+ drm_mode_probed_add(connector, mode);
+
+ return 1;
+}
+
+static const struct drm_panel_funcs feiyang_funcs = {
+ .disable = feiyang_disable,
+ .unprepare = feiyang_unprepare,
+ .prepare = feiyang_prepare,
+ .enable = feiyang_enable,
+ .get_modes = feiyang_get_modes,
+};
+
+static int feiyang_dsi_probe(struct mipi_dsi_device *dsi)
+{
+ struct feiyang *ctx;
+ int ret;
+
+ ctx = devm_kzalloc(&dsi->dev, sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return -ENOMEM;
+
+ mipi_dsi_set_drvdata(dsi, ctx);
+ ctx->dsi = dsi;
+
+ drm_panel_init(&ctx->panel);
+ ctx->panel.dev = &dsi->dev;
+ ctx->panel.funcs = &feiyang_funcs;
+
+ ctx->dvdd = devm_regulator_get(&dsi->dev, "dvdd");
+ if (IS_ERR(ctx->dvdd)) {
+ DRM_DEV_ERROR(&dsi->dev, "Couldn't get dvdd regulator\n");
+ return PTR_ERR(ctx->dvdd);
+ }
+
+ ctx->avdd = devm_regulator_get(&dsi->dev, "avdd");
+ if (IS_ERR(ctx->avdd)) {
+ DRM_DEV_ERROR(&dsi->dev, "Couldn't get avdd regulator\n");
+ return PTR_ERR(ctx->avdd);
+ }
+
+ ctx->reset = devm_gpiod_get(&dsi->dev, "reset", GPIOD_OUT_LOW);
+ if (IS_ERR(ctx->reset)) {
+ DRM_DEV_ERROR(&dsi->dev, "Couldn't get our reset GPIO\n");
+ return PTR_ERR(ctx->reset);
+ }
+
+ ctx->backlight = devm_of_find_backlight(&dsi->dev);
+ if (IS_ERR(ctx->backlight))
+ return PTR_ERR(ctx->backlight);
+
+ ret = drm_panel_add(&ctx->panel);
+ if (ret < 0)
+ return ret;
+
+ dsi->mode_flags = MIPI_DSI_MODE_VIDEO_BURST;
+ dsi->format = MIPI_DSI_FMT_RGB888;
+ dsi->lanes = 4;
+
+ return mipi_dsi_attach(dsi);
+}
+
+static int feiyang_dsi_remove(struct mipi_dsi_device *dsi)
+{
+ struct feiyang *ctx = mipi_dsi_get_drvdata(dsi);
+
+ mipi_dsi_detach(dsi);
+ drm_panel_remove(&ctx->panel);
+
+ return 0;
+}
+
+static const struct of_device_id feiyang_of_match[] = {
+ { .compatible = "feiyang,fy07024di26a30d", },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, feiyang_of_match);
+
+static struct mipi_dsi_driver feiyang_driver = {
+ .probe = feiyang_dsi_probe,
+ .remove = feiyang_dsi_remove,
+ .driver = {
+ .name = "feiyang-fy07024di26a30d",
+ .of_match_table = feiyang_of_match,
+ },
+};
+module_mipi_dsi_driver(feiyang_driver);
+
+MODULE_AUTHOR("Jagan Teki <jagan@amarulasolutions.com>");
+MODULE_DESCRIPTION("Feiyang FY07024DI26A30-D MIPI-DSI LCD panel");
+MODULE_LICENSE("GPL");
if (ili->conf->dclk_active_high) {
reg = ILI9322_POL_DCLK;
connector->display_info.bus_flags |=
- DRM_BUS_FLAG_PIXDATA_POSEDGE;
+ DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE;
} else {
reg = 0;
connector->display_info.bus_flags |=
- DRM_BUS_FLAG_PIXDATA_NEGEDGE;
+ DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE;
}
if (ili->conf->de_active_high) {
reg |= ILI9322_POL_DE;
struct ili9322 *ili = panel_to_ili9322(panel);
struct drm_display_mode *mode;
- strncpy(connector->display_info.name, "ILI9322 TFT LCD driver\0",
- DRM_DISPLAY_INFO_LEN);
connector->display_info.width_mm = ili->conf->width_mm;
connector->display_info.height_mm = ili->conf->height_mm;
num++;
}
- memcpy(connector->display_info.name, lcd_info->name, 32);
connector->display_info.width_mm = lcd_info->width_mm;
connector->display_info.height_mm = lcd_info->height_mm;
connector->display_info.bpc = lcd_info->bpc;
};
static const struct drm_display_mode default_mode = {
- .clock = 32729,
+ .clock = 29700,
.hdisplay = 480,
- .hsync_start = 480 + 120,
- .hsync_end = 480 + 120 + 63,
- .htotal = 480 + 120 + 63 + 120,
+ .hsync_start = 480 + 98,
+ .hsync_end = 480 + 98 + 32,
+ .htotal = 480 + 98 + 32 + 98,
.vdisplay = 800,
- .vsync_start = 800 + 12,
- .vsync_end = 800 + 12 + 12,
- .vtotal = 800 + 12 + 12 + 12,
+ .vsync_start = 800 + 15,
+ .vsync_end = 800 + 15 + 10,
+ .vtotal = 800 + 15 + 10 + 14,
.vrefresh = 50,
.flags = 0,
.width_mm = 52,
/* Send Command GRAM memory write (no parameters) */
dcs_write_seq(ctx, MIPI_DCS_WRITE_MEMORY_START);
+ /* Wait a short while to let the panel be ready before the 1st frame */
+ mdelay(10);
+
return 0;
}
ctx->supply = devm_regulator_get(dev, "power");
if (IS_ERR(ctx->supply)) {
ret = PTR_ERR(ctx->supply);
- dev_err(dev, "failed to request regulator: %d\n", ret);
+ if (ret != -EPROBE_DEFER)
+ dev_err(dev, "failed to request regulator: %d\n", ret);
return ret;
}
ctx->supply = devm_regulator_get(dev, "power");
if (IS_ERR(ctx->supply)) {
ret = PTR_ERR(ctx->supply);
- dev_err(dev, "cannot get regulator: %d\n", ret);
+ if (ret != -EPROBE_DEFER)
+ dev_err(dev, "cannot get regulator: %d\n", ret);
return ret;
}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Rockteck jh057n00900 5.5" MIPI-DSI panel driver
+ *
+ * Copyright (C) Purism SPC 2019
+ */
+
+#include <drm/drm_mipi_dsi.h>
+#include <drm/drm_modes.h>
+#include <drm/drm_panel.h>
+#include <drm/drm_print.h>
+#include <linux/backlight.h>
+#include <linux/debugfs.h>
+#include <linux/delay.h>
+#include <linux/gpio/consumer.h>
+#include <linux/media-bus-format.h>
+#include <linux/module.h>
+#include <video/display_timing.h>
+#include <video/mipi_display.h>
+
+#define DRV_NAME "panel-rocktech-jh057n00900"
+
+/* Manufacturer specific Commands send via DSI */
+#define ST7703_CMD_ALL_PIXEL_OFF 0x22
+#define ST7703_CMD_ALL_PIXEL_ON 0x23
+#define ST7703_CMD_SETDISP 0xB2
+#define ST7703_CMD_SETRGBIF 0xB3
+#define ST7703_CMD_SETCYC 0xB4
+#define ST7703_CMD_SETBGP 0xB5
+#define ST7703_CMD_SETVCOM 0xB6
+#define ST7703_CMD_SETOTP 0xB7
+#define ST7703_CMD_SETPOWER_EXT 0xB8
+#define ST7703_CMD_SETEXTC 0xB9
+#define ST7703_CMD_SETMIPI 0xBA
+#define ST7703_CMD_SETVDC 0xBC
+#define ST7703_CMD_SETSCR 0xC0
+#define ST7703_CMD_SETPOWER 0xC1
+#define ST7703_CMD_SETPANEL 0xCC
+#define ST7703_CMD_SETGAMMA 0xE0
+#define ST7703_CMD_SETEQ 0xE3
+#define ST7703_CMD_SETGIP1 0xE9
+#define ST7703_CMD_SETGIP2 0xEA
+
+struct jh057n {
+ struct device *dev;
+ struct drm_panel panel;
+ struct gpio_desc *reset_gpio;
+ struct backlight_device *backlight;
+ bool prepared;
+
+ struct dentry *debugfs;
+};
+
+static inline struct jh057n *panel_to_jh057n(struct drm_panel *panel)
+{
+ return container_of(panel, struct jh057n, panel);
+}
+
+#define dsi_generic_write_seq(dsi, seq...) do { \
+ static const u8 d[] = { seq }; \
+ int ret; \
+ ret = mipi_dsi_generic_write(dsi, d, ARRAY_SIZE(d)); \
+ if (ret < 0) \
+ return ret; \
+ } while (0)
+
+static int jh057n_init_sequence(struct jh057n *ctx)
+{
+ struct mipi_dsi_device *dsi = to_mipi_dsi_device(ctx->dev);
+ struct device *dev = ctx->dev;
+ int ret;
+
+ /*
+ * Init sequence was supplied by the panel vendor. Most of the commands
+ * resemble the ST7703 but the number of parameters often don't match
+ * so it's likely a clone.
+ */
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETEXTC,
+ 0xF1, 0x12, 0x83);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETRGBIF,
+ 0x10, 0x10, 0x05, 0x05, 0x03, 0xFF, 0x00, 0x00,
+ 0x00, 0x00);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETSCR,
+ 0x73, 0x73, 0x50, 0x50, 0x00, 0x00, 0x08, 0x70,
+ 0x00);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETVDC, 0x4E);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETPANEL, 0x0B);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETCYC, 0x80);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETDISP, 0xF0, 0x12, 0x30);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETEQ,
+ 0x07, 0x07, 0x0B, 0x0B, 0x03, 0x0B, 0x00, 0x00,
+ 0x00, 0x00, 0xFF, 0x00, 0xC0, 0x10);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETBGP, 0x08, 0x08);
+ msleep(20);
+
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETVCOM, 0x3F, 0x3F);
+ dsi_generic_write_seq(dsi, 0xBF, 0x02, 0x11, 0x00);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETGIP1,
+ 0x82, 0x10, 0x06, 0x05, 0x9E, 0x0A, 0xA5, 0x12,
+ 0x31, 0x23, 0x37, 0x83, 0x04, 0xBC, 0x27, 0x38,
+ 0x0C, 0x00, 0x03, 0x00, 0x00, 0x00, 0x0C, 0x00,
+ 0x03, 0x00, 0x00, 0x00, 0x75, 0x75, 0x31, 0x88,
+ 0x88, 0x88, 0x88, 0x88, 0x88, 0x13, 0x88, 0x64,
+ 0x64, 0x20, 0x88, 0x88, 0x88, 0x88, 0x88, 0x88,
+ 0x02, 0x88, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETGIP2,
+ 0x02, 0x21, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x02, 0x46, 0x02, 0x88,
+ 0x88, 0x88, 0x88, 0x88, 0x88, 0x64, 0x88, 0x13,
+ 0x57, 0x13, 0x88, 0x88, 0x88, 0x88, 0x88, 0x88,
+ 0x75, 0x88, 0x23, 0x14, 0x00, 0x00, 0x02, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x30, 0x0A,
+ 0xA5, 0x00, 0x00, 0x00, 0x00);
+ dsi_generic_write_seq(dsi, ST7703_CMD_SETGAMMA,
+ 0x00, 0x09, 0x0E, 0x29, 0x2D, 0x3C, 0x41, 0x37,
+ 0x07, 0x0B, 0x0D, 0x10, 0x11, 0x0F, 0x10, 0x11,
+ 0x18, 0x00, 0x09, 0x0E, 0x29, 0x2D, 0x3C, 0x41,
+ 0x37, 0x07, 0x0B, 0x0D, 0x10, 0x11, 0x0F, 0x10,
+ 0x11, 0x18);
+ msleep(20);
+
+ ret = mipi_dsi_dcs_exit_sleep_mode(dsi);
+ if (ret < 0) {
+ DRM_DEV_ERROR(dev, "Failed to exit sleep mode\n");
+ return ret;
+ }
+ /* Panel is operational 120 msec after reset */
+ msleep(60);
+ ret = mipi_dsi_dcs_set_display_on(dsi);
+ if (ret)
+ return ret;
+
+ DRM_DEV_DEBUG_DRIVER(dev, "Panel init sequence done\n");
+ return 0;
+}
+
+static int jh057n_enable(struct drm_panel *panel)
+{
+ struct jh057n *ctx = panel_to_jh057n(panel);
+
+ return backlight_enable(ctx->backlight);
+}
+
+static int jh057n_disable(struct drm_panel *panel)
+{
+ struct jh057n *ctx = panel_to_jh057n(panel);
+
+ return backlight_disable(ctx->backlight);
+}
+
+static int jh057n_unprepare(struct drm_panel *panel)
+{
+ struct jh057n *ctx = panel_to_jh057n(panel);
+ struct mipi_dsi_device *dsi = to_mipi_dsi_device(ctx->dev);
+
+ if (!ctx->prepared)
+ return 0;
+
+ mipi_dsi_dcs_set_display_off(dsi);
+ ctx->prepared = false;
+
+ return 0;
+}
+
+static int jh057n_prepare(struct drm_panel *panel)
+{
+ struct jh057n *ctx = panel_to_jh057n(panel);
+ int ret;
+
+ if (ctx->prepared)
+ return 0;
+
+ DRM_DEV_DEBUG_DRIVER(ctx->dev, "Resetting the panel\n");
+ gpiod_set_value_cansleep(ctx->reset_gpio, 1);
+ usleep_range(20, 40);
+ gpiod_set_value_cansleep(ctx->reset_gpio, 0);
+ msleep(20);
+
+ ret = jh057n_init_sequence(ctx);
+ if (ret < 0) {
+ DRM_DEV_ERROR(ctx->dev, "Panel init sequence failed: %d\n",
+ ret);
+ return ret;
+ }
+
+ ctx->prepared = true;
+
+ return 0;
+}
+
+static const struct drm_display_mode default_mode = {
+ .hdisplay = 720,
+ .hsync_start = 720 + 90,
+ .hsync_end = 720 + 90 + 20,
+ .htotal = 720 + 90 + 20 + 20,
+ .vdisplay = 1440,
+ .vsync_start = 1440 + 20,
+ .vsync_end = 1440 + 20 + 4,
+ .vtotal = 1440 + 20 + 4 + 12,
+ .vrefresh = 60,
+ .clock = 75276,
+ .flags = DRM_MODE_FLAG_NHSYNC | DRM_MODE_FLAG_NVSYNC,
+ .width_mm = 65,
+ .height_mm = 130,
+};
+
+static int jh057n_get_modes(struct drm_panel *panel)
+{
+ struct jh057n *ctx = panel_to_jh057n(panel);
+ struct drm_display_mode *mode;
+
+ mode = drm_mode_duplicate(panel->drm, &default_mode);
+ if (!mode) {
+ DRM_DEV_ERROR(ctx->dev, "Failed to add mode %ux%u@%u\n",
+ default_mode.hdisplay, default_mode.vdisplay,
+ default_mode.vrefresh);
+ return -ENOMEM;
+ }
+
+ drm_mode_set_name(mode);
+
+ mode->type = DRM_MODE_TYPE_DRIVER | DRM_MODE_TYPE_PREFERRED;
+ panel->connector->display_info.width_mm = mode->width_mm;
+ panel->connector->display_info.height_mm = mode->height_mm;
+ drm_mode_probed_add(panel->connector, mode);
+
+ return 1;
+}
+
+static const struct drm_panel_funcs jh057n_drm_funcs = {
+ .disable = jh057n_disable,
+ .unprepare = jh057n_unprepare,
+ .prepare = jh057n_prepare,
+ .enable = jh057n_enable,
+ .get_modes = jh057n_get_modes,
+};
+
+static int allpixelson_set(void *data, u64 val)
+{
+ struct jh057n *ctx = data;
+ struct mipi_dsi_device *dsi = to_mipi_dsi_device(ctx->dev);
+
+ DRM_DEV_DEBUG_DRIVER(ctx->dev, "Setting all pixels on\n");
+ dsi_generic_write_seq(dsi, ST7703_CMD_ALL_PIXEL_ON);
+ msleep(val * 1000);
+ /* Reset the panel to get video back */
+ drm_panel_disable(&ctx->panel);
+ drm_panel_unprepare(&ctx->panel);
+ drm_panel_prepare(&ctx->panel);
+ drm_panel_enable(&ctx->panel);
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(allpixelson_fops, NULL,
+ allpixelson_set, "%llu\n");
+
+static int jh057n_debugfs_init(struct jh057n *ctx)
+{
+ struct dentry *f;
+
+ ctx->debugfs = debugfs_create_dir(DRV_NAME, NULL);
+ if (!ctx->debugfs)
+ return -ENOMEM;
+
+ f = debugfs_create_file("allpixelson", 0600,
+ ctx->debugfs, ctx, &allpixelson_fops);
+ if (!f)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static void jh057n_debugfs_remove(struct jh057n *ctx)
+{
+ debugfs_remove_recursive(ctx->debugfs);
+ ctx->debugfs = NULL;
+}
+
+static int jh057n_probe(struct mipi_dsi_device *dsi)
+{
+ struct device *dev = &dsi->dev;
+ struct jh057n *ctx;
+ int ret;
+
+ ctx = devm_kzalloc(dev, sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return -ENOMEM;
+
+ ctx->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_LOW);
+ if (IS_ERR(ctx->reset_gpio)) {
+ DRM_DEV_ERROR(dev, "cannot get reset gpio\n");
+ return PTR_ERR(ctx->reset_gpio);
+ }
+
+ mipi_dsi_set_drvdata(dsi, ctx);
+
+ ctx->dev = dev;
+
+ dsi->lanes = 4;
+ dsi->format = MIPI_DSI_FMT_RGB888;
+ dsi->mode_flags = MIPI_DSI_MODE_VIDEO |
+ MIPI_DSI_MODE_VIDEO_BURST | MIPI_DSI_MODE_VIDEO_SYNC_PULSE;
+
+ ctx->backlight = devm_of_find_backlight(dev);
+ if (IS_ERR(ctx->backlight))
+ return PTR_ERR(ctx->backlight);
+
+ drm_panel_init(&ctx->panel);
+ ctx->panel.dev = dev;
+ ctx->panel.funcs = &jh057n_drm_funcs;
+
+ drm_panel_add(&ctx->panel);
+
+ ret = mipi_dsi_attach(dsi);
+ if (ret < 0) {
+ DRM_DEV_ERROR(dev, "mipi_dsi_attach failed. Is host ready?\n");
+ drm_panel_remove(&ctx->panel);
+ return ret;
+ }
+
+ DRM_DEV_INFO(dev, "%ux%u@%u %ubpp dsi %udl - ready\n",
+ default_mode.hdisplay, default_mode.vdisplay,
+ default_mode.vrefresh,
+ mipi_dsi_pixel_format_to_bpp(dsi->format), dsi->lanes);
+
+ jh057n_debugfs_init(ctx);
+ return 0;
+}
+
+static void jh057n_shutdown(struct mipi_dsi_device *dsi)
+{
+ struct jh057n *ctx = mipi_dsi_get_drvdata(dsi);
+ int ret;
+
+ ret = jh057n_unprepare(&ctx->panel);
+ if (ret < 0)
+ DRM_DEV_ERROR(&dsi->dev, "Failed to unprepare panel: %d\n",
+ ret);
+
+ ret = jh057n_disable(&ctx->panel);
+ if (ret < 0)
+ DRM_DEV_ERROR(&dsi->dev, "Failed to disable panel: %d\n",
+ ret);
+}
+
+static int jh057n_remove(struct mipi_dsi_device *dsi)
+{
+ struct jh057n *ctx = mipi_dsi_get_drvdata(dsi);
+ int ret;
+
+ jh057n_shutdown(dsi);
+
+ ret = mipi_dsi_detach(dsi);
+ if (ret < 0)
+ DRM_DEV_ERROR(&dsi->dev, "Failed to detach from DSI host: %d\n",
+ ret);
+
+ drm_panel_remove(&ctx->panel);
+
+ jh057n_debugfs_remove(ctx);
+
+ return 0;
+}
+
+static const struct of_device_id jh057n_of_match[] = {
+ { .compatible = "rocktech,jh057n00900" },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, jh057n_of_match);
+
+static struct mipi_dsi_driver jh057n_driver = {
+ .probe = jh057n_probe,
+ .remove = jh057n_remove,
+ .shutdown = jh057n_shutdown,
+ .driver = {
+ .name = DRV_NAME,
+ .of_match_table = jh057n_of_match,
+ },
+};
+module_mipi_dsi_driver(jh057n_driver);
+
+MODULE_AUTHOR("Guido Günther <agx@sigxcpu.org>");
+MODULE_DESCRIPTION("DRM driver for Rocktech JH057N00900 MIPI DSI panel");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Copyright (C) 2018-2019, Bridge Systems BV
+ * Copyright (C) 2018-2019, Bootlin
+ * Copyright (C) 2017, Free Electrons
+ *
+ * This file based on panel-ilitek-ili9881c.c
+ */
+
+#include <linux/backlight.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/errno.h>
+#include <linux/fb.h>
+#include <linux/kernel.h>
+#include <linux/media-bus-format.h>
+#include <linux/module.h>
+
+#include <linux/gpio/consumer.h>
+#include <linux/regulator/consumer.h>
+
+#include <drm/drm_connector.h>
+#include <drm/drm_mipi_dsi.h>
+#include <drm/drm_modes.h>
+#include <drm/drm_panel.h>
+#include <drm/drm_print.h>
+
+struct rb070d30_panel {
+ struct drm_panel panel;
+ struct mipi_dsi_device *dsi;
+ struct backlight_device *backlight;
+ struct regulator *supply;
+
+ struct {
+ struct gpio_desc *power;
+ struct gpio_desc *reset;
+ struct gpio_desc *updn;
+ struct gpio_desc *shlr;
+ } gpios;
+};
+
+static inline struct rb070d30_panel *panel_to_rb070d30_panel(struct drm_panel *panel)
+{
+ return container_of(panel, struct rb070d30_panel, panel);
+}
+
+static int rb070d30_panel_prepare(struct drm_panel *panel)
+{
+ struct rb070d30_panel *ctx = panel_to_rb070d30_panel(panel);
+ int ret;
+
+ ret = regulator_enable(ctx->supply);
+ if (ret < 0) {
+ DRM_DEV_ERROR(&ctx->dsi->dev, "Failed to enable supply: %d\n", ret);
+ return ret;
+ }
+
+ msleep(20);
+ gpiod_set_value(ctx->gpios.power, 1);
+ msleep(20);
+ gpiod_set_value(ctx->gpios.reset, 1);
+ msleep(20);
+ return 0;
+}
+
+static int rb070d30_panel_unprepare(struct drm_panel *panel)
+{
+ struct rb070d30_panel *ctx = panel_to_rb070d30_panel(panel);
+
+ gpiod_set_value(ctx->gpios.reset, 0);
+ gpiod_set_value(ctx->gpios.power, 0);
+ regulator_disable(ctx->supply);
+
+ return 0;
+}
+
+static int rb070d30_panel_enable(struct drm_panel *panel)
+{
+ struct rb070d30_panel *ctx = panel_to_rb070d30_panel(panel);
+ int ret;
+
+ ret = mipi_dsi_dcs_exit_sleep_mode(ctx->dsi);
+ if (ret)
+ return ret;
+
+ ret = backlight_enable(ctx->backlight);
+ if (ret)
+ goto out;
+
+ return 0;
+
+out:
+ mipi_dsi_dcs_enter_sleep_mode(ctx->dsi);
+ return ret;
+}
+
+static int rb070d30_panel_disable(struct drm_panel *panel)
+{
+ struct rb070d30_panel *ctx = panel_to_rb070d30_panel(panel);
+
+ backlight_disable(ctx->backlight);
+ return mipi_dsi_dcs_enter_sleep_mode(ctx->dsi);
+}
+
+/* Default timings */
+static const struct drm_display_mode default_mode = {
+ .clock = 51206,
+ .hdisplay = 1024,
+ .hsync_start = 1024 + 160,
+ .hsync_end = 1024 + 160 + 80,
+ .htotal = 1024 + 160 + 80 + 80,
+ .vdisplay = 600,
+ .vsync_start = 600 + 12,
+ .vsync_end = 600 + 12 + 10,
+ .vtotal = 600 + 12 + 10 + 13,
+ .vrefresh = 60,
+
+ .width_mm = 154,
+ .height_mm = 85,
+};
+
+static int rb070d30_panel_get_modes(struct drm_panel *panel)
+{
+ struct drm_connector *connector = panel->connector;
+ struct rb070d30_panel *ctx = panel_to_rb070d30_panel(panel);
+ struct drm_display_mode *mode;
+ static const u32 bus_format = MEDIA_BUS_FMT_RGB888_1X24;
+
+ mode = drm_mode_duplicate(panel->drm, &default_mode);
+ if (!mode) {
+ DRM_DEV_ERROR(&ctx->dsi->dev,
+ "Failed to add mode " DRM_MODE_FMT "\n",
+ DRM_MODE_ARG(&default_mode));
+ return -EINVAL;
+ }
+
+ drm_mode_set_name(mode);
+
+ mode->type = DRM_MODE_TYPE_DRIVER | DRM_MODE_TYPE_PREFERRED;
+ drm_mode_probed_add(connector, mode);
+
+ panel->connector->display_info.bpc = 8;
+ panel->connector->display_info.width_mm = mode->width_mm;
+ panel->connector->display_info.height_mm = mode->height_mm;
+ drm_display_info_set_bus_formats(&connector->display_info,
+ &bus_format, 1);
+
+ return 1;
+}
+
+static const struct drm_panel_funcs rb070d30_panel_funcs = {
+ .get_modes = rb070d30_panel_get_modes,
+ .prepare = rb070d30_panel_prepare,
+ .enable = rb070d30_panel_enable,
+ .disable = rb070d30_panel_disable,
+ .unprepare = rb070d30_panel_unprepare,
+};
+
+static int rb070d30_panel_dsi_probe(struct mipi_dsi_device *dsi)
+{
+ struct rb070d30_panel *ctx;
+ int ret;
+
+ ctx = devm_kzalloc(&dsi->dev, sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return -ENOMEM;
+
+ ctx->supply = devm_regulator_get(&dsi->dev, "vcc-lcd");
+ if (IS_ERR(ctx->supply))
+ return PTR_ERR(ctx->supply);
+
+ mipi_dsi_set_drvdata(dsi, ctx);
+ ctx->dsi = dsi;
+
+ drm_panel_init(&ctx->panel);
+ ctx->panel.dev = &dsi->dev;
+ ctx->panel.funcs = &rb070d30_panel_funcs;
+
+ ctx->gpios.reset = devm_gpiod_get(&dsi->dev, "reset", GPIOD_OUT_LOW);
+ if (IS_ERR(ctx->gpios.reset)) {
+ DRM_DEV_ERROR(&dsi->dev, "Couldn't get our reset GPIO\n");
+ return PTR_ERR(ctx->gpios.reset);
+ }
+
+ ctx->gpios.power = devm_gpiod_get(&dsi->dev, "power", GPIOD_OUT_LOW);
+ if (IS_ERR(ctx->gpios.power)) {
+ DRM_DEV_ERROR(&dsi->dev, "Couldn't get our power GPIO\n");
+ return PTR_ERR(ctx->gpios.power);
+ }
+
+ /*
+ * We don't change the state of that GPIO later on but we need
+ * to force it into a low state.
+ */
+ ctx->gpios.updn = devm_gpiod_get(&dsi->dev, "updn", GPIOD_OUT_LOW);
+ if (IS_ERR(ctx->gpios.updn)) {
+ DRM_DEV_ERROR(&dsi->dev, "Couldn't get our updn GPIO\n");
+ return PTR_ERR(ctx->gpios.updn);
+ }
+
+ /*
+ * We don't change the state of that GPIO later on but we need
+ * to force it into a low state.
+ */
+ ctx->gpios.shlr = devm_gpiod_get(&dsi->dev, "shlr", GPIOD_OUT_LOW);
+ if (IS_ERR(ctx->gpios.shlr)) {
+ DRM_DEV_ERROR(&dsi->dev, "Couldn't get our shlr GPIO\n");
+ return PTR_ERR(ctx->gpios.shlr);
+ }
+
+ ctx->backlight = devm_of_find_backlight(&dsi->dev);
+ if (IS_ERR(ctx->backlight)) {
+ DRM_DEV_ERROR(&dsi->dev, "Couldn't get our backlight\n");
+ return PTR_ERR(ctx->backlight);
+ }
+
+ ret = drm_panel_add(&ctx->panel);
+ if (ret < 0)
+ return ret;
+
+ dsi->mode_flags = MIPI_DSI_MODE_VIDEO | MIPI_DSI_MODE_VIDEO_BURST | MIPI_DSI_MODE_LPM;
+ dsi->format = MIPI_DSI_FMT_RGB888;
+ dsi->lanes = 4;
+
+ return mipi_dsi_attach(dsi);
+}
+
+static int rb070d30_panel_dsi_remove(struct mipi_dsi_device *dsi)
+{
+ struct rb070d30_panel *ctx = mipi_dsi_get_drvdata(dsi);
+
+ mipi_dsi_detach(dsi);
+ drm_panel_remove(&ctx->panel);
+
+ return 0;
+}
+
+static const struct of_device_id rb070d30_panel_of_match[] = {
+ { .compatible = "ronbo,rb070d30" },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, rb070d30_panel_of_match);
+
+static struct mipi_dsi_driver rb070d30_panel_driver = {
+ .probe = rb070d30_panel_dsi_probe,
+ .remove = rb070d30_panel_dsi_remove,
+ .driver = {
+ .name = "panel-ronbo-rb070d30",
+ .of_match_table = rb070d30_panel_of_match,
+ },
+};
+module_mipi_dsi_driver(rb070d30_panel_driver);
+
+MODULE_AUTHOR("Boris Brezillon <boris.brezillon@bootlin.com>");
+MODULE_AUTHOR("Konstantin Sudakov <k.sudakov@integrasources.com>");
+MODULE_DESCRIPTION("Ronbo RB070D30 Panel Driver");
+MODULE_LICENSE("GPL");
struct drm_connector *connector = panel->connector;
struct drm_display_mode *mode;
- strncpy(connector->display_info.name, "Samsung S6D16D0\0",
- DRM_DISPLAY_INFO_LEN);
-
mode = drm_mode_duplicate(panel->drm, &samsung_s6d16d0_mode);
if (!mode) {
DRM_ERROR("bad mode or failed to add mode\n");
.height = 57,
},
.bus_format = MEDIA_BUS_FMT_RGB888_1X24,
- .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_NEGEDGE,
+ .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE,
};
static const struct of_device_id platform_of_match[] = {
.width = 95,
.height = 54,
},
- .bus_flags = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
};
static const struct drm_display_mode cdtech_s070wv95_ct16_mode = {
.height = 91,
},
.bus_format = MEDIA_BUS_FMT_RGB888_1X24,
- .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
};
static const struct display_timing dlc_dlc0700yzg_1_timing = {
.height = 86,
},
.bus_format = MEDIA_BUS_FMT_RGB666_1X18,
- .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_NEGEDGE,
+ .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE,
};
static const struct drm_display_mode edt_etm0700g0dh6_mode = {
.height = 91,
},
.bus_format = MEDIA_BUS_FMT_RGB666_1X18,
- .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_NEGEDGE,
+ .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE,
};
static const struct panel_desc edt_etm0700g0bdh6 = {
.height = 91,
},
.bus_format = MEDIA_BUS_FMT_RGB666_1X18,
- .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
};
static const struct drm_display_mode foxlink_fl500wvr00_a0t_mode = {
.height = 54,
},
.bus_format = MEDIA_BUS_FMT_RGB888_1X24,
- .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
};
static const struct drm_display_mode innolux_at070tn92_mode = {
.height = 54,
},
.bus_format = MEDIA_BUS_FMT_RGB888_1X24,
- .bus_flags = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
};
static const struct drm_display_mode netron_dy_e231732_mode = {
.height = 54,
},
.bus_format = MEDIA_BUS_FMT_RGB888_1X24,
- .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_POSEDGE |
- DRM_BUS_FLAG_SYNC_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE |
+ DRM_BUS_FLAG_SYNC_DRIVE_POSEDGE,
};
static const struct display_timing nlt_nl192108ac18_02d_timing = {
.height = 93,
},
.bus_format = MEDIA_BUS_FMT_RGB888_1X24,
- .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
+};
+
+static const struct drm_display_mode osddisplays_osd070t1718_19ts_mode = {
+ .clock = 33000,
+ .hdisplay = 800,
+ .hsync_start = 800 + 210,
+ .hsync_end = 800 + 210 + 30,
+ .htotal = 800 + 210 + 30 + 16,
+ .vdisplay = 480,
+ .vsync_start = 480 + 22,
+ .vsync_end = 480 + 22 + 13,
+ .vtotal = 480 + 22 + 13 + 10,
+ .vrefresh = 60,
+ .flags = DRM_MODE_FLAG_NVSYNC | DRM_MODE_FLAG_NHSYNC,
+};
+
+static const struct panel_desc osddisplays_osd070t1718_19ts = {
+ .modes = &osddisplays_osd070t1718_19ts_mode,
+ .num_modes = 1,
+ .bpc = 8,
+ .size = {
+ .width = 152,
+ .height = 91,
+ },
+ .bus_format = MEDIA_BUS_FMT_RGB888_1X24,
+ .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
};
static const struct drm_display_mode pda_91_00156_a0_mode = {
.height = 116,
},
.bus_format = MEDIA_BUS_FMT_RGB888_1X24,
- .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_DE_HIGH | DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
};
static const struct drm_display_mode tpk_f07a_0102_mode = {
.width = 152,
.height = 91,
},
- .bus_flags = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
};
static const struct drm_display_mode tpk_f10a_0102_mode = {
}, {
.compatible = "ortustech,com43h4m85ulc",
.data = &ortustech_com43h4m85ulc,
+ }, {
+ .compatible = "osddisplays,osd070t1718-19ts",
+ .data = &osddisplays_osd070t1718_19ts,
}, {
.compatible = "pda,91-00156-a0",
.data = &pda_91_00156_a0,
.lanes = 4,
};
+static const struct drm_display_mode lg_acx467akm_7_mode = {
+ .clock = 150000,
+ .hdisplay = 1080,
+ .hsync_start = 1080 + 2,
+ .hsync_end = 1080 + 2 + 2,
+ .htotal = 1080 + 2 + 2 + 2,
+ .vdisplay = 1920,
+ .vsync_start = 1920 + 2,
+ .vsync_end = 1920 + 2 + 2,
+ .vtotal = 1920 + 2 + 2 + 2,
+ .vrefresh = 60,
+};
+
+static const struct panel_desc_dsi lg_acx467akm_7 = {
+ .desc = {
+ .modes = &lg_acx467akm_7_mode,
+ .num_modes = 1,
+ .bpc = 8,
+ .size = {
+ .width = 62,
+ .height = 110,
+ },
+ },
+ .flags = 0,
+ .format = MIPI_DSI_FMT_RGB888,
+ .lanes = 4,
+};
+
static const struct of_device_id dsi_of_match[] = {
{
.compatible = "auo,b080uan01",
}, {
.compatible = "panasonic,vvx10f004b00",
.data = &panasonic_vvx10f004b00
+ }, {
+ .compatible = "lg,acx467akm-7",
+ .data = &lg_acx467akm_7
}, {
/* sentinel */
}
.vtotal = 480 + 10 + 1 + 35,
.vrefresh = 60,
},
- .bus_flags = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
},
{
.name = "640x480 RGB",
.vtotal = 480 + 18 + 1 + 27,
.vrefresh = 60,
},
- .bus_flags = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
},
{
.name = "480x272 RGB",
.vtotal = 272 + 2 + 1 + 12,
.vrefresh = 60,
},
- .bus_flags = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
},
{
.name = "480x640 RGB",
.vtotal = 640 + 4 + 1 + 8,
.vrefresh = 60,
},
- .bus_flags = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
},
{
.name = "400x240 RGB",
.vtotal = 240 + 2 + 1 + 20,
.vrefresh = 60,
},
- .bus_flags = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE,
},
};
struct tpg110 *tpg = to_tpg110(panel);
struct drm_display_mode *mode;
- strncpy(connector->display_info.name, tpg->panel_mode->name,
- DRM_DISPLAY_INFO_LEN);
connector->display_info.width_mm = tpg->width;
connector->display_info.height_mm = tpg->height;
connector->display_info.bus_flags = tpg->panel_mode->bus_flags;
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+
+config DRM_PANFROST
+ tristate "Panfrost (DRM support for ARM Mali Midgard/Bifrost GPUs)"
+ depends on DRM
+ depends on ARM || ARM64 || (COMPILE_TEST && !GENERIC_ATOMIC64)
+ depends on MMU
+ select DRM_SCHED
+ select IOMMU_SUPPORT
+ select IOMMU_IO_PGTABLE_LPAE
+ select DRM_GEM_SHMEM_HELPER
+ help
+ DRM driver for ARM Mali Midgard (T6xx, T7xx, T8xx) and
+ Bifrost (G3x, G5x, G7x) GPUs.
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+
+panfrost-y := \
+ panfrost_drv.o \
+ panfrost_device.o \
+ panfrost_devfreq.o \
+ panfrost_gem.o \
+ panfrost_gpu.o \
+ panfrost_job.o \
+ panfrost_mmu.o
+
+obj-$(CONFIG_DRM_PANFROST) += panfrost.o
--- /dev/null
+- Thermal support.
+
+- Bifrost support:
+ - DT bindings (Neil, WIP)
+ - MMU page table format and address space setup
+ - Bifrost specific feature and issue handling
+ - Coherent DMA support
+
+- Support for 2MB pages. The io-pgtable code already supports this. Finishing
+ support involves either copying or adapting the iommu API to handle passing
+ aligned addresses and sizes to the io-pgtable code.
+
+- Per FD address space support. The h/w supports multiple addresses spaces.
+ The hard part is handling when more address spaces are needed than what
+ the h/w provides.
+
+- Support pinning pages on demand (GPU page faults).
+
+- Support userspace controlled GPU virtual addresses. Needed for Vulkan. (Tomeu)
+
+- Support for madvise and a shrinker.
+
+- Compute job support. So called 'compute only' jobs need to be plumbed up to
+ userspace.
+
+- Performance counter support. (Boris)
+
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2019 Collabora ltd. */
+#include <linux/devfreq.h>
+#include <linux/platform_device.h>
+#include <linux/pm_opp.h>
+#include <linux/clk.h>
+#include <linux/regulator/consumer.h>
+
+#include "panfrost_device.h"
+#include "panfrost_devfreq.h"
+#include "panfrost_features.h"
+#include "panfrost_issues.h"
+#include "panfrost_gpu.h"
+#include "panfrost_regs.h"
+
+static void panfrost_devfreq_update_utilization(struct panfrost_device *pfdev, int slot);
+
+static int panfrost_devfreq_target(struct device *dev, unsigned long *freq,
+ u32 flags)
+{
+ struct panfrost_device *pfdev = platform_get_drvdata(to_platform_device(dev));
+ struct dev_pm_opp *opp;
+ unsigned long old_clk_rate = pfdev->devfreq.cur_freq;
+ unsigned long target_volt, target_rate;
+ int err;
+
+ opp = devfreq_recommended_opp(dev, freq, flags);
+ if (IS_ERR(opp))
+ return PTR_ERR(opp);
+
+ target_rate = dev_pm_opp_get_freq(opp);
+ target_volt = dev_pm_opp_get_voltage(opp);
+ dev_pm_opp_put(opp);
+
+ if (old_clk_rate == target_rate)
+ return 0;
+
+ /*
+ * If frequency scaling from low to high, adjust voltage first.
+ * If frequency scaling from high to low, adjust frequency first.
+ */
+ if (old_clk_rate < target_rate) {
+ err = regulator_set_voltage(pfdev->regulator, target_volt,
+ target_volt);
+ if (err) {
+ dev_err(dev, "Cannot set voltage %lu uV\n",
+ target_volt);
+ return err;
+ }
+ }
+
+ err = clk_set_rate(pfdev->clock, target_rate);
+ if (err) {
+ dev_err(dev, "Cannot set frequency %lu (%d)\n", target_rate,
+ err);
+ regulator_set_voltage(pfdev->regulator, pfdev->devfreq.cur_volt,
+ pfdev->devfreq.cur_volt);
+ return err;
+ }
+
+ if (old_clk_rate > target_rate) {
+ err = regulator_set_voltage(pfdev->regulator, target_volt,
+ target_volt);
+ if (err)
+ dev_err(dev, "Cannot set voltage %lu uV\n", target_volt);
+ }
+
+ pfdev->devfreq.cur_freq = target_rate;
+ pfdev->devfreq.cur_volt = target_volt;
+
+ return 0;
+}
+
+static void panfrost_devfreq_reset(struct panfrost_device *pfdev)
+{
+ ktime_t now = ktime_get();
+ int i;
+
+ for (i = 0; i < NUM_JOB_SLOTS; i++) {
+ pfdev->devfreq.slot[i].busy_time = 0;
+ pfdev->devfreq.slot[i].idle_time = 0;
+ pfdev->devfreq.slot[i].time_last_update = now;
+ }
+}
+
+static int panfrost_devfreq_get_dev_status(struct device *dev,
+ struct devfreq_dev_status *status)
+{
+ struct panfrost_device *pfdev = platform_get_drvdata(to_platform_device(dev));
+ int i;
+
+ for (i = 0; i < NUM_JOB_SLOTS; i++) {
+ panfrost_devfreq_update_utilization(pfdev, i);
+ }
+
+ status->current_frequency = clk_get_rate(pfdev->clock);
+ status->total_time = ktime_to_ns(ktime_add(pfdev->devfreq.slot[0].busy_time,
+ pfdev->devfreq.slot[0].idle_time));
+
+ status->busy_time = 0;
+ for (i = 0; i < NUM_JOB_SLOTS; i++) {
+ status->busy_time += ktime_to_ns(pfdev->devfreq.slot[i].busy_time);
+ }
+
+ /* We're scheduling only to one core atm, so don't divide for now */
+ /* status->busy_time /= NUM_JOB_SLOTS; */
+
+ panfrost_devfreq_reset(pfdev);
+
+ dev_dbg(pfdev->dev, "busy %lu total %lu %lu %% freq %lu MHz\n", status->busy_time,
+ status->total_time,
+ status->busy_time / (status->total_time / 100),
+ status->current_frequency / 1000 / 1000);
+
+ return 0;
+}
+
+static int panfrost_devfreq_get_cur_freq(struct device *dev, unsigned long *freq)
+{
+ struct panfrost_device *pfdev = platform_get_drvdata(to_platform_device(dev));
+
+ *freq = pfdev->devfreq.cur_freq;
+
+ return 0;
+}
+
+static struct devfreq_dev_profile panfrost_devfreq_profile = {
+ .polling_ms = 50, /* ~3 frames */
+ .target = panfrost_devfreq_target,
+ .get_dev_status = panfrost_devfreq_get_dev_status,
+ .get_cur_freq = panfrost_devfreq_get_cur_freq,
+};
+
+int panfrost_devfreq_init(struct panfrost_device *pfdev)
+{
+ int ret;
+ struct dev_pm_opp *opp;
+
+ if (!pfdev->regulator)
+ return 0;
+
+ ret = dev_pm_opp_of_add_table(&pfdev->pdev->dev);
+ if (ret == -ENODEV) /* Optional, continue without devfreq */
+ return 0;
+
+ panfrost_devfreq_reset(pfdev);
+
+ pfdev->devfreq.cur_freq = clk_get_rate(pfdev->clock);
+
+ opp = devfreq_recommended_opp(&pfdev->pdev->dev, &pfdev->devfreq.cur_freq, 0);
+ if (IS_ERR(opp))
+ return PTR_ERR(opp);
+
+ panfrost_devfreq_profile.initial_freq = pfdev->devfreq.cur_freq;
+ dev_pm_opp_put(opp);
+
+ pfdev->devfreq.devfreq = devm_devfreq_add_device(&pfdev->pdev->dev,
+ &panfrost_devfreq_profile, "simple_ondemand", NULL);
+ if (IS_ERR(pfdev->devfreq.devfreq)) {
+ DRM_DEV_ERROR(&pfdev->pdev->dev, "Couldn't initialize GPU devfreq\n");
+ ret = PTR_ERR(pfdev->devfreq.devfreq);
+ pfdev->devfreq.devfreq = NULL;
+ return ret;
+ }
+
+ return 0;
+}
+
+void panfrost_devfreq_resume(struct panfrost_device *pfdev)
+{
+ int i;
+
+ if (!pfdev->devfreq.devfreq)
+ return;
+
+ panfrost_devfreq_reset(pfdev);
+ for (i = 0; i < NUM_JOB_SLOTS; i++)
+ pfdev->devfreq.slot[i].busy = false;
+
+ devfreq_resume_device(pfdev->devfreq.devfreq);
+}
+
+void panfrost_devfreq_suspend(struct panfrost_device *pfdev)
+{
+ if (!pfdev->devfreq.devfreq)
+ return;
+
+ devfreq_suspend_device(pfdev->devfreq.devfreq);
+}
+
+static void panfrost_devfreq_update_utilization(struct panfrost_device *pfdev, int slot)
+{
+ struct panfrost_devfreq_slot *devfreq_slot = &pfdev->devfreq.slot[slot];
+ ktime_t now;
+ ktime_t last;
+
+ if (!pfdev->devfreq.devfreq)
+ return;
+
+ now = ktime_get();
+ last = pfdev->devfreq.slot[slot].time_last_update;
+
+ /* If we last recorded a transition to busy, we have been idle since */
+ if (devfreq_slot->busy)
+ pfdev->devfreq.slot[slot].busy_time += ktime_sub(now, last);
+ else
+ pfdev->devfreq.slot[slot].idle_time += ktime_sub(now, last);
+
+ pfdev->devfreq.slot[slot].time_last_update = now;
+}
+
+/* The job scheduler is expected to call this at every transition busy <-> idle */
+void panfrost_devfreq_record_transition(struct panfrost_device *pfdev, int slot)
+{
+ struct panfrost_devfreq_slot *devfreq_slot = &pfdev->devfreq.slot[slot];
+
+ panfrost_devfreq_update_utilization(pfdev, slot);
+ devfreq_slot->busy = !devfreq_slot->busy;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2019 Collabora ltd. */
+
+#ifndef __PANFROST_DEVFREQ_H__
+#define __PANFROST_DEVFREQ_H__
+
+int panfrost_devfreq_init(struct panfrost_device *pfdev);
+
+void panfrost_devfreq_resume(struct panfrost_device *pfdev);
+void panfrost_devfreq_suspend(struct panfrost_device *pfdev);
+
+void panfrost_devfreq_record_transition(struct panfrost_device *pfdev, int slot);
+
+#endif /* __PANFROST_DEVFREQ_H__ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
+/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
+
+#include <linux/clk.h>
+#include <linux/reset.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/regulator/consumer.h>
+
+#include "panfrost_device.h"
+#include "panfrost_devfreq.h"
+#include "panfrost_features.h"
+#include "panfrost_gpu.h"
+#include "panfrost_job.h"
+#include "panfrost_mmu.h"
+
+static int panfrost_reset_init(struct panfrost_device *pfdev)
+{
+ int err;
+
+ pfdev->rstc = devm_reset_control_array_get(pfdev->dev, false, true);
+ if (IS_ERR(pfdev->rstc)) {
+ dev_err(pfdev->dev, "get reset failed %ld\n", PTR_ERR(pfdev->rstc));
+ return PTR_ERR(pfdev->rstc);
+ }
+
+ err = reset_control_deassert(pfdev->rstc);
+ if (err)
+ return err;
+
+ return 0;
+}
+
+static void panfrost_reset_fini(struct panfrost_device *pfdev)
+{
+ reset_control_assert(pfdev->rstc);
+}
+
+static int panfrost_clk_init(struct panfrost_device *pfdev)
+{
+ int err;
+ unsigned long rate;
+
+ pfdev->clock = devm_clk_get(pfdev->dev, NULL);
+ if (IS_ERR(pfdev->clock)) {
+ dev_err(pfdev->dev, "get clock failed %ld\n", PTR_ERR(pfdev->clock));
+ return PTR_ERR(pfdev->clock);
+ }
+
+ rate = clk_get_rate(pfdev->clock);
+ dev_info(pfdev->dev, "clock rate = %lu\n", rate);
+
+ err = clk_prepare_enable(pfdev->clock);
+ if (err)
+ return err;
+
+ return 0;
+}
+
+static void panfrost_clk_fini(struct panfrost_device *pfdev)
+{
+ clk_disable_unprepare(pfdev->clock);
+}
+
+static int panfrost_regulator_init(struct panfrost_device *pfdev)
+{
+ int ret;
+
+ pfdev->regulator = devm_regulator_get_optional(pfdev->dev, "mali");
+ if (IS_ERR(pfdev->regulator)) {
+ ret = PTR_ERR(pfdev->regulator);
+ pfdev->regulator = NULL;
+ if (ret == -ENODEV)
+ return 0;
+ dev_err(pfdev->dev, "failed to get regulator: %d\n", ret);
+ return ret;
+ }
+
+ ret = regulator_enable(pfdev->regulator);
+ if (ret < 0) {
+ dev_err(pfdev->dev, "failed to enable regulator: %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void panfrost_regulator_fini(struct panfrost_device *pfdev)
+{
+ if (pfdev->regulator)
+ regulator_disable(pfdev->regulator);
+}
+
+int panfrost_device_init(struct panfrost_device *pfdev)
+{
+ int err;
+ struct resource *res;
+
+ mutex_init(&pfdev->sched_lock);
+ mutex_init(&pfdev->reset_lock);
+ INIT_LIST_HEAD(&pfdev->scheduled_jobs);
+
+ spin_lock_init(&pfdev->hwaccess_lock);
+
+ err = panfrost_clk_init(pfdev);
+ if (err) {
+ dev_err(pfdev->dev, "clk init failed %d\n", err);
+ return err;
+ }
+
+ err = panfrost_regulator_init(pfdev);
+ if (err) {
+ dev_err(pfdev->dev, "regulator init failed %d\n", err);
+ goto err_out0;
+ }
+
+ err = panfrost_reset_init(pfdev);
+ if (err) {
+ dev_err(pfdev->dev, "reset init failed %d\n", err);
+ goto err_out1;
+ }
+
+ res = platform_get_resource(pfdev->pdev, IORESOURCE_MEM, 0);
+ pfdev->iomem = devm_ioremap_resource(pfdev->dev, res);
+ if (IS_ERR(pfdev->iomem)) {
+ dev_err(pfdev->dev, "failed to ioremap iomem\n");
+ err = PTR_ERR(pfdev->iomem);
+ goto err_out2;
+ }
+
+ err = panfrost_gpu_init(pfdev);
+ if (err)
+ goto err_out2;
+
+ err = panfrost_mmu_init(pfdev);
+ if (err)
+ goto err_out3;
+
+ err = panfrost_job_init(pfdev);
+ if (err)
+ goto err_out4;
+
+ /* runtime PM will wake us up later */
+ panfrost_gpu_power_off(pfdev);
+
+ pm_runtime_set_active(pfdev->dev);
+ pm_runtime_get_sync(pfdev->dev);
+ pm_runtime_mark_last_busy(pfdev->dev);
+ pm_runtime_put_autosuspend(pfdev->dev);
+
+ return 0;
+err_out4:
+ panfrost_mmu_fini(pfdev);
+err_out3:
+ panfrost_gpu_fini(pfdev);
+err_out2:
+ panfrost_reset_fini(pfdev);
+err_out1:
+ panfrost_regulator_fini(pfdev);
+err_out0:
+ panfrost_clk_fini(pfdev);
+ return err;
+}
+
+void panfrost_device_fini(struct panfrost_device *pfdev)
+{
+ panfrost_regulator_fini(pfdev);
+ panfrost_clk_fini(pfdev);
+}
+
+const char *panfrost_exception_name(struct panfrost_device *pfdev, u32 exception_code)
+{
+ switch (exception_code) {
+ /* Non-Fault Status code */
+ case 0x00: return "NOT_STARTED/IDLE/OK";
+ case 0x01: return "DONE";
+ case 0x02: return "INTERRUPTED";
+ case 0x03: return "STOPPED";
+ case 0x04: return "TERMINATED";
+ case 0x08: return "ACTIVE";
+ /* Job exceptions */
+ case 0x40: return "JOB_CONFIG_FAULT";
+ case 0x41: return "JOB_POWER_FAULT";
+ case 0x42: return "JOB_READ_FAULT";
+ case 0x43: return "JOB_WRITE_FAULT";
+ case 0x44: return "JOB_AFFINITY_FAULT";
+ case 0x48: return "JOB_BUS_FAULT";
+ case 0x50: return "INSTR_INVALID_PC";
+ case 0x51: return "INSTR_INVALID_ENC";
+ case 0x52: return "INSTR_TYPE_MISMATCH";
+ case 0x53: return "INSTR_OPERAND_FAULT";
+ case 0x54: return "INSTR_TLS_FAULT";
+ case 0x55: return "INSTR_BARRIER_FAULT";
+ case 0x56: return "INSTR_ALIGN_FAULT";
+ case 0x58: return "DATA_INVALID_FAULT";
+ case 0x59: return "TILE_RANGE_FAULT";
+ case 0x5A: return "ADDR_RANGE_FAULT";
+ case 0x60: return "OUT_OF_MEMORY";
+ /* GPU exceptions */
+ case 0x80: return "DELAYED_BUS_FAULT";
+ case 0x88: return "SHAREABILITY_FAULT";
+ /* MMU exceptions */
+ case 0xC1: return "TRANSLATION_FAULT_LEVEL1";
+ case 0xC2: return "TRANSLATION_FAULT_LEVEL2";
+ case 0xC3: return "TRANSLATION_FAULT_LEVEL3";
+ case 0xC4: return "TRANSLATION_FAULT_LEVEL4";
+ case 0xC8: return "PERMISSION_FAULT";
+ case 0xC9 ... 0xCF: return "PERMISSION_FAULT";
+ case 0xD1: return "TRANSTAB_BUS_FAULT_LEVEL1";
+ case 0xD2: return "TRANSTAB_BUS_FAULT_LEVEL2";
+ case 0xD3: return "TRANSTAB_BUS_FAULT_LEVEL3";
+ case 0xD4: return "TRANSTAB_BUS_FAULT_LEVEL4";
+ case 0xD8: return "ACCESS_FLAG";
+ case 0xD9 ... 0xDF: return "ACCESS_FLAG";
+ case 0xE0 ... 0xE7: return "ADDRESS_SIZE_FAULT";
+ case 0xE8 ... 0xEF: return "MEMORY_ATTRIBUTES_FAULT";
+ }
+
+ return "UNKNOWN";
+}
+
+#ifdef CONFIG_PM
+int panfrost_device_resume(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct panfrost_device *pfdev = platform_get_drvdata(pdev);
+
+ panfrost_gpu_soft_reset(pfdev);
+
+ /* TODO: Re-enable all other address spaces */
+ panfrost_gpu_power_on(pfdev);
+ panfrost_mmu_enable(pfdev, 0);
+ panfrost_job_enable_interrupts(pfdev);
+ panfrost_devfreq_resume(pfdev);
+
+ return 0;
+}
+
+int panfrost_device_suspend(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct panfrost_device *pfdev = platform_get_drvdata(pdev);
+
+ if (!panfrost_job_is_idle(pfdev))
+ return -EBUSY;
+
+ panfrost_devfreq_suspend(pfdev);
+ panfrost_gpu_power_off(pfdev);
+
+ return 0;
+}
+#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
+/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
+
+#ifndef __PANFROST_DEVICE_H__
+#define __PANFROST_DEVICE_H__
+
+#include <linux/spinlock.h>
+#include <drm/drm_device.h>
+#include <drm/drm_mm.h>
+#include <drm/gpu_scheduler.h>
+
+struct panfrost_device;
+struct panfrost_mmu;
+struct panfrost_job_slot;
+struct panfrost_job;
+
+#define NUM_JOB_SLOTS 3
+
+struct panfrost_features {
+ u16 id;
+ u16 revision;
+
+ u64 shader_present;
+ u64 tiler_present;
+ u64 l2_present;
+ u64 stack_present;
+ u32 as_present;
+ u32 js_present;
+
+ u32 l2_features;
+ u32 core_features;
+ u32 tiler_features;
+ u32 mem_features;
+ u32 mmu_features;
+ u32 thread_features;
+ u32 max_threads;
+ u32 thread_max_workgroup_sz;
+ u32 thread_max_barrier_sz;
+ u32 coherency_features;
+ u32 texture_features[4];
+ u32 js_features[16];
+
+ u32 nr_core_groups;
+
+ unsigned long hw_features[64 / BITS_PER_LONG];
+ unsigned long hw_issues[64 / BITS_PER_LONG];
+};
+
+struct panfrost_devfreq_slot {
+ ktime_t busy_time;
+ ktime_t idle_time;
+ ktime_t time_last_update;
+ bool busy;
+};
+
+struct panfrost_device {
+ struct device *dev;
+ struct drm_device *ddev;
+ struct platform_device *pdev;
+
+ spinlock_t hwaccess_lock;
+
+ struct drm_mm mm;
+ spinlock_t mm_lock;
+
+ void __iomem *iomem;
+ struct clk *clock;
+ struct regulator *regulator;
+ struct reset_control *rstc;
+
+ struct panfrost_features features;
+
+ struct panfrost_mmu *mmu;
+ struct panfrost_job_slot *js;
+
+ struct panfrost_job *jobs[NUM_JOB_SLOTS];
+ struct list_head scheduled_jobs;
+
+ struct mutex sched_lock;
+ struct mutex reset_lock;
+
+ struct {
+ struct devfreq *devfreq;
+ struct thermal_cooling_device *cooling;
+ unsigned long cur_freq;
+ unsigned long cur_volt;
+ struct panfrost_devfreq_slot slot[NUM_JOB_SLOTS];
+ } devfreq;
+};
+
+struct panfrost_file_priv {
+ struct panfrost_device *pfdev;
+
+ struct drm_sched_entity sched_entity[NUM_JOB_SLOTS];
+};
+
+static inline struct panfrost_device *to_panfrost_device(struct drm_device *ddev)
+{
+ return ddev->dev_private;
+}
+
+static inline int panfrost_model_cmp(struct panfrost_device *pfdev, s32 id)
+{
+ s32 match_id = pfdev->features.id;
+
+ if (match_id & 0xf000)
+ match_id &= 0xf00f;
+ return match_id - id;
+}
+
+static inline bool panfrost_model_eq(struct panfrost_device *pfdev, s32 id)
+{
+ return !panfrost_model_cmp(pfdev, id);
+}
+
+int panfrost_device_init(struct panfrost_device *pfdev);
+void panfrost_device_fini(struct panfrost_device *pfdev);
+
+int panfrost_device_resume(struct device *dev);
+int panfrost_device_suspend(struct device *dev);
+
+const char *panfrost_exception_name(struct panfrost_device *pfdev, u32 exception_code);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
+/* Copyright 2019 Linaro, Ltd., Rob Herring <robh@kernel.org> */
+/* Copyright 2019 Collabora ltd. */
+
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/pagemap.h>
+#include <linux/pm_runtime.h>
+#include <drm/panfrost_drm.h>
+#include <drm/drm_drv.h>
+#include <drm/drm_ioctl.h>
+#include <drm/drm_syncobj.h>
+#include <drm/drm_utils.h>
+
+#include "panfrost_device.h"
+#include "panfrost_devfreq.h"
+#include "panfrost_gem.h"
+#include "panfrost_mmu.h"
+#include "panfrost_job.h"
+#include "panfrost_gpu.h"
+
+static int panfrost_ioctl_get_param(struct drm_device *ddev, void *data, struct drm_file *file)
+{
+ struct drm_panfrost_get_param *param = data;
+ struct panfrost_device *pfdev = ddev->dev_private;
+
+ if (param->pad != 0)
+ return -EINVAL;
+
+ switch (param->param) {
+ case DRM_PANFROST_PARAM_GPU_PROD_ID:
+ param->value = pfdev->features.id;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int panfrost_ioctl_create_bo(struct drm_device *dev, void *data,
+ struct drm_file *file)
+{
+ int ret;
+ struct drm_gem_shmem_object *shmem;
+ struct drm_panfrost_create_bo *args = data;
+
+ if (!args->size || args->flags || args->pad)
+ return -EINVAL;
+
+ shmem = drm_gem_shmem_create_with_handle(file, dev, args->size,
+ &args->handle);
+ if (IS_ERR(shmem))
+ return PTR_ERR(shmem);
+
+ ret = panfrost_mmu_map(to_panfrost_bo(&shmem->base));
+ if (ret)
+ goto err_free;
+
+ args->offset = to_panfrost_bo(&shmem->base)->node.start << PAGE_SHIFT;
+
+ return 0;
+
+err_free:
+ drm_gem_object_put_unlocked(&shmem->base);
+ return ret;
+}
+
+/**
+ * panfrost_lookup_bos() - Sets up job->bo[] with the GEM objects
+ * referenced by the job.
+ * @dev: DRM device
+ * @file_priv: DRM file for this fd
+ * @args: IOCTL args
+ * @job: job being set up
+ *
+ * Resolve handles from userspace to BOs and attach them to job.
+ *
+ * Note that this function doesn't need to unreference the BOs on
+ * failure, because that will happen at panfrost_job_cleanup() time.
+ */
+static int
+panfrost_lookup_bos(struct drm_device *dev,
+ struct drm_file *file_priv,
+ struct drm_panfrost_submit *args,
+ struct panfrost_job *job)
+{
+ job->bo_count = args->bo_handle_count;
+
+ if (!job->bo_count)
+ return 0;
+
+ job->implicit_fences = kvmalloc_array(job->bo_count,
+ sizeof(struct dma_fence *),
+ GFP_KERNEL | __GFP_ZERO);
+ if (!job->implicit_fences)
+ return -ENOMEM;
+
+ return drm_gem_objects_lookup(file_priv,
+ (void __user *)(uintptr_t)args->bo_handles,
+ job->bo_count, &job->bos);
+}
+
+/**
+ * panfrost_copy_in_sync() - Sets up job->in_fences[] with the sync objects
+ * referenced by the job.
+ * @dev: DRM device
+ * @file_priv: DRM file for this fd
+ * @args: IOCTL args
+ * @job: job being set up
+ *
+ * Resolve syncobjs from userspace to fences and attach them to job.
+ *
+ * Note that this function doesn't need to unreference the fences on
+ * failure, because that will happen at panfrost_job_cleanup() time.
+ */
+static int
+panfrost_copy_in_sync(struct drm_device *dev,
+ struct drm_file *file_priv,
+ struct drm_panfrost_submit *args,
+ struct panfrost_job *job)
+{
+ u32 *handles;
+ int ret = 0;
+ int i;
+
+ job->in_fence_count = args->in_sync_count;
+
+ if (!job->in_fence_count)
+ return 0;
+
+ job->in_fences = kvmalloc_array(job->in_fence_count,
+ sizeof(struct dma_fence *),
+ GFP_KERNEL | __GFP_ZERO);
+ if (!job->in_fences) {
+ DRM_DEBUG("Failed to allocate job in fences\n");
+ return -ENOMEM;
+ }
+
+ handles = kvmalloc_array(job->in_fence_count, sizeof(u32), GFP_KERNEL);
+ if (!handles) {
+ ret = -ENOMEM;
+ DRM_DEBUG("Failed to allocate incoming syncobj handles\n");
+ goto fail;
+ }
+
+ if (copy_from_user(handles,
+ (void __user *)(uintptr_t)args->in_syncs,
+ job->in_fence_count * sizeof(u32))) {
+ ret = -EFAULT;
+ DRM_DEBUG("Failed to copy in syncobj handles\n");
+ goto fail;
+ }
+
+ for (i = 0; i < job->in_fence_count; i++) {
+ ret = drm_syncobj_find_fence(file_priv, handles[i], 0, 0,
+ &job->in_fences[i]);
+ if (ret == -EINVAL)
+ goto fail;
+ }
+
+fail:
+ kvfree(handles);
+ return ret;
+}
+
+static int panfrost_ioctl_submit(struct drm_device *dev, void *data,
+ struct drm_file *file)
+{
+ struct panfrost_device *pfdev = dev->dev_private;
+ struct drm_panfrost_submit *args = data;
+ struct drm_syncobj *sync_out = NULL;
+ struct panfrost_job *job;
+ int ret = 0;
+
+ if (!args->jc)
+ return -EINVAL;
+
+ if (args->requirements && args->requirements != PANFROST_JD_REQ_FS)
+ return -EINVAL;
+
+ if (args->out_sync > 0) {
+ sync_out = drm_syncobj_find(file, args->out_sync);
+ if (!sync_out)
+ return -ENODEV;
+ }
+
+ job = kzalloc(sizeof(*job), GFP_KERNEL);
+ if (!job) {
+ ret = -ENOMEM;
+ goto fail_out_sync;
+ }
+
+ kref_init(&job->refcount);
+
+ job->pfdev = pfdev;
+ job->jc = args->jc;
+ job->requirements = args->requirements;
+ job->flush_id = panfrost_gpu_get_latest_flush_id(pfdev);
+ job->file_priv = file->driver_priv;
+
+ ret = panfrost_copy_in_sync(dev, file, args, job);
+ if (ret)
+ goto fail_job;
+
+ ret = panfrost_lookup_bos(dev, file, args, job);
+ if (ret)
+ goto fail_job;
+
+ ret = panfrost_job_push(job);
+ if (ret)
+ goto fail_job;
+
+ /* Update the return sync object for the job */
+ if (sync_out)
+ drm_syncobj_replace_fence(sync_out, job->render_done_fence);
+
+fail_job:
+ panfrost_job_put(job);
+fail_out_sync:
+ drm_syncobj_put(sync_out);
+
+ return ret;
+}
+
+static int
+panfrost_ioctl_wait_bo(struct drm_device *dev, void *data,
+ struct drm_file *file_priv)
+{
+ long ret;
+ struct drm_panfrost_wait_bo *args = data;
+ struct drm_gem_object *gem_obj;
+ unsigned long timeout = drm_timeout_abs_to_jiffies(args->timeout_ns);
+
+ if (args->pad)
+ return -EINVAL;
+
+ gem_obj = drm_gem_object_lookup(file_priv, args->handle);
+ if (!gem_obj)
+ return -ENOENT;
+
+ ret = reservation_object_wait_timeout_rcu(gem_obj->resv, true,
+ true, timeout);
+ if (!ret)
+ ret = timeout ? -ETIMEDOUT : -EBUSY;
+
+ drm_gem_object_put_unlocked(gem_obj);
+
+ return ret;
+}
+
+static int panfrost_ioctl_mmap_bo(struct drm_device *dev, void *data,
+ struct drm_file *file_priv)
+{
+ struct drm_panfrost_mmap_bo *args = data;
+ struct drm_gem_object *gem_obj;
+ int ret;
+
+ if (args->flags != 0) {
+ DRM_INFO("unknown mmap_bo flags: %d\n", args->flags);
+ return -EINVAL;
+ }
+
+ gem_obj = drm_gem_object_lookup(file_priv, args->handle);
+ if (!gem_obj) {
+ DRM_DEBUG("Failed to look up GEM BO %d\n", args->handle);
+ return -ENOENT;
+ }
+
+ ret = drm_gem_create_mmap_offset(gem_obj);
+ if (ret == 0)
+ args->offset = drm_vma_node_offset_addr(&gem_obj->vma_node);
+ drm_gem_object_put_unlocked(gem_obj);
+
+ return ret;
+}
+
+static int panfrost_ioctl_get_bo_offset(struct drm_device *dev, void *data,
+ struct drm_file *file_priv)
+{
+ struct drm_panfrost_get_bo_offset *args = data;
+ struct drm_gem_object *gem_obj;
+ struct panfrost_gem_object *bo;
+
+ gem_obj = drm_gem_object_lookup(file_priv, args->handle);
+ if (!gem_obj) {
+ DRM_DEBUG("Failed to look up GEM BO %d\n", args->handle);
+ return -ENOENT;
+ }
+ bo = to_panfrost_bo(gem_obj);
+
+ args->offset = bo->node.start << PAGE_SHIFT;
+
+ drm_gem_object_put_unlocked(gem_obj);
+ return 0;
+}
+
+static int
+panfrost_open(struct drm_device *dev, struct drm_file *file)
+{
+ struct panfrost_device *pfdev = dev->dev_private;
+ struct panfrost_file_priv *panfrost_priv;
+
+ panfrost_priv = kzalloc(sizeof(*panfrost_priv), GFP_KERNEL);
+ if (!panfrost_priv)
+ return -ENOMEM;
+
+ panfrost_priv->pfdev = pfdev;
+ file->driver_priv = panfrost_priv;
+
+ return panfrost_job_open(panfrost_priv);
+}
+
+static void
+panfrost_postclose(struct drm_device *dev, struct drm_file *file)
+{
+ struct panfrost_file_priv *panfrost_priv = file->driver_priv;
+
+ panfrost_job_close(panfrost_priv);
+
+ kfree(panfrost_priv);
+}
+
+/* DRM_AUTH is required on SUBMIT for now, while all clients share a single
+ * address space. Note that render nodes would be able to submit jobs that
+ * could access BOs from clients authenticated with the master node.
+ */
+static const struct drm_ioctl_desc panfrost_drm_driver_ioctls[] = {
+#define PANFROST_IOCTL(n, func, flags) \
+ DRM_IOCTL_DEF_DRV(PANFROST_##n, panfrost_ioctl_##func, flags)
+
+ PANFROST_IOCTL(SUBMIT, submit, DRM_RENDER_ALLOW | DRM_AUTH),
+ PANFROST_IOCTL(WAIT_BO, wait_bo, DRM_RENDER_ALLOW),
+ PANFROST_IOCTL(CREATE_BO, create_bo, DRM_RENDER_ALLOW),
+ PANFROST_IOCTL(MMAP_BO, mmap_bo, DRM_RENDER_ALLOW),
+ PANFROST_IOCTL(GET_PARAM, get_param, DRM_RENDER_ALLOW),
+ PANFROST_IOCTL(GET_BO_OFFSET, get_bo_offset, DRM_RENDER_ALLOW),
+};
+
+DEFINE_DRM_GEM_SHMEM_FOPS(panfrost_drm_driver_fops);
+
+static struct drm_driver panfrost_drm_driver = {
+ .driver_features = DRIVER_RENDER | DRIVER_GEM | DRIVER_PRIME |
+ DRIVER_SYNCOBJ,
+ .open = panfrost_open,
+ .postclose = panfrost_postclose,
+ .ioctls = panfrost_drm_driver_ioctls,
+ .num_ioctls = ARRAY_SIZE(panfrost_drm_driver_ioctls),
+ .fops = &panfrost_drm_driver_fops,
+ .name = "panfrost",
+ .desc = "panfrost DRM",
+ .date = "20180908",
+ .major = 1,
+ .minor = 0,
+
+ .gem_create_object = panfrost_gem_create_object,
+ .prime_handle_to_fd = drm_gem_prime_handle_to_fd,
+ .prime_fd_to_handle = drm_gem_prime_fd_to_handle,
+ .gem_prime_import_sg_table = panfrost_gem_prime_import_sg_table,
+ .gem_prime_mmap = drm_gem_prime_mmap,
+};
+
+static int panfrost_probe(struct platform_device *pdev)
+{
+ struct panfrost_device *pfdev;
+ struct drm_device *ddev;
+ int err;
+
+ pfdev = devm_kzalloc(&pdev->dev, sizeof(*pfdev), GFP_KERNEL);
+ if (!pfdev)
+ return -ENOMEM;
+
+ pfdev->pdev = pdev;
+ pfdev->dev = &pdev->dev;
+
+ platform_set_drvdata(pdev, pfdev);
+
+ /* Allocate and initialze the DRM device. */
+ ddev = drm_dev_alloc(&panfrost_drm_driver, &pdev->dev);
+ if (IS_ERR(ddev))
+ return PTR_ERR(ddev);
+
+ ddev->dev_private = pfdev;
+ pfdev->ddev = ddev;
+
+ spin_lock_init(&pfdev->mm_lock);
+
+ /* 4G enough for now. can be 48-bit */
+ drm_mm_init(&pfdev->mm, SZ_32M >> PAGE_SHIFT, (SZ_4G - SZ_32M) >> PAGE_SHIFT);
+
+ pm_runtime_use_autosuspend(pfdev->dev);
+ pm_runtime_set_autosuspend_delay(pfdev->dev, 50); /* ~3 frames */
+ pm_runtime_enable(pfdev->dev);
+
+ err = panfrost_device_init(pfdev);
+ if (err) {
+ if (err != -EPROBE_DEFER)
+ dev_err(&pdev->dev, "Fatal error during GPU init\n");
+ goto err_out0;
+ }
+
+ err = panfrost_devfreq_init(pfdev);
+ if (err) {
+ if (err != -EPROBE_DEFER)
+ dev_err(&pdev->dev, "Fatal error during devfreq init\n");
+ goto err_out1;
+ }
+
+ /*
+ * Register the DRM device with the core and the connectors with
+ * sysfs
+ */
+ err = drm_dev_register(ddev, 0);
+ if (err < 0)
+ goto err_out1;
+
+ return 0;
+
+err_out1:
+ panfrost_device_fini(pfdev);
+err_out0:
+ pm_runtime_disable(pfdev->dev);
+ drm_dev_put(ddev);
+ return err;
+}
+
+static int panfrost_remove(struct platform_device *pdev)
+{
+ struct panfrost_device *pfdev = platform_get_drvdata(pdev);
+ struct drm_device *ddev = pfdev->ddev;
+
+ drm_dev_unregister(ddev);
+ pm_runtime_get_sync(pfdev->dev);
+ pm_runtime_put_sync_autosuspend(pfdev->dev);
+ pm_runtime_disable(pfdev->dev);
+ panfrost_device_fini(pfdev);
+ drm_dev_put(ddev);
+ return 0;
+}
+
+static const struct of_device_id dt_match[] = {
+ { .compatible = "arm,mali-t604" },
+ { .compatible = "arm,mali-t624" },
+ { .compatible = "arm,mali-t628" },
+ { .compatible = "arm,mali-t720" },
+ { .compatible = "arm,mali-t760" },
+ { .compatible = "arm,mali-t820" },
+ { .compatible = "arm,mali-t830" },
+ { .compatible = "arm,mali-t860" },
+ { .compatible = "arm,mali-t880" },
+ {}
+};
+MODULE_DEVICE_TABLE(of, dt_match);
+
+static const struct dev_pm_ops panfrost_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend, pm_runtime_force_resume)
+ SET_RUNTIME_PM_OPS(panfrost_device_suspend, panfrost_device_resume, NULL)
+};
+
+static struct platform_driver panfrost_driver = {
+ .probe = panfrost_probe,
+ .remove = panfrost_remove,
+ .driver = {
+ .name = "panfrost",
+ .pm = &panfrost_pm_ops,
+ .of_match_table = dt_match,
+ },
+};
+module_platform_driver(panfrost_driver);
+
+MODULE_AUTHOR("Panfrost Project Developers");
+MODULE_DESCRIPTION("Panfrost DRM Driver");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* (C) COPYRIGHT 2014-2018 ARM Limited. All rights reserved. */
+/* Copyright 2019 Linaro, Ltd., Rob Herring <robh@kernel.org> */
+#ifndef __PANFROST_FEATURES_H__
+#define __PANFROST_FEATURES_H__
+
+#include <linux/bitops.h>
+
+#include "panfrost_device.h"
+
+enum panfrost_hw_feature {
+ HW_FEATURE_JOBCHAIN_DISAMBIGUATION,
+ HW_FEATURE_PWRON_DURING_PWROFF_TRANS,
+ HW_FEATURE_XAFFINITY,
+ HW_FEATURE_OUT_OF_ORDER_EXEC,
+ HW_FEATURE_MRT,
+ HW_FEATURE_BRNDOUT_CC,
+ HW_FEATURE_INTERPIPE_REG_ALIASING,
+ HW_FEATURE_LD_ST_TILEBUFFER,
+ HW_FEATURE_MSAA_16X,
+ HW_FEATURE_32_BIT_UNIFORM_ADDRESS,
+ HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL,
+ HW_FEATURE_OPTIMIZED_COVERAGE_MASK,
+ HW_FEATURE_T7XX_PAIRING_RULES,
+ HW_FEATURE_LD_ST_LEA_TEX,
+ HW_FEATURE_LINEAR_FILTER_FLOAT,
+ HW_FEATURE_WORKGROUP_ROUND_MULTIPLE_OF_4,
+ HW_FEATURE_IMAGES_IN_FRAGMENT_SHADERS,
+ HW_FEATURE_TEST4_DATUM_MODE,
+ HW_FEATURE_NEXT_INSTRUCTION_TYPE,
+ HW_FEATURE_BRNDOUT_KILL,
+ HW_FEATURE_WARPING,
+ HW_FEATURE_V4,
+ HW_FEATURE_FLUSH_REDUCTION,
+ HW_FEATURE_PROTECTED_MODE,
+ HW_FEATURE_COHERENCY_REG,
+ HW_FEATURE_PROTECTED_DEBUG_MODE,
+ HW_FEATURE_AARCH64_MMU,
+ HW_FEATURE_TLS_HASHING,
+ HW_FEATURE_THREAD_GROUP_SPLIT,
+ HW_FEATURE_3BIT_EXT_RW_L2_MMU_CONFIG,
+};
+
+#define hw_features_t600 (\
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT) | \
+ BIT_ULL(HW_FEATURE_V4))
+
+#define hw_features_t620 (\
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT) | \
+ BIT_ULL(HW_FEATURE_V4))
+
+#define hw_features_t720 (\
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_INTERPIPE_REG_ALIASING) | \
+ BIT_ULL(HW_FEATURE_OPTIMIZED_COVERAGE_MASK) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT) | \
+ BIT_ULL(HW_FEATURE_WORKGROUP_ROUND_MULTIPLE_OF_4) | \
+ BIT_ULL(HW_FEATURE_WARPING) | \
+ BIT_ULL(HW_FEATURE_V4))
+
+
+#define hw_features_t760 (\
+ BIT_ULL(HW_FEATURE_JOBCHAIN_DISAMBIGUATION) | \
+ BIT_ULL(HW_FEATURE_PWRON_DURING_PWROFF_TRANS) | \
+ BIT_ULL(HW_FEATURE_XAFFINITY) | \
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_CC) | \
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LD_ST_TILEBUFFER) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_MRT) | \
+ BIT_ULL(HW_FEATURE_MSAA_16X) | \
+ BIT_ULL(HW_FEATURE_OUT_OF_ORDER_EXEC) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_TEST4_DATUM_MODE) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT))
+
+// T860
+#define hw_features_t860 (\
+ BIT_ULL(HW_FEATURE_JOBCHAIN_DISAMBIGUATION) | \
+ BIT_ULL(HW_FEATURE_PWRON_DURING_PWROFF_TRANS) | \
+ BIT_ULL(HW_FEATURE_XAFFINITY) | \
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_CC) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_KILL) | \
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LD_ST_TILEBUFFER) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_MRT) | \
+ BIT_ULL(HW_FEATURE_MSAA_16X) | \
+ BIT_ULL(HW_FEATURE_NEXT_INSTRUCTION_TYPE) | \
+ BIT_ULL(HW_FEATURE_OUT_OF_ORDER_EXEC) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_TEST4_DATUM_MODE) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT))
+
+#define hw_features_t880 hw_features_t860
+
+#define hw_features_t830 (\
+ BIT_ULL(HW_FEATURE_JOBCHAIN_DISAMBIGUATION) | \
+ BIT_ULL(HW_FEATURE_PWRON_DURING_PWROFF_TRANS) | \
+ BIT_ULL(HW_FEATURE_XAFFINITY) | \
+ BIT_ULL(HW_FEATURE_WARPING) | \
+ BIT_ULL(HW_FEATURE_INTERPIPE_REG_ALIASING) | \
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_CC) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_KILL) | \
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LD_ST_TILEBUFFER) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_MRT) | \
+ BIT_ULL(HW_FEATURE_NEXT_INSTRUCTION_TYPE) | \
+ BIT_ULL(HW_FEATURE_OUT_OF_ORDER_EXEC) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_TEST4_DATUM_MODE) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT))
+
+#define hw_features_t820 (\
+ BIT_ULL(HW_FEATURE_JOBCHAIN_DISAMBIGUATION) | \
+ BIT_ULL(HW_FEATURE_PWRON_DURING_PWROFF_TRANS) | \
+ BIT_ULL(HW_FEATURE_XAFFINITY) | \
+ BIT_ULL(HW_FEATURE_WARPING) | \
+ BIT_ULL(HW_FEATURE_INTERPIPE_REG_ALIASING) | \
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_CC) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_KILL) | \
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LD_ST_TILEBUFFER) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_MRT) | \
+ BIT_ULL(HW_FEATURE_NEXT_INSTRUCTION_TYPE) | \
+ BIT_ULL(HW_FEATURE_OUT_OF_ORDER_EXEC) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_TEST4_DATUM_MODE) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT))
+
+#define hw_features_g71 (\
+ BIT_ULL(HW_FEATURE_JOBCHAIN_DISAMBIGUATION) | \
+ BIT_ULL(HW_FEATURE_PWRON_DURING_PWROFF_TRANS) | \
+ BIT_ULL(HW_FEATURE_XAFFINITY) | \
+ BIT_ULL(HW_FEATURE_WARPING) | \
+ BIT_ULL(HW_FEATURE_INTERPIPE_REG_ALIASING) | \
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_CC) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_KILL) | \
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LD_ST_TILEBUFFER) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_MRT) | \
+ BIT_ULL(HW_FEATURE_MSAA_16X) | \
+ BIT_ULL(HW_FEATURE_NEXT_INSTRUCTION_TYPE) | \
+ BIT_ULL(HW_FEATURE_OUT_OF_ORDER_EXEC) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_TEST4_DATUM_MODE) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT) | \
+ BIT_ULL(HW_FEATURE_FLUSH_REDUCTION) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_MODE) | \
+ BIT_ULL(HW_FEATURE_COHERENCY_REG))
+
+#define hw_features_g72 (\
+ BIT_ULL(HW_FEATURE_JOBCHAIN_DISAMBIGUATION) | \
+ BIT_ULL(HW_FEATURE_PWRON_DURING_PWROFF_TRANS) | \
+ BIT_ULL(HW_FEATURE_XAFFINITY) | \
+ BIT_ULL(HW_FEATURE_WARPING) | \
+ BIT_ULL(HW_FEATURE_INTERPIPE_REG_ALIASING) | \
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_CC) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_KILL) | \
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LD_ST_TILEBUFFER) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_MRT) | \
+ BIT_ULL(HW_FEATURE_MSAA_16X) | \
+ BIT_ULL(HW_FEATURE_NEXT_INSTRUCTION_TYPE) | \
+ BIT_ULL(HW_FEATURE_OUT_OF_ORDER_EXEC) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_TEST4_DATUM_MODE) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT) | \
+ BIT_ULL(HW_FEATURE_FLUSH_REDUCTION) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_MODE) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_DEBUG_MODE) | \
+ BIT_ULL(HW_FEATURE_COHERENCY_REG))
+
+#define hw_features_g51 (\
+ BIT_ULL(HW_FEATURE_JOBCHAIN_DISAMBIGUATION) | \
+ BIT_ULL(HW_FEATURE_PWRON_DURING_PWROFF_TRANS) | \
+ BIT_ULL(HW_FEATURE_XAFFINITY) | \
+ BIT_ULL(HW_FEATURE_WARPING) | \
+ BIT_ULL(HW_FEATURE_INTERPIPE_REG_ALIASING) | \
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_CC) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_KILL) | \
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LD_ST_TILEBUFFER) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_MRT) | \
+ BIT_ULL(HW_FEATURE_MSAA_16X) | \
+ BIT_ULL(HW_FEATURE_NEXT_INSTRUCTION_TYPE) | \
+ BIT_ULL(HW_FEATURE_OUT_OF_ORDER_EXEC) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_TEST4_DATUM_MODE) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT) | \
+ BIT_ULL(HW_FEATURE_FLUSH_REDUCTION) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_MODE) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_DEBUG_MODE) | \
+ BIT_ULL(HW_FEATURE_COHERENCY_REG))
+
+#define hw_features_g52 (\
+ BIT_ULL(HW_FEATURE_JOBCHAIN_DISAMBIGUATION) | \
+ BIT_ULL(HW_FEATURE_PWRON_DURING_PWROFF_TRANS) | \
+ BIT_ULL(HW_FEATURE_XAFFINITY) | \
+ BIT_ULL(HW_FEATURE_WARPING) | \
+ BIT_ULL(HW_FEATURE_INTERPIPE_REG_ALIASING) | \
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_CC) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_KILL) | \
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LD_ST_TILEBUFFER) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_MRT) | \
+ BIT_ULL(HW_FEATURE_MSAA_16X) | \
+ BIT_ULL(HW_FEATURE_NEXT_INSTRUCTION_TYPE) | \
+ BIT_ULL(HW_FEATURE_OUT_OF_ORDER_EXEC) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_TEST4_DATUM_MODE) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT) | \
+ BIT_ULL(HW_FEATURE_FLUSH_REDUCTION) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_MODE) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_DEBUG_MODE) | \
+ BIT_ULL(HW_FEATURE_COHERENCY_REG))
+
+#define hw_features_g76 (\
+ BIT_ULL(HW_FEATURE_JOBCHAIN_DISAMBIGUATION) | \
+ BIT_ULL(HW_FEATURE_PWRON_DURING_PWROFF_TRANS) | \
+ BIT_ULL(HW_FEATURE_XAFFINITY) | \
+ BIT_ULL(HW_FEATURE_WARPING) | \
+ BIT_ULL(HW_FEATURE_INTERPIPE_REG_ALIASING) | \
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_CC) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_KILL) | \
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LD_ST_TILEBUFFER) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_MRT) | \
+ BIT_ULL(HW_FEATURE_MSAA_16X) | \
+ BIT_ULL(HW_FEATURE_NEXT_INSTRUCTION_TYPE) | \
+ BIT_ULL(HW_FEATURE_OUT_OF_ORDER_EXEC) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_TEST4_DATUM_MODE) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT) | \
+ BIT_ULL(HW_FEATURE_FLUSH_REDUCTION) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_MODE) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_DEBUG_MODE) | \
+ BIT_ULL(HW_FEATURE_COHERENCY_REG) | \
+ BIT_ULL(HW_FEATURE_AARCH64_MMU) | \
+ BIT_ULL(HW_FEATURE_TLS_HASHING) | \
+ BIT_ULL(HW_FEATURE_3BIT_EXT_RW_L2_MMU_CONFIG))
+
+#define hw_features_g31 (\
+ BIT_ULL(HW_FEATURE_JOBCHAIN_DISAMBIGUATION) | \
+ BIT_ULL(HW_FEATURE_PWRON_DURING_PWROFF_TRANS) | \
+ BIT_ULL(HW_FEATURE_XAFFINITY) | \
+ BIT_ULL(HW_FEATURE_WARPING) | \
+ BIT_ULL(HW_FEATURE_INTERPIPE_REG_ALIASING) | \
+ BIT_ULL(HW_FEATURE_32_BIT_UNIFORM_ADDRESS) | \
+ BIT_ULL(HW_FEATURE_ATTR_AUTO_TYPE_INFERRAL) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_CC) | \
+ BIT_ULL(HW_FEATURE_BRNDOUT_KILL) | \
+ BIT_ULL(HW_FEATURE_LD_ST_LEA_TEX) | \
+ BIT_ULL(HW_FEATURE_LD_ST_TILEBUFFER) | \
+ BIT_ULL(HW_FEATURE_LINEAR_FILTER_FLOAT) | \
+ BIT_ULL(HW_FEATURE_MRT) | \
+ BIT_ULL(HW_FEATURE_MSAA_16X) | \
+ BIT_ULL(HW_FEATURE_NEXT_INSTRUCTION_TYPE) | \
+ BIT_ULL(HW_FEATURE_OUT_OF_ORDER_EXEC) | \
+ BIT_ULL(HW_FEATURE_T7XX_PAIRING_RULES) | \
+ BIT_ULL(HW_FEATURE_TEST4_DATUM_MODE) | \
+ BIT_ULL(HW_FEATURE_THREAD_GROUP_SPLIT) | \
+ BIT_ULL(HW_FEATURE_FLUSH_REDUCTION) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_MODE) | \
+ BIT_ULL(HW_FEATURE_PROTECTED_DEBUG_MODE) | \
+ BIT_ULL(HW_FEATURE_COHERENCY_REG) | \
+ BIT_ULL(HW_FEATURE_AARCH64_MMU) | \
+ BIT_ULL(HW_FEATURE_TLS_HASHING) | \
+ BIT_ULL(HW_FEATURE_3BIT_EXT_RW_L2_MMU_CONFIG))
+
+static inline bool panfrost_has_hw_feature(struct panfrost_device *pfdev,
+ enum panfrost_hw_feature feat)
+{
+ return test_bit(feat, pfdev->features.hw_features);
+}
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
+
+#include <linux/err.h>
+#include <linux/slab.h>
+#include <linux/dma-buf.h>
+#include <linux/dma-mapping.h>
+
+#include <drm/panfrost_drm.h>
+#include "panfrost_device.h"
+#include "panfrost_gem.h"
+#include "panfrost_mmu.h"
+
+/* Called DRM core on the last userspace/kernel unreference of the
+ * BO.
+ */
+static void panfrost_gem_free_object(struct drm_gem_object *obj)
+{
+ struct panfrost_gem_object *bo = to_panfrost_bo(obj);
+ struct panfrost_device *pfdev = obj->dev->dev_private;
+
+ panfrost_mmu_unmap(bo);
+
+ spin_lock(&pfdev->mm_lock);
+ drm_mm_remove_node(&bo->node);
+ spin_unlock(&pfdev->mm_lock);
+
+ drm_gem_shmem_free_object(obj);
+}
+
+static const struct drm_gem_object_funcs panfrost_gem_funcs = {
+ .free = panfrost_gem_free_object,
+ .print_info = drm_gem_shmem_print_info,
+ .pin = drm_gem_shmem_pin,
+ .unpin = drm_gem_shmem_unpin,
+ .get_sg_table = drm_gem_shmem_get_sg_table,
+ .vmap = drm_gem_shmem_vmap,
+ .vunmap = drm_gem_shmem_vunmap,
+ .vm_ops = &drm_gem_shmem_vm_ops,
+};
+
+/**
+ * panfrost_gem_create_object - Implementation of driver->gem_create_object.
+ * @dev: DRM device
+ * @size: Size in bytes of the memory the object will reference
+ *
+ * This lets the GEM helpers allocate object structs for us, and keep
+ * our BO stats correct.
+ */
+struct drm_gem_object *panfrost_gem_create_object(struct drm_device *dev, size_t size)
+{
+ int ret;
+ struct panfrost_device *pfdev = dev->dev_private;
+ struct panfrost_gem_object *obj;
+
+ obj = kzalloc(sizeof(*obj), GFP_KERNEL);
+ if (!obj)
+ return NULL;
+
+ obj->base.base.funcs = &panfrost_gem_funcs;
+
+ spin_lock(&pfdev->mm_lock);
+ ret = drm_mm_insert_node(&pfdev->mm, &obj->node,
+ roundup(size, PAGE_SIZE) >> PAGE_SHIFT);
+ spin_unlock(&pfdev->mm_lock);
+ if (ret)
+ goto free_obj;
+
+ return &obj->base.base;
+
+free_obj:
+ kfree(obj);
+ return ERR_PTR(ret);
+}
+
+struct drm_gem_object *
+panfrost_gem_prime_import_sg_table(struct drm_device *dev,
+ struct dma_buf_attachment *attach,
+ struct sg_table *sgt)
+{
+ struct drm_gem_object *obj;
+ struct panfrost_gem_object *pobj;
+
+ obj = drm_gem_shmem_prime_import_sg_table(dev, attach, sgt);
+ if (IS_ERR(obj))
+ return ERR_CAST(obj);
+
+ pobj = to_panfrost_bo(obj);
+
+ obj->resv = attach->dmabuf->resv;
+
+ panfrost_mmu_map(pobj);
+
+ return obj;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
+
+#ifndef __PANFROST_GEM_H__
+#define __PANFROST_GEM_H__
+
+#include <drm/drm_gem_shmem_helper.h>
+#include <drm/drm_mm.h>
+
+struct panfrost_gem_object {
+ struct drm_gem_shmem_object base;
+
+ struct drm_mm_node node;
+};
+
+static inline
+struct panfrost_gem_object *to_panfrost_bo(struct drm_gem_object *obj)
+{
+ return container_of(to_drm_gem_shmem_obj(obj), struct panfrost_gem_object, base);
+}
+
+struct drm_gem_object *panfrost_gem_create_object(struct drm_device *dev, size_t size);
+
+struct drm_gem_object *
+panfrost_gem_prime_import_sg_table(struct drm_device *dev,
+ struct dma_buf_attachment *attach,
+ struct sg_table *sgt);
+
+#endif /* __PANFROST_GEM_H__ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
+/* Copyright 2019 Linaro, Ltd., Rob Herring <robh@kernel.org> */
+/* Copyright 2019 Collabora ltd. */
+#include <linux/bitfield.h>
+#include <linux/bitmap.h>
+#include <linux/delay.h>
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iopoll.h>
+#include <linux/platform_device.h>
+
+#include "panfrost_device.h"
+#include "panfrost_features.h"
+#include "panfrost_issues.h"
+#include "panfrost_gpu.h"
+#include "panfrost_regs.h"
+
+#define gpu_write(dev, reg, data) writel(data, dev->iomem + reg)
+#define gpu_read(dev, reg) readl(dev->iomem + reg)
+
+static irqreturn_t panfrost_gpu_irq_handler(int irq, void *data)
+{
+ struct panfrost_device *pfdev = data;
+ u32 state = gpu_read(pfdev, GPU_INT_STAT);
+ u32 fault_status = gpu_read(pfdev, GPU_FAULT_STATUS);
+
+ if (!state)
+ return IRQ_NONE;
+
+ if (state & GPU_IRQ_MASK_ERROR) {
+ u64 address = (u64) gpu_read(pfdev, GPU_FAULT_ADDRESS_HI) << 32;
+ address |= gpu_read(pfdev, GPU_FAULT_ADDRESS_LO);
+
+ dev_warn(pfdev->dev, "GPU Fault 0x%08x (%s) at 0x%016llx\n",
+ fault_status & 0xFF, panfrost_exception_name(pfdev, fault_status),
+ address);
+
+ if (state & GPU_IRQ_MULTIPLE_FAULT)
+ dev_warn(pfdev->dev, "There were multiple GPU faults - some have not been reported\n");
+
+ gpu_write(pfdev, GPU_INT_MASK, 0);
+ }
+
+ gpu_write(pfdev, GPU_INT_CLEAR, state);
+
+ return IRQ_HANDLED;
+}
+
+int panfrost_gpu_soft_reset(struct panfrost_device *pfdev)
+{
+ int ret;
+ u32 val;
+
+ gpu_write(pfdev, GPU_INT_MASK, 0);
+ gpu_write(pfdev, GPU_INT_CLEAR, GPU_IRQ_RESET_COMPLETED);
+ gpu_write(pfdev, GPU_CMD, GPU_CMD_SOFT_RESET);
+
+ ret = readl_relaxed_poll_timeout(pfdev->iomem + GPU_INT_RAWSTAT,
+ val, val & GPU_IRQ_RESET_COMPLETED, 100, 10000);
+
+ if (ret) {
+ dev_err(pfdev->dev, "gpu soft reset timed out\n");
+ return ret;
+ }
+
+ gpu_write(pfdev, GPU_INT_CLEAR, GPU_IRQ_MASK_ALL);
+ gpu_write(pfdev, GPU_INT_MASK, GPU_IRQ_MASK_ALL);
+
+ return 0;
+}
+
+static void panfrost_gpu_init_quirks(struct panfrost_device *pfdev)
+{
+ u32 quirks = 0;
+
+ if (panfrost_has_hw_issue(pfdev, HW_ISSUE_8443) ||
+ panfrost_has_hw_issue(pfdev, HW_ISSUE_11035))
+ quirks |= SC_LS_PAUSEBUFFER_DISABLE;
+
+ if (panfrost_has_hw_issue(pfdev, HW_ISSUE_10327))
+ quirks |= SC_SDC_DISABLE_OQ_DISCARD;
+
+ if (panfrost_has_hw_issue(pfdev, HW_ISSUE_10797))
+ quirks |= SC_ENABLE_TEXGRD_FLAGS;
+
+ if (!panfrost_has_hw_issue(pfdev, GPUCORE_1619)) {
+ if (panfrost_model_cmp(pfdev, 0x750) < 0) /* T60x, T62x, T72x */
+ quirks |= SC_LS_ATTR_CHECK_DISABLE;
+ else if (panfrost_model_cmp(pfdev, 0x880) <= 0) /* T76x, T8xx */
+ quirks |= SC_LS_ALLOW_ATTR_TYPES;
+ }
+
+ if (panfrost_has_hw_feature(pfdev, HW_FEATURE_TLS_HASHING))
+ quirks |= SC_TLS_HASH_ENABLE;
+
+ if (quirks)
+ gpu_write(pfdev, GPU_SHADER_CONFIG, quirks);
+
+
+ quirks = gpu_read(pfdev, GPU_TILER_CONFIG);
+
+ /* Set tiler clock gate override if required */
+ if (panfrost_has_hw_issue(pfdev, HW_ISSUE_T76X_3953))
+ quirks |= TC_CLOCK_GATE_OVERRIDE;
+
+ gpu_write(pfdev, GPU_TILER_CONFIG, quirks);
+
+
+ quirks = gpu_read(pfdev, GPU_L2_MMU_CONFIG);
+
+ /* Limit read & write ID width for AXI */
+ if (panfrost_has_hw_feature(pfdev, HW_FEATURE_3BIT_EXT_RW_L2_MMU_CONFIG))
+ quirks &= ~(L2_MMU_CONFIG_3BIT_LIMIT_EXTERNAL_READS |
+ L2_MMU_CONFIG_3BIT_LIMIT_EXTERNAL_WRITES);
+ else
+ quirks &= ~(L2_MMU_CONFIG_LIMIT_EXTERNAL_READS |
+ L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES);
+
+ gpu_write(pfdev, GPU_L2_MMU_CONFIG, quirks);
+
+ quirks = 0;
+ if ((panfrost_model_eq(pfdev, 0x860) || panfrost_model_eq(pfdev, 0x880)) &&
+ pfdev->features.revision >= 0x2000)
+ quirks |= JM_MAX_JOB_THROTTLE_LIMIT << JM_JOB_THROTTLE_LIMIT_SHIFT;
+ else if (panfrost_model_eq(pfdev, 0x6000) &&
+ pfdev->features.coherency_features == COHERENCY_ACE)
+ quirks |= (COHERENCY_ACE_LITE | COHERENCY_ACE) <<
+ JM_FORCE_COHERENCY_FEATURES_SHIFT;
+
+ if (quirks)
+ gpu_write(pfdev, GPU_JM_CONFIG, quirks);
+}
+
+#define MAX_HW_REVS 6
+
+struct panfrost_model {
+ const char *name;
+ u32 id;
+ u32 id_mask;
+ u64 features;
+ u64 issues;
+ struct {
+ u32 revision;
+ u64 issues;
+ } revs[MAX_HW_REVS];
+};
+
+#define GPU_MODEL(_name, _id, ...) \
+{\
+ .name = __stringify(_name), \
+ .id = _id, \
+ .features = hw_features_##_name, \
+ .issues = hw_issues_##_name, \
+ .revs = { __VA_ARGS__ }, \
+}
+
+#define GPU_REV_EXT(name, _rev, _p, _s, stat) \
+{\
+ .revision = (_rev) << 12 | (_p) << 4 | (_s), \
+ .issues = hw_issues_##name##_r##_rev##p##_p##stat, \
+}
+#define GPU_REV(name, r, p) GPU_REV_EXT(name, r, p, 0, )
+
+static const struct panfrost_model gpu_models[] = {
+ /* T60x has an oddball version */
+ GPU_MODEL(t600, 0x600,
+ GPU_REV_EXT(t600, 0, 0, 1, _15dev0)),
+ GPU_MODEL(t620, 0x620,
+ GPU_REV(t620, 0, 1), GPU_REV(t620, 1, 0)),
+ GPU_MODEL(t720, 0x720),
+ GPU_MODEL(t760, 0x750,
+ GPU_REV(t760, 0, 0), GPU_REV(t760, 0, 1),
+ GPU_REV_EXT(t760, 0, 1, 0, _50rel0),
+ GPU_REV(t760, 0, 2), GPU_REV(t760, 0, 3)),
+ GPU_MODEL(t820, 0x820),
+ GPU_MODEL(t830, 0x830),
+ GPU_MODEL(t860, 0x860),
+ GPU_MODEL(t880, 0x880),
+
+ GPU_MODEL(g71, 0x6000,
+ GPU_REV_EXT(g71, 0, 0, 1, _05dev0)),
+ GPU_MODEL(g72, 0x6001),
+ GPU_MODEL(g51, 0x7000),
+ GPU_MODEL(g76, 0x7001),
+ GPU_MODEL(g52, 0x7002),
+ GPU_MODEL(g31, 0x7003,
+ GPU_REV(g31, 1, 0)),
+};
+
+static void panfrost_gpu_init_features(struct panfrost_device *pfdev)
+{
+ u32 gpu_id, num_js, major, minor, status, rev;
+ const char *name = "unknown";
+ u64 hw_feat = 0;
+ u64 hw_issues = hw_issues_all;
+ const struct panfrost_model *model;
+ int i;
+
+ pfdev->features.l2_features = gpu_read(pfdev, GPU_L2_FEATURES);
+ pfdev->features.core_features = gpu_read(pfdev, GPU_CORE_FEATURES);
+ pfdev->features.tiler_features = gpu_read(pfdev, GPU_TILER_FEATURES);
+ pfdev->features.mem_features = gpu_read(pfdev, GPU_MEM_FEATURES);
+ pfdev->features.mmu_features = gpu_read(pfdev, GPU_MMU_FEATURES);
+ pfdev->features.thread_features = gpu_read(pfdev, GPU_THREAD_FEATURES);
+ pfdev->features.coherency_features = gpu_read(pfdev, GPU_COHERENCY_FEATURES);
+ for (i = 0; i < 4; i++)
+ pfdev->features.texture_features[i] = gpu_read(pfdev, GPU_TEXTURE_FEATURES(i));
+
+ pfdev->features.as_present = gpu_read(pfdev, GPU_AS_PRESENT);
+
+ pfdev->features.js_present = gpu_read(pfdev, GPU_JS_PRESENT);
+ num_js = hweight32(pfdev->features.js_present);
+ for (i = 0; i < num_js; i++)
+ pfdev->features.js_features[i] = gpu_read(pfdev, GPU_JS_FEATURES(i));
+
+ pfdev->features.shader_present = gpu_read(pfdev, GPU_SHADER_PRESENT_LO);
+ pfdev->features.shader_present |= (u64)gpu_read(pfdev, GPU_SHADER_PRESENT_HI) << 32;
+
+ pfdev->features.tiler_present = gpu_read(pfdev, GPU_TILER_PRESENT_LO);
+ pfdev->features.tiler_present |= (u64)gpu_read(pfdev, GPU_TILER_PRESENT_HI) << 32;
+
+ pfdev->features.l2_present = gpu_read(pfdev, GPU_L2_PRESENT_LO);
+ pfdev->features.l2_present |= (u64)gpu_read(pfdev, GPU_L2_PRESENT_HI) << 32;
+ pfdev->features.nr_core_groups = hweight64(pfdev->features.l2_present);
+
+ pfdev->features.stack_present = gpu_read(pfdev, GPU_STACK_PRESENT_LO);
+ pfdev->features.stack_present |= (u64)gpu_read(pfdev, GPU_STACK_PRESENT_HI) << 32;
+
+ gpu_id = gpu_read(pfdev, GPU_ID);
+ pfdev->features.revision = gpu_id & 0xffff;
+ pfdev->features.id = gpu_id >> 16;
+
+ /* The T60x has an oddball ID value. Fix it up to the standard Midgard
+ * format so we (and userspace) don't have to special case it.
+ */
+ if (pfdev->features.id == 0x6956)
+ pfdev->features.id = 0x0600;
+
+ major = (pfdev->features.revision >> 12) & 0xf;
+ minor = (pfdev->features.revision >> 4) & 0xff;
+ status = pfdev->features.revision & 0xf;
+ rev = pfdev->features.revision;
+
+ gpu_id = pfdev->features.id;
+
+ for (model = gpu_models; model->name; model++) {
+ int best = -1;
+
+ if (!panfrost_model_eq(pfdev, model->id))
+ continue;
+
+ name = model->name;
+ hw_feat = model->features;
+ hw_issues |= model->issues;
+ for (i = 0; i < MAX_HW_REVS; i++) {
+ if (model->revs[i].revision == rev) {
+ best = i;
+ break;
+ } else if (model->revs[i].revision == (rev & ~0xf))
+ best = i;
+ }
+
+ if (best >= 0)
+ hw_issues |= model->revs[best].issues;
+
+ break;
+ }
+
+ bitmap_from_u64(pfdev->features.hw_features, hw_feat);
+ bitmap_from_u64(pfdev->features.hw_issues, hw_issues);
+
+ dev_info(pfdev->dev, "mali-%s id 0x%x major 0x%x minor 0x%x status 0x%x",
+ name, gpu_id, major, minor, status);
+ dev_info(pfdev->dev, "features: %64pb, issues: %64pb",
+ pfdev->features.hw_features,
+ pfdev->features.hw_issues);
+
+ dev_info(pfdev->dev, "Features: L2:0x%08x Shader:0x%08x Tiler:0x%08x Mem:0x%0x MMU:0x%08x AS:0x%x JS:0x%x",
+ pfdev->features.l2_features,
+ pfdev->features.core_features,
+ pfdev->features.tiler_features,
+ pfdev->features.mem_features,
+ pfdev->features.mmu_features,
+ pfdev->features.as_present,
+ pfdev->features.js_present);
+
+ dev_info(pfdev->dev, "shader_present=0x%0llx l2_present=0x%0llx",
+ pfdev->features.shader_present, pfdev->features.l2_present);
+}
+
+void panfrost_gpu_power_on(struct panfrost_device *pfdev)
+{
+ int ret;
+ u32 val;
+
+ /* Just turn on everything for now */
+ gpu_write(pfdev, L2_PWRON_LO, pfdev->features.l2_present);
+ ret = readl_relaxed_poll_timeout(pfdev->iomem + L2_READY_LO,
+ val, val == pfdev->features.l2_present, 100, 1000);
+
+ gpu_write(pfdev, STACK_PWRON_LO, pfdev->features.stack_present);
+ ret |= readl_relaxed_poll_timeout(pfdev->iomem + STACK_READY_LO,
+ val, val == pfdev->features.stack_present, 100, 1000);
+
+ gpu_write(pfdev, SHADER_PWRON_LO, pfdev->features.shader_present);
+ ret |= readl_relaxed_poll_timeout(pfdev->iomem + SHADER_READY_LO,
+ val, val == pfdev->features.shader_present, 100, 1000);
+
+ gpu_write(pfdev, TILER_PWRON_LO, pfdev->features.tiler_present);
+ ret |= readl_relaxed_poll_timeout(pfdev->iomem + TILER_READY_LO,
+ val, val == pfdev->features.tiler_present, 100, 1000);
+
+ if (ret)
+ dev_err(pfdev->dev, "error powering up gpu");
+}
+
+void panfrost_gpu_power_off(struct panfrost_device *pfdev)
+{
+ gpu_write(pfdev, TILER_PWROFF_LO, 0);
+ gpu_write(pfdev, SHADER_PWROFF_LO, 0);
+ gpu_write(pfdev, STACK_PWROFF_LO, 0);
+ gpu_write(pfdev, L2_PWROFF_LO, 0);
+}
+
+int panfrost_gpu_init(struct panfrost_device *pfdev)
+{
+ int err, irq;
+
+ err = panfrost_gpu_soft_reset(pfdev);
+ if (err)
+ return err;
+
+ panfrost_gpu_init_features(pfdev);
+
+ dma_set_mask_and_coherent(pfdev->dev,
+ DMA_BIT_MASK(FIELD_GET(0xff00, pfdev->features.mmu_features)));
+
+ irq = platform_get_irq_byname(to_platform_device(pfdev->dev), "gpu");
+ if (irq <= 0)
+ return -ENODEV;
+
+ err = devm_request_irq(pfdev->dev, irq, panfrost_gpu_irq_handler,
+ IRQF_SHARED, "gpu", pfdev);
+ if (err) {
+ dev_err(pfdev->dev, "failed to request gpu irq");
+ return err;
+ }
+
+ panfrost_gpu_init_quirks(pfdev);
+ panfrost_gpu_power_on(pfdev);
+
+ return 0;
+}
+
+void panfrost_gpu_fini(struct panfrost_device *pfdev)
+{
+ panfrost_gpu_power_off(pfdev);
+}
+
+u32 panfrost_gpu_get_latest_flush_id(struct panfrost_device *pfdev)
+{
+ if (panfrost_has_hw_feature(pfdev, HW_FEATURE_FLUSH_REDUCTION))
+ return gpu_read(pfdev, GPU_LATEST_FLUSH_ID);
+ return 0;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
+/* Copyright 2019 Collabora ltd. */
+
+#ifndef __PANFROST_GPU_H__
+#define __PANFROST_GPU_H__
+
+struct panfrost_device;
+
+int panfrost_gpu_init(struct panfrost_device *pfdev);
+void panfrost_gpu_fini(struct panfrost_device *pfdev);
+
+u32 panfrost_gpu_get_latest_flush_id(struct panfrost_device *pfdev);
+
+int panfrost_gpu_soft_reset(struct panfrost_device *pfdev);
+void panfrost_gpu_power_on(struct panfrost_device *pfdev);
+void panfrost_gpu_power_off(struct panfrost_device *pfdev);
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* (C) COPYRIGHT 2014-2018 ARM Limited. All rights reserved. */
+/* Copyright 2019 Linaro, Ltd., Rob Herring <robh@kernel.org> */
+#ifndef __PANFROST_ISSUES_H__
+#define __PANFROST_ISSUES_H__
+
+#include <linux/bitops.h>
+
+#include "panfrost_device.h"
+
+/*
+ * This is not a complete list of issues, but only the ones the driver needs
+ * to care about.
+ */
+enum panfrost_hw_issue {
+ HW_ISSUE_6367,
+ HW_ISSUE_6787,
+ HW_ISSUE_8186,
+ HW_ISSUE_8245,
+ HW_ISSUE_8316,
+ HW_ISSUE_8394,
+ HW_ISSUE_8401,
+ HW_ISSUE_8408,
+ HW_ISSUE_8443,
+ HW_ISSUE_8987,
+ HW_ISSUE_9435,
+ HW_ISSUE_9510,
+ HW_ISSUE_9630,
+ HW_ISSUE_10327,
+ HW_ISSUE_10649,
+ HW_ISSUE_10676,
+ HW_ISSUE_10797,
+ HW_ISSUE_10817,
+ HW_ISSUE_10883,
+ HW_ISSUE_10959,
+ HW_ISSUE_10969,
+ HW_ISSUE_11020,
+ HW_ISSUE_11024,
+ HW_ISSUE_11035,
+ HW_ISSUE_11056,
+ HW_ISSUE_T76X_3542,
+ HW_ISSUE_T76X_3953,
+ HW_ISSUE_TMIX_8463,
+ GPUCORE_1619,
+ HW_ISSUE_TMIX_8438,
+ HW_ISSUE_TGOX_R1_1234,
+ HW_ISSUE_END
+};
+
+#define hw_issues_all (\
+ BIT_ULL(HW_ISSUE_9435))
+
+#define hw_issues_t600 (\
+ BIT_ULL(HW_ISSUE_6367) | \
+ BIT_ULL(HW_ISSUE_6787) | \
+ BIT_ULL(HW_ISSUE_8408) | \
+ BIT_ULL(HW_ISSUE_9510) | \
+ BIT_ULL(HW_ISSUE_10649) | \
+ BIT_ULL(HW_ISSUE_10676) | \
+ BIT_ULL(HW_ISSUE_10883) | \
+ BIT_ULL(HW_ISSUE_11020) | \
+ BIT_ULL(HW_ISSUE_11035) | \
+ BIT_ULL(HW_ISSUE_11056) | \
+ BIT_ULL(HW_ISSUE_TMIX_8438))
+
+#define hw_issues_t600_r0p0_15dev0 (\
+ BIT_ULL(HW_ISSUE_8186) | \
+ BIT_ULL(HW_ISSUE_8245) | \
+ BIT_ULL(HW_ISSUE_8316) | \
+ BIT_ULL(HW_ISSUE_8394) | \
+ BIT_ULL(HW_ISSUE_8401) | \
+ BIT_ULL(HW_ISSUE_8443) | \
+ BIT_ULL(HW_ISSUE_8987) | \
+ BIT_ULL(HW_ISSUE_9630) | \
+ BIT_ULL(HW_ISSUE_10969) | \
+ BIT_ULL(GPUCORE_1619))
+
+#define hw_issues_t620 (\
+ BIT_ULL(HW_ISSUE_10649) | \
+ BIT_ULL(HW_ISSUE_10883) | \
+ BIT_ULL(HW_ISSUE_10959) | \
+ BIT_ULL(HW_ISSUE_11056) | \
+ BIT_ULL(HW_ISSUE_TMIX_8438))
+
+#define hw_issues_t620_r0p1 (\
+ BIT_ULL(HW_ISSUE_10327) | \
+ BIT_ULL(HW_ISSUE_10676) | \
+ BIT_ULL(HW_ISSUE_10817) | \
+ BIT_ULL(HW_ISSUE_11020) | \
+ BIT_ULL(HW_ISSUE_11024) | \
+ BIT_ULL(HW_ISSUE_11035))
+
+#define hw_issues_t620_r1p0 (\
+ BIT_ULL(HW_ISSUE_11020) | \
+ BIT_ULL(HW_ISSUE_11024))
+
+#define hw_issues_t720 (\
+ BIT_ULL(HW_ISSUE_10649) | \
+ BIT_ULL(HW_ISSUE_10797) | \
+ BIT_ULL(HW_ISSUE_10883) | \
+ BIT_ULL(HW_ISSUE_11056) | \
+ BIT_ULL(HW_ISSUE_TMIX_8438))
+
+#define hw_issues_t760 (\
+ BIT_ULL(HW_ISSUE_10883) | \
+ BIT_ULL(HW_ISSUE_T76X_3953) | \
+ BIT_ULL(HW_ISSUE_TMIX_8438))
+
+#define hw_issues_t760_r0p0 (\
+ BIT_ULL(HW_ISSUE_11020) | \
+ BIT_ULL(HW_ISSUE_11024) | \
+ BIT_ULL(HW_ISSUE_T76X_3542))
+
+#define hw_issues_t760_r0p1 (\
+ BIT_ULL(HW_ISSUE_11020) | \
+ BIT_ULL(HW_ISSUE_11024) | \
+ BIT_ULL(HW_ISSUE_T76X_3542))
+
+#define hw_issues_t760_r0p1_50rel0 (\
+ BIT_ULL(HW_ISSUE_T76X_3542))
+
+#define hw_issues_t760_r0p2 (\
+ BIT_ULL(HW_ISSUE_11020) | \
+ BIT_ULL(HW_ISSUE_11024) | \
+ BIT_ULL(HW_ISSUE_T76X_3542))
+
+#define hw_issues_t760_r0p3 (\
+ BIT_ULL(HW_ISSUE_T76X_3542))
+
+#define hw_issues_t820 (\
+ BIT_ULL(HW_ISSUE_10883) | \
+ BIT_ULL(HW_ISSUE_T76X_3953) | \
+ BIT_ULL(HW_ISSUE_TMIX_8438))
+
+#define hw_issues_t830 (\
+ BIT_ULL(HW_ISSUE_10883) | \
+ BIT_ULL(HW_ISSUE_T76X_3953) | \
+ BIT_ULL(HW_ISSUE_TMIX_8438))
+
+#define hw_issues_t860 (\
+ BIT_ULL(HW_ISSUE_10883) | \
+ BIT_ULL(HW_ISSUE_T76X_3953) | \
+ BIT_ULL(HW_ISSUE_TMIX_8438))
+
+#define hw_issues_t880 (\
+ BIT_ULL(HW_ISSUE_10883) | \
+ BIT_ULL(HW_ISSUE_T76X_3953) | \
+ BIT_ULL(HW_ISSUE_TMIX_8438))
+
+#define hw_issues_g31 0
+
+#define hw_issues_g31_r1p0 (\
+ BIT_ULL(HW_ISSUE_TGOX_R1_1234))
+
+#define hw_issues_g51 0
+
+#define hw_issues_g52 0
+
+#define hw_issues_g71 (\
+ BIT_ULL(HW_ISSUE_TMIX_8463) | \
+ BIT_ULL(HW_ISSUE_TMIX_8438))
+
+#define hw_issues_g71_r0p0_05dev0 (\
+ BIT_ULL(HW_ISSUE_T76X_3953))
+
+#define hw_issues_g72 0
+
+#define hw_issues_g76 0
+
+static inline bool panfrost_has_hw_issue(struct panfrost_device *pfdev,
+ enum panfrost_hw_issue issue)
+{
+ return test_bit(issue, pfdev->features.hw_issues);
+}
+
+#endif /* __PANFROST_ISSUES_H__ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
+/* Copyright 2019 Collabora ltd. */
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/reservation.h>
+#include <drm/gpu_scheduler.h>
+#include <drm/panfrost_drm.h>
+
+#include "panfrost_device.h"
+#include "panfrost_devfreq.h"
+#include "panfrost_job.h"
+#include "panfrost_features.h"
+#include "panfrost_issues.h"
+#include "panfrost_gem.h"
+#include "panfrost_regs.h"
+#include "panfrost_gpu.h"
+#include "panfrost_mmu.h"
+
+#define job_write(dev, reg, data) writel(data, dev->iomem + (reg))
+#define job_read(dev, reg) readl(dev->iomem + (reg))
+
+struct panfrost_queue_state {
+ struct drm_gpu_scheduler sched;
+
+ u64 fence_context;
+ u64 emit_seqno;
+};
+
+struct panfrost_job_slot {
+ struct panfrost_queue_state queue[NUM_JOB_SLOTS];
+ spinlock_t job_lock;
+};
+
+static struct panfrost_job *
+to_panfrost_job(struct drm_sched_job *sched_job)
+{
+ return container_of(sched_job, struct panfrost_job, base);
+}
+
+struct panfrost_fence {
+ struct dma_fence base;
+ struct drm_device *dev;
+ /* panfrost seqno for signaled() test */
+ u64 seqno;
+ int queue;
+};
+
+static inline struct panfrost_fence *
+to_panfrost_fence(struct dma_fence *fence)
+{
+ return (struct panfrost_fence *)fence;
+}
+
+static const char *panfrost_fence_get_driver_name(struct dma_fence *fence)
+{
+ return "panfrost";
+}
+
+static const char *panfrost_fence_get_timeline_name(struct dma_fence *fence)
+{
+ struct panfrost_fence *f = to_panfrost_fence(fence);
+
+ switch (f->queue) {
+ case 0:
+ return "panfrost-js-0";
+ case 1:
+ return "panfrost-js-1";
+ case 2:
+ return "panfrost-js-2";
+ default:
+ return NULL;
+ }
+}
+
+static const struct dma_fence_ops panfrost_fence_ops = {
+ .get_driver_name = panfrost_fence_get_driver_name,
+ .get_timeline_name = panfrost_fence_get_timeline_name,
+};
+
+static struct dma_fence *panfrost_fence_create(struct panfrost_device *pfdev, int js_num)
+{
+ struct panfrost_fence *fence;
+ struct panfrost_job_slot *js = pfdev->js;
+
+ fence = kzalloc(sizeof(*fence), GFP_KERNEL);
+ if (!fence)
+ return ERR_PTR(-ENOMEM);
+
+ fence->dev = pfdev->ddev;
+ fence->queue = js_num;
+ fence->seqno = ++js->queue[js_num].emit_seqno;
+ dma_fence_init(&fence->base, &panfrost_fence_ops, &js->job_lock,
+ js->queue[js_num].fence_context, fence->seqno);
+
+ return &fence->base;
+}
+
+static int panfrost_job_get_slot(struct panfrost_job *job)
+{
+ /* JS0: fragment jobs.
+ * JS1: vertex/tiler jobs
+ * JS2: compute jobs
+ */
+ if (job->requirements & PANFROST_JD_REQ_FS)
+ return 0;
+
+/* Not exposed to userspace yet */
+#if 0
+ if (job->requirements & PANFROST_JD_REQ_ONLY_COMPUTE) {
+ if ((job->requirements & PANFROST_JD_REQ_CORE_GRP_MASK) &&
+ (job->pfdev->features.nr_core_groups == 2))
+ return 2;
+ if (panfrost_has_hw_issue(job->pfdev, HW_ISSUE_8987))
+ return 2;
+ }
+#endif
+ return 1;
+}
+
+static void panfrost_job_write_affinity(struct panfrost_device *pfdev,
+ u32 requirements,
+ int js)
+{
+ u64 affinity;
+
+ /*
+ * Use all cores for now.
+ * Eventually we may need to support tiler only jobs and h/w with
+ * multiple (2) coherent core groups
+ */
+ affinity = pfdev->features.shader_present;
+
+ job_write(pfdev, JS_AFFINITY_NEXT_LO(js), affinity & 0xFFFFFFFF);
+ job_write(pfdev, JS_AFFINITY_NEXT_HI(js), affinity >> 32);
+}
+
+static void panfrost_job_hw_submit(struct panfrost_job *job, int js)
+{
+ struct panfrost_device *pfdev = job->pfdev;
+ unsigned long flags;
+ u32 cfg;
+ u64 jc_head = job->jc;
+ int ret;
+
+ ret = pm_runtime_get_sync(pfdev->dev);
+ if (ret < 0)
+ return;
+
+ if (WARN_ON(job_read(pfdev, JS_COMMAND_NEXT(js))))
+ goto end;
+
+ panfrost_devfreq_record_transition(pfdev, js);
+ spin_lock_irqsave(&pfdev->hwaccess_lock, flags);
+
+ job_write(pfdev, JS_HEAD_NEXT_LO(js), jc_head & 0xFFFFFFFF);
+ job_write(pfdev, JS_HEAD_NEXT_HI(js), jc_head >> 32);
+
+ panfrost_job_write_affinity(pfdev, job->requirements, js);
+
+ /* start MMU, medium priority, cache clean/flush on end, clean/flush on
+ * start */
+ /* TODO: different address spaces */
+ cfg = JS_CONFIG_THREAD_PRI(8) |
+ JS_CONFIG_START_FLUSH_CLEAN_INVALIDATE |
+ JS_CONFIG_END_FLUSH_CLEAN_INVALIDATE;
+
+ if (panfrost_has_hw_feature(pfdev, HW_FEATURE_FLUSH_REDUCTION))
+ cfg |= JS_CONFIG_ENABLE_FLUSH_REDUCTION;
+
+ if (panfrost_has_hw_issue(pfdev, HW_ISSUE_10649))
+ cfg |= JS_CONFIG_START_MMU;
+
+ job_write(pfdev, JS_CONFIG_NEXT(js), cfg);
+
+ if (panfrost_has_hw_feature(pfdev, HW_FEATURE_FLUSH_REDUCTION))
+ job_write(pfdev, JS_FLUSH_ID_NEXT(js), job->flush_id);
+
+ /* GO ! */
+ dev_dbg(pfdev->dev, "JS: Submitting atom %p to js[%d] with head=0x%llx",
+ job, js, jc_head);
+
+ job_write(pfdev, JS_COMMAND_NEXT(js), JS_COMMAND_START);
+
+ spin_unlock_irqrestore(&pfdev->hwaccess_lock, flags);
+
+end:
+ pm_runtime_mark_last_busy(pfdev->dev);
+ pm_runtime_put_autosuspend(pfdev->dev);
+}
+
+static void panfrost_acquire_object_fences(struct drm_gem_object **bos,
+ int bo_count,
+ struct dma_fence **implicit_fences)
+{
+ int i;
+
+ for (i = 0; i < bo_count; i++)
+ implicit_fences[i] = reservation_object_get_excl_rcu(bos[i]->resv);
+}
+
+static void panfrost_attach_object_fences(struct drm_gem_object **bos,
+ int bo_count,
+ struct dma_fence *fence)
+{
+ int i;
+
+ for (i = 0; i < bo_count; i++)
+ reservation_object_add_excl_fence(bos[i]->resv, fence);
+}
+
+int panfrost_job_push(struct panfrost_job *job)
+{
+ struct panfrost_device *pfdev = job->pfdev;
+ int slot = panfrost_job_get_slot(job);
+ struct drm_sched_entity *entity = &job->file_priv->sched_entity[slot];
+ struct ww_acquire_ctx acquire_ctx;
+ int ret = 0;
+
+ mutex_lock(&pfdev->sched_lock);
+
+ ret = drm_gem_lock_reservations(job->bos, job->bo_count,
+ &acquire_ctx);
+ if (ret) {
+ mutex_unlock(&pfdev->sched_lock);
+ return ret;
+ }
+
+ ret = drm_sched_job_init(&job->base, entity, NULL);
+ if (ret) {
+ mutex_unlock(&pfdev->sched_lock);
+ goto unlock;
+ }
+
+ job->render_done_fence = dma_fence_get(&job->base.s_fence->finished);
+
+ kref_get(&job->refcount); /* put by scheduler job completion */
+
+ panfrost_acquire_object_fences(job->bos, job->bo_count,
+ job->implicit_fences);
+
+ drm_sched_entity_push_job(&job->base, entity);
+
+ mutex_unlock(&pfdev->sched_lock);
+
+ panfrost_attach_object_fences(job->bos, job->bo_count,
+ job->render_done_fence);
+
+unlock:
+ drm_gem_unlock_reservations(job->bos, job->bo_count, &acquire_ctx);
+
+ return ret;
+}
+
+static void panfrost_job_cleanup(struct kref *ref)
+{
+ struct panfrost_job *job = container_of(ref, struct panfrost_job,
+ refcount);
+ unsigned int i;
+
+ if (job->in_fences) {
+ for (i = 0; i < job->in_fence_count; i++)
+ dma_fence_put(job->in_fences[i]);
+ kvfree(job->in_fences);
+ }
+ if (job->implicit_fences) {
+ for (i = 0; i < job->bo_count; i++)
+ dma_fence_put(job->implicit_fences[i]);
+ kvfree(job->implicit_fences);
+ }
+ dma_fence_put(job->done_fence);
+ dma_fence_put(job->render_done_fence);
+
+ if (job->bos) {
+ for (i = 0; i < job->bo_count; i++)
+ drm_gem_object_put_unlocked(job->bos[i]);
+ kvfree(job->bos);
+ }
+
+ kfree(job);
+}
+
+void panfrost_job_put(struct panfrost_job *job)
+{
+ kref_put(&job->refcount, panfrost_job_cleanup);
+}
+
+static void panfrost_job_free(struct drm_sched_job *sched_job)
+{
+ struct panfrost_job *job = to_panfrost_job(sched_job);
+
+ drm_sched_job_cleanup(sched_job);
+
+ panfrost_job_put(job);
+}
+
+static struct dma_fence *panfrost_job_dependency(struct drm_sched_job *sched_job,
+ struct drm_sched_entity *s_entity)
+{
+ struct panfrost_job *job = to_panfrost_job(sched_job);
+ struct dma_fence *fence;
+ unsigned int i;
+
+ /* Explicit fences */
+ for (i = 0; i < job->in_fence_count; i++) {
+ if (job->in_fences[i]) {
+ fence = job->in_fences[i];
+ job->in_fences[i] = NULL;
+ return fence;
+ }
+ }
+
+ /* Implicit fences, max. one per BO */
+ for (i = 0; i < job->bo_count; i++) {
+ if (job->implicit_fences[i]) {
+ fence = job->implicit_fences[i];
+ job->implicit_fences[i] = NULL;
+ return fence;
+ }
+ }
+
+ return NULL;
+}
+
+static struct dma_fence *panfrost_job_run(struct drm_sched_job *sched_job)
+{
+ struct panfrost_job *job = to_panfrost_job(sched_job);
+ struct panfrost_device *pfdev = job->pfdev;
+ int slot = panfrost_job_get_slot(job);
+ struct dma_fence *fence = NULL;
+
+ if (unlikely(job->base.s_fence->finished.error))
+ return NULL;
+
+ pfdev->jobs[slot] = job;
+
+ fence = panfrost_fence_create(pfdev, slot);
+ if (IS_ERR(fence))
+ return NULL;
+
+ if (job->done_fence)
+ dma_fence_put(job->done_fence);
+ job->done_fence = dma_fence_get(fence);
+
+ panfrost_job_hw_submit(job, slot);
+
+ return fence;
+}
+
+void panfrost_job_enable_interrupts(struct panfrost_device *pfdev)
+{
+ int j;
+ u32 irq_mask = 0;
+
+ for (j = 0; j < NUM_JOB_SLOTS; j++) {
+ irq_mask |= MK_JS_MASK(j);
+ }
+
+ job_write(pfdev, JOB_INT_CLEAR, irq_mask);
+ job_write(pfdev, JOB_INT_MASK, irq_mask);
+}
+
+static void panfrost_job_timedout(struct drm_sched_job *sched_job)
+{
+ struct panfrost_job *job = to_panfrost_job(sched_job);
+ struct panfrost_device *pfdev = job->pfdev;
+ int js = panfrost_job_get_slot(job);
+ int i;
+
+ /*
+ * If the GPU managed to complete this jobs fence, the timeout is
+ * spurious. Bail out.
+ */
+ if (dma_fence_is_signaled(job->done_fence))
+ return;
+
+ dev_err(pfdev->dev, "gpu sched timeout, js=%d, status=0x%x, head=0x%x, tail=0x%x, sched_job=%p",
+ js,
+ job_read(pfdev, JS_STATUS(js)),
+ job_read(pfdev, JS_HEAD_LO(js)),
+ job_read(pfdev, JS_TAIL_LO(js)),
+ sched_job);
+
+ mutex_lock(&pfdev->reset_lock);
+
+ for (i = 0; i < NUM_JOB_SLOTS; i++)
+ drm_sched_stop(&pfdev->js->queue[i].sched);
+
+ if (sched_job)
+ drm_sched_increase_karma(sched_job);
+
+ /* panfrost_core_dump(pfdev); */
+
+ panfrost_devfreq_record_transition(pfdev, js);
+ panfrost_gpu_soft_reset(pfdev);
+
+ /* TODO: Re-enable all other address spaces */
+ panfrost_mmu_enable(pfdev, 0);
+ panfrost_gpu_power_on(pfdev);
+ panfrost_job_enable_interrupts(pfdev);
+
+ for (i = 0; i < NUM_JOB_SLOTS; i++)
+ drm_sched_resubmit_jobs(&pfdev->js->queue[i].sched);
+
+ /* restart scheduler after GPU is usable again */
+ for (i = 0; i < NUM_JOB_SLOTS; i++)
+ drm_sched_start(&pfdev->js->queue[i].sched, true);
+
+ mutex_unlock(&pfdev->reset_lock);
+}
+
+static const struct drm_sched_backend_ops panfrost_sched_ops = {
+ .dependency = panfrost_job_dependency,
+ .run_job = panfrost_job_run,
+ .timedout_job = panfrost_job_timedout,
+ .free_job = panfrost_job_free
+};
+
+static irqreturn_t panfrost_job_irq_handler(int irq, void *data)
+{
+ struct panfrost_device *pfdev = data;
+ u32 status = job_read(pfdev, JOB_INT_STAT);
+ int j;
+
+ dev_dbg(pfdev->dev, "jobslot irq status=%x\n", status);
+
+ if (!status)
+ return IRQ_NONE;
+
+ pm_runtime_mark_last_busy(pfdev->dev);
+
+ for (j = 0; status; j++) {
+ u32 mask = MK_JS_MASK(j);
+
+ if (!(status & mask))
+ continue;
+
+ job_write(pfdev, JOB_INT_CLEAR, mask);
+
+ if (status & JOB_INT_MASK_ERR(j)) {
+ job_write(pfdev, JS_COMMAND_NEXT(j), JS_COMMAND_NOP);
+
+ dev_err(pfdev->dev, "js fault, js=%d, status=%s, head=0x%x, tail=0x%x",
+ j,
+ panfrost_exception_name(pfdev, job_read(pfdev, JS_STATUS(j))),
+ job_read(pfdev, JS_HEAD_LO(j)),
+ job_read(pfdev, JS_TAIL_LO(j)));
+
+ drm_sched_fault(&pfdev->js->queue[j].sched);
+ }
+
+ if (status & JOB_INT_MASK_DONE(j)) {
+ panfrost_devfreq_record_transition(pfdev, j);
+ dma_fence_signal(pfdev->jobs[j]->done_fence);
+ }
+
+ status &= ~mask;
+ }
+
+ return IRQ_HANDLED;
+}
+
+int panfrost_job_init(struct panfrost_device *pfdev)
+{
+ struct panfrost_job_slot *js;
+ int ret, j, irq;
+
+ pfdev->js = js = devm_kzalloc(pfdev->dev, sizeof(*js), GFP_KERNEL);
+ if (!js)
+ return -ENOMEM;
+
+ spin_lock_init(&js->job_lock);
+
+ irq = platform_get_irq_byname(to_platform_device(pfdev->dev), "job");
+ if (irq <= 0)
+ return -ENODEV;
+
+ ret = devm_request_irq(pfdev->dev, irq, panfrost_job_irq_handler,
+ IRQF_SHARED, "job", pfdev);
+ if (ret) {
+ dev_err(pfdev->dev, "failed to request job irq");
+ return ret;
+ }
+
+ for (j = 0; j < NUM_JOB_SLOTS; j++) {
+ js->queue[j].fence_context = dma_fence_context_alloc(1);
+
+ ret = drm_sched_init(&js->queue[j].sched,
+ &panfrost_sched_ops,
+ 1, 0, msecs_to_jiffies(500),
+ "pan_js");
+ if (ret) {
+ dev_err(pfdev->dev, "Failed to create scheduler: %d.", ret);
+ goto err_sched;
+ }
+ }
+
+ panfrost_job_enable_interrupts(pfdev);
+
+ return 0;
+
+err_sched:
+ for (j--; j >= 0; j--)
+ drm_sched_fini(&js->queue[j].sched);
+
+ return ret;
+}
+
+void panfrost_job_fini(struct panfrost_device *pfdev)
+{
+ struct panfrost_job_slot *js = pfdev->js;
+ int j;
+
+ job_write(pfdev, JOB_INT_MASK, 0);
+
+ for (j = 0; j < NUM_JOB_SLOTS; j++)
+ drm_sched_fini(&js->queue[j].sched);
+
+}
+
+int panfrost_job_open(struct panfrost_file_priv *panfrost_priv)
+{
+ struct panfrost_device *pfdev = panfrost_priv->pfdev;
+ struct panfrost_job_slot *js = pfdev->js;
+ struct drm_sched_rq *rq;
+ int ret, i;
+
+ for (i = 0; i < NUM_JOB_SLOTS; i++) {
+ rq = &js->queue[i].sched.sched_rq[DRM_SCHED_PRIORITY_NORMAL];
+ ret = drm_sched_entity_init(&panfrost_priv->sched_entity[i], &rq, 1, NULL);
+ if (WARN_ON(ret))
+ return ret;
+ }
+ return 0;
+}
+
+void panfrost_job_close(struct panfrost_file_priv *panfrost_priv)
+{
+ int i;
+
+ for (i = 0; i < NUM_JOB_SLOTS; i++)
+ drm_sched_entity_destroy(&panfrost_priv->sched_entity[i]);
+}
+
+int panfrost_job_is_idle(struct panfrost_device *pfdev)
+{
+ struct panfrost_job_slot *js = pfdev->js;
+ int i;
+
+ for (i = 0; i < NUM_JOB_SLOTS; i++) {
+ /* If there are any jobs in the HW queue, we're not idle */
+ if (atomic_read(&js->queue[i].sched.hw_rq_count))
+ return false;
+
+ /* Check whether the hardware is idle */
+ if (pfdev->devfreq.slot[i].busy)
+ return false;
+ }
+
+ return true;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2019 Collabora ltd. */
+
+#ifndef __PANFROST_JOB_H__
+#define __PANFROST_JOB_H__
+
+#include <uapi/drm/panfrost_drm.h>
+#include <drm/gpu_scheduler.h>
+
+struct panfrost_device;
+struct panfrost_gem_object;
+struct panfrost_file_priv;
+
+struct panfrost_job {
+ struct drm_sched_job base;
+
+ struct kref refcount;
+
+ struct panfrost_device *pfdev;
+ struct panfrost_file_priv *file_priv;
+
+ /* Optional fences userspace can pass in for the job to depend on. */
+ struct dma_fence **in_fences;
+ u32 in_fence_count;
+
+ /* Fence to be signaled by IRQ handler when the job is complete. */
+ struct dma_fence *done_fence;
+
+ __u64 jc;
+ __u32 requirements;
+ __u32 flush_id;
+
+ /* Exclusive fences we have taken from the BOs to wait for */
+ struct dma_fence **implicit_fences;
+ struct drm_gem_object **bos;
+ u32 bo_count;
+
+ /* Fence to be signaled by drm-sched once its done with the job */
+ struct dma_fence *render_done_fence;
+};
+
+int panfrost_job_init(struct panfrost_device *pfdev);
+void panfrost_job_fini(struct panfrost_device *pfdev);
+int panfrost_job_open(struct panfrost_file_priv *panfrost_priv);
+void panfrost_job_close(struct panfrost_file_priv *panfrost_priv);
+int panfrost_job_push(struct panfrost_job *job);
+void panfrost_job_put(struct panfrost_job *job);
+void panfrost_job_enable_interrupts(struct panfrost_device *pfdev);
+int panfrost_job_is_idle(struct panfrost_device *pfdev);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
+#include <linux/bitfield.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iopoll.h>
+#include <linux/io-pgtable.h>
+#include <linux/iommu.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/sizes.h>
+
+#include "panfrost_device.h"
+#include "panfrost_mmu.h"
+#include "panfrost_gem.h"
+#include "panfrost_features.h"
+#include "panfrost_regs.h"
+
+#define mmu_write(dev, reg, data) writel(data, dev->iomem + reg)
+#define mmu_read(dev, reg) readl(dev->iomem + reg)
+
+struct panfrost_mmu {
+ struct io_pgtable_cfg pgtbl_cfg;
+ struct io_pgtable_ops *pgtbl_ops;
+ struct mutex lock;
+};
+
+static int wait_ready(struct panfrost_device *pfdev, u32 as_nr)
+{
+ int ret;
+ u32 val;
+
+ /* Wait for the MMU status to indicate there is no active command, in
+ * case one is pending. */
+ ret = readl_relaxed_poll_timeout_atomic(pfdev->iomem + AS_STATUS(as_nr),
+ val, !(val & AS_STATUS_AS_ACTIVE), 10, 1000);
+
+ if (ret)
+ dev_err(pfdev->dev, "AS_ACTIVE bit stuck\n");
+
+ return ret;
+}
+
+static int write_cmd(struct panfrost_device *pfdev, u32 as_nr, u32 cmd)
+{
+ int status;
+
+ /* write AS_COMMAND when MMU is ready to accept another command */
+ status = wait_ready(pfdev, as_nr);
+ if (!status)
+ mmu_write(pfdev, AS_COMMAND(as_nr), cmd);
+
+ return status;
+}
+
+static void lock_region(struct panfrost_device *pfdev, u32 as_nr,
+ u64 iova, size_t size)
+{
+ u8 region_width;
+ u64 region = iova & PAGE_MASK;
+ /*
+ * fls returns:
+ * 1 .. 32
+ *
+ * 10 + fls(num_pages)
+ * results in the range (11 .. 42)
+ */
+
+ size = round_up(size, PAGE_SIZE);
+
+ region_width = 10 + fls(size >> PAGE_SHIFT);
+ if ((size >> PAGE_SHIFT) != (1ul << (region_width - 11))) {
+ /* not pow2, so must go up to the next pow2 */
+ region_width += 1;
+ }
+ region |= region_width;
+
+ /* Lock the region that needs to be updated */
+ mmu_write(pfdev, AS_LOCKADDR_LO(as_nr), region & 0xFFFFFFFFUL);
+ mmu_write(pfdev, AS_LOCKADDR_HI(as_nr), (region >> 32) & 0xFFFFFFFFUL);
+ write_cmd(pfdev, as_nr, AS_COMMAND_LOCK);
+}
+
+
+static int mmu_hw_do_operation(struct panfrost_device *pfdev, u32 as_nr,
+ u64 iova, size_t size, u32 op)
+{
+ unsigned long flags;
+ int ret;
+
+ spin_lock_irqsave(&pfdev->hwaccess_lock, flags);
+
+ if (op != AS_COMMAND_UNLOCK)
+ lock_region(pfdev, as_nr, iova, size);
+
+ /* Run the MMU operation */
+ write_cmd(pfdev, as_nr, op);
+
+ /* Wait for the flush to complete */
+ ret = wait_ready(pfdev, as_nr);
+
+ spin_unlock_irqrestore(&pfdev->hwaccess_lock, flags);
+
+ return ret;
+}
+
+void panfrost_mmu_enable(struct panfrost_device *pfdev, u32 as_nr)
+{
+ struct io_pgtable_cfg *cfg = &pfdev->mmu->pgtbl_cfg;
+ u64 transtab = cfg->arm_mali_lpae_cfg.transtab;
+ u64 memattr = cfg->arm_mali_lpae_cfg.memattr;
+
+ mmu_write(pfdev, MMU_INT_CLEAR, ~0);
+ mmu_write(pfdev, MMU_INT_MASK, ~0);
+
+ mmu_write(pfdev, AS_TRANSTAB_LO(as_nr), transtab & 0xffffffffUL);
+ mmu_write(pfdev, AS_TRANSTAB_HI(as_nr), transtab >> 32);
+
+ /* Need to revisit mem attrs.
+ * NC is the default, Mali driver is inner WT.
+ */
+ mmu_write(pfdev, AS_MEMATTR_LO(as_nr), memattr & 0xffffffffUL);
+ mmu_write(pfdev, AS_MEMATTR_HI(as_nr), memattr >> 32);
+
+ write_cmd(pfdev, as_nr, AS_COMMAND_UPDATE);
+}
+
+static void mmu_disable(struct panfrost_device *pfdev, u32 as_nr)
+{
+ mmu_write(pfdev, AS_TRANSTAB_LO(as_nr), 0);
+ mmu_write(pfdev, AS_TRANSTAB_HI(as_nr), 0);
+
+ mmu_write(pfdev, AS_MEMATTR_LO(as_nr), 0);
+ mmu_write(pfdev, AS_MEMATTR_HI(as_nr), 0);
+
+ write_cmd(pfdev, as_nr, AS_COMMAND_UPDATE);
+}
+
+static size_t get_pgsize(u64 addr, size_t size)
+{
+ if (addr & (SZ_2M - 1) || size < SZ_2M)
+ return SZ_4K;
+
+ return SZ_2M;
+}
+
+int panfrost_mmu_map(struct panfrost_gem_object *bo)
+{
+ struct drm_gem_object *obj = &bo->base.base;
+ struct panfrost_device *pfdev = to_panfrost_device(obj->dev);
+ struct io_pgtable_ops *ops = pfdev->mmu->pgtbl_ops;
+ u64 iova = bo->node.start << PAGE_SHIFT;
+ unsigned int count;
+ struct scatterlist *sgl;
+ struct sg_table *sgt;
+ int ret;
+
+ sgt = drm_gem_shmem_get_pages_sgt(obj);
+ if (WARN_ON(IS_ERR(sgt)))
+ return PTR_ERR(sgt);
+
+ ret = pm_runtime_get_sync(pfdev->dev);
+ if (ret < 0)
+ return ret;
+
+ mutex_lock(&pfdev->mmu->lock);
+
+ for_each_sg(sgt->sgl, sgl, sgt->nents, count) {
+ unsigned long paddr = sg_dma_address(sgl);
+ size_t len = sg_dma_len(sgl);
+
+ dev_dbg(pfdev->dev, "map: iova=%llx, paddr=%lx, len=%zx", iova, paddr, len);
+
+ while (len) {
+ size_t pgsize = get_pgsize(iova | paddr, len);
+
+ ops->map(ops, iova, paddr, pgsize, IOMMU_WRITE | IOMMU_READ);
+ iova += pgsize;
+ paddr += pgsize;
+ len -= pgsize;
+ }
+ }
+
+ mmu_hw_do_operation(pfdev, 0, bo->node.start << PAGE_SHIFT,
+ bo->node.size << PAGE_SHIFT, AS_COMMAND_FLUSH_PT);
+
+ mutex_unlock(&pfdev->mmu->lock);
+
+ pm_runtime_mark_last_busy(pfdev->dev);
+ pm_runtime_put_autosuspend(pfdev->dev);
+
+ return 0;
+}
+
+void panfrost_mmu_unmap(struct panfrost_gem_object *bo)
+{
+ struct drm_gem_object *obj = &bo->base.base;
+ struct panfrost_device *pfdev = to_panfrost_device(obj->dev);
+ struct io_pgtable_ops *ops = pfdev->mmu->pgtbl_ops;
+ u64 iova = bo->node.start << PAGE_SHIFT;
+ size_t len = bo->node.size << PAGE_SHIFT;
+ size_t unmapped_len = 0;
+ int ret;
+
+ dev_dbg(pfdev->dev, "unmap: iova=%llx, len=%zx", iova, len);
+
+ ret = pm_runtime_get_sync(pfdev->dev);
+ if (ret < 0)
+ return;
+
+ mutex_lock(&pfdev->mmu->lock);
+
+ while (unmapped_len < len) {
+ size_t unmapped_page;
+ size_t pgsize = get_pgsize(iova, len - unmapped_len);
+
+ unmapped_page = ops->unmap(ops, iova, pgsize);
+ if (!unmapped_page)
+ break;
+
+ iova += unmapped_page;
+ unmapped_len += unmapped_page;
+ }
+
+ mmu_hw_do_operation(pfdev, 0, bo->node.start << PAGE_SHIFT,
+ bo->node.size << PAGE_SHIFT, AS_COMMAND_FLUSH_PT);
+
+ mutex_unlock(&pfdev->mmu->lock);
+
+ pm_runtime_mark_last_busy(pfdev->dev);
+ pm_runtime_put_autosuspend(pfdev->dev);
+}
+
+static void mmu_tlb_inv_context_s1(void *cookie)
+{
+ struct panfrost_device *pfdev = cookie;
+
+ mmu_hw_do_operation(pfdev, 0, 0, ~0UL, AS_COMMAND_FLUSH_MEM);
+}
+
+static void mmu_tlb_inv_range_nosync(unsigned long iova, size_t size,
+ size_t granule, bool leaf, void *cookie)
+{}
+
+static void mmu_tlb_sync_context(void *cookie)
+{
+ //struct panfrost_device *pfdev = cookie;
+ // TODO: Wait 1000 GPU cycles for HW_ISSUE_6367/T60X
+}
+
+static const struct iommu_gather_ops mmu_tlb_ops = {
+ .tlb_flush_all = mmu_tlb_inv_context_s1,
+ .tlb_add_flush = mmu_tlb_inv_range_nosync,
+ .tlb_sync = mmu_tlb_sync_context,
+};
+
+static const char *access_type_name(struct panfrost_device *pfdev,
+ u32 fault_status)
+{
+ switch (fault_status & AS_FAULTSTATUS_ACCESS_TYPE_MASK) {
+ case AS_FAULTSTATUS_ACCESS_TYPE_ATOMIC:
+ if (panfrost_has_hw_feature(pfdev, HW_FEATURE_AARCH64_MMU))
+ return "ATOMIC";
+ else
+ return "UNKNOWN";
+ case AS_FAULTSTATUS_ACCESS_TYPE_READ:
+ return "READ";
+ case AS_FAULTSTATUS_ACCESS_TYPE_WRITE:
+ return "WRITE";
+ case AS_FAULTSTATUS_ACCESS_TYPE_EX:
+ return "EXECUTE";
+ default:
+ WARN_ON(1);
+ return NULL;
+ }
+}
+
+static irqreturn_t panfrost_mmu_irq_handler(int irq, void *data)
+{
+ struct panfrost_device *pfdev = data;
+ u32 status = mmu_read(pfdev, MMU_INT_STAT);
+ int i;
+
+ if (!status)
+ return IRQ_NONE;
+
+ dev_err(pfdev->dev, "mmu irq status=%x\n", status);
+
+ for (i = 0; status; i++) {
+ u32 mask = BIT(i) | BIT(i + 16);
+ u64 addr;
+ u32 fault_status;
+ u32 exception_type;
+ u32 access_type;
+ u32 source_id;
+
+ if (!(status & mask))
+ continue;
+
+ fault_status = mmu_read(pfdev, AS_FAULTSTATUS(i));
+ addr = mmu_read(pfdev, AS_FAULTADDRESS_LO(i));
+ addr |= (u64)mmu_read(pfdev, AS_FAULTADDRESS_HI(i)) << 32;
+
+ /* decode the fault status */
+ exception_type = fault_status & 0xFF;
+ access_type = (fault_status >> 8) & 0x3;
+ source_id = (fault_status >> 16);
+
+ /* terminal fault, print info about the fault */
+ dev_err(pfdev->dev,
+ "Unhandled Page fault in AS%d at VA 0x%016llX\n"
+ "Reason: %s\n"
+ "raw fault status: 0x%X\n"
+ "decoded fault status: %s\n"
+ "exception type 0x%X: %s\n"
+ "access type 0x%X: %s\n"
+ "source id 0x%X\n",
+ i, addr,
+ "TODO",
+ fault_status,
+ (fault_status & (1 << 10) ? "DECODER FAULT" : "SLAVE FAULT"),
+ exception_type, panfrost_exception_name(pfdev, exception_type),
+ access_type, access_type_name(pfdev, fault_status),
+ source_id);
+
+ mmu_write(pfdev, MMU_INT_CLEAR, mask);
+
+ status &= ~mask;
+ }
+
+ return IRQ_HANDLED;
+};
+
+int panfrost_mmu_init(struct panfrost_device *pfdev)
+{
+ struct io_pgtable_ops *pgtbl_ops;
+ int err, irq;
+
+ pfdev->mmu = devm_kzalloc(pfdev->dev, sizeof(*pfdev->mmu), GFP_KERNEL);
+ if (!pfdev->mmu)
+ return -ENOMEM;
+
+ mutex_init(&pfdev->mmu->lock);
+
+ irq = platform_get_irq_byname(to_platform_device(pfdev->dev), "mmu");
+ if (irq <= 0)
+ return -ENODEV;
+
+ err = devm_request_irq(pfdev->dev, irq, panfrost_mmu_irq_handler,
+ IRQF_SHARED, "mmu", pfdev);
+
+ if (err) {
+ dev_err(pfdev->dev, "failed to request mmu irq");
+ return err;
+ }
+ mmu_write(pfdev, MMU_INT_CLEAR, ~0);
+ mmu_write(pfdev, MMU_INT_MASK, ~0);
+
+ pfdev->mmu->pgtbl_cfg = (struct io_pgtable_cfg) {
+ .pgsize_bitmap = SZ_4K | SZ_2M,
+ .ias = FIELD_GET(0xff, pfdev->features.mmu_features),
+ .oas = FIELD_GET(0xff00, pfdev->features.mmu_features),
+ .tlb = &mmu_tlb_ops,
+ .iommu_dev = pfdev->dev,
+ };
+
+ pgtbl_ops = alloc_io_pgtable_ops(ARM_MALI_LPAE, &pfdev->mmu->pgtbl_cfg,
+ pfdev);
+ if (!pgtbl_ops)
+ return -ENOMEM;
+
+ pfdev->mmu->pgtbl_ops = pgtbl_ops;
+
+ panfrost_mmu_enable(pfdev, 0);
+
+ return 0;
+}
+
+void panfrost_mmu_fini(struct panfrost_device *pfdev)
+{
+ mmu_write(pfdev, MMU_INT_MASK, 0);
+ mmu_disable(pfdev, 0);
+
+ free_io_pgtable_ops(pfdev->mmu->pgtbl_ops);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
+
+#ifndef __PANFROST_MMU_H__
+#define __PANFROST_MMU_H__
+
+struct panfrost_gem_object;
+
+int panfrost_mmu_map(struct panfrost_gem_object *bo);
+void panfrost_mmu_unmap(struct panfrost_gem_object *bo);
+
+int panfrost_mmu_init(struct panfrost_device *pfdev);
+void panfrost_mmu_fini(struct panfrost_device *pfdev);
+
+void panfrost_mmu_enable(struct panfrost_device *pfdev, u32 as_nr);
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright 2018 Marty E. Plummer <hanetzer@startmail.com> */
+/* Copyright 2019 Linaro, Ltd, Rob Herring <robh@kernel.org> */
+/*
+ * Register definitions based on mali_midg_regmap.h
+ * (C) COPYRIGHT 2010-2018 ARM Limited. All rights reserved.
+ */
+#ifndef __PANFROST_REGS_H__
+#define __PANFROST_REGS_H__
+
+#define GPU_ID 0x00
+#define GPU_L2_FEATURES 0x004 /* (RO) Level 2 cache features */
+#define GPU_CORE_FEATURES 0x008 /* (RO) Shader Core Features */
+#define GPU_TILER_FEATURES 0x00C /* (RO) Tiler Features */
+#define GPU_MEM_FEATURES 0x010 /* (RO) Memory system features */
+#define GROUPS_L2_COHERENT BIT(0) /* Cores groups are l2 coherent */
+
+#define GPU_MMU_FEATURES 0x014 /* (RO) MMU features */
+#define GPU_AS_PRESENT 0x018 /* (RO) Address space slots present */
+#define GPU_JS_PRESENT 0x01C /* (RO) Job slots present */
+
+#define GPU_INT_RAWSTAT 0x20
+#define GPU_INT_CLEAR 0x24
+#define GPU_INT_MASK 0x28
+#define GPU_INT_STAT 0x2c
+#define GPU_IRQ_FAULT BIT(0)
+#define GPU_IRQ_MULTIPLE_FAULT BIT(7)
+#define GPU_IRQ_RESET_COMPLETED BIT(8)
+#define GPU_IRQ_POWER_CHANGED BIT(9)
+#define GPU_IRQ_POWER_CHANGED_ALL BIT(10)
+#define GPU_IRQ_PERFCNT_SAMPLE_COMPLETED BIT(16)
+#define GPU_IRQ_CLEAN_CACHES_COMPLETED BIT(17)
+#define GPU_IRQ_MASK_ALL \
+ (GPU_IRQ_FAULT |\
+ GPU_IRQ_MULTIPLE_FAULT |\
+ GPU_IRQ_RESET_COMPLETED |\
+ GPU_IRQ_POWER_CHANGED |\
+ GPU_IRQ_POWER_CHANGED_ALL |\
+ GPU_IRQ_PERFCNT_SAMPLE_COMPLETED |\
+ GPU_IRQ_CLEAN_CACHES_COMPLETED)
+#define GPU_IRQ_MASK_ERROR \
+ ( \
+ GPU_IRQ_FAULT |\
+ GPU_IRQ_MULTIPLE_FAULT)
+#define GPU_CMD 0x30
+#define GPU_CMD_SOFT_RESET 0x01
+#define GPU_STATUS 0x34
+#define GPU_LATEST_FLUSH_ID 0x38
+#define GPU_FAULT_STATUS 0x3C
+#define GPU_FAULT_ADDRESS_LO 0x40
+#define GPU_FAULT_ADDRESS_HI 0x44
+
+#define GPU_THREAD_MAX_THREADS 0x0A0 /* (RO) Maximum number of threads per core */
+#define GPU_THREAD_MAX_WORKGROUP_SIZE 0x0A4 /* (RO) Maximum workgroup size */
+#define GPU_THREAD_MAX_BARRIER_SIZE 0x0A8 /* (RO) Maximum threads waiting at a barrier */
+#define GPU_THREAD_FEATURES 0x0AC /* (RO) Thread features */
+#define GPU_THREAD_TLS_ALLOC 0x310 /* (RO) Number of threads per core that
+ * TLS must be allocated for */
+
+#define GPU_TEXTURE_FEATURES(n) (0x0B0 + ((n) * 4))
+#define GPU_JS_FEATURES(n) (0x0C0 + ((n) * 4))
+
+#define GPU_SHADER_PRESENT_LO 0x100 /* (RO) Shader core present bitmap, low word */
+#define GPU_SHADER_PRESENT_HI 0x104 /* (RO) Shader core present bitmap, high word */
+#define GPU_TILER_PRESENT_LO 0x110 /* (RO) Tiler core present bitmap, low word */
+#define GPU_TILER_PRESENT_HI 0x114 /* (RO) Tiler core present bitmap, high word */
+
+#define GPU_L2_PRESENT_LO 0x120 /* (RO) Level 2 cache present bitmap, low word */
+#define GPU_L2_PRESENT_HI 0x124 /* (RO) Level 2 cache present bitmap, high word */
+
+#define GPU_COHERENCY_FEATURES 0x300 /* (RO) Coherency features present */
+#define COHERENCY_ACE_LITE BIT(0)
+#define COHERENCY_ACE BIT(1)
+
+#define GPU_STACK_PRESENT_LO 0xE00 /* (RO) Core stack present bitmap, low word */
+#define GPU_STACK_PRESENT_HI 0xE04 /* (RO) Core stack present bitmap, high word */
+
+#define SHADER_READY_LO 0x140 /* (RO) Shader core ready bitmap, low word */
+#define SHADER_READY_HI 0x144 /* (RO) Shader core ready bitmap, high word */
+
+#define TILER_READY_LO 0x150 /* (RO) Tiler core ready bitmap, low word */
+#define TILER_READY_HI 0x154 /* (RO) Tiler core ready bitmap, high word */
+
+#define L2_READY_LO 0x160 /* (RO) Level 2 cache ready bitmap, low word */
+#define L2_READY_HI 0x164 /* (RO) Level 2 cache ready bitmap, high word */
+
+#define STACK_READY_LO 0xE10 /* (RO) Core stack ready bitmap, low word */
+#define STACK_READY_HI 0xE14 /* (RO) Core stack ready bitmap, high word */
+
+
+#define SHADER_PWRON_LO 0x180 /* (WO) Shader core power on bitmap, low word */
+#define SHADER_PWRON_HI 0x184 /* (WO) Shader core power on bitmap, high word */
+
+#define TILER_PWRON_LO 0x190 /* (WO) Tiler core power on bitmap, low word */
+#define TILER_PWRON_HI 0x194 /* (WO) Tiler core power on bitmap, high word */
+
+#define L2_PWRON_LO 0x1A0 /* (WO) Level 2 cache power on bitmap, low word */
+#define L2_PWRON_HI 0x1A4 /* (WO) Level 2 cache power on bitmap, high word */
+
+#define STACK_PWRON_LO 0xE20 /* (RO) Core stack power on bitmap, low word */
+#define STACK_PWRON_HI 0xE24 /* (RO) Core stack power on bitmap, high word */
+
+
+#define SHADER_PWROFF_LO 0x1C0 /* (WO) Shader core power off bitmap, low word */
+#define SHADER_PWROFF_HI 0x1C4 /* (WO) Shader core power off bitmap, high word */
+
+#define TILER_PWROFF_LO 0x1D0 /* (WO) Tiler core power off bitmap, low word */
+#define TILER_PWROFF_HI 0x1D4 /* (WO) Tiler core power off bitmap, high word */
+
+#define L2_PWROFF_LO 0x1E0 /* (WO) Level 2 cache power off bitmap, low word */
+#define L2_PWROFF_HI 0x1E4 /* (WO) Level 2 cache power off bitmap, high word */
+
+#define STACK_PWROFF_LO 0xE30 /* (RO) Core stack power off bitmap, low word */
+#define STACK_PWROFF_HI 0xE34 /* (RO) Core stack power off bitmap, high word */
+
+
+#define SHADER_PWRTRANS_LO 0x200 /* (RO) Shader core power transition bitmap, low word */
+#define SHADER_PWRTRANS_HI 0x204 /* (RO) Shader core power transition bitmap, high word */
+
+#define TILER_PWRTRANS_LO 0x210 /* (RO) Tiler core power transition bitmap, low word */
+#define TILER_PWRTRANS_HI 0x214 /* (RO) Tiler core power transition bitmap, high word */
+
+#define L2_PWRTRANS_LO 0x220 /* (RO) Level 2 cache power transition bitmap, low word */
+#define L2_PWRTRANS_HI 0x224 /* (RO) Level 2 cache power transition bitmap, high word */
+
+#define STACK_PWRTRANS_LO 0xE40 /* (RO) Core stack power transition bitmap, low word */
+#define STACK_PWRTRANS_HI 0xE44 /* (RO) Core stack power transition bitmap, high word */
+
+
+#define SHADER_PWRACTIVE_LO 0x240 /* (RO) Shader core active bitmap, low word */
+#define SHADER_PWRACTIVE_HI 0x244 /* (RO) Shader core active bitmap, high word */
+
+#define TILER_PWRACTIVE_LO 0x250 /* (RO) Tiler core active bitmap, low word */
+#define TILER_PWRACTIVE_HI 0x254 /* (RO) Tiler core active bitmap, high word */
+
+#define L2_PWRACTIVE_LO 0x260 /* (RO) Level 2 cache active bitmap, low word */
+#define L2_PWRACTIVE_HI 0x264 /* (RO) Level 2 cache active bitmap, high word */
+
+#define GPU_JM_CONFIG 0xF00 /* (RW) Job Manager configuration register (Implementation specific register) */
+#define GPU_SHADER_CONFIG 0xF04 /* (RW) Shader core configuration settings (Implementation specific register) */
+#define GPU_TILER_CONFIG 0xF08 /* (RW) Tiler core configuration settings (Implementation specific register) */
+#define GPU_L2_MMU_CONFIG 0xF0C /* (RW) Configuration of the L2 cache and MMU (Implementation specific register) */
+
+/* L2_MMU_CONFIG register */
+#define L2_MMU_CONFIG_ALLOW_SNOOP_DISPARITY_SHIFT 23
+#define L2_MMU_CONFIG_ALLOW_SNOOP_DISPARITY (0x1 << L2_MMU_CONFIG_ALLOW_SNOOP_DISPARITY_SHIFT)
+#define L2_MMU_CONFIG_LIMIT_EXTERNAL_READS_SHIFT 24
+#define L2_MMU_CONFIG_LIMIT_EXTERNAL_READS (0x3 << L2_MMU_CONFIG_LIMIT_EXTERNAL_READS_SHIFT)
+#define L2_MMU_CONFIG_LIMIT_EXTERNAL_READS_OCTANT (0x1 << L2_MMU_CONFIG_LIMIT_EXTERNAL_READS_SHIFT)
+#define L2_MMU_CONFIG_LIMIT_EXTERNAL_READS_QUARTER (0x2 << L2_MMU_CONFIG_LIMIT_EXTERNAL_READS_SHIFT)
+#define L2_MMU_CONFIG_LIMIT_EXTERNAL_READS_HALF (0x3 << L2_MMU_CONFIG_LIMIT_EXTERNAL_READS_SHIFT)
+
+#define L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES_SHIFT 26
+#define L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES (0x3 << L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES_SHIFT)
+#define L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES_OCTANT (0x1 << L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES_SHIFT)
+#define L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES_QUARTER (0x2 << L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES_SHIFT)
+#define L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES_HALF (0x3 << L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES_SHIFT)
+
+#define L2_MMU_CONFIG_3BIT_LIMIT_EXTERNAL_READS_SHIFT 12
+#define L2_MMU_CONFIG_3BIT_LIMIT_EXTERNAL_READS (0x7 << L2_MMU_CONFIG_LIMIT_EXTERNAL_READS_SHIFT)
+
+#define L2_MMU_CONFIG_3BIT_LIMIT_EXTERNAL_WRITES_SHIFT 15
+#define L2_MMU_CONFIG_3BIT_LIMIT_EXTERNAL_WRITES (0x7 << L2_MMU_CONFIG_LIMIT_EXTERNAL_WRITES_SHIFT)
+
+/* SHADER_CONFIG register */
+#define SC_ALT_COUNTERS BIT(3)
+#define SC_OVERRIDE_FWD_PIXEL_KILL BIT(4)
+#define SC_SDC_DISABLE_OQ_DISCARD BIT(6)
+#define SC_LS_ALLOW_ATTR_TYPES BIT(16)
+#define SC_LS_PAUSEBUFFER_DISABLE BIT(16)
+#define SC_TLS_HASH_ENABLE BIT(17)
+#define SC_LS_ATTR_CHECK_DISABLE BIT(18)
+#define SC_ENABLE_TEXGRD_FLAGS BIT(25)
+/* End SHADER_CONFIG register */
+
+/* TILER_CONFIG register */
+#define TC_CLOCK_GATE_OVERRIDE BIT(0)
+
+/* JM_CONFIG register */
+#define JM_TIMESTAMP_OVERRIDE BIT(0)
+#define JM_CLOCK_GATE_OVERRIDE BIT(1)
+#define JM_JOB_THROTTLE_ENABLE BIT(2)
+#define JM_JOB_THROTTLE_LIMIT_SHIFT 3
+#define JM_MAX_JOB_THROTTLE_LIMIT 0x3F
+#define JM_FORCE_COHERENCY_FEATURES_SHIFT 2
+#define JM_IDVS_GROUP_SIZE_SHIFT 16
+#define JM_MAX_IDVS_GROUP_SIZE 0x3F
+
+
+/* Job Control regs */
+#define JOB_INT_RAWSTAT 0x1000
+#define JOB_INT_CLEAR 0x1004
+#define JOB_INT_MASK 0x1008
+#define JOB_INT_STAT 0x100c
+#define JOB_INT_JS_STATE 0x1010
+#define JOB_INT_THROTTLE 0x1014
+
+#define MK_JS_MASK(j) (0x10001 << (j))
+#define JOB_INT_MASK_ERR(j) BIT((j) + 16)
+#define JOB_INT_MASK_DONE(j) BIT(j)
+
+#define JS_BASE 0x1800
+#define JS_HEAD_LO(n) (JS_BASE + ((n) * 0x80) + 0x00)
+#define JS_HEAD_HI(n) (JS_BASE + ((n) * 0x80) + 0x04)
+#define JS_TAIL_LO(n) (JS_BASE + ((n) * 0x80) + 0x08)
+#define JS_TAIL_HI(n) (JS_BASE + ((n) * 0x80) + 0x0c)
+#define JS_AFFINITY_LO(n) (JS_BASE + ((n) * 0x80) + 0x10)
+#define JS_AFFINITY_HI(n) (JS_BASE + ((n) * 0x80) + 0x14)
+#define JS_CONFIG(n) (JS_BASE + ((n) * 0x80) + 0x18)
+#define JS_XAFFINITY(n) (JS_BASE + ((n) * 0x80) + 0x1c)
+#define JS_COMMAND(n) (JS_BASE + ((n) * 0x80) + 0x20)
+#define JS_STATUS(n) (JS_BASE + ((n) * 0x80) + 0x24)
+#define JS_HEAD_NEXT_LO(n) (JS_BASE + ((n) * 0x80) + 0x40)
+#define JS_HEAD_NEXT_HI(n) (JS_BASE + ((n) * 0x80) + 0x44)
+#define JS_AFFINITY_NEXT_LO(n) (JS_BASE + ((n) * 0x80) + 0x50)
+#define JS_AFFINITY_NEXT_HI(n) (JS_BASE + ((n) * 0x80) + 0x54)
+#define JS_CONFIG_NEXT(n) (JS_BASE + ((n) * 0x80) + 0x58)
+#define JS_COMMAND_NEXT(n) (JS_BASE + ((n) * 0x80) + 0x60)
+#define JS_FLUSH_ID_NEXT(n) (JS_BASE + ((n) * 0x80) + 0x70)
+
+/* Possible values of JS_CONFIG and JS_CONFIG_NEXT registers */
+#define JS_CONFIG_START_FLUSH_CLEAN BIT(8)
+#define JS_CONFIG_START_FLUSH_CLEAN_INVALIDATE (3u << 8)
+#define JS_CONFIG_START_MMU BIT(10)
+#define JS_CONFIG_JOB_CHAIN_FLAG BIT(11)
+#define JS_CONFIG_END_FLUSH_CLEAN BIT(12)
+#define JS_CONFIG_END_FLUSH_CLEAN_INVALIDATE (3u << 12)
+#define JS_CONFIG_ENABLE_FLUSH_REDUCTION BIT(14)
+#define JS_CONFIG_DISABLE_DESCRIPTOR_WR_BK BIT(15)
+#define JS_CONFIG_THREAD_PRI(n) ((n) << 16)
+
+#define JS_COMMAND_NOP 0x00
+#define JS_COMMAND_START 0x01
+#define JS_COMMAND_SOFT_STOP 0x02 /* Gently stop processing a job chain */
+#define JS_COMMAND_HARD_STOP 0x03 /* Rudely stop processing a job chain */
+#define JS_COMMAND_SOFT_STOP_0 0x04 /* Execute SOFT_STOP if JOB_CHAIN_FLAG is 0 */
+#define JS_COMMAND_HARD_STOP_0 0x05 /* Execute HARD_STOP if JOB_CHAIN_FLAG is 0 */
+#define JS_COMMAND_SOFT_STOP_1 0x06 /* Execute SOFT_STOP if JOB_CHAIN_FLAG is 1 */
+#define JS_COMMAND_HARD_STOP_1 0x07 /* Execute HARD_STOP if JOB_CHAIN_FLAG is 1 */
+
+#define JS_STATUS_EVENT_ACTIVE 0x08
+
+
+/* MMU regs */
+#define MMU_INT_RAWSTAT 0x2000
+#define MMU_INT_CLEAR 0x2004
+#define MMU_INT_MASK 0x2008
+#define MMU_INT_STAT 0x200c
+
+/* AS_COMMAND register commands */
+#define AS_COMMAND_NOP 0x00 /* NOP Operation */
+#define AS_COMMAND_UPDATE 0x01 /* Broadcasts the values in AS_TRANSTAB and ASn_MEMATTR to all MMUs */
+#define AS_COMMAND_LOCK 0x02 /* Issue a lock region command to all MMUs */
+#define AS_COMMAND_UNLOCK 0x03 /* Issue a flush region command to all MMUs */
+#define AS_COMMAND_FLUSH 0x04 /* Flush all L2 caches then issue a flush region command to all MMUs
+ (deprecated - only for use with T60x) */
+#define AS_COMMAND_FLUSH_PT 0x04 /* Flush all L2 caches then issue a flush region command to all MMUs */
+#define AS_COMMAND_FLUSH_MEM 0x05 /* Wait for memory accesses to complete, flush all the L1s cache then
+ flush all L2 caches then issue a flush region command to all MMUs */
+
+#define MMU_AS(as) (0x2400 + ((as) << 6))
+
+#define AS_TRANSTAB_LO(as) (MMU_AS(as) + 0x00) /* (RW) Translation Table Base Address for address space n, low word */
+#define AS_TRANSTAB_HI(as) (MMU_AS(as) + 0x04) /* (RW) Translation Table Base Address for address space n, high word */
+#define AS_MEMATTR_LO(as) (MMU_AS(as) + 0x08) /* (RW) Memory attributes for address space n, low word. */
+#define AS_MEMATTR_HI(as) (MMU_AS(as) + 0x0C) /* (RW) Memory attributes for address space n, high word. */
+#define AS_LOCKADDR_LO(as) (MMU_AS(as) + 0x10) /* (RW) Lock region address for address space n, low word */
+#define AS_LOCKADDR_HI(as) (MMU_AS(as) + 0x14) /* (RW) Lock region address for address space n, high word */
+#define AS_COMMAND(as) (MMU_AS(as) + 0x18) /* (WO) MMU command register for address space n */
+#define AS_FAULTSTATUS(as) (MMU_AS(as) + 0x1C) /* (RO) MMU fault status register for address space n */
+#define AS_FAULTADDRESS_LO(as) (MMU_AS(as) + 0x20) /* (RO) Fault Address for address space n, low word */
+#define AS_FAULTADDRESS_HI(as) (MMU_AS(as) + 0x24) /* (RO) Fault Address for address space n, high word */
+#define AS_STATUS(as) (MMU_AS(as) + 0x28) /* (RO) Status flags for address space n */
+/* Additional Bifrost AS regsiters */
+#define AS_TRANSCFG_LO(as) (MMU_AS(as) + 0x30) /* (RW) Translation table configuration for address space n, low word */
+#define AS_TRANSCFG_HI(as) (MMU_AS(as) + 0x34) /* (RW) Translation table configuration for address space n, high word */
+#define AS_FAULTEXTRA_LO(as) (MMU_AS(as) + 0x38) /* (RO) Secondary fault address for address space n, low word */
+#define AS_FAULTEXTRA_HI(as) (MMU_AS(as) + 0x3C) /* (RO) Secondary fault address for address space n, high word */
+
+/*
+ * Begin LPAE MMU TRANSTAB register values
+ */
+#define AS_TRANSTAB_LPAE_ADDR_SPACE_MASK 0xfffffffffffff000
+#define AS_TRANSTAB_LPAE_ADRMODE_IDENTITY 0x2
+#define AS_TRANSTAB_LPAE_ADRMODE_TABLE 0x3
+#define AS_TRANSTAB_LPAE_ADRMODE_MASK 0x3
+#define AS_TRANSTAB_LPAE_READ_INNER BIT(2)
+#define AS_TRANSTAB_LPAE_SHARE_OUTER BIT(4)
+
+#define AS_STATUS_AS_ACTIVE 0x01
+
+#define AS_FAULTSTATUS_ACCESS_TYPE_MASK (0x3 << 8)
+#define AS_FAULTSTATUS_ACCESS_TYPE_ATOMIC (0x0 << 8)
+#define AS_FAULTSTATUS_ACCESS_TYPE_EX (0x1 << 8)
+#define AS_FAULTSTATUS_ACCESS_TYPE_READ (0x2 << 8)
+#define AS_FAULTSTATUS_ACCESS_TYPE_WRITE (0x3 << 8)
+
+#endif
tim2 |= TIM2_IOE;
if (connector->display_info.bus_flags &
- DRM_BUS_FLAG_PIXDATA_NEGEDGE)
+ DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE)
tim2 |= TIM2_IPC;
}
ret = vexpress_muxfpga_init();
if (ret) {
dev_err(dev, "unable to initialize muxfpga driver\n");
+ of_node_put(np);
return ret;
}
pdev = of_find_device_by_node(np);
if (!pdev) {
dev_err(dev, "can't find the sysreg device, deferring\n");
+ of_node_put(np);
return -EPROBE_DEFER;
}
map = dev_get_drvdata(&pdev->dev);
if (!map) {
dev_err(dev, "sysreg has not yet probed\n");
platform_device_put(pdev);
+ of_node_put(np);
return -EPROBE_DEFER;
}
} else {
map = syscon_node_to_regmap(np);
}
+ of_node_put(np);
if (IS_ERR(map)) {
dev_err(dev, "no Versatile syscon regmap\n");
{
struct qxl_device *qdev = plane->dev->dev_private;
struct qxl_bo *bo = gem_to_qxl_bo(plane->state->fb->obj[0]);
- struct qxl_bo *bo_old, *primary;
+ struct qxl_bo *primary;
struct drm_clip_rect norect = {
.x1 = 0,
.y1 = 0,
};
uint32_t dumb_shadow_offset = 0;
- if (old_state->fb) {
- bo_old = gem_to_qxl_bo(old_state->fb->obj[0]);
- } else {
- bo_old = NULL;
- }
-
primary = bo->shadow ? bo->shadow : bo;
if (!primary->is_primary) {
extern int qxl_num_crtc;
extern int qxl_max_ioctls;
-#define DRM_FILE_OFFSET 0x100000000ULL
-#define DRM_FILE_PAGE_OFFSET (DRM_FILE_OFFSET >> PAGE_SHIFT)
-
#define QXL_INTERRUPT_MASK (\
QXL_INTERRUPT_DISPLAY |\
QXL_INTERRUPT_CURSOR |\
int qxl_mmap(struct file *filp, struct vm_area_struct *vma)
{
- struct drm_file *file_priv;
- struct qxl_device *qdev;
int r;
+ struct drm_file *file_priv = filp->private_data;
+ struct qxl_device *qdev = file_priv->minor->dev->dev_private;
- if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET))
- return -EINVAL;
-
- file_priv = filp->private_data;
- qdev = file_priv->minor->dev->dev_private;
if (qdev == NULL) {
DRM_ERROR(
"filp->private_data->minor->dev->dev_private == NULL\n");
r = ttm_bo_device_init(&qdev->mman.bdev,
&qxl_bo_driver,
qdev->ddev.anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET, 0);
+ false);
if (r) {
DRM_ERROR("failed initializing buffer object driver(%d).\n", r);
return r;
pci_set_consistent_dma_mask(rdev->pdev, DMA_BIT_MASK(32));
pr_warn("radeon: No coherent DMA available\n");
}
- rdev->need_swiotlb = drm_get_max_iomem() > ((u64)1 << dma_bits);
+ rdev->need_swiotlb = drm_need_swiotlb(dma_bits);
/* Registers mapping */
/* TODO: block userspace mapping of io register */
#include <linux/platform_device.h>
#include <drm/drm_legacy.h>
-#include <drm/ati_pcigart.h>
#include "radeon_family.h"
/* General customization:
* the helper contains a pointer to radeon framebuffer baseclass.
*/
struct radeon_fbdev {
- struct drm_fb_helper helper;
+ struct drm_fb_helper helper; /* must be first */
struct drm_framebuffer fb;
struct radeon_device *rdev;
};
goto out;
}
- info->par = rfbdev;
+ /* radeon resume is fragile and needs a vt switch to help it along */
+ info->skip_vt_switch = false;
ret = radeon_framebuffer_init(rdev->ddev, &rfbdev->fb, &mode_cmd, gobj);
if (ret) {
memset_io(rbo->kptr, 0x0, radeon_bo_size(rbo));
- strcpy(info->fix.id, "radeondrmfb");
-
- drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
-
info->fbops = &radeonfb_ops;
tmp = radeon_bo_gpu_offset(rbo) - rdev->mc.vram_start;
info->screen_base = rbo->kptr;
info->screen_size = radeon_bo_size(rbo);
- drm_fb_helper_fill_var(info, &rfbdev->helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(info, &rfbdev->helper, sizes);
/* setup aperture base/size for vesafb takeover */
info->apertures->ranges[0].base = rdev->ddev->mode_config.fb_base;
#include "radeon_reg.h"
#include "radeon.h"
-#define DRM_FILE_PAGE_OFFSET (0x100000000ULL >> PAGE_SHIFT)
-
static int radeon_ttm_debugfs_init(struct radeon_device *rdev);
static void radeon_ttm_debugfs_fini(struct radeon_device *rdev);
struct ttm_mem_reg *new_mem)
{
struct ttm_operation_ctx ctx = { interruptible, no_wait_gpu };
- struct radeon_device *rdev;
struct ttm_mem_reg *old_mem = &bo->mem;
struct ttm_mem_reg tmp_mem;
struct ttm_place placements;
struct ttm_placement placement;
int r;
- rdev = radeon_get_rdev(bo->bdev);
tmp_mem = *new_mem;
tmp_mem.mm_node = NULL;
placement.num_placement = 1;
struct ttm_mem_reg *new_mem)
{
struct ttm_operation_ctx ctx = { interruptible, no_wait_gpu };
- struct radeon_device *rdev;
struct ttm_mem_reg *old_mem = &bo->mem;
struct ttm_mem_reg tmp_mem;
struct ttm_placement placement;
struct ttm_place placements;
int r;
- rdev = radeon_get_rdev(bo->bdev);
tmp_mem = *new_mem;
tmp_mem.mm_node = NULL;
placement.num_placement = 1;
r = ttm_bo_device_init(&rdev->mman.bdev,
&radeon_bo_driver,
rdev->ddev->anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET,
rdev->need_dma32);
if (r) {
DRM_ERROR("failed initializing buffer object driver(%d).\n", r);
int radeon_mmap(struct file *filp, struct vm_area_struct *vma)
{
- struct drm_file *file_priv;
- struct radeon_device *rdev;
int r;
+ struct drm_file *file_priv = filp->private_data;
+ struct radeon_device *rdev = file_priv->minor->dev->dev_private;
- if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET)) {
- return -EINVAL;
- }
-
- file_priv = filp->private_data;
- rdev = file_priv->minor->dev->dev_private;
if (rdev == NULL) {
return -EINVAL;
}
depends on VIDEO_RENESAS_VSP1=y || (VIDEO_RENESAS_VSP1 && DRM_RCAR_DU=m)
help
Enable support to expose the R-Car VSP Compositor as KMS planes.
+
+config DRM_RCAR_WRITEBACK
+ bool
+ default y if ARM64
rcar_du_encoder.o \
rcar_du_group.o \
rcar_du_kms.o \
- rcar_du_plane.o
+ rcar_du_plane.o \
rcar-du-drm-$(CONFIG_DRM_RCAR_LVDS) += rcar_du_of.o \
rcar_du_of_lvds_r8a7790.dtb.o \
rcar_du_of_lvds_r8a7795.dtb.o \
rcar_du_of_lvds_r8a7796.dtb.o
rcar-du-drm-$(CONFIG_DRM_RCAR_VSP) += rcar_du_vsp.o
+rcar-du-drm-$(CONFIG_DRM_RCAR_WRITEBACK) += rcar_du_writeback.o
obj-$(CONFIG_DRM_RCAR_DU) += rcar-du-drm.o
obj-$(CONFIG_DRM_RCAR_DW_HDMI) += rcar_dw_hdmi.o
static u32 rcar_du_crtc_read(struct rcar_du_crtc *rcrtc, u32 reg)
{
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
return rcar_du_read(rcdu, rcrtc->mmio_offset + reg);
}
static void rcar_du_crtc_write(struct rcar_du_crtc *rcrtc, u32 reg, u32 data)
{
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
rcar_du_write(rcdu, rcrtc->mmio_offset + reg, data);
}
static void rcar_du_crtc_clr(struct rcar_du_crtc *rcrtc, u32 reg, u32 clr)
{
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
rcar_du_write(rcdu, rcrtc->mmio_offset + reg,
rcar_du_read(rcdu, rcrtc->mmio_offset + reg) & ~clr);
static void rcar_du_crtc_set(struct rcar_du_crtc *rcrtc, u32 reg, u32 set)
{
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
rcar_du_write(rcdu, rcrtc->mmio_offset + reg,
rcar_du_read(rcdu, rcrtc->mmio_offset + reg) | set);
void rcar_du_crtc_dsysr_clr_set(struct rcar_du_crtc *rcrtc, u32 clr, u32 set)
{
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
rcrtc->dsysr = (rcrtc->dsysr & ~clr) | set;
rcar_du_write(rcdu, rcrtc->mmio_offset + DSYSR, rcrtc->dsysr);
}
done:
- dev_dbg(rcrtc->group->dev->dev,
+ dev_dbg(rcrtc->dev->dev,
"output:%u, fdpll:%u, n:%u, m:%u, diff:%lu\n",
- dpll->output, dpll->fdpll, dpll->n, dpll->m,
- best_diff);
+ dpll->output, dpll->fdpll, dpll->n, dpll->m, best_diff);
}
struct du_clk_params {
static void rcar_du_crtc_set_display_timing(struct rcar_du_crtc *rcrtc)
{
const struct drm_display_mode *mode = &rcrtc->crtc.state->adjusted_mode;
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
unsigned long mode_clock = mode->clock * 1000;
u32 dsmr;
u32 escr;
rcar_du_escr_divider(rcrtc->extclock, mode_clock,
ESCR_DCLKSEL_DCLKIN, ¶ms);
- dev_dbg(rcrtc->group->dev->dev, "mode clock %lu %s rate %lu\n",
+ dev_dbg(rcrtc->dev->dev, "mode clock %lu %s rate %lu\n",
mode_clock, params.clk == rcrtc->clock ? "cpg" : "ext",
params.rate);
escr = params.escr;
}
- dev_dbg(rcrtc->group->dev->dev, "%s: ESCR 0x%08x\n", __func__, escr);
+ dev_dbg(rcrtc->dev->dev, "%s: ESCR 0x%08x\n", __func__, escr);
rcar_du_crtc_write(rcrtc, rcrtc->index % 2 ? ESCR13 : ESCR02, escr);
rcar_du_crtc_write(rcrtc, rcrtc->index % 2 ? OTAR13 : OTAR02, 0);
static void rcar_du_crtc_update_planes(struct rcar_du_crtc *rcrtc)
{
struct rcar_du_plane *planes[RCAR_DU_NUM_HW_PLANES];
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
unsigned int num_planes = 0;
unsigned int dptsr_planes;
unsigned int hwplanes = 0;
static void rcar_du_crtc_wait_page_flip(struct rcar_du_crtc *rcrtc)
{
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
if (wait_event_timeout(rcrtc->flip_wait,
!rcar_du_crtc_page_flip_pending(rcrtc),
rcar_du_group_write(rcrtc->group, rcrtc->index % 2 ? DS2PR : DS1PR, 0);
/* Enable the VSP compositor. */
- if (rcar_du_has(rcrtc->group->dev, RCAR_DU_FEATURE_VSP1_SOURCE))
+ if (rcar_du_has(rcrtc->dev, RCAR_DU_FEATURE_VSP1_SOURCE))
rcar_du_vsp_enable(rcrtc);
/* Turn vertical blanking interrupt reporting on. */
static void rcar_du_crtc_disable_planes(struct rcar_du_crtc *rcrtc)
{
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
struct drm_crtc *crtc = &rcrtc->crtc;
u32 status;
drm_crtc_vblank_off(crtc);
/* Disable the VSP compositor. */
- if (rcar_du_has(rcrtc->group->dev, RCAR_DU_FEATURE_VSP1_SOURCE))
+ if (rcar_du_has(rcrtc->dev, RCAR_DU_FEATURE_VSP1_SOURCE))
rcar_du_vsp_disable(rcrtc);
/*
* TODO: Find another way to stop the display for DUs that don't support
* TVM sync.
*/
- if (rcar_du_has(rcrtc->group->dev, RCAR_DU_FEATURE_TVM_SYNC))
+ if (rcar_du_has(rcrtc->dev, RCAR_DU_FEATURE_TVM_SYNC))
rcar_du_crtc_dsysr_clr_set(rcrtc, DSYSR_TVM_MASK,
DSYSR_TVM_SWITCH);
rstate->outputs = 0;
drm_for_each_encoder_mask(encoder, crtc->dev, state->encoder_mask) {
- struct rcar_du_encoder *renc = to_rcar_encoder(encoder);
+ struct rcar_du_encoder *renc;
+ /* Skip the writeback encoder. */
+ if (encoder->encoder_type == DRM_MODE_ENCODER_VIRTUAL)
+ continue;
+
+ renc = to_rcar_encoder(encoder);
rstate->outputs |= BIT(renc->output);
}
{
struct rcar_du_crtc *rcrtc = to_rcar_crtc(crtc);
struct rcar_du_crtc_state *rstate = to_rcar_crtc_state(crtc->state);
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
rcar_du_crtc_get(rcrtc);
{
struct rcar_du_crtc *rcrtc = to_rcar_crtc(crtc);
struct rcar_du_crtc_state *rstate = to_rcar_crtc_state(old_state);
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
rcar_du_crtc_stop(rcrtc);
rcar_du_crtc_put(rcrtc);
*/
rcar_du_crtc_get(rcrtc);
- if (rcar_du_has(rcrtc->group->dev, RCAR_DU_FEATURE_VSP1_SOURCE))
+ if (rcar_du_has(rcrtc->dev, RCAR_DU_FEATURE_VSP1_SOURCE))
rcar_du_vsp_atomic_begin(rcrtc);
}
spin_unlock_irqrestore(&dev->event_lock, flags);
}
- if (rcar_du_has(rcrtc->group->dev, RCAR_DU_FEATURE_VSP1_SOURCE))
+ if (rcar_du_has(rcrtc->dev, RCAR_DU_FEATURE_VSP1_SOURCE))
rcar_du_vsp_atomic_flush(rcrtc);
}
-enum drm_mode_status rcar_du_crtc_mode_valid(struct drm_crtc *crtc,
- const struct drm_display_mode *mode)
+static enum drm_mode_status
+rcar_du_crtc_mode_valid(struct drm_crtc *crtc,
+ const struct drm_display_mode *mode)
{
struct rcar_du_crtc *rcrtc = to_rcar_crtc(crtc);
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
bool interlaced = mode->flags & DRM_MODE_FLAG_INTERLACE;
unsigned int vbp;
static void rcar_du_crtc_crc_init(struct rcar_du_crtc *rcrtc)
{
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
const char **sources;
unsigned int count;
int i = -1;
return 0;
}
-const char *const *rcar_du_crtc_get_crc_sources(struct drm_crtc *crtc,
- size_t *count)
+static const char *const *
+rcar_du_crtc_get_crc_sources(struct drm_crtc *crtc, size_t *count)
{
struct rcar_du_crtc *rcrtc = to_rcar_crtc(crtc);
static irqreturn_t rcar_du_crtc_irq(int irq, void *arg)
{
struct rcar_du_crtc *rcrtc = arg;
- struct rcar_du_device *rcdu = rcrtc->group->dev;
+ struct rcar_du_device *rcdu = rcrtc->dev;
irqreturn_t ret = IRQ_NONE;
u32 status;
init_waitqueue_head(&rcrtc->vblank_wait);
spin_lock_init(&rcrtc->vblank_lock);
+ rcrtc->dev = rcdu;
rcrtc->group = rgrp;
rcrtc->mmio_offset = mmio_offsets[hwindex];
rcrtc->index = hwindex;
#include <linux/wait.h>
#include <drm/drm_crtc.h>
+#include <drm/drm_writeback.h>
#include <media/vsp1.h>
/**
* struct rcar_du_crtc - the CRTC, representing a DU superposition processor
* @crtc: base DRM CRTC
+ * @dev: the DU device
* @clock: the CRTC functional clock
* @extclock: external pixel dot clock (optional)
* @mmio_offset: offset of the CRTC registers in the DU MMIO block
- * @index: CRTC software and hardware index
+ * @index: CRTC hardware index
* @initialized: whether the CRTC has been initialized and clocks enabled
* @dsysr: cached value of the DSYSR register
* @vblank_enable: whether vblank events are enabled on this CRTC
* @group: CRTC group this CRTC belongs to
* @vsp: VSP feeding video to this CRTC
* @vsp_pipe: index of the VSP pipeline feeding video to this CRTC
+ * @writeback: the writeback connector
*/
struct rcar_du_crtc {
struct drm_crtc crtc;
+ struct rcar_du_device *dev;
struct clk *clock;
struct clk *extclock;
unsigned int mmio_offset;
const char *const *sources;
unsigned int sources_count;
+
+ struct drm_writeback_connector writeback;
};
-#define to_rcar_crtc(c) container_of(c, struct rcar_du_crtc, crtc)
+#define to_rcar_crtc(c) container_of(c, struct rcar_du_crtc, crtc)
+#define wb_to_rcar_crtc(c) container_of(c, struct rcar_du_crtc, writeback)
/**
* struct rcar_du_crtc_state - Driver-specific CRTC state
int rcar_du_crtc_create(struct rcar_du_group *rgrp, unsigned int swindex,
unsigned int hwindex);
-void rcar_du_crtc_suspend(struct rcar_du_crtc *rcrtc);
-void rcar_du_crtc_resume(struct rcar_du_crtc *rcrtc);
void rcar_du_crtc_finish_page_flip(struct rcar_du_crtc *rcrtc);
.destroy = drm_encoder_cleanup,
};
+static unsigned int rcar_du_encoder_count_ports(struct device_node *node)
+{
+ struct device_node *ports;
+ struct device_node *port;
+ unsigned int num_ports = 0;
+
+ ports = of_get_child_by_name(node, "ports");
+ if (!ports)
+ ports = of_node_get(node);
+
+ for_each_child_of_node(ports, port) {
+ if (of_node_name_eq(port, "port"))
+ num_ports++;
+ }
+
+ of_node_put(ports);
+
+ return num_ports;
+}
+
int rcar_du_encoder_init(struct rcar_du_device *rcdu,
enum rcar_du_output output,
struct device_node *enc_node)
{
struct rcar_du_encoder *renc;
struct drm_encoder *encoder;
- struct drm_bridge *bridge = NULL;
+ struct drm_bridge *bridge;
int ret;
renc = devm_kzalloc(rcdu->dev, sizeof(*renc), GFP_KERNEL);
dev_dbg(rcdu->dev, "initializing encoder %pOF for output %u\n",
enc_node, output);
- /* Locate the DRM bridge from the encoder DT node. */
- bridge = of_drm_find_bridge(enc_node);
- if (!bridge) {
- ret = -EPROBE_DEFER;
- goto done;
+ /*
+ * Locate the DRM bridge from the DT node. For the DPAD outputs, if the
+ * DT node has a single port, assume that it describes a panel and
+ * create a panel bridge.
+ */
+ if ((output == RCAR_DU_OUTPUT_DPAD0 ||
+ output == RCAR_DU_OUTPUT_DPAD1) &&
+ rcar_du_encoder_count_ports(enc_node) == 1) {
+ struct drm_panel *panel = of_drm_find_panel(enc_node);
+
+ if (IS_ERR(panel)) {
+ ret = PTR_ERR(panel);
+ goto done;
+ }
+
+ bridge = devm_drm_panel_bridge_add(rcdu->dev, panel,
+ DRM_MODE_CONNECTOR_DPI);
+ if (IS_ERR(bridge)) {
+ ret = PTR_ERR(bridge);
+ goto done;
+ }
+ } else {
+ bridge = of_drm_find_bridge(enc_node);
+ if (!bridge) {
+ ret = -EPROBE_DEFER;
+ goto done;
+ }
}
ret = drm_encoder_init(rcdu->ddev, encoder, &encoder_funcs,
#include "rcar_du_kms.h"
#include "rcar_du_regs.h"
#include "rcar_du_vsp.h"
+#include "rcar_du_writeback.h"
/* -----------------------------------------------------------------------------
* Format helpers
static const struct rcar_du_format_info rcar_du_format_infos[] = {
{
.fourcc = DRM_FORMAT_RGB565,
+ .v4l2 = V4L2_PIX_FMT_RGB565,
.bpp = 16,
.planes = 1,
.pnmr = PnMR_SPIM_TP | PnMR_DDDF_16BPP,
.edf = PnDDCR4_EDF_NONE,
}, {
.fourcc = DRM_FORMAT_ARGB1555,
+ .v4l2 = V4L2_PIX_FMT_ARGB555,
.bpp = 16,
.planes = 1,
.pnmr = PnMR_SPIM_ALP | PnMR_DDDF_ARGB,
.edf = PnDDCR4_EDF_NONE,
}, {
.fourcc = DRM_FORMAT_XRGB1555,
+ .v4l2 = V4L2_PIX_FMT_XRGB555,
.bpp = 16,
.planes = 1,
.pnmr = PnMR_SPIM_ALP | PnMR_DDDF_ARGB,
.edf = PnDDCR4_EDF_NONE,
}, {
.fourcc = DRM_FORMAT_XRGB8888,
+ .v4l2 = V4L2_PIX_FMT_XBGR32,
.bpp = 32,
.planes = 1,
.pnmr = PnMR_SPIM_TP | PnMR_DDDF_16BPP,
.edf = PnDDCR4_EDF_RGB888,
}, {
.fourcc = DRM_FORMAT_ARGB8888,
+ .v4l2 = V4L2_PIX_FMT_ABGR32,
.bpp = 32,
.planes = 1,
.pnmr = PnMR_SPIM_ALP | PnMR_DDDF_16BPP,
.edf = PnDDCR4_EDF_ARGB8888,
}, {
.fourcc = DRM_FORMAT_UYVY,
+ .v4l2 = V4L2_PIX_FMT_UYVY,
.bpp = 16,
.planes = 1,
.pnmr = PnMR_SPIM_TP_OFF | PnMR_DDDF_YC,
.edf = PnDDCR4_EDF_NONE,
}, {
.fourcc = DRM_FORMAT_YUYV,
+ .v4l2 = V4L2_PIX_FMT_YUYV,
.bpp = 16,
.planes = 1,
.pnmr = PnMR_SPIM_TP_OFF | PnMR_DDDF_YC,
.edf = PnDDCR4_EDF_NONE,
}, {
.fourcc = DRM_FORMAT_NV12,
+ .v4l2 = V4L2_PIX_FMT_NV12M,
.bpp = 12,
.planes = 2,
.pnmr = PnMR_SPIM_TP_OFF | PnMR_DDDF_YC,
.edf = PnDDCR4_EDF_NONE,
}, {
.fourcc = DRM_FORMAT_NV21,
+ .v4l2 = V4L2_PIX_FMT_NV21M,
.bpp = 12,
.planes = 2,
.pnmr = PnMR_SPIM_TP_OFF | PnMR_DDDF_YC,
.edf = PnDDCR4_EDF_NONE,
}, {
.fourcc = DRM_FORMAT_NV16,
+ .v4l2 = V4L2_PIX_FMT_NV16M,
.bpp = 16,
.planes = 2,
.pnmr = PnMR_SPIM_TP_OFF | PnMR_DDDF_YC,
*/
{
.fourcc = DRM_FORMAT_RGB332,
+ .v4l2 = V4L2_PIX_FMT_RGB332,
.bpp = 8,
.planes = 1,
}, {
.fourcc = DRM_FORMAT_ARGB4444,
+ .v4l2 = V4L2_PIX_FMT_ARGB444,
.bpp = 16,
.planes = 1,
}, {
.fourcc = DRM_FORMAT_XRGB4444,
+ .v4l2 = V4L2_PIX_FMT_XRGB444,
.bpp = 16,
.planes = 1,
}, {
.fourcc = DRM_FORMAT_BGR888,
+ .v4l2 = V4L2_PIX_FMT_RGB24,
.bpp = 24,
.planes = 1,
}, {
.fourcc = DRM_FORMAT_RGB888,
+ .v4l2 = V4L2_PIX_FMT_BGR24,
.bpp = 24,
.planes = 1,
}, {
.fourcc = DRM_FORMAT_BGRA8888,
+ .v4l2 = V4L2_PIX_FMT_ARGB32,
.bpp = 32,
.planes = 1,
}, {
.fourcc = DRM_FORMAT_BGRX8888,
+ .v4l2 = V4L2_PIX_FMT_XRGB32,
.bpp = 32,
.planes = 1,
}, {
.fourcc = DRM_FORMAT_YVYU,
+ .v4l2 = V4L2_PIX_FMT_YVYU,
.bpp = 16,
.planes = 1,
}, {
.fourcc = DRM_FORMAT_NV61,
+ .v4l2 = V4L2_PIX_FMT_NV61M,
.bpp = 16,
.planes = 2,
}, {
.fourcc = DRM_FORMAT_YUV420,
+ .v4l2 = V4L2_PIX_FMT_YUV420M,
.bpp = 12,
.planes = 3,
}, {
.fourcc = DRM_FORMAT_YVU420,
+ .v4l2 = V4L2_PIX_FMT_YVU420M,
.bpp = 12,
.planes = 3,
}, {
.fourcc = DRM_FORMAT_YUV422,
+ .v4l2 = V4L2_PIX_FMT_YUV422M,
.bpp = 16,
.planes = 3,
}, {
.fourcc = DRM_FORMAT_YVU422,
+ .v4l2 = V4L2_PIX_FMT_YVU422M,
.bpp = 16,
.planes = 3,
}, {
.fourcc = DRM_FORMAT_YUV444,
+ .v4l2 = V4L2_PIX_FMT_YUV444M,
.bpp = 24,
.planes = 3,
}, {
.fourcc = DRM_FORMAT_YVU444,
+ .v4l2 = V4L2_PIX_FMT_YVU444M,
.bpp = 24,
.planes = 3,
},
encoder->possible_clones = (1 << num_encoders) - 1;
}
+ /* Create the writeback connectors. */
+ if (rcdu->info->gen >= 3) {
+ for (i = 0; i < rcdu->num_crtcs; ++i) {
+ struct rcar_du_crtc *rcrtc = &rcdu->crtcs[i];
+
+ ret = rcar_du_writeback_init(rcdu, rcrtc);
+ if (ret < 0)
+ return ret;
+ }
+ }
+
/*
* Initialize the default DPAD0 source to the index of the first DU
* channel that can be connected to DPAD0. The exact value doesn't
struct rcar_du_format_info {
u32 fourcc;
+ u32 v4l2;
unsigned int bpp;
unsigned int planes;
unsigned int pnmr;
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
#include <drm/drm_fb_cma_helper.h>
+#include <drm/drm_fourcc.h>
#include <drm/drm_gem_cma_helper.h>
#include <drm/drm_gem_framebuffer_helper.h>
#include <drm/drm_plane_helper.h>
#include "rcar_du_drv.h"
#include "rcar_du_kms.h"
#include "rcar_du_vsp.h"
+#include "rcar_du_writeback.h"
-static void rcar_du_vsp_complete(void *private, bool completed, u32 crc)
+static void rcar_du_vsp_complete(void *private, unsigned int status, u32 crc)
{
struct rcar_du_crtc *crtc = private;
if (crtc->vblank_enable)
drm_crtc_handle_vblank(&crtc->crtc);
- if (completed)
+ if (status & VSP1_DU_STATUS_COMPLETE)
rcar_du_crtc_finish_page_flip(crtc);
+ if (status & VSP1_DU_STATUS_WRITEBACK)
+ rcar_du_writeback_complete(crtc);
drm_crtc_add_crc_entry(&crtc->crtc, false, 0, &crc);
}
void rcar_du_vsp_enable(struct rcar_du_crtc *crtc)
{
const struct drm_display_mode *mode = &crtc->crtc.state->adjusted_mode;
- struct rcar_du_device *rcdu = crtc->group->dev;
+ struct rcar_du_device *rcdu = crtc->dev;
struct vsp1_du_lif_config cfg = {
.width = mode->hdisplay,
.height = mode->vdisplay,
state = to_rcar_crtc_state(crtc->crtc.state);
cfg.crc = state->crc;
+ rcar_du_writeback_setup(crtc, &cfg.writeback);
+
vsp1_du_atomic_flush(crtc->vsp->vsp, crtc->vsp_pipe, &cfg);
}
-/* Keep the two tables in sync. */
-static const u32 formats_kms[] = {
+static const u32 rcar_du_vsp_formats[] = {
DRM_FORMAT_RGB332,
DRM_FORMAT_ARGB4444,
DRM_FORMAT_XRGB4444,
DRM_FORMAT_YVU444,
};
-static const u32 formats_v4l2[] = {
- V4L2_PIX_FMT_RGB332,
- V4L2_PIX_FMT_ARGB444,
- V4L2_PIX_FMT_XRGB444,
- V4L2_PIX_FMT_ARGB555,
- V4L2_PIX_FMT_XRGB555,
- V4L2_PIX_FMT_RGB565,
- V4L2_PIX_FMT_RGB24,
- V4L2_PIX_FMT_BGR24,
- V4L2_PIX_FMT_ARGB32,
- V4L2_PIX_FMT_XRGB32,
- V4L2_PIX_FMT_ABGR32,
- V4L2_PIX_FMT_XBGR32,
- V4L2_PIX_FMT_UYVY,
- V4L2_PIX_FMT_YUYV,
- V4L2_PIX_FMT_YVYU,
- V4L2_PIX_FMT_NV12M,
- V4L2_PIX_FMT_NV21M,
- V4L2_PIX_FMT_NV16M,
- V4L2_PIX_FMT_NV61M,
- V4L2_PIX_FMT_YUV420M,
- V4L2_PIX_FMT_YVU420M,
- V4L2_PIX_FMT_YUV422M,
- V4L2_PIX_FMT_YVU422M,
- V4L2_PIX_FMT_YUV444M,
- V4L2_PIX_FMT_YVU444M,
-};
-
static void rcar_du_vsp_plane_setup(struct rcar_du_vsp_plane *plane)
{
struct rcar_du_vsp_plane_state *state =
to_rcar_vsp_plane_state(plane->plane.state);
struct rcar_du_crtc *crtc = to_rcar_crtc(state->state.crtc);
struct drm_framebuffer *fb = plane->plane.state->fb;
+ const struct rcar_du_format_info *format;
struct vsp1_du_atomic_config cfg = {
.pixelformat = 0,
.pitch = fb->pitches[0],
cfg.mem[i] = sg_dma_address(state->sg_tables[i].sgl)
+ fb->offsets[i];
- for (i = 0; i < ARRAY_SIZE(formats_kms); ++i) {
- if (formats_kms[i] == state->format->fourcc) {
- cfg.pixelformat = formats_v4l2[i];
- break;
- }
- }
+ format = rcar_du_format_info(state->format->fourcc);
+ cfg.pixelformat = format->v4l2;
vsp1_du_atomic_update(plane->vsp->vsp, crtc->vsp_pipe,
plane->index, &cfg);
}
-static int rcar_du_vsp_plane_prepare_fb(struct drm_plane *plane,
- struct drm_plane_state *state)
+int rcar_du_vsp_map_fb(struct rcar_du_vsp *vsp, struct drm_framebuffer *fb,
+ struct sg_table sg_tables[3])
{
- struct rcar_du_vsp_plane_state *rstate = to_rcar_vsp_plane_state(state);
- struct rcar_du_vsp *vsp = to_rcar_vsp_plane(plane)->vsp;
struct rcar_du_device *rcdu = vsp->dev;
unsigned int i;
int ret;
- /*
- * There's no need to prepare (and unprepare) the framebuffer when the
- * plane is not visible, as it will not be displayed.
- */
- if (!state->visible)
- return 0;
-
- for (i = 0; i < rstate->format->planes; ++i) {
- struct drm_gem_cma_object *gem =
- drm_fb_cma_get_gem_obj(state->fb, i);
- struct sg_table *sgt = &rstate->sg_tables[i];
+ for (i = 0; i < fb->format->num_planes; ++i) {
+ struct drm_gem_cma_object *gem = drm_fb_cma_get_gem_obj(fb, i);
+ struct sg_table *sgt = &sg_tables[i];
ret = dma_get_sgtable(rcdu->dev, sgt, gem->vaddr, gem->paddr,
gem->base.size);
}
}
- ret = drm_gem_fb_prepare_fb(plane, state);
- if (ret)
- goto fail;
-
return 0;
fail:
while (i--) {
- struct sg_table *sgt = &rstate->sg_tables[i];
+ struct sg_table *sgt = &sg_tables[i];
vsp1_du_unmap_sg(vsp->vsp, sgt);
sg_free_table(sgt);
return ret;
}
-static void rcar_du_vsp_plane_cleanup_fb(struct drm_plane *plane,
- struct drm_plane_state *state)
+static int rcar_du_vsp_plane_prepare_fb(struct drm_plane *plane,
+ struct drm_plane_state *state)
{
struct rcar_du_vsp_plane_state *rstate = to_rcar_vsp_plane_state(state);
struct rcar_du_vsp *vsp = to_rcar_vsp_plane(plane)->vsp;
- unsigned int i;
+ int ret;
+ /*
+ * There's no need to prepare (and unprepare) the framebuffer when the
+ * plane is not visible, as it will not be displayed.
+ */
if (!state->visible)
- return;
+ return 0;
+
+ ret = rcar_du_vsp_map_fb(vsp, state->fb, rstate->sg_tables);
+ if (ret < 0)
+ return ret;
+
+ return drm_gem_fb_prepare_fb(plane, state);
+}
+
+void rcar_du_vsp_unmap_fb(struct rcar_du_vsp *vsp, struct drm_framebuffer *fb,
+ struct sg_table sg_tables[3])
+{
+ unsigned int i;
- for (i = 0; i < rstate->format->planes; ++i) {
- struct sg_table *sgt = &rstate->sg_tables[i];
+ for (i = 0; i < fb->format->num_planes; ++i) {
+ struct sg_table *sgt = &sg_tables[i];
vsp1_du_unmap_sg(vsp->vsp, sgt);
sg_free_table(sgt);
}
}
+static void rcar_du_vsp_plane_cleanup_fb(struct drm_plane *plane,
+ struct drm_plane_state *state)
+{
+ struct rcar_du_vsp_plane_state *rstate = to_rcar_vsp_plane_state(state);
+ struct rcar_du_vsp *vsp = to_rcar_vsp_plane(plane)->vsp;
+
+ if (!state->visible)
+ return;
+
+ rcar_du_vsp_unmap_fb(vsp, state->fb, rstate->sg_tables);
+}
+
static int rcar_du_vsp_plane_atomic_check(struct drm_plane *plane,
struct drm_plane_state *state)
{
ret = drm_universal_plane_init(rcdu->ddev, &plane->plane, crtcs,
&rcar_du_vsp_plane_funcs,
- formats_kms,
- ARRAY_SIZE(formats_kms),
+ rcar_du_vsp_formats,
+ ARRAY_SIZE(rcar_du_vsp_formats),
NULL, type, NULL);
if (ret < 0)
return ret;
#include <drm/drm_plane.h>
+struct drm_framebuffer;
struct rcar_du_format_info;
struct rcar_du_vsp;
+struct sg_table;
struct rcar_du_vsp_plane {
struct drm_plane plane;
void rcar_du_vsp_disable(struct rcar_du_crtc *crtc);
void rcar_du_vsp_atomic_begin(struct rcar_du_crtc *crtc);
void rcar_du_vsp_atomic_flush(struct rcar_du_crtc *crtc);
+int rcar_du_vsp_map_fb(struct rcar_du_vsp *vsp, struct drm_framebuffer *fb,
+ struct sg_table sg_tables[3]);
+void rcar_du_vsp_unmap_fb(struct rcar_du_vsp *vsp, struct drm_framebuffer *fb,
+ struct sg_table sg_tables[3]);
#else
static inline int rcar_du_vsp_init(struct rcar_du_vsp *vsp,
struct device_node *np,
static inline void rcar_du_vsp_disable(struct rcar_du_crtc *crtc) { };
static inline void rcar_du_vsp_atomic_begin(struct rcar_du_crtc *crtc) { };
static inline void rcar_du_vsp_atomic_flush(struct rcar_du_crtc *crtc) { };
+static inline int rcar_du_vsp_map_fb(struct rcar_du_vsp *vsp,
+ struct drm_framebuffer *fb,
+ struct sg_table sg_tables[3])
+{
+ return -ENXIO;
+}
+static inline void rcar_du_vsp_unmap_fb(struct rcar_du_vsp *vsp,
+ struct drm_framebuffer *fb,
+ struct sg_table sg_tables[3])
+{
+}
#endif
#endif /* __RCAR_DU_VSP_H__ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * rcar_du_writeback.c -- R-Car Display Unit Writeback Support
+ *
+ * Copyright (C) 2019 Laurent Pinchart <laurent.pinchart@ideasonboard.com>
+ */
+
+#include <drm/drm_atomic_helper.h>
+#include <drm/drm_device.h>
+#include <drm/drm_fourcc.h>
+#include <drm/drm_probe_helper.h>
+#include <drm/drm_writeback.h>
+
+#include "rcar_du_crtc.h"
+#include "rcar_du_drv.h"
+#include "rcar_du_kms.h"
+
+/**
+ * struct rcar_du_wb_conn_state - Driver-specific writeback connector state
+ * @state: base DRM connector state
+ * @format: format of the writeback framebuffer
+ */
+struct rcar_du_wb_conn_state {
+ struct drm_connector_state state;
+ const struct rcar_du_format_info *format;
+};
+
+#define to_rcar_wb_conn_state(s) \
+ container_of(s, struct rcar_du_wb_conn_state, state)
+
+/**
+ * struct rcar_du_wb_job - Driver-private data for writeback jobs
+ * @sg_tables: scatter-gather tables for the framebuffer memory
+ */
+struct rcar_du_wb_job {
+ struct sg_table sg_tables[3];
+};
+
+static int rcar_du_wb_conn_get_modes(struct drm_connector *connector)
+{
+ struct drm_device *dev = connector->dev;
+
+ return drm_add_modes_noedid(connector, dev->mode_config.max_width,
+ dev->mode_config.max_height);
+}
+
+static int rcar_du_wb_prepare_job(struct drm_writeback_connector *connector,
+ struct drm_writeback_job *job)
+{
+ struct rcar_du_crtc *rcrtc = wb_to_rcar_crtc(connector);
+ struct rcar_du_wb_job *rjob;
+ int ret;
+
+ if (!job->fb)
+ return 0;
+
+ rjob = kzalloc(sizeof(*rjob), GFP_KERNEL);
+ if (!rjob)
+ return -ENOMEM;
+
+ /* Map the framebuffer to the VSP. */
+ ret = rcar_du_vsp_map_fb(rcrtc->vsp, job->fb, rjob->sg_tables);
+ if (ret < 0) {
+ kfree(rjob);
+ return ret;
+ }
+
+ job->priv = rjob;
+ return 0;
+}
+
+static void rcar_du_wb_cleanup_job(struct drm_writeback_connector *connector,
+ struct drm_writeback_job *job)
+{
+ struct rcar_du_crtc *rcrtc = wb_to_rcar_crtc(connector);
+ struct rcar_du_wb_job *rjob = job->priv;
+
+ if (!job->fb)
+ return;
+
+ rcar_du_vsp_unmap_fb(rcrtc->vsp, job->fb, rjob->sg_tables);
+ kfree(rjob);
+}
+
+static const struct drm_connector_helper_funcs rcar_du_wb_conn_helper_funcs = {
+ .get_modes = rcar_du_wb_conn_get_modes,
+ .prepare_writeback_job = rcar_du_wb_prepare_job,
+ .cleanup_writeback_job = rcar_du_wb_cleanup_job,
+};
+
+static struct drm_connector_state *
+rcar_du_wb_conn_duplicate_state(struct drm_connector *connector)
+{
+ struct rcar_du_wb_conn_state *copy;
+
+ if (WARN_ON(!connector->state))
+ return NULL;
+
+ copy = kzalloc(sizeof(*copy), GFP_KERNEL);
+ if (!copy)
+ return NULL;
+
+ __drm_atomic_helper_connector_duplicate_state(connector, ©->state);
+
+ return ©->state;
+}
+
+static void rcar_du_wb_conn_destroy_state(struct drm_connector *connector,
+ struct drm_connector_state *state)
+{
+ __drm_atomic_helper_connector_destroy_state(state);
+ kfree(to_rcar_wb_conn_state(state));
+}
+
+static void rcar_du_wb_conn_reset(struct drm_connector *connector)
+{
+ struct rcar_du_wb_conn_state *state;
+
+ if (connector->state) {
+ rcar_du_wb_conn_destroy_state(connector, connector->state);
+ connector->state = NULL;
+ }
+
+ state = kzalloc(sizeof(*state), GFP_KERNEL);
+ if (state == NULL)
+ return;
+
+ __drm_atomic_helper_connector_reset(connector, &state->state);
+}
+
+static const struct drm_connector_funcs rcar_du_wb_conn_funcs = {
+ .reset = rcar_du_wb_conn_reset,
+ .fill_modes = drm_helper_probe_single_connector_modes,
+ .destroy = drm_connector_cleanup,
+ .atomic_duplicate_state = rcar_du_wb_conn_duplicate_state,
+ .atomic_destroy_state = rcar_du_wb_conn_destroy_state,
+};
+
+static int rcar_du_wb_enc_atomic_check(struct drm_encoder *encoder,
+ struct drm_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state)
+{
+ struct rcar_du_wb_conn_state *wb_state =
+ to_rcar_wb_conn_state(conn_state);
+ const struct drm_display_mode *mode = &crtc_state->mode;
+ struct drm_device *dev = encoder->dev;
+ struct drm_framebuffer *fb;
+
+ if (!conn_state->writeback_job || !conn_state->writeback_job->fb)
+ return 0;
+
+ fb = conn_state->writeback_job->fb;
+
+ /*
+ * Verify that the framebuffer format is supported and that its size
+ * matches the current mode.
+ */
+ if (fb->width != mode->hdisplay || fb->height != mode->vdisplay) {
+ dev_dbg(dev->dev, "%s: invalid framebuffer size %ux%u\n",
+ __func__, fb->width, fb->height);
+ return -EINVAL;
+ }
+
+ wb_state->format = rcar_du_format_info(fb->format->format);
+ if (wb_state->format == NULL) {
+ dev_dbg(dev->dev, "%s: unsupported format %08x\n", __func__,
+ fb->format->format);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static const struct drm_encoder_helper_funcs rcar_du_wb_enc_helper_funcs = {
+ .atomic_check = rcar_du_wb_enc_atomic_check,
+};
+
+/*
+ * Only RGB formats are currently supported as the VSP outputs RGB to the DU
+ * and can't convert to YUV separately for writeback.
+ */
+static const u32 writeback_formats[] = {
+ DRM_FORMAT_RGB332,
+ DRM_FORMAT_ARGB4444,
+ DRM_FORMAT_XRGB4444,
+ DRM_FORMAT_ARGB1555,
+ DRM_FORMAT_XRGB1555,
+ DRM_FORMAT_RGB565,
+ DRM_FORMAT_BGR888,
+ DRM_FORMAT_RGB888,
+ DRM_FORMAT_BGRA8888,
+ DRM_FORMAT_BGRX8888,
+ DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_XRGB8888,
+};
+
+int rcar_du_writeback_init(struct rcar_du_device *rcdu,
+ struct rcar_du_crtc *rcrtc)
+{
+ struct drm_writeback_connector *wb_conn = &rcrtc->writeback;
+
+ wb_conn->encoder.possible_crtcs = 1 << drm_crtc_index(&rcrtc->crtc);
+ drm_connector_helper_add(&wb_conn->base,
+ &rcar_du_wb_conn_helper_funcs);
+
+ return drm_writeback_connector_init(rcdu->ddev, wb_conn,
+ &rcar_du_wb_conn_funcs,
+ &rcar_du_wb_enc_helper_funcs,
+ writeback_formats,
+ ARRAY_SIZE(writeback_formats));
+}
+
+void rcar_du_writeback_setup(struct rcar_du_crtc *rcrtc,
+ struct vsp1_du_writeback_config *cfg)
+{
+ struct rcar_du_wb_conn_state *wb_state;
+ struct drm_connector_state *state;
+ struct rcar_du_wb_job *rjob;
+ struct drm_framebuffer *fb;
+ unsigned int i;
+
+ state = rcrtc->writeback.base.state;
+ if (!state || !state->writeback_job || !state->writeback_job->fb)
+ return;
+
+ fb = state->writeback_job->fb;
+ rjob = state->writeback_job->priv;
+ wb_state = to_rcar_wb_conn_state(state);
+
+ cfg->pixelformat = wb_state->format->v4l2;
+ cfg->pitch = fb->pitches[0];
+
+ for (i = 0; i < wb_state->format->planes; ++i)
+ cfg->mem[i] = sg_dma_address(rjob->sg_tables[i].sgl)
+ + fb->offsets[i];
+
+ drm_writeback_queue_job(&rcrtc->writeback, state);
+}
+
+void rcar_du_writeback_complete(struct rcar_du_crtc *rcrtc)
+{
+ drm_writeback_signal_completion(&rcrtc->writeback, 0);
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * rcar_du_writeback.h -- R-Car Display Unit Writeback Support
+ *
+ * Copyright (C) 2019 Laurent Pinchart <laurent.pinchart@ideasonboard.com>
+ */
+
+#ifndef __RCAR_DU_WRITEBACK_H__
+#define __RCAR_DU_WRITEBACK_H__
+
+#include <drm/drm_plane.h>
+
+struct rcar_du_crtc;
+struct rcar_du_device;
+struct vsp1_du_atomic_pipe_config;
+
+#ifdef CONFIG_DRM_RCAR_WRITEBACK
+int rcar_du_writeback_init(struct rcar_du_device *rcdu,
+ struct rcar_du_crtc *rcrtc);
+void rcar_du_writeback_setup(struct rcar_du_crtc *rcrtc,
+ struct vsp1_du_writeback_config *cfg);
+void rcar_du_writeback_complete(struct rcar_du_crtc *rcrtc);
+#else
+static inline int rcar_du_writeback_init(struct rcar_du_device *rcdu,
+ struct rcar_du_crtc *rcrtc)
+{
+ return -ENXIO;
+}
+static inline void
+rcar_du_writeback_setup(struct rcar_du_crtc *rcrtc,
+ struct vsp1_du_writeback_config *cfg)
+{
+}
+static inline void rcar_du_writeback_complete(struct rcar_du_crtc *rcrtc)
+{
+}
+#endif
+
+#endif /* __RCAR_DU_WRITEBACK_H__ */
* divider.
*/
fout = fvco / (1 << e) / div7;
- div = DIV_ROUND_CLOSEST(fout, target);
+ div = max(1UL, DIV_ROUND_CLOSEST(fout, target));
diff = abs(fout / div - target);
if (diff < pll->diff) {
}
if (lvds->info->quirks & RCAR_LVDS_QUIRK_GEN3_LVEN) {
- /* Turn on the LVDS PHY. */
+ /*
+ * Turn on the LVDS PHY. On D3, the LVEN and LVRES bit must be
+ * set at the same time, so don't write the register yet.
+ */
lvdcr0 |= LVDCR0_LVEN;
- rcar_lvds_write(lvds, LVDCR0, lvdcr0);
+ if (!(lvds->info->quirks & RCAR_LVDS_QUIRK_PWD))
+ rcar_lvds_write(lvds, LVDCR0, lvdcr0);
}
if (!(lvds->info->quirks & RCAR_LVDS_QUIRK_EXT_PLL)) {
const struct drm_display_mode *mode,
struct drm_display_mode *adjusted_mode)
{
+ struct rcar_lvds *lvds = bridge_to_rcar_lvds(bridge);
+ int min_freq;
+
/*
* The internal LVDS encoder has a restricted clock frequency operating
- * range (31MHz to 148.5MHz). Clamp the clock accordingly.
+ * range, from 5MHz to 148.5MHz on D3 and E3, and from 31MHz to
+ * 148.5MHz on all other platforms. Clamp the clock accordingly.
*/
- adjusted_mode->clock = clamp(adjusted_mode->clock, 31000, 148500);
+ min_freq = lvds->info->quirks & RCAR_LVDS_QUIRK_EXT_PLL ? 5000 : 31000;
+ adjusted_mode->clock = clamp(adjusted_mode->clock, min_freq, 148500);
return true;
}
Some Rockchip CRTCs, like rv1108, can directly output parallel
and serial RGB format to panel or connect to a conversion chip.
say Y to enable its driver.
+
+config ROCKCHIP_RK3066_HDMI
+ bool "Rockchip specific extensions for RK3066 HDMI"
+ depends on DRM_ROCKCHIP
+ help
+ This selects support for Rockchip SoC specific extensions
+ for the RK3066 HDMI driver. If you want to enable
+ HDMI on RK3066 based SoC, you should select this option.
endif
rockchipdrm-$(CONFIG_ROCKCHIP_INNO_HDMI) += inno_hdmi.o
rockchipdrm-$(CONFIG_ROCKCHIP_LVDS) += rockchip_lvds.o
rockchipdrm-$(CONFIG_ROCKCHIP_RGB) += rockchip_rgb.o
+rockchipdrm-$(CONFIG_ROCKCHIP_RK3066_HDMI) += rk3066_hdmi.o
obj-$(CONFIG_DRM_ROCKCHIP) += rockchipdrm.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Zheng Yang <zhengyang@rock-chips.com>
+ */
+
+#include <drm/drm_of.h>
+#include <drm/drm_probe_helper.h>
+
+#include <linux/clk.h>
+#include <linux/mfd/syscon.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+
+#include "rk3066_hdmi.h"
+
+#include "rockchip_drm_drv.h"
+#include "rockchip_drm_vop.h"
+
+#define DEFAULT_PLLA_RATE 30000000
+
+struct hdmi_data_info {
+ int vic; /* The CEA Video ID (VIC) of the current drm display mode. */
+ bool sink_is_hdmi;
+ unsigned int enc_out_format;
+ unsigned int colorimetry;
+};
+
+struct rk3066_hdmi_i2c {
+ struct i2c_adapter adap;
+
+ u8 ddc_addr;
+ u8 segment_addr;
+ u8 stat;
+
+ struct mutex i2c_lock; /* For i2c operation. */
+ struct completion cmpltn;
+};
+
+struct rk3066_hdmi {
+ struct device *dev;
+ struct drm_device *drm_dev;
+ struct regmap *grf_regmap;
+ int irq;
+ struct clk *hclk;
+ void __iomem *regs;
+
+ struct drm_connector connector;
+ struct drm_encoder encoder;
+
+ struct rk3066_hdmi_i2c *i2c;
+ struct i2c_adapter *ddc;
+
+ unsigned int tmdsclk;
+
+ struct hdmi_data_info hdmi_data;
+ struct drm_display_mode previous_mode;
+};
+
+#define to_rk3066_hdmi(x) container_of(x, struct rk3066_hdmi, x)
+
+static inline u8 hdmi_readb(struct rk3066_hdmi *hdmi, u16 offset)
+{
+ return readl_relaxed(hdmi->regs + offset);
+}
+
+static inline void hdmi_writeb(struct rk3066_hdmi *hdmi, u16 offset, u32 val)
+{
+ writel_relaxed(val, hdmi->regs + offset);
+}
+
+static inline void hdmi_modb(struct rk3066_hdmi *hdmi, u16 offset,
+ u32 msk, u32 val)
+{
+ u8 temp = hdmi_readb(hdmi, offset) & ~msk;
+
+ temp |= val & msk;
+ hdmi_writeb(hdmi, offset, temp);
+}
+
+static void rk3066_hdmi_i2c_init(struct rk3066_hdmi *hdmi)
+{
+ int ddc_bus_freq;
+
+ ddc_bus_freq = (hdmi->tmdsclk >> 2) / HDMI_SCL_RATE;
+
+ hdmi_writeb(hdmi, HDMI_DDC_BUS_FREQ_L, ddc_bus_freq & 0xFF);
+ hdmi_writeb(hdmi, HDMI_DDC_BUS_FREQ_H, (ddc_bus_freq >> 8) & 0xFF);
+
+ /* Clear the EDID interrupt flag and mute the interrupt. */
+ hdmi_modb(hdmi, HDMI_INTR_MASK1, HDMI_INTR_EDID_MASK, 0);
+ hdmi_writeb(hdmi, HDMI_INTR_STATUS1, HDMI_INTR_EDID_MASK);
+}
+
+static inline u8 rk3066_hdmi_get_power_mode(struct rk3066_hdmi *hdmi)
+{
+ return hdmi_readb(hdmi, HDMI_SYS_CTRL) & HDMI_SYS_POWER_MODE_MASK;
+}
+
+static void rk3066_hdmi_set_power_mode(struct rk3066_hdmi *hdmi, int mode)
+{
+ u8 current_mode, next_mode;
+ u8 i = 0;
+
+ current_mode = rk3066_hdmi_get_power_mode(hdmi);
+
+ DRM_DEV_DEBUG(hdmi->dev, "mode :%d\n", mode);
+ DRM_DEV_DEBUG(hdmi->dev, "current_mode :%d\n", current_mode);
+
+ if (current_mode == mode)
+ return;
+
+ do {
+ if (current_mode > mode) {
+ next_mode = current_mode / 2;
+ } else {
+ if (current_mode < HDMI_SYS_POWER_MODE_A)
+ next_mode = HDMI_SYS_POWER_MODE_A;
+ else
+ next_mode = current_mode * 2;
+ }
+
+ DRM_DEV_DEBUG(hdmi->dev, "%d: next_mode :%d\n", i, next_mode);
+
+ if (next_mode != HDMI_SYS_POWER_MODE_D) {
+ hdmi_modb(hdmi, HDMI_SYS_CTRL,
+ HDMI_SYS_POWER_MODE_MASK, next_mode);
+ } else {
+ hdmi_writeb(hdmi, HDMI_SYS_CTRL,
+ HDMI_SYS_POWER_MODE_D |
+ HDMI_SYS_PLL_RESET_MASK);
+ usleep_range(90, 100);
+ hdmi_writeb(hdmi, HDMI_SYS_CTRL,
+ HDMI_SYS_POWER_MODE_D |
+ HDMI_SYS_PLLB_RESET);
+ usleep_range(90, 100);
+ hdmi_writeb(hdmi, HDMI_SYS_CTRL,
+ HDMI_SYS_POWER_MODE_D);
+ }
+ current_mode = next_mode;
+ i = i + 1;
+ } while ((next_mode != mode) && (i < 5));
+
+ /*
+ * When the IP controller isn't configured with accurate video timing,
+ * DDC_CLK should be equal to the PLLA frequency, which is 30MHz,
+ * so we need to init the TMDS rate to the PCLK rate and reconfigure
+ * the DDC clock.
+ */
+ if (mode < HDMI_SYS_POWER_MODE_D)
+ hdmi->tmdsclk = DEFAULT_PLLA_RATE;
+}
+
+static int
+rk3066_hdmi_upload_frame(struct rk3066_hdmi *hdmi, int setup_rc,
+ union hdmi_infoframe *frame, u32 frame_index,
+ u32 mask, u32 disable, u32 enable)
+{
+ if (mask)
+ hdmi_modb(hdmi, HDMI_CP_AUTO_SEND_CTRL, mask, disable);
+
+ hdmi_writeb(hdmi, HDMI_CP_BUF_INDEX, frame_index);
+
+ if (setup_rc >= 0) {
+ u8 packed_frame[HDMI_MAXIMUM_INFO_FRAME_SIZE];
+ ssize_t rc, i;
+
+ rc = hdmi_infoframe_pack(frame, packed_frame,
+ sizeof(packed_frame));
+ if (rc < 0)
+ return rc;
+
+ for (i = 0; i < rc; i++)
+ hdmi_writeb(hdmi, HDMI_CP_BUF_ACC_HB0 + i * 4,
+ packed_frame[i]);
+
+ if (mask)
+ hdmi_modb(hdmi, HDMI_CP_AUTO_SEND_CTRL, mask, enable);
+ }
+
+ return setup_rc;
+}
+
+static int rk3066_hdmi_config_avi(struct rk3066_hdmi *hdmi,
+ struct drm_display_mode *mode)
+{
+ union hdmi_infoframe frame;
+ int rc;
+
+ rc = drm_hdmi_avi_infoframe_from_display_mode(&frame.avi,
+ &hdmi->connector, mode);
+
+ if (hdmi->hdmi_data.enc_out_format == HDMI_COLORSPACE_YUV444)
+ frame.avi.colorspace = HDMI_COLORSPACE_YUV444;
+ else if (hdmi->hdmi_data.enc_out_format == HDMI_COLORSPACE_YUV422)
+ frame.avi.colorspace = HDMI_COLORSPACE_YUV422;
+ else
+ frame.avi.colorspace = HDMI_COLORSPACE_RGB;
+
+ frame.avi.colorimetry = hdmi->hdmi_data.colorimetry;
+ frame.avi.scan_mode = HDMI_SCAN_MODE_NONE;
+
+ return rk3066_hdmi_upload_frame(hdmi, rc, &frame,
+ HDMI_INFOFRAME_AVI, 0, 0, 0);
+}
+
+static int rk3066_hdmi_config_video_timing(struct rk3066_hdmi *hdmi,
+ struct drm_display_mode *mode)
+{
+ int value, vsync_offset;
+
+ /* Set the details for the external polarity and interlace mode. */
+ value = HDMI_EXT_VIDEO_SET_EN;
+ value |= mode->flags & DRM_MODE_FLAG_PHSYNC ?
+ HDMI_VIDEO_HSYNC_ACTIVE_HIGH : HDMI_VIDEO_HSYNC_ACTIVE_LOW;
+ value |= mode->flags & DRM_MODE_FLAG_PVSYNC ?
+ HDMI_VIDEO_VSYNC_ACTIVE_HIGH : HDMI_VIDEO_VSYNC_ACTIVE_LOW;
+ value |= mode->flags & DRM_MODE_FLAG_INTERLACE ?
+ HDMI_VIDEO_MODE_INTERLACE : HDMI_VIDEO_MODE_PROGRESSIVE;
+
+ if (hdmi->hdmi_data.vic == 2 || hdmi->hdmi_data.vic == 3)
+ vsync_offset = 6;
+ else
+ vsync_offset = 0;
+
+ value |= vsync_offset << HDMI_VIDEO_VSYNC_OFFSET_SHIFT;
+ hdmi_writeb(hdmi, HDMI_EXT_VIDEO_PARA, value);
+
+ /* Set the details for the external video timing. */
+ value = mode->htotal;
+ hdmi_writeb(hdmi, HDMI_EXT_HTOTAL_L, value & 0xFF);
+ hdmi_writeb(hdmi, HDMI_EXT_HTOTAL_H, (value >> 8) & 0xFF);
+
+ value = mode->htotal - mode->hdisplay;
+ hdmi_writeb(hdmi, HDMI_EXT_HBLANK_L, value & 0xFF);
+ hdmi_writeb(hdmi, HDMI_EXT_HBLANK_H, (value >> 8) & 0xFF);
+
+ value = mode->htotal - mode->hsync_start;
+ hdmi_writeb(hdmi, HDMI_EXT_HDELAY_L, value & 0xFF);
+ hdmi_writeb(hdmi, HDMI_EXT_HDELAY_H, (value >> 8) & 0xFF);
+
+ value = mode->hsync_end - mode->hsync_start;
+ hdmi_writeb(hdmi, HDMI_EXT_HDURATION_L, value & 0xFF);
+ hdmi_writeb(hdmi, HDMI_EXT_HDURATION_H, (value >> 8) & 0xFF);
+
+ value = mode->vtotal;
+ hdmi_writeb(hdmi, HDMI_EXT_VTOTAL_L, value & 0xFF);
+ hdmi_writeb(hdmi, HDMI_EXT_VTOTAL_H, (value >> 8) & 0xFF);
+
+ value = mode->vtotal - mode->vdisplay;
+ hdmi_writeb(hdmi, HDMI_EXT_VBLANK_L, value & 0xFF);
+
+ value = mode->vtotal - mode->vsync_start + vsync_offset;
+ hdmi_writeb(hdmi, HDMI_EXT_VDELAY, value & 0xFF);
+
+ value = mode->vsync_end - mode->vsync_start;
+ hdmi_writeb(hdmi, HDMI_EXT_VDURATION, value & 0xFF);
+
+ return 0;
+}
+
+static void
+rk3066_hdmi_phy_write(struct rk3066_hdmi *hdmi, u16 offset, u8 value)
+{
+ hdmi_writeb(hdmi, offset, value);
+ hdmi_modb(hdmi, HDMI_SYS_CTRL,
+ HDMI_SYS_PLL_RESET_MASK, HDMI_SYS_PLL_RESET);
+ usleep_range(90, 100);
+ hdmi_modb(hdmi, HDMI_SYS_CTRL, HDMI_SYS_PLL_RESET_MASK, 0);
+ usleep_range(900, 1000);
+}
+
+static void rk3066_hdmi_config_phy(struct rk3066_hdmi *hdmi)
+{
+ /* TMDS uses the same frequency as dclk. */
+ hdmi_writeb(hdmi, HDMI_DEEP_COLOR_MODE, 0x22);
+
+ /*
+ * The semi-public documentation does not describe the hdmi registers
+ * used by the function rk3066_hdmi_phy_write(), so we keep using
+ * these magic values for now.
+ */
+ if (hdmi->tmdsclk > 100000000) {
+ rk3066_hdmi_phy_write(hdmi, 0x158, 0x0E);
+ rk3066_hdmi_phy_write(hdmi, 0x15c, 0x00);
+ rk3066_hdmi_phy_write(hdmi, 0x160, 0x60);
+ rk3066_hdmi_phy_write(hdmi, 0x164, 0x00);
+ rk3066_hdmi_phy_write(hdmi, 0x168, 0xDA);
+ rk3066_hdmi_phy_write(hdmi, 0x16c, 0xA1);
+ rk3066_hdmi_phy_write(hdmi, 0x170, 0x0e);
+ rk3066_hdmi_phy_write(hdmi, 0x174, 0x22);
+ rk3066_hdmi_phy_write(hdmi, 0x178, 0x00);
+ } else if (hdmi->tmdsclk > 50000000) {
+ rk3066_hdmi_phy_write(hdmi, 0x158, 0x06);
+ rk3066_hdmi_phy_write(hdmi, 0x15c, 0x00);
+ rk3066_hdmi_phy_write(hdmi, 0x160, 0x60);
+ rk3066_hdmi_phy_write(hdmi, 0x164, 0x00);
+ rk3066_hdmi_phy_write(hdmi, 0x168, 0xCA);
+ rk3066_hdmi_phy_write(hdmi, 0x16c, 0xA3);
+ rk3066_hdmi_phy_write(hdmi, 0x170, 0x0e);
+ rk3066_hdmi_phy_write(hdmi, 0x174, 0x20);
+ rk3066_hdmi_phy_write(hdmi, 0x178, 0x00);
+ } else {
+ rk3066_hdmi_phy_write(hdmi, 0x158, 0x02);
+ rk3066_hdmi_phy_write(hdmi, 0x15c, 0x00);
+ rk3066_hdmi_phy_write(hdmi, 0x160, 0x60);
+ rk3066_hdmi_phy_write(hdmi, 0x164, 0x00);
+ rk3066_hdmi_phy_write(hdmi, 0x168, 0xC2);
+ rk3066_hdmi_phy_write(hdmi, 0x16c, 0xA2);
+ rk3066_hdmi_phy_write(hdmi, 0x170, 0x0e);
+ rk3066_hdmi_phy_write(hdmi, 0x174, 0x20);
+ rk3066_hdmi_phy_write(hdmi, 0x178, 0x00);
+ }
+}
+
+static int rk3066_hdmi_setup(struct rk3066_hdmi *hdmi,
+ struct drm_display_mode *mode)
+{
+ hdmi->hdmi_data.vic = drm_match_cea_mode(mode);
+ hdmi->hdmi_data.enc_out_format = HDMI_COLORSPACE_RGB;
+
+ if (hdmi->hdmi_data.vic == 6 || hdmi->hdmi_data.vic == 7 ||
+ hdmi->hdmi_data.vic == 21 || hdmi->hdmi_data.vic == 22 ||
+ hdmi->hdmi_data.vic == 2 || hdmi->hdmi_data.vic == 3 ||
+ hdmi->hdmi_data.vic == 17 || hdmi->hdmi_data.vic == 18)
+ hdmi->hdmi_data.colorimetry = HDMI_COLORIMETRY_ITU_601;
+ else
+ hdmi->hdmi_data.colorimetry = HDMI_COLORIMETRY_ITU_709;
+
+ hdmi->tmdsclk = mode->clock * 1000;
+
+ /* Mute video and audio output. */
+ hdmi_modb(hdmi, HDMI_VIDEO_CTRL2, HDMI_VIDEO_AUDIO_DISABLE_MASK,
+ HDMI_AUDIO_DISABLE | HDMI_VIDEO_DISABLE);
+
+ /* Set power state to mode B. */
+ if (rk3066_hdmi_get_power_mode(hdmi) != HDMI_SYS_POWER_MODE_B)
+ rk3066_hdmi_set_power_mode(hdmi, HDMI_SYS_POWER_MODE_B);
+
+ /* Input video mode is RGB 24 bit. Use external data enable signal. */
+ hdmi_modb(hdmi, HDMI_AV_CTRL1,
+ HDMI_VIDEO_DE_MASK, HDMI_VIDEO_EXTERNAL_DE);
+ hdmi_writeb(hdmi, HDMI_VIDEO_CTRL1,
+ HDMI_VIDEO_OUTPUT_RGB444 |
+ HDMI_VIDEO_INPUT_DATA_DEPTH_8BIT |
+ HDMI_VIDEO_INPUT_COLOR_RGB);
+ hdmi_writeb(hdmi, HDMI_DEEP_COLOR_MODE, 0x20);
+
+ rk3066_hdmi_config_video_timing(hdmi, mode);
+
+ if (hdmi->hdmi_data.sink_is_hdmi) {
+ hdmi_modb(hdmi, HDMI_HDCP_CTRL, HDMI_VIDEO_MODE_MASK,
+ HDMI_VIDEO_MODE_HDMI);
+ rk3066_hdmi_config_avi(hdmi, mode);
+ } else {
+ hdmi_modb(hdmi, HDMI_HDCP_CTRL, HDMI_VIDEO_MODE_MASK, 0);
+ }
+
+ rk3066_hdmi_config_phy(hdmi);
+
+ rk3066_hdmi_set_power_mode(hdmi, HDMI_SYS_POWER_MODE_E);
+
+ /*
+ * When the IP controller is configured with accurate video
+ * timing, the TMDS clock source should be switched to
+ * DCLK_LCDC, so we need to init the TMDS rate to the pixel mode
+ * clock rate and reconfigure the DDC clock.
+ */
+ rk3066_hdmi_i2c_init(hdmi);
+
+ /* Unmute video output. */
+ hdmi_modb(hdmi, HDMI_VIDEO_CTRL2,
+ HDMI_VIDEO_AUDIO_DISABLE_MASK, HDMI_AUDIO_DISABLE);
+ return 0;
+}
+
+static void
+rk3066_hdmi_encoder_mode_set(struct drm_encoder *encoder,
+ struct drm_display_mode *mode,
+ struct drm_display_mode *adj_mode)
+{
+ struct rk3066_hdmi *hdmi = to_rk3066_hdmi(encoder);
+
+ /* Store the display mode for plugin/DPMS poweron events. */
+ memcpy(&hdmi->previous_mode, adj_mode, sizeof(hdmi->previous_mode));
+}
+
+static void rk3066_hdmi_encoder_enable(struct drm_encoder *encoder)
+{
+ struct rk3066_hdmi *hdmi = to_rk3066_hdmi(encoder);
+ int mux, val;
+
+ mux = drm_of_encoder_active_endpoint_id(hdmi->dev->of_node, encoder);
+ if (mux)
+ val = (HDMI_VIDEO_SEL << 16) | HDMI_VIDEO_SEL;
+ else
+ val = HDMI_VIDEO_SEL << 16;
+
+ regmap_write(hdmi->grf_regmap, GRF_SOC_CON0, val);
+
+ DRM_DEV_DEBUG(hdmi->dev, "hdmi encoder enable select: vop%s\n",
+ (mux) ? "1" : "0");
+
+ rk3066_hdmi_setup(hdmi, &hdmi->previous_mode);
+}
+
+static void rk3066_hdmi_encoder_disable(struct drm_encoder *encoder)
+{
+ struct rk3066_hdmi *hdmi = to_rk3066_hdmi(encoder);
+
+ DRM_DEV_DEBUG(hdmi->dev, "hdmi encoder disable\n");
+
+ if (rk3066_hdmi_get_power_mode(hdmi) == HDMI_SYS_POWER_MODE_E) {
+ hdmi_writeb(hdmi, HDMI_VIDEO_CTRL2,
+ HDMI_VIDEO_AUDIO_DISABLE_MASK);
+ hdmi_modb(hdmi, HDMI_VIDEO_CTRL2,
+ HDMI_AUDIO_CP_LOGIC_RESET_MASK,
+ HDMI_AUDIO_CP_LOGIC_RESET);
+ usleep_range(500, 510);
+ }
+ rk3066_hdmi_set_power_mode(hdmi, HDMI_SYS_POWER_MODE_A);
+}
+
+static bool
+rk3066_hdmi_encoder_mode_fixup(struct drm_encoder *encoder,
+ const struct drm_display_mode *mode,
+ struct drm_display_mode *adj_mode)
+{
+ return true;
+}
+
+static int
+rk3066_hdmi_encoder_atomic_check(struct drm_encoder *encoder,
+ struct drm_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state)
+{
+ struct rockchip_crtc_state *s = to_rockchip_crtc_state(crtc_state);
+
+ s->output_mode = ROCKCHIP_OUT_MODE_P888;
+ s->output_type = DRM_MODE_CONNECTOR_HDMIA;
+
+ return 0;
+}
+
+static const
+struct drm_encoder_helper_funcs rk3066_hdmi_encoder_helper_funcs = {
+ .enable = rk3066_hdmi_encoder_enable,
+ .disable = rk3066_hdmi_encoder_disable,
+ .mode_fixup = rk3066_hdmi_encoder_mode_fixup,
+ .mode_set = rk3066_hdmi_encoder_mode_set,
+ .atomic_check = rk3066_hdmi_encoder_atomic_check,
+};
+
+static const struct drm_encoder_funcs rk3066_hdmi_encoder_funcs = {
+ .destroy = drm_encoder_cleanup,
+};
+
+static enum drm_connector_status
+rk3066_hdmi_connector_detect(struct drm_connector *connector, bool force)
+{
+ struct rk3066_hdmi *hdmi = to_rk3066_hdmi(connector);
+
+ return (hdmi_readb(hdmi, HDMI_HPG_MENS_STA) & HDMI_HPG_IN_STATUS_HIGH) ?
+ connector_status_connected : connector_status_disconnected;
+}
+
+static int rk3066_hdmi_connector_get_modes(struct drm_connector *connector)
+{
+ struct rk3066_hdmi *hdmi = to_rk3066_hdmi(connector);
+ struct edid *edid;
+ int ret = 0;
+
+ if (!hdmi->ddc)
+ return 0;
+
+ edid = drm_get_edid(connector, hdmi->ddc);
+ if (edid) {
+ hdmi->hdmi_data.sink_is_hdmi = drm_detect_hdmi_monitor(edid);
+ drm_connector_update_edid_property(connector, edid);
+ ret = drm_add_edid_modes(connector, edid);
+ kfree(edid);
+ }
+
+ return ret;
+}
+
+static enum drm_mode_status
+rk3066_hdmi_connector_mode_valid(struct drm_connector *connector,
+ struct drm_display_mode *mode)
+{
+ u32 vic = drm_match_cea_mode(mode);
+
+ if (vic > 1)
+ return MODE_OK;
+ else
+ return MODE_BAD;
+}
+
+static struct drm_encoder *
+rk3066_hdmi_connector_best_encoder(struct drm_connector *connector)
+{
+ struct rk3066_hdmi *hdmi = to_rk3066_hdmi(connector);
+
+ return &hdmi->encoder;
+}
+
+static int
+rk3066_hdmi_probe_single_connector_modes(struct drm_connector *connector,
+ uint32_t maxX, uint32_t maxY)
+{
+ if (maxX > 1920)
+ maxX = 1920;
+ if (maxY > 1080)
+ maxY = 1080;
+
+ return drm_helper_probe_single_connector_modes(connector, maxX, maxY);
+}
+
+static void rk3066_hdmi_connector_destroy(struct drm_connector *connector)
+{
+ drm_connector_unregister(connector);
+ drm_connector_cleanup(connector);
+}
+
+static const struct drm_connector_funcs rk3066_hdmi_connector_funcs = {
+ .fill_modes = rk3066_hdmi_probe_single_connector_modes,
+ .detect = rk3066_hdmi_connector_detect,
+ .destroy = rk3066_hdmi_connector_destroy,
+ .reset = drm_atomic_helper_connector_reset,
+ .atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state,
+ .atomic_destroy_state = drm_atomic_helper_connector_destroy_state,
+};
+
+static const
+struct drm_connector_helper_funcs rk3066_hdmi_connector_helper_funcs = {
+ .get_modes = rk3066_hdmi_connector_get_modes,
+ .mode_valid = rk3066_hdmi_connector_mode_valid,
+ .best_encoder = rk3066_hdmi_connector_best_encoder,
+};
+
+static int
+rk3066_hdmi_register(struct drm_device *drm, struct rk3066_hdmi *hdmi)
+{
+ struct drm_encoder *encoder = &hdmi->encoder;
+ struct device *dev = hdmi->dev;
+
+ encoder->possible_crtcs =
+ drm_of_find_possible_crtcs(drm, dev->of_node);
+
+ /*
+ * If we failed to find the CRTC(s) which this encoder is
+ * supposed to be connected to, it's because the CRTC has
+ * not been registered yet. Defer probing, and hope that
+ * the required CRTC is added later.
+ */
+ if (encoder->possible_crtcs == 0)
+ return -EPROBE_DEFER;
+
+ drm_encoder_helper_add(encoder, &rk3066_hdmi_encoder_helper_funcs);
+ drm_encoder_init(drm, encoder, &rk3066_hdmi_encoder_funcs,
+ DRM_MODE_ENCODER_TMDS, NULL);
+
+ hdmi->connector.polled = DRM_CONNECTOR_POLL_HPD;
+
+ drm_connector_helper_add(&hdmi->connector,
+ &rk3066_hdmi_connector_helper_funcs);
+ drm_connector_init(drm, &hdmi->connector,
+ &rk3066_hdmi_connector_funcs,
+ DRM_MODE_CONNECTOR_HDMIA);
+
+ drm_connector_attach_encoder(&hdmi->connector, encoder);
+
+ return 0;
+}
+
+static irqreturn_t rk3066_hdmi_hardirq(int irq, void *dev_id)
+{
+ struct rk3066_hdmi *hdmi = dev_id;
+ irqreturn_t ret = IRQ_NONE;
+ u8 interrupt;
+
+ if (rk3066_hdmi_get_power_mode(hdmi) == HDMI_SYS_POWER_MODE_A)
+ hdmi_writeb(hdmi, HDMI_SYS_CTRL, HDMI_SYS_POWER_MODE_B);
+
+ interrupt = hdmi_readb(hdmi, HDMI_INTR_STATUS1);
+ if (interrupt)
+ hdmi_writeb(hdmi, HDMI_INTR_STATUS1, interrupt);
+
+ if (interrupt & HDMI_INTR_EDID_MASK) {
+ hdmi->i2c->stat = interrupt;
+ complete(&hdmi->i2c->cmpltn);
+ }
+
+ if (interrupt & (HDMI_INTR_HOTPLUG | HDMI_INTR_MSENS))
+ ret = IRQ_WAKE_THREAD;
+
+ return ret;
+}
+
+static irqreturn_t rk3066_hdmi_irq(int irq, void *dev_id)
+{
+ struct rk3066_hdmi *hdmi = dev_id;
+
+ drm_helper_hpd_irq_event(hdmi->connector.dev);
+
+ return IRQ_HANDLED;
+}
+
+static int rk3066_hdmi_i2c_read(struct rk3066_hdmi *hdmi, struct i2c_msg *msgs)
+{
+ int length = msgs->len;
+ u8 *buf = msgs->buf;
+ int ret;
+
+ ret = wait_for_completion_timeout(&hdmi->i2c->cmpltn, HZ / 10);
+ if (!ret || hdmi->i2c->stat & HDMI_INTR_EDID_ERR)
+ return -EAGAIN;
+
+ while (length--)
+ *buf++ = hdmi_readb(hdmi, HDMI_DDC_READ_FIFO_ADDR);
+
+ return 0;
+}
+
+static int rk3066_hdmi_i2c_write(struct rk3066_hdmi *hdmi, struct i2c_msg *msgs)
+{
+ /*
+ * The DDC module only supports read EDID message, so
+ * we assume that each word write to this i2c adapter
+ * should be the offset of the EDID word address.
+ */
+ if (msgs->len != 1 ||
+ (msgs->addr != DDC_ADDR && msgs->addr != DDC_SEGMENT_ADDR))
+ return -EINVAL;
+
+ reinit_completion(&hdmi->i2c->cmpltn);
+
+ if (msgs->addr == DDC_SEGMENT_ADDR)
+ hdmi->i2c->segment_addr = msgs->buf[0];
+ if (msgs->addr == DDC_ADDR)
+ hdmi->i2c->ddc_addr = msgs->buf[0];
+
+ /* Set edid word address 0x00/0x80. */
+ hdmi_writeb(hdmi, HDMI_EDID_WORD_ADDR, hdmi->i2c->ddc_addr);
+
+ /* Set edid segment pointer. */
+ hdmi_writeb(hdmi, HDMI_EDID_SEGMENT_POINTER, hdmi->i2c->segment_addr);
+
+ return 0;
+}
+
+static int rk3066_hdmi_i2c_xfer(struct i2c_adapter *adap,
+ struct i2c_msg *msgs, int num)
+{
+ struct rk3066_hdmi *hdmi = i2c_get_adapdata(adap);
+ struct rk3066_hdmi_i2c *i2c = hdmi->i2c;
+ int i, ret = 0;
+
+ mutex_lock(&i2c->i2c_lock);
+
+ rk3066_hdmi_i2c_init(hdmi);
+
+ /* Unmute HDMI EDID interrupt. */
+ hdmi_modb(hdmi, HDMI_INTR_MASK1,
+ HDMI_INTR_EDID_MASK, HDMI_INTR_EDID_MASK);
+ i2c->stat = 0;
+
+ for (i = 0; i < num; i++) {
+ DRM_DEV_DEBUG(hdmi->dev,
+ "xfer: num: %d/%d, len: %d, flags: %#x\n",
+ i + 1, num, msgs[i].len, msgs[i].flags);
+
+ if (msgs[i].flags & I2C_M_RD)
+ ret = rk3066_hdmi_i2c_read(hdmi, &msgs[i]);
+ else
+ ret = rk3066_hdmi_i2c_write(hdmi, &msgs[i]);
+
+ if (ret < 0)
+ break;
+ }
+
+ if (!ret)
+ ret = num;
+
+ /* Mute HDMI EDID interrupt. */
+ hdmi_modb(hdmi, HDMI_INTR_MASK1, HDMI_INTR_EDID_MASK, 0);
+
+ mutex_unlock(&i2c->i2c_lock);
+
+ return ret;
+}
+
+static u32 rk3066_hdmi_i2c_func(struct i2c_adapter *adapter)
+{
+ return I2C_FUNC_I2C | I2C_FUNC_SMBUS_EMUL;
+}
+
+static const struct i2c_algorithm rk3066_hdmi_algorithm = {
+ .master_xfer = rk3066_hdmi_i2c_xfer,
+ .functionality = rk3066_hdmi_i2c_func,
+};
+
+static struct i2c_adapter *rk3066_hdmi_i2c_adapter(struct rk3066_hdmi *hdmi)
+{
+ struct i2c_adapter *adap;
+ struct rk3066_hdmi_i2c *i2c;
+ int ret;
+
+ i2c = devm_kzalloc(hdmi->dev, sizeof(*i2c), GFP_KERNEL);
+ if (!i2c)
+ return ERR_PTR(-ENOMEM);
+
+ mutex_init(&i2c->i2c_lock);
+ init_completion(&i2c->cmpltn);
+
+ adap = &i2c->adap;
+ adap->class = I2C_CLASS_DDC;
+ adap->owner = THIS_MODULE;
+ adap->dev.parent = hdmi->dev;
+ adap->dev.of_node = hdmi->dev->of_node;
+ adap->algo = &rk3066_hdmi_algorithm;
+ strlcpy(adap->name, "RK3066 HDMI", sizeof(adap->name));
+ i2c_set_adapdata(adap, hdmi);
+
+ ret = i2c_add_adapter(adap);
+ if (ret) {
+ DRM_DEV_ERROR(hdmi->dev, "cannot add %s I2C adapter\n",
+ adap->name);
+ devm_kfree(hdmi->dev, i2c);
+ return ERR_PTR(ret);
+ }
+
+ hdmi->i2c = i2c;
+
+ DRM_DEV_DEBUG(hdmi->dev, "registered %s I2C bus driver\n", adap->name);
+
+ return adap;
+}
+
+static int rk3066_hdmi_bind(struct device *dev, struct device *master,
+ void *data)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct drm_device *drm = data;
+ struct rk3066_hdmi *hdmi;
+ struct resource *iores;
+ int irq;
+ int ret;
+
+ hdmi = devm_kzalloc(dev, sizeof(*hdmi), GFP_KERNEL);
+ if (!hdmi)
+ return -ENOMEM;
+
+ hdmi->dev = dev;
+ hdmi->drm_dev = drm;
+
+ iores = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!iores)
+ return -ENXIO;
+
+ hdmi->regs = devm_ioremap_resource(dev, iores);
+ if (IS_ERR(hdmi->regs))
+ return PTR_ERR(hdmi->regs);
+
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0)
+ return irq;
+
+ hdmi->hclk = devm_clk_get(dev, "hclk");
+ if (IS_ERR(hdmi->hclk)) {
+ DRM_DEV_ERROR(dev, "unable to get HDMI hclk clock\n");
+ return PTR_ERR(hdmi->hclk);
+ }
+
+ ret = clk_prepare_enable(hdmi->hclk);
+ if (ret) {
+ DRM_DEV_ERROR(dev, "cannot enable HDMI hclk clock: %d\n", ret);
+ return ret;
+ }
+
+ hdmi->grf_regmap = syscon_regmap_lookup_by_phandle(dev->of_node,
+ "rockchip,grf");
+ if (IS_ERR(hdmi->grf_regmap)) {
+ DRM_DEV_ERROR(dev, "unable to get rockchip,grf\n");
+ ret = PTR_ERR(hdmi->grf_regmap);
+ goto err_disable_hclk;
+ }
+
+ /* internal hclk = hdmi_hclk / 25 */
+ hdmi_writeb(hdmi, HDMI_INTERNAL_CLK_DIVIDER, 25);
+
+ hdmi->ddc = rk3066_hdmi_i2c_adapter(hdmi);
+ if (IS_ERR(hdmi->ddc)) {
+ ret = PTR_ERR(hdmi->ddc);
+ hdmi->ddc = NULL;
+ goto err_disable_hclk;
+ }
+
+ rk3066_hdmi_set_power_mode(hdmi, HDMI_SYS_POWER_MODE_B);
+ usleep_range(999, 1000);
+ hdmi_writeb(hdmi, HDMI_INTR_MASK1, HDMI_INTR_HOTPLUG);
+ hdmi_writeb(hdmi, HDMI_INTR_MASK2, 0);
+ hdmi_writeb(hdmi, HDMI_INTR_MASK3, 0);
+ hdmi_writeb(hdmi, HDMI_INTR_MASK4, 0);
+ rk3066_hdmi_set_power_mode(hdmi, HDMI_SYS_POWER_MODE_A);
+
+ ret = rk3066_hdmi_register(drm, hdmi);
+ if (ret)
+ goto err_disable_i2c;
+
+ dev_set_drvdata(dev, hdmi);
+
+ ret = devm_request_threaded_irq(dev, irq, rk3066_hdmi_hardirq,
+ rk3066_hdmi_irq, IRQF_SHARED,
+ dev_name(dev), hdmi);
+ if (ret) {
+ DRM_DEV_ERROR(dev, "failed to request hdmi irq: %d\n", ret);
+ goto err_cleanup_hdmi;
+ }
+
+ return 0;
+
+err_cleanup_hdmi:
+ hdmi->connector.funcs->destroy(&hdmi->connector);
+ hdmi->encoder.funcs->destroy(&hdmi->encoder);
+err_disable_i2c:
+ i2c_put_adapter(hdmi->ddc);
+err_disable_hclk:
+ clk_disable_unprepare(hdmi->hclk);
+
+ return ret;
+}
+
+static void rk3066_hdmi_unbind(struct device *dev, struct device *master,
+ void *data)
+{
+ struct rk3066_hdmi *hdmi = dev_get_drvdata(dev);
+
+ hdmi->connector.funcs->destroy(&hdmi->connector);
+ hdmi->encoder.funcs->destroy(&hdmi->encoder);
+
+ i2c_put_adapter(hdmi->ddc);
+ clk_disable_unprepare(hdmi->hclk);
+}
+
+static const struct component_ops rk3066_hdmi_ops = {
+ .bind = rk3066_hdmi_bind,
+ .unbind = rk3066_hdmi_unbind,
+};
+
+static int rk3066_hdmi_probe(struct platform_device *pdev)
+{
+ return component_add(&pdev->dev, &rk3066_hdmi_ops);
+}
+
+static int rk3066_hdmi_remove(struct platform_device *pdev)
+{
+ component_del(&pdev->dev, &rk3066_hdmi_ops);
+
+ return 0;
+}
+
+static const struct of_device_id rk3066_hdmi_dt_ids[] = {
+ { .compatible = "rockchip,rk3066-hdmi" },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, rk3066_hdmi_dt_ids);
+
+struct platform_driver rk3066_hdmi_driver = {
+ .probe = rk3066_hdmi_probe,
+ .remove = rk3066_hdmi_remove,
+ .driver = {
+ .name = "rockchip-rk3066-hdmi",
+ .of_match_table = rk3066_hdmi_dt_ids,
+ },
+};
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) Fuzhou Rockchip Electronics Co.Ltd
+ * Zheng Yang <zhengyang@rock-chips.com>
+ */
+
+#ifndef __RK3066_HDMI_H__
+#define __RK3066_HDMI_H__
+
+#define GRF_SOC_CON0 0x150
+#define HDMI_VIDEO_SEL BIT(14)
+
+#define DDC_SEGMENT_ADDR 0x30
+#define HDMI_SCL_RATE (50 * 1000)
+#define HDMI_MAXIMUM_INFO_FRAME_SIZE 0x11
+
+#define N_32K 0x1000
+#define N_441K 0x1880
+#define N_882K 0x3100
+#define N_1764K 0x6200
+#define N_48K 0x1800
+#define N_96K 0x3000
+#define N_192K 0x6000
+
+#define HDMI_SYS_CTRL 0x000
+#define HDMI_LR_SWAP_N3 0x004
+#define HDMI_N2 0x008
+#define HDMI_N1 0x00c
+#define HDMI_SPDIF_FS_CTS_INT3 0x010
+#define HDMI_CTS_INT2 0x014
+#define HDMI_CTS_INT1 0x018
+#define HDMI_CTS_EXT3 0x01c
+#define HDMI_CTS_EXT2 0x020
+#define HDMI_CTS_EXT1 0x024
+#define HDMI_AUDIO_CTRL1 0x028
+#define HDMI_AUDIO_CTRL2 0x02c
+#define HDMI_I2S_AUDIO_CTRL 0x030
+#define HDMI_I2S_SWAP 0x040
+#define HDMI_AUDIO_STA_BIT_CTRL1 0x044
+#define HDMI_AUDIO_STA_BIT_CTRL2 0x048
+#define HDMI_AUDIO_SRC_NUM_AND_LENGTH 0x050
+#define HDMI_AV_CTRL1 0x054
+#define HDMI_VIDEO_CTRL1 0x058
+#define HDMI_DEEP_COLOR_MODE 0x05c
+
+#define HDMI_EXT_VIDEO_PARA 0x0c0
+#define HDMI_EXT_HTOTAL_L 0x0c4
+#define HDMI_EXT_HTOTAL_H 0x0c8
+#define HDMI_EXT_HBLANK_L 0x0cc
+#define HDMI_EXT_HBLANK_H 0x0d0
+#define HDMI_EXT_HDELAY_L 0x0d4
+#define HDMI_EXT_HDELAY_H 0x0d8
+#define HDMI_EXT_HDURATION_L 0x0dc
+#define HDMI_EXT_HDURATION_H 0x0e0
+#define HDMI_EXT_VTOTAL_L 0x0e4
+#define HDMI_EXT_VTOTAL_H 0x0e8
+#define HDMI_AV_CTRL2 0x0ec
+#define HDMI_EXT_VBLANK_L 0x0f4
+#define HDMI_EXT_VBLANK_H 0x10c
+#define HDMI_EXT_VDELAY 0x0f8
+#define HDMI_EXT_VDURATION 0x0fc
+
+#define HDMI_CP_MANU_SEND_CTRL 0x100
+#define HDMI_CP_AUTO_SEND_CTRL 0x104
+#define HDMI_AUTO_CHECKSUM_OPT 0x108
+
+#define HDMI_VIDEO_CTRL2 0x114
+
+#define HDMI_PHY_OPTION 0x144
+
+#define HDMI_CP_BUF_INDEX 0x17c
+#define HDMI_CP_BUF_ACC_HB0 0x180
+#define HDMI_CP_BUF_ACC_HB1 0x184
+#define HDMI_CP_BUF_ACC_HB2 0x188
+#define HDMI_CP_BUF_ACC_PB0 0x18c
+
+#define HDMI_DDC_READ_FIFO_ADDR 0x200
+#define HDMI_DDC_BUS_FREQ_L 0x204
+#define HDMI_DDC_BUS_FREQ_H 0x208
+#define HDMI_DDC_BUS_CTRL 0x2dc
+#define HDMI_DDC_I2C_LEN 0x278
+#define HDMI_DDC_I2C_OFFSET 0x280
+#define HDMI_DDC_I2C_CTRL 0x284
+#define HDMI_DDC_I2C_READ_BUF0 0x288
+#define HDMI_DDC_I2C_READ_BUF1 0x28c
+#define HDMI_DDC_I2C_READ_BUF2 0x290
+#define HDMI_DDC_I2C_READ_BUF3 0x294
+#define HDMI_DDC_I2C_WRITE_BUF0 0x298
+#define HDMI_DDC_I2C_WRITE_BUF1 0x29c
+#define HDMI_DDC_I2C_WRITE_BUF2 0x2a0
+#define HDMI_DDC_I2C_WRITE_BUF3 0x2a4
+#define HDMI_DDC_I2C_WRITE_BUF4 0x2ac
+#define HDMI_DDC_I2C_WRITE_BUF5 0x2b0
+#define HDMI_DDC_I2C_WRITE_BUF6 0x2b4
+
+#define HDMI_INTR_MASK1 0x248
+#define HDMI_INTR_MASK2 0x24c
+#define HDMI_INTR_STATUS1 0x250
+#define HDMI_INTR_STATUS2 0x254
+#define HDMI_INTR_MASK3 0x258
+#define HDMI_INTR_MASK4 0x25c
+#define HDMI_INTR_STATUS3 0x260
+#define HDMI_INTR_STATUS4 0x264
+
+#define HDMI_HDCP_CTRL 0x2bc
+
+#define HDMI_EDID_SEGMENT_POINTER 0x310
+#define HDMI_EDID_WORD_ADDR 0x314
+#define HDMI_EDID_FIFO_ADDR 0x318
+
+#define HDMI_HPG_MENS_STA 0x37c
+
+#define HDMI_INTERNAL_CLK_DIVIDER 0x800
+
+enum {
+ /* HDMI_SYS_CTRL */
+ HDMI_SYS_POWER_MODE_MASK = 0xf0,
+ HDMI_SYS_POWER_MODE_A = 0x10,
+ HDMI_SYS_POWER_MODE_B = 0x20,
+ HDMI_SYS_POWER_MODE_D = 0x40,
+ HDMI_SYS_POWER_MODE_E = 0x80,
+ HDMI_SYS_PLL_RESET_MASK = 0x0c,
+ HDMI_SYS_PLL_RESET = 0x0c,
+ HDMI_SYS_PLLB_RESET = 0x08,
+
+ /* HDMI_LR_SWAP_N3 */
+ HDMI_AUDIO_LR_SWAP_MASK = 0xf0,
+ HDMI_AUDIO_LR_SWAP_SUBPACKET0 = 0x10,
+ HDMI_AUDIO_LR_SWAP_SUBPACKET1 = 0x20,
+ HDMI_AUDIO_LR_SWAP_SUBPACKET2 = 0x40,
+ HDMI_AUDIO_LR_SWAP_SUBPACKET3 = 0x80,
+ HDMI_AUDIO_N_19_16_MASK = 0x0f,
+
+ /* HDMI_AUDIO_CTRL1 */
+ HDMI_AUDIO_EXTERNAL_CTS = BIT(7),
+ HDMI_AUDIO_INPUT_IIS = 0,
+ HDMI_AUDIO_INPUT_SPDIF = 0x08,
+ HDMI_AUDIO_INPUT_MCLK_ACTIVE = 0x04,
+ HDMI_AUDIO_INPUT_MCLK_DEACTIVE = 0,
+ HDMI_AUDIO_INPUT_MCLK_RATE_128X = 0,
+ HDMI_AUDIO_INPUT_MCLK_RATE_256X = 1,
+ HDMI_AUDIO_INPUT_MCLK_RATE_384X = 2,
+ HDMI_AUDIO_INPUT_MCLK_RATE_512X = 3,
+
+ /* HDMI_I2S_AUDIO_CTRL */
+ HDMI_AUDIO_I2S_FORMAT_STANDARD = 0,
+ HDMI_AUDIO_I2S_CHANNEL_1_2 = 0x04,
+ HDMI_AUDIO_I2S_CHANNEL_3_4 = 0x0c,
+ HDMI_AUDIO_I2S_CHANNEL_5_6 = 0x1c,
+ HDMI_AUDIO_I2S_CHANNEL_7_8 = 0x3c,
+
+ /* HDMI_AV_CTRL1 */
+ HDMI_AUDIO_SAMPLE_FRE_MASK = 0xf0,
+ HDMI_AUDIO_SAMPLE_FRE_32000 = 0x30,
+ HDMI_AUDIO_SAMPLE_FRE_44100 = 0,
+ HDMI_AUDIO_SAMPLE_FRE_48000 = 0x20,
+ HDMI_AUDIO_SAMPLE_FRE_88200 = 0x80,
+ HDMI_AUDIO_SAMPLE_FRE_96000 = 0xa0,
+ HDMI_AUDIO_SAMPLE_FRE_176400 = 0xc0,
+ HDMI_AUDIO_SAMPLE_FRE_192000 = 0xe0,
+ HDMI_AUDIO_SAMPLE_FRE_768000 = 0x90,
+
+ HDMI_VIDEO_INPUT_FORMAT_MASK = 0x0e,
+ HDMI_VIDEO_INPUT_RGB_YCBCR444 = 0,
+ HDMI_VIDEO_INPUT_YCBCR422 = 0x02,
+ HDMI_VIDEO_DE_MASK = 0x1,
+ HDMI_VIDEO_INTERNAL_DE = 0,
+ HDMI_VIDEO_EXTERNAL_DE = 0x01,
+
+ /* HDMI_VIDEO_CTRL1 */
+ HDMI_VIDEO_OUTPUT_FORMAT_MASK = 0xc0,
+ HDMI_VIDEO_OUTPUT_RGB444 = 0,
+ HDMI_VIDEO_OUTPUT_YCBCR444 = 0x40,
+ HDMI_VIDEO_OUTPUT_YCBCR422 = 0x80,
+ HDMI_VIDEO_INPUT_DATA_DEPTH_MASK = 0x30,
+ HDMI_VIDEO_INPUT_DATA_DEPTH_12BIT = 0,
+ HDMI_VIDEO_INPUT_DATA_DEPTH_10BIT = 0x10,
+ HDMI_VIDEO_INPUT_DATA_DEPTH_8BIT = 0x30,
+ HDMI_VIDEO_INPUT_COLOR_MASK = 1,
+ HDMI_VIDEO_INPUT_COLOR_RGB = 0,
+ HDMI_VIDEO_INPUT_COLOR_YCBCR = 1,
+
+ /* HDMI_EXT_VIDEO_PARA */
+ HDMI_VIDEO_VSYNC_OFFSET_SHIFT = 4,
+ HDMI_VIDEO_VSYNC_ACTIVE_HIGH = BIT(3),
+ HDMI_VIDEO_VSYNC_ACTIVE_LOW = 0,
+ HDMI_VIDEO_HSYNC_ACTIVE_HIGH = BIT(2),
+ HDMI_VIDEO_HSYNC_ACTIVE_LOW = 0,
+ HDMI_VIDEO_MODE_INTERLACE = BIT(1),
+ HDMI_VIDEO_MODE_PROGRESSIVE = 0,
+ HDMI_EXT_VIDEO_SET_EN = BIT(0),
+
+ /* HDMI_CP_AUTO_SEND_CTRL */
+
+ /* HDMI_VIDEO_CTRL2 */
+ HDMI_VIDEO_AV_MUTE_MASK = 0xc0,
+ HDMI_VIDEO_CLR_AV_MUTE = BIT(7),
+ HDMI_VIDEO_SET_AV_MUTE = BIT(6),
+ HDMI_AUDIO_CP_LOGIC_RESET_MASK = BIT(2),
+ HDMI_AUDIO_CP_LOGIC_RESET = BIT(2),
+ HDMI_VIDEO_AUDIO_DISABLE_MASK = 0x3,
+ HDMI_AUDIO_DISABLE = BIT(1),
+ HDMI_VIDEO_DISABLE = BIT(0),
+
+ /* HDMI_CP_BUF_INDEX */
+ HDMI_INFOFRAME_VSI = 0x05,
+ HDMI_INFOFRAME_AVI = 0x06,
+ HDMI_INFOFRAME_AAI = 0x08,
+
+ /* HDMI_INTR_MASK1 */
+ /* HDMI_INTR_STATUS1 */
+ HDMI_INTR_HOTPLUG = BIT(7),
+ HDMI_INTR_MSENS = BIT(6),
+ HDMI_INTR_VSYNC = BIT(5),
+ HDMI_INTR_AUDIO_FIFO_FULL = BIT(4),
+ HDMI_INTR_EDID_MASK = 0x6,
+ HDMI_INTR_EDID_READY = BIT(2),
+ HDMI_INTR_EDID_ERR = BIT(1),
+
+ /* HDMI_HDCP_CTRL */
+ HDMI_VIDEO_MODE_MASK = BIT(1),
+ HDMI_VIDEO_MODE_HDMI = BIT(1),
+
+ /* HDMI_HPG_MENS_STA */
+ HDMI_HPG_IN_STATUS_HIGH = BIT(7),
+ HDMI_MSENS_IN_STATUS_HIGH = BIT(6),
+};
+
+#endif /* __RK3066_HDMI_H__ */
return 0;
}
+static void rockchip_drm_platform_shutdown(struct platform_device *pdev)
+{
+ struct drm_device *drm = platform_get_drvdata(pdev);
+
+ if (drm)
+ drm_atomic_helper_shutdown(drm);
+}
+
static const struct of_device_id rockchip_drm_dt_ids[] = {
{ .compatible = "rockchip,display-subsystem", },
{ /* sentinel */ },
static struct platform_driver rockchip_drm_platform_driver = {
.probe = rockchip_drm_platform_probe,
.remove = rockchip_drm_platform_remove,
+ .shutdown = rockchip_drm_platform_shutdown,
.driver = {
.name = "rockchip-drm",
.of_match_table = rockchip_drm_dt_ids,
ADD_ROCKCHIP_SUB_DRIVER(dw_mipi_dsi_rockchip_driver,
CONFIG_ROCKCHIP_DW_MIPI_DSI);
ADD_ROCKCHIP_SUB_DRIVER(inno_hdmi_driver, CONFIG_ROCKCHIP_INNO_HDMI);
+ ADD_ROCKCHIP_SUB_DRIVER(rk3066_hdmi_driver,
+ CONFIG_ROCKCHIP_RK3066_HDMI);
ret = platform_register_drivers(rockchip_sub_drivers,
num_rockchip_sub_drivers);
extern struct platform_driver rockchip_dp_driver;
extern struct platform_driver rockchip_lvds_driver;
extern struct platform_driver vop_platform_driver;
+extern struct platform_driver rk3066_hdmi_driver;
#endif /* _ROCKCHIP_DRM_DRV_H_ */
goto out;
}
- fbi->par = helper;
fbi->fbops = &rockchip_drm_fbdev_ops;
fb = helper->fb;
- drm_fb_helper_fill_fix(fbi, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(fbi, helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(fbi, helper, sizes);
offset = fbi->var.xoffset * bytes_per_pixel;
offset += fbi->var.yoffset * fb->pitches[0];
rk_obj->kvaddr,
offset, size);
- fbi->skip_vt_switch = true;
-
return 0;
out:
u16 vact_st = adjusted_mode->vtotal - adjusted_mode->vsync_start;
u16 vact_end = vact_st + vdisplay;
uint32_t pin_pol, val;
+ int dither_bpc = s->output_bpc ? s->output_bpc : 10;
int ret;
mutex_lock(&vop->vop_lock);
!(vop_data->feature & VOP_FEATURE_OUTPUT_RGB10))
s->output_mode = ROCKCHIP_OUT_MODE_P888;
- if (s->output_mode == ROCKCHIP_OUT_MODE_AAAA && s->output_bpc == 8)
+ if (s->output_mode == ROCKCHIP_OUT_MODE_AAAA && dither_bpc <= 8)
VOP_REG_SET(vop, common, pre_dither_down, 1);
else
VOP_REG_SET(vop, common, pre_dither_down, 0);
+ if (dither_bpc == 6) {
+ VOP_REG_SET(vop, common, dither_down_sel, DITHER_DOWN_ALLEGRO);
+ VOP_REG_SET(vop, common, dither_down_mode, RGB888_TO_RGB666);
+ VOP_REG_SET(vop, common, dither_down_en, 1);
+ } else {
+ VOP_REG_SET(vop, common, dither_down_en, 0);
+ }
+
VOP_REG_SET(vop, common, out_mode, s->output_mode);
VOP_REG_SET(vop, modeset, htotal_pw, (htotal << 16) | hsync_len);
struct vop_reg dsp_blank;
struct vop_reg data_blank;
struct vop_reg pre_dither_down;
- struct vop_reg dither_down;
+ struct vop_reg dither_down_sel;
+ struct vop_reg dither_down_mode;
+ struct vop_reg dither_down_en;
struct vop_reg dither_up;
struct vop_reg gate_en;
struct vop_reg mmu_en;
SCALE_DOWN_AVG = 0x1
};
+enum dither_down_mode {
+ RGB888_TO_RGB565 = 0x0,
+ RGB888_TO_RGB666 = 0x1
+};
+
+enum dither_down_mode_sel {
+ DITHER_DOWN_ALLEGRO = 0x0,
+ DITHER_DOWN_FRC = 0x1
+};
+
enum vop_pol {
HSYNC_POSITIVE = 0,
VSYNC_POSITIVE = 1,
.standby = VOP_REG_SYNC(RK3036_SYS_CTRL, 0x1, 30),
.out_mode = VOP_REG(RK3036_DSP_CTRL0, 0xf, 0),
.dsp_blank = VOP_REG(RK3036_DSP_CTRL1, 0x1, 24),
+ .dither_down_sel = VOP_REG(RK3036_DSP_CTRL0, 0x1, 27),
+ .dither_down_en = VOP_REG(RK3036_DSP_CTRL0, 0x1, 11),
+ .dither_down_mode = VOP_REG(RK3036_DSP_CTRL0, 0x1, 10),
.cfg_done = VOP_REG_SYNC(RK3036_REG_CFG_DONE, 0x1, 0),
};
.standby = VOP_REG_SYNC(PX30_SYS_CTRL2, 0x1, 1),
.out_mode = VOP_REG(PX30_DSP_CTRL2, 0xf, 16),
.dsp_blank = VOP_REG(PX30_DSP_CTRL2, 0x1, 14),
+ .dither_down_en = VOP_REG(PX30_DSP_CTRL2, 0x1, 8),
+ .dither_down_sel = VOP_REG(PX30_DSP_CTRL2, 0x1, 7),
+ .dither_down_mode = VOP_REG(PX30_DSP_CTRL2, 0x1, 6),
.cfg_done = VOP_REG_SYNC(PX30_REG_CFG_DONE, 0x1, 0),
};
.standby = VOP_REG(RK3066_SYS_CTRL0, 0x1, 1),
.out_mode = VOP_REG(RK3066_DSP_CTRL0, 0xf, 0),
.cfg_done = VOP_REG(RK3066_REG_CFG_DONE, 0x1, 0),
+ .dither_down_en = VOP_REG(RK3066_DSP_CTRL0, 0x1, 11),
+ .dither_down_mode = VOP_REG(RK3066_DSP_CTRL0, 0x1, 10),
.dsp_blank = VOP_REG(RK3066_DSP_CTRL1, 0x1, 24),
};
.standby = VOP_REG(RK3188_SYS_CTRL, 0x1, 30),
.out_mode = VOP_REG(RK3188_DSP_CTRL0, 0xf, 0),
.cfg_done = VOP_REG(RK3188_REG_CFG_DONE, 0x1, 0),
+ .dither_down_sel = VOP_REG(RK3188_DSP_CTRL0, 0x1, 27),
+ .dither_down_en = VOP_REG(RK3188_DSP_CTRL0, 0x1, 11),
+ .dither_down_mode = VOP_REG(RK3188_DSP_CTRL0, 0x1, 10),
.dsp_blank = VOP_REG(RK3188_DSP_CTRL1, 0x3, 24),
};
.standby = VOP_REG_SYNC(RK3288_SYS_CTRL, 0x1, 22),
.gate_en = VOP_REG(RK3288_SYS_CTRL, 0x1, 23),
.mmu_en = VOP_REG(RK3288_SYS_CTRL, 0x1, 20),
+ .dither_down_sel = VOP_REG(RK3288_DSP_CTRL1, 0x1, 4),
+ .dither_down_mode = VOP_REG(RK3288_DSP_CTRL1, 0x1, 3),
+ .dither_down_en = VOP_REG(RK3288_DSP_CTRL1, 0x1, 2),
.pre_dither_down = VOP_REG(RK3288_DSP_CTRL1, 0x1, 1),
- .dither_down = VOP_REG(RK3288_DSP_CTRL1, 0xf, 1),
.dither_up = VOP_REG(RK3288_DSP_CTRL1, 0x1, 6),
.data_blank = VOP_REG(RK3288_DSP_CTRL0, 0x1, 19),
.dsp_blank = VOP_REG(RK3288_DSP_CTRL0, 0x3, 18),
static const struct vop_common rk3328_common = {
.standby = VOP_REG_SYNC(RK3328_SYS_CTRL, 0x1, 22),
- .dither_down = VOP_REG(RK3328_DSP_CTRL1, 0xf, 1),
+ .dither_down_sel = VOP_REG(RK3328_DSP_CTRL1, 0x1, 4),
+ .dither_down_mode = VOP_REG(RK3328_DSP_CTRL1, 0x1, 3),
+ .dither_down_en = VOP_REG(RK3328_DSP_CTRL1, 0x1, 2),
+ .pre_dither_down = VOP_REG(RK3328_DSP_CTRL1, 0x1, 1),
.dither_up = VOP_REG(RK3328_DSP_CTRL1, 0x1, 6),
.dsp_blank = VOP_REG(RK3328_DSP_CTRL0, 0x3, 18),
.out_mode = VOP_REG(RK3328_DSP_CTRL0, 0xf, 0),
DRM_RND_STATE(prng, random_seed);
const unsigned int count = 8192;
unsigned int size;
- unsigned long *bitmap = NULL;
+ unsigned long *bitmap;
struct drm_mm mm;
struct drm_mm_node *nodes, *node, *next;
unsigned int *order, n, m, o = 0;
if (!nodes)
goto err;
- bitmap = kcalloc(count / BITS_PER_LONG, sizeof(unsigned long),
- GFP_KERNEL);
+ bitmap = bitmap_zalloc(count, GFP_KERNEL);
if (!bitmap)
goto err_nodes;
drm_mm_takedown(&mm);
kfree(order);
err_bitmap:
- kfree(bitmap);
+ bitmap_free(bitmap);
err_nodes:
vfree(nodes);
err:
if (!nodes)
goto err;
- bitmap = kcalloc(count / BITS_PER_LONG, sizeof(unsigned long),
- GFP_KERNEL);
+ bitmap = bitmap_zalloc(count, GFP_KERNEL);
if (!bitmap)
goto err_nodes;
drm_mm_takedown(&mm);
kfree(order);
err_bitmap:
- kfree(bitmap);
+ bitmap_free(bitmap);
err_nodes:
vfree(nodes);
err:
select DRM_KMS_CMA_HELPER
select DRM_PANEL_BRIDGE
select VIDEOMODE_HELPERS
- select FB_PROVIDE_GET_FB_UNMAPPED_AREA
+ select FB_PROVIDE_GET_FB_UNMAPPED_AREA if FB
help
Enable support for the on-chip display controller on
drm_mode_config_cleanup(ddev);
}
+static __maybe_unused int drv_suspend(struct device *dev)
+{
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct ltdc_device *ldev = ddev->dev_private;
+ struct drm_atomic_state *state;
+
+ drm_kms_helper_poll_disable(ddev);
+ state = drm_atomic_helper_suspend(ddev);
+ if (IS_ERR(state)) {
+ drm_kms_helper_poll_enable(ddev);
+ return PTR_ERR(state);
+ }
+ ldev->suspend_state = state;
+ ltdc_suspend(ddev);
+
+ return 0;
+}
+
+static __maybe_unused int drv_resume(struct device *dev)
+{
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct ltdc_device *ldev = ddev->dev_private;
+
+ ltdc_resume(ddev);
+ drm_atomic_helper_resume(ddev, ldev->suspend_state);
+ drm_kms_helper_poll_enable(ddev);
+
+ return 0;
+}
+
+static const struct dev_pm_ops drv_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(drv_suspend, drv_resume)
+};
+
static int stm_drm_platform_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
.driver = {
.name = "stm32-display",
.of_match_table = drv_dt_ids,
+ .pm = &drv_pm_ops,
},
};
return 0;
}
+static int __maybe_unused dw_mipi_dsi_stm_suspend(struct device *dev)
+{
+ struct dw_mipi_dsi_stm *dsi = dw_mipi_dsi_stm_plat_data.priv_data;
+
+ DRM_DEBUG_DRIVER("\n");
+
+ clk_disable_unprepare(dsi->pllref_clk);
+
+ return 0;
+}
+
+static int __maybe_unused dw_mipi_dsi_stm_resume(struct device *dev)
+{
+ struct dw_mipi_dsi_stm *dsi = dw_mipi_dsi_stm_plat_data.priv_data;
+
+ DRM_DEBUG_DRIVER("\n");
+
+ clk_prepare_enable(dsi->pllref_clk);
+
+ return 0;
+}
+
+static const struct dev_pm_ops dw_mipi_dsi_stm_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(dw_mipi_dsi_stm_suspend,
+ dw_mipi_dsi_stm_resume)
+};
+
static struct platform_driver dw_mipi_dsi_stm_driver = {
.probe = dw_mipi_dsi_stm_probe,
.remove = dw_mipi_dsi_stm_remove,
.driver = {
.of_match_table = dw_mipi_dsi_stm_dt_ids,
.name = "stm32-display-dsi",
+ .pm = &dw_mipi_dsi_stm_pm_ops,
},
};
return 0;
}
+void ltdc_suspend(struct drm_device *ddev)
+{
+ struct ltdc_device *ldev = ddev->dev_private;
+
+ DRM_DEBUG_DRIVER("\n");
+ clk_disable_unprepare(ldev->pixel_clk);
+}
+
+int ltdc_resume(struct drm_device *ddev)
+{
+ struct ltdc_device *ldev = ddev->dev_private;
+ int ret;
+
+ DRM_DEBUG_DRIVER("\n");
+
+ ret = clk_prepare_enable(ldev->pixel_clk);
+ if (ret) {
+ DRM_ERROR("failed to enable pixel clock (%d)\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
int ltdc_load(struct drm_device *ddev)
{
struct platform_device *pdev = to_platform_device(ddev->dev);
u32 error_status;
u32 irq_status;
struct fps_info plane_fpsi[LTDC_MAX_LAYER];
+ struct drm_atomic_state *suspend_state;
};
bool ltdc_crtc_scanoutpos(struct drm_device *dev, unsigned int pipe,
int ltdc_load(struct drm_device *ddev);
void ltdc_unload(struct drm_device *ddev);
+void ltdc_suspend(struct drm_device *ddev);
+int ltdc_resume(struct drm_device *ddev);
#endif
paddr = drm_fb_cma_get_gem_addr(fb, state, 0);
DRM_DEBUG_DRIVER("Setting buffer address to %pad\n", &paddr);
- /*
- * backend DMA accesses DRAM directly, bypassing the system
- * bus. As such, the address range is different and the buffer
- * address needs to be corrected.
- */
- paddr -= PHYS_OFFSET;
-
if (fb->format->is_yuv)
return sun4i_backend_update_yuv_buffer(backend, fb, paddr);
*/
static int sun4i_backend_of_get_id(struct device_node *node)
{
- struct device_node *port, *ep;
- int ret = -EINVAL;
+ struct device_node *ep, *remote;
+ struct of_endpoint of_ep;
- /* input is port 0 */
- port = of_graph_get_port_by_id(node, 0);
- if (!port)
+ /* Input port is 0, and we want the first endpoint. */
+ ep = of_graph_get_endpoint_by_regs(node, 0, -1);
+ if (!ep)
return -EINVAL;
- /* try finding an upstream endpoint */
- for_each_available_child_of_node(port, ep) {
- struct device_node *remote;
- u32 reg;
-
- remote = of_graph_get_remote_endpoint(ep);
- if (!remote)
- continue;
-
- ret = of_property_read_u32(remote, "reg", ®);
- if (ret)
- continue;
-
- ret = reg;
- }
-
- of_node_put(port);
+ remote = of_graph_get_remote_endpoint(ep);
+ of_node_put(ep);
+ if (!remote)
+ return -EINVAL;
- return ret;
+ of_graph_parse_endpoint(remote, &of_ep);
+ of_node_put(remote);
+ return of_ep.id;
}
/* TODO: This needs to take multiple pipelines into account */
dev_set_drvdata(dev, backend);
spin_lock_init(&backend->frontend_lock);
+ if (of_find_property(dev->of_node, "interconnects", NULL)) {
+ /*
+ * This assume we have the same DMA constraints for all our the
+ * devices in our pipeline (all the backends, but also the
+ * frontends). This sounds bad, but it has always been the case
+ * for us, and DRM doesn't do per-device allocation either, so
+ * we would need to fix DRM first...
+ */
+ ret = of_dma_configure(drm->dev, dev->of_node, true);
+ if (ret)
+ return ret;
+ } else {
+ /*
+ * If we don't have the interconnect property, most likely
+ * because of an old DT, we need to set the DMA offset by hand
+ * on our device since the RAM mapping is at 0 for the DMA bus,
+ * unlike the CPU.
+ */
+ drm->dev->dma_pfn_offset = PHYS_PFN_OFFSET;
+ }
+
backend->engine.node = dev->of_node;
backend->engine.ops = &sun4i_backend_engine_ops;
backend->engine.id = sun4i_backend_of_get_id(dev->of_node);
struct clk *tmds_clk;
struct i2c_adapter *i2c;
+ struct i2c_adapter *ddc_i2c;
/* Regmap fields for I2C adapter */
struct regmap_field *field_ddc_en;
struct edid *edid;
int ret;
- edid = drm_get_edid(connector, hdmi->i2c);
+ edid = drm_get_edid(connector, hdmi->ddc_i2c ?: hdmi->i2c);
if (!edid)
return 0;
return ret;
}
+static struct i2c_adapter *sun4i_hdmi_get_ddc(struct device *dev)
+{
+ struct device_node *phandle, *remote;
+ struct i2c_adapter *ddc;
+
+ remote = of_graph_get_remote_node(dev->of_node, 1, -1);
+ if (!remote)
+ return ERR_PTR(-EINVAL);
+
+ phandle = of_parse_phandle(remote, "ddc-i2c-bus", 0);
+ of_node_put(remote);
+ if (!phandle)
+ return ERR_PTR(-ENODEV);
+
+ ddc = of_get_i2c_adapter_by_node(phandle);
+ of_node_put(phandle);
+ if (!ddc)
+ return ERR_PTR(-EPROBE_DEFER);
+
+ return ddc;
+}
+
static const struct drm_connector_helper_funcs sun4i_hdmi_connector_helper_funcs = {
.get_modes = sun4i_hdmi_get_modes,
};
goto err_disable_mod_clk;
}
+ hdmi->ddc_i2c = sun4i_hdmi_get_ddc(dev);
+ if (IS_ERR(hdmi->ddc_i2c)) {
+ ret = PTR_ERR(hdmi->ddc_i2c);
+ if (ret == -ENODEV)
+ hdmi->ddc_i2c = NULL;
+ else
+ goto err_del_i2c_adapter;
+ }
+
drm_encoder_helper_add(&hdmi->encoder,
&sun4i_hdmi_helper_funcs);
ret = drm_encoder_init(drm,
NULL);
if (ret) {
dev_err(dev, "Couldn't initialise the HDMI encoder\n");
- goto err_del_i2c_adapter;
+ goto err_put_ddc_i2c;
}
hdmi->encoder.possible_crtcs = drm_of_find_possible_crtcs(drm,
dev->of_node);
if (!hdmi->encoder.possible_crtcs) {
ret = -EPROBE_DEFER;
- goto err_del_i2c_adapter;
+ goto err_put_ddc_i2c;
}
#ifdef CONFIG_DRM_SUN4I_HDMI_CEC
err_cleanup_connector:
cec_delete_adapter(hdmi->cec_adap);
drm_encoder_cleanup(&hdmi->encoder);
+err_put_ddc_i2c:
+ i2c_put_adapter(hdmi->ddc_i2c);
err_del_i2c_adapter:
i2c_del_adapter(hdmi->i2c);
err_disable_mod_clk:
drm_connector_cleanup(&hdmi->connector);
drm_encoder_cleanup(&hdmi->encoder);
i2c_del_adapter(hdmi->i2c);
+ i2c_put_adapter(hdmi->ddc_i2c);
clk_disable_unprepare(hdmi->mod_clk);
clk_disable_unprepare(hdmi->bus_clk);
}
struct drm_connector connector;
struct drm_encoder encoder;
- struct sun4i_tcon *tcon;
+ struct drm_panel *panel;
};
static inline struct sun4i_lvds *
{
struct sun4i_lvds *lvds =
drm_connector_to_sun4i_lvds(connector);
- struct sun4i_tcon *tcon = lvds->tcon;
- return drm_panel_get_modes(tcon->panel);
+ return drm_panel_get_modes(lvds->panel);
}
static struct drm_connector_helper_funcs sun4i_lvds_con_helper_funcs = {
sun4i_lvds_connector_destroy(struct drm_connector *connector)
{
struct sun4i_lvds *lvds = drm_connector_to_sun4i_lvds(connector);
- struct sun4i_tcon *tcon = lvds->tcon;
- drm_panel_detach(tcon->panel);
+ drm_panel_detach(lvds->panel);
drm_connector_cleanup(connector);
}
static void sun4i_lvds_encoder_enable(struct drm_encoder *encoder)
{
struct sun4i_lvds *lvds = drm_encoder_to_sun4i_lvds(encoder);
- struct sun4i_tcon *tcon = lvds->tcon;
DRM_DEBUG_DRIVER("Enabling LVDS output\n");
- if (tcon->panel) {
- drm_panel_prepare(tcon->panel);
- drm_panel_enable(tcon->panel);
+ if (lvds->panel) {
+ drm_panel_prepare(lvds->panel);
+ drm_panel_enable(lvds->panel);
}
}
static void sun4i_lvds_encoder_disable(struct drm_encoder *encoder)
{
struct sun4i_lvds *lvds = drm_encoder_to_sun4i_lvds(encoder);
- struct sun4i_tcon *tcon = lvds->tcon;
DRM_DEBUG_DRIVER("Disabling LVDS output\n");
- if (tcon->panel) {
- drm_panel_disable(tcon->panel);
- drm_panel_unprepare(tcon->panel);
+ if (lvds->panel) {
+ drm_panel_disable(lvds->panel);
+ drm_panel_unprepare(lvds->panel);
}
}
lvds = devm_kzalloc(drm->dev, sizeof(*lvds), GFP_KERNEL);
if (!lvds)
return -ENOMEM;
- lvds->tcon = tcon;
encoder = &lvds->encoder;
ret = drm_of_find_panel_or_bridge(tcon->dev->of_node, 1, 0,
- &tcon->panel, &bridge);
+ &lvds->panel, &bridge);
if (ret) {
dev_info(drm->dev, "No panel or bridge found... LVDS output disabled\n");
return 0;
/* The LVDS encoder can only work with the TCON channel 0 */
lvds->encoder.possible_crtcs = drm_crtc_mask(&tcon->crtc->crtc);
- if (tcon->panel) {
+ if (lvds->panel) {
drm_connector_helper_add(&lvds->connector,
&sun4i_lvds_con_helper_funcs);
ret = drm_connector_init(drm, &lvds->connector,
drm_connector_attach_encoder(&lvds->connector,
&lvds->encoder);
- ret = drm_panel_attach(tcon->panel, &lvds->connector);
+ ret = drm_panel_attach(lvds->panel, &lvds->connector);
if (ret) {
dev_err(drm->dev, "Couldn't attach our panel\n");
goto err_cleanup_connector;
struct drm_encoder encoder;
struct sun4i_tcon *tcon;
+ struct drm_panel *panel;
+ struct drm_bridge *bridge;
};
static inline struct sun4i_rgb *
{
struct sun4i_rgb *rgb =
drm_connector_to_sun4i_rgb(connector);
- struct sun4i_tcon *tcon = rgb->tcon;
- return drm_panel_get_modes(tcon->panel);
+ return drm_panel_get_modes(rgb->panel);
}
+/*
+ * VESA DMT defines a tolerance of 0.5% on the pixel clock, while the
+ * CVT spec reuses that tolerance in its examples, so it looks to be a
+ * good default tolerance for the EDID-based modes. Define it to 5 per
+ * mille to avoid floating point operations.
+ */
+#define SUN4I_RGB_DOTCLOCK_TOLERANCE_PER_MILLE 5
+
static enum drm_mode_status sun4i_rgb_mode_valid(struct drm_encoder *crtc,
const struct drm_display_mode *mode)
{
struct sun4i_tcon *tcon = rgb->tcon;
u32 hsync = mode->hsync_end - mode->hsync_start;
u32 vsync = mode->vsync_end - mode->vsync_start;
- unsigned long rate = mode->clock * 1000;
- long rounded_rate;
+ unsigned long long rate = mode->clock * 1000;
+ unsigned long long lowest, highest;
+ unsigned long long rounded_rate;
DRM_DEBUG_DRIVER("Validating modes...\n");
DRM_DEBUG_DRIVER("Vertical parameters OK\n");
+ /*
+ * TODO: We should use the struct display_timing if available
+ * and / or trying to stretch the timings within that
+ * tolerancy to take care of panels that we wouldn't be able
+ * to have a exact match for.
+ */
+ if (rgb->panel) {
+ DRM_DEBUG_DRIVER("RGB panel used, skipping clock rate checks");
+ goto out;
+ }
+
+ /*
+ * That shouldn't ever happen unless something is really wrong, but it
+ * doesn't harm to check.
+ */
+ if (!rgb->bridge)
+ goto out;
+
tcon->dclk_min_div = 6;
tcon->dclk_max_div = 127;
rounded_rate = clk_round_rate(tcon->dclk, rate);
- if (rounded_rate < rate)
+
+ lowest = rate * (1000 - SUN4I_RGB_DOTCLOCK_TOLERANCE_PER_MILLE);
+ do_div(lowest, 1000);
+ if (rounded_rate < lowest)
return MODE_CLOCK_LOW;
- if (rounded_rate > rate)
+ highest = rate * (1000 + SUN4I_RGB_DOTCLOCK_TOLERANCE_PER_MILLE);
+ do_div(highest, 1000);
+ if (rounded_rate > highest)
return MODE_CLOCK_HIGH;
+out:
DRM_DEBUG_DRIVER("Clock rate OK\n");
return MODE_OK;
sun4i_rgb_connector_destroy(struct drm_connector *connector)
{
struct sun4i_rgb *rgb = drm_connector_to_sun4i_rgb(connector);
- struct sun4i_tcon *tcon = rgb->tcon;
- drm_panel_detach(tcon->panel);
+ drm_panel_detach(rgb->panel);
drm_connector_cleanup(connector);
}
static void sun4i_rgb_encoder_enable(struct drm_encoder *encoder)
{
struct sun4i_rgb *rgb = drm_encoder_to_sun4i_rgb(encoder);
- struct sun4i_tcon *tcon = rgb->tcon;
DRM_DEBUG_DRIVER("Enabling RGB output\n");
- if (tcon->panel) {
- drm_panel_prepare(tcon->panel);
- drm_panel_enable(tcon->panel);
+ if (rgb->panel) {
+ drm_panel_prepare(rgb->panel);
+ drm_panel_enable(rgb->panel);
}
}
static void sun4i_rgb_encoder_disable(struct drm_encoder *encoder)
{
struct sun4i_rgb *rgb = drm_encoder_to_sun4i_rgb(encoder);
- struct sun4i_tcon *tcon = rgb->tcon;
DRM_DEBUG_DRIVER("Disabling RGB output\n");
- if (tcon->panel) {
- drm_panel_disable(tcon->panel);
- drm_panel_unprepare(tcon->panel);
+ if (rgb->panel) {
+ drm_panel_disable(rgb->panel);
+ drm_panel_unprepare(rgb->panel);
}
}
int sun4i_rgb_init(struct drm_device *drm, struct sun4i_tcon *tcon)
{
struct drm_encoder *encoder;
- struct drm_bridge *bridge;
struct sun4i_rgb *rgb;
int ret;
encoder = &rgb->encoder;
ret = drm_of_find_panel_or_bridge(tcon->dev->of_node, 1, 0,
- &tcon->panel, &bridge);
+ &rgb->panel, &rgb->bridge);
if (ret) {
dev_info(drm->dev, "No panel or bridge found... RGB output disabled\n");
return 0;
/* The RGB encoder can only work with the TCON channel 0 */
rgb->encoder.possible_crtcs = drm_crtc_mask(&tcon->crtc->crtc);
- if (tcon->panel) {
+ if (rgb->panel) {
drm_connector_helper_add(&rgb->connector,
&sun4i_rgb_con_helper_funcs);
ret = drm_connector_init(drm, &rgb->connector,
drm_connector_attach_encoder(&rgb->connector,
&rgb->encoder);
- ret = drm_panel_attach(tcon->panel, &rgb->connector);
+ ret = drm_panel_attach(rgb->panel, &rgb->connector);
if (ret) {
dev_err(drm->dev, "Couldn't attach our panel\n");
goto err_cleanup_connector;
}
}
- if (bridge) {
- ret = drm_bridge_attach(encoder, bridge, NULL);
+ if (rgb->bridge) {
+ ret = drm_bridge_attach(encoder, rgb->bridge, NULL);
if (ret) {
dev_err(drm->dev, "Couldn't attach our bridge\n");
goto err_cleanup_connector;
return NULL;
}
-void sun4i_tcon_set_mux(struct sun4i_tcon *tcon, int channel,
- const struct drm_encoder *encoder)
+static void sun4i_tcon_set_mux(struct sun4i_tcon *tcon, int channel,
+ const struct drm_encoder *encoder)
{
int ret = -ENOTSUPP;
u32 block_space, start_delay;
u32 tcon_div;
- tcon->dclk_min_div = 4;
- tcon->dclk_max_div = 127;
+ tcon->dclk_min_div = SUN6I_DSI_TCON_DIV;
+ tcon->dclk_max_div = SUN6I_DSI_TCON_DIV;
sun4i_tcon0_mode_set_common(tcon, mode);
* Following code is a way to avoid quirks all around TCON
* and DOTCLOCK drivers.
*/
- if (display_info.bus_flags & DRM_BUS_FLAG_PIXDATA_POSEDGE)
+ if (display_info.bus_flags & DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE)
clk_set_phase(tcon->dclk, 240);
- if (display_info.bus_flags & DRM_BUS_FLAG_PIXDATA_NEGEDGE)
+ if (display_info.bus_flags & DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE)
clk_set_phase(tcon->dclk, 0);
regmap_update_bits(tcon->regs, SUN4I_TCON0_IO_POL_REG,
struct reset_control *lcd_rst;
struct reset_control *lvds_rst;
- struct drm_panel *panel;
-
/* Platform adjustments */
const struct sun4i_tcon_quirks *quirks;
#include <drm/drm_panel.h>
#include <drm/drm_probe_helper.h>
+#include "sun4i_crtc.h"
#include "sun4i_drv.h"
+#include "sun4i_tcon.h"
#include "sun6i_mipi_dsi.h"
#include <video/mipi_display.h>
#define SUN6I_DSI_CTL_EN BIT(0)
#define SUN6I_DSI_BASIC_CTL_REG 0x00c
+#define SUN6I_DSI_BASIC_CTL_TRAIL_INV(n) (((n) & 0xf) << 4)
+#define SUN6I_DSI_BASIC_CTL_TRAIL_FILL BIT(3)
#define SUN6I_DSI_BASIC_CTL_HBP_DIS BIT(2)
#define SUN6I_DSI_BASIC_CTL_HSA_HSE_DIS BIT(1)
#define SUN6I_DSI_BASIC_CTL_VIDEO_BURST BIT(0)
#define SUN6I_DSI_CMD_TX_REG(n) (0x300 + (n) * 0x04)
+#define SUN6I_DSI_SYNC_POINT 40
+
enum sun6i_dsi_start_inst {
DSI_START_LPRX,
DSI_START_LPTX,
static u16 sun6i_dsi_get_video_start_delay(struct sun6i_dsi *dsi,
struct drm_display_mode *mode)
{
- return mode->vtotal - (mode->vsync_end - mode->vdisplay) + 1;
+ u16 start = clamp(mode->vtotal - mode->vdisplay - 10, 8, 100);
+ u16 delay = mode->vtotal - (mode->vsync_end - mode->vdisplay) + start;
+
+ if (delay > mode->vtotal)
+ delay = delay % mode->vtotal;
+
+ return max_t(u16, delay, 1);
+}
+
+static u16 sun6i_dsi_get_line_num(struct sun6i_dsi *dsi,
+ struct drm_display_mode *mode)
+{
+ struct mipi_dsi_device *device = dsi->device;
+ unsigned int Bpp = mipi_dsi_pixel_format_to_bpp(device->format) / 8;
+
+ return mode->htotal * Bpp / device->lanes;
+}
+
+static u16 sun6i_dsi_get_drq_edge0(struct sun6i_dsi *dsi,
+ struct drm_display_mode *mode,
+ u16 line_num, u16 edge1)
+{
+ u16 edge0 = edge1;
+
+ edge0 += (mode->hdisplay + 40) * SUN6I_DSI_TCON_DIV / 8;
+
+ if (edge0 > line_num)
+ return edge0 - line_num;
+
+ return 1;
+}
+
+static u16 sun6i_dsi_get_drq_edge1(struct sun6i_dsi *dsi,
+ struct drm_display_mode *mode,
+ u16 line_num)
+{
+ struct mipi_dsi_device *device = dsi->device;
+ unsigned int Bpp = mipi_dsi_pixel_format_to_bpp(device->format) / 8;
+ unsigned int hbp = mode->htotal - mode->hsync_end;
+ u16 edge1;
+
+ edge1 = SUN6I_DSI_SYNC_POINT;
+ edge1 += (mode->hdisplay + hbp + 20) * Bpp / device->lanes;
+
+ if (edge1 > line_num)
+ return line_num;
+
+ return edge1;
}
static void sun6i_dsi_setup_burst(struct sun6i_dsi *dsi,
struct mipi_dsi_device *device = dsi->device;
u32 val = 0;
- if ((mode->hsync_end - mode->hdisplay) > 20) {
+ if (device->mode_flags & MIPI_DSI_MODE_VIDEO_BURST) {
+ u16 line_num = sun6i_dsi_get_line_num(dsi, mode);
+ u16 edge0, edge1;
+
+ edge1 = sun6i_dsi_get_drq_edge1(dsi, mode, line_num);
+ edge0 = sun6i_dsi_get_drq_edge0(dsi, mode, line_num, edge1);
+
+ regmap_write(dsi->regs, SUN6I_DSI_BURST_DRQ_REG,
+ SUN6I_DSI_BURST_DRQ_EDGE0(edge0) |
+ SUN6I_DSI_BURST_DRQ_EDGE1(edge1));
+
+ regmap_write(dsi->regs, SUN6I_DSI_BURST_LINE_REG,
+ SUN6I_DSI_BURST_LINE_NUM(line_num) |
+ SUN6I_DSI_BURST_LINE_SYNC_POINT(SUN6I_DSI_SYNC_POINT));
+
+ val = SUN6I_DSI_TCON_DRQ_ENABLE_MODE;
+ } else if ((mode->hsync_end - mode->hdisplay) > 20) {
/* Maaaaaagic */
u16 drq = (mode->hsync_end - mode->hdisplay) - 20;
static void sun6i_dsi_setup_inst_loop(struct sun6i_dsi *dsi,
struct drm_display_mode *mode)
{
+ struct mipi_dsi_device *device = dsi->device;
u16 delay = 50 - 1;
+ if (device->mode_flags & MIPI_DSI_MODE_VIDEO_BURST) {
+ delay = (mode->htotal - mode->hdisplay) * 150;
+ delay /= (mode->clock / 1000) * 8;
+ delay -= 50;
+ }
+
+ regmap_write(dsi->regs, SUN6I_DSI_INST_LOOP_SEL_REG,
+ 2 << (4 * DSI_INST_ID_LP11) |
+ 3 << (4 * DSI_INST_ID_DLY));
+
regmap_write(dsi->regs, SUN6I_DSI_INST_LOOP_NUM_REG(0),
SUN6I_DSI_INST_LOOP_NUM_N0(50 - 1) |
SUN6I_DSI_INST_LOOP_NUM_N1(delay));
{
struct mipi_dsi_device *device = dsi->device;
unsigned int Bpp = mipi_dsi_pixel_format_to_bpp(device->format) / 8;
- u16 hbp, hfp, hsa, hblk, vblk;
+ u16 hbp = 0, hfp = 0, hsa = 0, hblk = 0, vblk = 0;
+ u32 basic_ctl = 0;
size_t bytes;
u8 *buffer;
/* Do all timing calculations up front to allocate buffer space */
- /*
- * A sync period is composed of a blanking packet (4 bytes +
- * payload + 2 bytes) and a sync event packet (4 bytes). Its
- * minimal size is therefore 10 bytes
- */
-#define HSA_PACKET_OVERHEAD 10
- hsa = max((unsigned int)HSA_PACKET_OVERHEAD,
- (mode->hsync_end - mode->hsync_start) * Bpp - HSA_PACKET_OVERHEAD);
+ if (device->mode_flags & MIPI_DSI_MODE_VIDEO_BURST) {
+ hblk = mode->hdisplay * Bpp;
+ basic_ctl = SUN6I_DSI_BASIC_CTL_VIDEO_BURST |
+ SUN6I_DSI_BASIC_CTL_HSA_HSE_DIS |
+ SUN6I_DSI_BASIC_CTL_HBP_DIS;
- /*
- * The backporch is set using a blanking packet (4 bytes +
- * payload + 2 bytes). Its minimal size is therefore 6 bytes
- */
+ if (device->lanes == 4)
+ basic_ctl |= SUN6I_DSI_BASIC_CTL_TRAIL_FILL |
+ SUN6I_DSI_BASIC_CTL_TRAIL_INV(0xc);
+ } else {
+ /*
+ * A sync period is composed of a blanking packet (4
+ * bytes + payload + 2 bytes) and a sync event packet
+ * (4 bytes). Its minimal size is therefore 10 bytes
+ */
+#define HSA_PACKET_OVERHEAD 10
+ hsa = max((unsigned int)HSA_PACKET_OVERHEAD,
+ (mode->hsync_end - mode->hsync_start) * Bpp - HSA_PACKET_OVERHEAD);
+
+ /*
+ * The backporch is set using a blanking packet (4
+ * bytes + payload + 2 bytes). Its minimal size is
+ * therefore 6 bytes
+ */
#define HBP_PACKET_OVERHEAD 6
- hbp = max((unsigned int)HBP_PACKET_OVERHEAD,
- (mode->hsync_start - mode->hdisplay) * Bpp - HBP_PACKET_OVERHEAD);
-
- /*
- * The frontporch is set using a blanking packet (4 bytes +
- * payload + 2 bytes). Its minimal size is therefore 6 bytes
- */
+ hbp = max((unsigned int)HBP_PACKET_OVERHEAD,
+ (mode->htotal - mode->hsync_end) * Bpp - HBP_PACKET_OVERHEAD);
+
+ /*
+ * The frontporch is set using a blanking packet (4
+ * bytes + payload + 2 bytes). Its minimal size is
+ * therefore 6 bytes
+ */
#define HFP_PACKET_OVERHEAD 6
- hfp = max((unsigned int)HFP_PACKET_OVERHEAD,
- (mode->htotal - mode->hsync_end) * Bpp - HFP_PACKET_OVERHEAD);
-
- /*
- * hblk seems to be the line + porches length.
- */
- hblk = mode->htotal * Bpp - hsa;
-
- /*
- * And I'm not entirely sure what vblk is about. The driver in
- * Allwinner BSP is using a rather convoluted calculation
- * there only for 4 lanes. However, using 0 (the !4 lanes
- * case) even with a 4 lanes screen seems to work...
- */
- vblk = 0;
+ hfp = max((unsigned int)HFP_PACKET_OVERHEAD,
+ (mode->hsync_start - mode->hdisplay) * Bpp - HFP_PACKET_OVERHEAD);
+
+ /*
+ * The blanking is set using a sync event (4 bytes)
+ * and a blanking packet (4 bytes + payload + 2
+ * bytes). Its minimal size is therefore 10 bytes.
+ */
+#define HBLK_PACKET_OVERHEAD 10
+ hblk = max((unsigned int)HBLK_PACKET_OVERHEAD,
+ (mode->htotal - (mode->hsync_end - mode->hsync_start)) * Bpp -
+ HBLK_PACKET_OVERHEAD);
+
+ /*
+ * And I'm not entirely sure what vblk is about. The driver in
+ * Allwinner BSP is using a rather convoluted calculation
+ * there only for 4 lanes. However, using 0 (the !4 lanes
+ * case) even with a 4 lanes screen seems to work...
+ */
+ vblk = 0;
+ }
/* How many bytes do we need to send all payloads? */
bytes = max_t(size_t, max(max(hfp, hblk), max(hsa, hbp)), vblk);
if (WARN_ON(!buffer))
return;
- regmap_write(dsi->regs, SUN6I_DSI_BASIC_CTL_REG, 0);
+ regmap_write(dsi->regs, SUN6I_DSI_BASIC_CTL_REG, basic_ctl);
regmap_write(dsi->regs, SUN6I_DSI_SYNC_HSS_REG,
sun6i_dsi_build_sync_pkt(MIPI_DSI_H_SYNC_START,
regmap_write(dsi->regs, SUN6I_DSI_BASIC_SIZE0_REG,
SUN6I_DSI_BASIC_SIZE0_VSA(mode->vsync_end -
mode->vsync_start) |
- SUN6I_DSI_BASIC_SIZE0_VBP(mode->vsync_start -
- mode->vdisplay));
+ SUN6I_DSI_BASIC_SIZE0_VBP(mode->vtotal -
+ mode->vsync_end));
regmap_write(dsi->regs, SUN6I_DSI_BASIC_SIZE1_REG,
SUN6I_DSI_BASIC_SIZE1_VACT(mode->vdisplay) |
#include <drm/drm_encoder.h>
#include <drm/drm_mipi_dsi.h>
+#define SUN6I_DSI_TCON_DIV 4
+
struct sun6i_dsi {
struct drm_connector connector;
struct drm_encoder encoder;
static int sun8i_mixer_of_get_id(struct device_node *node)
{
- struct device_node *port, *ep;
- int ret = -EINVAL;
+ struct device_node *ep, *remote;
+ struct of_endpoint of_ep;
- /* output is port 1 */
- port = of_graph_get_port_by_id(node, 1);
- if (!port)
+ /* Output port is 1, and we want the first endpoint. */
+ ep = of_graph_get_endpoint_by_regs(node, 1, -1);
+ if (!ep)
return -EINVAL;
- /* try to find downstream endpoint */
- for_each_available_child_of_node(port, ep) {
- struct device_node *remote;
- u32 reg;
-
- remote = of_graph_get_remote_endpoint(ep);
- if (!remote)
- continue;
-
- ret = of_property_read_u32(remote, "reg", ®);
- if (!ret) {
- of_node_put(remote);
- of_node_put(ep);
- of_node_put(port);
-
- return reg;
- }
-
- of_node_put(remote);
- }
-
- of_node_put(port);
+ remote = of_graph_get_remote_endpoint(ep);
+ of_node_put(ep);
+ if (!remote)
+ return -EINVAL;
- return ret;
+ of_graph_parse_endpoint(remote, &of_ep);
+ of_node_put(remote);
+ return of_ep.id;
}
static int sun8i_mixer_bind(struct device *dev, struct device *master,
static const struct sun8i_mixer_cfg sun8i_a83t_mixer0_cfg = {
.ccsc = 0,
.scaler_mask = 0xf,
+ .scanline_yuv = 2048,
.ui_num = 3,
.vi_num = 1,
};
static const struct sun8i_mixer_cfg sun8i_a83t_mixer1_cfg = {
.ccsc = 1,
.scaler_mask = 0x3,
+ .scanline_yuv = 2048,
.ui_num = 1,
.vi_num = 1,
};
.ccsc = 0,
.mod_rate = 432000000,
.scaler_mask = 0xf,
+ .scanline_yuv = 2048,
.ui_num = 3,
.vi_num = 1,
};
.ccsc = 0,
.mod_rate = 297000000,
.scaler_mask = 0xf,
+ .scanline_yuv = 2048,
.ui_num = 3,
.vi_num = 1,
};
.ccsc = 1,
.mod_rate = 297000000,
.scaler_mask = 0x3,
+ .scanline_yuv = 2048,
.ui_num = 1,
.vi_num = 1,
};
.vi_num = 2,
.ui_num = 1,
.scaler_mask = 0x3,
+ .scanline_yuv = 2048,
.ccsc = 0,
.mod_rate = 150000000,
};
.ccsc = 0,
.mod_rate = 297000000,
.scaler_mask = 0xf,
+ .scanline_yuv = 4096,
.ui_num = 3,
.vi_num = 1,
};
.ccsc = 1,
.mod_rate = 297000000,
.scaler_mask = 0x3,
+ .scanline_yuv = 2048,
.ui_num = 1,
.vi_num = 1,
};
.is_de3 = true,
.mod_rate = 600000000,
.scaler_mask = 0xf,
+ .scanline_yuv = 4096,
.ui_num = 3,
.vi_num = 1,
};
* @mod_rate: module clock rate that needs to be set in order to have
* a functional block.
* @is_de3: true, if this is next gen display engine 3.0, false otherwise.
+ * @scaline_yuv: size of a scanline for VI scaler for YUV formats.
*/
struct sun8i_mixer_cfg {
int vi_num;
int ccsc;
unsigned long mod_rate;
unsigned int is_de3 : 1;
+ unsigned int scanline_yuv;
};
struct sun8i_mixer {
return 0;
}
-const struct sun8i_tcon_top_quirks sun8i_r40_tcon_top_quirks = {
+static const struct sun8i_tcon_top_quirks sun8i_r40_tcon_top_quirks = {
.has_tcon_tv1 = true,
.has_dsi = true,
};
-const struct sun8i_tcon_top_quirks sun50i_h6_tcon_top_quirks = {
+static const struct sun8i_tcon_top_quirks sun50i_h6_tcon_top_quirks = {
/* Nothing special */
};
u32 bld_base, ch_base;
u32 outsize, insize;
u32 hphase, vphase;
+ u32 hn = 0, hm = 0;
+ u32 vn = 0, vm = 0;
bool subsampled;
DRM_DEBUG_DRIVER("Updating VI channel %d overlay %d\n",
subsampled = format->hsub > 1 || format->vsub > 1;
if (insize != outsize || subsampled || hphase || vphase) {
- u32 hscale, vscale;
+ unsigned int scanline, required;
+ struct drm_display_mode *mode;
+ u32 hscale, vscale, fps;
+ u64 ability;
DRM_DEBUG_DRIVER("HW scaling is enabled\n");
- hscale = state->src_w / state->crtc_w;
- vscale = state->src_h / state->crtc_h;
+ mode = &plane->state->crtc->state->mode;
+ fps = (mode->clock * 1000) / (mode->vtotal * mode->htotal);
+ ability = clk_get_rate(mixer->mod_clk);
+ /* BSP algorithm assumes 80% efficiency of VI scaler unit */
+ ability *= 80;
+ do_div(ability, mode->vdisplay * fps * max(src_w, dst_w));
+
+ required = src_h * 100 / dst_h;
+
+ if (ability < required) {
+ DRM_DEBUG_DRIVER("Using vertical coarse scaling\n");
+ vm = src_h;
+ vn = (u32)ability * dst_h / 100;
+ src_h = vn;
+ }
+
+ /* it seems that every RGB scaler has buffer for 2048 pixels */
+ scanline = subsampled ? mixer->cfg->scanline_yuv : 2048;
+
+ if (src_w > scanline) {
+ DRM_DEBUG_DRIVER("Using horizontal coarse scaling\n");
+ hm = src_w;
+ hn = scanline;
+ src_w = hn;
+ }
+
+ hscale = (src_w << 16) / dst_w;
+ vscale = (src_h << 16) / dst_h;
sun8i_vi_scaler_setup(mixer, channel, src_w, src_h, dst_w,
dst_h, hscale, vscale, hphase, vphase,
sun8i_vi_scaler_enable(mixer, channel, false);
}
+ regmap_write(mixer->engine.regs,
+ SUN8I_MIXER_CHAN_VI_HDS_Y(ch_base),
+ SUN8I_MIXER_CHAN_VI_DS_N(hn) |
+ SUN8I_MIXER_CHAN_VI_DS_M(hm));
+ regmap_write(mixer->engine.regs,
+ SUN8I_MIXER_CHAN_VI_HDS_UV(ch_base),
+ SUN8I_MIXER_CHAN_VI_DS_N(hn) |
+ SUN8I_MIXER_CHAN_VI_DS_M(hm));
+ regmap_write(mixer->engine.regs,
+ SUN8I_MIXER_CHAN_VI_VDS_Y(ch_base),
+ SUN8I_MIXER_CHAN_VI_DS_N(vn) |
+ SUN8I_MIXER_CHAN_VI_DS_M(vm));
+ regmap_write(mixer->engine.regs,
+ SUN8I_MIXER_CHAN_VI_VDS_UV(ch_base),
+ SUN8I_MIXER_CHAN_VI_DS_N(vn) |
+ SUN8I_MIXER_CHAN_VI_DS_M(vm));
+
/* Set base coordinates */
DRM_DEBUG_DRIVER("Layer destination coordinates X: %d Y: %d\n",
state->dst.x1, state->dst.y1);
((base) + 0x30 * (layer) + 0x18 + 4 * (plane))
#define SUN8I_MIXER_CHAN_VI_OVL_SIZE(base) \
((base) + 0xe8)
+#define SUN8I_MIXER_CHAN_VI_HDS_Y(base) \
+ ((base) + 0xf0)
+#define SUN8I_MIXER_CHAN_VI_HDS_UV(base) \
+ ((base) + 0xf4)
+#define SUN8I_MIXER_CHAN_VI_VDS_Y(base) \
+ ((base) + 0xf8)
+#define SUN8I_MIXER_CHAN_VI_VDS_UV(base) \
+ ((base) + 0xfc)
#define SUN8I_MIXER_CHAN_VI_LAYER_ATTR_EN BIT(0)
/* RGB mode should be set for RGB formats and cleared for YCbCr */
#define SUN50I_MIXER_CHAN_VI_LAYER_ATTR_ALPHA_MASK GENMASK(31, 24)
#define SUN50I_MIXER_CHAN_VI_LAYER_ATTR_ALPHA(x) ((x) << 24)
+#define SUN8I_MIXER_CHAN_VI_DS_N(x) ((x) << 16)
+#define SUN8I_MIXER_CHAN_VI_DS_M(x) ((x) << 0)
+
struct sun8i_mixer;
struct sun8i_vi_layer {
helper->fb = fb;
helper->fbdev = info;
- info->par = helper;
info->fbops = &tegra_fb_ops;
- drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(info, helper, fb->width, fb->height);
+ drm_fb_helper_fill_info(info, helper, sizes);
offset = info->var.xoffset * bytes_per_pixel +
info->var.yoffset * fb->pitches[0];
{
if (bo->pages) {
dma_unmap_sg(drm->dev, bo->sgt->sgl, bo->sgt->nents,
- DMA_BIDIRECTIONAL);
+ DMA_FROM_DEVICE);
drm_gem_put_pages(&bo->gem, bo->pages, true, true);
sg_free_table(bo->sgt);
kfree(bo->sgt);
}
err = dma_map_sg(drm->dev, bo->sgt->sgl, bo->sgt->nents,
- DMA_BIDIRECTIONAL);
+ DMA_FROM_DEVICE);
if (err == 0) {
err = -EFAULT;
goto free_sgt;
* kernel is possible.
*/
if (sor->rst) {
+ err = reset_control_acquire(sor->rst);
+ if (err < 0) {
+ dev_err(sor->dev, "failed to acquire SOR reset: %d\n",
+ err);
+ return err;
+ }
+
err = reset_control_assert(sor->rst);
if (err < 0) {
dev_err(sor->dev, "failed to assert SOR reset: %d\n",
err);
return err;
}
+
+ reset_control_release(sor->rst);
}
err = clk_prepare_enable(sor->clk_safe);
goto remove;
}
- sor->rst = devm_reset_control_get(&pdev->dev, "sor");
+ sor->rst = devm_reset_control_get_exclusive_released(&pdev->dev, "sor");
if (IS_ERR(sor->rst)) {
err = PTR_ERR(sor->rst);
dev_err(dev, "failed to assert reset: %d\n", err);
return err;
}
+
+ reset_control_release(sor->rst);
}
usleep_range(1000, 2000);
usleep_range(1000, 2000);
if (sor->rst) {
+ err = reset_control_acquire(sor->rst);
+ if (err < 0) {
+ dev_err(dev, "failed to acquire reset: %d\n", err);
+ clk_disable_unprepare(sor->clk);
+ return err;
+ }
+
err = reset_control_deassert(sor->rst);
if (err < 0) {
dev_err(dev, "failed to deassert reset: %d\n", err);
+ reset_control_release(sor->rst);
clk_disable_unprepare(sor->clk);
return err;
}
-tinydrm-y := tinydrm-core.o tinydrm-pipe.o tinydrm-helpers.o
+tinydrm-y := tinydrm-pipe.o tinydrm-helpers.o
obj-$(CONFIG_DRM_TINYDRM) += tinydrm.o
+++ /dev/null
-/*
- * Copyright (C) 2016 Noralf Trønnes
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <drm/drm_atomic.h>
-#include <drm/drm_atomic_helper.h>
-#include <drm/drm_drv.h>
-#include <drm/drm_fb_helper.h>
-#include <drm/drm_gem_framebuffer_helper.h>
-#include <drm/drm_probe_helper.h>
-#include <drm/drm_print.h>
-#include <drm/tinydrm/tinydrm.h>
-#include <linux/device.h>
-#include <linux/dma-buf.h>
-#include <linux/module.h>
-
-/**
- * DOC: overview
- *
- * This library provides driver helpers for very simple display hardware.
- *
- * It is based on &drm_simple_display_pipe coupled with a &drm_connector which
- * has only one fixed &drm_display_mode. The framebuffers are backed by the
- * cma helper and have support for framebuffer flushing (dirty).
- * fbdev support is also included.
- *
- */
-
-/**
- * DOC: core
- *
- * The driver allocates &tinydrm_device, initializes it using
- * devm_tinydrm_init(), sets up the pipeline using tinydrm_display_pipe_init()
- * and registers the DRM device using devm_tinydrm_register().
- */
-
-static const struct drm_mode_config_funcs tinydrm_mode_config_funcs = {
- .fb_create = drm_gem_fb_create_with_dirty,
- .atomic_check = drm_atomic_helper_check,
- .atomic_commit = drm_atomic_helper_commit,
-};
-
-static int tinydrm_init(struct device *parent, struct tinydrm_device *tdev,
- struct drm_driver *driver)
-{
- struct drm_device *drm;
-
- /*
- * We don't embed drm_device, because that prevent us from using
- * devm_kzalloc() to allocate tinydrm_device in the driver since
- * drm_dev_put() frees the structure. The devm_ functions provide
- * for easy error handling.
- */
- drm = drm_dev_alloc(driver, parent);
- if (IS_ERR(drm))
- return PTR_ERR(drm);
-
- tdev->drm = drm;
- drm->dev_private = tdev;
- drm_mode_config_init(drm);
- drm->mode_config.funcs = &tinydrm_mode_config_funcs;
- drm->mode_config.allow_fb_modifiers = true;
-
- return 0;
-}
-
-static void tinydrm_fini(struct tinydrm_device *tdev)
-{
- drm_mode_config_cleanup(tdev->drm);
- tdev->drm->dev_private = NULL;
- drm_dev_put(tdev->drm);
-}
-
-static void devm_tinydrm_release(void *data)
-{
- tinydrm_fini(data);
-}
-
-/**
- * devm_tinydrm_init - Initialize tinydrm device
- * @parent: Parent device object
- * @tdev: tinydrm device
- * @driver: DRM driver
- *
- * This function initializes @tdev, the underlying DRM device and it's
- * mode_config. Resources will be automatically freed on driver detach (devres)
- * using drm_mode_config_cleanup() and drm_dev_put().
- *
- * Returns:
- * Zero on success, negative error code on failure.
- */
-int devm_tinydrm_init(struct device *parent, struct tinydrm_device *tdev,
- struct drm_driver *driver)
-{
- int ret;
-
- ret = tinydrm_init(parent, tdev, driver);
- if (ret)
- return ret;
-
- ret = devm_add_action(parent, devm_tinydrm_release, tdev);
- if (ret)
- tinydrm_fini(tdev);
-
- return ret;
-}
-EXPORT_SYMBOL(devm_tinydrm_init);
-
-static int tinydrm_register(struct tinydrm_device *tdev)
-{
- struct drm_device *drm = tdev->drm;
- int ret;
-
- ret = drm_dev_register(tdev->drm, 0);
- if (ret)
- return ret;
-
- ret = drm_fbdev_generic_setup(drm, 0);
- if (ret)
- DRM_ERROR("Failed to initialize fbdev: %d\n", ret);
-
- return 0;
-}
-
-static void tinydrm_unregister(struct tinydrm_device *tdev)
-{
- drm_atomic_helper_shutdown(tdev->drm);
- drm_dev_unregister(tdev->drm);
-}
-
-static void devm_tinydrm_register_release(void *data)
-{
- tinydrm_unregister(data);
-}
-
-/**
- * devm_tinydrm_register - Register tinydrm device
- * @tdev: tinydrm device
- *
- * This function registers the underlying DRM device and fbdev.
- * These resources will be automatically unregistered on driver detach (devres)
- * and the display pipeline will be disabled.
- *
- * Returns:
- * Zero on success, negative error code on failure.
- */
-int devm_tinydrm_register(struct tinydrm_device *tdev)
-{
- struct device *dev = tdev->drm->dev;
- int ret;
-
- ret = tinydrm_register(tdev);
- if (ret)
- return ret;
-
- ret = devm_add_action(dev, devm_tinydrm_register_release, tdev);
- if (ret)
- tinydrm_unregister(tdev);
-
- return ret;
-}
-EXPORT_SYMBOL(devm_tinydrm_register);
-
-/**
- * tinydrm_shutdown - Shutdown tinydrm
- * @tdev: tinydrm device
- *
- * This function makes sure that the display pipeline is disabled.
- * Used by drivers in their shutdown callback to turn off the display
- * on machine shutdown and reboot.
- */
-void tinydrm_shutdown(struct tinydrm_device *tdev)
-{
- drm_atomic_helper_shutdown(tdev->drm);
-}
-EXPORT_SYMBOL(tinydrm_shutdown);
-
-MODULE_LICENSE("GPL");
module_param(spi_max, uint, 0400);
MODULE_PARM_DESC(spi_max, "Set a lower SPI max transfer size");
-/**
- * tinydrm_memcpy - Copy clip buffer
- * @dst: Destination buffer
- * @vaddr: Source buffer
- * @fb: DRM framebuffer
- * @clip: Clip rectangle area to copy
- */
-void tinydrm_memcpy(void *dst, void *vaddr, struct drm_framebuffer *fb,
- struct drm_rect *clip)
-{
- unsigned int cpp = drm_format_plane_cpp(fb->format->format, 0);
- unsigned int pitch = fb->pitches[0];
- void *src = vaddr + (clip->y1 * pitch) + (clip->x1 * cpp);
- size_t len = (clip->x2 - clip->x1) * cpp;
- unsigned int y;
-
- for (y = clip->y1; y < clip->y2; y++) {
- memcpy(dst, src, len);
- src += pitch;
- dst += len;
- }
-}
-EXPORT_SYMBOL(tinydrm_memcpy);
-
-/**
- * tinydrm_swab16 - Swap bytes into clip buffer
- * @dst: RGB565 destination buffer
- * @vaddr: RGB565 source buffer
- * @fb: DRM framebuffer
- * @clip: Clip rectangle area to copy
- */
-void tinydrm_swab16(u16 *dst, void *vaddr, struct drm_framebuffer *fb,
- struct drm_rect *clip)
-{
- size_t len = (clip->x2 - clip->x1) * sizeof(u16);
- unsigned int x, y;
- u16 *src, *buf;
-
- /*
- * The cma memory is write-combined so reads are uncached.
- * Speed up by fetching one line at a time.
- */
- buf = kmalloc(len, GFP_KERNEL);
- if (!buf)
- return;
-
- for (y = clip->y1; y < clip->y2; y++) {
- src = vaddr + (y * fb->pitches[0]);
- src += clip->x1;
- memcpy(buf, src, len);
- src = buf;
- for (x = clip->x1; x < clip->x2; x++)
- *dst++ = swab16(*src++);
- }
-
- kfree(buf);
-}
-EXPORT_SYMBOL(tinydrm_swab16);
-
-/**
- * tinydrm_xrgb8888_to_rgb565 - Convert XRGB8888 to RGB565 clip buffer
- * @dst: RGB565 destination buffer
- * @vaddr: XRGB8888 source buffer
- * @fb: DRM framebuffer
- * @clip: Clip rectangle area to copy
- * @swap: Swap bytes
- *
- * Drivers can use this function for RGB565 devices that don't natively
- * support XRGB8888.
- */
-void tinydrm_xrgb8888_to_rgb565(u16 *dst, void *vaddr,
- struct drm_framebuffer *fb,
- struct drm_rect *clip, bool swap)
-{
- size_t len = (clip->x2 - clip->x1) * sizeof(u32);
- unsigned int x, y;
- u32 *src, *buf;
- u16 val16;
-
- buf = kmalloc(len, GFP_KERNEL);
- if (!buf)
- return;
-
- for (y = clip->y1; y < clip->y2; y++) {
- src = vaddr + (y * fb->pitches[0]);
- src += clip->x1;
- memcpy(buf, src, len);
- src = buf;
- for (x = clip->x1; x < clip->x2; x++) {
- val16 = ((*src & 0x00F80000) >> 8) |
- ((*src & 0x0000FC00) >> 5) |
- ((*src & 0x000000F8) >> 3);
- src++;
- if (swap)
- *dst++ = swab16(val16);
- else
- *dst++ = val16;
- }
- }
-
- kfree(buf);
-}
-EXPORT_SYMBOL(tinydrm_xrgb8888_to_rgb565);
-
-/**
- * tinydrm_xrgb8888_to_gray8 - Convert XRGB8888 to grayscale
- * @dst: 8-bit grayscale destination buffer
- * @vaddr: XRGB8888 source buffer
- * @fb: DRM framebuffer
- * @clip: Clip rectangle area to copy
- *
- * Drm doesn't have native monochrome or grayscale support.
- * Such drivers can announce the commonly supported XR24 format to userspace
- * and use this function to convert to the native format.
- *
- * Monochrome drivers will use the most significant bit,
- * where 1 means foreground color and 0 background color.
- *
- * ITU BT.601 is used for the RGB -> luma (brightness) conversion.
- */
-void tinydrm_xrgb8888_to_gray8(u8 *dst, void *vaddr, struct drm_framebuffer *fb,
- struct drm_rect *clip)
-{
- unsigned int len = (clip->x2 - clip->x1) * sizeof(u32);
- unsigned int x, y;
- void *buf;
- u32 *src;
-
- if (WARN_ON(fb->format->format != DRM_FORMAT_XRGB8888))
- return;
- /*
- * The cma memory is write-combined so reads are uncached.
- * Speed up by fetching one line at a time.
- */
- buf = kmalloc(len, GFP_KERNEL);
- if (!buf)
- return;
-
- for (y = clip->y1; y < clip->y2; y++) {
- src = vaddr + (y * fb->pitches[0]);
- src += clip->x1;
- memcpy(buf, src, len);
- src = buf;
- for (x = clip->x1; x < clip->x2; x++) {
- u8 r = (*src & 0x00ff0000) >> 16;
- u8 g = (*src & 0x0000ff00) >> 8;
- u8 b = *src & 0x000000ff;
-
- /* ITU BT.601: Y = 0.299 R + 0.587 G + 0.114 B */
- *dst++ = (3 * r + 6 * g + b) / 10;
- src++;
- }
- }
-
- kfree(buf);
-}
-EXPORT_SYMBOL(tinydrm_xrgb8888_to_gray8);
-
#if IS_ENABLED(CONFIG_SPI)
/**
EXPORT_SYMBOL(tinydrm_spi_transfer);
#endif /* CONFIG_SPI */
+
+MODULE_LICENSE("GPL");
#include <drm/drm_modes.h>
#include <drm/drm_probe_helper.h>
#include <drm/drm_print.h>
-#include <drm/tinydrm/tinydrm.h>
+#include <drm/drm_simple_kms_helper.h>
struct tinydrm_connector {
struct drm_connector base;
/**
* tinydrm_display_pipe_init - Initialize display pipe
- * @tdev: tinydrm device
+ * @drm: DRM device
+ * @pipe: Display pipe
* @funcs: Display pipe functions
* @connector_type: Connector type
* @formats: Array of supported formats (DRM_FORMAT\_\*)
* Returns:
* Zero on success, negative error code on failure.
*/
-int
-tinydrm_display_pipe_init(struct tinydrm_device *tdev,
- const struct drm_simple_display_pipe_funcs *funcs,
- int connector_type,
- const uint32_t *formats,
- unsigned int format_count,
- const struct drm_display_mode *mode,
- unsigned int rotation)
+int tinydrm_display_pipe_init(struct drm_device *drm,
+ struct drm_simple_display_pipe *pipe,
+ const struct drm_simple_display_pipe_funcs *funcs,
+ int connector_type,
+ const uint32_t *formats,
+ unsigned int format_count,
+ const struct drm_display_mode *mode,
+ unsigned int rotation)
{
- struct drm_device *drm = tdev->drm;
struct drm_display_mode mode_copy;
struct drm_connector *connector;
int ret;
if (IS_ERR(connector))
return PTR_ERR(connector);
- return drm_simple_display_pipe_init(drm, &tdev->pipe, funcs, formats,
+ return drm_simple_display_pipe_init(drm, pipe, funcs, formats,
format_count, modifiers, connector);
}
EXPORT_SYMBOL(tinydrm_display_pipe_init);
#include <linux/property.h>
#include <linux/spi/spi.h>
+#include <drm/drm_atomic_helper.h>
#include <drm/drm_drv.h>
+#include <drm/drm_fb_helper.h>
#include <drm/drm_gem_cma_helper.h>
#include <drm/drm_gem_framebuffer_helper.h>
#include <drm/drm_modeset_helper.h>
struct drm_crtc_state *crtc_state,
struct drm_plane_state *plane_state)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(pipe->crtc.dev);
u8 addr_mode;
- int ret;
+ int ret, idx;
+
+ if (!drm_dev_enter(pipe->crtc.dev, &idx))
+ return;
DRM_DEBUG_KMS("\n");
ret = mipi_dbi_poweron_conditional_reset(mipi);
if (ret < 0)
- return;
+ goto out_exit;
if (ret == 1)
goto out_enable;
}
mipi_dbi_command(mipi, MIPI_DCS_SET_ADDRESS_MODE, addr_mode);
mipi_dbi_enable_flush(mipi, crtc_state, plane_state);
+out_exit:
+ drm_dev_exit(idx);
}
static const struct drm_simple_display_pipe_funcs hx8357d_pipe_funcs = {
};
static const struct drm_display_mode yx350hv15_mode = {
- TINYDRM_MODE(320, 480, 60, 75),
+ DRM_SIMPLE_MODE(320, 480, 60, 75),
};
DEFINE_DRM_GEM_CMA_FOPS(hx8357d_fops);
static struct drm_driver hx8357d_driver = {
.driver_features = DRIVER_GEM | DRIVER_MODESET | DRIVER_PRIME | DRIVER_ATOMIC,
.fops = &hx8357d_fops,
+ .release = mipi_dbi_release,
DRM_GEM_CMA_VMAP_DRIVER_OPS,
.debugfs_init = mipi_dbi_debugfs_init,
.name = "hx8357d",
static int hx8357d_probe(struct spi_device *spi)
{
struct device *dev = &spi->dev;
+ struct drm_device *drm;
struct mipi_dbi *mipi;
struct gpio_desc *dc;
u32 rotation = 0;
int ret;
- mipi = devm_kzalloc(dev, sizeof(*mipi), GFP_KERNEL);
+ mipi = kzalloc(sizeof(*mipi), GFP_KERNEL);
if (!mipi)
return -ENOMEM;
+ drm = &mipi->drm;
+ ret = devm_drm_dev_init(dev, drm, &hx8357d_driver);
+ if (ret) {
+ kfree(mipi);
+ return ret;
+ }
+
+ drm_mode_config_init(drm);
+
dc = devm_gpiod_get(dev, "dc", GPIOD_OUT_LOW);
if (IS_ERR(dc)) {
DRM_DEV_ERROR(dev, "Failed to get gpio 'dc'\n");
if (ret)
return ret;
- ret = mipi_dbi_init(&spi->dev, mipi, &hx8357d_pipe_funcs,
- &hx8357d_driver, &yx350hv15_mode, rotation);
+ ret = mipi_dbi_init(mipi, &hx8357d_pipe_funcs, &yx350hv15_mode, rotation);
if (ret)
return ret;
- spi_set_drvdata(spi, mipi);
+ drm_mode_config_reset(drm);
- return devm_tinydrm_register(&mipi->tinydrm);
+ ret = drm_dev_register(drm, 0);
+ if (ret)
+ return ret;
+
+ spi_set_drvdata(spi, drm);
+
+ drm_fbdev_generic_setup(drm, 0);
+
+ return 0;
}
-static void hx8357d_shutdown(struct spi_device *spi)
+static int hx8357d_remove(struct spi_device *spi)
{
- struct mipi_dbi *mipi = spi_get_drvdata(spi);
+ struct drm_device *drm = spi_get_drvdata(spi);
+
+ drm_dev_unplug(drm);
+ drm_atomic_helper_shutdown(drm);
- tinydrm_shutdown(&mipi->tinydrm);
+ return 0;
+}
+
+static void hx8357d_shutdown(struct spi_device *spi)
+{
+ drm_atomic_helper_shutdown(spi_get_drvdata(spi));
}
static struct spi_driver hx8357d_spi_driver = {
},
.id_table = hx8357d_id,
.probe = hx8357d_probe,
+ .remove = hx8357d_remove,
.shutdown = hx8357d_shutdown,
};
module_spi_driver(hx8357d_spi_driver);
#include <linux/spi/spi.h>
#include <video/mipi_display.h>
+#include <drm/drm_atomic_helper.h>
#include <drm/drm_damage_helper.h>
#include <drm/drm_drv.h>
#include <drm/drm_fb_cma_helper.h>
+#include <drm/drm_fb_helper.h>
#include <drm/drm_fourcc.h>
#include <drm/drm_gem_cma_helper.h>
#include <drm/drm_gem_framebuffer_helper.h>
static void ili9225_fb_dirty(struct drm_framebuffer *fb, struct drm_rect *rect)
{
struct drm_gem_cma_object *cma_obj = drm_fb_cma_get_gem_obj(fb, 0);
- struct tinydrm_device *tdev = fb->dev->dev_private;
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(fb->dev);
unsigned int height = rect->y2 - rect->y1;
unsigned int width = rect->x2 - rect->x1;
bool swap = mipi->swap_bytes;
u16 x_start, y_start;
u16 x1, x2, y1, y2;
- int ret = 0;
+ int idx, ret = 0;
bool full;
void *tr;
if (!mipi->enabled)
return;
+ if (!drm_dev_enter(fb->dev, &idx))
+ return;
+
full = width == fb->width && height == fb->height;
DRM_DEBUG_KMS("Flushing [FB:%d] " DRM_RECT_FMT "\n", fb->base.id, DRM_RECT_ARG(rect));
err_msg:
if (ret)
dev_err_once(fb->dev->dev, "Failed to update display %d\n", ret);
+
+ drm_dev_exit(idx);
}
static void ili9225_pipe_update(struct drm_simple_display_pipe *pipe,
struct drm_crtc_state *crtc_state,
struct drm_plane_state *plane_state)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(pipe->crtc.dev);
struct drm_framebuffer *fb = plane_state->fb;
- struct device *dev = tdev->drm->dev;
+ struct device *dev = pipe->crtc.dev->dev;
struct drm_rect rect = {
.x1 = 0,
.x2 = fb->width,
.y1 = 0,
.y2 = fb->height,
};
- int ret;
+ int ret, idx;
u8 am_id;
+ if (!drm_dev_enter(pipe->crtc.dev, &idx))
+ return;
+
DRM_DEBUG_KMS("\n");
mipi_dbi_hw_reset(mipi);
ret = ili9225_command(mipi, ILI9225_POWER_CONTROL_1, 0x0000);
if (ret) {
DRM_DEV_ERROR(dev, "Error sending command %d\n", ret);
- return;
+ goto out_exit;
}
ili9225_command(mipi, ILI9225_POWER_CONTROL_2, 0x0000);
ili9225_command(mipi, ILI9225_POWER_CONTROL_3, 0x0000);
mipi->enabled = true;
ili9225_fb_dirty(fb, &rect);
+out_exit:
+ drm_dev_exit(idx);
}
static void ili9225_pipe_disable(struct drm_simple_display_pipe *pipe)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(pipe->crtc.dev);
DRM_DEBUG_KMS("\n");
+ /*
+ * This callback is not protected by drm_dev_enter/exit since we want to
+ * turn off the display on regular driver unload. It's highly unlikely
+ * that the underlying SPI controller is gone should this be called after
+ * unplug.
+ */
+
if (!mipi->enabled)
return;
mipi->enabled = false;
}
-static int ili9225_dbi_command(struct mipi_dbi *mipi, u8 cmd, u8 *par,
+static int ili9225_dbi_command(struct mipi_dbi *mipi, u8 *cmd, u8 *par,
size_t num)
{
struct spi_device *spi = mipi->spi;
gpiod_set_value_cansleep(mipi->dc, 0);
speed_hz = mipi_dbi_spi_cmd_max_speed(spi, 1);
- ret = tinydrm_spi_transfer(spi, speed_hz, NULL, 8, &cmd, 1);
+ ret = tinydrm_spi_transfer(spi, speed_hz, NULL, 8, cmd, 1);
if (ret || !num)
return ret;
- if (cmd == ILI9225_WRITE_DATA_TO_GRAM && !mipi->swap_bytes)
+ if (*cmd == ILI9225_WRITE_DATA_TO_GRAM && !mipi->swap_bytes)
bpw = 16;
gpiod_set_value_cansleep(mipi->dc, 1);
};
static const struct drm_display_mode ili9225_mode = {
- TINYDRM_MODE(176, 220, 35, 44),
+ DRM_SIMPLE_MODE(176, 220, 35, 44),
};
DEFINE_DRM_GEM_CMA_FOPS(ili9225_fops);
.driver_features = DRIVER_GEM | DRIVER_MODESET | DRIVER_PRIME |
DRIVER_ATOMIC,
.fops = &ili9225_fops,
+ .release = mipi_dbi_release,
DRM_GEM_CMA_VMAP_DRIVER_OPS,
.name = "ili9225",
.desc = "Ilitek ILI9225",
static int ili9225_probe(struct spi_device *spi)
{
struct device *dev = &spi->dev;
+ struct drm_device *drm;
struct mipi_dbi *mipi;
struct gpio_desc *rs;
u32 rotation = 0;
int ret;
- mipi = devm_kzalloc(dev, sizeof(*mipi), GFP_KERNEL);
+ mipi = kzalloc(sizeof(*mipi), GFP_KERNEL);
if (!mipi)
return -ENOMEM;
+ drm = &mipi->drm;
+ ret = devm_drm_dev_init(dev, drm, &ili9225_driver);
+ if (ret) {
+ kfree(mipi);
+ return ret;
+ }
+
+ drm_mode_config_init(drm);
+
mipi->reset = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
if (IS_ERR(mipi->reset)) {
DRM_DEV_ERROR(dev, "Failed to get gpio 'reset'\n");
/* override the command function set in mipi_dbi_spi_init() */
mipi->command = ili9225_dbi_command;
- ret = mipi_dbi_init(&spi->dev, mipi, &ili9225_pipe_funcs,
- &ili9225_driver, &ili9225_mode, rotation);
+ ret = mipi_dbi_init(mipi, &ili9225_pipe_funcs, &ili9225_mode, rotation);
if (ret)
return ret;
- spi_set_drvdata(spi, mipi);
+ drm_mode_config_reset(drm);
+
+ ret = drm_dev_register(drm, 0);
+ if (ret)
+ return ret;
+
+ spi_set_drvdata(spi, drm);
+
+ drm_fbdev_generic_setup(drm, 0);
- return devm_tinydrm_register(&mipi->tinydrm);
+ return 0;
}
-static void ili9225_shutdown(struct spi_device *spi)
+static int ili9225_remove(struct spi_device *spi)
{
- struct mipi_dbi *mipi = spi_get_drvdata(spi);
+ struct drm_device *drm = spi_get_drvdata(spi);
+
+ drm_dev_unplug(drm);
+ drm_atomic_helper_shutdown(drm);
+
+ return 0;
+}
- tinydrm_shutdown(&mipi->tinydrm);
+static void ili9225_shutdown(struct spi_device *spi)
+{
+ drm_atomic_helper_shutdown(spi_get_drvdata(spi));
}
static struct spi_driver ili9225_spi_driver = {
},
.id_table = ili9225_id,
.probe = ili9225_probe,
+ .remove = ili9225_remove,
.shutdown = ili9225_shutdown,
};
module_spi_driver(ili9225_spi_driver);
#include <linux/property.h>
#include <linux/spi/spi.h>
+#include <drm/drm_atomic_helper.h>
#include <drm/drm_drv.h>
+#include <drm/drm_fb_helper.h>
#include <drm/drm_gem_cma_helper.h>
#include <drm/drm_gem_framebuffer_helper.h>
#include <drm/drm_modeset_helper.h>
struct drm_crtc_state *crtc_state,
struct drm_plane_state *plane_state)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(pipe->crtc.dev);
u8 addr_mode;
- int ret;
+ int ret, idx;
+
+ if (!drm_dev_enter(pipe->crtc.dev, &idx))
+ return;
DRM_DEBUG_KMS("\n");
ret = mipi_dbi_poweron_conditional_reset(mipi);
if (ret < 0)
- return;
+ goto out_exit;
if (ret == 1)
goto out_enable;
addr_mode |= ILI9341_MADCTL_BGR;
mipi_dbi_command(mipi, MIPI_DCS_SET_ADDRESS_MODE, addr_mode);
mipi_dbi_enable_flush(mipi, crtc_state, plane_state);
+out_exit:
+ drm_dev_exit(idx);
}
static const struct drm_simple_display_pipe_funcs ili9341_pipe_funcs = {
};
static const struct drm_display_mode yx240qv29_mode = {
- TINYDRM_MODE(240, 320, 37, 49),
+ DRM_SIMPLE_MODE(240, 320, 37, 49),
};
DEFINE_DRM_GEM_CMA_FOPS(ili9341_fops);
static struct drm_driver ili9341_driver = {
.driver_features = DRIVER_GEM | DRIVER_MODESET | DRIVER_PRIME | DRIVER_ATOMIC,
.fops = &ili9341_fops,
+ .release = mipi_dbi_release,
DRM_GEM_CMA_VMAP_DRIVER_OPS,
.debugfs_init = mipi_dbi_debugfs_init,
.name = "ili9341",
static int ili9341_probe(struct spi_device *spi)
{
struct device *dev = &spi->dev;
+ struct drm_device *drm;
struct mipi_dbi *mipi;
struct gpio_desc *dc;
u32 rotation = 0;
int ret;
- mipi = devm_kzalloc(dev, sizeof(*mipi), GFP_KERNEL);
+ mipi = kzalloc(sizeof(*mipi), GFP_KERNEL);
if (!mipi)
return -ENOMEM;
+ drm = &mipi->drm;
+ ret = devm_drm_dev_init(dev, drm, &ili9341_driver);
+ if (ret) {
+ kfree(mipi);
+ return ret;
+ }
+
+ drm_mode_config_init(drm);
+
mipi->reset = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_HIGH);
if (IS_ERR(mipi->reset)) {
DRM_DEV_ERROR(dev, "Failed to get gpio 'reset'\n");
if (ret)
return ret;
- ret = mipi_dbi_init(&spi->dev, mipi, &ili9341_pipe_funcs,
- &ili9341_driver, &yx240qv29_mode, rotation);
+ ret = mipi_dbi_init(mipi, &ili9341_pipe_funcs, &yx240qv29_mode, rotation);
if (ret)
return ret;
- spi_set_drvdata(spi, mipi);
+ drm_mode_config_reset(drm);
- return devm_tinydrm_register(&mipi->tinydrm);
+ ret = drm_dev_register(drm, 0);
+ if (ret)
+ return ret;
+
+ spi_set_drvdata(spi, drm);
+
+ drm_fbdev_generic_setup(drm, 0);
+
+ return 0;
}
-static void ili9341_shutdown(struct spi_device *spi)
+static int ili9341_remove(struct spi_device *spi)
{
- struct mipi_dbi *mipi = spi_get_drvdata(spi);
+ struct drm_device *drm = spi_get_drvdata(spi);
+
+ drm_dev_unplug(drm);
+ drm_atomic_helper_shutdown(drm);
- tinydrm_shutdown(&mipi->tinydrm);
+ return 0;
+}
+
+static void ili9341_shutdown(struct spi_device *spi)
+{
+ drm_atomic_helper_shutdown(spi_get_drvdata(spi));
}
static struct spi_driver ili9341_spi_driver = {
},
.id_table = ili9341_id,
.probe = ili9341_probe,
+ .remove = ili9341_remove,
.shutdown = ili9341_shutdown,
};
module_spi_driver(ili9341_spi_driver);
#include <linux/regulator/consumer.h>
#include <linux/spi/spi.h>
+#include <drm/drm_atomic_helper.h>
#include <drm/drm_drv.h>
+#include <drm/drm_fb_helper.h>
#include <drm/drm_gem_cma_helper.h>
#include <drm/drm_gem_framebuffer_helper.h>
#include <drm/drm_modeset_helper.h>
struct drm_crtc_state *crtc_state,
struct drm_plane_state *plane_state)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(pipe->crtc.dev);
u8 addr_mode;
- int ret;
+ int ret, idx;
+
+ if (!drm_dev_enter(pipe->crtc.dev, &idx))
+ return;
DRM_DEBUG_KMS("\n");
ret = mipi_dbi_poweron_conditional_reset(mipi);
if (ret < 0)
- return;
+ goto out_exit;
if (ret == 1)
goto out_enable;
addr_mode |= ILI9341_MADCTL_BGR;
mipi_dbi_command(mipi, MIPI_DCS_SET_ADDRESS_MODE, addr_mode);
mipi_dbi_enable_flush(mipi, crtc_state, plane_state);
+out_exit:
+ drm_dev_exit(idx);
}
static const struct drm_simple_display_pipe_funcs mi0283qt_pipe_funcs = {
};
static const struct drm_display_mode mi0283qt_mode = {
- TINYDRM_MODE(320, 240, 58, 43),
+ DRM_SIMPLE_MODE(320, 240, 58, 43),
};
DEFINE_DRM_GEM_CMA_FOPS(mi0283qt_fops);
.driver_features = DRIVER_GEM | DRIVER_MODESET | DRIVER_PRIME |
DRIVER_ATOMIC,
.fops = &mi0283qt_fops,
+ .release = mipi_dbi_release,
DRM_GEM_CMA_VMAP_DRIVER_OPS,
.debugfs_init = mipi_dbi_debugfs_init,
.name = "mi0283qt",
static int mi0283qt_probe(struct spi_device *spi)
{
struct device *dev = &spi->dev;
+ struct drm_device *drm;
struct mipi_dbi *mipi;
struct gpio_desc *dc;
u32 rotation = 0;
int ret;
- mipi = devm_kzalloc(dev, sizeof(*mipi), GFP_KERNEL);
+ mipi = kzalloc(sizeof(*mipi), GFP_KERNEL);
if (!mipi)
return -ENOMEM;
+ drm = &mipi->drm;
+ ret = devm_drm_dev_init(dev, drm, &mi0283qt_driver);
+ if (ret) {
+ kfree(mipi);
+ return ret;
+ }
+
+ drm_mode_config_init(drm);
+
mipi->reset = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_HIGH);
if (IS_ERR(mipi->reset)) {
DRM_DEV_ERROR(dev, "Failed to get gpio 'reset'\n");
if (ret)
return ret;
- ret = mipi_dbi_init(&spi->dev, mipi, &mi0283qt_pipe_funcs,
- &mi0283qt_driver, &mi0283qt_mode, rotation);
+ ret = mipi_dbi_init(mipi, &mi0283qt_pipe_funcs, &mi0283qt_mode, rotation);
+ if (ret)
+ return ret;
+
+ drm_mode_config_reset(drm);
+
+ ret = drm_dev_register(drm, 0);
if (ret)
return ret;
- spi_set_drvdata(spi, mipi);
+ spi_set_drvdata(spi, drm);
- return devm_tinydrm_register(&mipi->tinydrm);
+ drm_fbdev_generic_setup(drm, 0);
+
+ return 0;
}
-static void mi0283qt_shutdown(struct spi_device *spi)
+static int mi0283qt_remove(struct spi_device *spi)
{
- struct mipi_dbi *mipi = spi_get_drvdata(spi);
+ struct drm_device *drm = spi_get_drvdata(spi);
- tinydrm_shutdown(&mipi->tinydrm);
+ drm_dev_unplug(drm);
+ drm_atomic_helper_shutdown(drm);
+
+ return 0;
}
-static int __maybe_unused mi0283qt_pm_suspend(struct device *dev)
+static void mi0283qt_shutdown(struct spi_device *spi)
{
- struct mipi_dbi *mipi = dev_get_drvdata(dev);
+ drm_atomic_helper_shutdown(spi_get_drvdata(spi));
+}
- return drm_mode_config_helper_suspend(mipi->tinydrm.drm);
+static int __maybe_unused mi0283qt_pm_suspend(struct device *dev)
+{
+ return drm_mode_config_helper_suspend(dev_get_drvdata(dev));
}
static int __maybe_unused mi0283qt_pm_resume(struct device *dev)
{
- struct mipi_dbi *mipi = dev_get_drvdata(dev);
-
- drm_mode_config_helper_resume(mipi->tinydrm.drm);
+ drm_mode_config_helper_resume(dev_get_drvdata(dev));
return 0;
}
},
.id_table = mi0283qt_id,
.probe = mi0283qt_probe,
+ .remove = mi0283qt_remove,
.shutdown = mi0283qt_shutdown,
};
module_spi_driver(mi0283qt_spi_driver);
#include <drm/drm_drv.h>
#include <drm/drm_fb_cma_helper.h>
#include <drm/drm_gem_cma_helper.h>
+#include <drm/drm_format_helper.h>
#include <drm/drm_fourcc.h>
#include <drm/drm_gem_framebuffer_helper.h>
#include <drm/drm_vblank.h>
*/
int mipi_dbi_command_buf(struct mipi_dbi *mipi, u8 cmd, u8 *data, size_t len)
{
+ u8 *cmdbuf;
int ret;
+ /* SPI requires dma-safe buffers */
+ cmdbuf = kmemdup(&cmd, 1, GFP_KERNEL);
+ if (!cmdbuf)
+ return -ENOMEM;
+
mutex_lock(&mipi->cmdlock);
- ret = mipi->command(mipi, cmd, data, len);
+ ret = mipi->command(mipi, cmdbuf, data, len);
mutex_unlock(&mipi->cmdlock);
+ kfree(cmdbuf);
+
return ret;
}
EXPORT_SYMBOL(mipi_dbi_command_buf);
+/* This should only be used by mipi_dbi_command() */
+int mipi_dbi_command_stackbuf(struct mipi_dbi *mipi, u8 cmd, u8 *data, size_t len)
+{
+ u8 *buf;
+ int ret;
+
+ buf = kmemdup(data, len, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ ret = mipi_dbi_command_buf(mipi, cmd, buf, len);
+
+ kfree(buf);
+
+ return ret;
+}
+EXPORT_SYMBOL(mipi_dbi_command_stackbuf);
+
/**
* mipi_dbi_buf_copy - Copy a framebuffer, transforming it if necessary
* @dst: The destination buffer
switch (fb->format->format) {
case DRM_FORMAT_RGB565:
if (swap)
- tinydrm_swab16(dst, src, fb, clip);
+ drm_fb_swab16(dst, src, fb, clip);
else
- tinydrm_memcpy(dst, src, fb, clip);
+ drm_fb_memcpy(dst, src, fb, clip);
break;
case DRM_FORMAT_XRGB8888:
- tinydrm_xrgb8888_to_rgb565(dst, src, fb, clip, swap);
+ drm_fb_xrgb8888_to_rgb565(dst, src, fb, clip, swap);
break;
default:
dev_err_once(fb->dev->dev, "Format is not supported: %s\n",
static void mipi_dbi_fb_dirty(struct drm_framebuffer *fb, struct drm_rect *rect)
{
struct drm_gem_cma_object *cma_obj = drm_fb_cma_get_gem_obj(fb, 0);
- struct tinydrm_device *tdev = fb->dev->dev_private;
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(fb->dev);
unsigned int height = rect->y2 - rect->y1;
unsigned int width = rect->x2 - rect->x1;
bool swap = mipi->swap_bytes;
- int ret = 0;
+ int idx, ret = 0;
bool full;
void *tr;
if (!mipi->enabled)
return;
+ if (!drm_dev_enter(fb->dev, &idx))
+ return;
+
full = width == fb->width && height == fb->height;
DRM_DEBUG_KMS("Flushing [FB:%d] " DRM_RECT_FMT "\n", fb->base.id, DRM_RECT_ARG(rect));
err_msg:
if (ret)
dev_err_once(fb->dev->dev, "Failed to update display %d\n", ret);
+
+ drm_dev_exit(idx);
}
/**
.y1 = 0,
.y2 = fb->height,
};
+ int idx;
+
+ if (!drm_dev_enter(&mipi->drm, &idx))
+ return;
mipi->enabled = true;
mipi_dbi_fb_dirty(fb, &rect);
backlight_enable(mipi->backlight);
+
+ drm_dev_exit(idx);
}
EXPORT_SYMBOL(mipi_dbi_enable_flush);
static void mipi_dbi_blank(struct mipi_dbi *mipi)
{
- struct drm_device *drm = mipi->tinydrm.drm;
+ struct drm_device *drm = &mipi->drm;
u16 height = drm->mode_config.min_height;
u16 width = drm->mode_config.min_width;
size_t len = width * height * 2;
+ int idx;
+
+ if (!drm_dev_enter(drm, &idx))
+ return;
memset(mipi->tx_buf, 0, len);
(height >> 8) & 0xFF, (height - 1) & 0xFF);
mipi_dbi_command_buf(mipi, MIPI_DCS_WRITE_MEMORY_START,
(u8 *)mipi->tx_buf, len);
+
+ drm_dev_exit(idx);
}
/**
*/
void mipi_dbi_pipe_disable(struct drm_simple_display_pipe *pipe)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(pipe->crtc.dev);
+
+ if (!mipi->enabled)
+ return;
DRM_DEBUG_KMS("\n");
}
EXPORT_SYMBOL(mipi_dbi_pipe_disable);
+static const struct drm_mode_config_funcs mipi_dbi_mode_config_funcs = {
+ .fb_create = drm_gem_fb_create_with_dirty,
+ .atomic_check = drm_atomic_helper_check,
+ .atomic_commit = drm_atomic_helper_commit,
+};
+
static const uint32_t mipi_dbi_formats[] = {
DRM_FORMAT_RGB565,
DRM_FORMAT_XRGB8888,
/**
* mipi_dbi_init - MIPI DBI initialization
- * @dev: Parent device
* @mipi: &mipi_dbi structure to initialize
- * @pipe_funcs: Display pipe functions
- * @driver: DRM driver
+ * @funcs: Display pipe functions
* @mode: Display mode
* @rotation: Initial rotation in degrees Counter Clock Wise
*
- * This function initializes a &mipi_dbi structure and it's underlying
- * @tinydrm_device. It also sets up the display pipeline.
+ * This function sets up a &drm_simple_display_pipe with a &drm_connector that
+ * has one fixed &drm_display_mode which is rotated according to @rotation.
+ * This mode is used to set the mode config min/max width/height properties.
+ * Additionally &mipi_dbi.tx_buf is allocated.
*
* Supported formats: Native RGB565 and emulated XRGB8888.
*
- * Objects created by this function will be automatically freed on driver
- * detach (devres).
- *
* Returns:
* Zero on success, negative error code on failure.
*/
-int mipi_dbi_init(struct device *dev, struct mipi_dbi *mipi,
- const struct drm_simple_display_pipe_funcs *pipe_funcs,
- struct drm_driver *driver,
+int mipi_dbi_init(struct mipi_dbi *mipi,
+ const struct drm_simple_display_pipe_funcs *funcs,
const struct drm_display_mode *mode, unsigned int rotation)
{
size_t bufsize = mode->vdisplay * mode->hdisplay * sizeof(u16);
- struct tinydrm_device *tdev = &mipi->tinydrm;
+ struct drm_device *drm = &mipi->drm;
int ret;
if (!mipi->command)
mutex_init(&mipi->cmdlock);
- mipi->tx_buf = devm_kmalloc(dev, bufsize, GFP_KERNEL);
+ mipi->tx_buf = devm_kmalloc(drm->dev, bufsize, GFP_KERNEL);
if (!mipi->tx_buf)
return -ENOMEM;
- ret = devm_tinydrm_init(dev, tdev, driver);
- if (ret)
- return ret;
-
/* TODO: Maybe add DRM_MODE_CONNECTOR_SPI */
- ret = tinydrm_display_pipe_init(tdev, pipe_funcs,
+ ret = tinydrm_display_pipe_init(drm, &mipi->pipe, funcs,
DRM_MODE_CONNECTOR_VIRTUAL,
mipi_dbi_formats,
ARRAY_SIZE(mipi_dbi_formats), mode,
if (ret)
return ret;
- drm_plane_enable_fb_damage_clips(&tdev->pipe.plane);
+ drm_plane_enable_fb_damage_clips(&mipi->pipe.plane);
- tdev->drm->mode_config.preferred_depth = 16;
+ drm->mode_config.funcs = &mipi_dbi_mode_config_funcs;
+ drm->mode_config.preferred_depth = 16;
mipi->rotation = rotation;
- drm_mode_config_reset(tdev->drm);
-
DRM_DEBUG_KMS("preferred_depth=%u, rotation = %u\n",
- tdev->drm->mode_config.preferred_depth, rotation);
+ drm->mode_config.preferred_depth, rotation);
return 0;
}
EXPORT_SYMBOL(mipi_dbi_init);
+/**
+ * mipi_dbi_release - DRM driver release helper
+ * @drm: DRM device
+ *
+ * This function finalizes and frees &mipi_dbi.
+ *
+ * Drivers can use this as their &drm_driver->release callback.
+ */
+void mipi_dbi_release(struct drm_device *drm)
+{
+ struct mipi_dbi *dbi = drm_to_mipi_dbi(drm);
+
+ DRM_DEBUG_DRIVER("\n");
+
+ drm_mode_config_cleanup(drm);
+ drm_dev_fini(drm);
+ kfree(dbi);
+}
+EXPORT_SYMBOL(mipi_dbi_release);
+
/**
* mipi_dbi_hw_reset - Hardware reset of controller
* @mipi: MIPI DBI structure
static int mipi_dbi_poweron_reset_conditional(struct mipi_dbi *mipi, bool cond)
{
- struct device *dev = mipi->tinydrm.drm->dev;
+ struct device *dev = mipi->drm.dev;
int ret;
if (mipi->regulator) {
return 0;
}
-static int mipi_dbi_typec1_command(struct mipi_dbi *mipi, u8 cmd,
+static int mipi_dbi_typec1_command(struct mipi_dbi *mipi, u8 *cmd,
u8 *parameters, size_t num)
{
- unsigned int bpw = (cmd == MIPI_DCS_WRITE_MEMORY_START) ? 16 : 8;
+ unsigned int bpw = (*cmd == MIPI_DCS_WRITE_MEMORY_START) ? 16 : 8;
int ret;
- if (mipi_dbi_command_is_read(mipi, cmd))
+ if (mipi_dbi_command_is_read(mipi, *cmd))
return -ENOTSUPP;
- MIPI_DBI_DEBUG_COMMAND(cmd, parameters, num);
+ MIPI_DBI_DEBUG_COMMAND(*cmd, parameters, num);
- ret = mipi_dbi_spi1_transfer(mipi, 0, &cmd, 1, 8);
+ ret = mipi_dbi_spi1_transfer(mipi, 0, cmd, 1, 8);
if (ret || !num)
return ret;
/* MIPI DBI Type C Option 3 */
-static int mipi_dbi_typec3_command_read(struct mipi_dbi *mipi, u8 cmd,
+static int mipi_dbi_typec3_command_read(struct mipi_dbi *mipi, u8 *cmd,
u8 *data, size_t len)
{
struct spi_device *spi = mipi->spi;
struct spi_transfer tr[2] = {
{
.speed_hz = speed_hz,
- .tx_buf = &cmd,
+ .tx_buf = cmd,
.len = 1,
}, {
.speed_hz = speed_hz,
* Support non-standard 24-bit and 32-bit Nokia read commands which
* start with a dummy clock, so we need to read an extra byte.
*/
- if (cmd == MIPI_DCS_GET_DISPLAY_ID ||
- cmd == MIPI_DCS_GET_DISPLAY_STATUS) {
+ if (*cmd == MIPI_DCS_GET_DISPLAY_ID ||
+ *cmd == MIPI_DCS_GET_DISPLAY_STATUS) {
if (!(len == 3 || len == 4))
return -EINVAL;
data[i] = (buf[i] << 1) | !!(buf[i + 1] & BIT(7));
}
- MIPI_DBI_DEBUG_COMMAND(cmd, data, len);
+ MIPI_DBI_DEBUG_COMMAND(*cmd, data, len);
err_free:
kfree(buf);
return ret;
}
-static int mipi_dbi_typec3_command(struct mipi_dbi *mipi, u8 cmd,
+static int mipi_dbi_typec3_command(struct mipi_dbi *mipi, u8 *cmd,
u8 *par, size_t num)
{
struct spi_device *spi = mipi->spi;
u32 speed_hz;
int ret;
- if (mipi_dbi_command_is_read(mipi, cmd))
+ if (mipi_dbi_command_is_read(mipi, *cmd))
return mipi_dbi_typec3_command_read(mipi, cmd, par, num);
- MIPI_DBI_DEBUG_COMMAND(cmd, par, num);
+ MIPI_DBI_DEBUG_COMMAND(*cmd, par, num);
gpiod_set_value_cansleep(mipi->dc, 0);
speed_hz = mipi_dbi_spi_cmd_max_speed(spi, 1);
- ret = tinydrm_spi_transfer(spi, speed_hz, NULL, 8, &cmd, 1);
+ ret = tinydrm_spi_transfer(spi, speed_hz, NULL, 8, cmd, 1);
if (ret || !num)
return ret;
- if (cmd == MIPI_DCS_WRITE_MEMORY_START && !mipi->swap_bytes)
+ if (*cmd == MIPI_DCS_WRITE_MEMORY_START && !mipi->swap_bytes)
bpw = 16;
gpiod_set_value_cansleep(mipi->dc, 1);
* Even though it's not the SPI device that does DMA (the master does),
* the dma mask is necessary for the dma_alloc_wc() in
* drm_gem_cma_create(). The dma_addr returned will be a physical
- * adddress which might be different from the bus address, but this is
+ * address which might be different from the bus address, but this is
* not a problem since the address will not be used.
* The virtual address is used in the transfer and the SPI core
* re-maps it on the SPI master device using the DMA streaming API
u8 val, cmd = 0, parameters[64];
char *buf, *pos, *token;
unsigned int i;
- int ret;
+ int ret, idx;
+
+ if (!drm_dev_enter(&mipi->drm, &idx))
+ return -ENODEV;
buf = memdup_user_nul(ubuf, count);
- if (IS_ERR(buf))
- return PTR_ERR(buf);
+ if (IS_ERR(buf)) {
+ ret = PTR_ERR(buf);
+ goto err_exit;
+ }
/* strip trailing whitespace */
for (i = count - 1; i > 0; i--)
err_free:
kfree(buf);
+err_exit:
+ drm_dev_exit(idx);
return ret < 0 ? ret : count;
}
{
struct mipi_dbi *mipi = m->private;
u8 cmd, val[4];
+ int ret, idx;
size_t len;
- int ret;
+
+ if (!drm_dev_enter(&mipi->drm, &idx))
+ return -ENODEV;
for (cmd = 0; cmd < 255; cmd++) {
if (!mipi_dbi_command_is_read(mipi, cmd))
seq_printf(m, "%*phN\n", (int)len, val);
}
+ drm_dev_exit(idx);
+
return 0;
}
*/
int mipi_dbi_debugfs_init(struct drm_minor *minor)
{
- struct tinydrm_device *tdev = minor->dev->dev_private;
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(minor->dev);
umode_t mode = S_IFREG | S_IWUSR;
if (mipi->read_commands)
#include <linux/spi/spi.h>
#include <linux/thermal.h>
+#include <drm/drm_atomic_helper.h>
#include <drm/drm_damage_helper.h>
#include <drm/drm_drv.h>
#include <drm/drm_fb_cma_helper.h>
+#include <drm/drm_fb_helper.h>
+#include <drm/drm_format_helper.h>
#include <drm/drm_gem_cma_helper.h>
#include <drm/drm_gem_framebuffer_helper.h>
#include <drm/drm_rect.h>
#include <drm/drm_vblank.h>
-#include <drm/tinydrm/tinydrm.h>
+#include <drm/drm_simple_kms_helper.h>
#include <drm/tinydrm/tinydrm-helpers.h>
#define REPAPER_RID_G2_COG_ID 0x12
};
struct repaper_epd {
- struct tinydrm_device tinydrm;
+ struct drm_device drm;
+ struct drm_simple_display_pipe pipe;
struct spi_device *spi;
struct gpio_desc *panel_on;
bool partial;
};
-static inline struct repaper_epd *
-epd_from_tinydrm(struct tinydrm_device *tdev)
+static inline struct repaper_epd *drm_to_epd(struct drm_device *drm)
{
- return container_of(tdev, struct repaper_epd, tinydrm);
+ return container_of(drm, struct repaper_epd, drm);
}
static int repaper_spi_transfer(struct spi_device *spi, u8 header,
{
struct drm_gem_cma_object *cma_obj = drm_fb_cma_get_gem_obj(fb, 0);
struct dma_buf_attachment *import_attach = cma_obj->base.import_attach;
- struct tinydrm_device *tdev = fb->dev->dev_private;
- struct repaper_epd *epd = epd_from_tinydrm(tdev);
+ struct repaper_epd *epd = drm_to_epd(fb->dev);
struct drm_rect clip;
+ int idx, ret = 0;
u8 *buf = NULL;
- int ret = 0;
+
+ if (!epd->enabled)
+ return 0;
+
+ if (!drm_dev_enter(fb->dev, &idx))
+ return -ENODEV;
/* repaper can't do partial updates */
clip.x1 = 0;
clip.y1 = 0;
clip.y2 = fb->height;
- if (!epd->enabled)
- return 0;
-
repaper_get_temperature(epd);
DRM_DEBUG("Flushing [FB:%d] st=%ums\n", fb->base.id,
epd->factored_stage_time);
buf = kmalloc_array(fb->width, fb->height, GFP_KERNEL);
- if (!buf)
- return -ENOMEM;
+ if (!buf) {
+ ret = -ENOMEM;
+ goto out_exit;
+ }
if (import_attach) {
ret = dma_buf_begin_cpu_access(import_attach->dmabuf,
goto out_free;
}
- tinydrm_xrgb8888_to_gray8(buf, cma_obj->vaddr, fb, &clip);
+ drm_fb_xrgb8888_to_gray8(buf, cma_obj->vaddr, fb, &clip);
if (import_attach) {
ret = dma_buf_end_cpu_access(import_attach->dmabuf,
out_free:
kfree(buf);
+out_exit:
+ drm_dev_exit(idx);
return ret;
}
struct drm_crtc_state *crtc_state,
struct drm_plane_state *plane_state)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct repaper_epd *epd = epd_from_tinydrm(tdev);
+ struct repaper_epd *epd = drm_to_epd(pipe->crtc.dev);
struct spi_device *spi = epd->spi;
struct device *dev = &spi->dev;
bool dc_ok = false;
- int i, ret;
+ int i, ret, idx;
+
+ if (!drm_dev_enter(pipe->crtc.dev, &idx))
+ return;
DRM_DEBUG_DRIVER("\n");
if (!i) {
DRM_DEV_ERROR(dev, "timeout waiting for panel to become ready.\n");
power_off(epd);
- return;
+ goto out_exit;
}
repaper_read_id(spi);
else
dev_err(dev, "wrong COG ID 0x%02x\n", ret);
power_off(epd);
- return;
+ goto out_exit;
}
/* Disable OE */
else
DRM_DEV_ERROR(dev, "panel is reported broken\n");
power_off(epd);
- return;
+ goto out_exit;
}
/* Power saving mode */
if (ret < 0) {
DRM_DEV_ERROR(dev, "failed to read chip (%d)\n", ret);
power_off(epd);
- return;
+ goto out_exit;
}
if (ret & 0x40) {
if (!dc_ok) {
DRM_DEV_ERROR(dev, "dc/dc failed\n");
power_off(epd);
- return;
+ goto out_exit;
}
/*
epd->enabled = true;
epd->partial = false;
+out_exit:
+ drm_dev_exit(idx);
}
static void repaper_pipe_disable(struct drm_simple_display_pipe *pipe)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct repaper_epd *epd = epd_from_tinydrm(tdev);
+ struct repaper_epd *epd = drm_to_epd(pipe->crtc.dev);
struct spi_device *spi = epd->spi;
unsigned int line;
+ /*
+ * This callback is not protected by drm_dev_enter/exit since we want to
+ * turn off the display on regular driver unload. It's highly unlikely
+ * that the underlying SPI controller is gone should this be called after
+ * unplug.
+ */
+
+ if (!epd->enabled)
+ return;
+
DRM_DEBUG_DRIVER("\n");
epd->enabled = false;
.prepare_fb = drm_gem_fb_simple_display_pipe_prepare_fb,
};
+static const struct drm_mode_config_funcs repaper_mode_config_funcs = {
+ .fb_create = drm_gem_fb_create_with_dirty,
+ .atomic_check = drm_atomic_helper_check,
+ .atomic_commit = drm_atomic_helper_commit,
+};
+
+static void repaper_release(struct drm_device *drm)
+{
+ struct repaper_epd *epd = drm_to_epd(drm);
+
+ DRM_DEBUG_DRIVER("\n");
+
+ drm_mode_config_cleanup(drm);
+ drm_dev_fini(drm);
+ kfree(epd);
+}
+
static const uint32_t repaper_formats[] = {
DRM_FORMAT_XRGB8888,
};
static const struct drm_display_mode repaper_e1144cs021_mode = {
- TINYDRM_MODE(128, 96, 29, 22),
+ DRM_SIMPLE_MODE(128, 96, 29, 22),
};
static const u8 repaper_e1144cs021_cs[] = { 0x00, 0x00, 0x00, 0x00,
0x00, 0x0f, 0xff, 0x00 };
static const struct drm_display_mode repaper_e1190cs021_mode = {
- TINYDRM_MODE(144, 128, 36, 32),
+ DRM_SIMPLE_MODE(144, 128, 36, 32),
};
static const u8 repaper_e1190cs021_cs[] = { 0x00, 0x00, 0x00, 0x03,
0xfc, 0x00, 0x00, 0xff };
static const struct drm_display_mode repaper_e2200cs021_mode = {
- TINYDRM_MODE(200, 96, 46, 22),
+ DRM_SIMPLE_MODE(200, 96, 46, 22),
};
static const u8 repaper_e2200cs021_cs[] = { 0x00, 0x00, 0x00, 0x00,
0x01, 0xff, 0xe0, 0x00 };
static const struct drm_display_mode repaper_e2271cs021_mode = {
- TINYDRM_MODE(264, 176, 57, 38),
+ DRM_SIMPLE_MODE(264, 176, 57, 38),
};
static const u8 repaper_e2271cs021_cs[] = { 0x00, 0x00, 0x00, 0x7f,
.driver_features = DRIVER_GEM | DRIVER_MODESET | DRIVER_PRIME |
DRIVER_ATOMIC,
.fops = &repaper_fops,
+ .release = repaper_release,
DRM_GEM_CMA_VMAP_DRIVER_OPS,
.name = "repaper",
.desc = "Pervasive Displays RePaper e-ink panels",
const struct spi_device_id *spi_id;
const struct of_device_id *match;
struct device *dev = &spi->dev;
- struct tinydrm_device *tdev;
enum repaper_model model;
const char *thermal_zone;
struct repaper_epd *epd;
size_t line_buffer_size;
+ struct drm_device *drm;
int ret;
match = of_match_device(repaper_of_match, dev);
}
}
- epd = devm_kzalloc(dev, sizeof(*epd), GFP_KERNEL);
+ epd = kzalloc(sizeof(*epd), GFP_KERNEL);
if (!epd)
return -ENOMEM;
+ drm = &epd->drm;
+
+ ret = devm_drm_dev_init(dev, drm, &repaper_driver);
+ if (ret) {
+ kfree(epd);
+ return ret;
+ }
+
+ drm_mode_config_init(drm);
+ drm->mode_config.funcs = &repaper_mode_config_funcs;
+
epd->spi = spi;
epd->panel_on = devm_gpiod_get(dev, "panel-on", GPIOD_OUT_LOW);
if (!epd->current_frame)
return -ENOMEM;
- tdev = &epd->tinydrm;
-
- ret = devm_tinydrm_init(dev, tdev, &repaper_driver);
- if (ret)
- return ret;
-
- ret = tinydrm_display_pipe_init(tdev, &repaper_pipe_funcs,
+ ret = tinydrm_display_pipe_init(drm, &epd->pipe, &repaper_pipe_funcs,
DRM_MODE_CONNECTOR_VIRTUAL,
repaper_formats,
ARRAY_SIZE(repaper_formats), mode, 0);
if (ret)
return ret;
- drm_mode_config_reset(tdev->drm);
- spi_set_drvdata(spi, tdev);
+ drm_mode_config_reset(drm);
+
+ ret = drm_dev_register(drm, 0);
+ if (ret)
+ return ret;
+
+ spi_set_drvdata(spi, drm);
DRM_DEBUG_DRIVER("SPI speed: %uMHz\n", spi->max_speed_hz / 1000000);
- return devm_tinydrm_register(tdev);
+ drm_fbdev_generic_setup(drm, 0);
+
+ return 0;
}
-static void repaper_shutdown(struct spi_device *spi)
+static int repaper_remove(struct spi_device *spi)
{
- struct tinydrm_device *tdev = spi_get_drvdata(spi);
+ struct drm_device *drm = spi_get_drvdata(spi);
+
+ drm_dev_unplug(drm);
+ drm_atomic_helper_shutdown(drm);
+
+ return 0;
+}
- tinydrm_shutdown(tdev);
+static void repaper_shutdown(struct spi_device *spi)
+{
+ drm_atomic_helper_shutdown(spi_get_drvdata(spi));
}
static struct spi_driver repaper_spi_driver = {
},
.id_table = repaper_id,
.probe = repaper_probe,
+ .remove = repaper_remove,
.shutdown = repaper_shutdown,
};
module_spi_driver(repaper_spi_driver);
#include <linux/spi/spi.h>
#include <video/mipi_display.h>
+#include <drm/drm_atomic_helper.h>
#include <drm/drm_damage_helper.h>
#include <drm/drm_drv.h>
#include <drm/drm_fb_cma_helper.h>
+#include <drm/drm_fb_helper.h>
+#include <drm/drm_format_helper.h>
#include <drm/drm_gem_cma_helper.h>
#include <drm/drm_gem_framebuffer_helper.h>
#include <drm/drm_rect.h>
if (!buf)
return;
- tinydrm_xrgb8888_to_gray8(buf, vaddr, fb, clip);
+ drm_fb_xrgb8888_to_gray8(buf, vaddr, fb, clip);
src = buf;
for (y = clip->y1; y < clip->y2; y++) {
static void st7586_fb_dirty(struct drm_framebuffer *fb, struct drm_rect *rect)
{
- struct tinydrm_device *tdev = fb->dev->dev_private;
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
- int start, end;
- int ret = 0;
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(fb->dev);
+ int start, end, idx, ret = 0;
if (!mipi->enabled)
return;
+ if (!drm_dev_enter(fb->dev, &idx))
+ return;
+
/* 3 pixels per byte, so grow clip to nearest multiple of 3 */
rect->x1 = rounddown(rect->x1, 3);
rect->x2 = roundup(rect->x2, 3);
err_msg:
if (ret)
dev_err_once(fb->dev->dev, "Failed to update display %d\n", ret);
+
+ drm_dev_exit(idx);
}
static void st7586_pipe_update(struct drm_simple_display_pipe *pipe,
struct drm_crtc_state *crtc_state,
struct drm_plane_state *plane_state)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(pipe->crtc.dev);
struct drm_framebuffer *fb = plane_state->fb;
struct drm_rect rect = {
.x1 = 0,
.y1 = 0,
.y2 = fb->height,
};
- int ret;
+ int idx, ret;
u8 addr_mode;
+ if (!drm_dev_enter(pipe->crtc.dev, &idx))
+ return;
+
DRM_DEBUG_KMS("\n");
ret = mipi_dbi_poweron_reset(mipi);
if (ret)
- return;
+ goto out_exit;
mipi_dbi_command(mipi, ST7586_AUTO_READ_CTRL, 0x9f);
mipi_dbi_command(mipi, ST7586_OTP_RW_CTRL, 0x00);
st7586_fb_dirty(fb, &rect);
mipi_dbi_command(mipi, MIPI_DCS_SET_DISPLAY_ON);
+out_exit:
+ drm_dev_exit(idx);
}
static void st7586_pipe_disable(struct drm_simple_display_pipe *pipe)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(pipe->crtc.dev);
+
+ /*
+ * This callback is not protected by drm_dev_enter/exit since we want to
+ * turn off the display on regular driver unload. It's highly unlikely
+ * that the underlying SPI controller is gone should this be called after
+ * unplug.
+ */
DRM_DEBUG_KMS("\n");
DRM_FORMAT_XRGB8888,
};
-static int st7586_init(struct device *dev, struct mipi_dbi *mipi,
- const struct drm_simple_display_pipe_funcs *pipe_funcs,
- struct drm_driver *driver, const struct drm_display_mode *mode,
- unsigned int rotation)
-{
- size_t bufsize = (mode->vdisplay + 2) / 3 * mode->hdisplay;
- struct tinydrm_device *tdev = &mipi->tinydrm;
- int ret;
-
- mutex_init(&mipi->cmdlock);
-
- mipi->tx_buf = devm_kmalloc(dev, bufsize, GFP_KERNEL);
- if (!mipi->tx_buf)
- return -ENOMEM;
-
- ret = devm_tinydrm_init(dev, tdev, driver);
- if (ret)
- return ret;
-
- ret = tinydrm_display_pipe_init(tdev, pipe_funcs,
- DRM_MODE_CONNECTOR_VIRTUAL,
- st7586_formats,
- ARRAY_SIZE(st7586_formats),
- mode, rotation);
- if (ret)
- return ret;
-
- drm_plane_enable_fb_damage_clips(&tdev->pipe.plane);
-
- tdev->drm->mode_config.preferred_depth = 32;
- mipi->rotation = rotation;
-
- drm_mode_config_reset(tdev->drm);
-
- DRM_DEBUG_KMS("preferred_depth=%u, rotation = %u\n",
- tdev->drm->mode_config.preferred_depth, rotation);
-
- return 0;
-}
-
static const struct drm_simple_display_pipe_funcs st7586_pipe_funcs = {
.enable = st7586_pipe_enable,
.disable = st7586_pipe_disable,
.prepare_fb = drm_gem_fb_simple_display_pipe_prepare_fb,
};
+static const struct drm_mode_config_funcs st7586_mode_config_funcs = {
+ .fb_create = drm_gem_fb_create_with_dirty,
+ .atomic_check = drm_atomic_helper_check,
+ .atomic_commit = drm_atomic_helper_commit,
+};
+
static const struct drm_display_mode st7586_mode = {
- TINYDRM_MODE(178, 128, 37, 27),
+ DRM_SIMPLE_MODE(178, 128, 37, 27),
};
DEFINE_DRM_GEM_CMA_FOPS(st7586_fops);
.driver_features = DRIVER_GEM | DRIVER_MODESET | DRIVER_PRIME |
DRIVER_ATOMIC,
.fops = &st7586_fops,
+ .release = mipi_dbi_release,
DRM_GEM_CMA_VMAP_DRIVER_OPS,
.debugfs_init = mipi_dbi_debugfs_init,
.name = "st7586",
static int st7586_probe(struct spi_device *spi)
{
struct device *dev = &spi->dev;
+ struct drm_device *drm;
struct mipi_dbi *mipi;
struct gpio_desc *a0;
u32 rotation = 0;
+ size_t bufsize;
int ret;
- mipi = devm_kzalloc(dev, sizeof(*mipi), GFP_KERNEL);
+ mipi = kzalloc(sizeof(*mipi), GFP_KERNEL);
if (!mipi)
return -ENOMEM;
+ drm = &mipi->drm;
+ ret = devm_drm_dev_init(dev, drm, &st7586_driver);
+ if (ret) {
+ kfree(mipi);
+ return ret;
+ }
+
+ drm_mode_config_init(drm);
+ drm->mode_config.preferred_depth = 32;
+ drm->mode_config.funcs = &st7586_mode_config_funcs;
+
+ mutex_init(&mipi->cmdlock);
+
+ bufsize = (st7586_mode.vdisplay + 2) / 3 * st7586_mode.hdisplay;
+ mipi->tx_buf = devm_kmalloc(dev, bufsize, GFP_KERNEL);
+ if (!mipi->tx_buf)
+ return -ENOMEM;
+
mipi->reset = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
if (IS_ERR(mipi->reset)) {
DRM_DEV_ERROR(dev, "Failed to get gpio 'reset'\n");
}
device_property_read_u32(dev, "rotation", &rotation);
+ mipi->rotation = rotation;
ret = mipi_dbi_spi_init(spi, mipi, a0);
if (ret)
*/
mipi->swap_bytes = true;
- ret = st7586_init(&spi->dev, mipi, &st7586_pipe_funcs, &st7586_driver,
- &st7586_mode, rotation);
+ ret = tinydrm_display_pipe_init(drm, &mipi->pipe, &st7586_pipe_funcs,
+ DRM_MODE_CONNECTOR_VIRTUAL,
+ st7586_formats, ARRAY_SIZE(st7586_formats),
+ &st7586_mode, rotation);
if (ret)
return ret;
- spi_set_drvdata(spi, mipi);
+ drm_plane_enable_fb_damage_clips(&mipi->pipe.plane);
- return devm_tinydrm_register(&mipi->tinydrm);
+ drm_mode_config_reset(drm);
+
+ ret = drm_dev_register(drm, 0);
+ if (ret)
+ return ret;
+
+ spi_set_drvdata(spi, drm);
+
+ DRM_DEBUG_KMS("preferred_depth=%u, rotation = %u\n",
+ drm->mode_config.preferred_depth, rotation);
+
+ drm_fbdev_generic_setup(drm, 0);
+
+ return 0;
}
-static void st7586_shutdown(struct spi_device *spi)
+static int st7586_remove(struct spi_device *spi)
{
- struct mipi_dbi *mipi = spi_get_drvdata(spi);
+ struct drm_device *drm = spi_get_drvdata(spi);
+
+ drm_dev_unplug(drm);
+ drm_atomic_helper_shutdown(drm);
+
+ return 0;
+}
- tinydrm_shutdown(&mipi->tinydrm);
+static void st7586_shutdown(struct spi_device *spi)
+{
+ drm_atomic_helper_shutdown(spi_get_drvdata(spi));
}
static struct spi_driver st7586_spi_driver = {
},
.id_table = st7586_id,
.probe = st7586_probe,
+ .remove = st7586_remove,
.shutdown = st7586_shutdown,
};
module_spi_driver(st7586_spi_driver);
#include <linux/spi/spi.h>
#include <video/mipi_display.h>
+#include <drm/drm_atomic_helper.h>
#include <drm/drm_drv.h>
+#include <drm/drm_fb_helper.h>
#include <drm/drm_gem_cma_helper.h>
#include <drm/drm_gem_framebuffer_helper.h>
#include <drm/tinydrm/mipi-dbi.h>
struct drm_crtc_state *crtc_state,
struct drm_plane_state *plane_state)
{
- struct tinydrm_device *tdev = pipe_to_tinydrm(pipe);
- struct mipi_dbi *mipi = mipi_dbi_from_tinydrm(tdev);
- int ret;
+ struct mipi_dbi *mipi = drm_to_mipi_dbi(pipe->crtc.dev);
+ int ret, idx;
u8 addr_mode;
+ if (!drm_dev_enter(pipe->crtc.dev, &idx))
+ return;
+
DRM_DEBUG_KMS("\n");
ret = mipi_dbi_poweron_reset(mipi);
if (ret)
- return;
+ goto out_exit;
msleep(150);
msleep(20);
mipi_dbi_enable_flush(mipi, crtc_state, plane_state);
+out_exit:
+ drm_dev_exit(idx);
}
static const struct drm_simple_display_pipe_funcs jd_t18003_t01_pipe_funcs = {
};
static const struct drm_display_mode jd_t18003_t01_mode = {
- TINYDRM_MODE(128, 160, 28, 35),
+ DRM_SIMPLE_MODE(128, 160, 28, 35),
};
DEFINE_DRM_GEM_CMA_FOPS(st7735r_fops);
.driver_features = DRIVER_GEM | DRIVER_MODESET | DRIVER_PRIME |
DRIVER_ATOMIC,
.fops = &st7735r_fops,
+ .release = mipi_dbi_release,
DRM_GEM_CMA_VMAP_DRIVER_OPS,
.debugfs_init = mipi_dbi_debugfs_init,
.name = "st7735r",
static int st7735r_probe(struct spi_device *spi)
{
struct device *dev = &spi->dev;
+ struct drm_device *drm;
struct mipi_dbi *mipi;
struct gpio_desc *dc;
u32 rotation = 0;
int ret;
- mipi = devm_kzalloc(dev, sizeof(*mipi), GFP_KERNEL);
+ mipi = kzalloc(sizeof(*mipi), GFP_KERNEL);
if (!mipi)
return -ENOMEM;
+ drm = &mipi->drm;
+ ret = devm_drm_dev_init(dev, drm, &st7735r_driver);
+ if (ret) {
+ kfree(mipi);
+ return ret;
+ }
+
+ drm_mode_config_init(drm);
+
mipi->reset = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
if (IS_ERR(mipi->reset)) {
DRM_DEV_ERROR(dev, "Failed to get gpio 'reset'\n");
/* Cannot read from Adafruit 1.8" display via SPI */
mipi->read_commands = NULL;
- ret = mipi_dbi_init(&spi->dev, mipi, &jd_t18003_t01_pipe_funcs,
- &st7735r_driver, &jd_t18003_t01_mode, rotation);
+ ret = mipi_dbi_init(mipi, &jd_t18003_t01_pipe_funcs, &jd_t18003_t01_mode, rotation);
if (ret)
return ret;
- spi_set_drvdata(spi, mipi);
+ drm_mode_config_reset(drm);
- return devm_tinydrm_register(&mipi->tinydrm);
+ ret = drm_dev_register(drm, 0);
+ if (ret)
+ return ret;
+
+ spi_set_drvdata(spi, drm);
+
+ drm_fbdev_generic_setup(drm, 0);
+
+ return 0;
}
-static void st7735r_shutdown(struct spi_device *spi)
+static int st7735r_remove(struct spi_device *spi)
{
- struct mipi_dbi *mipi = spi_get_drvdata(spi);
+ struct drm_device *drm = spi_get_drvdata(spi);
+
+ drm_dev_unplug(drm);
+ drm_atomic_helper_shutdown(drm);
- tinydrm_shutdown(&mipi->tinydrm);
+ return 0;
+}
+
+static void st7735r_shutdown(struct spi_device *spi)
+{
+ drm_atomic_helper_shutdown(spi_get_drvdata(spi));
}
static struct spi_driver st7735r_spi_driver = {
},
.id_table = st7735r_id,
.probe = st7735r_probe,
+ .remove = st7735r_remove,
.shutdown = st7735r_shutdown,
};
module_spi_driver(st7735r_spi_driver);
int ttm_bo_device_init(struct ttm_bo_device *bdev,
struct ttm_bo_driver *driver,
struct address_space *mapping,
- uint64_t file_page_offset,
bool need_dma32)
{
struct ttm_bo_global *glob = &ttm_bo_glob;
if (unlikely(ret != 0))
goto out_no_sys;
- drm_vma_offset_manager_init(&bdev->vma_manager, file_page_offset,
- 0x10000000);
+ drm_vma_offset_manager_init(&bdev->vma_manager,
+ DRM_FILE_PAGE_OFFSET_START,
+ DRM_FILE_PAGE_OFFSET_SIZE);
INIT_DELAYED_WORK(&bdev->wq, ttm_bo_delayed_workqueue);
INIT_LIST_HEAD(&bdev->ddestroy);
bdev->dev_mapping = mapping;
struct ttm_buffer_object *bo;
int ret;
+ if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET_START))
+ return -EINVAL;
+
bo = ttm_bo_vm_lookup(bdev, vma->vm_pgoff, vma_pages(vma));
if (unlikely(!bo))
return -EINVAL;
struct ttm_validate_buffer *entry;
struct ttm_buffer_object *bo;
struct ttm_bo_global *glob;
- struct ttm_bo_device *bdev;
if (list_empty(list))
return;
bo = list_first_entry(list, struct ttm_validate_buffer, head)->bo;
- bdev = bo->bdev;
glob = bo->bdev->glob;
spin_lock(&glob->lru_lock);
struct ttm_mem_zone *zone =
container_of(kobj, struct ttm_mem_zone, kobj);
- pr_info("Zone %7s: Used memory at exit: %llu kiB\n",
+ pr_info("Zone %7s: Used memory at exit: %llu KiB\n",
zone->name, (unsigned long long)zone->used_mem >> 10);
kfree(zone);
}
#endif
for (i = 0; i < glob->num_zones; ++i) {
zone = glob->zones[i];
- pr_info("Zone %7s: Available graphics memory: %llu kiB\n",
+ pr_info("Zone %7s: Available graphics memory: %llu KiB\n",
zone->name, (unsigned long long)zone->max_mem >> 10);
}
ttm_page_alloc_init(glob, glob->zone_kernel->max_mem/(2*PAGE_SIZE));
void ttm_mem_global_free(struct ttm_mem_global *glob,
uint64_t amount)
{
- return ttm_mem_global_free_zone(glob, NULL, amount);
+ return ttm_mem_global_free_zone(glob, glob->zone_kernel, amount);
}
EXPORT_SYMBOL(ttm_mem_global_free);
{
/**
* Normal allocations of kernel memory are registered in
- * all zones.
+ * the kernel zone.
*/
- return ttm_mem_global_alloc_zone(glob, NULL, memory, ctx);
+ return ttm_mem_global_alloc_zone(glob, glob->zone_kernel, memory, ctx);
}
EXPORT_SYMBOL(ttm_mem_global_alloc);
/* Vsync IRQ at start of Vsync at first */
ctrl1 |= TVE200_VSTSTYPE_VSYNC;
- if (connector->display_info.bus_flags & DRM_BUS_FLAG_PIXDATA_NEGEDGE)
+ if (connector->display_info.bus_flags &
+ DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE)
ctrl1 |= TVE200_CTRL_TVCLKP;
if ((mode->hdisplay == 352 && mode->vdisplay == 240) || /* SIF(525) */
.llseek = noop_llseek,
};
+static void udl_driver_release(struct drm_device *dev)
+{
+ udl_fini(dev);
+ udl_modeset_cleanup(dev);
+ drm_dev_fini(dev);
+ kfree(dev);
+}
+
static struct drm_driver driver = {
.driver_features = DRIVER_MODESET | DRIVER_GEM | DRIVER_PRIME,
- .load = udl_driver_load,
- .unload = udl_driver_unload,
.release = udl_driver_release,
/* gem hooks */
.patchlevel = DRIVER_PATCHLEVEL,
};
+static struct udl_device *udl_driver_create(struct usb_interface *interface)
+{
+ struct usb_device *udev = interface_to_usbdev(interface);
+ struct udl_device *udl;
+ int r;
+
+ udl = kzalloc(sizeof(*udl), GFP_KERNEL);
+ if (!udl)
+ return ERR_PTR(-ENOMEM);
+
+ r = drm_dev_init(&udl->drm, &driver, &interface->dev);
+ if (r) {
+ kfree(udl);
+ return ERR_PTR(r);
+ }
+
+ udl->udev = udev;
+ udl->drm.dev_private = udl;
+
+ r = udl_init(udl);
+ if (r) {
+ drm_dev_fini(&udl->drm);
+ kfree(udl);
+ return ERR_PTR(r);
+ }
+
+ usb_set_intfdata(interface, udl);
+ return udl;
+}
+
static int udl_usb_probe(struct usb_interface *interface,
const struct usb_device_id *id)
{
- struct usb_device *udev = interface_to_usbdev(interface);
- struct drm_device *dev;
int r;
+ struct udl_device *udl;
- dev = drm_dev_alloc(&driver, &interface->dev);
- if (IS_ERR(dev))
- return PTR_ERR(dev);
+ udl = udl_driver_create(interface);
+ if (IS_ERR(udl))
+ return PTR_ERR(udl);
- r = drm_dev_register(dev, (unsigned long)udev);
+ r = drm_dev_register(&udl->drm, 0);
if (r)
goto err_free;
- usb_set_intfdata(interface, dev);
- DRM_INFO("Initialized udl on minor %d\n", dev->primary->index);
+ DRM_INFO("Initialized udl on minor %d\n", udl->drm.primary->index);
return 0;
err_free:
- drm_dev_put(dev);
+ drm_dev_put(&udl->drm);
return r;
}
udl_fbdev_unplug(dev);
udl_drop_usb(dev);
drm_dev_unplug(dev);
+ drm_dev_put(dev);
}
/*
struct udl_fbdev;
struct udl_device {
+ struct drm_device drm;
struct device *dev;
- struct drm_device *ddev;
struct usb_device *udev;
struct drm_crtc *crtc;
atomic_t cpu_kcycles_used; /* transpired during pixel processing */
};
+#define to_udl(x) container_of(x, struct udl_device, drm)
+
struct udl_gem_object {
struct drm_gem_object base;
struct page **pages;
int udl_submit_urb(struct drm_device *dev, struct urb *urb, size_t len);
void udl_urb_completion(struct urb *urb);
-int udl_driver_load(struct drm_device *dev, unsigned long flags);
-void udl_driver_unload(struct drm_device *dev);
-void udl_driver_release(struct drm_device *dev);
+int udl_init(struct udl_device *udl);
+void udl_fini(struct drm_device *dev);
int udl_fbdev_init(struct drm_device *dev);
void udl_fbdev_cleanup(struct drm_device *dev);
module_param(fb_defio, int, S_IWUSR | S_IRUSR | S_IWGRP | S_IRGRP);
struct udl_fbdev {
- struct drm_fb_helper helper;
+ struct drm_fb_helper helper; /* must be first */
struct udl_framebuffer ufb;
int fb_count;
};
int width, int height)
{
struct drm_device *dev = fb->base.dev;
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
int i, ret;
char *cmd;
cycles_t start_cycles, end_cycles;
{
struct udl_fbdev *ufbdev = info->par;
struct drm_device *dev = ufbdev->ufb.base.dev;
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
/* If the USB device is gone, we don't accept new opens */
- if (drm_dev_is_unplugged(udl->ddev))
+ if (drm_dev_is_unplugged(&udl->drm))
return -ENODEV;
ufbdev->fb_count++;
ret = PTR_ERR(info);
goto out_gfree;
}
- info->par = ufbdev;
ret = udl_framebuffer_init(dev, &ufbdev->ufb, &mode_cmd, obj);
if (ret)
ufbdev->helper.fb = fb;
- strcpy(info->fix.id, "udldrmfb");
-
info->screen_base = ufbdev->ufb.obj->vmapping;
info->fix.smem_len = size;
info->fix.smem_start = (unsigned long)ufbdev->ufb.obj->vmapping;
info->fbops = &udlfb_ops;
- drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(info, &ufbdev->helper, sizes->fb_width, sizes->fb_height);
+ drm_fb_helper_fill_info(info, &ufbdev->helper, sizes);
DRM_DEBUG_KMS("allocated %dx%d vmal %p\n",
fb->width, fb->height,
int udl_fbdev_init(struct drm_device *dev)
{
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
int bpp_sel = fb_bpp;
struct udl_fbdev *ufbdev;
int ret;
void udl_fbdev_cleanup(struct drm_device *dev)
{
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
if (!udl->fbdev)
return;
void udl_fbdev_unplug(struct drm_device *dev)
{
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
struct udl_fbdev *ufbdev;
if (!udl->fbdev)
return;
{
struct udl_gem_object *gobj;
struct drm_gem_object *obj;
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
int ret = 0;
mutex_lock(&udl->gem_lock);
static int udl_parse_vendor_descriptor(struct drm_device *dev,
struct usb_device *usbdev)
{
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
char *desc;
char *buf;
char *desc_end;
static void udl_free_urb_list(struct drm_device *dev)
{
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
int count = udl->urbs.count;
struct list_head *node;
struct urb_node *unode;
static int udl_alloc_urb_list(struct drm_device *dev, int count, size_t size)
{
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
struct urb *urb;
struct urb_node *unode;
char *buf;
struct urb *udl_get_urb(struct drm_device *dev)
{
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
int ret = 0;
struct list_head *entry;
struct urb_node *unode;
int udl_submit_urb(struct drm_device *dev, struct urb *urb, size_t len)
{
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
int ret;
BUG_ON(len > udl->urbs.size);
return ret;
}
-int udl_driver_load(struct drm_device *dev, unsigned long flags)
+int udl_init(struct udl_device *udl)
{
- struct usb_device *udev = (void*)flags;
- struct udl_device *udl;
+ struct drm_device *dev = &udl->drm;
int ret = -ENOMEM;
DRM_DEBUG("\n");
- udl = kzalloc(sizeof(struct udl_device), GFP_KERNEL);
- if (!udl)
- return -ENOMEM;
-
- udl->udev = udev;
- udl->ddev = dev;
- dev->dev_private = udl;
mutex_init(&udl->gem_lock);
err:
if (udl->urbs.count)
udl_free_urb_list(dev);
- kfree(udl);
DRM_ERROR("%d\n", ret);
return ret;
}
return 0;
}
-void udl_driver_unload(struct drm_device *dev)
+void udl_fini(struct drm_device *dev)
{
- struct udl_device *udl = dev->dev_private;
+ struct udl_device *udl = to_udl(dev);
drm_kms_helper_poll_fini(dev);
udl_free_urb_list(dev);
udl_fbdev_cleanup(dev);
- kfree(udl);
-}
-
-void udl_driver_release(struct drm_device *dev)
-{
- udl_modeset_cleanup(dev);
- drm_dev_fini(dev);
- kfree(dev);
}
depends on COMMON_CLK
depends on MMU
select DRM_SCHED
+ select DRM_GEM_SHMEM_HELPER
help
Choose this option if you have a system that has a Broadcom
V3D 3.x or newer GPU, such as BCM7268.
#include "v3d_drv.h"
#include "uapi/drm/v3d_drm.h"
-/* Pins the shmem pages, fills in the .pages and .sgt fields of the BO, and maps
- * it for DMA.
- */
-static int
-v3d_bo_get_pages(struct v3d_bo *bo)
-{
- struct drm_gem_object *obj = &bo->base;
- struct drm_device *dev = obj->dev;
- int npages = obj->size >> PAGE_SHIFT;
- int ret = 0;
-
- mutex_lock(&bo->lock);
- if (bo->pages_refcount++ != 0)
- goto unlock;
-
- if (!obj->import_attach) {
- bo->pages = drm_gem_get_pages(obj);
- if (IS_ERR(bo->pages)) {
- ret = PTR_ERR(bo->pages);
- goto unlock;
- }
-
- bo->sgt = drm_prime_pages_to_sg(bo->pages, npages);
- if (IS_ERR(bo->sgt)) {
- ret = PTR_ERR(bo->sgt);
- goto put_pages;
- }
-
- /* Map the pages for use by the GPU. */
- dma_map_sg(dev->dev, bo->sgt->sgl,
- bo->sgt->nents, DMA_BIDIRECTIONAL);
- } else {
- bo->pages = kcalloc(npages, sizeof(*bo->pages), GFP_KERNEL);
- if (!bo->pages)
- goto put_pages;
-
- drm_prime_sg_to_page_addr_arrays(bo->sgt, bo->pages,
- NULL, npages);
-
- /* Note that dma-bufs come in mapped. */
- }
-
- mutex_unlock(&bo->lock);
-
- return 0;
-
-put_pages:
- drm_gem_put_pages(obj, bo->pages, true, true);
- bo->pages = NULL;
-unlock:
- bo->pages_refcount--;
- mutex_unlock(&bo->lock);
- return ret;
-}
-
-static void
-v3d_bo_put_pages(struct v3d_bo *bo)
-{
- struct drm_gem_object *obj = &bo->base;
-
- mutex_lock(&bo->lock);
- if (--bo->pages_refcount == 0) {
- if (!obj->import_attach) {
- dma_unmap_sg(obj->dev->dev, bo->sgt->sgl,
- bo->sgt->nents, DMA_BIDIRECTIONAL);
- sg_free_table(bo->sgt);
- kfree(bo->sgt);
- drm_gem_put_pages(obj, bo->pages, true, true);
- } else {
- kfree(bo->pages);
- }
- }
- mutex_unlock(&bo->lock);
-}
-
-static struct v3d_bo *v3d_bo_create_struct(struct drm_device *dev,
- size_t unaligned_size)
-{
- struct v3d_dev *v3d = to_v3d_dev(dev);
- struct drm_gem_object *obj;
- struct v3d_bo *bo;
- size_t size = roundup(unaligned_size, PAGE_SIZE);
- int ret;
-
- if (size == 0)
- return ERR_PTR(-EINVAL);
-
- bo = kzalloc(sizeof(*bo), GFP_KERNEL);
- if (!bo)
- return ERR_PTR(-ENOMEM);
- obj = &bo->base;
-
- INIT_LIST_HEAD(&bo->vmas);
- INIT_LIST_HEAD(&bo->unref_head);
- mutex_init(&bo->lock);
-
- ret = drm_gem_object_init(dev, obj, size);
- if (ret)
- goto free_bo;
-
- spin_lock(&v3d->mm_lock);
- ret = drm_mm_insert_node_generic(&v3d->mm, &bo->node,
- obj->size >> PAGE_SHIFT,
- GMP_GRANULARITY >> PAGE_SHIFT, 0, 0);
- spin_unlock(&v3d->mm_lock);
- if (ret)
- goto free_obj;
-
- return bo;
-
-free_obj:
- drm_gem_object_release(obj);
-free_bo:
- kfree(bo);
- return ERR_PTR(ret);
-}
-
-struct v3d_bo *v3d_bo_create(struct drm_device *dev, struct drm_file *file_priv,
- size_t unaligned_size)
-{
- struct v3d_dev *v3d = to_v3d_dev(dev);
- struct drm_gem_object *obj;
- struct v3d_bo *bo;
- int ret;
-
- bo = v3d_bo_create_struct(dev, unaligned_size);
- if (IS_ERR(bo))
- return bo;
- obj = &bo->base;
-
- bo->resv = &bo->_resv;
- reservation_object_init(bo->resv);
-
- ret = v3d_bo_get_pages(bo);
- if (ret)
- goto free_mm;
-
- v3d_mmu_insert_ptes(bo);
-
- mutex_lock(&v3d->bo_lock);
- v3d->bo_stats.num_allocated++;
- v3d->bo_stats.pages_allocated += obj->size >> PAGE_SHIFT;
- mutex_unlock(&v3d->bo_lock);
-
- return bo;
-
-free_mm:
- spin_lock(&v3d->mm_lock);
- drm_mm_remove_node(&bo->node);
- spin_unlock(&v3d->mm_lock);
-
- drm_gem_object_release(obj);
- kfree(bo);
- return ERR_PTR(ret);
-}
-
/* Called DRM core on the last userspace/kernel unreference of the
* BO.
*/
struct v3d_dev *v3d = to_v3d_dev(obj->dev);
struct v3d_bo *bo = to_v3d_bo(obj);
+ v3d_mmu_remove_ptes(bo);
+
mutex_lock(&v3d->bo_lock);
v3d->bo_stats.num_allocated--;
v3d->bo_stats.pages_allocated -= obj->size >> PAGE_SHIFT;
mutex_unlock(&v3d->bo_lock);
- reservation_object_fini(&bo->_resv);
-
- v3d_bo_put_pages(bo);
-
- if (obj->import_attach)
- drm_prime_gem_destroy(obj, bo->sgt);
-
- v3d_mmu_remove_ptes(bo);
spin_lock(&v3d->mm_lock);
drm_mm_remove_node(&bo->node);
spin_unlock(&v3d->mm_lock);
- mutex_destroy(&bo->lock);
+ /* GPU execution may have dirtied any pages in the BO. */
+ bo->base.pages_mark_dirty_on_put = true;
- drm_gem_object_release(obj);
- kfree(bo);
+ drm_gem_shmem_free_object(obj);
}
-struct reservation_object *v3d_prime_res_obj(struct drm_gem_object *obj)
+static const struct drm_gem_object_funcs v3d_gem_funcs = {
+ .free = v3d_free_object,
+ .print_info = drm_gem_shmem_print_info,
+ .pin = drm_gem_shmem_pin,
+ .unpin = drm_gem_shmem_unpin,
+ .get_sg_table = drm_gem_shmem_get_sg_table,
+ .vmap = drm_gem_shmem_vmap,
+ .vunmap = drm_gem_shmem_vunmap,
+ .vm_ops = &drm_gem_shmem_vm_ops,
+};
+
+/* gem_create_object function for allocating a BO struct and doing
+ * early setup.
+ */
+struct drm_gem_object *v3d_create_object(struct drm_device *dev, size_t size)
{
- struct v3d_bo *bo = to_v3d_bo(obj);
+ struct v3d_bo *bo;
+ struct drm_gem_object *obj;
- return bo->resv;
-}
+ if (size == 0)
+ return NULL;
-static void
-v3d_set_mmap_vma_flags(struct vm_area_struct *vma)
-{
- vma->vm_flags &= ~VM_PFNMAP;
- vma->vm_flags |= VM_MIXEDMAP;
- vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags));
-}
+ bo = kzalloc(sizeof(*bo), GFP_KERNEL);
+ if (!bo)
+ return NULL;
+ obj = &bo->base.base;
-vm_fault_t v3d_gem_fault(struct vm_fault *vmf)
-{
- struct vm_area_struct *vma = vmf->vma;
- struct drm_gem_object *obj = vma->vm_private_data;
- struct v3d_bo *bo = to_v3d_bo(obj);
- pfn_t pfn;
- pgoff_t pgoff;
+ obj->funcs = &v3d_gem_funcs;
- /* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
- pfn = __pfn_to_pfn_t(page_to_pfn(bo->pages[pgoff]), PFN_DEV);
+ INIT_LIST_HEAD(&bo->unref_head);
- return vmf_insert_mixed(vma, vmf->address, pfn);
+ return &bo->base.base;
}
-int v3d_mmap(struct file *filp, struct vm_area_struct *vma)
+static int
+v3d_bo_create_finish(struct drm_gem_object *obj)
{
+ struct v3d_dev *v3d = to_v3d_dev(obj->dev);
+ struct v3d_bo *bo = to_v3d_bo(obj);
+ struct sg_table *sgt;
int ret;
- ret = drm_gem_mmap(filp, vma);
+ /* So far we pin the BO in the MMU for its lifetime, so use
+ * shmem's helper for getting a lifetime sgt.
+ */
+ sgt = drm_gem_shmem_get_pages_sgt(&bo->base.base);
+ if (IS_ERR(sgt))
+ return PTR_ERR(sgt);
+
+ spin_lock(&v3d->mm_lock);
+ /* Allocate the object's space in the GPU's page tables.
+ * Inserting PTEs will happen later, but the offset is for the
+ * lifetime of the BO.
+ */
+ ret = drm_mm_insert_node_generic(&v3d->mm, &bo->node,
+ obj->size >> PAGE_SHIFT,
+ GMP_GRANULARITY >> PAGE_SHIFT, 0, 0);
+ spin_unlock(&v3d->mm_lock);
if (ret)
return ret;
- v3d_set_mmap_vma_flags(vma);
+ /* Track stats for /debug/dri/n/bo_stats. */
+ mutex_lock(&v3d->bo_lock);
+ v3d->bo_stats.num_allocated++;
+ v3d->bo_stats.pages_allocated += obj->size >> PAGE_SHIFT;
+ mutex_unlock(&v3d->bo_lock);
- return ret;
+ v3d_mmu_insert_ptes(bo);
+
+ return 0;
}
-int v3d_prime_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma)
+struct v3d_bo *v3d_bo_create(struct drm_device *dev, struct drm_file *file_priv,
+ size_t unaligned_size)
{
+ struct drm_gem_shmem_object *shmem_obj;
+ struct v3d_bo *bo;
int ret;
- ret = drm_gem_mmap_obj(obj, obj->size, vma);
- if (ret < 0)
- return ret;
-
- v3d_set_mmap_vma_flags(vma);
+ shmem_obj = drm_gem_shmem_create(dev, unaligned_size);
+ if (IS_ERR(shmem_obj))
+ return ERR_CAST(shmem_obj);
+ bo = to_v3d_bo(&shmem_obj->base);
- return 0;
-}
+ ret = v3d_bo_create_finish(&shmem_obj->base);
+ if (ret)
+ goto free_obj;
-struct sg_table *
-v3d_prime_get_sg_table(struct drm_gem_object *obj)
-{
- struct v3d_bo *bo = to_v3d_bo(obj);
- int npages = obj->size >> PAGE_SHIFT;
+ return bo;
- return drm_prime_pages_to_sg(bo->pages, npages);
+free_obj:
+ drm_gem_shmem_free_object(&shmem_obj->base);
+ return ERR_PTR(ret);
}
struct drm_gem_object *
struct sg_table *sgt)
{
struct drm_gem_object *obj;
- struct v3d_bo *bo;
-
- bo = v3d_bo_create_struct(dev, attach->dmabuf->size);
- if (IS_ERR(bo))
- return ERR_CAST(bo);
- obj = &bo->base;
-
- bo->resv = attach->dmabuf->resv;
+ int ret;
- bo->sgt = sgt;
- obj->import_attach = attach;
- v3d_bo_get_pages(bo);
+ obj = drm_gem_shmem_prime_import_sg_table(dev, attach, sgt);
+ if (IS_ERR(obj))
+ return obj;
- v3d_mmu_insert_ptes(bo);
+ ret = v3d_bo_create_finish(obj);
+ if (ret) {
+ drm_gem_shmem_free_object(obj);
+ return ERR_PTR(ret);
+ }
return obj;
}
args->offset = bo->node.start << PAGE_SHIFT;
- ret = drm_gem_handle_create(file_priv, &bo->base, &args->handle);
- drm_gem_object_put_unlocked(&bo->base);
+ ret = drm_gem_handle_create(file_priv, &bo->base.base, &args->handle);
+ drm_gem_object_put_unlocked(&bo->base.base);
return ret;
}
{
struct drm_v3d_mmap_bo *args = data;
struct drm_gem_object *gem_obj;
- int ret;
if (args->flags != 0) {
DRM_INFO("unknown mmap_bo flags: %d\n", args->flags);
return -ENOENT;
}
- ret = drm_gem_create_mmap_offset(gem_obj);
- if (ret == 0)
- args->offset = drm_vma_node_offset_addr(&gem_obj->vma_node);
+ args->offset = drm_vma_node_offset_addr(&gem_obj->vma_node);
drm_gem_object_put_unlocked(gem_obj);
- return ret;
+ return 0;
}
int v3d_get_bo_offset_ioctl(struct drm_device *dev, void *data,
uint32_t cycles;
int core = 0;
int measure_ms = 1000;
+ int ret;
+
+ ret = pm_runtime_get_sync(v3d->dev);
+ if (ret < 0)
+ return ret;
if (v3d->ver >= 40) {
V3D_CORE_WRITE(core, V3D_V4_PCTR_0_SRC_0_3,
cycles / (measure_ms * 1000),
(cycles / (measure_ms * 100)) % 10);
+ pm_runtime_mark_last_busy(v3d->dev);
+ pm_runtime_put_autosuspend(v3d->dev);
+
return 0;
}
* This driver supports the Broadcom V3D 3.3 and 4.1 OpenGL ES GPUs.
* For V3D 2.x support, see the VC4 driver.
*
- * Currently only single-core rendering using the binner and renderer
- * is supported. The TFU (texture formatting unit) and V3D 4.x's CSD
- * (compute shader dispatch) are not yet supported.
+ * Currently only single-core rendering using the binner and renderer,
+ * along with TFU (texture formatting unit) rendering is supported.
+ * V3D 4.x's CSD (compute shader dispatch) is not yet supported.
*/
#include <linux/clk.h>
#include <linux/of_platform.h>
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
+#include <linux/reset.h>
#include <drm/drm_fb_cma_helper.h>
#include <drm/drm_fb_helper.h>
return -EINVAL;
ret = pm_runtime_get_sync(v3d->dev);
+ if (ret < 0)
+ return ret;
if (args->param >= DRM_V3D_PARAM_V3D_CORE0_IDENT0 &&
args->param <= DRM_V3D_PARAM_V3D_CORE0_IDENT2) {
args->value = V3D_CORE_READ(0, offset);
kfree(v3d_priv);
}
-static const struct file_operations v3d_drm_fops = {
- .owner = THIS_MODULE,
- .open = drm_open,
- .release = drm_release,
- .unlocked_ioctl = drm_ioctl,
- .mmap = v3d_mmap,
- .poll = drm_poll,
- .read = drm_read,
- .compat_ioctl = drm_compat_ioctl,
- .llseek = noop_llseek,
-};
+DEFINE_DRM_GEM_SHMEM_FOPS(v3d_drm_fops);
/* DRM_AUTH is required on SUBMIT_CL for now, while we don't have GMP
* protection between clients. Note that render nodes would be be
DRM_IOCTL_DEF_DRV(V3D_SUBMIT_TFU, v3d_submit_tfu_ioctl, DRM_RENDER_ALLOW | DRM_AUTH),
};
-static const struct vm_operations_struct v3d_vm_ops = {
- .fault = v3d_gem_fault,
- .open = drm_gem_vm_open,
- .close = drm_gem_vm_close,
-};
-
static struct drm_driver v3d_drm_driver = {
.driver_features = (DRIVER_GEM |
DRIVER_RENDER |
.debugfs_init = v3d_debugfs_init,
#endif
- .gem_free_object_unlocked = v3d_free_object,
- .gem_vm_ops = &v3d_vm_ops,
-
+ .gem_create_object = v3d_create_object,
.prime_handle_to_fd = drm_gem_prime_handle_to_fd,
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
- .gem_prime_import = drm_gem_prime_import,
- .gem_prime_export = drm_gem_prime_export,
- .gem_prime_res_obj = v3d_prime_res_obj,
- .gem_prime_get_sg_table = v3d_prime_get_sg_table,
.gem_prime_import_sg_table = v3d_prime_import_sg_table,
- .gem_prime_mmap = v3d_prime_mmap,
+ .gem_prime_mmap = drm_gem_prime_mmap,
.ioctls = v3d_drm_ioctls,
.num_ioctls = ARRAY_SIZE(v3d_drm_ioctls),
v3d->pdev = pdev;
drm = &v3d->drm;
- ret = map_regs(v3d, &v3d->bridge_regs, "bridge");
- if (ret)
- goto dev_free;
-
ret = map_regs(v3d, &v3d->hub_regs, "hub");
if (ret)
goto dev_free;
v3d->cores = V3D_GET_FIELD(ident1, V3D_HUB_IDENT1_NCORES);
WARN_ON(v3d->cores > 1); /* multicore not yet implemented */
+ v3d->reset = devm_reset_control_get_exclusive(dev, NULL);
+ if (IS_ERR(v3d->reset)) {
+ ret = PTR_ERR(v3d->reset);
+
+ if (ret == -EPROBE_DEFER)
+ goto dev_free;
+
+ v3d->reset = NULL;
+ ret = map_regs(v3d, &v3d->bridge_regs, "bridge");
+ if (ret) {
+ dev_err(dev,
+ "Failed to get reset control or bridge regs\n");
+ goto dev_free;
+ }
+ }
+
if (v3d->ver < 41) {
ret = map_regs(v3d, &v3d->gca_regs, "gca");
if (ret)
if (ret)
goto dev_destroy;
- v3d_irq_init(v3d);
+ ret = v3d_irq_init(v3d);
+ if (ret)
+ goto gem_destroy;
ret = drm_dev_register(drm, 0);
if (ret)
- goto gem_destroy;
+ goto irq_disable;
return 0;
+irq_disable:
+ v3d_irq_disable(v3d);
gem_destroy:
v3d_gem_destroy(drm);
dev_destroy:
// SPDX-License-Identifier: GPL-2.0+
/* Copyright (C) 2015-2018 Broadcom */
-#include <linux/reservation.h>
#include <linux/mm_types.h>
#include <drm/drmP.h>
#include <drm/drm_encoder.h>
#include <drm/drm_gem.h>
+#include <drm/drm_gem_shmem_helper.h>
#include <drm/gpu_scheduler.h>
#include "uapi/drm/v3d_drm.h"
* and revision.
*/
int ver;
+ bool single_irq_line;
struct device *dev;
struct platform_device *pdev;
void __iomem *bridge_regs;
void __iomem *gca_regs;
struct clk *clk;
+ struct reset_control *reset;
/* Virtual and DMA addresses of the single shared page table. */
volatile u32 *pt;
struct drm_sched_entity sched_entity[V3D_MAX_QUEUES];
};
-/* Tracks a mapping of a BO into a per-fd address space */
-struct v3d_vma {
- struct v3d_page_table *pt;
- struct list_head list; /* entry in v3d_bo.vmas */
-};
-
struct v3d_bo {
- struct drm_gem_object base;
-
- struct mutex lock;
+ struct drm_gem_shmem_object base;
struct drm_mm_node node;
- u32 pages_refcount;
- struct page **pages;
- struct sg_table *sgt;
- void *vaddr;
-
- struct list_head vmas; /* list of v3d_vma */
-
/* List entry for the BO's position in
* v3d_exec_info->unref_list
*/
struct list_head unref_head;
-
- /* normally (resv == &_resv) except for imported bo's */
- struct reservation_object *resv;
- struct reservation_object _resv;
};
static inline struct v3d_bo *
struct dma_fence *in_fence;
/* v3d fence to be signaled by IRQ handler when the job is complete. */
- struct dma_fence *done_fence;
+ struct dma_fence *irq_fence;
/* GPU virtual addresses of the start/end of the CL job. */
u32 start, end;
struct dma_fence *in_fence;
/* v3d fence to be signaled by IRQ handler when the job is complete. */
- struct dma_fence *done_fence;
+ struct dma_fence *irq_fence;
struct v3d_dev *v3d;
}
/* v3d_bo.c */
+struct drm_gem_object *v3d_create_object(struct drm_device *dev, size_t size);
void v3d_free_object(struct drm_gem_object *gem_obj);
struct v3d_bo *v3d_bo_create(struct drm_device *dev, struct drm_file *file_priv,
size_t size);
struct drm_file *file_priv);
int v3d_get_bo_offset_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_priv);
-vm_fault_t v3d_gem_fault(struct vm_fault *vmf);
-int v3d_mmap(struct file *filp, struct vm_area_struct *vma);
-struct reservation_object *v3d_prime_res_obj(struct drm_gem_object *obj);
-int v3d_prime_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma);
-struct sg_table *v3d_prime_get_sg_table(struct drm_gem_object *obj);
struct drm_gem_object *v3d_prime_import_sg_table(struct drm_device *dev,
struct dma_buf_attachment *attach,
struct sg_table *sgt);
void v3d_invalidate_caches(struct v3d_dev *v3d);
/* v3d_irq.c */
-void v3d_irq_init(struct v3d_dev *v3d);
+int v3d_irq_init(struct v3d_dev *v3d);
void v3d_irq_enable(struct v3d_dev *v3d);
void v3d_irq_disable(struct v3d_dev *v3d);
void v3d_irq_reset(struct v3d_dev *v3d);
#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
+#include <linux/reset.h>
#include <linux/device.h>
#include <linux/io.h>
#include <linux/sched/signal.h>
* type. If you want the default behavior, you can still put
* "2" in the indirect texture state's output_type field.
*/
- V3D_CORE_WRITE(core, V3D_CTL_MISCCFG, V3D_MISCCFG_OVRTMUOUT);
+ if (v3d->ver < 40)
+ V3D_CORE_WRITE(core, V3D_CTL_MISCCFG, V3D_MISCCFG_OVRTMUOUT);
/* Whenever we flush the L2T cache, we always want to flush
* the whole thing.
}
static void
-v3d_reset_v3d(struct v3d_dev *v3d)
+v3d_reset_by_bridge(struct v3d_dev *v3d)
{
int version = V3D_BRIDGE_READ(V3D_TOP_GR_BRIDGE_REVISION);
V3D_TOP_GR_BRIDGE_SW_INIT_1_V3D_CLK_108_SW_INIT);
V3D_BRIDGE_WRITE(V3D_TOP_GR_BRIDGE_SW_INIT_1, 0);
}
+}
+
+static void
+v3d_reset_v3d(struct v3d_dev *v3d)
+{
+ if (v3d->reset)
+ reset_control_reset(v3d->reset);
+ else
+ v3d_reset_by_bridge(v3d);
v3d_init_hw_state(v3d);
}
for (i = 0; i < bo_count; i++) {
/* XXX: Use shared fences for read-only objects. */
- reservation_object_add_excl_fence(bos[i]->resv, fence);
+ reservation_object_add_excl_fence(bos[i]->base.base.resv,
+ fence);
}
}
int bo_count,
struct ww_acquire_ctx *acquire_ctx)
{
- int i;
-
- for (i = 0; i < bo_count; i++)
- ww_mutex_unlock(&bos[i]->resv->lock);
-
- ww_acquire_fini(acquire_ctx);
+ drm_gem_unlock_reservations((struct drm_gem_object **)bos, bo_count,
+ acquire_ctx);
}
/* Takes the reservation lock on all the BOs being referenced, so that
int bo_count,
struct ww_acquire_ctx *acquire_ctx)
{
- int contended_lock = -1;
int i, ret;
- ww_acquire_init(acquire_ctx, &reservation_ww_class);
-
-retry:
- if (contended_lock != -1) {
- struct v3d_bo *bo = bos[contended_lock];
-
- ret = ww_mutex_lock_slow_interruptible(&bo->resv->lock,
- acquire_ctx);
- if (ret) {
- ww_acquire_done(acquire_ctx);
- return ret;
- }
- }
-
- for (i = 0; i < bo_count; i++) {
- if (i == contended_lock)
- continue;
-
- ret = ww_mutex_lock_interruptible(&bos[i]->resv->lock,
- acquire_ctx);
- if (ret) {
- int j;
-
- for (j = 0; j < i; j++)
- ww_mutex_unlock(&bos[j]->resv->lock);
-
- if (contended_lock != -1 && contended_lock >= i) {
- struct v3d_bo *bo = bos[contended_lock];
-
- ww_mutex_unlock(&bo->resv->lock);
- }
-
- if (ret == -EDEADLK) {
- contended_lock = i;
- goto retry;
- }
-
- ww_acquire_done(acquire_ctx);
- return ret;
- }
- }
-
- ww_acquire_done(acquire_ctx);
+ ret = drm_gem_lock_reservations((struct drm_gem_object **)bos,
+ bo_count, acquire_ctx);
+ if (ret)
+ return ret;
/* Reserve space for our shared (read-only) fence references,
* before we commit the CL to the hardware.
*/
for (i = 0; i < bo_count; i++) {
- ret = reservation_object_reserve_shared(bos[i]->resv, 1);
+ ret = reservation_object_reserve_shared(bos[i]->base.base.resv,
+ 1);
if (ret) {
v3d_unlock_bo_reservations(bos, bo_count,
acquire_ctx);
dma_fence_put(exec->bin.in_fence);
dma_fence_put(exec->render.in_fence);
- dma_fence_put(exec->bin.done_fence);
- dma_fence_put(exec->render.done_fence);
+ dma_fence_put(exec->bin.irq_fence);
+ dma_fence_put(exec->render.irq_fence);
dma_fence_put(exec->bin_done_fence);
dma_fence_put(exec->render_done_fence);
for (i = 0; i < exec->bo_count; i++)
- drm_gem_object_put_unlocked(&exec->bo[i]->base);
+ drm_gem_object_put_unlocked(&exec->bo[i]->base.base);
kvfree(exec->bo);
list_for_each_entry_safe(bo, save, &exec->unref_list, unref_head) {
- drm_gem_object_put_unlocked(&bo->base);
+ drm_gem_object_put_unlocked(&bo->base.base);
}
pm_runtime_mark_last_busy(v3d->dev);
unsigned int i;
dma_fence_put(job->in_fence);
- dma_fence_put(job->done_fence);
+ dma_fence_put(job->irq_fence);
for (i = 0; i < ARRAY_SIZE(job->bo); i++) {
if (job->bo[i])
- drm_gem_object_put_unlocked(&job->bo[i]->base);
+ drm_gem_object_put_unlocked(&job->bo[i]->base.base);
}
pm_runtime_mark_last_busy(v3d->dev);
{
int ret;
struct drm_v3d_wait_bo *args = data;
- struct drm_gem_object *gem_obj;
- struct v3d_bo *bo;
ktime_t start = ktime_get();
u64 delta_ns;
unsigned long timeout_jiffies =
if (args->pad != 0)
return -EINVAL;
- gem_obj = drm_gem_object_lookup(file_priv, args->handle);
- if (!gem_obj) {
- DRM_DEBUG("Failed to look up GEM BO %d\n", args->handle);
- return -EINVAL;
- }
- bo = to_v3d_bo(gem_obj);
-
- ret = reservation_object_wait_timeout_rcu(bo->resv,
- true, true,
- timeout_jiffies);
-
- if (ret == 0)
- ret = -ETIME;
- else if (ret > 0)
- ret = 0;
+ ret = drm_gem_reservation_object_wait(file_priv, args->handle,
+ true, timeout_jiffies);
/* Decrement the user's timeout, in case we got interrupted
* such that the ioctl will be restarted.
if (ret == -ETIME && args->timeout_ns)
ret = -EAGAIN;
- drm_gem_object_put_unlocked(gem_obj);
-
return ret;
}
V3D_HUB_INT_MMU_CAP | \
V3D_HUB_INT_TFUC))
+static irqreturn_t
+v3d_hub_irq(int irq, void *arg);
+
static void
v3d_overflow_mem_work(struct work_struct *work)
{
container_of(work, struct v3d_dev, overflow_mem_work);
struct drm_device *dev = &v3d->drm;
struct v3d_bo *bo = v3d_bo_create(dev, NULL /* XXX: GMP */, 256 * 1024);
+ struct drm_gem_object *obj;
unsigned long irqflags;
if (IS_ERR(bo)) {
DRM_ERROR("Couldn't allocate binner overflow mem\n");
return;
}
+ obj = &bo->base.base;
/* We lost a race, and our work task came in after the bin job
* completed and exited. This can happen because the HW
goto out;
}
- drm_gem_object_get(&bo->base);
+ drm_gem_object_get(obj);
list_add_tail(&bo->unref_head, &v3d->bin_job->unref_list);
spin_unlock_irqrestore(&v3d->job_lock, irqflags);
V3D_CORE_WRITE(0, V3D_PTB_BPOA, bo->node.start << PAGE_SHIFT);
- V3D_CORE_WRITE(0, V3D_PTB_BPOS, bo->base.size);
+ V3D_CORE_WRITE(0, V3D_PTB_BPOS, obj->size);
out:
- drm_gem_object_put_unlocked(&bo->base);
+ drm_gem_object_put_unlocked(obj);
}
static irqreturn_t
if (intsts & V3D_INT_OUTOMEM) {
/* Note that the OOM status is edge signaled, so the
* interrupt won't happen again until the we actually
- * add more memory.
+ * add more memory. Also, as of V3D 4.1, FLDONE won't
+ * be reported until any OOM state has been cleared.
*/
schedule_work(&v3d->overflow_mem_work);
status = IRQ_HANDLED;
if (intsts & V3D_INT_FLDONE) {
struct v3d_fence *fence =
- to_v3d_fence(v3d->bin_job->bin.done_fence);
+ to_v3d_fence(v3d->bin_job->bin.irq_fence);
trace_v3d_bcl_irq(&v3d->drm, fence->seqno);
dma_fence_signal(&fence->base);
if (intsts & V3D_INT_FRDONE) {
struct v3d_fence *fence =
- to_v3d_fence(v3d->render_job->render.done_fence);
+ to_v3d_fence(v3d->render_job->render.irq_fence);
trace_v3d_rcl_irq(&v3d->drm, fence->seqno);
dma_fence_signal(&fence->base);
if (intsts & V3D_INT_GMPV)
dev_err(v3d->dev, "GMP violation\n");
+ /* V3D 4.2 wires the hub and core IRQs together, so if we &
+ * didn't see the common one then check hub for MMU IRQs.
+ */
+ if (v3d->single_irq_line && status == IRQ_NONE)
+ return v3d_hub_irq(irq, arg);
+
return status;
}
if (intsts & V3D_HUB_INT_TFUC) {
struct v3d_fence *fence =
- to_v3d_fence(v3d->tfu_job->done_fence);
+ to_v3d_fence(v3d->tfu_job->irq_fence);
trace_v3d_tfu_irq(&v3d->drm, fence->seqno);
dma_fence_signal(&fence->base);
return status;
}
-void
+int
v3d_irq_init(struct v3d_dev *v3d)
{
- int ret, core;
+ int irq1, ret, core;
INIT_WORK(&v3d->overflow_mem_work, v3d_overflow_mem_work);
V3D_CORE_WRITE(core, V3D_CTL_INT_CLR, V3D_CORE_IRQS);
V3D_WRITE(V3D_HUB_INT_CLR, V3D_HUB_IRQS);
- ret = devm_request_irq(v3d->dev, platform_get_irq(v3d->pdev, 0),
- v3d_hub_irq, IRQF_SHARED,
- "v3d_hub", v3d);
- ret = devm_request_irq(v3d->dev, platform_get_irq(v3d->pdev, 1),
- v3d_irq, IRQF_SHARED,
- "v3d_core0", v3d);
- if (ret)
- dev_err(v3d->dev, "IRQ setup failed: %d\n", ret);
+ irq1 = platform_get_irq(v3d->pdev, 1);
+ if (irq1 == -EPROBE_DEFER)
+ return irq1;
+ if (irq1 > 0) {
+ ret = devm_request_irq(v3d->dev, irq1,
+ v3d_irq, IRQF_SHARED,
+ "v3d_core0", v3d);
+ if (ret)
+ goto fail;
+ ret = devm_request_irq(v3d->dev, platform_get_irq(v3d->pdev, 0),
+ v3d_hub_irq, IRQF_SHARED,
+ "v3d_hub", v3d);
+ if (ret)
+ goto fail;
+ } else {
+ v3d->single_irq_line = true;
+
+ ret = devm_request_irq(v3d->dev, platform_get_irq(v3d->pdev, 0),
+ v3d_irq, IRQF_SHARED,
+ "v3d", v3d);
+ if (ret)
+ goto fail;
+ }
v3d_irq_enable(v3d);
+ return 0;
+
+fail:
+ if (ret != -EPROBE_DEFER)
+ dev_err(v3d->dev, "IRQ setup failed: %d\n", ret);
+ return ret;
}
void
void v3d_mmu_insert_ptes(struct v3d_bo *bo)
{
- struct v3d_dev *v3d = to_v3d_dev(bo->base.dev);
+ struct drm_gem_shmem_object *shmem_obj = &bo->base;
+ struct v3d_dev *v3d = to_v3d_dev(shmem_obj->base.dev);
u32 page = bo->node.start;
u32 page_prot = V3D_PTE_WRITEABLE | V3D_PTE_VALID;
unsigned int count;
struct scatterlist *sgl;
- for_each_sg(bo->sgt->sgl, sgl, bo->sgt->nents, count) {
+ for_each_sg(shmem_obj->sgt->sgl, sgl, shmem_obj->sgt->nents, count) {
u32 page_address = sg_dma_address(sgl) >> V3D_MMU_PAGE_SHIFT;
u32 pte = page_prot | page_address;
u32 i;
}
WARN_ON_ONCE(page - bo->node.start !=
- bo->base.size >> V3D_MMU_PAGE_SHIFT);
+ shmem_obj->base.size >> V3D_MMU_PAGE_SHIFT);
if (v3d_mmu_flush_all(v3d))
dev_err(v3d->dev, "MMU flush timeout\n");
void v3d_mmu_remove_ptes(struct v3d_bo *bo)
{
- struct v3d_dev *v3d = to_v3d_dev(bo->base.dev);
- u32 npages = bo->base.size >> V3D_MMU_PAGE_SHIFT;
+ struct v3d_dev *v3d = to_v3d_dev(bo->base.base.dev);
+ u32 npages = bo->base.base.size >> V3D_MMU_PAGE_SHIFT;
u32 page;
for (page = bo->node.start; page < bo->node.start + npages; page++)
# define V3D_IDENT2_BCG_INT BIT(28)
#define V3D_CTL_MISCCFG 0x00018
+# define V3D_CTL_MISCCFG_QRMAXCNT_MASK V3D_MASK(3, 1)
+# define V3D_CTL_MISCCFG_QRMAXCNT_SHIFT 1
# define V3D_MISCCFG_OVRTMUOUT BIT(0)
#define V3D_CTL_L2CACTL 0x00020
if (IS_ERR(fence))
return NULL;
- if (job->done_fence)
- dma_fence_put(job->done_fence);
- job->done_fence = dma_fence_get(fence);
+ if (job->irq_fence)
+ dma_fence_put(job->irq_fence);
+ job->irq_fence = dma_fence_get(fence);
trace_v3d_submit_cl(dev, q == V3D_RENDER, to_v3d_fence(fence)->seqno,
job->start, job->end);
return NULL;
v3d->tfu_job = job;
- if (job->done_fence)
- dma_fence_put(job->done_fence);
- job->done_fence = dma_fence_get(fence);
+ if (job->irq_fence)
+ dma_fence_put(job->irq_fence);
+ job->irq_fence = dma_fence_get(fence);
trace_v3d_submit_tfu(dev, to_v3d_fence(fence)->seqno);
mutex_lock(&v3d->reset_lock);
/* block scheduler */
- for (q = 0; q < V3D_MAX_QUEUES; q++) {
- struct drm_gpu_scheduler *sched = &v3d->queue[q].sched;
-
- drm_sched_stop(sched);
+ for (q = 0; q < V3D_MAX_QUEUES; q++)
+ drm_sched_stop(&v3d->queue[q].sched);
- if(sched_job)
- drm_sched_increase_karma(sched_job);
- }
+ if (sched_job)
+ drm_sched_increase_karma(sched_job);
/* get the GPU back into the init state */
v3d_reset(v3d);
for (q = 0; q < V3D_MAX_QUEUES; q++)
- drm_sched_resubmit_jobs(sched_job->sched);
+ drm_sched_resubmit_jobs(&v3d->queue[q].sched);
/* Unblock schedulers and restart their jobs. */
for (q = 0; q < V3D_MAX_QUEUES; q++) {
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+config DRM_VBOXVIDEO
+ tristate "Virtual Box Graphics Card"
+ depends on DRM && X86 && PCI
+ select DRM_KMS_HELPER
+ select DRM_TTM
+ select GENERIC_ALLOCATOR
+ help
+ This is a KMS driver for the virtual Graphics Card used in
+ Virtual Box virtual machines.
+
+ Although it is possible to build this driver built-in to the
+ kernel, it is advised to build it as a module, so that it can
+ be updated independently of the kernel. Select M to build this
+ driver as a module and add support for these devices via drm/kms
+ interfaces.
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+vboxvideo-y := hgsmi_base.o modesetting.o vbva_base.o \
+ vbox_drv.o vbox_fb.o vbox_hgsmi.o vbox_irq.o vbox_main.o \
+ vbox_mode.o vbox_prime.o vbox_ttm.o
+
+obj-$(CONFIG_DRM_VBOXVIDEO) += vboxvideo.o
--- /dev/null
+// SPDX-License-Identifier: MIT
+/* Copyright (C) 2006-2017 Oracle Corporation */
+
+#include <linux/vbox_err.h>
+#include "vbox_drv.h"
+#include "vboxvideo_guest.h"
+#include "vboxvideo_vbe.h"
+#include "hgsmi_channels.h"
+#include "hgsmi_ch_setup.h"
+
+/**
+ * Inform the host of the location of the host flags in VRAM via an HGSMI cmd.
+ * Return: 0 or negative errno value.
+ * @ctx: The context of the guest heap to use.
+ * @location: The offset chosen for the flags within guest VRAM.
+ */
+int hgsmi_report_flags_location(struct gen_pool *ctx, u32 location)
+{
+ struct hgsmi_buffer_location *p;
+
+ p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_HGSMI,
+ HGSMI_CC_HOST_FLAGS_LOCATION);
+ if (!p)
+ return -ENOMEM;
+
+ p->buf_location = location;
+ p->buf_len = sizeof(struct hgsmi_host_flags);
+
+ hgsmi_buffer_submit(ctx, p);
+ hgsmi_buffer_free(ctx, p);
+
+ return 0;
+}
+
+/**
+ * Notify the host of HGSMI-related guest capabilities via an HGSMI command.
+ * Return: 0 or negative errno value.
+ * @ctx: The context of the guest heap to use.
+ * @caps: The capabilities to report, see vbva_caps.
+ */
+int hgsmi_send_caps_info(struct gen_pool *ctx, u32 caps)
+{
+ struct vbva_caps *p;
+
+ p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA, VBVA_INFO_CAPS);
+ if (!p)
+ return -ENOMEM;
+
+ p->rc = VERR_NOT_IMPLEMENTED;
+ p->caps = caps;
+
+ hgsmi_buffer_submit(ctx, p);
+
+ WARN_ON_ONCE(p->rc < 0);
+
+ hgsmi_buffer_free(ctx, p);
+
+ return 0;
+}
+
+int hgsmi_test_query_conf(struct gen_pool *ctx)
+{
+ u32 value = 0;
+ int ret;
+
+ ret = hgsmi_query_conf(ctx, U32_MAX, &value);
+ if (ret)
+ return ret;
+
+ return value == U32_MAX ? 0 : -EIO;
+}
+
+/**
+ * Query the host for an HGSMI configuration parameter via an HGSMI command.
+ * Return: 0 or negative errno value.
+ * @ctx: The context containing the heap used.
+ * @index: The index of the parameter to query.
+ * @value_ret: Where to store the value of the parameter on success.
+ */
+int hgsmi_query_conf(struct gen_pool *ctx, u32 index, u32 *value_ret)
+{
+ struct vbva_conf32 *p;
+
+ p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA,
+ VBVA_QUERY_CONF32);
+ if (!p)
+ return -ENOMEM;
+
+ p->index = index;
+ p->value = U32_MAX;
+
+ hgsmi_buffer_submit(ctx, p);
+
+ *value_ret = p->value;
+
+ hgsmi_buffer_free(ctx, p);
+
+ return 0;
+}
+
+/**
+ * Pass the host a new mouse pointer shape via an HGSMI command.
+ * Return: 0 or negative errno value.
+ * @ctx: The context containing the heap to be used.
+ * @flags: Cursor flags.
+ * @hot_x: Horizontal position of the hot spot.
+ * @hot_y: Vertical position of the hot spot.
+ * @width: Width in pixels of the cursor.
+ * @height: Height in pixels of the cursor.
+ * @pixels: Pixel data, @see VMMDevReqMousePointer for the format.
+ * @len: Size in bytes of the pixel data.
+ */
+int hgsmi_update_pointer_shape(struct gen_pool *ctx, u32 flags,
+ u32 hot_x, u32 hot_y, u32 width, u32 height,
+ u8 *pixels, u32 len)
+{
+ struct vbva_mouse_pointer_shape *p;
+ u32 pixel_len = 0;
+ int rc;
+
+ if (flags & VBOX_MOUSE_POINTER_SHAPE) {
+ /*
+ * Size of the pointer data:
+ * sizeof (AND mask) + sizeof (XOR_MASK)
+ */
+ pixel_len = ((((width + 7) / 8) * height + 3) & ~3) +
+ width * 4 * height;
+ if (pixel_len > len)
+ return -EINVAL;
+
+ /*
+ * If shape is supplied, then always create the pointer visible.
+ * See comments in 'vboxUpdatePointerShape'
+ */
+ flags |= VBOX_MOUSE_POINTER_VISIBLE;
+ }
+
+ p = hgsmi_buffer_alloc(ctx, sizeof(*p) + pixel_len, HGSMI_CH_VBVA,
+ VBVA_MOUSE_POINTER_SHAPE);
+ if (!p)
+ return -ENOMEM;
+
+ p->result = VINF_SUCCESS;
+ p->flags = flags;
+ p->hot_X = hot_x;
+ p->hot_y = hot_y;
+ p->width = width;
+ p->height = height;
+ if (pixel_len)
+ memcpy(p->data, pixels, pixel_len);
+
+ hgsmi_buffer_submit(ctx, p);
+
+ switch (p->result) {
+ case VINF_SUCCESS:
+ rc = 0;
+ break;
+ case VERR_NO_MEMORY:
+ rc = -ENOMEM;
+ break;
+ case VERR_NOT_SUPPORTED:
+ rc = -EBUSY;
+ break;
+ default:
+ rc = -EINVAL;
+ }
+
+ hgsmi_buffer_free(ctx, p);
+
+ return rc;
+}
+
+/**
+ * Report the guest cursor position. The host may wish to use this information
+ * to re-position its own cursor (though this is currently unlikely). The
+ * current host cursor position is returned.
+ * Return: 0 or negative errno value.
+ * @ctx: The context containing the heap used.
+ * @report_position: Are we reporting a position?
+ * @x: Guest cursor X position.
+ * @y: Guest cursor Y position.
+ * @x_host: Host cursor X position is stored here. Optional.
+ * @y_host: Host cursor Y position is stored here. Optional.
+ */
+int hgsmi_cursor_position(struct gen_pool *ctx, bool report_position,
+ u32 x, u32 y, u32 *x_host, u32 *y_host)
+{
+ struct vbva_cursor_position *p;
+
+ p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA,
+ VBVA_CURSOR_POSITION);
+ if (!p)
+ return -ENOMEM;
+
+ p->report_position = report_position;
+ p->x = x;
+ p->y = y;
+
+ hgsmi_buffer_submit(ctx, p);
+
+ *x_host = p->x;
+ *y_host = p->y;
+
+ hgsmi_buffer_free(ctx, p);
+
+ return 0;
+}
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/* Copyright (C) 2006-2017 Oracle Corporation */
+
+#ifndef __HGSMI_CH_SETUP_H__
+#define __HGSMI_CH_SETUP_H__
+
+/*
+ * Tell the host the location of hgsmi_host_flags structure, where the host
+ * can write information about pending buffers, etc, and which can be quickly
+ * polled by the guest without a need to port IO.
+ */
+#define HGSMI_CC_HOST_FLAGS_LOCATION 0
+
+struct hgsmi_buffer_location {
+ u32 buf_location;
+ u32 buf_len;
+} __packed;
+
+/* HGSMI setup and configuration data structures. */
+
+#define HGSMIHOSTFLAGS_COMMANDS_PENDING 0x01u
+#define HGSMIHOSTFLAGS_IRQ 0x02u
+#define HGSMIHOSTFLAGS_VSYNC 0x10u
+#define HGSMIHOSTFLAGS_HOTPLUG 0x20u
+#define HGSMIHOSTFLAGS_CURSOR_CAPABILITIES 0x40u
+
+struct hgsmi_host_flags {
+ u32 host_flags;
+ u32 reserved[3];
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/* Copyright (C) 2006-2017 Oracle Corporation */
+
+#ifndef __HGSMI_CHANNELS_H__
+#define __HGSMI_CHANNELS_H__
+
+/*
+ * Each channel has an 8 bit identifier. There are a number of predefined
+ * (hardcoded) channels.
+ *
+ * HGSMI_CH_HGSMI channel can be used to map a string channel identifier
+ * to a free 16 bit numerical value. values are allocated in range
+ * [HGSMI_CH_STRING_FIRST;HGSMI_CH_STRING_LAST].
+ */
+
+/* A reserved channel value */
+#define HGSMI_CH_RESERVED 0x00
+/* HGCMI: setup and configuration */
+#define HGSMI_CH_HGSMI 0x01
+/* Graphics: VBVA */
+#define HGSMI_CH_VBVA 0x02
+/* Graphics: Seamless with a single guest region */
+#define HGSMI_CH_SEAMLESS 0x03
+/* Graphics: Seamless with separate host windows */
+#define HGSMI_CH_SEAMLESS2 0x04
+/* Graphics: OpenGL HW acceleration */
+#define HGSMI_CH_OPENGL 0x05
+
+/* The first channel index to be used for string mappings (inclusive) */
+#define HGSMI_CH_STRING_FIRST 0x20
+/* The last channel index for string mappings (inclusive) */
+#define HGSMI_CH_STRING_LAST 0xff
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/* Copyright (C) 2006-2017 Oracle Corporation */
+
+#ifndef __HGSMI_DEFS_H__
+#define __HGSMI_DEFS_H__
+
+/* Buffer sequence type mask. */
+#define HGSMI_BUFFER_HEADER_F_SEQ_MASK 0x03
+/* Single buffer, not a part of a sequence. */
+#define HGSMI_BUFFER_HEADER_F_SEQ_SINGLE 0x00
+/* The first buffer in a sequence. */
+#define HGSMI_BUFFER_HEADER_F_SEQ_START 0x01
+/* A middle buffer in a sequence. */
+#define HGSMI_BUFFER_HEADER_F_SEQ_CONTINUE 0x02
+/* The last buffer in a sequence. */
+#define HGSMI_BUFFER_HEADER_F_SEQ_END 0x03
+
+/* 16 bytes buffer header. */
+struct hgsmi_buffer_header {
+ u32 data_size; /* Size of data that follows the header. */
+ u8 flags; /* HGSMI_BUFFER_HEADER_F_* */
+ u8 channel; /* The channel the data must be routed to. */
+ u16 channel_info; /* Opaque to the HGSMI, used by the channel. */
+
+ union {
+ /* Opaque placeholder to make the union 8 bytes. */
+ u8 header_data[8];
+
+ /* HGSMI_BUFFER_HEADER_F_SEQ_SINGLE */
+ struct {
+ u32 reserved1; /* A reserved field, initialize to 0. */
+ u32 reserved2; /* A reserved field, initialize to 0. */
+ } buffer;
+
+ /* HGSMI_BUFFER_HEADER_F_SEQ_START */
+ struct {
+ /* Must be the same for all buffers in the sequence. */
+ u32 sequence_number;
+ /* The total size of the sequence. */
+ u32 sequence_size;
+ } sequence_start;
+
+ /*
+ * HGSMI_BUFFER_HEADER_F_SEQ_CONTINUE and
+ * HGSMI_BUFFER_HEADER_F_SEQ_END
+ */
+ struct {
+ /* Must be the same for all buffers in the sequence. */
+ u32 sequence_number;
+ /* Data offset in the entire sequence. */
+ u32 sequence_offset;
+ } sequence_continue;
+ } u;
+} __packed;
+
+/* 8 bytes buffer tail. */
+struct hgsmi_buffer_tail {
+ /* Reserved, must be initialized to 0. */
+ u32 reserved;
+ /*
+ * One-at-a-Time Hash: http://www.burtleburtle.net/bob/hash/doobs.html
+ * Over the header, offset and for first 4 bytes of the tail.
+ */
+ u32 checksum;
+} __packed;
+
+/*
+ * The size of the array of channels. Array indexes are u8.
+ * Note: the value must not be changed.
+ */
+#define HGSMI_NUMBER_OF_CHANNELS 0x100
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: MIT
+/* Copyright (C) 2006-2017 Oracle Corporation */
+
+#include <linux/vbox_err.h>
+#include "vbox_drv.h"
+#include "vboxvideo_guest.h"
+#include "vboxvideo_vbe.h"
+#include "hgsmi_channels.h"
+
+/**
+ * Set a video mode via an HGSMI request. The views must have been
+ * initialised first using @a VBoxHGSMISendViewInfo and if the mode is being
+ * set on the first display then it must be set first using registers.
+ * @ctx: The context containing the heap to use.
+ * @display: The screen number.
+ * @origin_x: The horizontal displacement relative to the first scrn.
+ * @origin_y: The vertical displacement relative to the first screen.
+ * @start_offset: The offset of the visible area of the framebuffer
+ * relative to the framebuffer start.
+ * @pitch: The offset in bytes between the starts of two adjecent
+ * scan lines in video RAM.
+ * @width: The mode width.
+ * @height: The mode height.
+ * @bpp: The colour depth of the mode.
+ * @flags: Flags.
+ */
+void hgsmi_process_display_info(struct gen_pool *ctx, u32 display,
+ s32 origin_x, s32 origin_y, u32 start_offset,
+ u32 pitch, u32 width, u32 height,
+ u16 bpp, u16 flags)
+{
+ struct vbva_infoscreen *p;
+
+ p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA,
+ VBVA_INFO_SCREEN);
+ if (!p)
+ return;
+
+ p->view_index = display;
+ p->origin_x = origin_x;
+ p->origin_y = origin_y;
+ p->start_offset = start_offset;
+ p->line_size = pitch;
+ p->width = width;
+ p->height = height;
+ p->bits_per_pixel = bpp;
+ p->flags = flags;
+
+ hgsmi_buffer_submit(ctx, p);
+ hgsmi_buffer_free(ctx, p);
+}
+
+/**
+ * Report the rectangle relative to which absolute pointer events should be
+ * expressed. This information remains valid until the next VBVA resize event
+ * for any screen, at which time it is reset to the bounding rectangle of all
+ * virtual screens.
+ * Return: 0 or negative errno value.
+ * @ctx: The context containing the heap to use.
+ * @origin_x: Upper left X co-ordinate relative to the first screen.
+ * @origin_y: Upper left Y co-ordinate relative to the first screen.
+ * @width: Rectangle width.
+ * @height: Rectangle height.
+ */
+int hgsmi_update_input_mapping(struct gen_pool *ctx, s32 origin_x, s32 origin_y,
+ u32 width, u32 height)
+{
+ struct vbva_report_input_mapping *p;
+
+ p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA,
+ VBVA_REPORT_INPUT_MAPPING);
+ if (!p)
+ return -ENOMEM;
+
+ p->x = origin_x;
+ p->y = origin_y;
+ p->cx = width;
+ p->cy = height;
+
+ hgsmi_buffer_submit(ctx, p);
+ hgsmi_buffer_free(ctx, p);
+
+ return 0;
+}
+
+/**
+ * Get most recent video mode hints.
+ * Return: 0 or negative errno value.
+ * @ctx: The context containing the heap to use.
+ * @screens: The number of screens to query hints for, starting at 0.
+ * @hints: Array of vbva_modehint structures for receiving the hints.
+ */
+int hgsmi_get_mode_hints(struct gen_pool *ctx, unsigned int screens,
+ struct vbva_modehint *hints)
+{
+ struct vbva_query_mode_hints *p;
+ size_t size;
+
+ if (WARN_ON(!hints))
+ return -EINVAL;
+
+ size = screens * sizeof(struct vbva_modehint);
+ p = hgsmi_buffer_alloc(ctx, sizeof(*p) + size, HGSMI_CH_VBVA,
+ VBVA_QUERY_MODE_HINTS);
+ if (!p)
+ return -ENOMEM;
+
+ p->hints_queried_count = screens;
+ p->hint_structure_guest_size = sizeof(struct vbva_modehint);
+ p->rc = VERR_NOT_SUPPORTED;
+
+ hgsmi_buffer_submit(ctx, p);
+
+ if (p->rc < 0) {
+ hgsmi_buffer_free(ctx, p);
+ return -EIO;
+ }
+
+ memcpy(hints, ((u8 *)p) + sizeof(struct vbva_query_mode_hints), size);
+ hgsmi_buffer_free(ctx, p);
+
+ return 0;
+}
--- /dev/null
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright (C) 2013-2017 Oracle Corporation
+ * This file is based on ast_drv.c
+ * Copyright 2012 Red Hat Inc.
+ * Authors: Dave Airlie <airlied@redhat.com>
+ * Michael Thayer <michael.thayer@oracle.com,
+ * Hans de Goede <hdegoede@redhat.com>
+ */
+#include <linux/console.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/vt_kern.h>
+
+#include <drm/drm_crtc_helper.h>
+#include <drm/drm_drv.h>
+#include <drm/drm_file.h>
+#include <drm/drm_ioctl.h>
+
+#include "vbox_drv.h"
+
+static int vbox_modeset = -1;
+
+MODULE_PARM_DESC(modeset, "Disable/Enable modesetting");
+module_param_named(modeset, vbox_modeset, int, 0400);
+
+static struct drm_driver driver;
+
+static const struct pci_device_id pciidlist[] = {
+ { PCI_DEVICE(0x80ee, 0xbeef) },
+ { }
+};
+MODULE_DEVICE_TABLE(pci, pciidlist);
+
+static struct drm_fb_helper_funcs vbox_fb_helper_funcs = {
+ .fb_probe = vboxfb_create,
+};
+
+static int vbox_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
+{
+ struct vbox_private *vbox;
+ int ret = 0;
+
+ if (!vbox_check_supported(VBE_DISPI_ID_HGSMI))
+ return -ENODEV;
+
+ vbox = kzalloc(sizeof(*vbox), GFP_KERNEL);
+ if (!vbox)
+ return -ENOMEM;
+
+ ret = drm_dev_init(&vbox->ddev, &driver, &pdev->dev);
+ if (ret) {
+ kfree(vbox);
+ return ret;
+ }
+
+ vbox->ddev.pdev = pdev;
+ vbox->ddev.dev_private = vbox;
+ pci_set_drvdata(pdev, vbox);
+ mutex_init(&vbox->hw_mutex);
+
+ ret = pci_enable_device(pdev);
+ if (ret)
+ goto err_dev_put;
+
+ ret = vbox_hw_init(vbox);
+ if (ret)
+ goto err_pci_disable;
+
+ ret = vbox_mm_init(vbox);
+ if (ret)
+ goto err_hw_fini;
+
+ ret = vbox_mode_init(vbox);
+ if (ret)
+ goto err_mm_fini;
+
+ ret = vbox_irq_init(vbox);
+ if (ret)
+ goto err_mode_fini;
+
+ ret = drm_fb_helper_fbdev_setup(&vbox->ddev, &vbox->fb_helper,
+ &vbox_fb_helper_funcs, 32,
+ vbox->num_crtcs);
+ if (ret)
+ goto err_irq_fini;
+
+ ret = drm_dev_register(&vbox->ddev, 0);
+ if (ret)
+ goto err_fbdev_fini;
+
+ return 0;
+
+err_fbdev_fini:
+ vbox_fbdev_fini(vbox);
+err_irq_fini:
+ vbox_irq_fini(vbox);
+err_mode_fini:
+ vbox_mode_fini(vbox);
+err_mm_fini:
+ vbox_mm_fini(vbox);
+err_hw_fini:
+ vbox_hw_fini(vbox);
+err_pci_disable:
+ pci_disable_device(pdev);
+err_dev_put:
+ drm_dev_put(&vbox->ddev);
+ return ret;
+}
+
+static void vbox_pci_remove(struct pci_dev *pdev)
+{
+ struct vbox_private *vbox = pci_get_drvdata(pdev);
+
+ drm_dev_unregister(&vbox->ddev);
+ vbox_fbdev_fini(vbox);
+ vbox_irq_fini(vbox);
+ vbox_mode_fini(vbox);
+ vbox_mm_fini(vbox);
+ vbox_hw_fini(vbox);
+ drm_dev_put(&vbox->ddev);
+}
+
+#ifdef CONFIG_PM_SLEEP
+static int vbox_pm_suspend(struct device *dev)
+{
+ struct vbox_private *vbox = dev_get_drvdata(dev);
+ int error;
+
+ error = drm_mode_config_helper_suspend(&vbox->ddev);
+ if (error)
+ return error;
+
+ pci_save_state(vbox->ddev.pdev);
+ pci_disable_device(vbox->ddev.pdev);
+ pci_set_power_state(vbox->ddev.pdev, PCI_D3hot);
+
+ return 0;
+}
+
+static int vbox_pm_resume(struct device *dev)
+{
+ struct vbox_private *vbox = dev_get_drvdata(dev);
+
+ if (pci_enable_device(vbox->ddev.pdev))
+ return -EIO;
+
+ return drm_mode_config_helper_resume(&vbox->ddev);
+}
+
+static int vbox_pm_freeze(struct device *dev)
+{
+ struct vbox_private *vbox = dev_get_drvdata(dev);
+
+ return drm_mode_config_helper_suspend(&vbox->ddev);
+}
+
+static int vbox_pm_thaw(struct device *dev)
+{
+ struct vbox_private *vbox = dev_get_drvdata(dev);
+
+ return drm_mode_config_helper_resume(&vbox->ddev);
+}
+
+static int vbox_pm_poweroff(struct device *dev)
+{
+ struct vbox_private *vbox = dev_get_drvdata(dev);
+
+ return drm_mode_config_helper_suspend(&vbox->ddev);
+}
+
+static const struct dev_pm_ops vbox_pm_ops = {
+ .suspend = vbox_pm_suspend,
+ .resume = vbox_pm_resume,
+ .freeze = vbox_pm_freeze,
+ .thaw = vbox_pm_thaw,
+ .poweroff = vbox_pm_poweroff,
+ .restore = vbox_pm_resume,
+};
+#endif
+
+static struct pci_driver vbox_pci_driver = {
+ .name = DRIVER_NAME,
+ .id_table = pciidlist,
+ .probe = vbox_pci_probe,
+ .remove = vbox_pci_remove,
+#ifdef CONFIG_PM_SLEEP
+ .driver.pm = &vbox_pm_ops,
+#endif
+};
+
+static const struct file_operations vbox_fops = {
+ .owner = THIS_MODULE,
+ .open = drm_open,
+ .release = drm_release,
+ .unlocked_ioctl = drm_ioctl,
+ .compat_ioctl = drm_compat_ioctl,
+ .mmap = vbox_mmap,
+ .poll = drm_poll,
+ .read = drm_read,
+};
+
+static struct drm_driver driver = {
+ .driver_features =
+ DRIVER_MODESET | DRIVER_GEM | DRIVER_PRIME | DRIVER_ATOMIC,
+
+ .lastclose = drm_fb_helper_lastclose,
+
+ .fops = &vbox_fops,
+ .irq_handler = vbox_irq_handler,
+ .name = DRIVER_NAME,
+ .desc = DRIVER_DESC,
+ .date = DRIVER_DATE,
+ .major = DRIVER_MAJOR,
+ .minor = DRIVER_MINOR,
+ .patchlevel = DRIVER_PATCHLEVEL,
+
+ .gem_free_object_unlocked = vbox_gem_free_object,
+ .dumb_create = vbox_dumb_create,
+ .dumb_map_offset = vbox_dumb_mmap_offset,
+ .prime_handle_to_fd = drm_gem_prime_handle_to_fd,
+ .prime_fd_to_handle = drm_gem_prime_fd_to_handle,
+ .gem_prime_export = drm_gem_prime_export,
+ .gem_prime_import = drm_gem_prime_import,
+ .gem_prime_pin = vbox_gem_prime_pin,
+ .gem_prime_unpin = vbox_gem_prime_unpin,
+ .gem_prime_get_sg_table = vbox_gem_prime_get_sg_table,
+ .gem_prime_import_sg_table = vbox_gem_prime_import_sg_table,
+ .gem_prime_vmap = vbox_gem_prime_vmap,
+ .gem_prime_vunmap = vbox_gem_prime_vunmap,
+ .gem_prime_mmap = vbox_gem_prime_mmap,
+};
+
+static int __init vbox_init(void)
+{
+#ifdef CONFIG_VGA_CONSOLE
+ if (vgacon_text_force() && vbox_modeset == -1)
+ return -EINVAL;
+#endif
+
+ if (vbox_modeset == 0)
+ return -EINVAL;
+
+ return pci_register_driver(&vbox_pci_driver);
+}
+
+static void __exit vbox_exit(void)
+{
+ pci_unregister_driver(&vbox_pci_driver);
+}
+
+module_init(vbox_init);
+module_exit(vbox_exit);
+
+MODULE_AUTHOR("Oracle Corporation");
+MODULE_AUTHOR("Hans de Goede <hdegoede@redhat.com>");
+MODULE_DESCRIPTION(DRIVER_DESC);
+MODULE_LICENSE("GPL and additional rights");
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright (C) 2013-2017 Oracle Corporation
+ * This file is based on ast_drv.h
+ * Copyright 2012 Red Hat Inc.
+ * Authors: Dave Airlie <airlied@redhat.com>
+ * Michael Thayer <michael.thayer@oracle.com,
+ * Hans de Goede <hdegoede@redhat.com>
+ */
+#ifndef __VBOX_DRV_H__
+#define __VBOX_DRV_H__
+
+#include <linux/genalloc.h>
+#include <linux/io.h>
+#include <linux/irqreturn.h>
+#include <linux/string.h>
+
+#include <drm/drm_encoder.h>
+#include <drm/drm_fb_helper.h>
+#include <drm/drm_gem.h>
+
+#include <drm/ttm/ttm_bo_api.h>
+#include <drm/ttm/ttm_bo_driver.h>
+#include <drm/ttm/ttm_placement.h>
+#include <drm/ttm/ttm_memory.h>
+#include <drm/ttm/ttm_module.h>
+
+#include "vboxvideo_guest.h"
+#include "vboxvideo_vbe.h"
+#include "hgsmi_ch_setup.h"
+
+#define DRIVER_NAME "vboxvideo"
+#define DRIVER_DESC "Oracle VM VirtualBox Graphics Card"
+#define DRIVER_DATE "20130823"
+
+#define DRIVER_MAJOR 1
+#define DRIVER_MINOR 0
+#define DRIVER_PATCHLEVEL 0
+
+#define VBOX_MAX_CURSOR_WIDTH 64
+#define VBOX_MAX_CURSOR_HEIGHT 64
+#define CURSOR_PIXEL_COUNT (VBOX_MAX_CURSOR_WIDTH * VBOX_MAX_CURSOR_HEIGHT)
+#define CURSOR_DATA_SIZE (CURSOR_PIXEL_COUNT * 4 + CURSOR_PIXEL_COUNT / 8)
+
+#define VBOX_MAX_SCREENS 32
+
+#define GUEST_HEAP_OFFSET(vbox) ((vbox)->full_vram_size - \
+ VBVA_ADAPTER_INFORMATION_SIZE)
+#define GUEST_HEAP_SIZE VBVA_ADAPTER_INFORMATION_SIZE
+#define GUEST_HEAP_USABLE_SIZE (VBVA_ADAPTER_INFORMATION_SIZE - \
+ sizeof(struct hgsmi_host_flags))
+#define HOST_FLAGS_OFFSET GUEST_HEAP_USABLE_SIZE
+
+struct vbox_framebuffer {
+ struct drm_framebuffer base;
+ struct drm_gem_object *obj;
+};
+
+struct vbox_private {
+ /* Must be first; or we must define our own release callback */
+ struct drm_device ddev;
+ struct drm_fb_helper fb_helper;
+ struct vbox_framebuffer afb;
+
+ u8 __iomem *guest_heap;
+ u8 __iomem *vbva_buffers;
+ struct gen_pool *guest_pool;
+ struct vbva_buf_ctx *vbva_info;
+ bool any_pitch;
+ u32 num_crtcs;
+ /* Amount of available VRAM, including space used for buffers. */
+ u32 full_vram_size;
+ /* Amount of available VRAM, not including space used for buffers. */
+ u32 available_vram_size;
+ /* Array of structures for receiving mode hints. */
+ struct vbva_modehint *last_mode_hints;
+
+ int fb_mtrr;
+
+ struct {
+ struct ttm_bo_device bdev;
+ } ttm;
+
+ struct mutex hw_mutex; /* protects modeset and accel/vbva accesses */
+ struct work_struct hotplug_work;
+ u32 input_mapping_width;
+ u32 input_mapping_height;
+ /*
+ * Is user-space using an X.Org-style layout of one large frame-buffer
+ * encompassing all screen ones or is the fbdev console active?
+ */
+ bool single_framebuffer;
+ u8 cursor_data[CURSOR_DATA_SIZE];
+};
+
+#undef CURSOR_PIXEL_COUNT
+#undef CURSOR_DATA_SIZE
+
+struct vbox_gem_object;
+
+struct vbox_connector {
+ struct drm_connector base;
+ char name[32];
+ struct vbox_crtc *vbox_crtc;
+ struct {
+ u32 width;
+ u32 height;
+ bool disconnected;
+ } mode_hint;
+};
+
+struct vbox_crtc {
+ struct drm_crtc base;
+ bool disconnected;
+ unsigned int crtc_id;
+ u32 fb_offset;
+ bool cursor_enabled;
+ u32 x_hint;
+ u32 y_hint;
+ /*
+ * When setting a mode we not only pass the mode to the hypervisor,
+ * but also information on how to map / translate input coordinates
+ * for the emulated USB tablet. This input-mapping may change when
+ * the mode on *another* crtc changes.
+ *
+ * This means that sometimes we must do a modeset on other crtc-s then
+ * the one being changed to update the input-mapping. Including crtc-s
+ * which may be disabled inside the guest (shown as a black window
+ * on the host unless closed by the user).
+ *
+ * With atomic modesetting the mode-info of disabled crtcs gets zeroed
+ * yet we need it when updating the input-map to avoid resizing the
+ * window as a side effect of a mode_set on another crtc. Therefor we
+ * cache the info of the last mode below.
+ */
+ u32 width;
+ u32 height;
+ u32 x;
+ u32 y;
+};
+
+struct vbox_encoder {
+ struct drm_encoder base;
+};
+
+#define to_vbox_crtc(x) container_of(x, struct vbox_crtc, base)
+#define to_vbox_connector(x) container_of(x, struct vbox_connector, base)
+#define to_vbox_encoder(x) container_of(x, struct vbox_encoder, base)
+#define to_vbox_framebuffer(x) container_of(x, struct vbox_framebuffer, base)
+
+bool vbox_check_supported(u16 id);
+int vbox_hw_init(struct vbox_private *vbox);
+void vbox_hw_fini(struct vbox_private *vbox);
+
+int vbox_mode_init(struct vbox_private *vbox);
+void vbox_mode_fini(struct vbox_private *vbox);
+
+void vbox_report_caps(struct vbox_private *vbox);
+
+void vbox_framebuffer_dirty_rectangles(struct drm_framebuffer *fb,
+ struct drm_clip_rect *rects,
+ unsigned int num_rects);
+
+int vbox_framebuffer_init(struct vbox_private *vbox,
+ struct vbox_framebuffer *vbox_fb,
+ const struct drm_mode_fb_cmd2 *mode_cmd,
+ struct drm_gem_object *obj);
+
+int vboxfb_create(struct drm_fb_helper *helper,
+ struct drm_fb_helper_surface_size *sizes);
+void vbox_fbdev_fini(struct vbox_private *vbox);
+
+struct vbox_bo {
+ struct ttm_buffer_object bo;
+ struct ttm_placement placement;
+ struct ttm_bo_kmap_obj kmap;
+ struct drm_gem_object gem;
+ struct ttm_place placements[3];
+ int pin_count;
+};
+
+#define gem_to_vbox_bo(gobj) container_of((gobj), struct vbox_bo, gem)
+
+static inline struct vbox_bo *vbox_bo(struct ttm_buffer_object *bo)
+{
+ return container_of(bo, struct vbox_bo, bo);
+}
+
+#define to_vbox_obj(x) container_of(x, struct vbox_gem_object, base)
+
+static inline u64 vbox_bo_gpu_offset(struct vbox_bo *bo)
+{
+ return bo->bo.offset;
+}
+
+int vbox_dumb_create(struct drm_file *file,
+ struct drm_device *dev,
+ struct drm_mode_create_dumb *args);
+
+void vbox_gem_free_object(struct drm_gem_object *obj);
+int vbox_dumb_mmap_offset(struct drm_file *file,
+ struct drm_device *dev,
+ u32 handle, u64 *offset);
+
+int vbox_mm_init(struct vbox_private *vbox);
+void vbox_mm_fini(struct vbox_private *vbox);
+
+int vbox_bo_create(struct vbox_private *vbox, int size, int align,
+ u32 flags, struct vbox_bo **pvboxbo);
+
+int vbox_gem_create(struct vbox_private *vbox,
+ u32 size, bool iskernel, struct drm_gem_object **obj);
+
+int vbox_bo_pin(struct vbox_bo *bo, u32 pl_flag);
+int vbox_bo_unpin(struct vbox_bo *bo);
+
+static inline int vbox_bo_reserve(struct vbox_bo *bo, bool no_wait)
+{
+ int ret;
+
+ ret = ttm_bo_reserve(&bo->bo, true, no_wait, NULL);
+ if (ret) {
+ if (ret != -ERESTARTSYS && ret != -EBUSY)
+ DRM_ERROR("reserve failed %p\n", bo);
+ return ret;
+ }
+ return 0;
+}
+
+static inline void vbox_bo_unreserve(struct vbox_bo *bo)
+{
+ ttm_bo_unreserve(&bo->bo);
+}
+
+void vbox_ttm_placement(struct vbox_bo *bo, int domain);
+int vbox_bo_push_sysram(struct vbox_bo *bo);
+int vbox_mmap(struct file *filp, struct vm_area_struct *vma);
+void *vbox_bo_kmap(struct vbox_bo *bo);
+void vbox_bo_kunmap(struct vbox_bo *bo);
+
+/* vbox_prime.c */
+int vbox_gem_prime_pin(struct drm_gem_object *obj);
+void vbox_gem_prime_unpin(struct drm_gem_object *obj);
+struct sg_table *vbox_gem_prime_get_sg_table(struct drm_gem_object *obj);
+struct drm_gem_object *vbox_gem_prime_import_sg_table(
+ struct drm_device *dev, struct dma_buf_attachment *attach,
+ struct sg_table *table);
+void *vbox_gem_prime_vmap(struct drm_gem_object *obj);
+void vbox_gem_prime_vunmap(struct drm_gem_object *obj, void *vaddr);
+int vbox_gem_prime_mmap(struct drm_gem_object *obj,
+ struct vm_area_struct *area);
+
+/* vbox_irq.c */
+int vbox_irq_init(struct vbox_private *vbox);
+void vbox_irq_fini(struct vbox_private *vbox);
+void vbox_report_hotplug(struct vbox_private *vbox);
+irqreturn_t vbox_irq_handler(int irq, void *arg);
+
+/* vbox_hgsmi.c */
+void *hgsmi_buffer_alloc(struct gen_pool *guest_pool, size_t size,
+ u8 channel, u16 channel_info);
+void hgsmi_buffer_free(struct gen_pool *guest_pool, void *buf);
+int hgsmi_buffer_submit(struct gen_pool *guest_pool, void *buf);
+
+static inline void vbox_write_ioport(u16 index, u16 data)
+{
+ outw(index, VBE_DISPI_IOPORT_INDEX);
+ outw(data, VBE_DISPI_IOPORT_DATA);
+}
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright (C) 2013-2017 Oracle Corporation
+ * This file is based on ast_fb.c
+ * Copyright 2012 Red Hat Inc.
+ * Authors: Dave Airlie <airlied@redhat.com>
+ * Michael Thayer <michael.thayer@oracle.com,
+ */
+#include <linux/delay.h>
+#include <linux/errno.h>
+#include <linux/fb.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/string.h>
+#include <linux/sysrq.h>
+#include <linux/tty.h>
+
+#include <drm/drm_crtc.h>
+#include <drm/drm_crtc_helper.h>
+#include <drm/drm_fb_helper.h>
+#include <drm/drm_fourcc.h>
+
+#include "vbox_drv.h"
+#include "vboxvideo.h"
+
+#ifdef CONFIG_DRM_KMS_FB_HELPER
+static struct fb_deferred_io vbox_defio = {
+ .delay = HZ / 30,
+ .deferred_io = drm_fb_helper_deferred_io,
+};
+#endif
+
+static struct fb_ops vboxfb_ops = {
+ .owner = THIS_MODULE,
+ DRM_FB_HELPER_DEFAULT_OPS,
+ .fb_fillrect = drm_fb_helper_sys_fillrect,
+ .fb_copyarea = drm_fb_helper_sys_copyarea,
+ .fb_imageblit = drm_fb_helper_sys_imageblit,
+};
+
+int vboxfb_create(struct drm_fb_helper *helper,
+ struct drm_fb_helper_surface_size *sizes)
+{
+ struct vbox_private *vbox =
+ container_of(helper, struct vbox_private, fb_helper);
+ struct pci_dev *pdev = vbox->ddev.pdev;
+ struct drm_mode_fb_cmd2 mode_cmd;
+ struct drm_framebuffer *fb;
+ struct fb_info *info;
+ struct drm_gem_object *gobj;
+ struct vbox_bo *bo;
+ int size, ret;
+ u64 gpu_addr;
+ u32 pitch;
+
+ mode_cmd.width = sizes->surface_width;
+ mode_cmd.height = sizes->surface_height;
+ pitch = mode_cmd.width * ((sizes->surface_bpp + 7) / 8);
+ mode_cmd.pixel_format = drm_mode_legacy_fb_format(sizes->surface_bpp,
+ sizes->surface_depth);
+ mode_cmd.pitches[0] = pitch;
+
+ size = pitch * mode_cmd.height;
+
+ ret = vbox_gem_create(vbox, size, true, &gobj);
+ if (ret) {
+ DRM_ERROR("failed to create fbcon backing object %d\n", ret);
+ return ret;
+ }
+
+ ret = vbox_framebuffer_init(vbox, &vbox->afb, &mode_cmd, gobj);
+ if (ret)
+ return ret;
+
+ bo = gem_to_vbox_bo(gobj);
+
+ ret = vbox_bo_pin(bo, TTM_PL_FLAG_VRAM);
+ if (ret)
+ return ret;
+
+ info = drm_fb_helper_alloc_fbi(helper);
+ if (IS_ERR(info))
+ return PTR_ERR(info);
+
+ info->screen_size = size;
+ info->screen_base = (char __iomem *)vbox_bo_kmap(bo);
+ if (IS_ERR(info->screen_base))
+ return PTR_ERR(info->screen_base);
+
+ fb = &vbox->afb.base;
+ helper->fb = fb;
+
+ info->fbops = &vboxfb_ops;
+
+ /*
+ * This seems to be done for safety checking that the framebuffer
+ * is not registered twice by different drivers.
+ */
+ info->apertures->ranges[0].base = pci_resource_start(pdev, 0);
+ info->apertures->ranges[0].size = pci_resource_len(pdev, 0);
+
+ drm_fb_helper_fill_info(info, helper, sizes);
+
+ gpu_addr = vbox_bo_gpu_offset(bo);
+ info->fix.smem_start = info->apertures->ranges[0].base + gpu_addr;
+ info->fix.smem_len = vbox->available_vram_size - gpu_addr;
+
+#ifdef CONFIG_DRM_KMS_FB_HELPER
+ info->fbdefio = &vbox_defio;
+ fb_deferred_io_init(info);
+#endif
+
+ info->pixmap.flags = FB_PIXMAP_SYSTEM;
+
+ DRM_DEBUG_KMS("allocated %dx%d\n", fb->width, fb->height);
+
+ return 0;
+}
+
+void vbox_fbdev_fini(struct vbox_private *vbox)
+{
+ struct vbox_framebuffer *afb = &vbox->afb;
+
+#ifdef CONFIG_DRM_KMS_FB_HELPER
+ if (vbox->fb_helper.fbdev && vbox->fb_helper.fbdev->fbdefio)
+ fb_deferred_io_cleanup(vbox->fb_helper.fbdev);
+#endif
+
+ drm_fb_helper_unregister_fbi(&vbox->fb_helper);
+
+ if (afb->obj) {
+ struct vbox_bo *bo = gem_to_vbox_bo(afb->obj);
+
+ vbox_bo_kunmap(bo);
+
+ if (bo->pin_count)
+ vbox_bo_unpin(bo);
+
+ drm_gem_object_put_unlocked(afb->obj);
+ afb->obj = NULL;
+ }
+ drm_fb_helper_fini(&vbox->fb_helper);
+
+ drm_framebuffer_unregister_private(&afb->base);
+ drm_framebuffer_cleanup(&afb->base);
+}
--- /dev/null
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright (C) 2017 Oracle Corporation
+ * Authors: Hans de Goede <hdegoede@redhat.com>
+ */
+
+#include "vbox_drv.h"
+#include "vboxvideo_vbe.h"
+#include "hgsmi_defs.h"
+
+/* One-at-a-Time Hash from http://www.burtleburtle.net/bob/hash/doobs.html */
+static u32 hgsmi_hash_process(u32 hash, const u8 *data, int size)
+{
+ while (size--) {
+ hash += *data++;
+ hash += (hash << 10);
+ hash ^= (hash >> 6);
+ }
+
+ return hash;
+}
+
+static u32 hgsmi_hash_end(u32 hash)
+{
+ hash += (hash << 3);
+ hash ^= (hash >> 11);
+ hash += (hash << 15);
+
+ return hash;
+}
+
+/* Not really a checksum but that is the naming used in all vbox code */
+static u32 hgsmi_checksum(u32 offset,
+ const struct hgsmi_buffer_header *header,
+ const struct hgsmi_buffer_tail *tail)
+{
+ u32 checksum;
+
+ checksum = hgsmi_hash_process(0, (u8 *)&offset, sizeof(offset));
+ checksum = hgsmi_hash_process(checksum, (u8 *)header, sizeof(*header));
+ /* 4 -> Do not checksum the checksum itself */
+ checksum = hgsmi_hash_process(checksum, (u8 *)tail, 4);
+
+ return hgsmi_hash_end(checksum);
+}
+
+void *hgsmi_buffer_alloc(struct gen_pool *guest_pool, size_t size,
+ u8 channel, u16 channel_info)
+{
+ struct hgsmi_buffer_header *h;
+ struct hgsmi_buffer_tail *t;
+ size_t total_size;
+ dma_addr_t offset;
+
+ total_size = size + sizeof(*h) + sizeof(*t);
+ h = gen_pool_dma_alloc(guest_pool, total_size, &offset);
+ if (!h)
+ return NULL;
+
+ t = (struct hgsmi_buffer_tail *)((u8 *)h + sizeof(*h) + size);
+
+ h->flags = HGSMI_BUFFER_HEADER_F_SEQ_SINGLE;
+ h->data_size = size;
+ h->channel = channel;
+ h->channel_info = channel_info;
+ memset(&h->u.header_data, 0, sizeof(h->u.header_data));
+
+ t->reserved = 0;
+ t->checksum = hgsmi_checksum(offset, h, t);
+
+ return (u8 *)h + sizeof(*h);
+}
+
+void hgsmi_buffer_free(struct gen_pool *guest_pool, void *buf)
+{
+ struct hgsmi_buffer_header *h =
+ (struct hgsmi_buffer_header *)((u8 *)buf - sizeof(*h));
+ size_t total_size = h->data_size + sizeof(*h) +
+ sizeof(struct hgsmi_buffer_tail);
+
+ gen_pool_free(guest_pool, (unsigned long)h, total_size);
+}
+
+int hgsmi_buffer_submit(struct gen_pool *guest_pool, void *buf)
+{
+ phys_addr_t offset;
+
+ offset = gen_pool_virt_to_phys(guest_pool, (unsigned long)buf -
+ sizeof(struct hgsmi_buffer_header));
+ outl(offset, VGA_PORT_HGSMI_GUEST);
+ /* Make the compiler aware that the host has changed memory. */
+ mb();
+
+ return 0;
+}
--- /dev/null
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright (C) 2016-2017 Oracle Corporation
+ * This file is based on qxl_irq.c
+ * Copyright 2013 Red Hat Inc.
+ * Authors: Dave Airlie
+ * Alon Levy
+ * Michael Thayer <michael.thayer@oracle.com,
+ * Hans de Goede <hdegoede@redhat.com>
+ */
+
+#include <linux/pci.h>
+#include <drm/drm_irq.h>
+#include <drm/drm_probe_helper.h>
+
+#include "vbox_drv.h"
+#include "vboxvideo.h"
+
+static void vbox_clear_irq(void)
+{
+ outl((u32)~0, VGA_PORT_HGSMI_HOST);
+}
+
+static u32 vbox_get_flags(struct vbox_private *vbox)
+{
+ return readl(vbox->guest_heap + HOST_FLAGS_OFFSET);
+}
+
+void vbox_report_hotplug(struct vbox_private *vbox)
+{
+ schedule_work(&vbox->hotplug_work);
+}
+
+irqreturn_t vbox_irq_handler(int irq, void *arg)
+{
+ struct drm_device *dev = (struct drm_device *)arg;
+ struct vbox_private *vbox = (struct vbox_private *)dev->dev_private;
+ u32 host_flags = vbox_get_flags(vbox);
+
+ if (!(host_flags & HGSMIHOSTFLAGS_IRQ))
+ return IRQ_NONE;
+
+ /*
+ * Due to a bug in the initial host implementation of hot-plug irqs,
+ * the hot-plug and cursor capability flags were never cleared.
+ * Fortunately we can tell when they would have been set by checking
+ * that the VSYNC flag is not set.
+ */
+ if (host_flags &
+ (HGSMIHOSTFLAGS_HOTPLUG | HGSMIHOSTFLAGS_CURSOR_CAPABILITIES) &&
+ !(host_flags & HGSMIHOSTFLAGS_VSYNC))
+ vbox_report_hotplug(vbox);
+
+ vbox_clear_irq();
+
+ return IRQ_HANDLED;
+}
+
+/*
+ * Check that the position hints provided by the host are suitable for GNOME
+ * shell (i.e. all screens disjoint and hints for all enabled screens) and if
+ * not replace them with default ones. Providing valid hints improves the
+ * chances that we will get a known screen layout for pointer mapping.
+ */
+static void validate_or_set_position_hints(struct vbox_private *vbox)
+{
+ struct vbva_modehint *hintsi, *hintsj;
+ bool valid = true;
+ u16 currentx = 0;
+ int i, j;
+
+ for (i = 0; i < vbox->num_crtcs; ++i) {
+ for (j = 0; j < i; ++j) {
+ hintsi = &vbox->last_mode_hints[i];
+ hintsj = &vbox->last_mode_hints[j];
+
+ if (hintsi->enabled && hintsj->enabled) {
+ if (hintsi->dx >= 0xffff ||
+ hintsi->dy >= 0xffff ||
+ hintsj->dx >= 0xffff ||
+ hintsj->dy >= 0xffff ||
+ (hintsi->dx <
+ hintsj->dx + (hintsj->cx & 0x8fff) &&
+ hintsi->dx + (hintsi->cx & 0x8fff) >
+ hintsj->dx) ||
+ (hintsi->dy <
+ hintsj->dy + (hintsj->cy & 0x8fff) &&
+ hintsi->dy + (hintsi->cy & 0x8fff) >
+ hintsj->dy))
+ valid = false;
+ }
+ }
+ }
+ if (!valid)
+ for (i = 0; i < vbox->num_crtcs; ++i) {
+ if (vbox->last_mode_hints[i].enabled) {
+ vbox->last_mode_hints[i].dx = currentx;
+ vbox->last_mode_hints[i].dy = 0;
+ currentx +=
+ vbox->last_mode_hints[i].cx & 0x8fff;
+ }
+ }
+}
+
+/* Query the host for the most recent video mode hints. */
+static void vbox_update_mode_hints(struct vbox_private *vbox)
+{
+ struct drm_connector_list_iter conn_iter;
+ struct drm_device *dev = &vbox->ddev;
+ struct drm_connector *connector;
+ struct vbox_connector *vbox_conn;
+ struct vbva_modehint *hints;
+ u16 flags;
+ bool disconnected;
+ unsigned int crtc_id;
+ int ret;
+
+ ret = hgsmi_get_mode_hints(vbox->guest_pool, vbox->num_crtcs,
+ vbox->last_mode_hints);
+ if (ret) {
+ DRM_ERROR("vboxvideo: hgsmi_get_mode_hints failed: %d\n", ret);
+ return;
+ }
+
+ validate_or_set_position_hints(vbox);
+
+ drm_modeset_lock(&dev->mode_config.connection_mutex, NULL);
+ drm_connector_list_iter_begin(dev, &conn_iter);
+ drm_for_each_connector_iter(connector, &conn_iter) {
+ vbox_conn = to_vbox_connector(connector);
+
+ hints = &vbox->last_mode_hints[vbox_conn->vbox_crtc->crtc_id];
+ if (hints->magic != VBVAMODEHINT_MAGIC)
+ continue;
+
+ disconnected = !(hints->enabled);
+ crtc_id = vbox_conn->vbox_crtc->crtc_id;
+ vbox_conn->mode_hint.width = hints->cx;
+ vbox_conn->mode_hint.height = hints->cy;
+ vbox_conn->vbox_crtc->x_hint = hints->dx;
+ vbox_conn->vbox_crtc->y_hint = hints->dy;
+ vbox_conn->mode_hint.disconnected = disconnected;
+
+ if (vbox_conn->vbox_crtc->disconnected == disconnected)
+ continue;
+
+ if (disconnected)
+ flags = VBVA_SCREEN_F_ACTIVE | VBVA_SCREEN_F_DISABLED;
+ else
+ flags = VBVA_SCREEN_F_ACTIVE | VBVA_SCREEN_F_BLANK;
+
+ hgsmi_process_display_info(vbox->guest_pool, crtc_id, 0, 0, 0,
+ hints->cx * 4, hints->cx,
+ hints->cy, 0, flags);
+
+ vbox_conn->vbox_crtc->disconnected = disconnected;
+ }
+ drm_connector_list_iter_end(&conn_iter);
+ drm_modeset_unlock(&dev->mode_config.connection_mutex);
+}
+
+static void vbox_hotplug_worker(struct work_struct *work)
+{
+ struct vbox_private *vbox = container_of(work, struct vbox_private,
+ hotplug_work);
+
+ vbox_update_mode_hints(vbox);
+ drm_kms_helper_hotplug_event(&vbox->ddev);
+}
+
+int vbox_irq_init(struct vbox_private *vbox)
+{
+ INIT_WORK(&vbox->hotplug_work, vbox_hotplug_worker);
+ vbox_update_mode_hints(vbox);
+
+ return drm_irq_install(&vbox->ddev, vbox->ddev.pdev->irq);
+}
+
+void vbox_irq_fini(struct vbox_private *vbox)
+{
+ drm_irq_uninstall(&vbox->ddev);
+ flush_work(&vbox->hotplug_work);
+}
--- /dev/null
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright (C) 2013-2017 Oracle Corporation
+ * This file is based on ast_main.c
+ * Copyright 2012 Red Hat Inc.
+ * Authors: Dave Airlie <airlied@redhat.com>,
+ * Michael Thayer <michael.thayer@oracle.com,
+ * Hans de Goede <hdegoede@redhat.com>
+ */
+
+#include <linux/vbox_err.h>
+#include <drm/drm_fb_helper.h>
+#include <drm/drm_crtc_helper.h>
+
+#include "vbox_drv.h"
+#include "vboxvideo_guest.h"
+#include "vboxvideo_vbe.h"
+
+static void vbox_user_framebuffer_destroy(struct drm_framebuffer *fb)
+{
+ struct vbox_framebuffer *vbox_fb = to_vbox_framebuffer(fb);
+
+ if (vbox_fb->obj)
+ drm_gem_object_put_unlocked(vbox_fb->obj);
+
+ drm_framebuffer_cleanup(fb);
+ kfree(fb);
+}
+
+void vbox_report_caps(struct vbox_private *vbox)
+{
+ u32 caps = VBVACAPS_DISABLE_CURSOR_INTEGRATION |
+ VBVACAPS_IRQ | VBVACAPS_USE_VBVA_ONLY;
+
+ /* The host only accepts VIDEO_MODE_HINTS if it is send separately. */
+ hgsmi_send_caps_info(vbox->guest_pool, caps);
+ caps |= VBVACAPS_VIDEO_MODE_HINTS;
+ hgsmi_send_caps_info(vbox->guest_pool, caps);
+}
+
+/* Send information about dirty rectangles to VBVA. */
+void vbox_framebuffer_dirty_rectangles(struct drm_framebuffer *fb,
+ struct drm_clip_rect *rects,
+ unsigned int num_rects)
+{
+ struct vbox_private *vbox = fb->dev->dev_private;
+ struct drm_display_mode *mode;
+ struct drm_crtc *crtc;
+ int crtc_x, crtc_y;
+ unsigned int i;
+
+ mutex_lock(&vbox->hw_mutex);
+ list_for_each_entry(crtc, &fb->dev->mode_config.crtc_list, head) {
+ if (crtc->primary->state->fb != fb)
+ continue;
+
+ mode = &crtc->state->mode;
+ crtc_x = crtc->primary->state->src_x >> 16;
+ crtc_y = crtc->primary->state->src_y >> 16;
+
+ for (i = 0; i < num_rects; ++i) {
+ struct vbva_cmd_hdr cmd_hdr;
+ unsigned int crtc_id = to_vbox_crtc(crtc)->crtc_id;
+
+ if (rects[i].x1 > crtc_x + mode->hdisplay ||
+ rects[i].y1 > crtc_y + mode->vdisplay ||
+ rects[i].x2 < crtc_x ||
+ rects[i].y2 < crtc_y)
+ continue;
+
+ cmd_hdr.x = (s16)rects[i].x1;
+ cmd_hdr.y = (s16)rects[i].y1;
+ cmd_hdr.w = (u16)rects[i].x2 - rects[i].x1;
+ cmd_hdr.h = (u16)rects[i].y2 - rects[i].y1;
+
+ if (!vbva_buffer_begin_update(&vbox->vbva_info[crtc_id],
+ vbox->guest_pool))
+ continue;
+
+ vbva_write(&vbox->vbva_info[crtc_id], vbox->guest_pool,
+ &cmd_hdr, sizeof(cmd_hdr));
+ vbva_buffer_end_update(&vbox->vbva_info[crtc_id]);
+ }
+ }
+ mutex_unlock(&vbox->hw_mutex);
+}
+
+static int vbox_user_framebuffer_dirty(struct drm_framebuffer *fb,
+ struct drm_file *file_priv,
+ unsigned int flags, unsigned int color,
+ struct drm_clip_rect *rects,
+ unsigned int num_rects)
+{
+ vbox_framebuffer_dirty_rectangles(fb, rects, num_rects);
+
+ return 0;
+}
+
+static const struct drm_framebuffer_funcs vbox_fb_funcs = {
+ .destroy = vbox_user_framebuffer_destroy,
+ .dirty = vbox_user_framebuffer_dirty,
+};
+
+int vbox_framebuffer_init(struct vbox_private *vbox,
+ struct vbox_framebuffer *vbox_fb,
+ const struct drm_mode_fb_cmd2 *mode_cmd,
+ struct drm_gem_object *obj)
+{
+ int ret;
+
+ drm_helper_mode_fill_fb_struct(&vbox->ddev, &vbox_fb->base, mode_cmd);
+ vbox_fb->obj = obj;
+ ret = drm_framebuffer_init(&vbox->ddev, &vbox_fb->base, &vbox_fb_funcs);
+ if (ret) {
+ DRM_ERROR("framebuffer init failed %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static int vbox_accel_init(struct vbox_private *vbox)
+{
+ struct vbva_buffer *vbva;
+ unsigned int i;
+
+ vbox->vbva_info = devm_kcalloc(vbox->ddev.dev, vbox->num_crtcs,
+ sizeof(*vbox->vbva_info), GFP_KERNEL);
+ if (!vbox->vbva_info)
+ return -ENOMEM;
+
+ /* Take a command buffer for each screen from the end of usable VRAM. */
+ vbox->available_vram_size -= vbox->num_crtcs * VBVA_MIN_BUFFER_SIZE;
+
+ vbox->vbva_buffers = pci_iomap_range(vbox->ddev.pdev, 0,
+ vbox->available_vram_size,
+ vbox->num_crtcs *
+ VBVA_MIN_BUFFER_SIZE);
+ if (!vbox->vbva_buffers)
+ return -ENOMEM;
+
+ for (i = 0; i < vbox->num_crtcs; ++i) {
+ vbva_setup_buffer_context(&vbox->vbva_info[i],
+ vbox->available_vram_size +
+ i * VBVA_MIN_BUFFER_SIZE,
+ VBVA_MIN_BUFFER_SIZE);
+ vbva = (void __force *)vbox->vbva_buffers +
+ i * VBVA_MIN_BUFFER_SIZE;
+ if (!vbva_enable(&vbox->vbva_info[i],
+ vbox->guest_pool, vbva, i)) {
+ /* very old host or driver error. */
+ DRM_ERROR("vboxvideo: vbva_enable failed\n");
+ }
+ }
+
+ return 0;
+}
+
+static void vbox_accel_fini(struct vbox_private *vbox)
+{
+ unsigned int i;
+
+ for (i = 0; i < vbox->num_crtcs; ++i)
+ vbva_disable(&vbox->vbva_info[i], vbox->guest_pool, i);
+
+ pci_iounmap(vbox->ddev.pdev, vbox->vbva_buffers);
+}
+
+/* Do we support the 4.3 plus mode hint reporting interface? */
+static bool have_hgsmi_mode_hints(struct vbox_private *vbox)
+{
+ u32 have_hints, have_cursor;
+ int ret;
+
+ ret = hgsmi_query_conf(vbox->guest_pool,
+ VBOX_VBVA_CONF32_MODE_HINT_REPORTING,
+ &have_hints);
+ if (ret)
+ return false;
+
+ ret = hgsmi_query_conf(vbox->guest_pool,
+ VBOX_VBVA_CONF32_GUEST_CURSOR_REPORTING,
+ &have_cursor);
+ if (ret)
+ return false;
+
+ return have_hints == VINF_SUCCESS && have_cursor == VINF_SUCCESS;
+}
+
+bool vbox_check_supported(u16 id)
+{
+ u16 dispi_id;
+
+ vbox_write_ioport(VBE_DISPI_INDEX_ID, id);
+ dispi_id = inw(VBE_DISPI_IOPORT_DATA);
+
+ return dispi_id == id;
+}
+
+int vbox_hw_init(struct vbox_private *vbox)
+{
+ int ret = -ENOMEM;
+
+ vbox->full_vram_size = inl(VBE_DISPI_IOPORT_DATA);
+ vbox->any_pitch = vbox_check_supported(VBE_DISPI_ID_ANYX);
+
+ DRM_INFO("VRAM %08x\n", vbox->full_vram_size);
+
+ /* Map guest-heap at end of vram */
+ vbox->guest_heap =
+ pci_iomap_range(vbox->ddev.pdev, 0, GUEST_HEAP_OFFSET(vbox),
+ GUEST_HEAP_SIZE);
+ if (!vbox->guest_heap)
+ return -ENOMEM;
+
+ /* Create guest-heap mem-pool use 2^4 = 16 byte chunks */
+ vbox->guest_pool = gen_pool_create(4, -1);
+ if (!vbox->guest_pool)
+ goto err_unmap_guest_heap;
+
+ ret = gen_pool_add_virt(vbox->guest_pool,
+ (unsigned long)vbox->guest_heap,
+ GUEST_HEAP_OFFSET(vbox),
+ GUEST_HEAP_USABLE_SIZE, -1);
+ if (ret)
+ goto err_destroy_guest_pool;
+
+ ret = hgsmi_test_query_conf(vbox->guest_pool);
+ if (ret) {
+ DRM_ERROR("vboxvideo: hgsmi_test_query_conf failed\n");
+ goto err_destroy_guest_pool;
+ }
+
+ /* Reduce available VRAM size to reflect the guest heap. */
+ vbox->available_vram_size = GUEST_HEAP_OFFSET(vbox);
+ /* Linux drm represents monitors as a 32-bit array. */
+ hgsmi_query_conf(vbox->guest_pool, VBOX_VBVA_CONF32_MONITOR_COUNT,
+ &vbox->num_crtcs);
+ vbox->num_crtcs = clamp_t(u32, vbox->num_crtcs, 1, VBOX_MAX_SCREENS);
+
+ if (!have_hgsmi_mode_hints(vbox)) {
+ ret = -ENOTSUPP;
+ goto err_destroy_guest_pool;
+ }
+
+ vbox->last_mode_hints = devm_kcalloc(vbox->ddev.dev, vbox->num_crtcs,
+ sizeof(struct vbva_modehint),
+ GFP_KERNEL);
+ if (!vbox->last_mode_hints) {
+ ret = -ENOMEM;
+ goto err_destroy_guest_pool;
+ }
+
+ ret = vbox_accel_init(vbox);
+ if (ret)
+ goto err_destroy_guest_pool;
+
+ return 0;
+
+err_destroy_guest_pool:
+ gen_pool_destroy(vbox->guest_pool);
+err_unmap_guest_heap:
+ pci_iounmap(vbox->ddev.pdev, vbox->guest_heap);
+ return ret;
+}
+
+void vbox_hw_fini(struct vbox_private *vbox)
+{
+ vbox_accel_fini(vbox);
+ gen_pool_destroy(vbox->guest_pool);
+ pci_iounmap(vbox->ddev.pdev, vbox->guest_heap);
+}
+
+int vbox_gem_create(struct vbox_private *vbox,
+ u32 size, bool iskernel, struct drm_gem_object **obj)
+{
+ struct vbox_bo *vboxbo;
+ int ret;
+
+ *obj = NULL;
+
+ size = roundup(size, PAGE_SIZE);
+ if (size == 0)
+ return -EINVAL;
+
+ ret = vbox_bo_create(vbox, size, 0, 0, &vboxbo);
+ if (ret) {
+ if (ret != -ERESTARTSYS)
+ DRM_ERROR("failed to allocate GEM object\n");
+ return ret;
+ }
+
+ *obj = &vboxbo->gem;
+
+ return 0;
+}
+
+int vbox_dumb_create(struct drm_file *file,
+ struct drm_device *dev, struct drm_mode_create_dumb *args)
+{
+ struct vbox_private *vbox =
+ container_of(dev, struct vbox_private, ddev);
+ struct drm_gem_object *gobj;
+ u32 handle;
+ int ret;
+
+ args->pitch = args->width * ((args->bpp + 7) / 8);
+ args->size = args->pitch * args->height;
+
+ ret = vbox_gem_create(vbox, args->size, false, &gobj);
+ if (ret)
+ return ret;
+
+ ret = drm_gem_handle_create(file, gobj, &handle);
+ drm_gem_object_put_unlocked(gobj);
+ if (ret)
+ return ret;
+
+ args->handle = handle;
+
+ return 0;
+}
+
+void vbox_gem_free_object(struct drm_gem_object *obj)
+{
+ struct vbox_bo *vbox_bo = gem_to_vbox_bo(obj);
+
+ ttm_bo_put(&vbox_bo->bo);
+}
+
+static inline u64 vbox_bo_mmap_offset(struct vbox_bo *bo)
+{
+ return drm_vma_node_offset_addr(&bo->bo.vma_node);
+}
+
+int
+vbox_dumb_mmap_offset(struct drm_file *file,
+ struct drm_device *dev,
+ u32 handle, u64 *offset)
+{
+ struct drm_gem_object *obj;
+ int ret;
+ struct vbox_bo *bo;
+
+ mutex_lock(&dev->struct_mutex);
+ obj = drm_gem_object_lookup(file, handle);
+ if (!obj) {
+ ret = -ENOENT;
+ goto out_unlock;
+ }
+
+ bo = gem_to_vbox_bo(obj);
+ *offset = vbox_bo_mmap_offset(bo);
+
+ drm_gem_object_put(obj);
+ ret = 0;
+
+out_unlock:
+ mutex_unlock(&dev->struct_mutex);
+ return ret;
+}
--- /dev/null
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright (C) 2013-2017 Oracle Corporation
+ * This file is based on ast_mode.c
+ * Copyright 2012 Red Hat Inc.
+ * Parts based on xf86-video-ast
+ * Copyright (c) 2005 ASPEED Technology Inc.
+ * Authors: Dave Airlie <airlied@redhat.com>
+ * Michael Thayer <michael.thayer@oracle.com,
+ * Hans de Goede <hdegoede@redhat.com>
+ */
+#include <linux/export.h>
+
+#include <drm/drm_atomic.h>
+#include <drm/drm_atomic_helper.h>
+#include <drm/drm_fourcc.h>
+#include <drm/drm_plane_helper.h>
+#include <drm/drm_probe_helper.h>
+#include <drm/drm_vblank.h>
+
+#include "hgsmi_channels.h"
+#include "vbox_drv.h"
+#include "vboxvideo.h"
+
+/*
+ * Set a graphics mode. Poke any required values into registers, do an HGSMI
+ * mode set and tell the host we support advanced graphics functions.
+ */
+static void vbox_do_modeset(struct drm_crtc *crtc)
+{
+ struct drm_framebuffer *fb = crtc->primary->state->fb;
+ struct vbox_crtc *vbox_crtc = to_vbox_crtc(crtc);
+ struct vbox_private *vbox;
+ int width, height, bpp, pitch;
+ u16 flags;
+ s32 x_offset, y_offset;
+
+ vbox = crtc->dev->dev_private;
+ width = vbox_crtc->width ? vbox_crtc->width : 640;
+ height = vbox_crtc->height ? vbox_crtc->height : 480;
+ bpp = fb ? fb->format->cpp[0] * 8 : 32;
+ pitch = fb ? fb->pitches[0] : width * bpp / 8;
+ x_offset = vbox->single_framebuffer ? vbox_crtc->x : vbox_crtc->x_hint;
+ y_offset = vbox->single_framebuffer ? vbox_crtc->y : vbox_crtc->y_hint;
+
+ /*
+ * This is the old way of setting graphics modes. It assumed one screen
+ * and a frame-buffer at the start of video RAM. On older versions of
+ * VirtualBox, certain parts of the code still assume that the first
+ * screen is programmed this way, so try to fake it.
+ */
+ if (vbox_crtc->crtc_id == 0 && fb &&
+ vbox_crtc->fb_offset / pitch < 0xffff - crtc->y &&
+ vbox_crtc->fb_offset % (bpp / 8) == 0) {
+ vbox_write_ioport(VBE_DISPI_INDEX_XRES, width);
+ vbox_write_ioport(VBE_DISPI_INDEX_YRES, height);
+ vbox_write_ioport(VBE_DISPI_INDEX_VIRT_WIDTH, pitch * 8 / bpp);
+ vbox_write_ioport(VBE_DISPI_INDEX_BPP, bpp);
+ vbox_write_ioport(VBE_DISPI_INDEX_ENABLE, VBE_DISPI_ENABLED);
+ vbox_write_ioport(VBE_DISPI_INDEX_X_OFFSET,
+ vbox_crtc->fb_offset % pitch / bpp * 8 + vbox_crtc->x);
+ vbox_write_ioport(VBE_DISPI_INDEX_Y_OFFSET,
+ vbox_crtc->fb_offset / pitch + vbox_crtc->y);
+ }
+
+ flags = VBVA_SCREEN_F_ACTIVE;
+ flags |= (fb && crtc->state->enable) ? 0 : VBVA_SCREEN_F_BLANK;
+ flags |= vbox_crtc->disconnected ? VBVA_SCREEN_F_DISABLED : 0;
+ hgsmi_process_display_info(vbox->guest_pool, vbox_crtc->crtc_id,
+ x_offset, y_offset,
+ vbox_crtc->x * bpp / 8 +
+ vbox_crtc->y * pitch,
+ pitch, width, height, bpp, flags);
+}
+
+static int vbox_set_view(struct drm_crtc *crtc)
+{
+ struct vbox_crtc *vbox_crtc = to_vbox_crtc(crtc);
+ struct vbox_private *vbox = crtc->dev->dev_private;
+ struct vbva_infoview *p;
+
+ /*
+ * Tell the host about the view. This design originally targeted the
+ * Windows XP driver architecture and assumed that each screen would
+ * have a dedicated frame buffer with the command buffer following it,
+ * the whole being a "view". The host works out which screen a command
+ * buffer belongs to by checking whether it is in the first view, then
+ * whether it is in the second and so on. The first match wins. We
+ * cheat around this by making the first view be the managed memory
+ * plus the first command buffer, the second the same plus the second
+ * buffer and so on.
+ */
+ p = hgsmi_buffer_alloc(vbox->guest_pool, sizeof(*p),
+ HGSMI_CH_VBVA, VBVA_INFO_VIEW);
+ if (!p)
+ return -ENOMEM;
+
+ p->view_index = vbox_crtc->crtc_id;
+ p->view_offset = vbox_crtc->fb_offset;
+ p->view_size = vbox->available_vram_size - vbox_crtc->fb_offset +
+ vbox_crtc->crtc_id * VBVA_MIN_BUFFER_SIZE;
+ p->max_screen_size = vbox->available_vram_size - vbox_crtc->fb_offset;
+
+ hgsmi_buffer_submit(vbox->guest_pool, p);
+ hgsmi_buffer_free(vbox->guest_pool, p);
+
+ return 0;
+}
+
+/*
+ * Try to map the layout of virtual screens to the range of the input device.
+ * Return true if we need to re-set the crtc modes due to screen offset
+ * changes.
+ */
+static bool vbox_set_up_input_mapping(struct vbox_private *vbox)
+{
+ struct drm_crtc *crtci;
+ struct drm_connector *connectori;
+ struct drm_framebuffer *fb, *fb1 = NULL;
+ bool single_framebuffer = true;
+ bool old_single_framebuffer = vbox->single_framebuffer;
+ u16 width = 0, height = 0;
+
+ /*
+ * Are we using an X.Org-style single large frame-buffer for all crtcs?
+ * If so then screen layout can be deduced from the crtc offsets.
+ * Same fall-back if this is the fbdev frame-buffer.
+ */
+ list_for_each_entry(crtci, &vbox->ddev.mode_config.crtc_list, head) {
+ fb = crtci->primary->state->fb;
+ if (!fb)
+ continue;
+
+ if (!fb1) {
+ fb1 = fb;
+ if (to_vbox_framebuffer(fb1) == &vbox->afb)
+ break;
+ } else if (fb != fb1) {
+ single_framebuffer = false;
+ }
+ }
+ if (!fb1)
+ return false;
+
+ if (single_framebuffer) {
+ vbox->single_framebuffer = true;
+ vbox->input_mapping_width = fb1->width;
+ vbox->input_mapping_height = fb1->height;
+ return old_single_framebuffer != vbox->single_framebuffer;
+ }
+ /* Otherwise calculate the total span of all screens. */
+ list_for_each_entry(connectori, &vbox->ddev.mode_config.connector_list,
+ head) {
+ struct vbox_connector *vbox_connector =
+ to_vbox_connector(connectori);
+ struct vbox_crtc *vbox_crtc = vbox_connector->vbox_crtc;
+
+ width = max_t(u16, width, vbox_crtc->x_hint +
+ vbox_connector->mode_hint.width);
+ height = max_t(u16, height, vbox_crtc->y_hint +
+ vbox_connector->mode_hint.height);
+ }
+
+ vbox->single_framebuffer = false;
+ vbox->input_mapping_width = width;
+ vbox->input_mapping_height = height;
+
+ return old_single_framebuffer != vbox->single_framebuffer;
+}
+
+static void vbox_crtc_set_base_and_mode(struct drm_crtc *crtc,
+ struct drm_framebuffer *fb,
+ int x, int y)
+{
+ struct vbox_bo *bo = gem_to_vbox_bo(to_vbox_framebuffer(fb)->obj);
+ struct vbox_private *vbox = crtc->dev->dev_private;
+ struct vbox_crtc *vbox_crtc = to_vbox_crtc(crtc);
+ bool needs_modeset = drm_atomic_crtc_needs_modeset(crtc->state);
+
+ mutex_lock(&vbox->hw_mutex);
+
+ if (crtc->state->enable) {
+ vbox_crtc->width = crtc->state->mode.hdisplay;
+ vbox_crtc->height = crtc->state->mode.vdisplay;
+ }
+
+ vbox_crtc->x = x;
+ vbox_crtc->y = y;
+ vbox_crtc->fb_offset = vbox_bo_gpu_offset(bo);
+
+ /* vbox_do_modeset() checks vbox->single_framebuffer so update it now */
+ if (needs_modeset && vbox_set_up_input_mapping(vbox)) {
+ struct drm_crtc *crtci;
+
+ list_for_each_entry(crtci, &vbox->ddev.mode_config.crtc_list,
+ head) {
+ if (crtci == crtc)
+ continue;
+ vbox_do_modeset(crtci);
+ }
+ }
+
+ vbox_set_view(crtc);
+ vbox_do_modeset(crtc);
+
+ if (needs_modeset)
+ hgsmi_update_input_mapping(vbox->guest_pool, 0, 0,
+ vbox->input_mapping_width,
+ vbox->input_mapping_height);
+
+ mutex_unlock(&vbox->hw_mutex);
+}
+
+static void vbox_crtc_atomic_enable(struct drm_crtc *crtc,
+ struct drm_crtc_state *old_crtc_state)
+{
+}
+
+static void vbox_crtc_atomic_disable(struct drm_crtc *crtc,
+ struct drm_crtc_state *old_crtc_state)
+{
+}
+
+static void vbox_crtc_atomic_flush(struct drm_crtc *crtc,
+ struct drm_crtc_state *old_crtc_state)
+{
+ struct drm_pending_vblank_event *event;
+ unsigned long flags;
+
+ if (crtc->state && crtc->state->event) {
+ event = crtc->state->event;
+ crtc->state->event = NULL;
+
+ spin_lock_irqsave(&crtc->dev->event_lock, flags);
+ drm_crtc_send_vblank_event(crtc, event);
+ spin_unlock_irqrestore(&crtc->dev->event_lock, flags);
+ }
+}
+
+static const struct drm_crtc_helper_funcs vbox_crtc_helper_funcs = {
+ .atomic_enable = vbox_crtc_atomic_enable,
+ .atomic_disable = vbox_crtc_atomic_disable,
+ .atomic_flush = vbox_crtc_atomic_flush,
+};
+
+static void vbox_crtc_destroy(struct drm_crtc *crtc)
+{
+ drm_crtc_cleanup(crtc);
+ kfree(crtc);
+}
+
+static const struct drm_crtc_funcs vbox_crtc_funcs = {
+ .set_config = drm_atomic_helper_set_config,
+ .page_flip = drm_atomic_helper_page_flip,
+ /* .gamma_set = vbox_crtc_gamma_set, */
+ .destroy = vbox_crtc_destroy,
+ .reset = drm_atomic_helper_crtc_reset,
+ .atomic_duplicate_state = drm_atomic_helper_crtc_duplicate_state,
+ .atomic_destroy_state = drm_atomic_helper_crtc_destroy_state,
+};
+
+static int vbox_primary_atomic_check(struct drm_plane *plane,
+ struct drm_plane_state *new_state)
+{
+ struct drm_crtc_state *crtc_state = NULL;
+
+ if (new_state->crtc) {
+ crtc_state = drm_atomic_get_existing_crtc_state(
+ new_state->state, new_state->crtc);
+ if (WARN_ON(!crtc_state))
+ return -EINVAL;
+ }
+
+ return drm_atomic_helper_check_plane_state(new_state, crtc_state,
+ DRM_PLANE_HELPER_NO_SCALING,
+ DRM_PLANE_HELPER_NO_SCALING,
+ false, true);
+}
+
+static void vbox_primary_atomic_update(struct drm_plane *plane,
+ struct drm_plane_state *old_state)
+{
+ struct drm_crtc *crtc = plane->state->crtc;
+ struct drm_framebuffer *fb = plane->state->fb;
+
+ vbox_crtc_set_base_and_mode(crtc, fb,
+ plane->state->src_x >> 16,
+ plane->state->src_y >> 16);
+}
+
+static void vbox_primary_atomic_disable(struct drm_plane *plane,
+ struct drm_plane_state *old_state)
+{
+ struct drm_crtc *crtc = old_state->crtc;
+
+ /* vbox_do_modeset checks plane->state->fb and will disable if NULL */
+ vbox_crtc_set_base_and_mode(crtc, old_state->fb,
+ old_state->src_x >> 16,
+ old_state->src_y >> 16);
+}
+
+static int vbox_primary_prepare_fb(struct drm_plane *plane,
+ struct drm_plane_state *new_state)
+{
+ struct vbox_bo *bo;
+ int ret;
+
+ if (!new_state->fb)
+ return 0;
+
+ bo = gem_to_vbox_bo(to_vbox_framebuffer(new_state->fb)->obj);
+ ret = vbox_bo_pin(bo, TTM_PL_FLAG_VRAM);
+ if (ret)
+ DRM_WARN("Error %d pinning new fb, out of video mem?\n", ret);
+
+ return ret;
+}
+
+static void vbox_primary_cleanup_fb(struct drm_plane *plane,
+ struct drm_plane_state *old_state)
+{
+ struct vbox_bo *bo;
+
+ if (!old_state->fb)
+ return;
+
+ bo = gem_to_vbox_bo(to_vbox_framebuffer(old_state->fb)->obj);
+ vbox_bo_unpin(bo);
+}
+
+static int vbox_cursor_atomic_check(struct drm_plane *plane,
+ struct drm_plane_state *new_state)
+{
+ struct drm_crtc_state *crtc_state = NULL;
+ u32 width = new_state->crtc_w;
+ u32 height = new_state->crtc_h;
+ int ret;
+
+ if (new_state->crtc) {
+ crtc_state = drm_atomic_get_existing_crtc_state(
+ new_state->state, new_state->crtc);
+ if (WARN_ON(!crtc_state))
+ return -EINVAL;
+ }
+
+ ret = drm_atomic_helper_check_plane_state(new_state, crtc_state,
+ DRM_PLANE_HELPER_NO_SCALING,
+ DRM_PLANE_HELPER_NO_SCALING,
+ true, true);
+ if (ret)
+ return ret;
+
+ if (!new_state->fb)
+ return 0;
+
+ if (width > VBOX_MAX_CURSOR_WIDTH || height > VBOX_MAX_CURSOR_HEIGHT ||
+ width == 0 || height == 0)
+ return -EINVAL;
+
+ return 0;
+}
+
+/*
+ * Copy the ARGB image and generate the mask, which is needed in case the host
+ * does not support ARGB cursors. The mask is a 1BPP bitmap with the bit set
+ * if the corresponding alpha value in the ARGB image is greater than 0xF0.
+ */
+static void copy_cursor_image(u8 *src, u8 *dst, u32 width, u32 height,
+ size_t mask_size)
+{
+ size_t line_size = (width + 7) / 8;
+ u32 i, j;
+
+ memcpy(dst + mask_size, src, width * height * 4);
+ for (i = 0; i < height; ++i)
+ for (j = 0; j < width; ++j)
+ if (((u32 *)src)[i * width + j] > 0xf0000000)
+ dst[i * line_size + j / 8] |= (0x80 >> (j % 8));
+}
+
+static void vbox_cursor_atomic_update(struct drm_plane *plane,
+ struct drm_plane_state *old_state)
+{
+ struct vbox_private *vbox =
+ container_of(plane->dev, struct vbox_private, ddev);
+ struct vbox_crtc *vbox_crtc = to_vbox_crtc(plane->state->crtc);
+ struct drm_framebuffer *fb = plane->state->fb;
+ struct vbox_bo *bo = gem_to_vbox_bo(to_vbox_framebuffer(fb)->obj);
+ u32 width = plane->state->crtc_w;
+ u32 height = plane->state->crtc_h;
+ size_t data_size, mask_size;
+ u32 flags;
+ u8 *src;
+
+ /*
+ * VirtualBox uses the host windowing system to draw the cursor so
+ * moves are a no-op, we only need to upload new cursor sprites.
+ */
+ if (fb == old_state->fb)
+ return;
+
+ mutex_lock(&vbox->hw_mutex);
+
+ vbox_crtc->cursor_enabled = true;
+
+ /* pinning is done in prepare/cleanup framebuffer */
+ src = vbox_bo_kmap(bo);
+ if (IS_ERR(src)) {
+ mutex_unlock(&vbox->hw_mutex);
+ DRM_WARN("Could not kmap cursor bo, skipping update\n");
+ return;
+ }
+
+ /*
+ * The mask must be calculated based on the alpha
+ * channel, one bit per ARGB word, and must be 32-bit
+ * padded.
+ */
+ mask_size = ((width + 7) / 8 * height + 3) & ~3;
+ data_size = width * height * 4 + mask_size;
+
+ copy_cursor_image(src, vbox->cursor_data, width, height, mask_size);
+ vbox_bo_kunmap(bo);
+
+ flags = VBOX_MOUSE_POINTER_VISIBLE | VBOX_MOUSE_POINTER_SHAPE |
+ VBOX_MOUSE_POINTER_ALPHA;
+ hgsmi_update_pointer_shape(vbox->guest_pool, flags,
+ min_t(u32, max(fb->hot_x, 0), width),
+ min_t(u32, max(fb->hot_y, 0), height),
+ width, height, vbox->cursor_data, data_size);
+
+ mutex_unlock(&vbox->hw_mutex);
+}
+
+static void vbox_cursor_atomic_disable(struct drm_plane *plane,
+ struct drm_plane_state *old_state)
+{
+ struct vbox_private *vbox =
+ container_of(plane->dev, struct vbox_private, ddev);
+ struct vbox_crtc *vbox_crtc = to_vbox_crtc(old_state->crtc);
+ bool cursor_enabled = false;
+ struct drm_crtc *crtci;
+
+ mutex_lock(&vbox->hw_mutex);
+
+ vbox_crtc->cursor_enabled = false;
+
+ list_for_each_entry(crtci, &vbox->ddev.mode_config.crtc_list, head) {
+ if (to_vbox_crtc(crtci)->cursor_enabled)
+ cursor_enabled = true;
+ }
+
+ if (!cursor_enabled)
+ hgsmi_update_pointer_shape(vbox->guest_pool, 0, 0, 0,
+ 0, 0, NULL, 0);
+
+ mutex_unlock(&vbox->hw_mutex);
+}
+
+static int vbox_cursor_prepare_fb(struct drm_plane *plane,
+ struct drm_plane_state *new_state)
+{
+ struct vbox_bo *bo;
+
+ if (!new_state->fb)
+ return 0;
+
+ bo = gem_to_vbox_bo(to_vbox_framebuffer(new_state->fb)->obj);
+ return vbox_bo_pin(bo, TTM_PL_FLAG_SYSTEM);
+}
+
+static void vbox_cursor_cleanup_fb(struct drm_plane *plane,
+ struct drm_plane_state *old_state)
+{
+ struct vbox_bo *bo;
+
+ if (!plane->state->fb)
+ return;
+
+ bo = gem_to_vbox_bo(to_vbox_framebuffer(plane->state->fb)->obj);
+ vbox_bo_unpin(bo);
+}
+
+static const u32 vbox_cursor_plane_formats[] = {
+ DRM_FORMAT_ARGB8888,
+};
+
+static const struct drm_plane_helper_funcs vbox_cursor_helper_funcs = {
+ .atomic_check = vbox_cursor_atomic_check,
+ .atomic_update = vbox_cursor_atomic_update,
+ .atomic_disable = vbox_cursor_atomic_disable,
+ .prepare_fb = vbox_cursor_prepare_fb,
+ .cleanup_fb = vbox_cursor_cleanup_fb,
+};
+
+static const struct drm_plane_funcs vbox_cursor_plane_funcs = {
+ .update_plane = drm_atomic_helper_update_plane,
+ .disable_plane = drm_atomic_helper_disable_plane,
+ .destroy = drm_primary_helper_destroy,
+ .reset = drm_atomic_helper_plane_reset,
+ .atomic_duplicate_state = drm_atomic_helper_plane_duplicate_state,
+ .atomic_destroy_state = drm_atomic_helper_plane_destroy_state,
+};
+
+static const u32 vbox_primary_plane_formats[] = {
+ DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_ARGB8888,
+};
+
+static const struct drm_plane_helper_funcs vbox_primary_helper_funcs = {
+ .atomic_check = vbox_primary_atomic_check,
+ .atomic_update = vbox_primary_atomic_update,
+ .atomic_disable = vbox_primary_atomic_disable,
+ .prepare_fb = vbox_primary_prepare_fb,
+ .cleanup_fb = vbox_primary_cleanup_fb,
+};
+
+static const struct drm_plane_funcs vbox_primary_plane_funcs = {
+ .update_plane = drm_atomic_helper_update_plane,
+ .disable_plane = drm_atomic_helper_disable_plane,
+ .destroy = drm_primary_helper_destroy,
+ .reset = drm_atomic_helper_plane_reset,
+ .atomic_duplicate_state = drm_atomic_helper_plane_duplicate_state,
+ .atomic_destroy_state = drm_atomic_helper_plane_destroy_state,
+};
+
+static struct drm_plane *vbox_create_plane(struct vbox_private *vbox,
+ unsigned int possible_crtcs,
+ enum drm_plane_type type)
+{
+ const struct drm_plane_helper_funcs *helper_funcs = NULL;
+ const struct drm_plane_funcs *funcs;
+ struct drm_plane *plane;
+ const u32 *formats;
+ int num_formats;
+ int err;
+
+ if (type == DRM_PLANE_TYPE_PRIMARY) {
+ funcs = &vbox_primary_plane_funcs;
+ formats = vbox_primary_plane_formats;
+ helper_funcs = &vbox_primary_helper_funcs;
+ num_formats = ARRAY_SIZE(vbox_primary_plane_formats);
+ } else if (type == DRM_PLANE_TYPE_CURSOR) {
+ funcs = &vbox_cursor_plane_funcs;
+ formats = vbox_cursor_plane_formats;
+ helper_funcs = &vbox_cursor_helper_funcs;
+ num_formats = ARRAY_SIZE(vbox_cursor_plane_formats);
+ } else {
+ return ERR_PTR(-EINVAL);
+ }
+
+ plane = kzalloc(sizeof(*plane), GFP_KERNEL);
+ if (!plane)
+ return ERR_PTR(-ENOMEM);
+
+ err = drm_universal_plane_init(&vbox->ddev, plane, possible_crtcs,
+ funcs, formats, num_formats,
+ NULL, type, NULL);
+ if (err)
+ goto free_plane;
+
+ drm_plane_helper_add(plane, helper_funcs);
+
+ return plane;
+
+free_plane:
+ kfree(plane);
+ return ERR_PTR(-EINVAL);
+}
+
+static struct vbox_crtc *vbox_crtc_init(struct drm_device *dev, unsigned int i)
+{
+ struct vbox_private *vbox =
+ container_of(dev, struct vbox_private, ddev);
+ struct drm_plane *cursor = NULL;
+ struct vbox_crtc *vbox_crtc;
+ struct drm_plane *primary;
+ u32 caps = 0;
+ int ret;
+
+ ret = hgsmi_query_conf(vbox->guest_pool,
+ VBOX_VBVA_CONF32_CURSOR_CAPABILITIES, &caps);
+ if (ret)
+ return ERR_PTR(ret);
+
+ vbox_crtc = kzalloc(sizeof(*vbox_crtc), GFP_KERNEL);
+ if (!vbox_crtc)
+ return ERR_PTR(-ENOMEM);
+
+ primary = vbox_create_plane(vbox, 1 << i, DRM_PLANE_TYPE_PRIMARY);
+ if (IS_ERR(primary)) {
+ ret = PTR_ERR(primary);
+ goto free_mem;
+ }
+
+ if ((caps & VBOX_VBVA_CURSOR_CAPABILITY_HARDWARE)) {
+ cursor = vbox_create_plane(vbox, 1 << i, DRM_PLANE_TYPE_CURSOR);
+ if (IS_ERR(cursor)) {
+ ret = PTR_ERR(cursor);
+ goto clean_primary;
+ }
+ } else {
+ DRM_WARN("VirtualBox host is too old, no cursor support\n");
+ }
+
+ vbox_crtc->crtc_id = i;
+
+ ret = drm_crtc_init_with_planes(dev, &vbox_crtc->base, primary, cursor,
+ &vbox_crtc_funcs, NULL);
+ if (ret)
+ goto clean_cursor;
+
+ drm_mode_crtc_set_gamma_size(&vbox_crtc->base, 256);
+ drm_crtc_helper_add(&vbox_crtc->base, &vbox_crtc_helper_funcs);
+
+ return vbox_crtc;
+
+clean_cursor:
+ if (cursor) {
+ drm_plane_cleanup(cursor);
+ kfree(cursor);
+ }
+clean_primary:
+ drm_plane_cleanup(primary);
+ kfree(primary);
+free_mem:
+ kfree(vbox_crtc);
+ return ERR_PTR(ret);
+}
+
+static void vbox_encoder_destroy(struct drm_encoder *encoder)
+{
+ drm_encoder_cleanup(encoder);
+ kfree(encoder);
+}
+
+static const struct drm_encoder_funcs vbox_enc_funcs = {
+ .destroy = vbox_encoder_destroy,
+};
+
+static struct drm_encoder *vbox_encoder_init(struct drm_device *dev,
+ unsigned int i)
+{
+ struct vbox_encoder *vbox_encoder;
+
+ vbox_encoder = kzalloc(sizeof(*vbox_encoder), GFP_KERNEL);
+ if (!vbox_encoder)
+ return NULL;
+
+ drm_encoder_init(dev, &vbox_encoder->base, &vbox_enc_funcs,
+ DRM_MODE_ENCODER_DAC, NULL);
+
+ vbox_encoder->base.possible_crtcs = 1 << i;
+ return &vbox_encoder->base;
+}
+
+/*
+ * Generate EDID data with a mode-unique serial number for the virtual
+ * monitor to try to persuade Unity that different modes correspond to
+ * different monitors and it should not try to force the same resolution on
+ * them.
+ */
+static void vbox_set_edid(struct drm_connector *connector, int width,
+ int height)
+{
+ enum { EDID_SIZE = 128 };
+ unsigned char edid[EDID_SIZE] = {
+ 0x00, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x00, /* header */
+ 0x58, 0x58, /* manufacturer (VBX) */
+ 0x00, 0x00, /* product code */
+ 0x00, 0x00, 0x00, 0x00, /* serial number goes here */
+ 0x01, /* week of manufacture */
+ 0x00, /* year of manufacture */
+ 0x01, 0x03, /* EDID version */
+ 0x80, /* capabilities - digital */
+ 0x00, /* horiz. res in cm, zero for projectors */
+ 0x00, /* vert. res in cm */
+ 0x78, /* display gamma (120 == 2.2). */
+ 0xEE, /* features (standby, suspend, off, RGB, std */
+ /* colour space, preferred timing mode) */
+ 0xEE, 0x91, 0xA3, 0x54, 0x4C, 0x99, 0x26, 0x0F, 0x50, 0x54,
+ /* chromaticity for standard colour space. */
+ 0x00, 0x00, 0x00, /* no default timings */
+ 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+ 0x01, 0x01,
+ 0x01, 0x01, 0x01, 0x01, /* no standard timings */
+ 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x06, 0x00, 0x02, 0x02,
+ 0x02, 0x02,
+ /* descriptor block 1 goes below */
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ /* descriptor block 2, monitor ranges */
+ 0x00, 0x00, 0x00, 0xFD, 0x00,
+ 0x00, 0xC8, 0x00, 0xC8, 0x64, 0x00, 0x0A, 0x20, 0x20, 0x20,
+ 0x20, 0x20,
+ /* 0-200Hz vertical, 0-200KHz horizontal, 1000MHz pixel clock */
+ 0x20,
+ /* descriptor block 3, monitor name */
+ 0x00, 0x00, 0x00, 0xFC, 0x00,
+ 'V', 'B', 'O', 'X', ' ', 'm', 'o', 'n', 'i', 't', 'o', 'r',
+ '\n',
+ /* descriptor block 4: dummy data */
+ 0x00, 0x00, 0x00, 0x10, 0x00,
+ 0x0A, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
+ 0x20,
+ 0x00, /* number of extensions */
+ 0x00 /* checksum goes here */
+ };
+ int clock = (width + 6) * (height + 6) * 60 / 10000;
+ unsigned int i, sum = 0;
+
+ edid[12] = width & 0xff;
+ edid[13] = width >> 8;
+ edid[14] = height & 0xff;
+ edid[15] = height >> 8;
+ edid[54] = clock & 0xff;
+ edid[55] = clock >> 8;
+ edid[56] = width & 0xff;
+ edid[58] = (width >> 4) & 0xf0;
+ edid[59] = height & 0xff;
+ edid[61] = (height >> 4) & 0xf0;
+ for (i = 0; i < EDID_SIZE - 1; ++i)
+ sum += edid[i];
+ edid[EDID_SIZE - 1] = (0x100 - (sum & 0xFF)) & 0xFF;
+ drm_connector_update_edid_property(connector, (struct edid *)edid);
+}
+
+static int vbox_get_modes(struct drm_connector *connector)
+{
+ struct vbox_connector *vbox_connector = NULL;
+ struct drm_display_mode *mode = NULL;
+ struct vbox_private *vbox = NULL;
+ unsigned int num_modes = 0;
+ int preferred_width, preferred_height;
+
+ vbox_connector = to_vbox_connector(connector);
+ vbox = connector->dev->dev_private;
+
+ hgsmi_report_flags_location(vbox->guest_pool, GUEST_HEAP_OFFSET(vbox) +
+ HOST_FLAGS_OFFSET);
+ if (vbox_connector->vbox_crtc->crtc_id == 0)
+ vbox_report_caps(vbox);
+
+ num_modes = drm_add_modes_noedid(connector, 2560, 1600);
+ preferred_width = vbox_connector->mode_hint.width ?
+ vbox_connector->mode_hint.width : 1024;
+ preferred_height = vbox_connector->mode_hint.height ?
+ vbox_connector->mode_hint.height : 768;
+ mode = drm_cvt_mode(connector->dev, preferred_width, preferred_height,
+ 60, false, false, false);
+ if (mode) {
+ mode->type |= DRM_MODE_TYPE_PREFERRED;
+ drm_mode_probed_add(connector, mode);
+ ++num_modes;
+ }
+ vbox_set_edid(connector, preferred_width, preferred_height);
+
+ if (vbox_connector->vbox_crtc->x_hint != -1)
+ drm_object_property_set_value(&connector->base,
+ vbox->ddev.mode_config.suggested_x_property,
+ vbox_connector->vbox_crtc->x_hint);
+ else
+ drm_object_property_set_value(&connector->base,
+ vbox->ddev.mode_config.suggested_x_property, 0);
+
+ if (vbox_connector->vbox_crtc->y_hint != -1)
+ drm_object_property_set_value(&connector->base,
+ vbox->ddev.mode_config.suggested_y_property,
+ vbox_connector->vbox_crtc->y_hint);
+ else
+ drm_object_property_set_value(&connector->base,
+ vbox->ddev.mode_config.suggested_y_property, 0);
+
+ return num_modes;
+}
+
+static void vbox_connector_destroy(struct drm_connector *connector)
+{
+ drm_connector_unregister(connector);
+ drm_connector_cleanup(connector);
+ kfree(connector);
+}
+
+static enum drm_connector_status
+vbox_connector_detect(struct drm_connector *connector, bool force)
+{
+ struct vbox_connector *vbox_connector;
+
+ vbox_connector = to_vbox_connector(connector);
+
+ return vbox_connector->mode_hint.disconnected ?
+ connector_status_disconnected : connector_status_connected;
+}
+
+static int vbox_fill_modes(struct drm_connector *connector, u32 max_x,
+ u32 max_y)
+{
+ struct vbox_connector *vbox_connector;
+ struct drm_device *dev;
+ struct drm_display_mode *mode, *iterator;
+
+ vbox_connector = to_vbox_connector(connector);
+ dev = vbox_connector->base.dev;
+ list_for_each_entry_safe(mode, iterator, &connector->modes, head) {
+ list_del(&mode->head);
+ drm_mode_destroy(dev, mode);
+ }
+
+ return drm_helper_probe_single_connector_modes(connector, max_x, max_y);
+}
+
+static const struct drm_connector_helper_funcs vbox_connector_helper_funcs = {
+ .get_modes = vbox_get_modes,
+};
+
+static const struct drm_connector_funcs vbox_connector_funcs = {
+ .detect = vbox_connector_detect,
+ .fill_modes = vbox_fill_modes,
+ .destroy = vbox_connector_destroy,
+ .reset = drm_atomic_helper_connector_reset,
+ .atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state,
+ .atomic_destroy_state = drm_atomic_helper_connector_destroy_state,
+};
+
+static int vbox_connector_init(struct drm_device *dev,
+ struct vbox_crtc *vbox_crtc,
+ struct drm_encoder *encoder)
+{
+ struct vbox_connector *vbox_connector;
+ struct drm_connector *connector;
+
+ vbox_connector = kzalloc(sizeof(*vbox_connector), GFP_KERNEL);
+ if (!vbox_connector)
+ return -ENOMEM;
+
+ connector = &vbox_connector->base;
+ vbox_connector->vbox_crtc = vbox_crtc;
+
+ drm_connector_init(dev, connector, &vbox_connector_funcs,
+ DRM_MODE_CONNECTOR_VGA);
+ drm_connector_helper_add(connector, &vbox_connector_helper_funcs);
+
+ connector->interlace_allowed = 0;
+ connector->doublescan_allowed = 0;
+
+ drm_mode_create_suggested_offset_properties(dev);
+ drm_object_attach_property(&connector->base,
+ dev->mode_config.suggested_x_property, 0);
+ drm_object_attach_property(&connector->base,
+ dev->mode_config.suggested_y_property, 0);
+
+ drm_connector_attach_encoder(connector, encoder);
+
+ return 0;
+}
+
+static struct drm_framebuffer *vbox_user_framebuffer_create(
+ struct drm_device *dev,
+ struct drm_file *filp,
+ const struct drm_mode_fb_cmd2 *mode_cmd)
+{
+ struct vbox_private *vbox =
+ container_of(dev, struct vbox_private, ddev);
+ struct drm_gem_object *obj;
+ struct vbox_framebuffer *vbox_fb;
+ int ret = -ENOMEM;
+
+ obj = drm_gem_object_lookup(filp, mode_cmd->handles[0]);
+ if (!obj)
+ return ERR_PTR(-ENOENT);
+
+ vbox_fb = kzalloc(sizeof(*vbox_fb), GFP_KERNEL);
+ if (!vbox_fb)
+ goto err_unref_obj;
+
+ ret = vbox_framebuffer_init(vbox, vbox_fb, mode_cmd, obj);
+ if (ret)
+ goto err_free_vbox_fb;
+
+ return &vbox_fb->base;
+
+err_free_vbox_fb:
+ kfree(vbox_fb);
+err_unref_obj:
+ drm_gem_object_put_unlocked(obj);
+ return ERR_PTR(ret);
+}
+
+static const struct drm_mode_config_funcs vbox_mode_funcs = {
+ .fb_create = vbox_user_framebuffer_create,
+ .atomic_check = drm_atomic_helper_check,
+ .atomic_commit = drm_atomic_helper_commit,
+};
+
+int vbox_mode_init(struct vbox_private *vbox)
+{
+ struct drm_device *dev = &vbox->ddev;
+ struct drm_encoder *encoder;
+ struct vbox_crtc *vbox_crtc;
+ unsigned int i;
+ int ret;
+
+ drm_mode_config_init(dev);
+
+ dev->mode_config.funcs = (void *)&vbox_mode_funcs;
+ dev->mode_config.min_width = 0;
+ dev->mode_config.min_height = 0;
+ dev->mode_config.preferred_depth = 24;
+ dev->mode_config.max_width = VBE_DISPI_MAX_XRES;
+ dev->mode_config.max_height = VBE_DISPI_MAX_YRES;
+
+ for (i = 0; i < vbox->num_crtcs; ++i) {
+ vbox_crtc = vbox_crtc_init(dev, i);
+ if (IS_ERR(vbox_crtc)) {
+ ret = PTR_ERR(vbox_crtc);
+ goto err_drm_mode_cleanup;
+ }
+ encoder = vbox_encoder_init(dev, i);
+ if (!encoder) {
+ ret = -ENOMEM;
+ goto err_drm_mode_cleanup;
+ }
+ ret = vbox_connector_init(dev, vbox_crtc, encoder);
+ if (ret)
+ goto err_drm_mode_cleanup;
+ }
+
+ drm_mode_config_reset(dev);
+ return 0;
+
+err_drm_mode_cleanup:
+ drm_mode_config_cleanup(dev);
+ return ret;
+}
+
+void vbox_mode_fini(struct vbox_private *vbox)
+{
+ drm_mode_config_cleanup(&vbox->ddev);
+}
--- /dev/null
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright (C) 2017 Oracle Corporation
+ * Copyright 2017 Canonical
+ * Authors: Andreas Pokorny
+ */
+
+#include "vbox_drv.h"
+
+/*
+ * Based on qxl_prime.c:
+ * Empty Implementations as there should not be any other driver for a virtual
+ * device that might share buffers with vboxvideo
+ */
+
+int vbox_gem_prime_pin(struct drm_gem_object *obj)
+{
+ WARN_ONCE(1, "not implemented");
+ return -ENODEV;
+}
+
+void vbox_gem_prime_unpin(struct drm_gem_object *obj)
+{
+ WARN_ONCE(1, "not implemented");
+}
+
+struct sg_table *vbox_gem_prime_get_sg_table(struct drm_gem_object *obj)
+{
+ WARN_ONCE(1, "not implemented");
+ return ERR_PTR(-ENODEV);
+}
+
+struct drm_gem_object *vbox_gem_prime_import_sg_table(
+ struct drm_device *dev, struct dma_buf_attachment *attach,
+ struct sg_table *table)
+{
+ WARN_ONCE(1, "not implemented");
+ return ERR_PTR(-ENODEV);
+}
+
+void *vbox_gem_prime_vmap(struct drm_gem_object *obj)
+{
+ WARN_ONCE(1, "not implemented");
+ return ERR_PTR(-ENODEV);
+}
+
+void vbox_gem_prime_vunmap(struct drm_gem_object *obj, void *vaddr)
+{
+ WARN_ONCE(1, "not implemented");
+}
+
+int vbox_gem_prime_mmap(struct drm_gem_object *obj, struct vm_area_struct *area)
+{
+ WARN_ONCE(1, "not implemented");
+ return -ENODEV;
+}
--- /dev/null
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright (C) 2013-2017 Oracle Corporation
+ * This file is based on ast_ttm.c
+ * Copyright 2012 Red Hat Inc.
+ * Authors: Dave Airlie <airlied@redhat.com>
+ * Michael Thayer <michael.thayer@oracle.com>
+ */
+#include <linux/pci.h>
+#include <drm/drm_file.h>
+#include <drm/ttm/ttm_page_alloc.h>
+#include "vbox_drv.h"
+
+static inline struct vbox_private *vbox_bdev(struct ttm_bo_device *bd)
+{
+ return container_of(bd, struct vbox_private, ttm.bdev);
+}
+
+static void vbox_bo_ttm_destroy(struct ttm_buffer_object *tbo)
+{
+ struct vbox_bo *bo;
+
+ bo = container_of(tbo, struct vbox_bo, bo);
+
+ drm_gem_object_release(&bo->gem);
+ kfree(bo);
+}
+
+static bool vbox_ttm_bo_is_vbox_bo(struct ttm_buffer_object *bo)
+{
+ if (bo->destroy == &vbox_bo_ttm_destroy)
+ return true;
+
+ return false;
+}
+
+static int
+vbox_bo_init_mem_type(struct ttm_bo_device *bdev, u32 type,
+ struct ttm_mem_type_manager *man)
+{
+ switch (type) {
+ case TTM_PL_SYSTEM:
+ man->flags = TTM_MEMTYPE_FLAG_MAPPABLE;
+ man->available_caching = TTM_PL_MASK_CACHING;
+ man->default_caching = TTM_PL_FLAG_CACHED;
+ break;
+ case TTM_PL_VRAM:
+ man->func = &ttm_bo_manager_func;
+ man->flags = TTM_MEMTYPE_FLAG_FIXED | TTM_MEMTYPE_FLAG_MAPPABLE;
+ man->available_caching = TTM_PL_FLAG_UNCACHED | TTM_PL_FLAG_WC;
+ man->default_caching = TTM_PL_FLAG_WC;
+ break;
+ default:
+ DRM_ERROR("Unsupported memory type %u\n", (unsigned int)type);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void
+vbox_bo_evict_flags(struct ttm_buffer_object *bo, struct ttm_placement *pl)
+{
+ struct vbox_bo *vboxbo = vbox_bo(bo);
+
+ if (!vbox_ttm_bo_is_vbox_bo(bo))
+ return;
+
+ vbox_ttm_placement(vboxbo, TTM_PL_FLAG_SYSTEM);
+ *pl = vboxbo->placement;
+}
+
+static int vbox_bo_verify_access(struct ttm_buffer_object *bo,
+ struct file *filp)
+{
+ return 0;
+}
+
+static int vbox_ttm_io_mem_reserve(struct ttm_bo_device *bdev,
+ struct ttm_mem_reg *mem)
+{
+ struct ttm_mem_type_manager *man = &bdev->man[mem->mem_type];
+ struct vbox_private *vbox = vbox_bdev(bdev);
+
+ mem->bus.addr = NULL;
+ mem->bus.offset = 0;
+ mem->bus.size = mem->num_pages << PAGE_SHIFT;
+ mem->bus.base = 0;
+ mem->bus.is_iomem = false;
+ if (!(man->flags & TTM_MEMTYPE_FLAG_MAPPABLE))
+ return -EINVAL;
+ switch (mem->mem_type) {
+ case TTM_PL_SYSTEM:
+ /* system memory */
+ return 0;
+ case TTM_PL_VRAM:
+ mem->bus.offset = mem->start << PAGE_SHIFT;
+ mem->bus.base = pci_resource_start(vbox->ddev.pdev, 0);
+ mem->bus.is_iomem = true;
+ break;
+ default:
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static void vbox_ttm_io_mem_free(struct ttm_bo_device *bdev,
+ struct ttm_mem_reg *mem)
+{
+}
+
+static void vbox_ttm_backend_destroy(struct ttm_tt *tt)
+{
+ ttm_tt_fini(tt);
+ kfree(tt);
+}
+
+static struct ttm_backend_func vbox_tt_backend_func = {
+ .destroy = &vbox_ttm_backend_destroy,
+};
+
+static struct ttm_tt *vbox_ttm_tt_create(struct ttm_buffer_object *bo,
+ u32 page_flags)
+{
+ struct ttm_tt *tt;
+
+ tt = kzalloc(sizeof(*tt), GFP_KERNEL);
+ if (!tt)
+ return NULL;
+
+ tt->func = &vbox_tt_backend_func;
+ if (ttm_tt_init(tt, bo, page_flags)) {
+ kfree(tt);
+ return NULL;
+ }
+
+ return tt;
+}
+
+static struct ttm_bo_driver vbox_bo_driver = {
+ .ttm_tt_create = vbox_ttm_tt_create,
+ .init_mem_type = vbox_bo_init_mem_type,
+ .eviction_valuable = ttm_bo_eviction_valuable,
+ .evict_flags = vbox_bo_evict_flags,
+ .verify_access = vbox_bo_verify_access,
+ .io_mem_reserve = &vbox_ttm_io_mem_reserve,
+ .io_mem_free = &vbox_ttm_io_mem_free,
+};
+
+int vbox_mm_init(struct vbox_private *vbox)
+{
+ int ret;
+ struct drm_device *dev = &vbox->ddev;
+ struct ttm_bo_device *bdev = &vbox->ttm.bdev;
+
+ ret = ttm_bo_device_init(&vbox->ttm.bdev,
+ &vbox_bo_driver,
+ dev->anon_inode->i_mapping,
+ true);
+ if (ret) {
+ DRM_ERROR("Error initialising bo driver; %d\n", ret);
+ return ret;
+ }
+
+ ret = ttm_bo_init_mm(bdev, TTM_PL_VRAM,
+ vbox->available_vram_size >> PAGE_SHIFT);
+ if (ret) {
+ DRM_ERROR("Failed ttm VRAM init: %d\n", ret);
+ goto err_device_release;
+ }
+
+#ifdef DRM_MTRR_WC
+ vbox->fb_mtrr = drm_mtrr_add(pci_resource_start(dev->pdev, 0),
+ pci_resource_len(dev->pdev, 0),
+ DRM_MTRR_WC);
+#else
+ vbox->fb_mtrr = arch_phys_wc_add(pci_resource_start(dev->pdev, 0),
+ pci_resource_len(dev->pdev, 0));
+#endif
+ return 0;
+
+err_device_release:
+ ttm_bo_device_release(&vbox->ttm.bdev);
+ return ret;
+}
+
+void vbox_mm_fini(struct vbox_private *vbox)
+{
+#ifdef DRM_MTRR_WC
+ drm_mtrr_del(vbox->fb_mtrr,
+ pci_resource_start(vbox->ddev.pdev, 0),
+ pci_resource_len(vbox->ddev.pdev, 0), DRM_MTRR_WC);
+#else
+ arch_phys_wc_del(vbox->fb_mtrr);
+#endif
+ ttm_bo_device_release(&vbox->ttm.bdev);
+}
+
+void vbox_ttm_placement(struct vbox_bo *bo, int domain)
+{
+ unsigned int i;
+ u32 c = 0;
+
+ bo->placement.placement = bo->placements;
+ bo->placement.busy_placement = bo->placements;
+
+ if (domain & TTM_PL_FLAG_VRAM)
+ bo->placements[c++].flags =
+ TTM_PL_FLAG_WC | TTM_PL_FLAG_UNCACHED | TTM_PL_FLAG_VRAM;
+ if (domain & TTM_PL_FLAG_SYSTEM)
+ bo->placements[c++].flags =
+ TTM_PL_MASK_CACHING | TTM_PL_FLAG_SYSTEM;
+ if (!c)
+ bo->placements[c++].flags =
+ TTM_PL_MASK_CACHING | TTM_PL_FLAG_SYSTEM;
+
+ bo->placement.num_placement = c;
+ bo->placement.num_busy_placement = c;
+
+ for (i = 0; i < c; ++i) {
+ bo->placements[i].fpfn = 0;
+ bo->placements[i].lpfn = 0;
+ }
+}
+
+int vbox_bo_create(struct vbox_private *vbox, int size, int align,
+ u32 flags, struct vbox_bo **pvboxbo)
+{
+ struct vbox_bo *vboxbo;
+ size_t acc_size;
+ int ret;
+
+ vboxbo = kzalloc(sizeof(*vboxbo), GFP_KERNEL);
+ if (!vboxbo)
+ return -ENOMEM;
+
+ ret = drm_gem_object_init(&vbox->ddev, &vboxbo->gem, size);
+ if (ret)
+ goto err_free_vboxbo;
+
+ vboxbo->bo.bdev = &vbox->ttm.bdev;
+
+ vbox_ttm_placement(vboxbo, TTM_PL_FLAG_VRAM | TTM_PL_FLAG_SYSTEM);
+
+ acc_size = ttm_bo_dma_acc_size(&vbox->ttm.bdev, size,
+ sizeof(struct vbox_bo));
+
+ ret = ttm_bo_init(&vbox->ttm.bdev, &vboxbo->bo, size,
+ ttm_bo_type_device, &vboxbo->placement,
+ align >> PAGE_SHIFT, false, acc_size,
+ NULL, NULL, vbox_bo_ttm_destroy);
+ if (ret)
+ goto err_free_vboxbo;
+
+ *pvboxbo = vboxbo;
+
+ return 0;
+
+err_free_vboxbo:
+ kfree(vboxbo);
+ return ret;
+}
+
+int vbox_bo_pin(struct vbox_bo *bo, u32 pl_flag)
+{
+ struct ttm_operation_ctx ctx = { false, false };
+ int i, ret;
+
+ if (bo->pin_count) {
+ bo->pin_count++;
+ return 0;
+ }
+
+ ret = vbox_bo_reserve(bo, false);
+ if (ret)
+ return ret;
+
+ vbox_ttm_placement(bo, pl_flag);
+
+ for (i = 0; i < bo->placement.num_placement; i++)
+ bo->placements[i].flags |= TTM_PL_FLAG_NO_EVICT;
+
+ ret = ttm_bo_validate(&bo->bo, &bo->placement, &ctx);
+ if (ret == 0)
+ bo->pin_count = 1;
+
+ vbox_bo_unreserve(bo);
+
+ return ret;
+}
+
+int vbox_bo_unpin(struct vbox_bo *bo)
+{
+ struct ttm_operation_ctx ctx = { false, false };
+ int i, ret;
+
+ if (!bo->pin_count) {
+ DRM_ERROR("unpin bad %p\n", bo);
+ return 0;
+ }
+ bo->pin_count--;
+ if (bo->pin_count)
+ return 0;
+
+ ret = vbox_bo_reserve(bo, false);
+ if (ret) {
+ DRM_ERROR("Error %d reserving bo, leaving it pinned\n", ret);
+ return ret;
+ }
+
+ for (i = 0; i < bo->placement.num_placement; i++)
+ bo->placements[i].flags &= ~TTM_PL_FLAG_NO_EVICT;
+
+ ret = ttm_bo_validate(&bo->bo, &bo->placement, &ctx);
+
+ vbox_bo_unreserve(bo);
+
+ return ret;
+}
+
+/*
+ * Move a vbox-owned buffer object to system memory if no one else has it
+ * pinned. The caller must have pinned it previously, and this call will
+ * release the caller's pin.
+ */
+int vbox_bo_push_sysram(struct vbox_bo *bo)
+{
+ struct ttm_operation_ctx ctx = { false, false };
+ int i, ret;
+
+ if (!bo->pin_count) {
+ DRM_ERROR("unpin bad %p\n", bo);
+ return 0;
+ }
+ bo->pin_count--;
+ if (bo->pin_count)
+ return 0;
+
+ if (bo->kmap.virtual) {
+ ttm_bo_kunmap(&bo->kmap);
+ bo->kmap.virtual = NULL;
+ }
+
+ vbox_ttm_placement(bo, TTM_PL_FLAG_SYSTEM);
+
+ for (i = 0; i < bo->placement.num_placement; i++)
+ bo->placements[i].flags |= TTM_PL_FLAG_NO_EVICT;
+
+ ret = ttm_bo_validate(&bo->bo, &bo->placement, &ctx);
+ if (ret) {
+ DRM_ERROR("pushing to VRAM failed\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+int vbox_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+ struct drm_file *file_priv = filp->private_data;
+ struct vbox_private *vbox = file_priv->minor->dev->dev_private;
+
+ return ttm_bo_mmap(filp, vma, &vbox->ttm.bdev);
+}
+
+void *vbox_bo_kmap(struct vbox_bo *bo)
+{
+ int ret;
+
+ if (bo->kmap.virtual)
+ return bo->kmap.virtual;
+
+ ret = ttm_bo_kmap(&bo->bo, 0, bo->bo.num_pages, &bo->kmap);
+ if (ret) {
+ DRM_ERROR("Error kmapping bo: %d\n", ret);
+ return NULL;
+ }
+
+ return bo->kmap.virtual;
+}
+
+void vbox_bo_kunmap(struct vbox_bo *bo)
+{
+ if (bo->kmap.virtual) {
+ ttm_bo_kunmap(&bo->kmap);
+ bo->kmap.virtual = NULL;
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/* Copyright (C) 2006-2016 Oracle Corporation */
+
+#ifndef __VBOXVIDEO_H__
+#define __VBOXVIDEO_H__
+
+#define VBOX_VIDEO_MAX_SCREENS 64
+
+/*
+ * The last 4096 bytes of the guest VRAM contains the generic info for all
+ * DualView chunks: sizes and offsets of chunks. This is filled by miniport.
+ *
+ * Last 4096 bytes of each chunk contain chunk specific data: framebuffer info,
+ * etc. This is used exclusively by the corresponding instance of a display
+ * driver.
+ *
+ * The VRAM layout:
+ * Last 4096 bytes - Adapter information area.
+ * 4096 bytes aligned miniport heap (value specified in the config rouded up).
+ * Slack - what left after dividing the VRAM.
+ * 4096 bytes aligned framebuffers:
+ * last 4096 bytes of each framebuffer is the display information area.
+ *
+ * The Virtual Graphics Adapter information in the guest VRAM is stored by the
+ * guest video driver using structures prepended by VBOXVIDEOINFOHDR.
+ *
+ * When the guest driver writes dword 0 to the VBE_DISPI_INDEX_VBOX_VIDEO
+ * the host starts to process the info. The first element at the start of
+ * the 4096 bytes region should be normally be a LINK that points to
+ * actual information chain. That way the guest driver can have some
+ * fixed layout of the information memory block and just rewrite
+ * the link to point to relevant memory chain.
+ *
+ * The processing stops at the END element.
+ *
+ * The host can access the memory only when the port IO is processed.
+ * All data that will be needed later must be copied from these 4096 bytes.
+ * But other VRAM can be used by host until the mode is disabled.
+ *
+ * The guest driver writes dword 0xffffffff to the VBE_DISPI_INDEX_VBOX_VIDEO
+ * to disable the mode.
+ *
+ * VBE_DISPI_INDEX_VBOX_VIDEO is used to read the configuration information
+ * from the host and issue commands to the host.
+ *
+ * The guest writes the VBE_DISPI_INDEX_VBOX_VIDEO index register, the the
+ * following operations with the VBE data register can be performed:
+ *
+ * Operation Result
+ * write 16 bit value NOP
+ * read 16 bit value count of monitors
+ * write 32 bit value set the vbox cmd value and the cmd processed by the host
+ * read 32 bit value result of the last vbox command is returned
+ */
+
+struct vbva_cmd_hdr {
+ s16 x;
+ s16 y;
+ u16 w;
+ u16 h;
+} __packed;
+
+/*
+ * The VBVA ring buffer is suitable for transferring large (< 2GB) amount of
+ * data. For example big bitmaps which do not fit to the buffer.
+ *
+ * Guest starts writing to the buffer by initializing a record entry in the
+ * records queue. VBVA_F_RECORD_PARTIAL indicates that the record is being
+ * written. As data is written to the ring buffer, the guest increases
+ * free_offset.
+ *
+ * The host reads the records on flushes and processes all completed records.
+ * When host encounters situation when only a partial record presents and
+ * len_and_flags & ~VBVA_F_RECORD_PARTIAL >= VBVA_RING_BUFFER_SIZE -
+ * VBVA_RING_BUFFER_THRESHOLD, the host fetched all record data and updates
+ * data_offset. After that on each flush the host continues fetching the data
+ * until the record is completed.
+ */
+
+#define VBVA_RING_BUFFER_SIZE (4194304 - 1024)
+#define VBVA_RING_BUFFER_THRESHOLD (4096)
+
+#define VBVA_MAX_RECORDS (64)
+
+#define VBVA_F_MODE_ENABLED 0x00000001u
+#define VBVA_F_MODE_VRDP 0x00000002u
+#define VBVA_F_MODE_VRDP_RESET 0x00000004u
+#define VBVA_F_MODE_VRDP_ORDER_MASK 0x00000008u
+
+#define VBVA_F_STATE_PROCESSING 0x00010000u
+
+#define VBVA_F_RECORD_PARTIAL 0x80000000u
+
+struct vbva_record {
+ u32 len_and_flags;
+} __packed;
+
+/*
+ * The minimum HGSMI heap size is PAGE_SIZE (4096 bytes) and is a restriction of
+ * the runtime heapsimple API. Use minimum 2 pages here, because the info area
+ * also may contain other data (for example hgsmi_host_flags structure).
+ */
+#define VBVA_ADAPTER_INFORMATION_SIZE 65536
+#define VBVA_MIN_BUFFER_SIZE 65536
+
+/* The value for port IO to let the adapter to interpret the adapter memory. */
+#define VBOX_VIDEO_DISABLE_ADAPTER_MEMORY 0xFFFFFFFF
+
+/* The value for port IO to let the adapter to interpret the adapter memory. */
+#define VBOX_VIDEO_INTERPRET_ADAPTER_MEMORY 0x00000000
+
+/*
+ * The value for port IO to let the adapter to interpret the display memory.
+ * The display number is encoded in low 16 bits.
+ */
+#define VBOX_VIDEO_INTERPRET_DISPLAY_MEMORY_BASE 0x00010000
+
+struct vbva_host_flags {
+ u32 host_events;
+ u32 supported_orders;
+} __packed;
+
+struct vbva_buffer {
+ struct vbva_host_flags host_flags;
+
+ /* The offset where the data start in the buffer. */
+ u32 data_offset;
+ /* The offset where next data must be placed in the buffer. */
+ u32 free_offset;
+
+ /* The queue of record descriptions. */
+ struct vbva_record records[VBVA_MAX_RECORDS];
+ u32 record_first_index;
+ u32 record_free_index;
+
+ /* Space to leave free when large partial records are transferred. */
+ u32 partial_write_tresh;
+
+ u32 data_len;
+ /* variable size for the rest of the vbva_buffer area in VRAM. */
+ u8 data[0];
+} __packed;
+
+#define VBVA_MAX_RECORD_SIZE (128 * 1024 * 1024)
+
+/* guest->host commands */
+#define VBVA_QUERY_CONF32 1
+#define VBVA_SET_CONF32 2
+#define VBVA_INFO_VIEW 3
+#define VBVA_INFO_HEAP 4
+#define VBVA_FLUSH 5
+#define VBVA_INFO_SCREEN 6
+#define VBVA_ENABLE 7
+#define VBVA_MOUSE_POINTER_SHAPE 8
+/* informs host about HGSMI caps. see vbva_caps below */
+#define VBVA_INFO_CAPS 12
+/* configures scanline, see VBVASCANLINECFG below */
+#define VBVA_SCANLINE_CFG 13
+/* requests scanline info, see VBVASCANLINEINFO below */
+#define VBVA_SCANLINE_INFO 14
+/* inform host about VBVA Command submission */
+#define VBVA_CMDVBVA_SUBMIT 16
+/* inform host about VBVA Command submission */
+#define VBVA_CMDVBVA_FLUSH 17
+/* G->H DMA command */
+#define VBVA_CMDVBVA_CTL 18
+/* Query most recent mode hints sent */
+#define VBVA_QUERY_MODE_HINTS 19
+/*
+ * Report the guest virtual desktop position and size for mapping host and
+ * guest pointer positions.
+ */
+#define VBVA_REPORT_INPUT_MAPPING 20
+/* Report the guest cursor position and query the host position. */
+#define VBVA_CURSOR_POSITION 21
+
+/* host->guest commands */
+#define VBVAHG_EVENT 1
+#define VBVAHG_DISPLAY_CUSTOM 2
+
+/* vbva_conf32::index */
+#define VBOX_VBVA_CONF32_MONITOR_COUNT 0
+#define VBOX_VBVA_CONF32_HOST_HEAP_SIZE 1
+/*
+ * Returns VINF_SUCCESS if the host can report mode hints via VBVA.
+ * Set value to VERR_NOT_SUPPORTED before calling.
+ */
+#define VBOX_VBVA_CONF32_MODE_HINT_REPORTING 2
+/*
+ * Returns VINF_SUCCESS if the host can report guest cursor enabled status via
+ * VBVA. Set value to VERR_NOT_SUPPORTED before calling.
+ */
+#define VBOX_VBVA_CONF32_GUEST_CURSOR_REPORTING 3
+/*
+ * Returns the currently available host cursor capabilities. Available if
+ * VBOX_VBVA_CONF32_GUEST_CURSOR_REPORTING returns success.
+ */
+#define VBOX_VBVA_CONF32_CURSOR_CAPABILITIES 4
+/* Returns the supported flags in vbva_infoscreen.flags. */
+#define VBOX_VBVA_CONF32_SCREEN_FLAGS 5
+/* Returns the max size of VBVA record. */
+#define VBOX_VBVA_CONF32_MAX_RECORD_SIZE 6
+
+struct vbva_conf32 {
+ u32 index;
+ u32 value;
+} __packed;
+
+/* Reserved for historical reasons. */
+#define VBOX_VBVA_CURSOR_CAPABILITY_RESERVED0 BIT(0)
+/*
+ * Guest cursor capability: can the host show a hardware cursor at the host
+ * pointer location?
+ */
+#define VBOX_VBVA_CURSOR_CAPABILITY_HARDWARE BIT(1)
+/* Reserved for historical reasons. */
+#define VBOX_VBVA_CURSOR_CAPABILITY_RESERVED2 BIT(2)
+/* Reserved for historical reasons. Must always be unset. */
+#define VBOX_VBVA_CURSOR_CAPABILITY_RESERVED3 BIT(3)
+/* Reserved for historical reasons. */
+#define VBOX_VBVA_CURSOR_CAPABILITY_RESERVED4 BIT(4)
+/* Reserved for historical reasons. */
+#define VBOX_VBVA_CURSOR_CAPABILITY_RESERVED5 BIT(5)
+
+struct vbva_infoview {
+ /* Index of the screen, assigned by the guest. */
+ u32 view_index;
+
+ /* The screen offset in VRAM, the framebuffer starts here. */
+ u32 view_offset;
+
+ /* The size of the VRAM memory that can be used for the view. */
+ u32 view_size;
+
+ /* The recommended maximum size of the VRAM memory for the screen. */
+ u32 max_screen_size;
+} __packed;
+
+struct vbva_flush {
+ u32 reserved;
+} __packed;
+
+/* vbva_infoscreen.flags */
+#define VBVA_SCREEN_F_NONE 0x0000
+#define VBVA_SCREEN_F_ACTIVE 0x0001
+/*
+ * The virtual monitor has been disabled by the guest and should be removed
+ * by the host and ignored for purposes of pointer position calculation.
+ */
+#define VBVA_SCREEN_F_DISABLED 0x0002
+/*
+ * The virtual monitor has been blanked by the guest and should be blacked
+ * out by the host using width, height, etc values from the vbva_infoscreen
+ * request.
+ */
+#define VBVA_SCREEN_F_BLANK 0x0004
+/*
+ * The virtual monitor has been blanked by the guest and should be blacked
+ * out by the host using the previous mode values for width. height, etc.
+ */
+#define VBVA_SCREEN_F_BLANK2 0x0008
+
+struct vbva_infoscreen {
+ /* Which view contains the screen. */
+ u32 view_index;
+
+ /* Physical X origin relative to the primary screen. */
+ s32 origin_x;
+
+ /* Physical Y origin relative to the primary screen. */
+ s32 origin_y;
+
+ /* Offset of visible framebuffer relative to the framebuffer start. */
+ u32 start_offset;
+
+ /* The scan line size in bytes. */
+ u32 line_size;
+
+ /* Width of the screen. */
+ u32 width;
+
+ /* Height of the screen. */
+ u32 height;
+
+ /* Color depth. */
+ u16 bits_per_pixel;
+
+ /* VBVA_SCREEN_F_* */
+ u16 flags;
+} __packed;
+
+/* vbva_enable.flags */
+#define VBVA_F_NONE 0x00000000
+#define VBVA_F_ENABLE 0x00000001
+#define VBVA_F_DISABLE 0x00000002
+/* extended VBVA to be used with WDDM */
+#define VBVA_F_EXTENDED 0x00000004
+/* vbva offset is absolute VRAM offset */
+#define VBVA_F_ABSOFFSET 0x00000008
+
+struct vbva_enable {
+ u32 flags;
+ u32 offset;
+ s32 result;
+} __packed;
+
+struct vbva_enable_ex {
+ struct vbva_enable base;
+ u32 screen_id;
+} __packed;
+
+struct vbva_mouse_pointer_shape {
+ /* The host result. */
+ s32 result;
+
+ /* VBOX_MOUSE_POINTER_* bit flags. */
+ u32 flags;
+
+ /* X coordinate of the hot spot. */
+ u32 hot_X;
+
+ /* Y coordinate of the hot spot. */
+ u32 hot_y;
+
+ /* Width of the pointer in pixels. */
+ u32 width;
+
+ /* Height of the pointer in scanlines. */
+ u32 height;
+
+ /* Pointer data.
+ *
+ * The data consists of 1 bpp AND mask followed by 32 bpp XOR (color)
+ * mask.
+ *
+ * For pointers without alpha channel the XOR mask pixels are 32 bit
+ * values: (lsb)BGR0(msb). For pointers with alpha channel the XOR mask
+ * consists of (lsb)BGRA(msb) 32 bit values.
+ *
+ * Guest driver must create the AND mask for pointers with alpha chan.,
+ * so if host does not support alpha, the pointer could be displayed as
+ * a normal color pointer. The AND mask can be constructed from alpha
+ * values. For example alpha value >= 0xf0 means bit 0 in the AND mask.
+ *
+ * The AND mask is 1 bpp bitmap with byte aligned scanlines. Size of AND
+ * mask, therefore, is and_len = (width + 7) / 8 * height. The padding
+ * bits at the end of any scanline are undefined.
+ *
+ * The XOR mask follows the AND mask on the next 4 bytes aligned offset:
+ * u8 *xor = and + (and_len + 3) & ~3
+ * Bytes in the gap between the AND and the XOR mask are undefined.
+ * XOR mask scanlines have no gap between them and size of XOR mask is:
+ * xor_len = width * 4 * height.
+ *
+ * Preallocate 4 bytes for accessing actual data as p->data.
+ */
+ u8 data[4];
+} __packed;
+
+/* pointer is visible */
+#define VBOX_MOUSE_POINTER_VISIBLE 0x0001
+/* pointer has alpha channel */
+#define VBOX_MOUSE_POINTER_ALPHA 0x0002
+/* pointerData contains new pointer shape */
+#define VBOX_MOUSE_POINTER_SHAPE 0x0004
+
+/*
+ * The guest driver can handle asynch guest cmd completion by reading the
+ * command offset from io port.
+ */
+#define VBVACAPS_COMPLETEGCMD_BY_IOREAD 0x00000001
+/* the guest driver can handle video adapter IRQs */
+#define VBVACAPS_IRQ 0x00000002
+/* The guest can read video mode hints sent via VBVA. */
+#define VBVACAPS_VIDEO_MODE_HINTS 0x00000004
+/* The guest can switch to a software cursor on demand. */
+#define VBVACAPS_DISABLE_CURSOR_INTEGRATION 0x00000008
+/* The guest does not depend on host handling the VBE registers. */
+#define VBVACAPS_USE_VBVA_ONLY 0x00000010
+
+struct vbva_caps {
+ s32 rc;
+ u32 caps;
+} __packed;
+
+/* Query the most recent mode hints received from the host. */
+struct vbva_query_mode_hints {
+ /* The maximum number of screens to return hints for. */
+ u16 hints_queried_count;
+ /* The size of the mode hint structures directly following this one. */
+ u16 hint_structure_guest_size;
+ /* Return code for the operation. Initialise to VERR_NOT_SUPPORTED. */
+ s32 rc;
+} __packed;
+
+/*
+ * Structure in which a mode hint is returned. The guest allocates an array
+ * of these immediately after the vbva_query_mode_hints structure.
+ * To accommodate future extensions, the vbva_query_mode_hints structure
+ * specifies the size of the vbva_modehint structures allocated by the guest,
+ * and the host only fills out structure elements which fit into that size. The
+ * host should fill any unused members (e.g. dx, dy) or structure space on the
+ * end with ~0. The whole structure can legally be set to ~0 to skip a screen.
+ */
+struct vbva_modehint {
+ u32 magic;
+ u32 cx;
+ u32 cy;
+ u32 bpp; /* Which has never been used... */
+ u32 display;
+ u32 dx; /* X offset into the virtual frame-buffer. */
+ u32 dy; /* Y offset into the virtual frame-buffer. */
+ u32 enabled; /* Not flags. Add new members for new flags. */
+} __packed;
+
+#define VBVAMODEHINT_MAGIC 0x0801add9u
+
+/*
+ * Report the rectangle relative to which absolute pointer events should be
+ * expressed. This information remains valid until the next VBVA resize event
+ * for any screen, at which time it is reset to the bounding rectangle of all
+ * virtual screens and must be re-set.
+ */
+struct vbva_report_input_mapping {
+ s32 x; /* Upper left X co-ordinate relative to the first screen. */
+ s32 y; /* Upper left Y co-ordinate relative to the first screen. */
+ u32 cx; /* Rectangle width. */
+ u32 cy; /* Rectangle height. */
+} __packed;
+
+/*
+ * Report the guest cursor position and query the host one. The host may wish
+ * to use the guest information to re-position its own cursor (though this is
+ * currently unlikely).
+ */
+struct vbva_cursor_position {
+ u32 report_position; /* Are we reporting a position? */
+ u32 x; /* Guest cursor X position */
+ u32 y; /* Guest cursor Y position */
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/* Copyright (C) 2006-2016 Oracle Corporation */
+
+#ifndef __VBOXVIDEO_GUEST_H__
+#define __VBOXVIDEO_GUEST_H__
+
+#include <linux/genalloc.h>
+#include "vboxvideo.h"
+
+/*
+ * Structure grouping the context needed for sending graphics acceleration
+ * information to the host via VBVA. Each screen has its own VBVA buffer.
+ */
+struct vbva_buf_ctx {
+ /* Offset of the buffer in the VRAM section for the screen */
+ u32 buffer_offset;
+ /* Length of the buffer in bytes */
+ u32 buffer_length;
+ /* Set if we wrote to the buffer faster than the host could read it */
+ bool buffer_overflow;
+ /* VBVA record that we are currently preparing for the host, or NULL */
+ struct vbva_record *record;
+ /*
+ * Pointer to the VBVA buffer mapped into the current address space.
+ * Will be NULL if VBVA is not enabled.
+ */
+ struct vbva_buffer *vbva;
+};
+
+int hgsmi_report_flags_location(struct gen_pool *ctx, u32 location);
+int hgsmi_send_caps_info(struct gen_pool *ctx, u32 caps);
+int hgsmi_test_query_conf(struct gen_pool *ctx);
+int hgsmi_query_conf(struct gen_pool *ctx, u32 index, u32 *value_ret);
+int hgsmi_update_pointer_shape(struct gen_pool *ctx, u32 flags,
+ u32 hot_x, u32 hot_y, u32 width, u32 height,
+ u8 *pixels, u32 len);
+int hgsmi_cursor_position(struct gen_pool *ctx, bool report_position,
+ u32 x, u32 y, u32 *x_host, u32 *y_host);
+
+bool vbva_enable(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
+ struct vbva_buffer *vbva, s32 screen);
+void vbva_disable(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
+ s32 screen);
+bool vbva_buffer_begin_update(struct vbva_buf_ctx *vbva_ctx,
+ struct gen_pool *ctx);
+void vbva_buffer_end_update(struct vbva_buf_ctx *vbva_ctx);
+bool vbva_write(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
+ const void *p, u32 len);
+void vbva_setup_buffer_context(struct vbva_buf_ctx *vbva_ctx,
+ u32 buffer_offset, u32 buffer_length);
+
+void hgsmi_process_display_info(struct gen_pool *ctx, u32 display,
+ s32 origin_x, s32 origin_y, u32 start_offset,
+ u32 pitch, u32 width, u32 height,
+ u16 bpp, u16 flags);
+int hgsmi_update_input_mapping(struct gen_pool *ctx, s32 origin_x, s32 origin_y,
+ u32 width, u32 height);
+int hgsmi_get_mode_hints(struct gen_pool *ctx, unsigned int screens,
+ struct vbva_modehint *hints);
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/* Copyright (C) 2006-2016 Oracle Corporation */
+
+#ifndef __VBOXVIDEO_VBE_H__
+#define __VBOXVIDEO_VBE_H__
+
+/* GUEST <-> HOST Communication API */
+
+#define VBE_DISPI_BANK_ADDRESS 0xA0000
+#define VBE_DISPI_BANK_SIZE_KB 64
+
+#define VBE_DISPI_MAX_XRES 16384
+#define VBE_DISPI_MAX_YRES 16384
+#define VBE_DISPI_MAX_BPP 32
+
+#define VBE_DISPI_IOPORT_INDEX 0x01CE
+#define VBE_DISPI_IOPORT_DATA 0x01CF
+
+#define VBE_DISPI_IOPORT_DAC_WRITE_INDEX 0x03C8
+#define VBE_DISPI_IOPORT_DAC_DATA 0x03C9
+
+#define VBE_DISPI_INDEX_ID 0x0
+#define VBE_DISPI_INDEX_XRES 0x1
+#define VBE_DISPI_INDEX_YRES 0x2
+#define VBE_DISPI_INDEX_BPP 0x3
+#define VBE_DISPI_INDEX_ENABLE 0x4
+#define VBE_DISPI_INDEX_BANK 0x5
+#define VBE_DISPI_INDEX_VIRT_WIDTH 0x6
+#define VBE_DISPI_INDEX_VIRT_HEIGHT 0x7
+#define VBE_DISPI_INDEX_X_OFFSET 0x8
+#define VBE_DISPI_INDEX_Y_OFFSET 0x9
+#define VBE_DISPI_INDEX_VBOX_VIDEO 0xa
+#define VBE_DISPI_INDEX_FB_BASE_HI 0xb
+
+#define VBE_DISPI_ID0 0xB0C0
+#define VBE_DISPI_ID1 0xB0C1
+#define VBE_DISPI_ID2 0xB0C2
+#define VBE_DISPI_ID3 0xB0C3
+#define VBE_DISPI_ID4 0xB0C4
+
+#define VBE_DISPI_ID_VBOX_VIDEO 0xBE00
+/* The VBOX interface id. Indicates support for VBVA shared memory interface. */
+#define VBE_DISPI_ID_HGSMI 0xBE01
+#define VBE_DISPI_ID_ANYX 0xBE02
+
+#define VBE_DISPI_DISABLED 0x00
+#define VBE_DISPI_ENABLED 0x01
+#define VBE_DISPI_GETCAPS 0x02
+#define VBE_DISPI_8BIT_DAC 0x20
+
+#define VGA_PORT_HGSMI_HOST 0x3b0
+#define VGA_PORT_HGSMI_GUEST 0x3d0
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: MIT
+/* Copyright (C) 2006-2017 Oracle Corporation */
+
+#include <linux/vbox_err.h>
+#include "vbox_drv.h"
+#include "vboxvideo_guest.h"
+#include "hgsmi_channels.h"
+
+/*
+ * There is a hardware ring buffer in the graphics device video RAM, formerly
+ * in the VBox VMMDev PCI memory space.
+ * All graphics commands go there serialized by vbva_buffer_begin_update.
+ * and vbva_buffer_end_update.
+ *
+ * free_offset is writing position. data_offset is reading position.
+ * free_offset == data_offset means buffer is empty.
+ * There must be always gap between data_offset and free_offset when data
+ * are in the buffer.
+ * Guest only changes free_offset, host changes data_offset.
+ */
+
+static u32 vbva_buffer_available(const struct vbva_buffer *vbva)
+{
+ s32 diff = vbva->data_offset - vbva->free_offset;
+
+ return diff > 0 ? diff : vbva->data_len + diff;
+}
+
+static void vbva_buffer_place_data_at(struct vbva_buf_ctx *vbva_ctx,
+ const void *p, u32 len, u32 offset)
+{
+ struct vbva_buffer *vbva = vbva_ctx->vbva;
+ u32 bytes_till_boundary = vbva->data_len - offset;
+ u8 *dst = &vbva->data[offset];
+ s32 diff = len - bytes_till_boundary;
+
+ if (diff <= 0) {
+ /* Chunk will not cross buffer boundary. */
+ memcpy(dst, p, len);
+ } else {
+ /* Chunk crosses buffer boundary. */
+ memcpy(dst, p, bytes_till_boundary);
+ memcpy(&vbva->data[0], (u8 *)p + bytes_till_boundary, diff);
+ }
+}
+
+static void vbva_buffer_flush(struct gen_pool *ctx)
+{
+ struct vbva_flush *p;
+
+ p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA, VBVA_FLUSH);
+ if (!p)
+ return;
+
+ p->reserved = 0;
+
+ hgsmi_buffer_submit(ctx, p);
+ hgsmi_buffer_free(ctx, p);
+}
+
+bool vbva_write(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
+ const void *p, u32 len)
+{
+ struct vbva_record *record;
+ struct vbva_buffer *vbva;
+ u32 available;
+
+ vbva = vbva_ctx->vbva;
+ record = vbva_ctx->record;
+
+ if (!vbva || vbva_ctx->buffer_overflow ||
+ !record || !(record->len_and_flags & VBVA_F_RECORD_PARTIAL))
+ return false;
+
+ available = vbva_buffer_available(vbva);
+
+ while (len > 0) {
+ u32 chunk = len;
+
+ if (chunk >= available) {
+ vbva_buffer_flush(ctx);
+ available = vbva_buffer_available(vbva);
+ }
+
+ if (chunk >= available) {
+ if (WARN_ON(available <= vbva->partial_write_tresh)) {
+ vbva_ctx->buffer_overflow = true;
+ return false;
+ }
+ chunk = available - vbva->partial_write_tresh;
+ }
+
+ vbva_buffer_place_data_at(vbva_ctx, p, chunk,
+ vbva->free_offset);
+
+ vbva->free_offset = (vbva->free_offset + chunk) %
+ vbva->data_len;
+ record->len_and_flags += chunk;
+ available -= chunk;
+ len -= chunk;
+ p += chunk;
+ }
+
+ return true;
+}
+
+static bool vbva_inform_host(struct vbva_buf_ctx *vbva_ctx,
+ struct gen_pool *ctx, s32 screen, bool enable)
+{
+ struct vbva_enable_ex *p;
+ bool ret;
+
+ p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA, VBVA_ENABLE);
+ if (!p)
+ return false;
+
+ p->base.flags = enable ? VBVA_F_ENABLE : VBVA_F_DISABLE;
+ p->base.offset = vbva_ctx->buffer_offset;
+ p->base.result = VERR_NOT_SUPPORTED;
+ if (screen >= 0) {
+ p->base.flags |= VBVA_F_EXTENDED | VBVA_F_ABSOFFSET;
+ p->screen_id = screen;
+ }
+
+ hgsmi_buffer_submit(ctx, p);
+
+ if (enable)
+ ret = p->base.result >= 0;
+ else
+ ret = true;
+
+ hgsmi_buffer_free(ctx, p);
+
+ return ret;
+}
+
+bool vbva_enable(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
+ struct vbva_buffer *vbva, s32 screen)
+{
+ bool ret = false;
+
+ memset(vbva, 0, sizeof(*vbva));
+ vbva->partial_write_tresh = 256;
+ vbva->data_len = vbva_ctx->buffer_length - sizeof(struct vbva_buffer);
+ vbva_ctx->vbva = vbva;
+
+ ret = vbva_inform_host(vbva_ctx, ctx, screen, true);
+ if (!ret)
+ vbva_disable(vbva_ctx, ctx, screen);
+
+ return ret;
+}
+
+void vbva_disable(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
+ s32 screen)
+{
+ vbva_ctx->buffer_overflow = false;
+ vbva_ctx->record = NULL;
+ vbva_ctx->vbva = NULL;
+
+ vbva_inform_host(vbva_ctx, ctx, screen, false);
+}
+
+bool vbva_buffer_begin_update(struct vbva_buf_ctx *vbva_ctx,
+ struct gen_pool *ctx)
+{
+ struct vbva_record *record;
+ u32 next;
+
+ if (!vbva_ctx->vbva ||
+ !(vbva_ctx->vbva->host_flags.host_events & VBVA_F_MODE_ENABLED))
+ return false;
+
+ WARN_ON(vbva_ctx->buffer_overflow || vbva_ctx->record);
+
+ next = (vbva_ctx->vbva->record_free_index + 1) % VBVA_MAX_RECORDS;
+
+ /* Flush if all slots in the records queue are used */
+ if (next == vbva_ctx->vbva->record_first_index)
+ vbva_buffer_flush(ctx);
+
+ /* If even after flush there is no place then fail the request */
+ if (next == vbva_ctx->vbva->record_first_index)
+ return false;
+
+ record = &vbva_ctx->vbva->records[vbva_ctx->vbva->record_free_index];
+ record->len_and_flags = VBVA_F_RECORD_PARTIAL;
+ vbva_ctx->vbva->record_free_index = next;
+ /* Remember which record we are using. */
+ vbva_ctx->record = record;
+
+ return true;
+}
+
+void vbva_buffer_end_update(struct vbva_buf_ctx *vbva_ctx)
+{
+ struct vbva_record *record = vbva_ctx->record;
+
+ WARN_ON(!vbva_ctx->vbva || !record ||
+ !(record->len_and_flags & VBVA_F_RECORD_PARTIAL));
+
+ /* Mark the record completed. */
+ record->len_and_flags &= ~VBVA_F_RECORD_PARTIAL;
+
+ vbva_ctx->buffer_overflow = false;
+ vbva_ctx->record = NULL;
+}
+
+void vbva_setup_buffer_context(struct vbva_buf_ctx *vbva_ctx,
+ u32 buffer_offset, u32 buffer_length)
+{
+ vbva_ctx->buffer_offset = buffer_offset;
+ vbva_ctx->buffer_length = buffer_length;
+}
return label >= VC4_BO_TYPE_COUNT;
}
-static void vc4_bo_stats_dump(struct vc4_dev *vc4)
+static void vc4_bo_stats_print(struct drm_printer *p, struct vc4_dev *vc4)
{
int i;
if (!vc4->bo_labels[i].num_allocated)
continue;
- DRM_INFO("%30s: %6dkb BOs (%d)\n",
- vc4->bo_labels[i].name,
- vc4->bo_labels[i].size_allocated / 1024,
- vc4->bo_labels[i].num_allocated);
+ drm_printf(p, "%30s: %6dkb BOs (%d)\n",
+ vc4->bo_labels[i].name,
+ vc4->bo_labels[i].size_allocated / 1024,
+ vc4->bo_labels[i].num_allocated);
}
mutex_lock(&vc4->purgeable.lock);
if (vc4->purgeable.num)
- DRM_INFO("%30s: %6zdkb BOs (%d)\n", "userspace BO cache",
- vc4->purgeable.size / 1024, vc4->purgeable.num);
+ drm_printf(p, "%30s: %6zdkb BOs (%d)\n", "userspace BO cache",
+ vc4->purgeable.size / 1024, vc4->purgeable.num);
if (vc4->purgeable.purged_num)
- DRM_INFO("%30s: %6zdkb BOs (%d)\n", "total purged BO",
- vc4->purgeable.purged_size / 1024,
- vc4->purgeable.purged_num);
+ drm_printf(p, "%30s: %6zdkb BOs (%d)\n", "total purged BO",
+ vc4->purgeable.purged_size / 1024,
+ vc4->purgeable.purged_num);
mutex_unlock(&vc4->purgeable.lock);
}
-#ifdef CONFIG_DEBUG_FS
-int vc4_bo_stats_debugfs(struct seq_file *m, void *unused)
+static int vc4_bo_stats_debugfs(struct seq_file *m, void *unused)
{
struct drm_info_node *node = (struct drm_info_node *)m->private;
struct drm_device *dev = node->minor->dev;
struct vc4_dev *vc4 = to_vc4_dev(dev);
- int i;
-
- mutex_lock(&vc4->bo_lock);
- for (i = 0; i < vc4->num_labels; i++) {
- if (!vc4->bo_labels[i].num_allocated)
- continue;
-
- seq_printf(m, "%30s: %6dkb BOs (%d)\n",
- vc4->bo_labels[i].name,
- vc4->bo_labels[i].size_allocated / 1024,
- vc4->bo_labels[i].num_allocated);
- }
- mutex_unlock(&vc4->bo_lock);
-
- mutex_lock(&vc4->purgeable.lock);
- if (vc4->purgeable.num)
- seq_printf(m, "%30s: %6zdkb BOs (%d)\n", "userspace BO cache",
- vc4->purgeable.size / 1024, vc4->purgeable.num);
+ struct drm_printer p = drm_seq_file_printer(m);
- if (vc4->purgeable.purged_num)
- seq_printf(m, "%30s: %6zdkb BOs (%d)\n", "total purged BO",
- vc4->purgeable.purged_size / 1024,
- vc4->purgeable.purged_num);
- mutex_unlock(&vc4->purgeable.lock);
+ vc4_bo_stats_print(&p, vc4);
return 0;
}
-#endif
/* Takes ownership of *name and returns the appropriate slot for it in
* the bo_labels[] array, extending it as necessary.
bo->validated_shader = NULL;
}
- reservation_object_fini(&bo->_resv);
-
drm_gem_cma_free_object(obj);
}
vc4->bo_labels[VC4_BO_TYPE_KERNEL].num_allocated++;
vc4->bo_labels[VC4_BO_TYPE_KERNEL].size_allocated += size;
mutex_unlock(&vc4->bo_lock);
- bo->resv = &bo->_resv;
- reservation_object_init(bo->resv);
return &bo->base.base;
}
}
if (IS_ERR(cma_obj)) {
+ struct drm_printer p = drm_info_printer(vc4->dev->dev);
DRM_ERROR("Failed to allocate from CMA:\n");
- vc4_bo_stats_dump(vc4);
+ vc4_bo_stats_print(&p, vc4);
return ERR_PTR(-ENOMEM);
}
bo = to_vc4_bo(&cma_obj->base);
schedule_work(&vc4->bo_cache.time_work);
}
-struct reservation_object *vc4_prime_res_obj(struct drm_gem_object *obj)
-{
- struct vc4_bo *bo = to_vc4_bo(obj);
-
- return bo->resv;
-}
-
struct dma_buf *
vc4_prime_export(struct drm_device *dev, struct drm_gem_object *obj, int flags)
{
struct sg_table *sgt)
{
struct drm_gem_object *obj;
- struct vc4_bo *bo;
obj = drm_gem_cma_prime_import_sg_table(dev, attach, sgt);
if (IS_ERR(obj))
return obj;
- bo = to_vc4_bo(obj);
- bo->resv = attach->dmabuf->resv;
+ obj->resv = attach->dmabuf->resv;
return obj;
}
mutex_init(&vc4->bo_lock);
+ vc4_debugfs_add_file(dev, "bo_stats", vc4_bo_stats_debugfs, NULL);
+
INIT_LIST_HEAD(&vc4->bo_cache.time_list);
INIT_WORK(&vc4->bo_cache.time_work, vc4_bo_cache_time_work);
#include <drm/drm_atomic.h>
#include <drm/drm_atomic_helper.h>
#include <drm/drm_atomic_uapi.h>
+#include <drm/drm_print.h>
#include <drm/drm_probe_helper.h>
#include <linux/clk.h>
#include <drm/drm_fb_cma_helper.h>
#define CRTC_WRITE(offset, val) writel(val, vc4_crtc->regs + (offset))
#define CRTC_READ(offset) readl(vc4_crtc->regs + (offset))
-#define CRTC_REG(reg) { reg, #reg }
-static const struct {
- u32 reg;
- const char *name;
-} crtc_regs[] = {
- CRTC_REG(PV_CONTROL),
- CRTC_REG(PV_V_CONTROL),
- CRTC_REG(PV_VSYNCD_EVEN),
- CRTC_REG(PV_HORZA),
- CRTC_REG(PV_HORZB),
- CRTC_REG(PV_VERTA),
- CRTC_REG(PV_VERTB),
- CRTC_REG(PV_VERTA_EVEN),
- CRTC_REG(PV_VERTB_EVEN),
- CRTC_REG(PV_INTEN),
- CRTC_REG(PV_INTSTAT),
- CRTC_REG(PV_STAT),
- CRTC_REG(PV_HACT_ACT),
+static const struct debugfs_reg32 crtc_regs[] = {
+ VC4_REG32(PV_CONTROL),
+ VC4_REG32(PV_V_CONTROL),
+ VC4_REG32(PV_VSYNCD_EVEN),
+ VC4_REG32(PV_HORZA),
+ VC4_REG32(PV_HORZB),
+ VC4_REG32(PV_VERTA),
+ VC4_REG32(PV_VERTB),
+ VC4_REG32(PV_VERTA_EVEN),
+ VC4_REG32(PV_VERTB_EVEN),
+ VC4_REG32(PV_INTEN),
+ VC4_REG32(PV_INTSTAT),
+ VC4_REG32(PV_STAT),
+ VC4_REG32(PV_HACT_ACT),
};
-static void vc4_crtc_dump_regs(struct vc4_crtc *vc4_crtc)
-{
- int i;
-
- for (i = 0; i < ARRAY_SIZE(crtc_regs); i++) {
- DRM_INFO("0x%04x (%s): 0x%08x\n",
- crtc_regs[i].reg, crtc_regs[i].name,
- CRTC_READ(crtc_regs[i].reg));
- }
-}
-
-#ifdef CONFIG_DEBUG_FS
-int vc4_crtc_debugfs_regs(struct seq_file *m, void *unused)
-{
- struct drm_info_node *node = (struct drm_info_node *)m->private;
- struct drm_device *dev = node->minor->dev;
- int crtc_index = (uintptr_t)node->info_ent->data;
- struct drm_crtc *crtc;
- struct vc4_crtc *vc4_crtc;
- int i;
-
- i = 0;
- list_for_each_entry(crtc, &dev->mode_config.crtc_list, head) {
- if (i == crtc_index)
- break;
- i++;
- }
- if (!crtc)
- return 0;
- vc4_crtc = to_vc4_crtc(crtc);
-
- for (i = 0; i < ARRAY_SIZE(crtc_regs); i++) {
- seq_printf(m, "%s (0x%04x): 0x%08x\n",
- crtc_regs[i].name, crtc_regs[i].reg,
- CRTC_READ(crtc_regs[i].reg));
- }
-
- return 0;
-}
-#endif
-
bool vc4_crtc_get_scanoutpos(struct drm_device *dev, unsigned int crtc_id,
bool in_vblank_irq, int *vpos, int *hpos,
ktime_t *stime, ktime_t *etime,
bool debug_dump_regs = false;
if (debug_dump_regs) {
- DRM_INFO("CRTC %d regs before:\n", drm_crtc_index(crtc));
- vc4_crtc_dump_regs(vc4_crtc);
+ struct drm_printer p = drm_info_printer(&vc4_crtc->pdev->dev);
+ dev_info(&vc4_crtc->pdev->dev, "CRTC %d regs before:\n",
+ drm_crtc_index(crtc));
+ drm_print_regset32(&p, &vc4_crtc->regset);
}
if (vc4_crtc->channel == 2) {
vc4_crtc_lut_load(crtc);
if (debug_dump_regs) {
- DRM_INFO("CRTC %d regs after:\n", drm_crtc_index(crtc));
- vc4_crtc_dump_regs(vc4_crtc);
+ struct drm_printer p = drm_info_printer(&vc4_crtc->pdev->dev);
+ dev_info(&vc4_crtc->pdev->dev, "CRTC %d regs after:\n",
+ drm_crtc_index(crtc));
+ drm_print_regset32(&p, &vc4_crtc->regset);
}
}
drm_crtc_send_vblank_event(crtc, vc4_crtc->event);
vc4_crtc->event = NULL;
drm_crtc_vblank_put(crtc);
+
+ /* Wait for the page flip to unmask the underrun to ensure that
+ * the display list was updated by the hardware. Before that
+ * happens, the HVS will be using the previous display list with
+ * the CRTC and encoder already reconfigured, leading to
+ * underruns. This can be seen when reconfiguring the CRTC.
+ */
+ vc4_hvs_unmask_underrun(dev, vc4_crtc->channel);
}
spin_unlock_irqrestore(&dev->event_lock, flags);
}
static const struct vc4_crtc_data pv0_data = {
.hvs_channel = 0,
+ .debugfs_name = "crtc0_regs",
.encoder_types = {
[PV_CONTROL_CLK_SELECT_DSI] = VC4_ENCODER_TYPE_DSI0,
[PV_CONTROL_CLK_SELECT_DPI_SMI_HDMI] = VC4_ENCODER_TYPE_DPI,
static const struct vc4_crtc_data pv1_data = {
.hvs_channel = 2,
+ .debugfs_name = "crtc1_regs",
.encoder_types = {
[PV_CONTROL_CLK_SELECT_DSI] = VC4_ENCODER_TYPE_DSI1,
[PV_CONTROL_CLK_SELECT_DPI_SMI_HDMI] = VC4_ENCODER_TYPE_SMI,
static const struct vc4_crtc_data pv2_data = {
.hvs_channel = 1,
+ .debugfs_name = "crtc2_regs",
.encoder_types = {
[PV_CONTROL_CLK_SELECT_DPI_SMI_HDMI] = VC4_ENCODER_TYPE_HDMI,
[PV_CONTROL_CLK_SELECT_VEC] = VC4_ENCODER_TYPE_VEC,
if (!match)
return -ENODEV;
vc4_crtc->data = match->data;
+ vc4_crtc->pdev = pdev;
vc4_crtc->regs = vc4_ioremap_regs(pdev, 0);
if (IS_ERR(vc4_crtc->regs))
return PTR_ERR(vc4_crtc->regs);
+ vc4_crtc->regset.base = vc4_crtc->regs;
+ vc4_crtc->regset.regs = crtc_regs;
+ vc4_crtc->regset.nregs = ARRAY_SIZE(crtc_regs);
+
/* For now, we create just the primary and the legacy cursor
* planes. We should be able to stack more planes on easily,
* but to do that we would need to compute the bandwidth
platform_set_drvdata(pdev, vc4_crtc);
+ vc4_debugfs_add_regset32(drm, vc4_crtc->data->debugfs_name,
+ &vc4_crtc->regset);
+
return 0;
err_destroy_planes:
#include "vc4_drv.h"
#include "vc4_regs.h"
-static const struct drm_info_list vc4_debugfs_list[] = {
- {"bo_stats", vc4_bo_stats_debugfs, 0},
- {"dpi_regs", vc4_dpi_debugfs_regs, 0},
- {"dsi1_regs", vc4_dsi_debugfs_regs, 0, (void *)(uintptr_t)1},
- {"hdmi_regs", vc4_hdmi_debugfs_regs, 0},
- {"vec_regs", vc4_vec_debugfs_regs, 0},
- {"txp_regs", vc4_txp_debugfs_regs, 0},
- {"hvs_regs", vc4_hvs_debugfs_regs, 0},
- {"crtc0_regs", vc4_crtc_debugfs_regs, 0, (void *)(uintptr_t)0},
- {"crtc1_regs", vc4_crtc_debugfs_regs, 0, (void *)(uintptr_t)1},
- {"crtc2_regs", vc4_crtc_debugfs_regs, 0, (void *)(uintptr_t)2},
- {"v3d_ident", vc4_v3d_debugfs_ident, 0},
- {"v3d_regs", vc4_v3d_debugfs_regs, 0},
+struct vc4_debugfs_info_entry {
+ struct list_head link;
+ struct drm_info_list info;
};
-#define VC4_DEBUGFS_ENTRIES ARRAY_SIZE(vc4_debugfs_list)
-
+/**
+ * Called at drm_dev_register() time on each of the minors registered
+ * by the DRM device, to attach the debugfs files.
+ */
int
vc4_debugfs_init(struct drm_minor *minor)
{
- return drm_debugfs_create_files(vc4_debugfs_list, VC4_DEBUGFS_ENTRIES,
- minor->debugfs_root, minor);
+ struct vc4_dev *vc4 = to_vc4_dev(minor->dev);
+ struct vc4_debugfs_info_entry *entry;
+ struct dentry *dentry;
+
+ dentry = debugfs_create_bool("hvs_load_tracker", S_IRUGO | S_IWUSR,
+ minor->debugfs_root,
+ &vc4->load_tracker_enabled);
+ if (!dentry)
+ return -ENOMEM;
+
+ list_for_each_entry(entry, &vc4->debugfs_list, link) {
+ int ret = drm_debugfs_create_files(&entry->info, 1,
+ minor->debugfs_root, minor);
+
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int vc4_debugfs_regset32(struct seq_file *m, void *unused)
+{
+ struct drm_info_node *node = (struct drm_info_node *)m->private;
+ struct debugfs_regset32 *regset = node->info_ent->data;
+ struct drm_printer p = drm_seq_file_printer(m);
+
+ drm_print_regset32(&p, regset);
+
+ return 0;
+}
+
+/**
+ * Registers a debugfs file with a callback function for a vc4 component.
+ *
+ * This is like drm_debugfs_create_files(), but that can only be
+ * called a given DRM minor, while the various VC4 components want to
+ * register their debugfs files during the component bind process. We
+ * track the request and delay it to be called on each minor during
+ * vc4_debugfs_init().
+ */
+void vc4_debugfs_add_file(struct drm_device *dev,
+ const char *name,
+ int (*show)(struct seq_file*, void*),
+ void *data)
+{
+ struct vc4_dev *vc4 = to_vc4_dev(dev);
+
+ struct vc4_debugfs_info_entry *entry =
+ devm_kzalloc(dev->dev, sizeof(*entry), GFP_KERNEL);
+
+ if (!entry)
+ return;
+
+ entry->info.name = name;
+ entry->info.show = show;
+ entry->info.data = data;
+
+ list_add(&entry->link, &vc4->debugfs_list);
+}
+
+void vc4_debugfs_add_regset32(struct drm_device *drm,
+ const char *name,
+ struct debugfs_regset32 *regset)
+{
+ vc4_debugfs_add_file(drm, name, vc4_debugfs_regset32, regset);
}
struct clk *pixel_clock;
struct clk *core_clock;
+
+ struct debugfs_regset32 regset;
};
#define DPI_READ(offset) readl(dpi->regs + (offset))
return container_of(encoder, struct vc4_dpi_encoder, base.base);
}
-#define DPI_REG(reg) { reg, #reg }
-static const struct {
- u32 reg;
- const char *name;
-} dpi_regs[] = {
- DPI_REG(DPI_C),
- DPI_REG(DPI_ID),
+static const struct debugfs_reg32 dpi_regs[] = {
+ VC4_REG32(DPI_C),
+ VC4_REG32(DPI_ID),
};
-#ifdef CONFIG_DEBUG_FS
-int vc4_dpi_debugfs_regs(struct seq_file *m, void *unused)
-{
- struct drm_info_node *node = (struct drm_info_node *)m->private;
- struct drm_device *dev = node->minor->dev;
- struct vc4_dev *vc4 = to_vc4_dev(dev);
- struct vc4_dpi *dpi = vc4->dpi;
- int i;
-
- if (!dpi)
- return 0;
-
- for (i = 0; i < ARRAY_SIZE(dpi_regs); i++) {
- seq_printf(m, "%s (0x%04x): 0x%08x\n",
- dpi_regs[i].name, dpi_regs[i].reg,
- DPI_READ(dpi_regs[i].reg));
- }
-
- return 0;
-}
-#endif
-
static const struct drm_encoder_funcs vc4_dpi_encoder_funcs = {
.destroy = drm_encoder_cleanup,
};
dpi->regs = vc4_ioremap_regs(pdev, 0);
if (IS_ERR(dpi->regs))
return PTR_ERR(dpi->regs);
+ dpi->regset.base = dpi->regs;
+ dpi->regset.regs = dpi_regs;
+ dpi->regset.nregs = ARRAY_SIZE(dpi_regs);
if (DPI_READ(DPI_ID) != DPI_ID_VALUE) {
dev_err(dev, "Port returned 0x%08x for ID instead of 0x%08x\n",
vc4->dpi = dpi;
+ vc4_debugfs_add_regset32(drm, "dpi_regs", &dpi->regset);
+
return 0;
err_destroy_encoder:
if (args->pad != 0)
return -EINVAL;
+ if (!vc4->v3d)
+ return -ENODEV;
+
switch (args->param) {
case DRM_VC4_PARAM_V3D_IDENT0:
- ret = pm_runtime_get_sync(&vc4->v3d->pdev->dev);
- if (ret < 0)
+ ret = vc4_v3d_pm_get(vc4);
+ if (ret)
return ret;
args->value = V3D_READ(V3D_IDENT0);
- pm_runtime_mark_last_busy(&vc4->v3d->pdev->dev);
- pm_runtime_put_autosuspend(&vc4->v3d->pdev->dev);
+ vc4_v3d_pm_put(vc4);
break;
case DRM_VC4_PARAM_V3D_IDENT1:
- ret = pm_runtime_get_sync(&vc4->v3d->pdev->dev);
- if (ret < 0)
+ ret = vc4_v3d_pm_get(vc4);
+ if (ret)
return ret;
args->value = V3D_READ(V3D_IDENT1);
- pm_runtime_mark_last_busy(&vc4->v3d->pdev->dev);
- pm_runtime_put_autosuspend(&vc4->v3d->pdev->dev);
+ vc4_v3d_pm_put(vc4);
break;
case DRM_VC4_PARAM_V3D_IDENT2:
- ret = pm_runtime_get_sync(&vc4->v3d->pdev->dev);
- if (ret < 0)
+ ret = vc4_v3d_pm_get(vc4);
+ if (ret)
return ret;
args->value = V3D_READ(V3D_IDENT2);
- pm_runtime_mark_last_busy(&vc4->v3d->pdev->dev);
- pm_runtime_put_autosuspend(&vc4->v3d->pdev->dev);
+ vc4_v3d_pm_put(vc4);
break;
case DRM_VC4_PARAM_SUPPORTS_BRANCHES:
case DRM_VC4_PARAM_SUPPORTS_ETC1:
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
.gem_prime_import = drm_gem_prime_import,
.gem_prime_export = vc4_prime_export,
- .gem_prime_res_obj = vc4_prime_res_obj,
.gem_prime_get_sg_table = drm_gem_cma_prime_get_sg_table,
.gem_prime_import_sg_table = vc4_prime_import_sg_table,
.gem_prime_vmap = vc4_prime_vmap,
struct platform_device *pdev = to_platform_device(dev);
struct drm_device *drm;
struct vc4_dev *vc4;
+ struct device_node *node;
int ret = 0;
dev->coherent_dma_mask = DMA_BIT_MASK(32);
if (!vc4)
return -ENOMEM;
+ /* If VC4 V3D is missing, don't advertise render nodes. */
+ node = of_find_matching_node_and_match(NULL, vc4_v3d_dt_match, NULL);
+ if (!node || !of_device_is_available(node))
+ vc4_drm_driver.driver_features &= ~DRIVER_RENDER;
+ of_node_put(node);
+
drm = drm_dev_alloc(&vc4_drm_driver, dev);
if (IS_ERR(drm))
return PTR_ERR(drm);
platform_set_drvdata(pdev, drm);
vc4->dev = drm;
drm->dev_private = vc4;
+ INIT_LIST_HEAD(&vc4->debugfs_list);
ret = vc4_bo_cache_init(drm);
if (ret)
drm_fb_helper_remove_conflicting_framebuffers(NULL, "vc4drmfb", false);
- ret = drm_dev_register(drm, 0);
+ ret = vc4_kms_load(drm);
if (ret < 0)
goto unbind_all;
- vc4_kms_load(drm);
+ ret = drm_dev_register(drm, 0);
+ if (ret < 0)
+ goto unbind_all;
- drm_fbdev_generic_setup(drm, 32);
+ drm_fbdev_generic_setup(drm, 16);
return 0;
drm_mode_config_cleanup(drm);
+ drm_atomic_private_obj_fini(&vc4->load_tracker);
drm_atomic_private_obj_fini(&vc4->ctm_manager);
drm_dev_put(drm);
*/
#include <linux/mm_types.h>
-#include <linux/reservation.h>
#include <drm/drmP.h>
#include <drm/drm_util.h>
#include <drm/drm_encoder.h>
/* Bitmask of the current bin_alloc used for overflow memory. */
uint32_t bin_alloc_overflow;
+ /* Incremented when an underrun error happened after an atomic commit.
+ * This is particularly useful to detect when a specific modeset is too
+ * demanding in term of memory or HVS bandwidth which is hard to guess
+ * at atomic check time.
+ */
+ atomic_t underrun;
+
struct work_struct overflow_mem_work;
int power_refcount;
+ /* Set to true when the load tracker is active. */
+ bool load_tracker_enabled;
+
/* Mutex controlling the power refcount. */
struct mutex power_lock;
struct drm_modeset_lock ctm_state_lock;
struct drm_private_obj ctm_manager;
+ struct drm_private_obj load_tracker;
+
+ /* List of vc4_debugfs_info_entry for adding to debugfs once
+ * the minor is available (after drm_dev_register()).
+ */
+ struct list_head debugfs_list;
};
static inline struct vc4_dev *
*/
struct vc4_validated_shader_info *validated_shader;
- /* normally (resv == &_resv) except for imported bo's */
- struct reservation_object *resv;
- struct reservation_object _resv;
-
/* One of enum vc4_kernel_bo_type, or VC4_BO_TYPE_COUNT + i
* for user-allocated labels.
*/
struct platform_device *pdev;
void __iomem *regs;
struct clk *clk;
+ struct debugfs_regset32 regset;
};
struct vc4_hvs {
spinlock_t mm_lock;
struct drm_mm_node mitchell_netravali_filter;
+ struct debugfs_regset32 regset;
};
struct vc4_plane {
* when async update is not possible.
*/
bool dlist_initialized;
+
+ /* Load of this plane on the HVS block. The load is expressed in HVS
+ * cycles/sec.
+ */
+ u64 hvs_load;
+
+ /* Memory bandwidth needed for this plane. This is expressed in
+ * bytes/sec.
+ */
+ u64 membus_load;
};
static inline struct vc4_plane_state *
int hvs_channel;
enum vc4_encoder_type encoder_types[4];
+ const char *debugfs_name;
};
struct vc4_crtc {
struct drm_crtc base;
+ struct platform_device *pdev;
const struct vc4_crtc_data *data;
void __iomem *regs;
u32 cob_size;
struct drm_pending_vblank_event *event;
+
+ struct debugfs_regset32 regset;
};
static inline struct vc4_crtc *
#define HVS_READ(offset) readl(vc4->hvs->regs + offset)
#define HVS_WRITE(offset, val) writel(val, vc4->hvs->regs + offset)
+#define VC4_REG32(reg) { .name = #reg, .offset = reg }
+
struct vc4_exec_info {
/* Sequence number for this bin/render job. */
uint64_t seqno;
struct drm_file *file_priv);
vm_fault_t vc4_fault(struct vm_fault *vmf);
int vc4_mmap(struct file *filp, struct vm_area_struct *vma);
-struct reservation_object *vc4_prime_res_obj(struct drm_gem_object *obj);
int vc4_prime_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma);
struct drm_gem_object *vc4_prime_import_sg_table(struct drm_device *dev,
struct dma_buf_attachment *attach,
void *vc4_prime_vmap(struct drm_gem_object *obj);
int vc4_bo_cache_init(struct drm_device *dev);
void vc4_bo_cache_destroy(struct drm_device *dev);
-int vc4_bo_stats_debugfs(struct seq_file *m, void *arg);
int vc4_bo_inc_usecnt(struct vc4_bo *bo);
void vc4_bo_dec_usecnt(struct vc4_bo *bo);
void vc4_bo_add_to_purgeable_pool(struct vc4_bo *bo);
/* vc4_crtc.c */
extern struct platform_driver vc4_crtc_driver;
-int vc4_crtc_debugfs_regs(struct seq_file *m, void *arg);
bool vc4_crtc_get_scanoutpos(struct drm_device *dev, unsigned int crtc_id,
bool in_vblank_irq, int *vpos, int *hpos,
ktime_t *stime, ktime_t *etime,
/* vc4_debugfs.c */
int vc4_debugfs_init(struct drm_minor *minor);
+#ifdef CONFIG_DEBUG_FS
+void vc4_debugfs_add_file(struct drm_device *drm,
+ const char *filename,
+ int (*show)(struct seq_file*, void*),
+ void *data);
+void vc4_debugfs_add_regset32(struct drm_device *drm,
+ const char *filename,
+ struct debugfs_regset32 *regset);
+#else
+static inline void vc4_debugfs_add_file(struct drm_device *drm,
+ const char *filename,
+ int (*show)(struct seq_file*, void*),
+ void *data)
+{
+}
+
+static inline void vc4_debugfs_add_regset32(struct drm_device *drm,
+ const char *filename,
+ struct debugfs_regset32 *regset)
+{
+}
+#endif
/* vc4_drv.c */
void __iomem *vc4_ioremap_regs(struct platform_device *dev, int index);
/* vc4_dpi.c */
extern struct platform_driver vc4_dpi_driver;
-int vc4_dpi_debugfs_regs(struct seq_file *m, void *unused);
/* vc4_dsi.c */
extern struct platform_driver vc4_dsi_driver;
-int vc4_dsi_debugfs_regs(struct seq_file *m, void *unused);
/* vc4_fence.c */
extern const struct dma_fence_ops vc4_fence_ops;
/* vc4_hdmi.c */
extern struct platform_driver vc4_hdmi_driver;
-int vc4_hdmi_debugfs_regs(struct seq_file *m, void *unused);
/* vc4_vec.c */
extern struct platform_driver vc4_vec_driver;
-int vc4_vec_debugfs_regs(struct seq_file *m, void *unused);
/* vc4_txp.c */
extern struct platform_driver vc4_txp_driver;
-int vc4_txp_debugfs_regs(struct seq_file *m, void *unused);
/* vc4_irq.c */
irqreturn_t vc4_irq(int irq, void *arg);
/* vc4_hvs.c */
extern struct platform_driver vc4_hvs_driver;
void vc4_hvs_dump_state(struct drm_device *dev);
-int vc4_hvs_debugfs_regs(struct seq_file *m, void *unused);
+void vc4_hvs_unmask_underrun(struct drm_device *dev, int channel);
+void vc4_hvs_mask_underrun(struct drm_device *dev, int channel);
/* vc4_kms.c */
int vc4_kms_load(struct drm_device *dev);
/* vc4_v3d.c */
extern struct platform_driver vc4_v3d_driver;
-int vc4_v3d_debugfs_ident(struct seq_file *m, void *unused);
-int vc4_v3d_debugfs_regs(struct seq_file *m, void *unused);
+extern const struct of_device_id vc4_v3d_dt_match[];
int vc4_v3d_get_bin_slot(struct vc4_dev *vc4);
+int vc4_v3d_pm_get(struct vc4_dev *vc4);
+void vc4_v3d_pm_put(struct vc4_dev *vc4);
/* vc4_validate.c */
int
struct completion xfer_completion;
int xfer_result;
+
+ struct debugfs_regset32 regset;
};
#define host_to_dsi(host) container_of(host, struct vc4_dsi, dsi_host)
return container_of(encoder, struct vc4_dsi_encoder, base.base);
}
-#define DSI_REG(reg) { reg, #reg }
-static const struct {
- u32 reg;
- const char *name;
-} dsi0_regs[] = {
- DSI_REG(DSI0_CTRL),
- DSI_REG(DSI0_STAT),
- DSI_REG(DSI0_HSTX_TO_CNT),
- DSI_REG(DSI0_LPRX_TO_CNT),
- DSI_REG(DSI0_TA_TO_CNT),
- DSI_REG(DSI0_PR_TO_CNT),
- DSI_REG(DSI0_DISP0_CTRL),
- DSI_REG(DSI0_DISP1_CTRL),
- DSI_REG(DSI0_INT_STAT),
- DSI_REG(DSI0_INT_EN),
- DSI_REG(DSI0_PHYC),
- DSI_REG(DSI0_HS_CLT0),
- DSI_REG(DSI0_HS_CLT1),
- DSI_REG(DSI0_HS_CLT2),
- DSI_REG(DSI0_HS_DLT3),
- DSI_REG(DSI0_HS_DLT4),
- DSI_REG(DSI0_HS_DLT5),
- DSI_REG(DSI0_HS_DLT6),
- DSI_REG(DSI0_HS_DLT7),
- DSI_REG(DSI0_PHY_AFEC0),
- DSI_REG(DSI0_PHY_AFEC1),
- DSI_REG(DSI0_ID),
+static const struct debugfs_reg32 dsi0_regs[] = {
+ VC4_REG32(DSI0_CTRL),
+ VC4_REG32(DSI0_STAT),
+ VC4_REG32(DSI0_HSTX_TO_CNT),
+ VC4_REG32(DSI0_LPRX_TO_CNT),
+ VC4_REG32(DSI0_TA_TO_CNT),
+ VC4_REG32(DSI0_PR_TO_CNT),
+ VC4_REG32(DSI0_DISP0_CTRL),
+ VC4_REG32(DSI0_DISP1_CTRL),
+ VC4_REG32(DSI0_INT_STAT),
+ VC4_REG32(DSI0_INT_EN),
+ VC4_REG32(DSI0_PHYC),
+ VC4_REG32(DSI0_HS_CLT0),
+ VC4_REG32(DSI0_HS_CLT1),
+ VC4_REG32(DSI0_HS_CLT2),
+ VC4_REG32(DSI0_HS_DLT3),
+ VC4_REG32(DSI0_HS_DLT4),
+ VC4_REG32(DSI0_HS_DLT5),
+ VC4_REG32(DSI0_HS_DLT6),
+ VC4_REG32(DSI0_HS_DLT7),
+ VC4_REG32(DSI0_PHY_AFEC0),
+ VC4_REG32(DSI0_PHY_AFEC1),
+ VC4_REG32(DSI0_ID),
};
-static const struct {
- u32 reg;
- const char *name;
-} dsi1_regs[] = {
- DSI_REG(DSI1_CTRL),
- DSI_REG(DSI1_STAT),
- DSI_REG(DSI1_HSTX_TO_CNT),
- DSI_REG(DSI1_LPRX_TO_CNT),
- DSI_REG(DSI1_TA_TO_CNT),
- DSI_REG(DSI1_PR_TO_CNT),
- DSI_REG(DSI1_DISP0_CTRL),
- DSI_REG(DSI1_DISP1_CTRL),
- DSI_REG(DSI1_INT_STAT),
- DSI_REG(DSI1_INT_EN),
- DSI_REG(DSI1_PHYC),
- DSI_REG(DSI1_HS_CLT0),
- DSI_REG(DSI1_HS_CLT1),
- DSI_REG(DSI1_HS_CLT2),
- DSI_REG(DSI1_HS_DLT3),
- DSI_REG(DSI1_HS_DLT4),
- DSI_REG(DSI1_HS_DLT5),
- DSI_REG(DSI1_HS_DLT6),
- DSI_REG(DSI1_HS_DLT7),
- DSI_REG(DSI1_PHY_AFEC0),
- DSI_REG(DSI1_PHY_AFEC1),
- DSI_REG(DSI1_ID),
+static const struct debugfs_reg32 dsi1_regs[] = {
+ VC4_REG32(DSI1_CTRL),
+ VC4_REG32(DSI1_STAT),
+ VC4_REG32(DSI1_HSTX_TO_CNT),
+ VC4_REG32(DSI1_LPRX_TO_CNT),
+ VC4_REG32(DSI1_TA_TO_CNT),
+ VC4_REG32(DSI1_PR_TO_CNT),
+ VC4_REG32(DSI1_DISP0_CTRL),
+ VC4_REG32(DSI1_DISP1_CTRL),
+ VC4_REG32(DSI1_INT_STAT),
+ VC4_REG32(DSI1_INT_EN),
+ VC4_REG32(DSI1_PHYC),
+ VC4_REG32(DSI1_HS_CLT0),
+ VC4_REG32(DSI1_HS_CLT1),
+ VC4_REG32(DSI1_HS_CLT2),
+ VC4_REG32(DSI1_HS_DLT3),
+ VC4_REG32(DSI1_HS_DLT4),
+ VC4_REG32(DSI1_HS_DLT5),
+ VC4_REG32(DSI1_HS_DLT6),
+ VC4_REG32(DSI1_HS_DLT7),
+ VC4_REG32(DSI1_PHY_AFEC0),
+ VC4_REG32(DSI1_PHY_AFEC1),
+ VC4_REG32(DSI1_ID),
};
-static void vc4_dsi_dump_regs(struct vc4_dsi *dsi)
-{
- int i;
-
- if (dsi->port == 0) {
- for (i = 0; i < ARRAY_SIZE(dsi0_regs); i++) {
- DRM_INFO("0x%04x (%s): 0x%08x\n",
- dsi0_regs[i].reg, dsi0_regs[i].name,
- DSI_READ(dsi0_regs[i].reg));
- }
- } else {
- for (i = 0; i < ARRAY_SIZE(dsi1_regs); i++) {
- DRM_INFO("0x%04x (%s): 0x%08x\n",
- dsi1_regs[i].reg, dsi1_regs[i].name,
- DSI_READ(dsi1_regs[i].reg));
- }
- }
-}
-
-#ifdef CONFIG_DEBUG_FS
-int vc4_dsi_debugfs_regs(struct seq_file *m, void *unused)
-{
- struct drm_info_node *node = (struct drm_info_node *)m->private;
- struct drm_device *drm = node->minor->dev;
- struct vc4_dev *vc4 = to_vc4_dev(drm);
- int dsi_index = (uintptr_t)node->info_ent->data;
- struct vc4_dsi *dsi = (dsi_index == 1 ? vc4->dsi1 : NULL);
- int i;
-
- if (!dsi)
- return 0;
-
- if (dsi->port == 0) {
- for (i = 0; i < ARRAY_SIZE(dsi0_regs); i++) {
- seq_printf(m, "0x%04x (%s): 0x%08x\n",
- dsi0_regs[i].reg, dsi0_regs[i].name,
- DSI_READ(dsi0_regs[i].reg));
- }
- } else {
- for (i = 0; i < ARRAY_SIZE(dsi1_regs); i++) {
- seq_printf(m, "0x%04x (%s): 0x%08x\n",
- dsi1_regs[i].reg, dsi1_regs[i].name,
- DSI_READ(dsi1_regs[i].reg));
- }
- }
-
- return 0;
-}
-#endif
-
static void vc4_dsi_encoder_destroy(struct drm_encoder *encoder)
{
drm_encoder_cleanup(encoder);
}
if (debug_dump_regs) {
- DRM_INFO("DSI regs before:\n");
- vc4_dsi_dump_regs(dsi);
+ struct drm_printer p = drm_info_printer(&dsi->pdev->dev);
+ dev_info(&dsi->pdev->dev, "DSI regs before:\n");
+ drm_print_regset32(&p, &dsi->regset);
}
/* Round up the clk_set_rate() request slightly, since
drm_bridge_enable(dsi->bridge);
if (debug_dump_regs) {
- DRM_INFO("DSI regs after:\n");
- vc4_dsi_dump_regs(dsi);
+ struct drm_printer p = drm_info_printer(&dsi->pdev->dev);
+ dev_info(&dsi->pdev->dev, "DSI regs after:\n");
+ drm_print_regset32(&p, &dsi->regset);
}
}
if (IS_ERR(dsi->regs))
return PTR_ERR(dsi->regs);
+ dsi->regset.base = dsi->regs;
+ if (dsi->port == 0) {
+ dsi->regset.regs = dsi0_regs;
+ dsi->regset.nregs = ARRAY_SIZE(dsi0_regs);
+ } else {
+ dsi->regset.regs = dsi1_regs;
+ dsi->regset.nregs = ARRAY_SIZE(dsi1_regs);
+ }
+
if (DSI_PORT_READ(ID) != DSI_ID_VALUE) {
dev_err(dev, "Port returned 0x%08x for ID instead of 0x%08x\n",
DSI_PORT_READ(ID), DSI_ID_VALUE);
*/
dsi->encoder->bridge = NULL;
+ if (dsi->port == 0)
+ vc4_debugfs_add_regset32(drm, "dsi0_regs", &dsi->regset);
+ else
+ vc4_debugfs_add_regset32(drm, "dsi1_regs", &dsi->regset);
+
pm_runtime_enable(dev);
return 0;
u32 i;
int ret = 0;
+ if (!vc4->v3d) {
+ DRM_DEBUG("VC4_GET_HANG_STATE with no VC4 V3D probed\n");
+ return -ENODEV;
+ }
+
spin_lock_irqsave(&vc4->job_lock, irqflags);
kernel_state = vc4->hang_state;
if (!kernel_state) {
bo = to_vc4_bo(&exec->bo[i]->base);
bo->seqno = seqno;
- reservation_object_add_shared_fence(bo->resv, exec->fence);
+ reservation_object_add_shared_fence(bo->base.base.resv, exec->fence);
}
list_for_each_entry(bo, &exec->unref_list, unref_head) {
bo = to_vc4_bo(&exec->rcl_write_bo[i]->base);
bo->write_seqno = seqno;
- reservation_object_add_excl_fence(bo->resv, exec->fence);
+ reservation_object_add_excl_fence(bo->base.base.resv, exec->fence);
}
}
int i;
for (i = 0; i < exec->bo_count; i++) {
- struct vc4_bo *bo = to_vc4_bo(&exec->bo[i]->base);
+ struct drm_gem_object *bo = &exec->bo[i]->base;
ww_mutex_unlock(&bo->resv->lock);
}
{
int contended_lock = -1;
int i, ret;
- struct vc4_bo *bo;
+ struct drm_gem_object *bo;
ww_acquire_init(acquire_ctx, &reservation_ww_class);
retry:
if (contended_lock != -1) {
- bo = to_vc4_bo(&exec->bo[contended_lock]->base);
+ bo = &exec->bo[contended_lock]->base;
ret = ww_mutex_lock_slow_interruptible(&bo->resv->lock,
acquire_ctx);
if (ret) {
if (i == contended_lock)
continue;
- bo = to_vc4_bo(&exec->bo[i]->base);
+ bo = &exec->bo[i]->base;
ret = ww_mutex_lock_interruptible(&bo->resv->lock, acquire_ctx);
if (ret) {
int j;
for (j = 0; j < i; j++) {
- bo = to_vc4_bo(&exec->bo[j]->base);
+ bo = &exec->bo[j]->base;
ww_mutex_unlock(&bo->resv->lock);
}
if (contended_lock != -1 && contended_lock >= i) {
- bo = to_vc4_bo(&exec->bo[contended_lock]->base);
+ bo = &exec->bo[contended_lock]->base;
ww_mutex_unlock(&bo->resv->lock);
}
* before we commit the CL to the hardware.
*/
for (i = 0; i < exec->bo_count; i++) {
- bo = to_vc4_bo(&exec->bo[i]->base);
+ bo = &exec->bo[i]->base;
ret = reservation_object_reserve_shared(bo->resv, 1);
if (ret) {
/* Release the reference we had on the perf monitor. */
vc4_perfmon_put(exec->perfmon);
- mutex_lock(&vc4->power_lock);
- if (--vc4->power_refcount == 0) {
- pm_runtime_mark_last_busy(&vc4->v3d->pdev->dev);
- pm_runtime_put_autosuspend(&vc4->v3d->pdev->dev);
- }
- mutex_unlock(&vc4->power_lock);
+ vc4_v3d_pm_put(vc4);
kfree(exec);
}
struct dma_fence *in_fence;
int ret = 0;
+ if (!vc4->v3d) {
+ DRM_DEBUG("VC4_SUBMIT_CL with no VC4 V3D probed\n");
+ return -ENODEV;
+ }
+
if ((args->flags & ~(VC4_SUBMIT_CL_USE_CLEAR_COLOR |
VC4_SUBMIT_CL_FIXED_RCL_ORDER |
VC4_SUBMIT_CL_RCL_ORDER_INCREASING_X |
return -ENOMEM;
}
- mutex_lock(&vc4->power_lock);
- if (vc4->power_refcount++ == 0) {
- ret = pm_runtime_get_sync(&vc4->v3d->pdev->dev);
- if (ret < 0) {
- mutex_unlock(&vc4->power_lock);
- vc4->power_refcount--;
- kfree(exec);
- return ret;
- }
+ ret = vc4_v3d_pm_get(vc4);
+ if (ret) {
+ kfree(exec);
+ return ret;
}
- mutex_unlock(&vc4->power_lock);
exec->args = args;
INIT_LIST_HEAD(&exec->unref_list);
struct clk *pixel_clock;
struct clk *hsm_clock;
+
+ struct debugfs_regset32 hdmi_regset;
+ struct debugfs_regset32 hd_regset;
};
#define HDMI_READ(offset) readl(vc4->hdmi->hdmicore_regs + offset)
return container_of(connector, struct vc4_hdmi_connector, base);
}
-#define HDMI_REG(reg) { reg, #reg }
-static const struct {
- u32 reg;
- const char *name;
-} hdmi_regs[] = {
- HDMI_REG(VC4_HDMI_CORE_REV),
- HDMI_REG(VC4_HDMI_SW_RESET_CONTROL),
- HDMI_REG(VC4_HDMI_HOTPLUG_INT),
- HDMI_REG(VC4_HDMI_HOTPLUG),
- HDMI_REG(VC4_HDMI_MAI_CHANNEL_MAP),
- HDMI_REG(VC4_HDMI_MAI_CONFIG),
- HDMI_REG(VC4_HDMI_MAI_FORMAT),
- HDMI_REG(VC4_HDMI_AUDIO_PACKET_CONFIG),
- HDMI_REG(VC4_HDMI_RAM_PACKET_CONFIG),
- HDMI_REG(VC4_HDMI_HORZA),
- HDMI_REG(VC4_HDMI_HORZB),
- HDMI_REG(VC4_HDMI_FIFO_CTL),
- HDMI_REG(VC4_HDMI_SCHEDULER_CONTROL),
- HDMI_REG(VC4_HDMI_VERTA0),
- HDMI_REG(VC4_HDMI_VERTA1),
- HDMI_REG(VC4_HDMI_VERTB0),
- HDMI_REG(VC4_HDMI_VERTB1),
- HDMI_REG(VC4_HDMI_TX_PHY_RESET_CTL),
- HDMI_REG(VC4_HDMI_TX_PHY_CTL0),
-
- HDMI_REG(VC4_HDMI_CEC_CNTRL_1),
- HDMI_REG(VC4_HDMI_CEC_CNTRL_2),
- HDMI_REG(VC4_HDMI_CEC_CNTRL_3),
- HDMI_REG(VC4_HDMI_CEC_CNTRL_4),
- HDMI_REG(VC4_HDMI_CEC_CNTRL_5),
- HDMI_REG(VC4_HDMI_CPU_STATUS),
- HDMI_REG(VC4_HDMI_CPU_MASK_STATUS),
-
- HDMI_REG(VC4_HDMI_CEC_RX_DATA_1),
- HDMI_REG(VC4_HDMI_CEC_RX_DATA_2),
- HDMI_REG(VC4_HDMI_CEC_RX_DATA_3),
- HDMI_REG(VC4_HDMI_CEC_RX_DATA_4),
- HDMI_REG(VC4_HDMI_CEC_TX_DATA_1),
- HDMI_REG(VC4_HDMI_CEC_TX_DATA_2),
- HDMI_REG(VC4_HDMI_CEC_TX_DATA_3),
- HDMI_REG(VC4_HDMI_CEC_TX_DATA_4),
+static const struct debugfs_reg32 hdmi_regs[] = {
+ VC4_REG32(VC4_HDMI_CORE_REV),
+ VC4_REG32(VC4_HDMI_SW_RESET_CONTROL),
+ VC4_REG32(VC4_HDMI_HOTPLUG_INT),
+ VC4_REG32(VC4_HDMI_HOTPLUG),
+ VC4_REG32(VC4_HDMI_MAI_CHANNEL_MAP),
+ VC4_REG32(VC4_HDMI_MAI_CONFIG),
+ VC4_REG32(VC4_HDMI_MAI_FORMAT),
+ VC4_REG32(VC4_HDMI_AUDIO_PACKET_CONFIG),
+ VC4_REG32(VC4_HDMI_RAM_PACKET_CONFIG),
+ VC4_REG32(VC4_HDMI_HORZA),
+ VC4_REG32(VC4_HDMI_HORZB),
+ VC4_REG32(VC4_HDMI_FIFO_CTL),
+ VC4_REG32(VC4_HDMI_SCHEDULER_CONTROL),
+ VC4_REG32(VC4_HDMI_VERTA0),
+ VC4_REG32(VC4_HDMI_VERTA1),
+ VC4_REG32(VC4_HDMI_VERTB0),
+ VC4_REG32(VC4_HDMI_VERTB1),
+ VC4_REG32(VC4_HDMI_TX_PHY_RESET_CTL),
+ VC4_REG32(VC4_HDMI_TX_PHY_CTL0),
+
+ VC4_REG32(VC4_HDMI_CEC_CNTRL_1),
+ VC4_REG32(VC4_HDMI_CEC_CNTRL_2),
+ VC4_REG32(VC4_HDMI_CEC_CNTRL_3),
+ VC4_REG32(VC4_HDMI_CEC_CNTRL_4),
+ VC4_REG32(VC4_HDMI_CEC_CNTRL_5),
+ VC4_REG32(VC4_HDMI_CPU_STATUS),
+ VC4_REG32(VC4_HDMI_CPU_MASK_STATUS),
+
+ VC4_REG32(VC4_HDMI_CEC_RX_DATA_1),
+ VC4_REG32(VC4_HDMI_CEC_RX_DATA_2),
+ VC4_REG32(VC4_HDMI_CEC_RX_DATA_3),
+ VC4_REG32(VC4_HDMI_CEC_RX_DATA_4),
+ VC4_REG32(VC4_HDMI_CEC_TX_DATA_1),
+ VC4_REG32(VC4_HDMI_CEC_TX_DATA_2),
+ VC4_REG32(VC4_HDMI_CEC_TX_DATA_3),
+ VC4_REG32(VC4_HDMI_CEC_TX_DATA_4),
};
-static const struct {
- u32 reg;
- const char *name;
-} hd_regs[] = {
- HDMI_REG(VC4_HD_M_CTL),
- HDMI_REG(VC4_HD_MAI_CTL),
- HDMI_REG(VC4_HD_MAI_THR),
- HDMI_REG(VC4_HD_MAI_FMT),
- HDMI_REG(VC4_HD_MAI_SMP),
- HDMI_REG(VC4_HD_VID_CTL),
- HDMI_REG(VC4_HD_CSC_CTL),
- HDMI_REG(VC4_HD_FRAME_COUNT),
+static const struct debugfs_reg32 hd_regs[] = {
+ VC4_REG32(VC4_HD_M_CTL),
+ VC4_REG32(VC4_HD_MAI_CTL),
+ VC4_REG32(VC4_HD_MAI_THR),
+ VC4_REG32(VC4_HD_MAI_FMT),
+ VC4_REG32(VC4_HD_MAI_SMP),
+ VC4_REG32(VC4_HD_VID_CTL),
+ VC4_REG32(VC4_HD_CSC_CTL),
+ VC4_REG32(VC4_HD_FRAME_COUNT),
};
-#ifdef CONFIG_DEBUG_FS
-int vc4_hdmi_debugfs_regs(struct seq_file *m, void *unused)
+static int vc4_hdmi_debugfs_regs(struct seq_file *m, void *unused)
{
struct drm_info_node *node = (struct drm_info_node *)m->private;
struct drm_device *dev = node->minor->dev;
struct vc4_dev *vc4 = to_vc4_dev(dev);
- int i;
-
- for (i = 0; i < ARRAY_SIZE(hdmi_regs); i++) {
- seq_printf(m, "%s (0x%04x): 0x%08x\n",
- hdmi_regs[i].name, hdmi_regs[i].reg,
- HDMI_READ(hdmi_regs[i].reg));
- }
+ struct vc4_hdmi *hdmi = vc4->hdmi;
+ struct drm_printer p = drm_seq_file_printer(m);
- for (i = 0; i < ARRAY_SIZE(hd_regs); i++) {
- seq_printf(m, "%s (0x%04x): 0x%08x\n",
- hd_regs[i].name, hd_regs[i].reg,
- HD_READ(hd_regs[i].reg));
- }
+ drm_print_regset32(&p, &hdmi->hdmi_regset);
+ drm_print_regset32(&p, &hdmi->hd_regset);
return 0;
}
-#endif /* CONFIG_DEBUG_FS */
-
-static void vc4_hdmi_dump_regs(struct drm_device *dev)
-{
- struct vc4_dev *vc4 = to_vc4_dev(dev);
- int i;
-
- for (i = 0; i < ARRAY_SIZE(hdmi_regs); i++) {
- DRM_INFO("0x%04x (%s): 0x%08x\n",
- hdmi_regs[i].reg, hdmi_regs[i].name,
- HDMI_READ(hdmi_regs[i].reg));
- }
- for (i = 0; i < ARRAY_SIZE(hd_regs); i++) {
- DRM_INFO("0x%04x (%s): 0x%08x\n",
- hd_regs[i].reg, hd_regs[i].name,
- HD_READ(hd_regs[i].reg));
- }
-}
static enum drm_connector_status
vc4_hdmi_connector_detect(struct drm_connector *connector, bool force)
HDMI_WRITE(VC4_HDMI_TX_PHY_RESET_CTL, 0);
if (debug_dump_regs) {
- DRM_INFO("HDMI regs before:\n");
- vc4_hdmi_dump_regs(dev);
+ struct drm_printer p = drm_info_printer(&hdmi->pdev->dev);
+
+ dev_info(&hdmi->pdev->dev, "HDMI regs before:\n");
+ drm_print_regset32(&p, &hdmi->hdmi_regset);
+ drm_print_regset32(&p, &hdmi->hd_regset);
}
HD_WRITE(VC4_HD_VID_CTL, 0);
HDMI_WRITE(VC4_HDMI_FIFO_CTL, VC4_HDMI_FIFO_CTL_MASTER_SLAVE_N);
if (debug_dump_regs) {
- DRM_INFO("HDMI regs after:\n");
- vc4_hdmi_dump_regs(dev);
+ struct drm_printer p = drm_info_printer(&hdmi->pdev->dev);
+
+ dev_info(&hdmi->pdev->dev, "HDMI regs after:\n");
+ drm_print_regset32(&p, &hdmi->hdmi_regset);
+ drm_print_regset32(&p, &hdmi->hd_regset);
}
HD_WRITE(VC4_HD_VID_CTL,
if (IS_ERR(hdmi->hd_regs))
return PTR_ERR(hdmi->hd_regs);
+ hdmi->hdmi_regset.base = hdmi->hdmicore_regs;
+ hdmi->hdmi_regset.regs = hdmi_regs;
+ hdmi->hdmi_regset.nregs = ARRAY_SIZE(hdmi_regs);
+ hdmi->hd_regset.base = hdmi->hd_regs;
+ hdmi->hd_regset.regs = hd_regs;
+ hdmi->hd_regset.nregs = ARRAY_SIZE(hd_regs);
+
hdmi->pixel_clock = devm_clk_get(dev, "pixel");
if (IS_ERR(hdmi->pixel_clock)) {
DRM_ERROR("Failed to get pixel clock\n");
if (ret)
goto err_destroy_encoder;
+ vc4_debugfs_add_file(drm, "hdmi_regs", vc4_hdmi_debugfs_regs, hdmi);
+
return 0;
#ifdef CONFIG_DRM_VC4_HDMI_CEC
* each CRTC.
*/
+#include <drm/drm_atomic_helper.h>
#include <linux/component.h>
#include "vc4_drv.h"
#include "vc4_regs.h"
-#define HVS_REG(reg) { reg, #reg }
-static const struct {
- u32 reg;
- const char *name;
-} hvs_regs[] = {
- HVS_REG(SCALER_DISPCTRL),
- HVS_REG(SCALER_DISPSTAT),
- HVS_REG(SCALER_DISPID),
- HVS_REG(SCALER_DISPECTRL),
- HVS_REG(SCALER_DISPPROF),
- HVS_REG(SCALER_DISPDITHER),
- HVS_REG(SCALER_DISPEOLN),
- HVS_REG(SCALER_DISPLIST0),
- HVS_REG(SCALER_DISPLIST1),
- HVS_REG(SCALER_DISPLIST2),
- HVS_REG(SCALER_DISPLSTAT),
- HVS_REG(SCALER_DISPLACT0),
- HVS_REG(SCALER_DISPLACT1),
- HVS_REG(SCALER_DISPLACT2),
- HVS_REG(SCALER_DISPCTRL0),
- HVS_REG(SCALER_DISPBKGND0),
- HVS_REG(SCALER_DISPSTAT0),
- HVS_REG(SCALER_DISPBASE0),
- HVS_REG(SCALER_DISPCTRL1),
- HVS_REG(SCALER_DISPBKGND1),
- HVS_REG(SCALER_DISPSTAT1),
- HVS_REG(SCALER_DISPBASE1),
- HVS_REG(SCALER_DISPCTRL2),
- HVS_REG(SCALER_DISPBKGND2),
- HVS_REG(SCALER_DISPSTAT2),
- HVS_REG(SCALER_DISPBASE2),
- HVS_REG(SCALER_DISPALPHA2),
- HVS_REG(SCALER_OLEDOFFS),
- HVS_REG(SCALER_OLEDCOEF0),
- HVS_REG(SCALER_OLEDCOEF1),
- HVS_REG(SCALER_OLEDCOEF2),
+static const struct debugfs_reg32 hvs_regs[] = {
+ VC4_REG32(SCALER_DISPCTRL),
+ VC4_REG32(SCALER_DISPSTAT),
+ VC4_REG32(SCALER_DISPID),
+ VC4_REG32(SCALER_DISPECTRL),
+ VC4_REG32(SCALER_DISPPROF),
+ VC4_REG32(SCALER_DISPDITHER),
+ VC4_REG32(SCALER_DISPEOLN),
+ VC4_REG32(SCALER_DISPLIST0),
+ VC4_REG32(SCALER_DISPLIST1),
+ VC4_REG32(SCALER_DISPLIST2),
+ VC4_REG32(SCALER_DISPLSTAT),
+ VC4_REG32(SCALER_DISPLACT0),
+ VC4_REG32(SCALER_DISPLACT1),
+ VC4_REG32(SCALER_DISPLACT2),
+ VC4_REG32(SCALER_DISPCTRL0),
+ VC4_REG32(SCALER_DISPBKGND0),
+ VC4_REG32(SCALER_DISPSTAT0),
+ VC4_REG32(SCALER_DISPBASE0),
+ VC4_REG32(SCALER_DISPCTRL1),
+ VC4_REG32(SCALER_DISPBKGND1),
+ VC4_REG32(SCALER_DISPSTAT1),
+ VC4_REG32(SCALER_DISPBASE1),
+ VC4_REG32(SCALER_DISPCTRL2),
+ VC4_REG32(SCALER_DISPBKGND2),
+ VC4_REG32(SCALER_DISPSTAT2),
+ VC4_REG32(SCALER_DISPBASE2),
+ VC4_REG32(SCALER_DISPALPHA2),
+ VC4_REG32(SCALER_OLEDOFFS),
+ VC4_REG32(SCALER_OLEDCOEF0),
+ VC4_REG32(SCALER_OLEDCOEF1),
+ VC4_REG32(SCALER_OLEDCOEF2),
};
void vc4_hvs_dump_state(struct drm_device *dev)
{
struct vc4_dev *vc4 = to_vc4_dev(dev);
+ struct drm_printer p = drm_info_printer(&vc4->hvs->pdev->dev);
int i;
- for (i = 0; i < ARRAY_SIZE(hvs_regs); i++) {
- DRM_INFO("0x%04x (%s): 0x%08x\n",
- hvs_regs[i].reg, hvs_regs[i].name,
- HVS_READ(hvs_regs[i].reg));
- }
+ drm_print_regset32(&p, &vc4->hvs->regset);
DRM_INFO("HVS ctx:\n");
for (i = 0; i < 64; i += 4) {
}
}
-#ifdef CONFIG_DEBUG_FS
-int vc4_hvs_debugfs_regs(struct seq_file *m, void *unused)
+static int vc4_hvs_debugfs_underrun(struct seq_file *m, void *data)
{
- struct drm_info_node *node = (struct drm_info_node *)m->private;
+ struct drm_info_node *node = m->private;
struct drm_device *dev = node->minor->dev;
struct vc4_dev *vc4 = to_vc4_dev(dev);
- int i;
+ struct drm_printer p = drm_seq_file_printer(m);
- for (i = 0; i < ARRAY_SIZE(hvs_regs); i++) {
- seq_printf(m, "%s (0x%04x): 0x%08x\n",
- hvs_regs[i].name, hvs_regs[i].reg,
- HVS_READ(hvs_regs[i].reg));
- }
+ drm_printf(&p, "%d\n", atomic_read(&vc4->underrun));
return 0;
}
-#endif
/* The filter kernel is composed of dwords each containing 3 9-bit
* signed integers packed next to each other.
return 0;
}
+void vc4_hvs_mask_underrun(struct drm_device *dev, int channel)
+{
+ struct vc4_dev *vc4 = to_vc4_dev(dev);
+ u32 dispctrl = HVS_READ(SCALER_DISPCTRL);
+
+ dispctrl &= ~SCALER_DISPCTRL_DSPEISLUR(channel);
+
+ HVS_WRITE(SCALER_DISPCTRL, dispctrl);
+}
+
+void vc4_hvs_unmask_underrun(struct drm_device *dev, int channel)
+{
+ struct vc4_dev *vc4 = to_vc4_dev(dev);
+ u32 dispctrl = HVS_READ(SCALER_DISPCTRL);
+
+ dispctrl |= SCALER_DISPCTRL_DSPEISLUR(channel);
+
+ HVS_WRITE(SCALER_DISPSTAT,
+ SCALER_DISPSTAT_EUFLOW(channel));
+ HVS_WRITE(SCALER_DISPCTRL, dispctrl);
+}
+
+static void vc4_hvs_report_underrun(struct drm_device *dev)
+{
+ struct vc4_dev *vc4 = to_vc4_dev(dev);
+
+ atomic_inc(&vc4->underrun);
+ DRM_DEV_ERROR(dev->dev, "HVS underrun\n");
+}
+
+static irqreturn_t vc4_hvs_irq_handler(int irq, void *data)
+{
+ struct drm_device *dev = data;
+ struct vc4_dev *vc4 = to_vc4_dev(dev);
+ irqreturn_t irqret = IRQ_NONE;
+ int channel;
+ u32 control;
+ u32 status;
+
+ status = HVS_READ(SCALER_DISPSTAT);
+ control = HVS_READ(SCALER_DISPCTRL);
+
+ for (channel = 0; channel < SCALER_CHANNELS_COUNT; channel++) {
+ /* Interrupt masking is not always honored, so check it here. */
+ if (status & SCALER_DISPSTAT_EUFLOW(channel) &&
+ control & SCALER_DISPCTRL_DSPEISLUR(channel)) {
+ vc4_hvs_mask_underrun(dev, channel);
+ vc4_hvs_report_underrun(dev);
+
+ irqret = IRQ_HANDLED;
+ }
+ }
+
+ /* Clear every per-channel interrupt flag. */
+ HVS_WRITE(SCALER_DISPSTAT, SCALER_DISPSTAT_IRQMASK(0) |
+ SCALER_DISPSTAT_IRQMASK(1) |
+ SCALER_DISPSTAT_IRQMASK(2));
+
+ return irqret;
+}
+
static int vc4_hvs_bind(struct device *dev, struct device *master, void *data)
{
struct platform_device *pdev = to_platform_device(dev);
if (IS_ERR(hvs->regs))
return PTR_ERR(hvs->regs);
+ hvs->regset.base = hvs->regs;
+ hvs->regset.regs = hvs_regs;
+ hvs->regset.nregs = ARRAY_SIZE(hvs_regs);
+
hvs->dlist = hvs->regs + SCALER_DLIST_START;
spin_lock_init(&hvs->mm_lock);
dispctrl = HVS_READ(SCALER_DISPCTRL);
dispctrl |= SCALER_DISPCTRL_ENABLE;
+ dispctrl |= SCALER_DISPCTRL_DISPEIRQ(0) |
+ SCALER_DISPCTRL_DISPEIRQ(1) |
+ SCALER_DISPCTRL_DISPEIRQ(2);
/* Set DSP3 (PV1) to use HVS channel 2, which would otherwise
* be unused.
*/
dispctrl &= ~SCALER_DISPCTRL_DSP3_MUX_MASK;
+ dispctrl &= ~(SCALER_DISPCTRL_DMAEIRQ |
+ SCALER_DISPCTRL_SLVWREIRQ |
+ SCALER_DISPCTRL_SLVRDEIRQ |
+ SCALER_DISPCTRL_DSPEIEOF(0) |
+ SCALER_DISPCTRL_DSPEIEOF(1) |
+ SCALER_DISPCTRL_DSPEIEOF(2) |
+ SCALER_DISPCTRL_DSPEIEOLN(0) |
+ SCALER_DISPCTRL_DSPEIEOLN(1) |
+ SCALER_DISPCTRL_DSPEIEOLN(2) |
+ SCALER_DISPCTRL_DSPEISLUR(0) |
+ SCALER_DISPCTRL_DSPEISLUR(1) |
+ SCALER_DISPCTRL_DSPEISLUR(2) |
+ SCALER_DISPCTRL_SCLEIRQ);
dispctrl |= VC4_SET_FIELD(2, SCALER_DISPCTRL_DSP3_MUX);
HVS_WRITE(SCALER_DISPCTRL, dispctrl);
+ ret = devm_request_irq(dev, platform_get_irq(pdev, 0),
+ vc4_hvs_irq_handler, 0, "vc4 hvs", drm);
+ if (ret)
+ return ret;
+
+ vc4_debugfs_add_regset32(drm, "hvs_regs", &hvs->regset);
+ vc4_debugfs_add_file(drm, "hvs_underrun", vc4_hvs_debugfs_underrun,
+ NULL);
+
return 0;
}
{
struct vc4_dev *vc4 = to_vc4_dev(dev);
+ if (!vc4->v3d)
+ return;
+
init_waitqueue_head(&vc4->job_wait_queue);
INIT_WORK(&vc4->overflow_mem_work, vc4_overflow_mem_work);
{
struct vc4_dev *vc4 = to_vc4_dev(dev);
+ if (!vc4->v3d)
+ return 0;
+
/* Enable both the render done and out of memory interrupts. */
V3D_WRITE(V3D_INTENA, V3D_DRIVER_IRQS);
{
struct vc4_dev *vc4 = to_vc4_dev(dev);
+ if (!vc4->v3d)
+ return;
+
/* Disable sending interrupts for our driver's IRQs. */
V3D_WRITE(V3D_INTDIS, V3D_DRIVER_IRQS);
return container_of(priv, struct vc4_ctm_state, base);
}
+struct vc4_load_tracker_state {
+ struct drm_private_state base;
+ u64 hvs_load;
+ u64 membus_load;
+};
+
+static struct vc4_load_tracker_state *
+to_vc4_load_tracker_state(struct drm_private_state *priv)
+{
+ return container_of(priv, struct vc4_load_tracker_state, base);
+}
+
static struct vc4_ctm_state *vc4_get_ctm_state(struct drm_atomic_state *state,
struct drm_private_obj *manager)
{
{
struct drm_device *dev = state->dev;
struct vc4_dev *vc4 = to_vc4_dev(dev);
+ struct vc4_crtc *vc4_crtc;
+ int i;
+
+ for (i = 0; i < dev->mode_config.num_crtc; i++) {
+ if (!state->crtcs[i].ptr || !state->crtcs[i].commit)
+ continue;
+
+ vc4_crtc = to_vc4_crtc(state->crtcs[i].ptr);
+ vc4_hvs_mask_underrun(dev, vc4_crtc->channel);
+ }
drm_atomic_helper_wait_for_fences(dev, state, false);
return 0;
}
+static int vc4_load_tracker_atomic_check(struct drm_atomic_state *state)
+{
+ struct drm_plane_state *old_plane_state, *new_plane_state;
+ struct vc4_dev *vc4 = to_vc4_dev(state->dev);
+ struct vc4_load_tracker_state *load_state;
+ struct drm_private_state *priv_state;
+ struct drm_plane *plane;
+ int i;
+
+ priv_state = drm_atomic_get_private_obj_state(state,
+ &vc4->load_tracker);
+ if (IS_ERR(priv_state))
+ return PTR_ERR(priv_state);
+
+ load_state = to_vc4_load_tracker_state(priv_state);
+ for_each_oldnew_plane_in_state(state, plane, old_plane_state,
+ new_plane_state, i) {
+ struct vc4_plane_state *vc4_plane_state;
+
+ if (old_plane_state->fb && old_plane_state->crtc) {
+ vc4_plane_state = to_vc4_plane_state(old_plane_state);
+ load_state->membus_load -= vc4_plane_state->membus_load;
+ load_state->hvs_load -= vc4_plane_state->hvs_load;
+ }
+
+ if (new_plane_state->fb && new_plane_state->crtc) {
+ vc4_plane_state = to_vc4_plane_state(new_plane_state);
+ load_state->membus_load += vc4_plane_state->membus_load;
+ load_state->hvs_load += vc4_plane_state->hvs_load;
+ }
+ }
+
+ /* Don't check the load when the tracker is disabled. */
+ if (!vc4->load_tracker_enabled)
+ return 0;
+
+ /* The absolute limit is 2Gbyte/sec, but let's take a margin to let
+ * the system work when other blocks are accessing the memory.
+ */
+ if (load_state->membus_load > SZ_1G + SZ_512M)
+ return -ENOSPC;
+
+ /* HVS clock is supposed to run @ 250Mhz, let's take a margin and
+ * consider the maximum number of cycles is 240M.
+ */
+ if (load_state->hvs_load > 240000000ULL)
+ return -ENOSPC;
+
+ return 0;
+}
+
+static struct drm_private_state *
+vc4_load_tracker_duplicate_state(struct drm_private_obj *obj)
+{
+ struct vc4_load_tracker_state *state;
+
+ state = kmemdup(obj->state, sizeof(*state), GFP_KERNEL);
+ if (!state)
+ return NULL;
+
+ __drm_atomic_helper_private_obj_duplicate_state(obj, &state->base);
+
+ return &state->base;
+}
+
+static void vc4_load_tracker_destroy_state(struct drm_private_obj *obj,
+ struct drm_private_state *state)
+{
+ struct vc4_load_tracker_state *load_state;
+
+ load_state = to_vc4_load_tracker_state(state);
+ kfree(load_state);
+}
+
+static const struct drm_private_state_funcs vc4_load_tracker_state_funcs = {
+ .atomic_duplicate_state = vc4_load_tracker_duplicate_state,
+ .atomic_destroy_state = vc4_load_tracker_destroy_state,
+};
+
static int
vc4_atomic_check(struct drm_device *dev, struct drm_atomic_state *state)
{
if (ret < 0)
return ret;
- return drm_atomic_helper_check(dev, state);
+ ret = drm_atomic_helper_check(dev, state);
+ if (ret)
+ return ret;
+
+ return vc4_load_tracker_atomic_check(state);
}
static const struct drm_mode_config_funcs vc4_mode_funcs = {
{
struct vc4_dev *vc4 = to_vc4_dev(dev);
struct vc4_ctm_state *ctm_state;
+ struct vc4_load_tracker_state *load_state;
int ret;
+ /* Start with the load tracker enabled. Can be disabled through the
+ * debugfs load_tracker file.
+ */
+ vc4->load_tracker_enabled = true;
+
sema_init(&vc4->async_modeset, 1);
/* Set support for vblank irq fast disable, before drm_vblank_init() */
dev->vblank_disable_immediate = true;
+ dev->irq_enabled = true;
ret = drm_vblank_init(dev, dev->mode_config.num_crtc);
if (ret < 0) {
dev_err(dev->dev, "failed to initialize vblank\n");
drm_atomic_private_obj_init(dev, &vc4->ctm_manager, &ctm_state->base,
&vc4_ctm_state_funcs);
+ load_state = kzalloc(sizeof(*load_state), GFP_KERNEL);
+ if (!load_state) {
+ drm_atomic_private_obj_fini(&vc4->ctm_manager);
+ return -ENOMEM;
+ }
+
+ drm_atomic_private_obj_init(dev, &vc4->load_tracker, &load_state->base,
+ &vc4_load_tracker_state_funcs);
+
drm_mode_config_reset(dev);
drm_kms_helper_poll_init(dev);
int vc4_perfmon_create_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_priv)
{
+ struct vc4_dev *vc4 = to_vc4_dev(dev);
struct vc4_file *vc4file = file_priv->driver_priv;
struct drm_vc4_perfmon_create *req = data;
struct vc4_perfmon *perfmon;
unsigned int i;
int ret;
+ if (!vc4->v3d) {
+ DRM_DEBUG("Creating perfmon no VC4 V3D probed\n");
+ return -ENODEV;
+ }
+
/* Number of monitored counters cannot exceed HW limits. */
if (req->ncounters > DRM_VC4_MAX_PERF_COUNTERS ||
!req->ncounters)
int vc4_perfmon_destroy_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_priv)
{
+ struct vc4_dev *vc4 = to_vc4_dev(dev);
struct vc4_file *vc4file = file_priv->driver_priv;
struct drm_vc4_perfmon_destroy *req = data;
struct vc4_perfmon *perfmon;
+ if (!vc4->v3d) {
+ DRM_DEBUG("Destroying perfmon no VC4 V3D probed\n");
+ return -ENODEV;
+ }
+
mutex_lock(&vc4file->perfmon.lock);
perfmon = idr_remove(&vc4file->perfmon.idr, req->id);
mutex_unlock(&vc4file->perfmon.lock);
int vc4_perfmon_get_values_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_priv)
{
+ struct vc4_dev *vc4 = to_vc4_dev(dev);
struct vc4_file *vc4file = file_priv->driver_priv;
struct drm_vc4_perfmon_get_values *req = data;
struct vc4_perfmon *perfmon;
int ret;
+ if (!vc4->v3d) {
+ DRM_DEBUG("Getting perfmon no VC4 V3D probed\n");
+ return -ENODEV;
+ }
+
mutex_lock(&vc4file->perfmon.lock);
perfmon = idr_find(&vc4file->perfmon.idr, req->id);
vc4_perfmon_get(perfmon);
}
}
+static void vc4_plane_calc_load(struct drm_plane_state *state)
+{
+ unsigned int hvs_load_shift, vrefresh, i;
+ struct drm_framebuffer *fb = state->fb;
+ struct vc4_plane_state *vc4_state;
+ struct drm_crtc_state *crtc_state;
+ unsigned int vscale_factor;
+
+ vc4_state = to_vc4_plane_state(state);
+ crtc_state = drm_atomic_get_existing_crtc_state(state->state,
+ state->crtc);
+ vrefresh = drm_mode_vrefresh(&crtc_state->adjusted_mode);
+
+ /* The HVS is able to process 2 pixels/cycle when scaling the source,
+ * 4 pixels/cycle otherwise.
+ * Alpha blending step seems to be pipelined and it's always operating
+ * at 4 pixels/cycle, so the limiting aspect here seems to be the
+ * scaler block.
+ * HVS load is expressed in clk-cycles/sec (AKA Hz).
+ */
+ if (vc4_state->x_scaling[0] != VC4_SCALING_NONE ||
+ vc4_state->x_scaling[1] != VC4_SCALING_NONE ||
+ vc4_state->y_scaling[0] != VC4_SCALING_NONE ||
+ vc4_state->y_scaling[1] != VC4_SCALING_NONE)
+ hvs_load_shift = 1;
+ else
+ hvs_load_shift = 2;
+
+ vc4_state->membus_load = 0;
+ vc4_state->hvs_load = 0;
+ for (i = 0; i < fb->format->num_planes; i++) {
+ /* Even if the bandwidth/plane required for a single frame is
+ *
+ * vc4_state->src_w[i] * vc4_state->src_h[i] * cpp * vrefresh
+ *
+ * when downscaling, we have to read more pixels per line in
+ * the time frame reserved for a single line, so the bandwidth
+ * demand can be punctually higher. To account for that, we
+ * calculate the down-scaling factor and multiply the plane
+ * load by this number. We're likely over-estimating the read
+ * demand, but that's better than under-estimating it.
+ */
+ vscale_factor = DIV_ROUND_UP(vc4_state->src_h[i],
+ vc4_state->crtc_h);
+ vc4_state->membus_load += vc4_state->src_w[i] *
+ vc4_state->src_h[i] * vscale_factor *
+ fb->format->cpp[i];
+ vc4_state->hvs_load += vc4_state->crtc_h * vc4_state->crtc_w;
+ }
+
+ vc4_state->hvs_load *= vrefresh;
+ vc4_state->hvs_load >>= hvs_load_shift;
+ vc4_state->membus_load *= vrefresh;
+}
+
static int vc4_plane_allocate_lbm(struct drm_plane_state *state)
{
struct vc4_dev *vc4 = to_vc4_dev(state->plane->dev);
*/
vc4_state->dlist_initialized = 1;
+ vc4_plane_calc_load(state);
+
return 0;
}
bo = to_vc4_bo(&drm_fb_cma_get_gem_obj(state->fb, 0)->base);
- fence = reservation_object_get_excl_rcu(bo->resv);
+ fence = reservation_object_get_excl_rcu(bo->base.base.resv);
drm_atomic_set_fence_for_plane(state, fence);
if (plane->state->fb == state->fb)
#define PV_HACT_ACT 0x30
+#define SCALER_CHANNELS_COUNT 3
+
#define SCALER_DISPCTRL 0x00000000
/* Global register for clock gating the HVS */
# define SCALER_DISPCTRL_ENABLE BIT(31)
-# define SCALER_DISPCTRL_DSP2EISLUR BIT(15)
-# define SCALER_DISPCTRL_DSP1EISLUR BIT(14)
# define SCALER_DISPCTRL_DSP3_MUX_MASK VC4_MASK(19, 18)
# define SCALER_DISPCTRL_DSP3_MUX_SHIFT 18
* SCALER_DISPSTAT_IRQDISP0. Note that short frame contributions are
* always enabled.
*/
-# define SCALER_DISPCTRL_DSP0EISLUR BIT(13)
-# define SCALER_DISPCTRL_DSP2EIEOLN BIT(12)
-# define SCALER_DISPCTRL_DSP2EIEOF BIT(11)
-# define SCALER_DISPCTRL_DSP1EIEOLN BIT(10)
-# define SCALER_DISPCTRL_DSP1EIEOF BIT(9)
+# define SCALER_DISPCTRL_DSPEISLUR(x) BIT(13 + (x))
/* Enables Display 0 end-of-line-N contribution to
* SCALER_DISPSTAT_IRQDISP0
*/
-# define SCALER_DISPCTRL_DSP0EIEOLN BIT(8)
+# define SCALER_DISPCTRL_DSPEIEOLN(x) BIT(8 + ((x) * 2))
/* Enables Display 0 EOF contribution to SCALER_DISPSTAT_IRQDISP0 */
-# define SCALER_DISPCTRL_DSP0EIEOF BIT(7)
+# define SCALER_DISPCTRL_DSPEIEOF(x) BIT(7 + ((x) * 2))
# define SCALER_DISPCTRL_SLVRDEIRQ BIT(6)
# define SCALER_DISPCTRL_SLVWREIRQ BIT(5)
# define SCALER_DISPCTRL_DMAEIRQ BIT(4)
-# define SCALER_DISPCTRL_DISP2EIRQ BIT(3)
-# define SCALER_DISPCTRL_DISP1EIRQ BIT(2)
/* Enables interrupt generation on the enabled EOF/EOLN/EISLUR
* bits and short frames..
*/
-# define SCALER_DISPCTRL_DISP0EIRQ BIT(1)
+# define SCALER_DISPCTRL_DISPEIRQ(x) BIT(1 + (x))
/* Enables interrupt generation on scaler profiler interrupt. */
# define SCALER_DISPCTRL_SCLEIRQ BIT(0)
#define SCALER_DISPSTAT 0x00000004
-# define SCALER_DISPSTAT_COBLOW2 BIT(29)
-# define SCALER_DISPSTAT_EOLN2 BIT(28)
-# define SCALER_DISPSTAT_ESFRAME2 BIT(27)
-# define SCALER_DISPSTAT_ESLINE2 BIT(26)
-# define SCALER_DISPSTAT_EUFLOW2 BIT(25)
-# define SCALER_DISPSTAT_EOF2 BIT(24)
-
-# define SCALER_DISPSTAT_COBLOW1 BIT(21)
-# define SCALER_DISPSTAT_EOLN1 BIT(20)
-# define SCALER_DISPSTAT_ESFRAME1 BIT(19)
-# define SCALER_DISPSTAT_ESLINE1 BIT(18)
-# define SCALER_DISPSTAT_EUFLOW1 BIT(17)
-# define SCALER_DISPSTAT_EOF1 BIT(16)
-
# define SCALER_DISPSTAT_RESP_MASK VC4_MASK(15, 14)
# define SCALER_DISPSTAT_RESP_SHIFT 14
# define SCALER_DISPSTAT_RESP_OKAY 0
# define SCALER_DISPSTAT_RESP_SLVERR 2
# define SCALER_DISPSTAT_RESP_DECERR 3
-# define SCALER_DISPSTAT_COBLOW0 BIT(13)
+# define SCALER_DISPSTAT_COBLOW(x) BIT(13 + ((x) * 8))
/* Set when the DISPEOLN line is done compositing. */
-# define SCALER_DISPSTAT_EOLN0 BIT(12)
+# define SCALER_DISPSTAT_EOLN(x) BIT(12 + ((x) * 8))
/* Set when VSTART is seen but there are still pixels in the current
* output line.
*/
-# define SCALER_DISPSTAT_ESFRAME0 BIT(11)
+# define SCALER_DISPSTAT_ESFRAME(x) BIT(11 + ((x) * 8))
/* Set when HSTART is seen but there are still pixels in the current
* output line.
*/
-# define SCALER_DISPSTAT_ESLINE0 BIT(10)
+# define SCALER_DISPSTAT_ESLINE(x) BIT(10 + ((x) * 8))
/* Set when the the downstream tries to read from the display FIFO
* while it's empty.
*/
-# define SCALER_DISPSTAT_EUFLOW0 BIT(9)
+# define SCALER_DISPSTAT_EUFLOW(x) BIT(9 + ((x) * 8))
/* Set when the display mode changes from RUN to EOF */
-# define SCALER_DISPSTAT_EOF0 BIT(8)
+# define SCALER_DISPSTAT_EOF(x) BIT(8 + ((x) * 8))
+
+# define SCALER_DISPSTAT_IRQMASK(x) VC4_MASK(13 + ((x) * 8), \
+ 8 + ((x) * 8))
/* Set on AXI invalid DMA ID error. */
# define SCALER_DISPSTAT_DMA_ERROR BIT(7)
* SCALER_DISPSTAT_RESP_ERROR is not SCALER_DISPSTAT_RESP_OKAY.
*/
# define SCALER_DISPSTAT_IRQDMA BIT(4)
-# define SCALER_DISPSTAT_IRQDISP2 BIT(3)
-# define SCALER_DISPSTAT_IRQDISP1 BIT(2)
/* Set when any of the EOF/EOLN/ESFRAME/ESLINE bits are set and their
* corresponding interrupt bit is enabled in DISPCTRL.
*/
-# define SCALER_DISPSTAT_IRQDISP0 BIT(1)
+# define SCALER_DISPSTAT_IRQDISP(x) BIT(1 + (x))
/* On read, the profiler interrupt. On write, clear *all* interrupt bits. */
# define SCALER_DISPSTAT_IRQSCL BIT(0)
}
if (setup->zs_read) {
+ if (setup->color_read) {
+ /* Exec previous load. */
+ vc4_tile_coordinates(setup, x, y);
+ vc4_store_before_load(setup);
+ }
+
if (args->zs_read.flags &
VC4_SUBMIT_RCL_SURFACE_READ_IS_FULL_RES) {
rcl_u8(setup, VC4_PACKET_LOAD_FULL_RES_TILE_BUFFER);
&args->zs_read, x, y) |
VC4_LOADSTORE_FULL_RES_DISABLE_COLOR);
} else {
- if (setup->color_read) {
- /* Exec previous load. */
- vc4_tile_coordinates(setup, x, y);
- vc4_store_before_load(setup);
- }
-
rcl_u8(setup, VC4_PACKET_LOAD_TILE_BUFFER_GENERAL);
rcl_u16(setup, args->zs_read.bits);
rcl_u32(setup, setup->zs_read->paddr +
}
}
if (setup->zs_read) {
+ if (setup->color_read) {
+ loop_body_size += VC4_PACKET_TILE_COORDINATES_SIZE;
+ loop_body_size += VC4_PACKET_STORE_TILE_BUFFER_GENERAL_SIZE;
+ }
+
if (args->zs_read.flags &
VC4_SUBMIT_RCL_SURFACE_READ_IS_FULL_RES) {
loop_body_size += VC4_PACKET_LOAD_FULL_RES_TILE_BUFFER_SIZE;
} else {
- if (setup->color_read &&
- !(args->color_read.flags &
- VC4_SUBMIT_RCL_SURFACE_READ_IS_FULL_RES)) {
- loop_body_size += VC4_PACKET_TILE_COORDINATES_SIZE;
- loop_body_size += VC4_PACKET_STORE_TILE_BUFFER_GENERAL_SIZE;
- }
loop_body_size += VC4_PACKET_LOAD_TILE_BUFFER_GENERAL_SIZE;
}
}
struct drm_writeback_connector connector;
void __iomem *regs;
+ struct debugfs_regset32 regset;
};
static inline struct vc4_txp *encoder_to_vc4_txp(struct drm_encoder *encoder)
return container_of(conn, struct vc4_txp, connector.base);
}
-#define TXP_REG(reg) { reg, #reg }
-static const struct {
- u32 reg;
- const char *name;
-} txp_regs[] = {
- TXP_REG(TXP_DST_PTR),
- TXP_REG(TXP_DST_PITCH),
- TXP_REG(TXP_DIM),
- TXP_REG(TXP_DST_CTRL),
- TXP_REG(TXP_PROGRESS),
+static const struct debugfs_reg32 txp_regs[] = {
+ VC4_REG32(TXP_DST_PTR),
+ VC4_REG32(TXP_DST_PITCH),
+ VC4_REG32(TXP_DIM),
+ VC4_REG32(TXP_DST_CTRL),
+ VC4_REG32(TXP_PROGRESS),
};
-#ifdef CONFIG_DEBUG_FS
-int vc4_txp_debugfs_regs(struct seq_file *m, void *unused)
-{
- struct drm_info_node *node = (struct drm_info_node *)m->private;
- struct drm_device *dev = node->minor->dev;
- struct vc4_dev *vc4 = to_vc4_dev(dev);
- struct vc4_txp *txp = vc4->txp;
- int i;
-
- if (!txp)
- return 0;
-
- for (i = 0; i < ARRAY_SIZE(txp_regs); i++) {
- seq_printf(m, "%s (0x%04x): 0x%08x\n",
- txp_regs[i].name, txp_regs[i].reg,
- TXP_READ(txp_regs[i].reg));
- }
-
- return 0;
-}
-#endif
-
static int vc4_txp_connector_get_modes(struct drm_connector *connector)
{
struct drm_device *dev = connector->dev;
struct drm_connector_state *conn_state)
{
struct drm_crtc_state *crtc_state;
- struct drm_gem_cma_object *gem;
struct drm_framebuffer *fb;
int i;
if (i == ARRAY_SIZE(drm_fmts))
return -EINVAL;
- gem = drm_fb_cma_get_gem_obj(fb, 0);
-
/* Pitch must be aligned on 16 bytes. */
if (fb->pitches[0] & GENMASK(3, 0))
return -EINVAL;
TXP_WRITE(TXP_DST_CTRL, ctrl);
- drm_writeback_queue_job(&txp->connector, conn_state->writeback_job);
+ drm_writeback_queue_job(&txp->connector, conn_state);
}
static const struct drm_connector_helper_funcs vc4_txp_connector_helper_funcs = {
txp->regs = vc4_ioremap_regs(pdev, 0);
if (IS_ERR(txp->regs))
return PTR_ERR(txp->regs);
+ txp->regset.base = txp->regs;
+ txp->regset.regs = txp_regs;
+ txp->regset.nregs = ARRAY_SIZE(txp_regs);
drm_connector_helper_add(&txp->connector.base,
&vc4_txp_connector_helper_funcs);
dev_set_drvdata(dev, txp);
vc4->txp = txp;
+ vc4_debugfs_add_regset32(drm, "txp_regs", &txp->regset);
+
return 0;
}
#include "vc4_drv.h"
#include "vc4_regs.h"
-#ifdef CONFIG_DEBUG_FS
-#define REGDEF(reg) { reg, #reg }
-static const struct {
- uint32_t reg;
- const char *name;
-} vc4_reg_defs[] = {
- REGDEF(V3D_IDENT0),
- REGDEF(V3D_IDENT1),
- REGDEF(V3D_IDENT2),
- REGDEF(V3D_SCRATCH),
- REGDEF(V3D_L2CACTL),
- REGDEF(V3D_SLCACTL),
- REGDEF(V3D_INTCTL),
- REGDEF(V3D_INTENA),
- REGDEF(V3D_INTDIS),
- REGDEF(V3D_CT0CS),
- REGDEF(V3D_CT1CS),
- REGDEF(V3D_CT0EA),
- REGDEF(V3D_CT1EA),
- REGDEF(V3D_CT0CA),
- REGDEF(V3D_CT1CA),
- REGDEF(V3D_CT00RA0),
- REGDEF(V3D_CT01RA0),
- REGDEF(V3D_CT0LC),
- REGDEF(V3D_CT1LC),
- REGDEF(V3D_CT0PC),
- REGDEF(V3D_CT1PC),
- REGDEF(V3D_PCS),
- REGDEF(V3D_BFC),
- REGDEF(V3D_RFC),
- REGDEF(V3D_BPCA),
- REGDEF(V3D_BPCS),
- REGDEF(V3D_BPOA),
- REGDEF(V3D_BPOS),
- REGDEF(V3D_BXCF),
- REGDEF(V3D_SQRSV0),
- REGDEF(V3D_SQRSV1),
- REGDEF(V3D_SQCNTL),
- REGDEF(V3D_SRQPC),
- REGDEF(V3D_SRQUA),
- REGDEF(V3D_SRQUL),
- REGDEF(V3D_SRQCS),
- REGDEF(V3D_VPACNTL),
- REGDEF(V3D_VPMBASE),
- REGDEF(V3D_PCTRC),
- REGDEF(V3D_PCTRE),
- REGDEF(V3D_PCTR(0)),
- REGDEF(V3D_PCTRS(0)),
- REGDEF(V3D_PCTR(1)),
- REGDEF(V3D_PCTRS(1)),
- REGDEF(V3D_PCTR(2)),
- REGDEF(V3D_PCTRS(2)),
- REGDEF(V3D_PCTR(3)),
- REGDEF(V3D_PCTRS(3)),
- REGDEF(V3D_PCTR(4)),
- REGDEF(V3D_PCTRS(4)),
- REGDEF(V3D_PCTR(5)),
- REGDEF(V3D_PCTRS(5)),
- REGDEF(V3D_PCTR(6)),
- REGDEF(V3D_PCTRS(6)),
- REGDEF(V3D_PCTR(7)),
- REGDEF(V3D_PCTRS(7)),
- REGDEF(V3D_PCTR(8)),
- REGDEF(V3D_PCTRS(8)),
- REGDEF(V3D_PCTR(9)),
- REGDEF(V3D_PCTRS(9)),
- REGDEF(V3D_PCTR(10)),
- REGDEF(V3D_PCTRS(10)),
- REGDEF(V3D_PCTR(11)),
- REGDEF(V3D_PCTRS(11)),
- REGDEF(V3D_PCTR(12)),
- REGDEF(V3D_PCTRS(12)),
- REGDEF(V3D_PCTR(13)),
- REGDEF(V3D_PCTRS(13)),
- REGDEF(V3D_PCTR(14)),
- REGDEF(V3D_PCTRS(14)),
- REGDEF(V3D_PCTR(15)),
- REGDEF(V3D_PCTRS(15)),
- REGDEF(V3D_DBGE),
- REGDEF(V3D_FDBGO),
- REGDEF(V3D_FDBGB),
- REGDEF(V3D_FDBGR),
- REGDEF(V3D_FDBGS),
- REGDEF(V3D_ERRSTAT),
+static const struct debugfs_reg32 v3d_regs[] = {
+ VC4_REG32(V3D_IDENT0),
+ VC4_REG32(V3D_IDENT1),
+ VC4_REG32(V3D_IDENT2),
+ VC4_REG32(V3D_SCRATCH),
+ VC4_REG32(V3D_L2CACTL),
+ VC4_REG32(V3D_SLCACTL),
+ VC4_REG32(V3D_INTCTL),
+ VC4_REG32(V3D_INTENA),
+ VC4_REG32(V3D_INTDIS),
+ VC4_REG32(V3D_CT0CS),
+ VC4_REG32(V3D_CT1CS),
+ VC4_REG32(V3D_CT0EA),
+ VC4_REG32(V3D_CT1EA),
+ VC4_REG32(V3D_CT0CA),
+ VC4_REG32(V3D_CT1CA),
+ VC4_REG32(V3D_CT00RA0),
+ VC4_REG32(V3D_CT01RA0),
+ VC4_REG32(V3D_CT0LC),
+ VC4_REG32(V3D_CT1LC),
+ VC4_REG32(V3D_CT0PC),
+ VC4_REG32(V3D_CT1PC),
+ VC4_REG32(V3D_PCS),
+ VC4_REG32(V3D_BFC),
+ VC4_REG32(V3D_RFC),
+ VC4_REG32(V3D_BPCA),
+ VC4_REG32(V3D_BPCS),
+ VC4_REG32(V3D_BPOA),
+ VC4_REG32(V3D_BPOS),
+ VC4_REG32(V3D_BXCF),
+ VC4_REG32(V3D_SQRSV0),
+ VC4_REG32(V3D_SQRSV1),
+ VC4_REG32(V3D_SQCNTL),
+ VC4_REG32(V3D_SRQPC),
+ VC4_REG32(V3D_SRQUA),
+ VC4_REG32(V3D_SRQUL),
+ VC4_REG32(V3D_SRQCS),
+ VC4_REG32(V3D_VPACNTL),
+ VC4_REG32(V3D_VPMBASE),
+ VC4_REG32(V3D_PCTRC),
+ VC4_REG32(V3D_PCTRE),
+ VC4_REG32(V3D_PCTR(0)),
+ VC4_REG32(V3D_PCTRS(0)),
+ VC4_REG32(V3D_PCTR(1)),
+ VC4_REG32(V3D_PCTRS(1)),
+ VC4_REG32(V3D_PCTR(2)),
+ VC4_REG32(V3D_PCTRS(2)),
+ VC4_REG32(V3D_PCTR(3)),
+ VC4_REG32(V3D_PCTRS(3)),
+ VC4_REG32(V3D_PCTR(4)),
+ VC4_REG32(V3D_PCTRS(4)),
+ VC4_REG32(V3D_PCTR(5)),
+ VC4_REG32(V3D_PCTRS(5)),
+ VC4_REG32(V3D_PCTR(6)),
+ VC4_REG32(V3D_PCTRS(6)),
+ VC4_REG32(V3D_PCTR(7)),
+ VC4_REG32(V3D_PCTRS(7)),
+ VC4_REG32(V3D_PCTR(8)),
+ VC4_REG32(V3D_PCTRS(8)),
+ VC4_REG32(V3D_PCTR(9)),
+ VC4_REG32(V3D_PCTRS(9)),
+ VC4_REG32(V3D_PCTR(10)),
+ VC4_REG32(V3D_PCTRS(10)),
+ VC4_REG32(V3D_PCTR(11)),
+ VC4_REG32(V3D_PCTRS(11)),
+ VC4_REG32(V3D_PCTR(12)),
+ VC4_REG32(V3D_PCTRS(12)),
+ VC4_REG32(V3D_PCTR(13)),
+ VC4_REG32(V3D_PCTRS(13)),
+ VC4_REG32(V3D_PCTR(14)),
+ VC4_REG32(V3D_PCTRS(14)),
+ VC4_REG32(V3D_PCTR(15)),
+ VC4_REG32(V3D_PCTRS(15)),
+ VC4_REG32(V3D_DBGE),
+ VC4_REG32(V3D_FDBGO),
+ VC4_REG32(V3D_FDBGB),
+ VC4_REG32(V3D_FDBGR),
+ VC4_REG32(V3D_FDBGS),
+ VC4_REG32(V3D_ERRSTAT),
};
-int vc4_v3d_debugfs_regs(struct seq_file *m, void *unused)
+static int vc4_v3d_debugfs_ident(struct seq_file *m, void *unused)
{
struct drm_info_node *node = (struct drm_info_node *)m->private;
struct drm_device *dev = node->minor->dev;
struct vc4_dev *vc4 = to_vc4_dev(dev);
- int i;
-
- for (i = 0; i < ARRAY_SIZE(vc4_reg_defs); i++) {
- seq_printf(m, "%s (0x%04x): 0x%08x\n",
- vc4_reg_defs[i].name, vc4_reg_defs[i].reg,
- V3D_READ(vc4_reg_defs[i].reg));
+ int ret = vc4_v3d_pm_get(vc4);
+
+ if (ret == 0) {
+ uint32_t ident1 = V3D_READ(V3D_IDENT1);
+ uint32_t nslc = VC4_GET_FIELD(ident1, V3D_IDENT1_NSLC);
+ uint32_t tups = VC4_GET_FIELD(ident1, V3D_IDENT1_TUPS);
+ uint32_t qups = VC4_GET_FIELD(ident1, V3D_IDENT1_QUPS);
+
+ seq_printf(m, "Revision: %d\n",
+ VC4_GET_FIELD(ident1, V3D_IDENT1_REV));
+ seq_printf(m, "Slices: %d\n", nslc);
+ seq_printf(m, "TMUs: %d\n", nslc * tups);
+ seq_printf(m, "QPUs: %d\n", nslc * qups);
+ seq_printf(m, "Semaphores: %d\n",
+ VC4_GET_FIELD(ident1, V3D_IDENT1_NSEM));
+ vc4_v3d_pm_put(vc4);
}
return 0;
}
-int vc4_v3d_debugfs_ident(struct seq_file *m, void *unused)
+/**
+ * Wraps pm_runtime_get_sync() in a refcount, so that we can reliably
+ * get the pm_runtime refcount to 0 in vc4_reset().
+ */
+int
+vc4_v3d_pm_get(struct vc4_dev *vc4)
{
- struct drm_info_node *node = (struct drm_info_node *)m->private;
- struct drm_device *dev = node->minor->dev;
- struct vc4_dev *vc4 = to_vc4_dev(dev);
- uint32_t ident1 = V3D_READ(V3D_IDENT1);
- uint32_t nslc = VC4_GET_FIELD(ident1, V3D_IDENT1_NSLC);
- uint32_t tups = VC4_GET_FIELD(ident1, V3D_IDENT1_TUPS);
- uint32_t qups = VC4_GET_FIELD(ident1, V3D_IDENT1_QUPS);
-
- seq_printf(m, "Revision: %d\n",
- VC4_GET_FIELD(ident1, V3D_IDENT1_REV));
- seq_printf(m, "Slices: %d\n", nslc);
- seq_printf(m, "TMUs: %d\n", nslc * tups);
- seq_printf(m, "QPUs: %d\n", nslc * qups);
- seq_printf(m, "Semaphores: %d\n",
- VC4_GET_FIELD(ident1, V3D_IDENT1_NSEM));
+ mutex_lock(&vc4->power_lock);
+ if (vc4->power_refcount++ == 0) {
+ int ret = pm_runtime_get_sync(&vc4->v3d->pdev->dev);
+
+ if (ret < 0) {
+ vc4->power_refcount--;
+ mutex_unlock(&vc4->power_lock);
+ return ret;
+ }
+ }
+ mutex_unlock(&vc4->power_lock);
return 0;
}
-#endif /* CONFIG_DEBUG_FS */
+
+void
+vc4_v3d_pm_put(struct vc4_dev *vc4)
+{
+ mutex_lock(&vc4->power_lock);
+ if (--vc4->power_refcount == 0) {
+ pm_runtime_mark_last_busy(&vc4->v3d->pdev->dev);
+ pm_runtime_put_autosuspend(&vc4->v3d->pdev->dev);
+ }
+ mutex_unlock(&vc4->power_lock);
+}
static void vc4_v3d_init_hw(struct drm_device *dev)
{
v3d->regs = vc4_ioremap_regs(pdev, 0);
if (IS_ERR(v3d->regs))
return PTR_ERR(v3d->regs);
+ v3d->regset.base = v3d->regs;
+ v3d->regset.regs = v3d_regs;
+ v3d->regset.nregs = ARRAY_SIZE(v3d_regs);
vc4->v3d = v3d;
v3d->vc4 = vc4;
pm_runtime_set_autosuspend_delay(dev, 40); /* a little over 2 frames. */
pm_runtime_enable(dev);
+ vc4_debugfs_add_file(drm, "v3d_ident", vc4_v3d_debugfs_ident, NULL);
+ vc4_debugfs_add_regset32(drm, "v3d_regs", &v3d->regset);
+
return 0;
}
return 0;
}
-static const struct of_device_id vc4_v3d_dt_match[] = {
+const struct of_device_id vc4_v3d_dt_match[] = {
{ .compatible = "brcm,bcm2835-v3d" },
{ .compatible = "brcm,cygnus-v3d" },
{ .compatible = "brcm,vc4-v3d" },
struct clk *clock;
const struct vc4_vec_tv_mode *tv_mode;
+
+ struct debugfs_regset32 regset;
};
#define VEC_READ(offset) readl(vec->regs + (offset))
void (*mode_set)(struct vc4_vec *vec);
};
-#define VEC_REG(reg) { reg, #reg }
-static const struct {
- u32 reg;
- const char *name;
-} vec_regs[] = {
- VEC_REG(VEC_WSE_CONTROL),
- VEC_REG(VEC_WSE_WSS_DATA),
- VEC_REG(VEC_WSE_VPS_DATA1),
- VEC_REG(VEC_WSE_VPS_CONTROL),
- VEC_REG(VEC_REVID),
- VEC_REG(VEC_CONFIG0),
- VEC_REG(VEC_SCHPH),
- VEC_REG(VEC_CLMP0_START),
- VEC_REG(VEC_CLMP0_END),
- VEC_REG(VEC_FREQ3_2),
- VEC_REG(VEC_FREQ1_0),
- VEC_REG(VEC_CONFIG1),
- VEC_REG(VEC_CONFIG2),
- VEC_REG(VEC_INTERRUPT_CONTROL),
- VEC_REG(VEC_INTERRUPT_STATUS),
- VEC_REG(VEC_FCW_SECAM_B),
- VEC_REG(VEC_SECAM_GAIN_VAL),
- VEC_REG(VEC_CONFIG3),
- VEC_REG(VEC_STATUS0),
- VEC_REG(VEC_MASK0),
- VEC_REG(VEC_CFG),
- VEC_REG(VEC_DAC_TEST),
- VEC_REG(VEC_DAC_CONFIG),
- VEC_REG(VEC_DAC_MISC),
+static const struct debugfs_reg32 vec_regs[] = {
+ VC4_REG32(VEC_WSE_CONTROL),
+ VC4_REG32(VEC_WSE_WSS_DATA),
+ VC4_REG32(VEC_WSE_VPS_DATA1),
+ VC4_REG32(VEC_WSE_VPS_CONTROL),
+ VC4_REG32(VEC_REVID),
+ VC4_REG32(VEC_CONFIG0),
+ VC4_REG32(VEC_SCHPH),
+ VC4_REG32(VEC_CLMP0_START),
+ VC4_REG32(VEC_CLMP0_END),
+ VC4_REG32(VEC_FREQ3_2),
+ VC4_REG32(VEC_FREQ1_0),
+ VC4_REG32(VEC_CONFIG1),
+ VC4_REG32(VEC_CONFIG2),
+ VC4_REG32(VEC_INTERRUPT_CONTROL),
+ VC4_REG32(VEC_INTERRUPT_STATUS),
+ VC4_REG32(VEC_FCW_SECAM_B),
+ VC4_REG32(VEC_SECAM_GAIN_VAL),
+ VC4_REG32(VEC_CONFIG3),
+ VC4_REG32(VEC_STATUS0),
+ VC4_REG32(VEC_MASK0),
+ VC4_REG32(VEC_CFG),
+ VC4_REG32(VEC_DAC_TEST),
+ VC4_REG32(VEC_DAC_CONFIG),
+ VC4_REG32(VEC_DAC_MISC),
};
-#ifdef CONFIG_DEBUG_FS
-int vc4_vec_debugfs_regs(struct seq_file *m, void *unused)
-{
- struct drm_info_node *node = (struct drm_info_node *)m->private;
- struct drm_device *dev = node->minor->dev;
- struct vc4_dev *vc4 = to_vc4_dev(dev);
- struct vc4_vec *vec = vc4->vec;
- int i;
-
- if (!vec)
- return 0;
-
- for (i = 0; i < ARRAY_SIZE(vec_regs); i++) {
- seq_printf(m, "%s (0x%04x): 0x%08x\n",
- vec_regs[i].name, vec_regs[i].reg,
- VEC_READ(vec_regs[i].reg));
- }
-
- return 0;
-}
-#endif
-
static void vc4_vec_ntsc_mode_set(struct vc4_vec *vec)
{
VEC_WRITE(VEC_CONFIG0, VEC_CONFIG0_NTSC_STD | VEC_CONFIG0_PDEN);
vec->regs = vc4_ioremap_regs(pdev, 0);
if (IS_ERR(vec->regs))
return PTR_ERR(vec->regs);
+ vec->regset.base = vec->regs;
+ vec->regset.regs = vec_regs;
+ vec->regset.nregs = ARRAY_SIZE(vec_regs);
vec->clock = devm_clk_get(dev, NULL);
if (IS_ERR(vec->clock)) {
vc4->vec = vec;
+ vc4_debugfs_add_regset32(drm, "vec_regs", &vec->regset);
+
return 0;
err_destroy_encoder:
#include "virtgpu_drv.h"
+static void virtio_add_bool(struct seq_file *m, const char *name,
+ bool value)
+{
+ seq_printf(m, "%-16s : %s\n", name, value ? "yes" : "no");
+}
+
+static void virtio_add_int(struct seq_file *m, const char *name,
+ int value)
+{
+ seq_printf(m, "%-16s : %d\n", name, value);
+}
+
+static int virtio_gpu_features(struct seq_file *m, void *data)
+{
+ struct drm_info_node *node = (struct drm_info_node *) m->private;
+ struct virtio_gpu_device *vgdev = node->minor->dev->dev_private;
+
+ virtio_add_bool(m, "virgl", vgdev->has_virgl_3d);
+ virtio_add_bool(m, "edid", vgdev->has_edid);
+ virtio_add_int(m, "cap sets", vgdev->num_capsets);
+ virtio_add_int(m, "scanouts", vgdev->num_scanouts);
+ return 0;
+}
+
static int
virtio_gpu_debugfs_irq_info(struct seq_file *m, void *data)
{
}
static struct drm_info_list virtio_gpu_debugfs_list[] = {
- { "irq_fence", virtio_gpu_debugfs_irq_info, 0, NULL },
+ { "virtio-gpu-features", virtio_gpu_features },
+ { "virtio-gpu-irq-fence", virtio_gpu_debugfs_irq_info, 0, NULL },
};
#define VIRTIO_GPU_DEBUGFS_ENTRIES ARRAY_SIZE(virtio_gpu_debugfs_list)
for (i = 0 ; i < vgdev->num_scanouts; ++i)
kfree(vgdev->outputs[i].edid);
+ drm_atomic_helper_shutdown(vgdev->ddev);
drm_mode_config_cleanup(vgdev->ddev);
}
.prime_fd_to_handle = drm_gem_prime_fd_to_handle,
.gem_prime_export = drm_gem_prime_export,
.gem_prime_import = drm_gem_prime_import,
- .gem_prime_pin = virtgpu_gem_prime_pin,
- .gem_prime_unpin = virtgpu_gem_prime_unpin,
.gem_prime_get_sg_table = virtgpu_gem_prime_get_sg_table,
.gem_prime_import_sg_table = virtgpu_gem_prime_import_sg_table,
.gem_prime_vmap = virtgpu_gem_prime_vmap,
#define DRIVER_MINOR 1
#define DRIVER_PATCHLEVEL 0
+struct virtio_gpu_object_params {
+ uint32_t format;
+ uint32_t width;
+ uint32_t height;
+ unsigned long size;
+ bool dumb;
+ /* 3d */
+ bool virgl;
+ uint32_t target;
+ uint32_t bind;
+ uint32_t depth;
+ uint32_t array_size;
+ uint32_t last_level;
+ uint32_t nr_samples;
+ uint32_t flags;
+};
+
struct virtio_gpu_object {
struct drm_gem_object gem_base;
uint32_t hw_res_handle;
/* virtio_ioctl.c */
#define DRM_VIRTIO_NUM_IOCTLS 10
extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS];
+int virtio_gpu_object_list_validate(struct ww_acquire_ctx *ticket,
+ struct list_head *head);
+void virtio_gpu_unref_list(struct list_head *head);
/* virtio_kms.c */
int virtio_gpu_init(struct drm_device *dev);
void virtio_gpu_gem_fini(struct virtio_gpu_device *vgdev);
int virtio_gpu_gem_create(struct drm_file *file,
struct drm_device *dev,
- uint64_t size,
+ struct virtio_gpu_object_params *params,
struct drm_gem_object **obj_p,
uint32_t *handle_p);
int virtio_gpu_gem_object_open(struct drm_gem_object *obj,
struct drm_file *file);
void virtio_gpu_gem_object_close(struct drm_gem_object *obj,
struct drm_file *file);
-struct virtio_gpu_object *virtio_gpu_alloc_object(struct drm_device *dev,
- size_t size, bool kernel,
- bool pinned);
+struct virtio_gpu_object*
+virtio_gpu_alloc_object(struct drm_device *dev,
+ struct virtio_gpu_object_params *params,
+ struct virtio_gpu_fence *fence);
int virtio_gpu_mode_dumb_create(struct drm_file *file_priv,
struct drm_device *dev,
struct drm_mode_create_dumb *args);
void virtio_gpu_free_vbufs(struct virtio_gpu_device *vgdev);
void virtio_gpu_cmd_create_resource(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object *bo,
- uint32_t format,
- uint32_t width,
- uint32_t height);
+ struct virtio_gpu_object_params *params,
+ struct virtio_gpu_fence *fence);
void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,
uint32_t resource_id);
void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,
void
virtio_gpu_cmd_resource_create_3d(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object *bo,
- struct virtio_gpu_resource_create_3d *rc_3d);
+ struct virtio_gpu_object_params *params,
+ struct virtio_gpu_fence *fence);
void virtio_gpu_ctrl_ack(struct virtqueue *vq);
void virtio_gpu_cursor_ack(struct virtqueue *vq);
void virtio_gpu_fence_ack(struct virtqueue *vq);
int virtio_gpu_mmap(struct file *filp, struct vm_area_struct *vma);
/* virtio_gpu_fence.c */
+bool virtio_fence_signaled(struct dma_fence *f);
struct virtio_gpu_fence *virtio_gpu_fence_alloc(
struct virtio_gpu_device *vgdev);
int virtio_gpu_fence_emit(struct virtio_gpu_device *vgdev,
/* virtio_gpu_object */
int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,
- unsigned long size, bool kernel, bool pinned,
- struct virtio_gpu_object **bo_ptr);
+ struct virtio_gpu_object_params *params,
+ struct virtio_gpu_object **bo_ptr,
+ struct virtio_gpu_fence *fence);
void virtio_gpu_object_kunmap(struct virtio_gpu_object *bo);
int virtio_gpu_object_kmap(struct virtio_gpu_object *bo);
int virtio_gpu_object_get_sg_table(struct virtio_gpu_device *qdev,
int virtio_gpu_object_wait(struct virtio_gpu_object *bo, bool no_wait);
/* virtgpu_prime.c */
-int virtgpu_gem_prime_pin(struct drm_gem_object *obj);
-void virtgpu_gem_prime_unpin(struct drm_gem_object *obj);
struct sg_table *virtgpu_gem_prime_get_sg_table(struct drm_gem_object *obj);
struct drm_gem_object *virtgpu_gem_prime_import_sg_table(
struct drm_device *dev, struct dma_buf_attachment *attach,
return "controlq";
}
-static bool virtio_signaled(struct dma_fence *f)
+bool virtio_fence_signaled(struct dma_fence *f)
{
struct virtio_gpu_fence *fence = to_virtio_fence(f);
static const struct dma_fence_ops virtio_fence_ops = {
.get_driver_name = virtio_get_driver_name,
.get_timeline_name = virtio_get_timeline_name,
- .signaled = virtio_signaled,
+ .signaled = virtio_fence_signaled,
.fence_value_str = virtio_fence_value_str,
.timeline_value_str = virtio_timeline_value_str,
};
virtio_gpu_object_unref(&obj);
}
-struct virtio_gpu_object *virtio_gpu_alloc_object(struct drm_device *dev,
- size_t size, bool kernel,
- bool pinned)
+struct virtio_gpu_object*
+virtio_gpu_alloc_object(struct drm_device *dev,
+ struct virtio_gpu_object_params *params,
+ struct virtio_gpu_fence *fence)
{
struct virtio_gpu_device *vgdev = dev->dev_private;
struct virtio_gpu_object *obj;
int ret;
- ret = virtio_gpu_object_create(vgdev, size, kernel, pinned, &obj);
+ ret = virtio_gpu_object_create(vgdev, params, &obj, fence);
if (ret)
return ERR_PTR(ret);
int virtio_gpu_gem_create(struct drm_file *file,
struct drm_device *dev,
- uint64_t size,
+ struct virtio_gpu_object_params *params,
struct drm_gem_object **obj_p,
uint32_t *handle_p)
{
int ret;
u32 handle;
- obj = virtio_gpu_alloc_object(dev, size, false, false);
+ obj = virtio_gpu_alloc_object(dev, params, NULL);
if (IS_ERR(obj))
return PTR_ERR(obj);
struct drm_device *dev,
struct drm_mode_create_dumb *args)
{
- struct virtio_gpu_device *vgdev = dev->dev_private;
struct drm_gem_object *gobj;
- struct virtio_gpu_object *obj;
+ struct virtio_gpu_object_params params = { 0 };
int ret;
uint32_t pitch;
- uint32_t format;
if (args->bpp != 32)
return -EINVAL;
args->size = pitch * args->height;
args->size = ALIGN(args->size, PAGE_SIZE);
- ret = virtio_gpu_gem_create(file_priv, dev, args->size, &gobj,
+ params.format = virtio_gpu_translate_format(DRM_FORMAT_HOST_XRGB8888);
+ params.width = args->width;
+ params.height = args->height;
+ params.size = args->size;
+ params.dumb = true;
+ ret = virtio_gpu_gem_create(file_priv, dev, ¶ms, &gobj,
&args->handle);
if (ret)
goto fail;
- format = virtio_gpu_translate_format(DRM_FORMAT_HOST_XRGB8888);
- obj = gem_to_virtio_gpu_obj(gobj);
- virtio_gpu_cmd_create_resource(vgdev, obj, format,
- args->width, args->height);
-
- /* attach the object to the resource */
- ret = virtio_gpu_object_attach(vgdev, obj, NULL);
- if (ret)
- goto fail;
-
- obj->dumb = true;
args->pitch = pitch;
return ret;
&virtio_gpu_map->offset);
}
-static int virtio_gpu_object_list_validate(struct ww_acquire_ctx *ticket,
- struct list_head *head)
+int virtio_gpu_object_list_validate(struct ww_acquire_ctx *ticket,
+ struct list_head *head)
{
struct ttm_operation_ctx ctx = { false, false };
struct ttm_validate_buffer *buf;
return 0;
}
-static void virtio_gpu_unref_list(struct list_head *head)
+void virtio_gpu_unref_list(struct list_head *head)
{
struct ttm_validate_buffer *buf;
struct ttm_buffer_object *bo;
{
struct virtio_gpu_device *vgdev = dev->dev_private;
struct drm_virtgpu_resource_create *rc = data;
+ struct virtio_gpu_fence *fence;
int ret;
struct virtio_gpu_object *qobj;
struct drm_gem_object *obj;
uint32_t handle = 0;
- uint32_t size;
- struct list_head validate_list;
- struct ttm_validate_buffer mainbuf;
- struct virtio_gpu_fence *fence = NULL;
- struct ww_acquire_ctx ticket;
- struct virtio_gpu_resource_create_3d rc_3d;
+ struct virtio_gpu_object_params params = { 0 };
if (vgdev->has_virgl_3d == false) {
if (rc->depth > 1)
return -EINVAL;
}
- INIT_LIST_HEAD(&validate_list);
- memset(&mainbuf, 0, sizeof(struct ttm_validate_buffer));
-
- size = rc->size;
-
+ params.format = rc->format;
+ params.width = rc->width;
+ params.height = rc->height;
+ params.size = rc->size;
+ if (vgdev->has_virgl_3d) {
+ params.virgl = true;
+ params.target = rc->target;
+ params.bind = rc->bind;
+ params.depth = rc->depth;
+ params.array_size = rc->array_size;
+ params.last_level = rc->last_level;
+ params.nr_samples = rc->nr_samples;
+ params.flags = rc->flags;
+ }
/* allocate a single page size object */
- if (size == 0)
- size = PAGE_SIZE;
+ if (params.size == 0)
+ params.size = PAGE_SIZE;
- qobj = virtio_gpu_alloc_object(dev, size, false, false);
+ fence = virtio_gpu_fence_alloc(vgdev);
+ if (!fence)
+ return -ENOMEM;
+ qobj = virtio_gpu_alloc_object(dev, ¶ms, fence);
+ dma_fence_put(&fence->f);
if (IS_ERR(qobj))
return PTR_ERR(qobj);
obj = &qobj->gem_base;
- if (!vgdev->has_virgl_3d) {
- virtio_gpu_cmd_create_resource(vgdev, qobj, rc->format,
- rc->width, rc->height);
-
- ret = virtio_gpu_object_attach(vgdev, qobj, NULL);
- } else {
- /* use a gem reference since unref list undoes them */
- drm_gem_object_get(&qobj->gem_base);
- mainbuf.bo = &qobj->tbo;
- list_add(&mainbuf.head, &validate_list);
-
- ret = virtio_gpu_object_list_validate(&ticket, &validate_list);
- if (ret) {
- DRM_DEBUG("failed to validate\n");
- goto fail_unref;
- }
-
- rc_3d.resource_id = cpu_to_le32(qobj->hw_res_handle);
- rc_3d.target = cpu_to_le32(rc->target);
- rc_3d.format = cpu_to_le32(rc->format);
- rc_3d.bind = cpu_to_le32(rc->bind);
- rc_3d.width = cpu_to_le32(rc->width);
- rc_3d.height = cpu_to_le32(rc->height);
- rc_3d.depth = cpu_to_le32(rc->depth);
- rc_3d.array_size = cpu_to_le32(rc->array_size);
- rc_3d.last_level = cpu_to_le32(rc->last_level);
- rc_3d.nr_samples = cpu_to_le32(rc->nr_samples);
- rc_3d.flags = cpu_to_le32(rc->flags);
-
- fence = virtio_gpu_fence_alloc(vgdev);
- if (!fence) {
- ret = -ENOMEM;
- goto fail_backoff;
- }
-
- virtio_gpu_cmd_resource_create_3d(vgdev, qobj, &rc_3d);
- ret = virtio_gpu_object_attach(vgdev, qobj, fence);
- if (ret) {
- dma_fence_put(&fence->f);
- goto fail_backoff;
- }
- ttm_eu_fence_buffer_objects(&ticket, &validate_list, &fence->f);
- }
-
ret = drm_gem_handle_create(file_priv, obj, &handle);
if (ret) {
-
drm_gem_object_release(obj);
- if (vgdev->has_virgl_3d) {
- virtio_gpu_unref_list(&validate_list);
- dma_fence_put(&fence->f);
- }
return ret;
}
drm_gem_object_put_unlocked(obj);
rc->res_handle = qobj->hw_res_handle; /* similiar to a VM address */
rc->bo_handle = handle;
-
- if (vgdev->has_virgl_3d) {
- virtio_gpu_unref_list(&validate_list);
- dma_fence_put(&fence->f);
- }
return 0;
-fail_backoff:
- ttm_eu_backoff_reservation(&ticket, &validate_list);
-fail_unref:
- if (vgdev->has_virgl_3d) {
- virtio_gpu_unref_list(&validate_list);
- dma_fence_put(&fence->f);
- }
-//fail_obj:
-// drm_gem_object_handle_unreference_unlocked(obj);
- return ret;
}
static int virtio_gpu_resource_info_ioctl(struct drm_device *dev, void *data,
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
+#include <drm/ttm/ttm_execbuf_util.h>
+
#include "virtgpu_drv.h"
static int virtio_gpu_resource_id_get(struct virtio_gpu_device *vgdev,
kfree(bo);
}
-static void virtio_gpu_init_ttm_placement(struct virtio_gpu_object *vgbo,
- bool pinned)
+static void virtio_gpu_init_ttm_placement(struct virtio_gpu_object *vgbo)
{
u32 c = 1;
- u32 pflag = pinned ? TTM_PL_FLAG_NO_EVICT : 0;
vgbo->placement.placement = &vgbo->placement_code;
vgbo->placement.busy_placement = &vgbo->placement_code;
vgbo->placement_code.fpfn = 0;
vgbo->placement_code.lpfn = 0;
vgbo->placement_code.flags =
- TTM_PL_MASK_CACHING | TTM_PL_FLAG_TT | pflag;
+ TTM_PL_MASK_CACHING | TTM_PL_FLAG_TT |
+ TTM_PL_FLAG_NO_EVICT;
vgbo->placement.num_placement = c;
vgbo->placement.num_busy_placement = c;
}
int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,
- unsigned long size, bool kernel, bool pinned,
- struct virtio_gpu_object **bo_ptr)
+ struct virtio_gpu_object_params *params,
+ struct virtio_gpu_object **bo_ptr,
+ struct virtio_gpu_fence *fence)
{
struct virtio_gpu_object *bo;
- enum ttm_bo_type type;
size_t acc_size;
int ret;
- if (kernel)
- type = ttm_bo_type_kernel;
- else
- type = ttm_bo_type_device;
*bo_ptr = NULL;
- acc_size = ttm_bo_dma_acc_size(&vgdev->mman.bdev, size,
+ acc_size = ttm_bo_dma_acc_size(&vgdev->mman.bdev, params->size,
sizeof(struct virtio_gpu_object));
bo = kzalloc(sizeof(struct virtio_gpu_object), GFP_KERNEL);
kfree(bo);
return ret;
}
- size = roundup(size, PAGE_SIZE);
- ret = drm_gem_object_init(vgdev->ddev, &bo->gem_base, size);
+ params->size = roundup(params->size, PAGE_SIZE);
+ ret = drm_gem_object_init(vgdev->ddev, &bo->gem_base, params->size);
if (ret != 0) {
virtio_gpu_resource_id_put(vgdev, bo->hw_res_handle);
kfree(bo);
return ret;
}
- bo->dumb = false;
- virtio_gpu_init_ttm_placement(bo, pinned);
+ bo->dumb = params->dumb;
+
+ if (params->virgl) {
+ virtio_gpu_cmd_resource_create_3d(vgdev, bo, params, fence);
+ } else {
+ virtio_gpu_cmd_create_resource(vgdev, bo, params, fence);
+ }
- ret = ttm_bo_init(&vgdev->mman.bdev, &bo->tbo, size, type,
- &bo->placement, 0, !kernel, acc_size,
- NULL, NULL, &virtio_gpu_ttm_bo_destroy);
+ virtio_gpu_init_ttm_placement(bo);
+ ret = ttm_bo_init(&vgdev->mman.bdev, &bo->tbo, params->size,
+ ttm_bo_type_device, &bo->placement, 0,
+ true, acc_size, NULL, NULL,
+ &virtio_gpu_ttm_bo_destroy);
/* ttm_bo_init failure will call the destroy */
if (ret != 0)
return ret;
+ if (fence) {
+ struct virtio_gpu_fence_driver *drv = &vgdev->fence_drv;
+ struct list_head validate_list;
+ struct ttm_validate_buffer mainbuf;
+ struct ww_acquire_ctx ticket;
+ unsigned long irq_flags;
+ bool signaled;
+
+ INIT_LIST_HEAD(&validate_list);
+ memset(&mainbuf, 0, sizeof(struct ttm_validate_buffer));
+
+ /* use a gem reference since unref list undoes them */
+ drm_gem_object_get(&bo->gem_base);
+ mainbuf.bo = &bo->tbo;
+ list_add(&mainbuf.head, &validate_list);
+
+ ret = virtio_gpu_object_list_validate(&ticket, &validate_list);
+ if (ret == 0) {
+ spin_lock_irqsave(&drv->lock, irq_flags);
+ signaled = virtio_fence_signaled(&fence->f);
+ if (!signaled)
+ /* virtio create command still in flight */
+ ttm_eu_fence_buffer_objects(&ticket, &validate_list,
+ &fence->f);
+ spin_unlock_irqrestore(&drv->lock, irq_flags);
+ if (signaled)
+ /* virtio create command finished */
+ ttm_eu_backoff_reservation(&ticket, &validate_list);
+ }
+ virtio_gpu_unref_list(&validate_list);
+ }
+
*bo_ptr = bo;
return 0;
}
* device that might share buffers with virtgpu
*/
-int virtgpu_gem_prime_pin(struct drm_gem_object *obj)
+struct sg_table *virtgpu_gem_prime_get_sg_table(struct drm_gem_object *obj)
{
- WARN_ONCE(1, "not implemented");
- return -ENODEV;
-}
+ struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(obj);
-void virtgpu_gem_prime_unpin(struct drm_gem_object *obj)
-{
- WARN_ONCE(1, "not implemented");
-}
+ if (!bo->tbo.ttm->pages || !bo->tbo.ttm->num_pages)
+ /* should not happen */
+ return ERR_PTR(-EINVAL);
-struct sg_table *virtgpu_gem_prime_get_sg_table(struct drm_gem_object *obj)
-{
- return ERR_PTR(-ENODEV);
+ return drm_prime_pages_to_sg(bo->tbo.ttm->pages,
+ bo->tbo.ttm->num_pages);
}
struct drm_gem_object *virtgpu_gem_prime_import_sg_table(
}
int virtgpu_gem_prime_mmap(struct drm_gem_object *obj,
- struct vm_area_struct *area)
+ struct vm_area_struct *vma)
{
- return -ENODEV;
+ struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(obj);
+
+ bo->gem_base.vma_node.vm_node.start = bo->tbo.vma_node.vm_node.start;
+ return drm_gem_prime_mmap(obj, vma);
}
#include <linux/delay.h>
-#define DRM_FILE_PAGE_OFFSET (0x100000000ULL >> PAGE_SHIFT)
-
static struct
virtio_gpu_device *virtio_gpu_get_vgdev(struct ttm_bo_device *bdev)
{
static int virtio_gpu_init_mem_type(struct ttm_bo_device *bdev, uint32_t type,
struct ttm_mem_type_manager *man)
{
- struct virtio_gpu_device *vgdev;
-
- vgdev = virtio_gpu_get_vgdev(bdev);
-
switch (type) {
case TTM_PL_SYSTEM:
/* System memory */
*/
struct virtio_gpu_ttm_tt {
struct ttm_dma_tt ttm;
- struct virtio_gpu_device *vgdev;
- u64 offset;
+ struct virtio_gpu_object *obj;
};
-static int virtio_gpu_ttm_backend_bind(struct ttm_tt *ttm,
- struct ttm_mem_reg *bo_mem)
+static int virtio_gpu_ttm_tt_bind(struct ttm_tt *ttm,
+ struct ttm_mem_reg *bo_mem)
{
- struct virtio_gpu_ttm_tt *gtt = (void *)ttm;
+ struct virtio_gpu_ttm_tt *gtt =
+ container_of(ttm, struct virtio_gpu_ttm_tt, ttm.ttm);
+ struct virtio_gpu_device *vgdev =
+ virtio_gpu_get_vgdev(gtt->obj->tbo.bdev);
- gtt->offset = (unsigned long)(bo_mem->start << PAGE_SHIFT);
- if (!ttm->num_pages)
- WARN(1, "nothing to bind %lu pages for mreg %p back %p!\n",
- ttm->num_pages, bo_mem, ttm);
-
- /* Not implemented */
+ virtio_gpu_object_attach(vgdev, gtt->obj, NULL);
return 0;
}
-static int virtio_gpu_ttm_backend_unbind(struct ttm_tt *ttm)
+static int virtio_gpu_ttm_tt_unbind(struct ttm_tt *ttm)
{
- /* Not implemented */
+ struct virtio_gpu_ttm_tt *gtt =
+ container_of(ttm, struct virtio_gpu_ttm_tt, ttm.ttm);
+ struct virtio_gpu_device *vgdev =
+ virtio_gpu_get_vgdev(gtt->obj->tbo.bdev);
+
+ virtio_gpu_object_detach(vgdev, gtt->obj);
return 0;
}
-static void virtio_gpu_ttm_backend_destroy(struct ttm_tt *ttm)
+static void virtio_gpu_ttm_tt_destroy(struct ttm_tt *ttm)
{
- struct virtio_gpu_ttm_tt *gtt = (void *)ttm;
+ struct virtio_gpu_ttm_tt *gtt =
+ container_of(ttm, struct virtio_gpu_ttm_tt, ttm.ttm);
ttm_dma_tt_fini(>t->ttm);
kfree(gtt);
}
-static struct ttm_backend_func virtio_gpu_backend_func = {
- .bind = &virtio_gpu_ttm_backend_bind,
- .unbind = &virtio_gpu_ttm_backend_unbind,
- .destroy = &virtio_gpu_ttm_backend_destroy,
+static struct ttm_backend_func virtio_gpu_tt_func = {
+ .bind = &virtio_gpu_ttm_tt_bind,
+ .unbind = &virtio_gpu_ttm_tt_unbind,
+ .destroy = &virtio_gpu_ttm_tt_destroy,
};
static struct ttm_tt *virtio_gpu_ttm_tt_create(struct ttm_buffer_object *bo,
gtt = kzalloc(sizeof(struct virtio_gpu_ttm_tt), GFP_KERNEL);
if (gtt == NULL)
return NULL;
- gtt->ttm.ttm.func = &virtio_gpu_backend_func;
- gtt->vgdev = vgdev;
+ gtt->ttm.ttm.func = &virtio_gpu_tt_func;
+ gtt->obj = container_of(bo, struct virtio_gpu_object, tbo);
if (ttm_dma_tt_init(>t->ttm, bo, page_flags)) {
kfree(gtt);
return NULL;
return >t->ttm.ttm;
}
-static void virtio_gpu_move_null(struct ttm_buffer_object *bo,
- struct ttm_mem_reg *new_mem)
-{
- struct ttm_mem_reg *old_mem = &bo->mem;
-
- BUG_ON(old_mem->mm_node != NULL);
- *old_mem = *new_mem;
- new_mem->mm_node = NULL;
-}
-
-static int virtio_gpu_bo_move(struct ttm_buffer_object *bo, bool evict,
- struct ttm_operation_ctx *ctx,
- struct ttm_mem_reg *new_mem)
-{
- int ret;
-
- ret = ttm_bo_wait(bo, ctx->interruptible, ctx->no_wait_gpu);
- if (ret)
- return ret;
-
- virtio_gpu_move_null(bo, new_mem);
- return 0;
-}
-
-static void virtio_gpu_bo_move_notify(struct ttm_buffer_object *tbo,
- bool evict,
- struct ttm_mem_reg *new_mem)
-{
- struct virtio_gpu_object *bo;
- struct virtio_gpu_device *vgdev;
-
- bo = container_of(tbo, struct virtio_gpu_object, tbo);
- vgdev = (struct virtio_gpu_device *)bo->gem_base.dev->dev_private;
-
- if (!new_mem || (new_mem->placement & TTM_PL_FLAG_SYSTEM)) {
- if (bo->hw_res_handle)
- virtio_gpu_object_detach(vgdev, bo);
-
- } else if (new_mem->placement & TTM_PL_FLAG_TT) {
- if (bo->hw_res_handle) {
- virtio_gpu_object_attach(vgdev, bo, NULL);
- }
- }
-}
-
static void virtio_gpu_bo_swap_notify(struct ttm_buffer_object *tbo)
{
struct virtio_gpu_object *bo;
- struct virtio_gpu_device *vgdev;
bo = container_of(tbo, struct virtio_gpu_object, tbo);
- vgdev = (struct virtio_gpu_device *)bo->gem_base.dev->dev_private;
if (bo->pages)
virtio_gpu_object_free_sg_table(bo);
.init_mem_type = &virtio_gpu_init_mem_type,
.eviction_valuable = ttm_bo_eviction_valuable,
.evict_flags = &virtio_gpu_evict_flags,
- .move = &virtio_gpu_bo_move,
.verify_access = &virtio_gpu_verify_access,
.io_mem_reserve = &virtio_gpu_ttm_io_mem_reserve,
.io_mem_free = &virtio_gpu_ttm_io_mem_free,
- .move_notify = &virtio_gpu_bo_move_notify,
.swap_notify = &virtio_gpu_bo_swap_notify,
};
r = ttm_bo_device_init(&vgdev->mman.bdev,
&virtio_gpu_bo_driver,
vgdev->ddev->anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET, 0);
+ false);
if (r) {
DRM_ERROR("failed initializing buffer object driver(%d).\n", r);
goto err_dev_init;
/* create a basic resource */
void virtio_gpu_cmd_create_resource(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object *bo,
- uint32_t format,
- uint32_t width,
- uint32_t height)
+ struct virtio_gpu_object_params *params,
+ struct virtio_gpu_fence *fence)
{
struct virtio_gpu_resource_create_2d *cmd_p;
struct virtio_gpu_vbuffer *vbuf;
cmd_p->hdr.type = cpu_to_le32(VIRTIO_GPU_CMD_RESOURCE_CREATE_2D);
cmd_p->resource_id = cpu_to_le32(bo->hw_res_handle);
- cmd_p->format = cpu_to_le32(format);
- cmd_p->width = cpu_to_le32(width);
- cmd_p->height = cpu_to_le32(height);
+ cmd_p->format = cpu_to_le32(params->format);
+ cmd_p->width = cpu_to_le32(params->width);
+ cmd_p->height = cpu_to_le32(params->height);
- virtio_gpu_queue_ctrl_buffer(vgdev, vbuf);
+ virtio_gpu_queue_fenced_ctrl_buffer(vgdev, vbuf, &cmd_p->hdr, fence);
bo->created = true;
}
void
virtio_gpu_cmd_resource_create_3d(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object *bo,
- struct virtio_gpu_resource_create_3d *rc_3d)
+ struct virtio_gpu_object_params *params,
+ struct virtio_gpu_fence *fence)
{
struct virtio_gpu_resource_create_3d *cmd_p;
struct virtio_gpu_vbuffer *vbuf;
cmd_p = virtio_gpu_alloc_cmd(vgdev, &vbuf, sizeof(*cmd_p));
memset(cmd_p, 0, sizeof(*cmd_p));
- *cmd_p = *rc_3d;
cmd_p->hdr.type = cpu_to_le32(VIRTIO_GPU_CMD_RESOURCE_CREATE_3D);
- cmd_p->hdr.flags = 0;
+ cmd_p->resource_id = cpu_to_le32(bo->hw_res_handle);
+ cmd_p->format = cpu_to_le32(params->format);
+ cmd_p->width = cpu_to_le32(params->width);
+ cmd_p->height = cpu_to_le32(params->height);
+
+ cmd_p->target = cpu_to_le32(params->target);
+ cmd_p->bind = cpu_to_le32(params->bind);
+ cmd_p->depth = cpu_to_le32(params->depth);
+ cmd_p->array_size = cpu_to_le32(params->array_size);
+ cmd_p->last_level = cpu_to_le32(params->last_level);
+ cmd_p->nr_samples = cpu_to_le32(params->nr_samples);
+ cmd_p->flags = cpu_to_le32(params->flags);
- virtio_gpu_queue_ctrl_buffer(vgdev, vbuf);
+ virtio_gpu_queue_fenced_ctrl_buffer(vgdev, vbuf, &cmd_p->hdr, fence);
bo->created = true;
}
struct scatterlist *sg;
int si, nents;
- if (!obj->created)
- return 0;
+ if (WARN_ON_ONCE(!obj->created))
+ return -EINVAL;
if (!obj->pages) {
int ret;
spin_lock_init(&vkms_out->state_lock);
vkms_out->crc_workq = alloc_ordered_workqueue("vkms_crc_workq", 0);
+ if (!vkms_out->crc_workq)
+ return -ENOMEM;
return ret;
}
SVGA3dCmdSetShader body;
} *cmd;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for shader "
- "unbinding.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_SET_SHADER;
cmd->header.size = sizeof(cmd->body);
SVGA3dCmdSetRenderTarget body;
} *cmd;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for render target "
- "unbinding.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_SETRENDERTARGET;
cmd->header.size = sizeof(cmd->body);
} body;
} *cmd;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for texture "
- "unbinding.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_SETTEXTURESTATE;
cmd->header.size = sizeof(cmd->body);
SVGA3dCmdDXSetShader body;
} *cmd;
- cmd = vmw_fifo_reserve_dx(dev_priv, sizeof(*cmd), bi->ctx->id);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for DX shader "
- "unbinding.\n");
+ cmd = VMW_FIFO_RESERVE_DX(dev_priv, sizeof(*cmd), bi->ctx->id);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
+
cmd->header.id = SVGA_3D_CMD_DX_SET_SHADER;
cmd->header.size = sizeof(cmd->body);
cmd->body.type = binding->shader_slot + SVGA3D_SHADERTYPE_MIN;
SVGA3dCmdDXSetSingleConstantBuffer body;
} *cmd;
- cmd = vmw_fifo_reserve_dx(dev_priv, sizeof(*cmd), bi->ctx->id);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for DX shader "
- "unbinding.\n");
+ cmd = VMW_FIFO_RESERVE_DX(dev_priv, sizeof(*cmd), bi->ctx->id);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_SET_SINGLE_CONSTANT_BUFFER;
cmd->header.size = sizeof(cmd->body);
view_id_size = cbs->bind_cmd_count*sizeof(uint32);
cmd_size = sizeof(*cmd) + view_id_size;
- cmd = vmw_fifo_reserve_dx(ctx->dev_priv, cmd_size, ctx->id);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for DX shader"
- " resource binding.\n");
+ cmd = VMW_FIFO_RESERVE_DX(ctx->dev_priv, cmd_size, ctx->id);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_SET_SHADER_RESOURCES;
cmd->header.size = sizeof(cmd->body) + view_id_size;
vmw_collect_view_ids(cbs, loc, SVGA3D_MAX_SIMULTANEOUS_RENDER_TARGETS);
view_id_size = cbs->bind_cmd_count*sizeof(uint32);
cmd_size = sizeof(*cmd) + view_id_size;
- cmd = vmw_fifo_reserve_dx(ctx->dev_priv, cmd_size, ctx->id);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for DX render-target"
- " binding.\n");
+ cmd = VMW_FIFO_RESERVE_DX(ctx->dev_priv, cmd_size, ctx->id);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_SET_RENDERTARGETS;
cmd->header.size = sizeof(cmd->body) + view_id_size;
so_target_size = cbs->bind_cmd_count*sizeof(SVGA3dSoTarget);
cmd_size = sizeof(*cmd) + so_target_size;
- cmd = vmw_fifo_reserve_dx(ctx->dev_priv, cmd_size, ctx->id);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for DX SO target"
- " binding.\n");
+ cmd = VMW_FIFO_RESERVE_DX(ctx->dev_priv, cmd_size, ctx->id);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_SET_SOTARGETS;
cmd->header.size = sizeof(cmd->body) + so_target_size;
set_vb_size = cbs->bind_cmd_count*sizeof(SVGA3dVertexBuffer);
cmd_size = sizeof(*cmd) + set_vb_size;
- cmd = vmw_fifo_reserve_dx(ctx->dev_priv, cmd_size, ctx->id);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for DX vertex buffer"
- " binding.\n");
+ cmd = VMW_FIFO_RESERVE_DX(ctx->dev_priv, cmd_size, ctx->id);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_SET_VERTEX_BUFFERS;
cmd->header.size = sizeof(cmd->body) + set_vb_size;
SVGA3dCmdDXSetIndexBuffer body;
} *cmd;
- cmd = vmw_fifo_reserve_dx(dev_priv, sizeof(*cmd), bi->ctx->id);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for DX index buffer "
- "binding.\n");
+ cmd = VMW_FIFO_RESERVE_DX(dev_priv, sizeof(*cmd), bi->ctx->id);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
+
cmd->header.id = SVGA_3D_CMD_DX_SET_INDEX_BUFFER;
cmd->header.size = sizeof(cmd->body);
if (rebind) {
vmw_binding_drop(entry);
}
+/**
+ * vmw_binding_dirtying - Return whether a binding type is dirtying its resource
+ * @binding_type: The binding type
+ *
+ * Each time a resource is put on the validation list as the result of a
+ * context binding referencing it, we need to determine whether that resource
+ * will be dirtied (written to by the GPU) as a result of the corresponding
+ * GPU operation. Currently rendertarget-, depth-stencil-, and
+ * stream-output-target bindings are capable of dirtying its resource.
+ *
+ * Return: Whether the binding type dirties the resource its binding points to.
+ */
+u32 vmw_binding_dirtying(enum vmw_ctx_binding_type binding_type)
+{
+ static u32 is_binding_dirtying[vmw_ctx_binding_max] = {
+ [vmw_ctx_binding_rt] = VMW_RES_DIRTY_SET,
+ [vmw_ctx_binding_dx_rt] = VMW_RES_DIRTY_SET,
+ [vmw_ctx_binding_ds] = VMW_RES_DIRTY_SET,
+ [vmw_ctx_binding_so] = VMW_RES_DIRTY_SET,
+ };
+
+ /* Review this function as new bindings are added. */
+ BUILD_BUG_ON(vmw_ctx_binding_max != 11);
+ return is_binding_dirtying[binding_type];
+}
+
/*
* This function is unused at run-time, and only used to hold various build
* asserts important for code optimization assumptions.
extern struct list_head *
vmw_binding_state_list(struct vmw_ctx_binding_state *cbs);
extern void vmw_binding_state_reset(struct vmw_ctx_binding_state *cbs);
+extern u32 vmw_binding_dirtying(enum vmw_ctx_binding_type binding_type);
+
#endif
__vmw_cmdbuf_header_free(entry);
break;
case SVGA_CB_STATUS_COMMAND_ERROR:
+ WARN_ONCE(true, "Command buffer error.\n");
entry->cb_header->status = SVGA_CB_STATUS_NONE;
list_add_tail(&entry->list, &man->error);
schedule_work(&man->work);
container_of(work, struct vmw_cmdbuf_man, work);
struct vmw_cmdbuf_header *entry, *next;
uint32_t dummy;
- bool restart[SVGA_CB_CONTEXT_MAX];
bool send_fence = false;
struct list_head restart_head[SVGA_CB_CONTEXT_MAX];
int i;
struct vmw_cmdbuf_context *ctx;
bool global_block = false;
- for_each_cmdbuf_ctx(man, i, ctx) {
+ for_each_cmdbuf_ctx(man, i, ctx)
INIT_LIST_HEAD(&restart_head[i]);
- restart[i] = false;
- }
mutex_lock(&man->error_mutex);
spin_lock(&man->lock);
const char *cmd_name;
list_del_init(&entry->list);
- restart[entry->cb_context] = true;
global_block = true;
if (!vmw_cmd_describe(header, &error_cmd_size, &cmd_name)) {
- DRM_ERROR("Unknown command causing device error.\n");
- DRM_ERROR("Command buffer offset is %lu\n",
- (unsigned long) cb_hdr->errorOffset);
+ VMW_DEBUG_USER("Unknown command causing device error.\n");
+ VMW_DEBUG_USER("Command buffer offset is %lu\n",
+ (unsigned long) cb_hdr->errorOffset);
__vmw_cmdbuf_header_free(entry);
send_fence = true;
continue;
}
- DRM_ERROR("Command \"%s\" causing device error.\n", cmd_name);
- DRM_ERROR("Command buffer offset is %lu\n",
- (unsigned long) cb_hdr->errorOffset);
- DRM_ERROR("Command size is %lu\n",
- (unsigned long) error_cmd_size);
+ VMW_DEBUG_USER("Command \"%s\" causing device error.\n",
+ cmd_name);
+ VMW_DEBUG_USER("Command buffer offset is %lu\n",
+ (unsigned long) cb_hdr->errorOffset);
+ VMW_DEBUG_USER("Command size is %lu\n",
+ (unsigned long) error_cmd_size);
new_start_offset = cb_hdr->errorOffset + error_cmd_size;
}
vmw_execbuf_release_pinned_bo(dev_priv);
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for surface "
- "destruction.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return;
- }
cmd->header.id = SVGA_3D_CMD_CONTEXT_DESTROY;
cmd->header.size = sizeof(cmd->body);
for (i = 0; i < SVGA_COTABLE_DX10_MAX; ++i) {
uctx->cotables[i] = vmw_cotable_alloc(dev_priv,
&uctx->res, i);
- if (unlikely(IS_ERR(uctx->cotables[i]))) {
+ if (IS_ERR(uctx->cotables[i])) {
ret = PTR_ERR(uctx->cotables[i]);
goto out_cotables;
}
return -ENOMEM;
}
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
if (unlikely(cmd == NULL)) {
- DRM_ERROR("Fifo reserve failed.\n");
vmw_resource_unreference(&res);
return -ENOMEM;
}
goto out_no_fifo;
}
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for context "
- "creation.\n");
ret = -ENOMEM;
goto out_no_fifo;
}
BUG_ON(bo->mem.mem_type != VMW_PL_MOB);
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for context "
- "binding.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
+
cmd->header.id = SVGA_3D_CMD_BIND_GB_CONTEXT;
cmd->header.size = sizeof(cmd->body);
cmd->body.cid = res->id;
submit_size = sizeof(*cmd2) + (readback ? sizeof(*cmd1) : 0);
- cmd = vmw_fifo_reserve(dev_priv, submit_size);
+ cmd = VMW_FIFO_RESERVE(dev_priv, submit_size);
if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for context "
- "unbinding.\n");
mutex_unlock(&dev_priv->binding_mutex);
return -ENOMEM;
}
if (likely(res->id == -1))
return 0;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for context "
- "destruction.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DESTROY_GB_CONTEXT;
cmd->header.size = sizeof(cmd->body);
goto out_no_fifo;
}
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for context "
- "creation.\n");
ret = -ENOMEM;
goto out_no_fifo;
}
BUG_ON(bo->mem.mem_type != VMW_PL_MOB);
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for context "
- "binding.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_BIND_CONTEXT;
cmd->header.size = sizeof(cmd->body);
submit_size = sizeof(*cmd2) + (readback ? sizeof(*cmd1) : 0);
- cmd = vmw_fifo_reserve(dev_priv, submit_size);
+ cmd = VMW_FIFO_RESERVE(dev_priv, submit_size);
if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for context "
- "unbinding.\n");
mutex_unlock(&dev_priv->binding_mutex);
return -ENOMEM;
}
if (likely(res->id == -1))
return 0;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for context "
- "destruction.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_DESTROY_CONTEXT;
cmd->header.size = sizeof(cmd->body);
int ret;
if (!dev_priv->has_dx && dx) {
- DRM_ERROR("DX contexts not supported by device.\n");
+ VMW_DEBUG_USER("DX contexts not supported by device.\n");
return -EINVAL;
}
WARN_ON_ONCE(bo->mem.mem_type != VMW_PL_MOB);
lockdep_assert_held(&bo->resv->lock.base);
- cmd = vmw_fifo_reserve_dx(dev_priv, sizeof(*cmd), SVGA3D_INVALID_ID);
- if (!cmd) {
- DRM_ERROR("Failed reserving FIFO space for cotable "
- "binding.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (!cmd)
return -ENOMEM;
- }
WARN_ON(vcotbl->ctx->id == SVGA3D_INVALID_ID);
WARN_ON(bo->mem.mem_type != VMW_PL_MOB);
if (readback)
submit_size += sizeof(*cmd0);
- cmd1 = vmw_fifo_reserve_dx(dev_priv, submit_size, SVGA3D_INVALID_ID);
- if (!cmd1) {
- DRM_ERROR("Failed reserving FIFO space for cotable "
- "unbinding.\n");
+ cmd1 = VMW_FIFO_RESERVE(dev_priv, submit_size);
+ if (!cmd1)
return -ENOMEM;
- }
vcotbl->size_read_back = 0;
if (readback) {
struct vmw_fence_obj *fence;
if (!vcotbl->scrubbed) {
- cmd = vmw_fifo_reserve_dx(dev_priv, sizeof(*cmd),
- SVGA3D_INVALID_ID);
- if (!cmd) {
- DRM_ERROR("Failed reserving FIFO space for cotable "
- "readback.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (!cmd)
return -ENOMEM;
- }
+
cmd->header.id = SVGA_3D_CMD_DX_READBACK_COTABLE;
cmd->header.size = sizeof(cmd->body);
cmd->body.cid = vcotbl->ctx->id;
ret = ttm_bo_device_init(&dev_priv->bdev,
&vmw_bo_driver,
dev->anon_inode->i_mapping,
- VMWGFX_FILE_PAGE_OFFSET,
false);
if (unlikely(ret != 0)) {
DRM_ERROR("Failed initializing TTM buffer object driver.\n");
#define VMWGFX_DRIVER_MAJOR 2
#define VMWGFX_DRIVER_MINOR 15
#define VMWGFX_DRIVER_PATCHLEVEL 0
-#define VMWGFX_FILE_PAGE_OFFSET 0x00100000
#define VMWGFX_FIFO_STATIC_SIZE (1024*1024)
#define VMWGFX_MAX_RELOCATIONS 2048
#define VMWGFX_MAX_VALIDATIONS 2048
uint32_t *inout_id,
struct vmw_resource **out);
extern void vmw_resource_unreserve(struct vmw_resource *res,
+ bool dirty_set,
+ bool dirty,
bool switch_backup,
struct vmw_buffer_object *new_backup,
unsigned long new_backup_offset);
struct vmw_fifo_state *fifo);
extern void vmw_fifo_release(struct vmw_private *dev_priv,
struct vmw_fifo_state *fifo);
-extern void *vmw_fifo_reserve(struct vmw_private *dev_priv, uint32_t bytes);
extern void *
vmw_fifo_reserve_dx(struct vmw_private *dev_priv, uint32_t bytes, int ctx_id);
extern void vmw_fifo_commit(struct vmw_private *dev_priv, uint32_t bytes);
extern int vmw_fifo_flush(struct vmw_private *dev_priv,
bool interruptible);
+#define VMW_FIFO_RESERVE_DX(__priv, __bytes, __ctx_id) \
+({ \
+ vmw_fifo_reserve_dx(__priv, __bytes, __ctx_id) ? : ({ \
+ DRM_ERROR("FIFO reserve failed at %s for %u bytes\n", \
+ __func__, (unsigned int) __bytes); \
+ NULL; \
+ }); \
+})
+
+#define VMW_FIFO_RESERVE(__priv, __bytes) \
+ VMW_FIFO_RESERVE_DX(__priv, __bytes, SVGA3D_INVALID_ID)
+
/**
* TTM glue - vmwgfx_ttm_glue.c
*/
char *buffer, size_t *length);
int vmw_host_log(const char *log);
+/* VMW logging */
+
+/**
+ * VMW_DEBUG_USER - Debug output for user-space debugging.
+ *
+ * @fmt: printf() like format string.
+ *
+ * This macro is for logging user-space error and debugging messages for e.g.
+ * command buffer execution errors due to malformed commands, invalid context,
+ * etc.
+ */
+#define VMW_DEBUG_USER(fmt, ...) \
+ DRM_DEBUG_DRIVER(fmt, ##__VA_ARGS__)
+
/**
* Inline helper functions
*/
#define VMW_RES_HT_ORDER 12
/*
+ * Helper macro to get dx_ctx_node if available otherwise print an error
+ * message. This is for use in command verifier function where if dx_ctx_node
+ * is not set then command is invalid.
+ */
+#define VMW_GET_CTX_NODE(__sw_context) \
+({ \
+ __sw_context->dx_ctx_node ? __sw_context->dx_ctx_node : ({ \
+ VMW_DEBUG_USER("SM context is not set at %s\n", __func__); \
+ __sw_context->dx_ctx_node; \
+ }); \
+})
+
+#define VMW_DECLARE_CMD_VAR(__var, __type) \
+ struct { \
+ SVGA3dCmdHeader header; \
+ __type body; \
+ } __var
+
+/**
* struct vmw_relocation - Buffer object relocation
*
* @head: List head for the command submission context's relocation list
* command stream is replaced with the actual id after validation.
* @vmw_res_rel_nop: NOP relocation. The command is unconditionally replaced
* with a NOP.
- * @vmw_res_rel_cond_nop: Conditional NOP relocation. If the resource id
- * after validation is -1, the command is replaced with a NOP. Otherwise no
- * action.
+ * @vmw_res_rel_cond_nop: Conditional NOP relocation. If the resource id after
+ * validation is -1, the command is replaced with a NOP. Otherwise no action.
*/
enum vmw_resource_relocation_type {
vmw_res_rel_normal,
*
* @head: List head for the software context's relocation list.
* @res: Non-ref-counted pointer to the resource.
- * @offset: Offset of single byte entries into the command buffer where the
- * id that needs fixup is located.
+ * @offset: Offset of single byte entries into the command buffer where the id
+ * that needs fixup is located.
* @rel_type: Type of relocation.
*/
struct vmw_resource_relocation {
enum vmw_resource_relocation_type rel_type:3;
};
-/*
+/**
* struct vmw_ctx_validation_info - Extra validation metadata for contexts
+ *
* @head: List head of context list
* @ctx: The context resource
* @cur: The context's persistent binding state
/**
* vmw_execbuf_bindings_commit - Commit modified binding state
+ *
* @sw_context: The command submission context
- * @backoff: Whether this is part of the error path and binding state
- * changes should be ignored
+ * @backoff: Whether this is part of the error path and binding state changes
+ * should be ignored
*/
static void vmw_execbuf_bindings_commit(struct vmw_sw_context *sw_context,
bool backoff)
list_for_each_entry(entry, &sw_context->ctx_list, head) {
if (!backoff)
vmw_binding_state_commit(entry->cur, entry->staged);
+
if (entry->staged != sw_context->staged_bindings)
vmw_binding_state_free(entry->staged);
else
/**
* vmw_bind_dx_query_mob - Bind the DX query MOB if referenced
+ *
* @sw_context: The command submission context
*/
static void vmw_bind_dx_query_mob(struct vmw_sw_context *sw_context)
}
/**
- * vmw_cmd_ctx_first_setup - Perform the setup needed when a context is
- * added to the validate list.
+ * vmw_cmd_ctx_first_setup - Perform the setup needed when a context is added to
+ * the validate list.
*
* @dev_priv: Pointer to the device private:
* @sw_context: The command submission context
goto out_err;
if (!sw_context->staged_bindings) {
- sw_context->staged_bindings =
- vmw_binding_state_alloc(dev_priv);
+ sw_context->staged_bindings = vmw_binding_state_alloc(dev_priv);
if (IS_ERR(sw_context->staged_bindings)) {
- DRM_ERROR("Failed to allocate context binding "
- "information.\n");
ret = PTR_ERR(sw_context->staged_bindings);
sw_context->staged_bindings = NULL;
goto out_err;
if (sw_context->staged_bindings_inuse) {
node->staged = vmw_binding_state_alloc(dev_priv);
if (IS_ERR(node->staged)) {
- DRM_ERROR("Failed to allocate context binding "
- "information.\n");
ret = PTR_ERR(node->staged);
node->staged = NULL;
goto out_err;
list_add_tail(&node->head, &sw_context->ctx_list);
return 0;
+
out_err:
return ret;
}
/**
- * vmw_execbuf_res_size - calculate extra size fore the resource validation
- * node
+ * vmw_execbuf_res_size - calculate extra size fore the resource validation node
+ *
* @dev_priv: Pointer to the device private struct.
* @res_type: The resource type.
*
- * Guest-backed contexts and DX contexts require extra size to store
- * execbuf private information in the validation node. Typically the
- * binding manager associated data structures.
+ * Guest-backed contexts and DX contexts require extra size to store execbuf
+ * private information in the validation node. Typically the binding manager
+ * associated data structures.
*
* Returns: The extra size requirement based on resource type.
*/
*
* @rcache: Pointer to the entry to update.
* @res: Pointer to the resource.
- * @private: Pointer to the execbuf-private space in the resource
- * validation node.
+ * @private: Pointer to the execbuf-private space in the resource validation
+ * node.
*/
static void vmw_execbuf_rcache_update(struct vmw_res_cache_entry *rcache,
struct vmw_resource *res,
}
/**
- * vmw_execbuf_res_noref_val_add - Add a resource described by an
- * unreferenced rcu-protected pointer to the validation list.
+ * vmw_execbuf_res_noref_val_add - Add a resource described by an unreferenced
+ * rcu-protected pointer to the validation list.
+ *
* @sw_context: Pointer to the software context.
* @res: Unreferenced rcu-protected pointer to the resource.
+ * @dirty: Whether to change dirty status.
*
- * Returns: 0 on success. Negative error code on failure. Typical error
- * codes are %-EINVAL on inconsistency and %-ESRCH if the resource was
- * doomed.
+ * Returns: 0 on success. Negative error code on failure. Typical error codes
+ * are %-EINVAL on inconsistency and %-ESRCH if the resource was doomed.
*/
static int vmw_execbuf_res_noref_val_add(struct vmw_sw_context *sw_context,
- struct vmw_resource *res)
+ struct vmw_resource *res,
+ u32 dirty)
{
struct vmw_private *dev_priv = res->dev_priv;
int ret;
rcache = &sw_context->res_cache[res_type];
if (likely(rcache->valid && rcache->res == res)) {
+ if (dirty)
+ vmw_validation_res_set_dirty(sw_context->ctx,
+ rcache->private, dirty);
vmw_user_resource_noref_release();
return 0;
}
priv_size = vmw_execbuf_res_size(dev_priv, res_type);
ret = vmw_validation_add_resource(sw_context->ctx, res, priv_size,
- (void **)&ctx_info, &first_usage);
+ dirty, (void **)&ctx_info,
+ &first_usage);
vmw_user_resource_noref_release();
if (ret)
return ret;
if (priv_size && first_usage) {
ret = vmw_cmd_ctx_first_setup(dev_priv, sw_context, res,
ctx_info);
- if (ret)
+ if (ret) {
+ VMW_DEBUG_USER("Failed first usage context setup.\n");
return ret;
+ }
}
vmw_execbuf_rcache_update(rcache, res, ctx_info);
/**
* vmw_execbuf_res_noctx_val_add - Add a non-context resource to the resource
* validation list if it's not already on it
+ *
* @sw_context: Pointer to the software context.
* @res: Pointer to the resource.
+ * @dirty: Whether to change dirty status.
*
* Returns: Zero on success. Negative error code on failure.
*/
static int vmw_execbuf_res_noctx_val_add(struct vmw_sw_context *sw_context,
- struct vmw_resource *res)
+ struct vmw_resource *res,
+ u32 dirty)
{
struct vmw_res_cache_entry *rcache;
enum vmw_res_type res_type = vmw_res_type(res);
int ret;
rcache = &sw_context->res_cache[res_type];
- if (likely(rcache->valid && rcache->res == res))
+ if (likely(rcache->valid && rcache->res == res)) {
+ if (dirty)
+ vmw_validation_res_set_dirty(sw_context->ctx,
+ rcache->private, dirty);
return 0;
+ }
- ret = vmw_validation_add_resource(sw_context->ctx, res, 0, &ptr, NULL);
+ ret = vmw_validation_add_resource(sw_context->ctx, res, 0, dirty,
+ &ptr, NULL);
if (ret)
return ret;
}
/**
- * vmw_view_res_val_add - Add a view and the surface it's pointing to
- * to the validation list
+ * vmw_view_res_val_add - Add a view and the surface it's pointing to to the
+ * validation list
*
* @sw_context: The software context holding the validation list.
* @view: Pointer to the view resource.
int ret;
/*
- * First add the resource the view is pointing to, otherwise
- * it may be swapped out when the view is validated.
+ * First add the resource the view is pointing to, otherwise it may be
+ * swapped out when the view is validated.
*/
- ret = vmw_execbuf_res_noctx_val_add(sw_context, vmw_view_srf(view));
+ ret = vmw_execbuf_res_noctx_val_add(sw_context, vmw_view_srf(view),
+ vmw_view_dirtying(view));
if (ret)
return ret;
- return vmw_execbuf_res_noctx_val_add(sw_context, view);
+ return vmw_execbuf_res_noctx_val_add(sw_context, view,
+ VMW_RES_DIRTY_NONE);
}
/**
- * vmw_view_id_val_add - Look up a view and add it and the surface it's
- * pointing to to the validation list.
+ * vmw_view_id_val_add - Look up a view and add it and the surface it's pointing
+ * to to the validation list.
*
* @sw_context: The software context holding the validation list.
* @view_type: The view type to look up.
* @id: view id of the view.
*
- * The view is represented by a view id and the DX context it's created on,
- * or scheduled for creation on. If there is no DX context set, the function
- * will return an -EINVAL error pointer.
+ * The view is represented by a view id and the DX context it's created on, or
+ * scheduled for creation on. If there is no DX context set, the function will
+ * return an -EINVAL error pointer.
*
* Returns: Unreferenced pointer to the resource on success, negative error
* pointer on failure.
struct vmw_resource *view;
int ret;
- if (!ctx_node) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return ERR_PTR(-EINVAL);
- }
view = vmw_view_lookup(sw_context->man, view_type, id);
if (IS_ERR(view))
* @sw_context: Pointer to a software context used for this command submission
* @ctx: Pointer to the context resource
*
- * This function puts all resources that were previously bound to @ctx on
- * the resource validation list. This is part of the context state reemission
+ * This function puts all resources that were previously bound to @ctx on the
+ * resource validation list. This is part of the context state reemission
*/
static int vmw_resource_context_res_add(struct vmw_private *dev_priv,
struct vmw_sw_context *sw_context,
if (IS_ERR(res))
continue;
- ret = vmw_execbuf_res_noctx_val_add(sw_context, res);
+ ret = vmw_execbuf_res_noctx_val_add(sw_context, res,
+ VMW_RES_DIRTY_SET);
if (unlikely(ret != 0))
return ret;
}
}
-
/* Add all resources bound to the context to the validation list */
mutex_lock(&dev_priv->binding_mutex);
binding_list = vmw_context_binding_list(ctx);
if (vmw_res_type(entry->res) == vmw_res_view)
ret = vmw_view_res_val_add(sw_context, entry->res);
else
- ret = vmw_execbuf_res_noctx_val_add(sw_context,
- entry->res);
+ ret = vmw_execbuf_res_noctx_val_add
+ (sw_context, entry->res,
+ vmw_binding_dirtying(entry->bt));
if (unlikely(ret != 0))
break;
}
*
* @list: Pointer to head of relocation list.
* @res: The resource.
- * @offset: Offset into the command buffer currently being parsed where the
- * id that needs fixup is located. Granularity is one byte.
+ * @offset: Offset into the command buffer currently being parsed where the id
+ * that needs fixup is located. Granularity is one byte.
* @rel_type: Relocation type.
*/
static int vmw_resource_relocation_add(struct vmw_sw_context *sw_context,
rel = vmw_validation_mem_alloc(sw_context->ctx, sizeof(*rel));
if (unlikely(!rel)) {
- DRM_ERROR("Failed to allocate a resource relocation.\n");
+ VMW_DEBUG_USER("Failed to allocate a resource relocation.\n");
return -ENOMEM;
}
static void vmw_resource_relocations_free(struct list_head *list)
{
/* Memory is validation context memory, so no need to free it */
-
INIT_LIST_HEAD(list);
}
/**
* vmw_resource_relocations_apply - Apply all relocations on a list
*
- * @cb: Pointer to the start of the command buffer bein patch. This need
- * not be the same buffer as the one being parsed when the relocation
- * list was built, but the contents must be the same modulo the
- * resource ids.
+ * @cb: Pointer to the start of the command buffer bein patch. This need not be
+ * the same buffer as the one being parsed when the relocation list was built,
+ * but the contents must be the same modulo the resource ids.
* @list: Pointer to the head of the relocation list.
*/
static void vmw_resource_relocations_apply(uint32_t *cb,
}
/**
- * vmw_resources_reserve - Reserve all resources on the sw_context's
- * resource list.
+ * vmw_resources_reserve - Reserve all resources on the sw_context's resource
+ * list.
*
* @sw_context: Pointer to the software context.
*
- * Note that since vmware's command submission currently is protected by
- * the cmdbuf mutex, no fancy deadlock avoidance is required for resources,
- * since only a single thread at once will attempt this.
+ * Note that since vmware's command submission currently is protected by the
+ * cmdbuf mutex, no fancy deadlock avoidance is required for resources, since
+ * only a single thread at once will attempt this.
*/
static int vmw_resources_reserve(struct vmw_sw_context *sw_context)
{
}
/**
- * vmw_cmd_res_check - Check that a resource is present and if so, put it
- * on the resource validate list unless it's already there.
+ * vmw_cmd_res_check - Check that a resource is present and if so, put it on the
+ * resource validate list unless it's already there.
*
* @dev_priv: Pointer to a device private structure.
* @sw_context: Pointer to the software context.
* @res_type: Resource type.
+ * @dirty: Whether to change dirty status.
* @converter: User-space visisble type specific information.
- * @id_loc: Pointer to the location in the command buffer currently being
- * parsed from where the user-space resource id handle is located.
- * @p_val: Pointer to pointer to resource validalidation node. Populated
- * on exit.
+ * @id_loc: Pointer to the location in the command buffer currently being parsed
+ * from where the user-space resource id handle is located.
+ * @p_val: Pointer to pointer to resource validalidation node. Populated on
+ * exit.
*/
static int
vmw_cmd_res_check(struct vmw_private *dev_priv,
struct vmw_sw_context *sw_context,
enum vmw_res_type res_type,
+ u32 dirty,
const struct vmw_user_resource_conv *converter,
uint32_t *id_loc,
struct vmw_resource **p_res)
if (*id_loc == SVGA3D_INVALID_ID) {
if (res_type == vmw_res_context) {
- DRM_ERROR("Illegal context invalid id.\n");
+ VMW_DEBUG_USER("Illegal context invalid id.\n");
return -EINVAL;
}
return 0;
if (likely(rcache->valid_handle && *id_loc == rcache->handle)) {
res = rcache->res;
+ if (dirty)
+ vmw_validation_res_set_dirty(sw_context->ctx,
+ rcache->private, dirty);
} else {
unsigned int size = vmw_execbuf_res_size(dev_priv, res_type);
res = vmw_user_resource_noref_lookup_handle
(dev_priv, sw_context->fp->tfile, *id_loc, converter);
- if (unlikely(IS_ERR(res))) {
- DRM_ERROR("Could not find or use resource 0x%08x.\n",
- (unsigned int) *id_loc);
+ if (IS_ERR(res)) {
+ VMW_DEBUG_USER("Could not find/use resource 0x%08x.\n",
+ (unsigned int) *id_loc);
return PTR_ERR(res);
}
- ret = vmw_execbuf_res_noref_val_add(sw_context, res);
+ ret = vmw_execbuf_res_noref_val_add(sw_context, res, dirty);
if (unlikely(ret != 0))
return ret;
{
struct vmw_private *dev_priv = ctx_res->dev_priv;
struct vmw_buffer_object *dx_query_mob;
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXBindAllQuery body;
- } *cmd;
-
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXBindAllQuery);
dx_query_mob = vmw_context_get_dx_query_mob(ctx_res);
if (!dx_query_mob || dx_query_mob->dx_query_ctx)
return 0;
- cmd = vmw_fifo_reserve_dx(dev_priv, sizeof(*cmd), ctx_res->id);
-
- if (cmd == NULL) {
- DRM_ERROR("Failed to rebind queries.\n");
+ cmd = VMW_FIFO_RESERVE_DX(dev_priv, sizeof(*cmd), ctx_res->id);
+ if (cmd == NULL)
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_BIND_ALL_QUERY;
cmd->header.size = sizeof(cmd->body);
}
/**
- * vmw_rebind_contexts - Rebind all resources previously bound to
- * referenced contexts.
+ * vmw_rebind_contexts - Rebind all resources previously bound to referenced
+ * contexts.
*
* @sw_context: Pointer to the software context.
*
ret = vmw_binding_rebind_all(val->cur);
if (unlikely(ret != 0)) {
if (ret != -ERESTARTSYS)
- DRM_ERROR("Failed to rebind context.\n");
+ VMW_DEBUG_USER("Failed to rebind context.\n");
return ret;
}
ret = vmw_rebind_all_dx_query(val->ctx);
- if (ret != 0)
+ if (ret != 0) {
+ VMW_DEBUG_USER("Failed to rebind queries.\n");
return ret;
+ }
}
return 0;
}
/**
- * vmw_view_bindings_add - Add an array of view bindings to a context
- * binding state tracker.
+ * vmw_view_bindings_add - Add an array of view bindings to a context binding
+ * state tracker.
*
* @sw_context: The execbuf state used for this command.
* @view_type: View type for the bindings.
uint32 view_ids[], u32 num_views,
u32 first_slot)
{
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
u32 i;
- if (!ctx_node) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
for (i = 0; i < num_views; ++i) {
struct vmw_ctx_bindinfo_view binding;
view = vmw_view_id_val_add(sw_context, view_type,
view_ids[i]);
if (IS_ERR(view)) {
- DRM_ERROR("View not found.\n");
+ VMW_DEBUG_USER("View not found.\n");
return PTR_ERR(view);
}
}
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_cid_cmd {
- SVGA3dCmdHeader header;
- uint32_t cid;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, uint32_t) =
+ container_of(header, typeof(*cmd), header);
- cmd = container_of(header, struct vmw_cid_cmd, header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_context,
- user_context_converter, &cmd->cid, NULL);
+ VMW_RES_DIRTY_SET, user_context_converter,
+ &cmd->body, NULL);
}
/**
* vmw_execbuf_info_from_res - Get the private validation metadata for a
* recently validated resource
+ *
* @sw_context: Pointer to the command submission context
* @res: The resource
*
* context's resource cache and hence the last resource of that type to be
* processed by the validation code.
*
- * Return: a pointer to the private metadata of the resource, or NULL
- * if it wasn't found
+ * Return: a pointer to the private metadata of the resource, or NULL if it
+ * wasn't found
*/
static struct vmw_ctx_validation_info *
vmw_execbuf_info_from_res(struct vmw_sw_context *sw_context,
return NULL;
}
-
static int vmw_cmd_set_render_target_check(struct vmw_private *dev_priv,
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_sid_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdSetRenderTarget body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdSetRenderTarget);
struct vmw_resource *ctx;
struct vmw_resource *res;
int ret;
- cmd = container_of(header, struct vmw_sid_cmd, header);
+ cmd = container_of(header, typeof(*cmd), header);
if (cmd->body.type >= SVGA3D_RT_MAX) {
- DRM_ERROR("Illegal render target type %u.\n",
- (unsigned) cmd->body.type);
+ VMW_DEBUG_USER("Illegal render target type %u.\n",
+ (unsigned int) cmd->body.type);
return -EINVAL;
}
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_context,
- user_context_converter, &cmd->body.cid,
- &ctx);
+ VMW_RES_DIRTY_SET, user_context_converter,
+ &cmd->body.cid, &ctx);
if (unlikely(ret != 0))
return ret;
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter, &cmd->body.target.sid,
- &res);
+ VMW_RES_DIRTY_SET, user_surface_converter,
+ &cmd->body.target.sid, &res);
if (unlikely(ret))
return ret;
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_sid_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdSurfaceCopy body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdSurfaceCopy);
int ret;
- cmd = container_of(header, struct vmw_sid_cmd, header);
+ cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
- &cmd->body.src.sid, NULL);
+ VMW_RES_DIRTY_NONE, user_surface_converter,
+ &cmd->body.src.sid, NULL);
if (ret)
return ret;
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_SET, user_surface_converter,
&cmd->body.dest.sid, NULL);
}
static int vmw_cmd_buffer_copy_check(struct vmw_private *dev_priv,
- struct vmw_sw_context *sw_context,
- SVGA3dCmdHeader *header)
+ struct vmw_sw_context *sw_context,
+ SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXBufferCopy body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXBufferCopy);
int ret;
cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->body.src, NULL);
if (ret != 0)
return ret;
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_SET, user_surface_converter,
&cmd->body.dest, NULL);
}
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXPredCopyRegion body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXPredCopyRegion);
int ret;
cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->body.srcSid, NULL);
if (ret != 0)
return ret;
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_SET, user_surface_converter,
&cmd->body.dstSid, NULL);
}
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_sid_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdSurfaceStretchBlt body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdSurfaceStretchBlt);
int ret;
- cmd = container_of(header, struct vmw_sid_cmd, header);
+ cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->body.src.sid, NULL);
if (unlikely(ret != 0))
return ret;
+
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_SET, user_surface_converter,
&cmd->body.dest.sid, NULL);
}
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_sid_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdBlitSurfaceToScreen body;
- } *cmd;
-
- cmd = container_of(header, struct vmw_sid_cmd, header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdBlitSurfaceToScreen) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->body.srcImage.sid, NULL);
}
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_sid_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdPresent body;
- } *cmd;
-
-
- cmd = container_of(header, struct vmw_sid_cmd, header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdPresent) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter, &cmd->body.sid,
- NULL);
+ VMW_RES_DIRTY_NONE, user_surface_converter,
+ &cmd->body.sid, NULL);
}
/**
* @new_query_bo: The new buffer holding query results.
* @sw_context: The software context used for this command submission.
*
- * This function checks whether @new_query_bo is suitable for holding
- * query results, and if another buffer currently is pinned for query
- * results. If so, the function prepares the state of @sw_context for
- * switching pinned buffers after successful submission of the current
- * command batch.
+ * This function checks whether @new_query_bo is suitable for holding query
+ * results, and if another buffer currently is pinned for query results. If so,
+ * the function prepares the state of @sw_context for switching pinned buffers
+ * after successful submission of the current command batch.
*/
static int vmw_query_bo_switch_prepare(struct vmw_private *dev_priv,
struct vmw_buffer_object *new_query_bo,
if (unlikely(new_query_bo != sw_context->cur_query_bo)) {
if (unlikely(new_query_bo->base.num_pages > 4)) {
- DRM_ERROR("Query buffer too large.\n");
+ VMW_DEBUG_USER("Query buffer too large.\n");
return -EINVAL;
}
dev_priv->has_mob, false);
if (unlikely(ret != 0))
return ret;
-
}
return 0;
}
-
/**
* vmw_query_bo_switch_commit - Finalize switching pinned query buffer
*
*
* This function will check if we're switching query buffers, and will then,
* issue a dummy occlusion query wait used as a query barrier. When the fence
- * object following that query wait has signaled, we are sure that all
- * preceding queries have finished, and the old query buffer can be unpinned.
- * However, since both the new query buffer and the old one are fenced with
- * that fence, we can do an asynchronus unpin now, and be sure that the
- * old query buffer won't be moved until the fence has signaled.
+ * object following that query wait has signaled, we are sure that all preceding
+ * queries have finished, and the old query buffer can be unpinned. However,
+ * since both the new query buffer and the old one are fenced with that fence,
+ * we can do an asynchronus unpin now, and be sure that the old query buffer
+ * won't be moved until the fence has signaled.
*
* As mentioned above, both the new - and old query buffers need to be fenced
* using a sequence emitted *after* calling this function.
* The validate list should still hold references to all
* contexts here.
*/
-
if (sw_context->needs_post_query_barrier) {
struct vmw_res_cache_entry *ctx_entry =
&sw_context->res_cache[vmw_res_context];
ret = vmw_fifo_emit_dummy_query(dev_priv, ctx->id);
if (unlikely(ret != 0))
- DRM_ERROR("Out of fifo space for dummy query.\n");
+ VMW_DEBUG_USER("Out of fifo space for dummy query.\n");
}
if (dev_priv->pinned_bo != sw_context->cur_query_bo) {
/*
* We pin also the dummy_query_bo buffer so that we
- * don't need to validate it when emitting
- * dummy queries in context destroy paths.
+ * don't need to validate it when emitting dummy queries
+ * in context destroy paths.
*/
-
if (!dev_priv->dummy_query_bo_pinned) {
vmw_bo_pin_reserved(dev_priv->dummy_query_bo,
true);
}
/**
- * vmw_translate_mob_pointer - Prepare to translate a user-space buffer
- * handle to a MOB id.
+ * vmw_translate_mob_pointer - Prepare to translate a user-space buffer handle
+ * to a MOB id.
*
* @dev_priv: Pointer to a device private structure.
* @sw_context: The software context used for this command batch validation.
* @id: Pointer to the user-space handle to be translated.
- * @vmw_bo_p: Points to a location that, on successful return will carry
- * a non-reference-counted pointer to the buffer object identified by the
+ * @vmw_bo_p: Points to a location that, on successful return will carry a
+ * non-reference-counted pointer to the buffer object identified by the
* user-space handle in @id.
*
* This function saves information needed to translate a user-space buffer
* handle to a MOB id. The translation does not take place immediately, but
- * during a call to vmw_apply_relocations(). This function builds a relocation
- * list and a list of buffers to validate. The former needs to be freed using
- * either vmw_apply_relocations() or vmw_free_relocations(). The latter
- * needs to be freed using vmw_clear_validations.
+ * during a call to vmw_apply_relocations().
+ *
+ * This function builds a relocation list and a list of buffers to validate. The
+ * former needs to be freed using either vmw_apply_relocations() or
+ * vmw_free_relocations(). The latter needs to be freed using
+ * vmw_clear_validations.
*/
static int vmw_translate_mob_ptr(struct vmw_private *dev_priv,
struct vmw_sw_context *sw_context,
vmw_validation_preload_bo(sw_context->ctx);
vmw_bo = vmw_user_bo_noref_lookup(sw_context->fp->tfile, handle);
if (IS_ERR(vmw_bo)) {
- DRM_ERROR("Could not find or use MOB buffer.\n");
+ VMW_DEBUG_USER("Could not find or use MOB buffer.\n");
return PTR_ERR(vmw_bo);
}
}
/**
- * vmw_translate_guest_pointer - Prepare to translate a user-space buffer
- * handle to a valid SVGAGuestPtr
+ * vmw_translate_guest_pointer - Prepare to translate a user-space buffer handle
+ * to a valid SVGAGuestPtr
*
* @dev_priv: Pointer to a device private structure.
* @sw_context: The software context used for this command batch validation.
* @ptr: Pointer to the user-space handle to be translated.
- * @vmw_bo_p: Points to a location that, on successful return will carry
- * a non-reference-counted pointer to the DMA buffer identified by the
- * user-space handle in @id.
+ * @vmw_bo_p: Points to a location that, on successful return will carry a
+ * non-reference-counted pointer to the DMA buffer identified by the user-space
+ * handle in @id.
*
* This function saves information needed to translate a user-space buffer
* handle to a valid SVGAGuestPtr. The translation does not take place
* immediately, but during a call to vmw_apply_relocations().
+ *
* This function builds a relocation list and a list of buffers to validate.
* The former needs to be freed using either vmw_apply_relocations() or
* vmw_free_relocations(). The latter needs to be freed using
vmw_validation_preload_bo(sw_context->ctx);
vmw_bo = vmw_user_bo_noref_lookup(sw_context->fp->tfile, handle);
if (IS_ERR(vmw_bo)) {
- DRM_ERROR("Could not find or use GMR region.\n");
+ VMW_DEBUG_USER("Could not find or use GMR region.\n");
return PTR_ERR(vmw_bo);
}
return 0;
}
-
-
/**
- * vmw_cmd_dx_define_query - validate a SVGA_3D_CMD_DX_DEFINE_QUERY command.
+ * vmw_cmd_dx_define_query - validate SVGA_3D_CMD_DX_DEFINE_QUERY command.
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context used for this command submission.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_dx_define_query_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXDefineQuery q;
- } *cmd;
-
- int ret;
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXDefineQuery);
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct vmw_resource *cotable_res;
+ int ret;
-
- if (ctx_node == NULL) {
- DRM_ERROR("DX Context not set for query.\n");
+ if (!ctx_node)
return -EINVAL;
- }
- cmd = container_of(header, struct vmw_dx_define_query_cmd, header);
+ cmd = container_of(header, typeof(*cmd), header);
- if (cmd->q.type < SVGA3D_QUERYTYPE_MIN ||
- cmd->q.type >= SVGA3D_QUERYTYPE_MAX)
+ if (cmd->body.type < SVGA3D_QUERYTYPE_MIN ||
+ cmd->body.type >= SVGA3D_QUERYTYPE_MAX)
return -EINVAL;
cotable_res = vmw_context_cotable(ctx_node->ctx, SVGA_COTABLE_DXQUERY);
- ret = vmw_cotable_notify(cotable_res, cmd->q.queryId);
+ ret = vmw_cotable_notify(cotable_res, cmd->body.queryId);
return ret;
}
-
-
/**
- * vmw_cmd_dx_bind_query - validate a SVGA_3D_CMD_DX_BIND_QUERY command.
+ * vmw_cmd_dx_bind_query - validate SVGA_3D_CMD_DX_BIND_QUERY command.
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context used for this command submission.
* @header: Pointer to the command header in the command stream.
*
- * The query bind operation will eventually associate the query ID
- * with its backing MOB. In this function, we take the user mode
- * MOB ID and use vmw_translate_mob_ptr() to translate it to its
- * kernel mode equivalent.
+ * The query bind operation will eventually associate the query ID with its
+ * backing MOB. In this function, we take the user mode MOB ID and use
+ * vmw_translate_mob_ptr() to translate it to its kernel mode equivalent.
*/
static int vmw_cmd_dx_bind_query(struct vmw_private *dev_priv,
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_dx_bind_query_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXBindQuery q;
- } *cmd;
-
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXBindQuery);
struct vmw_buffer_object *vmw_bo;
- int ret;
-
+ int ret;
- cmd = container_of(header, struct vmw_dx_bind_query_cmd, header);
+ cmd = container_of(header, typeof(*cmd), header);
/*
* Look up the buffer pointed to by q.mobid, put it on the relocation
* list so its kernel mode MOB ID can be filled in later
*/
- ret = vmw_translate_mob_ptr(dev_priv, sw_context, &cmd->q.mobid,
+ ret = vmw_translate_mob_ptr(dev_priv, sw_context, &cmd->body.mobid,
&vmw_bo);
if (ret != 0)
return 0;
}
-
-
/**
- * vmw_cmd_begin_gb_query - validate a SVGA_3D_CMD_BEGIN_GB_QUERY command.
+ * vmw_cmd_begin_gb_query - validate SVGA_3D_CMD_BEGIN_GB_QUERY command.
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context used for this command submission.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_begin_gb_query_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdBeginGBQuery q;
- } *cmd;
-
- cmd = container_of(header, struct vmw_begin_gb_query_cmd,
- header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdBeginGBQuery) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_context,
- user_context_converter, &cmd->q.cid,
- NULL);
+ VMW_RES_DIRTY_SET, user_context_converter,
+ &cmd->body.cid, NULL);
}
/**
- * vmw_cmd_begin_query - validate a SVGA_3D_CMD_BEGIN_QUERY command.
+ * vmw_cmd_begin_query - validate SVGA_3D_CMD_BEGIN_QUERY command.
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context used for this command submission.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_begin_query_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdBeginQuery q;
- } *cmd;
-
- cmd = container_of(header, struct vmw_begin_query_cmd,
- header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdBeginQuery) =
+ container_of(header, typeof(*cmd), header);
if (unlikely(dev_priv->has_mob)) {
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdBeginGBQuery q;
- } gb_cmd;
+ VMW_DECLARE_CMD_VAR(gb_cmd, SVGA3dCmdBeginGBQuery);
BUG_ON(sizeof(gb_cmd) != sizeof(*cmd));
gb_cmd.header.id = SVGA_3D_CMD_BEGIN_GB_QUERY;
gb_cmd.header.size = cmd->header.size;
- gb_cmd.q.cid = cmd->q.cid;
- gb_cmd.q.type = cmd->q.type;
+ gb_cmd.body.cid = cmd->body.cid;
+ gb_cmd.body.type = cmd->body.type;
memcpy(cmd, &gb_cmd, sizeof(*cmd));
return vmw_cmd_begin_gb_query(dev_priv, sw_context, header);
}
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_context,
- user_context_converter, &cmd->q.cid,
- NULL);
+ VMW_RES_DIRTY_SET, user_context_converter,
+ &cmd->body.cid, NULL);
}
/**
- * vmw_cmd_end_gb_query - validate a SVGA_3D_CMD_END_GB_QUERY command.
+ * vmw_cmd_end_gb_query - validate SVGA_3D_CMD_END_GB_QUERY command.
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context used for this command submission.
SVGA3dCmdHeader *header)
{
struct vmw_buffer_object *vmw_bo;
- struct vmw_query_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdEndGBQuery q;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdEndGBQuery);
int ret;
- cmd = container_of(header, struct vmw_query_cmd, header);
+ cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_cid_check(dev_priv, sw_context, header);
if (unlikely(ret != 0))
return ret;
- ret = vmw_translate_mob_ptr(dev_priv, sw_context,
- &cmd->q.mobid,
+ ret = vmw_translate_mob_ptr(dev_priv, sw_context, &cmd->body.mobid,
&vmw_bo);
if (unlikely(ret != 0))
return ret;
}
/**
- * vmw_cmd_end_query - validate a SVGA_3D_CMD_END_QUERY command.
+ * vmw_cmd_end_query - validate SVGA_3D_CMD_END_QUERY command.
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context used for this command submission.
SVGA3dCmdHeader *header)
{
struct vmw_buffer_object *vmw_bo;
- struct vmw_query_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdEndQuery q;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdEndQuery);
int ret;
- cmd = container_of(header, struct vmw_query_cmd, header);
+ cmd = container_of(header, typeof(*cmd), header);
if (dev_priv->has_mob) {
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdEndGBQuery q;
- } gb_cmd;
+ VMW_DECLARE_CMD_VAR(gb_cmd, SVGA3dCmdEndGBQuery);
BUG_ON(sizeof(gb_cmd) != sizeof(*cmd));
gb_cmd.header.id = SVGA_3D_CMD_END_GB_QUERY;
gb_cmd.header.size = cmd->header.size;
- gb_cmd.q.cid = cmd->q.cid;
- gb_cmd.q.type = cmd->q.type;
- gb_cmd.q.mobid = cmd->q.guestResult.gmrId;
- gb_cmd.q.offset = cmd->q.guestResult.offset;
+ gb_cmd.body.cid = cmd->body.cid;
+ gb_cmd.body.type = cmd->body.type;
+ gb_cmd.body.mobid = cmd->body.guestResult.gmrId;
+ gb_cmd.body.offset = cmd->body.guestResult.offset;
memcpy(cmd, &gb_cmd, sizeof(*cmd));
return vmw_cmd_end_gb_query(dev_priv, sw_context, header);
return ret;
ret = vmw_translate_guest_ptr(dev_priv, sw_context,
- &cmd->q.guestResult,
- &vmw_bo);
+ &cmd->body.guestResult, &vmw_bo);
if (unlikely(ret != 0))
return ret;
}
/**
- * vmw_cmd_wait_gb_query - validate a SVGA_3D_CMD_WAIT_GB_QUERY command.
+ * vmw_cmd_wait_gb_query - validate SVGA_3D_CMD_WAIT_GB_QUERY command.
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context used for this command submission.
SVGA3dCmdHeader *header)
{
struct vmw_buffer_object *vmw_bo;
- struct vmw_query_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdWaitForGBQuery q;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdWaitForGBQuery);
int ret;
- cmd = container_of(header, struct vmw_query_cmd, header);
+ cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_cid_check(dev_priv, sw_context, header);
if (unlikely(ret != 0))
return ret;
- ret = vmw_translate_mob_ptr(dev_priv, sw_context,
- &cmd->q.mobid,
+ ret = vmw_translate_mob_ptr(dev_priv, sw_context, &cmd->body.mobid,
&vmw_bo);
if (unlikely(ret != 0))
return ret;
}
/**
- * vmw_cmd_wait_query - validate a SVGA_3D_CMD_WAIT_QUERY command.
+ * vmw_cmd_wait_query - validate SVGA_3D_CMD_WAIT_QUERY command.
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context used for this command submission.
SVGA3dCmdHeader *header)
{
struct vmw_buffer_object *vmw_bo;
- struct vmw_query_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdWaitForQuery q;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdWaitForQuery);
int ret;
- cmd = container_of(header, struct vmw_query_cmd, header);
+ cmd = container_of(header, typeof(*cmd), header);
if (dev_priv->has_mob) {
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdWaitForGBQuery q;
- } gb_cmd;
+ VMW_DECLARE_CMD_VAR(gb_cmd, SVGA3dCmdWaitForGBQuery);
BUG_ON(sizeof(gb_cmd) != sizeof(*cmd));
gb_cmd.header.id = SVGA_3D_CMD_WAIT_FOR_GB_QUERY;
gb_cmd.header.size = cmd->header.size;
- gb_cmd.q.cid = cmd->q.cid;
- gb_cmd.q.type = cmd->q.type;
- gb_cmd.q.mobid = cmd->q.guestResult.gmrId;
- gb_cmd.q.offset = cmd->q.guestResult.offset;
+ gb_cmd.body.cid = cmd->body.cid;
+ gb_cmd.body.type = cmd->body.type;
+ gb_cmd.body.mobid = cmd->body.guestResult.gmrId;
+ gb_cmd.body.offset = cmd->body.guestResult.offset;
memcpy(cmd, &gb_cmd, sizeof(*cmd));
return vmw_cmd_wait_gb_query(dev_priv, sw_context, header);
return ret;
ret = vmw_translate_guest_ptr(dev_priv, sw_context,
- &cmd->q.guestResult,
- &vmw_bo);
+ &cmd->body.guestResult, &vmw_bo);
if (unlikely(ret != 0))
return ret;
{
struct vmw_buffer_object *vmw_bo = NULL;
struct vmw_surface *srf = NULL;
- struct vmw_dma_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdSurfaceDMA dma;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdSurfaceDMA);
int ret;
SVGA3dCmdSurfaceDMASuffix *suffix;
uint32_t bo_size;
+ bool dirty;
- cmd = container_of(header, struct vmw_dma_cmd, header);
- suffix = (SVGA3dCmdSurfaceDMASuffix *)((unsigned long) &cmd->dma +
+ cmd = container_of(header, typeof(*cmd), header);
+ suffix = (SVGA3dCmdSurfaceDMASuffix *)((unsigned long) &cmd->body +
header->size - sizeof(*suffix));
/* Make sure device and verifier stays in sync. */
if (unlikely(suffix->suffixSize != sizeof(*suffix))) {
- DRM_ERROR("Invalid DMA suffix size.\n");
+ VMW_DEBUG_USER("Invalid DMA suffix size.\n");
return -EINVAL;
}
ret = vmw_translate_guest_ptr(dev_priv, sw_context,
- &cmd->dma.guest.ptr,
- &vmw_bo);
+ &cmd->body.guest.ptr, &vmw_bo);
if (unlikely(ret != 0))
return ret;
/* Make sure DMA doesn't cross BO boundaries. */
bo_size = vmw_bo->base.num_pages * PAGE_SIZE;
- if (unlikely(cmd->dma.guest.ptr.offset > bo_size)) {
- DRM_ERROR("Invalid DMA offset.\n");
+ if (unlikely(cmd->body.guest.ptr.offset > bo_size)) {
+ VMW_DEBUG_USER("Invalid DMA offset.\n");
return -EINVAL;
}
- bo_size -= cmd->dma.guest.ptr.offset;
+ bo_size -= cmd->body.guest.ptr.offset;
if (unlikely(suffix->maximumOffset > bo_size))
suffix->maximumOffset = bo_size;
+ dirty = (cmd->body.transfer == SVGA3D_WRITE_HOST_VRAM) ?
+ VMW_RES_DIRTY_SET : 0;
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter, &cmd->dma.host.sid,
- NULL);
+ dirty, user_surface_converter,
+ &cmd->body.host.sid, NULL);
if (unlikely(ret != 0)) {
if (unlikely(ret != -ERESTARTSYS))
- DRM_ERROR("could not find surface for DMA.\n");
+ VMW_DEBUG_USER("could not find surface for DMA.\n");
return ret;
}
srf = vmw_res_to_srf(sw_context->res_cache[vmw_res_surface].res);
- vmw_kms_cursor_snoop(srf, sw_context->fp->tfile, &vmw_bo->base,
- header);
+ vmw_kms_cursor_snoop(srf, sw_context->fp->tfile, &vmw_bo->base, header);
return 0;
}
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_draw_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdDrawPrimitives body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDrawPrimitives);
SVGA3dVertexDecl *decl = (SVGA3dVertexDecl *)(
(unsigned long)header + sizeof(*cmd));
SVGA3dPrimitiveRange *range;
if (unlikely(ret != 0))
return ret;
- cmd = container_of(header, struct vmw_draw_cmd, header);
+ cmd = container_of(header, typeof(*cmd), header);
maxnum = (header->size - sizeof(cmd->body)) / sizeof(*decl);
if (unlikely(cmd->body.numVertexDecls > maxnum)) {
- DRM_ERROR("Illegal number of vertex declarations.\n");
+ VMW_DEBUG_USER("Illegal number of vertex declarations.\n");
return -EINVAL;
}
for (i = 0; i < cmd->body.numVertexDecls; ++i, ++decl) {
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
+ VMW_RES_DIRTY_NONE,
user_surface_converter,
&decl->array.surfaceId, NULL);
if (unlikely(ret != 0))
maxnum = (header->size - sizeof(cmd->body) -
cmd->body.numVertexDecls * sizeof(*decl)) / sizeof(*range);
if (unlikely(cmd->body.numRanges > maxnum)) {
- DRM_ERROR("Illegal number of index ranges.\n");
+ VMW_DEBUG_USER("Illegal number of index ranges.\n");
return -EINVAL;
}
range = (SVGA3dPrimitiveRange *) decl;
for (i = 0; i < cmd->body.numRanges; ++i, ++range) {
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
+ VMW_RES_DIRTY_NONE,
user_surface_converter,
&range->indexArray.surfaceId, NULL);
if (unlikely(ret != 0))
return 0;
}
-
static int vmw_cmd_tex_state(struct vmw_private *dev_priv,
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_tex_state_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdSetTextureState state;
- } *cmd;
-
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdSetTextureState);
SVGA3dTextureState *last_state = (SVGA3dTextureState *)
((unsigned long) header + header->size + sizeof(header));
SVGA3dTextureState *cur_state = (SVGA3dTextureState *)
- ((unsigned long) header + sizeof(struct vmw_tex_state_cmd));
+ ((unsigned long) header + sizeof(*cmd));
struct vmw_resource *ctx;
struct vmw_resource *res;
int ret;
- cmd = container_of(header, struct vmw_tex_state_cmd,
- header);
+ cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_context,
- user_context_converter, &cmd->state.cid,
- &ctx);
+ VMW_RES_DIRTY_SET, user_context_converter,
+ &cmd->body.cid, &ctx);
if (unlikely(ret != 0))
return ret;
continue;
if (cur_state->stage >= SVGA3D_NUM_TEXTURE_UNITS) {
- DRM_ERROR("Illegal texture/sampler unit %u.\n",
- (unsigned) cur_state->stage);
+ VMW_DEBUG_USER("Illegal texture/sampler unit %u.\n",
+ (unsigned int) cur_state->stage);
return -EINVAL;
}
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
+ VMW_RES_DIRTY_NONE,
user_surface_converter,
&cur_state->value, &res);
if (unlikely(ret != 0))
SVGAFifoCmdDefineGMRFB body;
} *cmd = buf;
- return vmw_translate_guest_ptr(dev_priv, sw_context,
- &cmd->body.ptr,
+ return vmw_translate_guest_ptr(dev_priv, sw_context, &cmd->body.ptr,
&vmw_bo);
}
-
/**
* vmw_cmd_res_switch_backup - Utility function to handle backup buffer
* switching
* stream.
* @backup_offset: Offset of backup into MOB.
*
- * This function prepares for registering a switch of backup buffers
- * in the resource metadata just prior to unreserving. It's basically a wrapper
- * around vmw_cmd_res_switch_backup with a different interface.
+ * This function prepares for registering a switch of backup buffers in the
+ * resource metadata just prior to unreserving. It's basically a wrapper around
+ * vmw_cmd_res_switch_backup with a different interface.
*/
static int vmw_cmd_res_switch_backup(struct vmw_private *dev_priv,
struct vmw_sw_context *sw_context,
- struct vmw_resource *res,
- uint32_t *buf_id,
+ struct vmw_resource *res, uint32_t *buf_id,
unsigned long backup_offset)
{
struct vmw_buffer_object *vbo;
return 0;
}
-
/**
* vmw_cmd_switch_backup - Utility function to handle backup buffer switching
*
* stream.
* @backup_offset: Offset of backup into MOB.
*
- * This function prepares for registering a switch of backup buffers
- * in the resource metadata just prior to unreserving. It's basically a wrapper
- * around vmw_cmd_res_switch_backup with a different interface.
+ * This function prepares for registering a switch of backup buffers in the
+ * resource metadata just prior to unreserving. It's basically a wrapper around
+ * vmw_cmd_res_switch_backup with a different interface.
*/
static int vmw_cmd_switch_backup(struct vmw_private *dev_priv,
struct vmw_sw_context *sw_context,
enum vmw_res_type res_type,
const struct vmw_user_resource_conv
- *converter,
- uint32_t *res_id,
- uint32_t *buf_id,
+ *converter, uint32_t *res_id, uint32_t *buf_id,
unsigned long backup_offset)
{
struct vmw_resource *res;
int ret;
ret = vmw_cmd_res_check(dev_priv, sw_context, res_type,
- converter, res_id, &res);
+ VMW_RES_DIRTY_NONE, converter, res_id, &res);
if (ret)
return ret;
- return vmw_cmd_res_switch_backup(dev_priv, sw_context, res,
- buf_id, backup_offset);
+ return vmw_cmd_res_switch_backup(dev_priv, sw_context, res, buf_id,
+ backup_offset);
}
/**
- * vmw_cmd_bind_gb_surface - Validate an SVGA_3D_CMD_BIND_GB_SURFACE
- * command
+ * vmw_cmd_bind_gb_surface - Validate SVGA_3D_CMD_BIND_GB_SURFACE command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_bind_gb_surface_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdBindGBSurface body;
- } *cmd;
-
- cmd = container_of(header, struct vmw_bind_gb_surface_cmd, header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdBindGBSurface) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_switch_backup(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
- &cmd->body.sid, &cmd->body.mobid,
- 0);
+ user_surface_converter, &cmd->body.sid,
+ &cmd->body.mobid, 0);
}
/**
- * vmw_cmd_update_gb_image - Validate an SVGA_3D_CMD_UPDATE_GB_IMAGE
- * command
+ * vmw_cmd_update_gb_image - Validate SVGA_3D_CMD_UPDATE_GB_IMAGE command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_gb_surface_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdUpdateGBImage body;
- } *cmd;
-
- cmd = container_of(header, struct vmw_gb_surface_cmd, header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdUpdateGBImage) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->body.image.sid, NULL);
}
/**
- * vmw_cmd_update_gb_surface - Validate an SVGA_3D_CMD_UPDATE_GB_SURFACE
- * command
+ * vmw_cmd_update_gb_surface - Validate SVGA_3D_CMD_UPDATE_GB_SURFACE command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_gb_surface_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdUpdateGBSurface body;
- } *cmd;
-
- cmd = container_of(header, struct vmw_gb_surface_cmd, header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdUpdateGBSurface) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_CLEAR, user_surface_converter,
&cmd->body.sid, NULL);
}
/**
- * vmw_cmd_readback_gb_image - Validate an SVGA_3D_CMD_READBACK_GB_IMAGE
- * command
+ * vmw_cmd_readback_gb_image - Validate SVGA_3D_CMD_READBACK_GB_IMAGE command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_gb_surface_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdReadbackGBImage body;
- } *cmd;
-
- cmd = container_of(header, struct vmw_gb_surface_cmd, header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdReadbackGBImage) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->body.image.sid, NULL);
}
/**
- * vmw_cmd_readback_gb_surface - Validate an SVGA_3D_CMD_READBACK_GB_SURFACE
+ * vmw_cmd_readback_gb_surface - Validate SVGA_3D_CMD_READBACK_GB_SURFACE
* command
*
* @dev_priv: Pointer to a device private struct.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_gb_surface_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdReadbackGBSurface body;
- } *cmd;
-
- cmd = container_of(header, struct vmw_gb_surface_cmd, header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdReadbackGBSurface) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_CLEAR, user_surface_converter,
&cmd->body.sid, NULL);
}
/**
- * vmw_cmd_invalidate_gb_image - Validate an SVGA_3D_CMD_INVALIDATE_GB_IMAGE
+ * vmw_cmd_invalidate_gb_image - Validate SVGA_3D_CMD_INVALIDATE_GB_IMAGE
* command
*
* @dev_priv: Pointer to a device private struct.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_gb_surface_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdInvalidateGBImage body;
- } *cmd;
-
- cmd = container_of(header, struct vmw_gb_surface_cmd, header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdInvalidateGBImage) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->body.image.sid, NULL);
}
/**
- * vmw_cmd_invalidate_gb_surface - Validate an
- * SVGA_3D_CMD_INVALIDATE_GB_SURFACE command
+ * vmw_cmd_invalidate_gb_surface - Validate SVGA_3D_CMD_INVALIDATE_GB_SURFACE
+ * command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_gb_surface_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdInvalidateGBSurface body;
- } *cmd;
-
- cmd = container_of(header, struct vmw_gb_surface_cmd, header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdInvalidateGBSurface) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_CLEAR, user_surface_converter,
&cmd->body.sid, NULL);
}
-
/**
- * vmw_cmd_shader_define - Validate an SVGA_3D_CMD_SHADER_DEFINE
- * command
+ * vmw_cmd_shader_define - Validate SVGA_3D_CMD_SHADER_DEFINE command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_shader_define_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdDefineShader body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDefineShader);
int ret;
size_t size;
struct vmw_resource *ctx;
- cmd = container_of(header, struct vmw_shader_define_cmd,
- header);
+ cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_context,
- user_context_converter, &cmd->body.cid,
- &ctx);
+ VMW_RES_DIRTY_SET, user_context_converter,
+ &cmd->body.cid, &ctx);
if (unlikely(ret != 0))
return ret;
return 0;
size = cmd->header.size - sizeof(cmd->body);
- ret = vmw_compat_shader_add(dev_priv,
- vmw_context_res_man(ctx),
- cmd->body.shid, cmd + 1,
- cmd->body.type, size,
- &sw_context->staged_cmd_res);
+ ret = vmw_compat_shader_add(dev_priv, vmw_context_res_man(ctx),
+ cmd->body.shid, cmd + 1, cmd->body.type,
+ size, &sw_context->staged_cmd_res);
if (unlikely(ret != 0))
return ret;
- return vmw_resource_relocation_add(sw_context,
- NULL,
+ return vmw_resource_relocation_add(sw_context, NULL,
vmw_ptr_diff(sw_context->buf_start,
&cmd->header.id),
vmw_res_rel_nop);
}
/**
- * vmw_cmd_shader_destroy - Validate an SVGA_3D_CMD_SHADER_DESTROY
- * command
+ * vmw_cmd_shader_destroy - Validate SVGA_3D_CMD_SHADER_DESTROY command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_shader_destroy_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdDestroyShader body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDestroyShader);
int ret;
struct vmw_resource *ctx;
- cmd = container_of(header, struct vmw_shader_destroy_cmd,
- header);
+ cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_context,
- user_context_converter, &cmd->body.cid,
- &ctx);
+ VMW_RES_DIRTY_SET, user_context_converter,
+ &cmd->body.cid, &ctx);
if (unlikely(ret != 0))
return ret;
if (unlikely(!dev_priv->has_mob))
return 0;
- ret = vmw_shader_remove(vmw_context_res_man(ctx),
- cmd->body.shid,
- cmd->body.type,
- &sw_context->staged_cmd_res);
+ ret = vmw_shader_remove(vmw_context_res_man(ctx), cmd->body.shid,
+ cmd->body.type, &sw_context->staged_cmd_res);
if (unlikely(ret != 0))
return ret;
- return vmw_resource_relocation_add(sw_context,
- NULL,
+ return vmw_resource_relocation_add(sw_context, NULL,
vmw_ptr_diff(sw_context->buf_start,
&cmd->header.id),
vmw_res_rel_nop);
}
/**
- * vmw_cmd_set_shader - Validate an SVGA_3D_CMD_SET_SHADER
- * command
+ * vmw_cmd_set_shader - Validate SVGA_3D_CMD_SET_SHADER command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_set_shader_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdSetShader body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdSetShader);
struct vmw_ctx_bindinfo_shader binding;
struct vmw_resource *ctx, *res = NULL;
struct vmw_ctx_validation_info *ctx_info;
int ret;
- cmd = container_of(header, struct vmw_set_shader_cmd,
- header);
+ cmd = container_of(header, typeof(*cmd), header);
if (cmd->body.type >= SVGA3D_SHADERTYPE_PREDX_MAX) {
- DRM_ERROR("Illegal shader type %u.\n",
- (unsigned) cmd->body.type);
+ VMW_DEBUG_USER("Illegal shader type %u.\n",
+ (unsigned int) cmd->body.type);
return -EINVAL;
}
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_context,
- user_context_converter, &cmd->body.cid,
- &ctx);
+ VMW_RES_DIRTY_SET, user_context_converter,
+ &cmd->body.cid, &ctx);
if (unlikely(ret != 0))
return ret;
if (cmd->body.shid != SVGA3D_INVALID_ID) {
res = vmw_shader_lookup(vmw_context_res_man(ctx),
- cmd->body.shid,
- cmd->body.type);
-
+ cmd->body.shid, cmd->body.type);
if (!IS_ERR(res)) {
- ret = vmw_execbuf_res_noctx_val_add(sw_context, res);
+ ret = vmw_execbuf_res_noctx_val_add(sw_context, res,
+ VMW_RES_DIRTY_NONE);
if (unlikely(ret != 0))
return ret;
}
}
if (IS_ERR_OR_NULL(res)) {
- ret = vmw_cmd_res_check(dev_priv, sw_context,
- vmw_res_shader,
- user_shader_converter,
- &cmd->body.shid, &res);
+ ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_shader,
+ VMW_RES_DIRTY_NONE,
+ user_shader_converter, &cmd->body.shid,
+ &res);
if (unlikely(ret != 0))
return ret;
}
binding.bi.res = res;
binding.bi.bt = vmw_ctx_binding_shader;
binding.shader_slot = cmd->body.type - SVGA3D_SHADERTYPE_MIN;
- vmw_binding_add(ctx_info->staged, &binding.bi,
- binding.shader_slot, 0);
+ vmw_binding_add(ctx_info->staged, &binding.bi, binding.shader_slot, 0);
+
return 0;
}
/**
- * vmw_cmd_set_shader_const - Validate an SVGA_3D_CMD_SET_SHADER_CONST
- * command
+ * vmw_cmd_set_shader_const - Validate SVGA_3D_CMD_SET_SHADER_CONST command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_set_shader_const_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdSetShaderConst body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdSetShaderConst);
int ret;
- cmd = container_of(header, struct vmw_set_shader_const_cmd,
- header);
+ cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_context,
- user_context_converter, &cmd->body.cid,
- NULL);
+ VMW_RES_DIRTY_SET, user_context_converter,
+ &cmd->body.cid, NULL);
if (unlikely(ret != 0))
return ret;
}
/**
- * vmw_cmd_bind_gb_shader - Validate an SVGA_3D_CMD_BIND_GB_SHADER
- * command
+ * vmw_cmd_bind_gb_shader - Validate SVGA_3D_CMD_BIND_GB_SHADER command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_bind_gb_shader_cmd {
- SVGA3dCmdHeader header;
- SVGA3dCmdBindGBShader body;
- } *cmd;
-
- cmd = container_of(header, struct vmw_bind_gb_shader_cmd,
- header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdBindGBShader) =
+ container_of(header, typeof(*cmd), header);
return vmw_cmd_switch_backup(dev_priv, sw_context, vmw_res_shader,
- user_shader_converter,
- &cmd->body.shid, &cmd->body.mobid,
- cmd->body.offsetInBytes);
+ user_shader_converter, &cmd->body.shid,
+ &cmd->body.mobid, cmd->body.offsetInBytes);
}
/**
- * vmw_cmd_dx_set_single_constant_buffer - Validate an
+ * vmw_cmd_dx_set_single_constant_buffer - Validate
* SVGA_3D_CMD_DX_SET_SINGLE_CONSTANT_BUFFER command.
*
* @dev_priv: Pointer to a device private struct.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXSetSingleConstantBuffer body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXSetSingleConstantBuffer);
struct vmw_resource *res = NULL;
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct vmw_ctx_bindinfo_cb binding;
int ret;
- if (unlikely(ctx_node == NULL)) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->body.sid, &res);
if (unlikely(ret != 0))
return ret;
if (binding.shader_slot >= SVGA3D_NUM_SHADERTYPE_DX10 ||
binding.slot >= SVGA3D_DX_MAX_CONSTBUFFERS) {
- DRM_ERROR("Illegal const buffer shader %u slot %u.\n",
- (unsigned) cmd->body.type,
- (unsigned) binding.slot);
+ VMW_DEBUG_USER("Illegal const buffer shader %u slot %u.\n",
+ (unsigned int) cmd->body.type,
+ (unsigned int) binding.slot);
return -EINVAL;
}
- vmw_binding_add(ctx_node->staged, &binding.bi,
- binding.shader_slot, binding.slot);
+ vmw_binding_add(ctx_node->staged, &binding.bi, binding.shader_slot,
+ binding.slot);
return 0;
}
/**
- * vmw_cmd_dx_set_shader_res - Validate an
- * SVGA_3D_CMD_DX_SET_SHADER_RESOURCES command
+ * vmw_cmd_dx_set_shader_res - Validate SVGA_3D_CMD_DX_SET_SHADER_RESOURCES
+ * command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXSetShaderResources body;
- } *cmd = container_of(header, typeof(*cmd), header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXSetShaderResources) =
+ container_of(header, typeof(*cmd), header);
u32 num_sr_view = (cmd->header.size - sizeof(cmd->body)) /
sizeof(SVGA3dShaderResourceViewId);
if ((u64) cmd->body.startView + (u64) num_sr_view >
(u64) SVGA3D_DX_MAX_SRVIEWS ||
cmd->body.type >= SVGA3D_SHADERTYPE_DX10_MAX) {
- DRM_ERROR("Invalid shader binding.\n");
+ VMW_DEBUG_USER("Invalid shader binding.\n");
return -EINVAL;
}
}
/**
- * vmw_cmd_dx_set_shader - Validate an SVGA_3D_CMD_DX_SET_SHADER
- * command
+ * vmw_cmd_dx_set_shader - Validate SVGA_3D_CMD_DX_SET_SHADER command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXSetShader body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXSetShader);
struct vmw_resource *res = NULL;
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct vmw_ctx_bindinfo_shader binding;
int ret = 0;
- if (unlikely(ctx_node == NULL)) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
cmd = container_of(header, typeof(*cmd), header);
if (cmd->body.type >= SVGA3D_SHADERTYPE_DX10_MAX) {
- DRM_ERROR("Illegal shader type %u.\n",
- (unsigned) cmd->body.type);
+ VMW_DEBUG_USER("Illegal shader type %u.\n",
+ (unsigned int) cmd->body.type);
return -EINVAL;
}
if (cmd->body.shaderId != SVGA3D_INVALID_ID) {
res = vmw_shader_lookup(sw_context->man, cmd->body.shaderId, 0);
if (IS_ERR(res)) {
- DRM_ERROR("Could not find shader for binding.\n");
+ VMW_DEBUG_USER("Could not find shader for binding.\n");
return PTR_ERR(res);
}
- ret = vmw_execbuf_res_noctx_val_add(sw_context, res);
+ ret = vmw_execbuf_res_noctx_val_add(sw_context, res,
+ VMW_RES_DIRTY_NONE);
if (ret)
return ret;
}
binding.bi.bt = vmw_ctx_binding_dx_shader;
binding.shader_slot = cmd->body.type - SVGA3D_SHADERTYPE_MIN;
- vmw_binding_add(ctx_node->staged, &binding.bi,
- binding.shader_slot, 0);
+ vmw_binding_add(ctx_node->staged, &binding.bi, binding.shader_slot, 0);
return 0;
}
/**
- * vmw_cmd_dx_set_vertex_buffers - Validates an
- * SVGA_3D_CMD_DX_SET_VERTEX_BUFFERS command
+ * vmw_cmd_dx_set_vertex_buffers - Validates SVGA_3D_CMD_DX_SET_VERTEX_BUFFERS
+ * command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct vmw_ctx_bindinfo_vb binding;
struct vmw_resource *res;
struct {
} *cmd;
int i, ret, num;
- if (unlikely(ctx_node == NULL)) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
cmd = container_of(header, typeof(*cmd), header);
num = (cmd->header.size - sizeof(cmd->body)) /
sizeof(SVGA3dVertexBuffer);
if ((u64)num + (u64)cmd->body.startBuffer >
(u64)SVGA3D_DX_MAX_VERTEXBUFFERS) {
- DRM_ERROR("Invalid number of vertex buffers.\n");
+ VMW_DEBUG_USER("Invalid number of vertex buffers.\n");
return -EINVAL;
}
for (i = 0; i < num; i++) {
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
+ VMW_RES_DIRTY_NONE,
user_surface_converter,
&cmd->buf[i].sid, &res);
if (unlikely(ret != 0))
binding.stride = cmd->buf[i].stride;
binding.slot = i + cmd->body.startBuffer;
- vmw_binding_add(ctx_node->staged, &binding.bi,
- 0, binding.slot);
+ vmw_binding_add(ctx_node->staged, &binding.bi, 0, binding.slot);
}
return 0;
}
/**
- * vmw_cmd_dx_ia_set_vertex_buffers - Validate an
+ * vmw_cmd_dx_ia_set_vertex_buffers - Validate
* SVGA_3D_CMD_DX_IA_SET_INDEX_BUFFER command.
*
* @dev_priv: Pointer to a device private struct.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct vmw_ctx_bindinfo_ib binding;
struct vmw_resource *res;
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXSetIndexBuffer body;
- } *cmd;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXSetIndexBuffer);
int ret;
- if (unlikely(ctx_node == NULL)) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->body.sid, &res);
if (unlikely(ret != 0))
return ret;
}
/**
- * vmw_cmd_dx_set_rendertarget - Validate an
- * SVGA_3D_CMD_DX_SET_RENDERTARGETS command
+ * vmw_cmd_dx_set_rendertarget - Validate SVGA_3D_CMD_DX_SET_RENDERTARGETS
+ * command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXSetRenderTargets body;
- } *cmd = container_of(header, typeof(*cmd), header);
- int ret;
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXSetRenderTargets) =
+ container_of(header, typeof(*cmd), header);
u32 num_rt_view = (cmd->header.size - sizeof(cmd->body)) /
sizeof(SVGA3dRenderTargetViewId);
+ int ret;
if (num_rt_view > SVGA3D_MAX_SIMULTANEOUS_RENDER_TARGETS) {
- DRM_ERROR("Invalid DX Rendertarget binding.\n");
+ VMW_DEBUG_USER("Invalid DX Rendertarget binding.\n");
return -EINVAL;
}
- ret = vmw_view_bindings_add(sw_context, vmw_view_ds,
- vmw_ctx_binding_ds, 0,
- &cmd->body.depthStencilViewId, 1, 0);
+ ret = vmw_view_bindings_add(sw_context, vmw_view_ds, vmw_ctx_binding_ds,
+ 0, &cmd->body.depthStencilViewId, 1, 0);
if (ret)
return ret;
return vmw_view_bindings_add(sw_context, vmw_view_rt,
- vmw_ctx_binding_dx_rt, 0,
- (void *)&cmd[1], num_rt_view, 0);
+ vmw_ctx_binding_dx_rt, 0, (void *)&cmd[1],
+ num_rt_view, 0);
}
/**
- * vmw_cmd_dx_clear_rendertarget_view - Validate an
+ * vmw_cmd_dx_clear_rendertarget_view - Validate
* SVGA_3D_CMD_DX_CLEAR_RENDERTARGET_VIEW command
*
* @dev_priv: Pointer to a device private struct.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXClearRenderTargetView body;
- } *cmd = container_of(header, typeof(*cmd), header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXClearRenderTargetView) =
+ container_of(header, typeof(*cmd), header);
return PTR_RET(vmw_view_id_val_add(sw_context, vmw_view_rt,
cmd->body.renderTargetViewId));
}
/**
- * vmw_cmd_dx_clear_rendertarget_view - Validate an
+ * vmw_cmd_dx_clear_rendertarget_view - Validate
* SVGA_3D_CMD_DX_CLEAR_DEPTHSTENCIL_VIEW command
*
* @dev_priv: Pointer to a device private struct.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXClearDepthStencilView body;
- } *cmd = container_of(header, typeof(*cmd), header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXClearDepthStencilView) =
+ container_of(header, typeof(*cmd), header);
return PTR_RET(vmw_view_id_val_add(sw_context, vmw_view_ds,
cmd->body.depthStencilViewId));
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct vmw_resource *srf;
struct vmw_resource *res;
enum vmw_view_type view_type;
int ret;
/*
- * This is based on the fact that all affected define commands have
- * the same initial command body layout.
+ * This is based on the fact that all affected define commands have the
+ * same initial command body layout.
*/
struct {
SVGA3dCmdHeader header;
uint32 sid;
} *cmd;
- if (unlikely(ctx_node == NULL)) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
view_type = vmw_view_cmd_to_type(header->id);
if (view_type == vmw_view_max)
return -EINVAL;
+
cmd = container_of(header, typeof(*cmd), header);
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->sid, &srf);
if (unlikely(ret != 0))
return ret;
if (unlikely(ret != 0))
return ret;
- return vmw_view_add(sw_context->man,
- ctx_node->ctx,
- srf,
- view_type,
- cmd->defined_id,
- header,
+ return vmw_view_add(sw_context->man, ctx_node->ctx, srf, view_type,
+ cmd->defined_id, header,
header->size + sizeof(*header),
&sw_context->staged_cmd_res);
}
/**
- * vmw_cmd_dx_set_so_targets - Validate an
- * SVGA_3D_CMD_DX_SET_SOTARGETS command.
+ * vmw_cmd_dx_set_so_targets - Validate SVGA_3D_CMD_DX_SET_SOTARGETS command.
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct vmw_ctx_bindinfo_so binding;
struct vmw_resource *res;
struct {
} *cmd;
int i, ret, num;
- if (unlikely(ctx_node == NULL)) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
cmd = container_of(header, typeof(*cmd), header);
- num = (cmd->header.size - sizeof(cmd->body)) /
- sizeof(SVGA3dSoTarget);
+ num = (cmd->header.size - sizeof(cmd->body)) / sizeof(SVGA3dSoTarget);
if (num > SVGA3D_DX_MAX_SOTARGETS) {
- DRM_ERROR("Invalid DX SO binding.\n");
+ VMW_DEBUG_USER("Invalid DX SO binding.\n");
return -EINVAL;
}
for (i = 0; i < num; i++) {
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
+ VMW_RES_DIRTY_SET,
user_surface_converter,
&cmd->targets[i].sid, &res);
if (unlikely(ret != 0))
binding.size = cmd->targets[i].sizeInBytes;
binding.slot = i;
- vmw_binding_add(ctx_node->staged, &binding.bi,
- 0, binding.slot);
+ vmw_binding_add(ctx_node->staged, &binding.bi, 0, binding.slot);
}
return 0;
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct vmw_resource *res;
/*
* This is based on the fact that all affected define commands have
enum vmw_so_type so_type;
int ret;
- if (unlikely(ctx_node == NULL)) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
so_type = vmw_so_cmd_to_type(header->id);
res = vmw_context_cotable(ctx_node->ctx, vmw_so_cotables[so_type]);
}
/**
- * vmw_cmd_dx_check_subresource - Validate an
- * SVGA_3D_CMD_DX_[X]_SUBRESOURCE command
+ * vmw_cmd_dx_check_subresource - Validate SVGA_3D_CMD_DX_[X]_SUBRESOURCE
+ * command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
cmd = container_of(header, typeof(*cmd), header);
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->sid, NULL);
}
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
- if (unlikely(ctx_node == NULL)) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
return 0;
}
/**
- * vmw_cmd_dx_view_remove - validate a view remove command and
- * schedule the view resource for removal.
+ * vmw_cmd_dx_view_remove - validate a view remove command and schedule the view
+ * resource for removal.
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
* @header: Pointer to the command header in the command stream.
*
- * Check that the view exists, and if it was not created using this
- * command batch, conditionally make this command a NOP.
+ * Check that the view exists, and if it was not created using this command
+ * batch, conditionally make this command a NOP.
*/
static int vmw_cmd_dx_view_remove(struct vmw_private *dev_priv,
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct {
SVGA3dCmdHeader header;
union vmw_view_destroy body;
struct vmw_resource *view;
int ret;
- if (!ctx_node) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
- ret = vmw_view_remove(sw_context->man,
- cmd->body.view_id, view_type,
- &sw_context->staged_cmd_res,
- &view);
+ ret = vmw_view_remove(sw_context->man, cmd->body.view_id, view_type,
+ &sw_context->staged_cmd_res, &view);
if (ret || !view)
return ret;
* relocation to conditionally make this command a NOP to avoid
* device errors.
*/
- return vmw_resource_relocation_add(sw_context,
- view,
+ return vmw_resource_relocation_add(sw_context, view,
vmw_ptr_diff(sw_context->buf_start,
&cmd->header.id),
vmw_res_rel_cond_nop);
}
/**
- * vmw_cmd_dx_define_shader - Validate an SVGA_3D_CMD_DX_DEFINE_SHADER
- * command
+ * vmw_cmd_dx_define_shader - Validate SVGA_3D_CMD_DX_DEFINE_SHADER command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct vmw_resource *res;
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXDefineShader body;
- } *cmd = container_of(header, typeof(*cmd), header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXDefineShader) =
+ container_of(header, typeof(*cmd), header);
int ret;
- if (!ctx_node) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
res = vmw_context_cotable(ctx_node->ctx, SVGA_COTABLE_DXSHADER);
ret = vmw_cotable_notify(res, cmd->body.shaderId);
}
/**
- * vmw_cmd_dx_destroy_shader - Validate an SVGA_3D_CMD_DX_DESTROY_SHADER
- * command
+ * vmw_cmd_dx_destroy_shader - Validate SVGA_3D_CMD_DX_DESTROY_SHADER command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct vmw_ctx_validation_info *ctx_node = sw_context->dx_ctx_node;
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXDestroyShader body;
- } *cmd = container_of(header, typeof(*cmd), header);
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXDestroyShader) =
+ container_of(header, typeof(*cmd), header);
int ret;
- if (!ctx_node) {
- DRM_ERROR("DX Context not set.\n");
+ if (!ctx_node)
return -EINVAL;
- }
ret = vmw_shader_remove(sw_context->man, cmd->body.shaderId, 0,
&sw_context->staged_cmd_res);
- if (ret)
- DRM_ERROR("Could not find shader to remove.\n");
return ret;
}
/**
- * vmw_cmd_dx_bind_shader - Validate an SVGA_3D_CMD_DX_BIND_SHADER
- * command
+ * vmw_cmd_dx_bind_shader - Validate SVGA_3D_CMD_DX_BIND_SHADER command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
{
struct vmw_resource *ctx;
struct vmw_resource *res;
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXBindShader body;
- } *cmd = container_of(header, typeof(*cmd), header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXBindShader) =
+ container_of(header, typeof(*cmd), header);
int ret;
if (cmd->body.cid != SVGA3D_INVALID_ID) {
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_context,
- user_context_converter,
- &cmd->body.cid, &ctx);
+ VMW_RES_DIRTY_SET,
+ user_context_converter, &cmd->body.cid,
+ &ctx);
if (ret)
return ret;
} else {
- if (!sw_context->dx_ctx_node) {
- DRM_ERROR("DX Context not set.\n");
+ struct vmw_ctx_validation_info *ctx_node =
+ VMW_GET_CTX_NODE(sw_context);
+
+ if (!ctx_node)
return -EINVAL;
- }
- ctx = sw_context->dx_ctx_node->ctx;
+
+ ctx = ctx_node->ctx;
}
- res = vmw_shader_lookup(vmw_context_res_man(ctx),
- cmd->body.shid, 0);
+ res = vmw_shader_lookup(vmw_context_res_man(ctx), cmd->body.shid, 0);
if (IS_ERR(res)) {
- DRM_ERROR("Could not find shader to bind.\n");
+ VMW_DEBUG_USER("Could not find shader to bind.\n");
return PTR_ERR(res);
}
- ret = vmw_execbuf_res_noctx_val_add(sw_context, res);
+ ret = vmw_execbuf_res_noctx_val_add(sw_context, res,
+ VMW_RES_DIRTY_NONE);
if (ret) {
- DRM_ERROR("Error creating resource validation node.\n");
+ VMW_DEBUG_USER("Error creating resource validation node.\n");
return ret;
}
}
/**
- * vmw_cmd_dx_genmips - Validate an SVGA_3D_CMD_DX_GENMIPS command
+ * vmw_cmd_dx_genmips - Validate SVGA_3D_CMD_DX_GENMIPS command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXGenMips body;
- } *cmd = container_of(header, typeof(*cmd), header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXGenMips) =
+ container_of(header, typeof(*cmd), header);
return PTR_RET(vmw_view_id_val_add(sw_context, vmw_view_sr,
cmd->body.shaderResourceViewId));
}
/**
- * vmw_cmd_dx_transfer_from_buffer -
- * Validate an SVGA_3D_CMD_DX_TRANSFER_FROM_BUFFER command
+ * vmw_cmd_dx_transfer_from_buffer - Validate
+ * SVGA_3D_CMD_DX_TRANSFER_FROM_BUFFER command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdDXTransferFromBuffer body;
- } *cmd = container_of(header, typeof(*cmd), header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXTransferFromBuffer) =
+ container_of(header, typeof(*cmd), header);
int ret;
ret = vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_NONE, user_surface_converter,
&cmd->body.srcSid, NULL);
if (ret != 0)
return ret;
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
+ VMW_RES_DIRTY_SET, user_surface_converter,
&cmd->body.destSid, NULL);
}
/**
- * vmw_cmd_intra_surface_copy -
- * Validate an SVGA_3D_CMD_INTRA_SURFACE_COPY command
+ * vmw_cmd_intra_surface_copy - Validate SVGA_3D_CMD_INTRA_SURFACE_COPY command
*
* @dev_priv: Pointer to a device private struct.
* @sw_context: The software context being used for this batch.
struct vmw_sw_context *sw_context,
SVGA3dCmdHeader *header)
{
- struct {
- SVGA3dCmdHeader header;
- SVGA3dCmdIntraSurfaceCopy body;
- } *cmd = container_of(header, typeof(*cmd), header);
+ VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdIntraSurfaceCopy) =
+ container_of(header, typeof(*cmd), header);
if (!(dev_priv->capabilities2 & SVGA_CAP2_INTRA_SURFACE_COPY))
return -EINVAL;
return vmw_cmd_res_check(dev_priv, sw_context, vmw_res_surface,
- user_surface_converter,
- &cmd->body.surface.sid, NULL);
+ VMW_RES_DIRTY_SET, user_surface_converter,
+ &cmd->body.surface.sid, NULL);
}
-
static int vmw_cmd_check_not_3d(struct vmw_private *dev_priv,
struct vmw_sw_context *sw_context,
void *buf, uint32_t *size)
*size = sizeof(uint32_t) + sizeof(SVGAFifoCmdBlitGMRFBToScreen);
break;
default:
- DRM_ERROR("Unsupported SVGA command: %u.\n", cmd_id);
+ VMW_DEBUG_USER("Unsupported SVGA command: %u.\n", cmd_id);
return -EINVAL;
}
if (*size > size_remaining) {
- DRM_ERROR("Invalid SVGA command (size mismatch):"
- " %u.\n", cmd_id);
+ VMW_DEBUG_USER("Invalid SVGA command (size mismatch): %u.\n",
+ cmd_id);
return -EINVAL;
}
if (unlikely(!sw_context->kernel)) {
- DRM_ERROR("Kernel only SVGA command: %u.\n", cmd_id);
+ VMW_DEBUG_USER("Kernel only SVGA command: %u.\n", cmd_id);
return -EPERM;
}
VMW_CMD_DEF(SVGA_3D_CMD_DEFINE_GB_SURFACE_V2, &vmw_cmd_invalid,
false, false, true),
- /*
- * DX commands
- */
+ /* SM commands */
VMW_CMD_DEF(SVGA_3D_CMD_DX_DEFINE_CONTEXT, &vmw_cmd_invalid,
false, false, true),
VMW_CMD_DEF(SVGA_3D_CMD_DX_DESTROY_CONTEXT, &vmw_cmd_invalid,
}
static int vmw_cmd_check(struct vmw_private *dev_priv,
- struct vmw_sw_context *sw_context,
- void *buf, uint32_t *size)
+ struct vmw_sw_context *sw_context, void *buf,
+ uint32_t *size)
{
uint32_t cmd_id;
uint32_t size_remaining = *size;
goto out_new;
ret = entry->func(dev_priv, sw_context, header);
- if (unlikely(ret != 0))
- goto out_invalid;
+ if (unlikely(ret != 0)) {
+ VMW_DEBUG_USER("SVGA3D command: %d failed with error %d\n",
+ cmd_id + SVGA_3D_CMD_BASE, ret);
+ return ret;
+ }
return 0;
out_invalid:
- DRM_ERROR("Invalid SVGA3D command: %d\n",
- cmd_id + SVGA_3D_CMD_BASE);
+ VMW_DEBUG_USER("Invalid SVGA3D command: %d\n",
+ cmd_id + SVGA_3D_CMD_BASE);
return -EINVAL;
out_privileged:
- DRM_ERROR("Privileged SVGA3D command: %d\n",
- cmd_id + SVGA_3D_CMD_BASE);
+ VMW_DEBUG_USER("Privileged SVGA3D command: %d\n",
+ cmd_id + SVGA_3D_CMD_BASE);
return -EPERM;
out_old:
- DRM_ERROR("Deprecated (disallowed) SVGA3D command: %d\n",
- cmd_id + SVGA_3D_CMD_BASE);
+ VMW_DEBUG_USER("Deprecated (disallowed) SVGA3D command: %d\n",
+ cmd_id + SVGA_3D_CMD_BASE);
return -EINVAL;
out_new:
- DRM_ERROR("SVGA3D command: %d not supported by virtual hardware.\n",
- cmd_id + SVGA_3D_CMD_BASE);
+ VMW_DEBUG_USER("SVGA3D command: %d not supported by virtual device.\n",
+ cmd_id + SVGA_3D_CMD_BASE);
return -EINVAL;
}
static int vmw_cmd_check_all(struct vmw_private *dev_priv,
- struct vmw_sw_context *sw_context,
- void *buf,
+ struct vmw_sw_context *sw_context, void *buf,
uint32_t size)
{
int32_t cur_size = size;
}
if (unlikely(cur_size != 0)) {
- DRM_ERROR("Command verifier out of sync.\n");
+ VMW_DEBUG_USER("Command verifier out of sync.\n");
return -EINVAL;
}
static void vmw_free_relocations(struct vmw_sw_context *sw_context)
{
/* Memory is validation context memory, so no need to free it */
-
INIT_LIST_HEAD(&sw_context->bo_relocations);
}
sw_context->cmd_bounce = vmalloc(sw_context->cmd_bounce_size);
if (sw_context->cmd_bounce == NULL) {
- DRM_ERROR("Failed to allocate command bounce buffer.\n");
+ VMW_DEBUG_USER("Failed to allocate command bounce buffer.\n");
sw_context->cmd_bounce_size = 0;
return -ENOMEM;
}
* If this fails for some reason, We sync the fifo and return NULL.
* It is then safe to fence buffers with a NULL pointer.
*
- * If @p_handle is not NULL @file_priv must also not be NULL. Creates
- * a userspace handle if @p_handle is not NULL, otherwise not.
+ * If @p_handle is not NULL @file_priv must also not be NULL. Creates a
+ * userspace handle if @p_handle is not NULL, otherwise not.
*/
int vmw_execbuf_fence_commands(struct drm_file *file_priv,
ret = vmw_fifo_send_fence(dev_priv, &sequence);
if (unlikely(ret != 0)) {
- DRM_ERROR("Fence submission error. Syncing.\n");
+ VMW_DEBUG_USER("Fence submission error. Syncing.\n");
synced = true;
}
ret = vmw_fence_create(dev_priv->fman, sequence, p_fence);
if (unlikely(ret != 0 && !synced)) {
- (void) vmw_fallback_wait(dev_priv, false, false,
- sequence, false,
- VMW_FENCE_WAIT_TIMEOUT);
+ (void) vmw_fallback_wait(dev_priv, false, false, sequence,
+ false, VMW_FENCE_WAIT_TIMEOUT);
*p_fence = NULL;
}
}
/**
- * vmw_execbuf_copy_fence_user - copy fence object information to
- * user-space.
+ * vmw_execbuf_copy_fence_user - copy fence object information to user-space.
*
* @dev_priv: Pointer to a vmw_private struct.
* @vmw_fp: Pointer to the struct vmw_fpriv representing the calling file.
* @ret: Return value from fence object creation.
- * @user_fence_rep: User space address of a struct drm_vmw_fence_rep to
- * which the information should be copied.
+ * @user_fence_rep: User space address of a struct drm_vmw_fence_rep to which
+ * the information should be copied.
* @fence: Pointer to the fenc object.
* @fence_handle: User-space fence handle.
* @out_fence_fd: exported file descriptor for the fence. -1 if not used
* @sync_file: Only used to clean up in case of an error in this function.
*
- * This function copies fence information to user-space. If copying fails,
- * The user-space struct drm_vmw_fence_rep::error member is hopefully
- * left untouched, and if it's preloaded with an -EFAULT by user-space,
- * the error will hopefully be detected.
- * Also if copying fails, user-space will be unable to signal the fence
- * object so we wait for it immediately, and then unreference the
- * user-space reference.
+ * This function copies fence information to user-space. If copying fails, the
+ * user-space struct drm_vmw_fence_rep::error member is hopefully left
+ * untouched, and if it's preloaded with an -EFAULT by user-space, the error
+ * will hopefully be detected.
+ *
+ * Also if copying fails, user-space will be unable to signal the fence object
+ * so we wait for it immediately, and then unreference the user-space reference.
*/
void
vmw_execbuf_copy_fence_user(struct vmw_private *dev_priv,
- struct vmw_fpriv *vmw_fp,
- int ret,
+ struct vmw_fpriv *vmw_fp, int ret,
struct drm_vmw_fence_rep __user *user_fence_rep,
- struct vmw_fence_obj *fence,
- uint32_t fence_handle,
- int32_t out_fence_fd,
- struct sync_file *sync_file)
+ struct vmw_fence_obj *fence, uint32_t fence_handle,
+ int32_t out_fence_fd, struct sync_file *sync_file)
{
struct drm_vmw_fence_rep fence_rep;
}
/*
- * copy_to_user errors will be detected by user space not
- * seeing fence_rep::error filled in. Typically
- * user-space would have pre-set that member to -EFAULT.
+ * copy_to_user errors will be detected by user space not seeing
+ * fence_rep::error filled in. Typically user-space would have pre-set
+ * that member to -EFAULT.
*/
ret = copy_to_user(user_fence_rep, &fence_rep,
sizeof(fence_rep));
/*
- * User-space lost the fence object. We need to sync
- * and unreference the handle.
+ * User-space lost the fence object. We need to sync and unreference the
+ * handle.
*/
if (unlikely(ret != 0) && (fence_rep.error == 0)) {
if (sync_file)
fence_rep.fd = -1;
}
- ttm_ref_object_base_unref(vmw_fp->tfile,
- fence_handle, TTM_REF_USAGE);
- DRM_ERROR("Fence copy error. Syncing.\n");
+ ttm_ref_object_base_unref(vmw_fp->tfile, fence_handle,
+ TTM_REF_USAGE);
+ VMW_DEBUG_USER("Fence copy error. Syncing.\n");
(void) vmw_fence_obj_wait(fence, false, false,
VMW_FENCE_WAIT_TIMEOUT);
}
}
/**
- * vmw_execbuf_submit_fifo - Patch a command batch and submit it using
- * the fifo.
+ * vmw_execbuf_submit_fifo - Patch a command batch and submit it using the fifo.
*
* @dev_priv: Pointer to a device private structure.
* @kernel_commands: Pointer to the unpatched command batch.
* @command_size: Size of the unpatched command batch.
* @sw_context: Structure holding the relocation lists.
*
- * Side effects: If this function returns 0, then the command batch
- * pointed to by @kernel_commands will have been modified.
+ * Side effects: If this function returns 0, then the command batch pointed to
+ * by @kernel_commands will have been modified.
*/
static int vmw_execbuf_submit_fifo(struct vmw_private *dev_priv,
- void *kernel_commands,
- u32 command_size,
+ void *kernel_commands, u32 command_size,
struct vmw_sw_context *sw_context)
{
void *cmd;
if (sw_context->dx_ctx_node)
- cmd = vmw_fifo_reserve_dx(dev_priv, command_size,
+ cmd = VMW_FIFO_RESERVE_DX(dev_priv, command_size,
sw_context->dx_ctx_node->ctx->id);
else
- cmd = vmw_fifo_reserve(dev_priv, command_size);
- if (!cmd) {
- DRM_ERROR("Failed reserving fifo space for commands.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, command_size);
+
+ if (!cmd)
return -ENOMEM;
- }
vmw_apply_relocations(sw_context);
memcpy(cmd, kernel_commands, command_size);
}
/**
- * vmw_execbuf_submit_cmdbuf - Patch a command batch and submit it using
- * the command buffer manager.
+ * vmw_execbuf_submit_cmdbuf - Patch a command batch and submit it using the
+ * command buffer manager.
*
* @dev_priv: Pointer to a device private structure.
* @header: Opaque handle to the command buffer allocation.
* @command_size: Size of the unpatched command batch.
* @sw_context: Structure holding the relocation lists.
*
- * Side effects: If this function returns 0, then the command buffer
- * represented by @header will have been modified.
+ * Side effects: If this function returns 0, then the command buffer represented
+ * by @header will have been modified.
*/
static int vmw_execbuf_submit_cmdbuf(struct vmw_private *dev_priv,
struct vmw_cmdbuf_header *header,
{
u32 id = ((sw_context->dx_ctx_node) ? sw_context->dx_ctx_node->ctx->id :
SVGA3D_INVALID_ID);
- void *cmd = vmw_cmdbuf_reserve(dev_priv->cman, command_size,
- id, false, header);
+ void *cmd = vmw_cmdbuf_reserve(dev_priv->cman, command_size, id, false,
+ header);
vmw_apply_relocations(sw_context);
vmw_resource_relocations_apply(cmd, &sw_context->res_relocations);
* @header: Out parameter returning the opaque pointer to the command buffer.
*
* This function checks whether we can use the command buffer manager for
- * submission and if so, creates a command buffer of suitable size and
- * copies the user data into that buffer.
+ * submission and if so, creates a command buffer of suitable size and copies
+ * the user data into that buffer.
*
* On successful return, the function returns a pointer to the data in the
* command buffer and *@header is set to non-NULL.
- * If command buffers could not be used, the function will return the value
- * of @kernel_commands on function call. That value may be NULL. In that case,
- * the value of *@header will be set to NULL.
+ *
+ * If command buffers could not be used, the function will return the value of
+ * @kernel_commands on function call. That value may be NULL. In that case, the
+ * value of *@header will be set to NULL.
+ *
* If an error is encountered, the function will return a pointer error value.
* If the function is interrupted by a signal while sleeping, it will return
* -ERESTARTSYS casted to a pointer error value.
*/
static void *vmw_execbuf_cmdbuf(struct vmw_private *dev_priv,
void __user *user_commands,
- void *kernel_commands,
- u32 command_size,
+ void *kernel_commands, u32 command_size,
struct vmw_cmdbuf_header **header)
{
size_t cmdbuf_size;
*header = NULL;
if (command_size > SVGA_CB_MAX_SIZE) {
- DRM_ERROR("Command buffer is too large.\n");
+ VMW_DEBUG_USER("Command buffer is too large.\n");
return ERR_PTR(-EINVAL);
}
/* If possible, add a little space for fencing. */
cmdbuf_size = command_size + 512;
cmdbuf_size = min_t(size_t, cmdbuf_size, SVGA_CB_MAX_SIZE);
- kernel_commands = vmw_cmdbuf_alloc(dev_priv->cman, cmdbuf_size,
- true, header);
+ kernel_commands = vmw_cmdbuf_alloc(dev_priv->cman, cmdbuf_size, true,
+ header);
if (IS_ERR(kernel_commands))
return kernel_commands;
- ret = copy_from_user(kernel_commands, user_commands,
- command_size);
+ ret = copy_from_user(kernel_commands, user_commands, command_size);
if (ret) {
- DRM_ERROR("Failed copying commands.\n");
+ VMW_DEBUG_USER("Failed copying commands.\n");
vmw_cmdbuf_header_free(*header);
*header = NULL;
return ERR_PTR(-EFAULT);
res = vmw_user_resource_noref_lookup_handle
(dev_priv, sw_context->fp->tfile, handle,
user_context_converter);
- if (unlikely(IS_ERR(res))) {
- DRM_ERROR("Could not find or user DX context 0x%08x.\n",
- (unsigned) handle);
+ if (IS_ERR(res)) {
+ VMW_DEBUG_USER("Could not find or user DX context 0x%08x.\n",
+ (unsigned int) handle);
return PTR_ERR(res);
}
- ret = vmw_execbuf_res_noref_val_add(sw_context, res);
+ ret = vmw_execbuf_res_noref_val_add(sw_context, res, VMW_RES_DIRTY_SET);
if (unlikely(ret != 0))
return ret;
int vmw_execbuf_process(struct drm_file *file_priv,
struct vmw_private *dev_priv,
- void __user *user_commands,
- void *kernel_commands,
- uint32_t command_size,
- uint64_t throttle_us,
+ void __user *user_commands, void *kernel_commands,
+ uint32_t command_size, uint64_t throttle_us,
uint32_t dx_context_handle,
struct drm_vmw_fence_rep __user *user_fence_rep,
- struct vmw_fence_obj **out_fence,
- uint32_t flags)
+ struct vmw_fence_obj **out_fence, uint32_t flags)
{
struct vmw_sw_context *sw_context = &dev_priv->ctx;
struct vmw_fence_obj *fence = NULL;
struct vmw_cmdbuf_header *header;
- uint32_t handle;
+ uint32_t handle = 0;
int ret;
int32_t out_fence_fd = -1;
struct sync_file *sync_file = NULL;
if (flags & DRM_VMW_EXECBUF_FLAG_EXPORT_FENCE_FD) {
out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
if (out_fence_fd < 0) {
- DRM_ERROR("Failed to get a fence file descriptor.\n");
+ VMW_DEBUG_USER("Failed to get a fence fd.\n");
return out_fence_fd;
}
}
if (unlikely(ret != 0))
goto out_unlock;
-
- ret = copy_from_user(sw_context->cmd_bounce,
- user_commands, command_size);
-
+ ret = copy_from_user(sw_context->cmd_bounce, user_commands,
+ command_size);
if (unlikely(ret != 0)) {
ret = -EFAULT;
- DRM_ERROR("Failed copying commands.\n");
+ VMW_DEBUG_USER("Failed copying commands.\n");
goto out_unlock;
}
+
kernel_commands = sw_context->cmd_bounce;
- } else if (!header)
+ } else if (!header) {
sw_context->kernel = true;
+ }
sw_context->fp = vmw_fpriv(file_priv);
INIT_LIST_HEAD(&sw_context->ctx_list);
memset(sw_context->res_cache, 0, sizeof(sw_context->res_cache));
INIT_LIST_HEAD(&sw_context->res_relocations);
INIT_LIST_HEAD(&sw_context->bo_relocations);
+
if (sw_context->staged_bindings)
vmw_binding_state_reset(sw_context->staged_bindings);
ret = drm_ht_create(&sw_context->res_ht, VMW_RES_HT_ORDER);
if (unlikely(ret != 0))
goto out_unlock;
+
sw_context->res_ht_initialized = true;
}
+
INIT_LIST_HEAD(&sw_context->staged_cmd_res);
sw_context->ctx = &val_ctx;
ret = vmw_execbuf_tie_context(dev_priv, sw_context, dx_context_handle);
ret = vmw_validation_res_validate(&val_ctx, true);
if (unlikely(ret != 0))
goto out_err;
+
vmw_validation_drop_ht(&val_ctx);
ret = mutex_lock_interruptible(&dev_priv->binding_mutex);
goto out_err;
vmw_query_bo_switch_commit(dev_priv, sw_context);
- ret = vmw_execbuf_fence_commands(file_priv, dev_priv,
- &fence,
+ ret = vmw_execbuf_fence_commands(file_priv, dev_priv, &fence,
(user_fence_rep) ? &handle : NULL);
/*
* This error is harmless, because if fence submission fails,
* vmw_fifo_send_fence will sync. The error will be propagated to
* user-space in @fence_rep
*/
-
if (ret != 0)
- DRM_ERROR("Fence submission error. Syncing.\n");
+ VMW_DEBUG_USER("Fence submission error. Syncing.\n");
vmw_execbuf_bindings_commit(sw_context, false);
vmw_bind_dx_query_mob(sw_context);
vmw_validation_bo_fence(sw_context->ctx, fence);
- if (unlikely(dev_priv->pinned_bo != NULL &&
- !dev_priv->query_cid_valid))
+ if (unlikely(dev_priv->pinned_bo != NULL && !dev_priv->query_cid_valid))
__vmw_execbuf_release_pinned_bo(dev_priv, fence);
/*
- * If anything fails here, give up trying to export the fence
- * and do a sync since the user mode will not be able to sync
- * the fence itself. This ensures we are still functionally
- * correct.
+ * If anything fails here, give up trying to export the fence and do a
+ * sync since the user mode will not be able to sync the fence itself.
+ * This ensures we are still functionally correct.
*/
if (flags & DRM_VMW_EXECBUF_FLAG_EXPORT_FENCE_FD) {
sync_file = sync_file_create(&fence->base);
if (!sync_file) {
- DRM_ERROR("Unable to create sync file for fence\n");
+ VMW_DEBUG_USER("Sync file create failed for fence\n");
put_unused_fd(out_fence_fd);
out_fence_fd = -1;
}
vmw_execbuf_copy_fence_user(dev_priv, vmw_fpriv(file_priv), ret,
- user_fence_rep, fence, handle,
- out_fence_fd, sync_file);
+ user_fence_rep, fence, handle, out_fence_fd,
+ sync_file);
/* Don't unreference when handing fence out */
if (unlikely(out_fence != NULL)) {
mutex_unlock(&dev_priv->cmdbuf_mutex);
/*
- * Unreference resources outside of the cmdbuf_mutex to
- * avoid deadlocks in resource destruction paths.
+ * Unreference resources outside of the cmdbuf_mutex to avoid deadlocks
+ * in resource destruction paths.
*/
vmw_validation_unref_lists(&val_ctx);
vmw_validation_res_unreserve(&val_ctx, true);
vmw_resource_relocations_free(&sw_context->res_relocations);
vmw_free_relocations(sw_context);
- if (unlikely(dev_priv->pinned_bo != NULL &&
- !dev_priv->query_cid_valid))
+ if (unlikely(dev_priv->pinned_bo != NULL && !dev_priv->query_cid_valid))
__vmw_execbuf_release_pinned_bo(dev_priv, NULL);
out_unlock:
vmw_cmdbuf_res_revert(&sw_context->staged_cmd_res);
mutex_unlock(&dev_priv->cmdbuf_mutex);
/*
- * Unreference resources outside of the cmdbuf_mutex to
- * avoid deadlocks in resource destruction paths.
+ * Unreference resources outside of the cmdbuf_mutex to avoid deadlocks
+ * in resource destruction paths.
*/
vmw_validation_unref_lists(&val_ctx);
out_free_header:
*
* @dev_priv: The device private structure.
*
- * This function is called to idle the fifo and unpin the query buffer
- * if the normal way to do this hits an error, which should typically be
- * extremely rare.
+ * This function is called to idle the fifo and unpin the query buffer if the
+ * normal way to do this hits an error, which should typically be extremely
+ * rare.
*/
static void vmw_execbuf_unpin_panic(struct vmw_private *dev_priv)
{
- DRM_ERROR("Can't unpin query buffer. Trying to recover.\n");
+ VMW_DEBUG_USER("Can't unpin query buffer. Trying to recover.\n");
(void) vmw_fallback_wait(dev_priv, false, true, 0, false, 10*HZ);
vmw_bo_pin_reserved(dev_priv->pinned_bo, false);
/**
- * __vmw_execbuf_release_pinned_bo - Flush queries and unpin the pinned
- * query bo.
+ * __vmw_execbuf_release_pinned_bo - Flush queries and unpin the pinned query
+ * bo.
*
* @dev_priv: The device private structure.
- * @fence: If non-NULL should point to a struct vmw_fence_obj issued
- * _after_ a query barrier that flushes all queries touching the current
- * buffer pointed to by @dev_priv->pinned_bo
+ * @fence: If non-NULL should point to a struct vmw_fence_obj issued _after_ a
+ * query barrier that flushes all queries touching the current buffer pointed to
+ * by @dev_priv->pinned_bo
*
- * This function should be used to unpin the pinned query bo, or
- * as a query barrier when we need to make sure that all queries have
- * finished before the next fifo command. (For example on hardware
- * context destructions where the hardware may otherwise leak unfinished
- * queries).
+ * This function should be used to unpin the pinned query bo, or as a query
+ * barrier when we need to make sure that all queries have finished before the
+ * next fifo command. (For example on hardware context destructions where the
+ * hardware may otherwise leak unfinished queries).
*
- * This function does not return any failure codes, but make attempts
- * to do safe unpinning in case of errors.
+ * This function does not return any failure codes, but make attempts to do safe
+ * unpinning in case of errors.
*
- * The function will synchronize on the previous query barrier, and will
- * thus not finish until that barrier has executed.
+ * The function will synchronize on the previous query barrier, and will thus
+ * not finish until that barrier has executed.
*
- * the @dev_priv->cmdbuf_mutex needs to be held by the current thread
- * before calling this function.
+ * the @dev_priv->cmdbuf_mutex needs to be held by the current thread before
+ * calling this function.
*/
void __vmw_execbuf_release_pinned_bo(struct vmw_private *dev_priv,
struct vmw_fence_obj *fence)
vmw_validation_unref_lists(&val_ctx);
vmw_bo_unreference(&dev_priv->pinned_bo);
+
out_unlock:
return;
-
out_no_emit:
vmw_validation_bo_backoff(&val_ctx);
out_no_reserve:
vmw_validation_unref_lists(&val_ctx);
vmw_execbuf_unpin_panic(dev_priv);
vmw_bo_unreference(&dev_priv->pinned_bo);
-
}
/**
- * vmw_execbuf_release_pinned_bo - Flush queries and unpin the pinned
- * query bo.
+ * vmw_execbuf_release_pinned_bo - Flush queries and unpin the pinned query bo.
*
* @dev_priv: The device private structure.
*
- * This function should be used to unpin the pinned query bo, or
- * as a query barrier when we need to make sure that all queries have
- * finished before the next fifo command. (For example on hardware
- * context destructions where the hardware may otherwise leak unfinished
- * queries).
+ * This function should be used to unpin the pinned query bo, or as a query
+ * barrier when we need to make sure that all queries have finished before the
+ * next fifo command. (For example on hardware context destructions where the
+ * hardware may otherwise leak unfinished queries).
*
- * This function does not return any failure codes, but make attempts
- * to do safe unpinning in case of errors.
+ * This function does not return any failure codes, but make attempts to do safe
+ * unpinning in case of errors.
*
- * The function will synchronize on the previous query barrier, and will
- * thus not finish until that barrier has executed.
+ * The function will synchronize on the previous query barrier, and will thus
+ * not finish until that barrier has executed.
*/
void vmw_execbuf_release_pinned_bo(struct vmw_private *dev_priv)
{
struct dma_fence *in_fence = NULL;
if (unlikely(size < copy_offset[0])) {
- DRM_ERROR("Invalid command size, ioctl %d\n",
- DRM_VMW_EXECBUF);
+ VMW_DEBUG_USER("Invalid command size, ioctl %d\n",
+ DRM_VMW_EXECBUF);
return -EINVAL;
}
return -EFAULT;
/*
- * Extend the ioctl argument while
- * maintaining backwards compatibility:
- * We take different code paths depending on the value of
- * arg.version.
+ * Extend the ioctl argument while maintaining backwards compatibility:
+ * We take different code paths depending on the value of arg.version.
*/
-
if (unlikely(arg.version > DRM_VMW_EXECBUF_VERSION ||
arg.version == 0)) {
- DRM_ERROR("Incorrect execbuf version.\n");
+ VMW_DEBUG_USER("Incorrect execbuf version.\n");
return -EINVAL;
}
if (arg.version > 1 &&
copy_from_user(&arg.context_handle,
(void __user *) (data + copy_offset[0]),
- copy_offset[arg.version - 1] -
- copy_offset[0]) != 0)
+ copy_offset[arg.version - 1] - copy_offset[0]) != 0)
return -EFAULT;
switch (arg.version) {
break;
}
-
/* If imported a fence FD from elsewhere, then wait on it */
if (arg.flags & DRM_VMW_EXECBUF_FLAG_IMPORT_FENCE_FD) {
in_fence = sync_file_get_fence(arg.imported_fence_fd);
if (!in_fence) {
- DRM_ERROR("Cannot get imported fence\n");
+ VMW_DEBUG_USER("Cannot get imported fence\n");
return -EINVAL;
}
NULL, arg.command_size, arg.throttle_us,
arg.context_handle,
(void __user *)(unsigned long)arg.fence_rep,
- NULL,
- arg.flags);
+ NULL, arg.flags);
+
ttm_read_unlock(&dev_priv->reservation_sem);
if (unlikely(ret != 0))
goto out;
struct vmw_fb_par *par;
struct fb_info *info;
unsigned fb_width, fb_height;
- unsigned fb_bpp, fb_depth, fb_offset, fb_pitch, fb_size;
+ unsigned int fb_bpp, fb_pitch, fb_size;
struct drm_display_mode *init_mode;
int ret;
fb_bpp = 32;
- fb_depth = 24;
/* XXX As shouldn't these be as well. */
fb_width = min(vmw_priv->fb_max_width, (unsigned)2048);
fb_pitch = fb_width * fb_bpp / 8;
fb_size = fb_pitch * fb_height;
- fb_offset = vmw_read(vmw_priv, SVGA_REG_FB_OFFSET);
info = framebuffer_alloc(sizeof(*par), device);
if (!info)
WARN(1, "Command buffer has not been allocated.\n");
ret = NULL;
}
- if (IS_ERR_OR_NULL(ret)) {
- DRM_ERROR("Fifo reserve failure of %u bytes.\n",
- (unsigned) bytes);
- dump_stack();
+ if (IS_ERR_OR_NULL(ret))
return NULL;
- }
return ret;
}
int ret = 0;
uint32_t bytes = sizeof(u32) + sizeof(*cmd_fence);
- fm = vmw_fifo_reserve(dev_priv, bytes);
+ fm = VMW_FIFO_RESERVE(dev_priv, bytes);
if (unlikely(fm == NULL)) {
*seqno = atomic_read(&dev_priv->marker_seq);
ret = -ENOMEM;
SVGA3dCmdWaitForQuery body;
} *cmd;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
-
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Out of fifo space for dummy query.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_WAIT_FOR_QUERY;
cmd->header.size = sizeof(cmd->body);
SVGA3dCmdWaitForGBQuery body;
} *cmd;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
-
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Out of fifo space for dummy query.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_WAIT_FOR_GB_QUERY;
cmd->header.size = sizeof(cmd->body);
return vmw_fifo_emit_dummy_legacy_query(dev_priv, cid);
}
-
-void *vmw_fifo_reserve(struct vmw_private *dev_priv, uint32_t bytes)
-{
- return vmw_fifo_reserve_dx(dev_priv, bytes, SVGA3D_INVALID_ID);
-}
uint32_t cmd_size = define_size + remap_size;
uint32_t i;
- cmd_orig = cmd = vmw_fifo_reserve(dev_priv, cmd_size);
+ cmd_orig = cmd = VMW_FIFO_RESERVE(dev_priv, cmd_size);
if (unlikely(cmd == NULL))
return -ENOMEM;
uint32_t define_size = sizeof(define_cmd) + 4;
uint32_t *cmd;
- cmd = vmw_fifo_reserve(dev_priv, define_size);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("GMR2 unbind failed.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, define_size);
+ if (unlikely(cmd == NULL))
return;
- }
+
define_cmd.gmrId = gmr_id;
define_cmd.numPages = 0;
struct vmw_fpriv *vmw_fp = vmw_fpriv(file_priv);
if (unlikely(arg->pad64 != 0 || arg->max_size == 0)) {
- DRM_ERROR("Illegal GET_3D_CAP argument.\n");
+ VMW_DEBUG_USER("Illegal GET_3D_CAP argument.\n");
return -EINVAL;
}
return 0;
if (clips_ptr == NULL) {
- DRM_ERROR("Variable clips_ptr must be specified.\n");
+ VMW_DEBUG_USER("Variable clips_ptr must be specified.\n");
ret = -EINVAL;
goto out_clips;
}
fb = drm_framebuffer_lookup(dev, file_priv, arg->fb_id);
if (!fb) {
- DRM_ERROR("Invalid framebuffer id.\n");
+ VMW_DEBUG_USER("Invalid framebuffer id.\n");
ret = -ENOENT;
goto out_no_fb;
}
return 0;
if (clips_ptr == NULL) {
- DRM_ERROR("Argument clips_ptr must be specified.\n");
+ VMW_DEBUG_USER("Argument clips_ptr must be specified.\n");
ret = -EINVAL;
goto out_clips;
}
fb = drm_framebuffer_lookup(dev, file_priv, arg->fb_id);
if (!fb) {
- DRM_ERROR("Invalid framebuffer id.\n");
+ VMW_DEBUG_USER("Invalid framebuffer id.\n");
ret = -ENOENT;
goto out_no_fb;
}
vfb = vmw_framebuffer_to_vfb(fb);
if (!vfb->bo) {
- DRM_ERROR("Framebuffer not buffer backed.\n");
+ VMW_DEBUG_USER("Framebuffer not buffer backed.\n");
ret = -EINVAL;
goto out_no_ttm_lock;
}
if (!image)
return -EINVAL;
- cmd = vmw_fifo_reserve(dev_priv, cmd_size);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Fifo reserve failed.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, cmd_size);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
memset(cmd, 0, sizeof(*cmd));
vmw_bo_unreference(&res->backup);
res->backup = vmw_bo_reference(bo_mob);
res->backup_offset = 0;
- vmw_resource_unreserve(res, false, NULL, 0);
+ vmw_resource_unreserve(res, false, false, false, NULL, 0);
mutex_unlock(&res->dev_priv->cmdbuf_mutex);
return 0;
dirty->unit = unit;
if (dirty->fifo_reserve_size > 0) {
- dirty->cmd = vmw_fifo_reserve(dev_priv,
+ dirty->cmd = VMW_FIFO_RESERVE(dev_priv,
dirty->fifo_reserve_size);
- if (!dirty->cmd) {
- DRM_ERROR("Couldn't reserve fifo space "
- "for dirty blits.\n");
+ if (!dirty->cmd)
return -ENOMEM;
- }
+
memset(dirty->cmd, 0, dirty->fifo_reserve_size);
}
dirty->num_hits = 0;
if (!clips)
return 0;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd) * num_clips);
- if (!cmd) {
- DRM_ERROR("Couldn't reserve fifo space for proxy surface "
- "update.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd) * num_clips);
+ if (!cmd)
return -ENOMEM;
- }
for (i = 0; i < num_clips; ++i, clips += increment, ++cmd) {
box = &cmd->body.box;
container_of(update->vfb, typeof(*vfbs), base);
ret = vmw_validation_add_resource(&val_ctx, &vfbs->surface->res,
- 0, NULL, NULL);
+ 0, VMW_RES_DIRTY_NONE, NULL,
+ NULL);
}
if (ret)
goto out_unref;
reserved_size = update->calc_fifo_size(update, num_hits);
- cmd_start = vmw_fifo_reserve(update->dev_priv, reserved_size);
+ cmd_start = VMW_FIFO_RESERVE(update->dev_priv, reserved_size);
if (!cmd_start) {
ret = -ENOMEM;
goto out_revert;
} *cmd;
fifo_size = sizeof(*cmd) * num_clips;
- cmd = vmw_fifo_reserve(dev_priv, fifo_size);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Fifo reserve failed.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, fifo_size);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
memset(cmd, 0, fifo_size);
for (i = 0; i < num_clips; i++, clips += increment) {
mob->pt_level += VMW_MOBFMT_PTDEPTH_1 - SVGA3D_MOBFMT_PTDEPTH_1;
}
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for OTable setup.\n");
ret = -ENOMEM;
goto out_no_fifo;
}
return;
bo = otable->page_table->pt_bo;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for OTable "
- "takedown.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return;
- }
memset(cmd, 0, sizeof(*cmd));
cmd->header.id = SVGA_3D_CMD_SET_OTABLE_BASE;
BUG_ON(ret != 0);
}
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for Memory "
- "Object unbinding.\n");
- } else {
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (cmd) {
cmd->header.id = SVGA_3D_CMD_DESTROY_GB_MOB;
cmd->header.size = sizeof(cmd->body);
cmd->body.mobid = mob->id;
vmw_fifo_commit(dev_priv, sizeof(*cmd));
}
+
if (bo) {
vmw_bo_fence_single(bo, NULL);
ttm_bo_unreserve(bo);
vmw_fifo_resource_inc(dev_priv);
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for Memory "
- "Object binding.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
goto out_no_cmd_space;
- }
cmd->header.id = SVGA_3D_CMD_DEFINE_GB_MOB64;
cmd->header.size = sizeof(cmd->body);
fifo_size = sizeof(*cmds) + sizeof(*flush) + sizeof(*items) * num_items;
- cmds = vmw_fifo_reserve(dev_priv, fifo_size);
+ cmds = VMW_FIFO_RESERVE(dev_priv, fifo_size);
/* hardware has hung, can't do anything here */
if (!cmds)
return -ENOMEM;
int ret;
for (;;) {
- cmds = vmw_fifo_reserve(dev_priv, sizeof(*cmds));
+ cmds = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmds));
if (cmds)
break;
list_add_tail(&res->mob_head, &res->backup->res_list);
}
- /*
- * Only do this on write operations, and move to
- * vmw_resource_unreserve if it can be called after
- * backup buffers have been unreserved. Otherwise
- * sort out locking.
- */
- res->res_dirty = true;
-
return 0;
out_bind_failed:
* command submission.
*
* @res: Pointer to the struct vmw_resource to unreserve.
+ * @dirty_set: Change dirty status of the resource.
+ * @dirty: When changing dirty status indicates the new status.
* @switch_backup: Backup buffer has been switched.
* @new_backup: Pointer to new backup buffer if command submission
* switched. May be NULL.
* resource lru list, so that it can be evicted if necessary.
*/
void vmw_resource_unreserve(struct vmw_resource *res,
+ bool dirty_set,
+ bool dirty,
bool switch_backup,
struct vmw_buffer_object *new_backup,
unsigned long new_backup_offset)
if (switch_backup)
res->backup_offset = new_backup_offset;
+ if (dirty_set)
+ res->res_dirty = dirty;
+
if (!res->func->may_evict || res->id == -1 || res->pin_count)
return;
if (!res->func->unbind)
continue;
- (void) res->func->unbind(res, true, &val_buf);
+ (void) res->func->unbind(res, res->res_dirty, &val_buf);
res->backup_dirty = true;
res->res_dirty = false;
list_del_init(&res->mob_head);
dx_query_ctx = dx_query_mob->dx_query_ctx;
dev_priv = dx_query_ctx->dev_priv;
- cmd = vmw_fifo_reserve_dx(dev_priv, sizeof(*cmd), dx_query_ctx->id);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for "
- "query MOB read back.\n");
+ cmd = VMW_FIFO_RESERVE_DX(dev_priv, sizeof(*cmd), dx_query_ctx->id);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_READBACK_ALL_QUERY;
cmd->header.size = sizeof(cmd->body);
res->pin_count++;
out_no_validate:
- vmw_resource_unreserve(res, false, NULL, 0UL);
+ vmw_resource_unreserve(res, false, false, false, NULL, 0UL);
out_no_reserve:
mutex_unlock(&dev_priv->cmdbuf_mutex);
ttm_write_unlock(&dev_priv->reservation_sem);
ttm_bo_unreserve(&vbo->base);
}
- vmw_resource_unreserve(res, false, NULL, 0UL);
+ vmw_resource_unreserve(res, false, false, false, NULL, 0UL);
mutex_unlock(&dev_priv->cmdbuf_mutex);
ttm_read_unlock(&dev_priv->reservation_sem);
BUG_ON(!sou->buffer);
fifo_size = sizeof(*cmd);
- cmd = vmw_fifo_reserve(dev_priv, fifo_size);
- /* The hardware has hung, nothing we can do about it here. */
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Fifo reserve failed.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, fifo_size);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
memset(cmd, 0, fifo_size);
cmd->header.cmdType = SVGA_CMD_DEFINE_SCREEN;
return 0;
fifo_size = sizeof(*cmd);
- cmd = vmw_fifo_reserve(dev_priv, fifo_size);
- /* the hardware has hung, nothing we can do about it here */
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Fifo reserve failed.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, fifo_size);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
memset(cmd, 0, fifo_size);
cmd->header.cmdType = SVGA_CMD_DESTROY_SCREEN;
if (depth == 32)
depth = 24;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (!cmd) {
- DRM_ERROR("Out of fifo space for dirty framebuffer command.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (!cmd)
return -ENOMEM;
- }
cmd->header = SVGA_CMD_DEFINE_GMRFB;
cmd->body.format.bitsPerPixel = framebuffer->base.format->cpp[0] * 8;
if (!srf)
srf = &vfbs->surface->res;
- ret = vmw_validation_add_resource(&val_ctx, srf, 0, NULL, NULL);
+ ret = vmw_validation_add_resource(&val_ctx, srf, 0, VMW_RES_DIRTY_NONE,
+ NULL, NULL);
if (ret)
return ret;
goto out_no_fifo;
}
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for shader "
- "creation.\n");
ret = -ENOMEM;
goto out_no_fifo;
}
BUG_ON(bo->mem.mem_type != VMW_PL_MOB);
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for shader "
- "binding.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_BIND_GB_SHADER;
cmd->header.size = sizeof(cmd->body);
BUG_ON(res->backup->base.mem.mem_type != VMW_PL_MOB);
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for shader "
- "unbinding.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_BIND_GB_SHADER;
cmd->header.size = sizeof(cmd->body);
mutex_lock(&dev_priv->binding_mutex);
vmw_binding_res_list_scrub(&res->binding_head);
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for shader "
- "destruction.\n");
mutex_unlock(&dev_priv->binding_mutex);
return -ENOMEM;
}
if (!list_empty(&shader->cotable_head) || !shader->committed)
return 0;
- cmd = vmw_fifo_reserve_dx(dev_priv, sizeof(*cmd),
- shader->ctx->id);
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for shader "
- "scrubbing.\n");
+ cmd = VMW_FIFO_RESERVE_DX(dev_priv, sizeof(*cmd), shader->ctx->id);
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_BIND_SHADER;
cmd->header.size = sizeof(cmd->body);
return 0;
WARN_ON_ONCE(!shader->committed);
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Failed reserving FIFO space for shader "
- "scrubbing.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DX_BIND_SHADER;
cmd->header.size = sizeof(cmd->body);
ret = vmw_user_bo_lookup(tfile, buffer_handle,
&buffer, NULL);
if (unlikely(ret != 0)) {
- DRM_ERROR("Could not find buffer for shader "
- "creation.\n");
+ VMW_DEBUG_USER("Couldn't find buffer for shader creation.\n");
return ret;
}
if ((u64)buffer->base.num_pages * PAGE_SIZE <
(u64)size + (u64)offset) {
- DRM_ERROR("Illegal buffer- or shader size.\n");
+ VMW_DEBUG_USER("Illegal buffer- or shader size.\n");
ret = -EINVAL;
goto out_bad_arg;
}
shader_type = SVGA3D_SHADERTYPE_PS;
break;
default:
- DRM_ERROR("Illegal shader type.\n");
+ VMW_DEBUG_USER("Illegal shader type.\n");
ret = -EINVAL;
goto out_bad_arg;
}
base = ttm_base_object_lookup(tfile, handle);
if (!base) {
- DRM_ERROR("Invalid %s handle 0x%08lx.\n",
- func->res_func.type_name,
- (unsigned long) handle);
+ VMW_DEBUG_USER("Invalid %s handle 0x%08lx.\n",
+ func->res_func.type_name,
+ (unsigned long) handle);
return ERR_PTR(-ESRCH);
}
if (ttm_base_object_type(base) != func->ttm_res_type) {
ttm_base_object_unref(&base);
- DRM_ERROR("Invalid type of %s handle 0x%08lx.\n",
- func->res_func.type_name,
- (unsigned long) handle);
+ VMW_DEBUG_USER("Invalid type of %s handle 0x%08lx.\n",
+ func->res_func.type_name,
+ (unsigned long) handle);
return ERR_PTR(-EINVAL);
}
return 0;
}
- cmd = vmw_fifo_reserve_dx(res->dev_priv, view->cmd_size,
- view->ctx->id);
+ cmd = VMW_FIFO_RESERVE_DX(res->dev_priv, view->cmd_size, view->ctx->id);
if (!cmd) {
- DRM_ERROR("Failed reserving FIFO space for view creation.\n");
mutex_unlock(&dev_priv->binding_mutex);
return -ENOMEM;
}
+
memcpy(cmd, &view->cmd, view->cmd_size);
WARN_ON(cmd->body.view_id != view->view_id);
/* Sid may have changed due to surface eviction. */
if (!view->committed || res->id == -1)
return 0;
- cmd = vmw_fifo_reserve_dx(dev_priv, sizeof(*cmd), view->ctx->id);
- if (!cmd) {
- DRM_ERROR("Failed reserving FIFO space for view "
- "destruction.\n");
+ cmd = VMW_FIFO_RESERVE_DX(dev_priv, sizeof(*cmd), view->ctx->id);
+ if (!cmd)
return -ENOMEM;
- }
cmd->header.id = vmw_view_destroy_cmds[view->view_type];
cmd->header.size = sizeof(cmd->body);
if (cmd_size != vmw_view_define_sizes[view_type] +
sizeof(SVGA3dCmdHeader)) {
- DRM_ERROR("Illegal view create command size.\n");
+ VMW_DEBUG_USER("Illegal view create command size.\n");
return -EINVAL;
}
if (!vmw_view_id_ok(user_key, view_type)) {
- DRM_ERROR("Illegal view add view id.\n");
+ VMW_DEBUG_USER("Illegal view add view id.\n");
return -EINVAL;
}
ret = ttm_mem_global_alloc(vmw_mem_glob(dev_priv), size, &ttm_opt_ctx);
if (ret) {
if (ret != -ERESTARTSYS)
- DRM_ERROR("Out of graphics memory for view"
- " creation.\n");
+ DRM_ERROR("Out of graphics memory for view creation\n");
return ret;
}
struct vmw_resource **res_p)
{
if (!vmw_view_id_ok(user_key, view_type)) {
- DRM_ERROR("Illegal view remove view id.\n");
+ VMW_DEBUG_USER("Illegal view remove view id.\n");
return -EINVAL;
}
vmw_view_key(user_key, view_type));
}
+/**
+ * vmw_view_dirtying - Return whether a view type is dirtying its resource
+ * @res: Pointer to the view
+ *
+ * Each time a resource is put on the validation list as the result of a
+ * view pointing to it, we need to determine whether that resource will
+ * be dirtied (written to by the GPU) as a result of the corresponding
+ * GPU operation. Currently only rendertarget- and depth-stencil views are
+ * capable of dirtying its resource.
+ *
+ * Return: Whether the view type of @res dirties the resource it points to.
+ */
+u32 vmw_view_dirtying(struct vmw_resource *res)
+{
+ static u32 view_is_dirtying[vmw_view_max] = {
+ [vmw_view_rt] = VMW_RES_DIRTY_SET,
+ [vmw_view_ds] = VMW_RES_DIRTY_SET,
+ };
+
+ /* Update this function as we add more view types */
+ BUILD_BUG_ON(vmw_view_max != 3);
+ return view_is_dirtying[vmw_view(res)->view_type];
+}
+
const u32 vmw_view_destroy_cmds[] = {
[vmw_view_sr] = SVGA_3D_CMD_DX_DESTROY_SHADERRESOURCE_VIEW,
[vmw_view_rt] = SVGA_3D_CMD_DX_DESTROY_RENDERTARGET_VIEW,
extern struct vmw_resource *vmw_view_lookup(struct vmw_cmdbuf_res_manager *man,
enum vmw_view_type view_type,
u32 user_key);
+extern u32 vmw_view_dirtying(struct vmw_resource *res);
#endif
*/
struct vmw_screen_target_display_unit {
struct vmw_display_unit base;
- const struct vmw_surface *display_srf;
+ struct vmw_surface *display_srf;
enum stdu_content_type content_fb_type;
s32 display_width, display_height;
SVGA3dCmdDefineGBScreenTarget body;
} *cmd;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
-
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Out of FIFO space defining Screen Target\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DEFINE_GB_SCREENTARGET;
cmd->header.size = sizeof(cmd->body);
memset(&image, 0, sizeof(image));
image.sid = res ? res->id : SVGA3D_INVALID_ID;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
-
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Out of FIFO space binding a screen target\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_BIND_GB_SCREENTARGET;
cmd->header.size = sizeof(cmd->body);
return -EINVAL;
}
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
-
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Out of FIFO space updating a Screen Target\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
vmw_stdu_populate_update(cmd, stdu->base.unit,
0, stdu->display_width,
if (unlikely(!stdu->defined))
return 0;
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
-
- if (unlikely(cmd == NULL)) {
- DRM_ERROR("Out of FIFO space, screen target not destroyed\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (unlikely(cmd == NULL))
return -ENOMEM;
- }
cmd->header.id = SVGA_3D_CMD_DESTROY_GB_SCREENTARGET;
cmd->header.size = sizeof(cmd->body);
vmw_fifo_commit(dirty->dev_priv, sizeof(*cmd) + blit_size);
+ stdu->display_srf->res.res_dirty = true;
ddirty->left = ddirty->top = S32_MAX;
ddirty->right = ddirty->bottom = S32_MIN;
}
region.x2 = diff.rect.x2;
region.y1 = diff.rect.y1;
region.y2 = diff.rect.y2;
- ret = vmw_kms_update_proxy(
- (struct vmw_resource *) &stdu->display_srf->res,
- (const struct drm_clip_rect *) ®ion, 1, 1);
+ ret = vmw_kms_update_proxy(&stdu->display_srf->res, ®ion,
+ 1, 1);
if (ret)
goto out_cleanup;
dev_priv = vmw_priv(stdu->base.crtc.dev);
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
-
- if (!cmd) {
- DRM_ERROR("Cannot reserve FIFO space to update STDU");
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
+ if (!cmd)
goto out_cleanup;
- }
vmw_stdu_populate_update(cmd, stdu->base.unit,
region.x1, region.x2,
cmd->body.dest.sid = stdu->display_srf->res.id;
update = (struct vmw_stdu_update *) &blit[dirty->num_hits];
commit_size = sizeof(*cmd) + blit_size + sizeof(*update);
+ stdu->display_srf->res.res_dirty = true;
} else {
update = dirty->cmd;
commit_size = sizeof(*update);
if (!srf)
srf = &vfbs->surface->res;
- ret = vmw_validation_add_resource(&val_ctx, srf, 0, NULL, NULL);
+ ret = vmw_validation_add_resource(&val_ctx, srf, 0, VMW_RES_DIRTY_NONE,
+ NULL, NULL);
if (ret)
return ret;
if (res->id != -1) {
- cmd = vmw_fifo_reserve(dev_priv, vmw_surface_destroy_size());
- if (unlikely(!cmd)) {
- DRM_ERROR("Failed reserving FIFO space for surface "
- "destruction.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, vmw_surface_destroy_size());
+ if (unlikely(!cmd))
return;
- }
vmw_surface_destroy_encode(res->id, cmd);
vmw_fifo_commit(dev_priv, vmw_surface_destroy_size());
*/
submit_size = vmw_surface_define_size(srf);
- cmd = vmw_fifo_reserve(dev_priv, submit_size);
+ cmd = VMW_FIFO_RESERVE(dev_priv, submit_size);
if (unlikely(!cmd)) {
- DRM_ERROR("Failed reserving FIFO space for surface "
- "creation.\n");
ret = -ENOMEM;
goto out_no_fifo;
}
BUG_ON(!val_buf->bo);
submit_size = vmw_surface_dma_size(srf);
- cmd = vmw_fifo_reserve(dev_priv, submit_size);
- if (unlikely(!cmd)) {
- DRM_ERROR("Failed reserving FIFO space for surface "
- "DMA.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, submit_size);
+ if (unlikely(!cmd))
return -ENOMEM;
- }
+
vmw_bo_get_guest_ptr(val_buf->bo, &ptr);
vmw_surface_dma_encode(srf, cmd, &ptr, bind);
*/
submit_size = vmw_surface_destroy_size();
- cmd = vmw_fifo_reserve(dev_priv, submit_size);
- if (unlikely(!cmd)) {
- DRM_ERROR("Failed reserving FIFO space for surface "
- "eviction.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, submit_size);
+ if (unlikely(!cmd))
return -ENOMEM;
- }
vmw_surface_destroy_encode(res->id, cmd);
vmw_fifo_commit(dev_priv, submit_size);
ttm_round_pot(num_sizes * sizeof(struct drm_vmw_size)) +
ttm_round_pot(num_sizes * sizeof(struct vmw_surface_offset));
-
desc = svga3dsurface_get_desc(req->format);
if (unlikely(desc->block_desc == SVGA3DBLOCKDESC_NONE)) {
- DRM_ERROR("Invalid surface format for surface creation.\n");
- DRM_ERROR("Format requested is: %d\n", req->format);
+ VMW_DEBUG_USER("Invalid format %d for surface creation.\n",
+ req->format);
return -EINVAL;
}
size, &ctx);
if (unlikely(ret != 0)) {
if (ret != -ERESTARTSYS)
- DRM_ERROR("Out of graphics memory for surface"
- " creation.\n");
+ DRM_ERROR("Out of graphics memory for surface.\n");
goto out_unlock;
}
ret = -EINVAL;
base = ttm_base_object_lookup_for_ref(dev_priv->tdev, handle);
if (unlikely(!base)) {
- DRM_ERROR("Could not find surface to reference.\n");
+ VMW_DEBUG_USER("Could not find surface to reference.\n");
goto out_no_lookup;
}
if (unlikely(ttm_base_object_type(base) != VMW_RES_SURFACE)) {
- DRM_ERROR("Referenced object is not a surface.\n");
+ VMW_DEBUG_USER("Referenced object is not a surface.\n");
goto out_bad_resource;
}
ret = copy_to_user(user_sizes, &srf->base_size,
sizeof(srf->base_size));
if (unlikely(ret != 0)) {
- DRM_ERROR("copy_to_user failed %p %u\n",
- user_sizes, srf->num_sizes);
+ VMW_DEBUG_USER("copy_to_user failed %p %u\n", user_sizes,
+ srf->num_sizes);
ttm_ref_object_base_unref(tfile, base->handle, TTM_REF_USAGE);
ret = -EFAULT;
}
submit_len = sizeof(*cmd);
}
- cmd = vmw_fifo_reserve(dev_priv, submit_len);
+ cmd = VMW_FIFO_RESERVE(dev_priv, submit_len);
cmd2 = (typeof(cmd2))cmd;
cmd3 = (typeof(cmd3))cmd;
if (unlikely(!cmd)) {
- DRM_ERROR("Failed reserving FIFO space for surface "
- "creation.\n");
ret = -ENOMEM;
goto out_no_fifo;
}
submit_size = sizeof(*cmd1) + (res->backup_dirty ? sizeof(*cmd2) : 0);
- cmd1 = vmw_fifo_reserve(dev_priv, submit_size);
- if (unlikely(!cmd1)) {
- DRM_ERROR("Failed reserving FIFO space for surface "
- "binding.\n");
+ cmd1 = VMW_FIFO_RESERVE(dev_priv, submit_size);
+ if (unlikely(!cmd1))
return -ENOMEM;
- }
cmd1->header.id = SVGA_3D_CMD_BIND_GB_SURFACE;
cmd1->header.size = sizeof(cmd1->body);
BUG_ON(bo->mem.mem_type != VMW_PL_MOB);
submit_size = sizeof(*cmd3) + (readback ? sizeof(*cmd1) : sizeof(*cmd2));
- cmd = vmw_fifo_reserve(dev_priv, submit_size);
- if (unlikely(!cmd)) {
- DRM_ERROR("Failed reserving FIFO space for surface "
- "unbinding.\n");
+ cmd = VMW_FIFO_RESERVE(dev_priv, submit_size);
+ if (unlikely(!cmd))
return -ENOMEM;
- }
if (readback) {
cmd1 = (void *) cmd;
vmw_view_surface_list_destroy(dev_priv, &srf->view_list);
vmw_binding_res_list_scrub(&res->binding_head);
- cmd = vmw_fifo_reserve(dev_priv, sizeof(*cmd));
+ cmd = VMW_FIFO_RESERVE(dev_priv, sizeof(*cmd));
if (unlikely(!cmd)) {
- DRM_ERROR("Failed reserving FIFO space for surface "
- "destruction.\n");
mutex_unlock(&dev_priv->binding_mutex);
return -ENOMEM;
}
if (for_scanout) {
if (!svga3dsurface_is_screen_target_format(format)) {
- DRM_ERROR("Invalid Screen Target surface format.");
+ VMW_DEBUG_USER("Invalid Screen Target surface format.");
return -EINVAL;
}
if (size.width > dev_priv->texture_max_width ||
size.height > dev_priv->texture_max_height) {
- DRM_ERROR("%ux%u\n, exceeds max surface size %ux%u",
- size.width, size.height,
- dev_priv->texture_max_width,
- dev_priv->texture_max_height);
+ VMW_DEBUG_USER("%ux%u\n, exceeds max surface size %ux%u",
+ size.width, size.height,
+ dev_priv->texture_max_width,
+ dev_priv->texture_max_height);
return -EINVAL;
}
} else {
desc = svga3dsurface_get_desc(format);
if (unlikely(desc->block_desc == SVGA3DBLOCKDESC_NONE)) {
- DRM_ERROR("Invalid surface format.\n");
+ VMW_DEBUG_USER("Invalid surface format.\n");
return -EINVAL;
}
}
/* array_size must be null for non-GL3 host. */
if (array_size > 0 && !dev_priv->has_dx) {
- DRM_ERROR("Tried to create DX surface on non-DX host.\n");
+ VMW_DEBUG_USER("Tried to create DX surface on non-DX host.\n");
return -EINVAL;
}
if (ret == 0) {
if (res->backup->base.num_pages * PAGE_SIZE <
res->backup_size) {
- DRM_ERROR("Surface backup buffer too small.\n");
+ VMW_DEBUG_USER("Surface backup buffer too small.\n");
vmw_bo_unreference(&res->backup);
ret = -EINVAL;
goto out_unlock;
int vmw_mmap(struct file *filp, struct vm_area_struct *vma)
{
- struct drm_file *file_priv;
- struct vmw_private *dev_priv;
+ struct drm_file *file_priv = filp->private_data;
+ struct vmw_private *dev_priv = vmw_priv(file_priv->minor->dev);
- if (unlikely(vma->vm_pgoff < VMWGFX_FILE_PAGE_OFFSET)) {
- DRM_ERROR("Illegal attempt to mmap old fifo space.\n");
- return -EINVAL;
- }
-
- file_priv = filp->private_data;
- dev_priv = vmw_priv(file_priv->minor->dev);
return ttm_bo_mmap(filp, vma, &dev_priv->bdev);
}
u32 switching_backup : 1;
u32 first_usage : 1;
u32 reserved : 1;
+ u32 dirty : 1;
+ u32 dirty_set : 1;
unsigned long private[0];
};
* @ctx: The validation context.
* @res: The resource.
* @priv_size: Size of private, additional metadata.
+ * @dirty: Whether to change dirty status.
* @p_node: Output pointer of additional metadata address.
* @first_usage: Whether this was the first time this resource was seen.
*
int vmw_validation_add_resource(struct vmw_validation_context *ctx,
struct vmw_resource *res,
size_t priv_size,
+ u32 dirty,
void **p_node,
bool *first_usage)
{
node = vmw_validation_mem_alloc(ctx, sizeof(*node) + priv_size);
if (!node) {
- DRM_ERROR("Failed to allocate a resource validation "
- "entry.\n");
+ VMW_DEBUG_USER("Failed to allocate a resource validation entry.\n");
return -ENOMEM;
}
}
out_fill:
+ if (dirty) {
+ node->dirty_set = 1;
+ /* Overwriting previous information here is intentional! */
+ node->dirty = (dirty & VMW_RES_DIRTY_SET) ? 1 : 0;
+ }
if (first_usage)
*first_usage = node->first_usage;
if (p_node)
return 0;
}
+/**
+ * vmw_validation_res_set_dirty - Register a resource dirty set or clear during
+ * validation.
+ * @ctx: The validation context.
+ * @val_private: The additional meta-data pointer returned when the
+ * resource was registered with the validation context. Used to identify
+ * the resource.
+ * @dirty: Dirty information VMW_RES_DIRTY_XX
+ */
+void vmw_validation_res_set_dirty(struct vmw_validation_context *ctx,
+ void *val_private, u32 dirty)
+{
+ struct vmw_validation_res_node *val;
+
+ if (!dirty)
+ return;
+
+ val = container_of(val_private, typeof(*val), private);
+ val->dirty_set = 1;
+ /* Overwriting previous information here is intentional! */
+ val->dirty = (dirty & VMW_RES_DIRTY_SET) ? 1 : 0;
+}
+
/**
* vmw_validation_res_switch_backup - Register a backup MOB switch during
* validation.
struct vmw_validation_res_node *val;
list_splice_init(&ctx->resource_ctx_list, &ctx->resource_list);
-
- list_for_each_entry(val, &ctx->resource_list, head) {
- if (val->reserved)
- vmw_resource_unreserve(val->res,
- !backoff &&
- val->switching_backup,
- val->new_backup,
- val->new_backup_offset);
- }
+ if (backoff)
+ list_for_each_entry(val, &ctx->resource_list, head) {
+ if (val->reserved)
+ vmw_resource_unreserve(val->res,
+ false, false, false,
+ NULL, 0);
+ }
+ else
+ list_for_each_entry(val, &ctx->resource_list, head) {
+ if (val->reserved)
+ vmw_resource_unreserve(val->res,
+ val->dirty_set,
+ val->dirty,
+ val->switching_backup,
+ val->new_backup,
+ val->new_backup_offset);
+ }
}
/**
#include <linux/ww_mutex.h>
#include <drm/ttm/ttm_execbuf_util.h>
+#define VMW_RES_DIRTY_NONE 0
+#define VMW_RES_DIRTY_SET BIT(0)
+#define VMW_RES_DIRTY_CLEAR BIT(1)
+
/**
* struct vmw_validation_mem - Custom interface to provide memory reservations
* for the validation code.
int vmw_validation_add_resource(struct vmw_validation_context *ctx,
struct vmw_resource *res,
size_t priv_size,
+ u32 dirty,
void **p_node,
bool *first_usage);
void vmw_validation_drop_ht(struct vmw_validation_context *ctx);
int vmw_validation_preload_bo(struct vmw_validation_context *ctx);
int vmw_validation_preload_res(struct vmw_validation_context *ctx,
unsigned int size);
+void vmw_validation_res_set_dirty(struct vmw_validation_context *ctx,
+ void *val_private, u32 dirty);
#endif
drm_kms_helper_poll_fini(dev);
drm_dev_unplug(dev);
+ drm_dev_put(dev);
front_info->drm_info = NULL;
return ret;
}
+/*
+ * We print a warning when we are not flagged to support atomic transfers but
+ * will try anyhow. That's what the I2C core would do as well. Sadly, we can't
+ * modify the algorithm struct at probe time because this struct is exported
+ * 'const'.
+ */
+static int bit_xfer_atomic(struct i2c_adapter *i2c_adap, struct i2c_msg msgs[],
+ int num)
+{
+ struct i2c_algo_bit_data *adap = i2c_adap->algo_data;
+
+ if (!adap->can_do_atomic)
+ dev_warn(&i2c_adap->dev, "not flagged for atomic transfers\n");
+
+ return bit_xfer(i2c_adap, msgs, num);
+}
+
static u32 bit_func(struct i2c_adapter *adap)
{
return I2C_FUNC_I2C | I2C_FUNC_NOSTART | I2C_FUNC_SMBUS_EMUL |
/* -----exported algorithm data: ------------------------------------- */
const struct i2c_algorithm i2c_bit_algo = {
- .master_xfer = bit_xfer,
- .functionality = bit_func,
+ .master_xfer = bit_xfer,
+ .master_xfer_atomic = bit_xfer_atomic,
+ .functionality = bit_func,
};
EXPORT_SYMBOL(i2c_bit_algo);
This driver can also be built as a module. If so, the module
will be called i2c-amd8111.
+config I2C_AMD_MP2
+ tristate "AMD MP2 PCIe"
+ depends on PCI && ACPI
+ help
+ If you say yes to this option, support will be included for the AMD
+ MP2 PCIe I2C adapter.
+
+ This driver can also be built as modules. If so, the modules will
+ be called i2c-amd-mp2-pci and i2c-amd-mp2-plat.
+
config I2C_HIX5HD2
tristate "Hix5hd2 high-speed I2C driver"
depends on ARCH_HISI || ARCH_HIX5HD2 || COMPILE_TEST
AMD Hudson-2
AMD ML
AMD CZ
+ Hygon CZ
Serverworks OSB4
Serverworks CSB5
Serverworks CSB6
the latency to fill the transmission register is too long. If you
are facing this situation, use the i2c-gpio driver.
+config I2C_AT91_SLAVE_EXPERIMENTAL
+ tristate "Microchip AT91 I2C experimental slave mode"
+ depends on I2C_AT91
+ select I2C_SLAVE
+ help
+ If you say yes to this option, support for the slave mode will be
+ added. Caution: do not use it for production. This feature has not
+ been tested in a heavy way, help wanted.
+ There are known bugs:
+ - It can hang, on a SAMA5D4, after several transfers.
+ - There are some mismtaches with a SAMA5D4 as slave and a SAMA5D2 as
+ master.
+
config I2C_AU1550
tristate "Au1550/Au1200/Au1300 SMBus interface"
depends on MIPS_ALCHEMY
tristate "Broadcom iProc I2C controller"
depends on ARCH_BCM_IPROC || COMPILE_TEST
default ARCH_BCM_IPROC
+ select I2C_SLAVE
help
If you say yes to this option, support will be included for the
Broadcom iProc I2C controller.
# Embedded system I2C/SMBus host controller drivers
obj-$(CONFIG_I2C_ALTERA) += i2c-altera.o
+obj-$(CONFIG_I2C_AMD_MP2) += i2c-amd-mp2-pci.o i2c-amd-mp2-plat.o
obj-$(CONFIG_I2C_ASPEED) += i2c-aspeed.o
obj-$(CONFIG_I2C_AT91) += i2c-at91.o
+i2c-at91-objs := i2c-at91-core.o i2c-at91-master.o
+ifeq ($(CONFIG_I2C_AT91_SLAVE_EXPERIMENTAL),y)
+ i2c-at91-objs += i2c-at91-slave.o
+endif
obj-$(CONFIG_I2C_AU1550) += i2c-au1550.o
obj-$(CONFIG_I2C_AXXIA) += i2c-axxia.o
obj-$(CONFIG_I2C_BCM2835) += i2c-bcm2835.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * AMD MP2 PCIe communication driver
+ *
+ * Authors: Shyam Sundar S K <Shyam-sundar.S-k@amd.com>
+ * Elie Morisse <syniurge@gmail.com>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/slab.h>
+
+#include "i2c-amd-mp2.h"
+
+#include <linux/io-64-nonatomic-lo-hi.h>
+
+static void amd_mp2_c2p_mutex_lock(struct amd_i2c_common *i2c_common)
+{
+ struct amd_mp2_dev *privdata = i2c_common->mp2_dev;
+
+ /* there is only one data mailbox for two i2c adapters */
+ mutex_lock(&privdata->c2p_lock);
+ privdata->c2p_lock_busid = i2c_common->bus_id;
+}
+
+static void amd_mp2_c2p_mutex_unlock(struct amd_i2c_common *i2c_common)
+{
+ struct amd_mp2_dev *privdata = i2c_common->mp2_dev;
+
+ if (unlikely(privdata->c2p_lock_busid != i2c_common->bus_id)) {
+ dev_warn(ndev_dev(privdata),
+ "bus %d attempting to unlock C2P locked by bus %d\n",
+ i2c_common->bus_id, privdata->c2p_lock_busid);
+ return;
+ }
+
+ mutex_unlock(&privdata->c2p_lock);
+}
+
+static int amd_mp2_cmd(struct amd_i2c_common *i2c_common,
+ union i2c_cmd_base i2c_cmd_base)
+{
+ struct amd_mp2_dev *privdata = i2c_common->mp2_dev;
+ void __iomem *reg;
+
+ i2c_common->reqcmd = i2c_cmd_base.s.i2c_cmd;
+
+ reg = privdata->mmio + ((i2c_cmd_base.s.bus_id == 1) ?
+ AMD_C2P_MSG1 : AMD_C2P_MSG0);
+ writel(i2c_cmd_base.ul, reg);
+
+ return 0;
+}
+
+int amd_mp2_bus_enable_set(struct amd_i2c_common *i2c_common, bool enable)
+{
+ struct amd_mp2_dev *privdata = i2c_common->mp2_dev;
+ union i2c_cmd_base i2c_cmd_base;
+
+ dev_dbg(ndev_dev(privdata), "%s id: %d\n", __func__,
+ i2c_common->bus_id);
+
+ i2c_cmd_base.ul = 0;
+ i2c_cmd_base.s.i2c_cmd = enable ? i2c_enable : i2c_disable;
+ i2c_cmd_base.s.bus_id = i2c_common->bus_id;
+ i2c_cmd_base.s.i2c_speed = i2c_common->i2c_speed;
+
+ amd_mp2_c2p_mutex_lock(i2c_common);
+
+ return amd_mp2_cmd(i2c_common, i2c_cmd_base);
+}
+EXPORT_SYMBOL_GPL(amd_mp2_bus_enable_set);
+
+static void amd_mp2_cmd_rw_fill(struct amd_i2c_common *i2c_common,
+ union i2c_cmd_base *i2c_cmd_base,
+ enum i2c_cmd reqcmd)
+{
+ i2c_cmd_base->s.i2c_cmd = reqcmd;
+ i2c_cmd_base->s.bus_id = i2c_common->bus_id;
+ i2c_cmd_base->s.i2c_speed = i2c_common->i2c_speed;
+ i2c_cmd_base->s.slave_addr = i2c_common->msg->addr;
+ i2c_cmd_base->s.length = i2c_common->msg->len;
+}
+
+int amd_mp2_rw(struct amd_i2c_common *i2c_common, enum i2c_cmd reqcmd)
+{
+ struct amd_mp2_dev *privdata = i2c_common->mp2_dev;
+ union i2c_cmd_base i2c_cmd_base;
+
+ amd_mp2_cmd_rw_fill(i2c_common, &i2c_cmd_base, reqcmd);
+ amd_mp2_c2p_mutex_lock(i2c_common);
+
+ if (i2c_common->msg->len <= 32) {
+ i2c_cmd_base.s.mem_type = use_c2pmsg;
+ if (reqcmd == i2c_write)
+ memcpy_toio(privdata->mmio + AMD_C2P_MSG2,
+ i2c_common->msg->buf,
+ i2c_common->msg->len);
+ } else {
+ i2c_cmd_base.s.mem_type = use_dram;
+ writeq((u64)i2c_common->dma_addr,
+ privdata->mmio + AMD_C2P_MSG2);
+ }
+
+ return amd_mp2_cmd(i2c_common, i2c_cmd_base);
+}
+EXPORT_SYMBOL_GPL(amd_mp2_rw);
+
+static void amd_mp2_pci_check_rw_event(struct amd_i2c_common *i2c_common)
+{
+ struct amd_mp2_dev *privdata = i2c_common->mp2_dev;
+ int len = i2c_common->eventval.r.length;
+ u32 slave_addr = i2c_common->eventval.r.slave_addr;
+ bool err = false;
+
+ if (unlikely(len != i2c_common->msg->len)) {
+ dev_err(ndev_dev(privdata),
+ "length %d in event doesn't match buffer length %d!\n",
+ len, i2c_common->msg->len);
+ err = true;
+ }
+
+ if (unlikely(slave_addr != i2c_common->msg->addr)) {
+ dev_err(ndev_dev(privdata),
+ "unexpected slave address %x (expected: %x)!\n",
+ slave_addr, i2c_common->msg->addr);
+ err = true;
+ }
+
+ if (!err)
+ i2c_common->cmd_success = true;
+}
+
+static void __amd_mp2_process_event(struct amd_i2c_common *i2c_common)
+{
+ struct amd_mp2_dev *privdata = i2c_common->mp2_dev;
+ enum status_type sts = i2c_common->eventval.r.status;
+ enum response_type res = i2c_common->eventval.r.response;
+ int len = i2c_common->eventval.r.length;
+
+ if (res != command_success) {
+ if (res != command_failed)
+ dev_err(ndev_dev(privdata), "invalid response to i2c command!\n");
+ return;
+ }
+
+ switch (i2c_common->reqcmd) {
+ case i2c_read:
+ if (sts == i2c_readcomplete_event) {
+ amd_mp2_pci_check_rw_event(i2c_common);
+ if (len <= 32)
+ memcpy_fromio(i2c_common->msg->buf,
+ privdata->mmio + AMD_C2P_MSG2,
+ len);
+ } else if (sts != i2c_readfail_event) {
+ dev_err(ndev_dev(privdata),
+ "invalid i2c status after read (%d)!\n", sts);
+ }
+ break;
+ case i2c_write:
+ if (sts == i2c_writecomplete_event)
+ amd_mp2_pci_check_rw_event(i2c_common);
+ else if (sts != i2c_writefail_event)
+ dev_err(ndev_dev(privdata),
+ "invalid i2c status after write (%d)!\n", sts);
+ break;
+ case i2c_enable:
+ if (sts == i2c_busenable_complete)
+ i2c_common->cmd_success = true;
+ else if (sts != i2c_busenable_failed)
+ dev_err(ndev_dev(privdata),
+ "invalid i2c status after bus enable (%d)!\n",
+ sts);
+ break;
+ case i2c_disable:
+ if (sts == i2c_busdisable_complete)
+ i2c_common->cmd_success = true;
+ else if (sts != i2c_busdisable_failed)
+ dev_err(ndev_dev(privdata),
+ "invalid i2c status after bus disable (%d)!\n",
+ sts);
+ break;
+ default:
+ break;
+ }
+}
+
+void amd_mp2_process_event(struct amd_i2c_common *i2c_common)
+{
+ struct amd_mp2_dev *privdata = i2c_common->mp2_dev;
+
+ if (unlikely(i2c_common->reqcmd == i2c_none)) {
+ dev_warn(ndev_dev(privdata),
+ "received msg but no cmd was sent (bus = %d)!\n",
+ i2c_common->bus_id);
+ return;
+ }
+
+ __amd_mp2_process_event(i2c_common);
+
+ i2c_common->reqcmd = i2c_none;
+ amd_mp2_c2p_mutex_unlock(i2c_common);
+}
+EXPORT_SYMBOL_GPL(amd_mp2_process_event);
+
+static irqreturn_t amd_mp2_irq_isr(int irq, void *dev)
+{
+ struct amd_mp2_dev *privdata = dev;
+ struct amd_i2c_common *i2c_common;
+ u32 val;
+ unsigned int bus_id;
+ void __iomem *reg;
+ enum irqreturn ret = IRQ_NONE;
+
+ for (bus_id = 0; bus_id < 2; bus_id++) {
+ i2c_common = privdata->busses[bus_id];
+ if (!i2c_common)
+ continue;
+
+ reg = privdata->mmio + ((bus_id == 0) ?
+ AMD_P2C_MSG1 : AMD_P2C_MSG2);
+ val = readl(reg);
+ if (val != 0) {
+ writel(0, reg);
+ writel(0, privdata->mmio + AMD_P2C_MSG_INTEN);
+ i2c_common->eventval.ul = val;
+ i2c_common->cmd_completion(i2c_common);
+
+ ret = IRQ_HANDLED;
+ }
+ }
+
+ if (ret != IRQ_HANDLED) {
+ val = readl(privdata->mmio + AMD_P2C_MSG_INTEN);
+ if (val != 0) {
+ writel(0, privdata->mmio + AMD_P2C_MSG_INTEN);
+ dev_warn(ndev_dev(privdata),
+ "received irq without message\n");
+ ret = IRQ_HANDLED;
+ }
+ }
+
+ return ret;
+}
+
+void amd_mp2_rw_timeout(struct amd_i2c_common *i2c_common)
+{
+ i2c_common->reqcmd = i2c_none;
+ amd_mp2_c2p_mutex_unlock(i2c_common);
+}
+EXPORT_SYMBOL_GPL(amd_mp2_rw_timeout);
+
+int amd_mp2_register_cb(struct amd_i2c_common *i2c_common)
+{
+ struct amd_mp2_dev *privdata = i2c_common->mp2_dev;
+
+ if (i2c_common->bus_id > 1)
+ return -EINVAL;
+
+ if (privdata->busses[i2c_common->bus_id]) {
+ dev_err(ndev_dev(privdata),
+ "Bus %d already taken!\n", i2c_common->bus_id);
+ return -EINVAL;
+ }
+
+ privdata->busses[i2c_common->bus_id] = i2c_common;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(amd_mp2_register_cb);
+
+int amd_mp2_unregister_cb(struct amd_i2c_common *i2c_common)
+{
+ struct amd_mp2_dev *privdata = i2c_common->mp2_dev;
+
+ privdata->busses[i2c_common->bus_id] = NULL;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(amd_mp2_unregister_cb);
+
+static void amd_mp2_clear_reg(struct amd_mp2_dev *privdata)
+{
+ int reg;
+
+ for (reg = AMD_C2P_MSG0; reg <= AMD_C2P_MSG9; reg += 4)
+ writel(0, privdata->mmio + reg);
+
+ for (reg = AMD_P2C_MSG1; reg <= AMD_P2C_MSG2; reg += 4)
+ writel(0, privdata->mmio + reg);
+}
+
+static int amd_mp2_pci_init(struct amd_mp2_dev *privdata,
+ struct pci_dev *pci_dev)
+{
+ int rc;
+
+ pci_set_drvdata(pci_dev, privdata);
+
+ rc = pcim_enable_device(pci_dev);
+ if (rc) {
+ dev_err(ndev_dev(privdata), "Failed to enable MP2 PCI device\n");
+ goto err_pci_enable;
+ }
+
+ rc = pcim_iomap_regions(pci_dev, 1 << 2, pci_name(pci_dev));
+ if (rc) {
+ dev_err(ndev_dev(privdata), "I/O memory remapping failed\n");
+ goto err_pci_enable;
+ }
+ privdata->mmio = pcim_iomap_table(pci_dev)[2];
+
+ pci_set_master(pci_dev);
+
+ rc = pci_set_dma_mask(pci_dev, DMA_BIT_MASK(64));
+ if (rc) {
+ rc = pci_set_dma_mask(pci_dev, DMA_BIT_MASK(32));
+ if (rc)
+ goto err_dma_mask;
+ }
+
+ /* Set up intx irq */
+ writel(0, privdata->mmio + AMD_P2C_MSG_INTEN);
+ pci_intx(pci_dev, 1);
+ rc = devm_request_irq(&pci_dev->dev, pci_dev->irq, amd_mp2_irq_isr,
+ IRQF_SHARED, dev_name(&pci_dev->dev), privdata);
+ if (rc)
+ dev_err(&pci_dev->dev, "Failure requesting irq %i: %d\n",
+ pci_dev->irq, rc);
+
+ return rc;
+
+err_dma_mask:
+ pci_clear_master(pci_dev);
+err_pci_enable:
+ pci_set_drvdata(pci_dev, NULL);
+ return rc;
+}
+
+static int amd_mp2_pci_probe(struct pci_dev *pci_dev,
+ const struct pci_device_id *id)
+{
+ struct amd_mp2_dev *privdata;
+ int rc;
+
+ privdata = devm_kzalloc(&pci_dev->dev, sizeof(*privdata), GFP_KERNEL);
+ if (!privdata)
+ return -ENOMEM;
+
+ rc = amd_mp2_pci_init(privdata, pci_dev);
+ if (rc)
+ return rc;
+
+ mutex_init(&privdata->c2p_lock);
+ privdata->pci_dev = pci_dev;
+
+ pm_runtime_set_autosuspend_delay(&pci_dev->dev, 1000);
+ pm_runtime_use_autosuspend(&pci_dev->dev);
+ pm_runtime_put_autosuspend(&pci_dev->dev);
+ pm_runtime_allow(&pci_dev->dev);
+
+ privdata->probed = true;
+
+ dev_info(&pci_dev->dev, "MP2 device registered.\n");
+ return 0;
+}
+
+static void amd_mp2_pci_remove(struct pci_dev *pci_dev)
+{
+ struct amd_mp2_dev *privdata = pci_get_drvdata(pci_dev);
+
+ pm_runtime_forbid(&pci_dev->dev);
+ pm_runtime_get_noresume(&pci_dev->dev);
+
+ pci_intx(pci_dev, 0);
+ pci_clear_master(pci_dev);
+
+ amd_mp2_clear_reg(privdata);
+}
+
+#ifdef CONFIG_PM
+static int amd_mp2_pci_suspend(struct device *dev)
+{
+ struct pci_dev *pci_dev = to_pci_dev(dev);
+ struct amd_mp2_dev *privdata = pci_get_drvdata(pci_dev);
+ struct amd_i2c_common *i2c_common;
+ unsigned int bus_id;
+ int ret = 0;
+
+ for (bus_id = 0; bus_id < 2; bus_id++) {
+ i2c_common = privdata->busses[bus_id];
+ if (i2c_common)
+ i2c_common->suspend(i2c_common);
+ }
+
+ ret = pci_save_state(pci_dev);
+ if (ret) {
+ dev_err(ndev_dev(privdata),
+ "pci_save_state failed = %d\n", ret);
+ return ret;
+ }
+
+ pci_disable_device(pci_dev);
+ return ret;
+}
+
+static int amd_mp2_pci_resume(struct device *dev)
+{
+ struct pci_dev *pci_dev = to_pci_dev(dev);
+ struct amd_mp2_dev *privdata = pci_get_drvdata(pci_dev);
+ struct amd_i2c_common *i2c_common;
+ unsigned int bus_id;
+ int ret = 0;
+
+ pci_restore_state(pci_dev);
+ ret = pci_enable_device(pci_dev);
+ if (ret < 0) {
+ dev_err(ndev_dev(privdata),
+ "pci_enable_device failed = %d\n", ret);
+ return ret;
+ }
+
+ for (bus_id = 0; bus_id < 2; bus_id++) {
+ i2c_common = privdata->busses[bus_id];
+ if (i2c_common) {
+ ret = i2c_common->resume(i2c_common);
+ if (ret < 0)
+ return ret;
+ }
+ }
+
+ return ret;
+}
+
+static UNIVERSAL_DEV_PM_OPS(amd_mp2_pci_pm_ops, amd_mp2_pci_suspend,
+ amd_mp2_pci_resume, NULL);
+#endif /* CONFIG_PM */
+
+static const struct pci_device_id amd_mp2_pci_tbl[] = {
+ {PCI_VDEVICE(AMD, PCI_DEVICE_ID_AMD_MP2)},
+ {0}
+};
+MODULE_DEVICE_TABLE(pci, amd_mp2_pci_tbl);
+
+static struct pci_driver amd_mp2_pci_driver = {
+ .name = "i2c_amd_mp2",
+ .id_table = amd_mp2_pci_tbl,
+ .probe = amd_mp2_pci_probe,
+ .remove = amd_mp2_pci_remove,
+#ifdef CONFIG_PM
+ .driver = {
+ .pm = &amd_mp2_pci_pm_ops,
+ },
+#endif
+};
+module_pci_driver(amd_mp2_pci_driver);
+
+static int amd_mp2_device_match(struct device *dev, void *data)
+{
+ return 1;
+}
+
+struct amd_mp2_dev *amd_mp2_find_device(void)
+{
+ struct device *dev;
+ struct pci_dev *pci_dev;
+
+ dev = driver_find_device(&amd_mp2_pci_driver.driver, NULL, NULL,
+ amd_mp2_device_match);
+ if (!dev)
+ return NULL;
+
+ pci_dev = to_pci_dev(dev);
+ return (struct amd_mp2_dev *)pci_get_drvdata(pci_dev);
+}
+EXPORT_SYMBOL_GPL(amd_mp2_find_device);
+
+MODULE_DESCRIPTION("AMD(R) PCI-E MP2 I2C Controller Driver");
+MODULE_AUTHOR("Shyam Sundar S K <Shyam-sundar.S-k@amd.com>");
+MODULE_AUTHOR("Elie Morisse <syniurge@gmail.com>");
+MODULE_LICENSE("Dual BSD/GPL");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * AMD MP2 platform driver
+ *
+ * Setup the I2C adapters enumerated in the ACPI namespace.
+ * MP2 controllers have 2 separate busses, up to 2 I2C adapters may be listed.
+ *
+ * Authors: Nehal Bakulchandra Shah <Nehal-bakulchandra.shah@amd.com>
+ * Elie Morisse <syniurge@gmail.com>
+ */
+
+#include <linux/acpi.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+
+#include "i2c-amd-mp2.h"
+
+#define AMD_MP2_I2C_MAX_RW_LENGTH ((1 << 12) - 1)
+#define AMD_I2C_TIMEOUT (msecs_to_jiffies(250))
+
+/**
+ * struct amd_i2c_dev - MP2 bus/i2c adapter context
+ * @common: shared context with the MP2 PCI driver
+ * @pdev: platform driver node
+ * @adap: i2c adapter
+ * @cmd_complete: xfer completion object
+ */
+struct amd_i2c_dev {
+ struct amd_i2c_common common;
+ struct platform_device *pdev;
+ struct i2c_adapter adap;
+ struct completion cmd_complete;
+};
+
+#define amd_i2c_dev_common(__common) \
+ container_of(__common, struct amd_i2c_dev, common)
+
+static int i2c_amd_dma_map(struct amd_i2c_common *i2c_common)
+{
+ struct device *dev_pci = &i2c_common->mp2_dev->pci_dev->dev;
+ struct amd_i2c_dev *i2c_dev = amd_i2c_dev_common(i2c_common);
+ enum dma_data_direction dma_direction =
+ i2c_common->msg->flags & I2C_M_RD ?
+ DMA_FROM_DEVICE : DMA_TO_DEVICE;
+
+ i2c_common->dma_buf = i2c_get_dma_safe_msg_buf(i2c_common->msg, 0);
+ i2c_common->dma_addr = dma_map_single(dev_pci, i2c_common->dma_buf,
+ i2c_common->msg->len,
+ dma_direction);
+
+ if (unlikely(dma_mapping_error(dev_pci, i2c_common->dma_addr))) {
+ dev_err(&i2c_dev->pdev->dev,
+ "Error while mapping dma buffer %p\n",
+ i2c_common->dma_buf);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static void i2c_amd_dma_unmap(struct amd_i2c_common *i2c_common)
+{
+ struct device *dev_pci = &i2c_common->mp2_dev->pci_dev->dev;
+ enum dma_data_direction dma_direction =
+ i2c_common->msg->flags & I2C_M_RD ?
+ DMA_FROM_DEVICE : DMA_TO_DEVICE;
+
+ dma_unmap_single(dev_pci, i2c_common->dma_addr,
+ i2c_common->msg->len, dma_direction);
+
+ i2c_put_dma_safe_msg_buf(i2c_common->dma_buf, i2c_common->msg, true);
+}
+
+static void i2c_amd_start_cmd(struct amd_i2c_dev *i2c_dev)
+{
+ struct amd_i2c_common *i2c_common = &i2c_dev->common;
+
+ reinit_completion(&i2c_dev->cmd_complete);
+ i2c_common->cmd_success = false;
+}
+
+static void i2c_amd_cmd_completion(struct amd_i2c_common *i2c_common)
+{
+ struct amd_i2c_dev *i2c_dev = amd_i2c_dev_common(i2c_common);
+ union i2c_event *event = &i2c_common->eventval;
+
+ if (event->r.status == i2c_readcomplete_event)
+ dev_dbg(&i2c_dev->pdev->dev, "%s readdata:%*ph\n",
+ __func__, event->r.length,
+ i2c_common->msg->buf);
+
+ complete(&i2c_dev->cmd_complete);
+}
+
+static int i2c_amd_check_cmd_completion(struct amd_i2c_dev *i2c_dev)
+{
+ struct amd_i2c_common *i2c_common = &i2c_dev->common;
+ unsigned long timeout;
+
+ timeout = wait_for_completion_timeout(&i2c_dev->cmd_complete,
+ i2c_dev->adap.timeout);
+
+ if ((i2c_common->reqcmd == i2c_read ||
+ i2c_common->reqcmd == i2c_write) &&
+ i2c_common->msg->len > 32)
+ i2c_amd_dma_unmap(i2c_common);
+
+ if (timeout == 0) {
+ amd_mp2_rw_timeout(i2c_common);
+ return -ETIMEDOUT;
+ }
+
+ amd_mp2_process_event(i2c_common);
+
+ if (!i2c_common->cmd_success)
+ return -EIO;
+
+ return 0;
+}
+
+static int i2c_amd_enable_set(struct amd_i2c_dev *i2c_dev, bool enable)
+{
+ struct amd_i2c_common *i2c_common = &i2c_dev->common;
+
+ i2c_amd_start_cmd(i2c_dev);
+ amd_mp2_bus_enable_set(i2c_common, enable);
+
+ return i2c_amd_check_cmd_completion(i2c_dev);
+}
+
+static int i2c_amd_xfer_msg(struct amd_i2c_dev *i2c_dev, struct i2c_msg *pmsg)
+{
+ struct amd_i2c_common *i2c_common = &i2c_dev->common;
+
+ i2c_amd_start_cmd(i2c_dev);
+ i2c_common->msg = pmsg;
+
+ if (pmsg->len > 32)
+ if (i2c_amd_dma_map(i2c_common))
+ return -EIO;
+
+ if (pmsg->flags & I2C_M_RD)
+ amd_mp2_rw(i2c_common, i2c_read);
+ else
+ amd_mp2_rw(i2c_common, i2c_write);
+
+ return i2c_amd_check_cmd_completion(i2c_dev);
+}
+
+static int i2c_amd_xfer(struct i2c_adapter *adap, struct i2c_msg *msgs, int num)
+{
+ struct amd_i2c_dev *i2c_dev = i2c_get_adapdata(adap);
+ int i;
+ struct i2c_msg *pmsg;
+ int err;
+
+ /* the adapter might have been deleted while waiting for the bus lock */
+ if (unlikely(!i2c_dev->common.mp2_dev))
+ return -EINVAL;
+
+ amd_mp2_pm_runtime_get(i2c_dev->common.mp2_dev);
+
+ for (i = 0; i < num; i++) {
+ pmsg = &msgs[i];
+ err = i2c_amd_xfer_msg(i2c_dev, pmsg);
+ if (err)
+ break;
+ }
+
+ amd_mp2_pm_runtime_put(i2c_dev->common.mp2_dev);
+ return err ? err : num;
+}
+
+static u32 i2c_amd_func(struct i2c_adapter *a)
+{
+ return I2C_FUNC_I2C | I2C_FUNC_SMBUS_EMUL;
+}
+
+static const struct i2c_algorithm i2c_amd_algorithm = {
+ .master_xfer = i2c_amd_xfer,
+ .functionality = i2c_amd_func,
+};
+
+#ifdef CONFIG_PM
+static int i2c_amd_suspend(struct amd_i2c_common *i2c_common)
+{
+ struct amd_i2c_dev *i2c_dev = amd_i2c_dev_common(i2c_common);
+
+ i2c_amd_enable_set(i2c_dev, false);
+ return 0;
+}
+
+static int i2c_amd_resume(struct amd_i2c_common *i2c_common)
+{
+ struct amd_i2c_dev *i2c_dev = amd_i2c_dev_common(i2c_common);
+
+ return i2c_amd_enable_set(i2c_dev, true);
+}
+#endif
+
+static enum speed_enum i2c_amd_get_bus_speed(struct platform_device *pdev)
+{
+ u32 acpi_speed;
+ int i;
+ static const u32 supported_speeds[] = {
+ 0, 100000, 400000, 1000000, 1400000, 3400000
+ };
+
+ acpi_speed = i2c_acpi_find_bus_speed(&pdev->dev);
+ /* round down to the lowest standard speed */
+ for (i = 1; i < ARRAY_SIZE(supported_speeds); i++) {
+ if (acpi_speed < supported_speeds[i])
+ break;
+ }
+ acpi_speed = supported_speeds[i - 1];
+
+ switch (acpi_speed) {
+ case 100000:
+ return speed100k;
+ case 400000:
+ return speed400k;
+ case 1000000:
+ return speed1000k;
+ case 1400000:
+ return speed1400k;
+ case 3400000:
+ return speed3400k;
+ default:
+ return speed400k;
+ }
+}
+
+static const struct i2c_adapter_quirks amd_i2c_dev_quirks = {
+ .max_read_len = AMD_MP2_I2C_MAX_RW_LENGTH,
+ .max_write_len = AMD_MP2_I2C_MAX_RW_LENGTH,
+};
+
+static int i2c_amd_probe(struct platform_device *pdev)
+{
+ int ret;
+ struct amd_i2c_dev *i2c_dev;
+ acpi_handle handle = ACPI_HANDLE(&pdev->dev);
+ struct acpi_device *adev;
+ struct amd_mp2_dev *mp2_dev;
+ const char *uid;
+
+ if (acpi_bus_get_device(handle, &adev))
+ return -ENODEV;
+
+ /* The ACPI namespace doesn't contain information about which MP2 PCI
+ * device an AMDI0011 ACPI device is related to, so assume that there's
+ * only one MP2 PCI device per system.
+ */
+ mp2_dev = amd_mp2_find_device();
+ if (!mp2_dev || !mp2_dev->probed)
+ /* The MP2 PCI device should get probed later */
+ return -EPROBE_DEFER;
+
+ i2c_dev = devm_kzalloc(&pdev->dev, sizeof(*i2c_dev), GFP_KERNEL);
+ if (!i2c_dev)
+ return -ENOMEM;
+
+ i2c_dev->common.mp2_dev = mp2_dev;
+ i2c_dev->pdev = pdev;
+ platform_set_drvdata(pdev, i2c_dev);
+
+ i2c_dev->common.cmd_completion = &i2c_amd_cmd_completion;
+#ifdef CONFIG_PM
+ i2c_dev->common.suspend = &i2c_amd_suspend;
+ i2c_dev->common.resume = &i2c_amd_resume;
+#endif
+
+ uid = adev->pnp.unique_id;
+ if (!uid) {
+ dev_err(&pdev->dev, "missing UID/bus id!\n");
+ return -EINVAL;
+ } else if (strcmp(uid, "0") == 0) {
+ i2c_dev->common.bus_id = 0;
+ } else if (strcmp(uid, "1") == 0) {
+ i2c_dev->common.bus_id = 1;
+ } else {
+ dev_err(&pdev->dev, "incorrect UID/bus id \"%s\"!\n", uid);
+ return -EINVAL;
+ }
+ dev_dbg(&pdev->dev, "bus id is %u\n", i2c_dev->common.bus_id);
+
+ /* Register the adapter */
+ amd_mp2_pm_runtime_get(mp2_dev);
+
+ i2c_dev->common.reqcmd = i2c_none;
+ if (amd_mp2_register_cb(&i2c_dev->common))
+ return -EINVAL;
+ device_link_add(&i2c_dev->pdev->dev, &mp2_dev->pci_dev->dev,
+ DL_FLAG_AUTOREMOVE_CONSUMER);
+
+ i2c_dev->common.i2c_speed = i2c_amd_get_bus_speed(pdev);
+
+ /* Setup i2c adapter description */
+ i2c_dev->adap.owner = THIS_MODULE;
+ i2c_dev->adap.algo = &i2c_amd_algorithm;
+ i2c_dev->adap.quirks = &amd_i2c_dev_quirks;
+ i2c_dev->adap.dev.parent = &pdev->dev;
+ i2c_dev->adap.algo_data = i2c_dev;
+ i2c_dev->adap.timeout = AMD_I2C_TIMEOUT;
+ ACPI_COMPANION_SET(&i2c_dev->adap.dev, ACPI_COMPANION(&pdev->dev));
+ i2c_dev->adap.dev.of_node = pdev->dev.of_node;
+ snprintf(i2c_dev->adap.name, sizeof(i2c_dev->adap.name),
+ "AMD MP2 i2c bus %u", i2c_dev->common.bus_id);
+ i2c_set_adapdata(&i2c_dev->adap, i2c_dev);
+
+ init_completion(&i2c_dev->cmd_complete);
+
+ /* Enable the bus */
+ if (i2c_amd_enable_set(i2c_dev, true))
+ dev_err(&pdev->dev, "initial bus enable failed\n");
+
+ /* Attach to the i2c layer */
+ ret = i2c_add_adapter(&i2c_dev->adap);
+
+ amd_mp2_pm_runtime_put(mp2_dev);
+
+ if (ret < 0)
+ dev_err(&pdev->dev, "i2c add adapter failed = %d\n", ret);
+
+ return ret;
+}
+
+static int i2c_amd_remove(struct platform_device *pdev)
+{
+ struct amd_i2c_dev *i2c_dev = platform_get_drvdata(pdev);
+ struct amd_i2c_common *i2c_common = &i2c_dev->common;
+
+ i2c_lock_bus(&i2c_dev->adap, I2C_LOCK_ROOT_ADAPTER);
+
+ i2c_amd_enable_set(i2c_dev, false);
+ amd_mp2_unregister_cb(i2c_common);
+ i2c_common->mp2_dev = NULL;
+
+ i2c_unlock_bus(&i2c_dev->adap, I2C_LOCK_ROOT_ADAPTER);
+
+ i2c_del_adapter(&i2c_dev->adap);
+ return 0;
+}
+
+static const struct acpi_device_id i2c_amd_acpi_match[] = {
+ { "AMDI0011" },
+ { },
+};
+MODULE_DEVICE_TABLE(acpi, i2c_amd_acpi_match);
+
+static struct platform_driver i2c_amd_plat_driver = {
+ .probe = i2c_amd_probe,
+ .remove = i2c_amd_remove,
+ .driver = {
+ .name = "i2c_amd_mp2",
+ .acpi_match_table = ACPI_PTR(i2c_amd_acpi_match),
+ },
+};
+module_platform_driver(i2c_amd_plat_driver);
+
+MODULE_DESCRIPTION("AMD(R) MP2 I2C Platform Driver");
+MODULE_AUTHOR("Nehal Shah <nehal-bakulchandra.shah@amd.com>");
+MODULE_AUTHOR("Elie Morisse <syniurge@gmail.com>");
+MODULE_LICENSE("Dual BSD/GPL");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * AMD MP2 I2C adapter driver
+ *
+ * Authors: Shyam Sundar S K <Shyam-sundar.S-k@amd.com>
+ * Elie Morisse <syniurge@gmail.com>
+ */
+
+#ifndef I2C_AMD_PCI_MP2_H
+#define I2C_AMD_PCI_MP2_H
+
+#include <linux/i2c.h>
+#include <linux/pci.h>
+#include <linux/pm_runtime.h>
+
+#define PCI_DEVICE_ID_AMD_MP2 0x15E6
+
+struct amd_i2c_common;
+struct amd_mp2_dev;
+
+enum {
+ /* MP2 C2P Message Registers */
+ AMD_C2P_MSG0 = 0x10500, /* MP2 Message for I2C0 */
+ AMD_C2P_MSG1 = 0x10504, /* MP2 Message for I2C1 */
+ AMD_C2P_MSG2 = 0x10508, /* DRAM Address Lo / Data 0 */
+ AMD_C2P_MSG3 = 0x1050c, /* DRAM Address HI / Data 1 */
+ AMD_C2P_MSG4 = 0x10510, /* Data 2 */
+ AMD_C2P_MSG5 = 0x10514, /* Data 3 */
+ AMD_C2P_MSG6 = 0x10518, /* Data 4 */
+ AMD_C2P_MSG7 = 0x1051c, /* Data 5 */
+ AMD_C2P_MSG8 = 0x10520, /* Data 6 */
+ AMD_C2P_MSG9 = 0x10524, /* Data 7 */
+
+ /* MP2 P2C Message Registers */
+ AMD_P2C_MSG0 = 0x10680, /* Do not use */
+ AMD_P2C_MSG1 = 0x10684, /* I2C0 interrupt register */
+ AMD_P2C_MSG2 = 0x10688, /* I2C1 interrupt register */
+ AMD_P2C_MSG3 = 0x1068C, /* MP2 debug info */
+ AMD_P2C_MSG_INTEN = 0x10690, /* MP2 interrupt gen register */
+ AMD_P2C_MSG_INTSTS = 0x10694, /* Interrupt status */
+};
+
+/* Command register data structures */
+
+#define i2c_none (-1)
+enum i2c_cmd {
+ i2c_read = 0,
+ i2c_write,
+ i2c_enable,
+ i2c_disable,
+ number_of_sensor_discovered,
+ is_mp2_active,
+ invalid_cmd = 0xF,
+};
+
+enum speed_enum {
+ speed100k = 0,
+ speed400k = 1,
+ speed1000k = 2,
+ speed1400k = 3,
+ speed3400k = 4
+};
+
+enum mem_type {
+ use_dram = 0,
+ use_c2pmsg = 1,
+};
+
+/**
+ * union i2c_cmd_base : bit access of C2P commands
+ * @i2c_cmd: bit 0..3 i2c R/W command
+ * @bus_id: bit 4..7 i2c bus index
+ * @slave_addr: bit 8..15 slave address
+ * @length: bit 16..27 read/write length
+ * @i2c_speed: bit 28..30 bus speed
+ * @mem_type: bit 31 0-DRAM; 1-C2P msg o/p
+ */
+union i2c_cmd_base {
+ u32 ul;
+ struct {
+ enum i2c_cmd i2c_cmd : 4;
+ u8 bus_id : 4;
+ u32 slave_addr : 8;
+ u32 length : 12;
+ enum speed_enum i2c_speed : 3;
+ enum mem_type mem_type : 1;
+ } s;
+};
+
+enum response_type {
+ invalid_response = 0,
+ command_success = 1,
+ command_failed = 2,
+};
+
+enum status_type {
+ i2c_readcomplete_event = 0,
+ i2c_readfail_event = 1,
+ i2c_writecomplete_event = 2,
+ i2c_writefail_event = 3,
+ i2c_busenable_complete = 4,
+ i2c_busenable_failed = 5,
+ i2c_busdisable_complete = 6,
+ i2c_busdisable_failed = 7,
+ invalid_data_length = 8,
+ invalid_slave_address = 9,
+ invalid_i2cbus_id = 10,
+ invalid_dram_addr = 11,
+ invalid_command = 12,
+ mp2_active = 13,
+ numberof_sensors_discovered_resp = 14,
+ i2c_bus_notinitialized
+};
+
+/**
+ * union i2c_event : bit access of P2C events
+ * @response: bit 0..1 i2c response type
+ * @status: bit 2..6 status_type
+ * @mem_type: bit 7 0-DRAM; 1-C2P msg o/p
+ * @bus_id: bit 8..11 i2c bus id
+ * @length: bit 12..23 message length
+ * @slave_addr: bit 24-31 slave address
+ */
+union i2c_event {
+ u32 ul;
+ struct {
+ enum response_type response : 2;
+ enum status_type status : 5;
+ enum mem_type mem_type : 1;
+ u8 bus_id : 4;
+ u32 length : 12;
+ u32 slave_addr : 8;
+ } r;
+};
+
+/**
+ * struct amd_i2c_common - per bus/i2c adapter context, shared
+ * between the pci and the platform driver
+ * @eventval: MP2 event value set by the IRQ handler
+ * @mp2_dev: MP2 pci device this adapter is part of
+ * @msg: i2c message
+ * @cmd_completion: function called by the IRQ handler to signal
+ * the platform driver
+ * @reqcmd: requested i2c command type
+ * @cmd_success: set to true if the MP2 responded to a command with
+ * the expected status and response type
+ * @bus_id: bus index
+ * @i2c_speed: i2c bus speed determined by the slowest slave
+ * @dma_buf: if msg length > 32, holds the DMA buffer virtual address
+ * @dma_addr: if msg length > 32, holds the DMA buffer address
+ */
+struct amd_i2c_common {
+ union i2c_event eventval;
+ struct amd_mp2_dev *mp2_dev;
+ struct i2c_msg *msg;
+ void (*cmd_completion)(struct amd_i2c_common *i2c_common);
+ enum i2c_cmd reqcmd;
+ u8 cmd_success;
+ u8 bus_id;
+ enum speed_enum i2c_speed;
+ u8 *dma_buf;
+ dma_addr_t dma_addr;
+#ifdef CONFIG_PM
+ int (*suspend)(struct amd_i2c_common *i2c_common);
+ int (*resume)(struct amd_i2c_common *i2c_common);
+#endif /* CONFIG_PM */
+};
+
+/**
+ * struct amd_mp2_dev - per PCI device context
+ * @pci_dev: PCI driver node
+ * @busses: MP2 devices may have up to two busses,
+ * each bus corresponding to an i2c adapter
+ * @mmio: iommapped registers
+ * @c2p_lock: controls access to the C2P mailbox shared between
+ * the two adapters
+ * @c2p_lock_busid: id of the adapter which locked c2p_lock
+ */
+struct amd_mp2_dev {
+ struct pci_dev *pci_dev;
+ struct amd_i2c_common *busses[2];
+ void __iomem *mmio;
+ struct mutex c2p_lock;
+ u8 c2p_lock_busid;
+ unsigned int probed;
+};
+
+#define ndev_pdev(ndev) ((ndev)->pci_dev)
+#define ndev_name(ndev) pci_name(ndev_pdev(ndev))
+#define ndev_dev(ndev) (&ndev_pdev(ndev)->dev)
+#define work_amd_i2c_common(__work) \
+ container_of(__work, struct amd_i2c_common, work.work)
+
+/* PCIe communication driver */
+
+int amd_mp2_rw(struct amd_i2c_common *i2c_common, enum i2c_cmd reqcmd);
+int amd_mp2_bus_enable_set(struct amd_i2c_common *i2c_common, bool enable);
+
+void amd_mp2_process_event(struct amd_i2c_common *i2c_common);
+
+void amd_mp2_rw_timeout(struct amd_i2c_common *i2c_common);
+
+int amd_mp2_register_cb(struct amd_i2c_common *i2c_common);
+int amd_mp2_unregister_cb(struct amd_i2c_common *i2c_common);
+
+struct amd_mp2_dev *amd_mp2_find_device(void);
+
+static inline void amd_mp2_pm_runtime_get(struct amd_mp2_dev *mp2_dev)
+{
+ pm_runtime_get_sync(&mp2_dev->pci_dev->dev);
+}
+
+static inline void amd_mp2_pm_runtime_put(struct amd_mp2_dev *mp2_dev)
+{
+ pm_runtime_mark_last_busy(&mp2_dev->pci_dev->dev);
+ pm_runtime_put_autosuspend(&mp2_dev->pci_dev->dev);
+}
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * i2c Support for Atmel's AT91 Two-Wire Interface (TWI)
+ *
+ * Copyright (C) 2011 Weinmann Medical GmbH
+ * Author: Nikolaus Voss <n.voss@weinmann.de>
+ *
+ * Evolved from original work by:
+ * Copyright (C) 2004 Rick Bronson
+ * Converted to 2.6 by Andrew Victor <andrew@sanpeople.com>
+ *
+ * Borrowed heavily from original work by:
+ * Copyright (C) 2000 Philip Edelbrock <phil@stimpy.netroedge.com>
+ */
+
+#include <linux/clk.h>
+#include <linux/err.h>
+#include <linux/i2c.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/pinctrl/consumer.h>
+
+#include "i2c-at91.h"
+
+unsigned at91_twi_read(struct at91_twi_dev *dev, unsigned reg)
+{
+ return readl_relaxed(dev->base + reg);
+}
+
+void at91_twi_write(struct at91_twi_dev *dev, unsigned reg, unsigned val)
+{
+ writel_relaxed(val, dev->base + reg);
+}
+
+void at91_disable_twi_interrupts(struct at91_twi_dev *dev)
+{
+ at91_twi_write(dev, AT91_TWI_IDR, AT91_TWI_INT_MASK);
+}
+
+void at91_twi_irq_save(struct at91_twi_dev *dev)
+{
+ dev->imr = at91_twi_read(dev, AT91_TWI_IMR) & AT91_TWI_INT_MASK;
+ at91_disable_twi_interrupts(dev);
+}
+
+void at91_twi_irq_restore(struct at91_twi_dev *dev)
+{
+ at91_twi_write(dev, AT91_TWI_IER, dev->imr);
+}
+
+void at91_init_twi_bus(struct at91_twi_dev *dev)
+{
+ at91_disable_twi_interrupts(dev);
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_SWRST);
+ if (dev->slave_detected)
+ at91_init_twi_bus_slave(dev);
+ else
+ at91_init_twi_bus_master(dev);
+}
+
+static struct at91_twi_pdata at91rm9200_config = {
+ .clk_max_div = 5,
+ .clk_offset = 3,
+ .has_unre_flag = true,
+ .has_alt_cmd = false,
+ .has_hold_field = false,
+};
+
+static struct at91_twi_pdata at91sam9261_config = {
+ .clk_max_div = 5,
+ .clk_offset = 4,
+ .has_unre_flag = false,
+ .has_alt_cmd = false,
+ .has_hold_field = false,
+};
+
+static struct at91_twi_pdata at91sam9260_config = {
+ .clk_max_div = 7,
+ .clk_offset = 4,
+ .has_unre_flag = false,
+ .has_alt_cmd = false,
+ .has_hold_field = false,
+};
+
+static struct at91_twi_pdata at91sam9g20_config = {
+ .clk_max_div = 7,
+ .clk_offset = 4,
+ .has_unre_flag = false,
+ .has_alt_cmd = false,
+ .has_hold_field = false,
+};
+
+static struct at91_twi_pdata at91sam9g10_config = {
+ .clk_max_div = 7,
+ .clk_offset = 4,
+ .has_unre_flag = false,
+ .has_alt_cmd = false,
+ .has_hold_field = false,
+};
+
+static const struct platform_device_id at91_twi_devtypes[] = {
+ {
+ .name = "i2c-at91rm9200",
+ .driver_data = (unsigned long) &at91rm9200_config,
+ }, {
+ .name = "i2c-at91sam9261",
+ .driver_data = (unsigned long) &at91sam9261_config,
+ }, {
+ .name = "i2c-at91sam9260",
+ .driver_data = (unsigned long) &at91sam9260_config,
+ }, {
+ .name = "i2c-at91sam9g20",
+ .driver_data = (unsigned long) &at91sam9g20_config,
+ }, {
+ .name = "i2c-at91sam9g10",
+ .driver_data = (unsigned long) &at91sam9g10_config,
+ }, {
+ /* sentinel */
+ }
+};
+
+#if defined(CONFIG_OF)
+static struct at91_twi_pdata at91sam9x5_config = {
+ .clk_max_div = 7,
+ .clk_offset = 4,
+ .has_unre_flag = false,
+ .has_alt_cmd = false,
+ .has_hold_field = false,
+};
+
+static struct at91_twi_pdata sama5d4_config = {
+ .clk_max_div = 7,
+ .clk_offset = 4,
+ .has_unre_flag = false,
+ .has_alt_cmd = false,
+ .has_hold_field = true,
+};
+
+static struct at91_twi_pdata sama5d2_config = {
+ .clk_max_div = 7,
+ .clk_offset = 4,
+ .has_unre_flag = true,
+ .has_alt_cmd = true,
+ .has_hold_field = true,
+};
+
+static const struct of_device_id atmel_twi_dt_ids[] = {
+ {
+ .compatible = "atmel,at91rm9200-i2c",
+ .data = &at91rm9200_config,
+ }, {
+ .compatible = "atmel,at91sam9260-i2c",
+ .data = &at91sam9260_config,
+ }, {
+ .compatible = "atmel,at91sam9261-i2c",
+ .data = &at91sam9261_config,
+ }, {
+ .compatible = "atmel,at91sam9g20-i2c",
+ .data = &at91sam9g20_config,
+ }, {
+ .compatible = "atmel,at91sam9g10-i2c",
+ .data = &at91sam9g10_config,
+ }, {
+ .compatible = "atmel,at91sam9x5-i2c",
+ .data = &at91sam9x5_config,
+ }, {
+ .compatible = "atmel,sama5d4-i2c",
+ .data = &sama5d4_config,
+ }, {
+ .compatible = "atmel,sama5d2-i2c",
+ .data = &sama5d2_config,
+ }, {
+ /* sentinel */
+ }
+};
+MODULE_DEVICE_TABLE(of, atmel_twi_dt_ids);
+#endif
+
+static struct at91_twi_pdata *at91_twi_get_driver_data(
+ struct platform_device *pdev)
+{
+ if (pdev->dev.of_node) {
+ const struct of_device_id *match;
+ match = of_match_node(atmel_twi_dt_ids, pdev->dev.of_node);
+ if (!match)
+ return NULL;
+ return (struct at91_twi_pdata *)match->data;
+ }
+ return (struct at91_twi_pdata *) platform_get_device_id(pdev)->driver_data;
+}
+
+static int at91_twi_probe(struct platform_device *pdev)
+{
+ struct at91_twi_dev *dev;
+ struct resource *mem;
+ int rc;
+ u32 phy_addr;
+
+ dev = devm_kzalloc(&pdev->dev, sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return -ENOMEM;
+
+ dev->dev = &pdev->dev;
+
+ mem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!mem)
+ return -ENODEV;
+ phy_addr = mem->start;
+
+ dev->pdata = at91_twi_get_driver_data(pdev);
+ if (!dev->pdata)
+ return -ENODEV;
+
+ dev->base = devm_ioremap_resource(&pdev->dev, mem);
+ if (IS_ERR(dev->base))
+ return PTR_ERR(dev->base);
+
+ dev->irq = platform_get_irq(pdev, 0);
+ if (dev->irq < 0)
+ return dev->irq;
+
+ platform_set_drvdata(pdev, dev);
+
+ dev->clk = devm_clk_get(dev->dev, NULL);
+ if (IS_ERR(dev->clk)) {
+ dev_err(dev->dev, "no clock defined\n");
+ return -ENODEV;
+ }
+ clk_prepare_enable(dev->clk);
+
+ snprintf(dev->adapter.name, sizeof(dev->adapter.name), "AT91");
+ i2c_set_adapdata(&dev->adapter, dev);
+ dev->adapter.owner = THIS_MODULE;
+ dev->adapter.class = I2C_CLASS_DEPRECATED;
+ dev->adapter.dev.parent = dev->dev;
+ dev->adapter.nr = pdev->id;
+ dev->adapter.timeout = AT91_I2C_TIMEOUT;
+ dev->adapter.dev.of_node = pdev->dev.of_node;
+
+ dev->slave_detected = i2c_detect_slave_mode(&pdev->dev);
+
+ if (dev->slave_detected)
+ rc = at91_twi_probe_slave(pdev, phy_addr, dev);
+ else
+ rc = at91_twi_probe_master(pdev, phy_addr, dev);
+ if (rc)
+ return rc;
+
+ at91_init_twi_bus(dev);
+
+ pm_runtime_set_autosuspend_delay(dev->dev, AUTOSUSPEND_TIMEOUT);
+ pm_runtime_use_autosuspend(dev->dev);
+ pm_runtime_set_active(dev->dev);
+ pm_runtime_enable(dev->dev);
+
+ rc = i2c_add_numbered_adapter(&dev->adapter);
+ if (rc) {
+ clk_disable_unprepare(dev->clk);
+
+ pm_runtime_disable(dev->dev);
+ pm_runtime_set_suspended(dev->dev);
+
+ return rc;
+ }
+
+ dev_info(dev->dev, "AT91 i2c bus driver (hw version: %#x).\n",
+ at91_twi_read(dev, AT91_TWI_VER));
+ return 0;
+}
+
+static int at91_twi_remove(struct platform_device *pdev)
+{
+ struct at91_twi_dev *dev = platform_get_drvdata(pdev);
+
+ i2c_del_adapter(&dev->adapter);
+ clk_disable_unprepare(dev->clk);
+
+ pm_runtime_disable(dev->dev);
+ pm_runtime_set_suspended(dev->dev);
+
+ return 0;
+}
+
+#ifdef CONFIG_PM
+
+static int at91_twi_runtime_suspend(struct device *dev)
+{
+ struct at91_twi_dev *twi_dev = dev_get_drvdata(dev);
+
+ clk_disable_unprepare(twi_dev->clk);
+
+ pinctrl_pm_select_sleep_state(dev);
+
+ return 0;
+}
+
+static int at91_twi_runtime_resume(struct device *dev)
+{
+ struct at91_twi_dev *twi_dev = dev_get_drvdata(dev);
+
+ pinctrl_pm_select_default_state(dev);
+
+ return clk_prepare_enable(twi_dev->clk);
+}
+
+static int at91_twi_suspend_noirq(struct device *dev)
+{
+ if (!pm_runtime_status_suspended(dev))
+ at91_twi_runtime_suspend(dev);
+
+ return 0;
+}
+
+static int at91_twi_resume_noirq(struct device *dev)
+{
+ struct at91_twi_dev *twi_dev = dev_get_drvdata(dev);
+ int ret;
+
+ if (!pm_runtime_status_suspended(dev)) {
+ ret = at91_twi_runtime_resume(dev);
+ if (ret)
+ return ret;
+ }
+
+ pm_runtime_mark_last_busy(dev);
+ pm_request_autosuspend(dev);
+
+ at91_init_twi_bus(twi_dev);
+
+ return 0;
+}
+
+static const struct dev_pm_ops at91_twi_pm = {
+ .suspend_noirq = at91_twi_suspend_noirq,
+ .resume_noirq = at91_twi_resume_noirq,
+ .runtime_suspend = at91_twi_runtime_suspend,
+ .runtime_resume = at91_twi_runtime_resume,
+};
+
+#define at91_twi_pm_ops (&at91_twi_pm)
+#else
+#define at91_twi_pm_ops NULL
+#endif
+
+static struct platform_driver at91_twi_driver = {
+ .probe = at91_twi_probe,
+ .remove = at91_twi_remove,
+ .id_table = at91_twi_devtypes,
+ .driver = {
+ .name = "at91_i2c",
+ .of_match_table = of_match_ptr(atmel_twi_dt_ids),
+ .pm = at91_twi_pm_ops,
+ },
+};
+
+static int __init at91_twi_init(void)
+{
+ return platform_driver_register(&at91_twi_driver);
+}
+
+static void __exit at91_twi_exit(void)
+{
+ platform_driver_unregister(&at91_twi_driver);
+}
+
+subsys_initcall(at91_twi_init);
+module_exit(at91_twi_exit);
+
+MODULE_AUTHOR("Nikolaus Voss <n.voss@weinmann.de>");
+MODULE_DESCRIPTION("I2C (TWI) driver for Atmel AT91");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:at91_i2c");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * i2c Support for Atmel's AT91 Two-Wire Interface (TWI)
+ *
+ * Copyright (C) 2011 Weinmann Medical GmbH
+ * Author: Nikolaus Voss <n.voss@weinmann.de>
+ *
+ * Evolved from original work by:
+ * Copyright (C) 2004 Rick Bronson
+ * Converted to 2.6 by Andrew Victor <andrew@sanpeople.com>
+ *
+ * Borrowed heavily from original work by:
+ * Copyright (C) 2000 Philip Edelbrock <phil@stimpy.netroedge.com>
+ */
+
+#include <linux/clk.h>
+#include <linux/completion.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmaengine.h>
+#include <linux/err.h>
+#include <linux/i2c.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/platform_data/dma-atmel.h>
+#include <linux/pm_runtime.h>
+
+#include "i2c-at91.h"
+
+void at91_init_twi_bus_master(struct at91_twi_dev *dev)
+{
+ /* FIFO should be enabled immediately after the software reset */
+ if (dev->fifo_size)
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_FIFOEN);
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_MSEN);
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_SVDIS);
+ at91_twi_write(dev, AT91_TWI_CWGR, dev->twi_cwgr_reg);
+}
+
+/*
+ * Calculate symmetric clock as stated in datasheet:
+ * twi_clk = F_MAIN / (2 * (cdiv * (1 << ckdiv) + offset))
+ */
+static void at91_calc_twi_clock(struct at91_twi_dev *dev)
+{
+ int ckdiv, cdiv, div, hold = 0;
+ struct at91_twi_pdata *pdata = dev->pdata;
+ int offset = pdata->clk_offset;
+ int max_ckdiv = pdata->clk_max_div;
+ struct i2c_timings timings, *t = &timings;
+
+ i2c_parse_fw_timings(dev->dev, t, true);
+
+ div = max(0, (int)DIV_ROUND_UP(clk_get_rate(dev->clk),
+ 2 * t->bus_freq_hz) - offset);
+ ckdiv = fls(div >> 8);
+ cdiv = div >> ckdiv;
+
+ if (ckdiv > max_ckdiv) {
+ dev_warn(dev->dev, "%d exceeds ckdiv max value which is %d.\n",
+ ckdiv, max_ckdiv);
+ ckdiv = max_ckdiv;
+ cdiv = 255;
+ }
+
+ if (pdata->has_hold_field) {
+ /*
+ * hold time = HOLD + 3 x T_peripheral_clock
+ * Use clk rate in kHz to prevent overflows when computing
+ * hold.
+ */
+ hold = DIV_ROUND_UP(t->sda_hold_ns
+ * (clk_get_rate(dev->clk) / 1000), 1000000);
+ hold -= 3;
+ if (hold < 0)
+ hold = 0;
+ if (hold > AT91_TWI_CWGR_HOLD_MAX) {
+ dev_warn(dev->dev,
+ "HOLD field set to its maximum value (%d instead of %d)\n",
+ AT91_TWI_CWGR_HOLD_MAX, hold);
+ hold = AT91_TWI_CWGR_HOLD_MAX;
+ }
+ }
+
+ dev->twi_cwgr_reg = (ckdiv << 16) | (cdiv << 8) | cdiv
+ | AT91_TWI_CWGR_HOLD(hold);
+
+ dev_dbg(dev->dev, "cdiv %d ckdiv %d hold %d (%d ns)\n",
+ cdiv, ckdiv, hold, t->sda_hold_ns);
+}
+
+static void at91_twi_dma_cleanup(struct at91_twi_dev *dev)
+{
+ struct at91_twi_dma *dma = &dev->dma;
+
+ at91_twi_irq_save(dev);
+
+ if (dma->xfer_in_progress) {
+ if (dma->direction == DMA_FROM_DEVICE)
+ dmaengine_terminate_all(dma->chan_rx);
+ else
+ dmaengine_terminate_all(dma->chan_tx);
+ dma->xfer_in_progress = false;
+ }
+ if (dma->buf_mapped) {
+ dma_unmap_single(dev->dev, sg_dma_address(&dma->sg[0]),
+ dev->buf_len, dma->direction);
+ dma->buf_mapped = false;
+ }
+
+ at91_twi_irq_restore(dev);
+}
+
+static void at91_twi_write_next_byte(struct at91_twi_dev *dev)
+{
+ if (!dev->buf_len)
+ return;
+
+ /* 8bit write works with and without FIFO */
+ writeb_relaxed(*dev->buf, dev->base + AT91_TWI_THR);
+
+ /* send stop when last byte has been written */
+ if (--dev->buf_len == 0)
+ if (!dev->use_alt_cmd)
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_STOP);
+
+ dev_dbg(dev->dev, "wrote 0x%x, to go %zu\n", *dev->buf, dev->buf_len);
+
+ ++dev->buf;
+}
+
+static void at91_twi_write_data_dma_callback(void *data)
+{
+ struct at91_twi_dev *dev = (struct at91_twi_dev *)data;
+
+ dma_unmap_single(dev->dev, sg_dma_address(&dev->dma.sg[0]),
+ dev->buf_len, DMA_TO_DEVICE);
+
+ /*
+ * When this callback is called, THR/TX FIFO is likely not to be empty
+ * yet. So we have to wait for TXCOMP or NACK bits to be set into the
+ * Status Register to be sure that the STOP bit has been sent and the
+ * transfer is completed. The NACK interrupt has already been enabled,
+ * we just have to enable TXCOMP one.
+ */
+ at91_twi_write(dev, AT91_TWI_IER, AT91_TWI_TXCOMP);
+ if (!dev->use_alt_cmd)
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_STOP);
+}
+
+static void at91_twi_write_data_dma(struct at91_twi_dev *dev)
+{
+ dma_addr_t dma_addr;
+ struct dma_async_tx_descriptor *txdesc;
+ struct at91_twi_dma *dma = &dev->dma;
+ struct dma_chan *chan_tx = dma->chan_tx;
+ unsigned int sg_len = 1;
+
+ if (!dev->buf_len)
+ return;
+
+ dma->direction = DMA_TO_DEVICE;
+
+ at91_twi_irq_save(dev);
+ dma_addr = dma_map_single(dev->dev, dev->buf, dev->buf_len,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(dev->dev, dma_addr)) {
+ dev_err(dev->dev, "dma map failed\n");
+ return;
+ }
+ dma->buf_mapped = true;
+ at91_twi_irq_restore(dev);
+
+ if (dev->fifo_size) {
+ size_t part1_len, part2_len;
+ struct scatterlist *sg;
+ unsigned fifo_mr;
+
+ sg_len = 0;
+
+ part1_len = dev->buf_len & ~0x3;
+ if (part1_len) {
+ sg = &dma->sg[sg_len++];
+ sg_dma_len(sg) = part1_len;
+ sg_dma_address(sg) = dma_addr;
+ }
+
+ part2_len = dev->buf_len & 0x3;
+ if (part2_len) {
+ sg = &dma->sg[sg_len++];
+ sg_dma_len(sg) = part2_len;
+ sg_dma_address(sg) = dma_addr + part1_len;
+ }
+
+ /*
+ * DMA controller is triggered when at least 4 data can be
+ * written into the TX FIFO
+ */
+ fifo_mr = at91_twi_read(dev, AT91_TWI_FMR);
+ fifo_mr &= ~AT91_TWI_FMR_TXRDYM_MASK;
+ fifo_mr |= AT91_TWI_FMR_TXRDYM(AT91_TWI_FOUR_DATA);
+ at91_twi_write(dev, AT91_TWI_FMR, fifo_mr);
+ } else {
+ sg_dma_len(&dma->sg[0]) = dev->buf_len;
+ sg_dma_address(&dma->sg[0]) = dma_addr;
+ }
+
+ txdesc = dmaengine_prep_slave_sg(chan_tx, dma->sg, sg_len,
+ DMA_MEM_TO_DEV,
+ DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
+ if (!txdesc) {
+ dev_err(dev->dev, "dma prep slave sg failed\n");
+ goto error;
+ }
+
+ txdesc->callback = at91_twi_write_data_dma_callback;
+ txdesc->callback_param = dev;
+
+ dma->xfer_in_progress = true;
+ dmaengine_submit(txdesc);
+ dma_async_issue_pending(chan_tx);
+
+ return;
+
+error:
+ at91_twi_dma_cleanup(dev);
+}
+
+static void at91_twi_read_next_byte(struct at91_twi_dev *dev)
+{
+ /*
+ * If we are in this case, it means there is garbage data in RHR, so
+ * delete them.
+ */
+ if (!dev->buf_len) {
+ at91_twi_read(dev, AT91_TWI_RHR);
+ return;
+ }
+
+ /* 8bit read works with and without FIFO */
+ *dev->buf = readb_relaxed(dev->base + AT91_TWI_RHR);
+ --dev->buf_len;
+
+ /* return if aborting, we only needed to read RHR to clear RXRDY*/
+ if (dev->recv_len_abort)
+ return;
+
+ /* handle I2C_SMBUS_BLOCK_DATA */
+ if (unlikely(dev->msg->flags & I2C_M_RECV_LEN)) {
+ /* ensure length byte is a valid value */
+ if (*dev->buf <= I2C_SMBUS_BLOCK_MAX && *dev->buf > 0) {
+ dev->msg->flags &= ~I2C_M_RECV_LEN;
+ dev->buf_len += *dev->buf;
+ dev->msg->len = dev->buf_len + 1;
+ dev_dbg(dev->dev, "received block length %zu\n",
+ dev->buf_len);
+ } else {
+ /* abort and send the stop by reading one more byte */
+ dev->recv_len_abort = true;
+ dev->buf_len = 1;
+ }
+ }
+
+ /* send stop if second but last byte has been read */
+ if (!dev->use_alt_cmd && dev->buf_len == 1)
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_STOP);
+
+ dev_dbg(dev->dev, "read 0x%x, to go %zu\n", *dev->buf, dev->buf_len);
+
+ ++dev->buf;
+}
+
+static void at91_twi_read_data_dma_callback(void *data)
+{
+ struct at91_twi_dev *dev = (struct at91_twi_dev *)data;
+ unsigned ier = AT91_TWI_TXCOMP;
+
+ dma_unmap_single(dev->dev, sg_dma_address(&dev->dma.sg[0]),
+ dev->buf_len, DMA_FROM_DEVICE);
+
+ if (!dev->use_alt_cmd) {
+ /* The last two bytes have to be read without using dma */
+ dev->buf += dev->buf_len - 2;
+ dev->buf_len = 2;
+ ier |= AT91_TWI_RXRDY;
+ }
+ at91_twi_write(dev, AT91_TWI_IER, ier);
+}
+
+static void at91_twi_read_data_dma(struct at91_twi_dev *dev)
+{
+ dma_addr_t dma_addr;
+ struct dma_async_tx_descriptor *rxdesc;
+ struct at91_twi_dma *dma = &dev->dma;
+ struct dma_chan *chan_rx = dma->chan_rx;
+ size_t buf_len;
+
+ buf_len = (dev->use_alt_cmd) ? dev->buf_len : dev->buf_len - 2;
+ dma->direction = DMA_FROM_DEVICE;
+
+ /* Keep in mind that we won't use dma to read the last two bytes */
+ at91_twi_irq_save(dev);
+ dma_addr = dma_map_single(dev->dev, dev->buf, buf_len, DMA_FROM_DEVICE);
+ if (dma_mapping_error(dev->dev, dma_addr)) {
+ dev_err(dev->dev, "dma map failed\n");
+ return;
+ }
+ dma->buf_mapped = true;
+ at91_twi_irq_restore(dev);
+
+ if (dev->fifo_size && IS_ALIGNED(buf_len, 4)) {
+ unsigned fifo_mr;
+
+ /*
+ * DMA controller is triggered when at least 4 data can be
+ * read from the RX FIFO
+ */
+ fifo_mr = at91_twi_read(dev, AT91_TWI_FMR);
+ fifo_mr &= ~AT91_TWI_FMR_RXRDYM_MASK;
+ fifo_mr |= AT91_TWI_FMR_RXRDYM(AT91_TWI_FOUR_DATA);
+ at91_twi_write(dev, AT91_TWI_FMR, fifo_mr);
+ }
+
+ sg_dma_len(&dma->sg[0]) = buf_len;
+ sg_dma_address(&dma->sg[0]) = dma_addr;
+
+ rxdesc = dmaengine_prep_slave_sg(chan_rx, dma->sg, 1, DMA_DEV_TO_MEM,
+ DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
+ if (!rxdesc) {
+ dev_err(dev->dev, "dma prep slave sg failed\n");
+ goto error;
+ }
+
+ rxdesc->callback = at91_twi_read_data_dma_callback;
+ rxdesc->callback_param = dev;
+
+ dma->xfer_in_progress = true;
+ dmaengine_submit(rxdesc);
+ dma_async_issue_pending(dma->chan_rx);
+
+ return;
+
+error:
+ at91_twi_dma_cleanup(dev);
+}
+
+static irqreturn_t atmel_twi_interrupt(int irq, void *dev_id)
+{
+ struct at91_twi_dev *dev = dev_id;
+ const unsigned status = at91_twi_read(dev, AT91_TWI_SR);
+ const unsigned irqstatus = status & at91_twi_read(dev, AT91_TWI_IMR);
+
+ if (!irqstatus)
+ return IRQ_NONE;
+ /*
+ * In reception, the behavior of the twi device (before sama5d2) is
+ * weird. There is some magic about RXRDY flag! When a data has been
+ * almost received, the reception of a new one is anticipated if there
+ * is no stop command to send. That is the reason why ask for sending
+ * the stop command not on the last data but on the second last one.
+ *
+ * Unfortunately, we could still have the RXRDY flag set even if the
+ * transfer is done and we have read the last data. It might happen
+ * when the i2c slave device sends too quickly data after receiving the
+ * ack from the master. The data has been almost received before having
+ * the order to send stop. In this case, sending the stop command could
+ * cause a RXRDY interrupt with a TXCOMP one. It is better to manage
+ * the RXRDY interrupt first in order to not keep garbage data in the
+ * Receive Holding Register for the next transfer.
+ */
+ if (irqstatus & AT91_TWI_RXRDY) {
+ /*
+ * Read all available bytes at once by polling RXRDY usable w/
+ * and w/o FIFO. With FIFO enabled we could also read RXFL and
+ * avoid polling RXRDY.
+ */
+ do {
+ at91_twi_read_next_byte(dev);
+ } while (at91_twi_read(dev, AT91_TWI_SR) & AT91_TWI_RXRDY);
+ }
+
+ /*
+ * When a NACK condition is detected, the I2C controller sets the NACK,
+ * TXCOMP and TXRDY bits all together in the Status Register (SR).
+ *
+ * 1 - Handling NACK errors with CPU write transfer.
+ *
+ * In such case, we should not write the next byte into the Transmit
+ * Holding Register (THR) otherwise the I2C controller would start a new
+ * transfer and the I2C slave is likely to reply by another NACK.
+ *
+ * 2 - Handling NACK errors with DMA write transfer.
+ *
+ * By setting the TXRDY bit in the SR, the I2C controller also triggers
+ * the DMA controller to write the next data into the THR. Then the
+ * result depends on the hardware version of the I2C controller.
+ *
+ * 2a - Without support of the Alternative Command mode.
+ *
+ * This is the worst case: the DMA controller is triggered to write the
+ * next data into the THR, hence starting a new transfer: the I2C slave
+ * is likely to reply by another NACK.
+ * Concurrently, this interrupt handler is likely to be called to manage
+ * the first NACK before the I2C controller detects the second NACK and
+ * sets once again the NACK bit into the SR.
+ * When handling the first NACK, this interrupt handler disables the I2C
+ * controller interruptions, especially the NACK interrupt.
+ * Hence, the NACK bit is pending into the SR. This is why we should
+ * read the SR to clear all pending interrupts at the beginning of
+ * at91_do_twi_transfer() before actually starting a new transfer.
+ *
+ * 2b - With support of the Alternative Command mode.
+ *
+ * When a NACK condition is detected, the I2C controller also locks the
+ * THR (and sets the LOCK bit in the SR): even though the DMA controller
+ * is triggered by the TXRDY bit to write the next data into the THR,
+ * this data actually won't go on the I2C bus hence a second NACK is not
+ * generated.
+ */
+ if (irqstatus & (AT91_TWI_TXCOMP | AT91_TWI_NACK)) {
+ at91_disable_twi_interrupts(dev);
+ complete(&dev->cmd_complete);
+ } else if (irqstatus & AT91_TWI_TXRDY) {
+ at91_twi_write_next_byte(dev);
+ }
+
+ /* catch error flags */
+ dev->transfer_status |= status;
+
+ return IRQ_HANDLED;
+}
+
+static int at91_do_twi_transfer(struct at91_twi_dev *dev)
+{
+ int ret;
+ unsigned long time_left;
+ bool has_unre_flag = dev->pdata->has_unre_flag;
+ bool has_alt_cmd = dev->pdata->has_alt_cmd;
+
+ /*
+ * WARNING: the TXCOMP bit in the Status Register is NOT a clear on
+ * read flag but shows the state of the transmission at the time the
+ * Status Register is read. According to the programmer datasheet,
+ * TXCOMP is set when both holding register and internal shifter are
+ * empty and STOP condition has been sent.
+ * Consequently, we should enable NACK interrupt rather than TXCOMP to
+ * detect transmission failure.
+ * Indeed let's take the case of an i2c write command using DMA.
+ * Whenever the slave doesn't acknowledge a byte, the LOCK, NACK and
+ * TXCOMP bits are set together into the Status Register.
+ * LOCK is a clear on write bit, which is set to prevent the DMA
+ * controller from sending new data on the i2c bus after a NACK
+ * condition has happened. Once locked, this i2c peripheral stops
+ * triggering the DMA controller for new data but it is more than
+ * likely that a new DMA transaction is already in progress, writing
+ * into the Transmit Holding Register. Since the peripheral is locked,
+ * these new data won't be sent to the i2c bus but they will remain
+ * into the Transmit Holding Register, so TXCOMP bit is cleared.
+ * Then when the interrupt handler is called, the Status Register is
+ * read: the TXCOMP bit is clear but NACK bit is still set. The driver
+ * manage the error properly, without waiting for timeout.
+ * This case can be reproduced easyly when writing into an at24 eeprom.
+ *
+ * Besides, the TXCOMP bit is already set before the i2c transaction
+ * has been started. For read transactions, this bit is cleared when
+ * writing the START bit into the Control Register. So the
+ * corresponding interrupt can safely be enabled just after.
+ * However for write transactions managed by the CPU, we first write
+ * into THR, so TXCOMP is cleared. Then we can safely enable TXCOMP
+ * interrupt. If TXCOMP interrupt were enabled before writing into THR,
+ * the interrupt handler would be called immediately and the i2c command
+ * would be reported as completed.
+ * Also when a write transaction is managed by the DMA controller,
+ * enabling the TXCOMP interrupt in this function may lead to a race
+ * condition since we don't know whether the TXCOMP interrupt is enabled
+ * before or after the DMA has started to write into THR. So the TXCOMP
+ * interrupt is enabled later by at91_twi_write_data_dma_callback().
+ * Immediately after in that DMA callback, if the alternative command
+ * mode is not used, we still need to send the STOP condition manually
+ * writing the corresponding bit into the Control Register.
+ */
+
+ dev_dbg(dev->dev, "transfer: %s %zu bytes.\n",
+ (dev->msg->flags & I2C_M_RD) ? "read" : "write", dev->buf_len);
+
+ reinit_completion(&dev->cmd_complete);
+ dev->transfer_status = 0;
+
+ /* Clear pending interrupts, such as NACK. */
+ at91_twi_read(dev, AT91_TWI_SR);
+
+ if (dev->fifo_size) {
+ unsigned fifo_mr = at91_twi_read(dev, AT91_TWI_FMR);
+
+ /* Reset FIFO mode register */
+ fifo_mr &= ~(AT91_TWI_FMR_TXRDYM_MASK |
+ AT91_TWI_FMR_RXRDYM_MASK);
+ fifo_mr |= AT91_TWI_FMR_TXRDYM(AT91_TWI_ONE_DATA);
+ fifo_mr |= AT91_TWI_FMR_RXRDYM(AT91_TWI_ONE_DATA);
+ at91_twi_write(dev, AT91_TWI_FMR, fifo_mr);
+
+ /* Flush FIFOs */
+ at91_twi_write(dev, AT91_TWI_CR,
+ AT91_TWI_THRCLR | AT91_TWI_RHRCLR);
+ }
+
+ if (!dev->buf_len) {
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_QUICK);
+ at91_twi_write(dev, AT91_TWI_IER, AT91_TWI_TXCOMP);
+ } else if (dev->msg->flags & I2C_M_RD) {
+ unsigned start_flags = AT91_TWI_START;
+
+ /* if only one byte is to be read, immediately stop transfer */
+ if (!dev->use_alt_cmd && dev->buf_len <= 1 &&
+ !(dev->msg->flags & I2C_M_RECV_LEN))
+ start_flags |= AT91_TWI_STOP;
+ at91_twi_write(dev, AT91_TWI_CR, start_flags);
+ /*
+ * When using dma without alternative command mode, the last
+ * byte has to be read manually in order to not send the stop
+ * command too late and then to receive extra data.
+ * In practice, there are some issues if you use the dma to
+ * read n-1 bytes because of latency.
+ * Reading n-2 bytes with dma and the two last ones manually
+ * seems to be the best solution.
+ */
+ if (dev->use_dma && (dev->buf_len > AT91_I2C_DMA_THRESHOLD)) {
+ at91_twi_write(dev, AT91_TWI_IER, AT91_TWI_NACK);
+ at91_twi_read_data_dma(dev);
+ } else {
+ at91_twi_write(dev, AT91_TWI_IER,
+ AT91_TWI_TXCOMP |
+ AT91_TWI_NACK |
+ AT91_TWI_RXRDY);
+ }
+ } else {
+ if (dev->use_dma && (dev->buf_len > AT91_I2C_DMA_THRESHOLD)) {
+ at91_twi_write(dev, AT91_TWI_IER, AT91_TWI_NACK);
+ at91_twi_write_data_dma(dev);
+ } else {
+ at91_twi_write_next_byte(dev);
+ at91_twi_write(dev, AT91_TWI_IER,
+ AT91_TWI_TXCOMP |
+ AT91_TWI_NACK |
+ AT91_TWI_TXRDY);
+ }
+ }
+
+ time_left = wait_for_completion_timeout(&dev->cmd_complete,
+ dev->adapter.timeout);
+ if (time_left == 0) {
+ dev->transfer_status |= at91_twi_read(dev, AT91_TWI_SR);
+ dev_err(dev->dev, "controller timed out\n");
+ at91_init_twi_bus(dev);
+ ret = -ETIMEDOUT;
+ goto error;
+ }
+ if (dev->transfer_status & AT91_TWI_NACK) {
+ dev_dbg(dev->dev, "received nack\n");
+ ret = -EREMOTEIO;
+ goto error;
+ }
+ if (dev->transfer_status & AT91_TWI_OVRE) {
+ dev_err(dev->dev, "overrun while reading\n");
+ ret = -EIO;
+ goto error;
+ }
+ if (has_unre_flag && dev->transfer_status & AT91_TWI_UNRE) {
+ dev_err(dev->dev, "underrun while writing\n");
+ ret = -EIO;
+ goto error;
+ }
+ if ((has_alt_cmd || dev->fifo_size) &&
+ (dev->transfer_status & AT91_TWI_LOCK)) {
+ dev_err(dev->dev, "tx locked\n");
+ ret = -EIO;
+ goto error;
+ }
+ if (dev->recv_len_abort) {
+ dev_err(dev->dev, "invalid smbus block length recvd\n");
+ ret = -EPROTO;
+ goto error;
+ }
+
+ dev_dbg(dev->dev, "transfer complete\n");
+
+ return 0;
+
+error:
+ /* first stop DMA transfer if still in progress */
+ at91_twi_dma_cleanup(dev);
+ /* then flush THR/FIFO and unlock TX if locked */
+ if ((has_alt_cmd || dev->fifo_size) &&
+ (dev->transfer_status & AT91_TWI_LOCK)) {
+ dev_dbg(dev->dev, "unlock tx\n");
+ at91_twi_write(dev, AT91_TWI_CR,
+ AT91_TWI_THRCLR | AT91_TWI_LOCKCLR);
+ }
+ return ret;
+}
+
+static int at91_twi_xfer(struct i2c_adapter *adap, struct i2c_msg *msg, int num)
+{
+ struct at91_twi_dev *dev = i2c_get_adapdata(adap);
+ int ret;
+ unsigned int_addr_flag = 0;
+ struct i2c_msg *m_start = msg;
+ bool is_read;
+
+ dev_dbg(&adap->dev, "at91_xfer: processing %d messages:\n", num);
+
+ ret = pm_runtime_get_sync(dev->dev);
+ if (ret < 0)
+ goto out;
+
+ if (num == 2) {
+ int internal_address = 0;
+ int i;
+
+ /* 1st msg is put into the internal address, start with 2nd */
+ m_start = &msg[1];
+ for (i = 0; i < msg->len; ++i) {
+ const unsigned addr = msg->buf[msg->len - 1 - i];
+
+ internal_address |= addr << (8 * i);
+ int_addr_flag += AT91_TWI_IADRSZ_1;
+ }
+ at91_twi_write(dev, AT91_TWI_IADR, internal_address);
+ }
+
+ dev->use_alt_cmd = false;
+ is_read = (m_start->flags & I2C_M_RD);
+ if (dev->pdata->has_alt_cmd) {
+ if (m_start->len > 0 &&
+ m_start->len < AT91_I2C_MAX_ALT_CMD_DATA_SIZE) {
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_ACMEN);
+ at91_twi_write(dev, AT91_TWI_ACR,
+ AT91_TWI_ACR_DATAL(m_start->len) |
+ ((is_read) ? AT91_TWI_ACR_DIR : 0));
+ dev->use_alt_cmd = true;
+ } else {
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_ACMDIS);
+ }
+ }
+
+ at91_twi_write(dev, AT91_TWI_MMR,
+ (m_start->addr << 16) |
+ int_addr_flag |
+ ((!dev->use_alt_cmd && is_read) ? AT91_TWI_MREAD : 0));
+
+ dev->buf_len = m_start->len;
+ dev->buf = m_start->buf;
+ dev->msg = m_start;
+ dev->recv_len_abort = false;
+
+ ret = at91_do_twi_transfer(dev);
+
+ ret = (ret < 0) ? ret : num;
+out:
+ pm_runtime_mark_last_busy(dev->dev);
+ pm_runtime_put_autosuspend(dev->dev);
+
+ return ret;
+}
+
+/*
+ * The hardware can handle at most two messages concatenated by a
+ * repeated start via it's internal address feature.
+ */
+static const struct i2c_adapter_quirks at91_twi_quirks = {
+ .flags = I2C_AQ_COMB | I2C_AQ_COMB_WRITE_FIRST | I2C_AQ_COMB_SAME_ADDR,
+ .max_comb_1st_msg_len = 3,
+};
+
+static u32 at91_twi_func(struct i2c_adapter *adapter)
+{
+ return I2C_FUNC_I2C | I2C_FUNC_SMBUS_EMUL
+ | I2C_FUNC_SMBUS_READ_BLOCK_DATA;
+}
+
+static const struct i2c_algorithm at91_twi_algorithm = {
+ .master_xfer = at91_twi_xfer,
+ .functionality = at91_twi_func,
+};
+
+static int at91_twi_configure_dma(struct at91_twi_dev *dev, u32 phy_addr)
+{
+ int ret = 0;
+ struct dma_slave_config slave_config;
+ struct at91_twi_dma *dma = &dev->dma;
+ enum dma_slave_buswidth addr_width = DMA_SLAVE_BUSWIDTH_1_BYTE;
+
+ /*
+ * The actual width of the access will be chosen in
+ * dmaengine_prep_slave_sg():
+ * for each buffer in the scatter-gather list, if its size is aligned
+ * to addr_width then addr_width accesses will be performed to transfer
+ * the buffer. On the other hand, if the buffer size is not aligned to
+ * addr_width then the buffer is transferred using single byte accesses.
+ * Please refer to the Atmel eXtended DMA controller driver.
+ * When FIFOs are used, the TXRDYM threshold can always be set to
+ * trigger the XDMAC when at least 4 data can be written into the TX
+ * FIFO, even if single byte accesses are performed.
+ * However the RXRDYM threshold must be set to fit the access width,
+ * deduced from buffer length, so the XDMAC is triggered properly to
+ * read data from the RX FIFO.
+ */
+ if (dev->fifo_size)
+ addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
+
+ memset(&slave_config, 0, sizeof(slave_config));
+ slave_config.src_addr = (dma_addr_t)phy_addr + AT91_TWI_RHR;
+ slave_config.src_addr_width = addr_width;
+ slave_config.src_maxburst = 1;
+ slave_config.dst_addr = (dma_addr_t)phy_addr + AT91_TWI_THR;
+ slave_config.dst_addr_width = addr_width;
+ slave_config.dst_maxburst = 1;
+ slave_config.device_fc = false;
+
+ dma->chan_tx = dma_request_slave_channel_reason(dev->dev, "tx");
+ if (IS_ERR(dma->chan_tx)) {
+ ret = PTR_ERR(dma->chan_tx);
+ dma->chan_tx = NULL;
+ goto error;
+ }
+
+ dma->chan_rx = dma_request_slave_channel_reason(dev->dev, "rx");
+ if (IS_ERR(dma->chan_rx)) {
+ ret = PTR_ERR(dma->chan_rx);
+ dma->chan_rx = NULL;
+ goto error;
+ }
+
+ slave_config.direction = DMA_MEM_TO_DEV;
+ if (dmaengine_slave_config(dma->chan_tx, &slave_config)) {
+ dev_err(dev->dev, "failed to configure tx channel\n");
+ ret = -EINVAL;
+ goto error;
+ }
+
+ slave_config.direction = DMA_DEV_TO_MEM;
+ if (dmaengine_slave_config(dma->chan_rx, &slave_config)) {
+ dev_err(dev->dev, "failed to configure rx channel\n");
+ ret = -EINVAL;
+ goto error;
+ }
+
+ sg_init_table(dma->sg, 2);
+ dma->buf_mapped = false;
+ dma->xfer_in_progress = false;
+ dev->use_dma = true;
+
+ dev_info(dev->dev, "using %s (tx) and %s (rx) for DMA transfers\n",
+ dma_chan_name(dma->chan_tx), dma_chan_name(dma->chan_rx));
+
+ return ret;
+
+error:
+ if (ret != -EPROBE_DEFER)
+ dev_info(dev->dev, "can't get DMA channel, continue without DMA support\n");
+ if (dma->chan_rx)
+ dma_release_channel(dma->chan_rx);
+ if (dma->chan_tx)
+ dma_release_channel(dma->chan_tx);
+ return ret;
+}
+
+int at91_twi_probe_master(struct platform_device *pdev,
+ u32 phy_addr, struct at91_twi_dev *dev)
+{
+ int rc;
+
+ init_completion(&dev->cmd_complete);
+
+ rc = devm_request_irq(&pdev->dev, dev->irq, atmel_twi_interrupt, 0,
+ dev_name(dev->dev), dev);
+ if (rc) {
+ dev_err(dev->dev, "Cannot get irq %d: %d\n", dev->irq, rc);
+ return rc;
+ }
+
+ if (dev->dev->of_node) {
+ rc = at91_twi_configure_dma(dev, phy_addr);
+ if (rc == -EPROBE_DEFER)
+ return rc;
+ }
+
+ if (!of_property_read_u32(pdev->dev.of_node, "atmel,fifo-size",
+ &dev->fifo_size)) {
+ dev_info(dev->dev, "Using FIFO (%u data)\n", dev->fifo_size);
+ }
+
+ at91_calc_twi_clock(dev);
+
+ dev->adapter.algo = &at91_twi_algorithm;
+ dev->adapter.quirks = &at91_twi_quirks;
+
+ return 0;
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * i2c slave support for Atmel's AT91 Two-Wire Interface (TWI)
+ *
+ * Copyright (C) 2017 Juergen Fitschen <me@jue.yt>
+ */
+
+#include <linux/err.h>
+#include <linux/i2c.h>
+#include <linux/interrupt.h>
+#include <linux/pm_runtime.h>
+
+#include "i2c-at91.h"
+
+static irqreturn_t atmel_twi_interrupt_slave(int irq, void *dev_id)
+{
+ struct at91_twi_dev *dev = dev_id;
+ const unsigned status = at91_twi_read(dev, AT91_TWI_SR);
+ const unsigned irqstatus = status & at91_twi_read(dev, AT91_TWI_IMR);
+ u8 value;
+
+ if (!irqstatus)
+ return IRQ_NONE;
+
+ /* slave address has been detected on I2C bus */
+ if (irqstatus & AT91_TWI_SVACC) {
+ if (status & AT91_TWI_SVREAD) {
+ i2c_slave_event(dev->slave,
+ I2C_SLAVE_READ_REQUESTED, &value);
+ writeb_relaxed(value, dev->base + AT91_TWI_THR);
+ at91_twi_write(dev, AT91_TWI_IER,
+ AT91_TWI_TXRDY | AT91_TWI_EOSACC);
+ } else {
+ i2c_slave_event(dev->slave,
+ I2C_SLAVE_WRITE_REQUESTED, &value);
+ at91_twi_write(dev, AT91_TWI_IER,
+ AT91_TWI_RXRDY | AT91_TWI_EOSACC);
+ }
+ at91_twi_write(dev, AT91_TWI_IDR, AT91_TWI_SVACC);
+ }
+
+ /* byte transmitted to remote master */
+ if (irqstatus & AT91_TWI_TXRDY) {
+ i2c_slave_event(dev->slave, I2C_SLAVE_READ_PROCESSED, &value);
+ writeb_relaxed(value, dev->base + AT91_TWI_THR);
+ }
+
+ /* byte received from remote master */
+ if (irqstatus & AT91_TWI_RXRDY) {
+ value = readb_relaxed(dev->base + AT91_TWI_RHR);
+ i2c_slave_event(dev->slave, I2C_SLAVE_WRITE_RECEIVED, &value);
+ }
+
+ /* master sent stop */
+ if (irqstatus & AT91_TWI_EOSACC) {
+ at91_twi_write(dev, AT91_TWI_IDR,
+ AT91_TWI_TXRDY | AT91_TWI_RXRDY | AT91_TWI_EOSACC);
+ at91_twi_write(dev, AT91_TWI_IER, AT91_TWI_SVACC);
+ i2c_slave_event(dev->slave, I2C_SLAVE_STOP, &value);
+ }
+
+ return IRQ_HANDLED;
+}
+
+static int at91_reg_slave(struct i2c_client *slave)
+{
+ struct at91_twi_dev *dev = i2c_get_adapdata(slave->adapter);
+
+ if (dev->slave)
+ return -EBUSY;
+
+ if (slave->flags & I2C_CLIENT_TEN)
+ return -EAFNOSUPPORT;
+
+ /* Make sure twi_clk doesn't get turned off! */
+ pm_runtime_get_sync(dev->dev);
+
+ dev->slave = slave;
+ dev->smr = AT91_TWI_SMR_SADR(slave->addr);
+
+ at91_init_twi_bus(dev);
+ at91_twi_write(dev, AT91_TWI_IER, AT91_TWI_SVACC);
+
+ dev_info(dev->dev, "entered slave mode (ADR=%d)\n", slave->addr);
+
+ return 0;
+}
+
+static int at91_unreg_slave(struct i2c_client *slave)
+{
+ struct at91_twi_dev *dev = i2c_get_adapdata(slave->adapter);
+
+ WARN_ON(!dev->slave);
+
+ dev_info(dev->dev, "leaving slave mode\n");
+
+ dev->slave = NULL;
+ dev->smr = 0;
+
+ at91_init_twi_bus(dev);
+
+ pm_runtime_put(dev->dev);
+
+ return 0;
+}
+
+static u32 at91_twi_func(struct i2c_adapter *adapter)
+{
+ return I2C_FUNC_SLAVE | I2C_FUNC_I2C | I2C_FUNC_SMBUS_EMUL
+ | I2C_FUNC_SMBUS_READ_BLOCK_DATA;
+}
+
+static const struct i2c_algorithm at91_twi_algorithm_slave = {
+ .reg_slave = at91_reg_slave,
+ .unreg_slave = at91_unreg_slave,
+ .functionality = at91_twi_func,
+};
+
+int at91_twi_probe_slave(struct platform_device *pdev,
+ u32 phy_addr, struct at91_twi_dev *dev)
+{
+ int rc;
+
+ rc = devm_request_irq(&pdev->dev, dev->irq, atmel_twi_interrupt_slave,
+ 0, dev_name(dev->dev), dev);
+ if (rc) {
+ dev_err(dev->dev, "Cannot get irq %d: %d\n", dev->irq, rc);
+ return rc;
+ }
+
+ dev->adapter.algo = &at91_twi_algorithm_slave;
+
+ return 0;
+}
+
+void at91_init_twi_bus_slave(struct at91_twi_dev *dev)
+{
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_MSDIS);
+ if (dev->slave_detected && dev->smr) {
+ at91_twi_write(dev, AT91_TWI_SMR, dev->smr);
+ at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_SVEN);
+ }
+}
+++ /dev/null
-/*
- * i2c Support for Atmel's AT91 Two-Wire Interface (TWI)
- *
- * Copyright (C) 2011 Weinmann Medical GmbH
- * Author: Nikolaus Voss <n.voss@weinmann.de>
- *
- * Evolved from original work by:
- * Copyright (C) 2004 Rick Bronson
- * Converted to 2.6 by Andrew Victor <andrew@sanpeople.com>
- *
- * Borrowed heavily from original work by:
- * Copyright (C) 2000 Philip Edelbrock <phil@stimpy.netroedge.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#include <linux/clk.h>
-#include <linux/completion.h>
-#include <linux/dma-mapping.h>
-#include <linux/dmaengine.h>
-#include <linux/err.h>
-#include <linux/i2c.h>
-#include <linux/interrupt.h>
-#include <linux/io.h>
-#include <linux/module.h>
-#include <linux/of.h>
-#include <linux/of_device.h>
-#include <linux/platform_device.h>
-#include <linux/slab.h>
-#include <linux/platform_data/dma-atmel.h>
-#include <linux/pm_runtime.h>
-#include <linux/pinctrl/consumer.h>
-
-#define DEFAULT_TWI_CLK_HZ 100000 /* max 400 Kbits/s */
-#define AT91_I2C_TIMEOUT msecs_to_jiffies(100) /* transfer timeout */
-#define AT91_I2C_DMA_THRESHOLD 8 /* enable DMA if transfer size is bigger than this threshold */
-#define AUTOSUSPEND_TIMEOUT 2000
-#define AT91_I2C_MAX_ALT_CMD_DATA_SIZE 256
-
-/* AT91 TWI register definitions */
-#define AT91_TWI_CR 0x0000 /* Control Register */
-#define AT91_TWI_START BIT(0) /* Send a Start Condition */
-#define AT91_TWI_STOP BIT(1) /* Send a Stop Condition */
-#define AT91_TWI_MSEN BIT(2) /* Master Transfer Enable */
-#define AT91_TWI_MSDIS BIT(3) /* Master Transfer Disable */
-#define AT91_TWI_SVEN BIT(4) /* Slave Transfer Enable */
-#define AT91_TWI_SVDIS BIT(5) /* Slave Transfer Disable */
-#define AT91_TWI_QUICK BIT(6) /* SMBus quick command */
-#define AT91_TWI_SWRST BIT(7) /* Software Reset */
-#define AT91_TWI_ACMEN BIT(16) /* Alternative Command Mode Enable */
-#define AT91_TWI_ACMDIS BIT(17) /* Alternative Command Mode Disable */
-#define AT91_TWI_THRCLR BIT(24) /* Transmit Holding Register Clear */
-#define AT91_TWI_RHRCLR BIT(25) /* Receive Holding Register Clear */
-#define AT91_TWI_LOCKCLR BIT(26) /* Lock Clear */
-#define AT91_TWI_FIFOEN BIT(28) /* FIFO Enable */
-#define AT91_TWI_FIFODIS BIT(29) /* FIFO Disable */
-
-#define AT91_TWI_MMR 0x0004 /* Master Mode Register */
-#define AT91_TWI_IADRSZ_1 0x0100 /* Internal Device Address Size */
-#define AT91_TWI_MREAD BIT(12) /* Master Read Direction */
-
-#define AT91_TWI_IADR 0x000c /* Internal Address Register */
-
-#define AT91_TWI_CWGR 0x0010 /* Clock Waveform Generator Reg */
-#define AT91_TWI_CWGR_HOLD_MAX 0x1f
-#define AT91_TWI_CWGR_HOLD(x) (((x) & AT91_TWI_CWGR_HOLD_MAX) << 24)
-
-#define AT91_TWI_SR 0x0020 /* Status Register */
-#define AT91_TWI_TXCOMP BIT(0) /* Transmission Complete */
-#define AT91_TWI_RXRDY BIT(1) /* Receive Holding Register Ready */
-#define AT91_TWI_TXRDY BIT(2) /* Transmit Holding Register Ready */
-#define AT91_TWI_OVRE BIT(6) /* Overrun Error */
-#define AT91_TWI_UNRE BIT(7) /* Underrun Error */
-#define AT91_TWI_NACK BIT(8) /* Not Acknowledged */
-#define AT91_TWI_LOCK BIT(23) /* TWI Lock due to Frame Errors */
-
-#define AT91_TWI_INT_MASK \
- (AT91_TWI_TXCOMP | AT91_TWI_RXRDY | AT91_TWI_TXRDY | AT91_TWI_NACK)
-
-#define AT91_TWI_IER 0x0024 /* Interrupt Enable Register */
-#define AT91_TWI_IDR 0x0028 /* Interrupt Disable Register */
-#define AT91_TWI_IMR 0x002c /* Interrupt Mask Register */
-#define AT91_TWI_RHR 0x0030 /* Receive Holding Register */
-#define AT91_TWI_THR 0x0034 /* Transmit Holding Register */
-
-#define AT91_TWI_ACR 0x0040 /* Alternative Command Register */
-#define AT91_TWI_ACR_DATAL(len) ((len) & 0xff)
-#define AT91_TWI_ACR_DIR BIT(8)
-
-#define AT91_TWI_FMR 0x0050 /* FIFO Mode Register */
-#define AT91_TWI_FMR_TXRDYM(mode) (((mode) & 0x3) << 0)
-#define AT91_TWI_FMR_TXRDYM_MASK (0x3 << 0)
-#define AT91_TWI_FMR_RXRDYM(mode) (((mode) & 0x3) << 4)
-#define AT91_TWI_FMR_RXRDYM_MASK (0x3 << 4)
-#define AT91_TWI_ONE_DATA 0x0
-#define AT91_TWI_TWO_DATA 0x1
-#define AT91_TWI_FOUR_DATA 0x2
-
-#define AT91_TWI_FLR 0x0054 /* FIFO Level Register */
-
-#define AT91_TWI_FSR 0x0060 /* FIFO Status Register */
-#define AT91_TWI_FIER 0x0064 /* FIFO Interrupt Enable Register */
-#define AT91_TWI_FIDR 0x0068 /* FIFO Interrupt Disable Register */
-#define AT91_TWI_FIMR 0x006c /* FIFO Interrupt Mask Register */
-
-#define AT91_TWI_VER 0x00fc /* Version Register */
-
-struct at91_twi_pdata {
- unsigned clk_max_div;
- unsigned clk_offset;
- bool has_unre_flag;
- bool has_alt_cmd;
- bool has_hold_field;
- struct at_dma_slave dma_slave;
-};
-
-struct at91_twi_dma {
- struct dma_chan *chan_rx;
- struct dma_chan *chan_tx;
- struct scatterlist sg[2];
- struct dma_async_tx_descriptor *data_desc;
- enum dma_data_direction direction;
- bool buf_mapped;
- bool xfer_in_progress;
-};
-
-struct at91_twi_dev {
- struct device *dev;
- void __iomem *base;
- struct completion cmd_complete;
- struct clk *clk;
- u8 *buf;
- size_t buf_len;
- struct i2c_msg *msg;
- int irq;
- unsigned imr;
- unsigned transfer_status;
- struct i2c_adapter adapter;
- unsigned twi_cwgr_reg;
- struct at91_twi_pdata *pdata;
- bool use_dma;
- bool use_alt_cmd;
- bool recv_len_abort;
- u32 fifo_size;
- struct at91_twi_dma dma;
-};
-
-static unsigned at91_twi_read(struct at91_twi_dev *dev, unsigned reg)
-{
- return readl_relaxed(dev->base + reg);
-}
-
-static void at91_twi_write(struct at91_twi_dev *dev, unsigned reg, unsigned val)
-{
- writel_relaxed(val, dev->base + reg);
-}
-
-static void at91_disable_twi_interrupts(struct at91_twi_dev *dev)
-{
- at91_twi_write(dev, AT91_TWI_IDR, AT91_TWI_INT_MASK);
-}
-
-static void at91_twi_irq_save(struct at91_twi_dev *dev)
-{
- dev->imr = at91_twi_read(dev, AT91_TWI_IMR) & AT91_TWI_INT_MASK;
- at91_disable_twi_interrupts(dev);
-}
-
-static void at91_twi_irq_restore(struct at91_twi_dev *dev)
-{
- at91_twi_write(dev, AT91_TWI_IER, dev->imr);
-}
-
-static void at91_init_twi_bus(struct at91_twi_dev *dev)
-{
- at91_disable_twi_interrupts(dev);
- at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_SWRST);
- /* FIFO should be enabled immediately after the software reset */
- if (dev->fifo_size)
- at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_FIFOEN);
- at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_MSEN);
- at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_SVDIS);
- at91_twi_write(dev, AT91_TWI_CWGR, dev->twi_cwgr_reg);
-}
-
-/*
- * Calculate symmetric clock as stated in datasheet:
- * twi_clk = F_MAIN / (2 * (cdiv * (1 << ckdiv) + offset))
- */
-static void at91_calc_twi_clock(struct at91_twi_dev *dev, int twi_clk)
-{
- int ckdiv, cdiv, div, hold = 0;
- struct at91_twi_pdata *pdata = dev->pdata;
- int offset = pdata->clk_offset;
- int max_ckdiv = pdata->clk_max_div;
- u32 twd_hold_time_ns = 0;
-
- div = max(0, (int)DIV_ROUND_UP(clk_get_rate(dev->clk),
- 2 * twi_clk) - offset);
- ckdiv = fls(div >> 8);
- cdiv = div >> ckdiv;
-
- if (ckdiv > max_ckdiv) {
- dev_warn(dev->dev, "%d exceeds ckdiv max value which is %d.\n",
- ckdiv, max_ckdiv);
- ckdiv = max_ckdiv;
- cdiv = 255;
- }
-
- if (pdata->has_hold_field) {
- of_property_read_u32(dev->dev->of_node, "i2c-sda-hold-time-ns",
- &twd_hold_time_ns);
-
- /*
- * hold time = HOLD + 3 x T_peripheral_clock
- * Use clk rate in kHz to prevent overflows when computing
- * hold.
- */
- hold = DIV_ROUND_UP(twd_hold_time_ns
- * (clk_get_rate(dev->clk) / 1000), 1000000);
- hold -= 3;
- if (hold < 0)
- hold = 0;
- if (hold > AT91_TWI_CWGR_HOLD_MAX) {
- dev_warn(dev->dev,
- "HOLD field set to its maximum value (%d instead of %d)\n",
- AT91_TWI_CWGR_HOLD_MAX, hold);
- hold = AT91_TWI_CWGR_HOLD_MAX;
- }
- }
-
- dev->twi_cwgr_reg = (ckdiv << 16) | (cdiv << 8) | cdiv
- | AT91_TWI_CWGR_HOLD(hold);
-
- dev_dbg(dev->dev, "cdiv %d ckdiv %d hold %d (%d ns)\n",
- cdiv, ckdiv, hold, twd_hold_time_ns);
-}
-
-static void at91_twi_dma_cleanup(struct at91_twi_dev *dev)
-{
- struct at91_twi_dma *dma = &dev->dma;
-
- at91_twi_irq_save(dev);
-
- if (dma->xfer_in_progress) {
- if (dma->direction == DMA_FROM_DEVICE)
- dmaengine_terminate_all(dma->chan_rx);
- else
- dmaengine_terminate_all(dma->chan_tx);
- dma->xfer_in_progress = false;
- }
- if (dma->buf_mapped) {
- dma_unmap_single(dev->dev, sg_dma_address(&dma->sg[0]),
- dev->buf_len, dma->direction);
- dma->buf_mapped = false;
- }
-
- at91_twi_irq_restore(dev);
-}
-
-static void at91_twi_write_next_byte(struct at91_twi_dev *dev)
-{
- if (!dev->buf_len)
- return;
-
- /* 8bit write works with and without FIFO */
- writeb_relaxed(*dev->buf, dev->base + AT91_TWI_THR);
-
- /* send stop when last byte has been written */
- if (--dev->buf_len == 0)
- if (!dev->use_alt_cmd)
- at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_STOP);
-
- dev_dbg(dev->dev, "wrote 0x%x, to go %zu\n", *dev->buf, dev->buf_len);
-
- ++dev->buf;
-}
-
-static void at91_twi_write_data_dma_callback(void *data)
-{
- struct at91_twi_dev *dev = (struct at91_twi_dev *)data;
-
- dma_unmap_single(dev->dev, sg_dma_address(&dev->dma.sg[0]),
- dev->buf_len, DMA_TO_DEVICE);
-
- /*
- * When this callback is called, THR/TX FIFO is likely not to be empty
- * yet. So we have to wait for TXCOMP or NACK bits to be set into the
- * Status Register to be sure that the STOP bit has been sent and the
- * transfer is completed. The NACK interrupt has already been enabled,
- * we just have to enable TXCOMP one.
- */
- at91_twi_write(dev, AT91_TWI_IER, AT91_TWI_TXCOMP);
- if (!dev->use_alt_cmd)
- at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_STOP);
-}
-
-static void at91_twi_write_data_dma(struct at91_twi_dev *dev)
-{
- dma_addr_t dma_addr;
- struct dma_async_tx_descriptor *txdesc;
- struct at91_twi_dma *dma = &dev->dma;
- struct dma_chan *chan_tx = dma->chan_tx;
- unsigned int sg_len = 1;
-
- if (!dev->buf_len)
- return;
-
- dma->direction = DMA_TO_DEVICE;
-
- at91_twi_irq_save(dev);
- dma_addr = dma_map_single(dev->dev, dev->buf, dev->buf_len,
- DMA_TO_DEVICE);
- if (dma_mapping_error(dev->dev, dma_addr)) {
- dev_err(dev->dev, "dma map failed\n");
- return;
- }
- dma->buf_mapped = true;
- at91_twi_irq_restore(dev);
-
- if (dev->fifo_size) {
- size_t part1_len, part2_len;
- struct scatterlist *sg;
- unsigned fifo_mr;
-
- sg_len = 0;
-
- part1_len = dev->buf_len & ~0x3;
- if (part1_len) {
- sg = &dma->sg[sg_len++];
- sg_dma_len(sg) = part1_len;
- sg_dma_address(sg) = dma_addr;
- }
-
- part2_len = dev->buf_len & 0x3;
- if (part2_len) {
- sg = &dma->sg[sg_len++];
- sg_dma_len(sg) = part2_len;
- sg_dma_address(sg) = dma_addr + part1_len;
- }
-
- /*
- * DMA controller is triggered when at least 4 data can be
- * written into the TX FIFO
- */
- fifo_mr = at91_twi_read(dev, AT91_TWI_FMR);
- fifo_mr &= ~AT91_TWI_FMR_TXRDYM_MASK;
- fifo_mr |= AT91_TWI_FMR_TXRDYM(AT91_TWI_FOUR_DATA);
- at91_twi_write(dev, AT91_TWI_FMR, fifo_mr);
- } else {
- sg_dma_len(&dma->sg[0]) = dev->buf_len;
- sg_dma_address(&dma->sg[0]) = dma_addr;
- }
-
- txdesc = dmaengine_prep_slave_sg(chan_tx, dma->sg, sg_len,
- DMA_MEM_TO_DEV,
- DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
- if (!txdesc) {
- dev_err(dev->dev, "dma prep slave sg failed\n");
- goto error;
- }
-
- txdesc->callback = at91_twi_write_data_dma_callback;
- txdesc->callback_param = dev;
-
- dma->xfer_in_progress = true;
- dmaengine_submit(txdesc);
- dma_async_issue_pending(chan_tx);
-
- return;
-
-error:
- at91_twi_dma_cleanup(dev);
-}
-
-static void at91_twi_read_next_byte(struct at91_twi_dev *dev)
-{
- /*
- * If we are in this case, it means there is garbage data in RHR, so
- * delete them.
- */
- if (!dev->buf_len) {
- at91_twi_read(dev, AT91_TWI_RHR);
- return;
- }
-
- /* 8bit read works with and without FIFO */
- *dev->buf = readb_relaxed(dev->base + AT91_TWI_RHR);
- --dev->buf_len;
-
- /* return if aborting, we only needed to read RHR to clear RXRDY*/
- if (dev->recv_len_abort)
- return;
-
- /* handle I2C_SMBUS_BLOCK_DATA */
- if (unlikely(dev->msg->flags & I2C_M_RECV_LEN)) {
- /* ensure length byte is a valid value */
- if (*dev->buf <= I2C_SMBUS_BLOCK_MAX && *dev->buf > 0) {
- dev->msg->flags &= ~I2C_M_RECV_LEN;
- dev->buf_len += *dev->buf;
- dev->msg->len = dev->buf_len + 1;
- dev_dbg(dev->dev, "received block length %zu\n",
- dev->buf_len);
- } else {
- /* abort and send the stop by reading one more byte */
- dev->recv_len_abort = true;
- dev->buf_len = 1;
- }
- }
-
- /* send stop if second but last byte has been read */
- if (!dev->use_alt_cmd && dev->buf_len == 1)
- at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_STOP);
-
- dev_dbg(dev->dev, "read 0x%x, to go %zu\n", *dev->buf, dev->buf_len);
-
- ++dev->buf;
-}
-
-static void at91_twi_read_data_dma_callback(void *data)
-{
- struct at91_twi_dev *dev = (struct at91_twi_dev *)data;
- unsigned ier = AT91_TWI_TXCOMP;
-
- dma_unmap_single(dev->dev, sg_dma_address(&dev->dma.sg[0]),
- dev->buf_len, DMA_FROM_DEVICE);
-
- if (!dev->use_alt_cmd) {
- /* The last two bytes have to be read without using dma */
- dev->buf += dev->buf_len - 2;
- dev->buf_len = 2;
- ier |= AT91_TWI_RXRDY;
- }
- at91_twi_write(dev, AT91_TWI_IER, ier);
-}
-
-static void at91_twi_read_data_dma(struct at91_twi_dev *dev)
-{
- dma_addr_t dma_addr;
- struct dma_async_tx_descriptor *rxdesc;
- struct at91_twi_dma *dma = &dev->dma;
- struct dma_chan *chan_rx = dma->chan_rx;
- size_t buf_len;
-
- buf_len = (dev->use_alt_cmd) ? dev->buf_len : dev->buf_len - 2;
- dma->direction = DMA_FROM_DEVICE;
-
- /* Keep in mind that we won't use dma to read the last two bytes */
- at91_twi_irq_save(dev);
- dma_addr = dma_map_single(dev->dev, dev->buf, buf_len, DMA_FROM_DEVICE);
- if (dma_mapping_error(dev->dev, dma_addr)) {
- dev_err(dev->dev, "dma map failed\n");
- return;
- }
- dma->buf_mapped = true;
- at91_twi_irq_restore(dev);
-
- if (dev->fifo_size && IS_ALIGNED(buf_len, 4)) {
- unsigned fifo_mr;
-
- /*
- * DMA controller is triggered when at least 4 data can be
- * read from the RX FIFO
- */
- fifo_mr = at91_twi_read(dev, AT91_TWI_FMR);
- fifo_mr &= ~AT91_TWI_FMR_RXRDYM_MASK;
- fifo_mr |= AT91_TWI_FMR_RXRDYM(AT91_TWI_FOUR_DATA);
- at91_twi_write(dev, AT91_TWI_FMR, fifo_mr);
- }
-
- sg_dma_len(&dma->sg[0]) = buf_len;
- sg_dma_address(&dma->sg[0]) = dma_addr;
-
- rxdesc = dmaengine_prep_slave_sg(chan_rx, dma->sg, 1, DMA_DEV_TO_MEM,
- DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
- if (!rxdesc) {
- dev_err(dev->dev, "dma prep slave sg failed\n");
- goto error;
- }
-
- rxdesc->callback = at91_twi_read_data_dma_callback;
- rxdesc->callback_param = dev;
-
- dma->xfer_in_progress = true;
- dmaengine_submit(rxdesc);
- dma_async_issue_pending(dma->chan_rx);
-
- return;
-
-error:
- at91_twi_dma_cleanup(dev);
-}
-
-static irqreturn_t atmel_twi_interrupt(int irq, void *dev_id)
-{
- struct at91_twi_dev *dev = dev_id;
- const unsigned status = at91_twi_read(dev, AT91_TWI_SR);
- const unsigned irqstatus = status & at91_twi_read(dev, AT91_TWI_IMR);
-
- if (!irqstatus)
- return IRQ_NONE;
- /*
- * In reception, the behavior of the twi device (before sama5d2) is
- * weird. There is some magic about RXRDY flag! When a data has been
- * almost received, the reception of a new one is anticipated if there
- * is no stop command to send. That is the reason why ask for sending
- * the stop command not on the last data but on the second last one.
- *
- * Unfortunately, we could still have the RXRDY flag set even if the
- * transfer is done and we have read the last data. It might happen
- * when the i2c slave device sends too quickly data after receiving the
- * ack from the master. The data has been almost received before having
- * the order to send stop. In this case, sending the stop command could
- * cause a RXRDY interrupt with a TXCOMP one. It is better to manage
- * the RXRDY interrupt first in order to not keep garbage data in the
- * Receive Holding Register for the next transfer.
- */
- if (irqstatus & AT91_TWI_RXRDY) {
- /*
- * Read all available bytes at once by polling RXRDY usable w/
- * and w/o FIFO. With FIFO enabled we could also read RXFL and
- * avoid polling RXRDY.
- */
- do {
- at91_twi_read_next_byte(dev);
- } while (at91_twi_read(dev, AT91_TWI_SR) & AT91_TWI_RXRDY);
- }
-
- /*
- * When a NACK condition is detected, the I2C controller sets the NACK,
- * TXCOMP and TXRDY bits all together in the Status Register (SR).
- *
- * 1 - Handling NACK errors with CPU write transfer.
- *
- * In such case, we should not write the next byte into the Transmit
- * Holding Register (THR) otherwise the I2C controller would start a new
- * transfer and the I2C slave is likely to reply by another NACK.
- *
- * 2 - Handling NACK errors with DMA write transfer.
- *
- * By setting the TXRDY bit in the SR, the I2C controller also triggers
- * the DMA controller to write the next data into the THR. Then the
- * result depends on the hardware version of the I2C controller.
- *
- * 2a - Without support of the Alternative Command mode.
- *
- * This is the worst case: the DMA controller is triggered to write the
- * next data into the THR, hence starting a new transfer: the I2C slave
- * is likely to reply by another NACK.
- * Concurrently, this interrupt handler is likely to be called to manage
- * the first NACK before the I2C controller detects the second NACK and
- * sets once again the NACK bit into the SR.
- * When handling the first NACK, this interrupt handler disables the I2C
- * controller interruptions, especially the NACK interrupt.
- * Hence, the NACK bit is pending into the SR. This is why we should
- * read the SR to clear all pending interrupts at the beginning of
- * at91_do_twi_transfer() before actually starting a new transfer.
- *
- * 2b - With support of the Alternative Command mode.
- *
- * When a NACK condition is detected, the I2C controller also locks the
- * THR (and sets the LOCK bit in the SR): even though the DMA controller
- * is triggered by the TXRDY bit to write the next data into the THR,
- * this data actually won't go on the I2C bus hence a second NACK is not
- * generated.
- */
- if (irqstatus & (AT91_TWI_TXCOMP | AT91_TWI_NACK)) {
- at91_disable_twi_interrupts(dev);
- complete(&dev->cmd_complete);
- } else if (irqstatus & AT91_TWI_TXRDY) {
- at91_twi_write_next_byte(dev);
- }
-
- /* catch error flags */
- dev->transfer_status |= status;
-
- return IRQ_HANDLED;
-}
-
-static int at91_do_twi_transfer(struct at91_twi_dev *dev)
-{
- int ret;
- unsigned long time_left;
- bool has_unre_flag = dev->pdata->has_unre_flag;
- bool has_alt_cmd = dev->pdata->has_alt_cmd;
-
- /*
- * WARNING: the TXCOMP bit in the Status Register is NOT a clear on
- * read flag but shows the state of the transmission at the time the
- * Status Register is read. According to the programmer datasheet,
- * TXCOMP is set when both holding register and internal shifter are
- * empty and STOP condition has been sent.
- * Consequently, we should enable NACK interrupt rather than TXCOMP to
- * detect transmission failure.
- * Indeed let's take the case of an i2c write command using DMA.
- * Whenever the slave doesn't acknowledge a byte, the LOCK, NACK and
- * TXCOMP bits are set together into the Status Register.
- * LOCK is a clear on write bit, which is set to prevent the DMA
- * controller from sending new data on the i2c bus after a NACK
- * condition has happened. Once locked, this i2c peripheral stops
- * triggering the DMA controller for new data but it is more than
- * likely that a new DMA transaction is already in progress, writing
- * into the Transmit Holding Register. Since the peripheral is locked,
- * these new data won't be sent to the i2c bus but they will remain
- * into the Transmit Holding Register, so TXCOMP bit is cleared.
- * Then when the interrupt handler is called, the Status Register is
- * read: the TXCOMP bit is clear but NACK bit is still set. The driver
- * manage the error properly, without waiting for timeout.
- * This case can be reproduced easyly when writing into an at24 eeprom.
- *
- * Besides, the TXCOMP bit is already set before the i2c transaction
- * has been started. For read transactions, this bit is cleared when
- * writing the START bit into the Control Register. So the
- * corresponding interrupt can safely be enabled just after.
- * However for write transactions managed by the CPU, we first write
- * into THR, so TXCOMP is cleared. Then we can safely enable TXCOMP
- * interrupt. If TXCOMP interrupt were enabled before writing into THR,
- * the interrupt handler would be called immediately and the i2c command
- * would be reported as completed.
- * Also when a write transaction is managed by the DMA controller,
- * enabling the TXCOMP interrupt in this function may lead to a race
- * condition since we don't know whether the TXCOMP interrupt is enabled
- * before or after the DMA has started to write into THR. So the TXCOMP
- * interrupt is enabled later by at91_twi_write_data_dma_callback().
- * Immediately after in that DMA callback, if the alternative command
- * mode is not used, we still need to send the STOP condition manually
- * writing the corresponding bit into the Control Register.
- */
-
- dev_dbg(dev->dev, "transfer: %s %zu bytes.\n",
- (dev->msg->flags & I2C_M_RD) ? "read" : "write", dev->buf_len);
-
- reinit_completion(&dev->cmd_complete);
- dev->transfer_status = 0;
-
- /* Clear pending interrupts, such as NACK. */
- at91_twi_read(dev, AT91_TWI_SR);
-
- if (dev->fifo_size) {
- unsigned fifo_mr = at91_twi_read(dev, AT91_TWI_FMR);
-
- /* Reset FIFO mode register */
- fifo_mr &= ~(AT91_TWI_FMR_TXRDYM_MASK |
- AT91_TWI_FMR_RXRDYM_MASK);
- fifo_mr |= AT91_TWI_FMR_TXRDYM(AT91_TWI_ONE_DATA);
- fifo_mr |= AT91_TWI_FMR_RXRDYM(AT91_TWI_ONE_DATA);
- at91_twi_write(dev, AT91_TWI_FMR, fifo_mr);
-
- /* Flush FIFOs */
- at91_twi_write(dev, AT91_TWI_CR,
- AT91_TWI_THRCLR | AT91_TWI_RHRCLR);
- }
-
- if (!dev->buf_len) {
- at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_QUICK);
- at91_twi_write(dev, AT91_TWI_IER, AT91_TWI_TXCOMP);
- } else if (dev->msg->flags & I2C_M_RD) {
- unsigned start_flags = AT91_TWI_START;
-
- /* if only one byte is to be read, immediately stop transfer */
- if (!dev->use_alt_cmd && dev->buf_len <= 1 &&
- !(dev->msg->flags & I2C_M_RECV_LEN))
- start_flags |= AT91_TWI_STOP;
- at91_twi_write(dev, AT91_TWI_CR, start_flags);
- /*
- * When using dma without alternative command mode, the last
- * byte has to be read manually in order to not send the stop
- * command too late and then to receive extra data.
- * In practice, there are some issues if you use the dma to
- * read n-1 bytes because of latency.
- * Reading n-2 bytes with dma and the two last ones manually
- * seems to be the best solution.
- */
- if (dev->use_dma && (dev->buf_len > AT91_I2C_DMA_THRESHOLD)) {
- at91_twi_write(dev, AT91_TWI_IER, AT91_TWI_NACK);
- at91_twi_read_data_dma(dev);
- } else {
- at91_twi_write(dev, AT91_TWI_IER,
- AT91_TWI_TXCOMP |
- AT91_TWI_NACK |
- AT91_TWI_RXRDY);
- }
- } else {
- if (dev->use_dma && (dev->buf_len > AT91_I2C_DMA_THRESHOLD)) {
- at91_twi_write(dev, AT91_TWI_IER, AT91_TWI_NACK);
- at91_twi_write_data_dma(dev);
- } else {
- at91_twi_write_next_byte(dev);
- at91_twi_write(dev, AT91_TWI_IER,
- AT91_TWI_TXCOMP |
- AT91_TWI_NACK |
- AT91_TWI_TXRDY);
- }
- }
-
- time_left = wait_for_completion_timeout(&dev->cmd_complete,
- dev->adapter.timeout);
- if (time_left == 0) {
- dev->transfer_status |= at91_twi_read(dev, AT91_TWI_SR);
- dev_err(dev->dev, "controller timed out\n");
- at91_init_twi_bus(dev);
- ret = -ETIMEDOUT;
- goto error;
- }
- if (dev->transfer_status & AT91_TWI_NACK) {
- dev_dbg(dev->dev, "received nack\n");
- ret = -EREMOTEIO;
- goto error;
- }
- if (dev->transfer_status & AT91_TWI_OVRE) {
- dev_err(dev->dev, "overrun while reading\n");
- ret = -EIO;
- goto error;
- }
- if (has_unre_flag && dev->transfer_status & AT91_TWI_UNRE) {
- dev_err(dev->dev, "underrun while writing\n");
- ret = -EIO;
- goto error;
- }
- if ((has_alt_cmd || dev->fifo_size) &&
- (dev->transfer_status & AT91_TWI_LOCK)) {
- dev_err(dev->dev, "tx locked\n");
- ret = -EIO;
- goto error;
- }
- if (dev->recv_len_abort) {
- dev_err(dev->dev, "invalid smbus block length recvd\n");
- ret = -EPROTO;
- goto error;
- }
-
- dev_dbg(dev->dev, "transfer complete\n");
-
- return 0;
-
-error:
- /* first stop DMA transfer if still in progress */
- at91_twi_dma_cleanup(dev);
- /* then flush THR/FIFO and unlock TX if locked */
- if ((has_alt_cmd || dev->fifo_size) &&
- (dev->transfer_status & AT91_TWI_LOCK)) {
- dev_dbg(dev->dev, "unlock tx\n");
- at91_twi_write(dev, AT91_TWI_CR,
- AT91_TWI_THRCLR | AT91_TWI_LOCKCLR);
- }
- return ret;
-}
-
-static int at91_twi_xfer(struct i2c_adapter *adap, struct i2c_msg *msg, int num)
-{
- struct at91_twi_dev *dev = i2c_get_adapdata(adap);
- int ret;
- unsigned int_addr_flag = 0;
- struct i2c_msg *m_start = msg;
- bool is_read;
-
- dev_dbg(&adap->dev, "at91_xfer: processing %d messages:\n", num);
-
- ret = pm_runtime_get_sync(dev->dev);
- if (ret < 0)
- goto out;
-
- if (num == 2) {
- int internal_address = 0;
- int i;
-
- /* 1st msg is put into the internal address, start with 2nd */
- m_start = &msg[1];
- for (i = 0; i < msg->len; ++i) {
- const unsigned addr = msg->buf[msg->len - 1 - i];
-
- internal_address |= addr << (8 * i);
- int_addr_flag += AT91_TWI_IADRSZ_1;
- }
- at91_twi_write(dev, AT91_TWI_IADR, internal_address);
- }
-
- dev->use_alt_cmd = false;
- is_read = (m_start->flags & I2C_M_RD);
- if (dev->pdata->has_alt_cmd) {
- if (m_start->len > 0 &&
- m_start->len < AT91_I2C_MAX_ALT_CMD_DATA_SIZE) {
- at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_ACMEN);
- at91_twi_write(dev, AT91_TWI_ACR,
- AT91_TWI_ACR_DATAL(m_start->len) |
- ((is_read) ? AT91_TWI_ACR_DIR : 0));
- dev->use_alt_cmd = true;
- } else {
- at91_twi_write(dev, AT91_TWI_CR, AT91_TWI_ACMDIS);
- }
- }
-
- at91_twi_write(dev, AT91_TWI_MMR,
- (m_start->addr << 16) |
- int_addr_flag |
- ((!dev->use_alt_cmd && is_read) ? AT91_TWI_MREAD : 0));
-
- dev->buf_len = m_start->len;
- dev->buf = m_start->buf;
- dev->msg = m_start;
- dev->recv_len_abort = false;
-
- ret = at91_do_twi_transfer(dev);
-
- ret = (ret < 0) ? ret : num;
-out:
- pm_runtime_mark_last_busy(dev->dev);
- pm_runtime_put_autosuspend(dev->dev);
-
- return ret;
-}
-
-/*
- * The hardware can handle at most two messages concatenated by a
- * repeated start via it's internal address feature.
- */
-static const struct i2c_adapter_quirks at91_twi_quirks = {
- .flags = I2C_AQ_COMB | I2C_AQ_COMB_WRITE_FIRST | I2C_AQ_COMB_SAME_ADDR,
- .max_comb_1st_msg_len = 3,
-};
-
-static u32 at91_twi_func(struct i2c_adapter *adapter)
-{
- return I2C_FUNC_I2C | I2C_FUNC_SMBUS_EMUL
- | I2C_FUNC_SMBUS_READ_BLOCK_DATA;
-}
-
-static const struct i2c_algorithm at91_twi_algorithm = {
- .master_xfer = at91_twi_xfer,
- .functionality = at91_twi_func,
-};
-
-static struct at91_twi_pdata at91rm9200_config = {
- .clk_max_div = 5,
- .clk_offset = 3,
- .has_unre_flag = true,
- .has_alt_cmd = false,
- .has_hold_field = false,
-};
-
-static struct at91_twi_pdata at91sam9261_config = {
- .clk_max_div = 5,
- .clk_offset = 4,
- .has_unre_flag = false,
- .has_alt_cmd = false,
- .has_hold_field = false,
-};
-
-static struct at91_twi_pdata at91sam9260_config = {
- .clk_max_div = 7,
- .clk_offset = 4,
- .has_unre_flag = false,
- .has_alt_cmd = false,
- .has_hold_field = false,
-};
-
-static struct at91_twi_pdata at91sam9g20_config = {
- .clk_max_div = 7,
- .clk_offset = 4,
- .has_unre_flag = false,
- .has_alt_cmd = false,
- .has_hold_field = false,
-};
-
-static struct at91_twi_pdata at91sam9g10_config = {
- .clk_max_div = 7,
- .clk_offset = 4,
- .has_unre_flag = false,
- .has_alt_cmd = false,
- .has_hold_field = false,
-};
-
-static const struct platform_device_id at91_twi_devtypes[] = {
- {
- .name = "i2c-at91rm9200",
- .driver_data = (unsigned long) &at91rm9200_config,
- }, {
- .name = "i2c-at91sam9261",
- .driver_data = (unsigned long) &at91sam9261_config,
- }, {
- .name = "i2c-at91sam9260",
- .driver_data = (unsigned long) &at91sam9260_config,
- }, {
- .name = "i2c-at91sam9g20",
- .driver_data = (unsigned long) &at91sam9g20_config,
- }, {
- .name = "i2c-at91sam9g10",
- .driver_data = (unsigned long) &at91sam9g10_config,
- }, {
- /* sentinel */
- }
-};
-
-#if defined(CONFIG_OF)
-static struct at91_twi_pdata at91sam9x5_config = {
- .clk_max_div = 7,
- .clk_offset = 4,
- .has_unre_flag = false,
- .has_alt_cmd = false,
- .has_hold_field = false,
-};
-
-static struct at91_twi_pdata sama5d4_config = {
- .clk_max_div = 7,
- .clk_offset = 4,
- .has_unre_flag = false,
- .has_alt_cmd = false,
- .has_hold_field = true,
-};
-
-static struct at91_twi_pdata sama5d2_config = {
- .clk_max_div = 7,
- .clk_offset = 4,
- .has_unre_flag = true,
- .has_alt_cmd = true,
- .has_hold_field = true,
-};
-
-static const struct of_device_id atmel_twi_dt_ids[] = {
- {
- .compatible = "atmel,at91rm9200-i2c",
- .data = &at91rm9200_config,
- } , {
- .compatible = "atmel,at91sam9260-i2c",
- .data = &at91sam9260_config,
- } , {
- .compatible = "atmel,at91sam9261-i2c",
- .data = &at91sam9261_config,
- } , {
- .compatible = "atmel,at91sam9g20-i2c",
- .data = &at91sam9g20_config,
- } , {
- .compatible = "atmel,at91sam9g10-i2c",
- .data = &at91sam9g10_config,
- }, {
- .compatible = "atmel,at91sam9x5-i2c",
- .data = &at91sam9x5_config,
- }, {
- .compatible = "atmel,sama5d4-i2c",
- .data = &sama5d4_config,
- }, {
- .compatible = "atmel,sama5d2-i2c",
- .data = &sama5d2_config,
- }, {
- /* sentinel */
- }
-};
-MODULE_DEVICE_TABLE(of, atmel_twi_dt_ids);
-#endif
-
-static int at91_twi_configure_dma(struct at91_twi_dev *dev, u32 phy_addr)
-{
- int ret = 0;
- struct dma_slave_config slave_config;
- struct at91_twi_dma *dma = &dev->dma;
- enum dma_slave_buswidth addr_width = DMA_SLAVE_BUSWIDTH_1_BYTE;
-
- /*
- * The actual width of the access will be chosen in
- * dmaengine_prep_slave_sg():
- * for each buffer in the scatter-gather list, if its size is aligned
- * to addr_width then addr_width accesses will be performed to transfer
- * the buffer. On the other hand, if the buffer size is not aligned to
- * addr_width then the buffer is transferred using single byte accesses.
- * Please refer to the Atmel eXtended DMA controller driver.
- * When FIFOs are used, the TXRDYM threshold can always be set to
- * trigger the XDMAC when at least 4 data can be written into the TX
- * FIFO, even if single byte accesses are performed.
- * However the RXRDYM threshold must be set to fit the access width,
- * deduced from buffer length, so the XDMAC is triggered properly to
- * read data from the RX FIFO.
- */
- if (dev->fifo_size)
- addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
-
- memset(&slave_config, 0, sizeof(slave_config));
- slave_config.src_addr = (dma_addr_t)phy_addr + AT91_TWI_RHR;
- slave_config.src_addr_width = addr_width;
- slave_config.src_maxburst = 1;
- slave_config.dst_addr = (dma_addr_t)phy_addr + AT91_TWI_THR;
- slave_config.dst_addr_width = addr_width;
- slave_config.dst_maxburst = 1;
- slave_config.device_fc = false;
-
- dma->chan_tx = dma_request_slave_channel_reason(dev->dev, "tx");
- if (IS_ERR(dma->chan_tx)) {
- ret = PTR_ERR(dma->chan_tx);
- dma->chan_tx = NULL;
- goto error;
- }
-
- dma->chan_rx = dma_request_slave_channel_reason(dev->dev, "rx");
- if (IS_ERR(dma->chan_rx)) {
- ret = PTR_ERR(dma->chan_rx);
- dma->chan_rx = NULL;
- goto error;
- }
-
- slave_config.direction = DMA_MEM_TO_DEV;
- if (dmaengine_slave_config(dma->chan_tx, &slave_config)) {
- dev_err(dev->dev, "failed to configure tx channel\n");
- ret = -EINVAL;
- goto error;
- }
-
- slave_config.direction = DMA_DEV_TO_MEM;
- if (dmaengine_slave_config(dma->chan_rx, &slave_config)) {
- dev_err(dev->dev, "failed to configure rx channel\n");
- ret = -EINVAL;
- goto error;
- }
-
- sg_init_table(dma->sg, 2);
- dma->buf_mapped = false;
- dma->xfer_in_progress = false;
- dev->use_dma = true;
-
- dev_info(dev->dev, "using %s (tx) and %s (rx) for DMA transfers\n",
- dma_chan_name(dma->chan_tx), dma_chan_name(dma->chan_rx));
-
- return ret;
-
-error:
- if (ret != -EPROBE_DEFER)
- dev_info(dev->dev, "can't get DMA channel, continue without DMA support\n");
- if (dma->chan_rx)
- dma_release_channel(dma->chan_rx);
- if (dma->chan_tx)
- dma_release_channel(dma->chan_tx);
- return ret;
-}
-
-static struct at91_twi_pdata *at91_twi_get_driver_data(
- struct platform_device *pdev)
-{
- if (pdev->dev.of_node) {
- const struct of_device_id *match;
- match = of_match_node(atmel_twi_dt_ids, pdev->dev.of_node);
- if (!match)
- return NULL;
- return (struct at91_twi_pdata *)match->data;
- }
- return (struct at91_twi_pdata *) platform_get_device_id(pdev)->driver_data;
-}
-
-static int at91_twi_probe(struct platform_device *pdev)
-{
- struct at91_twi_dev *dev;
- struct resource *mem;
- int rc;
- u32 phy_addr;
- u32 bus_clk_rate;
-
- dev = devm_kzalloc(&pdev->dev, sizeof(*dev), GFP_KERNEL);
- if (!dev)
- return -ENOMEM;
- init_completion(&dev->cmd_complete);
- dev->dev = &pdev->dev;
-
- mem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!mem)
- return -ENODEV;
- phy_addr = mem->start;
-
- dev->pdata = at91_twi_get_driver_data(pdev);
- if (!dev->pdata)
- return -ENODEV;
-
- dev->base = devm_ioremap_resource(&pdev->dev, mem);
- if (IS_ERR(dev->base))
- return PTR_ERR(dev->base);
-
- dev->irq = platform_get_irq(pdev, 0);
- if (dev->irq < 0)
- return dev->irq;
-
- rc = devm_request_irq(&pdev->dev, dev->irq, atmel_twi_interrupt, 0,
- dev_name(dev->dev), dev);
- if (rc) {
- dev_err(dev->dev, "Cannot get irq %d: %d\n", dev->irq, rc);
- return rc;
- }
-
- platform_set_drvdata(pdev, dev);
-
- dev->clk = devm_clk_get(dev->dev, NULL);
- if (IS_ERR(dev->clk)) {
- dev_err(dev->dev, "no clock defined\n");
- return -ENODEV;
- }
- rc = clk_prepare_enable(dev->clk);
- if (rc)
- return rc;
-
- if (dev->dev->of_node) {
- rc = at91_twi_configure_dma(dev, phy_addr);
- if (rc == -EPROBE_DEFER) {
- clk_disable_unprepare(dev->clk);
- return rc;
- }
- }
-
- if (!of_property_read_u32(pdev->dev.of_node, "atmel,fifo-size",
- &dev->fifo_size)) {
- dev_info(dev->dev, "Using FIFO (%u data)\n", dev->fifo_size);
- }
-
- rc = of_property_read_u32(dev->dev->of_node, "clock-frequency",
- &bus_clk_rate);
- if (rc)
- bus_clk_rate = DEFAULT_TWI_CLK_HZ;
-
- at91_calc_twi_clock(dev, bus_clk_rate);
- at91_init_twi_bus(dev);
-
- snprintf(dev->adapter.name, sizeof(dev->adapter.name), "AT91");
- i2c_set_adapdata(&dev->adapter, dev);
- dev->adapter.owner = THIS_MODULE;
- dev->adapter.class = I2C_CLASS_DEPRECATED;
- dev->adapter.algo = &at91_twi_algorithm;
- dev->adapter.quirks = &at91_twi_quirks;
- dev->adapter.dev.parent = dev->dev;
- dev->adapter.nr = pdev->id;
- dev->adapter.timeout = AT91_I2C_TIMEOUT;
- dev->adapter.dev.of_node = pdev->dev.of_node;
-
- pm_runtime_set_autosuspend_delay(dev->dev, AUTOSUSPEND_TIMEOUT);
- pm_runtime_use_autosuspend(dev->dev);
- pm_runtime_set_active(dev->dev);
- pm_runtime_enable(dev->dev);
-
- rc = i2c_add_numbered_adapter(&dev->adapter);
- if (rc) {
- clk_disable_unprepare(dev->clk);
-
- pm_runtime_disable(dev->dev);
- pm_runtime_set_suspended(dev->dev);
-
- return rc;
- }
-
- dev_info(dev->dev, "AT91 i2c bus driver (hw version: %#x).\n",
- at91_twi_read(dev, AT91_TWI_VER));
- return 0;
-}
-
-static int at91_twi_remove(struct platform_device *pdev)
-{
- struct at91_twi_dev *dev = platform_get_drvdata(pdev);
-
- i2c_del_adapter(&dev->adapter);
- clk_disable_unprepare(dev->clk);
-
- pm_runtime_disable(dev->dev);
- pm_runtime_set_suspended(dev->dev);
-
- return 0;
-}
-
-#ifdef CONFIG_PM
-
-static int at91_twi_runtime_suspend(struct device *dev)
-{
- struct at91_twi_dev *twi_dev = dev_get_drvdata(dev);
-
- clk_disable_unprepare(twi_dev->clk);
-
- pinctrl_pm_select_sleep_state(dev);
-
- return 0;
-}
-
-static int at91_twi_runtime_resume(struct device *dev)
-{
- struct at91_twi_dev *twi_dev = dev_get_drvdata(dev);
-
- pinctrl_pm_select_default_state(dev);
-
- return clk_prepare_enable(twi_dev->clk);
-}
-
-static int at91_twi_suspend_noirq(struct device *dev)
-{
- if (!pm_runtime_status_suspended(dev))
- at91_twi_runtime_suspend(dev);
-
- return 0;
-}
-
-static int at91_twi_resume_noirq(struct device *dev)
-{
- struct at91_twi_dev *twi_dev = dev_get_drvdata(dev);
- int ret;
-
- if (!pm_runtime_status_suspended(dev)) {
- ret = at91_twi_runtime_resume(dev);
- if (ret)
- return ret;
- }
-
- pm_runtime_mark_last_busy(dev);
- pm_request_autosuspend(dev);
-
- at91_init_twi_bus(twi_dev);
-
- return 0;
-}
-
-static const struct dev_pm_ops at91_twi_pm = {
- .suspend_noirq = at91_twi_suspend_noirq,
- .resume_noirq = at91_twi_resume_noirq,
- .runtime_suspend = at91_twi_runtime_suspend,
- .runtime_resume = at91_twi_runtime_resume,
-};
-
-#define at91_twi_pm_ops (&at91_twi_pm)
-#else
-#define at91_twi_pm_ops NULL
-#endif
-
-static struct platform_driver at91_twi_driver = {
- .probe = at91_twi_probe,
- .remove = at91_twi_remove,
- .id_table = at91_twi_devtypes,
- .driver = {
- .name = "at91_i2c",
- .of_match_table = of_match_ptr(atmel_twi_dt_ids),
- .pm = at91_twi_pm_ops,
- },
-};
-
-static int __init at91_twi_init(void)
-{
- return platform_driver_register(&at91_twi_driver);
-}
-
-static void __exit at91_twi_exit(void)
-{
- platform_driver_unregister(&at91_twi_driver);
-}
-
-subsys_initcall(at91_twi_init);
-module_exit(at91_twi_exit);
-
-MODULE_AUTHOR("Nikolaus Voss <n.voss@weinmann.de>");
-MODULE_DESCRIPTION("I2C (TWI) driver for Atmel AT91");
-MODULE_LICENSE("GPL");
-MODULE_ALIAS("platform:at91_i2c");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * i2c Support for Atmel's AT91 Two-Wire Interface (TWI)
+ *
+ * Copyright (C) 2011 Weinmann Medical GmbH
+ * Author: Nikolaus Voss <n.voss@weinmann.de>
+ *
+ * Evolved from original work by:
+ * Copyright (C) 2004 Rick Bronson
+ * Converted to 2.6 by Andrew Victor <andrew@sanpeople.com>
+ *
+ * Borrowed heavily from original work by:
+ * Copyright (C) 2000 Philip Edelbrock <phil@stimpy.netroedge.com>
+ */
+
+#include <linux/clk.h>
+#include <linux/completion.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmaengine.h>
+#include <linux/i2c.h>
+#include <linux/platform_data/dma-atmel.h>
+#include <linux/platform_device.h>
+
+#define AT91_I2C_TIMEOUT msecs_to_jiffies(100) /* transfer timeout */
+#define AT91_I2C_DMA_THRESHOLD 8 /* enable DMA if transfer size is bigger than this threshold */
+#define AUTOSUSPEND_TIMEOUT 2000
+#define AT91_I2C_MAX_ALT_CMD_DATA_SIZE 256
+
+/* AT91 TWI register definitions */
+#define AT91_TWI_CR 0x0000 /* Control Register */
+#define AT91_TWI_START BIT(0) /* Send a Start Condition */
+#define AT91_TWI_STOP BIT(1) /* Send a Stop Condition */
+#define AT91_TWI_MSEN BIT(2) /* Master Transfer Enable */
+#define AT91_TWI_MSDIS BIT(3) /* Master Transfer Disable */
+#define AT91_TWI_SVEN BIT(4) /* Slave Transfer Enable */
+#define AT91_TWI_SVDIS BIT(5) /* Slave Transfer Disable */
+#define AT91_TWI_QUICK BIT(6) /* SMBus quick command */
+#define AT91_TWI_SWRST BIT(7) /* Software Reset */
+#define AT91_TWI_ACMEN BIT(16) /* Alternative Command Mode Enable */
+#define AT91_TWI_ACMDIS BIT(17) /* Alternative Command Mode Disable */
+#define AT91_TWI_THRCLR BIT(24) /* Transmit Holding Register Clear */
+#define AT91_TWI_RHRCLR BIT(25) /* Receive Holding Register Clear */
+#define AT91_TWI_LOCKCLR BIT(26) /* Lock Clear */
+#define AT91_TWI_FIFOEN BIT(28) /* FIFO Enable */
+#define AT91_TWI_FIFODIS BIT(29) /* FIFO Disable */
+
+#define AT91_TWI_MMR 0x0004 /* Master Mode Register */
+#define AT91_TWI_IADRSZ_1 0x0100 /* Internal Device Address Size */
+#define AT91_TWI_MREAD BIT(12) /* Master Read Direction */
+
+#define AT91_TWI_SMR 0x0008 /* Slave Mode Register */
+#define AT91_TWI_SMR_SADR_MAX 0x007f
+#define AT91_TWI_SMR_SADR(x) (((x) & AT91_TWI_SMR_SADR_MAX) << 16)
+
+#define AT91_TWI_IADR 0x000c /* Internal Address Register */
+
+#define AT91_TWI_CWGR 0x0010 /* Clock Waveform Generator Reg */
+#define AT91_TWI_CWGR_HOLD_MAX 0x1f
+#define AT91_TWI_CWGR_HOLD(x) (((x) & AT91_TWI_CWGR_HOLD_MAX) << 24)
+
+#define AT91_TWI_SR 0x0020 /* Status Register */
+#define AT91_TWI_TXCOMP BIT(0) /* Transmission Complete */
+#define AT91_TWI_RXRDY BIT(1) /* Receive Holding Register Ready */
+#define AT91_TWI_TXRDY BIT(2) /* Transmit Holding Register Ready */
+#define AT91_TWI_SVREAD BIT(3) /* Slave Read */
+#define AT91_TWI_SVACC BIT(4) /* Slave Access */
+#define AT91_TWI_OVRE BIT(6) /* Overrun Error */
+#define AT91_TWI_UNRE BIT(7) /* Underrun Error */
+#define AT91_TWI_NACK BIT(8) /* Not Acknowledged */
+#define AT91_TWI_EOSACC BIT(11) /* End Of Slave Access */
+#define AT91_TWI_LOCK BIT(23) /* TWI Lock due to Frame Errors */
+
+#define AT91_TWI_INT_MASK \
+ (AT91_TWI_TXCOMP | AT91_TWI_RXRDY | AT91_TWI_TXRDY | AT91_TWI_NACK \
+ | AT91_TWI_SVACC | AT91_TWI_EOSACC)
+
+#define AT91_TWI_IER 0x0024 /* Interrupt Enable Register */
+#define AT91_TWI_IDR 0x0028 /* Interrupt Disable Register */
+#define AT91_TWI_IMR 0x002c /* Interrupt Mask Register */
+#define AT91_TWI_RHR 0x0030 /* Receive Holding Register */
+#define AT91_TWI_THR 0x0034 /* Transmit Holding Register */
+
+#define AT91_TWI_ACR 0x0040 /* Alternative Command Register */
+#define AT91_TWI_ACR_DATAL(len) ((len) & 0xff)
+#define AT91_TWI_ACR_DIR BIT(8)
+
+#define AT91_TWI_FMR 0x0050 /* FIFO Mode Register */
+#define AT91_TWI_FMR_TXRDYM(mode) (((mode) & 0x3) << 0)
+#define AT91_TWI_FMR_TXRDYM_MASK (0x3 << 0)
+#define AT91_TWI_FMR_RXRDYM(mode) (((mode) & 0x3) << 4)
+#define AT91_TWI_FMR_RXRDYM_MASK (0x3 << 4)
+#define AT91_TWI_ONE_DATA 0x0
+#define AT91_TWI_TWO_DATA 0x1
+#define AT91_TWI_FOUR_DATA 0x2
+
+#define AT91_TWI_FLR 0x0054 /* FIFO Level Register */
+
+#define AT91_TWI_FSR 0x0060 /* FIFO Status Register */
+#define AT91_TWI_FIER 0x0064 /* FIFO Interrupt Enable Register */
+#define AT91_TWI_FIDR 0x0068 /* FIFO Interrupt Disable Register */
+#define AT91_TWI_FIMR 0x006c /* FIFO Interrupt Mask Register */
+
+#define AT91_TWI_VER 0x00fc /* Version Register */
+
+struct at91_twi_pdata {
+ unsigned clk_max_div;
+ unsigned clk_offset;
+ bool has_unre_flag;
+ bool has_alt_cmd;
+ bool has_hold_field;
+ struct at_dma_slave dma_slave;
+};
+
+struct at91_twi_dma {
+ struct dma_chan *chan_rx;
+ struct dma_chan *chan_tx;
+ struct scatterlist sg[2];
+ struct dma_async_tx_descriptor *data_desc;
+ enum dma_data_direction direction;
+ bool buf_mapped;
+ bool xfer_in_progress;
+};
+
+struct at91_twi_dev {
+ struct device *dev;
+ void __iomem *base;
+ struct completion cmd_complete;
+ struct clk *clk;
+ u8 *buf;
+ size_t buf_len;
+ struct i2c_msg *msg;
+ int irq;
+ unsigned imr;
+ unsigned transfer_status;
+ struct i2c_adapter adapter;
+ unsigned twi_cwgr_reg;
+ struct at91_twi_pdata *pdata;
+ bool use_dma;
+ bool use_alt_cmd;
+ bool recv_len_abort;
+ u32 fifo_size;
+ struct at91_twi_dma dma;
+ bool slave_detected;
+#ifdef CONFIG_I2C_AT91_SLAVE_EXPERIMENTAL
+ unsigned smr;
+ struct i2c_client *slave;
+#endif
+};
+
+unsigned at91_twi_read(struct at91_twi_dev *dev, unsigned reg);
+void at91_twi_write(struct at91_twi_dev *dev, unsigned reg, unsigned val);
+void at91_disable_twi_interrupts(struct at91_twi_dev *dev);
+void at91_twi_irq_save(struct at91_twi_dev *dev);
+void at91_twi_irq_restore(struct at91_twi_dev *dev);
+void at91_init_twi_bus(struct at91_twi_dev *dev);
+
+void at91_init_twi_bus_master(struct at91_twi_dev *dev);
+int at91_twi_probe_master(struct platform_device *pdev, u32 phy_addr,
+ struct at91_twi_dev *dev);
+
+#ifdef CONFIG_I2C_AT91_SLAVE_EXPERIMENTAL
+void at91_init_twi_bus_slave(struct at91_twi_dev *dev);
+int at91_twi_probe_slave(struct platform_device *pdev, u32 phy_addr,
+ struct at91_twi_dev *dev);
+
+#else
+static inline void at91_init_twi_bus_slave(struct at91_twi_dev *dev) {}
+static inline int at91_twi_probe_slave(struct platform_device *pdev,
+ u32 phy_addr, struct at91_twi_dev *dev)
+{
+ return -EINVAL;
+}
+
+#endif
* @adapter: core i2c abstraction
* @i2c_clk: clock reference for i2c input clock
* @bus_clk_rate: current i2c bus clock rate
+ * @last: a flag indicating is this is last message in transfer
*/
struct axxia_i2c_dev {
void __iomem *base;
struct i2c_adapter adapter;
struct clk *i2c_clk;
u32 bus_clk_rate;
+ bool last;
};
static void i2c_int_disable(struct axxia_i2c_dev *idev, u32 mask)
/* Stop completed */
i2c_int_disable(idev, ~MST_STATUS_TSS);
complete(&idev->msg_complete);
- } else if (status & MST_STATUS_SNS) {
+ } else if (status & (MST_STATUS_SNS | MST_STATUS_SS)) {
/* Transfer done */
- i2c_int_disable(idev, ~MST_STATUS_TSS);
+ int mask = idev->last ? ~0 : ~MST_STATUS_TSS;
+
+ i2c_int_disable(idev, mask);
if (i2c_m_rd(idev->msg_r) && idev->msg_xfrd_r < idev->msg_r->len)
axxia_i2c_empty_rx_fifo(idev);
complete(&idev->msg_complete);
- } else if (status & MST_STATUS_SS) {
- /* Auto/Sequence transfer done */
- complete(&idev->msg_complete);
} else if (status & MST_STATUS_TSS) {
/* Transfer timeout */
idev->msg_err = -ETIMEDOUT;
idev->msg_r = &msgs[1];
idev->msg_xfrd = 0;
idev->msg_xfrd_r = 0;
+ idev->last = true;
axxia_i2c_fill_tx_fifo(idev);
writel(CMD_SEQUENCE, idev->base + MST_COMMAND);
time_left = wait_for_completion_timeout(&idev->msg_complete,
I2C_XFER_TIMEOUT);
- i2c_int_disable(idev, int_mask);
-
- axxia_i2c_empty_rx_fifo(idev);
-
if (idev->msg_err == -ENXIO) {
if (axxia_i2c_handle_seq_nak(idev))
axxia_i2c_init(idev);
return idev->msg_err;
}
-static int axxia_i2c_xfer_msg(struct axxia_i2c_dev *idev, struct i2c_msg *msg)
+static int axxia_i2c_xfer_msg(struct axxia_i2c_dev *idev, struct i2c_msg *msg,
+ bool last)
{
- u32 int_mask = MST_STATUS_ERR | MST_STATUS_SNS;
+ u32 int_mask = MST_STATUS_ERR;
u32 rx_xfer, tx_xfer;
unsigned long time_left;
unsigned int wt_value;
idev->msg_r = msg;
idev->msg_xfrd = 0;
idev->msg_xfrd_r = 0;
+ idev->last = last;
reinit_completion(&idev->msg_complete);
axxia_i2c_set_addr(idev, msg);
if (idev->msg_err)
goto out;
- /* Start manual mode */
- writel(CMD_MANUAL, idev->base + MST_COMMAND);
+ if (!last) {
+ writel(CMD_MANUAL, idev->base + MST_COMMAND);
+ int_mask |= MST_STATUS_SNS;
+ } else {
+ writel(CMD_AUTO, idev->base + MST_COMMAND);
+ int_mask |= MST_STATUS_SS;
+ }
writel(WT_EN | wt_value, idev->base + WAIT_TIMER_CONTROL);
return idev->msg_err;
}
-static int axxia_i2c_stop(struct axxia_i2c_dev *idev)
-{
- u32 int_mask = MST_STATUS_ERR | MST_STATUS_SCC | MST_STATUS_TSS;
- unsigned long time_left;
-
- reinit_completion(&idev->msg_complete);
-
- /* Issue stop */
- writel(0xb, idev->base + MST_COMMAND);
- i2c_int_enable(idev, int_mask);
- time_left = wait_for_completion_timeout(&idev->msg_complete,
- I2C_STOP_TIMEOUT);
- i2c_int_disable(idev, int_mask);
- if (time_left == 0)
- return -ETIMEDOUT;
-
- if (readl(idev->base + MST_COMMAND) & CMD_BUSY)
- dev_warn(idev->dev, "busy after stop\n");
-
- return 0;
-}
-
/* This function checks if the msgs[] array contains messages compatible with
* Sequence mode of operation. This mode assumes there will be exactly one
* write of non-zero length followed by exactly one read of non-zero length,
i2c_int_enable(idev, MST_STATUS_TSS);
for (i = 0; ret == 0 && i < num; ++i)
- ret = axxia_i2c_xfer_msg(idev, &msgs[i]);
-
- axxia_i2c_stop(idev);
+ ret = axxia_i2c_xfer_msg(idev, &msgs[i], i == (num - 1));
return ret ? : i;
}
#include <linux/io.h>
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/platform_device.h>
#include <linux/slab.h>
+#define IDM_CTRL_DIRECT_OFFSET 0x00
#define CFG_OFFSET 0x00
#define CFG_RESET_SHIFT 31
#define CFG_EN_SHIFT 30
+#define CFG_SLAVE_ADDR_0_SHIFT 28
#define CFG_M_RETRY_CNT_SHIFT 16
#define CFG_M_RETRY_CNT_MASK 0x0f
#define TIM_CFG_OFFSET 0x04
#define TIM_CFG_MODE_400_SHIFT 31
+#define TIM_RAND_SLAVE_STRETCH_SHIFT 24
+#define TIM_RAND_SLAVE_STRETCH_MASK 0x7f
+#define TIM_PERIODIC_SLAVE_STRETCH_SHIFT 16
+#define TIM_PERIODIC_SLAVE_STRETCH_MASK 0x7f
+
+#define S_CFG_SMBUS_ADDR_OFFSET 0x08
+#define S_CFG_EN_NIC_SMB_ADDR3_SHIFT 31
+#define S_CFG_NIC_SMB_ADDR3_SHIFT 24
+#define S_CFG_NIC_SMB_ADDR3_MASK 0x7f
+#define S_CFG_EN_NIC_SMB_ADDR2_SHIFT 23
+#define S_CFG_NIC_SMB_ADDR2_SHIFT 16
+#define S_CFG_NIC_SMB_ADDR2_MASK 0x7f
+#define S_CFG_EN_NIC_SMB_ADDR1_SHIFT 15
+#define S_CFG_NIC_SMB_ADDR1_SHIFT 8
+#define S_CFG_NIC_SMB_ADDR1_MASK 0x7f
+#define S_CFG_EN_NIC_SMB_ADDR0_SHIFT 7
+#define S_CFG_NIC_SMB_ADDR0_SHIFT 0
+#define S_CFG_NIC_SMB_ADDR0_MASK 0x7f
#define M_FIFO_CTRL_OFFSET 0x0c
#define M_FIFO_RX_FLUSH_SHIFT 31
#define M_FIFO_RX_THLD_SHIFT 8
#define M_FIFO_RX_THLD_MASK 0x3f
+#define S_FIFO_CTRL_OFFSET 0x10
+#define S_FIFO_RX_FLUSH_SHIFT 31
+#define S_FIFO_TX_FLUSH_SHIFT 30
+#define S_FIFO_RX_CNT_SHIFT 16
+#define S_FIFO_RX_CNT_MASK 0x7f
+#define S_FIFO_RX_THLD_SHIFT 8
+#define S_FIFO_RX_THLD_MASK 0x3f
+
#define M_CMD_OFFSET 0x30
#define M_CMD_START_BUSY_SHIFT 31
#define M_CMD_STATUS_SHIFT 25
#define M_CMD_STATUS_NACK_ADDR 0x2
#define M_CMD_STATUS_NACK_DATA 0x3
#define M_CMD_STATUS_TIMEOUT 0x4
+#define M_CMD_STATUS_FIFO_UNDERRUN 0x5
+#define M_CMD_STATUS_RX_FIFO_FULL 0x6
#define M_CMD_PROTOCOL_SHIFT 9
#define M_CMD_PROTOCOL_MASK 0xf
#define M_CMD_PROTOCOL_BLK_WR 0x7
#define M_CMD_RD_CNT_SHIFT 0
#define M_CMD_RD_CNT_MASK 0xff
+#define S_CMD_OFFSET 0x34
+#define S_CMD_START_BUSY_SHIFT 31
+#define S_CMD_STATUS_SHIFT 23
+#define S_CMD_STATUS_MASK 0x07
+#define S_CMD_STATUS_SUCCESS 0x0
+#define S_CMD_STATUS_TIMEOUT 0x5
+
#define IE_OFFSET 0x38
#define IE_M_RX_FIFO_FULL_SHIFT 31
#define IE_M_RX_THLD_SHIFT 30
#define IE_M_START_BUSY_SHIFT 28
#define IE_M_TX_UNDERRUN_SHIFT 27
+#define IE_S_RX_FIFO_FULL_SHIFT 26
+#define IE_S_RX_THLD_SHIFT 25
+#define IE_S_RX_EVENT_SHIFT 24
+#define IE_S_START_BUSY_SHIFT 23
+#define IE_S_TX_UNDERRUN_SHIFT 22
+#define IE_S_RD_EVENT_SHIFT 21
#define IS_OFFSET 0x3c
#define IS_M_RX_FIFO_FULL_SHIFT 31
#define IS_M_RX_THLD_SHIFT 30
#define IS_M_START_BUSY_SHIFT 28
#define IS_M_TX_UNDERRUN_SHIFT 27
+#define IS_S_RX_FIFO_FULL_SHIFT 26
+#define IS_S_RX_THLD_SHIFT 25
+#define IS_S_RX_EVENT_SHIFT 24
+#define IS_S_START_BUSY_SHIFT 23
+#define IS_S_TX_UNDERRUN_SHIFT 22
+#define IS_S_RD_EVENT_SHIFT 21
#define M_TX_OFFSET 0x40
#define M_TX_WR_STATUS_SHIFT 31
#define M_RX_DATA_SHIFT 0
#define M_RX_DATA_MASK 0xff
+#define S_TX_OFFSET 0x48
+#define S_TX_WR_STATUS_SHIFT 31
+#define S_TX_DATA_SHIFT 0
+#define S_TX_DATA_MASK 0xff
+
+#define S_RX_OFFSET 0x4c
+#define S_RX_STATUS_SHIFT 30
+#define S_RX_STATUS_MASK 0x03
+#define S_RX_PEC_ERR_SHIFT 29
+#define S_RX_DATA_SHIFT 0
+#define S_RX_DATA_MASK 0xff
+
#define I2C_TIMEOUT_MSEC 50000
#define M_TX_RX_FIFO_SIZE 64
+#define M_RX_FIFO_MAX_THLD_VALUE (M_TX_RX_FIFO_SIZE - 1)
+
+#define M_RX_MAX_READ_LEN 255
+#define M_RX_FIFO_THLD_VALUE 50
+
+#define IE_M_ALL_INTERRUPT_SHIFT 27
+#define IE_M_ALL_INTERRUPT_MASK 0x1e
+
+#define SLAVE_READ_WRITE_BIT_MASK 0x1
+#define SLAVE_READ_WRITE_BIT_SHIFT 0x1
+#define SLAVE_MAX_SIZE_TRANSACTION 64
+#define SLAVE_CLOCK_STRETCH_TIME 25
+
+#define IE_S_ALL_INTERRUPT_SHIFT 21
+#define IE_S_ALL_INTERRUPT_MASK 0x3f
+
+enum i2c_slave_read_status {
+ I2C_SLAVE_RX_FIFO_EMPTY = 0,
+ I2C_SLAVE_RX_START,
+ I2C_SLAVE_RX_DATA,
+ I2C_SLAVE_RX_END,
+};
+
+enum i2c_slave_xfer_dir {
+ I2C_SLAVE_DIR_READ = 0,
+ I2C_SLAVE_DIR_WRITE,
+ I2C_SLAVE_DIR_NONE,
+};
enum bus_speed_index {
I2C_SPD_100K = 0,
I2C_SPD_400K,
};
+enum bcm_iproc_i2c_type {
+ IPROC_I2C,
+ IPROC_I2C_NIC
+};
+
struct bcm_iproc_i2c_dev {
struct device *device;
+ enum bcm_iproc_i2c_type type;
int irq;
void __iomem *base;
+ void __iomem *idm_base;
+
+ u32 ape_addr_mask;
+
+ /* lock for indirect access through IDM */
+ spinlock_t idm_lock;
struct i2c_adapter adapter;
unsigned int bus_speed;
struct i2c_msg *msg;
+ struct i2c_client *slave;
+ enum i2c_slave_xfer_dir xfer_dir;
+
/* bytes that have been transferred */
unsigned int tx_bytes;
+ /* bytes that have been read */
+ unsigned int rx_bytes;
+ unsigned int thld_bytes;
};
/*
* Can be expanded in the future if more interrupt status bits are utilized
*/
-#define ISR_MASK (BIT(IS_M_START_BUSY_SHIFT) | BIT(IS_M_TX_UNDERRUN_SHIFT))
+#define ISR_MASK (BIT(IS_M_START_BUSY_SHIFT) | BIT(IS_M_TX_UNDERRUN_SHIFT)\
+ | BIT(IS_M_RX_THLD_SHIFT))
-static irqreturn_t bcm_iproc_i2c_isr(int irq, void *data)
+#define ISR_MASK_SLAVE (BIT(IS_S_START_BUSY_SHIFT)\
+ | BIT(IS_S_RX_EVENT_SHIFT) | BIT(IS_S_RD_EVENT_SHIFT))
+
+static int bcm_iproc_i2c_reg_slave(struct i2c_client *slave);
+static int bcm_iproc_i2c_unreg_slave(struct i2c_client *slave);
+static void bcm_iproc_i2c_enable_disable(struct bcm_iproc_i2c_dev *iproc_i2c,
+ bool enable);
+
+static inline u32 iproc_i2c_rd_reg(struct bcm_iproc_i2c_dev *iproc_i2c,
+ u32 offset)
{
- struct bcm_iproc_i2c_dev *iproc_i2c = data;
- u32 status = readl(iproc_i2c->base + IS_OFFSET);
+ u32 val;
- status &= ISR_MASK;
+ if (iproc_i2c->idm_base) {
+ spin_lock(&iproc_i2c->idm_lock);
+ writel(iproc_i2c->ape_addr_mask,
+ iproc_i2c->idm_base + IDM_CTRL_DIRECT_OFFSET);
+ val = readl(iproc_i2c->base + offset);
+ spin_unlock(&iproc_i2c->idm_lock);
+ } else {
+ val = readl(iproc_i2c->base + offset);
+ }
- if (!status)
- return IRQ_NONE;
+ return val;
+}
- /* TX FIFO is empty and we have more data to send */
- if (status & BIT(IS_M_TX_UNDERRUN_SHIFT)) {
- struct i2c_msg *msg = iproc_i2c->msg;
- unsigned int tx_bytes = msg->len - iproc_i2c->tx_bytes;
- unsigned int i;
- u32 val;
-
- /* can only fill up to the FIFO size */
- tx_bytes = min_t(unsigned int, tx_bytes, M_TX_RX_FIFO_SIZE);
- for (i = 0; i < tx_bytes; i++) {
- /* start from where we left over */
- unsigned int idx = iproc_i2c->tx_bytes + i;
+static inline void iproc_i2c_wr_reg(struct bcm_iproc_i2c_dev *iproc_i2c,
+ u32 offset, u32 val)
+{
+ if (iproc_i2c->idm_base) {
+ spin_lock(&iproc_i2c->idm_lock);
+ writel(iproc_i2c->ape_addr_mask,
+ iproc_i2c->idm_base + IDM_CTRL_DIRECT_OFFSET);
+ writel(val, iproc_i2c->base + offset);
+ spin_unlock(&iproc_i2c->idm_lock);
+ } else {
+ writel(val, iproc_i2c->base + offset);
+ }
+}
- val = msg->buf[idx];
+static void bcm_iproc_i2c_slave_init(
+ struct bcm_iproc_i2c_dev *iproc_i2c, bool need_reset)
+{
+ u32 val;
- /* mark the last byte */
- if (idx == msg->len - 1) {
- u32 tmp;
+ if (need_reset) {
+ /* put controller in reset */
+ val = iproc_i2c_rd_reg(iproc_i2c, CFG_OFFSET);
+ val |= BIT(CFG_RESET_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, CFG_OFFSET, val);
- val |= BIT(M_TX_WR_STATUS_SHIFT);
+ /* wait 100 usec per spec */
+ udelay(100);
+
+ /* bring controller out of reset */
+ val &= ~(BIT(CFG_RESET_SHIFT));
+ iproc_i2c_wr_reg(iproc_i2c, CFG_OFFSET, val);
+ }
+
+ /* flush TX/RX FIFOs */
+ val = (BIT(S_FIFO_RX_FLUSH_SHIFT) | BIT(S_FIFO_TX_FLUSH_SHIFT));
+ iproc_i2c_wr_reg(iproc_i2c, S_FIFO_CTRL_OFFSET, val);
+
+ /* Maximum slave stretch time */
+ val = iproc_i2c_rd_reg(iproc_i2c, TIM_CFG_OFFSET);
+ val &= ~(TIM_RAND_SLAVE_STRETCH_MASK << TIM_RAND_SLAVE_STRETCH_SHIFT);
+ val |= (SLAVE_CLOCK_STRETCH_TIME << TIM_RAND_SLAVE_STRETCH_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, TIM_CFG_OFFSET, val);
+
+ /* Configure the slave address */
+ val = iproc_i2c_rd_reg(iproc_i2c, S_CFG_SMBUS_ADDR_OFFSET);
+ val |= BIT(S_CFG_EN_NIC_SMB_ADDR3_SHIFT);
+ val &= ~(S_CFG_NIC_SMB_ADDR3_MASK << S_CFG_NIC_SMB_ADDR3_SHIFT);
+ val |= (iproc_i2c->slave->addr << S_CFG_NIC_SMB_ADDR3_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, S_CFG_SMBUS_ADDR_OFFSET, val);
+
+ /* clear all pending slave interrupts */
+ iproc_i2c_wr_reg(iproc_i2c, IS_OFFSET, ISR_MASK_SLAVE);
+
+ /* Enable interrupt register for any READ event */
+ val = BIT(IE_S_RD_EVENT_SHIFT);
+ /* Enable interrupt register to indicate a valid byte in receive fifo */
+ val |= BIT(IE_S_RX_EVENT_SHIFT);
+ /* Enable interrupt register for the Slave BUSY command */
+ val |= BIT(IE_S_START_BUSY_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, IE_OFFSET, val);
+
+ iproc_i2c->xfer_dir = I2C_SLAVE_DIR_NONE;
+}
+
+static void bcm_iproc_i2c_check_slave_status(
+ struct bcm_iproc_i2c_dev *iproc_i2c)
+{
+ u32 val;
+
+ val = iproc_i2c_rd_reg(iproc_i2c, S_CMD_OFFSET);
+ val = (val >> S_CMD_STATUS_SHIFT) & S_CMD_STATUS_MASK;
+
+ if (val == S_CMD_STATUS_TIMEOUT) {
+ dev_err(iproc_i2c->device, "slave random stretch time timeout\n");
+
+ /* re-initialize i2c for recovery */
+ bcm_iproc_i2c_enable_disable(iproc_i2c, false);
+ bcm_iproc_i2c_slave_init(iproc_i2c, true);
+ bcm_iproc_i2c_enable_disable(iproc_i2c, true);
+ }
+}
+
+static bool bcm_iproc_i2c_slave_isr(struct bcm_iproc_i2c_dev *iproc_i2c,
+ u32 status)
+{
+ u8 value;
+ u32 val;
+ u32 rd_status;
+ u32 tmp;
+
+ /* Start of transaction. check address and populate the direction */
+ if (iproc_i2c->xfer_dir == I2C_SLAVE_DIR_NONE) {
+ tmp = iproc_i2c_rd_reg(iproc_i2c, S_RX_OFFSET);
+ rd_status = (tmp >> S_RX_STATUS_SHIFT) & S_RX_STATUS_MASK;
+ /* This condition checks whether the request is a new request */
+ if (((rd_status == I2C_SLAVE_RX_START) &&
+ (status & BIT(IS_S_RX_EVENT_SHIFT))) ||
+ ((rd_status == I2C_SLAVE_RX_END) &&
+ (status & BIT(IS_S_RD_EVENT_SHIFT)))) {
+
+ /* Last bit is W/R bit.
+ * If 1 then its a read request(by master).
+ */
+ iproc_i2c->xfer_dir = tmp & SLAVE_READ_WRITE_BIT_MASK;
+ if (iproc_i2c->xfer_dir == I2C_SLAVE_DIR_WRITE)
+ i2c_slave_event(iproc_i2c->slave,
+ I2C_SLAVE_READ_REQUESTED, &value);
+ else
+ i2c_slave_event(iproc_i2c->slave,
+ I2C_SLAVE_WRITE_REQUESTED, &value);
+ }
+ }
+
+ /* read request from master */
+ if ((status & BIT(IS_S_RD_EVENT_SHIFT)) &&
+ (iproc_i2c->xfer_dir == I2C_SLAVE_DIR_WRITE)) {
+ i2c_slave_event(iproc_i2c->slave,
+ I2C_SLAVE_READ_PROCESSED, &value);
+ iproc_i2c_wr_reg(iproc_i2c, S_TX_OFFSET, value);
+
+ val = BIT(S_CMD_START_BUSY_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, S_CMD_OFFSET, val);
+ }
+
+ /* write request from master */
+ if ((status & BIT(IS_S_RX_EVENT_SHIFT)) &&
+ (iproc_i2c->xfer_dir == I2C_SLAVE_DIR_READ)) {
+ val = iproc_i2c_rd_reg(iproc_i2c, S_RX_OFFSET);
+ /* Its a write request by Master to Slave.
+ * We read data present in receive FIFO
+ */
+ value = (u8)((val >> S_RX_DATA_SHIFT) & S_RX_DATA_MASK);
+ i2c_slave_event(iproc_i2c->slave,
+ I2C_SLAVE_WRITE_RECEIVED, &value);
+
+ /* check the status for the last byte of the transaction */
+ rd_status = (val >> S_RX_STATUS_SHIFT) & S_RX_STATUS_MASK;
+ if (rd_status == I2C_SLAVE_RX_END)
+ iproc_i2c->xfer_dir = I2C_SLAVE_DIR_NONE;
+
+ dev_dbg(iproc_i2c->device, "\nread value = 0x%x\n", value);
+ }
+
+ /* Stop */
+ if (status & BIT(IS_S_START_BUSY_SHIFT)) {
+ i2c_slave_event(iproc_i2c->slave, I2C_SLAVE_STOP, &value);
+ iproc_i2c->xfer_dir = I2C_SLAVE_DIR_NONE;
+ }
+
+ /* clear interrupt status */
+ iproc_i2c_wr_reg(iproc_i2c, IS_OFFSET, status);
+
+ bcm_iproc_i2c_check_slave_status(iproc_i2c);
+ return true;
+}
+
+static void bcm_iproc_i2c_read_valid_bytes(struct bcm_iproc_i2c_dev *iproc_i2c)
+{
+ struct i2c_msg *msg = iproc_i2c->msg;
+
+ /* Read valid data from RX FIFO */
+ while (iproc_i2c->rx_bytes < msg->len) {
+ if (!((iproc_i2c_rd_reg(iproc_i2c, M_FIFO_CTRL_OFFSET) >> M_FIFO_RX_CNT_SHIFT)
+ & M_FIFO_RX_CNT_MASK))
+ break;
+
+ msg->buf[iproc_i2c->rx_bytes] =
+ (iproc_i2c_rd_reg(iproc_i2c, M_RX_OFFSET) >>
+ M_RX_DATA_SHIFT) & M_RX_DATA_MASK;
+ iproc_i2c->rx_bytes++;
+ }
+}
+
+static void bcm_iproc_i2c_send(struct bcm_iproc_i2c_dev *iproc_i2c)
+{
+ struct i2c_msg *msg = iproc_i2c->msg;
+ unsigned int tx_bytes = msg->len - iproc_i2c->tx_bytes;
+ unsigned int i;
+ u32 val;
+
+ /* can only fill up to the FIFO size */
+ tx_bytes = min_t(unsigned int, tx_bytes, M_TX_RX_FIFO_SIZE);
+ for (i = 0; i < tx_bytes; i++) {
+ /* start from where we left over */
+ unsigned int idx = iproc_i2c->tx_bytes + i;
+
+ val = msg->buf[idx];
+
+ /* mark the last byte */
+ if (idx == msg->len - 1) {
+ val |= BIT(M_TX_WR_STATUS_SHIFT);
+
+ if (iproc_i2c->irq) {
+ u32 tmp;
/*
- * Since this is the last byte, we should
- * now disable TX FIFO underrun interrupt
+ * Since this is the last byte, we should now
+ * disable TX FIFO underrun interrupt
*/
- tmp = readl(iproc_i2c->base + IE_OFFSET);
+ tmp = iproc_i2c_rd_reg(iproc_i2c, IE_OFFSET);
tmp &= ~BIT(IE_M_TX_UNDERRUN_SHIFT);
- writel(tmp, iproc_i2c->base + IE_OFFSET);
+ iproc_i2c_wr_reg(iproc_i2c, IE_OFFSET,
+ tmp);
}
+ }
+
+ /* load data into TX FIFO */
+ iproc_i2c_wr_reg(iproc_i2c, M_TX_OFFSET, val);
+ }
- /* load data into TX FIFO */
- writel(val, iproc_i2c->base + M_TX_OFFSET);
+ /* update number of transferred bytes */
+ iproc_i2c->tx_bytes += tx_bytes;
+}
+
+static void bcm_iproc_i2c_read(struct bcm_iproc_i2c_dev *iproc_i2c)
+{
+ struct i2c_msg *msg = iproc_i2c->msg;
+ u32 bytes_left, val;
+
+ bcm_iproc_i2c_read_valid_bytes(iproc_i2c);
+ bytes_left = msg->len - iproc_i2c->rx_bytes;
+ if (bytes_left == 0) {
+ if (iproc_i2c->irq) {
+ /* finished reading all data, disable rx thld event */
+ val = iproc_i2c_rd_reg(iproc_i2c, IE_OFFSET);
+ val &= ~BIT(IS_M_RX_THLD_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, IE_OFFSET, val);
}
- /* update number of transferred bytes */
- iproc_i2c->tx_bytes += tx_bytes;
+ } else if (bytes_left < iproc_i2c->thld_bytes) {
+ /* set bytes left as threshold */
+ val = iproc_i2c_rd_reg(iproc_i2c, M_FIFO_CTRL_OFFSET);
+ val &= ~(M_FIFO_RX_THLD_MASK << M_FIFO_RX_THLD_SHIFT);
+ val |= (bytes_left << M_FIFO_RX_THLD_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, M_FIFO_CTRL_OFFSET, val);
+ iproc_i2c->thld_bytes = bytes_left;
}
+ /*
+ * bytes_left >= iproc_i2c->thld_bytes,
+ * hence no need to change the THRESHOLD SET.
+ * It will remain as iproc_i2c->thld_bytes itself
+ */
+}
+
+static void bcm_iproc_i2c_process_m_event(struct bcm_iproc_i2c_dev *iproc_i2c,
+ u32 status)
+{
+ /* TX FIFO is empty and we have more data to send */
+ if (status & BIT(IS_M_TX_UNDERRUN_SHIFT))
+ bcm_iproc_i2c_send(iproc_i2c);
+ /* RX FIFO threshold is reached and data needs to be read out */
+ if (status & BIT(IS_M_RX_THLD_SHIFT))
+ bcm_iproc_i2c_read(iproc_i2c);
+
+ /* transfer is done */
if (status & BIT(IS_M_START_BUSY_SHIFT)) {
iproc_i2c->xfer_is_done = 1;
- complete(&iproc_i2c->done);
+ if (iproc_i2c->irq)
+ complete(&iproc_i2c->done);
}
+}
- writel(status, iproc_i2c->base + IS_OFFSET);
+static irqreturn_t bcm_iproc_i2c_isr(int irq, void *data)
+{
+ struct bcm_iproc_i2c_dev *iproc_i2c = data;
+ u32 status = iproc_i2c_rd_reg(iproc_i2c, IS_OFFSET);
+ bool ret;
+ u32 sl_status = status & ISR_MASK_SLAVE;
+
+ if (sl_status) {
+ ret = bcm_iproc_i2c_slave_isr(iproc_i2c, sl_status);
+ if (ret)
+ return IRQ_HANDLED;
+ else
+ return IRQ_NONE;
+ }
+
+ status &= ISR_MASK;
+ if (!status)
+ return IRQ_NONE;
+
+ /* process all master based events */
+ bcm_iproc_i2c_process_m_event(iproc_i2c, status);
+ iproc_i2c_wr_reg(iproc_i2c, IS_OFFSET, status);
return IRQ_HANDLED;
}
u32 val;
/* put controller in reset */
- val = readl(iproc_i2c->base + CFG_OFFSET);
- val |= 1 << CFG_RESET_SHIFT;
- val &= ~(1 << CFG_EN_SHIFT);
- writel(val, iproc_i2c->base + CFG_OFFSET);
+ val = iproc_i2c_rd_reg(iproc_i2c, CFG_OFFSET);
+ val |= BIT(CFG_RESET_SHIFT);
+ val &= ~(BIT(CFG_EN_SHIFT));
+ iproc_i2c_wr_reg(iproc_i2c, CFG_OFFSET, val);
/* wait 100 usec per spec */
udelay(100);
/* bring controller out of reset */
- val &= ~(1 << CFG_RESET_SHIFT);
- writel(val, iproc_i2c->base + CFG_OFFSET);
+ val &= ~(BIT(CFG_RESET_SHIFT));
+ iproc_i2c_wr_reg(iproc_i2c, CFG_OFFSET, val);
/* flush TX/RX FIFOs and set RX FIFO threshold to zero */
- val = (1 << M_FIFO_RX_FLUSH_SHIFT) | (1 << M_FIFO_TX_FLUSH_SHIFT);
- writel(val, iproc_i2c->base + M_FIFO_CTRL_OFFSET);
+ val = (BIT(M_FIFO_RX_FLUSH_SHIFT) | BIT(M_FIFO_TX_FLUSH_SHIFT));
+ iproc_i2c_wr_reg(iproc_i2c, M_FIFO_CTRL_OFFSET, val);
/* disable all interrupts */
- writel(0, iproc_i2c->base + IE_OFFSET);
+ val = iproc_i2c_rd_reg(iproc_i2c, IE_OFFSET);
+ val &= ~(IE_M_ALL_INTERRUPT_MASK <<
+ IE_M_ALL_INTERRUPT_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, IE_OFFSET, val);
/* clear all pending interrupts */
- writel(0xffffffff, iproc_i2c->base + IS_OFFSET);
+ iproc_i2c_wr_reg(iproc_i2c, IS_OFFSET, 0xffffffff);
return 0;
}
{
u32 val;
- val = readl(iproc_i2c->base + CFG_OFFSET);
+ val = iproc_i2c_rd_reg(iproc_i2c, CFG_OFFSET);
if (enable)
val |= BIT(CFG_EN_SHIFT);
else
val &= ~BIT(CFG_EN_SHIFT);
- writel(val, iproc_i2c->base + CFG_OFFSET);
+ iproc_i2c_wr_reg(iproc_i2c, CFG_OFFSET, val);
}
static int bcm_iproc_i2c_check_status(struct bcm_iproc_i2c_dev *iproc_i2c,
{
u32 val;
- val = readl(iproc_i2c->base + M_CMD_OFFSET);
+ val = iproc_i2c_rd_reg(iproc_i2c, M_CMD_OFFSET);
val = (val >> M_CMD_STATUS_SHIFT) & M_CMD_STATUS_MASK;
switch (val) {
dev_dbg(iproc_i2c->device, "bus timeout\n");
return -ETIMEDOUT;
+ case M_CMD_STATUS_FIFO_UNDERRUN:
+ dev_dbg(iproc_i2c->device, "FIFO under-run\n");
+ return -ENXIO;
+
+ case M_CMD_STATUS_RX_FIFO_FULL:
+ dev_dbg(iproc_i2c->device, "RX FIFO full\n");
+ return -ETIMEDOUT;
+
default:
dev_dbg(iproc_i2c->device, "unknown error code=%d\n", val);
}
}
+static int bcm_iproc_i2c_xfer_wait(struct bcm_iproc_i2c_dev *iproc_i2c,
+ struct i2c_msg *msg,
+ u32 cmd)
+{
+ unsigned long time_left = msecs_to_jiffies(I2C_TIMEOUT_MSEC);
+ u32 val, status;
+ int ret;
+
+ iproc_i2c_wr_reg(iproc_i2c, M_CMD_OFFSET, cmd);
+
+ if (iproc_i2c->irq) {
+ time_left = wait_for_completion_timeout(&iproc_i2c->done,
+ time_left);
+ /* disable all interrupts */
+ iproc_i2c_wr_reg(iproc_i2c, IE_OFFSET, 0);
+ /* read it back to flush the write */
+ iproc_i2c_rd_reg(iproc_i2c, IE_OFFSET);
+ /* make sure the interrupt handler isn't running */
+ synchronize_irq(iproc_i2c->irq);
+
+ } else { /* polling mode */
+ unsigned long timeout = jiffies + time_left;
+
+ do {
+ status = iproc_i2c_rd_reg(iproc_i2c,
+ IS_OFFSET) & ISR_MASK;
+ bcm_iproc_i2c_process_m_event(iproc_i2c, status);
+ iproc_i2c_wr_reg(iproc_i2c, IS_OFFSET, status);
+
+ if (time_after(jiffies, timeout)) {
+ time_left = 0;
+ break;
+ }
+
+ cpu_relax();
+ cond_resched();
+ } while (!iproc_i2c->xfer_is_done);
+ }
+
+ if (!time_left && !iproc_i2c->xfer_is_done) {
+ dev_err(iproc_i2c->device, "transaction timed out\n");
+
+ /* flush both TX/RX FIFOs */
+ val = BIT(M_FIFO_RX_FLUSH_SHIFT) | BIT(M_FIFO_TX_FLUSH_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, M_FIFO_CTRL_OFFSET, val);
+ return -ETIMEDOUT;
+ }
+
+ ret = bcm_iproc_i2c_check_status(iproc_i2c, msg);
+ if (ret) {
+ /* flush both TX/RX FIFOs */
+ val = BIT(M_FIFO_RX_FLUSH_SHIFT) | BIT(M_FIFO_TX_FLUSH_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, M_FIFO_CTRL_OFFSET, val);
+ return ret;
+ }
+
+ return 0;
+}
+
static int bcm_iproc_i2c_xfer_single_msg(struct bcm_iproc_i2c_dev *iproc_i2c,
struct i2c_msg *msg)
{
- int ret, i;
+ int i;
u8 addr;
- u32 val;
+ u32 val, tmp, val_intr_en;
unsigned int tx_bytes;
- unsigned long time_left = msecs_to_jiffies(I2C_TIMEOUT_MSEC);
/* check if bus is busy */
- if (!!(readl(iproc_i2c->base + M_CMD_OFFSET) &
- BIT(M_CMD_START_BUSY_SHIFT))) {
+ if (!!(iproc_i2c_rd_reg(iproc_i2c,
+ M_CMD_OFFSET) & BIT(M_CMD_START_BUSY_SHIFT))) {
dev_warn(iproc_i2c->device, "bus is busy\n");
return -EBUSY;
}
/* format and load slave address into the TX FIFO */
addr = i2c_8bit_addr_from_msg(msg);
- writel(addr, iproc_i2c->base + M_TX_OFFSET);
+ iproc_i2c_wr_reg(iproc_i2c, M_TX_OFFSET, addr);
/*
* For a write transaction, load data into the TX FIFO. Only allow
/* mark the last byte */
if (i == msg->len - 1)
- val |= 1 << M_TX_WR_STATUS_SHIFT;
+ val |= BIT(M_TX_WR_STATUS_SHIFT);
- writel(val, iproc_i2c->base + M_TX_OFFSET);
+ iproc_i2c_wr_reg(iproc_i2c, M_TX_OFFSET, val);
}
iproc_i2c->tx_bytes = tx_bytes;
}
/* mark as incomplete before starting the transaction */
- reinit_completion(&iproc_i2c->done);
+ if (iproc_i2c->irq)
+ reinit_completion(&iproc_i2c->done);
+
iproc_i2c->xfer_is_done = 0;
/*
* transaction is done, i.e., the internal start_busy bit, transitions
* from 1 to 0.
*/
- val = BIT(IE_M_START_BUSY_SHIFT);
+ val_intr_en = BIT(IE_M_START_BUSY_SHIFT);
/*
* If TX data size is larger than the TX FIFO, need to enable TX
*/
if (!(msg->flags & I2C_M_RD) &&
msg->len > iproc_i2c->tx_bytes)
- val |= BIT(IE_M_TX_UNDERRUN_SHIFT);
-
- writel(val, iproc_i2c->base + IE_OFFSET);
+ val_intr_en |= BIT(IE_M_TX_UNDERRUN_SHIFT);
/*
* Now we can activate the transfer. For a read operation, specify the
*/
val = BIT(M_CMD_START_BUSY_SHIFT);
if (msg->flags & I2C_M_RD) {
+ iproc_i2c->rx_bytes = 0;
+ if (msg->len > M_RX_FIFO_MAX_THLD_VALUE)
+ iproc_i2c->thld_bytes = M_RX_FIFO_THLD_VALUE;
+ else
+ iproc_i2c->thld_bytes = msg->len;
+
+ /* set threshold value */
+ tmp = iproc_i2c_rd_reg(iproc_i2c, M_FIFO_CTRL_OFFSET);
+ tmp &= ~(M_FIFO_RX_THLD_MASK << M_FIFO_RX_THLD_SHIFT);
+ tmp |= iproc_i2c->thld_bytes << M_FIFO_RX_THLD_SHIFT;
+ iproc_i2c_wr_reg(iproc_i2c, M_FIFO_CTRL_OFFSET, tmp);
+
+ /* enable the RX threshold interrupt */
+ val_intr_en |= BIT(IE_M_RX_THLD_SHIFT);
+
val |= (M_CMD_PROTOCOL_BLK_RD << M_CMD_PROTOCOL_SHIFT) |
(msg->len << M_CMD_RD_CNT_SHIFT);
} else {
val |= (M_CMD_PROTOCOL_BLK_WR << M_CMD_PROTOCOL_SHIFT);
}
- writel(val, iproc_i2c->base + M_CMD_OFFSET);
-
- time_left = wait_for_completion_timeout(&iproc_i2c->done, time_left);
-
- /* disable all interrupts */
- writel(0, iproc_i2c->base + IE_OFFSET);
- /* read it back to flush the write */
- readl(iproc_i2c->base + IE_OFFSET);
-
- /* make sure the interrupt handler isn't running */
- synchronize_irq(iproc_i2c->irq);
- if (!time_left && !iproc_i2c->xfer_is_done) {
- dev_err(iproc_i2c->device, "transaction timed out\n");
+ if (iproc_i2c->irq)
+ iproc_i2c_wr_reg(iproc_i2c, IE_OFFSET, val_intr_en);
- /* flush FIFOs */
- val = (1 << M_FIFO_RX_FLUSH_SHIFT) |
- (1 << M_FIFO_TX_FLUSH_SHIFT);
- writel(val, iproc_i2c->base + M_FIFO_CTRL_OFFSET);
- return -ETIMEDOUT;
- }
-
- ret = bcm_iproc_i2c_check_status(iproc_i2c, msg);
- if (ret) {
- /* flush both TX/RX FIFOs */
- val = (1 << M_FIFO_RX_FLUSH_SHIFT) |
- (1 << M_FIFO_TX_FLUSH_SHIFT);
- writel(val, iproc_i2c->base + M_FIFO_CTRL_OFFSET);
- return ret;
- }
-
- /*
- * For a read operation, we now need to load the data from FIFO
- * into the memory buffer
- */
- if (msg->flags & I2C_M_RD) {
- for (i = 0; i < msg->len; i++) {
- msg->buf[i] = (readl(iproc_i2c->base + M_RX_OFFSET) >>
- M_RX_DATA_SHIFT) & M_RX_DATA_MASK;
- }
- }
-
- return 0;
+ return bcm_iproc_i2c_xfer_wait(iproc_i2c, msg, val);
}
static int bcm_iproc_i2c_xfer(struct i2c_adapter *adapter,
static uint32_t bcm_iproc_i2c_functionality(struct i2c_adapter *adap)
{
- return I2C_FUNC_I2C | I2C_FUNC_SMBUS_EMUL;
+ u32 val = I2C_FUNC_I2C | I2C_FUNC_SMBUS_EMUL;
+
+ if (adap->algo->reg_slave)
+ val |= I2C_FUNC_SLAVE;
+
+ return val;
}
-static const struct i2c_algorithm bcm_iproc_algo = {
+static struct i2c_algorithm bcm_iproc_algo = {
.master_xfer = bcm_iproc_i2c_xfer,
.functionality = bcm_iproc_i2c_functionality,
+ .reg_slave = bcm_iproc_i2c_reg_slave,
+ .unreg_slave = bcm_iproc_i2c_unreg_slave,
};
-static const struct i2c_adapter_quirks bcm_iproc_i2c_quirks = {
- /* need to reserve one byte in the FIFO for the slave address */
- .max_read_len = M_TX_RX_FIFO_SIZE - 1,
+static struct i2c_adapter_quirks bcm_iproc_i2c_quirks = {
+ .max_read_len = M_RX_MAX_READ_LEN,
};
static int bcm_iproc_i2c_cfg_speed(struct bcm_iproc_i2c_dev *iproc_i2c)
}
iproc_i2c->bus_speed = bus_speed;
- val = readl(iproc_i2c->base + TIM_CFG_OFFSET);
- val &= ~(1 << TIM_CFG_MODE_400_SHIFT);
+ val = iproc_i2c_rd_reg(iproc_i2c, TIM_CFG_OFFSET);
+ val &= ~BIT(TIM_CFG_MODE_400_SHIFT);
val |= (bus_speed == 400000) << TIM_CFG_MODE_400_SHIFT;
- writel(val, iproc_i2c->base + TIM_CFG_OFFSET);
+ iproc_i2c_wr_reg(iproc_i2c, TIM_CFG_OFFSET, val);
dev_info(iproc_i2c->device, "bus set to %u Hz\n", bus_speed);
platform_set_drvdata(pdev, iproc_i2c);
iproc_i2c->device = &pdev->dev;
+ iproc_i2c->type =
+ (enum bcm_iproc_i2c_type)of_device_get_match_data(&pdev->dev);
init_completion(&iproc_i2c->done);
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (IS_ERR(iproc_i2c->base))
return PTR_ERR(iproc_i2c->base);
+ if (iproc_i2c->type == IPROC_I2C_NIC) {
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
+ iproc_i2c->idm_base = devm_ioremap_resource(iproc_i2c->device,
+ res);
+ if (IS_ERR(iproc_i2c->idm_base))
+ return PTR_ERR(iproc_i2c->idm_base);
+
+ ret = of_property_read_u32(iproc_i2c->device->of_node,
+ "brcm,ape-hsls-addr-mask",
+ &iproc_i2c->ape_addr_mask);
+ if (ret < 0) {
+ dev_err(iproc_i2c->device,
+ "'brcm,ape-hsls-addr-mask' missing\n");
+ return -EINVAL;
+ }
+
+ spin_lock_init(&iproc_i2c->idm_lock);
+
+ /* no slave support */
+ bcm_iproc_algo.reg_slave = NULL;
+ bcm_iproc_algo.unreg_slave = NULL;
+ }
+
ret = bcm_iproc_i2c_init(iproc_i2c);
if (ret)
return ret;
return ret;
irq = platform_get_irq(pdev, 0);
- if (irq <= 0) {
- dev_err(iproc_i2c->device, "no irq resource\n");
- return irq;
- }
- iproc_i2c->irq = irq;
+ if (irq > 0) {
+ ret = devm_request_irq(iproc_i2c->device, irq,
+ bcm_iproc_i2c_isr, 0, pdev->name,
+ iproc_i2c);
+ if (ret < 0) {
+ dev_err(iproc_i2c->device,
+ "unable to request irq %i\n", irq);
+ return ret;
+ }
- ret = devm_request_irq(iproc_i2c->device, irq, bcm_iproc_i2c_isr, 0,
- pdev->name, iproc_i2c);
- if (ret < 0) {
- dev_err(iproc_i2c->device, "unable to request irq %i\n", irq);
- return ret;
+ iproc_i2c->irq = irq;
+ } else {
+ dev_warn(iproc_i2c->device,
+ "no irq resource, falling back to poll mode\n");
}
bcm_iproc_i2c_enable_disable(iproc_i2c, true);
{
struct bcm_iproc_i2c_dev *iproc_i2c = platform_get_drvdata(pdev);
- /* make sure there's no pending interrupt when we remove the adapter */
- writel(0, iproc_i2c->base + IE_OFFSET);
- readl(iproc_i2c->base + IE_OFFSET);
- synchronize_irq(iproc_i2c->irq);
+ if (iproc_i2c->irq) {
+ /*
+ * Make sure there's no pending interrupt when we remove the
+ * adapter
+ */
+ iproc_i2c_wr_reg(iproc_i2c, IE_OFFSET, 0);
+ iproc_i2c_rd_reg(iproc_i2c, IE_OFFSET);
+ synchronize_irq(iproc_i2c->irq);
+ }
i2c_del_adapter(&iproc_i2c->adapter);
bcm_iproc_i2c_enable_disable(iproc_i2c, false);
{
struct bcm_iproc_i2c_dev *iproc_i2c = dev_get_drvdata(dev);
- /* make sure there's no pending interrupt when we go into suspend */
- writel(0, iproc_i2c->base + IE_OFFSET);
- readl(iproc_i2c->base + IE_OFFSET);
- synchronize_irq(iproc_i2c->irq);
+ if (iproc_i2c->irq) {
+ /*
+ * Make sure there's no pending interrupt when we go into
+ * suspend
+ */
+ iproc_i2c_wr_reg(iproc_i2c, IE_OFFSET, 0);
+ iproc_i2c_rd_reg(iproc_i2c, IE_OFFSET);
+ synchronize_irq(iproc_i2c->irq);
+ }
/* now disable the controller */
bcm_iproc_i2c_enable_disable(iproc_i2c, false);
return ret;
/* configure to the desired bus speed */
- val = readl(iproc_i2c->base + TIM_CFG_OFFSET);
- val &= ~(1 << TIM_CFG_MODE_400_SHIFT);
+ val = iproc_i2c_rd_reg(iproc_i2c, TIM_CFG_OFFSET);
+ val &= ~BIT(TIM_CFG_MODE_400_SHIFT);
val |= (iproc_i2c->bus_speed == 400000) << TIM_CFG_MODE_400_SHIFT;
- writel(val, iproc_i2c->base + TIM_CFG_OFFSET);
+ iproc_i2c_wr_reg(iproc_i2c, TIM_CFG_OFFSET, val);
bcm_iproc_i2c_enable_disable(iproc_i2c, true);
#define BCM_IPROC_I2C_PM_OPS NULL
#endif /* CONFIG_PM_SLEEP */
+
+static int bcm_iproc_i2c_reg_slave(struct i2c_client *slave)
+{
+ struct bcm_iproc_i2c_dev *iproc_i2c = i2c_get_adapdata(slave->adapter);
+
+ if (iproc_i2c->slave)
+ return -EBUSY;
+
+ if (slave->flags & I2C_CLIENT_TEN)
+ return -EAFNOSUPPORT;
+
+ iproc_i2c->slave = slave;
+ bcm_iproc_i2c_slave_init(iproc_i2c, false);
+ return 0;
+}
+
+static int bcm_iproc_i2c_unreg_slave(struct i2c_client *slave)
+{
+ u32 tmp;
+ struct bcm_iproc_i2c_dev *iproc_i2c = i2c_get_adapdata(slave->adapter);
+
+ if (!iproc_i2c->slave)
+ return -EINVAL;
+
+ iproc_i2c->slave = NULL;
+
+ /* disable all slave interrupts */
+ tmp = iproc_i2c_rd_reg(iproc_i2c, IE_OFFSET);
+ tmp &= ~(IE_S_ALL_INTERRUPT_MASK <<
+ IE_S_ALL_INTERRUPT_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, IE_OFFSET, tmp);
+
+ /* Erase the slave address programmed */
+ tmp = iproc_i2c_rd_reg(iproc_i2c, S_CFG_SMBUS_ADDR_OFFSET);
+ tmp &= ~BIT(S_CFG_EN_NIC_SMB_ADDR3_SHIFT);
+ iproc_i2c_wr_reg(iproc_i2c, S_CFG_SMBUS_ADDR_OFFSET, tmp);
+
+ return 0;
+}
+
static const struct of_device_id bcm_iproc_i2c_of_match[] = {
- { .compatible = "brcm,iproc-i2c" },
+ {
+ .compatible = "brcm,iproc-i2c",
+ .data = (int *)IPROC_I2C,
+ }, {
+ .compatible = "brcm,iproc-nic-i2c",
+ .data = (int *)IPROC_I2C_NIC,
+ },
{ /* sentinel */ }
};
MODULE_DEVICE_TABLE(of, bcm_iproc_i2c_of_match);
struct brcmstb_i2c_dev {
struct device *device;
void __iomem *base;
- void __iomem *irq_base;
int irq;
struct bsc_regs *bsc_regmap;
struct i2c_adapter adapter;
int i2c_dw_prepare_clk(struct dw_i2c_dev *dev, bool prepare)
{
+ int ret;
+
if (IS_ERR(dev->clk))
return PTR_ERR(dev->clk);
- if (prepare)
- return clk_prepare_enable(dev->clk);
+ if (prepare) {
+ /* Optional interface clock */
+ ret = clk_prepare_enable(dev->pclk);
+ if (ret)
+ return ret;
+
+ ret = clk_prepare_enable(dev->clk);
+ if (ret)
+ clk_disable_unprepare(dev->pclk);
+
+ return ret;
+ }
clk_disable_unprepare(dev->clk);
+ clk_disable_unprepare(dev->pclk);
+
return 0;
}
EXPORT_SYMBOL_GPL(i2c_dw_prepare_clk);
* @base: IO registers pointer
* @cmd_complete: tx completion indicator
* @clk: input reference clock
+ * @pclk: clock required to access the registers
* @slave: represent an I2C slave device
* @cmd_err: run time hadware error code
* @msgs: points to an array of messages currently being transferred
void __iomem *ext;
struct completion cmd_complete;
struct clk *clk;
+ struct clk *pclk;
struct reset_control *rst;
struct i2c_client *slave;
u32 (*get_clk_rate_khz) (struct dw_i2c_dev *dev);
else
i2c_dw_configure_master(dev);
+ /* Optional interface clock */
+ dev->pclk = devm_clk_get_optional(&pdev->dev, "pclk");
+ if (IS_ERR(dev->pclk))
+ return PTR_ERR(dev->pclk);
+
dev->clk = devm_clk_get(&pdev->dev, NULL);
if (!i2c_dw_prepare_clk(dev, true)) {
u64 clk_khz;
if (gpiod_cansleep(priv->sda) || gpiod_cansleep(priv->scl))
dev_warn(dev, "Slow GPIO pins might wreak havoc into I2C/SMBus bus timing");
+ else
+ bit_data->can_do_atomic = true;
bit_data->setsda = i2c_gpio_setsda_val;
bit_data->setscl = i2c_gpio_setscl_val;
return 0;
}
-#ifdef CONFIG_PM_SLEEP
-static int lpi2c_runtime_suspend(struct device *dev)
+static int __maybe_unused lpi2c_runtime_suspend(struct device *dev)
{
struct lpi2c_imx_struct *lpi2c_imx = dev_get_drvdata(dev);
return 0;
}
-static int lpi2c_runtime_resume(struct device *dev)
+static int __maybe_unused lpi2c_runtime_resume(struct device *dev)
{
struct lpi2c_imx_struct *lpi2c_imx = dev_get_drvdata(dev);
int ret;
SET_RUNTIME_PM_OPS(lpi2c_runtime_suspend,
lpi2c_runtime_resume, NULL)
};
-#define IMX_LPI2C_PM (&lpi2c_pm_ops)
-#else
-#define IMX_LPI2C_PM NULL
-#endif
static struct platform_driver lpi2c_imx_driver = {
.probe = lpi2c_imx_probe,
.driver = {
.name = DRIVER_NAME,
.of_match_table = lpi2c_imx_of_match,
- .pm = IMX_LPI2C_PM,
+ .pm = &lpi2c_pm_ops,
},
};
#include <linux/ioport.h>
#include <linux/i2c.h>
#include <linux/io.h>
-#include <linux/acpi.h>
/* SCH SMBus address offsets */
#define SMBHSTCNT (0 + sch_smba)
#include <linux/slab.h>
#define I2C_RS_TRANSFER (1 << 4)
+#define I2C_ARB_LOST (1 << 3)
#define I2C_HS_NACKERR (1 << 2)
#define I2C_ACKERR (1 << 1)
#define I2C_TRANSAC_COMP (1 << 0)
#define I2C_CONTROL_DIR_CHANGE (0x1 << 4)
#define I2C_CONTROL_ACKERR_DET_EN (0x1 << 5)
#define I2C_CONTROL_TRANSFER_LEN_CHANGE (0x1 << 6)
+#define I2C_CONTROL_DMAACK_EN (0x1 << 8)
+#define I2C_CONTROL_ASYNC_MODE (0x1 << 9)
#define I2C_CONTROL_WRAPPER (0x1 << 0)
#define I2C_DRV_NAME "i2c-mt65xx"
};
enum I2C_REGS_OFFSET {
- OFFSET_DATA_PORT = 0x0,
- OFFSET_SLAVE_ADDR = 0x04,
- OFFSET_INTR_MASK = 0x08,
- OFFSET_INTR_STAT = 0x0c,
- OFFSET_CONTROL = 0x10,
- OFFSET_TRANSFER_LEN = 0x14,
- OFFSET_TRANSAC_LEN = 0x18,
- OFFSET_DELAY_LEN = 0x1c,
- OFFSET_TIMING = 0x20,
- OFFSET_START = 0x24,
- OFFSET_EXT_CONF = 0x28,
- OFFSET_FIFO_STAT = 0x30,
- OFFSET_FIFO_THRESH = 0x34,
- OFFSET_FIFO_ADDR_CLR = 0x38,
- OFFSET_IO_CONFIG = 0x40,
- OFFSET_RSV_DEBUG = 0x44,
- OFFSET_HS = 0x48,
- OFFSET_SOFTRESET = 0x50,
- OFFSET_DCM_EN = 0x54,
- OFFSET_PATH_DIR = 0x60,
- OFFSET_DEBUGSTAT = 0x64,
- OFFSET_DEBUGCTRL = 0x68,
- OFFSET_TRANSFER_LEN_AUX = 0x6c,
- OFFSET_CLOCK_DIV = 0x70,
+ OFFSET_DATA_PORT,
+ OFFSET_SLAVE_ADDR,
+ OFFSET_INTR_MASK,
+ OFFSET_INTR_STAT,
+ OFFSET_CONTROL,
+ OFFSET_TRANSFER_LEN,
+ OFFSET_TRANSAC_LEN,
+ OFFSET_DELAY_LEN,
+ OFFSET_TIMING,
+ OFFSET_START,
+ OFFSET_EXT_CONF,
+ OFFSET_FIFO_STAT,
+ OFFSET_FIFO_THRESH,
+ OFFSET_FIFO_ADDR_CLR,
+ OFFSET_IO_CONFIG,
+ OFFSET_RSV_DEBUG,
+ OFFSET_HS,
+ OFFSET_SOFTRESET,
+ OFFSET_DCM_EN,
+ OFFSET_PATH_DIR,
+ OFFSET_DEBUGSTAT,
+ OFFSET_DEBUGCTRL,
+ OFFSET_TRANSFER_LEN_AUX,
+ OFFSET_CLOCK_DIV,
+ OFFSET_LTIMING,
+};
+
+static const u16 mt_i2c_regs_v1[] = {
+ [OFFSET_DATA_PORT] = 0x0,
+ [OFFSET_SLAVE_ADDR] = 0x4,
+ [OFFSET_INTR_MASK] = 0x8,
+ [OFFSET_INTR_STAT] = 0xc,
+ [OFFSET_CONTROL] = 0x10,
+ [OFFSET_TRANSFER_LEN] = 0x14,
+ [OFFSET_TRANSAC_LEN] = 0x18,
+ [OFFSET_DELAY_LEN] = 0x1c,
+ [OFFSET_TIMING] = 0x20,
+ [OFFSET_START] = 0x24,
+ [OFFSET_EXT_CONF] = 0x28,
+ [OFFSET_FIFO_STAT] = 0x30,
+ [OFFSET_FIFO_THRESH] = 0x34,
+ [OFFSET_FIFO_ADDR_CLR] = 0x38,
+ [OFFSET_IO_CONFIG] = 0x40,
+ [OFFSET_RSV_DEBUG] = 0x44,
+ [OFFSET_HS] = 0x48,
+ [OFFSET_SOFTRESET] = 0x50,
+ [OFFSET_DCM_EN] = 0x54,
+ [OFFSET_PATH_DIR] = 0x60,
+ [OFFSET_DEBUGSTAT] = 0x64,
+ [OFFSET_DEBUGCTRL] = 0x68,
+ [OFFSET_TRANSFER_LEN_AUX] = 0x6c,
+ [OFFSET_CLOCK_DIV] = 0x70,
+};
+
+static const u16 mt_i2c_regs_v2[] = {
+ [OFFSET_DATA_PORT] = 0x0,
+ [OFFSET_SLAVE_ADDR] = 0x4,
+ [OFFSET_INTR_MASK] = 0x8,
+ [OFFSET_INTR_STAT] = 0xc,
+ [OFFSET_CONTROL] = 0x10,
+ [OFFSET_TRANSFER_LEN] = 0x14,
+ [OFFSET_TRANSAC_LEN] = 0x18,
+ [OFFSET_DELAY_LEN] = 0x1c,
+ [OFFSET_TIMING] = 0x20,
+ [OFFSET_START] = 0x24,
+ [OFFSET_EXT_CONF] = 0x28,
+ [OFFSET_LTIMING] = 0x2c,
+ [OFFSET_HS] = 0x30,
+ [OFFSET_IO_CONFIG] = 0x34,
+ [OFFSET_FIFO_ADDR_CLR] = 0x38,
+ [OFFSET_TRANSFER_LEN_AUX] = 0x44,
+ [OFFSET_CLOCK_DIV] = 0x48,
+ [OFFSET_SOFTRESET] = 0x50,
+ [OFFSET_DEBUGSTAT] = 0xe0,
+ [OFFSET_DEBUGCTRL] = 0xe8,
+ [OFFSET_FIFO_STAT] = 0xf4,
+ [OFFSET_FIFO_THRESH] = 0xf8,
+ [OFFSET_DCM_EN] = 0xf88,
};
struct mtk_i2c_compatible {
const struct i2c_adapter_quirks *quirks;
+ const u16 *regs;
unsigned char pmic_i2c: 1;
unsigned char dcm: 1;
unsigned char auto_restart: 1;
unsigned char aux_len_reg: 1;
unsigned char support_33bits: 1;
unsigned char timing_adjust: 1;
+ unsigned char dma_sync: 1;
+ unsigned char ltiming_adjust: 1;
};
struct mtk_i2c {
struct clk *clk_main; /* main clock for i2c bus */
struct clk *clk_dma; /* DMA clock for i2c via DMA */
struct clk *clk_pmic; /* PMIC clock for i2c from PMIC */
+ struct clk *clk_arb; /* Arbitrator clock for i2c */
bool have_pmic; /* can use i2c pins from PMIC */
bool use_push_pull; /* IO config push-pull mode */
enum mtk_trans_op op;
u16 timing_reg;
u16 high_speed_reg;
+ u16 ltiming_reg;
unsigned char auto_restart;
bool ignore_restart_irq;
const struct mtk_i2c_compatible *dev_comp;
};
static const struct mtk_i2c_compatible mt2712_compat = {
+ .regs = mt_i2c_regs_v1,
.pmic_i2c = 0,
.dcm = 1,
.auto_restart = 1,
.aux_len_reg = 1,
.support_33bits = 1,
.timing_adjust = 1,
+ .dma_sync = 0,
+ .ltiming_adjust = 0,
};
static const struct mtk_i2c_compatible mt6577_compat = {
.quirks = &mt6577_i2c_quirks,
+ .regs = mt_i2c_regs_v1,
.pmic_i2c = 0,
.dcm = 1,
.auto_restart = 0,
.aux_len_reg = 0,
.support_33bits = 0,
.timing_adjust = 0,
+ .dma_sync = 0,
+ .ltiming_adjust = 0,
};
static const struct mtk_i2c_compatible mt6589_compat = {
.quirks = &mt6577_i2c_quirks,
+ .regs = mt_i2c_regs_v1,
.pmic_i2c = 1,
.dcm = 0,
.auto_restart = 0,
.aux_len_reg = 0,
.support_33bits = 0,
.timing_adjust = 0,
+ .dma_sync = 0,
+ .ltiming_adjust = 0,
};
static const struct mtk_i2c_compatible mt7622_compat = {
.quirks = &mt7622_i2c_quirks,
+ .regs = mt_i2c_regs_v1,
.pmic_i2c = 0,
.dcm = 1,
.auto_restart = 1,
.aux_len_reg = 1,
.support_33bits = 0,
.timing_adjust = 0,
+ .dma_sync = 0,
+ .ltiming_adjust = 0,
};
static const struct mtk_i2c_compatible mt8173_compat = {
+ .regs = mt_i2c_regs_v1,
.pmic_i2c = 0,
.dcm = 1,
.auto_restart = 1,
.aux_len_reg = 1,
.support_33bits = 1,
.timing_adjust = 0,
+ .dma_sync = 0,
+ .ltiming_adjust = 0,
+};
+
+static const struct mtk_i2c_compatible mt8183_compat = {
+ .regs = mt_i2c_regs_v2,
+ .pmic_i2c = 0,
+ .dcm = 0,
+ .auto_restart = 1,
+ .aux_len_reg = 1,
+ .support_33bits = 1,
+ .timing_adjust = 1,
+ .dma_sync = 1,
+ .ltiming_adjust = 1,
};
static const struct of_device_id mtk_i2c_of_match[] = {
{ .compatible = "mediatek,mt6589-i2c", .data = &mt6589_compat },
{ .compatible = "mediatek,mt7622-i2c", .data = &mt7622_compat },
{ .compatible = "mediatek,mt8173-i2c", .data = &mt8173_compat },
+ { .compatible = "mediatek,mt8183-i2c", .data = &mt8183_compat },
{}
};
MODULE_DEVICE_TABLE(of, mtk_i2c_of_match);
+static u16 mtk_i2c_readw(struct mtk_i2c *i2c, enum I2C_REGS_OFFSET reg)
+{
+ return readw(i2c->base + i2c->dev_comp->regs[reg]);
+}
+
+static void mtk_i2c_writew(struct mtk_i2c *i2c, u16 val,
+ enum I2C_REGS_OFFSET reg)
+{
+ writew(val, i2c->base + i2c->dev_comp->regs[reg]);
+}
+
static int mtk_i2c_clock_enable(struct mtk_i2c *i2c)
{
int ret;
if (ret)
goto err_pmic;
}
+
+ if (i2c->clk_arb) {
+ ret = clk_prepare_enable(i2c->clk_arb);
+ if (ret)
+ goto err_arb;
+ }
+
return 0;
+err_arb:
+ if (i2c->have_pmic)
+ clk_disable_unprepare(i2c->clk_pmic);
err_pmic:
clk_disable_unprepare(i2c->clk_main);
err_main:
static void mtk_i2c_clock_disable(struct mtk_i2c *i2c)
{
+ if (i2c->clk_arb)
+ clk_disable_unprepare(i2c->clk_arb);
+
if (i2c->have_pmic)
clk_disable_unprepare(i2c->clk_pmic);
{
u16 control_reg;
- writew(I2C_SOFT_RST, i2c->base + OFFSET_SOFTRESET);
+ mtk_i2c_writew(i2c, I2C_SOFT_RST, OFFSET_SOFTRESET);
/* Set ioconfig */
if (i2c->use_push_pull)
- writew(I2C_IO_CONFIG_PUSH_PULL, i2c->base + OFFSET_IO_CONFIG);
+ mtk_i2c_writew(i2c, I2C_IO_CONFIG_PUSH_PULL, OFFSET_IO_CONFIG);
else
- writew(I2C_IO_CONFIG_OPEN_DRAIN, i2c->base + OFFSET_IO_CONFIG);
+ mtk_i2c_writew(i2c, I2C_IO_CONFIG_OPEN_DRAIN, OFFSET_IO_CONFIG);
if (i2c->dev_comp->dcm)
- writew(I2C_DCM_DISABLE, i2c->base + OFFSET_DCM_EN);
+ mtk_i2c_writew(i2c, I2C_DCM_DISABLE, OFFSET_DCM_EN);
if (i2c->dev_comp->timing_adjust)
- writew(I2C_DEFAULT_CLK_DIV - 1, i2c->base + OFFSET_CLOCK_DIV);
+ mtk_i2c_writew(i2c, I2C_DEFAULT_CLK_DIV - 1, OFFSET_CLOCK_DIV);
- writew(i2c->timing_reg, i2c->base + OFFSET_TIMING);
- writew(i2c->high_speed_reg, i2c->base + OFFSET_HS);
+ mtk_i2c_writew(i2c, i2c->timing_reg, OFFSET_TIMING);
+ mtk_i2c_writew(i2c, i2c->high_speed_reg, OFFSET_HS);
+ if (i2c->dev_comp->ltiming_adjust)
+ mtk_i2c_writew(i2c, i2c->ltiming_reg, OFFSET_LTIMING);
/* If use i2c pin from PMIC mt6397 side, need set PATH_DIR first */
if (i2c->have_pmic)
- writew(I2C_CONTROL_WRAPPER, i2c->base + OFFSET_PATH_DIR);
+ mtk_i2c_writew(i2c, I2C_CONTROL_WRAPPER, OFFSET_PATH_DIR);
control_reg = I2C_CONTROL_ACKERR_DET_EN |
I2C_CONTROL_CLK_EXT_EN | I2C_CONTROL_DMA_EN;
- writew(control_reg, i2c->base + OFFSET_CONTROL);
- writew(I2C_DELAY_LEN, i2c->base + OFFSET_DELAY_LEN);
+ if (i2c->dev_comp->dma_sync)
+ control_reg |= I2C_CONTROL_DMAACK_EN | I2C_CONTROL_ASYNC_MODE;
+
+ mtk_i2c_writew(i2c, control_reg, OFFSET_CONTROL);
+ mtk_i2c_writew(i2c, I2C_DELAY_LEN, OFFSET_DELAY_LEN);
writel(I2C_DMA_HARD_RST, i2c->pdmabase + OFFSET_RST);
udelay(50);
unsigned int clk_src;
unsigned int step_cnt;
unsigned int sample_cnt;
+ unsigned int l_step_cnt;
+ unsigned int l_sample_cnt;
unsigned int target_speed;
int ret;
if (target_speed > MAX_FS_MODE_SPEED) {
/* Set master code speed register */
ret = mtk_i2c_calculate_speed(i2c, clk_src, MAX_FS_MODE_SPEED,
- &step_cnt, &sample_cnt);
+ &l_step_cnt, &l_sample_cnt);
if (ret < 0)
return ret;
- i2c->timing_reg = (sample_cnt << 8) | step_cnt;
+ i2c->timing_reg = (l_sample_cnt << 8) | l_step_cnt;
/* Set the high speed mode register */
ret = mtk_i2c_calculate_speed(i2c, clk_src, target_speed,
i2c->high_speed_reg = I2C_TIME_DEFAULT_VALUE |
(sample_cnt << 12) | (step_cnt << 8);
+
+ if (i2c->dev_comp->ltiming_adjust)
+ i2c->ltiming_reg = (l_sample_cnt << 6) | l_step_cnt |
+ (sample_cnt << 12) | (step_cnt << 9);
} else {
ret = mtk_i2c_calculate_speed(i2c, clk_src, target_speed,
&step_cnt, &sample_cnt);
/* Disable the high speed transaction */
i2c->high_speed_reg = I2C_TIME_CLR_VALUE;
+
+ if (i2c->dev_comp->ltiming_adjust)
+ i2c->ltiming_reg = (sample_cnt << 6) | step_cnt;
}
return 0;
reinit_completion(&i2c->msg_complete);
- control_reg = readw(i2c->base + OFFSET_CONTROL) &
+ control_reg = mtk_i2c_readw(i2c, OFFSET_CONTROL) &
~(I2C_CONTROL_DIR_CHANGE | I2C_CONTROL_RS);
if ((i2c->speed_hz > MAX_FS_MODE_SPEED) || (left_num >= 1))
control_reg |= I2C_CONTROL_RS;
if (i2c->op == I2C_MASTER_WRRD)
control_reg |= I2C_CONTROL_DIR_CHANGE | I2C_CONTROL_RS;
- writew(control_reg, i2c->base + OFFSET_CONTROL);
+ mtk_i2c_writew(i2c, control_reg, OFFSET_CONTROL);
/* set start condition */
if (i2c->speed_hz <= I2C_DEFAULT_SPEED)
- writew(I2C_ST_START_CON, i2c->base + OFFSET_EXT_CONF);
+ mtk_i2c_writew(i2c, I2C_ST_START_CON, OFFSET_EXT_CONF);
else
- writew(I2C_FS_START_CON, i2c->base + OFFSET_EXT_CONF);
+ mtk_i2c_writew(i2c, I2C_FS_START_CON, OFFSET_EXT_CONF);
addr_reg = i2c_8bit_addr_from_msg(msgs);
- writew(addr_reg, i2c->base + OFFSET_SLAVE_ADDR);
+ mtk_i2c_writew(i2c, addr_reg, OFFSET_SLAVE_ADDR);
/* Clear interrupt status */
- writew(restart_flag | I2C_HS_NACKERR | I2C_ACKERR |
- I2C_TRANSAC_COMP, i2c->base + OFFSET_INTR_STAT);
- writew(I2C_FIFO_ADDR_CLR, i2c->base + OFFSET_FIFO_ADDR_CLR);
+ mtk_i2c_writew(i2c, restart_flag | I2C_HS_NACKERR | I2C_ACKERR |
+ I2C_ARB_LOST | I2C_TRANSAC_COMP, OFFSET_INTR_STAT);
+
+ mtk_i2c_writew(i2c, I2C_FIFO_ADDR_CLR, OFFSET_FIFO_ADDR_CLR);
/* Enable interrupt */
- writew(restart_flag | I2C_HS_NACKERR | I2C_ACKERR |
- I2C_TRANSAC_COMP, i2c->base + OFFSET_INTR_MASK);
+ mtk_i2c_writew(i2c, restart_flag | I2C_HS_NACKERR | I2C_ACKERR |
+ I2C_ARB_LOST | I2C_TRANSAC_COMP, OFFSET_INTR_MASK);
/* Set transfer and transaction len */
if (i2c->op == I2C_MASTER_WRRD) {
if (i2c->dev_comp->aux_len_reg) {
- writew(msgs->len, i2c->base + OFFSET_TRANSFER_LEN);
- writew((msgs + 1)->len, i2c->base +
- OFFSET_TRANSFER_LEN_AUX);
+ mtk_i2c_writew(i2c, msgs->len, OFFSET_TRANSFER_LEN);
+ mtk_i2c_writew(i2c, (msgs + 1)->len,
+ OFFSET_TRANSFER_LEN_AUX);
} else {
- writew(msgs->len | ((msgs + 1)->len) << 8,
- i2c->base + OFFSET_TRANSFER_LEN);
+ mtk_i2c_writew(i2c, msgs->len | ((msgs + 1)->len) << 8,
+ OFFSET_TRANSFER_LEN);
}
- writew(I2C_WRRD_TRANAC_VALUE, i2c->base + OFFSET_TRANSAC_LEN);
+ mtk_i2c_writew(i2c, I2C_WRRD_TRANAC_VALUE, OFFSET_TRANSAC_LEN);
} else {
- writew(msgs->len, i2c->base + OFFSET_TRANSFER_LEN);
- writew(num, i2c->base + OFFSET_TRANSAC_LEN);
+ mtk_i2c_writew(i2c, msgs->len, OFFSET_TRANSFER_LEN);
+ mtk_i2c_writew(i2c, num, OFFSET_TRANSAC_LEN);
}
/* Prepare buffer data to start transfer */
if (left_num >= 1)
start_reg |= I2C_RS_MUL_CNFG;
}
- writew(start_reg, i2c->base + OFFSET_START);
+ mtk_i2c_writew(i2c, start_reg, OFFSET_START);
ret = wait_for_completion_timeout(&i2c->msg_complete,
i2c->adap.timeout);
/* Clear interrupt mask */
- writew(~(restart_flag | I2C_HS_NACKERR | I2C_ACKERR |
- I2C_TRANSAC_COMP), i2c->base + OFFSET_INTR_MASK);
+ mtk_i2c_writew(i2c, ~(restart_flag | I2C_HS_NACKERR | I2C_ACKERR |
+ I2C_ARB_LOST | I2C_TRANSAC_COMP), OFFSET_INTR_MASK);
if (i2c->op == I2C_MASTER_WR) {
dma_unmap_single(i2c->dev, wpaddr,
if (i2c->auto_restart)
restart_flag = I2C_RS_TRANSFER;
- intr_stat = readw(i2c->base + OFFSET_INTR_STAT);
- writew(intr_stat, i2c->base + OFFSET_INTR_STAT);
+ intr_stat = mtk_i2c_readw(i2c, OFFSET_INTR_STAT);
+ mtk_i2c_writew(i2c, intr_stat, OFFSET_INTR_STAT);
/*
* when occurs ack error, i2c controller generate two interrupts
if (i2c->ignore_restart_irq && (i2c->irq_stat & restart_flag)) {
i2c->ignore_restart_irq = false;
i2c->irq_stat = 0;
- writew(I2C_RS_MUL_CNFG | I2C_RS_MUL_TRIG | I2C_TRANSAC_START,
- i2c->base + OFFSET_START);
+ mtk_i2c_writew(i2c, I2C_RS_MUL_CNFG | I2C_RS_MUL_TRIG |
+ I2C_TRANSAC_START, OFFSET_START);
} else {
if (i2c->irq_stat & (I2C_TRANSAC_COMP | restart_flag))
complete(&i2c->msg_complete);
return PTR_ERR(i2c->clk_dma);
}
+ i2c->clk_arb = devm_clk_get(&pdev->dev, "arb");
+ if (IS_ERR(i2c->clk_arb))
+ i2c->clk_arb = NULL;
+
clk = i2c->clk_main;
if (i2c->have_pmic) {
i2c->clk_pmic = devm_clk_get(&pdev->dev, "pmic");
/* disable the controller */
i2c_clr_bit(dev->virtbase + I2C_CR, I2C_CR_PE);
clk_disable_unprepare(dev->clk);
- if (res)
- release_mem_region(res->start, resource_size(res));
+ release_mem_region(res->start, resource_size(res));
return 0;
}
#include <linux/spinlock.h>
#include <linux/jiffies.h>
-#define OCORES_FLAG_POLL BIT(0)
-
/*
* 'process_lock' exists because ocores_process() and ocores_process_timeout()
* can't run in parallel.
int iobase;
u32 reg_shift;
u32 reg_io_width;
- unsigned long flags;
wait_queue_head_t wait;
struct i2c_adapter adap;
struct i2c_msg *msg;
static int ocores_xfer(struct i2c_adapter *adap,
struct i2c_msg *msgs, int num)
{
- struct ocores_i2c *i2c = i2c_get_adapdata(adap);
-
- if (i2c->flags & OCORES_FLAG_POLL)
- return ocores_xfer_polling(adap, msgs, num);
- return ocores_xfer_core(i2c, msgs, num, false);
+ return ocores_xfer_core(i2c_get_adapdata(adap), msgs, num, false);
}
static int ocores_init(struct device *dev, struct ocores_i2c *i2c)
return I2C_FUNC_I2C | I2C_FUNC_SMBUS_EMUL;
}
-static const struct i2c_algorithm ocores_algorithm = {
+static struct i2c_algorithm ocores_algorithm = {
.master_xfer = ocores_xfer,
+ .master_xfer_atomic = ocores_xfer_polling,
.functionality = ocores_func,
};
irq = platform_get_irq(pdev, 0);
if (irq == -ENXIO) {
- i2c->flags |= OCORES_FLAG_POLL;
+ ocores_algorithm.master_xfer = ocores_xfer_polling;
} else {
if (irq < 0)
return irq;
}
- if (!(i2c->flags & OCORES_FLAG_POLL)) {
+ if (ocores_algorithm.master_xfer != ocores_xfer_polling) {
ret = devm_request_irq(&pdev->dev, irq, ocores_isr, 0,
pdev->name, i2c);
if (ret) {
[OMAP_I2C_IP_V2_IRQENABLE_CLR] = 0x30,
};
+static int omap_i2c_xfer_data(struct omap_i2c_dev *omap);
+
static inline void omap_i2c_write_reg(struct omap_i2c_dev *omap,
int reg, u16 val)
{
(1000 * omap->speed / 8);
}
+static void omap_i2c_wait(struct omap_i2c_dev *omap)
+{
+ u16 stat;
+ u16 mask = omap_i2c_read_reg(omap, OMAP_I2C_IE_REG);
+ int count = 0;
+
+ do {
+ stat = omap_i2c_read_reg(omap, OMAP_I2C_STAT_REG);
+ count++;
+ } while (!(stat & mask) && count < 5);
+}
+
/*
* Low level master read/write transaction.
*/
static int omap_i2c_xfer_msg(struct i2c_adapter *adap,
- struct i2c_msg *msg, int stop)
+ struct i2c_msg *msg, int stop, bool polling)
{
struct omap_i2c_dev *omap = i2c_get_adapdata(adap);
unsigned long timeout;
u16 w;
+ int ret;
dev_dbg(omap->dev, "addr: 0x%04x, len: %d, flags: 0x%x, stop: %d\n",
msg->addr, msg->len, msg->flags, stop);
w |= OMAP_I2C_BUF_RXFIF_CLR | OMAP_I2C_BUF_TXFIF_CLR;
omap_i2c_write_reg(omap, OMAP_I2C_BUF_REG, w);
- reinit_completion(&omap->cmd_complete);
+ if (!polling)
+ reinit_completion(&omap->cmd_complete);
omap->cmd_err = 0;
w = OMAP_I2C_CON_EN | OMAP_I2C_CON_MST | OMAP_I2C_CON_STT;
* REVISIT: We should abort the transfer on signals, but the bus goes
* into arbitration and we're currently unable to recover from it.
*/
- timeout = wait_for_completion_timeout(&omap->cmd_complete,
- OMAP_I2C_TIMEOUT);
+ if (!polling) {
+ timeout = wait_for_completion_timeout(&omap->cmd_complete,
+ OMAP_I2C_TIMEOUT);
+ } else {
+ do {
+ omap_i2c_wait(omap);
+ ret = omap_i2c_xfer_data(omap);
+ } while (ret == -EAGAIN);
+
+ timeout = !ret;
+ }
+
if (timeout == 0) {
dev_err(omap->dev, "controller timed out\n");
omap_i2c_reset(omap);
* to do the work during IRQ processing.
*/
static int
-omap_i2c_xfer(struct i2c_adapter *adap, struct i2c_msg msgs[], int num)
+omap_i2c_xfer_common(struct i2c_adapter *adap, struct i2c_msg msgs[], int num,
+ bool polling)
{
struct omap_i2c_dev *omap = i2c_get_adapdata(adap);
int i;
omap->set_mpu_wkup_lat(omap->dev, omap->latency);
for (i = 0; i < num; i++) {
- r = omap_i2c_xfer_msg(adap, &msgs[i], (i == (num - 1)));
+ r = omap_i2c_xfer_msg(adap, &msgs[i], (i == (num - 1)),
+ polling);
if (r != 0)
break;
}
return r;
}
+static int
+omap_i2c_xfer_irq(struct i2c_adapter *adap, struct i2c_msg msgs[], int num)
+{
+ return omap_i2c_xfer_common(adap, msgs, num, false);
+}
+
+static int
+omap_i2c_xfer_polling(struct i2c_adapter *adap, struct i2c_msg msgs[], int num)
+{
+ return omap_i2c_xfer_common(adap, msgs, num, true);
+}
+
static u32
omap_i2c_func(struct i2c_adapter *adap)
{
return ret;
}
-static irqreturn_t
-omap_i2c_isr_thread(int this_irq, void *dev_id)
+static int omap_i2c_xfer_data(struct omap_i2c_dev *omap)
{
- struct omap_i2c_dev *omap = dev_id;
u16 bits;
u16 stat;
int err = 0, count = 0;
if (!stat) {
/* my work here is done */
- goto out;
+ err = -EAGAIN;
+ break;
}
dev_dbg(omap->dev, "IRQ (ISR = 0x%04x)\n", stat);
}
} while (stat);
- omap_i2c_complete_cmd(omap, err);
+ return err;
+}
+
+static irqreturn_t
+omap_i2c_isr_thread(int this_irq, void *dev_id)
+{
+ int ret;
+ struct omap_i2c_dev *omap = dev_id;
+
+ ret = omap_i2c_xfer_data(omap);
+ if (ret != -EAGAIN)
+ omap_i2c_complete_cmd(omap, ret);
-out:
return IRQ_HANDLED;
}
static const struct i2c_algorithm omap_i2c_algo = {
- .master_xfer = omap_i2c_xfer,
+ .master_xfer = omap_i2c_xfer_irq,
+ .master_xfer_atomic = omap_i2c_xfer_polling,
.functionality = omap_i2c_func,
};
Serverworks OSB4, CSB5, CSB6, HT-1000, HT-1100
ATI IXP200, IXP300, IXP400, SB600, SB700/SP5100, SB800
AMD Hudson-2, ML, CZ
+ Hygon CZ
SMSC Victory66
Note: we assume there can only be one device, with one or more
PIIX4_dev->revision >= 0x41) ||
(PIIX4_dev->vendor == PCI_VENDOR_ID_AMD &&
PIIX4_dev->device == PCI_DEVICE_ID_AMD_KERNCZ_SMBUS &&
- PIIX4_dev->revision >= 0x49))
+ PIIX4_dev->revision >= 0x49) ||
+ (PIIX4_dev->vendor == PCI_VENDOR_ID_HYGON &&
+ PIIX4_dev->device == PCI_DEVICE_ID_AMD_KERNCZ_SMBUS))
smb_en = 0x00;
else
smb_en = (aux) ? 0x28 : 0x2c;
piix4_smba, i2ccfg >> 4);
/* Find which register is used for port selection */
- if (PIIX4_dev->vendor == PCI_VENDOR_ID_AMD) {
+ if (PIIX4_dev->vendor == PCI_VENDOR_ID_AMD ||
+ PIIX4_dev->vendor == PCI_VENDOR_ID_HYGON) {
switch (PIIX4_dev->device) {
case PCI_DEVICE_ID_AMD_KERNCZ_SMBUS:
piix4_port_sel_sb800 = SB800_PIIX4_PORT_IDX_KERNCZ;
{ PCI_DEVICE(PCI_VENDOR_ID_ATI, PCI_DEVICE_ID_ATI_SBX00_SMBUS) },
{ PCI_DEVICE(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_HUDSON2_SMBUS) },
{ PCI_DEVICE(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_KERNCZ_SMBUS) },
+ { PCI_DEVICE(PCI_VENDOR_ID_HYGON, PCI_DEVICE_ID_AMD_KERNCZ_SMBUS) },
{ PCI_DEVICE(PCI_VENDOR_ID_SERVERWORKS,
PCI_DEVICE_ID_SERVERWORKS_OSB4) },
{ PCI_DEVICE(PCI_VENDOR_ID_SERVERWORKS,
if ((dev->vendor == PCI_VENDOR_ID_ATI &&
dev->device == PCI_DEVICE_ID_ATI_SBX00_SMBUS &&
dev->revision >= 0x40) ||
- dev->vendor == PCI_VENDOR_ID_AMD) {
+ dev->vendor == PCI_VENDOR_ID_AMD ||
+ dev->vendor == PCI_VENDOR_ID_HYGON) {
bool notify_imc = false;
is_sb800 = true;
- if (dev->vendor == PCI_VENDOR_ID_AMD &&
+ if ((dev->vendor == PCI_VENDOR_ID_AMD ||
+ dev->vendor == PCI_VENDOR_ID_HYGON) &&
dev->device == PCI_DEVICE_ID_AMD_KERNCZ_SMBUS) {
u8 imc;
/* ICFBSCR */
#define TCYC17 0x0f /* 17*Tcyc delay 1st bit between SDA and SCL */
+#define RCAR_MIN_DMA_LEN 8
#define RCAR_BUS_PHASE_START (MDBS | MIE | ESG)
#define RCAR_BUS_PHASE_DATA (MDBS | MIE)
rcar_i2c_dma_unmap(priv);
}
-static void rcar_i2c_dma(struct rcar_i2c_priv *priv)
+static bool rcar_i2c_dma(struct rcar_i2c_priv *priv)
{
struct device *dev = rcar_i2c_priv_to_dev(priv);
struct i2c_msg *msg = priv->msg;
int len;
/* Do various checks to see if DMA is feasible at all */
- if (IS_ERR(chan) || msg->len < 8 || !(msg->flags & I2C_M_DMA_SAFE) ||
- (read && priv->flags & ID_P_NO_RXDMA))
- return;
+ if (IS_ERR(chan) || msg->len < RCAR_MIN_DMA_LEN ||
+ !(msg->flags & I2C_M_DMA_SAFE) || (read && priv->flags & ID_P_NO_RXDMA))
+ return false;
if (read) {
/*
dma_addr = dma_map_single(chan->device->dev, buf, len, dir);
if (dma_mapping_error(chan->device->dev, dma_addr)) {
dev_dbg(dev, "dma map failed, using PIO\n");
- return;
+ return false;
}
sg_dma_len(&priv->sg) = len;
if (!txdesc) {
dev_dbg(dev, "dma prep slave sg failed, using PIO\n");
rcar_i2c_cleanup_dma(priv);
- return;
+ return false;
}
txdesc->callback = rcar_i2c_dma_callback;
if (dma_submit_error(cookie)) {
dev_dbg(dev, "submitting dma failed, using PIO\n");
rcar_i2c_cleanup_dma(priv);
- return;
+ return false;
}
/* Enable DMA Master Received/Transmitted */
rcar_i2c_write(priv, ICDMAER, TMDMAE);
dma_async_issue_pending(chan);
+ return true;
}
static void rcar_i2c_irq_send(struct rcar_i2c_priv *priv, u32 msr)
if (!(msr & MDE))
return;
+ /* Check if DMA can be enabled and take over */
+ if (priv->pos == 1 && rcar_i2c_dma(priv))
+ return;
+
if (priv->pos < msg->len) {
/*
* Prepare next data to ICRXTX register.
*/
rcar_i2c_write(priv, ICRXTX, msg->buf[priv->pos]);
priv->pos++;
-
- /*
- * Try to use DMA to transmit the rest of the data if
- * address transfer phase just finished.
- */
- if (msr & MAT)
- rcar_i2c_dma(priv);
} else {
/*
* The last data was pushed to ICRXTX on _PREV_ empty irq.
struct i2c_timings i2c_t;
int irq, ret;
+ /* Otherwise logic will break because some bytes must always use PIO */
+ BUILD_BUG_ON_MSG(RCAR_MIN_DMA_LEN < 3, "Invalid min DMA length");
+
priv = devm_kzalloc(dev, sizeof(struct rcar_i2c_priv), GFP_KERNEL);
if (!priv)
return -ENOMEM;
#include <linux/module.h>
#include <linux/of.h>
#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
#define RIIC_ICCR1 0x00
#define RIIC_ICCR2 0x04
{
struct riic_dev *riic = i2c_get_adapdata(adap);
unsigned long time_left;
- int i, ret;
+ int i;
u8 start_bit;
- ret = clk_prepare_enable(riic->clk);
- if (ret)
- return ret;
+ pm_runtime_get_sync(adap->dev.parent);
if (readb(riic->base + RIIC_ICCR2) & ICCR2_BBSY) {
riic->err = -EBUSY;
}
out:
- clk_disable_unprepare(riic->clk);
+ pm_runtime_put(adap->dev.parent);
return riic->err ?: num;
}
static int riic_init_hw(struct riic_dev *riic, struct i2c_timings *t)
{
- int ret;
+ int ret = 0;
unsigned long rate;
int total_ticks, cks, brl, brh;
- ret = clk_prepare_enable(riic->clk);
- if (ret)
- return ret;
+ pm_runtime_get_sync(riic->adapter.dev.parent);
if (t->bus_freq_hz > 400000) {
dev_err(&riic->adapter.dev,
"unsupported bus speed (%dHz). 400000 max\n",
t->bus_freq_hz);
- clk_disable_unprepare(riic->clk);
- return -EINVAL;
+ ret = -EINVAL;
+ goto out;
}
rate = clk_get_rate(riic->clk);
if (brl > (0x1F + 3)) {
dev_err(&riic->adapter.dev, "invalid speed (%lu). Too slow.\n",
(unsigned long)t->bus_freq_hz);
- clk_disable_unprepare(riic->clk);
- return -EINVAL;
+ ret = -EINVAL;
+ goto out;
}
brh = total_ticks - brl;
riic_clear_set_bit(riic, ICCR1_IICRST, 0, RIIC_ICCR1);
- clk_disable_unprepare(riic->clk);
-
- return 0;
+out:
+ pm_runtime_put(riic->adapter.dev.parent);
+ return ret;
}
static struct riic_irq_desc riic_irqs[] = {
i2c_parse_fw_timings(&pdev->dev, &i2c_t, true);
+ pm_runtime_enable(&pdev->dev);
+
ret = riic_init_hw(riic, &i2c_t);
if (ret)
- return ret;
-
+ goto out;
ret = i2c_add_adapter(adap);
if (ret)
- return ret;
+ goto out;
platform_set_drvdata(pdev, riic);
dev_info(&pdev->dev, "registered with %dHz bus speed\n",
i2c_t.bus_freq_hz);
return 0;
+
+out:
+ pm_runtime_disable(&pdev->dev);
+ return ret;
}
static int riic_i2c_remove(struct platform_device *pdev)
{
struct riic_dev *riic = platform_get_drvdata(pdev);
+ pm_runtime_get_sync(&pdev->dev);
writeb(0, riic->base + RIIC_ICIER);
+ pm_runtime_put(&pdev->dev);
i2c_del_adapter(&riic->adapter);
+ pm_runtime_disable(&pdev->dev);
return 0;
}
list_add_tail(&v->node,
&solutions);
+ break;
}
}
+
+ if (p_prev == p)
+ break;
}
}
{
int ret;
- if (unlikely(irqs_disabled())) {
- /* TODO: implement polling for this case if need be. */
- WARN(1, "irqs are disabled, cannot poll for event\n");
- return -EIO;
- }
-
/* Lock command issue, fill in an event we wait for */
spin_lock_irq(&dev->cmd_issue_lock);
init_completion(&dev->cmd_complete);
{
int ret;
- if (unlikely(irqs_disabled())) {
- /* TODO: implement polling for this case if need be. */
- dev_err(&dev->pdev->dev, "irqs are disabled on this "
- "system!\n");
- return -EIO;
- }
-
/* Is it already here? */
spin_lock_irq(&dev->cmd_issue_lock);
dev->cmd_err = STU300_ERROR_NONE;
return num;
}
+static int stu300_xfer_todo(struct i2c_adapter *adap, struct i2c_msg *msgs, int num)
+{
+ /* TODO: implement polling for this case if need be. */
+ WARN(1, "%s: atomic transfers not implemented\n", dev_name(&adap->dev));
+ return -EOPNOTSUPP;
+}
+
static u32 stu300_func(struct i2c_adapter *adap)
{
/* This is the simplest thing you can think of... */
}
static const struct i2c_algorithm stu300_algo = {
- .master_xfer = stu300_xfer,
- .functionality = stu300_func,
+ .master_xfer = stu300_xfer,
+ .master_xfer_atomic = stu300_xfer_todo,
+ .functionality = stu300_func,
};
static const struct i2c_adapter_quirks stu300_quirks = {
static int tegra_bpmp_i2c_msg_xfer(struct tegra_bpmp_i2c *i2c,
struct mrq_i2c_request *request,
- struct mrq_i2c_response *response)
+ struct mrq_i2c_response *response,
+ bool atomic)
{
struct tegra_bpmp_message msg;
int err;
msg.rx.data = response;
msg.rx.size = sizeof(*response);
- if (irqs_disabled())
+ if (atomic)
err = tegra_bpmp_transfer_atomic(i2c->bpmp, &msg);
else
err = tegra_bpmp_transfer(i2c->bpmp, &msg);
return err;
}
-static int tegra_bpmp_i2c_xfer(struct i2c_adapter *adapter,
- struct i2c_msg *msgs, int num)
+static int tegra_bpmp_i2c_xfer_common(struct i2c_adapter *adapter,
+ struct i2c_msg *msgs, int num,
+ bool atomic)
{
struct tegra_bpmp_i2c *i2c = i2c_get_adapdata(adapter);
struct mrq_i2c_response response;
return err;
}
- err = tegra_bpmp_i2c_msg_xfer(i2c, &request, &response);
+ err = tegra_bpmp_i2c_msg_xfer(i2c, &request, &response, atomic);
if (err < 0) {
dev_err(i2c->dev, "failed to transfer message: %d\n", err);
return err;
return num;
}
+static int tegra_bpmp_i2c_xfer(struct i2c_adapter *adapter,
+ struct i2c_msg *msgs, int num)
+{
+ return tegra_bpmp_i2c_xfer_common(adapter, msgs, num, false);
+}
+
+static int tegra_bpmp_i2c_xfer_atomic(struct i2c_adapter *adapter,
+ struct i2c_msg *msgs, int num)
+{
+ return tegra_bpmp_i2c_xfer_common(adapter, msgs, num, true);
+}
+
static u32 tegra_bpmp_i2c_func(struct i2c_adapter *adapter)
{
return I2C_FUNC_I2C | I2C_FUNC_SMBUS_EMUL | I2C_FUNC_10BIT_ADDR |
static const struct i2c_algorithm tegra_bpmp_i2c_algo = {
.master_xfer = tegra_bpmp_i2c_xfer,
+ .master_xfer_atomic = tegra_bpmp_i2c_xfer_atomic,
.functionality = tegra_bpmp_i2c_func,
};
if (WARN_ON(!msgs || num < 1))
return -EINVAL;
- if (WARN_ON(test_bit(I2C_ALF_IS_SUSPENDED, &adap->locked_flags)))
- return -ESHUTDOWN;
+
+ ret = __i2c_check_suspended(adap);
+ if (ret)
+ return ret;
if (adap->quirks && i2c_check_for_quirks(adap, msgs, num))
return -EOPNOTSUPP;
/* Retry automatically on arbitration loss */
orig_jiffies = jiffies;
for (ret = 0, try = 0; try <= adap->retries; try++) {
- ret = adap->algo->master_xfer(adap, msgs, num);
+ if (i2c_in_atomic_xfer_mode() && adap->algo->master_xfer_atomic)
+ ret = adap->algo->master_xfer_atomic(adap, msgs, num);
+ else
+ ret = adap->algo->master_xfer(adap, msgs, num);
+
if (ret != -EAGAIN)
break;
if (time_after(jiffies, orig_jiffies + adap->timeout))
* one (discarding status on the second message) or errno
* (discarding status on the first one).
*/
- if (in_atomic() || irqs_disabled()) {
- ret = i2c_trylock_bus(adap, I2C_LOCK_SEGMENT);
- if (!ret)
- /* I2C activity is ongoing. */
- return -EAGAIN;
- } else {
- i2c_lock_bus(adap, I2C_LOCK_SEGMENT);
- }
+ ret = __i2c_lock_bus_helper(adap);
+ if (ret)
+ return ret;
ret = __i2c_transfer(adap, msgs, num);
i2c_unlock_bus(adap, I2C_LOCK_SEGMENT);
#include <linux/i2c-smbus.h>
#include <linux/slab.h>
+#include "i2c-core.h"
+
#define CREATE_TRACE_POINTS
#include <trace/events/smbus.h>
{
s32 res;
- i2c_lock_bus(adapter, I2C_LOCK_SEGMENT);
+ res = __i2c_lock_bus_helper(adapter);
+ if (res)
+ return res;
+
res = __i2c_smbus_xfer(adapter, addr, flags, read_write,
command, protocol, data);
i2c_unlock_bus(adapter, I2C_LOCK_SEGMENT);
unsigned short flags, char read_write,
u8 command, int protocol, union i2c_smbus_data *data)
{
+ int (*xfer_func)(struct i2c_adapter *adap, u16 addr,
+ unsigned short flags, char read_write,
+ u8 command, int size, union i2c_smbus_data *data);
unsigned long orig_jiffies;
int try;
s32 res;
+ res = __i2c_check_suspended(adapter);
+ if (res)
+ return res;
+
/* If enabled, the following two tracepoints are conditional on
* read_write and protocol.
*/
flags &= I2C_M_TEN | I2C_CLIENT_PEC | I2C_CLIENT_SCCB;
- if (adapter->algo->smbus_xfer) {
+ xfer_func = adapter->algo->smbus_xfer;
+ if (i2c_in_atomic_xfer_mode()) {
+ if (adapter->algo->smbus_xfer_atomic)
+ xfer_func = adapter->algo->smbus_xfer_atomic;
+ else if (adapter->algo->master_xfer_atomic)
+ xfer_func = NULL; /* fallback to I2C emulation */
+ }
+
+ if (xfer_func) {
/* Retry automatically on arbitration loss */
orig_jiffies = jiffies;
for (res = 0, try = 0; try <= adapter->retries; try++) {
- res = adapter->algo->smbus_xfer(adapter, addr, flags,
- read_write, command,
- protocol, data);
+ res = xfer_func(adapter, addr, flags, read_write,
+ command, protocol, data);
if (res != -EAGAIN)
break;
if (time_after(jiffies,
int i2c_check_7bit_addr_validity_strict(unsigned short addr);
+/*
+ * We only allow atomic transfers for very late communication, e.g. to send
+ * the powerdown command to a PMIC. Atomic transfers are a corner case and not
+ * for generic use!
+ */
+static inline bool i2c_in_atomic_xfer_mode(void)
+{
+ return system_state > SYSTEM_RUNNING && irqs_disabled();
+}
+
+static inline int __i2c_lock_bus_helper(struct i2c_adapter *adap)
+{
+ int ret = 0;
+
+ if (i2c_in_atomic_xfer_mode()) {
+ WARN(!adap->algo->master_xfer_atomic && !adap->algo->smbus_xfer_atomic,
+ "No atomic I2C transfer handler for '%s'\n", dev_name(&adap->dev));
+ ret = i2c_trylock_bus(adap, I2C_LOCK_SEGMENT) ? 0 : -EAGAIN;
+ } else {
+ i2c_lock_bus(adap, I2C_LOCK_SEGMENT);
+ }
+
+ return ret;
+}
+
+static inline int __i2c_check_suspended(struct i2c_adapter *adap)
+{
+ if (test_bit(I2C_ALF_IS_SUSPENDED, &adap->locked_flags)) {
+ if (!test_and_set_bit(I2C_ALF_SUSPEND_REPORTED, &adap->locked_flags))
+ dev_WARN(&adap->dev, "Transfer while suspended\n");
+ return -ESHUTDOWN;
+ }
+
+ return 0;
+}
+
#ifdef CONFIG_ACPI
const struct acpi_device_id *
i2c_acpi_match_device(const struct acpi_device_id *matches,
else
priv->algo.master_xfer = __i2c_mux_master_xfer;
}
+ if (parent->algo->master_xfer_atomic)
+ priv->algo.master_xfer_atomic = priv->algo.master_xfer;
+
if (parent->algo->smbus_xfer) {
if (muxc->mux_locked)
priv->algo.smbus_xfer = i2c_mux_smbus_xfer;
else
priv->algo.smbus_xfer = __i2c_mux_smbus_xfer;
}
+ if (parent->algo->smbus_xfer_atomic)
+ priv->algo.smbus_xfer_atomic = priv->algo.smbus_xfer;
+
priv->algo.functionality = i2c_mux_functionality;
/* Now fill out new adapter structure */
/* Now fill out current adapter structure. cur_chan must be up to date */
priv->algo.master_xfer = i2c_demux_master_xfer;
+ if (adap->algo->master_xfer_atomic)
+ priv->algo.master_xfer_atomic = i2c_demux_master_xfer;
priv->algo.functionality = i2c_demux_functionality;
snprintf(priv->cur_adap.name, sizeof(priv->cur_adap.name),
return -EINVAL;
}
- priv = devm_kzalloc(&pdev->dev, sizeof(*priv)
- + num_chan * sizeof(struct i2c_demux_pinctrl_chan), GFP_KERNEL);
+ priv = devm_kzalloc(&pdev->dev, struct_size(priv, chan, num_chan),
+ GFP_KERNEL);
props = devm_kcalloc(&pdev->dev, num_chan, sizeof(*props), GFP_KERNEL);
#include <linux/i2c-mux.h>
#include <linux/jiffies.h>
#include <linux/module.h>
-#include <linux/platform_data/pca954x.h>
#include <linux/slab.h>
/*
const struct i2c_device_id *id)
{
struct i2c_adapter *adap = client->adapter;
- struct pca954x_platform_data *pdata = dev_get_platdata(&client->dev);
struct i2c_mux_core *muxc;
struct pca9541 *data;
- int force;
int ret;
if (!i2c_check_functionality(adap, I2C_FUNC_SMBUS_BYTE_DATA))
/* Create mux adapter */
- force = 0;
- if (pdata)
- force = pdata->modes[0].adap_id;
muxc = i2c_mux_alloc(adap, &client->dev, 1, sizeof(*data),
I2C_MUX_ARBITRATOR,
pca9541_select_chan, pca9541_release_chan);
i2c_set_clientdata(client, muxc);
- ret = i2c_mux_add_adapter(muxc, force, 0, 0);
+ ret = i2c_mux_add_adapter(muxc, 0, 0, 0);
if (ret)
return ret;
#include <linux/of.h>
#include <linux/of_device.h>
#include <linux/of_irq.h>
-#include <linux/platform_data/pca954x.h>
#include <linux/pm.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
+#include <dt-bindings/mux/mux.h>
#define PCA954X_MAX_NCHANS 8
const struct chip_desc *chip;
u8 last_chan; /* last register value */
- u8 deselect;
+ /* MUX_IDLE_AS_IS, MUX_IDLE_DISCONNECT or >= 0 for channel */
+ s8 idle_state;
+
struct i2c_client *client;
struct irq_domain *irq;
{
struct pca954x *data = i2c_mux_priv(muxc);
struct i2c_client *client = data->client;
+ s8 idle_state;
+
+ idle_state = READ_ONCE(data->idle_state);
+ if (idle_state >= 0)
+ /* Set the mux back to a predetermined channel */
+ return pca954x_select_chan(muxc, idle_state);
+
+ if (idle_state == MUX_IDLE_DISCONNECT) {
+ /* Deselect active channel */
+ data->last_chan = 0;
+ return pca954x_reg_write(muxc->parent, client,
+ data->last_chan);
+ }
- if (!(data->deselect & (1 << chan)))
- return 0;
+ /* otherwise leave as-is */
- /* Deselect active channel */
- data->last_chan = 0;
- return pca954x_reg_write(muxc->parent, client, data->last_chan);
+ return 0;
+}
+
+static ssize_t idle_state_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct i2c_mux_core *muxc = i2c_get_clientdata(client);
+ struct pca954x *data = i2c_mux_priv(muxc);
+
+ return sprintf(buf, "%d\n", READ_ONCE(data->idle_state));
}
+static ssize_t idle_state_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct i2c_mux_core *muxc = i2c_get_clientdata(client);
+ struct pca954x *data = i2c_mux_priv(muxc);
+ int val;
+ int ret;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (ret < 0)
+ return ret;
+
+ if (val != MUX_IDLE_AS_IS && val != MUX_IDLE_DISCONNECT &&
+ (val < 0 || val >= data->chip->nchans))
+ return -EINVAL;
+
+ i2c_lock_bus(muxc->parent, I2C_LOCK_SEGMENT);
+
+ WRITE_ONCE(data->idle_state, val);
+ /*
+ * Set the mux into a state consistent with the new
+ * idle_state.
+ */
+ if (data->last_chan || val != MUX_IDLE_DISCONNECT)
+ ret = pca954x_deselect_mux(muxc, 0);
+
+ i2c_unlock_bus(muxc->parent, I2C_LOCK_SEGMENT);
+
+ return ret < 0 ? ret : count;
+}
+
+static DEVICE_ATTR_RW(idle_state);
+
static irqreturn_t pca954x_irq_handler(int irq, void *dev_id)
{
struct pca954x *data = dev_id;
static void pca954x_cleanup(struct i2c_mux_core *muxc)
{
struct pca954x *data = i2c_mux_priv(muxc);
+ struct i2c_client *client = data->client;
int c, irq;
+ device_remove_file(&client->dev, &dev_attr_idle_state);
+
if (data->irq) {
for (c = 0; c < data->chip->nchans; c++) {
irq = irq_find_mapping(data->irq, c);
const struct i2c_device_id *id)
{
struct i2c_adapter *adap = client->adapter;
- struct pca954x_platform_data *pdata = dev_get_platdata(&client->dev);
struct device *dev = &client->dev;
struct device_node *np = dev->of_node;
bool idle_disconnect_dt;
struct gpio_desc *gpio;
- int num, force, class;
struct i2c_mux_core *muxc;
struct pca954x *data;
+ int num;
int ret;
if (!i2c_check_functionality(adap, I2C_FUNC_SMBUS_BYTE))
}
data->last_chan = 0; /* force the first selection */
+ data->idle_state = MUX_IDLE_AS_IS;
idle_disconnect_dt = np &&
of_property_read_bool(np, "i2c-mux-idle-disconnect");
+ if (idle_disconnect_dt)
+ data->idle_state = MUX_IDLE_DISCONNECT;
ret = pca954x_irq_setup(muxc);
if (ret)
/* Now create an adapter for each channel */
for (num = 0; num < data->chip->nchans; num++) {
- bool idle_disconnect_pd = false;
-
- force = 0; /* dynamic adap number */
- class = 0; /* no class by default */
- if (pdata) {
- if (num < pdata->num_modes) {
- /* force static number */
- force = pdata->modes[num].adap_id;
- class = pdata->modes[num].class;
- } else
- /* discard unconfigured channels */
- break;
- idle_disconnect_pd = pdata->modes[num].deselect_on_exit;
- }
- data->deselect |= (idle_disconnect_pd ||
- idle_disconnect_dt) << num;
-
- ret = i2c_mux_add_adapter(muxc, force, num, class);
+ ret = i2c_mux_add_adapter(muxc, 0, num, 0);
if (ret)
goto fail_cleanup;
}
goto fail_cleanup;
}
+ /*
+ * The attr probably isn't going to be needed in most cases,
+ * so don't fail completely on error.
+ */
+ device_create_file(dev, &dev_attr_idle_state);
+
dev_info(dev, "registered %d multiplexed busses for I2C %s %s\n",
num, data->chip->muxtype == pca954x_ismux
? "mux" : "switch", client->name);
ide_hwif_t *hwif, *mate = NULL;
int i, j = 0;
+ pr_warn("legacy IDE will be removed in 2021, please switch to libata\n"
+ "Report any missing HW support to linux-ide@vger.kernel.org\n");
+
ide_host_for_each_port(i, hwif, host) {
if (hwif == NULL) {
mate = NULL;
source "drivers/infiniband/hw/qib/Kconfig"
source "drivers/infiniband/hw/cxgb3/Kconfig"
source "drivers/infiniband/hw/cxgb4/Kconfig"
+source "drivers/infiniband/hw/efa/Kconfig"
source "drivers/infiniband/hw/i40iw/Kconfig"
source "drivers/infiniband/hw/mlx4/Kconfig"
source "drivers/infiniband/hw/mlx5/Kconfig"
#include <net/ipv6_stubs.h>
#include <net/ip6_route.h>
#include <rdma/ib_addr.h>
+#include <rdma/ib_cache.h>
#include <rdma/ib_sa.h>
#include <rdma/ib.h>
#include <rdma/rdma_netlink.h>
GID_TABLE_ENTRY_PENDING_DEL = 3,
};
+struct roce_gid_ndev_storage {
+ struct rcu_head rcu_head;
+ struct net_device *ndev;
+};
+
struct ib_gid_table_entry {
struct kref kref;
struct work_struct del_work;
struct ib_gid_attr attr;
void *context;
+ /* Store the ndev pointer to release reference later on in
+ * call_rcu context because by that time gid_table_entry
+ * and attr might be already freed. So keep a copy of it.
+ * ndev_storage is freed by rcu callback.
+ */
+ struct roce_gid_ndev_storage *ndev_storage;
enum gid_table_entry_state state;
};
queue_work(ib_wq, &entry->del_work);
}
+static void put_gid_ndev(struct rcu_head *head)
+{
+ struct roce_gid_ndev_storage *storage =
+ container_of(head, struct roce_gid_ndev_storage, rcu_head);
+
+ WARN_ON(!storage->ndev);
+ /* At this point its safe to release netdev reference,
+ * as all callers working on gid_attr->ndev are done
+ * using this netdev.
+ */
+ dev_put(storage->ndev);
+ kfree(storage);
+}
+
static void free_gid_entry_locked(struct ib_gid_table_entry *entry)
{
struct ib_device *device = entry->attr.device;
/* Now this index is ready to be allocated */
write_unlock_irq(&table->rwlock);
- if (entry->attr.ndev)
- dev_put(entry->attr.ndev);
+ if (entry->ndev_storage)
+ call_rcu(&entry->ndev_storage->rcu_head, put_gid_ndev);
kfree(entry);
}
alloc_gid_entry(const struct ib_gid_attr *attr)
{
struct ib_gid_table_entry *entry;
+ struct net_device *ndev;
entry = kzalloc(sizeof(*entry), GFP_KERNEL);
if (!entry)
return NULL;
+
+ ndev = rcu_dereference_protected(attr->ndev, 1);
+ if (ndev) {
+ entry->ndev_storage = kzalloc(sizeof(*entry->ndev_storage),
+ GFP_KERNEL);
+ if (!entry->ndev_storage) {
+ kfree(entry);
+ return NULL;
+ }
+ dev_hold(ndev);
+ entry->ndev_storage->ndev = ndev;
+ }
kref_init(&entry->kref);
memcpy(&entry->attr, attr, sizeof(*attr));
- if (entry->attr.ndev)
- dev_hold(entry->attr.ndev);
INIT_WORK(&entry->del_work, free_gid_work);
entry->state = GID_TABLE_ENTRY_INVALID;
return entry;
static void del_gid(struct ib_device *ib_dev, u8 port,
struct ib_gid_table *table, int ix)
{
+ struct roce_gid_ndev_storage *ndev_storage;
struct ib_gid_table_entry *entry;
lockdep_assert_held(&table->lock);
table->data_vec[ix] = NULL;
write_unlock_irq(&table->rwlock);
+ ndev_storage = entry->ndev_storage;
+ if (ndev_storage) {
+ entry->ndev_storage = NULL;
+ rcu_assign_pointer(entry->attr.ndev, NULL);
+ call_rcu(&ndev_storage->rcu_head, put_gid_ndev);
+ }
+
if (rdma_cap_roce_gid_table(ib_dev, port))
ib_dev->ops.del_gid(&entry->attr, &entry->context);
int ib_cache_gid_add(struct ib_device *ib_dev, u8 port,
union ib_gid *gid, struct ib_gid_attr *attr)
{
- struct net_device *idev;
- unsigned long mask;
- int ret;
-
- idev = ib_device_get_netdev(ib_dev, port);
- if (idev && attr->ndev != idev) {
- union ib_gid default_gid;
-
- /* Adding default GIDs is not permitted */
- make_default_gid(idev, &default_gid);
- if (!memcmp(gid, &default_gid, sizeof(*gid))) {
- dev_put(idev);
- return -EPERM;
- }
- }
- if (idev)
- dev_put(idev);
-
- mask = GID_ATTR_FIND_MASK_GID |
- GID_ATTR_FIND_MASK_GID_TYPE |
- GID_ATTR_FIND_MASK_NETDEV;
+ unsigned long mask = GID_ATTR_FIND_MASK_GID |
+ GID_ATTR_FIND_MASK_GID_TYPE |
+ GID_ATTR_FIND_MASK_NETDEV;
- ret = __ib_cache_gid_add(ib_dev, port, gid, attr, mask, false);
- return ret;
+ return __ib_cache_gid_add(ib_dev, port, gid, attr, mask, false);
}
static int
read_lock_irqsave(&table->rwlock, flags);
valid = is_gid_entry_valid(table->data_vec[attr->index]);
- if (valid && attr->ndev && (READ_ONCE(attr->ndev->flags) & IFF_UP))
- ndev = attr->ndev;
+ if (valid) {
+ ndev = rcu_dereference(attr->ndev);
+ if (!ndev ||
+ (ndev && ((READ_ONCE(ndev->flags) & IFF_UP) == 0)))
+ ndev = ERR_PTR(-ENODEV);
+ }
read_unlock_irqrestore(&table->rwlock, flags);
return ndev;
}
+EXPORT_SYMBOL(rdma_read_gid_attr_ndev_rcu);
+
+static int get_lower_dev_vlan(struct net_device *lower_dev, void *data)
+{
+ u16 *vlan_id = data;
+
+ if (is_vlan_dev(lower_dev))
+ *vlan_id = vlan_dev_vlan_id(lower_dev);
+
+ /* We are interested only in first level vlan device, so
+ * always return 1 to stop iterating over next level devices.
+ */
+ return 1;
+}
+
+/**
+ * rdma_read_gid_l2_fields - Read the vlan ID and source MAC address
+ * of a GID entry.
+ *
+ * @attr: GID attribute pointer whose L2 fields to be read
+ * @vlan_id: Pointer to vlan id to fill up if the GID entry has
+ * vlan id. It is optional.
+ * @smac: Pointer to smac to fill up for a GID entry. It is optional.
+ *
+ * rdma_read_gid_l2_fields() returns 0 on success and returns vlan id
+ * (if gid entry has vlan) and source MAC, or returns error.
+ */
+int rdma_read_gid_l2_fields(const struct ib_gid_attr *attr,
+ u16 *vlan_id, u8 *smac)
+{
+ struct net_device *ndev;
+
+ rcu_read_lock();
+ ndev = rcu_dereference(attr->ndev);
+ if (!ndev) {
+ rcu_read_unlock();
+ return -ENODEV;
+ }
+ if (smac)
+ ether_addr_copy(smac, ndev->dev_addr);
+ if (vlan_id) {
+ *vlan_id = 0xffff;
+ if (is_vlan_dev(ndev)) {
+ *vlan_id = vlan_dev_vlan_id(ndev);
+ } else {
+ /* If the netdev is upper device and if it's lower
+ * device is vlan device, consider vlan id of the
+ * the lower vlan device for this gid entry.
+ */
+ netdev_walk_all_lower_dev_rcu(attr->ndev,
+ get_lower_dev_vlan, vlan_id);
+ }
+ }
+ rcu_read_unlock();
+ return 0;
+}
+EXPORT_SYMBOL(rdma_read_gid_l2_fields);
static int config_non_roce_gid_cache(struct ib_device *device,
u8 port, int gid_tbl_len)
event->event == IB_EVENT_PORT_ACTIVE ||
event->event == IB_EVENT_LID_CHANGE ||
event->event == IB_EVENT_PKEY_CHANGE ||
- event->event == IB_EVENT_SM_CHANGE ||
event->event == IB_EVENT_CLIENT_REREGISTER ||
event->event == IB_EVENT_GID_CHANGE) {
work = kmalloc(sizeof *work, GFP_ATOMIC);
#include <rdma/ib_cache.h>
#include <rdma/ib_cm.h>
#include "cm_msgs.h"
+#include "core_priv.h"
MODULE_AUTHOR("Sean Hefty");
MODULE_DESCRIPTION("InfiniBand CM");
struct rb_root remote_qp_table;
struct rb_root remote_id_table;
struct rb_root remote_sidr_table;
- struct idr local_id_table;
+ struct xarray local_id_table;
+ u32 local_id_next;
__be32 random_id_operand;
struct list_head timewait_list;
struct workqueue_struct *wq;
struct cm_device {
struct list_head list;
struct ib_device *ib_device;
- struct device *device;
u8 ack_delay;
int going_down;
struct cm_port *port[0];
static int cm_alloc_id(struct cm_id_private *cm_id_priv)
{
- unsigned long flags;
- int id;
-
- idr_preload(GFP_KERNEL);
- spin_lock_irqsave(&cm.lock, flags);
+ int err;
+ u32 id;
- id = idr_alloc_cyclic(&cm.local_id_table, cm_id_priv, 0, 0, GFP_NOWAIT);
-
- spin_unlock_irqrestore(&cm.lock, flags);
- idr_preload_end();
+ err = xa_alloc_cyclic_irq(&cm.local_id_table, &id, cm_id_priv,
+ xa_limit_32b, &cm.local_id_next, GFP_KERNEL);
cm_id_priv->id.local_id = (__force __be32)id ^ cm.random_id_operand;
- return id < 0 ? id : 0;
+ return err;
+}
+
+static u32 cm_local_id(__be32 local_id)
+{
+ return (__force u32) (local_id ^ cm.random_id_operand);
}
static void cm_free_id(__be32 local_id)
{
- spin_lock_irq(&cm.lock);
- idr_remove(&cm.local_id_table,
- (__force int) (local_id ^ cm.random_id_operand));
- spin_unlock_irq(&cm.lock);
+ xa_erase_irq(&cm.local_id_table, cm_local_id(local_id));
}
static struct cm_id_private * cm_get_id(__be32 local_id, __be32 remote_id)
{
struct cm_id_private *cm_id_priv;
- cm_id_priv = idr_find(&cm.local_id_table,
- (__force int) (local_id ^ cm.random_id_operand));
+ cm_id_priv = xa_load(&cm.local_id_table, cm_local_id(local_id));
if (cm_id_priv) {
if (cm_id_priv->id.remote_id == remote_id)
atomic_inc(&cm_id_priv->refcount);
grh = rdma_ah_read_grh(&cm_id_priv->av.ah_attr);
gid_attr = grh->sgid_attr;
- if (gid_attr && gid_attr->ndev) {
+ if (gid_attr &&
+ rdma_protocol_roce(work->port->cm_dev->ib_device,
+ work->port->port_num)) {
work->path[0].rec_type =
sa_conv_gid_to_pathrec_type(gid_attr->gid_type);
} else {
- /* If no GID attribute or ndev is null, it is not RoCE. */
cm_path_set_rec_type(work->port->cm_dev->ib_device,
work->port->port_num,
&work->path[0],
spin_unlock_irq(&cm.lock);
return NULL;
}
- cm_id_priv = idr_find(&cm.local_id_table, (__force int)
- (timewait_info->work.local_id ^
- cm.random_id_operand));
+ cm_id_priv = xa_load(&cm.local_id_table,
+ cm_local_id(timewait_info->work.local_id));
if (cm_id_priv) {
if (cm_id_priv->id.remote_id == remote_id)
atomic_inc(&cm_id_priv->refcount);
.default_attrs = cm_counter_default_attrs
};
-static void cm_release_port_obj(struct kobject *obj)
-{
- struct cm_port *cm_port;
-
- cm_port = container_of(obj, struct cm_port, port_obj);
- kfree(cm_port);
-}
-
-static struct kobj_type cm_port_obj_type = {
- .release = cm_release_port_obj
-};
-
static char *cm_devnode(struct device *dev, umode_t *mode)
{
if (mode)
{
int i, ret;
- ret = kobject_init_and_add(&port->port_obj, &cm_port_obj_type,
- &port->cm_dev->device->kobj,
- "%d", port->port_num);
- if (ret) {
- kfree(port);
- return ret;
- }
-
for (i = 0; i < CM_COUNTER_GROUPS; i++) {
- ret = kobject_init_and_add(&port->counter_group[i].obj,
- &cm_counter_obj_type,
- &port->port_obj,
- "%s", counter_group_names[i]);
+ ret = ib_port_register_module_stat(port->cm_dev->ib_device,
+ port->port_num,
+ &port->counter_group[i].obj,
+ &cm_counter_obj_type,
+ counter_group_names[i]);
if (ret)
goto error;
}
error:
while (i--)
- kobject_put(&port->counter_group[i].obj);
- kobject_put(&port->port_obj);
+ ib_port_unregister_module_stat(&port->counter_group[i].obj);
return ret;
}
int i;
for (i = 0; i < CM_COUNTER_GROUPS; i++)
- kobject_put(&port->counter_group[i].obj);
+ ib_port_unregister_module_stat(&port->counter_group[i].obj);
- kobject_put(&port->port_obj);
}
static void cm_add_one(struct ib_device *ib_device)
cm_dev->ib_device = ib_device;
cm_dev->ack_delay = ib_device->attrs.local_ca_ack_delay;
cm_dev->going_down = 0;
- cm_dev->device = device_create(&cm_class, &ib_device->dev,
- MKDEV(0, 0), NULL,
- "%s", dev_name(&ib_device->dev));
- if (IS_ERR(cm_dev->device)) {
- kfree(cm_dev);
- return;
- }
set_bit(IB_MGMT_METHOD_SEND, reg_req.method_mask);
for (i = 1; i <= ib_device->phys_port_cnt; i++) {
cm_remove_port_fs(port);
}
free:
- device_unregister(cm_dev->device);
kfree(cm_dev);
}
cm_remove_port_fs(port);
}
- device_unregister(cm_dev->device);
kfree(cm_dev);
}
{
int ret;
- memset(&cm, 0, sizeof cm);
INIT_LIST_HEAD(&cm.device_list);
rwlock_init(&cm.device_lock);
spin_lock_init(&cm.lock);
cm.remote_id_table = RB_ROOT;
cm.remote_qp_table = RB_ROOT;
cm.remote_sidr_table = RB_ROOT;
- idr_init(&cm.local_id_table);
+ xa_init_flags(&cm.local_id_table, XA_FLAGS_ALLOC | XA_FLAGS_LOCK_IRQ);
get_random_bytes(&cm.random_id_operand, sizeof cm.random_id_operand);
INIT_LIST_HEAD(&cm.timewait_list);
error2:
class_unregister(&cm_class);
error1:
- idr_destroy(&cm.local_id_table);
return ret;
}
}
class_unregister(&cm_class);
- idr_destroy(&cm.local_id_table);
+ WARN_ON(!xa_empty(&cm.local_id_table));
}
module_init(ib_cm_init);
module_exit(ib_cm_cleanup);
-
u32 private_data[IB_CM_REQ_PRIVATE_DATA_SIZE / sizeof(u32)];
-} __attribute__ ((packed));
+} __packed;
static inline __be32 cm_req_get_local_qpn(struct cm_req_msg *req_msg)
{
u8 private_data[IB_CM_MRA_PRIVATE_DATA_SIZE];
-} __attribute__ ((packed));
+} __packed;
static inline u8 cm_mra_get_msg_mraed(struct cm_mra_msg *mra_msg)
{
u8 private_data[IB_CM_REJ_PRIVATE_DATA_SIZE];
-} __attribute__ ((packed));
+} __packed;
static inline u8 cm_rej_get_msg_rejected(struct cm_rej_msg *rej_msg)
{
u8 private_data[IB_CM_REP_PRIVATE_DATA_SIZE];
-} __attribute__ ((packed));
+} __packed;
static inline __be32 cm_rep_get_local_qpn(struct cm_rep_msg *rep_msg)
{
u8 private_data[IB_CM_RTU_PRIVATE_DATA_SIZE];
-} __attribute__ ((packed));
+} __packed;
struct cm_dreq_msg {
struct ib_mad_hdr hdr;
u8 private_data[IB_CM_DREQ_PRIVATE_DATA_SIZE];
-} __attribute__ ((packed));
+} __packed;
static inline __be32 cm_dreq_get_remote_qpn(struct cm_dreq_msg *dreq_msg)
{
u8 private_data[IB_CM_DREP_PRIVATE_DATA_SIZE];
-} __attribute__ ((packed));
+} __packed;
struct cm_lap_msg {
struct ib_mad_hdr hdr;
u8 offset63;
u8 private_data[IB_CM_LAP_PRIVATE_DATA_SIZE];
-} __attribute__ ((packed));
+} __packed;
static inline __be32 cm_lap_get_remote_qpn(struct cm_lap_msg *lap_msg)
{
u8 info[IB_CM_APR_INFO_LENGTH];
u8 private_data[IB_CM_APR_PRIVATE_DATA_SIZE];
-} __attribute__ ((packed));
+} __packed;
struct cm_sidr_req_msg {
struct ib_mad_hdr hdr;
__be64 service_id;
u32 private_data[IB_CM_SIDR_REQ_PRIVATE_DATA_SIZE / sizeof(u32)];
-} __attribute__ ((packed));
+} __packed;
struct cm_sidr_rep_msg {
struct ib_mad_hdr hdr;
u8 info[IB_CM_SIDR_REP_INFO_LENGTH];
u8 private_data[IB_CM_SIDR_REP_PRIVATE_DATA_SIZE];
-} __attribute__ ((packed));
+} __packed;
static inline __be32 cm_sidr_rep_get_qpn(struct cm_sidr_rep_msg *sidr_rep_msg)
{
#include <linux/mutex.h>
#include <linux/random.h>
#include <linux/igmp.h>
-#include <linux/idr.h>
+#include <linux/xarray.h>
#include <linux/inetdevice.h>
#include <linux/slab.h>
#include <linux/module.h>
static unsigned int cma_pernet_id;
struct cma_pernet {
- struct idr tcp_ps;
- struct idr udp_ps;
- struct idr ipoib_ps;
- struct idr ib_ps;
+ struct xarray tcp_ps;
+ struct xarray udp_ps;
+ struct xarray ipoib_ps;
+ struct xarray ib_ps;
};
static struct cma_pernet *cma_pernet(struct net *net)
return net_generic(net, cma_pernet_id);
}
-static struct idr *cma_pernet_idr(struct net *net, enum rdma_ucm_port_space ps)
+static
+struct xarray *cma_pernet_xa(struct net *net, enum rdma_ucm_port_space ps)
{
struct cma_pernet *pernet = cma_pernet(net);
static int cma_ps_alloc(struct net *net, enum rdma_ucm_port_space ps,
struct rdma_bind_list *bind_list, int snum)
{
- struct idr *idr = cma_pernet_idr(net, ps);
+ struct xarray *xa = cma_pernet_xa(net, ps);
- return idr_alloc(idr, bind_list, snum, snum + 1, GFP_KERNEL);
+ return xa_insert(xa, snum, bind_list, GFP_KERNEL);
}
static struct rdma_bind_list *cma_ps_find(struct net *net,
enum rdma_ucm_port_space ps, int snum)
{
- struct idr *idr = cma_pernet_idr(net, ps);
+ struct xarray *xa = cma_pernet_xa(net, ps);
- return idr_find(idr, snum);
+ return xa_load(xa, snum);
}
static void cma_ps_remove(struct net *net, enum rdma_ucm_port_space ps,
int snum)
{
- struct idr *idr = cma_pernet_idr(net, ps);
+ struct xarray *xa = cma_pernet_xa(net, ps);
- idr_remove(idr, snum);
+ xa_erase(xa, snum);
}
enum {
int dev_type = dev_addr->dev_type;
struct net_device *ndev = NULL;
+ if (!rdma_dev_access_netns(device, id_priv->id.route.addr.dev_addr.net))
+ return ERR_PTR(-ENODEV);
+
if ((dev_type == ARPHRD_INFINIBAND) && !rdma_protocol_ib(device, port))
return ERR_PTR(-ENODEV);
return cma_zero_addr(addr) || cma_loopback_addr(addr);
}
-static int cma_addr_cmp(struct sockaddr *src, struct sockaddr *dst)
+static int cma_addr_cmp(const struct sockaddr *src, const struct sockaddr *dst)
{
if (src->sa_family != dst->sa_family)
return -1;
switch (src->sa_family) {
case AF_INET:
- return ((struct sockaddr_in *) src)->sin_addr.s_addr !=
- ((struct sockaddr_in *) dst)->sin_addr.s_addr;
- case AF_INET6:
- return ipv6_addr_cmp(&((struct sockaddr_in6 *) src)->sin6_addr,
- &((struct sockaddr_in6 *) dst)->sin6_addr);
+ return ((struct sockaddr_in *)src)->sin_addr.s_addr !=
+ ((struct sockaddr_in *)dst)->sin_addr.s_addr;
+ case AF_INET6: {
+ struct sockaddr_in6 *src_addr6 = (struct sockaddr_in6 *)src;
+ struct sockaddr_in6 *dst_addr6 = (struct sockaddr_in6 *)dst;
+ bool link_local;
+
+ if (ipv6_addr_cmp(&src_addr6->sin6_addr,
+ &dst_addr6->sin6_addr))
+ return 1;
+ link_local = ipv6_addr_type(&dst_addr6->sin6_addr) &
+ IPV6_ADDR_LINKLOCAL;
+ /* Link local must match their scope_ids */
+ return link_local ? (src_addr6->sin6_scope_id !=
+ dst_addr6->sin6_scope_id) :
+ 0;
+ }
+
default:
return ib_addr_cmp(&((struct sockaddr_ib *) src)->sib_addr,
&((struct sockaddr_ib *) dst)->sib_addr);
roce_get_net_dev_by_cm_event(const struct ib_cm_event *ib_event)
{
const struct ib_gid_attr *sgid_attr = NULL;
+ struct net_device *ndev;
if (ib_event->event == IB_CM_REQ_RECEIVED)
sgid_attr = ib_event->param.req_rcvd.ppath_sgid_attr;
if (!sgid_attr)
return NULL;
- dev_hold(sgid_attr->ndev);
- return sgid_attr->ndev;
+
+ rcu_read_lock();
+ ndev = rdma_read_gid_attr_ndev_rcu(sgid_attr);
+ if (IS_ERR(ndev))
+ ndev = NULL;
+ else
+ dev_hold(ndev);
+ rcu_read_unlock();
+ return ndev;
}
static struct net_device *cma_get_net_dev(const struct ib_cm_event *ib_event,
goto err;
bind_list->ps = ps;
- bind_list->port = (unsigned short)ret;
+ bind_list->port = snum;
cma_bind_port(bind_list, id_priv);
return 0;
err:
{
struct cma_pernet *pernet = cma_pernet(net);
- idr_init(&pernet->tcp_ps);
- idr_init(&pernet->udp_ps);
- idr_init(&pernet->ipoib_ps);
- idr_init(&pernet->ib_ps);
+ xa_init(&pernet->tcp_ps);
+ xa_init(&pernet->udp_ps);
+ xa_init(&pernet->ipoib_ps);
+ xa_init(&pernet->ib_ps);
return 0;
}
{
struct cma_pernet *pernet = cma_pernet(net);
- idr_destroy(&pernet->tcp_ps);
- idr_destroy(&pernet->udp_ps);
- idr_destroy(&pernet->ipoib_ps);
- idr_destroy(&pernet->ib_ps);
+ WARN_ON(!xa_empty(&pernet->tcp_ps));
+ WARN_ON(!xa_empty(&pernet->udp_ps));
+ WARN_ON(!xa_empty(&pernet->ipoib_ps));
+ WARN_ON(!xa_empty(&pernet->ib_ps));
}
static struct pernet_operations cma_pernet_operations = {
};
extern const struct attribute_group ib_dev_attr_group;
+extern bool ib_devices_shared_netns;
int ib_device_register_sysfs(struct ib_device *device);
void ib_device_unregister_sysfs(struct ib_device *device);
}
#endif
-struct ib_device *ib_device_get_by_index(u32 ifindex);
+struct ib_device *ib_device_get_by_index(const struct net *net, u32 index);
+
/* RDMA device netlink */
void nldev_init(void);
void nldev_exit(void);
qp->device = dev;
qp->pd = pd;
qp->uobject = uobj;
+ qp->real_qp = qp;
/*
* We don't track XRC QPs for now, because they don't have PD
* and more importantly they are created internaly by driver,
const struct ib_gid_attr *attr);
struct net_device *rdma_read_gid_attr_ndev_rcu(const struct ib_gid_attr *attr);
+
+void ib_free_port_attrs(struct ib_core_device *coredev);
+int ib_setup_port_attrs(struct ib_core_device *coredev);
+
+int rdma_compatdev_set(u8 enable);
+
+int ib_port_register_module_stat(struct ib_device *device, u8 port_num,
+ struct kobject *kobj, struct kobj_type *ktype,
+ const char *name);
+void ib_port_unregister_module_stat(struct kobject *kobj);
+
+int ib_device_set_netns_put(struct sk_buff *skb,
+ struct ib_device *dev, u32 ns_fd);
#endif /* _CORE_PRIV_H */
* @comp_vector: HCA completion vectors for this CQ
* @poll_ctx: context to poll the CQ from.
* @caller: module owner name.
+ * @udata: Valid user data or NULL for kernel object
*
* This is the proper interface to allocate a CQ for in-kernel users. A
* CQ allocated with this interface will automatically be polled from the
* specified context. The ULP must use wr->wr_cqe instead of wr->wr_id
* to use this CQ abstraction.
*/
-struct ib_cq *__ib_alloc_cq(struct ib_device *dev, void *private,
- int nr_cqe, int comp_vector,
- enum ib_poll_context poll_ctx, const char *caller)
+struct ib_cq *__ib_alloc_cq_user(struct ib_device *dev, void *private,
+ int nr_cqe, int comp_vector,
+ enum ib_poll_context poll_ctx,
+ const char *caller, struct ib_udata *udata)
{
struct ib_cq_init_attr cq_attr = {
.cqe = nr_cqe,
struct ib_cq *cq;
int ret = -ENOMEM;
- cq = dev->ops.create_cq(dev, &cq_attr, NULL, NULL);
+ cq = dev->ops.create_cq(dev, &cq_attr, NULL);
if (IS_ERR(cq))
return cq;
kfree(cq->wc);
rdma_restrack_del(&cq->res);
out_destroy_cq:
- cq->device->ops.destroy_cq(cq);
+ cq->device->ops.destroy_cq(cq, udata);
return ERR_PTR(ret);
}
-EXPORT_SYMBOL(__ib_alloc_cq);
+EXPORT_SYMBOL(__ib_alloc_cq_user);
/**
* ib_free_cq - free a completion queue
* @cq: completion queue to free.
+ * @udata: User data or NULL for kernel object
*/
-void ib_free_cq(struct ib_cq *cq)
+void ib_free_cq_user(struct ib_cq *cq, struct ib_udata *udata)
{
int ret;
kfree(cq->wc);
rdma_restrack_del(&cq->res);
- ret = cq->device->ops.destroy_cq(cq);
+ ret = cq->device->ops.destroy_cq(cq, udata);
WARN_ON_ONCE(ret);
}
-EXPORT_SYMBOL(ib_free_cq);
+EXPORT_SYMBOL(ib_free_cq_user);
#include <linux/slab.h>
#include <linux/init.h>
#include <linux/netdevice.h>
+#include <net/net_namespace.h>
+#include <net/netns/generic.h>
#include <linux/security.h>
#include <linux/notifier.h>
#include <linux/hashtable.h>
* be registered.
*/
#define CLIENT_DATA_REGISTERED XA_MARK_1
+
+/**
+ * struct rdma_dev_net - rdma net namespace metadata for a net
+ * @net: Pointer to owner net namespace
+ * @id: xarray id to identify the net namespace.
+ */
+struct rdma_dev_net {
+ possible_net_t net;
+ u32 id;
+};
+
+static unsigned int rdma_dev_net_id;
+
+/*
+ * A list of net namespaces is maintained in an xarray. This is necessary
+ * because we can't get the locking right using the existing net ns list. We
+ * would require a init_net callback after the list is updated.
+ */
+static DEFINE_XARRAY_FLAGS(rdma_nets, XA_FLAGS_ALLOC);
+/*
+ * rwsem to protect accessing the rdma_nets xarray entries.
+ */
+static DECLARE_RWSEM(rdma_nets_rwsem);
+
+bool ib_devices_shared_netns = true;
+module_param_named(netns_mode, ib_devices_shared_netns, bool, 0444);
+MODULE_PARM_DESC(netns_mode,
+ "Share device among net namespaces; default=1 (shared)");
+/**
+ * rdma_dev_access_netns() - Return whether a rdma device can be accessed
+ * from a specified net namespace or not.
+ * @device: Pointer to rdma device which needs to be checked
+ * @net: Pointer to net namesapce for which access to be checked
+ *
+ * rdma_dev_access_netns() - Return whether a rdma device can be accessed
+ * from a specified net namespace or not. When
+ * rdma device is in shared mode, it ignores the
+ * net namespace. When rdma device is exclusive
+ * to a net namespace, rdma device net namespace is
+ * checked against the specified one.
+ */
+bool rdma_dev_access_netns(const struct ib_device *dev, const struct net *net)
+{
+ return (ib_devices_shared_netns ||
+ net_eq(read_pnet(&dev->coredev.rdma_net), net));
+}
+EXPORT_SYMBOL(rdma_dev_access_netns);
+
/*
* xarray has this behavior where it won't iterate over NULL values stored in
* allocated arrays. So we need our own iterator to see all values stored in
static void ib_policy_change_task(struct work_struct *work);
static DECLARE_WORK(ib_policy_change_work, ib_policy_change_task);
+static void __ibdev_printk(const char *level, const struct ib_device *ibdev,
+ struct va_format *vaf)
+{
+ if (ibdev && ibdev->dev.parent)
+ dev_printk_emit(level[1] - '0',
+ ibdev->dev.parent,
+ "%s %s %s: %pV",
+ dev_driver_string(ibdev->dev.parent),
+ dev_name(ibdev->dev.parent),
+ dev_name(&ibdev->dev),
+ vaf);
+ else if (ibdev)
+ printk("%s%s: %pV",
+ level, dev_name(&ibdev->dev), vaf);
+ else
+ printk("%s(NULL ib_device): %pV", level, vaf);
+}
+
+void ibdev_printk(const char *level, const struct ib_device *ibdev,
+ const char *format, ...)
+{
+ struct va_format vaf;
+ va_list args;
+
+ va_start(args, format);
+
+ vaf.fmt = format;
+ vaf.va = &args;
+
+ __ibdev_printk(level, ibdev, &vaf);
+
+ va_end(args);
+}
+EXPORT_SYMBOL(ibdev_printk);
+
+#define define_ibdev_printk_level(func, level) \
+void func(const struct ib_device *ibdev, const char *fmt, ...) \
+{ \
+ struct va_format vaf; \
+ va_list args; \
+ \
+ va_start(args, fmt); \
+ \
+ vaf.fmt = fmt; \
+ vaf.va = &args; \
+ \
+ __ibdev_printk(level, ibdev, &vaf); \
+ \
+ va_end(args); \
+} \
+EXPORT_SYMBOL(func);
+
+define_ibdev_printk_level(ibdev_emerg, KERN_EMERG);
+define_ibdev_printk_level(ibdev_alert, KERN_ALERT);
+define_ibdev_printk_level(ibdev_crit, KERN_CRIT);
+define_ibdev_printk_level(ibdev_err, KERN_ERR);
+define_ibdev_printk_level(ibdev_warn, KERN_WARNING);
+define_ibdev_printk_level(ibdev_notice, KERN_NOTICE);
+define_ibdev_printk_level(ibdev_info, KERN_INFO);
+
static struct notifier_block ibdev_lsm_nb = {
.notifier_call = ib_security_change,
};
+static int rdma_dev_change_netns(struct ib_device *device, struct net *cur_net,
+ struct net *net);
+
/* Pointer to the RCU head at the start of the ib_port_data array */
struct ib_port_data_rcu {
struct rcu_head rcu_head;
* Caller must perform ib_device_put() to return the device reference count
* when ib_device_get_by_index() returns valid device pointer.
*/
-struct ib_device *ib_device_get_by_index(u32 index)
+struct ib_device *ib_device_get_by_index(const struct net *net, u32 index)
{
struct ib_device *device;
down_read(&devices_rwsem);
device = xa_load(&devices, index);
if (device) {
+ if (!rdma_dev_access_netns(device, net)) {
+ device = NULL;
+ goto out;
+ }
+
if (!ib_device_try_get(device))
device = NULL;
}
+out:
up_read(&devices_rwsem);
return device;
}
}
EXPORT_SYMBOL(ib_device_get_by_name);
+static int rename_compat_devs(struct ib_device *device)
+{
+ struct ib_core_device *cdev;
+ unsigned long index;
+ int ret = 0;
+
+ mutex_lock(&device->compat_devs_mutex);
+ xa_for_each (&device->compat_devs, index, cdev) {
+ ret = device_rename(&cdev->dev, dev_name(&device->dev));
+ if (ret) {
+ dev_warn(&cdev->dev,
+ "Fail to rename compatdev to new name %s\n",
+ dev_name(&device->dev));
+ break;
+ }
+ }
+ mutex_unlock(&device->compat_devs_mutex);
+ return ret;
+}
+
int ib_device_rename(struct ib_device *ibdev, const char *name)
{
int ret;
if (ret)
goto out;
strlcpy(ibdev->name, name, IB_DEVICE_NAME_MAX);
+ ret = rename_compat_devs(ibdev);
out:
up_write(&devices_rwsem);
return ret;
WARN_ON(refcount_read(&dev->refcount));
ib_cache_release_one(dev);
ib_security_release_port_pkey_list(dev);
+ xa_destroy(&dev->compat_devs);
xa_destroy(&dev->client_data);
if (dev->port_data)
kfree_rcu(container_of(dev->port_data, struct ib_port_data_rcu,
return 0;
}
+static const void *net_namespace(struct device *d)
+{
+ struct ib_core_device *coredev =
+ container_of(d, struct ib_core_device, dev);
+
+ return read_pnet(&coredev->rdma_net);
+}
+
static struct class ib_class = {
.name = "infiniband",
.dev_release = ib_device_release,
.dev_uevent = ib_device_uevent,
+ .ns_type = &net_ns_type_operations,
+ .namespace = net_namespace,
};
+static void rdma_init_coredev(struct ib_core_device *coredev,
+ struct ib_device *dev, struct net *net)
+{
+ /* This BUILD_BUG_ON is intended to catch layout change
+ * of union of ib_core_device and device.
+ * dev must be the first element as ib_core and providers
+ * driver uses it. Adding anything in ib_core_device before
+ * device will break this assumption.
+ */
+ BUILD_BUG_ON(offsetof(struct ib_device, coredev.dev) !=
+ offsetof(struct ib_device, dev));
+
+ coredev->dev.class = &ib_class;
+ coredev->dev.groups = dev->groups;
+ device_initialize(&coredev->dev);
+ coredev->owner = dev;
+ INIT_LIST_HEAD(&coredev->port_list);
+ write_pnet(&coredev->rdma_net, net);
+}
+
/**
* _ib_alloc_device - allocate an IB device struct
* @size:size of structure to allocate
return NULL;
}
- device->dev.class = &ib_class;
device->groups[0] = &ib_dev_attr_group;
- device->dev.groups = device->groups;
- device_initialize(&device->dev);
+ rdma_init_coredev(&device->coredev, device, &init_net);
INIT_LIST_HEAD(&device->event_handler_list);
spin_lock_init(&device->event_handler_lock);
*/
xa_init_flags(&device->client_data, XA_FLAGS_ALLOC);
init_rwsem(&device->client_data_rwsem);
- INIT_LIST_HEAD(&device->port_list);
+ xa_init_flags(&device->compat_devs, XA_FLAGS_ALLOC);
+ mutex_init(&device->compat_devs_mutex);
init_completion(&device->unreg_completion);
INIT_WORK(&device->unregistration_work, ib_unregister_work);
/* Expedite releasing netdev references */
free_netdevs(device);
+ WARN_ON(!xa_empty(&device->compat_devs));
WARN_ON(!xa_empty(&device->client_data));
WARN_ON(refcount_read(&device->refcount));
rdma_restrack_clean(device);
return NOTIFY_OK;
}
+static void compatdev_release(struct device *dev)
+{
+ struct ib_core_device *cdev =
+ container_of(dev, struct ib_core_device, dev);
+
+ kfree(cdev);
+}
+
+static int add_one_compat_dev(struct ib_device *device,
+ struct rdma_dev_net *rnet)
+{
+ struct ib_core_device *cdev;
+ int ret;
+
+ lockdep_assert_held(&rdma_nets_rwsem);
+ if (!ib_devices_shared_netns)
+ return 0;
+
+ /*
+ * Create and add compat device in all namespaces other than where it
+ * is currently bound to.
+ */
+ if (net_eq(read_pnet(&rnet->net),
+ read_pnet(&device->coredev.rdma_net)))
+ return 0;
+
+ /*
+ * The first of init_net() or ib_register_device() to take the
+ * compat_devs_mutex wins and gets to add the device. Others will wait
+ * for completion here.
+ */
+ mutex_lock(&device->compat_devs_mutex);
+ cdev = xa_load(&device->compat_devs, rnet->id);
+ if (cdev) {
+ ret = 0;
+ goto done;
+ }
+ ret = xa_reserve(&device->compat_devs, rnet->id, GFP_KERNEL);
+ if (ret)
+ goto done;
+
+ cdev = kzalloc(sizeof(*cdev), GFP_KERNEL);
+ if (!cdev) {
+ ret = -ENOMEM;
+ goto cdev_err;
+ }
+
+ cdev->dev.parent = device->dev.parent;
+ rdma_init_coredev(cdev, device, read_pnet(&rnet->net));
+ cdev->dev.release = compatdev_release;
+ dev_set_name(&cdev->dev, "%s", dev_name(&device->dev));
+
+ ret = device_add(&cdev->dev);
+ if (ret)
+ goto add_err;
+ ret = ib_setup_port_attrs(cdev);
+ if (ret)
+ goto port_err;
+
+ ret = xa_err(xa_store(&device->compat_devs, rnet->id,
+ cdev, GFP_KERNEL));
+ if (ret)
+ goto insert_err;
+
+ mutex_unlock(&device->compat_devs_mutex);
+ return 0;
+
+insert_err:
+ ib_free_port_attrs(cdev);
+port_err:
+ device_del(&cdev->dev);
+add_err:
+ put_device(&cdev->dev);
+cdev_err:
+ xa_release(&device->compat_devs, rnet->id);
+done:
+ mutex_unlock(&device->compat_devs_mutex);
+ return ret;
+}
+
+static void remove_one_compat_dev(struct ib_device *device, u32 id)
+{
+ struct ib_core_device *cdev;
+
+ mutex_lock(&device->compat_devs_mutex);
+ cdev = xa_erase(&device->compat_devs, id);
+ mutex_unlock(&device->compat_devs_mutex);
+ if (cdev) {
+ ib_free_port_attrs(cdev);
+ device_del(&cdev->dev);
+ put_device(&cdev->dev);
+ }
+}
+
+static void remove_compat_devs(struct ib_device *device)
+{
+ struct ib_core_device *cdev;
+ unsigned long index;
+
+ xa_for_each (&device->compat_devs, index, cdev)
+ remove_one_compat_dev(device, index);
+}
+
+static int add_compat_devs(struct ib_device *device)
+{
+ struct rdma_dev_net *rnet;
+ unsigned long index;
+ int ret = 0;
+
+ lockdep_assert_held(&devices_rwsem);
+
+ down_read(&rdma_nets_rwsem);
+ xa_for_each (&rdma_nets, index, rnet) {
+ ret = add_one_compat_dev(device, rnet);
+ if (ret)
+ break;
+ }
+ up_read(&rdma_nets_rwsem);
+ return ret;
+}
+
+static void remove_all_compat_devs(void)
+{
+ struct ib_compat_device *cdev;
+ struct ib_device *dev;
+ unsigned long index;
+
+ down_read(&devices_rwsem);
+ xa_for_each (&devices, index, dev) {
+ unsigned long c_index = 0;
+
+ /* Hold nets_rwsem so that any other thread modifying this
+ * system param can sync with this thread.
+ */
+ down_read(&rdma_nets_rwsem);
+ xa_for_each (&dev->compat_devs, c_index, cdev)
+ remove_one_compat_dev(dev, c_index);
+ up_read(&rdma_nets_rwsem);
+ }
+ up_read(&devices_rwsem);
+}
+
+static int add_all_compat_devs(void)
+{
+ struct rdma_dev_net *rnet;
+ struct ib_device *dev;
+ unsigned long index;
+ int ret = 0;
+
+ down_read(&devices_rwsem);
+ xa_for_each_marked (&devices, index, dev, DEVICE_REGISTERED) {
+ unsigned long net_index = 0;
+
+ /* Hold nets_rwsem so that any other thread modifying this
+ * system param can sync with this thread.
+ */
+ down_read(&rdma_nets_rwsem);
+ xa_for_each (&rdma_nets, net_index, rnet) {
+ ret = add_one_compat_dev(dev, rnet);
+ if (ret)
+ break;
+ }
+ up_read(&rdma_nets_rwsem);
+ }
+ up_read(&devices_rwsem);
+ if (ret)
+ remove_all_compat_devs();
+ return ret;
+}
+
+int rdma_compatdev_set(u8 enable)
+{
+ struct rdma_dev_net *rnet;
+ unsigned long index;
+ int ret = 0;
+
+ down_write(&rdma_nets_rwsem);
+ if (ib_devices_shared_netns == enable) {
+ up_write(&rdma_nets_rwsem);
+ return 0;
+ }
+
+ /* enable/disable of compat devices is not supported
+ * when more than default init_net exists.
+ */
+ xa_for_each (&rdma_nets, index, rnet) {
+ ret++;
+ break;
+ }
+ if (!ret)
+ ib_devices_shared_netns = enable;
+ up_write(&rdma_nets_rwsem);
+ if (ret)
+ return -EBUSY;
+
+ if (enable)
+ ret = add_all_compat_devs();
+ else
+ remove_all_compat_devs();
+ return ret;
+}
+
+static void rdma_dev_exit_net(struct net *net)
+{
+ struct rdma_dev_net *rnet = net_generic(net, rdma_dev_net_id);
+ struct ib_device *dev;
+ unsigned long index;
+ int ret;
+
+ down_write(&rdma_nets_rwsem);
+ /*
+ * Prevent the ID from being re-used and hide the id from xa_for_each.
+ */
+ ret = xa_err(xa_store(&rdma_nets, rnet->id, NULL, GFP_KERNEL));
+ WARN_ON(ret);
+ up_write(&rdma_nets_rwsem);
+
+ down_read(&devices_rwsem);
+ xa_for_each (&devices, index, dev) {
+ get_device(&dev->dev);
+ /*
+ * Release the devices_rwsem so that pontentially blocking
+ * device_del, doesn't hold the devices_rwsem for too long.
+ */
+ up_read(&devices_rwsem);
+
+ remove_one_compat_dev(dev, rnet->id);
+
+ /*
+ * If the real device is in the NS then move it back to init.
+ */
+ rdma_dev_change_netns(dev, net, &init_net);
+
+ put_device(&dev->dev);
+ down_read(&devices_rwsem);
+ }
+ up_read(&devices_rwsem);
+
+ xa_erase(&rdma_nets, rnet->id);
+}
+
+static __net_init int rdma_dev_init_net(struct net *net)
+{
+ struct rdma_dev_net *rnet = net_generic(net, rdma_dev_net_id);
+ unsigned long index;
+ struct ib_device *dev;
+ int ret;
+
+ /* No need to create any compat devices in default init_net. */
+ if (net_eq(net, &init_net))
+ return 0;
+
+ write_pnet(&rnet->net, net);
+
+ ret = xa_alloc(&rdma_nets, &rnet->id, rnet, xa_limit_32b, GFP_KERNEL);
+ if (ret)
+ return ret;
+
+ down_read(&devices_rwsem);
+ xa_for_each_marked (&devices, index, dev, DEVICE_REGISTERED) {
+ /* Hold nets_rwsem so that netlink command cannot change
+ * system configuration for device sharing mode.
+ */
+ down_read(&rdma_nets_rwsem);
+ ret = add_one_compat_dev(dev, rnet);
+ up_read(&rdma_nets_rwsem);
+ if (ret)
+ break;
+ }
+ up_read(&devices_rwsem);
+
+ if (ret)
+ rdma_dev_exit_net(net);
+
+ return ret;
+}
+
/*
* Assign the unique string device name and the unique device index. This is
* undone by ib_dealloc_device.
WARN_ON_ONCE(!parent);
device->dma_device = parent;
}
+ /* Setup default max segment size for all IB devices */
+ dma_set_max_seg_size(device->dma_device, SZ_2G);
+
}
/*
ib_device_put(device);
wait_for_completion(&device->unreg_completion);
- /* Expedite removing unregistered pointers from the hash table */
- free_netdevs(device);
+ /*
+ * compat devices must be removed after device refcount drops to zero.
+ * Otherwise init_net() may add more compatdevs after removing compat
+ * devices and before device is disabled.
+ */
+ remove_compat_devs(device);
}
/*
break;
}
up_read(&clients_rwsem);
-
+ if (!ret)
+ ret = add_compat_devs(device);
out:
up_read(&devices_rwsem);
return ret;
ib_device_register_rdmacg(device);
+ /*
+ * Ensure that ADD uevent is not fired because it
+ * is too early amd device is not initialized yet.
+ */
+ dev_set_uevent_suppress(&device->dev, true);
ret = device_add(&device->dev);
if (ret)
goto cg_cleanup;
}
ret = enable_device_and_get(device);
+ dev_set_uevent_suppress(&device->dev, false);
+ /* Mark for userspace that device is ready */
+ kobject_uevent(&device->dev.kobj, KOBJ_ADD);
if (ret) {
void (*dealloc_fn)(struct ib_device *);
dev_cleanup:
device_del(&device->dev);
cg_cleanup:
+ dev_set_uevent_suppress(&device->dev, false);
ib_device_unregister_rdmacg(device);
ib_cache_cleanup_one(device);
return ret;
goto out;
disable_device(ib_dev);
+
+ /* Expedite removing unregistered pointers from the hash table */
+ free_netdevs(ib_dev);
+
ib_device_unregister_sysfs(ib_dev);
device_del(&ib_dev->dev);
ib_device_unregister_rdmacg(ib_dev);
}
EXPORT_SYMBOL(ib_unregister_device_queued);
+/*
+ * The caller must pass in a device that has the kref held and the refcount
+ * released. If the device is in cur_net and still registered then it is moved
+ * into net.
+ */
+static int rdma_dev_change_netns(struct ib_device *device, struct net *cur_net,
+ struct net *net)
+{
+ int ret2 = -EINVAL;
+ int ret;
+
+ mutex_lock(&device->unregistration_lock);
+
+ /*
+ * If a device not under ib_device_get() or if the unregistration_lock
+ * is not held, the namespace can be changed, or it can be unregistered.
+ * Check again under the lock.
+ */
+ if (refcount_read(&device->refcount) == 0 ||
+ !net_eq(cur_net, read_pnet(&device->coredev.rdma_net))) {
+ ret = -ENODEV;
+ goto out;
+ }
+
+ kobject_uevent(&device->dev.kobj, KOBJ_REMOVE);
+ disable_device(device);
+
+ /*
+ * At this point no one can be using the device, so it is safe to
+ * change the namespace.
+ */
+ write_pnet(&device->coredev.rdma_net, net);
+
+ down_read(&devices_rwsem);
+ /*
+ * Currently rdma devices are system wide unique. So the device name
+ * is guaranteed free in the new namespace. Publish the new namespace
+ * at the sysfs level.
+ */
+ ret = device_rename(&device->dev, dev_name(&device->dev));
+ up_read(&devices_rwsem);
+ if (ret) {
+ dev_warn(&device->dev,
+ "%s: Couldn't rename device after namespace change\n",
+ __func__);
+ /* Try and put things back and re-enable the device */
+ write_pnet(&device->coredev.rdma_net, cur_net);
+ }
+
+ ret2 = enable_device_and_get(device);
+ if (ret2) {
+ /*
+ * This shouldn't really happen, but if it does, let the user
+ * retry at later point. So don't disable the device.
+ */
+ dev_warn(&device->dev,
+ "%s: Couldn't re-enable device after namespace change\n",
+ __func__);
+ }
+ kobject_uevent(&device->dev.kobj, KOBJ_ADD);
+
+ ib_device_put(device);
+out:
+ mutex_unlock(&device->unregistration_lock);
+ if (ret)
+ return ret;
+ return ret2;
+}
+
+int ib_device_set_netns_put(struct sk_buff *skb,
+ struct ib_device *dev, u32 ns_fd)
+{
+ struct net *net;
+ int ret;
+
+ net = get_net_ns_by_fd(ns_fd);
+ if (IS_ERR(net)) {
+ ret = PTR_ERR(net);
+ goto net_err;
+ }
+
+ if (!netlink_ns_capable(skb, net->user_ns, CAP_NET_ADMIN)) {
+ ret = -EPERM;
+ goto ns_err;
+ }
+
+ /*
+ * Currently supported only for those providers which support
+ * disassociation and don't do port specific sysfs init. Once a
+ * port_cleanup infrastructure is implemented, this limitation will be
+ * removed.
+ */
+ if (!dev->ops.disassociate_ucontext || dev->ops.init_port ||
+ ib_devices_shared_netns) {
+ ret = -EOPNOTSUPP;
+ goto ns_err;
+ }
+
+ get_device(&dev->dev);
+ ib_device_put(dev);
+ ret = rdma_dev_change_netns(dev, current->nsproxy->net_ns, net);
+ put_device(&dev->dev);
+
+ put_net(net);
+ return ret;
+
+ns_err:
+ put_net(net);
+net_err:
+ ib_device_put(dev);
+ return ret;
+}
+
+static struct pernet_operations rdma_dev_net_ops = {
+ .init = rdma_dev_init_net,
+ .exit = rdma_dev_exit_net,
+ .id = &rdma_dev_net_id,
+ .size = sizeof(struct rdma_dev_net),
+};
+
static int assign_client_id(struct ib_client *client)
{
int ret;
down_read(&devices_rwsem);
xa_for_each_marked (&devices, index, dev, DEVICE_REGISTERED) {
+ if (!rdma_dev_access_netns(dev, sock_net(skb->sk)))
+ continue;
+
ret = nldev_cb(dev, skb, cb, idx);
if (ret)
break;
SET_DEVICE_OP(dev_ops, get_vf_config);
SET_DEVICE_OP(dev_ops, get_vf_stats);
SET_DEVICE_OP(dev_ops, init_port);
+ SET_DEVICE_OP(dev_ops, iw_accept);
+ SET_DEVICE_OP(dev_ops, iw_add_ref);
+ SET_DEVICE_OP(dev_ops, iw_connect);
+ SET_DEVICE_OP(dev_ops, iw_create_listen);
+ SET_DEVICE_OP(dev_ops, iw_destroy_listen);
+ SET_DEVICE_OP(dev_ops, iw_get_qp);
+ SET_DEVICE_OP(dev_ops, iw_reject);
+ SET_DEVICE_OP(dev_ops, iw_rem_ref);
SET_DEVICE_OP(dev_ops, map_mr_sg);
SET_DEVICE_OP(dev_ops, map_phys_fmr);
SET_DEVICE_OP(dev_ops, mmap);
SET_DEVICE_OP(dev_ops, set_vf_link_state);
SET_DEVICE_OP(dev_ops, unmap_fmr);
+ SET_OBJ_SIZE(dev_ops, ib_ah);
SET_OBJ_SIZE(dev_ops, ib_pd);
+ SET_OBJ_SIZE(dev_ops, ib_srq);
SET_OBJ_SIZE(dev_ops, ib_ucontext);
}
EXPORT_SYMBOL(ib_set_device_ops);
goto err_sa;
}
+ ret = register_pernet_device(&rdma_dev_net_ops);
+ if (ret) {
+ pr_warn("Couldn't init compat dev. ret %d\n", ret);
+ goto err_compat;
+ }
+
nldev_init();
rdma_nl_register(RDMA_NL_LS, ibnl_ls_cb_table);
roce_gid_mgmt_init();
return 0;
+err_compat:
+ unregister_lsm_notifier(&ibdev_lsm_nb);
err_sa:
ib_sa_cleanup();
err_mad:
roce_gid_mgmt_cleanup();
nldev_exit();
rdma_nl_unregister(RDMA_NL_LS);
+ unregister_pernet_device(&rdma_dev_net_ops);
unregister_lsm_notifier(&ibdev_lsm_nb);
ib_sa_cleanup();
ib_mad_cleanup();
MODULE_ALIAS_RDMA_NETLINK(RDMA_NL_LS, 4);
-subsys_initcall(ib_core_init);
+/* ib core relies on netdev stack to first register net_ns_type_operations
+ * ns kobject type before ib_core initialization.
+ */
+fs_initcall(ib_core_init);
module_exit(ib_core_cleanup);
cm_id_priv->state = IW_CM_STATE_DESTROYING;
spin_unlock_irqrestore(&cm_id_priv->lock, flags);
/* destroy the listening endpoint */
- cm_id->device->iwcm->destroy_listen(cm_id);
+ cm_id->device->ops.iw_destroy_listen(cm_id);
spin_lock_irqsave(&cm_id_priv->lock, flags);
break;
case IW_CM_STATE_ESTABLISHED:
*/
cm_id_priv->state = IW_CM_STATE_DESTROYING;
spin_unlock_irqrestore(&cm_id_priv->lock, flags);
- cm_id->device->iwcm->reject(cm_id, NULL, 0);
+ cm_id->device->ops.iw_reject(cm_id, NULL, 0);
spin_lock_irqsave(&cm_id_priv->lock, flags);
break;
case IW_CM_STATE_CONN_SENT:
break;
}
if (cm_id_priv->qp) {
- cm_id_priv->id.device->iwcm->rem_ref(cm_id_priv->qp);
+ cm_id_priv->id.device->ops.iw_rem_ref(cm_id_priv->qp);
cm_id_priv->qp = NULL;
}
spin_unlock_irqrestore(&cm_id_priv->lock, flags);
static int iw_cm_map(struct iw_cm_id *cm_id, bool active)
{
const char *devname = dev_name(&cm_id->device->dev);
- const char *ifname = cm_id->device->iwcm->ifname;
+ const char *ifname = cm_id->device->iw_ifname;
struct iwpm_dev_data pm_reg_msg = {};
struct iwpm_sa_data pm_msg;
int status;
cm_id->mapped = true;
pm_msg.loc_addr = cm_id->local_addr;
pm_msg.rem_addr = cm_id->remote_addr;
- pm_msg.flags = (cm_id->device->iwcm->driver_flags & IW_F_NO_PORT_MAP) ?
+ pm_msg.flags = (cm_id->device->iw_driver_flags & IW_F_NO_PORT_MAP) ?
IWPM_FLAGS_NO_PORT_MAP : 0;
if (active)
status = iwpm_add_and_query_mapping(&pm_msg,
spin_unlock_irqrestore(&cm_id_priv->lock, flags);
ret = iw_cm_map(cm_id, false);
if (!ret)
- ret = cm_id->device->iwcm->create_listen(cm_id, backlog);
+ ret = cm_id->device->ops.iw_create_listen(cm_id,
+ backlog);
if (ret)
cm_id_priv->state = IW_CM_STATE_IDLE;
spin_lock_irqsave(&cm_id_priv->lock, flags);
cm_id_priv->state = IW_CM_STATE_IDLE;
spin_unlock_irqrestore(&cm_id_priv->lock, flags);
- ret = cm_id->device->iwcm->reject(cm_id, private_data,
+ ret = cm_id->device->ops.iw_reject(cm_id, private_data,
private_data_len);
clear_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
return -EINVAL;
}
/* Get the ib_qp given the QPN */
- qp = cm_id->device->iwcm->get_qp(cm_id->device, iw_param->qpn);
+ qp = cm_id->device->ops.iw_get_qp(cm_id->device, iw_param->qpn);
if (!qp) {
spin_unlock_irqrestore(&cm_id_priv->lock, flags);
clear_bit(IWCM_F_CONNECT_WAIT, &cm_id_priv->flags);
wake_up_all(&cm_id_priv->connect_wait);
return -EINVAL;
}
- cm_id->device->iwcm->add_ref(qp);
+ cm_id->device->ops.iw_add_ref(qp);
cm_id_priv->qp = qp;
spin_unlock_irqrestore(&cm_id_priv->lock, flags);
- ret = cm_id->device->iwcm->accept(cm_id, iw_param);
+ ret = cm_id->device->ops.iw_accept(cm_id, iw_param);
if (ret) {
/* An error on accept precludes provider events */
BUG_ON(cm_id_priv->state != IW_CM_STATE_CONN_RECV);
cm_id_priv->state = IW_CM_STATE_IDLE;
spin_lock_irqsave(&cm_id_priv->lock, flags);
if (cm_id_priv->qp) {
- cm_id->device->iwcm->rem_ref(qp);
+ cm_id->device->ops.iw_rem_ref(qp);
cm_id_priv->qp = NULL;
}
spin_unlock_irqrestore(&cm_id_priv->lock, flags);
}
/* Get the ib_qp given the QPN */
- qp = cm_id->device->iwcm->get_qp(cm_id->device, iw_param->qpn);
+ qp = cm_id->device->ops.iw_get_qp(cm_id->device, iw_param->qpn);
if (!qp) {
ret = -EINVAL;
goto err;
}
- cm_id->device->iwcm->add_ref(qp);
+ cm_id->device->ops.iw_add_ref(qp);
cm_id_priv->qp = qp;
cm_id_priv->state = IW_CM_STATE_CONN_SENT;
spin_unlock_irqrestore(&cm_id_priv->lock, flags);
ret = iw_cm_map(cm_id, true);
if (!ret)
- ret = cm_id->device->iwcm->connect(cm_id, iw_param);
+ ret = cm_id->device->ops.iw_connect(cm_id, iw_param);
if (!ret)
return 0; /* success */
spin_lock_irqsave(&cm_id_priv->lock, flags);
if (cm_id_priv->qp) {
- cm_id->device->iwcm->rem_ref(qp);
+ cm_id->device->ops.iw_rem_ref(qp);
cm_id_priv->qp = NULL;
}
cm_id_priv->state = IW_CM_STATE_IDLE;
cm_id_priv->state = IW_CM_STATE_ESTABLISHED;
} else {
/* REJECTED or RESET */
- cm_id_priv->id.device->iwcm->rem_ref(cm_id_priv->qp);
+ cm_id_priv->id.device->ops.iw_rem_ref(cm_id_priv->qp);
cm_id_priv->qp = NULL;
cm_id_priv->state = IW_CM_STATE_IDLE;
}
spin_lock_irqsave(&cm_id_priv->lock, flags);
if (cm_id_priv->qp) {
- cm_id_priv->id.device->iwcm->rem_ref(cm_id_priv->qp);
+ cm_id_priv->id.device->ops.iw_rem_ref(cm_id_priv->qp);
cm_id_priv->qp = NULL;
}
switch (cm_id_priv->state) {
* Copyright (c) 2005 Intel Corporation. All rights reserved.
* Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved.
* Copyright (c) 2009 HNR Consulting. All rights reserved.
- * Copyright (c) 2014 Intel Corporation. All rights reserved.
+ * Copyright (c) 2014,2018 Intel Corporation. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/dma-mapping.h>
-#include <linux/idr.h>
#include <linux/slab.h>
#include <linux/module.h>
#include <linux/security.h>
+#include <linux/xarray.h>
#include <rdma/ib_cache.h>
#include "mad_priv.h"
#include "opa_smi.h"
#include "agent.h"
+#define CREATE_TRACE_POINTS
+#include <trace/events/ib_mad.h>
+
+#ifdef CONFIG_TRACEPOINTS
+static void create_mad_addr_info(struct ib_mad_send_wr_private *mad_send_wr,
+ struct ib_mad_qp_info *qp_info,
+ struct trace_event_raw_ib_mad_send_template *entry)
+{
+ u16 pkey;
+ struct ib_device *dev = qp_info->port_priv->device;
+ u8 pnum = qp_info->port_priv->port_num;
+ struct ib_ud_wr *wr = &mad_send_wr->send_wr;
+ struct rdma_ah_attr attr = {};
+
+ rdma_query_ah(wr->ah, &attr);
+
+ /* These are common */
+ entry->sl = attr.sl;
+ ib_query_pkey(dev, pnum, wr->pkey_index, &pkey);
+ entry->pkey = pkey;
+ entry->rqpn = wr->remote_qpn;
+ entry->rqkey = wr->remote_qkey;
+ entry->dlid = rdma_ah_get_dlid(&attr);
+}
+#endif
+
static int mad_sendq_size = IB_MAD_QP_SEND_SIZE;
static int mad_recvq_size = IB_MAD_QP_RECV_SIZE;
module_param_named(recv_queue_size, mad_recvq_size, int, 0444);
MODULE_PARM_DESC(recv_queue_size, "Size of receive queue in number of work requests");
-/*
- * The mlx4 driver uses the top byte to distinguish which virtual function
- * generated the MAD, so we must avoid using it.
- */
-#define AGENT_ID_LIMIT (1 << 24)
-static DEFINE_IDR(ib_mad_clients);
+/* Client ID 0 is used for snoop-only clients */
+static DEFINE_XARRAY_ALLOC1(ib_mad_clients);
+static u32 ib_mad_client_next;
static struct list_head ib_mad_port_list;
/* Port list lock */
goto error4;
}
- idr_preload(GFP_KERNEL);
- idr_lock(&ib_mad_clients);
- ret2 = idr_alloc_cyclic(&ib_mad_clients, mad_agent_priv, 0,
- AGENT_ID_LIMIT, GFP_ATOMIC);
- idr_unlock(&ib_mad_clients);
- idr_preload_end();
-
+ /*
+ * The mlx4 driver uses the top byte to distinguish which virtual
+ * function generated the MAD, so we must avoid using it.
+ */
+ ret2 = xa_alloc_cyclic(&ib_mad_clients, &mad_agent_priv->agent.hi_tid,
+ mad_agent_priv, XA_LIMIT(0, (1 << 24) - 1),
+ &ib_mad_client_next, GFP_KERNEL);
if (ret2 < 0) {
ret = ERR_PTR(ret2);
goto error5;
}
- mad_agent_priv->agent.hi_tid = ret2;
/*
* Make sure MAD registration (if supplied)
}
spin_unlock_irq(&port_priv->reg_lock);
+ trace_ib_mad_create_agent(mad_agent_priv);
return &mad_agent_priv->agent;
error6:
spin_unlock_irq(&port_priv->reg_lock);
- idr_lock(&ib_mad_clients);
- idr_remove(&ib_mad_clients, mad_agent_priv->agent.hi_tid);
- idr_unlock(&ib_mad_clients);
+ xa_erase(&ib_mad_clients, mad_agent_priv->agent.hi_tid);
error5:
ib_mad_agent_security_cleanup(&mad_agent_priv->agent);
error4:
struct ib_mad_port_private *port_priv;
/* Note that we could still be handling received MADs */
+ trace_ib_mad_unregister_agent(mad_agent_priv);
/*
* Canceling all sends results in dropping received response
spin_lock_irq(&port_priv->reg_lock);
remove_mad_reg_req(mad_agent_priv);
spin_unlock_irq(&port_priv->reg_lock);
- idr_lock(&ib_mad_clients);
- idr_remove(&ib_mad_clients, mad_agent_priv->agent.hi_tid);
- idr_unlock(&ib_mad_clients);
+ xa_erase(&ib_mad_clients, mad_agent_priv->agent.hi_tid);
flush_workqueue(port_priv->wq);
ib_cancel_rmpp_recvs(mad_agent_priv);
if (opa && smp->class_version == OPA_SM_CLASS_VERSION) {
u32 opa_drslid;
+ trace_ib_mad_handle_out_opa_smi(opa_smp);
+
if ((opa_get_smp_direction(opa_smp)
? opa_smp->route.dr.dr_dlid : opa_smp->route.dr.dr_slid) ==
OPA_LID_PERMISSIVE &&
opa_smi_check_local_returning_smp(opa_smp, device) == IB_SMI_DISCARD)
goto out;
} else {
+ trace_ib_mad_handle_out_ib_smi(smp);
+
if ((ib_get_smp_direction(smp) ? smp->dr_dlid : smp->dr_slid) ==
IB_LID_PERMISSIVE &&
smi_handle_dr_smp_send(smp, rdma_cap_ib_switch(device), port_num) ==
spin_lock_irqsave(&qp_info->send_queue.lock, flags);
if (qp_info->send_queue.count < qp_info->send_queue.max_active) {
+ trace_ib_mad_ib_send_mad(mad_send_wr, qp_info);
ret = ib_post_send(mad_agent->qp, &mad_send_wr->send_wr.wr,
NULL);
list = &qp_info->send_queue.list;
*/
hi_tid = be64_to_cpu(mad_hdr->tid) >> 32;
rcu_read_lock();
- mad_agent = idr_find(&ib_mad_clients, hi_tid);
+ mad_agent = xa_load(&ib_mad_clients, hi_tid);
if (mad_agent && !atomic_inc_not_zero(&mad_agent->refcount))
mad_agent = NULL;
rcu_read_unlock();
enum smi_forward_action retsmi;
struct ib_smp *smp = (struct ib_smp *)recv->mad;
+ trace_ib_mad_handle_ib_smi(smp);
+
if (smi_handle_dr_smp_recv(smp,
rdma_cap_ib_switch(port_priv->device),
port_num,
enum smi_forward_action retsmi;
struct opa_smp *smp = (struct opa_smp *)recv->mad;
+ trace_ib_mad_handle_opa_smi(smp);
+
if (opa_smi_handle_dr_smp_recv(smp,
rdma_cap_ib_switch(port_priv->device),
port_num,
if (!validate_mad((const struct ib_mad_hdr *)recv->mad, qp_info, opa))
goto out;
+ trace_ib_mad_recv_done_handler(qp_info, wc,
+ (struct ib_mad_hdr *)recv->mad);
+
mad_size = recv->mad_size;
response = alloc_mad_private(mad_size, GFP_KERNEL);
if (!response)
mad_agent = find_mad_agent(port_priv, (const struct ib_mad_hdr *)recv->mad);
if (mad_agent) {
+ trace_ib_mad_recv_done_agent(mad_agent);
ib_mad_complete_recv(mad_agent, &recv->header.recv_wc);
/*
* recv is freed up in error cases in ib_mad_complete_recv
send_queue = mad_list->mad_queue;
qp_info = send_queue->qp_info;
+ trace_ib_mad_send_done_agent(mad_send_wr->mad_agent_priv);
+ trace_ib_mad_send_done_handler(mad_send_wr, wc);
+
retry:
ib_dma_unmap_single(mad_send_wr->send_buf.mad_agent->device,
mad_send_wr->header_mapping,
ib_mad_complete_send_wr(mad_send_wr, &mad_send_wc);
if (queued_send_wr) {
+ trace_ib_mad_send_done_resend(queued_send_wr, qp_info);
ret = ib_post_send(qp_info->qp, &queued_send_wr->send_wr.wr,
NULL);
if (ret) {
if (mad_send_wr->retry) {
/* Repost send */
mad_send_wr->retry = 0;
+ trace_ib_mad_error_handler(mad_send_wr, qp_info);
ret = ib_post_send(qp_info->qp, &mad_send_wr->send_wr.wr,
NULL);
if (!ret)
INIT_LIST_HEAD(&ib_mad_port_list);
- /* Client ID 0 is used for snoop-only clients */
- idr_alloc(&ib_mad_clients, NULL, 0, 0, GFP_KERNEL);
-
if (ib_register_client(&mad_client)) {
pr_err("Couldn't register ib_mad client\n");
return -EINVAL;
struct ib_mad_recv_wc recv_wc;
struct ib_wc wc;
u64 mapping;
-} __attribute__ ((packed));
+} __packed;
struct ib_mad_private {
struct ib_mad_private_header header;
size_t mad_size;
struct ib_grh grh;
u8 mad[0];
-} __attribute__ ((packed));
+} __packed;
struct ib_rmpp_segment {
struct list_head list;
switch (event->event) {
case IB_EVENT_PORT_ERR:
case IB_EVENT_LID_CHANGE:
- case IB_EVENT_SM_CHANGE:
case IB_EVENT_CLIENT_REREGISTER:
mcast_groups_event(&dev->port[index], MCAST_GROUP_ERROR);
break;
[RDMA_NLDEV_ATTR_RES_CTXN] = { .type = NLA_U32 },
[RDMA_NLDEV_ATTR_LINK_TYPE] = { .type = NLA_NUL_STRING,
.len = RDMA_NLDEV_ATTR_ENTRY_STRLEN },
+ [RDMA_NLDEV_SYS_ATTR_NETNS_MODE] = { .type = NLA_U8 },
+ [RDMA_NLDEV_ATTR_DEV_PROTOCOL] = { .type = NLA_NUL_STRING,
+ .len = RDMA_NLDEV_ATTR_ENTRY_STRLEN },
+ [RDMA_NLDEV_NET_NS_FD] = { .type = NLA_U32 },
};
static int put_driver_name_print_type(struct sk_buff *msg, const char *name,
static int fill_dev_info(struct sk_buff *msg, struct ib_device *device)
{
char fw[IB_FW_VERSION_NAME_MAX];
+ int ret = 0;
+ u8 port;
if (fill_nldev_handle(msg, device))
return -EMSGSIZE;
return -EMSGSIZE;
if (nla_put_u8(msg, RDMA_NLDEV_ATTR_DEV_NODE_TYPE, device->node_type))
return -EMSGSIZE;
- return 0;
+
+ /*
+ * Link type is determined on first port and mlx4 device
+ * which can potentially have two different link type for the same
+ * IB device is considered as better to be avoided in the future,
+ */
+ port = rdma_start_port(device);
+ if (rdma_cap_opa_mad(device, port))
+ ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL, "opa");
+ else if (rdma_protocol_ib(device, port))
+ ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL, "ib");
+ else if (rdma_protocol_iwarp(device, port))
+ ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL, "iw");
+ else if (rdma_protocol_roce(device, port))
+ ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL, "roce");
+ else if (rdma_protocol_usnic(device, port))
+ ret = nla_put_string(msg, RDMA_NLDEV_ATTR_DEV_PROTOCOL,
+ "usnic");
+ return ret;
}
static int fill_port_info(struct sk_buff *msg,
index = nla_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
- device = ib_device_get_by_index(index);
+ device = ib_device_get_by_index(sock_net(skb->sk), index);
if (!device)
return -EINVAL;
return -EINVAL;
index = nla_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
- device = ib_device_get_by_index(index);
+ device = ib_device_get_by_index(sock_net(skb->sk), index);
if (!device)
return -EINVAL;
nla_strlcpy(name, tb[RDMA_NLDEV_ATTR_DEV_NAME],
IB_DEVICE_NAME_MAX);
err = ib_device_rename(device, name);
+ goto done;
}
+ if (tb[RDMA_NLDEV_NET_NS_FD]) {
+ u32 ns_fd;
+
+ ns_fd = nla_get_u32(tb[RDMA_NLDEV_NET_NS_FD]);
+ err = ib_device_set_netns_put(skb, device, ns_fd);
+ goto put_done;
+ }
+
+done:
ib_device_put(device);
+put_done:
return err;
}
{
/*
* There is no need to take lock, because
- * we are relying on ib_core's lists_rwsem
+ * we are relying on ib_core's locking.
*/
return ib_enum_all_devs(_nldev_get_dumpit, skb, cb);
}
return -EINVAL;
index = nla_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
- device = ib_device_get_by_index(index);
+ device = ib_device_get_by_index(sock_net(skb->sk), index);
if (!device)
return -EINVAL;
return -EINVAL;
ifindex = nla_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
- device = ib_device_get_by_index(ifindex);
+ device = ib_device_get_by_index(sock_net(skb->sk), ifindex);
if (!device)
return -EINVAL;
return -EINVAL;
index = nla_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
- device = ib_device_get_by_index(index);
+ device = ib_device_get_by_index(sock_net(skb->sk), index);
if (!device)
return -EINVAL;
nlmsg_cancel(skb, nlh);
goto out;
}
-
nlmsg_end(skb, nlh);
idx++;
return -EINVAL;
index = nla_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
- device = ib_device_get_by_index(index);
+ device = ib_device_get_by_index(sock_net(skb->sk), index);
if (!device)
return -EINVAL;
return -EINVAL;
index = nla_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
- device = ib_device_get_by_index(index);
+ device = ib_device_get_by_index(sock_net(skb->sk), index);
if (!device)
return -EINVAL;
return -EINVAL;
index = nla_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
- device = ib_device_get_by_index(index);
+ device = ib_device_get_by_index(sock_net(skb->sk), index);
if (!device)
return -EINVAL;
return 0;
}
+static int nldev_get_sys_get_dumpit(struct sk_buff *skb,
+ struct netlink_callback *cb)
+{
+ struct nlattr *tb[RDMA_NLDEV_ATTR_MAX];
+ struct nlmsghdr *nlh;
+ int err;
+
+ err = nlmsg_parse(cb->nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1,
+ nldev_policy, NULL);
+ if (err)
+ return err;
+
+ nlh = nlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
+ RDMA_NL_GET_TYPE(RDMA_NL_NLDEV,
+ RDMA_NLDEV_CMD_SYS_GET),
+ 0, 0);
+
+ err = nla_put_u8(skb, RDMA_NLDEV_SYS_ATTR_NETNS_MODE,
+ (u8)ib_devices_shared_netns);
+ if (err) {
+ nlmsg_cancel(skb, nlh);
+ return err;
+ }
+
+ nlmsg_end(skb, nlh);
+ return skb->len;
+}
+
+static int nldev_set_sys_set_doit(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
+{
+ struct nlattr *tb[RDMA_NLDEV_ATTR_MAX];
+ u8 enable;
+ int err;
+
+ err = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1,
+ nldev_policy, extack);
+ if (err || !tb[RDMA_NLDEV_SYS_ATTR_NETNS_MODE])
+ return -EINVAL;
+
+ enable = nla_get_u8(tb[RDMA_NLDEV_SYS_ATTR_NETNS_MODE]);
+ /* Only 0 and 1 are supported */
+ if (enable > 1)
+ return -EINVAL;
+
+ err = rdma_compatdev_set(enable);
+ return err;
+}
+
static const struct rdma_nl_cbs nldev_cb_table[RDMA_NLDEV_NUM_OPS] = {
[RDMA_NLDEV_CMD_GET] = {
.doit = nldev_get_doit,
.doit = nldev_res_get_pd_doit,
.dump = nldev_res_get_pd_dumpit,
},
+ [RDMA_NLDEV_CMD_SYS_GET] = {
+ .dump = nldev_get_sys_get_dumpit,
+ },
+ [RDMA_NLDEV_CMD_SYS_SET] = {
+ .doit = nldev_set_sys_set_doit,
+ .flags = RDMA_NL_ADMIN_PERM,
+ },
};
void __init nldev_init(void)
* and consumes the kref on the uobj.
*/
static int uverbs_destroy_uobject(struct ib_uobject *uobj,
- enum rdma_remove_reason reason)
+ enum rdma_remove_reason reason,
+ struct uverbs_attr_bundle *attrs)
{
- struct ib_uverbs_file *ufile = uobj->ufile;
+ struct ib_uverbs_file *ufile = attrs->ufile;
unsigned long flags;
int ret;
assert_uverbs_usecnt(uobj, UVERBS_LOOKUP_WRITE);
if (uobj->object) {
- ret = uobj->uapi_object->type_class->destroy_hw(uobj, reason);
+ ret = uobj->uapi_object->type_class->destroy_hw(uobj, reason,
+ attrs);
if (ret) {
if (ib_is_destroy_retryable(ret, reason, uobj))
return ret;
* version requires the caller to have already obtained an
* LOOKUP_DESTROY uobject kref.
*/
-int uobj_destroy(struct ib_uobject *uobj)
+int uobj_destroy(struct ib_uobject *uobj, struct uverbs_attr_bundle *attrs)
{
- struct ib_uverbs_file *ufile = uobj->ufile;
+ struct ib_uverbs_file *ufile = attrs->ufile;
int ret;
down_read(&ufile->hw_destroy_rwsem);
if (ret)
goto out_unlock;
- ret = uverbs_destroy_uobject(uobj, RDMA_REMOVE_DESTROY);
+ ret = uverbs_destroy_uobject(uobj, RDMA_REMOVE_DESTROY, attrs);
if (ret) {
atomic_set(&uobj->usecnt, 0);
goto out_unlock;
* uverbs_put_destroy.
*/
struct ib_uobject *__uobj_get_destroy(const struct uverbs_api_object *obj,
- u32 id,
- const struct uverbs_attr_bundle *attrs)
+ u32 id, struct uverbs_attr_bundle *attrs)
{
struct ib_uobject *uobj;
int ret;
uobj = rdma_lookup_get_uobject(obj, attrs->ufile, id,
- UVERBS_LOOKUP_DESTROY);
+ UVERBS_LOOKUP_DESTROY, attrs);
if (IS_ERR(uobj))
return uobj;
- ret = uobj_destroy(uobj);
+ ret = uobj_destroy(uobj, attrs);
if (ret) {
rdma_lookup_put_uobject(uobj, UVERBS_LOOKUP_DESTROY);
return ERR_PTR(ret);
* (negative errno on failure). For use by callers that do not need the uobj.
*/
int __uobj_perform_destroy(const struct uverbs_api_object *obj, u32 id,
- const struct uverbs_attr_bundle *attrs)
+ struct uverbs_attr_bundle *attrs)
{
struct ib_uobject *uobj;
static int idr_add_uobj(struct ib_uobject *uobj)
{
- int ret;
-
- idr_preload(GFP_KERNEL);
- spin_lock(&uobj->ufile->idr_lock);
-
- /*
- * We start with allocating an idr pointing to NULL. This represents an
- * object which isn't initialized yet. We'll replace it later on with
- * the real object once we commit.
- */
- ret = idr_alloc(&uobj->ufile->idr, NULL, 0,
- min_t(unsigned long, U32_MAX - 1, INT_MAX), GFP_NOWAIT);
- if (ret >= 0)
- uobj->id = ret;
-
- spin_unlock(&uobj->ufile->idr_lock);
- idr_preload_end();
-
- return ret < 0 ? ret : 0;
+ /*
+ * We start with allocating an idr pointing to NULL. This represents an
+ * object which isn't initialized yet. We'll replace it later on with
+ * the real object once we commit.
+ */
+ return xa_alloc(&uobj->ufile->idr, &uobj->id, NULL, xa_limit_32b,
+ GFP_KERNEL);
}
/* Returns the ib_uobject or an error. The caller should check for IS_ERR. */
enum rdma_lookup_mode mode)
{
struct ib_uobject *uobj;
- unsigned long idrno = id;
if (id < 0 || id > ULONG_MAX)
return ERR_PTR(-EINVAL);
rcu_read_lock();
- /* object won't be released as we're protected in rcu */
- uobj = idr_find(&ufile->idr, idrno);
- if (!uobj) {
- uobj = ERR_PTR(-ENOENT);
- goto free;
- }
-
/*
* The idr_find is guaranteed to return a pointer to something that
* isn't freed yet, or NULL, as the free after idr_remove goes through
* kfree_rcu(). However the object may still have been released and
* kfree() could be called at any time.
*/
- if (!kref_get_unless_zero(&uobj->ref))
+ uobj = xa_load(&ufile->idr, id);
+ if (!uobj || !kref_get_unless_zero(&uobj->ref))
uobj = ERR_PTR(-ENOENT);
-
-free:
rcu_read_unlock();
return uobj;
}
struct ib_uobject *rdma_lookup_get_uobject(const struct uverbs_api_object *obj,
struct ib_uverbs_file *ufile, s64 id,
- enum rdma_lookup_mode mode)
+ enum rdma_lookup_mode mode,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_uobject *uobj;
int ret;
- if (IS_ERR(obj) && PTR_ERR(obj) == -ENOMSG) {
+ if (obj == ERR_PTR(-ENOMSG)) {
/* must be UVERBS_IDR_ANY_OBJECT, see uapi_get_object() */
uobj = lookup_get_idr_uobject(NULL, ufile, id, mode);
if (IS_ERR(uobj))
ret = uverbs_try_lock_object(uobj, mode);
if (ret)
goto free;
+ if (attrs)
+ attrs->context = uobj->context;
return uobj;
free:
uverbs_uobject_put(uobj);
return ERR_PTR(ret);
}
-struct ib_uobject *_uobj_get_read(enum uverbs_default_objects type,
- u32 object_id,
- struct uverbs_attr_bundle *attrs)
-{
- struct ib_uobject *uobj;
-
- uobj = rdma_lookup_get_uobject(uobj_get_type(attrs, type), attrs->ufile,
- object_id, UVERBS_LOOKUP_READ);
- if (IS_ERR(uobj))
- return uobj;
-
- attrs->context = uobj->context;
-
- return uobj;
-}
-
-struct ib_uobject *_uobj_get_write(enum uverbs_default_objects type,
- u32 object_id,
- struct uverbs_attr_bundle *attrs)
-{
- struct ib_uobject *uobj;
-
- uobj = rdma_lookup_get_uobject(uobj_get_type(attrs, type), attrs->ufile,
- object_id, UVERBS_LOOKUP_WRITE);
-
- if (IS_ERR(uobj))
- return uobj;
-
- attrs->context = uobj->context;
-
- return uobj;
-}
static struct ib_uobject *
alloc_begin_idr_uobject(const struct uverbs_api_object *obj,
ret = ib_rdmacg_try_charge(&uobj->cg_obj, uobj->context->device,
RDMACG_RESOURCE_HCA_OBJECT);
if (ret)
- goto idr_remove;
+ goto remove;
return uobj;
-idr_remove:
- spin_lock(&ufile->idr_lock);
- idr_remove(&ufile->idr, uobj->id);
- spin_unlock(&ufile->idr_lock);
+remove:
+ xa_erase(&ufile->idr, uobj->id);
uobj_put:
uverbs_uobject_put(uobj);
return ERR_PTR(ret);
}
struct ib_uobject *rdma_alloc_begin_uobject(const struct uverbs_api_object *obj,
- struct ib_uverbs_file *ufile)
+ struct ib_uverbs_file *ufile,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_uobject *ret;
up_read(&ufile->hw_destroy_rwsem);
return ret;
}
+ if (attrs)
+ attrs->context = ret->context;
return ret;
}
ib_rdmacg_uncharge(&uobj->cg_obj, uobj->context->device,
RDMACG_RESOURCE_HCA_OBJECT);
- spin_lock(&uobj->ufile->idr_lock);
- idr_remove(&uobj->ufile->idr, uobj->id);
- spin_unlock(&uobj->ufile->idr_lock);
+ xa_erase(&uobj->ufile->idr, uobj->id);
}
static int __must_check destroy_hw_idr_uobject(struct ib_uobject *uobj,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
const struct uverbs_obj_idr_type *idr_type =
container_of(uobj->uapi_object->type_attrs,
struct uverbs_obj_idr_type, type);
- int ret = idr_type->destroy_object(uobj, why);
+ int ret = idr_type->destroy_object(uobj, why, attrs);
/*
* We can only fail gracefully if the user requested to destroy the
static void remove_handle_idr_uobject(struct ib_uobject *uobj)
{
- spin_lock(&uobj->ufile->idr_lock);
- idr_remove(&uobj->ufile->idr, uobj->id);
- spin_unlock(&uobj->ufile->idr_lock);
+ xa_erase(&uobj->ufile->idr, uobj->id);
/* Matches the kref in alloc_commit_idr_uobject */
uverbs_uobject_put(uobj);
}
}
static int __must_check destroy_hw_fd_uobject(struct ib_uobject *uobj,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
const struct uverbs_obj_fd_type *fd_type = container_of(
uobj->uapi_object->type_attrs, struct uverbs_obj_fd_type, type);
static int alloc_commit_idr_uobject(struct ib_uobject *uobj)
{
struct ib_uverbs_file *ufile = uobj->ufile;
+ void *old;
- spin_lock(&ufile->idr_lock);
/*
* We already allocated this IDR with a NULL object, so
* this shouldn't fail.
*
- * NOTE: Once we set the IDR we loose ownership of our kref on uobj.
+ * NOTE: Storing the uobj transfers our kref on uobj to the XArray.
* It will be put by remove_commit_idr_uobject()
*/
- WARN_ON(idr_replace(&ufile->idr, uobj, uobj->id));
- spin_unlock(&ufile->idr_lock);
+ old = xa_store(&ufile->idr, uobj->id, uobj, GFP_KERNEL);
+ WARN_ON(old != NULL);
return 0;
}
* caller can no longer assume uobj is valid. If this function fails it
* destroys the uboject, including the attached HW object.
*/
-int __must_check rdma_alloc_commit_uobject(struct ib_uobject *uobj)
+int __must_check rdma_alloc_commit_uobject(struct ib_uobject *uobj,
+ struct uverbs_attr_bundle *attrs)
{
- struct ib_uverbs_file *ufile = uobj->ufile;
+ struct ib_uverbs_file *ufile = attrs->ufile;
int ret;
/* alloc_commit consumes the uobj kref */
ret = uobj->uapi_object->type_class->alloc_commit(uobj);
if (ret) {
- uverbs_destroy_uobject(uobj, RDMA_REMOVE_ABORT);
+ uverbs_destroy_uobject(uobj, RDMA_REMOVE_ABORT, attrs);
up_read(&ufile->hw_destroy_rwsem);
return ret;
}
* This consumes the kref for uobj. It is up to the caller to unwind the HW
* object and anything else connected to uobj before calling this.
*/
-void rdma_alloc_abort_uobject(struct ib_uobject *uobj)
+void rdma_alloc_abort_uobject(struct ib_uobject *uobj,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_uverbs_file *ufile = uobj->ufile;
uobj->object = NULL;
- uverbs_destroy_uobject(uobj, RDMA_REMOVE_ABORT);
+ uverbs_destroy_uobject(uobj, RDMA_REMOVE_ABORT, attrs);
/* Matches the down_read in rdma_alloc_begin_uobject */
up_read(&ufile->hw_destroy_rwsem);
void setup_ufile_idr_uobject(struct ib_uverbs_file *ufile)
{
- spin_lock_init(&ufile->idr_lock);
- idr_init(&ufile->idr);
+ xa_init_flags(&ufile->idr, XA_FLAGS_ALLOC);
}
void release_ufile_idr_uobject(struct ib_uverbs_file *ufile)
{
struct ib_uobject *entry;
- int id;
+ unsigned long id;
/*
* At this point uverbs_cleanup_ufile() is guaranteed to have run, and
- * there are no HW objects left, however the IDR is still populated
+ * there are no HW objects left, however the xarray is still populated
* with anything that has not been cleaned up by userspace. Since the
* kref on ufile is 0, nothing is allowed to call lookup_get.
*
* This is an optimized equivalent to remove_handle_idr_uobject
*/
- idr_for_each_entry(&ufile->idr, entry, id) {
+ xa_for_each(&ufile->idr, id, entry) {
WARN_ON(entry->object);
uverbs_uobject_put(entry);
}
- idr_destroy(&ufile->idr);
+ xa_destroy(&ufile->idr);
}
const struct uverbs_obj_type_class uverbs_idr_class = {
{
struct ib_uobject *uobj = f->private_data;
struct ib_uverbs_file *ufile = uobj->ufile;
+ struct uverbs_attr_bundle attrs = {
+ .context = uobj->context,
+ .ufile = ufile,
+ };
if (down_read_trylock(&ufile->hw_destroy_rwsem)) {
/*
* write lock here, or we have a kernel bug.
*/
WARN_ON(uverbs_try_lock_object(uobj, UVERBS_LOOKUP_WRITE));
- uverbs_destroy_uobject(uobj, RDMA_REMOVE_CLOSE);
+ uverbs_destroy_uobject(uobj, RDMA_REMOVE_CLOSE, &attrs);
up_read(&ufile->hw_destroy_rwsem);
}
{
struct ib_uobject *obj, *next_obj;
int ret = -EINVAL;
+ struct uverbs_attr_bundle attrs = { .ufile = ufile };
/*
* This shouldn't run while executing other commands on this
* other threads (which might still use the FDs) chance to run.
*/
list_for_each_entry_safe(obj, next_obj, &ufile->uobjects, list) {
+ attrs.context = obj->context;
/*
* if we hit this WARN_ON, that means we are
* racing with a lookup_get.
*/
WARN_ON(uverbs_try_lock_object(obj, UVERBS_LOOKUP_WRITE));
- if (!uverbs_destroy_uobject(obj, reason))
+ if (!uverbs_destroy_uobject(obj, reason, &attrs))
ret = 0;
else
atomic_set(&obj->usecnt, 0);
EXPORT_SYMBOL(uverbs_fd_class);
struct ib_uobject *
-uverbs_get_uobject_from_file(u16 object_id,
- struct ib_uverbs_file *ufile,
- enum uverbs_obj_access access, s64 id)
+uverbs_get_uobject_from_file(u16 object_id, enum uverbs_obj_access access,
+ s64 id, struct uverbs_attr_bundle *attrs)
{
const struct uverbs_api_object *obj =
- uapi_get_object(ufile->device->uapi, object_id);
+ uapi_get_object(attrs->ufile->device->uapi, object_id);
switch (access) {
case UVERBS_ACCESS_READ:
- return rdma_lookup_get_uobject(obj, ufile, id,
- UVERBS_LOOKUP_READ);
+ return rdma_lookup_get_uobject(obj, attrs->ufile, id,
+ UVERBS_LOOKUP_READ, attrs);
case UVERBS_ACCESS_DESTROY:
/* Actual destruction is done inside uverbs_handle_method */
- return rdma_lookup_get_uobject(obj, ufile, id,
- UVERBS_LOOKUP_DESTROY);
+ return rdma_lookup_get_uobject(obj, attrs->ufile, id,
+ UVERBS_LOOKUP_DESTROY, attrs);
case UVERBS_ACCESS_WRITE:
- return rdma_lookup_get_uobject(obj, ufile, id,
- UVERBS_LOOKUP_WRITE);
+ return rdma_lookup_get_uobject(obj, attrs->ufile, id,
+ UVERBS_LOOKUP_WRITE, attrs);
case UVERBS_ACCESS_NEW:
- return rdma_alloc_begin_uobject(obj, ufile);
+ return rdma_alloc_begin_uobject(obj, attrs->ufile, attrs);
default:
WARN_ON(true);
return ERR_PTR(-EOPNOTSUPP);
}
int uverbs_finalize_object(struct ib_uobject *uobj,
- enum uverbs_obj_access access,
- bool commit)
+ enum uverbs_obj_access access, bool commit,
+ struct uverbs_attr_bundle *attrs)
{
int ret = 0;
break;
case UVERBS_ACCESS_NEW:
if (commit)
- ret = rdma_alloc_commit_uobject(uobj);
+ ret = rdma_alloc_commit_uobject(uobj, attrs);
else
- rdma_alloc_abort_uobject(uobj);
+ rdma_alloc_abort_uobject(uobj, attrs);
break;
default:
WARN_ON(true);
void uverbs_destroy_ufile_hw(struct ib_uverbs_file *ufile,
enum rdma_remove_reason reason);
-int uobj_destroy(struct ib_uobject *uobj);
+int uobj_destroy(struct ib_uobject *uobj, struct uverbs_attr_bundle *attrs);
/*
* uverbs_uobject_get is called in order to increase the reference count on
* uverbs_finalize_objects are called.
*/
struct ib_uobject *
-uverbs_get_uobject_from_file(u16 object_id,
- struct ib_uverbs_file *ufile,
- enum uverbs_obj_access access, s64 id);
+uverbs_get_uobject_from_file(u16 object_id, enum uverbs_obj_access access,
+ s64 id, struct uverbs_attr_bundle *attrs);
/*
* Note that certain finalize stages could return a status:
* object.
*/
int uverbs_finalize_object(struct ib_uobject *uobj,
- enum uverbs_obj_access access,
- bool commit);
+ enum uverbs_obj_access access, bool commit,
+ struct uverbs_attr_bundle *attrs);
int uverbs_output_written(const struct uverbs_attr_bundle *bundle, size_t idx);
#include <linux/slab.h>
#include <linux/dma-mapping.h>
#include <linux/kref.h>
-#include <linux/idr.h>
+#include <linux/xarray.h>
#include <linux/workqueue.h>
#include <uapi/linux/if_ether.h>
#include <rdma/ib_pack.h>
.remove = ib_sa_remove_one
};
-static DEFINE_SPINLOCK(idr_lock);
-static DEFINE_IDR(query_idr);
+static DEFINE_XARRAY_FLAGS(queries, XA_FLAGS_ALLOC | XA_FLAGS_LOCK_IRQ);
static DEFINE_SPINLOCK(tid_lock);
static u32 tid;
struct ib_mad_agent *agent;
struct ib_mad_send_buf *mad_buf;
- spin_lock_irqsave(&idr_lock, flags);
- if (idr_find(&query_idr, id) != query) {
- spin_unlock_irqrestore(&idr_lock, flags);
+ xa_lock_irqsave(&queries, flags);
+ if (xa_load(&queries, id) != query) {
+ xa_unlock_irqrestore(&queries, flags);
return;
}
agent = query->port->agent;
mad_buf = query->mad_buf;
- spin_unlock_irqrestore(&idr_lock, flags);
+ xa_unlock_irqrestore(&queries, flags);
/*
* If the query is still on the netlink request list, schedule
static int send_mad(struct ib_sa_query *query, unsigned long timeout_ms,
gfp_t gfp_mask)
{
- bool preload = gfpflags_allow_blocking(gfp_mask);
unsigned long flags;
int ret, id;
- if (preload)
- idr_preload(gfp_mask);
- spin_lock_irqsave(&idr_lock, flags);
-
- id = idr_alloc(&query_idr, query, 0, 0, GFP_NOWAIT);
-
- spin_unlock_irqrestore(&idr_lock, flags);
- if (preload)
- idr_preload_end();
- if (id < 0)
- return id;
+ xa_lock_irqsave(&queries, flags);
+ ret = __xa_alloc(&queries, &id, query, xa_limit_32b, gfp_mask);
+ xa_unlock_irqrestore(&queries, flags);
+ if (ret < 0)
+ return ret;
query->mad_buf->timeout_ms = timeout_ms;
query->mad_buf->context[0] = query;
ret = ib_post_send_mad(query->mad_buf, NULL);
if (ret) {
- spin_lock_irqsave(&idr_lock, flags);
- idr_remove(&query_idr, id);
- spin_unlock_irqrestore(&idr_lock, flags);
+ xa_lock_irqsave(&queries, flags);
+ __xa_erase(&queries, id);
+ xa_unlock_irqrestore(&queries, flags);
}
/*
break;
}
- spin_lock_irqsave(&idr_lock, flags);
- idr_remove(&query_idr, query->id);
- spin_unlock_irqrestore(&idr_lock, flags);
+ xa_lock_irqsave(&queries, flags);
+ __xa_erase(&queries, query->id);
+ xa_unlock_irqrestore(&queries, flags);
free_mad(query);
if (query->client)
destroy_workqueue(ib_nl_wq);
mcast_cleanup();
ib_unregister_client(&sa_client);
- idr_destroy(&query_idr);
+ WARN_ON(!xa_empty(&queries));
}
static size_t print_ndev(const struct ib_gid_attr *gid_attr, char *buf)
{
- if (!gid_attr->ndev)
- return -EINVAL;
-
- return sprintf(buf, "%s\n", gid_attr->ndev->name);
+ struct net_device *ndev;
+ size_t ret = -EINVAL;
+
+ rcu_read_lock();
+ ndev = rcu_dereference(gid_attr->ndev);
+ if (ndev)
+ ret = sprintf(buf, "%s\n", ndev->name);
+ rcu_read_unlock();
+ return ret;
}
static size_t print_gid_type(const struct ib_gid_attr *gid_attr, char *buf)
return;
}
-static int add_port(struct ib_device *device, int port_num)
+static int add_port(struct ib_core_device *coredev, int port_num)
{
+ struct ib_device *device = rdma_device_to_ibdev(&coredev->dev);
+ bool is_full_dev = &device->coredev == coredev;
struct ib_port *p;
struct ib_port_attr attr;
int i;
p->port_num = port_num;
ret = kobject_init_and_add(&p->kobj, &port_type,
- device->ports_kobj,
+ coredev->ports_kobj,
"%d", port_num);
if (ret) {
kfree(p);
goto err_put;
}
- if (device->ops.process_mad) {
+ if (device->ops.process_mad && is_full_dev) {
p->pma_table = get_counter_table(device, port_num);
ret = sysfs_create_group(&p->kobj, p->pma_table);
if (ret)
if (ret)
goto err_free_pkey;
- if (device->ops.init_port) {
+ if (device->ops.init_port && is_full_dev) {
ret = device->ops.init_port(device, port_num, &p->kobj);
if (ret)
goto err_remove_pkey;
* port, so holder should be device. Therefore skip per port conunter
* initialization.
*/
- if (device->ops.alloc_hw_stats && port_num)
+ if (device->ops.alloc_hw_stats && port_num && is_full_dev)
setup_hw_stats(device, p, port_num);
- list_add_tail(&p->kobj.entry, &device->port_list);
+ list_add_tail(&p->kobj.entry, &coredev->port_list);
kobject_uevent(&p->kobj, KOBJ_ADD);
return 0;
case RDMA_NODE_RNIC: return sprintf(buf, "%d: RNIC\n", dev->node_type);
case RDMA_NODE_USNIC: return sprintf(buf, "%d: usNIC\n", dev->node_type);
case RDMA_NODE_USNIC_UDP: return sprintf(buf, "%d: usNIC UDP\n", dev->node_type);
+ case RDMA_NODE_UNSPECIFIED: return sprintf(buf, "%d: unspecified\n", dev->node_type);
case RDMA_NODE_IB_SWITCH: return sprintf(buf, "%d: switch\n", dev->node_type);
case RDMA_NODE_IB_ROUTER: return sprintf(buf, "%d: router\n", dev->node_type);
default: return sprintf(buf, "%d: <unknown>\n", dev->node_type);
.attrs = ib_dev_attrs,
};
-static void ib_free_port_attrs(struct ib_device *device)
+void ib_free_port_attrs(struct ib_core_device *coredev)
{
struct kobject *p, *t;
- list_for_each_entry_safe(p, t, &device->port_list, entry) {
+ list_for_each_entry_safe(p, t, &coredev->port_list, entry) {
struct ib_port *port = container_of(p, struct ib_port, kobj);
list_del(&p->entry);
kobject_put(p);
}
- kobject_put(device->ports_kobj);
+ kobject_put(coredev->ports_kobj);
}
-static int ib_setup_port_attrs(struct ib_device *device)
+int ib_setup_port_attrs(struct ib_core_device *coredev)
{
+ struct ib_device *device = rdma_device_to_ibdev(&coredev->dev);
unsigned int port;
int ret;
- device->ports_kobj = kobject_create_and_add("ports", &device->dev.kobj);
- if (!device->ports_kobj)
+ coredev->ports_kobj = kobject_create_and_add("ports",
+ &coredev->dev.kobj);
+ if (!coredev->ports_kobj)
return -ENOMEM;
rdma_for_each_port (device, port) {
- ret = add_port(device, port);
+ ret = add_port(coredev, port);
if (ret)
goto err_put;
}
return 0;
err_put:
- ib_free_port_attrs(device);
+ ib_free_port_attrs(coredev);
return ret;
}
{
int ret;
- ret = ib_setup_port_attrs(device);
+ ret = ib_setup_port_attrs(&device->coredev);
if (ret)
return ret;
free_hsag(&device->dev.kobj, device->hw_stats_ag);
kfree(device->hw_stats);
- ib_free_port_attrs(device);
+ ib_free_port_attrs(&device->coredev);
+}
+
+/**
+ * ib_port_register_module_stat - add module counters under relevant port
+ * of IB device.
+ *
+ * @device: IB device to add counters
+ * @port_num: valid port number
+ * @kobj: pointer to the kobject to initialize
+ * @ktype: pointer to the ktype for this kobject.
+ * @name: the name of the kobject
+ */
+int ib_port_register_module_stat(struct ib_device *device, u8 port_num,
+ struct kobject *kobj, struct kobj_type *ktype,
+ const char *name)
+{
+ struct kobject *p, *t;
+ int ret;
+
+ list_for_each_entry_safe(p, t, &device->coredev.port_list, entry) {
+ struct ib_port *port = container_of(p, struct ib_port, kobj);
+
+ if (port->port_num != port_num)
+ continue;
+
+ ret = kobject_init_and_add(kobj, ktype, &port->kobj, "%s",
+ name);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(ib_port_register_module_stat);
+
+/**
+ * ib_port_unregister_module_stat - release module counters
+ * @kobj: pointer to the kobject to release
+ */
+void ib_port_unregister_module_stat(struct kobject *kobj)
+{
+ kobject_put(kobj);
}
+EXPORT_SYMBOL(ib_port_unregister_module_stat);
#include <linux/file.h>
#include <linux/mount.h>
#include <linux/cdev.h>
-#include <linux/idr.h>
+#include <linux/xarray.h>
#include <linux/mutex.h>
#include <linux/slab.h>
.remove = ib_ucm_remove_one
};
-static DEFINE_MUTEX(ctx_id_mutex);
-static DEFINE_IDR(ctx_id_table);
+static DEFINE_XARRAY_ALLOC(ctx_id_table);
static DECLARE_BITMAP(dev_map, IB_UCM_MAX_DEVICES);
static struct ib_ucm_context *ib_ucm_ctx_get(struct ib_ucm_file *file, int id)
{
struct ib_ucm_context *ctx;
- mutex_lock(&ctx_id_mutex);
- ctx = idr_find(&ctx_id_table, id);
+ xa_lock(&ctx_id_table);
+ ctx = xa_load(&ctx_id_table, id);
if (!ctx)
ctx = ERR_PTR(-ENOENT);
else if (ctx->file != file)
ctx = ERR_PTR(-EINVAL);
else
atomic_inc(&ctx->ref);
- mutex_unlock(&ctx_id_mutex);
+ xa_unlock(&ctx_id_table);
return ctx;
}
ctx->file = file;
INIT_LIST_HEAD(&ctx->events);
- mutex_lock(&ctx_id_mutex);
- ctx->id = idr_alloc(&ctx_id_table, ctx, 0, 0, GFP_KERNEL);
- mutex_unlock(&ctx_id_mutex);
- if (ctx->id < 0)
+ if (xa_alloc(&ctx_id_table, &ctx->id, ctx, xa_limit_32b, GFP_KERNEL))
goto error;
list_add_tail(&ctx->file_list, &file->ctxs);
err2:
ib_destroy_cm_id(ctx->cm_id);
err1:
- mutex_lock(&ctx_id_mutex);
- idr_remove(&ctx_id_table, ctx->id);
- mutex_unlock(&ctx_id_mutex);
+ xa_erase(&ctx_id_table, ctx->id);
kfree(ctx);
return result;
}
if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
return -EFAULT;
- mutex_lock(&ctx_id_mutex);
- ctx = idr_find(&ctx_id_table, cmd.id);
+ xa_lock(&ctx_id_table);
+ ctx = xa_load(&ctx_id_table, cmd.id);
if (!ctx)
ctx = ERR_PTR(-ENOENT);
else if (ctx->file != file)
ctx = ERR_PTR(-EINVAL);
else
- idr_remove(&ctx_id_table, ctx->id);
- mutex_unlock(&ctx_id_mutex);
+ __xa_erase(&ctx_id_table, ctx->id);
+ xa_unlock(&ctx_id_table);
if (IS_ERR(ctx))
return PTR_ERR(ctx);
struct ib_ucm_context, file_list);
mutex_unlock(&file->file_mutex);
- mutex_lock(&ctx_id_mutex);
- idr_remove(&ctx_id_table, ctx->id);
- mutex_unlock(&ctx_id_mutex);
-
+ xa_erase(&ctx_id_table, ctx->id);
ib_destroy_cm_id(ctx->cm_id);
ib_ucm_cleanup_events(ctx);
kfree(ctx);
class_remove_file(&cm_class, &class_attr_abi_version.attr);
unregister_chrdev_region(IB_UCM_BASE_DEV, IB_UCM_NUM_FIXED_MINOR);
unregister_chrdev_region(dynamic_ucm_dev, IB_UCM_NUM_DYNAMIC_MINOR);
- idr_destroy(&ctx_id_table);
+ WARN_ON(!xa_empty(&ctx_id_table));
}
module_init(ib_ucm_init);
#include <linux/sched/signal.h>
#include <linux/sched/mm.h>
#include <linux/export.h>
-#include <linux/hugetlb.h>
#include <linux/slab.h>
+#include <linux/pagemap.h>
#include <rdma/ib_umem_odp.h>
#include "uverbs.h"
-
static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dirty)
{
- struct scatterlist *sg;
+ struct sg_page_iter sg_iter;
struct page *page;
- int i;
if (umem->nmap > 0)
- ib_dma_unmap_sg(dev, umem->sg_head.sgl,
- umem->npages,
+ ib_dma_unmap_sg(dev, umem->sg_head.sgl, umem->sg_nents,
DMA_BIDIRECTIONAL);
- for_each_sg(umem->sg_head.sgl, sg, umem->npages, i) {
-
- page = sg_page(sg);
+ for_each_sg_page(umem->sg_head.sgl, &sg_iter, umem->sg_nents, 0) {
+ page = sg_page_iter_page(&sg_iter);
if (!PageDirty(page) && umem->writable && dirty)
set_page_dirty_lock(page);
put_page(page);
sg_free_table(&umem->sg_head);
}
+/* ib_umem_add_sg_table - Add N contiguous pages to scatter table
+ *
+ * sg: current scatterlist entry
+ * page_list: array of npage struct page pointers
+ * npages: number of pages in page_list
+ * max_seg_sz: maximum segment size in bytes
+ * nents: [out] number of entries in the scatterlist
+ *
+ * Return new end of scatterlist
+ */
+static struct scatterlist *ib_umem_add_sg_table(struct scatterlist *sg,
+ struct page **page_list,
+ unsigned long npages,
+ unsigned int max_seg_sz,
+ int *nents)
+{
+ unsigned long first_pfn;
+ unsigned long i = 0;
+ bool update_cur_sg = false;
+ bool first = !sg_page(sg);
+
+ /* Check if new page_list is contiguous with end of previous page_list.
+ * sg->length here is a multiple of PAGE_SIZE and sg->offset is 0.
+ */
+ if (!first && (page_to_pfn(sg_page(sg)) + (sg->length >> PAGE_SHIFT) ==
+ page_to_pfn(page_list[0])))
+ update_cur_sg = true;
+
+ while (i != npages) {
+ unsigned long len;
+ struct page *first_page = page_list[i];
+
+ first_pfn = page_to_pfn(first_page);
+
+ /* Compute the number of contiguous pages we have starting
+ * at i
+ */
+ for (len = 0; i != npages &&
+ first_pfn + len == page_to_pfn(page_list[i]) &&
+ len < (max_seg_sz >> PAGE_SHIFT);
+ len++)
+ i++;
+
+ /* Squash N contiguous pages from page_list into current sge */
+ if (update_cur_sg) {
+ if ((max_seg_sz - sg->length) >= (len << PAGE_SHIFT)) {
+ sg_set_page(sg, sg_page(sg),
+ sg->length + (len << PAGE_SHIFT),
+ 0);
+ update_cur_sg = false;
+ continue;
+ }
+ update_cur_sg = false;
+ }
+
+ /* Squash N contiguous pages into next sge or first sge */
+ if (!first)
+ sg = sg_next(sg);
+
+ (*nents)++;
+ sg_set_page(sg, first_page, len << PAGE_SHIFT, 0);
+ first = false;
+ }
+
+ return sg;
+}
+
+/**
+ * ib_umem_find_best_pgsz - Find best HW page size to use for this MR
+ *
+ * @umem: umem struct
+ * @pgsz_bitmap: bitmap of HW supported page sizes
+ * @virt: IOVA
+ *
+ * This helper is intended for HW that support multiple page
+ * sizes but can do only a single page size in an MR.
+ *
+ * Returns 0 if the umem requires page sizes not supported by
+ * the driver to be mapped. Drivers always supporting PAGE_SIZE
+ * or smaller will never see a 0 result.
+ */
+unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,
+ unsigned long pgsz_bitmap,
+ unsigned long virt)
+{
+ struct scatterlist *sg;
+ unsigned int best_pg_bit;
+ unsigned long va, pgoff;
+ dma_addr_t mask;
+ int i;
+
+ /* At minimum, drivers must support PAGE_SIZE or smaller */
+ if (WARN_ON(!(pgsz_bitmap & GENMASK(PAGE_SHIFT, 0))))
+ return 0;
+
+ va = virt;
+ /* max page size not to exceed MR length */
+ mask = roundup_pow_of_two(umem->length);
+ /* offset into first SGL */
+ pgoff = umem->address & ~PAGE_MASK;
+
+ for_each_sg(umem->sg_head.sgl, sg, umem->nmap, i) {
+ /* Walk SGL and reduce max page size if VA/PA bits differ
+ * for any address.
+ */
+ mask |= (sg_dma_address(sg) + pgoff) ^ va;
+ if (i && i != (umem->nmap - 1))
+ /* restrict by length as well for interior SGEs */
+ mask |= sg_dma_len(sg);
+ va += sg_dma_len(sg) - pgoff;
+ pgoff = 0;
+ }
+ best_pg_bit = rdma_find_pg_bit(mask, pgsz_bitmap);
+
+ return BIT_ULL(best_pg_bit);
+}
+EXPORT_SYMBOL(ib_umem_find_best_pgsz);
+
/**
* ib_umem_get - Pin and DMA map userspace memory.
*
struct ib_ucontext *context;
struct ib_umem *umem;
struct page **page_list;
- struct vm_area_struct **vma_list;
unsigned long lock_limit;
unsigned long new_pinned;
unsigned long cur_base;
struct mm_struct *mm;
unsigned long npages;
int ret;
- int i;
unsigned long dma_attrs = 0;
- struct scatterlist *sg, *sg_list_start;
+ struct scatterlist *sg;
unsigned int gup_flags = FOLL_WRITE;
if (!udata)
mmgrab(mm);
if (access & IB_ACCESS_ON_DEMAND) {
+ if (WARN_ON_ONCE(!context->invalidate_range)) {
+ ret = -EINVAL;
+ goto umem_kfree;
+ }
+
ret = ib_umem_odp_get(to_ib_umem_odp(umem), access);
if (ret)
goto umem_kfree;
return umem;
}
- /* We assume the memory is from hugetlb until proved otherwise */
- umem->hugetlb = 1;
-
page_list = (struct page **) __get_free_page(GFP_KERNEL);
if (!page_list) {
ret = -ENOMEM;
goto umem_kfree;
}
- /*
- * if we can't alloc the vma_list, it's not so bad;
- * just assume the memory is not hugetlb memory
- */
- vma_list = (struct vm_area_struct **) __get_free_page(GFP_KERNEL);
- if (!vma_list)
- umem->hugetlb = 0;
-
npages = ib_umem_num_pages(umem);
if (npages == 0 || npages > UINT_MAX) {
ret = -EINVAL;
if (!umem->writable)
gup_flags |= FOLL_FORCE;
- sg_list_start = umem->sg_head.sgl;
+ sg = umem->sg_head.sgl;
while (npages) {
down_read(&mm->mmap_sem);
ret = get_user_pages_longterm(cur_base,
min_t(unsigned long, npages,
PAGE_SIZE / sizeof (struct page *)),
- gup_flags, page_list, vma_list);
+ gup_flags, page_list, NULL);
if (ret < 0) {
up_read(&mm->mmap_sem);
goto umem_release;
}
- umem->npages += ret;
cur_base += ret * PAGE_SIZE;
npages -= ret;
- /* Continue to hold the mmap_sem as vma_list access
- * needs to be protected.
- */
- for_each_sg(sg_list_start, sg, ret, i) {
- if (vma_list && !is_vm_hugetlb_page(vma_list[i]))
- umem->hugetlb = 0;
+ sg = ib_umem_add_sg_table(sg, page_list, ret,
+ dma_get_max_seg_size(context->device->dma_device),
+ &umem->sg_nents);
- sg_set_page(sg, page_list[i], PAGE_SIZE, 0);
- }
up_read(&mm->mmap_sem);
-
- /* preparing for next loop */
- sg_list_start = sg;
}
+ sg_mark_end(sg);
+
umem->nmap = ib_dma_map_sg_attrs(context->device,
umem->sg_head.sgl,
- umem->npages,
+ umem->sg_nents,
DMA_BIDIRECTIONAL,
dma_attrs);
vma:
atomic64_sub(ib_umem_num_pages(umem), &mm->pinned_vm);
out:
- if (vma_list)
- free_page((unsigned long) vma_list);
free_page((unsigned long) page_list);
umem_kfree:
if (ret) {
return -EINVAL;
}
- ret = sg_pcopy_to_buffer(umem->sg_head.sgl, umem->npages, dst, length,
+ ret = sg_pcopy_to_buffer(umem->sg_head.sgl, umem->sg_nents, dst, length,
offset + ib_umem_offset(umem));
if (ret < 0)
per_mm->mm = mm;
per_mm->umem_tree = RB_ROOT_CACHED;
init_rwsem(&per_mm->umem_rwsem);
- per_mm->active = ctx->invalidate_range;
+ per_mm->active = true;
rcu_read_lock();
per_mm->tgid = get_task_pid(current->group_leader, PIDTYPE_PID);
h = hstate_vma(vma);
umem->page_shift = huge_page_shift(h);
up_read(&mm->mmap_sem);
- umem->hugetlb = 1;
- } else {
- umem->hugetlb = 0;
}
mutex_init(&umem_odp->umem_mutex);
struct ib_umem *umem = &umem_odp->umem;
struct ib_device *dev = umem->context->device;
dma_addr_t dma_addr;
- int stored_page = 0;
int remove_existing_mapping = 0;
int ret = 0;
}
umem_odp->dma_list[page_index] = dma_addr | access_mask;
umem_odp->page_list[page_index] = page;
- umem->npages++;
- stored_page = 1;
+ umem_odp->npages++;
} else if (umem_odp->page_list[page_index] == page) {
umem_odp->dma_list[page_index] |= access_mask;
} else {
}
out:
- /* On Demand Paging - avoid pinning the page */
- if (umem->context->invalidate_range || !stored_page)
- put_page(page);
+ put_page(page);
- if (remove_existing_mapping && umem->context->invalidate_range) {
+ if (remove_existing_mapping) {
ib_umem_notifier_start_account(umem_odp);
umem->context->invalidate_range(
umem_odp,
*/
set_page_dirty(head_page);
}
- /* on demand pinning support */
- if (!umem->context->invalidate_range)
- put_page(page);
umem_odp->page_list[idx] = NULL;
umem_odp->dma_list[idx] = 0;
- umem->npages--;
+ umem_odp->npages--;
}
}
mutex_unlock(&umem_odp->umem_mutex);
struct ib_user_mad mad;
};
+#define CREATE_TRACE_POINTS
+#include <trace/events/ib_umad.h>
+
static const dev_t base_umad_dev = MKDEV(IB_UMAD_MAJOR, IB_UMAD_MINOR_BASE);
static const dev_t base_issm_dev = MKDEV(IB_UMAD_MAJOR, IB_UMAD_MINOR_BASE) +
IB_UMAD_NUM_FIXED_MINOR;
return -EFAULT;
}
}
+
+ trace_ib_umad_read_recv(file, &packet->mad.hdr, &recv_buf->mad->mad_hdr);
+
return hdr_size(file) + packet->length;
}
if (copy_to_user(buf, packet->mad.data, packet->length))
return -EFAULT;
+ trace_ib_umad_read_send(file, &packet->mad.hdr,
+ (struct ib_mad_hdr *)&packet->mad.data);
+
return size;
}
mutex_lock(&file->mutex);
+ trace_ib_umad_write(file, &packet->mad.hdr,
+ (struct ib_mad_hdr *)&packet->mad.data);
+
agent = __get_agent(file, packet->mad.hdr.id);
if (!agent) {
ret = -EINVAL;
goto out;
}
+ if (!rdma_dev_access_netns(port->ib_dev, current->nsproxy->net_ns)) {
+ ret = -EPERM;
+ goto out;
+ }
+
file = kzalloc(sizeof(*file), GFP_KERNEL);
if (!file) {
ret = -ENOMEM;
}
}
+ if (!rdma_dev_access_netns(port->ib_dev, current->nsproxy->net_ns)) {
+ ret = -EPERM;
+ goto err_up_sem;
+ }
+
ret = ib_modify_port(port->ib_dev, port->port_num, 0, &props);
if (ret)
goto err_up_sem;
struct list_head umaps;
struct page *disassociate_page;
- struct idr idr;
- /* spinlock protects write access to idr */
- spinlock_t idr_lock;
+ struct xarray idr;
};
struct ib_uverbs_event {
void ib_uverbs_event_handler(struct ib_event_handler *handler,
struct ib_event *event);
int ib_uverbs_dealloc_xrcd(struct ib_uobject *uobject, struct ib_xrcd *xrcd,
- enum rdma_remove_reason why);
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs);
int uverbs_dealloc_mw(struct ib_mw *mw);
void ib_uverbs_detach_umcast(struct ib_qp *qp,
const void __user *res = iter->cur;
if (iter->cur + len > iter->end)
- return ERR_PTR(-ENOSPC);
+ return (void __force __user *)ERR_PTR(-ENOSPC);
iter->cur += len;
return res;
}
}
static struct ib_uverbs_completion_event_file *
-_ib_uverbs_lookup_comp_file(s32 fd, const struct uverbs_attr_bundle *attrs)
+_ib_uverbs_lookup_comp_file(s32 fd, struct uverbs_attr_bundle *attrs)
{
struct ib_uobject *uobj = ufd_get_read(UVERBS_OBJECT_COMP_CHANNEL,
fd, attrs);
goto err_alloc;
}
+ attrs->context = ucontext;
+
ucontext->res.type = RDMA_RESTRACK_CTX;
ucontext->device = ib_dev;
ucontext->cg_obj = cg_obj;
atomic_set(&pd->usecnt, 0);
pd->res.type = RDMA_RESTRACK_PD;
- ret = ib_dev->ops.alloc_pd(pd, uobj->context, &attrs->driver_udata);
+ ret = ib_dev->ops.alloc_pd(pd, &attrs->driver_udata);
if (ret)
goto err_alloc;
if (ret)
goto err_copy;
- return uobj_alloc_commit(uobj);
+ return uobj_alloc_commit(uobj, attrs);
err_copy:
- ib_dealloc_pd(pd);
+ ib_dealloc_pd_user(pd, &attrs->driver_udata);
pd = NULL;
err_alloc:
kfree(pd);
err:
- uobj_alloc_abort(uobj);
+ uobj_alloc_abort(uobj, attrs);
return ret;
}
}
if (!xrcd) {
- xrcd = ib_dev->ops.alloc_xrcd(ib_dev, obj->uobject.context,
- &attrs->driver_udata);
+ xrcd = ib_dev->ops.alloc_xrcd(ib_dev, &attrs->driver_udata);
if (IS_ERR(xrcd)) {
ret = PTR_ERR(xrcd);
goto err;
mutex_unlock(&ibudev->xrcd_tree_mutex);
- return uobj_alloc_commit(&obj->uobject);
+ return uobj_alloc_commit(&obj->uobject, attrs);
err_copy:
if (inode) {
}
err_dealloc_xrcd:
- ib_dealloc_xrcd(xrcd);
+ ib_dealloc_xrcd(xrcd, &attrs->driver_udata);
err:
- uobj_alloc_abort(&obj->uobject);
+ uobj_alloc_abort(&obj->uobject, attrs);
err_tree_mutex_unlock:
if (f.file)
return uobj_perform_destroy(UVERBS_OBJECT_XRCD, cmd.xrcd_handle, attrs);
}
-int ib_uverbs_dealloc_xrcd(struct ib_uobject *uobject,
- struct ib_xrcd *xrcd,
- enum rdma_remove_reason why)
+int ib_uverbs_dealloc_xrcd(struct ib_uobject *uobject, struct ib_xrcd *xrcd,
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct inode *inode;
int ret;
- struct ib_uverbs_device *dev = uobject->context->ufile->device;
+ struct ib_uverbs_device *dev = attrs->ufile->device;
inode = xrcd->inode;
if (inode && !atomic_dec_and_test(&xrcd->usecnt))
return 0;
- ret = ib_dealloc_xrcd(xrcd);
+ ret = ib_dealloc_xrcd(xrcd, &attrs->driver_udata);
if (ib_is_destroy_retryable(ret, why, uobject)) {
atomic_inc(&xrcd->usecnt);
uobj_put_obj_read(pd);
- return uobj_alloc_commit(uobj);
+ return uobj_alloc_commit(uobj, attrs);
err_copy:
- ib_dereg_mr(mr);
+ ib_dereg_mr_user(mr, &attrs->driver_udata);
err_put:
uobj_put_obj_read(pd);
err_free:
- uobj_alloc_abort(uobj);
+ uobj_alloc_abort(uobj, attrs);
return ret;
}
goto err_copy;
uobj_put_obj_read(pd);
- return uobj_alloc_commit(uobj);
+ return uobj_alloc_commit(uobj, attrs);
err_copy:
uverbs_dealloc_mw(mw);
err_put:
uobj_put_obj_read(pd);
err_free:
- uobj_alloc_abort(uobj);
+ uobj_alloc_abort(uobj, attrs);
return ret;
}
ret = uverbs_response(attrs, &resp, sizeof(resp));
if (ret) {
- uobj_alloc_abort(uobj);
+ uobj_alloc_abort(uobj, attrs);
return ret;
}
- return uobj_alloc_commit(uobj);
+ return uobj_alloc_commit(uobj, attrs);
}
static struct ib_ucq_object *create_cq(struct uverbs_attr_bundle *attrs,
attr.comp_vector = cmd->comp_vector;
attr.flags = cmd->flags;
- cq = ib_dev->ops.create_cq(ib_dev, &attr, obj->uobject.context,
- &attrs->driver_udata);
+ cq = ib_dev->ops.create_cq(ib_dev, &attr, &attrs->driver_udata);
if (IS_ERR(cq)) {
ret = PTR_ERR(cq);
goto err_file;
if (ret)
goto err_cb;
- ret = uobj_alloc_commit(&obj->uobject);
+ ret = uobj_alloc_commit(&obj->uobject, attrs);
if (ret)
return ERR_PTR(ret);
return obj;
ib_uverbs_release_ucq(attrs->ufile, ev_file, obj);
err:
- uobj_alloc_abort(&obj->uobject);
+ uobj_alloc_abort(&obj->uobject, attrs);
return ERR_PTR(ret);
}
if (ret)
goto err_cb;
- qp->real_qp = qp;
qp->pd = pd;
qp->send_cq = attr.send_cq;
qp->recv_cq = attr.recv_cq;
if (ind_tbl)
uobj_put_obj_read(ind_tbl);
- return uobj_alloc_commit(&obj->uevent.uobject);
+ return uobj_alloc_commit(&obj->uevent.uobject, attrs);
err_cb:
ib_destroy_qp(qp);
if (ind_tbl)
uobj_put_obj_read(ind_tbl);
- uobj_alloc_abort(&obj->uevent.uobject);
+ uobj_alloc_abort(&obj->uevent.uobject, attrs);
return ret;
}
qp->uobject = &obj->uevent.uobject;
uobj_put_read(xrcd_uobj);
- return uobj_alloc_commit(&obj->uevent.uobject);
+ return uobj_alloc_commit(&obj->uevent.uobject, attrs);
err_destroy:
ib_destroy_qp(qp);
err_xrcd:
uobj_put_read(xrcd_uobj);
err_put:
- uobj_alloc_abort(&obj->uevent.uobject);
+ uobj_alloc_abort(&obj->uevent.uobject, attrs);
return ret;
}
goto err_copy;
uobj_put_obj_read(pd);
- return uobj_alloc_commit(uobj);
+ return uobj_alloc_commit(uobj, attrs);
err_copy:
rdma_destroy_ah(ah, RDMA_DESTROY_AH_SLEEPABLE);
uobj_put_obj_read(pd);
err:
- uobj_alloc_abort(uobj);
+ uobj_alloc_abort(uobj, attrs);
return ret;
}
uobj_put_obj_read(pd);
uobj_put_obj_read(cq);
- return uobj_alloc_commit(&obj->uevent.uobject);
+ return uobj_alloc_commit(&obj->uevent.uobject, attrs);
err_copy:
- ib_destroy_wq(wq);
+ ib_destroy_wq(wq, &attrs->driver_udata);
err_put_cq:
uobj_put_obj_read(cq);
err_put_pd:
uobj_put_obj_read(pd);
err_uobj:
- uobj_alloc_abort(&obj->uevent.uobject);
+ uobj_alloc_abort(&obj->uevent.uobject, attrs);
return err;
}
for (j = 0; j < num_read_wqs; j++)
uobj_put_obj_read(wqs[j]);
- return uobj_alloc_commit(uobj);
+ return uobj_alloc_commit(uobj, attrs);
err_copy:
ib_destroy_rwq_ind_table(rwq_ind_tbl);
err_uobj:
- uobj_alloc_abort(uobj);
+ uobj_alloc_abort(uobj, attrs);
put_wqs:
for (j = 0; j < num_read_wqs; j++)
uobj_put_obj_read(wqs[j]);
kfree(flow_attr);
if (cmd.flow_attr.num_of_specs)
kfree(kern_flow_attr);
- return uobj_alloc_commit(uobj);
+ return uobj_alloc_commit(uobj, attrs);
err_copy:
if (!qp->device->ops.destroy_flow(flow_id))
atomic_dec(&qp->usecnt);
err_put:
uobj_put_obj_read(qp);
err_uobj:
- uobj_alloc_abort(uobj);
+ uobj_alloc_abort(uobj, attrs);
err_free_attr:
if (cmd.flow_attr.num_of_specs)
kfree(kern_flow_attr);
obj->uevent.events_reported = 0;
INIT_LIST_HEAD(&obj->uevent.event_list);
- srq = pd->device->ops.create_srq(pd, &attr, udata);
- if (IS_ERR(srq)) {
- ret = PTR_ERR(srq);
+ srq = rdma_zalloc_drv_obj(ib_dev, ib_srq);
+ if (!srq) {
+ ret = -ENOMEM;
goto err_put;
}
srq->event_handler = attr.event_handler;
srq->srq_context = attr.srq_context;
+ ret = pd->device->ops.create_srq(srq, &attr, udata);
+ if (ret)
+ goto err_free;
+
if (ib_srq_has_cq(cmd->srq_type)) {
srq->ext.cq = attr.ext.cq;
atomic_inc(&attr.ext.cq->usecnt);
uobj_put_obj_read(attr.ext.cq);
uobj_put_obj_read(pd);
- return uobj_alloc_commit(&obj->uevent.uobject);
+ return uobj_alloc_commit(&obj->uevent.uobject, attrs);
err_copy:
- ib_destroy_srq(srq);
+ ib_destroy_srq_user(srq, &attrs->driver_udata);
+err_free:
+ kfree(srq);
err_put:
uobj_put_obj_read(pd);
}
err:
- uobj_alloc_abort(&obj->uevent.uobject);
+ uobj_alloc_abort(&obj->uevent.uobject, attrs);
return ret;
}
for (i = 0; i != array_len; i++) {
attr->uobjects[i] = uverbs_get_uobject_from_file(
- spec->u2.objs_arr.obj_type, pbundle->bundle.ufile,
- spec->u2.objs_arr.access, idr_vals[i]);
+ spec->u2.objs_arr.obj_type, spec->u2.objs_arr.access,
+ idr_vals[i], &pbundle->bundle);
if (IS_ERR(attr->uobjects[i])) {
ret = PTR_ERR(attr->uobjects[i]);
break;
}
- pbundle->bundle.context = attr->uobjects[i]->context;
}
attr->len = i;
static int uverbs_free_idrs_array(const struct uverbs_api_attr *attr_uapi,
struct uverbs_objs_arr_attr *attr,
- bool commit)
+ bool commit, struct uverbs_attr_bundle *attrs)
{
const struct uverbs_attr_spec *spec = &attr_uapi->spec;
int current_ret;
size_t i;
for (i = 0; i != attr->len; i++) {
- current_ret = uverbs_finalize_object(
- attr->uobjects[i], spec->u2.objs_arr.access, commit);
+ current_ret = uverbs_finalize_object(attr->uobjects[i],
+ spec->u2.objs_arr.access,
+ commit, attrs);
if (!ret)
ret = current_ret;
}
* IDR implementation today rejects negative IDs
*/
o_attr->uobject = uverbs_get_uobject_from_file(
- spec->u.obj.obj_type,
- pbundle->bundle.ufile,
- spec->u.obj.access,
- uattr->data_s64);
+ spec->u.obj.obj_type, spec->u.obj.access,
+ uattr->data_s64, &pbundle->bundle);
if (IS_ERR(o_attr->uobject))
return PTR_ERR(o_attr->uobject);
- pbundle->bundle.context = o_attr->uobject->context;
__set_bit(attr_bkey, pbundle->uobj_finalize);
if (spec->u.obj.access == UVERBS_ACCESS_NEW) {
uverbs_fill_udata(&pbundle->bundle,
&pbundle->bundle.driver_udata,
UVERBS_ATTR_UHW_IN, UVERBS_ATTR_UHW_OUT);
+ else
+ pbundle->bundle.driver_udata = (struct ib_udata){};
if (destroy_bkey != UVERBS_API_ATTR_BKEY_LEN) {
struct uverbs_obj_attr *destroy_attr =
&pbundle->bundle.attrs[destroy_bkey].obj_attr;
- ret = uobj_destroy(destroy_attr->uobject);
+ ret = uobj_destroy(destroy_attr->uobject, &pbundle->bundle);
if (ret)
return ret;
__clear_bit(destroy_bkey, pbundle->uobj_finalize);
current_ret = uverbs_finalize_object(
attr->obj_attr.uobject,
- attr->obj_attr.attr_elm->spec.u.obj.access, commit);
+ attr->obj_attr.attr_elm->spec.u.obj.access, commit,
+ &pbundle->bundle);
if (!ret)
ret = current_ret;
}
if (attr_uapi->spec.type == UVERBS_ATTR_TYPE_IDRS_ARRAY) {
current_ret = uverbs_free_idrs_array(
- attr_uapi, &attr->objs_arr_attr, commit);
+ attr_uapi, &attr->objs_arr_attr, commit,
+ &pbundle->bundle);
if (!ret)
ret = current_ret;
}
* then the command request structure starts
* with a '__aligned u64 response' member.
*/
- ret = get_user(response, (const u64 *)buf);
+ ret = get_user(response, (const u64 __user *)buf);
if (ret)
goto out_unlock;
.fault = rdma_umap_fault,
};
-static struct rdma_umap_priv *rdma_user_mmap_pre(struct ib_ucontext *ucontext,
- struct vm_area_struct *vma,
- unsigned long size)
+/*
+ * Map IO memory into a process. This is to be called by drivers as part of
+ * their mmap() functions if they wish to send something like PCI-E BAR memory
+ * to userspace.
+ */
+int rdma_user_mmap_io(struct ib_ucontext *ucontext, struct vm_area_struct *vma,
+ unsigned long pfn, unsigned long size, pgprot_t prot)
{
struct ib_uverbs_file *ufile = ucontext->ufile;
struct rdma_umap_priv *priv;
if (!(vma->vm_flags & VM_SHARED))
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
if (vma->vm_end - vma->vm_start != size)
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
/* Driver is using this wrong, must be called by ib_uverbs_mmap */
if (WARN_ON(!vma->vm_file ||
vma->vm_file->private_data != ufile))
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
lockdep_assert_held(&ufile->device->disassociate_srcu);
priv = kzalloc(sizeof(*priv), GFP_KERNEL);
if (!priv)
- return ERR_PTR(-ENOMEM);
- return priv;
-}
-
-/*
- * Map IO memory into a process. This is to be called by drivers as part of
- * their mmap() functions if they wish to send something like PCI-E BAR memory
- * to userspace.
- */
-int rdma_user_mmap_io(struct ib_ucontext *ucontext, struct vm_area_struct *vma,
- unsigned long pfn, unsigned long size, pgprot_t prot)
-{
- struct rdma_umap_priv *priv = rdma_user_mmap_pre(ucontext, vma, size);
-
- if (IS_ERR(priv))
- return PTR_ERR(priv);
+ return -ENOMEM;
vma->vm_page_prot = prot;
if (io_remap_pfn_range(vma, vma->vm_start, pfn, size, prot)) {
}
EXPORT_SYMBOL(rdma_user_mmap_io);
-/*
- * The page case is here for a slightly different reason, the driver expects
- * to be able to free the page it is sharing to user space when it destroys
- * its ucontext, which means we need to zap the user space references.
- *
- * We could handle this differently by providing an API to allocate a shared
- * page and then only freeing the shared page when the last ufile is
- * destroyed.
- */
-int rdma_user_mmap_page(struct ib_ucontext *ucontext,
- struct vm_area_struct *vma, struct page *page,
- unsigned long size)
-{
- struct rdma_umap_priv *priv = rdma_user_mmap_pre(ucontext, vma, size);
-
- if (IS_ERR(priv))
- return PTR_ERR(priv);
-
- if (remap_pfn_range(vma, vma->vm_start, page_to_pfn(page), size,
- vma->vm_page_prot)) {
- kfree(priv);
- return -EAGAIN;
- }
-
- rdma_umap_priv_init(priv, vma);
- return 0;
-}
-EXPORT_SYMBOL(rdma_user_mmap_page);
-
void uverbs_user_mmap_disassociate(struct ib_uverbs_file *ufile)
{
struct rdma_umap_priv *priv, *next_priv;
goto err;
}
+ if (!rdma_dev_access_netns(ib_dev, current->nsproxy->net_ns)) {
+ ret = -EPERM;
+ goto err;
+ }
+
/* In case IB device supports disassociate ucontext, there is no hard
* dependency between uverbs device and its low level device.
*/
#include "uverbs.h"
static int uverbs_free_ah(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
- return rdma_destroy_ah((struct ib_ah *)uobject->object,
- RDMA_DESTROY_AH_SLEEPABLE);
+ return rdma_destroy_ah_user((struct ib_ah *)uobject->object,
+ RDMA_DESTROY_AH_SLEEPABLE,
+ &attrs->driver_udata);
}
static int uverbs_free_flow(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_flow *flow = (struct ib_flow *)uobject->object;
struct ib_uflow_object *uflow =
}
static int uverbs_free_mw(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
return uverbs_dealloc_mw((struct ib_mw *)uobject->object);
}
static int uverbs_free_qp(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_qp *qp = uobject->object;
struct ib_uqp_object *uqp =
ib_uverbs_detach_umcast(qp, uqp);
}
- ret = ib_destroy_qp(qp);
+ ret = ib_destroy_qp_user(qp, &attrs->driver_udata);
if (ib_is_destroy_retryable(ret, why, uobject))
return ret;
if (uqp->uxrcd)
atomic_dec(&uqp->uxrcd->refcnt);
- ib_uverbs_release_uevent(uobject->context->ufile, &uqp->uevent);
+ ib_uverbs_release_uevent(attrs->ufile, &uqp->uevent);
return ret;
}
static int uverbs_free_rwq_ind_tbl(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_rwq_ind_table *rwq_ind_tbl = uobject->object;
struct ib_wq **ind_tbl = rwq_ind_tbl->ind_tbl;
}
static int uverbs_free_wq(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_wq *wq = uobject->object;
struct ib_uwq_object *uwq =
container_of(uobject, struct ib_uwq_object, uevent.uobject);
int ret;
- ret = ib_destroy_wq(wq);
+ ret = ib_destroy_wq(wq, &attrs->driver_udata);
if (ib_is_destroy_retryable(ret, why, uobject))
return ret;
- ib_uverbs_release_uevent(uobject->context->ufile, &uwq->uevent);
+ ib_uverbs_release_uevent(attrs->ufile, &uwq->uevent);
return ret;
}
static int uverbs_free_srq(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_srq *srq = uobject->object;
struct ib_uevent_object *uevent =
enum ib_srq_type srq_type = srq->srq_type;
int ret;
- ret = ib_destroy_srq(srq);
+ ret = ib_destroy_srq_user(srq, &attrs->driver_udata);
if (ib_is_destroy_retryable(ret, why, uobject))
return ret;
atomic_dec(&us->uxrcd->refcnt);
}
- ib_uverbs_release_uevent(uobject->context->ufile, uevent);
+ ib_uverbs_release_uevent(attrs->ufile, uevent);
return ret;
}
static int uverbs_free_xrcd(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_xrcd *xrcd = uobject->object;
struct ib_uxrcd_object *uxrcd =
if (ret)
return ret;
- mutex_lock(&uobject->context->ufile->device->xrcd_tree_mutex);
- ret = ib_uverbs_dealloc_xrcd(uobject, xrcd, why);
- mutex_unlock(&uobject->context->ufile->device->xrcd_tree_mutex);
+ mutex_lock(&attrs->ufile->device->xrcd_tree_mutex);
+ ret = ib_uverbs_dealloc_xrcd(uobject, xrcd, why, attrs);
+ mutex_unlock(&attrs->ufile->device->xrcd_tree_mutex);
return ret;
}
static int uverbs_free_pd(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_pd *pd = uobject->object;
int ret;
if (ret)
return ret;
- ib_dealloc_pd(pd);
+ ib_dealloc_pd_user(pd, &attrs->driver_udata);
return 0;
}
* SOFTWARE.
*/
+#include "rdma_core.h"
#include "uverbs.h"
#include <rdma/uverbs_std_types.h>
static int uverbs_free_counters(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_counters *counters = uobject->object;
int ret;
{
struct ib_uobject *uobj = uverbs_attr_get_uobject(
attrs, UVERBS_ATTR_CREATE_COUNTERS_HANDLE);
- struct ib_device *ib_dev = uobj->context->device;
+ struct ib_device *ib_dev = attrs->context->device;
struct ib_counters *counters;
int ret;
#include "uverbs.h"
static int uverbs_free_cq(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_cq *cq = uobject->object;
struct ib_uverbs_event_queue *ev_queue = cq->cq_context;
container_of(uobject, struct ib_ucq_object, uobject);
int ret;
- ret = ib_destroy_cq(cq);
+ ret = ib_destroy_cq_user(cq, &attrs->driver_udata);
if (ib_is_destroy_retryable(ret, why, uobject))
return ret;
ib_uverbs_release_ucq(
- uobject->context->ufile,
+ attrs->ufile,
ev_queue ? container_of(ev_queue,
struct ib_uverbs_completion_event_file,
ev_queue) :
struct ib_ucq_object *obj = container_of(
uverbs_attr_get_uobject(attrs, UVERBS_ATTR_CREATE_CQ_HANDLE),
typeof(*obj), uobject);
- struct ib_device *ib_dev = obj->uobject.context->device;
+ struct ib_device *ib_dev = attrs->context->device;
int ret;
u64 user_handle;
struct ib_cq_init_attr attr = {};
INIT_LIST_HEAD(&obj->comp_list);
INIT_LIST_HEAD(&obj->async_list);
- cq = ib_dev->ops.create_cq(ib_dev, &attr, obj->uobject.context,
- &attrs->driver_udata);
+ cq = ib_dev->ops.create_cq(ib_dev, &attr, &attrs->driver_udata);
if (IS_ERR(cq)) {
ret = PTR_ERR(cq);
goto err_event_file;
* SOFTWARE.
*/
+#include "rdma_core.h"
#include "uverbs.h"
#include <rdma/uverbs_std_types.h>
static int uverbs_free_dm(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_dm *dm = uobject->object;
int ret;
if (ret)
return ret;
- return dm->device->ops.dealloc_dm(dm);
+ return dm->device->ops.dealloc_dm(dm, attrs);
}
static int UVERBS_HANDLER(UVERBS_METHOD_DM_ALLOC)(
struct ib_uobject *uobj =
uverbs_attr_get(attrs, UVERBS_ATTR_ALLOC_DM_HANDLE)
->obj_attr.uobject;
- struct ib_device *ib_dev = uobj->context->device;
+ struct ib_device *ib_dev = attrs->context->device;
struct ib_dm *dm;
int ret;
if (ret)
return ret;
- dm = ib_dev->ops.alloc_dm(ib_dev, uobj->context, &attr, attrs);
+ dm = ib_dev->ops.alloc_dm(ib_dev, attrs->context, &attr, attrs);
if (IS_ERR(dm))
return PTR_ERR(dm);
* SOFTWARE.
*/
+#include "rdma_core.h"
#include "uverbs.h"
#include <rdma/uverbs_std_types.h>
static int uverbs_free_flow_action(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct ib_flow_action *action = uobject->object;
int ret;
{
struct ib_uobject *uobj = uverbs_attr_get_uobject(
attrs, UVERBS_ATTR_CREATE_FLOW_ACTION_ESP_HANDLE);
- struct ib_device *ib_dev = uobj->context->device;
+ struct ib_device *ib_dev = attrs->context->device;
int ret;
struct ib_flow_action *action;
struct ib_flow_action_esp_attr esp_attr = {};
* SOFTWARE.
*/
+#include "rdma_core.h"
#include "uverbs.h"
#include <rdma/uverbs_std_types.h>
static int uverbs_free_mr(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
- return ib_dereg_mr((struct ib_mr *)uobject->object);
+ return ib_dereg_mr_user((struct ib_mr *)uobject->object,
+ &attrs->driver_udata);
}
static int UVERBS_HANDLER(UVERBS_METHOD_ADVISE_MR)(
return 0;
err_dereg:
- ib_dereg_mr(mr);
+ ib_dereg_mr_user(mr, &attrs->driver_udata);
return ret;
}
return RDMA_TRANSPORT_USNIC_UDP;
if (node_type == RDMA_NODE_RNIC)
return RDMA_TRANSPORT_IWARP;
+ if (node_type == RDMA_NODE_UNSPECIFIED)
+ return RDMA_TRANSPORT_UNSPECIFIED;
return RDMA_TRANSPORT_IB;
}
pd->res.type = RDMA_RESTRACK_PD;
rdma_restrack_set_task(&pd->res, caller);
- ret = device->ops.alloc_pd(pd, NULL, NULL);
+ ret = device->ops.alloc_pd(pd, NULL);
if (ret) {
kfree(pd);
return ERR_PTR(ret);
/**
* ib_dealloc_pd - Deallocates a protection domain.
* @pd: The protection domain to deallocate.
+ * @udata: Valid user data or NULL for kernel object
*
* It is an error to call this function while any resources in the pd still
* exist. The caller is responsible to synchronously destroy them and
* guarantee no new allocations will happen.
*/
-void ib_dealloc_pd(struct ib_pd *pd)
+void ib_dealloc_pd_user(struct ib_pd *pd, struct ib_udata *udata)
{
int ret;
if (pd->__internal_mr) {
- ret = pd->device->ops.dereg_mr(pd->__internal_mr);
+ ret = pd->device->ops.dereg_mr(pd->__internal_mr, NULL);
WARN_ON(ret);
pd->__internal_mr = NULL;
}
WARN_ON(atomic_read(&pd->usecnt));
rdma_restrack_del(&pd->res);
- pd->device->ops.dealloc_pd(pd);
+ pd->device->ops.dealloc_pd(pd, udata);
kfree(pd);
}
-EXPORT_SYMBOL(ib_dealloc_pd);
+EXPORT_SYMBOL(ib_dealloc_pd_user);
/* Address handles */
u32 flags,
struct ib_udata *udata)
{
+ struct ib_device *device = pd->device;
struct ib_ah *ah;
+ int ret;
might_sleep_if(flags & RDMA_CREATE_AH_SLEEPABLE);
- if (!pd->device->ops.create_ah)
+ if (!device->ops.create_ah)
return ERR_PTR(-EOPNOTSUPP);
- ah = pd->device->ops.create_ah(pd, ah_attr, flags, udata);
+ ah = rdma_zalloc_drv_obj_gfp(
+ device, ib_ah,
+ (flags & RDMA_CREATE_AH_SLEEPABLE) ? GFP_KERNEL : GFP_ATOMIC);
+ if (!ah)
+ return ERR_PTR(-ENOMEM);
- if (!IS_ERR(ah)) {
- ah->device = pd->device;
- ah->pd = pd;
- ah->uobject = NULL;
- ah->type = ah_attr->type;
- ah->sgid_attr = rdma_update_sgid_attr(ah_attr, NULL);
+ ah->device = device;
+ ah->pd = pd;
+ ah->type = ah_attr->type;
+ ah->sgid_attr = rdma_update_sgid_attr(ah_attr, NULL);
- atomic_inc(&pd->usecnt);
+ ret = device->ops.create_ah(ah, ah_attr, flags, udata);
+ if (ret) {
+ kfree(ah);
+ return ERR_PTR(ret);
}
+ atomic_inc(&pd->usecnt);
return ah;
}
}
EXPORT_SYMBOL(rdma_query_ah);
-int rdma_destroy_ah(struct ib_ah *ah, u32 flags)
+int rdma_destroy_ah_user(struct ib_ah *ah, u32 flags, struct ib_udata *udata)
{
const struct ib_gid_attr *sgid_attr = ah->sgid_attr;
struct ib_pd *pd;
- int ret;
might_sleep_if(flags & RDMA_DESTROY_AH_SLEEPABLE);
pd = ah->pd;
- ret = ah->device->ops.destroy_ah(ah, flags);
- if (!ret) {
- atomic_dec(&pd->usecnt);
- if (sgid_attr)
- rdma_put_gid_attr(sgid_attr);
- }
- return ret;
+ ah->device->ops.destroy_ah(ah, flags);
+ atomic_dec(&pd->usecnt);
+ if (sgid_attr)
+ rdma_put_gid_attr(sgid_attr);
+
+ kfree(ah);
+ return 0;
}
-EXPORT_SYMBOL(rdma_destroy_ah);
+EXPORT_SYMBOL(rdma_destroy_ah_user);
/* Shared receive queues */
struct ib_srq_init_attr *srq_init_attr)
{
struct ib_srq *srq;
+ int ret;
if (!pd->device->ops.create_srq)
return ERR_PTR(-EOPNOTSUPP);
- srq = pd->device->ops.create_srq(pd, srq_init_attr, NULL);
-
- if (!IS_ERR(srq)) {
- srq->device = pd->device;
- srq->pd = pd;
- srq->uobject = NULL;
- srq->event_handler = srq_init_attr->event_handler;
- srq->srq_context = srq_init_attr->srq_context;
- srq->srq_type = srq_init_attr->srq_type;
- if (ib_srq_has_cq(srq->srq_type)) {
- srq->ext.cq = srq_init_attr->ext.cq;
- atomic_inc(&srq->ext.cq->usecnt);
- }
- if (srq->srq_type == IB_SRQT_XRC) {
- srq->ext.xrc.xrcd = srq_init_attr->ext.xrc.xrcd;
- atomic_inc(&srq->ext.xrc.xrcd->usecnt);
- }
- atomic_inc(&pd->usecnt);
- atomic_set(&srq->usecnt, 0);
+ srq = rdma_zalloc_drv_obj(pd->device, ib_srq);
+ if (!srq)
+ return ERR_PTR(-ENOMEM);
+
+ srq->device = pd->device;
+ srq->pd = pd;
+ srq->event_handler = srq_init_attr->event_handler;
+ srq->srq_context = srq_init_attr->srq_context;
+ srq->srq_type = srq_init_attr->srq_type;
+
+ if (ib_srq_has_cq(srq->srq_type)) {
+ srq->ext.cq = srq_init_attr->ext.cq;
+ atomic_inc(&srq->ext.cq->usecnt);
+ }
+ if (srq->srq_type == IB_SRQT_XRC) {
+ srq->ext.xrc.xrcd = srq_init_attr->ext.xrc.xrcd;
+ atomic_inc(&srq->ext.xrc.xrcd->usecnt);
+ }
+ atomic_inc(&pd->usecnt);
+
+ ret = pd->device->ops.create_srq(srq, srq_init_attr, NULL);
+ if (ret) {
+ atomic_dec(&srq->pd->usecnt);
+ if (srq->srq_type == IB_SRQT_XRC)
+ atomic_dec(&srq->ext.xrc.xrcd->usecnt);
+ if (ib_srq_has_cq(srq->srq_type))
+ atomic_dec(&srq->ext.cq->usecnt);
+ kfree(srq);
+ return ERR_PTR(ret);
}
return srq;
}
EXPORT_SYMBOL(ib_query_srq);
-int ib_destroy_srq(struct ib_srq *srq)
+int ib_destroy_srq_user(struct ib_srq *srq, struct ib_udata *udata)
{
- struct ib_pd *pd;
- enum ib_srq_type srq_type;
- struct ib_xrcd *uninitialized_var(xrcd);
- struct ib_cq *uninitialized_var(cq);
- int ret;
-
if (atomic_read(&srq->usecnt))
return -EBUSY;
- pd = srq->pd;
- srq_type = srq->srq_type;
- if (ib_srq_has_cq(srq_type))
- cq = srq->ext.cq;
- if (srq_type == IB_SRQT_XRC)
- xrcd = srq->ext.xrc.xrcd;
+ srq->device->ops.destroy_srq(srq, udata);
- ret = srq->device->ops.destroy_srq(srq);
- if (!ret) {
- atomic_dec(&pd->usecnt);
- if (srq_type == IB_SRQT_XRC)
- atomic_dec(&xrcd->usecnt);
- if (ib_srq_has_cq(srq_type))
- atomic_dec(&cq->usecnt);
- }
+ atomic_dec(&srq->pd->usecnt);
+ if (srq->srq_type == IB_SRQT_XRC)
+ atomic_dec(&srq->ext.xrc.xrcd->usecnt);
+ if (ib_srq_has_cq(srq->srq_type))
+ atomic_dec(&srq->ext.cq->usecnt);
+ kfree(srq);
- return ret;
+ return 0;
}
-EXPORT_SYMBOL(ib_destroy_srq);
+EXPORT_SYMBOL(ib_destroy_srq_user);
/* Queue pairs */
}
EXPORT_SYMBOL(ib_open_qp);
-static struct ib_qp *create_xrc_qp(struct ib_qp *qp,
- struct ib_qp_init_attr *qp_init_attr)
+static struct ib_qp *create_xrc_qp_user(struct ib_qp *qp,
+ struct ib_qp_init_attr *qp_init_attr,
+ struct ib_udata *udata)
{
struct ib_qp *real_qp = qp;
return qp;
}
-struct ib_qp *ib_create_qp(struct ib_pd *pd,
- struct ib_qp_init_attr *qp_init_attr)
+struct ib_qp *ib_create_qp_user(struct ib_pd *pd,
+ struct ib_qp_init_attr *qp_init_attr,
+ struct ib_udata *udata)
{
struct ib_device *device = pd ? pd->device : qp_init_attr->xrcd->device;
struct ib_qp *qp;
if (ret)
goto err;
- qp->real_qp = qp;
qp->qp_type = qp_init_attr->qp_type;
qp->rwq_ind_tbl = qp_init_attr->rwq_ind_tbl;
qp->port = 0;
if (qp_init_attr->qp_type == IB_QPT_XRC_TGT) {
- struct ib_qp *xrc_qp = create_xrc_qp(qp, qp_init_attr);
+ struct ib_qp *xrc_qp =
+ create_xrc_qp_user(qp, qp_init_attr, udata);
if (IS_ERR(xrc_qp)) {
ret = PTR_ERR(xrc_qp);
return ERR_PTR(ret);
}
-EXPORT_SYMBOL(ib_create_qp);
+EXPORT_SYMBOL(ib_create_qp_user);
static const struct {
int valid;
return 0;
}
-int ib_destroy_qp(struct ib_qp *qp)
+int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata)
{
const struct ib_gid_attr *alt_path_sgid_attr = qp->alt_path_sgid_attr;
const struct ib_gid_attr *av_sgid_attr = qp->av_sgid_attr;
rdma_rw_cleanup_mrs(qp);
rdma_restrack_del(&qp->res);
- ret = qp->device->ops.destroy_qp(qp);
+ ret = qp->device->ops.destroy_qp(qp, udata);
if (!ret) {
if (alt_path_sgid_attr)
rdma_put_gid_attr(alt_path_sgid_attr);
return ret;
}
-EXPORT_SYMBOL(ib_destroy_qp);
+EXPORT_SYMBOL(ib_destroy_qp_user);
/* Completion queues */
{
struct ib_cq *cq;
- cq = device->ops.create_cq(device, cq_attr, NULL, NULL);
+ cq = device->ops.create_cq(device, cq_attr, NULL);
if (!IS_ERR(cq)) {
cq->device = device;
}
EXPORT_SYMBOL(rdma_set_cq_moderation);
-int ib_destroy_cq(struct ib_cq *cq)
+int ib_destroy_cq_user(struct ib_cq *cq, struct ib_udata *udata)
{
if (atomic_read(&cq->usecnt))
return -EBUSY;
rdma_restrack_del(&cq->res);
- return cq->device->ops.destroy_cq(cq);
+ return cq->device->ops.destroy_cq(cq, udata);
}
-EXPORT_SYMBOL(ib_destroy_cq);
+EXPORT_SYMBOL(ib_destroy_cq_user);
int ib_resize_cq(struct ib_cq *cq, int cqe)
{
/* Memory regions */
-int ib_dereg_mr(struct ib_mr *mr)
+int ib_dereg_mr_user(struct ib_mr *mr, struct ib_udata *udata)
{
struct ib_pd *pd = mr->pd;
struct ib_dm *dm = mr->dm;
int ret;
rdma_restrack_del(&mr->res);
- ret = mr->device->ops.dereg_mr(mr);
+ ret = mr->device->ops.dereg_mr(mr, udata);
if (!ret) {
atomic_dec(&pd->usecnt);
if (dm)
return ret;
}
-EXPORT_SYMBOL(ib_dereg_mr);
+EXPORT_SYMBOL(ib_dereg_mr_user);
/**
* ib_alloc_mr() - Allocates a memory region
* @pd: protection domain associated with the region
* @mr_type: memory region type
* @max_num_sg: maximum sg entries available for registration.
+ * @udata: user data or null for kernel objects
*
* Notes:
* Memory registeration page/sg lists must not exceed max_num_sg.
* max_num_sg * used_page_size.
*
*/
-struct ib_mr *ib_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg)
+struct ib_mr *ib_alloc_mr_user(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
struct ib_mr *mr;
if (!pd->device->ops.alloc_mr)
return ERR_PTR(-EOPNOTSUPP);
- mr = pd->device->ops.alloc_mr(pd, mr_type, max_num_sg);
+ mr = pd->device->ops.alloc_mr(pd, mr_type, max_num_sg, udata);
if (!IS_ERR(mr)) {
mr->device = pd->device;
mr->pd = pd;
return mr;
}
-EXPORT_SYMBOL(ib_alloc_mr);
+EXPORT_SYMBOL(ib_alloc_mr_user);
/* "Fast" memory regions */
if (!device->ops.alloc_xrcd)
return ERR_PTR(-EOPNOTSUPP);
- xrcd = device->ops.alloc_xrcd(device, NULL, NULL);
+ xrcd = device->ops.alloc_xrcd(device, NULL);
if (!IS_ERR(xrcd)) {
xrcd->device = device;
xrcd->inode = NULL;
}
EXPORT_SYMBOL(__ib_alloc_xrcd);
-int ib_dealloc_xrcd(struct ib_xrcd *xrcd)
+int ib_dealloc_xrcd(struct ib_xrcd *xrcd, struct ib_udata *udata)
{
struct ib_qp *qp;
int ret;
return ret;
}
- return xrcd->device->ops.dealloc_xrcd(xrcd);
+ return xrcd->device->ops.dealloc_xrcd(xrcd, udata);
}
EXPORT_SYMBOL(ib_dealloc_xrcd);
EXPORT_SYMBOL(ib_create_wq);
/**
- * ib_destroy_wq - Destroys the specified WQ.
+ * ib_destroy_wq - Destroys the specified user WQ.
* @wq: The WQ to destroy.
+ * @udata: Valid user data
*/
-int ib_destroy_wq(struct ib_wq *wq)
+int ib_destroy_wq(struct ib_wq *wq, struct ib_udata *udata)
{
int err;
struct ib_cq *cq = wq->cq;
if (atomic_read(&wq->usecnt))
return -EBUSY;
- err = wq->device->ops.destroy_wq(wq);
+ err = wq->device->ops.destroy_wq(wq, udata);
if (!err) {
atomic_dec(&pd->usecnt);
atomic_dec(&cq->usecnt);
netdev, params.param);
}
EXPORT_SYMBOL(rdma_init_netdev);
+
+void __rdma_block_iter_start(struct ib_block_iter *biter,
+ struct scatterlist *sglist, unsigned int nents,
+ unsigned long pgsz)
+{
+ memset(biter, 0, sizeof(struct ib_block_iter));
+ biter->__sg = sglist;
+ biter->__sg_nents = nents;
+
+ /* Driver provides best block size to use */
+ biter->__pg_bit = __fls(pgsz);
+}
+EXPORT_SYMBOL(__rdma_block_iter_start);
+
+bool __rdma_block_iter_next(struct ib_block_iter *biter)
+{
+ unsigned int block_offset;
+
+ if (!biter->__sg_nents || !biter->__sg)
+ return false;
+
+ biter->__dma_addr = sg_dma_address(biter->__sg) + biter->__sg_advance;
+ block_offset = biter->__dma_addr & (BIT_ULL(biter->__pg_bit) - 1);
+ biter->__sg_advance += BIT_ULL(biter->__pg_bit) - block_offset;
+
+ if (biter->__sg_advance >= sg_dma_len(biter->__sg)) {
+ biter->__sg_advance = 0;
+ biter->__sg = sg_next(biter->__sg);
+ biter->__sg_nents--;
+ }
+
+ return true;
+}
+EXPORT_SYMBOL(__rdma_block_iter_next);
obj-$(CONFIG_INFINIBAND_QIB) += qib/
obj-$(CONFIG_INFINIBAND_CXGB3) += cxgb3/
obj-$(CONFIG_INFINIBAND_CXGB4) += cxgb4/
+obj-$(CONFIG_INFINIBAND_EFA) += efa/
obj-$(CONFIG_INFINIBAND_I40IW) += i40iw/
obj-$(CONFIG_MLX4_INFINIBAND) += mlx4/
obj-$(CONFIG_MLX5_INFINIBAND) += mlx5/
config INFINIBAND_BNXT_RE
- tristate "Broadcom Netxtreme HCA support"
- depends on 64BIT
- depends on ETHERNET && NETDEVICES && PCI && INET && DCB
- select NET_VENDOR_BROADCOM
- select BNXT
- ---help---
+ tristate "Broadcom Netxtreme HCA support"
+ depends on 64BIT
+ depends on ETHERNET && NETDEVICES && PCI && INET && DCB
+ select NET_VENDOR_BROADCOM
+ select BNXT
+ ---help---
This driver supports Broadcom NetXtreme-E 10/25/40/50 gigabit
RoCE HCAs. To compile this driver as a module, choose M here:
the module will be called bnxt_re.
}
/* Device */
-struct net_device *bnxt_re_get_netdev(struct ib_device *ibdev, u8 port_num)
-{
- struct bnxt_re_dev *rdev = to_bnxt_re_dev(ibdev, ibdev);
- struct net_device *netdev = NULL;
-
- rcu_read_lock();
- if (rdev)
- netdev = rdev->netdev;
- if (netdev)
- dev_hold(netdev);
-
- rcu_read_unlock();
- return netdev;
-}
-
int bnxt_re_query_device(struct ib_device *ibdev,
struct ib_device_attr *ib_attr,
struct ib_udata *udata)
struct bnxt_re_dev *rdev = to_bnxt_re_dev(attr->device, ibdev);
struct bnxt_qplib_sgid_tbl *sgid_tbl = &rdev->qplib_res.sgid_tbl;
- if ((attr->ndev) && is_vlan_dev(attr->ndev))
- vlan_id = vlan_dev_vlan_id(attr->ndev);
+ rc = rdma_read_gid_l2_fields(attr, &vlan_id, NULL);
+ if (rc)
+ return rc;
rc = bnxt_qplib_add_sgid(sgid_tbl, (struct bnxt_qplib_gid *)&attr->gid,
rdev->qplib_res.netdev->dev_addr,
}
/* Protection Domains */
-void bnxt_re_dealloc_pd(struct ib_pd *ib_pd)
+void bnxt_re_dealloc_pd(struct ib_pd *ib_pd, struct ib_udata *udata)
{
struct bnxt_re_pd *pd = container_of(ib_pd, struct bnxt_re_pd, ib_pd);
struct bnxt_re_dev *rdev = pd->rdev;
&pd->qplib_pd);
}
-int bnxt_re_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *ucontext,
- struct ib_udata *udata)
+int bnxt_re_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct ib_device *ibdev = ibpd->device;
struct bnxt_re_dev *rdev = to_bnxt_re_dev(ibdev, ibdev);
- struct bnxt_re_ucontext *ucntx = container_of(ucontext,
- struct bnxt_re_ucontext,
- ib_uctx);
+ struct bnxt_re_ucontext *ucntx = rdma_udata_to_drv_context(
+ udata, struct bnxt_re_ucontext, ib_uctx);
struct bnxt_re_pd *pd = container_of(ibpd, struct bnxt_re_pd, ib_pd);
int rc;
}
/* Address Handles */
-int bnxt_re_destroy_ah(struct ib_ah *ib_ah, u32 flags)
+void bnxt_re_destroy_ah(struct ib_ah *ib_ah, u32 flags)
{
struct bnxt_re_ah *ah = container_of(ib_ah, struct bnxt_re_ah, ib_ah);
struct bnxt_re_dev *rdev = ah->rdev;
- int rc;
- rc = bnxt_qplib_destroy_ah(&rdev->qplib_res, &ah->qplib_ah,
- !(flags & RDMA_DESTROY_AH_SLEEPABLE));
- if (rc) {
- dev_err(rdev_to_dev(rdev), "Failed to destroy HW AH");
- return rc;
- }
- kfree(ah);
- return 0;
+ bnxt_qplib_destroy_ah(&rdev->qplib_res, &ah->qplib_ah,
+ !(flags & RDMA_DESTROY_AH_SLEEPABLE));
}
static u8 bnxt_re_stack_to_dev_nw_type(enum rdma_network_type ntype)
return nw_type;
}
-struct ib_ah *bnxt_re_create_ah(struct ib_pd *ib_pd,
- struct rdma_ah_attr *ah_attr,
- u32 flags,
- struct ib_udata *udata)
+int bnxt_re_create_ah(struct ib_ah *ib_ah, struct rdma_ah_attr *ah_attr,
+ u32 flags, struct ib_udata *udata)
{
+ struct ib_pd *ib_pd = ib_ah->pd;
struct bnxt_re_pd *pd = container_of(ib_pd, struct bnxt_re_pd, ib_pd);
const struct ib_global_route *grh = rdma_ah_read_grh(ah_attr);
struct bnxt_re_dev *rdev = pd->rdev;
const struct ib_gid_attr *sgid_attr;
- struct bnxt_re_ah *ah;
+ struct bnxt_re_ah *ah = container_of(ib_ah, struct bnxt_re_ah, ib_ah);
u8 nw_type;
int rc;
if (!(rdma_ah_get_ah_flags(ah_attr) & IB_AH_GRH)) {
dev_err(rdev_to_dev(rdev), "Failed to alloc AH: GRH not set");
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
}
- ah = kzalloc(sizeof(*ah), GFP_ATOMIC);
- if (!ah)
- return ERR_PTR(-ENOMEM);
ah->rdev = rdev;
ah->qplib_ah.pd = &pd->qplib_pd;
!(flags & RDMA_CREATE_AH_SLEEPABLE));
if (rc) {
dev_err(rdev_to_dev(rdev), "Failed to allocate HW AH");
- goto fail;
+ return rc;
}
/* Write AVID to shared page. */
spin_unlock_irqrestore(&uctx->sh_lock, flag);
}
- return &ah->ib_ah;
-
-fail:
- kfree(ah);
- return ERR_PTR(rc);
+ return 0;
}
int bnxt_re_modify_ah(struct ib_ah *ib_ah, struct rdma_ah_attr *ah_attr)
}
/* Queue Pairs */
-int bnxt_re_destroy_qp(struct ib_qp *ib_qp)
+int bnxt_re_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
{
struct bnxt_re_qp *qp = container_of(ib_qp, struct bnxt_re_qp, ib_qp);
struct bnxt_re_dev *rdev = qp->rdev;
bnxt_qplib_free_qp_res(&rdev->qplib_res, &qp->qplib_qp);
if (ib_qp->qp_type == IB_QPT_GSI && rdev->qp1_sqp) {
- rc = bnxt_qplib_destroy_ah(&rdev->qplib_res,
- &rdev->sqp_ah->qplib_ah, false);
- if (rc) {
- dev_err(rdev_to_dev(rdev),
- "Failed to destroy HW AH for shadow QP");
- return rc;
- }
+ bnxt_qplib_destroy_ah(&rdev->qplib_res, &rdev->sqp_ah->qplib_ah,
+ false);
bnxt_qplib_clean_qp(&qp->qplib_qp);
rc = bnxt_qplib_destroy_qp(&rdev->qplib_res,
return PTR_ERR(umem);
qp->sumem = umem;
- qplib_qp->sq.sglist = umem->sg_head.sgl;
- qplib_qp->sq.nmap = umem->nmap;
+ qplib_qp->sq.sg_info.sglist = umem->sg_head.sgl;
+ qplib_qp->sq.sg_info.npages = ib_umem_num_pages(umem);
+ qplib_qp->sq.sg_info.nmap = umem->nmap;
qplib_qp->qp_handle = ureq.qp_handle;
if (!qp->qplib_qp.srq) {
if (IS_ERR(umem))
goto rqfail;
qp->rumem = umem;
- qplib_qp->rq.sglist = umem->sg_head.sgl;
- qplib_qp->rq.nmap = umem->nmap;
+ qplib_qp->rq.sg_info.sglist = umem->sg_head.sgl;
+ qplib_qp->rq.sg_info.npages = ib_umem_num_pages(umem);
+ qplib_qp->rq.sg_info.nmap = umem->nmap;
}
qplib_qp->dpi = &cntx->dpi;
rqfail:
ib_umem_release(qp->sumem);
qp->sumem = NULL;
- qplib_qp->sq.sglist = NULL;
- qplib_qp->sq.nmap = 0;
+ memset(&qplib_qp->sq.sg_info, 0, sizeof(qplib_qp->sq.sg_info));
return PTR_ERR(umem);
}
}
/* Shared Receive Queues */
-int bnxt_re_destroy_srq(struct ib_srq *ib_srq)
+void bnxt_re_destroy_srq(struct ib_srq *ib_srq, struct ib_udata *udata)
{
struct bnxt_re_srq *srq = container_of(ib_srq, struct bnxt_re_srq,
ib_srq);
struct bnxt_re_dev *rdev = srq->rdev;
struct bnxt_qplib_srq *qplib_srq = &srq->qplib_srq;
struct bnxt_qplib_nq *nq = NULL;
- int rc;
if (qplib_srq->cq)
nq = qplib_srq->cq->nq;
- rc = bnxt_qplib_destroy_srq(&rdev->qplib_res, qplib_srq);
- if (rc) {
- dev_err(rdev_to_dev(rdev), "Destroy HW SRQ failed!");
- return rc;
- }
-
+ bnxt_qplib_destroy_srq(&rdev->qplib_res, qplib_srq);
if (srq->umem)
ib_umem_release(srq->umem);
- kfree(srq);
atomic_dec(&rdev->srq_count);
if (nq)
nq->budget--;
- return 0;
}
static int bnxt_re_init_user_srq(struct bnxt_re_dev *rdev,
return PTR_ERR(umem);
srq->umem = umem;
- qplib_srq->nmap = umem->nmap;
- qplib_srq->sglist = umem->sg_head.sgl;
+ qplib_srq->sg_info.sglist = umem->sg_head.sgl;
+ qplib_srq->sg_info.npages = ib_umem_num_pages(umem);
+ qplib_srq->sg_info.nmap = umem->nmap;
qplib_srq->srq_handle = ureq.srq_handle;
qplib_srq->dpi = &cntx->dpi;
return 0;
}
-struct ib_srq *bnxt_re_create_srq(struct ib_pd *ib_pd,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata)
+int bnxt_re_create_srq(struct ib_srq *ib_srq,
+ struct ib_srq_init_attr *srq_init_attr,
+ struct ib_udata *udata)
{
+ struct ib_pd *ib_pd = ib_srq->pd;
struct bnxt_re_pd *pd = container_of(ib_pd, struct bnxt_re_pd, ib_pd);
struct bnxt_re_dev *rdev = pd->rdev;
struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr;
- struct bnxt_re_srq *srq;
+ struct bnxt_re_srq *srq =
+ container_of(ib_srq, struct bnxt_re_srq, ib_srq);
struct bnxt_qplib_nq *nq = NULL;
int rc, entries;
goto exit;
}
- srq = kzalloc(sizeof(*srq), GFP_KERNEL);
- if (!srq) {
- rc = -ENOMEM;
- goto exit;
- }
srq->rdev = rdev;
srq->qplib_srq.pd = &pd->qplib_pd;
srq->qplib_srq.dpi = &rdev->dpi_privileged;
nq->budget++;
atomic_inc(&rdev->srq_count);
- return &srq->ib_srq;
+ return 0;
fail:
if (srq->umem)
ib_umem_release(srq->umem);
- kfree(srq);
exit:
- return ERR_PTR(rc);
+ return rc;
}
int bnxt_re_modify_srq(struct ib_srq *ib_srq, struct ib_srq_attr *srq_attr,
qp_attr->ah_attr.roce.dmac);
sgid_attr = qp_attr->ah_attr.grh.sgid_attr;
- memcpy(qp->qplib_qp.smac, sgid_attr->ndev->dev_addr,
- ETH_ALEN);
+ rc = rdma_read_gid_l2_fields(sgid_attr, NULL,
+ &qp->qplib_qp.smac[0]);
+ if (rc)
+ return rc;
+
nw_type = rdma_gid_attr_network_type(sgid_attr);
switch (nw_type) {
case RDMA_NETWORK_IPV4:
memset(&qp->qp1_hdr, 0, sizeof(qp->qp1_hdr));
- if (is_vlan_dev(sgid_attr->ndev))
- vlan_id = vlan_dev_vlan_id(sgid_attr->ndev);
+ rc = rdma_read_gid_l2_fields(sgid_attr, &vlan_id, NULL);
+ if (rc)
+ return rc;
+
/* Get network header type for this GID */
nw_type = rdma_gid_attr_network_type(sgid_attr);
switch (nw_type) {
}
/* Completion Queues */
-int bnxt_re_destroy_cq(struct ib_cq *ib_cq)
+int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
{
int rc;
struct bnxt_re_cq *cq;
struct ib_cq *bnxt_re_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
struct bnxt_re_dev *rdev = to_bnxt_re_dev(ibdev, ibdev);
if (entries > dev_attr->max_cq_wqes + 1)
entries = dev_attr->max_cq_wqes + 1;
- if (context) {
+ if (udata) {
struct bnxt_re_cq_req req;
- struct bnxt_re_ucontext *uctx = container_of
- (context,
- struct bnxt_re_ucontext,
- ib_uctx);
+ struct bnxt_re_ucontext *uctx = rdma_udata_to_drv_context(
+ udata, struct bnxt_re_ucontext, ib_uctx);
if (ib_copy_from_udata(&req, udata, sizeof(req))) {
rc = -EFAULT;
goto fail;
rc = PTR_ERR(cq->umem);
goto fail;
}
- cq->qplib_cq.sghead = cq->umem->sg_head.sgl;
- cq->qplib_cq.nmap = cq->umem->nmap;
+ cq->qplib_cq.sg_info.sglist = cq->umem->sg_head.sgl;
+ cq->qplib_cq.sg_info.npages = ib_umem_num_pages(cq->umem);
+ cq->qplib_cq.sg_info.nmap = cq->umem->nmap;
cq->qplib_cq.dpi = &uctx->dpi;
} else {
cq->max_cql = min_t(u32, entries, MAX_CQL_PER_POLL);
}
cq->qplib_cq.dpi = &rdev->dpi_privileged;
- cq->qplib_cq.sghead = NULL;
- cq->qplib_cq.nmap = 0;
}
/*
* Allocating the NQ in a round robin fashion. nq_alloc_cnt is a
atomic_inc(&rdev->cq_count);
spin_lock_init(&cq->cq_lock);
- if (context) {
+ if (udata) {
struct bnxt_re_cq_resp resp;
resp.cqid = cq->qplib_cq.id;
return &cq->ib_cq;
c2fail:
- if (context)
+ if (udata)
ib_umem_release(cq->umem);
fail:
kfree(cq->cql);
return ERR_PTR(rc);
}
-int bnxt_re_dereg_mr(struct ib_mr *ib_mr)
+int bnxt_re_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
{
struct bnxt_re_mr *mr = container_of(ib_mr, struct bnxt_re_mr, ib_mr);
struct bnxt_re_dev *rdev = mr->rdev;
}
struct ib_mr *bnxt_re_alloc_mr(struct ib_pd *ib_pd, enum ib_mr_type type,
- u32 max_num_sg)
+ u32 max_num_sg, struct ib_udata *udata)
{
struct bnxt_re_pd *pd = container_of(ib_pd, struct bnxt_re_pd, ib_pd);
struct bnxt_re_dev *rdev = pd->rdev;
int page_shift)
{
u64 *pbl_tbl = pbl_tbl_orig;
- u64 paddr;
- u64 page_mask = (1ULL << page_shift) - 1;
- struct sg_dma_page_iter sg_iter;
+ u64 page_size = BIT_ULL(page_shift);
+ struct ib_block_iter biter;
+
+ rdma_for_each_block(umem->sg_head.sgl, &biter, umem->nmap, page_size)
+ *pbl_tbl++ = rdma_block_iter_dma_address(&biter);
- for_each_sg_dma_page (umem->sg_head.sgl, &sg_iter, umem->nmap, 0) {
- paddr = sg_page_iter_dma_address(&sg_iter);
- if (pbl_tbl == pbl_tbl_orig)
- *pbl_tbl++ = paddr & ~page_mask;
- else if ((paddr & page_mask) == 0)
- *pbl_tbl++ = paddr;
- }
return pbl_tbl - pbl_tbl_orig;
}
goto free_umem;
}
- page_shift = PAGE_SHIFT;
+ page_shift = __ffs(ib_umem_find_best_pgsz(umem,
+ BNXT_RE_PAGE_SIZE_4K | BNXT_RE_PAGE_SIZE_2M,
+ virt_addr));
if (!bnxt_re_page_size_ok(page_shift)) {
dev_err(rdev_to_dev(rdev), "umem page size unsupported!");
goto fail;
}
- if (!umem->hugetlb && length > BNXT_RE_MAX_MR_SIZE_LOW) {
+ if (page_shift == BNXT_RE_PAGE_SHIFT_4K &&
+ length > BNXT_RE_MAX_MR_SIZE_LOW) {
dev_err(rdev_to_dev(rdev), "Requested MR Sz:%llu Max sup:%llu",
length, (u64)BNXT_RE_MAX_MR_SIZE_LOW);
rc = -EINVAL;
goto fail;
}
- if (umem->hugetlb && length > BNXT_RE_PAGE_SIZE_2M) {
- page_shift = BNXT_RE_PAGE_SHIFT_2M;
- dev_warn(rdev_to_dev(rdev), "umem hugetlb set page_size %x",
- 1 << page_shift);
- }
/* Map umem buf ptrs to the PBL */
umem_pgs = fill_umem_pbl_tbl(umem, pbl_tbl, page_shift);
resp.chip_id0 = chip_met_rev_num;
/* Future extension of chip info */
resp.chip_id1 = 0;
- /*Temp, Use idr_alloc instead */
+ /*Temp, Use xa_alloc instead */
resp.dev_id = rdev->en_dev->pdev->devfn;
resp.max_qp = rdev->qplib_ctx.qpc_count;
resp.pg_size = PAGE_SIZE;
};
struct bnxt_re_ah {
- struct bnxt_re_dev *rdev;
struct ib_ah ib_ah;
+ struct bnxt_re_dev *rdev;
struct bnxt_qplib_ah qplib_ah;
};
struct bnxt_re_srq {
+ struct ib_srq ib_srq;
struct bnxt_re_dev *rdev;
u32 srq_limit;
- struct ib_srq ib_srq;
struct bnxt_qplib_srq qplib_srq;
struct ib_umem *umem;
spinlock_t lock; /* protect srq */
spinlock_t sh_lock; /* protect shpg */
};
-struct net_device *bnxt_re_get_netdev(struct ib_device *ibdev, u8 port_num);
-
int bnxt_re_query_device(struct ib_device *ibdev,
struct ib_device_attr *ib_attr,
struct ib_udata *udata);
int index, union ib_gid *gid);
enum rdma_link_layer bnxt_re_get_link_layer(struct ib_device *ibdev,
u8 port_num);
-int bnxt_re_alloc_pd(struct ib_pd *pd, struct ib_ucontext *context,
- struct ib_udata *udata);
-void bnxt_re_dealloc_pd(struct ib_pd *pd);
-struct ib_ah *bnxt_re_create_ah(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
- u32 flags,
- struct ib_udata *udata);
+int bnxt_re_alloc_pd(struct ib_pd *pd, struct ib_udata *udata);
+void bnxt_re_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata);
+int bnxt_re_create_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr, u32 flags,
+ struct ib_udata *udata);
int bnxt_re_modify_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr);
int bnxt_re_query_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr);
-int bnxt_re_destroy_ah(struct ib_ah *ah, u32 flags);
-struct ib_srq *bnxt_re_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata);
+void bnxt_re_destroy_ah(struct ib_ah *ah, u32 flags);
+int bnxt_re_create_srq(struct ib_srq *srq,
+ struct ib_srq_init_attr *srq_init_attr,
+ struct ib_udata *udata);
int bnxt_re_modify_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr,
enum ib_srq_attr_mask srq_attr_mask,
struct ib_udata *udata);
int bnxt_re_query_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr);
-int bnxt_re_destroy_srq(struct ib_srq *srq);
+void bnxt_re_destroy_srq(struct ib_srq *srq, struct ib_udata *udata);
int bnxt_re_post_srq_recv(struct ib_srq *srq, const struct ib_recv_wr *recv_wr,
const struct ib_recv_wr **bad_recv_wr);
struct ib_qp *bnxt_re_create_qp(struct ib_pd *pd,
int qp_attr_mask, struct ib_udata *udata);
int bnxt_re_query_qp(struct ib_qp *qp, struct ib_qp_attr *qp_attr,
int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr);
-int bnxt_re_destroy_qp(struct ib_qp *qp);
+int bnxt_re_destroy_qp(struct ib_qp *qp, struct ib_udata *udata);
int bnxt_re_post_send(struct ib_qp *qp, const struct ib_send_wr *send_wr,
const struct ib_send_wr **bad_send_wr);
int bnxt_re_post_recv(struct ib_qp *qp, const struct ib_recv_wr *recv_wr,
const struct ib_recv_wr **bad_recv_wr);
struct ib_cq *bnxt_re_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata);
-int bnxt_re_destroy_cq(struct ib_cq *cq);
+int bnxt_re_destroy_cq(struct ib_cq *cq, struct ib_udata *udata);
int bnxt_re_poll_cq(struct ib_cq *cq, int num_entries, struct ib_wc *wc);
int bnxt_re_req_notify_cq(struct ib_cq *cq, enum ib_cq_notify_flags flags);
struct ib_mr *bnxt_re_get_dma_mr(struct ib_pd *pd, int mr_access_flags);
int bnxt_re_map_mr_sg(struct ib_mr *ib_mr, struct scatterlist *sg, int sg_nents,
unsigned int *sg_offset);
struct ib_mr *bnxt_re_alloc_mr(struct ib_pd *ib_pd, enum ib_mr_type mr_type,
- u32 max_num_sg);
-int bnxt_re_dereg_mr(struct ib_mr *mr);
+ u32 max_num_sg, struct ib_udata *udata);
+int bnxt_re_dereg_mr(struct ib_mr *mr, struct ib_udata *udata);
struct ib_mw *bnxt_re_alloc_mw(struct ib_pd *ib_pd, enum ib_mw_type type,
struct ib_udata *udata);
int bnxt_re_dealloc_mw(struct ib_mw *mw);
.get_dma_mr = bnxt_re_get_dma_mr,
.get_hw_stats = bnxt_re_ib_get_hw_stats,
.get_link_layer = bnxt_re_get_link_layer,
- .get_netdev = bnxt_re_get_netdev,
.get_port_immutable = bnxt_re_get_port_immutable,
.map_mr_sg = bnxt_re_map_mr_sg,
.mmap = bnxt_re_mmap,
.query_srq = bnxt_re_query_srq,
.reg_user_mr = bnxt_re_reg_user_mr,
.req_notify_cq = bnxt_re_req_notify_cq,
+ INIT_RDMA_OBJ_SIZE(ib_ah, bnxt_re_ah, ib_ah),
INIT_RDMA_OBJ_SIZE(ib_pd, bnxt_re_pd, ib_pd),
+ INIT_RDMA_OBJ_SIZE(ib_srq, bnxt_re_srq, ib_srq),
INIT_RDMA_OBJ_SIZE(ib_ucontext, bnxt_re_ucontext, ib_uctx),
};
static int bnxt_re_register_ib(struct bnxt_re_dev *rdev)
{
struct ib_device *ibdev = &rdev->ibdev;
+ int ret;
/* ib device init */
ibdev->owner = THIS_MODULE;
rdma_set_device_sysfs_group(ibdev, &bnxt_re_dev_attr_group);
ibdev->driver_id = RDMA_DRIVER_BNXT_RE;
ib_set_device_ops(ibdev, &bnxt_re_dev_ops);
+ ret = ib_device_set_netdev(&rdev->ibdev, rdev->netdev, 1);
+ if (ret)
+ return ret;
+
return ib_register_device(ibdev, "bnxt_re%d");
}
nq->hwq.max_elements > BNXT_QPLIB_NQE_MAX_CNT)
nq->hwq.max_elements = BNXT_QPLIB_NQE_MAX_CNT;
hwq_type = bnxt_qplib_get_hwq_type(nq->res);
- if (bnxt_qplib_alloc_init_hwq(nq->pdev, &nq->hwq, NULL, 0,
+ if (bnxt_qplib_alloc_init_hwq(nq->pdev, &nq->hwq, NULL,
&nq->hwq.max_elements,
BNXT_QPLIB_MAX_NQE_ENTRY_SIZE, 0,
PAGE_SIZE, hwq_type))
writeq(val, db);
}
-int bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res,
+void bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res,
struct bnxt_qplib_srq *srq)
{
struct bnxt_qplib_rcfw *rcfw = res->rcfw;
/* Configure the request */
req.srq_cid = cpu_to_le32(srq->id);
- rc = bnxt_qplib_rcfw_send_message(rcfw, (void *)&req,
- (void *)&resp, NULL, 0);
+ rc = bnxt_qplib_rcfw_send_message(rcfw, (struct cmdq_base *)&req,
+ (struct creq_base *)&resp, NULL, 0);
+ kfree(srq->swq);
if (rc)
- return rc;
-
+ return;
bnxt_qplib_free_hwq(res->pdev, &srq->hwq);
- kfree(srq->swq);
- return 0;
}
int bnxt_qplib_create_srq(struct bnxt_qplib_res *res,
int rc, idx;
srq->hwq.max_elements = srq->max_wqe;
- rc = bnxt_qplib_alloc_init_hwq(res->pdev, &srq->hwq, srq->sglist,
- srq->nmap, &srq->hwq.max_elements,
+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, &srq->hwq, &srq->sg_info,
+ &srq->hwq.max_elements,
BNXT_QPLIB_MAX_RQE_ENTRY_SIZE, 0,
PAGE_SIZE, HWQ_TYPE_QUEUE);
if (rc)
/* SQ */
sq->hwq.max_elements = sq->max_wqe;
- rc = bnxt_qplib_alloc_init_hwq(res->pdev, &sq->hwq, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, &sq->hwq, NULL,
&sq->hwq.max_elements,
BNXT_QPLIB_MAX_SQE_ENTRY_SIZE, 0,
PAGE_SIZE, HWQ_TYPE_QUEUE);
/* RQ */
if (rq->max_wqe) {
rq->hwq.max_elements = qp->rq.max_wqe;
- rc = bnxt_qplib_alloc_init_hwq(res->pdev, &rq->hwq, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, &rq->hwq, NULL,
&rq->hwq.max_elements,
BNXT_QPLIB_MAX_RQE_ENTRY_SIZE, 0,
PAGE_SIZE, HWQ_TYPE_QUEUE);
sizeof(struct sq_psn_search);
}
sq->hwq.max_elements = sq->max_wqe;
- rc = bnxt_qplib_alloc_init_hwq(res->pdev, &sq->hwq, sq->sglist,
- sq->nmap, &sq->hwq.max_elements,
+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, &sq->hwq, &sq->sg_info,
+ &sq->hwq.max_elements,
BNXT_QPLIB_MAX_SQE_ENTRY_SIZE,
psn_sz,
PAGE_SIZE, HWQ_TYPE_QUEUE);
/* RQ */
if (rq->max_wqe) {
rq->hwq.max_elements = rq->max_wqe;
- rc = bnxt_qplib_alloc_init_hwq(res->pdev, &rq->hwq, rq->sglist,
- rq->nmap, &rq->hwq.max_elements,
+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, &rq->hwq,
+ &rq->sg_info,
+ &rq->hwq.max_elements,
BNXT_QPLIB_MAX_RQE_ENTRY_SIZE, 0,
PAGE_SIZE, HWQ_TYPE_QUEUE);
if (rc)
req_size = xrrq->max_elements *
BNXT_QPLIB_MAX_ORRQE_ENTRY_SIZE + PAGE_SIZE - 1;
req_size &= ~(PAGE_SIZE - 1);
- rc = bnxt_qplib_alloc_init_hwq(res->pdev, xrrq, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, xrrq, NULL,
&xrrq->max_elements,
BNXT_QPLIB_MAX_ORRQE_ENTRY_SIZE,
0, req_size, HWQ_TYPE_CTX);
BNXT_QPLIB_MAX_IRRQE_ENTRY_SIZE + PAGE_SIZE - 1;
req_size &= ~(PAGE_SIZE - 1);
- rc = bnxt_qplib_alloc_init_hwq(res->pdev, xrrq, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, xrrq, NULL,
&xrrq->max_elements,
BNXT_QPLIB_MAX_IRRQE_ENTRY_SIZE,
0, req_size, HWQ_TYPE_CTX);
int rc;
cq->hwq.max_elements = cq->max_wqe;
- rc = bnxt_qplib_alloc_init_hwq(res->pdev, &cq->hwq, cq->sghead,
- cq->nmap, &cq->hwq.max_elements,
+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, &cq->hwq, &cq->sg_info,
+ &cq->hwq.max_elements,
BNXT_QPLIB_MAX_CQE_ENTRY_SIZE, 0,
PAGE_SIZE, HWQ_TYPE_QUEUE);
if (rc)
struct bnxt_qplib_cq *cq;
struct bnxt_qplib_hwq hwq;
struct bnxt_qplib_swq *swq;
- struct scatterlist *sglist;
int start_idx;
int last_idx;
- u32 nmap;
+ struct bnxt_qplib_sg_info sg_info;
u16 eventq_hw_ring_id;
spinlock_t lock; /* protect SRQE link list */
};
struct bnxt_qplib_q {
struct bnxt_qplib_hwq hwq;
struct bnxt_qplib_swq *swq;
- struct scatterlist *sglist;
- u32 nmap;
+ struct bnxt_qplib_sg_info sg_info;
u32 max_wqe;
u16 q_full_delta;
u16 max_sge;
u32 cnq_hw_ring_id;
struct bnxt_qplib_nq *nq;
bool resize_in_progress;
- struct scatterlist *sghead;
- u32 nmap;
+ struct bnxt_qplib_sg_info sg_info;
u64 cq_handle;
#define CQ_RESIZE_WAIT_TIME_MS 500
struct bnxt_qplib_srq *srq);
int bnxt_qplib_query_srq(struct bnxt_qplib_res *res,
struct bnxt_qplib_srq *srq);
-int bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res,
- struct bnxt_qplib_srq *srq);
+void bnxt_qplib_destroy_srq(struct bnxt_qplib_res *res,
+ struct bnxt_qplib_srq *srq);
int bnxt_qplib_post_srq_recv(struct bnxt_qplib_srq *srq,
struct bnxt_qplib_swqe *wqe);
int bnxt_qplib_create_qp1(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp);
rcfw->pdev = pdev;
rcfw->creq.max_elements = BNXT_QPLIB_CREQE_MAX_CNT;
hwq_type = bnxt_qplib_get_hwq_type(rcfw->res);
- if (bnxt_qplib_alloc_init_hwq(rcfw->pdev, &rcfw->creq, NULL, 0,
+ if (bnxt_qplib_alloc_init_hwq(rcfw->pdev, &rcfw->creq, NULL,
&rcfw->creq.max_elements,
BNXT_QPLIB_CREQE_UNITS,
0, PAGE_SIZE, hwq_type)) {
rcfw->cmdq.max_elements = rcfw->cmdq_depth;
if (bnxt_qplib_alloc_init_hwq
- (rcfw->pdev, &rcfw->cmdq, NULL, 0,
+ (rcfw->pdev, &rcfw->cmdq, NULL,
&rcfw->cmdq.max_elements,
BNXT_QPLIB_CMDQE_UNITS, 0,
bnxt_qplib_cmdqe_page_size(rcfw->cmdq_depth),
}
static int __alloc_pbl(struct pci_dev *pdev, struct bnxt_qplib_pbl *pbl,
- struct scatterlist *sghead, u32 pages, u32 pg_size)
+ struct scatterlist *sghead, u32 pages,
+ u32 nmaps, u32 pg_size)
{
struct sg_dma_page_iter sg_iter;
bool is_umem = false;
} else {
i = 0;
is_umem = true;
- for_each_sg_dma_page (sghead, &sg_iter, pages, 0) {
+ for_each_sg_dma_page(sghead, &sg_iter, nmaps, 0) {
pbl->pg_map_arr[i] = sg_page_iter_dma_address(&sg_iter);
pbl->pg_arr[i] = NULL;
pbl->pg_count++;
/* All HWQs are power of 2 in size */
int bnxt_qplib_alloc_init_hwq(struct pci_dev *pdev, struct bnxt_qplib_hwq *hwq,
- struct scatterlist *sghead, int nmap,
+ struct bnxt_qplib_sg_info *sg_info,
u32 *elements, u32 element_size, u32 aux,
u32 pg_size, enum bnxt_qplib_hwq_type hwq_type)
{
- u32 pages, slots, size, aux_pages = 0, aux_size = 0;
+ u32 pages, maps, slots, size, aux_pages = 0, aux_size = 0;
dma_addr_t *src_phys_ptr, **dst_virt_ptr;
+ struct scatterlist *sghead = NULL;
int i, rc;
hwq->level = PBL_LVL_MAX;
}
size = roundup_pow_of_two(element_size);
+ if (sg_info)
+ sghead = sg_info->sglist;
+
if (!sghead) {
hwq->is_user = false;
pages = (slots * size) / pg_size + aux_pages;
pages++;
if (!pages)
return -EINVAL;
+ maps = 0;
} else {
hwq->is_user = true;
- pages = nmap;
+ pages = sg_info->npages;
+ maps = sg_info->nmap;
}
/* Alloc the 1st memory block; can be a PDL/PTL/PBL */
if (sghead && (pages == MAX_PBL_LVL_0_PGS))
rc = __alloc_pbl(pdev, &hwq->pbl[PBL_LVL_0], sghead,
- pages, pg_size);
+ pages, maps, pg_size);
else
- rc = __alloc_pbl(pdev, &hwq->pbl[PBL_LVL_0], NULL, 1, pg_size);
+ rc = __alloc_pbl(pdev, &hwq->pbl[PBL_LVL_0], NULL,
+ 1, 0, pg_size);
if (rc)
goto fail;
if (pages > MAX_PBL_LVL_1_PGS) {
/* 2 levels of indirection */
rc = __alloc_pbl(pdev, &hwq->pbl[PBL_LVL_1], NULL,
- MAX_PBL_LVL_1_PGS_FOR_LVL_2, pg_size);
+ MAX_PBL_LVL_1_PGS_FOR_LVL_2,
+ 0, pg_size);
if (rc)
goto fail;
/* Fill in lvl0 PBL */
hwq->level = PBL_LVL_1;
rc = __alloc_pbl(pdev, &hwq->pbl[PBL_LVL_2], sghead,
- pages, pg_size);
+ pages, maps, pg_size);
if (rc)
goto fail;
/* 1 level of indirection */
rc = __alloc_pbl(pdev, &hwq->pbl[PBL_LVL_1], sghead,
- pages, pg_size);
+ pages, maps, pg_size);
if (rc)
goto fail;
/* Fill in lvl0 PBL */
/* QPC Tables */
ctx->qpc_tbl.max_elements = ctx->qpc_count;
- rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->qpc_tbl, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->qpc_tbl, NULL,
&ctx->qpc_tbl.max_elements,
BNXT_QPLIB_MAX_QP_CTX_ENTRY_SIZE, 0,
PAGE_SIZE, HWQ_TYPE_CTX);
/* MRW Tables */
ctx->mrw_tbl.max_elements = ctx->mrw_count;
- rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->mrw_tbl, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->mrw_tbl, NULL,
&ctx->mrw_tbl.max_elements,
BNXT_QPLIB_MAX_MRW_CTX_ENTRY_SIZE, 0,
PAGE_SIZE, HWQ_TYPE_CTX);
/* SRQ Tables */
ctx->srqc_tbl.max_elements = ctx->srqc_count;
- rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->srqc_tbl, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->srqc_tbl, NULL,
&ctx->srqc_tbl.max_elements,
BNXT_QPLIB_MAX_SRQ_CTX_ENTRY_SIZE, 0,
PAGE_SIZE, HWQ_TYPE_CTX);
/* CQ Tables */
ctx->cq_tbl.max_elements = ctx->cq_count;
- rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->cq_tbl, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->cq_tbl, NULL,
&ctx->cq_tbl.max_elements,
BNXT_QPLIB_MAX_CQ_CTX_ENTRY_SIZE, 0,
PAGE_SIZE, HWQ_TYPE_CTX);
/* TQM Buffer */
ctx->tqm_pde.max_elements = 512;
- rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->tqm_pde, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->tqm_pde, NULL,
&ctx->tqm_pde.max_elements, sizeof(u64),
0, PAGE_SIZE, HWQ_TYPE_CTX);
if (rc)
continue;
ctx->tqm_tbl[i].max_elements = ctx->qpc_count *
ctx->tqm_count[i];
- rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->tqm_tbl[i], NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->tqm_tbl[i], NULL,
&ctx->tqm_tbl[i].max_elements, 1,
0, PAGE_SIZE, HWQ_TYPE_CTX);
if (rc)
/* TIM Buffer */
ctx->tim_tbl.max_elements = ctx->qpc_count * 16;
- rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->tim_tbl, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(pdev, &ctx->tim_tbl, NULL,
&ctx->tim_tbl.max_elements, 1,
0, PAGE_SIZE, HWQ_TYPE_CTX);
if (rc)
RING_ALLOC_REQ_RING_TYPE_ROCE_CMPL;
}
+struct bnxt_qplib_sg_info {
+ struct scatterlist *sglist;
+ u32 nmap;
+ u32 npages;
+};
+
#define to_bnxt_qplib(ptr, type, member) \
container_of(ptr, type, member)
void bnxt_qplib_free_hwq(struct pci_dev *pdev, struct bnxt_qplib_hwq *hwq);
int bnxt_qplib_alloc_init_hwq(struct pci_dev *pdev, struct bnxt_qplib_hwq *hwq,
- struct scatterlist *sl, int nmap, u32 *elements,
+ struct bnxt_qplib_sg_info *sg_info, u32 *elements,
u32 elements_per_page, u32 aux, u32 pg_size,
enum bnxt_qplib_hwq_type hwq_type);
void bnxt_qplib_get_guid(u8 *dev_addr, u8 *guid);
return 0;
}
-int bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah,
- bool block)
+void bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah,
+ bool block)
{
struct bnxt_qplib_rcfw *rcfw = res->rcfw;
struct cmdq_destroy_ah req;
struct creq_destroy_ah_resp resp;
u16 cmd_flags = 0;
- int rc;
/* Clean up the AH table in the device */
RCFW_CMD_PREP(req, DESTROY_AH, cmd_flags);
req.ah_cid = cpu_to_le32(ah->id);
- rc = bnxt_qplib_rcfw_send_message(rcfw, (void *)&req, (void *)&resp,
- NULL, block);
- if (rc)
- return rc;
- return 0;
+ bnxt_qplib_rcfw_send_message(rcfw, (void *)&req, (void *)&resp, NULL,
+ block);
}
/* MRW */
mr->hwq.max_elements = pages;
/* Use system PAGE_SIZE */
- rc = bnxt_qplib_alloc_init_hwq(res->pdev, &mr->hwq, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, &mr->hwq, NULL,
&mr->hwq.max_elements,
PAGE_SIZE, 0, PAGE_SIZE,
HWQ_TYPE_CTX);
return -ENOMEM;
frpl->hwq.max_elements = pages;
- rc = bnxt_qplib_alloc_init_hwq(res->pdev, &frpl->hwq, NULL, 0,
+ rc = bnxt_qplib_alloc_init_hwq(res->pdev, &frpl->hwq, NULL,
&frpl->hwq.max_elements, PAGE_SIZE, 0,
PAGE_SIZE, HWQ_TYPE_CTX);
if (!rc)
struct bnxt_qplib_ctx *ctx);
int bnxt_qplib_create_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah,
bool block);
-int bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah,
- bool block);
+void bnxt_qplib_destroy_ah(struct bnxt_qplib_res *res, struct bnxt_qplib_ah *ah,
+ bool block);
int bnxt_qplib_alloc_mrw(struct bnxt_qplib_res *res,
struct bnxt_qplib_mrw *mrw);
int bnxt_qplib_dereg_mrw(struct bnxt_qplib_res *res, struct bnxt_qplib_mrw *mrw,
T3_SOLICITED_EVENT_FLAG = 0x04,
T3_READ_FENCE_FLAG = 0x08,
T3_LOCAL_FENCE_FLAG = 0x10
-} __attribute__ ((packed));
+} __packed;
enum t3_wr_opcode {
T3_WR_BP = FW_WROPCODE_RI_BYPASS,
T3_WR_INIT = FW_WROPCODE_RI_RDMA_INIT,
T3_WR_QP_MOD = FW_WROPCODE_RI_MODIFY_QP,
T3_WR_FASTREG = FW_WROPCODE_RI_FASTREGISTER_MR
-} __attribute__ ((packed));
+} __packed;
enum t3_rdma_opcode {
T3_RDMA_WRITE, /* IETF RDMAP v1.0 ... */
T3_QP_MOD,
T3_BYPASS,
T3_RDMA_READ_REQ_WITH_INV,
-} __attribute__ ((packed));
+} __packed;
static inline enum t3_rdma_opcode wr2opcode(enum t3_wr_opcode wrop)
{
uP_RI_MPA_TX_MARKER_ENABLE = 0x2,
uP_RI_MPA_CRC_ENABLE = 0x4,
uP_RI_MPA_IETF_ENABLE = 0x8
-} __attribute__ ((packed));
+} __packed;
enum t3_qp_caps {
uP_RI_QP_RDMA_READ_ENABLE = 0x01,
uP_RI_QP_BIND_ENABLE = 0x04,
uP_RI_QP_FAST_REGISTER_ENABLE = 0x08,
uP_RI_QP_STAG0_ENABLE = 0x10
-} __attribute__ ((packed));
+} __packed;
enum rdma_init_rtr_types {
RTR_READ = 1,
static LIST_HEAD(dev_list);
static DEFINE_MUTEX(dev_mutex);
-static int disable_qp_db(int id, void *p, void *data)
-{
- struct iwch_qp *qhp = p;
-
- cxio_disable_wq_db(&qhp->wq);
- return 0;
-}
-
-static int enable_qp_db(int id, void *p, void *data)
-{
- struct iwch_qp *qhp = p;
-
- if (data)
- ring_doorbell(qhp->rhp->rdev.ctrl_qp.doorbell, qhp->wq.qpid);
- cxio_enable_wq_db(&qhp->wq);
- return 0;
-}
-
static void disable_dbs(struct iwch_dev *rnicp)
{
- spin_lock_irq(&rnicp->lock);
- idr_for_each(&rnicp->qpidr, disable_qp_db, NULL);
- spin_unlock_irq(&rnicp->lock);
+ unsigned long index;
+ struct iwch_qp *qhp;
+
+ xa_lock_irq(&rnicp->qps);
+ xa_for_each(&rnicp->qps, index, qhp)
+ cxio_disable_wq_db(&qhp->wq);
+ xa_unlock_irq(&rnicp->qps);
}
static void enable_dbs(struct iwch_dev *rnicp, int ring_db)
{
- spin_lock_irq(&rnicp->lock);
- idr_for_each(&rnicp->qpidr, enable_qp_db,
- (void *)(unsigned long)ring_db);
- spin_unlock_irq(&rnicp->lock);
+ unsigned long index;
+ struct iwch_qp *qhp;
+
+ xa_lock_irq(&rnicp->qps);
+ xa_for_each(&rnicp->qps, index, qhp) {
+ if (ring_db)
+ ring_doorbell(qhp->rhp->rdev.ctrl_qp.doorbell,
+ qhp->wq.qpid);
+ cxio_enable_wq_db(&qhp->wq);
+ }
+ xa_unlock_irq(&rnicp->qps);
}
static void iwch_db_drop_task(struct work_struct *work)
static void rnic_init(struct iwch_dev *rnicp)
{
pr_debug("%s iwch_dev %p\n", __func__, rnicp);
- idr_init(&rnicp->cqidr);
- idr_init(&rnicp->qpidr);
- idr_init(&rnicp->mmidr);
- spin_lock_init(&rnicp->lock);
+ xa_init_flags(&rnicp->cqs, XA_FLAGS_LOCK_IRQ);
+ xa_init_flags(&rnicp->qps, XA_FLAGS_LOCK_IRQ);
+ xa_init_flags(&rnicp->mrs, XA_FLAGS_LOCK_IRQ);
INIT_DELAYED_WORK(&rnicp->db_drop_task, iwch_db_drop_task);
rnicp->attr.max_qps = T3_MAX_NUM_QP - 32;
list_del(&dev->entry);
iwch_unregister_device(dev);
cxio_rdev_close(&dev->rdev);
- idr_destroy(&dev->cqidr);
- idr_destroy(&dev->qpidr);
- idr_destroy(&dev->mmidr);
+ WARN_ON(!xa_empty(&dev->cqs));
+ WARN_ON(!xa_empty(&dev->qps));
+ WARN_ON(!xa_empty(&dev->mrs));
ib_dealloc_device(&dev->ibdev);
break;
}
#include <linux/mutex.h>
#include <linux/list.h>
#include <linux/spinlock.h>
-#include <linux/idr.h>
+#include <linux/xarray.h>
#include <linux/workqueue.h>
#include <rdma/ib_verbs.h>
struct cxio_rdev rdev;
u32 device_cap_flags;
struct iwch_rnic_attributes attr;
- struct idr cqidr;
- struct idr qpidr;
- struct idr mmidr;
- spinlock_t lock;
+ struct xarray cqs;
+ struct xarray qps;
+ struct xarray mrs;
struct list_head entry;
struct delayed_work db_drop_task;
};
static inline struct iwch_cq *get_chp(struct iwch_dev *rhp, u32 cqid)
{
- return idr_find(&rhp->cqidr, cqid);
+ return xa_load(&rhp->cqs, cqid);
}
static inline struct iwch_qp *get_qhp(struct iwch_dev *rhp, u32 qpid)
{
- return idr_find(&rhp->qpidr, qpid);
+ return xa_load(&rhp->qps, qpid);
}
static inline struct iwch_mr *get_mhp(struct iwch_dev *rhp, u32 mmid)
{
- return idr_find(&rhp->mmidr, mmid);
-}
-
-static inline int insert_handle(struct iwch_dev *rhp, struct idr *idr,
- void *handle, u32 id)
-{
- int ret;
-
- idr_preload(GFP_KERNEL);
- spin_lock_irq(&rhp->lock);
-
- ret = idr_alloc(idr, handle, id, id + 1, GFP_NOWAIT);
-
- spin_unlock_irq(&rhp->lock);
- idr_preload_end();
-
- return ret < 0 ? ret : 0;
-}
-
-static inline void remove_handle(struct iwch_dev *rhp, struct idr *idr, u32 id)
-{
- spin_lock_irq(&rhp->lock);
- idr_remove(idr, id);
- spin_unlock_irq(&rhp->lock);
+ return xa_load(&rhp->mrs, mmid);
}
extern struct cxgb3_client t3c_client;
struct iwch_qp *qhp;
unsigned long flag;
- spin_lock(&rnicp->lock);
- qhp = get_qhp(rnicp, CQE_QPID(rsp_msg->cqe));
+ xa_lock(&rnicp->qps);
+ qhp = xa_load(&rnicp->qps, CQE_QPID(rsp_msg->cqe));
if (!qhp) {
pr_err("%s unaffiliated error 0x%x qpid 0x%x\n",
__func__, CQE_STATUS(rsp_msg->cqe),
CQE_QPID(rsp_msg->cqe));
- spin_unlock(&rnicp->lock);
+ xa_unlock(&rnicp->qps);
return;
}
__func__,
qhp->attr.state, qhp->wq.qpid,
CQE_STATUS(rsp_msg->cqe));
- spin_unlock(&rnicp->lock);
+ xa_unlock(&rnicp->qps);
return;
}
CQE_WRID_HI(rsp_msg->cqe), CQE_WRID_LOW(rsp_msg->cqe));
atomic_inc(&qhp->refcnt);
- spin_unlock(&rnicp->lock);
+ xa_unlock(&rnicp->qps);
if (qhp->attr.state == IWCH_QP_STATE_RTS) {
attrs.next_state = IWCH_QP_STATE_TERMINATE;
unsigned long flag;
rnicp = (struct iwch_dev *) rdev_p->ulp;
- spin_lock(&rnicp->lock);
+ xa_lock(&rnicp->qps);
chp = get_chp(rnicp, cqid);
- qhp = get_qhp(rnicp, CQE_QPID(rsp_msg->cqe));
+ qhp = xa_load(&rnicp->qps, CQE_QPID(rsp_msg->cqe));
if (!chp || !qhp) {
pr_err("BAD AE cqid 0x%x qpid 0x%x opcode %d status 0x%x type %d wrid.hi 0x%x wrid.lo 0x%x\n",
cqid, CQE_QPID(rsp_msg->cqe),
CQE_OPCODE(rsp_msg->cqe), CQE_STATUS(rsp_msg->cqe),
CQE_TYPE(rsp_msg->cqe), CQE_WRID_HI(rsp_msg->cqe),
CQE_WRID_LOW(rsp_msg->cqe));
- spin_unlock(&rnicp->lock);
+ xa_unlock(&rnicp->qps);
goto out;
}
iwch_qp_add_ref(&qhp->ibqp);
atomic_inc(&chp->refcnt);
- spin_unlock(&rnicp->lock);
+ xa_unlock(&rnicp->qps);
/*
* 1) completion of our sending a TERMINATE.
mmid = stag >> 8;
mhp->ibmr.rkey = mhp->ibmr.lkey = stag;
pr_debug("%s mmid 0x%x mhp %p\n", __func__, mmid, mhp);
- return insert_handle(mhp->rhp, &mhp->rhp->mmidr, mhp, mmid);
+ return xa_insert_irq(&mhp->rhp->mrs, mmid, mhp, GFP_KERNEL);
}
int iwch_register_mem(struct iwch_dev *rhp, struct iwch_pd *php,
return 0;
}
-static int iwch_destroy_cq(struct ib_cq *ib_cq)
+static int iwch_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
{
struct iwch_cq *chp;
pr_debug("%s ib_cq %p\n", __func__, ib_cq);
chp = to_iwch_cq(ib_cq);
- remove_handle(chp->rhp, &chp->rhp->cqidr, chp->cq.cqid);
+ xa_erase_irq(&chp->rhp->cqs, chp->cq.cqid);
atomic_dec(&chp->refcnt);
wait_event(chp->wait, !atomic_read(&chp->refcnt));
static struct ib_cq *iwch_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *ib_context,
struct ib_udata *udata)
{
int entries = attr->cqe;
struct iwch_cq *chp;
struct iwch_create_cq_resp uresp;
struct iwch_create_cq_req ureq;
- struct iwch_ucontext *ucontext = NULL;
static int warned;
size_t resplen;
if (!chp)
return ERR_PTR(-ENOMEM);
- if (ib_context) {
- ucontext = to_iwch_ucontext(ib_context);
+ if (udata) {
if (!t3a_device(rhp)) {
if (ib_copy_from_udata(&ureq, udata, sizeof (ureq))) {
kfree(chp);
entries = roundup_pow_of_two(entries);
chp->cq.size_log2 = ilog2(entries);
- if (cxio_create_cq(&rhp->rdev, &chp->cq, !ucontext)) {
+ if (cxio_create_cq(&rhp->rdev, &chp->cq, !udata)) {
kfree(chp);
return ERR_PTR(-ENOMEM);
}
spin_lock_init(&chp->comp_handler_lock);
atomic_set(&chp->refcnt, 1);
init_waitqueue_head(&chp->wait);
- if (insert_handle(rhp, &rhp->cqidr, chp, chp->cq.cqid)) {
+ if (xa_store_irq(&rhp->cqs, chp->cq.cqid, chp, GFP_KERNEL)) {
cxio_destroy_cq(&chp->rhp->rdev, &chp->cq);
kfree(chp);
return ERR_PTR(-ENOMEM);
}
- if (ucontext) {
+ if (udata) {
struct iwch_mm_entry *mm;
+ struct iwch_ucontext *ucontext = rdma_udata_to_drv_context(
+ udata, struct iwch_ucontext, ibucontext);
mm = kmalloc(sizeof *mm, GFP_KERNEL);
if (!mm) {
- iwch_destroy_cq(&chp->ibcq);
+ iwch_destroy_cq(&chp->ibcq, udata);
return ERR_PTR(-ENOMEM);
}
uresp.cqid = chp->cq.cqid;
}
if (ib_copy_to_udata(udata, &uresp, resplen)) {
kfree(mm);
- iwch_destroy_cq(&chp->ibcq);
+ iwch_destroy_cq(&chp->ibcq, udata);
return ERR_PTR(-EFAULT);
}
insert_mmap(ucontext, mm);
return ret;
}
-static void iwch_deallocate_pd(struct ib_pd *pd)
+static void iwch_deallocate_pd(struct ib_pd *pd, struct ib_udata *udata)
{
struct iwch_dev *rhp;
struct iwch_pd *php;
cxio_hal_put_pdid(rhp->rdev.rscp, php->pdid);
}
-static int iwch_allocate_pd(struct ib_pd *pd, struct ib_ucontext *context,
- struct ib_udata *udata)
+static int iwch_allocate_pd(struct ib_pd *pd, struct ib_udata *udata)
{
struct iwch_pd *php = to_iwch_pd(pd);
struct ib_device *ibdev = pd->device;
php->pdid = pdid;
php->rhp = rhp;
- if (context) {
+ if (udata) {
struct iwch_alloc_pd_resp resp = {.pdid = php->pdid};
if (ib_copy_to_udata(udata, &resp, sizeof(resp))) {
- iwch_deallocate_pd(&php->ibpd);
+ iwch_deallocate_pd(&php->ibpd, udata);
return -EFAULT;
}
}
return 0;
}
-static int iwch_dereg_mr(struct ib_mr *ib_mr)
+static int iwch_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
{
struct iwch_dev *rhp;
struct iwch_mr *mhp;
cxio_dereg_mem(&rhp->rdev, mhp->attr.stag, mhp->attr.pbl_size,
mhp->attr.pbl_addr);
iwch_free_pbl(mhp);
- remove_handle(rhp, &rhp->mmidr, mmid);
+ xa_erase_irq(&rhp->mrs, mmid);
if (mhp->kva)
kfree((void *) (unsigned long) mhp->kva);
if (mhp->umem)
shift = PAGE_SHIFT;
- n = mhp->umem->nmap;
+ n = ib_umem_num_pages(mhp->umem);
err = iwch_alloc_pbl(mhp, n);
if (err)
uresp.pbl_addr);
if (ib_copy_to_udata(udata, &uresp, sizeof (uresp))) {
- iwch_dereg_mr(&mhp->ibmr);
+ iwch_dereg_mr(&mhp->ibmr, udata);
err = -EFAULT;
goto err;
}
mhp->attr.stag = stag;
mmid = (stag) >> 8;
mhp->ibmw.rkey = stag;
- if (insert_handle(rhp, &rhp->mmidr, mhp, mmid)) {
+ if (xa_insert_irq(&rhp->mrs, mmid, mhp, GFP_KERNEL)) {
cxio_deallocate_window(&rhp->rdev, mhp->attr.stag);
kfree(mhp);
return ERR_PTR(-ENOMEM);
rhp = mhp->rhp;
mmid = (mw->rkey) >> 8;
cxio_deallocate_window(&rhp->rdev, mhp->attr.stag);
- remove_handle(rhp, &rhp->mmidr, mmid);
+ xa_erase_irq(&rhp->mrs, mmid);
pr_debug("%s ib_mw %p mmid 0x%x ptr %p\n", __func__, mw, mmid, mhp);
kfree(mhp);
return 0;
}
-static struct ib_mr *iwch_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg)
+static struct ib_mr *iwch_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
struct iwch_dev *rhp;
struct iwch_pd *php;
mhp->attr.state = 1;
mmid = (stag) >> 8;
mhp->ibmr.rkey = mhp->ibmr.lkey = stag;
- ret = insert_handle(rhp, &rhp->mmidr, mhp, mmid);
+ ret = xa_insert_irq(&rhp->mrs, mmid, mhp, GFP_KERNEL);
if (ret)
goto err3;
return ib_sg_to_pages(ibmr, sg, sg_nents, sg_offset, iwch_set_page);
}
-static int iwch_destroy_qp(struct ib_qp *ib_qp)
+static int iwch_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
{
struct iwch_dev *rhp;
struct iwch_qp *qhp;
iwch_modify_qp(rhp, qhp, IWCH_QP_ATTR_NEXT_STATE, &attrs, 0);
wait_event(qhp->wait, !qhp->ep);
- remove_handle(rhp, &rhp->qpidr, qhp->wq.qpid);
+ xa_erase_irq(&rhp->qps, qhp->wq.qpid);
atomic_dec(&qhp->refcnt);
wait_event(qhp->wait, !atomic_read(&qhp->refcnt));
- ucontext = ib_qp->uobject ? to_iwch_ucontext(ib_qp->uobject->context)
- : NULL;
+ ucontext = rdma_udata_to_drv_context(udata, struct iwch_ucontext,
+ ibucontext);
cxio_destroy_qp(&rhp->rdev, &qhp->wq,
ucontext ? &ucontext->uctx : &rhp->rdev.uctx);
init_waitqueue_head(&qhp->wait);
atomic_set(&qhp->refcnt, 1);
- if (insert_handle(rhp, &rhp->qpidr, qhp, qhp->wq.qpid)) {
+ if (xa_store_irq(&rhp->qps, qhp->wq.qpid, qhp, GFP_KERNEL)) {
cxio_destroy_qp(&rhp->rdev, &qhp->wq,
ucontext ? &ucontext->uctx : &rhp->rdev.uctx);
kfree(qhp);
mm1 = kmalloc(sizeof *mm1, GFP_KERNEL);
if (!mm1) {
- iwch_destroy_qp(&qhp->ibqp);
+ iwch_destroy_qp(&qhp->ibqp, udata);
return ERR_PTR(-ENOMEM);
}
mm2 = kmalloc(sizeof *mm2, GFP_KERNEL);
if (!mm2) {
kfree(mm1);
- iwch_destroy_qp(&qhp->ibqp);
+ iwch_destroy_qp(&qhp->ibqp, udata);
return ERR_PTR(-ENOMEM);
}
if (ib_copy_to_udata(udata, &uresp, sizeof (uresp))) {
kfree(mm1);
kfree(mm2);
- iwch_destroy_qp(&qhp->ibqp);
+ iwch_destroy_qp(&qhp->ibqp, udata);
return ERR_PTR(-EFAULT);
}
mm1->key = uresp.key;
.get_dma_mr = iwch_get_dma_mr,
.get_hw_stats = iwch_get_mib,
.get_port_immutable = iwch_port_immutable,
+ .iw_accept = iwch_accept_cr,
+ .iw_add_ref = iwch_qp_add_ref,
+ .iw_connect = iwch_connect,
+ .iw_create_listen = iwch_create_listen,
+ .iw_destroy_listen = iwch_destroy_listen,
+ .iw_get_qp = iwch_get_qp,
+ .iw_reject = iwch_reject_cr,
+ .iw_rem_ref = iwch_qp_rem_ref,
.map_mr_sg = iwch_map_mr_sg,
.mmap = iwch_mmap,
.modify_qp = iwch_ib_modify_qp,
int iwch_register_device(struct iwch_dev *dev)
{
- int ret;
-
pr_debug("%s iwch_dev %p\n", __func__, dev);
memset(&dev->ibdev.node_guid, 0, sizeof(dev->ibdev.node_guid));
memcpy(&dev->ibdev.node_guid, dev->rdev.t3cdev_p->lldev->dev_addr, 6);
dev->ibdev.dev.parent = &dev->rdev.rnic_info.pdev->dev;
dev->ibdev.uverbs_abi_ver = IWCH_UVERBS_ABI_VERSION;
- dev->ibdev.iwcm = kzalloc(sizeof(struct iw_cm_verbs), GFP_KERNEL);
- if (!dev->ibdev.iwcm)
- return -ENOMEM;
-
- dev->ibdev.iwcm->connect = iwch_connect;
- dev->ibdev.iwcm->accept = iwch_accept_cr;
- dev->ibdev.iwcm->reject = iwch_reject_cr;
- dev->ibdev.iwcm->create_listen = iwch_create_listen;
- dev->ibdev.iwcm->destroy_listen = iwch_destroy_listen;
- dev->ibdev.iwcm->add_ref = iwch_qp_add_ref;
- dev->ibdev.iwcm->rem_ref = iwch_qp_rem_ref;
- dev->ibdev.iwcm->get_qp = iwch_get_qp;
- memcpy(dev->ibdev.iwcm->ifname, dev->rdev.t3cdev_p->lldev->name,
- sizeof(dev->ibdev.iwcm->ifname));
+ memcpy(dev->ibdev.iw_ifname, dev->rdev.t3cdev_p->lldev->name,
+ sizeof(dev->ibdev.iw_ifname));
dev->ibdev.driver_id = RDMA_DRIVER_CXGB3;
rdma_set_device_sysfs_group(&dev->ibdev, &iwch_attr_group);
ib_set_device_ops(&dev->ibdev, &iwch_dev_ops);
- ret = ib_register_device(&dev->ibdev, "cxgb3_%d");
- if (ret)
- kfree(dev->ibdev.iwcm);
- return ret;
+ return ib_register_device(&dev->ibdev, "cxgb3_%d");
}
void iwch_unregister_device(struct iwch_dev *dev)
{
pr_debug("%s iwch_dev %p\n", __func__, dev);
ib_unregister_device(&dev->ibdev);
- kfree(dev->ibdev.iwcm);
return;
}
{
unsigned long flags;
- spin_lock_irqsave(&ep->com.dev->lock, flags);
- _remove_handle(ep->com.dev, &ep->com.dev->hwtid_idr, ep->hwtid, 0);
- if (idr_is_empty(&ep->com.dev->hwtid_idr))
+ xa_lock_irqsave(&ep->com.dev->hwtids, flags);
+ __xa_erase(&ep->com.dev->hwtids, ep->hwtid);
+ if (xa_empty(&ep->com.dev->hwtids))
wake_up(&ep->com.dev->wait);
- spin_unlock_irqrestore(&ep->com.dev->lock, flags);
+ xa_unlock_irqrestore(&ep->com.dev->hwtids, flags);
}
-static void insert_ep_tid(struct c4iw_ep *ep)
+static int insert_ep_tid(struct c4iw_ep *ep)
{
unsigned long flags;
+ int err;
+
+ xa_lock_irqsave(&ep->com.dev->hwtids, flags);
+ err = __xa_insert(&ep->com.dev->hwtids, ep->hwtid, ep, GFP_KERNEL);
+ xa_unlock_irqrestore(&ep->com.dev->hwtids, flags);
- spin_lock_irqsave(&ep->com.dev->lock, flags);
- _insert_handle(ep->com.dev, &ep->com.dev->hwtid_idr, ep, ep->hwtid, 0);
- spin_unlock_irqrestore(&ep->com.dev->lock, flags);
+ return err;
}
/*
struct c4iw_ep *ep;
unsigned long flags;
- spin_lock_irqsave(&dev->lock, flags);
- ep = idr_find(&dev->hwtid_idr, tid);
+ xa_lock_irqsave(&dev->hwtids, flags);
+ ep = xa_load(&dev->hwtids, tid);
if (ep)
c4iw_get_ep(&ep->com);
- spin_unlock_irqrestore(&dev->lock, flags);
+ xa_unlock_irqrestore(&dev->hwtids, flags);
return ep;
}
struct c4iw_listen_ep *ep;
unsigned long flags;
- spin_lock_irqsave(&dev->lock, flags);
- ep = idr_find(&dev->stid_idr, stid);
+ xa_lock_irqsave(&dev->stids, flags);
+ ep = xa_load(&dev->stids, stid);
if (ep)
c4iw_get_ep(&ep->com);
- spin_unlock_irqrestore(&dev->lock, flags);
+ xa_unlock_irqrestore(&dev->stids, flags);
return ep;
}
skb_reset_transport_header(skb);
} else {
skb = alloc_skb(len, gfp);
+ if (!skb)
+ return NULL;
}
t4_set_arp_err_handler(skb, NULL, NULL);
return skb;
cxgb4_clip_release(ep->com.dev->rdev.lldi.ports[0],
(const u32 *)&sin6->sin6_addr.s6_addr, 1);
}
- remove_handle(ep->com.dev, &ep->com.dev->atid_idr, ep->atid);
+ xa_erase_irq(&ep->com.dev->atids, ep->atid);
cxgb4_free_atid(ep->com.dev->rdev.lldi.tids, ep->atid);
queue_arp_failure_cpl(ep, skb, FAKE_CPL_PUT_EP_SAFE);
}
set_emss(ep, tcp_opt);
/* dealloc the atid */
- remove_handle(ep->com.dev, &ep->com.dev->atid_idr, atid);
+ xa_erase_irq(&ep->com.dev->atids, atid);
cxgb4_free_atid(t, atid);
set_bit(ACT_ESTAB, &ep->com.history);
err = -ENOMEM;
goto fail2;
}
- insert_handle(ep->com.dev, &ep->com.dev->atid_idr, ep, ep->atid);
+ err = xa_insert_irq(&ep->com.dev->atids, ep->atid, ep, GFP_KERNEL);
+ if (err)
+ goto fail2a;
/* find a route */
if (ep->com.cm_id->m_local_addr.ss_family == AF_INET) {
fail4:
dst_release(ep->dst);
fail3:
- remove_handle(ep->com.dev, &ep->com.dev->atid_idr, ep->atid);
+ xa_erase_irq(&ep->com.dev->atids, ep->atid);
+fail2a:
cxgb4_free_atid(ep->com.dev->rdev.lldi.tids, ep->atid);
fail2:
/*
(const u32 *)
&sin6->sin6_addr.s6_addr, 1);
}
- remove_handle(ep->com.dev, &ep->com.dev->atid_idr,
- atid);
+ xa_erase_irq(&ep->com.dev->atids, atid);
cxgb4_free_atid(t, atid);
dst_release(ep->dst);
cxgb4_l2t_release(ep->l2t);
cxgb4_remove_tid(ep->com.dev->rdev.lldi.tids, 0, GET_TID(rpl),
ep->com.local_addr.ss_family);
- remove_handle(ep->com.dev, &ep->com.dev->atid_idr, atid);
+ xa_erase_irq(&ep->com.dev->atids, atid);
cxgb4_free_atid(t, atid);
dst_release(ep->dst);
cxgb4_l2t_release(ep->l2t);
(const u32 *)&sin6->sin6_addr.s6_addr,
1);
}
- remove_handle(ep->com.dev, &ep->com.dev->hwtid_idr, ep->hwtid);
+ xa_erase_irq(&ep->com.dev->hwtids, ep->hwtid);
cxgb4_remove_tid(ep->com.dev->rdev.lldi.tids, 0, ep->hwtid,
ep->com.local_addr.ss_family);
dst_release(ep->dst);
err = -ENOMEM;
goto fail2;
}
- insert_handle(dev, &dev->atid_idr, ep, ep->atid);
+ err = xa_insert_irq(&dev->atids, ep->atid, ep, GFP_KERNEL);
+ if (err)
+ goto fail5;
memcpy(&ep->com.local_addr, &cm_id->m_local_addr,
sizeof(ep->com.local_addr));
fail4:
dst_release(ep->dst);
fail3:
- remove_handle(ep->com.dev, &ep->com.dev->atid_idr, ep->atid);
+ xa_erase_irq(&ep->com.dev->atids, ep->atid);
+fail5:
cxgb4_free_atid(ep->com.dev->rdev.lldi.tids, ep->atid);
fail2:
skb_queue_purge(&ep->com.ep_skb_list);
err = -ENOMEM;
goto fail2;
}
- insert_handle(dev, &dev->stid_idr, ep, ep->stid);
+ err = xa_insert_irq(&dev->stids, ep->stid, ep, GFP_KERNEL);
+ if (err)
+ goto fail3;
state_set(&ep->com, LISTEN);
if (ep->com.local_addr.ss_family == AF_INET)
cm_id->provider_data = ep;
goto out;
}
- remove_handle(ep->com.dev, &ep->com.dev->stid_idr, ep->stid);
+ xa_erase_irq(&ep->com.dev->stids, ep->stid);
+fail3:
cxgb4_free_stid(ep->com.dev->rdev.lldi.tids, ep->stid,
ep->com.local_addr.ss_family);
fail2:
cxgb4_clip_release(ep->com.dev->rdev.lldi.ports[0],
(const u32 *)&sin6->sin6_addr.s6_addr, 1);
}
- remove_handle(ep->com.dev, &ep->com.dev->stid_idr, ep->stid);
+ xa_erase_irq(&ep->com.dev->stids, ep->stid);
cxgb4_free_stid(ep->com.dev->rdev.lldi.tids, ep->stid,
ep->com.local_addr.ss_family);
done:
cxgb4_clip_release(ep->com.dev->rdev.lldi.ports[0],
(const u32 *)&sin6->sin6_addr.s6_addr, 1);
}
- remove_handle(dev, &dev->atid_idr, atid);
+ xa_erase_irq(&dev->atids, atid);
cxgb4_free_atid(dev->rdev.lldi.tids, atid);
dst_release(ep->dst);
cxgb4_l2t_release(ep->l2t);
* SOFTWARE.
*/
+#include <rdma/uverbs_ioctl.h>
+
#include "iw_cxgb4.h"
static int destroy_cq(struct c4iw_rdev *rdev, struct t4_cq *cq,
return !err || err == -ENODATA ? npolled : err;
}
-int c4iw_destroy_cq(struct ib_cq *ib_cq)
+int c4iw_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
{
struct c4iw_cq *chp;
struct c4iw_ucontext *ucontext;
pr_debug("ib_cq %p\n", ib_cq);
chp = to_c4iw_cq(ib_cq);
- remove_handle(chp->rhp, &chp->rhp->cqidr, chp->cq.cqid);
+ xa_erase_irq(&chp->rhp->cqs, chp->cq.cqid);
atomic_dec(&chp->refcnt);
wait_event(chp->wait, !atomic_read(&chp->refcnt));
- ucontext = ib_cq->uobject ? to_c4iw_ucontext(ib_cq->uobject->context)
- : NULL;
+ ucontext = rdma_udata_to_drv_context(udata, struct c4iw_ucontext,
+ ibucontext);
destroy_cq(&chp->rhp->rdev, &chp->cq,
ucontext ? &ucontext->uctx : &chp->cq.rdev->uctx,
chp->destroy_skb, chp->wr_waitp);
struct ib_cq *c4iw_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *ib_context,
struct ib_udata *udata)
{
int entries = attr->cqe;
struct c4iw_cq *chp;
struct c4iw_create_cq ucmd;
struct c4iw_create_cq_resp uresp;
- struct c4iw_ucontext *ucontext = NULL;
int ret, wr_len;
size_t memsize, hwentries;
struct c4iw_mm_entry *mm, *mm2;
+ struct c4iw_ucontext *ucontext = rdma_udata_to_drv_context(
+ udata, struct c4iw_ucontext, ibucontext);
pr_debug("ib_dev %p entries %d\n", ibdev, entries);
if (attr->flags)
if (vector >= rhp->rdev.lldi.nciq)
return ERR_PTR(-EINVAL);
- if (ib_context) {
- ucontext = to_c4iw_ucontext(ib_context);
+ if (udata) {
if (udata->inlen < sizeof(ucmd))
ucontext->is_32b_cqe = 1;
}
/*
* memsize must be a multiple of the page size if its a user cq.
*/
- if (ucontext)
+ if (udata)
memsize = roundup(memsize, PAGE_SIZE);
chp->cq.size = hwentries;
spin_lock_init(&chp->comp_handler_lock);
atomic_set(&chp->refcnt, 1);
init_waitqueue_head(&chp->wait);
- ret = insert_handle(rhp, &rhp->cqidr, chp, chp->cq.cqid);
+ ret = xa_insert_irq(&rhp->cqs, chp->cq.cqid, chp, GFP_KERNEL);
if (ret)
goto err_destroy_cq;
err_free_mm:
kfree(mm);
err_remove_handle:
- remove_handle(rhp, &rhp->cqidr, chp->cq.cqid);
+ xa_erase_irq(&rhp->cqs, chp->cq.cqid);
err_destroy_cq:
destroy_cq(&chp->rhp->rdev, &chp->cq,
ucontext ? &ucontext->uctx : &rhp->rdev.uctx,
int pos;
};
-static int count_idrs(int id, void *p, void *data)
-{
- int *countp = data;
-
- *countp = *countp + 1;
- return 0;
-}
-
static ssize_t debugfs_read(struct file *file, char __user *buf, size_t count,
loff_t *ppos)
{
}
}
-static int dump_qp(int id, void *p, void *data)
+static int dump_qp(struct c4iw_qp *qp, struct c4iw_debugfs_data *qpd)
{
- struct c4iw_qp *qp = p;
- struct c4iw_debugfs_data *qpd = data;
int space;
int cc;
- if (id != qp->wq.sq.qid)
- return 0;
-
space = qpd->bufsize - qpd->pos - 1;
if (space == 0)
return 1;
static int qp_open(struct inode *inode, struct file *file)
{
+ struct c4iw_qp *qp;
struct c4iw_debugfs_data *qpd;
+ unsigned long index;
int count = 1;
qpd = kmalloc(sizeof *qpd, GFP_KERNEL);
qpd->devp = inode->i_private;
qpd->pos = 0;
- spin_lock_irq(&qpd->devp->lock);
- idr_for_each(&qpd->devp->qpidr, count_idrs, &count);
- spin_unlock_irq(&qpd->devp->lock);
+ /*
+ * No need to lock; we drop the lock to call vmalloc so it's racy
+ * anyway. Someone who cares should switch this over to seq_file
+ */
+ xa_for_each(&qpd->devp->qps, index, qp)
+ count++;
qpd->bufsize = count * 180;
qpd->buf = vmalloc(qpd->bufsize);
return -ENOMEM;
}
- spin_lock_irq(&qpd->devp->lock);
- idr_for_each(&qpd->devp->qpidr, dump_qp, qpd);
- spin_unlock_irq(&qpd->devp->lock);
+ xa_lock_irq(&qpd->devp->qps);
+ xa_for_each(&qpd->devp->qps, index, qp)
+ dump_qp(qp, qpd);
+ xa_unlock_irq(&qpd->devp->qps);
qpd->buf[qpd->pos++] = 0;
file->private_data = qpd;
.llseek = default_llseek,
};
-static int dump_stag(int id, void *p, void *data)
+static int dump_stag(unsigned long id, struct c4iw_debugfs_data *stagd)
{
- struct c4iw_debugfs_data *stagd = data;
int space;
int cc;
struct fw_ri_tpte tpte;
static int stag_open(struct inode *inode, struct file *file)
{
struct c4iw_debugfs_data *stagd;
+ void *p;
+ unsigned long index;
int ret = 0;
int count = 1;
stagd->devp = inode->i_private;
stagd->pos = 0;
- spin_lock_irq(&stagd->devp->lock);
- idr_for_each(&stagd->devp->mmidr, count_idrs, &count);
- spin_unlock_irq(&stagd->devp->lock);
+ xa_for_each(&stagd->devp->mrs, index, p)
+ count++;
stagd->bufsize = count * 256;
stagd->buf = vmalloc(stagd->bufsize);
goto err1;
}
- spin_lock_irq(&stagd->devp->lock);
- idr_for_each(&stagd->devp->mmidr, dump_stag, stagd);
- spin_unlock_irq(&stagd->devp->lock);
+ xa_lock_irq(&stagd->devp->mrs);
+ xa_for_each(&stagd->devp->mrs, index, p)
+ dump_stag(index, stagd);
+ xa_unlock_irq(&stagd->devp->mrs);
stagd->buf[stagd->pos++] = 0;
file->private_data = stagd;
.write = stats_clear,
};
-static int dump_ep(int id, void *p, void *data)
+static int dump_ep(struct c4iw_ep *ep, struct c4iw_debugfs_data *epd)
{
- struct c4iw_ep *ep = p;
- struct c4iw_debugfs_data *epd = data;
int space;
int cc;
return 0;
}
-static int dump_listen_ep(int id, void *p, void *data)
+static
+int dump_listen_ep(struct c4iw_listen_ep *ep, struct c4iw_debugfs_data *epd)
{
- struct c4iw_listen_ep *ep = p;
- struct c4iw_debugfs_data *epd = data;
int space;
int cc;
static int ep_open(struct inode *inode, struct file *file)
{
+ struct c4iw_ep *ep;
+ struct c4iw_listen_ep *lep;
+ unsigned long index;
struct c4iw_debugfs_data *epd;
int ret = 0;
int count = 1;
epd->devp = inode->i_private;
epd->pos = 0;
- spin_lock_irq(&epd->devp->lock);
- idr_for_each(&epd->devp->hwtid_idr, count_idrs, &count);
- idr_for_each(&epd->devp->atid_idr, count_idrs, &count);
- idr_for_each(&epd->devp->stid_idr, count_idrs, &count);
- spin_unlock_irq(&epd->devp->lock);
+ xa_for_each(&epd->devp->hwtids, index, ep)
+ count++;
+ xa_for_each(&epd->devp->atids, index, ep)
+ count++;
+ xa_for_each(&epd->devp->stids, index, lep)
+ count++;
epd->bufsize = count * 240;
epd->buf = vmalloc(epd->bufsize);
goto err1;
}
- spin_lock_irq(&epd->devp->lock);
- idr_for_each(&epd->devp->hwtid_idr, dump_ep, epd);
- idr_for_each(&epd->devp->atid_idr, dump_ep, epd);
- idr_for_each(&epd->devp->stid_idr, dump_listen_ep, epd);
- spin_unlock_irq(&epd->devp->lock);
+ xa_lock_irq(&epd->devp->hwtids);
+ xa_for_each(&epd->devp->hwtids, index, ep)
+ dump_ep(ep, epd);
+ xa_unlock_irq(&epd->devp->hwtids);
+ xa_lock_irq(&epd->devp->atids);
+ xa_for_each(&epd->devp->atids, index, ep)
+ dump_ep(ep, epd);
+ xa_unlock_irq(&epd->devp->atids);
+ xa_lock_irq(&epd->devp->stids);
+ xa_for_each(&epd->devp->stids, index, lep)
+ dump_listen_ep(lep, epd);
+ xa_unlock_irq(&epd->devp->stids);
file->private_data = epd;
goto out;
void c4iw_dealloc(struct uld_ctx *ctx)
{
c4iw_rdev_close(&ctx->dev->rdev);
- WARN_ON_ONCE(!idr_is_empty(&ctx->dev->cqidr));
- idr_destroy(&ctx->dev->cqidr);
- WARN_ON_ONCE(!idr_is_empty(&ctx->dev->qpidr));
- idr_destroy(&ctx->dev->qpidr);
- WARN_ON_ONCE(!idr_is_empty(&ctx->dev->mmidr));
- idr_destroy(&ctx->dev->mmidr);
- wait_event(ctx->dev->wait, idr_is_empty(&ctx->dev->hwtid_idr));
- idr_destroy(&ctx->dev->hwtid_idr);
- idr_destroy(&ctx->dev->stid_idr);
- idr_destroy(&ctx->dev->atid_idr);
+ WARN_ON(!xa_empty(&ctx->dev->cqs));
+ WARN_ON(!xa_empty(&ctx->dev->qps));
+ WARN_ON(!xa_empty(&ctx->dev->mrs));
+ wait_event(ctx->dev->wait, xa_empty(&ctx->dev->hwtids));
+ WARN_ON(!xa_empty(&ctx->dev->stids));
+ WARN_ON(!xa_empty(&ctx->dev->atids));
if (ctx->dev->rdev.bar2_kva)
iounmap(ctx->dev->rdev.bar2_kva);
if (ctx->dev->rdev.oc_mw_kva)
return ERR_PTR(ret);
}
- idr_init(&devp->cqidr);
- idr_init(&devp->qpidr);
- idr_init(&devp->mmidr);
- idr_init(&devp->hwtid_idr);
- idr_init(&devp->stid_idr);
- idr_init(&devp->atid_idr);
- spin_lock_init(&devp->lock);
+ xa_init_flags(&devp->cqs, XA_FLAGS_LOCK_IRQ);
+ xa_init_flags(&devp->qps, XA_FLAGS_LOCK_IRQ);
+ xa_init_flags(&devp->mrs, XA_FLAGS_LOCK_IRQ);
+ xa_init_flags(&devp->hwtids, XA_FLAGS_LOCK_IRQ);
+ xa_init_flags(&devp->atids, XA_FLAGS_LOCK_IRQ);
+ xa_init_flags(&devp->stids, XA_FLAGS_LOCK_IRQ);
mutex_init(&devp->rdev.stats.lock);
mutex_init(&devp->db_mutex);
INIT_LIST_HEAD(&devp->db_fc_list);
return 0;
}
-static int disable_qp_db(int id, void *p, void *data)
-{
- struct c4iw_qp *qp = p;
-
- t4_disable_wq_db(&qp->wq);
- return 0;
-}
-
static void stop_queues(struct uld_ctx *ctx)
{
- unsigned long flags;
+ struct c4iw_qp *qp;
+ unsigned long index, flags;
- spin_lock_irqsave(&ctx->dev->lock, flags);
+ xa_lock_irqsave(&ctx->dev->qps, flags);
ctx->dev->rdev.stats.db_state_transitions++;
ctx->dev->db_state = STOPPED;
- if (ctx->dev->rdev.flags & T4_STATUS_PAGE_DISABLED)
- idr_for_each(&ctx->dev->qpidr, disable_qp_db, NULL);
- else
+ if (ctx->dev->rdev.flags & T4_STATUS_PAGE_DISABLED) {
+ xa_for_each(&ctx->dev->qps, index, qp)
+ t4_disable_wq_db(&qp->wq);
+ } else {
ctx->dev->rdev.status_page->db_off = 1;
- spin_unlock_irqrestore(&ctx->dev->lock, flags);
-}
-
-static int enable_qp_db(int id, void *p, void *data)
-{
- struct c4iw_qp *qp = p;
-
- t4_enable_wq_db(&qp->wq);
- return 0;
+ }
+ xa_unlock_irqrestore(&ctx->dev->qps, flags);
}
static void resume_rc_qp(struct c4iw_qp *qp)
static void resume_queues(struct uld_ctx *ctx)
{
- spin_lock_irq(&ctx->dev->lock);
+ xa_lock_irq(&ctx->dev->qps);
if (ctx->dev->db_state != STOPPED)
goto out;
ctx->dev->db_state = FLOW_CONTROL;
while (1) {
if (list_empty(&ctx->dev->db_fc_list)) {
+ struct c4iw_qp *qp;
+ unsigned long index;
+
WARN_ON(ctx->dev->db_state != FLOW_CONTROL);
ctx->dev->db_state = NORMAL;
ctx->dev->rdev.stats.db_state_transitions++;
if (ctx->dev->rdev.flags & T4_STATUS_PAGE_DISABLED) {
- idr_for_each(&ctx->dev->qpidr, enable_qp_db,
- NULL);
+ xa_for_each(&ctx->dev->qps, index, qp)
+ t4_enable_wq_db(&qp->wq);
} else {
ctx->dev->rdev.status_page->db_off = 0;
}
resume_a_chunk(ctx);
}
if (!list_empty(&ctx->dev->db_fc_list)) {
- spin_unlock_irq(&ctx->dev->lock);
+ xa_unlock_irq(&ctx->dev->qps);
if (DB_FC_RESUME_DELAY) {
set_current_state(TASK_UNINTERRUPTIBLE);
schedule_timeout(DB_FC_RESUME_DELAY);
}
- spin_lock_irq(&ctx->dev->lock);
+ xa_lock_irq(&ctx->dev->qps);
if (ctx->dev->db_state != FLOW_CONTROL)
break;
}
out:
if (ctx->dev->db_state != NORMAL)
ctx->dev->rdev.stats.db_fc_interruptions++;
- spin_unlock_irq(&ctx->dev->lock);
+ xa_unlock_irq(&ctx->dev->qps);
}
struct qp_list {
struct c4iw_qp **qps;
};
-static int add_and_ref_qp(int id, void *p, void *data)
-{
- struct qp_list *qp_listp = data;
- struct c4iw_qp *qp = p;
-
- c4iw_qp_add_ref(&qp->ibqp);
- qp_listp->qps[qp_listp->idx++] = qp;
- return 0;
-}
-
-static int count_qps(int id, void *p, void *data)
-{
- unsigned *countp = data;
- (*countp)++;
- return 0;
-}
-
static void deref_qps(struct qp_list *qp_list)
{
int idx;
for (idx = 0; idx < qp_list->idx; idx++) {
struct c4iw_qp *qp = qp_list->qps[idx];
- spin_lock_irq(&qp->rhp->lock);
+ xa_lock_irq(&qp->rhp->qps);
spin_lock(&qp->lock);
ret = cxgb4_sync_txq_pidx(qp->rhp->rdev.lldi.ports[0],
qp->wq.sq.qid,
pr_err("%s: Fatal error - DB overflow recovery failed - error syncing SQ qid %u\n",
pci_name(ctx->lldi.pdev), qp->wq.sq.qid);
spin_unlock(&qp->lock);
- spin_unlock_irq(&qp->rhp->lock);
+ xa_unlock_irq(&qp->rhp->qps);
return;
}
qp->wq.sq.wq_pidx_inc = 0;
pr_err("%s: Fatal error - DB overflow recovery failed - error syncing RQ qid %u\n",
pci_name(ctx->lldi.pdev), qp->wq.rq.qid);
spin_unlock(&qp->lock);
- spin_unlock_irq(&qp->rhp->lock);
+ xa_unlock_irq(&qp->rhp->qps);
return;
}
qp->wq.rq.wq_pidx_inc = 0;
spin_unlock(&qp->lock);
- spin_unlock_irq(&qp->rhp->lock);
+ xa_unlock_irq(&qp->rhp->qps);
/* Wait for the dbfifo to drain */
while (cxgb4_dbfifo_count(qp->rhp->rdev.lldi.ports[0], 1) > 0) {
static void recover_queues(struct uld_ctx *ctx)
{
+ struct c4iw_qp *qp;
+ unsigned long index;
int count = 0;
struct qp_list qp_list;
int ret;
}
/* Count active queues so we can build a list of queues to recover */
- spin_lock_irq(&ctx->dev->lock);
+ xa_lock_irq(&ctx->dev->qps);
WARN_ON(ctx->dev->db_state != STOPPED);
ctx->dev->db_state = RECOVERY;
- idr_for_each(&ctx->dev->qpidr, count_qps, &count);
+ xa_for_each(&ctx->dev->qps, index, qp)
+ count++;
qp_list.qps = kcalloc(count, sizeof(*qp_list.qps), GFP_ATOMIC);
if (!qp_list.qps) {
- spin_unlock_irq(&ctx->dev->lock);
+ xa_unlock_irq(&ctx->dev->qps);
return;
}
qp_list.idx = 0;
/* add and ref each qp so it doesn't get freed */
- idr_for_each(&ctx->dev->qpidr, add_and_ref_qp, &qp_list);
+ xa_for_each(&ctx->dev->qps, index, qp) {
+ c4iw_qp_add_ref(&qp->ibqp);
+ qp_list.qps[qp_list.idx++] = qp;
+ }
- spin_unlock_irq(&ctx->dev->lock);
+ xa_unlock_irq(&ctx->dev->qps);
/* now traverse the list in a safe context to recover the db state*/
recover_lost_dbs(ctx, &qp_list);
deref_qps(&qp_list);
kfree(qp_list.qps);
- spin_lock_irq(&ctx->dev->lock);
+ xa_lock_irq(&ctx->dev->qps);
WARN_ON(ctx->dev->db_state != RECOVERY);
ctx->dev->db_state = STOPPED;
- spin_unlock_irq(&ctx->dev->lock);
+ xa_unlock_irq(&ctx->dev->qps);
}
static int c4iw_uld_control(void *handle, enum cxgb4_control control, ...)
struct c4iw_qp *qhp;
u32 cqid;
- spin_lock_irq(&dev->lock);
- qhp = get_qhp(dev, CQE_QPID(err_cqe));
+ xa_lock_irq(&dev->qps);
+ qhp = xa_load(&dev->qps, CQE_QPID(err_cqe));
if (!qhp) {
pr_err("BAD AE qpid 0x%x opcode %d status 0x%x type %d wrid.hi 0x%x wrid.lo 0x%x\n",
CQE_QPID(err_cqe),
CQE_OPCODE(err_cqe), CQE_STATUS(err_cqe),
CQE_TYPE(err_cqe), CQE_WRID_HI(err_cqe),
CQE_WRID_LOW(err_cqe));
- spin_unlock_irq(&dev->lock);
+ xa_unlock_irq(&dev->qps);
goto out;
}
CQE_OPCODE(err_cqe), CQE_STATUS(err_cqe),
CQE_TYPE(err_cqe), CQE_WRID_HI(err_cqe),
CQE_WRID_LOW(err_cqe));
- spin_unlock_irq(&dev->lock);
+ xa_unlock_irq(&dev->qps);
goto out;
}
c4iw_qp_add_ref(&qhp->ibqp);
atomic_inc(&chp->refcnt);
- spin_unlock_irq(&dev->lock);
+ xa_unlock_irq(&dev->qps);
/* Bad incoming write */
if (RQ_TYPE(err_cqe) &&
struct c4iw_cq *chp;
unsigned long flag;
- spin_lock_irqsave(&dev->lock, flag);
- chp = get_chp(dev, qid);
+ xa_lock_irqsave(&dev->cqs, flag);
+ chp = xa_load(&dev->cqs, qid);
if (chp) {
atomic_inc(&chp->refcnt);
- spin_unlock_irqrestore(&dev->lock, flag);
+ xa_unlock_irqrestore(&dev->cqs, flag);
t4_clear_cq_armed(&chp->cq);
spin_lock_irqsave(&chp->comp_handler_lock, flag);
(*chp->ibcq.comp_handler)(&chp->ibcq, chp->ibcq.cq_context);
wake_up(&chp->wait);
} else {
pr_debug("unknown cqid 0x%x\n", qid);
- spin_unlock_irqrestore(&dev->lock, flag);
+ xa_unlock_irqrestore(&dev->cqs, flag);
}
return 0;
}
#include <linux/mutex.h>
#include <linux/list.h>
#include <linux/spinlock.h>
-#include <linux/idr.h>
+#include <linux/xarray.h>
#include <linux/completion.h>
#include <linux/netdevice.h>
#include <linux/sched/mm.h>
struct ib_device ibdev;
struct c4iw_rdev rdev;
u32 device_cap_flags;
- struct idr cqidr;
- struct idr qpidr;
- struct idr mmidr;
- spinlock_t lock;
+ struct xarray cqs;
+ struct xarray qps;
+ struct xarray mrs;
struct mutex db_mutex;
struct dentry *debugfs_root;
enum db_state db_state;
- struct idr hwtid_idr;
- struct idr atid_idr;
- struct idr stid_idr;
+ struct xarray hwtids;
+ struct xarray atids;
+ struct xarray stids;
struct list_head db_fc_list;
u32 avail_ird;
wait_queue_head_t wait;
static inline struct c4iw_cq *get_chp(struct c4iw_dev *rhp, u32 cqid)
{
- return idr_find(&rhp->cqidr, cqid);
+ return xa_load(&rhp->cqs, cqid);
}
static inline struct c4iw_qp *get_qhp(struct c4iw_dev *rhp, u32 qpid)
{
- return idr_find(&rhp->qpidr, qpid);
-}
-
-static inline struct c4iw_mr *get_mhp(struct c4iw_dev *rhp, u32 mmid)
-{
- return idr_find(&rhp->mmidr, mmid);
-}
-
-static inline int _insert_handle(struct c4iw_dev *rhp, struct idr *idr,
- void *handle, u32 id, int lock)
-{
- int ret;
-
- if (lock) {
- idr_preload(GFP_KERNEL);
- spin_lock_irq(&rhp->lock);
- }
-
- ret = idr_alloc(idr, handle, id, id + 1, GFP_ATOMIC);
-
- if (lock) {
- spin_unlock_irq(&rhp->lock);
- idr_preload_end();
- }
-
- return ret < 0 ? ret : 0;
-}
-
-static inline int insert_handle(struct c4iw_dev *rhp, struct idr *idr,
- void *handle, u32 id)
-{
- return _insert_handle(rhp, idr, handle, id, 1);
-}
-
-static inline int insert_handle_nolock(struct c4iw_dev *rhp, struct idr *idr,
- void *handle, u32 id)
-{
- return _insert_handle(rhp, idr, handle, id, 0);
-}
-
-static inline void _remove_handle(struct c4iw_dev *rhp, struct idr *idr,
- u32 id, int lock)
-{
- if (lock)
- spin_lock_irq(&rhp->lock);
- idr_remove(idr, id);
- if (lock)
- spin_unlock_irq(&rhp->lock);
-}
-
-static inline void remove_handle(struct c4iw_dev *rhp, struct idr *idr, u32 id)
-{
- _remove_handle(rhp, idr, id, 1);
-}
-
-static inline void remove_handle_nolock(struct c4iw_dev *rhp,
- struct idr *idr, u32 id)
-{
- _remove_handle(rhp, idr, id, 0);
+ return xa_load(&rhp->qps, qpid);
}
extern uint c4iw_max_read_depth;
int c4iw_reject_cr(struct iw_cm_id *cm_id, const void *pdata, u8 pdata_len);
void c4iw_qp_add_ref(struct ib_qp *qp);
void c4iw_qp_rem_ref(struct ib_qp *qp);
-struct ib_mr *c4iw_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg);
+struct ib_mr *c4iw_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata);
int c4iw_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
unsigned int *sg_offset);
int c4iw_dealloc_mw(struct ib_mw *mw);
u64 length, u64 virt, int acc,
struct ib_udata *udata);
struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc);
-int c4iw_dereg_mr(struct ib_mr *ib_mr);
-int c4iw_destroy_cq(struct ib_cq *ib_cq);
+int c4iw_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata);
+int c4iw_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata);
struct ib_cq *c4iw_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *ib_context,
struct ib_udata *udata);
int c4iw_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags);
int c4iw_modify_srq(struct ib_srq *ib_srq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask srq_attr_mask,
struct ib_udata *udata);
-int c4iw_destroy_srq(struct ib_srq *ib_srq);
-struct ib_srq *c4iw_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *attrs,
- struct ib_udata *udata);
-int c4iw_destroy_qp(struct ib_qp *ib_qp);
+void c4iw_destroy_srq(struct ib_srq *ib_srq, struct ib_udata *udata);
+int c4iw_create_srq(struct ib_srq *srq, struct ib_srq_init_attr *attrs,
+ struct ib_udata *udata);
+int c4iw_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata);
struct ib_qp *c4iw_create_qp(struct ib_pd *pd,
struct ib_qp_init_attr *attrs,
struct ib_udata *udata);
mhp->ibmr.iova = mhp->attr.va_fbo;
mhp->ibmr.page_size = 1U << (mhp->attr.page_size + 12);
pr_debug("mmid 0x%x mhp %p\n", mmid, mhp);
- return insert_handle(mhp->rhp, &mhp->rhp->mmidr, mhp, mmid);
+ return xa_insert_irq(&mhp->rhp->mrs, mmid, mhp, GFP_KERNEL);
}
static int register_mem(struct c4iw_dev *rhp, struct c4iw_pd *php,
shift = PAGE_SHIFT;
- n = mhp->umem->nmap;
+ n = ib_umem_num_pages(mhp->umem);
err = alloc_pbl(mhp, n);
if (err)
goto err_umem_release;
mhp->attr.stag = stag;
mmid = (stag) >> 8;
mhp->ibmw.rkey = stag;
- if (insert_handle(rhp, &rhp->mmidr, mhp, mmid)) {
+ if (xa_insert_irq(&rhp->mrs, mmid, mhp, GFP_KERNEL)) {
ret = -ENOMEM;
goto dealloc_win;
}
mhp = to_c4iw_mw(mw);
rhp = mhp->rhp;
mmid = (mw->rkey) >> 8;
- remove_handle(rhp, &rhp->mmidr, mmid);
+ xa_erase_irq(&rhp->mrs, mmid);
deallocate_window(&rhp->rdev, mhp->attr.stag, mhp->dereg_skb,
mhp->wr_waitp);
kfree_skb(mhp->dereg_skb);
return 0;
}
-struct ib_mr *c4iw_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg)
+struct ib_mr *c4iw_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
struct c4iw_dev *rhp;
struct c4iw_pd *php;
mhp->attr.state = 0;
mmid = (stag) >> 8;
mhp->ibmr.rkey = mhp->ibmr.lkey = stag;
- if (insert_handle(rhp, &rhp->mmidr, mhp, mmid)) {
+ if (xa_insert_irq(&rhp->mrs, mmid, mhp, GFP_KERNEL)) {
ret = -ENOMEM;
goto err_dereg;
}
return ib_sg_to_pages(ibmr, sg, sg_nents, sg_offset, c4iw_set_page);
}
-int c4iw_dereg_mr(struct ib_mr *ib_mr)
+int c4iw_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
{
struct c4iw_dev *rhp;
struct c4iw_mr *mhp;
mhp = to_c4iw_mr(ib_mr);
rhp = mhp->rhp;
mmid = mhp->attr.stag >> 8;
- remove_handle(rhp, &rhp->mmidr, mmid);
+ xa_erase_irq(&rhp->mrs, mmid);
if (mhp->mpl)
dma_free_coherent(&mhp->rhp->rdev.lldi.pdev->dev,
mhp->max_mpl_len, mhp->mpl, mhp->mpl_addr);
struct c4iw_mr *mhp;
unsigned long flags;
- spin_lock_irqsave(&rhp->lock, flags);
- mhp = get_mhp(rhp, rkey >> 8);
+ xa_lock_irqsave(&rhp->mrs, flags);
+ mhp = xa_load(&rhp->mrs, rkey >> 8);
if (mhp)
mhp->attr.state = 0;
- spin_unlock_irqrestore(&rhp->lock, flags);
+ xa_unlock_irqrestore(&rhp->mrs, flags);
}
return ret;
}
-static void c4iw_deallocate_pd(struct ib_pd *pd)
+static void c4iw_deallocate_pd(struct ib_pd *pd, struct ib_udata *udata)
{
struct c4iw_dev *rhp;
struct c4iw_pd *php;
mutex_unlock(&rhp->rdev.stats.lock);
}
-static int c4iw_allocate_pd(struct ib_pd *pd, struct ib_ucontext *context,
- struct ib_udata *udata)
+static int c4iw_allocate_pd(struct ib_pd *pd, struct ib_udata *udata)
{
struct c4iw_pd *php = to_c4iw_pd(pd);
struct ib_device *ibdev = pd->device;
php->pdid = pdid;
php->rhp = rhp;
- if (context) {
+ if (udata) {
struct c4iw_alloc_pd_resp uresp = {.pdid = php->pdid};
if (ib_copy_to_udata(udata, &uresp, sizeof(uresp))) {
- c4iw_deallocate_pd(&php->ibpd);
+ c4iw_deallocate_pd(&php->ibpd, udata);
return -EFAULT;
}
}
FW_HDR_FW_VER_BUILD_G(c4iw_dev->rdev.lldi.fw_vers));
}
-static struct net_device *get_netdev(struct ib_device *dev, u8 port)
-{
- struct c4iw_dev *c4iw_dev = container_of(dev, struct c4iw_dev, ibdev);
- struct c4iw_rdev *rdev = &c4iw_dev->rdev;
- struct net_device *ndev;
-
- if (!port || port > rdev->lldi.nports)
- return NULL;
-
- rcu_read_lock();
- ndev = rdev->lldi.ports[port - 1];
- if (ndev)
- dev_hold(ndev);
- rcu_read_unlock();
-
- return ndev;
-}
-
static int fill_res_entry(struct sk_buff *msg, struct rdma_restrack_entry *res)
{
return (res->type < ARRAY_SIZE(c4iw_restrack_funcs) &&
.get_dev_fw_str = get_dev_fw_str,
.get_dma_mr = c4iw_get_dma_mr,
.get_hw_stats = c4iw_get_mib,
- .get_netdev = get_netdev,
.get_port_immutable = c4iw_port_immutable,
+ .iw_accept = c4iw_accept_cr,
+ .iw_add_ref = c4iw_qp_add_ref,
+ .iw_connect = c4iw_connect,
+ .iw_create_listen = c4iw_create_listen,
+ .iw_destroy_listen = c4iw_destroy_listen,
+ .iw_get_qp = c4iw_get_qp,
+ .iw_reject = c4iw_reject_cr,
+ .iw_rem_ref = c4iw_qp_rem_ref,
.map_mr_sg = c4iw_map_mr_sg,
.mmap = c4iw_mmap,
.modify_qp = c4iw_ib_modify_qp,
.reg_user_mr = c4iw_reg_user_mr,
.req_notify_cq = c4iw_arm_cq,
INIT_RDMA_OBJ_SIZE(ib_pd, c4iw_pd, ibpd),
+ INIT_RDMA_OBJ_SIZE(ib_srq, c4iw_srq, ibsrq),
INIT_RDMA_OBJ_SIZE(ib_ucontext, c4iw_ucontext, ibucontext),
};
+static int set_netdevs(struct ib_device *ib_dev, struct c4iw_rdev *rdev)
+{
+ int ret;
+ int i;
+
+ for (i = 0; i < rdev->lldi.nports; i++) {
+ ret = ib_device_set_netdev(ib_dev, rdev->lldi.ports[i],
+ i + 1);
+ if (ret)
+ return ret;
+ }
+ return 0;
+}
+
void c4iw_register_device(struct work_struct *work)
{
int ret;
dev->ibdev.dev.parent = &dev->rdev.lldi.pdev->dev;
dev->ibdev.uverbs_abi_ver = C4IW_UVERBS_ABI_VERSION;
- dev->ibdev.iwcm = kzalloc(sizeof(struct iw_cm_verbs), GFP_KERNEL);
- if (!dev->ibdev.iwcm) {
- ret = -ENOMEM;
- goto err_dealloc_ctx;
- }
-
- dev->ibdev.iwcm->connect = c4iw_connect;
- dev->ibdev.iwcm->accept = c4iw_accept_cr;
- dev->ibdev.iwcm->reject = c4iw_reject_cr;
- dev->ibdev.iwcm->create_listen = c4iw_create_listen;
- dev->ibdev.iwcm->destroy_listen = c4iw_destroy_listen;
- dev->ibdev.iwcm->add_ref = c4iw_qp_add_ref;
- dev->ibdev.iwcm->rem_ref = c4iw_qp_rem_ref;
- dev->ibdev.iwcm->get_qp = c4iw_get_qp;
- memcpy(dev->ibdev.iwcm->ifname, dev->rdev.lldi.ports[0]->name,
- sizeof(dev->ibdev.iwcm->ifname));
+ memcpy(dev->ibdev.iw_ifname, dev->rdev.lldi.ports[0]->name,
+ sizeof(dev->ibdev.iw_ifname));
rdma_set_device_sysfs_group(&dev->ibdev, &c4iw_attr_group);
dev->ibdev.driver_id = RDMA_DRIVER_CXGB4;
ib_set_device_ops(&dev->ibdev, &c4iw_dev_ops);
+ ret = set_netdevs(&dev->ibdev, &dev->rdev);
+ if (ret)
+ goto err_dealloc_ctx;
ret = ib_register_device(&dev->ibdev, "cxgb4_%d");
if (ret)
- goto err_kfree_iwcm;
+ goto err_dealloc_ctx;
return;
-err_kfree_iwcm:
- kfree(dev->ibdev.iwcm);
err_dealloc_ctx:
pr_err("%s - Failed registering iwarp device: %d\n",
pci_name(ctx->lldi.pdev), ret);
{
pr_debug("c4iw_dev %p\n", dev);
ib_unregister_device(&dev->ibdev);
- kfree(dev->ibdev.iwcm);
return;
}
static int max_fr_immd = T4_MAX_FR_IMMD;
module_param(max_fr_immd, int, 0644);
-MODULE_PARM_DESC(max_fr_immd, "fastreg threshold for using DSGL instead of immedate");
+MODULE_PARM_DESC(max_fr_immd, "fastreg threshold for using DSGL instead of immediate");
static int alloc_ird(struct c4iw_dev *dev, u32 ird)
{
int ret = 0;
- spin_lock_irq(&dev->lock);
+ xa_lock_irq(&dev->qps);
if (ird <= dev->avail_ird)
dev->avail_ird -= ird;
else
ret = -ENOMEM;
- spin_unlock_irq(&dev->lock);
+ xa_unlock_irq(&dev->qps);
if (ret)
dev_warn(&dev->rdev.lldi.pdev->dev,
static void free_ird(struct c4iw_dev *dev, int ird)
{
- spin_lock_irq(&dev->lock);
+ xa_lock_irq(&dev->qps);
dev->avail_ird += ird;
- spin_unlock_irq(&dev->lock);
+ xa_unlock_irq(&dev->qps);
}
static void set_state(struct c4iw_qp *qhp, enum c4iw_qp_state state)
{
unsigned long flags;
- spin_lock_irqsave(&qhp->rhp->lock, flags);
+ xa_lock_irqsave(&qhp->rhp->qps, flags);
spin_lock(&qhp->lock);
if (qhp->rhp->db_state == NORMAL)
t4_ring_sq_db(&qhp->wq, inc, NULL);
qhp->wq.sq.wq_pidx_inc += inc;
}
spin_unlock(&qhp->lock);
- spin_unlock_irqrestore(&qhp->rhp->lock, flags);
+ xa_unlock_irqrestore(&qhp->rhp->qps, flags);
return 0;
}
{
unsigned long flags;
- spin_lock_irqsave(&qhp->rhp->lock, flags);
+ xa_lock_irqsave(&qhp->rhp->qps, flags);
spin_lock(&qhp->lock);
if (qhp->rhp->db_state == NORMAL)
t4_ring_rq_db(&qhp->wq, inc, NULL);
qhp->wq.rq.wq_pidx_inc += inc;
}
spin_unlock(&qhp->lock);
- spin_unlock_irqrestore(&qhp->rhp->lock, flags);
+ xa_unlock_irqrestore(&qhp->rhp->qps, flags);
return 0;
}
qhp->attr.layer_etype = attrs->layer_etype;
qhp->attr.ecode = attrs->ecode;
ep = qhp->ep;
+ c4iw_get_ep(&ep->com);
+ disconnect = 1;
if (!internal) {
- c4iw_get_ep(&qhp->ep->com);
terminate = 1;
- disconnect = 1;
} else {
terminate = qhp->attr.send_term;
ret = rdma_fini(rhp, qhp, ep);
return ret;
}
-int c4iw_destroy_qp(struct ib_qp *ib_qp)
+int c4iw_destroy_qp(struct ib_qp *ib_qp, struct ib_udata *udata)
{
struct c4iw_dev *rhp;
struct c4iw_qp *qhp;
c4iw_modify_qp(rhp, qhp, C4IW_QP_ATTR_NEXT_STATE, &attrs, 0);
wait_event(qhp->wait, !qhp->ep);
- remove_handle(rhp, &rhp->qpidr, qhp->wq.sq.qid);
-
- spin_lock_irq(&rhp->lock);
+ xa_lock_irq(&rhp->qps);
+ __xa_erase(&rhp->qps, qhp->wq.sq.qid);
if (!list_empty(&qhp->db_fc_entry))
list_del_init(&qhp->db_fc_entry);
- spin_unlock_irq(&rhp->lock);
+ xa_unlock_irq(&rhp->qps);
free_ird(rhp, qhp->attr.max_ird);
c4iw_qp_rem_ref(ib_qp);
kref_init(&qhp->kref);
INIT_WORK(&qhp->free_work, free_qp_work);
- ret = insert_handle(rhp, &rhp->qpidr, qhp, qhp->wq.sq.qid);
+ ret = xa_insert_irq(&rhp->qps, qhp->wq.sq.qid, qhp, GFP_KERNEL);
if (ret)
goto err_destroy_qp;
err_free_sq_key:
kfree(sq_key_mm);
err_remove_handle:
- remove_handle(rhp, &rhp->qpidr, qhp->wq.sq.qid);
+ xa_erase_irq(&rhp->qps, qhp->wq.sq.qid);
err_destroy_qp:
destroy_qp(&rhp->rdev, &qhp->wq,
ucontext ? &ucontext->uctx : &rhp->rdev.uctx, !attrs->srq);
}
}
-struct ib_srq *c4iw_create_srq(struct ib_pd *pd, struct ib_srq_init_attr *attrs,
+int c4iw_create_srq(struct ib_srq *ib_srq, struct ib_srq_init_attr *attrs,
struct ib_udata *udata)
{
+ struct ib_pd *pd = ib_srq->pd;
struct c4iw_dev *rhp;
- struct c4iw_srq *srq;
+ struct c4iw_srq *srq = to_c4iw_srq(ib_srq);
struct c4iw_pd *php;
struct c4iw_create_srq_resp uresp;
struct c4iw_ucontext *ucontext;
rhp = php->rhp;
if (!rhp->rdev.lldi.vr->srq.size)
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
if (attrs->attr.max_wr > rhp->rdev.hw_queue.t4_max_rq_size)
- return ERR_PTR(-E2BIG);
+ return -E2BIG;
if (attrs->attr.max_sge > T4_MAX_RECV_SGE)
- return ERR_PTR(-E2BIG);
+ return -E2BIG;
/*
* SRQ RQT and RQ must be a power of 2 and at least 16 deep.
ucontext = rdma_udata_to_drv_context(udata, struct c4iw_ucontext,
ibucontext);
- srq = kzalloc(sizeof(*srq), GFP_KERNEL);
- if (!srq)
- return ERR_PTR(-ENOMEM);
-
srq->wr_waitp = c4iw_alloc_wr_wait(GFP_KERNEL);
- if (!srq->wr_waitp) {
- ret = -ENOMEM;
- goto err_free_srq;
- }
+ if (!srq->wr_waitp)
+ return -ENOMEM;
srq->idx = c4iw_alloc_srq_idx(&rhp->rdev);
if (srq->idx < 0) {
if (CHELSIO_CHIP_VERSION(rhp->rdev.lldi.adapter_type) > CHELSIO_T6)
srq->flags = T4_SRQ_LIMIT_SUPPORT;
- ret = insert_handle(rhp, &rhp->qpidr, srq, srq->wq.qid);
+ ret = xa_insert_irq(&rhp->qps, srq->wq.qid, srq, GFP_KERNEL);
if (ret)
goto err_free_queue;
(unsigned long)srq->wq.memsize, attrs->attr.max_wr);
spin_lock_init(&srq->lock);
- return &srq->ibsrq;
+ return 0;
+
err_free_srq_db_key_mm:
kfree(srq_db_key_mm);
err_free_srq_key_mm:
kfree(srq_key_mm);
err_remove_handle:
- remove_handle(rhp, &rhp->qpidr, srq->wq.qid);
+ xa_erase_irq(&rhp->qps, srq->wq.qid);
err_free_queue:
free_srq_queue(srq, ucontext ? &ucontext->uctx : &rhp->rdev.uctx,
srq->wr_waitp);
c4iw_free_srq_idx(&rhp->rdev, srq->idx);
err_free_wr_wait:
c4iw_put_wr_wait(srq->wr_waitp);
-err_free_srq:
- kfree(srq);
- return ERR_PTR(ret);
+ return ret;
}
-int c4iw_destroy_srq(struct ib_srq *ibsrq)
+void c4iw_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)
{
struct c4iw_dev *rhp;
struct c4iw_srq *srq;
pr_debug("%s id %d\n", __func__, srq->wq.qid);
- remove_handle(rhp, &rhp->qpidr, srq->wq.qid);
- ucontext = ibsrq->uobject ?
- to_c4iw_ucontext(ibsrq->uobject->context) : NULL;
+ xa_erase_irq(&rhp->qps, srq->wq.qid);
+ ucontext = rdma_udata_to_drv_context(udata, struct c4iw_ucontext,
+ ibucontext);
free_srq_queue(srq, ucontext ? &ucontext->uctx : &rhp->rdev.uctx,
srq->wr_waitp);
c4iw_free_srq_idx(&rhp->rdev, srq->idx);
c4iw_put_wr_wait(srq->wr_waitp);
- kfree(srq);
- return 0;
}
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+# Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+#
+# Amazon fabric device configuration
+#
+
+config INFINIBAND_EFA
+ tristate "Amazon Elastic Fabric Adapter (EFA) support"
+ depends on PCI_MSI && 64BIT && !CPU_BIG_ENDIAN
+ depends on INFINIBAND_USER_ACCESS
+ help
+ This driver supports Amazon Elastic Fabric Adapter (EFA).
+
+ To compile this driver as a module, choose M here.
+ The module will be called efa.
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+# Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+#
+# Makefile for Amazon Elastic Fabric Adapter (EFA) device driver.
+#
+
+obj-$(CONFIG_INFINIBAND_EFA) += efa.o
+
+efa-y := efa_com_cmd.o efa_com.o efa_main.o efa_verbs.o
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#ifndef _EFA_H_
+#define _EFA_H_
+
+#include <linux/bitops.h>
+#include <linux/idr.h>
+#include <linux/interrupt.h>
+#include <linux/pci.h>
+#include <linux/sched.h>
+
+#include <rdma/efa-abi.h>
+#include <rdma/ib_verbs.h>
+
+#include "efa_com_cmd.h"
+
+#define DRV_MODULE_NAME "efa"
+#define DEVICE_NAME "Elastic Fabric Adapter (EFA)"
+
+#define EFA_IRQNAME_SIZE 40
+
+/* 1 for AENQ + ADMIN */
+#define EFA_NUM_MSIX_VEC 1
+#define EFA_MGMNT_MSIX_VEC_IDX 0
+
+struct efa_irq {
+ irq_handler_t handler;
+ void *data;
+ int cpu;
+ u32 vector;
+ cpumask_t affinity_hint_mask;
+ char name[EFA_IRQNAME_SIZE];
+};
+
+struct efa_sw_stats {
+ atomic64_t alloc_pd_err;
+ atomic64_t create_qp_err;
+ atomic64_t create_cq_err;
+ atomic64_t reg_mr_err;
+ atomic64_t alloc_ucontext_err;
+ atomic64_t create_ah_err;
+};
+
+/* Don't use anything other than atomic64 */
+struct efa_stats {
+ struct efa_sw_stats sw_stats;
+ atomic64_t keep_alive_rcvd;
+};
+
+struct efa_dev {
+ struct ib_device ibdev;
+ struct efa_com_dev edev;
+ struct pci_dev *pdev;
+ struct efa_com_get_device_attr_result dev_attr;
+
+ u64 reg_bar_addr;
+ u64 reg_bar_len;
+ u64 mem_bar_addr;
+ u64 mem_bar_len;
+ u64 db_bar_addr;
+ u64 db_bar_len;
+ u8 addr[EFA_GID_SIZE];
+ u32 mtu;
+
+ int admin_msix_vector_idx;
+ struct efa_irq admin_irq;
+
+ struct efa_stats stats;
+};
+
+struct efa_ucontext {
+ struct ib_ucontext ibucontext;
+ struct xarray mmap_xa;
+ u32 mmap_xa_page;
+ u16 uarn;
+};
+
+struct efa_pd {
+ struct ib_pd ibpd;
+ u16 pdn;
+};
+
+struct efa_mr {
+ struct ib_mr ibmr;
+ struct ib_umem *umem;
+};
+
+struct efa_cq {
+ struct ib_cq ibcq;
+ struct efa_ucontext *ucontext;
+ dma_addr_t dma_addr;
+ void *cpu_addr;
+ size_t size;
+ u16 cq_idx;
+};
+
+struct efa_qp {
+ struct ib_qp ibqp;
+ dma_addr_t rq_dma_addr;
+ void *rq_cpu_addr;
+ size_t rq_size;
+ enum ib_qp_state state;
+ u32 qp_handle;
+ u32 max_send_wr;
+ u32 max_recv_wr;
+ u32 max_send_sge;
+ u32 max_recv_sge;
+ u32 max_inline_data;
+};
+
+struct efa_ah {
+ struct ib_ah ibah;
+ u16 ah;
+ /* dest_addr */
+ u8 id[EFA_GID_SIZE];
+};
+
+int efa_query_device(struct ib_device *ibdev,
+ struct ib_device_attr *props,
+ struct ib_udata *udata);
+int efa_query_port(struct ib_device *ibdev, u8 port,
+ struct ib_port_attr *props);
+int efa_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
+ int qp_attr_mask,
+ struct ib_qp_init_attr *qp_init_attr);
+int efa_query_gid(struct ib_device *ibdev, u8 port, int index,
+ union ib_gid *gid);
+int efa_query_pkey(struct ib_device *ibdev, u8 port, u16 index,
+ u16 *pkey);
+int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata);
+void efa_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata);
+int efa_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata);
+struct ib_qp *efa_create_qp(struct ib_pd *ibpd,
+ struct ib_qp_init_attr *init_attr,
+ struct ib_udata *udata);
+int efa_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata);
+struct ib_cq *efa_create_cq(struct ib_device *ibdev,
+ const struct ib_cq_init_attr *attr,
+ struct ib_udata *udata);
+struct ib_mr *efa_reg_mr(struct ib_pd *ibpd, u64 start, u64 length,
+ u64 virt_addr, int access_flags,
+ struct ib_udata *udata);
+int efa_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata);
+int efa_get_port_immutable(struct ib_device *ibdev, u8 port_num,
+ struct ib_port_immutable *immutable);
+int efa_alloc_ucontext(struct ib_ucontext *ibucontext, struct ib_udata *udata);
+void efa_dealloc_ucontext(struct ib_ucontext *ibucontext);
+int efa_mmap(struct ib_ucontext *ibucontext,
+ struct vm_area_struct *vma);
+int efa_create_ah(struct ib_ah *ibah,
+ struct rdma_ah_attr *ah_attr,
+ u32 flags,
+ struct ib_udata *udata);
+void efa_destroy_ah(struct ib_ah *ibah, u32 flags);
+int efa_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
+ int qp_attr_mask, struct ib_udata *udata);
+enum rdma_link_layer efa_port_link_layer(struct ib_device *ibdev,
+ u8 port_num);
+
+#endif /* _EFA_H_ */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#ifndef _EFA_ADMIN_CMDS_H_
+#define _EFA_ADMIN_CMDS_H_
+
+#define EFA_ADMIN_API_VERSION_MAJOR 0
+#define EFA_ADMIN_API_VERSION_MINOR 1
+
+/* EFA admin queue opcodes */
+enum efa_admin_aq_opcode {
+ EFA_ADMIN_CREATE_QP = 1,
+ EFA_ADMIN_MODIFY_QP = 2,
+ EFA_ADMIN_QUERY_QP = 3,
+ EFA_ADMIN_DESTROY_QP = 4,
+ EFA_ADMIN_CREATE_AH = 5,
+ EFA_ADMIN_DESTROY_AH = 6,
+ EFA_ADMIN_REG_MR = 7,
+ EFA_ADMIN_DEREG_MR = 8,
+ EFA_ADMIN_CREATE_CQ = 9,
+ EFA_ADMIN_DESTROY_CQ = 10,
+ EFA_ADMIN_GET_FEATURE = 11,
+ EFA_ADMIN_SET_FEATURE = 12,
+ EFA_ADMIN_GET_STATS = 13,
+ EFA_ADMIN_ALLOC_PD = 14,
+ EFA_ADMIN_DEALLOC_PD = 15,
+ EFA_ADMIN_ALLOC_UAR = 16,
+ EFA_ADMIN_DEALLOC_UAR = 17,
+ EFA_ADMIN_MAX_OPCODE = 17,
+};
+
+enum efa_admin_aq_feature_id {
+ EFA_ADMIN_DEVICE_ATTR = 1,
+ EFA_ADMIN_AENQ_CONFIG = 2,
+ EFA_ADMIN_NETWORK_ATTR = 3,
+ EFA_ADMIN_QUEUE_ATTR = 4,
+ EFA_ADMIN_HW_HINTS = 5,
+ EFA_ADMIN_FEATURES_OPCODE_NUM = 8,
+};
+
+/* QP transport type */
+enum efa_admin_qp_type {
+ /* Unreliable Datagram */
+ EFA_ADMIN_QP_TYPE_UD = 1,
+ /* Scalable Reliable Datagram */
+ EFA_ADMIN_QP_TYPE_SRD = 2,
+};
+
+/* QP state */
+enum efa_admin_qp_state {
+ EFA_ADMIN_QP_STATE_RESET = 0,
+ EFA_ADMIN_QP_STATE_INIT = 1,
+ EFA_ADMIN_QP_STATE_RTR = 2,
+ EFA_ADMIN_QP_STATE_RTS = 3,
+ EFA_ADMIN_QP_STATE_SQD = 4,
+ EFA_ADMIN_QP_STATE_SQE = 5,
+ EFA_ADMIN_QP_STATE_ERR = 6,
+};
+
+enum efa_admin_get_stats_type {
+ EFA_ADMIN_GET_STATS_TYPE_BASIC = 0,
+};
+
+enum efa_admin_get_stats_scope {
+ EFA_ADMIN_GET_STATS_SCOPE_ALL = 0,
+ EFA_ADMIN_GET_STATS_SCOPE_QUEUE = 1,
+};
+
+enum efa_admin_modify_qp_mask_bits {
+ EFA_ADMIN_QP_STATE_BIT = 0,
+ EFA_ADMIN_CUR_QP_STATE_BIT = 1,
+ EFA_ADMIN_QKEY_BIT = 2,
+ EFA_ADMIN_SQ_PSN_BIT = 3,
+ EFA_ADMIN_SQ_DRAINED_ASYNC_NOTIFY_BIT = 4,
+};
+
+/*
+ * QP allocation sizes, converted by fabric QueuePair (QP) create command
+ * from QP capabilities.
+ */
+struct efa_admin_qp_alloc_size {
+ /* Send descriptor ring size in bytes */
+ u32 send_queue_ring_size;
+
+ /* Max number of WQEs that can be outstanding on send queue. */
+ u32 send_queue_depth;
+
+ /*
+ * Recv descriptor ring size in bytes, sufficient for user-provided
+ * number of WQEs
+ */
+ u32 recv_queue_ring_size;
+
+ /* Max number of WQEs that can be outstanding on recv queue */
+ u32 recv_queue_depth;
+};
+
+struct efa_admin_create_qp_cmd {
+ /* Common Admin Queue descriptor */
+ struct efa_admin_aq_common_desc aq_common_desc;
+
+ /* Protection Domain associated with this QP */
+ u16 pd;
+
+ /* QP type */
+ u8 qp_type;
+
+ /*
+ * 0 : sq_virt - If set, SQ ring base address is
+ * virtual (IOVA returned by MR registration)
+ * 1 : rq_virt - If set, RQ ring base address is
+ * virtual (IOVA returned by MR registration)
+ * 7:2 : reserved - MBZ
+ */
+ u8 flags;
+
+ /*
+ * Send queue (SQ) ring base physical address. This field is not
+ * used if this is a Low Latency Queue(LLQ).
+ */
+ u64 sq_base_addr;
+
+ /* Receive queue (RQ) ring base address. */
+ u64 rq_base_addr;
+
+ /* Index of CQ to be associated with Send Queue completions */
+ u32 send_cq_idx;
+
+ /* Index of CQ to be associated with Recv Queue completions */
+ u32 recv_cq_idx;
+
+ /*
+ * Memory registration key for the SQ ring, used only when not in
+ * LLQ mode and base address is virtual
+ */
+ u32 sq_l_key;
+
+ /*
+ * Memory registration key for the RQ ring, used only when base
+ * address is virtual
+ */
+ u32 rq_l_key;
+
+ /* Requested QP allocation sizes */
+ struct efa_admin_qp_alloc_size qp_alloc_size;
+
+ /* UAR number */
+ u16 uar;
+
+ /* MBZ */
+ u16 reserved;
+
+ /* MBZ */
+ u32 reserved2;
+};
+
+struct efa_admin_create_qp_resp {
+ /* Common Admin Queue completion descriptor */
+ struct efa_admin_acq_common_desc acq_common_desc;
+
+ /* Opaque handle to be used for consequent operations on the QP */
+ u32 qp_handle;
+
+ /* QP number in the given EFA virtual device */
+ u16 qp_num;
+
+ /* MBZ */
+ u16 reserved;
+
+ /* Index of sub-CQ for Send Queue completions */
+ u16 send_sub_cq_idx;
+
+ /* Index of sub-CQ for Receive Queue completions */
+ u16 recv_sub_cq_idx;
+
+ /* SQ doorbell address, as offset to PCIe DB BAR */
+ u32 sq_db_offset;
+
+ /* RQ doorbell address, as offset to PCIe DB BAR */
+ u32 rq_db_offset;
+
+ /*
+ * low latency send queue ring base address as an offset to PCIe
+ * MMIO LLQ_MEM BAR
+ */
+ u32 llq_descriptors_offset;
+};
+
+struct efa_admin_modify_qp_cmd {
+ /* Common Admin Queue descriptor */
+ struct efa_admin_aq_common_desc aq_common_desc;
+
+ /*
+ * Mask indicating which fields should be updated see enum
+ * efa_admin_modify_qp_mask_bits
+ */
+ u32 modify_mask;
+
+ /* QP handle returned by create_qp command */
+ u32 qp_handle;
+
+ /* QP state */
+ u32 qp_state;
+
+ /* Override current QP state (before applying the transition) */
+ u32 cur_qp_state;
+
+ /* QKey */
+ u32 qkey;
+
+ /* SQ PSN */
+ u32 sq_psn;
+
+ /* Enable async notification when SQ is drained */
+ u8 sq_drained_async_notify;
+
+ /* MBZ */
+ u8 reserved1;
+
+ /* MBZ */
+ u16 reserved2;
+};
+
+struct efa_admin_modify_qp_resp {
+ /* Common Admin Queue completion descriptor */
+ struct efa_admin_acq_common_desc acq_common_desc;
+};
+
+struct efa_admin_query_qp_cmd {
+ /* Common Admin Queue descriptor */
+ struct efa_admin_aq_common_desc aq_common_desc;
+
+ /* QP handle returned by create_qp command */
+ u32 qp_handle;
+};
+
+struct efa_admin_query_qp_resp {
+ /* Common Admin Queue completion descriptor */
+ struct efa_admin_acq_common_desc acq_common_desc;
+
+ /* QP state */
+ u32 qp_state;
+
+ /* QKey */
+ u32 qkey;
+
+ /* SQ PSN */
+ u32 sq_psn;
+
+ /* Indicates that draining is in progress */
+ u8 sq_draining;
+
+ /* MBZ */
+ u8 reserved1;
+
+ /* MBZ */
+ u16 reserved2;
+};
+
+struct efa_admin_destroy_qp_cmd {
+ /* Common Admin Queue descriptor */
+ struct efa_admin_aq_common_desc aq_common_desc;
+
+ /* QP handle returned by create_qp command */
+ u32 qp_handle;
+};
+
+struct efa_admin_destroy_qp_resp {
+ /* Common Admin Queue completion descriptor */
+ struct efa_admin_acq_common_desc acq_common_desc;
+};
+
+/*
+ * Create Address Handle command parameters. Must not be called more than
+ * once for the same destination
+ */
+struct efa_admin_create_ah_cmd {
+ /* Common Admin Queue descriptor */
+ struct efa_admin_aq_common_desc aq_common_desc;
+
+ /* Destination address in network byte order */
+ u8 dest_addr[16];
+
+ /* PD number */
+ u16 pd;
+
+ u16 reserved;
+};
+
+struct efa_admin_create_ah_resp {
+ /* Common Admin Queue completion descriptor */
+ struct efa_admin_acq_common_desc acq_common_desc;
+
+ /* Target interface address handle (opaque) */
+ u16 ah;
+
+ u16 reserved;
+};
+
+struct efa_admin_destroy_ah_cmd {
+ /* Common Admin Queue descriptor */
+ struct efa_admin_aq_common_desc aq_common_desc;
+
+ /* Target interface address handle (opaque) */
+ u16 ah;
+
+ /* PD number */
+ u16 pd;
+};
+
+struct efa_admin_destroy_ah_resp {
+ /* Common Admin Queue completion descriptor */
+ struct efa_admin_acq_common_desc acq_common_desc;
+};
+
+/*
+ * Registration of MemoryRegion, required for QP working with Virtual
+ * Addresses. In standard verbs semantics, region length is limited to 2GB
+ * space, but EFA offers larger MR support for large memory space, to ease
+ * on users working with very large datasets (i.e. full GPU memory mapping).
+ */
+struct efa_admin_reg_mr_cmd {
+ /* Common Admin Queue descriptor */
+ struct efa_admin_aq_common_desc aq_common_desc;
+
+ /* Protection Domain */
+ u16 pd;
+
+ /* MBZ */
+ u16 reserved16_w1;
+
+ /* Physical Buffer List, each element is page-aligned. */
+ union {
+ /*
+ * Inline array of guest-physical page addresses of user
+ * memory pages (optimization for short region
+ * registrations)
+ */
+ u64 inline_pbl_array[4];
+
+ /* points to PBL (direct or indirect, chained if needed) */
+ struct efa_admin_ctrl_buff_info pbl;
+ } pbl;
+
+ /* Memory region length, in bytes. */
+ u64 mr_length;
+
+ /*
+ * flags and page size
+ * 4:0 : phys_page_size_shift - page size is (1 <<
+ * phys_page_size_shift). Page size is used for
+ * building the Virtual to Physical address mapping
+ * 6:5 : reserved - MBZ
+ * 7 : mem_addr_phy_mode_en - Enable bit for physical
+ * memory registration (no translation), can be used
+ * only by privileged clients. If set, PBL must
+ * contain a single entry.
+ */
+ u8 flags;
+
+ /*
+ * permissions
+ * 0 : local_write_enable - Write permissions: value
+ * of 1 needed for RQ buffers and for RDMA write
+ * 7:1 : reserved1 - remote access flags, etc
+ */
+ u8 permissions;
+
+ u16 reserved16_w5;
+
+ /* number of pages in PBL (redundant, could be calculated) */
+ u32 page_num;
+
+ /*
+ * IO Virtual Address associated with this MR. If
+ * mem_addr_phy_mode_en is set, contains the physical address of
+ * the region.
+ */
+ u64 iova;
+};
+
+struct efa_admin_reg_mr_resp {
+ /* Common Admin Queue completion descriptor */
+ struct efa_admin_acq_common_desc acq_common_desc;
+
+ /*
+ * L_Key, to be used in conjunction with local buffer references in
+ * SQ and RQ WQE, or with virtual RQ/CQ rings
+ */
+ u32 l_key;
+
+ /*
+ * R_Key, to be used in RDMA messages to refer to remotely accessed
+ * memory region
+ */
+ u32 r_key;
+};
+
+struct efa_admin_dereg_mr_cmd {
+ /* Common Admin Queue descriptor */
+ struct efa_admin_aq_common_desc aq_common_desc;
+
+ /* L_Key, memory region's l_key */
+ u32 l_key;
+};
+
+struct efa_admin_dereg_mr_resp {
+ /* Common Admin Queue completion descriptor */
+ struct efa_admin_acq_common_desc acq_common_desc;
+};
+
+struct efa_admin_create_cq_cmd {
+ struct efa_admin_aq_common_desc aq_common_desc;
+
+ /*
+ * 4:0 : reserved5
+ * 5 : interrupt_mode_enabled - if set, cq operates
+ * in interrupt mode (i.e. CQ events and MSI-X are
+ * generated), otherwise - polling
+ * 6 : virt - If set, ring base address is virtual
+ * (IOVA returned by MR registration)
+ * 7 : reserved6
+ */
+ u8 cq_caps_1;
+
+ /*
+ * 4:0 : cq_entry_size_words - size of CQ entry in
+ * 32-bit words, valid values: 4, 8.
+ * 7:5 : reserved7
+ */
+ u8 cq_caps_2;
+
+ /* completion queue depth in # of entries. must be power of 2 */
+ u16 cq_depth;
+
+ /* msix vector assigned to this cq */
+ u32 msix_vector_idx;
+
+ /*
+ * CQ ring base address, virtual or physical depending on 'virt'
+ * flag
+ */
+ struct efa_common_mem_addr cq_ba;
+
+ /*
+ * Memory registration key for the ring, used only when base
+ * address is virtual
+ */
+ u32 l_key;
+
+ /*
+ * number of sub cqs - must be equal to sub_cqs_per_cq of queue
+ * attributes.
+ */
+ u16 num_sub_cqs;
+
+ /* UAR number */
+ u16 uar;
+};
+
+struct efa_admin_create_cq_resp {
+ struct efa_admin_acq_common_desc acq_common_desc;
+
+ u16 cq_idx;
+
+ /* actual cq depth in number of entries */
+ u16 cq_actual_depth;
+};
+
+struct efa_admin_destroy_cq_cmd {
+ struct efa_admin_aq_common_desc aq_common_desc;
+
+ u16 cq_idx;
+
+ u16 reserved1;
+};
+
+struct efa_admin_destroy_cq_resp {
+ struct efa_admin_acq_common_desc acq_common_desc;
+};
+
+/*
+ * EFA AQ Get Statistics command. Extended statistics are placed in control
+ * buffer pointed by AQ entry
+ */
+struct efa_admin_aq_get_stats_cmd {
+ struct efa_admin_aq_common_desc aq_common_descriptor;
+
+ union {
+ /* command specific inline data */
+ u32 inline_data_w1[3];
+
+ struct efa_admin_ctrl_buff_info control_buffer;
+ } u;
+
+ /* stats type as defined in enum efa_admin_get_stats_type */
+ u8 type;
+
+ /* stats scope defined in enum efa_admin_get_stats_scope */
+ u8 scope;
+
+ u16 scope_modifier;
+};
+
+struct efa_admin_basic_stats {
+ u64 tx_bytes;
+
+ u64 tx_pkts;
+
+ u64 rx_bytes;
+
+ u64 rx_pkts;
+
+ u64 rx_drops;
+};
+
+struct efa_admin_acq_get_stats_resp {
+ struct efa_admin_acq_common_desc acq_common_desc;
+
+ struct efa_admin_basic_stats basic_stats;
+};
+
+struct efa_admin_get_set_feature_common_desc {
+ /*
+ * 1:0 : select - 0x1 - current value; 0x3 - default
+ * value
+ * 7:3 : reserved3
+ */
+ u8 flags;
+
+ /* as appears in efa_admin_aq_feature_id */
+ u8 feature_id;
+
+ /* MBZ */
+ u16 reserved16;
+};
+
+struct efa_admin_feature_device_attr_desc {
+ /* Bitmap of efa_admin_aq_feature_id */
+ u64 supported_features;
+
+ /* Bitmap of supported page sizes in MR registrations */
+ u64 page_size_cap;
+
+ u32 fw_version;
+
+ u32 admin_api_version;
+
+ u32 device_version;
+
+ /* Bar used for SQ and RQ doorbells */
+ u16 db_bar;
+
+ /* Indicates how many bits are used physical address access */
+ u8 phys_addr_width;
+
+ /* Indicates how many bits are used virtual address access */
+ u8 virt_addr_width;
+};
+
+struct efa_admin_feature_queue_attr_desc {
+ /* The maximum number of queue pairs supported */
+ u32 max_qp;
+
+ u32 max_sq_depth;
+
+ /* max send wr used in inline-buf */
+ u32 inline_buf_size;
+
+ u32 max_rq_depth;
+
+ /* The maximum number of completion queues supported per VF */
+ u32 max_cq;
+
+ u32 max_cq_depth;
+
+ /* Number of sub-CQs to be created for each CQ */
+ u16 sub_cqs_per_cq;
+
+ u16 reserved;
+
+ /*
+ * Maximum number of SGEs (buffs) allowed for a single send work
+ * queue element (WQE)
+ */
+ u16 max_wr_send_sges;
+
+ /* Maximum number of SGEs allowed for a single recv WQE */
+ u16 max_wr_recv_sges;
+
+ /* The maximum number of memory regions supported */
+ u32 max_mr;
+
+ /* The maximum number of pages can be registered */
+ u32 max_mr_pages;
+
+ /* The maximum number of protection domains supported */
+ u32 max_pd;
+
+ /* The maximum number of address handles supported */
+ u32 max_ah;
+
+ /* The maximum size of LLQ in bytes */
+ u32 max_llq_size;
+};
+
+struct efa_admin_feature_aenq_desc {
+ /* bitmask for AENQ groups the device can report */
+ u32 supported_groups;
+
+ /* bitmask for AENQ groups to report */
+ u32 enabled_groups;
+};
+
+struct efa_admin_feature_network_attr_desc {
+ /* Raw address data in network byte order */
+ u8 addr[16];
+
+ u32 mtu;
+};
+
+/*
+ * When hint value is 0, hints capabilities are not supported or driver
+ * should use its own predefined value
+ */
+struct efa_admin_hw_hints {
+ /* value in ms */
+ u16 mmio_read_timeout;
+
+ /* value in ms */
+ u16 driver_watchdog_timeout;
+
+ /* value in ms */
+ u16 admin_completion_timeout;
+
+ /* poll interval in ms */
+ u16 poll_interval;
+};
+
+struct efa_admin_get_feature_cmd {
+ struct efa_admin_aq_common_desc aq_common_descriptor;
+
+ struct efa_admin_ctrl_buff_info control_buffer;
+
+ struct efa_admin_get_set_feature_common_desc feature_common;
+
+ u32 raw[11];
+};
+
+struct efa_admin_get_feature_resp {
+ struct efa_admin_acq_common_desc acq_common_desc;
+
+ union {
+ u32 raw[14];
+
+ struct efa_admin_feature_device_attr_desc device_attr;
+
+ struct efa_admin_feature_aenq_desc aenq;
+
+ struct efa_admin_feature_network_attr_desc network_attr;
+
+ struct efa_admin_feature_queue_attr_desc queue_attr;
+
+ struct efa_admin_hw_hints hw_hints;
+ } u;
+};
+
+struct efa_admin_set_feature_cmd {
+ struct efa_admin_aq_common_desc aq_common_descriptor;
+
+ struct efa_admin_ctrl_buff_info control_buffer;
+
+ struct efa_admin_get_set_feature_common_desc feature_common;
+
+ union {
+ u32 raw[11];
+
+ /* AENQ configuration */
+ struct efa_admin_feature_aenq_desc aenq;
+ } u;
+};
+
+struct efa_admin_set_feature_resp {
+ struct efa_admin_acq_common_desc acq_common_desc;
+
+ union {
+ u32 raw[14];
+ } u;
+};
+
+struct efa_admin_alloc_pd_cmd {
+ struct efa_admin_aq_common_desc aq_common_descriptor;
+};
+
+struct efa_admin_alloc_pd_resp {
+ struct efa_admin_acq_common_desc acq_common_desc;
+
+ /* PD number */
+ u16 pd;
+
+ /* MBZ */
+ u16 reserved;
+};
+
+struct efa_admin_dealloc_pd_cmd {
+ struct efa_admin_aq_common_desc aq_common_descriptor;
+
+ /* PD number */
+ u16 pd;
+
+ /* MBZ */
+ u16 reserved;
+};
+
+struct efa_admin_dealloc_pd_resp {
+ struct efa_admin_acq_common_desc acq_common_desc;
+};
+
+struct efa_admin_alloc_uar_cmd {
+ struct efa_admin_aq_common_desc aq_common_descriptor;
+};
+
+struct efa_admin_alloc_uar_resp {
+ struct efa_admin_acq_common_desc acq_common_desc;
+
+ /* UAR number */
+ u16 uar;
+
+ /* MBZ */
+ u16 reserved;
+};
+
+struct efa_admin_dealloc_uar_cmd {
+ struct efa_admin_aq_common_desc aq_common_descriptor;
+
+ /* UAR number */
+ u16 uar;
+
+ /* MBZ */
+ u16 reserved;
+};
+
+struct efa_admin_dealloc_uar_resp {
+ struct efa_admin_acq_common_desc acq_common_desc;
+};
+
+/* asynchronous event notification groups */
+enum efa_admin_aenq_group {
+ EFA_ADMIN_FATAL_ERROR = 1,
+ EFA_ADMIN_WARNING = 2,
+ EFA_ADMIN_NOTIFICATION = 3,
+ EFA_ADMIN_KEEP_ALIVE = 4,
+ EFA_ADMIN_AENQ_GROUPS_NUM = 5,
+};
+
+enum efa_admin_aenq_notification_syndrom {
+ EFA_ADMIN_SUSPEND = 0,
+ EFA_ADMIN_RESUME = 1,
+ EFA_ADMIN_UPDATE_HINTS = 2,
+};
+
+struct efa_admin_mmio_req_read_less_resp {
+ u16 req_id;
+
+ u16 reg_off;
+
+ /* value is valid when poll is cleared */
+ u32 reg_val;
+};
+
+/* create_qp_cmd */
+#define EFA_ADMIN_CREATE_QP_CMD_SQ_VIRT_MASK BIT(0)
+#define EFA_ADMIN_CREATE_QP_CMD_RQ_VIRT_SHIFT 1
+#define EFA_ADMIN_CREATE_QP_CMD_RQ_VIRT_MASK BIT(1)
+
+/* reg_mr_cmd */
+#define EFA_ADMIN_REG_MR_CMD_PHYS_PAGE_SIZE_SHIFT_MASK GENMASK(4, 0)
+#define EFA_ADMIN_REG_MR_CMD_MEM_ADDR_PHY_MODE_EN_SHIFT 7
+#define EFA_ADMIN_REG_MR_CMD_MEM_ADDR_PHY_MODE_EN_MASK BIT(7)
+#define EFA_ADMIN_REG_MR_CMD_LOCAL_WRITE_ENABLE_MASK BIT(0)
+
+/* create_cq_cmd */
+#define EFA_ADMIN_CREATE_CQ_CMD_INTERRUPT_MODE_ENABLED_SHIFT 5
+#define EFA_ADMIN_CREATE_CQ_CMD_INTERRUPT_MODE_ENABLED_MASK BIT(5)
+#define EFA_ADMIN_CREATE_CQ_CMD_VIRT_SHIFT 6
+#define EFA_ADMIN_CREATE_CQ_CMD_VIRT_MASK BIT(6)
+#define EFA_ADMIN_CREATE_CQ_CMD_CQ_ENTRY_SIZE_WORDS_MASK GENMASK(4, 0)
+
+/* get_set_feature_common_desc */
+#define EFA_ADMIN_GET_SET_FEATURE_COMMON_DESC_SELECT_MASK GENMASK(1, 0)
+
+#endif /* _EFA_ADMIN_CMDS_H_ */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#ifndef _EFA_ADMIN_H_
+#define _EFA_ADMIN_H_
+
+enum efa_admin_aq_completion_status {
+ EFA_ADMIN_SUCCESS = 0,
+ EFA_ADMIN_RESOURCE_ALLOCATION_FAILURE = 1,
+ EFA_ADMIN_BAD_OPCODE = 2,
+ EFA_ADMIN_UNSUPPORTED_OPCODE = 3,
+ EFA_ADMIN_MALFORMED_REQUEST = 4,
+ /* Additional status is provided in ACQ entry extended_status */
+ EFA_ADMIN_ILLEGAL_PARAMETER = 5,
+ EFA_ADMIN_UNKNOWN_ERROR = 6,
+ EFA_ADMIN_RESOURCE_BUSY = 7,
+};
+
+struct efa_admin_aq_common_desc {
+ /*
+ * 11:0 : command_id
+ * 15:12 : reserved12
+ */
+ u16 command_id;
+
+ /* as appears in efa_admin_aq_opcode */
+ u8 opcode;
+
+ /*
+ * 0 : phase
+ * 1 : ctrl_data - control buffer address valid
+ * 2 : ctrl_data_indirect - control buffer address
+ * points to list of pages with addresses of control
+ * buffers
+ * 7:3 : reserved3
+ */
+ u8 flags;
+};
+
+/*
+ * used in efa_admin_aq_entry. Can point directly to control data, or to a
+ * page list chunk. Used also at the end of indirect mode page list chunks,
+ * for chaining.
+ */
+struct efa_admin_ctrl_buff_info {
+ u32 length;
+
+ struct efa_common_mem_addr address;
+};
+
+struct efa_admin_aq_entry {
+ struct efa_admin_aq_common_desc aq_common_descriptor;
+
+ union {
+ u32 inline_data_w1[3];
+
+ struct efa_admin_ctrl_buff_info control_buffer;
+ } u;
+
+ u32 inline_data_w4[12];
+};
+
+struct efa_admin_acq_common_desc {
+ /*
+ * command identifier to associate it with the aq descriptor
+ * 11:0 : command_id
+ * 15:12 : reserved12
+ */
+ u16 command;
+
+ u8 status;
+
+ /*
+ * 0 : phase
+ * 7:1 : reserved1
+ */
+ u8 flags;
+
+ u16 extended_status;
+
+ /*
+ * indicates to the driver which AQ entry has been consumed by the
+ * device and could be reused
+ */
+ u16 sq_head_indx;
+};
+
+struct efa_admin_acq_entry {
+ struct efa_admin_acq_common_desc acq_common_descriptor;
+
+ u32 response_specific_data[14];
+};
+
+struct efa_admin_aenq_common_desc {
+ u16 group;
+
+ u16 syndrom;
+
+ /*
+ * 0 : phase
+ * 7:1 : reserved - MBZ
+ */
+ u8 flags;
+
+ u8 reserved1[3];
+
+ u32 timestamp_low;
+
+ u32 timestamp_high;
+};
+
+struct efa_admin_aenq_entry {
+ struct efa_admin_aenq_common_desc aenq_common_desc;
+
+ /* command specific inline data */
+ u32 inline_data_w4[12];
+};
+
+/* aq_common_desc */
+#define EFA_ADMIN_AQ_COMMON_DESC_COMMAND_ID_MASK GENMASK(11, 0)
+#define EFA_ADMIN_AQ_COMMON_DESC_PHASE_MASK BIT(0)
+#define EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA_SHIFT 1
+#define EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA_MASK BIT(1)
+#define EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA_INDIRECT_SHIFT 2
+#define EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA_INDIRECT_MASK BIT(2)
+
+/* acq_common_desc */
+#define EFA_ADMIN_ACQ_COMMON_DESC_COMMAND_ID_MASK GENMASK(11, 0)
+#define EFA_ADMIN_ACQ_COMMON_DESC_PHASE_MASK BIT(0)
+
+/* aenq_common_desc */
+#define EFA_ADMIN_AENQ_COMMON_DESC_PHASE_MASK BIT(0)
+
+#endif /* _EFA_ADMIN_H_ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#include "efa_com.h"
+#include "efa_regs_defs.h"
+
+#define ADMIN_CMD_TIMEOUT_US 30000000 /* usecs */
+
+#define EFA_REG_READ_TIMEOUT_US 50000 /* usecs */
+#define EFA_MMIO_READ_INVALID 0xffffffff
+
+#define EFA_POLL_INTERVAL_MS 100 /* msecs */
+
+#define EFA_ASYNC_QUEUE_DEPTH 16
+#define EFA_ADMIN_QUEUE_DEPTH 32
+
+#define MIN_EFA_VER\
+ ((EFA_ADMIN_API_VERSION_MAJOR << EFA_REGS_VERSION_MAJOR_VERSION_SHIFT) | \
+ (EFA_ADMIN_API_VERSION_MINOR & EFA_REGS_VERSION_MINOR_VERSION_MASK))
+
+#define EFA_CTRL_MAJOR 0
+#define EFA_CTRL_MINOR 0
+#define EFA_CTRL_SUB_MINOR 1
+
+#define MIN_EFA_CTRL_VER \
+ (((EFA_CTRL_MAJOR) << \
+ (EFA_REGS_CONTROLLER_VERSION_MAJOR_VERSION_SHIFT)) | \
+ ((EFA_CTRL_MINOR) << \
+ (EFA_REGS_CONTROLLER_VERSION_MINOR_VERSION_SHIFT)) | \
+ (EFA_CTRL_SUB_MINOR))
+
+#define EFA_DMA_ADDR_TO_UINT32_LOW(x) ((u32)((u64)(x)))
+#define EFA_DMA_ADDR_TO_UINT32_HIGH(x) ((u32)(((u64)(x)) >> 32))
+
+#define EFA_REGS_ADMIN_INTR_MASK 1
+
+enum efa_cmd_status {
+ EFA_CMD_SUBMITTED,
+ EFA_CMD_COMPLETED,
+ /* Abort - canceled by the driver */
+ EFA_CMD_ABORTED,
+};
+
+struct efa_comp_ctx {
+ struct completion wait_event;
+ struct efa_admin_acq_entry *user_cqe;
+ u32 comp_size;
+ enum efa_cmd_status status;
+ /* status from the device */
+ u8 comp_status;
+ u8 cmd_opcode;
+ u8 occupied;
+};
+
+static const char *efa_com_cmd_str(u8 cmd)
+{
+#define EFA_CMD_STR_CASE(_cmd) case EFA_ADMIN_##_cmd: return #_cmd
+
+ switch (cmd) {
+ EFA_CMD_STR_CASE(CREATE_QP);
+ EFA_CMD_STR_CASE(MODIFY_QP);
+ EFA_CMD_STR_CASE(QUERY_QP);
+ EFA_CMD_STR_CASE(DESTROY_QP);
+ EFA_CMD_STR_CASE(CREATE_AH);
+ EFA_CMD_STR_CASE(DESTROY_AH);
+ EFA_CMD_STR_CASE(REG_MR);
+ EFA_CMD_STR_CASE(DEREG_MR);
+ EFA_CMD_STR_CASE(CREATE_CQ);
+ EFA_CMD_STR_CASE(DESTROY_CQ);
+ EFA_CMD_STR_CASE(GET_FEATURE);
+ EFA_CMD_STR_CASE(SET_FEATURE);
+ EFA_CMD_STR_CASE(GET_STATS);
+ EFA_CMD_STR_CASE(ALLOC_PD);
+ EFA_CMD_STR_CASE(DEALLOC_PD);
+ EFA_CMD_STR_CASE(ALLOC_UAR);
+ EFA_CMD_STR_CASE(DEALLOC_UAR);
+ default: return "unknown command opcode";
+ }
+#undef EFA_CMD_STR_CASE
+}
+
+static u32 efa_com_reg_read32(struct efa_com_dev *edev, u16 offset)
+{
+ struct efa_com_mmio_read *mmio_read = &edev->mmio_read;
+ struct efa_admin_mmio_req_read_less_resp *read_resp;
+ unsigned long exp_time;
+ u32 mmio_read_reg;
+ u32 err;
+
+ read_resp = mmio_read->read_resp;
+
+ spin_lock(&mmio_read->lock);
+ mmio_read->seq_num++;
+
+ /* trash DMA req_id to identify when hardware is done */
+ read_resp->req_id = mmio_read->seq_num + 0x9aL;
+ mmio_read_reg = (offset << EFA_REGS_MMIO_REG_READ_REG_OFF_SHIFT) &
+ EFA_REGS_MMIO_REG_READ_REG_OFF_MASK;
+ mmio_read_reg |= mmio_read->seq_num &
+ EFA_REGS_MMIO_REG_READ_REQ_ID_MASK;
+
+ writel(mmio_read_reg, edev->reg_bar + EFA_REGS_MMIO_REG_READ_OFF);
+
+ exp_time = jiffies + usecs_to_jiffies(mmio_read->mmio_read_timeout);
+ do {
+ if (READ_ONCE(read_resp->req_id) == mmio_read->seq_num)
+ break;
+ udelay(1);
+ } while (time_is_after_jiffies(exp_time));
+
+ if (read_resp->req_id != mmio_read->seq_num) {
+ ibdev_err(edev->efa_dev,
+ "Reading register timed out. expected: req id[%u] offset[%#x] actual: req id[%u] offset[%#x]\n",
+ mmio_read->seq_num, offset, read_resp->req_id,
+ read_resp->reg_off);
+ err = EFA_MMIO_READ_INVALID;
+ goto out;
+ }
+
+ if (read_resp->reg_off != offset) {
+ ibdev_err(edev->efa_dev,
+ "Reading register failed: wrong offset provided\n");
+ err = EFA_MMIO_READ_INVALID;
+ goto out;
+ }
+
+ err = read_resp->reg_val;
+out:
+ spin_unlock(&mmio_read->lock);
+ return err;
+}
+
+static int efa_com_admin_init_sq(struct efa_com_dev *edev)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_com_admin_sq *sq = &aq->sq;
+ u16 size = aq->depth * sizeof(*sq->entries);
+ u32 addr_high;
+ u32 addr_low;
+ u32 aq_caps;
+
+ sq->entries =
+ dma_alloc_coherent(aq->dmadev, size, &sq->dma_addr, GFP_KERNEL);
+ if (!sq->entries)
+ return -ENOMEM;
+
+ spin_lock_init(&sq->lock);
+
+ sq->cc = 0;
+ sq->pc = 0;
+ sq->phase = 1;
+
+ sq->db_addr = (u32 __iomem *)(edev->reg_bar + EFA_REGS_AQ_PROD_DB_OFF);
+
+ addr_high = EFA_DMA_ADDR_TO_UINT32_HIGH(sq->dma_addr);
+ addr_low = EFA_DMA_ADDR_TO_UINT32_LOW(sq->dma_addr);
+
+ writel(addr_low, edev->reg_bar + EFA_REGS_AQ_BASE_LO_OFF);
+ writel(addr_high, edev->reg_bar + EFA_REGS_AQ_BASE_HI_OFF);
+
+ aq_caps = aq->depth & EFA_REGS_AQ_CAPS_AQ_DEPTH_MASK;
+ aq_caps |= (sizeof(struct efa_admin_aq_entry) <<
+ EFA_REGS_AQ_CAPS_AQ_ENTRY_SIZE_SHIFT) &
+ EFA_REGS_AQ_CAPS_AQ_ENTRY_SIZE_MASK;
+
+ writel(aq_caps, edev->reg_bar + EFA_REGS_AQ_CAPS_OFF);
+
+ return 0;
+}
+
+static int efa_com_admin_init_cq(struct efa_com_dev *edev)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_com_admin_cq *cq = &aq->cq;
+ u16 size = aq->depth * sizeof(*cq->entries);
+ u32 addr_high;
+ u32 addr_low;
+ u32 acq_caps;
+
+ cq->entries =
+ dma_alloc_coherent(aq->dmadev, size, &cq->dma_addr, GFP_KERNEL);
+ if (!cq->entries)
+ return -ENOMEM;
+
+ spin_lock_init(&cq->lock);
+
+ cq->cc = 0;
+ cq->phase = 1;
+
+ addr_high = EFA_DMA_ADDR_TO_UINT32_HIGH(cq->dma_addr);
+ addr_low = EFA_DMA_ADDR_TO_UINT32_LOW(cq->dma_addr);
+
+ writel(addr_low, edev->reg_bar + EFA_REGS_ACQ_BASE_LO_OFF);
+ writel(addr_high, edev->reg_bar + EFA_REGS_ACQ_BASE_HI_OFF);
+
+ acq_caps = aq->depth & EFA_REGS_ACQ_CAPS_ACQ_DEPTH_MASK;
+ acq_caps |= (sizeof(struct efa_admin_acq_entry) <<
+ EFA_REGS_ACQ_CAPS_ACQ_ENTRY_SIZE_SHIFT) &
+ EFA_REGS_ACQ_CAPS_ACQ_ENTRY_SIZE_MASK;
+ acq_caps |= (aq->msix_vector_idx <<
+ EFA_REGS_ACQ_CAPS_ACQ_MSIX_VECTOR_SHIFT) &
+ EFA_REGS_ACQ_CAPS_ACQ_MSIX_VECTOR_MASK;
+
+ writel(acq_caps, edev->reg_bar + EFA_REGS_ACQ_CAPS_OFF);
+
+ return 0;
+}
+
+static int efa_com_admin_init_aenq(struct efa_com_dev *edev,
+ struct efa_aenq_handlers *aenq_handlers)
+{
+ struct efa_com_aenq *aenq = &edev->aenq;
+ u32 addr_low, addr_high, aenq_caps;
+ u16 size;
+
+ if (!aenq_handlers) {
+ ibdev_err(edev->efa_dev, "aenq handlers pointer is NULL\n");
+ return -EINVAL;
+ }
+
+ size = EFA_ASYNC_QUEUE_DEPTH * sizeof(*aenq->entries);
+ aenq->entries = dma_alloc_coherent(edev->dmadev, size, &aenq->dma_addr,
+ GFP_KERNEL);
+ if (!aenq->entries)
+ return -ENOMEM;
+
+ aenq->aenq_handlers = aenq_handlers;
+ aenq->depth = EFA_ASYNC_QUEUE_DEPTH;
+ aenq->cc = 0;
+ aenq->phase = 1;
+
+ addr_low = EFA_DMA_ADDR_TO_UINT32_LOW(aenq->dma_addr);
+ addr_high = EFA_DMA_ADDR_TO_UINT32_HIGH(aenq->dma_addr);
+
+ writel(addr_low, edev->reg_bar + EFA_REGS_AENQ_BASE_LO_OFF);
+ writel(addr_high, edev->reg_bar + EFA_REGS_AENQ_BASE_HI_OFF);
+
+ aenq_caps = aenq->depth & EFA_REGS_AENQ_CAPS_AENQ_DEPTH_MASK;
+ aenq_caps |= (sizeof(struct efa_admin_aenq_entry) <<
+ EFA_REGS_AENQ_CAPS_AENQ_ENTRY_SIZE_SHIFT) &
+ EFA_REGS_AENQ_CAPS_AENQ_ENTRY_SIZE_MASK;
+ aenq_caps |= (aenq->msix_vector_idx
+ << EFA_REGS_AENQ_CAPS_AENQ_MSIX_VECTOR_SHIFT) &
+ EFA_REGS_AENQ_CAPS_AENQ_MSIX_VECTOR_MASK;
+ writel(aenq_caps, edev->reg_bar + EFA_REGS_AENQ_CAPS_OFF);
+
+ /*
+ * Init cons_db to mark that all entries in the queue
+ * are initially available
+ */
+ writel(edev->aenq.cc, edev->reg_bar + EFA_REGS_AENQ_CONS_DB_OFF);
+
+ return 0;
+}
+
+/* ID to be used with efa_com_get_comp_ctx */
+static u16 efa_com_alloc_ctx_id(struct efa_com_admin_queue *aq)
+{
+ u16 ctx_id;
+
+ spin_lock(&aq->comp_ctx_lock);
+ ctx_id = aq->comp_ctx_pool[aq->comp_ctx_pool_next];
+ aq->comp_ctx_pool_next++;
+ spin_unlock(&aq->comp_ctx_lock);
+
+ return ctx_id;
+}
+
+static void efa_com_dealloc_ctx_id(struct efa_com_admin_queue *aq,
+ u16 ctx_id)
+{
+ spin_lock(&aq->comp_ctx_lock);
+ aq->comp_ctx_pool_next--;
+ aq->comp_ctx_pool[aq->comp_ctx_pool_next] = ctx_id;
+ spin_unlock(&aq->comp_ctx_lock);
+}
+
+static inline void efa_com_put_comp_ctx(struct efa_com_admin_queue *aq,
+ struct efa_comp_ctx *comp_ctx)
+{
+ u16 comp_id = comp_ctx->user_cqe->acq_common_descriptor.command &
+ EFA_ADMIN_ACQ_COMMON_DESC_COMMAND_ID_MASK;
+
+ ibdev_dbg(aq->efa_dev, "Putting completion command_id %d\n", comp_id);
+ comp_ctx->occupied = 0;
+ efa_com_dealloc_ctx_id(aq, comp_id);
+}
+
+static struct efa_comp_ctx *efa_com_get_comp_ctx(struct efa_com_admin_queue *aq,
+ u16 command_id, bool capture)
+{
+ if (command_id >= aq->depth) {
+ ibdev_err(aq->efa_dev,
+ "command id is larger than the queue size. cmd_id: %u queue size %d\n",
+ command_id, aq->depth);
+ return NULL;
+ }
+
+ if (aq->comp_ctx[command_id].occupied && capture) {
+ ibdev_err(aq->efa_dev, "Completion context is occupied\n");
+ return NULL;
+ }
+
+ if (capture) {
+ aq->comp_ctx[command_id].occupied = 1;
+ ibdev_dbg(aq->efa_dev, "Taking completion ctxt command_id %d\n",
+ command_id);
+ }
+
+ return &aq->comp_ctx[command_id];
+}
+
+static struct efa_comp_ctx *__efa_com_submit_admin_cmd(struct efa_com_admin_queue *aq,
+ struct efa_admin_aq_entry *cmd,
+ size_t cmd_size_in_bytes,
+ struct efa_admin_acq_entry *comp,
+ size_t comp_size_in_bytes)
+{
+ struct efa_comp_ctx *comp_ctx;
+ u16 queue_size_mask;
+ u16 ctx_id;
+ u16 pi;
+
+ queue_size_mask = aq->depth - 1;
+ pi = aq->sq.pc & queue_size_mask;
+
+ ctx_id = efa_com_alloc_ctx_id(aq);
+
+ cmd->aq_common_descriptor.flags |= aq->sq.phase &
+ EFA_ADMIN_AQ_COMMON_DESC_PHASE_MASK;
+
+ cmd->aq_common_descriptor.command_id |= ctx_id &
+ EFA_ADMIN_AQ_COMMON_DESC_COMMAND_ID_MASK;
+
+ comp_ctx = efa_com_get_comp_ctx(aq, ctx_id, true);
+ if (!comp_ctx) {
+ efa_com_dealloc_ctx_id(aq, ctx_id);
+ return ERR_PTR(-EINVAL);
+ }
+
+ comp_ctx->status = EFA_CMD_SUBMITTED;
+ comp_ctx->comp_size = comp_size_in_bytes;
+ comp_ctx->user_cqe = comp;
+ comp_ctx->cmd_opcode = cmd->aq_common_descriptor.opcode;
+
+ reinit_completion(&comp_ctx->wait_event);
+
+ memcpy(&aq->sq.entries[pi], cmd, cmd_size_in_bytes);
+
+ aq->sq.pc++;
+ atomic64_inc(&aq->stats.submitted_cmd);
+
+ if ((aq->sq.pc & queue_size_mask) == 0)
+ aq->sq.phase = !aq->sq.phase;
+
+ /* barrier not needed in case of writel */
+ writel(aq->sq.pc, aq->sq.db_addr);
+
+ return comp_ctx;
+}
+
+static inline int efa_com_init_comp_ctxt(struct efa_com_admin_queue *aq)
+{
+ size_t pool_size = aq->depth * sizeof(*aq->comp_ctx_pool);
+ size_t size = aq->depth * sizeof(struct efa_comp_ctx);
+ struct efa_comp_ctx *comp_ctx;
+ u16 i;
+
+ aq->comp_ctx = devm_kzalloc(aq->dmadev, size, GFP_KERNEL);
+ aq->comp_ctx_pool = devm_kzalloc(aq->dmadev, pool_size, GFP_KERNEL);
+ if (!aq->comp_ctx || !aq->comp_ctx_pool) {
+ devm_kfree(aq->dmadev, aq->comp_ctx_pool);
+ devm_kfree(aq->dmadev, aq->comp_ctx);
+ return -ENOMEM;
+ }
+
+ for (i = 0; i < aq->depth; i++) {
+ comp_ctx = efa_com_get_comp_ctx(aq, i, false);
+ if (comp_ctx)
+ init_completion(&comp_ctx->wait_event);
+
+ aq->comp_ctx_pool[i] = i;
+ }
+
+ spin_lock_init(&aq->comp_ctx_lock);
+
+ aq->comp_ctx_pool_next = 0;
+
+ return 0;
+}
+
+static struct efa_comp_ctx *efa_com_submit_admin_cmd(struct efa_com_admin_queue *aq,
+ struct efa_admin_aq_entry *cmd,
+ size_t cmd_size_in_bytes,
+ struct efa_admin_acq_entry *comp,
+ size_t comp_size_in_bytes)
+{
+ struct efa_comp_ctx *comp_ctx;
+
+ spin_lock(&aq->sq.lock);
+ if (!test_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state)) {
+ ibdev_err(aq->efa_dev, "Admin queue is closed\n");
+ spin_unlock(&aq->sq.lock);
+ return ERR_PTR(-ENODEV);
+ }
+
+ comp_ctx = __efa_com_submit_admin_cmd(aq, cmd, cmd_size_in_bytes, comp,
+ comp_size_in_bytes);
+ spin_unlock(&aq->sq.lock);
+ if (IS_ERR(comp_ctx))
+ clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state);
+
+ return comp_ctx;
+}
+
+static void efa_com_handle_single_admin_completion(struct efa_com_admin_queue *aq,
+ struct efa_admin_acq_entry *cqe)
+{
+ struct efa_comp_ctx *comp_ctx;
+ u16 cmd_id;
+
+ cmd_id = cqe->acq_common_descriptor.command &
+ EFA_ADMIN_ACQ_COMMON_DESC_COMMAND_ID_MASK;
+
+ comp_ctx = efa_com_get_comp_ctx(aq, cmd_id, false);
+ if (!comp_ctx) {
+ ibdev_err(aq->efa_dev,
+ "comp_ctx is NULL. Changing the admin queue running state\n");
+ clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state);
+ return;
+ }
+
+ comp_ctx->status = EFA_CMD_COMPLETED;
+ comp_ctx->comp_status = cqe->acq_common_descriptor.status;
+ if (comp_ctx->user_cqe)
+ memcpy(comp_ctx->user_cqe, cqe, comp_ctx->comp_size);
+
+ if (!test_bit(EFA_AQ_STATE_POLLING_BIT, &aq->state))
+ complete(&comp_ctx->wait_event);
+}
+
+static void efa_com_handle_admin_completion(struct efa_com_admin_queue *aq)
+{
+ struct efa_admin_acq_entry *cqe;
+ u16 queue_size_mask;
+ u16 comp_num = 0;
+ u8 phase;
+ u16 ci;
+
+ queue_size_mask = aq->depth - 1;
+
+ ci = aq->cq.cc & queue_size_mask;
+ phase = aq->cq.phase;
+
+ cqe = &aq->cq.entries[ci];
+
+ /* Go over all the completions */
+ while ((READ_ONCE(cqe->acq_common_descriptor.flags) &
+ EFA_ADMIN_ACQ_COMMON_DESC_PHASE_MASK) == phase) {
+ /*
+ * Do not read the rest of the completion entry before the
+ * phase bit was validated
+ */
+ dma_rmb();
+ efa_com_handle_single_admin_completion(aq, cqe);
+
+ ci++;
+ comp_num++;
+ if (ci == aq->depth) {
+ ci = 0;
+ phase = !phase;
+ }
+
+ cqe = &aq->cq.entries[ci];
+ }
+
+ aq->cq.cc += comp_num;
+ aq->cq.phase = phase;
+ aq->sq.cc += comp_num;
+ atomic64_add(comp_num, &aq->stats.completed_cmd);
+}
+
+static int efa_com_comp_status_to_errno(u8 comp_status)
+{
+ switch (comp_status) {
+ case EFA_ADMIN_SUCCESS:
+ return 0;
+ case EFA_ADMIN_RESOURCE_ALLOCATION_FAILURE:
+ return -ENOMEM;
+ case EFA_ADMIN_UNSUPPORTED_OPCODE:
+ return -EOPNOTSUPP;
+ case EFA_ADMIN_BAD_OPCODE:
+ case EFA_ADMIN_MALFORMED_REQUEST:
+ case EFA_ADMIN_ILLEGAL_PARAMETER:
+ case EFA_ADMIN_UNKNOWN_ERROR:
+ return -EINVAL;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int efa_com_wait_and_process_admin_cq_polling(struct efa_comp_ctx *comp_ctx,
+ struct efa_com_admin_queue *aq)
+{
+ unsigned long timeout;
+ unsigned long flags;
+ int err;
+
+ timeout = jiffies + usecs_to_jiffies(aq->completion_timeout);
+
+ while (1) {
+ spin_lock_irqsave(&aq->cq.lock, flags);
+ efa_com_handle_admin_completion(aq);
+ spin_unlock_irqrestore(&aq->cq.lock, flags);
+
+ if (comp_ctx->status != EFA_CMD_SUBMITTED)
+ break;
+
+ if (time_is_before_jiffies(timeout)) {
+ ibdev_err(aq->efa_dev,
+ "Wait for completion (polling) timeout\n");
+ /* EFA didn't have any completion */
+ atomic64_inc(&aq->stats.no_completion);
+
+ clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state);
+ err = -ETIME;
+ goto out;
+ }
+
+ msleep(aq->poll_interval);
+ }
+
+ if (comp_ctx->status == EFA_CMD_ABORTED) {
+ ibdev_err(aq->efa_dev, "Command was aborted\n");
+ atomic64_inc(&aq->stats.aborted_cmd);
+ err = -ENODEV;
+ goto out;
+ }
+
+ WARN_ONCE(comp_ctx->status != EFA_CMD_COMPLETED,
+ "Invalid completion status %d\n", comp_ctx->status);
+
+ err = efa_com_comp_status_to_errno(comp_ctx->comp_status);
+out:
+ efa_com_put_comp_ctx(aq, comp_ctx);
+ return err;
+}
+
+static int efa_com_wait_and_process_admin_cq_interrupts(struct efa_comp_ctx *comp_ctx,
+ struct efa_com_admin_queue *aq)
+{
+ unsigned long flags;
+ int err;
+
+ wait_for_completion_timeout(&comp_ctx->wait_event,
+ usecs_to_jiffies(aq->completion_timeout));
+
+ /*
+ * In case the command wasn't completed find out the root cause.
+ * There might be 2 kinds of errors
+ * 1) No completion (timeout reached)
+ * 2) There is completion but the device didn't get any msi-x interrupt.
+ */
+ if (comp_ctx->status == EFA_CMD_SUBMITTED) {
+ spin_lock_irqsave(&aq->cq.lock, flags);
+ efa_com_handle_admin_completion(aq);
+ spin_unlock_irqrestore(&aq->cq.lock, flags);
+
+ atomic64_inc(&aq->stats.no_completion);
+
+ if (comp_ctx->status == EFA_CMD_COMPLETED)
+ ibdev_err(aq->efa_dev,
+ "The device sent a completion but the driver didn't receive any MSI-X interrupt for admin cmd %s(%d) status %d (ctx: 0x%p, sq producer: %d, sq consumer: %d, cq consumer: %d)\n",
+ efa_com_cmd_str(comp_ctx->cmd_opcode),
+ comp_ctx->cmd_opcode, comp_ctx->status,
+ comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc);
+ else
+ ibdev_err(aq->efa_dev,
+ "The device didn't send any completion for admin cmd %s(%d) status %d (ctx 0x%p, sq producer: %d, sq consumer: %d, cq consumer: %d)\n",
+ efa_com_cmd_str(comp_ctx->cmd_opcode),
+ comp_ctx->cmd_opcode, comp_ctx->status,
+ comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc);
+
+ clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state);
+ err = -ETIME;
+ goto out;
+ }
+
+ err = efa_com_comp_status_to_errno(comp_ctx->comp_status);
+out:
+ efa_com_put_comp_ctx(aq, comp_ctx);
+ return err;
+}
+
+/*
+ * There are two types to wait for completion.
+ * Polling mode - wait until the completion is available.
+ * Async mode - wait on wait queue until the completion is ready
+ * (or the timeout expired).
+ * It is expected that the IRQ called efa_com_handle_admin_completion
+ * to mark the completions.
+ */
+static int efa_com_wait_and_process_admin_cq(struct efa_comp_ctx *comp_ctx,
+ struct efa_com_admin_queue *aq)
+{
+ if (test_bit(EFA_AQ_STATE_POLLING_BIT, &aq->state))
+ return efa_com_wait_and_process_admin_cq_polling(comp_ctx, aq);
+
+ return efa_com_wait_and_process_admin_cq_interrupts(comp_ctx, aq);
+}
+
+/**
+ * efa_com_cmd_exec - Execute admin command
+ * @aq: admin queue.
+ * @cmd: the admin command to execute.
+ * @cmd_size: the command size.
+ * @comp: command completion return entry.
+ * @comp_size: command completion size.
+ * Submit an admin command and then wait until the device will return a
+ * completion.
+ * The completion will be copied into comp.
+ *
+ * @return - 0 on success, negative value on failure.
+ */
+int efa_com_cmd_exec(struct efa_com_admin_queue *aq,
+ struct efa_admin_aq_entry *cmd,
+ size_t cmd_size,
+ struct efa_admin_acq_entry *comp,
+ size_t comp_size)
+{
+ struct efa_comp_ctx *comp_ctx;
+ int err;
+
+ might_sleep();
+
+ /* In case of queue FULL */
+ down(&aq->avail_cmds);
+
+ ibdev_dbg(aq->efa_dev, "%s (opcode %d)\n",
+ efa_com_cmd_str(cmd->aq_common_descriptor.opcode),
+ cmd->aq_common_descriptor.opcode);
+ comp_ctx = efa_com_submit_admin_cmd(aq, cmd, cmd_size, comp, comp_size);
+ if (IS_ERR(comp_ctx)) {
+ ibdev_err(aq->efa_dev,
+ "Failed to submit command %s (opcode %u) err %ld\n",
+ efa_com_cmd_str(cmd->aq_common_descriptor.opcode),
+ cmd->aq_common_descriptor.opcode, PTR_ERR(comp_ctx));
+
+ up(&aq->avail_cmds);
+ return PTR_ERR(comp_ctx);
+ }
+
+ err = efa_com_wait_and_process_admin_cq(comp_ctx, aq);
+ if (err)
+ ibdev_err(aq->efa_dev,
+ "Failed to process command %s (opcode %u) comp_status %d err %d\n",
+ efa_com_cmd_str(cmd->aq_common_descriptor.opcode),
+ cmd->aq_common_descriptor.opcode,
+ comp_ctx->comp_status, err);
+
+ up(&aq->avail_cmds);
+
+ return err;
+}
+
+/**
+ * efa_com_abort_admin_commands - Abort all the outstanding admin commands.
+ * @edev: EFA communication layer struct
+ *
+ * This method aborts all the outstanding admin commands.
+ * The caller should then call efa_com_wait_for_abort_completion to make sure
+ * all the commands were completed.
+ */
+static void efa_com_abort_admin_commands(struct efa_com_dev *edev)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_comp_ctx *comp_ctx;
+ unsigned long flags;
+ u16 i;
+
+ spin_lock(&aq->sq.lock);
+ spin_lock_irqsave(&aq->cq.lock, flags);
+ for (i = 0; i < aq->depth; i++) {
+ comp_ctx = efa_com_get_comp_ctx(aq, i, false);
+ if (!comp_ctx)
+ break;
+
+ comp_ctx->status = EFA_CMD_ABORTED;
+
+ complete(&comp_ctx->wait_event);
+ }
+ spin_unlock_irqrestore(&aq->cq.lock, flags);
+ spin_unlock(&aq->sq.lock);
+}
+
+/**
+ * efa_com_wait_for_abort_completion - Wait for admin commands abort.
+ * @edev: EFA communication layer struct
+ *
+ * This method wait until all the outstanding admin commands will be completed.
+ */
+static void efa_com_wait_for_abort_completion(struct efa_com_dev *edev)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ int i;
+
+ /* all mine */
+ for (i = 0; i < aq->depth; i++)
+ down(&aq->avail_cmds);
+
+ /* let it go */
+ for (i = 0; i < aq->depth; i++)
+ up(&aq->avail_cmds);
+}
+
+static void efa_com_admin_flush(struct efa_com_dev *edev)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+
+ clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state);
+
+ efa_com_abort_admin_commands(edev);
+ efa_com_wait_for_abort_completion(edev);
+}
+
+/**
+ * efa_com_admin_destroy - Destroy the admin and the async events queues.
+ * @edev: EFA communication layer struct
+ */
+void efa_com_admin_destroy(struct efa_com_dev *edev)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_com_aenq *aenq = &edev->aenq;
+ struct efa_com_admin_cq *cq = &aq->cq;
+ struct efa_com_admin_sq *sq = &aq->sq;
+ u16 size;
+
+ efa_com_admin_flush(edev);
+
+ devm_kfree(edev->dmadev, aq->comp_ctx_pool);
+ devm_kfree(edev->dmadev, aq->comp_ctx);
+
+ size = aq->depth * sizeof(*sq->entries);
+ dma_free_coherent(edev->dmadev, size, sq->entries, sq->dma_addr);
+
+ size = aq->depth * sizeof(*cq->entries);
+ dma_free_coherent(edev->dmadev, size, cq->entries, cq->dma_addr);
+
+ size = aenq->depth * sizeof(*aenq->entries);
+ dma_free_coherent(edev->dmadev, size, aenq->entries, aenq->dma_addr);
+}
+
+/**
+ * efa_com_set_admin_polling_mode - Set the admin completion queue polling mode
+ * @edev: EFA communication layer struct
+ * @polling: Enable/Disable polling mode
+ *
+ * Set the admin completion mode.
+ */
+void efa_com_set_admin_polling_mode(struct efa_com_dev *edev, bool polling)
+{
+ u32 mask_value = 0;
+
+ if (polling)
+ mask_value = EFA_REGS_ADMIN_INTR_MASK;
+
+ writel(mask_value, edev->reg_bar + EFA_REGS_INTR_MASK_OFF);
+ if (polling)
+ set_bit(EFA_AQ_STATE_POLLING_BIT, &edev->aq.state);
+ else
+ clear_bit(EFA_AQ_STATE_POLLING_BIT, &edev->aq.state);
+}
+
+static void efa_com_stats_init(struct efa_com_dev *edev)
+{
+ atomic64_t *s = (atomic64_t *)&edev->aq.stats;
+ int i;
+
+ for (i = 0; i < sizeof(edev->aq.stats) / sizeof(*s); i++, s++)
+ atomic64_set(s, 0);
+}
+
+/**
+ * efa_com_admin_init - Init the admin and the async queues
+ * @edev: EFA communication layer struct
+ * @aenq_handlers: Those handlers to be called upon event.
+ *
+ * Initialize the admin submission and completion queues.
+ * Initialize the asynchronous events notification queues.
+ *
+ * @return - 0 on success, negative value on failure.
+ */
+int efa_com_admin_init(struct efa_com_dev *edev,
+ struct efa_aenq_handlers *aenq_handlers)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ u32 timeout;
+ u32 dev_sts;
+ u32 cap;
+ int err;
+
+ dev_sts = efa_com_reg_read32(edev, EFA_REGS_DEV_STS_OFF);
+ if (!(dev_sts & EFA_REGS_DEV_STS_READY_MASK)) {
+ ibdev_err(edev->efa_dev,
+ "Device isn't ready, abort com init %#x\n", dev_sts);
+ return -ENODEV;
+ }
+
+ aq->depth = EFA_ADMIN_QUEUE_DEPTH;
+
+ aq->dmadev = edev->dmadev;
+ aq->efa_dev = edev->efa_dev;
+ set_bit(EFA_AQ_STATE_POLLING_BIT, &aq->state);
+
+ sema_init(&aq->avail_cmds, aq->depth);
+
+ efa_com_stats_init(edev);
+
+ err = efa_com_init_comp_ctxt(aq);
+ if (err)
+ return err;
+
+ err = efa_com_admin_init_sq(edev);
+ if (err)
+ goto err_destroy_comp_ctxt;
+
+ err = efa_com_admin_init_cq(edev);
+ if (err)
+ goto err_destroy_sq;
+
+ efa_com_set_admin_polling_mode(edev, false);
+
+ err = efa_com_admin_init_aenq(edev, aenq_handlers);
+ if (err)
+ goto err_destroy_cq;
+
+ cap = efa_com_reg_read32(edev, EFA_REGS_CAPS_OFF);
+ timeout = (cap & EFA_REGS_CAPS_ADMIN_CMD_TO_MASK) >>
+ EFA_REGS_CAPS_ADMIN_CMD_TO_SHIFT;
+ if (timeout)
+ /* the resolution of timeout reg is 100ms */
+ aq->completion_timeout = timeout * 100000;
+ else
+ aq->completion_timeout = ADMIN_CMD_TIMEOUT_US;
+
+ aq->poll_interval = EFA_POLL_INTERVAL_MS;
+
+ set_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state);
+
+ return 0;
+
+err_destroy_cq:
+ dma_free_coherent(edev->dmadev, aq->depth * sizeof(*aq->cq.entries),
+ aq->cq.entries, aq->cq.dma_addr);
+err_destroy_sq:
+ dma_free_coherent(edev->dmadev, aq->depth * sizeof(*aq->sq.entries),
+ aq->sq.entries, aq->sq.dma_addr);
+err_destroy_comp_ctxt:
+ devm_kfree(edev->dmadev, aq->comp_ctx);
+
+ return err;
+}
+
+/**
+ * efa_com_admin_q_comp_intr_handler - admin queue interrupt handler
+ * @edev: EFA communication layer struct
+ *
+ * This method goes over the admin completion queue and wakes up
+ * all the pending threads that wait on the commands wait event.
+ *
+ * @note: Should be called after MSI-X interrupt.
+ */
+void efa_com_admin_q_comp_intr_handler(struct efa_com_dev *edev)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&edev->aq.cq.lock, flags);
+ efa_com_handle_admin_completion(&edev->aq);
+ spin_unlock_irqrestore(&edev->aq.cq.lock, flags);
+}
+
+/*
+ * efa_handle_specific_aenq_event:
+ * return the handler that is relevant to the specific event group
+ */
+static efa_aenq_handler efa_com_get_specific_aenq_cb(struct efa_com_dev *edev,
+ u16 group)
+{
+ struct efa_aenq_handlers *aenq_handlers = edev->aenq.aenq_handlers;
+
+ if (group < EFA_MAX_HANDLERS && aenq_handlers->handlers[group])
+ return aenq_handlers->handlers[group];
+
+ return aenq_handlers->unimplemented_handler;
+}
+
+/**
+ * efa_com_aenq_intr_handler - AENQ interrupt handler
+ * @edev: EFA communication layer struct
+ * @data: Data of interrupt handler.
+ *
+ * Go over the async event notification queue and call the proper aenq handler.
+ */
+void efa_com_aenq_intr_handler(struct efa_com_dev *edev, void *data)
+{
+ struct efa_admin_aenq_common_desc *aenq_common;
+ struct efa_com_aenq *aenq = &edev->aenq;
+ struct efa_admin_aenq_entry *aenq_e;
+ efa_aenq_handler handler_cb;
+ u32 processed = 0;
+ u8 phase;
+ u32 ci;
+
+ ci = aenq->cc & (aenq->depth - 1);
+ phase = aenq->phase;
+ aenq_e = &aenq->entries[ci]; /* Get first entry */
+ aenq_common = &aenq_e->aenq_common_desc;
+
+ /* Go over all the events */
+ while ((READ_ONCE(aenq_common->flags) &
+ EFA_ADMIN_AENQ_COMMON_DESC_PHASE_MASK) == phase) {
+ /*
+ * Do not read the rest of the completion entry before the
+ * phase bit was validated
+ */
+ dma_rmb();
+
+ /* Handle specific event*/
+ handler_cb = efa_com_get_specific_aenq_cb(edev,
+ aenq_common->group);
+ handler_cb(data, aenq_e); /* call the actual event handler*/
+
+ /* Get next event entry */
+ ci++;
+ processed++;
+
+ if (ci == aenq->depth) {
+ ci = 0;
+ phase = !phase;
+ }
+ aenq_e = &aenq->entries[ci];
+ aenq_common = &aenq_e->aenq_common_desc;
+ }
+
+ aenq->cc += processed;
+ aenq->phase = phase;
+
+ /* Don't update aenq doorbell if there weren't any processed events */
+ if (!processed)
+ return;
+
+ /* barrier not needed in case of writel */
+ writel(aenq->cc, edev->reg_bar + EFA_REGS_AENQ_CONS_DB_OFF);
+}
+
+static void efa_com_mmio_reg_read_resp_addr_init(struct efa_com_dev *edev)
+{
+ struct efa_com_mmio_read *mmio_read = &edev->mmio_read;
+ u32 addr_high;
+ u32 addr_low;
+
+ /* dma_addr_bits is unknown at this point */
+ addr_high = (mmio_read->read_resp_dma_addr >> 32) & GENMASK(31, 0);
+ addr_low = mmio_read->read_resp_dma_addr & GENMASK(31, 0);
+
+ writel(addr_high, edev->reg_bar + EFA_REGS_MMIO_RESP_HI_OFF);
+ writel(addr_low, edev->reg_bar + EFA_REGS_MMIO_RESP_LO_OFF);
+}
+
+int efa_com_mmio_reg_read_init(struct efa_com_dev *edev)
+{
+ struct efa_com_mmio_read *mmio_read = &edev->mmio_read;
+
+ spin_lock_init(&mmio_read->lock);
+ mmio_read->read_resp =
+ dma_alloc_coherent(edev->dmadev, sizeof(*mmio_read->read_resp),
+ &mmio_read->read_resp_dma_addr, GFP_KERNEL);
+ if (!mmio_read->read_resp)
+ return -ENOMEM;
+
+ efa_com_mmio_reg_read_resp_addr_init(edev);
+
+ mmio_read->read_resp->req_id = 0;
+ mmio_read->seq_num = 0;
+ mmio_read->mmio_read_timeout = EFA_REG_READ_TIMEOUT_US;
+
+ return 0;
+}
+
+void efa_com_mmio_reg_read_destroy(struct efa_com_dev *edev)
+{
+ struct efa_com_mmio_read *mmio_read = &edev->mmio_read;
+
+ dma_free_coherent(edev->dmadev, sizeof(*mmio_read->read_resp),
+ mmio_read->read_resp, mmio_read->read_resp_dma_addr);
+}
+
+int efa_com_validate_version(struct efa_com_dev *edev)
+{
+ u32 ctrl_ver_masked;
+ u32 ctrl_ver;
+ u32 ver;
+
+ /*
+ * Make sure the EFA version and the controller version are at least
+ * as the driver expects
+ */
+ ver = efa_com_reg_read32(edev, EFA_REGS_VERSION_OFF);
+ ctrl_ver = efa_com_reg_read32(edev,
+ EFA_REGS_CONTROLLER_VERSION_OFF);
+
+ ibdev_dbg(edev->efa_dev, "efa device version: %d.%d\n",
+ (ver & EFA_REGS_VERSION_MAJOR_VERSION_MASK) >>
+ EFA_REGS_VERSION_MAJOR_VERSION_SHIFT,
+ ver & EFA_REGS_VERSION_MINOR_VERSION_MASK);
+
+ if (ver < MIN_EFA_VER) {
+ ibdev_err(edev->efa_dev,
+ "EFA version is lower than the minimal version the driver supports\n");
+ return -EOPNOTSUPP;
+ }
+
+ ibdev_dbg(edev->efa_dev,
+ "efa controller version: %d.%d.%d implementation version %d\n",
+ (ctrl_ver & EFA_REGS_CONTROLLER_VERSION_MAJOR_VERSION_MASK) >>
+ EFA_REGS_CONTROLLER_VERSION_MAJOR_VERSION_SHIFT,
+ (ctrl_ver & EFA_REGS_CONTROLLER_VERSION_MINOR_VERSION_MASK) >>
+ EFA_REGS_CONTROLLER_VERSION_MINOR_VERSION_SHIFT,
+ (ctrl_ver & EFA_REGS_CONTROLLER_VERSION_SUBMINOR_VERSION_MASK),
+ (ctrl_ver & EFA_REGS_CONTROLLER_VERSION_IMPL_ID_MASK) >>
+ EFA_REGS_CONTROLLER_VERSION_IMPL_ID_SHIFT);
+
+ ctrl_ver_masked =
+ (ctrl_ver & EFA_REGS_CONTROLLER_VERSION_MAJOR_VERSION_MASK) |
+ (ctrl_ver & EFA_REGS_CONTROLLER_VERSION_MINOR_VERSION_MASK) |
+ (ctrl_ver & EFA_REGS_CONTROLLER_VERSION_SUBMINOR_VERSION_MASK);
+
+ /* Validate the ctrl version without the implementation ID */
+ if (ctrl_ver_masked < MIN_EFA_CTRL_VER) {
+ ibdev_err(edev->efa_dev,
+ "EFA ctrl version is lower than the minimal ctrl version the driver supports\n");
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+/**
+ * efa_com_get_dma_width - Retrieve physical dma address width the device
+ * supports.
+ * @edev: EFA communication layer struct
+ *
+ * Retrieve the maximum physical address bits the device can handle.
+ *
+ * @return: > 0 on Success and negative value otherwise.
+ */
+int efa_com_get_dma_width(struct efa_com_dev *edev)
+{
+ u32 caps = efa_com_reg_read32(edev, EFA_REGS_CAPS_OFF);
+ int width;
+
+ width = (caps & EFA_REGS_CAPS_DMA_ADDR_WIDTH_MASK) >>
+ EFA_REGS_CAPS_DMA_ADDR_WIDTH_SHIFT;
+
+ ibdev_dbg(edev->efa_dev, "DMA width: %d\n", width);
+
+ if (width < 32 || width > 64) {
+ ibdev_err(edev->efa_dev, "DMA width illegal value: %d\n", width);
+ return -EINVAL;
+ }
+
+ edev->dma_addr_bits = width;
+
+ return width;
+}
+
+static int wait_for_reset_state(struct efa_com_dev *edev, u32 timeout,
+ u16 exp_state)
+{
+ u32 val, i;
+
+ for (i = 0; i < timeout; i++) {
+ val = efa_com_reg_read32(edev, EFA_REGS_DEV_STS_OFF);
+
+ if ((val & EFA_REGS_DEV_STS_RESET_IN_PROGRESS_MASK) ==
+ exp_state)
+ return 0;
+
+ ibdev_dbg(edev->efa_dev, "Reset indication val %d\n", val);
+ msleep(EFA_POLL_INTERVAL_MS);
+ }
+
+ return -ETIME;
+}
+
+/**
+ * efa_com_dev_reset - Perform device FLR to the device.
+ * @edev: EFA communication layer struct
+ * @reset_reason: Specify what is the trigger for the reset in case of an error.
+ *
+ * @return - 0 on success, negative value on failure.
+ */
+int efa_com_dev_reset(struct efa_com_dev *edev,
+ enum efa_regs_reset_reason_types reset_reason)
+{
+ u32 stat, timeout, cap, reset_val;
+ int err;
+
+ stat = efa_com_reg_read32(edev, EFA_REGS_DEV_STS_OFF);
+ cap = efa_com_reg_read32(edev, EFA_REGS_CAPS_OFF);
+
+ if (!(stat & EFA_REGS_DEV_STS_READY_MASK)) {
+ ibdev_err(edev->efa_dev,
+ "Device isn't ready, can't reset device\n");
+ return -EINVAL;
+ }
+
+ timeout = (cap & EFA_REGS_CAPS_RESET_TIMEOUT_MASK) >>
+ EFA_REGS_CAPS_RESET_TIMEOUT_SHIFT;
+ if (!timeout) {
+ ibdev_err(edev->efa_dev, "Invalid timeout value\n");
+ return -EINVAL;
+ }
+
+ /* start reset */
+ reset_val = EFA_REGS_DEV_CTL_DEV_RESET_MASK;
+ reset_val |= (reset_reason << EFA_REGS_DEV_CTL_RESET_REASON_SHIFT) &
+ EFA_REGS_DEV_CTL_RESET_REASON_MASK;
+ writel(reset_val, edev->reg_bar + EFA_REGS_DEV_CTL_OFF);
+
+ /* reset clears the mmio readless address, restore it */
+ efa_com_mmio_reg_read_resp_addr_init(edev);
+
+ err = wait_for_reset_state(edev, timeout,
+ EFA_REGS_DEV_STS_RESET_IN_PROGRESS_MASK);
+ if (err) {
+ ibdev_err(edev->efa_dev, "Reset indication didn't turn on\n");
+ return err;
+ }
+
+ /* reset done */
+ writel(0, edev->reg_bar + EFA_REGS_DEV_CTL_OFF);
+ err = wait_for_reset_state(edev, timeout, 0);
+ if (err) {
+ ibdev_err(edev->efa_dev, "Reset indication didn't turn off\n");
+ return err;
+ }
+
+ timeout = (cap & EFA_REGS_CAPS_ADMIN_CMD_TO_MASK) >>
+ EFA_REGS_CAPS_ADMIN_CMD_TO_SHIFT;
+ if (timeout)
+ /* the resolution of timeout reg is 100ms */
+ edev->aq.completion_timeout = timeout * 100000;
+ else
+ edev->aq.completion_timeout = ADMIN_CMD_TIMEOUT_US;
+
+ return 0;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#ifndef _EFA_COM_H_
+#define _EFA_COM_H_
+
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/semaphore.h>
+#include <linux/sched.h>
+
+#include <rdma/ib_verbs.h>
+
+#include "efa_common_defs.h"
+#include "efa_admin_defs.h"
+#include "efa_admin_cmds_defs.h"
+#include "efa_regs_defs.h"
+
+#define EFA_MAX_HANDLERS 256
+
+struct efa_com_admin_cq {
+ struct efa_admin_acq_entry *entries;
+ dma_addr_t dma_addr;
+ spinlock_t lock; /* Protects ACQ */
+
+ u16 cc; /* consumer counter */
+ u8 phase;
+};
+
+struct efa_com_admin_sq {
+ struct efa_admin_aq_entry *entries;
+ dma_addr_t dma_addr;
+ spinlock_t lock; /* Protects ASQ */
+
+ u32 __iomem *db_addr;
+
+ u16 cc; /* consumer counter */
+ u16 pc; /* producer counter */
+ u8 phase;
+
+};
+
+/* Don't use anything other than atomic64 */
+struct efa_com_stats_admin {
+ atomic64_t aborted_cmd;
+ atomic64_t submitted_cmd;
+ atomic64_t completed_cmd;
+ atomic64_t no_completion;
+};
+
+enum {
+ EFA_AQ_STATE_RUNNING_BIT = 0,
+ EFA_AQ_STATE_POLLING_BIT = 1,
+};
+
+struct efa_com_admin_queue {
+ void *dmadev;
+ void *efa_dev;
+ struct efa_comp_ctx *comp_ctx;
+ u32 completion_timeout; /* usecs */
+ u16 poll_interval; /* msecs */
+ u16 depth;
+ struct efa_com_admin_cq cq;
+ struct efa_com_admin_sq sq;
+ u16 msix_vector_idx;
+
+ unsigned long state;
+
+ /* Count the number of available admin commands */
+ struct semaphore avail_cmds;
+
+ struct efa_com_stats_admin stats;
+
+ spinlock_t comp_ctx_lock; /* Protects completion context pool */
+ u32 *comp_ctx_pool;
+ u16 comp_ctx_pool_next;
+};
+
+struct efa_aenq_handlers;
+
+struct efa_com_aenq {
+ struct efa_admin_aenq_entry *entries;
+ struct efa_aenq_handlers *aenq_handlers;
+ dma_addr_t dma_addr;
+ u32 cc; /* consumer counter */
+ u16 msix_vector_idx;
+ u16 depth;
+ u8 phase;
+};
+
+struct efa_com_mmio_read {
+ struct efa_admin_mmio_req_read_less_resp *read_resp;
+ dma_addr_t read_resp_dma_addr;
+ u16 seq_num;
+ u16 mmio_read_timeout; /* usecs */
+ /* serializes mmio reads */
+ spinlock_t lock;
+};
+
+struct efa_com_dev {
+ struct efa_com_admin_queue aq;
+ struct efa_com_aenq aenq;
+ u8 __iomem *reg_bar;
+ void *dmadev;
+ void *efa_dev;
+ u32 supported_features;
+ u32 dma_addr_bits;
+
+ struct efa_com_mmio_read mmio_read;
+};
+
+typedef void (*efa_aenq_handler)(void *data,
+ struct efa_admin_aenq_entry *aenq_e);
+
+/* Holds aenq handlers. Indexed by AENQ event group */
+struct efa_aenq_handlers {
+ efa_aenq_handler handlers[EFA_MAX_HANDLERS];
+ efa_aenq_handler unimplemented_handler;
+};
+
+int efa_com_admin_init(struct efa_com_dev *edev,
+ struct efa_aenq_handlers *aenq_handlers);
+void efa_com_admin_destroy(struct efa_com_dev *edev);
+int efa_com_dev_reset(struct efa_com_dev *edev,
+ enum efa_regs_reset_reason_types reset_reason);
+void efa_com_set_admin_polling_mode(struct efa_com_dev *edev, bool polling);
+void efa_com_admin_q_comp_intr_handler(struct efa_com_dev *edev);
+int efa_com_mmio_reg_read_init(struct efa_com_dev *edev);
+void efa_com_mmio_reg_read_destroy(struct efa_com_dev *edev);
+
+int efa_com_validate_version(struct efa_com_dev *edev);
+int efa_com_get_dma_width(struct efa_com_dev *edev);
+
+int efa_com_cmd_exec(struct efa_com_admin_queue *aq,
+ struct efa_admin_aq_entry *cmd,
+ size_t cmd_size,
+ struct efa_admin_acq_entry *comp,
+ size_t comp_size);
+void efa_com_aenq_intr_handler(struct efa_com_dev *edev, void *data);
+
+#endif /* _EFA_COM_H_ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#include "efa.h"
+#include "efa_com.h"
+#include "efa_com_cmd.h"
+
+void efa_com_set_dma_addr(dma_addr_t addr, u32 *addr_high, u32 *addr_low)
+{
+ *addr_low = lower_32_bits(addr);
+ *addr_high = upper_32_bits(addr);
+}
+
+int efa_com_create_qp(struct efa_com_dev *edev,
+ struct efa_com_create_qp_params *params,
+ struct efa_com_create_qp_result *res)
+{
+ struct efa_admin_create_qp_cmd create_qp_cmd = {};
+ struct efa_admin_create_qp_resp cmd_completion;
+ struct efa_com_admin_queue *aq = &edev->aq;
+ int err;
+
+ create_qp_cmd.aq_common_desc.opcode = EFA_ADMIN_CREATE_QP;
+
+ create_qp_cmd.pd = params->pd;
+ create_qp_cmd.qp_type = params->qp_type;
+ create_qp_cmd.rq_base_addr = params->rq_base_addr;
+ create_qp_cmd.send_cq_idx = params->send_cq_idx;
+ create_qp_cmd.recv_cq_idx = params->recv_cq_idx;
+ create_qp_cmd.qp_alloc_size.send_queue_ring_size =
+ params->sq_ring_size_in_bytes;
+ create_qp_cmd.qp_alloc_size.send_queue_depth =
+ params->sq_depth;
+ create_qp_cmd.qp_alloc_size.recv_queue_ring_size =
+ params->rq_ring_size_in_bytes;
+ create_qp_cmd.qp_alloc_size.recv_queue_depth =
+ params->rq_depth;
+ create_qp_cmd.uar = params->uarn;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&create_qp_cmd,
+ sizeof(create_qp_cmd),
+ (struct efa_admin_acq_entry *)&cmd_completion,
+ sizeof(cmd_completion));
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to create qp [%d]\n", err);
+ return err;
+ }
+
+ res->qp_handle = cmd_completion.qp_handle;
+ res->qp_num = cmd_completion.qp_num;
+ res->sq_db_offset = cmd_completion.sq_db_offset;
+ res->rq_db_offset = cmd_completion.rq_db_offset;
+ res->llq_descriptors_offset = cmd_completion.llq_descriptors_offset;
+ res->send_sub_cq_idx = cmd_completion.send_sub_cq_idx;
+ res->recv_sub_cq_idx = cmd_completion.recv_sub_cq_idx;
+
+ return err;
+}
+
+int efa_com_modify_qp(struct efa_com_dev *edev,
+ struct efa_com_modify_qp_params *params)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_admin_modify_qp_cmd cmd = {};
+ struct efa_admin_modify_qp_resp resp;
+ int err;
+
+ cmd.aq_common_desc.opcode = EFA_ADMIN_MODIFY_QP;
+ cmd.modify_mask = params->modify_mask;
+ cmd.qp_handle = params->qp_handle;
+ cmd.qp_state = params->qp_state;
+ cmd.cur_qp_state = params->cur_qp_state;
+ cmd.qkey = params->qkey;
+ cmd.sq_psn = params->sq_psn;
+ cmd.sq_drained_async_notify = params->sq_drained_async_notify;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&cmd,
+ sizeof(cmd),
+ (struct efa_admin_acq_entry *)&resp,
+ sizeof(resp));
+ if (err) {
+ ibdev_err(edev->efa_dev,
+ "Failed to modify qp-%u modify_mask[%#x] [%d]\n",
+ cmd.qp_handle, cmd.modify_mask, err);
+ return err;
+ }
+
+ return 0;
+}
+
+int efa_com_query_qp(struct efa_com_dev *edev,
+ struct efa_com_query_qp_params *params,
+ struct efa_com_query_qp_result *result)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_admin_query_qp_cmd cmd = {};
+ struct efa_admin_query_qp_resp resp;
+ int err;
+
+ cmd.aq_common_desc.opcode = EFA_ADMIN_QUERY_QP;
+ cmd.qp_handle = params->qp_handle;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&cmd,
+ sizeof(cmd),
+ (struct efa_admin_acq_entry *)&resp,
+ sizeof(resp));
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to query qp-%u [%d]\n",
+ cmd.qp_handle, err);
+ return err;
+ }
+
+ result->qp_state = resp.qp_state;
+ result->qkey = resp.qkey;
+ result->sq_draining = resp.sq_draining;
+ result->sq_psn = resp.sq_psn;
+
+ return 0;
+}
+
+int efa_com_destroy_qp(struct efa_com_dev *edev,
+ struct efa_com_destroy_qp_params *params)
+{
+ struct efa_admin_destroy_qp_resp cmd_completion;
+ struct efa_admin_destroy_qp_cmd qp_cmd = {};
+ struct efa_com_admin_queue *aq = &edev->aq;
+ int err;
+
+ qp_cmd.aq_common_desc.opcode = EFA_ADMIN_DESTROY_QP;
+ qp_cmd.qp_handle = params->qp_handle;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&qp_cmd,
+ sizeof(qp_cmd),
+ (struct efa_admin_acq_entry *)&cmd_completion,
+ sizeof(cmd_completion));
+ if (err)
+ ibdev_err(edev->efa_dev, "Failed to destroy qp-%u [%d]\n",
+ qp_cmd.qp_handle, err);
+
+ return 0;
+}
+
+int efa_com_create_cq(struct efa_com_dev *edev,
+ struct efa_com_create_cq_params *params,
+ struct efa_com_create_cq_result *result)
+{
+ struct efa_admin_create_cq_resp cmd_completion;
+ struct efa_admin_create_cq_cmd create_cmd = {};
+ struct efa_com_admin_queue *aq = &edev->aq;
+ int err;
+
+ create_cmd.aq_common_desc.opcode = EFA_ADMIN_CREATE_CQ;
+ create_cmd.cq_caps_2 = (params->entry_size_in_bytes / 4) &
+ EFA_ADMIN_CREATE_CQ_CMD_CQ_ENTRY_SIZE_WORDS_MASK;
+ create_cmd.cq_depth = params->cq_depth;
+ create_cmd.num_sub_cqs = params->num_sub_cqs;
+ create_cmd.uar = params->uarn;
+
+ efa_com_set_dma_addr(params->dma_addr,
+ &create_cmd.cq_ba.mem_addr_high,
+ &create_cmd.cq_ba.mem_addr_low);
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&create_cmd,
+ sizeof(create_cmd),
+ (struct efa_admin_acq_entry *)&cmd_completion,
+ sizeof(cmd_completion));
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to create cq[%d]\n", err);
+ return err;
+ }
+
+ result->cq_idx = cmd_completion.cq_idx;
+ result->actual_depth = params->cq_depth;
+
+ return err;
+}
+
+int efa_com_destroy_cq(struct efa_com_dev *edev,
+ struct efa_com_destroy_cq_params *params)
+{
+ struct efa_admin_destroy_cq_cmd destroy_cmd = {};
+ struct efa_admin_destroy_cq_resp destroy_resp;
+ struct efa_com_admin_queue *aq = &edev->aq;
+ int err;
+
+ destroy_cmd.cq_idx = params->cq_idx;
+ destroy_cmd.aq_common_desc.opcode = EFA_ADMIN_DESTROY_CQ;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&destroy_cmd,
+ sizeof(destroy_cmd),
+ (struct efa_admin_acq_entry *)&destroy_resp,
+ sizeof(destroy_resp));
+
+ if (err)
+ ibdev_err(edev->efa_dev, "Failed to destroy CQ-%u [%d]\n",
+ params->cq_idx, err);
+
+ return 0;
+}
+
+int efa_com_register_mr(struct efa_com_dev *edev,
+ struct efa_com_reg_mr_params *params,
+ struct efa_com_reg_mr_result *result)
+{
+ struct efa_admin_reg_mr_resp cmd_completion;
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_admin_reg_mr_cmd mr_cmd = {};
+ int err;
+
+ mr_cmd.aq_common_desc.opcode = EFA_ADMIN_REG_MR;
+ mr_cmd.pd = params->pd;
+ mr_cmd.mr_length = params->mr_length_in_bytes;
+ mr_cmd.flags |= params->page_shift &
+ EFA_ADMIN_REG_MR_CMD_PHYS_PAGE_SIZE_SHIFT_MASK;
+ mr_cmd.iova = params->iova;
+ mr_cmd.permissions |= params->permissions &
+ EFA_ADMIN_REG_MR_CMD_LOCAL_WRITE_ENABLE_MASK;
+
+ if (params->inline_pbl) {
+ memcpy(mr_cmd.pbl.inline_pbl_array,
+ params->pbl.inline_pbl_array,
+ sizeof(mr_cmd.pbl.inline_pbl_array));
+ } else {
+ mr_cmd.pbl.pbl.length = params->pbl.pbl.length;
+ mr_cmd.pbl.pbl.address.mem_addr_low =
+ params->pbl.pbl.address.mem_addr_low;
+ mr_cmd.pbl.pbl.address.mem_addr_high =
+ params->pbl.pbl.address.mem_addr_high;
+ mr_cmd.aq_common_desc.flags |=
+ EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA_MASK;
+ if (params->indirect)
+ mr_cmd.aq_common_desc.flags |=
+ EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA_INDIRECT_MASK;
+ }
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&mr_cmd,
+ sizeof(mr_cmd),
+ (struct efa_admin_acq_entry *)&cmd_completion,
+ sizeof(cmd_completion));
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to register mr [%d]\n", err);
+ return err;
+ }
+
+ result->l_key = cmd_completion.l_key;
+ result->r_key = cmd_completion.r_key;
+
+ return 0;
+}
+
+int efa_com_dereg_mr(struct efa_com_dev *edev,
+ struct efa_com_dereg_mr_params *params)
+{
+ struct efa_admin_dereg_mr_resp cmd_completion;
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_admin_dereg_mr_cmd mr_cmd = {};
+ int err;
+
+ mr_cmd.aq_common_desc.opcode = EFA_ADMIN_DEREG_MR;
+ mr_cmd.l_key = params->l_key;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&mr_cmd,
+ sizeof(mr_cmd),
+ (struct efa_admin_acq_entry *)&cmd_completion,
+ sizeof(cmd_completion));
+ if (err)
+ ibdev_err(edev->efa_dev,
+ "Failed to de-register mr(lkey-%u) [%d]\n",
+ mr_cmd.l_key, err);
+
+ return 0;
+}
+
+int efa_com_create_ah(struct efa_com_dev *edev,
+ struct efa_com_create_ah_params *params,
+ struct efa_com_create_ah_result *result)
+{
+ struct efa_admin_create_ah_resp cmd_completion;
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_admin_create_ah_cmd ah_cmd = {};
+ int err;
+
+ ah_cmd.aq_common_desc.opcode = EFA_ADMIN_CREATE_AH;
+
+ memcpy(ah_cmd.dest_addr, params->dest_addr, sizeof(ah_cmd.dest_addr));
+ ah_cmd.pd = params->pdn;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&ah_cmd,
+ sizeof(ah_cmd),
+ (struct efa_admin_acq_entry *)&cmd_completion,
+ sizeof(cmd_completion));
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to create ah [%d]\n", err);
+ return err;
+ }
+
+ result->ah = cmd_completion.ah;
+
+ return 0;
+}
+
+int efa_com_destroy_ah(struct efa_com_dev *edev,
+ struct efa_com_destroy_ah_params *params)
+{
+ struct efa_admin_destroy_ah_resp cmd_completion;
+ struct efa_admin_destroy_ah_cmd ah_cmd = {};
+ struct efa_com_admin_queue *aq = &edev->aq;
+ int err;
+
+ ah_cmd.aq_common_desc.opcode = EFA_ADMIN_DESTROY_AH;
+ ah_cmd.ah = params->ah;
+ ah_cmd.pd = params->pdn;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&ah_cmd,
+ sizeof(ah_cmd),
+ (struct efa_admin_acq_entry *)&cmd_completion,
+ sizeof(cmd_completion));
+ if (err)
+ ibdev_err(edev->efa_dev, "Failed to destroy ah-%d pd-%d [%d]\n",
+ ah_cmd.ah, ah_cmd.pd, err);
+
+ return 0;
+}
+
+static bool
+efa_com_check_supported_feature_id(struct efa_com_dev *edev,
+ enum efa_admin_aq_feature_id feature_id)
+{
+ u32 feature_mask = 1 << feature_id;
+
+ /* Device attributes is always supported */
+ if (feature_id != EFA_ADMIN_DEVICE_ATTR &&
+ !(edev->supported_features & feature_mask))
+ return false;
+
+ return true;
+}
+
+static int efa_com_get_feature_ex(struct efa_com_dev *edev,
+ struct efa_admin_get_feature_resp *get_resp,
+ enum efa_admin_aq_feature_id feature_id,
+ dma_addr_t control_buf_dma_addr,
+ u32 control_buff_size)
+{
+ struct efa_admin_get_feature_cmd get_cmd = {};
+ struct efa_com_admin_queue *aq;
+ int err;
+
+ if (!efa_com_check_supported_feature_id(edev, feature_id)) {
+ ibdev_err(edev->efa_dev, "Feature %d isn't supported\n",
+ feature_id);
+ return -EOPNOTSUPP;
+ }
+
+ aq = &edev->aq;
+
+ get_cmd.aq_common_descriptor.opcode = EFA_ADMIN_GET_FEATURE;
+
+ if (control_buff_size)
+ get_cmd.aq_common_descriptor.flags =
+ EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA_INDIRECT_MASK;
+
+
+ efa_com_set_dma_addr(control_buf_dma_addr,
+ &get_cmd.control_buffer.address.mem_addr_high,
+ &get_cmd.control_buffer.address.mem_addr_low);
+
+ get_cmd.control_buffer.length = control_buff_size;
+ get_cmd.feature_common.feature_id = feature_id;
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)
+ &get_cmd,
+ sizeof(get_cmd),
+ (struct efa_admin_acq_entry *)
+ get_resp,
+ sizeof(*get_resp));
+
+ if (err)
+ ibdev_err(edev->efa_dev,
+ "Failed to submit get_feature command %d [%d]\n",
+ feature_id, err);
+
+ return 0;
+}
+
+static int efa_com_get_feature(struct efa_com_dev *edev,
+ struct efa_admin_get_feature_resp *get_resp,
+ enum efa_admin_aq_feature_id feature_id)
+{
+ return efa_com_get_feature_ex(edev, get_resp, feature_id, 0, 0);
+}
+
+int efa_com_get_network_attr(struct efa_com_dev *edev,
+ struct efa_com_get_network_attr_result *result)
+{
+ struct efa_admin_get_feature_resp resp;
+ int err;
+
+ err = efa_com_get_feature(edev, &resp,
+ EFA_ADMIN_NETWORK_ATTR);
+ if (err) {
+ ibdev_err(edev->efa_dev,
+ "Failed to get network attributes %d\n", err);
+ return err;
+ }
+
+ memcpy(result->addr, resp.u.network_attr.addr,
+ sizeof(resp.u.network_attr.addr));
+ result->mtu = resp.u.network_attr.mtu;
+
+ return 0;
+}
+
+int efa_com_get_device_attr(struct efa_com_dev *edev,
+ struct efa_com_get_device_attr_result *result)
+{
+ struct efa_admin_get_feature_resp resp;
+ int err;
+
+ err = efa_com_get_feature(edev, &resp, EFA_ADMIN_DEVICE_ATTR);
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to get device attributes %d\n",
+ err);
+ return err;
+ }
+
+ result->page_size_cap = resp.u.device_attr.page_size_cap;
+ result->fw_version = resp.u.device_attr.fw_version;
+ result->admin_api_version = resp.u.device_attr.admin_api_version;
+ result->device_version = resp.u.device_attr.device_version;
+ result->supported_features = resp.u.device_attr.supported_features;
+ result->phys_addr_width = resp.u.device_attr.phys_addr_width;
+ result->virt_addr_width = resp.u.device_attr.virt_addr_width;
+ result->db_bar = resp.u.device_attr.db_bar;
+
+ if (result->admin_api_version < 1) {
+ ibdev_err(edev->efa_dev,
+ "Failed to get device attr api version [%u < 1]\n",
+ result->admin_api_version);
+ return -EINVAL;
+ }
+
+ edev->supported_features = resp.u.device_attr.supported_features;
+ err = efa_com_get_feature(edev, &resp,
+ EFA_ADMIN_QUEUE_ATTR);
+ if (err) {
+ ibdev_err(edev->efa_dev,
+ "Failed to get network attributes %d\n", err);
+ return err;
+ }
+
+ result->max_qp = resp.u.queue_attr.max_qp;
+ result->max_sq_depth = resp.u.queue_attr.max_sq_depth;
+ result->max_rq_depth = resp.u.queue_attr.max_rq_depth;
+ result->max_cq = resp.u.queue_attr.max_cq;
+ result->max_cq_depth = resp.u.queue_attr.max_cq_depth;
+ result->inline_buf_size = resp.u.queue_attr.inline_buf_size;
+ result->max_sq_sge = resp.u.queue_attr.max_wr_send_sges;
+ result->max_rq_sge = resp.u.queue_attr.max_wr_recv_sges;
+ result->max_mr = resp.u.queue_attr.max_mr;
+ result->max_mr_pages = resp.u.queue_attr.max_mr_pages;
+ result->max_pd = resp.u.queue_attr.max_pd;
+ result->max_ah = resp.u.queue_attr.max_ah;
+ result->max_llq_size = resp.u.queue_attr.max_llq_size;
+ result->sub_cqs_per_cq = resp.u.queue_attr.sub_cqs_per_cq;
+
+ return 0;
+}
+
+int efa_com_get_hw_hints(struct efa_com_dev *edev,
+ struct efa_com_get_hw_hints_result *result)
+{
+ struct efa_admin_get_feature_resp resp;
+ int err;
+
+ err = efa_com_get_feature(edev, &resp, EFA_ADMIN_HW_HINTS);
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to get hw hints %d\n", err);
+ return err;
+ }
+
+ result->admin_completion_timeout = resp.u.hw_hints.admin_completion_timeout;
+ result->driver_watchdog_timeout = resp.u.hw_hints.driver_watchdog_timeout;
+ result->mmio_read_timeout = resp.u.hw_hints.mmio_read_timeout;
+ result->poll_interval = resp.u.hw_hints.poll_interval;
+
+ return 0;
+}
+
+static int efa_com_set_feature_ex(struct efa_com_dev *edev,
+ struct efa_admin_set_feature_resp *set_resp,
+ struct efa_admin_set_feature_cmd *set_cmd,
+ enum efa_admin_aq_feature_id feature_id,
+ dma_addr_t control_buf_dma_addr,
+ u32 control_buff_size)
+{
+ struct efa_com_admin_queue *aq;
+ int err;
+
+ if (!efa_com_check_supported_feature_id(edev, feature_id)) {
+ ibdev_err(edev->efa_dev, "Feature %d isn't supported\n",
+ feature_id);
+ return -EOPNOTSUPP;
+ }
+
+ aq = &edev->aq;
+
+ set_cmd->aq_common_descriptor.opcode = EFA_ADMIN_SET_FEATURE;
+ if (control_buff_size) {
+ set_cmd->aq_common_descriptor.flags =
+ EFA_ADMIN_AQ_COMMON_DESC_CTRL_DATA_INDIRECT_MASK;
+ efa_com_set_dma_addr(control_buf_dma_addr,
+ &set_cmd->control_buffer.address.mem_addr_high,
+ &set_cmd->control_buffer.address.mem_addr_low);
+ }
+
+ set_cmd->control_buffer.length = control_buff_size;
+ set_cmd->feature_common.feature_id = feature_id;
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)set_cmd,
+ sizeof(*set_cmd),
+ (struct efa_admin_acq_entry *)set_resp,
+ sizeof(*set_resp));
+
+ if (err)
+ ibdev_err(edev->efa_dev,
+ "Failed to submit set_feature command %d error: %d\n",
+ feature_id, err);
+
+ return 0;
+}
+
+static int efa_com_set_feature(struct efa_com_dev *edev,
+ struct efa_admin_set_feature_resp *set_resp,
+ struct efa_admin_set_feature_cmd *set_cmd,
+ enum efa_admin_aq_feature_id feature_id)
+{
+ return efa_com_set_feature_ex(edev, set_resp, set_cmd, feature_id,
+ 0, 0);
+}
+
+int efa_com_set_aenq_config(struct efa_com_dev *edev, u32 groups)
+{
+ struct efa_admin_get_feature_resp get_resp;
+ struct efa_admin_set_feature_resp set_resp;
+ struct efa_admin_set_feature_cmd cmd = {};
+ int err;
+
+ ibdev_dbg(edev->efa_dev, "Configuring aenq with groups[%#x]\n", groups);
+
+ err = efa_com_get_feature(edev, &get_resp, EFA_ADMIN_AENQ_CONFIG);
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to get aenq attributes: %d\n",
+ err);
+ return err;
+ }
+
+ ibdev_dbg(edev->efa_dev,
+ "Get aenq groups: supported[%#x] enabled[%#x]\n",
+ get_resp.u.aenq.supported_groups,
+ get_resp.u.aenq.enabled_groups);
+
+ if ((get_resp.u.aenq.supported_groups & groups) != groups) {
+ ibdev_err(edev->efa_dev,
+ "Trying to set unsupported aenq groups[%#x] supported[%#x]\n",
+ groups, get_resp.u.aenq.supported_groups);
+ return -EOPNOTSUPP;
+ }
+
+ cmd.u.aenq.enabled_groups = groups;
+ err = efa_com_set_feature(edev, &set_resp, &cmd,
+ EFA_ADMIN_AENQ_CONFIG);
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to set aenq attributes: %d\n",
+ err);
+ return err;
+ }
+
+ return 0;
+}
+
+int efa_com_alloc_pd(struct efa_com_dev *edev,
+ struct efa_com_alloc_pd_result *result)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_admin_alloc_pd_cmd cmd = {};
+ struct efa_admin_alloc_pd_resp resp;
+ int err;
+
+ cmd.aq_common_descriptor.opcode = EFA_ADMIN_ALLOC_PD;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&cmd,
+ sizeof(cmd),
+ (struct efa_admin_acq_entry *)&resp,
+ sizeof(resp));
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to allocate pd[%d]\n", err);
+ return err;
+ }
+
+ result->pdn = resp.pd;
+
+ return 0;
+}
+
+int efa_com_dealloc_pd(struct efa_com_dev *edev,
+ struct efa_com_dealloc_pd_params *params)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_admin_dealloc_pd_cmd cmd = {};
+ struct efa_admin_dealloc_pd_resp resp;
+ int err;
+
+ cmd.aq_common_descriptor.opcode = EFA_ADMIN_DEALLOC_PD;
+ cmd.pd = params->pdn;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&cmd,
+ sizeof(cmd),
+ (struct efa_admin_acq_entry *)&resp,
+ sizeof(resp));
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to deallocate pd-%u [%d]\n",
+ cmd.pd, err);
+ return err;
+ }
+
+ return 0;
+}
+
+int efa_com_alloc_uar(struct efa_com_dev *edev,
+ struct efa_com_alloc_uar_result *result)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_admin_alloc_uar_cmd cmd = {};
+ struct efa_admin_alloc_uar_resp resp;
+ int err;
+
+ cmd.aq_common_descriptor.opcode = EFA_ADMIN_ALLOC_UAR;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&cmd,
+ sizeof(cmd),
+ (struct efa_admin_acq_entry *)&resp,
+ sizeof(resp));
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to allocate uar[%d]\n", err);
+ return err;
+ }
+
+ result->uarn = resp.uar;
+
+ return 0;
+}
+
+int efa_com_dealloc_uar(struct efa_com_dev *edev,
+ struct efa_com_dealloc_uar_params *params)
+{
+ struct efa_com_admin_queue *aq = &edev->aq;
+ struct efa_admin_dealloc_uar_cmd cmd = {};
+ struct efa_admin_dealloc_uar_resp resp;
+ int err;
+
+ cmd.aq_common_descriptor.opcode = EFA_ADMIN_DEALLOC_UAR;
+ cmd.uar = params->uarn;
+
+ err = efa_com_cmd_exec(aq,
+ (struct efa_admin_aq_entry *)&cmd,
+ sizeof(cmd),
+ (struct efa_admin_acq_entry *)&resp,
+ sizeof(resp));
+ if (err) {
+ ibdev_err(edev->efa_dev, "Failed to deallocate uar-%u [%d]\n",
+ cmd.uar, err);
+ return err;
+ }
+
+ return 0;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#ifndef _EFA_COM_CMD_H_
+#define _EFA_COM_CMD_H_
+
+#include "efa_com.h"
+
+#define EFA_GID_SIZE 16
+
+struct efa_com_create_qp_params {
+ u64 rq_base_addr;
+ u32 send_cq_idx;
+ u32 recv_cq_idx;
+ /*
+ * Send descriptor ring size in bytes,
+ * sufficient for user-provided number of WQEs and SGL size
+ */
+ u32 sq_ring_size_in_bytes;
+ /* Max number of WQEs that will be posted on send queue */
+ u32 sq_depth;
+ /* Recv descriptor ring size in bytes */
+ u32 rq_ring_size_in_bytes;
+ u32 rq_depth;
+ u16 pd;
+ u16 uarn;
+ u8 qp_type;
+};
+
+struct efa_com_create_qp_result {
+ u32 qp_handle;
+ u32 qp_num;
+ u32 sq_db_offset;
+ u32 rq_db_offset;
+ u32 llq_descriptors_offset;
+ u16 send_sub_cq_idx;
+ u16 recv_sub_cq_idx;
+};
+
+struct efa_com_modify_qp_params {
+ u32 modify_mask;
+ u32 qp_handle;
+ u32 qp_state;
+ u32 cur_qp_state;
+ u32 qkey;
+ u32 sq_psn;
+ u8 sq_drained_async_notify;
+};
+
+struct efa_com_query_qp_params {
+ u32 qp_handle;
+};
+
+struct efa_com_query_qp_result {
+ u32 qp_state;
+ u32 qkey;
+ u32 sq_draining;
+ u32 sq_psn;
+};
+
+struct efa_com_destroy_qp_params {
+ u32 qp_handle;
+};
+
+struct efa_com_create_cq_params {
+ /* cq physical base address in OS memory */
+ dma_addr_t dma_addr;
+ /* completion queue depth in # of entries */
+ u16 cq_depth;
+ u16 num_sub_cqs;
+ u16 uarn;
+ u8 entry_size_in_bytes;
+};
+
+struct efa_com_create_cq_result {
+ /* cq identifier */
+ u16 cq_idx;
+ /* actual cq depth in # of entries */
+ u16 actual_depth;
+};
+
+struct efa_com_destroy_cq_params {
+ u16 cq_idx;
+};
+
+struct efa_com_create_ah_params {
+ u16 pdn;
+ /* Destination address in network byte order */
+ u8 dest_addr[EFA_GID_SIZE];
+};
+
+struct efa_com_create_ah_result {
+ u16 ah;
+};
+
+struct efa_com_destroy_ah_params {
+ u16 ah;
+ u16 pdn;
+};
+
+struct efa_com_get_network_attr_result {
+ u8 addr[EFA_GID_SIZE];
+ u32 mtu;
+};
+
+struct efa_com_get_device_attr_result {
+ u64 page_size_cap;
+ u64 max_mr_pages;
+ u32 fw_version;
+ u32 admin_api_version;
+ u32 device_version;
+ u32 supported_features;
+ u32 phys_addr_width;
+ u32 virt_addr_width;
+ u32 max_qp;
+ u32 max_sq_depth; /* wqes */
+ u32 max_rq_depth; /* wqes */
+ u32 max_cq;
+ u32 max_cq_depth; /* cqes */
+ u32 inline_buf_size;
+ u32 max_mr;
+ u32 max_pd;
+ u32 max_ah;
+ u32 max_llq_size;
+ u16 sub_cqs_per_cq;
+ u16 max_sq_sge;
+ u16 max_rq_sge;
+ u8 db_bar;
+};
+
+struct efa_com_get_hw_hints_result {
+ u16 mmio_read_timeout;
+ u16 driver_watchdog_timeout;
+ u16 admin_completion_timeout;
+ u16 poll_interval;
+ u32 reserved[4];
+};
+
+struct efa_com_mem_addr {
+ u32 mem_addr_low;
+ u32 mem_addr_high;
+};
+
+/* Used at indirect mode page list chunks for chaining */
+struct efa_com_ctrl_buff_info {
+ /* indicates length of the buffer pointed by control_buffer_address. */
+ u32 length;
+ /* points to control buffer (direct or indirect) */
+ struct efa_com_mem_addr address;
+};
+
+struct efa_com_reg_mr_params {
+ /* Memory region length, in bytes. */
+ u64 mr_length_in_bytes;
+ /* IO Virtual Address associated with this MR. */
+ u64 iova;
+ /* words 8:15: Physical Buffer List, each element is page-aligned. */
+ union {
+ /*
+ * Inline array of physical addresses of app pages
+ * (optimization for short region reservations)
+ */
+ u64 inline_pbl_array[4];
+ /*
+ * Describes the next physically contiguous chunk of indirect
+ * page list. A page list contains physical addresses of command
+ * data pages. Data pages are 4KB; page list chunks are
+ * variable-sized.
+ */
+ struct efa_com_ctrl_buff_info pbl;
+ } pbl;
+ /* number of pages in PBL (redundant, could be calculated) */
+ u32 page_num;
+ /* Protection Domain */
+ u16 pd;
+ /*
+ * phys_page_size_shift - page size is (1 << phys_page_size_shift)
+ * Page size is used for building the Virtual to Physical
+ * address mapping
+ */
+ u8 page_shift;
+ /*
+ * permissions
+ * 0: local_write_enable - Write permissions: value of 1 needed
+ * for RQ buffers and for RDMA write:1: reserved1 - remote
+ * access flags, etc
+ */
+ u8 permissions;
+ u8 inline_pbl;
+ u8 indirect;
+};
+
+struct efa_com_reg_mr_result {
+ /*
+ * To be used in conjunction with local buffers references in SQ and
+ * RQ WQE
+ */
+ u32 l_key;
+ /*
+ * To be used in incoming RDMA semantics messages to refer to remotely
+ * accessed memory region
+ */
+ u32 r_key;
+};
+
+struct efa_com_dereg_mr_params {
+ u32 l_key;
+};
+
+struct efa_com_alloc_pd_result {
+ u16 pdn;
+};
+
+struct efa_com_dealloc_pd_params {
+ u16 pdn;
+};
+
+struct efa_com_alloc_uar_result {
+ u16 uarn;
+};
+
+struct efa_com_dealloc_uar_params {
+ u16 uarn;
+};
+
+void efa_com_set_dma_addr(dma_addr_t addr, u32 *addr_high, u32 *addr_low);
+int efa_com_create_qp(struct efa_com_dev *edev,
+ struct efa_com_create_qp_params *params,
+ struct efa_com_create_qp_result *res);
+int efa_com_modify_qp(struct efa_com_dev *edev,
+ struct efa_com_modify_qp_params *params);
+int efa_com_query_qp(struct efa_com_dev *edev,
+ struct efa_com_query_qp_params *params,
+ struct efa_com_query_qp_result *result);
+int efa_com_destroy_qp(struct efa_com_dev *edev,
+ struct efa_com_destroy_qp_params *params);
+int efa_com_create_cq(struct efa_com_dev *edev,
+ struct efa_com_create_cq_params *params,
+ struct efa_com_create_cq_result *result);
+int efa_com_destroy_cq(struct efa_com_dev *edev,
+ struct efa_com_destroy_cq_params *params);
+int efa_com_register_mr(struct efa_com_dev *edev,
+ struct efa_com_reg_mr_params *params,
+ struct efa_com_reg_mr_result *result);
+int efa_com_dereg_mr(struct efa_com_dev *edev,
+ struct efa_com_dereg_mr_params *params);
+int efa_com_create_ah(struct efa_com_dev *edev,
+ struct efa_com_create_ah_params *params,
+ struct efa_com_create_ah_result *result);
+int efa_com_destroy_ah(struct efa_com_dev *edev,
+ struct efa_com_destroy_ah_params *params);
+int efa_com_get_network_attr(struct efa_com_dev *edev,
+ struct efa_com_get_network_attr_result *result);
+int efa_com_get_device_attr(struct efa_com_dev *edev,
+ struct efa_com_get_device_attr_result *result);
+int efa_com_get_hw_hints(struct efa_com_dev *edev,
+ struct efa_com_get_hw_hints_result *result);
+int efa_com_set_aenq_config(struct efa_com_dev *edev, u32 groups);
+int efa_com_alloc_pd(struct efa_com_dev *edev,
+ struct efa_com_alloc_pd_result *result);
+int efa_com_dealloc_pd(struct efa_com_dev *edev,
+ struct efa_com_dealloc_pd_params *params);
+int efa_com_alloc_uar(struct efa_com_dev *edev,
+ struct efa_com_alloc_uar_result *result);
+int efa_com_dealloc_uar(struct efa_com_dev *edev,
+ struct efa_com_dealloc_uar_params *params);
+
+#endif /* _EFA_COM_CMD_H_ */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#ifndef _EFA_COMMON_H_
+#define _EFA_COMMON_H_
+
+#define EFA_COMMON_SPEC_VERSION_MAJOR 2
+#define EFA_COMMON_SPEC_VERSION_MINOR 0
+
+struct efa_common_mem_addr {
+ u32 mem_addr_low;
+
+ u32 mem_addr_high;
+};
+
+#endif /* _EFA_COMMON_H_ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+
+#include <rdma/ib_user_verbs.h>
+
+#include "efa.h"
+
+#define PCI_DEV_ID_EFA_VF 0xefa0
+
+static const struct pci_device_id efa_pci_tbl[] = {
+ { PCI_VDEVICE(AMAZON, PCI_DEV_ID_EFA_VF) },
+ { }
+};
+
+MODULE_AUTHOR("Amazon.com, Inc. or its affiliates");
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_DESCRIPTION(DEVICE_NAME);
+MODULE_DEVICE_TABLE(pci, efa_pci_tbl);
+
+#define EFA_REG_BAR 0
+#define EFA_MEM_BAR 2
+#define EFA_BASE_BAR_MASK (BIT(EFA_REG_BAR) | BIT(EFA_MEM_BAR))
+
+#define EFA_AENQ_ENABLED_GROUPS \
+ (BIT(EFA_ADMIN_FATAL_ERROR) | BIT(EFA_ADMIN_WARNING) | \
+ BIT(EFA_ADMIN_NOTIFICATION) | BIT(EFA_ADMIN_KEEP_ALIVE))
+
+static void efa_update_network_attr(struct efa_dev *dev,
+ struct efa_com_get_network_attr_result *network_attr)
+{
+ memcpy(dev->addr, network_attr->addr, sizeof(network_attr->addr));
+ dev->mtu = network_attr->mtu;
+
+ dev_dbg(&dev->pdev->dev, "Full address %pI6\n", dev->addr);
+}
+
+/* This handler will called for unknown event group or unimplemented handlers */
+static void unimplemented_aenq_handler(void *data,
+ struct efa_admin_aenq_entry *aenq_e)
+{
+ struct efa_dev *dev = (struct efa_dev *)data;
+
+ ibdev_err(&dev->ibdev,
+ "Unknown event was received or event with unimplemented handler\n");
+}
+
+static void efa_keep_alive(void *data, struct efa_admin_aenq_entry *aenq_e)
+{
+ struct efa_dev *dev = (struct efa_dev *)data;
+
+ atomic64_inc(&dev->stats.keep_alive_rcvd);
+}
+
+static struct efa_aenq_handlers aenq_handlers = {
+ .handlers = {
+ [EFA_ADMIN_KEEP_ALIVE] = efa_keep_alive,
+ },
+ .unimplemented_handler = unimplemented_aenq_handler
+};
+
+static void efa_release_bars(struct efa_dev *dev, int bars_mask)
+{
+ struct pci_dev *pdev = dev->pdev;
+ int release_bars;
+
+ release_bars = pci_select_bars(pdev, IORESOURCE_MEM) & bars_mask;
+ pci_release_selected_regions(pdev, release_bars);
+}
+
+static irqreturn_t efa_intr_msix_mgmnt(int irq, void *data)
+{
+ struct efa_dev *dev = data;
+
+ efa_com_admin_q_comp_intr_handler(&dev->edev);
+ efa_com_aenq_intr_handler(&dev->edev, data);
+
+ return IRQ_HANDLED;
+}
+
+static int efa_request_mgmnt_irq(struct efa_dev *dev)
+{
+ struct efa_irq *irq;
+ int err;
+
+ irq = &dev->admin_irq;
+ err = request_irq(irq->vector, irq->handler, 0, irq->name,
+ irq->data);
+ if (err) {
+ dev_err(&dev->pdev->dev, "Failed to request admin irq (%d)\n",
+ err);
+ return err;
+ }
+
+ dev_dbg(&dev->pdev->dev, "Set affinity hint of mgmnt irq to %*pbl (irq vector: %d)\n",
+ nr_cpumask_bits, &irq->affinity_hint_mask, irq->vector);
+ irq_set_affinity_hint(irq->vector, &irq->affinity_hint_mask);
+
+ return err;
+}
+
+static void efa_setup_mgmnt_irq(struct efa_dev *dev)
+{
+ u32 cpu;
+
+ snprintf(dev->admin_irq.name, EFA_IRQNAME_SIZE,
+ "efa-mgmnt@pci:%s", pci_name(dev->pdev));
+ dev->admin_irq.handler = efa_intr_msix_mgmnt;
+ dev->admin_irq.data = dev;
+ dev->admin_irq.vector =
+ pci_irq_vector(dev->pdev, dev->admin_msix_vector_idx);
+ cpu = cpumask_first(cpu_online_mask);
+ dev->admin_irq.cpu = cpu;
+ cpumask_set_cpu(cpu,
+ &dev->admin_irq.affinity_hint_mask);
+ dev_info(&dev->pdev->dev, "Setup irq:0x%p vector:%d name:%s\n",
+ &dev->admin_irq,
+ dev->admin_irq.vector,
+ dev->admin_irq.name);
+}
+
+static void efa_free_mgmnt_irq(struct efa_dev *dev)
+{
+ struct efa_irq *irq;
+
+ irq = &dev->admin_irq;
+ irq_set_affinity_hint(irq->vector, NULL);
+ free_irq(irq->vector, irq->data);
+}
+
+static int efa_set_mgmnt_irq(struct efa_dev *dev)
+{
+ efa_setup_mgmnt_irq(dev);
+
+ return efa_request_mgmnt_irq(dev);
+}
+
+static int efa_request_doorbell_bar(struct efa_dev *dev)
+{
+ u8 db_bar_idx = dev->dev_attr.db_bar;
+ struct pci_dev *pdev = dev->pdev;
+ int bars;
+ int err;
+
+ if (!(BIT(db_bar_idx) & EFA_BASE_BAR_MASK)) {
+ bars = pci_select_bars(pdev, IORESOURCE_MEM) & BIT(db_bar_idx);
+
+ err = pci_request_selected_regions(pdev, bars, DRV_MODULE_NAME);
+ if (err) {
+ dev_err(&dev->pdev->dev,
+ "pci_request_selected_regions for bar %d failed %d\n",
+ db_bar_idx, err);
+ return err;
+ }
+ }
+
+ dev->db_bar_addr = pci_resource_start(dev->pdev, db_bar_idx);
+ dev->db_bar_len = pci_resource_len(dev->pdev, db_bar_idx);
+
+ return 0;
+}
+
+static void efa_release_doorbell_bar(struct efa_dev *dev)
+{
+ if (!(BIT(dev->dev_attr.db_bar) & EFA_BASE_BAR_MASK))
+ efa_release_bars(dev, BIT(dev->dev_attr.db_bar));
+}
+
+static void efa_update_hw_hints(struct efa_dev *dev,
+ struct efa_com_get_hw_hints_result *hw_hints)
+{
+ struct efa_com_dev *edev = &dev->edev;
+
+ if (hw_hints->mmio_read_timeout)
+ edev->mmio_read.mmio_read_timeout =
+ hw_hints->mmio_read_timeout * 1000;
+
+ if (hw_hints->poll_interval)
+ edev->aq.poll_interval = hw_hints->poll_interval;
+
+ if (hw_hints->admin_completion_timeout)
+ edev->aq.completion_timeout =
+ hw_hints->admin_completion_timeout;
+}
+
+static void efa_stats_init(struct efa_dev *dev)
+{
+ atomic64_t *s = (atomic64_t *)&dev->stats;
+ int i;
+
+ for (i = 0; i < sizeof(dev->stats) / sizeof(*s); i++, s++)
+ atomic64_set(s, 0);
+}
+
+static const struct ib_device_ops efa_dev_ops = {
+ .alloc_pd = efa_alloc_pd,
+ .alloc_ucontext = efa_alloc_ucontext,
+ .create_ah = efa_create_ah,
+ .create_cq = efa_create_cq,
+ .create_qp = efa_create_qp,
+ .dealloc_pd = efa_dealloc_pd,
+ .dealloc_ucontext = efa_dealloc_ucontext,
+ .dereg_mr = efa_dereg_mr,
+ .destroy_ah = efa_destroy_ah,
+ .destroy_cq = efa_destroy_cq,
+ .destroy_qp = efa_destroy_qp,
+ .get_link_layer = efa_port_link_layer,
+ .get_port_immutable = efa_get_port_immutable,
+ .mmap = efa_mmap,
+ .modify_qp = efa_modify_qp,
+ .query_device = efa_query_device,
+ .query_gid = efa_query_gid,
+ .query_pkey = efa_query_pkey,
+ .query_port = efa_query_port,
+ .query_qp = efa_query_qp,
+ .reg_user_mr = efa_reg_mr,
+
+ INIT_RDMA_OBJ_SIZE(ib_ah, efa_ah, ibah),
+ INIT_RDMA_OBJ_SIZE(ib_pd, efa_pd, ibpd),
+ INIT_RDMA_OBJ_SIZE(ib_ucontext, efa_ucontext, ibucontext),
+};
+
+static int efa_ib_device_add(struct efa_dev *dev)
+{
+ struct efa_com_get_network_attr_result network_attr;
+ struct efa_com_get_hw_hints_result hw_hints;
+ struct pci_dev *pdev = dev->pdev;
+ int err;
+
+ efa_stats_init(dev);
+
+ err = efa_com_get_device_attr(&dev->edev, &dev->dev_attr);
+ if (err)
+ return err;
+
+ dev_dbg(&dev->pdev->dev, "Doorbells bar (%d)\n", dev->dev_attr.db_bar);
+ err = efa_request_doorbell_bar(dev);
+ if (err)
+ return err;
+
+ err = efa_com_get_network_attr(&dev->edev, &network_attr);
+ if (err)
+ goto err_release_doorbell_bar;
+
+ efa_update_network_attr(dev, &network_attr);
+
+ err = efa_com_get_hw_hints(&dev->edev, &hw_hints);
+ if (err)
+ goto err_release_doorbell_bar;
+
+ efa_update_hw_hints(dev, &hw_hints);
+
+ /* Try to enable all the available aenq groups */
+ err = efa_com_set_aenq_config(&dev->edev, EFA_AENQ_ENABLED_GROUPS);
+ if (err)
+ goto err_release_doorbell_bar;
+
+ dev->ibdev.owner = THIS_MODULE;
+ dev->ibdev.node_type = RDMA_NODE_UNSPECIFIED;
+ dev->ibdev.phys_port_cnt = 1;
+ dev->ibdev.num_comp_vectors = 1;
+ dev->ibdev.dev.parent = &pdev->dev;
+ dev->ibdev.uverbs_abi_ver = EFA_UVERBS_ABI_VERSION;
+
+ dev->ibdev.uverbs_cmd_mask =
+ (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) |
+ (1ull << IB_USER_VERBS_CMD_QUERY_DEVICE) |
+ (1ull << IB_USER_VERBS_CMD_QUERY_PORT) |
+ (1ull << IB_USER_VERBS_CMD_ALLOC_PD) |
+ (1ull << IB_USER_VERBS_CMD_DEALLOC_PD) |
+ (1ull << IB_USER_VERBS_CMD_REG_MR) |
+ (1ull << IB_USER_VERBS_CMD_DEREG_MR) |
+ (1ull << IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL) |
+ (1ull << IB_USER_VERBS_CMD_CREATE_CQ) |
+ (1ull << IB_USER_VERBS_CMD_DESTROY_CQ) |
+ (1ull << IB_USER_VERBS_CMD_CREATE_QP) |
+ (1ull << IB_USER_VERBS_CMD_MODIFY_QP) |
+ (1ull << IB_USER_VERBS_CMD_QUERY_QP) |
+ (1ull << IB_USER_VERBS_CMD_DESTROY_QP) |
+ (1ull << IB_USER_VERBS_CMD_CREATE_AH) |
+ (1ull << IB_USER_VERBS_CMD_DESTROY_AH);
+
+ dev->ibdev.uverbs_ex_cmd_mask =
+ (1ull << IB_USER_VERBS_EX_CMD_QUERY_DEVICE);
+
+ dev->ibdev.driver_id = RDMA_DRIVER_EFA;
+ ib_set_device_ops(&dev->ibdev, &efa_dev_ops);
+
+ err = ib_register_device(&dev->ibdev, "efa_%d");
+ if (err)
+ goto err_release_doorbell_bar;
+
+ ibdev_info(&dev->ibdev, "IB device registered\n");
+
+ return 0;
+
+err_release_doorbell_bar:
+ efa_release_doorbell_bar(dev);
+ return err;
+}
+
+static void efa_ib_device_remove(struct efa_dev *dev)
+{
+ efa_com_dev_reset(&dev->edev, EFA_REGS_RESET_NORMAL);
+ ibdev_info(&dev->ibdev, "Unregister ib device\n");
+ ib_unregister_device(&dev->ibdev);
+ efa_release_doorbell_bar(dev);
+}
+
+static void efa_disable_msix(struct efa_dev *dev)
+{
+ pci_free_irq_vectors(dev->pdev);
+}
+
+static int efa_enable_msix(struct efa_dev *dev)
+{
+ int msix_vecs, irq_num;
+
+ /* Reserve the max msix vectors we might need */
+ msix_vecs = EFA_NUM_MSIX_VEC;
+ dev_dbg(&dev->pdev->dev, "Trying to enable MSI-X, vectors %d\n",
+ msix_vecs);
+
+ dev->admin_msix_vector_idx = EFA_MGMNT_MSIX_VEC_IDX;
+ irq_num = pci_alloc_irq_vectors(dev->pdev, msix_vecs,
+ msix_vecs, PCI_IRQ_MSIX);
+
+ if (irq_num < 0) {
+ dev_err(&dev->pdev->dev, "Failed to enable MSI-X. irq_num %d\n",
+ irq_num);
+ return -ENOSPC;
+ }
+
+ if (irq_num != msix_vecs) {
+ dev_err(&dev->pdev->dev,
+ "Allocated %d MSI-X (out of %d requested)\n",
+ irq_num, msix_vecs);
+ return -ENOSPC;
+ }
+
+ return 0;
+}
+
+static int efa_device_init(struct efa_com_dev *edev, struct pci_dev *pdev)
+{
+ int dma_width;
+ int err;
+
+ err = efa_com_dev_reset(edev, EFA_REGS_RESET_NORMAL);
+ if (err)
+ return err;
+
+ err = efa_com_validate_version(edev);
+ if (err)
+ return err;
+
+ dma_width = efa_com_get_dma_width(edev);
+ if (dma_width < 0) {
+ err = dma_width;
+ return err;
+ }
+
+ err = pci_set_dma_mask(pdev, DMA_BIT_MASK(dma_width));
+ if (err) {
+ dev_err(&pdev->dev, "pci_set_dma_mask failed %d\n", err);
+ return err;
+ }
+
+ err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(dma_width));
+ if (err) {
+ dev_err(&pdev->dev,
+ "err_pci_set_consistent_dma_mask failed %d\n",
+ err);
+ return err;
+ }
+
+ return 0;
+}
+
+static struct efa_dev *efa_probe_device(struct pci_dev *pdev)
+{
+ struct efa_com_dev *edev;
+ struct efa_dev *dev;
+ int bars;
+ int err;
+
+ err = pci_enable_device_mem(pdev);
+ if (err) {
+ dev_err(&pdev->dev, "pci_enable_device_mem() failed!\n");
+ return ERR_PTR(err);
+ }
+
+ pci_set_master(pdev);
+
+ dev = ib_alloc_device(efa_dev, ibdev);
+ if (!dev) {
+ dev_err(&pdev->dev, "Device alloc failed\n");
+ err = -ENOMEM;
+ goto err_disable_device;
+ }
+
+ pci_set_drvdata(pdev, dev);
+ edev = &dev->edev;
+ edev->efa_dev = dev;
+ edev->dmadev = &pdev->dev;
+ dev->pdev = pdev;
+
+ bars = pci_select_bars(pdev, IORESOURCE_MEM) & EFA_BASE_BAR_MASK;
+ err = pci_request_selected_regions(pdev, bars, DRV_MODULE_NAME);
+ if (err) {
+ dev_err(&pdev->dev, "pci_request_selected_regions failed %d\n",
+ err);
+ goto err_ibdev_destroy;
+ }
+
+ dev->reg_bar_addr = pci_resource_start(pdev, EFA_REG_BAR);
+ dev->reg_bar_len = pci_resource_len(pdev, EFA_REG_BAR);
+ dev->mem_bar_addr = pci_resource_start(pdev, EFA_MEM_BAR);
+ dev->mem_bar_len = pci_resource_len(pdev, EFA_MEM_BAR);
+
+ edev->reg_bar = devm_ioremap(&pdev->dev,
+ dev->reg_bar_addr,
+ dev->reg_bar_len);
+ if (!edev->reg_bar) {
+ dev_err(&pdev->dev, "Failed to remap register bar\n");
+ err = -EFAULT;
+ goto err_release_bars;
+ }
+
+ err = efa_com_mmio_reg_read_init(edev);
+ if (err) {
+ dev_err(&pdev->dev, "Failed to init readless MMIO\n");
+ goto err_iounmap;
+ }
+
+ err = efa_device_init(edev, pdev);
+ if (err) {
+ dev_err(&pdev->dev, "EFA device init failed\n");
+ if (err == -ETIME)
+ err = -EPROBE_DEFER;
+ goto err_reg_read_destroy;
+ }
+
+ err = efa_enable_msix(dev);
+ if (err)
+ goto err_reg_read_destroy;
+
+ edev->aq.msix_vector_idx = dev->admin_msix_vector_idx;
+ edev->aenq.msix_vector_idx = dev->admin_msix_vector_idx;
+
+ err = efa_set_mgmnt_irq(dev);
+ if (err)
+ goto err_disable_msix;
+
+ err = efa_com_admin_init(edev, &aenq_handlers);
+ if (err)
+ goto err_free_mgmnt_irq;
+
+ return dev;
+
+err_free_mgmnt_irq:
+ efa_free_mgmnt_irq(dev);
+err_disable_msix:
+ efa_disable_msix(dev);
+err_reg_read_destroy:
+ efa_com_mmio_reg_read_destroy(edev);
+err_iounmap:
+ devm_iounmap(&pdev->dev, edev->reg_bar);
+err_release_bars:
+ efa_release_bars(dev, EFA_BASE_BAR_MASK);
+err_ibdev_destroy:
+ ib_dealloc_device(&dev->ibdev);
+err_disable_device:
+ pci_disable_device(pdev);
+ return ERR_PTR(err);
+}
+
+static void efa_remove_device(struct pci_dev *pdev)
+{
+ struct efa_dev *dev = pci_get_drvdata(pdev);
+ struct efa_com_dev *edev;
+
+ edev = &dev->edev;
+ efa_com_admin_destroy(edev);
+ efa_free_mgmnt_irq(dev);
+ efa_disable_msix(dev);
+ efa_com_mmio_reg_read_destroy(edev);
+ devm_iounmap(&pdev->dev, edev->reg_bar);
+ efa_release_bars(dev, EFA_BASE_BAR_MASK);
+ ib_dealloc_device(&dev->ibdev);
+ pci_disable_device(pdev);
+}
+
+static int efa_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
+{
+ struct efa_dev *dev;
+ int err;
+
+ dev = efa_probe_device(pdev);
+ if (IS_ERR(dev))
+ return PTR_ERR(dev);
+
+ err = efa_ib_device_add(dev);
+ if (err)
+ goto err_remove_device;
+
+ return 0;
+
+err_remove_device:
+ efa_remove_device(pdev);
+ return err;
+}
+
+static void efa_remove(struct pci_dev *pdev)
+{
+ struct efa_dev *dev = pci_get_drvdata(pdev);
+
+ efa_ib_device_remove(dev);
+ efa_remove_device(pdev);
+}
+
+static struct pci_driver efa_pci_driver = {
+ .name = DRV_MODULE_NAME,
+ .id_table = efa_pci_tbl,
+ .probe = efa_probe,
+ .remove = efa_remove,
+};
+
+module_pci_driver(efa_pci_driver);
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause */
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#ifndef _EFA_REGS_H_
+#define _EFA_REGS_H_
+
+enum efa_regs_reset_reason_types {
+ EFA_REGS_RESET_NORMAL = 0,
+ /* Keep alive timeout */
+ EFA_REGS_RESET_KEEP_ALIVE_TO = 1,
+ EFA_REGS_RESET_ADMIN_TO = 2,
+ EFA_REGS_RESET_INIT_ERR = 3,
+ EFA_REGS_RESET_DRIVER_INVALID_STATE = 4,
+ EFA_REGS_RESET_OS_TRIGGER = 5,
+ EFA_REGS_RESET_SHUTDOWN = 6,
+ EFA_REGS_RESET_USER_TRIGGER = 7,
+ EFA_REGS_RESET_GENERIC = 8,
+};
+
+/* efa_registers offsets */
+
+/* 0 base */
+#define EFA_REGS_VERSION_OFF 0x0
+#define EFA_REGS_CONTROLLER_VERSION_OFF 0x4
+#define EFA_REGS_CAPS_OFF 0x8
+#define EFA_REGS_AQ_BASE_LO_OFF 0x10
+#define EFA_REGS_AQ_BASE_HI_OFF 0x14
+#define EFA_REGS_AQ_CAPS_OFF 0x18
+#define EFA_REGS_ACQ_BASE_LO_OFF 0x20
+#define EFA_REGS_ACQ_BASE_HI_OFF 0x24
+#define EFA_REGS_ACQ_CAPS_OFF 0x28
+#define EFA_REGS_AQ_PROD_DB_OFF 0x2c
+#define EFA_REGS_AENQ_CAPS_OFF 0x34
+#define EFA_REGS_AENQ_BASE_LO_OFF 0x38
+#define EFA_REGS_AENQ_BASE_HI_OFF 0x3c
+#define EFA_REGS_AENQ_CONS_DB_OFF 0x40
+#define EFA_REGS_INTR_MASK_OFF 0x4c
+#define EFA_REGS_DEV_CTL_OFF 0x54
+#define EFA_REGS_DEV_STS_OFF 0x58
+#define EFA_REGS_MMIO_REG_READ_OFF 0x5c
+#define EFA_REGS_MMIO_RESP_LO_OFF 0x60
+#define EFA_REGS_MMIO_RESP_HI_OFF 0x64
+
+/* version register */
+#define EFA_REGS_VERSION_MINOR_VERSION_MASK 0xff
+#define EFA_REGS_VERSION_MAJOR_VERSION_SHIFT 8
+#define EFA_REGS_VERSION_MAJOR_VERSION_MASK 0xff00
+
+/* controller_version register */
+#define EFA_REGS_CONTROLLER_VERSION_SUBMINOR_VERSION_MASK 0xff
+#define EFA_REGS_CONTROLLER_VERSION_MINOR_VERSION_SHIFT 8
+#define EFA_REGS_CONTROLLER_VERSION_MINOR_VERSION_MASK 0xff00
+#define EFA_REGS_CONTROLLER_VERSION_MAJOR_VERSION_SHIFT 16
+#define EFA_REGS_CONTROLLER_VERSION_MAJOR_VERSION_MASK 0xff0000
+#define EFA_REGS_CONTROLLER_VERSION_IMPL_ID_SHIFT 24
+#define EFA_REGS_CONTROLLER_VERSION_IMPL_ID_MASK 0xff000000
+
+/* caps register */
+#define EFA_REGS_CAPS_CONTIGUOUS_QUEUE_REQUIRED_MASK 0x1
+#define EFA_REGS_CAPS_RESET_TIMEOUT_SHIFT 1
+#define EFA_REGS_CAPS_RESET_TIMEOUT_MASK 0x3e
+#define EFA_REGS_CAPS_DMA_ADDR_WIDTH_SHIFT 8
+#define EFA_REGS_CAPS_DMA_ADDR_WIDTH_MASK 0xff00
+#define EFA_REGS_CAPS_ADMIN_CMD_TO_SHIFT 16
+#define EFA_REGS_CAPS_ADMIN_CMD_TO_MASK 0xf0000
+
+/* aq_caps register */
+#define EFA_REGS_AQ_CAPS_AQ_DEPTH_MASK 0xffff
+#define EFA_REGS_AQ_CAPS_AQ_ENTRY_SIZE_SHIFT 16
+#define EFA_REGS_AQ_CAPS_AQ_ENTRY_SIZE_MASK 0xffff0000
+
+/* acq_caps register */
+#define EFA_REGS_ACQ_CAPS_ACQ_DEPTH_MASK 0xffff
+#define EFA_REGS_ACQ_CAPS_ACQ_ENTRY_SIZE_SHIFT 16
+#define EFA_REGS_ACQ_CAPS_ACQ_ENTRY_SIZE_MASK 0xff0000
+#define EFA_REGS_ACQ_CAPS_ACQ_MSIX_VECTOR_SHIFT 24
+#define EFA_REGS_ACQ_CAPS_ACQ_MSIX_VECTOR_MASK 0xff000000
+
+/* aenq_caps register */
+#define EFA_REGS_AENQ_CAPS_AENQ_DEPTH_MASK 0xffff
+#define EFA_REGS_AENQ_CAPS_AENQ_ENTRY_SIZE_SHIFT 16
+#define EFA_REGS_AENQ_CAPS_AENQ_ENTRY_SIZE_MASK 0xff0000
+#define EFA_REGS_AENQ_CAPS_AENQ_MSIX_VECTOR_SHIFT 24
+#define EFA_REGS_AENQ_CAPS_AENQ_MSIX_VECTOR_MASK 0xff000000
+
+/* dev_ctl register */
+#define EFA_REGS_DEV_CTL_DEV_RESET_MASK 0x1
+#define EFA_REGS_DEV_CTL_AQ_RESTART_SHIFT 1
+#define EFA_REGS_DEV_CTL_AQ_RESTART_MASK 0x2
+#define EFA_REGS_DEV_CTL_RESET_REASON_SHIFT 28
+#define EFA_REGS_DEV_CTL_RESET_REASON_MASK 0xf0000000
+
+/* dev_sts register */
+#define EFA_REGS_DEV_STS_READY_MASK 0x1
+#define EFA_REGS_DEV_STS_AQ_RESTART_IN_PROGRESS_SHIFT 1
+#define EFA_REGS_DEV_STS_AQ_RESTART_IN_PROGRESS_MASK 0x2
+#define EFA_REGS_DEV_STS_AQ_RESTART_FINISHED_SHIFT 2
+#define EFA_REGS_DEV_STS_AQ_RESTART_FINISHED_MASK 0x4
+#define EFA_REGS_DEV_STS_RESET_IN_PROGRESS_SHIFT 3
+#define EFA_REGS_DEV_STS_RESET_IN_PROGRESS_MASK 0x8
+#define EFA_REGS_DEV_STS_RESET_FINISHED_SHIFT 4
+#define EFA_REGS_DEV_STS_RESET_FINISHED_MASK 0x10
+#define EFA_REGS_DEV_STS_FATAL_ERROR_SHIFT 5
+#define EFA_REGS_DEV_STS_FATAL_ERROR_MASK 0x20
+
+/* mmio_reg_read register */
+#define EFA_REGS_MMIO_REG_READ_REQ_ID_MASK 0xffff
+#define EFA_REGS_MMIO_REG_READ_REG_OFF_SHIFT 16
+#define EFA_REGS_MMIO_REG_READ_REG_OFF_MASK 0xffff0000
+
+#endif /* _EFA_REGS_H_ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#include <linux/vmalloc.h>
+
+#include <rdma/ib_addr.h>
+#include <rdma/ib_umem.h>
+#include <rdma/ib_user_verbs.h>
+#include <rdma/ib_verbs.h>
+#include <rdma/uverbs_ioctl.h>
+
+#include "efa.h"
+
+#define EFA_MMAP_FLAG_SHIFT 56
+#define EFA_MMAP_PAGE_MASK GENMASK(EFA_MMAP_FLAG_SHIFT - 1, 0)
+#define EFA_MMAP_INVALID U64_MAX
+
+enum {
+ EFA_MMAP_DMA_PAGE = 0,
+ EFA_MMAP_IO_WC,
+ EFA_MMAP_IO_NC,
+};
+
+#define EFA_AENQ_ENABLED_GROUPS \
+ (BIT(EFA_ADMIN_FATAL_ERROR) | BIT(EFA_ADMIN_WARNING) | \
+ BIT(EFA_ADMIN_NOTIFICATION) | BIT(EFA_ADMIN_KEEP_ALIVE))
+
+struct efa_mmap_entry {
+ void *obj;
+ u64 address;
+ u64 length;
+ u32 mmap_page;
+ u8 mmap_flag;
+};
+
+static inline u64 get_mmap_key(const struct efa_mmap_entry *efa)
+{
+ return ((u64)efa->mmap_flag << EFA_MMAP_FLAG_SHIFT) |
+ ((u64)efa->mmap_page << PAGE_SHIFT);
+}
+
+#define EFA_CHUNK_PAYLOAD_SHIFT 12
+#define EFA_CHUNK_PAYLOAD_SIZE BIT(EFA_CHUNK_PAYLOAD_SHIFT)
+#define EFA_CHUNK_PAYLOAD_PTR_SIZE 8
+
+#define EFA_CHUNK_SHIFT 12
+#define EFA_CHUNK_SIZE BIT(EFA_CHUNK_SHIFT)
+#define EFA_CHUNK_PTR_SIZE sizeof(struct efa_com_ctrl_buff_info)
+
+#define EFA_PTRS_PER_CHUNK \
+ ((EFA_CHUNK_SIZE - EFA_CHUNK_PTR_SIZE) / EFA_CHUNK_PAYLOAD_PTR_SIZE)
+
+#define EFA_CHUNK_USED_SIZE \
+ ((EFA_PTRS_PER_CHUNK * EFA_CHUNK_PAYLOAD_PTR_SIZE) + EFA_CHUNK_PTR_SIZE)
+
+#define EFA_SUPPORTED_ACCESS_FLAGS IB_ACCESS_LOCAL_WRITE
+
+struct pbl_chunk {
+ dma_addr_t dma_addr;
+ u64 *buf;
+ u32 length;
+};
+
+struct pbl_chunk_list {
+ struct pbl_chunk *chunks;
+ unsigned int size;
+};
+
+struct pbl_context {
+ union {
+ struct {
+ dma_addr_t dma_addr;
+ } continuous;
+ struct {
+ u32 pbl_buf_size_in_pages;
+ struct scatterlist *sgl;
+ int sg_dma_cnt;
+ struct pbl_chunk_list chunk_list;
+ } indirect;
+ } phys;
+ u64 *pbl_buf;
+ u32 pbl_buf_size_in_bytes;
+ u8 physically_continuous;
+};
+
+static inline struct efa_dev *to_edev(struct ib_device *ibdev)
+{
+ return container_of(ibdev, struct efa_dev, ibdev);
+}
+
+static inline struct efa_ucontext *to_eucontext(struct ib_ucontext *ibucontext)
+{
+ return container_of(ibucontext, struct efa_ucontext, ibucontext);
+}
+
+static inline struct efa_pd *to_epd(struct ib_pd *ibpd)
+{
+ return container_of(ibpd, struct efa_pd, ibpd);
+}
+
+static inline struct efa_mr *to_emr(struct ib_mr *ibmr)
+{
+ return container_of(ibmr, struct efa_mr, ibmr);
+}
+
+static inline struct efa_qp *to_eqp(struct ib_qp *ibqp)
+{
+ return container_of(ibqp, struct efa_qp, ibqp);
+}
+
+static inline struct efa_cq *to_ecq(struct ib_cq *ibcq)
+{
+ return container_of(ibcq, struct efa_cq, ibcq);
+}
+
+static inline struct efa_ah *to_eah(struct ib_ah *ibah)
+{
+ return container_of(ibah, struct efa_ah, ibah);
+}
+
+#define field_avail(x, fld, sz) (offsetof(typeof(x), fld) + \
+ sizeof(((typeof(x) *)0)->fld) <= (sz))
+
+#define is_reserved_cleared(reserved) \
+ !memchr_inv(reserved, 0, sizeof(reserved))
+
+static void *efa_zalloc_mapped(struct efa_dev *dev, dma_addr_t *dma_addr,
+ size_t size, enum dma_data_direction dir)
+{
+ void *addr;
+
+ addr = alloc_pages_exact(size, GFP_KERNEL | __GFP_ZERO);
+ if (!addr)
+ return NULL;
+
+ *dma_addr = dma_map_single(&dev->pdev->dev, addr, size, dir);
+ if (dma_mapping_error(&dev->pdev->dev, *dma_addr)) {
+ ibdev_err(&dev->ibdev, "Failed to map DMA address\n");
+ free_pages_exact(addr, size);
+ return NULL;
+ }
+
+ return addr;
+}
+
+/*
+ * This is only called when the ucontext is destroyed and there can be no
+ * concurrent query via mmap or allocate on the xarray, thus we can be sure no
+ * other thread is using the entry pointer. We also know that all the BAR
+ * pages have either been zap'd or munmaped at this point. Normal pages are
+ * refcounted and will be freed at the proper time.
+ */
+static void mmap_entries_remove_free(struct efa_dev *dev,
+ struct efa_ucontext *ucontext)
+{
+ struct efa_mmap_entry *entry;
+ unsigned long mmap_page;
+
+ xa_for_each(&ucontext->mmap_xa, mmap_page, entry) {
+ xa_erase(&ucontext->mmap_xa, mmap_page);
+
+ ibdev_dbg(
+ &dev->ibdev,
+ "mmap: obj[0x%p] key[%#llx] addr[%#llx] len[%#llx] removed\n",
+ entry->obj, get_mmap_key(entry), entry->address,
+ entry->length);
+ if (entry->mmap_flag == EFA_MMAP_DMA_PAGE)
+ /* DMA mapping is already gone, now free the pages */
+ free_pages_exact(phys_to_virt(entry->address),
+ entry->length);
+ kfree(entry);
+ }
+}
+
+static struct efa_mmap_entry *mmap_entry_get(struct efa_dev *dev,
+ struct efa_ucontext *ucontext,
+ u64 key, u64 len)
+{
+ struct efa_mmap_entry *entry;
+ u64 mmap_page;
+
+ mmap_page = (key & EFA_MMAP_PAGE_MASK) >> PAGE_SHIFT;
+ if (mmap_page > U32_MAX)
+ return NULL;
+
+ entry = xa_load(&ucontext->mmap_xa, mmap_page);
+ if (!entry || get_mmap_key(entry) != key || entry->length != len)
+ return NULL;
+
+ ibdev_dbg(&dev->ibdev,
+ "mmap: obj[0x%p] key[%#llx] addr[%#llx] len[%#llx] removed\n",
+ entry->obj, key, entry->address, entry->length);
+
+ return entry;
+}
+
+/*
+ * Note this locking scheme cannot support removal of entries, except during
+ * ucontext destruction when the core code guarentees no concurrency.
+ */
+static u64 mmap_entry_insert(struct efa_dev *dev, struct efa_ucontext *ucontext,
+ void *obj, u64 address, u64 length, u8 mmap_flag)
+{
+ struct efa_mmap_entry *entry;
+ int err;
+
+ entry = kmalloc(sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ return EFA_MMAP_INVALID;
+
+ entry->obj = obj;
+ entry->address = address;
+ entry->length = length;
+ entry->mmap_flag = mmap_flag;
+
+ xa_lock(&ucontext->mmap_xa);
+ entry->mmap_page = ucontext->mmap_xa_page;
+ ucontext->mmap_xa_page += DIV_ROUND_UP(length, PAGE_SIZE);
+ err = __xa_insert(&ucontext->mmap_xa, entry->mmap_page, entry,
+ GFP_KERNEL);
+ xa_unlock(&ucontext->mmap_xa);
+ if (err){
+ kfree(entry);
+ return EFA_MMAP_INVALID;
+ }
+
+ ibdev_dbg(
+ &dev->ibdev,
+ "mmap: obj[0x%p] addr[%#llx], len[%#llx], key[%#llx] inserted\n",
+ entry->obj, entry->address, entry->length, get_mmap_key(entry));
+
+ return get_mmap_key(entry);
+}
+
+int efa_query_device(struct ib_device *ibdev,
+ struct ib_device_attr *props,
+ struct ib_udata *udata)
+{
+ struct efa_com_get_device_attr_result *dev_attr;
+ struct efa_ibv_ex_query_device_resp resp = {};
+ struct efa_dev *dev = to_edev(ibdev);
+ int err;
+
+ if (udata && udata->inlen &&
+ !ib_is_udata_cleared(udata, 0, udata->inlen)) {
+ ibdev_dbg(ibdev,
+ "Incompatible ABI params, udata not cleared\n");
+ return -EINVAL;
+ }
+
+ dev_attr = &dev->dev_attr;
+
+ memset(props, 0, sizeof(*props));
+ props->max_mr_size = dev_attr->max_mr_pages * PAGE_SIZE;
+ props->page_size_cap = dev_attr->page_size_cap;
+ props->vendor_id = dev->pdev->vendor;
+ props->vendor_part_id = dev->pdev->device;
+ props->hw_ver = dev->pdev->subsystem_device;
+ props->max_qp = dev_attr->max_qp;
+ props->max_cq = dev_attr->max_cq;
+ props->max_pd = dev_attr->max_pd;
+ props->max_mr = dev_attr->max_mr;
+ props->max_ah = dev_attr->max_ah;
+ props->max_cqe = dev_attr->max_cq_depth;
+ props->max_qp_wr = min_t(u32, dev_attr->max_sq_depth,
+ dev_attr->max_rq_depth);
+ props->max_send_sge = dev_attr->max_sq_sge;
+ props->max_recv_sge = dev_attr->max_rq_sge;
+
+ if (udata && udata->outlen) {
+ resp.max_sq_sge = dev_attr->max_sq_sge;
+ resp.max_rq_sge = dev_attr->max_rq_sge;
+ resp.max_sq_wr = dev_attr->max_sq_depth;
+ resp.max_rq_wr = dev_attr->max_rq_depth;
+
+ err = ib_copy_to_udata(udata, &resp,
+ min(sizeof(resp), udata->outlen));
+ if (err) {
+ ibdev_dbg(ibdev,
+ "Failed to copy udata for query_device\n");
+ return err;
+ }
+ }
+
+ return 0;
+}
+
+int efa_query_port(struct ib_device *ibdev, u8 port,
+ struct ib_port_attr *props)
+{
+ struct efa_dev *dev = to_edev(ibdev);
+
+ props->lmc = 1;
+
+ props->state = IB_PORT_ACTIVE;
+ props->phys_state = 5;
+ props->gid_tbl_len = 1;
+ props->pkey_tbl_len = 1;
+ props->active_speed = IB_SPEED_EDR;
+ props->active_width = IB_WIDTH_4X;
+ props->max_mtu = ib_mtu_int_to_enum(dev->mtu);
+ props->active_mtu = ib_mtu_int_to_enum(dev->mtu);
+ props->max_msg_sz = dev->mtu;
+ props->max_vl_num = 1;
+
+ return 0;
+}
+
+int efa_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
+ int qp_attr_mask,
+ struct ib_qp_init_attr *qp_init_attr)
+{
+ struct efa_dev *dev = to_edev(ibqp->device);
+ struct efa_com_query_qp_params params = {};
+ struct efa_com_query_qp_result result;
+ struct efa_qp *qp = to_eqp(ibqp);
+ int err;
+
+#define EFA_QUERY_QP_SUPP_MASK \
+ (IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_PORT | \
+ IB_QP_QKEY | IB_QP_SQ_PSN | IB_QP_CAP)
+
+ if (qp_attr_mask & ~EFA_QUERY_QP_SUPP_MASK) {
+ ibdev_dbg(&dev->ibdev,
+ "Unsupported qp_attr_mask[%#x] supported[%#x]\n",
+ qp_attr_mask, EFA_QUERY_QP_SUPP_MASK);
+ return -EOPNOTSUPP;
+ }
+
+ memset(qp_attr, 0, sizeof(*qp_attr));
+ memset(qp_init_attr, 0, sizeof(*qp_init_attr));
+
+ params.qp_handle = qp->qp_handle;
+ err = efa_com_query_qp(&dev->edev, ¶ms, &result);
+ if (err)
+ return err;
+
+ qp_attr->qp_state = result.qp_state;
+ qp_attr->qkey = result.qkey;
+ qp_attr->sq_psn = result.sq_psn;
+ qp_attr->sq_draining = result.sq_draining;
+ qp_attr->port_num = 1;
+
+ qp_attr->cap.max_send_wr = qp->max_send_wr;
+ qp_attr->cap.max_recv_wr = qp->max_recv_wr;
+ qp_attr->cap.max_send_sge = qp->max_send_sge;
+ qp_attr->cap.max_recv_sge = qp->max_recv_sge;
+ qp_attr->cap.max_inline_data = qp->max_inline_data;
+
+ qp_init_attr->qp_type = ibqp->qp_type;
+ qp_init_attr->recv_cq = ibqp->recv_cq;
+ qp_init_attr->send_cq = ibqp->send_cq;
+ qp_init_attr->qp_context = ibqp->qp_context;
+ qp_init_attr->cap = qp_attr->cap;
+
+ return 0;
+}
+
+int efa_query_gid(struct ib_device *ibdev, u8 port, int index,
+ union ib_gid *gid)
+{
+ struct efa_dev *dev = to_edev(ibdev);
+
+ memcpy(gid->raw, dev->addr, sizeof(dev->addr));
+
+ return 0;
+}
+
+int efa_query_pkey(struct ib_device *ibdev, u8 port, u16 index,
+ u16 *pkey)
+{
+ if (index > 0)
+ return -EINVAL;
+
+ *pkey = 0xffff;
+ return 0;
+}
+
+static int efa_pd_dealloc(struct efa_dev *dev, u16 pdn)
+{
+ struct efa_com_dealloc_pd_params params = {
+ .pdn = pdn,
+ };
+
+ return efa_com_dealloc_pd(&dev->edev, ¶ms);
+}
+
+int efa_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
+{
+ struct efa_dev *dev = to_edev(ibpd->device);
+ struct efa_ibv_alloc_pd_resp resp = {};
+ struct efa_com_alloc_pd_result result;
+ struct efa_pd *pd = to_epd(ibpd);
+ int err;
+
+ if (udata->inlen &&
+ !ib_is_udata_cleared(udata, 0, udata->inlen)) {
+ ibdev_dbg(&dev->ibdev,
+ "Incompatible ABI params, udata not cleared\n");
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ err = efa_com_alloc_pd(&dev->edev, &result);
+ if (err)
+ goto err_out;
+
+ pd->pdn = result.pdn;
+ resp.pdn = result.pdn;
+
+ if (udata->outlen) {
+ err = ib_copy_to_udata(udata, &resp,
+ min(sizeof(resp), udata->outlen));
+ if (err) {
+ ibdev_dbg(&dev->ibdev,
+ "Failed to copy udata for alloc_pd\n");
+ goto err_dealloc_pd;
+ }
+ }
+
+ ibdev_dbg(&dev->ibdev, "Allocated pd[%d]\n", pd->pdn);
+
+ return 0;
+
+err_dealloc_pd:
+ efa_pd_dealloc(dev, result.pdn);
+err_out:
+ atomic64_inc(&dev->stats.sw_stats.alloc_pd_err);
+ return err;
+}
+
+void efa_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
+{
+ struct efa_dev *dev = to_edev(ibpd->device);
+ struct efa_pd *pd = to_epd(ibpd);
+
+ if (udata->inlen &&
+ !ib_is_udata_cleared(udata, 0, udata->inlen)) {
+ ibdev_dbg(&dev->ibdev, "Incompatible ABI params\n");
+ return;
+ }
+
+ ibdev_dbg(&dev->ibdev, "Dealloc pd[%d]\n", pd->pdn);
+ efa_pd_dealloc(dev, pd->pdn);
+}
+
+static int efa_destroy_qp_handle(struct efa_dev *dev, u32 qp_handle)
+{
+ struct efa_com_destroy_qp_params params = { .qp_handle = qp_handle };
+
+ return efa_com_destroy_qp(&dev->edev, ¶ms);
+}
+
+int efa_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
+{
+ struct efa_dev *dev = to_edev(ibqp->pd->device);
+ struct efa_qp *qp = to_eqp(ibqp);
+ int err;
+
+ if (udata->inlen &&
+ !ib_is_udata_cleared(udata, 0, udata->inlen)) {
+ ibdev_dbg(&dev->ibdev, "Incompatible ABI params\n");
+ return -EINVAL;
+ }
+
+ ibdev_dbg(&dev->ibdev, "Destroy qp[%u]\n", ibqp->qp_num);
+ err = efa_destroy_qp_handle(dev, qp->qp_handle);
+ if (err)
+ return err;
+
+ if (qp->rq_cpu_addr) {
+ ibdev_dbg(&dev->ibdev,
+ "qp->cpu_addr[0x%p] freed: size[%lu], dma[%pad]\n",
+ qp->rq_cpu_addr, qp->rq_size,
+ &qp->rq_dma_addr);
+ dma_unmap_single(&dev->pdev->dev, qp->rq_dma_addr, qp->rq_size,
+ DMA_TO_DEVICE);
+ }
+
+ kfree(qp);
+ return 0;
+}
+
+static int qp_mmap_entries_setup(struct efa_qp *qp,
+ struct efa_dev *dev,
+ struct efa_ucontext *ucontext,
+ struct efa_com_create_qp_params *params,
+ struct efa_ibv_create_qp_resp *resp)
+{
+ /*
+ * Once an entry is inserted it might be mmapped, hence cannot be
+ * cleaned up until dealloc_ucontext.
+ */
+ resp->sq_db_mmap_key =
+ mmap_entry_insert(dev, ucontext, qp,
+ dev->db_bar_addr + resp->sq_db_offset,
+ PAGE_SIZE, EFA_MMAP_IO_NC);
+ if (resp->sq_db_mmap_key == EFA_MMAP_INVALID)
+ return -ENOMEM;
+
+ resp->sq_db_offset &= ~PAGE_MASK;
+
+ resp->llq_desc_mmap_key =
+ mmap_entry_insert(dev, ucontext, qp,
+ dev->mem_bar_addr + resp->llq_desc_offset,
+ PAGE_ALIGN(params->sq_ring_size_in_bytes +
+ (resp->llq_desc_offset & ~PAGE_MASK)),
+ EFA_MMAP_IO_WC);
+ if (resp->llq_desc_mmap_key == EFA_MMAP_INVALID)
+ return -ENOMEM;
+
+ resp->llq_desc_offset &= ~PAGE_MASK;
+
+ if (qp->rq_size) {
+ resp->rq_db_mmap_key =
+ mmap_entry_insert(dev, ucontext, qp,
+ dev->db_bar_addr + resp->rq_db_offset,
+ PAGE_SIZE, EFA_MMAP_IO_NC);
+ if (resp->rq_db_mmap_key == EFA_MMAP_INVALID)
+ return -ENOMEM;
+
+ resp->rq_db_offset &= ~PAGE_MASK;
+
+ resp->rq_mmap_key =
+ mmap_entry_insert(dev, ucontext, qp,
+ virt_to_phys(qp->rq_cpu_addr),
+ qp->rq_size, EFA_MMAP_DMA_PAGE);
+ if (resp->rq_mmap_key == EFA_MMAP_INVALID)
+ return -ENOMEM;
+
+ resp->rq_mmap_size = qp->rq_size;
+ }
+
+ return 0;
+}
+
+static int efa_qp_validate_cap(struct efa_dev *dev,
+ struct ib_qp_init_attr *init_attr)
+{
+ if (init_attr->cap.max_send_wr > dev->dev_attr.max_sq_depth) {
+ ibdev_dbg(&dev->ibdev,
+ "qp: requested send wr[%u] exceeds the max[%u]\n",
+ init_attr->cap.max_send_wr,
+ dev->dev_attr.max_sq_depth);
+ return -EINVAL;
+ }
+ if (init_attr->cap.max_recv_wr > dev->dev_attr.max_rq_depth) {
+ ibdev_dbg(&dev->ibdev,
+ "qp: requested receive wr[%u] exceeds the max[%u]\n",
+ init_attr->cap.max_recv_wr,
+ dev->dev_attr.max_rq_depth);
+ return -EINVAL;
+ }
+ if (init_attr->cap.max_send_sge > dev->dev_attr.max_sq_sge) {
+ ibdev_dbg(&dev->ibdev,
+ "qp: requested sge send[%u] exceeds the max[%u]\n",
+ init_attr->cap.max_send_sge, dev->dev_attr.max_sq_sge);
+ return -EINVAL;
+ }
+ if (init_attr->cap.max_recv_sge > dev->dev_attr.max_rq_sge) {
+ ibdev_dbg(&dev->ibdev,
+ "qp: requested sge recv[%u] exceeds the max[%u]\n",
+ init_attr->cap.max_recv_sge, dev->dev_attr.max_rq_sge);
+ return -EINVAL;
+ }
+ if (init_attr->cap.max_inline_data > dev->dev_attr.inline_buf_size) {
+ ibdev_dbg(&dev->ibdev,
+ "qp: requested inline data[%u] exceeds the max[%u]\n",
+ init_attr->cap.max_inline_data,
+ dev->dev_attr.inline_buf_size);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int efa_qp_validate_attr(struct efa_dev *dev,
+ struct ib_qp_init_attr *init_attr)
+{
+ if (init_attr->qp_type != IB_QPT_DRIVER &&
+ init_attr->qp_type != IB_QPT_UD) {
+ ibdev_dbg(&dev->ibdev,
+ "Unsupported qp type %d\n", init_attr->qp_type);
+ return -EOPNOTSUPP;
+ }
+
+ if (init_attr->srq) {
+ ibdev_dbg(&dev->ibdev, "SRQ is not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (init_attr->create_flags) {
+ ibdev_dbg(&dev->ibdev, "Unsupported create flags\n");
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+struct ib_qp *efa_create_qp(struct ib_pd *ibpd,
+ struct ib_qp_init_attr *init_attr,
+ struct ib_udata *udata)
+{
+ struct efa_com_create_qp_params create_qp_params = {};
+ struct efa_com_create_qp_result create_qp_resp;
+ struct efa_dev *dev = to_edev(ibpd->device);
+ struct efa_ibv_create_qp_resp resp = {};
+ struct efa_ibv_create_qp cmd = {};
+ bool rq_entry_inserted = false;
+ struct efa_ucontext *ucontext;
+ struct efa_qp *qp;
+ int err;
+
+ ucontext = rdma_udata_to_drv_context(udata, struct efa_ucontext,
+ ibucontext);
+
+ err = efa_qp_validate_cap(dev, init_attr);
+ if (err)
+ goto err_out;
+
+ err = efa_qp_validate_attr(dev, init_attr);
+ if (err)
+ goto err_out;
+
+ if (!field_avail(cmd, driver_qp_type, udata->inlen)) {
+ ibdev_dbg(&dev->ibdev,
+ "Incompatible ABI params, no input udata\n");
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ if (udata->inlen > sizeof(cmd) &&
+ !ib_is_udata_cleared(udata, sizeof(cmd),
+ udata->inlen - sizeof(cmd))) {
+ ibdev_dbg(&dev->ibdev,
+ "Incompatible ABI params, unknown fields in udata\n");
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ err = ib_copy_from_udata(&cmd, udata,
+ min(sizeof(cmd), udata->inlen));
+ if (err) {
+ ibdev_dbg(&dev->ibdev,
+ "Cannot copy udata for create_qp\n");
+ goto err_out;
+ }
+
+ if (cmd.comp_mask) {
+ ibdev_dbg(&dev->ibdev,
+ "Incompatible ABI params, unknown fields in udata\n");
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ qp = kzalloc(sizeof(*qp), GFP_KERNEL);
+ if (!qp) {
+ err = -ENOMEM;
+ goto err_out;
+ }
+
+ create_qp_params.uarn = ucontext->uarn;
+ create_qp_params.pd = to_epd(ibpd)->pdn;
+
+ if (init_attr->qp_type == IB_QPT_UD) {
+ create_qp_params.qp_type = EFA_ADMIN_QP_TYPE_UD;
+ } else if (cmd.driver_qp_type == EFA_QP_DRIVER_TYPE_SRD) {
+ create_qp_params.qp_type = EFA_ADMIN_QP_TYPE_SRD;
+ } else {
+ ibdev_dbg(&dev->ibdev,
+ "Unsupported qp type %d driver qp type %d\n",
+ init_attr->qp_type, cmd.driver_qp_type);
+ err = -EOPNOTSUPP;
+ goto err_free_qp;
+ }
+
+ ibdev_dbg(&dev->ibdev, "Create QP: qp type %d driver qp type %#x\n",
+ init_attr->qp_type, cmd.driver_qp_type);
+ create_qp_params.send_cq_idx = to_ecq(init_attr->send_cq)->cq_idx;
+ create_qp_params.recv_cq_idx = to_ecq(init_attr->recv_cq)->cq_idx;
+ create_qp_params.sq_depth = init_attr->cap.max_send_wr;
+ create_qp_params.sq_ring_size_in_bytes = cmd.sq_ring_size;
+
+ create_qp_params.rq_depth = init_attr->cap.max_recv_wr;
+ create_qp_params.rq_ring_size_in_bytes = cmd.rq_ring_size;
+ qp->rq_size = PAGE_ALIGN(create_qp_params.rq_ring_size_in_bytes);
+ if (qp->rq_size) {
+ qp->rq_cpu_addr = efa_zalloc_mapped(dev, &qp->rq_dma_addr,
+ qp->rq_size, DMA_TO_DEVICE);
+ if (!qp->rq_cpu_addr) {
+ err = -ENOMEM;
+ goto err_free_qp;
+ }
+
+ ibdev_dbg(&dev->ibdev,
+ "qp->cpu_addr[0x%p] allocated: size[%lu], dma[%pad]\n",
+ qp->rq_cpu_addr, qp->rq_size, &qp->rq_dma_addr);
+ create_qp_params.rq_base_addr = qp->rq_dma_addr;
+ }
+
+ err = efa_com_create_qp(&dev->edev, &create_qp_params,
+ &create_qp_resp);
+ if (err)
+ goto err_free_mapped;
+
+ resp.sq_db_offset = create_qp_resp.sq_db_offset;
+ resp.rq_db_offset = create_qp_resp.rq_db_offset;
+ resp.llq_desc_offset = create_qp_resp.llq_descriptors_offset;
+ resp.send_sub_cq_idx = create_qp_resp.send_sub_cq_idx;
+ resp.recv_sub_cq_idx = create_qp_resp.recv_sub_cq_idx;
+
+ err = qp_mmap_entries_setup(qp, dev, ucontext, &create_qp_params,
+ &resp);
+ if (err)
+ goto err_destroy_qp;
+
+ rq_entry_inserted = true;
+ qp->qp_handle = create_qp_resp.qp_handle;
+ qp->ibqp.qp_num = create_qp_resp.qp_num;
+ qp->ibqp.qp_type = init_attr->qp_type;
+ qp->max_send_wr = init_attr->cap.max_send_wr;
+ qp->max_recv_wr = init_attr->cap.max_recv_wr;
+ qp->max_send_sge = init_attr->cap.max_send_sge;
+ qp->max_recv_sge = init_attr->cap.max_recv_sge;
+ qp->max_inline_data = init_attr->cap.max_inline_data;
+
+ if (udata->outlen) {
+ err = ib_copy_to_udata(udata, &resp,
+ min(sizeof(resp), udata->outlen));
+ if (err) {
+ ibdev_dbg(&dev->ibdev,
+ "Failed to copy udata for qp[%u]\n",
+ create_qp_resp.qp_num);
+ goto err_destroy_qp;
+ }
+ }
+
+ ibdev_dbg(&dev->ibdev, "Created qp[%d]\n", qp->ibqp.qp_num);
+
+ return &qp->ibqp;
+
+err_destroy_qp:
+ efa_destroy_qp_handle(dev, create_qp_resp.qp_handle);
+err_free_mapped:
+ if (qp->rq_size) {
+ dma_unmap_single(&dev->pdev->dev, qp->rq_dma_addr, qp->rq_size,
+ DMA_TO_DEVICE);
+ if (!rq_entry_inserted)
+ free_pages_exact(qp->rq_cpu_addr, qp->rq_size);
+ }
+err_free_qp:
+ kfree(qp);
+err_out:
+ atomic64_inc(&dev->stats.sw_stats.create_qp_err);
+ return ERR_PTR(err);
+}
+
+static int efa_modify_qp_validate(struct efa_dev *dev, struct efa_qp *qp,
+ struct ib_qp_attr *qp_attr, int qp_attr_mask,
+ enum ib_qp_state cur_state,
+ enum ib_qp_state new_state)
+{
+#define EFA_MODIFY_QP_SUPP_MASK \
+ (IB_QP_STATE | IB_QP_CUR_STATE | IB_QP_EN_SQD_ASYNC_NOTIFY | \
+ IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_QKEY | IB_QP_SQ_PSN)
+
+ if (qp_attr_mask & ~EFA_MODIFY_QP_SUPP_MASK) {
+ ibdev_dbg(&dev->ibdev,
+ "Unsupported qp_attr_mask[%#x] supported[%#x]\n",
+ qp_attr_mask, EFA_MODIFY_QP_SUPP_MASK);
+ return -EOPNOTSUPP;
+ }
+
+ if (!ib_modify_qp_is_ok(cur_state, new_state, IB_QPT_UD,
+ qp_attr_mask)) {
+ ibdev_dbg(&dev->ibdev, "Invalid modify QP parameters\n");
+ return -EINVAL;
+ }
+
+ if ((qp_attr_mask & IB_QP_PORT) && qp_attr->port_num != 1) {
+ ibdev_dbg(&dev->ibdev, "Can't change port num\n");
+ return -EOPNOTSUPP;
+ }
+
+ if ((qp_attr_mask & IB_QP_PKEY_INDEX) && qp_attr->pkey_index) {
+ ibdev_dbg(&dev->ibdev, "Can't change pkey index\n");
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+int efa_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
+ int qp_attr_mask, struct ib_udata *udata)
+{
+ struct efa_dev *dev = to_edev(ibqp->device);
+ struct efa_com_modify_qp_params params = {};
+ struct efa_qp *qp = to_eqp(ibqp);
+ enum ib_qp_state cur_state;
+ enum ib_qp_state new_state;
+ int err;
+
+ if (udata->inlen &&
+ !ib_is_udata_cleared(udata, 0, udata->inlen)) {
+ ibdev_dbg(&dev->ibdev,
+ "Incompatible ABI params, udata not cleared\n");
+ return -EINVAL;
+ }
+
+ cur_state = qp_attr_mask & IB_QP_CUR_STATE ? qp_attr->cur_qp_state :
+ qp->state;
+ new_state = qp_attr_mask & IB_QP_STATE ? qp_attr->qp_state : cur_state;
+
+ err = efa_modify_qp_validate(dev, qp, qp_attr, qp_attr_mask, cur_state,
+ new_state);
+ if (err)
+ return err;
+
+ params.qp_handle = qp->qp_handle;
+
+ if (qp_attr_mask & IB_QP_STATE) {
+ params.modify_mask |= BIT(EFA_ADMIN_QP_STATE_BIT) |
+ BIT(EFA_ADMIN_CUR_QP_STATE_BIT);
+ params.cur_qp_state = qp_attr->cur_qp_state;
+ params.qp_state = qp_attr->qp_state;
+ }
+
+ if (qp_attr_mask & IB_QP_EN_SQD_ASYNC_NOTIFY) {
+ params.modify_mask |=
+ BIT(EFA_ADMIN_SQ_DRAINED_ASYNC_NOTIFY_BIT);
+ params.sq_drained_async_notify = qp_attr->en_sqd_async_notify;
+ }
+
+ if (qp_attr_mask & IB_QP_QKEY) {
+ params.modify_mask |= BIT(EFA_ADMIN_QKEY_BIT);
+ params.qkey = qp_attr->qkey;
+ }
+
+ if (qp_attr_mask & IB_QP_SQ_PSN) {
+ params.modify_mask |= BIT(EFA_ADMIN_SQ_PSN_BIT);
+ params.sq_psn = qp_attr->sq_psn;
+ }
+
+ err = efa_com_modify_qp(&dev->edev, ¶ms);
+ if (err)
+ return err;
+
+ qp->state = new_state;
+
+ return 0;
+}
+
+static int efa_destroy_cq_idx(struct efa_dev *dev, int cq_idx)
+{
+ struct efa_com_destroy_cq_params params = { .cq_idx = cq_idx };
+
+ return efa_com_destroy_cq(&dev->edev, ¶ms);
+}
+
+int efa_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
+{
+ struct efa_dev *dev = to_edev(ibcq->device);
+ struct efa_cq *cq = to_ecq(ibcq);
+ int err;
+
+ if (udata->inlen &&
+ !ib_is_udata_cleared(udata, 0, udata->inlen)) {
+ ibdev_dbg(&dev->ibdev, "Incompatible ABI params\n");
+ return -EINVAL;
+ }
+
+ ibdev_dbg(&dev->ibdev,
+ "Destroy cq[%d] virt[0x%p] freed: size[%lu], dma[%pad]\n",
+ cq->cq_idx, cq->cpu_addr, cq->size, &cq->dma_addr);
+
+ err = efa_destroy_cq_idx(dev, cq->cq_idx);
+ if (err)
+ return err;
+
+ dma_unmap_single(&dev->pdev->dev, cq->dma_addr, cq->size,
+ DMA_FROM_DEVICE);
+
+ kfree(cq);
+ return 0;
+}
+
+static int cq_mmap_entries_setup(struct efa_dev *dev, struct efa_cq *cq,
+ struct efa_ibv_create_cq_resp *resp)
+{
+ resp->q_mmap_size = cq->size;
+ resp->q_mmap_key = mmap_entry_insert(dev, cq->ucontext, cq,
+ virt_to_phys(cq->cpu_addr),
+ cq->size, EFA_MMAP_DMA_PAGE);
+ if (resp->q_mmap_key == EFA_MMAP_INVALID)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static struct ib_cq *do_create_cq(struct ib_device *ibdev, int entries,
+ int vector, struct ib_ucontext *ibucontext,
+ struct ib_udata *udata)
+{
+ struct efa_ibv_create_cq_resp resp = {};
+ struct efa_com_create_cq_params params;
+ struct efa_com_create_cq_result result;
+ struct efa_dev *dev = to_edev(ibdev);
+ struct efa_ibv_create_cq cmd = {};
+ bool cq_entry_inserted = false;
+ struct efa_cq *cq;
+ int err;
+
+ ibdev_dbg(ibdev, "create_cq entries %d\n", entries);
+
+ if (entries < 1 || entries > dev->dev_attr.max_cq_depth) {
+ ibdev_dbg(ibdev,
+ "cq: requested entries[%u] non-positive or greater than max[%u]\n",
+ entries, dev->dev_attr.max_cq_depth);
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ if (!field_avail(cmd, num_sub_cqs, udata->inlen)) {
+ ibdev_dbg(ibdev,
+ "Incompatible ABI params, no input udata\n");
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ if (udata->inlen > sizeof(cmd) &&
+ !ib_is_udata_cleared(udata, sizeof(cmd),
+ udata->inlen - sizeof(cmd))) {
+ ibdev_dbg(ibdev,
+ "Incompatible ABI params, unknown fields in udata\n");
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ err = ib_copy_from_udata(&cmd, udata,
+ min(sizeof(cmd), udata->inlen));
+ if (err) {
+ ibdev_dbg(ibdev, "Cannot copy udata for create_cq\n");
+ goto err_out;
+ }
+
+ if (cmd.comp_mask || !is_reserved_cleared(cmd.reserved_50)) {
+ ibdev_dbg(ibdev,
+ "Incompatible ABI params, unknown fields in udata\n");
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ if (!cmd.cq_entry_size) {
+ ibdev_dbg(ibdev,
+ "Invalid entry size [%u]\n", cmd.cq_entry_size);
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ if (cmd.num_sub_cqs != dev->dev_attr.sub_cqs_per_cq) {
+ ibdev_dbg(ibdev,
+ "Invalid number of sub cqs[%u] expected[%u]\n",
+ cmd.num_sub_cqs, dev->dev_attr.sub_cqs_per_cq);
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ cq = kzalloc(sizeof(*cq), GFP_KERNEL);
+ if (!cq) {
+ err = -ENOMEM;
+ goto err_out;
+ }
+
+ cq->ucontext = to_eucontext(ibucontext);
+ cq->size = PAGE_ALIGN(cmd.cq_entry_size * entries * cmd.num_sub_cqs);
+ cq->cpu_addr = efa_zalloc_mapped(dev, &cq->dma_addr, cq->size,
+ DMA_FROM_DEVICE);
+ if (!cq->cpu_addr) {
+ err = -ENOMEM;
+ goto err_free_cq;
+ }
+
+ params.uarn = cq->ucontext->uarn;
+ params.cq_depth = entries;
+ params.dma_addr = cq->dma_addr;
+ params.entry_size_in_bytes = cmd.cq_entry_size;
+ params.num_sub_cqs = cmd.num_sub_cqs;
+ err = efa_com_create_cq(&dev->edev, ¶ms, &result);
+ if (err)
+ goto err_free_mapped;
+
+ resp.cq_idx = result.cq_idx;
+ cq->cq_idx = result.cq_idx;
+ cq->ibcq.cqe = result.actual_depth;
+ WARN_ON_ONCE(entries != result.actual_depth);
+
+ err = cq_mmap_entries_setup(dev, cq, &resp);
+ if (err) {
+ ibdev_dbg(ibdev,
+ "Could not setup cq[%u] mmap entries\n", cq->cq_idx);
+ goto err_destroy_cq;
+ }
+
+ cq_entry_inserted = true;
+
+ if (udata->outlen) {
+ err = ib_copy_to_udata(udata, &resp,
+ min(sizeof(resp), udata->outlen));
+ if (err) {
+ ibdev_dbg(ibdev,
+ "Failed to copy udata for create_cq\n");
+ goto err_destroy_cq;
+ }
+ }
+
+ ibdev_dbg(ibdev,
+ "Created cq[%d], cq depth[%u]. dma[%pad] virt[0x%p]\n",
+ cq->cq_idx, result.actual_depth, &cq->dma_addr, cq->cpu_addr);
+
+ return &cq->ibcq;
+
+err_destroy_cq:
+ efa_destroy_cq_idx(dev, cq->cq_idx);
+err_free_mapped:
+ dma_unmap_single(&dev->pdev->dev, cq->dma_addr, cq->size,
+ DMA_FROM_DEVICE);
+ if (!cq_entry_inserted)
+ free_pages_exact(cq->cpu_addr, cq->size);
+err_free_cq:
+ kfree(cq);
+err_out:
+ atomic64_inc(&dev->stats.sw_stats.create_cq_err);
+ return ERR_PTR(err);
+}
+
+struct ib_cq *efa_create_cq(struct ib_device *ibdev,
+ const struct ib_cq_init_attr *attr,
+ struct ib_udata *udata)
+{
+ struct efa_ucontext *ucontext = rdma_udata_to_drv_context(udata,
+ struct efa_ucontext,
+ ibucontext);
+
+ return do_create_cq(ibdev, attr->cqe, attr->comp_vector,
+ &ucontext->ibucontext, udata);
+}
+
+static int umem_to_page_list(struct efa_dev *dev,
+ struct ib_umem *umem,
+ u64 *page_list,
+ u32 hp_cnt,
+ u8 hp_shift)
+{
+ u32 pages_in_hp = BIT(hp_shift - PAGE_SHIFT);
+ struct sg_dma_page_iter sg_iter;
+ unsigned int page_idx = 0;
+ unsigned int hp_idx = 0;
+
+ ibdev_dbg(&dev->ibdev, "hp_cnt[%u], pages_in_hp[%u]\n",
+ hp_cnt, pages_in_hp);
+
+ for_each_sg_dma_page(umem->sg_head.sgl, &sg_iter, umem->nmap, 0) {
+ if (page_idx % pages_in_hp == 0) {
+ page_list[hp_idx] = sg_page_iter_dma_address(&sg_iter);
+ hp_idx++;
+ }
+
+ page_idx++;
+ }
+
+ return 0;
+}
+
+static struct scatterlist *efa_vmalloc_buf_to_sg(u64 *buf, int page_cnt)
+{
+ struct scatterlist *sglist;
+ struct page *pg;
+ int i;
+
+ sglist = kcalloc(page_cnt, sizeof(*sglist), GFP_KERNEL);
+ if (!sglist)
+ return NULL;
+ sg_init_table(sglist, page_cnt);
+ for (i = 0; i < page_cnt; i++) {
+ pg = vmalloc_to_page(buf);
+ if (!pg)
+ goto err;
+ sg_set_page(&sglist[i], pg, PAGE_SIZE, 0);
+ buf += PAGE_SIZE / sizeof(*buf);
+ }
+ return sglist;
+
+err:
+ kfree(sglist);
+ return NULL;
+}
+
+/*
+ * create a chunk list of physical pages dma addresses from the supplied
+ * scatter gather list
+ */
+static int pbl_chunk_list_create(struct efa_dev *dev, struct pbl_context *pbl)
+{
+ unsigned int entry, payloads_in_sg, chunk_list_size, chunk_idx, payload_idx;
+ struct pbl_chunk_list *chunk_list = &pbl->phys.indirect.chunk_list;
+ int page_cnt = pbl->phys.indirect.pbl_buf_size_in_pages;
+ struct scatterlist *pages_sgl = pbl->phys.indirect.sgl;
+ int sg_dma_cnt = pbl->phys.indirect.sg_dma_cnt;
+ struct efa_com_ctrl_buff_info *ctrl_buf;
+ u64 *cur_chunk_buf, *prev_chunk_buf;
+ struct scatterlist *sg;
+ dma_addr_t dma_addr;
+ int i;
+
+ /* allocate a chunk list that consists of 4KB chunks */
+ chunk_list_size = DIV_ROUND_UP(page_cnt, EFA_PTRS_PER_CHUNK);
+
+ chunk_list->size = chunk_list_size;
+ chunk_list->chunks = kcalloc(chunk_list_size,
+ sizeof(*chunk_list->chunks),
+ GFP_KERNEL);
+ if (!chunk_list->chunks)
+ return -ENOMEM;
+
+ ibdev_dbg(&dev->ibdev,
+ "chunk_list_size[%u] - pages[%u]\n", chunk_list_size,
+ page_cnt);
+
+ /* allocate chunk buffers: */
+ for (i = 0; i < chunk_list_size; i++) {
+ chunk_list->chunks[i].buf = kzalloc(EFA_CHUNK_SIZE, GFP_KERNEL);
+ if (!chunk_list->chunks[i].buf)
+ goto chunk_list_dealloc;
+
+ chunk_list->chunks[i].length = EFA_CHUNK_USED_SIZE;
+ }
+ chunk_list->chunks[chunk_list_size - 1].length =
+ ((page_cnt % EFA_PTRS_PER_CHUNK) * EFA_CHUNK_PAYLOAD_PTR_SIZE) +
+ EFA_CHUNK_PTR_SIZE;
+
+ /* fill the dma addresses of sg list pages to chunks: */
+ chunk_idx = 0;
+ payload_idx = 0;
+ cur_chunk_buf = chunk_list->chunks[0].buf;
+ for_each_sg(pages_sgl, sg, sg_dma_cnt, entry) {
+ payloads_in_sg = sg_dma_len(sg) >> EFA_CHUNK_PAYLOAD_SHIFT;
+ for (i = 0; i < payloads_in_sg; i++) {
+ cur_chunk_buf[payload_idx++] =
+ (sg_dma_address(sg) & ~(EFA_CHUNK_PAYLOAD_SIZE - 1)) +
+ (EFA_CHUNK_PAYLOAD_SIZE * i);
+
+ if (payload_idx == EFA_PTRS_PER_CHUNK) {
+ chunk_idx++;
+ cur_chunk_buf = chunk_list->chunks[chunk_idx].buf;
+ payload_idx = 0;
+ }
+ }
+ }
+
+ /* map chunks to dma and fill chunks next ptrs */
+ for (i = chunk_list_size - 1; i >= 0; i--) {
+ dma_addr = dma_map_single(&dev->pdev->dev,
+ chunk_list->chunks[i].buf,
+ chunk_list->chunks[i].length,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(&dev->pdev->dev, dma_addr)) {
+ ibdev_err(&dev->ibdev,
+ "chunk[%u] dma_map_failed\n", i);
+ goto chunk_list_unmap;
+ }
+
+ chunk_list->chunks[i].dma_addr = dma_addr;
+ ibdev_dbg(&dev->ibdev,
+ "chunk[%u] mapped at [%pad]\n", i, &dma_addr);
+
+ if (!i)
+ break;
+
+ prev_chunk_buf = chunk_list->chunks[i - 1].buf;
+
+ ctrl_buf = (struct efa_com_ctrl_buff_info *)
+ &prev_chunk_buf[EFA_PTRS_PER_CHUNK];
+ ctrl_buf->length = chunk_list->chunks[i].length;
+
+ efa_com_set_dma_addr(dma_addr,
+ &ctrl_buf->address.mem_addr_high,
+ &ctrl_buf->address.mem_addr_low);
+ }
+
+ return 0;
+
+chunk_list_unmap:
+ for (; i < chunk_list_size; i++) {
+ dma_unmap_single(&dev->pdev->dev, chunk_list->chunks[i].dma_addr,
+ chunk_list->chunks[i].length, DMA_TO_DEVICE);
+ }
+chunk_list_dealloc:
+ for (i = 0; i < chunk_list_size; i++)
+ kfree(chunk_list->chunks[i].buf);
+
+ kfree(chunk_list->chunks);
+ return -ENOMEM;
+}
+
+static void pbl_chunk_list_destroy(struct efa_dev *dev, struct pbl_context *pbl)
+{
+ struct pbl_chunk_list *chunk_list = &pbl->phys.indirect.chunk_list;
+ int i;
+
+ for (i = 0; i < chunk_list->size; i++) {
+ dma_unmap_single(&dev->pdev->dev, chunk_list->chunks[i].dma_addr,
+ chunk_list->chunks[i].length, DMA_TO_DEVICE);
+ kfree(chunk_list->chunks[i].buf);
+ }
+
+ kfree(chunk_list->chunks);
+}
+
+/* initialize pbl continuous mode: map pbl buffer to a dma address. */
+static int pbl_continuous_initialize(struct efa_dev *dev,
+ struct pbl_context *pbl)
+{
+ dma_addr_t dma_addr;
+
+ dma_addr = dma_map_single(&dev->pdev->dev, pbl->pbl_buf,
+ pbl->pbl_buf_size_in_bytes, DMA_TO_DEVICE);
+ if (dma_mapping_error(&dev->pdev->dev, dma_addr)) {
+ ibdev_err(&dev->ibdev, "Unable to map pbl to DMA address\n");
+ return -ENOMEM;
+ }
+
+ pbl->phys.continuous.dma_addr = dma_addr;
+ ibdev_dbg(&dev->ibdev,
+ "pbl continuous - dma_addr = %pad, size[%u]\n",
+ &dma_addr, pbl->pbl_buf_size_in_bytes);
+
+ return 0;
+}
+
+/*
+ * initialize pbl indirect mode:
+ * create a chunk list out of the dma addresses of the physical pages of
+ * pbl buffer.
+ */
+static int pbl_indirect_initialize(struct efa_dev *dev, struct pbl_context *pbl)
+{
+ u32 size_in_pages = DIV_ROUND_UP(pbl->pbl_buf_size_in_bytes, PAGE_SIZE);
+ struct scatterlist *sgl;
+ int sg_dma_cnt, err;
+
+ BUILD_BUG_ON(EFA_CHUNK_PAYLOAD_SIZE > PAGE_SIZE);
+ sgl = efa_vmalloc_buf_to_sg(pbl->pbl_buf, size_in_pages);
+ if (!sgl)
+ return -ENOMEM;
+
+ sg_dma_cnt = dma_map_sg(&dev->pdev->dev, sgl, size_in_pages, DMA_TO_DEVICE);
+ if (!sg_dma_cnt) {
+ err = -EINVAL;
+ goto err_map;
+ }
+
+ pbl->phys.indirect.pbl_buf_size_in_pages = size_in_pages;
+ pbl->phys.indirect.sgl = sgl;
+ pbl->phys.indirect.sg_dma_cnt = sg_dma_cnt;
+ err = pbl_chunk_list_create(dev, pbl);
+ if (err) {
+ ibdev_dbg(&dev->ibdev,
+ "chunk_list creation failed[%d]\n", err);
+ goto err_chunk;
+ }
+
+ ibdev_dbg(&dev->ibdev,
+ "pbl indirect - size[%u], chunks[%u]\n",
+ pbl->pbl_buf_size_in_bytes,
+ pbl->phys.indirect.chunk_list.size);
+
+ return 0;
+
+err_chunk:
+ dma_unmap_sg(&dev->pdev->dev, sgl, size_in_pages, DMA_TO_DEVICE);
+err_map:
+ kfree(sgl);
+ return err;
+}
+
+static void pbl_indirect_terminate(struct efa_dev *dev, struct pbl_context *pbl)
+{
+ pbl_chunk_list_destroy(dev, pbl);
+ dma_unmap_sg(&dev->pdev->dev, pbl->phys.indirect.sgl,
+ pbl->phys.indirect.pbl_buf_size_in_pages, DMA_TO_DEVICE);
+ kfree(pbl->phys.indirect.sgl);
+}
+
+/* create a page buffer list from a mapped user memory region */
+static int pbl_create(struct efa_dev *dev,
+ struct pbl_context *pbl,
+ struct ib_umem *umem,
+ int hp_cnt,
+ u8 hp_shift)
+{
+ int err;
+
+ pbl->pbl_buf_size_in_bytes = hp_cnt * EFA_CHUNK_PAYLOAD_PTR_SIZE;
+ pbl->pbl_buf = kzalloc(pbl->pbl_buf_size_in_bytes,
+ GFP_KERNEL | __GFP_NOWARN);
+ if (pbl->pbl_buf) {
+ pbl->physically_continuous = 1;
+ err = umem_to_page_list(dev, umem, pbl->pbl_buf, hp_cnt,
+ hp_shift);
+ if (err)
+ goto err_continuous;
+ err = pbl_continuous_initialize(dev, pbl);
+ if (err)
+ goto err_continuous;
+ } else {
+ pbl->physically_continuous = 0;
+ pbl->pbl_buf = vzalloc(pbl->pbl_buf_size_in_bytes);
+ if (!pbl->pbl_buf)
+ return -ENOMEM;
+
+ err = umem_to_page_list(dev, umem, pbl->pbl_buf, hp_cnt,
+ hp_shift);
+ if (err)
+ goto err_indirect;
+ err = pbl_indirect_initialize(dev, pbl);
+ if (err)
+ goto err_indirect;
+ }
+
+ ibdev_dbg(&dev->ibdev,
+ "user_pbl_created: user_pages[%u], continuous[%u]\n",
+ hp_cnt, pbl->physically_continuous);
+
+ return 0;
+
+err_continuous:
+ kfree(pbl->pbl_buf);
+ return err;
+err_indirect:
+ vfree(pbl->pbl_buf);
+ return err;
+}
+
+static void pbl_destroy(struct efa_dev *dev, struct pbl_context *pbl)
+{
+ if (pbl->physically_continuous) {
+ dma_unmap_single(&dev->pdev->dev, pbl->phys.continuous.dma_addr,
+ pbl->pbl_buf_size_in_bytes, DMA_TO_DEVICE);
+ kfree(pbl->pbl_buf);
+ } else {
+ pbl_indirect_terminate(dev, pbl);
+ vfree(pbl->pbl_buf);
+ }
+}
+
+static int efa_create_inline_pbl(struct efa_dev *dev, struct efa_mr *mr,
+ struct efa_com_reg_mr_params *params)
+{
+ int err;
+
+ params->inline_pbl = 1;
+ err = umem_to_page_list(dev, mr->umem, params->pbl.inline_pbl_array,
+ params->page_num, params->page_shift);
+ if (err)
+ return err;
+
+ ibdev_dbg(&dev->ibdev,
+ "inline_pbl_array - pages[%u]\n", params->page_num);
+
+ return 0;
+}
+
+static int efa_create_pbl(struct efa_dev *dev,
+ struct pbl_context *pbl,
+ struct efa_mr *mr,
+ struct efa_com_reg_mr_params *params)
+{
+ int err;
+
+ err = pbl_create(dev, pbl, mr->umem, params->page_num,
+ params->page_shift);
+ if (err) {
+ ibdev_dbg(&dev->ibdev, "Failed to create pbl[%d]\n", err);
+ return err;
+ }
+
+ params->inline_pbl = 0;
+ params->indirect = !pbl->physically_continuous;
+ if (pbl->physically_continuous) {
+ params->pbl.pbl.length = pbl->pbl_buf_size_in_bytes;
+
+ efa_com_set_dma_addr(pbl->phys.continuous.dma_addr,
+ ¶ms->pbl.pbl.address.mem_addr_high,
+ ¶ms->pbl.pbl.address.mem_addr_low);
+ } else {
+ params->pbl.pbl.length =
+ pbl->phys.indirect.chunk_list.chunks[0].length;
+
+ efa_com_set_dma_addr(pbl->phys.indirect.chunk_list.chunks[0].dma_addr,
+ ¶ms->pbl.pbl.address.mem_addr_high,
+ ¶ms->pbl.pbl.address.mem_addr_low);
+ }
+
+ return 0;
+}
+
+static void efa_cont_pages(struct ib_umem *umem, u64 addr,
+ unsigned long max_page_shift,
+ int *count, u8 *shift, u32 *ncont)
+{
+ struct scatterlist *sg;
+ u64 base = ~0, p = 0;
+ unsigned long tmp;
+ unsigned long m;
+ u64 len, pfn;
+ int i = 0;
+ int entry;
+
+ addr = addr >> PAGE_SHIFT;
+ tmp = (unsigned long)addr;
+ m = find_first_bit(&tmp, BITS_PER_LONG);
+ if (max_page_shift)
+ m = min_t(unsigned long, max_page_shift - PAGE_SHIFT, m);
+
+ for_each_sg(umem->sg_head.sgl, sg, umem->nmap, entry) {
+ len = DIV_ROUND_UP(sg_dma_len(sg), PAGE_SIZE);
+ pfn = sg_dma_address(sg) >> PAGE_SHIFT;
+ if (base + p != pfn) {
+ /*
+ * If either the offset or the new
+ * base are unaligned update m
+ */
+ tmp = (unsigned long)(pfn | p);
+ if (!IS_ALIGNED(tmp, 1 << m))
+ m = find_first_bit(&tmp, BITS_PER_LONG);
+
+ base = pfn;
+ p = 0;
+ }
+
+ p += len;
+ i += len;
+ }
+
+ if (i) {
+ m = min_t(unsigned long, ilog2(roundup_pow_of_two(i)), m);
+ *ncont = DIV_ROUND_UP(i, (1 << m));
+ } else {
+ m = 0;
+ *ncont = 0;
+ }
+
+ *shift = PAGE_SHIFT + m;
+ *count = i;
+}
+
+struct ib_mr *efa_reg_mr(struct ib_pd *ibpd, u64 start, u64 length,
+ u64 virt_addr, int access_flags,
+ struct ib_udata *udata)
+{
+ struct efa_dev *dev = to_edev(ibpd->device);
+ struct efa_com_reg_mr_params params = {};
+ struct efa_com_reg_mr_result result = {};
+ unsigned long max_page_shift;
+ struct pbl_context pbl;
+ struct efa_mr *mr;
+ int inline_size;
+ int npages;
+ int err;
+
+ if (udata->inlen &&
+ !ib_is_udata_cleared(udata, 0, sizeof(udata->inlen))) {
+ ibdev_dbg(&dev->ibdev,
+ "Incompatible ABI params, udata not cleared\n");
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ if (access_flags & ~EFA_SUPPORTED_ACCESS_FLAGS) {
+ ibdev_dbg(&dev->ibdev,
+ "Unsupported access flags[%#x], supported[%#x]\n",
+ access_flags, EFA_SUPPORTED_ACCESS_FLAGS);
+ err = -EOPNOTSUPP;
+ goto err_out;
+ }
+
+ mr = kzalloc(sizeof(*mr), GFP_KERNEL);
+ if (!mr) {
+ err = -ENOMEM;
+ goto err_out;
+ }
+
+ mr->umem = ib_umem_get(udata, start, length, access_flags, 0);
+ if (IS_ERR(mr->umem)) {
+ err = PTR_ERR(mr->umem);
+ ibdev_dbg(&dev->ibdev,
+ "Failed to pin and map user space memory[%d]\n", err);
+ goto err_free;
+ }
+
+ params.pd = to_epd(ibpd)->pdn;
+ params.iova = virt_addr;
+ params.mr_length_in_bytes = length;
+ params.permissions = access_flags & 0x1;
+ max_page_shift = fls64(dev->dev_attr.page_size_cap);
+
+ efa_cont_pages(mr->umem, start, max_page_shift, &npages,
+ ¶ms.page_shift, ¶ms.page_num);
+ ibdev_dbg(&dev->ibdev,
+ "start %#llx length %#llx npages %d params.page_shift %u params.page_num %u\n",
+ start, length, npages, params.page_shift, params.page_num);
+
+ inline_size = ARRAY_SIZE(params.pbl.inline_pbl_array);
+ if (params.page_num <= inline_size) {
+ err = efa_create_inline_pbl(dev, mr, ¶ms);
+ if (err)
+ goto err_unmap;
+
+ err = efa_com_register_mr(&dev->edev, ¶ms, &result);
+ if (err)
+ goto err_unmap;
+ } else {
+ err = efa_create_pbl(dev, &pbl, mr, ¶ms);
+ if (err)
+ goto err_unmap;
+
+ err = efa_com_register_mr(&dev->edev, ¶ms, &result);
+ pbl_destroy(dev, &pbl);
+
+ if (err)
+ goto err_unmap;
+ }
+
+ mr->ibmr.lkey = result.l_key;
+ mr->ibmr.rkey = result.r_key;
+ mr->ibmr.length = length;
+ ibdev_dbg(&dev->ibdev, "Registered mr[%d]\n", mr->ibmr.lkey);
+
+ return &mr->ibmr;
+
+err_unmap:
+ ib_umem_release(mr->umem);
+err_free:
+ kfree(mr);
+err_out:
+ atomic64_inc(&dev->stats.sw_stats.reg_mr_err);
+ return ERR_PTR(err);
+}
+
+int efa_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
+{
+ struct efa_dev *dev = to_edev(ibmr->device);
+ struct efa_com_dereg_mr_params params;
+ struct efa_mr *mr = to_emr(ibmr);
+ int err;
+
+ if (udata->inlen &&
+ !ib_is_udata_cleared(udata, 0, udata->inlen)) {
+ ibdev_dbg(&dev->ibdev, "Incompatible ABI params\n");
+ return -EINVAL;
+ }
+
+ ibdev_dbg(&dev->ibdev, "Deregister mr[%d]\n", ibmr->lkey);
+
+ if (mr->umem) {
+ params.l_key = mr->ibmr.lkey;
+ err = efa_com_dereg_mr(&dev->edev, ¶ms);
+ if (err)
+ return err;
+ ib_umem_release(mr->umem);
+ }
+
+ kfree(mr);
+
+ return 0;
+}
+
+int efa_get_port_immutable(struct ib_device *ibdev, u8 port_num,
+ struct ib_port_immutable *immutable)
+{
+ struct ib_port_attr attr;
+ int err;
+
+ err = ib_query_port(ibdev, port_num, &attr);
+ if (err) {
+ ibdev_dbg(ibdev, "Couldn't query port err[%d]\n", err);
+ return err;
+ }
+
+ immutable->pkey_tbl_len = attr.pkey_tbl_len;
+ immutable->gid_tbl_len = attr.gid_tbl_len;
+
+ return 0;
+}
+
+static int efa_dealloc_uar(struct efa_dev *dev, u16 uarn)
+{
+ struct efa_com_dealloc_uar_params params = {
+ .uarn = uarn,
+ };
+
+ return efa_com_dealloc_uar(&dev->edev, ¶ms);
+}
+
+int efa_alloc_ucontext(struct ib_ucontext *ibucontext, struct ib_udata *udata)
+{
+ struct efa_ucontext *ucontext = to_eucontext(ibucontext);
+ struct efa_dev *dev = to_edev(ibucontext->device);
+ struct efa_ibv_alloc_ucontext_resp resp = {};
+ struct efa_com_alloc_uar_result result;
+ int err;
+
+ /*
+ * it's fine if the driver does not know all request fields,
+ * we will ack input fields in our response.
+ */
+
+ err = efa_com_alloc_uar(&dev->edev, &result);
+ if (err)
+ goto err_out;
+
+ ucontext->uarn = result.uarn;
+ xa_init(&ucontext->mmap_xa);
+
+ resp.cmds_supp_udata_mask |= EFA_USER_CMDS_SUPP_UDATA_QUERY_DEVICE;
+ resp.cmds_supp_udata_mask |= EFA_USER_CMDS_SUPP_UDATA_CREATE_AH;
+ resp.sub_cqs_per_cq = dev->dev_attr.sub_cqs_per_cq;
+ resp.inline_buf_size = dev->dev_attr.inline_buf_size;
+ resp.max_llq_size = dev->dev_attr.max_llq_size;
+
+ if (udata && udata->outlen) {
+ err = ib_copy_to_udata(udata, &resp,
+ min(sizeof(resp), udata->outlen));
+ if (err)
+ goto err_dealloc_uar;
+ }
+
+ return 0;
+
+err_dealloc_uar:
+ efa_dealloc_uar(dev, result.uarn);
+err_out:
+ atomic64_inc(&dev->stats.sw_stats.alloc_ucontext_err);
+ return err;
+}
+
+void efa_dealloc_ucontext(struct ib_ucontext *ibucontext)
+{
+ struct efa_ucontext *ucontext = to_eucontext(ibucontext);
+ struct efa_dev *dev = to_edev(ibucontext->device);
+
+ mmap_entries_remove_free(dev, ucontext);
+ efa_dealloc_uar(dev, ucontext->uarn);
+}
+
+static int __efa_mmap(struct efa_dev *dev, struct efa_ucontext *ucontext,
+ struct vm_area_struct *vma, u64 key, u64 length)
+{
+ struct efa_mmap_entry *entry;
+ unsigned long va;
+ u64 pfn;
+ int err;
+
+ entry = mmap_entry_get(dev, ucontext, key, length);
+ if (!entry) {
+ ibdev_dbg(&dev->ibdev, "key[%#llx] does not have valid entry\n",
+ key);
+ return -EINVAL;
+ }
+
+ ibdev_dbg(&dev->ibdev,
+ "Mapping address[%#llx], length[%#llx], mmap_flag[%d]\n",
+ entry->address, length, entry->mmap_flag);
+
+ pfn = entry->address >> PAGE_SHIFT;
+ switch (entry->mmap_flag) {
+ case EFA_MMAP_IO_NC:
+ err = rdma_user_mmap_io(&ucontext->ibucontext, vma, pfn, length,
+ pgprot_noncached(vma->vm_page_prot));
+ break;
+ case EFA_MMAP_IO_WC:
+ err = rdma_user_mmap_io(&ucontext->ibucontext, vma, pfn, length,
+ pgprot_writecombine(vma->vm_page_prot));
+ break;
+ case EFA_MMAP_DMA_PAGE:
+ for (va = vma->vm_start; va < vma->vm_end;
+ va += PAGE_SIZE, pfn++) {
+ err = vm_insert_page(vma, va, pfn_to_page(pfn));
+ if (err)
+ break;
+ }
+ break;
+ default:
+ err = -EINVAL;
+ }
+
+ if (err)
+ ibdev_dbg(
+ &dev->ibdev,
+ "Couldn't mmap address[%#llx] length[%#llx] mmap_flag[%d] err[%d]\n",
+ entry->address, length, entry->mmap_flag, err);
+
+ return err;
+}
+
+int efa_mmap(struct ib_ucontext *ibucontext,
+ struct vm_area_struct *vma)
+{
+ struct efa_ucontext *ucontext = to_eucontext(ibucontext);
+ struct efa_dev *dev = to_edev(ibucontext->device);
+ u64 length = vma->vm_end - vma->vm_start;
+ u64 key = vma->vm_pgoff << PAGE_SHIFT;
+
+ ibdev_dbg(&dev->ibdev,
+ "start %#lx, end %#lx, length = %#llx, key = %#llx\n",
+ vma->vm_start, vma->vm_end, length, key);
+
+ if (length % PAGE_SIZE != 0 || !(vma->vm_flags & VM_SHARED)) {
+ ibdev_dbg(&dev->ibdev,
+ "length[%#llx] is not page size aligned[%#lx] or VM_SHARED is not set [%#lx]\n",
+ length, PAGE_SIZE, vma->vm_flags);
+ return -EINVAL;
+ }
+
+ if (vma->vm_flags & VM_EXEC) {
+ ibdev_dbg(&dev->ibdev, "Mapping executable pages is not permitted\n");
+ return -EPERM;
+ }
+ vma->vm_flags &= ~VM_MAYEXEC;
+
+ return __efa_mmap(dev, ucontext, vma, key, length);
+}
+
+static int efa_ah_destroy(struct efa_dev *dev, struct efa_ah *ah)
+{
+ struct efa_com_destroy_ah_params params = {
+ .ah = ah->ah,
+ .pdn = to_epd(ah->ibah.pd)->pdn,
+ };
+
+ return efa_com_destroy_ah(&dev->edev, ¶ms);
+}
+
+int efa_create_ah(struct ib_ah *ibah,
+ struct rdma_ah_attr *ah_attr,
+ u32 flags,
+ struct ib_udata *udata)
+{
+ struct efa_dev *dev = to_edev(ibah->device);
+ struct efa_com_create_ah_params params = {};
+ struct efa_ibv_create_ah_resp resp = {};
+ struct efa_com_create_ah_result result;
+ struct efa_ah *ah = to_eah(ibah);
+ int err;
+
+ if (!(flags & RDMA_CREATE_AH_SLEEPABLE)) {
+ ibdev_dbg(&dev->ibdev,
+ "Create address handle is not supported in atomic context\n");
+ err = -EOPNOTSUPP;
+ goto err_out;
+ }
+
+ if (udata->inlen &&
+ !ib_is_udata_cleared(udata, 0, udata->inlen)) {
+ ibdev_dbg(&dev->ibdev, "Incompatible ABI params\n");
+ err = -EINVAL;
+ goto err_out;
+ }
+
+ memcpy(params.dest_addr, ah_attr->grh.dgid.raw,
+ sizeof(params.dest_addr));
+ params.pdn = to_epd(ibah->pd)->pdn;
+ err = efa_com_create_ah(&dev->edev, ¶ms, &result);
+ if (err)
+ goto err_out;
+
+ memcpy(ah->id, ah_attr->grh.dgid.raw, sizeof(ah->id));
+ ah->ah = result.ah;
+
+ resp.efa_address_handle = result.ah;
+
+ if (udata->outlen) {
+ err = ib_copy_to_udata(udata, &resp,
+ min(sizeof(resp), udata->outlen));
+ if (err) {
+ ibdev_dbg(&dev->ibdev,
+ "Failed to copy udata for create_ah response\n");
+ goto err_destroy_ah;
+ }
+ }
+ ibdev_dbg(&dev->ibdev, "Created ah[%d]\n", ah->ah);
+
+ return 0;
+
+err_destroy_ah:
+ efa_ah_destroy(dev, ah);
+err_out:
+ atomic64_inc(&dev->stats.sw_stats.create_ah_err);
+ return err;
+}
+
+void efa_destroy_ah(struct ib_ah *ibah, u32 flags)
+{
+ struct efa_dev *dev = to_edev(ibah->pd->device);
+ struct efa_ah *ah = to_eah(ibah);
+
+ ibdev_dbg(&dev->ibdev, "Destroy ah[%d]\n", ah->ah);
+
+ if (!(flags & RDMA_DESTROY_AH_SLEEPABLE)) {
+ ibdev_dbg(&dev->ibdev,
+ "Destroy address handle is not supported in atomic context\n");
+ return;
+ }
+
+ efa_ah_destroy(dev, ah);
+}
+
+enum rdma_link_layer efa_port_link_layer(struct ib_device *ibdev,
+ u8 port_num)
+{
+ return IB_LINK_LAYER_UNSPECIFIED;
+}
+
static struct cntr_entry dev_cntrs[DEV_CNTR_LAST] = {
[C_RCV_OVF] = RXE32_DEV_CNTR_ELEM(RcvOverflow, RCV_BUF_OVFL_CNT, CNTR_SYNTH),
+[C_RX_LEN_ERR] = RXE32_DEV_CNTR_ELEM(RxLenErr, RCV_LENGTH_ERR_CNT, CNTR_SYNTH),
+[C_RX_ICRC_ERR] = RXE32_DEV_CNTR_ELEM(RxICrcErr, RCV_ICRC_ERR_CNT, CNTR_SYNTH),
+[C_RX_EBP] = RXE32_DEV_CNTR_ELEM(RxEbpCnt, RCV_EBP_CNT, CNTR_SYNTH),
[C_RX_TID_FULL] = RXE32_DEV_CNTR_ELEM(RxTIDFullEr, RCV_TID_FULL_ERR_CNT,
CNTR_NORMAL),
[C_RX_TID_INVALID] = RXE32_DEV_CNTR_ELEM(RxTIDInvalid, RCV_TID_VALID_ERR_CNT,
/*
* The RMT entries are currently allocated as shown below:
* 1. QOS (0 to 128 entries);
- * 2. FECN for PSM (num_user_contexts + num_vnic_contexts);
+ * 2. FECN (num_kernel_context - 1 + num_user_contexts +
+ * num_vnic_contexts);
* 3. VNIC (num_vnic_contexts).
- * It should be noted that PSM FECN oversubscribe num_vnic_contexts
+ * It should be noted that FECN oversubscribe num_vnic_contexts
* entries of RMT because both VNIC and PSM could allocate any receive
* context between dd->first_dyn_alloc_text and dd->num_rcv_contexts,
* and PSM FECN must reserve an RMT entry for each possible PSM receive
* context.
*/
rmt_count = qos_rmt_entries(dd, NULL, NULL) + (num_vnic_contexts * 2);
+ if (HFI1_CAP_IS_KSET(TID_RDMA))
+ rmt_count += num_kernel_contexts - 1;
if (rmt_count + n_usr_ctxts > NUM_MAP_ENTRIES) {
user_rmt_reduced = NUM_MAP_ENTRIES - rmt_count;
dd_dev_err(dd,
init_qpmap_table(dd, FIRST_KERNEL_KCTXT, dd->n_krcv_queues - 1);
}
-static void init_user_fecn_handling(struct hfi1_devdata *dd,
- struct rsm_map_table *rmt)
+static void init_fecn_handling(struct hfi1_devdata *dd,
+ struct rsm_map_table *rmt)
{
struct rsm_rule_data rrd;
u64 reg;
- int i, idx, regoff, regidx;
+ int i, idx, regoff, regidx, start;
u8 offset;
u32 total_cnt;
+ if (HFI1_CAP_IS_KSET(TID_RDMA))
+ /* Exclude context 0 */
+ start = 1;
+ else
+ start = dd->first_dyn_alloc_ctxt;
+
+ total_cnt = dd->num_rcv_contexts - start;
+
/* there needs to be enough room in the map table */
- total_cnt = dd->num_rcv_contexts - dd->first_dyn_alloc_ctxt;
if (rmt->used + total_cnt >= NUM_MAP_ENTRIES) {
- dd_dev_err(dd, "User FECN handling disabled - too many user contexts allocated\n");
+ dd_dev_err(dd, "FECN handling disabled - too many contexts allocated\n");
return;
}
/*
* RSM will extract the destination context as an index into the
* map table. The destination contexts are a sequential block
- * in the range first_dyn_alloc_ctxt...num_rcv_contexts-1 (inclusive).
+ * in the range start...num_rcv_contexts-1 (inclusive).
* Map entries are accessed as offset + extracted value. Adjust
* the added offset so this sequence can be placed anywhere in
* the table - as long as the entries themselves do not wrap.
* There are only enough bits in offset for the table size, so
* start with that to allow for a "negative" offset.
*/
- offset = (u8)(NUM_MAP_ENTRIES + (int)rmt->used -
- (int)dd->first_dyn_alloc_ctxt);
+ offset = (u8)(NUM_MAP_ENTRIES + rmt->used - start);
- for (i = dd->first_dyn_alloc_ctxt, idx = rmt->used;
- i < dd->num_rcv_contexts; i++, idx++) {
+ for (i = start, idx = rmt->used; i < dd->num_rcv_contexts;
+ i++, idx++) {
/* replace with identity mapping */
regoff = (idx % 8) * 8;
regidx = idx / 8;
rmt = alloc_rsm_map_table(dd);
/* set up QOS, including the QPN map table */
init_qos(dd, rmt);
- init_user_fecn_handling(dd, rmt);
+ init_fecn_handling(dd, rmt);
complete_rsm_map_table(dd, rmt);
/* record number of used rsm map entries for vnic */
dd->vnic.rmt_start = rmt->used;
*/
static int init_asic_data(struct hfi1_devdata *dd)
{
- unsigned long flags;
- struct hfi1_devdata *tmp, *peer = NULL;
+ unsigned long index;
+ struct hfi1_devdata *peer;
struct hfi1_asic_data *asic_data;
int ret = 0;
if (!asic_data)
return -ENOMEM;
- spin_lock_irqsave(&hfi1_devs_lock, flags);
+ xa_lock_irq(&hfi1_dev_table);
/* Find our peer device */
- list_for_each_entry(tmp, &hfi1_dev_list, list) {
- if ((HFI_BASE_GUID(dd) == HFI_BASE_GUID(tmp)) &&
- dd->unit != tmp->unit) {
- peer = tmp;
+ xa_for_each(&hfi1_dev_table, index, peer) {
+ if ((HFI_BASE_GUID(dd) == HFI_BASE_GUID(peer)) &&
+ dd->unit != peer->unit)
break;
- }
}
if (peer) {
mutex_init(&dd->asic_data->asic_resource_mutex);
}
dd->asic_data->dds[dd->hfi1_id] = dd; /* self back-pointer */
- spin_unlock_irqrestore(&hfi1_devs_lock, flags);
+ xa_unlock_irq(&hfi1_dev_table);
/* first one through - set up i2c devices */
if (!peer)
/* Per device counter indexes */
enum {
C_RCV_OVF = 0,
+ C_RX_LEN_ERR,
+ C_RX_ICRC_ERR,
+ C_RX_EBP,
C_RX_TID_FULL,
C_RX_TID_INVALID,
C_RX_TID_FLGMS,
#define DC_LCB_PRF_TX_FLIT_CNT (DC_LCB_CSRS + 0x000000000418)
#define DC_LCB_STS_LINK_TRANSFER_ACTIVE (DC_LCB_CSRS + 0x000000000468)
#define DC_LCB_STS_ROUND_TRIP_LTP_CNT (DC_LCB_CSRS + 0x0000000004B0)
+#define RCV_LENGTH_ERR_CNT 0
+#define RCV_ICRC_ERR_CNT 6
+#define RCV_EBP_CNT 9
#define RCV_BUF_OVFL_CNT 10
#define RCV_CONTEXT_EGR_STALL 22
#define RCV_DATA_PKT_CNT 0
#define RHF_TID_ERR (0x1ull << 59)
#define RHF_LEN_ERR (0x1ull << 60)
#define RHF_ECC_ERR (0x1ull << 61)
-#define RHF_VCRC_ERR (0x1ull << 62)
+#define RHF_RESERVED (0x1ull << 62)
#define RHF_ICRC_ERR (0x1ull << 63)
#define RHF_ERROR_SMASK 0xffe0000000000000ull /* bits 63:53 */
return __qsfp_debugfs_release(in, fp, 1);
}
+#define EXPROM_WRITE_ENABLE BIT_ULL(14)
+
+static bool exprom_wp_disabled;
+
+static int exprom_wp_set(struct hfi1_devdata *dd, bool disable)
+{
+ u64 gpio_val = 0;
+
+ if (disable) {
+ gpio_val = EXPROM_WRITE_ENABLE;
+ exprom_wp_disabled = true;
+ dd_dev_info(dd, "Disable Expansion ROM Write Protection\n");
+ } else {
+ exprom_wp_disabled = false;
+ dd_dev_info(dd, "Enable Expansion ROM Write Protection\n");
+ }
+
+ write_csr(dd, ASIC_GPIO_OUT, gpio_val);
+ write_csr(dd, ASIC_GPIO_OE, gpio_val);
+
+ return 0;
+}
+
+static ssize_t exprom_wp_debugfs_read(struct file *file, char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ return 0;
+}
+
+static ssize_t exprom_wp_debugfs_write(struct file *file,
+ const char __user *buf, size_t count,
+ loff_t *ppos)
+{
+ struct hfi1_pportdata *ppd = private2ppd(file);
+ char cdata;
+
+ if (count != 1)
+ return -EINVAL;
+ if (get_user(cdata, buf))
+ return -EFAULT;
+ if (cdata == '0')
+ exprom_wp_set(ppd->dd, false);
+ else if (cdata == '1')
+ exprom_wp_set(ppd->dd, true);
+ else
+ return -EINVAL;
+
+ return 1;
+}
+
+static unsigned long exprom_in_use;
+
+static int exprom_wp_debugfs_open(struct inode *in, struct file *fp)
+{
+ if (test_and_set_bit(0, &exprom_in_use))
+ return -EBUSY;
+
+ return 0;
+}
+
+static int exprom_wp_debugfs_release(struct inode *in, struct file *fp)
+{
+ struct hfi1_pportdata *ppd = private2ppd(fp);
+
+ if (exprom_wp_disabled)
+ exprom_wp_set(ppd->dd, false);
+ clear_bit(0, &exprom_in_use);
+
+ return 0;
+}
+
#define DEBUGFS_OPS(nm, readroutine, writeroutine) \
{ \
.name = nm, \
qsfp1_debugfs_open, qsfp1_debugfs_release),
DEBUGFS_XOPS("qsfp2", qsfp2_debugfs_read, qsfp2_debugfs_write,
qsfp2_debugfs_open, qsfp2_debugfs_release),
+ DEBUGFS_XOPS("exprom_wp", exprom_wp_debugfs_read,
+ exprom_wp_debugfs_write, exprom_wp_debugfs_open,
+ exprom_wp_debugfs_release),
DEBUGFS_OPS("asic_flags", asic_flags_read, asic_flags_write),
DEBUGFS_OPS("dc8051_memory", dc8051_memory_read, NULL),
DEBUGFS_OPS("lcb", debugfs_lcb_read, debugfs_lcb_write),
static u64 hfi1_sps_ints(void)
{
- unsigned long flags;
+ unsigned long index, flags;
struct hfi1_devdata *dd;
u64 sps_ints = 0;
- spin_lock_irqsave(&hfi1_devs_lock, flags);
- list_for_each_entry(dd, &hfi1_dev_list, list) {
+ xa_lock_irqsave(&hfi1_dev_table, flags);
+ xa_for_each(&hfi1_dev_table, index, dd) {
sps_ints += get_all_cpu_total(dd->int_counter);
}
- spin_unlock_irqrestore(&hfi1_devs_lock, flags);
+ xa_unlock_irqrestore(&hfi1_dev_table, flags);
return sps_ints;
}
*/
const char ib_hfi1_version[] = HFI1_DRIVER_VERSION "\n";
-DEFINE_SPINLOCK(hfi1_devs_lock);
-LIST_HEAD(hfi1_dev_list);
DEFINE_MUTEX(hfi1_mutex); /* general driver use */
unsigned int hfi1_max_mtu = HFI1_DEFAULT_MAX_MTU;
{
struct hfi1_devdata *dd;
struct hfi1_pportdata *ppd;
- unsigned long flags;
+ unsigned long index, flags;
int pidx, nunits_active = 0;
- spin_lock_irqsave(&hfi1_devs_lock, flags);
- list_for_each_entry(dd, &hfi1_dev_list, list) {
+ xa_lock_irqsave(&hfi1_dev_table, flags);
+ xa_for_each(&hfi1_dev_table, index, dd) {
if (!(dd->flags & HFI1_PRESENT) || !dd->kregbase1)
continue;
for (pidx = 0; pidx < dd->num_pports; ++pidx) {
}
}
}
- spin_unlock_irqrestore(&hfi1_devs_lock, flags);
+ xa_unlock_irqrestore(&hfi1_dev_table, flags);
return nunits_active;
}
hfi1_dbg_fault_suppress_err(verbs_dev))
return;
- if (packet->rhf & (RHF_VCRC_ERR | RHF_ICRC_ERR))
+ if (packet->rhf & RHF_ICRC_ERR)
return;
if (packet->etype == RHF_RCV_TYPE_BYPASS) {
*/
do_cnp = prescan ||
(opcode >= IB_OPCODE_RC_RDMA_READ_RESPONSE_FIRST &&
- opcode <= IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE);
+ opcode <= IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE) ||
+ opcode == TID_OP(READ_RESP) ||
+ opcode == TID_OP(ACK);
/* Call appropriate CNP handler */
if (!ignore_fecn && do_cnp && fecn)
u32 rte = rhf_rcv_type_err(packet->rhf);
dd_dev_err(rcd->dd,
- "receive context %d: rhf 0x%016llx, errs [ %s%s%s%s%s%s%s%s] rte 0x%x\n",
+ "receive context %d: rhf 0x%016llx, errs [ %s%s%s%s%s%s%s] rte 0x%x\n",
rcd->ctxt, packet->rhf,
packet->rhf & RHF_K_HDR_LEN_ERR ? "k_hdr_len " : "",
packet->rhf & RHF_DC_UNC_ERR ? "dc_unc " : "",
packet->rhf & RHF_TID_ERR ? "tid " : "",
packet->rhf & RHF_LEN_ERR ? "len " : "",
packet->rhf & RHF_ECC_ERR ? "ecc " : "",
- packet->rhf & RHF_VCRC_ERR ? "vcrc " : "",
packet->rhf & RHF_ICRC_ERR ? "icrc " : "",
rte);
}
*/
void hfi1_free_ctxt_rcv_groups(struct hfi1_ctxtdata *rcd)
{
- WARN_ON(!EXP_TID_SET_EMPTY(rcd->tid_full_list));
- WARN_ON(!EXP_TID_SET_EMPTY(rcd->tid_used_list));
-
kfree(rcd->groups);
rcd->groups = NULL;
hfi1_exp_tid_group_init(rcd);
#include <linux/list.h>
#include <linux/scatterlist.h>
#include <linux/slab.h>
-#include <linux/idr.h>
#include <linux/io.h>
#include <linux/fs.h>
#include <linux/completion.h>
#include <linux/kthread.h>
#include <linux/i2c.h>
#include <linux/i2c-algo-bit.h>
+#include <linux/xarray.h>
#include <rdma/ib_hdrs.h>
#include <rdma/opa_addr.h>
#include <linux/rhashtable.h>
struct hfi1_vnic_data {
struct hfi1_ctxtdata *ctxt[HFI1_NUM_VNIC_CTXT];
struct kmem_cache *txreq_cache;
+ struct xarray vesws;
u8 num_vports;
- struct idr vesw_idr;
u8 rmt_start;
u8 num_ctxt;
};
typedef int (*send_routine)(struct rvt_qp *, struct hfi1_pkt_state *, u64);
struct hfi1_devdata {
struct hfi1_ibdev verbs_dev; /* must be first */
- struct list_head list;
/* pointers to related structs for this device */
/* pci access data structure */
struct pci_dev *pcidev;
struct mm_struct *mm;
};
-extern struct list_head hfi1_dev_list;
-extern spinlock_t hfi1_devs_lock;
+extern struct xarray hfi1_dev_table;
struct hfi1_devdata *hfi1_lookup(int unit);
static inline unsigned long uctxt_offset(struct hfi1_ctxtdata *uctxt)
#include <linux/netdevice.h>
#include <linux/vmalloc.h>
#include <linux/delay.h>
-#include <linux/idr.h>
+#include <linux/xarray.h>
#include <linux/module.h>
#include <linux/printk.h>
#include <linux/hrtimer.h>
static inline u64 encode_rcv_header_entry_size(u16 size);
-static struct idr hfi1_unit_table;
+DEFINE_XARRAY_FLAGS(hfi1_dev_table, XA_FLAGS_ALLOC | XA_FLAGS_LOCK_IRQ);
static int hfi1_create_kctxt(struct hfi1_devdata *dd,
struct hfi1_pportdata *ppd)
if (rcd->egrbufs.size < hfi1_max_mtu) {
rcd->egrbufs.size = __roundup_pow_of_two(hfi1_max_mtu);
hfi1_cdbg(PROC,
- "ctxt%u: eager bufs size too small. Adjusting to %zu\n",
+ "ctxt%u: eager bufs size too small. Adjusting to %u\n",
rcd->ctxt, rcd->egrbufs.size);
}
rcd->egrbufs.rcvtid_size = HFI1_MAX_EAGER_BUFFER_SIZE;
ppd->hfi1_wq =
alloc_workqueue(
"hfi%d_%d",
- WQ_SYSFS | WQ_HIGHPRI | WQ_CPU_INTENSIVE,
+ WQ_SYSFS | WQ_HIGHPRI | WQ_CPU_INTENSIVE |
+ WQ_MEM_RECLAIM,
HFI1_MAX_ACTIVE_WORKQUEUE_ENTRIES,
dd->unit, pidx);
if (!ppd->hfi1_wq)
return ret;
}
-static inline struct hfi1_devdata *__hfi1_lookup(int unit)
-{
- return idr_find(&hfi1_unit_table, unit);
-}
-
struct hfi1_devdata *hfi1_lookup(int unit)
{
- struct hfi1_devdata *dd;
- unsigned long flags;
-
- spin_lock_irqsave(&hfi1_devs_lock, flags);
- dd = __hfi1_lookup(unit);
- spin_unlock_irqrestore(&hfi1_devs_lock, flags);
-
- return dd;
+ return xa_load(&hfi1_dev_table, unit);
}
/*
/*
* Release our hold on the shared asic data. If we are the last one,
* return the structure to be finalized outside the lock. Must be
- * holding hfi1_devs_lock.
+ * holding hfi1_dev_table lock.
*/
static struct hfi1_asic_data *release_asic_data(struct hfi1_devdata *dd)
{
struct hfi1_asic_data *ad;
unsigned long flags;
- spin_lock_irqsave(&hfi1_devs_lock, flags);
- if (!list_empty(&dd->list)) {
- idr_remove(&hfi1_unit_table, dd->unit);
- list_del_init(&dd->list);
- }
+ xa_lock_irqsave(&hfi1_dev_table, flags);
+ __xa_erase(&hfi1_dev_table, dd->unit);
ad = release_asic_data(dd);
- spin_unlock_irqrestore(&hfi1_devs_lock, flags);
+ xa_unlock_irqrestore(&hfi1_dev_table, flags);
finalize_asic_data(dd, ad);
free_platform_config(dd);
* Must be done via verbs allocator, because the verbs cleanup process
* both does cleanup and free of the data structure.
* "extra" is for chip-specific data.
- *
- * Use the idr mechanism to get a unit number for this unit.
*/
static struct hfi1_devdata *hfi1_alloc_devdata(struct pci_dev *pdev,
size_t extra)
{
- unsigned long flags;
struct hfi1_devdata *dd;
int ret, nports;
dd->pport = (struct hfi1_pportdata *)(dd + 1);
dd->pcidev = pdev;
pci_set_drvdata(pdev, dd);
-
- INIT_LIST_HEAD(&dd->list);
- idr_preload(GFP_KERNEL);
- spin_lock_irqsave(&hfi1_devs_lock, flags);
-
- ret = idr_alloc(&hfi1_unit_table, dd, 0, 0, GFP_NOWAIT);
- if (ret >= 0) {
- dd->unit = ret;
- list_add(&dd->list, &hfi1_dev_list);
- }
dd->node = NUMA_NO_NODE;
- spin_unlock_irqrestore(&hfi1_devs_lock, flags);
- idr_preload_end();
-
+ ret = xa_alloc_irq(&hfi1_dev_table, &dd->unit, dd, xa_limit_32b,
+ GFP_KERNEL);
if (ret < 0) {
dev_err(&pdev->dev,
"Could not allocate unit ID: error %d\n", -ret);
* These must be called before the driver is registered with
* the PCI subsystem.
*/
- idr_init(&hfi1_unit_table);
-
hfi1_dbg_init();
ret = pci_register_driver(&hfi1_pci_driver);
if (ret < 0) {
bail_dev:
hfi1_dbg_exit();
- idr_destroy(&hfi1_unit_table);
dev_cleanup();
bail:
return ret;
node_affinity_destroy_all();
hfi1_dbg_exit();
- idr_destroy(&hfi1_unit_table);
+ WARN_ON(!xa_empty(&hfi1_dev_table));
dispose_firmware(); /* asymmetric with obtain_firmware() */
dev_cleanup();
}
rcd->egrbufs.size = alloced_bytes;
hfi1_cdbg(PROC,
- "ctxt%u: Alloced %u rcv tid entries @ %uKB, total %zuKB\n",
+ "ctxt%u: Alloced %u rcv tid entries @ %uKB, total %uKB\n",
rcd->ctxt, rcd->egrbufs.alloced,
rcd->egrbufs.rcvtid_size / 1024, rcd->egrbufs.size / 1024);
* for future transactions
*/
+#include <linux/workqueue.h>
+#include <rdma/ib_verbs.h>
+#include <rdma/rdmavt_qp.h>
+
/* STL Verbs Extended */
#define IB_BTHE_E_SHIFT 24
#define HFI1_VERBS_E_ATOMIC_VADDR U64_MAX
-struct ib_atomic_eth;
-
enum hfi1_opfn_codes {
STL_VERBS_EXTD_NONE = 0,
STL_VERBS_EXTD_TID_RDMA,
iowait_wakeup,
iowait_sdma_drained,
hfi1_init_priority);
+ /* Init to a value to start the running average correctly */
+ priv->s_running_pkt_size = piothreshold / 2;
return priv;
}
case OP(RDMA_READ_RESPONSE_LAST):
case OP(RDMA_READ_RESPONSE_ONLY):
e = &qp->s_ack_queue[qp->s_tail_ack_queue];
- if (e->rdma_sge.mr) {
- rvt_put_mr(e->rdma_sge.mr);
- e->rdma_sge.mr = NULL;
- }
+ release_rdma_sge_mr(e);
/* FALLTHROUGH */
case OP(ATOMIC_ACKNOWLEDGE):
/*
break;
e->sent = 1;
- qp->s_ack_state = OP(RDMA_READ_RESPONSE_LAST);
+ /* Do not free e->rdma_sge until all data are received */
+ qp->s_ack_state = OP(ATOMIC_ACKNOWLEDGE);
break;
case TID_OP(READ_RESP):
qp->s_last = s_last;
/* see post_send() */
barrier();
- rvt_put_swqe(wqe);
+ rvt_put_qp_swqe(qp, wqe);
rvt_qp_swqe_complete(qp,
wqe,
ib_hfi1_wc_opcode[wqe->wr.opcode],
u32 s_last;
trdma_clean_swqe(qp, wqe);
- rvt_put_swqe(wqe);
+ rvt_put_qp_swqe(qp, wqe);
rvt_qp_wqe_unreserve(qp, wqe);
s_last = qp->s_last;
trace_hfi1_qp_send_completion(qp, wqe, s_last);
len = be32_to_cpu(reth->length);
if (unlikely(offset + len != e->rdma_sge.sge_length))
goto unlock_done;
- if (e->rdma_sge.mr) {
- rvt_put_mr(e->rdma_sge.mr);
- e->rdma_sge.mr = NULL;
- }
+ release_rdma_sge_mr(e);
if (len != 0) {
u32 rkey = be32_to_cpu(reth->rkey);
u64 vaddr = get_ib_reth_vaddr(reth);
update_ack_queue(qp, next);
}
e = &qp->s_ack_queue[qp->r_head_ack_queue];
- if (e->rdma_sge.mr) {
- rvt_put_mr(e->rdma_sge.mr);
- e->rdma_sge.mr = NULL;
- }
+ release_rdma_sge_mr(e);
reth = &ohdr->u.rc.reth;
len = be32_to_cpu(reth->length);
if (len) {
update_ack_queue(qp, next);
}
e = &qp->s_ack_queue[qp->r_head_ack_queue];
- if (e->rdma_sge.mr) {
- rvt_put_mr(e->rdma_sge.mr);
- e->rdma_sge.mr = NULL;
- }
+ release_rdma_sge_mr(e);
/* Process OPFN special virtual address */
if (opfn) {
opfn_conn_response(qp, e, ateth);
return rvt_restart_sge(ss, wqe, len);
}
+static inline void release_rdma_sge_mr(struct rvt_ack_entry *e)
+{
+ if (e->rdma_sge.mr) {
+ rvt_put_mr(e->rdma_sge.mr);
+ e->rdma_sge.mr = NULL;
+ }
+}
+
struct rvt_ack_entry *find_prev_entry(struct rvt_qp *qp, u32 psn, u8 *prev,
u8 *prev_ack, bool *scheduled);
int do_rc_ack(struct rvt_qp *qp, u32 aeth, u32 psn, int opcode, u64 val,
/**
* hfi1_do_send - perform a send on a QP
- * @work: contains a pointer to the QP
+ * @qp: a pointer to the QP
* @in_thread: true if in a workqueue thread
*
* Process entries in the send work queue until credit or queue is
#define TID_RDMA_DESTQP_FLOW_SHIFT 11
#define TID_RDMA_DESTQP_FLOW_MASK 0x1f
-#define TID_FLOW_SW_PSN BIT(0)
-
#define TID_OPFN_QP_CTXT_MASK 0xff
#define TID_OPFN_QP_CTXT_SHIFT 56
#define TID_OPFN_QP_KDETH_MASK 0xff
struct ib_other_headers *ohdr,
struct hfi1_pkt_state *ps);
static void hfi1_do_tid_send(struct rvt_qp *qp);
+static u32 read_r_next_psn(struct hfi1_devdata *dd, u8 ctxt, u8 fidx);
+static void tid_rdma_rcv_err(struct hfi1_packet *packet,
+ struct ib_other_headers *ohdr,
+ struct rvt_qp *qp, u32 psn, int diff, bool fecn);
+static void update_r_next_psn_fecn(struct hfi1_packet *packet,
+ struct hfi1_qp_priv *priv,
+ struct hfi1_ctxtdata *rcd,
+ struct tid_rdma_flow *flow,
+ bool fecn);
static u64 tid_rdma_opfn_encode(struct tid_rdma_params *p)
{
rcd->flows[fs->index].generation = fs->generation;
fs->generation = kern_setup_hw_flow(rcd, fs->index);
fs->psn = 0;
- fs->flags = 0;
dequeue_tid_waiter(rcd, &rcd->flow_queue, qp);
/* get head before dropping lock */
fqp = first_qp(rcd, &rcd->flow_queue);
goto done;
hfi1_kern_clear_hw_flow(req->rcd, qp);
+ qpriv->s_flags &= ~HFI1_R_TID_SW_PSN;
req->state = TID_REQUEST_ACTIVE;
}
if (psn != e->psn || len != req->total_len)
goto unlock;
- if (e->rdma_sge.mr) {
- rvt_put_mr(e->rdma_sge.mr);
- e->rdma_sge.mr = NULL;
- }
+ release_rdma_sge_mr(e);
rkey = be32_to_cpu(reth->rkey);
vaddr = get_ib_reth_vaddr(reth);
struct ib_reth *reth;
struct hfi1_qp_priv *qpriv = qp->priv;
u32 bth0, psn, len, rkey;
- bool is_fecn;
+ bool fecn;
u8 next;
u64 vaddr;
int diff;
if (hfi1_ruc_check_hdr(ibp, packet))
return;
- is_fecn = process_ecn(qp, packet);
+ fecn = process_ecn(qp, packet);
psn = mask_psn(be32_to_cpu(ohdr->bth[2]));
trace_hfi1_rsp_rcv_tid_read_req(qp, psn);
diff = delta_psn(psn, qp->r_psn);
if (unlikely(diff)) {
- if (tid_rdma_rcv_error(packet, ohdr, qp, psn, diff))
- return;
- goto send_ack;
+ tid_rdma_rcv_err(packet, ohdr, qp, psn, diff, fecn);
+ return;
}
/* We've verified the request, insert it into the ack queue. */
update_ack_queue(qp, next);
}
e = &qp->s_ack_queue[qp->r_head_ack_queue];
- if (e->rdma_sge.mr) {
- rvt_put_mr(e->rdma_sge.mr);
- e->rdma_sge.mr = NULL;
- }
+ release_rdma_sge_mr(e);
rkey = be32_to_cpu(reth->rkey);
qp->r_len = len;
/* Schedule the send tasklet. */
qp->s_flags |= RVT_S_RESP_PENDING;
+ if (fecn)
+ qp->s_flags |= RVT_S_ECN;
hfi1_schedule_send(qp);
spin_unlock_irqrestore(&qp->s_lock, flags);
- if (is_fecn)
- goto send_ack;
return;
nack_inv_unlock:
rvt_rc_error(qp, IB_WC_LOC_PROT_ERR);
qp->r_nak_state = IB_NAK_REMOTE_ACCESS_ERROR;
qp->r_ack_psn = qp->r_psn;
-send_ack:
- hfi1_send_rc_ack(packet, is_fecn);
}
u32 hfi1_build_tid_rdma_read_resp(struct rvt_qp *qp, struct rvt_ack_entry *e,
struct tid_rdma_request *req;
struct tid_rdma_flow *flow;
u32 opcode, aeth;
- bool is_fecn;
+ bool fecn;
unsigned long flags;
u32 kpsn, ipsn;
trace_hfi1_sender_rcv_tid_read_resp(qp);
- is_fecn = process_ecn(qp, packet);
+ fecn = process_ecn(qp, packet);
kpsn = mask_psn(be32_to_cpu(ohdr->bth[2]));
aeth = be32_to_cpu(ohdr->u.tid_rdma.r_rsp.aeth);
opcode = (be32_to_cpu(ohdr->bth[0]) >> 24) & 0xff;
flow = &req->flows[req->clear_tail];
/* When header suppression is disabled */
- if (cmp_psn(ipsn, flow->flow_state.ib_lpsn))
+ if (cmp_psn(ipsn, flow->flow_state.ib_lpsn)) {
+ update_r_next_psn_fecn(packet, priv, rcd, flow, fecn);
+
+ if (cmp_psn(kpsn, flow->flow_state.r_next_psn))
+ goto ack_done;
+ flow->flow_state.r_next_psn = mask_psn(kpsn + 1);
+ /*
+ * Copy the payload to destination buffer if this packet is
+ * delivered as an eager packet due to RSM rule and FECN.
+ * The RSM rule selects FECN bit in BTH and SH bit in
+ * KDETH header and therefore will not match the last
+ * packet of each segment that has SH bit cleared.
+ */
+ if (fecn && packet->etype == RHF_RCV_TYPE_EAGER) {
+ struct rvt_sge_state ss;
+ u32 len;
+ u32 tlen = packet->tlen;
+ u16 hdrsize = packet->hlen;
+ u8 pad = packet->pad;
+ u8 extra_bytes = pad + packet->extra_byte +
+ (SIZE_OF_CRC << 2);
+ u32 pmtu = qp->pmtu;
+
+ if (unlikely(tlen != (hdrsize + pmtu + extra_bytes)))
+ goto ack_op_err;
+ len = restart_sge(&ss, req->e.swqe, ipsn, pmtu);
+ if (unlikely(len < pmtu))
+ goto ack_op_err;
+ rvt_copy_sge(qp, &ss, packet->payload, pmtu, false,
+ false);
+ /* Raise the sw sequence check flag for next packet */
+ priv->s_flags |= HFI1_R_TID_SW_PSN;
+ }
+
goto ack_done;
+ }
+ flow->flow_state.r_next_psn = mask_psn(kpsn + 1);
req->ack_pending--;
priv->pending_tid_r_segs--;
qp->s_num_rd_atomic--;
req->comp_seg == req->cur_seg) ||
priv->tid_r_comp == priv->tid_r_reqs) {
hfi1_kern_clear_hw_flow(priv->rcd, qp);
+ priv->s_flags &= ~HFI1_R_TID_SW_PSN;
if (req->state == TID_REQUEST_SYNC)
req->state = TID_REQUEST_ACTIVE;
}
ack_done:
spin_unlock_irqrestore(&qp->s_lock, flags);
- if (is_fecn)
- hfi1_send_rc_ack(packet, is_fecn);
}
void hfi1_kern_read_tid_flow_free(struct rvt_qp *qp)
rvt_rc_error(qp, IB_WC_LOC_QP_OP_ERR);
return ret;
}
- if (priv->flow_state.flags & TID_FLOW_SW_PSN) {
+ if (priv->s_flags & HFI1_R_TID_SW_PSN) {
diff = cmp_psn(psn,
- priv->flow_state.r_next_psn);
+ flow->flow_state.r_next_psn);
if (diff > 0) {
if (!(qp->r_flags & RVT_R_RDMAR_SEQ))
restart_tid_rdma_read_req(rcd,
qp->r_flags &=
~RVT_R_RDMAR_SEQ;
}
- priv->flow_state.r_next_psn++;
+ flow->flow_state.r_next_psn =
+ mask_psn(psn + 1);
} else {
- u64 reg;
u32 last_psn;
- /*
- * The only sane way to get the amount of
- * progress is to read the HW flow state.
- */
- reg = read_uctxt_csr(dd, rcd->ctxt,
- RCV_TID_FLOW_TABLE +
- (8 * flow->idx));
- last_psn = mask_psn(reg);
-
- priv->flow_state.r_next_psn = last_psn;
- priv->flow_state.flags |= TID_FLOW_SW_PSN;
+ last_psn = read_r_next_psn(dd, rcd->ctxt,
+ flow->idx);
+ flow->flow_state.r_next_psn = last_psn;
+ priv->s_flags |= HFI1_R_TID_SW_PSN;
/*
* If no request has been restarted yet,
* restart the current one.
struct rvt_ack_entry *e;
struct tid_rdma_request *req;
struct tid_rdma_flow *flow;
+ int diff = 0;
trace_hfi1_msg_handle_kdeth_eflags(NULL, "Kdeth error: rhf ",
packet->rhf);
- if (packet->rhf & (RHF_VCRC_ERR | RHF_ICRC_ERR))
+ if (packet->rhf & RHF_ICRC_ERR)
return ret;
packet->ohdr = &hdr->u.oth;
switch (rte) {
case RHF_RTE_EXPECTED_FLOW_SEQ_ERR:
if (!(qpriv->s_flags & HFI1_R_TID_SW_PSN)) {
- u64 reg;
-
qpriv->s_flags |= HFI1_R_TID_SW_PSN;
- /*
- * The only sane way to get the amount of
- * progress is to read the HW flow state.
- */
- reg = read_uctxt_csr(dd, rcd->ctxt,
- RCV_TID_FLOW_TABLE +
- (8 * flow->idx));
- flow->flow_state.r_next_psn = mask_psn(reg);
+ flow->flow_state.r_next_psn =
+ read_r_next_psn(dd, rcd->ctxt,
+ flow->idx);
qpriv->r_next_psn_kdeth =
flow->flow_state.r_next_psn;
goto nak_psn;
* mismatch could be due to packets that were
* already in flight.
*/
- if (psn != flow->flow_state.r_next_psn) {
- psn = flow->flow_state.r_next_psn;
+ diff = cmp_psn(psn,
+ flow->flow_state.r_next_psn);
+ if (diff > 0)
goto nak_psn;
- }
+ else if (diff < 0)
+ break;
qpriv->s_nak_state = 0;
/*
if (psn == full_flow_psn(flow,
flow->flow_state.lpsn))
ret = false;
+ flow->flow_state.r_next_psn =
+ mask_psn(psn + 1);
qpriv->r_next_psn_kdeth =
- ++flow->flow_state.r_next_psn;
+ flow->flow_state.r_next_psn;
}
break;
if (qpriv->r_tid_alloc == qpriv->r_tid_head) {
/* If all data has been received, clear the flow */
if (qpriv->flow_state.index < RXE_NUM_TID_FLOWS &&
- !qpriv->alloc_w_segs)
+ !qpriv->alloc_w_segs) {
hfi1_kern_clear_hw_flow(rcd, qp);
+ qpriv->s_flags &= ~HFI1_R_TID_SW_PSN;
+ }
break;
}
if (qpriv->sync_pt && !qpriv->alloc_w_segs) {
hfi1_kern_clear_hw_flow(rcd, qp);
qpriv->sync_pt = false;
- if (qpriv->s_flags & HFI1_R_TID_SW_PSN)
- qpriv->s_flags &= ~HFI1_R_TID_SW_PSN;
+ qpriv->s_flags &= ~HFI1_R_TID_SW_PSN;
}
/* Allocate flow if we don't have one */
struct hfi1_qp_priv *qpriv = qp->priv;
struct tid_rdma_request *req;
u32 bth0, psn, len, rkey, num_segs;
- bool is_fecn;
+ bool fecn;
u8 next;
u64 vaddr;
int diff;
if (hfi1_ruc_check_hdr(ibp, packet))
return;
- is_fecn = process_ecn(qp, packet);
+ fecn = process_ecn(qp, packet);
psn = mask_psn(be32_to_cpu(ohdr->bth[2]));
trace_hfi1_rsp_rcv_tid_write_req(qp, psn);
num_segs = DIV_ROUND_UP(len, qpriv->tid_rdma.local.max_len);
diff = delta_psn(psn, qp->r_psn);
if (unlikely(diff)) {
- if (tid_rdma_rcv_error(packet, ohdr, qp, psn, diff))
- return;
- goto send_ack;
+ tid_rdma_rcv_err(packet, ohdr, qp, psn, diff, fecn);
+ return;
}
/*
goto update_head;
}
- if (e->rdma_sge.mr) {
- rvt_put_mr(e->rdma_sge.mr);
- e->rdma_sge.mr = NULL;
- }
+ release_rdma_sge_mr(e);
/* The length needs to be in multiples of PAGE_SIZE */
if (!len || len & ~PAGE_MASK)
/* Schedule the send tasklet. */
qp->s_flags |= RVT_S_RESP_PENDING;
+ if (fecn)
+ qp->s_flags |= RVT_S_ECN;
hfi1_schedule_send(qp);
spin_unlock_irqrestore(&qp->s_lock, flags);
- if (is_fecn)
- goto send_ack;
return;
nack_inv_unlock:
rvt_rc_error(qp, IB_WC_LOC_PROT_ERR);
qp->r_nak_state = IB_NAK_REMOTE_ACCESS_ERROR;
qp->r_ack_psn = qp->r_psn;
-send_ack:
- hfi1_send_rc_ack(packet, is_fecn);
}
u32 hfi1_build_tid_rdma_write_resp(struct rvt_qp *qp, struct rvt_ack_entry *e,
struct tid_rdma_flow *flow;
enum ib_wc_status status;
u32 opcode, aeth, psn, flow_psn, i, tidlen = 0, pktlen;
- bool is_fecn;
+ bool fecn;
unsigned long flags;
- is_fecn = process_ecn(qp, packet);
+ fecn = process_ecn(qp, packet);
psn = mask_psn(be32_to_cpu(ohdr->bth[2]));
aeth = be32_to_cpu(ohdr->u.tid_rdma.w_rsp.aeth);
opcode = (be32_to_cpu(ohdr->bth[0]) >> 24) & 0xff;
qpriv->s_tid_cur = i;
}
qp->s_flags &= ~HFI1_S_WAIT_TID_RESP;
-
hfi1_schedule_tid_send(qp);
goto ack_done;
ack_err:
rvt_error_qp(qp, status);
ack_done:
+ if (fecn)
+ qp->s_flags |= RVT_S_ECN;
spin_unlock_irqrestore(&qp->s_lock, flags);
- if (is_fecn)
- hfi1_send_rc_ack(packet, is_fecn);
}
bool hfi1_build_tid_rdma_packet(struct rvt_swqe *wqe,
unsigned long flags;
u32 psn, next;
u8 opcode;
+ bool fecn;
+ fecn = process_ecn(qp, packet);
psn = mask_psn(be32_to_cpu(ohdr->bth[2]));
opcode = (be32_to_cpu(ohdr->bth[0]) >> 24) & 0xff;
req = ack_to_tid_req(e);
flow = &req->flows[req->clear_tail];
if (cmp_psn(psn, full_flow_psn(flow, flow->flow_state.lpsn))) {
+ update_r_next_psn_fecn(packet, priv, rcd, flow, fecn);
+
if (cmp_psn(psn, flow->flow_state.r_next_psn))
goto send_nak;
- flow->flow_state.r_next_psn++;
+
+ flow->flow_state.r_next_psn = mask_psn(psn + 1);
+ /*
+ * Copy the payload to destination buffer if this packet is
+ * delivered as an eager packet due to RSM rule and FECN.
+ * The RSM rule selects FECN bit in BTH and SH bit in
+ * KDETH header and therefore will not match the last
+ * packet of each segment that has SH bit cleared.
+ */
+ if (fecn && packet->etype == RHF_RCV_TYPE_EAGER) {
+ struct rvt_sge_state ss;
+ u32 len;
+ u32 tlen = packet->tlen;
+ u16 hdrsize = packet->hlen;
+ u8 pad = packet->pad;
+ u8 extra_bytes = pad + packet->extra_byte +
+ (SIZE_OF_CRC << 2);
+ u32 pmtu = qp->pmtu;
+
+ if (unlikely(tlen != (hdrsize + pmtu + extra_bytes)))
+ goto send_nak;
+ len = req->comp_seg * req->seg_len;
+ len += delta_psn(psn,
+ full_flow_psn(flow, flow->flow_state.spsn)) *
+ pmtu;
+ if (unlikely(req->total_len - len < pmtu))
+ goto send_nak;
+
+ /*
+ * The e->rdma_sge field is set when TID RDMA WRITE REQ
+ * is first received and is never modified thereafter.
+ */
+ ss.sge = e->rdma_sge;
+ ss.sg_list = NULL;
+ ss.num_sge = 1;
+ ss.total_len = req->total_len;
+ rvt_skip_sge(&ss, len, false);
+ rvt_copy_sge(qp, &ss, packet->payload, pmtu, false,
+ false);
+ /* Raise the sw sequence check flag for next packet */
+ priv->r_next_psn_kdeth = mask_psn(psn + 1);
+ priv->s_flags |= HFI1_R_TID_SW_PSN;
+ }
goto exit;
}
flow->flow_state.r_next_psn = mask_psn(psn + 1);
priv->r_tid_ack = priv->r_tid_tail;
if (opcode == TID_OP(WRITE_DATA_LAST)) {
+ release_rdma_sge_mr(e);
for (next = priv->r_tid_tail + 1; ; next++) {
if (next > rvt_size_atomic(&dev->rdi))
next = 0;
hfi1_schedule_tid_send(qp);
exit:
priv->r_next_psn_kdeth = flow->flow_state.r_next_psn;
+ if (fecn)
+ qp->s_flags |= RVT_S_ECN;
spin_unlock_irqrestore(&qp->s_lock, flags);
return;
struct tid_rdma_request *req;
struct tid_rdma_flow *flow;
u32 aeth, psn, req_psn, ack_psn, fspsn, resync_psn, ack_kpsn;
- bool is_fecn;
unsigned long flags;
u16 fidx;
trace_hfi1_tid_write_sender_rcv_tid_ack(qp, 0);
- is_fecn = process_ecn(qp, packet);
+ process_ecn(qp, packet);
psn = mask_psn(be32_to_cpu(ohdr->bth[2]));
aeth = be32_to_cpu(ohdr->u.tid_rdma.ack.aeth);
req_psn = mask_psn(be32_to_cpu(ohdr->u.tid_rdma.ack.verbs_psn));
struct tid_rdma_flow *flow;
struct tid_flow_state *fs = &qpriv->flow_state;
u32 psn, generation, idx, gen_next;
- bool is_fecn;
+ bool fecn;
unsigned long flags;
- is_fecn = process_ecn(qp, packet);
+ fecn = process_ecn(qp, packet);
psn = mask_psn(be32_to_cpu(ohdr->bth[2]));
generation = mask_psn(psn + 1) >> HFI1_KDETH_BTH_SEQ_SHIFT;
qpriv->s_flags |= RVT_S_ACK_PENDING;
hfi1_schedule_tid_send(qp);
bail:
+ if (fecn)
+ qp->s_flags |= RVT_S_ECN;
spin_unlock_irqrestore(&qp->s_lock, flags);
}
}
return false;
}
+
+static u32 read_r_next_psn(struct hfi1_devdata *dd, u8 ctxt, u8 fidx)
+{
+ u64 reg;
+
+ /*
+ * The only sane way to get the amount of
+ * progress is to read the HW flow state.
+ */
+ reg = read_uctxt_csr(dd, ctxt, RCV_TID_FLOW_TABLE + (8 * fidx));
+ return mask_psn(reg);
+}
+
+static void tid_rdma_rcv_err(struct hfi1_packet *packet,
+ struct ib_other_headers *ohdr,
+ struct rvt_qp *qp, u32 psn, int diff, bool fecn)
+{
+ unsigned long flags;
+
+ tid_rdma_rcv_error(packet, ohdr, qp, psn, diff);
+ if (fecn) {
+ spin_lock_irqsave(&qp->s_lock, flags);
+ qp->s_flags |= RVT_S_ECN;
+ spin_unlock_irqrestore(&qp->s_lock, flags);
+ }
+}
+
+static void update_r_next_psn_fecn(struct hfi1_packet *packet,
+ struct hfi1_qp_priv *priv,
+ struct hfi1_ctxtdata *rcd,
+ struct tid_rdma_flow *flow,
+ bool fecn)
+{
+ /*
+ * If a start/middle packet is delivered here due to
+ * RSM rule and FECN, we need to update the r_next_psn.
+ */
+ if (fecn && packet->etype == RHF_RCV_TYPE_EAGER &&
+ !(priv->s_flags & HFI1_R_TID_SW_PSN)) {
+ struct hfi1_devdata *dd = rcd->dd;
+
+ flow->flow_state.r_next_psn =
+ read_r_next_psn(dd, rcd->ctxt, flow->idx);
+ }
+}
struct tid_flow_state {
u32 generation;
u32 psn;
- u32 r_next_psn; /* next PSN to be received (in TID space) */
u8 index;
u8 last_index;
- u8 flags;
};
enum tid_rdma_req_state {
* actual function to work and can not be in a macro.
*/
#define __hfi1_trace_def(lvl) \
-void __hfi1_trace_##lvl(const char *funct, char *fmt, ...); \
+void __printf(2, 3) __hfi1_trace_##lvl(const char *funct, char *fmt, ...); \
\
DEFINE_EVENT(hfi1_trace_template, hfi1_ ##lvl, \
TP_PROTO(const char *function, struct va_format *vaf), \
TP_ARGS(function, vaf))
#define __hfi1_trace_fn(lvl) \
-void __hfi1_trace_##lvl(const char *func, char *fmt, ...) \
+void __printf(2, 3) __hfi1_trace_##lvl(const char *func, char *fmt, ...)\
{ \
struct va_format vaf = { \
.fmt = fmt, \
"tid_r_comp %u pending_tid_r_segs %u " \
"s_flags 0x%x ps_flags 0x%x iow_flags 0x%lx " \
"s_state 0x%x hw_flow_index %u generation 0x%x " \
- "fpsn 0x%x flow_flags 0x%x"
+ "fpsn 0x%x"
#define TID_REQ_PRN "[%s] qpn 0x%x newreq %u opcode 0x%x psn 0x%x lpsn 0x%x " \
"cur_seg %u comp_seg %u ack_seg %u alloc_seg %u " \
"pending_tid_w_segs %u sync_pt %s " \
"ps_nak_psn 0x%x ps_nak_state 0x%x " \
"prnr_nak_state 0x%x hw_flow_index %u generation "\
- "0x%x fpsn 0x%x flow_flags 0x%x resync %s" \
+ "0x%x fpsn 0x%x resync %s" \
"r_next_psn_kdeth 0x%x"
#define TID_WRITE_SENDER_PRN "[%s] qpn 0x%x newreq %u s_tid_cur %u " \
__field(u32, hw_flow_index)
__field(u32, generation)
__field(u32, fpsn)
- __field(u32, flow_flags)
),
TP_fast_assign(/* assign */
struct hfi1_qp_priv *priv = qp->priv;
__entry->hw_flow_index = priv->flow_state.index;
__entry->generation = priv->flow_state.generation;
__entry->fpsn = priv->flow_state.psn;
- __entry->flow_flags = priv->flow_state.flags;
),
TP_printk(/* print */
TID_READ_SENDER_PRN,
__entry->s_state,
__entry->hw_flow_index,
__entry->generation,
- __entry->fpsn,
- __entry->flow_flags
+ __entry->fpsn
)
);
__field(u32, hw_flow_index)
__field(u32, generation)
__field(u32, fpsn)
- __field(u32, flow_flags)
__field(bool, resync)
__field(u32, r_next_psn_kdeth)
),
__entry->hw_flow_index = priv->flow_state.index;
__entry->generation = priv->flow_state.generation;
__entry->fpsn = priv->flow_state.psn;
- __entry->flow_flags = priv->flow_state.flags;
__entry->resync = priv->resync;
__entry->r_next_psn_kdeth = priv->r_next_psn_kdeth;
),
__entry->hw_flow_index,
__entry->generation,
__entry->fpsn,
- __entry->flow_flags,
__entry->resync ? "yes" : "no",
__entry->r_next_psn_kdeth
)
case IB_QPT_UD:
break;
case IB_QPT_UC:
- case IB_QPT_RC: {
+ case IB_QPT_RC:
+ priv->s_running_pkt_size =
+ (tx->s_cur_size + priv->s_running_pkt_size) / 2;
if (piothreshold &&
- tx->s_cur_size <= min(piothreshold, qp->pmtu) &&
+ priv->s_running_pkt_size <= min(piothreshold, qp->pmtu) &&
(BIT(ps->opcode & OPMASK) & pio_opmask[ps->opcode >> 5]) &&
iowait_sdma_pending(&priv->s_iowait) == 0 &&
!sdma_txreq_built(&tx->txreq))
return dd->process_pio_send;
break;
- }
default:
break;
}
static u64 hfi1_sps_ints(void)
{
- unsigned long flags;
+ unsigned long index, flags;
struct hfi1_devdata *dd;
u64 sps_ints = 0;
- spin_lock_irqsave(&hfi1_devs_lock, flags);
- list_for_each_entry(dd, &hfi1_dev_list, list) {
+ xa_lock_irqsave(&hfi1_dev_table, flags);
+ xa_for_each(&hfi1_dev_table, index, dd) {
sps_ints += get_all_cpu_total(dd->int_counter);
}
- spin_unlock_irqrestore(&hfi1_devs_lock, flags);
+ xa_unlock_irqrestore(&hfi1_dev_table, flags);
return sps_ints;
}
struct tid_flow_state flow_state;
struct tid_rdma_qp_params tid_rdma;
struct rvt_qp *owner;
+ u16 s_running_pkt_size;
u8 hdr_type; /* 9B or 16B */
struct rvt_sge_state tid_ss; /* SGE state pointer for 2nd leg */
atomic_t n_requests; /* # of TID RDMA requests in the */
void hfi1_vnic_setup(struct hfi1_devdata *dd)
{
- idr_init(&dd->vnic.vesw_idr);
+ xa_init(&dd->vnic.vesws);
}
void hfi1_vnic_cleanup(struct hfi1_devdata *dd)
{
- idr_destroy(&dd->vnic.vesw_idr);
+ WARN_ON(!xa_empty(&dd->vnic.vesws));
}
#define SUM_GRP_COUNTERS(stats, qstats, x_grp) do { \
l4_type = hfi1_16B_get_l4(packet->ebuf);
if (likely(l4_type == OPA_16B_L4_ETHR)) {
vesw_id = HFI1_VNIC_GET_VESWID(packet->ebuf);
- vinfo = idr_find(&dd->vnic.vesw_idr, vesw_id);
+ vinfo = xa_load(&dd->vnic.vesws, vesw_id);
/*
* In case of invalid vesw id, count the error on
*/
if (unlikely(!vinfo)) {
struct hfi1_vnic_vport_info *vinfo_tmp;
- int id_tmp = 0;
+ unsigned long index = 0;
- vinfo_tmp = idr_get_next(&dd->vnic.vesw_idr, &id_tmp);
+ vinfo_tmp = xa_find(&dd->vnic.vesws, &index, ULONG_MAX,
+ XA_PRESENT);
if (vinfo_tmp) {
spin_lock(&vport_cntr_lock);
vinfo_tmp->stats[0].netstats.rx_nohandler++;
if (!vinfo->vesw_id)
return -EINVAL;
- rc = idr_alloc(&dd->vnic.vesw_idr, vinfo, vinfo->vesw_id,
- vinfo->vesw_id + 1, GFP_NOWAIT);
+ rc = xa_insert(&dd->vnic.vesws, vinfo->vesw_id, vinfo, GFP_KERNEL);
if (rc < 0)
return rc;
clear_bit(HFI1_VNIC_UP, &vinfo->flags);
netif_carrier_off(vinfo->netdev);
netif_tx_disable(vinfo->netdev);
- idr_remove(&dd->vnic.vesw_idr, vinfo->vesw_id);
+ xa_erase(&dd->vnic.vesws, vinfo->vesw_id);
/* ensure irqs see the change */
msix_vnic_synchronize_irq(dd);
obj-$(CONFIG_INFINIBAND_HNS) += hns-roce.o
hns-roce-objs := hns_roce_main.o hns_roce_cmd.o hns_roce_pd.o \
hns_roce_ah.o hns_roce_hem.o hns_roce_mr.o hns_roce_qp.o \
- hns_roce_cq.o hns_roce_alloc.o hns_roce_db.o hns_roce_srq.o
+ hns_roce_cq.o hns_roce_alloc.o hns_roce_db.o hns_roce_srq.o hns_roce_restrack.o
obj-$(CONFIG_INFINIBAND_HNS_HIP06) += hns-roce-hw-v1.o
hns-roce-hw-v1-objs := hns_roce_hw_v1.o
obj-$(CONFIG_INFINIBAND_HNS_HIP08) += hns-roce-hw-v2.o
-hns-roce-hw-v2-objs := hns_roce_hw_v2.o
+hns-roce-hw-v2-objs := hns_roce_hw_v2.o hns_roce_hw_v2_dfx.o
#define HNS_ROCE_VLAN_SL_BIT_MASK 7
#define HNS_ROCE_VLAN_SL_SHIFT 13
-struct ib_ah *hns_roce_create_ah(struct ib_pd *ibpd,
- struct rdma_ah_attr *ah_attr,
- u32 flags,
- struct ib_udata *udata)
+int hns_roce_create_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr,
+ u32 flags, struct ib_udata *udata)
{
- struct hns_roce_dev *hr_dev = to_hr_dev(ibpd->device);
+ struct hns_roce_dev *hr_dev = to_hr_dev(ibah->device);
const struct ib_gid_attr *gid_attr;
struct device *dev = hr_dev->dev;
- struct hns_roce_ah *ah;
+ struct hns_roce_ah *ah = to_hr_ah(ibah);
u16 vlan_tag = 0xffff;
const struct ib_global_route *grh = rdma_ah_read_grh(ah_attr);
bool vlan_en = false;
+ int ret;
- ah = kzalloc(sizeof(*ah), GFP_ATOMIC);
- if (!ah)
- return ERR_PTR(-ENOMEM);
+ gid_attr = ah_attr->grh.sgid_attr;
+ ret = rdma_read_gid_l2_fields(gid_attr, &vlan_tag, NULL);
+ if (ret)
+ return ret;
/* Get mac address */
memcpy(ah->av.mac, ah_attr->roce.dmac, ETH_ALEN);
- gid_attr = ah_attr->grh.sgid_attr;
- if (is_vlan_dev(gid_attr->ndev)) {
- vlan_tag = vlan_dev_vlan_id(gid_attr->ndev);
+ if (vlan_tag < VLAN_CFI_MASK) {
vlan_en = true;
- }
-
- if (vlan_tag < 0x1000)
vlan_tag |= (rdma_ah_get_sl(ah_attr) &
HNS_ROCE_VLAN_SL_BIT_MASK) <<
HNS_ROCE_VLAN_SL_SHIFT;
+ }
- ah->av.port_pd = cpu_to_be32(to_hr_pd(ibpd)->pdn |
+ ah->av.port_pd = cpu_to_le32(to_hr_pd(ibah->pd)->pdn |
(rdma_ah_get_port_num(ah_attr) <<
HNS_ROCE_PORT_NUM_SHIFT));
ah->av.gid_index = grh->sgid_index;
ah->av.sl_tclass_flowlabel = cpu_to_le32(rdma_ah_get_sl(ah_attr) <<
HNS_ROCE_SL_SHIFT);
- return &ah->ibah;
+ return 0;
}
int hns_roce_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr)
return 0;
}
-int hns_roce_destroy_ah(struct ib_ah *ah, u32 flags)
+void hns_roce_destroy_ah(struct ib_ah *ah, u32 flags)
{
- kfree(to_hr_ah(ah));
-
- return 0;
+ return;
}
HNS_ROCE_CMD_QUERY_QPC = 0x42,
HNS_ROCE_CMD_MODIFY_CQC = 0x52,
+ HNS_ROCE_CMD_QUERY_CQC = 0x53,
/* CQC BT commands */
HNS_ROCE_CMD_WRITE_CQC_BT0 = 0x10,
HNS_ROCE_CMD_WRITE_CQC_BT1 = 0x11,
#define roce_set_bit(origin, shift, val) \
roce_set_field((origin), (1ul << (shift)), (shift), (val))
-/*
- * roce_hw_index_cmp_lt - Compare two hardware index values in hisilicon
- * SOC, check if a is less than b.
- * @a: hardware index value
- * @b: hardware index value
- * @bits: the number of bits of a and b, range: 0~31.
- *
- * Hardware index increases continuously till max value, and then restart
- * from zero, again and again. Because the bits of reg field is often
- * limited, the reg field can only hold the low bits of the hardware index
- * in hisilicon SOC.
- * In some scenes we need to compare two values(a,b) getted from two reg
- * fields in this driver, for example:
- * If a equals 0xfffe, b equals 0x1 and bits equals 16, we think b has
- * incresed from 0xffff to 0x1 and a is less than b.
- * If a equals 0xfffe, b equals 0x0xf001 and bits equals 16, we think a
- * is bigger than b.
- *
- * Return true on a less than b, otherwise false.
- */
-#define roce_hw_index_mask(bits) ((1ul << (bits)) - 1)
-#define roce_hw_index_shift(bits) (32 - (bits))
-#define roce_hw_index_cmp_lt(a, b, bits) \
- ((int)((((a) - (b)) & roce_hw_index_mask(bits)) << \
- roce_hw_index_shift(bits)) < 0)
-
#define ROCEE_GLB_CFG_ROCEE_DB_SQ_MODE_S 3
#define ROCEE_GLB_CFG_ROCEE_DB_OTH_MODE_S 4
#define ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M \
(((1UL << 28) - 1) << ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S)
-#define ROCEE_SDB_PTR_CMP_BITS 28
-
#define ROCEE_SDB_INV_CNT_SDB_INV_CNT_S 0
#define ROCEE_SDB_INV_CNT_SDB_INV_CNT_M \
(((1UL << 16) - 1) << ROCEE_SDB_INV_CNT_SDB_INV_CNT_S)
#define ROCEE_CAEP_AE_MASK_REG 0x6C8
#define ROCEE_CAEP_AE_ST_REG 0x6CC
-#define ROCEE_SDB_ISSUE_PTR_REG 0x758
-#define ROCEE_SDB_SEND_PTR_REG 0x75C
#define ROCEE_CAEP_CQE_WCMD_EMPTY 0x850
#define ROCEE_SCAEP_WR_CQE_CNT 0x8D0
-#define ROCEE_SDB_INV_CNT_REG 0x9A4
-#define ROCEE_SDB_RETRY_CNT_REG 0x9AC
-#define ROCEE_TSP_BP_ST_REG 0x9EC
#define ROCEE_ECC_UCERR_ALM0_REG 0xB34
#define ROCEE_ECC_CERR_ALM0_REG 0xB40
#include <linux/platform_device.h>
#include <rdma/ib_umem.h>
+#include <rdma/uverbs_ioctl.h>
#include "hns_roce_device.h"
#include "hns_roce_cmd.h"
#include "hns_roce_hem.h"
goto err_out;
}
- /* The cq insert radix tree */
- spin_lock_irq(&cq_table->lock);
- /* Radix_tree: The associated pointer and long integer key value like */
- ret = radix_tree_insert(&cq_table->tree, hr_cq->cqn, hr_cq);
- spin_unlock_irq(&cq_table->lock);
+ ret = xa_err(xa_store(&cq_table->array, hr_cq->cqn, hr_cq, GFP_KERNEL));
if (ret) {
- dev_err(dev, "CQ alloc.Failed to radix_tree_insert.\n");
+ dev_err(dev, "CQ alloc failed xa_store.\n");
goto err_put;
}
mailbox = hns_roce_alloc_cmd_mailbox(hr_dev);
if (IS_ERR(mailbox)) {
ret = PTR_ERR(mailbox);
- goto err_radix;
+ goto err_xa;
}
hr_dev->hw->write_cqc(hr_dev, hr_cq, mailbox->buf, mtts, dma_handle,
hns_roce_free_cmd_mailbox(hr_dev, mailbox);
if (ret) {
dev_err(dev, "CQ alloc.Failed to cmd mailbox.\n");
- goto err_radix;
+ goto err_xa;
}
hr_cq->cons_index = 0;
return 0;
-err_radix:
- spin_lock_irq(&cq_table->lock);
- radix_tree_delete(&cq_table->tree, hr_cq->cqn);
- spin_unlock_irq(&cq_table->lock);
+err_xa:
+ xa_erase(&cq_table->array, hr_cq->cqn);
err_put:
hns_roce_table_put(hr_dev, &cq_table->table, hr_cq->cqn);
dev_err(dev, "HW2SW_CQ failed (%d) for CQN %06lx\n", ret,
hr_cq->cqn);
+ xa_erase(&cq_table->array, hr_cq->cqn);
+
/* Waiting interrupt process procedure carried out */
synchronize_irq(hr_dev->eq_table.eq[hr_cq->vector].irq);
complete(&hr_cq->free);
wait_for_completion(&hr_cq->free);
- spin_lock_irq(&cq_table->lock);
- radix_tree_delete(&cq_table->tree, hr_cq->cqn);
- spin_unlock_irq(&cq_table->lock);
-
hns_roce_table_put(hr_dev, &cq_table->table, hr_cq->cqn);
hns_roce_bitmap_free(&cq_table->bitmap, hr_cq->cqn, BITMAP_NO_RR);
}
struct ib_cq *hns_roce_ib_create_cq(struct ib_device *ib_dev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
struct hns_roce_dev *hr_dev = to_hr_dev(ib_dev);
int vector = attr->comp_vector;
int cq_entries = attr->cqe;
int ret;
+ struct hns_roce_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct hns_roce_ucontext, ibucontext);
if (cq_entries < 1 || cq_entries > hr_dev->caps.max_cqes) {
dev_err(dev, "Creat CQ failed. entries=%d, max=%d\n",
hr_cq->ib_cq.cqe = cq_entries - 1;
spin_lock_init(&hr_cq->lock);
- if (context) {
+ if (udata) {
if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd))) {
dev_err(dev, "Failed to copy_from_udata.\n");
ret = -EFAULT;
if ((hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_RECORD_DB) &&
(udata->outlen >= sizeof(resp))) {
- ret = hns_roce_db_map_user(to_hr_ucontext(context),
- udata, ucmd.db_addr,
+ ret = hns_roce_db_map_user(context, udata, ucmd.db_addr,
&hr_cq->db);
if (ret) {
dev_err(dev, "cq record doorbell map failed!\n");
}
/* Get user space parameters */
- uar = &to_hr_ucontext(context)->uar;
+ uar = &context->uar;
} else {
if (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_RECORD_DB) {
ret = hns_roce_alloc_db(hr_dev, &hr_cq->db, 1);
* problems if tptr is set to zero here, so we initialze it in user
* space.
*/
- if (!context && hr_cq->tptr_addr)
+ if (!udata && hr_cq->tptr_addr)
*hr_cq->tptr_addr = 0;
/* Get created cq handler and carry out event */
hr_cq->event = hns_roce_ib_cq_event;
hr_cq->cq_depth = cq_entries;
- if (context) {
+ if (udata) {
resp.cqn = hr_cq->cqn;
ret = ib_copy_to_udata(udata, &resp, sizeof(resp));
if (ret)
hns_roce_free_cq(hr_dev, hr_cq);
err_dbmap:
- if (context && (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_RECORD_DB) &&
+ if (udata && (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_RECORD_DB) &&
(udata->outlen >= sizeof(resp)))
- hns_roce_db_unmap_user(to_hr_ucontext(context),
- &hr_cq->db);
+ hns_roce_db_unmap_user(context, &hr_cq->db);
err_mtt:
hns_roce_mtt_cleanup(hr_dev, &hr_cq->hr_buf.hr_mtt);
- if (context)
+ if (udata)
ib_umem_release(hr_cq->umem);
else
hns_roce_ib_free_cq_buf(hr_dev, &hr_cq->hr_buf,
hr_cq->ib_cq.cqe);
err_db:
- if (!context && (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_RECORD_DB))
+ if (!udata && (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_RECORD_DB))
hns_roce_free_db(hr_dev, &hr_cq->db);
err_cq:
}
EXPORT_SYMBOL_GPL(hns_roce_ib_create_cq);
-int hns_roce_ib_destroy_cq(struct ib_cq *ib_cq)
+int hns_roce_ib_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
{
struct hns_roce_dev *hr_dev = to_hr_dev(ib_cq->device);
struct hns_roce_cq *hr_cq = to_hr_cq(ib_cq);
int ret = 0;
if (hr_dev->hw->destroy_cq) {
- ret = hr_dev->hw->destroy_cq(ib_cq);
+ ret = hr_dev->hw->destroy_cq(ib_cq, udata);
} else {
hns_roce_free_cq(hr_dev, hr_cq);
hns_roce_mtt_cleanup(hr_dev, &hr_cq->hr_buf.hr_mtt);
- if (ib_cq->uobject) {
+ if (udata) {
ib_umem_release(hr_cq->umem);
if (hr_cq->db_en == 1)
hns_roce_db_unmap_user(
- to_hr_ucontext(ib_cq->uobject->context),
+ rdma_udata_to_drv_context(
+ udata,
+ struct hns_roce_ucontext,
+ ibucontext),
&hr_cq->db);
} else {
/* Free the buff of stored cq */
struct device *dev = hr_dev->dev;
struct hns_roce_cq *cq;
- cq = radix_tree_lookup(&hr_dev->cq_table.tree,
- cqn & (hr_dev->caps.num_cqs - 1));
+ cq = xa_load(&hr_dev->cq_table.array, cqn & (hr_dev->caps.num_cqs - 1));
if (!cq) {
dev_warn(dev, "Completion event for bogus CQ 0x%08x\n", cqn);
return;
struct device *dev = hr_dev->dev;
struct hns_roce_cq *cq;
- cq = radix_tree_lookup(&cq_table->tree,
- cqn & (hr_dev->caps.num_cqs - 1));
+ cq = xa_load(&cq_table->array, cqn & (hr_dev->caps.num_cqs - 1));
if (cq)
atomic_inc(&cq->refcount);
{
struct hns_roce_cq_table *cq_table = &hr_dev->cq_table;
- spin_lock_init(&cq_table->lock);
- INIT_RADIX_TREE(&cq_table->tree, GFP_ATOMIC);
+ xa_init(&cq_table->array);
return hns_roce_bitmap_init(&cq_table->bitmap, hr_dev->caps.num_cqs,
hr_dev->caps.num_cqs - 1,
struct hns_roce_qp_table {
struct hns_roce_bitmap bitmap;
- spinlock_t lock;
struct hns_roce_hem_table qp_table;
struct hns_roce_hem_table irrl_table;
struct hns_roce_hem_table trrl_table;
struct hns_roce_cq_table {
struct hns_roce_bitmap bitmap;
- spinlock_t lock;
- struct radix_tree_root tree;
+ struct xarray array;
struct hns_roce_hem_table table;
};
int sub_type;
};
+struct hns_roce_dfx_hw {
+ int (*query_cqc_info)(struct hns_roce_dev *hr_dev, u32 cqn,
+ int *buffer);
+};
+
struct hns_roce_hw {
int (*reset)(struct hns_roce_dev *hr_dev, bool enable);
int (*cmq_init)(struct hns_roce_dev *hr_dev);
int (*modify_qp)(struct ib_qp *ibqp, const struct ib_qp_attr *attr,
int attr_mask, enum ib_qp_state cur_state,
enum ib_qp_state new_state);
- int (*destroy_qp)(struct ib_qp *ibqp);
+ int (*destroy_qp)(struct ib_qp *ibqp, struct ib_udata *udata);
int (*qp_flow_control_init)(struct hns_roce_dev *hr_dev,
struct hns_roce_qp *hr_qp);
int (*post_send)(struct ib_qp *ibqp, const struct ib_send_wr *wr,
const struct ib_recv_wr **bad_recv_wr);
int (*req_notify_cq)(struct ib_cq *ibcq, enum ib_cq_notify_flags flags);
int (*poll_cq)(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc);
- int (*dereg_mr)(struct hns_roce_dev *hr_dev, struct hns_roce_mr *mr);
- int (*destroy_cq)(struct ib_cq *ibcq);
+ int (*dereg_mr)(struct hns_roce_dev *hr_dev, struct hns_roce_mr *mr,
+ struct ib_udata *udata);
+ int (*destroy_cq)(struct ib_cq *ibcq, struct ib_udata *udata);
int (*modify_cq)(struct ib_cq *cq, u16 cq_count, u16 cq_period);
int (*init_eq)(struct hns_roce_dev *hr_dev);
void (*cleanup_eq)(struct hns_roce_dev *hr_dev);
int irq[HNS_ROCE_MAX_IRQ_NUM];
u8 __iomem *reg_base;
struct hns_roce_caps caps;
- struct radix_tree_root qp_table_tree;
+ struct xarray qp_table_xa;
unsigned char dev_addr[HNS_ROCE_MAX_PORTS][MAC_ADDR_OCTET_NUM];
u64 sys_image_guid;
const struct hns_roce_hw *hw;
void *priv;
struct workqueue_struct *irq_workq;
+ const struct hns_roce_dfx_hw *dfx;
};
static inline struct hns_roce_dev *to_hr_dev(struct ib_device *ib_dev)
static inline struct hns_roce_qp
*__hns_roce_qp_lookup(struct hns_roce_dev *hr_dev, u32 qpn)
{
- return radix_tree_lookup(&hr_dev->qp_table_tree,
- qpn & (hr_dev->caps.num_qps - 1));
+ return xa_load(&hr_dev->qp_table_xa, qpn & (hr_dev->caps.num_qps - 1));
}
static inline void *hns_roce_buf_offset(struct hns_roce_buf *buf, int offset)
unsigned long obj, int cnt,
int rr);
-struct ib_ah *hns_roce_create_ah(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
- u32 flags,
- struct ib_udata *udata);
+int hns_roce_create_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr,
+ u32 flags, struct ib_udata *udata);
int hns_roce_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr);
-int hns_roce_destroy_ah(struct ib_ah *ah, u32 flags);
+void hns_roce_destroy_ah(struct ib_ah *ah, u32 flags);
-int hns_roce_alloc_pd(struct ib_pd *pd, struct ib_ucontext *context,
- struct ib_udata *udata);
-void hns_roce_dealloc_pd(struct ib_pd *pd);
+int hns_roce_alloc_pd(struct ib_pd *pd, struct ib_udata *udata);
+void hns_roce_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata);
struct ib_mr *hns_roce_get_dma_mr(struct ib_pd *pd, int acc);
struct ib_mr *hns_roce_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
u64 virt_addr, int mr_access_flags, struct ib_pd *pd,
struct ib_udata *udata);
struct ib_mr *hns_roce_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
- u32 max_num_sg);
+ u32 max_num_sg, struct ib_udata *udata);
int hns_roce_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
unsigned int *sg_offset);
-int hns_roce_dereg_mr(struct ib_mr *ibmr);
+int hns_roce_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata);
int hns_roce_hw2sw_mpt(struct hns_roce_dev *hr_dev,
struct hns_roce_cmd_mailbox *mailbox,
unsigned long mpt_index);
int hns_roce_ib_umem_write_mtt(struct hns_roce_dev *hr_dev,
struct hns_roce_mtt *mtt, struct ib_umem *umem);
-struct ib_srq *hns_roce_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata);
+int hns_roce_create_srq(struct ib_srq *srq,
+ struct ib_srq_init_attr *srq_init_attr,
+ struct ib_udata *udata);
int hns_roce_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *srq_attr,
enum ib_srq_attr_mask srq_attr_mask,
struct ib_udata *udata);
-int hns_roce_destroy_srq(struct ib_srq *ibsrq);
+void hns_roce_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata);
struct ib_qp *hns_roce_create_qp(struct ib_pd *ib_pd,
struct ib_qp_init_attr *init_attr,
struct ib_cq *hns_roce_ib_create_cq(struct ib_device *ib_dev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata);
-int hns_roce_ib_destroy_cq(struct ib_cq *ib_cq);
+int hns_roce_ib_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata);
void hns_roce_free_cq(struct hns_roce_dev *hr_dev, struct hns_roce_cq *hr_cq);
int hns_roce_db_map_user(struct hns_roce_ucontext *context,
int hns_roce_init(struct hns_roce_dev *hr_dev);
void hns_roce_exit(struct hns_roce_dev *hr_dev);
+int hns_roce_fill_res_entry(struct sk_buff *msg,
+ struct rdma_restrack_entry *res);
#endif /* _HNS_ROCE_DEVICE_H */
/* Reserved cq for loop qp */
cq_init_attr.cqe = HNS_ROCE_MIN_WQE_NUM * 2;
cq_init_attr.comp_vector = 0;
- cq = hns_roce_ib_create_cq(&hr_dev->ib_dev, &cq_init_attr, NULL, NULL);
+ cq = hns_roce_ib_create_cq(&hr_dev->ib_dev, &cq_init_attr, NULL);
if (IS_ERR(cq)) {
dev_err(dev, "Create cq for reserved loop qp failed!");
return -ENOMEM;
goto alloc_mem_failed;
pd->device = ibdev;
- ret = hns_roce_alloc_pd(pd, NULL, NULL);
+ ret = hns_roce_alloc_pd(pd, NULL);
if (ret)
goto alloc_pd_failed;
create_lp_qp_failed:
for (i -= 1; i >= 0; i--) {
hr_qp = free_mr->mr_free_qp[i];
- if (hns_roce_v1_destroy_qp(&hr_qp->ibqp))
+ if (hns_roce_v1_destroy_qp(&hr_qp->ibqp, NULL))
dev_err(dev, "Destroy qp %d for mr free failed!\n", i);
}
- hns_roce_dealloc_pd(pd);
+ hns_roce_dealloc_pd(pd, NULL);
alloc_pd_failed:
kfree(pd);
alloc_mem_failed:
- if (hns_roce_ib_destroy_cq(cq))
+ if (hns_roce_ib_destroy_cq(cq, NULL))
dev_err(dev, "Destroy cq for create_lp_qp failed!\n");
return ret;
if (!hr_qp)
continue;
- ret = hns_roce_v1_destroy_qp(&hr_qp->ibqp);
+ ret = hns_roce_v1_destroy_qp(&hr_qp->ibqp, NULL);
if (ret)
dev_err(dev, "Destroy qp %d for mr free failed(%d)!\n",
i, ret);
}
- ret = hns_roce_ib_destroy_cq(&free_mr->mr_free_cq->ib_cq);
+ ret = hns_roce_ib_destroy_cq(&free_mr->mr_free_cq->ib_cq, NULL);
if (ret)
dev_err(dev, "Destroy cq for mr_free failed(%d)!\n", ret);
- hns_roce_dealloc_pd(&free_mr->mr_free_pd->ibpd);
+ hns_roce_dealloc_pd(&free_mr->mr_free_pd->ibpd, NULL);
}
static int hns_roce_db_init(struct hns_roce_dev *hr_dev)
}
static int hns_roce_v1_dereg_mr(struct hns_roce_dev *hr_dev,
- struct hns_roce_mr *mr)
+ struct hns_roce_mr *mr, struct ib_udata *udata)
{
struct device *dev = &hr_dev->pdev->dev;
struct hns_roce_mr_free_work *mr_work;
return ret;
}
-static int hns_roce_des_qp_init(struct hns_roce_dev *hr_dev)
-{
- struct device *dev = &hr_dev->pdev->dev;
- struct hns_roce_v1_priv *priv;
- struct hns_roce_des_qp *des_qp;
-
- priv = (struct hns_roce_v1_priv *)hr_dev->priv;
- des_qp = &priv->des_qp;
-
- des_qp->requeue_flag = 1;
- des_qp->qp_wq = create_singlethread_workqueue("hns_roce_destroy_qp");
- if (!des_qp->qp_wq) {
- dev_err(dev, "Create destroy qp workqueue failed!\n");
- return -ENOMEM;
- }
-
- return 0;
-}
-
-static void hns_roce_des_qp_free(struct hns_roce_dev *hr_dev)
-{
- struct hns_roce_v1_priv *priv;
- struct hns_roce_des_qp *des_qp;
-
- priv = (struct hns_roce_v1_priv *)hr_dev->priv;
- des_qp = &priv->des_qp;
-
- des_qp->requeue_flag = 0;
- flush_workqueue(des_qp->qp_wq);
- destroy_workqueue(des_qp->qp_wq);
-}
-
static int hns_roce_v1_profile(struct hns_roce_dev *hr_dev)
{
int i = 0;
goto error_failed_tptr_init;
}
- ret = hns_roce_des_qp_init(hr_dev);
- if (ret) {
- dev_err(dev, "des qp init failed!\n");
- goto error_failed_des_qp_init;
- }
-
ret = hns_roce_free_mr_init(hr_dev);
if (ret) {
dev_err(dev, "free mr init failed!\n");
return 0;
error_failed_free_mr_init:
- hns_roce_des_qp_free(hr_dev);
-
-error_failed_des_qp_init:
hns_roce_tptr_free(hr_dev);
error_failed_tptr_init:
{
hns_roce_port_enable(hr_dev, HNS_ROCE_PORT_DOWN);
hns_roce_free_mr_free(hr_dev);
- hns_roce_des_qp_free(hr_dev);
hns_roce_tptr_free(hr_dev);
hns_roce_bt_free(hr_dev);
hns_roce_raq_free(hr_dev);
hns_roce_v1_q_qp(ibqp, qp_attr, qp_attr_mask, qp_init_attr);
}
-static void hns_roce_check_sdb_status(struct hns_roce_dev *hr_dev,
- u32 *old_send, u32 *old_retry,
- u32 *tsp_st, u32 *success_flags)
-{
- __le32 *old_send_tmp, *old_retry_tmp;
- u32 sdb_retry_cnt;
- u32 sdb_send_ptr;
- u32 cur_cnt, old_cnt;
- __le32 tmp, tmp1;
- u32 send_ptr;
-
- sdb_send_ptr = roce_read(hr_dev, ROCEE_SDB_SEND_PTR_REG);
- sdb_retry_cnt = roce_read(hr_dev, ROCEE_SDB_RETRY_CNT_REG);
- tmp = cpu_to_le32(sdb_send_ptr);
- tmp1 = cpu_to_le32(sdb_retry_cnt);
- cur_cnt = roce_get_field(tmp, ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S) +
- roce_get_field(tmp1, ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_M,
- ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_S);
-
- old_send_tmp = (__le32 *)old_send;
- old_retry_tmp = (__le32 *)old_retry;
- if (!roce_get_bit(*tsp_st, ROCEE_CNT_CLR_CE_CNT_CLR_CE_S)) {
- old_cnt = roce_get_field(*old_send_tmp,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S) +
- roce_get_field(*old_retry_tmp,
- ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_M,
- ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_S);
- if (cur_cnt - old_cnt > SDB_ST_CMP_VAL)
- *success_flags = 1;
- } else {
- old_cnt = roce_get_field(*old_send_tmp,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S);
- if (cur_cnt - old_cnt > SDB_ST_CMP_VAL) {
- *success_flags = 1;
- } else {
- send_ptr = roce_get_field(*old_send_tmp,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S) +
- roce_get_field(tmp1,
- ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_M,
- ROCEE_SDB_RETRY_CNT_SDB_RETRY_CT_S);
- roce_set_field(*old_send_tmp,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S,
- send_ptr);
- }
- }
-}
-
-static int check_qp_db_process_status(struct hns_roce_dev *hr_dev,
- struct hns_roce_qp *hr_qp,
- u32 sdb_issue_ptr,
- u32 *sdb_inv_cnt,
- u32 *wait_stage)
-{
- struct device *dev = &hr_dev->pdev->dev;
- u32 sdb_send_ptr, old_send;
- __le32 sdb_issue_ptr_tmp;
- __le32 sdb_send_ptr_tmp;
- u32 success_flags = 0;
- unsigned long end;
- u32 old_retry;
- u32 inv_cnt;
- u32 tsp_st;
- __le32 tmp;
-
- if (*wait_stage > HNS_ROCE_V1_DB_STAGE2 ||
- *wait_stage < HNS_ROCE_V1_DB_STAGE1) {
- dev_err(dev, "QP(0x%lx) db status wait stage(%d) error!\n",
- hr_qp->qpn, *wait_stage);
- return -EINVAL;
- }
-
- /* Calculate the total timeout for the entire verification process */
- end = msecs_to_jiffies(HNS_ROCE_V1_CHECK_DB_TIMEOUT_MSECS) + jiffies;
-
- if (*wait_stage == HNS_ROCE_V1_DB_STAGE1) {
- /* Query db process status, until hw process completely */
- sdb_send_ptr = roce_read(hr_dev, ROCEE_SDB_SEND_PTR_REG);
- while (roce_hw_index_cmp_lt(sdb_send_ptr, sdb_issue_ptr,
- ROCEE_SDB_PTR_CMP_BITS)) {
- if (!time_before(jiffies, end)) {
- dev_dbg(dev, "QP(0x%lx) db process stage1 timeout. issue 0x%x send 0x%x.\n",
- hr_qp->qpn, sdb_issue_ptr,
- sdb_send_ptr);
- return 0;
- }
-
- msleep(HNS_ROCE_V1_CHECK_DB_SLEEP_MSECS);
- sdb_send_ptr = roce_read(hr_dev,
- ROCEE_SDB_SEND_PTR_REG);
- }
-
- sdb_send_ptr_tmp = cpu_to_le32(sdb_send_ptr);
- sdb_issue_ptr_tmp = cpu_to_le32(sdb_issue_ptr);
- if (roce_get_field(sdb_issue_ptr_tmp,
- ROCEE_SDB_ISSUE_PTR_SDB_ISSUE_PTR_M,
- ROCEE_SDB_ISSUE_PTR_SDB_ISSUE_PTR_S) ==
- roce_get_field(sdb_send_ptr_tmp,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_M,
- ROCEE_SDB_SEND_PTR_SDB_SEND_PTR_S)) {
- old_send = roce_read(hr_dev, ROCEE_SDB_SEND_PTR_REG);
- old_retry = roce_read(hr_dev, ROCEE_SDB_RETRY_CNT_REG);
-
- do {
- tsp_st = roce_read(hr_dev, ROCEE_TSP_BP_ST_REG);
- tmp = cpu_to_le32(tsp_st);
- if (roce_get_bit(tmp,
- ROCEE_TSP_BP_ST_QH_FIFO_ENTRY_S) == 1) {
- *wait_stage = HNS_ROCE_V1_DB_WAIT_OK;
- return 0;
- }
-
- if (!time_before(jiffies, end)) {
- dev_dbg(dev, "QP(0x%lx) db process stage1 timeout when send ptr equals issue ptr.\n"
- "issue 0x%x send 0x%x.\n",
- hr_qp->qpn,
- le32_to_cpu(sdb_issue_ptr_tmp),
- le32_to_cpu(sdb_send_ptr_tmp));
- return 0;
- }
-
- msleep(HNS_ROCE_V1_CHECK_DB_SLEEP_MSECS);
-
- hns_roce_check_sdb_status(hr_dev, &old_send,
- &old_retry, &tsp_st,
- &success_flags);
- } while (!success_flags);
- }
-
- *wait_stage = HNS_ROCE_V1_DB_STAGE2;
-
- /* Get list pointer */
- *sdb_inv_cnt = roce_read(hr_dev, ROCEE_SDB_INV_CNT_REG);
- dev_dbg(dev, "QP(0x%lx) db process stage2. inv cnt = 0x%x.\n",
- hr_qp->qpn, *sdb_inv_cnt);
- }
-
- if (*wait_stage == HNS_ROCE_V1_DB_STAGE2) {
- /* Query db's list status, until hw reversal */
- inv_cnt = roce_read(hr_dev, ROCEE_SDB_INV_CNT_REG);
- while (roce_hw_index_cmp_lt(inv_cnt,
- *sdb_inv_cnt + SDB_INV_CNT_OFFSET,
- ROCEE_SDB_CNT_CMP_BITS)) {
- if (!time_before(jiffies, end)) {
- dev_dbg(dev, "QP(0x%lx) db process stage2 timeout. inv cnt 0x%x.\n",
- hr_qp->qpn, inv_cnt);
- return 0;
- }
-
- msleep(HNS_ROCE_V1_CHECK_DB_SLEEP_MSECS);
- inv_cnt = roce_read(hr_dev, ROCEE_SDB_INV_CNT_REG);
- }
-
- *wait_stage = HNS_ROCE_V1_DB_WAIT_OK;
- }
-
- return 0;
-}
-
-static int check_qp_reset_state(struct hns_roce_dev *hr_dev,
- struct hns_roce_qp *hr_qp,
- struct hns_roce_qp_work *qp_work_entry,
- int *is_timeout)
-{
- struct device *dev = &hr_dev->pdev->dev;
- u32 sdb_issue_ptr;
- int ret;
-
- if (hr_qp->state != IB_QPS_RESET) {
- /* Set qp to ERR, waiting for hw complete processing all dbs */
- ret = hns_roce_v1_modify_qp(&hr_qp->ibqp, NULL, 0, hr_qp->state,
- IB_QPS_ERR);
- if (ret) {
- dev_err(dev, "Modify QP(0x%lx) to ERR failed!\n",
- hr_qp->qpn);
- return ret;
- }
-
- /* Record issued doorbell */
- sdb_issue_ptr = roce_read(hr_dev, ROCEE_SDB_ISSUE_PTR_REG);
- qp_work_entry->sdb_issue_ptr = sdb_issue_ptr;
- qp_work_entry->db_wait_stage = HNS_ROCE_V1_DB_STAGE1;
-
- /* Query db process status, until hw process completely */
- ret = check_qp_db_process_status(hr_dev, hr_qp, sdb_issue_ptr,
- &qp_work_entry->sdb_inv_cnt,
- &qp_work_entry->db_wait_stage);
- if (ret) {
- dev_err(dev, "Check QP(0x%lx) db process status failed!\n",
- hr_qp->qpn);
- return ret;
- }
-
- if (qp_work_entry->db_wait_stage != HNS_ROCE_V1_DB_WAIT_OK) {
- qp_work_entry->sche_cnt = 0;
- *is_timeout = 1;
- return 0;
- }
-
- /* Modify qp to reset before destroying qp */
- ret = hns_roce_v1_modify_qp(&hr_qp->ibqp, NULL, 0, hr_qp->state,
- IB_QPS_RESET);
- if (ret) {
- dev_err(dev, "Modify QP(0x%lx) to RST failed!\n",
- hr_qp->qpn);
- return ret;
- }
- }
-
- return 0;
-}
-
-static void hns_roce_v1_destroy_qp_work_fn(struct work_struct *work)
-{
- struct hns_roce_qp_work *qp_work_entry;
- struct hns_roce_v1_priv *priv;
- struct hns_roce_dev *hr_dev;
- struct hns_roce_qp *hr_qp;
- struct device *dev;
- unsigned long qpn;
- int ret;
-
- qp_work_entry = container_of(work, struct hns_roce_qp_work, work);
- hr_dev = to_hr_dev(qp_work_entry->ib_dev);
- dev = &hr_dev->pdev->dev;
- priv = (struct hns_roce_v1_priv *)hr_dev->priv;
- hr_qp = qp_work_entry->qp;
- qpn = hr_qp->qpn;
-
- dev_dbg(dev, "Schedule destroy QP(0x%lx) work.\n", qpn);
-
- qp_work_entry->sche_cnt++;
-
- /* Query db process status, until hw process completely */
- ret = check_qp_db_process_status(hr_dev, hr_qp,
- qp_work_entry->sdb_issue_ptr,
- &qp_work_entry->sdb_inv_cnt,
- &qp_work_entry->db_wait_stage);
- if (ret) {
- dev_err(dev, "Check QP(0x%lx) db process status failed!\n",
- qpn);
- return;
- }
-
- if (qp_work_entry->db_wait_stage != HNS_ROCE_V1_DB_WAIT_OK &&
- priv->des_qp.requeue_flag) {
- queue_work(priv->des_qp.qp_wq, work);
- return;
- }
-
- /* Modify qp to reset before destroying qp */
- ret = hns_roce_v1_modify_qp(&hr_qp->ibqp, NULL, 0, hr_qp->state,
- IB_QPS_RESET);
- if (ret) {
- dev_err(dev, "Modify QP(0x%lx) to RST failed!\n", qpn);
- return;
- }
-
- hns_roce_qp_remove(hr_dev, hr_qp);
- hns_roce_qp_free(hr_dev, hr_qp);
-
- if (hr_qp->ibqp.qp_type == IB_QPT_RC) {
- /* RC QP, release QPN */
- hns_roce_release_range_qp(hr_dev, qpn, 1);
- kfree(hr_qp);
- } else
- kfree(hr_to_hr_sqp(hr_qp));
-
- kfree(qp_work_entry);
-
- dev_dbg(dev, "Accomplished destroy QP(0x%lx) work.\n", qpn);
-}
-
-int hns_roce_v1_destroy_qp(struct ib_qp *ibqp)
+int hns_roce_v1_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
{
struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);
struct hns_roce_qp *hr_qp = to_hr_qp(ibqp);
- struct device *dev = &hr_dev->pdev->dev;
- struct hns_roce_qp_work qp_work_entry;
- struct hns_roce_qp_work *qp_work;
- struct hns_roce_v1_priv *priv;
struct hns_roce_cq *send_cq, *recv_cq;
- bool is_user = ibqp->uobject;
- int is_timeout = 0;
int ret;
- ret = check_qp_reset_state(hr_dev, hr_qp, &qp_work_entry, &is_timeout);
- if (ret) {
- dev_err(dev, "QP reset state check failed(%d)!\n", ret);
+ ret = hns_roce_v1_modify_qp(ibqp, NULL, 0, hr_qp->state, IB_QPS_RESET);
+ if (ret)
return ret;
- }
send_cq = to_hr_cq(hr_qp->ibqp.send_cq);
recv_cq = to_hr_cq(hr_qp->ibqp.recv_cq);
hns_roce_lock_cqs(send_cq, recv_cq);
- if (!is_user) {
+ if (!udata) {
__hns_roce_v1_cq_clean(recv_cq, hr_qp->qpn, hr_qp->ibqp.srq ?
to_hr_srq(hr_qp->ibqp.srq) : NULL);
if (send_cq != recv_cq)
}
hns_roce_unlock_cqs(send_cq, recv_cq);
- if (!is_timeout) {
- hns_roce_qp_remove(hr_dev, hr_qp);
- hns_roce_qp_free(hr_dev, hr_qp);
+ hns_roce_qp_remove(hr_dev, hr_qp);
+ hns_roce_qp_free(hr_dev, hr_qp);
- /* RC QP, release QPN */
- if (hr_qp->ibqp.qp_type == IB_QPT_RC)
- hns_roce_release_range_qp(hr_dev, hr_qp->qpn, 1);
- }
+ /* RC QP, release QPN */
+ if (hr_qp->ibqp.qp_type == IB_QPT_RC)
+ hns_roce_release_range_qp(hr_dev, hr_qp->qpn, 1);
hns_roce_mtt_cleanup(hr_dev, &hr_qp->mtt);
- if (is_user)
+ if (udata)
ib_umem_release(hr_qp->umem);
else {
kfree(hr_qp->sq.wrid);
hns_roce_buf_free(hr_dev, hr_qp->buff_size, &hr_qp->hr_buf);
}
- if (!is_timeout) {
- if (hr_qp->ibqp.qp_type == IB_QPT_RC)
- kfree(hr_qp);
- else
- kfree(hr_to_hr_sqp(hr_qp));
- } else {
- qp_work = kzalloc(sizeof(*qp_work), GFP_KERNEL);
- if (!qp_work)
- return -ENOMEM;
-
- INIT_WORK(&qp_work->work, hns_roce_v1_destroy_qp_work_fn);
- qp_work->ib_dev = &hr_dev->ib_dev;
- qp_work->qp = hr_qp;
- qp_work->db_wait_stage = qp_work_entry.db_wait_stage;
- qp_work->sdb_issue_ptr = qp_work_entry.sdb_issue_ptr;
- qp_work->sdb_inv_cnt = qp_work_entry.sdb_inv_cnt;
- qp_work->sche_cnt = qp_work_entry.sche_cnt;
-
- priv = (struct hns_roce_v1_priv *)hr_dev->priv;
- queue_work(priv->des_qp.qp_wq, &qp_work->work);
- dev_dbg(dev, "Begin destroy QP(0x%lx) work.\n", hr_qp->qpn);
- }
-
+ if (hr_qp->ibqp.qp_type == IB_QPT_RC)
+ kfree(hr_qp);
+ else
+ kfree(hr_to_hr_sqp(hr_qp));
return 0;
}
-static int hns_roce_v1_destroy_cq(struct ib_cq *ibcq)
+static int hns_roce_v1_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
{
struct hns_roce_dev *hr_dev = to_hr_dev(ibcq->device);
struct hns_roce_cq *hr_cq = to_hr_cq(ibcq);
#define HNS_ROCE_V1_EXT_ODB_ALFUL \
(HNS_ROCE_V1_EXT_ODB_DEPTH - HNS_ROCE_V1_DB_RSVD)
-#define HNS_ROCE_V1_DB_WAIT_OK 0
-#define HNS_ROCE_V1_DB_STAGE1 1
-#define HNS_ROCE_V1_DB_STAGE2 2
-#define HNS_ROCE_V1_CHECK_DB_TIMEOUT_MSECS 10000
-#define HNS_ROCE_V1_CHECK_DB_SLEEP_MSECS 20
#define HNS_ROCE_V1_FREE_MR_TIMEOUT_MSECS 50000
#define HNS_ROCE_V1_RECREATE_LP_QP_TIMEOUT_MSECS 10000
#define HNS_ROCE_V1_FREE_MR_WAIT_VALUE 5
#define SQ_PSN_SHIFT 8
#define QKEY_VAL 0x80010000
#define SDB_INV_CNT_OFFSET 8
-#define SDB_ST_CMP_VAL 8
#define HNS_ROCE_CEQ_DEFAULT_INTERVAL 0x10
#define HNS_ROCE_CEQ_DEFAULT_BURST_NUM 0x10
u32 sche_cnt;
};
-struct hns_roce_des_qp {
- struct workqueue_struct *qp_wq;
- int requeue_flag;
-};
-
struct hns_roce_mr_free_work {
struct work_struct work;
struct ib_device *ib_dev;
struct hns_roce_raq_table raq_table;
struct hns_roce_bt_table bt_table;
struct hns_roce_tptr_table tptr_table;
- struct hns_roce_des_qp des_qp;
struct hns_roce_free_mr free_mr;
};
int hns_dsaf_roce_reset(struct fwnode_handle *dsaf_fwnode, bool dereset);
int hns_roce_v1_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc);
-int hns_roce_v1_destroy_qp(struct ib_qp *ibqp);
+int hns_roce_v1_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata);
#endif
#include <linux/types.h>
#include <net/addrconf.h>
#include <rdma/ib_addr.h>
+#include <rdma/ib_cache.h>
#include <rdma/ib_umem.h>
+#include <rdma/uverbs_ioctl.h>
#include "hnae3.h"
#include "hns_roce_common.h"
return ret;
}
-int hns_roce_cmq_send(struct hns_roce_dev *hr_dev,
+static int hns_roce_cmq_send(struct hns_roce_dev *hr_dev,
struct hns_roce_cmq_desc *desc, int num)
{
int retval;
caps->qpc_hop_num = HNS_ROCE_CONTEXT_HOP_NUM;
caps->srqc_ba_pg_sz = 0;
caps->srqc_buf_pg_sz = 0;
- caps->srqc_hop_num = HNS_ROCE_HOP_NUM_0;
+ caps->srqc_hop_num = HNS_ROCE_CONTEXT_HOP_NUM;
caps->cqc_ba_pg_sz = 0;
caps->cqc_buf_pg_sz = 0;
caps->cqc_hop_num = HNS_ROCE_CONTEXT_HOP_NUM;
V2_MPT_BYTE_4_PD_S, mr->pd);
roce_set_bit(mpt_entry->byte_8_mw_cnt_en, V2_MPT_BYTE_8_RA_EN_S, 0);
- roce_set_bit(mpt_entry->byte_8_mw_cnt_en, V2_MPT_BYTE_8_R_INV_EN_S, 1);
+ roce_set_bit(mpt_entry->byte_8_mw_cnt_en, V2_MPT_BYTE_8_R_INV_EN_S, 0);
roce_set_bit(mpt_entry->byte_8_mw_cnt_en, V2_MPT_BYTE_8_L_INV_EN_S, 1);
roce_set_bit(mpt_entry->byte_8_mw_cnt_en, V2_MPT_BYTE_8_BIND_EN_S,
(mr->access & IB_ACCESS_MW_BIND ? 1 : 0));
roce_set_bit(qpc_mask->byte_28_at_fl, V2_QPC_BYTE_28_CNP_TX_FLAG_S, 0);
roce_set_bit(qpc_mask->byte_28_at_fl, V2_QPC_BYTE_28_CE_FLAG_S, 0);
- if (attr_mask & IB_QP_QKEY) {
- context->qkey_xrcd = attr->qkey;
- qpc_mask->qkey_xrcd = 0;
- hr_qp->qkey = attr->qkey;
- }
-
if (hr_qp->rdb_en) {
roce_set_bit(context->byte_68_rq_db,
V2_QPC_BYTE_68_RQ_RECORD_EN_S, 1);
0);
hr_qp->access_flags = attr->qp_access_flags;
- hr_qp->pkey_index = attr->pkey_index;
roce_set_field(context->byte_252_err_txcqn, V2_QPC_BYTE_252_TX_CQN_M,
V2_QPC_BYTE_252_TX_CQN_S, to_hr_cq(ibqp->send_cq)->cqn);
roce_set_field(qpc_mask->byte_252_err_txcqn, V2_QPC_BYTE_252_TX_CQN_M,
V2_QPC_BYTE_76_SRQN_M, V2_QPC_BYTE_76_SRQN_S, 0);
}
- if (attr_mask & IB_QP_QKEY) {
- context->qkey_xrcd = attr->qkey;
- qpc_mask->qkey_xrcd = 0;
- }
-
roce_set_field(context->byte_4_sqpn_tst, V2_QPC_BYTE_4_SQPN_M,
V2_QPC_BYTE_4_SQPN_S, hr_qp->qpn);
roce_set_field(qpc_mask->byte_4_sqpn_tst, V2_QPC_BYTE_4_SQPN_M,
V2_QPC_BYTE_16_WQE_SGE_BUF_PG_SZ_M,
V2_QPC_BYTE_16_WQE_SGE_BUF_PG_SZ_S, 0);
- roce_set_field(context->byte_80_rnr_rx_cqn,
- V2_QPC_BYTE_80_MIN_RNR_TIME_M,
- V2_QPC_BYTE_80_MIN_RNR_TIME_S, attr->min_rnr_timer);
- roce_set_field(qpc_mask->byte_80_rnr_rx_cqn,
- V2_QPC_BYTE_80_MIN_RNR_TIME_M,
- V2_QPC_BYTE_80_MIN_RNR_TIME_S, 0);
-
page_size = 1 << (hr_dev->caps.mtt_buf_pg_sz + PAGE_SHIFT);
context->rq_cur_blk_addr = (u32)(mtts[hr_qp->rq.offset / page_size]
>> PAGE_ADDR_SHIFT);
V2_QPC_BYTE_104_RQ_NXT_BLK_ADDR_M,
V2_QPC_BYTE_104_RQ_NXT_BLK_ADDR_S, 0);
- roce_set_field(context->byte_108_rx_reqepsn,
- V2_QPC_BYTE_108_RX_REQ_EPSN_M,
- V2_QPC_BYTE_108_RX_REQ_EPSN_S, attr->rq_psn);
- roce_set_field(qpc_mask->byte_108_rx_reqepsn,
- V2_QPC_BYTE_108_RX_REQ_EPSN_M,
- V2_QPC_BYTE_108_RX_REQ_EPSN_S, 0);
-
roce_set_field(context->byte_132_trrl, V2_QPC_BYTE_132_TRRL_BA_M,
V2_QPC_BYTE_132_TRRL_BA_S, dma_handle_3 >> 4);
roce_set_field(qpc_mask->byte_132_trrl, V2_QPC_BYTE_132_TRRL_BA_M,
roce_set_bit(qpc_mask->byte_28_at_fl, V2_QPC_BYTE_28_LBI_S, 0);
}
- if ((attr_mask & IB_QP_MAX_DEST_RD_ATOMIC) &&
- attr->max_dest_rd_atomic) {
- roce_set_field(context->byte_140_raq, V2_QPC_BYTE_140_RR_MAX_M,
- V2_QPC_BYTE_140_RR_MAX_S,
- fls(attr->max_dest_rd_atomic - 1));
- roce_set_field(qpc_mask->byte_140_raq, V2_QPC_BYTE_140_RR_MAX_M,
- V2_QPC_BYTE_140_RR_MAX_S, 0);
- }
-
if (attr_mask & IB_QP_DEST_QPN) {
roce_set_field(context->byte_56_dqpn_err, V2_QPC_BYTE_56_DQPN_M,
V2_QPC_BYTE_56_DQPN_S, attr->dest_qp_num);
context->rq_rnr_timer = 0;
qpc_mask->rq_rnr_timer = 0;
- roce_set_field(context->byte_152_raq, V2_QPC_BYTE_152_RAQ_PSN_M,
- V2_QPC_BYTE_152_RAQ_PSN_S, attr->rq_psn - 1);
- roce_set_field(qpc_mask->byte_152_raq, V2_QPC_BYTE_152_RAQ_PSN_M,
- V2_QPC_BYTE_152_RAQ_PSN_S, 0);
-
roce_set_field(qpc_mask->byte_132_trrl, V2_QPC_BYTE_132_TRRL_HEAD_MAX_M,
V2_QPC_BYTE_132_TRRL_HEAD_MAX_S, 0);
roce_set_field(qpc_mask->byte_132_trrl, V2_QPC_BYTE_132_TRRL_TAIL_MAX_M,
V2_QPC_BYTE_240_RX_ACK_MSN_M,
V2_QPC_BYTE_240_RX_ACK_MSN_S, 0);
- roce_set_field(context->byte_244_rnr_rxack,
- V2_QPC_BYTE_244_RX_ACK_EPSN_M,
- V2_QPC_BYTE_244_RX_ACK_EPSN_S, attr->sq_psn);
- roce_set_field(qpc_mask->byte_244_rnr_rxack,
- V2_QPC_BYTE_244_RX_ACK_EPSN_M,
- V2_QPC_BYTE_244_RX_ACK_EPSN_S, 0);
-
roce_set_field(qpc_mask->byte_248_ack_psn,
V2_QPC_BYTE_248_ACK_LAST_OPTYPE_M,
V2_QPC_BYTE_248_ACK_LAST_OPTYPE_S, 0);
V2_QPC_BYTE_240_IRRL_TAIL_REAL_M,
V2_QPC_BYTE_240_IRRL_TAIL_REAL_S, 0);
- roce_set_field(context->byte_220_retry_psn_msn,
- V2_QPC_BYTE_220_RETRY_MSG_PSN_M,
- V2_QPC_BYTE_220_RETRY_MSG_PSN_S, attr->sq_psn);
- roce_set_field(qpc_mask->byte_220_retry_psn_msn,
- V2_QPC_BYTE_220_RETRY_MSG_PSN_M,
- V2_QPC_BYTE_220_RETRY_MSG_PSN_S, 0);
-
- roce_set_field(context->byte_224_retry_msg,
- V2_QPC_BYTE_224_RETRY_MSG_PSN_M,
- V2_QPC_BYTE_224_RETRY_MSG_PSN_S, attr->sq_psn >> 16);
- roce_set_field(qpc_mask->byte_224_retry_msg,
- V2_QPC_BYTE_224_RETRY_MSG_PSN_M,
- V2_QPC_BYTE_224_RETRY_MSG_PSN_S, 0);
-
- roce_set_field(context->byte_224_retry_msg,
- V2_QPC_BYTE_224_RETRY_MSG_FPKT_PSN_M,
- V2_QPC_BYTE_224_RETRY_MSG_FPKT_PSN_S, attr->sq_psn);
- roce_set_field(qpc_mask->byte_224_retry_msg,
- V2_QPC_BYTE_224_RETRY_MSG_FPKT_PSN_M,
- V2_QPC_BYTE_224_RETRY_MSG_FPKT_PSN_S, 0);
-
roce_set_field(qpc_mask->byte_220_retry_psn_msn,
V2_QPC_BYTE_220_RETRY_MSG_MSN_M,
V2_QPC_BYTE_220_RETRY_MSG_MSN_S, 0);
roce_set_field(qpc_mask->byte_212_lsn, V2_QPC_BYTE_212_CHECK_FLG_M,
V2_QPC_BYTE_212_CHECK_FLG_S, 0);
- roce_set_field(context->byte_212_lsn, V2_QPC_BYTE_212_RETRY_CNT_M,
- V2_QPC_BYTE_212_RETRY_CNT_S, attr->retry_cnt);
- roce_set_field(qpc_mask->byte_212_lsn, V2_QPC_BYTE_212_RETRY_CNT_M,
- V2_QPC_BYTE_212_RETRY_CNT_S, 0);
-
- roce_set_field(context->byte_212_lsn, V2_QPC_BYTE_212_RETRY_NUM_INIT_M,
- V2_QPC_BYTE_212_RETRY_NUM_INIT_S, attr->retry_cnt);
- roce_set_field(qpc_mask->byte_212_lsn, V2_QPC_BYTE_212_RETRY_NUM_INIT_M,
- V2_QPC_BYTE_212_RETRY_NUM_INIT_S, 0);
-
- roce_set_field(context->byte_244_rnr_rxack,
- V2_QPC_BYTE_244_RNR_NUM_INIT_M,
- V2_QPC_BYTE_244_RNR_NUM_INIT_S, attr->rnr_retry);
- roce_set_field(qpc_mask->byte_244_rnr_rxack,
- V2_QPC_BYTE_244_RNR_NUM_INIT_M,
- V2_QPC_BYTE_244_RNR_NUM_INIT_S, 0);
-
- roce_set_field(context->byte_244_rnr_rxack, V2_QPC_BYTE_244_RNR_CNT_M,
- V2_QPC_BYTE_244_RNR_CNT_S, attr->rnr_retry);
- roce_set_field(qpc_mask->byte_244_rnr_rxack, V2_QPC_BYTE_244_RNR_CNT_M,
- V2_QPC_BYTE_244_RNR_CNT_S, 0);
-
roce_set_field(context->byte_212_lsn, V2_QPC_BYTE_212_LSN_M,
V2_QPC_BYTE_212_LSN_S, 0x100);
roce_set_field(qpc_mask->byte_212_lsn, V2_QPC_BYTE_212_LSN_M,
V2_QPC_BYTE_212_LSN_S, 0);
- if (attr_mask & IB_QP_TIMEOUT) {
- if (attr->timeout < 31) {
- roce_set_field(context->byte_28_at_fl,
- V2_QPC_BYTE_28_AT_M, V2_QPC_BYTE_28_AT_S,
- attr->timeout);
- roce_set_field(qpc_mask->byte_28_at_fl,
- V2_QPC_BYTE_28_AT_M, V2_QPC_BYTE_28_AT_S,
- 0);
- } else {
- dev_warn(dev, "Local ACK timeout shall be 0 to 30.\n");
- }
- }
-
- roce_set_field(context->byte_172_sq_psn, V2_QPC_BYTE_172_SQ_CUR_PSN_M,
- V2_QPC_BYTE_172_SQ_CUR_PSN_S, attr->sq_psn);
- roce_set_field(qpc_mask->byte_172_sq_psn, V2_QPC_BYTE_172_SQ_CUR_PSN_M,
- V2_QPC_BYTE_172_SQ_CUR_PSN_S, 0);
-
roce_set_field(qpc_mask->byte_196_sq_psn, V2_QPC_BYTE_196_IRRL_HEAD_M,
V2_QPC_BYTE_196_IRRL_HEAD_S, 0);
- roce_set_field(context->byte_196_sq_psn, V2_QPC_BYTE_196_SQ_MAX_PSN_M,
- V2_QPC_BYTE_196_SQ_MAX_PSN_S, attr->sq_psn);
- roce_set_field(qpc_mask->byte_196_sq_psn, V2_QPC_BYTE_196_SQ_MAX_PSN_M,
- V2_QPC_BYTE_196_SQ_MAX_PSN_S, 0);
- if ((attr_mask & IB_QP_MAX_QP_RD_ATOMIC) && attr->max_rd_atomic) {
- roce_set_field(context->byte_208_irrl, V2_QPC_BYTE_208_SR_MAX_M,
- V2_QPC_BYTE_208_SR_MAX_S,
- fls(attr->max_rd_atomic - 1));
- roce_set_field(qpc_mask->byte_208_irrl,
- V2_QPC_BYTE_208_SR_MAX_M,
- V2_QPC_BYTE_208_SR_MAX_S, 0);
- }
return 0;
}
const struct ib_global_route *grh =
rdma_ah_read_grh(&attr->ah_attr);
const struct ib_gid_attr *gid_attr = NULL;
- u8 src_mac[ETH_ALEN];
int is_roce_protocol;
u16 vlan = 0xffff;
u8 ib_port;
if (is_roce_protocol) {
gid_attr = attr->ah_attr.grh.sgid_attr;
- vlan = rdma_vlan_dev_vlan_id(gid_attr->ndev);
- memcpy(src_mac, gid_attr->ndev->dev_addr, ETH_ALEN);
+ ret = rdma_read_gid_l2_fields(gid_attr, &vlan, NULL);
+ if (ret)
+ goto out;
}
- if (is_vlan_dev(gid_attr->ndev)) {
+ if (vlan < VLAN_CFI_MASK) {
roce_set_bit(context->byte_76_srqn_op_en,
V2_QPC_BYTE_76_RQ_VLAN_EN_S, 1);
roce_set_bit(qpc_mask->byte_76_srqn_op_en,
hr_qp->sl = rdma_ah_get_sl(&attr->ah_attr);
}
+ if (attr_mask & IB_QP_TIMEOUT) {
+ if (attr->timeout < 31) {
+ roce_set_field(context->byte_28_at_fl,
+ V2_QPC_BYTE_28_AT_M, V2_QPC_BYTE_28_AT_S,
+ attr->timeout);
+ roce_set_field(qpc_mask->byte_28_at_fl,
+ V2_QPC_BYTE_28_AT_M, V2_QPC_BYTE_28_AT_S,
+ 0);
+ } else {
+ dev_warn(dev, "Local ACK timeout shall be 0 to 30.\n");
+ }
+ }
+
+ if (attr_mask & IB_QP_RETRY_CNT) {
+ roce_set_field(context->byte_212_lsn,
+ V2_QPC_BYTE_212_RETRY_NUM_INIT_M,
+ V2_QPC_BYTE_212_RETRY_NUM_INIT_S,
+ attr->retry_cnt);
+ roce_set_field(qpc_mask->byte_212_lsn,
+ V2_QPC_BYTE_212_RETRY_NUM_INIT_M,
+ V2_QPC_BYTE_212_RETRY_NUM_INIT_S, 0);
+
+ roce_set_field(context->byte_212_lsn,
+ V2_QPC_BYTE_212_RETRY_CNT_M,
+ V2_QPC_BYTE_212_RETRY_CNT_S,
+ attr->retry_cnt);
+ roce_set_field(qpc_mask->byte_212_lsn,
+ V2_QPC_BYTE_212_RETRY_CNT_M,
+ V2_QPC_BYTE_212_RETRY_CNT_S, 0);
+ }
+
+ if (attr_mask & IB_QP_RNR_RETRY) {
+ roce_set_field(context->byte_244_rnr_rxack,
+ V2_QPC_BYTE_244_RNR_NUM_INIT_M,
+ V2_QPC_BYTE_244_RNR_NUM_INIT_S, attr->rnr_retry);
+ roce_set_field(qpc_mask->byte_244_rnr_rxack,
+ V2_QPC_BYTE_244_RNR_NUM_INIT_M,
+ V2_QPC_BYTE_244_RNR_NUM_INIT_S, 0);
+
+ roce_set_field(context->byte_244_rnr_rxack,
+ V2_QPC_BYTE_244_RNR_CNT_M,
+ V2_QPC_BYTE_244_RNR_CNT_S, attr->rnr_retry);
+ roce_set_field(qpc_mask->byte_244_rnr_rxack,
+ V2_QPC_BYTE_244_RNR_CNT_M,
+ V2_QPC_BYTE_244_RNR_CNT_S, 0);
+ }
+
+ if (attr_mask & IB_QP_SQ_PSN) {
+ roce_set_field(context->byte_172_sq_psn,
+ V2_QPC_BYTE_172_SQ_CUR_PSN_M,
+ V2_QPC_BYTE_172_SQ_CUR_PSN_S, attr->sq_psn);
+ roce_set_field(qpc_mask->byte_172_sq_psn,
+ V2_QPC_BYTE_172_SQ_CUR_PSN_M,
+ V2_QPC_BYTE_172_SQ_CUR_PSN_S, 0);
+
+ roce_set_field(context->byte_196_sq_psn,
+ V2_QPC_BYTE_196_SQ_MAX_PSN_M,
+ V2_QPC_BYTE_196_SQ_MAX_PSN_S, attr->sq_psn);
+ roce_set_field(qpc_mask->byte_196_sq_psn,
+ V2_QPC_BYTE_196_SQ_MAX_PSN_M,
+ V2_QPC_BYTE_196_SQ_MAX_PSN_S, 0);
+
+ roce_set_field(context->byte_220_retry_psn_msn,
+ V2_QPC_BYTE_220_RETRY_MSG_PSN_M,
+ V2_QPC_BYTE_220_RETRY_MSG_PSN_S, attr->sq_psn);
+ roce_set_field(qpc_mask->byte_220_retry_psn_msn,
+ V2_QPC_BYTE_220_RETRY_MSG_PSN_M,
+ V2_QPC_BYTE_220_RETRY_MSG_PSN_S, 0);
+
+ roce_set_field(context->byte_224_retry_msg,
+ V2_QPC_BYTE_224_RETRY_MSG_PSN_M,
+ V2_QPC_BYTE_224_RETRY_MSG_PSN_S,
+ attr->sq_psn >> 16);
+ roce_set_field(qpc_mask->byte_224_retry_msg,
+ V2_QPC_BYTE_224_RETRY_MSG_PSN_M,
+ V2_QPC_BYTE_224_RETRY_MSG_PSN_S, 0);
+
+ roce_set_field(context->byte_224_retry_msg,
+ V2_QPC_BYTE_224_RETRY_MSG_FPKT_PSN_M,
+ V2_QPC_BYTE_224_RETRY_MSG_FPKT_PSN_S,
+ attr->sq_psn);
+ roce_set_field(qpc_mask->byte_224_retry_msg,
+ V2_QPC_BYTE_224_RETRY_MSG_FPKT_PSN_M,
+ V2_QPC_BYTE_224_RETRY_MSG_FPKT_PSN_S, 0);
+
+ roce_set_field(context->byte_244_rnr_rxack,
+ V2_QPC_BYTE_244_RX_ACK_EPSN_M,
+ V2_QPC_BYTE_244_RX_ACK_EPSN_S, attr->sq_psn);
+ roce_set_field(qpc_mask->byte_244_rnr_rxack,
+ V2_QPC_BYTE_244_RX_ACK_EPSN_M,
+ V2_QPC_BYTE_244_RX_ACK_EPSN_S, 0);
+ }
+
+ if ((attr_mask & IB_QP_MAX_DEST_RD_ATOMIC) &&
+ attr->max_dest_rd_atomic) {
+ roce_set_field(context->byte_140_raq, V2_QPC_BYTE_140_RR_MAX_M,
+ V2_QPC_BYTE_140_RR_MAX_S,
+ fls(attr->max_dest_rd_atomic - 1));
+ roce_set_field(qpc_mask->byte_140_raq, V2_QPC_BYTE_140_RR_MAX_M,
+ V2_QPC_BYTE_140_RR_MAX_S, 0);
+ }
+
+ if ((attr_mask & IB_QP_MAX_QP_RD_ATOMIC) && attr->max_rd_atomic) {
+ roce_set_field(context->byte_208_irrl, V2_QPC_BYTE_208_SR_MAX_M,
+ V2_QPC_BYTE_208_SR_MAX_S,
+ fls(attr->max_rd_atomic - 1));
+ roce_set_field(qpc_mask->byte_208_irrl,
+ V2_QPC_BYTE_208_SR_MAX_M,
+ V2_QPC_BYTE_208_SR_MAX_S, 0);
+ }
+
if (attr_mask & (IB_QP_ACCESS_FLAGS | IB_QP_MAX_DEST_RD_ATOMIC))
set_access_flags(hr_qp, context, qpc_mask, attr, attr_mask);
+ if (attr_mask & IB_QP_MIN_RNR_TIMER) {
+ roce_set_field(context->byte_80_rnr_rx_cqn,
+ V2_QPC_BYTE_80_MIN_RNR_TIME_M,
+ V2_QPC_BYTE_80_MIN_RNR_TIME_S,
+ attr->min_rnr_timer);
+ roce_set_field(qpc_mask->byte_80_rnr_rx_cqn,
+ V2_QPC_BYTE_80_MIN_RNR_TIME_M,
+ V2_QPC_BYTE_80_MIN_RNR_TIME_S, 0);
+ }
+
+ /* RC&UC required attr */
+ if (attr_mask & IB_QP_RQ_PSN) {
+ roce_set_field(context->byte_108_rx_reqepsn,
+ V2_QPC_BYTE_108_RX_REQ_EPSN_M,
+ V2_QPC_BYTE_108_RX_REQ_EPSN_S, attr->rq_psn);
+ roce_set_field(qpc_mask->byte_108_rx_reqepsn,
+ V2_QPC_BYTE_108_RX_REQ_EPSN_M,
+ V2_QPC_BYTE_108_RX_REQ_EPSN_S, 0);
+
+ roce_set_field(context->byte_152_raq, V2_QPC_BYTE_152_RAQ_PSN_M,
+ V2_QPC_BYTE_152_RAQ_PSN_S, attr->rq_psn - 1);
+ roce_set_field(qpc_mask->byte_152_raq,
+ V2_QPC_BYTE_152_RAQ_PSN_M,
+ V2_QPC_BYTE_152_RAQ_PSN_S, 0);
+ }
+
+ if (attr_mask & IB_QP_QKEY) {
+ context->qkey_xrcd = attr->qkey;
+ qpc_mask->qkey_xrcd = 0;
+ hr_qp->qkey = attr->qkey;
+ }
+
roce_set_bit(context->byte_108_rx_reqepsn, V2_QPC_BYTE_108_INV_CREDIT_S,
ibqp->srq ? 1 : 0);
roce_set_bit(qpc_mask->byte_108_rx_reqepsn,
static int hns_roce_v2_destroy_qp_common(struct hns_roce_dev *hr_dev,
struct hns_roce_qp *hr_qp,
- bool is_user)
+ struct ib_udata *udata)
{
struct hns_roce_cq *send_cq, *recv_cq;
struct device *dev = hr_dev->dev;
hns_roce_lock_cqs(send_cq, recv_cq);
- if (!is_user) {
+ if (!udata) {
__hns_roce_v2_cq_clean(recv_cq, hr_qp->qpn, hr_qp->ibqp.srq ?
to_hr_srq(hr_qp->ibqp.srq) : NULL);
if (send_cq != recv_cq)
hns_roce_mtt_cleanup(hr_dev, &hr_qp->mtt);
- if (is_user) {
+ if (udata) {
+ struct hns_roce_ucontext *context =
+ rdma_udata_to_drv_context(
+ udata,
+ struct hns_roce_ucontext,
+ ibucontext);
+
if (hr_qp->sq.wqe_cnt && (hr_qp->sdb_en == 1))
- hns_roce_db_unmap_user(
- to_hr_ucontext(hr_qp->ibqp.uobject->context),
- &hr_qp->sdb);
+ hns_roce_db_unmap_user(context, &hr_qp->sdb);
if (hr_qp->rq.wqe_cnt && (hr_qp->rdb_en == 1))
- hns_roce_db_unmap_user(
- to_hr_ucontext(hr_qp->ibqp.uobject->context),
- &hr_qp->rdb);
+ hns_roce_db_unmap_user(context, &hr_qp->rdb);
ib_umem_release(hr_qp->umem);
} else {
kfree(hr_qp->sq.wrid);
return 0;
}
-static int hns_roce_v2_destroy_qp(struct ib_qp *ibqp)
+static int hns_roce_v2_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
{
struct hns_roce_dev *hr_dev = to_hr_dev(ibqp->device);
struct hns_roce_qp *hr_qp = to_hr_qp(ibqp);
int ret;
- ret = hns_roce_v2_destroy_qp_common(hr_dev, hr_qp, ibqp->uobject);
+ ret = hns_roce_v2_destroy_qp_common(hr_dev, hr_qp, udata);
if (ret) {
dev_err(hr_dev->dev, "Destroy qp failed(%d)\n", ret);
return ret;
return ret;
}
+static const struct hns_roce_dfx_hw hns_roce_dfx_hw_v2 = {
+ .query_cqc_info = hns_roce_v2_query_cqc_info,
+};
+
static const struct ib_device_ops hns_roce_v2_dev_ops = {
.destroy_qp = hns_roce_v2_destroy_qp,
.modify_cq = hns_roce_v2_modify_cq,
struct hnae3_handle *handle)
{
struct hns_roce_v2_priv *priv = hr_dev->priv;
- const struct pci_device_id *id;
int i;
- id = pci_match_id(hns_roce_hw_v2_pci_tbl, hr_dev->pci_dev);
- if (!id) {
- dev_err(hr_dev->dev, "device is not compatible!\n");
- return -ENXIO;
- }
-
hr_dev->hw = &hns_roce_hw_v2;
+ hr_dev->dfx = &hns_roce_dfx_hw_v2;
hr_dev->sdb_offset = ROCEE_DB_SQ_L_0_REG;
hr_dev->odb_offset = hr_dev->sdb_offset;
static int hns_roce_hw_v2_init_instance(struct hnae3_handle *handle)
{
const struct hnae3_ae_ops *ops = handle->ae_algo->ops;
+ const struct pci_device_id *id;
struct device *dev = &handle->pdev->dev;
int ret;
goto reset_chk_err;
}
+ id = pci_match_id(hns_roce_hw_v2_pci_tbl, handle->pdev);
+ if (!id)
+ return 0;
+
ret = __hns_roce_hw_v2_init_instance(handle);
if (ret) {
handle->rinfo.instance_state = HNS_ROCE_STATE_NON_INIT;
#define V2_QPC_BYTE_148_RAQ_SYNDROME_S 24
#define V2_QPC_BYTE_148_RAQ_SYNDROME_M GENMASK(31, 24)
-#define V2_QPC_BYTE_152_RAQ_PSN_S 8
-#define V2_QPC_BYTE_152_RAQ_PSN_M GENMASK(31, 8)
+#define V2_QPC_BYTE_152_RAQ_PSN_S 0
+#define V2_QPC_BYTE_152_RAQ_PSN_M GENMASK(23, 0)
#define V2_QPC_BYTE_152_RAQ_TRRL_RTY_HEAD_S 24
#define V2_QPC_BYTE_152_RAQ_TRRL_RTY_HEAD_M GENMASK(31, 24)
__le32 rsv[5];
};
+int hns_roce_v2_query_cqc_info(struct hns_roce_dev *hr_dev, u32 cqn,
+ int *buffer);
+
static inline void hns_roce_write64(struct hns_roce_dev *hr_dev, __le32 val[2],
void __iomem *dest)
{
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+// Copyright (c) 2019 Hisilicon Limited.
+
+#include "hnae3.h"
+#include "hns_roce_device.h"
+#include "hns_roce_cmd.h"
+#include "hns_roce_hw_v2.h"
+
+int hns_roce_v2_query_cqc_info(struct hns_roce_dev *hr_dev, u32 cqn,
+ int *buffer)
+{
+ struct hns_roce_v2_cq_context *cq_context;
+ struct hns_roce_cmd_mailbox *mailbox;
+ int ret;
+
+ mailbox = hns_roce_alloc_cmd_mailbox(hr_dev);
+ if (IS_ERR(mailbox))
+ return PTR_ERR(mailbox);
+
+ cq_context = mailbox->buf;
+ ret = hns_roce_cmd_mbox(hr_dev, 0, mailbox->dma, cqn, 0,
+ HNS_ROCE_CMD_QUERY_CQC,
+ HNS_ROCE_CMD_TIMEOUT_MSECS);
+ if (ret) {
+ dev_err(hr_dev->dev, "QUERY cqc cmd process error\n");
+ goto err_mailbox;
+ }
+
+ memcpy(buffer, cq_context, sizeof(*cq_context));
+
+err_mailbox:
+ hns_roce_free_cmd_mailbox(hr_dev, mailbox);
+
+ return ret;
+}
return 0;
}
-static struct net_device *hns_roce_get_netdev(struct ib_device *ib_dev,
- u8 port_num)
-{
- struct hns_roce_dev *hr_dev = to_hr_dev(ib_dev);
- struct net_device *ndev;
-
- if (port_num < 1 || port_num > hr_dev->caps.num_ports)
- return NULL;
-
- rcu_read_lock();
-
- ndev = hr_dev->iboe.netdevs[port_num - 1];
- if (ndev)
- dev_hold(ndev);
-
- rcu_read_unlock();
- return ndev;
-}
-
static int hns_roce_query_port(struct ib_device *ib_dev, u8 port_num,
struct ib_port_attr *props)
{
.destroy_ah = hns_roce_destroy_ah,
.destroy_cq = hns_roce_ib_destroy_cq,
.disassociate_ucontext = hns_roce_disassociate_ucontext,
+ .fill_res_entry = hns_roce_fill_res_entry,
.get_dma_mr = hns_roce_get_dma_mr,
.get_link_layer = hns_roce_get_link_layer,
- .get_netdev = hns_roce_get_netdev,
.get_port_immutable = hns_roce_port_immutable,
.mmap = hns_roce_mmap,
.modify_device = hns_roce_modify_device,
.query_pkey = hns_roce_query_pkey,
.query_port = hns_roce_query_port,
.reg_user_mr = hns_roce_reg_user_mr,
+
+ INIT_RDMA_OBJ_SIZE(ib_ah, hns_roce_ah, ibah),
INIT_RDMA_OBJ_SIZE(ib_pd, hns_roce_pd, ibpd),
INIT_RDMA_OBJ_SIZE(ib_ucontext, hns_roce_ucontext, ibucontext),
};
static const struct ib_device_ops hns_roce_dev_srq_ops = {
.create_srq = hns_roce_create_srq,
.destroy_srq = hns_roce_destroy_srq,
+
+ INIT_RDMA_OBJ_SIZE(ib_srq, hns_roce_srq, ibsrq),
};
static int hns_roce_register_device(struct hns_roce_dev *hr_dev)
struct hns_roce_ib_iboe *iboe = NULL;
struct ib_device *ib_dev = NULL;
struct device *dev = hr_dev->dev;
+ unsigned int i;
iboe = &hr_dev->iboe;
spin_lock_init(&iboe->lock);
ib_dev->driver_id = RDMA_DRIVER_HNS;
ib_set_device_ops(ib_dev, hr_dev->hw->hns_roce_dev_ops);
ib_set_device_ops(ib_dev, &hns_roce_dev_ops);
+ for (i = 0; i < hr_dev->caps.num_ports; i++) {
+ if (!hr_dev->iboe.netdevs[i])
+ continue;
+
+ ret = ib_device_set_netdev(ib_dev, hr_dev->iboe.netdevs[i],
+ i + 1);
+ if (ret)
+ return ret;
+ }
ret = ib_register_device(ib_dev, "hns_%d");
if (ret) {
dev_err(dev, "ib_register_device failed!\n");
return ret;
}
-int hns_roce_dereg_mr(struct ib_mr *ibmr)
+int hns_roce_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
{
struct hns_roce_dev *hr_dev = to_hr_dev(ibmr->device);
struct hns_roce_mr *mr = to_hr_mr(ibmr);
int ret = 0;
if (hr_dev->hw->dereg_mr) {
- ret = hr_dev->hw->dereg_mr(hr_dev, mr);
+ ret = hr_dev->hw->dereg_mr(hr_dev, mr, udata);
} else {
hns_roce_mr_free(hr_dev, mr);
}
struct ib_mr *hns_roce_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
- u32 max_num_sg)
+ u32 max_num_sg, struct ib_udata *udata)
{
struct hns_roce_dev *hr_dev = to_hr_dev(pd->device);
struct device *dev = hr_dev->dev;
hns_roce_bitmap_cleanup(&hr_dev->pd_bitmap);
}
-int hns_roce_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata)
+int hns_roce_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct ib_device *ib_dev = ibpd->device;
struct hns_roce_dev *hr_dev = to_hr_dev(ib_dev);
return ret;
}
- if (context) {
+ if (udata) {
struct hns_roce_ib_alloc_pd_resp uresp = {.pdn = pd->pdn};
if (ib_copy_to_udata(udata, &uresp, sizeof(uresp))) {
}
EXPORT_SYMBOL_GPL(hns_roce_alloc_pd);
-void hns_roce_dealloc_pd(struct ib_pd *pd)
+void hns_roce_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata)
{
hns_roce_pd_free(to_hr_dev(pd->device), to_hr_pd(pd)->pdn);
}
void hns_roce_qp_event(struct hns_roce_dev *hr_dev, u32 qpn, int event_type)
{
- struct hns_roce_qp_table *qp_table = &hr_dev->qp_table;
struct device *dev = hr_dev->dev;
struct hns_roce_qp *qp;
- spin_lock(&qp_table->lock);
-
+ xa_lock(&hr_dev->qp_table_xa);
qp = __hns_roce_qp_lookup(hr_dev, qpn);
if (qp)
atomic_inc(&qp->refcount);
-
- spin_unlock(&qp_table->lock);
+ xa_unlock(&hr_dev->qp_table_xa);
if (!qp) {
dev_warn(dev, "Async event for bogus QP %08x\n", qpn);
static int hns_roce_gsi_qp_alloc(struct hns_roce_dev *hr_dev, unsigned long qpn,
struct hns_roce_qp *hr_qp)
{
- struct hns_roce_qp_table *qp_table = &hr_dev->qp_table;
+ struct xarray *xa = &hr_dev->qp_table_xa;
int ret;
if (!qpn)
return -EINVAL;
hr_qp->qpn = qpn;
-
- spin_lock_irq(&qp_table->lock);
- ret = radix_tree_insert(&hr_dev->qp_table_tree,
- hr_qp->qpn & (hr_dev->caps.num_qps - 1), hr_qp);
- spin_unlock_irq(&qp_table->lock);
- if (ret) {
- dev_err(hr_dev->dev, "QPC radix_tree_insert failed\n");
- goto err_put_irrl;
- }
-
atomic_set(&hr_qp->refcount, 1);
init_completion(&hr_qp->free);
- return 0;
-
-err_put_irrl:
+ ret = xa_err(xa_store_irq(xa, hr_qp->qpn & (hr_dev->caps.num_qps - 1),
+ hr_qp, GFP_KERNEL));
+ if (ret)
+ dev_err(hr_dev->dev, "QPC xa_store failed\n");
return ret;
}
}
}
- spin_lock_irq(&qp_table->lock);
- ret = radix_tree_insert(&hr_dev->qp_table_tree,
- hr_qp->qpn & (hr_dev->caps.num_qps - 1), hr_qp);
- spin_unlock_irq(&qp_table->lock);
- if (ret) {
- dev_err(dev, "QPC radix_tree_insert failed\n");
+ ret = hns_roce_gsi_qp_alloc(hr_dev, qpn, hr_qp);
+ if (ret)
goto err_put_sccc;
- }
-
- atomic_set(&hr_qp->refcount, 1);
- init_completion(&hr_qp->free);
return 0;
void hns_roce_qp_remove(struct hns_roce_dev *hr_dev, struct hns_roce_qp *hr_qp)
{
- struct hns_roce_qp_table *qp_table = &hr_dev->qp_table;
+ struct xarray *xa = &hr_dev->qp_table_xa;
unsigned long flags;
- spin_lock_irqsave(&qp_table->lock, flags);
- radix_tree_delete(&hr_dev->qp_table_tree,
- hr_qp->qpn & (hr_dev->caps.num_qps - 1));
- spin_unlock_irqrestore(&qp_table->lock, flags);
+ xa_lock_irqsave(xa, flags);
+ __xa_erase(xa, hr_qp->qpn & (hr_dev->caps.num_qps - 1));
+ xa_unlock_irqrestore(xa, flags);
}
EXPORT_SYMBOL_GPL(hns_roce_qp_remove);
int ret;
mutex_init(&qp_table->scc_mutex);
- spin_lock_init(&qp_table->lock);
- INIT_RADIX_TREE(&hr_dev->qp_table_tree, GFP_ATOMIC);
+ xa_init(&hr_dev->qp_table_xa);
/* In hw v1, a port include two SQP, six ports total 12 */
if (hr_dev->caps.max_sq_sg <= 2)
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+// Copyright (c) 2019 Hisilicon Limited.
+
+#include <rdma/rdma_cm.h>
+#include <rdma/restrack.h>
+#include <uapi/rdma/rdma_netlink.h>
+#include "hnae3.h"
+#include "hns_roce_common.h"
+#include "hns_roce_device.h"
+#include "hns_roce_hw_v2.h"
+
+static int hns_roce_fill_cq(struct sk_buff *msg,
+ struct hns_roce_v2_cq_context *context)
+{
+ if (rdma_nl_put_driver_u32(msg, "state",
+ roce_get_field(context->byte_4_pg_ceqn,
+ V2_CQC_BYTE_4_ARM_ST_M,
+ V2_CQC_BYTE_4_ARM_ST_S)))
+ goto err;
+
+ if (rdma_nl_put_driver_u32(msg, "ceqn",
+ roce_get_field(context->byte_4_pg_ceqn,
+ V2_CQC_BYTE_4_CEQN_M,
+ V2_CQC_BYTE_4_CEQN_S)))
+ goto err;
+
+ if (rdma_nl_put_driver_u32(msg, "cqn",
+ roce_get_field(context->byte_8_cqn,
+ V2_CQC_BYTE_8_CQN_M,
+ V2_CQC_BYTE_8_CQN_S)))
+ goto err;
+
+ if (rdma_nl_put_driver_u32(msg, "hopnum",
+ roce_get_field(context->byte_16_hop_addr,
+ V2_CQC_BYTE_16_CQE_HOP_NUM_M,
+ V2_CQC_BYTE_16_CQE_HOP_NUM_S)))
+ goto err;
+
+ if (rdma_nl_put_driver_u32(
+ msg, "pi",
+ roce_get_field(context->byte_28_cq_pi,
+ V2_CQC_BYTE_28_CQ_PRODUCER_IDX_M,
+ V2_CQC_BYTE_28_CQ_PRODUCER_IDX_S)))
+ goto err;
+
+ if (rdma_nl_put_driver_u32(
+ msg, "ci",
+ roce_get_field(context->byte_32_cq_ci,
+ V2_CQC_BYTE_32_CQ_CONSUMER_IDX_M,
+ V2_CQC_BYTE_32_CQ_CONSUMER_IDX_S)))
+ goto err;
+
+ if (rdma_nl_put_driver_u32(
+ msg, "coalesce",
+ roce_get_field(context->byte_56_cqe_period_maxcnt,
+ V2_CQC_BYTE_56_CQ_MAX_CNT_M,
+ V2_CQC_BYTE_56_CQ_MAX_CNT_S)))
+ goto err;
+
+ if (rdma_nl_put_driver_u32(
+ msg, "period",
+ roce_get_field(context->byte_56_cqe_period_maxcnt,
+ V2_CQC_BYTE_56_CQ_PERIOD_M,
+ V2_CQC_BYTE_56_CQ_PERIOD_S)))
+ goto err;
+
+ if (rdma_nl_put_driver_u32(msg, "cnt",
+ roce_get_field(context->byte_52_cqe_cnt,
+ V2_CQC_BYTE_52_CQE_CNT_M,
+ V2_CQC_BYTE_52_CQE_CNT_S)))
+ goto err;
+
+ return 0;
+
+err:
+ return -EMSGSIZE;
+}
+
+static int hns_roce_fill_res_cq_entry(struct sk_buff *msg,
+ struct rdma_restrack_entry *res)
+{
+ struct ib_cq *ib_cq = container_of(res, struct ib_cq, res);
+ struct hns_roce_dev *hr_dev = to_hr_dev(ib_cq->device);
+ struct hns_roce_cq *hr_cq = to_hr_cq(ib_cq);
+ struct hns_roce_v2_cq_context *context;
+ struct nlattr *table_attr;
+ int ret;
+
+ if (!hr_dev->dfx->query_cqc_info)
+ return -EINVAL;
+
+ context = kzalloc(sizeof(struct hns_roce_v2_cq_context), GFP_KERNEL);
+ if (!context)
+ return -ENOMEM;
+
+ ret = hr_dev->dfx->query_cqc_info(hr_dev, hr_cq->cqn, (int *)context);
+ if (ret)
+ goto err;
+
+ table_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_DRIVER);
+ if (!table_attr)
+ goto err;
+
+ if (hns_roce_fill_cq(msg, context))
+ goto err_cancel_table;
+
+ nla_nest_end(msg, table_attr);
+ kfree(context);
+
+ return 0;
+
+err_cancel_table:
+ nla_nest_cancel(msg, table_attr);
+err:
+ kfree(context);
+ return -EMSGSIZE;
+}
+
+int hns_roce_fill_res_entry(struct sk_buff *msg,
+ struct rdma_restrack_entry *res)
+{
+ if (res->type == RDMA_RESTRACK_CQ)
+ return hns_roce_fill_res_cq_entry(msg, res);
+
+ return 0;
+}
return 0;
}
-struct ib_srq *hns_roce_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata)
+int hns_roce_create_srq(struct ib_srq *ib_srq,
+ struct ib_srq_init_attr *srq_init_attr,
+ struct ib_udata *udata)
{
- struct hns_roce_dev *hr_dev = to_hr_dev(pd->device);
+ struct hns_roce_dev *hr_dev = to_hr_dev(ib_srq->device);
struct hns_roce_ib_create_srq_resp resp = {};
- struct hns_roce_srq *srq;
+ struct hns_roce_srq *srq = to_hr_srq(ib_srq);
int srq_desc_size;
int srq_buf_size;
u32 page_shift;
/* Check the actual SRQ wqe and SRQ sge num */
if (srq_init_attr->attr.max_wr >= hr_dev->caps.max_srq_wrs ||
srq_init_attr->attr.max_sge > hr_dev->caps.max_srq_sges)
- return ERR_PTR(-EINVAL);
-
- srq = kzalloc(sizeof(*srq), GFP_KERNEL);
- if (!srq)
- return ERR_PTR(-ENOMEM);
+ return -EINVAL;
mutex_init(&srq->mutex);
spin_lock_init(&srq->lock);
if (udata) {
struct hns_roce_ib_create_srq ucmd;
- if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd))) {
- ret = -EFAULT;
- goto err_srq;
- }
+ if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd)))
+ return -EFAULT;
srq->umem =
ib_umem_get(udata, ucmd.buf_addr, srq_buf_size, 0, 0);
- if (IS_ERR(srq->umem)) {
- ret = PTR_ERR(srq->umem);
- goto err_srq;
- }
+ if (IS_ERR(srq->umem))
+ return PTR_ERR(srq->umem);
if (hr_dev->caps.srqwqe_buf_pg_sz) {
npages = (ib_umem_page_count(srq->umem) +
} else {
page_shift = PAGE_SHIFT + hr_dev->caps.srqwqe_buf_pg_sz;
if (hns_roce_buf_alloc(hr_dev, srq_buf_size,
- (1 << page_shift) * 2,
- &srq->buf, page_shift)) {
- ret = -ENOMEM;
- goto err_srq;
- }
+ (1 << page_shift) * 2, &srq->buf,
+ page_shift))
+ return -ENOMEM;
srq->head = 0;
srq->tail = srq->max - 1;
goto err_srq_mtt;
page_shift = PAGE_SHIFT + hr_dev->caps.idx_buf_pg_sz;
- ret = hns_roce_create_idx_que(pd, srq, page_shift);
+ ret = hns_roce_create_idx_que(ib_srq->pd, srq, page_shift);
if (ret) {
dev_err(hr_dev->dev, "Create idx queue fail(%d)!\n",
ret);
srq->db_reg_l = hr_dev->reg_base + SRQ_DB_REG;
- ret = hns_roce_srq_alloc(hr_dev, to_hr_pd(pd)->pdn, cqn, 0,
+ ret = hns_roce_srq_alloc(hr_dev, to_hr_pd(ib_srq->pd)->pdn, cqn, 0,
&srq->mtt, 0, srq);
if (ret)
goto err_wrid;
}
}
- return &srq->ibsrq;
+ return 0;
err_srqc_alloc:
hns_roce_srq_free(hr_dev, srq);
else
hns_roce_buf_free(hr_dev, srq_buf_size, &srq->buf);
-err_srq:
- kfree(srq);
- return ERR_PTR(ret);
+ return ret;
}
-int hns_roce_destroy_srq(struct ib_srq *ibsrq)
+void hns_roce_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)
{
struct hns_roce_dev *hr_dev = to_hr_dev(ibsrq->device);
struct hns_roce_srq *srq = to_hr_srq(ibsrq);
hns_roce_buf_free(hr_dev, srq->max << srq->wqe_shift,
&srq->buf);
}
-
- kfree(srq);
-
- return 0;
}
int hns_roce_init_srq_table(struct hns_roce_dev *hr_dev)
void i40iw_request_reset(struct i40iw_device *iwdev);
void i40iw_destroy_rdma_device(struct i40iw_ib_device *iwibdev);
-void i40iw_setup_cm_core(struct i40iw_device *iwdev);
+int i40iw_setup_cm_core(struct i40iw_device *iwdev);
void i40iw_cleanup_cm_core(struct i40iw_cm_core *cm_core);
void i40iw_process_ceq(struct i40iw_device *, struct i40iw_ceq *iwceq);
void i40iw_process_aeq(struct i40iw_device *);
* core
* @iwdev: iwarp device structure
*/
-void i40iw_setup_cm_core(struct i40iw_device *iwdev)
+int i40iw_setup_cm_core(struct i40iw_device *iwdev)
{
struct i40iw_cm_core *cm_core = &iwdev->cm_core;
cm_core->event_wq = alloc_ordered_workqueue("iwewq",
WQ_MEM_RECLAIM);
+ if (!cm_core->event_wq)
+ goto error;
cm_core->disconn_wq = alloc_ordered_workqueue("iwdwq",
WQ_MEM_RECLAIM);
+ if (!cm_core->disconn_wq)
+ goto error;
+
+ return 0;
+error:
+ i40iw_cleanup_cm_core(&iwdev->cm_core);
+
+ return -ENOMEM;
}
/**
del_timer_sync(&cm_core->tcp_timer);
spin_unlock_irqrestore(&cm_core->ht_lock, flags);
- destroy_workqueue(cm_core->event_wq);
- destroy_workqueue(cm_core->disconn_wq);
+ if (cm_core->event_wq)
+ destroy_workqueue(cm_core->event_wq);
+ if (cm_core->disconn_wq)
+ destroy_workqueue(cm_core->disconn_wq);
}
/**
/* Need to free the Last Streaming Mode Message */
if (iwqp->ietf_mem.va) {
if (iwqp->lsmm_mr)
- iwibdev->ibdev.ops.dereg_mr(iwqp->lsmm_mr);
+ iwibdev->ibdev.ops.dereg_mr(iwqp->lsmm_mr,
+ NULL);
i40iw_free_dma_mem(iwdev->sc_dev.hw, &iwqp->ietf_mem);
}
}
iwdev = &hdl->device;
iwdev->hdl = hdl;
dev = &iwdev->sc_dev;
- i40iw_setup_cm_core(iwdev);
+ if (i40iw_setup_cm_core(iwdev)) {
+ kfree(iwdev->hdl);
+ return -ENOMEM;
+ }
dev->back_dev = (void *)iwdev;
iwdev->ldev = &hdl->ldev;
/**
* i40iw_alloc_pd - allocate protection domain
* @pd: PD pointer
- * @context: user context created during alloc
* @udata: user data
*/
-static int i40iw_alloc_pd(struct ib_pd *pd, struct ib_ucontext *context,
- struct ib_udata *udata)
+static int i40iw_alloc_pd(struct ib_pd *pd, struct ib_udata *udata)
{
struct i40iw_pd *iwpd = to_iwpd(pd);
struct i40iw_device *iwdev = to_iwdev(pd->device);
struct i40iw_sc_dev *dev = &iwdev->sc_dev;
struct i40iw_alloc_pd_resp uresp;
struct i40iw_sc_pd *sc_pd;
- struct i40iw_ucontext *ucontext;
u32 pd_id = 0;
int err;
sc_pd = &iwpd->sc_pd;
- if (context) {
- ucontext = to_ucontext(context);
+ if (udata) {
+ struct i40iw_ucontext *ucontext = rdma_udata_to_drv_context(
+ udata, struct i40iw_ucontext, ibucontext);
dev->iw_pd_ops->pd_init(dev, sc_pd, pd_id, ucontext->abi_ver);
memset(&uresp, 0, sizeof(uresp));
uresp.pd_id = pd_id;
/**
* i40iw_dealloc_pd - deallocate pd
* @ibpd: ptr of pd to be deallocated
+ * @udata: user data or null for kernel object
*/
-static void i40iw_dealloc_pd(struct ib_pd *ibpd)
+static void i40iw_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct i40iw_pd *iwpd = to_iwpd(ibpd);
struct i40iw_device *iwdev = to_iwdev(ibpd->device);
* i40iw_destroy_qp - destroy qp
* @ibqp: qp's ib pointer also to get to device's qp address
*/
-static int i40iw_destroy_qp(struct ib_qp *ibqp)
+static int i40iw_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
{
struct i40iw_qp *iwqp = to_iwqp(ibqp);
err_code = ib_copy_to_udata(udata, &uresp, sizeof(uresp));
if (err_code) {
i40iw_pr_err("copy_to_udata failed\n");
- i40iw_destroy_qp(&iwqp->ibqp);
- /* let the completion of the qp destroy free the qp */
+ i40iw_destroy_qp(&iwqp->ibqp, udata);
+ /* let the completion of the qp destroy free the qp */
return ERR_PTR(err_code);
}
}
/**
* i40iw_destroy_cq - destroy cq
* @ib_cq: cq pointer
+ * @udata: user data or NULL for kernel object
*/
-static int i40iw_destroy_cq(struct ib_cq *ib_cq)
+static int i40iw_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
{
struct i40iw_cq *iwcq;
struct i40iw_device *iwdev;
* i40iw_create_cq - create cq
* @ibdev: device pointer from stack
* @attr: attributes for cq
- * @context: user context created during alloc
* @udata: user data
*/
static struct ib_cq *i40iw_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
struct i40iw_device *iwdev = to_iwdev(ibdev);
info.ceq_id_valid = true;
info.ceqe_mask = 1;
info.type = I40IW_CQ_TYPE_IWARP;
- if (context) {
- struct i40iw_ucontext *ucontext;
+ if (udata) {
+ struct i40iw_ucontext *ucontext = rdma_udata_to_drv_context(
+ udata, struct i40iw_ucontext, ibucontext);
struct i40iw_create_cq_req req;
struct i40iw_cq_mr *cqmr;
memset(&req, 0, sizeof(req));
iwcq->user_mode = true;
- ucontext = to_ucontext(context);
if (ib_copy_from_udata(&req, udata, sizeof(struct i40iw_create_cq_req))) {
err_code = -EFAULT;
goto cq_free_resources;
goto cq_free_resources;
}
- if (context) {
+ if (udata) {
struct i40iw_create_cq_resp resp;
memset(&resp, 0, sizeof(resp));
struct i40iw_pbl *iwpbl = &iwmr->iwpbl;
struct i40iw_pble_alloc *palloc = &iwpbl->pble_alloc;
struct i40iw_pble_info *pinfo;
- struct sg_dma_page_iter sg_iter;
- u64 pg_addr = 0;
+ struct ib_block_iter biter;
u32 idx = 0;
- bool first_pg = true;
pinfo = (level == I40IW_LEVEL_1) ? NULL : palloc->level2.leaf;
if (iwmr->type == IW_MEMREG_TYPE_QP)
iwpbl->qp_mr.sq_page = sg_page(region->sg_head.sgl);
- for_each_sg_dma_page (region->sg_head.sgl, &sg_iter, region->nmap, 0) {
- pg_addr = sg_page_iter_dma_address(&sg_iter);
- if (first_pg)
- *pbl = cpu_to_le64(pg_addr & iwmr->page_msk);
- else if (!(pg_addr & ~iwmr->page_msk))
- *pbl = cpu_to_le64(pg_addr);
- else
- continue;
-
- first_pg = false;
+ rdma_for_each_block(region->sg_head.sgl, &biter, region->nmap,
+ iwmr->page_size) {
+ *pbl = rdma_block_iter_dma_address(&biter);
pbl = i40iw_next_pbl_addr(pbl, &pinfo, &idx);
}
}
-/**
- * i40iw_set_hugetlb_params - set MR pg size and mask to huge pg values.
- * @addr: virtual address
- * @iwmr: mr pointer for this memory registration
- */
-static void i40iw_set_hugetlb_values(u64 addr, struct i40iw_mr *iwmr)
-{
- struct vm_area_struct *vma;
- struct hstate *h;
-
- down_read(¤t->mm->mmap_sem);
- vma = find_vma(current->mm, addr);
- if (vma && is_vm_hugetlb_page(vma)) {
- h = hstate_vma(vma);
- if (huge_page_size(h) == 0x200000) {
- iwmr->page_size = huge_page_size(h);
- iwmr->page_msk = huge_page_mask(h);
- }
- }
- up_read(¤t->mm->mmap_sem);
-}
-
/**
* i40iw_check_mem_contiguous - check if pbls stored in arr are contiguous
* @arr: lvl1 pbl array
* @pd: ibpd pointer
* @mr_type: memory for stag registrion
* @max_num_sg: man number of pages
+ * @udata: user data or NULL for kernel objects
*/
-static struct ib_mr *i40iw_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg)
+static struct ib_mr *i40iw_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
struct i40iw_pd *iwpd = to_iwpd(pd);
struct i40iw_device *iwdev = to_iwdev(pd->device);
iwmr->ibmr.device = pd->device;
iwmr->page_size = PAGE_SIZE;
- iwmr->page_msk = PAGE_MASK;
-
- if (region->hugetlb && (req.reg_type == IW_MEMREG_TYPE_MEM))
- i40iw_set_hugetlb_values(start, iwmr);
+ if (req.reg_type == IW_MEMREG_TYPE_MEM)
+ iwmr->page_size = ib_umem_find_best_pgsz(region, SZ_4K | SZ_2M,
+ virt);
region_length = region->length + (start & (iwmr->page_size - 1));
pg_shift = ffs(iwmr->page_size) - 1;
* i40iw_dereg_mr - deregister mr
* @ib_mr: mr ptr for dereg
*/
-static int i40iw_dereg_mr(struct ib_mr *ib_mr)
+static int i40iw_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
{
struct ib_pd *ibpd = ib_mr->pd;
struct i40iw_pd *iwpd = to_iwpd(ibpd);
if (iwmr->type != IW_MEMREG_TYPE_MEM) {
/* region is released. only test for userness. */
if (iwmr->region) {
- struct i40iw_ucontext *ucontext;
+ struct i40iw_ucontext *ucontext =
+ rdma_udata_to_drv_context(
+ udata,
+ struct i40iw_ucontext,
+ ibucontext);
- ucontext = to_ucontext(ibpd->uobject->context);
i40iw_del_memlist(iwmr, ucontext);
}
if (iwpbl->pbl_allocated && iwmr->type != IW_MEMREG_TYPE_QP)
.get_dma_mr = i40iw_get_dma_mr,
.get_hw_stats = i40iw_get_hw_stats,
.get_port_immutable = i40iw_port_immutable,
+ .iw_accept = i40iw_accept,
+ .iw_add_ref = i40iw_add_ref,
+ .iw_connect = i40iw_connect,
+ .iw_create_listen = i40iw_create_listen,
+ .iw_destroy_listen = i40iw_destroy_listen,
+ .iw_get_qp = i40iw_get_qp,
+ .iw_reject = i40iw_reject,
+ .iw_rem_ref = i40iw_rem_ref,
.map_mr_sg = i40iw_map_mr_sg,
.mmap = i40iw_mmap,
.modify_qp = i40iw_modify_qp,
iwibdev->ibdev.phys_port_cnt = 1;
iwibdev->ibdev.num_comp_vectors = iwdev->ceqs_count;
iwibdev->ibdev.dev.parent = &pcidev->dev;
- iwibdev->ibdev.iwcm = kzalloc(sizeof(*iwibdev->ibdev.iwcm), GFP_KERNEL);
- if (!iwibdev->ibdev.iwcm) {
- ib_dealloc_device(&iwibdev->ibdev);
- return NULL;
- }
-
- iwibdev->ibdev.iwcm->add_ref = i40iw_add_ref;
- iwibdev->ibdev.iwcm->rem_ref = i40iw_rem_ref;
- iwibdev->ibdev.iwcm->get_qp = i40iw_get_qp;
- iwibdev->ibdev.iwcm->connect = i40iw_connect;
- iwibdev->ibdev.iwcm->accept = i40iw_accept;
- iwibdev->ibdev.iwcm->reject = i40iw_reject;
- iwibdev->ibdev.iwcm->create_listen = i40iw_create_listen;
- iwibdev->ibdev.iwcm->destroy_listen = i40iw_destroy_listen;
- memcpy(iwibdev->ibdev.iwcm->ifname, netdev->name,
- sizeof(iwibdev->ibdev.iwcm->ifname));
+ memcpy(iwibdev->ibdev.iw_ifname, netdev->name,
+ sizeof(iwibdev->ibdev.iw_ifname));
ib_set_device_ops(&iwibdev->ibdev, &i40iw_dev_ops);
return iwibdev;
return;
ib_unregister_device(&iwibdev->ibdev);
- kfree(iwibdev->ibdev.iwcm);
- iwibdev->ibdev.iwcm = NULL;
wait_event_timeout(iwibdev->iwdev->close_wq,
!atomic64_read(&iwibdev->iwdev->use_count),
I40IW_EVENT_TIMEOUT);
return 0;
error:
- kfree(iwdev->iwibdev->ibdev.iwcm);
- iwdev->iwibdev->ibdev.iwcm = NULL;
ib_dealloc_device(&iwdev->iwibdev->ibdev);
return ret;
}
struct ib_umem *region;
u16 type;
u32 page_cnt;
- u32 page_size;
- u64 page_msk;
+ u64 page_size;
u32 npages;
u32 stag;
u64 length;
#include "mlx4_ib.h"
-static struct ib_ah *create_ib_ah(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
- struct mlx4_ib_ah *ah)
+static void create_ib_ah(struct ib_ah *ib_ah, struct rdma_ah_attr *ah_attr)
{
- struct mlx4_dev *dev = to_mdev(pd->device)->dev;
+ struct mlx4_ib_ah *ah = to_mah(ib_ah);
+ struct mlx4_dev *dev = to_mdev(ib_ah->device)->dev;
- ah->av.ib.port_pd = cpu_to_be32(to_mpd(pd)->pdn |
+ ah->av.ib.port_pd = cpu_to_be32(to_mpd(ib_ah->pd)->pdn |
(rdma_ah_get_port_num(ah_attr) << 24));
ah->av.ib.g_slid = rdma_ah_get_path_bits(ah_attr);
ah->av.ib.sl_tclass_flowlabel =
--static_rate;
ah->av.ib.stat_rate = static_rate;
}
-
- return &ah->ibah;
}
-static struct ib_ah *create_iboe_ah(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
- struct mlx4_ib_ah *ah)
+static int create_iboe_ah(struct ib_ah *ib_ah, struct rdma_ah_attr *ah_attr)
{
- struct mlx4_ib_dev *ibdev = to_mdev(pd->device);
+ struct mlx4_ib_dev *ibdev = to_mdev(ib_ah->device);
+ struct mlx4_ib_ah *ah = to_mah(ib_ah);
const struct ib_gid_attr *gid_attr;
struct mlx4_dev *dev = ibdev->dev;
int is_mcast = 0;
*/
gid_attr = ah_attr->grh.sgid_attr;
if (gid_attr) {
- if (is_vlan_dev(gid_attr->ndev))
- vlan_tag = vlan_dev_vlan_id(gid_attr->ndev);
- memcpy(ah->av.eth.s_mac, gid_attr->ndev->dev_addr, ETH_ALEN);
+ ret = rdma_read_gid_l2_fields(gid_attr, &vlan_tag,
+ &ah->av.eth.s_mac[0]);
+ if (ret)
+ return ret;
+
ret = mlx4_ib_gid_index_to_real_index(ibdev, gid_attr);
if (ret < 0)
- return ERR_PTR(ret);
+ return ret;
ah->av.eth.gid_index = ret;
} else {
/* mlx4_ib_create_ah_slave fills in the s_mac and the vlan */
if (vlan_tag < 0x1000)
vlan_tag |= (rdma_ah_get_sl(ah_attr) & 7) << 13;
- ah->av.eth.port_pd = cpu_to_be32(to_mpd(pd)->pdn |
+ ah->av.eth.port_pd = cpu_to_be32(to_mpd(ib_ah->pd)->pdn |
(rdma_ah_get_port_num(ah_attr) << 24));
ah->av.eth.vlan = cpu_to_be16(vlan_tag);
ah->av.eth.hop_limit = grh->hop_limit;
memcpy(ah->av.eth.dgid, grh->dgid.raw, 16);
ah->av.eth.sl_tclass_flowlabel |= cpu_to_be32(rdma_ah_get_sl(ah_attr)
<< 29);
- return &ah->ibah;
+ return 0;
}
-struct ib_ah *mlx4_ib_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
- u32 flags, struct ib_udata *udata)
+int mlx4_ib_create_ah(struct ib_ah *ib_ah, struct rdma_ah_attr *ah_attr,
+ u32 flags, struct ib_udata *udata)
{
- struct mlx4_ib_ah *ah;
- struct ib_ah *ret;
-
- ah = kzalloc(sizeof *ah, GFP_ATOMIC);
- if (!ah)
- return ERR_PTR(-ENOMEM);
-
if (ah_attr->type == RDMA_AH_ATTR_TYPE_ROCE) {
- if (!(rdma_ah_get_ah_flags(ah_attr) & IB_AH_GRH)) {
- ret = ERR_PTR(-EINVAL);
- } else {
- /*
- * TBD: need to handle the case when we get
- * called in an atomic context and there we
- * might sleep. We don't expect this
- * currently since we're working with link
- * local addresses which we can translate
- * without going to sleep.
- */
- ret = create_iboe_ah(pd, ah_attr, ah);
- }
-
- if (IS_ERR(ret))
- kfree(ah);
+ if (!(rdma_ah_get_ah_flags(ah_attr) & IB_AH_GRH))
+ return -EINVAL;
+ /*
+ * TBD: need to handle the case when we get
+ * called in an atomic context and there we
+ * might sleep. We don't expect this
+ * currently since we're working with link
+ * local addresses which we can translate
+ * without going to sleep.
+ */
+ return create_iboe_ah(ib_ah, ah_attr);
+ }
- return ret;
- } else
- return create_ib_ah(pd, ah_attr, ah); /* never fails */
+ create_ib_ah(ib_ah, ah_attr);
+ return 0;
}
-/* AH's created via this call must be free'd by mlx4_ib_destroy_ah. */
-struct ib_ah *mlx4_ib_create_ah_slave(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
- int slave_sgid_index, u8 *s_mac,
- u16 vlan_tag)
+int mlx4_ib_create_ah_slave(struct ib_ah *ah, struct rdma_ah_attr *ah_attr,
+ int slave_sgid_index, u8 *s_mac, u16 vlan_tag)
{
struct rdma_ah_attr slave_attr = *ah_attr;
- struct mlx4_ib_ah *mah;
- struct ib_ah *ah;
+ struct mlx4_ib_ah *mah = to_mah(ah);
+ int ret;
slave_attr.grh.sgid_attr = NULL;
slave_attr.grh.sgid_index = slave_sgid_index;
- ah = mlx4_ib_create_ah(pd, &slave_attr, 0, NULL);
- if (IS_ERR(ah))
- return ah;
+ ret = mlx4_ib_create_ah(ah, &slave_attr, 0, NULL);
+ if (ret)
+ return ret;
- ah->device = pd->device;
- ah->pd = pd;
ah->type = ah_attr->type;
- mah = to_mah(ah);
/* get rid of force-loopback bit */
mah->av.ib.port_pd &= cpu_to_be32(0x7FFFFFFF);
vlan_tag |= (rdma_ah_get_sl(ah_attr) & 7) << 13;
mah->av.eth.vlan = cpu_to_be16(vlan_tag);
- return ah;
+ return 0;
}
int mlx4_ib_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr)
return 0;
}
-int mlx4_ib_destroy_ah(struct ib_ah *ah, u32 flags)
+void mlx4_ib_destroy_ah(struct ib_ah *ah, u32 flags)
{
- kfree(to_mah(ah));
- return 0;
+ return;
}
{
struct delayed_work *delay = to_delayed_work(work);
struct id_map_entry *ent = container_of(delay, struct id_map_entry, timeout);
- struct id_map_entry *db_ent, *found_ent;
+ struct id_map_entry *found_ent;
struct mlx4_ib_dev *dev = ent->dev;
struct mlx4_ib_sriov *sriov = &dev->sriov;
struct rb_root *sl_id_map = &sriov->sl_id_map;
- int pv_id = (int) ent->pv_cm_id;
spin_lock(&sriov->id_map_lock);
- db_ent = (struct id_map_entry *)idr_find(&sriov->pv_id_table, pv_id);
- if (!db_ent)
+ if (!xa_erase(&sriov->pv_id_table, ent->pv_cm_id))
goto out;
found_ent = id_map_find_by_sl_id(&dev->ib_dev, ent->slave_id, ent->sl_cm_id);
if (found_ent && found_ent == ent)
rb_erase(&found_ent->node, sl_id_map);
- idr_remove(&sriov->pv_id_table, pv_id);
out:
list_del(&ent->list);
struct id_map_entry *ent, *found_ent;
spin_lock(&sriov->id_map_lock);
- ent = (struct id_map_entry *)idr_find(&sriov->pv_id_table, pv_cm_id);
+ ent = xa_erase(&sriov->pv_id_table, pv_cm_id);
if (!ent)
goto out;
found_ent = id_map_find_by_sl_id(ibdev, ent->slave_id, ent->sl_cm_id);
if (found_ent && found_ent == ent)
rb_erase(&found_ent->node, sl_id_map);
- idr_remove(&sriov->pv_id_table, pv_cm_id);
out:
spin_unlock(&sriov->id_map_lock);
}
ent->dev = to_mdev(ibdev);
INIT_DELAYED_WORK(&ent->timeout, id_map_ent_timeout);
- idr_preload(GFP_KERNEL);
- spin_lock(&to_mdev(ibdev)->sriov.id_map_lock);
-
- ret = idr_alloc_cyclic(&sriov->pv_id_table, ent, 0, 0, GFP_NOWAIT);
+ ret = xa_alloc_cyclic(&sriov->pv_id_table, &ent->pv_cm_id, ent,
+ xa_limit_32b, &sriov->pv_id_next, GFP_KERNEL);
if (ret >= 0) {
- ent->pv_cm_id = (u32)ret;
+ spin_lock(&sriov->id_map_lock);
sl_id_map_add(ibdev, ent);
list_add_tail(&ent->list, &sriov->cm_list);
- }
-
- spin_unlock(&sriov->id_map_lock);
- idr_preload_end();
-
- if (ret >= 0)
+ spin_unlock(&sriov->id_map_lock);
return ent;
+ }
/*error flow*/
kfree(ent);
- mlx4_ib_warn(ibdev, "No more space in the idr (err:0x%x)\n", ret);
+ mlx4_ib_warn(ibdev, "Allocation failed (err:0x%x)\n", ret);
return ERR_PTR(-ENOMEM);
}
if (ent)
*pv_cm_id = (int) ent->pv_cm_id;
} else
- ent = (struct id_map_entry *)idr_find(&sriov->pv_id_table, *pv_cm_id);
+ ent = xa_load(&sriov->pv_id_table, *pv_cm_id);
spin_unlock(&sriov->id_map_lock);
return ent;
spin_lock_init(&dev->sriov.id_map_lock);
INIT_LIST_HEAD(&dev->sriov.cm_list);
dev->sriov.sl_id_map = RB_ROOT;
- idr_init(&dev->sriov.pv_id_table);
+ xa_init_flags(&dev->sriov.pv_id_table, XA_FLAGS_ALLOC);
}
/* slave = -1 ==> all slaves */
struct id_map_entry, node);
rb_erase(&ent->node, sl_id_map);
- idr_remove(&sriov->pv_id_table, (int) ent->pv_cm_id);
+ xa_erase(&sriov->pv_id_table, ent->pv_cm_id);
}
list_splice_init(&dev->sriov.cm_list, &lh);
} else {
/* remove those nodes from databases */
list_for_each_entry_safe(map, tmp_map, &lh, list) {
rb_erase(&map->node, sl_id_map);
- idr_remove(&sriov->pv_id_table, (int) map->pv_cm_id);
+ xa_erase(&sriov->pv_id_table, map->pv_cm_id);
}
/* add remaining nodes from cm_list */
#include "mlx4_ib.h"
#include <rdma/mlx4-abi.h>
+#include <rdma/uverbs_ioctl.h>
static void mlx4_ib_cq_comp(struct mlx4_cq *cq)
{
#define CQ_CREATE_FLAGS_SUPPORTED IB_UVERBS_CQ_FLAGS_TIMESTAMP_COMPLETION
struct ib_cq *mlx4_ib_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
int entries = attr->cqe;
struct mlx4_uar *uar;
void *buf_addr;
int err;
+ struct mlx4_ib_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct mlx4_ib_ucontext, ibucontext);
if (entries < 1 || entries > dev->dev->caps.max_cqes)
return ERR_PTR(-EINVAL);
INIT_LIST_HEAD(&cq->send_qp_list);
INIT_LIST_HEAD(&cq->recv_qp_list);
- if (context) {
+ if (udata) {
struct mlx4_ib_create_cq ucmd;
if (ib_copy_from_udata(&ucmd, udata, sizeof ucmd)) {
if (err)
goto err_cq;
- err = mlx4_ib_db_map_user(to_mucontext(context), udata,
- ucmd.db_addr, &cq->db);
+ err = mlx4_ib_db_map_user(udata, ucmd.db_addr, &cq->db);
if (err)
goto err_mtt;
- uar = &to_mucontext(context)->uar;
+ uar = &context->uar;
cq->mcq.usage = MLX4_RES_USAGE_USER_VERBS;
} else {
err = mlx4_db_alloc(dev->dev, &cq->db, 1);
if (dev->eq_table)
vector = dev->eq_table[vector % ibdev->num_comp_vectors];
- err = mlx4_cq_alloc(dev->dev, entries, &cq->buf.mtt, uar,
- cq->db.dma, &cq->mcq, vector, 0,
+ err = mlx4_cq_alloc(dev->dev, entries, &cq->buf.mtt, uar, cq->db.dma,
+ &cq->mcq, vector, 0,
!!(cq->create_flags &
IB_UVERBS_CQ_FLAGS_TIMESTAMP_COMPLETION),
- buf_addr, !!context);
+ buf_addr, !!udata);
if (err)
goto err_dbmap;
- if (context)
+ if (udata)
cq->mcq.tasklet_ctx.comp = mlx4_ib_cq_comp;
else
cq->mcq.comp = mlx4_ib_cq_comp;
cq->mcq.event = mlx4_ib_cq_event;
- if (context)
+ if (udata)
if (ib_copy_to_udata(udata, &cq->mcq.cqn, sizeof (__u32))) {
err = -EFAULT;
goto err_cq_free;
mlx4_cq_free(dev->dev, &cq->mcq);
err_dbmap:
- if (context)
- mlx4_ib_db_unmap_user(to_mucontext(context), &cq->db);
+ if (udata)
+ mlx4_ib_db_unmap_user(context, &cq->db);
err_mtt:
mlx4_mtt_cleanup(dev->dev, &cq->buf.mtt);
- if (context)
+ if (udata)
ib_umem_release(cq->umem);
else
mlx4_ib_free_cq_buf(dev, &cq->buf, cq->ibcq.cqe);
err_db:
- if (!context)
+ if (!udata)
mlx4_db_free(dev->dev, &cq->db);
err_cq:
return err;
}
-int mlx4_ib_destroy_cq(struct ib_cq *cq)
+int mlx4_ib_destroy_cq(struct ib_cq *cq, struct ib_udata *udata)
{
struct mlx4_ib_dev *dev = to_mdev(cq->device);
struct mlx4_ib_cq *mcq = to_mcq(cq);
mlx4_cq_free(dev->dev, &mcq->mcq);
mlx4_mtt_cleanup(dev->dev, &mcq->buf.mtt);
- if (cq->uobject) {
- mlx4_ib_db_unmap_user(to_mucontext(cq->uobject->context), &mcq->db);
+ if (udata) {
+ mlx4_ib_db_unmap_user(
+ rdma_udata_to_drv_context(
+ udata,
+ struct mlx4_ib_ucontext,
+ ibucontext),
+ &mcq->db);
ib_umem_release(mcq->umem);
} else {
mlx4_ib_free_cq_buf(dev, &mcq->buf, cq->cqe);
*/
#include <linux/slab.h>
+#include <rdma/uverbs_ioctl.h>
#include "mlx4_ib.h"
int refcnt;
};
-int mlx4_ib_db_map_user(struct mlx4_ib_ucontext *context,
- struct ib_udata *udata, unsigned long virt,
+int mlx4_ib_db_map_user(struct ib_udata *udata, unsigned long virt,
struct mlx4_db *db)
{
struct mlx4_ib_user_db_page *page;
int err = 0;
+ struct mlx4_ib_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct mlx4_ib_ucontext, ibucontext);
mutex_lock(&context->db_page_mutex);
struct ib_ah *ah;
struct ib_qp *send_qp = NULL;
unsigned wire_tx_ix = 0;
- int ret = 0;
u16 wire_pkey_ix;
int src_qpnum;
+ int ret;
sqp_ctx = dev->sriov.sqps[port-1];
send_qp = sqp->qp;
- /* create ah */
- ah = mlx4_ib_create_ah_slave(sqp_ctx->pd, attr,
- rdma_ah_retrieve_grh(attr)->sgid_index,
- s_mac, vlan_id);
- if (IS_ERR(ah))
+ ah = rdma_zalloc_drv_obj(sqp_ctx->pd->device, ib_ah);
+ if (!ah)
return -ENOMEM;
+
+ ah->device = sqp_ctx->pd->device;
+ ah->pd = sqp_ctx->pd;
+
+ /* create ah */
+ ret = mlx4_ib_create_ah_slave(ah, attr,
+ rdma_ah_retrieve_grh(attr)->sgid_index,
+ s_mac, vlan_id);
+ if (ret)
+ goto out;
+
spin_lock(&sqp->tx_lock);
if (sqp->tx_ix_head - sqp->tx_ix_tail >=
(MLX4_NUM_TUNNEL_BUFS - 1))
goto out;
sqp_mad = (struct mlx4_mad_snd_buf *) (sqp->tx_ring[wire_tx_ix].buf.addr);
- if (sqp->tx_ring[wire_tx_ix].ah)
- mlx4_ib_destroy_ah(sqp->tx_ring[wire_tx_ix].ah, 0);
+ kfree(sqp->tx_ring[wire_tx_ix].ah);
sqp->tx_ring[wire_tx_ix].ah = ah;
ib_dma_sync_single_for_cpu(&dev->ib_dev,
sqp->tx_ring[wire_tx_ix].buf.map,
spin_unlock(&sqp->tx_lock);
sqp->tx_ring[wire_tx_ix].ah = NULL;
out:
- mlx4_ib_destroy_ah(ah, 0);
+ kfree(ah);
return ret;
}
if (wc.status == IB_WC_SUCCESS) {
switch (wc.opcode) {
case IB_WC_SEND:
- mlx4_ib_destroy_ah(sqp->tx_ring[wc.wr_id &
- (MLX4_NUM_TUNNEL_BUFS - 1)].ah, 0);
+ kfree(sqp->tx_ring[wc.wr_id &
+ (MLX4_NUM_TUNNEL_BUFS - 1)].ah);
sqp->tx_ring[wc.wr_id & (MLX4_NUM_TUNNEL_BUFS - 1)].ah
= NULL;
spin_lock(&sqp->tx_lock);
" status = %d, wrid = 0x%llx\n",
ctx->slave, wc.status, wc.wr_id);
if (!MLX4_TUN_IS_RECV(wc.wr_id)) {
- mlx4_ib_destroy_ah(sqp->tx_ring[wc.wr_id &
- (MLX4_NUM_TUNNEL_BUFS - 1)].ah, 0);
+ kfree(sqp->tx_ring[wc.wr_id &
+ (MLX4_NUM_TUNNEL_BUFS - 1)].ah);
sqp->tx_ring[wc.wr_id & (MLX4_NUM_TUNNEL_BUFS - 1)].ah
= NULL;
spin_lock(&sqp->tx_lock);
}
}
-static int mlx4_ib_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata)
+static int mlx4_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct mlx4_ib_pd *pd = to_mpd(ibpd);
struct ib_device *ibdev = ibpd->device;
if (err)
return err;
- if (context && ib_copy_to_udata(udata, &pd->pdn, sizeof(__u32))) {
+ if (udata && ib_copy_to_udata(udata, &pd->pdn, sizeof(__u32))) {
mlx4_pd_free(to_mdev(ibdev)->dev, pd->pdn);
return -EFAULT;
}
return 0;
}
-static void mlx4_ib_dealloc_pd(struct ib_pd *pd)
+static void mlx4_ib_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata)
{
mlx4_pd_free(to_mdev(pd->device)->dev, to_mpd(pd)->pdn);
}
static struct ib_xrcd *mlx4_ib_alloc_xrcd(struct ib_device *ibdev,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
struct mlx4_ib_xrcd *xrcd;
return ERR_PTR(err);
}
-static int mlx4_ib_dealloc_xrcd(struct ib_xrcd *xrcd)
+static int mlx4_ib_dealloc_xrcd(struct ib_xrcd *xrcd, struct ib_udata *udata)
{
ib_destroy_cq(to_mxrcd(xrcd)->cq);
ib_dealloc_pd(to_mxrcd(xrcd)->pd);
.req_notify_cq = mlx4_ib_arm_cq,
.rereg_user_mr = mlx4_ib_rereg_user_mr,
.resize_cq = mlx4_ib_resize_cq,
+
+ INIT_RDMA_OBJ_SIZE(ib_ah, mlx4_ib_ah, ibah),
INIT_RDMA_OBJ_SIZE(ib_pd, mlx4_ib_pd, ibpd),
+ INIT_RDMA_OBJ_SIZE(ib_srq, mlx4_ib_srq, ibsrq),
INIT_RDMA_OBJ_SIZE(ib_ucontext, mlx4_ib_ucontext, ibucontext),
};
struct mlx4_sriov_alias_guid alias_guid;
/* CM paravirtualization fields */
- struct list_head cm_list;
+ struct xarray pv_id_table;
+ u32 pv_id_next;
spinlock_t id_map_lock;
struct rb_root sl_id_map;
- struct idr pv_id_table;
+ struct list_head cm_list;
};
struct gid_cache_context {
int mlx4_ib_init_sriov(struct mlx4_ib_dev *dev);
void mlx4_ib_close_sriov(struct mlx4_ib_dev *dev);
-int mlx4_ib_db_map_user(struct mlx4_ib_ucontext *context,
- struct ib_udata *udata, unsigned long virt,
+int mlx4_ib_db_map_user(struct ib_udata *udata, unsigned long virt,
struct mlx4_db *db);
void mlx4_ib_db_unmap_user(struct mlx4_ib_ucontext *context, struct mlx4_db *db);
struct ib_mr *mlx4_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
u64 virt_addr, int access_flags,
struct ib_udata *udata);
-int mlx4_ib_dereg_mr(struct ib_mr *mr);
+int mlx4_ib_dereg_mr(struct ib_mr *mr, struct ib_udata *udata);
struct ib_mw *mlx4_ib_alloc_mw(struct ib_pd *pd, enum ib_mw_type type,
struct ib_udata *udata);
int mlx4_ib_dealloc_mw(struct ib_mw *mw);
-struct ib_mr *mlx4_ib_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg);
+struct ib_mr *mlx4_ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata);
int mlx4_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
unsigned int *sg_offset);
int mlx4_ib_modify_cq(struct ib_cq *cq, u16 cq_count, u16 cq_period);
int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata);
struct ib_cq *mlx4_ib_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata);
-int mlx4_ib_destroy_cq(struct ib_cq *cq);
+int mlx4_ib_destroy_cq(struct ib_cq *cq, struct ib_udata *udata);
int mlx4_ib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc);
int mlx4_ib_arm_cq(struct ib_cq *cq, enum ib_cq_notify_flags flags);
void __mlx4_ib_cq_clean(struct mlx4_ib_cq *cq, u32 qpn, struct mlx4_ib_srq *srq);
void mlx4_ib_cq_clean(struct mlx4_ib_cq *cq, u32 qpn, struct mlx4_ib_srq *srq);
-struct ib_ah *mlx4_ib_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
- u32 flags, struct ib_udata *udata);
-struct ib_ah *mlx4_ib_create_ah_slave(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
- int slave_sgid_index, u8 *s_mac,
- u16 vlan_tag);
+int mlx4_ib_create_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr, u32 flags,
+ struct ib_udata *udata);
+int mlx4_ib_create_ah_slave(struct ib_ah *ah, struct rdma_ah_attr *ah_attr,
+ int slave_sgid_index, u8 *s_mac, u16 vlan_tag);
int mlx4_ib_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr);
-int mlx4_ib_destroy_ah(struct ib_ah *ah, u32 flags);
+void mlx4_ib_destroy_ah(struct ib_ah *ah, u32 flags);
-struct ib_srq *mlx4_ib_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *init_attr,
- struct ib_udata *udata);
+int mlx4_ib_create_srq(struct ib_srq *srq, struct ib_srq_init_attr *init_attr,
+ struct ib_udata *udata);
int mlx4_ib_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask attr_mask, struct ib_udata *udata);
int mlx4_ib_query_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr);
-int mlx4_ib_destroy_srq(struct ib_srq *srq);
+void mlx4_ib_destroy_srq(struct ib_srq *srq, struct ib_udata *udata);
void mlx4_ib_free_srq_wqe(struct mlx4_ib_srq *srq, int wqe_index);
int mlx4_ib_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr,
const struct ib_recv_wr **bad_wr);
struct ib_qp *mlx4_ib_create_qp(struct ib_pd *pd,
struct ib_qp_init_attr *init_attr,
struct ib_udata *udata);
-int mlx4_ib_destroy_qp(struct ib_qp *qp);
+int mlx4_ib_destroy_qp(struct ib_qp *qp, struct ib_udata *udata);
void mlx4_ib_drain_sq(struct ib_qp *qp);
void mlx4_ib_drain_rq(struct ib_qp *qp);
int mlx4_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
struct ib_wq *mlx4_ib_create_wq(struct ib_pd *pd,
struct ib_wq_init_attr *init_attr,
struct ib_udata *udata);
-int mlx4_ib_destroy_wq(struct ib_wq *wq);
+int mlx4_ib_destroy_wq(struct ib_wq *wq, struct ib_udata *udata);
int mlx4_ib_modify_wq(struct ib_wq *wq, struct ib_wq_attr *wq_attr,
u32 wq_attr_mask, struct ib_udata *udata);
}
}
-int mlx4_ib_dereg_mr(struct ib_mr *ibmr)
+int mlx4_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
{
struct mlx4_ib_mr *mr = to_mmr(ibmr);
int ret;
return 0;
}
-struct ib_mr *mlx4_ib_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg)
+struct ib_mr *mlx4_ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
struct mlx4_ib_dev *dev = to_mdev(pd->device);
struct mlx4_ib_mr *mr;
goto err_mtt;
if (qp_has_rq(init_attr)) {
- err = mlx4_ib_db_map_user(
- context, udata,
- (src == MLX4_IB_QP_SRC) ? ucmd.qp.db_addr :
+ err = mlx4_ib_db_map_user(udata,
+ (src == MLX4_IB_QP_SRC) ?
+ ucmd.qp.db_addr :
ucmd.wq.db_addr,
- &qp->db);
+ &qp->db);
if (err)
goto err_mtt;
}
}
static void destroy_qp_common(struct mlx4_ib_dev *dev, struct mlx4_ib_qp *qp,
- enum mlx4_ib_source_type src, bool is_user)
+ enum mlx4_ib_source_type src,
+ struct ib_udata *udata)
{
struct mlx4_ib_cq *send_cq, *recv_cq;
unsigned long flags;
list_del(&qp->qps_list);
list_del(&qp->cq_send_list);
list_del(&qp->cq_recv_list);
- if (!is_user) {
+ if (!udata) {
__mlx4_ib_cq_clean(recv_cq, qp->mqp.qpn,
qp->ibqp.srq ? to_msrq(qp->ibqp.srq): NULL);
if (send_cq != recv_cq)
if (qp->flags & MLX4_IB_QP_NETIF)
mlx4_ib_steer_qp_free(dev, qp->mqp.qpn, 1);
else if (src == MLX4_IB_RWQ_SRC)
- mlx4_ib_release_wqn(to_mucontext(
- qp->ibwq.uobject->context), qp, 1);
+ mlx4_ib_release_wqn(
+ rdma_udata_to_drv_context(
+ udata,
+ struct mlx4_ib_ucontext,
+ ibucontext),
+ qp, 1);
else
mlx4_qp_release_range(dev->dev, qp->mqp.qpn, 1);
}
mlx4_mtt_cleanup(dev->dev, &qp->mtt);
- if (is_user) {
+ if (udata) {
if (qp->rq.wqe_cnt) {
- struct mlx4_ib_ucontext *mcontext = !src ?
- to_mucontext(qp->ibqp.uobject->context) :
- to_mucontext(qp->ibwq.uobject->context);
+ struct mlx4_ib_ucontext *mcontext =
+ rdma_udata_to_drv_context(
+ udata,
+ struct mlx4_ib_ucontext,
+ ibucontext);
+
mlx4_ib_db_unmap_user(mcontext, &qp->db);
}
ib_umem_release(qp->umem);
return ibqp;
}
-static int _mlx4_ib_destroy_qp(struct ib_qp *qp)
+static int _mlx4_ib_destroy_qp(struct ib_qp *qp, struct ib_udata *udata)
{
struct mlx4_ib_dev *dev = to_mdev(qp->device);
struct mlx4_ib_qp *mqp = to_mqp(qp);
if (qp->rwq_ind_tbl) {
destroy_qp_rss(dev, mqp);
} else {
- destroy_qp_common(dev, mqp, MLX4_IB_QP_SRC, qp->uobject);
+ destroy_qp_common(dev, mqp, MLX4_IB_QP_SRC, udata);
}
if (is_sqp(dev, mqp))
return 0;
}
-int mlx4_ib_destroy_qp(struct ib_qp *qp)
+int mlx4_ib_destroy_qp(struct ib_qp *qp, struct ib_udata *udata)
{
struct mlx4_ib_qp *mqp = to_mqp(qp);
ib_destroy_qp(sqp->roce_v2_gsi);
}
- return _mlx4_ib_destroy_qp(qp);
+ return _mlx4_ib_destroy_qp(qp, udata);
}
static int to_mlx4_st(struct mlx4_ib_dev *dev, enum mlx4_ib_qp_type type)
if (is_eth) {
gid_attr = attr->ah_attr.grh.sgid_attr;
- vlan = rdma_vlan_dev_vlan_id(gid_attr->ndev);
- memcpy(smac, gid_attr->ndev->dev_addr, ETH_ALEN);
+ err = rdma_read_gid_l2_fields(gid_attr, &vlan,
+ &smac[0]);
+ if (err)
+ goto out;
}
if (mlx4_set_path(dev, attr, attr_mask, qp, &context->pri_path,
return err;
}
-int mlx4_ib_destroy_wq(struct ib_wq *ibwq)
+int mlx4_ib_destroy_wq(struct ib_wq *ibwq, struct ib_udata *udata)
{
struct mlx4_ib_dev *dev = to_mdev(ibwq->device);
struct mlx4_ib_qp *qp = to_mqp((struct ib_qp *)ibwq);
if (qp->counter_index)
mlx4_ib_free_qp_counter(dev, qp);
- destroy_qp_common(dev, qp, MLX4_IB_RWQ_SRC, 1);
+ destroy_qp_common(dev, qp, MLX4_IB_RWQ_SRC, udata);
kfree(qp);
}
}
-struct ib_srq *mlx4_ib_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *init_attr,
- struct ib_udata *udata)
+int mlx4_ib_create_srq(struct ib_srq *ib_srq,
+ struct ib_srq_init_attr *init_attr,
+ struct ib_udata *udata)
{
- struct mlx4_ib_dev *dev = to_mdev(pd->device);
+ struct mlx4_ib_dev *dev = to_mdev(ib_srq->device);
struct mlx4_ib_ucontext *ucontext = rdma_udata_to_drv_context(
udata, struct mlx4_ib_ucontext, ibucontext);
- struct mlx4_ib_srq *srq;
+ struct mlx4_ib_srq *srq = to_msrq(ib_srq);
struct mlx4_wqe_srq_next_seg *next;
struct mlx4_wqe_data_seg *scatter;
u32 cqn;
/* Sanity check SRQ size before proceeding */
if (init_attr->attr.max_wr >= dev->dev->caps.max_srq_wqes ||
init_attr->attr.max_sge > dev->dev->caps.max_srq_sge)
- return ERR_PTR(-EINVAL);
-
- srq = kmalloc(sizeof *srq, GFP_KERNEL);
- if (!srq)
- return ERR_PTR(-ENOMEM);
+ return -EINVAL;
mutex_init(&srq->mutex);
spin_lock_init(&srq->lock);
if (udata) {
struct mlx4_ib_create_srq ucmd;
- if (ib_copy_from_udata(&ucmd, udata, sizeof ucmd)) {
- err = -EFAULT;
- goto err_srq;
- }
+ if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd)))
+ return -EFAULT;
srq->umem = ib_umem_get(udata, ucmd.buf_addr, buf_size, 0, 0);
- if (IS_ERR(srq->umem)) {
- err = PTR_ERR(srq->umem);
- goto err_srq;
- }
+ if (IS_ERR(srq->umem))
+ return PTR_ERR(srq->umem);
err = mlx4_mtt_init(dev->dev, ib_umem_page_count(srq->umem),
srq->umem->page_shift, &srq->mtt);
if (err)
goto err_mtt;
- err = mlx4_ib_db_map_user(ucontext, udata, ucmd.db_addr,
- &srq->db);
+ err = mlx4_ib_db_map_user(udata, ucmd.db_addr, &srq->db);
if (err)
goto err_mtt;
} else {
err = mlx4_db_alloc(dev->dev, &srq->db, 0);
if (err)
- goto err_srq;
+ return err;
*srq->db.db = 0;
xrcdn = (init_attr->srq_type == IB_SRQT_XRC) ?
to_mxrcd(init_attr->ext.xrc.xrcd)->xrcdn :
(u16) dev->dev->caps.reserved_xrcds;
- err = mlx4_srq_alloc(dev->dev, to_mpd(pd)->pdn, cqn, xrcdn, &srq->mtt,
- srq->db.dma, &srq->msrq);
+ err = mlx4_srq_alloc(dev->dev, to_mpd(ib_srq->pd)->pdn, cqn, xrcdn,
+ &srq->mtt, srq->db.dma, &srq->msrq);
if (err)
goto err_wrid;
init_attr->attr.max_wr = srq->msrq.max - 1;
- return &srq->ibsrq;
+ return 0;
err_wrid:
if (udata)
if (!udata)
mlx4_db_free(dev->dev, &srq->db);
-err_srq:
- kfree(srq);
-
- return ERR_PTR(err);
+ return err;
}
int mlx4_ib_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
return 0;
}
-int mlx4_ib_destroy_srq(struct ib_srq *srq)
+void mlx4_ib_destroy_srq(struct ib_srq *srq, struct ib_udata *udata)
{
struct mlx4_ib_dev *dev = to_mdev(srq->device);
struct mlx4_ib_srq *msrq = to_msrq(srq);
mlx4_srq_free(dev->dev, &msrq->msrq);
mlx4_mtt_cleanup(dev->dev, &msrq->mtt);
- if (srq->uobject) {
- mlx4_ib_db_unmap_user(to_mucontext(srq->uobject->context), &msrq->db);
+ if (udata) {
+ mlx4_ib_db_unmap_user(
+ rdma_udata_to_drv_context(
+ udata,
+ struct mlx4_ib_ucontext,
+ ibucontext),
+ &msrq->db);
ib_umem_release(msrq->umem);
} else {
kvfree(msrq->wrid);
&msrq->buf);
mlx4_db_free(dev->dev, &msrq->db);
}
-
- kfree(msrq);
-
- return 0;
}
void mlx4_ib_free_srq_wqe(struct mlx4_ib_srq *srq, int wqe_index)
#include "mlx5_ib.h"
-static struct ib_ah *create_ib_ah(struct mlx5_ib_dev *dev,
- struct mlx5_ib_ah *ah,
- struct rdma_ah_attr *ah_attr)
+static void create_ib_ah(struct mlx5_ib_dev *dev, struct mlx5_ib_ah *ah,
+ struct rdma_ah_attr *ah_attr)
{
enum ib_gid_type gid_type;
ah->av.fl_mlid = rdma_ah_get_path_bits(ah_attr) & 0x7f;
ah->av.stat_rate_sl |= (rdma_ah_get_sl(ah_attr) & 0xf);
}
-
- return &ah->ibah;
}
-struct ib_ah *mlx5_ib_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
- u32 flags, struct ib_udata *udata)
+int mlx5_ib_create_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr,
+ u32 flags, struct ib_udata *udata)
{
- struct mlx5_ib_ah *ah;
- struct mlx5_ib_dev *dev = to_mdev(pd->device);
+ struct mlx5_ib_ah *ah = to_mah(ibah);
+ struct mlx5_ib_dev *dev = to_mdev(ibah->device);
enum rdma_ah_attr_type ah_type = ah_attr->type;
if ((ah_type == RDMA_AH_ATTR_TYPE_ROCE) &&
!(rdma_ah_get_ah_flags(ah_attr) & IB_AH_GRH))
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
if (ah_type == RDMA_AH_ATTR_TYPE_ROCE && udata) {
int err;
sizeof(resp.dmac);
if (udata->outlen < min_resp_len)
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
resp.response_length = min_resp_len;
memcpy(resp.dmac, ah_attr->roce.dmac, ETH_ALEN);
err = ib_copy_to_udata(udata, &resp, resp.response_length);
if (err)
- return ERR_PTR(err);
+ return err;
}
- ah = kzalloc(sizeof(*ah), GFP_ATOMIC);
- if (!ah)
- return ERR_PTR(-ENOMEM);
-
- return create_ib_ah(dev, ah, ah_attr); /* never fails */
+ create_ib_ah(dev, ah, ah_attr);
+ return 0;
}
int mlx5_ib_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr)
return 0;
}
-int mlx5_ib_destroy_ah(struct ib_ah *ah, u32 flags)
+void mlx5_ib_destroy_ah(struct ib_ah *ah, u32 flags)
{
- kfree(to_mah(ah));
- return 0;
+ return;
}
return mlx5_cmd_exec(dev, in, in_size, out, sizeof(out));
}
-int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr,
- u64 length, u32 alignment)
+int mlx5_cmd_alloc_memic(struct mlx5_dm *dm, phys_addr_t *addr,
+ u64 length, u32 alignment)
{
- struct mlx5_core_dev *dev = memic->dev;
+ struct mlx5_core_dev *dev = dm->dev;
u64 num_memic_hw_pages = MLX5_CAP_DEV_MEM(dev, memic_bar_size)
>> PAGE_SHIFT;
u64 hw_start_addr = MLX5_CAP64_DEV_MEM(dev, memic_bar_start_addr);
mlx5_alignment);
while (page_idx < num_memic_hw_pages) {
- spin_lock(&memic->memic_lock);
- page_idx = bitmap_find_next_zero_area(memic->memic_alloc_pages,
+ spin_lock(&dm->lock);
+ page_idx = bitmap_find_next_zero_area(dm->memic_alloc_pages,
num_memic_hw_pages,
page_idx,
num_pages, 0);
if (page_idx < num_memic_hw_pages)
- bitmap_set(memic->memic_alloc_pages,
+ bitmap_set(dm->memic_alloc_pages,
page_idx, num_pages);
- spin_unlock(&memic->memic_lock);
+ spin_unlock(&dm->lock);
if (page_idx >= num_memic_hw_pages)
break;
ret = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
if (ret) {
- spin_lock(&memic->memic_lock);
- bitmap_clear(memic->memic_alloc_pages,
+ spin_lock(&dm->lock);
+ bitmap_clear(dm->memic_alloc_pages,
page_idx, num_pages);
- spin_unlock(&memic->memic_lock);
+ spin_unlock(&dm->lock);
if (ret == -EAGAIN) {
page_idx++;
return -ENOMEM;
}
-int mlx5_cmd_dealloc_memic(struct mlx5_memic *memic, u64 addr, u64 length)
+int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, phys_addr_t addr, u64 length)
{
- struct mlx5_core_dev *dev = memic->dev;
+ struct mlx5_core_dev *dev = dm->dev;
u64 hw_start_addr = MLX5_CAP64_DEV_MEM(dev, memic_bar_start_addr);
u32 num_pages = DIV_ROUND_UP(length, PAGE_SIZE);
u32 out[MLX5_ST_SZ_DW(dealloc_memic_out)] = {0};
err = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
if (!err) {
- spin_lock(&memic->memic_lock);
- bitmap_clear(memic->memic_alloc_pages,
+ spin_lock(&dm->lock);
+ bitmap_clear(dm->memic_alloc_pages,
start_page_idx, num_pages);
- spin_unlock(&memic->memic_lock);
+ spin_unlock(&dm->lock);
}
return err;
}
+int mlx5_cmd_alloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
+ u16 uid, phys_addr_t *addr, u32 *obj_id)
+{
+ struct mlx5_core_dev *dev = dm->dev;
+ u32 num_blocks = DIV_ROUND_UP(length, MLX5_SW_ICM_BLOCK_SIZE(dev));
+ u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
+ u32 in[MLX5_ST_SZ_DW(create_sw_icm_in)] = {};
+ unsigned long *block_map;
+ u64 icm_start_addr;
+ u32 log_icm_size;
+ u32 max_blocks;
+ u64 block_idx;
+ void *sw_icm;
+ int ret;
+
+ MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
+ MLX5_CMD_OP_CREATE_GENERAL_OBJECT);
+ MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
+ MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
+
+ switch (type) {
+ case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+ icm_start_addr = MLX5_CAP64_DEV_MEM(dev,
+ steering_sw_icm_start_address);
+ log_icm_size = MLX5_CAP_DEV_MEM(dev, log_steering_sw_icm_size);
+ block_map = dm->steering_sw_icm_alloc_blocks;
+ break;
+ case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
+ icm_start_addr = MLX5_CAP64_DEV_MEM(dev,
+ header_modify_sw_icm_start_address);
+ log_icm_size = MLX5_CAP_DEV_MEM(dev,
+ log_header_modify_sw_icm_size);
+ block_map = dm->header_modify_sw_icm_alloc_blocks;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ max_blocks = BIT(log_icm_size - MLX5_LOG_SW_ICM_BLOCK_SIZE(dev));
+ spin_lock(&dm->lock);
+ block_idx = bitmap_find_next_zero_area(block_map,
+ max_blocks,
+ 0,
+ num_blocks, 0);
+
+ if (block_idx < max_blocks)
+ bitmap_set(block_map,
+ block_idx, num_blocks);
+
+ spin_unlock(&dm->lock);
+
+ if (block_idx >= max_blocks)
+ return -ENOMEM;
+
+ sw_icm = MLX5_ADDR_OF(create_sw_icm_in, in, sw_icm);
+ icm_start_addr += block_idx << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
+ MLX5_SET64(sw_icm, sw_icm, sw_icm_start_addr,
+ icm_start_addr);
+ MLX5_SET(sw_icm, sw_icm, log_sw_icm_size, ilog2(length));
+
+ ret = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+ if (ret) {
+ spin_lock(&dm->lock);
+ bitmap_clear(block_map,
+ block_idx, num_blocks);
+ spin_unlock(&dm->lock);
+
+ return ret;
+ }
+
+ *addr = icm_start_addr;
+ *obj_id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id);
+
+ return 0;
+}
+
+int mlx5_cmd_dealloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
+ u16 uid, phys_addr_t addr, u32 obj_id)
+{
+ struct mlx5_core_dev *dev = dm->dev;
+ u32 num_blocks = DIV_ROUND_UP(length, MLX5_SW_ICM_BLOCK_SIZE(dev));
+ u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)] = {};
+ u32 in[MLX5_ST_SZ_DW(general_obj_in_cmd_hdr)] = {};
+ unsigned long *block_map;
+ u64 start_idx;
+ int err;
+
+ switch (type) {
+ case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+ start_idx =
+ (addr - MLX5_CAP64_DEV_MEM(
+ dev, steering_sw_icm_start_address)) >>
+ MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
+ block_map = dm->steering_sw_icm_alloc_blocks;
+ break;
+ case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
+ start_idx =
+ (addr -
+ MLX5_CAP64_DEV_MEM(
+ dev, header_modify_sw_icm_start_address)) >>
+ MLX5_LOG_SW_ICM_BLOCK_SIZE(dev);
+ block_map = dm->header_modify_sw_icm_alloc_blocks;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ MLX5_SET(general_obj_in_cmd_hdr, in, opcode,
+ MLX5_CMD_OP_DESTROY_GENERAL_OBJECT);
+ MLX5_SET(general_obj_in_cmd_hdr, in, obj_type, MLX5_OBJ_TYPE_SW_ICM);
+ MLX5_SET(general_obj_in_cmd_hdr, in, obj_id, obj_id);
+ MLX5_SET(general_obj_in_cmd_hdr, in, uid, uid);
+
+ err = mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+ if (err)
+ return err;
+
+ spin_lock(&dm->lock);
+ bitmap_clear(block_map,
+ start_idx, num_blocks);
+ spin_unlock(&dm->lock);
+
+ return 0;
+}
+
int mlx5_cmd_query_ext_ppcnt_counters(struct mlx5_core_dev *dev, void *out)
{
u32 in[MLX5_ST_SZ_DW(ppcnt_reg)] = {};
int mlx5_cmd_query_ext_ppcnt_counters(struct mlx5_core_dev *dev, void *out);
int mlx5_cmd_modify_cong_params(struct mlx5_core_dev *mdev,
void *in, int in_size);
-int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr,
+int mlx5_cmd_alloc_memic(struct mlx5_dm *dm, phys_addr_t *addr,
u64 length, u32 alignment);
-int mlx5_cmd_dealloc_memic(struct mlx5_memic *memic, u64 addr, u64 length);
+int mlx5_cmd_dealloc_memic(struct mlx5_dm *dm, phys_addr_t addr, u64 length);
void mlx5_cmd_dealloc_pd(struct mlx5_core_dev *dev, u32 pdn, u16 uid);
void mlx5_cmd_destroy_tir(struct mlx5_core_dev *dev, u32 tirn, u16 uid);
void mlx5_cmd_destroy_tis(struct mlx5_core_dev *dev, u32 tisn, u16 uid);
u16 uid);
int mlx5_cmd_mad_ifc(struct mlx5_core_dev *dev, const void *inb, void *outb,
u16 opmod, u8 port);
+int mlx5_cmd_alloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
+ u16 uid, phys_addr_t *addr, u32 *obj_id);
+int mlx5_cmd_dealloc_sw_icm(struct mlx5_dm *dm, int type, u64 length,
+ u16 uid, phys_addr_t addr, u32 obj_id);
#endif /* MLX5_IB_CMD_H */
}
static int create_cq_user(struct mlx5_ib_dev *dev, struct ib_udata *udata,
- struct ib_ucontext *context, struct mlx5_ib_cq *cq,
- int entries, u32 **cqb,
+ struct mlx5_ib_cq *cq, int entries, u32 **cqb,
int *cqe_size, int *index, int *inlen)
{
struct mlx5_ib_create_cq ucmd = {};
int ncont;
void *cqc;
int err;
+ struct mlx5_ib_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct mlx5_ib_ucontext, ibucontext);
ucmdlen = udata->inlen < sizeof(ucmd) ?
(sizeof(ucmd) - sizeof(ucmd.flags)) : sizeof(ucmd);
return err;
}
- err = mlx5_ib_db_map_user(to_mucontext(context), udata, ucmd.db_addr,
- &cq->db);
+ err = mlx5_ib_db_map_user(context, udata, ucmd.db_addr, &cq->db);
if (err)
goto err_umem;
MLX5_SET(cqc, cqc, log_page_size,
page_shift - MLX5_ADAPTER_PAGE_SHIFT);
- *index = to_mucontext(context)->bfregi.sys_pages[0];
+ *index = context->bfregi.sys_pages[0];
if (ucmd.cqe_comp_en == 1) {
int mini_cqe_format;
cq->private_flags |= MLX5_IB_CQ_PR_FLAGS_CQE_128_PAD;
}
- MLX5_SET(create_cq_in, *cqb, uid, to_mucontext(context)->devx_uid);
+ MLX5_SET(create_cq_in, *cqb, uid, context->devx_uid);
return 0;
err_cqb:
kvfree(*cqb);
err_db:
- mlx5_ib_db_unmap_user(to_mucontext(context), &cq->db);
+ mlx5_ib_db_unmap_user(context, &cq->db);
err_umem:
ib_umem_release(cq->buf.umem);
return err;
}
-static void destroy_cq_user(struct mlx5_ib_cq *cq, struct ib_ucontext *context)
+static void destroy_cq_user(struct mlx5_ib_cq *cq, struct ib_udata *udata)
{
- mlx5_ib_db_unmap_user(to_mucontext(context), &cq->db);
+ struct mlx5_ib_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct mlx5_ib_ucontext, ibucontext);
+
+ mlx5_ib_db_unmap_user(context, &cq->db);
ib_umem_release(cq->buf.umem);
}
struct ib_cq *mlx5_ib_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
int entries = attr->cqe;
INIT_LIST_HEAD(&cq->list_send_qp);
INIT_LIST_HEAD(&cq->list_recv_qp);
- if (context) {
- err = create_cq_user(dev, udata, context, cq, entries,
- &cqb, &cqe_size, &index, &inlen);
+ if (udata) {
+ err = create_cq_user(dev, udata, cq, entries, &cqb, &cqe_size,
+ &index, &inlen);
if (err)
goto err_create;
} else {
mlx5_ib_dbg(dev, "cqn 0x%x\n", cq->mcq.cqn);
cq->mcq.irqn = irqn;
- if (context)
+ if (udata)
cq->mcq.tasklet_ctx.comp = mlx5_ib_cq_comp;
else
cq->mcq.comp = mlx5_ib_cq_comp;
INIT_LIST_HEAD(&cq->wc_list);
- if (context)
+ if (udata)
if (ib_copy_to_udata(udata, &cq->mcq.cqn, sizeof(__u32))) {
err = -EFAULT;
goto err_cmd;
err_cqb:
kvfree(cqb);
- if (context)
- destroy_cq_user(cq, context);
+ if (udata)
+ destroy_cq_user(cq, udata);
else
destroy_cq_kernel(dev, cq);
return ERR_PTR(err);
}
-
-int mlx5_ib_destroy_cq(struct ib_cq *cq)
+int mlx5_ib_destroy_cq(struct ib_cq *cq, struct ib_udata *udata)
{
struct mlx5_ib_dev *dev = to_mdev(cq->device);
struct mlx5_ib_cq *mcq = to_mcq(cq);
- struct ib_ucontext *context = NULL;
-
- if (cq->uobject)
- context = cq->uobject->context;
mlx5_core_destroy_cq(dev->mdev, &mcq->mcq);
- if (context)
- destroy_cq_user(mcq, context);
+ if (udata)
+ destroy_cq_user(mcq, udata);
else
destroy_cq_kernel(dev, mcq);
if (is_user && capable(CAP_NET_RAW) &&
(MLX5_CAP_GEN(dev->mdev, uctx_cap) & MLX5_UCTX_CAP_RAW_TX))
cap |= MLX5_UCTX_CAP_RAW_TX;
+ if (is_user && capable(CAP_SYS_RAWIO) &&
+ (MLX5_CAP_GEN(dev->mdev, uctx_cap) &
+ MLX5_UCTX_CAP_INTERNAL_DEV_RES))
+ cap |= MLX5_UCTX_CAP_INTERNAL_DEV_RES;
MLX5_SET(create_uctx_in, in, opcode, MLX5_CMD_OP_CREATE_UCTX);
MLX5_SET(uctx, uctx, cap, cap);
return obj_id;
}
-static bool devx_is_valid_obj_id(struct ib_uobject *uobj, const void *in)
+static bool devx_is_valid_obj_id(struct uverbs_attr_bundle *attrs,
+ struct ib_uobject *uobj, const void *in)
{
+ struct mlx5_ib_dev *dev = mlx5_udata_to_mdev(&attrs->driver_udata);
u64 obj_id = devx_get_obj_id(in);
if (!obj_id)
case UVERBS_OBJECT_SRQ:
{
struct mlx5_core_srq *srq = &(to_msrq(uobj->object)->msrq);
- struct mlx5_ib_dev *dev = to_mdev(uobj->context->device);
u16 opcode;
switch (srq->common.res) {
switch (opcode) {
case MLX5_CMD_OP_QUERY_HCA_CAP:
case MLX5_CMD_OP_QUERY_HCA_VPORT_CONTEXT:
+ case MLX5_CMD_OP_QUERY_ESW_VPORT_CONTEXT:
return true;
default:
return false;
switch (opcode) {
case MLX5_CMD_OP_QUERY_HCA_CAP:
case MLX5_CMD_OP_QUERY_HCA_VPORT_CONTEXT:
+ case MLX5_CMD_OP_QUERY_ESW_VPORT_CONTEXT:
case MLX5_CMD_OP_QUERY_VPORT_STATE:
case MLX5_CMD_OP_QUERY_ADAPTER:
case MLX5_CMD_OP_QUERY_ISSI:
}
static int devx_obj_cleanup(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)];
struct devx_obj *obj = uobject->object;
return ret;
if (obj->flags & DEVX_OBJ_FLAGS_INDIRECT_MKEY) {
- struct mlx5_ib_dev *dev = to_mdev(uobject->context->device);
+ struct mlx5_ib_dev *dev =
+ mlx5_udata_to_mdev(&attrs->driver_udata);
call_srcu(&dev->mr_srcu, &obj->devx_mr.rcu,
devx_free_indirect_mkey);
if (!devx_is_obj_modify_cmd(cmd_in))
return -EINVAL;
- if (!devx_is_valid_obj_id(uobj, cmd_in))
+ if (!devx_is_valid_obj_id(attrs, uobj, cmd_in))
return -EINVAL;
cmd_out = uverbs_zalloc(attrs, cmd_out_len);
if (!devx_is_obj_query_cmd(cmd_in))
return -EINVAL;
- if (!devx_is_valid_obj_id(uobj, cmd_in))
+ if (!devx_is_valid_obj_id(attrs, uobj, cmd_in))
return -EINVAL;
cmd_out = uverbs_zalloc(attrs, cmd_out_len);
struct ib_uobject *uobj = uverbs_attr_get_uobject(
attrs, MLX5_IB_ATTR_DEVX_ASYNC_CMD_FD_ALLOC_HANDLE);
- struct mlx5_ib_dev *mdev = to_mdev(uobj->context->device);
+ struct mlx5_ib_dev *mdev = mlx5_udata_to_mdev(&attrs->driver_udata);
ev_file = container_of(uobj, struct devx_async_cmd_event_file,
uobj);
if (err)
return err;
- if (!devx_is_valid_obj_id(uobj, cmd_in))
+ if (!devx_is_valid_obj_id(attrs, uobj, cmd_in))
return -EINVAL;
fd_uobj = uverbs_attr_get_uobject(attrs,
}
static int devx_umem_cleanup(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct devx_umem *obj = uobject->object;
u32 out[MLX5_ST_SZ_DW(general_obj_out_cmd_hdr)];
return pollflags;
}
-const struct file_operations devx_async_cmd_event_fops = {
+static const struct file_operations devx_async_cmd_event_fops = {
.owner = THIS_MODULE,
.read = devx_async_cmd_event_read,
.poll = devx_async_cmd_event_poll,
{
struct mlx5_ib_dev *dev = to_mdev(device);
- return !dev->rep && MLX5_CAP_GEN(dev->mdev, log_max_uctx);
+ return MLX5_CAP_GEN(dev->mdev, log_max_uctx);
}
const struct uapi_definition mlx5_ib_devx_defs[] = {
case MLX5_IB_UAPI_FLOW_TABLE_TYPE_NIC_TX:
*namespace = MLX5_FLOW_NAMESPACE_EGRESS;
break;
+ case MLX5_IB_UAPI_FLOW_TABLE_TYPE_FDB:
+ *namespace = MLX5_FLOW_NAMESPACE_FDB;
+ break;
default:
return -EINVAL;
}
struct ib_qp *qp = NULL;
struct ib_uobject *uobj =
uverbs_attr_get_uobject(attrs, MLX5_IB_ATTR_CREATE_FLOW_HANDLE);
- struct mlx5_ib_dev *dev = to_mdev(uobj->context->device);
+ struct mlx5_ib_dev *dev = mlx5_udata_to_mdev(&attrs->driver_udata);
int len, ret, i;
u32 counter_id = 0;
((dest_devx && dest_qp) || (!dest_devx && !dest_qp)))
return -EINVAL;
+ /* Allow only DEVX object as dest when inserting to FDB */
+ if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_FDB && !dest_devx)
+ return -EINVAL;
+
if (dest_devx) {
devx_obj = uverbs_attr_get_obj(
attrs, MLX5_IB_ATTR_CREATE_FLOW_DEST_DEVX);
*/
if (!mlx5_ib_devx_is_flow_dest(devx_obj, &dest_id, &dest_type))
return -EINVAL;
+ /* Allow only flow table as dest when inserting to FDB */
+ if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_FDB &&
+ dest_type != MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE)
+ return -EINVAL;
} else if (dest_qp) {
struct mlx5_ib_qp *mqp;
}
static int flow_matcher_cleanup(struct ib_uobject *uobject,
- enum rdma_remove_reason why)
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs)
{
struct mlx5_ib_flow_matcher *obj = uobject->object;
int ret;
return 0;
}
+static int mlx5_ib_matcher_ns(struct uverbs_attr_bundle *attrs,
+ struct mlx5_ib_flow_matcher *obj)
+{
+ enum mlx5_ib_uapi_flow_table_type ft_type =
+ MLX5_IB_UAPI_FLOW_TABLE_TYPE_NIC_RX;
+ u32 flags;
+ int err;
+
+ /* New users should use MLX5_IB_ATTR_FLOW_MATCHER_FT_TYPE and older
+ * users should switch to it. We leave this to not break userspace
+ */
+ if (uverbs_attr_is_valid(attrs, MLX5_IB_ATTR_FLOW_MATCHER_FT_TYPE) &&
+ uverbs_attr_is_valid(attrs, MLX5_IB_ATTR_FLOW_MATCHER_FLOW_FLAGS))
+ return -EINVAL;
+
+ if (uverbs_attr_is_valid(attrs, MLX5_IB_ATTR_FLOW_MATCHER_FT_TYPE)) {
+ err = uverbs_get_const(&ft_type, attrs,
+ MLX5_IB_ATTR_FLOW_MATCHER_FT_TYPE);
+ if (err)
+ return err;
+
+ err = mlx5_ib_ft_type_to_namespace(ft_type, &obj->ns_type);
+ if (err)
+ return err;
+
+ return 0;
+ }
+
+ if (uverbs_attr_is_valid(attrs, MLX5_IB_ATTR_FLOW_MATCHER_FLOW_FLAGS)) {
+ err = uverbs_get_flags32(&flags, attrs,
+ MLX5_IB_ATTR_FLOW_MATCHER_FLOW_FLAGS,
+ IB_FLOW_ATTR_FLAGS_EGRESS);
+ if (err)
+ return err;
+
+ if (flags) {
+ mlx5_ib_ft_type_to_namespace(
+ MLX5_IB_UAPI_FLOW_TABLE_TYPE_NIC_TX,
+ &obj->ns_type);
+ return 0;
+ }
+ }
+
+ obj->ns_type = MLX5_FLOW_NAMESPACE_BYPASS;
+
+ return 0;
+}
+
static int UVERBS_HANDLER(MLX5_IB_METHOD_FLOW_MATCHER_CREATE)(
struct uverbs_attr_bundle *attrs)
{
struct ib_uobject *uobj = uverbs_attr_get_uobject(
attrs, MLX5_IB_ATTR_FLOW_MATCHER_CREATE_HANDLE);
- struct mlx5_ib_dev *dev = to_mdev(uobj->context->device);
+ struct mlx5_ib_dev *dev = mlx5_udata_to_mdev(&attrs->driver_udata);
struct mlx5_ib_flow_matcher *obj;
- u32 flags;
int err;
obj = kzalloc(sizeof(struct mlx5_ib_flow_matcher), GFP_KERNEL);
if (!obj)
return -ENOMEM;
- obj->ns_type = MLX5_FLOW_NAMESPACE_BYPASS;
obj->mask_len = uverbs_attr_get_len(
attrs, MLX5_IB_ATTR_FLOW_MATCHER_MATCH_MASK);
err = uverbs_copy_from(&obj->matcher_mask,
if (err)
goto end;
- err = uverbs_get_flags32(&flags, attrs,
- MLX5_IB_ATTR_FLOW_MATCHER_FLOW_FLAGS,
- IB_FLOW_ATTR_FLAGS_EGRESS);
+ err = mlx5_ib_matcher_ns(attrs, obj);
if (err)
goto end;
- if (flags) {
- err = mlx5_ib_ft_type_to_namespace(
- MLX5_IB_UAPI_FLOW_TABLE_TYPE_NIC_TX, &obj->ns_type);
- if (err)
- goto end;
- }
-
uobj->object = obj;
obj->mdev = dev->mdev;
atomic_set(&obj->usecnt, 0);
{
struct ib_uobject *uobj = uverbs_attr_get_uobject(
attrs, MLX5_IB_ATTR_CREATE_MODIFY_HEADER_HANDLE);
- struct mlx5_ib_dev *mdev = to_mdev(uobj->context->device);
+ struct mlx5_ib_dev *mdev = mlx5_udata_to_mdev(&attrs->driver_udata);
enum mlx5_ib_uapi_flow_table_type ft_type;
struct ib_flow_action *action;
int num_actions;
if (IS_ERR(action))
return PTR_ERR(action);
- uverbs_flow_action_fill_action(action, uobj, uobj->context->device,
+ uverbs_flow_action_fill_action(action, uobj, &mdev->ib_dev,
IB_FLOW_ACTION_UNSPECIFIED);
return 0;
{
struct ib_uobject *uobj = uverbs_attr_get_uobject(attrs,
MLX5_IB_ATTR_CREATE_PACKET_REFORMAT_HANDLE);
- struct mlx5_ib_dev *mdev = to_mdev(uobj->context->device);
+ struct mlx5_ib_dev *mdev = mlx5_udata_to_mdev(&attrs->driver_udata);
enum mlx5_ib_uapi_flow_action_packet_reformat_type dv_prt;
enum mlx5_ib_uapi_flow_table_type ft_type;
struct mlx5_ib_flow_action *maction;
goto free_maction;
}
- uverbs_flow_action_fill_action(&maction->ib_action, uobj,
- uobj->context->device,
+ uverbs_flow_action_fill_action(&maction->ib_action, uobj, &mdev->ib_dev,
IB_FLOW_ACTION_UNSPECIFIED);
return 0;
UA_MANDATORY),
UVERBS_ATTR_FLAGS_IN(MLX5_IB_ATTR_FLOW_MATCHER_FLOW_FLAGS,
enum ib_flow_flags,
+ UA_OPTIONAL),
+ UVERBS_ATTR_CONST_IN(MLX5_IB_ATTR_FLOW_MATCHER_FT_TYPE,
+ enum mlx5_ib_uapi_flow_table_type,
UA_OPTIONAL));
DECLARE_UVERBS_NAMED_METHOD_DESTROY(
&UVERBS_METHOD(MLX5_IB_METHOD_FLOW_MATCHER_CREATE),
&UVERBS_METHOD(MLX5_IB_METHOD_FLOW_MATCHER_DESTROY));
-static bool flow_is_supported(struct ib_device *device)
-{
- return !to_mdev(device)->rep;
-}
-
const struct uapi_definition mlx5_ib_flow_defs[] = {
UAPI_DEF_CHAIN_OBJ_TREE_NAMED(
- MLX5_IB_OBJECT_FLOW_MATCHER,
- UAPI_DEF_IS_OBJ_SUPPORTED(flow_is_supported)),
+ MLX5_IB_OBJECT_FLOW_MATCHER),
UAPI_DEF_CHAIN_OBJ_TREE(
UVERBS_OBJECT_FLOW,
&mlx5_ib_fs),
#include "ib_rep.h"
#include "srq.h"
-static const struct mlx5_ib_profile vf_rep_profile = {
- STAGE_CREATE(MLX5_IB_STAGE_INIT,
- mlx5_ib_stage_init_init,
- mlx5_ib_stage_init_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_FLOW_DB,
- mlx5_ib_stage_rep_flow_db_init,
- NULL),
- STAGE_CREATE(MLX5_IB_STAGE_CAPS,
- mlx5_ib_stage_caps_init,
- NULL),
- STAGE_CREATE(MLX5_IB_STAGE_NON_DEFAULT_CB,
- mlx5_ib_stage_rep_non_default_cb,
- NULL),
- STAGE_CREATE(MLX5_IB_STAGE_ROCE,
- mlx5_ib_stage_rep_roce_init,
- mlx5_ib_stage_rep_roce_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_SRQ,
- mlx5_init_srq_table,
- mlx5_cleanup_srq_table),
- STAGE_CREATE(MLX5_IB_STAGE_DEVICE_RESOURCES,
- mlx5_ib_stage_dev_res_init,
- mlx5_ib_stage_dev_res_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_COUNTERS,
- mlx5_ib_stage_counters_init,
- mlx5_ib_stage_counters_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_BFREG,
- mlx5_ib_stage_bfrag_init,
- mlx5_ib_stage_bfrag_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_PRE_IB_REG_UMR,
- NULL,
- mlx5_ib_stage_pre_ib_reg_umr_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_IB_REG,
- mlx5_ib_stage_ib_reg_init,
- mlx5_ib_stage_ib_reg_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_POST_IB_REG_UMR,
- mlx5_ib_stage_post_ib_reg_umr_init,
- NULL),
-};
+static int
+mlx5_ib_set_vport_rep(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
+{
+ struct mlx5_ib_dev *ibdev;
+ int vport_index;
+
+ ibdev = mlx5_ib_get_uplink_ibdev(dev->priv.eswitch);
+ vport_index = ibdev->free_port++;
+
+ ibdev->port[vport_index].rep = rep;
+ write_lock(&ibdev->port[vport_index].roce.netdev_lock);
+ ibdev->port[vport_index].roce.netdev =
+ mlx5_ib_get_rep_netdev(dev->priv.eswitch, rep->vport);
+ write_unlock(&ibdev->port[vport_index].roce.netdev_lock);
+
+ return 0;
+}
static int
mlx5_ib_vport_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
{
+ int num_ports = MLX5_TOTAL_VPORTS(dev);
const struct mlx5_ib_profile *profile;
struct mlx5_ib_dev *ibdev;
+ int vport_index;
if (rep->vport == MLX5_VPORT_UPLINK)
profile = &uplink_rep_profile;
else
- profile = &vf_rep_profile;
+ return mlx5_ib_set_vport_rep(dev, rep);
ibdev = ib_alloc_device(mlx5_ib_dev, ib_dev);
if (!ibdev)
return -ENOMEM;
- ibdev->rep = rep;
- ibdev->mdev = dev;
- ibdev->num_ports = max(MLX5_CAP_GEN(dev, num_ports),
- MLX5_CAP_GEN(dev, num_vhca_ports));
- if (!__mlx5_ib_add(ibdev, profile)) {
+ ibdev->port = kcalloc(num_ports, sizeof(*ibdev->port),
+ GFP_KERNEL);
+ if (!ibdev->port) {
ib_dealloc_device(&ibdev->ib_dev);
- return -EINVAL;
+ return -ENOMEM;
}
+ ibdev->is_rep = true;
+ vport_index = ibdev->free_port++;
+ ibdev->port[vport_index].rep = rep;
+ ibdev->port[vport_index].roce.netdev =
+ mlx5_ib_get_rep_netdev(dev->priv.eswitch, rep->vport);
+ ibdev->mdev = dev;
+ ibdev->num_ports = num_ports;
+
+ if (!__mlx5_ib_add(ibdev, profile))
+ return -EINVAL;
+
rep->rep_if[REP_IB].priv = ibdev;
return 0;
{
struct mlx5_ib_dev *dev;
- if (!rep->rep_if[REP_IB].priv)
+ if (!rep->rep_if[REP_IB].priv ||
+ rep->vport != MLX5_VPORT_UPLINK)
return;
dev = mlx5_ib_rep_to_dev(rep);
__mlx5_ib_remove(dev, dev->profile, MLX5_IB_STAGE_MAX);
rep->rep_if[REP_IB].priv = NULL;
- ib_dealloc_device(&dev->ib_dev);
}
static void *mlx5_ib_vport_get_proto_dev(struct mlx5_eswitch_rep *rep)
return mlx5_eswitch_vport_rep(esw, vport);
}
-int create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
- struct mlx5_ib_sq *sq)
+struct mlx5_flow_handle *create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
+ struct mlx5_ib_sq *sq,
+ u16 port)
{
- struct mlx5_flow_handle *flow_rule;
struct mlx5_eswitch *esw = dev->mdev->priv.eswitch;
+ struct mlx5_eswitch_rep *rep;
- if (!dev->rep)
- return 0;
+ if (!dev->is_rep || !port)
+ return NULL;
- flow_rule =
- mlx5_eswitch_add_send_to_vport_rule(esw,
- dev->rep->vport,
- sq->base.mqp.qpn);
- if (IS_ERR(flow_rule))
- return PTR_ERR(flow_rule);
- sq->flow_rule = flow_rule;
+ if (!dev->port[port - 1].rep)
+ return ERR_PTR(-EINVAL);
- return 0;
+ rep = dev->port[port - 1].rep;
+
+ return mlx5_eswitch_add_send_to_vport_rule(esw, rep->vport,
+ sq->base.mqp.qpn);
}
int vport_index);
void mlx5_ib_register_vport_reps(struct mlx5_core_dev *mdev);
void mlx5_ib_unregister_vport_reps(struct mlx5_core_dev *mdev);
-int create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
- struct mlx5_ib_sq *sq);
+struct mlx5_flow_handle *create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
+ struct mlx5_ib_sq *sq,
+ u16 port);
struct net_device *mlx5_ib_get_rep_netdev(struct mlx5_eswitch *esw,
int vport_index);
#else /* CONFIG_MLX5_ESWITCH */
static inline void mlx5_ib_register_vport_reps(struct mlx5_core_dev *mdev) {}
static inline void mlx5_ib_unregister_vport_reps(struct mlx5_core_dev *mdev) {}
-static inline int create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
- struct mlx5_ib_sq *sq)
+static inline
+struct mlx5_flow_handle *create_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
+ struct mlx5_ib_sq *sq,
+ u16 port)
{
- return 0;
+ return NULL;
}
static inline
return ret;
}
+static struct mlx5_roce *mlx5_get_rep_roce(struct mlx5_ib_dev *dev,
+ struct net_device *ndev,
+ u8 *port_num)
+{
+ struct mlx5_eswitch *esw = dev->mdev->priv.eswitch;
+ struct net_device *rep_ndev;
+ struct mlx5_ib_port *port;
+ int i;
+
+ for (i = 0; i < dev->num_ports; i++) {
+ port = &dev->port[i];
+ if (!port->rep)
+ continue;
+
+ read_lock(&port->roce.netdev_lock);
+ rep_ndev = mlx5_ib_get_rep_netdev(esw,
+ port->rep->vport);
+ if (rep_ndev == ndev) {
+ read_unlock(&port->roce.netdev_lock);
+ *port_num = i + 1;
+ return &port->roce;
+ }
+ read_unlock(&port->roce.netdev_lock);
+ }
+
+ return NULL;
+}
+
static int mlx5_netdev_event(struct notifier_block *this,
unsigned long event, void *ptr)
{
switch (event) {
case NETDEV_REGISTER:
+ /* Should already be registered during the load */
+ if (ibdev->is_rep)
+ break;
write_lock(&roce->netdev_lock);
- if (ibdev->rep) {
- struct mlx5_eswitch *esw = ibdev->mdev->priv.eswitch;
- struct net_device *rep_ndev;
-
- rep_ndev = mlx5_ib_get_rep_netdev(esw,
- ibdev->rep->vport);
- if (rep_ndev == ndev)
- roce->netdev = ndev;
- } else if (ndev->dev.parent == mdev->device) {
+ if (ndev->dev.parent == mdev->device)
roce->netdev = ndev;
- }
write_unlock(&roce->netdev_lock);
break;
case NETDEV_UNREGISTER:
+ /* In case of reps, ib device goes away before the netdevs */
write_lock(&roce->netdev_lock);
if (roce->netdev == ndev)
roce->netdev = NULL;
dev_put(lag_ndev);
}
+ if (ibdev->is_rep)
+ roce = mlx5_get_rep_roce(ibdev, ndev, &port_num);
+ if (!roce)
+ return NOTIFY_DONE;
if ((upper == ndev || (!upper && ndev == roce->netdev))
&& ibdev->ib_active) {
struct ib_event ibev = { };
/* Ensure ndev does not disappear before we invoke dev_hold()
*/
- read_lock(&ibdev->roce[port_num - 1].netdev_lock);
- ndev = ibdev->roce[port_num - 1].netdev;
+ read_lock(&ibdev->port[port_num - 1].roce.netdev_lock);
+ ndev = ibdev->port[port_num - 1].roce.netdev;
if (ndev)
dev_hold(ndev);
- read_unlock(&ibdev->roce[port_num - 1].netdev_lock);
+ read_unlock(&ibdev->port[port_num - 1].roce.netdev_lock);
out:
mlx5_ib_put_native_port_mdev(ibdev, port_num);
/* Possible bad flows are checked before filling out props so in case
* of an error it will still be zeroed out.
+ * Use native port in case of reps
*/
- err = mlx5_query_port_ptys(mdev, out, sizeof(out), MLX5_PTYS_EN,
- mdev_port_num);
+ if (dev->is_rep)
+ err = mlx5_query_port_ptys(mdev, out, sizeof(out), MLX5_PTYS_EN,
+ 1);
+ else
+ err = mlx5_query_port_ptys(mdev, out, sizeof(out), MLX5_PTYS_EN,
+ mdev_port_num);
if (err)
goto out;
ext = MLX5_CAP_PCAM_FEATURE(dev->mdev, ptys_extended_ethernet);
return err;
}
-struct mlx5_ib_vlan_info {
- u16 vlan_id;
- bool vlan;
-};
-
-static int get_lower_dev_vlan(struct net_device *lower_dev, void *data)
-{
- struct mlx5_ib_vlan_info *vlan_info = data;
-
- if (is_vlan_dev(lower_dev)) {
- vlan_info->vlan = true;
- vlan_info->vlan_id = vlan_dev_vlan_id(lower_dev);
- }
- /* We are interested only in first level vlan device, so
- * always return 1 to stop iterating over next level devices.
- */
- return 1;
-}
-
static int set_roce_addr(struct mlx5_ib_dev *dev, u8 port_num,
unsigned int index, const union ib_gid *gid,
const struct ib_gid_attr *attr)
{
enum ib_gid_type gid_type = IB_GID_TYPE_IB;
- struct mlx5_ib_vlan_info vlan_info = { };
+ u16 vlan_id = 0xffff;
u8 roce_version = 0;
u8 roce_l3_type = 0;
u8 mac[ETH_ALEN];
+ int ret;
if (gid) {
gid_type = attr->gid_type;
- ether_addr_copy(mac, attr->ndev->dev_addr);
-
- if (is_vlan_dev(attr->ndev)) {
- vlan_info.vlan = true;
- vlan_info.vlan_id = vlan_dev_vlan_id(attr->ndev);
- } else {
- /* If the netdev is upper device and if it's lower
- * lower device is vlan device, consider vlan id of
- * the lower vlan device for this gid entry.
- */
- rcu_read_lock();
- netdev_walk_all_lower_dev_rcu(attr->ndev,
- get_lower_dev_vlan, &vlan_info);
- rcu_read_unlock();
- }
+ ret = rdma_read_gid_l2_fields(attr, &vlan_id, &mac[0]);
+ if (ret)
+ return ret;
}
switch (gid_type) {
return mlx5_core_roce_gid_set(dev->mdev, index, roce_version,
roce_l3_type, gid->raw, mac,
- vlan_info.vlan, vlan_info.vlan_id,
+ vlan_id < VLAN_CFI_MASK, vlan_id,
port_num);
}
{
int ret;
- /* Only link layer == ethernet is valid for representors */
+ /* Only link layer == ethernet is valid for representors
+ * and we always use port 1
+ */
ret = mlx5_query_port_roce(ibdev, port, props);
if (ret || !props)
return ret;
print_lib_caps(dev, context->lib_caps);
if (dev->lag_active) {
- u8 port = mlx5_core_native_port_num(dev->mdev);
+ u8 port = mlx5_core_native_port_num(dev->mdev) - 1;
atomic_set(&context->tx_port_affinity,
atomic_add_return(
- 1, &dev->roce[port].tx_port_affinity));
+ 1, &dev->port[port].roce.tx_port_affinity));
}
return 0;
struct vm_area_struct *vma,
struct mlx5_ib_ucontext *context)
{
- if (vma->vm_end - vma->vm_start != PAGE_SIZE)
+ if ((vma->vm_end - vma->vm_start != PAGE_SIZE) ||
+ !(vma->vm_flags & VM_SHARED))
return -EINVAL;
if (get_index(vma->vm_pgoff) != MLX5_IB_CLOCK_INFO_V1)
return -EOPNOTSUPP;
- if (vma->vm_flags & VM_WRITE)
+ if (vma->vm_flags & (VM_WRITE | VM_EXEC))
return -EPERM;
vma->vm_flags &= ~VM_MAYWRITE;
- if (!dev->mdev->clock_info_page)
+ if (!dev->mdev->clock_info)
return -EOPNOTSUPP;
- return rdma_user_mmap_page(&context->ibucontext, vma,
- dev->mdev->clock_info_page, PAGE_SIZE);
+ return vm_insert_page(vma, vma->vm_start,
+ virt_to_page(dev->mdev->clock_info));
}
static int uar_mmap(struct mlx5_ib_dev *dev, enum mlx5_ib_mmap_cmd cmd,
return 0;
}
-struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev,
- struct ib_ucontext *context,
- struct ib_dm_alloc_attr *attr,
- struct uverbs_attr_bundle *attrs)
+static inline int check_dm_type_support(struct mlx5_ib_dev *dev,
+ u32 type)
{
- u64 act_size = roundup(attr->length, MLX5_MEMIC_BASE_SIZE);
- struct mlx5_memic *memic = &to_mdev(ibdev)->memic;
- phys_addr_t memic_addr;
- struct mlx5_ib_dm *dm;
+ switch (type) {
+ case MLX5_IB_UAPI_DM_TYPE_MEMIC:
+ if (!MLX5_CAP_DEV_MEM(dev->mdev, memic))
+ return -EOPNOTSUPP;
+ break;
+ case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+ if (!capable(CAP_SYS_RAWIO) ||
+ !capable(CAP_NET_RAW))
+ return -EPERM;
+
+ if (!(MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner) ||
+ MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, sw_owner)))
+ return -EOPNOTSUPP;
+ break;
+ }
+
+ return 0;
+}
+
+static int handle_alloc_dm_memic(struct ib_ucontext *ctx,
+ struct mlx5_ib_dm *dm,
+ struct ib_dm_alloc_attr *attr,
+ struct uverbs_attr_bundle *attrs)
+{
+ struct mlx5_dm *dm_db = &to_mdev(ctx->device)->dm;
u64 start_offset;
u32 page_idx;
int err;
- dm = kzalloc(sizeof(*dm), GFP_KERNEL);
- if (!dm)
- return ERR_PTR(-ENOMEM);
-
- mlx5_ib_dbg(to_mdev(ibdev), "alloc_memic req: user_length=0x%llx act_length=0x%llx log_alignment=%d\n",
- attr->length, act_size, attr->alignment);
+ dm->size = roundup(attr->length, MLX5_MEMIC_BASE_SIZE);
- err = mlx5_cmd_alloc_memic(memic, &memic_addr,
- act_size, attr->alignment);
+ err = mlx5_cmd_alloc_memic(dm_db, &dm->dev_addr,
+ dm->size, attr->alignment);
if (err)
- goto err_free;
+ return err;
- start_offset = memic_addr & ~PAGE_MASK;
- page_idx = (memic_addr - memic->dev->bar_addr -
- MLX5_CAP64_DEV_MEM(memic->dev, memic_bar_start_addr)) >>
+ page_idx = (dm->dev_addr - pci_resource_start(dm_db->dev->pdev, 0) -
+ MLX5_CAP64_DEV_MEM(dm_db->dev, memic_bar_start_addr)) >>
PAGE_SHIFT;
+ err = uverbs_copy_to(attrs,
+ MLX5_IB_ATTR_ALLOC_DM_RESP_PAGE_INDEX,
+ &page_idx, sizeof(page_idx));
+ if (err)
+ goto err_dealloc;
+
+ start_offset = dm->dev_addr & ~PAGE_MASK;
err = uverbs_copy_to(attrs,
MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET,
&start_offset, sizeof(start_offset));
if (err)
goto err_dealloc;
+ bitmap_set(to_mucontext(ctx)->dm_pages, page_idx,
+ DIV_ROUND_UP(dm->size, PAGE_SIZE));
+
+ return 0;
+
+err_dealloc:
+ mlx5_cmd_dealloc_memic(dm_db, dm->dev_addr, dm->size);
+
+ return err;
+}
+
+static int handle_alloc_dm_sw_icm(struct ib_ucontext *ctx,
+ struct mlx5_ib_dm *dm,
+ struct ib_dm_alloc_attr *attr,
+ struct uverbs_attr_bundle *attrs,
+ int type)
+{
+ struct mlx5_dm *dm_db = &to_mdev(ctx->device)->dm;
+ u64 act_size;
+ int err;
+
+ /* Allocation size must a multiple of the basic block size
+ * and a power of 2.
+ */
+ act_size = roundup(attr->length, MLX5_SW_ICM_BLOCK_SIZE(dm_db->dev));
+ act_size = roundup_pow_of_two(act_size);
+
+ dm->size = act_size;
+ err = mlx5_cmd_alloc_sw_icm(dm_db, type, act_size,
+ to_mucontext(ctx)->devx_uid, &dm->dev_addr,
+ &dm->icm_dm.obj_id);
+ if (err)
+ return err;
+
err = uverbs_copy_to(attrs,
- MLX5_IB_ATTR_ALLOC_DM_RESP_PAGE_INDEX,
- &page_idx, sizeof(page_idx));
+ MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET,
+ &dm->dev_addr, sizeof(dm->dev_addr));
if (err)
- goto err_dealloc;
+ mlx5_cmd_dealloc_sw_icm(dm_db, type, dm->size,
+ to_mucontext(ctx)->devx_uid,
+ dm->dev_addr, dm->icm_dm.obj_id);
+
+ return err;
+}
+
+struct ib_dm *mlx5_ib_alloc_dm(struct ib_device *ibdev,
+ struct ib_ucontext *context,
+ struct ib_dm_alloc_attr *attr,
+ struct uverbs_attr_bundle *attrs)
+{
+ struct mlx5_ib_dm *dm;
+ enum mlx5_ib_uapi_dm_type type;
+ int err;
- bitmap_set(to_mucontext(context)->dm_pages, page_idx,
- DIV_ROUND_UP(act_size, PAGE_SIZE));
+ err = uverbs_get_const_default(&type, attrs,
+ MLX5_IB_ATTR_ALLOC_DM_REQ_TYPE,
+ MLX5_IB_UAPI_DM_TYPE_MEMIC);
+ if (err)
+ return ERR_PTR(err);
- dm->dev_addr = memic_addr;
+ mlx5_ib_dbg(to_mdev(ibdev), "alloc_dm req: dm_type=%d user_length=0x%llx log_alignment=%d\n",
+ type, attr->length, attr->alignment);
+
+ err = check_dm_type_support(to_mdev(ibdev), type);
+ if (err)
+ return ERR_PTR(err);
+
+ dm = kzalloc(sizeof(*dm), GFP_KERNEL);
+ if (!dm)
+ return ERR_PTR(-ENOMEM);
+
+ dm->type = type;
+
+ switch (type) {
+ case MLX5_IB_UAPI_DM_TYPE_MEMIC:
+ err = handle_alloc_dm_memic(context, dm,
+ attr,
+ attrs);
+ break;
+ case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+ case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
+ err = handle_alloc_dm_sw_icm(context, dm, attr, attrs, type);
+ break;
+ default:
+ err = -EOPNOTSUPP;
+ }
+
+ if (err)
+ goto err_free;
return &dm->ibdm;
-err_dealloc:
- mlx5_cmd_dealloc_memic(memic, memic_addr,
- act_size);
err_free:
kfree(dm);
return ERR_PTR(err);
}
-int mlx5_ib_dealloc_dm(struct ib_dm *ibdm)
+int mlx5_ib_dealloc_dm(struct ib_dm *ibdm, struct uverbs_attr_bundle *attrs)
{
- struct mlx5_memic *memic = &to_mdev(ibdm->device)->memic;
+ struct mlx5_ib_ucontext *ctx = rdma_udata_to_drv_context(
+ &attrs->driver_udata, struct mlx5_ib_ucontext, ibucontext);
+ struct mlx5_dm *dm_db = &to_mdev(ibdm->device)->dm;
struct mlx5_ib_dm *dm = to_mdm(ibdm);
- u64 act_size = roundup(dm->ibdm.length, MLX5_MEMIC_BASE_SIZE);
u32 page_idx;
int ret;
- ret = mlx5_cmd_dealloc_memic(memic, dm->dev_addr, act_size);
- if (ret)
- return ret;
+ switch (dm->type) {
+ case MLX5_IB_UAPI_DM_TYPE_MEMIC:
+ ret = mlx5_cmd_dealloc_memic(dm_db, dm->dev_addr, dm->size);
+ if (ret)
+ return ret;
- page_idx = (dm->dev_addr - memic->dev->bar_addr -
- MLX5_CAP64_DEV_MEM(memic->dev, memic_bar_start_addr)) >>
- PAGE_SHIFT;
- bitmap_clear(to_mucontext(ibdm->uobject->context)->dm_pages,
- page_idx,
- DIV_ROUND_UP(act_size, PAGE_SIZE));
+ page_idx = (dm->dev_addr -
+ pci_resource_start(dm_db->dev->pdev, 0) -
+ MLX5_CAP64_DEV_MEM(dm_db->dev,
+ memic_bar_start_addr)) >>
+ PAGE_SHIFT;
+ bitmap_clear(ctx->dm_pages, page_idx,
+ DIV_ROUND_UP(dm->size, PAGE_SIZE));
+ break;
+ case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+ case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
+ ret = mlx5_cmd_dealloc_sw_icm(dm_db, dm->type, dm->size,
+ ctx->devx_uid, dm->dev_addr,
+ dm->icm_dm.obj_id);
+ if (ret)
+ return ret;
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
kfree(dm);
return 0;
}
-static int mlx5_ib_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata)
+static int mlx5_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct mlx5_ib_pd *pd = to_mpd(ibpd);
struct ib_device *ibdev = ibpd->device;
u32 out[MLX5_ST_SZ_DW(alloc_pd_out)] = {};
u32 in[MLX5_ST_SZ_DW(alloc_pd_in)] = {};
u16 uid = 0;
+ struct mlx5_ib_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct mlx5_ib_ucontext, ibucontext);
- uid = context ? to_mucontext(context)->devx_uid : 0;
+ uid = context ? context->devx_uid : 0;
MLX5_SET(alloc_pd_in, in, opcode, MLX5_CMD_OP_ALLOC_PD);
MLX5_SET(alloc_pd_in, in, uid, uid);
err = mlx5_cmd_exec(to_mdev(ibdev)->mdev, in, sizeof(in),
pd->pdn = MLX5_GET(alloc_pd_out, out, pd);
pd->uid = uid;
- if (context) {
+ if (udata) {
resp.pdn = pd->pdn;
if (ib_copy_to_udata(udata, &resp, sizeof(resp))) {
mlx5_cmd_dealloc_pd(to_mdev(ibdev)->mdev, pd->pdn, uid);
return 0;
}
-static void mlx5_ib_dealloc_pd(struct ib_pd *pd)
+static void mlx5_ib_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata)
{
struct mlx5_ib_dev *mdev = to_mdev(pd->device);
struct mlx5_ib_pd *mpd = to_mpd(pd);
if (ft_type == MLX5_IB_FT_RX) {
fn_type = MLX5_FLOW_NAMESPACE_BYPASS;
prio = &dev->flow_db->prios[priority];
- if (!dev->rep &&
+ if (!dev->is_rep &&
MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, decap))
flags |= MLX5_FLOW_TABLE_TUNNEL_EN_DECAP;
- if (!dev->rep &&
+ if (!dev->is_rep &&
MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev,
reformat_l3_tunnel_to_l2))
flags |= MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
log_max_ft_size));
fn_type = MLX5_FLOW_NAMESPACE_EGRESS;
prio = &dev->flow_db->egress_prios[priority];
- if (!dev->rep &&
+ if (!dev->is_rep &&
MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, reformat))
flags |= MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
}
if (!ns)
return ERR_PTR(-ENOTSUPP);
- if (num_entries > max_table_size)
- return ERR_PTR(-ENOMEM);
+ max_table_size = min_t(int, num_entries, max_table_size);
ft = prio->flow_table;
if (!ft)
- return _get_prio(ns, prio, priority, num_entries, num_groups,
+ return _get_prio(ns, prio, priority, max_table_size, num_groups,
flags);
return prio;
if (!is_valid_attr(dev->mdev, flow_attr))
return ERR_PTR(-EINVAL);
- if (dev->rep && is_egress)
+ if (dev->is_rep && is_egress)
return ERR_PTR(-EINVAL);
spec = kvzalloc(sizeof(*spec), GFP_KERNEL);
if (!flow_is_multicast_only(flow_attr))
set_underlay_qp(dev, spec, underlay_qpn);
- if (dev->rep) {
+ if (dev->is_rep) {
void *misc;
+ if (!dev->port[flow_attr->port - 1].rep) {
+ err = -EINVAL;
+ goto free;
+ }
misc = MLX5_ADDR_OF(fte_match_param, spec->match_value,
misc_parameters);
MLX5_SET(fte_match_set_misc, misc, source_port,
- dev->rep->vport);
+ dev->port[flow_attr->port - 1].rep->vport);
misc = MLX5_ADDR_OF(fte_match_param, spec->match_criteria,
misc_parameters);
MLX5_SET_TO_ONES(fte_match_set_misc, misc, source_port);
bool mcast)
{
struct mlx5_flow_namespace *ns = NULL;
- struct mlx5_ib_flow_prio *prio;
- int max_table_size;
+ struct mlx5_ib_flow_prio *prio = NULL;
+ int max_table_size = 0;
u32 flags = 0;
int priority;
+ if (mcast)
+ priority = MLX5_IB_FLOW_MCAST_PRIO;
+ else
+ priority = ib_prio_to_core_prio(fs_matcher->priority, false);
+
if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_BYPASS) {
max_table_size = BIT(MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev,
log_max_ft_size));
if (MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev,
reformat_l3_tunnel_to_l2))
flags |= MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
- } else { /* Can only be MLX5_FLOW_NAMESPACE_EGRESS */
- max_table_size = BIT(MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev,
- log_max_ft_size));
+ } else if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_EGRESS) {
+ max_table_size = BIT(
+ MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, log_max_ft_size));
if (MLX5_CAP_FLOWTABLE_NIC_TX(dev->mdev, reformat))
flags |= MLX5_FLOW_TABLE_TUNNEL_EN_REFORMAT;
+ } else if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_FDB) {
+ max_table_size = BIT(
+ MLX5_CAP_ESW_FLOWTABLE_FDB(dev->mdev, log_max_ft_size));
+ priority = FDB_BYPASS_PATH;
}
- if (max_table_size < MLX5_FS_MAX_ENTRIES)
- return ERR_PTR(-ENOMEM);
-
- if (mcast)
- priority = MLX5_IB_FLOW_MCAST_PRIO;
- else
- priority = ib_prio_to_core_prio(fs_matcher->priority, false);
+ max_table_size = min_t(int, max_table_size, MLX5_FS_MAX_ENTRIES);
ns = mlx5_get_flow_namespace(dev->mdev, fs_matcher->ns_type);
if (!ns)
if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_BYPASS)
prio = &dev->flow_db->prios[priority];
- else
+ else if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_EGRESS)
prio = &dev->flow_db->egress_prios[priority];
+ else if (fs_matcher->ns_type == MLX5_FLOW_NAMESPACE_FDB)
+ prio = &dev->flow_db->fdb;
+
+ if (!prio)
+ return ERR_PTR(-EINVAL);
if (prio->flow_table)
return prio;
- return _get_prio(ns, prio, priority, MLX5_FS_MAX_ENTRIES,
+ return _get_prio(ns, prio, priority, max_table_size,
MLX5_FS_MAX_TYPES, flags);
}
int err;
int port;
- for (port = 1; port <= dev->num_ports; port++) {
+ for (port = 1; port <= ARRAY_SIZE(dev->mdev->port_caps); port++) {
dev->mdev->port_caps[port - 1].has_smi = false;
if (MLX5_CAP_GEN(dev->mdev, port_type) ==
MLX5_CAP_PORT_TYPE_IB) {
mlx5_query_ext_port_caps(dev, port);
}
-static int get_port_caps(struct mlx5_ib_dev *dev, u8 port)
+static int __get_port_caps(struct mlx5_ib_dev *dev, u8 port)
{
struct ib_device_attr *dprops = NULL;
struct ib_port_attr *pprops = NULL;
if (!dprops)
goto out;
- err = set_has_smi_cap(dev);
- if (err)
- goto out;
-
err = mlx5_ib_query_device(&dev->ib_dev, dprops, &uhw);
if (err) {
mlx5_ib_warn(dev, "query_device failed %d\n", err);
return err;
}
+static int get_port_caps(struct mlx5_ib_dev *dev, u8 port)
+{
+ /* For representors use port 1, is this is the only native
+ * port
+ */
+ if (dev->is_rep)
+ return __get_port_caps(dev, 1);
+ return __get_port_caps(dev, port);
+}
+
static void destroy_umrc_res(struct mlx5_ib_dev *dev)
{
int err;
mlx5_ib_warn(dev, "mr cache cleanup failed\n");
if (dev->umrc.qp)
- mlx5_ib_destroy_qp(dev->umrc.qp);
+ mlx5_ib_destroy_qp(dev->umrc.qp, NULL);
if (dev->umrc.cq)
ib_free_cq(dev->umrc.cq);
if (dev->umrc.pd)
return 0;
error_4:
- mlx5_ib_destroy_qp(qp);
+ mlx5_ib_destroy_qp(qp, NULL);
dev->umrc.qp = NULL;
error_3:
devr->p0->uobject = NULL;
atomic_set(&devr->p0->usecnt, 0);
- ret = mlx5_ib_alloc_pd(devr->p0, NULL, NULL);
+ ret = mlx5_ib_alloc_pd(devr->p0, NULL);
if (ret)
goto error0;
- devr->c0 = mlx5_ib_create_cq(&dev->ib_dev, &cq_attr, NULL, NULL);
+ devr->c0 = mlx5_ib_create_cq(&dev->ib_dev, &cq_attr, NULL);
if (IS_ERR(devr->c0)) {
ret = PTR_ERR(devr->c0);
goto error1;
devr->c0->cq_context = NULL;
atomic_set(&devr->c0->usecnt, 0);
- devr->x0 = mlx5_ib_alloc_xrcd(&dev->ib_dev, NULL, NULL);
+ devr->x0 = mlx5_ib_alloc_xrcd(&dev->ib_dev, NULL);
if (IS_ERR(devr->x0)) {
ret = PTR_ERR(devr->x0);
goto error2;
mutex_init(&devr->x0->tgt_qp_mutex);
INIT_LIST_HEAD(&devr->x0->tgt_qp_list);
- devr->x1 = mlx5_ib_alloc_xrcd(&dev->ib_dev, NULL, NULL);
+ devr->x1 = mlx5_ib_alloc_xrcd(&dev->ib_dev, NULL);
if (IS_ERR(devr->x1)) {
ret = PTR_ERR(devr->x1);
goto error3;
attr.ext.cq = devr->c0;
attr.ext.xrc.xrcd = devr->x0;
- devr->s0 = mlx5_ib_create_srq(devr->p0, &attr, NULL);
- if (IS_ERR(devr->s0)) {
- ret = PTR_ERR(devr->s0);
+ devr->s0 = rdma_zalloc_drv_obj(ibdev, ib_srq);
+ if (!devr->s0) {
+ ret = -ENOMEM;
goto error4;
}
+
devr->s0->device = &dev->ib_dev;
devr->s0->pd = devr->p0;
- devr->s0->uobject = NULL;
- devr->s0->event_handler = NULL;
- devr->s0->srq_context = NULL;
devr->s0->srq_type = IB_SRQT_XRC;
devr->s0->ext.xrc.xrcd = devr->x0;
devr->s0->ext.cq = devr->c0;
+ ret = mlx5_ib_create_srq(devr->s0, &attr, NULL);
+ if (ret)
+ goto err_create;
+
atomic_inc(&devr->s0->ext.xrc.xrcd->usecnt);
atomic_inc(&devr->s0->ext.cq->usecnt);
atomic_inc(&devr->p0->usecnt);
attr.attr.max_sge = 1;
attr.attr.max_wr = 1;
attr.srq_type = IB_SRQT_BASIC;
- devr->s1 = mlx5_ib_create_srq(devr->p0, &attr, NULL);
- if (IS_ERR(devr->s1)) {
- ret = PTR_ERR(devr->s1);
+ devr->s1 = rdma_zalloc_drv_obj(ibdev, ib_srq);
+ if (!devr->s1) {
+ ret = -ENOMEM;
goto error5;
}
+
devr->s1->device = &dev->ib_dev;
devr->s1->pd = devr->p0;
- devr->s1->uobject = NULL;
- devr->s1->event_handler = NULL;
- devr->s1->srq_context = NULL;
devr->s1->srq_type = IB_SRQT_BASIC;
devr->s1->ext.cq = devr->c0;
+
+ ret = mlx5_ib_create_srq(devr->s1, &attr, NULL);
+ if (ret)
+ goto error6;
+
atomic_inc(&devr->p0->usecnt);
atomic_set(&devr->s1->usecnt, 0);
return 0;
+error6:
+ kfree(devr->s1);
error5:
- mlx5_ib_destroy_srq(devr->s0);
+ mlx5_ib_destroy_srq(devr->s0, NULL);
+err_create:
+ kfree(devr->s0);
error4:
- mlx5_ib_dealloc_xrcd(devr->x1);
+ mlx5_ib_dealloc_xrcd(devr->x1, NULL);
error3:
- mlx5_ib_dealloc_xrcd(devr->x0);
+ mlx5_ib_dealloc_xrcd(devr->x0, NULL);
error2:
- mlx5_ib_destroy_cq(devr->c0);
+ mlx5_ib_destroy_cq(devr->c0, NULL);
error1:
- mlx5_ib_dealloc_pd(devr->p0);
+ mlx5_ib_dealloc_pd(devr->p0, NULL);
error0:
kfree(devr->p0);
return ret;
static void destroy_dev_resources(struct mlx5_ib_resources *devr)
{
- struct mlx5_ib_dev *dev =
- container_of(devr, struct mlx5_ib_dev, devr);
int port;
- mlx5_ib_destroy_srq(devr->s1);
- mlx5_ib_destroy_srq(devr->s0);
- mlx5_ib_dealloc_xrcd(devr->x0);
- mlx5_ib_dealloc_xrcd(devr->x1);
- mlx5_ib_destroy_cq(devr->c0);
- mlx5_ib_dealloc_pd(devr->p0);
+ mlx5_ib_destroy_srq(devr->s1, NULL);
+ kfree(devr->s1);
+ mlx5_ib_destroy_srq(devr->s0, NULL);
+ kfree(devr->s0);
+ mlx5_ib_dealloc_xrcd(devr->x0, NULL);
+ mlx5_ib_dealloc_xrcd(devr->x1, NULL);
+ mlx5_ib_destroy_cq(devr->c0, NULL);
+ mlx5_ib_dealloc_pd(devr->p0, NULL);
kfree(devr->p0);
/* Make sure no change P_Key work items are still executing */
- for (port = 0; port < dev->num_ports; ++port)
+ for (port = 0; port < ARRAY_SIZE(devr->ports); ++port)
cancel_work_sync(&devr->ports[port].pkey_change_work);
}
{
int err;
- dev->roce[port_num].nb.notifier_call = mlx5_netdev_event;
- err = register_netdevice_notifier(&dev->roce[port_num].nb);
+ dev->port[port_num].roce.nb.notifier_call = mlx5_netdev_event;
+ err = register_netdevice_notifier(&dev->port[port_num].roce.nb);
if (err) {
- dev->roce[port_num].nb.notifier_call = NULL;
+ dev->port[port_num].roce.nb.notifier_call = NULL;
return err;
}
static void mlx5_remove_netdev_notifier(struct mlx5_ib_dev *dev, u8 port_num)
{
- if (dev->roce[port_num].nb.notifier_call) {
- unregister_netdevice_notifier(&dev->roce[port_num].nb);
- dev->roce[port_num].nb.notifier_call = NULL;
+ if (dev->port[port_num].roce.nb.notifier_call) {
+ unregister_netdevice_notifier(&dev->port[port_num].roce.nb);
+ dev->port[port_num].roce.nb.notifier_call = NULL;
}
}
mlx5_ib_err(ibdev, "Failed to unaffiliate port %u\n",
port_num + 1);
- ibdev->roce[port_num].last_port_state = IB_PORT_DOWN;
+ ibdev->port[port_num].roce.last_port_state = IB_PORT_DOWN;
}
/* The mlx5_ib_multiport_mutex should be held when calling this function */
UA_MANDATORY),
UVERBS_ATTR_PTR_OUT(MLX5_IB_ATTR_ALLOC_DM_RESP_PAGE_INDEX,
UVERBS_ATTR_TYPE(u16),
- UA_MANDATORY));
+ UA_OPTIONAL),
+ UVERBS_ATTR_CONST_IN(MLX5_IB_ATTR_ALLOC_DM_REQ_TYPE,
+ enum mlx5_ib_uapi_dm_type,
+ UA_OPTIONAL));
ADD_UVERBS_ATTRIBUTES_SIMPLE(
mlx5_ib_flow_action,
return &mcounters->ibcntrs;
}
-void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)
+static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)
{
+ struct mlx5_core_dev *mdev = dev->mdev;
+
mlx5_ib_cleanup_multiport_master(dev);
if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
srcu_barrier(&dev->mr_srcu);
cleanup_srcu_struct(&dev->mr_srcu);
}
- kfree(dev->port);
+
+ WARN_ON(!bitmap_empty(dev->dm.memic_alloc_pages, MLX5_MAX_MEMIC_PAGES));
+
+ WARN_ON(dev->dm.steering_sw_icm_alloc_blocks &&
+ !bitmap_empty(
+ dev->dm.steering_sw_icm_alloc_blocks,
+ BIT(MLX5_CAP_DEV_MEM(mdev, log_steering_sw_icm_size) -
+ MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev))));
+
+ kfree(dev->dm.steering_sw_icm_alloc_blocks);
+
+ WARN_ON(dev->dm.header_modify_sw_icm_alloc_blocks &&
+ !bitmap_empty(dev->dm.header_modify_sw_icm_alloc_blocks,
+ BIT(MLX5_CAP_DEV_MEM(
+ mdev, log_header_modify_sw_icm_size) -
+ MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev))));
+
+ kfree(dev->dm.header_modify_sw_icm_alloc_blocks);
}
-int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
+static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
{
struct mlx5_core_dev *mdev = dev->mdev;
+ u64 header_modify_icm_blocks = 0;
+ u64 steering_icm_blocks = 0;
int err;
int i;
- dev->port = kcalloc(dev->num_ports, sizeof(*dev->port),
- GFP_KERNEL);
- if (!dev->port)
- return -ENOMEM;
-
for (i = 0; i < dev->num_ports; i++) {
spin_lock_init(&dev->port[i].mp.mpi_lock);
- rwlock_init(&dev->roce[i].netdev_lock);
+ rwlock_init(&dev->port[i].roce.netdev_lock);
+ dev->port[i].roce.dev = dev;
+ dev->port[i].roce.native_port_num = i + 1;
+ dev->port[i].roce.last_port_state = IB_PORT_DOWN;
}
err = mlx5_ib_init_multiport_master(dev);
if (err)
- goto err_free_port;
+ return err;
+
+ err = set_has_smi_cap(dev);
+ if (err)
+ return err;
if (!mlx5_core_mp_enabled(mdev)) {
for (i = 1; i <= dev->num_ports; i++) {
INIT_LIST_HEAD(&dev->qp_list);
spin_lock_init(&dev->reset_flow_resource_lock);
- spin_lock_init(&dev->memic.memic_lock);
- dev->memic.dev = mdev;
+ if (MLX5_CAP_GEN_64(mdev, general_obj_types) &
+ MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM) {
+ if (MLX5_CAP64_DEV_MEM(mdev, steering_sw_icm_start_address)) {
+ steering_icm_blocks =
+ BIT(MLX5_CAP_DEV_MEM(mdev,
+ log_steering_sw_icm_size) -
+ MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev));
+
+ dev->dm.steering_sw_icm_alloc_blocks =
+ kcalloc(BITS_TO_LONGS(steering_icm_blocks),
+ sizeof(unsigned long), GFP_KERNEL);
+ if (!dev->dm.steering_sw_icm_alloc_blocks)
+ goto err_mp;
+ }
+
+ if (MLX5_CAP64_DEV_MEM(mdev,
+ header_modify_sw_icm_start_address)) {
+ header_modify_icm_blocks = BIT(
+ MLX5_CAP_DEV_MEM(
+ mdev, log_header_modify_sw_icm_size) -
+ MLX5_LOG_SW_ICM_BLOCK_SIZE(mdev));
+
+ dev->dm.header_modify_sw_icm_alloc_blocks =
+ kcalloc(BITS_TO_LONGS(header_modify_icm_blocks),
+ sizeof(unsigned long), GFP_KERNEL);
+ if (!dev->dm.header_modify_sw_icm_alloc_blocks)
+ goto err_dm;
+ }
+ }
+
+ spin_lock_init(&dev->dm.lock);
+ dev->dm.dev = mdev;
if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
err = init_srcu_struct(&dev->mr_srcu);
if (err)
- goto err_mp;
+ goto err_dm;
}
return 0;
+
+err_dm:
+ kfree(dev->dm.steering_sw_icm_alloc_blocks);
+ kfree(dev->dm.header_modify_sw_icm_alloc_blocks);
+
err_mp:
mlx5_ib_cleanup_multiport_master(dev);
-err_free_port:
- kfree(dev->port);
-
return -ENOMEM;
}
return 0;
}
-int mlx5_ib_stage_rep_flow_db_init(struct mlx5_ib_dev *dev)
-{
- struct mlx5_ib_dev *nic_dev;
-
- nic_dev = mlx5_ib_get_uplink_ibdev(dev->mdev->priv.eswitch);
-
- if (!nic_dev)
- return -EINVAL;
-
- dev->flow_db = nic_dev->flow_db;
-
- return 0;
-}
-
static void mlx5_ib_stage_flow_db_cleanup(struct mlx5_ib_dev *dev)
{
kfree(dev->flow_db);
.req_notify_cq = mlx5_ib_arm_cq,
.rereg_user_mr = mlx5_ib_rereg_user_mr,
.resize_cq = mlx5_ib_resize_cq,
+
+ INIT_RDMA_OBJ_SIZE(ib_ah, mlx5_ib_ah, ibah),
INIT_RDMA_OBJ_SIZE(ib_pd, mlx5_ib_pd, ibpd),
+ INIT_RDMA_OBJ_SIZE(ib_srq, mlx5_ib_srq, ibsrq),
INIT_RDMA_OBJ_SIZE(ib_ucontext, mlx5_ib_ucontext, ibucontext),
};
.reg_dm_mr = mlx5_ib_reg_dm_mr,
};
-int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)
+static int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)
{
struct mlx5_core_dev *mdev = dev->mdev;
int err;
ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_xrc_ops);
}
- if (MLX5_CAP_DEV_MEM(mdev, memic))
+ if (MLX5_CAP_DEV_MEM(mdev, memic) ||
+ MLX5_CAP_GEN_64(dev->mdev, general_obj_types) &
+ MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM)
ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_dm_ops);
if (mlx5_accel_ipsec_device_caps(dev->mdev) &
.query_port = mlx5_ib_rep_query_port,
};
-int mlx5_ib_stage_rep_non_default_cb(struct mlx5_ib_dev *dev)
+static int mlx5_ib_stage_rep_non_default_cb(struct mlx5_ib_dev *dev)
{
ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_port_rep_ops);
return 0;
static int mlx5_ib_stage_common_roce_init(struct mlx5_ib_dev *dev)
{
u8 port_num;
- int i;
-
- for (i = 0; i < dev->num_ports; i++) {
- dev->roce[i].dev = dev;
- dev->roce[i].native_port_num = i + 1;
- dev->roce[i].last_port_state = IB_PORT_DOWN;
- }
dev->ib_dev.uverbs_ex_cmd_mask |=
(1ull << IB_USER_VERBS_EX_CMD_CREATE_WQ) |
port_num = mlx5_core_native_port_num(dev->mdev) - 1;
+ /* Register only for native ports */
return mlx5_add_netdev_notifier(dev, port_num);
}
mlx5_remove_netdev_notifier(dev, port_num);
}
-int mlx5_ib_stage_rep_roce_init(struct mlx5_ib_dev *dev)
+static int mlx5_ib_stage_rep_roce_init(struct mlx5_ib_dev *dev)
{
struct mlx5_core_dev *mdev = dev->mdev;
enum rdma_link_layer ll;
return err;
}
-void mlx5_ib_stage_rep_roce_cleanup(struct mlx5_ib_dev *dev)
+static void mlx5_ib_stage_rep_roce_cleanup(struct mlx5_ib_dev *dev)
{
mlx5_ib_stage_common_roce_cleanup(dev);
}
}
}
-int mlx5_ib_stage_dev_res_init(struct mlx5_ib_dev *dev)
+static int mlx5_ib_stage_dev_res_init(struct mlx5_ib_dev *dev)
{
return create_dev_resources(&dev->devr);
}
-void mlx5_ib_stage_dev_res_cleanup(struct mlx5_ib_dev *dev)
+static void mlx5_ib_stage_dev_res_cleanup(struct mlx5_ib_dev *dev)
{
destroy_dev_resources(&dev->devr);
}
.get_hw_stats = mlx5_ib_get_hw_stats,
};
-int mlx5_ib_stage_counters_init(struct mlx5_ib_dev *dev)
+static int mlx5_ib_stage_counters_init(struct mlx5_ib_dev *dev)
{
if (MLX5_CAP_GEN(dev->mdev, max_qp_cnt)) {
ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_hw_stats_ops);
return 0;
}
-void mlx5_ib_stage_counters_cleanup(struct mlx5_ib_dev *dev)
+static void mlx5_ib_stage_counters_cleanup(struct mlx5_ib_dev *dev)
{
if (MLX5_CAP_GEN(dev->mdev, max_qp_cnt))
mlx5_ib_dealloc_counters(dev);
mlx5_put_uars_page(dev->mdev, dev->mdev->priv.uar);
}
-int mlx5_ib_stage_bfrag_init(struct mlx5_ib_dev *dev)
+static int mlx5_ib_stage_bfrag_init(struct mlx5_ib_dev *dev)
{
int err;
return err;
}
-void mlx5_ib_stage_bfrag_cleanup(struct mlx5_ib_dev *dev)
+static void mlx5_ib_stage_bfrag_cleanup(struct mlx5_ib_dev *dev)
{
mlx5_free_bfreg(dev->mdev, &dev->fp_bfreg);
mlx5_free_bfreg(dev->mdev, &dev->bfreg);
}
-int mlx5_ib_stage_ib_reg_init(struct mlx5_ib_dev *dev)
+static int mlx5_ib_stage_ib_reg_init(struct mlx5_ib_dev *dev)
{
const char *name;
return ib_register_device(&dev->ib_dev, name);
}
-void mlx5_ib_stage_pre_ib_reg_umr_cleanup(struct mlx5_ib_dev *dev)
+static void mlx5_ib_stage_pre_ib_reg_umr_cleanup(struct mlx5_ib_dev *dev)
{
destroy_umrc_res(dev);
}
-void mlx5_ib_stage_ib_reg_cleanup(struct mlx5_ib_dev *dev)
+static void mlx5_ib_stage_ib_reg_cleanup(struct mlx5_ib_dev *dev)
{
ib_unregister_device(&dev->ib_dev);
}
-int mlx5_ib_stage_post_ib_reg_umr_init(struct mlx5_ib_dev *dev)
+static int mlx5_ib_stage_post_ib_reg_umr_init(struct mlx5_ib_dev *dev)
{
return create_umr_res(dev);
}
if (profile->stage[stage].cleanup)
profile->stage[stage].cleanup(dev);
}
+
+ kfree(dev->port);
+ ib_dealloc_device(&dev->ib_dev);
}
void *__mlx5_ib_add(struct mlx5_ib_dev *dev,
STAGE_CREATE(MLX5_IB_STAGE_PRE_IB_REG_UMR,
NULL,
mlx5_ib_stage_pre_ib_reg_umr_cleanup),
+ STAGE_CREATE(MLX5_IB_STAGE_WHITELIST_UID,
+ mlx5_ib_stage_devx_init,
+ mlx5_ib_stage_devx_cleanup),
STAGE_CREATE(MLX5_IB_STAGE_IB_REG,
mlx5_ib_stage_ib_reg_init,
mlx5_ib_stage_ib_reg_cleanup),
enum rdma_link_layer ll;
struct mlx5_ib_dev *dev;
int port_type_cap;
+ int num_ports;
printk_once(KERN_INFO "%s", mlx5_version);
if (MLX5_ESWITCH_MANAGER(mdev) &&
mlx5_ib_eswitch_mode(mdev->priv.eswitch) == SRIOV_OFFLOADS) {
- mlx5_ib_register_vport_reps(mdev);
+ if (!mlx5_core_mp_enabled(mdev))
+ mlx5_ib_register_vport_reps(mdev);
return mdev;
}
if (mlx5_core_is_mp_slave(mdev) && ll == IB_LINK_LAYER_ETHERNET)
return mlx5_ib_add_slave_port(mdev);
+ num_ports = max(MLX5_CAP_GEN(mdev, num_ports),
+ MLX5_CAP_GEN(mdev, num_vhca_ports));
dev = ib_alloc_device(mlx5_ib_dev, ib_dev);
if (!dev)
return NULL;
+ dev->port = kcalloc(num_ports, sizeof(*dev->port),
+ GFP_KERNEL);
+ if (!dev->port) {
+ ib_dealloc_device((struct ib_device *)dev);
+ return NULL;
+ }
dev->mdev = mdev;
- dev->num_ports = max(MLX5_CAP_GEN(mdev, num_ports),
- MLX5_CAP_GEN(mdev, num_vhca_ports));
+ dev->num_ports = num_ports;
return __mlx5_ib_add(dev, &pf_profile);
}
dev = context;
__mlx5_ib_remove(dev, dev->profile, MLX5_IB_STAGE_MAX);
-
- ib_dealloc_device((struct ib_device *)dev);
}
static struct mlx5_interface mlx5_ib_interface = {
#include <rdma/mlx5-abi.h>
#include <rdma/uverbs_ioctl.h>
#include <rdma/mlx5_user_ioctl_cmds.h>
+#include <rdma/mlx5_user_ioctl_verbs.h>
#include "srq.h"
MLX5_MEMIC_BASE_SIZE = 1 << MLX5_MEMIC_BASE_ALIGN,
};
+#define MLX5_LOG_SW_ICM_BLOCK_SIZE(dev) \
+ (MLX5_CAP_DEV_MEM(dev, log_sw_icm_alloc_granularity))
+#define MLX5_SW_ICM_BLOCK_SIZE(dev) (1 << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev))
+
struct mlx5_ib_ucontext {
struct ib_ucontext ibucontext;
struct list_head db_page_list;
struct mlx5_ib_flow_prio egress_prios[MLX5_IB_NUM_FLOW_FT];
struct mlx5_ib_flow_prio sniffer[MLX5_IB_NUM_SNIFFER_FTS];
struct mlx5_ib_flow_prio egress[MLX5_IB_NUM_EGRESS_FTS];
+ struct mlx5_ib_flow_prio fdb;
struct mlx5_flow_table *lag_demux_ft;
/* Protect flow steering bypass flow tables
* when add/del flow rules.
struct mlx5_ib_dm {
struct ib_dm ibdm;
phys_addr_t dev_addr;
+ u32 type;
+ size_t size;
+ union {
+ struct {
+ u32 obj_id;
+ } icm_dm;
+ /* other dm types specific params should be added here */
+ };
};
#define MLX5_IB_MTT_PRESENT (MLX5_IB_MTT_READ | MLX5_IB_MTT_WRITE)
-#define MLX5_IB_DM_ALLOWED_ACCESS (IB_ACCESS_LOCAL_WRITE |\
- IB_ACCESS_REMOTE_WRITE |\
- IB_ACCESS_REMOTE_READ |\
- IB_ACCESS_REMOTE_ATOMIC |\
- IB_ZERO_BASED)
+#define MLX5_IB_DM_MEMIC_ALLOWED_ACCESS (IB_ACCESS_LOCAL_WRITE |\
+ IB_ACCESS_REMOTE_WRITE |\
+ IB_ACCESS_REMOTE_READ |\
+ IB_ACCESS_REMOTE_ATOMIC |\
+ IB_ZERO_BASED)
+
+#define MLX5_IB_DM_SW_ICM_ALLOWED_ACCESS (IB_ACCESS_LOCAL_WRITE |\
+ IB_ACCESS_REMOTE_WRITE |\
+ IB_ACCESS_REMOTE_READ |\
+ IB_ZERO_BASED)
struct mlx5_ib_mr {
struct ib_mr ibmr;
spinlock_t mpi_lock;
};
-struct mlx5_ib_port {
- struct mlx5_ib_counters cnts;
- struct mlx5_ib_multiport mp;
- struct mlx5_ib_dbg_cc_params *dbg_cc_params;
-};
-
struct mlx5_roce {
/* Protect mlx5_ib_get_netdev from invoking dev_hold() with a NULL
* netdev pointer
u8 native_port_num;
};
+struct mlx5_ib_port {
+ struct mlx5_ib_counters cnts;
+ struct mlx5_ib_multiport mp;
+ struct mlx5_ib_dbg_cc_params *dbg_cc_params;
+ struct mlx5_roce roce;
+ struct mlx5_eswitch_rep *rep;
+};
+
struct mlx5_ib_dbg_param {
int offset;
struct mlx5_ib_dev *dev;
};
};
-struct mlx5_memic {
+struct mlx5_dm {
struct mlx5_core_dev *dev;
- spinlock_t memic_lock;
+ /* This lock is used to protect the access to the shared
+ * allocation map when concurrent requests by different
+ * processes are handled.
+ */
+ spinlock_t lock;
DECLARE_BITMAP(memic_alloc_pages, MLX5_MAX_MEMIC_PAGES);
+ unsigned long *steering_sw_icm_alloc_blocks;
+ unsigned long *header_modify_sw_icm_alloc_blocks;
};
struct mlx5_read_counters_attr {
struct ib_device ib_dev;
struct mlx5_core_dev *mdev;
struct notifier_block mdev_events;
- struct mlx5_roce roce[MLX5_MAX_PORTS];
int num_ports;
/* serialize update of capability mask
*/
struct mlx5_sq_bfreg fp_bfreg;
struct mlx5_ib_delay_drop delay_drop;
const struct mlx5_ib_profile *profile;
- struct mlx5_eswitch_rep *rep;
+ bool is_rep;
int lag_active;
struct mlx5_ib_lb_state lb;
u8 umr_fence;
struct list_head ib_dev_list;
u64 sys_image_guid;
- struct mlx5_memic memic;
+ struct mlx5_dm dm;
u16 devx_whitelist_uid;
struct mlx5_srq_table srq_table;
struct mlx5_async_ctx async_ctx;
+ int free_port;
};
static inline struct mlx5_ib_cq *to_mibcq(struct mlx5_core_cq *mcq)
return container_of(ibdev, struct mlx5_ib_dev, ib_dev);
}
+static inline struct mlx5_ib_dev *mlx5_udata_to_mdev(struct ib_udata *udata)
+{
+ struct mlx5_ib_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct mlx5_ib_ucontext, ibucontext);
+
+ return to_mdev(context->ibucontext.device);
+}
+
static inline struct mlx5_ib_cq *to_mcq(struct ib_cq *ibcq)
{
return container_of(ibcq, struct mlx5_ib_cq, ibcq);
void __mlx5_ib_cq_clean(struct mlx5_ib_cq *cq, u32 qpn, struct mlx5_ib_srq *srq);
void mlx5_ib_cq_clean(struct mlx5_ib_cq *cq, u32 qpn, struct mlx5_ib_srq *srq);
void mlx5_ib_free_srq_wqe(struct mlx5_ib_srq *srq, int wqe_index);
-struct ib_ah *mlx5_ib_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
- u32 flags, struct ib_udata *udata);
+int mlx5_ib_create_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr, u32 flags,
+ struct ib_udata *udata);
int mlx5_ib_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr);
-int mlx5_ib_destroy_ah(struct ib_ah *ah, u32 flags);
-struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *init_attr,
- struct ib_udata *udata);
+void mlx5_ib_destroy_ah(struct ib_ah *ah, u32 flags);
+int mlx5_ib_create_srq(struct ib_srq *srq, struct ib_srq_init_attr *init_attr,
+ struct ib_udata *udata);
int mlx5_ib_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask attr_mask, struct ib_udata *udata);
int mlx5_ib_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *srq_attr);
-int mlx5_ib_destroy_srq(struct ib_srq *srq);
+void mlx5_ib_destroy_srq(struct ib_srq *srq, struct ib_udata *udata);
int mlx5_ib_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr,
const struct ib_recv_wr **bad_wr);
int mlx5_ib_enable_lb(struct mlx5_ib_dev *dev, bool td, bool qp);
int attr_mask, struct ib_udata *udata);
int mlx5_ib_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr, int qp_attr_mask,
struct ib_qp_init_attr *qp_init_attr);
-int mlx5_ib_destroy_qp(struct ib_qp *qp);
+int mlx5_ib_destroy_qp(struct ib_qp *qp, struct ib_udata *udata);
void mlx5_ib_drain_sq(struct ib_qp *qp);
void mlx5_ib_drain_rq(struct ib_qp *qp);
int mlx5_ib_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
void *buffer, int buflen, size_t *bc);
struct ib_cq *mlx5_ib_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata);
-int mlx5_ib_destroy_cq(struct ib_cq *cq);
+int mlx5_ib_destroy_cq(struct ib_cq *cq, struct ib_udata *udata);
int mlx5_ib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc);
int mlx5_ib_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags);
int mlx5_ib_modify_cq(struct ib_cq *cq, u16 cq_count, u16 cq_period);
int mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start,
u64 length, u64 virt_addr, int access_flags,
struct ib_pd *pd, struct ib_udata *udata);
-int mlx5_ib_dereg_mr(struct ib_mr *ibmr);
-struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg);
+int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata);
+struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata);
int mlx5_ib_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
unsigned int *sg_offset);
int mlx5_ib_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num,
struct ib_mad_hdr *out, size_t *out_mad_size,
u16 *out_mad_pkey_index);
struct ib_xrcd *mlx5_ib_alloc_xrcd(struct ib_device *ibdev,
- struct ib_ucontext *context,
- struct ib_udata *udata);
-int mlx5_ib_dealloc_xrcd(struct ib_xrcd *xrcd);
+ struct ib_udata *udata);
+int mlx5_ib_dealloc_xrcd(struct ib_xrcd *xrcd, struct ib_udata *udata);
int mlx5_ib_get_buf_offset(u64 addr, int page_shift, u32 *offset);
int mlx5_query_ext_port_caps(struct mlx5_ib_dev *dev, u8 port);
int mlx5_query_mad_ifc_smp_attr_node_info(struct ib_device *ibdev,
struct ib_wq *mlx5_ib_create_wq(struct ib_pd *pd,
struct ib_wq_init_attr *init_attr,
struct ib_udata *udata);
-int mlx5_ib_destroy_wq(struct ib_wq *wq);
+int mlx5_ib_destroy_wq(struct ib_wq *wq, struct ib_udata *udata);
int mlx5_ib_modify_wq(struct ib_wq *wq, struct ib_wq_attr *wq_attr,
u32 wq_attr_mask, struct ib_udata *udata);
struct ib_rwq_ind_table *mlx5_ib_create_rwq_ind_table(struct ib_device *device,
struct ib_ucontext *context,
struct ib_dm_alloc_attr *attr,
struct uverbs_attr_bundle *attrs);
-int mlx5_ib_dealloc_dm(struct ib_dm *ibdm);
+int mlx5_ib_dealloc_dm(struct ib_dm *ibdm, struct uverbs_attr_bundle *attrs);
struct ib_mr *mlx5_ib_reg_dm_mr(struct ib_pd *pd, struct ib_dm *dm,
struct ib_dm_mr_attr *attr,
struct uverbs_attr_bundle *attrs);
#endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */
/* Needed for rep profile */
-int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_rep_flow_db_init(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_rep_non_default_cb(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_rep_roce_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_rep_roce_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_dev_res_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_dev_res_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_counters_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_counters_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_bfrag_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_bfrag_cleanup(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_pre_ib_reg_umr_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_ib_reg_init(struct mlx5_ib_dev *dev);
-void mlx5_ib_stage_ib_reg_cleanup(struct mlx5_ib_dev *dev);
-int mlx5_ib_stage_post_ib_reg_umr_init(struct mlx5_ib_dev *dev);
void __mlx5_ib_remove(struct mlx5_ib_dev *dev,
const struct mlx5_ib_profile *profile,
int stage);
static void mlx5_mr_cache_debugfs_cleanup(struct mlx5_ib_dev *dev)
{
- if (!mlx5_debugfs_root || dev->rep)
+ if (!mlx5_debugfs_root || dev->is_rep)
return;
debugfs_remove_recursive(dev->cache.root);
struct dentry *dir;
int i;
- if (!mlx5_debugfs_root || dev->rep)
+ if (!mlx5_debugfs_root || dev->is_rep)
return;
cache->root = debugfs_create_dir("mr_cache", dev->mdev->priv.dbg_root);
MLX5_IB_UMR_OCTOWORD;
ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT;
if ((dev->mdev->profile->mask & MLX5_PROF_MASK_MR_CACHE) &&
- !dev->rep &&
+ !dev->is_rep &&
mlx5_core_is_pf(dev->mdev))
ent->limit = dev->mdev->profile->mr_cache[i].limit;
else
mr->access_flags = access_flags;
}
-static struct ib_mr *mlx5_ib_get_memic_mr(struct ib_pd *pd, u64 memic_addr,
- u64 length, int acc)
+static struct ib_mr *mlx5_ib_get_dm_mr(struct ib_pd *pd, u64 start_addr,
+ u64 length, int acc, int mode)
{
struct mlx5_ib_dev *dev = to_mdev(pd->device);
int inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
- MLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_MEMIC & 0x3);
- MLX5_SET(mkc, mkc, access_mode_4_2,
- (MLX5_MKC_ACCESS_MODE_MEMIC >> 2) & 0x7);
+ MLX5_SET(mkc, mkc, access_mode_1_0, mode & 0x3);
+ MLX5_SET(mkc, mkc, access_mode_4_2, (mode >> 2) & 0x7);
MLX5_SET(mkc, mkc, a, !!(acc & IB_ACCESS_REMOTE_ATOMIC));
MLX5_SET(mkc, mkc, rw, !!(acc & IB_ACCESS_REMOTE_WRITE));
MLX5_SET(mkc, mkc, rr, !!(acc & IB_ACCESS_REMOTE_READ));
MLX5_SET64(mkc, mkc, len, length);
MLX5_SET(mkc, mkc, pd, to_mpd(pd)->pdn);
MLX5_SET(mkc, mkc, qpn, 0xffffff);
- MLX5_SET64(mkc, mkc, start_addr, memic_addr - dev->mdev->bar_addr);
+ MLX5_SET64(mkc, mkc, start_addr, start_addr);
err = mlx5_core_create_mkey(mdev, &mr->mmkey, in, inlen);
if (err)
struct uverbs_attr_bundle *attrs)
{
struct mlx5_ib_dm *mdm = to_mdm(dm);
- u64 memic_addr;
+ struct mlx5_core_dev *dev = to_mdev(dm->device)->mdev;
+ u64 start_addr = mdm->dev_addr + attr->offset;
+ int mode;
- if (attr->access_flags & ~MLX5_IB_DM_ALLOWED_ACCESS)
- return ERR_PTR(-EINVAL);
+ switch (mdm->type) {
+ case MLX5_IB_UAPI_DM_TYPE_MEMIC:
+ if (attr->access_flags & ~MLX5_IB_DM_MEMIC_ALLOWED_ACCESS)
+ return ERR_PTR(-EINVAL);
+
+ mode = MLX5_MKC_ACCESS_MODE_MEMIC;
+ start_addr -= pci_resource_start(dev->pdev, 0);
+ break;
+ case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM:
+ case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM:
+ if (attr->access_flags & ~MLX5_IB_DM_SW_ICM_ALLOWED_ACCESS)
+ return ERR_PTR(-EINVAL);
- memic_addr = mdm->dev_addr + attr->offset;
+ mode = MLX5_MKC_ACCESS_MODE_SW_ICM;
+ break;
+ default:
+ return ERR_PTR(-EINVAL);
+ }
- return mlx5_ib_get_memic_mr(pd, memic_addr, attr->length,
- attr->access_flags);
+ return mlx5_ib_get_dm_mr(pd, start_addr, attr->length,
+ attr->access_flags, mode);
}
struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
kfree(mr);
}
-int mlx5_ib_dereg_mr(struct ib_mr *ibmr)
+int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
{
dereg_mr(to_mdev(ibmr->device), to_mmr(ibmr));
return 0;
}
-struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg)
+struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
struct mlx5_ib_dev *dev = to_mdev(pd->device);
int inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
ib_umem_odp_unmap_dma_pages(umem_odp, start, end);
- if (unlikely(!umem->npages && mr->parent &&
+ if (unlikely(!umem_odp->npages && mr->parent &&
!umem_odp->dying)) {
WRITE_ONCE(umem_odp->dying, 1);
atomic_inc(&mr->parent->num_leaf_free);
int depth;
};
+static bool mkey_is_eq(struct mlx5_core_mkey *mmkey, u32 key)
+{
+ if (!mmkey)
+ return false;
+ if (mmkey->type == MLX5_MKEY_MW)
+ return mlx5_base_mkey(mmkey->key) == mlx5_base_mkey(key);
+ return mmkey->key == key;
+}
+
static int get_indirect_num_descs(struct mlx5_core_mkey *mmkey)
{
struct mlx5_ib_mw *mw;
next_mr:
mmkey = __mlx5_mr_lookup(dev->mdev, mlx5_base_mkey(key));
- if (!mmkey || mmkey->key != key) {
+ if (!mkey_is_eq(mmkey, key)) {
mlx5_ib_dbg(dev, "failed to find mkey %x\n", key);
ret = -EFAULT;
goto srcu_unlock;
struct mlx5_pagefault *pfault,
void *wqe,
void *wqe_end, u32 *bytes_mapped,
- u32 *total_wqe_bytes, int receive_queue)
+ u32 *total_wqe_bytes, bool receive_queue)
{
int ret = 0, npages = 0;
u64 io_virt;
static void mlx5_ib_mr_wqe_pfault_handler(struct mlx5_ib_dev *dev,
struct mlx5_pagefault *pfault)
{
- int ret;
- void *wqe, *wqe_end;
+ bool sq = pfault->type & MLX5_PFAULT_REQUESTOR;
+ u16 wqe_index = pfault->wqe.wqe_index;
+ void *wqe = NULL, *wqe_end = NULL;
u32 bytes_mapped, total_wqe_bytes;
- char *buffer = NULL;
+ struct mlx5_core_rsc_common *res;
int resume_with_error = 1;
- u16 wqe_index = pfault->wqe.wqe_index;
- int requestor = pfault->type & MLX5_PFAULT_REQUESTOR;
- struct mlx5_core_rsc_common *res = NULL;
- struct mlx5_ib_qp *qp = NULL;
- struct mlx5_ib_srq *srq = NULL;
+ struct mlx5_ib_qp *qp;
size_t bytes_copied;
+ int ret = 0;
res = odp_get_rsc(dev, pfault->wqe.wq_num, pfault->type);
if (!res) {
return;
}
- switch (res->res) {
- case MLX5_RES_QP:
- qp = res_to_qp(res);
- break;
- case MLX5_RES_SRQ:
- case MLX5_RES_XSRQ:
- srq = res_to_srq(res);
- break;
- default:
- mlx5_ib_err(dev, "wqe page fault for unsupported type %d\n", pfault->type);
+ if (res->res != MLX5_RES_QP && res->res != MLX5_RES_SRQ &&
+ res->res != MLX5_RES_XSRQ) {
+ mlx5_ib_err(dev, "wqe page fault for unsupported type %d\n",
+ pfault->type);
goto resolve_page_fault;
}
- buffer = (char *)__get_free_page(GFP_KERNEL);
- if (!buffer) {
+ wqe = (void *)__get_free_page(GFP_KERNEL);
+ if (!wqe) {
mlx5_ib_err(dev, "Error allocating memory for IO page fault handling.\n");
goto resolve_page_fault;
}
- if (qp) {
- if (requestor) {
- ret = mlx5_ib_read_user_wqe_sq(qp, wqe_index,
- buffer, PAGE_SIZE,
- &bytes_copied);
- } else {
- ret = mlx5_ib_read_user_wqe_rq(qp, wqe_index,
- buffer, PAGE_SIZE,
- &bytes_copied);
- }
- } else {
- ret = mlx5_ib_read_user_wqe_srq(srq, wqe_index,
- buffer, PAGE_SIZE,
+ qp = (res->res == MLX5_RES_QP) ? res_to_qp(res) : NULL;
+ if (qp && sq) {
+ ret = mlx5_ib_read_user_wqe_sq(qp, wqe_index, wqe, PAGE_SIZE,
+ &bytes_copied);
+ if (ret)
+ goto read_user;
+ ret = mlx5_ib_mr_initiator_pfault_handler(
+ dev, pfault, qp, &wqe, &wqe_end, bytes_copied);
+ } else if (qp && !sq) {
+ ret = mlx5_ib_read_user_wqe_rq(qp, wqe_index, wqe, PAGE_SIZE,
+ &bytes_copied);
+ if (ret)
+ goto read_user;
+ ret = mlx5_ib_mr_responder_pfault_handler_rq(
+ dev, qp, wqe, &wqe_end, bytes_copied);
+ } else if (!qp) {
+ struct mlx5_ib_srq *srq = res_to_srq(res);
+
+ ret = mlx5_ib_read_user_wqe_srq(srq, wqe_index, wqe, PAGE_SIZE,
&bytes_copied);
+ if (ret)
+ goto read_user;
+ ret = mlx5_ib_mr_responder_pfault_handler_srq(
+ dev, srq, &wqe, &wqe_end, bytes_copied);
}
- if (ret) {
- mlx5_ib_err(dev, "Failed reading a WQE following page fault, error=%d, wqe_index=%x, qpn=%x\n",
- ret, wqe_index, pfault->token);
+ if (ret < 0 || wqe >= wqe_end)
goto resolve_page_fault;
- }
- wqe = buffer;
- if (requestor)
- ret = mlx5_ib_mr_initiator_pfault_handler(dev, pfault, qp,
- &wqe, &wqe_end,
- bytes_copied);
- else if (qp)
- ret = mlx5_ib_mr_responder_pfault_handler_rq(dev, qp,
- wqe, &wqe_end,
- bytes_copied);
- else
- ret = mlx5_ib_mr_responder_pfault_handler_srq(dev, srq,
- &wqe, &wqe_end,
- bytes_copied);
+ ret = pagefault_data_segments(dev, pfault, wqe, wqe_end, &bytes_mapped,
+ &total_wqe_bytes, !sq);
+ if (ret == -EAGAIN)
+ goto out;
- if (ret < 0)
+ if (ret < 0 || total_wqe_bytes > bytes_mapped)
goto resolve_page_fault;
- if (wqe >= wqe_end) {
- mlx5_ib_err(dev, "ODP fault on invalid WQE.\n");
- goto resolve_page_fault;
- }
+out:
+ ret = 0;
+ resume_with_error = 0;
- ret = pagefault_data_segments(dev, pfault, wqe, wqe_end,
- &bytes_mapped, &total_wqe_bytes,
- !requestor);
- if (ret == -EAGAIN) {
- resume_with_error = 0;
- goto resolve_page_fault;
- } else if (ret < 0 || total_wqe_bytes > bytes_mapped) {
- goto resolve_page_fault;
- }
+read_user:
+ if (ret)
+ mlx5_ib_err(
+ dev,
+ "Failed reading a WQE following page fault, error %d, wqe_index %x, qpn %x\n",
+ ret, wqe_index, pfault->token);
- resume_with_error = 0;
resolve_page_fault:
mlx5_ib_page_fault_resume(dev, pfault, resume_with_error);
mlx5_ib_dbg(dev, "PAGE FAULT completed. QP 0x%x resume_with_error=%d, type: 0x%x\n",
pfault->wqe.wq_num, resume_with_error,
pfault->type);
mlx5_core_res_put(res);
- free_page((unsigned long)buffer);
+ free_page((unsigned long)wqe);
}
static int pages_in_range(u64 address, u32 length)
struct mlx5_rate_limit rl;
u8 rq_q_ctr_id;
+ u16 port;
};
static void get_cqs(enum ib_qp_type qp_type,
}
static void destroy_user_rq(struct mlx5_ib_dev *dev, struct ib_pd *pd,
- struct mlx5_ib_rwq *rwq)
+ struct mlx5_ib_rwq *rwq, struct ib_udata *udata)
{
- struct mlx5_ib_ucontext *context;
+ struct mlx5_ib_ucontext *context =
+ rdma_udata_to_drv_context(
+ udata,
+ struct mlx5_ib_ucontext,
+ ibucontext);
if (rwq->create_flags & MLX5_IB_WQ_FLAGS_DELAY_DROP)
atomic_dec(&dev->delay_drop.rqs_cnt);
- context = to_mucontext(pd->uobject->context);
mlx5_ib_db_unmap_user(context, &rwq->db);
if (rwq->umem)
ib_umem_release(rwq->umem);
}
static void destroy_qp_user(struct mlx5_ib_dev *dev, struct ib_pd *pd,
- struct mlx5_ib_qp *qp, struct mlx5_ib_qp_base *base)
+ struct mlx5_ib_qp *qp, struct mlx5_ib_qp_base *base,
+ struct ib_udata *udata)
{
- struct mlx5_ib_ucontext *context;
+ struct mlx5_ib_ucontext *context =
+ rdma_udata_to_drv_context(
+ udata,
+ struct mlx5_ib_ucontext,
+ ibucontext);
- context = to_mucontext(pd->uobject->context);
mlx5_ib_db_unmap_user(context, &qp->db);
if (base->ubuffer.umem)
ib_umem_release(base->ubuffer.umem);
mlx5_cmd_destroy_tis(dev->mdev, sq->tisn, to_mpd(pd)->uid);
}
-static void destroy_flow_rule_vport_sq(struct mlx5_ib_dev *dev,
- struct mlx5_ib_sq *sq)
+static void destroy_flow_rule_vport_sq(struct mlx5_ib_sq *sq)
{
if (sq->flow_rule)
mlx5_del_flow_rules(sq->flow_rule);
+ sq->flow_rule = NULL;
}
static int create_raw_packet_qp_sq(struct mlx5_ib_dev *dev,
if (err)
goto err_umem;
- err = create_flow_rule_vport_sq(dev, sq);
- if (err)
- goto err_flow;
-
return 0;
-err_flow:
- mlx5_core_destroy_sq_tracked(dev->mdev, &sq->base.mqp);
-
err_umem:
ib_umem_release(sq->ubuffer.umem);
sq->ubuffer.umem = NULL;
static void destroy_raw_packet_qp_sq(struct mlx5_ib_dev *dev,
struct mlx5_ib_sq *sq)
{
- destroy_flow_rule_vport_sq(dev, sq);
+ destroy_flow_rule_vport_sq(sq);
mlx5_core_destroy_sq_tracked(dev->mdev, &sq->base.mqp);
ib_umem_release(sq->ubuffer.umem);
}
static int create_raw_packet_qp_tir(struct mlx5_ib_dev *dev,
struct mlx5_ib_rq *rq, u32 tdn,
u32 *qp_flags_en,
- struct ib_pd *pd)
+ struct ib_pd *pd,
+ u32 *out, int outlen)
{
u8 lb_flag = 0;
u32 *in;
if (*qp_flags_en & MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_MC)
lb_flag |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_MULTICAST;
- if (dev->rep) {
+ if (dev->is_rep) {
lb_flag |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST;
*qp_flags_en |= MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC;
}
MLX5_SET(tirc, tirc, self_lb_block, lb_flag);
- err = mlx5_core_create_tir(dev->mdev, in, inlen, &rq->tirn);
+ err = mlx5_core_create_tir_out(dev->mdev, in, inlen, out, outlen);
+ rq->tirn = MLX5_GET(create_tir_out, out, tirn);
if (!err && MLX5_GET(tirc, tirc, self_lb_block)) {
err = mlx5_ib_enable_lb(dev, false, true);
int err;
u32 tdn = mucontext->tdn;
u16 uid = to_mpd(pd)->uid;
+ u32 out[MLX5_ST_SZ_DW(create_tir_out)] = {};
if (qp->sq.wqe_cnt) {
err = create_raw_packet_qp_tis(dev, qp, sq, tdn, pd);
if (err)
goto err_destroy_sq;
- err = create_raw_packet_qp_tir(dev, rq, tdn, &qp->flags_en, pd);
+ err = create_raw_packet_qp_tir(
+ dev, rq, tdn, &qp->flags_en, pd, out,
+ MLX5_ST_SZ_BYTES(create_tir_out));
if (err)
goto err_destroy_rq;
resp->comp_mask |= MLX5_IB_CREATE_QP_RESP_MASK_RQN;
resp->tirn = rq->tirn;
resp->comp_mask |= MLX5_IB_CREATE_QP_RESP_MASK_TIRN;
+ if (MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner)) {
+ resp->tir_icm_addr = MLX5_GET(
+ create_tir_out, out, icm_address_31_0);
+ resp->tir_icm_addr |=
+ (u64)MLX5_GET(create_tir_out, out,
+ icm_address_39_32)
+ << 32;
+ resp->tir_icm_addr |=
+ (u64)MLX5_GET(create_tir_out, out,
+ icm_address_63_40)
+ << 40;
+ resp->comp_mask |=
+ MLX5_IB_CREATE_QP_RESP_MASK_TIR_ICM_ADDR;
+ }
}
}
udata, struct mlx5_ib_ucontext, ibucontext);
struct mlx5_ib_create_qp_resp resp = {};
int inlen;
+ int outlen;
int err;
u32 *in;
+ u32 *out;
void *tirc;
void *hfso;
u32 selected_fields = 0;
return -EOPNOTSUPP;
}
- if (ucmd.flags & MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC || dev->rep) {
+ if (ucmd.flags & MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC || dev->is_rep) {
lb_flag |= MLX5_TIRC_SELF_LB_BLOCK_BLOCK_UNICAST;
qp->flags_en |= MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC;
}
}
inlen = MLX5_ST_SZ_BYTES(create_tir_in);
- in = kvzalloc(inlen, GFP_KERNEL);
+ outlen = MLX5_ST_SZ_BYTES(create_tir_out);
+ in = kvzalloc(inlen + outlen, GFP_KERNEL);
if (!in)
return -ENOMEM;
+ out = in + MLX5_ST_SZ_DW(create_tir_in);
MLX5_SET(create_tir_in, in, uid, to_mpd(pd)->uid);
tirc = MLX5_ADDR_OF(create_tir_in, in, ctx);
MLX5_SET(tirc, tirc, disp_type,
MLX5_SET(rx_hash_field_select, hfso, selected_fields, selected_fields);
create_tir:
- err = mlx5_core_create_tir(dev->mdev, in, inlen, &qp->rss_qp.tirn);
+ err = mlx5_core_create_tir_out(dev->mdev, in, inlen, out, outlen);
+ qp->rss_qp.tirn = MLX5_GET(create_tir_out, out, tirn);
if (!err && MLX5_GET(tirc, tirc, self_lb_block)) {
err = mlx5_ib_enable_lb(dev, false, true);
if (mucontext->devx_uid) {
resp.comp_mask |= MLX5_IB_CREATE_QP_RESP_MASK_TIRN;
resp.tirn = qp->rss_qp.tirn;
+ if (MLX5_CAP_FLOWTABLE_NIC_RX(dev->mdev, sw_owner)) {
+ resp.tir_icm_addr =
+ MLX5_GET(create_tir_out, out, icm_address_31_0);
+ resp.tir_icm_addr |= (u64)MLX5_GET(create_tir_out, out,
+ icm_address_39_32)
+ << 32;
+ resp.tir_icm_addr |= (u64)MLX5_GET(create_tir_out, out,
+ icm_address_63_40)
+ << 40;
+ resp.comp_mask |=
+ MLX5_IB_CREATE_QP_RESP_MASK_TIR_ICM_ADDR;
+ }
}
err = ib_copy_to_udata(udata, &resp, min(udata->outlen, sizeof(resp)));
err_create:
if (qp->create_type == MLX5_QP_USER)
- destroy_qp_user(dev, pd, qp, base);
+ destroy_qp_user(dev, pd, qp, base, udata);
else if (qp->create_type == MLX5_QP_KERNEL)
destroy_qp_kernel(dev, qp);
const struct mlx5_modify_raw_qp_param *raw_qp_param,
u8 lag_tx_affinity);
-static void destroy_qp_common(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp)
+static void destroy_qp_common(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,
+ struct ib_udata *udata)
{
struct mlx5_ib_cq *send_cq, *recv_cq;
struct mlx5_ib_qp_base *base;
if (qp->create_type == MLX5_QP_KERNEL)
destroy_qp_kernel(dev, qp);
else if (qp->create_type == MLX5_QP_USER)
- destroy_qp_user(dev, &get_pd(qp)->ibpd, qp, base);
+ destroy_qp_user(dev, &get_pd(qp)->ibpd, qp, base, udata);
}
static const char *ib_qp_type_str(enum ib_qp_type type)
return 0;
}
-int mlx5_ib_destroy_qp(struct ib_qp *qp)
+int mlx5_ib_destroy_qp(struct ib_qp *qp, struct ib_udata *udata)
{
struct mlx5_ib_dev *dev = to_mdev(qp->device);
struct mlx5_ib_qp *mqp = to_mqp(qp);
if (mqp->qp_sub_type == MLX5_IB_QPT_DCT)
return mlx5_ib_destroy_dct(mqp);
- destroy_qp_common(dev, mqp);
+ destroy_qp_common(dev, mqp, udata);
kfree(mqp);
[MLX5_QP_ST_UD] = MLX5_QP_OPTPAR_PKEY_INDEX |
MLX5_QP_OPTPAR_Q_KEY |
MLX5_QP_OPTPAR_PRI_PORT,
+ [MLX5_QP_ST_XRC] = MLX5_QP_OPTPAR_RRE |
+ MLX5_QP_OPTPAR_RAE |
+ MLX5_QP_OPTPAR_RWE |
+ MLX5_QP_OPTPAR_PKEY_INDEX |
+ MLX5_QP_OPTPAR_PRI_PORT,
},
[MLX5_QP_STATE_RTR] = {
[MLX5_QP_ST_RC] = MLX5_QP_OPTPAR_ALT_ADDR_PATH |
MLX5_QP_OPTPAR_RWE |
MLX5_QP_OPTPAR_PM_STATE,
[MLX5_QP_ST_UD] = MLX5_QP_OPTPAR_Q_KEY,
+ [MLX5_QP_ST_XRC] = MLX5_QP_OPTPAR_ALT_ADDR_PATH |
+ MLX5_QP_OPTPAR_RRE |
+ MLX5_QP_OPTPAR_RAE |
+ MLX5_QP_OPTPAR_RWE |
+ MLX5_QP_OPTPAR_PM_STATE |
+ MLX5_QP_OPTPAR_RNR_TIMEOUT,
},
},
[MLX5_QP_STATE_RTS] = {
[MLX5_QP_ST_UD] = MLX5_QP_OPTPAR_Q_KEY |
MLX5_QP_OPTPAR_SRQN |
MLX5_QP_OPTPAR_CQN_RCV,
+ [MLX5_QP_ST_XRC] = MLX5_QP_OPTPAR_RRE |
+ MLX5_QP_OPTPAR_RAE |
+ MLX5_QP_OPTPAR_RWE |
+ MLX5_QP_OPTPAR_RNR_TIMEOUT |
+ MLX5_QP_OPTPAR_PM_STATE |
+ MLX5_QP_OPTPAR_ALT_ADDR_PATH,
},
},
[MLX5_QP_STATE_SQER] = {
MLX5_QP_OPTPAR_RWE |
MLX5_QP_OPTPAR_RAE |
MLX5_QP_OPTPAR_RRE,
+ [MLX5_QP_ST_XRC] = MLX5_QP_OPTPAR_RNR_TIMEOUT |
+ MLX5_QP_OPTPAR_RWE |
+ MLX5_QP_OPTPAR_RAE |
+ MLX5_QP_OPTPAR_RRE,
},
},
};
}
if (modify_sq) {
+ struct mlx5_flow_handle *flow_rule;
+
if (tx_affinity) {
err = modify_raw_packet_tx_affinity(dev->mdev, sq,
tx_affinity,
return err;
}
- return modify_raw_packet_qp_sq(dev->mdev, sq, sq_state,
- raw_qp_param, qp->ibqp.pd);
+ flow_rule = create_flow_rule_vport_sq(dev, sq,
+ raw_qp_param->port);
+ if (IS_ERR(flow_rule))
+ return PTR_ERR(flow_rule);
+
+ err = modify_raw_packet_qp_sq(dev->mdev, sq, sq_state,
+ raw_qp_param, qp->ibqp.pd);
+ if (err) {
+ if (flow_rule)
+ mlx5_del_flow_rules(flow_rule);
+ return err;
+ }
+
+ if (flow_rule) {
+ destroy_flow_rule_vport_sq(sq);
+ sq->flow_rule = flow_rule;
+ }
+
+ return err;
}
return 0;
} else {
tx_port_affinity =
(unsigned int)atomic_add_return(
- 1, &dev->roce[port_num].tx_port_affinity) %
+ 1, &dev->port[port_num].roce.tx_port_affinity) %
MLX5_MAX_PORTS +
1;
mlx5_ib_dbg(dev, "Set tx affinity 0x%x to qpn 0x%x\n",
(ibqp->qp_type == IB_QPT_XRC_INI) ||
(ibqp->qp_type == IB_QPT_XRC_TGT)) {
if (dev->lag_active) {
- u8 p = mlx5_core_native_port_num(dev->mdev);
+ u8 p = mlx5_core_native_port_num(dev->mdev) - 1;
tx_affinity = get_tx_affinity(dev, pd, base, p,
udata);
context->flags |= cpu_to_be32(tx_affinity << 24);
raw_qp_param.set_mask |= MLX5_RAW_QP_MOD_SET_RQ_Q_CTR_ID;
}
+ if (attr_mask & IB_QP_PORT)
+ raw_qp_param.port = attr->port_num;
+
if (attr_mask & IB_QP_RATE_LIMIT) {
raw_qp_param.rl.rate = attr->rate_limit;
static void dump_wqe(struct mlx5_ib_qp *qp, u32 idx, int size_16)
{
__be32 *p = NULL;
- u32 tidx = idx;
int i, j;
pr_debug("dump WQE index %u:\n", idx);
for (i = 0, j = 0; i < size_16 * 4; i += 4, j += 4) {
if ((i & 0xf) == 0) {
- tidx = (tidx + 1) & (qp->sq.wqe_cnt - 1);
- p = mlx5_frag_buf_get_wqe(&qp->sq.fbc, tidx);
+ p = mlx5_frag_buf_get_wqe(&qp->sq.fbc, idx);
pr_debug("WQBB at %p:\n", (void *)p);
j = 0;
+ idx = (idx + 1) & (qp->sq.wqe_cnt - 1);
}
pr_debug("%08x %08x %08x %08x\n", be32_to_cpu(p[j]),
be32_to_cpu(p[j + 1]), be32_to_cpu(p[j + 2]),
}
struct ib_xrcd *mlx5_ib_alloc_xrcd(struct ib_device *ibdev,
- struct ib_ucontext *context,
- struct ib_udata *udata)
+ struct ib_udata *udata)
{
struct mlx5_ib_dev *dev = to_mdev(ibdev);
struct mlx5_ib_xrcd *xrcd;
return &xrcd->ibxrcd;
}
-int mlx5_ib_dealloc_xrcd(struct ib_xrcd *xrcd)
+int mlx5_ib_dealloc_xrcd(struct ib_xrcd *xrcd, struct ib_udata *udata)
{
struct mlx5_ib_dev *dev = to_mdev(xrcd->device);
u32 xrcdn = to_mxrcd(xrcd)->xrcdn;
err_copy:
mlx5_core_destroy_rq_tracked(dev->mdev, &rwq->core_qp);
err_user_rq:
- destroy_user_rq(dev, pd, rwq);
+ destroy_user_rq(dev, pd, rwq, udata);
err:
kfree(rwq);
return ERR_PTR(err);
}
-int mlx5_ib_destroy_wq(struct ib_wq *wq)
+int mlx5_ib_destroy_wq(struct ib_wq *wq, struct ib_udata *udata)
{
struct mlx5_ib_dev *dev = to_mdev(wq->device);
struct mlx5_ib_rwq *rwq = to_mrwq(wq);
mlx5_core_destroy_rq_tracked(dev->mdev, &rwq->core_qp);
- destroy_user_rq(dev, wq->pd, rwq);
+ destroy_user_rq(dev, wq->pd, rwq, udata);
kfree(rwq);
return 0;
return err;
}
-static void destroy_srq_user(struct ib_pd *pd, struct mlx5_ib_srq *srq)
+static void destroy_srq_user(struct ib_pd *pd, struct mlx5_ib_srq *srq,
+ struct ib_udata *udata)
{
- mlx5_ib_db_unmap_user(to_mucontext(pd->uobject->context), &srq->db);
+ mlx5_ib_db_unmap_user(
+ rdma_udata_to_drv_context(
+ udata,
+ struct mlx5_ib_ucontext,
+ ibucontext),
+ &srq->db);
ib_umem_release(srq->umem);
}
mlx5_db_free(dev->mdev, &srq->db);
}
-struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *init_attr,
- struct ib_udata *udata)
+int mlx5_ib_create_srq(struct ib_srq *ib_srq,
+ struct ib_srq_init_attr *init_attr,
+ struct ib_udata *udata)
{
- struct mlx5_ib_dev *dev = to_mdev(pd->device);
- struct mlx5_ib_srq *srq;
+ struct mlx5_ib_dev *dev = to_mdev(ib_srq->device);
+ struct mlx5_ib_srq *srq = to_msrq(ib_srq);
size_t desc_size;
size_t buf_size;
int err;
- struct mlx5_srq_attr in = {0};
+ struct mlx5_srq_attr in = {};
__u32 max_srq_wqes = 1 << MLX5_CAP_GEN(dev->mdev, log_max_srq_sz);
/* Sanity check SRQ size before proceeding */
mlx5_ib_dbg(dev, "max_wr %d, cap %d\n",
init_attr->attr.max_wr,
max_srq_wqes);
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
}
- srq = kmalloc(sizeof(*srq), GFP_KERNEL);
- if (!srq)
- return ERR_PTR(-ENOMEM);
-
mutex_init(&srq->mutex);
spin_lock_init(&srq->lock);
srq->msrq.max = roundup_pow_of_two(init_attr->attr.max_wr + 1);
desc_size = sizeof(struct mlx5_wqe_srq_next_seg) +
srq->msrq.max_gs * sizeof(struct mlx5_wqe_data_seg);
- if (desc_size == 0 || srq->msrq.max_gs > desc_size) {
- err = -EINVAL;
- goto err_srq;
- }
+ if (desc_size == 0 || srq->msrq.max_gs > desc_size)
+ return -EINVAL;
+
desc_size = roundup_pow_of_two(desc_size);
desc_size = max_t(size_t, 32, desc_size);
- if (desc_size < sizeof(struct mlx5_wqe_srq_next_seg)) {
- err = -EINVAL;
- goto err_srq;
- }
+ if (desc_size < sizeof(struct mlx5_wqe_srq_next_seg))
+ return -EINVAL;
+
srq->msrq.max_avail_gather = (desc_size - sizeof(struct mlx5_wqe_srq_next_seg)) /
sizeof(struct mlx5_wqe_data_seg);
srq->msrq.wqe_shift = ilog2(desc_size);
buf_size = srq->msrq.max * desc_size;
- if (buf_size < desc_size) {
- err = -EINVAL;
- goto err_srq;
- }
+ if (buf_size < desc_size)
+ return -EINVAL;
+
in.type = init_attr->srq_type;
if (udata)
- err = create_srq_user(pd, srq, &in, udata, buf_size);
+ err = create_srq_user(ib_srq->pd, srq, &in, udata, buf_size);
else
err = create_srq_kernel(dev, srq, &in, buf_size);
if (err) {
mlx5_ib_warn(dev, "create srq %s failed, err %d\n",
udata ? "user" : "kernel", err);
- goto err_srq;
+ return err;
}
in.log_size = ilog2(srq->msrq.max);
else
in.cqn = to_mcq(dev->devr.c0)->mcq.cqn;
- in.pd = to_mpd(pd)->pdn;
+ in.pd = to_mpd(ib_srq->pd)->pdn;
in.db_record = srq->db.dma;
err = mlx5_cmd_create_srq(dev, &srq->msrq, &in);
kvfree(in.pas);
init_attr->attr.max_wr = srq->msrq.max - 1;
- return &srq->ibsrq;
+ return 0;
err_core:
mlx5_cmd_destroy_srq(dev, &srq->msrq);
err_usr_kern_srq:
if (udata)
- destroy_srq_user(pd, srq);
+ destroy_srq_user(ib_srq->pd, srq, udata);
else
destroy_srq_kernel(dev, srq);
-err_srq:
- kfree(srq);
-
- return ERR_PTR(err);
+ return err;
}
int mlx5_ib_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
return ret;
}
-int mlx5_ib_destroy_srq(struct ib_srq *srq)
+void mlx5_ib_destroy_srq(struct ib_srq *srq, struct ib_udata *udata)
{
struct mlx5_ib_dev *dev = to_mdev(srq->device);
struct mlx5_ib_srq *msrq = to_msrq(srq);
mlx5_cmd_destroy_srq(dev, &msrq->msrq);
if (srq->uobject) {
- mlx5_ib_db_unmap_user(to_mucontext(srq->uobject->context), &msrq->db);
+ mlx5_ib_db_unmap_user(
+ rdma_udata_to_drv_context(
+ udata,
+ struct mlx5_ib_ucontext,
+ ibucontext),
+ &msrq->db);
ib_umem_release(msrq->umem);
} else {
destroy_srq_kernel(dev, msrq);
}
-
- kfree(srq);
- return 0;
}
void mlx5_ib_free_srq_wqe(struct mlx5_ib_srq *srq, int wqe_index)
struct mlx5_srq_table {
struct notifier_block nb;
- /* protect radix tree
- */
- spinlock_t lock;
- struct radix_tree_root tree;
+ struct xarray array;
};
int mlx5_cmd_create_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
struct mlx5_srq_attr *in);
-int mlx5_cmd_destroy_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq);
+void mlx5_cmd_destroy_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq);
int mlx5_cmd_query_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
struct mlx5_srq_attr *out);
int mlx5_cmd_arm_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
struct mlx5_srq_table *table = &dev->srq_table;
struct mlx5_core_srq *srq;
- spin_lock(&table->lock);
-
- srq = radix_tree_lookup(&table->tree, srqn);
+ xa_lock(&table->array);
+ srq = xa_load(&table->array, srqn);
if (srq)
atomic_inc(&srq->common.refcount);
-
- spin_unlock(&table->lock);
+ xa_unlock(&table->array);
return srq;
}
atomic_set(&srq->common.refcount, 1);
init_completion(&srq->common.free);
- spin_lock_irq(&table->lock);
- err = radix_tree_insert(&table->tree, srq->srqn, srq);
- spin_unlock_irq(&table->lock);
+ err = xa_err(xa_store_irq(&table->array, srq->srqn, srq, GFP_KERNEL));
if (err)
goto err_destroy_srq_split;
return err;
}
-int mlx5_cmd_destroy_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq)
+void mlx5_cmd_destroy_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq)
{
struct mlx5_srq_table *table = &dev->srq_table;
struct mlx5_core_srq *tmp;
int err;
- spin_lock_irq(&table->lock);
- tmp = radix_tree_delete(&table->tree, srq->srqn);
- spin_unlock_irq(&table->lock);
+ tmp = xa_erase_irq(&table->array, srq->srqn);
if (!tmp || tmp != srq)
- return -EINVAL;
+ return;
err = destroy_srq_split(dev, srq);
if (err)
- return err;
+ return;
mlx5_core_res_put(&srq->common);
wait_for_completion(&srq->common.free);
-
- return 0;
}
int mlx5_cmd_query_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
eqe = data;
srqn = be32_to_cpu(eqe->data.qp_srq.qp_srq_n) & 0xffffff;
- spin_lock(&table->lock);
-
- srq = radix_tree_lookup(&table->tree, srqn);
+ xa_lock(&table->array);
+ srq = xa_load(&table->array, srqn);
if (srq)
atomic_inc(&srq->common.refcount);
-
- spin_unlock(&table->lock);
+ xa_unlock(&table->array);
if (!srq)
return NOTIFY_OK;
struct mlx5_srq_table *table = &dev->srq_table;
memset(table, 0, sizeof(*table));
- spin_lock_init(&table->lock);
- INIT_RADIX_TREE(&table->tree, GFP_ATOMIC);
+ xa_init_flags(&table->array, XA_FLAGS_LOCK_IRQ);
table->nb.notifier_call = srq_event_notifier;
mlx5_notifier_register(dev->mdev, &table->nb);
__be32 ci_db; /* Arbel only */
__be32 state_db; /* Arbel only */
u32 reserved;
-} __attribute__((packed));
+} __packed;
#define MTHCA_CQ_STATUS_OK ( 0 << 28)
#define MTHCA_CQ_STATUS_OVERFLOW ( 9 << 28)
__be32 consumer_index;
__be32 producer_index;
u32 reserved3[4];
-} __attribute__((packed));
+} __packed;
#define MTHCA_EQ_STATUS_OK ( 0 << 28)
#define MTHCA_EQ_STATUS_OVERFLOW ( 9 << 28)
u32 raw[6];
struct {
__be32 cqn;
- } __attribute__((packed)) comp;
+ } __packed comp;
struct {
u16 reserved1;
__be16 token;
u8 reserved3[3];
u8 status;
__be64 out_param;
- } __attribute__((packed)) cmd;
+ } __packed cmd;
struct {
__be32 qpn;
- } __attribute__((packed)) qp;
+ } __packed qp;
struct {
__be32 srqn;
- } __attribute__((packed)) srq;
+ } __packed srq;
struct {
__be32 cqn;
u32 reserved1;
u8 reserved2[3];
u8 syndrome;
- } __attribute__((packed)) cq_err;
+ } __packed cq_err;
struct {
u32 reserved1[2];
__be32 port;
- } __attribute__((packed)) port_change;
+ } __packed port_change;
} event;
u8 reserved3[3];
u8 owner;
-} __attribute__((packed));
+} __packed;
#define MTHCA_EQ_ENTRY_OWNER_SW (0 << 7)
#define MTHCA_EQ_ENTRY_OWNER_HW (1 << 7)
__be64 mtt_seg;
__be32 mtt_sz; /* Arbel only */
u32 reserved[2];
-} __attribute__((packed));
+} __packed;
#define MTHCA_MPT_FLAG_SW_OWNS (0xfUL << 28)
#define MTHCA_MPT_FLAG_MIO (1 << 17)
return 0;
}
-static int mthca_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata)
+static int mthca_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct ib_device *ibdev = ibpd->device;
struct mthca_pd *pd = to_mpd(ibpd);
int err;
- err = mthca_pd_alloc(to_mdev(ibdev), !context, pd);
+ err = mthca_pd_alloc(to_mdev(ibdev), !udata, pd);
if (err)
return err;
- if (context) {
+ if (udata) {
if (ib_copy_to_udata(udata, &pd->pd_num, sizeof (__u32))) {
mthca_pd_free(to_mdev(ibdev), pd);
return -EFAULT;
return 0;
}
-static void mthca_dealloc_pd(struct ib_pd *pd)
+static void mthca_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata)
{
mthca_pd_free(to_mdev(pd->device), to_mpd(pd));
}
-static struct ib_ah *mthca_ah_create(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
- u32 flags,
- struct ib_udata *udata)
+static int mthca_ah_create(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr,
+ u32 flags, struct ib_udata *udata)
{
- int err;
- struct mthca_ah *ah;
+ struct mthca_ah *ah = to_mah(ibah);
- ah = kmalloc(sizeof *ah, GFP_ATOMIC);
- if (!ah)
- return ERR_PTR(-ENOMEM);
-
- err = mthca_create_ah(to_mdev(pd->device), to_mpd(pd), ah_attr, ah);
- if (err) {
- kfree(ah);
- return ERR_PTR(err);
- }
-
- return &ah->ibah;
+ return mthca_create_ah(to_mdev(ibah->device), to_mpd(ibah->pd), ah_attr,
+ ah);
}
-static int mthca_ah_destroy(struct ib_ah *ah, u32 flags)
+static void mthca_ah_destroy(struct ib_ah *ah, u32 flags)
{
mthca_destroy_ah(to_mdev(ah->device), to_mah(ah));
- kfree(ah);
-
- return 0;
}
-static struct ib_srq *mthca_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *init_attr,
- struct ib_udata *udata)
+static int mthca_create_srq(struct ib_srq *ibsrq,
+ struct ib_srq_init_attr *init_attr,
+ struct ib_udata *udata)
{
struct mthca_create_srq ucmd;
struct mthca_ucontext *context = rdma_udata_to_drv_context(
udata, struct mthca_ucontext, ibucontext);
- struct mthca_srq *srq;
+ struct mthca_srq *srq = to_msrq(ibsrq);
int err;
if (init_attr->srq_type != IB_SRQT_BASIC)
- return ERR_PTR(-EOPNOTSUPP);
-
- srq = kmalloc(sizeof *srq, GFP_KERNEL);
- if (!srq)
- return ERR_PTR(-ENOMEM);
+ return -EOPNOTSUPP;
if (udata) {
- if (ib_copy_from_udata(&ucmd, udata, sizeof ucmd)) {
- err = -EFAULT;
- goto err_free;
- }
+ if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd)))
+ return -EFAULT;
- err = mthca_map_user_db(to_mdev(pd->device), &context->uar,
+ err = mthca_map_user_db(to_mdev(ibsrq->device), &context->uar,
context->db_tab, ucmd.db_index,
ucmd.db_page);
if (err)
- goto err_free;
+ return err;
srq->mr.ibmr.lkey = ucmd.lkey;
srq->db_index = ucmd.db_index;
}
- err = mthca_alloc_srq(to_mdev(pd->device), to_mpd(pd),
+ err = mthca_alloc_srq(to_mdev(ibsrq->device), to_mpd(ibsrq->pd),
&init_attr->attr, srq, udata);
if (err && udata)
- mthca_unmap_user_db(to_mdev(pd->device), &context->uar,
+ mthca_unmap_user_db(to_mdev(ibsrq->device), &context->uar,
context->db_tab, ucmd.db_index);
if (err)
- goto err_free;
+ return err;
- if (context && ib_copy_to_udata(udata, &srq->srqn, sizeof (__u32))) {
- mthca_free_srq(to_mdev(pd->device), srq);
- err = -EFAULT;
- goto err_free;
+ if (context && ib_copy_to_udata(udata, &srq->srqn, sizeof(__u32))) {
+ mthca_free_srq(to_mdev(ibsrq->device), srq);
+ return -EFAULT;
}
- return &srq->ibsrq;
-
-err_free:
- kfree(srq);
-
- return ERR_PTR(err);
+ return 0;
}
-static int mthca_destroy_srq(struct ib_srq *srq)
+static void mthca_destroy_srq(struct ib_srq *srq, struct ib_udata *udata)
{
- struct mthca_ucontext *context;
-
- if (srq->uobject) {
- context = to_mucontext(srq->uobject->context);
+ if (udata) {
+ struct mthca_ucontext *context =
+ rdma_udata_to_drv_context(
+ udata,
+ struct mthca_ucontext,
+ ibucontext);
mthca_unmap_user_db(to_mdev(srq->device), &context->uar,
context->db_tab, to_msrq(srq)->db_index);
}
mthca_free_srq(to_mdev(srq->device), to_msrq(srq));
- kfree(srq);
-
- return 0;
}
static struct ib_qp *mthca_create_qp(struct ib_pd *pd,
return &qp->ibqp;
}
-static int mthca_destroy_qp(struct ib_qp *qp)
+static int mthca_destroy_qp(struct ib_qp *qp, struct ib_udata *udata)
{
- if (qp->uobject) {
+ if (udata) {
+ struct mthca_ucontext *context =
+ rdma_udata_to_drv_context(
+ udata,
+ struct mthca_ucontext,
+ ibucontext);
+
mthca_unmap_user_db(to_mdev(qp->device),
- &to_mucontext(qp->uobject->context)->uar,
- to_mucontext(qp->uobject->context)->db_tab,
+ &context->uar,
+ context->db_tab,
to_mqp(qp)->sq.db_index);
mthca_unmap_user_db(to_mdev(qp->device),
- &to_mucontext(qp->uobject->context)->uar,
- to_mucontext(qp->uobject->context)->db_tab,
+ &context->uar,
+ context->db_tab,
to_mqp(qp)->rq.db_index);
}
mthca_free_qp(to_mdev(qp->device), to_mqp(qp));
static struct ib_cq *mthca_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
int entries = attr->cqe;
struct mthca_cq *cq;
int nent;
int err;
+ struct mthca_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct mthca_ucontext, ibucontext);
if (attr->flags)
return ERR_PTR(-EINVAL);
if (entries < 1 || entries > to_mdev(ibdev)->limits.max_cqes)
return ERR_PTR(-EINVAL);
- if (context) {
+ if (udata) {
if (ib_copy_from_udata(&ucmd, udata, sizeof ucmd))
return ERR_PTR(-EFAULT);
- err = mthca_map_user_db(to_mdev(ibdev), &to_mucontext(context)->uar,
- to_mucontext(context)->db_tab,
- ucmd.set_db_index, ucmd.set_db_page);
+ err = mthca_map_user_db(to_mdev(ibdev), &context->uar,
+ context->db_tab, ucmd.set_db_index,
+ ucmd.set_db_page);
if (err)
return ERR_PTR(err);
- err = mthca_map_user_db(to_mdev(ibdev), &to_mucontext(context)->uar,
- to_mucontext(context)->db_tab,
- ucmd.arm_db_index, ucmd.arm_db_page);
+ err = mthca_map_user_db(to_mdev(ibdev), &context->uar,
+ context->db_tab, ucmd.arm_db_index,
+ ucmd.arm_db_page);
if (err)
goto err_unmap_set;
}
goto err_unmap_arm;
}
- if (context) {
+ if (udata) {
cq->buf.mr.ibmr.lkey = ucmd.lkey;
cq->set_ci_db_index = ucmd.set_db_index;
cq->arm_db_index = ucmd.arm_db_index;
for (nent = 1; nent <= entries; nent <<= 1)
; /* nothing */
- err = mthca_init_cq(to_mdev(ibdev), nent,
- context ? to_mucontext(context) : NULL,
- context ? ucmd.pdn : to_mdev(ibdev)->driver_pd.pd_num,
+ err = mthca_init_cq(to_mdev(ibdev), nent, context,
+ udata ? ucmd.pdn : to_mdev(ibdev)->driver_pd.pd_num,
cq);
if (err)
goto err_free;
- if (context && ib_copy_to_udata(udata, &cq->cqn, sizeof (__u32))) {
+ if (udata && ib_copy_to_udata(udata, &cq->cqn, sizeof(__u32))) {
mthca_free_cq(to_mdev(ibdev), cq);
err = -EFAULT;
goto err_free;
kfree(cq);
err_unmap_arm:
- if (context)
- mthca_unmap_user_db(to_mdev(ibdev), &to_mucontext(context)->uar,
- to_mucontext(context)->db_tab, ucmd.arm_db_index);
+ if (udata)
+ mthca_unmap_user_db(to_mdev(ibdev), &context->uar,
+ context->db_tab, ucmd.arm_db_index);
err_unmap_set:
- if (context)
- mthca_unmap_user_db(to_mdev(ibdev), &to_mucontext(context)->uar,
- to_mucontext(context)->db_tab, ucmd.set_db_index);
+ if (udata)
+ mthca_unmap_user_db(to_mdev(ibdev), &context->uar,
+ context->db_tab, ucmd.set_db_index);
return ERR_PTR(err);
}
return ret;
}
-static int mthca_destroy_cq(struct ib_cq *cq)
+static int mthca_destroy_cq(struct ib_cq *cq, struct ib_udata *udata)
{
- if (cq->uobject) {
+ if (udata) {
+ struct mthca_ucontext *context =
+ rdma_udata_to_drv_context(
+ udata,
+ struct mthca_ucontext,
+ ibucontext);
+
mthca_unmap_user_db(to_mdev(cq->device),
- &to_mucontext(cq->uobject->context)->uar,
- to_mucontext(cq->uobject->context)->db_tab,
+ &context->uar,
+ context->db_tab,
to_mcq(cq)->arm_db_index);
mthca_unmap_user_db(to_mdev(cq->device),
- &to_mucontext(cq->uobject->context)->uar,
- to_mucontext(cq->uobject->context)->db_tab,
+ &context->uar,
+ context->db_tab,
to_mcq(cq)->set_ci_db_index);
}
mthca_free_cq(to_mdev(cq->device), to_mcq(cq));
goto err;
}
- n = mr->umem->nmap;
+ n = ib_umem_num_pages(mr->umem);
mr->mtt = mthca_alloc_mtt(dev, n);
if (IS_ERR(mr->mtt)) {
return ERR_PTR(err);
}
-static int mthca_dereg_mr(struct ib_mr *mr)
+static int mthca_dereg_mr(struct ib_mr *mr, struct ib_udata *udata)
{
struct mthca_mr *mmr = to_mmr(mr);
.query_qp = mthca_query_qp,
.reg_user_mr = mthca_reg_user_mr,
.resize_cq = mthca_resize_cq,
+
+ INIT_RDMA_OBJ_SIZE(ib_ah, mthca_ah, ibah),
INIT_RDMA_OBJ_SIZE(ib_pd, mthca_pd, ibpd),
INIT_RDMA_OBJ_SIZE(ib_ucontext, mthca_ucontext, ibucontext),
};
.modify_srq = mthca_modify_srq,
.post_srq_recv = mthca_arbel_post_srq_recv,
.query_srq = mthca_query_srq,
+
+ INIT_RDMA_OBJ_SIZE(ib_srq, mthca_srq, ibsrq),
};
static const struct ib_device_ops mthca_dev_tavor_srq_ops = {
.modify_srq = mthca_modify_srq,
.post_srq_recv = mthca_tavor_post_srq_recv,
.query_srq = mthca_query_srq,
+
+ INIT_RDMA_OBJ_SIZE(ib_srq, mthca_srq, ibsrq),
};
static const struct ib_device_ops mthca_dev_arbel_fmr_ops = {
u8 hop_limit;
__be32 sl_tclass_flowlabel;
u8 rgid[16];
-} __attribute__((packed));
+} __packed;
struct mthca_qp_context {
__be32 flags;
__be16 rq_wqe_counter; /* reserved on Tavor */
__be16 sq_wqe_counter; /* reserved on Tavor */
u32 reserved3[18];
-} __attribute__((packed));
+} __packed;
struct mthca_qp_param {
__be32 opt_param_mask;
u32 reserved1;
struct mthca_qp_context context;
u32 reserved2[62];
-} __attribute__((packed));
+} __packed;
enum {
MTHCA_QP_OPTPAR_ALT_ADDR_PATH = 1 << 0,
/* Need to free the Last Streaming Mode Message */
if (nesqp->ietf_frame) {
if (nesqp->lsmm_mr)
- nesibdev->ibdev.ops.dereg_mr(nesqp->lsmm_mr);
+ nesibdev->ibdev.ops.dereg_mr(nesqp->lsmm_mr,
+ NULL);
pci_free_consistent(nesdev->pcidev,
nesqp->private_data_len + nesqp->ietf_frame_size,
nesqp->ietf_frame, nesqp->ietf_frame_pbase);
atomic_t sw_qps_destroyed;
static void nes_unregister_ofa_device(struct nes_ib_device *nesibdev);
-static int nes_dereg_mr(struct ib_mr *ib_mr);
+static int nes_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata);
/**
* nes_alloc_mw
/*
* nes_alloc_mr
*/
-static struct ib_mr *nes_alloc_mr(struct ib_pd *ibpd,
- enum ib_mr_type mr_type,
- u32 max_num_sg)
+static struct ib_mr *nes_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
struct nes_pd *nespd = to_nespd(ibpd);
struct nes_vnic *nesvnic = to_nesvnic(ibpd->device);
return ibmr;
err:
- nes_dereg_mr(ibmr);
+ nes_dereg_mr(ibmr, udata);
return ERR_PTR(-ENOMEM);
}
/**
* nes_alloc_pd
*/
-static int nes_alloc_pd(struct ib_pd *pd, struct ib_ucontext *context,
- struct ib_udata *udata)
+static int nes_alloc_pd(struct ib_pd *pd, struct ib_udata *udata)
{
struct ib_device *ibdev = pd->device;
struct nes_pd *nespd = to_nespd(pd);
struct nes_vnic *nesvnic = to_nesvnic(ibdev);
struct nes_device *nesdev = nesvnic->nesdev;
struct nes_adapter *nesadapter = nesdev->nesadapter;
- struct nes_ucontext *nesucontext;
struct nes_alloc_pd_resp uresp;
u32 pd_num = 0;
int err;
+ struct nes_ucontext *nesucontext = rdma_udata_to_drv_context(
+ udata, struct nes_ucontext, ibucontext);
- nes_debug(NES_DBG_PD, "nesvnic=%p, netdev=%p %s, ibdev=%p, context=%p, netdev refcnt=%u\n",
- nesvnic, nesdev->netdev[0], nesdev->netdev[0]->name, ibdev, context,
- netdev_refcnt_read(nesvnic->netdev));
+ nes_debug(
+ NES_DBG_PD,
+ "nesvnic=%p, netdev=%p %s, ibdev=%p, context=%p, netdev refcnt=%u\n",
+ nesvnic, nesdev->netdev[0], nesdev->netdev[0]->name, ibdev,
+ &nesucontext->ibucontext, netdev_refcnt_read(nesvnic->netdev));
err = nes_alloc_resource(nesadapter, nesadapter->allocated_pds,
nesadapter->max_pd, &pd_num, &nesadapter->next_pd, NES_RESOURCE_PD);
nespd->pd_id = (pd_num << (PAGE_SHIFT-12)) + nesadapter->base_pd;
- if (context) {
- nesucontext = to_nesucontext(context);
+ if (udata) {
nespd->mmap_db_index = find_next_zero_bit(nesucontext->allocated_doorbells,
NES_MAX_USER_DB_REGIONS, nesucontext->first_free_db);
nes_debug(NES_DBG_PD, "find_first_zero_biton doorbells returned %u, mapping pd_id %u.\n",
/**
* nes_dealloc_pd
*/
-static void nes_dealloc_pd(struct ib_pd *ibpd)
+static void nes_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct nes_ucontext *nesucontext;
struct nes_pd *nespd = to_nespd(ibpd);
struct nes_device *nesdev = nesvnic->nesdev;
struct nes_adapter *nesadapter = nesdev->nesadapter;
- if ((ibpd->uobject) && (ibpd->uobject->context)) {
- nesucontext = to_nesucontext(ibpd->uobject->context);
+ if (udata) {
+ nesucontext =
+ rdma_udata_to_drv_context(
+ udata,
+ struct nes_ucontext,
+ ibucontext);
nes_debug(NES_DBG_PD, "Clearing bit %u from allocated doorbells\n",
nespd->mmap_db_index);
clear_bit(nespd->mmap_db_index, nesucontext->allocated_doorbells);
}
if (req.user_qp_buffer)
nesqp->nesuqp_addr = req.user_qp_buffer;
- if (udata) {
- nesqp->user_mode = 1;
- if (virt_wqs) {
- err = 1;
- list_for_each_entry(nespbl, &nes_ucontext->qp_reg_mem_list, list) {
- if (nespbl->user_base == (unsigned long )req.user_wqe_buffers) {
- list_del(&nespbl->list);
- err = 0;
- nes_debug(NES_DBG_QP, "Found PBL for virtual QP. nespbl=%p. user_base=0x%lx\n",
- nespbl, nespbl->user_base);
- break;
- }
- }
- if (err) {
- nes_debug(NES_DBG_QP, "Didn't Find PBL for virtual QP. address = %llx.\n",
- (long long unsigned int)req.user_wqe_buffers);
- nes_free_resource(nesadapter, nesadapter->allocated_qps, qp_num);
- kfree(nesqp->allocated_buffer);
- return ERR_PTR(-EFAULT);
+
+ nesqp->user_mode = 1;
+ if (virt_wqs) {
+ err = 1;
+ list_for_each_entry(nespbl, &nes_ucontext->qp_reg_mem_list, list) {
+ if (nespbl->user_base == (unsigned long )req.user_wqe_buffers) {
+ list_del(&nespbl->list);
+ err = 0;
+ nes_debug(NES_DBG_QP, "Found PBL for virtual QP. nespbl=%p. user_base=0x%lx\n",
+ nespbl, nespbl->user_base);
+ break;
}
}
-
- nesqp->mmap_sq_db_index =
- find_next_zero_bit(nes_ucontext->allocated_wqs,
- NES_MAX_USER_WQ_REGIONS, nes_ucontext->first_free_wq);
- /* nes_debug(NES_DBG_QP, "find_first_zero_biton wqs returned %u\n",
- nespd->mmap_db_index); */
- if (nesqp->mmap_sq_db_index >= NES_MAX_USER_WQ_REGIONS) {
- nes_debug(NES_DBG_QP,
- "db index > max user regions, failing create QP\n");
+ if (err) {
+ nes_debug(NES_DBG_QP, "Didn't Find PBL for virtual QP. address = %llx.\n",
+ (long long unsigned int)req.user_wqe_buffers);
nes_free_resource(nesadapter, nesadapter->allocated_qps, qp_num);
- if (virt_wqs) {
- pci_free_consistent(nesdev->pcidev, nespbl->pbl_size, nespbl->pbl_vbase,
- nespbl->pbl_pbase);
- kfree(nespbl);
- }
kfree(nesqp->allocated_buffer);
- return ERR_PTR(-ENOMEM);
+ return ERR_PTR(-EFAULT);
}
- set_bit(nesqp->mmap_sq_db_index, nes_ucontext->allocated_wqs);
- nes_ucontext->mmap_nesqp[nesqp->mmap_sq_db_index] = nesqp;
- nes_ucontext->first_free_wq = nesqp->mmap_sq_db_index + 1;
- } else {
+ }
+
+ nesqp->mmap_sq_db_index =
+ find_next_zero_bit(nes_ucontext->allocated_wqs,
+ NES_MAX_USER_WQ_REGIONS, nes_ucontext->first_free_wq);
+ /* nes_debug(NES_DBG_QP, "find_first_zero_biton wqs returned %u\n",
+ nespd->mmap_db_index); */
+ if (nesqp->mmap_sq_db_index >= NES_MAX_USER_WQ_REGIONS) {
+ nes_debug(NES_DBG_QP,
+ "db index > max user regions, failing create QP\n");
nes_free_resource(nesadapter, nesadapter->allocated_qps, qp_num);
+ if (virt_wqs) {
+ pci_free_consistent(nesdev->pcidev, nespbl->pbl_size, nespbl->pbl_vbase,
+ nespbl->pbl_pbase);
+ kfree(nespbl);
+ }
kfree(nesqp->allocated_buffer);
- return ERR_PTR(-EFAULT);
+ return ERR_PTR(-ENOMEM);
}
+ set_bit(nesqp->mmap_sq_db_index, nes_ucontext->allocated_wqs);
+ nes_ucontext->mmap_nesqp[nesqp->mmap_sq_db_index] = nesqp;
+ nes_ucontext->first_free_wq = nesqp->mmap_sq_db_index + 1;
}
err = (!virt_wqs) ? nes_setup_mmap_qp(nesqp, nesvnic, sq_size, rq_size) :
nes_setup_virt_qp(nesqp, nespbl, nesvnic, sq_size, rq_size);
/**
* nes_destroy_qp
*/
-static int nes_destroy_qp(struct ib_qp *ibqp)
+static int nes_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
{
struct nes_qp *nesqp = to_nesqp(ibqp);
struct nes_ucontext *nes_ucontext;
}
if (nesqp->user_mode) {
- if ((ibqp->uobject)&&(ibqp->uobject->context)) {
- nes_ucontext = to_nesucontext(ibqp->uobject->context);
+ if (udata) {
+ nes_ucontext =
+ rdma_udata_to_drv_context(
+ udata,
+ struct nes_ucontext,
+ ibucontext);
clear_bit(nesqp->mmap_sq_db_index, nes_ucontext->allocated_wqs);
nes_ucontext->mmap_nesqp[nesqp->mmap_sq_db_index] = NULL;
if (nes_ucontext->first_free_wq > nesqp->mmap_sq_db_index) {
*/
static struct ib_cq *nes_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
int entries = attr->cqe;
nescq->hw_cq.cq_number = cq_num;
nescq->ibcq.cqe = nescq->hw_cq.cq_size - 1;
+ if (udata) {
+ struct nes_ucontext *nes_ucontext = rdma_udata_to_drv_context(
+ udata, struct nes_ucontext, ibucontext);
- if (context) {
- nes_ucontext = to_nesucontext(context);
if (ib_copy_from_udata(&req, udata, sizeof (struct nes_create_cq_req))) {
nes_free_resource(nesadapter, nesadapter->allocated_cqs, cq_num);
kfree(nescq);
cqp_request = nes_get_cqp_request(nesdev);
if (cqp_request == NULL) {
nes_debug(NES_DBG_CQ, "Failed to get a cqp_request.\n");
- if (!context)
+ if (!udata)
pci_free_consistent(nesdev->pcidev, nescq->cq_mem_size, mem,
nescq->hw_cq.cq_pbase);
else {
if (nesadapter->free_4kpbl == 0) {
spin_unlock_irqrestore(&nesadapter->pbl_lock, flags);
nes_free_cqp_request(nesdev, cqp_request);
- if (!context)
+ if (!udata)
pci_free_consistent(nesdev->pcidev, nescq->cq_mem_size, mem,
nescq->hw_cq.cq_pbase);
else {
if (nesadapter->free_256pbl == 0) {
spin_unlock_irqrestore(&nesadapter->pbl_lock, flags);
nes_free_cqp_request(nesdev, cqp_request);
- if (!context)
+ if (!udata)
pci_free_consistent(nesdev->pcidev, nescq->cq_mem_size, mem,
nescq->hw_cq.cq_pbase);
else {
set_wqe_32bit_value(cqp_wqe->wqe_words, NES_CQP_WQE_ID_IDX,
(nescq->hw_cq.cq_number | ((u32)nesdev->ceq_index << 16)));
- if (context) {
+ if (udata) {
if (pbl_entries != 1)
u64temp = (u64)nespbl->pbl_pbase;
else
nescq->hw_cq.cq_number, ret);
if ((!ret) || (cqp_request->major_code)) {
nes_put_cqp_request(nesdev, cqp_request);
- if (!context)
+ if (!udata)
pci_free_consistent(nesdev->pcidev, nescq->cq_mem_size, mem,
nescq->hw_cq.cq_pbase);
else {
}
nes_put_cqp_request(nesdev, cqp_request);
- if (context) {
+ if (udata) {
/* free the nespbl */
pci_free_consistent(nesdev->pcidev, nespbl->pbl_size, nespbl->pbl_vbase,
nespbl->pbl_pbase);
/**
* nes_destroy_cq
*/
-static int nes_destroy_cq(struct ib_cq *ib_cq)
+static int nes_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
{
struct nes_cq *nescq;
struct nes_device *nesdev;
/**
* nes_dereg_mr
*/
-static int nes_dereg_mr(struct ib_mr *ib_mr)
+static int nes_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
{
struct nes_mr *nesmr = to_nesmr(ib_mr);
struct nes_vnic *nesvnic = to_nesvnic(ib_mr->device);
.get_dev_fw_str = get_dev_fw_str,
.get_dma_mr = nes_get_dma_mr,
.get_port_immutable = nes_port_immutable,
+ .iw_accept = nes_accept,
+ .iw_add_ref = nes_add_ref,
+ .iw_connect = nes_connect,
+ .iw_create_listen = nes_create_listen,
+ .iw_destroy_listen = nes_destroy_listen,
+ .iw_get_qp = nes_get_qp,
+ .iw_reject = nes_reject,
+ .iw_rem_ref = nes_rem_ref,
.map_mr_sg = nes_map_mr_sg,
.mmap = nes_mmap,
.modify_qp = nes_modify_qp,
nesibdev->ibdev.num_comp_vectors = 1;
nesibdev->ibdev.dev.parent = &nesdev->pcidev->dev;
- nesibdev->ibdev.iwcm = kzalloc(sizeof(*nesibdev->ibdev.iwcm), GFP_KERNEL);
- if (nesibdev->ibdev.iwcm == NULL) {
- ib_dealloc_device(&nesibdev->ibdev);
- return NULL;
- }
- nesibdev->ibdev.iwcm->add_ref = nes_add_ref;
- nesibdev->ibdev.iwcm->rem_ref = nes_rem_ref;
- nesibdev->ibdev.iwcm->get_qp = nes_get_qp;
- nesibdev->ibdev.iwcm->connect = nes_connect;
- nesibdev->ibdev.iwcm->accept = nes_accept;
- nesibdev->ibdev.iwcm->reject = nes_reject;
- nesibdev->ibdev.iwcm->create_listen = nes_create_listen;
- nesibdev->ibdev.iwcm->destroy_listen = nes_destroy_listen;
-
ib_set_device_ops(&nesibdev->ibdev, &nes_dev_ops);
- memcpy(nesibdev->ibdev.iwcm->ifname, netdev->name,
- sizeof(nesibdev->ibdev.iwcm->ifname));
+ memcpy(nesibdev->ibdev.iw_ifname, netdev->name,
+ sizeof(nesibdev->ibdev.iw_ifname));
return nesibdev;
}
nes_unregister_ofa_device(nesibdev);
- kfree(nesibdev->ibdev.iwcm);
ib_dealloc_device(&nesibdev->ibdev);
}
return status;
}
-struct ib_ah *ocrdma_create_ah(struct ib_pd *ibpd, struct rdma_ah_attr *attr,
- u32 flags, struct ib_udata *udata)
+int ocrdma_create_ah(struct ib_ah *ibah, struct rdma_ah_attr *attr, u32 flags,
+ struct ib_udata *udata)
{
u32 *ahid_addr;
int status;
- struct ocrdma_ah *ah;
+ struct ocrdma_ah *ah = get_ocrdma_ah(ibah);
bool isvlan = false;
u16 vlan_tag = 0xffff;
const struct ib_gid_attr *sgid_attr;
- struct ocrdma_pd *pd = get_ocrdma_pd(ibpd);
- struct ocrdma_dev *dev = get_ocrdma_dev(ibpd->device);
+ struct ocrdma_pd *pd = get_ocrdma_pd(ibah->pd);
+ struct ocrdma_dev *dev = get_ocrdma_dev(ibah->device);
if ((attr->type != RDMA_AH_ATTR_TYPE_ROCE) ||
!(rdma_ah_get_ah_flags(attr) & IB_AH_GRH))
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
if (atomic_cmpxchg(&dev->update_sl, 1, 0))
ocrdma_init_service_level(dev);
- ah = kzalloc(sizeof(*ah), GFP_ATOMIC);
- if (!ah)
- return ERR_PTR(-ENOMEM);
+ sgid_attr = attr->grh.sgid_attr;
+ status = rdma_read_gid_l2_fields(sgid_attr, &vlan_tag, NULL);
+ if (status)
+ return status;
status = ocrdma_alloc_av(dev, ah);
if (status)
goto av_err;
- sgid_attr = attr->grh.sgid_attr;
- if (is_vlan_dev(sgid_attr->ndev))
- vlan_tag = vlan_dev_vlan_id(sgid_attr->ndev);
-
/* Get network header type for this GID */
ah->hdr_type = rdma_gid_attr_network_type(sgid_attr);
OCRDMA_AH_VLAN_VALID_SHIFT);
}
- return &ah->ibah;
+ return 0;
av_conf_err:
ocrdma_free_av(dev, ah);
av_err:
- kfree(ah);
- return ERR_PTR(status);
+ return status;
}
-int ocrdma_destroy_ah(struct ib_ah *ibah, u32 flags)
+void ocrdma_destroy_ah(struct ib_ah *ibah, u32 flags)
{
struct ocrdma_ah *ah = get_ocrdma_ah(ibah);
struct ocrdma_dev *dev = get_ocrdma_dev(ibah->device);
ocrdma_free_av(dev, ah);
- kfree(ah);
- return 0;
}
int ocrdma_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *attr)
OCRDMA_AH_L3_TYPE_SHIFT = 0x1D /* 29 bits */
};
-struct ib_ah *ocrdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
- u32 flags, struct ib_udata *udata);
-int ocrdma_destroy_ah(struct ib_ah *ah, u32 flags);
+int ocrdma_create_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr, u32 flags,
+ struct ib_udata *udata);
+void ocrdma_destroy_ah(struct ib_ah *ah, u32 flags);
int ocrdma_query_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr);
int ocrdma_process_mad(struct ib_device *,
int status;
struct rdma_ah_attr *ah_attr = &attrs->ah_attr;
const struct ib_gid_attr *sgid_attr;
- u32 vlan_id = 0xFFFF;
+ u16 vlan_id = 0xFFFF;
u8 mac_addr[6], hdr_type;
union {
struct sockaddr _sockaddr;
sizeof(cmd->params.dgid));
sgid_attr = ah_attr->grh.sgid_attr;
- vlan_id = rdma_vlan_dev_vlan_id(sgid_attr->ndev);
- memcpy(mac_addr, sgid_attr->ndev->dev_addr, ETH_ALEN);
+ status = rdma_read_gid_l2_fields(sgid_attr, &vlan_id, &mac_addr[0]);
+ if (status)
+ return status;
qp->sgid_idx = grh->sgid_index;
memcpy(&cmd->params.sgid[0], &sgid_attr->gid.raw[0],
return status;
}
-int ocrdma_mbx_destroy_srq(struct ocrdma_dev *dev, struct ocrdma_srq *srq)
+void ocrdma_mbx_destroy_srq(struct ocrdma_dev *dev, struct ocrdma_srq *srq)
{
- int status = -ENOMEM;
struct ocrdma_destroy_srq *cmd;
struct pci_dev *pdev = dev->nic_info.pdev;
cmd = ocrdma_init_emb_mqe(OCRDMA_CMD_DELETE_SRQ, sizeof(*cmd));
if (!cmd)
- return status;
+ return;
cmd->id = srq->id;
- status = ocrdma_mbx_cmd(dev, (struct ocrdma_mqe *)cmd);
+ ocrdma_mbx_cmd(dev, (struct ocrdma_mqe *)cmd);
if (srq->rq.va)
dma_free_coherent(&pdev->dev, srq->rq.len,
srq->rq.va, srq->rq.pa);
kfree(cmd);
- return status;
}
static int ocrdma_mbx_get_dcbx_config(struct ocrdma_dev *dev, u32 ptype,
return status;
}
-int ocrdma_free_av(struct ocrdma_dev *dev, struct ocrdma_ah *ah)
+void ocrdma_free_av(struct ocrdma_dev *dev, struct ocrdma_ah *ah)
{
unsigned long flags;
spin_lock_irqsave(&dev->av_tbl.lock, flags);
ah->av->valid = 0;
spin_unlock_irqrestore(&dev->av_tbl.lock, flags);
- return 0;
}
static int ocrdma_create_eqs(struct ocrdma_dev *dev)
struct ocrdma_pd *);
int ocrdma_mbx_modify_srq(struct ocrdma_srq *, struct ib_srq_attr *);
int ocrdma_mbx_query_srq(struct ocrdma_srq *, struct ib_srq_attr *);
-int ocrdma_mbx_destroy_srq(struct ocrdma_dev *, struct ocrdma_srq *);
+void ocrdma_mbx_destroy_srq(struct ocrdma_dev *dev, struct ocrdma_srq *srq);
-int ocrdma_alloc_av(struct ocrdma_dev *, struct ocrdma_ah *);
-int ocrdma_free_av(struct ocrdma_dev *, struct ocrdma_ah *);
+int ocrdma_alloc_av(struct ocrdma_dev *dev, struct ocrdma_ah *ah);
+void ocrdma_free_av(struct ocrdma_dev *dev, struct ocrdma_ah *ah);
int ocrdma_qp_state_change(struct ocrdma_qp *, enum ib_qp_state new_state,
enum ib_qp_state *old_ib_state);
MODULE_AUTHOR("Emulex Corporation");
MODULE_LICENSE("Dual BSD/GPL");
-static DEFINE_IDR(ocrdma_dev_id);
-
void ocrdma_get_guid(struct ocrdma_dev *dev, u8 *guid)
{
u8 mac_addr[6];
.get_dev_fw_str = get_dev_fw_str,
.get_dma_mr = ocrdma_get_dma_mr,
.get_link_layer = ocrdma_link_layer,
- .get_netdev = ocrdma_get_netdev,
.get_port_immutable = ocrdma_port_immutable,
.map_mr_sg = ocrdma_map_mr_sg,
.mmap = ocrdma_mmap,
.reg_user_mr = ocrdma_reg_user_mr,
.req_notify_cq = ocrdma_arm_cq,
.resize_cq = ocrdma_resize_cq,
+
+ INIT_RDMA_OBJ_SIZE(ib_ah, ocrdma_ah, ibah),
INIT_RDMA_OBJ_SIZE(ib_pd, ocrdma_pd, ibpd),
INIT_RDMA_OBJ_SIZE(ib_ucontext, ocrdma_ucontext, ibucontext),
};
.modify_srq = ocrdma_modify_srq,
.post_srq_recv = ocrdma_post_srq_recv,
.query_srq = ocrdma_query_srq,
+
+ INIT_RDMA_OBJ_SIZE(ib_srq, ocrdma_srq, ibsrq),
};
static int ocrdma_register_device(struct ocrdma_dev *dev)
{
+ int ret;
+
ocrdma_get_guid(dev, (u8 *)&dev->ibdev.node_guid);
BUILD_BUG_ON(sizeof(OCRDMA_NODE_DESC) > IB_DEVICE_NODE_DESC_MAX);
memcpy(dev->ibdev.node_desc, OCRDMA_NODE_DESC,
}
rdma_set_device_sysfs_group(&dev->ibdev, &ocrdma_attr_group);
dev->ibdev.driver_id = RDMA_DRIVER_OCRDMA;
+ ret = ib_device_set_netdev(&dev->ibdev, dev->nic_info.netdev, 1);
+ if (ret)
+ return ret;
+
return ib_register_device(&dev->ibdev, "ocrdma%d");
}
pr_err("Unable to allocate ib device\n");
return NULL;
}
+
dev->mbx_cmd = kzalloc(sizeof(struct ocrdma_mqe_emb_cmd), GFP_KERNEL);
if (!dev->mbx_cmd)
- goto idr_err;
+ goto init_err;
memcpy(&dev->nic_info, dev_info, sizeof(*dev_info));
- dev->id = idr_alloc(&ocrdma_dev_id, NULL, 0, 0, GFP_KERNEL);
- if (dev->id < 0)
- goto idr_err;
-
+ dev->id = PCI_FUNC(dev->nic_info.pdev->devfn);
status = ocrdma_init_hw(dev);
if (status)
goto init_err;
ocrdma_free_resources(dev);
ocrdma_cleanup_hw(dev);
init_err:
- idr_remove(&ocrdma_dev_id, dev->id);
-idr_err:
kfree(dev->mbx_cmd);
ib_dealloc_device(&dev->ibdev);
pr_err("%s() leaving. ret=%d\n", __func__, status);
static void ocrdma_remove_free(struct ocrdma_dev *dev)
{
- idr_remove(&ocrdma_dev_id, dev->id);
kfree(dev->mbx_cmd);
ib_dealloc_device(&dev->ibdev);
}
{
be_roce_unregister_driver(&ocrdma_drv);
ocrdma_rem_debugfs();
- idr_destroy(&ocrdma_dev_id);
}
module_init(ocrdma_init_module);
#include <rdma/ib_umem.h>
#include <rdma/ib_addr.h>
#include <rdma/ib_cache.h>
+#include <rdma/uverbs_ioctl.h>
#include "ocrdma.h"
#include "ocrdma_hw.h"
return 0;
}
-struct net_device *ocrdma_get_netdev(struct ib_device *ibdev, u8 port_num)
-{
- struct ocrdma_dev *dev;
- struct net_device *ndev = NULL;
-
- rcu_read_lock();
-
- dev = get_ocrdma_dev(ibdev);
- if (dev)
- ndev = dev->nic_info.netdev;
- if (ndev)
- dev_hold(ndev);
-
- rcu_read_unlock();
-
- return ndev;
-}
-
static inline void get_link_speed_and_width(struct ocrdma_dev *dev,
u8 *ib_speed, u8 *ib_width)
{
return status;
}
+/*
+ * NOTE:
+ *
+ * ocrdma_ucontext must be used here because this function is also
+ * called from ocrdma_alloc_ucontext where ib_udata does not have
+ * valid ib_ucontext pointer. ib_uverbs_get_context does not call
+ * uobj_{alloc|get_xxx} helpers which are used to store the
+ * ib_ucontext in uverbs_attr_bundle wrapping the ib_udata. so
+ * ib_udata does NOT imply valid ib_ucontext here!
+ */
static int _ocrdma_alloc_pd(struct ocrdma_dev *dev, struct ocrdma_pd *pd,
struct ocrdma_ucontext *uctx,
struct ib_udata *udata)
}
static int ocrdma_copy_pd_uresp(struct ocrdma_dev *dev, struct ocrdma_pd *pd,
- struct ib_ucontext *ib_ctx,
struct ib_udata *udata)
{
int status;
u64 dpp_page_addr = 0;
u32 db_page_size;
struct ocrdma_alloc_pd_uresp rsp;
- struct ocrdma_ucontext *uctx = get_ocrdma_ucontext(ib_ctx);
+ struct ocrdma_ucontext *uctx = rdma_udata_to_drv_context(
+ udata, struct ocrdma_ucontext, ibucontext);
memset(&rsp, 0, sizeof(rsp));
rsp.id = pd->id;
return status;
}
-int ocrdma_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata)
+int ocrdma_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct ib_device *ibdev = ibpd->device;
struct ocrdma_dev *dev = get_ocrdma_dev(ibdev);
struct ocrdma_pd *pd;
- struct ocrdma_ucontext *uctx = NULL;
int status;
u8 is_uctx_pd = false;
+ struct ocrdma_ucontext *uctx = rdma_udata_to_drv_context(
+ udata, struct ocrdma_ucontext, ibucontext);
- if (udata && context) {
- uctx = get_ocrdma_ucontext(context);
+ if (udata) {
pd = ocrdma_get_ucontext_pd(uctx);
if (pd) {
is_uctx_pd = true;
goto exit;
pd_mapping:
- if (udata && context) {
- status = ocrdma_copy_pd_uresp(dev, pd, context, udata);
+ if (udata) {
+ status = ocrdma_copy_pd_uresp(dev, pd, udata);
if (status)
goto err;
}
return status;
}
-void ocrdma_dealloc_pd(struct ib_pd *ibpd)
+void ocrdma_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct ocrdma_pd *pd = get_ocrdma_pd(ibpd);
struct ocrdma_dev *dev = get_ocrdma_dev(ibpd->device);
return ERR_PTR(status);
}
-int ocrdma_dereg_mr(struct ib_mr *ib_mr)
+int ocrdma_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
{
struct ocrdma_mr *mr = get_ocrdma_mr(ib_mr);
struct ocrdma_dev *dev = get_ocrdma_dev(ib_mr->device);
}
static int ocrdma_copy_cq_uresp(struct ocrdma_dev *dev, struct ocrdma_cq *cq,
- struct ib_udata *udata,
- struct ib_ucontext *ib_ctx)
+ struct ib_udata *udata)
{
int status;
- struct ocrdma_ucontext *uctx = get_ocrdma_ucontext(ib_ctx);
+ struct ocrdma_ucontext *uctx = rdma_udata_to_drv_context(
+ udata, struct ocrdma_ucontext, ibucontext);
struct ocrdma_create_cq_uresp uresp;
+ /* this must be user flow! */
+ if (!udata)
+ return -EINVAL;
+
memset(&uresp, 0, sizeof(uresp));
uresp.cq_id = cq->id;
uresp.page_size = PAGE_ALIGN(cq->len);
struct ib_cq *ocrdma_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *ib_ctx,
struct ib_udata *udata)
{
int entries = attr->cqe;
struct ocrdma_cq *cq;
struct ocrdma_dev *dev = get_ocrdma_dev(ibdev);
- struct ocrdma_ucontext *uctx = NULL;
+ struct ocrdma_ucontext *uctx = rdma_udata_to_drv_context(
+ udata, struct ocrdma_ucontext, ibucontext);
u16 pd_id = 0;
int status;
struct ocrdma_create_cq_ureq ureq;
INIT_LIST_HEAD(&cq->sq_head);
INIT_LIST_HEAD(&cq->rq_head);
- if (ib_ctx) {
- uctx = get_ocrdma_ucontext(ib_ctx);
+ if (udata)
pd_id = uctx->cntxt_pd->id;
- }
status = ocrdma_mbx_create_cq(dev, cq, entries, ureq.dpp_cq, pd_id);
if (status) {
kfree(cq);
return ERR_PTR(status);
}
- if (ib_ctx) {
- status = ocrdma_copy_cq_uresp(dev, cq, udata, ib_ctx);
+ if (udata) {
+ status = ocrdma_copy_cq_uresp(dev, cq, udata);
if (status)
goto ctx_err;
}
spin_unlock_irqrestore(&cq->cq_lock, flags);
}
-int ocrdma_destroy_cq(struct ib_cq *ibcq)
+int ocrdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
{
struct ocrdma_cq *cq = get_ocrdma_cq(ibcq);
struct ocrdma_eq *eq = NULL;
spin_unlock_irqrestore(&dev->flush_q_lock, flags);
}
-int ocrdma_destroy_qp(struct ib_qp *ibqp)
+int ocrdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
{
struct ocrdma_pd *pd;
struct ocrdma_qp *qp;
return status;
}
-struct ib_srq *ocrdma_create_srq(struct ib_pd *ibpd,
- struct ib_srq_init_attr *init_attr,
- struct ib_udata *udata)
+int ocrdma_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *init_attr,
+ struct ib_udata *udata)
{
- int status = -ENOMEM;
- struct ocrdma_pd *pd = get_ocrdma_pd(ibpd);
- struct ocrdma_dev *dev = get_ocrdma_dev(ibpd->device);
- struct ocrdma_srq *srq;
+ int status;
+ struct ocrdma_pd *pd = get_ocrdma_pd(ibsrq->pd);
+ struct ocrdma_dev *dev = get_ocrdma_dev(ibsrq->device);
+ struct ocrdma_srq *srq = get_ocrdma_srq(ibsrq);
if (init_attr->attr.max_sge > dev->attr.max_recv_sge)
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
if (init_attr->attr.max_wr > dev->attr.max_rqe)
- return ERR_PTR(-EINVAL);
-
- srq = kzalloc(sizeof(*srq), GFP_KERNEL);
- if (!srq)
- return ERR_PTR(status);
+ return -EINVAL;
spin_lock_init(&srq->q_lock);
srq->pd = pd;
srq->db = dev->nic_info.db + (pd->id * dev->nic_info.db_page_size);
status = ocrdma_mbx_create_srq(dev, srq, init_attr, pd);
if (status)
- goto err;
+ return status;
- if (udata == NULL) {
- status = -ENOMEM;
+ if (!udata) {
srq->rqe_wr_id_tbl = kcalloc(srq->rq.max_cnt, sizeof(u64),
GFP_KERNEL);
- if (srq->rqe_wr_id_tbl == NULL)
+ if (!srq->rqe_wr_id_tbl) {
+ status = -ENOMEM;
goto arm_err;
+ }
srq->bit_fields_len = (srq->rq.max_cnt / 32) +
(srq->rq.max_cnt % 32 ? 1 : 0);
srq->idx_bit_fields =
kmalloc_array(srq->bit_fields_len, sizeof(u32),
GFP_KERNEL);
- if (srq->idx_bit_fields == NULL)
+ if (!srq->idx_bit_fields) {
+ status = -ENOMEM;
goto arm_err;
+ }
memset(srq->idx_bit_fields, 0xff,
srq->bit_fields_len * sizeof(u32));
}
goto arm_err;
}
- return &srq->ibsrq;
+ return 0;
arm_err:
ocrdma_mbx_destroy_srq(dev, srq);
-err:
kfree(srq->rqe_wr_id_tbl);
kfree(srq->idx_bit_fields);
- kfree(srq);
- return ERR_PTR(status);
+ return status;
}
int ocrdma_modify_srq(struct ib_srq *ibsrq,
return status;
}
-int ocrdma_destroy_srq(struct ib_srq *ibsrq)
+void ocrdma_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)
{
- int status;
struct ocrdma_srq *srq;
struct ocrdma_dev *dev = get_ocrdma_dev(ibsrq->device);
srq = get_ocrdma_srq(ibsrq);
- status = ocrdma_mbx_destroy_srq(dev, srq);
+ ocrdma_mbx_destroy_srq(dev, srq);
if (srq->pd->uctx)
ocrdma_del_mmap(srq->pd->uctx, (u64) srq->rq.pa,
kfree(srq->idx_bit_fields);
kfree(srq->rqe_wr_id_tbl);
- kfree(srq);
- return status;
}
/* unprivileged verbs and their support functions. */
return 0;
}
-struct ib_mr *ocrdma_alloc_mr(struct ib_pd *ibpd,
- enum ib_mr_type mr_type,
- u32 max_num_sg)
+struct ib_mr *ocrdma_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
int status;
struct ocrdma_mr *mr;
ocrdma_query_protocol(struct ib_device *device, u8 port_num);
void ocrdma_get_guid(struct ocrdma_dev *, u8 *guid);
-struct net_device *ocrdma_get_netdev(struct ib_device *device, u8 port_num);
int ocrdma_query_pkey(struct ib_device *, u8 port, u16 index, u16 *pkey);
int ocrdma_alloc_ucontext(struct ib_ucontext *uctx, struct ib_udata *udata);
int ocrdma_mmap(struct ib_ucontext *, struct vm_area_struct *vma);
-int ocrdma_alloc_pd(struct ib_pd *pd, struct ib_ucontext *uctx,
- struct ib_udata *udata);
-void ocrdma_dealloc_pd(struct ib_pd *pd);
+int ocrdma_alloc_pd(struct ib_pd *pd, struct ib_udata *udata);
+void ocrdma_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata);
struct ib_cq *ocrdma_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *ib_ctx,
struct ib_udata *udata);
int ocrdma_resize_cq(struct ib_cq *, int cqe, struct ib_udata *);
-int ocrdma_destroy_cq(struct ib_cq *);
+int ocrdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata);
struct ib_qp *ocrdma_create_qp(struct ib_pd *,
struct ib_qp_init_attr *attrs,
int ocrdma_query_qp(struct ib_qp *,
struct ib_qp_attr *qp_attr,
int qp_attr_mask, struct ib_qp_init_attr *);
-int ocrdma_destroy_qp(struct ib_qp *);
+int ocrdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata);
void ocrdma_del_flush_qp(struct ocrdma_qp *qp);
-struct ib_srq *ocrdma_create_srq(struct ib_pd *, struct ib_srq_init_attr *,
- struct ib_udata *);
+int ocrdma_create_srq(struct ib_srq *srq, struct ib_srq_init_attr *attr,
+ struct ib_udata *udata);
int ocrdma_modify_srq(struct ib_srq *, struct ib_srq_attr *,
enum ib_srq_attr_mask, struct ib_udata *);
int ocrdma_query_srq(struct ib_srq *, struct ib_srq_attr *);
-int ocrdma_destroy_srq(struct ib_srq *);
+void ocrdma_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata);
int ocrdma_post_srq_recv(struct ib_srq *, const struct ib_recv_wr *,
const struct ib_recv_wr **bad_recv_wr);
-int ocrdma_dereg_mr(struct ib_mr *);
+int ocrdma_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata);
struct ib_mr *ocrdma_get_dma_mr(struct ib_pd *, int acc);
struct ib_mr *ocrdma_reg_user_mr(struct ib_pd *, u64 start, u64 length,
u64 virt, int acc, struct ib_udata *);
-struct ib_mr *ocrdma_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg);
+struct ib_mr *ocrdma_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata);
int ocrdma_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
unsigned int *sg_offset);
#include <linux/iommu.h>
#include <linux/pci.h>
#include <net/addrconf.h>
-#include <linux/idr.h>
#include <linux/qed/qed_chain.h>
#include <linux/qed/qed_if.h>
(fw_ver >> 8) & 0xFF, fw_ver & 0xFF);
}
-static struct net_device *qedr_get_netdev(struct ib_device *dev, u8 port_num)
-{
- struct qedr_dev *qdev;
-
- qdev = get_qedr_dev(dev);
- dev_hold(qdev->ndev);
-
- /* The HW vendor's device driver must guarantee
- * that this function returns NULL before the net device has finished
- * NETDEV_UNREGISTER state.
- */
- return qdev->ndev;
-}
-
static int qedr_roce_port_immutable(struct ib_device *ibdev, u8 port_num,
struct ib_port_immutable *immutable)
{
static const struct ib_device_ops qedr_iw_dev_ops = {
.get_port_immutable = qedr_iw_port_immutable,
+ .iw_accept = qedr_iw_accept,
+ .iw_add_ref = qedr_iw_qp_add_ref,
+ .iw_connect = qedr_iw_connect,
+ .iw_create_listen = qedr_iw_create_listen,
+ .iw_destroy_listen = qedr_iw_destroy_listen,
+ .iw_get_qp = qedr_iw_get_qp,
+ .iw_reject = qedr_iw_reject,
+ .iw_rem_ref = qedr_iw_qp_rem_ref,
.query_gid = qedr_iw_query_gid,
};
ib_set_device_ops(&dev->ibdev, &qedr_iw_dev_ops);
- dev->ibdev.iwcm = kzalloc(sizeof(*dev->ibdev.iwcm), GFP_KERNEL);
- if (!dev->ibdev.iwcm)
- return -ENOMEM;
-
- dev->ibdev.iwcm->connect = qedr_iw_connect;
- dev->ibdev.iwcm->accept = qedr_iw_accept;
- dev->ibdev.iwcm->reject = qedr_iw_reject;
- dev->ibdev.iwcm->create_listen = qedr_iw_create_listen;
- dev->ibdev.iwcm->destroy_listen = qedr_iw_destroy_listen;
- dev->ibdev.iwcm->add_ref = qedr_iw_qp_add_ref;
- dev->ibdev.iwcm->rem_ref = qedr_iw_qp_rem_ref;
- dev->ibdev.iwcm->get_qp = qedr_iw_get_qp;
-
- memcpy(dev->ibdev.iwcm->ifname,
- dev->ndev->name, sizeof(dev->ibdev.iwcm->ifname));
+ memcpy(dev->ibdev.iw_ifname,
+ dev->ndev->name, sizeof(dev->ibdev.iw_ifname));
return 0;
}
.get_dev_fw_str = qedr_get_dev_fw_str,
.get_dma_mr = qedr_get_dma_mr,
.get_link_layer = qedr_link_layer,
- .get_netdev = qedr_get_netdev,
.map_mr_sg = qedr_map_mr_sg,
.mmap = qedr_mmap,
.modify_port = qedr_modify_port,
.reg_user_mr = qedr_reg_user_mr,
.req_notify_cq = qedr_arm_cq,
.resize_cq = qedr_resize_cq,
+
+ INIT_RDMA_OBJ_SIZE(ib_ah, qedr_ah, ibah),
INIT_RDMA_OBJ_SIZE(ib_pd, qedr_pd, ibpd),
+ INIT_RDMA_OBJ_SIZE(ib_srq, qedr_srq, ibsrq),
INIT_RDMA_OBJ_SIZE(ib_ucontext, qedr_ucontext, ibucontext),
};
ib_set_device_ops(&dev->ibdev, &qedr_dev_ops);
dev->ibdev.driver_id = RDMA_DRIVER_QEDR;
+ rc = ib_device_set_netdev(&dev->ibdev, dev->ndev, 1);
+ if (rc)
+ return rc;
+
return ib_register_device(&dev->ibdev, "qedr%d");
}
spin_lock_init(&dev->sgid_lock);
if (IS_IWARP(dev)) {
- spin_lock_init(&dev->qpidr.idr_lock);
- idr_init(&dev->qpidr.idr);
+ xa_init_flags(&dev->qps, XA_FLAGS_LOCK_IRQ);
dev->iwarp_wq = create_singlethread_workqueue("qedr_iwarpq");
}
break;
case EVENT_TYPE_SRQ:
srq_id = (u16)roce_handle64;
- spin_lock_irqsave(&dev->srqidr.idr_lock, flags);
- srq = idr_find(&dev->srqidr.idr, srq_id);
+ xa_lock_irqsave(&dev->srqs, flags);
+ srq = xa_load(&dev->srqs, srq_id);
if (srq) {
ibsrq = &srq->ibsrq;
if (ibsrq->event_handler) {
"SRQ event with NULL pointer ibsrq. Handle=%llx\n",
roce_handle64);
}
- spin_unlock_irqrestore(&dev->srqidr.idr_lock, flags);
+ xa_unlock_irqrestore(&dev->srqs, flags);
DP_NOTICE(dev, "SRQ event %d on handle %p\n", e_code, srq);
default:
break;
#define __QEDR_H__
#include <linux/pci.h>
-#include <linux/idr.h>
+#include <linux/xarray.h>
#include <rdma/ib_addr.h>
#include <linux/qed/qed_if.h>
#include <linux/qed/qed_chain.h>
#define QEDR_ENET_STATE_BIT (0)
-struct qedr_idr {
- spinlock_t idr_lock; /* Protect idr data-structure */
- struct idr idr;
-};
-
struct qedr_dev {
struct ib_device ibdev;
struct qed_dev *cdev;
struct qedr_cq *gsi_rqcq;
struct qedr_qp *gsi_qp;
enum qed_rdma_type rdma_type;
- struct qedr_idr qpidr;
- struct qedr_idr srqidr;
+ struct xarray qps;
+ struct xarray srqs;
struct workqueue_struct *iwarp_wq;
u16 iwarp_max_mtu;
int rc = 0;
int i;
- qp = idr_find(&dev->qpidr.idr, conn_param->qpn);
+ qp = xa_load(&dev->qps, conn_param->qpn);
if (unlikely(!qp))
return -EINVAL;
DP_DEBUG(dev, QEDR_MSG_IWARP, "Accept on qpid=%d\n", conn_param->qpn);
- qp = idr_find(&dev->qpidr.idr, conn_param->qpn);
+ qp = xa_load(&dev->qps, conn_param->qpn);
if (!qp) {
DP_ERR(dev, "Invalid QP number %d\n", conn_param->qpn);
return -EINVAL;
struct qedr_qp *qp = get_qedr_qp(ibqp);
if (atomic_dec_and_test(&qp->refcnt)) {
- spin_lock_irq(&qp->dev->qpidr.idr_lock);
- idr_remove(&qp->dev->qpidr.idr, qp->qp_id);
- spin_unlock_irq(&qp->dev->qpidr.idr_lock);
+ xa_erase_irq(&qp->dev->qps, qp->qp_id);
kfree(qp);
}
}
{
struct qedr_dev *dev = get_qedr_dev(ibdev);
- return idr_find(&dev->qpidr.idr, qpn);
+ return xa_load(&dev->qps, qpn);
}
bool has_udp = false;
int i;
- send_size = 0;
- for (i = 0; i < swr->num_sge; ++i)
- send_size += swr->sg_list[i].length;
+ rc = rdma_read_gid_l2_fields(sgid_attr, &vlan_id, NULL);
+ if (rc)
+ return rc;
- vlan_id = rdma_vlan_dev_vlan_id(sgid_attr->ndev);
if (vlan_id < VLAN_CFI_MASK)
has_vlan = true;
+ send_size = 0;
+ for (i = 0; i < swr->num_sge; ++i)
+ send_size += swr->sg_list[i].length;
+
has_udp = (sgid_attr->gid_type == IB_GID_TYPE_ROCE_UDP_ENCAP);
if (!has_udp) {
/* RoCE v1 */
#include <rdma/ib_umem.h>
#include <rdma/ib_addr.h>
#include <rdma/ib_cache.h>
+#include <rdma/uverbs_ioctl.h>
#include <linux/qed/common_hsi.h>
#include "qedr_hsi_rdma.h"
vma->vm_page_prot);
}
-int qedr_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata)
+int qedr_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct ib_device *ibdev = ibpd->device;
struct qedr_dev *dev = get_qedr_dev(ibdev);
int rc;
DP_DEBUG(dev, QEDR_MSG_INIT, "Function called from: %s\n",
- (udata && context) ? "User Lib" : "Kernel");
+ udata ? "User Lib" : "Kernel");
if (!dev->rdma_ctx) {
DP_ERR(dev, "invalid RDMA context\n");
pd->pd_id = pd_id;
- if (udata && context) {
+ if (udata) {
struct qedr_alloc_pd_uresp uresp = {
.pd_id = pd_id,
};
+ struct qedr_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct qedr_ucontext, ibucontext);
rc = qedr_ib_copy_to_udata(udata, &uresp, sizeof(uresp));
if (rc) {
return rc;
}
- pd->uctx = get_qedr_ucontext(context);
+ pd->uctx = context;
pd->uctx->pd = pd;
}
return 0;
}
-void qedr_dealloc_pd(struct ib_pd *ibpd)
+void qedr_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct qedr_dev *dev = get_qedr_dev(ibpd->device);
struct qedr_pd *pd = get_qedr_pd(ibpd);
struct ib_cq *qedr_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *ib_ctx, struct ib_udata *udata)
+ struct ib_udata *udata)
{
- struct qedr_ucontext *ctx = get_qedr_ucontext(ib_ctx);
+ struct qedr_ucontext *ctx = rdma_udata_to_drv_context(
+ udata, struct qedr_ucontext, ibucontext);
struct qed_rdma_destroy_cq_out_params destroy_oparams;
struct qed_rdma_destroy_cq_in_params destroy_iparams;
struct qedr_dev *dev = get_qedr_dev(ibdev);
cq->sig = QEDR_CQ_MAGIC_NUMBER;
spin_lock_init(&cq->cq_lock);
- if (ib_ctx) {
+ if (udata) {
rc = qedr_copy_cq_uresp(dev, cq, udata);
if (rc)
goto err3;
#define QEDR_DESTROY_CQ_MAX_ITERATIONS (10)
#define QEDR_DESTROY_CQ_ITER_DURATION (10)
-int qedr_destroy_cq(struct ib_cq *ibcq)
+int qedr_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
{
struct qedr_dev *dev = get_qedr_dev(ibcq->device);
struct qed_rdma_destroy_cq_out_params oparams;
dev->ops->common->chain_free(dev->cdev, &cq->pbl);
- if (ibcq->uobject && ibcq->uobject->context) {
+ if (udata) {
qedr_free_pbl(dev, &cq->q.pbl_info, cq->q.pbl_tbl);
ib_umem_release(cq->q.umem);
}
enum rdma_network_type nw_type;
const struct ib_global_route *grh = rdma_ah_read_grh(&attr->ah_attr);
u32 ipv4_addr;
+ int ret;
int i;
gid_attr = grh->sgid_attr;
- qp_params->vlan_id = rdma_vlan_dev_vlan_id(gid_attr->ndev);
+ ret = rdma_read_gid_l2_fields(gid_attr, &qp_params->vlan_id, NULL);
+ if (ret)
+ return ret;
nw_type = rdma_gid_attr_network_type(gid_attr);
switch (nw_type) {
}
}
-static int qedr_check_srq_params(struct ib_pd *ibpd, struct qedr_dev *dev,
+static int qedr_check_srq_params(struct qedr_dev *dev,
struct ib_srq_init_attr *attrs,
struct ib_udata *udata)
{
return rc;
}
-static int qedr_idr_add(struct qedr_dev *dev, struct qedr_idr *qidr,
- void *ptr, u32 id);
-static void qedr_idr_remove(struct qedr_dev *dev,
- struct qedr_idr *qidr, u32 id);
-
-struct ib_srq *qedr_create_srq(struct ib_pd *ibpd,
- struct ib_srq_init_attr *init_attr,
- struct ib_udata *udata)
+int qedr_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *init_attr,
+ struct ib_udata *udata)
{
struct qed_rdma_destroy_srq_in_params destroy_in_params;
struct qed_rdma_create_srq_in_params in_params = {};
- struct qedr_dev *dev = get_qedr_dev(ibpd->device);
+ struct qedr_dev *dev = get_qedr_dev(ibsrq->device);
struct qed_rdma_create_srq_out_params out_params;
- struct qedr_pd *pd = get_qedr_pd(ibpd);
+ struct qedr_pd *pd = get_qedr_pd(ibsrq->pd);
struct qedr_create_srq_ureq ureq = {};
u64 pbl_base_addr, phy_prod_pair_addr;
struct qedr_srq_hwq_info *hw_srq;
u32 page_cnt, page_size;
- struct qedr_srq *srq;
+ struct qedr_srq *srq = get_qedr_srq(ibsrq);
int rc = 0;
DP_DEBUG(dev, QEDR_MSG_QP,
"create SRQ called from %s (pd %p)\n",
(udata) ? "User lib" : "kernel", pd);
- rc = qedr_check_srq_params(ibpd, dev, init_attr, udata);
+ rc = qedr_check_srq_params(dev, init_attr, udata);
if (rc)
- return ERR_PTR(-EINVAL);
-
- srq = kzalloc(sizeof(*srq), GFP_KERNEL);
- if (!srq)
- return ERR_PTR(-ENOMEM);
+ return -EINVAL;
srq->dev = dev;
hw_srq = &srq->hw_srq;
goto err2;
}
- rc = qedr_idr_add(dev, &dev->srqidr, srq, srq->srq_id);
+ rc = xa_insert_irq(&dev->srqs, srq->srq_id, srq, GFP_KERNEL);
if (rc)
goto err2;
DP_DEBUG(dev, QEDR_MSG_SRQ,
"create srq: created srq with srq_id=0x%0x\n", srq->srq_id);
- return &srq->ibsrq;
+ return 0;
err2:
destroy_in_params.srq_id = srq->srq_id;
else
qedr_free_srq_kernel_params(srq);
err0:
- kfree(srq);
-
- return ERR_PTR(-EFAULT);
+ return -EFAULT;
}
-int qedr_destroy_srq(struct ib_srq *ibsrq)
+void qedr_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)
{
struct qed_rdma_destroy_srq_in_params in_params = {};
struct qedr_dev *dev = get_qedr_dev(ibsrq->device);
struct qedr_srq *srq = get_qedr_srq(ibsrq);
- qedr_idr_remove(dev, &dev->srqidr, srq->srq_id);
+ xa_erase_irq(&dev->srqs, srq->srq_id);
in_params.srq_id = srq->srq_id;
dev->ops->rdma_destroy_srq(dev->rdma_ctx, &in_params);
DP_DEBUG(dev, QEDR_MSG_SRQ,
"destroy srq: destroyed srq with srq_id=0x%0x\n",
srq->srq_id);
- kfree(srq);
-
- return 0;
}
int qedr_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
qp->usq.buf_len, qp->urq.buf_addr, qp->urq.buf_len);
}
-static int qedr_idr_add(struct qedr_dev *dev, struct qedr_idr *qidr,
- void *ptr, u32 id)
-{
- int rc;
-
- idr_preload(GFP_KERNEL);
- spin_lock_irq(&qidr->idr_lock);
-
- rc = idr_alloc(&qidr->idr, ptr, id, id + 1, GFP_ATOMIC);
-
- spin_unlock_irq(&qidr->idr_lock);
- idr_preload_end();
-
- return rc < 0 ? rc : 0;
-}
-
-static void qedr_idr_remove(struct qedr_dev *dev, struct qedr_idr *qidr, u32 id)
-{
- spin_lock_irq(&qidr->idr_lock);
- idr_remove(&qidr->idr, id);
- spin_unlock_irq(&qidr->idr_lock);
-}
-
static inline void
qedr_iwarp_populate_user_qp(struct qedr_dev *dev,
struct qedr_qp *qp,
qp->ibqp.qp_num = qp->qp_id;
if (rdma_protocol_iwarp(&dev->ibdev, 1)) {
- rc = qedr_idr_add(dev, &dev->qpidr, qp, qp->qp_id);
+ rc = xa_insert_irq(&dev->qps, qp->qp_id, qp, GFP_KERNEL);
if (rc)
goto err;
}
return rc;
}
-static int qedr_free_qp_resources(struct qedr_dev *dev, struct qedr_qp *qp)
+static int qedr_free_qp_resources(struct qedr_dev *dev, struct qedr_qp *qp,
+ struct ib_udata *udata)
{
int rc = 0;
return rc;
}
- if (qp->ibqp.uobject && qp->ibqp.uobject->context)
+ if (udata)
qedr_cleanup_user(dev, qp);
else
qedr_cleanup_kernel(dev, qp);
return 0;
}
-int qedr_destroy_qp(struct ib_qp *ibqp)
+int qedr_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
{
struct qedr_qp *qp = get_qedr_qp(ibqp);
struct qedr_dev *dev = qp->dev;
if (qp->qp_type == IB_QPT_GSI)
qedr_destroy_gsi_qp(dev);
- qedr_free_qp_resources(dev, qp);
+ qedr_free_qp_resources(dev, qp, udata);
if (atomic_dec_and_test(&qp->refcnt) &&
rdma_protocol_iwarp(&dev->ibdev, 1)) {
- qedr_idr_remove(dev, &dev->qpidr, qp->qp_id);
+ xa_erase_irq(&dev->qps, qp->qp_id);
kfree(qp);
}
return rc;
}
-struct ib_ah *qedr_create_ah(struct ib_pd *ibpd, struct rdma_ah_attr *attr,
- u32 flags, struct ib_udata *udata)
+int qedr_create_ah(struct ib_ah *ibah, struct rdma_ah_attr *attr, u32 flags,
+ struct ib_udata *udata)
{
- struct qedr_ah *ah;
-
- ah = kzalloc(sizeof(*ah), GFP_ATOMIC);
- if (!ah)
- return ERR_PTR(-ENOMEM);
+ struct qedr_ah *ah = get_qedr_ah(ibah);
rdma_copy_ah_attr(&ah->attr, attr);
- return &ah->ibah;
+ return 0;
}
-int qedr_destroy_ah(struct ib_ah *ibah, u32 flags)
+void qedr_destroy_ah(struct ib_ah *ibah, u32 flags)
{
struct qedr_ah *ah = get_qedr_ah(ibah);
rdma_destroy_ah_attr(&ah->attr);
- kfree(ah);
- return 0;
}
static void free_mr_info(struct qedr_dev *dev, struct mr_info *info)
return ERR_PTR(rc);
}
-int qedr_dereg_mr(struct ib_mr *ib_mr)
+int qedr_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata)
{
struct qedr_mr *mr = get_qedr_mr(ib_mr);
struct qedr_dev *dev = get_qedr_dev(ib_mr->device);
return ERR_PTR(rc);
}
-struct ib_mr *qedr_alloc_mr(struct ib_pd *ibpd,
- enum ib_mr_type mr_type, u32 max_num_sg)
+struct ib_mr *qedr_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
struct qedr_mr *mr;
void qedr_dealloc_ucontext(struct ib_ucontext *uctx);
int qedr_mmap(struct ib_ucontext *, struct vm_area_struct *vma);
-int qedr_alloc_pd(struct ib_pd *pd, struct ib_ucontext *uctx,
- struct ib_udata *udata);
-void qedr_dealloc_pd(struct ib_pd *pd);
+int qedr_alloc_pd(struct ib_pd *pd, struct ib_udata *udata);
+void qedr_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata);
struct ib_cq *qedr_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *ib_ctx,
struct ib_udata *udata);
int qedr_resize_cq(struct ib_cq *, int cqe, struct ib_udata *);
-int qedr_destroy_cq(struct ib_cq *);
+int qedr_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata);
int qedr_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags);
struct ib_qp *qedr_create_qp(struct ib_pd *, struct ib_qp_init_attr *attrs,
struct ib_udata *);
int attr_mask, struct ib_udata *udata);
int qedr_query_qp(struct ib_qp *, struct ib_qp_attr *qp_attr,
int qp_attr_mask, struct ib_qp_init_attr *);
-int qedr_destroy_qp(struct ib_qp *ibqp);
+int qedr_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata);
-struct ib_srq *qedr_create_srq(struct ib_pd *ibpd,
- struct ib_srq_init_attr *attr,
- struct ib_udata *udata);
+int qedr_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *attr,
+ struct ib_udata *udata);
int qedr_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask attr_mask, struct ib_udata *udata);
int qedr_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr);
-int qedr_destroy_srq(struct ib_srq *ibsrq);
+void qedr_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata);
int qedr_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr,
const struct ib_recv_wr **bad_recv_wr);
-struct ib_ah *qedr_create_ah(struct ib_pd *ibpd, struct rdma_ah_attr *attr,
- u32 flags, struct ib_udata *udata);
-int qedr_destroy_ah(struct ib_ah *ibah, u32 flags);
+int qedr_create_ah(struct ib_ah *ibah, struct rdma_ah_attr *attr, u32 flags,
+ struct ib_udata *udata);
+void qedr_destroy_ah(struct ib_ah *ibah, u32 flags);
-int qedr_dereg_mr(struct ib_mr *);
+int qedr_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata);
struct ib_mr *qedr_get_dma_mr(struct ib_pd *, int acc);
struct ib_mr *qedr_reg_user_mr(struct ib_pd *, u64 start, u64 length,
int sg_nents, unsigned int *sg_offset);
struct ib_mr *qedr_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
- u32 max_num_sg);
+ u32 max_num_sg, struct ib_udata *udata);
int qedr_poll_cq(struct ib_cq *, int num_entries, struct ib_wc *wc);
int qedr_post_send(struct ib_qp *, const struct ib_send_wr *,
const struct ib_send_wr **bad_wr);
#include <linux/kref.h>
#include <linux/sched.h>
#include <linux/kthread.h>
+#include <linux/xarray.h>
#include <rdma/ib_hdrs.h>
#include <rdma/rdma_vt.h>
int rec_cpu_num; /* for cpu affinity; -1 if none */
};
-extern struct list_head qib_dev_list;
-extern spinlock_t qib_devs_lock;
+extern struct xarray qib_dev_table;
extern struct qib_devdata *qib_lookup(int unit);
extern u32 qib_cpulist_count;
extern unsigned long *qib_cpulist;
* QIB_VERBOSE_TRACING define as 1 if you want additional tracing in
* fastpath code
* QIB_TRACE_REGWRITES define as 1 if you want register writes to be
- * traced in faspath code
+ * traced in fastpath code
* _QIB_TRACING define as 0 if you want to remove all tracing in a
* compilation unit
*/
*/
const char ib_qib_version[] = QIB_DRIVER_VERSION "\n";
-DEFINE_SPINLOCK(qib_devs_lock);
-LIST_HEAD(qib_dev_list);
DEFINE_MUTEX(qib_mutex); /* general driver use */
unsigned qib_ibmtu;
{
struct qib_devdata *dd;
struct qib_pportdata *ppd;
- unsigned long flags;
+ unsigned long index, flags;
int pidx, nunits_active = 0;
- spin_lock_irqsave(&qib_devs_lock, flags);
- list_for_each_entry(dd, &qib_dev_list, list) {
+ xa_lock_irqsave(&qib_dev_table, flags);
+ xa_for_each(&qib_dev_table, index, dd) {
if (!(dd->flags & QIB_PRESENT) || !dd->kregbase)
continue;
for (pidx = 0; pidx < dd->num_pports; ++pidx) {
}
}
}
- spin_unlock_irqrestore(&qib_devs_lock, flags);
+ xa_unlock_irqrestore(&qib_dev_table, flags);
return nunits_active;
}
{
int nunits = 0, npresent = 0, nup = 0;
struct qib_devdata *dd;
- unsigned long flags;
+ unsigned long index, flags;
int pidx;
struct qib_pportdata *ppd;
- spin_lock_irqsave(&qib_devs_lock, flags);
-
- list_for_each_entry(dd, &qib_dev_list, list) {
+ xa_lock_irqsave(&qib_dev_table, flags);
+ xa_for_each(&qib_dev_table, index, dd) {
nunits++;
if ((dd->flags & QIB_PRESENT) && dd->kregbase)
npresent++;
nup++;
}
}
-
- spin_unlock_irqrestore(&qib_devs_lock, flags);
+ xa_unlock_irqrestore(&qib_dev_table, flags);
if (npresentp)
*npresentp = npresent;
*/
static int qibfs_fill_super(struct super_block *sb, void *data, int silent)
{
- struct qib_devdata *dd, *tmp;
- unsigned long flags;
+ struct qib_devdata *dd;
+ unsigned long index;
int ret;
static const struct tree_descr files[] = {
goto bail;
}
- spin_lock_irqsave(&qib_devs_lock, flags);
-
- list_for_each_entry_safe(dd, tmp, &qib_dev_list, list) {
- spin_unlock_irqrestore(&qib_devs_lock, flags);
+ xa_for_each(&qib_dev_table, index, dd) {
ret = add_cntr_files(sb, dd);
if (ret)
goto bail;
- spin_lock_irqsave(&qib_devs_lock, flags);
}
- spin_unlock_irqrestore(&qib_devs_lock, flags);
-
bail:
return ret;
}
static int setup_txselect(const char *str, const struct kernel_param *kp)
{
struct qib_devdata *dd;
- unsigned long val;
+ unsigned long index, val;
char *n;
if (strlen(str) >= ARRAY_SIZE(txselect_list)) {
}
strncpy(txselect_list, str, ARRAY_SIZE(txselect_list) - 1);
- list_for_each_entry(dd, &qib_dev_list, list)
+ xa_for_each(&qib_dev_table, index, dd)
if (dd->deviceid == PCI_DEVICE_ID_QLOGIC_IB_7322)
set_no_qsfp_atten(dd, 1);
return 0;
#include <linux/netdevice.h>
#include <linux/vmalloc.h>
#include <linux/delay.h>
-#include <linux/idr.h>
#include <linux/module.h>
#include <linux/printk.h>
#ifdef CONFIG_INFINIBAND_QIB_DCA
static void verify_interrupt(struct timer_list *);
-static struct idr qib_unit_table;
+DEFINE_XARRAY_FLAGS(qib_dev_table, XA_FLAGS_ALLOC | XA_FLAGS_LOCK_IRQ);
u32 qib_cpulist_count;
unsigned long *qib_cpulist;
{
}
-static inline struct qib_devdata *__qib_lookup(int unit)
-{
- return idr_find(&qib_unit_table, unit);
-}
-
struct qib_devdata *qib_lookup(int unit)
{
- struct qib_devdata *dd;
- unsigned long flags;
-
- spin_lock_irqsave(&qib_devs_lock, flags);
- dd = __qib_lookup(unit);
- spin_unlock_irqrestore(&qib_devs_lock, flags);
-
- return dd;
+ return xa_load(&qib_dev_table, unit);
}
/*
{
unsigned long flags;
- spin_lock_irqsave(&qib_devs_lock, flags);
- idr_remove(&qib_unit_table, dd->unit);
- list_del(&dd->list);
- spin_unlock_irqrestore(&qib_devs_lock, flags);
+ xa_lock_irqsave(&qib_dev_table, flags);
+ __xa_erase(&qib_dev_table, dd->unit);
+ xa_unlock_irqrestore(&qib_dev_table, flags);
#ifdef CONFIG_DEBUG_FS
qib_dbg_ibdev_exit(&dd->verbs_dev);
u64 qib_sps_ints(void)
{
- unsigned long flags;
+ unsigned long index, flags;
struct qib_devdata *dd;
u64 sps_ints = 0;
- spin_lock_irqsave(&qib_devs_lock, flags);
- list_for_each_entry(dd, &qib_dev_list, list) {
+ xa_lock_irqsave(&qib_dev_table, flags);
+ xa_for_each(&qib_dev_table, index, dd) {
sps_ints += qib_int_counter(dd);
}
- spin_unlock_irqrestore(&qib_devs_lock, flags);
+ xa_unlock_irqrestore(&qib_dev_table, flags);
return sps_ints;
}
* allocator, because the verbs cleanup process both does cleanup and
* free of the data structure.
* "extra" is for chip-specific data.
- *
- * Use the idr mechanism to get a unit number for this unit.
*/
struct qib_devdata *qib_alloc_devdata(struct pci_dev *pdev, size_t extra)
{
- unsigned long flags;
struct qib_devdata *dd;
int ret, nports;
if (!dd)
return ERR_PTR(-ENOMEM);
- INIT_LIST_HEAD(&dd->list);
-
- idr_preload(GFP_KERNEL);
- spin_lock_irqsave(&qib_devs_lock, flags);
-
- ret = idr_alloc(&qib_unit_table, dd, 0, 0, GFP_NOWAIT);
- if (ret >= 0) {
- dd->unit = ret;
- list_add(&dd->list, &qib_dev_list);
- }
-
- spin_unlock_irqrestore(&qib_devs_lock, flags);
- idr_preload_end();
-
+ ret = xa_alloc_irq(&qib_dev_table, &dd->unit, dd, xa_limit_32b,
+ GFP_KERNEL);
if (ret < 0) {
qib_early_err(&pdev->dev,
"Could not allocate unit ID: error %d\n", -ret);
* These must be called before the driver is registered with
* the PCI subsystem.
*/
- idr_init(&qib_unit_table);
-
#ifdef CONFIG_INFINIBAND_QIB_DCA
dca_register_notify(&dca_notifier);
#endif
#ifdef CONFIG_DEBUG_FS
qib_dbg_exit();
#endif
- idr_destroy(&qib_unit_table);
qib_dev_cleanup();
bail:
return ret;
qib_cpulist_count = 0;
kfree(qib_cpulist);
- idr_destroy(&qib_unit_table);
+ WARN_ON(!xa_empty(&qib_dev_table));
qib_dev_cleanup();
}
qp->s_last = s_last;
/* see post_send() */
barrier();
- rvt_put_swqe(wqe);
+ rvt_put_qp_swqe(qp, wqe);
rvt_qp_swqe_complete(qp,
wqe,
ib_qib_wc_opcode[wqe->wr.opcode],
qib_cmp24(qp->s_sending_psn, qp->s_sending_hpsn) > 0) {
u32 s_last;
- rvt_put_swqe(wqe);
+ rvt_put_qp_swqe(qp, wqe);
s_last = qp->s_last;
if (++s_last >= qp->s_size)
s_last = 0;
if (sdma_rb_node) {
sdma_rb_node->refcount++;
} else {
- int ret;
-
sdma_rb_node = kmalloc(sizeof(
struct qib_user_sdma_rb_node), GFP_KERNEL);
if (!sdma_rb_node)
sdma_rb_node->refcount = 1;
sdma_rb_node->pid = current->pid;
- ret = qib_user_sdma_rb_insert(&qib_user_sdma_rb_root,
- sdma_rb_node);
+ qib_user_sdma_rb_insert(&qib_user_sdma_rb_root, sdma_rb_node);
}
pq->sdma_rb_node = sdma_rb_node;
#include <rdma/ib_pack.h>
#include <rdma/ib_user_verbs.h>
#include <rdma/ib_hdrs.h>
-#include <rdma/rdma_vt.h>
+#include <rdma/rdmavt_qp.h>
#include <rdma/rdmavt_cq.h>
struct qib_ctxtdata;
return 0;
}
-int usnic_ib_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata)
+int usnic_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct usnic_ib_pd *pd = to_upd(ibpd);
void *umem_pd;
return 0;
}
-void usnic_ib_dealloc_pd(struct ib_pd *pd)
+void usnic_ib_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata)
{
usnic_uiom_dealloc_pd((to_upd(pd))->umem_pd);
}
return ERR_PTR(err);
}
-int usnic_ib_destroy_qp(struct ib_qp *qp)
+int usnic_ib_destroy_qp(struct ib_qp *qp, struct ib_udata *udata)
{
struct usnic_ib_qp_grp *qp_grp;
struct usnic_ib_vf *vf;
struct ib_cq *usnic_ib_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
struct ib_cq *cq;
return cq;
}
-int usnic_ib_destroy_cq(struct ib_cq *cq)
+int usnic_ib_destroy_cq(struct ib_cq *cq, struct ib_udata *udata)
{
usnic_dbg("\n");
kfree(cq);
return ERR_PTR(err);
}
-int usnic_ib_dereg_mr(struct ib_mr *ibmr)
+int usnic_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
{
struct usnic_ib_mr *mr = to_umr(ibmr);
usnic_dbg("va 0x%lx length 0x%zx\n", mr->umem->va, mr->umem->length);
- usnic_uiom_reg_release(mr->umem, ibmr->uobject->context);
+ usnic_uiom_reg_release(mr->umem);
kfree(mr);
return 0;
}
return -EINVAL;
}
-/* End of ib callbacks section */
union ib_gid *gid);
int usnic_ib_query_pkey(struct ib_device *ibdev, u8 port, u16 index,
u16 *pkey);
-int usnic_ib_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata);
-void usnic_ib_dealloc_pd(struct ib_pd *pd);
+int usnic_ib_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata);
+void usnic_ib_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata);
struct ib_qp *usnic_ib_create_qp(struct ib_pd *pd,
struct ib_qp_init_attr *init_attr,
struct ib_udata *udata);
-int usnic_ib_destroy_qp(struct ib_qp *qp);
+int usnic_ib_destroy_qp(struct ib_qp *qp, struct ib_udata *udata);
int usnic_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
int attr_mask, struct ib_udata *udata);
struct ib_cq *usnic_ib_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata);
-int usnic_ib_destroy_cq(struct ib_cq *cq);
+int usnic_ib_destroy_cq(struct ib_cq *cq, struct ib_udata *udata);
struct ib_mr *usnic_ib_reg_mr(struct ib_pd *pd, u64 start, u64 length,
u64 virt_addr, int access_flags,
struct ib_udata *udata);
-int usnic_ib_dereg_mr(struct ib_mr *ibmr);
+int usnic_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata);
int usnic_ib_alloc_ucontext(struct ib_ucontext *uctx, struct ib_udata *udata);
void usnic_ib_dealloc_ucontext(struct ib_ucontext *ibcontext);
int usnic_ib_mmap(struct ib_ucontext *context,
return PAGE_ALIGN(uiomr->length + uiomr->offset) >> PAGE_SHIFT;
}
-void usnic_uiom_reg_release(struct usnic_uiom_reg *uiomr,
- struct ib_ucontext *context)
+void usnic_uiom_reg_release(struct usnic_uiom_reg *uiomr)
{
__usnic_uiom_reg_release(uiomr->pd, uiomr, 1);
struct usnic_uiom_reg *usnic_uiom_reg_get(struct usnic_uiom_pd *pd,
unsigned long addr, size_t size,
int access, int dmasync);
-void usnic_uiom_reg_release(struct usnic_uiom_reg *uiomr,
- struct ib_ucontext *ucontext);
+void usnic_uiom_reg_release(struct usnic_uiom_reg *uiomr);
int usnic_uiom_init(char *drv_name);
#endif /* USNIC_UIOM_H_ */
#include <rdma/ib_addr.h>
#include <rdma/ib_smi.h>
#include <rdma/ib_user_verbs.h>
+#include <rdma/uverbs_ioctl.h>
#include "pvrdma.h"
* pvrdma_create_cq - create completion queue
* @ibdev: the device
* @attr: completion queue attributes
- * @context: user context
* @udata: user data
*
* @return: ib_cq completion queue pointer on success,
*/
struct ib_cq *pvrdma_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
int entries = attr->cqe;
struct pvrdma_cmd_create_cq_resp *resp = &rsp.create_cq_resp;
struct pvrdma_create_cq_resp cq_resp = {0};
struct pvrdma_create_cq ucmd;
+ struct pvrdma_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct pvrdma_ucontext, ibucontext);
BUILD_BUG_ON(sizeof(struct pvrdma_cqe) != 64);
}
cq->ibcq.cqe = entries;
- cq->is_kernel = !context;
+ cq->is_kernel = !udata;
if (!cq->is_kernel) {
if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd))) {
memset(cmd, 0, sizeof(*cmd));
cmd->hdr.cmd = PVRDMA_CMD_CREATE_CQ;
cmd->nchunks = npages;
- cmd->ctx_handle = (context) ?
- (u64)to_vucontext(context)->ctx_handle : 0;
+ cmd->ctx_handle = context ? context->ctx_handle : 0;
cmd->cqe = entries;
cmd->pdir_dma = cq->pdir.dir_dma;
ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_CREATE_CQ_RESP);
spin_unlock_irqrestore(&dev->cq_tbl_lock, flags);
if (!cq->is_kernel) {
- cq->uar = &(to_vucontext(context)->uar);
+ cq->uar = &context->uar;
/* Copy udata back. */
if (ib_copy_to_udata(udata, &cq_resp, sizeof(cq_resp))) {
dev_warn(&dev->pdev->dev,
"failed to copy back udata\n");
- pvrdma_destroy_cq(&cq->ibcq);
+ pvrdma_destroy_cq(&cq->ibcq, udata);
return ERR_PTR(-EINVAL);
}
}
/**
* pvrdma_destroy_cq - destroy completion queue
* @cq: the completion queue to destroy.
+ * @udata: user data or null for kernel object
*
* @return: 0 for success.
*/
-int pvrdma_destroy_cq(struct ib_cq *cq)
+int pvrdma_destroy_cq(struct ib_cq *cq, struct ib_udata *udata)
{
struct pvrdma_cq *vcq = to_vcq(cq);
union pvrdma_cmd_req req;
return 0;
}
-static struct net_device *pvrdma_get_netdev(struct ib_device *ibdev,
- u8 port_num)
-{
- struct net_device *netdev;
- struct pvrdma_dev *dev = to_vdev(ibdev);
-
- if (port_num != 1)
- return NULL;
-
- rcu_read_lock();
- netdev = dev->netdev;
- if (netdev)
- dev_hold(netdev);
- rcu_read_unlock();
-
- return netdev;
-}
-
static const struct ib_device_ops pvrdma_dev_ops = {
.add_gid = pvrdma_add_gid,
.alloc_mr = pvrdma_alloc_mr,
.get_dev_fw_str = pvrdma_get_fw_ver_str,
.get_dma_mr = pvrdma_get_dma_mr,
.get_link_layer = pvrdma_port_link_layer,
- .get_netdev = pvrdma_get_netdev,
.get_port_immutable = pvrdma_port_immutable,
.map_mr_sg = pvrdma_map_mr_sg,
.mmap = pvrdma_mmap,
.query_qp = pvrdma_query_qp,
.reg_user_mr = pvrdma_reg_user_mr,
.req_notify_cq = pvrdma_req_notify_cq,
+
+ INIT_RDMA_OBJ_SIZE(ib_ah, pvrdma_ah, ibah),
INIT_RDMA_OBJ_SIZE(ib_pd, pvrdma_pd, ibpd),
INIT_RDMA_OBJ_SIZE(ib_ucontext, pvrdma_ucontext, ibucontext),
};
.destroy_srq = pvrdma_destroy_srq,
.modify_srq = pvrdma_modify_srq,
.query_srq = pvrdma_query_srq,
+
+ INIT_RDMA_OBJ_SIZE(ib_srq, pvrdma_srq, ibsrq),
};
static int pvrdma_register_device(struct pvrdma_dev *dev)
goto err_qp_free;
}
dev->ib_dev.driver_id = RDMA_DRIVER_VMW_PVRDMA;
+ ret = ib_device_set_netdev(&dev->ib_dev, dev->netdev, 1);
+ if (ret)
+ return ret;
spin_lock_init(&dev->srq_tbl_lock);
rdma_set_device_sysfs_group(&dev->ib_dev, &pvrdma_attr_group);
pvrdma_dispatch_event(dev, 1, IB_EVENT_PORT_ACTIVE);
break;
case NETDEV_UNREGISTER:
+ ib_device_set_netdev(&dev->ib_dev, NULL, 1);
dev_put(dev->netdev);
dev->netdev = NULL;
break;
if ((dev->netdev == NULL) &&
(pci_get_drvdata(pdev_net) == ndev)) {
/* this is our netdev */
+ ib_device_set_netdev(&dev->ib_dev, ndev, 1);
dev->netdev = ndev;
dev_hold(ndev);
}
union pvrdma_cmd_resp rsp;
struct pvrdma_cmd_create_mr *cmd = &req.create_mr;
struct pvrdma_cmd_create_mr_resp *resp = &rsp.create_mr_resp;
- int ret;
+ int ret, npages;
if (length == 0 || length > dev->dsr->caps.max_mr_size) {
dev_warn(&dev->pdev->dev, "invalid mem region length\n");
return ERR_CAST(umem);
}
- if (umem->npages < 0 || umem->npages > PVRDMA_PAGE_DIR_MAX_PAGES) {
+ npages = ib_umem_num_pages(umem);
+ if (npages < 0 || npages > PVRDMA_PAGE_DIR_MAX_PAGES) {
dev_warn(&dev->pdev->dev, "overflow %d pages in mem region\n",
- umem->npages);
+ npages);
ret = -EINVAL;
goto err_umem;
}
mr->mmr.size = length;
mr->umem = umem;
- ret = pvrdma_page_dir_init(dev, &mr->pdir, umem->npages, false);
+ ret = pvrdma_page_dir_init(dev, &mr->pdir, npages, false);
if (ret) {
dev_warn(&dev->pdev->dev,
"could not allocate page directory\n");
cmd->length = length;
cmd->pd_handle = to_vpd(pd)->pd_handle;
cmd->access_flags = access_flags;
- cmd->nchunks = umem->npages;
+ cmd->nchunks = npages;
cmd->pdir_dma = mr->pdir.dir_dma;
ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_CREATE_MR_RESP);
* @return: ib_mr pointer on success, otherwise returns an errno.
*/
struct ib_mr *pvrdma_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
- u32 max_num_sg)
+ u32 max_num_sg, struct ib_udata *udata)
{
struct pvrdma_dev *dev = to_vdev(pd->device);
struct pvrdma_user_mr *mr;
*
* @return: 0 on success.
*/
-int pvrdma_dereg_mr(struct ib_mr *ibmr)
+int pvrdma_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
{
struct pvrdma_user_mr *mr = to_vmr(ibmr);
struct pvrdma_dev *dev = to_vdev(ibmr->device);
/**
* pvrdma_destroy_qp - destroy a queue pair
* @qp: the queue pair to destroy
+ * @udata: user data or null for kernel object
*
* @return: 0 on success.
*/
-int pvrdma_destroy_qp(struct ib_qp *qp)
+int pvrdma_destroy_qp(struct ib_qp *qp, struct ib_udata *udata)
{
struct pvrdma_qp *vqp = to_vqp(qp);
union pvrdma_cmd_req req;
* @init_attr: shared receive queue attributes
* @udata: user data
*
- * @return: the ib_srq pointer on success, otherwise returns an errno.
+ * @return: 0 on success, otherwise returns an errno.
*/
-struct ib_srq *pvrdma_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *init_attr,
- struct ib_udata *udata)
+int pvrdma_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *init_attr,
+ struct ib_udata *udata)
{
- struct pvrdma_srq *srq = NULL;
- struct pvrdma_dev *dev = to_vdev(pd->device);
+ struct pvrdma_srq *srq = to_vsrq(ibsrq);
+ struct pvrdma_dev *dev = to_vdev(ibsrq->device);
union pvrdma_cmd_req req;
union pvrdma_cmd_resp rsp;
struct pvrdma_cmd_create_srq *cmd = &req.create_srq;
struct pvrdma_cmd_create_srq_resp *resp = &rsp.create_srq_resp;
- struct pvrdma_create_srq_resp srq_resp = {0};
+ struct pvrdma_create_srq_resp srq_resp = {};
struct pvrdma_create_srq ucmd;
unsigned long flags;
int ret;
/* No support for kernel clients. */
dev_warn(&dev->pdev->dev,
"no shared receive queue support for kernel client\n");
- return ERR_PTR(-EOPNOTSUPP);
+ return -EOPNOTSUPP;
}
if (init_attr->srq_type != IB_SRQT_BASIC) {
dev_warn(&dev->pdev->dev,
"shared receive queue type %d not supported\n",
init_attr->srq_type);
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
}
if (init_attr->attr.max_wr > dev->dsr->caps.max_srq_wr ||
init_attr->attr.max_sge > dev->dsr->caps.max_srq_sge) {
dev_warn(&dev->pdev->dev,
"shared receive queue size invalid\n");
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
}
if (!atomic_add_unless(&dev->num_srqs, 1, dev->dsr->caps.max_srq))
- return ERR_PTR(-ENOMEM);
-
- srq = kmalloc(sizeof(*srq), GFP_KERNEL);
- if (!srq) {
- ret = -ENOMEM;
- goto err_srq;
- }
+ return -ENOMEM;
spin_lock_init(&srq->lock);
refcount_set(&srq->refcnt, 1);
cmd->hdr.cmd = PVRDMA_CMD_CREATE_SRQ;
cmd->srq_type = init_attr->srq_type;
cmd->nchunks = srq->npages;
- cmd->pd_handle = to_vpd(pd)->pd_handle;
+ cmd->pd_handle = to_vpd(ibsrq->pd)->pd_handle;
cmd->attrs.max_wr = init_attr->attr.max_wr;
cmd->attrs.max_sge = init_attr->attr.max_sge;
cmd->attrs.srq_limit = init_attr->attr.srq_limit;
/* Copy udata back. */
if (ib_copy_to_udata(udata, &srq_resp, sizeof(srq_resp))) {
dev_warn(&dev->pdev->dev, "failed to copy back udata\n");
- pvrdma_destroy_srq(&srq->ibsrq);
- return ERR_PTR(-EINVAL);
+ pvrdma_destroy_srq(&srq->ibsrq, udata);
+ return -EINVAL;
}
- return &srq->ibsrq;
+ return 0;
err_page_dir:
pvrdma_page_dir_cleanup(dev, &srq->pdir);
err_umem:
ib_umem_release(srq->umem);
err_srq:
- kfree(srq);
atomic_dec(&dev->num_srqs);
- return ERR_PTR(ret);
+ return ret;
}
static void pvrdma_free_srq(struct pvrdma_dev *dev, struct pvrdma_srq *srq)
/**
* pvrdma_destroy_srq - destroy shared receive queue
* @srq: the shared receive queue to destroy
+ * @udata: user data or null for kernel object
*
* @return: 0 for success.
*/
-int pvrdma_destroy_srq(struct ib_srq *srq)
+void pvrdma_destroy_srq(struct ib_srq *srq, struct ib_udata *udata)
{
struct pvrdma_srq *vsrq = to_vsrq(srq);
union pvrdma_cmd_req req;
ret);
pvrdma_free_srq(dev, vsrq);
-
- return 0;
}
/**
#include <rdma/ib_smi.h>
#include <rdma/ib_user_verbs.h>
#include <rdma/vmw_pvrdma-abi.h>
+#include <rdma/uverbs_ioctl.h>
#include "pvrdma.h"
if (uhw->inlen || uhw->outlen)
return -EINVAL;
- memset(props, 0, sizeof(*props));
-
props->fw_ver = dev->dsr->caps.fw_ver;
props->sys_image_guid = dev->dsr->caps.sys_image_guid;
props->max_mr_size = dev->dsr->caps.max_mr_size;
/**
* pvrdma_alloc_pd - allocate protection domain
* @ibpd: PD pointer
- * @context: user context
* @udata: user data
*
* @return: the ib_pd protection domain pointer on success, otherwise errno.
*/
-int pvrdma_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata)
+int pvrdma_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct ib_device *ibdev = ibpd->device;
struct pvrdma_pd *pd = to_vpd(ibpd);
struct pvrdma_cmd_create_pd_resp *resp = &rsp.create_pd_resp;
struct pvrdma_alloc_pd_resp pd_resp = {0};
int ret;
+ struct pvrdma_ucontext *context = rdma_udata_to_drv_context(
+ udata, struct pvrdma_ucontext, ibucontext);
/* Check allowed max pds */
if (!atomic_add_unless(&dev->num_pds, 1, dev->dsr->caps.max_pd))
return -ENOMEM;
cmd->hdr.cmd = PVRDMA_CMD_CREATE_PD;
- cmd->ctx_handle = (context) ? to_vucontext(context)->ctx_handle : 0;
+ cmd->ctx_handle = context ? context->ctx_handle : 0;
ret = pvrdma_cmd_post(dev, &req, &rsp, PVRDMA_CMD_CREATE_PD_RESP);
if (ret < 0) {
dev_warn(&dev->pdev->dev,
goto err;
}
- pd->privileged = !context;
+ pd->privileged = !udata;
pd->pd_handle = resp->pd_handle;
pd->pdn = resp->pd_handle;
pd_resp.pdn = resp->pd_handle;
- if (context) {
+ if (udata) {
if (ib_copy_to_udata(udata, &pd_resp, sizeof(pd_resp))) {
dev_warn(&dev->pdev->dev,
"failed to copy back protection domain\n");
- pvrdma_dealloc_pd(&pd->ibpd);
+ pvrdma_dealloc_pd(&pd->ibpd, udata);
return -EFAULT;
}
}
/**
* pvrdma_dealloc_pd - deallocate protection domain
* @pd: the protection domain to be released
+ * @udata: user data or null for kernel object
*
* @return: 0 on success, otherwise errno.
*/
-void pvrdma_dealloc_pd(struct ib_pd *pd)
+void pvrdma_dealloc_pd(struct ib_pd *pd, struct ib_udata *udata)
{
struct pvrdma_dev *dev = to_vdev(pd->device);
union pvrdma_cmd_req req = {};
* @udata: user data blob
* @flags: create address handle flags (see enum rdma_create_ah_flags)
*
- * @return: the ib_ah pointer on success, otherwise errno.
+ * @return: 0 on success, otherwise errno.
*/
-struct ib_ah *pvrdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
- u32 flags, struct ib_udata *udata)
+int pvrdma_create_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr,
+ u32 flags, struct ib_udata *udata)
{
- struct pvrdma_dev *dev = to_vdev(pd->device);
- struct pvrdma_ah *ah;
+ struct pvrdma_dev *dev = to_vdev(ibah->device);
+ struct pvrdma_ah *ah = to_vah(ibah);
const struct ib_global_route *grh;
u8 port_num = rdma_ah_get_port_num(ah_attr);
if (!(rdma_ah_get_ah_flags(ah_attr) & IB_AH_GRH))
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
grh = rdma_ah_read_grh(ah_attr);
if ((ah_attr->type != RDMA_AH_ATTR_TYPE_ROCE) ||
rdma_is_multicast_addr((struct in6_addr *)grh->dgid.raw))
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
if (!atomic_add_unless(&dev->num_ahs, 1, dev->dsr->caps.max_ah))
- return ERR_PTR(-ENOMEM);
-
- ah = kzalloc(sizeof(*ah), GFP_ATOMIC);
- if (!ah) {
- atomic_dec(&dev->num_ahs);
- return ERR_PTR(-ENOMEM);
- }
+ return -ENOMEM;
- ah->av.port_pd = to_vpd(pd)->pd_handle | (port_num << 24);
+ ah->av.port_pd = to_vpd(ibah->pd)->pd_handle | (port_num << 24);
ah->av.src_path_bits = rdma_ah_get_path_bits(ah_attr);
ah->av.src_path_bits |= 0x80;
ah->av.gid_index = grh->sgid_index;
memcpy(ah->av.dgid, grh->dgid.raw, 16);
memcpy(ah->av.dmac, ah_attr->roce.dmac, ETH_ALEN);
- ah->ibah.device = pd->device;
- ah->ibah.pd = pd;
- ah->ibah.uobject = NULL;
-
- return &ah->ibah;
+ return 0;
}
/**
* @ah: the address handle to destroyed
* @flags: destroy address handle flags (see enum rdma_destroy_ah_flags)
*
- * @return: 0 on success.
*/
-int pvrdma_destroy_ah(struct ib_ah *ah, u32 flags)
+void pvrdma_destroy_ah(struct ib_ah *ah, u32 flags)
{
struct pvrdma_dev *dev = to_vdev(ah->device);
- kfree(to_vah(ah));
atomic_dec(&dev->num_ahs);
-
- return 0;
}
int pvrdma_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
int pvrdma_alloc_ucontext(struct ib_ucontext *uctx, struct ib_udata *udata);
void pvrdma_dealloc_ucontext(struct ib_ucontext *context);
-int pvrdma_alloc_pd(struct ib_pd *pd, struct ib_ucontext *context,
- struct ib_udata *udata);
-void pvrdma_dealloc_pd(struct ib_pd *ibpd);
+int pvrdma_alloc_pd(struct ib_pd *pd, struct ib_udata *udata);
+void pvrdma_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata);
struct ib_mr *pvrdma_get_dma_mr(struct ib_pd *pd, int acc);
struct ib_mr *pvrdma_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
u64 virt_addr, int access_flags,
struct ib_udata *udata);
-int pvrdma_dereg_mr(struct ib_mr *mr);
+int pvrdma_dereg_mr(struct ib_mr *mr, struct ib_udata *udata);
struct ib_mr *pvrdma_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
- u32 max_num_sg);
+ u32 max_num_sg, struct ib_udata *udata);
int pvrdma_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg,
int sg_nents, unsigned int *sg_offset);
struct ib_cq *pvrdma_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata);
-int pvrdma_destroy_cq(struct ib_cq *cq);
+int pvrdma_destroy_cq(struct ib_cq *cq, struct ib_udata *udata);
int pvrdma_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc);
int pvrdma_req_notify_cq(struct ib_cq *cq, enum ib_cq_notify_flags flags);
-struct ib_ah *pvrdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr,
- u32 flags, struct ib_udata *udata);
-int pvrdma_destroy_ah(struct ib_ah *ah, u32 flags);
+int pvrdma_create_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr, u32 flags,
+ struct ib_udata *udata);
+void pvrdma_destroy_ah(struct ib_ah *ah, u32 flags);
-struct ib_srq *pvrdma_create_srq(struct ib_pd *pd,
- struct ib_srq_init_attr *init_attr,
- struct ib_udata *udata);
+int pvrdma_create_srq(struct ib_srq *srq, struct ib_srq_init_attr *init_attr,
+ struct ib_udata *udata);
int pvrdma_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask attr_mask, struct ib_udata *udata);
int pvrdma_query_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr);
-int pvrdma_destroy_srq(struct ib_srq *srq);
+void pvrdma_destroy_srq(struct ib_srq *srq, struct ib_udata *udata);
struct ib_qp *pvrdma_create_qp(struct ib_pd *pd,
struct ib_qp_init_attr *init_attr,
int attr_mask, struct ib_udata *udata);
int pvrdma_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr);
-int pvrdma_destroy_qp(struct ib_qp *qp);
+int pvrdma_destroy_qp(struct ib_qp *qp, struct ib_udata *udata);
int pvrdma_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
const struct ib_send_wr **bad_wr);
int pvrdma_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
/**
* rvt_create_ah - create an address handle
- * @pd: the protection domain
+ * @ibah: the IB address handle
* @ah_attr: the attributes of the AH
* @create_flags: create address handle flags (see enum rdma_create_ah_flags)
* @udata: pointer to user's input output buffer information.
*
* This may be called from interrupt context.
*
- * Return: newly allocated ah
+ * Return: 0 on success
*/
-struct ib_ah *rvt_create_ah(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
- u32 create_flags,
- struct ib_udata *udata)
+int rvt_create_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr,
+ u32 create_flags, struct ib_udata *udata)
{
- struct rvt_ah *ah;
- struct rvt_dev_info *dev = ib_to_rvt(pd->device);
+ struct rvt_ah *ah = ibah_to_rvtah(ibah);
+ struct rvt_dev_info *dev = ib_to_rvt(ibah->device);
unsigned long flags;
- if (rvt_check_ah(pd->device, ah_attr))
- return ERR_PTR(-EINVAL);
-
- ah = kmalloc(sizeof(*ah), GFP_ATOMIC);
- if (!ah)
- return ERR_PTR(-ENOMEM);
+ if (rvt_check_ah(ibah->device, ah_attr))
+ return -EINVAL;
spin_lock_irqsave(&dev->n_ahs_lock, flags);
if (dev->n_ahs_allocated == dev->dparms.props.max_ah) {
spin_unlock_irqrestore(&dev->n_ahs_lock, flags);
- kfree(ah);
- return ERR_PTR(-ENOMEM);
+ return -ENOMEM;
}
dev->n_ahs_allocated++;
atomic_set(&ah->refcount, 0);
if (dev->driver_f.notify_new_ah)
- dev->driver_f.notify_new_ah(pd->device, ah_attr, ah);
+ dev->driver_f.notify_new_ah(ibah->device, ah_attr, ah);
- return &ah->ibah;
+ return 0;
}
/**
* rvt_destory_ah - Destory an address handle
* @ibah: address handle
* @destroy_flags: destroy address handle flags (see enum rdma_destroy_ah_flags)
+ * @udata: user data or NULL for kernel object
*
* Return: 0 on success
*/
-int rvt_destroy_ah(struct ib_ah *ibah, u32 destroy_flags)
+void rvt_destroy_ah(struct ib_ah *ibah, u32 destroy_flags)
{
struct rvt_dev_info *dev = ib_to_rvt(ibah->device);
struct rvt_ah *ah = ibah_to_rvtah(ibah);
unsigned long flags;
- if (atomic_read(&ah->refcount) != 0)
- return -EBUSY;
+ WARN_ON_ONCE(atomic_read(&ah->refcount));
spin_lock_irqsave(&dev->n_ahs_lock, flags);
dev->n_ahs_allocated--;
spin_unlock_irqrestore(&dev->n_ahs_lock, flags);
rdma_destroy_ah_attr(&ah->attr);
- kfree(ah);
-
- return 0;
}
/**
#include <rdma/rdma_vt.h>
-struct ib_ah *rvt_create_ah(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr,
- u32 create_flags,
- struct ib_udata *udata);
-int rvt_destroy_ah(struct ib_ah *ibah, u32 destroy_flags);
+int rvt_create_ah(struct ib_ah *ah, struct rdma_ah_attr *ah_attr,
+ u32 create_flags, struct ib_udata *udata);
+void rvt_destroy_ah(struct ib_ah *ibah, u32 destroy_flags);
int rvt_modify_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr);
int rvt_query_ah(struct ib_ah *ibah, struct rdma_ah_attr *ah_attr);
* rvt_create_cq - create a completion queue
* @ibdev: the device this completion queue is attached to
* @attr: creation attributes
- * @context: unused by the QLogic_IB driver
* @udata: user data for libibverbs.so
*
* Called by ib_create_cq() in the generic verbs code.
*/
struct ib_cq *rvt_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
if (udata && udata->outlen >= sizeof(__u64)) {
int err;
- cq->ip = rvt_create_mmap_info(rdi, sz, context, wc);
+ cq->ip = rvt_create_mmap_info(rdi, sz, udata, wc);
if (!cq->ip) {
ret = ERR_PTR(-ENOMEM);
goto bail_wc;
/**
* rvt_destroy_cq - destroy a completion queue
* @ibcq: the completion queue to destroy.
+ * @udata: user data or NULL for kernel object
*
* Called by ib_destroy_cq() in the generic verbs code.
*
* Return: always 0
*/
-int rvt_destroy_cq(struct ib_cq *ibcq)
+int rvt_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
{
struct rvt_cq *cq = ibcq_to_rvtcq(ibcq);
struct rvt_dev_info *rdi = cq->rdi;
struct ib_cq *rvt_create_cq(struct ib_device *ibdev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata);
-int rvt_destroy_cq(struct ib_cq *ibcq);
+int rvt_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata);
int rvt_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags);
int rvt_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata);
int rvt_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *entry);
#include <linux/vmalloc.h>
#include <linux/mm.h>
#include <asm/pgtable.h>
+#include <rdma/uverbs_ioctl.h>
#include "mmap.h"
/**
* rvt_create_mmap_info - allocate information for hfi1_mmap
* @rdi: rvt dev struct
* @size: size in bytes to map
- * @context: user context
+ * @udata: user data (must be valid!)
* @obj: opaque pointer to a cq, wq etc
*
* Return: rvt_mmap struct on success
*/
-struct rvt_mmap_info *rvt_create_mmap_info(struct rvt_dev_info *rdi,
- u32 size,
- struct ib_ucontext *context,
- void *obj)
+struct rvt_mmap_info *rvt_create_mmap_info(struct rvt_dev_info *rdi, u32 size,
+ struct ib_udata *udata, void *obj)
{
struct rvt_mmap_info *ip;
+ if (!udata)
+ return ERR_PTR(-EINVAL);
+
ip = kmalloc_node(sizeof(*ip), GFP_KERNEL, rdi->dparms.node);
if (!ip)
return ip;
INIT_LIST_HEAD(&ip->pending_mmaps);
ip->size = size;
- ip->context = context;
+ ip->context =
+ container_of(udata, struct uverbs_attr_bundle, driver_udata)
+ ->context;
ip->obj = obj;
kref_init(&ip->ref);
void rvt_mmap_init(struct rvt_dev_info *rdi);
void rvt_release_mmap_info(struct kref *ref);
int rvt_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
-struct rvt_mmap_info *rvt_create_mmap_info(struct rvt_dev_info *rdi,
- u32 size,
- struct ib_ucontext *context,
- void *obj);
+struct rvt_mmap_info *rvt_create_mmap_info(struct rvt_dev_info *rdi, u32 size,
+ struct ib_udata *udata, void *obj);
void rvt_update_mmap_info(struct rvt_dev_info *rdi, struct rvt_mmap_info *ip,
u32 size, void *obj);
if (IS_ERR(umem))
return (void *)umem;
- n = umem->nmap;
+ n = ib_umem_num_pages(umem);
mr = __rvt_alloc_mr(n, pd);
if (IS_ERR(mr)) {
*
* Returns 0 on success.
*/
-int rvt_dereg_mr(struct ib_mr *ibmr)
+int rvt_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
{
struct rvt_mr *mr = to_imr(ibmr);
int ret;
*
* Return: the memory region on success, otherwise return an errno.
*/
-struct ib_mr *rvt_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg)
+struct ib_mr *rvt_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
struct rvt_mr *mr;
struct ib_mr *rvt_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
u64 virt_addr, int mr_access_flags,
struct ib_udata *udata);
-int rvt_dereg_mr(struct ib_mr *ibmr);
-struct ib_mr *rvt_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg);
+int rvt_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata);
+struct ib_mr *rvt_alloc_mr(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata);
int rvt_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg,
int sg_nents, unsigned int *sg_offset);
struct ib_fmr *rvt_alloc_fmr(struct ib_pd *pd, int mr_access_flags,
/**
* rvt_alloc_pd - allocate a protection domain
* @ibpd: PD
- * @context: optional user context
* @udata: optional user data
*
* Allocate and keep track of a PD.
*
* Return: 0 on success
*/
-int rvt_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata)
+int rvt_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct ib_device *ibdev = ibpd->device;
struct rvt_dev_info *dev = ib_to_rvt(ibdev);
/**
* rvt_dealloc_pd - Free PD
* @ibpd: Free up PD
+ * @udata: Valid user data or NULL for kernel object
*
* Return: always 0
*/
-void rvt_dealloc_pd(struct ib_pd *ibpd)
+void rvt_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct rvt_dev_info *dev = ib_to_rvt(ibpd->device);
#include <rdma/rdma_vt.h>
-int rvt_alloc_pd(struct ib_pd *pd, struct ib_ucontext *context,
- struct ib_udata *udata);
-void rvt_dealloc_pd(struct ib_pd *ibpd);
+int rvt_alloc_pd(struct ib_pd *pd, struct ib_udata *udata);
+void rvt_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata);
#endif /* DEF_RDMAVTPD_H */
while (qp->s_last != qp->s_head) {
struct rvt_swqe *wqe = rvt_get_swqe_ptr(qp, qp->s_last);
- rvt_put_swqe(wqe);
-
- if (qp->ibqp.qp_type == IB_QPT_UD ||
- qp->ibqp.qp_type == IB_QPT_SMI ||
- qp->ibqp.qp_type == IB_QPT_GSI)
- atomic_dec(&ibah_to_rvtah(
- wqe->ud_wr.ah)->refcount);
+ rvt_put_qp_swqe(qp, wqe);
if (++qp->s_last >= qp->s_size)
qp->s_last = 0;
smp_wmb(); /* see qp_set_savail */
size_t sg_list_sz;
struct ib_qp *ret = ERR_PTR(-ENOMEM);
struct rvt_dev_info *rdi = ib_to_rvt(ibpd->device);
- struct rvt_ucontext *ucontext = rdma_udata_to_drv_context(
- udata, struct rvt_ucontext, ibucontext);
void *priv = NULL;
size_t sqsize;
} else {
u32 s = sizeof(struct rvt_rwq) + qp->r_rq.size * sz;
- qp->ip = rvt_create_mmap_info(rdi, s,
- &ucontext->ibucontext,
+ qp->ip = rvt_create_mmap_info(rdi, s, udata,
qp->r_rq.wq);
if (!qp->ip) {
ret = ERR_PTR(-ENOMEM);
*
* Return: 0 on success.
*/
-int rvt_destroy_qp(struct ib_qp *ibqp)
+int rvt_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
{
struct rvt_qp *qp = ibqp_to_rvtqp(ibqp);
struct rvt_dev_info *rdi = ib_to_rvt(ibqp->device);
* opportunity to adjust PSN values based on internal checks.
*/
log_pmtu = qp->log_pmtu;
- if (qp->ibqp.qp_type != IB_QPT_UC &&
- qp->ibqp.qp_type != IB_QPT_RC) {
+ if (qp->allowed_ops == IB_OPCODE_UD) {
struct rvt_ah *ah = ibah_to_rvtah(wqe->ud_wr.ah);
log_pmtu = ah->log_pmtu;
return 0;
bail_inval_free_ref:
- if (qp->ibqp.qp_type != IB_QPT_UC &&
- qp->ibqp.qp_type != IB_QPT_RC)
+ if (qp->allowed_ops == IB_OPCODE_UD)
atomic_dec(&ibah_to_rvtah(ud_wr(wr)->ah)->refcount);
bail_inval_free:
/* release mr holds */
qp->s_last = last;
/* See post_send() */
barrier();
- rvt_put_swqe(wqe);
- if (qp->ibqp.qp_type == IB_QPT_UD ||
- qp->ibqp.qp_type == IB_QPT_SMI ||
- qp->ibqp.qp_type == IB_QPT_GSI)
- atomic_dec(&ibah_to_rvtah(wqe->ud_wr.ah)->refcount);
+ rvt_put_qp_swqe(qp, wqe);
rvt_qp_swqe_complete(qp,
wqe,
*
*/
-#include <rdma/rdma_vt.h>
+#include <rdma/rdmavt_qp.h>
int rvt_driver_qp_init(struct rvt_dev_info *rdi);
void rvt_qp_exit(struct rvt_dev_info *rdi);
struct ib_udata *udata);
int rvt_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
int attr_mask, struct ib_udata *udata);
-int rvt_destroy_qp(struct ib_qp *ibqp);
+int rvt_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata);
int rvt_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
int attr_mask, struct ib_qp_init_attr *init_attr);
int rvt_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr,
*
*/
-#include <rdma/rdma_vt.h>
+#include <rdma/rdmavt_qp.h>
#include <rdma/ib_hdrs.h>
/*
* @srq_init_attr: the attributes of the SRQ
* @udata: data from libibverbs when creating a user SRQ
*
- * Return: Allocated srq object
+ * Return: 0 on success
*/
-struct ib_srq *rvt_create_srq(struct ib_pd *ibpd,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata)
+int rvt_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *srq_init_attr,
+ struct ib_udata *udata)
{
- struct rvt_dev_info *dev = ib_to_rvt(ibpd->device);
- struct rvt_ucontext *ucontext = rdma_udata_to_drv_context(
- udata, struct rvt_ucontext, ibucontext);
- struct rvt_srq *srq;
+ struct rvt_dev_info *dev = ib_to_rvt(ibsrq->device);
+ struct rvt_srq *srq = ibsrq_to_rvtsrq(ibsrq);
u32 sz;
- struct ib_srq *ret;
+ int ret;
if (srq_init_attr->srq_type != IB_SRQT_BASIC)
- return ERR_PTR(-EOPNOTSUPP);
+ return -EOPNOTSUPP;
if (srq_init_attr->attr.max_sge == 0 ||
srq_init_attr->attr.max_sge > dev->dparms.props.max_srq_sge ||
srq_init_attr->attr.max_wr == 0 ||
srq_init_attr->attr.max_wr > dev->dparms.props.max_srq_wr)
- return ERR_PTR(-EINVAL);
-
- srq = kzalloc_node(sizeof(*srq), GFP_KERNEL, dev->dparms.node);
- if (!srq)
- return ERR_PTR(-ENOMEM);
+ return -EINVAL;
/*
* Need to use vmalloc() if we want to support large #s of entries.
vzalloc_node(sizeof(struct rvt_rwq) + srq->rq.size * sz,
dev->dparms.node);
if (!srq->rq.wq) {
- ret = ERR_PTR(-ENOMEM);
+ ret = -ENOMEM;
goto bail_srq;
}
* See rvt_mmap() for details.
*/
if (udata && udata->outlen >= sizeof(__u64)) {
- int err;
u32 s = sizeof(struct rvt_rwq) + srq->rq.size * sz;
- srq->ip =
- rvt_create_mmap_info(dev, s, &ucontext->ibucontext,
- srq->rq.wq);
+ srq->ip = rvt_create_mmap_info(dev, s, udata, srq->rq.wq);
if (!srq->ip) {
- ret = ERR_PTR(-ENOMEM);
+ ret = -ENOMEM;
goto bail_wq;
}
- err = ib_copy_to_udata(udata, &srq->ip->offset,
+ ret = ib_copy_to_udata(udata, &srq->ip->offset,
sizeof(srq->ip->offset));
- if (err) {
- ret = ERR_PTR(err);
+ if (ret)
goto bail_ip;
- }
}
/*
spin_lock(&dev->n_srqs_lock);
if (dev->n_srqs_allocated == dev->dparms.props.max_srq) {
spin_unlock(&dev->n_srqs_lock);
- ret = ERR_PTR(-ENOMEM);
+ ret = -ENOMEM;
goto bail_ip;
}
spin_unlock_irq(&dev->pending_lock);
}
- return &srq->ibsrq;
+ return 0;
bail_ip:
kfree(srq->ip);
bail_wq:
vfree(srq->rq.wq);
bail_srq:
- kfree(srq);
return ret;
}
* rvt_destroy_srq - destory an srq
* @ibsrq: srq object to destroy
*
- * Return always 0
*/
-int rvt_destroy_srq(struct ib_srq *ibsrq)
+void rvt_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)
{
struct rvt_srq *srq = ibsrq_to_rvtsrq(ibsrq);
struct rvt_dev_info *dev = ib_to_rvt(ibsrq->device);
kref_put(&srq->ip->ref, rvt_release_mmap_info);
else
vfree(srq->rq.wq);
- kfree(srq);
-
- return 0;
}
#include <rdma/rdma_vt.h>
void rvt_driver_srq_init(struct rvt_dev_info *rdi);
-struct ib_srq *rvt_create_srq(struct ib_pd *ibpd,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata);
+int rvt_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *srq_init_attr,
+ struct ib_udata *udata);
int rvt_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
enum ib_srq_attr_mask attr_mask,
struct ib_udata *udata);
int rvt_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr);
-int rvt_destroy_srq(struct ib_srq *ibsrq);
+void rvt_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata);
#endif /* DEF_RVTSRQ_H */
#include <linux/trace_seq.h>
#include <rdma/ib_verbs.h>
-#include <rdma/rdma_vt.h>
+#include <rdma/rdmavt_qp.h>
#undef TRACE_SYSTEM
#define TRACE_SYSTEM rvt_qp
#include <linux/trace_seq.h>
#include <rdma/ib_verbs.h>
-#include <rdma/rdma_vt.h>
+#include <rdma/rdmavt_qp.h>
#undef TRACE_SYSTEM
#define TRACE_SYSTEM rvt_rc
#include <linux/trace_seq.h>
#include <rdma/ib_verbs.h>
-#include <rdma/rdma_vt.h>
+#include <rdma/rdmavt_qp.h>
#undef TRACE_SYSTEM
#define TRACE_SYSTEM rvt_tx
.req_notify_cq = rvt_req_notify_cq,
.resize_cq = rvt_resize_cq,
.unmap_fmr = rvt_unmap_fmr,
+
+ INIT_RDMA_OBJ_SIZE(ib_ah, rvt_ah, ibah),
INIT_RDMA_OBJ_SIZE(ib_pd, rvt_pd, ibpd),
+ INIT_RDMA_OBJ_SIZE(ib_srq, rvt_srq, ibsrq),
INIT_RDMA_OBJ_SIZE(ib_ucontext, rvt_ucontext, ibucontext),
};
}
int rxe_cq_from_init(struct rxe_dev *rxe, struct rxe_cq *cq, int cqe,
- int comp_vector, struct ib_ucontext *context,
+ int comp_vector, struct ib_udata *udata,
struct rxe_create_cq_resp __user *uresp)
{
int err;
return -ENOMEM;
}
- err = do_mmap_info(rxe, uresp ? &uresp->mi : NULL, context,
+ err = do_mmap_info(rxe, uresp ? &uresp->mi : NULL, udata,
cq->queue->buf, cq->queue->buf_size, &cq->queue->ip);
if (err) {
vfree(cq->queue->buf);
}
int rxe_cq_resize_queue(struct rxe_cq *cq, int cqe,
- struct rxe_resize_cq_resp __user *uresp)
+ struct rxe_resize_cq_resp __user *uresp,
+ struct ib_udata *udata)
{
int err;
err = rxe_queue_resize(cq->queue, (unsigned int *)&cqe,
- sizeof(struct rxe_cqe),
- cq->queue->ip ? cq->queue->ip->context : NULL,
+ sizeof(struct rxe_cqe), udata,
uresp ? &uresp->mi : NULL, NULL, &cq->cq_lock);
if (!err)
cq->ibcq.cqe = cqe;
__be32 rkey;
__be64 swap_add;
__be64 comp;
-} __attribute__((__packed__));
+} __packed;
static inline u64 __atmeth_va(void *arg)
{
int cqe, int comp_vector);
int rxe_cq_from_init(struct rxe_dev *rxe, struct rxe_cq *cq, int cqe,
- int comp_vector, struct ib_ucontext *context,
+ int comp_vector, struct ib_udata *udata,
struct rxe_create_cq_resp __user *uresp);
int rxe_cq_resize_queue(struct rxe_cq *cq, int new_cqe,
- struct rxe_resize_cq_resp __user *uresp);
+ struct rxe_resize_cq_resp __user *uresp,
+ struct ib_udata *udata);
int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited);
void rxe_mmap_release(struct kref *ref);
-struct rxe_mmap_info *rxe_create_mmap_info(struct rxe_dev *dev,
- u32 size,
- struct ib_ucontext *context,
- void *obj);
+struct rxe_mmap_info *rxe_create_mmap_info(struct rxe_dev *dev, u32 size,
+ struct ib_udata *udata, void *obj);
int rxe_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
struct ib_srq_attr *attr, enum ib_srq_attr_mask mask);
int rxe_srq_from_init(struct rxe_dev *rxe, struct rxe_srq *srq,
- struct ib_srq_init_attr *init,
- struct ib_ucontext *context,
+ struct ib_srq_init_attr *init, struct ib_udata *udata,
struct rxe_create_srq_resp __user *uresp);
int rxe_srq_from_attr(struct rxe_dev *rxe, struct rxe_srq *srq,
struct ib_srq_attr *attr, enum ib_srq_attr_mask mask,
- struct rxe_modify_srq_cmd *ucmd);
+ struct rxe_modify_srq_cmd *ucmd, struct ib_udata *udata);
void rxe_dealloc(struct ib_device *ib_dev);
#include <linux/mm.h>
#include <linux/errno.h>
#include <asm/pgtable.h>
+#include <rdma/uverbs_ioctl.h>
#include "rxe.h"
#include "rxe_loc.h"
/*
* Allocate information for rxe_mmap
*/
-struct rxe_mmap_info *rxe_create_mmap_info(struct rxe_dev *rxe,
- u32 size,
- struct ib_ucontext *context,
- void *obj)
+struct rxe_mmap_info *rxe_create_mmap_info(struct rxe_dev *rxe, u32 size,
+ struct ib_udata *udata, void *obj)
{
struct rxe_mmap_info *ip;
+ if (!udata)
+ return ERR_PTR(-EINVAL);
+
ip = kmalloc(sizeof(*ip), GFP_KERNEL);
if (!ip)
return NULL;
INIT_LIST_HEAD(&ip->pending_mmaps);
ip->info.size = size;
- ip->context = context;
+ ip->context =
+ container_of(udata, struct uverbs_attr_bundle, driver_udata)
+ ->context;
ip->obj = obj;
kref_init(&ip->ref);
}
mem->umem = umem;
- num_buf = umem->nmap;
+ num_buf = ib_umem_num_pages(umem);
rxe_mem_init(access, mem);
buf = map[0]->buf;
for_each_sg_page(umem->sg_head.sgl, &sg_iter, umem->nmap, 0) {
+ if (num_buf >= RXE_BUF_PER_MAP) {
+ map++;
+ buf = map[0]->buf;
+ num_buf = 0;
+ }
+
vaddr = page_address(sg_page_iter_page(&sg_iter));
if (!vaddr) {
pr_warn("null vaddr\n");
num_buf++;
buf++;
- if (num_buf >= RXE_BUF_PER_MAP) {
- map++;
- buf = map[0]->buf;
- num_buf = 0;
- }
}
}
ip6h->payload_len = htons(skb->len - sizeof(*ip6h));
}
-static int prepare4(struct rxe_pkt_info *pkt, struct sk_buff *skb,
- struct rxe_av *av)
+static int prepare4(struct rxe_pkt_info *pkt, struct sk_buff *skb)
{
struct rxe_qp *qp = pkt->qp;
struct dst_entry *dst;
bool xnet = false;
__be16 df = htons(IP_DF);
+ struct rxe_av *av = rxe_get_av(pkt);
struct in_addr *saddr = &av->sgid_addr._sockaddr_in.sin_addr;
struct in_addr *daddr = &av->dgid_addr._sockaddr_in.sin_addr;
return 0;
}
-static int prepare6(struct rxe_pkt_info *pkt, struct sk_buff *skb,
- struct rxe_av *av)
+static int prepare6(struct rxe_pkt_info *pkt, struct sk_buff *skb)
{
struct rxe_qp *qp = pkt->qp;
struct dst_entry *dst;
+ struct rxe_av *av = rxe_get_av(pkt);
struct in6_addr *saddr = &av->sgid_addr._sockaddr_in6.sin6_addr;
struct in6_addr *daddr = &av->dgid_addr._sockaddr_in6.sin6_addr;
int rxe_prepare(struct rxe_pkt_info *pkt, struct sk_buff *skb, u32 *crc)
{
int err = 0;
- struct rxe_av *av = rxe_get_av(pkt);
- if (av->network_type == RDMA_NETWORK_IPV4)
- err = prepare4(pkt, skb, av);
- else if (av->network_type == RDMA_NETWORK_IPV6)
- err = prepare6(pkt, skb, av);
+ if (skb->protocol == htons(ETH_P_IP))
+ err = prepare4(pkt, skb);
+ else if (skb->protocol == htons(ETH_P_IPV6))
+ err = prepare6(pkt, skb);
*crc = rxe_icrc_hdr(pkt, skb);
- if (ether_addr_equal(skb->dev->dev_addr, av->dmac))
+ if (ether_addr_equal(skb->dev->dev_addr, rxe_get_av(pkt)->dmac))
pkt->mask |= RXE_LOOPBACK_MASK;
return err;
int rxe_send(struct rxe_pkt_info *pkt, struct sk_buff *skb)
{
- struct rxe_av *av;
int err;
- av = rxe_get_av(pkt);
-
skb->destructor = rxe_skb_tx_dtor;
skb->sk = pkt->qp->sk->sk;
rxe_add_ref(pkt->qp);
atomic_inc(&pkt->qp->skb_out);
- if (av->network_type == RDMA_NETWORK_IPV4) {
+ if (skb->protocol == htons(ETH_P_IP)) {
err = ip_local_out(dev_net(skb_dst(skb)->dev), skb->sk, skb);
- } else if (av->network_type == RDMA_NETWORK_IPV6) {
+ } else if (skb->protocol == htons(ETH_P_IPV6)) {
err = ip6_local_out(dev_net(skb_dst(skb)->dev), skb->sk, skb);
} else {
- pr_err("Unknown layer 3 protocol: %d\n", av->network_type);
+ pr_err("Unknown layer 3 protocol: %d\n", skb->protocol);
atomic_dec(&pkt->qp->skb_out);
rxe_drop_ref(pkt->qp);
kfree_skb(skb);
int paylen, struct rxe_pkt_info *pkt)
{
unsigned int hdr_len;
- struct sk_buff *skb;
+ struct sk_buff *skb = NULL;
struct net_device *ndev;
const struct ib_gid_attr *attr;
const int port_num = 1;
attr = rdma_get_gid_attr(&rxe->ib_dev, port_num, av->grh.sgid_index);
if (IS_ERR(attr))
return NULL;
- ndev = attr->ndev;
if (av->network_type == RDMA_NETWORK_IPV4)
hdr_len = ETH_HLEN + sizeof(struct udphdr) +
hdr_len = ETH_HLEN + sizeof(struct udphdr) +
sizeof(struct ipv6hdr);
+ rcu_read_lock();
+ ndev = rdma_read_gid_attr_ndev_rcu(attr);
+ if (IS_ERR(ndev)) {
+ rcu_read_unlock();
+ goto out;
+ }
skb = alloc_skb(paylen + hdr_len + LL_RESERVED_SPACE(ndev),
GFP_ATOMIC);
- if (unlikely(!skb))
+ if (unlikely(!skb)) {
+ rcu_read_unlock();
goto out;
+ }
- skb_reserve(skb, hdr_len + LL_RESERVED_SPACE(rxe->ndev));
+ skb_reserve(skb, hdr_len + LL_RESERVED_SPACE(ndev));
+ /* FIXME: hold reference to this netdev until life of this skb. */
skb->dev = ndev;
+ rcu_read_unlock();
+
if (av->network_type == RDMA_NETWORK_IPV4)
skb->protocol = htons(ETH_P_IP);
else
[RXE_TYPE_AH] = {
.name = "rxe-ah",
.size = sizeof(struct rxe_ah),
- .flags = RXE_POOL_ATOMIC,
+ .flags = RXE_POOL_ATOMIC | RXE_POOL_NO_ALLOC,
},
[RXE_TYPE_SRQ] = {
.name = "rxe-srq",
.size = sizeof(struct rxe_srq),
- .flags = RXE_POOL_INDEX,
+ .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC,
.min_index = RXE_MIN_SRQ_INDEX,
.max_index = RXE_MAX_SRQ_INDEX,
},
}
static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp,
- struct ib_qp_init_attr *init,
- struct ib_ucontext *context,
+ struct ib_qp_init_attr *init, struct ib_udata *udata,
struct rxe_create_qp_resp __user *uresp)
{
int err;
if (!qp->sq.queue)
return -ENOMEM;
- err = do_mmap_info(rxe, uresp ? &uresp->sq_mi : NULL, context,
+ err = do_mmap_info(rxe, uresp ? &uresp->sq_mi : NULL, udata,
qp->sq.queue->buf, qp->sq.queue->buf_size,
&qp->sq.queue->ip);
static int rxe_qp_init_resp(struct rxe_dev *rxe, struct rxe_qp *qp,
struct ib_qp_init_attr *init,
- struct ib_ucontext *context,
+ struct ib_udata *udata,
struct rxe_create_qp_resp __user *uresp)
{
int err;
if (!qp->rq.queue)
return -ENOMEM;
- err = do_mmap_info(rxe, uresp ? &uresp->rq_mi : NULL, context,
+ err = do_mmap_info(rxe, uresp ? &uresp->rq_mi : NULL, udata,
qp->rq.queue->buf, qp->rq.queue->buf_size,
&qp->rq.queue->ip);
if (err) {
struct rxe_cq *rcq = to_rcq(init->recv_cq);
struct rxe_cq *scq = to_rcq(init->send_cq);
struct rxe_srq *srq = init->srq ? to_rsrq(init->srq) : NULL;
- struct rxe_ucontext *ucontext =
- rdma_udata_to_drv_context(udata, struct rxe_ucontext, ibuc);
rxe_add_ref(pd);
rxe_add_ref(rcq);
rxe_qp_init_misc(rxe, qp, init);
- err = rxe_qp_init_req(rxe, qp, init, &ucontext->ibuc, uresp);
+ err = rxe_qp_init_req(rxe, qp, init, udata, uresp);
if (err)
goto err1;
- err = rxe_qp_init_resp(rxe, qp, init, &ucontext->ibuc, uresp);
+ err = rxe_qp_init_resp(rxe, qp, init, udata, uresp);
if (err)
goto err2;
#include "rxe_loc.h"
#include "rxe_queue.h"
-int do_mmap_info(struct rxe_dev *rxe,
- struct mminfo __user *outbuf,
- struct ib_ucontext *context,
- struct rxe_queue_buf *buf,
- size_t buf_size,
- struct rxe_mmap_info **ip_p)
+int do_mmap_info(struct rxe_dev *rxe, struct mminfo __user *outbuf,
+ struct ib_udata *udata, struct rxe_queue_buf *buf,
+ size_t buf_size, struct rxe_mmap_info **ip_p)
{
int err;
struct rxe_mmap_info *ip = NULL;
if (outbuf) {
- ip = rxe_create_mmap_info(rxe, buf_size, context, buf);
+ ip = rxe_create_mmap_info(rxe, buf_size, udata, buf);
if (!ip)
goto err1;
return 0;
}
-int rxe_queue_resize(struct rxe_queue *q,
- unsigned int *num_elem_p,
- unsigned int elem_size,
- struct ib_ucontext *context,
- struct mminfo __user *outbuf,
- spinlock_t *producer_lock,
+int rxe_queue_resize(struct rxe_queue *q, unsigned int *num_elem_p,
+ unsigned int elem_size, struct ib_udata *udata,
+ struct mminfo __user *outbuf, spinlock_t *producer_lock,
spinlock_t *consumer_lock)
{
struct rxe_queue *new_q;
if (!new_q)
return -ENOMEM;
- err = do_mmap_info(new_q->rxe, outbuf, context, new_q->buf,
+ err = do_mmap_info(new_q->rxe, outbuf, udata, new_q->buf,
new_q->buf_size, &new_q->ip);
if (err) {
vfree(new_q->buf);
unsigned int index_mask;
};
-int do_mmap_info(struct rxe_dev *rxe,
- struct mminfo __user *outbuf,
- struct ib_ucontext *context,
- struct rxe_queue_buf *buf,
- size_t buf_size,
- struct rxe_mmap_info **ip_p);
+int do_mmap_info(struct rxe_dev *rxe, struct mminfo __user *outbuf,
+ struct ib_udata *udata, struct rxe_queue_buf *buf,
+ size_t buf_size, struct rxe_mmap_info **ip_p);
void rxe_queue_reset(struct rxe_queue *q);
int *num_elem,
unsigned int elem_size);
-int rxe_queue_resize(struct rxe_queue *q,
- unsigned int *num_elem_p,
- unsigned int elem_size,
- struct ib_ucontext *context,
+int rxe_queue_resize(struct rxe_queue *q, unsigned int *num_elem_p,
+ unsigned int elem_size, struct ib_udata *udata,
struct mminfo __user *outbuf,
/* Protect producers while resizing queue */
spinlock_t *producer_lock,
}
int rxe_srq_from_init(struct rxe_dev *rxe, struct rxe_srq *srq,
- struct ib_srq_init_attr *init,
- struct ib_ucontext *context,
+ struct ib_srq_init_attr *init, struct ib_udata *udata,
struct rxe_create_srq_resp __user *uresp)
{
int err;
srq->rq.queue = q;
- err = do_mmap_info(rxe, uresp ? &uresp->mi : NULL, context, q->buf,
+ err = do_mmap_info(rxe, uresp ? &uresp->mi : NULL, udata, q->buf,
q->buf_size, &q->ip);
if (err) {
vfree(q->buf);
int rxe_srq_from_attr(struct rxe_dev *rxe, struct rxe_srq *srq,
struct ib_srq_attr *attr, enum ib_srq_attr_mask mask,
- struct rxe_modify_srq_cmd *ucmd)
+ struct rxe_modify_srq_cmd *ucmd, struct ib_udata *udata)
{
int err;
struct rxe_queue *q = srq->rq.queue;
mi = u64_to_user_ptr(ucmd->mmap_info_addr);
err = rxe_queue_resize(q, &attr->max_wr,
- rcv_wqe_size(srq->rq.max_sge),
- srq->rq.queue->ip ?
- srq->rq.queue->ip->context :
- NULL,
- mi, &srq->rq.producer_lock,
+ rcv_wqe_size(srq->rq.max_sge), udata, mi,
+ &srq->rq.producer_lock,
&srq->rq.consumer_lock);
if (err)
goto err2;
return 0;
}
-static int rxe_alloc_pd(struct ib_pd *ibpd, struct ib_ucontext *context,
- struct ib_udata *udata)
+static int rxe_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct rxe_dev *rxe = to_rdev(ibpd->device);
struct rxe_pd *pd = to_rpd(ibpd);
return rxe_add_to_pool(&rxe->pd_pool, &pd->pelem);
}
-static void rxe_dealloc_pd(struct ib_pd *ibpd)
+static void rxe_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata)
{
struct rxe_pd *pd = to_rpd(ibpd);
rxe_drop_ref(pd);
}
-static struct ib_ah *rxe_create_ah(struct ib_pd *ibpd,
- struct rdma_ah_attr *attr,
- u32 flags,
- struct ib_udata *udata)
+static int rxe_create_ah(struct ib_ah *ibah, struct rdma_ah_attr *attr,
+ u32 flags, struct ib_udata *udata)
{
int err;
- struct rxe_dev *rxe = to_rdev(ibpd->device);
- struct rxe_pd *pd = to_rpd(ibpd);
- struct rxe_ah *ah;
+ struct rxe_dev *rxe = to_rdev(ibah->device);
+ struct rxe_ah *ah = to_rah(ibah);
err = rxe_av_chk_attr(rxe, attr);
if (err)
- return ERR_PTR(err);
-
- ah = rxe_alloc(&rxe->ah_pool);
- if (!ah)
- return ERR_PTR(-ENOMEM);
+ return err;
- rxe_add_ref(pd);
- ah->pd = pd;
+ err = rxe_add_to_pool(&rxe->ah_pool, &ah->pelem);
+ if (err)
+ return err;
rxe_init_av(attr, &ah->av);
- return &ah->ibah;
+ return 0;
}
static int rxe_modify_ah(struct ib_ah *ibah, struct rdma_ah_attr *attr)
return 0;
}
-static int rxe_destroy_ah(struct ib_ah *ibah, u32 flags)
+static void rxe_destroy_ah(struct ib_ah *ibah, u32 flags)
{
struct rxe_ah *ah = to_rah(ibah);
- rxe_drop_ref(ah->pd);
rxe_drop_ref(ah);
- return 0;
}
static int post_one_recv(struct rxe_rq *rq, const struct ib_recv_wr *ibwr)
return err;
}
-static struct ib_srq *rxe_create_srq(struct ib_pd *ibpd,
- struct ib_srq_init_attr *init,
- struct ib_udata *udata)
+static int rxe_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *init,
+ struct ib_udata *udata)
{
int err;
- struct rxe_dev *rxe = to_rdev(ibpd->device);
- struct rxe_pd *pd = to_rpd(ibpd);
- struct rxe_ucontext *ucontext =
- rdma_udata_to_drv_context(udata, struct rxe_ucontext, ibuc);
- struct rxe_srq *srq;
+ struct rxe_dev *rxe = to_rdev(ibsrq->device);
+ struct rxe_pd *pd = to_rpd(ibsrq->pd);
+ struct rxe_srq *srq = to_rsrq(ibsrq);
struct rxe_create_srq_resp __user *uresp = NULL;
if (udata) {
if (udata->outlen < sizeof(*uresp))
- return ERR_PTR(-EINVAL);
+ return -EINVAL;
uresp = udata->outbuf;
}
if (err)
goto err1;
- srq = rxe_alloc(&rxe->srq_pool);
- if (!srq) {
- err = -ENOMEM;
+ err = rxe_add_to_pool(&rxe->srq_pool, &srq->pelem);
+ if (err)
goto err1;
- }
- rxe_add_index(srq);
rxe_add_ref(pd);
srq->pd = pd;
- err = rxe_srq_from_init(rxe, srq, init, &ucontext->ibuc, uresp);
+ err = rxe_srq_from_init(rxe, srq, init, udata, uresp);
if (err)
goto err2;
- return &srq->ibsrq;
+ return 0;
err2:
rxe_drop_ref(pd);
- rxe_drop_index(srq);
rxe_drop_ref(srq);
err1:
- return ERR_PTR(err);
+ return err;
}
static int rxe_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
if (err)
goto err1;
- err = rxe_srq_from_attr(rxe, srq, attr, mask, &ucmd);
+ err = rxe_srq_from_attr(rxe, srq, attr, mask, &ucmd, udata);
if (err)
goto err1;
return 0;
}
-static int rxe_destroy_srq(struct ib_srq *ibsrq)
+static void rxe_destroy_srq(struct ib_srq *ibsrq, struct ib_udata *udata)
{
struct rxe_srq *srq = to_rsrq(ibsrq);
rxe_queue_cleanup(srq->rq.queue);
rxe_drop_ref(srq->pd);
- rxe_drop_index(srq);
rxe_drop_ref(srq);
-
- return 0;
}
static int rxe_post_srq_recv(struct ib_srq *ibsrq, const struct ib_recv_wr *wr,
return 0;
}
-static int rxe_destroy_qp(struct ib_qp *ibqp)
+static int rxe_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
{
struct rxe_qp *qp = to_rqp(ibqp);
static struct ib_cq *rxe_create_cq(struct ib_device *dev,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata)
{
int err;
goto err1;
}
- err = rxe_cq_from_init(rxe, cq, attr->cqe, attr->comp_vector,
- context, uresp);
+ err = rxe_cq_from_init(rxe, cq, attr->cqe, attr->comp_vector, udata,
+ uresp);
if (err)
goto err2;
return ERR_PTR(err);
}
-static int rxe_destroy_cq(struct ib_cq *ibcq)
+static int rxe_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
{
struct rxe_cq *cq = to_rcq(ibcq);
if (err)
goto err1;
- err = rxe_cq_resize_queue(cq, cqe, uresp);
+ err = rxe_cq_resize_queue(cq, cqe, uresp, udata);
if (err)
goto err1;
return ERR_PTR(err);
}
-static int rxe_dereg_mr(struct ib_mr *ibmr)
+static int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
{
struct rxe_mem *mr = to_rmr(ibmr);
return 0;
}
-static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd,
- enum ib_mr_type mr_type,
- u32 max_num_sg)
+static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata)
{
struct rxe_dev *rxe = to_rdev(ibpd->device);
struct rxe_pd *pd = to_rpd(ibpd);
.reg_user_mr = rxe_reg_user_mr,
.req_notify_cq = rxe_req_notify_cq,
.resize_cq = rxe_resize_cq,
+
+ INIT_RDMA_OBJ_SIZE(ib_ah, rxe_ah, ibah),
INIT_RDMA_OBJ_SIZE(ib_pd, rxe_pd, ibpd),
+ INIT_RDMA_OBJ_SIZE(ib_srq, rxe_srq, ibsrq),
INIT_RDMA_OBJ_SIZE(ib_ucontext, rxe_ucontext, ibuc),
};
};
struct rxe_ah {
- struct rxe_pool_entry pelem;
struct ib_ah ibah;
+ struct rxe_pool_entry pelem;
struct rxe_pd *pd;
struct rxe_av av;
};
};
struct rxe_srq {
- struct rxe_pool_entry pelem;
struct ib_srq ibsrq;
+ struct rxe_pool_entry pelem;
struct rxe_pd *pd;
struct rxe_rq rq;
u32 srq_num;
{
struct net_device *ndev = to_net_dev(dev);
- if (ndev->dev_id == ndev->dev_port)
+ /*
+ * ndev->dev_port will be equal to 0 in old kernel prior to commit
+ * 9b8b2a323008 ("IB/ipoib: Use dev_port to expose network interface
+ * port numbers") Zero was chosen as special case for user space
+ * applications to fallback and query dev_id to check if it has
+ * different value or not.
+ *
+ * Don't print warning in such scenario.
+ *
+ * https://github.com/systemd/systemd/blob/master/src/udev/udev-builtin-net_id.c#L358
+ */
+ if (ndev->dev_port && ndev->dev_id == ndev->dev_port)
netdev_info_once(ndev,
"\"%s\" wants to know my dev_id. Should it look at dev_port instead? See Documentation/ABI/testing/sysfs-class-net for more info.\n",
current->comm);
ipoib_dbg(priv, "Event %d on device %s port %d\n", record->event,
dev_name(&record->device->dev), record->element.port_num);
- if (record->event == IB_EVENT_SM_CHANGE ||
- record->event == IB_EVENT_CLIENT_REREGISTER) {
+ if (record->event == IB_EVENT_CLIENT_REREGISTER) {
queue_work(ipoib_workqueue, &priv->flush_light);
} else if (record->event == IB_EVENT_PORT_ERR ||
record->event == IB_EVENT_PORT_ACTIVE ||
select SCSI_ISCSI_ATTRS
---help---
Support for the iSCSI Extensions for RDMA (iSER) Protocol
- over InfiniBand. This allows you to access storage devices
- that speak iSCSI over iSER over InfiniBand.
+ over InfiniBand. This allows you to access storage devices
+ that speak iSCSI over iSER over InfiniBand.
The iSER protocol is defined by IETF.
See <http://www.ietf.org/rfc/rfc5046.txt>
enum iscsi_param param, char *buf)
{
struct iser_conn *iser_conn = ep->dd_data;
- int len;
switch (param) {
case ISCSI_PARAM_CONN_PORT:
return iscsi_conn_get_addr_param((struct sockaddr_storage *)
&iser_conn->ib_conn.cma_id->route.addr.dst_addr,
param, buf);
- break;
default:
- return -ENOSYS;
+ break;
}
-
- return len;
+ return -ENOSYS;
}
/**
u64 rsp_dma;
struct ib_sge sge;
struct ib_cqe cqe;
-} __attribute__((packed));
+} __packed;
struct iser_conn;
struct ib_conn;
*/
#include <linux/module.h>
+#include <linux/xarray.h>
#include <rdma/ib_addr.h>
#include <rdma/ib_verbs.h>
#include <rdma/opa_smi.h>
* @class_port_info: Class port info information.
* @tid: Transaction id
* @port_num: OPA port number
- * @vport_idr: vnic ports idr
+ * @vports: vnic ports
* @event_handler: ib event handler
* @lock: adapter interface lock
*/
struct opa_class_port_info class_port_info;
u64 tid;
u8 port_num;
- struct idr vport_idr;
+ struct xarray vports;
struct ib_event_handler event_handler;
/* Lock to query/update network adapter */
{
u8 vport_num = vema_get_vport_num(recvd_mad);
- return idr_find(&port->vport_idr, vport_num);
+ return xa_load(&port->vports, vport_num);
}
/**
int rc;
adapter->cport = cport;
- rc = idr_alloc(&port->vport_idr, adapter, vport_num,
- vport_num + 1, GFP_NOWAIT);
+ rc = xa_insert(&port->vports, vport_num, adapter, GFP_KERNEL);
if (rc < 0) {
opa_vnic_rem_netdev(adapter);
adapter = ERR_PTR(rc);
v_err("Aborting trap\n");
}
-static int vema_rem_vport(int id, void *p, void *data)
-{
- struct opa_vnic_adapter *adapter = p;
-
- opa_vnic_rem_netdev(adapter);
- return 0;
-}
-
-static int vema_enable_vport(int id, void *p, void *data)
-{
- struct opa_vnic_adapter *adapter = p;
-
- netif_carrier_on(adapter->netdev);
- return 0;
-}
-
-static int vema_disable_vport(int id, void *p, void *data)
-{
- struct opa_vnic_adapter *adapter = p;
-
- netif_carrier_off(adapter->netdev);
- return 0;
-}
-
static void opa_vnic_event(struct ib_event_handler *handler,
struct ib_event *record)
{
struct opa_vnic_vema_port *port =
container_of(handler, struct opa_vnic_vema_port, event_handler);
struct opa_vnic_ctrl_port *cport = port->cport;
+ struct opa_vnic_adapter *adapter;
+ unsigned long index;
if (record->element.port_num != port->port_num)
return;
record->event, dev_name(&record->device->dev),
record->element.port_num);
- if (record->event == IB_EVENT_PORT_ERR)
- idr_for_each(&port->vport_idr, vema_disable_vport, NULL);
- if (record->event == IB_EVENT_PORT_ACTIVE)
- idr_for_each(&port->vport_idr, vema_enable_vport, NULL);
+ if (record->event != IB_EVENT_PORT_ERR &&
+ record->event != IB_EVENT_PORT_ACTIVE)
+ return;
+
+ xa_for_each(&port->vports, index, adapter) {
+ if (record->event == IB_EVENT_PORT_ACTIVE)
+ netif_carrier_on(adapter->netdev);
+ else
+ netif_carrier_off(adapter->netdev);
+ }
}
/**
*/
static void vema_unregister(struct opa_vnic_ctrl_port *cport)
{
+ struct opa_vnic_adapter *adapter;
+ unsigned long index;
int i;
for (i = 1; i <= cport->num_ports; i++) {
/* Lock ensures no MAD is being processed */
mutex_lock(&port->lock);
- idr_for_each(&port->vport_idr, vema_rem_vport, NULL);
+ xa_for_each(&port->vports, index, adapter)
+ opa_vnic_rem_netdev(adapter);
mutex_unlock(&port->lock);
ib_unregister_mad_agent(port->mad_agent);
port->mad_agent = NULL;
mutex_destroy(&port->lock);
- idr_destroy(&port->vport_idr);
+ xa_destroy(&port->vports);
ib_unregister_event_handler(&port->event_handler);
}
}
cport->ibdev, opa_vnic_event);
ib_register_event_handler(&port->event_handler);
- idr_init(&port->vport_idr);
+ xa_init(&port->vports);
mutex_init(&port->lock);
port->mad_agent = ib_register_mad_agent(cport->ibdev, i,
IB_QPT_GSI, ®_req,
ret = PTR_ERR(port->mad_agent);
port->mad_agent = NULL;
mutex_destroy(&port->lock);
- idr_destroy(&port->vport_idr);
vema_unregister(cport);
return ret;
}
#define ARM_LPAE_MAIR_ATTR_IDX_CACHE 1
#define ARM_LPAE_MAIR_ATTR_IDX_DEV 2
+#define ARM_MALI_LPAE_TTBR_ADRMODE_TABLE (3u << 0)
+#define ARM_MALI_LPAE_TTBR_READ_INNER BIT(2)
+#define ARM_MALI_LPAE_TTBR_SHARE_OUTER BIT(4)
+
/* IOPTE accessors */
#define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d))
#define iopte_prot(pte) ((pte) & ARM_LPAE_PTE_ATTR_MASK)
-#define iopte_leaf(pte,l) \
- (l == (ARM_LPAE_MAX_LEVELS - 1) ? \
- (iopte_type(pte,l) == ARM_LPAE_PTE_TYPE_PAGE) : \
- (iopte_type(pte,l) == ARM_LPAE_PTE_TYPE_BLOCK))
-
struct arm_lpae_io_pgtable {
struct io_pgtable iop;
typedef u64 arm_lpae_iopte;
+static inline bool iopte_leaf(arm_lpae_iopte pte, int lvl,
+ enum io_pgtable_fmt fmt)
+{
+ if (lvl == (ARM_LPAE_MAX_LEVELS - 1) && fmt != ARM_MALI_LPAE)
+ return iopte_type(pte, lvl) == ARM_LPAE_PTE_TYPE_PAGE;
+
+ return iopte_type(pte, lvl) == ARM_LPAE_PTE_TYPE_BLOCK;
+}
+
static arm_lpae_iopte paddr_to_iopte(phys_addr_t paddr,
struct arm_lpae_io_pgtable *data)
{
if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_ARM_NS)
pte |= ARM_LPAE_PTE_NS;
- if (lvl == ARM_LPAE_MAX_LEVELS - 1)
+ if (data->iop.fmt != ARM_MALI_LPAE && lvl == ARM_LPAE_MAX_LEVELS - 1)
pte |= ARM_LPAE_PTE_TYPE_PAGE;
else
pte |= ARM_LPAE_PTE_TYPE_BLOCK;
- pte |= ARM_LPAE_PTE_AF | ARM_LPAE_PTE_SH_IS;
+ if (data->iop.fmt != ARM_MALI_LPAE)
+ pte |= ARM_LPAE_PTE_AF;
+ pte |= ARM_LPAE_PTE_SH_IS;
pte |= paddr_to_iopte(paddr, data);
__arm_lpae_set_pte(ptep, pte, &data->iop.cfg);
{
arm_lpae_iopte pte = *ptep;
- if (iopte_leaf(pte, lvl)) {
+ if (iopte_leaf(pte, lvl, data->iop.fmt)) {
/* We require an unmap first */
WARN_ON(!selftest_running);
return -EEXIST;
__arm_lpae_sync_pte(ptep, cfg);
}
- if (pte && !iopte_leaf(pte, lvl)) {
+ if (pte && !iopte_leaf(pte, lvl, data->iop.fmt)) {
cptep = iopte_deref(pte, data);
} else if (pte) {
/* We require an unmap first */
if (data->iop.fmt == ARM_64_LPAE_S1 ||
data->iop.fmt == ARM_32_LPAE_S1) {
pte = ARM_LPAE_PTE_nG;
-
if (!(prot & IOMMU_WRITE) && (prot & IOMMU_READ))
pte |= ARM_LPAE_PTE_AP_RDONLY;
-
if (!(prot & IOMMU_PRIV))
pte |= ARM_LPAE_PTE_AP_UNPRIV;
-
- if (prot & IOMMU_MMIO)
- pte |= (ARM_LPAE_MAIR_ATTR_IDX_DEV
- << ARM_LPAE_PTE_ATTRINDX_SHIFT);
- else if (prot & IOMMU_CACHE)
- pte |= (ARM_LPAE_MAIR_ATTR_IDX_CACHE
- << ARM_LPAE_PTE_ATTRINDX_SHIFT);
} else {
pte = ARM_LPAE_PTE_HAP_FAULT;
if (prot & IOMMU_READ)
pte |= ARM_LPAE_PTE_HAP_READ;
if (prot & IOMMU_WRITE)
pte |= ARM_LPAE_PTE_HAP_WRITE;
+ }
+
+ /*
+ * Note that this logic is structured to accommodate Mali LPAE
+ * having stage-1-like attributes but stage-2-like permissions.
+ */
+ if (data->iop.fmt == ARM_64_LPAE_S2 ||
+ data->iop.fmt == ARM_32_LPAE_S2) {
if (prot & IOMMU_MMIO)
pte |= ARM_LPAE_PTE_MEMATTR_DEV;
else if (prot & IOMMU_CACHE)
pte |= ARM_LPAE_PTE_MEMATTR_OIWB;
else
pte |= ARM_LPAE_PTE_MEMATTR_NC;
+ } else {
+ if (prot & IOMMU_MMIO)
+ pte |= (ARM_LPAE_MAIR_ATTR_IDX_DEV
+ << ARM_LPAE_PTE_ATTRINDX_SHIFT);
+ else if (prot & IOMMU_CACHE)
+ pte |= (ARM_LPAE_MAIR_ATTR_IDX_CACHE
+ << ARM_LPAE_PTE_ATTRINDX_SHIFT);
}
if (prot & IOMMU_NOEXEC)
while (ptep != end) {
arm_lpae_iopte pte = *ptep++;
- if (!pte || iopte_leaf(pte, lvl))
+ if (!pte || iopte_leaf(pte, lvl, data->iop.fmt))
continue;
__arm_lpae_free_pgtable(data, lvl + 1, iopte_deref(pte, data));
if (size == ARM_LPAE_BLOCK_SIZE(lvl, data)) {
__arm_lpae_set_pte(ptep, 0, &iop->cfg);
- if (!iopte_leaf(pte, lvl)) {
+ if (!iopte_leaf(pte, lvl, iop->fmt)) {
/* Also flush any partial walks */
io_pgtable_tlb_add_flush(iop, iova, size,
ARM_LPAE_GRANULE(data), false);
}
return size;
- } else if (iopte_leaf(pte, lvl)) {
+ } else if (iopte_leaf(pte, lvl, iop->fmt)) {
/*
* Insert a table at the next level to map the old region,
* minus the part we want to unmap
return 0;
/* Leaf entry? */
- if (iopte_leaf(pte,lvl))
+ if (iopte_leaf(pte, lvl, data->iop.fmt))
goto found_translation;
/* Take it to the next level */
return iop;
}
+static struct io_pgtable *
+arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie)
+{
+ struct io_pgtable *iop;
+
+ if (cfg->ias != 48 || cfg->oas > 40)
+ return NULL;
+
+ cfg->pgsize_bitmap &= (SZ_4K | SZ_2M | SZ_1G);
+ iop = arm_64_lpae_alloc_pgtable_s1(cfg, cookie);
+ if (iop) {
+ u64 mair, ttbr;
+
+ /* Copy values as union fields overlap */
+ mair = cfg->arm_lpae_s1_cfg.mair[0];
+ ttbr = cfg->arm_lpae_s1_cfg.ttbr[0];
+
+ cfg->arm_mali_lpae_cfg.memattr = mair;
+ cfg->arm_mali_lpae_cfg.transtab = ttbr |
+ ARM_MALI_LPAE_TTBR_READ_INNER |
+ ARM_MALI_LPAE_TTBR_ADRMODE_TABLE;
+ }
+
+ return iop;
+}
+
struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s1_init_fns = {
.alloc = arm_64_lpae_alloc_pgtable_s1,
.free = arm_lpae_free_pgtable,
.free = arm_lpae_free_pgtable,
};
+struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns = {
+ .alloc = arm_mali_lpae_alloc_pgtable,
+ .free = arm_lpae_free_pgtable,
+};
+
#ifdef CONFIG_IOMMU_IO_PGTABLE_LPAE_SELFTEST
static struct io_pgtable_cfg *cfg_cookie;
[ARM_32_LPAE_S2] = &io_pgtable_arm_32_lpae_s2_init_fns,
[ARM_64_LPAE_S1] = &io_pgtable_arm_64_lpae_s1_init_fns,
[ARM_64_LPAE_S2] = &io_pgtable_arm_64_lpae_s2_init_fns,
+ [ARM_MALI_LPAE] = &io_pgtable_arm_mali_lpae_init_fns,
#endif
#ifdef CONFIG_IOMMU_IO_PGTABLE_ARMV7S
[ARM_V7S] = &io_pgtable_arm_v7s_init_fns,
static void brx_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
struct vsp1_brx *brx = to_brx(&entity->subdev);
static void clu_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
struct vsp1_clu *clu = to_clu(&entity->subdev);
* @post_cmd: post command to be issued through extended dl header
* @has_chain: if true, indicates that there's a partition chain
* @chain: entry in the display list partition chain
- * @internal: whether the display list is used for internal purpose
+ * @flags: display list flags, a combination of VSP1_DL_FRAME_END_*
*/
struct vsp1_dl_list {
struct list_head list;
bool has_chain;
struct list_head chain;
- bool internal;
+ unsigned int flags;
};
/**
* which bodies are added.
*
* Adding a body to a display list passes ownership of the body to the list. The
- * caller retains its reference to the fragment when adding it to the display
- * list, but is not allowed to add new entries to the body.
+ * caller retains its reference to the body when adding it to the display list,
+ * but is not allowed to add new entries to the body.
*
* The reference must be explicitly released by a call to vsp1_dl_body_put()
* when the body isn't needed anymore.
}
dl->header->num_lists = num_lists;
+ dl->header->flags = 0;
- if (!list_empty(&dl->chain) && !is_last) {
+ /*
+ * Enable the interrupt for the end of each frame. In continuous mode
+ * chained lists are used with one list per frame, so enable the
+ * interrupt for each list. In singleshot mode chained lists are used
+ * to partition a single frame, so enable the interrupt for the last
+ * list only.
+ */
+ if (!dlm->singleshot || is_last)
+ dl->header->flags |= VSP1_DLH_INT_ENABLE;
+
+ /*
+ * In continuous mode enable auto-start for all lists, as the VSP must
+ * loop on the same list until a new one is queued. In singleshot mode
+ * enable auto-start for all lists but the last to chain processing of
+ * partitions without software intervention.
+ */
+ if (!dlm->singleshot || !is_last)
+ dl->header->flags |= VSP1_DLH_AUTO_START;
+
+ if (!is_last) {
/*
- * If this display list's chain is not empty, we are on a list,
- * and the next item is the display list that we must queue for
- * automatic processing by the hardware.
+ * If this is not the last display list in the chain, queue the
+ * next item for automatic processing by the hardware.
*/
struct vsp1_dl_list *next = list_next_entry(dl, chain);
dl->header->next_header = next->dma;
- dl->header->flags = VSP1_DLH_AUTO_START;
} else if (!dlm->singleshot) {
/*
* if the display list manager works in continuous mode, the VSP
* instructed to do otherwise.
*/
dl->header->next_header = dl->dma;
- dl->header->flags = VSP1_DLH_INT_ENABLE | VSP1_DLH_AUTO_START;
- } else {
- /*
- * Otherwise, in mem-to-mem mode, we work in single-shot mode
- * and the next display list must not be started automatically.
- */
- dl->header->flags = VSP1_DLH_INT_ENABLE;
}
if (!dl->extension)
*
* If a display list is already pending we simply drop it as the new
* display list is assumed to contain a more recent configuration. It is
- * an error if the already pending list has the internal flag set, as
- * there is then a process waiting for that list to complete. This
- * shouldn't happen as the waiting process should perform proper
- * locking, but warn just in case.
+ * an error if the already pending list has the
+ * VSP1_DL_FRAME_END_INTERNAL flag set, as there is then a process
+ * waiting for that list to complete. This shouldn't happen as the
+ * waiting process should perform proper locking, but warn just in
+ * case.
*/
if (vsp1_dl_list_hw_update_pending(dlm)) {
- WARN_ON(dlm->pending && dlm->pending->internal);
+ WARN_ON(dlm->pending &&
+ (dlm->pending->flags & VSP1_DL_FRAME_END_INTERNAL));
__vsp1_dl_list_put(dlm->pending);
dlm->pending = dl;
return;
dlm->active = dl;
}
-void vsp1_dl_list_commit(struct vsp1_dl_list *dl, bool internal)
+void vsp1_dl_list_commit(struct vsp1_dl_list *dl, unsigned int dl_flags)
{
struct vsp1_dl_manager *dlm = dl->dlm;
struct vsp1_dl_list *dl_next;
vsp1_dl_list_fill_header(dl_next, last);
}
- dl->internal = internal;
+ dl->flags = dl_flags & ~VSP1_DL_FRAME_END_COMPLETED;
spin_lock_irqsave(&dlm->lock, flags);
* set in single-shot mode as display list processing is then not continuous and
* races never occur.
*
- * The VSP1_DL_FRAME_END_INTERNAL flag indicates that the previous display list
- * has completed and had been queued with the internal notification flag.
- * Internal notification is only supported for continuous mode.
+ * The following flags are only supported for continuous mode.
+ *
+ * The VSP1_DL_FRAME_END_INTERNAL flag indicates that the display list that just
+ * became active had been queued with the internal notification flag.
+ *
+ * The VSP1_DL_FRAME_END_WRITEBACK flag indicates that the previously active
+ * display list had been queued with the writeback flag.
*/
unsigned int vsp1_dlm_irq_frame_end(struct vsp1_dl_manager *dlm)
{
if (status & VI6_STATUS_FLD_STD(dlm->index))
goto done;
+ /*
+ * If the active display list has the writeback flag set, the frame
+ * completion marks the end of the writeback capture. Return the
+ * VSP1_DL_FRAME_END_WRITEBACK flag and reset the display list's
+ * writeback flag.
+ */
+ if (dlm->active && (dlm->active->flags & VSP1_DL_FRAME_END_WRITEBACK)) {
+ flags |= VSP1_DL_FRAME_END_WRITEBACK;
+ dlm->active->flags &= ~VSP1_DL_FRAME_END_WRITEBACK;
+ }
+
/*
* The device starts processing the queued display list right after the
* frame end interrupt. The display list thus becomes active.
*/
if (dlm->queued) {
- if (dlm->queued->internal)
+ if (dlm->queued->flags & VSP1_DL_FRAME_END_INTERNAL)
flags |= VSP1_DL_FRAME_END_INTERNAL;
- dlm->queued->internal = false;
+ dlm->queued->flags &= ~VSP1_DL_FRAME_END_INTERNAL;
__vsp1_dl_list_put(dlm->active);
dlm->active = dlm->queued;
struct vsp1_dl_list;
struct vsp1_dl_manager;
+/* Keep these flags in sync with VSP1_DU_STATUS_* in include/media/vsp1.h. */
#define VSP1_DL_FRAME_END_COMPLETED BIT(0)
-#define VSP1_DL_FRAME_END_INTERNAL BIT(1)
+#define VSP1_DL_FRAME_END_WRITEBACK BIT(1)
+#define VSP1_DL_FRAME_END_INTERNAL BIT(2)
/**
* struct vsp1_dl_ext_cmd - Extended Display command
void vsp1_dl_list_put(struct vsp1_dl_list *dl);
struct vsp1_dl_body *vsp1_dl_list_get_body0(struct vsp1_dl_list *dl);
struct vsp1_dl_ext_cmd *vsp1_dl_get_pre_cmd(struct vsp1_dl_list *dl);
-void vsp1_dl_list_commit(struct vsp1_dl_list *dl, bool internal);
+void vsp1_dl_list_commit(struct vsp1_dl_list *dl, unsigned int dl_flags);
struct vsp1_dl_body_pool *
vsp1_dl_body_pool_create(struct vsp1_device *vsp1, unsigned int num_bodies,
unsigned int completion)
{
struct vsp1_drm_pipeline *drm_pipe = to_vsp1_drm_pipeline(pipe);
- bool complete = completion == VSP1_DL_FRAME_END_COMPLETED;
if (drm_pipe->du_complete) {
struct vsp1_entity *uif = drm_pipe->uif;
+ unsigned int status = completion
+ & (VSP1_DU_STATUS_COMPLETE |
+ VSP1_DU_STATUS_WRITEBACK);
u32 crc;
crc = uif ? vsp1_uif_get_crc(to_uif(&uif->subdev)) : 0;
- drm_pipe->du_complete(drm_pipe->du_private, complete, crc);
+ drm_pipe->du_complete(drm_pipe->du_private, status, crc);
}
if (completion & VSP1_DL_FRAME_END_INTERNAL) {
struct vsp1_entity *next;
struct vsp1_dl_list *dl;
struct vsp1_dl_body *dlb;
+ unsigned int dl_flags = 0;
+
+ if (drm_pipe->force_brx_release)
+ dl_flags |= VSP1_DL_FRAME_END_INTERNAL;
+ if (pipe->output->writeback)
+ dl_flags |= VSP1_DL_FRAME_END_WRITEBACK;
dl = vsp1_dl_list_get(pipe->output->dlm);
dlb = vsp1_dl_list_get_body0(dl);
}
vsp1_entity_route_setup(entity, pipe, dlb);
- vsp1_entity_configure_stream(entity, pipe, dlb);
+ vsp1_entity_configure_stream(entity, pipe, dl, dlb);
vsp1_entity_configure_frame(entity, pipe, dl, dlb);
vsp1_entity_configure_partition(entity, pipe, dl, dlb);
}
- vsp1_dl_list_commit(dl, drm_pipe->force_brx_release);
+ vsp1_dl_list_commit(dl, dl_flags);
+}
+
+static int vsp1_du_pipeline_set_rwpf_format(struct vsp1_device *vsp1,
+ struct vsp1_rwpf *rwpf,
+ u32 pixelformat, unsigned int pitch)
+{
+ const struct vsp1_format_info *fmtinfo;
+ unsigned int chroma_hsub;
+
+ fmtinfo = vsp1_get_format_info(vsp1, pixelformat);
+ if (!fmtinfo) {
+ dev_dbg(vsp1->dev, "Unsupported pixel format %08x\n",
+ pixelformat);
+ return -EINVAL;
+ }
+
+ /*
+ * Only formats with three planes can affect the chroma planes pitch.
+ * All formats with two planes have a horizontal subsampling value of 2,
+ * but combine U and V in a single chroma plane, which thus results in
+ * the luma plane and chroma plane having the same pitch.
+ */
+ chroma_hsub = (fmtinfo->planes == 3) ? fmtinfo->hsub : 1;
+
+ rwpf->fmtinfo = fmtinfo;
+ rwpf->format.num_planes = fmtinfo->planes;
+ rwpf->format.plane_fmt[0].bytesperline = pitch;
+ rwpf->format.plane_fmt[1].bytesperline = pitch / chroma_hsub;
+
+ return 0;
}
/* -----------------------------------------------------------------------------
drm_pipe->du_private = cfg->callback_data;
/* Disable the display interrupts. */
- vsp1_write(vsp1, VI6_DISP_IRQ_STA, 0);
- vsp1_write(vsp1, VI6_DISP_IRQ_ENB, 0);
+ vsp1_write(vsp1, VI6_DISP_IRQ_STA(pipe_index), 0);
+ vsp1_write(vsp1, VI6_DISP_IRQ_ENB(pipe_index), 0);
/* Configure all entities in the pipeline. */
vsp1_du_pipeline_configure(pipe);
{
struct vsp1_device *vsp1 = dev_get_drvdata(dev);
struct vsp1_drm_pipeline *drm_pipe = &vsp1->drm->pipe[pipe_index];
- const struct vsp1_format_info *fmtinfo;
- unsigned int chroma_hsub;
struct vsp1_rwpf *rpf;
+ int ret;
if (rpf_index >= vsp1->info->rpf_count)
return -EINVAL;
* Store the format, stride, memory buffer address, crop and compose
* rectangles and Z-order position and for the input.
*/
- fmtinfo = vsp1_get_format_info(vsp1, cfg->pixelformat);
- if (!fmtinfo) {
- dev_dbg(vsp1->dev, "Unsupported pixel format %08x for RPF\n",
- cfg->pixelformat);
- return -EINVAL;
- }
-
- /*
- * Only formats with three planes can affect the chroma planes pitch.
- * All formats with two planes have a horizontal subsampling value of 2,
- * but combine U and V in a single chroma plane, which thus results in
- * the luma plane and chroma plane having the same pitch.
- */
- chroma_hsub = (fmtinfo->planes == 3) ? fmtinfo->hsub : 1;
+ ret = vsp1_du_pipeline_set_rwpf_format(vsp1, rpf, cfg->pixelformat,
+ cfg->pitch);
+ if (ret < 0)
+ return ret;
- rpf->fmtinfo = fmtinfo;
- rpf->format.num_planes = fmtinfo->planes;
- rpf->format.plane_fmt[0].bytesperline = cfg->pitch;
- rpf->format.plane_fmt[1].bytesperline = cfg->pitch / chroma_hsub;
rpf->alpha = cfg->alpha;
rpf->mem.addr[0] = cfg->mem[0];
struct vsp1_device *vsp1 = dev_get_drvdata(dev);
struct vsp1_drm_pipeline *drm_pipe = &vsp1->drm->pipe[pipe_index];
struct vsp1_pipeline *pipe = &drm_pipe->pipe;
+ int ret;
drm_pipe->crc = cfg->crc;
mutex_lock(&vsp1->drm->lock);
+
+ if (cfg->writeback.pixelformat) {
+ const struct vsp1_du_writeback_config *wb_cfg = &cfg->writeback;
+
+ ret = vsp1_du_pipeline_set_rwpf_format(vsp1, pipe->output,
+ wb_cfg->pixelformat,
+ wb_cfg->pitch);
+ if (WARN_ON(ret < 0))
+ goto done;
+
+ pipe->output->mem.addr[0] = wb_cfg->mem[0];
+ pipe->output->mem.addr[1] = wb_cfg->mem[1];
+ pipe->output->mem.addr[2] = wb_cfg->mem[2];
+ pipe->output->writeback = true;
+ }
+
vsp1_du_pipeline_setup_inputs(vsp1, pipe);
vsp1_du_pipeline_configure(pipe);
+
+done:
mutex_unlock(&vsp1->drm->lock);
}
EXPORT_SYMBOL_GPL(vsp1_du_atomic_flush);
struct vsp1_du_crc_config crc;
/* Frame synchronisation */
- void (*du_complete)(void *data, bool completed, u32 crc);
+ void (*du_complete)(void *data, unsigned int status, u32 crc);
void *du_private;
};
void vsp1_entity_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
if (entity->ops->configure_stream)
- entity->ops->configure_stream(entity, pipe, dlb);
+ entity->ops->configure_stream(entity, pipe, dl, dlb);
}
void vsp1_entity_configure_frame(struct vsp1_entity *entity,
* struct vsp1_entity_operations - Entity operations
* @destroy: Destroy the entity.
* @configure_stream: Setup the hardware parameters for the stream which do
- * not vary between frames (pipeline, formats).
+ * not vary between frames (pipeline, formats). Note that
+ * the vsp1_dl_list argument is only valid for display
+ * pipeline and will be NULL for mem-to-mem pipelines.
* @configure_frame: Configure the runtime parameters for each frame.
* @configure_partition: Configure partition specific parameters.
* @max_width: Return the max supported width of data that the entity can
struct vsp1_entity_operations {
void (*destroy)(struct vsp1_entity *);
void (*configure_stream)(struct vsp1_entity *, struct vsp1_pipeline *,
- struct vsp1_dl_body *);
+ struct vsp1_dl_list *, struct vsp1_dl_body *);
void (*configure_frame)(struct vsp1_entity *, struct vsp1_pipeline *,
struct vsp1_dl_list *, struct vsp1_dl_body *);
void (*configure_partition)(struct vsp1_entity *,
void vsp1_entity_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb);
void vsp1_entity_configure_frame(struct vsp1_entity *entity,
static void hgo_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
struct vsp1_hgo *hgo = to_hgo(&entity->subdev);
static void hgt_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
struct vsp1_hgt *hgt = to_hgt(&entity->subdev);
static void hsit_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
struct vsp1_hsit *hsit = to_hsit(&entity->subdev);
static void lif_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
const struct v4l2_mbus_framefmt *format;
static void lut_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
struct vsp1_lut *lut = to_lut(&entity->subdev);
#define VI6_WFP_IRQ_STA_DFE (1 << 1)
#define VI6_WFP_IRQ_STA_FRE (1 << 0)
-#define VI6_DISP_IRQ_ENB 0x0078
+#define VI6_DISP_IRQ_ENB(n) (0x0078 + (n) * 60)
#define VI6_DISP_IRQ_ENB_DSTE (1 << 8)
#define VI6_DISP_IRQ_ENB_MAEE (1 << 5)
#define VI6_DISP_IRQ_ENB_LNEE(n) (1 << (n))
-#define VI6_DISP_IRQ_STA 0x007c
+#define VI6_DISP_IRQ_STA(n) (0x007c + (n) * 60)
#define VI6_DISP_IRQ_STA_DST (1 << 8)
#define VI6_DISP_IRQ_STA_MAE (1 << 5)
#define VI6_DISP_IRQ_STA_LNE(n) (1 << (n))
#define VI6_WPF_DSTM_ADDR_C0 0x1028
#define VI6_WPF_DSTM_ADDR_C1 0x102c
-#define VI6_WPF_WRBCK_CTRL 0x1034
+#define VI6_WPF_WRBCK_CTRL(n) (0x1034 + (n) * 0x100)
#define VI6_WPF_WRBCK_CTRL_WBMD (1 << 0)
/* -----------------------------------------------------------------------------
static void rpf_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
struct vsp1_rwpf *rpf = to_rwpf(&entity->subdev);
} flip;
struct vsp1_rwpf_memory mem;
+ bool writeback;
struct vsp1_dl_manager *dlm;
};
static void sru_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
const struct vsp1_sru_param *param;
static void uds_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
struct vsp1_uds *uds = to_uds(&entity->subdev);
static void uif_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
struct vsp1_uif *uif = to_uif(&entity->subdev);
* This function completes the current buffer by filling its sequence number,
* time stamp and payload size, and hands it back to the videobuf core.
*
- * When operating in DU output mode (deep pipeline to the DU through the LIF),
- * the VSP1 needs to constantly supply frames to the display. In that case, if
- * no other buffer is queued, reuse the one that has just been processed instead
- * of handing it back to the videobuf core.
- *
* Return the next queued buffer or NULL if the queue is empty.
*/
static struct vsp1_vb2_buffer *
done = list_first_entry(&video->irqqueue,
struct vsp1_vb2_buffer, queue);
- /* In DU output mode reuse the buffer if the list is singular. */
- if (pipe->lif && list_is_singular(&video->irqqueue)) {
- spin_unlock_irqrestore(&video->irqlock, flags);
- return done;
- }
-
list_del(&done->queue);
if (!list_empty(&video->irqqueue))
}
/* Complete, and commit the head display list. */
- vsp1_dl_list_commit(dl, false);
+ vsp1_dl_list_commit(dl, 0);
pipe->configured = true;
vsp1_pipeline_run(pipe);
list_for_each_entry(entity, &pipe->entities, list_pipe) {
vsp1_entity_route_setup(entity, pipe, pipe->stream_config);
- vsp1_entity_configure_stream(entity, pipe, pipe->stream_config);
+ vsp1_entity_configure_stream(entity, pipe, NULL,
+ pipe->stream_config);
}
return 0;
vsp1_dlm_destroy(wpf->dlm);
}
+static int wpf_configure_writeback_chain(struct vsp1_rwpf *wpf,
+ struct vsp1_dl_list *dl)
+{
+ unsigned int index = wpf->entity.index;
+ struct vsp1_dl_list *dl_next;
+ struct vsp1_dl_body *dlb;
+
+ dl_next = vsp1_dl_list_get(wpf->dlm);
+ if (!dl_next) {
+ dev_err(wpf->entity.vsp1->dev,
+ "Failed to obtain a dl list, disabling writeback\n");
+ return -ENOMEM;
+ }
+
+ dlb = vsp1_dl_list_get_body0(dl_next);
+ vsp1_dl_body_write(dlb, VI6_WPF_WRBCK_CTRL(index), 0);
+ vsp1_dl_list_add_chain(dl, dl_next);
+
+ return 0;
+}
+
static void wpf_configure_stream(struct vsp1_entity *entity,
struct vsp1_pipeline *pipe,
+ struct vsp1_dl_list *dl,
struct vsp1_dl_body *dlb)
{
struct vsp1_rwpf *wpf = to_rwpf(&entity->subdev);
struct vsp1_device *vsp1 = wpf->entity.vsp1;
const struct v4l2_mbus_framefmt *source_format;
const struct v4l2_mbus_framefmt *sink_format;
+ unsigned int index = wpf->entity.index;
unsigned int i;
u32 outfmt = 0;
u32 srcrpf = 0;
+ int ret;
sink_format = vsp1_entity_get_pad_format(&wpf->entity,
wpf->entity.config,
source_format = vsp1_entity_get_pad_format(&wpf->entity,
wpf->entity.config,
RWPF_PAD_SOURCE);
+
/* Format */
- if (!pipe->lif) {
+ if (!pipe->lif || wpf->writeback) {
const struct v4l2_pix_format_mplane *format = &wpf->format;
const struct vsp1_format_info *fmtinfo = wpf->fmtinfo;
vsp1_wpf_write(wpf, dlb, VI6_WPF_DSWAP, fmtinfo->swap);
- if (vsp1_feature(vsp1, VSP1_HAS_WPF_HFLIP) &&
- wpf->entity.index == 0)
+ if (vsp1_feature(vsp1, VSP1_HAS_WPF_HFLIP) && index == 0)
vsp1_wpf_write(wpf, dlb, VI6_WPF_ROT_CTRL,
VI6_WPF_ROT_CTRL_LN16 |
(256 << VI6_WPF_ROT_CTRL_LMEM_WD_SHIFT));
wpf->outfmt = outfmt;
- vsp1_dl_body_write(dlb, VI6_DPR_WPF_FPORCH(wpf->entity.index),
+ vsp1_dl_body_write(dlb, VI6_DPR_WPF_FPORCH(index),
VI6_DPR_WPF_FPORCH_FP_WPFN);
- vsp1_dl_body_write(dlb, VI6_WPF_WRBCK_CTRL, 0);
-
/*
* Sources. If the pipeline has a single input and BRx is not used,
* configure it as the master layer. Otherwise configure all
vsp1_wpf_write(wpf, dlb, VI6_WPF_SRCRPF, srcrpf);
/* Enable interrupts. */
- vsp1_dl_body_write(dlb, VI6_WPF_IRQ_STA(wpf->entity.index), 0);
- vsp1_dl_body_write(dlb, VI6_WPF_IRQ_ENB(wpf->entity.index),
+ vsp1_dl_body_write(dlb, VI6_WPF_IRQ_STA(index), 0);
+ vsp1_dl_body_write(dlb, VI6_WPF_IRQ_ENB(index),
VI6_WFP_IRQ_ENB_DFEE);
+
+ /*
+ * Configure writeback for display pipelines (the wpf writeback flag is
+ * never set for memory-to-memory pipelines). Start by adding a chained
+ * display list to disable writeback after a single frame, and process
+ * to enable writeback. If the display list allocation fails don't
+ * enable writeback as we wouldn't be able to safely disable it,
+ * resulting in possible memory corruption.
+ */
+ if (wpf->writeback) {
+ ret = wpf_configure_writeback_chain(wpf, dl);
+ if (ret < 0)
+ wpf->writeback = false;
+ }
+
+ vsp1_dl_body_write(dlb, VI6_WPF_WRBCK_CTRL(index),
+ wpf->writeback ? VI6_WPF_WRBCK_CTRL_WBMD : 0);
}
static void wpf_configure_frame(struct vsp1_entity *entity,
const struct vsp1_format_info *fmtinfo = wpf->fmtinfo;
unsigned int width;
unsigned int height;
+ unsigned int left;
unsigned int offset;
unsigned int flip;
unsigned int i;
RWPF_PAD_SINK);
width = sink_format->width;
height = sink_format->height;
+ left = 0;
/*
* Cropping. The partition algorithm can split the image into
* multiple slices.
*/
- if (pipe->partitions > 1)
+ if (pipe->partitions > 1) {
width = pipe->partition->wpf.width;
+ left = pipe->partition->wpf.left;
+ }
vsp1_wpf_write(wpf, dlb, VI6_WPF_HSZCLIP, VI6_WPF_SZCLIP_EN |
(0 << VI6_WPF_SZCLIP_OFST_SHIFT) |
(0 << VI6_WPF_SZCLIP_OFST_SHIFT) |
(height << VI6_WPF_SZCLIP_SIZE_SHIFT));
- if (pipe->lif)
+ /*
+ * For display pipelines without writeback enabled there's no memory
+ * address to configure, return now.
+ */
+ if (pipe->lif && !wpf->writeback)
return;
/*
flip = wpf->flip.active;
if (flip & BIT(WPF_CTRL_HFLIP) && !wpf->flip.rotate)
- offset = format->width - pipe->partition->wpf.left
- - pipe->partition->wpf.width;
+ offset = format->width - left - width;
else if (flip & BIT(WPF_CTRL_VFLIP) && wpf->flip.rotate)
- offset = format->height - pipe->partition->wpf.left
- - pipe->partition->wpf.width;
+ offset = format->height - left - width;
else
- offset = pipe->partition->wpf.left;
+ offset = left;
for (i = 0; i < format->num_planes; ++i) {
unsigned int hsub = i > 0 ? fmtinfo->hsub : 1;
* image height.
*/
if (wpf->flip.rotate)
- height = pipe->partition->wpf.width;
+ height = width;
else
height = format->height;
vsp1_wpf_write(wpf, dlb, VI6_WPF_DSTM_ADDR_Y, mem.addr[0]);
vsp1_wpf_write(wpf, dlb, VI6_WPF_DSTM_ADDR_C0, mem.addr[1]);
vsp1_wpf_write(wpf, dlb, VI6_WPF_DSTM_ADDR_C1, mem.addr[2]);
+
+ /*
+ * Writeback operates in single-shot mode and lasts for a single frame,
+ * reset the writeback flag to false for the next frame.
+ */
+ wpf->writeback = false;
}
static unsigned int wpf_max_width(struct vsp1_entity *entity,
#include <linux/seq_file.h>
#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/dma/idma64.h>
+
#include "intel-lpss.h"
#define LPSS_DEV_OFFSET 0x000
DEFINE_RES_IRQ(0),
};
-#define LPSS_IDMA64_DRIVER_NAME "idma64"
-
/*
* Cells needs to be ordered so that the iDMA is created first. This is
* because we need to be sure the DMA is available when the host controller
do_div(ns, NSEC_PER_SEC / HZ);
clock->overflow_period = ns;
- mdev->clock_info_page = alloc_page(GFP_KERNEL);
- if (mdev->clock_info_page) {
- mdev->clock_info = kmap(mdev->clock_info_page);
- if (!mdev->clock_info) {
- __free_page(mdev->clock_info_page);
- mlx5_core_warn(mdev, "failed to map clock page\n");
- } else {
- mdev->clock_info->sign = 0;
- mdev->clock_info->nsec = clock->tc.nsec;
- mdev->clock_info->cycles = clock->tc.cycle_last;
- mdev->clock_info->mask = clock->cycles.mask;
- mdev->clock_info->mult = clock->nominal_c_mult;
- mdev->clock_info->shift = clock->cycles.shift;
- mdev->clock_info->frac = clock->tc.frac;
- mdev->clock_info->overflow_period =
- clock->overflow_period;
- }
+ mdev->clock_info =
+ (struct mlx5_ib_clock_info *)get_zeroed_page(GFP_KERNEL);
+ if (mdev->clock_info) {
+ mdev->clock_info->nsec = clock->tc.nsec;
+ mdev->clock_info->cycles = clock->tc.cycle_last;
+ mdev->clock_info->mask = clock->cycles.mask;
+ mdev->clock_info->mult = clock->nominal_c_mult;
+ mdev->clock_info->shift = clock->cycles.shift;
+ mdev->clock_info->frac = clock->tc.frac;
+ mdev->clock_info->overflow_period = clock->overflow_period;
}
INIT_WORK(&clock->pps_info.out_work, mlx5_pps_out);
cancel_delayed_work_sync(&clock->overflow_work);
if (mdev->clock_info) {
- kunmap(mdev->clock_info_page);
- __free_page(mdev->clock_info_page);
+ free_page((unsigned long)mdev->clock_info);
mdev->clock_info = NULL;
}
/**
* parport_ip32_dma_start - begins a DMA transfer
+ * @p: partport to work on
* @dir: DMA direction: DMA_TO_DEVICE or DMA_FROM_DEVICE
* @addr: pointer to data buffer
* @count: buffer size
* Calls to parport_ip32_dma_start() and parport_ip32_dma_stop() must be
* correctly balanced.
*/
-static int parport_ip32_dma_start(enum dma_data_direction dir,
- void *addr, size_t count)
+static int parport_ip32_dma_start(struct parport *p,
+ enum dma_data_direction dir, void *addr, size_t count)
{
unsigned int limit;
u64 ctrl;
/* Prepare DMA pointers */
parport_ip32_dma.dir = dir;
- parport_ip32_dma.buf = dma_map_single(NULL, addr, count, dir);
+ parport_ip32_dma.buf = dma_map_single(&p->bus_dev, addr, count, dir);
parport_ip32_dma.len = count;
parport_ip32_dma.next = parport_ip32_dma.buf;
parport_ip32_dma.left = parport_ip32_dma.len;
/**
* parport_ip32_dma_stop - ends a running DMA transfer
+ * @p: partport to work on
*
* Calls to parport_ip32_dma_start() and parport_ip32_dma_stop() must be
* correctly balanced.
*/
-static void parport_ip32_dma_stop(void)
+static void parport_ip32_dma_stop(struct parport *p)
{
u64 ctx_a;
u64 ctx_b;
enable_irq(MACEISA_PAR_CTXB_IRQ);
parport_ip32_dma.irq_on = 1;
- dma_unmap_single(NULL, parport_ip32_dma.buf, parport_ip32_dma.len,
- parport_ip32_dma.dir);
+ dma_unmap_single(&p->bus_dev, parport_ip32_dma.buf,
+ parport_ip32_dma.len, parport_ip32_dma.dir);
}
/**
priv->irq_mode = PARPORT_IP32_IRQ_HERE;
- parport_ip32_dma_start(DMA_TO_DEVICE, (void *)buf, len);
+ parport_ip32_dma_start(p, DMA_TO_DEVICE, (void *)buf, len);
reinit_completion(&priv->irq_complete);
parport_ip32_frob_econtrol(p, ECR_DMAEN | ECR_SERVINTR, ECR_DMAEN);
if (ecr & ECR_SERVINTR)
break; /* DMA transfer just finished */
}
- parport_ip32_dma_stop();
+ parport_ip32_dma_stop(p);
written = len - parport_ip32_dma_get_residue();
priv->irq_mode = PARPORT_IP32_IRQ_FWD;
init.name = name;
init.ops = &clk_mux_ops;
- init.flags = CLK_IS_BASIC;
+ init.flags = 0;
init.parent_names = meson->data->parent_names;
init.num_parents = meson->data->num_parents;
* @id: ID of the reset controller in the reset
* controller device
* @refcnt: Number of gets of this reset_control
+ * @acquired: Only one reset_control may be acquired for a given rcdev and id.
* @shared: Is this a shared (1), or an exclusive (0) reset_control?
* @deassert_cnt: Number of times this reset line has been deasserted
* @triggered_count: Number of times this reset line has been reset. Currently
struct list_head list;
unsigned int id;
struct kref refcnt;
+ bool acquired;
bool shared;
bool array;
atomic_t deassert_count;
struct reset_control *rstc[];
};
+static const char *rcdev_name(struct reset_controller_dev *rcdev)
+{
+ if (rcdev->dev)
+ return dev_name(rcdev->dev);
+
+ if (rcdev->of_node)
+ return rcdev->of_node->full_name;
+
+ return NULL;
+}
+
/**
* of_reset_simple_xlate - translate reset_spec to the reset line number
* @rcdev: a pointer to the reset controller device
return ret;
}
+static int reset_control_array_acquire(struct reset_control_array *resets)
+{
+ unsigned int i;
+ int err;
+
+ for (i = 0; i < resets->num_rstcs; i++) {
+ err = reset_control_acquire(resets->rstc[i]);
+ if (err < 0)
+ goto release;
+ }
+
+ return 0;
+
+release:
+ while (i--)
+ reset_control_release(resets->rstc[i]);
+
+ return err;
+}
+
+static void reset_control_array_release(struct reset_control_array *resets)
+{
+ unsigned int i;
+
+ for (i = 0; i < resets->num_rstcs; i++)
+ reset_control_release(resets->rstc[i]);
+}
+
static inline bool reset_control_is_array(struct reset_control *rstc)
{
return rstc->array;
if (atomic_inc_return(&rstc->triggered_count) != 1)
return 0;
+ } else {
+ if (!rstc->acquired)
+ return -EPERM;
}
ret = rstc->rcdev->ops->reset(rstc->rcdev, rstc->id);
*/
if (!rstc->rcdev->ops->assert)
return -ENOTSUPP;
+
+ if (!rstc->acquired) {
+ WARN(1, "reset %s (ID: %u) is not acquired\n",
+ rcdev_name(rstc->rcdev), rstc->id);
+ return -EPERM;
+ }
}
return rstc->rcdev->ops->assert(rstc->rcdev, rstc->id);
if (atomic_inc_return(&rstc->deassert_count) != 1)
return 0;
+ } else {
+ if (!rstc->acquired) {
+ WARN(1, "reset %s (ID: %u) is not acquired\n",
+ rcdev_name(rstc->rcdev), rstc->id);
+ return -EPERM;
+ }
}
/*
}
EXPORT_SYMBOL_GPL(reset_control_status);
+/**
+ * reset_control_acquire() - acquires a reset control for exclusive use
+ * @rstc: reset control
+ *
+ * This is used to explicitly acquire a reset control for exclusive use. Note
+ * that exclusive resets are requested as acquired by default. In order for a
+ * second consumer to be able to control the reset, the first consumer has to
+ * release it first. Typically the easiest way to achieve this is to call the
+ * reset_control_get_exclusive_released() to obtain an instance of the reset
+ * control. Such reset controls are not acquired by default.
+ *
+ * Consumers implementing shared access to an exclusive reset need to follow
+ * a specific protocol in order to work together. Before consumers can change
+ * a reset they must acquire exclusive access using reset_control_acquire().
+ * After they are done operating the reset, they must release exclusive access
+ * with a call to reset_control_release(). Consumers are not granted exclusive
+ * access to the reset as long as another consumer hasn't released a reset.
+ *
+ * See also: reset_control_release()
+ */
+int reset_control_acquire(struct reset_control *rstc)
+{
+ struct reset_control *rc;
+
+ if (!rstc)
+ return 0;
+
+ if (WARN_ON(IS_ERR(rstc)))
+ return -EINVAL;
+
+ if (reset_control_is_array(rstc))
+ return reset_control_array_acquire(rstc_to_array(rstc));
+
+ mutex_lock(&reset_list_mutex);
+
+ if (rstc->acquired) {
+ mutex_unlock(&reset_list_mutex);
+ return 0;
+ }
+
+ list_for_each_entry(rc, &rstc->rcdev->reset_control_head, list) {
+ if (rstc != rc && rstc->id == rc->id) {
+ if (rc->acquired) {
+ mutex_unlock(&reset_list_mutex);
+ return -EBUSY;
+ }
+ }
+ }
+
+ rstc->acquired = true;
+
+ mutex_unlock(&reset_list_mutex);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(reset_control_acquire);
+
+/**
+ * reset_control_release() - releases exclusive access to a reset control
+ * @rstc: reset control
+ *
+ * Releases exclusive access right to a reset control previously obtained by a
+ * call to reset_control_acquire(). Until a consumer calls this function, no
+ * other consumers will be granted exclusive access.
+ *
+ * See also: reset_control_acquire()
+ */
+void reset_control_release(struct reset_control *rstc)
+{
+ if (!rstc || WARN_ON(IS_ERR(rstc)))
+ return;
+
+ if (reset_control_is_array(rstc))
+ reset_control_array_release(rstc_to_array(rstc));
+ else
+ rstc->acquired = false;
+}
+EXPORT_SYMBOL_GPL(reset_control_release);
+
static struct reset_control *__reset_control_get_internal(
struct reset_controller_dev *rcdev,
- unsigned int index, bool shared)
+ unsigned int index, bool shared, bool acquired)
{
struct reset_control *rstc;
list_for_each_entry(rstc, &rcdev->reset_control_head, list) {
if (rstc->id == index) {
+ /*
+ * Allow creating a secondary exclusive reset_control
+ * that is initially not acquired for an already
+ * controlled reset line.
+ */
+ if (!rstc->shared && !shared && !acquired)
+ break;
+
if (WARN_ON(!rstc->shared || !shared))
return ERR_PTR(-EBUSY);
list_add(&rstc->list, &rcdev->reset_control_head);
rstc->id = index;
kref_init(&rstc->refcnt);
+ rstc->acquired = acquired;
rstc->shared = shared;
return rstc;
struct reset_control *__of_reset_control_get(struct device_node *node,
const char *id, int index, bool shared,
- bool optional)
+ bool optional, bool acquired)
{
struct reset_control *rstc;
struct reset_controller_dev *r, *rcdev;
}
/* reset_list_mutex also protects the rcdev's reset_control list */
- rstc = __reset_control_get_internal(rcdev, rstc_id, shared);
+ rstc = __reset_control_get_internal(rcdev, rstc_id, shared, acquired);
out:
mutex_unlock(&reset_list_mutex);
static struct reset_control *
__reset_control_get_from_lookup(struct device *dev, const char *con_id,
- bool shared, bool optional)
+ bool shared, bool optional, bool acquired)
{
const struct reset_control_lookup *lookup;
struct reset_controller_dev *rcdev;
rstc = __reset_control_get_internal(rcdev,
lookup->index,
- shared);
+ shared, acquired);
mutex_unlock(&reset_list_mutex);
break;
}
}
struct reset_control *__reset_control_get(struct device *dev, const char *id,
- int index, bool shared, bool optional)
+ int index, bool shared, bool optional,
+ bool acquired)
{
+ if (WARN_ON(shared && acquired))
+ return ERR_PTR(-EINVAL);
+
if (dev->of_node)
return __of_reset_control_get(dev->of_node, id, index, shared,
- optional);
+ optional, acquired);
- return __reset_control_get_from_lookup(dev, id, shared, optional);
+ return __reset_control_get_from_lookup(dev, id, shared, optional,
+ acquired);
}
EXPORT_SYMBOL_GPL(__reset_control_get);
struct reset_control *__devm_reset_control_get(struct device *dev,
const char *id, int index, bool shared,
- bool optional)
+ bool optional, bool acquired)
{
struct reset_control **ptr, *rstc;
if (!ptr)
return ERR_PTR(-ENOMEM);
- rstc = __reset_control_get(dev, id, index, shared, optional);
+ rstc = __reset_control_get(dev, id, index, shared, optional, acquired);
if (!IS_ERR(rstc)) {
*ptr = rstc;
devres_add(dev, ptr);
struct reset_control *rstc;
int ret;
- rstc = __reset_control_get(dev, NULL, 0, 0, optional);
+ rstc = __reset_control_get(dev, NULL, 0, 0, optional, true);
if (IS_ERR(rstc))
return PTR_ERR(rstc);
* @np: device node for the device that requests the reset controls array
* @shared: whether reset controls are shared or not
* @optional: whether it is optional to get the reset controls
+ * @acquired: only one reset control may be acquired for a given controller
+ * and ID
*
* Returns pointer to allocated reset_control_array on success or
* error on failure
*/
struct reset_control *
-of_reset_control_array_get(struct device_node *np, bool shared, bool optional)
+of_reset_control_array_get(struct device_node *np, bool shared, bool optional,
+ bool acquired)
{
struct reset_control_array *resets;
struct reset_control *rstc;
return ERR_PTR(-ENOMEM);
for (i = 0; i < num; i++) {
- rstc = __of_reset_control_get(np, NULL, i, shared, optional);
+ rstc = __of_reset_control_get(np, NULL, i, shared, optional,
+ acquired);
if (IS_ERR(rstc))
goto err_rst;
resets->rstc[i] = rstc;
if (!devres)
return ERR_PTR(-ENOMEM);
- rstc = of_reset_control_array_get(dev->of_node, shared, optional);
+ rstc = of_reset_control_array_get(dev->of_node, shared, optional, true);
if (IS_ERR(rstc)) {
devres_free(devres);
return rstc;
config RTC_DRV_PCF85063
tristate "NXP PCF85063"
+ select REGMAP_I2C
help
If you say yes here you get support for the PCF85063 RTC chip
config RTC_DRV_PCF85363
tristate "NXP PCF85363"
- depends on I2C
select REGMAP_I2C
help
If you say yes here you get support for the PCF85363 RTC chip.
config RTC_DRV_RX8010
tristate "Epson RX8010SJ"
- depends on I2C
help
If you say yes here you get support for the Epson RX8010SJ RTC
chip.
config RTC_DRV_AT91SAM9
tristate "AT91SAM9 RTT as RTC"
depends on ARCH_AT91 || COMPILE_TEST
- depends on HAS_IOMEM
+ depends on OF && HAS_IOMEM
select MFD_SYSCON
help
Some AT91SAM9 SoCs provide an RTT (Real Time Timer) block which
If you say yes here, you get support for the RTD1295 SoC
Real Time Clock.
+config RTC_DRV_ASPEED
+ tristate "ASPEED RTC"
+ depends on OF
+ depends on ARCH_ASPEED || COMPILE_TEST
+ help
+ If you say yes here you get support for the ASPEED BMC SoC real time
+ clocks.
+
+ This driver can also be built as a module, if so, the module
+ will be called "rtc-aspeed".
+
comment "HID Sensor RTC drivers"
config RTC_DRV_HID_SENSOR_TIME
config RTC_DRV_GOLDFISH
tristate "Goldfish Real Time Clock"
- depends on MIPS && (GOLDFISH || COMPILE_TEST)
+ depends on OF && HAS_IOMEM
+ depends on GOLDFISH || COMPILE_TEST
help
Say yes to enable RTC driver for the Goldfish based virtual platform.
obj-$(CONFIG_RTC_DRV_ARMADA38X) += rtc-armada38x.o
obj-$(CONFIG_RTC_DRV_AS3722) += rtc-as3722.o
obj-$(CONFIG_RTC_DRV_ASM9260) += rtc-asm9260.o
+obj-$(CONFIG_RTC_DRV_ASPEED) += rtc-aspeed.o
obj-$(CONFIG_RTC_DRV_AT91RM9200)+= rtc-at91rm9200.o
obj-$(CONFIG_RTC_DRV_AT91SAM9) += rtc-at91sam9.o
obj-$(CONFIG_RTC_DRV_AU1XXX) += rtc-au1xxx.o
+// SPDX-License-Identifier: GPL-2.0
/*
* RTC subsystem, base class
*
* Author: Alessandro Zummo <a.zummo@towertech.it>
*
* class skeleton from drivers/hwmon/hwmon.c
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+ */
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include "rtc-core.h"
-
static DEFINE_IDA(rtc_ida);
struct class *rtc_class;
static void rtc_device_release(struct device *dev)
{
struct rtc_device *rtc = to_rtc_device(dev);
+
ida_simple_remove(&rtc_ida, rtc->id);
kfree(rtc);
}
static struct timespec64 old_rtc, old_system, old_delta;
-
static int rtc_suspend(struct device *dev)
{
struct rtc_device *rtc = to_rtc_device(dev);
ktime_get_real_ts64(&old_system);
old_rtc.tv_sec = rtc_tm_to_time64(&tm);
-
/*
* To avoid drift caused by repeated suspend/resumes,
* which each can add ~1 second drift error,
if (delta_delta.tv_sec < -2 || delta_delta.tv_sec >= 2) {
/*
* if delta_delta is too large, assume time correction
- * has occured and set old_delta to the current delta.
+ * has occurred and set old_delta to the current delta.
*/
old_delta = delta;
} else {
* to keep things accurate.
*/
sleep_time = timespec64_sub(sleep_time,
- timespec64_sub(new_system, old_system));
+ timespec64_sub(new_system, old_system));
if (sleep_time.tv_sec >= 0)
timekeeping_inject_sleeptime64(&sleep_time);
* rtc_register_device instead
*/
struct rtc_device *devm_rtc_device_register(struct device *dev,
- const char *name,
- const struct rtc_class_ops *ops,
- struct module *owner)
+ const char *name,
+ const struct rtc_class_ops *ops,
+ struct module *owner)
{
struct rtc_device *rtc;
int err;
+// SPDX-License-Identifier: GPL-2.0
/*
* RTC subsystem, dev interface
*
* Author: Alessandro Zummo <a.zummo@towertech.it>
*
* based on arch/arm/common/rtctime.c
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+ */
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
} else if (rtc->oldsecs != tm.tm_sec) {
num = (tm.tm_sec + 60 - rtc->oldsecs) % 60;
rtc->oldsecs = tm.tm_sec;
- rtc->uie_timer.expires = jiffies + HZ - (HZ/10);
+ rtc->uie_timer.expires = jiffies + HZ - (HZ / 10);
rtc->uie_timer_active = 1;
rtc->uie_task_active = 0;
add_timer(&rtc->uie_timer);
if (num)
rtc_handle_legacy_irq(rtc, num, RTC_UF);
}
+
static void rtc_uie_timer(struct timer_list *t)
{
struct rtc_device *rtc = from_timer(rtc, t, uie_timer);
}
static long rtc_dev_ioctl(struct file *file,
- unsigned int cmd, unsigned long arg)
+ unsigned int cmd, unsigned long arg)
{
int err = 0;
struct rtc_device *rtc = file->private_data;
const struct rtc_class_ops *ops = rtc->ops;
struct rtc_time tm;
struct rtc_wkalrm alarm;
- void __user *uarg = (void __user *) arg;
+ void __user *uarg = (void __user *)arg;
err = mutex_lock_interruptible(&rtc->ops_lock);
if (err)
case RTC_PIE_ON:
if (rtc->irq_freq > rtc->max_user_freq &&
- !capable(CAP_SYS_RESOURCE))
+ !capable(CAP_SYS_RESOURCE))
err = -EACCES;
break;
}
err = ops->ioctl(rtc->dev.parent, cmd, arg);
if (err == -ENOIOCTLCMD)
err = -ENOTTY;
- } else
+ } else {
err = -ENOTTY;
+ }
break;
}
static int rtc_dev_fasync(int fd, struct file *file, int on)
{
struct rtc_device *rtc = file->private_data;
+
return fasync_helper(fd, file, on, &rtc->async_queue);
}
+// SPDX-License-Identifier: GPL-2.0
/*
* RTC subsystem, initialize system time on startup
*
* Copyright (C) 2005 Tower Technologies
* Author: Alessandro Zummo <a.zummo@towertech.it>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+ */
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
};
struct rtc_device *rtc = rtc_class_open(CONFIG_RTC_HCTOSYS_DEVICE);
- if (rtc == NULL) {
+ if (!rtc) {
pr_info("unable to open rtc device (%s)\n",
CONFIG_RTC_HCTOSYS_DEVICE);
goto err_open;
dev_err(rtc->dev.parent,
"hctosys: unable to read the hardware clock\n");
goto err_read;
-
}
tv64.tv_sec = rtc_tm_to_time64(&tm);
+// SPDX-License-Identifier: GPL-2.0
/*
* RTC subsystem, interface functions
*
* Author: Alessandro Zummo <a.zummo@towertech.it>
*
* based on arch/arm/common/rtctime.c
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+ */
#include <linux/rtc.h>
#include <linux/sched.h>
static int __rtc_read_time(struct rtc_device *rtc, struct rtc_time *tm)
{
int err;
- if (!rtc->ops)
+
+ if (!rtc->ops) {
err = -ENODEV;
- else if (!rtc->ops->read_time)
+ } else if (!rtc->ops->read_time) {
err = -EINVAL;
- else {
+ } else {
memset(tm, 0, sizeof(struct rtc_time));
err = rtc->ops->read_time(rtc->dev.parent, tm);
if (err < 0) {
err = -ENODEV;
else if (rtc->ops->set_time)
err = rtc->ops->set_time(rtc->dev.parent, tm);
- else if (rtc->ops->set_mmss64) {
- time64_t secs64 = rtc_tm_to_time64(tm);
-
- err = rtc->ops->set_mmss64(rtc->dev.parent, secs64);
- } else if (rtc->ops->set_mmss) {
- time64_t secs64 = rtc_tm_to_time64(tm);
- err = rtc->ops->set_mmss(rtc->dev.parent, secs64);
- } else
+ else
err = -EINVAL;
pm_stay_awake(rtc->dev.parent);
}
EXPORT_SYMBOL_GPL(rtc_set_time);
-static int rtc_read_alarm_internal(struct rtc_device *rtc, struct rtc_wkalrm *alarm)
+static int rtc_read_alarm_internal(struct rtc_device *rtc,
+ struct rtc_wkalrm *alarm)
{
int err;
if (err)
return err;
- if (rtc->ops == NULL)
+ if (!rtc->ops) {
err = -ENODEV;
- else if (!rtc->ops->read_alarm)
+ } else if (!rtc->ops->read_alarm) {
err = -EINVAL;
- else {
+ } else {
alarm->enabled = 0;
alarm->pending = 0;
alarm->time.tm_sec = -1;
int first_time = 1;
time64_t t_now, t_alm;
enum { none, day, month, year } missing = none;
- unsigned days;
+ unsigned int days;
/* The lower level RTC driver may return -1 in some fields,
* creating invalid alarm->time values, for reasons like:
return err;
/* note that tm_sec is a "don't care" value here: */
- } while ( before.tm_min != now.tm_min
- || before.tm_hour != now.tm_hour
- || before.tm_mon != now.tm_mon
- || before.tm_year != now.tm_year);
+ } while (before.tm_min != now.tm_min ||
+ before.tm_hour != now.tm_hour ||
+ before.tm_mon != now.tm_mon ||
+ before.tm_year != now.tm_year);
/* Fill in the missing alarm fields using the timestamp; we
* know there's at least one since alarm->time is invalid.
alarm->time.tm_mday = now.tm_mday;
missing = day;
}
- if ((unsigned)alarm->time.tm_mon >= 12) {
+ if ((unsigned int)alarm->time.tm_mon >= 12) {
alarm->time.tm_mon = now.tm_mon;
if (missing == none)
missing = month;
goto done;
switch (missing) {
-
/* 24 hour rollover ... if it's now 10am Monday, an alarm that
* that will trigger at 5am will do so at 5am Tuesday, which
* could also be in the next month or year. This is a common
case month:
dev_dbg(&rtc->dev, "alarm rollover: %s\n", "month");
do {
- if (alarm->time.tm_mon < 11)
+ if (alarm->time.tm_mon < 11) {
alarm->time.tm_mon++;
- else {
+ } else {
alarm->time.tm_mon = 0;
alarm->time.tm_year++;
}
days = rtc_month_days(alarm->time.tm_mon,
- alarm->time.tm_year);
+ alarm->time.tm_year);
} while (days < alarm->time.tm_mday);
break;
dev_dbg(&rtc->dev, "alarm rollover: %s\n", "year");
do {
alarm->time.tm_year++;
- } while (!is_leap_year(alarm->time.tm_year + 1900)
- && rtc_valid_tm(&alarm->time) != 0);
+ } while (!is_leap_year(alarm->time.tm_year + 1900) &&
+ rtc_valid_tm(&alarm->time) != 0);
break;
default:
done:
if (err)
- dev_warn(&rtc->dev, "invalid alarm value: %ptR\n", &alarm->time);
+ dev_warn(&rtc->dev, "invalid alarm value: %ptR\n",
+ &alarm->time);
return err;
}
err = mutex_lock_interruptible(&rtc->ops_lock);
if (err)
return err;
- if (rtc->ops == NULL)
+ if (!rtc->ops) {
err = -ENODEV;
- else if (!rtc->ops->read_alarm)
+ } else if (!rtc->ops->read_alarm) {
err = -EINVAL;
- else {
+ } else {
memset(alarm, 0, sizeof(struct rtc_wkalrm));
alarm->enabled = rtc->aie_timer.enabled;
alarm->time = rtc_ktime_to_tm(rtc->aie_timer.node.expires);
/* Alarm has to be enabled & in the future for us to enqueue it */
if (alarm->enabled && (rtc_tm_to_ktime(now) <
rtc->aie_timer.node.expires)) {
-
rtc->aie_timer.enabled = 1;
timerqueue_add(&rtc->timerqueue, &rtc->aie_timer.node);
trace_rtc_timer_enqueue(&rtc->aie_timer);
int rtc_alarm_irq_enable(struct rtc_device *rtc, unsigned int enabled)
{
- int err = mutex_lock_interruptible(&rtc->ops_lock);
+ int err;
+
+ err = mutex_lock_interruptible(&rtc->ops_lock);
if (err)
return err;
int rtc_update_irq_enable(struct rtc_device *rtc, unsigned int enabled)
{
- int err = mutex_lock_interruptible(&rtc->ops_lock);
+ int err;
+
+ err = mutex_lock_interruptible(&rtc->ops_lock);
if (err)
return err;
rtc->uie_rtctimer.node.expires = ktime_add(now, onesec);
rtc->uie_rtctimer.period = ktime_set(1, 0);
err = rtc_timer_enqueue(rtc, &rtc->uie_rtctimer);
- } else
+ } else {
rtc_timer_remove(rtc, &rtc->uie_rtctimer);
+ }
out:
mutex_unlock(&rtc->ops_lock);
#ifdef CONFIG_RTC_INTF_DEV_UIE_EMUL
/*
- * Enable emulation if the driver did not provide
- * the update_irq_enable function pointer or if returned
- * -EINVAL to signal that it has been configured without
- * interrupts or that are not available at the moment.
+ * Enable emulation if the driver returned -EINVAL to signal that it has
+ * been configured without interrupts or they are not available at the
+ * moment.
*/
if (err == -EINVAL)
err = rtc_dev_update_irq_enable_emul(rtc, enabled);
#endif
return err;
-
}
EXPORT_SYMBOL_GPL(rtc_update_irq_enable);
-
/**
* rtc_handle_legacy_irq - AIE, UIE and PIE event hook
* @rtc: pointer to the rtc device
/* mark one irq of the appropriate mode */
spin_lock_irqsave(&rtc->irq_lock, flags);
- rtc->irq_data = (rtc->irq_data + (num << 8)) | (RTC_IRQF|mode);
+ rtc->irq_data = (rtc->irq_data + (num << 8)) | (RTC_IRQF | mode);
spin_unlock_irqrestore(&rtc->irq_lock, flags);
wake_up_interruptible(&rtc->irq_queue);
kill_fasync(&rtc->async_queue, SIGIO, POLL_IN);
}
-
/**
* rtc_aie_update_irq - AIE mode rtctimer hook
* @rtc: pointer to the rtc_device
rtc_handle_legacy_irq(rtc, 1, RTC_AF);
}
-
/**
* rtc_uie_update_irq - UIE mode rtctimer hook
* @rtc: pointer to the rtc_device
rtc_handle_legacy_irq(rtc, 1, RTC_UF);
}
-
/**
* rtc_pie_update_irq - PIE mode hrtimer hook
* @timer: pointer to the pie mode hrtimer
struct rtc_device *rtc;
ktime_t period;
int count;
+
rtc = container_of(timer, struct rtc_device, pie_timer);
period = NSEC_PER_SEC / rtc->irq_freq;
* Context: any
*/
void rtc_update_irq(struct rtc_device *rtc,
- unsigned long num, unsigned long events)
+ unsigned long num, unsigned long events)
{
if (IS_ERR_OR_NULL(rtc))
return;
if (!next || ktime_before(timer->node.expires, next->expires)) {
struct rtc_wkalrm alarm;
int err;
+
alarm.time = rtc_ktime_to_tm(timer->node.expires);
alarm.enabled = 1;
err = __rtc_set_alarm(rtc, &alarm);
static void rtc_timer_remove(struct rtc_device *rtc, struct rtc_timer *timer)
{
struct timerqueue_node *next = timerqueue_getnext(&rtc->timerqueue);
+
timerqueue_del(&rtc->timerqueue, &timer->node);
trace_rtc_timer_dequeue(timer);
timer->enabled = 0;
if (next == &timer->node) {
struct rtc_wkalrm alarm;
int err;
+
next = timerqueue_getnext(&rtc->timerqueue);
if (!next) {
rtc_alarm_disable(rtc);
alarm.enabled = 1;
reprogram:
err = __rtc_set_alarm(rtc, &alarm);
- if (err == -ETIME)
+ if (err == -ETIME) {
goto again;
- else if (err) {
+ } else if (err) {
if (retry-- > 0)
goto reprogram;
dev_err(&rtc->dev, "__rtc_set_alarm: err=%d\n", err);
goto again;
}
- } else
+ } else {
rtc_alarm_disable(rtc);
+ }
pm_relax(rtc->dev.parent);
mutex_unlock(&rtc->ops_lock);
}
-
/* rtc_timer_init - Initializes an rtc_timer
* @timer: timer to be intiialized
* @f: function pointer to be called when timer fires
* Kernel interface to set an rtc_timer
*/
int rtc_timer_start(struct rtc_device *rtc, struct rtc_timer *timer,
- ktime_t expires, ktime_t period)
+ ktime_t expires, ktime_t period)
{
int ret = 0;
+
mutex_lock(&rtc->ops_lock);
if (timer->enabled)
rtc_timer_remove(rtc, timer);
+// SPDX-License-Identifier: GPL-2.0
/*
* rtc and date/time utility functions
*
* Author: Alessandro Zummo <a.zummo@towertech.it>
*
* based on arch/arm/common/rtctime.c and other bits
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+ */
#include <linux/export.h>
#include <linux/rtc.h>
{ 0, 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366 }
};
-#define LEAPS_THRU_END_OF(y) ((y)/4 - (y)/100 + (y)/400)
+#define LEAPS_THRU_END_OF(y) ((y) / 4 - (y) / 100 + (y) / 400)
/*
* The number of days in the month.
*/
int rtc_year_days(unsigned int day, unsigned int month, unsigned int year)
{
- return rtc_ydays[is_leap_year(year)][month] + day-1;
+ return rtc_ydays[is_leap_year(year)][month] + day - 1;
}
EXPORT_SYMBOL(rtc_year_days);
-
/*
* rtc_time64_to_tm - Converts time64_t to rtc_time.
* Convert seconds since 01-01-1970 00:00:00 to Gregorian date.
*/
int rtc_valid_tm(struct rtc_time *tm)
{
- if (tm->tm_year < 70
- || ((unsigned)tm->tm_mon) >= 12
- || tm->tm_mday < 1
- || tm->tm_mday > rtc_month_days(tm->tm_mon, ((unsigned)tm->tm_year + 1900))
- || ((unsigned)tm->tm_hour) >= 24
- || ((unsigned)tm->tm_min) >= 60
- || ((unsigned)tm->tm_sec) >= 60)
+ if (tm->tm_year < 70 ||
+ tm->tm_year > (INT_MAX - 1900) ||
+ ((unsigned int)tm->tm_mon) >= 12 ||
+ tm->tm_mday < 1 ||
+ tm->tm_mday > rtc_month_days(tm->tm_mon,
+ ((unsigned int)tm->tm_year + 1900)) ||
+ ((unsigned int)tm->tm_hour) >= 24 ||
+ ((unsigned int)tm->tm_min) >= 60 ||
+ ((unsigned int)tm->tm_sec) >= 60)
return -EINVAL;
return 0;
*/
time64_t rtc_tm_to_time64(struct rtc_time *tm)
{
- return mktime64(((unsigned)tm->tm_year + 1900), tm->tm_mon + 1,
+ return mktime64(((unsigned int)tm->tm_year + 1900), tm->tm_mon + 1,
tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec);
}
EXPORT_SYMBOL(rtc_tm_to_time64);
+// SPDX-License-Identifier: GPL-2.0
/*
* RTC subsystem, nvmem interface
*
* Copyright (C) 2017 Alexandre Belloni
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/err.h>
{
int err;
- rtc->nvram = kzalloc(sizeof(struct bin_attribute), GFP_KERNEL);
+ rtc->nvram = kzalloc(sizeof(*rtc->nvram), GFP_KERNEL);
if (!rtc->nvram)
return -ENOMEM;
+// SPDX-License-Identifier: GPL-2.0
/*
* RTC subsystem, proc interface
*
* Author: Alessandro Zummo <a.zummo@towertech.it>
*
* based on arch/arm/common/rtctime.c
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+ */
#include <linux/module.h>
#include <linux/rtc.h>
seq_printf(seq, "alrm_time\t: %ptRt\n", &alrm.time);
seq_printf(seq, "alrm_date\t: %ptRd\n", &alrm.time);
seq_printf(seq, "alarm_IRQ\t: %s\n",
- alrm.enabled ? "yes" : "no");
+ alrm.enabled ? "yes" : "no");
seq_printf(seq, "alrm_pending\t: %s\n",
- alrm.pending ? "yes" : "no");
+ alrm.pending ? "yes" : "no");
seq_printf(seq, "update IRQ enabled\t: %s\n",
- (rtc->uie_rtctimer.enabled) ? "yes" : "no");
+ (rtc->uie_rtctimer.enabled) ? "yes" : "no");
seq_printf(seq, "periodic IRQ enabled\t: %s\n",
- (rtc->pie_enabled) ? "yes" : "no");
+ (rtc->pie_enabled) ? "yes" : "no");
seq_printf(seq, "periodic IRQ frequency\t: %d\n",
- rtc->irq_freq);
+ rtc->irq_freq);
seq_printf(seq, "max user IRQ frequency\t: %d\n",
- rtc->max_user_freq);
+ rtc->max_user_freq);
}
seq_printf(seq, "24hr\t\t: yes\n");
{
if (is_rtc_hctosys(rtc))
proc_create_single_data("driver/rtc", 0, NULL, rtc_proc_show,
- rtc);
+ rtc);
}
void rtc_proc_del_device(struct rtc_device *rtc)
+// SPDX-License-Identifier: GPL-2.0
/*
* Real Time Clock driver for Marvell 88PM80x PMIC
*
* Copyright (c) 2012 Marvell International Ltd.
* Wenzeng Chen<wzch@marvell.com>
* Qiao Zhou <zhouqiao@marvell.com>
- *
- * This file is subject to the terms and conditions of the GNU General
- * Public License. See the file "COPYING" in the main directory of this
- * archive for more details.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/
#include <linux/kernel.h>
struct pm860x_rtc_info *info = platform_get_drvdata(pdev);
#ifdef VRTC_CALIBRATION
- flush_scheduled_work();
+ cancel_delayed_work_sync(&info->calib_work);
/* disable measurement */
pm860x_set_bits(info->i2c, PM8607_MEAS_EN2, MEAS2_VRTC, 0);
#endif /* VRTC_CALIBRATION */
+// SPDX-License-Identifier: GPL-2.0+
/*
* rtc-ab-b5ze-s3 - Driver for Abracon AB-RTCMC-32.768Khz-B5ZE-S3
* I2C RTC / Alarm chip
*
* This work is based on ISL12057 driver (drivers/rtc/rtc-isl12057.c).
*
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
*/
#include <linux/module.h>
-#include <linux/mutex.h>
#include <linux/rtc.h>
#include <linux/i2c.h>
#include <linux/bcd.h>
struct abb5zes3_rtc_data {
struct rtc_device *rtc;
struct regmap *regmap;
- struct mutex lock;
int irq;
/*
* Try and match register bits w/ fixed null values to see whether we
- * are dealing with an ABB5ZES3. Note: this function is called early
- * during init and hence does need mutex protection.
+ * are dealing with an ABB5ZES3.
*/
static int abb5zes3_i2c_validate_chip(struct regmap *regmap)
{
if (ret) {
dev_err(dev, "%s: reading RTC time failed (%d)\n",
__func__, ret);
- goto err;
+ return ret;
}
/* If clock integrity is not guaranteed, do not return a time value */
- if (regs[ABB5ZES3_REG_RTC_SC] & ABB5ZES3_REG_RTC_SC_OSC) {
- ret = -ENODATA;
- goto err;
- }
+ if (regs[ABB5ZES3_REG_RTC_SC] & ABB5ZES3_REG_RTC_SC_OSC)
+ return -ENODATA;
tm->tm_sec = bcd2bin(regs[ABB5ZES3_REG_RTC_SC] & 0x7F);
tm->tm_min = bcd2bin(regs[ABB5ZES3_REG_RTC_MN]);
tm->tm_mon = bcd2bin(regs[ABB5ZES3_REG_RTC_MO]) - 1; /* starts at 1 */
tm->tm_year = bcd2bin(regs[ABB5ZES3_REG_RTC_YR]) + 100;
-err:
return ret;
}
regs[ABB5ZES3_REG_RTC_MO] = bin2bcd(tm->tm_mon + 1);
regs[ABB5ZES3_REG_RTC_YR] = bin2bcd(tm->tm_year - 100);
- mutex_lock(&data->lock);
ret = regmap_bulk_write(data->regmap, ABB5ZES3_REG_RTC_SC,
regs + ABB5ZES3_REG_RTC_SC,
ABB5ZES3_RTC_SEC_LEN);
- mutex_unlock(&data->lock);
-
return ret;
}
if (ret) {
dev_err(dev, "%s: reading Timer A section failed (%d)\n",
__func__, ret);
- goto err;
+ return ret;
}
/* get current time ... */
ret = _abb5zes3_rtc_read_time(dev, &rtc_tm);
if (ret)
- goto err;
+ return ret;
/* ... convert to seconds ... */
- ret = rtc_tm_to_time(&rtc_tm, &rtc_secs);
- if (ret)
- goto err;
+ rtc_secs = rtc_tm_to_time64(&rtc_tm);
/* ... add remaining timer A time ... */
ret = sec_from_timer_a(&timer_secs, regs[1], regs[2]);
if (ret)
- goto err;
+ return ret;
/* ... and convert back. */
- rtc_time_to_tm(rtc_secs + timer_secs, alarm_tm);
+ rtc_time64_to_tm(rtc_secs + timer_secs, alarm_tm);
ret = regmap_read(data->regmap, ABB5ZES3_REG_CTRL2, ®);
if (ret) {
dev_err(dev, "%s: reading ctrl reg failed (%d)\n",
__func__, ret);
- goto err;
+ return ret;
}
alarm->enabled = !!(reg & ABB5ZES3_REG_CTRL2_WTAIE);
-err:
- return ret;
+ return 0;
}
/* Read alarm currently configured via a RTC alarm registers. */
if (ret) {
dev_err(dev, "%s: reading alarm section failed (%d)\n",
__func__, ret);
- goto err;
+ return ret;
}
alarm_tm->tm_sec = 0;
*/
ret = _abb5zes3_rtc_read_time(dev, &rtc_tm);
if (ret)
- goto err;
+ return ret;
alarm_tm->tm_year = rtc_tm.tm_year;
alarm_tm->tm_mon = rtc_tm.tm_mon;
- ret = rtc_tm_to_time(&rtc_tm, &rtc_secs);
- if (ret)
- goto err;
-
- ret = rtc_tm_to_time(alarm_tm, &alarm_secs);
- if (ret)
- goto err;
+ rtc_secs = rtc_tm_to_time64(&rtc_tm);
+ alarm_secs = rtc_tm_to_time64(alarm_tm);
if (alarm_secs < rtc_secs) {
if (alarm_tm->tm_mon == 11) {
if (ret) {
dev_err(dev, "%s: reading ctrl reg failed (%d)\n",
__func__, ret);
- goto err;
+ return ret;
}
alarm->enabled = !!(reg & ABB5ZES3_REG_CTRL1_AIE);
-err:
- return ret;
+ return 0;
}
/*
struct abb5zes3_rtc_data *data = dev_get_drvdata(dev);
int ret;
- mutex_lock(&data->lock);
if (data->timer_alarm)
ret = _abb5zes3_rtc_read_timer(dev, alarm);
else
ret = _abb5zes3_rtc_read_alarm(dev, alarm);
- mutex_unlock(&data->lock);
return ret;
}
{
struct abb5zes3_rtc_data *data = dev_get_drvdata(dev);
struct rtc_time *alarm_tm = &alarm->time;
- unsigned long rtc_secs, alarm_secs;
u8 regs[ABB5ZES3_ALRM_SEC_LEN];
struct rtc_time rtc_tm;
int ret, enable = 1;
- ret = _abb5zes3_rtc_read_time(dev, &rtc_tm);
- if (ret)
- goto err;
-
- ret = rtc_tm_to_time(&rtc_tm, &rtc_secs);
- if (ret)
- goto err;
-
- ret = rtc_tm_to_time(alarm_tm, &alarm_secs);
- if (ret)
- goto err;
-
- /* If alarm time is before current time, disable the alarm */
- if (!alarm->enabled || alarm_secs <= rtc_secs) {
+ if (!alarm->enabled) {
enable = 0;
} else {
+ unsigned long rtc_secs, alarm_secs;
+
/*
* Chip only support alarms up to one month in the future. Let's
* return an error if we get something after that limit.
* Comparison is done by incrementing rtc_tm month field by one
* and checking alarm value is still below.
*/
+ ret = _abb5zes3_rtc_read_time(dev, &rtc_tm);
+ if (ret)
+ return ret;
+
if (rtc_tm.tm_mon == 11) { /* handle year wrapping */
rtc_tm.tm_mon = 0;
rtc_tm.tm_year += 1;
rtc_tm.tm_mon += 1;
}
- ret = rtc_tm_to_time(&rtc_tm, &rtc_secs);
- if (ret)
- goto err;
+ rtc_secs = rtc_tm_to_time64(&rtc_tm);
+ alarm_secs = rtc_tm_to_time64(alarm_tm);
if (alarm_secs > rtc_secs) {
- dev_err(dev, "%s: alarm maximum is one month in the "
- "future (%d)\n", __func__, ret);
- ret = -EINVAL;
- goto err;
+ dev_err(dev, "%s: alarm maximum is one month in the future (%d)\n",
+ __func__, ret);
+ return -EINVAL;
}
}
if (ret < 0) {
dev_err(dev, "%s: writing ALARM section failed (%d)\n",
__func__, ret);
- goto err;
+ return ret;
}
/* Record currently configured alarm is not a timer */
data->timer_alarm = 0;
/* Enable or disable alarm interrupt generation */
- ret = _abb5zes3_rtc_update_alarm(dev, enable);
-
-err:
- return ret;
+ return _abb5zes3_rtc_update_alarm(dev, enable);
}
/*
ABB5ZES3_TIMA_SEC_LEN);
if (ret < 0) {
dev_err(dev, "%s: writing timer section failed\n", __func__);
- goto err;
+ return ret;
}
/* Configure Timer A as a watchdog timer */
data->timer_alarm = 1;
/* Enable or disable timer interrupt generation */
- ret = _abb5zes3_rtc_update_timer(dev, alarm->enabled);
-
-err:
- return ret;
+ return _abb5zes3_rtc_update_timer(dev, alarm->enabled);
}
/*
struct rtc_time rtc_tm;
int ret;
- mutex_lock(&data->lock);
ret = _abb5zes3_rtc_read_time(dev, &rtc_tm);
if (ret)
- goto err;
+ return ret;
- ret = rtc_tm_to_time(&rtc_tm, &rtc_secs);
- if (ret)
- goto err;
-
- ret = rtc_tm_to_time(alarm_tm, &alarm_secs);
- if (ret)
- goto err;
+ rtc_secs = rtc_tm_to_time64(&rtc_tm);
+ alarm_secs = rtc_tm_to_time64(alarm_tm);
/* Let's first disable both the alarm and the timer interrupts */
ret = _abb5zes3_rtc_update_alarm(dev, false);
if (ret < 0) {
dev_err(dev, "%s: unable to disable alarm (%d)\n", __func__,
ret);
- goto err;
+ return ret;
}
ret = _abb5zes3_rtc_update_timer(dev, false);
if (ret < 0) {
dev_err(dev, "%s: unable to disable timer (%d)\n", __func__,
ret);
- goto err;
+ return ret;
}
data->timer_alarm = 0;
else
ret = _abb5zes3_rtc_set_alarm(dev, alarm);
- err:
- mutex_unlock(&data->lock);
-
if (ret)
dev_err(dev, "%s: unable to configure alarm (%d)\n", __func__,
ret);
/*
* Check current RTC status and enable/disable what needs to be. Return 0 if
- * everything went ok and a negative value upon error. Note: this function
- * is called early during init and hence does need mutex protection.
+ * everything went ok and a negative value upon error.
*/
static int abb5zes3_rtc_check_setup(struct device *dev)
{
ABB5ZES3_REG_TIM_CLK_COF1 | ABB5ZES3_REG_TIM_CLK_COF2 |
ABB5ZES3_REG_TIM_CLK_TBM | ABB5ZES3_REG_TIM_CLK_TAM);
ret = regmap_update_bits(regmap, ABB5ZES3_REG_TIM_CLK, mask,
- ABB5ZES3_REG_TIM_CLK_COF0 | ABB5ZES3_REG_TIM_CLK_COF1 |
- ABB5ZES3_REG_TIM_CLK_COF2);
+ ABB5ZES3_REG_TIM_CLK_COF0 |
+ ABB5ZES3_REG_TIM_CLK_COF1 |
+ ABB5ZES3_REG_TIM_CLK_COF2);
if (ret < 0) {
dev_err(dev, "%s: unable to initialize clkout register (%d)\n",
__func__, ret);
* switchover flag but not battery low flag. The latter is checked
* later below.
*/
- mask = (ABB5ZES3_REG_CTRL3_PM0 | ABB5ZES3_REG_CTRL3_PM1 |
- ABB5ZES3_REG_CTRL3_PM2 | ABB5ZES3_REG_CTRL3_BLIE |
- ABB5ZES3_REG_CTRL3_BSIE| ABB5ZES3_REG_CTRL3_BSF);
+ mask = (ABB5ZES3_REG_CTRL3_PM0 | ABB5ZES3_REG_CTRL3_PM1 |
+ ABB5ZES3_REG_CTRL3_PM2 | ABB5ZES3_REG_CTRL3_BLIE |
+ ABB5ZES3_REG_CTRL3_BSIE | ABB5ZES3_REG_CTRL3_BSF);
ret = regmap_update_bits(regmap, ABB5ZES3_REG_CTRL3, mask, 0);
if (ret < 0) {
dev_err(dev, "%s: unable to initialize CTRL3 register (%d)\n",
}
if (reg & ABB5ZES3_REG_RTC_SC_OSC) {
- dev_err(dev, "clock integrity not guaranteed. Osc. has stopped "
- "or has been interrupted.\n");
- dev_err(dev, "change battery (if not already done) and "
- "then set time to reset osc. failure flag.\n");
+ dev_err(dev, "clock integrity not guaranteed. Osc. has stopped or has been interrupted.\n");
+ dev_err(dev, "change battery (if not already done) and then set time to reset osc. failure flag.\n");
}
/*
data->battery_low = reg & ABB5ZES3_REG_CTRL3_BLF;
if (data->battery_low) {
- dev_err(dev, "RTC battery is low; please, consider "
- "changing it!\n");
+ dev_err(dev, "RTC battery is low; please, consider changing it!\n");
ret = _abb5zes3_rtc_battery_low_irq_enable(regmap, false);
if (ret)
- dev_err(dev, "%s: disabling battery low interrupt "
- "generation failed (%d)\n", __func__, ret);
+ dev_err(dev, "%s: disabling battery low interrupt generation failed (%d)\n",
+ __func__, ret);
}
return ret;
int ret = 0;
if (rtc_data->irq) {
- mutex_lock(&rtc_data->lock);
if (rtc_data->timer_alarm)
ret = _abb5zes3_rtc_update_timer(dev, enable);
else
ret = _abb5zes3_rtc_update_alarm(dev, enable);
- mutex_unlock(&rtc_data->lock);
}
return ret;
if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C |
I2C_FUNC_SMBUS_BYTE_DATA |
- I2C_FUNC_SMBUS_I2C_BLOCK)) {
- ret = -ENODEV;
- goto err;
- }
+ I2C_FUNC_SMBUS_I2C_BLOCK))
+ return -ENODEV;
regmap = devm_regmap_init_i2c(client, &abb5zes3_rtc_regmap_config);
if (IS_ERR(regmap)) {
ret = PTR_ERR(regmap);
dev_err(dev, "%s: regmap allocation failed: %d\n",
__func__, ret);
- goto err;
+ return ret;
}
ret = abb5zes3_i2c_validate_chip(regmap);
if (ret)
- goto err;
+ return ret;
data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
- if (!data) {
- ret = -ENOMEM;
- goto err;
- }
+ if (!data)
+ return -ENOMEM;
- mutex_init(&data->lock);
data->regmap = regmap;
dev_set_drvdata(dev, data);
ret = abb5zes3_rtc_check_setup(dev);
if (ret)
- goto err;
+ return ret;
data->rtc = devm_rtc_allocate_device(dev);
ret = PTR_ERR_OR_ZERO(data->rtc);
if (ret) {
dev_err(dev, "%s: unable to allocate RTC device (%d)\n",
__func__, ret);
- goto err;
+ return ret;
}
if (client->irq > 0) {
ret = devm_request_threaded_irq(dev, client->irq, NULL,
_abb5zes3_rtc_interrupt,
- IRQF_SHARED|IRQF_ONESHOT,
+ IRQF_SHARED | IRQF_ONESHOT,
DRV_NAME, client);
if (!ret) {
device_init_wakeup(dev, true);
if (!data->battery_low && data->irq) {
ret = _abb5zes3_rtc_battery_low_irq_enable(regmap, true);
if (ret) {
- dev_err(dev, "%s: enabling battery low interrupt "
- "generation failed (%d)\n", __func__, ret);
+ dev_err(dev, "%s: enabling battery low interrupt generation failed (%d)\n",
+ __func__, ret);
goto err;
}
}
ret = rtc_register_device(data->rtc);
err:
- if (ret && data && data->irq)
+ if (ret && data->irq)
device_init_wakeup(dev, false);
return ret;
}
+// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2007-2009 ST-Ericsson AB
- * License terms: GNU General Public License (GPL) version 2
* RTC clock driver for the AB3100 Analog Baseband Chip
* Author: Linus Walleij <linus.walleij@stericsson.com>
*/
/*
* RTC clock functions and device struct declaration
*/
-static int ab3100_rtc_set_mmss(struct device *dev, time64_t secs)
+static int ab3100_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
u8 regs[] = {AB3100_TI0, AB3100_TI1, AB3100_TI2,
AB3100_TI3, AB3100_TI4, AB3100_TI5};
unsigned char buf[6];
- u64 hw_counter = secs * AB3100_RTC_CLOCK_RATE * 2;
+ u64 hw_counter = rtc_tm_to_time64(tm) * AB3100_RTC_CLOCK_RATE * 2;
int err = 0;
int i;
static const struct rtc_class_ops ab3100_rtc_ops = {
.read_time = ab3100_rtc_read_time,
- .set_mmss64 = ab3100_rtc_set_mmss,
+ .set_time = ab3100_rtc_set_time,
.read_alarm = ab3100_rtc_read_alarm,
.set_alarm = ab3100_rtc_set_alarm,
.alarm_irq_enable = ab3100_rtc_irq_enable,
/* Ignore any error on this write */
}
- rtc = devm_rtc_device_register(&pdev->dev, "ab3100-rtc",
- &ab3100_rtc_ops, THIS_MODULE);
- if (IS_ERR(rtc)) {
- err = PTR_ERR(rtc);
- return err;
- }
+ rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(rtc))
+ return PTR_ERR(rtc);
+
+ rtc->ops = &ab3100_rtc_ops;
+ /* 48bit counter at (AB3100_RTC_CLOCK_RATE * 2) */
+ rtc->range_max = U32_MAX;
+
platform_set_drvdata(pdev, rtc);
- return 0;
+ return rtc_register_device(rtc);
}
static struct platform_driver ab3100_rtc_driver = {
+// SPDX-License-Identifier: GPL-2.0
/*
* A driver for the I2C members of the Abracon AB x8xx RTC family,
* and compatible: AB 1805 and AB 0805
* Author: Philippe De Muyter <phdm@macqel.be>
* Author: Alexandre Belloni <alexandre.belloni@bootlin.com>
*
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
*/
#include <linux/bcd.h>
return -EINVAL;
}
- retval = abx80x_rtc_set_autocalibration(dev, autocalibration);
+ retval = abx80x_rtc_set_autocalibration(dev->parent, autocalibration);
return retval ? retval : count;
}
{
int autocalibration = 0;
- autocalibration = abx80x_rtc_get_autocalibration(dev);
+ autocalibration = abx80x_rtc_get_autocalibration(dev->parent);
if (autocalibration < 0) {
dev_err(dev, "Failed to read RTC autocalibration\n");
sprintf(buf, "0\n");
struct device_attribute *attr,
const char *buf, size_t count)
{
- struct i2c_client *client = to_i2c_client(dev);
+ struct i2c_client *client = to_i2c_client(dev->parent);
int retval, flags, rc_mode = 0;
if (strncmp(buf, "rc", 2) == 0) {
struct device_attribute *attr, char *buf)
{
int rc_mode = 0;
- struct i2c_client *client = to_i2c_client(dev);
+ struct i2c_client *client = to_i2c_client(dev->parent);
rc_mode = abx80x_is_rc_mode(client);
return (trickle_cfg | i);
}
-static void rtc_calib_remove_sysfs_group(void *_dev)
-{
- struct device *dev = _dev;
-
- sysfs_remove_group(&dev->kobj, &rtc_calib_attr_group);
-}
-
#ifdef CONFIG_WATCHDOG
static inline u8 timeout_bits(unsigned int timeout)
}
}
- /* Export sysfs entries */
- err = sysfs_create_group(&(&client->dev)->kobj, &rtc_calib_attr_group);
+ err = rtc_add_group(priv->rtc, &rtc_calib_attr_group);
if (err) {
dev_err(&client->dev, "Failed to create sysfs group: %d\n",
err);
return err;
}
- err = devm_add_action_or_reset(&client->dev,
- rtc_calib_remove_sysfs_group,
- &client->dev);
- if (err) {
- dev_err(&client->dev,
- "Failed to add sysfs cleanup action: %d\n",
- err);
- return err;
- }
-
- err = rtc_register_device(priv->rtc);
-
- return err;
-}
-
-static int abx80x_remove(struct i2c_client *client)
-{
- return 0;
+ return rtc_register_device(priv->rtc);
}
static const struct i2c_device_id abx80x_id[] = {
.name = "rtc-abx80x",
},
.probe = abx80x_probe,
- .remove = abx80x_remove,
.id_table = abx80x_id,
};
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+// Copyright 2015 IBM Corp.
+
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/rtc.h>
+#include <linux/io.h>
+
+struct aspeed_rtc {
+ struct rtc_device *rtc_dev;
+ void __iomem *base;
+};
+
+#define RTC_TIME 0x00
+#define RTC_YEAR 0x04
+#define RTC_CTRL 0x10
+
+#define RTC_UNLOCK BIT(1)
+#define RTC_ENABLE BIT(0)
+
+static int aspeed_rtc_read_time(struct device *dev, struct rtc_time *tm)
+{
+ struct aspeed_rtc *rtc = dev_get_drvdata(dev);
+ unsigned int cent, year;
+ u32 reg1, reg2;
+
+ if (!(readl(rtc->base + RTC_CTRL) & RTC_ENABLE)) {
+ dev_dbg(dev, "%s failing as rtc disabled\n", __func__);
+ return -EINVAL;
+ }
+
+ do {
+ reg2 = readl(rtc->base + RTC_YEAR);
+ reg1 = readl(rtc->base + RTC_TIME);
+ } while (reg2 != readl(rtc->base + RTC_YEAR));
+
+ tm->tm_mday = (reg1 >> 24) & 0x1f;
+ tm->tm_hour = (reg1 >> 16) & 0x1f;
+ tm->tm_min = (reg1 >> 8) & 0x3f;
+ tm->tm_sec = (reg1 >> 0) & 0x3f;
+
+ cent = (reg2 >> 16) & 0x1f;
+ year = (reg2 >> 8) & 0x7f;
+ tm->tm_mon = ((reg2 >> 0) & 0x0f) - 1;
+ tm->tm_year = year + (cent * 100) - 1900;
+
+ dev_dbg(dev, "%s %ptR", __func__, tm);
+
+ return 0;
+}
+
+static int aspeed_rtc_set_time(struct device *dev, struct rtc_time *tm)
+{
+ struct aspeed_rtc *rtc = dev_get_drvdata(dev);
+ u32 reg1, reg2, ctrl;
+ int year, cent;
+
+ cent = (tm->tm_year + 1900) / 100;
+ year = tm->tm_year % 100;
+
+ reg1 = (tm->tm_mday << 24) | (tm->tm_hour << 16) | (tm->tm_min << 8) |
+ tm->tm_sec;
+
+ reg2 = ((cent & 0x1f) << 16) | ((year & 0x7f) << 8) |
+ ((tm->tm_mon + 1) & 0xf);
+
+ ctrl = readl(rtc->base + RTC_CTRL);
+ writel(ctrl | RTC_UNLOCK, rtc->base + RTC_CTRL);
+
+ writel(reg1, rtc->base + RTC_TIME);
+ writel(reg2, rtc->base + RTC_YEAR);
+
+ /* Re-lock and ensure enable is set now that a time is programmed */
+ writel(ctrl | RTC_ENABLE, rtc->base + RTC_CTRL);
+
+ return 0;
+}
+
+static const struct rtc_class_ops aspeed_rtc_ops = {
+ .read_time = aspeed_rtc_read_time,
+ .set_time = aspeed_rtc_set_time,
+};
+
+static int aspeed_rtc_probe(struct platform_device *pdev)
+{
+ struct aspeed_rtc *rtc;
+ struct resource *res;
+ int ret;
+
+ rtc = devm_kzalloc(&pdev->dev, sizeof(*rtc), GFP_KERNEL);
+ if (!rtc)
+ return -ENOMEM;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ rtc->base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(rtc->base))
+ return PTR_ERR(rtc->base);
+
+ rtc->rtc_dev = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(rtc->rtc_dev))
+ return PTR_ERR(rtc->rtc_dev);
+
+ platform_set_drvdata(pdev, rtc);
+
+ rtc->rtc_dev->ops = &aspeed_rtc_ops;
+ rtc->rtc_dev->range_min = RTC_TIMESTAMP_BEGIN_1900;
+ rtc->rtc_dev->range_max = 38814989399LL; /* 3199-12-31 23:59:59 */
+
+ ret = rtc_register_device(rtc->rtc_dev);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static const struct of_device_id aspeed_rtc_match[] = {
+ { .compatible = "aspeed,ast2400-rtc", },
+ { .compatible = "aspeed,ast2500-rtc", },
+ { .compatible = "aspeed,ast2600-rtc", },
+ {}
+};
+MODULE_DEVICE_TABLE(of, aspeed_rtc_match);
+
+static struct platform_driver aspeed_rtc_driver = {
+ .driver = {
+ .name = "aspeed-rtc",
+ .of_match_table = of_match_ptr(aspeed_rtc_match),
+ },
+};
+
+module_platform_driver_probe(aspeed_rtc_driver, aspeed_rtc_probe);
+
+MODULE_DESCRIPTION("ASPEED RTC driver");
+MODULE_AUTHOR("Joel Stanley <joel@jms.id.au>");
+MODULE_LICENSE("GPL");
+// SPDX-License-Identifier: GPL-2.0+
/*
* "RTT as Real Time Clock" driver for AT91SAM9 SoC family
*
* (C) 2007 Michel Benoit
*
* Based on rtc-at91rm9200.c by Rick Bronson
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version
- * 2 of the License, or (at your option) any later version.
*/
#include <linux/clk.h>
* registers available, likewise usable for more than "RTC" support.
*/
-#define AT91_RTT_MR 0x00 /* Real-time Mode Register */
-#define AT91_RTT_RTPRES (0xffff << 0) /* Real-time Timer Prescaler Value */
-#define AT91_RTT_ALMIEN (1 << 16) /* Alarm Interrupt Enable */
-#define AT91_RTT_RTTINCIEN (1 << 17) /* Real Time Timer Increment Interrupt Enable */
-#define AT91_RTT_RTTRST (1 << 18) /* Real Time Timer Restart */
+#define AT91_RTT_MR 0x00 /* Real-time Mode Register */
+#define AT91_RTT_RTPRES (0xffff << 0) /* Timer Prescaler Value */
+#define AT91_RTT_ALMIEN BIT(16) /* Alarm Interrupt Enable */
+#define AT91_RTT_RTTINCIEN BIT(17) /* Increment Interrupt Enable */
+#define AT91_RTT_RTTRST BIT(18) /* Timer Restart */
-#define AT91_RTT_AR 0x04 /* Real-time Alarm Register */
-#define AT91_RTT_ALMV (0xffffffff) /* Alarm Value */
+#define AT91_RTT_AR 0x04 /* Real-time Alarm Register */
+#define AT91_RTT_ALMV (0xffffffff) /* Alarm Value */
-#define AT91_RTT_VR 0x08 /* Real-time Value Register */
-#define AT91_RTT_CRTV (0xffffffff) /* Current Real-time Value */
+#define AT91_RTT_VR 0x08 /* Real-time Value Register */
+#define AT91_RTT_CRTV (0xffffffff) /* Current Real-time Value */
-#define AT91_RTT_SR 0x0c /* Real-time Status Register */
-#define AT91_RTT_ALMS (1 << 0) /* Real-time Alarm Status */
-#define AT91_RTT_RTTINC (1 << 1) /* Real-time Timer Increment */
+#define AT91_RTT_SR 0x0c /* Real-time Status Register */
+#define AT91_RTT_ALMS BIT(0) /* Alarm Status */
+#define AT91_RTT_RTTINC BIT(1) /* Timer Increment */
/*
* We store ALARM_DISABLED in ALMV to record that no alarm is set.
*/
#define ALARM_DISABLED ((u32)~0)
-
struct sam9_rtc {
void __iomem *rtt;
struct rtc_device *rtcdev;
u32 imr;
struct regmap *gpbr;
unsigned int gpbr_offset;
- int irq;
+ int irq;
struct clk *sclk;
bool suspended;
unsigned long events;
if (secs != secs2)
secs = rtt_readl(rtc, VR);
- rtc_time_to_tm(offset + secs, tm);
+ rtc_time64_to_tm(offset + secs, tm);
dev_dbg(dev, "%s: %ptR\n", __func__, tm);
static int at91_rtc_settime(struct device *dev, struct rtc_time *tm)
{
struct sam9_rtc *rtc = dev_get_drvdata(dev);
- int err;
u32 offset, alarm, mr;
unsigned long secs;
dev_dbg(dev, "%s: %ptR\n", __func__, tm);
- err = rtc_tm_to_time(tm, &secs);
- if (err != 0)
- return err;
+ secs = rtc_tm_to_time64(tm);
mr = rtt_readl(rtc, MR);
memset(alrm, 0, sizeof(*alrm));
if (alarm != ALARM_DISABLED && offset != 0) {
- rtc_time_to_tm(offset + alarm, tm);
+ rtc_time64_to_tm(offset + alarm, tm);
dev_dbg(dev, "%s: %ptR\n", __func__, tm);
unsigned long secs;
u32 offset;
u32 mr;
- int err;
- err = rtc_tm_to_time(tm, &secs);
- if (err != 0)
- return err;
+ secs = rtc_tm_to_time64(tm);
offset = gpbr_readl(rtc);
if (offset == 0) {
u32 mr = rtt_readl(rtc, MR);
seq_printf(seq, "update_IRQ\t: %s\n",
- (mr & AT91_RTT_RTTINCIEN) ? "yes" : "no");
+ (mr & AT91_RTT_RTTINCIEN) ? "yes" : "no");
return 0;
}
rtc->events = 0;
pr_debug("%s: num=%ld, events=0x%02lx\n", __func__,
- rtc->events >> 8, rtc->events & 0x000000FF);
+ rtc->events >> 8, rtc->events & 0x000000FF);
}
/*
.alarm_irq_enable = at91_rtc_alarm_irq_enable,
};
-static const struct regmap_config gpbr_regmap_config = {
- .name = "gpbr",
- .reg_bits = 32,
- .val_bits = 32,
- .reg_stride = 4,
-};
-
/*
* Initialize and install RTC driver
*/
int ret, irq;
u32 mr;
unsigned int sclk_rate;
+ struct of_phandle_args args;
irq = platform_get_irq(pdev, 0);
if (irq < 0) {
if (IS_ERR(rtc->rtt))
return PTR_ERR(rtc->rtt);
- if (!pdev->dev.of_node) {
- /*
- * TODO: Remove this code chunk when removing non DT board
- * support. Remember to remove the gpbr_regmap_config
- * variable too.
- */
- void __iomem *gpbr;
-
- r = platform_get_resource(pdev, IORESOURCE_MEM, 1);
- gpbr = devm_ioremap_resource(&pdev->dev, r);
- if (IS_ERR(gpbr))
- return PTR_ERR(gpbr);
-
- rtc->gpbr = regmap_init_mmio(NULL, gpbr,
- &gpbr_regmap_config);
- } else {
- struct of_phandle_args args;
-
- ret = of_parse_phandle_with_fixed_args(pdev->dev.of_node,
- "atmel,rtt-rtc-time-reg", 1, 0,
- &args);
- if (ret)
- return ret;
-
- rtc->gpbr = syscon_node_to_regmap(args.np);
- rtc->gpbr_offset = args.args[0];
- }
+ ret = of_parse_phandle_with_fixed_args(pdev->dev.of_node,
+ "atmel,rtt-rtc-time-reg", 1, 0,
+ &args);
+ if (ret)
+ return ret;
+ rtc->gpbr = syscon_node_to_regmap(args.np);
+ rtc->gpbr_offset = args.args[0];
if (IS_ERR(rtc->gpbr)) {
dev_err(&pdev->dev, "failed to retrieve gpbr regmap, aborting.\n");
return -ENOMEM;
mr &= ~(AT91_RTT_ALMIEN | AT91_RTT_RTTINCIEN);
rtt_writel(rtc, MR, mr);
- rtc->rtcdev = devm_rtc_device_register(&pdev->dev, pdev->name,
- &at91_rtc_ops, THIS_MODULE);
+ rtc->rtcdev = devm_rtc_allocate_device(&pdev->dev);
if (IS_ERR(rtc->rtcdev)) {
ret = PTR_ERR(rtc->rtcdev);
goto err_clk;
}
+ rtc->rtcdev->ops = &at91_rtc_ops;
+ rtc->rtcdev->range_max = U32_MAX;
+
/* register irq handler after we know what name we'll use */
ret = devm_request_irq(&pdev->dev, rtc->irq, at91_rtc_interrupt,
IRQF_SHARED | IRQF_COND_SUSPEND,
if (gpbr_readl(rtc) == 0)
dev_warn(&pdev->dev, "%s: SET TIME!\n",
- dev_name(&rtc->rtcdev->dev));
+ dev_name(&rtc->rtcdev->dev));
- return 0;
+ return rtc_register_device(rtc->rtcdev);
err_clk:
clk_disable_unprepare(rtc->sclk);
/* don't let RTTINC cause wakeups */
if (mr & AT91_RTT_RTTINCIEN)
rtt_writel(rtc, MR, mr & ~AT91_RTT_RTTINCIEN);
- } else
+ } else {
rtt_writel(rtc, MR, mr & ~rtc->imr);
+ }
}
return 0;
static SIMPLE_DEV_PM_OPS(at91_rtc_pm_ops, at91_rtc_suspend, at91_rtc_resume);
-#ifdef CONFIG_OF
static const struct of_device_id at91_rtc_dt_ids[] = {
{ .compatible = "atmel,at91sam9260-rtt" },
{ /* sentinel */ }
};
MODULE_DEVICE_TABLE(of, at91_rtc_dt_ids);
-#endif
static struct platform_driver at91_rtc_driver = {
.probe = at91_rtc_probe,
wktmr_read(timer, &now);
- rtc_time_to_tm(now.sec, tm);
+ rtc_time64_to_tm(now.sec, tm);
return 0;
}
+// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2007-2009 ST-Ericsson AB
- * License terms: GNU General Public License (GPL) version 2
* Real Time Clock interface for ST-Ericsson AB COH 901 331 RTC.
* Author: Linus Walleij <linus.walleij@stericsson.com>
* Based on rtc-pl031.c by Deepak Saxena <dsaxena@plexity.net>
clk_enable(rtap->clk);
/* Check if the time is valid */
- if (readl(rtap->virtbase + COH901331_VALID)) {
- rtc_time_to_tm(readl(rtap->virtbase + COH901331_CUR_TIME), tm);
+ if (!readl(rtap->virtbase + COH901331_VALID)) {
clk_disable(rtap->clk);
- return 0;
+ return -EINVAL;
}
+
+ rtc_time64_to_tm(readl(rtap->virtbase + COH901331_CUR_TIME), tm);
clk_disable(rtap->clk);
- return -EINVAL;
+ return 0;
}
-static int coh901331_set_mmss(struct device *dev, unsigned long secs)
+static int coh901331_set_time(struct device *dev, struct rtc_time *tm)
{
struct coh901331_port *rtap = dev_get_drvdata(dev);
clk_enable(rtap->clk);
- writel(secs, rtap->virtbase + COH901331_SET_TIME);
+ writel(rtc_tm_to_time64(tm), rtap->virtbase + COH901331_SET_TIME);
clk_disable(rtap->clk);
return 0;
struct coh901331_port *rtap = dev_get_drvdata(dev);
clk_enable(rtap->clk);
- rtc_time_to_tm(readl(rtap->virtbase + COH901331_ALARM), &alarm->time);
+ rtc_time64_to_tm(readl(rtap->virtbase + COH901331_ALARM), &alarm->time);
alarm->pending = readl(rtap->virtbase + COH901331_IRQ_EVENT) & 1U;
alarm->enabled = readl(rtap->virtbase + COH901331_IRQ_MASK) & 1U;
clk_disable(rtap->clk);
static int coh901331_set_alarm(struct device *dev, struct rtc_wkalrm *alarm)
{
struct coh901331_port *rtap = dev_get_drvdata(dev);
- unsigned long time;
+ unsigned long time = rtc_tm_to_time64(&alarm->time);
- rtc_tm_to_time(&alarm->time, &time);
clk_enable(rtap->clk);
writel(time, rtap->virtbase + COH901331_ALARM);
writel(alarm->enabled, rtap->virtbase + COH901331_IRQ_MASK);
static const struct rtc_class_ops coh901331_ops = {
.read_time = coh901331_read_time,
- .set_mmss = coh901331_set_mmss,
+ .set_time = coh901331_set_time,
.read_alarm = coh901331_read_alarm,
.set_alarm = coh901331_set_alarm,
.alarm_irq_enable = coh901331_alarm_irq_enable,
return ret;
}
+ rtap->rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(rtap->rtc))
+ return PTR_ERR(rtap->rtc);
+
+ rtap->rtc->ops = &coh901331_ops;
+ rtap->rtc->range_max = U32_MAX;
+
/* We enable/disable the clock only to assure it works */
ret = clk_prepare_enable(rtap->clk);
if (ret) {
clk_disable(rtap->clk);
platform_set_drvdata(pdev, rtap);
- rtap->rtc = devm_rtc_device_register(&pdev->dev, "coh901331",
- &coh901331_ops, THIS_MODULE);
- if (IS_ERR(rtap->rtc)) {
- ret = PTR_ERR(rtap->rtc);
+
+ ret = rtc_register_device(rtap->rtc);
+ if (ret)
goto out_no_rtc;
- }
return 0;
-/* rtc-da9063.c - Real time clock device driver for DA9063
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Real time clock device driver for DA9063
* Copyright (C) 2013-2015 Dialog Semiconductor Ltd.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version 2
- * of the License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
*/
#include <linux/delay.h>
da9063_data_to_tm(data, tm, rtc);
- rtc_tm_to_time(tm, &tm_secs);
- rtc_tm_to_time(&rtc->alarm_time, &al_secs);
+ tm_secs = rtc_tm_to_time64(tm);
+ al_secs = rtc_tm_to_time64(&rtc->alarm_time);
/* handle the rtc synchronisation delay */
if (rtc->rtc_sync == true && al_secs - tm_secs == 1)
platform_set_drvdata(pdev, rtc);
- rtc->rtc_dev = devm_rtc_device_register(&pdev->dev, DA9063_DRVNAME_RTC,
- &da9063_rtc_ops, THIS_MODULE);
+ rtc->rtc_dev = devm_rtc_allocate_device(&pdev->dev);
if (IS_ERR(rtc->rtc_dev))
return PTR_ERR(rtc->rtc_dev);
+ rtc->rtc_dev->ops = &da9063_rtc_ops;
+ rtc->rtc_dev->range_min = RTC_TIMESTAMP_BEGIN_2000;
+ rtc->rtc_dev->range_max = RTC_TIMESTAMP_END_2063;
+
da9063_data_to_tm(data, &rtc->alarm_time, rtc);
rtc->rtc_sync = false;
dev_err(&pdev->dev, "Failed to request ALARM IRQ %d: %d\n",
irq_alarm, ret);
- return ret;
+ return rtc_register_device(rtc->rtc_dev);
}
static struct platform_driver da9063_rtc_driver = {
+// SPDX-License-Identifier: GPL-2.0+
/*
* Real Time Clock driver for Conexant Digicolor
*
* Copyright (C) 2015 Paradox Innovation Ltd.
*
* Author: Baruch Siach <baruch@tkos.co.il>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
*/
#include <linux/io.h>
return 0;
}
-static int dc_rtc_set_mmss(struct device *dev, unsigned long secs)
+static int dc_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
struct dc_rtc *rtc = dev_get_drvdata(dev);
- return dc_rtc_write(rtc, secs);
+ return dc_rtc_write(rtc, rtc_tm_to_time64(tm));
}
static int dc_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alarm)
static const struct rtc_class_ops dc_rtc_ops = {
.read_time = dc_rtc_read_time,
- .set_mmss = dc_rtc_set_mmss,
+ .set_time = dc_rtc_set_time,
.read_alarm = dc_rtc_read_alarm,
.set_alarm = dc_rtc_set_alarm,
.alarm_irq_enable = dc_rtc_alarm_irq_enable,
if (IS_ERR(rtc->regs))
return PTR_ERR(rtc->regs);
+ rtc->rtc_dev = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(rtc->rtc_dev))
+ return PTR_ERR(rtc->rtc_dev);
+
irq = platform_get_irq(pdev, 0);
if (irq < 0)
return irq;
return ret;
platform_set_drvdata(pdev, rtc);
- rtc->rtc_dev = devm_rtc_device_register(&pdev->dev, pdev->name,
- &dc_rtc_ops, THIS_MODULE);
- if (IS_ERR(rtc->rtc_dev))
- return PTR_ERR(rtc->rtc_dev);
- return 0;
+ rtc->rtc_dev->ops = &dc_rtc_ops;
+ rtc->rtc_dev->range_max = U32_MAX;
+
+ return rtc_register_device(rtc->rtc_dev);
}
static const struct of_device_id dc_dt_ids[] = {
+// SPDX-License-Identifier: GPL-2.0+
/*
* rtc-dm355evm.c - access battery-backed counter in MSP430 firmware
*
* Copyright (c) 2008 by David Brownell
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version
- * 2 of the License, or (at your option) any later version.
*/
#include <linux/kernel.h>
#include <linux/init.h>
dev_dbg(dev, "read timestamp %08x\n", time.value);
- rtc_time_to_tm(le32_to_cpu(time.value), tm);
+ rtc_time64_to_tm(le32_to_cpu(time.value), tm);
return 0;
}
unsigned long value;
int status;
- rtc_tm_to_time(tm, &value);
+ value = rtc_tm_to_time64(tm);
time.value = cpu_to_le32(value);
dev_dbg(dev, "write timestamp %08x\n", time.value);
{
struct rtc_device *rtc;
- rtc = devm_rtc_device_register(&pdev->dev, pdev->name,
- &dm355evm_rtc_ops, THIS_MODULE);
- if (IS_ERR(rtc)) {
- dev_err(&pdev->dev, "can't register RTC device, err %ld\n",
- PTR_ERR(rtc));
+ rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(rtc))
return PTR_ERR(rtc);
- }
+
platform_set_drvdata(pdev, rtc);
- return 0;
+ rtc->ops = &dm355evm_rtc_ops;
+ rtc->range_max = U32_MAX;
+
+ return rtc_register_device(rtc);
}
/*
+// SPDX-License-Identifier: GPL-2.0
/*
* An rtc/i2c driver for the Dallas DS1672
* Copyright 2005-06 Tower Technologies
*
* Author: Alessandro Zummo <a.zummo@towertech.it>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/i2c.h>
#define DS1672_REG_CONTROL_EOSC 0x80
-static struct i2c_driver ds1672_driver;
-
/*
* In the routines that deal directly with the ds1672 hardware, we use
* rtc_time -- month 0-11, hour 0-23, yr = calendar year-epoch
- * Epoch is initialized as 2000. Time is set to UTC.
+ * Time is set to UTC.
*/
-static int ds1672_get_datetime(struct i2c_client *client, struct rtc_time *tm)
+static int ds1672_read_time(struct device *dev, struct rtc_time *tm)
{
+ struct i2c_client *client = to_i2c_client(dev);
unsigned long time;
- unsigned char addr = DS1672_REG_CNT_BASE;
+ unsigned char addr = DS1672_REG_CONTROL;
unsigned char buf[4];
struct i2c_msg msgs[] = {
{/* read date */
.addr = client->addr,
.flags = I2C_M_RD,
- .len = 4,
+ .len = 1,
.buf = buf
},
};
+ /* read control register */
+ if ((i2c_transfer(client->adapter, &msgs[0], 2)) != 2) {
+ dev_warn(&client->dev, "Unable to read the control register\n");
+ return -EIO;
+ }
+
+ if (buf[0] & DS1672_REG_CONTROL_EOSC) {
+ dev_warn(&client->dev, "Oscillator not enabled. Set time to enable.\n");
+ return -EINVAL;
+ }
+
+ addr = DS1672_REG_CNT_BASE;
+ msgs[1].len = 4;
+
/* read date registers */
if ((i2c_transfer(client->adapter, &msgs[0], 2)) != 2) {
dev_err(&client->dev, "%s: read error\n", __func__);
time = ((unsigned long)buf[3] << 24) | (buf[2] << 16) |
(buf[1] << 8) | buf[0];
- rtc_time_to_tm(time, tm);
+ rtc_time64_to_tm(time, tm);
- dev_dbg(&client->dev, "%s: tm is secs=%d, mins=%d, hours=%d, "
- "mday=%d, mon=%d, year=%d, wday=%d\n",
- __func__, tm->tm_sec, tm->tm_min, tm->tm_hour,
- tm->tm_mday, tm->tm_mon, tm->tm_year, tm->tm_wday);
+ dev_dbg(&client->dev, "%s: tm is %ptR\n", __func__, tm);
return 0;
}
-static int ds1672_set_mmss(struct i2c_client *client, unsigned long secs)
+static int ds1672_set_time(struct device *dev, struct rtc_time *tm)
{
+ struct i2c_client *client = to_i2c_client(dev);
int xfer;
unsigned char buf[6];
+ unsigned long secs = rtc_tm_to_time64(tm);
buf[0] = DS1672_REG_CNT_BASE;
buf[1] = secs & 0x000000FF;
return 0;
}
-static int ds1672_rtc_read_time(struct device *dev, struct rtc_time *tm)
-{
- return ds1672_get_datetime(to_i2c_client(dev), tm);
-}
-
-static int ds1672_rtc_set_mmss(struct device *dev, unsigned long secs)
-{
- return ds1672_set_mmss(to_i2c_client(dev), secs);
-}
-
-static int ds1672_get_control(struct i2c_client *client, u8 *status)
-{
- unsigned char addr = DS1672_REG_CONTROL;
-
- struct i2c_msg msgs[] = {
- {/* setup read ptr */
- .addr = client->addr,
- .len = 1,
- .buf = &addr
- },
- {/* read control */
- .addr = client->addr,
- .flags = I2C_M_RD,
- .len = 1,
- .buf = status
- },
- };
-
- /* read control register */
- if ((i2c_transfer(client->adapter, &msgs[0], 2)) != 2) {
- dev_err(&client->dev, "%s: read error\n", __func__);
- return -EIO;
- }
-
- return 0;
-}
-
-/* following are the sysfs callback functions */
-static ssize_t show_control(struct device *dev, struct device_attribute *attr,
- char *buf)
-{
- struct i2c_client *client = to_i2c_client(dev);
- u8 control;
- int err;
-
- err = ds1672_get_control(client, &control);
- if (err)
- return err;
-
- return sprintf(buf, "%s\n", (control & DS1672_REG_CONTROL_EOSC)
- ? "disabled" : "enabled");
-}
-
-static DEVICE_ATTR(control, S_IRUGO, show_control, NULL);
-
static const struct rtc_class_ops ds1672_rtc_ops = {
- .read_time = ds1672_rtc_read_time,
- .set_mmss = ds1672_rtc_set_mmss,
+ .read_time = ds1672_read_time,
+ .set_time = ds1672_set_time,
};
static int ds1672_probe(struct i2c_client *client,
const struct i2c_device_id *id)
{
int err = 0;
- u8 control;
struct rtc_device *rtc;
dev_dbg(&client->dev, "%s\n", __func__);
if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C))
return -ENODEV;
- rtc = devm_rtc_device_register(&client->dev, ds1672_driver.driver.name,
- &ds1672_rtc_ops, THIS_MODULE);
-
+ rtc = devm_rtc_allocate_device(&client->dev);
if (IS_ERR(rtc))
return PTR_ERR(rtc);
- i2c_set_clientdata(client, rtc);
+ rtc->ops = &ds1672_rtc_ops;
+ rtc->range_max = U32_MAX;
- /* read control register */
- err = ds1672_get_control(client, &control);
- if (err) {
- dev_warn(&client->dev, "Unable to read the control register\n");
- }
+ err = rtc_register_device(rtc);
+ if (err)
+ return err;
- if (control & DS1672_REG_CONTROL_EOSC)
- dev_warn(&client->dev, "Oscillator not enabled. "
- "Set time to enable.\n");
+ if (IS_ERR(rtc))
+ return PTR_ERR(rtc);
- /* Register sysfs hooks */
- err = device_create_file(&client->dev, &dev_attr_control);
- if (err)
- dev_err(&client->dev, "Unable to create sysfs entry: %s\n",
- dev_attr_control.attr.name);
+ i2c_set_clientdata(client, rtc);
return 0;
}
(rtc->read(rtc, RTC_CTRL_B) & ~(RTC_CTRL_B_SET)));
}
-/**
- * ds1685_rtc_begin_ctrl_access - prepare the rtc for ctrl access.
- * @rtc: pointer to the ds1685 rtc structure.
- * @flags: irq flags variable for spin_lock_irqsave.
- *
- * This takes several steps to prepare the rtc for access to read just the
- * control registers:
- * - Sets a spinlock on the rtc IRQ.
- * - Switches the rtc to bank 1. This allows access to the two extended
- * control registers.
- *
- * Only use this where you are certain another lock will not be held.
- */
-static inline void
-ds1685_rtc_begin_ctrl_access(struct ds1685_priv *rtc, unsigned long *flags)
-{
- spin_lock_irqsave(&rtc->lock, *flags);
- ds1685_rtc_switch_to_bank1(rtc);
-}
-
-/**
- * ds1685_rtc_end_ctrl_access - end ctrl access on the rtc.
- * @rtc: pointer to the ds1685 rtc structure.
- * @flags: irq flags variable for spin_unlock_irqrestore.
- *
- * This ends what was started by ds1685_rtc_begin_ctrl_access:
- * - Switches the rtc back to bank 0.
- * - Unsets the spinlock on the rtc IRQ.
- */
-static inline void
-ds1685_rtc_end_ctrl_access(struct ds1685_priv *rtc, unsigned long flags)
-{
- ds1685_rtc_switch_to_bank0(rtc);
- spin_unlock_irqrestore(&rtc->lock, flags);
-}
-
/**
* ds1685_rtc_get_ssn - retrieve the silicon serial number.
* @rtc: pointer to the ds1685 rtc structure.
ds1685_rtc_alarm_irq_enable(struct device *dev, unsigned int enabled)
{
struct ds1685_priv *rtc = dev_get_drvdata(dev);
- unsigned long flags = 0;
-
- /* Enable/disable the Alarm IRQ-Enable flag. */
- spin_lock_irqsave(&rtc->lock, flags);
/* Flip the requisite interrupt-enable bit. */
if (enabled)
/* Read Control C to clear all the flag bits. */
rtc->read(rtc, RTC_CTRL_C);
- spin_unlock_irqrestore(&rtc->lock, flags);
return 0;
}
/* ----------------------------------------------------------------------- */
-/* IRQ handler & workqueue. */
-
-/**
- * ds1685_rtc_irq_handler - IRQ handler.
- * @irq: IRQ number.
- * @dev_id: platform device pointer.
- */
-static irqreturn_t
-ds1685_rtc_irq_handler(int irq, void *dev_id)
-{
- struct platform_device *pdev = dev_id;
- struct ds1685_priv *rtc = platform_get_drvdata(pdev);
- u8 ctrlb, ctrlc;
- unsigned long events = 0;
- u8 num_irqs = 0;
-
- /* Abort early if the device isn't ready yet (i.e., DEBUG_SHIRQ). */
- if (unlikely(!rtc))
- return IRQ_HANDLED;
-
- /* Ctrlb holds the interrupt-enable bits and ctrlc the flag bits. */
- spin_lock(&rtc->lock);
- ctrlb = rtc->read(rtc, RTC_CTRL_B);
- ctrlc = rtc->read(rtc, RTC_CTRL_C);
-
- /* Is the IRQF bit set? */
- if (likely(ctrlc & RTC_CTRL_C_IRQF)) {
- /*
- * We need to determine if it was one of the standard
- * events: PF, AF, or UF. If so, we handle them and
- * update the RTC core.
- */
- if (likely(ctrlc & RTC_CTRL_B_PAU_MASK)) {
- events = RTC_IRQF;
-
- /* Check for a periodic interrupt. */
- if ((ctrlb & RTC_CTRL_B_PIE) &&
- (ctrlc & RTC_CTRL_C_PF)) {
- events |= RTC_PF;
- num_irqs++;
- }
-
- /* Check for an alarm interrupt. */
- if ((ctrlb & RTC_CTRL_B_AIE) &&
- (ctrlc & RTC_CTRL_C_AF)) {
- events |= RTC_AF;
- num_irqs++;
- }
-
- /* Check for an update interrupt. */
- if ((ctrlb & RTC_CTRL_B_UIE) &&
- (ctrlc & RTC_CTRL_C_UF)) {
- events |= RTC_UF;
- num_irqs++;
- }
-
- rtc_update_irq(rtc->dev, num_irqs, events);
- } else {
- /*
- * One of the "extended" interrupts was received that
- * is not recognized by the RTC core. These need to
- * be handled in task context as they can call other
- * functions and the time spent in irq context needs
- * to be minimized. Schedule them into a workqueue
- * and inform the RTC core that the IRQs were handled.
- */
- spin_unlock(&rtc->lock);
- schedule_work(&rtc->work);
- rtc_update_irq(rtc->dev, 0, 0);
- return IRQ_HANDLED;
- }
- }
- spin_unlock(&rtc->lock);
-
- return events ? IRQ_HANDLED : IRQ_NONE;
-}
+/* IRQ handler */
/**
- * ds1685_rtc_work_queue - work queue handler.
- * @work: work_struct containing data to work on in task context.
+ * ds1685_rtc_extended_irq - take care of extended interrupts
+ * @rtc: pointer to the ds1685 rtc structure.
+ * @pdev: platform device pointer.
*/
static void
-ds1685_rtc_work_queue(struct work_struct *work)
+ds1685_rtc_extended_irq(struct ds1685_priv *rtc, struct platform_device *pdev)
{
- struct ds1685_priv *rtc = container_of(work,
- struct ds1685_priv, work);
- struct platform_device *pdev = to_platform_device(&rtc->dev->dev);
- struct mutex *rtc_mutex = &rtc->dev->ops_lock;
u8 ctrl4a, ctrl4b;
- mutex_lock(rtc_mutex);
-
ds1685_rtc_switch_to_bank1(rtc);
ctrl4a = rtc->read(rtc, RTC_EXT_CTRL_4A);
ctrl4b = rtc->read(rtc, RTC_EXT_CTRL_4B);
"RAM-Clear IRQ just occurred!\n");
}
ds1685_rtc_switch_to_bank0(rtc);
+}
+
+/**
+ * ds1685_rtc_irq_handler - IRQ handler.
+ * @irq: IRQ number.
+ * @dev_id: platform device pointer.
+ */
+static irqreturn_t
+ds1685_rtc_irq_handler(int irq, void *dev_id)
+{
+ struct platform_device *pdev = dev_id;
+ struct ds1685_priv *rtc = platform_get_drvdata(pdev);
+ struct mutex *rtc_mutex;
+ u8 ctrlb, ctrlc;
+ unsigned long events = 0;
+ u8 num_irqs = 0;
+
+ /* Abort early if the device isn't ready yet (i.e., DEBUG_SHIRQ). */
+ if (unlikely(!rtc))
+ return IRQ_HANDLED;
+
+ rtc_mutex = &rtc->dev->ops_lock;
+ mutex_lock(rtc_mutex);
+ /* Ctrlb holds the interrupt-enable bits and ctrlc the flag bits. */
+ ctrlb = rtc->read(rtc, RTC_CTRL_B);
+ ctrlc = rtc->read(rtc, RTC_CTRL_C);
+
+ /* Is the IRQF bit set? */
+ if (likely(ctrlc & RTC_CTRL_C_IRQF)) {
+ /*
+ * We need to determine if it was one of the standard
+ * events: PF, AF, or UF. If so, we handle them and
+ * update the RTC core.
+ */
+ if (likely(ctrlc & RTC_CTRL_B_PAU_MASK)) {
+ events = RTC_IRQF;
+
+ /* Check for a periodic interrupt. */
+ if ((ctrlb & RTC_CTRL_B_PIE) &&
+ (ctrlc & RTC_CTRL_C_PF)) {
+ events |= RTC_PF;
+ num_irqs++;
+ }
+
+ /* Check for an alarm interrupt. */
+ if ((ctrlb & RTC_CTRL_B_AIE) &&
+ (ctrlc & RTC_CTRL_C_AF)) {
+ events |= RTC_AF;
+ num_irqs++;
+ }
+
+ /* Check for an update interrupt. */
+ if ((ctrlb & RTC_CTRL_B_UIE) &&
+ (ctrlc & RTC_CTRL_C_UF)) {
+ events |= RTC_UF;
+ num_irqs++;
+ }
+ } else {
+ /*
+ * One of the "extended" interrupts was received that
+ * is not recognized by the RTC core.
+ */
+ ds1685_rtc_extended_irq(rtc, pdev);
+ }
+ }
+ rtc_update_irq(rtc->dev, num_irqs, events);
mutex_unlock(rtc_mutex);
+
+ return events ? IRQ_HANDLED : IRQ_NONE;
}
/* ----------------------------------------------------------------------- */
size_t size)
{
struct ds1685_priv *rtc = priv;
+ struct mutex *rtc_mutex = &rtc->dev->ops_lock;
ssize_t count;
- unsigned long flags = 0;
u8 *buf = val;
+ int err;
+
+ err = mutex_lock_interruptible(rtc_mutex);
+ if (err)
+ return err;
- spin_lock_irqsave(&rtc->lock, flags);
ds1685_rtc_switch_to_bank0(rtc);
/* Read NVRAM in time and bank0 registers. */
ds1685_rtc_switch_to_bank0(rtc);
}
#endif /* !CONFIG_RTC_DRV_DS1689 */
- spin_unlock_irqrestore(&rtc->lock, flags);
+ mutex_unlock(rtc_mutex);
return 0;
}
size_t size)
{
struct ds1685_priv *rtc = priv;
+ struct mutex *rtc_mutex = &rtc->dev->ops_lock;
ssize_t count;
- unsigned long flags = 0;
u8 *buf = val;
+ int err;
+
+ err = mutex_lock_interruptible(rtc_mutex);
+ if (err)
+ return err;
- spin_lock_irqsave(&rtc->lock, flags);
ds1685_rtc_switch_to_bank0(rtc);
/* Write NVRAM in time and bank0 registers. */
ds1685_rtc_switch_to_bank0(rtc);
}
#endif /* !CONFIG_RTC_DRV_DS1689 */
- spin_unlock_irqrestore(&rtc->lock, flags);
+ mutex_unlock(rtc_mutex);
return 0;
}
ds1685_rtc_sysfs_battery_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- struct ds1685_priv *rtc = dev_get_drvdata(dev);
+ struct ds1685_priv *rtc = dev_get_drvdata(dev->parent);
u8 ctrld;
ctrld = rtc->read(rtc, RTC_CTRL_D);
ds1685_rtc_sysfs_auxbatt_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- struct ds1685_priv *rtc = dev_get_drvdata(dev);
+ struct ds1685_priv *rtc = dev_get_drvdata(dev->parent);
u8 ctrl4a;
ds1685_rtc_switch_to_bank1(rtc);
ds1685_rtc_sysfs_serial_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- struct ds1685_priv *rtc = dev_get_drvdata(dev);
+ struct ds1685_priv *rtc = dev_get_drvdata(dev->parent);
u8 ssn[8];
ds1685_rtc_switch_to_bank1(rtc);
if (pdata->plat_post_ram_clear)
rtc->post_ram_clear = pdata->plat_post_ram_clear;
- /* Init the spinlock, workqueue, & set the driver data. */
- spin_lock_init(&rtc->lock);
- INIT_WORK(&rtc->work, ds1685_rtc_work_queue);
+ /* set the driver data. */
platform_set_drvdata(pdev, rtc);
/* Turn the oscillator on if is not already on (DV1 = 1). */
*/
if (!pdata->no_irq) {
ret = platform_get_irq(pdev, 0);
- if (ret > 0) {
- rtc->irq_num = ret;
-
- /* Request an IRQ. */
- ret = devm_request_irq(&pdev->dev, rtc->irq_num,
- ds1685_rtc_irq_handler,
- IRQF_SHARED, pdev->name, pdev);
-
- /* Check to see if something came back. */
- if (unlikely(ret)) {
- dev_warn(&pdev->dev,
- "RTC interrupt not available\n");
- rtc->irq_num = 0;
- }
- } else
+ if (ret <= 0)
return ret;
+
+ rtc->irq_num = ret;
+
+ /* Request an IRQ. */
+ ret = devm_request_threaded_irq(&pdev->dev, rtc->irq_num,
+ NULL, ds1685_rtc_irq_handler,
+ IRQF_SHARED | IRQF_ONESHOT,
+ pdev->name, pdev);
+
+ /* Check to see if something came back. */
+ if (unlikely(ret)) {
+ dev_warn(&pdev->dev,
+ "RTC interrupt not available\n");
+ rtc->irq_num = 0;
+ }
}
rtc->no_irq = pdata->no_irq;
(rtc->read(rtc, RTC_EXT_CTRL_4A) &
~(RTC_CTRL_4A_RWK_MASK)));
- cancel_work_sync(&rtc->work);
-
return 0;
}
-/*
- * Copyright (C) 2012 Sven Schnelle <svens@stackframe.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- */
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2012 Sven Schnelle <svens@stackframe.org>
#include <linux/platform_device.h>
#include <linux/module.h>
#define DS2404_COPY_SCRATCHPAD_CMD 0x55
#define DS2404_READ_MEMORY_CMD 0xf0
-struct ds2404;
-
-struct ds2404_chip_ops {
- int (*map_io)(struct ds2404 *chip, struct platform_device *pdev,
- struct ds2404_platform_data *pdata);
- void (*unmap_io)(struct ds2404 *chip);
-};
-
#define DS2404_RST 0
#define DS2404_CLK 1
#define DS2404_DQ 2
struct ds2404 {
struct ds2404_gpio *gpio;
- const struct ds2404_chip_ops *ops;
struct rtc_device *rtc;
};
return err;
}
-static void ds2404_gpio_unmap(struct ds2404 *chip)
+static void ds2404_gpio_unmap(void *data)
{
int i;
gpio_free(ds2404_gpio[i].gpio);
}
-static const struct ds2404_chip_ops ds2404_gpio_ops = {
- .map_io = ds2404_gpio_map,
- .unmap_io = ds2404_gpio_unmap,
-};
-
static void ds2404_reset(struct device *dev)
{
gpio_set_value(ds2404_gpio[DS2404_RST].gpio, 0);
ds2404_read_memory(dev, 0x203, 4, (u8 *)&time);
time = le32_to_cpu(time);
- rtc_time_to_tm(time, dt);
+ rtc_time64_to_tm(time, dt);
return 0;
}
-static int ds2404_set_mmss(struct device *dev, unsigned long secs)
+static int ds2404_set_time(struct device *dev, struct rtc_time *dt)
{
- u32 time = cpu_to_le32(secs);
+ u32 time = cpu_to_le32(rtc_tm_to_time64(dt));
ds2404_write_memory(dev, 0x203, 4, (u8 *)&time);
return 0;
}
static const struct rtc_class_ops ds2404_rtc_ops = {
.read_time = ds2404_read_time,
- .set_mmss = ds2404_set_mmss,
+ .set_time = ds2404_set_time,
};
static int rtc_probe(struct platform_device *pdev)
if (!chip)
return -ENOMEM;
- chip->ops = &ds2404_gpio_ops;
+ chip->rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(chip->rtc))
+ return PTR_ERR(chip->rtc);
- retval = chip->ops->map_io(chip, pdev, pdata);
+ retval = ds2404_gpio_map(chip, pdev, pdata);
if (retval)
- goto err_chip;
+ return retval;
+
+ retval = devm_add_action_or_reset(&pdev->dev, ds2404_gpio_unmap, chip);
+ if (retval)
+ return retval;
dev_info(&pdev->dev, "using GPIOs RST:%d, CLK:%d, DQ:%d\n",
chip->gpio[DS2404_RST].gpio, chip->gpio[DS2404_CLK].gpio,
platform_set_drvdata(pdev, chip);
- chip->rtc = devm_rtc_device_register(&pdev->dev, "ds2404",
- &ds2404_rtc_ops, THIS_MODULE);
- if (IS_ERR(chip->rtc)) {
- retval = PTR_ERR(chip->rtc);
- goto err_io;
- }
+ chip->rtc->ops = &ds2404_rtc_ops;
+ chip->rtc->range_max = U32_MAX;
- ds2404_enable_osc(&pdev->dev);
- return 0;
-
-err_io:
- chip->ops->unmap_io(chip);
-err_chip:
- return retval;
-}
-
-static int rtc_remove(struct platform_device *dev)
-{
- struct ds2404 *chip = platform_get_drvdata(dev);
-
- chip->ops->unmap_io(chip);
+ retval = rtc_register_device(chip->rtc);
+ if (retval)
+ return retval;
+ ds2404_enable_osc(&pdev->dev);
return 0;
}
static struct platform_driver rtc_device_driver = {
.probe = rtc_probe,
- .remove = rtc_remove,
.driver = {
.name = "ds2404",
},
# define DS3232_REG_SR_A1F 0x01
#define DS3232_REG_TEMPERATURE 0x11
+#define DS3232_REG_SRAM_START 0x14
+#define DS3232_REG_SRAM_END 0xFF
+
+#define DS3232_REG_SRAM_SIZE 236
struct ds3232 {
struct device *dev;
.alarm_irq_enable = ds3232_alarm_irq_enable,
};
+static int ds3232_nvmem_read(void *priv, unsigned int offset, void *val,
+ size_t bytes)
+{
+ struct regmap *ds3232_regmap = (struct regmap *)priv;
+
+ return regmap_bulk_read(ds3232_regmap, DS3232_REG_SRAM_START + offset,
+ val, bytes);
+}
+
+static int ds3232_nvmem_write(void *priv, unsigned int offset, void *val,
+ size_t bytes)
+{
+ struct regmap *ds3232_regmap = (struct regmap *)priv;
+
+ return regmap_bulk_write(ds3232_regmap, DS3232_REG_SRAM_START + offset,
+ val, bytes);
+}
+
static int ds3232_probe(struct device *dev, struct regmap *regmap, int irq,
const char *name)
{
struct ds3232 *ds3232;
int ret;
+ struct nvmem_config nvmem_cfg = {
+ .name = "ds3232_sram",
+ .stride = 1,
+ .size = DS3232_REG_SRAM_SIZE,
+ .word_size = 1,
+ .reg_read = ds3232_nvmem_read,
+ .reg_write = ds3232_nvmem_write,
+ .priv = regmap,
+ .type = NVMEM_TYPE_BATTERY_BACKED
+ };
ds3232 = devm_kzalloc(dev, sizeof(*ds3232), GFP_KERNEL);
if (!ds3232)
if (IS_ERR(ds3232->rtc))
return PTR_ERR(ds3232->rtc);
+ ret = rtc_nvmem_register(ds3232->rtc, &nvmem_cfg);
+ if(ret)
+ return ret;
+
if (ds3232->irq > 0) {
ret = devm_request_threaded_irq(dev, ds3232->irq, NULL,
ds3232_irq,
static const struct regmap_config config = {
.reg_bits = 8,
.val_bits = 8,
- .max_register = 0x13,
+ .max_register = DS3232_REG_SRAM_END,
};
regmap = devm_regmap_init_i2c(client, &config);
static const struct regmap_config config = {
.reg_bits = 8,
.val_bits = 8,
- .max_register = 0x13,
+ .max_register = DS3232_REG_SRAM_END,
.write_flag_mask = 0x80,
};
struct regmap *regmap;
+// SPDX-License-Identifier: GPL-2.0
/*
* A driver for the RTC embedded in the Cirrus Logic EP93XX processors
* Copyright (c) 2006 Tower Technologies
*
* Author: Alessandro Zummo <a.zummo@towertech.it>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/module.h>
#define EP93XX_RTC_DATA 0x000
#define EP93XX_RTC_MATCH 0x004
#define EP93XX_RTC_STATUS 0x008
-#define EP93XX_RTC_STATUS_INTR (1<<0)
+#define EP93XX_RTC_STATUS_INTR BIT(0)
#define EP93XX_RTC_LOAD 0x00C
#define EP93XX_RTC_CONTROL 0x010
-#define EP93XX_RTC_CONTROL_MIE (1<<0)
+#define EP93XX_RTC_CONTROL_MIE BIT(0)
#define EP93XX_RTC_SWCOMP 0x108
#define EP93XX_RTC_SWCOMP_DEL_MASK 0x001f0000
#define EP93XX_RTC_SWCOMP_DEL_SHIFT 16
#define EP93XX_RTC_SWCOMP_INT_MASK 0x0000ffff
#define EP93XX_RTC_SWCOMP_INT_SHIFT 0
-/*
- * struct device dev.platform_data is used to store our private data
- * because struct rtc_device does not have a variable to hold it.
- */
struct ep93xx_rtc {
void __iomem *mmio_base;
struct rtc_device *rtc;
};
static int ep93xx_rtc_get_swcomp(struct device *dev, unsigned short *preload,
- unsigned short *delete)
+ unsigned short *delete)
{
struct ep93xx_rtc *ep93xx_rtc = dev_get_platdata(dev);
unsigned long comp;
time = readl(ep93xx_rtc->mmio_base + EP93XX_RTC_DATA);
- rtc_time_to_tm(time, tm);
+ rtc_time64_to_tm(time, tm);
return 0;
}
-static int ep93xx_rtc_set_mmss(struct device *dev, unsigned long secs)
+static int ep93xx_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
struct ep93xx_rtc *ep93xx_rtc = dev_get_platdata(dev);
+ unsigned long secs = rtc_tm_to_time64(tm);
writel(secs + 1, ep93xx_rtc->mmio_base + EP93XX_RTC_LOAD);
return 0;
static const struct rtc_class_ops ep93xx_rtc_ops = {
.read_time = ep93xx_rtc_read_time,
- .set_mmss = ep93xx_rtc_set_mmss,
+ .set_time = ep93xx_rtc_set_time,
.proc = ep93xx_rtc_proc,
};
-static ssize_t ep93xx_rtc_show_comp_preload(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t comp_preload_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
unsigned short preload;
- ep93xx_rtc_get_swcomp(dev, &preload, NULL);
+ ep93xx_rtc_get_swcomp(dev->parent, &preload, NULL);
return sprintf(buf, "%d\n", preload);
}
-static DEVICE_ATTR(comp_preload, S_IRUGO, ep93xx_rtc_show_comp_preload, NULL);
+static DEVICE_ATTR_RO(comp_preload);
-static ssize_t ep93xx_rtc_show_comp_delete(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t comp_delete_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
unsigned short delete;
- ep93xx_rtc_get_swcomp(dev, NULL, &delete);
+ ep93xx_rtc_get_swcomp(dev->parent, NULL, &delete);
return sprintf(buf, "%d\n", delete);
}
-static DEVICE_ATTR(comp_delete, S_IRUGO, ep93xx_rtc_show_comp_delete, NULL);
+static DEVICE_ATTR_RO(comp_delete);
static struct attribute *ep93xx_rtc_attrs[] = {
&dev_attr_comp_preload.attr,
if (IS_ERR(ep93xx_rtc->mmio_base))
return PTR_ERR(ep93xx_rtc->mmio_base);
- pdev->dev.platform_data = ep93xx_rtc;
platform_set_drvdata(pdev, ep93xx_rtc);
- ep93xx_rtc->rtc = devm_rtc_device_register(&pdev->dev,
- pdev->name, &ep93xx_rtc_ops, THIS_MODULE);
- if (IS_ERR(ep93xx_rtc->rtc)) {
- err = PTR_ERR(ep93xx_rtc->rtc);
- goto exit;
- }
+ ep93xx_rtc->rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(ep93xx_rtc->rtc))
+ return PTR_ERR(ep93xx_rtc->rtc);
- err = sysfs_create_group(&pdev->dev.kobj, &ep93xx_rtc_sysfs_files);
- if (err)
- goto exit;
-
- return 0;
+ ep93xx_rtc->rtc->ops = &ep93xx_rtc_ops;
+ ep93xx_rtc->rtc->range_max = U32_MAX;
-exit:
- pdev->dev.platform_data = NULL;
- return err;
-}
-
-static int ep93xx_rtc_remove(struct platform_device *pdev)
-{
- sysfs_remove_group(&pdev->dev.kobj, &ep93xx_rtc_sysfs_files);
- pdev->dev.platform_data = NULL;
+ err = rtc_add_group(ep93xx_rtc->rtc, &ep93xx_rtc_sysfs_files);
+ if (err)
+ return err;
- return 0;
+ return rtc_register_device(ep93xx_rtc->rtc);
}
static struct platform_driver ep93xx_rtc_driver = {
.name = "ep93xx-rtc",
},
.probe = ep93xx_rtc_probe,
- .remove = ep93xx_rtc_remove,
};
module_platform_driver(ep93xx_rtc_driver);
+// SPDX-License-Identifier: GPL-2.0
/* drivers/rtc/rtc-goldfish.c
*
* Copyright (C) 2007 Google, Inc.
* Copyright (C) 2017 Imagination Technologies Ltd.
- *
- * This software is licensed under the terms of the GNU General Public
- * License version 2, as published by the Free Software Foundation, and
- * may be copied, distributed, and modified under those terms.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
*/
+#include <linux/io.h>
#include <linux/module.h>
+#include <linux/of.h>
#include <linux/platform_device.h>
#include <linux/rtc.h>
-#include <linux/io.h>
#define TIMER_TIME_LOW 0x00 /* get low bits of current time */
/* and update TIMER_TIME_HIGH */
do_div(rtc_alarm, NSEC_PER_SEC);
memset(alrm, 0, sizeof(struct rtc_wkalrm));
- rtc_time_to_tm(rtc_alarm, &alrm->time);
+ rtc_time64_to_tm(rtc_alarm, &alrm->time);
if (readl(base + TIMER_ALARM_STATUS))
alrm->enabled = 1;
struct rtc_wkalrm *alrm)
{
struct goldfish_rtc *rtcdrv;
- unsigned long rtc_alarm;
u64 rtc_alarm64;
u64 rtc_status_reg;
void __iomem *base;
- int ret = 0;
rtcdrv = dev_get_drvdata(dev);
base = rtcdrv->base;
if (alrm->enabled) {
- ret = rtc_tm_to_time(&alrm->time, &rtc_alarm);
- if (ret != 0)
- return ret;
-
- rtc_alarm64 = rtc_alarm * NSEC_PER_SEC;
+ rtc_alarm64 = rtc_tm_to_time64(&alrm->time) * NSEC_PER_SEC;
writel((rtc_alarm64 >> 32), base + TIMER_ALARM_HIGH);
writel(rtc_alarm64, base + TIMER_ALARM_LOW);
} else {
writel(1, base + TIMER_CLEAR_ALARM);
}
- return ret;
+ return 0;
}
static int goldfish_rtc_alarm_irq_enable(struct device *dev,
do_div(time, NSEC_PER_SEC);
- rtc_time_to_tm(time, tm);
+ rtc_time64_to_tm(time, tm);
return 0;
}
{
struct goldfish_rtc *rtcdrv;
void __iomem *base;
- unsigned long now;
u64 now64;
- int ret;
rtcdrv = dev_get_drvdata(dev);
base = rtcdrv->base;
- ret = rtc_tm_to_time(tm, &now);
- if (ret == 0) {
- now64 = now * NSEC_PER_SEC;
- writel((now64 >> 32), base + TIMER_TIME_HIGH);
- writel(now64, base + TIMER_TIME_LOW);
- }
+ now64 = rtc_tm_to_time64(tm) * NSEC_PER_SEC;
+ writel((now64 >> 32), base + TIMER_TIME_HIGH);
+ writel(now64, base + TIMER_TIME_LOW);
- return ret;
+ return 0;
}
static const struct rtc_class_ops goldfish_rtc_ops = {
if (rtcdrv->irq < 0)
return -ENODEV;
- rtcdrv->rtc = devm_rtc_device_register(&pdev->dev, pdev->name,
- &goldfish_rtc_ops,
- THIS_MODULE);
+ rtcdrv->rtc = devm_rtc_allocate_device(&pdev->dev);
if (IS_ERR(rtcdrv->rtc))
return PTR_ERR(rtcdrv->rtc);
+ rtcdrv->rtc->ops = &goldfish_rtc_ops;
+ rtcdrv->rtc->range_max = U64_MAX / NSEC_PER_SEC;
+
err = devm_request_irq(&pdev->dev, rtcdrv->irq,
goldfish_rtc_interrupt,
0, pdev->name, rtcdrv);
if (err)
return err;
- return 0;
+ return rtc_register_device(rtcdrv->rtc);
}
static const struct of_device_id goldfish_rtc_of_match[] = {
static int hid_rtc_read_time(struct device *dev, struct rtc_time *tm)
{
unsigned long flags;
- struct hid_time_state *time_state =
- platform_get_drvdata(to_platform_device(dev));
+ struct hid_time_state *time_state = dev_get_drvdata(dev);
int ret;
reinit_completion(&time_state->comp_last_time);
+// SPDX-License-Identifier: GPL-2.0+
/*
* Copyright 2008-2009 Freescale Semiconductor, Inc. All Rights Reserved.
* Copyright 2010 Orex Computed Radiography
*/
-/*
- * The code contained herein is licensed under the GNU General Public
- * License. You may obtain a copy of the GNU General Public License
- * Version 2 or later at the following locations:
- *
- * http://www.opensource.org/licenses/gpl-license.html
- * http://www.gnu.org/copyleft/gpl.html
- */
-
-/* based on rtc-mc13892.c */
-
/*
* This driver uses the 47-bit 32 kHz counter in the Freescale DryIce block
* to implement a Linux RTC. Times and alarms are truncated to seconds.
unsigned long now;
now = readl(imxdi->ioaddr + DTCMR);
- rtc_time_to_tm(now, tm);
+ rtc_time64_to_tm(now, tm);
return 0;
}
* set the seconds portion of dryice time counter and clear the
* fractional part.
*/
-static int dryice_rtc_set_mmss(struct device *dev, unsigned long secs)
+static int dryice_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
struct imxdi_dev *imxdi = dev_get_drvdata(dev);
u32 dcr, dsr;
if (rc != 0)
return rc;
- rc = di_write_wait(imxdi, secs, DTCMR);
+ rc = di_write_wait(imxdi, rtc_tm_to_time64(tm), DTCMR);
if (rc != 0)
return rc;
u32 dcamr;
dcamr = readl(imxdi->ioaddr + DCAMR);
- rtc_time_to_tm(dcamr, &alarm->time);
+ rtc_time64_to_tm(dcamr, &alarm->time);
/* alarm is enabled if the interrupt is enabled */
alarm->enabled = (readl(imxdi->ioaddr + DIER) & DIER_CAIE) != 0;
static int dryice_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alarm)
{
struct imxdi_dev *imxdi = dev_get_drvdata(dev);
- unsigned long now;
- unsigned long alarm_time;
int rc;
- rc = rtc_tm_to_time(&alarm->time, &alarm_time);
- if (rc)
- return rc;
-
- /* don't allow setting alarm in the past */
- now = readl(imxdi->ioaddr + DTCMR);
- if (alarm_time < now)
- return -EINVAL;
-
/* write the new alarm time */
- rc = di_write_wait(imxdi, (u32)alarm_time, DCAMR);
+ rc = di_write_wait(imxdi, rtc_tm_to_time64(&alarm->time), DCAMR);
if (rc)
return rc;
static const struct rtc_class_ops dryice_rtc_ops = {
.read_time = dryice_rtc_read_time,
- .set_mmss = dryice_rtc_set_mmss,
+ .set_time = dryice_rtc_set_time,
.alarm_irq_enable = dryice_rtc_alarm_irq_enable,
.read_alarm = dryice_rtc_read_alarm,
.set_alarm = dryice_rtc_set_alarm,
mutex_init(&imxdi->write_mutex);
+ imxdi->rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(imxdi->rtc))
+ return PTR_ERR(imxdi->rtc);
+
imxdi->clk = devm_clk_get(&pdev->dev, NULL);
if (IS_ERR(imxdi->clk))
return PTR_ERR(imxdi->clk);
}
platform_set_drvdata(pdev, imxdi);
- imxdi->rtc = devm_rtc_device_register(&pdev->dev, pdev->name,
- &dryice_rtc_ops, THIS_MODULE);
- if (IS_ERR(imxdi->rtc)) {
- rc = PTR_ERR(imxdi->rtc);
+
+ imxdi->rtc->ops = &dryice_rtc_ops;
+ imxdi->rtc->range_max = U32_MAX;
+
+ rc = rtc_register_device(imxdi->rtc);
+ if (rc)
goto err;
- }
return 0;
+// SPDX-License-Identifier: GPL-2.0+
/*
* Copyright (C) 2009-2010, Lars-Peter Clausen <lars@metafoo.de>
* Copyright (C) 2010, Paul Cercueil <paul@crapouillou.net>
* JZ4740 SoC RTC driver
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- * You should have received a copy of the GNU General Public License along
- * with this program; if not, write to the Free Software Foundation, Inc.,
- * 675 Mass Ave, Cambridge, MA 02139, USA.
- *
*/
#include <linux/clk.h>
#include <linux/module.h>
#include <linux/of_device.h>
#include <linux/platform_device.h>
+#include <linux/pm_wakeirq.h>
#include <linux/reboot.h>
#include <linux/rtc.h>
#include <linux/slab.h>
uint32_t secs, secs2;
int timeout = 5;
+ if (jz4740_rtc_reg_read(rtc, JZ_REG_RTC_SCRATCHPAD) != 0x12345678)
+ return -EINVAL;
+
/* If the seconds register is read while it is updated, it can contain a
* bogus value. This can be avoided by making sure that two consecutive
* reads have the same value.
if (timeout == 0)
return -EIO;
- rtc_time_to_tm(secs, time);
+ rtc_time64_to_tm(secs, time);
return 0;
}
-static int jz4740_rtc_set_mmss(struct device *dev, unsigned long secs)
+static int jz4740_rtc_set_time(struct device *dev, struct rtc_time *time)
{
struct jz4740_rtc *rtc = dev_get_drvdata(dev);
+ int ret;
- return jz4740_rtc_reg_write(rtc, JZ_REG_RTC_SEC, secs);
+ ret = jz4740_rtc_reg_write(rtc, JZ_REG_RTC_SEC, rtc_tm_to_time64(time));
+ if (ret)
+ return ret;
+
+ return jz4740_rtc_reg_write(rtc, JZ_REG_RTC_SCRATCHPAD, 0x12345678);
}
static int jz4740_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
alrm->enabled = !!(ctrl & JZ_RTC_CTRL_AE);
alrm->pending = !!(ctrl & JZ_RTC_CTRL_AF);
- rtc_time_to_tm(secs, &alrm->time);
+ rtc_time64_to_tm(secs, &alrm->time);
- return rtc_valid_tm(&alrm->time);
+ return 0;
}
static int jz4740_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm)
{
int ret;
struct jz4740_rtc *rtc = dev_get_drvdata(dev);
- unsigned long secs;
-
- rtc_tm_to_time(&alrm->time, &secs);
+ uint32_t secs = lower_32_bits(rtc_tm_to_time64(&alrm->time));
ret = jz4740_rtc_reg_write(rtc, JZ_REG_RTC_SEC_ALARM, secs);
if (!ret)
static const struct rtc_class_ops jz4740_rtc_ops = {
.read_time = jz4740_rtc_read_time,
- .set_mmss = jz4740_rtc_set_mmss,
+ .set_time = jz4740_rtc_set_time,
.read_alarm = jz4740_rtc_read_alarm,
.set_alarm = jz4740_rtc_set_alarm,
.alarm_irq_enable = jz4740_rtc_alarm_irq_enable,
{
int ret;
struct jz4740_rtc *rtc;
- uint32_t scratchpad;
struct resource *mem;
const struct platform_device_id *id = platform_get_device_id(pdev);
const struct of_device_id *of_id = of_match_device(
device_init_wakeup(&pdev->dev, 1);
- rtc->rtc = devm_rtc_device_register(&pdev->dev, pdev->name,
- &jz4740_rtc_ops, THIS_MODULE);
+ ret = dev_pm_set_wake_irq(&pdev->dev, rtc->irq);
+ if (ret) {
+ dev_err(&pdev->dev, "Failed to set wake irq: %d\n", ret);
+ return ret;
+ }
+
+ rtc->rtc = devm_rtc_allocate_device(&pdev->dev);
if (IS_ERR(rtc->rtc)) {
ret = PTR_ERR(rtc->rtc);
+ dev_err(&pdev->dev, "Failed to allocate rtc device: %d\n", ret);
+ return ret;
+ }
+
+ rtc->rtc->ops = &jz4740_rtc_ops;
+ rtc->rtc->range_max = U32_MAX;
+
+ ret = rtc_register_device(rtc->rtc);
+ if (ret) {
dev_err(&pdev->dev, "Failed to register rtc device: %d\n", ret);
return ret;
}
return ret;
}
- scratchpad = jz4740_rtc_reg_read(rtc, JZ_REG_RTC_SCRATCHPAD);
- if (scratchpad != 0x12345678) {
- ret = jz4740_rtc_reg_write(rtc, JZ_REG_RTC_SCRATCHPAD, 0x12345678);
- ret = jz4740_rtc_reg_write(rtc, JZ_REG_RTC_SEC, 0);
- if (ret) {
- dev_err(&pdev->dev, "Could not write to RTC registers\n");
- return ret;
- }
- }
-
if (np && of_device_is_system_power_controller(np)) {
if (!pm_power_off) {
/* Default: 60ms */
return 0;
}
-#ifdef CONFIG_PM
-static int jz4740_rtc_suspend(struct device *dev)
-{
- struct jz4740_rtc *rtc = dev_get_drvdata(dev);
-
- if (device_may_wakeup(dev))
- enable_irq_wake(rtc->irq);
- return 0;
-}
-
-static int jz4740_rtc_resume(struct device *dev)
-{
- struct jz4740_rtc *rtc = dev_get_drvdata(dev);
-
- if (device_may_wakeup(dev))
- disable_irq_wake(rtc->irq);
- return 0;
-}
-
-static const struct dev_pm_ops jz4740_pm_ops = {
- .suspend = jz4740_rtc_suspend,
- .resume = jz4740_rtc_resume,
-};
-#define JZ4740_RTC_PM_OPS (&jz4740_pm_ops)
-
-#else
-#define JZ4740_RTC_PM_OPS NULL
-#endif /* CONFIG_PM */
-
static const struct platform_device_id jz4740_rtc_ids[] = {
{ "jz4740-rtc", ID_JZ4740 },
{ "jz4780-rtc", ID_JZ4780 },
.probe = jz4740_rtc_probe,
.driver = {
.name = "jz4740-rtc",
- .pm = JZ4740_RTC_PM_OPS,
.of_match_table = of_match_ptr(jz4740_rtc_of_match),
},
.id_table = jz4740_rtc_ids,
+// SPDX-License-Identifier: GPL-2.0+
/*
* Copyright (C) 2010 NXP Semiconductors
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * You should have received a copy of the GNU General Public License along
- * with this program; if not, write to the Free Software Foundation, Inc.,
- * 675 Mass Ave, Cambridge, MA 02139, USA.
*/
#include <linux/kernel.h>
#define LPC32XX_RTC_KEY_ONSW_LOADVAL 0xB5C13F27
-#define RTC_NAME "rtc-lpc32xx"
-
#define rtc_readl(dev, reg) \
__raw_readl((dev)->rtc_base + (reg))
#define rtc_writel(dev, reg, val) \
struct lpc32xx_rtc *rtc = dev_get_drvdata(dev);
elapsed_sec = rtc_readl(rtc, LPC32XX_RTC_UCOUNT);
- rtc_time_to_tm(elapsed_sec, time);
+ rtc_time64_to_tm(elapsed_sec, time);
return 0;
}
-static int lpc32xx_rtc_set_mmss(struct device *dev, unsigned long secs)
+static int lpc32xx_rtc_set_time(struct device *dev, struct rtc_time *time)
{
struct lpc32xx_rtc *rtc = dev_get_drvdata(dev);
+ u32 secs = rtc_tm_to_time64(time);
u32 tmp;
spin_lock_irq(&rtc->lock);
{
struct lpc32xx_rtc *rtc = dev_get_drvdata(dev);
- rtc_time_to_tm(rtc_readl(rtc, LPC32XX_RTC_MATCH0), &wkalrm->time);
+ rtc_time64_to_tm(rtc_readl(rtc, LPC32XX_RTC_MATCH0), &wkalrm->time);
wkalrm->enabled = rtc->alarm_enabled;
wkalrm->pending = !!(rtc_readl(rtc, LPC32XX_RTC_INTSTAT) &
LPC32XX_RTC_INTSTAT_MATCH0);
struct lpc32xx_rtc *rtc = dev_get_drvdata(dev);
unsigned long alarmsecs;
u32 tmp;
- int ret;
- ret = rtc_tm_to_time(&wkalrm->time, &alarmsecs);
- if (ret < 0) {
- dev_warn(dev, "Failed to convert time: %d\n", ret);
- return ret;
- }
+ alarmsecs = rtc_tm_to_time64(&wkalrm->time);
spin_lock_irq(&rtc->lock);
static const struct rtc_class_ops lpc32xx_rtc_ops = {
.read_time = lpc32xx_rtc_read_time,
- .set_mmss = lpc32xx_rtc_set_mmss,
+ .set_time = lpc32xx_rtc_set_time,
.read_alarm = lpc32xx_rtc_read_alarm,
.set_alarm = lpc32xx_rtc_set_alarm,
.alarm_irq_enable = lpc32xx_rtc_alarm_irq_enable,
{
struct resource *res;
struct lpc32xx_rtc *rtc;
- int rtcirq;
+ int err;
u32 tmp;
- rtcirq = platform_get_irq(pdev, 0);
- if (rtcirq < 0) {
- dev_warn(&pdev->dev, "Can't get interrupt resource\n");
- rtcirq = -1;
- }
-
rtc = devm_kzalloc(&pdev->dev, sizeof(*rtc), GFP_KERNEL);
if (unlikely(!rtc))
return -ENOMEM;
- rtc->irq = rtcirq;
-
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
rtc->rtc_base = devm_ioremap_resource(&pdev->dev, res);
if (IS_ERR(rtc->rtc_base))
platform_set_drvdata(pdev, rtc);
- rtc->rtc = devm_rtc_device_register(&pdev->dev, RTC_NAME,
- &lpc32xx_rtc_ops, THIS_MODULE);
- if (IS_ERR(rtc->rtc)) {
- dev_err(&pdev->dev, "Can't get RTC\n");
+ rtc->rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(rtc->rtc))
return PTR_ERR(rtc->rtc);
- }
+
+ rtc->rtc->ops = &lpc32xx_rtc_ops;
+ rtc->rtc->range_max = U32_MAX;
+
+ err = rtc_register_device(rtc->rtc);
+ if (err)
+ return err;
/*
* IRQ is enabled after device registration in case alarm IRQ
* is pending upon suspend exit.
*/
- if (rtc->irq >= 0) {
+ rtc->irq = platform_get_irq(pdev, 0);
+ if (rtc->irq < 0) {
+ dev_warn(&pdev->dev, "Can't get interrupt resource\n");
+ } else {
if (devm_request_irq(&pdev->dev, rtc->irq,
lpc32xx_rtc_alarm_interrupt,
0, pdev->name, rtc) < 0) {
.probe = lpc32xx_rtc_probe,
.remove = lpc32xx_rtc_remove,
.driver = {
- .name = RTC_NAME,
+ .name = "rtc-lpc32xx",
.pm = LPC32XX_RTC_PM_OPS,
.of_match_table = of_match_ptr(lpc32xx_rtc_match),
},
+// SPDX-License-Identifier: GPL-2.0
/*
* Real Time Clock driver for Freescale MC13XXX PMIC
*
* (C) 2009 Sascha Hauer, Pengutronix
* (C) 2009 Uwe Kleine-Koenig, Pengutronix
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/mfd/mc13xxx.h>
return 0;
}
-static int mc13xxx_rtc_set_mmss(struct device *dev, time64_t secs)
+static int mc13xxx_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
struct mc13xxx_rtc *priv = dev_get_drvdata(dev);
unsigned int seconds, days;
unsigned int alarmseconds;
int ret;
- days = div_s64_rem(secs, SEC_PER_DAY, &seconds);
+ days = div_s64_rem(rtc_tm_to_time64(tm), SEC_PER_DAY, &seconds);
mc13xxx_lock(priv->mc13xxx);
static int mc13xxx_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alarm)
{
struct mc13xxx_rtc *priv = dev_get_drvdata(dev);
- unsigned seconds, days;
+ unsigned int seconds, days;
time64_t s1970;
int enabled, pending;
int ret;
static const struct rtc_class_ops mc13xxx_rtc_ops = {
.read_time = mc13xxx_rtc_read_time,
- .set_mmss64 = mc13xxx_rtc_set_mmss,
+ .set_time = mc13xxx_rtc_set_time,
.read_alarm = mc13xxx_rtc_read_alarm,
.set_alarm = mc13xxx_rtc_set_alarm,
.alarm_irq_enable = mc13xxx_rtc_alarm_irq_enable,
priv->mc13xxx = mc13xxx;
priv->valid = 1;
+ priv->rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(priv->rtc))
+ return PTR_ERR(priv->rtc);
platform_set_drvdata(pdev, priv);
+ priv->rtc->ops = &mc13xxx_rtc_ops;
+ /* 15bit days + hours, minutes, seconds */
+ priv->rtc->range_max = (timeu64_t)(1 << 15) * SEC_PER_DAY - 1;
+
mc13xxx_lock(mc13xxx);
mc13xxx_irq_ack(mc13xxx, MC13XXX_IRQ_RTCRST);
mc13xxx_unlock(mc13xxx);
- priv->rtc = devm_rtc_device_register(&pdev->dev, pdev->name,
- &mc13xxx_rtc_ops, THIS_MODULE);
+ ret = rtc_register_device(priv->rtc);
+ if (ret)
+ goto err_irq_request;
return 0;
if (ret) {
dev_err(&pdev->dev, "Failed to request alarm IRQ: %d: %d\n",
rtc->irq, ret);
- goto out_dispose_irq;
+ return ret;
}
device_init_wakeup(&pdev->dev, 1);
return 0;
out_free_irq:
- free_irq(rtc->irq, rtc->rtc_dev);
-out_dispose_irq:
- irq_dispose_mapping(rtc->irq);
+ free_irq(rtc->irq, rtc);
return ret;
}
{
struct mt6397_rtc *rtc = platform_get_drvdata(pdev);
- free_irq(rtc->irq, rtc->rtc_dev);
- irq_dispose_mapping(rtc->irq);
+ free_irq(rtc->irq, rtc);
return 0;
}
+// SPDX-License-Identifier: GPL-2.0
/*
* Driver for the RTC in Marvell SoCs.
- *
- * This file is licensed under the terms of the GNU General Public
- * License version 2. This program is licensed "as is" without any
- * warranty of any kind, whether express or implied.
*/
#include <linux/init.h>
rtc_reg = (bin2bcd(tm->tm_mday) << RTC_MDAY_OFFS) |
(bin2bcd(tm->tm_mon + 1) << RTC_MONTH_OFFS) |
- (bin2bcd(tm->tm_year % 100) << RTC_YEAR_OFFS);
+ (bin2bcd(tm->tm_year - 100) << RTC_YEAR_OFFS);
writel(rtc_reg, ioaddr + RTC_DATE_REG_OFFS);
return 0;
<< RTC_MONTH_OFFS;
if (alm->time.tm_year >= 0)
- rtc_reg |= (RTC_ALARM_VALID | bin2bcd(alm->time.tm_year % 100))
+ rtc_reg |= (RTC_ALARM_VALID | bin2bcd(alm->time.tm_year - 100))
<< RTC_YEAR_OFFS;
writel(rtc_reg, ioaddr + RTC_ALARM_DATE_REG_OFFS);
platform_set_drvdata(pdev, pdata);
- if (pdata->irq >= 0) {
- device_init_wakeup(&pdev->dev, 1);
- pdata->rtc = devm_rtc_device_register(&pdev->dev, pdev->name,
- &mv_rtc_alarm_ops,
- THIS_MODULE);
- } else {
- pdata->rtc = devm_rtc_device_register(&pdev->dev, pdev->name,
- &mv_rtc_ops, THIS_MODULE);
- }
+ pdata->rtc = devm_rtc_allocate_device(&pdev->dev);
if (IS_ERR(pdata->rtc)) {
ret = PTR_ERR(pdata->rtc);
goto out;
}
}
- return 0;
+ if (pdata->irq >= 0) {
+ device_init_wakeup(&pdev->dev, 1);
+ pdata->rtc->ops = &mv_rtc_alarm_ops;
+ } else {
+ pdata->rtc->ops = &mv_rtc_ops;
+ }
+
+ pdata->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
+ pdata->rtc->range_max = RTC_TIMESTAMP_END_2099;
+
+ ret = rtc_register_device(pdata->rtc);
+ if (!ret)
+ return 0;
out:
if (!IS_ERR(pdata->clk))
clk_disable_unprepare(pdata->clk);
#include <linux/slab.h>
#include <linux/interrupt.h>
#include <linux/platform_device.h>
+#include <linux/pm_wakeirq.h>
#include <linux/clk.h>
#include <linux/of.h>
#include <linux/of_device.h>
/*
* This function sets the internal RTC time based on tm in Gregorian date.
*/
-static int mxc_rtc_set_mmss(struct device *dev, time64_t time)
+static int mxc_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
- struct rtc_plat_data *pdata = dev_get_drvdata(dev);
-
- /*
- * TTC_DAYR register is 9-bit in MX1 SoC, save time and day of year only
- */
- if (is_imx1_rtc(pdata)) {
- struct rtc_time tm;
-
- rtc_time64_to_tm(time, &tm);
- tm.tm_year = 70;
- time = rtc_tm_to_time64(&tm);
- }
+ time64_t time = rtc_tm_to_time64(tm);
/* Avoid roll-over from reading the different registers */
do {
/* RTC layer */
static const struct rtc_class_ops mxc_rtc_ops = {
.read_time = mxc_rtc_read_time,
- .set_mmss64 = mxc_rtc_set_mmss,
+ .set_time = mxc_rtc_set_time,
.read_alarm = mxc_rtc_read_alarm,
.set_alarm = mxc_rtc_set_alarm,
.alarm_irq_enable = mxc_rtc_alarm_irq_enable,
static int mxc_rtc_probe(struct platform_device *pdev)
{
- struct resource *res;
struct rtc_device *rtc;
struct rtc_plat_data *pdata = NULL;
u32 reg;
else
pdata->devtype = pdev->id_entry->driver_data;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- pdata->ioaddr = devm_ioremap_resource(&pdev->dev, res);
+ pdata->ioaddr = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(pdata->ioaddr))
return PTR_ERR(pdata->ioaddr);
+ rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(rtc))
+ return PTR_ERR(rtc);
+
+ pdata->rtc = rtc;
+ rtc->ops = &mxc_rtc_ops;
+ if (is_imx1_rtc(pdata)) {
+ struct rtc_time tm;
+
+ /* 9bit days + hours minutes seconds */
+ rtc->range_max = (1 << 9) * 86400 - 1;
+
+ /*
+ * Set the start date as beginning of the current year. This can
+ * be overridden using device tree.
+ */
+ rtc_time64_to_tm(ktime_get_real_seconds(), &tm);
+ rtc->start_secs = mktime64(tm.tm_year, 1, 1, 0, 0, 0);
+ rtc->set_start_time = true;
+ } else {
+ /* 16bit days + hours minutes seconds */
+ rtc->range_max = (1 << 16) * 86400ULL - 1;
+ }
+
pdata->clk_ipg = devm_clk_get(&pdev->dev, "ipg");
if (IS_ERR(pdata->clk_ipg)) {
dev_err(&pdev->dev, "unable to get ipg clock!\n");
pdata->irq = -1;
}
- if (pdata->irq >= 0)
+ if (pdata->irq >= 0) {
device_init_wakeup(&pdev->dev, 1);
-
- rtc = devm_rtc_device_register(&pdev->dev, pdev->name, &mxc_rtc_ops,
- THIS_MODULE);
- if (IS_ERR(rtc)) {
- ret = PTR_ERR(rtc);
- goto exit_put_clk_ref;
+ ret = dev_pm_set_wake_irq(&pdev->dev, pdata->irq);
+ if (ret)
+ dev_err(&pdev->dev, "failed to enable irq wake\n");
}
- pdata->rtc = rtc;
+ ret = rtc_register_device(rtc);
+ if (ret)
+ goto exit_put_clk_ref;
return 0;
return 0;
}
-#ifdef CONFIG_PM_SLEEP
-static int mxc_rtc_suspend(struct device *dev)
-{
- struct rtc_plat_data *pdata = dev_get_drvdata(dev);
-
- if (device_may_wakeup(dev))
- enable_irq_wake(pdata->irq);
-
- return 0;
-}
-
-static int mxc_rtc_resume(struct device *dev)
-{
- struct rtc_plat_data *pdata = dev_get_drvdata(dev);
-
- if (device_may_wakeup(dev))
- disable_irq_wake(pdata->irq);
-
- return 0;
-}
-#endif
-
-static SIMPLE_DEV_PM_OPS(mxc_rtc_pm_ops, mxc_rtc_suspend, mxc_rtc_resume);
-
static struct platform_driver mxc_rtc_driver = {
.driver = {
.name = "mxc_rtc",
.of_match_table = of_match_ptr(imx_rtc_dt_ids),
- .pm = &mxc_rtc_pm_ops,
},
.id_table = imx_rtc_devtype,
.probe = mxc_rtc_probe,
#include <linux/module.h>
#include <linux/mod_devicetable.h>
#include <linux/platform_device.h>
+#include <linux/pm_wakeirq.h>
#include <linux/rtc.h>
#define SRTC_LPPDR_INIT 0x41736166 /* init for glitch detect */
return pdata->irq;
device_init_wakeup(&pdev->dev, 1);
+ ret = dev_pm_set_wake_irq(&pdev->dev, pdata->irq);
+ if (ret)
+ dev_err(&pdev->dev, "failed to enable irq wake\n");
ret = clk_prepare_enable(pdata->clk);
if (ret)
return 0;
}
-#ifdef CONFIG_PM_SLEEP
-static int mxc_rtc_suspend(struct device *dev)
-{
- struct mxc_rtc_data *pdata = dev_get_drvdata(dev);
-
- if (device_may_wakeup(dev))
- enable_irq_wake(pdata->irq);
-
- return 0;
-}
-
-static int mxc_rtc_resume(struct device *dev)
-{
- struct mxc_rtc_data *pdata = dev_get_drvdata(dev);
-
- if (device_may_wakeup(dev))
- disable_irq_wake(pdata->irq);
-
- return 0;
-}
-#endif
-
-static SIMPLE_DEV_PM_OPS(mxc_rtc_pm_ops, mxc_rtc_suspend, mxc_rtc_resume);
-
static const struct of_device_id mxc_ids[] = {
{ .compatible = "fsl,imx53-rtc", },
{}
.driver = {
.name = "mxc_rtc_v2",
.of_match_table = mxc_ids,
- .pm = &mxc_rtc_pm_ops,
},
.probe = mxc_rtc_probe,
.remove = mxc_rtc_remove,
+// SPDX-License-Identifier: GPL-2.0+
/*
* TI OMAP Real Time Clock interface for Linux
*
*
* Copyright (C) 2006 David Brownell (new RTC framework)
* Copyright (C) 2014 Johan Hovold <johan@kernel.org>
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version
- * 2 of the License, or (at your option) any later version.
*/
#include <dt-bindings/gpio/gpio.h>
}
/* this hardware doesn't support "don't care" alarm fields */
-static int tm2bcd(struct rtc_time *tm)
+static void tm2bcd(struct rtc_time *tm)
{
tm->tm_sec = bin2bcd(tm->tm_sec);
tm->tm_min = bin2bcd(tm->tm_min);
tm->tm_mday = bin2bcd(tm->tm_mday);
tm->tm_mon = bin2bcd(tm->tm_mon + 1);
-
- /* epoch == 1900 */
- if (tm->tm_year < 100 || tm->tm_year > 199)
- return -EINVAL;
tm->tm_year = bin2bcd(tm->tm_year - 100);
-
- return 0;
}
static void bcd2tm(struct rtc_time *tm)
{
struct omap_rtc *rtc = dev_get_drvdata(dev);
- if (tm2bcd(tm) < 0)
- return -EINVAL;
+ tm2bcd(tm);
local_irq_disable();
rtc_wait_not_busy(rtc);
struct omap_rtc *rtc = dev_get_drvdata(dev);
u8 reg, irqwake_reg = 0;
- if (tm2bcd(&alm->time) < 0)
- return -EINVAL;
+ tm2bcd(&alm->time);
local_irq_disable();
rtc_wait_not_busy(rtc);
omap_rtc_read_time_raw(rtc, &tm);
seconds = tm.tm_sec;
bcd2tm(&tm);
- rtc_tm_to_time(&tm, &now);
- rtc_time_to_tm(now + 1, &tm);
+ now = rtc_tm_to_time64(&tm);
+ rtc_time64_to_tm(now + 1, &tm);
- if (tm2bcd(&tm) < 0) {
- dev_err(&rtc->rtc->dev, "power off failed\n");
- rtc->type->lock(rtc);
- return;
- }
+ tm2bcd(&tm);
rtc_wait_not_busy(rtc);
}
rtc->rtc->ops = &omap_rtc_ops;
+ rtc->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
+ rtc->rtc->range_max = RTC_TIMESTAMP_END_2099;
omap_rtc_nvmem_config.priv = rtc;
/* handle periodic and alarm irqs */
return rc;
}
-int opal_tpo_alarm_irq_enable(struct device *dev, unsigned int enabled)
+static int opal_tpo_alarm_irq_enable(struct device *dev, unsigned int enabled)
{
struct rtc_wkalrm alarm = { .enabled = 0 };
+// SPDX-License-Identifier: GPL-2.0
/*
* pcap rtc code for Motorola EZX phones
*
* Copyright (c) 2009 Daniel Ribeiro <drwyrm@gmail.com>
*
* Based on Motorola's rtc.c Copyright (c) 2003-2005 Motorola
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
*/
#include <linux/kernel.h>
ezx_pcap_read(pcap_rtc->pcap, PCAP_REG_RTC_DAYA, &days);
secs += (days & PCAP_RTC_DAY_MASK) * SEC_PER_DAY;
- rtc_time_to_tm(secs, tm);
+ rtc_time64_to_tm(secs, tm);
return 0;
}
static int pcap_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm)
{
struct pcap_rtc *pcap_rtc = dev_get_drvdata(dev);
- struct rtc_time *tm = &alrm->time;
- unsigned long secs;
+ unsigned long secs = rtc_tm_to_time64(&alrm->time);
u32 tod, days;
- rtc_tm_to_time(tm, &secs);
-
tod = secs % SEC_PER_DAY;
ezx_pcap_write(pcap_rtc->pcap, PCAP_REG_RTC_TODA, tod);
ezx_pcap_read(pcap_rtc->pcap, PCAP_REG_RTC_DAY, &days);
secs += (days & PCAP_RTC_DAY_MASK) * SEC_PER_DAY;
- rtc_time_to_tm(secs, tm);
+ rtc_time64_to_tm(secs, tm);
return 0;
}
-static int pcap_rtc_set_mmss(struct device *dev, unsigned long secs)
+static int pcap_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
struct pcap_rtc *pcap_rtc = dev_get_drvdata(dev);
+ unsigned long secs = rtc_tm_to_time64(tm);
u32 tod, days;
tod = secs % SEC_PER_DAY;
static const struct rtc_class_ops pcap_rtc_ops = {
.read_time = pcap_rtc_read_time,
+ .set_time = pcap_rtc_set_time,
.read_alarm = pcap_rtc_read_alarm,
.set_alarm = pcap_rtc_set_alarm,
- .set_mmss = pcap_rtc_set_mmss,
.alarm_irq_enable = pcap_rtc_alarm_irq_enable,
};
platform_set_drvdata(pdev, pcap_rtc);
- pcap_rtc->rtc = devm_rtc_device_register(&pdev->dev, "pcap",
- &pcap_rtc_ops, THIS_MODULE);
+ pcap_rtc->rtc = devm_rtc_allocate_device(&pdev->dev);
if (IS_ERR(pcap_rtc->rtc))
return PTR_ERR(pcap_rtc->rtc);
+ pcap_rtc->rtc->ops = &pcap_rtc_ops;
+ pcap_rtc->rtc->range_max = (1 << 14) * 86400ULL - 1;
+
timer_irq = pcap_to_irq(pcap_rtc->pcap, PCAP_IRQ_1HZ);
alarm_irq = pcap_to_irq(pcap_rtc->pcap, PCAP_IRQ_TODA);
if (err)
return err;
- return 0;
+ return rtc_register_device(pcap_rtc->rtc);
}
static int __exit pcap_rtc_remove(struct platform_device *pdev)
+// SPDX-License-Identifier: GPL-2.0
/*
* An I2C driver for the PCF85063 RTC
* Copyright 2014 Rose Technology
* Author: Søren Andersen <san@rosetechnology.dk>
* Maintainers: http://www.nslu2-linux.org/
*
- * based on the other drivers in this same directory.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
+ * Copyright (C) 2019 Micro Crystal AG
+ * Author: Alexandre Belloni <alexandre.belloni@bootlin.com>
*/
#include <linux/i2c.h>
#include <linux/bcd.h>
#include <linux/rtc.h>
#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/pm_wakeirq.h>
+#include <linux/regmap.h>
/*
* Information for this driver was pulled from the following datasheets.
*
* PCF85063A -- Rev. 6 — 18 November 2015
* PCF85063TP -- Rev. 4 — 6 May 2015
-*/
+ *
+ * https://www.microcrystal.com/fileadmin/Media/Products/RTC/App.Manual/RV-8263-C7_App-Manual.pdf
+ * RV8263 -- Rev. 1.0 — January 2019
+ */
#define PCF85063_REG_CTRL1 0x00 /* status */
#define PCF85063_REG_CTRL1_CAP_SEL BIT(0)
#define PCF85063_REG_CTRL1_STOP BIT(5)
-#define PCF85063_REG_SC 0x04 /* datetime */
-#define PCF85063_REG_SC_OS 0x80
-
-static struct i2c_driver pcf85063_driver;
-
-static int pcf85063_stop_clock(struct i2c_client *client, u8 *ctrl1)
-{
- int rc;
- u8 reg;
+#define PCF85063_REG_CTRL2 0x01
+#define PCF85063_CTRL2_AF BIT(6)
+#define PCF85063_CTRL2_AIE BIT(7)
- rc = i2c_smbus_read_byte_data(client, PCF85063_REG_CTRL1);
- if (rc < 0) {
- dev_err(&client->dev, "Failing to stop the clock\n");
- return -EIO;
- }
+#define PCF85063_REG_OFFSET 0x02
+#define PCF85063_OFFSET_SIGN_BIT 6 /* 2's complement sign bit */
+#define PCF85063_OFFSET_MODE BIT(7)
+#define PCF85063_OFFSET_STEP0 4340
+#define PCF85063_OFFSET_STEP1 4069
- /* stop the clock */
- reg = rc | PCF85063_REG_CTRL1_STOP;
+#define PCF85063_REG_RAM 0x03
- rc = i2c_smbus_write_byte_data(client, PCF85063_REG_CTRL1, reg);
- if (rc < 0) {
- dev_err(&client->dev, "Failing to stop the clock\n");
- return -EIO;
- }
-
- *ctrl1 = reg;
-
- return 0;
-}
-
-static int pcf85063_start_clock(struct i2c_client *client, u8 ctrl1)
-{
- int rc;
+#define PCF85063_REG_SC 0x04 /* datetime */
+#define PCF85063_REG_SC_OS 0x80
- /* start the clock */
- ctrl1 &= ~PCF85063_REG_CTRL1_STOP;
+#define PCF85063_REG_ALM_S 0x0b
+#define PCF85063_AEN BIT(7)
- rc = i2c_smbus_write_byte_data(client, PCF85063_REG_CTRL1, ctrl1);
- if (rc < 0) {
- dev_err(&client->dev, "Failing to start the clock\n");
- return -EIO;
- }
+struct pcf85063_config {
+ struct regmap_config regmap;
+ unsigned has_alarms:1;
+ unsigned force_cap_7000:1;
+};
- return 0;
-}
+struct pcf85063 {
+ struct rtc_device *rtc;
+ struct regmap *regmap;
+};
static int pcf85063_rtc_read_time(struct device *dev, struct rtc_time *tm)
{
- struct i2c_client *client = to_i2c_client(dev);
+ struct pcf85063 *pcf85063 = dev_get_drvdata(dev);
int rc;
u8 regs[7];
* event, the access must be finished within one second. So, read all
* time/date registers in one turn.
*/
- rc = i2c_smbus_read_i2c_block_data(client, PCF85063_REG_SC,
- sizeof(regs), regs);
- if (rc != sizeof(regs)) {
- dev_err(&client->dev, "date/time register read error\n");
- return -EIO;
- }
+ rc = regmap_bulk_read(pcf85063->regmap, PCF85063_REG_SC, regs,
+ sizeof(regs));
+ if (rc)
+ return rc;
/* if the clock has lost its power it makes no sense to use its time */
if (regs[0] & PCF85063_REG_SC_OS) {
- dev_warn(&client->dev, "Power loss detected, invalid time\n");
+ dev_warn(&pcf85063->rtc->dev, "Power loss detected, invalid time\n");
return -EINVAL;
}
static int pcf85063_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
- struct i2c_client *client = to_i2c_client(dev);
+ struct pcf85063 *pcf85063 = dev_get_drvdata(dev);
int rc;
u8 regs[7];
- u8 ctrl1;
-
- if ((tm->tm_year < 100) || (tm->tm_year > 199))
- return -EINVAL;
/*
* to accurately set the time, reset the divider chain and keep it in
* reset state until all time/date registers are written
*/
- rc = pcf85063_stop_clock(client, &ctrl1);
- if (rc != 0)
+ rc = regmap_update_bits(pcf85063->regmap, PCF85063_REG_CTRL1,
+ PCF85063_REG_CTRL1_STOP,
+ PCF85063_REG_CTRL1_STOP);
+ if (rc)
return rc;
/* hours, minutes and seconds */
regs[6] = bin2bcd(tm->tm_year - 100);
/* write all registers at once */
- rc = i2c_smbus_write_i2c_block_data(client, PCF85063_REG_SC,
- sizeof(regs), regs);
- if (rc < 0) {
- dev_err(&client->dev, "date/time register write error\n");
+ rc = regmap_bulk_write(pcf85063->regmap, PCF85063_REG_SC,
+ regs, sizeof(regs));
+ if (rc)
return rc;
- }
/*
* Write the control register as a separate action since the size of
* the register space is different between the PCF85063TP and
* PCF85063A devices. The rollover point can not be used.
*/
- rc = pcf85063_start_clock(client, ctrl1);
- if (rc != 0)
- return rc;
+ return regmap_update_bits(pcf85063->regmap, PCF85063_REG_CTRL1,
+ PCF85063_REG_CTRL1_STOP, 0);
+}
+
+static int pcf85063_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
+{
+ struct pcf85063 *pcf85063 = dev_get_drvdata(dev);
+ u8 buf[4];
+ unsigned int val;
+ int ret;
+
+ ret = regmap_bulk_read(pcf85063->regmap, PCF85063_REG_ALM_S,
+ buf, sizeof(buf));
+ if (ret)
+ return ret;
+
+ alrm->time.tm_sec = bcd2bin(buf[0]);
+ alrm->time.tm_min = bcd2bin(buf[1]);
+ alrm->time.tm_hour = bcd2bin(buf[2]);
+ alrm->time.tm_mday = bcd2bin(buf[3]);
+
+ ret = regmap_read(pcf85063->regmap, PCF85063_REG_CTRL2, &val);
+ if (ret)
+ return ret;
+
+ alrm->enabled = !!(val & PCF85063_CTRL2_AIE);
return 0;
}
+static int pcf85063_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm)
+{
+ struct pcf85063 *pcf85063 = dev_get_drvdata(dev);
+ u8 buf[5];
+ int ret;
+
+ buf[0] = bin2bcd(alrm->time.tm_sec);
+ buf[1] = bin2bcd(alrm->time.tm_min);
+ buf[2] = bin2bcd(alrm->time.tm_hour);
+ buf[3] = bin2bcd(alrm->time.tm_mday);
+ buf[4] = PCF85063_AEN; /* Do not match on week day */
+
+ ret = regmap_update_bits(pcf85063->regmap, PCF85063_REG_CTRL2,
+ PCF85063_CTRL2_AIE | PCF85063_CTRL2_AF, 0);
+ if (ret)
+ return ret;
+
+ ret = regmap_bulk_write(pcf85063->regmap, PCF85063_REG_ALM_S,
+ buf, sizeof(buf));
+ if (ret)
+ return ret;
+
+ return regmap_update_bits(pcf85063->regmap, PCF85063_REG_CTRL2,
+ PCF85063_CTRL2_AIE | PCF85063_CTRL2_AF,
+ alrm->enabled ? PCF85063_CTRL2_AIE | PCF85063_CTRL2_AF : PCF85063_CTRL2_AF);
+}
+
+static int pcf85063_rtc_alarm_irq_enable(struct device *dev,
+ unsigned int enabled)
+{
+ struct pcf85063 *pcf85063 = dev_get_drvdata(dev);
+
+ return regmap_update_bits(pcf85063->regmap, PCF85063_REG_CTRL2,
+ PCF85063_CTRL2_AIE,
+ enabled ? PCF85063_CTRL2_AIE : 0);
+}
+
+static irqreturn_t pcf85063_rtc_handle_irq(int irq, void *dev_id)
+{
+ struct pcf85063 *pcf85063 = dev_id;
+ unsigned int val;
+ int err;
+
+ err = regmap_read(pcf85063->regmap, PCF85063_REG_CTRL2, &val);
+ if (err)
+ return IRQ_NONE;
+
+ if (val & PCF85063_CTRL2_AF) {
+ rtc_update_irq(pcf85063->rtc, 1, RTC_IRQF | RTC_AF);
+ regmap_update_bits(pcf85063->regmap, PCF85063_REG_CTRL2,
+ PCF85063_CTRL2_AIE | PCF85063_CTRL2_AF,
+ 0);
+ return IRQ_HANDLED;
+ }
+
+ return IRQ_NONE;
+}
+
+static int pcf85063_read_offset(struct device *dev, long *offset)
+{
+ struct pcf85063 *pcf85063 = dev_get_drvdata(dev);
+ long val;
+ u32 reg;
+ int ret;
+
+ ret = regmap_read(pcf85063->regmap, PCF85063_REG_OFFSET, ®);
+ if (ret < 0)
+ return ret;
+
+ val = sign_extend32(reg & ~PCF85063_OFFSET_MODE,
+ PCF85063_OFFSET_SIGN_BIT);
+
+ if (reg & PCF85063_OFFSET_MODE)
+ *offset = val * PCF85063_OFFSET_STEP1;
+ else
+ *offset = val * PCF85063_OFFSET_STEP0;
+
+ return 0;
+}
+
+static int pcf85063_set_offset(struct device *dev, long offset)
+{
+ struct pcf85063 *pcf85063 = dev_get_drvdata(dev);
+ s8 mode0, mode1, reg;
+ unsigned int error0, error1;
+
+ if (offset > PCF85063_OFFSET_STEP0 * 63)
+ return -ERANGE;
+ if (offset < PCF85063_OFFSET_STEP0 * -64)
+ return -ERANGE;
+
+ mode0 = DIV_ROUND_CLOSEST(offset, PCF85063_OFFSET_STEP0);
+ mode1 = DIV_ROUND_CLOSEST(offset, PCF85063_OFFSET_STEP1);
+
+ error0 = abs(offset - (mode0 * PCF85063_OFFSET_STEP0));
+ error1 = abs(offset - (mode1 * PCF85063_OFFSET_STEP1));
+ if (mode1 > 63 || mode1 < -64 || error0 < error1)
+ reg = mode0 & ~PCF85063_OFFSET_MODE;
+ else
+ reg = mode1 | PCF85063_OFFSET_MODE;
+
+ return regmap_write(pcf85063->regmap, PCF85063_REG_OFFSET, reg);
+}
+
+static int pcf85063_ioctl(struct device *dev, unsigned int cmd,
+ unsigned long arg)
+{
+ struct pcf85063 *pcf85063 = dev_get_drvdata(dev);
+ int status, ret = 0;
+
+ switch (cmd) {
+ case RTC_VL_READ:
+ ret = regmap_read(pcf85063->regmap, PCF85063_REG_SC, &status);
+ if (ret < 0)
+ return ret;
+
+ if (status & PCF85063_REG_SC_OS)
+ dev_warn(&pcf85063->rtc->dev, "Voltage low, data loss detected.\n");
+
+ status &= PCF85063_REG_SC_OS;
+
+ if (copy_to_user((void __user *)arg, &status, sizeof(int)))
+ return -EFAULT;
+
+ return 0;
+
+ case RTC_VL_CLR:
+ ret = regmap_update_bits(pcf85063->regmap, PCF85063_REG_SC,
+ PCF85063_REG_SC_OS, 0);
+
+ return ret;
+
+ default:
+ return -ENOIOCTLCMD;
+ }
+}
+
static const struct rtc_class_ops pcf85063_rtc_ops = {
.read_time = pcf85063_rtc_read_time,
- .set_time = pcf85063_rtc_set_time
+ .set_time = pcf85063_rtc_set_time,
+ .read_offset = pcf85063_read_offset,
+ .set_offset = pcf85063_set_offset,
+ .ioctl = pcf85063_ioctl,
+};
+
+static const struct rtc_class_ops pcf85063_rtc_ops_alarm = {
+ .read_time = pcf85063_rtc_read_time,
+ .set_time = pcf85063_rtc_set_time,
+ .read_offset = pcf85063_read_offset,
+ .set_offset = pcf85063_set_offset,
+ .read_alarm = pcf85063_rtc_read_alarm,
+ .set_alarm = pcf85063_rtc_set_alarm,
+ .alarm_irq_enable = pcf85063_rtc_alarm_irq_enable,
+ .ioctl = pcf85063_ioctl,
};
-static int pcf85063_load_capacitance(struct i2c_client *client)
+static int pcf85063_nvmem_read(void *priv, unsigned int offset,
+ void *val, size_t bytes)
{
- u32 load;
- int rc;
- u8 reg;
+ return regmap_read(priv, PCF85063_REG_RAM, val);
+}
- rc = i2c_smbus_read_byte_data(client, PCF85063_REG_CTRL1);
- if (rc < 0)
- return rc;
+static int pcf85063_nvmem_write(void *priv, unsigned int offset,
+ void *val, size_t bytes)
+{
+ return regmap_write(priv, PCF85063_REG_RAM, *(u8 *)val);
+}
- reg = rc;
- load = 7000;
- of_property_read_u32(client->dev.of_node, "quartz-load-femtofarads",
- &load);
+static int pcf85063_load_capacitance(struct pcf85063 *pcf85063,
+ const struct device_node *np,
+ unsigned int force_cap)
+{
+ u32 load = 7000;
+ u8 reg = 0;
+
+ if (force_cap)
+ load = force_cap;
+ else
+ of_property_read_u32(np, "quartz-load-femtofarads", &load);
switch (load) {
default:
- dev_warn(&client->dev, "Unknown quartz-load-femtofarads value: %d. Assuming 7000",
+ dev_warn(&pcf85063->rtc->dev, "Unknown quartz-load-femtofarads value: %d. Assuming 7000",
load);
/* fall through */
case 7000:
- reg &= ~PCF85063_REG_CTRL1_CAP_SEL;
break;
case 12500:
- reg |= PCF85063_REG_CTRL1_CAP_SEL;
+ reg = PCF85063_REG_CTRL1_CAP_SEL;
break;
}
- rc = i2c_smbus_write_byte_data(client, PCF85063_REG_CTRL1, reg);
-
- return rc;
+ return regmap_update_bits(pcf85063->regmap, PCF85063_REG_CTRL1,
+ PCF85063_REG_CTRL1_CAP_SEL, reg);
}
-static int pcf85063_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static const struct pcf85063_config pcf85063a_config = {
+ .regmap = {
+ .reg_bits = 8,
+ .val_bits = 8,
+ .max_register = 0x11,
+ },
+ .has_alarms = 1,
+};
+
+static const struct pcf85063_config pcf85063tp_config = {
+ .regmap = {
+ .reg_bits = 8,
+ .val_bits = 8,
+ .max_register = 0x0a,
+ },
+};
+
+static const struct pcf85063_config rv8263_config = {
+ .regmap = {
+ .reg_bits = 8,
+ .val_bits = 8,
+ .max_register = 0x11,
+ },
+ .has_alarms = 1,
+ .force_cap_7000 = 1,
+};
+
+static int pcf85063_probe(struct i2c_client *client)
{
- struct rtc_device *rtc;
+ struct pcf85063 *pcf85063;
+ unsigned int tmp;
int err;
+ const struct pcf85063_config *config = &pcf85063tp_config;
+ const void *data = of_device_get_match_data(&client->dev);
+ struct nvmem_config nvmem_cfg = {
+ .name = "pcf85063_nvram",
+ .reg_read = pcf85063_nvmem_read,
+ .reg_write = pcf85063_nvmem_write,
+ .type = NVMEM_TYPE_BATTERY_BACKED,
+ .size = 1,
+ };
dev_dbg(&client->dev, "%s\n", __func__);
- if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C))
- return -ENODEV;
+ pcf85063 = devm_kzalloc(&client->dev, sizeof(struct pcf85063),
+ GFP_KERNEL);
+ if (!pcf85063)
+ return -ENOMEM;
+
+ if (data)
+ config = data;
+
+ pcf85063->regmap = devm_regmap_init_i2c(client, &config->regmap);
+ if (IS_ERR(pcf85063->regmap))
+ return PTR_ERR(pcf85063->regmap);
- err = i2c_smbus_read_byte_data(client, PCF85063_REG_CTRL1);
- if (err < 0) {
+ i2c_set_clientdata(client, pcf85063);
+
+ err = regmap_read(pcf85063->regmap, PCF85063_REG_CTRL1, &tmp);
+ if (err) {
dev_err(&client->dev, "RTC chip is not present\n");
return err;
}
- err = pcf85063_load_capacitance(client);
+ pcf85063->rtc = devm_rtc_allocate_device(&client->dev);
+ if (IS_ERR(pcf85063->rtc))
+ return PTR_ERR(pcf85063->rtc);
+
+ err = pcf85063_load_capacitance(pcf85063, client->dev.of_node,
+ config->force_cap_7000 ? 7000 : 0);
if (err < 0)
dev_warn(&client->dev, "failed to set xtal load capacitance: %d",
err);
- rtc = devm_rtc_device_register(&client->dev,
- pcf85063_driver.driver.name,
- &pcf85063_rtc_ops, THIS_MODULE);
+ pcf85063->rtc->ops = &pcf85063_rtc_ops;
+ pcf85063->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
+ pcf85063->rtc->range_max = RTC_TIMESTAMP_END_2099;
+ pcf85063->rtc->uie_unsupported = 1;
+
+ if (config->has_alarms && client->irq > 0) {
+ err = devm_request_threaded_irq(&client->dev, client->irq,
+ NULL, pcf85063_rtc_handle_irq,
+ IRQF_TRIGGER_LOW | IRQF_ONESHOT,
+ "pcf85063", pcf85063);
+ if (err) {
+ dev_warn(&pcf85063->rtc->dev,
+ "unable to request IRQ, alarms disabled\n");
+ } else {
+ pcf85063->rtc->ops = &pcf85063_rtc_ops_alarm;
+ device_init_wakeup(&client->dev, true);
+ err = dev_pm_set_wake_irq(&client->dev, client->irq);
+ if (err)
+ dev_err(&pcf85063->rtc->dev,
+ "failed to enable irq wake\n");
+ }
+ }
+
+ nvmem_cfg.priv = pcf85063->regmap;
+ rtc_nvmem_register(pcf85063->rtc, &nvmem_cfg);
- return PTR_ERR_OR_ZERO(rtc);
+ return rtc_register_device(pcf85063->rtc);
}
-static const struct i2c_device_id pcf85063_id[] = {
- { "pcf85063", 0 },
- { }
-};
-MODULE_DEVICE_TABLE(i2c, pcf85063_id);
-
#ifdef CONFIG_OF
static const struct of_device_id pcf85063_of_match[] = {
- { .compatible = "nxp,pcf85063" },
+ { .compatible = "nxp,pcf85063", .data = &pcf85063tp_config },
+ { .compatible = "nxp,pcf85063tp", .data = &pcf85063tp_config },
+ { .compatible = "nxp,pcf85063a", .data = &pcf85063a_config },
+ { .compatible = "microcrystal,rv8263", .data = &rv8263_config },
{}
};
MODULE_DEVICE_TABLE(of, pcf85063_of_match);
.name = "rtc-pcf85063",
.of_match_table = of_match_ptr(pcf85063_of_match),
},
- .probe = pcf85063_probe,
- .id_table = pcf85063_id,
+ .probe_new = pcf85063_probe,
};
module_i2c_driver(pcf85063_driver);
+// SPDX-License-Identifier: GPL-2.0
/*
* drivers/rtc/rtc-pcf85363.c
*
* Driver for NXP PCF85363 real-time clock.
*
* Copyright (C) 2017 Eric Nelson
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * Based loosely on rtc-8583 by Russell King, Wolfram Sang and Juergen Beisert
*/
#include <linux/module.h>
#include <linux/i2c.h>
#define NVRAM_SIZE 0x40
-static struct i2c_driver pcf85363_driver;
-
struct pcf85363 {
- struct device *dev;
struct rtc_device *rtc;
struct regmap *regmap;
};
if (data)
config = data;
- if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C))
- return -ENODEV;
-
pcf85363 = devm_kzalloc(&client->dev, sizeof(struct pcf85363),
GFP_KERNEL);
if (!pcf85363)
return PTR_ERR(pcf85363->regmap);
}
- pcf85363->dev = &client->dev;
i2c_set_clientdata(client, pcf85363);
- pcf85363->rtc = devm_rtc_allocate_device(pcf85363->dev);
+ pcf85363->rtc = devm_rtc_allocate_device(&client->dev);
if (IS_ERR(pcf85363->rtc))
return PTR_ERR(pcf85363->rtc);
pcf85363->rtc->ops = &rtc_ops;
+ pcf85363->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
+ pcf85363->rtc->range_max = RTC_TIMESTAMP_END_2099;
if (client->irq > 0) {
regmap_write(pcf85363->regmap, CTRL_FLAGS, 0);
regmap_update_bits(pcf85363->regmap, CTRL_PIN_IO,
PIN_IO_INTA_OUT, PIN_IO_INTAPM);
- ret = devm_request_threaded_irq(pcf85363->dev, client->irq,
+ ret = devm_request_threaded_irq(&client->dev, client->irq,
NULL, pcf85363_rtc_handle_irq,
IRQF_TRIGGER_LOW | IRQF_ONESHOT,
"pcf85363", client);
+// SPDX-License-Identifier: GPL-2.0
/*
* PS3 RTC Driver
*
* Copyright 2009 Sony Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; version 2 of the License.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.
- * If not, see <http://www.gnu.org/licenses/>.
*/
#include <linux/kernel.h>
static int ps3_get_time(struct device *dev, struct rtc_time *tm)
{
- rtc_time_to_tm(read_rtc() + ps3_os_area_get_rtc_diff(), tm);
+ rtc_time64_to_tm(read_rtc() + ps3_os_area_get_rtc_diff(), tm);
return 0;
}
static int ps3_set_time(struct device *dev, struct rtc_time *tm)
{
- unsigned long now;
-
- rtc_tm_to_time(tm, &now);
- ps3_os_area_set_rtc_diff(now - read_rtc());
+ ps3_os_area_set_rtc_diff(rtc_tm_to_time64(tm) - read_rtc());
return 0;
}
{
struct rtc_device *rtc;
- rtc = devm_rtc_device_register(&dev->dev, "rtc-ps3", &ps3_rtc_ops,
- THIS_MODULE);
+ rtc = devm_rtc_allocate_device(&dev->dev);
if (IS_ERR(rtc))
return PTR_ERR(rtc);
+ rtc->ops = &ps3_rtc_ops;
+ rtc->range_max = U64_MAX;
+
platform_set_drvdata(dev, rtc);
- return 0;
+
+ return rtc_register_device(rtc);
}
static struct platform_driver ps3_rtc_driver = {
static irqreturn_t pxa_rtc_irq(int irq, void *dev_id)
{
- struct platform_device *pdev = to_platform_device(dev_id);
- struct pxa_rtc *pxa_rtc = platform_get_drvdata(pdev);
+ struct pxa_rtc *pxa_rtc = dev_get_drvdata(dev_id);
u32 rtsr;
unsigned long events = 0;
/* Turn off the alarm if it should not be a wake source. */
static int rk808_rtc_suspend(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct rk808_rtc *rk808_rtc = dev_get_drvdata(&pdev->dev);
+ struct rk808_rtc *rk808_rtc = dev_get_drvdata(dev);
if (device_may_wakeup(dev))
enable_irq_wake(rk808_rtc->irq);
*/
static int rk808_rtc_resume(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct rk808_rtc *rk808_rtc = dev_get_drvdata(&pdev->dev);
+ struct rk808_rtc *rk808_rtc = dev_get_drvdata(dev);
if (device_may_wakeup(dev))
disable_irq_wake(rk808_rtc->irq);
#include <linux/of_gpio.h>
#include <linux/regmap.h>
#include <linux/rtc.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
#include <linux/spi/spi.h>
/* RX-6110 Register definitions */
};
MODULE_DEVICE_TABLE(spi, rx6110_id);
+static const struct of_device_id rx6110_spi_of_match[] = {
+ { .compatible = "epson,rx6110" },
+ { },
+};
+MODULE_DEVICE_TABLE(of, rx6110_spi_of_match);
+
static struct spi_driver rx6110_driver = {
.driver = {
.name = RX6110_DRIVER_NAME,
+ .of_match_table = of_match_ptr(rx6110_spi_of_match),
},
.probe = rx6110_probe,
.remove = rx6110_remove,
t->time.tm_hour = bcd2bin(ald[1] & 0x1f) % 12
+ (ald[1] & 0x20 ? 12 : 0);
- dev_dbg(dev, "%s: date: %ptRr\n", __func__, t);
+ dev_dbg(dev, "%s: date: %ptRr\n", __func__, &t->time);
t->enabled = !!(rx8025->ctrl1 & RX8025_BIT_CTRL1_DALE);
t->pending = (ctrl2 & RX8025_BIT_CTRL2_DAFG) && t->enabled;
struct sh_rtc *rtc = dev_get_drvdata(dev);
unsigned int sec128, sec2, yr, yr100, cf_bit;
+ if (!(readb(rtc->regbase + RCR2) & RCR2_RTCEN))
+ return -EINVAL;
+
do {
unsigned int tmp;
{
struct sh_rtc *rtc;
struct resource *res;
- struct rtc_time r;
char clk_name[6];
int clk_id, ret;
rtc->clk = NULL;
}
+ rtc->rtc_dev = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(rtc->rtc_dev))
+ return PTR_ERR(rtc->rtc_dev);
+
clk_enable(rtc->clk);
rtc->capabilities = RTC_DEF_CAPABILITIES;
sh_rtc_setaie(&pdev->dev, 0);
sh_rtc_setcie(&pdev->dev, 0);
- rtc->rtc_dev = devm_rtc_device_register(&pdev->dev, "sh",
- &sh_rtc_ops, THIS_MODULE);
- if (IS_ERR(rtc->rtc_dev)) {
- ret = PTR_ERR(rtc->rtc_dev);
- goto err_unmap;
- }
-
+ rtc->rtc_dev->ops = &sh_rtc_ops;
rtc->rtc_dev->max_user_freq = 256;
- /* reset rtc to epoch 0 if time is invalid */
- if (rtc_read_time(rtc->rtc_dev, &r) < 0) {
- rtc_time_to_tm(0, &r);
- rtc_set_time(rtc->rtc_dev, &r);
+ if (rtc->capabilities & RTC_CAP_4_DIGIT_YEAR) {
+ rtc->rtc_dev->range_min = RTC_TIMESTAMP_BEGIN_1900;
+ rtc->rtc_dev->range_max = RTC_TIMESTAMP_END_9999;
+ } else {
+ rtc->rtc_dev->range_min = mktime64(1999, 1, 1, 0, 0, 0);
+ rtc->rtc_dev->range_max = mktime64(2098, 12, 31, 23, 59, 59);
}
+ ret = rtc_register_device(rtc->rtc_dev);
+ if (ret)
+ goto err_unmap;
+
device_init_wakeup(&pdev->dev, 1);
return 0;
{},
};
-const struct regmap_config sysrtc_regmap_config = {
+static const struct regmap_config sysrtc_regmap_config = {
.reg_bits = 32,
.val_bits = 32,
.fast_io = true,
#include <linux/of.h>
#include <linux/of_device.h>
#include <linux/platform_device.h>
+#include <linux/pm_wakeirq.h>
#include <linux/rtc.h>
#include <linux/clk.h>
#include <linux/mfd/syscon.h>
static int snvs_rtc_probe(struct platform_device *pdev)
{
struct snvs_rtc_data *data;
- struct resource *res;
int ret;
void __iomem *mmio;
if (IS_ERR(data->regmap)) {
dev_warn(&pdev->dev, "snvs rtc: you use old dts file, please update it\n");
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- mmio = devm_ioremap_resource(&pdev->dev, res);
+ mmio = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(mmio))
return PTR_ERR(mmio);
}
device_init_wakeup(&pdev->dev, true);
+ ret = dev_pm_set_wake_irq(&pdev->dev, data->irq);
+ if (ret)
+ dev_err(&pdev->dev, "failed to enable irq wake\n");
ret = devm_request_irq(&pdev->dev, data->irq, snvs_rtc_irq_handler,
IRQF_SHARED, "rtc alarm", &pdev->dev);
return ret;
}
-#ifdef CONFIG_PM_SLEEP
-static int snvs_rtc_suspend(struct device *dev)
-{
- struct snvs_rtc_data *data = dev_get_drvdata(dev);
-
- if (device_may_wakeup(dev))
- return enable_irq_wake(data->irq);
-
- return 0;
-}
-
-static int snvs_rtc_suspend_noirq(struct device *dev)
+static int __maybe_unused snvs_rtc_suspend_noirq(struct device *dev)
{
struct snvs_rtc_data *data = dev_get_drvdata(dev);
return 0;
}
-static int snvs_rtc_resume(struct device *dev)
-{
- struct snvs_rtc_data *data = dev_get_drvdata(dev);
-
- if (device_may_wakeup(dev))
- return disable_irq_wake(data->irq);
-
- return 0;
-}
-
-static int snvs_rtc_resume_noirq(struct device *dev)
+static int __maybe_unused snvs_rtc_resume_noirq(struct device *dev)
{
struct snvs_rtc_data *data = dev_get_drvdata(dev);
}
static const struct dev_pm_ops snvs_rtc_pm_ops = {
- .suspend = snvs_rtc_suspend,
- .suspend_noirq = snvs_rtc_suspend_noirq,
- .resume = snvs_rtc_resume,
- .resume_noirq = snvs_rtc_resume_noirq,
+ SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(snvs_rtc_suspend_noirq, snvs_rtc_resume_noirq)
};
-#define SNVS_RTC_PM_OPS (&snvs_rtc_pm_ops)
-
-#else
-
-#define SNVS_RTC_PM_OPS NULL
-
-#endif
-
static const struct of_device_id snvs_dt_ids[] = {
{ .compatible = "fsl,sec-v4.0-mon-rtc-lp", },
{ /* sentinel */ }
static struct platform_driver snvs_rtc_driver = {
.driver = {
.name = "snvs_rtc",
- .pm = SNVS_RTC_PM_OPS,
+ .pm = &snvs_rtc_pm_ops,
.of_match_table = snvs_dt_ids,
},
.probe = snvs_rtc_probe,
ret = device_init_wakeup(&pdev->dev, true);
if (rtc->data->has_wakeirq) {
rtc->wakeirq_alarm = platform_get_irq(pdev, 1);
- if (rtc->wakeirq_alarm <= 0)
- ret = rtc->wakeirq_alarm;
- else
+ if (rtc->wakeirq_alarm > 0) {
ret = dev_pm_set_dedicated_wake_irq(&pdev->dev,
rtc->wakeirq_alarm);
+ } else {
+ ret = rtc->wakeirq_alarm;
+ if (rtc->wakeirq_alarm == -EPROBE_DEFER)
+ goto err;
+ }
}
if (ret)
dev_warn(&pdev->dev, "alarm can't wake up the system: %d", ret);
+// SPDX-License-Identifier: GPL-2.0+
/*
* Freescale STMP37XX/STMP378X Real Time Clock driver
*
* Copyright 2008 Embedded Alley Solutions, Inc All Rights Reserved.
* Copyright 2011 Wolfram Sang, Pengutronix e.K.
*/
-
-/*
- * The code contained herein is licensed under the GNU General Public
- * License. You may obtain a copy of the GNU General Public License
- * Version 2 or later at the following locations:
- *
- * http://www.opensource.org/licenses/gpl-license.html
- * http://www.gnu.org/copyleft/gpl.html
- */
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/io.h>
if (ret)
return ret;
- rtc_time_to_tm(readl(rtc_data->io + STMP3XXX_RTC_SECONDS), rtc_tm);
+ rtc_time64_to_tm(readl(rtc_data->io + STMP3XXX_RTC_SECONDS), rtc_tm);
return 0;
}
-static int stmp3xxx_rtc_set_mmss(struct device *dev, unsigned long t)
+static int stmp3xxx_rtc_settime(struct device *dev, struct rtc_time *rtc_tm)
{
struct stmp3xxx_rtc_data *rtc_data = dev_get_drvdata(dev);
- writel(t, rtc_data->io + STMP3XXX_RTC_SECONDS);
+ writel(rtc_tm_to_time64(rtc_tm), rtc_data->io + STMP3XXX_RTC_SECONDS);
return stmp3xxx_wait_time(rtc_data);
}
{
struct stmp3xxx_rtc_data *rtc_data = dev_get_drvdata(dev);
- rtc_time_to_tm(readl(rtc_data->io + STMP3XXX_RTC_ALARM), &alm->time);
+ rtc_time64_to_tm(readl(rtc_data->io + STMP3XXX_RTC_ALARM), &alm->time);
return 0;
}
static int stmp3xxx_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alm)
{
- unsigned long t;
struct stmp3xxx_rtc_data *rtc_data = dev_get_drvdata(dev);
- rtc_tm_to_time(&alm->time, &t);
- writel(t, rtc_data->io + STMP3XXX_RTC_ALARM);
+ writel(rtc_tm_to_time64(&alm->time), rtc_data->io + STMP3XXX_RTC_ALARM);
stmp3xxx_alarm_irq_enable(dev, alm->enabled);
.alarm_irq_enable =
stmp3xxx_alarm_irq_enable,
.read_time = stmp3xxx_rtc_gettime,
- .set_mmss = stmp3xxx_rtc_set_mmss,
+ .set_time = stmp3xxx_rtc_settime,
.read_alarm = stmp3xxx_rtc_read_alarm,
.set_alarm = stmp3xxx_rtc_set_alarm,
};
STMP3XXX_RTC_CTRL_ALARM_IRQ_EN,
rtc_data->io + STMP3XXX_RTC_CTRL + STMP_OFFSET_REG_CLR);
- rtc_data->rtc = devm_rtc_device_register(&pdev->dev, pdev->name,
- &stmp3xxx_rtc_ops, THIS_MODULE);
+ rtc_data->rtc = devm_rtc_allocate_device(&pdev->dev);
if (IS_ERR(rtc_data->rtc))
return PTR_ERR(rtc_data->rtc);
return err;
}
+ rtc_data->rtc->ops = &stmp3xxx_rtc_ops;
+ rtc_data->rtc->range_max = U32_MAX;
+
+ err = rtc_register_device(rtc_data->rtc);
+ if (err)
+ return err;
+
stmp3xxx_wdt_register(pdev);
return 0;
}
+// SPDX-License-Identifier: GPL-2.0
/* rtc-sun4v.c: Hypervisor based RTC for SUN4V systems.
*
* Author: David S. Miller
- * License: GPL
*
* Copyright (C) 2008 David S. Miller <davem@davemloft.net>
*/
static int sun4v_read_time(struct device *dev, struct rtc_time *tm)
{
- rtc_time_to_tm(hypervisor_get_time(), tm);
+ rtc_time64_to_tm(hypervisor_get_time(), tm);
return 0;
}
static int sun4v_set_time(struct device *dev, struct rtc_time *tm)
{
- unsigned long secs;
- int err;
-
- err = rtc_tm_to_time(tm, &secs);
- if (err)
- return err;
-
- return hypervisor_set_time(secs);
+ return hypervisor_set_time(rtc_tm_to_time64(tm));
}
static const struct rtc_class_ops sun4v_rtc_ops = {
{
struct rtc_device *rtc;
- rtc = devm_rtc_device_register(&pdev->dev, "sun4v",
- &sun4v_rtc_ops, THIS_MODULE);
+ rtc = devm_rtc_allocate_device(&pdev->dev);
if (IS_ERR(rtc))
return PTR_ERR(rtc);
+ rtc->ops = &sun4v_rtc_ops;
+ rtc->range_max = U64_MAX;
platform_set_drvdata(pdev, rtc);
- return 0;
+
+ return rtc_register_device(rtc);
}
static struct platform_driver sun4v_rtc_driver = {
+// SPDX-License-Identifier: GPL-2.0+
/*
* An RTC driver for the NVIDIA Tegra 200 series internal RTC.
*
* Copyright (c) 2010, NVIDIA Corporation.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
- * more details.
- *
- * You should have received a copy of the GNU General Public License along
- * with this program; if not, write to the Free Software Foundation, Inc.,
- * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include <linux/clk.h>
spin_unlock_irqrestore(&info->tegra_rtc_lock, sl_irq_flags);
- rtc_time_to_tm(sec, tm);
+ rtc_time64_to_tm(sec, tm);
dev_vdbg(dev, "time read as %lu. %ptR\n", sec, tm);
int ret;
/* convert tm to seconds. */
- rtc_tm_to_time(tm, &sec);
+ sec = rtc_tm_to_time64(tm);
dev_vdbg(dev, "time set to %lu. %ptR\n", sec, tm);
} else {
/* alarm is enabled. */
alarm->enabled = 1;
- rtc_time_to_tm(sec, &alarm->time);
+ rtc_time64_to_tm(sec, &alarm->time);
}
tmp = readl(info->rtc_base + TEGRA_RTC_REG_INTR_STATUS);
unsigned long sec;
if (alarm->enabled)
- rtc_tm_to_time(&alarm->time, &sec);
+ sec = rtc_tm_to_time64(&alarm->time);
else
sec = 0;
info->tegra_rtc_irq = ret;
+ info->rtc_dev = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(info->rtc_dev))
+ return PTR_ERR(info->rtc_dev);
+
+ info->rtc_dev->ops = &tegra_rtc_ops;
+ info->rtc_dev->range_max = U32_MAX;
+
info->clk = devm_clk_get(&pdev->dev, NULL);
if (IS_ERR(info->clk))
return PTR_ERR(info->clk);
device_init_wakeup(&pdev->dev, 1);
- info->rtc_dev = devm_rtc_device_register(&pdev->dev,
- dev_name(&pdev->dev), &tegra_rtc_ops,
- THIS_MODULE);
- if (IS_ERR(info->rtc_dev)) {
- ret = PTR_ERR(info->rtc_dev);
- dev_err(&pdev->dev, "Unable to register device (err=%d).\n",
- ret);
- goto disable_clk;
- }
-
ret = devm_request_irq(&pdev->dev, info->tegra_rtc_irq,
tegra_rtc_irq_handler, IRQF_TRIGGER_HIGH,
dev_name(&pdev->dev), &pdev->dev);
goto disable_clk;
}
+ ret = rtc_register_device(info->rtc_dev);
+ if (ret) {
+ dev_err(&pdev->dev, "Unable to register device (err=%d).\n",
+ ret);
+ goto disable_clk;
+ }
+
dev_notice(&pdev->dev, "Tegra internal Real Time Clock\n");
return 0;
return 0;
}
-static int test_rtc_set_mmss64(struct device *dev, time64_t secs)
+static int test_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
struct rtc_test_data *rtd = dev_get_drvdata(dev);
- rtd->offset = secs - ktime_get_real_seconds();
+ rtd->offset = rtc_tm_to_time64(tm) - ktime_get_real_seconds();
return 0;
}
static const struct rtc_class_ops test_rtc_ops_noalm = {
.read_time = test_rtc_read_time,
- .set_mmss64 = test_rtc_set_mmss64,
+ .set_time = test_rtc_set_time,
.alarm_irq_enable = test_rtc_alarm_irq_enable,
};
static const struct rtc_class_ops test_rtc_ops = {
.read_time = test_rtc_read_time,
+ .set_time = test_rtc_set_time,
.read_alarm = test_rtc_read_alarm,
.set_alarm = test_rtc_set_alarm,
- .set_mmss64 = test_rtc_set_mmss64,
.alarm_irq_enable = test_rtc_alarm_irq_enable,
};
{
int i, err;
- if ((err = platform_driver_register(&test_driver)))
+ err = platform_driver_register(&test_driver);
+ if (err)
return err;
err = -ENOMEM;
spinlock_t lock;
};
-static struct tx4939rtc_plat_data *get_tx4939rtc_plat_data(struct device *dev)
-{
- return platform_get_drvdata(to_platform_device(dev));
-}
-
static int tx4939_rtc_cmd(struct tx4939_rtc_reg __iomem *rtcreg, int cmd)
{
int i = 0;
static int tx4939_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
- struct tx4939rtc_plat_data *pdata = get_tx4939rtc_plat_data(dev);
+ struct tx4939rtc_plat_data *pdata = dev_get_drvdata(dev);
struct tx4939_rtc_reg __iomem *rtcreg = pdata->rtcreg;
unsigned long secs = rtc_tm_to_time64(tm);
int i, ret;
static int tx4939_rtc_read_time(struct device *dev, struct rtc_time *tm)
{
- struct tx4939rtc_plat_data *pdata = get_tx4939rtc_plat_data(dev);
+ struct tx4939rtc_plat_data *pdata = dev_get_drvdata(dev);
struct tx4939_rtc_reg __iomem *rtcreg = pdata->rtcreg;
int i, ret;
unsigned long sec;
static int tx4939_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm)
{
- struct tx4939rtc_plat_data *pdata = get_tx4939rtc_plat_data(dev);
+ struct tx4939rtc_plat_data *pdata = dev_get_drvdata(dev);
struct tx4939_rtc_reg __iomem *rtcreg = pdata->rtcreg;
int i, ret;
unsigned long sec;
static int tx4939_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
{
- struct tx4939rtc_plat_data *pdata = get_tx4939rtc_plat_data(dev);
+ struct tx4939rtc_plat_data *pdata = dev_get_drvdata(dev);
struct tx4939_rtc_reg __iomem *rtcreg = pdata->rtcreg;
int i, ret;
unsigned long sec;
static int tx4939_rtc_alarm_irq_enable(struct device *dev, unsigned int enabled)
{
- struct tx4939rtc_plat_data *pdata = get_tx4939rtc_plat_data(dev);
+ struct tx4939rtc_plat_data *pdata = dev_get_drvdata(dev);
spin_lock_irq(&pdata->lock);
tx4939_rtc_cmd(pdata->rtcreg,
static irqreturn_t tx4939_rtc_interrupt(int irq, void *dev_id)
{
- struct tx4939rtc_plat_data *pdata = get_tx4939rtc_plat_data(dev_id);
+ struct tx4939rtc_plat_data *pdata = dev_get_drvdata(dev_id);
struct tx4939_rtc_reg __iomem *rtcreg = pdata->rtcreg;
unsigned long events = RTC_IRQF;
+// SPDX-License-Identifier: GPL-2.0+
/*
* Real Time Clock driver for Wolfson Microelectronics WM831x
*
*
* Author: Mark Brown <broonie@opensource.wolfsonmicro.com>
*
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
*/
#include <linux/module.h>
if (memcmp(time1, time2, sizeof(time1)) == 0) {
u32 time = (time1[0] << 16) | time1[1];
- rtc_time_to_tm(time, tm);
+ rtc_time64_to_tm(time, tm);
return 0;
}
/*
* Set current time and date in RTC
*/
-static int wm831x_rtc_set_mmss(struct device *dev, unsigned long time)
+static int wm831x_rtc_settime(struct device *dev, struct rtc_time *tm)
{
struct wm831x_rtc *wm831x_rtc = dev_get_drvdata(dev);
struct wm831x *wm831x = wm831x_rtc->wm831x;
struct rtc_time new_tm;
- unsigned long new_time;
+ unsigned long time, new_time;
int ret;
int count = 0;
+ time = rtc_tm_to_time64(tm);
+
ret = wm831x_reg_write(wm831x, WM831X_RTC_TIME_1,
(time >> 16) & 0xffff);
if (ret < 0) {
if (ret < 0)
return ret;
- ret = rtc_tm_to_time(&new_tm, &new_time);
- if (ret < 0) {
- dev_err(dev, "Failed to convert time: %d\n", ret);
- return ret;
- }
+ new_time = rtc_tm_to_time64(&new_tm);
/* Allow a second of change in case of tick */
if (new_time - time > 1) {
time = (data[0] << 16) | data[1];
- rtc_time_to_tm(time, &alrm->time);
+ rtc_time64_to_tm(time, &alrm->time);
ret = wm831x_reg_read(wm831x_rtc->wm831x, WM831X_RTC_CONTROL);
if (ret < 0) {
int ret;
unsigned long time;
- ret = rtc_tm_to_time(&alrm->time, &time);
- if (ret < 0) {
- dev_err(dev, "Failed to convert time: %d\n", ret);
- return ret;
- }
+ time = rtc_tm_to_time64(&alrm->time);
ret = wm831x_rtc_stop_alarm(wm831x_rtc);
if (ret < 0) {
static const struct rtc_class_ops wm831x_rtc_ops = {
.read_time = wm831x_rtc_readtime,
- .set_mmss = wm831x_rtc_set_mmss,
+ .set_time = wm831x_rtc_settime,
.read_alarm = wm831x_rtc_readalarm,
.set_alarm = wm831x_rtc_setalarm,
.alarm_irq_enable = wm831x_rtc_alarm_irq_enable,
/* Turn off the alarm if it should not be a wake source. */
static int wm831x_rtc_suspend(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct wm831x_rtc *wm831x_rtc = dev_get_drvdata(&pdev->dev);
+ struct wm831x_rtc *wm831x_rtc = dev_get_drvdata(dev);
int ret, enable;
- if (wm831x_rtc->alarm_enabled && device_may_wakeup(&pdev->dev))
+ if (wm831x_rtc->alarm_enabled && device_may_wakeup(dev))
enable = WM831X_RTC_ALM_ENA;
else
enable = 0;
ret = wm831x_set_bits(wm831x_rtc->wm831x, WM831X_RTC_CONTROL,
WM831X_RTC_ALM_ENA, enable);
if (ret != 0)
- dev_err(&pdev->dev, "Failed to update RTC alarm: %d\n", ret);
+ dev_err(dev, "Failed to update RTC alarm: %d\n", ret);
return 0;
}
*/
static int wm831x_rtc_resume(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct wm831x_rtc *wm831x_rtc = dev_get_drvdata(&pdev->dev);
+ struct wm831x_rtc *wm831x_rtc = dev_get_drvdata(dev);
int ret;
if (wm831x_rtc->alarm_enabled) {
ret = wm831x_rtc_start_alarm(wm831x_rtc);
if (ret != 0)
- dev_err(&pdev->dev,
- "Failed to restart RTC alarm: %d\n", ret);
+ dev_err(dev, "Failed to restart RTC alarm: %d\n", ret);
}
return 0;
/* Unconditionally disable the alarm */
static int wm831x_rtc_freeze(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct wm831x_rtc *wm831x_rtc = dev_get_drvdata(&pdev->dev);
+ struct wm831x_rtc *wm831x_rtc = dev_get_drvdata(dev);
int ret;
ret = wm831x_set_bits(wm831x_rtc->wm831x, WM831X_RTC_CONTROL,
WM831X_RTC_ALM_ENA, 0);
if (ret != 0)
- dev_err(&pdev->dev, "Failed to stop RTC alarm: %d\n", ret);
+ dev_err(dev, "Failed to stop RTC alarm: %d\n", ret);
return 0;
}
ret = wm831x_reg_read(wm831x, WM831X_RTC_CONTROL);
if (ret < 0) {
dev_err(&pdev->dev, "Failed to read RTC control: %d\n", ret);
- goto err;
+ return ret;
}
if (ret & WM831X_RTC_ALM_ENA)
wm831x_rtc->alarm_enabled = 1;
device_init_wakeup(&pdev->dev, 1);
- wm831x_rtc->rtc = devm_rtc_device_register(&pdev->dev, "wm831x",
- &wm831x_rtc_ops, THIS_MODULE);
- if (IS_ERR(wm831x_rtc->rtc)) {
- ret = PTR_ERR(wm831x_rtc->rtc);
- goto err;
- }
+ wm831x_rtc->rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(wm831x_rtc->rtc))
+ return PTR_ERR(wm831x_rtc->rtc);
+
+ wm831x_rtc->rtc->ops = &wm831x_rtc_ops;
+ wm831x_rtc->rtc->range_max = U32_MAX;
+
+ ret = rtc_register_device(wm831x_rtc->rtc);
+ if (ret)
+ return ret;
ret = devm_request_threaded_irq(&pdev->dev, alm_irq, NULL,
wm831x_alm_irq,
wm831x_rtc_add_randomness(wm831x);
return 0;
-
-err:
- return ret;
}
static const struct dev_pm_ops wm831x_rtc_pm_ops = {
#ifdef CONFIG_PM_SLEEP
static int wm8350_rtc_suspend(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct wm8350 *wm8350 = dev_get_drvdata(&pdev->dev);
+ struct wm8350 *wm8350 = dev_get_drvdata(dev);
int ret = 0;
u16 reg;
reg & WM8350_RTC_ALMSTS) {
ret = wm8350_rtc_stop_alarm(wm8350);
if (ret != 0)
- dev_err(&pdev->dev, "Failed to stop RTC alarm: %d\n",
- ret);
+ dev_err(dev, "Failed to stop RTC alarm: %d\n", ret);
}
return ret;
static int wm8350_rtc_resume(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct wm8350 *wm8350 = dev_get_drvdata(&pdev->dev);
+ struct wm8350 *wm8350 = dev_get_drvdata(dev);
int ret;
if (wm8350->rtc.alarm_enabled) {
ret = wm8350_rtc_start_alarm(wm8350);
if (ret != 0)
- dev_err(&pdev->dev,
- "Failed to restart RTC alarm: %d\n", ret);
+ dev_err(dev, "Failed to restart RTC alarm: %d\n", ret);
}
return 0;
};
MODULE_DEVICE_TABLE(i2c, x1205_id);
+static const struct of_device_id x1205_dt_ids[] = {
+ { .compatible = "xircom,x1205", },
+ {},
+};
+MODULE_DEVICE_TABLE(of, x1205_dt_ids);
+
static struct i2c_driver x1205_driver = {
.driver = {
.name = "rtc-x1205",
+ .of_match_table = x1205_dt_ids,
},
.probe = x1205_probe,
.remove = x1205_remove,
+// SPDX-License-Identifier: GPL-2.0+
/*
* APM X-Gene SoC Real Time Clock Driver
*
* Copyright (c) 2014, Applied Micro Circuits Corporation
* Author: Rameshwar Prasad Sahu <rsahu@apm.com>
* Loc Ho <lho@apm.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- *
*/
+#include <linux/clk.h>
+#include <linux/delay.h>
#include <linux/init.h>
+#include <linux/io.h>
#include <linux/module.h>
#include <linux/of.h>
#include <linux/platform_device.h>
-#include <linux/io.h>
-#include <linux/slab.h>
-#include <linux/clk.h>
-#include <linux/delay.h>
#include <linux/rtc.h>
+#include <linux/slab.h>
/* RTC CSR Registers */
#define RTC_CCVR 0x00
struct xgene_rtc_dev {
struct rtc_device *rtc;
struct device *dev;
- unsigned long alarm_time;
void __iomem *csr_base;
struct clk *clk;
unsigned int irq_wake;
{
struct xgene_rtc_dev *pdata = dev_get_drvdata(dev);
- rtc_time_to_tm(readl(pdata->csr_base + RTC_CCVR), tm);
+ rtc_time64_to_tm(readl(pdata->csr_base + RTC_CCVR), tm);
return 0;
}
-static int xgene_rtc_set_mmss(struct device *dev, unsigned long secs)
+static int xgene_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
struct xgene_rtc_dev *pdata = dev_get_drvdata(dev);
* NOTE: After the following write, the RTC_CCVR is only reflected
* after the update cycle of 1 seconds.
*/
- writel((u32) secs, pdata->csr_base + RTC_CLR);
+ writel((u32)rtc_tm_to_time64(tm), pdata->csr_base + RTC_CLR);
readl(pdata->csr_base + RTC_CLR); /* Force a barrier */
return 0;
{
struct xgene_rtc_dev *pdata = dev_get_drvdata(dev);
- rtc_time_to_tm(pdata->alarm_time, &alrm->time);
+ /* If possible, CMR should be read here */
+ rtc_time64_to_tm(0, &alrm->time);
alrm->enabled = readl(pdata->csr_base + RTC_CCR) & RTC_CCR_IE;
return 0;
static int xgene_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm)
{
struct xgene_rtc_dev *pdata = dev_get_drvdata(dev);
- unsigned long alarm_time;
- rtc_tm_to_time(&alrm->time, &alarm_time);
- pdata->alarm_time = alarm_time;
- writel((u32) pdata->alarm_time, pdata->csr_base + RTC_CMR);
+ writel((u32)rtc_tm_to_time64(&alrm->time), pdata->csr_base + RTC_CMR);
xgene_rtc_alarm_irq_enable(dev, alrm->enabled);
static const struct rtc_class_ops xgene_rtc_ops = {
.read_time = xgene_rtc_read_time,
- .set_mmss = xgene_rtc_set_mmss,
+ .set_time = xgene_rtc_set_time,
.read_alarm = xgene_rtc_read_alarm,
.set_alarm = xgene_rtc_set_alarm,
.alarm_irq_enable = xgene_rtc_alarm_irq_enable,
static irqreturn_t xgene_rtc_interrupt(int irq, void *id)
{
- struct xgene_rtc_dev *pdata = (struct xgene_rtc_dev *) id;
+ struct xgene_rtc_dev *pdata = id;
/* Check if interrupt asserted */
if (!(readl(pdata->csr_base + RTC_STAT) & RTC_STAT_BIT))
if (IS_ERR(pdata->csr_base))
return PTR_ERR(pdata->csr_base);
+ pdata->rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(pdata->rtc))
+ return PTR_ERR(pdata->rtc);
+
irq = platform_get_irq(pdev, 0);
if (irq < 0) {
dev_err(&pdev->dev, "No IRQ resource\n");
return ret;
}
- pdata->rtc = devm_rtc_device_register(&pdev->dev, pdev->name,
- &xgene_rtc_ops, THIS_MODULE);
- if (IS_ERR(pdata->rtc)) {
- clk_disable_unprepare(pdata->clk);
- return PTR_ERR(pdata->rtc);
- }
-
/* HW does not support update faster than 1 seconds */
pdata->rtc->uie_unsupported = 1;
+ pdata->rtc->ops = &xgene_rtc_ops;
+ pdata->rtc->range_max = U32_MAX;
+
+ ret = rtc_register_device(pdata->rtc);
+ if (ret) {
+ clk_disable_unprepare(pdata->clk);
+ return ret;
+ }
return 0;
}
+// SPDX-License-Identifier: GPL-2.0
/*
* Xilinx Zynq Ultrascale+ MPSoC Real Time Clock Driver
*
* Copyright (C) 2015 Xilinx, Inc.
*
- * This program is free software; you can redistribute it and/or modify it
- * under the terms and conditions of the GNU General Public License,
- * version 2, as published by the Free Software Foundation.
- *
- * This program is distributed in the hope it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
- * more details.
- *
- * You should have received a copy of the GNU General Public License along with
- * this program. If not, see <http://www.gnu.org/licenses/>.
- *
*/
#include <linux/delay.h>
+// SPDX-License-Identifier: GPL-2.0
/*
* RTC subsystem, sysfs interface
*
* Copyright (C) 2005 Tower Technologies
* Author: Alessandro Zummo <a.zummo@towertech.it>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+ */
#include <linux/module.h>
#include <linux/rtc.h>
#include "rtc-core.h"
-
/* device attributes */
/*
static ssize_t
max_user_freq_store(struct device *dev, struct device_attribute *attr,
- const char *buf, size_t n)
+ const char *buf, size_t n)
{
struct rtc_device *rtc = to_rtc_device(dev);
unsigned long val;
{
#ifdef CONFIG_RTC_HCTOSYS_DEVICE
if (rtc_hctosys_ret == 0 &&
- strcmp(dev_name(&to_rtc_device(dev)->dev),
- CONFIG_RTC_HCTOSYS_DEVICE) == 0)
+ strcmp(dev_name(&to_rtc_device(dev)->dev),
+ CONFIG_RTC_HCTOSYS_DEVICE) == 0)
return sprintf(buf, "1\n");
- else
#endif
- return sprintf(buf, "0\n");
+ return sprintf(buf, "0\n");
}
static DEVICE_ATTR_RO(hctosys);
if (*buf_ptr == '=') {
buf_ptr++;
push = 1;
- } else
+ } else {
adjust = 1;
+ }
}
retval = kstrtos64(buf_ptr, 0, &alarm);
if (retval)
return retval;
- if (adjust) {
+ if (adjust)
alarm += now;
- }
if (alarm > now || push) {
/* Avoid accidentally clobbering active alarms; we can't
* entirely prevent that here, without even the minimal
-/*
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 as published by
- * the Free Software Foundation.
- *
- */
+// SPDX-License-Identifier: GPL-2.0
#include <linux/rtc.h>
#include <linux/time.h>
if (!rtc)
goto out_err;
- if (!rtc->ops || (!rtc->ops->set_time && !rtc->ops->set_mmss64 &&
- !rtc->ops->set_mmss))
+ if (!rtc->ops || !rtc->ops->set_time)
goto out_close;
/* Compute the value of tv_nsec we require the caller to supply in
rtc_time64_to_tm(to_set.tv_sec, &tm);
- /* rtc_hctosys exclusively uses UTC, so we call set_time here, not
- * set_mmss.
- */
err = rtc_set_time(rtc, &tm);
out_close:
static bool pxa2xx_spi_idma_filter(struct dma_chan *chan, void *param)
{
- struct device *dev = param;
-
- if (dev != chan->device->dev->parent)
- return false;
-
- return true;
+ return param == chan->device->dev;
}
#endif /* CONFIG_PCI */
source "drivers/staging/vc04_services/Kconfig"
-source "drivers/staging/vboxvideo/Kconfig"
-
source "drivers/staging/pi433/Kconfig"
source "drivers/staging/mt7621-pci/Kconfig"
obj-$(CONFIG_KS7010) += ks7010/
obj-$(CONFIG_GREYBUS) += greybus/
obj-$(CONFIG_BCM2835_VCHIQ) += vc04_services/
-obj-$(CONFIG_DRM_VBOXVIDEO) += vboxvideo/
obj-$(CONFIG_PI433) += pi433/
obj-$(CONFIG_PCI_MT7621) += mt7621-pci/
obj-$(CONFIG_PCI_MT7621_PHY) += mt7621-pci-phy/
+++ /dev/null
-# SPDX-License-Identifier: GPL-2.0
-config DRM_VBOXVIDEO
- tristate "Virtual Box Graphics Card"
- depends on DRM && X86 && PCI
- select DRM_KMS_HELPER
- select DRM_TTM
- select GENERIC_ALLOCATOR
- help
- This is a KMS driver for the virtual Graphics Card used in
- Virtual Box virtual machines.
-
- Although it is possible to build this driver built-in to the
- kernel, it is advised to build it as a module, so that it can
- be updated independently of the kernel. Select M to build this
- driver as a module and add support for these devices via drm/kms
- interfaces.
+++ /dev/null
-# SPDX-License-Identifier: GPL-2.0
-vboxvideo-y := hgsmi_base.o modesetting.o vbva_base.o \
- vbox_drv.o vbox_fb.o vbox_hgsmi.o vbox_irq.o vbox_main.o \
- vbox_mode.o vbox_prime.o vbox_ttm.o
-
-obj-$(CONFIG_DRM_VBOXVIDEO) += vboxvideo.o
+++ /dev/null
-TODO:
--Get a full review from the drm-maintainers on dri-devel done on this driver
--Drop all the logic around initial_mode_queried, the master_set and
- master_drop callbacks and everything related to this. kms clients can handle
- hotplugs.
--Extend this TODO with the results of that review
-
-Please send any patches to Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
-Hans de Goede <hdegoede@redhat.com>, Michael Thayer <michael.thayer@oracle.com>
-and dri-devel@lists.freedesktop.org .
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/* Copyright (C) 2006-2017 Oracle Corporation */
-
-#include <linux/vbox_err.h>
-#include "vbox_drv.h"
-#include "vboxvideo_guest.h"
-#include "vboxvideo_vbe.h"
-#include "hgsmi_channels.h"
-#include "hgsmi_ch_setup.h"
-
-/**
- * Inform the host of the location of the host flags in VRAM via an HGSMI cmd.
- * Return: 0 or negative errno value.
- * @ctx: The context of the guest heap to use.
- * @location: The offset chosen for the flags within guest VRAM.
- */
-int hgsmi_report_flags_location(struct gen_pool *ctx, u32 location)
-{
- struct hgsmi_buffer_location *p;
-
- p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_HGSMI,
- HGSMI_CC_HOST_FLAGS_LOCATION);
- if (!p)
- return -ENOMEM;
-
- p->buf_location = location;
- p->buf_len = sizeof(struct hgsmi_host_flags);
-
- hgsmi_buffer_submit(ctx, p);
- hgsmi_buffer_free(ctx, p);
-
- return 0;
-}
-
-/**
- * Notify the host of HGSMI-related guest capabilities via an HGSMI command.
- * Return: 0 or negative errno value.
- * @ctx: The context of the guest heap to use.
- * @caps: The capabilities to report, see vbva_caps.
- */
-int hgsmi_send_caps_info(struct gen_pool *ctx, u32 caps)
-{
- struct vbva_caps *p;
-
- p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA, VBVA_INFO_CAPS);
- if (!p)
- return -ENOMEM;
-
- p->rc = VERR_NOT_IMPLEMENTED;
- p->caps = caps;
-
- hgsmi_buffer_submit(ctx, p);
-
- WARN_ON_ONCE(p->rc < 0);
-
- hgsmi_buffer_free(ctx, p);
-
- return 0;
-}
-
-int hgsmi_test_query_conf(struct gen_pool *ctx)
-{
- u32 value = 0;
- int ret;
-
- ret = hgsmi_query_conf(ctx, U32_MAX, &value);
- if (ret)
- return ret;
-
- return value == U32_MAX ? 0 : -EIO;
-}
-
-/**
- * Query the host for an HGSMI configuration parameter via an HGSMI command.
- * Return: 0 or negative errno value.
- * @ctx: The context containing the heap used.
- * @index: The index of the parameter to query.
- * @value_ret: Where to store the value of the parameter on success.
- */
-int hgsmi_query_conf(struct gen_pool *ctx, u32 index, u32 *value_ret)
-{
- struct vbva_conf32 *p;
-
- p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA,
- VBVA_QUERY_CONF32);
- if (!p)
- return -ENOMEM;
-
- p->index = index;
- p->value = U32_MAX;
-
- hgsmi_buffer_submit(ctx, p);
-
- *value_ret = p->value;
-
- hgsmi_buffer_free(ctx, p);
-
- return 0;
-}
-
-/**
- * Pass the host a new mouse pointer shape via an HGSMI command.
- * Return: 0 or negative errno value.
- * @ctx: The context containing the heap to be used.
- * @flags: Cursor flags.
- * @hot_x: Horizontal position of the hot spot.
- * @hot_y: Vertical position of the hot spot.
- * @width: Width in pixels of the cursor.
- * @height: Height in pixels of the cursor.
- * @pixels: Pixel data, @see VMMDevReqMousePointer for the format.
- * @len: Size in bytes of the pixel data.
- */
-int hgsmi_update_pointer_shape(struct gen_pool *ctx, u32 flags,
- u32 hot_x, u32 hot_y, u32 width, u32 height,
- u8 *pixels, u32 len)
-{
- struct vbva_mouse_pointer_shape *p;
- u32 pixel_len = 0;
- int rc;
-
- if (flags & VBOX_MOUSE_POINTER_SHAPE) {
- /*
- * Size of the pointer data:
- * sizeof (AND mask) + sizeof (XOR_MASK)
- */
- pixel_len = ((((width + 7) / 8) * height + 3) & ~3) +
- width * 4 * height;
- if (pixel_len > len)
- return -EINVAL;
-
- /*
- * If shape is supplied, then always create the pointer visible.
- * See comments in 'vboxUpdatePointerShape'
- */
- flags |= VBOX_MOUSE_POINTER_VISIBLE;
- }
-
- p = hgsmi_buffer_alloc(ctx, sizeof(*p) + pixel_len, HGSMI_CH_VBVA,
- VBVA_MOUSE_POINTER_SHAPE);
- if (!p)
- return -ENOMEM;
-
- p->result = VINF_SUCCESS;
- p->flags = flags;
- p->hot_X = hot_x;
- p->hot_y = hot_y;
- p->width = width;
- p->height = height;
- if (pixel_len)
- memcpy(p->data, pixels, pixel_len);
-
- hgsmi_buffer_submit(ctx, p);
-
- switch (p->result) {
- case VINF_SUCCESS:
- rc = 0;
- break;
- case VERR_NO_MEMORY:
- rc = -ENOMEM;
- break;
- case VERR_NOT_SUPPORTED:
- rc = -EBUSY;
- break;
- default:
- rc = -EINVAL;
- }
-
- hgsmi_buffer_free(ctx, p);
-
- return rc;
-}
-
-/**
- * Report the guest cursor position. The host may wish to use this information
- * to re-position its own cursor (though this is currently unlikely). The
- * current host cursor position is returned.
- * Return: 0 or negative errno value.
- * @ctx: The context containing the heap used.
- * @report_position: Are we reporting a position?
- * @x: Guest cursor X position.
- * @y: Guest cursor Y position.
- * @x_host: Host cursor X position is stored here. Optional.
- * @y_host: Host cursor Y position is stored here. Optional.
- */
-int hgsmi_cursor_position(struct gen_pool *ctx, bool report_position,
- u32 x, u32 y, u32 *x_host, u32 *y_host)
-{
- struct vbva_cursor_position *p;
-
- p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA,
- VBVA_CURSOR_POSITION);
- if (!p)
- return -ENOMEM;
-
- p->report_position = report_position;
- p->x = x;
- p->y = y;
-
- hgsmi_buffer_submit(ctx, p);
-
- *x_host = p->x;
- *y_host = p->y;
-
- hgsmi_buffer_free(ctx, p);
-
- return 0;
-}
+++ /dev/null
-/* SPDX-License-Identifier: MIT */
-/* Copyright (C) 2006-2017 Oracle Corporation */
-
-#ifndef __HGSMI_CH_SETUP_H__
-#define __HGSMI_CH_SETUP_H__
-
-/*
- * Tell the host the location of hgsmi_host_flags structure, where the host
- * can write information about pending buffers, etc, and which can be quickly
- * polled by the guest without a need to port IO.
- */
-#define HGSMI_CC_HOST_FLAGS_LOCATION 0
-
-struct hgsmi_buffer_location {
- u32 buf_location;
- u32 buf_len;
-} __packed;
-
-/* HGSMI setup and configuration data structures. */
-
-#define HGSMIHOSTFLAGS_COMMANDS_PENDING 0x01u
-#define HGSMIHOSTFLAGS_IRQ 0x02u
-#define HGSMIHOSTFLAGS_VSYNC 0x10u
-#define HGSMIHOSTFLAGS_HOTPLUG 0x20u
-#define HGSMIHOSTFLAGS_CURSOR_CAPABILITIES 0x40u
-
-struct hgsmi_host_flags {
- u32 host_flags;
- u32 reserved[3];
-} __packed;
-
-#endif
+++ /dev/null
-/* SPDX-License-Identifier: MIT */
-/* Copyright (C) 2006-2017 Oracle Corporation */
-
-#ifndef __HGSMI_CHANNELS_H__
-#define __HGSMI_CHANNELS_H__
-
-/*
- * Each channel has an 8 bit identifier. There are a number of predefined
- * (hardcoded) channels.
- *
- * HGSMI_CH_HGSMI channel can be used to map a string channel identifier
- * to a free 16 bit numerical value. values are allocated in range
- * [HGSMI_CH_STRING_FIRST;HGSMI_CH_STRING_LAST].
- */
-
-/* A reserved channel value */
-#define HGSMI_CH_RESERVED 0x00
-/* HGCMI: setup and configuration */
-#define HGSMI_CH_HGSMI 0x01
-/* Graphics: VBVA */
-#define HGSMI_CH_VBVA 0x02
-/* Graphics: Seamless with a single guest region */
-#define HGSMI_CH_SEAMLESS 0x03
-/* Graphics: Seamless with separate host windows */
-#define HGSMI_CH_SEAMLESS2 0x04
-/* Graphics: OpenGL HW acceleration */
-#define HGSMI_CH_OPENGL 0x05
-
-/* The first channel index to be used for string mappings (inclusive) */
-#define HGSMI_CH_STRING_FIRST 0x20
-/* The last channel index for string mappings (inclusive) */
-#define HGSMI_CH_STRING_LAST 0xff
-
-#endif
+++ /dev/null
-/* SPDX-License-Identifier: MIT */
-/* Copyright (C) 2006-2017 Oracle Corporation */
-
-#ifndef __HGSMI_DEFS_H__
-#define __HGSMI_DEFS_H__
-
-/* Buffer sequence type mask. */
-#define HGSMI_BUFFER_HEADER_F_SEQ_MASK 0x03
-/* Single buffer, not a part of a sequence. */
-#define HGSMI_BUFFER_HEADER_F_SEQ_SINGLE 0x00
-/* The first buffer in a sequence. */
-#define HGSMI_BUFFER_HEADER_F_SEQ_START 0x01
-/* A middle buffer in a sequence. */
-#define HGSMI_BUFFER_HEADER_F_SEQ_CONTINUE 0x02
-/* The last buffer in a sequence. */
-#define HGSMI_BUFFER_HEADER_F_SEQ_END 0x03
-
-/* 16 bytes buffer header. */
-struct hgsmi_buffer_header {
- u32 data_size; /* Size of data that follows the header. */
- u8 flags; /* HGSMI_BUFFER_HEADER_F_* */
- u8 channel; /* The channel the data must be routed to. */
- u16 channel_info; /* Opaque to the HGSMI, used by the channel. */
-
- union {
- /* Opaque placeholder to make the union 8 bytes. */
- u8 header_data[8];
-
- /* HGSMI_BUFFER_HEADER_F_SEQ_SINGLE */
- struct {
- u32 reserved1; /* A reserved field, initialize to 0. */
- u32 reserved2; /* A reserved field, initialize to 0. */
- } buffer;
-
- /* HGSMI_BUFFER_HEADER_F_SEQ_START */
- struct {
- /* Must be the same for all buffers in the sequence. */
- u32 sequence_number;
- /* The total size of the sequence. */
- u32 sequence_size;
- } sequence_start;
-
- /*
- * HGSMI_BUFFER_HEADER_F_SEQ_CONTINUE and
- * HGSMI_BUFFER_HEADER_F_SEQ_END
- */
- struct {
- /* Must be the same for all buffers in the sequence. */
- u32 sequence_number;
- /* Data offset in the entire sequence. */
- u32 sequence_offset;
- } sequence_continue;
- } u;
-} __packed;
-
-/* 8 bytes buffer tail. */
-struct hgsmi_buffer_tail {
- /* Reserved, must be initialized to 0. */
- u32 reserved;
- /*
- * One-at-a-Time Hash: http://www.burtleburtle.net/bob/hash/doobs.html
- * Over the header, offset and for first 4 bytes of the tail.
- */
- u32 checksum;
-} __packed;
-
-/*
- * The size of the array of channels. Array indexes are u8.
- * Note: the value must not be changed.
- */
-#define HGSMI_NUMBER_OF_CHANNELS 0x100
-
-#endif
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/* Copyright (C) 2006-2017 Oracle Corporation */
-
-#include <linux/vbox_err.h>
-#include "vbox_drv.h"
-#include "vboxvideo_guest.h"
-#include "vboxvideo_vbe.h"
-#include "hgsmi_channels.h"
-
-/**
- * Set a video mode via an HGSMI request. The views must have been
- * initialised first using @a VBoxHGSMISendViewInfo and if the mode is being
- * set on the first display then it must be set first using registers.
- * @ctx: The context containing the heap to use.
- * @display: The screen number.
- * @origin_x: The horizontal displacement relative to the first scrn.
- * @origin_y: The vertical displacement relative to the first screen.
- * @start_offset: The offset of the visible area of the framebuffer
- * relative to the framebuffer start.
- * @pitch: The offset in bytes between the starts of two adjecent
- * scan lines in video RAM.
- * @width: The mode width.
- * @height: The mode height.
- * @bpp: The colour depth of the mode.
- * @flags: Flags.
- */
-void hgsmi_process_display_info(struct gen_pool *ctx, u32 display,
- s32 origin_x, s32 origin_y, u32 start_offset,
- u32 pitch, u32 width, u32 height,
- u16 bpp, u16 flags)
-{
- struct vbva_infoscreen *p;
-
- p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA,
- VBVA_INFO_SCREEN);
- if (!p)
- return;
-
- p->view_index = display;
- p->origin_x = origin_x;
- p->origin_y = origin_y;
- p->start_offset = start_offset;
- p->line_size = pitch;
- p->width = width;
- p->height = height;
- p->bits_per_pixel = bpp;
- p->flags = flags;
-
- hgsmi_buffer_submit(ctx, p);
- hgsmi_buffer_free(ctx, p);
-}
-
-/**
- * Report the rectangle relative to which absolute pointer events should be
- * expressed. This information remains valid until the next VBVA resize event
- * for any screen, at which time it is reset to the bounding rectangle of all
- * virtual screens.
- * Return: 0 or negative errno value.
- * @ctx: The context containing the heap to use.
- * @origin_x: Upper left X co-ordinate relative to the first screen.
- * @origin_y: Upper left Y co-ordinate relative to the first screen.
- * @width: Rectangle width.
- * @height: Rectangle height.
- */
-int hgsmi_update_input_mapping(struct gen_pool *ctx, s32 origin_x, s32 origin_y,
- u32 width, u32 height)
-{
- struct vbva_report_input_mapping *p;
-
- p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA,
- VBVA_REPORT_INPUT_MAPPING);
- if (!p)
- return -ENOMEM;
-
- p->x = origin_x;
- p->y = origin_y;
- p->cx = width;
- p->cy = height;
-
- hgsmi_buffer_submit(ctx, p);
- hgsmi_buffer_free(ctx, p);
-
- return 0;
-}
-
-/**
- * Get most recent video mode hints.
- * Return: 0 or negative errno value.
- * @ctx: The context containing the heap to use.
- * @screens: The number of screens to query hints for, starting at 0.
- * @hints: Array of vbva_modehint structures for receiving the hints.
- */
-int hgsmi_get_mode_hints(struct gen_pool *ctx, unsigned int screens,
- struct vbva_modehint *hints)
-{
- struct vbva_query_mode_hints *p;
- size_t size;
-
- if (WARN_ON(!hints))
- return -EINVAL;
-
- size = screens * sizeof(struct vbva_modehint);
- p = hgsmi_buffer_alloc(ctx, sizeof(*p) + size, HGSMI_CH_VBVA,
- VBVA_QUERY_MODE_HINTS);
- if (!p)
- return -ENOMEM;
-
- p->hints_queried_count = screens;
- p->hint_structure_guest_size = sizeof(struct vbva_modehint);
- p->rc = VERR_NOT_SUPPORTED;
-
- hgsmi_buffer_submit(ctx, p);
-
- if (p->rc < 0) {
- hgsmi_buffer_free(ctx, p);
- return -EIO;
- }
-
- memcpy(hints, ((u8 *)p) + sizeof(struct vbva_query_mode_hints), size);
- hgsmi_buffer_free(ctx, p);
-
- return 0;
-}
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/*
- * Copyright (C) 2013-2017 Oracle Corporation
- * This file is based on ast_drv.c
- * Copyright 2012 Red Hat Inc.
- * Authors: Dave Airlie <airlied@redhat.com>
- * Michael Thayer <michael.thayer@oracle.com,
- * Hans de Goede <hdegoede@redhat.com>
- */
-#include <linux/console.h>
-#include <linux/module.h>
-#include <linux/pci.h>
-#include <linux/vt_kern.h>
-
-#include <drm/drm_crtc_helper.h>
-#include <drm/drm_drv.h>
-#include <drm/drm_file.h>
-#include <drm/drm_ioctl.h>
-
-#include "vbox_drv.h"
-
-static int vbox_modeset = -1;
-
-MODULE_PARM_DESC(modeset, "Disable/Enable modesetting");
-module_param_named(modeset, vbox_modeset, int, 0400);
-
-static struct drm_driver driver;
-
-static const struct pci_device_id pciidlist[] = {
- { PCI_DEVICE(0x80ee, 0xbeef) },
- { }
-};
-MODULE_DEVICE_TABLE(pci, pciidlist);
-
-static struct drm_fb_helper_funcs vbox_fb_helper_funcs = {
- .fb_probe = vboxfb_create,
-};
-
-static int vbox_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
-{
- struct vbox_private *vbox;
- int ret = 0;
-
- if (!vbox_check_supported(VBE_DISPI_ID_HGSMI))
- return -ENODEV;
-
- vbox = kzalloc(sizeof(*vbox), GFP_KERNEL);
- if (!vbox)
- return -ENOMEM;
-
- ret = drm_dev_init(&vbox->ddev, &driver, &pdev->dev);
- if (ret) {
- kfree(vbox);
- return ret;
- }
-
- vbox->ddev.pdev = pdev;
- vbox->ddev.dev_private = vbox;
- pci_set_drvdata(pdev, vbox);
- mutex_init(&vbox->hw_mutex);
-
- ret = pci_enable_device(pdev);
- if (ret)
- goto err_dev_put;
-
- ret = vbox_hw_init(vbox);
- if (ret)
- goto err_pci_disable;
-
- ret = vbox_mm_init(vbox);
- if (ret)
- goto err_hw_fini;
-
- ret = vbox_mode_init(vbox);
- if (ret)
- goto err_mm_fini;
-
- ret = vbox_irq_init(vbox);
- if (ret)
- goto err_mode_fini;
-
- ret = drm_fb_helper_fbdev_setup(&vbox->ddev, &vbox->fb_helper,
- &vbox_fb_helper_funcs, 32,
- vbox->num_crtcs);
- if (ret)
- goto err_irq_fini;
-
- ret = drm_dev_register(&vbox->ddev, 0);
- if (ret)
- goto err_fbdev_fini;
-
- return 0;
-
-err_fbdev_fini:
- vbox_fbdev_fini(vbox);
-err_irq_fini:
- vbox_irq_fini(vbox);
-err_mode_fini:
- vbox_mode_fini(vbox);
-err_mm_fini:
- vbox_mm_fini(vbox);
-err_hw_fini:
- vbox_hw_fini(vbox);
-err_pci_disable:
- pci_disable_device(pdev);
-err_dev_put:
- drm_dev_put(&vbox->ddev);
- return ret;
-}
-
-static void vbox_pci_remove(struct pci_dev *pdev)
-{
- struct vbox_private *vbox = pci_get_drvdata(pdev);
-
- drm_dev_unregister(&vbox->ddev);
- vbox_fbdev_fini(vbox);
- vbox_irq_fini(vbox);
- vbox_mode_fini(vbox);
- vbox_mm_fini(vbox);
- vbox_hw_fini(vbox);
- drm_dev_put(&vbox->ddev);
-}
-
-#ifdef CONFIG_PM_SLEEP
-static int vbox_pm_suspend(struct device *dev)
-{
- struct vbox_private *vbox = dev_get_drvdata(dev);
- int error;
-
- error = drm_mode_config_helper_suspend(&vbox->ddev);
- if (error)
- return error;
-
- pci_save_state(vbox->ddev.pdev);
- pci_disable_device(vbox->ddev.pdev);
- pci_set_power_state(vbox->ddev.pdev, PCI_D3hot);
-
- return 0;
-}
-
-static int vbox_pm_resume(struct device *dev)
-{
- struct vbox_private *vbox = dev_get_drvdata(dev);
-
- if (pci_enable_device(vbox->ddev.pdev))
- return -EIO;
-
- return drm_mode_config_helper_resume(&vbox->ddev);
-}
-
-static int vbox_pm_freeze(struct device *dev)
-{
- struct vbox_private *vbox = dev_get_drvdata(dev);
-
- return drm_mode_config_helper_suspend(&vbox->ddev);
-}
-
-static int vbox_pm_thaw(struct device *dev)
-{
- struct vbox_private *vbox = dev_get_drvdata(dev);
-
- return drm_mode_config_helper_resume(&vbox->ddev);
-}
-
-static int vbox_pm_poweroff(struct device *dev)
-{
- struct vbox_private *vbox = dev_get_drvdata(dev);
-
- return drm_mode_config_helper_suspend(&vbox->ddev);
-}
-
-static const struct dev_pm_ops vbox_pm_ops = {
- .suspend = vbox_pm_suspend,
- .resume = vbox_pm_resume,
- .freeze = vbox_pm_freeze,
- .thaw = vbox_pm_thaw,
- .poweroff = vbox_pm_poweroff,
- .restore = vbox_pm_resume,
-};
-#endif
-
-static struct pci_driver vbox_pci_driver = {
- .name = DRIVER_NAME,
- .id_table = pciidlist,
- .probe = vbox_pci_probe,
- .remove = vbox_pci_remove,
-#ifdef CONFIG_PM_SLEEP
- .driver.pm = &vbox_pm_ops,
-#endif
-};
-
-static const struct file_operations vbox_fops = {
- .owner = THIS_MODULE,
- .open = drm_open,
- .release = drm_release,
- .unlocked_ioctl = drm_ioctl,
- .compat_ioctl = drm_compat_ioctl,
- .mmap = vbox_mmap,
- .poll = drm_poll,
- .read = drm_read,
-};
-
-static int vbox_master_set(struct drm_device *dev,
- struct drm_file *file_priv, bool from_open)
-{
- struct vbox_private *vbox = dev->dev_private;
-
- /*
- * We do not yet know whether the new owner can handle hotplug, so we
- * do not advertise dynamic modes on the first query and send a
- * tentative hotplug notification after that to see if they query again.
- */
- vbox->initial_mode_queried = false;
-
- return 0;
-}
-
-static void vbox_master_drop(struct drm_device *dev, struct drm_file *file_priv)
-{
- struct vbox_private *vbox = dev->dev_private;
-
- /* See vbox_master_set() */
- vbox->initial_mode_queried = false;
-}
-
-static struct drm_driver driver = {
- .driver_features =
- DRIVER_MODESET | DRIVER_GEM | DRIVER_PRIME | DRIVER_ATOMIC,
-
- .lastclose = drm_fb_helper_lastclose,
- .master_set = vbox_master_set,
- .master_drop = vbox_master_drop,
-
- .fops = &vbox_fops,
- .irq_handler = vbox_irq_handler,
- .name = DRIVER_NAME,
- .desc = DRIVER_DESC,
- .date = DRIVER_DATE,
- .major = DRIVER_MAJOR,
- .minor = DRIVER_MINOR,
- .patchlevel = DRIVER_PATCHLEVEL,
-
- .gem_free_object_unlocked = vbox_gem_free_object,
- .dumb_create = vbox_dumb_create,
- .dumb_map_offset = vbox_dumb_mmap_offset,
- .prime_handle_to_fd = drm_gem_prime_handle_to_fd,
- .prime_fd_to_handle = drm_gem_prime_fd_to_handle,
- .gem_prime_export = drm_gem_prime_export,
- .gem_prime_import = drm_gem_prime_import,
- .gem_prime_pin = vbox_gem_prime_pin,
- .gem_prime_unpin = vbox_gem_prime_unpin,
- .gem_prime_get_sg_table = vbox_gem_prime_get_sg_table,
- .gem_prime_import_sg_table = vbox_gem_prime_import_sg_table,
- .gem_prime_vmap = vbox_gem_prime_vmap,
- .gem_prime_vunmap = vbox_gem_prime_vunmap,
- .gem_prime_mmap = vbox_gem_prime_mmap,
-};
-
-static int __init vbox_init(void)
-{
-#ifdef CONFIG_VGA_CONSOLE
- if (vgacon_text_force() && vbox_modeset == -1)
- return -EINVAL;
-#endif
-
- if (vbox_modeset == 0)
- return -EINVAL;
-
- return pci_register_driver(&vbox_pci_driver);
-}
-
-static void __exit vbox_exit(void)
-{
- pci_unregister_driver(&vbox_pci_driver);
-}
-
-module_init(vbox_init);
-module_exit(vbox_exit);
-
-MODULE_AUTHOR("Oracle Corporation");
-MODULE_AUTHOR("Hans de Goede <hdegoede@redhat.com>");
-MODULE_DESCRIPTION(DRIVER_DESC);
-MODULE_LICENSE("GPL and additional rights");
+++ /dev/null
-/* SPDX-License-Identifier: MIT */
-/*
- * Copyright (C) 2013-2017 Oracle Corporation
- * This file is based on ast_drv.h
- * Copyright 2012 Red Hat Inc.
- * Authors: Dave Airlie <airlied@redhat.com>
- * Michael Thayer <michael.thayer@oracle.com,
- * Hans de Goede <hdegoede@redhat.com>
- */
-#ifndef __VBOX_DRV_H__
-#define __VBOX_DRV_H__
-
-#include <linux/genalloc.h>
-#include <linux/io.h>
-#include <linux/irqreturn.h>
-#include <linux/string.h>
-#include <linux/version.h>
-
-#include <drm/drm_encoder.h>
-#include <drm/drm_fb_helper.h>
-#include <drm/drm_gem.h>
-
-#include <drm/ttm/ttm_bo_api.h>
-#include <drm/ttm/ttm_bo_driver.h>
-#include <drm/ttm/ttm_placement.h>
-#include <drm/ttm/ttm_memory.h>
-#include <drm/ttm/ttm_module.h>
-
-#include "vboxvideo_guest.h"
-#include "vboxvideo_vbe.h"
-#include "hgsmi_ch_setup.h"
-
-#define DRIVER_NAME "vboxvideo"
-#define DRIVER_DESC "Oracle VM VirtualBox Graphics Card"
-#define DRIVER_DATE "20130823"
-
-#define DRIVER_MAJOR 1
-#define DRIVER_MINOR 0
-#define DRIVER_PATCHLEVEL 0
-
-#define VBOX_MAX_CURSOR_WIDTH 64
-#define VBOX_MAX_CURSOR_HEIGHT 64
-#define CURSOR_PIXEL_COUNT (VBOX_MAX_CURSOR_WIDTH * VBOX_MAX_CURSOR_HEIGHT)
-#define CURSOR_DATA_SIZE (CURSOR_PIXEL_COUNT * 4 + CURSOR_PIXEL_COUNT / 8)
-
-#define VBOX_MAX_SCREENS 32
-
-#define GUEST_HEAP_OFFSET(vbox) ((vbox)->full_vram_size - \
- VBVA_ADAPTER_INFORMATION_SIZE)
-#define GUEST_HEAP_SIZE VBVA_ADAPTER_INFORMATION_SIZE
-#define GUEST_HEAP_USABLE_SIZE (VBVA_ADAPTER_INFORMATION_SIZE - \
- sizeof(struct hgsmi_host_flags))
-#define HOST_FLAGS_OFFSET GUEST_HEAP_USABLE_SIZE
-
-struct vbox_framebuffer {
- struct drm_framebuffer base;
- struct drm_gem_object *obj;
-};
-
-struct vbox_private {
- /* Must be first; or we must define our own release callback */
- struct drm_device ddev;
- struct drm_fb_helper fb_helper;
- struct vbox_framebuffer afb;
-
- u8 __iomem *guest_heap;
- u8 __iomem *vbva_buffers;
- struct gen_pool *guest_pool;
- struct vbva_buf_ctx *vbva_info;
- bool any_pitch;
- u32 num_crtcs;
- /* Amount of available VRAM, including space used for buffers. */
- u32 full_vram_size;
- /* Amount of available VRAM, not including space used for buffers. */
- u32 available_vram_size;
- /* Array of structures for receiving mode hints. */
- struct vbva_modehint *last_mode_hints;
-
- int fb_mtrr;
-
- struct {
- struct ttm_bo_device bdev;
- } ttm;
-
- struct mutex hw_mutex; /* protects modeset and accel/vbva accesses */
- /*
- * We decide whether or not user-space supports display hot-plug
- * depending on whether they react to a hot-plug event after the initial
- * mode query.
- */
- bool initial_mode_queried;
- struct work_struct hotplug_work;
- u32 input_mapping_width;
- u32 input_mapping_height;
- /*
- * Is user-space using an X.Org-style layout of one large frame-buffer
- * encompassing all screen ones or is the fbdev console active?
- */
- bool single_framebuffer;
- u8 cursor_data[CURSOR_DATA_SIZE];
-};
-
-#undef CURSOR_PIXEL_COUNT
-#undef CURSOR_DATA_SIZE
-
-struct vbox_gem_object;
-
-struct vbox_connector {
- struct drm_connector base;
- char name[32];
- struct vbox_crtc *vbox_crtc;
- struct {
- u32 width;
- u32 height;
- bool disconnected;
- } mode_hint;
-};
-
-struct vbox_crtc {
- struct drm_crtc base;
- bool disconnected;
- unsigned int crtc_id;
- u32 fb_offset;
- bool cursor_enabled;
- u32 x_hint;
- u32 y_hint;
- /*
- * When setting a mode we not only pass the mode to the hypervisor,
- * but also information on how to map / translate input coordinates
- * for the emulated USB tablet. This input-mapping may change when
- * the mode on *another* crtc changes.
- *
- * This means that sometimes we must do a modeset on other crtc-s then
- * the one being changed to update the input-mapping. Including crtc-s
- * which may be disabled inside the guest (shown as a black window
- * on the host unless closed by the user).
- *
- * With atomic modesetting the mode-info of disabled crtcs gets zeroed
- * yet we need it when updating the input-map to avoid resizing the
- * window as a side effect of a mode_set on another crtc. Therefor we
- * cache the info of the last mode below.
- */
- u32 width;
- u32 height;
- u32 x;
- u32 y;
-};
-
-struct vbox_encoder {
- struct drm_encoder base;
-};
-
-#define to_vbox_crtc(x) container_of(x, struct vbox_crtc, base)
-#define to_vbox_connector(x) container_of(x, struct vbox_connector, base)
-#define to_vbox_encoder(x) container_of(x, struct vbox_encoder, base)
-#define to_vbox_framebuffer(x) container_of(x, struct vbox_framebuffer, base)
-
-bool vbox_check_supported(u16 id);
-int vbox_hw_init(struct vbox_private *vbox);
-void vbox_hw_fini(struct vbox_private *vbox);
-
-int vbox_mode_init(struct vbox_private *vbox);
-void vbox_mode_fini(struct vbox_private *vbox);
-
-void vbox_report_caps(struct vbox_private *vbox);
-
-void vbox_framebuffer_dirty_rectangles(struct drm_framebuffer *fb,
- struct drm_clip_rect *rects,
- unsigned int num_rects);
-
-int vbox_framebuffer_init(struct vbox_private *vbox,
- struct vbox_framebuffer *vbox_fb,
- const struct drm_mode_fb_cmd2 *mode_cmd,
- struct drm_gem_object *obj);
-
-int vboxfb_create(struct drm_fb_helper *helper,
- struct drm_fb_helper_surface_size *sizes);
-void vbox_fbdev_fini(struct vbox_private *vbox);
-
-struct vbox_bo {
- struct ttm_buffer_object bo;
- struct ttm_placement placement;
- struct ttm_bo_kmap_obj kmap;
- struct drm_gem_object gem;
- struct ttm_place placements[3];
- int pin_count;
-};
-
-#define gem_to_vbox_bo(gobj) container_of((gobj), struct vbox_bo, gem)
-
-static inline struct vbox_bo *vbox_bo(struct ttm_buffer_object *bo)
-{
- return container_of(bo, struct vbox_bo, bo);
-}
-
-#define to_vbox_obj(x) container_of(x, struct vbox_gem_object, base)
-
-static inline u64 vbox_bo_gpu_offset(struct vbox_bo *bo)
-{
- return bo->bo.offset;
-}
-
-int vbox_dumb_create(struct drm_file *file,
- struct drm_device *dev,
- struct drm_mode_create_dumb *args);
-
-void vbox_gem_free_object(struct drm_gem_object *obj);
-int vbox_dumb_mmap_offset(struct drm_file *file,
- struct drm_device *dev,
- u32 handle, u64 *offset);
-
-#define DRM_FILE_PAGE_OFFSET (0x10000000ULL >> PAGE_SHIFT)
-
-int vbox_mm_init(struct vbox_private *vbox);
-void vbox_mm_fini(struct vbox_private *vbox);
-
-int vbox_bo_create(struct vbox_private *vbox, int size, int align,
- u32 flags, struct vbox_bo **pvboxbo);
-
-int vbox_gem_create(struct vbox_private *vbox,
- u32 size, bool iskernel, struct drm_gem_object **obj);
-
-int vbox_bo_pin(struct vbox_bo *bo, u32 pl_flag);
-int vbox_bo_unpin(struct vbox_bo *bo);
-
-static inline int vbox_bo_reserve(struct vbox_bo *bo, bool no_wait)
-{
- int ret;
-
- ret = ttm_bo_reserve(&bo->bo, true, no_wait, NULL);
- if (ret) {
- if (ret != -ERESTARTSYS && ret != -EBUSY)
- DRM_ERROR("reserve failed %p\n", bo);
- return ret;
- }
- return 0;
-}
-
-static inline void vbox_bo_unreserve(struct vbox_bo *bo)
-{
- ttm_bo_unreserve(&bo->bo);
-}
-
-void vbox_ttm_placement(struct vbox_bo *bo, int domain);
-int vbox_bo_push_sysram(struct vbox_bo *bo);
-int vbox_mmap(struct file *filp, struct vm_area_struct *vma);
-void *vbox_bo_kmap(struct vbox_bo *bo);
-void vbox_bo_kunmap(struct vbox_bo *bo);
-
-/* vbox_prime.c */
-int vbox_gem_prime_pin(struct drm_gem_object *obj);
-void vbox_gem_prime_unpin(struct drm_gem_object *obj);
-struct sg_table *vbox_gem_prime_get_sg_table(struct drm_gem_object *obj);
-struct drm_gem_object *vbox_gem_prime_import_sg_table(
- struct drm_device *dev, struct dma_buf_attachment *attach,
- struct sg_table *table);
-void *vbox_gem_prime_vmap(struct drm_gem_object *obj);
-void vbox_gem_prime_vunmap(struct drm_gem_object *obj, void *vaddr);
-int vbox_gem_prime_mmap(struct drm_gem_object *obj,
- struct vm_area_struct *area);
-
-/* vbox_irq.c */
-int vbox_irq_init(struct vbox_private *vbox);
-void vbox_irq_fini(struct vbox_private *vbox);
-void vbox_report_hotplug(struct vbox_private *vbox);
-irqreturn_t vbox_irq_handler(int irq, void *arg);
-
-/* vbox_hgsmi.c */
-void *hgsmi_buffer_alloc(struct gen_pool *guest_pool, size_t size,
- u8 channel, u16 channel_info);
-void hgsmi_buffer_free(struct gen_pool *guest_pool, void *buf);
-int hgsmi_buffer_submit(struct gen_pool *guest_pool, void *buf);
-
-static inline void vbox_write_ioport(u16 index, u16 data)
-{
- outw(index, VBE_DISPI_IOPORT_INDEX);
- outw(data, VBE_DISPI_IOPORT_DATA);
-}
-
-#endif
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/*
- * Copyright (C) 2013-2017 Oracle Corporation
- * This file is based on ast_fb.c
- * Copyright 2012 Red Hat Inc.
- * Authors: Dave Airlie <airlied@redhat.com>
- * Michael Thayer <michael.thayer@oracle.com,
- */
-#include <linux/delay.h>
-#include <linux/errno.h>
-#include <linux/fb.h>
-#include <linux/init.h>
-#include <linux/kernel.h>
-#include <linux/mm.h>
-#include <linux/module.h>
-#include <linux/pci.h>
-#include <linux/string.h>
-#include <linux/sysrq.h>
-#include <linux/tty.h>
-
-#include <drm/drm_crtc.h>
-#include <drm/drm_crtc_helper.h>
-#include <drm/drm_fb_helper.h>
-#include <drm/drm_fourcc.h>
-
-#include "vbox_drv.h"
-#include "vboxvideo.h"
-
-#ifdef CONFIG_DRM_KMS_FB_HELPER
-static struct fb_deferred_io vbox_defio = {
- .delay = HZ / 30,
- .deferred_io = drm_fb_helper_deferred_io,
-};
-#endif
-
-static struct fb_ops vboxfb_ops = {
- .owner = THIS_MODULE,
- DRM_FB_HELPER_DEFAULT_OPS,
- .fb_fillrect = drm_fb_helper_sys_fillrect,
- .fb_copyarea = drm_fb_helper_sys_copyarea,
- .fb_imageblit = drm_fb_helper_sys_imageblit,
-};
-
-int vboxfb_create(struct drm_fb_helper *helper,
- struct drm_fb_helper_surface_size *sizes)
-{
- struct vbox_private *vbox =
- container_of(helper, struct vbox_private, fb_helper);
- struct pci_dev *pdev = vbox->ddev.pdev;
- struct drm_mode_fb_cmd2 mode_cmd;
- struct drm_framebuffer *fb;
- struct fb_info *info;
- struct drm_gem_object *gobj;
- struct vbox_bo *bo;
- int size, ret;
- u64 gpu_addr;
- u32 pitch;
-
- mode_cmd.width = sizes->surface_width;
- mode_cmd.height = sizes->surface_height;
- pitch = mode_cmd.width * ((sizes->surface_bpp + 7) / 8);
- mode_cmd.pixel_format = drm_mode_legacy_fb_format(sizes->surface_bpp,
- sizes->surface_depth);
- mode_cmd.pitches[0] = pitch;
-
- size = pitch * mode_cmd.height;
-
- ret = vbox_gem_create(vbox, size, true, &gobj);
- if (ret) {
- DRM_ERROR("failed to create fbcon backing object %d\n", ret);
- return ret;
- }
-
- ret = vbox_framebuffer_init(vbox, &vbox->afb, &mode_cmd, gobj);
- if (ret)
- return ret;
-
- bo = gem_to_vbox_bo(gobj);
-
- ret = vbox_bo_pin(bo, TTM_PL_FLAG_VRAM);
- if (ret)
- return ret;
-
- info = drm_fb_helper_alloc_fbi(helper);
- if (IS_ERR(info))
- return PTR_ERR(info);
-
- info->screen_size = size;
- info->screen_base = (char __iomem *)vbox_bo_kmap(bo);
- if (IS_ERR(info->screen_base))
- return PTR_ERR(info->screen_base);
-
- info->par = helper;
-
- fb = &vbox->afb.base;
- helper->fb = fb;
-
- strcpy(info->fix.id, "vboxdrmfb");
-
- info->fbops = &vboxfb_ops;
-
- /*
- * This seems to be done for safety checking that the framebuffer
- * is not registered twice by different drivers.
- */
- info->apertures->ranges[0].base = pci_resource_start(pdev, 0);
- info->apertures->ranges[0].size = pci_resource_len(pdev, 0);
-
- drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
- drm_fb_helper_fill_var(info, helper, sizes->fb_width,
- sizes->fb_height);
-
- gpu_addr = vbox_bo_gpu_offset(bo);
- info->fix.smem_start = info->apertures->ranges[0].base + gpu_addr;
- info->fix.smem_len = vbox->available_vram_size - gpu_addr;
-
-#ifdef CONFIG_DRM_KMS_FB_HELPER
- info->fbdefio = &vbox_defio;
- fb_deferred_io_init(info);
-#endif
-
- info->pixmap.flags = FB_PIXMAP_SYSTEM;
-
- DRM_DEBUG_KMS("allocated %dx%d\n", fb->width, fb->height);
-
- return 0;
-}
-
-void vbox_fbdev_fini(struct vbox_private *vbox)
-{
- struct vbox_framebuffer *afb = &vbox->afb;
-
-#ifdef CONFIG_DRM_KMS_FB_HELPER
- if (vbox->fb_helper.fbdev && vbox->fb_helper.fbdev->fbdefio)
- fb_deferred_io_cleanup(vbox->fb_helper.fbdev);
-#endif
-
- drm_fb_helper_unregister_fbi(&vbox->fb_helper);
-
- if (afb->obj) {
- struct vbox_bo *bo = gem_to_vbox_bo(afb->obj);
-
- vbox_bo_kunmap(bo);
-
- if (bo->pin_count)
- vbox_bo_unpin(bo);
-
- drm_gem_object_put_unlocked(afb->obj);
- afb->obj = NULL;
- }
- drm_fb_helper_fini(&vbox->fb_helper);
-
- drm_framebuffer_unregister_private(&afb->base);
- drm_framebuffer_cleanup(&afb->base);
-}
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/*
- * Copyright (C) 2017 Oracle Corporation
- * Authors: Hans de Goede <hdegoede@redhat.com>
- */
-
-#include "vbox_drv.h"
-#include "vboxvideo_vbe.h"
-#include "hgsmi_defs.h"
-
-/* One-at-a-Time Hash from http://www.burtleburtle.net/bob/hash/doobs.html */
-static u32 hgsmi_hash_process(u32 hash, const u8 *data, int size)
-{
- while (size--) {
- hash += *data++;
- hash += (hash << 10);
- hash ^= (hash >> 6);
- }
-
- return hash;
-}
-
-static u32 hgsmi_hash_end(u32 hash)
-{
- hash += (hash << 3);
- hash ^= (hash >> 11);
- hash += (hash << 15);
-
- return hash;
-}
-
-/* Not really a checksum but that is the naming used in all vbox code */
-static u32 hgsmi_checksum(u32 offset,
- const struct hgsmi_buffer_header *header,
- const struct hgsmi_buffer_tail *tail)
-{
- u32 checksum;
-
- checksum = hgsmi_hash_process(0, (u8 *)&offset, sizeof(offset));
- checksum = hgsmi_hash_process(checksum, (u8 *)header, sizeof(*header));
- /* 4 -> Do not checksum the checksum itself */
- checksum = hgsmi_hash_process(checksum, (u8 *)tail, 4);
-
- return hgsmi_hash_end(checksum);
-}
-
-void *hgsmi_buffer_alloc(struct gen_pool *guest_pool, size_t size,
- u8 channel, u16 channel_info)
-{
- struct hgsmi_buffer_header *h;
- struct hgsmi_buffer_tail *t;
- size_t total_size;
- dma_addr_t offset;
-
- total_size = size + sizeof(*h) + sizeof(*t);
- h = gen_pool_dma_alloc(guest_pool, total_size, &offset);
- if (!h)
- return NULL;
-
- t = (struct hgsmi_buffer_tail *)((u8 *)h + sizeof(*h) + size);
-
- h->flags = HGSMI_BUFFER_HEADER_F_SEQ_SINGLE;
- h->data_size = size;
- h->channel = channel;
- h->channel_info = channel_info;
- memset(&h->u.header_data, 0, sizeof(h->u.header_data));
-
- t->reserved = 0;
- t->checksum = hgsmi_checksum(offset, h, t);
-
- return (u8 *)h + sizeof(*h);
-}
-
-void hgsmi_buffer_free(struct gen_pool *guest_pool, void *buf)
-{
- struct hgsmi_buffer_header *h =
- (struct hgsmi_buffer_header *)((u8 *)buf - sizeof(*h));
- size_t total_size = h->data_size + sizeof(*h) +
- sizeof(struct hgsmi_buffer_tail);
-
- gen_pool_free(guest_pool, (unsigned long)h, total_size);
-}
-
-int hgsmi_buffer_submit(struct gen_pool *guest_pool, void *buf)
-{
- phys_addr_t offset;
-
- offset = gen_pool_virt_to_phys(guest_pool, (unsigned long)buf -
- sizeof(struct hgsmi_buffer_header));
- outl(offset, VGA_PORT_HGSMI_GUEST);
- /* Make the compiler aware that the host has changed memory. */
- mb();
-
- return 0;
-}
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/*
- * Copyright (C) 2016-2017 Oracle Corporation
- * This file is based on qxl_irq.c
- * Copyright 2013 Red Hat Inc.
- * Authors: Dave Airlie
- * Alon Levy
- * Michael Thayer <michael.thayer@oracle.com,
- * Hans de Goede <hdegoede@redhat.com>
- */
-
-#include <linux/pci.h>
-#include <drm/drm_irq.h>
-#include <drm/drm_probe_helper.h>
-
-#include "vbox_drv.h"
-#include "vboxvideo.h"
-
-static void vbox_clear_irq(void)
-{
- outl((u32)~0, VGA_PORT_HGSMI_HOST);
-}
-
-static u32 vbox_get_flags(struct vbox_private *vbox)
-{
- return readl(vbox->guest_heap + HOST_FLAGS_OFFSET);
-}
-
-void vbox_report_hotplug(struct vbox_private *vbox)
-{
- schedule_work(&vbox->hotplug_work);
-}
-
-irqreturn_t vbox_irq_handler(int irq, void *arg)
-{
- struct drm_device *dev = (struct drm_device *)arg;
- struct vbox_private *vbox = (struct vbox_private *)dev->dev_private;
- u32 host_flags = vbox_get_flags(vbox);
-
- if (!(host_flags & HGSMIHOSTFLAGS_IRQ))
- return IRQ_NONE;
-
- /*
- * Due to a bug in the initial host implementation of hot-plug irqs,
- * the hot-plug and cursor capability flags were never cleared.
- * Fortunately we can tell when they would have been set by checking
- * that the VSYNC flag is not set.
- */
- if (host_flags &
- (HGSMIHOSTFLAGS_HOTPLUG | HGSMIHOSTFLAGS_CURSOR_CAPABILITIES) &&
- !(host_flags & HGSMIHOSTFLAGS_VSYNC))
- vbox_report_hotplug(vbox);
-
- vbox_clear_irq();
-
- return IRQ_HANDLED;
-}
-
-/*
- * Check that the position hints provided by the host are suitable for GNOME
- * shell (i.e. all screens disjoint and hints for all enabled screens) and if
- * not replace them with default ones. Providing valid hints improves the
- * chances that we will get a known screen layout for pointer mapping.
- */
-static void validate_or_set_position_hints(struct vbox_private *vbox)
-{
- struct vbva_modehint *hintsi, *hintsj;
- bool valid = true;
- u16 currentx = 0;
- int i, j;
-
- for (i = 0; i < vbox->num_crtcs; ++i) {
- for (j = 0; j < i; ++j) {
- hintsi = &vbox->last_mode_hints[i];
- hintsj = &vbox->last_mode_hints[j];
-
- if (hintsi->enabled && hintsj->enabled) {
- if (hintsi->dx >= 0xffff ||
- hintsi->dy >= 0xffff ||
- hintsj->dx >= 0xffff ||
- hintsj->dy >= 0xffff ||
- (hintsi->dx <
- hintsj->dx + (hintsj->cx & 0x8fff) &&
- hintsi->dx + (hintsi->cx & 0x8fff) >
- hintsj->dx) ||
- (hintsi->dy <
- hintsj->dy + (hintsj->cy & 0x8fff) &&
- hintsi->dy + (hintsi->cy & 0x8fff) >
- hintsj->dy))
- valid = false;
- }
- }
- }
- if (!valid)
- for (i = 0; i < vbox->num_crtcs; ++i) {
- if (vbox->last_mode_hints[i].enabled) {
- vbox->last_mode_hints[i].dx = currentx;
- vbox->last_mode_hints[i].dy = 0;
- currentx +=
- vbox->last_mode_hints[i].cx & 0x8fff;
- }
- }
-}
-
-/* Query the host for the most recent video mode hints. */
-static void vbox_update_mode_hints(struct vbox_private *vbox)
-{
- struct drm_device *dev = &vbox->ddev;
- struct drm_connector *connector;
- struct vbox_connector *vbox_conn;
- struct vbva_modehint *hints;
- u16 flags;
- bool disconnected;
- unsigned int crtc_id;
- int ret;
-
- ret = hgsmi_get_mode_hints(vbox->guest_pool, vbox->num_crtcs,
- vbox->last_mode_hints);
- if (ret) {
- DRM_ERROR("vboxvideo: hgsmi_get_mode_hints failed: %d\n", ret);
- return;
- }
-
- validate_or_set_position_hints(vbox);
- drm_modeset_lock_all(dev);
- list_for_each_entry(connector, &dev->mode_config.connector_list, head) {
- vbox_conn = to_vbox_connector(connector);
-
- hints = &vbox->last_mode_hints[vbox_conn->vbox_crtc->crtc_id];
- if (hints->magic != VBVAMODEHINT_MAGIC)
- continue;
-
- disconnected = !(hints->enabled);
- crtc_id = vbox_conn->vbox_crtc->crtc_id;
- vbox_conn->mode_hint.width = hints->cx;
- vbox_conn->mode_hint.height = hints->cy;
- vbox_conn->vbox_crtc->x_hint = hints->dx;
- vbox_conn->vbox_crtc->y_hint = hints->dy;
- vbox_conn->mode_hint.disconnected = disconnected;
-
- if (vbox_conn->vbox_crtc->disconnected == disconnected)
- continue;
-
- if (disconnected)
- flags = VBVA_SCREEN_F_ACTIVE | VBVA_SCREEN_F_DISABLED;
- else
- flags = VBVA_SCREEN_F_ACTIVE | VBVA_SCREEN_F_BLANK;
-
- hgsmi_process_display_info(vbox->guest_pool, crtc_id, 0, 0, 0,
- hints->cx * 4, hints->cx,
- hints->cy, 0, flags);
-
- vbox_conn->vbox_crtc->disconnected = disconnected;
- }
- drm_modeset_unlock_all(dev);
-}
-
-static void vbox_hotplug_worker(struct work_struct *work)
-{
- struct vbox_private *vbox = container_of(work, struct vbox_private,
- hotplug_work);
-
- vbox_update_mode_hints(vbox);
- drm_kms_helper_hotplug_event(&vbox->ddev);
-}
-
-int vbox_irq_init(struct vbox_private *vbox)
-{
- INIT_WORK(&vbox->hotplug_work, vbox_hotplug_worker);
- vbox_update_mode_hints(vbox);
-
- return drm_irq_install(&vbox->ddev, vbox->ddev.pdev->irq);
-}
-
-void vbox_irq_fini(struct vbox_private *vbox)
-{
- drm_irq_uninstall(&vbox->ddev);
- flush_work(&vbox->hotplug_work);
-}
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/*
- * Copyright (C) 2013-2017 Oracle Corporation
- * This file is based on ast_main.c
- * Copyright 2012 Red Hat Inc.
- * Authors: Dave Airlie <airlied@redhat.com>,
- * Michael Thayer <michael.thayer@oracle.com,
- * Hans de Goede <hdegoede@redhat.com>
- */
-
-#include <linux/vbox_err.h>
-#include <drm/drm_fb_helper.h>
-#include <drm/drm_crtc_helper.h>
-
-#include "vbox_drv.h"
-#include "vboxvideo_guest.h"
-#include "vboxvideo_vbe.h"
-
-static void vbox_user_framebuffer_destroy(struct drm_framebuffer *fb)
-{
- struct vbox_framebuffer *vbox_fb = to_vbox_framebuffer(fb);
-
- if (vbox_fb->obj)
- drm_gem_object_put_unlocked(vbox_fb->obj);
-
- drm_framebuffer_cleanup(fb);
- kfree(fb);
-}
-
-void vbox_report_caps(struct vbox_private *vbox)
-{
- u32 caps = VBVACAPS_DISABLE_CURSOR_INTEGRATION |
- VBVACAPS_IRQ | VBVACAPS_USE_VBVA_ONLY;
-
- if (vbox->initial_mode_queried)
- caps |= VBVACAPS_VIDEO_MODE_HINTS;
-
- hgsmi_send_caps_info(vbox->guest_pool, caps);
-}
-
-/* Send information about dirty rectangles to VBVA. */
-void vbox_framebuffer_dirty_rectangles(struct drm_framebuffer *fb,
- struct drm_clip_rect *rects,
- unsigned int num_rects)
-{
- struct vbox_private *vbox = fb->dev->dev_private;
- struct drm_display_mode *mode;
- struct drm_crtc *crtc;
- int crtc_x, crtc_y;
- unsigned int i;
-
- mutex_lock(&vbox->hw_mutex);
- list_for_each_entry(crtc, &fb->dev->mode_config.crtc_list, head) {
- if (crtc->primary->state->fb != fb)
- continue;
-
- mode = &crtc->state->mode;
- crtc_x = crtc->primary->state->src_x >> 16;
- crtc_y = crtc->primary->state->src_y >> 16;
-
- for (i = 0; i < num_rects; ++i) {
- struct vbva_cmd_hdr cmd_hdr;
- unsigned int crtc_id = to_vbox_crtc(crtc)->crtc_id;
-
- if (rects[i].x1 > crtc_x + mode->hdisplay ||
- rects[i].y1 > crtc_y + mode->vdisplay ||
- rects[i].x2 < crtc_x ||
- rects[i].y2 < crtc_y)
- continue;
-
- cmd_hdr.x = (s16)rects[i].x1;
- cmd_hdr.y = (s16)rects[i].y1;
- cmd_hdr.w = (u16)rects[i].x2 - rects[i].x1;
- cmd_hdr.h = (u16)rects[i].y2 - rects[i].y1;
-
- if (!vbva_buffer_begin_update(&vbox->vbva_info[crtc_id],
- vbox->guest_pool))
- continue;
-
- vbva_write(&vbox->vbva_info[crtc_id], vbox->guest_pool,
- &cmd_hdr, sizeof(cmd_hdr));
- vbva_buffer_end_update(&vbox->vbva_info[crtc_id]);
- }
- }
- mutex_unlock(&vbox->hw_mutex);
-}
-
-static int vbox_user_framebuffer_dirty(struct drm_framebuffer *fb,
- struct drm_file *file_priv,
- unsigned int flags, unsigned int color,
- struct drm_clip_rect *rects,
- unsigned int num_rects)
-{
- vbox_framebuffer_dirty_rectangles(fb, rects, num_rects);
-
- return 0;
-}
-
-static const struct drm_framebuffer_funcs vbox_fb_funcs = {
- .destroy = vbox_user_framebuffer_destroy,
- .dirty = vbox_user_framebuffer_dirty,
-};
-
-int vbox_framebuffer_init(struct vbox_private *vbox,
- struct vbox_framebuffer *vbox_fb,
- const struct drm_mode_fb_cmd2 *mode_cmd,
- struct drm_gem_object *obj)
-{
- int ret;
-
- drm_helper_mode_fill_fb_struct(&vbox->ddev, &vbox_fb->base, mode_cmd);
- vbox_fb->obj = obj;
- ret = drm_framebuffer_init(&vbox->ddev, &vbox_fb->base, &vbox_fb_funcs);
- if (ret) {
- DRM_ERROR("framebuffer init failed %d\n", ret);
- return ret;
- }
-
- return 0;
-}
-
-static int vbox_accel_init(struct vbox_private *vbox)
-{
- struct vbva_buffer *vbva;
- unsigned int i;
-
- vbox->vbva_info = devm_kcalloc(vbox->ddev.dev, vbox->num_crtcs,
- sizeof(*vbox->vbva_info), GFP_KERNEL);
- if (!vbox->vbva_info)
- return -ENOMEM;
-
- /* Take a command buffer for each screen from the end of usable VRAM. */
- vbox->available_vram_size -= vbox->num_crtcs * VBVA_MIN_BUFFER_SIZE;
-
- vbox->vbva_buffers = pci_iomap_range(vbox->ddev.pdev, 0,
- vbox->available_vram_size,
- vbox->num_crtcs *
- VBVA_MIN_BUFFER_SIZE);
- if (!vbox->vbva_buffers)
- return -ENOMEM;
-
- for (i = 0; i < vbox->num_crtcs; ++i) {
- vbva_setup_buffer_context(&vbox->vbva_info[i],
- vbox->available_vram_size +
- i * VBVA_MIN_BUFFER_SIZE,
- VBVA_MIN_BUFFER_SIZE);
- vbva = (void __force *)vbox->vbva_buffers +
- i * VBVA_MIN_BUFFER_SIZE;
- if (!vbva_enable(&vbox->vbva_info[i],
- vbox->guest_pool, vbva, i)) {
- /* very old host or driver error. */
- DRM_ERROR("vboxvideo: vbva_enable failed\n");
- }
- }
-
- return 0;
-}
-
-static void vbox_accel_fini(struct vbox_private *vbox)
-{
- unsigned int i;
-
- for (i = 0; i < vbox->num_crtcs; ++i)
- vbva_disable(&vbox->vbva_info[i], vbox->guest_pool, i);
-
- pci_iounmap(vbox->ddev.pdev, vbox->vbva_buffers);
-}
-
-/* Do we support the 4.3 plus mode hint reporting interface? */
-static bool have_hgsmi_mode_hints(struct vbox_private *vbox)
-{
- u32 have_hints, have_cursor;
- int ret;
-
- ret = hgsmi_query_conf(vbox->guest_pool,
- VBOX_VBVA_CONF32_MODE_HINT_REPORTING,
- &have_hints);
- if (ret)
- return false;
-
- ret = hgsmi_query_conf(vbox->guest_pool,
- VBOX_VBVA_CONF32_GUEST_CURSOR_REPORTING,
- &have_cursor);
- if (ret)
- return false;
-
- return have_hints == VINF_SUCCESS && have_cursor == VINF_SUCCESS;
-}
-
-bool vbox_check_supported(u16 id)
-{
- u16 dispi_id;
-
- vbox_write_ioport(VBE_DISPI_INDEX_ID, id);
- dispi_id = inw(VBE_DISPI_IOPORT_DATA);
-
- return dispi_id == id;
-}
-
-int vbox_hw_init(struct vbox_private *vbox)
-{
- int ret = -ENOMEM;
-
- vbox->full_vram_size = inl(VBE_DISPI_IOPORT_DATA);
- vbox->any_pitch = vbox_check_supported(VBE_DISPI_ID_ANYX);
-
- DRM_INFO("VRAM %08x\n", vbox->full_vram_size);
-
- /* Map guest-heap at end of vram */
- vbox->guest_heap =
- pci_iomap_range(vbox->ddev.pdev, 0, GUEST_HEAP_OFFSET(vbox),
- GUEST_HEAP_SIZE);
- if (!vbox->guest_heap)
- return -ENOMEM;
-
- /* Create guest-heap mem-pool use 2^4 = 16 byte chunks */
- vbox->guest_pool = gen_pool_create(4, -1);
- if (!vbox->guest_pool)
- goto err_unmap_guest_heap;
-
- ret = gen_pool_add_virt(vbox->guest_pool,
- (unsigned long)vbox->guest_heap,
- GUEST_HEAP_OFFSET(vbox),
- GUEST_HEAP_USABLE_SIZE, -1);
- if (ret)
- goto err_destroy_guest_pool;
-
- ret = hgsmi_test_query_conf(vbox->guest_pool);
- if (ret) {
- DRM_ERROR("vboxvideo: hgsmi_test_query_conf failed\n");
- goto err_destroy_guest_pool;
- }
-
- /* Reduce available VRAM size to reflect the guest heap. */
- vbox->available_vram_size = GUEST_HEAP_OFFSET(vbox);
- /* Linux drm represents monitors as a 32-bit array. */
- hgsmi_query_conf(vbox->guest_pool, VBOX_VBVA_CONF32_MONITOR_COUNT,
- &vbox->num_crtcs);
- vbox->num_crtcs = clamp_t(u32, vbox->num_crtcs, 1, VBOX_MAX_SCREENS);
-
- if (!have_hgsmi_mode_hints(vbox)) {
- ret = -ENOTSUPP;
- goto err_destroy_guest_pool;
- }
-
- vbox->last_mode_hints = devm_kcalloc(vbox->ddev.dev, vbox->num_crtcs,
- sizeof(struct vbva_modehint),
- GFP_KERNEL);
- if (!vbox->last_mode_hints) {
- ret = -ENOMEM;
- goto err_destroy_guest_pool;
- }
-
- ret = vbox_accel_init(vbox);
- if (ret)
- goto err_destroy_guest_pool;
-
- return 0;
-
-err_destroy_guest_pool:
- gen_pool_destroy(vbox->guest_pool);
-err_unmap_guest_heap:
- pci_iounmap(vbox->ddev.pdev, vbox->guest_heap);
- return ret;
-}
-
-void vbox_hw_fini(struct vbox_private *vbox)
-{
- vbox_accel_fini(vbox);
- gen_pool_destroy(vbox->guest_pool);
- pci_iounmap(vbox->ddev.pdev, vbox->guest_heap);
-}
-
-int vbox_gem_create(struct vbox_private *vbox,
- u32 size, bool iskernel, struct drm_gem_object **obj)
-{
- struct vbox_bo *vboxbo;
- int ret;
-
- *obj = NULL;
-
- size = roundup(size, PAGE_SIZE);
- if (size == 0)
- return -EINVAL;
-
- ret = vbox_bo_create(vbox, size, 0, 0, &vboxbo);
- if (ret) {
- if (ret != -ERESTARTSYS)
- DRM_ERROR("failed to allocate GEM object\n");
- return ret;
- }
-
- *obj = &vboxbo->gem;
-
- return 0;
-}
-
-int vbox_dumb_create(struct drm_file *file,
- struct drm_device *dev, struct drm_mode_create_dumb *args)
-{
- struct vbox_private *vbox =
- container_of(dev, struct vbox_private, ddev);
- struct drm_gem_object *gobj;
- u32 handle;
- int ret;
-
- args->pitch = args->width * ((args->bpp + 7) / 8);
- args->size = args->pitch * args->height;
-
- ret = vbox_gem_create(vbox, args->size, false, &gobj);
- if (ret)
- return ret;
-
- ret = drm_gem_handle_create(file, gobj, &handle);
- drm_gem_object_put_unlocked(gobj);
- if (ret)
- return ret;
-
- args->handle = handle;
-
- return 0;
-}
-
-void vbox_gem_free_object(struct drm_gem_object *obj)
-{
- struct vbox_bo *vbox_bo = gem_to_vbox_bo(obj);
-
- ttm_bo_put(&vbox_bo->bo);
-}
-
-static inline u64 vbox_bo_mmap_offset(struct vbox_bo *bo)
-{
- return drm_vma_node_offset_addr(&bo->bo.vma_node);
-}
-
-int
-vbox_dumb_mmap_offset(struct drm_file *file,
- struct drm_device *dev,
- u32 handle, u64 *offset)
-{
- struct drm_gem_object *obj;
- int ret;
- struct vbox_bo *bo;
-
- mutex_lock(&dev->struct_mutex);
- obj = drm_gem_object_lookup(file, handle);
- if (!obj) {
- ret = -ENOENT;
- goto out_unlock;
- }
-
- bo = gem_to_vbox_bo(obj);
- *offset = vbox_bo_mmap_offset(bo);
-
- drm_gem_object_put(obj);
- ret = 0;
-
-out_unlock:
- mutex_unlock(&dev->struct_mutex);
- return ret;
-}
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/*
- * Copyright (C) 2013-2017 Oracle Corporation
- * This file is based on ast_mode.c
- * Copyright 2012 Red Hat Inc.
- * Parts based on xf86-video-ast
- * Copyright (c) 2005 ASPEED Technology Inc.
- * Authors: Dave Airlie <airlied@redhat.com>
- * Michael Thayer <michael.thayer@oracle.com,
- * Hans de Goede <hdegoede@redhat.com>
- */
-#include <linux/export.h>
-
-#include <drm/drm_atomic.h>
-#include <drm/drm_atomic_helper.h>
-#include <drm/drm_fourcc.h>
-#include <drm/drm_plane_helper.h>
-#include <drm/drm_probe_helper.h>
-#include <drm/drm_vblank.h>
-
-#include "hgsmi_channels.h"
-#include "vbox_drv.h"
-#include "vboxvideo.h"
-
-/*
- * Set a graphics mode. Poke any required values into registers, do an HGSMI
- * mode set and tell the host we support advanced graphics functions.
- */
-static void vbox_do_modeset(struct drm_crtc *crtc)
-{
- struct drm_framebuffer *fb = crtc->primary->state->fb;
- struct vbox_crtc *vbox_crtc = to_vbox_crtc(crtc);
- struct vbox_private *vbox;
- int width, height, bpp, pitch;
- u16 flags;
- s32 x_offset, y_offset;
-
- vbox = crtc->dev->dev_private;
- width = vbox_crtc->width ? vbox_crtc->width : 640;
- height = vbox_crtc->height ? vbox_crtc->height : 480;
- bpp = fb ? fb->format->cpp[0] * 8 : 32;
- pitch = fb ? fb->pitches[0] : width * bpp / 8;
- x_offset = vbox->single_framebuffer ? vbox_crtc->x : vbox_crtc->x_hint;
- y_offset = vbox->single_framebuffer ? vbox_crtc->y : vbox_crtc->y_hint;
-
- /*
- * This is the old way of setting graphics modes. It assumed one screen
- * and a frame-buffer at the start of video RAM. On older versions of
- * VirtualBox, certain parts of the code still assume that the first
- * screen is programmed this way, so try to fake it.
- */
- if (vbox_crtc->crtc_id == 0 && fb &&
- vbox_crtc->fb_offset / pitch < 0xffff - crtc->y &&
- vbox_crtc->fb_offset % (bpp / 8) == 0) {
- vbox_write_ioport(VBE_DISPI_INDEX_XRES, width);
- vbox_write_ioport(VBE_DISPI_INDEX_YRES, height);
- vbox_write_ioport(VBE_DISPI_INDEX_VIRT_WIDTH, pitch * 8 / bpp);
- vbox_write_ioport(VBE_DISPI_INDEX_BPP, bpp);
- vbox_write_ioport(VBE_DISPI_INDEX_ENABLE, VBE_DISPI_ENABLED);
- vbox_write_ioport(VBE_DISPI_INDEX_X_OFFSET,
- vbox_crtc->fb_offset % pitch / bpp * 8 + vbox_crtc->x);
- vbox_write_ioport(VBE_DISPI_INDEX_Y_OFFSET,
- vbox_crtc->fb_offset / pitch + vbox_crtc->y);
- }
-
- flags = VBVA_SCREEN_F_ACTIVE;
- flags |= (fb && crtc->state->enable) ? 0 : VBVA_SCREEN_F_BLANK;
- flags |= vbox_crtc->disconnected ? VBVA_SCREEN_F_DISABLED : 0;
- hgsmi_process_display_info(vbox->guest_pool, vbox_crtc->crtc_id,
- x_offset, y_offset,
- vbox_crtc->x * bpp / 8 +
- vbox_crtc->y * pitch,
- pitch, width, height, bpp, flags);
-}
-
-static int vbox_set_view(struct drm_crtc *crtc)
-{
- struct vbox_crtc *vbox_crtc = to_vbox_crtc(crtc);
- struct vbox_private *vbox = crtc->dev->dev_private;
- struct vbva_infoview *p;
-
- /*
- * Tell the host about the view. This design originally targeted the
- * Windows XP driver architecture and assumed that each screen would
- * have a dedicated frame buffer with the command buffer following it,
- * the whole being a "view". The host works out which screen a command
- * buffer belongs to by checking whether it is in the first view, then
- * whether it is in the second and so on. The first match wins. We
- * cheat around this by making the first view be the managed memory
- * plus the first command buffer, the second the same plus the second
- * buffer and so on.
- */
- p = hgsmi_buffer_alloc(vbox->guest_pool, sizeof(*p),
- HGSMI_CH_VBVA, VBVA_INFO_VIEW);
- if (!p)
- return -ENOMEM;
-
- p->view_index = vbox_crtc->crtc_id;
- p->view_offset = vbox_crtc->fb_offset;
- p->view_size = vbox->available_vram_size - vbox_crtc->fb_offset +
- vbox_crtc->crtc_id * VBVA_MIN_BUFFER_SIZE;
- p->max_screen_size = vbox->available_vram_size - vbox_crtc->fb_offset;
-
- hgsmi_buffer_submit(vbox->guest_pool, p);
- hgsmi_buffer_free(vbox->guest_pool, p);
-
- return 0;
-}
-
-/*
- * Try to map the layout of virtual screens to the range of the input device.
- * Return true if we need to re-set the crtc modes due to screen offset
- * changes.
- */
-static bool vbox_set_up_input_mapping(struct vbox_private *vbox)
-{
- struct drm_crtc *crtci;
- struct drm_connector *connectori;
- struct drm_framebuffer *fb, *fb1 = NULL;
- bool single_framebuffer = true;
- bool old_single_framebuffer = vbox->single_framebuffer;
- u16 width = 0, height = 0;
-
- /*
- * Are we using an X.Org-style single large frame-buffer for all crtcs?
- * If so then screen layout can be deduced from the crtc offsets.
- * Same fall-back if this is the fbdev frame-buffer.
- */
- list_for_each_entry(crtci, &vbox->ddev.mode_config.crtc_list, head) {
- fb = crtci->primary->state->fb;
- if (!fb)
- continue;
-
- if (!fb1) {
- fb1 = fb;
- if (to_vbox_framebuffer(fb1) == &vbox->afb)
- break;
- } else if (fb != fb1) {
- single_framebuffer = false;
- }
- }
- if (!fb1)
- return false;
-
- if (single_framebuffer) {
- vbox->single_framebuffer = true;
- vbox->input_mapping_width = fb1->width;
- vbox->input_mapping_height = fb1->height;
- return old_single_framebuffer != vbox->single_framebuffer;
- }
- /* Otherwise calculate the total span of all screens. */
- list_for_each_entry(connectori, &vbox->ddev.mode_config.connector_list,
- head) {
- struct vbox_connector *vbox_connector =
- to_vbox_connector(connectori);
- struct vbox_crtc *vbox_crtc = vbox_connector->vbox_crtc;
-
- width = max_t(u16, width, vbox_crtc->x_hint +
- vbox_connector->mode_hint.width);
- height = max_t(u16, height, vbox_crtc->y_hint +
- vbox_connector->mode_hint.height);
- }
-
- vbox->single_framebuffer = false;
- vbox->input_mapping_width = width;
- vbox->input_mapping_height = height;
-
- return old_single_framebuffer != vbox->single_framebuffer;
-}
-
-static void vbox_crtc_set_base_and_mode(struct drm_crtc *crtc,
- struct drm_framebuffer *fb,
- int x, int y)
-{
- struct vbox_bo *bo = gem_to_vbox_bo(to_vbox_framebuffer(fb)->obj);
- struct vbox_private *vbox = crtc->dev->dev_private;
- struct vbox_crtc *vbox_crtc = to_vbox_crtc(crtc);
- bool needs_modeset = drm_atomic_crtc_needs_modeset(crtc->state);
-
- mutex_lock(&vbox->hw_mutex);
-
- if (crtc->state->enable) {
- vbox_crtc->width = crtc->state->mode.hdisplay;
- vbox_crtc->height = crtc->state->mode.vdisplay;
- }
-
- vbox_crtc->x = x;
- vbox_crtc->y = y;
- vbox_crtc->fb_offset = vbox_bo_gpu_offset(bo);
-
- /* vbox_do_modeset() checks vbox->single_framebuffer so update it now */
- if (needs_modeset && vbox_set_up_input_mapping(vbox)) {
- struct drm_crtc *crtci;
-
- list_for_each_entry(crtci, &vbox->ddev.mode_config.crtc_list,
- head) {
- if (crtci == crtc)
- continue;
- vbox_do_modeset(crtci);
- }
- }
-
- vbox_set_view(crtc);
- vbox_do_modeset(crtc);
-
- if (needs_modeset)
- hgsmi_update_input_mapping(vbox->guest_pool, 0, 0,
- vbox->input_mapping_width,
- vbox->input_mapping_height);
-
- mutex_unlock(&vbox->hw_mutex);
-}
-
-static void vbox_crtc_atomic_enable(struct drm_crtc *crtc,
- struct drm_crtc_state *old_crtc_state)
-{
-}
-
-static void vbox_crtc_atomic_disable(struct drm_crtc *crtc,
- struct drm_crtc_state *old_crtc_state)
-{
-}
-
-static void vbox_crtc_atomic_flush(struct drm_crtc *crtc,
- struct drm_crtc_state *old_crtc_state)
-{
- struct drm_pending_vblank_event *event;
- unsigned long flags;
-
- if (crtc->state && crtc->state->event) {
- event = crtc->state->event;
- crtc->state->event = NULL;
-
- spin_lock_irqsave(&crtc->dev->event_lock, flags);
- drm_crtc_send_vblank_event(crtc, event);
- spin_unlock_irqrestore(&crtc->dev->event_lock, flags);
- }
-}
-
-static const struct drm_crtc_helper_funcs vbox_crtc_helper_funcs = {
- .atomic_enable = vbox_crtc_atomic_enable,
- .atomic_disable = vbox_crtc_atomic_disable,
- .atomic_flush = vbox_crtc_atomic_flush,
-};
-
-static void vbox_crtc_destroy(struct drm_crtc *crtc)
-{
- drm_crtc_cleanup(crtc);
- kfree(crtc);
-}
-
-static const struct drm_crtc_funcs vbox_crtc_funcs = {
- .set_config = drm_atomic_helper_set_config,
- .page_flip = drm_atomic_helper_page_flip,
- /* .gamma_set = vbox_crtc_gamma_set, */
- .destroy = vbox_crtc_destroy,
- .reset = drm_atomic_helper_crtc_reset,
- .atomic_duplicate_state = drm_atomic_helper_crtc_duplicate_state,
- .atomic_destroy_state = drm_atomic_helper_crtc_destroy_state,
-};
-
-static int vbox_primary_atomic_check(struct drm_plane *plane,
- struct drm_plane_state *new_state)
-{
- struct drm_crtc_state *crtc_state = NULL;
-
- if (new_state->crtc) {
- crtc_state = drm_atomic_get_existing_crtc_state(
- new_state->state, new_state->crtc);
- if (WARN_ON(!crtc_state))
- return -EINVAL;
- }
-
- return drm_atomic_helper_check_plane_state(new_state, crtc_state,
- DRM_PLANE_HELPER_NO_SCALING,
- DRM_PLANE_HELPER_NO_SCALING,
- false, true);
-}
-
-static void vbox_primary_atomic_update(struct drm_plane *plane,
- struct drm_plane_state *old_state)
-{
- struct drm_crtc *crtc = plane->state->crtc;
- struct drm_framebuffer *fb = plane->state->fb;
-
- vbox_crtc_set_base_and_mode(crtc, fb,
- plane->state->src_x >> 16,
- plane->state->src_y >> 16);
-}
-
-static void vbox_primary_atomic_disable(struct drm_plane *plane,
- struct drm_plane_state *old_state)
-{
- struct drm_crtc *crtc = old_state->crtc;
-
- /* vbox_do_modeset checks plane->state->fb and will disable if NULL */
- vbox_crtc_set_base_and_mode(crtc, old_state->fb,
- old_state->src_x >> 16,
- old_state->src_y >> 16);
-}
-
-static int vbox_primary_prepare_fb(struct drm_plane *plane,
- struct drm_plane_state *new_state)
-{
- struct vbox_bo *bo;
- int ret;
-
- if (!new_state->fb)
- return 0;
-
- bo = gem_to_vbox_bo(to_vbox_framebuffer(new_state->fb)->obj);
- ret = vbox_bo_pin(bo, TTM_PL_FLAG_VRAM);
- if (ret)
- DRM_WARN("Error %d pinning new fb, out of video mem?\n", ret);
-
- return ret;
-}
-
-static void vbox_primary_cleanup_fb(struct drm_plane *plane,
- struct drm_plane_state *old_state)
-{
- struct vbox_bo *bo;
-
- if (!old_state->fb)
- return;
-
- bo = gem_to_vbox_bo(to_vbox_framebuffer(old_state->fb)->obj);
- vbox_bo_unpin(bo);
-}
-
-static int vbox_cursor_atomic_check(struct drm_plane *plane,
- struct drm_plane_state *new_state)
-{
- struct drm_crtc_state *crtc_state = NULL;
- u32 width = new_state->crtc_w;
- u32 height = new_state->crtc_h;
- int ret;
-
- if (new_state->crtc) {
- crtc_state = drm_atomic_get_existing_crtc_state(
- new_state->state, new_state->crtc);
- if (WARN_ON(!crtc_state))
- return -EINVAL;
- }
-
- ret = drm_atomic_helper_check_plane_state(new_state, crtc_state,
- DRM_PLANE_HELPER_NO_SCALING,
- DRM_PLANE_HELPER_NO_SCALING,
- true, true);
- if (ret)
- return ret;
-
- if (!new_state->fb)
- return 0;
-
- if (width > VBOX_MAX_CURSOR_WIDTH || height > VBOX_MAX_CURSOR_HEIGHT ||
- width == 0 || height == 0)
- return -EINVAL;
-
- return 0;
-}
-
-/*
- * Copy the ARGB image and generate the mask, which is needed in case the host
- * does not support ARGB cursors. The mask is a 1BPP bitmap with the bit set
- * if the corresponding alpha value in the ARGB image is greater than 0xF0.
- */
-static void copy_cursor_image(u8 *src, u8 *dst, u32 width, u32 height,
- size_t mask_size)
-{
- size_t line_size = (width + 7) / 8;
- u32 i, j;
-
- memcpy(dst + mask_size, src, width * height * 4);
- for (i = 0; i < height; ++i)
- for (j = 0; j < width; ++j)
- if (((u32 *)src)[i * width + j] > 0xf0000000)
- dst[i * line_size + j / 8] |= (0x80 >> (j % 8));
-}
-
-static void vbox_cursor_atomic_update(struct drm_plane *plane,
- struct drm_plane_state *old_state)
-{
- struct vbox_private *vbox =
- container_of(plane->dev, struct vbox_private, ddev);
- struct vbox_crtc *vbox_crtc = to_vbox_crtc(plane->state->crtc);
- struct drm_framebuffer *fb = plane->state->fb;
- struct vbox_bo *bo = gem_to_vbox_bo(to_vbox_framebuffer(fb)->obj);
- u32 width = plane->state->crtc_w;
- u32 height = plane->state->crtc_h;
- size_t data_size, mask_size;
- u32 flags;
- u8 *src;
-
- /*
- * VirtualBox uses the host windowing system to draw the cursor so
- * moves are a no-op, we only need to upload new cursor sprites.
- */
- if (fb == old_state->fb)
- return;
-
- mutex_lock(&vbox->hw_mutex);
-
- vbox_crtc->cursor_enabled = true;
-
- /* pinning is done in prepare/cleanup framebuffer */
- src = vbox_bo_kmap(bo);
- if (IS_ERR(src)) {
- mutex_unlock(&vbox->hw_mutex);
- DRM_WARN("Could not kmap cursor bo, skipping update\n");
- return;
- }
-
- /*
- * The mask must be calculated based on the alpha
- * channel, one bit per ARGB word, and must be 32-bit
- * padded.
- */
- mask_size = ((width + 7) / 8 * height + 3) & ~3;
- data_size = width * height * 4 + mask_size;
-
- copy_cursor_image(src, vbox->cursor_data, width, height, mask_size);
- vbox_bo_kunmap(bo);
-
- flags = VBOX_MOUSE_POINTER_VISIBLE | VBOX_MOUSE_POINTER_SHAPE |
- VBOX_MOUSE_POINTER_ALPHA;
- hgsmi_update_pointer_shape(vbox->guest_pool, flags,
- min_t(u32, max(fb->hot_x, 0), width),
- min_t(u32, max(fb->hot_y, 0), height),
- width, height, vbox->cursor_data, data_size);
-
- mutex_unlock(&vbox->hw_mutex);
-}
-
-static void vbox_cursor_atomic_disable(struct drm_plane *plane,
- struct drm_plane_state *old_state)
-{
- struct vbox_private *vbox =
- container_of(plane->dev, struct vbox_private, ddev);
- struct vbox_crtc *vbox_crtc = to_vbox_crtc(old_state->crtc);
- bool cursor_enabled = false;
- struct drm_crtc *crtci;
-
- mutex_lock(&vbox->hw_mutex);
-
- vbox_crtc->cursor_enabled = false;
-
- list_for_each_entry(crtci, &vbox->ddev.mode_config.crtc_list, head) {
- if (to_vbox_crtc(crtci)->cursor_enabled)
- cursor_enabled = true;
- }
-
- if (!cursor_enabled)
- hgsmi_update_pointer_shape(vbox->guest_pool, 0, 0, 0,
- 0, 0, NULL, 0);
-
- mutex_unlock(&vbox->hw_mutex);
-}
-
-static int vbox_cursor_prepare_fb(struct drm_plane *plane,
- struct drm_plane_state *new_state)
-{
- struct vbox_bo *bo;
-
- if (!new_state->fb)
- return 0;
-
- bo = gem_to_vbox_bo(to_vbox_framebuffer(new_state->fb)->obj);
- return vbox_bo_pin(bo, TTM_PL_FLAG_SYSTEM);
-}
-
-static void vbox_cursor_cleanup_fb(struct drm_plane *plane,
- struct drm_plane_state *old_state)
-{
- struct vbox_bo *bo;
-
- if (!plane->state->fb)
- return;
-
- bo = gem_to_vbox_bo(to_vbox_framebuffer(plane->state->fb)->obj);
- vbox_bo_unpin(bo);
-}
-
-static const u32 vbox_cursor_plane_formats[] = {
- DRM_FORMAT_ARGB8888,
-};
-
-static const struct drm_plane_helper_funcs vbox_cursor_helper_funcs = {
- .atomic_check = vbox_cursor_atomic_check,
- .atomic_update = vbox_cursor_atomic_update,
- .atomic_disable = vbox_cursor_atomic_disable,
- .prepare_fb = vbox_cursor_prepare_fb,
- .cleanup_fb = vbox_cursor_cleanup_fb,
-};
-
-static const struct drm_plane_funcs vbox_cursor_plane_funcs = {
- .update_plane = drm_atomic_helper_update_plane,
- .disable_plane = drm_atomic_helper_disable_plane,
- .destroy = drm_primary_helper_destroy,
- .reset = drm_atomic_helper_plane_reset,
- .atomic_duplicate_state = drm_atomic_helper_plane_duplicate_state,
- .atomic_destroy_state = drm_atomic_helper_plane_destroy_state,
-};
-
-static const u32 vbox_primary_plane_formats[] = {
- DRM_FORMAT_XRGB8888,
- DRM_FORMAT_ARGB8888,
-};
-
-static const struct drm_plane_helper_funcs vbox_primary_helper_funcs = {
- .atomic_check = vbox_primary_atomic_check,
- .atomic_update = vbox_primary_atomic_update,
- .atomic_disable = vbox_primary_atomic_disable,
- .prepare_fb = vbox_primary_prepare_fb,
- .cleanup_fb = vbox_primary_cleanup_fb,
-};
-
-static const struct drm_plane_funcs vbox_primary_plane_funcs = {
- .update_plane = drm_atomic_helper_update_plane,
- .disable_plane = drm_atomic_helper_disable_plane,
- .destroy = drm_primary_helper_destroy,
- .reset = drm_atomic_helper_plane_reset,
- .atomic_duplicate_state = drm_atomic_helper_plane_duplicate_state,
- .atomic_destroy_state = drm_atomic_helper_plane_destroy_state,
-};
-
-static struct drm_plane *vbox_create_plane(struct vbox_private *vbox,
- unsigned int possible_crtcs,
- enum drm_plane_type type)
-{
- const struct drm_plane_helper_funcs *helper_funcs = NULL;
- const struct drm_plane_funcs *funcs;
- struct drm_plane *plane;
- const u32 *formats;
- int num_formats;
- int err;
-
- if (type == DRM_PLANE_TYPE_PRIMARY) {
- funcs = &vbox_primary_plane_funcs;
- formats = vbox_primary_plane_formats;
- helper_funcs = &vbox_primary_helper_funcs;
- num_formats = ARRAY_SIZE(vbox_primary_plane_formats);
- } else if (type == DRM_PLANE_TYPE_CURSOR) {
- funcs = &vbox_cursor_plane_funcs;
- formats = vbox_cursor_plane_formats;
- helper_funcs = &vbox_cursor_helper_funcs;
- num_formats = ARRAY_SIZE(vbox_cursor_plane_formats);
- } else {
- return ERR_PTR(-EINVAL);
- }
-
- plane = kzalloc(sizeof(*plane), GFP_KERNEL);
- if (!plane)
- return ERR_PTR(-ENOMEM);
-
- err = drm_universal_plane_init(&vbox->ddev, plane, possible_crtcs,
- funcs, formats, num_formats,
- NULL, type, NULL);
- if (err)
- goto free_plane;
-
- drm_plane_helper_add(plane, helper_funcs);
-
- return plane;
-
-free_plane:
- kfree(plane);
- return ERR_PTR(-EINVAL);
-}
-
-static struct vbox_crtc *vbox_crtc_init(struct drm_device *dev, unsigned int i)
-{
- struct vbox_private *vbox =
- container_of(dev, struct vbox_private, ddev);
- struct drm_plane *cursor = NULL;
- struct vbox_crtc *vbox_crtc;
- struct drm_plane *primary;
- u32 caps = 0;
- int ret;
-
- ret = hgsmi_query_conf(vbox->guest_pool,
- VBOX_VBVA_CONF32_CURSOR_CAPABILITIES, &caps);
- if (ret)
- return ERR_PTR(ret);
-
- vbox_crtc = kzalloc(sizeof(*vbox_crtc), GFP_KERNEL);
- if (!vbox_crtc)
- return ERR_PTR(-ENOMEM);
-
- primary = vbox_create_plane(vbox, 1 << i, DRM_PLANE_TYPE_PRIMARY);
- if (IS_ERR(primary)) {
- ret = PTR_ERR(primary);
- goto free_mem;
- }
-
- if ((caps & VBOX_VBVA_CURSOR_CAPABILITY_HARDWARE)) {
- cursor = vbox_create_plane(vbox, 1 << i, DRM_PLANE_TYPE_CURSOR);
- if (IS_ERR(cursor)) {
- ret = PTR_ERR(cursor);
- goto clean_primary;
- }
- } else {
- DRM_WARN("VirtualBox host is too old, no cursor support\n");
- }
-
- vbox_crtc->crtc_id = i;
-
- ret = drm_crtc_init_with_planes(dev, &vbox_crtc->base, primary, cursor,
- &vbox_crtc_funcs, NULL);
- if (ret)
- goto clean_cursor;
-
- drm_mode_crtc_set_gamma_size(&vbox_crtc->base, 256);
- drm_crtc_helper_add(&vbox_crtc->base, &vbox_crtc_helper_funcs);
-
- return vbox_crtc;
-
-clean_cursor:
- if (cursor) {
- drm_plane_cleanup(cursor);
- kfree(cursor);
- }
-clean_primary:
- drm_plane_cleanup(primary);
- kfree(primary);
-free_mem:
- kfree(vbox_crtc);
- return ERR_PTR(ret);
-}
-
-static void vbox_encoder_destroy(struct drm_encoder *encoder)
-{
- drm_encoder_cleanup(encoder);
- kfree(encoder);
-}
-
-static const struct drm_encoder_funcs vbox_enc_funcs = {
- .destroy = vbox_encoder_destroy,
-};
-
-static struct drm_encoder *vbox_encoder_init(struct drm_device *dev,
- unsigned int i)
-{
- struct vbox_encoder *vbox_encoder;
-
- vbox_encoder = kzalloc(sizeof(*vbox_encoder), GFP_KERNEL);
- if (!vbox_encoder)
- return NULL;
-
- drm_encoder_init(dev, &vbox_encoder->base, &vbox_enc_funcs,
- DRM_MODE_ENCODER_DAC, NULL);
-
- vbox_encoder->base.possible_crtcs = 1 << i;
- return &vbox_encoder->base;
-}
-
-/*
- * Generate EDID data with a mode-unique serial number for the virtual
- * monitor to try to persuade Unity that different modes correspond to
- * different monitors and it should not try to force the same resolution on
- * them.
- */
-static void vbox_set_edid(struct drm_connector *connector, int width,
- int height)
-{
- enum { EDID_SIZE = 128 };
- unsigned char edid[EDID_SIZE] = {
- 0x00, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x00, /* header */
- 0x58, 0x58, /* manufacturer (VBX) */
- 0x00, 0x00, /* product code */
- 0x00, 0x00, 0x00, 0x00, /* serial number goes here */
- 0x01, /* week of manufacture */
- 0x00, /* year of manufacture */
- 0x01, 0x03, /* EDID version */
- 0x80, /* capabilities - digital */
- 0x00, /* horiz. res in cm, zero for projectors */
- 0x00, /* vert. res in cm */
- 0x78, /* display gamma (120 == 2.2). */
- 0xEE, /* features (standby, suspend, off, RGB, std */
- /* colour space, preferred timing mode) */
- 0xEE, 0x91, 0xA3, 0x54, 0x4C, 0x99, 0x26, 0x0F, 0x50, 0x54,
- /* chromaticity for standard colour space. */
- 0x00, 0x00, 0x00, /* no default timings */
- 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
- 0x01, 0x01,
- 0x01, 0x01, 0x01, 0x01, /* no standard timings */
- 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x06, 0x00, 0x02, 0x02,
- 0x02, 0x02,
- /* descriptor block 1 goes below */
- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
- /* descriptor block 2, monitor ranges */
- 0x00, 0x00, 0x00, 0xFD, 0x00,
- 0x00, 0xC8, 0x00, 0xC8, 0x64, 0x00, 0x0A, 0x20, 0x20, 0x20,
- 0x20, 0x20,
- /* 0-200Hz vertical, 0-200KHz horizontal, 1000MHz pixel clock */
- 0x20,
- /* descriptor block 3, monitor name */
- 0x00, 0x00, 0x00, 0xFC, 0x00,
- 'V', 'B', 'O', 'X', ' ', 'm', 'o', 'n', 'i', 't', 'o', 'r',
- '\n',
- /* descriptor block 4: dummy data */
- 0x00, 0x00, 0x00, 0x10, 0x00,
- 0x0A, 0x20, 0x20, 0x20, 0x20, 0x20,
- 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
- 0x20,
- 0x00, /* number of extensions */
- 0x00 /* checksum goes here */
- };
- int clock = (width + 6) * (height + 6) * 60 / 10000;
- unsigned int i, sum = 0;
-
- edid[12] = width & 0xff;
- edid[13] = width >> 8;
- edid[14] = height & 0xff;
- edid[15] = height >> 8;
- edid[54] = clock & 0xff;
- edid[55] = clock >> 8;
- edid[56] = width & 0xff;
- edid[58] = (width >> 4) & 0xf0;
- edid[59] = height & 0xff;
- edid[61] = (height >> 4) & 0xf0;
- for (i = 0; i < EDID_SIZE - 1; ++i)
- sum += edid[i];
- edid[EDID_SIZE - 1] = (0x100 - (sum & 0xFF)) & 0xFF;
- drm_connector_update_edid_property(connector, (struct edid *)edid);
-}
-
-static int vbox_get_modes(struct drm_connector *connector)
-{
- struct vbox_connector *vbox_connector = NULL;
- struct drm_display_mode *mode = NULL;
- struct vbox_private *vbox = NULL;
- unsigned int num_modes = 0;
- int preferred_width, preferred_height;
-
- vbox_connector = to_vbox_connector(connector);
- vbox = connector->dev->dev_private;
- /*
- * Heuristic: we do not want to tell the host that we support dynamic
- * resizing unless we feel confident that the user space client using
- * the video driver can handle hot-plug events. So the first time modes
- * are queried after a "master" switch we tell the host that we do not,
- * and immediately after we send the client a hot-plug notification as
- * a test to see if they will respond and query again.
- * That is also the reason why capabilities are reported to the host at
- * this place in the code rather than elsewhere.
- * We need to report the flags location before reporting the IRQ
- * capability.
- */
- hgsmi_report_flags_location(vbox->guest_pool, GUEST_HEAP_OFFSET(vbox) +
- HOST_FLAGS_OFFSET);
- if (vbox_connector->vbox_crtc->crtc_id == 0)
- vbox_report_caps(vbox);
- if (!vbox->initial_mode_queried) {
- if (vbox_connector->vbox_crtc->crtc_id == 0) {
- vbox->initial_mode_queried = true;
- vbox_report_hotplug(vbox);
- }
- return drm_add_modes_noedid(connector, 800, 600);
- }
- num_modes = drm_add_modes_noedid(connector, 2560, 1600);
- preferred_width = vbox_connector->mode_hint.width ?
- vbox_connector->mode_hint.width : 1024;
- preferred_height = vbox_connector->mode_hint.height ?
- vbox_connector->mode_hint.height : 768;
- mode = drm_cvt_mode(connector->dev, preferred_width, preferred_height,
- 60, false, false, false);
- if (mode) {
- mode->type |= DRM_MODE_TYPE_PREFERRED;
- drm_mode_probed_add(connector, mode);
- ++num_modes;
- }
- vbox_set_edid(connector, preferred_width, preferred_height);
-
- if (vbox_connector->vbox_crtc->x_hint != -1)
- drm_object_property_set_value(&connector->base,
- vbox->ddev.mode_config.suggested_x_property,
- vbox_connector->vbox_crtc->x_hint);
- else
- drm_object_property_set_value(&connector->base,
- vbox->ddev.mode_config.suggested_x_property, 0);
-
- if (vbox_connector->vbox_crtc->y_hint != -1)
- drm_object_property_set_value(&connector->base,
- vbox->ddev.mode_config.suggested_y_property,
- vbox_connector->vbox_crtc->y_hint);
- else
- drm_object_property_set_value(&connector->base,
- vbox->ddev.mode_config.suggested_y_property, 0);
-
- return num_modes;
-}
-
-static void vbox_connector_destroy(struct drm_connector *connector)
-{
- drm_connector_unregister(connector);
- drm_connector_cleanup(connector);
- kfree(connector);
-}
-
-static enum drm_connector_status
-vbox_connector_detect(struct drm_connector *connector, bool force)
-{
- struct vbox_connector *vbox_connector;
-
- vbox_connector = to_vbox_connector(connector);
-
- return vbox_connector->mode_hint.disconnected ?
- connector_status_disconnected : connector_status_connected;
-}
-
-static int vbox_fill_modes(struct drm_connector *connector, u32 max_x,
- u32 max_y)
-{
- struct vbox_connector *vbox_connector;
- struct drm_device *dev;
- struct drm_display_mode *mode, *iterator;
-
- vbox_connector = to_vbox_connector(connector);
- dev = vbox_connector->base.dev;
- list_for_each_entry_safe(mode, iterator, &connector->modes, head) {
- list_del(&mode->head);
- drm_mode_destroy(dev, mode);
- }
-
- return drm_helper_probe_single_connector_modes(connector, max_x, max_y);
-}
-
-static const struct drm_connector_helper_funcs vbox_connector_helper_funcs = {
- .get_modes = vbox_get_modes,
-};
-
-static const struct drm_connector_funcs vbox_connector_funcs = {
- .detect = vbox_connector_detect,
- .fill_modes = vbox_fill_modes,
- .destroy = vbox_connector_destroy,
- .reset = drm_atomic_helper_connector_reset,
- .atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state,
- .atomic_destroy_state = drm_atomic_helper_connector_destroy_state,
-};
-
-static int vbox_connector_init(struct drm_device *dev,
- struct vbox_crtc *vbox_crtc,
- struct drm_encoder *encoder)
-{
- struct vbox_connector *vbox_connector;
- struct drm_connector *connector;
-
- vbox_connector = kzalloc(sizeof(*vbox_connector), GFP_KERNEL);
- if (!vbox_connector)
- return -ENOMEM;
-
- connector = &vbox_connector->base;
- vbox_connector->vbox_crtc = vbox_crtc;
-
- drm_connector_init(dev, connector, &vbox_connector_funcs,
- DRM_MODE_CONNECTOR_VGA);
- drm_connector_helper_add(connector, &vbox_connector_helper_funcs);
-
- connector->interlace_allowed = 0;
- connector->doublescan_allowed = 0;
-
- drm_mode_create_suggested_offset_properties(dev);
- drm_object_attach_property(&connector->base,
- dev->mode_config.suggested_x_property, 0);
- drm_object_attach_property(&connector->base,
- dev->mode_config.suggested_y_property, 0);
-
- drm_connector_attach_encoder(connector, encoder);
-
- return 0;
-}
-
-static struct drm_framebuffer *vbox_user_framebuffer_create(
- struct drm_device *dev,
- struct drm_file *filp,
- const struct drm_mode_fb_cmd2 *mode_cmd)
-{
- struct vbox_private *vbox =
- container_of(dev, struct vbox_private, ddev);
- struct drm_gem_object *obj;
- struct vbox_framebuffer *vbox_fb;
- int ret = -ENOMEM;
-
- obj = drm_gem_object_lookup(filp, mode_cmd->handles[0]);
- if (!obj)
- return ERR_PTR(-ENOENT);
-
- vbox_fb = kzalloc(sizeof(*vbox_fb), GFP_KERNEL);
- if (!vbox_fb)
- goto err_unref_obj;
-
- ret = vbox_framebuffer_init(vbox, vbox_fb, mode_cmd, obj);
- if (ret)
- goto err_free_vbox_fb;
-
- return &vbox_fb->base;
-
-err_free_vbox_fb:
- kfree(vbox_fb);
-err_unref_obj:
- drm_gem_object_put_unlocked(obj);
- return ERR_PTR(ret);
-}
-
-static const struct drm_mode_config_funcs vbox_mode_funcs = {
- .fb_create = vbox_user_framebuffer_create,
- .atomic_check = drm_atomic_helper_check,
- .atomic_commit = drm_atomic_helper_commit,
-};
-
-int vbox_mode_init(struct vbox_private *vbox)
-{
- struct drm_device *dev = &vbox->ddev;
- struct drm_encoder *encoder;
- struct vbox_crtc *vbox_crtc;
- unsigned int i;
- int ret;
-
- drm_mode_config_init(dev);
-
- dev->mode_config.funcs = (void *)&vbox_mode_funcs;
- dev->mode_config.min_width = 0;
- dev->mode_config.min_height = 0;
- dev->mode_config.preferred_depth = 24;
- dev->mode_config.max_width = VBE_DISPI_MAX_XRES;
- dev->mode_config.max_height = VBE_DISPI_MAX_YRES;
-
- for (i = 0; i < vbox->num_crtcs; ++i) {
- vbox_crtc = vbox_crtc_init(dev, i);
- if (IS_ERR(vbox_crtc)) {
- ret = PTR_ERR(vbox_crtc);
- goto err_drm_mode_cleanup;
- }
- encoder = vbox_encoder_init(dev, i);
- if (!encoder) {
- ret = -ENOMEM;
- goto err_drm_mode_cleanup;
- }
- ret = vbox_connector_init(dev, vbox_crtc, encoder);
- if (ret)
- goto err_drm_mode_cleanup;
- }
-
- drm_mode_config_reset(dev);
- return 0;
-
-err_drm_mode_cleanup:
- drm_mode_config_cleanup(dev);
- return ret;
-}
-
-void vbox_mode_fini(struct vbox_private *vbox)
-{
- drm_mode_config_cleanup(&vbox->ddev);
-}
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/*
- * Copyright (C) 2017 Oracle Corporation
- * Copyright 2017 Canonical
- * Authors: Andreas Pokorny
- */
-
-#include "vbox_drv.h"
-
-/*
- * Based on qxl_prime.c:
- * Empty Implementations as there should not be any other driver for a virtual
- * device that might share buffers with vboxvideo
- */
-
-int vbox_gem_prime_pin(struct drm_gem_object *obj)
-{
- WARN_ONCE(1, "not implemented");
- return -ENODEV;
-}
-
-void vbox_gem_prime_unpin(struct drm_gem_object *obj)
-{
- WARN_ONCE(1, "not implemented");
-}
-
-struct sg_table *vbox_gem_prime_get_sg_table(struct drm_gem_object *obj)
-{
- WARN_ONCE(1, "not implemented");
- return ERR_PTR(-ENODEV);
-}
-
-struct drm_gem_object *vbox_gem_prime_import_sg_table(
- struct drm_device *dev, struct dma_buf_attachment *attach,
- struct sg_table *table)
-{
- WARN_ONCE(1, "not implemented");
- return ERR_PTR(-ENODEV);
-}
-
-void *vbox_gem_prime_vmap(struct drm_gem_object *obj)
-{
- WARN_ONCE(1, "not implemented");
- return ERR_PTR(-ENODEV);
-}
-
-void vbox_gem_prime_vunmap(struct drm_gem_object *obj, void *vaddr)
-{
- WARN_ONCE(1, "not implemented");
-}
-
-int vbox_gem_prime_mmap(struct drm_gem_object *obj, struct vm_area_struct *area)
-{
- WARN_ONCE(1, "not implemented");
- return -ENODEV;
-}
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/*
- * Copyright (C) 2013-2017 Oracle Corporation
- * This file is based on ast_ttm.c
- * Copyright 2012 Red Hat Inc.
- * Authors: Dave Airlie <airlied@redhat.com>
- * Michael Thayer <michael.thayer@oracle.com>
- */
-#include <linux/pci.h>
-#include <drm/drm_file.h>
-#include <drm/ttm/ttm_page_alloc.h>
-#include "vbox_drv.h"
-
-static inline struct vbox_private *vbox_bdev(struct ttm_bo_device *bd)
-{
- return container_of(bd, struct vbox_private, ttm.bdev);
-}
-
-static void vbox_bo_ttm_destroy(struct ttm_buffer_object *tbo)
-{
- struct vbox_bo *bo;
-
- bo = container_of(tbo, struct vbox_bo, bo);
-
- drm_gem_object_release(&bo->gem);
- kfree(bo);
-}
-
-static bool vbox_ttm_bo_is_vbox_bo(struct ttm_buffer_object *bo)
-{
- if (bo->destroy == &vbox_bo_ttm_destroy)
- return true;
-
- return false;
-}
-
-static int
-vbox_bo_init_mem_type(struct ttm_bo_device *bdev, u32 type,
- struct ttm_mem_type_manager *man)
-{
- switch (type) {
- case TTM_PL_SYSTEM:
- man->flags = TTM_MEMTYPE_FLAG_MAPPABLE;
- man->available_caching = TTM_PL_MASK_CACHING;
- man->default_caching = TTM_PL_FLAG_CACHED;
- break;
- case TTM_PL_VRAM:
- man->func = &ttm_bo_manager_func;
- man->flags = TTM_MEMTYPE_FLAG_FIXED | TTM_MEMTYPE_FLAG_MAPPABLE;
- man->available_caching = TTM_PL_FLAG_UNCACHED | TTM_PL_FLAG_WC;
- man->default_caching = TTM_PL_FLAG_WC;
- break;
- default:
- DRM_ERROR("Unsupported memory type %u\n", (unsigned int)type);
- return -EINVAL;
- }
-
- return 0;
-}
-
-static void
-vbox_bo_evict_flags(struct ttm_buffer_object *bo, struct ttm_placement *pl)
-{
- struct vbox_bo *vboxbo = vbox_bo(bo);
-
- if (!vbox_ttm_bo_is_vbox_bo(bo))
- return;
-
- vbox_ttm_placement(vboxbo, TTM_PL_FLAG_SYSTEM);
- *pl = vboxbo->placement;
-}
-
-static int vbox_bo_verify_access(struct ttm_buffer_object *bo,
- struct file *filp)
-{
- return 0;
-}
-
-static int vbox_ttm_io_mem_reserve(struct ttm_bo_device *bdev,
- struct ttm_mem_reg *mem)
-{
- struct ttm_mem_type_manager *man = &bdev->man[mem->mem_type];
- struct vbox_private *vbox = vbox_bdev(bdev);
-
- mem->bus.addr = NULL;
- mem->bus.offset = 0;
- mem->bus.size = mem->num_pages << PAGE_SHIFT;
- mem->bus.base = 0;
- mem->bus.is_iomem = false;
- if (!(man->flags & TTM_MEMTYPE_FLAG_MAPPABLE))
- return -EINVAL;
- switch (mem->mem_type) {
- case TTM_PL_SYSTEM:
- /* system memory */
- return 0;
- case TTM_PL_VRAM:
- mem->bus.offset = mem->start << PAGE_SHIFT;
- mem->bus.base = pci_resource_start(vbox->ddev.pdev, 0);
- mem->bus.is_iomem = true;
- break;
- default:
- return -EINVAL;
- }
- return 0;
-}
-
-static void vbox_ttm_io_mem_free(struct ttm_bo_device *bdev,
- struct ttm_mem_reg *mem)
-{
-}
-
-static void vbox_ttm_backend_destroy(struct ttm_tt *tt)
-{
- ttm_tt_fini(tt);
- kfree(tt);
-}
-
-static struct ttm_backend_func vbox_tt_backend_func = {
- .destroy = &vbox_ttm_backend_destroy,
-};
-
-static struct ttm_tt *vbox_ttm_tt_create(struct ttm_buffer_object *bo,
- u32 page_flags)
-{
- struct ttm_tt *tt;
-
- tt = kzalloc(sizeof(*tt), GFP_KERNEL);
- if (!tt)
- return NULL;
-
- tt->func = &vbox_tt_backend_func;
- if (ttm_tt_init(tt, bo, page_flags)) {
- kfree(tt);
- return NULL;
- }
-
- return tt;
-}
-
-static struct ttm_bo_driver vbox_bo_driver = {
- .ttm_tt_create = vbox_ttm_tt_create,
- .init_mem_type = vbox_bo_init_mem_type,
- .eviction_valuable = ttm_bo_eviction_valuable,
- .evict_flags = vbox_bo_evict_flags,
- .verify_access = vbox_bo_verify_access,
- .io_mem_reserve = &vbox_ttm_io_mem_reserve,
- .io_mem_free = &vbox_ttm_io_mem_free,
-};
-
-int vbox_mm_init(struct vbox_private *vbox)
-{
- int ret;
- struct drm_device *dev = &vbox->ddev;
- struct ttm_bo_device *bdev = &vbox->ttm.bdev;
-
- ret = ttm_bo_device_init(&vbox->ttm.bdev,
- &vbox_bo_driver,
- dev->anon_inode->i_mapping,
- DRM_FILE_PAGE_OFFSET, true);
- if (ret) {
- DRM_ERROR("Error initialising bo driver; %d\n", ret);
- return ret;
- }
-
- ret = ttm_bo_init_mm(bdev, TTM_PL_VRAM,
- vbox->available_vram_size >> PAGE_SHIFT);
- if (ret) {
- DRM_ERROR("Failed ttm VRAM init: %d\n", ret);
- goto err_device_release;
- }
-
-#ifdef DRM_MTRR_WC
- vbox->fb_mtrr = drm_mtrr_add(pci_resource_start(dev->pdev, 0),
- pci_resource_len(dev->pdev, 0),
- DRM_MTRR_WC);
-#else
- vbox->fb_mtrr = arch_phys_wc_add(pci_resource_start(dev->pdev, 0),
- pci_resource_len(dev->pdev, 0));
-#endif
- return 0;
-
-err_device_release:
- ttm_bo_device_release(&vbox->ttm.bdev);
- return ret;
-}
-
-void vbox_mm_fini(struct vbox_private *vbox)
-{
-#ifdef DRM_MTRR_WC
- drm_mtrr_del(vbox->fb_mtrr,
- pci_resource_start(vbox->ddev.pdev, 0),
- pci_resource_len(vbox->ddev.pdev, 0), DRM_MTRR_WC);
-#else
- arch_phys_wc_del(vbox->fb_mtrr);
-#endif
- ttm_bo_device_release(&vbox->ttm.bdev);
-}
-
-void vbox_ttm_placement(struct vbox_bo *bo, int domain)
-{
- unsigned int i;
- u32 c = 0;
-
- bo->placement.placement = bo->placements;
- bo->placement.busy_placement = bo->placements;
-
- if (domain & TTM_PL_FLAG_VRAM)
- bo->placements[c++].flags =
- TTM_PL_FLAG_WC | TTM_PL_FLAG_UNCACHED | TTM_PL_FLAG_VRAM;
- if (domain & TTM_PL_FLAG_SYSTEM)
- bo->placements[c++].flags =
- TTM_PL_MASK_CACHING | TTM_PL_FLAG_SYSTEM;
- if (!c)
- bo->placements[c++].flags =
- TTM_PL_MASK_CACHING | TTM_PL_FLAG_SYSTEM;
-
- bo->placement.num_placement = c;
- bo->placement.num_busy_placement = c;
-
- for (i = 0; i < c; ++i) {
- bo->placements[i].fpfn = 0;
- bo->placements[i].lpfn = 0;
- }
-}
-
-int vbox_bo_create(struct vbox_private *vbox, int size, int align,
- u32 flags, struct vbox_bo **pvboxbo)
-{
- struct vbox_bo *vboxbo;
- size_t acc_size;
- int ret;
-
- vboxbo = kzalloc(sizeof(*vboxbo), GFP_KERNEL);
- if (!vboxbo)
- return -ENOMEM;
-
- ret = drm_gem_object_init(&vbox->ddev, &vboxbo->gem, size);
- if (ret)
- goto err_free_vboxbo;
-
- vboxbo->bo.bdev = &vbox->ttm.bdev;
-
- vbox_ttm_placement(vboxbo, TTM_PL_FLAG_VRAM | TTM_PL_FLAG_SYSTEM);
-
- acc_size = ttm_bo_dma_acc_size(&vbox->ttm.bdev, size,
- sizeof(struct vbox_bo));
-
- ret = ttm_bo_init(&vbox->ttm.bdev, &vboxbo->bo, size,
- ttm_bo_type_device, &vboxbo->placement,
- align >> PAGE_SHIFT, false, acc_size,
- NULL, NULL, vbox_bo_ttm_destroy);
- if (ret)
- goto err_free_vboxbo;
-
- *pvboxbo = vboxbo;
-
- return 0;
-
-err_free_vboxbo:
- kfree(vboxbo);
- return ret;
-}
-
-int vbox_bo_pin(struct vbox_bo *bo, u32 pl_flag)
-{
- struct ttm_operation_ctx ctx = { false, false };
- int i, ret;
-
- if (bo->pin_count) {
- bo->pin_count++;
- return 0;
- }
-
- ret = vbox_bo_reserve(bo, false);
- if (ret)
- return ret;
-
- vbox_ttm_placement(bo, pl_flag);
-
- for (i = 0; i < bo->placement.num_placement; i++)
- bo->placements[i].flags |= TTM_PL_FLAG_NO_EVICT;
-
- ret = ttm_bo_validate(&bo->bo, &bo->placement, &ctx);
- if (ret == 0)
- bo->pin_count = 1;
-
- vbox_bo_unreserve(bo);
-
- return ret;
-}
-
-int vbox_bo_unpin(struct vbox_bo *bo)
-{
- struct ttm_operation_ctx ctx = { false, false };
- int i, ret;
-
- if (!bo->pin_count) {
- DRM_ERROR("unpin bad %p\n", bo);
- return 0;
- }
- bo->pin_count--;
- if (bo->pin_count)
- return 0;
-
- ret = vbox_bo_reserve(bo, false);
- if (ret) {
- DRM_ERROR("Error %d reserving bo, leaving it pinned\n", ret);
- return ret;
- }
-
- for (i = 0; i < bo->placement.num_placement; i++)
- bo->placements[i].flags &= ~TTM_PL_FLAG_NO_EVICT;
-
- ret = ttm_bo_validate(&bo->bo, &bo->placement, &ctx);
-
- vbox_bo_unreserve(bo);
-
- return ret;
-}
-
-/*
- * Move a vbox-owned buffer object to system memory if no one else has it
- * pinned. The caller must have pinned it previously, and this call will
- * release the caller's pin.
- */
-int vbox_bo_push_sysram(struct vbox_bo *bo)
-{
- struct ttm_operation_ctx ctx = { false, false };
- int i, ret;
-
- if (!bo->pin_count) {
- DRM_ERROR("unpin bad %p\n", bo);
- return 0;
- }
- bo->pin_count--;
- if (bo->pin_count)
- return 0;
-
- if (bo->kmap.virtual) {
- ttm_bo_kunmap(&bo->kmap);
- bo->kmap.virtual = NULL;
- }
-
- vbox_ttm_placement(bo, TTM_PL_FLAG_SYSTEM);
-
- for (i = 0; i < bo->placement.num_placement; i++)
- bo->placements[i].flags |= TTM_PL_FLAG_NO_EVICT;
-
- ret = ttm_bo_validate(&bo->bo, &bo->placement, &ctx);
- if (ret) {
- DRM_ERROR("pushing to VRAM failed\n");
- return ret;
- }
-
- return 0;
-}
-
-int vbox_mmap(struct file *filp, struct vm_area_struct *vma)
-{
- struct drm_file *file_priv;
- struct vbox_private *vbox;
-
- if (unlikely(vma->vm_pgoff < DRM_FILE_PAGE_OFFSET))
- return -EINVAL;
-
- file_priv = filp->private_data;
- vbox = file_priv->minor->dev->dev_private;
-
- return ttm_bo_mmap(filp, vma, &vbox->ttm.bdev);
-}
-
-void *vbox_bo_kmap(struct vbox_bo *bo)
-{
- int ret;
-
- if (bo->kmap.virtual)
- return bo->kmap.virtual;
-
- ret = ttm_bo_kmap(&bo->bo, 0, bo->bo.num_pages, &bo->kmap);
- if (ret) {
- DRM_ERROR("Error kmapping bo: %d\n", ret);
- return NULL;
- }
-
- return bo->kmap.virtual;
-}
-
-void vbox_bo_kunmap(struct vbox_bo *bo)
-{
- if (bo->kmap.virtual) {
- ttm_bo_kunmap(&bo->kmap);
- bo->kmap.virtual = NULL;
- }
-}
+++ /dev/null
-/* SPDX-License-Identifier: MIT */
-/* Copyright (C) 2006-2016 Oracle Corporation */
-
-#ifndef __VBOXVIDEO_H__
-#define __VBOXVIDEO_H__
-
-#define VBOX_VIDEO_MAX_SCREENS 64
-
-/*
- * The last 4096 bytes of the guest VRAM contains the generic info for all
- * DualView chunks: sizes and offsets of chunks. This is filled by miniport.
- *
- * Last 4096 bytes of each chunk contain chunk specific data: framebuffer info,
- * etc. This is used exclusively by the corresponding instance of a display
- * driver.
- *
- * The VRAM layout:
- * Last 4096 bytes - Adapter information area.
- * 4096 bytes aligned miniport heap (value specified in the config rouded up).
- * Slack - what left after dividing the VRAM.
- * 4096 bytes aligned framebuffers:
- * last 4096 bytes of each framebuffer is the display information area.
- *
- * The Virtual Graphics Adapter information in the guest VRAM is stored by the
- * guest video driver using structures prepended by VBOXVIDEOINFOHDR.
- *
- * When the guest driver writes dword 0 to the VBE_DISPI_INDEX_VBOX_VIDEO
- * the host starts to process the info. The first element at the start of
- * the 4096 bytes region should be normally be a LINK that points to
- * actual information chain. That way the guest driver can have some
- * fixed layout of the information memory block and just rewrite
- * the link to point to relevant memory chain.
- *
- * The processing stops at the END element.
- *
- * The host can access the memory only when the port IO is processed.
- * All data that will be needed later must be copied from these 4096 bytes.
- * But other VRAM can be used by host until the mode is disabled.
- *
- * The guest driver writes dword 0xffffffff to the VBE_DISPI_INDEX_VBOX_VIDEO
- * to disable the mode.
- *
- * VBE_DISPI_INDEX_VBOX_VIDEO is used to read the configuration information
- * from the host and issue commands to the host.
- *
- * The guest writes the VBE_DISPI_INDEX_VBOX_VIDEO index register, the the
- * following operations with the VBE data register can be performed:
- *
- * Operation Result
- * write 16 bit value NOP
- * read 16 bit value count of monitors
- * write 32 bit value set the vbox cmd value and the cmd processed by the host
- * read 32 bit value result of the last vbox command is returned
- */
-
-struct vbva_cmd_hdr {
- s16 x;
- s16 y;
- u16 w;
- u16 h;
-} __packed;
-
-/*
- * The VBVA ring buffer is suitable for transferring large (< 2GB) amount of
- * data. For example big bitmaps which do not fit to the buffer.
- *
- * Guest starts writing to the buffer by initializing a record entry in the
- * records queue. VBVA_F_RECORD_PARTIAL indicates that the record is being
- * written. As data is written to the ring buffer, the guest increases
- * free_offset.
- *
- * The host reads the records on flushes and processes all completed records.
- * When host encounters situation when only a partial record presents and
- * len_and_flags & ~VBVA_F_RECORD_PARTIAL >= VBVA_RING_BUFFER_SIZE -
- * VBVA_RING_BUFFER_THRESHOLD, the host fetched all record data and updates
- * data_offset. After that on each flush the host continues fetching the data
- * until the record is completed.
- */
-
-#define VBVA_RING_BUFFER_SIZE (4194304 - 1024)
-#define VBVA_RING_BUFFER_THRESHOLD (4096)
-
-#define VBVA_MAX_RECORDS (64)
-
-#define VBVA_F_MODE_ENABLED 0x00000001u
-#define VBVA_F_MODE_VRDP 0x00000002u
-#define VBVA_F_MODE_VRDP_RESET 0x00000004u
-#define VBVA_F_MODE_VRDP_ORDER_MASK 0x00000008u
-
-#define VBVA_F_STATE_PROCESSING 0x00010000u
-
-#define VBVA_F_RECORD_PARTIAL 0x80000000u
-
-struct vbva_record {
- u32 len_and_flags;
-} __packed;
-
-/*
- * The minimum HGSMI heap size is PAGE_SIZE (4096 bytes) and is a restriction of
- * the runtime heapsimple API. Use minimum 2 pages here, because the info area
- * also may contain other data (for example hgsmi_host_flags structure).
- */
-#define VBVA_ADAPTER_INFORMATION_SIZE 65536
-#define VBVA_MIN_BUFFER_SIZE 65536
-
-/* The value for port IO to let the adapter to interpret the adapter memory. */
-#define VBOX_VIDEO_DISABLE_ADAPTER_MEMORY 0xFFFFFFFF
-
-/* The value for port IO to let the adapter to interpret the adapter memory. */
-#define VBOX_VIDEO_INTERPRET_ADAPTER_MEMORY 0x00000000
-
-/*
- * The value for port IO to let the adapter to interpret the display memory.
- * The display number is encoded in low 16 bits.
- */
-#define VBOX_VIDEO_INTERPRET_DISPLAY_MEMORY_BASE 0x00010000
-
-struct vbva_host_flags {
- u32 host_events;
- u32 supported_orders;
-} __packed;
-
-struct vbva_buffer {
- struct vbva_host_flags host_flags;
-
- /* The offset where the data start in the buffer. */
- u32 data_offset;
- /* The offset where next data must be placed in the buffer. */
- u32 free_offset;
-
- /* The queue of record descriptions. */
- struct vbva_record records[VBVA_MAX_RECORDS];
- u32 record_first_index;
- u32 record_free_index;
-
- /* Space to leave free when large partial records are transferred. */
- u32 partial_write_tresh;
-
- u32 data_len;
- /* variable size for the rest of the vbva_buffer area in VRAM. */
- u8 data[0];
-} __packed;
-
-#define VBVA_MAX_RECORD_SIZE (128 * 1024 * 1024)
-
-/* guest->host commands */
-#define VBVA_QUERY_CONF32 1
-#define VBVA_SET_CONF32 2
-#define VBVA_INFO_VIEW 3
-#define VBVA_INFO_HEAP 4
-#define VBVA_FLUSH 5
-#define VBVA_INFO_SCREEN 6
-#define VBVA_ENABLE 7
-#define VBVA_MOUSE_POINTER_SHAPE 8
-/* informs host about HGSMI caps. see vbva_caps below */
-#define VBVA_INFO_CAPS 12
-/* configures scanline, see VBVASCANLINECFG below */
-#define VBVA_SCANLINE_CFG 13
-/* requests scanline info, see VBVASCANLINEINFO below */
-#define VBVA_SCANLINE_INFO 14
-/* inform host about VBVA Command submission */
-#define VBVA_CMDVBVA_SUBMIT 16
-/* inform host about VBVA Command submission */
-#define VBVA_CMDVBVA_FLUSH 17
-/* G->H DMA command */
-#define VBVA_CMDVBVA_CTL 18
-/* Query most recent mode hints sent */
-#define VBVA_QUERY_MODE_HINTS 19
-/*
- * Report the guest virtual desktop position and size for mapping host and
- * guest pointer positions.
- */
-#define VBVA_REPORT_INPUT_MAPPING 20
-/* Report the guest cursor position and query the host position. */
-#define VBVA_CURSOR_POSITION 21
-
-/* host->guest commands */
-#define VBVAHG_EVENT 1
-#define VBVAHG_DISPLAY_CUSTOM 2
-
-/* vbva_conf32::index */
-#define VBOX_VBVA_CONF32_MONITOR_COUNT 0
-#define VBOX_VBVA_CONF32_HOST_HEAP_SIZE 1
-/*
- * Returns VINF_SUCCESS if the host can report mode hints via VBVA.
- * Set value to VERR_NOT_SUPPORTED before calling.
- */
-#define VBOX_VBVA_CONF32_MODE_HINT_REPORTING 2
-/*
- * Returns VINF_SUCCESS if the host can report guest cursor enabled status via
- * VBVA. Set value to VERR_NOT_SUPPORTED before calling.
- */
-#define VBOX_VBVA_CONF32_GUEST_CURSOR_REPORTING 3
-/*
- * Returns the currently available host cursor capabilities. Available if
- * VBOX_VBVA_CONF32_GUEST_CURSOR_REPORTING returns success.
- */
-#define VBOX_VBVA_CONF32_CURSOR_CAPABILITIES 4
-/* Returns the supported flags in vbva_infoscreen.flags. */
-#define VBOX_VBVA_CONF32_SCREEN_FLAGS 5
-/* Returns the max size of VBVA record. */
-#define VBOX_VBVA_CONF32_MAX_RECORD_SIZE 6
-
-struct vbva_conf32 {
- u32 index;
- u32 value;
-} __packed;
-
-/* Reserved for historical reasons. */
-#define VBOX_VBVA_CURSOR_CAPABILITY_RESERVED0 BIT(0)
-/*
- * Guest cursor capability: can the host show a hardware cursor at the host
- * pointer location?
- */
-#define VBOX_VBVA_CURSOR_CAPABILITY_HARDWARE BIT(1)
-/* Reserved for historical reasons. */
-#define VBOX_VBVA_CURSOR_CAPABILITY_RESERVED2 BIT(2)
-/* Reserved for historical reasons. Must always be unset. */
-#define VBOX_VBVA_CURSOR_CAPABILITY_RESERVED3 BIT(3)
-/* Reserved for historical reasons. */
-#define VBOX_VBVA_CURSOR_CAPABILITY_RESERVED4 BIT(4)
-/* Reserved for historical reasons. */
-#define VBOX_VBVA_CURSOR_CAPABILITY_RESERVED5 BIT(5)
-
-struct vbva_infoview {
- /* Index of the screen, assigned by the guest. */
- u32 view_index;
-
- /* The screen offset in VRAM, the framebuffer starts here. */
- u32 view_offset;
-
- /* The size of the VRAM memory that can be used for the view. */
- u32 view_size;
-
- /* The recommended maximum size of the VRAM memory for the screen. */
- u32 max_screen_size;
-} __packed;
-
-struct vbva_flush {
- u32 reserved;
-} __packed;
-
-/* vbva_infoscreen.flags */
-#define VBVA_SCREEN_F_NONE 0x0000
-#define VBVA_SCREEN_F_ACTIVE 0x0001
-/*
- * The virtual monitor has been disabled by the guest and should be removed
- * by the host and ignored for purposes of pointer position calculation.
- */
-#define VBVA_SCREEN_F_DISABLED 0x0002
-/*
- * The virtual monitor has been blanked by the guest and should be blacked
- * out by the host using width, height, etc values from the vbva_infoscreen
- * request.
- */
-#define VBVA_SCREEN_F_BLANK 0x0004
-/*
- * The virtual monitor has been blanked by the guest and should be blacked
- * out by the host using the previous mode values for width. height, etc.
- */
-#define VBVA_SCREEN_F_BLANK2 0x0008
-
-struct vbva_infoscreen {
- /* Which view contains the screen. */
- u32 view_index;
-
- /* Physical X origin relative to the primary screen. */
- s32 origin_x;
-
- /* Physical Y origin relative to the primary screen. */
- s32 origin_y;
-
- /* Offset of visible framebuffer relative to the framebuffer start. */
- u32 start_offset;
-
- /* The scan line size in bytes. */
- u32 line_size;
-
- /* Width of the screen. */
- u32 width;
-
- /* Height of the screen. */
- u32 height;
-
- /* Color depth. */
- u16 bits_per_pixel;
-
- /* VBVA_SCREEN_F_* */
- u16 flags;
-} __packed;
-
-/* vbva_enable.flags */
-#define VBVA_F_NONE 0x00000000
-#define VBVA_F_ENABLE 0x00000001
-#define VBVA_F_DISABLE 0x00000002
-/* extended VBVA to be used with WDDM */
-#define VBVA_F_EXTENDED 0x00000004
-/* vbva offset is absolute VRAM offset */
-#define VBVA_F_ABSOFFSET 0x00000008
-
-struct vbva_enable {
- u32 flags;
- u32 offset;
- s32 result;
-} __packed;
-
-struct vbva_enable_ex {
- struct vbva_enable base;
- u32 screen_id;
-} __packed;
-
-struct vbva_mouse_pointer_shape {
- /* The host result. */
- s32 result;
-
- /* VBOX_MOUSE_POINTER_* bit flags. */
- u32 flags;
-
- /* X coordinate of the hot spot. */
- u32 hot_X;
-
- /* Y coordinate of the hot spot. */
- u32 hot_y;
-
- /* Width of the pointer in pixels. */
- u32 width;
-
- /* Height of the pointer in scanlines. */
- u32 height;
-
- /* Pointer data.
- *
- * The data consists of 1 bpp AND mask followed by 32 bpp XOR (color)
- * mask.
- *
- * For pointers without alpha channel the XOR mask pixels are 32 bit
- * values: (lsb)BGR0(msb). For pointers with alpha channel the XOR mask
- * consists of (lsb)BGRA(msb) 32 bit values.
- *
- * Guest driver must create the AND mask for pointers with alpha chan.,
- * so if host does not support alpha, the pointer could be displayed as
- * a normal color pointer. The AND mask can be constructed from alpha
- * values. For example alpha value >= 0xf0 means bit 0 in the AND mask.
- *
- * The AND mask is 1 bpp bitmap with byte aligned scanlines. Size of AND
- * mask, therefore, is and_len = (width + 7) / 8 * height. The padding
- * bits at the end of any scanline are undefined.
- *
- * The XOR mask follows the AND mask on the next 4 bytes aligned offset:
- * u8 *xor = and + (and_len + 3) & ~3
- * Bytes in the gap between the AND and the XOR mask are undefined.
- * XOR mask scanlines have no gap between them and size of XOR mask is:
- * xor_len = width * 4 * height.
- *
- * Preallocate 4 bytes for accessing actual data as p->data.
- */
- u8 data[4];
-} __packed;
-
-/* pointer is visible */
-#define VBOX_MOUSE_POINTER_VISIBLE 0x0001
-/* pointer has alpha channel */
-#define VBOX_MOUSE_POINTER_ALPHA 0x0002
-/* pointerData contains new pointer shape */
-#define VBOX_MOUSE_POINTER_SHAPE 0x0004
-
-/*
- * The guest driver can handle asynch guest cmd completion by reading the
- * command offset from io port.
- */
-#define VBVACAPS_COMPLETEGCMD_BY_IOREAD 0x00000001
-/* the guest driver can handle video adapter IRQs */
-#define VBVACAPS_IRQ 0x00000002
-/* The guest can read video mode hints sent via VBVA. */
-#define VBVACAPS_VIDEO_MODE_HINTS 0x00000004
-/* The guest can switch to a software cursor on demand. */
-#define VBVACAPS_DISABLE_CURSOR_INTEGRATION 0x00000008
-/* The guest does not depend on host handling the VBE registers. */
-#define VBVACAPS_USE_VBVA_ONLY 0x00000010
-
-struct vbva_caps {
- s32 rc;
- u32 caps;
-} __packed;
-
-/* Query the most recent mode hints received from the host. */
-struct vbva_query_mode_hints {
- /* The maximum number of screens to return hints for. */
- u16 hints_queried_count;
- /* The size of the mode hint structures directly following this one. */
- u16 hint_structure_guest_size;
- /* Return code for the operation. Initialise to VERR_NOT_SUPPORTED. */
- s32 rc;
-} __packed;
-
-/*
- * Structure in which a mode hint is returned. The guest allocates an array
- * of these immediately after the vbva_query_mode_hints structure.
- * To accommodate future extensions, the vbva_query_mode_hints structure
- * specifies the size of the vbva_modehint structures allocated by the guest,
- * and the host only fills out structure elements which fit into that size. The
- * host should fill any unused members (e.g. dx, dy) or structure space on the
- * end with ~0. The whole structure can legally be set to ~0 to skip a screen.
- */
-struct vbva_modehint {
- u32 magic;
- u32 cx;
- u32 cy;
- u32 bpp; /* Which has never been used... */
- u32 display;
- u32 dx; /* X offset into the virtual frame-buffer. */
- u32 dy; /* Y offset into the virtual frame-buffer. */
- u32 enabled; /* Not flags. Add new members for new flags. */
-} __packed;
-
-#define VBVAMODEHINT_MAGIC 0x0801add9u
-
-/*
- * Report the rectangle relative to which absolute pointer events should be
- * expressed. This information remains valid until the next VBVA resize event
- * for any screen, at which time it is reset to the bounding rectangle of all
- * virtual screens and must be re-set.
- */
-struct vbva_report_input_mapping {
- s32 x; /* Upper left X co-ordinate relative to the first screen. */
- s32 y; /* Upper left Y co-ordinate relative to the first screen. */
- u32 cx; /* Rectangle width. */
- u32 cy; /* Rectangle height. */
-} __packed;
-
-/*
- * Report the guest cursor position and query the host one. The host may wish
- * to use the guest information to re-position its own cursor (though this is
- * currently unlikely).
- */
-struct vbva_cursor_position {
- u32 report_position; /* Are we reporting a position? */
- u32 x; /* Guest cursor X position */
- u32 y; /* Guest cursor Y position */
-} __packed;
-
-#endif
+++ /dev/null
-/* SPDX-License-Identifier: MIT */
-/* Copyright (C) 2006-2016 Oracle Corporation */
-
-#ifndef __VBOXVIDEO_GUEST_H__
-#define __VBOXVIDEO_GUEST_H__
-
-#include <linux/genalloc.h>
-#include "vboxvideo.h"
-
-/*
- * Structure grouping the context needed for sending graphics acceleration
- * information to the host via VBVA. Each screen has its own VBVA buffer.
- */
-struct vbva_buf_ctx {
- /* Offset of the buffer in the VRAM section for the screen */
- u32 buffer_offset;
- /* Length of the buffer in bytes */
- u32 buffer_length;
- /* Set if we wrote to the buffer faster than the host could read it */
- bool buffer_overflow;
- /* VBVA record that we are currently preparing for the host, or NULL */
- struct vbva_record *record;
- /*
- * Pointer to the VBVA buffer mapped into the current address space.
- * Will be NULL if VBVA is not enabled.
- */
- struct vbva_buffer *vbva;
-};
-
-int hgsmi_report_flags_location(struct gen_pool *ctx, u32 location);
-int hgsmi_send_caps_info(struct gen_pool *ctx, u32 caps);
-int hgsmi_test_query_conf(struct gen_pool *ctx);
-int hgsmi_query_conf(struct gen_pool *ctx, u32 index, u32 *value_ret);
-int hgsmi_update_pointer_shape(struct gen_pool *ctx, u32 flags,
- u32 hot_x, u32 hot_y, u32 width, u32 height,
- u8 *pixels, u32 len);
-int hgsmi_cursor_position(struct gen_pool *ctx, bool report_position,
- u32 x, u32 y, u32 *x_host, u32 *y_host);
-
-bool vbva_enable(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
- struct vbva_buffer *vbva, s32 screen);
-void vbva_disable(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
- s32 screen);
-bool vbva_buffer_begin_update(struct vbva_buf_ctx *vbva_ctx,
- struct gen_pool *ctx);
-void vbva_buffer_end_update(struct vbva_buf_ctx *vbva_ctx);
-bool vbva_write(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
- const void *p, u32 len);
-void vbva_setup_buffer_context(struct vbva_buf_ctx *vbva_ctx,
- u32 buffer_offset, u32 buffer_length);
-
-void hgsmi_process_display_info(struct gen_pool *ctx, u32 display,
- s32 origin_x, s32 origin_y, u32 start_offset,
- u32 pitch, u32 width, u32 height,
- u16 bpp, u16 flags);
-int hgsmi_update_input_mapping(struct gen_pool *ctx, s32 origin_x, s32 origin_y,
- u32 width, u32 height);
-int hgsmi_get_mode_hints(struct gen_pool *ctx, unsigned int screens,
- struct vbva_modehint *hints);
-
-#endif
+++ /dev/null
-/* SPDX-License-Identifier: MIT */
-/* Copyright (C) 2006-2016 Oracle Corporation */
-
-#ifndef __VBOXVIDEO_VBE_H__
-#define __VBOXVIDEO_VBE_H__
-
-/* GUEST <-> HOST Communication API */
-
-#define VBE_DISPI_BANK_ADDRESS 0xA0000
-#define VBE_DISPI_BANK_SIZE_KB 64
-
-#define VBE_DISPI_MAX_XRES 16384
-#define VBE_DISPI_MAX_YRES 16384
-#define VBE_DISPI_MAX_BPP 32
-
-#define VBE_DISPI_IOPORT_INDEX 0x01CE
-#define VBE_DISPI_IOPORT_DATA 0x01CF
-
-#define VBE_DISPI_IOPORT_DAC_WRITE_INDEX 0x03C8
-#define VBE_DISPI_IOPORT_DAC_DATA 0x03C9
-
-#define VBE_DISPI_INDEX_ID 0x0
-#define VBE_DISPI_INDEX_XRES 0x1
-#define VBE_DISPI_INDEX_YRES 0x2
-#define VBE_DISPI_INDEX_BPP 0x3
-#define VBE_DISPI_INDEX_ENABLE 0x4
-#define VBE_DISPI_INDEX_BANK 0x5
-#define VBE_DISPI_INDEX_VIRT_WIDTH 0x6
-#define VBE_DISPI_INDEX_VIRT_HEIGHT 0x7
-#define VBE_DISPI_INDEX_X_OFFSET 0x8
-#define VBE_DISPI_INDEX_Y_OFFSET 0x9
-#define VBE_DISPI_INDEX_VBOX_VIDEO 0xa
-#define VBE_DISPI_INDEX_FB_BASE_HI 0xb
-
-#define VBE_DISPI_ID0 0xB0C0
-#define VBE_DISPI_ID1 0xB0C1
-#define VBE_DISPI_ID2 0xB0C2
-#define VBE_DISPI_ID3 0xB0C3
-#define VBE_DISPI_ID4 0xB0C4
-
-#define VBE_DISPI_ID_VBOX_VIDEO 0xBE00
-/* The VBOX interface id. Indicates support for VBVA shared memory interface. */
-#define VBE_DISPI_ID_HGSMI 0xBE01
-#define VBE_DISPI_ID_ANYX 0xBE02
-
-#define VBE_DISPI_DISABLED 0x00
-#define VBE_DISPI_ENABLED 0x01
-#define VBE_DISPI_GETCAPS 0x02
-#define VBE_DISPI_8BIT_DAC 0x20
-
-#define VGA_PORT_HGSMI_HOST 0x3b0
-#define VGA_PORT_HGSMI_GUEST 0x3d0
-
-#endif
+++ /dev/null
-// SPDX-License-Identifier: MIT
-/* Copyright (C) 2006-2017 Oracle Corporation */
-
-#include <linux/vbox_err.h>
-#include "vbox_drv.h"
-#include "vboxvideo_guest.h"
-#include "hgsmi_channels.h"
-
-/*
- * There is a hardware ring buffer in the graphics device video RAM, formerly
- * in the VBox VMMDev PCI memory space.
- * All graphics commands go there serialized by vbva_buffer_begin_update.
- * and vbva_buffer_end_update.
- *
- * free_offset is writing position. data_offset is reading position.
- * free_offset == data_offset means buffer is empty.
- * There must be always gap between data_offset and free_offset when data
- * are in the buffer.
- * Guest only changes free_offset, host changes data_offset.
- */
-
-static u32 vbva_buffer_available(const struct vbva_buffer *vbva)
-{
- s32 diff = vbva->data_offset - vbva->free_offset;
-
- return diff > 0 ? diff : vbva->data_len + diff;
-}
-
-static void vbva_buffer_place_data_at(struct vbva_buf_ctx *vbva_ctx,
- const void *p, u32 len, u32 offset)
-{
- struct vbva_buffer *vbva = vbva_ctx->vbva;
- u32 bytes_till_boundary = vbva->data_len - offset;
- u8 *dst = &vbva->data[offset];
- s32 diff = len - bytes_till_boundary;
-
- if (diff <= 0) {
- /* Chunk will not cross buffer boundary. */
- memcpy(dst, p, len);
- } else {
- /* Chunk crosses buffer boundary. */
- memcpy(dst, p, bytes_till_boundary);
- memcpy(&vbva->data[0], (u8 *)p + bytes_till_boundary, diff);
- }
-}
-
-static void vbva_buffer_flush(struct gen_pool *ctx)
-{
- struct vbva_flush *p;
-
- p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA, VBVA_FLUSH);
- if (!p)
- return;
-
- p->reserved = 0;
-
- hgsmi_buffer_submit(ctx, p);
- hgsmi_buffer_free(ctx, p);
-}
-
-bool vbva_write(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
- const void *p, u32 len)
-{
- struct vbva_record *record;
- struct vbva_buffer *vbva;
- u32 available;
-
- vbva = vbva_ctx->vbva;
- record = vbva_ctx->record;
-
- if (!vbva || vbva_ctx->buffer_overflow ||
- !record || !(record->len_and_flags & VBVA_F_RECORD_PARTIAL))
- return false;
-
- available = vbva_buffer_available(vbva);
-
- while (len > 0) {
- u32 chunk = len;
-
- if (chunk >= available) {
- vbva_buffer_flush(ctx);
- available = vbva_buffer_available(vbva);
- }
-
- if (chunk >= available) {
- if (WARN_ON(available <= vbva->partial_write_tresh)) {
- vbva_ctx->buffer_overflow = true;
- return false;
- }
- chunk = available - vbva->partial_write_tresh;
- }
-
- vbva_buffer_place_data_at(vbva_ctx, p, chunk,
- vbva->free_offset);
-
- vbva->free_offset = (vbva->free_offset + chunk) %
- vbva->data_len;
- record->len_and_flags += chunk;
- available -= chunk;
- len -= chunk;
- p += chunk;
- }
-
- return true;
-}
-
-static bool vbva_inform_host(struct vbva_buf_ctx *vbva_ctx,
- struct gen_pool *ctx, s32 screen, bool enable)
-{
- struct vbva_enable_ex *p;
- bool ret;
-
- p = hgsmi_buffer_alloc(ctx, sizeof(*p), HGSMI_CH_VBVA, VBVA_ENABLE);
- if (!p)
- return false;
-
- p->base.flags = enable ? VBVA_F_ENABLE : VBVA_F_DISABLE;
- p->base.offset = vbva_ctx->buffer_offset;
- p->base.result = VERR_NOT_SUPPORTED;
- if (screen >= 0) {
- p->base.flags |= VBVA_F_EXTENDED | VBVA_F_ABSOFFSET;
- p->screen_id = screen;
- }
-
- hgsmi_buffer_submit(ctx, p);
-
- if (enable)
- ret = p->base.result >= 0;
- else
- ret = true;
-
- hgsmi_buffer_free(ctx, p);
-
- return ret;
-}
-
-bool vbva_enable(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
- struct vbva_buffer *vbva, s32 screen)
-{
- bool ret = false;
-
- memset(vbva, 0, sizeof(*vbva));
- vbva->partial_write_tresh = 256;
- vbva->data_len = vbva_ctx->buffer_length - sizeof(struct vbva_buffer);
- vbva_ctx->vbva = vbva;
-
- ret = vbva_inform_host(vbva_ctx, ctx, screen, true);
- if (!ret)
- vbva_disable(vbva_ctx, ctx, screen);
-
- return ret;
-}
-
-void vbva_disable(struct vbva_buf_ctx *vbva_ctx, struct gen_pool *ctx,
- s32 screen)
-{
- vbva_ctx->buffer_overflow = false;
- vbva_ctx->record = NULL;
- vbva_ctx->vbva = NULL;
-
- vbva_inform_host(vbva_ctx, ctx, screen, false);
-}
-
-bool vbva_buffer_begin_update(struct vbva_buf_ctx *vbva_ctx,
- struct gen_pool *ctx)
-{
- struct vbva_record *record;
- u32 next;
-
- if (!vbva_ctx->vbva ||
- !(vbva_ctx->vbva->host_flags.host_events & VBVA_F_MODE_ENABLED))
- return false;
-
- WARN_ON(vbva_ctx->buffer_overflow || vbva_ctx->record);
-
- next = (vbva_ctx->vbva->record_free_index + 1) % VBVA_MAX_RECORDS;
-
- /* Flush if all slots in the records queue are used */
- if (next == vbva_ctx->vbva->record_first_index)
- vbva_buffer_flush(ctx);
-
- /* If even after flush there is no place then fail the request */
- if (next == vbva_ctx->vbva->record_first_index)
- return false;
-
- record = &vbva_ctx->vbva->records[vbva_ctx->vbva->record_free_index];
- record->len_and_flags = VBVA_F_RECORD_PARTIAL;
- vbva_ctx->vbva->record_free_index = next;
- /* Remember which record we are using. */
- vbva_ctx->record = record;
-
- return true;
-}
-
-void vbva_buffer_end_update(struct vbva_buf_ctx *vbva_ctx)
-{
- struct vbva_record *record = vbva_ctx->record;
-
- WARN_ON(!vbva_ctx->vbva || !record ||
- !(record->len_and_flags & VBVA_F_RECORD_PARTIAL));
-
- /* Mark the record completed. */
- record->len_and_flags &= ~VBVA_F_RECORD_PARTIAL;
-
- vbva_ctx->buffer_overflow = false;
- vbva_ctx->record = NULL;
-}
-
-void vbva_setup_buffer_context(struct vbva_buf_ctx *vbva_ctx,
- u32 buffer_offset, u32 buffer_length)
-{
- vbva_ctx->buffer_offset = buffer_offset;
- vbva_ctx->buffer_length = buffer_length;
-}
static bool dw8250_idma_filter(struct dma_chan *chan, void *param)
{
- return param == chan->device->dev->parent;
+ return param == chan->device->dev;
}
/*
data->uart_16550_compatible = true;
}
- /* Platforms with iDMA */
+ /* Platforms with iDMA 64-bit */
if (platform_get_resource_byname(to_platform_device(p->dev),
IORESOURCE_MEM, "lpss_priv")) {
data->dma.rx_param = p->dev->parent;
simple->pulse_resets = true;
}
- simple->resets = of_reset_control_array_get(np, shared_resets, true);
+ simple->resets = of_reset_control_array_get(np, shared_resets, true,
+ true);
if (IS_ERR(simple->resets)) {
ret = PTR_ERR(simple->resets);
dev_err(dev, "failed to get device resets, err=%d\n", ret);
unregister_framebuffer(info);
fb_dealloc_cmap(&info->cmap);
- dma_free_coherent(NULL, PALETTE_SIZE, par->v_palette_base,
+ dma_free_coherent(par->dev, PALETTE_SIZE, par->v_palette_base,
par->p_palette_base);
- dma_free_coherent(NULL, par->vram_size, par->vram_virt,
+ dma_free_coherent(par->dev, par->vram_size, par->vram_virt,
par->vram_phys);
pm_runtime_put_sync(&dev->dev);
pm_runtime_disable(&dev->dev);
par->vram_size = roundup(par->vram_size/8, ulcm);
par->vram_size = par->vram_size * LCD_NUM_BUFFERS;
- par->vram_virt = dma_alloc_coherent(NULL,
+ par->vram_virt = dma_alloc_coherent(par->dev,
par->vram_size,
&par->vram_phys,
GFP_KERNEL | GFP_DMA);
da8xx_fb_fix.line_length - 1;
/* allocate palette buffer */
- par->v_palette_base = dma_alloc_coherent(NULL, PALETTE_SIZE,
+ par->v_palette_base = dma_alloc_coherent(par->dev, PALETTE_SIZE,
&par->p_palette_base,
GFP_KERNEL | GFP_DMA);
if (!par->v_palette_base) {
fb_dealloc_cmap(&da8xx_fb_info->cmap);
err_release_pl_mem:
- dma_free_coherent(NULL, PALETTE_SIZE, par->v_palette_base,
+ dma_free_coherent(par->dev, PALETTE_SIZE, par->v_palette_base,
par->p_palette_base);
err_release_fb_mem:
- dma_free_coherent(NULL, par->vram_size, par->vram_virt, par->vram_phys);
+ dma_free_coherent(par->dev, par->vram_size, par->vram_virt,
+ par->vram_phys);
err_release_fb:
framebuffer_release(da8xx_fb_info);
}
gbe_revision = gbe->ctrlstat & 15;
- gbe_tiles.cpu =
- dma_alloc_coherent(NULL, GBE_TLB_SIZE * sizeof(uint16_t),
- &gbe_tiles.dma, GFP_KERNEL);
+ gbe_tiles.cpu = dmam_alloc_coherent(&p_dev->dev,
+ GBE_TLB_SIZE * sizeof(uint16_t),
+ &gbe_tiles.dma, GFP_KERNEL);
if (!gbe_tiles.cpu) {
printk(KERN_ERR "gbefb: couldn't allocate tiles table\n");
ret = -ENOMEM;
if (!gbe_mem) {
printk(KERN_ERR "gbefb: couldn't map framebuffer\n");
ret = -ENOMEM;
- goto out_tiles_free;
+ goto out_release_mem_region;
}
gbe_dma_addr = 0;
} else {
/* try to allocate memory with the classical allocator
* this has high chance to fail on low memory machines */
- gbe_mem = dma_alloc_wc(NULL, gbe_mem_size, &gbe_dma_addr,
- GFP_KERNEL);
+ gbe_mem = dmam_alloc_attrs(&p_dev->dev, gbe_mem_size,
+ &gbe_dma_addr, GFP_KERNEL,
+ DMA_ATTR_WRITE_COMBINE);
if (!gbe_mem) {
printk(KERN_ERR "gbefb: couldn't allocate framebuffer memory\n");
ret = -ENOMEM;
- goto out_tiles_free;
+ goto out_release_mem_region;
}
gbe_mem_phys = (unsigned long) gbe_dma_addr;
out_gbe_unmap:
arch_phys_wc_del(par->wc_cookie);
- if (gbe_dma_addr)
- dma_free_wc(NULL, gbe_mem_size, gbe_mem, gbe_mem_phys);
-out_tiles_free:
- dma_free_coherent(NULL, GBE_TLB_SIZE * sizeof(uint16_t),
- (void *)gbe_tiles.cpu, gbe_tiles.dma);
out_release_mem_region:
release_mem_region(GBE_BASE, sizeof(struct sgi_gbe));
out_release_framebuffer:
unregister_framebuffer(info);
gbe_turn_off();
arch_phys_wc_del(par->wc_cookie);
- if (gbe_dma_addr)
- dma_free_wc(NULL, gbe_mem_size, gbe_mem, gbe_mem_phys);
- dma_free_coherent(NULL, GBE_TLB_SIZE * sizeof(uint16_t),
- (void *)gbe_tiles.cpu, gbe_tiles.dma);
release_mem_region(GBE_BASE, sizeof(struct sgi_gbe));
gbefb_remove_sysfs(&p_dev->dev);
framebuffer_release(info);
};
struct pxa3xx_gcu_priv {
+ struct device *dev;
void __iomem *mmio_base;
struct clk *clk;
struct pxa3xx_gcu_shared *shared;
if (size != SHARED_SIZE)
return -EINVAL;
- return dma_mmap_coherent(NULL, vma,
+ return dma_mmap_coherent(priv->dev, vma,
priv->shared, priv->shared_phys, size);
case SHARED_SIZE >> PAGE_SHIFT:
platform_set_drvdata(pdev, priv);
priv->resource_mem = r;
+ priv->dev = dev;
pxa3xx_gcu_reset(priv);
pxa3xx_gcu_init_debug_timer(priv);
host = nlmclnt_lookup_host(nlm_init->address, nlm_init->addrlen,
nlm_init->protocol, nlm_version,
nlm_init->hostname, nlm_init->noresvport,
- nlm_init->net);
+ nlm_init->net, nlm_init->cred);
if (host == NULL)
goto out_nohost;
if (host->h_rpcclnt == NULL && nlm_bind_host(host) == NULL)
struct nlm_rqst *req = data;
u32 status = ntohl(req->a_res.status);
- if (RPC_ASSASSINATED(task))
+ if (RPC_SIGNALLED(task))
goto die;
if (task->tk_status < 0) {
struct nlm_rqst *req = data;
u32 status = ntohl(req->a_res.status);
- if (RPC_ASSASSINATED(task))
+ if (RPC_SIGNALLED(task))
goto die;
if (task->tk_status < 0) {
const size_t hostname_len; /* it's length */
const int noresvport; /* use non-priv port */
struct net *net; /* network namespace to bind */
+ const struct cred *cred;
};
/*
host->h_nsmhandle = nsm;
host->h_addrbuf = nsm->sm_addrbuf;
host->net = ni->net;
+ host->h_cred = get_cred(ni->cred),
strlcpy(host->nodename, utsname()->nodename, sizeof(host->nodename));
out:
clnt = host->h_rpcclnt;
if (clnt != NULL)
rpc_shutdown_client(clnt);
+ put_cred(host->h_cred);
kfree(host);
ln->nrhosts--;
* @version: NLM protocol version
* @hostname: '\0'-terminated hostname of server
* @noresvport: 1 if non-privileged port should be used
+ * @net: pointer to net namespace
+ * @cred: pointer to cred
*
* Returns an nlm_host structure that matches the passed-in
* [server address, transport protocol, NLM version, server hostname].
const u32 version,
const char *hostname,
int noresvport,
- struct net *net)
+ struct net *net,
+ const struct cred *cred)
{
struct nlm_lookup_host_info ni = {
.server = 0,
.hostname_len = strlen(hostname),
.noresvport = noresvport,
.net = net,
+ .cred = cred,
};
struct hlist_head *chain;
struct nlm_host *host;
.authflavor = RPC_AUTH_UNIX,
.flags = (RPC_CLNT_CREATE_NOPING |
RPC_CLNT_CREATE_AUTOBIND),
+ .cred = host->h_cred,
};
/*
.version = NSM_VERSION,
.authflavor = RPC_AUTH_NULL,
.flags = RPC_CLNT_CREATE_NOPING,
+ .cred = current_cred(),
};
return rpc_create(&args);
struct nfs_client *clp;
const struct sockaddr *sap = data->addr;
struct nfs_net *nn = net_generic(data->net, nfs_net_id);
+ int error;
again:
list_for_each_entry(clp, &nn->nfs_client_list, cl_share_link) {
if (clp->cl_cons_state > NFS_CS_READY) {
refcount_inc(&clp->cl_count);
spin_unlock(&nn->nfs_client_lock);
- nfs_wait_client_init_complete(clp);
+ error = nfs_wait_client_init_complete(clp);
nfs_put_client(clp);
spin_lock(&nn->nfs_client_lock);
+ if (error < 0)
+ return ERR_PTR(error);
goto again;
}
clp = nfs_match_client(cl_init);
if (clp) {
spin_unlock(&nn->nfs_client_lock);
+ if (IS_ERR(clp))
+ return clp;
if (new)
new->rpc_ops->free_client(new);
return nfs_found_client(cl_init, clp);
.program = &nfs_program,
.version = clp->rpc_ops->version,
.authflavor = flavor,
+ .cred = cl_init->cred,
};
if (test_bit(NFS_CS_DISCRTRY, &clp->cl_flags))
sizeof(server->client->cl_timeout_default));
server->client->cl_timeout = &server->client->cl_timeout_default;
server->client->cl_softrtry = 0;
+ if (server->flags & NFS_MOUNT_SOFTERR)
+ server->client->cl_softerr = 1;
if (server->flags & NFS_MOUNT_SOFT)
server->client->cl_softrtry = 1;
.proto = data->nfs_server.protocol,
.net = data->net,
.timeparms = &timeparms,
+ .cred = server->cred,
};
struct nfs_client *clp;
int error;
ida_destroy(&server->lockowner_id);
ida_destroy(&server->openowner_id);
nfs_free_iostats(server->io_stats);
+ put_cred(server->cred);
kfree(server);
nfs_release_automount_timer();
}
if (!server)
return ERR_PTR(-ENOMEM);
+ server->cred = get_cred(current_cred());
+
error = -ENOMEM;
fattr = nfs_alloc_fattr();
if (fattr == NULL)
if (!server)
return ERR_PTR(-ENOMEM);
+ server->cred = get_cred(source->cred);
+
error = -ENOMEM;
fattr_fsinfo = nfs_alloc_fattr();
if (fattr_fsinfo == NULL)
rcu_read_unlock();
}
+/**
+ * nfs_test_expired_all_delegations - test all delegations for a client
+ * @clp: nfs_client to process
+ *
+ * Helper for handling "recallable state revoked" status from server.
+ */
+void nfs_test_expired_all_delegations(struct nfs_client *clp)
+{
+ nfs_mark_test_expired_all_delegations(clp);
+ nfs4_schedule_state_manager(clp);
+}
+
/**
* nfs_reap_expired_delegations - reap expired delegations
* @clp: nfs_client to process
void nfs_delegation_reap_unclaimed(struct nfs_client *clp);
void nfs_mark_test_expired_all_delegations(struct nfs_client *clp);
+void nfs_test_expired_all_delegations(struct nfs_client *clp);
void nfs_reap_expired_delegations(struct nfs_client *clp);
/* NFSv4 delegation-related procedures */
* We only need to convert from xdr once so future lookups are much simpler
*/
static
-int nfs_readdir_filler(nfs_readdir_descriptor_t *desc, struct page* page)
+int nfs_readdir_filler(void *data, struct page* page)
{
+ nfs_readdir_descriptor_t *desc = data;
struct inode *inode = file_inode(desc->file);
int ret;
static
struct page *get_cache_page(nfs_readdir_descriptor_t *desc)
{
- return read_cache_page(desc->file->f_mapping,
- desc->page_index, (filler_t *)nfs_readdir_filler, desc);
+ return read_cache_page(desc->file->f_mapping, desc->page_index,
+ nfs_readdir_filler, desc);
}
/*
struct nfs_page *req;
unsigned int req_len = min_t(size_t, bytes, PAGE_SIZE - pgbase);
/* XXX do we need to do the eof zeroing found in async_filler? */
- req = nfs_create_request(dreq->ctx, pagevec[i], NULL,
+ req = nfs_create_request(dreq->ctx, pagevec[i],
pgbase, req_len);
if (IS_ERR(req)) {
result = PTR_ERR(req);
}
list_for_each_entry_safe(req, tmp, &reqs, wb_list) {
+ /* Bump the transmission count */
+ req->wb_nio++;
if (!nfs_pageio_add_request(&desc, req)) {
nfs_list_move_request(req, &failed);
spin_lock(&cinfo.inode->i_lock);
req = nfs_list_entry(data->pages.next);
nfs_list_remove_request(req);
if (dreq->flags == NFS_ODIRECT_RESCHED_WRITES) {
+ /*
+ * Despite the reboot, the write was successful,
+ * so reset wb_nio.
+ */
+ req->wb_nio = 0;
/* Note the rewrite will go through mds */
nfs_mark_request_commit(req, NULL, &cinfo, 0);
} else
struct nfs_page *req;
unsigned int req_len = min_t(size_t, bytes, PAGE_SIZE - pgbase);
- req = nfs_create_request(dreq->ctx, pagevec[i], NULL,
+ req = nfs_create_request(dreq->ctx, pagevec[i],
pgbase, req_len);
if (IS_ERR(req)) {
result = PTR_ERR(req);
return 0;
/* Flush writes to the server and return any errors */
- return vfs_fsync(file, 0);
+ return nfs_wb_all(inode);
}
ssize_t
* Flush any dirty pages for this process, and check for write errors.
* The return status from this call provides a reliable indication of
* whether any write errors occurred for this process.
- *
- * Notice that it clears the NFS_CONTEXT_ERROR_WRITE before synching to
- * disk, but it retrieves and clears ctx->error after synching, despite
- * the two being set at the same time in nfs_context_set_write_error().
- * This is because the former is used to notify the _next_ call to
- * nfs_file_write() that a write error occurred, and hence cause it to
- * fall back to doing a synchronous write.
*/
static int
nfs_file_fsync_commit(struct file *file, int datasync)
nfs_inc_stats(inode, NFSIOS_VFSFSYNC);
do_resend = test_and_clear_bit(NFS_CONTEXT_RESEND_WRITES, &ctx->flags);
status = nfs_commit_inode(inode, FLUSH_SYNC);
- if (test_bit(NFS_CONTEXT_ERROR_WRITE, &ctx->flags)) {
- ret = xchg(&ctx->error, 0);
- if (ret)
- goto out;
- }
+ if (status == 0)
+ status = file_check_and_advance_wb_err(file);
if (status < 0) {
ret = status;
goto out;
trace_nfs_fsync_enter(inode);
do {
- struct nfs_open_context *ctx = nfs_file_open_context(file);
- ret = filemap_write_and_wait_range(inode->i_mapping, start, end);
- if (test_and_clear_bit(NFS_CONTEXT_ERROR_WRITE, &ctx->flags)) {
- int ret2 = xchg(&ctx->error, 0);
- if (ret2)
- ret = ret2;
- }
+ ret = file_write_and_wait_range(file, start, end);
if (ret != 0)
break;
ret = nfs_file_fsync_commit(file, datasync);
struct nfs_open_context *ctx;
ctx = nfs_file_open_context(filp);
- if (test_bit(NFS_CONTEXT_ERROR_WRITE, &ctx->flags) ||
- nfs_ctx_key_to_expire(ctx, inode))
+ if (nfs_ctx_key_to_expire(ctx, inode))
return 1;
return 0;
}
/* Return error values */
if (nfs_need_check_write(file, inode)) {
- int err = vfs_fsync(file, 0);
+ int err = nfs_wb_all(inode);
if (err < 0)
result = err;
}
* Flush all pending writes before doing anything
* with locks..
*/
- vfs_fsync(filp, 0);
+ nfs_wb_all(inode);
l_ctx = nfs_get_lock_context(nfs_file_open_context(filp));
if (!IS_ERR(l_ctx)) {
status = filelayout_check_deviceid(lo, fl, gfp_flags);
if (status) {
pnfs_put_lseg(lseg);
- lseg = ERR_PTR(status);
+ lseg = NULL;
}
out:
return lseg;
pnfs_generic_pg_check_layout(pgio);
if (!pgio->pg_lseg) {
pgio->pg_lseg = fl_pnfs_update_layout(pgio->pg_inode,
- req->wb_context,
+ nfs_req_openctx(req),
0,
NFS4_MAX_UINT64,
IOMODE_READ,
pnfs_generic_pg_check_layout(pgio);
if (!pgio->pg_lseg) {
pgio->pg_lseg = fl_pnfs_update_layout(pgio->pg_inode,
- req->wb_context,
+ nfs_req_openctx(req),
0,
NFS4_MAX_UINT64,
IOMODE_RW,
#define FF_LAYOUT_POLL_RETRY_MAX (15*HZ)
#define FF_LAYOUTRETURN_MAXERR 20
+static unsigned short io_maxretrans;
+
static void ff_layout_read_record_layoutstats_done(struct rpc_task *task,
struct nfs_pgio_header *hdr);
static int ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,
{
pnfs_put_lseg(pgio->pg_lseg);
pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode,
- req->wb_context,
+ nfs_req_openctx(req),
0,
NFS4_MAX_UINT64,
IOMODE_READ,
pgm = &pgio->pg_mirrors[0];
pgm->pg_bsize = mirror->mirror_ds->ds_versions[0].rsize;
+ pgio->pg_maxretrans = io_maxretrans;
return;
out_nolseg:
if (pgio->pg_error < 0)
pnfs_generic_pg_check_layout(pgio);
if (!pgio->pg_lseg) {
pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode,
- req->wb_context,
+ nfs_req_openctx(req),
0,
NFS4_MAX_UINT64,
IOMODE_RW,
pgm->pg_bsize = mirror->mirror_ds->ds_versions[0].wsize;
}
+ pgio->pg_maxretrans = io_maxretrans;
return;
out_mds:
{
if (!pgio->pg_lseg) {
pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode,
- req->wb_context,
+ nfs_req_openctx(req),
0,
NFS4_MAX_UINT64,
IOMODE_RW,
module_init(nfs4flexfilelayout_init);
module_exit(nfs4flexfilelayout_exit);
+
+module_param(io_maxretrans, ushort, 0644);
+MODULE_PARM_DESC(io_maxretrans, "The number of times the NFSv4.1 client "
+ "retries an I/O request before returning an error. ");
spin_lock(&inode->i_lock);
res = __nfs_find_lock_context(ctx);
if (res == NULL) {
- list_add_tail_rcu(&new->list, &ctx->lock_context.list);
- new->open_context = ctx;
- res = new;
- new = NULL;
+ new->open_context = get_nfs_open_context(ctx);
+ if (new->open_context) {
+ list_add_tail_rcu(&new->list,
+ &ctx->lock_context.list);
+ res = new;
+ new = NULL;
+ } else
+ res = ERR_PTR(-EBADF);
}
spin_unlock(&inode->i_lock);
kfree(new);
return;
list_del_rcu(&l_ctx->list);
spin_unlock(&inode->i_lock);
+ put_nfs_open_context(ctx);
kfree_rcu(l_ctx, rcu_head);
}
EXPORT_SYMBOL_GPL(nfs_put_lock_context);
u32 minorversion;
struct net *net;
const struct rpc_timeout *timeparms;
+ const struct cred *cred;
};
/*
case -ESTALE:
case -E2BIG:
case -ENOMEM:
+ case -ETIMEDOUT:
return true;
default:
return false;
}
}
-static inline void nfs_context_set_write_error(struct nfs_open_context *ctx, int error)
-{
- ctx->error = error;
- smp_wmb();
- set_bit(NFS_CONTEXT_ERROR_WRITE, &ctx->flags);
-}
.program = &mnt_program,
.version = info->version,
.authflavor = RPC_AUTH_UNIX,
+ .cred = current_cred(),
};
struct rpc_clnt *mnt_clnt;
int status;
.version = info->version,
.authflavor = RPC_AUTH_UNIX,
.flags = RPC_CLNT_CREATE_NOPING,
+ .cred = current_cred(),
};
struct rpc_message msg = {
.rpc_argp = info->dirpath,
* or decoded inline.
*/
+static struct user_namespace *rpc_userns(const struct rpc_clnt *clnt)
+{
+ if (clnt && clnt->cl_cred)
+ return clnt->cl_cred->user_ns;
+ return &init_user_ns;
+}
+
+static struct user_namespace *rpc_rqst_userns(const struct rpc_rqst *rqstp)
+{
+ if (rqstp->rq_task)
+ return rpc_userns(rqstp->rq_task->tk_client);
+ return &init_user_ns;
+}
+
/*
* typedef opaque nfsdata<>;
*/
* };
*
*/
-static int decode_fattr(struct xdr_stream *xdr, struct nfs_fattr *fattr)
+static int decode_fattr(struct xdr_stream *xdr, struct nfs_fattr *fattr,
+ struct user_namespace *userns)
{
u32 rdev, type;
__be32 *p;
fattr->mode = be32_to_cpup(p++);
fattr->nlink = be32_to_cpup(p++);
- fattr->uid = make_kuid(&init_user_ns, be32_to_cpup(p++));
+ fattr->uid = make_kuid(userns, be32_to_cpup(p++));
if (!uid_valid(fattr->uid))
goto out_uid;
- fattr->gid = make_kgid(&init_user_ns, be32_to_cpup(p++));
+ fattr->gid = make_kgid(userns, be32_to_cpup(p++));
if (!gid_valid(fattr->gid))
goto out_gid;
return p;
}
-static void encode_sattr(struct xdr_stream *xdr, const struct iattr *attr)
+static void encode_sattr(struct xdr_stream *xdr, const struct iattr *attr,
+ struct user_namespace *userns)
{
struct timespec ts;
__be32 *p;
else
*p++ = cpu_to_be32(NFS2_SATTR_NOT_SET);
if (attr->ia_valid & ATTR_UID)
- *p++ = cpu_to_be32(from_kuid(&init_user_ns, attr->ia_uid));
+ *p++ = cpu_to_be32(from_kuid_munged(userns, attr->ia_uid));
else
*p++ = cpu_to_be32(NFS2_SATTR_NOT_SET);
if (attr->ia_valid & ATTR_GID)
- *p++ = cpu_to_be32(from_kgid(&init_user_ns, attr->ia_gid));
+ *p++ = cpu_to_be32(from_kgid_munged(userns, attr->ia_gid));
else
*p++ = cpu_to_be32(NFS2_SATTR_NOT_SET);
if (attr->ia_valid & ATTR_SIZE)
* };
*/
static int decode_attrstat(struct xdr_stream *xdr, struct nfs_fattr *result,
- __u32 *op_status)
+ __u32 *op_status,
+ struct user_namespace *userns)
{
enum nfs_stat status;
int error;
*op_status = status;
if (status != NFS_OK)
goto out_default;
- error = decode_fattr(xdr, result);
+ error = decode_fattr(xdr, result, userns);
out:
return error;
out_default:
* void;
* };
*/
-static int decode_diropok(struct xdr_stream *xdr, struct nfs_diropok *result)
+static int decode_diropok(struct xdr_stream *xdr, struct nfs_diropok *result,
+ struct user_namespace *userns)
{
int error;
error = decode_fhandle(xdr, result->fh);
if (unlikely(error))
goto out;
- error = decode_fattr(xdr, result->fattr);
+ error = decode_fattr(xdr, result->fattr, userns);
out:
return error;
}
-static int decode_diropres(struct xdr_stream *xdr, struct nfs_diropok *result)
+static int decode_diropres(struct xdr_stream *xdr, struct nfs_diropok *result,
+ struct user_namespace *userns)
{
enum nfs_stat status;
int error;
goto out;
if (status != NFS_OK)
goto out_default;
- error = decode_diropok(xdr, result);
+ error = decode_diropok(xdr, result, userns);
out:
return error;
out_default:
const struct nfs_sattrargs *args = data;
encode_fhandle(xdr, args->fh);
- encode_sattr(xdr, args->sattr);
+ encode_sattr(xdr, args->sattr, rpc_rqst_userns(req));
}
static void nfs2_xdr_enc_diropargs(struct rpc_rqst *req,
const struct nfs_createargs *args = data;
encode_diropargs(xdr, args->fh, args->name, args->len);
- encode_sattr(xdr, args->sattr);
+ encode_sattr(xdr, args->sattr, rpc_rqst_userns(req));
}
static void nfs2_xdr_enc_removeargs(struct rpc_rqst *req,
encode_diropargs(xdr, args->fromfh, args->fromname, args->fromlen);
encode_path(xdr, args->pages, args->pathlen);
- encode_sattr(xdr, args->sattr);
+ encode_sattr(xdr, args->sattr, rpc_rqst_userns(req));
}
/*
static int nfs2_xdr_dec_attrstat(struct rpc_rqst *req, struct xdr_stream *xdr,
void *result)
{
- return decode_attrstat(xdr, result, NULL);
+ return decode_attrstat(xdr, result, NULL, rpc_rqst_userns(req));
}
static int nfs2_xdr_dec_diropres(struct rpc_rqst *req, struct xdr_stream *xdr,
void *result)
{
- return decode_diropres(xdr, result);
+ return decode_diropres(xdr, result, rpc_rqst_userns(req));
}
/*
result->op_status = status;
if (status != NFS_OK)
goto out_default;
- error = decode_fattr(xdr, result->fattr);
+ error = decode_fattr(xdr, result->fattr, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
error = decode_nfsdata(xdr, result);
/* All NFSv2 writes are "file sync" writes */
result->verf->committed = NFS_FILE_SYNC;
- return decode_attrstat(xdr, result->fattr, &result->op_status);
+ return decode_attrstat(xdr, result->fattr, &result->op_status,
+ rpc_rqst_userns(req));
}
/**
.proto = ds_proto,
.net = mds_clp->cl_net,
.timeparms = &ds_timeout,
+ .cred = mds_srv->cred,
};
struct nfs_client *clp;
char buf[INET6_ADDRSTRLEN + 1];
[NF3FIFO] = S_IFIFO,
};
+static struct user_namespace *rpc_userns(const struct rpc_clnt *clnt)
+{
+ if (clnt && clnt->cl_cred)
+ return clnt->cl_cred->user_ns;
+ return &init_user_ns;
+}
+
+static struct user_namespace *rpc_rqst_userns(const struct rpc_rqst *rqstp)
+{
+ if (rqstp->rq_task)
+ return rpc_userns(rqstp->rq_task->tk_client);
+ return &init_user_ns;
+}
+
/*
* Encode/decode NFSv3 basic data types
*
* set_mtime mtime;
* };
*/
-static void encode_sattr3(struct xdr_stream *xdr, const struct iattr *attr)
+static void encode_sattr3(struct xdr_stream *xdr, const struct iattr *attr,
+ struct user_namespace *userns)
{
struct timespec ts;
u32 nbytes;
if (attr->ia_valid & ATTR_UID) {
*p++ = xdr_one;
- *p++ = cpu_to_be32(from_kuid(&init_user_ns, attr->ia_uid));
+ *p++ = cpu_to_be32(from_kuid_munged(userns, attr->ia_uid));
} else
*p++ = xdr_zero;
if (attr->ia_valid & ATTR_GID) {
*p++ = xdr_one;
- *p++ = cpu_to_be32(from_kgid(&init_user_ns, attr->ia_gid));
+ *p++ = cpu_to_be32(from_kgid_munged(userns, attr->ia_gid));
} else
*p++ = xdr_zero;
* nfstime3 ctime;
* };
*/
-static int decode_fattr3(struct xdr_stream *xdr, struct nfs_fattr *fattr)
+static int decode_fattr3(struct xdr_stream *xdr, struct nfs_fattr *fattr,
+ struct user_namespace *userns)
{
umode_t fmode;
__be32 *p;
fattr->mode = (be32_to_cpup(p++) & ~S_IFMT) | fmode;
fattr->nlink = be32_to_cpup(p++);
- fattr->uid = make_kuid(&init_user_ns, be32_to_cpup(p++));
+ fattr->uid = make_kuid(userns, be32_to_cpup(p++));
if (!uid_valid(fattr->uid))
goto out_uid;
- fattr->gid = make_kgid(&init_user_ns, be32_to_cpup(p++));
+ fattr->gid = make_kgid(userns, be32_to_cpup(p++));
if (!gid_valid(fattr->gid))
goto out_gid;
* void;
* };
*/
-static int decode_post_op_attr(struct xdr_stream *xdr, struct nfs_fattr *fattr)
+static int decode_post_op_attr(struct xdr_stream *xdr, struct nfs_fattr *fattr,
+ struct user_namespace *userns)
{
__be32 *p;
if (unlikely(!p))
return -EIO;
if (*p != xdr_zero)
- return decode_fattr3(xdr, fattr);
+ return decode_fattr3(xdr, fattr, userns);
return 0;
}
return 0;
}
-static int decode_wcc_data(struct xdr_stream *xdr, struct nfs_fattr *fattr)
+static int decode_wcc_data(struct xdr_stream *xdr, struct nfs_fattr *fattr,
+ struct user_namespace *userns)
{
int error;
error = decode_pre_op_attr(xdr, fattr);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, fattr);
+ error = decode_post_op_attr(xdr, fattr, userns);
out:
return error;
}
{
const struct nfs3_sattrargs *args = data;
encode_nfs_fh3(xdr, args->fh);
- encode_sattr3(xdr, args->sattr);
+ encode_sattr3(xdr, args->sattr, rpc_rqst_userns(req));
encode_sattrguard3(xdr, args);
}
* };
*/
static void encode_createhow3(struct xdr_stream *xdr,
- const struct nfs3_createargs *args)
+ const struct nfs3_createargs *args,
+ struct user_namespace *userns)
{
encode_uint32(xdr, args->createmode);
switch (args->createmode) {
case NFS3_CREATE_UNCHECKED:
case NFS3_CREATE_GUARDED:
- encode_sattr3(xdr, args->sattr);
+ encode_sattr3(xdr, args->sattr, userns);
break;
case NFS3_CREATE_EXCLUSIVE:
encode_createverf3(xdr, args->verifier);
const struct nfs3_createargs *args = data;
encode_diropargs3(xdr, args->fh, args->name, args->len);
- encode_createhow3(xdr, args);
+ encode_createhow3(xdr, args, rpc_rqst_userns(req));
}
/*
const struct nfs3_mkdirargs *args = data;
encode_diropargs3(xdr, args->fh, args->name, args->len);
- encode_sattr3(xdr, args->sattr);
+ encode_sattr3(xdr, args->sattr, rpc_rqst_userns(req));
}
/*
* };
*/
static void encode_symlinkdata3(struct xdr_stream *xdr,
- const void *data)
+ const void *data,
+ struct user_namespace *userns)
{
const struct nfs3_symlinkargs *args = data;
- encode_sattr3(xdr, args->sattr);
+ encode_sattr3(xdr, args->sattr, userns);
encode_nfspath3(xdr, args->pages, args->pathlen);
}
const struct nfs3_symlinkargs *args = data;
encode_diropargs3(xdr, args->fromfh, args->fromname, args->fromlen);
- encode_symlinkdata3(xdr, args);
+ encode_symlinkdata3(xdr, args, rpc_rqst_userns(req));
xdr->buf->flags |= XDRBUF_WRITE;
}
* };
*/
static void encode_devicedata3(struct xdr_stream *xdr,
- const struct nfs3_mknodargs *args)
+ const struct nfs3_mknodargs *args,
+ struct user_namespace *userns)
{
- encode_sattr3(xdr, args->sattr);
+ encode_sattr3(xdr, args->sattr, userns);
encode_specdata3(xdr, args->rdev);
}
static void encode_mknoddata3(struct xdr_stream *xdr,
- const struct nfs3_mknodargs *args)
+ const struct nfs3_mknodargs *args,
+ struct user_namespace *userns)
{
encode_ftype3(xdr, args->type);
switch (args->type) {
case NF3CHR:
case NF3BLK:
- encode_devicedata3(xdr, args);
+ encode_devicedata3(xdr, args, userns);
break;
case NF3SOCK:
case NF3FIFO:
- encode_sattr3(xdr, args->sattr);
+ encode_sattr3(xdr, args->sattr, userns);
break;
case NF3REG:
case NF3DIR:
const struct nfs3_mknodargs *args = data;
encode_diropargs3(xdr, args->fh, args->name, args->len);
- encode_mknoddata3(xdr, args);
+ encode_mknoddata3(xdr, args, rpc_rqst_userns(req));
}
/*
goto out;
if (status != NFS3_OK)
goto out_default;
- error = decode_fattr3(xdr, result);
+ error = decode_fattr3(xdr, result, rpc_rqst_userns(req));
out:
return error;
out_default:
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_wcc_data(xdr, result);
+ error = decode_wcc_data(xdr, result, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
if (status != NFS3_OK)
struct xdr_stream *xdr,
void *data)
{
+ struct user_namespace *userns = rpc_rqst_userns(req);
struct nfs3_diropres *result = data;
enum nfs_stat status;
int error;
error = decode_nfs_fh3(xdr, result->fh);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, result->fattr);
+ error = decode_post_op_attr(xdr, result->fattr, userns);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, result->dir_attr);
+ error = decode_post_op_attr(xdr, result->dir_attr, userns);
out:
return error;
out_default:
- error = decode_post_op_attr(xdr, result->dir_attr);
+ error = decode_post_op_attr(xdr, result->dir_attr, userns);
if (unlikely(error))
goto out;
return nfs3_stat_to_errno(status);
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, result->fattr);
+ error = decode_post_op_attr(xdr, result->fattr, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
if (status != NFS3_OK)
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, result);
+ error = decode_post_op_attr(xdr, result, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
if (status != NFS3_OK)
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, result->fattr);
+ error = decode_post_op_attr(xdr, result->fattr, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
result->op_status = status;
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_wcc_data(xdr, result->fattr);
+ error = decode_wcc_data(xdr, result->fattr, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
result->op_status = status;
* };
*/
static int decode_create3resok(struct xdr_stream *xdr,
- struct nfs3_diropres *result)
+ struct nfs3_diropres *result,
+ struct user_namespace *userns)
{
int error;
error = decode_post_op_fh3(xdr, result->fh);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, result->fattr);
+ error = decode_post_op_attr(xdr, result->fattr, userns);
if (unlikely(error))
goto out;
/* The server isn't required to return a file handle.
* values for the new object. */
if (result->fh->size == 0)
result->fattr->valid = 0;
- error = decode_wcc_data(xdr, result->dir_attr);
+ error = decode_wcc_data(xdr, result->dir_attr, userns);
out:
return error;
}
struct xdr_stream *xdr,
void *data)
{
+ struct user_namespace *userns = rpc_rqst_userns(req);
struct nfs3_diropres *result = data;
enum nfs_stat status;
int error;
goto out;
if (status != NFS3_OK)
goto out_default;
- error = decode_create3resok(xdr, result);
+ error = decode_create3resok(xdr, result, userns);
out:
return error;
out_default:
- error = decode_wcc_data(xdr, result->dir_attr);
+ error = decode_wcc_data(xdr, result->dir_attr, userns);
if (unlikely(error))
goto out;
return nfs3_stat_to_errno(status);
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_wcc_data(xdr, result->dir_attr);
+ error = decode_wcc_data(xdr, result->dir_attr, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
if (status != NFS3_OK)
struct xdr_stream *xdr,
void *data)
{
+ struct user_namespace *userns = rpc_rqst_userns(req);
struct nfs_renameres *result = data;
enum nfs_stat status;
int error;
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_wcc_data(xdr, result->old_fattr);
+ error = decode_wcc_data(xdr, result->old_fattr, userns);
if (unlikely(error))
goto out;
- error = decode_wcc_data(xdr, result->new_fattr);
+ error = decode_wcc_data(xdr, result->new_fattr, userns);
if (unlikely(error))
goto out;
if (status != NFS3_OK)
static int nfs3_xdr_dec_link3res(struct rpc_rqst *req, struct xdr_stream *xdr,
void *data)
{
+ struct user_namespace *userns = rpc_rqst_userns(req);
struct nfs3_linkres *result = data;
enum nfs_stat status;
int error;
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, result->fattr);
+ error = decode_post_op_attr(xdr, result->fattr, userns);
if (unlikely(error))
goto out;
- error = decode_wcc_data(xdr, result->dir_attr);
+ error = decode_wcc_data(xdr, result->dir_attr, userns);
if (unlikely(error))
goto out;
if (status != NFS3_OK)
int nfs3_decode_dirent(struct xdr_stream *xdr, struct nfs_entry *entry,
bool plus)
{
+ struct user_namespace *userns = rpc_userns(entry->server->client);
struct nfs_entry old = *entry;
__be32 *p;
int error;
if (plus) {
entry->fattr->valid = 0;
- error = decode_post_op_attr(xdr, entry->fattr);
+ error = decode_post_op_attr(xdr, entry->fattr, userns);
if (unlikely(error))
return error;
if (entry->fattr->valid & NFS_ATTR_FATTR_V3)
}
static int decode_readdir3resok(struct xdr_stream *xdr,
- struct nfs3_readdirres *result)
+ struct nfs3_readdirres *result,
+ struct user_namespace *userns)
{
int error;
- error = decode_post_op_attr(xdr, result->dir_attr);
+ error = decode_post_op_attr(xdr, result->dir_attr, userns);
if (unlikely(error))
goto out;
/* XXX: do we need to check if result->verf != NULL ? */
goto out;
if (status != NFS3_OK)
goto out_default;
- error = decode_readdir3resok(xdr, result);
+ error = decode_readdir3resok(xdr, result, rpc_rqst_userns(req));
out:
return error;
out_default:
- error = decode_post_op_attr(xdr, result->dir_attr);
+ error = decode_post_op_attr(xdr, result->dir_attr, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
return nfs3_stat_to_errno(status);
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, result->fattr);
+ error = decode_post_op_attr(xdr, result->fattr, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
if (status != NFS3_OK)
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, result->fattr);
+ error = decode_post_op_attr(xdr, result->fattr, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
if (status != NFS3_OK)
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_post_op_attr(xdr, result->fattr);
+ error = decode_post_op_attr(xdr, result->fattr, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
if (status != NFS3_OK)
error = decode_nfsstat3(xdr, &status);
if (unlikely(error))
goto out;
- error = decode_wcc_data(xdr, result->fattr);
+ error = decode_wcc_data(xdr, result->fattr, rpc_rqst_userns(req));
if (unlikely(error))
goto out;
result->op_status = status;
#ifdef CONFIG_NFS_V3_ACL
static inline int decode_getacl3resok(struct xdr_stream *xdr,
- struct nfs3_getaclres *result)
+ struct nfs3_getaclres *result,
+ struct user_namespace *userns)
{
struct posix_acl **acl;
unsigned int *aclcnt;
size_t hdrlen;
int error;
- error = decode_post_op_attr(xdr, result->fattr);
+ error = decode_post_op_attr(xdr, result->fattr, userns);
if (unlikely(error))
goto out;
error = decode_uint32(xdr, &result->mask);
goto out;
if (status != NFS3_OK)
goto out_default;
- error = decode_getacl3resok(xdr, result);
+ error = decode_getacl3resok(xdr, result, rpc_rqst_userns(req));
out:
return error;
out_default:
goto out;
if (status != NFS3_OK)
goto out_default;
- error = decode_post_op_attr(xdr, result);
+ error = decode_post_op_attr(xdr, result, rpc_rqst_userns(req));
out:
return error;
out_default:
unsigned char delay : 1,
recovering : 1,
retry : 1;
+ bool interruptible;
};
struct nfs4_state_recovery_ops {
.minorversion = minorversion,
.net = net,
.timeparms = timeparms,
+ .cred = server->cred,
};
struct nfs_client *clp;
.minorversion = minor_version,
.net = mds_clp->cl_net,
.timeparms = &ds_timeout,
+ .cred = mds_srv->cred,
};
char buf[INET6_ADDRSTRLEN + 1];
if (!server)
return ERR_PTR(-ENOMEM);
+ server->cred = get_cred(current_cred());
+
auth_probe = mount_info->parsed->auth_info.flavor_len < 1;
/* set up the general RPC client */
parent_server = NFS_SB(data->sb);
parent_client = parent_server->nfs_client;
+ server->cred = get_cred(parent_server->cred);
+
/* Initialise the client representation from the parent server */
nfs_server_copy_userdata(server, parent_server);
return filemap_fdatawrite(file->f_mapping);
/* Flush writes to the server and return any errors */
- return vfs_fsync(file, 0);
+ return nfs_wb_all(inode);
}
#ifdef CONFIG_NFS_V4_2
bool same_inode = false;
int ret;
- if (remap_flags & ~REMAP_FILE_ADVISORY)
+ if (remap_flags & ~(REMAP_FILE_DEDUP | REMAP_FILE_ADVISORY))
return -EINVAL;
/* check alignment w.r.t. clone_blksize */
struct rpc_pipe *idmap_pipe;
struct idmap_legacy_upcalldata *idmap_upcall_data;
struct mutex idmap_mutex;
+ const struct cred *cred;
};
+static struct user_namespace *idmap_userns(const struct idmap *idmap)
+{
+ if (idmap && idmap->cred)
+ return idmap->cred->user_ns;
+ return &init_user_ns;
+}
+
/**
* nfs_fattr_init_names - initialise the nfs_fattr owner_name/group_name fields
* @fattr: fully initialised struct nfs_fattr
const char *type, struct idmap *idmap)
{
char *desc;
- struct key *rkey;
+ struct key *rkey = ERR_PTR(-EAGAIN);
ssize_t ret;
ret = nfs_idmap_get_desc(name, namelen, type, strlen(type), &desc);
if (ret < 0)
return ERR_PTR(ret);
- rkey = request_key(&key_type_id_resolver, desc, "");
+ if (!idmap->cred || idmap->cred->user_ns == &init_user_ns)
+ rkey = request_key(&key_type_id_resolver, desc, "");
if (IS_ERR(rkey)) {
mutex_lock(&idmap->idmap_mutex);
rkey = request_key_with_auxdata(&key_type_id_resolver_legacy,
if (idmap == NULL)
return -ENOMEM;
+ mutex_init(&idmap->idmap_mutex);
+ idmap->cred = get_cred(clp->cl_rpcclient->cl_cred);
+
rpc_init_pipe_dir_object(&idmap->idmap_pdo,
&nfs_idmap_pipe_dir_object_ops,
idmap);
goto err;
}
idmap->idmap_pipe = pipe;
- mutex_init(&idmap->idmap_mutex);
error = rpc_add_pipe_dir_object(clp->cl_net,
&clp->cl_rpcclient->cl_pipedir_objects,
err_destroy_pipe:
rpc_destroy_pipe_data(idmap->idmap_pipe);
err:
+ put_cred(idmap->cred);
kfree(idmap);
return error;
}
&clp->cl_rpcclient->cl_pipedir_objects,
&idmap->idmap_pdo);
rpc_destroy_pipe_data(idmap->idmap_pipe);
+ put_cred(idmap->cred);
kfree(idmap);
}
if (!nfs_map_string_to_numeric(name, namelen, &id))
ret = nfs_idmap_lookup_id(name, namelen, "uid", &id, idmap);
if (ret == 0) {
- *uid = make_kuid(&init_user_ns, id);
+ *uid = make_kuid(idmap_userns(idmap), id);
if (!uid_valid(*uid))
ret = -ERANGE;
}
if (!nfs_map_string_to_numeric(name, namelen, &id))
ret = nfs_idmap_lookup_id(name, namelen, "gid", &id, idmap);
if (ret == 0) {
- *gid = make_kgid(&init_user_ns, id);
+ *gid = make_kgid(idmap_userns(idmap), id);
if (!gid_valid(*gid))
ret = -ERANGE;
}
int ret = -EINVAL;
__u32 id;
- id = from_kuid(&init_user_ns, uid);
+ id = from_kuid_munged(idmap_userns(idmap), uid);
if (!(server->caps & NFS_CAP_UIDGID_NOMAP))
ret = nfs_idmap_lookup_name(id, "user", buf, buflen, idmap);
if (ret < 0)
int ret = -EINVAL;
__u32 id;
- id = from_kgid(&init_user_ns, gid);
+ id = from_kgid_munged(idmap_userns(idmap), gid);
if (!(server->caps & NFS_CAP_UIDGID_NOMAP))
ret = nfs_idmap_lookup_name(id, "group", buf, buflen, idmap);
if (ret < 0)
return ret;
}
-static int nfs4_delay(struct rpc_clnt *clnt, long *timeout)
+static int nfs4_delay_killable(long *timeout)
{
- int res = 0;
-
might_sleep();
freezable_schedule_timeout_killable_unsafe(
nfs4_update_delay(timeout));
- if (fatal_signal_pending(current))
- res = -ERESTARTSYS;
- return res;
+ if (!__fatal_signal_pending(current))
+ return 0;
+ return -EINTR;
+}
+
+static int nfs4_delay_interruptible(long *timeout)
+{
+ might_sleep();
+
+ freezable_schedule_timeout_interruptible(nfs4_update_delay(timeout));
+ if (!signal_pending(current))
+ return 0;
+ return __fatal_signal_pending(current) ? -EINTR :-ERESTARTSYS;
+}
+
+static int nfs4_delay(long *timeout, bool interruptible)
+{
+ if (interruptible)
+ return nfs4_delay_interruptible(timeout);
+ return nfs4_delay_killable(timeout);
}
/* This is the error handling routine for processes that are allowed
ret = nfs4_do_handle_exception(server, errorcode, exception);
if (exception->delay) {
- ret = nfs4_delay(server->client, &exception->timeout);
+ ret = nfs4_delay(&exception->timeout,
+ exception->interruptible);
goto out_retry;
}
if (exception->recovering) {
if (res->sr_slot != NULL)
goto out_start;
- if (session) {
+ if (session)
tbl = &session->fc_slot_table;
- task->tk_timeout = 0;
- }
spin_lock(&tbl->slot_tbl_lock);
/* The state manager will wait until the slot table is empty */
slot = nfs4_alloc_slot(tbl);
if (IS_ERR(slot)) {
- /* Try again in 1/4 second */
if (slot == ERR_PTR(-ENOMEM))
- task->tk_timeout = HZ >> 2;
+ goto out_sleep_timeout;
goto out_sleep;
}
spin_unlock(&tbl->slot_tbl_lock);
nfs41_sequence_res_init(res);
rpc_call_start(task);
return 0;
-
+out_sleep_timeout:
+ /* Try again in 1/4 second */
+ if (args->sa_privileged)
+ rpc_sleep_on_priority_timeout(&tbl->slot_tbl_waitq, task,
+ jiffies + (HZ >> 2), RPC_PRIORITY_PRIVILEGED);
+ else
+ rpc_sleep_on_timeout(&tbl->slot_tbl_waitq, task,
+ NULL, jiffies + (HZ >> 2));
+ spin_unlock(&tbl->slot_tbl_lock);
+ return -EAGAIN;
out_sleep:
if (args->sa_privileged)
rpc_sleep_on_priority(&tbl->slot_tbl_waitq, task,
- NULL, RPC_PRIORITY_PRIVILEGED);
+ RPC_PRIORITY_PRIVILEGED);
else
rpc_sleep_on(&tbl->slot_tbl_waitq, task, NULL);
spin_unlock(&tbl->slot_tbl_lock);
int *opened)
{
struct nfs_server *server = NFS_SERVER(dir);
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
struct nfs4_state *res;
struct nfs4_open_createattrs c = {
.label = label,
int nfs4_server_capabilities(struct nfs_server *server, struct nfs_fh *fhandle)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = nfs4_handle_exception(server,
static int nfs4_lookup_root(struct nfs_server *server, struct nfs_fh *fhandle,
struct nfs_fsinfo *info)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = _nfs4_lookup_root(server, fhandle, info);
struct nfs_fattr *fattr, struct nfs4_label *label,
struct inode *inode)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = _nfs4_proc_getattr(server, fhandle, fattr, label, inode);
const struct qstr *name, struct nfs_fh *fhandle,
struct nfs_fattr *fattr, struct nfs4_label *label)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
struct rpc_clnt *client = *clnt;
int err;
do {
static int nfs4_proc_lookupp(struct inode *inode, struct nfs_fh *fhandle,
struct nfs_fattr *fattr, struct nfs4_label *label)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = _nfs4_proc_lookupp(inode, fhandle, fattr, label);
static int nfs4_proc_access(struct inode *inode, struct nfs_access_entry *entry)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = _nfs4_proc_access(inode, entry);
static int nfs4_proc_readlink(struct inode *inode, struct page *page,
unsigned int pgbase, unsigned int pglen)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = _nfs4_proc_readlink(inode, page, pgbase, pglen);
static int nfs4_proc_remove(struct inode *dir, struct dentry *dentry)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
struct inode *inode = d_inode(dentry);
int err;
static int nfs4_proc_rmdir(struct inode *dir, const struct qstr *name)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
static int nfs4_proc_link(struct inode *inode, struct inode *dir, const struct qstr *name)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = nfs4_handle_exception(NFS_SERVER(inode),
static int nfs4_proc_symlink(struct inode *dir, struct dentry *dentry,
struct page *page, unsigned int len, struct iattr *sattr)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
struct nfs4_label l, *label = NULL;
int err;
struct iattr *sattr)
{
struct nfs_server *server = NFS_SERVER(dir);
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
struct nfs4_label l, *label = NULL;
int err;
static int nfs4_proc_readdir(struct dentry *dentry, const struct cred *cred,
u64 cookie, struct page **pages, unsigned int count, bool plus)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = _nfs4_proc_readdir(dentry, cred, cookie,
struct iattr *sattr, dev_t rdev)
{
struct nfs_server *server = NFS_SERVER(dir);
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
struct nfs4_label l, *label = NULL;
int err;
static int nfs4_proc_statfs(struct nfs_server *server, struct nfs_fh *fhandle, struct nfs_fsstat *fsstat)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = nfs4_handle_exception(server,
static int nfs4_do_fsinfo(struct nfs_server *server, struct nfs_fh *fhandle, struct nfs_fsinfo *fsinfo)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
unsigned long now = jiffies;
int err;
static int nfs4_proc_pathconf(struct nfs_server *server, struct nfs_fh *fhandle,
struct nfs_pathconf *pathconf)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
static ssize_t nfs4_get_acl_uncached(struct inode *inode, void *buf, size_t buflen)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
ssize_t ret;
do {
ret = __nfs4_get_acl_uncached(inode, buf, buflen);
static int nfs4_get_security_label(struct inode *inode, void *buf,
size_t buflen)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
if (!nfs_server_capable(inode, NFS_CAP_SECURITY_LABEL))
static int nfs4_proc_getlk(struct nfs4_state *state, int cmd, struct file_lock *request)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
struct nfs4_exception exception = {
.state = state,
.inode = state->inode,
+ .interruptible = true,
};
int err;
struct nfs4_fs_locations *fs_locations,
struct page *page)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = _nfs4_proc_fs_locations(client, dir, name,
struct nfs_client *clp = server->nfs_client;
const struct nfs4_mig_recovery_ops *ops =
clp->cl_mvops->mig_recovery_ops;
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int status;
dprintk("%s: FSID %llx:%llx on \"%s\"\n", __func__,
struct nfs_client *clp = server->nfs_client;
const struct nfs4_mig_recovery_ops *ops =
clp->cl_mvops->mig_recovery_ops;
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int status;
dprintk("%s: FSID %llx:%llx on \"%s\"\n", __func__,
int nfs4_proc_secinfo(struct inode *dir, const struct qstr *name,
struct nfs4_secinfo_flavors *flavors)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = -NFS4ERR_WRONGSEC;
nfs41_proc_secinfo_no_name(struct nfs_server *server, struct nfs_fh *fhandle,
struct nfs_fsinfo *info, struct nfs4_secinfo_flavors *flavors)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
/* first try using integrity protection */
nfs4_stateid *stateid,
const struct cred *cred)
{
- struct nfs4_exception exception = { };
+ struct nfs4_exception exception = {
+ .interruptible = true,
+ };
int err;
do {
err = _nfs41_test_stateid(server, stateid, cred);
/* Sustain the lease, even if it's empty. If the clientid4
* goes stale it's of no use for trunking discovery. */
nfs4_schedule_state_renewal(*result);
+
+ /* If the client state need to recover, do it. */
+ if (clp->cl_state)
+ nfs4_schedule_state_manager(clp);
}
out:
return status;
{
/* FIXME: For now, we destroy all layouts. */
pnfs_destroy_all_layouts(clp);
- /* FIXME: For now, we test all delegations+open state+locks. */
- nfs41_handle_some_state_revoked(clp);
+ nfs_test_expired_all_delegations(clp);
dprintk("%s: Recallable state revoked on server %s!\n", __func__,
clp->cl_hostname);
}
#include <linux/nfs.h>
#include <linux/nfs3.h>
#include <linux/nfs4.h>
-#include <linux/nfs_page.h>
#include <linux/nfs_fs.h>
+#include <linux/nfs_page.h>
#include <linux/nfs_mount.h>
#include <linux/export.h>
hdr->req = nfs_list_entry(mirror->pg_list.next);
hdr->inode = desc->pg_inode;
- hdr->cred = hdr->req->wb_context->cred;
+ hdr->cred = nfs_req_openctx(hdr->req)->cred;
hdr->io_start = req_offset(hdr->req);
hdr->good_bytes = mirror->pg_count;
hdr->io_completion = desc->pg_io_completion;
nfs_release_request(head);
}
-/**
- * nfs_create_request - Create an NFS read/write request.
- * @ctx: open context to use
- * @page: page to write
- * @last: last nfs request created for this page group or NULL if head
- * @offset: starting offset within the page for the write
- * @count: number of bytes to read/write
- *
- * The page must be locked by the caller. This makes sure we never
- * create two different requests for the same page.
- * User should ensure it is safe to sleep in this function.
- */
-struct nfs_page *
-nfs_create_request(struct nfs_open_context *ctx, struct page *page,
- struct nfs_page *last, unsigned int offset,
+static struct nfs_page *
+__nfs_create_request(struct nfs_lock_context *l_ctx, struct page *page,
+ unsigned int pgbase, unsigned int offset,
unsigned int count)
{
struct nfs_page *req;
- struct nfs_lock_context *l_ctx;
+ struct nfs_open_context *ctx = l_ctx->open_context;
if (test_bit(NFS_CONTEXT_BAD, &ctx->flags))
return ERR_PTR(-EBADF);
if (req == NULL)
return ERR_PTR(-ENOMEM);
- /* get lock context early so we can deal with alloc failures */
- l_ctx = nfs_get_lock_context(ctx);
- if (IS_ERR(l_ctx)) {
- nfs_page_free(req);
- return ERR_CAST(l_ctx);
- }
req->wb_lock_context = l_ctx;
+ refcount_inc(&l_ctx->count);
atomic_inc(&l_ctx->io_count);
/* Initialize the request struct. Initially, we assume a
get_page(page);
}
req->wb_offset = offset;
- req->wb_pgbase = offset;
+ req->wb_pgbase = pgbase;
req->wb_bytes = count;
- req->wb_context = get_nfs_open_context(ctx);
kref_init(&req->wb_kref);
- nfs_page_group_init(req, last);
+ req->wb_nio = 0;
return req;
}
+/**
+ * nfs_create_request - Create an NFS read/write request.
+ * @ctx: open context to use
+ * @page: page to write
+ * @offset: starting offset within the page for the write
+ * @count: number of bytes to read/write
+ *
+ * The page must be locked by the caller. This makes sure we never
+ * create two different requests for the same page.
+ * User should ensure it is safe to sleep in this function.
+ */
+struct nfs_page *
+nfs_create_request(struct nfs_open_context *ctx, struct page *page,
+ unsigned int offset, unsigned int count)
+{
+ struct nfs_lock_context *l_ctx = nfs_get_lock_context(ctx);
+ struct nfs_page *ret;
+
+ if (IS_ERR(l_ctx))
+ return ERR_CAST(l_ctx);
+ ret = __nfs_create_request(l_ctx, page, offset, offset, count);
+ if (!IS_ERR(ret))
+ nfs_page_group_init(ret, NULL);
+ nfs_put_lock_context(l_ctx);
+ return ret;
+}
+
+static struct nfs_page *
+nfs_create_subreq(struct nfs_page *req, struct nfs_page *last,
+ unsigned int pgbase, unsigned int offset,
+ unsigned int count)
+{
+ struct nfs_page *ret;
+
+ ret = __nfs_create_request(req->wb_lock_context, req->wb_page,
+ pgbase, offset, count);
+ if (!IS_ERR(ret)) {
+ nfs_lock_request(ret);
+ ret->wb_index = req->wb_index;
+ nfs_page_group_init(ret, last);
+ ret->wb_nio = req->wb_nio;
+ }
+ return ret;
+}
+
/**
* nfs_unlock_request - Unlock request and wake up sleepers.
* @req: pointer to request
static void nfs_clear_request(struct nfs_page *req)
{
struct page *page = req->wb_page;
- struct nfs_open_context *ctx = req->wb_context;
struct nfs_lock_context *l_ctx = req->wb_lock_context;
+ struct nfs_open_context *ctx;
if (page != NULL) {
put_page(page);
if (l_ctx != NULL) {
if (atomic_dec_and_test(&l_ctx->io_count)) {
wake_up_var(&l_ctx->io_count);
+ ctx = l_ctx->open_context;
if (test_bit(NFS_CONTEXT_UNLOCK, &ctx->flags))
rpc_wake_up(&NFS_SERVER(d_inode(ctx->dentry))->uoc_rpcwaitq);
}
nfs_put_lock_context(l_ctx);
req->wb_lock_context = NULL;
}
- if (ctx != NULL) {
- put_nfs_open_context(ctx);
- req->wb_context = NULL;
- }
}
/**
hdr->args.pgbase = req->wb_pgbase;
hdr->args.pages = hdr->page_array.pagevec;
hdr->args.count = count;
- hdr->args.context = get_nfs_open_context(req->wb_context);
+ hdr->args.context = get_nfs_open_context(nfs_req_openctx(req));
hdr->args.lock_context = req->wb_lock_context;
hdr->args.stable = NFS_UNSTABLE;
switch (how & (FLUSH_STABLE | FLUSH_COND_STABLE)) {
desc->pg_mirrors_dynamic = NULL;
desc->pg_mirrors = desc->pg_mirrors_static;
nfs_pageio_mirror_init(&desc->pg_mirrors[0], bsize);
+ desc->pg_maxretrans = 0;
}
/**
struct file_lock_context *flctx;
if (prev) {
- if (!nfs_match_open_context(req->wb_context, prev->wb_context))
+ if (!nfs_match_open_context(nfs_req_openctx(req), nfs_req_openctx(prev)))
return false;
- flctx = d_inode(req->wb_context->dentry)->i_flctx;
+ flctx = d_inode(nfs_req_openctx(req)->dentry)->i_flctx;
if (flctx != NULL &&
!(list_empty_careful(&flctx->flc_posix) &&
list_empty_careful(&flctx->flc_flock)) &&
return 0;
mirror->pg_base = req->wb_pgbase;
}
+
+ if (desc->pg_maxretrans && req->wb_nio > desc->pg_maxretrans) {
+ if (NFS_SERVER(desc->pg_inode)->flags & NFS_MOUNT_SOFTERR)
+ desc->pg_error = -ETIMEDOUT;
+ else
+ desc->pg_error = -EIO;
+ return 0;
+ }
+
if (!nfs_can_coalesce_requests(prev, req, desc))
return 0;
nfs_list_move_request(req, &mirror->pg_list);
pgbase += subreq->wb_bytes;
if (bytes_left) {
- subreq = nfs_create_request(req->wb_context,
- req->wb_page,
- subreq, pgbase, bytes_left);
+ subreq = nfs_create_subreq(req, subreq, pgbase,
+ offset, bytes_left);
if (IS_ERR(subreq))
goto err_ptr;
- nfs_lock_request(subreq);
- subreq->wb_offset = offset;
- subreq->wb_index = req->wb_index;
}
} while (bytes_left > 0);
lastreq = lastreq->wb_this_page)
;
- dupreq = nfs_create_request(req->wb_context,
- req->wb_page, lastreq, pgbase, bytes);
+ dupreq = nfs_create_subreq(req, lastreq,
+ pgbase, offset, bytes);
+ nfs_page_group_unlock(req);
if (IS_ERR(dupreq)) {
- nfs_page_group_unlock(req);
desc->pg_error = PTR_ERR(dupreq);
goto out_failed;
}
-
- nfs_lock_request(dupreq);
- nfs_page_group_unlock(req);
- dupreq->wb_offset = offset;
- dupreq->wb_index = req->wb_index;
} else
dupreq = req;
rd_size = nfs_dreq_bytes_left(pgio->pg_dreq);
pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode,
- req->wb_context,
+ nfs_req_openctx(req),
req_offset(req),
rd_size,
IOMODE_READ,
pnfs_generic_pg_check_range(pgio, req);
if (pgio->pg_lseg == NULL) {
pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode,
- req->wb_context,
+ nfs_req_openctx(req),
req_offset(req),
wb_size,
IOMODE_RW,
pnfs_mark_request_commit(struct nfs_page *req, struct pnfs_layout_segment *lseg,
struct nfs_commit_info *cinfo, u32 ds_commit_idx)
{
- struct inode *inode = d_inode(req->wb_context->dentry);
+ struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
struct pnfs_layoutdriver_type *ld = NFS_SERVER(inode)->pnfs_curr_ld;
if (lseg == NULL || ld->mark_request_commit == NULL)
static inline bool
pnfs_clear_request_commit(struct nfs_page *req, struct nfs_commit_info *cinfo)
{
- struct inode *inode = d_inode(req->wb_context->dentry);
+ struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
struct pnfs_layoutdriver_type *ld = NFS_SERVER(inode)->pnfs_curr_ld;
if (ld == NULL || ld->clear_request_commit == NULL)
static void nfs_readpage_release(struct nfs_page *req)
{
- struct inode *inode = d_inode(req->wb_context->dentry);
+ struct inode *inode = d_inode(nfs_req_openctx(req)->dentry);
dprintk("NFS: read done (%s/%llu %d@%lld)\n", inode->i_sb->s_id,
(unsigned long long)NFS_FILEID(inode), req->wb_bytes,
len = nfs_page_length(page);
if (len == 0)
return nfs_return_empty_page(page);
- new = nfs_create_request(ctx, page, NULL, 0, len);
+ new = nfs_create_request(ctx, page, 0, len);
if (IS_ERR(new)) {
unlock_page(page);
return PTR_ERR(new);
if (len == 0)
return nfs_return_empty_page(page);
- new = nfs_create_request(desc->ctx, page, NULL, 0, len);
+ new = nfs_create_request(desc->ctx, page, 0, len);
if (IS_ERR(new))
goto out_error;
enum {
/* Mount options that take no arguments */
- Opt_soft, Opt_hard,
+ Opt_soft, Opt_softerr, Opt_hard,
Opt_posix, Opt_noposix,
Opt_cto, Opt_nocto,
Opt_ac, Opt_noac,
{ Opt_sloppy, "sloppy" },
{ Opt_soft, "soft" },
+ { Opt_softerr, "softerr" },
{ Opt_hard, "hard" },
{ Opt_deprecated, "intr" },
{ Opt_deprecated, "nointr" },
const char *str;
const char *nostr;
} nfs_info[] = {
- { NFS_MOUNT_SOFT, ",soft", ",hard" },
+ { NFS_MOUNT_SOFT, ",soft", "" },
+ { NFS_MOUNT_SOFTERR, ",softerr", "" },
{ NFS_MOUNT_POSIX, ",posix", "" },
{ NFS_MOUNT_NOCTO, ",nocto", "" },
{ NFS_MOUNT_NOAC, ",noac", "" },
seq_printf(m, ",acdirmin=%u", nfss->acdirmin/HZ);
if (nfss->acdirmax != NFS_DEF_ACDIRMAX*HZ || showdefaults)
seq_printf(m, ",acdirmax=%u", nfss->acdirmax/HZ);
+ if (!(nfss->flags & (NFS_MOUNT_SOFT|NFS_MOUNT_SOFTERR)))
+ seq_puts(m, ",hard");
for (nfs_infop = nfs_info; nfs_infop->flag; nfs_infop++) {
if (nfss->flags & nfs_infop->flag)
seq_puts(m, nfs_infop->str);
*/
case Opt_soft:
mnt->flags |= NFS_MOUNT_SOFT;
+ mnt->flags &= ~NFS_MOUNT_SOFTERR;
break;
- case Opt_hard:
+ case Opt_softerr:
+ mnt->flags |= NFS_MOUNT_SOFTERR;
mnt->flags &= ~NFS_MOUNT_SOFT;
break;
+ case Opt_hard:
+ mnt->flags &= ~(NFS_MOUNT_SOFT|NFS_MOUNT_SOFTERR);
+ break;
case Opt_posix:
mnt->flags |= NFS_MOUNT_POSIX;
break;
return 1;
}
+static int nfs_compare_userns(const struct nfs_server *old,
+ const struct nfs_server *new)
+{
+ const struct user_namespace *oldns = &init_user_ns;
+ const struct user_namespace *newns = &init_user_ns;
+
+ if (old->client && old->client->cl_cred)
+ oldns = old->client->cl_cred->user_ns;
+ if (new->client && new->client->cl_cred)
+ newns = new->client->cl_cred->user_ns;
+ if (oldns != newns)
+ return 0;
+ return 1;
+}
+
static int nfs_compare_super(struct super_block *sb, void *data)
{
struct nfs_sb_mountdata *sb_mntdata = data;
return 0;
if (memcmp(&old->fsid, &server->fsid, sizeof(old->fsid)) != 0)
return 0;
+ if (!nfs_compare_userns(old, server))
+ return 0;
return nfs_compare_mount_options(sb, server, mntflags);
}
* and straight-forward than readdir caching.
*/
-static int nfs_symlink_filler(struct inode *inode, struct page *page)
+static int nfs_symlink_filler(void *data, struct page *page)
{
+ struct inode *inode = data;
int error;
error = NFS_PROTO(inode)->readlink(inode, page, 0, PAGE_SIZE);
err = ERR_PTR(nfs_revalidate_mapping(inode, inode->i_mapping));
if (err)
return err;
- page = read_cache_page(&inode->i_data, 0,
- (filler_t *)nfs_symlink_filler, inode);
+ page = read_cache_page(&inode->i_data, 0, nfs_symlink_filler,
+ inode);
if (IS_ERR(page))
return ERR_CAST(page);
}
nfs_zap_mapping(mapping->host, mapping);
}
+static void nfs_mapping_set_error(struct page *page, int error)
+{
+ SetPageError(page);
+ mapping_set_error(page_file_mapping(page), error);
+}
+
/*
* nfs_page_group_search_locked
* @head - head request of page group
return ERR_PTR(ret);
}
-static void nfs_write_error_remove_page(struct nfs_page *req)
+static void nfs_write_error(struct nfs_page *req, int error)
{
+ nfs_mapping_set_error(req->wb_page, error);
nfs_end_page_writeback(req);
- generic_error_remove_page(page_file_mapping(req->wb_page),
- req->wb_page);
nfs_release_request(req);
}
static int nfs_page_async_flush(struct nfs_pageio_descriptor *pgio,
struct page *page)
{
+ struct address_space *mapping;
struct nfs_page *req;
int ret = 0;
nfs_set_page_writeback(page);
WARN_ON_ONCE(test_bit(PG_CLEAN, &req->wb_flags));
- ret = req->wb_context->error;
/* If there is a fatal error that covers this write, just exit */
- if (nfs_error_is_fatal_on_server(ret))
+ ret = 0;
+ mapping = page_file_mapping(page);
+ if (test_bit(AS_ENOSPC, &mapping->flags) ||
+ test_bit(AS_EIO, &mapping->flags))
goto out_launder;
- ret = 0;
if (!nfs_pageio_add_request(pgio, req)) {
ret = pgio->pg_error;
/*
* Remove the problematic req upon fatal errors on the server
*/
if (nfs_error_is_fatal(ret)) {
- nfs_context_set_write_error(req->wb_context, ret);
if (nfs_error_is_fatal_on_server(ret))
goto out_launder;
} else
out:
return ret;
out_launder:
- nfs_write_error_remove_page(req);
- return ret;
+ nfs_write_error(req, ret);
+ return 0;
}
static int nfs_do_writepage(struct page *page, struct writeback_control *wbc,
nfs_clear_request_commit(struct nfs_page *req)
{
if (test_bit(PG_CLEAN, &req->wb_flags)) {
- struct inode *inode = d_inode(req->wb_context->dentry);
+ struct nfs_open_context *ctx = nfs_req_openctx(req);
+ struct inode *inode = d_inode(ctx->dentry);
struct nfs_commit_info cinfo;
nfs_init_cinfo_from_inode(&cinfo, inode);
if (test_bit(NFS_IOHDR_ERROR, &hdr->flags) &&
(hdr->good_bytes < bytes)) {
nfs_set_pageerror(page_file_mapping(req->wb_page));
- nfs_context_set_write_error(req->wb_context, hdr->error);
+ nfs_mapping_set_error(req->wb_page, hdr->error);
goto remove_req;
}
if (nfs_write_need_commit(hdr)) {
+ /* Reset wb_nio, since the write was successful. */
+ req->wb_nio = 0;
memcpy(&req->wb_verf, &hdr->verf.verifier, sizeof(req->wb_verf));
nfs_mark_request_commit(req, hdr->lseg, &cinfo,
hdr->pgio_mirror_idx);
req->wb_bytes = end - req->wb_offset;
else
req->wb_bytes = rqend - req->wb_offset;
+ req->wb_nio = 0;
return req;
out_flushme:
/*
req = nfs_try_to_update_request(inode, page, offset, bytes);
if (req != NULL)
goto out;
- req = nfs_create_request(ctx, page, NULL, offset, bytes);
+ req = nfs_create_request(ctx, page, offset, bytes);
if (IS_ERR(req))
goto out;
nfs_inode_add_request(inode, req);
return 0;
l_ctx = req->wb_lock_context;
do_flush = req->wb_page != page ||
- !nfs_match_open_context(req->wb_context, ctx);
+ !nfs_match_open_context(nfs_req_openctx(req), ctx);
if (l_ctx && flctx &&
!(list_empty_careful(&flctx->flc_posix) &&
list_empty_careful(&flctx->flc_flock))) {
*/
static void nfs_redirty_request(struct nfs_page *req)
{
+ /* Bump the transmission count */
+ req->wb_nio++;
nfs_mark_request_dirty(req);
- set_bit(NFS_CONTEXT_RESEND_WRITES, &req->wb_context->flags);
+ set_bit(NFS_CONTEXT_RESEND_WRITES, &nfs_req_openctx(req)->flags);
nfs_end_page_writeback(req);
nfs_release_request(req);
}
while (!list_empty(head)) {
req = nfs_list_entry(head->next);
nfs_list_remove_request(req);
- if (nfs_error_is_fatal(error)) {
- nfs_context_set_write_error(req->wb_context, error);
- if (nfs_error_is_fatal_on_server(error)) {
- nfs_write_error_remove_page(req);
- continue;
- }
- }
- nfs_redirty_request(req);
+ if (nfs_error_is_fatal(error))
+ nfs_write_error(req, error);
+ else
+ nfs_redirty_request(req);
}
}
struct nfs_commit_info *cinfo)
{
struct nfs_page *first = nfs_list_entry(head->next);
- struct inode *inode = d_inode(first->wb_context->dentry);
+ struct nfs_open_context *ctx = nfs_req_openctx(first);
+ struct inode *inode = d_inode(ctx->dentry);
/* Set up the RPC argument and reply structs
* NB: take care not to mess about with data->commit et al. */
list_splice_init(head, &data->pages);
data->inode = inode;
- data->cred = first->wb_context->cred;
+ data->cred = ctx->cred;
data->lseg = lseg; /* reference transferred */
/* only set lwb for pnfs commit */
if (lseg)
/* Note: we always request a commit of the entire inode */
data->args.offset = 0;
data->args.count = 0;
- data->context = get_nfs_open_context(first->wb_context);
+ data->context = get_nfs_open_context(ctx);
data->res.fattr = &data->fattr;
data->res.verf = &data->verf;
nfs_fattr_init(&data->fattr);
nfs_clear_page_commit(req->wb_page);
dprintk("NFS: commit (%s/%llu %d@%lld)",
- req->wb_context->dentry->d_sb->s_id,
- (unsigned long long)NFS_FILEID(d_inode(req->wb_context->dentry)),
+ nfs_req_openctx(req)->dentry->d_sb->s_id,
+ (unsigned long long)NFS_FILEID(d_inode(nfs_req_openctx(req)->dentry)),
req->wb_bytes,
(long long)req_offset(req));
if (status < 0) {
- nfs_context_set_write_error(req->wb_context, status);
- if (req->wb_page)
+ if (req->wb_page) {
+ nfs_mapping_set_error(req->wb_page, status);
nfs_inode_remove_request(req);
+ }
dprintk_cont(", error = %d\n", status);
goto next;
}
/* We have a mismatch. Write the page again */
dprintk_cont(" mismatch\n");
nfs_mark_request_dirty(req);
- set_bit(NFS_CONTEXT_RESEND_WRITES, &req->wb_context->flags);
+ set_bit(NFS_CONTEXT_RESEND_WRITES, &nfs_req_openctx(req)->flags);
next:
nfs_unlock_and_release_request(req);
/* Latency breaker */
.program = &cb_program,
.version = 1,
.flags = (RPC_CLNT_CREATE_NOPING | RPC_CLNT_CREATE_QUIET),
+ .cred = current_cred(),
};
struct rpc_clnt *client;
const struct cred *cred;
* the submission code will error out, so we don't need to
* handle that case here.
*/
- if (task->tk_flags & RPC_TASK_KILLED)
+ if (RPC_SIGNALLED(task))
goto need_restart;
return true;
dprintk("%s: freed slot, new seqid=%d\n", __func__,
clp->cl_cb_session->se_cb_seq_nr);
- if (task->tk_flags & RPC_TASK_KILLED)
+ if (RPC_SIGNALLED(task))
goto need_restart;
out:
return ret;
rc = __orangefs_set_acl(inode, acl, type);
} else {
iattr.ia_valid = ATTR_MODE;
- rc = orangefs_inode_setattr(inode, &iattr);
+ rc = __orangefs_setattr(inode, &iattr);
}
return rc;
inode->i_mode = mode;
iattr.ia_mode = mode;
iattr.ia_valid |= ATTR_MODE;
- orangefs_inode_setattr(inode, &iattr);
+ __orangefs_setattr(inode, &iattr);
}
return error;
// SPDX-License-Identifier: GPL-2.0
/*
* (C) 2001 Clemson University and The University of Chicago
+ * Copyright 2018 Omnibond Systems, L.L.C.
*
* See COPYING in top-level directory.
*/
/*
* Post and wait for the I/O upcall to finish
*/
-static ssize_t wait_for_direct_io(enum ORANGEFS_io_type type, struct inode *inode,
- loff_t *offset, struct iov_iter *iter,
- size_t total_size, loff_t readahead_size)
+ssize_t wait_for_direct_io(enum ORANGEFS_io_type type, struct inode *inode,
+ loff_t *offset, struct iov_iter *iter, size_t total_size,
+ loff_t readahead_size, struct orangefs_write_range *wr, int *index_return)
{
struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
struct orangefs_khandle *handle = &orangefs_inode->refn.khandle;
struct orangefs_kernel_op_s *new_op = NULL;
int buffer_index = -1;
ssize_t ret;
+ size_t copy_amount;
new_op = op_alloc(ORANGEFS_VFS_OP_FILE_IO);
if (!new_op)
new_op->upcall.req.io.buf_index = buffer_index;
new_op->upcall.req.io.count = total_size;
new_op->upcall.req.io.offset = *offset;
+ if (type == ORANGEFS_IO_WRITE && wr) {
+ new_op->upcall.uid = from_kuid(&init_user_ns, wr->uid);
+ new_op->upcall.gid = from_kgid(&init_user_ns, wr->gid);
+ }
gossip_debug(GOSSIP_FILE_DEBUG,
"%s(%pU): offset: %llu total_size: %zd\n",
* trigger the write.
*/
case OP_VFS_STATE_INPROGR:
- ret = total_size;
+ if (type == ORANGEFS_IO_READ)
+ ret = -EINTR;
+ else
+ ret = total_size;
break;
default:
gossip_err("%s: unexpected op state :%d:.\n",
* can futher be kernel-space or user-space addresses.
* or it can pointers to struct page's
*/
+
+ /*
+ * When reading, readahead_size will only be zero when
+ * we're doing O_DIRECT, otherwise we got here from
+ * orangefs_readpage.
+ *
+ * If we got here from orangefs_readpage we want to
+ * copy either a page or the whole file into the io
+ * vector, whichever is smaller.
+ */
+ if (readahead_size)
+ copy_amount =
+ min(new_op->downcall.resp.io.amt_complete,
+ (__s64)PAGE_SIZE);
+ else
+ copy_amount = new_op->downcall.resp.io.amt_complete;
+
ret = orangefs_bufmap_copy_to_iovec(iter, buffer_index,
- new_op->downcall.resp.io.amt_complete);
+ copy_amount);
if (ret < 0) {
gossip_err("%s: Failed to copy-out buffers. Please make sure that the pvfs2-client is running (%ld)\n",
__func__, (long)ret);
out:
if (buffer_index >= 0) {
- orangefs_bufmap_put(buffer_index);
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s(%pU): PUT buffer_index %d\n",
- __func__, handle, buffer_index);
+ if ((readahead_size) && (type == ORANGEFS_IO_READ)) {
+ /* readpage */
+ *index_return = buffer_index;
+ gossip_debug(GOSSIP_FILE_DEBUG,
+ "%s: hold on to buffer_index :%d:\n",
+ __func__, buffer_index);
+ } else {
+ /* O_DIRECT */
+ orangefs_bufmap_put(buffer_index);
+ gossip_debug(GOSSIP_FILE_DEBUG,
+ "%s(%pU): PUT buffer_index %d\n",
+ __func__, handle, buffer_index);
+ }
buffer_index = -1;
}
op_release(new_op);
return ret;
}
-/*
- * Common entry point for read/write/readv/writev
- * This function will dispatch it to either the direct I/O
- * or buffered I/O path depending on the mount options and/or
- * augmented/extended metadata attached to the file.
- * Note: File extended attributes override any mount options.
- */
-static ssize_t do_readv_writev(enum ORANGEFS_io_type type, struct file *file,
- loff_t *offset, struct iov_iter *iter)
+int orangefs_revalidate_mapping(struct inode *inode)
{
- struct inode *inode = file->f_mapping->host;
struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
- struct orangefs_khandle *handle = &orangefs_inode->refn.khandle;
- size_t count = iov_iter_count(iter);
- ssize_t total_count = 0;
- ssize_t ret = -EINVAL;
-
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s-BEGIN(%pU): count(%d) after estimate_max_iovecs.\n",
- __func__,
- handle,
- (int)count);
-
- if (type == ORANGEFS_IO_WRITE) {
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s(%pU): proceeding with offset : %llu, "
- "size %d\n",
- __func__,
- handle,
- llu(*offset),
- (int)count);
- }
+ struct address_space *mapping = inode->i_mapping;
+ unsigned long *bitlock = &orangefs_inode->bitlock;
+ int ret;
- if (count == 0) {
- ret = 0;
- goto out;
+ while (1) {
+ ret = wait_on_bit(bitlock, 1, TASK_KILLABLE);
+ if (ret)
+ return ret;
+ spin_lock(&inode->i_lock);
+ if (test_bit(1, bitlock)) {
+ spin_unlock(&inode->i_lock);
+ continue;
+ }
+ if (!time_before(jiffies, orangefs_inode->mapping_time))
+ break;
+ spin_unlock(&inode->i_lock);
+ return 0;
}
- while (iov_iter_count(iter)) {
- size_t each_count = iov_iter_count(iter);
- size_t amt_complete;
-
- /* how much to transfer in this loop iteration */
- if (each_count > orangefs_bufmap_size_query())
- each_count = orangefs_bufmap_size_query();
+ set_bit(1, bitlock);
+ smp_wmb();
+ spin_unlock(&inode->i_lock);
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s(%pU): size of each_count(%d)\n",
- __func__,
- handle,
- (int)each_count);
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s(%pU): BEFORE wait_for_io: offset is %d\n",
- __func__,
- handle,
- (int)*offset);
-
- ret = wait_for_direct_io(type, inode, offset, iter,
- each_count, 0);
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s(%pU): return from wait_for_io:%d\n",
- __func__,
- handle,
- (int)ret);
+ unmap_mapping_range(mapping, 0, 0, 0);
+ ret = filemap_write_and_wait(mapping);
+ if (!ret)
+ ret = invalidate_inode_pages2(mapping);
- if (ret < 0)
- goto out;
-
- *offset += ret;
- total_count += ret;
- amt_complete = ret;
+ orangefs_inode->mapping_time = jiffies +
+ orangefs_cache_timeout_msecs*HZ/1000;
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s(%pU): AFTER wait_for_io: offset is %d\n",
- __func__,
- handle,
- (int)*offset);
-
- /*
- * if we got a short I/O operations,
- * fall out and return what we got so far
- */
- if (amt_complete < each_count)
- break;
- } /*end while */
-
-out:
- if (total_count > 0)
- ret = total_count;
- if (ret > 0) {
- if (type == ORANGEFS_IO_READ) {
- file_accessed(file);
- } else {
- file_update_time(file);
- /*
- * Must invalidate to ensure write loop doesn't
- * prevent kernel from reading updated
- * attribute. Size probably changed because of
- * the write, and other clients could update
- * any other attribute.
- */
- orangefs_inode->getattr_time = jiffies - 1;
- }
- }
-
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s(%pU): Value(%d) returned.\n",
- __func__,
- handle,
- (int)ret);
+ clear_bit(1, bitlock);
+ smp_mb__after_atomic();
+ wake_up_bit(bitlock, 1);
return ret;
}
-/*
- * Read data from a specified offset in a file (referenced by inode).
- * Data may be placed either in a user or kernel buffer.
- */
-ssize_t orangefs_inode_read(struct inode *inode,
- struct iov_iter *iter,
- loff_t *offset,
- loff_t readahead_size)
+static ssize_t orangefs_file_read_iter(struct kiocb *iocb,
+ struct iov_iter *iter)
{
- struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
- size_t count = iov_iter_count(iter);
- size_t bufmap_size;
- ssize_t ret = -EINVAL;
+ int ret;
+ struct orangefs_read_options *ro;
orangefs_stats.reads++;
- bufmap_size = orangefs_bufmap_size_query();
- if (count > bufmap_size) {
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s: count is too large (%zd/%zd)!\n",
- __func__, count, bufmap_size);
- return -EINVAL;
+ /*
+ * Remember how they set "count" in read(2) or pread(2) or whatever -
+ * users can use count as a knob to control orangefs io size and later
+ * we can try to help them fill as many pages as possible in readpage.
+ */
+ if (!iocb->ki_filp->private_data) {
+ iocb->ki_filp->private_data = kmalloc(sizeof *ro, GFP_KERNEL);
+ if (!iocb->ki_filp->private_data)
+ return(ENOMEM);
+ ro = iocb->ki_filp->private_data;
+ ro->blksiz = iter->count;
}
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s(%pU) %zd@%llu\n",
- __func__,
- &orangefs_inode->refn.khandle,
- count,
- llu(*offset));
-
- ret = wait_for_direct_io(ORANGEFS_IO_READ, inode, offset, iter,
- count, readahead_size);
- if (ret > 0)
- *offset += ret;
-
- gossip_debug(GOSSIP_FILE_DEBUG,
- "%s(%pU): Value(%zd) returned.\n",
- __func__,
- &orangefs_inode->refn.khandle,
- ret);
+ down_read(&file_inode(iocb->ki_filp)->i_rwsem);
+ ret = orangefs_revalidate_mapping(file_inode(iocb->ki_filp));
+ if (ret)
+ goto out;
+ ret = generic_file_read_iter(iocb, iter);
+out:
+ up_read(&file_inode(iocb->ki_filp)->i_rwsem);
return ret;
}
-static ssize_t orangefs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
+static ssize_t orangefs_file_write_iter(struct kiocb *iocb,
+ struct iov_iter *iter)
{
- struct file *file = iocb->ki_filp;
- loff_t pos = iocb->ki_pos;
- ssize_t rc = 0;
-
- gossip_debug(GOSSIP_FILE_DEBUG, "orangefs_file_read_iter\n");
-
- orangefs_stats.reads++;
-
- rc = do_readv_writev(ORANGEFS_IO_READ, file, &pos, iter);
- iocb->ki_pos = pos;
-
- return rc;
-}
-
-static ssize_t orangefs_file_write_iter(struct kiocb *iocb, struct iov_iter *iter)
-{
- struct file *file = iocb->ki_filp;
- loff_t pos;
- ssize_t rc;
-
- gossip_debug(GOSSIP_FILE_DEBUG, "orangefs_file_write_iter\n");
-
- inode_lock(file->f_mapping->host);
-
- /* Make sure generic_write_checks sees an up to date inode size. */
- if (file->f_flags & O_APPEND) {
- rc = orangefs_inode_getattr(file->f_mapping->host, 0, 1,
- STATX_SIZE);
- if (rc == -ESTALE)
- rc = -EIO;
- if (rc) {
- gossip_err("%s: orangefs_inode_getattr failed, "
- "rc:%zd:.\n", __func__, rc);
- goto out;
- }
- }
-
- rc = generic_write_checks(iocb, iter);
-
- if (rc <= 0) {
- gossip_err("%s: generic_write_checks failed, rc:%zd:.\n",
- __func__, rc);
- goto out;
- }
-
- /*
- * if we are appending, generic_write_checks would have updated
- * pos to the end of the file, so we will wait till now to set
- * pos...
- */
- pos = iocb->ki_pos;
-
- rc = do_readv_writev(ORANGEFS_IO_WRITE,
- file,
- &pos,
- iter);
- if (rc < 0) {
- gossip_err("%s: do_readv_writev failed, rc:%zd:.\n",
- __func__, rc);
- goto out;
- }
-
- iocb->ki_pos = pos;
+ int ret;
orangefs_stats.writes++;
-out:
+ if (iocb->ki_pos > i_size_read(file_inode(iocb->ki_filp))) {
+ ret = orangefs_revalidate_mapping(file_inode(iocb->ki_filp));
+ if (ret)
+ return ret;
+ }
- inode_unlock(file->f_mapping->host);
- return rc;
+ ret = generic_file_write_iter(iocb, iter);
+ return ret;
}
/*
{
struct file *file = vmf->vma->vm_file;
int ret;
-
- ret = orangefs_inode_getattr(file->f_mapping->host, 0, 1,
- STATX_SIZE);
+ ret = orangefs_inode_getattr(file->f_mapping->host,
+ ORANGEFS_GETATTR_SIZE);
if (ret == -ESTALE)
ret = -EIO;
if (ret) {
- gossip_err("%s: orangefs_inode_getattr failed, ret:%d:.\n",
- __func__, ret);
+ gossip_err("%s: orangefs_inode_getattr failed, "
+ "ret:%d:.\n", __func__, ret);
return VM_FAULT_SIGBUS;
}
return filemap_fault(vmf);
static const struct vm_operations_struct orangefs_file_vm_ops = {
.fault = orangefs_fault,
.map_pages = filemap_map_pages,
- .page_mkwrite = filemap_page_mkwrite,
+ .page_mkwrite = orangefs_page_mkwrite,
};
/*
*/
static int orangefs_file_mmap(struct file *file, struct vm_area_struct *vma)
{
+ int ret;
+
+ ret = orangefs_revalidate_mapping(file_inode(file));
+ if (ret)
+ return ret;
+
gossip_debug(GOSSIP_FILE_DEBUG,
"orangefs_file_mmap: called on %s\n",
(file ?
(char *)file->f_path.dentry->d_name.name :
(char *)"Unknown"));
- if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_MAYWRITE))
- return -EINVAL;
-
/* set the sequential readahead hint */
vma->vm_flags |= VM_SEQ_READ;
vma->vm_flags &= ~VM_RAND_READ;
gossip_debug(GOSSIP_INODE_DEBUG,
"flush_racache finished\n");
}
- truncate_inode_pages(file_inode(file)->i_mapping,
- 0);
+
}
return 0;
}
ORANGEFS_I(file_inode(file));
struct orangefs_kernel_op_s *new_op = NULL;
+ ret = filemap_write_and_wait_range(file_inode(file)->i_mapping,
+ start, end);
+ if (ret < 0)
+ return ret;
+
new_op = op_alloc(ORANGEFS_VFS_OP_FSYNC);
if (!new_op)
return -ENOMEM;
* NOTE: We are only interested in file size here,
* so we set mask accordingly.
*/
- ret = orangefs_inode_getattr(file->f_mapping->host, 0, 1,
- STATX_SIZE);
+ ret = orangefs_inode_getattr(file->f_mapping->host,
+ ORANGEFS_GETATTR_SIZE);
if (ret == -ESTALE)
ret = -EIO;
if (ret) {
return rc;
}
+static int orangefs_file_open(struct inode * inode, struct file *file)
+{
+ file->private_data = NULL;
+ return generic_file_open(inode, file);
+}
+
+static int orangefs_flush(struct file *file, fl_owner_t id)
+{
+ /*
+ * This is vfs_fsync_range(file, 0, LLONG_MAX, 0) without the
+ * service_operation in orangefs_fsync.
+ *
+ * Do not send fsync to OrangeFS server on a close. Do send fsync
+ * on an explicit fsync call. This duplicates historical OrangeFS
+ * behavior.
+ */
+ struct inode *inode = file->f_mapping->host;
+ int r;
+
+ kfree(file->private_data);
+ file->private_data = NULL;
+
+ if (inode->i_state & I_DIRTY_TIME) {
+ spin_lock(&inode->i_lock);
+ inode->i_state &= ~I_DIRTY_TIME;
+ spin_unlock(&inode->i_lock);
+ mark_inode_dirty_sync(inode);
+ }
+
+ r = filemap_write_and_wait_range(file->f_mapping, 0, LLONG_MAX);
+ if (r > 0)
+ return 0;
+ else
+ return r;
+}
+
/** ORANGEFS implementation of VFS file operations */
const struct file_operations orangefs_file_operations = {
.llseek = orangefs_file_llseek,
.lock = orangefs_lock,
.unlocked_ioctl = orangefs_ioctl,
.mmap = orangefs_file_mmap,
- .open = generic_file_open,
+ .open = orangefs_file_open,
+ .flush = orangefs_flush,
.release = orangefs_file_release,
.fsync = orangefs_fsync,
};
// SPDX-License-Identifier: GPL-2.0
/*
* (C) 2001 Clemson University and The University of Chicago
+ * Copyright 2018 Omnibond Systems, L.L.C.
*
* See COPYING in top-level directory.
*/
#include "orangefs-kernel.h"
#include "orangefs-bufmap.h"
-static int read_one_page(struct page *page)
+static int orangefs_writepage_locked(struct page *page,
+ struct writeback_control *wbc)
{
- int ret;
- int max_block;
- ssize_t bytes_read = 0;
struct inode *inode = page->mapping->host;
- const __u32 blocksize = PAGE_SIZE;
- const __u32 blockbits = PAGE_SHIFT;
- struct iov_iter to;
- struct bio_vec bv = {.bv_page = page, .bv_len = PAGE_SIZE};
+ struct orangefs_write_range *wr = NULL;
+ struct iov_iter iter;
+ struct bio_vec bv;
+ size_t len, wlen;
+ ssize_t ret;
+ loff_t off;
+
+ set_page_writeback(page);
+
+ len = i_size_read(inode);
+ if (PagePrivate(page)) {
+ wr = (struct orangefs_write_range *)page_private(page);
+ WARN_ON(wr->pos >= len);
+ off = wr->pos;
+ if (off + wr->len > len)
+ wlen = len - off;
+ else
+ wlen = wr->len;
+ } else {
+ WARN_ON(1);
+ off = page_offset(page);
+ if (off + PAGE_SIZE > len)
+ wlen = len - off;
+ else
+ wlen = PAGE_SIZE;
+ }
+ /* Should've been handled in orangefs_invalidatepage. */
+ WARN_ON(off == len || off + wlen > len);
+
+ bv.bv_page = page;
+ bv.bv_len = wlen;
+ bv.bv_offset = off % PAGE_SIZE;
+ WARN_ON(wlen == 0);
+ iov_iter_bvec(&iter, WRITE, &bv, 1, wlen);
+
+ ret = wait_for_direct_io(ORANGEFS_IO_WRITE, inode, &off, &iter, wlen,
+ len, wr, NULL);
+ if (ret < 0) {
+ SetPageError(page);
+ mapping_set_error(page->mapping, ret);
+ } else {
+ ret = 0;
+ }
+ if (wr) {
+ kfree(wr);
+ set_page_private(page, 0);
+ ClearPagePrivate(page);
+ put_page(page);
+ }
+ return ret;
+}
+
+static int orangefs_writepage(struct page *page, struct writeback_control *wbc)
+{
+ int ret;
+ ret = orangefs_writepage_locked(page, wbc);
+ unlock_page(page);
+ end_page_writeback(page);
+ return ret;
+}
- iov_iter_bvec(&to, READ, &bv, 1, PAGE_SIZE);
+struct orangefs_writepages {
+ loff_t off;
+ size_t len;
+ kuid_t uid;
+ kgid_t gid;
+ int maxpages;
+ int npages;
+ struct page **pages;
+ struct bio_vec *bv;
+};
- gossip_debug(GOSSIP_INODE_DEBUG,
- "orangefs_readpage called with page %p\n",
- page);
+static int orangefs_writepages_work(struct orangefs_writepages *ow,
+ struct writeback_control *wbc)
+{
+ struct inode *inode = ow->pages[0]->mapping->host;
+ struct orangefs_write_range *wrp, wr;
+ struct iov_iter iter;
+ ssize_t ret;
+ size_t len;
+ loff_t off;
+ int i;
+
+ len = i_size_read(inode);
+
+ for (i = 0; i < ow->npages; i++) {
+ set_page_writeback(ow->pages[i]);
+ ow->bv[i].bv_page = ow->pages[i];
+ ow->bv[i].bv_len = min(page_offset(ow->pages[i]) + PAGE_SIZE,
+ ow->off + ow->len) -
+ max(ow->off, page_offset(ow->pages[i]));
+ if (i == 0)
+ ow->bv[i].bv_offset = ow->off -
+ page_offset(ow->pages[i]);
+ else
+ ow->bv[i].bv_offset = 0;
+ }
+ iov_iter_bvec(&iter, WRITE, ow->bv, ow->npages, ow->len);
+
+ WARN_ON(ow->off >= len);
+ if (ow->off + ow->len > len)
+ ow->len = len - ow->off;
+
+ off = ow->off;
+ wr.uid = ow->uid;
+ wr.gid = ow->gid;
+ ret = wait_for_direct_io(ORANGEFS_IO_WRITE, inode, &off, &iter, ow->len,
+ 0, &wr, NULL);
+ if (ret < 0) {
+ for (i = 0; i < ow->npages; i++) {
+ SetPageError(ow->pages[i]);
+ mapping_set_error(ow->pages[i]->mapping, ret);
+ if (PagePrivate(ow->pages[i])) {
+ wrp = (struct orangefs_write_range *)
+ page_private(ow->pages[i]);
+ ClearPagePrivate(ow->pages[i]);
+ put_page(ow->pages[i]);
+ kfree(wrp);
+ }
+ end_page_writeback(ow->pages[i]);
+ unlock_page(ow->pages[i]);
+ }
+ } else {
+ ret = 0;
+ for (i = 0; i < ow->npages; i++) {
+ if (PagePrivate(ow->pages[i])) {
+ wrp = (struct orangefs_write_range *)
+ page_private(ow->pages[i]);
+ ClearPagePrivate(ow->pages[i]);
+ put_page(ow->pages[i]);
+ kfree(wrp);
+ }
+ end_page_writeback(ow->pages[i]);
+ unlock_page(ow->pages[i]);
+ }
+ }
+ return ret;
+}
+
+static int orangefs_writepages_callback(struct page *page,
+ struct writeback_control *wbc, void *data)
+{
+ struct orangefs_writepages *ow = data;
+ struct orangefs_write_range *wr;
+ int ret;
+
+ if (!PagePrivate(page)) {
+ unlock_page(page);
+ /* It's not private so there's nothing to write, right? */
+ printk("writepages_callback not private!\n");
+ BUG();
+ return 0;
+ }
+ wr = (struct orangefs_write_range *)page_private(page);
+
+ ret = -1;
+ if (ow->npages == 0) {
+ ow->off = wr->pos;
+ ow->len = wr->len;
+ ow->uid = wr->uid;
+ ow->gid = wr->gid;
+ ow->pages[ow->npages++] = page;
+ ret = 0;
+ goto done;
+ }
+ if (!uid_eq(ow->uid, wr->uid) || !gid_eq(ow->gid, wr->gid)) {
+ orangefs_writepages_work(ow, wbc);
+ ow->npages = 0;
+ ret = -1;
+ goto done;
+ }
+ if (ow->off + ow->len == wr->pos) {
+ ow->len += wr->len;
+ ow->pages[ow->npages++] = page;
+ ret = 0;
+ goto done;
+ }
+done:
+ if (ret == -1) {
+ if (ow->npages) {
+ orangefs_writepages_work(ow, wbc);
+ ow->npages = 0;
+ }
+ ret = orangefs_writepage_locked(page, wbc);
+ mapping_set_error(page->mapping, ret);
+ unlock_page(page);
+ end_page_writeback(page);
+ } else {
+ if (ow->npages == ow->maxpages) {
+ orangefs_writepages_work(ow, wbc);
+ ow->npages = 0;
+ }
+ }
+ return ret;
+}
+
+static int orangefs_writepages(struct address_space *mapping,
+ struct writeback_control *wbc)
+{
+ struct orangefs_writepages *ow;
+ struct blk_plug plug;
+ int ret;
+ ow = kzalloc(sizeof(struct orangefs_writepages), GFP_KERNEL);
+ if (!ow)
+ return -ENOMEM;
+ ow->maxpages = orangefs_bufmap_size_query()/PAGE_SIZE;
+ ow->pages = kcalloc(ow->maxpages, sizeof(struct page *), GFP_KERNEL);
+ if (!ow->pages) {
+ kfree(ow);
+ return -ENOMEM;
+ }
+ ow->bv = kcalloc(ow->maxpages, sizeof(struct bio_vec), GFP_KERNEL);
+ if (!ow->bv) {
+ kfree(ow->pages);
+ kfree(ow);
+ return -ENOMEM;
+ }
+ blk_start_plug(&plug);
+ ret = write_cache_pages(mapping, wbc, orangefs_writepages_callback, ow);
+ if (ow->npages)
+ ret = orangefs_writepages_work(ow, wbc);
+ blk_finish_plug(&plug);
+ kfree(ow->pages);
+ kfree(ow->bv);
+ kfree(ow);
+ return ret;
+}
- max_block = ((inode->i_size / blocksize) + 1);
+static int orangefs_launder_page(struct page *);
- if (page->index < max_block) {
- loff_t blockptr_offset = (((loff_t) page->index) << blockbits);
+static int orangefs_readpage(struct file *file, struct page *page)
+{
+ struct inode *inode = page->mapping->host;
+ struct iov_iter iter;
+ struct bio_vec bv;
+ ssize_t ret;
+ loff_t off; /* offset into this page */
+ pgoff_t index; /* which page */
+ struct page *next_page;
+ char *kaddr;
+ struct orangefs_read_options *ro = file->private_data;
+ loff_t read_size;
+ loff_t roundedup;
+ int buffer_index = -1; /* orangefs shared memory slot */
+ int slot_index; /* index into slot */
+ int remaining;
- bytes_read = orangefs_inode_read(inode,
- &to,
- &blockptr_offset,
- inode->i_size);
+ /*
+ * If they set some miniscule size for "count" in read(2)
+ * (for example) then let's try to read a page, or the whole file
+ * if it is smaller than a page. Once "count" goes over a page
+ * then lets round up to the highest page size multiple that is
+ * less than or equal to "count" and do that much orangefs IO and
+ * try to fill as many pages as we can from it.
+ *
+ * "count" should be represented in ro->blksiz.
+ *
+ * inode->i_size = file size.
+ */
+ if (ro) {
+ if (ro->blksiz < PAGE_SIZE) {
+ if (inode->i_size < PAGE_SIZE)
+ read_size = inode->i_size;
+ else
+ read_size = PAGE_SIZE;
+ } else {
+ roundedup = ((PAGE_SIZE - 1) & ro->blksiz) ?
+ ((ro->blksiz + PAGE_SIZE) & ~(PAGE_SIZE -1)) :
+ ro->blksiz;
+ if (roundedup > inode->i_size)
+ read_size = inode->i_size;
+ else
+ read_size = roundedup;
+
+ }
+ } else {
+ read_size = PAGE_SIZE;
}
+ if (!read_size)
+ read_size = PAGE_SIZE;
+
+ if (PageDirty(page))
+ orangefs_launder_page(page);
+
+ off = page_offset(page);
+ index = off >> PAGE_SHIFT;
+ bv.bv_page = page;
+ bv.bv_len = PAGE_SIZE;
+ bv.bv_offset = 0;
+ iov_iter_bvec(&iter, READ, &bv, 1, PAGE_SIZE);
+
+ ret = wait_for_direct_io(ORANGEFS_IO_READ, inode, &off, &iter,
+ read_size, inode->i_size, NULL, &buffer_index);
+ remaining = ret;
/* this will only zero remaining unread portions of the page data */
- iov_iter_zero(~0U, &to);
+ iov_iter_zero(~0U, &iter);
/* takes care of potential aliasing */
flush_dcache_page(page);
- if (bytes_read < 0) {
- ret = bytes_read;
+ if (ret < 0) {
SetPageError(page);
+ unlock_page(page);
+ goto out;
} else {
SetPageUptodate(page);
if (PageError(page))
}
/* unlock the page after the ->readpage() routine completes */
unlock_page(page);
+
+ if (remaining > PAGE_SIZE) {
+ slot_index = 0;
+ while ((remaining - PAGE_SIZE) >= PAGE_SIZE) {
+ remaining -= PAGE_SIZE;
+ /*
+ * It is an optimization to try and fill more than one
+ * page... by now we've already gotten the single
+ * page we were after, if stuff doesn't seem to
+ * be going our way at this point just return
+ * and hope for the best.
+ *
+ * If we look for pages and they're already there is
+ * one reason to give up, and if they're not there
+ * and we can't create them is another reason.
+ */
+
+ index++;
+ slot_index++;
+ next_page = find_get_page(inode->i_mapping, index);
+ if (next_page) {
+ gossip_debug(GOSSIP_FILE_DEBUG,
+ "%s: found next page, quitting\n",
+ __func__);
+ put_page(next_page);
+ goto out;
+ }
+ next_page = find_or_create_page(inode->i_mapping,
+ index,
+ GFP_KERNEL);
+ /*
+ * I've never hit this, leave it as a printk for
+ * now so it will be obvious.
+ */
+ if (!next_page) {
+ printk("%s: can't create next page, quitting\n",
+ __func__);
+ goto out;
+ }
+ kaddr = kmap_atomic(next_page);
+ orangefs_bufmap_page_fill(kaddr,
+ buffer_index,
+ slot_index);
+ kunmap_atomic(kaddr);
+ SetPageUptodate(next_page);
+ unlock_page(next_page);
+ put_page(next_page);
+ }
+ }
+
+out:
+ if (buffer_index != -1)
+ orangefs_bufmap_put(buffer_index);
return ret;
}
-static int orangefs_readpage(struct file *file, struct page *page)
+static int orangefs_write_begin(struct file *file,
+ struct address_space *mapping,
+ loff_t pos, unsigned len, unsigned flags, struct page **pagep,
+ void **fsdata)
{
- return read_one_page(page);
+ struct orangefs_write_range *wr;
+ struct page *page;
+ pgoff_t index;
+ int ret;
+
+ index = pos >> PAGE_SHIFT;
+
+ page = grab_cache_page_write_begin(mapping, index, flags);
+ if (!page)
+ return -ENOMEM;
+
+ *pagep = page;
+
+ if (PageDirty(page) && !PagePrivate(page)) {
+ /*
+ * Should be impossible. If it happens, launder the page
+ * since we don't know what's dirty. This will WARN in
+ * orangefs_writepage_locked.
+ */
+ ret = orangefs_launder_page(page);
+ if (ret)
+ return ret;
+ }
+ if (PagePrivate(page)) {
+ struct orangefs_write_range *wr;
+ wr = (struct orangefs_write_range *)page_private(page);
+ if (wr->pos + wr->len == pos &&
+ uid_eq(wr->uid, current_fsuid()) &&
+ gid_eq(wr->gid, current_fsgid())) {
+ wr->len += len;
+ goto okay;
+ } else {
+ ret = orangefs_launder_page(page);
+ if (ret)
+ return ret;
+ }
+ }
+
+ wr = kmalloc(sizeof *wr, GFP_KERNEL);
+ if (!wr)
+ return -ENOMEM;
+
+ wr->pos = pos;
+ wr->len = len;
+ wr->uid = current_fsuid();
+ wr->gid = current_fsgid();
+ SetPagePrivate(page);
+ set_page_private(page, (unsigned long)wr);
+ get_page(page);
+okay:
+ return 0;
}
-static int orangefs_readpages(struct file *file,
- struct address_space *mapping,
- struct list_head *pages,
- unsigned nr_pages)
+static int orangefs_write_end(struct file *file, struct address_space *mapping,
+ loff_t pos, unsigned len, unsigned copied, struct page *page, void *fsdata)
{
- int page_idx;
- int ret;
+ struct inode *inode = page->mapping->host;
+ loff_t last_pos = pos + copied;
- gossip_debug(GOSSIP_INODE_DEBUG, "orangefs_readpages called\n");
-
- for (page_idx = 0; page_idx < nr_pages; page_idx++) {
- struct page *page;
-
- page = lru_to_page(pages);
- list_del(&page->lru);
- if (!add_to_page_cache(page,
- mapping,
- page->index,
- readahead_gfp_mask(mapping))) {
- ret = read_one_page(page);
- gossip_debug(GOSSIP_INODE_DEBUG,
- "failure adding page to cache, read_one_page returned: %d\n",
- ret);
- } else {
- put_page(page);
- }
- }
- BUG_ON(!list_empty(pages));
- return 0;
+ /*
+ * No need to use i_size_read() here, the i_size
+ * cannot change under us because we hold the i_mutex.
+ */
+ if (last_pos > inode->i_size)
+ i_size_write(inode, last_pos);
+
+ /* zero the stale part of the page if we did a short copy */
+ if (!PageUptodate(page)) {
+ unsigned from = pos & (PAGE_SIZE - 1);
+ if (copied < len) {
+ zero_user(page, from + copied, len - copied);
+ }
+ /* Set fully written pages uptodate. */
+ if (pos == page_offset(page) &&
+ (len == PAGE_SIZE || pos + len == inode->i_size)) {
+ zero_user_segment(page, from + copied, PAGE_SIZE);
+ SetPageUptodate(page);
+ }
+ }
+
+ set_page_dirty(page);
+ unlock_page(page);
+ put_page(page);
+
+ mark_inode_dirty_sync(file_inode(file));
+ return copied;
}
static void orangefs_invalidatepage(struct page *page,
unsigned int offset,
unsigned int length)
{
- gossip_debug(GOSSIP_INODE_DEBUG,
- "orangefs_invalidatepage called on page %p "
- "(offset is %u)\n",
- page,
- offset);
+ struct orangefs_write_range *wr;
+ wr = (struct orangefs_write_range *)page_private(page);
+
+ if (offset == 0 && length == PAGE_SIZE) {
+ kfree((struct orangefs_write_range *)page_private(page));
+ set_page_private(page, 0);
+ ClearPagePrivate(page);
+ put_page(page);
+ return;
+ /* write range entirely within invalidate range (or equal) */
+ } else if (page_offset(page) + offset <= wr->pos &&
+ wr->pos + wr->len <= page_offset(page) + offset + length) {
+ kfree((struct orangefs_write_range *)page_private(page));
+ set_page_private(page, 0);
+ ClearPagePrivate(page);
+ put_page(page);
+ /* XXX is this right? only caller in fs */
+ cancel_dirty_page(page);
+ return;
+ /* invalidate range chops off end of write range */
+ } else if (wr->pos < page_offset(page) + offset &&
+ wr->pos + wr->len <= page_offset(page) + offset + length &&
+ page_offset(page) + offset < wr->pos + wr->len) {
+ size_t x;
+ x = wr->pos + wr->len - (page_offset(page) + offset);
+ WARN_ON(x > wr->len);
+ wr->len -= x;
+ wr->uid = current_fsuid();
+ wr->gid = current_fsgid();
+ /* invalidate range chops off beginning of write range */
+ } else if (page_offset(page) + offset <= wr->pos &&
+ page_offset(page) + offset + length < wr->pos + wr->len &&
+ wr->pos < page_offset(page) + offset + length) {
+ size_t x;
+ x = page_offset(page) + offset + length - wr->pos;
+ WARN_ON(x > wr->len);
+ wr->pos += x;
+ wr->len -= x;
+ wr->uid = current_fsuid();
+ wr->gid = current_fsgid();
+ /* invalidate range entirely within write range (punch hole) */
+ } else if (wr->pos < page_offset(page) + offset &&
+ page_offset(page) + offset + length < wr->pos + wr->len) {
+ /* XXX what do we do here... should not WARN_ON */
+ WARN_ON(1);
+ /* punch hole */
+ /*
+ * should we just ignore this and write it out anyway?
+ * it hardly makes sense
+ */
+ return;
+ /* non-overlapping ranges */
+ } else {
+ /* WARN if they do overlap */
+ if (!((page_offset(page) + offset + length <= wr->pos) ^
+ (wr->pos + wr->len <= page_offset(page) + offset))) {
+ WARN_ON(1);
+ printk("invalidate range offset %llu length %u\n",
+ page_offset(page) + offset, length);
+ printk("write range offset %llu length %zu\n",
+ wr->pos, wr->len);
+ }
+ return;
+ }
- ClearPageUptodate(page);
- ClearPageMappedToDisk(page);
- return;
+ /*
+ * Above there are returns where wr is freed or where we WARN.
+ * Thus the following runs if wr was modified above.
+ */
+ orangefs_launder_page(page);
}
static int orangefs_releasepage(struct page *page, gfp_t foo)
{
- gossip_debug(GOSSIP_INODE_DEBUG,
- "orangefs_releasepage called on page %p\n",
- page);
- return 0;
+ return !PagePrivate(page);
}
-/*
- * Having a direct_IO entry point in the address_space_operations
- * struct causes the kernel to allows us to use O_DIRECT on
- * open. Nothing will ever call this thing, but in the future we
- * will need to be able to use O_DIRECT on open in order to support
- * AIO. Modeled after NFS, they do this too.
- */
+static void orangefs_freepage(struct page *page)
+{
+ if (PagePrivate(page)) {
+ kfree((struct orangefs_write_range *)page_private(page));
+ set_page_private(page, 0);
+ ClearPagePrivate(page);
+ put_page(page);
+ }
+}
+
+static int orangefs_launder_page(struct page *page)
+{
+ int r = 0;
+ struct writeback_control wbc = {
+ .sync_mode = WB_SYNC_ALL,
+ .nr_to_write = 0,
+ };
+ wait_on_page_writeback(page);
+ if (clear_page_dirty_for_io(page)) {
+ r = orangefs_writepage_locked(page, &wbc);
+ end_page_writeback(page);
+ }
+ return r;
+}
static ssize_t orangefs_direct_IO(struct kiocb *iocb,
struct iov_iter *iter)
{
- gossip_debug(GOSSIP_INODE_DEBUG,
- "orangefs_direct_IO: %pD\n",
- iocb->ki_filp);
+ /*
+ * Comment from original do_readv_writev:
+ * Common entry point for read/write/readv/writev
+ * This function will dispatch it to either the direct I/O
+ * or buffered I/O path depending on the mount options and/or
+ * augmented/extended metadata attached to the file.
+ * Note: File extended attributes override any mount options.
+ */
+ struct file *file = iocb->ki_filp;
+ loff_t pos = iocb->ki_pos;
+ enum ORANGEFS_io_type type = iov_iter_rw(iter) == WRITE ?
+ ORANGEFS_IO_WRITE : ORANGEFS_IO_READ;
+ loff_t *offset = &pos;
+ struct inode *inode = file->f_mapping->host;
+ struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
+ struct orangefs_khandle *handle = &orangefs_inode->refn.khandle;
+ size_t count = iov_iter_count(iter);
+ ssize_t total_count = 0;
+ ssize_t ret = -EINVAL;
+ int i = 0;
+
+ gossip_debug(GOSSIP_FILE_DEBUG,
+ "%s-BEGIN(%pU): count(%d) after estimate_max_iovecs.\n",
+ __func__,
+ handle,
+ (int)count);
+
+ if (type == ORANGEFS_IO_WRITE) {
+ gossip_debug(GOSSIP_FILE_DEBUG,
+ "%s(%pU): proceeding with offset : %llu, "
+ "size %d\n",
+ __func__,
+ handle,
+ llu(*offset),
+ (int)count);
+ }
+
+ if (count == 0) {
+ ret = 0;
+ goto out;
+ }
- return -EINVAL;
+ while (iov_iter_count(iter)) {
+ size_t each_count = iov_iter_count(iter);
+ size_t amt_complete;
+ i++;
+
+ /* how much to transfer in this loop iteration */
+ if (each_count > orangefs_bufmap_size_query())
+ each_count = orangefs_bufmap_size_query();
+
+ gossip_debug(GOSSIP_FILE_DEBUG,
+ "%s(%pU): size of each_count(%d)\n",
+ __func__,
+ handle,
+ (int)each_count);
+ gossip_debug(GOSSIP_FILE_DEBUG,
+ "%s(%pU): BEFORE wait_for_io: offset is %d\n",
+ __func__,
+ handle,
+ (int)*offset);
+
+ ret = wait_for_direct_io(type, inode, offset, iter,
+ each_count, 0, NULL, NULL);
+ gossip_debug(GOSSIP_FILE_DEBUG,
+ "%s(%pU): return from wait_for_io:%d\n",
+ __func__,
+ handle,
+ (int)ret);
+
+ if (ret < 0)
+ goto out;
+
+ *offset += ret;
+ total_count += ret;
+ amt_complete = ret;
+
+ gossip_debug(GOSSIP_FILE_DEBUG,
+ "%s(%pU): AFTER wait_for_io: offset is %d\n",
+ __func__,
+ handle,
+ (int)*offset);
+
+ /*
+ * if we got a short I/O operations,
+ * fall out and return what we got so far
+ */
+ if (amt_complete < each_count)
+ break;
+ } /*end while */
+
+out:
+ if (total_count > 0)
+ ret = total_count;
+ if (ret > 0) {
+ if (type == ORANGEFS_IO_READ) {
+ file_accessed(file);
+ } else {
+ file_update_time(file);
+ if (*offset > i_size_read(inode))
+ i_size_write(inode, *offset);
+ }
+ }
+
+ gossip_debug(GOSSIP_FILE_DEBUG,
+ "%s(%pU): Value(%d) returned.\n",
+ __func__,
+ handle,
+ (int)ret);
+
+ return ret;
}
/** ORANGEFS2 implementation of address space operations */
static const struct address_space_operations orangefs_address_operations = {
+ .writepage = orangefs_writepage,
.readpage = orangefs_readpage,
- .readpages = orangefs_readpages,
+ .writepages = orangefs_writepages,
+ .set_page_dirty = __set_page_dirty_nobuffers,
+ .write_begin = orangefs_write_begin,
+ .write_end = orangefs_write_end,
.invalidatepage = orangefs_invalidatepage,
.releasepage = orangefs_releasepage,
+ .freepage = orangefs_freepage,
+ .launder_page = orangefs_launder_page,
.direct_IO = orangefs_direct_IO,
};
+vm_fault_t orangefs_page_mkwrite(struct vm_fault *vmf)
+{
+ struct page *page = vmf->page;
+ struct inode *inode = file_inode(vmf->vma->vm_file);
+ struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
+ unsigned long *bitlock = &orangefs_inode->bitlock;
+ vm_fault_t ret;
+ struct orangefs_write_range *wr;
+
+ sb_start_pagefault(inode->i_sb);
+
+ if (wait_on_bit(bitlock, 1, TASK_KILLABLE)) {
+ ret = VM_FAULT_RETRY;
+ goto out;
+ }
+
+ lock_page(page);
+ if (PageDirty(page) && !PagePrivate(page)) {
+ /*
+ * Should be impossible. If it happens, launder the page
+ * since we don't know what's dirty. This will WARN in
+ * orangefs_writepage_locked.
+ */
+ if (orangefs_launder_page(page)) {
+ ret = VM_FAULT_LOCKED|VM_FAULT_RETRY;
+ goto out;
+ }
+ }
+ if (PagePrivate(page)) {
+ wr = (struct orangefs_write_range *)page_private(page);
+ if (uid_eq(wr->uid, current_fsuid()) &&
+ gid_eq(wr->gid, current_fsgid())) {
+ wr->pos = page_offset(page);
+ wr->len = PAGE_SIZE;
+ goto okay;
+ } else {
+ if (orangefs_launder_page(page)) {
+ ret = VM_FAULT_LOCKED|VM_FAULT_RETRY;
+ goto out;
+ }
+ }
+ }
+ wr = kmalloc(sizeof *wr, GFP_KERNEL);
+ if (!wr) {
+ ret = VM_FAULT_LOCKED|VM_FAULT_RETRY;
+ goto out;
+ }
+ wr->pos = page_offset(page);
+ wr->len = PAGE_SIZE;
+ wr->uid = current_fsuid();
+ wr->gid = current_fsgid();
+ SetPagePrivate(page);
+ set_page_private(page, (unsigned long)wr);
+ get_page(page);
+okay:
+
+ file_update_time(vmf->vma->vm_file);
+ if (page->mapping != inode->i_mapping) {
+ unlock_page(page);
+ ret = VM_FAULT_LOCKED|VM_FAULT_NOPAGE;
+ goto out;
+ }
+
+ /*
+ * We mark the page dirty already here so that when freeze is in
+ * progress, we are guaranteed that writeback during freezing will
+ * see the dirty page and writeprotect it again.
+ */
+ set_page_dirty(page);
+ wait_for_stable_page(page);
+ ret = VM_FAULT_LOCKED;
+out:
+ sb_end_pagefault(inode->i_sb);
+ return ret;
+}
+
static int orangefs_setattr_size(struct inode *inode, struct iattr *iattr)
{
struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
iattr->ia_size);
/* Ensure that we have a up to date size, so we know if it changed. */
- ret = orangefs_inode_getattr(inode, 0, 1, STATX_SIZE);
+ ret = orangefs_inode_getattr(inode, ORANGEFS_GETATTR_SIZE);
if (ret == -ESTALE)
ret = -EIO;
if (ret) {
}
orig_size = i_size_read(inode);
- truncate_setsize(inode, iattr->ia_size);
+ /* This is truncate_setsize in a different order. */
+ truncate_pagecache(inode, iattr->ia_size);
+ i_size_write(inode, iattr->ia_size);
+ if (iattr->ia_size > orig_size)
+ pagecache_isize_extended(inode, orig_size, iattr->ia_size);
new_op = op_alloc(ORANGEFS_VFS_OP_TRUNCATE);
if (!new_op)
return ret;
}
-/*
- * Change attributes of an object referenced by dentry.
- */
-int orangefs_setattr(struct dentry *dentry, struct iattr *iattr)
+int __orangefs_setattr(struct inode *inode, struct iattr *iattr)
{
- int ret = -EINVAL;
- struct inode *inode = dentry->d_inode;
-
- gossip_debug(GOSSIP_INODE_DEBUG,
- "%s: called on %pd\n",
- __func__,
- dentry);
+ int ret;
- ret = setattr_prepare(dentry, iattr);
- if (ret)
- goto out;
+ if (iattr->ia_valid & ATTR_MODE) {
+ if (iattr->ia_mode & (S_ISVTX)) {
+ if (is_root_handle(inode)) {
+ /*
+ * allow sticky bit to be set on root (since
+ * it shows up that way by default anyhow),
+ * but don't show it to the server
+ */
+ iattr->ia_mode -= S_ISVTX;
+ } else {
+ gossip_debug(GOSSIP_UTILS_DEBUG,
+ "User attempted to set sticky bit on non-root directory; returning EINVAL.\n");
+ ret = -EINVAL;
+ goto out;
+ }
+ }
+ if (iattr->ia_mode & (S_ISUID)) {
+ gossip_debug(GOSSIP_UTILS_DEBUG,
+ "Attempting to set setuid bit (not supported); returning EINVAL.\n");
+ ret = -EINVAL;
+ goto out;
+ }
+ }
if (iattr->ia_valid & ATTR_SIZE) {
ret = orangefs_setattr_size(inode, iattr);
goto out;
}
+again:
+ spin_lock(&inode->i_lock);
+ if (ORANGEFS_I(inode)->attr_valid) {
+ if (uid_eq(ORANGEFS_I(inode)->attr_uid, current_fsuid()) &&
+ gid_eq(ORANGEFS_I(inode)->attr_gid, current_fsgid())) {
+ ORANGEFS_I(inode)->attr_valid = iattr->ia_valid;
+ } else {
+ spin_unlock(&inode->i_lock);
+ write_inode_now(inode, 1);
+ goto again;
+ }
+ } else {
+ ORANGEFS_I(inode)->attr_valid = iattr->ia_valid;
+ ORANGEFS_I(inode)->attr_uid = current_fsuid();
+ ORANGEFS_I(inode)->attr_gid = current_fsgid();
+ }
setattr_copy(inode, iattr);
+ spin_unlock(&inode->i_lock);
mark_inode_dirty(inode);
- ret = orangefs_inode_setattr(inode, iattr);
- gossip_debug(GOSSIP_INODE_DEBUG,
- "%s: orangefs_inode_setattr returned %d\n",
- __func__,
- ret);
-
- if (!ret && (iattr->ia_valid & ATTR_MODE))
+ if (iattr->ia_valid & ATTR_MODE)
/* change mod on a file that has ACLs */
ret = posix_acl_chmod(inode, inode->i_mode);
+ ret = 0;
out:
- gossip_debug(GOSSIP_INODE_DEBUG, "%s: ret:%d:\n", __func__, ret);
+ return ret;
+}
+
+/*
+ * Change attributes of an object referenced by dentry.
+ */
+int orangefs_setattr(struct dentry *dentry, struct iattr *iattr)
+{
+ int ret;
+ gossip_debug(GOSSIP_INODE_DEBUG, "__orangefs_setattr: called on %pd\n",
+ dentry);
+ ret = setattr_prepare(dentry, iattr);
+ if (ret)
+ goto out;
+ ret = __orangefs_setattr(d_inode(dentry), iattr);
+ sync_inode_metadata(d_inode(dentry), 1);
+out:
+ gossip_debug(GOSSIP_INODE_DEBUG, "orangefs_setattr: returning %d\n",
+ ret);
return ret;
}
struct inode *inode = path->dentry->d_inode;
gossip_debug(GOSSIP_INODE_DEBUG,
- "orangefs_getattr: called on %pd\n",
- path->dentry);
+ "orangefs_getattr: called on %pd mask %u\n",
+ path->dentry, request_mask);
- ret = orangefs_inode_getattr(inode, 0, 0, request_mask);
+ ret = orangefs_inode_getattr(inode,
+ request_mask & STATX_SIZE ? ORANGEFS_GETATTR_SIZE : 0);
if (ret == 0) {
generic_fillattr(inode, stat);
gossip_debug(GOSSIP_INODE_DEBUG, "%s: refreshing\n", __func__);
/* Make sure the permission (and other common attrs) are up to date. */
- ret = orangefs_inode_getattr(inode, 0, 0, STATX_MODE);
+ ret = orangefs_inode_getattr(inode, 0);
if (ret < 0)
return ret;
iattr.ia_valid |= ATTR_CTIME;
if (flags & S_MTIME)
iattr.ia_valid |= ATTR_MTIME;
- return orangefs_inode_setattr(inode, &iattr);
+ return __orangefs_setattr(inode, &iattr);
}
/* ORANGEFS2 implementation of VFS inode operations for files */
struct orangefs_object_kref *ref = (struct orangefs_object_kref *) data;
ORANGEFS_I(inode)->refn.fs_id = ref->fs_id;
ORANGEFS_I(inode)->refn.khandle = ref->khandle;
+ ORANGEFS_I(inode)->attr_valid = 0;
+ hash_init(ORANGEFS_I(inode)->xattr_cache);
+ ORANGEFS_I(inode)->mapping_time = jiffies - 1;
+ ORANGEFS_I(inode)->bitlock = 0;
return 0;
}
if (!(inode->i_state & I_NEW))
return inode;
- error = orangefs_inode_getattr(inode, 1, 1, STATX_ALL);
+ error = orangefs_inode_getattr(inode, ORANGEFS_GETATTR_NEW);
if (error) {
iget_failed(inode);
return ERR_PTR(error);
orangefs_set_inode(inode, ref);
inode->i_ino = hash; /* needed for stat etc */
- error = orangefs_inode_getattr(inode, 1, 1, STATX_ALL);
+ error = orangefs_inode_getattr(inode, ORANGEFS_GETATTR_NEW);
if (error)
goto out_iput;
orangefs_init_iops(inode);
-
- inode->i_mode = mode;
- inode->i_uid = current_fsuid();
- inode->i_gid = current_fsgid();
- inode->i_atime = inode->i_mtime = inode->i_ctime = current_time(inode);
- inode->i_size = PAGE_SIZE;
inode->i_rdev = dev;
error = insert_inode_locked4(inode, hash, orangefs_test_inode, ref);
d_instantiate_new(dentry, inode);
orangefs_set_timeout(dentry);
- ORANGEFS_I(inode)->getattr_time = jiffies - 1;
- ORANGEFS_I(inode)->getattr_mask = STATX_BASIC_STATS;
gossip_debug(GOSSIP_NAME_DEBUG,
"%s: dentry instantiated for %pd\n",
__func__,
dentry);
- dir->i_mtime = dir->i_ctime = current_time(dir);
memset(&iattr, 0, sizeof iattr);
- iattr.ia_valid |= ATTR_MTIME;
- orangefs_inode_setattr(dir, &iattr);
- mark_inode_dirty_sync(dir);
+ iattr.ia_valid |= ATTR_MTIME | ATTR_CTIME;
+ iattr.ia_mtime = iattr.ia_ctime = current_time(dir);
+ __orangefs_setattr(dir, &iattr);
ret = 0;
out:
op_release(new_op);
if (!ret) {
drop_nlink(inode);
- dir->i_mtime = dir->i_ctime = current_time(dir);
memset(&iattr, 0, sizeof iattr);
- iattr.ia_valid |= ATTR_MTIME;
- orangefs_inode_setattr(dir, &iattr);
- mark_inode_dirty_sync(dir);
+ iattr.ia_valid |= ATTR_MTIME | ATTR_CTIME;
+ iattr.ia_mtime = iattr.ia_ctime = current_time(dir);
+ __orangefs_setattr(dir, &iattr);
}
return ret;
}
d_instantiate_new(dentry, inode);
orangefs_set_timeout(dentry);
- ORANGEFS_I(inode)->getattr_time = jiffies - 1;
- ORANGEFS_I(inode)->getattr_mask = STATX_BASIC_STATS;
gossip_debug(GOSSIP_NAME_DEBUG,
"Inode (Symlink) %pU -> %pd\n",
get_khandle_from_ino(inode),
dentry);
- dir->i_mtime = dir->i_ctime = current_time(dir);
memset(&iattr, 0, sizeof iattr);
- iattr.ia_valid |= ATTR_MTIME;
- orangefs_inode_setattr(dir, &iattr);
- mark_inode_dirty_sync(dir);
+ iattr.ia_valid |= ATTR_MTIME | ATTR_CTIME;
+ iattr.ia_mtime = iattr.ia_ctime = current_time(dir);
+ __orangefs_setattr(dir, &iattr);
ret = 0;
out:
op_release(new_op);
d_instantiate_new(dentry, inode);
orangefs_set_timeout(dentry);
- ORANGEFS_I(inode)->getattr_time = jiffies - 1;
- ORANGEFS_I(inode)->getattr_mask = STATX_BASIC_STATS;
gossip_debug(GOSSIP_NAME_DEBUG,
"Inode (Directory) %pU -> %pd\n",
* NOTE: we have no good way to keep nlink consistent for directories
* across clients; keep constant at 1.
*/
- dir->i_mtime = dir->i_ctime = current_time(dir);
memset(&iattr, 0, sizeof iattr);
- iattr.ia_valid |= ATTR_MTIME;
- orangefs_inode_setattr(dir, &iattr);
- mark_inode_dirty_sync(dir);
+ iattr.ia_valid |= ATTR_MTIME | ATTR_CTIME;
+ iattr.ia_mtime = iattr.ia_ctime = current_time(dir);
+ __orangefs_setattr(dir, &iattr);
out:
op_release(new_op);
return ret;
unsigned int flags)
{
struct orangefs_kernel_op_s *new_op;
+ struct iattr iattr;
int ret;
if (flags)
"orangefs_rename: called (%pd2 => %pd2) ct=%d\n",
old_dentry, new_dentry, d_count(new_dentry));
- ORANGEFS_I(new_dentry->d_parent->d_inode)->getattr_time = jiffies - 1;
+ memset(&iattr, 0, sizeof iattr);
+ iattr.ia_valid |= ATTR_MTIME | ATTR_CTIME;
+ iattr.ia_mtime = iattr.ia_ctime = current_time(new_dir);
+ __orangefs_setattr(new_dir, &iattr);
new_op = op_alloc(ORANGEFS_VFS_OP_RENAME);
if (!new_op)
}
return 0;
}
+
+void orangefs_bufmap_page_fill(void *page_to,
+ int buffer_index,
+ int slot_index)
+{
+ struct orangefs_bufmap_desc *from;
+ void *page_from;
+
+ from = &__orangefs_bufmap->desc_array[buffer_index];
+ page_from = kmap_atomic(from->page_array[slot_index]);
+ memcpy(page_to, page_from, PAGE_SIZE);
+ kunmap_atomic(page_from);
+}
int buffer_index,
size_t size);
+void orangefs_bufmap_page_fill(void *kaddr, int buffer_index, int slot_index);
+
#endif /* __ORANGEFS_BUFMAP_H */
return ret;
}
-int orangefs_debugfs_new_client_string(void __user *arg)
+int orangefs_debugfs_new_client_string(void __user *arg)
{
int ret;
return 0;
}
-int orangefs_debugfs_new_debug(void __user *arg)
+int orangefs_debugfs_new_debug(void __user *arg)
{
struct dev_mask_info_s mask_info = {0};
int ret;
#include <linux/rwsem.h>
#include <linux/xattr.h>
#include <linux/exportfs.h>
+#include <linux/hashtable.h>
#include <asm/unaligned.h>
sector_t last_failed_block_index_read;
unsigned long getattr_time;
- u32 getattr_mask;
+ unsigned long mapping_time;
+ int attr_valid;
+ kuid_t attr_uid;
+ kgid_t attr_gid;
+ unsigned long bitlock;
+
+ DECLARE_HASHTABLE(xattr_cache, 4);
};
/* per superblock private orangefs info */
unsigned long writes;
};
+struct orangefs_cached_xattr {
+ struct hlist_node node;
+ char key[ORANGEFS_MAX_XATTR_NAMELEN];
+ char val[ORANGEFS_MAX_XATTR_VALUELEN];
+ ssize_t length;
+ unsigned long timeout;
+};
+
+struct orangefs_write_range {
+ loff_t pos;
+ size_t len;
+ kuid_t uid;
+ kgid_t gid;
+};
+
+struct orangefs_read_options {
+ ssize_t blksiz;
+};
+
extern struct orangefs_stats orangefs_stats;
/*
/*
* defined in inode.c
*/
+vm_fault_t orangefs_page_mkwrite(struct vm_fault *);
struct inode *orangefs_new_inode(struct super_block *sb,
struct inode *dir,
int mode,
dev_t dev,
struct orangefs_object_kref *ref);
-int orangefs_setattr(struct dentry *dentry, struct iattr *iattr);
+int __orangefs_setattr(struct inode *, struct iattr *);
+int orangefs_setattr(struct dentry *, struct iattr *);
int orangefs_getattr(const struct path *path, struct kstat *stat,
u32 request_mask, unsigned int flags);
struct inode *orangefs_iget(struct super_block *sb,
struct orangefs_object_kref *ref);
-ssize_t orangefs_inode_read(struct inode *inode,
- struct iov_iter *iter,
- loff_t *offset,
- loff_t readahead_size);
-
/*
* defined in devorangefs-req.c
*/
int is_daemon_in_service(void);
bool __is_daemon_in_service(void);
+/*
+ * defined in file.c
+ */
+int orangefs_revalidate_mapping(struct inode *);
+ssize_t wait_for_direct_io(enum ORANGEFS_io_type, struct inode *, loff_t *,
+ struct iov_iter *, size_t, loff_t, struct orangefs_write_range *, int *);
+ssize_t do_readv_writev(enum ORANGEFS_io_type, struct file *, loff_t *,
+ struct iov_iter *);
+
/*
* defined in orangefs-utils.c
*/
size_t size,
int flags);
-int orangefs_inode_getattr(struct inode *inode, int new, int bypass,
- u32 request_mask);
+#define ORANGEFS_GETATTR_NEW 1
+#define ORANGEFS_GETATTR_SIZE 2
+
+int orangefs_inode_getattr(struct inode *, int);
int orangefs_inode_check_changed(struct inode *inode);
-int orangefs_inode_setattr(struct inode *inode, struct iattr *iattr);
+int orangefs_inode_setattr(struct inode *inode);
bool orangefs_cancel_op_in_progress(struct orangefs_kernel_op_s *op);
extern struct mutex orangefs_request_mutex;
extern int op_timeout_secs;
extern int slot_timeout_secs;
+extern int orangefs_cache_timeout_msecs;
extern int orangefs_dcache_timeout_msecs;
extern int orangefs_getattr_timeout_msecs;
extern struct list_head orangefs_superblocks;
#define ORANGEFS_OP_CANCELLATION 4 /* this is a cancellation */
#define ORANGEFS_OP_NO_MUTEX 8 /* don't acquire request_mutex */
#define ORANGEFS_OP_ASYNC 16 /* Queue it, but don't wait */
+#define ORANGEFS_OP_WRITEBACK 32
int service_operation(struct orangefs_kernel_op_s *op,
const char *op_name,
__u64 orangefs_gossip_debug_mask;
int op_timeout_secs = ORANGEFS_DEFAULT_OP_TIMEOUT_SECS;
int slot_timeout_secs = ORANGEFS_DEFAULT_SLOT_TIMEOUT_SECS;
+int orangefs_cache_timeout_msecs = 50;
int orangefs_dcache_timeout_msecs = 50;
int orangefs_getattr_timeout_msecs = 50;
* Slots are requested and waited for,
* the wait times out after slot_timeout_secs.
*
+ * What: /sys/fs/orangefs/cache_timeout_msecs
+ * Date: Mar 2018
+ * Contact: Martin Brandenburg <martin@omnibond.com>
+ * Description:
+ * Time in milliseconds between which
+ * orangefs_revalidate_mapping will invalidate the page
+ * cache.
+ *
* What: /sys/fs/orangefs/dcache_timeout_msecs
* Date: Jul 2016
* Contact: Martin Brandenburg <martin@omnibond.com>
"%d\n",
slot_timeout_secs);
goto out;
+ } else if (!strcmp(attr->attr.name,
+ "cache_timeout_msecs")) {
+ rc = scnprintf(buf,
+ PAGE_SIZE,
+ "%d\n",
+ orangefs_cache_timeout_msecs);
+ goto out;
} else if (!strcmp(attr->attr.name,
"dcache_timeout_msecs")) {
rc = scnprintf(buf,
} else if (!strcmp(attr->attr.name, "slot_timeout_secs")) {
rc = kstrtoint(buf, 0, &slot_timeout_secs);
goto out;
+ } else if (!strcmp(attr->attr.name, "cache_timeout_msecs")) {
+ rc = kstrtoint(buf, 0, &orangefs_cache_timeout_msecs);
+ goto out;
} else if (!strcmp(attr->attr.name, "dcache_timeout_msecs")) {
rc = kstrtoint(buf, 0, &orangefs_dcache_timeout_msecs);
goto out;
static struct orangefs_attribute slot_timeout_secs_attribute =
__ATTR(slot_timeout_secs, 0664, sysfs_int_show, sysfs_int_store);
+static struct orangefs_attribute cache_timeout_msecs_attribute =
+ __ATTR(cache_timeout_msecs, 0664, sysfs_int_show, sysfs_int_store);
+
static struct orangefs_attribute dcache_timeout_msecs_attribute =
__ATTR(dcache_timeout_msecs, 0664, sysfs_int_show, sysfs_int_store);
static struct attribute *orangefs_default_attrs[] = {
&op_timeout_secs_attribute.attr,
&slot_timeout_secs_attribute.attr,
+ &cache_timeout_msecs_attribute.attr,
&dcache_timeout_msecs_attribute.attr,
&getattr_timeout_msecs_attribute.attr,
&readahead_count_attribute.attr,
// SPDX-License-Identifier: GPL-2.0
/*
* (C) 2001 Clemson University and The University of Chicago
+ * Copyright 2018 Omnibond Systems, L.L.C.
*
* See COPYING in top-level directory.
*/
* NOTE: in kernel land, we never use the sys_attr->link_target for
* anything, so don't bother copying it into the sys_attr object here.
*/
-static inline int copy_attributes_from_inode(struct inode *inode,
- struct ORANGEFS_sys_attr_s *attrs,
- struct iattr *iattr)
+static inline void copy_attributes_from_inode(struct inode *inode,
+ struct ORANGEFS_sys_attr_s *attrs)
{
- umode_t tmp_mode;
-
- if (!iattr || !inode || !attrs) {
- gossip_err("NULL iattr (%p), inode (%p), attrs (%p) "
- "in copy_attributes_from_inode!\n",
- iattr,
- inode,
- attrs);
- return -EINVAL;
- }
- /*
- * We need to be careful to only copy the attributes out of the
- * iattr object that we know are valid.
- */
+ struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
attrs->mask = 0;
- if (iattr->ia_valid & ATTR_UID) {
- attrs->owner = from_kuid(&init_user_ns, iattr->ia_uid);
+ if (orangefs_inode->attr_valid & ATTR_UID) {
+ attrs->owner = from_kuid(&init_user_ns, inode->i_uid);
attrs->mask |= ORANGEFS_ATTR_SYS_UID;
gossip_debug(GOSSIP_UTILS_DEBUG, "(UID) %d\n", attrs->owner);
}
- if (iattr->ia_valid & ATTR_GID) {
- attrs->group = from_kgid(&init_user_ns, iattr->ia_gid);
+ if (orangefs_inode->attr_valid & ATTR_GID) {
+ attrs->group = from_kgid(&init_user_ns, inode->i_gid);
attrs->mask |= ORANGEFS_ATTR_SYS_GID;
gossip_debug(GOSSIP_UTILS_DEBUG, "(GID) %d\n", attrs->group);
}
- if (iattr->ia_valid & ATTR_ATIME) {
+ if (orangefs_inode->attr_valid & ATTR_ATIME) {
attrs->mask |= ORANGEFS_ATTR_SYS_ATIME;
- if (iattr->ia_valid & ATTR_ATIME_SET) {
- attrs->atime = (time64_t)iattr->ia_atime.tv_sec;
+ if (orangefs_inode->attr_valid & ATTR_ATIME_SET) {
+ attrs->atime = (time64_t)inode->i_atime.tv_sec;
attrs->mask |= ORANGEFS_ATTR_SYS_ATIME_SET;
}
}
- if (iattr->ia_valid & ATTR_MTIME) {
+ if (orangefs_inode->attr_valid & ATTR_MTIME) {
attrs->mask |= ORANGEFS_ATTR_SYS_MTIME;
- if (iattr->ia_valid & ATTR_MTIME_SET) {
- attrs->mtime = (time64_t)iattr->ia_mtime.tv_sec;
+ if (orangefs_inode->attr_valid & ATTR_MTIME_SET) {
+ attrs->mtime = (time64_t)inode->i_mtime.tv_sec;
attrs->mask |= ORANGEFS_ATTR_SYS_MTIME_SET;
}
}
- if (iattr->ia_valid & ATTR_CTIME)
+ if (orangefs_inode->attr_valid & ATTR_CTIME)
attrs->mask |= ORANGEFS_ATTR_SYS_CTIME;
/*
* worry about ATTR_SIZE
*/
- if (iattr->ia_valid & ATTR_MODE) {
- tmp_mode = iattr->ia_mode;
- if (tmp_mode & (S_ISVTX)) {
- if (is_root_handle(inode)) {
- /*
- * allow sticky bit to be set on root (since
- * it shows up that way by default anyhow),
- * but don't show it to the server
- */
- tmp_mode -= S_ISVTX;
- } else {
- gossip_debug(GOSSIP_UTILS_DEBUG,
- "%s: setting sticky bit not supported.\n",
- __func__);
- return -EINVAL;
- }
- }
-
- if (tmp_mode & (S_ISUID)) {
- gossip_debug(GOSSIP_UTILS_DEBUG,
- "%s: setting setuid bit not supported.\n",
- __func__);
- return -EINVAL;
- }
-
- attrs->perms = ORANGEFS_util_translate_mode(tmp_mode);
+ if (orangefs_inode->attr_valid & ATTR_MODE) {
+ attrs->perms = ORANGEFS_util_translate_mode(inode->i_mode);
attrs->mask |= ORANGEFS_ATTR_SYS_PERM;
}
-
- return 0;
}
static int orangefs_inode_type(enum orangefs_ds_type objtype)
return 0;
}
-int orangefs_inode_getattr(struct inode *inode, int new, int bypass,
- u32 request_mask)
+int orangefs_inode_getattr(struct inode *inode, int flags)
{
struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
struct orangefs_kernel_op_s *new_op;
loff_t inode_size;
int ret, type;
- gossip_debug(GOSSIP_UTILS_DEBUG, "%s: called on inode %pU\n", __func__,
- get_khandle_from_ino(inode));
+ gossip_debug(GOSSIP_UTILS_DEBUG, "%s: called on inode %pU flags %d\n",
+ __func__, get_khandle_from_ino(inode), flags);
- if (!new && !bypass) {
- /*
- * Must have all the attributes in the mask and be within cache
- * time.
- */
- if ((request_mask & orangefs_inode->getattr_mask) ==
- request_mask &&
- time_before(jiffies, orangefs_inode->getattr_time))
- return 0;
+again:
+ spin_lock(&inode->i_lock);
+ /* Must have all the attributes in the mask and be within cache time. */
+ if ((!flags && time_before(jiffies, orangefs_inode->getattr_time)) ||
+ orangefs_inode->attr_valid || inode->i_state & I_DIRTY_PAGES) {
+ if (orangefs_inode->attr_valid) {
+ spin_unlock(&inode->i_lock);
+ write_inode_now(inode, 1);
+ goto again;
+ }
+ spin_unlock(&inode->i_lock);
+ return 0;
}
+ spin_unlock(&inode->i_lock);
new_op = op_alloc(ORANGEFS_VFS_OP_GETATTR);
if (!new_op)
* Size is the hardest attribute to get. The incremental cost of any
* other attribute is essentially zero.
*/
- if (request_mask & STATX_SIZE || new)
+ if (flags)
new_op->upcall.req.getattr.mask = ORANGEFS_ATTR_SYS_ALL_NOHINT;
else
new_op->upcall.req.getattr.mask =
if (ret != 0)
goto out;
- if (!new) {
+again2:
+ spin_lock(&inode->i_lock);
+ /* Must have all the attributes in the mask and be within cache time. */
+ if ((!flags && time_before(jiffies, orangefs_inode->getattr_time)) ||
+ orangefs_inode->attr_valid || inode->i_state & I_DIRTY_PAGES) {
+ if (orangefs_inode->attr_valid) {
+ spin_unlock(&inode->i_lock);
+ write_inode_now(inode, 1);
+ goto again2;
+ }
+ if (inode->i_state & I_DIRTY_PAGES) {
+ ret = 0;
+ goto out_unlock;
+ }
+ gossip_debug(GOSSIP_UTILS_DEBUG, "%s: in cache or dirty\n",
+ __func__);
+ ret = 0;
+ goto out_unlock;
+ }
+
+ if (!(flags & ORANGEFS_GETATTR_NEW)) {
ret = orangefs_inode_is_stale(inode,
&new_op->downcall.resp.getattr.attributes,
new_op->downcall.resp.getattr.link_target);
if (ret) {
ret = -ESTALE;
- goto out;
+ goto out_unlock;
}
}
case S_IFREG:
inode->i_flags = orangefs_inode_flags(&new_op->
downcall.resp.getattr.attributes);
- if (request_mask & STATX_SIZE || new) {
+ if (flags) {
inode_size = (loff_t)new_op->
downcall.resp.getattr.attributes.size;
inode->i_size = inode_size;
inode->i_blkbits = ffs(new_op->downcall.resp.getattr.
attributes.blksize);
- spin_lock(&inode->i_lock);
inode->i_bytes = inode_size;
inode->i_blocks =
(inode_size + 512 - inode_size % 512)/512;
- spin_unlock(&inode->i_lock);
}
break;
case S_IFDIR:
- if (request_mask & STATX_SIZE || new) {
+ if (flags) {
inode->i_size = PAGE_SIZE;
- spin_lock(&inode->i_lock);
inode_set_bytes(inode, inode->i_size);
- spin_unlock(&inode->i_lock);
}
set_nlink(inode, 1);
break;
case S_IFLNK:
- if (new) {
+ if (flags & ORANGEFS_GETATTR_NEW) {
inode->i_size = (loff_t)strlen(new_op->
downcall.resp.getattr.link_target);
ret = strscpy(orangefs_inode->link_target,
ORANGEFS_NAME_MAX);
if (ret == -E2BIG) {
ret = -EIO;
- goto out;
+ goto out_unlock;
}
inode->i_link = orangefs_inode->link_target;
}
/* XXX: ESTALE? This is what is done if it is not new. */
orangefs_make_bad_inode(inode);
ret = -ESTALE;
- goto out;
+ goto out_unlock;
}
inode->i_uid = make_kuid(&init_user_ns, new_op->
orangefs_inode->getattr_time = jiffies +
orangefs_getattr_timeout_msecs*HZ/1000;
- if (request_mask & STATX_SIZE || new)
- orangefs_inode->getattr_mask = STATX_BASIC_STATS;
- else
- orangefs_inode->getattr_mask = STATX_BASIC_STATS & ~STATX_SIZE;
ret = 0;
+out_unlock:
+ spin_unlock(&inode->i_lock);
out:
op_release(new_op);
return ret;
* issues a orangefs setattr request to make sure the new attribute values
* take effect if successful. returns 0 on success; -errno otherwise
*/
-int orangefs_inode_setattr(struct inode *inode, struct iattr *iattr)
+int orangefs_inode_setattr(struct inode *inode)
{
struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
struct orangefs_kernel_op_s *new_op;
if (!new_op)
return -ENOMEM;
+ spin_lock(&inode->i_lock);
+ new_op->upcall.uid = from_kuid(&init_user_ns, orangefs_inode->attr_uid);
+ new_op->upcall.gid = from_kgid(&init_user_ns, orangefs_inode->attr_gid);
new_op->upcall.req.setattr.refn = orangefs_inode->refn;
- ret = copy_attributes_from_inode(inode,
- &new_op->upcall.req.setattr.attributes,
- iattr);
- if (ret >= 0) {
- ret = service_operation(new_op, __func__,
- get_interruptible_flag(inode));
-
- gossip_debug(GOSSIP_UTILS_DEBUG,
- "orangefs_inode_setattr: returning %d\n",
- ret);
+ copy_attributes_from_inode(inode,
+ &new_op->upcall.req.setattr.attributes);
+ orangefs_inode->attr_valid = 0;
+ if (!new_op->upcall.req.setattr.attributes.mask) {
+ spin_unlock(&inode->i_lock);
+ op_release(new_op);
+ return 0;
}
+ spin_unlock(&inode->i_lock);
+
+ ret = service_operation(new_op, __func__,
+ get_interruptible_flag(inode) | ORANGEFS_OP_WRITEBACK);
+ gossip_debug(GOSSIP_UTILS_DEBUG,
+ "orangefs_inode_setattr: returning %d\n", ret);
+ if (ret)
+ orangefs_make_bad_inode(inode);
op_release(new_op);
if (ret == 0)
orangefs_inode->getattr_time = jiffies - 1;
-
return ret;
}
#include "orangefs-bufmap.h"
#include <linux/parser.h>
+#include <linux/hashtable.h>
/* a cache for orangefs-inode objects (i.e. orangefs inode private data) */
static struct kmem_cache *orangefs_inode_cache;
static void orangefs_free_inode(struct inode *inode)
{
- kmem_cache_free(orangefs_inode_cache, ORANGEFS_I(inode));
+ struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
+ struct orangefs_cached_xattr *cx;
+ struct hlist_node *tmp;
+ int i;
+
+ hash_for_each_safe(orangefs_inode->xattr_cache, i, tmp, cx, node) {
+ hlist_del(&cx->node);
+ kfree(cx);
+ }
+
+ kmem_cache_free(orangefs_inode_cache, orangefs_inode);
}
static void orangefs_destroy_inode(struct inode *inode)
__func__, orangefs_inode, get_khandle_from_ino(inode));
}
+static int orangefs_write_inode(struct inode *inode,
+ struct writeback_control *wbc)
+{
+ gossip_debug(GOSSIP_SUPER_DEBUG, "orangefs_write_inode\n");
+ return orangefs_inode_setattr(inode);
+}
+
/*
* NOTE: information filled in here is typically reflected in the
* output of the system command 'df'
.alloc_inode = orangefs_alloc_inode,
.free_inode = orangefs_free_inode,
.destroy_inode = orangefs_destroy_inode,
+ .write_inode = orangefs_write_inode,
.drop_inode = generic_delete_inode,
.statfs = orangefs_statfs,
.remount_fs = orangefs_remount_fs,
struct orangefs_fs_mount_response *fs_mount,
void *data, int silent)
{
- int ret = -EINVAL;
- struct inode *root = NULL;
- struct dentry *root_dentry = NULL;
+ int ret;
+ struct inode *root;
+ struct dentry *root_dentry;
struct orangefs_object_kref root_object;
- /* alloc and init our private orangefs sb info */
- sb->s_fs_info = kzalloc(sizeof(struct orangefs_sb_info_s), GFP_KERNEL);
- if (!ORANGEFS_SB(sb))
- return -ENOMEM;
ORANGEFS_SB(sb)->sb = sb;
ORANGEFS_SB(sb)->root_khandle = fs_mount->root_khandle;
sb->s_blocksize_bits = PAGE_SHIFT;
sb->s_maxbytes = MAX_LFS_FILESIZE;
+ ret = super_setup_bdi(sb);
+ if (ret)
+ return ret;
+
root_object.khandle = ORANGEFS_SB(sb)->root_khandle;
root_object.fs_id = ORANGEFS_SB(sb)->fs_id;
gossip_debug(GOSSIP_SUPER_DEBUG,
goto free_op;
}
+ /* alloc and init our private orangefs sb info */
+ sb->s_fs_info = kzalloc(sizeof(struct orangefs_sb_info_s), GFP_KERNEL);
+ if (!ORANGEFS_SB(sb)) {
+ d = ERR_PTR(-ENOMEM);
+ goto free_op;
+ }
+
ret = orangefs_fill_sb(sb,
&new_op->downcall.resp.fs_mount, data,
flags & SB_SILENT ? 1 : 0);
static int wait_for_matching_downcall(struct orangefs_kernel_op_s *op,
long timeout,
- bool interruptible)
+ int flags)
__acquires(op->lock);
static void orangefs_clean_up_interrupted_operation(struct orangefs_kernel_op_s *op)
__releases(op->lock);
if (!(flags & ORANGEFS_OP_NO_MUTEX))
mutex_unlock(&orangefs_request_mutex);
- ret = wait_for_matching_downcall(op, timeout,
- flags & ORANGEFS_OP_INTERRUPTIBLE);
-
+ ret = wait_for_matching_downcall(op, timeout, flags);
gossip_debug(GOSSIP_WAIT_DEBUG,
"%s: wait_for_matching_downcall returned %d for %p\n",
__func__,
*/
static int wait_for_matching_downcall(struct orangefs_kernel_op_s *op,
long timeout,
- bool interruptible)
+ int flags)
__acquires(op->lock)
{
long n;
+ int writeback = flags & ORANGEFS_OP_WRITEBACK,
+ interruptible = flags & ORANGEFS_OP_INTERRUPTIBLE;
/*
* There's a "schedule_timeout" inside of these wait
* user process that needs something done and is being
* manipulated by the client-core process.
*/
- if (interruptible)
+ if (writeback)
+ n = wait_for_completion_io_timeout(&op->waitq, timeout);
+ else if (!writeback && interruptible)
n = wait_for_completion_interruptible_timeout(&op->waitq,
- timeout);
- else
+ timeout);
+ else /* !writeback && !interruptible but compiler complains */
n = wait_for_completion_killable_timeout(&op->waitq, timeout);
spin_lock(&op->lock);
// SPDX-License-Identifier: GPL-2.0
/*
* (C) 2001 Clemson University and The University of Chicago
+ * Copyright 2018 Omnibond Systems, L.L.C.
*
* See COPYING in top-level directory.
*/
#include "orangefs-bufmap.h"
#include <linux/posix_acl_xattr.h>
#include <linux/xattr.h>
-
+#include <linux/hashtable.h>
#define SYSTEM_ORANGEFS_KEY "system.pvfs2."
#define SYSTEM_ORANGEFS_KEY_LEN 13
return internal_flag;
}
+static unsigned int xattr_key(const char *key)
+{
+ unsigned int i = 0;
+ while (key)
+ i += *key++;
+ return i % 16;
+}
+
+static struct orangefs_cached_xattr *find_cached_xattr(struct inode *inode,
+ const char *key)
+{
+ struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
+ struct orangefs_cached_xattr *cx;
+ struct hlist_head *h;
+ struct hlist_node *tmp;
+ h = &orangefs_inode->xattr_cache[xattr_key(key)];
+ if (hlist_empty(h))
+ return NULL;
+ hlist_for_each_entry_safe(cx, tmp, h, node) {
+/* if (!time_before(jiffies, cx->timeout)) {
+ hlist_del(&cx->node);
+ kfree(cx);
+ continue;
+ }*/
+ if (!strcmp(cx->key, key))
+ return cx;
+ }
+ return NULL;
+}
/*
* Tries to get a specified key's attributes of a given
{
struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
struct orangefs_kernel_op_s *new_op = NULL;
+ struct orangefs_cached_xattr *cx;
ssize_t ret = -ENOMEM;
ssize_t length = 0;
int fsuid;
down_read(&orangefs_inode->xattr_sem);
+ cx = find_cached_xattr(inode, name);
+ if (cx && time_before(jiffies, cx->timeout)) {
+ if (cx->length == -1) {
+ ret = -ENODATA;
+ goto out_unlock;
+ } else {
+ if (size == 0) {
+ ret = cx->length;
+ goto out_unlock;
+ }
+ if (cx->length > size) {
+ ret = -ERANGE;
+ goto out_unlock;
+ }
+ memcpy(buffer, cx->val, cx->length);
+ memset(buffer + cx->length, 0, size - cx->length);
+ ret = cx->length;
+ goto out_unlock;
+ }
+ }
+
new_op = op_alloc(ORANGEFS_VFS_OP_GETXATTR);
if (!new_op)
goto out_unlock;
" does not exist!\n",
get_khandle_from_ino(inode),
(char *)new_op->upcall.req.getxattr.key);
+ cx = kmalloc(sizeof *cx, GFP_KERNEL);
+ if (cx) {
+ strcpy(cx->key, name);
+ cx->length = -1;
+ cx->timeout = jiffies +
+ orangefs_getattr_timeout_msecs*HZ/1000;
+ hash_add(orangefs_inode->xattr_cache, &cx->node,
+ xattr_key(cx->key));
+ }
}
goto out_release_op;
}
ret = length;
+ if (cx) {
+ strcpy(cx->key, name);
+ memcpy(cx->val, buffer, length);
+ cx->length = length;
+ cx->timeout = jiffies + HZ;
+ } else {
+ cx = kmalloc(sizeof *cx, GFP_KERNEL);
+ if (cx) {
+ strcpy(cx->key, name);
+ memcpy(cx->val, buffer, length);
+ cx->length = length;
+ cx->timeout = jiffies + HZ;
+ hash_add(orangefs_inode->xattr_cache, &cx->node,
+ xattr_key(cx->key));
+ }
+ }
+
out_release_op:
op_release(new_op);
out_unlock:
{
struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
struct orangefs_kernel_op_s *new_op = NULL;
+ struct orangefs_cached_xattr *cx;
+ struct hlist_head *h;
+ struct hlist_node *tmp;
int ret = -ENOMEM;
if (strlen(name) >= ORANGEFS_MAX_XATTR_NAMELEN)
"orangefs_inode_removexattr: returning %d\n", ret);
op_release(new_op);
+
+ h = &orangefs_inode->xattr_cache[xattr_key(name)];
+ hlist_for_each_entry_safe(cx, tmp, h, node) {
+ if (!strcmp(cx->key, name)) {
+ hlist_del(&cx->node);
+ kfree(cx);
+ break;
+ }
+ }
+
out_unlock:
up_write(&orangefs_inode->xattr_sem);
return ret;
struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
struct orangefs_kernel_op_s *new_op;
int internal_flag = 0;
+ struct orangefs_cached_xattr *cx;
+ struct hlist_head *h;
+ struct hlist_node *tmp;
int ret = -ENOMEM;
gossip_debug(GOSSIP_XATTR_DEBUG,
/* when request is serviced properly, free req op struct */
op_release(new_op);
+
+ h = &orangefs_inode->xattr_cache[xattr_key(name)];
+ hlist_for_each_entry_safe(cx, tmp, h, node) {
+ if (!strcmp(cx->key, name)) {
+ hlist_del(&cx->node);
+ kfree(cx);
+ break;
+ }
+ }
+
out_unlock:
up_write(&orangefs_inode->xattr_sem);
return ret;
struct drm_private_state * __must_check
drm_atomic_get_private_obj_state(struct drm_atomic_state *state,
struct drm_private_obj *obj);
+struct drm_private_state *
+drm_atomic_get_old_private_obj_state(struct drm_atomic_state *state,
+ struct drm_private_obj *obj);
+struct drm_private_state *
+drm_atomic_get_new_private_obj_state(struct drm_atomic_state *state,
+ struct drm_private_obj *obj);
/**
* drm_atomic_get_existing_crtc_state - get crtc state, if it exists
* @get_power: get the POWER_DOMAIN_AUDIO power well
*
* Request the power well to be turned on.
+ *
+ * Returns a wakeref cookie to be passed back to the corresponding
+ * call to @put_power.
*/
- void (*get_power)(struct device *);
+ unsigned long (*get_power)(struct device *);
/**
* @put_power: put the POWER_DOMAIN_AUDIO power well
*
* Allow the power well to be turned off.
*/
- void (*put_power)(struct device *);
+ void (*put_power)(struct device *, unsigned long);
/**
* @codec_wake_override: Enable/disable codec wake signal
*/
*
* @refcount: Refcount for this master object.
* @dev: Link back to the DRM device
- * @lock: DRI1 lock information.
* @driver_priv: Pointer to driver-private information.
* @lessor: Lease holder
* @lessee_id: id for lessees. Owners always have id 0
* &drm_device.master_mutex.
*/
struct idr magic_map;
- struct drm_lock_data lock;
void *driver_priv;
/* Tree of display resource leases, each of which is a drm_master struct
struct list_head lessees;
struct idr leases;
struct idr lessee_idr;
+ /* private: */
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
+ struct drm_lock_data lock;
+#endif
};
struct drm_master *drm_master_get(struct drm_master *master);
*/
struct drm_bridge_timings {
/**
- * @sampling_edge:
+ * @input_bus_flags:
*
- * Tells whether the bridge samples the digital input signal
- * from the display engine on the positive or negative edge of the
- * clock, this should reuse the DRM_BUS_FLAG_PIXDATA_[POS|NEG]EDGE
- * bitwise flags from the DRM connector (bit 2 and 3 valid).
+ * Tells what additional settings for the pixel data on the bus
+ * this bridge requires (like pixel signal polarity). See also
+ * &drm_display_info->bus_flags.
*/
- u32 sampling_edge;
+ u32 input_bus_flags;
/**
* @setup_time_ps:
*
void drm_clflush_pages(struct page *pages[], unsigned long num_pages);
void drm_clflush_sg(struct sg_table *st);
void drm_clflush_virt_range(void *addr, unsigned long length);
-u64 drm_get_max_iomem(void);
+bool drm_need_swiotlb(int dma_bits);
static inline bool drm_arch_can_wc_memory(void)
int drm_client_init(struct drm_device *dev, struct drm_client_dev *client,
const char *name, const struct drm_client_funcs *funcs);
void drm_client_release(struct drm_client_dev *client);
-void drm_client_add(struct drm_client_dev *client);
+void drm_client_register(struct drm_client_dev *client);
void drm_client_dev_unregister(struct drm_device *dev);
void drm_client_dev_hotplug(struct drm_device *dev);
DRM_MODE_PANEL_ORIENTATION_RIGHT_UP,
};
+/*
+ * This is a consolidated colorimetry list supported by HDMI and
+ * DP protocol standard. The respective connectors will register
+ * a property with the subset of this list (supported by that
+ * respective protocol). Userspace will set the colorspace through
+ * a colorspace property which will be created and exposed to
+ * userspace.
+ */
+
+/* For Default case, driver will set the colorspace */
+#define DRM_MODE_COLORIMETRY_DEFAULT 0
+/* CEA 861 Normal Colorimetry options */
+#define DRM_MODE_COLORIMETRY_NO_DATA 0
+#define DRM_MODE_COLORIMETRY_SMPTE_170M_YCC 1
+#define DRM_MODE_COLORIMETRY_BT709_YCC 2
+/* CEA 861 Extended Colorimetry Options */
+#define DRM_MODE_COLORIMETRY_XVYCC_601 3
+#define DRM_MODE_COLORIMETRY_XVYCC_709 4
+#define DRM_MODE_COLORIMETRY_SYCC_601 5
+#define DRM_MODE_COLORIMETRY_OPYCC_601 6
+#define DRM_MODE_COLORIMETRY_OPRGB 7
+#define DRM_MODE_COLORIMETRY_BT2020_CYCC 8
+#define DRM_MODE_COLORIMETRY_BT2020_RGB 9
+#define DRM_MODE_COLORIMETRY_BT2020_YCC 10
+/* Additional Colorimetry extension added as part of CTA 861.G */
+#define DRM_MODE_COLORIMETRY_DCI_P3_RGB_D65 11
+#define DRM_MODE_COLORIMETRY_DCI_P3_RGB_THEATER 12
+
+/**
+ * enum drm_bus_flags - bus_flags info for &drm_display_info
+ *
+ * This enum defines signal polarities and clock edge information for signals on
+ * a bus as bitmask flags.
+ *
+ * The clock edge information is conveyed by two sets of symbols,
+ * DRM_BUS_FLAGS_*_DRIVE_\* and DRM_BUS_FLAGS_*_SAMPLE_\*. When this enum is
+ * used to describe a bus from the point of view of the transmitter, the
+ * \*_DRIVE_\* flags should be used. When used from the point of view of the
+ * receiver, the \*_SAMPLE_\* flags should be used. The \*_DRIVE_\* and
+ * \*_SAMPLE_\* flags alias each other, with the \*_SAMPLE_POSEDGE and
+ * \*_SAMPLE_NEGEDGE flags being equal to \*_DRIVE_NEGEDGE and \*_DRIVE_POSEDGE
+ * respectively. This simplifies code as signals are usually sampled on the
+ * opposite edge of the driving edge. Transmitters and receivers may however
+ * need to take other signal timings into account to convert between driving
+ * and sample edges.
+ *
+ * @DRM_BUS_FLAG_DE_LOW: The Data Enable signal is active low
+ * @DRM_BUS_FLAG_DE_HIGH: The Data Enable signal is active high
+ * @DRM_BUS_FLAG_PIXDATA_POSEDGE: Legacy value, do not use
+ * @DRM_BUS_FLAG_PIXDATA_NEGEDGE: Legacy value, do not use
+ * @DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE: Data is driven on the rising edge of
+ * the pixel clock
+ * @DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE: Data is driven on the falling edge of
+ * the pixel clock
+ * @DRM_BUS_FLAG_PIXDATA_SAMPLE_POSEDGE: Data is sampled on the rising edge of
+ * the pixel clock
+ * @DRM_BUS_FLAG_PIXDATA_SAMPLE_NEGEDGE: Data is sampled on the falling edge of
+ * the pixel clock
+ * @DRM_BUS_FLAG_DATA_MSB_TO_LSB: Data is transmitted MSB to LSB on the bus
+ * @DRM_BUS_FLAG_DATA_LSB_TO_MSB: Data is transmitted LSB to MSB on the bus
+ * @DRM_BUS_FLAG_SYNC_POSEDGE: Legacy value, do not use
+ * @DRM_BUS_FLAG_SYNC_NEGEDGE: Legacy value, do not use
+ * @DRM_BUS_FLAG_SYNC_DRIVE_POSEDGE: Sync signals are driven on the rising
+ * edge of the pixel clock
+ * @DRM_BUS_FLAG_SYNC_DRIVE_NEGEDGE: Sync signals are driven on the falling
+ * edge of the pixel clock
+ * @DRM_BUS_FLAG_SYNC_SAMPLE_POSEDGE: Sync signals are sampled on the rising
+ * edge of the pixel clock
+ * @DRM_BUS_FLAG_SYNC_SAMPLE_NEGEDGE: Sync signals are sampled on the falling
+ * edge of the pixel clock
+ */
+enum drm_bus_flags {
+ DRM_BUS_FLAG_DE_LOW = BIT(0),
+ DRM_BUS_FLAG_DE_HIGH = BIT(1),
+ DRM_BUS_FLAG_PIXDATA_POSEDGE = BIT(2),
+ DRM_BUS_FLAG_PIXDATA_NEGEDGE = BIT(3),
+ DRM_BUS_FLAG_PIXDATA_DRIVE_POSEDGE = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ DRM_BUS_FLAG_PIXDATA_DRIVE_NEGEDGE = DRM_BUS_FLAG_PIXDATA_NEGEDGE,
+ DRM_BUS_FLAG_PIXDATA_SAMPLE_POSEDGE = DRM_BUS_FLAG_PIXDATA_NEGEDGE,
+ DRM_BUS_FLAG_PIXDATA_SAMPLE_NEGEDGE = DRM_BUS_FLAG_PIXDATA_POSEDGE,
+ DRM_BUS_FLAG_DATA_MSB_TO_LSB = BIT(4),
+ DRM_BUS_FLAG_DATA_LSB_TO_MSB = BIT(5),
+ DRM_BUS_FLAG_SYNC_POSEDGE = BIT(6),
+ DRM_BUS_FLAG_SYNC_NEGEDGE = BIT(7),
+ DRM_BUS_FLAG_SYNC_DRIVE_POSEDGE = DRM_BUS_FLAG_SYNC_POSEDGE,
+ DRM_BUS_FLAG_SYNC_DRIVE_NEGEDGE = DRM_BUS_FLAG_SYNC_NEGEDGE,
+ DRM_BUS_FLAG_SYNC_SAMPLE_POSEDGE = DRM_BUS_FLAG_SYNC_NEGEDGE,
+ DRM_BUS_FLAG_SYNC_SAMPLE_NEGEDGE = DRM_BUS_FLAG_SYNC_POSEDGE,
+};
+
/**
* struct drm_display_info - runtime data about the connected sink
*
* drm_add_edid_modes().
*/
struct drm_display_info {
- /**
- * @name: Name of the display.
- */
- char name[DRM_DISPLAY_INFO_LEN];
-
/**
* @width_mm: Physical width in mm.
*/
- unsigned int width_mm;
+ unsigned int width_mm;
+
/**
* @height_mm: Physical height in mm.
*/
unsigned int height_mm;
- /**
- * @pixel_clock: Maximum pixel clock supported by the sink, in units of
- * 100Hz. This mismatches the clock in &drm_display_mode (which is in
- * kHZ), because that's what the EDID uses as base unit.
- */
- unsigned int pixel_clock;
/**
* @bpc: Maximum bits per color channel. Used by HDMI and DP outputs.
*/
*/
unsigned int num_bus_formats;
-#define DRM_BUS_FLAG_DE_LOW (1<<0)
-#define DRM_BUS_FLAG_DE_HIGH (1<<1)
-/* drive data on pos. edge */
-#define DRM_BUS_FLAG_PIXDATA_POSEDGE (1<<2)
-/* drive data on neg. edge */
-#define DRM_BUS_FLAG_PIXDATA_NEGEDGE (1<<3)
-/* data is transmitted MSB to LSB on the bus */
-#define DRM_BUS_FLAG_DATA_MSB_TO_LSB (1<<4)
-/* data is transmitted LSB to MSB on the bus */
-#define DRM_BUS_FLAG_DATA_LSB_TO_MSB (1<<5)
-/* drive sync on pos. edge */
-#define DRM_BUS_FLAG_SYNC_POSEDGE (1<<6)
-/* drive sync on neg. edge */
-#define DRM_BUS_FLAG_SYNC_NEGEDGE (1<<7)
-
/**
* @bus_flags: Additional information (like pixel signal polarity) for
- * the pixel data on the bus, using DRM_BUS_FLAGS\_ defines.
+ * the pixel data on the bus, using &enum drm_bus_flags values
+ * DRM_BUS_FLAGS\_.
*/
u32 bus_flags;
*/
unsigned int content_protection;
+ /**
+ * @colorspace: State variable for Connector property to request
+ * colorspace change on Sink. This is most commonly used to switch
+ * to wider color gamuts like BT2020.
+ */
+ u32 colorspace;
+
/**
* @writeback_job: Writeback job for writeback connectors
*
*/
struct drm_property *content_protection_property;
+ /**
+ * @colorspace_property: Connector property to set the suitable
+ * colorspace supported by the sink.
+ */
+ struct drm_property *colorspace_property;
+
/**
* @path_blob_ptr:
*
int drm_connector_attach_content_protection_property(
struct drm_connector *connector);
int drm_mode_create_aspect_ratio_property(struct drm_device *dev);
+int drm_mode_create_colorspace_property(struct drm_connector *connector);
int drm_mode_create_content_type_property(struct drm_device *dev);
void drm_hdmi_avi_infoframe_content_type(struct hdmi_avi_infoframe *frame,
const struct drm_connector_state *conn_state);
/**
* struct drm_crtc_state - mutable CRTC state
*
- * Note that the distinction between @enable and @active is rather subtile:
+ * Note that the distinction between @enable and @active is rather subtle:
* Flipping @active while @enable is set without changing anything else may
* never return in a failure from the &drm_mode_config_funcs.atomic_check
* callback. Userspace assumes that a DPMS On will always succeed. In other
/**
* @destroy:
*
- * Clean up plane resources. This is only called at driver unload time
+ * Clean up CRTC resources. This is only called at driver unload time
* through drm_mode_config_cleanup() since a CRTC cannot be hotplugged
* in DRM.
*/
/* Everything below here is for legacy driver, never use! */
/* private: */
-
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
/* Context handle management - linked list of context handles */
struct list_head ctxlist;
/* Scatter gather memory */
struct drm_sg_mem *sg;
+#endif
};
#endif
* submission.
*/
DRIVER_SYNCOBJ = BIT(5),
+ /**
+ * @DRIVER_SYNCOBJ_TIMELINE:
+ *
+ * Driver supports the timeline flavor of &drm_syncobj for explicit
+ * synchronization of command submission.
+ */
+ DRIVER_SYNCOBJ_TIMELINE = BIT(6),
/* IMPORTANT: Below are all the legacy flags, add new ones above. */
/**
* struct drm_driver - DRM driver structure
*
- * This structure represent the common code for a family of cards. There will
- * one drm_device for each card present in this family. It contains lots of
- * vfunc entries, and a pile of those probably should be moved to more
+ * This structure represent the common code for a family of cards. There will be
+ * one &struct drm_device for each card present in this family. It contains lots
+ * of vfunc entries, and a pile of those probably should be moved to more
* appropriate places like &drm_mode_config_funcs or into a new operations
* structure for GEM drivers.
*/
int drm_dev_init(struct drm_device *dev,
struct drm_driver *driver,
struct device *parent);
+int devm_drm_dev_init(struct device *parent,
+ struct drm_device *dev,
+ struct drm_driver *driver);
void drm_dev_fini(struct drm_device *dev);
struct drm_device *drm_dev_alloc(struct drm_driver *driver,
*/
u16 slice_height;
/**
- * @enable422: True for 4_2_2 sampling, false for 4_4_4 sampling
+ * @simple_422: True if simple 4_2_2 mode is enabled else False
*/
- bool enable422;
+ bool simple_422;
/**
* @pic_width: Width of the input display frame in pixels
*/
struct drm_dsc_picture_parameter_set pps_payload;
} __packed;
-void drm_dsc_dp_pps_header_init(struct drm_dsc_pps_infoframe *pps_sdp);
-void drm_dsc_pps_infoframe_pack(struct drm_dsc_pps_infoframe *pps_sdp,
+void drm_dsc_dp_pps_header_init(struct dp_sdp_header *pps_header);
+void drm_dsc_pps_payload_pack(struct drm_dsc_picture_parameter_set *pps_sdp,
const struct drm_dsc_config *dsc_cfg);
+int drm_dsc_compute_rc_parameters(struct drm_dsc_config *vdsc_cfg);
#endif /* _DRM_DSC_H_ */
struct drm_encoder;
struct drm_connector;
+struct drm_connector_state;
struct drm_display_mode;
int drm_edid_to_sad(struct edid *edid, struct cea_sad **sads);
drm_hdmi_vendor_infoframe_from_display_mode(struct hdmi_vendor_infoframe *frame,
struct drm_connector *connector,
const struct drm_display_mode *mode);
+
+void
+drm_hdmi_avi_infoframe_colorspace(struct hdmi_avi_infoframe *frame,
+ const struct drm_connector_state *conn_state);
+
void
drm_hdmi_avi_infoframe_quant_range(struct hdmi_avi_infoframe *frame,
struct drm_connector *connector,
* according to the largest width/height (so it is large enough for all CRTCs
* to scanout). But the fbdev width/height is sized to the minimum width/
* height of all the displays. This ensures that fbcon fits on the smallest
- * of the attached displays.
- *
- * So what is passed to drm_fb_helper_fill_var() should be fb_width/fb_height,
- * rather than the surface size.
+ * of the attached displays. fb_width/fb_height is used by
+ * drm_fb_helper_fill_info() to fill out the &fb_info.var structure.
*/
struct drm_fb_helper_surface_size {
u32 fb_width;
*/
int (*fb_probe)(struct drm_fb_helper *helper,
struct drm_fb_helper_surface_size *sizes);
-
- /**
- * @initial_config:
- *
- * Driver callback to setup an initial fbdev display configuration.
- * Drivers can use this callback to tell the fbdev emulation what the
- * preferred initial configuration is. This is useful to implement
- * smooth booting where the fbdev (and subsequently all userspace) never
- * changes the mode, but always inherits the existing configuration.
- *
- * This callback is optional.
- *
- * RETURNS:
- *
- * The driver should return true if a suitable initial configuration has
- * been filled out and false when the fbdev helper should fall back to
- * the default probing logic.
- */
- bool (*initial_config)(struct drm_fb_helper *fb_helper,
- struct drm_fb_helper_crtc **crtcs,
- struct drm_display_mode **modes,
- struct drm_fb_offset *offsets,
- bool *enabled, int width, int height);
};
struct drm_fb_helper_connector {
struct fb_info *drm_fb_helper_alloc_fbi(struct drm_fb_helper *fb_helper);
void drm_fb_helper_unregister_fbi(struct drm_fb_helper *fb_helper);
-void drm_fb_helper_fill_var(struct fb_info *info, struct drm_fb_helper *fb_helper,
- uint32_t fb_width, uint32_t fb_height);
-void drm_fb_helper_fill_fix(struct fb_info *info, uint32_t pitch,
- uint32_t depth);
+void drm_fb_helper_fill_info(struct fb_info *info,
+ struct drm_fb_helper *fb_helper,
+ struct drm_fb_helper_surface_size *sizes);
void drm_fb_helper_unlink_fbi(struct drm_fb_helper *fb_helper);
{
}
-static inline void drm_fb_helper_fill_var(struct fb_info *info,
- struct drm_fb_helper *fb_helper,
- uint32_t fb_width, uint32_t fb_height)
-{
-}
-
-static inline void drm_fb_helper_fill_fix(struct fb_info *info, uint32_t pitch,
- uint32_t depth)
+static inline void
+drm_fb_helper_fill_info(struct fb_info *info,
+ struct drm_fb_helper *fb_helper,
+ struct drm_fb_helper_surface_size *sizes)
{
}
struct drm_prime_file_private prime;
/* private: */
+#if IS_ENABLED(CONFIG_DRM_LEGACY)
unsigned long lock_count; /* DRI1 legacy lock count */
+#endif
};
/**
--- /dev/null
+/*
+ * Copyright (C) 2016 Noralf Trønnes
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#ifndef __LINUX_DRM_FORMAT_HELPER_H
+#define __LINUX_DRM_FORMAT_HELPER_H
+
+struct drm_framebuffer;
+struct drm_rect;
+
+void drm_fb_memcpy(void *dst, void *vaddr, struct drm_framebuffer *fb,
+ struct drm_rect *clip);
+void drm_fb_memcpy_dstclip(void __iomem *dst, void *vaddr,
+ struct drm_framebuffer *fb,
+ struct drm_rect *clip);
+void drm_fb_swab16(u16 *dst, void *vaddr, struct drm_framebuffer *fb,
+ struct drm_rect *clip);
+void drm_fb_xrgb8888_to_rgb565(void *dst, void *vaddr,
+ struct drm_framebuffer *fb,
+ struct drm_rect *clip, bool swab);
+void drm_fb_xrgb8888_to_rgb565_dstclip(void __iomem *dst, unsigned int dst_pitch,
+ void *vaddr, struct drm_framebuffer *fb,
+ struct drm_rect *clip, bool swab);
+void drm_fb_xrgb8888_to_rgb888_dstclip(void __iomem *dst, unsigned int dst_pitch,
+ void *vaddr, struct drm_framebuffer *fb,
+ struct drm_rect *clip);
+void drm_fb_xrgb8888_to_gray8(u8 *dst, void *vaddr, struct drm_framebuffer *fb,
+ struct drm_rect *clip);
+
+#endif /* __LINUX_DRM_FORMAT_HELPER_H */
struct drm_clip_rect;
struct drm_device;
struct drm_file;
+struct drm_format_info;
struct drm_framebuffer;
struct drm_gem_object;
*/
#include <linux/kref.h>
+#include <linux/reservation.h>
#include <drm/drm_vma_manager.h>
*/
struct dma_buf_attachment *import_attach;
+ /**
+ * @resv:
+ *
+ * Pointer to reservation object associated with the this GEM object.
+ *
+ * Normally (@resv == &@_resv) except for imported GEM objects.
+ */
+ struct reservation_object *resv;
+
+ /**
+ * @_resv:
+ *
+ * A reservation object for this GEM object.
+ *
+ * This is unused for imported GEM objects.
+ */
+ struct reservation_object _resv;
+
/**
* @funcs:
*
void drm_gem_put_pages(struct drm_gem_object *obj, struct page **pages,
bool dirty, bool accessed);
+int drm_gem_objects_lookup(struct drm_file *filp, void __user *bo_handles,
+ int count, struct drm_gem_object ***objs_out);
struct drm_gem_object *drm_gem_object_lookup(struct drm_file *filp, u32 handle);
+long drm_gem_reservation_object_wait(struct drm_file *filep, u32 handle,
+ bool wait_all, unsigned long timeout);
+int drm_gem_lock_reservations(struct drm_gem_object **objs, int count,
+ struct ww_acquire_ctx *acquire_ctx);
+void drm_gem_unlock_reservations(struct drm_gem_object **objs, int count,
+ struct ww_acquire_ctx *acquire_ctx);
+int drm_gem_fence_array_add(struct xarray *fence_array,
+ struct dma_fence *fence);
+int drm_gem_fence_array_add_implicit(struct xarray *fence_array,
+ struct drm_gem_object *obj,
+ bool write);
int drm_gem_dumb_map_offset(struct drm_file *file, struct drm_device *dev,
u32 handle, u64 *offset);
int drm_gem_dumb_destroy(struct drm_file *file,
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __DRM_GEM_SHMEM_HELPER_H__
+#define __DRM_GEM_SHMEM_HELPER_H__
+
+#include <linux/fs.h>
+#include <linux/mm.h>
+#include <linux/mutex.h>
+
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_ioctl.h>
+#include <drm/drm_prime.h>
+
+struct dma_buf_attachment;
+struct drm_mode_create_dumb;
+struct drm_printer;
+struct sg_table;
+
+/**
+ * struct drm_gem_shmem_object - GEM object backed by shmem
+ */
+struct drm_gem_shmem_object {
+ /**
+ * @base: Base GEM object
+ */
+ struct drm_gem_object base;
+
+ /**
+ * @pages_lock: Protects the page table and use count
+ */
+ struct mutex pages_lock;
+
+ /**
+ * @pages: Page table
+ */
+ struct page **pages;
+
+ /**
+ * @pages_use_count:
+ *
+ * Reference count on the pages table.
+ * The pages are put when the count reaches zero.
+ */
+ unsigned int pages_use_count;
+
+ /**
+ * @pages_mark_dirty_on_put:
+ *
+ * Mark pages as dirty when they are put.
+ */
+ unsigned int pages_mark_dirty_on_put : 1;
+
+ /**
+ * @pages_mark_accessed_on_put:
+ *
+ * Mark pages as accessed when they are put.
+ */
+ unsigned int pages_mark_accessed_on_put : 1;
+
+ /**
+ * @sgt: Scatter/gather table for imported PRIME buffers
+ */
+ struct sg_table *sgt;
+
+ /**
+ * @vmap_lock: Protects the vmap address and use count
+ */
+ struct mutex vmap_lock;
+
+ /**
+ * @vaddr: Kernel virtual address of the backing memory
+ */
+ void *vaddr;
+
+ /**
+ * @vmap_use_count:
+ *
+ * Reference count on the virtual address.
+ * The address are un-mapped when the count reaches zero.
+ */
+ unsigned int vmap_use_count;
+};
+
+#define to_drm_gem_shmem_obj(obj) \
+ container_of(obj, struct drm_gem_shmem_object, base)
+
+/**
+ * DEFINE_DRM_GEM_SHMEM_FOPS() - Macro to generate file operations for shmem drivers
+ * @name: name for the generated structure
+ *
+ * This macro autogenerates a suitable &struct file_operations for shmem based
+ * drivers, which can be assigned to &drm_driver.fops. Note that this structure
+ * cannot be shared between drivers, because it contains a reference to the
+ * current module using THIS_MODULE.
+ *
+ * Note that the declaration is already marked as static - if you need a
+ * non-static version of this you're probably doing it wrong and will break the
+ * THIS_MODULE reference by accident.
+ */
+#define DEFINE_DRM_GEM_SHMEM_FOPS(name) \
+ static const struct file_operations name = {\
+ .owner = THIS_MODULE,\
+ .open = drm_open,\
+ .release = drm_release,\
+ .unlocked_ioctl = drm_ioctl,\
+ .compat_ioctl = drm_compat_ioctl,\
+ .poll = drm_poll,\
+ .read = drm_read,\
+ .llseek = noop_llseek,\
+ .mmap = drm_gem_shmem_mmap, \
+ }
+
+struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t size);
+void drm_gem_shmem_free_object(struct drm_gem_object *obj);
+
+int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem);
+void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_pin(struct drm_gem_object *obj);
+void drm_gem_shmem_unpin(struct drm_gem_object *obj);
+void *drm_gem_shmem_vmap(struct drm_gem_object *obj);
+void drm_gem_shmem_vunmap(struct drm_gem_object *obj, void *vaddr);
+
+struct drm_gem_shmem_object *
+drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
+ struct drm_device *dev, size_t size,
+ uint32_t *handle);
+int drm_gem_shmem_dumb_create(struct drm_file *file, struct drm_device *dev,
+ struct drm_mode_create_dumb *args);
+
+int drm_gem_shmem_mmap(struct file *filp, struct vm_area_struct *vma);
+
+extern const struct vm_operations_struct drm_gem_shmem_vm_ops;
+
+void drm_gem_shmem_print_info(struct drm_printer *p, unsigned int indent,
+ const struct drm_gem_object *obj);
+
+struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_object *obj);
+struct drm_gem_object *
+drm_gem_shmem_prime_import_sg_table(struct drm_device *dev,
+ struct dma_buf_attachment *attach,
+ struct sg_table *sgt);
+
+struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_object *obj);
+
+/**
+ * DRM_GEM_SHMEM_DRIVER_OPS - Default shmem GEM operations
+ *
+ * This macro provides a shortcut for setting the shmem GEM operations in
+ * the &drm_driver structure.
+ */
+#define DRM_GEM_SHMEM_DRIVER_OPS \
+ .prime_handle_to_fd = drm_gem_prime_handle_to_fd, \
+ .prime_fd_to_handle = drm_gem_prime_fd_to_handle, \
+ .gem_prime_import_sg_table = drm_gem_shmem_prime_import_sg_table, \
+ .gem_prime_mmap = drm_gem_prime_mmap, \
+ .dumb_create = drm_gem_shmem_dumb_create
+
+#endif /* __DRM_GEM_SHMEM_HELPER_H__ */
/* Period of hdcp checks (to ensure we're still authenticated) */
#define DRM_HDCP_CHECK_PERIOD_MS (128 * 16)
+#define DRM_HDCP2_CHECK_PERIOD_MS 500
/* Shared lengths/masks between HDMI/DVI/DisplayPort */
#define DRM_HDCP_AN_LEN 8
#define HDCP_2_2_REP_SEND_ACK 15
#define HDCP_2_2_REP_STREAM_MANAGE 16
#define HDCP_2_2_REP_STREAM_READY 17
-#define HDCP_2_2_ERRATA_DP_STREAM_TYPE 50
#define HDCP_2_2_RTX_LEN 8
#define HDCP_2_2_RRX_LEN 8
u8 m_prime[HDCP_2_2_MPRIME_LEN];
} __packed;
-struct hdcp2_dp_errata_stream_type {
- u8 msg_id;
- u8 stream_type;
-} __packed;
-
/* HDCP2.2 TIMEOUTs in mSec */
#define HDCP_2_2_CERT_TIMEOUT_MS 100
#define HDCP_2_2_HPRIME_NO_PAIRED_TIMEOUT_MS 1000
struct drm_local_map *drm_legacy_findmap(struct drm_device *dev, unsigned int token);
void drm_legacy_rmmap(struct drm_device *d, struct drm_local_map *map);
int drm_legacy_rmmap_locked(struct drm_device *d, struct drm_local_map *map);
-void drm_legacy_master_rmmaps(struct drm_device *dev,
- struct drm_master *master);
struct drm_local_map *drm_legacy_getsarea(struct drm_device *dev);
int drm_legacy_mmap(struct file *filp, struct vm_area_struct *vma);
.vsync_start = (vss), .vsync_end = (vse), .vtotal = (vt), \
.vscan = (vs), .flags = (f)
+/**
+ * DRM_SIMPLE_MODE - Simple display mode
+ * @hd: Horizontal resolution, width
+ * @vd: Vertical resolution, height
+ * @hd_mm: Display width in millimeters
+ * @vd_mm: Display height in millimeters
+ *
+ * This macro initializes a &drm_display_mode that only contains info about
+ * resolution and physical size.
+ */
+#define DRM_SIMPLE_MODE(hd, vd, hd_mm, vd_mm) \
+ .type = DRM_MODE_TYPE_DRIVER, .clock = 1 /* pass validation */, \
+ .hdisplay = (hd), .hsync_start = (hd), .hsync_end = (hd), \
+ .htotal = (hd), .vdisplay = (vd), .vsync_start = (vd), \
+ .vsync_end = (vd), .vtotal = (vd), .width_mm = (hd_mm), \
+ .height_mm = (vd_mm)
+
#define CRTC_INTERLACE_HALVE_V (1 << 0) /* halve V values for interlacing */
#define CRTC_STEREO_DOUBLE (1 << 1) /* adjust timings for stereo modes */
#define CRTC_NO_DBLSCAN (1 << 2) /* don't adjust doublescan */
*/
enum mode_set_atomic;
+struct drm_writeback_connector;
+struct drm_writeback_job;
/**
* struct drm_crtc_helper_funcs - helper operations for CRTCs
*/
void (*atomic_commit)(struct drm_connector *connector,
struct drm_connector_state *state);
+
+ int (*prepare_writeback_job)(struct drm_writeback_connector *connector,
+ struct drm_writeback_job *job);
+ void (*cleanup_writeback_job)(struct drm_writeback_connector *connector,
+ struct drm_writeback_job *job);
};
/**
#include <linux/printk.h>
#include <linux/seq_file.h>
#include <linux/device.h>
+#include <linux/debugfs.h>
/**
* DOC: print
__printf(2, 3)
void drm_printf(struct drm_printer *p, const char *f, ...);
void drm_puts(struct drm_printer *p, const char *str);
+void drm_print_regset32(struct drm_printer *p, struct debugfs_regset32 *regset);
__printf(2, 0)
/**
#define __DRM_SYNCOBJ_H__
#include <linux/dma-fence.h>
+#include <linux/dma-fence-chain.h>
struct drm_file;
struct drm_syncobj *drm_syncobj_find(struct drm_file *file_private,
u32 handle);
+void drm_syncobj_add_point(struct drm_syncobj *syncobj,
+ struct dma_fence_chain *chain,
+ struct dma_fence *fence,
+ uint64_t point);
void drm_syncobj_replace_fence(struct drm_syncobj *syncobj,
struct dma_fence *fence);
int drm_syncobj_find_fence(struct drm_file *file_private,
#ifndef __DRM_UTILS_H__
#define __DRM_UTILS_H__
+#include <linux/types.h>
+
int drm_get_panel_orientation_quirk(int width, int height);
+signed long drm_timeout_abs_to_jiffies(int64_t timeout_nsec);
+
#endif
#include <linux/spinlock.h>
#include <linux/types.h>
+/* We make up offsets for buffer objects so we can recognize them at
+ * mmap time. pgoff in mmap is an unsigned long, so we need to make sure
+ * that the faked up offset will fit
+ */
+#if BITS_PER_LONG == 64
+#define DRM_FILE_PAGE_OFFSET_START ((0xFFFFFFFFUL >> PAGE_SHIFT) + 1)
+#define DRM_FILE_PAGE_OFFSET_SIZE ((0xFFFFFFFFUL >> PAGE_SHIFT) * 256)
+#else
+#define DRM_FILE_PAGE_OFFSET_START ((0xFFFFFFFUL >> PAGE_SHIFT) + 1)
+#define DRM_FILE_PAGE_OFFSET_SIZE ((0xFFFFFFFUL >> PAGE_SHIFT) * 16)
+#endif
+
struct drm_file;
struct drm_vma_offset_file {
};
struct drm_writeback_job {
+ /**
+ * @connector:
+ *
+ * Back-pointer to the writeback connector associated with the job
+ */
+ struct drm_writeback_connector *connector;
+
+ /**
+ * @prepared:
+ *
+ * Set when the job has been prepared with drm_writeback_prepare_job()
+ */
+ bool prepared;
+
/**
* @cleanup_work:
*
* @fb:
*
* Framebuffer to be written to by the writeback connector. Do not set
- * directly, use drm_atomic_set_writeback_fb_for_connector()
+ * directly, use drm_writeback_set_fb()
*/
struct drm_framebuffer *fb;
* Fence which will signal once the writeback has completed
*/
struct dma_fence *out_fence;
+
+ /**
+ * @priv:
+ *
+ * Driver-private data
+ */
+ void *priv;
};
static inline struct drm_writeback_connector *
const struct drm_encoder_helper_funcs *enc_helper_funcs,
const u32 *formats, int n_formats);
+int drm_writeback_set_fb(struct drm_connector_state *conn_state,
+ struct drm_framebuffer *fb);
+
+int drm_writeback_prepare_job(struct drm_writeback_job *job);
+
void drm_writeback_queue_job(struct drm_writeback_connector *wb_connector,
- struct drm_writeback_job *job);
+ struct drm_connector_state *conn_state);
void drm_writeback_cleanup_job(struct drm_writeback_job *job);
INTEL_VGA_DEVICE(0x2e42, info), /* B43_G */ \
INTEL_VGA_DEVICE(0x2e92, info) /* B43_G.1 */
-#define INTEL_PINEVIEW_IDS(info) \
- INTEL_VGA_DEVICE(0xa001, info), \
+#define INTEL_PINEVIEW_G_IDS(info) \
+ INTEL_VGA_DEVICE(0xa001, info)
+
+#define INTEL_PINEVIEW_M_IDS(info) \
INTEL_VGA_DEVICE(0xa011, info)
#define INTEL_IRONLAKE_D_IDS(info) \
#define INTEL_IVB_Q_IDS(info) \
INTEL_QUANTA_VGA_DEVICE(info) /* Quanta transcode */
+#define INTEL_HSW_ULT_GT1_IDS(info) \
+ INTEL_VGA_DEVICE(0x0A02, info), /* ULT GT1 desktop */ \
+ INTEL_VGA_DEVICE(0x0A0A, info), /* ULT GT1 server */ \
+ INTEL_VGA_DEVICE(0x0A0B, info), /* ULT GT1 reserved */ \
+ INTEL_VGA_DEVICE(0x0A06, info) /* ULT GT1 mobile */
+
+#define INTEL_HSW_ULX_GT1_IDS(info) \
+ INTEL_VGA_DEVICE(0x0A0E, info) /* ULX GT1 mobile */
+
#define INTEL_HSW_GT1_IDS(info) \
+ INTEL_HSW_ULT_GT1_IDS(info), \
+ INTEL_HSW_ULX_GT1_IDS(info), \
INTEL_VGA_DEVICE(0x0402, info), /* GT1 desktop */ \
INTEL_VGA_DEVICE(0x040a, info), /* GT1 server */ \
INTEL_VGA_DEVICE(0x040B, info), /* GT1 reserved */ \
INTEL_VGA_DEVICE(0x0C0A, info), /* SDV GT1 server */ \
INTEL_VGA_DEVICE(0x0C0B, info), /* SDV GT1 reserved */ \
INTEL_VGA_DEVICE(0x0C0E, info), /* SDV GT1 reserved */ \
- INTEL_VGA_DEVICE(0x0A02, info), /* ULT GT1 desktop */ \
- INTEL_VGA_DEVICE(0x0A0A, info), /* ULT GT1 server */ \
- INTEL_VGA_DEVICE(0x0A0B, info), /* ULT GT1 reserved */ \
INTEL_VGA_DEVICE(0x0D02, info), /* CRW GT1 desktop */ \
INTEL_VGA_DEVICE(0x0D0A, info), /* CRW GT1 server */ \
INTEL_VGA_DEVICE(0x0D0B, info), /* CRW GT1 reserved */ \
INTEL_VGA_DEVICE(0x0D0E, info), /* CRW GT1 reserved */ \
INTEL_VGA_DEVICE(0x0406, info), /* GT1 mobile */ \
INTEL_VGA_DEVICE(0x0C06, info), /* SDV GT1 mobile */ \
- INTEL_VGA_DEVICE(0x0A06, info), /* ULT GT1 mobile */ \
- INTEL_VGA_DEVICE(0x0A0E, info), /* ULX GT1 mobile */ \
INTEL_VGA_DEVICE(0x0D06, info) /* CRW GT1 mobile */
+#define INTEL_HSW_ULT_GT2_IDS(info) \
+ INTEL_VGA_DEVICE(0x0A12, info), /* ULT GT2 desktop */ \
+ INTEL_VGA_DEVICE(0x0A1A, info), /* ULT GT2 server */ \
+ INTEL_VGA_DEVICE(0x0A1B, info), /* ULT GT2 reserved */ \
+ INTEL_VGA_DEVICE(0x0A16, info) /* ULT GT2 mobile */
+
+#define INTEL_HSW_ULX_GT2_IDS(info) \
+ INTEL_VGA_DEVICE(0x0A1E, info) /* ULX GT2 mobile */ \
+
#define INTEL_HSW_GT2_IDS(info) \
+ INTEL_HSW_ULT_GT2_IDS(info), \
+ INTEL_HSW_ULX_GT2_IDS(info), \
INTEL_VGA_DEVICE(0x0412, info), /* GT2 desktop */ \
INTEL_VGA_DEVICE(0x041a, info), /* GT2 server */ \
INTEL_VGA_DEVICE(0x041B, info), /* GT2 reserved */ \
INTEL_VGA_DEVICE(0x0C1A, info), /* SDV GT2 server */ \
INTEL_VGA_DEVICE(0x0C1B, info), /* SDV GT2 reserved */ \
INTEL_VGA_DEVICE(0x0C1E, info), /* SDV GT2 reserved */ \
- INTEL_VGA_DEVICE(0x0A12, info), /* ULT GT2 desktop */ \
- INTEL_VGA_DEVICE(0x0A1A, info), /* ULT GT2 server */ \
- INTEL_VGA_DEVICE(0x0A1B, info), /* ULT GT2 reserved */ \
INTEL_VGA_DEVICE(0x0D12, info), /* CRW GT2 desktop */ \
INTEL_VGA_DEVICE(0x0D1A, info), /* CRW GT2 server */ \
INTEL_VGA_DEVICE(0x0D1B, info), /* CRW GT2 reserved */ \
INTEL_VGA_DEVICE(0x0416, info), /* GT2 mobile */ \
INTEL_VGA_DEVICE(0x0426, info), /* GT2 mobile */ \
INTEL_VGA_DEVICE(0x0C16, info), /* SDV GT2 mobile */ \
- INTEL_VGA_DEVICE(0x0A16, info), /* ULT GT2 mobile */ \
- INTEL_VGA_DEVICE(0x0A1E, info), /* ULX GT2 mobile */ \
INTEL_VGA_DEVICE(0x0D16, info) /* CRW GT2 mobile */
+#define INTEL_HSW_ULT_GT3_IDS(info) \
+ INTEL_VGA_DEVICE(0x0A22, info), /* ULT GT3 desktop */ \
+ INTEL_VGA_DEVICE(0x0A2A, info), /* ULT GT3 server */ \
+ INTEL_VGA_DEVICE(0x0A2B, info), /* ULT GT3 reserved */ \
+ INTEL_VGA_DEVICE(0x0A26, info), /* ULT GT3 mobile */ \
+ INTEL_VGA_DEVICE(0x0A2E, info) /* ULT GT3 reserved */
+
#define INTEL_HSW_GT3_IDS(info) \
+ INTEL_HSW_ULT_GT3_IDS(info), \
INTEL_VGA_DEVICE(0x0422, info), /* GT3 desktop */ \
INTEL_VGA_DEVICE(0x042a, info), /* GT3 server */ \
INTEL_VGA_DEVICE(0x042B, info), /* GT3 reserved */ \
INTEL_VGA_DEVICE(0x0C2A, info), /* SDV GT3 server */ \
INTEL_VGA_DEVICE(0x0C2B, info), /* SDV GT3 reserved */ \
INTEL_VGA_DEVICE(0x0C2E, info), /* SDV GT3 reserved */ \
- INTEL_VGA_DEVICE(0x0A22, info), /* ULT GT3 desktop */ \
- INTEL_VGA_DEVICE(0x0A2A, info), /* ULT GT3 server */ \
- INTEL_VGA_DEVICE(0x0A2B, info), /* ULT GT3 reserved */ \
INTEL_VGA_DEVICE(0x0D22, info), /* CRW GT3 desktop */ \
INTEL_VGA_DEVICE(0x0D2A, info), /* CRW GT3 server */ \
INTEL_VGA_DEVICE(0x0D2B, info), /* CRW GT3 reserved */ \
INTEL_VGA_DEVICE(0x0D2E, info), /* CRW GT3 reserved */ \
INTEL_VGA_DEVICE(0x0C26, info), /* SDV GT3 mobile */ \
- INTEL_VGA_DEVICE(0x0A26, info), /* ULT GT3 mobile */ \
- INTEL_VGA_DEVICE(0x0A2E, info), /* ULT GT3 reserved */ \
INTEL_VGA_DEVICE(0x0D26, info) /* CRW GT3 mobile */
#define INTEL_HSW_IDS(info) \
INTEL_VGA_DEVICE(0x0157, info), \
INTEL_VGA_DEVICE(0x0155, info)
-#define INTEL_BDW_GT1_IDS(info) \
- INTEL_VGA_DEVICE(0x1602, info), /* GT1 ULT */ \
+#define INTEL_BDW_ULT_GT1_IDS(info) \
INTEL_VGA_DEVICE(0x1606, info), /* GT1 ULT */ \
- INTEL_VGA_DEVICE(0x160B, info), /* GT1 Iris */ \
- INTEL_VGA_DEVICE(0x160E, info), /* GT1 ULX */ \
+ INTEL_VGA_DEVICE(0x160B, info) /* GT1 Iris */
+
+#define INTEL_BDW_ULX_GT1_IDS(info) \
+ INTEL_VGA_DEVICE(0x160E, info) /* GT1 ULX */
+
+#define INTEL_BDW_GT1_IDS(info) \
+ INTEL_BDW_ULT_GT1_IDS(info), \
+ INTEL_BDW_ULX_GT1_IDS(info), \
+ INTEL_VGA_DEVICE(0x1602, info), /* GT1 ULT */ \
INTEL_VGA_DEVICE(0x160A, info), /* GT1 Server */ \
INTEL_VGA_DEVICE(0x160D, info) /* GT1 Workstation */
-#define INTEL_BDW_GT2_IDS(info) \
- INTEL_VGA_DEVICE(0x1612, info), /* GT2 Halo */ \
+#define INTEL_BDW_ULT_GT2_IDS(info) \
INTEL_VGA_DEVICE(0x1616, info), /* GT2 ULT */ \
- INTEL_VGA_DEVICE(0x161B, info), /* GT2 ULT */ \
- INTEL_VGA_DEVICE(0x161E, info), /* GT2 ULX */ \
+ INTEL_VGA_DEVICE(0x161B, info) /* GT2 ULT */
+
+#define INTEL_BDW_ULX_GT2_IDS(info) \
+ INTEL_VGA_DEVICE(0x161E, info) /* GT2 ULX */
+
+#define INTEL_BDW_GT2_IDS(info) \
+ INTEL_BDW_ULT_GT2_IDS(info), \
+ INTEL_BDW_ULX_GT2_IDS(info), \
+ INTEL_VGA_DEVICE(0x1612, info), /* GT2 Halo */ \
INTEL_VGA_DEVICE(0x161A, info), /* GT2 Server */ \
INTEL_VGA_DEVICE(0x161D, info) /* GT2 Workstation */
+#define INTEL_BDW_ULT_GT3_IDS(info) \
+ INTEL_VGA_DEVICE(0x1626, info), /* ULT */ \
+ INTEL_VGA_DEVICE(0x162B, info) /* Iris */ \
+
+#define INTEL_BDW_ULX_GT3_IDS(info) \
+ INTEL_VGA_DEVICE(0x162E, info) /* ULX */
+
#define INTEL_BDW_GT3_IDS(info) \
+ INTEL_BDW_ULT_GT3_IDS(info), \
+ INTEL_BDW_ULX_GT3_IDS(info), \
INTEL_VGA_DEVICE(0x1622, info), /* ULT */ \
- INTEL_VGA_DEVICE(0x1626, info), /* ULT */ \
- INTEL_VGA_DEVICE(0x162B, info), /* Iris */ \
- INTEL_VGA_DEVICE(0x162E, info), /* ULX */\
INTEL_VGA_DEVICE(0x162A, info), /* Server */ \
INTEL_VGA_DEVICE(0x162D, info) /* Workstation */
+#define INTEL_BDW_ULT_RSVD_IDS(info) \
+ INTEL_VGA_DEVICE(0x1636, info), /* ULT */ \
+ INTEL_VGA_DEVICE(0x163B, info) /* Iris */
+
+#define INTEL_BDW_ULX_RSVD_IDS(info) \
+ INTEL_VGA_DEVICE(0x163E, info) /* ULX */
+
#define INTEL_BDW_RSVD_IDS(info) \
+ INTEL_BDW_ULT_RSVD_IDS(info), \
+ INTEL_BDW_ULX_RSVD_IDS(info), \
INTEL_VGA_DEVICE(0x1632, info), /* ULT */ \
- INTEL_VGA_DEVICE(0x1636, info), /* ULT */ \
- INTEL_VGA_DEVICE(0x163B, info), /* Iris */ \
- INTEL_VGA_DEVICE(0x163E, info), /* ULX */ \
INTEL_VGA_DEVICE(0x163A, info), /* Server */ \
INTEL_VGA_DEVICE(0x163D, info) /* Workstation */
INTEL_VGA_DEVICE(0x22b2, info), \
INTEL_VGA_DEVICE(0x22b3, info)
+#define INTEL_SKL_ULT_GT1_IDS(info) \
+ INTEL_VGA_DEVICE(0x1906, info) /* ULT GT1 */
+
+#define INTEL_SKL_ULX_GT1_IDS(info) \
+ INTEL_VGA_DEVICE(0x190E, info) /* ULX GT1 */
+
#define INTEL_SKL_GT1_IDS(info) \
- INTEL_VGA_DEVICE(0x1906, info), /* ULT GT1 */ \
- INTEL_VGA_DEVICE(0x190E, info), /* ULX GT1 */ \
+ INTEL_SKL_ULT_GT1_IDS(info), \
+ INTEL_SKL_ULX_GT1_IDS(info), \
INTEL_VGA_DEVICE(0x1902, info), /* DT GT1 */ \
INTEL_VGA_DEVICE(0x190B, info), /* Halo GT1 */ \
INTEL_VGA_DEVICE(0x190A, info) /* SRV GT1 */
-#define INTEL_SKL_GT2_IDS(info) \
+#define INTEL_SKL_ULT_GT2_IDS(info) \
INTEL_VGA_DEVICE(0x1916, info), /* ULT GT2 */ \
- INTEL_VGA_DEVICE(0x1921, info), /* ULT GT2F */ \
- INTEL_VGA_DEVICE(0x191E, info), /* ULX GT2 */ \
+ INTEL_VGA_DEVICE(0x1921, info) /* ULT GT2F */
+
+#define INTEL_SKL_ULX_GT2_IDS(info) \
+ INTEL_VGA_DEVICE(0x191E, info) /* ULX GT2 */
+
+#define INTEL_SKL_GT2_IDS(info) \
+ INTEL_SKL_ULT_GT2_IDS(info), \
+ INTEL_SKL_ULX_GT2_IDS(info), \
INTEL_VGA_DEVICE(0x1912, info), /* DT GT2 */ \
INTEL_VGA_DEVICE(0x191B, info), /* Halo GT2 */ \
INTEL_VGA_DEVICE(0x191A, info), /* SRV GT2 */ \
INTEL_VGA_DEVICE(0x191D, info) /* WKS GT2 */
+#define INTEL_SKL_ULT_GT3_IDS(info) \
+ INTEL_VGA_DEVICE(0x1926, info) /* ULT GT3 */
+
#define INTEL_SKL_GT3_IDS(info) \
+ INTEL_SKL_ULT_GT3_IDS(info), \
INTEL_VGA_DEVICE(0x1923, info), /* ULT GT3 */ \
- INTEL_VGA_DEVICE(0x1926, info), /* ULT GT3 */ \
INTEL_VGA_DEVICE(0x1927, info), /* ULT GT3 */ \
INTEL_VGA_DEVICE(0x192B, info), /* Halo GT3 */ \
INTEL_VGA_DEVICE(0x192D, info) /* SRV GT3 */
INTEL_VGA_DEVICE(0x3184, info), \
INTEL_VGA_DEVICE(0x3185, info)
-#define INTEL_KBL_GT1_IDS(info) \
- INTEL_VGA_DEVICE(0x5913, info), /* ULT GT1.5 */ \
- INTEL_VGA_DEVICE(0x5915, info), /* ULX GT1.5 */ \
+#define INTEL_KBL_ULT_GT1_IDS(info) \
INTEL_VGA_DEVICE(0x5906, info), /* ULT GT1 */ \
+ INTEL_VGA_DEVICE(0x5913, info) /* ULT GT1.5 */
+
+#define INTEL_KBL_ULX_GT1_IDS(info) \
INTEL_VGA_DEVICE(0x590E, info), /* ULX GT1 */ \
+ INTEL_VGA_DEVICE(0x5915, info) /* ULX GT1.5 */
+
+#define INTEL_KBL_GT1_IDS(info) \
+ INTEL_KBL_ULT_GT1_IDS(info), \
+ INTEL_KBL_ULX_GT1_IDS(info), \
INTEL_VGA_DEVICE(0x5902, info), /* DT GT1 */ \
INTEL_VGA_DEVICE(0x5908, info), /* Halo GT1 */ \
INTEL_VGA_DEVICE(0x590B, info), /* Halo GT1 */ \
INTEL_VGA_DEVICE(0x590A, info) /* SRV GT1 */
-#define INTEL_KBL_GT2_IDS(info) \
+#define INTEL_KBL_ULT_GT2_IDS(info) \
INTEL_VGA_DEVICE(0x5916, info), /* ULT GT2 */ \
+ INTEL_VGA_DEVICE(0x5921, info) /* ULT GT2F */
+
+#define INTEL_KBL_ULX_GT2_IDS(info) \
+ INTEL_VGA_DEVICE(0x591E, info) /* ULX GT2 */
+
+#define INTEL_KBL_GT2_IDS(info) \
+ INTEL_KBL_ULT_GT2_IDS(info), \
+ INTEL_KBL_ULX_GT2_IDS(info), \
INTEL_VGA_DEVICE(0x5917, info), /* Mobile GT2 */ \
- INTEL_VGA_DEVICE(0x5921, info), /* ULT GT2F */ \
- INTEL_VGA_DEVICE(0x591E, info), /* ULX GT2 */ \
INTEL_VGA_DEVICE(0x5912, info), /* DT GT2 */ \
INTEL_VGA_DEVICE(0x591B, info), /* Halo GT2 */ \
INTEL_VGA_DEVICE(0x591A, info), /* SRV GT2 */ \
INTEL_VGA_DEVICE(0x591D, info) /* WKS GT2 */
+#define INTEL_KBL_ULT_GT3_IDS(info) \
+ INTEL_VGA_DEVICE(0x5926, info) /* ULT GT3 */
+
#define INTEL_KBL_GT3_IDS(info) \
+ INTEL_KBL_ULT_GT3_IDS(info), \
INTEL_VGA_DEVICE(0x5923, info), /* ULT GT3 */ \
- INTEL_VGA_DEVICE(0x5926, info), /* ULT GT3 */ \
INTEL_VGA_DEVICE(0x5927, info) /* ULT GT3 */
#define INTEL_KBL_GT4_IDS(info) \
#define INTEL_AML_CFL_GT2_IDS(info) \
INTEL_VGA_DEVICE(0x87CA, info)
+/* CML GT1 */
+#define INTEL_CML_GT1_IDS(info) \
+ INTEL_VGA_DEVICE(0x9B21, info), \
+ INTEL_VGA_DEVICE(0x9BAA, info), \
+ INTEL_VGA_DEVICE(0x9BAB, info), \
+ INTEL_VGA_DEVICE(0x9BAC, info), \
+ INTEL_VGA_DEVICE(0x9BA0, info), \
+ INTEL_VGA_DEVICE(0x9BA5, info), \
+ INTEL_VGA_DEVICE(0x9BA8, info), \
+ INTEL_VGA_DEVICE(0x9BA4, info), \
+ INTEL_VGA_DEVICE(0x9BA2, info)
+
+/* CML GT2 */
+#define INTEL_CML_GT2_IDS(info) \
+ INTEL_VGA_DEVICE(0x9B41, info), \
+ INTEL_VGA_DEVICE(0x9BCA, info), \
+ INTEL_VGA_DEVICE(0x9BCB, info), \
+ INTEL_VGA_DEVICE(0x9BCC, info), \
+ INTEL_VGA_DEVICE(0x9BC0, info), \
+ INTEL_VGA_DEVICE(0x9BC5, info), \
+ INTEL_VGA_DEVICE(0x9BC8, info), \
+ INTEL_VGA_DEVICE(0x9BC4, info), \
+ INTEL_VGA_DEVICE(0x9BC2, info)
+
#define INTEL_KBL_IDS(info) \
INTEL_KBL_GT1_IDS(info), \
INTEL_KBL_GT2_IDS(info), \
INTEL_WHL_U_GT1_IDS(info), \
INTEL_WHL_U_GT2_IDS(info), \
INTEL_WHL_U_GT3_IDS(info), \
- INTEL_AML_CFL_GT2_IDS(info)
+ INTEL_AML_CFL_GT2_IDS(info), \
+ INTEL_CML_GT1_IDS(info), \
+ INTEL_CML_GT2_IDS(info)
/* CNL */
+#define INTEL_CNL_PORT_F_IDS(info) \
+ INTEL_VGA_DEVICE(0x5A54, info), \
+ INTEL_VGA_DEVICE(0x5A5C, info), \
+ INTEL_VGA_DEVICE(0x5A44, info), \
+ INTEL_VGA_DEVICE(0x5A4C, info)
+
#define INTEL_CNL_IDS(info) \
+ INTEL_CNL_PORT_F_IDS(info), \
INTEL_VGA_DEVICE(0x5A51, info), \
INTEL_VGA_DEVICE(0x5A59, info), \
INTEL_VGA_DEVICE(0x5A41, info), \
INTEL_VGA_DEVICE(0x5A42, info), \
INTEL_VGA_DEVICE(0x5A4A, info), \
INTEL_VGA_DEVICE(0x5A50, info), \
- INTEL_VGA_DEVICE(0x5A40, info), \
- INTEL_VGA_DEVICE(0x5A54, info), \
- INTEL_VGA_DEVICE(0x5A5C, info), \
- INTEL_VGA_DEVICE(0x5A44, info), \
- INTEL_VGA_DEVICE(0x5A4C, info)
+ INTEL_VGA_DEVICE(0x5A40, info)
/* ICL */
-#define INTEL_ICL_11_IDS(info) \
+#define INTEL_ICL_PORT_F_IDS(info) \
INTEL_VGA_DEVICE(0x8A50, info), \
- INTEL_VGA_DEVICE(0x8A51, info), \
INTEL_VGA_DEVICE(0x8A5C, info), \
INTEL_VGA_DEVICE(0x8A5D, info), \
INTEL_VGA_DEVICE(0x8A59, info), \
INTEL_VGA_DEVICE(0x8A57, info), \
INTEL_VGA_DEVICE(0x8A56, info), \
INTEL_VGA_DEVICE(0x8A71, info), \
- INTEL_VGA_DEVICE(0x8A70, info)
+ INTEL_VGA_DEVICE(0x8A70, info), \
+ INTEL_VGA_DEVICE(0x8A53, info)
+
+#define INTEL_ICL_11_IDS(info) \
+ INTEL_ICL_PORT_F_IDS(info), \
+ INTEL_VGA_DEVICE(0x8A51, info)
+
+/* EHL */
+#define INTEL_EHL_IDS(info) \
+ INTEL_VGA_DEVICE(0x4500, info), \
+ INTEL_VGA_DEVICE(0x4571, info), \
+ INTEL_VGA_DEVICE(0x4551, info), \
+ INTEL_VGA_DEVICE(0x4541, info)
#endif /* _I915_PCIIDS_H */
#ifndef __LINUX_MIPI_DBI_H
#define __LINUX_MIPI_DBI_H
-#include <drm/tinydrm/tinydrm.h>
+#include <linux/mutex.h>
+#include <drm/drm_device.h>
+#include <drm/drm_simple_kms_helper.h>
struct drm_rect;
struct spi_device;
/**
* struct mipi_dbi - MIPI DBI controller
- * @tinydrm: tinydrm base
* @spi: SPI device
* @enabled: Pipeline is enabled
* @cmdlock: Command lock
* @regulator: power regulator (optional)
*/
struct mipi_dbi {
- struct tinydrm_device tinydrm;
+ /**
+ * @drm: DRM device
+ */
+ struct drm_device drm;
+
+ /**
+ * @pipe: Display pipe structure
+ */
+ struct drm_simple_display_pipe pipe;
+
struct spi_device *spi;
bool enabled;
struct mutex cmdlock;
- int (*command)(struct mipi_dbi *mipi, u8 cmd, u8 *param, size_t num);
+ int (*command)(struct mipi_dbi *mipi, u8 *cmd, u8 *param, size_t num);
const u8 *read_commands;
struct gpio_desc *dc;
u16 *tx_buf;
struct regulator *regulator;
};
-static inline struct mipi_dbi *
-mipi_dbi_from_tinydrm(struct tinydrm_device *tdev)
+static inline struct mipi_dbi *drm_to_mipi_dbi(struct drm_device *drm)
{
- return container_of(tdev, struct mipi_dbi, tinydrm);
+ return container_of(drm, struct mipi_dbi, drm);
}
int mipi_dbi_spi_init(struct spi_device *spi, struct mipi_dbi *mipi,
struct gpio_desc *dc);
-int mipi_dbi_init(struct device *dev, struct mipi_dbi *mipi,
- const struct drm_simple_display_pipe_funcs *pipe_funcs,
- struct drm_driver *driver,
+int mipi_dbi_init(struct mipi_dbi *mipi,
+ const struct drm_simple_display_pipe_funcs *funcs,
const struct drm_display_mode *mode, unsigned int rotation);
+void mipi_dbi_release(struct drm_device *drm);
void mipi_dbi_pipe_update(struct drm_simple_display_pipe *pipe,
struct drm_plane_state *old_state);
void mipi_dbi_enable_flush(struct mipi_dbi *mipi,
int mipi_dbi_command_read(struct mipi_dbi *mipi, u8 cmd, u8 *val);
int mipi_dbi_command_buf(struct mipi_dbi *mipi, u8 cmd, u8 *data, size_t len);
+int mipi_dbi_command_stackbuf(struct mipi_dbi *mipi, u8 cmd, u8 *data, size_t len);
int mipi_dbi_buf_copy(void *dst, struct drm_framebuffer *fb,
struct drm_rect *clip, bool swap);
/**
#define mipi_dbi_command(mipi, cmd, seq...) \
({ \
u8 d[] = { seq }; \
- mipi_dbi_command_buf(mipi, cmd, d, ARRAY_SIZE(d)); \
+ mipi_dbi_command_stackbuf(mipi, cmd, d, ARRAY_SIZE(d)); \
})
#ifdef CONFIG_DEBUG_FS
#define __LINUX_TINYDRM_HELPERS_H
struct backlight_device;
+struct drm_device;
+struct drm_display_mode;
struct drm_framebuffer;
struct drm_rect;
+struct drm_simple_display_pipe;
+struct drm_simple_display_pipe_funcs;
struct spi_transfer;
struct spi_message;
struct spi_device;
#endif
}
-void tinydrm_memcpy(void *dst, void *vaddr, struct drm_framebuffer *fb,
- struct drm_rect *clip);
-void tinydrm_swab16(u16 *dst, void *vaddr, struct drm_framebuffer *fb,
- struct drm_rect *clip);
-void tinydrm_xrgb8888_to_rgb565(u16 *dst, void *vaddr,
- struct drm_framebuffer *fb,
- struct drm_rect *clip, bool swap);
-void tinydrm_xrgb8888_to_gray8(u8 *dst, void *vaddr, struct drm_framebuffer *fb,
- struct drm_rect *clip);
+int tinydrm_display_pipe_init(struct drm_device *drm,
+ struct drm_simple_display_pipe *pipe,
+ const struct drm_simple_display_pipe_funcs *funcs,
+ int connector_type,
+ const uint32_t *formats,
+ unsigned int format_count,
+ const struct drm_display_mode *mode,
+ unsigned int rotation);
size_t tinydrm_spi_max_transfer_size(struct spi_device *spi, size_t max_len);
bool tinydrm_spi_bpw_supported(struct spi_device *spi, u8 bpw);
+++ /dev/null
-/*
- * Copyright (C) 2016 Noralf Trønnes
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- */
-
-#ifndef __LINUX_TINYDRM_H
-#define __LINUX_TINYDRM_H
-
-#include <drm/drm_simple_kms_helper.h>
-
-struct drm_driver;
-
-/**
- * struct tinydrm_device - tinydrm device
- */
-struct tinydrm_device {
- /**
- * @drm: DRM device
- */
- struct drm_device *drm;
-
- /**
- * @pipe: Display pipe structure
- */
- struct drm_simple_display_pipe pipe;
-};
-
-static inline struct tinydrm_device *
-pipe_to_tinydrm(struct drm_simple_display_pipe *pipe)
-{
- return container_of(pipe, struct tinydrm_device, pipe);
-}
-
-/**
- * TINYDRM_MODE - tinydrm display mode
- * @hd: Horizontal resolution, width
- * @vd: Vertical resolution, height
- * @hd_mm: Display width in millimeters
- * @vd_mm: Display height in millimeters
- *
- * This macro creates a &drm_display_mode for use with tinydrm.
- */
-#define TINYDRM_MODE(hd, vd, hd_mm, vd_mm) \
- .hdisplay = (hd), \
- .hsync_start = (hd), \
- .hsync_end = (hd), \
- .htotal = (hd), \
- .vdisplay = (vd), \
- .vsync_start = (vd), \
- .vsync_end = (vd), \
- .vtotal = (vd), \
- .width_mm = (hd_mm), \
- .height_mm = (vd_mm), \
- .type = DRM_MODE_TYPE_DRIVER, \
- .clock = 1 /* pass validation */
-
-int devm_tinydrm_init(struct device *parent, struct tinydrm_device *tdev,
- struct drm_driver *driver);
-int devm_tinydrm_register(struct tinydrm_device *tdev);
-void tinydrm_shutdown(struct tinydrm_device *tdev);
-
-int
-tinydrm_display_pipe_init(struct tinydrm_device *tdev,
- const struct drm_simple_display_pipe_funcs *funcs,
- int connector_type,
- const uint32_t *formats,
- unsigned int format_count,
- const struct drm_display_mode *mode,
- unsigned int rotation);
-
-#endif /* __LINUX_TINYDRM_H */
int ttm_bo_device_init(struct ttm_bo_device *bdev,
struct ttm_bo_driver *driver,
struct address_space *mapping,
- uint64_t file_page_offset, bool need_dma32);
+ bool need_dma32);
/**
* ttm_bo_unmap_virtual
#ifndef __AXG_AUDIO_CLKC_BINDINGS_H
#define __AXG_AUDIO_CLKC_BINDINGS_H
-#define AUD_CLKID_SLV_SCLK0 9
-#define AUD_CLKID_SLV_SCLK1 10
-#define AUD_CLKID_SLV_SCLK2 11
-#define AUD_CLKID_SLV_SCLK3 12
-#define AUD_CLKID_SLV_SCLK4 13
-#define AUD_CLKID_SLV_SCLK5 14
-#define AUD_CLKID_SLV_SCLK6 15
-#define AUD_CLKID_SLV_SCLK7 16
-#define AUD_CLKID_SLV_SCLK8 17
-#define AUD_CLKID_SLV_SCLK9 18
-#define AUD_CLKID_SLV_LRCLK0 19
-#define AUD_CLKID_SLV_LRCLK1 20
-#define AUD_CLKID_SLV_LRCLK2 21
-#define AUD_CLKID_SLV_LRCLK3 22
-#define AUD_CLKID_SLV_LRCLK4 23
-#define AUD_CLKID_SLV_LRCLK5 24
-#define AUD_CLKID_SLV_LRCLK6 25
-#define AUD_CLKID_SLV_LRCLK7 26
-#define AUD_CLKID_SLV_LRCLK8 27
-#define AUD_CLKID_SLV_LRCLK9 28
#define AUD_CLKID_DDR_ARB 29
#define AUD_CLKID_PDM 30
#define AUD_CLKID_TDMIN_A 31
#define AUD_CLKID_TDMOUT_A_LRCLK 134
#define AUD_CLKID_TDMOUT_B_LRCLK 135
#define AUD_CLKID_TDMOUT_C_LRCLK 136
+#define AUD_CLKID_SPDIFOUT_B 151
+#define AUD_CLKID_SPDIFOUT_B_CLK 152
+#define AUD_CLKID_TDM_MCLK_PAD0 155
+#define AUD_CLKID_TDM_MCLK_PAD1 156
+#define AUD_CLKID_TDM_LRCLK_PAD0 157
+#define AUD_CLKID_TDM_LRCLK_PAD1 158
+#define AUD_CLKID_TDM_LRCLK_PAD2 159
+#define AUD_CLKID_TDM_SCLK_PAD0 160
+#define AUD_CLKID_TDM_SCLK_PAD1 161
+#define AUD_CLKID_TDM_SCLK_PAD2 162
#endif /* __AXG_AUDIO_CLKC_BINDINGS_H */
#define CLK_UART0 257
#define CLK_UART1 258
#define CLK_UART2 259
+#define CLK_UART3 260
#define CLK_I2C0 261
#define CLK_I2C1 262
#define CLK_I2C2 263
#define CLK_USI1 266
#define CLK_USI2 267
#define CLK_USI3 268
-#define CLK_UART3 260
+#define CLK_TSADC 270
#define CLK_PWM 279
#define CLK_MCT 315
#define CLK_WDT 316
#define CLKID_AO_M4_FCLK 13
#define CLKID_AO_M4_HCLK 14
#define CLKID_AO_CLK81 15
+#define CLKID_AO_SAR_ADC_SEL 16
#define CLKID_AO_SAR_ADC_CLK 18
+#define CLKID_AO_CTS_OSCIN 19
#define CLKID_AO_32K 23
#define CLKID_AO_CEC 27
#define CLKID_AO_CTS_RTC_OSCIN 28
#define CLKID_MALI_1 174
#define CLKID_MALI 175
#define CLKID_MPLL_5OM 177
+#define CLKID_CPU_CLK 187
+#define CLKID_PCIE_PLL 201
+#define CLKID_VDEC_1 204
+#define CLKID_VDEC_HEVC 207
+#define CLKID_VDEC_HEVCF 210
#endif /* __G12A_CLKC_H */
#define IMX7ULP_CLK_FLEXBUS 2
#define IMX7ULP_CLK_SEMA42_1 3
#define IMX7ULP_CLK_DMA_MUX1 4
-#define IMX7ULP_CLK_SNVS 5
#define IMX7ULP_CLK_CAAM 6
#define IMX7ULP_CLK_LPTPM4 7
#define IMX7ULP_CLK_LPTPM5 8
#define JZ4725B_CLK_TCU 22
#define JZ4725B_CLK_EXT512 23
#define JZ4725B_CLK_RTC 24
+#define JZ4725B_CLK_UDC_PHY 25
#endif /* __DT_BINDINGS_CLOCK_JZ4725B_CGU_H__ */
#define CLKID_MPLL1 94
#define CLKID_MPLL2 95
#define CLKID_NAND_CLK 112
-#define CLKID_ABP 124
#define CLKID_APB 124
#define CLKID_PERIPH 126
#define CLKID_AXI 128
#define CLKID_L2_DRAM 130
+#define CLKID_VPU 190
+#define CLKID_VDEC_1 196
+#define CLKID_VDEC_HCODEC 199
+#define CLKID_VDEC_2 202
+#define CLKID_VDEC_HEVC 206
#endif /* __MESON8B_CLKC_H */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Weiyi Lu <weiyi.lu@mediatek.com>
+ */
+
+#ifndef _DT_BINDINGS_CLK_MT8183_H
+#define _DT_BINDINGS_CLK_MT8183_H
+
+/* APMIXED */
+#define CLK_APMIXED_ARMPLL_LL 0
+#define CLK_APMIXED_ARMPLL_L 1
+#define CLK_APMIXED_CCIPLL 2
+#define CLK_APMIXED_MAINPLL 3
+#define CLK_APMIXED_UNIV2PLL 4
+#define CLK_APMIXED_MSDCPLL 5
+#define CLK_APMIXED_MMPLL 6
+#define CLK_APMIXED_MFGPLL 7
+#define CLK_APMIXED_TVDPLL 8
+#define CLK_APMIXED_APLL1 9
+#define CLK_APMIXED_APLL2 10
+#define CLK_APMIXED_SSUSB_26M 11
+#define CLK_APMIXED_APPLL_26M 12
+#define CLK_APMIXED_MIPIC0_26M 13
+#define CLK_APMIXED_MDPLLGP_26M 14
+#define CLK_APMIXED_MMSYS_26M 15
+#define CLK_APMIXED_UFS_26M 16
+#define CLK_APMIXED_MIPIC1_26M 17
+#define CLK_APMIXED_MEMPLL_26M 18
+#define CLK_APMIXED_CLKSQ_LVPLL_26M 19
+#define CLK_APMIXED_MIPID0_26M 20
+#define CLK_APMIXED_MIPID1_26M 21
+#define CLK_APMIXED_NR_CLK 22
+
+/* TOPCKGEN */
+#define CLK_TOP_MUX_AXI 0
+#define CLK_TOP_MUX_MM 1
+#define CLK_TOP_MUX_CAM 2
+#define CLK_TOP_MUX_MFG 3
+#define CLK_TOP_MUX_CAMTG 4
+#define CLK_TOP_MUX_UART 5
+#define CLK_TOP_MUX_SPI 6
+#define CLK_TOP_MUX_MSDC50_0_HCLK 7
+#define CLK_TOP_MUX_MSDC50_0 8
+#define CLK_TOP_MUX_MSDC30_1 9
+#define CLK_TOP_MUX_MSDC30_2 10
+#define CLK_TOP_MUX_AUDIO 11
+#define CLK_TOP_MUX_AUD_INTBUS 12
+#define CLK_TOP_MUX_FPWRAP_ULPOSC 13
+#define CLK_TOP_MUX_SCP 14
+#define CLK_TOP_MUX_ATB 15
+#define CLK_TOP_MUX_SSPM 16
+#define CLK_TOP_MUX_DPI0 17
+#define CLK_TOP_MUX_SCAM 18
+#define CLK_TOP_MUX_AUD_1 19
+#define CLK_TOP_MUX_AUD_2 20
+#define CLK_TOP_MUX_DISP_PWM 21
+#define CLK_TOP_MUX_SSUSB_TOP_XHCI 22
+#define CLK_TOP_MUX_USB_TOP 23
+#define CLK_TOP_MUX_SPM 24
+#define CLK_TOP_MUX_I2C 25
+#define CLK_TOP_MUX_F52M_MFG 26
+#define CLK_TOP_MUX_SENINF 27
+#define CLK_TOP_MUX_DXCC 28
+#define CLK_TOP_MUX_CAMTG2 29
+#define CLK_TOP_MUX_AUD_ENG1 30
+#define CLK_TOP_MUX_AUD_ENG2 31
+#define CLK_TOP_MUX_FAES_UFSFDE 32
+#define CLK_TOP_MUX_FUFS 33
+#define CLK_TOP_MUX_IMG 34
+#define CLK_TOP_MUX_DSP 35
+#define CLK_TOP_MUX_DSP1 36
+#define CLK_TOP_MUX_DSP2 37
+#define CLK_TOP_MUX_IPU_IF 38
+#define CLK_TOP_MUX_CAMTG3 39
+#define CLK_TOP_MUX_CAMTG4 40
+#define CLK_TOP_MUX_PMICSPI 41
+#define CLK_TOP_SYSPLL_CK 42
+#define CLK_TOP_SYSPLL_D2 43
+#define CLK_TOP_SYSPLL_D3 44
+#define CLK_TOP_SYSPLL_D5 45
+#define CLK_TOP_SYSPLL_D7 46
+#define CLK_TOP_SYSPLL_D2_D2 47
+#define CLK_TOP_SYSPLL_D2_D4 48
+#define CLK_TOP_SYSPLL_D2_D8 49
+#define CLK_TOP_SYSPLL_D2_D16 50
+#define CLK_TOP_SYSPLL_D3_D2 51
+#define CLK_TOP_SYSPLL_D3_D4 52
+#define CLK_TOP_SYSPLL_D3_D8 53
+#define CLK_TOP_SYSPLL_D5_D2 54
+#define CLK_TOP_SYSPLL_D5_D4 55
+#define CLK_TOP_SYSPLL_D7_D2 56
+#define CLK_TOP_SYSPLL_D7_D4 57
+#define CLK_TOP_UNIVPLL_CK 58
+#define CLK_TOP_UNIVPLL_D2 59
+#define CLK_TOP_UNIVPLL_D3 60
+#define CLK_TOP_UNIVPLL_D5 61
+#define CLK_TOP_UNIVPLL_D7 62
+#define CLK_TOP_UNIVPLL_D2_D2 63
+#define CLK_TOP_UNIVPLL_D2_D4 64
+#define CLK_TOP_UNIVPLL_D2_D8 65
+#define CLK_TOP_UNIVPLL_D3_D2 66
+#define CLK_TOP_UNIVPLL_D3_D4 67
+#define CLK_TOP_UNIVPLL_D3_D8 68
+#define CLK_TOP_UNIVPLL_D5_D2 69
+#define CLK_TOP_UNIVPLL_D5_D4 70
+#define CLK_TOP_UNIVPLL_D5_D8 71
+#define CLK_TOP_APLL1_CK 72
+#define CLK_TOP_APLL1_D2 73
+#define CLK_TOP_APLL1_D4 74
+#define CLK_TOP_APLL1_D8 75
+#define CLK_TOP_APLL2_CK 76
+#define CLK_TOP_APLL2_D2 77
+#define CLK_TOP_APLL2_D4 78
+#define CLK_TOP_APLL2_D8 79
+#define CLK_TOP_TVDPLL_CK 80
+#define CLK_TOP_TVDPLL_D2 81
+#define CLK_TOP_TVDPLL_D4 82
+#define CLK_TOP_TVDPLL_D8 83
+#define CLK_TOP_TVDPLL_D16 84
+#define CLK_TOP_MSDCPLL_CK 85
+#define CLK_TOP_MSDCPLL_D2 86
+#define CLK_TOP_MSDCPLL_D4 87
+#define CLK_TOP_MSDCPLL_D8 88
+#define CLK_TOP_MSDCPLL_D16 89
+#define CLK_TOP_AD_OSC_CK 90
+#define CLK_TOP_OSC_D2 91
+#define CLK_TOP_OSC_D4 92
+#define CLK_TOP_OSC_D8 93
+#define CLK_TOP_OSC_D16 94
+#define CLK_TOP_F26M_CK_D2 95
+#define CLK_TOP_MFGPLL_CK 96
+#define CLK_TOP_UNIVP_192M_CK 97
+#define CLK_TOP_UNIVP_192M_D2 98
+#define CLK_TOP_UNIVP_192M_D4 99
+#define CLK_TOP_UNIVP_192M_D8 100
+#define CLK_TOP_UNIVP_192M_D16 101
+#define CLK_TOP_UNIVP_192M_D32 102
+#define CLK_TOP_MMPLL_CK 103
+#define CLK_TOP_MMPLL_D4 104
+#define CLK_TOP_MMPLL_D4_D2 105
+#define CLK_TOP_MMPLL_D4_D4 106
+#define CLK_TOP_MMPLL_D5 107
+#define CLK_TOP_MMPLL_D5_D2 108
+#define CLK_TOP_MMPLL_D5_D4 109
+#define CLK_TOP_MMPLL_D6 110
+#define CLK_TOP_MMPLL_D7 111
+#define CLK_TOP_CLK26M 112
+#define CLK_TOP_CLK13M 113
+#define CLK_TOP_ULPOSC 114
+#define CLK_TOP_UNIVP_192M 115
+#define CLK_TOP_MUX_APLL_I2S0 116
+#define CLK_TOP_MUX_APLL_I2S1 117
+#define CLK_TOP_MUX_APLL_I2S2 118
+#define CLK_TOP_MUX_APLL_I2S3 119
+#define CLK_TOP_MUX_APLL_I2S4 120
+#define CLK_TOP_MUX_APLL_I2S5 121
+#define CLK_TOP_APLL12_DIV0 122
+#define CLK_TOP_APLL12_DIV1 123
+#define CLK_TOP_APLL12_DIV2 124
+#define CLK_TOP_APLL12_DIV3 125
+#define CLK_TOP_APLL12_DIV4 126
+#define CLK_TOP_APLL12_DIVB 127
+#define CLK_TOP_UNIVPLL 128
+#define CLK_TOP_ARMPLL_DIV_PLL1 129
+#define CLK_TOP_ARMPLL_DIV_PLL2 130
+#define CLK_TOP_UNIVPLL_D3_D16 131
+#define CLK_TOP_NR_CLK 132
+
+/* CAMSYS */
+#define CLK_CAM_LARB6 0
+#define CLK_CAM_DFP_VAD 1
+#define CLK_CAM_CAM 2
+#define CLK_CAM_CAMTG 3
+#define CLK_CAM_SENINF 4
+#define CLK_CAM_CAMSV0 5
+#define CLK_CAM_CAMSV1 6
+#define CLK_CAM_CAMSV2 7
+#define CLK_CAM_CCU 8
+#define CLK_CAM_LARB3 9
+#define CLK_CAM_NR_CLK 10
+
+/* INFRACFG_AO */
+#define CLK_INFRA_PMIC_TMR 0
+#define CLK_INFRA_PMIC_AP 1
+#define CLK_INFRA_PMIC_MD 2
+#define CLK_INFRA_PMIC_CONN 3
+#define CLK_INFRA_SCPSYS 4
+#define CLK_INFRA_SEJ 5
+#define CLK_INFRA_APXGPT 6
+#define CLK_INFRA_ICUSB 7
+#define CLK_INFRA_GCE 8
+#define CLK_INFRA_THERM 9
+#define CLK_INFRA_I2C0 10
+#define CLK_INFRA_I2C1 11
+#define CLK_INFRA_I2C2 12
+#define CLK_INFRA_I2C3 13
+#define CLK_INFRA_PWM_HCLK 14
+#define CLK_INFRA_PWM1 15
+#define CLK_INFRA_PWM2 16
+#define CLK_INFRA_PWM3 17
+#define CLK_INFRA_PWM4 18
+#define CLK_INFRA_PWM 19
+#define CLK_INFRA_UART0 20
+#define CLK_INFRA_UART1 21
+#define CLK_INFRA_UART2 22
+#define CLK_INFRA_UART3 23
+#define CLK_INFRA_GCE_26M 24
+#define CLK_INFRA_CQ_DMA_FPC 25
+#define CLK_INFRA_BTIF 26
+#define CLK_INFRA_SPI0 27
+#define CLK_INFRA_MSDC0 28
+#define CLK_INFRA_MSDC1 29
+#define CLK_INFRA_MSDC2 30
+#define CLK_INFRA_MSDC0_SCK 31
+#define CLK_INFRA_DVFSRC 32
+#define CLK_INFRA_GCPU 33
+#define CLK_INFRA_TRNG 34
+#define CLK_INFRA_AUXADC 35
+#define CLK_INFRA_CPUM 36
+#define CLK_INFRA_CCIF1_AP 37
+#define CLK_INFRA_CCIF1_MD 38
+#define CLK_INFRA_AUXADC_MD 39
+#define CLK_INFRA_MSDC1_SCK 40
+#define CLK_INFRA_MSDC2_SCK 41
+#define CLK_INFRA_AP_DMA 42
+#define CLK_INFRA_XIU 43
+#define CLK_INFRA_DEVICE_APC 44
+#define CLK_INFRA_CCIF_AP 45
+#define CLK_INFRA_DEBUGSYS 46
+#define CLK_INFRA_AUDIO 47
+#define CLK_INFRA_CCIF_MD 48
+#define CLK_INFRA_DXCC_SEC_CORE 49
+#define CLK_INFRA_DXCC_AO 50
+#define CLK_INFRA_DRAMC_F26M 51
+#define CLK_INFRA_IRTX 52
+#define CLK_INFRA_DISP_PWM 53
+#define CLK_INFRA_CLDMA_BCLK 54
+#define CLK_INFRA_AUDIO_26M_BCLK 55
+#define CLK_INFRA_SPI1 56
+#define CLK_INFRA_I2C4 57
+#define CLK_INFRA_MODEM_TEMP_SHARE 58
+#define CLK_INFRA_SPI2 59
+#define CLK_INFRA_SPI3 60
+#define CLK_INFRA_UNIPRO_SCK 61
+#define CLK_INFRA_UNIPRO_TICK 62
+#define CLK_INFRA_UFS_MP_SAP_BCLK 63
+#define CLK_INFRA_MD32_BCLK 64
+#define CLK_INFRA_SSPM 65
+#define CLK_INFRA_UNIPRO_MBIST 66
+#define CLK_INFRA_SSPM_BUS_HCLK 67
+#define CLK_INFRA_I2C5 68
+#define CLK_INFRA_I2C5_ARBITER 69
+#define CLK_INFRA_I2C5_IMM 70
+#define CLK_INFRA_I2C1_ARBITER 71
+#define CLK_INFRA_I2C1_IMM 72
+#define CLK_INFRA_I2C2_ARBITER 73
+#define CLK_INFRA_I2C2_IMM 74
+#define CLK_INFRA_SPI4 75
+#define CLK_INFRA_SPI5 76
+#define CLK_INFRA_CQ_DMA 77
+#define CLK_INFRA_UFS 78
+#define CLK_INFRA_AES_UFSFDE 79
+#define CLK_INFRA_UFS_TICK 80
+#define CLK_INFRA_MSDC0_SELF 81
+#define CLK_INFRA_MSDC1_SELF 82
+#define CLK_INFRA_MSDC2_SELF 83
+#define CLK_INFRA_SSPM_26M_SELF 84
+#define CLK_INFRA_SSPM_32K_SELF 85
+#define CLK_INFRA_UFS_AXI 86
+#define CLK_INFRA_I2C6 87
+#define CLK_INFRA_AP_MSDC0 88
+#define CLK_INFRA_MD_MSDC0 89
+#define CLK_INFRA_USB 90
+#define CLK_INFRA_DEVMPU_BCLK 91
+#define CLK_INFRA_CCIF2_AP 92
+#define CLK_INFRA_CCIF2_MD 93
+#define CLK_INFRA_CCIF3_AP 94
+#define CLK_INFRA_CCIF3_MD 95
+#define CLK_INFRA_SEJ_F13M 96
+#define CLK_INFRA_AES_BCLK 97
+#define CLK_INFRA_I2C7 98
+#define CLK_INFRA_I2C8 99
+#define CLK_INFRA_FBIST2FPC 100
+#define CLK_INFRA_NR_CLK 101
+
+/* MFGCFG */
+#define CLK_MFG_BG3D 0
+#define CLK_MFG_NR_CLK 1
+
+/* IMG */
+#define CLK_IMG_OWE 0
+#define CLK_IMG_WPE_B 1
+#define CLK_IMG_WPE_A 2
+#define CLK_IMG_MFB 3
+#define CLK_IMG_RSC 4
+#define CLK_IMG_DPE 5
+#define CLK_IMG_FDVT 6
+#define CLK_IMG_DIP 7
+#define CLK_IMG_LARB2 8
+#define CLK_IMG_LARB5 9
+#define CLK_IMG_NR_CLK 10
+
+/* MMSYS_CONFIG */
+#define CLK_MM_SMI_COMMON 0
+#define CLK_MM_SMI_LARB0 1
+#define CLK_MM_SMI_LARB1 2
+#define CLK_MM_GALS_COMM0 3
+#define CLK_MM_GALS_COMM1 4
+#define CLK_MM_GALS_CCU2MM 5
+#define CLK_MM_GALS_IPU12MM 6
+#define CLK_MM_GALS_IMG2MM 7
+#define CLK_MM_GALS_CAM2MM 8
+#define CLK_MM_GALS_IPU2MM 9
+#define CLK_MM_MDP_DL_TXCK 10
+#define CLK_MM_IPU_DL_TXCK 11
+#define CLK_MM_MDP_RDMA0 12
+#define CLK_MM_MDP_RDMA1 13
+#define CLK_MM_MDP_RSZ0 14
+#define CLK_MM_MDP_RSZ1 15
+#define CLK_MM_MDP_TDSHP 16
+#define CLK_MM_MDP_WROT0 17
+#define CLK_MM_FAKE_ENG 18
+#define CLK_MM_DISP_OVL0 19
+#define CLK_MM_DISP_OVL0_2L 20
+#define CLK_MM_DISP_OVL1_2L 21
+#define CLK_MM_DISP_RDMA0 22
+#define CLK_MM_DISP_RDMA1 23
+#define CLK_MM_DISP_WDMA0 24
+#define CLK_MM_DISP_COLOR0 25
+#define CLK_MM_DISP_CCORR0 26
+#define CLK_MM_DISP_AAL0 27
+#define CLK_MM_DISP_GAMMA0 28
+#define CLK_MM_DISP_DITHER0 29
+#define CLK_MM_DISP_SPLIT 30
+#define CLK_MM_DSI0_MM 31
+#define CLK_MM_DSI0_IF 32
+#define CLK_MM_DPI_MM 33
+#define CLK_MM_DPI_IF 34
+#define CLK_MM_FAKE_ENG2 35
+#define CLK_MM_MDP_DL_RX 36
+#define CLK_MM_IPU_DL_RX 37
+#define CLK_MM_26M 38
+#define CLK_MM_MMSYS_R2Y 39
+#define CLK_MM_DISP_RSZ 40
+#define CLK_MM_MDP_WDMA0 41
+#define CLK_MM_MDP_AAL 42
+#define CLK_MM_MDP_CCORR 43
+#define CLK_MM_DBI_MM 44
+#define CLK_MM_DBI_IF 45
+#define CLK_MM_NR_CLK 46
+
+/* VDEC_GCON */
+#define CLK_VDEC_VDEC 0
+#define CLK_VDEC_LARB1 1
+#define CLK_VDEC_NR_CLK 2
+
+/* VENC_GCON */
+#define CLK_VENC_LARB 0
+#define CLK_VENC_VENC 1
+#define CLK_VENC_JPGENC 2
+#define CLK_VENC_NR_CLK 3
+
+/* AUDIO */
+#define CLK_AUDIO_TML 0
+#define CLK_AUDIO_DAC_PREDIS 1
+#define CLK_AUDIO_DAC 2
+#define CLK_AUDIO_ADC 3
+#define CLK_AUDIO_APLL_TUNER 4
+#define CLK_AUDIO_APLL2_TUNER 5
+#define CLK_AUDIO_24M 6
+#define CLK_AUDIO_22M 7
+#define CLK_AUDIO_AFE 8
+#define CLK_AUDIO_I2S4 9
+#define CLK_AUDIO_I2S3 10
+#define CLK_AUDIO_I2S2 11
+#define CLK_AUDIO_I2S1 12
+#define CLK_AUDIO_PDN_ADDA6_ADC 13
+#define CLK_AUDIO_TDM 14
+#define CLK_AUDIO_NR_CLK 15
+
+/* IPU_CONN */
+#define CLK_IPU_CONN_IPU 0
+#define CLK_IPU_CONN_AHB 1
+#define CLK_IPU_CONN_AXI 2
+#define CLK_IPU_CONN_ISP 3
+#define CLK_IPU_CONN_CAM_ADL 4
+#define CLK_IPU_CONN_IMG_ADL 5
+#define CLK_IPU_CONN_DAP_RX 6
+#define CLK_IPU_CONN_APB2AXI 7
+#define CLK_IPU_CONN_APB2AHB 8
+#define CLK_IPU_CONN_IPU_CAB1TO2 9
+#define CLK_IPU_CONN_IPU1_CAB1TO2 10
+#define CLK_IPU_CONN_IPU2_CAB1TO2 11
+#define CLK_IPU_CONN_CAB3TO3 12
+#define CLK_IPU_CONN_CAB2TO1 13
+#define CLK_IPU_CONN_CAB3TO1_SLICE 14
+#define CLK_IPU_CONN_NR_CLK 15
+
+/* IPU_ADL */
+#define CLK_IPU_ADL_CABGEN 0
+#define CLK_IPU_ADL_NR_CLK 1
+
+/* IPU_CORE0 */
+#define CLK_IPU_CORE0_JTAG 0
+#define CLK_IPU_CORE0_AXI 1
+#define CLK_IPU_CORE0_IPU 2
+#define CLK_IPU_CORE0_NR_CLK 3
+
+/* IPU_CORE1 */
+#define CLK_IPU_CORE1_JTAG 0
+#define CLK_IPU_CORE1_AXI 1
+#define CLK_IPU_CORE1_IPU 2
+#define CLK_IPU_CORE1_NR_CLK 3
+
+/* MCUCFG */
+#define CLK_MCU_MP0_SEL 0
+#define CLK_MCU_MP2_SEL 1
+#define CLK_MCU_BUS_SEL 2
+#define CLK_MCU_NR_CLK 3
+
+#endif /* _DT_BINDINGS_CLK_MT8183_H */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2019 MediaTek Inc.
+ * Copyright (c) 2019 BayLibre, SAS.
+ * Author: James Liao <jamesjj.liao@mediatek.com>
+ */
+
+#ifndef _DT_BINDINGS_CLK_MT8516_H
+#define _DT_BINDINGS_CLK_MT8516_H
+
+/* APMIXEDSYS */
+
+#define CLK_APMIXED_ARMPLL 0
+#define CLK_APMIXED_MAINPLL 1
+#define CLK_APMIXED_UNIVPLL 2
+#define CLK_APMIXED_MMPLL 3
+#define CLK_APMIXED_APLL1 4
+#define CLK_APMIXED_APLL2 5
+#define CLK_APMIXED_NR_CLK 6
+
+/* INFRACFG */
+
+#define CLK_IFR_MUX1_SEL 0
+#define CLK_IFR_ETH_25M_SEL 1
+#define CLK_IFR_I2C0_SEL 2
+#define CLK_IFR_I2C1_SEL 3
+#define CLK_IFR_I2C2_SEL 4
+#define CLK_IFR_NR_CLK 5
+
+/* TOPCKGEN */
+
+#define CLK_TOP_CLK_NULL 0
+#define CLK_TOP_I2S_INFRA_BCK 1
+#define CLK_TOP_MEMPLL 2
+#define CLK_TOP_DMPLL 3
+#define CLK_TOP_MAINPLL_D2 4
+#define CLK_TOP_MAINPLL_D4 5
+#define CLK_TOP_MAINPLL_D8 6
+#define CLK_TOP_MAINPLL_D16 7
+#define CLK_TOP_MAINPLL_D11 8
+#define CLK_TOP_MAINPLL_D22 9
+#define CLK_TOP_MAINPLL_D3 10
+#define CLK_TOP_MAINPLL_D6 11
+#define CLK_TOP_MAINPLL_D12 12
+#define CLK_TOP_MAINPLL_D5 13
+#define CLK_TOP_MAINPLL_D10 14
+#define CLK_TOP_MAINPLL_D20 15
+#define CLK_TOP_MAINPLL_D40 16
+#define CLK_TOP_MAINPLL_D7 17
+#define CLK_TOP_MAINPLL_D14 18
+#define CLK_TOP_UNIVPLL_D2 19
+#define CLK_TOP_UNIVPLL_D4 20
+#define CLK_TOP_UNIVPLL_D8 21
+#define CLK_TOP_UNIVPLL_D16 22
+#define CLK_TOP_UNIVPLL_D3 23
+#define CLK_TOP_UNIVPLL_D6 24
+#define CLK_TOP_UNIVPLL_D12 25
+#define CLK_TOP_UNIVPLL_D24 26
+#define CLK_TOP_UNIVPLL_D5 27
+#define CLK_TOP_UNIVPLL_D20 28
+#define CLK_TOP_MMPLL380M 29
+#define CLK_TOP_MMPLL_D2 30
+#define CLK_TOP_MMPLL_200M 31
+#define CLK_TOP_USB_PHY48M 32
+#define CLK_TOP_APLL1 33
+#define CLK_TOP_APLL1_D2 34
+#define CLK_TOP_APLL1_D4 35
+#define CLK_TOP_APLL1_D8 36
+#define CLK_TOP_APLL2 37
+#define CLK_TOP_APLL2_D2 38
+#define CLK_TOP_APLL2_D4 39
+#define CLK_TOP_APLL2_D8 40
+#define CLK_TOP_CLK26M 41
+#define CLK_TOP_CLK26M_D2 42
+#define CLK_TOP_AHB_INFRA_D2 43
+#define CLK_TOP_NFI1X 44
+#define CLK_TOP_ETH_D2 45
+#define CLK_TOP_THEM 46
+#define CLK_TOP_APDMA 47
+#define CLK_TOP_I2C0 48
+#define CLK_TOP_I2C1 49
+#define CLK_TOP_AUXADC1 50
+#define CLK_TOP_NFI 51
+#define CLK_TOP_NFIECC 52
+#define CLK_TOP_DEBUGSYS 53
+#define CLK_TOP_PWM 54
+#define CLK_TOP_UART0 55
+#define CLK_TOP_UART1 56
+#define CLK_TOP_BTIF 57
+#define CLK_TOP_USB 58
+#define CLK_TOP_FLASHIF_26M 59
+#define CLK_TOP_AUXADC2 60
+#define CLK_TOP_I2C2 61
+#define CLK_TOP_MSDC0 62
+#define CLK_TOP_MSDC1 63
+#define CLK_TOP_NFI2X 64
+#define CLK_TOP_PMICWRAP_AP 65
+#define CLK_TOP_SEJ 66
+#define CLK_TOP_MEMSLP_DLYER 67
+#define CLK_TOP_SPI 68
+#define CLK_TOP_APXGPT 69
+#define CLK_TOP_AUDIO 70
+#define CLK_TOP_PMICWRAP_MD 71
+#define CLK_TOP_PMICWRAP_CONN 72
+#define CLK_TOP_PMICWRAP_26M 73
+#define CLK_TOP_AUX_ADC 74
+#define CLK_TOP_AUX_TP 75
+#define CLK_TOP_MSDC2 76
+#define CLK_TOP_RBIST 77
+#define CLK_TOP_NFI_BUS 78
+#define CLK_TOP_GCE 79
+#define CLK_TOP_TRNG 80
+#define CLK_TOP_SEJ_13M 81
+#define CLK_TOP_AES 82
+#define CLK_TOP_PWM_B 83
+#define CLK_TOP_PWM1_FB 84
+#define CLK_TOP_PWM2_FB 85
+#define CLK_TOP_PWM3_FB 86
+#define CLK_TOP_PWM4_FB 87
+#define CLK_TOP_PWM5_FB 88
+#define CLK_TOP_USB_1P 89
+#define CLK_TOP_FLASHIF_FREERUN 90
+#define CLK_TOP_66M_ETH 91
+#define CLK_TOP_133M_ETH 92
+#define CLK_TOP_FETH_25M 93
+#define CLK_TOP_FETH_50M 94
+#define CLK_TOP_FLASHIF_AXI 95
+#define CLK_TOP_USBIF 96
+#define CLK_TOP_UART2 97
+#define CLK_TOP_BSI 98
+#define CLK_TOP_RG_SPINOR 99
+#define CLK_TOP_RG_MSDC2 100
+#define CLK_TOP_RG_ETH 101
+#define CLK_TOP_RG_AUD1 102
+#define CLK_TOP_RG_AUD2 103
+#define CLK_TOP_RG_AUD_ENGEN1 104
+#define CLK_TOP_RG_AUD_ENGEN2 105
+#define CLK_TOP_RG_I2C 106
+#define CLK_TOP_RG_PWM_INFRA 107
+#define CLK_TOP_RG_AUD_SPDIF_IN 108
+#define CLK_TOP_RG_UART2 109
+#define CLK_TOP_RG_BSI 110
+#define CLK_TOP_RG_DBG_ATCLK 111
+#define CLK_TOP_RG_NFIECC 112
+#define CLK_TOP_RG_APLL1_D2_EN 113
+#define CLK_TOP_RG_APLL1_D4_EN 114
+#define CLK_TOP_RG_APLL1_D8_EN 115
+#define CLK_TOP_RG_APLL2_D2_EN 116
+#define CLK_TOP_RG_APLL2_D4_EN 117
+#define CLK_TOP_RG_APLL2_D8_EN 118
+#define CLK_TOP_APLL12_DIV0 119
+#define CLK_TOP_APLL12_DIV1 120
+#define CLK_TOP_APLL12_DIV2 121
+#define CLK_TOP_APLL12_DIV3 122
+#define CLK_TOP_APLL12_DIV4 123
+#define CLK_TOP_APLL12_DIV4B 124
+#define CLK_TOP_APLL12_DIV5 125
+#define CLK_TOP_APLL12_DIV5B 126
+#define CLK_TOP_APLL12_DIV6 127
+#define CLK_TOP_UART0_SEL 128
+#define CLK_TOP_EMI_DDRPHY_SEL 129
+#define CLK_TOP_AHB_INFRA_SEL 130
+#define CLK_TOP_MSDC0_SEL 131
+#define CLK_TOP_UART1_SEL 132
+#define CLK_TOP_MSDC1_SEL 133
+#define CLK_TOP_PMICSPI_SEL 134
+#define CLK_TOP_QAXI_AUD26M_SEL 135
+#define CLK_TOP_AUD_INTBUS_SEL 136
+#define CLK_TOP_NFI2X_PAD_SEL 137
+#define CLK_TOP_NFI1X_PAD_SEL 138
+#define CLK_TOP_DDRPHYCFG_SEL 139
+#define CLK_TOP_USB_78M_SEL 140
+#define CLK_TOP_SPINOR_SEL 141
+#define CLK_TOP_MSDC2_SEL 142
+#define CLK_TOP_ETH_SEL 143
+#define CLK_TOP_AUD1_SEL 144
+#define CLK_TOP_AUD2_SEL 145
+#define CLK_TOP_AUD_ENGEN1_SEL 146
+#define CLK_TOP_AUD_ENGEN2_SEL 147
+#define CLK_TOP_I2C_SEL 148
+#define CLK_TOP_AUD_I2S0_M_SEL 149
+#define CLK_TOP_AUD_I2S1_M_SEL 150
+#define CLK_TOP_AUD_I2S2_M_SEL 151
+#define CLK_TOP_AUD_I2S3_M_SEL 152
+#define CLK_TOP_AUD_I2S4_M_SEL 153
+#define CLK_TOP_AUD_I2S5_M_SEL 154
+#define CLK_TOP_AUD_SPDIF_B_SEL 155
+#define CLK_TOP_PWM_SEL 156
+#define CLK_TOP_SPI_SEL 157
+#define CLK_TOP_AUD_SPDIFIN_SEL 158
+#define CLK_TOP_UART2_SEL 159
+#define CLK_TOP_BSI_SEL 160
+#define CLK_TOP_DBG_ATCLK_SEL 161
+#define CLK_TOP_CSW_NFIECC_SEL 162
+#define CLK_TOP_NFIECC_SEL 163
+#define CLK_TOP_APLL12_CK_DIV0 164
+#define CLK_TOP_APLL12_CK_DIV1 165
+#define CLK_TOP_APLL12_CK_DIV2 166
+#define CLK_TOP_APLL12_CK_DIV3 167
+#define CLK_TOP_APLL12_CK_DIV4 168
+#define CLK_TOP_APLL12_CK_DIV4B 169
+#define CLK_TOP_APLL12_CK_DIV5 170
+#define CLK_TOP_APLL12_CK_DIV5B 171
+#define CLK_TOP_APLL12_CK_DIV6 172
+#define CLK_TOP_USB_78M 173
+#define CLK_TOP_MSDC0_INFRA 174
+#define CLK_TOP_MSDC1_INFRA 175
+#define CLK_TOP_MSDC2_INFRA 176
+#define CLK_TOP_NR_CLK 177
+
+#endif /* _DT_BINDINGS_CLK_MT8516_H */
#define GCC_MDP_TBU_CLK 138
#define GCC_QDSS_DAP_CLK 139
#define GCC_DCC_XO_CLK 140
+#define GCC_CDSP_CFG_AHB_CLK 143
+#define GCC_BIMC_CDSP_CLK 144
+#define GCC_CDSP_TBU_CLK 145
+#define GCC_CDSP_BIMC_CLK_SRC 146
#define GCC_GENI_IR_BCR 0
#define GCC_USB_HS_BCR 1
#define GCC_PCIE_0_LINK_DOWN_BCR 11
#define GCC_PCIEPHY_0_PHY_BCR 12
#define GCC_EMAC_BCR 13
+#define GCC_CDSP_RESTART 14
#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2019, Linaro Ltd
+ */
+
+#ifndef _DT_BINDINGS_CLK_TURING_QCS404_H
+#define _DT_BINDINGS_CLK_TURING_QCS404_H
+
+#define TURING_Q6SS_Q6_AXIM_CLK 0
+#define TURING_Q6SS_AHBM_AON_CLK 1
+#define TURING_WRAPPER_AON_CLK 2
+#define TURING_Q6SS_AHBS_AON_CLK 3
+#define TURING_WRAPPER_QOS_AHBS_AON_CLK 4
+
+#endif
#define CLK_I2C3 28
#define CLK_I2C4 29
#define CLK_LPTIMER 30
-
-#define END_PRIMARY_CLK_F7 31
+#define CLK_PLL_SRC 31
+#define CLK_DFSDM1 32
+#define CLK_ADFSDM1 33
+#define CLK_F769_DSI 34
+#define END_PRIMARY_CLK_F7 35
#endif
#define CLK_AVS 96
#define CLK_HDMI 97
#define CLK_GPU 98
-
+#define CLK_MBUS 99
#define CLK_IEP 100
#endif /* _DT_BINDINGS_CLK_SUN5I_H_ */
* specified at mount time and thus is implemented here.
*/
CGRP_CPUSET_CLONE_CHILDREN,
+
+ /* Control group has to be frozen. */
+ CGRP_FREEZE,
+
+ /* Cgroup is frozen. */
+ CGRP_FROZEN,
};
/* cgroup_root->flags */
struct cgroup *updated_next; /* NULL iff not on the list */
};
+struct cgroup_freezer_state {
+ /* Should the cgroup and its descendants be frozen. */
+ bool freeze;
+
+ /* Should the cgroup actually be frozen? */
+ int e_freeze;
+
+ /* Fields below are protected by css_set_lock */
+
+ /* Number of frozen descendant cgroups */
+ int nr_frozen_descendants;
+
+ /*
+ * Number of tasks, which are counted as frozen:
+ * frozen, SIGSTOPped, and PTRACEd.
+ */
+ int nr_frozen_tasks;
+};
+
struct cgroup {
/* self css with NULL ->ss, points back to this cgroup */
struct cgroup_subsys_state self;
* Dying cgroups are cgroups which were deleted by a user,
* but are still existing because someone else is holding a reference.
* max_descendants is a maximum allowed number of descent cgroups.
+ *
+ * nr_descendants and nr_dying_descendants are protected
+ * by cgroup_mutex and css_set_lock. It's fine to read them holding
+ * any of cgroup_mutex and css_set_lock; for writing both locks
+ * should be held.
*/
int nr_descendants;
int nr_dying_descendants;
/* If there is block congestion on this cgroup. */
atomic_t congestion_count;
+ /* Used to store internal freezer state */
+ struct cgroup_freezer_state freezer;
+
/* ids of the ancestors at each level including self */
int ancestor_ids[];
};
free_cgroup_ns(ns);
}
+#ifdef CONFIG_CGROUPS
+
+void cgroup_enter_frozen(void);
+void cgroup_leave_frozen(bool always_leave);
+void cgroup_update_frozen(struct cgroup *cgrp);
+void cgroup_freeze(struct cgroup *cgrp, bool freeze);
+void cgroup_freezer_migrate_task(struct task_struct *task, struct cgroup *src,
+ struct cgroup *dst);
+
+static inline bool cgroup_task_freeze(struct task_struct *task)
+{
+ bool ret;
+
+ if (task->flags & PF_KTHREAD)
+ return false;
+
+ rcu_read_lock();
+ ret = test_bit(CGRP_FREEZE, &task_dfl_cgroup(task)->flags);
+ rcu_read_unlock();
+
+ return ret;
+}
+
+static inline bool cgroup_task_frozen(struct task_struct *task)
+{
+ return task->frozen;
+}
+
+#else /* !CONFIG_CGROUPS */
+
+static inline void cgroup_enter_frozen(void) { }
+static inline void cgroup_leave_frozen(bool always_leave) { }
+static inline bool cgroup_task_freeze(struct task_struct *task)
+{
+ return false;
+}
+static inline bool cgroup_task_frozen(struct task_struct *task)
+{
+ return false;
+}
+
+#endif /* !CONFIG_CGROUPS */
+
#endif /* _LINUX_CGROUP_H */
#define CLK_SET_RATE_PARENT BIT(2) /* propagate rate change up one level */
#define CLK_IGNORE_UNUSED BIT(3) /* do not gate even if unused */
/* unused */
-#define CLK_IS_BASIC BIT(5) /* Basic clk, can't do a to_clk_foo() */
+ /* unused */
#define CLK_GET_RATE_NOCACHE BIT(6) /* do not use the cached clk rate */
#define CLK_SET_RATE_NO_REPARENT BIT(7) /* don't re-parent on rate change */
#define CLK_GET_ACCURACY_NOCACHE BIT(8) /* do not use the cached clk accuracy */
void (*debug_init)(struct clk_hw *hw, struct dentry *dentry);
};
+/**
+ * struct clk_parent_data - clk parent information
+ * @hw: parent clk_hw pointer (used for clk providers with internal clks)
+ * @fw_name: parent name local to provider registering clk
+ * @name: globally unique parent name (used as a fallback)
+ * @index: parent index local to provider registering clk (if @fw_name absent)
+ */
+struct clk_parent_data {
+ const struct clk_hw *hw;
+ const char *fw_name;
+ const char *name;
+ int index;
+};
+
/**
* struct clk_init_data - holds init data that's common to all clocks and is
* shared between the clock provider and the common clock framework.
* @name: clock name
* @ops: operations this clock supports
* @parent_names: array of string names for all possible parents
+ * @parent_data: array of parent data for all possible parents (when some
+ * parents are external to the clk controller)
+ * @parent_hws: array of pointers to all possible parents (when all parents
+ * are internal to the clk controller)
* @num_parents: number of possible parents
* @flags: framework-level hints and quirks
*/
struct clk_init_data {
const char *name;
const struct clk_ops *ops;
+ /* Only one of the following three should be assigned */
const char * const *parent_names;
+ const struct clk_parent_data *parent_data;
+ const struct clk_hw **parent_hws;
u8 num_parents;
unsigned long flags;
};
struct clk_hw hw;
unsigned long fixed_rate;
unsigned long fixed_accuracy;
- u8 flags;
};
#define to_clk_fixed_rate(_hw) container_of(_hw, struct clk_fixed_rate, hw)
* of this register, and mask of gate bits are in higher 16-bit of this
* register. While setting the gate bits, higher 16-bit should also be
* updated to indicate changing gate bits.
+ * CLK_GATE_BIG_ENDIAN - by default little endian register accesses are used for
+ * the gate register. Setting this flag makes the register accesses big
+ * endian.
*/
struct clk_gate {
struct clk_hw hw;
#define CLK_GATE_SET_TO_DISABLE BIT(0)
#define CLK_GATE_HIWORD_MASK BIT(1)
+#define CLK_GATE_BIG_ENDIAN BIT(2)
extern const struct clk_ops clk_gate_ops;
struct clk *clk_register_gate(struct device *dev, const char *name,
* CLK_DIVIDER_MAX_AT_ZERO - For dividers which are like CLK_DIVIDER_ONE_BASED
* except when the value read from the register is zero, the divisor is
* 2^width of the field.
+ * CLK_DIVIDER_BIG_ENDIAN - By default little endian register accesses are used
+ * for the divider register. Setting this flag makes the register accesses
+ * big endian.
*/
struct clk_divider {
struct clk_hw hw;
#define CLK_DIVIDER_ROUND_CLOSEST BIT(4)
#define CLK_DIVIDER_READ_ONLY BIT(5)
#define CLK_DIVIDER_MAX_AT_ZERO BIT(6)
+#define CLK_DIVIDER_BIG_ENDIAN BIT(7)
extern const struct clk_ops clk_divider_ops;
extern const struct clk_ops clk_divider_ro_ops;
* register, and mask of mux bits are in higher 16-bit of this register.
* While setting the mux bits, higher 16-bit should also be updated to
* indicate changing mux bits.
+ * CLK_MUX_READ_ONLY - The mux registers can't be written, only read in the
+ * .get_parent clk_op.
* CLK_MUX_ROUND_CLOSEST - Use the parent rate that is closest to the desired
* frequency.
+ * CLK_MUX_BIG_ENDIAN - By default little endian register accesses are used for
+ * the mux register. Setting this flag makes the register accesses big
+ * endian.
*/
struct clk_mux {
struct clk_hw hw;
#define CLK_MUX_HIWORD_MASK BIT(2)
#define CLK_MUX_READ_ONLY BIT(3) /* mux can't be changed */
#define CLK_MUX_ROUND_CLOSEST BIT(4)
+#define CLK_MUX_BIG_ENDIAN BIT(5)
extern const struct clk_ops clk_mux_ops;
extern const struct clk_ops clk_mux_ro_ops;
* is the value read from the register. If CLK_FRAC_DIVIDER_ZERO_BASED
* is set then the numerator and denominator are both the value read
* plus one.
+ * CLK_FRAC_DIVIDER_BIG_ENDIAN - By default little endian register accesses are
+ * used for the divider register. Setting this flag makes the register
+ * accesses big endian.
*/
struct clk_fractional_divider {
struct clk_hw hw;
#define to_clk_fd(_hw) container_of(_hw, struct clk_fractional_divider, hw)
#define CLK_FRAC_DIVIDER_ZERO_BASED BIT(0)
+#define CLK_FRAC_DIVIDER_BIG_ENDIAN BIT(1)
extern const struct clk_ops clk_fractional_divider_ops;
struct clk *clk_register_fractional_divider(struct device *dev,
* leaving the parent rate unmodified.
* CLK_MULTIPLIER_ROUND_CLOSEST - Makes the best calculated divider to be
* rounded to the closest integer instead of the down one.
+ * CLK_MULTIPLIER_BIG_ENDIAN - By default little endian register accesses are
+ * used for the multiplier register. Setting this flag makes the register
+ * accesses big endian.
*/
struct clk_multiplier {
struct clk_hw hw;
#define CLK_MULTIPLIER_ZERO_BYPASS BIT(0)
#define CLK_MULTIPLIER_ROUND_CLOSEST BIT(1)
+#define CLK_MULTIPLIER_BIG_ENDIAN BIT(2)
extern const struct clk_ops clk_multiplier_ops;
unsigned long flags);
void clk_hw_unregister_composite(struct clk_hw *hw);
-/***
- * struct clk_gpio_gate - gpio gated clock
+/**
+ * struct clk_gpio - gpio gated clock
*
* @hw: handle between common and hardware-specific interfaces
* @gpiod: gpio descriptor
*
- * Clock with a gpio control for enabling and disabling the parent clock.
- * Implements .enable, .disable and .is_enabled
+ * Clock with a gpio control for enabling and disabling the parent clock
+ * or switching between two parents by asserting or deasserting the gpio.
+ *
+ * Implements .enable, .disable and .is_enabled or
+ * .get_parent, .set_parent and .determine_rate depending on which clk_ops
+ * is used.
*/
-
struct clk_gpio {
struct clk_hw hw;
struct gpio_desc *gpiod;
unsigned long flags);
void clk_hw_unregister_gpio_gate(struct clk_hw *hw);
-/**
- * struct clk_gpio_mux - gpio controlled clock multiplexer
- *
- * @hw: see struct clk_gpio
- * @gpiod: gpio descriptor to select the parent of this clock multiplexer
- *
- * Clock with a gpio control for selecting the parent clock.
- * Implements .get_parent, .set_parent and .determine_rate
- */
-
extern const struct clk_ops clk_gpio_mux_ops;
struct clk *clk_register_gpio_mux(struct device *dev, const char *name,
const char * const *parent_names, u8 num_parents, struct gpio_desc *gpiod,
unsigned long flags);
void clk_hw_unregister_gpio_mux(struct clk_hw *hw);
-/**
- * clk_register - allocate a new clock, register it and return an opaque cookie
- * @dev: device that is registering this clock
- * @hw: link to hardware-specific clock data
- *
- * clk_register is the primary interface for populating the clock tree with new
- * clock nodes. It returns a pointer to the newly allocated struct clk which
- * cannot be dereferenced by driver code but may be used in conjuction with the
- * rest of the clock API. In the event of an error clk_register will return an
- * error code; drivers must test for an error code after calling clk_register.
- */
struct clk *clk_register(struct device *dev, struct clk_hw *hw);
struct clk *devm_clk_register(struct device *dev, struct clk_hw *hw);
int __must_check clk_hw_register(struct device *dev, struct clk_hw *hw);
int __must_check devm_clk_hw_register(struct device *dev, struct clk_hw *hw);
+int __must_check of_clk_hw_register(struct device_node *node, struct clk_hw *hw);
void clk_unregister(struct clk *clk);
void devm_clk_unregister(struct device *dev, struct clk *clk);
}
#endif /* CONFIG_OF */
-/*
- * wrap access to peripherals in accessor routines
- * for improved portability across platforms
- */
-
-#if IS_ENABLED(CONFIG_PPC)
-
-static inline u32 clk_readl(u32 __iomem *reg)
-{
- return ioread32be(reg);
-}
-
-static inline void clk_writel(u32 val, u32 __iomem *reg)
-{
- iowrite32be(val, reg);
-}
-
-#else /* platform dependent I/O accessors */
-
-static inline u32 clk_readl(u32 __iomem *reg)
-{
- return readl(reg);
-}
-
-static inline void clk_writel(u32 val, u32 __iomem *reg)
-{
- writel(val, reg);
-}
-
-#endif /* platform dependent I/O accessors */
-
void clk_gate_restore_context(struct clk_hw *hw);
#endif /* CONFIG_COMMON_CLK */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2018-2019 SiFive, Inc.
+ * Wesley Terpstra
+ * Paul Walmsley
+ */
+
+#ifndef __LINUX_CLK_ANALOGBITS_WRPLL_CLN28HPC_H
+#define __LINUX_CLK_ANALOGBITS_WRPLL_CLN28HPC_H
+
+#include <linux/types.h>
+
+/* DIVQ_VALUES: number of valid DIVQ values */
+#define DIVQ_VALUES 6
+
+/*
+ * Bit definitions for struct wrpll_cfg.flags
+ *
+ * WRPLL_FLAGS_BYPASS_FLAG: if set, the PLL is either in bypass, or should be
+ * programmed to enter bypass
+ * WRPLL_FLAGS_RESET_FLAG: if set, the PLL is in reset
+ * WRPLL_FLAGS_INT_FEEDBACK_FLAG: if set, the PLL is configured for internal
+ * feedback mode
+ * WRPLL_FLAGS_EXT_FEEDBACK_FLAG: if set, the PLL is configured for external
+ * feedback mode (not yet supported by this driver)
+ */
+#define WRPLL_FLAGS_BYPASS_SHIFT 0
+#define WRPLL_FLAGS_BYPASS_MASK BIT(WRPLL_FLAGS_BYPASS_SHIFT)
+#define WRPLL_FLAGS_RESET_SHIFT 1
+#define WRPLL_FLAGS_RESET_MASK BIT(WRPLL_FLAGS_RESET_SHIFT)
+#define WRPLL_FLAGS_INT_FEEDBACK_SHIFT 2
+#define WRPLL_FLAGS_INT_FEEDBACK_MASK BIT(WRPLL_FLAGS_INT_FEEDBACK_SHIFT)
+#define WRPLL_FLAGS_EXT_FEEDBACK_SHIFT 3
+#define WRPLL_FLAGS_EXT_FEEDBACK_MASK BIT(WRPLL_FLAGS_EXT_FEEDBACK_SHIFT)
+
+/**
+ * struct wrpll_cfg - WRPLL configuration values
+ * @divr: reference divider value (6 bits), as presented to the PLL signals
+ * @divf: feedback divider value (9 bits), as presented to the PLL signals
+ * @divq: output divider value (3 bits), as presented to the PLL signals
+ * @flags: PLL configuration flags. See above for more information
+ * @range: PLL loop filter range. See below for more information
+ * @output_rate_cache: cached output rates, swept across DIVQ
+ * @parent_rate: PLL refclk rate for which values are valid
+ * @max_r: maximum possible R divider value, given @parent_rate
+ * @init_r: initial R divider value to start the search from
+ *
+ * @divr, @divq, @divq, @range represent what the PLL expects to see
+ * on its input signals. Thus @divr and @divf are the actual divisors
+ * minus one. @divq is a power-of-two divider; for example, 1 =
+ * divide-by-2 and 6 = divide-by-64. 0 is an invalid @divq value.
+ *
+ * When initially passing a struct wrpll_cfg record, the
+ * record should be zero-initialized with the exception of the @flags
+ * field. The only flag bits that need to be set are either
+ * WRPLL_FLAGS_INT_FEEDBACK or WRPLL_FLAGS_EXT_FEEDBACK.
+ */
+struct wrpll_cfg {
+ u8 divr;
+ u8 divq;
+ u8 range;
+ u8 flags;
+ u16 divf;
+/* private: */
+ u32 output_rate_cache[DIVQ_VALUES];
+ unsigned long parent_rate;
+ u8 max_r;
+ u8 init_r;
+};
+
+int wrpll_configure_for_rate(struct wrpll_cfg *c, u32 target_rate,
+ unsigned long parent_rate);
+
+unsigned int wrpll_calc_max_lock_us(const struct wrpll_cfg *c);
+
+unsigned long wrpll_calc_output_rate(const struct wrpll_cfg *c,
+ unsigned long parent_rate);
+
+#endif /* __LINUX_CLK_ANALOGBITS_WRPLL_CLN28HPC_H */
#define AT91_PMC_USBDIV_4 (2 << 28)
#define AT91_PMC_USB96M (1 << 28) /* Divider by 2 Enable (PLLB only) */
+#define AT91_PMC_CPU_CKR 0x28 /* CPU Clock Register */
+
#define AT91_PMC_MCKR 0x30 /* Master Clock Register */
#define AT91_PMC_CSS (3 << 0) /* Master Clock Selection */
#define AT91_PMC_CSS_SLOW (0 << 0)
#define AT91_PMC_PCR 0x10c /* Peripheral Control Register [some SAM9 and SAMA5] */
#define AT91_PMC_PCR_PID_MASK 0x3f
-#define AT91_PMC_PCR_GCKCSS_OFFSET 8
-#define AT91_PMC_PCR_GCKCSS_MASK (0x7 << AT91_PMC_PCR_GCKCSS_OFFSET)
-#define AT91_PMC_PCR_GCKCSS(n) ((n) << AT91_PMC_PCR_GCKCSS_OFFSET) /* GCK Clock Source Selection */
#define AT91_PMC_PCR_CMD (0x1 << 12) /* Command (read=0, write=1) */
-#define AT91_PMC_PCR_DIV_OFFSET 16
-#define AT91_PMC_PCR_DIV_MASK (0x3 << AT91_PMC_PCR_DIV_OFFSET)
-#define AT91_PMC_PCR_DIV(n) ((n) << AT91_PMC_PCR_DIV_OFFSET) /* Divisor Value */
-#define AT91_PMC_PCR_GCKDIV_OFFSET 20
-#define AT91_PMC_PCR_GCKDIV_MASK (0xff << AT91_PMC_PCR_GCKDIV_OFFSET)
-#define AT91_PMC_PCR_GCKDIV(n) ((n) << AT91_PMC_PCR_GCKDIV_OFFSET) /* Generated Clock Divisor Value */
+#define AT91_PMC_PCR_GCKDIV_MASK GENMASK(27, 20)
#define AT91_PMC_PCR_EN (0x1 << 28) /* Enable */
#define AT91_PMC_PCR_GCKEN (0x1 << 29) /* GCK Enable */
#define to_clk_hw_omap(_hw) container_of(_hw, struct clk_hw_omap, hw)
+bool omap2_clk_is_hw_omap(struct clk_hw *hw);
int omap2_clk_disable_autoidle_all(void);
int omap2_clk_enable_autoidle_all(void);
int omap2_clk_allow_idle(struct clk *clk);
#define TI_CLK_DISABLE_CLKDM_CONTROL BIT(2)
#define TI_CLK_ERRATA_I810 BIT(3)
#define TI_CLK_CLKCTRL_COMPAT BIT(4)
+#define TI_CLK_DEVICE_TYPE_GP BIT(5)
void ti_clk_setup_features(struct ti_clk_features *features);
const struct ti_clk_features *ti_clk_get_features(void);
static inline struct device_node *dev_of_node(struct device *dev)
{
- if (!IS_ENABLED(CONFIG_OF))
+ if (!IS_ENABLED(CONFIG_OF) || !dev)
return NULL;
return dev->of_node;
}
--- /dev/null
+/*
+ * fence-chain: chain fences together in a timeline
+ *
+ * Copyright (C) 2018 Advanced Micro Devices, Inc.
+ * Authors:
+ * Christian König <christian.koenig@amd.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ */
+
+#ifndef __LINUX_DMA_FENCE_CHAIN_H
+#define __LINUX_DMA_FENCE_CHAIN_H
+
+#include <linux/dma-fence.h>
+#include <linux/irq_work.h>
+
+/**
+ * struct dma_fence_chain - fence to represent an node of a fence chain
+ * @base: fence base class
+ * @lock: spinlock for fence handling
+ * @prev: previous fence of the chain
+ * @prev_seqno: original previous seqno before garbage collection
+ * @fence: encapsulated fence
+ * @cb: callback structure for signaling
+ * @work: irq work item for signaling
+ */
+struct dma_fence_chain {
+ struct dma_fence base;
+ spinlock_t lock;
+ struct dma_fence __rcu *prev;
+ u64 prev_seqno;
+ struct dma_fence *fence;
+ struct dma_fence_cb cb;
+ struct irq_work work;
+};
+
+extern const struct dma_fence_ops dma_fence_chain_ops;
+
+/**
+ * to_dma_fence_chain - cast a fence to a dma_fence_chain
+ * @fence: fence to cast to a dma_fence_array
+ *
+ * Returns NULL if the fence is not a dma_fence_chain,
+ * or the dma_fence_chain otherwise.
+ */
+static inline struct dma_fence_chain *
+to_dma_fence_chain(struct dma_fence *fence)
+{
+ if (!fence || fence->ops != &dma_fence_chain_ops)
+ return NULL;
+
+ return container_of(fence, struct dma_fence_chain, base);
+}
+
+/**
+ * dma_fence_chain_for_each - iterate over all fences in chain
+ * @iter: current fence
+ * @head: starting point
+ *
+ * Iterate over all fences in the chain. We keep a reference to the current
+ * fence while inside the loop which must be dropped when breaking out.
+ */
+#define dma_fence_chain_for_each(iter, head) \
+ for (iter = dma_fence_get(head); iter; \
+ iter = dma_fence_chain_walk(iter))
+
+struct dma_fence *dma_fence_chain_walk(struct dma_fence *fence);
+int dma_fence_chain_find_seqno(struct dma_fence **pfence, uint64_t seqno);
+void dma_fence_chain_init(struct dma_fence_chain *chain,
+ struct dma_fence *prev,
+ struct dma_fence *fence,
+ uint64_t seqno);
+
+#endif /* __LINUX_DMA_FENCE_CHAIN_H */
*
*/
struct dma_fence_ops {
+ /**
+ * @use_64bit_seqno:
+ *
+ * True if this dma_fence implementation uses 64bit seqno, false
+ * otherwise.
+ */
+ bool use_64bit_seqno;
+
/**
* @get_driver_name:
*
* __dma_fence_is_later - return if f1 is chronologically later than f2
* @f1: the first fence's seqno
* @f2: the second fence's seqno from the same context
+ * @ops: dma_fence_ops associated with the seqno
*
* Returns true if f1 is chronologically later than f2. Both fences must be
* from the same context, since a seqno is not common across contexts.
*/
-static inline bool __dma_fence_is_later(u64 f1, u64 f2)
+static inline bool __dma_fence_is_later(u64 f1, u64 f2,
+ const struct dma_fence_ops *ops)
{
/* This is for backward compatibility with drivers which can only handle
- * 32bit sequence numbers. Use a 64bit compare when any of the higher
- * bits are none zero, otherwise use a 32bit compare with wrap around
- * handling.
+ * 32bit sequence numbers. Use a 64bit compare when the driver says to
+ * do so.
*/
- if (upper_32_bits(f1) || upper_32_bits(f2))
+ if (ops->use_64bit_seqno)
return f1 > f2;
return (int)(lower_32_bits(f1) - lower_32_bits(f2)) > 0;
if (WARN_ON(f1->context != f2->context))
return false;
- return __dma_fence_is_later(f1->seqno, f2->seqno);
+ return __dma_fence_is_later(f1->seqno, f2->seqno, f1->ops);
}
/**
static inline const struct dma_map_ops *get_dma_ops(struct device *dev)
{
- if (dev && dev->dma_ops)
+ if (dev->dma_ops)
return dev->dma_ops;
- return get_arch_dma_ops(dev ? dev->bus : NULL);
+ return get_arch_dma_ops(dev->bus);
}
static inline void set_dma_ops(struct device *dev,
static inline u64 dma_get_mask(struct device *dev)
{
- if (dev && dev->dma_mask && *dev->dma_mask)
+ if (dev->dma_mask && *dev->dma_mask)
return *dev->dma_mask;
return DMA_BIT_MASK(32);
}
}
#endif /* CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL */
+#ifdef CONFIG_ARCH_HAS_DMA_PREP_COHERENT
void arch_dma_prep_coherent(struct page *page, size_t size);
+#else
+static inline void arch_dma_prep_coherent(struct page *page, size_t size)
+{
+}
+#endif /* CONFIG_ARCH_HAS_DMA_PREP_COHERENT */
#endif /* _LINUX_DMA_NONCOHERENT_H */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Definitions for the Intel integrated DMA 64-bit
+ *
+ * Copyright (C) 2019 Intel Corporation
+ */
+
+#ifndef __LINUX_DMA_IDMA64_H__
+#define __LINUX_DMA_IDMA64_H__
+
+/* Platform driver name */
+#define LPSS_IDMA64_DRIVER_NAME "idma64"
+
+#endif /* __LINUX_DMA_IDMA64_H__ */
const struct net_device *dev,
const char *fmt, ...);
+struct ib_device;
+
+extern __printf(3, 4)
+void __dynamic_ibdev_dbg(struct _ddebug *descriptor,
+ const struct ib_device *ibdev,
+ const char *fmt, ...);
+
#define DEFINE_DYNAMIC_DEBUG_METADATA(name, fmt) \
static struct _ddebug __aligned(8) \
__attribute__((section("__verbose"))) name = { \
_dynamic_func_call(fmt, __dynamic_netdev_dbg, \
dev, fmt, ##__VA_ARGS__)
+#define dynamic_ibdev_dbg(dev, fmt, ...) \
+ _dynamic_func_call(fmt, __dynamic_ibdev_dbg, \
+ dev, fmt, ##__VA_ARGS__)
+
#define dynamic_hex_dump(prefix_str, prefix_type, rowsize, \
groupsize, buf, len, ascii) \
_dynamic_func_call_no_desc(__builtin_constant_p(prefix_str) ? prefix_str : "hexdump", \
minimum 5 us for standard-mode I2C and SMBus,
maximum 50 us for SMBus */
int timeout; /* in jiffies */
+ bool can_do_atomic; /* callbacks don't sleep, we can be atomic */
};
int i2c_bit_add_bus(struct i2c_adapter *);
* @master_xfer: Issue a set of i2c transactions to the given I2C adapter
* defined by the msgs array, with num messages available to transfer via
* the adapter specified by adap.
+ * @master_xfer_atomic: same as @master_xfer. Yet, only using atomic context
+ * so e.g. PMICs can be accessed very late before shutdown. Optional.
* @smbus_xfer: Issue smbus transactions to the given I2C adapter. If this
* is not present, then the bus layer will try and convert the SMBus calls
* into I2C transfers instead.
+ * @smbus_xfer_atomic: same as @smbus_xfer. Yet, only using atomic context
+ * so e.g. PMICs can be accessed very late before shutdown. Optional.
* @functionality: Return the flags that this algorithm/adapter pair supports
* from the I2C_FUNC_* flags.
* @reg_slave: Register given client to I2C slave mode of this adapter
* be addressed using the same bus algorithms - i.e. bit-banging or the PCF8584
* to name two of the most common.
*
- * The return codes from the @master_xfer field should indicate the type of
- * error code that occurred during the transfer, as documented in the kernel
- * Documentation file Documentation/i2c/fault-codes.
+ * The return codes from the @master_xfer{_atomic} fields should indicate the
+ * type of error code that occurred during the transfer, as documented in the
+ * Kernel Documentation file Documentation/i2c/fault-codes.
*/
struct i2c_algorithm {
- /* If an adapter algorithm can't do I2C-level access, set master_xfer
- to NULL. If an adapter algorithm can do SMBus access, set
- smbus_xfer. If set to NULL, the SMBus protocol is simulated
- using common I2C messages */
- /* master_xfer should return the number of messages successfully
- processed, or a negative value on error */
+ /*
+ * If an adapter algorithm can't do I2C-level access, set master_xfer
+ * to NULL. If an adapter algorithm can do SMBus access, set
+ * smbus_xfer. If set to NULL, the SMBus protocol is simulated
+ * using common I2C messages.
+ *
+ * master_xfer should return the number of messages successfully
+ * processed, or a negative value on error
+ */
int (*master_xfer)(struct i2c_adapter *adap, struct i2c_msg *msgs,
int num);
- int (*smbus_xfer) (struct i2c_adapter *adap, u16 addr,
- unsigned short flags, char read_write,
- u8 command, int size, union i2c_smbus_data *data);
+ int (*master_xfer_atomic)(struct i2c_adapter *adap,
+ struct i2c_msg *msgs, int num);
+ int (*smbus_xfer)(struct i2c_adapter *adap, u16 addr,
+ unsigned short flags, char read_write,
+ u8 command, int size, union i2c_smbus_data *data);
+ int (*smbus_xfer_atomic)(struct i2c_adapter *adap, u16 addr,
+ unsigned short flags, char read_write,
+ u8 command, int size, union i2c_smbus_data *data);
/* To determine what the adapter supports */
- u32 (*functionality) (struct i2c_adapter *);
+ u32 (*functionality)(struct i2c_adapter *adap);
#if IS_ENABLED(CONFIG_I2C_SLAVE)
int (*reg_slave)(struct i2c_client *client);
int retries;
struct device dev; /* the adapter device */
unsigned long locked_flags; /* owned by the I2C core */
-#define I2C_ALF_IS_SUSPENDED 0
+#define I2C_ALF_IS_SUSPENDED 0
+#define I2C_ALF_SUSPEND_REPORTED 1
int nr;
char name[48];
ARM_64_LPAE_S1,
ARM_64_LPAE_S2,
ARM_V7S,
+ ARM_MALI_LPAE,
IO_PGTABLE_NUM_FMTS,
};
u32 nmrr;
u32 prrr;
} arm_v7s_cfg;
+
+ struct {
+ u64 transtab;
+ u64 memattr;
+ } arm_mali_lpae_cfg;
};
};
extern struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s1_init_fns;
extern struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s2_init_fns;
extern struct io_pgtable_init_fns io_pgtable_arm_v7s_init_fns;
+extern struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns;
#endif /* __IO_PGTABLE_H */
int noresvport;
struct net *net;
const struct nlmclnt_operations *nlmclnt_ops;
+ const struct cred *cred;
};
/*
struct nsm_handle *h_nsmhandle; /* NSM status handle */
char *h_addrbuf; /* address eyecatcher */
struct net *net; /* host net */
+ const struct cred *h_cred;
char nodename[UNX_MAXNODENAME + 1];
const struct nlmclnt_operations *h_nlmclnt_ops; /* Callback ops for NLM users */
};
const u32 version,
const char *hostname,
int noresvport,
- struct net *net);
+ struct net *net,
+ const struct cred *cred);
void nlmclnt_release_host(struct nlm_host *);
struct nlm_host *nlmsvc_lookup_host(const struct svc_rqst *rqstp,
const char *hostname,
#define DIV64_U64_ROUND_UP(ll, d) \
({ u64 _tmp = (d); div64_u64((ll) + _tmp - 1, _tmp); })
+/**
+ * DIV64_U64_ROUND_CLOSEST - unsigned 64bit divide with 64bit divisor rounded to nearest integer
+ * @dividend: unsigned 64bit dividend
+ * @divisor: unsigned 64bit divisor
+ *
+ * Divide unsigned 64bit dividend by unsigned 64bit divisor
+ * and round to closest integer.
+ *
+ * Return: dividend / divisor rounded to nearest integer
+ */
+#define DIV64_U64_ROUND_CLOSEST(dividend, divisor) \
+ ({ u64 _tmp = (divisor); div64_u64((dividend) + _tmp / 2, _tmp); })
+
#endif /* _LINUX_MATH64_H */
#endif
struct mlx5_clock clock;
struct mlx5_ib_clock_info *clock_info;
- struct page *clock_info_page;
struct mlx5_fw_tracer *tracer;
};
#endif /* CONFIG_LIVEPATCH */
bool is_module_sig_enforced(void);
+void set_module_sig_enforced(void);
#else /* !CONFIG_MODULES... */
return false;
}
+static inline void set_module_sig_enforced(void)
+{
+}
+
/* Dereference module function descriptor */
static inline
void *dereference_module_function_descriptor(struct module *mod, void *ptr)
fmode_t mode;
unsigned long flags;
-#define NFS_CONTEXT_ERROR_WRITE (0)
#define NFS_CONTEXT_RESEND_WRITES (1)
#define NFS_CONTEXT_BAD (2)
#define NFS_CONTEXT_UNLOCK (3)
struct nfs_iostats __percpu *io_stats; /* I/O statistics */
atomic_long_t writeback; /* number of writeback pages */
int flags; /* various flags */
+
+/* The following are for internal use only. Also see uapi/linux/nfs_mount.h */
+#define NFS_MOUNT_LOOKUP_CACHE_NONEG 0x10000
+#define NFS_MOUNT_LOOKUP_CACHE_NONE 0x20000
+#define NFS_MOUNT_NORESVPORT 0x40000
+#define NFS_MOUNT_LEGACY_INTERFACE 0x80000
+#define NFS_MOUNT_LOCAL_FLOCK 0x100000
+#define NFS_MOUNT_LOCAL_FCNTL 0x200000
+#define NFS_MOUNT_SOFTERR 0x400000
+
unsigned int caps; /* server capabilities */
unsigned int rsize; /* read size */
unsigned int rpages; /* read size (in pages) */
/* XDR related information */
unsigned int read_hdrsize;
+
+ /* User namespace info */
+ const struct cred *cred;
};
/* Server capabilities */
struct nfs_page {
struct list_head wb_list; /* Defines state of page: */
struct page *wb_page; /* page to read in/write out */
- struct nfs_open_context *wb_context; /* File state context info */
struct nfs_lock_context *wb_lock_context; /* lock context info */
pgoff_t wb_index; /* Offset >> PAGE_SHIFT */
unsigned int wb_offset, /* Offset & ~PAGE_MASK */
struct nfs_write_verifier wb_verf; /* Commit cookie */
struct nfs_page *wb_this_page; /* list of reqs for this page */
struct nfs_page *wb_head; /* head pointer for req list */
+ unsigned short wb_nio; /* Number of I/O attempts */
};
struct nfs_pageio_descriptor;
};
struct nfs_pageio_descriptor {
- unsigned char pg_moreio : 1;
struct inode *pg_inode;
const struct nfs_pageio_ops *pg_ops;
const struct nfs_rw_ops *pg_rw_ops;
struct nfs_pgio_mirror pg_mirrors_static[1];
struct nfs_pgio_mirror *pg_mirrors_dynamic;
u32 pg_mirror_idx; /* current mirror */
+ unsigned short pg_maxretrans;
+ unsigned char pg_moreio : 1;
};
/* arbitrarily selected limit to number of mirrors */
extern struct nfs_page *nfs_create_request(struct nfs_open_context *ctx,
struct page *page,
- struct nfs_page *last,
unsigned int offset,
unsigned int count);
extern void nfs_release_request(struct nfs_page *);
return (((loff_t)req->wb_index) << PAGE_SHIFT) + req->wb_offset;
}
+static inline struct nfs_open_context *
+nfs_req_openctx(struct nfs_page *req)
+{
+ return req->wb_lock_context->open_context;
+}
+
#endif /* _LINUX_NFS_PAGE_H */
#define type_max(T) ((T)((__type_half_max(T) - 1) + __type_half_max(T)))
#define type_min(T) ((T)((T)-type_max(T)-(T)1))
+/*
+ * Avoids triggering -Wtype-limits compilation warning,
+ * while using unsigned data types to check a < 0.
+ */
+#define is_non_negative(a) ((a) > 0 || (a) == 0)
+#define is_negative(a) (!(is_non_negative(a)))
#ifdef COMPILER_HAS_GENERIC_BUILTIN_OVERFLOW
/*
typeof(d) _d = d; \
u64 _a_full = _a; \
unsigned int _to_shift = \
- _s >= 0 && _s < 8 * sizeof(*d) ? _s : 0; \
+ is_non_negative(_s) && _s < 8 * sizeof(*d) ? _s : 0; \
*_d = (_a_full << _to_shift); \
- (_to_shift != _s || *_d < 0 || _a < 0 || \
- (*_d >> _to_shift) != _a); \
+ (_to_shift != _s || is_negative(*_d) || is_negative(_a) || \
+ (*_d >> _to_shift) != _a); \
})
/**
+++ /dev/null
-/*
- *
- * pca954x.h - I2C multiplexer/switch support
- *
- * Copyright (c) 2008-2009 Rodolfo Giometti <giometti@linux.it>
- * Copyright (c) 2008-2009 Eurotech S.p.A. <info@eurotech.it>
- * Michael Lawnick <michael.lawnick.ext@nsn.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
- */
-
-
-#ifndef _LINUX_I2C_PCA954X_H
-#define _LINUX_I2C_PCA954X_H
-
-/* Platform data for the PCA954x I2C multiplexers */
-
-/* Per channel initialisation data:
- * @adap_id: bus number for the adapter. 0 = don't care
- * @deselect_on_exit: set this entry to 1, if your H/W needs deselection
- * of this channel after transaction.
- *
- */
-struct pca954x_platform_mode {
- int adap_id;
- unsigned int deselect_on_exit:1;
- unsigned int class;
-};
-
-/* Per mux/switch data, used with i2c_register_board_info */
-struct pca954x_platform_data {
- struct pca954x_platform_mode *modes;
- int num_modes;
-};
-
-#endif /* _LINUX_I2C_PCA954X_H */
* @obj: the reservation object
*
* Returns the exclusive fence (if any). Does NOT take a
- * reference. The obj->lock must be held.
+ * reference. Writers must hold obj->lock, readers may only
+ * hold a RCU read side lock.
*
* RETURNS
* The exclusive fence or NULL
int reset_control_assert(struct reset_control *rstc);
int reset_control_deassert(struct reset_control *rstc);
int reset_control_status(struct reset_control *rstc);
+int reset_control_acquire(struct reset_control *rstc);
+void reset_control_release(struct reset_control *rstc);
struct reset_control *__of_reset_control_get(struct device_node *node,
const char *id, int index, bool shared,
- bool optional);
+ bool optional, bool acquired);
struct reset_control *__reset_control_get(struct device *dev, const char *id,
int index, bool shared,
- bool optional);
+ bool optional, bool acquired);
void reset_control_put(struct reset_control *rstc);
int __device_reset(struct device *dev, bool optional);
struct reset_control *__devm_reset_control_get(struct device *dev,
const char *id, int index, bool shared,
- bool optional);
+ bool optional, bool acquired);
struct reset_control *devm_reset_control_array_get(struct device *dev,
bool shared, bool optional);
struct reset_control *of_reset_control_array_get(struct device_node *np,
- bool shared, bool optional);
+ bool shared, bool optional,
+ bool acquired);
int reset_control_get_count(struct device *dev);
return 0;
}
+static inline int reset_control_acquire(struct reset_control *rstc)
+{
+ return 0;
+}
+
+static inline void reset_control_release(struct reset_control *rstc)
+{
+}
+
static inline void reset_control_put(struct reset_control *rstc)
{
}
static inline struct reset_control *__of_reset_control_get(
struct device_node *node,
const char *id, int index, bool shared,
- bool optional)
+ bool optional, bool acquired)
{
return optional ? NULL : ERR_PTR(-ENOTSUPP);
}
static inline struct reset_control *__reset_control_get(
struct device *dev, const char *id,
- int index, bool shared, bool optional)
+ int index, bool shared, bool optional,
+ bool acquired)
{
return optional ? NULL : ERR_PTR(-ENOTSUPP);
}
static inline struct reset_control *__devm_reset_control_get(
struct device *dev, const char *id,
- int index, bool shared, bool optional)
+ int index, bool shared, bool optional,
+ bool acquired)
{
return optional ? NULL : ERR_PTR(-ENOTSUPP);
}
}
static inline struct reset_control *
-of_reset_control_array_get(struct device_node *np, bool shared, bool optional)
+of_reset_control_array_get(struct device_node *np, bool shared, bool optional,
+ bool acquired)
{
return optional ? NULL : ERR_PTR(-ENOTSUPP);
}
static inline struct reset_control *
__must_check reset_control_get_exclusive(struct device *dev, const char *id)
{
- return __reset_control_get(dev, id, 0, false, false);
+ return __reset_control_get(dev, id, 0, false, false, true);
+}
+
+/**
+ * reset_control_get_exclusive_released - Lookup and obtain a temoprarily
+ * exclusive reference to a reset
+ * controller.
+ * @dev: device to be reset by the controller
+ * @id: reset line name
+ *
+ * Returns a struct reset_control or IS_ERR() condition containing errno.
+ * reset-controls returned by this function must be acquired via
+ * reset_control_acquire() before they can be used and should be released
+ * via reset_control_release() afterwards.
+ *
+ * Use of id names is optional.
+ */
+static inline struct reset_control *
+__must_check reset_control_get_exclusive_released(struct device *dev,
+ const char *id)
+{
+ return __reset_control_get(dev, id, 0, false, false, false);
}
/**
static inline struct reset_control *reset_control_get_shared(
struct device *dev, const char *id)
{
- return __reset_control_get(dev, id, 0, true, false);
+ return __reset_control_get(dev, id, 0, true, false, false);
}
static inline struct reset_control *reset_control_get_optional_exclusive(
struct device *dev, const char *id)
{
- return __reset_control_get(dev, id, 0, false, true);
+ return __reset_control_get(dev, id, 0, false, true, true);
}
static inline struct reset_control *reset_control_get_optional_shared(
struct device *dev, const char *id)
{
- return __reset_control_get(dev, id, 0, true, true);
+ return __reset_control_get(dev, id, 0, true, true, false);
}
/**
static inline struct reset_control *of_reset_control_get_exclusive(
struct device_node *node, const char *id)
{
- return __of_reset_control_get(node, id, 0, false, false);
+ return __of_reset_control_get(node, id, 0, false, false, true);
}
/**
static inline struct reset_control *of_reset_control_get_shared(
struct device_node *node, const char *id)
{
- return __of_reset_control_get(node, id, 0, true, false);
+ return __of_reset_control_get(node, id, 0, true, false, false);
}
/**
static inline struct reset_control *of_reset_control_get_exclusive_by_index(
struct device_node *node, int index)
{
- return __of_reset_control_get(node, NULL, index, false, false);
+ return __of_reset_control_get(node, NULL, index, false, false, true);
}
/**
static inline struct reset_control *of_reset_control_get_shared_by_index(
struct device_node *node, int index)
{
- return __of_reset_control_get(node, NULL, index, true, false);
+ return __of_reset_control_get(node, NULL, index, true, false, false);
}
/**
__must_check devm_reset_control_get_exclusive(struct device *dev,
const char *id)
{
- return __devm_reset_control_get(dev, id, 0, false, false);
+ return __devm_reset_control_get(dev, id, 0, false, false, true);
+}
+
+/**
+ * devm_reset_control_get_exclusive_released - resource managed
+ * reset_control_get_exclusive_released()
+ * @dev: device to be reset by the controller
+ * @id: reset line name
+ *
+ * Managed reset_control_get_exclusive_released(). For reset controllers
+ * returned from this function, reset_control_put() is called automatically on
+ * driver detach.
+ *
+ * See reset_control_get_exclusive_released() for more information.
+ */
+static inline struct reset_control *
+__must_check devm_reset_control_get_exclusive_released(struct device *dev,
+ const char *id)
+{
+ return __devm_reset_control_get(dev, id, 0, false, false, false);
}
/**
static inline struct reset_control *devm_reset_control_get_shared(
struct device *dev, const char *id)
{
- return __devm_reset_control_get(dev, id, 0, true, false);
+ return __devm_reset_control_get(dev, id, 0, true, false, false);
}
static inline struct reset_control *devm_reset_control_get_optional_exclusive(
struct device *dev, const char *id)
{
- return __devm_reset_control_get(dev, id, 0, false, true);
+ return __devm_reset_control_get(dev, id, 0, false, true, true);
}
static inline struct reset_control *devm_reset_control_get_optional_shared(
struct device *dev, const char *id)
{
- return __devm_reset_control_get(dev, id, 0, true, true);
+ return __devm_reset_control_get(dev, id, 0, true, true, false);
}
/**
static inline struct reset_control *
devm_reset_control_get_exclusive_by_index(struct device *dev, int index)
{
- return __devm_reset_control_get(dev, NULL, index, false, false);
+ return __devm_reset_control_get(dev, NULL, index, false, false, true);
}
/**
static inline struct reset_control *
devm_reset_control_get_shared_by_index(struct device *dev, int index)
{
- return __devm_reset_control_get(dev, NULL, index, true, false);
+ return __devm_reset_control_get(dev, NULL, index, true, false, false);
}
/*
static inline struct reset_control *
of_reset_control_array_get_exclusive(struct device_node *node)
{
- return of_reset_control_array_get(node, false, false);
+ return of_reset_control_array_get(node, false, false, true);
+}
+
+static inline struct reset_control *
+of_reset_control_array_get_exclusive_released(struct device_node *node)
+{
+ return of_reset_control_array_get(node, false, false, false);
}
static inline struct reset_control *
of_reset_control_array_get_shared(struct device_node *node)
{
- return of_reset_control_array_get(node, true, false);
+ return of_reset_control_array_get(node, true, false, true);
}
static inline struct reset_control *
of_reset_control_array_get_optional_exclusive(struct device_node *node)
{
- return of_reset_control_array_get(node, false, true);
+ return of_reset_control_array_get(node, false, true, true);
}
static inline struct reset_control *
of_reset_control_array_get_optional_shared(struct device_node *node)
{
- return of_reset_control_array_get(node, true, true);
+ return of_reset_control_array_get(node, true, true, true);
}
#endif
int (*read_alarm)(struct device *, struct rtc_wkalrm *);
int (*set_alarm)(struct device *, struct rtc_wkalrm *);
int (*proc)(struct device *, struct seq_file *);
- int (*set_mmss64)(struct device *, time64_t secs);
- int (*set_mmss)(struct device *, unsigned long secs);
int (*alarm_irq_enable)(struct device *, unsigned int enabled);
int (*read_offset)(struct device *, long *offset);
int (*set_offset)(struct device *, long offset);
#define to_rtc_device(d) container_of(d, struct rtc_device, dev)
/* useful timestamps */
-#define RTC_TIMESTAMP_BEGIN_1900 -2208989361LL /* 1900-01-01 00:00:00 */
+#define RTC_TIMESTAMP_BEGIN_1900 -2208988800LL /* 1900-01-01 00:00:00 */
#define RTC_TIMESTAMP_BEGIN_2000 946684800LL /* 2000-01-01 00:00:00 */
+#define RTC_TIMESTAMP_END_2063 2966371199LL /* 2063-12-31 23:59:59 */
#define RTC_TIMESTAMP_END_2099 4102444799LL /* 2099-12-31 23:59:59 */
+#define RTC_TIMESTAMP_END_9999 253402300799LL /* 9999-12-31 23:59:59 */
extern struct rtc_device *devm_rtc_device_register(struct device *dev,
const char *name,
u32 regstep;
resource_size_t baseaddr;
size_t size;
- spinlock_t lock;
- struct work_struct work;
int irq_num;
bool bcd_mode;
bool no_irq;
* sg page iterator
*
* Iterates over sg entries page-by-page. On each successful iteration, you
- * can call sg_page_iter_page(@piter) to get the current page and its dma
- * address. @piter->sg will point to the sg holding this page and
- * @piter->sg_pgoffset to the page's page offset within the sg. The iteration
- * will stop either when a maximum number of sg entries was reached or a
- * terminating sg (sg_last(sg) == true) was reached.
+ * can call sg_page_iter_page(@piter) to get the current page.
+ * @piter->sg will point to the sg holding this page and @piter->sg_pgoffset to
+ * the page's page offset within the sg. The iteration will stop either when a
+ * maximum number of sg entries was reached or a terminating sg
+ * (sg_last(sg) == true) was reached.
*/
struct sg_page_iter {
struct scatterlist *sg; /* sg holding the page */
#ifdef CONFIG_CGROUPS
/* disallow userland-initiated cgroup migration */
unsigned no_cgroup_migration:1;
+ /* task is frozen/stopped (used by the cgroup freezer) */
+ unsigned frozen:1;
#endif
#ifdef CONFIG_BLK_CGROUP
/* to be used once the psi infrastructure lands upstream. */
#define JOBCTL_TRAP_NOTIFY_BIT 20 /* trap for NOTIFY */
#define JOBCTL_TRAPPING_BIT 21 /* switching to TRACED */
#define JOBCTL_LISTENING_BIT 22 /* ptracer is listening for events */
+#define JOBCTL_TRAP_FREEZE_BIT 23 /* trap for cgroup freezer */
#define JOBCTL_STOP_DEQUEUED (1UL << JOBCTL_STOP_DEQUEUED_BIT)
#define JOBCTL_STOP_PENDING (1UL << JOBCTL_STOP_PENDING_BIT)
#define JOBCTL_TRAP_NOTIFY (1UL << JOBCTL_TRAP_NOTIFY_BIT)
#define JOBCTL_TRAPPING (1UL << JOBCTL_TRAPPING_BIT)
#define JOBCTL_LISTENING (1UL << JOBCTL_LISTENING_BIT)
+#define JOBCTL_TRAP_FREEZE (1UL << JOBCTL_TRAP_FREEZE_BIT)
#define JOBCTL_TRAP_MASK (JOBCTL_TRAP_STOP | JOBCTL_TRAP_NOTIFY)
#define JOBCTL_PENDING_MASK (JOBCTL_STOP_PENDING | JOBCTL_TRAP_MASK)
struct rpc_iostats * cl_metrics; /* per-client statistics */
unsigned int cl_softrtry : 1,/* soft timeouts */
+ cl_softerr : 1,/* Timeouts return errors */
cl_discrtry : 1,/* disconnect before retry */
cl_noretranstimeo: 1,/* No retransmit timeouts */
cl_autobind : 1,/* use getport() */
struct dentry *cl_debugfs; /* debugfs directory */
#endif
struct rpc_xprt_iter cl_xpi;
+ const struct cred *cl_cred;
};
/*
unsigned long flags;
char *client_name;
struct svc_xprt *bc_xprt; /* NFSv4.1 backchannel */
+ const struct cred *cred;
};
struct rpc_add_xprt_test {
#define RPC_CLNT_CREATE_INFINITE_SLOTS (1UL << 7)
#define RPC_CLNT_CREATE_NO_IDLE_TIMEOUT (1UL << 8)
#define RPC_CLNT_CREATE_NO_RETRANS_TIMEOUT (1UL << 9)
+#define RPC_CLNT_CREATE_SOFTERR (1UL << 10)
struct rpc_clnt *rpc_create(struct rpc_create_args *args);
struct rpc_clnt *rpc_bind_new_program(struct rpc_clnt *,
struct list_head list; /* wait queue links */
struct list_head links; /* Links to related tasks */
struct list_head timer_list; /* Timer list */
- unsigned long expires;
};
/*
struct rpc_wait tk_wait; /* RPC wait */
} u;
+ int tk_rpc_status; /* Result of last RPC operation */
+
/*
* RPC call state
*/
#define RPC_CALL_MAJORSEEN 0x0020 /* major timeout seen */
#define RPC_TASK_ROOTCREDS 0x0040 /* force root creds */
#define RPC_TASK_DYNAMIC 0x0080 /* task was kmalloc'ed */
-#define RPC_TASK_KILLED 0x0100 /* task was killed */
#define RPC_TASK_SOFT 0x0200 /* Use soft timeouts */
#define RPC_TASK_SOFTCONN 0x0400 /* Fail if can't connect */
#define RPC_TASK_SENT 0x0800 /* message was sent */
#define RPC_IS_ASYNC(t) ((t)->tk_flags & RPC_TASK_ASYNC)
#define RPC_IS_SWAPPER(t) ((t)->tk_flags & RPC_TASK_SWAPPER)
-#define RPC_ASSASSINATED(t) ((t)->tk_flags & RPC_TASK_KILLED)
#define RPC_IS_SOFT(t) ((t)->tk_flags & (RPC_TASK_SOFT|RPC_TASK_TIMEOUT))
#define RPC_IS_SOFTCONN(t) ((t)->tk_flags & RPC_TASK_SOFTCONN)
#define RPC_WAS_SENT(t) ((t)->tk_flags & RPC_TASK_SENT)
#define RPC_TASK_NEED_XMIT 3
#define RPC_TASK_NEED_RECV 4
#define RPC_TASK_MSG_PIN_WAIT 5
+#define RPC_TASK_SIGNALLED 6
#define RPC_IS_RUNNING(t) test_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)
#define rpc_set_running(t) set_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)
#define RPC_IS_ACTIVATED(t) test_bit(RPC_TASK_ACTIVE, &(t)->tk_runstate)
+#define RPC_SIGNALLED(t) test_bit(RPC_TASK_SIGNALLED, &(t)->tk_runstate)
+
/*
* Task priorities.
* Note: if you change these, you must also change
struct rpc_timer {
struct timer_list timer;
struct list_head list;
- unsigned long expires;
};
/*
struct rpc_task *rpc_run_bc_task(struct rpc_rqst *req);
void rpc_put_task(struct rpc_task *);
void rpc_put_task_async(struct rpc_task *);
+void rpc_signal_task(struct rpc_task *);
void rpc_exit_task(struct rpc_task *);
void rpc_exit(struct rpc_task *, int);
void rpc_release_calldata(const struct rpc_call_ops *, void *);
void rpc_init_priority_wait_queue(struct rpc_wait_queue *, const char *);
void rpc_init_wait_queue(struct rpc_wait_queue *, const char *);
void rpc_destroy_wait_queue(struct rpc_wait_queue *);
+unsigned long rpc_task_timeout(const struct rpc_task *task);
+void rpc_sleep_on_timeout(struct rpc_wait_queue *queue,
+ struct rpc_task *task,
+ rpc_action action,
+ unsigned long timeout);
void rpc_sleep_on(struct rpc_wait_queue *, struct rpc_task *,
rpc_action action);
+void rpc_sleep_on_priority_timeout(struct rpc_wait_queue *queue,
+ struct rpc_task *task,
+ unsigned long timeout,
+ int priority);
void rpc_sleep_on_priority(struct rpc_wait_queue *,
struct rpc_task *,
- rpc_action action,
int priority);
void rpc_wake_up_queued_task_on_wq(struct workqueue_struct *wq,
struct rpc_wait_queue *queue,
void (*buf_free)(struct rpc_task *task);
void (*prepare_request)(struct rpc_rqst *req);
int (*send_request)(struct rpc_rqst *req);
- void (*set_retrans_timeout)(struct rpc_task *task);
+ void (*wait_for_reply_request)(struct rpc_task *task);
void (*timer)(struct rpc_xprt *xprt, struct rpc_task *task);
void (*release_request)(struct rpc_task *task);
void (*close)(struct rpc_xprt *xprt);
int xprt_register_transport(struct xprt_class *type);
int xprt_unregister_transport(struct xprt_class *type);
int xprt_load_transport(const char *);
-void xprt_set_retrans_timeout_def(struct rpc_task *task);
-void xprt_set_retrans_timeout_rtt(struct rpc_task *task);
+void xprt_wait_for_reply_request_def(struct rpc_task *task);
+void xprt_wait_for_reply_request_rtt(struct rpc_task *task);
void xprt_wake_pending_tasks(struct rpc_xprt *xprt, int status);
void xprt_wait_for_buffer_space(struct rpc_xprt *xprt);
bool xprt_write_space(struct rpc_xprt *xprt);
int vsp1_du_init(struct device *dev);
+#define VSP1_DU_STATUS_COMPLETE BIT(0)
+#define VSP1_DU_STATUS_WRITEBACK BIT(1)
+
/**
* struct vsp1_du_lif_config - VSP LIF configuration
* @width: output frame width
unsigned int height;
bool interlaced;
- void (*callback)(void *data, bool completed, u32 crc);
+ void (*callback)(void *data, unsigned int status, u32 crc);
void *callback_data;
};
unsigned int index;
};
+/**
+ * struct vsp1_du_writeback_config - VSP writeback configuration parameters
+ * @pixelformat: plane pixel format (V4L2 4CC)
+ * @pitch: line pitch in bytes for the first plane
+ * @mem: DMA memory address for each plane of the frame buffer
+ */
+struct vsp1_du_writeback_config {
+ u32 pixelformat;
+ unsigned int pitch;
+ dma_addr_t mem[3];
+};
+
/**
* struct vsp1_du_atomic_pipe_config - VSP atomic pipe configuration parameters
* @crc: CRC computation configuration
+ * @writeback: writeback configuration
*/
struct vsp1_du_atomic_pipe_config {
struct vsp1_du_crc_config crc;
+ struct vsp1_du_writeback_config writeback;
};
void vsp1_du_atomic_begin(struct device *dev, unsigned int pipe_index);
void *),
void *context);
+int rdma_read_gid_l2_fields(const struct ib_gid_attr *attr,
+ u16 *vlan_id, u8 *smac);
+struct net_device *rdma_read_gid_attr_ndev_rcu(const struct ib_gid_attr *attr);
+
/**
* ib_get_cached_pkey - Returns a cached PKey table entry
* @device: The device to query.
__be16 attr_offset;
__be16 reserved;
ib_sa_comp_mask comp_mask;
-} __attribute__ ((packed));
+} __packed;
struct ib_mad {
struct ib_mad_hdr mad_hdr;
struct ib_rmpp_hdr rmpp_hdr;
struct ib_sa_hdr sa_hdr;
u8 data[IB_MGMT_SA_DATA];
-} __attribute__ ((packed));
+} __packed;
struct ib_vendor_mad {
struct ib_mad_hdr mad_hdr;
u8 data[IB_SMP_DATA_SIZE];
u8 initial_path[IB_SMP_MAX_PATH_HOPS];
u8 return_path[IB_SMP_MAX_PATH_HOPS];
-} __attribute__ ((packed));
+} __packed;
#define IB_SMP_DIRECTION cpu_to_be16(0x8000)
unsigned long address;
int page_shift;
u32 writable : 1;
- u32 hugetlb : 1;
u32 is_odp : 1;
struct work_struct work;
struct sg_table sg_head;
int nmap;
- int npages;
+ unsigned int sg_nents;
};
/* Returns the offset of the umem start relative to the first page. */
int ib_umem_page_count(struct ib_umem *umem);
int ib_umem_copy_from(void *dst, struct ib_umem *umem, size_t offset,
size_t length);
+unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,
+ unsigned long pgsz_bitmap,
+ unsigned long virt);
#else /* CONFIG_INFINIBAND_USER_MEM */
size_t length) {
return -EINVAL;
}
+static inline int ib_umem_find_best_pgsz(struct ib_umem *umem,
+ unsigned long pgsz_bitmap,
+ unsigned long virt) {
+ return -EINVAL;
+}
+
#endif /* CONFIG_INFINIBAND_USER_MEM */
#endif /* IB_UMEM_H */
int notifiers_seq;
int notifiers_count;
+ int npages;
/* Tree tracking */
struct umem_odp_node interval_tree;
#include <linux/mmu_notifier.h>
#include <linux/uaccess.h>
#include <linux/cgroup_rdma.h>
+#include <linux/irqflags.h>
+#include <linux/preempt.h>
#include <uapi/rdma/ib_user_verbs.h>
#include <rdma/restrack.h>
#include <uapi/rdma/rdma_user_ioctl.h>
extern struct workqueue_struct *ib_comp_wq;
extern struct workqueue_struct *ib_comp_unbound_wq;
+__printf(3, 4) __cold
+void ibdev_printk(const char *level, const struct ib_device *ibdev,
+ const char *format, ...);
+__printf(2, 3) __cold
+void ibdev_emerg(const struct ib_device *ibdev, const char *format, ...);
+__printf(2, 3) __cold
+void ibdev_alert(const struct ib_device *ibdev, const char *format, ...);
+__printf(2, 3) __cold
+void ibdev_crit(const struct ib_device *ibdev, const char *format, ...);
+__printf(2, 3) __cold
+void ibdev_err(const struct ib_device *ibdev, const char *format, ...);
+__printf(2, 3) __cold
+void ibdev_warn(const struct ib_device *ibdev, const char *format, ...);
+__printf(2, 3) __cold
+void ibdev_notice(const struct ib_device *ibdev, const char *format, ...);
+__printf(2, 3) __cold
+void ibdev_info(const struct ib_device *ibdev, const char *format, ...);
+
+#if defined(CONFIG_DYNAMIC_DEBUG)
+#define ibdev_dbg(__dev, format, args...) \
+ dynamic_ibdev_dbg(__dev, format, ##args)
+#elif defined(DEBUG)
+#define ibdev_dbg(__dev, format, args...) \
+ ibdev_printk(KERN_DEBUG, __dev, format, ##args)
+#else
+__printf(2, 3) __cold
+static inline
+void ibdev_dbg(const struct ib_device *ibdev, const char *format, ...) {}
+#endif
+
union ib_gid {
u8 raw[16];
struct {
#define ROCE_V2_UDP_DPORT 4791
struct ib_gid_attr {
- struct net_device *ndev;
+ struct net_device __rcu *ndev;
struct ib_device *device;
union ib_gid gid;
enum ib_gid_type gid_type;
RDMA_NODE_RNIC,
RDMA_NODE_USNIC,
RDMA_NODE_USNIC_UDP,
+ RDMA_NODE_UNSPECIFIED,
};
enum {
RDMA_TRANSPORT_IB,
RDMA_TRANSPORT_IWARP,
RDMA_TRANSPORT_USNIC,
- RDMA_TRANSPORT_USNIC_UDP
+ RDMA_TRANSPORT_USNIC_UDP,
+ RDMA_TRANSPORT_UNSPECIFIED,
};
enum rdma_protocol_type {
struct ib_event_handler event_handler;
};
-struct iw_cm_verbs;
-
struct ib_port_immutable {
int pkey_tbl_len;
int gid_tbl_len;
};
struct uverbs_attr_bundle;
+struct iw_cm_id;
+struct iw_cm_conn_param;
#define INIT_RDMA_OBJ_SIZE(ib_struct, drv_struct, member) \
.size_##ib_struct = \
!__same_type(((struct drv_struct *)NULL)->member, \
struct ib_struct)))
+#define rdma_zalloc_drv_obj_gfp(ib_dev, ib_type, gfp) \
+ ((struct ib_type *)kzalloc(ib_dev->ops.size_##ib_type, gfp))
+
#define rdma_zalloc_drv_obj(ib_dev, ib_type) \
- ((struct ib_type *)kzalloc(ib_dev->ops.size_##ib_type, GFP_KERNEL))
+ rdma_zalloc_drv_obj_gfp(ib_dev, ib_type, GFP_KERNEL)
#define DECLARE_RDMA_OBJ_SIZE(ib_struct) size_t size_##ib_struct
void (*dealloc_ucontext)(struct ib_ucontext *context);
int (*mmap)(struct ib_ucontext *context, struct vm_area_struct *vma);
void (*disassociate_ucontext)(struct ib_ucontext *ibcontext);
- int (*alloc_pd)(struct ib_pd *pd, struct ib_ucontext *context,
- struct ib_udata *udata);
- void (*dealloc_pd)(struct ib_pd *pd);
- struct ib_ah *(*create_ah)(struct ib_pd *pd,
- struct rdma_ah_attr *ah_attr, u32 flags,
- struct ib_udata *udata);
+ int (*alloc_pd)(struct ib_pd *pd, struct ib_udata *udata);
+ void (*dealloc_pd)(struct ib_pd *pd, struct ib_udata *udata);
+ int (*create_ah)(struct ib_ah *ah, struct rdma_ah_attr *ah_attr,
+ u32 flags, struct ib_udata *udata);
int (*modify_ah)(struct ib_ah *ah, struct rdma_ah_attr *ah_attr);
int (*query_ah)(struct ib_ah *ah, struct rdma_ah_attr *ah_attr);
- int (*destroy_ah)(struct ib_ah *ah, u32 flags);
- struct ib_srq *(*create_srq)(struct ib_pd *pd,
- struct ib_srq_init_attr *srq_init_attr,
- struct ib_udata *udata);
+ void (*destroy_ah)(struct ib_ah *ah, u32 flags);
+ int (*create_srq)(struct ib_srq *srq,
+ struct ib_srq_init_attr *srq_init_attr,
+ struct ib_udata *udata);
int (*modify_srq)(struct ib_srq *srq, struct ib_srq_attr *srq_attr,
enum ib_srq_attr_mask srq_attr_mask,
struct ib_udata *udata);
int (*query_srq)(struct ib_srq *srq, struct ib_srq_attr *srq_attr);
- int (*destroy_srq)(struct ib_srq *srq);
+ void (*destroy_srq)(struct ib_srq *srq, struct ib_udata *udata);
struct ib_qp *(*create_qp)(struct ib_pd *pd,
struct ib_qp_init_attr *qp_init_attr,
struct ib_udata *udata);
int qp_attr_mask, struct ib_udata *udata);
int (*query_qp)(struct ib_qp *qp, struct ib_qp_attr *qp_attr,
int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr);
- int (*destroy_qp)(struct ib_qp *qp);
+ int (*destroy_qp)(struct ib_qp *qp, struct ib_udata *udata);
struct ib_cq *(*create_cq)(struct ib_device *device,
const struct ib_cq_init_attr *attr,
- struct ib_ucontext *context,
struct ib_udata *udata);
int (*modify_cq)(struct ib_cq *cq, u16 cq_count, u16 cq_period);
- int (*destroy_cq)(struct ib_cq *cq);
+ int (*destroy_cq)(struct ib_cq *cq, struct ib_udata *udata);
int (*resize_cq)(struct ib_cq *cq, int cqe, struct ib_udata *udata);
struct ib_mr *(*get_dma_mr)(struct ib_pd *pd, int mr_access_flags);
struct ib_mr *(*reg_user_mr)(struct ib_pd *pd, u64 start, u64 length,
int (*rereg_user_mr)(struct ib_mr *mr, int flags, u64 start, u64 length,
u64 virt_addr, int mr_access_flags,
struct ib_pd *pd, struct ib_udata *udata);
- int (*dereg_mr)(struct ib_mr *mr);
+ int (*dereg_mr)(struct ib_mr *mr, struct ib_udata *udata);
struct ib_mr *(*alloc_mr)(struct ib_pd *pd, enum ib_mr_type mr_type,
- u32 max_num_sg);
+ u32 max_num_sg, struct ib_udata *udata);
int (*advise_mr)(struct ib_pd *pd,
enum ib_uverbs_advise_mr_advice advice, u32 flags,
struct ib_sge *sg_list, u32 num_sge,
int (*attach_mcast)(struct ib_qp *qp, union ib_gid *gid, u16 lid);
int (*detach_mcast)(struct ib_qp *qp, union ib_gid *gid, u16 lid);
struct ib_xrcd *(*alloc_xrcd)(struct ib_device *device,
- struct ib_ucontext *ucontext,
struct ib_udata *udata);
- int (*dealloc_xrcd)(struct ib_xrcd *xrcd);
+ int (*dealloc_xrcd)(struct ib_xrcd *xrcd, struct ib_udata *udata);
struct ib_flow *(*create_flow)(struct ib_qp *qp,
struct ib_flow_attr *flow_attr,
int domain, struct ib_udata *udata);
struct ib_wq *(*create_wq)(struct ib_pd *pd,
struct ib_wq_init_attr *init_attr,
struct ib_udata *udata);
- int (*destroy_wq)(struct ib_wq *wq);
+ int (*destroy_wq)(struct ib_wq *wq, struct ib_udata *udata);
int (*modify_wq)(struct ib_wq *wq, struct ib_wq_attr *attr,
u32 wq_attr_mask, struct ib_udata *udata);
struct ib_rwq_ind_table *(*create_rwq_ind_table)(
struct ib_ucontext *context,
struct ib_dm_alloc_attr *attr,
struct uverbs_attr_bundle *attrs);
- int (*dealloc_dm)(struct ib_dm *dm);
+ int (*dealloc_dm)(struct ib_dm *dm, struct uverbs_attr_bundle *attrs);
struct ib_mr *(*reg_dm_mr)(struct ib_pd *pd, struct ib_dm *dm,
struct ib_dm_mr_attr *attr,
struct uverbs_attr_bundle *attrs);
*/
void (*dealloc_driver)(struct ib_device *dev);
+ /* iWarp CM callbacks */
+ void (*iw_add_ref)(struct ib_qp *qp);
+ void (*iw_rem_ref)(struct ib_qp *qp);
+ struct ib_qp *(*iw_get_qp)(struct ib_device *device, int qpn);
+ int (*iw_connect)(struct iw_cm_id *cm_id,
+ struct iw_cm_conn_param *conn_param);
+ int (*iw_accept)(struct iw_cm_id *cm_id,
+ struct iw_cm_conn_param *conn_param);
+ int (*iw_reject)(struct iw_cm_id *cm_id, const void *pdata,
+ u8 pdata_len);
+ int (*iw_create_listen)(struct iw_cm_id *cm_id, int backlog);
+ int (*iw_destroy_listen)(struct iw_cm_id *cm_id);
+
+ DECLARE_RDMA_OBJ_SIZE(ib_ah);
DECLARE_RDMA_OBJ_SIZE(ib_pd);
+ DECLARE_RDMA_OBJ_SIZE(ib_srq);
DECLARE_RDMA_OBJ_SIZE(ib_ucontext);
};
-struct rdma_restrack_root;
+struct ib_core_device {
+ /* device must be the first element in structure until,
+ * union of ib_core_device and device exists in ib_device.
+ */
+ struct device dev;
+ possible_net_t rdma_net;
+ struct kobject *ports_kobj;
+ struct list_head port_list;
+ struct ib_device *owner; /* reach back to owner ib_device */
+};
+struct rdma_restrack_root;
struct ib_device {
/* Do not access @dma_device directly from ULP nor from HW drivers. */
struct device *dma_device;
int num_comp_vectors;
- struct iw_cm_verbs *iwcm;
-
struct module *owner;
- struct device dev;
+ union {
+ struct device dev;
+ struct ib_core_device coredev;
+ };
+
/* First group for device attributes,
* Second group for driver provided attributes (optional).
* It is NULL terminated array.
*/
const struct attribute_group *groups[3];
- struct kobject *ports_kobj;
- struct list_head port_list;
-
int uverbs_abi_ver;
u64 uverbs_cmd_mask;
u64 uverbs_ex_cmd_mask;
struct work_struct unregistration_work;
const struct rdma_link_ops *link_ops;
+
+ /* Protects compat_devs xarray modifications */
+ struct mutex compat_devs_mutex;
+ /* Maintains compat devices for each net namespace */
+ struct xarray compat_devs;
+
+ /* Used by iWarp CM */
+ char iw_ifname[IFNAMSIZ];
+ u32 iw_driver_flags;
};
struct ib_client {
u8 no_kverbs_req:1;
};
+/*
+ * IB block DMA iterator
+ *
+ * Iterates the DMA-mapped SGL in contiguous memory blocks aligned
+ * to a HW supported page size.
+ */
+struct ib_block_iter {
+ /* internal states */
+ struct scatterlist *__sg; /* sg holding the current aligned block */
+ dma_addr_t __dma_addr; /* unaligned DMA address of this block */
+ unsigned int __sg_nents; /* number of SG entries */
+ unsigned int __sg_advance; /* number of bytes to advance in sg in next step */
+ unsigned int __pg_bit; /* alignment of current block */
+};
+
struct ib_device *_ib_alloc_device(size_t size);
#define ib_alloc_device(drv_struct, member) \
container_of(_ib_alloc_device(sizeof(struct drv_struct) + \
int ib_register_client (struct ib_client *client);
void ib_unregister_client(struct ib_client *client);
+void __rdma_block_iter_start(struct ib_block_iter *biter,
+ struct scatterlist *sglist,
+ unsigned int nents,
+ unsigned long pgsz);
+bool __rdma_block_iter_next(struct ib_block_iter *biter);
+
+/**
+ * rdma_block_iter_dma_address - get the aligned dma address of the current
+ * block held by the block iterator.
+ * @biter: block iterator holding the memory block
+ */
+static inline dma_addr_t
+rdma_block_iter_dma_address(struct ib_block_iter *biter)
+{
+ return biter->__dma_addr & ~(BIT_ULL(biter->__pg_bit) - 1);
+}
+
+/**
+ * rdma_for_each_block - iterate over contiguous memory blocks of the sg list
+ * @sglist: sglist to iterate over
+ * @biter: block iterator holding the memory block
+ * @nents: maximum number of sg entries to iterate over
+ * @pgsz: best HW supported page size to use
+ *
+ * Callers may use rdma_block_iter_dma_address() to get each
+ * blocks aligned DMA address.
+ */
+#define rdma_for_each_block(sglist, biter, nents, pgsz) \
+ for (__rdma_block_iter_start(biter, sglist, nents, \
+ pgsz); \
+ __rdma_block_iter_next(biter);)
+
/**
* ib_get_client_data - Get IB client context
* @device:Device to get context for
#if IS_ENABLED(CONFIG_INFINIBAND_USER_ACCESS)
int rdma_user_mmap_io(struct ib_ucontext *ucontext, struct vm_area_struct *vma,
unsigned long pfn, unsigned long size, pgprot_t prot);
-int rdma_user_mmap_page(struct ib_ucontext *ucontext,
- struct vm_area_struct *vma, struct page *page,
- unsigned long size);
#else
static inline int rdma_user_mmap_io(struct ib_ucontext *ucontext,
struct vm_area_struct *vma,
{
return -EINVAL;
}
-static inline int rdma_user_mmap_page(struct ib_ucontext *ucontext,
- struct vm_area_struct *vma, struct page *page,
- unsigned long size)
-{
- return -EINVAL;
-}
#endif
static inline int ib_copy_from_udata(void *dest, struct ib_udata *udata, size_t len)
*/
static inline bool rdma_cap_opa_mad(struct ib_device *device, u8 port_num)
{
- return (device->port_data[port_num].immutable.core_cap_flags &
- RDMA_CORE_CAP_OPA_MAD) == RDMA_CORE_CAP_OPA_MAD;
+ return device->port_data[port_num].immutable.core_cap_flags &
+ RDMA_CORE_CAP_OPA_MAD;
}
/**
return rdma_protocol_iwarp(dev, port_num);
}
+/**
+ * rdma_find_pg_bit - Find page bit given address and HW supported page sizes
+ *
+ * @addr: address
+ * @pgsz_bitmap: bitmap of HW supported page sizes
+ */
+static inline unsigned int rdma_find_pg_bit(unsigned long addr,
+ unsigned long pgsz_bitmap)
+{
+ unsigned long align;
+ unsigned long pgsz;
+
+ align = addr & -addr;
+
+ /* Find page bit such that addr is aligned to the highest supported
+ * HW page size
+ */
+ pgsz = pgsz_bitmap & ~(-align << 1);
+ if (!pgsz)
+ return __ffs(pgsz_bitmap);
+
+ return __fls(pgsz);
+}
+
int ib_set_vf_link_state(struct ib_device *device, int vf, u8 port,
int state);
int ib_get_vf_config(struct ib_device *device, int vf, u8 port,
struct ib_pd *__ib_alloc_pd(struct ib_device *device, unsigned int flags,
const char *caller);
+
#define ib_alloc_pd(device, flags) \
__ib_alloc_pd((device), (flags), KBUILD_MODNAME)
-void ib_dealloc_pd(struct ib_pd *pd);
+
+/**
+ * ib_dealloc_pd_user - Deallocate kernel/user PD
+ * @pd: The protection domain
+ * @udata: Valid user data or NULL for kernel objects
+ */
+void ib_dealloc_pd_user(struct ib_pd *pd, struct ib_udata *udata);
+
+/**
+ * ib_dealloc_pd - Deallocate kernel PD
+ * @pd: The protection domain
+ *
+ * NOTE: for user PD use ib_dealloc_pd_user with valid udata!
+ */
+static inline void ib_dealloc_pd(struct ib_pd *pd)
+{
+ ib_dealloc_pd_user(pd, NULL);
+}
enum rdma_create_ah_flags {
/* In a sleepable context */
};
/**
- * rdma_destroy_ah - Destroys an address handle.
+ * rdma_destroy_ah_user - Destroys an address handle.
* @ah: The address handle to destroy.
* @flags: Destroy address handle flags (see enum rdma_destroy_ah_flags).
+ * @udata: Valid user data or NULL for kernel objects
*/
-int rdma_destroy_ah(struct ib_ah *ah, u32 flags);
+int rdma_destroy_ah_user(struct ib_ah *ah, u32 flags, struct ib_udata *udata);
+
+/**
+ * rdma_destroy_ah - Destroys an kernel address handle.
+ * @ah: The address handle to destroy.
+ * @flags: Destroy address handle flags (see enum rdma_destroy_ah_flags).
+ *
+ * NOTE: for user ah use rdma_destroy_ah_user with valid udata!
+ */
+static inline int rdma_destroy_ah(struct ib_ah *ah, u32 flags)
+{
+ return rdma_destroy_ah_user(ah, flags, NULL);
+}
/**
* ib_create_srq - Creates a SRQ associated with the specified protection
struct ib_srq_attr *srq_attr);
/**
- * ib_destroy_srq - Destroys the specified SRQ.
+ * ib_destroy_srq_user - Destroys the specified SRQ.
* @srq: The SRQ to destroy.
+ * @udata: Valid user data or NULL for kernel objects
*/
-int ib_destroy_srq(struct ib_srq *srq);
+int ib_destroy_srq_user(struct ib_srq *srq, struct ib_udata *udata);
+
+/**
+ * ib_destroy_srq - Destroys the specified kernel SRQ.
+ * @srq: The SRQ to destroy.
+ *
+ * NOTE: for user srq use ib_destroy_srq_user with valid udata!
+ */
+static inline int ib_destroy_srq(struct ib_srq *srq)
+{
+ return ib_destroy_srq_user(srq, NULL);
+}
/**
* ib_post_srq_recv - Posts a list of work requests to the specified SRQ.
}
/**
- * ib_create_qp - Creates a QP associated with the specified protection
+ * ib_create_qp_user - Creates a QP associated with the specified protection
* domain.
* @pd: The protection domain associated with the QP.
* @qp_init_attr: A list of initial attributes required to create the
* QP. If QP creation succeeds, then the attributes are updated to
* the actual capabilities of the created QP.
+ * @udata: Valid user data or NULL for kernel objects
*/
-struct ib_qp *ib_create_qp(struct ib_pd *pd,
- struct ib_qp_init_attr *qp_init_attr);
+struct ib_qp *ib_create_qp_user(struct ib_pd *pd,
+ struct ib_qp_init_attr *qp_init_attr,
+ struct ib_udata *udata);
+
+/**
+ * ib_create_qp - Creates a kernel QP associated with the specified protection
+ * domain.
+ * @pd: The protection domain associated with the QP.
+ * @qp_init_attr: A list of initial attributes required to create the
+ * QP. If QP creation succeeds, then the attributes are updated to
+ * the actual capabilities of the created QP.
+ * @udata: Valid user data or NULL for kernel objects
+ *
+ * NOTE: for user qp use ib_create_qp_user with valid udata!
+ */
+static inline struct ib_qp *ib_create_qp(struct ib_pd *pd,
+ struct ib_qp_init_attr *qp_init_attr)
+{
+ return ib_create_qp_user(pd, qp_init_attr, NULL);
+}
/**
* ib_modify_qp_with_udata - Modifies the attributes for the specified QP.
/**
* ib_destroy_qp - Destroys the specified QP.
* @qp: The QP to destroy.
+ * @udata: Valid udata or NULL for kernel objects
*/
-int ib_destroy_qp(struct ib_qp *qp);
+int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata);
+
+/**
+ * ib_destroy_qp - Destroys the specified kernel QP.
+ * @qp: The QP to destroy.
+ *
+ * NOTE: for user qp use ib_destroy_qp_user with valid udata!
+ */
+static inline int ib_destroy_qp(struct ib_qp *qp)
+{
+ return ib_destroy_qp_user(qp, NULL);
+}
/**
* ib_open_qp - Obtain a reference to an existing sharable QP.
return qp->device->ops.post_recv(qp, recv_wr, bad_recv_wr ? : &dummy);
}
-struct ib_cq *__ib_alloc_cq(struct ib_device *dev, void *private,
- int nr_cqe, int comp_vector,
- enum ib_poll_context poll_ctx, const char *caller);
-#define ib_alloc_cq(device, priv, nr_cqe, comp_vect, poll_ctx) \
- __ib_alloc_cq((device), (priv), (nr_cqe), (comp_vect), (poll_ctx), KBUILD_MODNAME)
+struct ib_cq *__ib_alloc_cq_user(struct ib_device *dev, void *private,
+ int nr_cqe, int comp_vector,
+ enum ib_poll_context poll_ctx,
+ const char *caller, struct ib_udata *udata);
+
+/**
+ * ib_alloc_cq_user: Allocate kernel/user CQ
+ * @dev: The IB device
+ * @private: Private data attached to the CQE
+ * @nr_cqe: Number of CQEs in the CQ
+ * @comp_vector: Completion vector used for the IRQs
+ * @poll_ctx: Context used for polling the CQ
+ * @udata: Valid user data or NULL for kernel objects
+ */
+static inline struct ib_cq *ib_alloc_cq_user(struct ib_device *dev,
+ void *private, int nr_cqe,
+ int comp_vector,
+ enum ib_poll_context poll_ctx,
+ struct ib_udata *udata)
+{
+ return __ib_alloc_cq_user(dev, private, nr_cqe, comp_vector, poll_ctx,
+ KBUILD_MODNAME, udata);
+}
+
+/**
+ * ib_alloc_cq: Allocate kernel CQ
+ * @dev: The IB device
+ * @private: Private data attached to the CQE
+ * @nr_cqe: Number of CQEs in the CQ
+ * @comp_vector: Completion vector used for the IRQs
+ * @poll_ctx: Context used for polling the CQ
+ *
+ * NOTE: for user cq use ib_alloc_cq_user with valid udata!
+ */
+static inline struct ib_cq *ib_alloc_cq(struct ib_device *dev, void *private,
+ int nr_cqe, int comp_vector,
+ enum ib_poll_context poll_ctx)
+{
+ return ib_alloc_cq_user(dev, private, nr_cqe, comp_vector, poll_ctx,
+ NULL);
+}
+
+/**
+ * ib_free_cq_user - Free kernel/user CQ
+ * @cq: The CQ to free
+ * @udata: Valid user data or NULL for kernel objects
+ */
+void ib_free_cq_user(struct ib_cq *cq, struct ib_udata *udata);
+
+/**
+ * ib_free_cq - Free kernel CQ
+ * @cq: The CQ to free
+ *
+ * NOTE: for user cq use ib_free_cq_user with valid udata!
+ */
+static inline void ib_free_cq(struct ib_cq *cq)
+{
+ ib_free_cq_user(cq, NULL);
+}
-void ib_free_cq(struct ib_cq *cq);
int ib_process_cq_direct(struct ib_cq *cq, int budget);
/**
int rdma_set_cq_moderation(struct ib_cq *cq, u16 cq_count, u16 cq_period);
/**
- * ib_destroy_cq - Destroys the specified CQ.
+ * ib_destroy_cq_user - Destroys the specified CQ.
* @cq: The CQ to destroy.
+ * @udata: Valid user data or NULL for kernel objects
*/
-int ib_destroy_cq(struct ib_cq *cq);
+int ib_destroy_cq_user(struct ib_cq *cq, struct ib_udata *udata);
+
+/**
+ * ib_destroy_cq - Destroys the specified kernel CQ.
+ * @cq: The CQ to destroy.
+ *
+ * NOTE: for user cq use ib_destroy_cq_user with valid udata!
+ */
+static inline int ib_destroy_cq(struct ib_cq *cq)
+{
+ return ib_destroy_cq_user(cq, NULL);
+}
/**
* ib_poll_cq - poll a CQ for completion(s)
}
/**
- * ib_dereg_mr - Deregisters a memory region and removes it from the
+ * ib_dereg_mr_user - Deregisters a memory region and removes it from the
+ * HCA translation table.
+ * @mr: The memory region to deregister.
+ * @udata: Valid user data or NULL for kernel object
+ *
+ * This function can fail, if the memory region has memory windows bound to it.
+ */
+int ib_dereg_mr_user(struct ib_mr *mr, struct ib_udata *udata);
+
+/**
+ * ib_dereg_mr - Deregisters a kernel memory region and removes it from the
* HCA translation table.
* @mr: The memory region to deregister.
*
* This function can fail, if the memory region has memory windows bound to it.
+ *
+ * NOTE: for user mr use ib_dereg_mr_user with valid udata!
*/
-int ib_dereg_mr(struct ib_mr *mr);
+static inline int ib_dereg_mr(struct ib_mr *mr)
+{
+ return ib_dereg_mr_user(mr, NULL);
+}
+
+struct ib_mr *ib_alloc_mr_user(struct ib_pd *pd, enum ib_mr_type mr_type,
+ u32 max_num_sg, struct ib_udata *udata);
-struct ib_mr *ib_alloc_mr(struct ib_pd *pd,
- enum ib_mr_type mr_type,
- u32 max_num_sg);
+static inline struct ib_mr *ib_alloc_mr(struct ib_pd *pd,
+ enum ib_mr_type mr_type, u32 max_num_sg)
+{
+ return ib_alloc_mr_user(pd, mr_type, max_num_sg, NULL);
+}
/**
* ib_update_fast_reg_key - updates the key portion of the fast_reg MR
/**
* ib_dealloc_xrcd - Deallocates an XRC domain.
* @xrcd: The XRC domain to deallocate.
+ * @udata: Valid user data or NULL for kernel object
*/
-int ib_dealloc_xrcd(struct ib_xrcd *xrcd);
+int ib_dealloc_xrcd(struct ib_xrcd *xrcd, struct ib_udata *udata);
static inline int ib_check_mr_access(int flags)
{
struct ib_wq *ib_create_wq(struct ib_pd *pd,
struct ib_wq_init_attr *init_attr);
-int ib_destroy_wq(struct ib_wq *wq);
+int ib_destroy_wq(struct ib_wq *wq, struct ib_udata *udata);
int ib_modify_wq(struct ib_wq *wq, struct ib_wq_attr *attr,
u32 wq_attr_mask);
struct ib_rwq_ind_table *ib_create_rwq_ind_table(struct ib_device *device,
*/
static inline struct ib_device *rdma_device_to_ibdev(struct device *device)
{
- return container_of(device, struct ib_device, dev);
+ struct ib_core_device *coredev =
+ container_of(device, struct ib_core_device, dev);
+
+ return coredev->owner;
}
/**
*/
#define rdma_device_to_drv_device(dev, drv_dev_struct, ibdev_member) \
container_of(rdma_device_to_ibdev(dev), drv_dev_struct, ibdev_member)
+
+bool rdma_dev_access_netns(const struct ib_device *device,
+ const struct net *net);
#endif /* IB_VERBS_H */
IW_F_NO_PORT_MAP = (1 << 0),
};
-struct iw_cm_verbs {
- void (*add_ref)(struct ib_qp *qp);
-
- void (*rem_ref)(struct ib_qp *qp);
-
- struct ib_qp * (*get_qp)(struct ib_device *device,
- int qpn);
-
- int (*connect)(struct iw_cm_id *cm_id,
- struct iw_cm_conn_param *conn_param);
-
- int (*accept)(struct iw_cm_id *cm_id,
- struct iw_cm_conn_param *conn_param);
-
- int (*reject)(struct iw_cm_id *cm_id,
- const void *pdata, u8 pdata_len);
-
- int (*create_listen)(struct iw_cm_id *cm_id,
- int backlog);
-
- int (*destroy_listen)(struct iw_cm_id *cm_id);
- char ifname[IFNAMSIZ];
- enum iw_flags driver_flags;
-};
-
/**
* iw_create_cm_id - Create an IW CM identifier.
*
u8 local_port_num;
u8 reserved12;
u8 reserved13; /* was guid_cap */
-} __attribute__ ((packed));
+} __packed;
#endif /* OPA_PORT_INFO_H */
struct opa_node_description {
u8 data[64];
-} __attribute__ ((packed));
+} __packed;
struct opa_node_info {
u8 base_version;
__be32 revision;
u8 local_port_num;
u8 vendor_id[3]; /* network byte order */
-} __attribute__ ((packed));
+} __packed;
#define OPA_PARTITION_TABLE_BLK_SIZE 32
#include <rdma/ib_verbs.h>
#include <rdma/ib_mad.h>
#include <rdma/rdmavt_mr.h>
-#include <rdma/rdmavt_qp.h>
#define RVT_MAX_PKEY_VALUES 16
struct list_head list;
};
+struct rvt_qp;
+struct rvt_qpn_table;
struct rvt_ibport {
struct rvt_qp __rcu *qp[2];
struct ib_mad_agent *send_agent; /* agent for SMI (traps) */
u8 log_pmtu;
};
+/*
+ * This structure is used by rvt_mmap() to validate an offset
+ * when an mmap() request is made. The vm_area_struct then uses
+ * this as its vm_private_data.
+ */
+struct rvt_mmap_info {
+ struct list_head pending_mmaps;
+ struct ib_ucontext *context;
+ void *obj;
+ __u64 offset;
+ struct kref ref;
+ u32 size;
+};
+
/* memory working set size */
struct rvt_wss {
unsigned long *entries;
return container_of(ibdev, struct rvt_dev_info, ibdev);
}
-static inline struct rvt_srq *ibsrq_to_rvtsrq(struct ib_srq *ibsrq)
-{
- return container_of(ibsrq, struct rvt_srq, ibsrq);
-}
-
-static inline struct rvt_qp *ibqp_to_rvtqp(struct ib_qp *ibqp)
-{
- return container_of(ibqp, struct rvt_qp, ibqp);
-}
-
static inline unsigned rvt_get_npkeys(struct rvt_dev_info *rdi)
{
/*
return rdi->ports[port_index]->pkey_table[index];
}
-/**
- * rvt_lookup_qpn - return the QP with the given QPN
- * @ibp: the ibport
- * @qpn: the QP number to look up
- *
- * The caller must hold the rcu_read_lock(), and keep the lock until
- * the returned qp is no longer in use.
- */
-/* TODO: Remove this and put in rdmavt/qp.h when no longer needed by drivers */
-static inline struct rvt_qp *rvt_lookup_qpn(struct rvt_dev_info *rdi,
- struct rvt_ibport *rvp,
- u32 qpn) __must_hold(RCU)
-{
- struct rvt_qp *qp = NULL;
-
- if (unlikely(qpn <= 1)) {
- qp = rcu_dereference(rvp->qp[qpn]);
- } else {
- u32 n = hash_32(qpn, rdi->qp_dev->qp_table_bits);
-
- for (qp = rcu_dereference(rdi->qp_dev->qp_table[n]); qp;
- qp = rcu_dereference(qp->next))
- if (qp->ibqp.qp_num == qpn)
- break;
- }
- return qp;
-}
-
-/**
- * rvt_mod_retry_timer - mod a retry timer
- * @qp - the QP
- * @shift - timeout shift to wait for multiple packets
- * Modify a potentially already running retry timer
- */
-static inline void rvt_mod_retry_timer_ext(struct rvt_qp *qp, u8 shift)
-{
- struct ib_qp *ibqp = &qp->ibqp;
- struct rvt_dev_info *rdi = ib_to_rvt(ibqp->device);
-
- lockdep_assert_held(&qp->s_lock);
- qp->s_flags |= RVT_S_TIMER;
- /* 4.096 usec. * (1 << qp->timeout) */
- mod_timer(&qp->s_timer, jiffies + rdi->busy_jiffies +
- (qp->timeout_jiffies << shift));
-}
-
-static inline void rvt_mod_retry_timer(struct rvt_qp *qp)
-{
- return rvt_mod_retry_timer_ext(qp, 0);
-}
-
struct rvt_dev_info *rvt_alloc_device(size_t size, int nports);
void rvt_dealloc_device(struct rvt_dev_info *rdi);
int rvt_register_device(struct rvt_dev_info *rvd, u32 driver_id);
* RVT_S_WAIT_DMA - waiting for send DMA queue to drain before generating
* next send completion entry not via send DMA
* RVT_S_WAIT_PIO - waiting for a send buffer to be available
- * RVT_S_WAIT_PIO_DRAIN - waiting for a qp to drain pio packets
* RVT_S_WAIT_TX - waiting for a struct verbs_txreq to be available
* RVT_S_WAIT_DMA_DESC - waiting for DMA descriptors to be available
* RVT_S_WAIT_KMEM - waiting for kernel memory to be available
spinlock_t lock ____cacheline_aligned_in_smp;
};
-/*
- * This structure is used by rvt_mmap() to validate an offset
- * when an mmap() request is made. The vm_area_struct then uses
- * this as its vm_private_data.
- */
-struct rvt_mmap_info {
- struct list_head pending_mmaps;
- struct ib_ucontext *context;
- void *obj;
- __u64 offset;
- struct kref ref;
- unsigned size;
-};
-
/*
* This structure holds the information that the send tasklet needs
* to send a RDMA read response or atomic operation.
u32 limit;
};
+static inline struct rvt_srq *ibsrq_to_rvtsrq(struct ib_srq *ibsrq)
+{
+ return container_of(ibsrq, struct rvt_srq, ibsrq);
+}
+
+static inline struct rvt_qp *ibqp_to_rvtqp(struct ib_qp *ibqp)
+{
+ return container_of(ibqp, struct rvt_qp, ibqp);
+}
+
#define RVT_QPN_MAX BIT(24)
#define RVT_QPNMAP_ENTRIES (RVT_QPN_MAX / PAGE_SIZE / BITS_PER_BYTE)
#define RVT_BITS_PER_PAGE (PAGE_SIZE * BITS_PER_BYTE)
return usecs_to_jiffies(1U << timeout) * 4096UL / 1000UL;
}
+/**
+ * rvt_lookup_qpn - return the QP with the given QPN
+ * @ibp: the ibport
+ * @qpn: the QP number to look up
+ *
+ * The caller must hold the rcu_read_lock(), and keep the lock until
+ * the returned qp is no longer in use.
+ */
+static inline struct rvt_qp *rvt_lookup_qpn(struct rvt_dev_info *rdi,
+ struct rvt_ibport *rvp,
+ u32 qpn) __must_hold(RCU)
+{
+ struct rvt_qp *qp = NULL;
+
+ if (unlikely(qpn <= 1)) {
+ qp = rcu_dereference(rvp->qp[qpn]);
+ } else {
+ u32 n = hash_32(qpn, rdi->qp_dev->qp_table_bits);
+
+ for (qp = rcu_dereference(rdi->qp_dev->qp_table[n]); qp;
+ qp = rcu_dereference(qp->next))
+ if (qp->ibqp.qp_num == qpn)
+ break;
+ }
+ return qp;
+}
+
+/**
+ * rvt_mod_retry_timer - mod a retry timer
+ * @qp - the QP
+ * @shift - timeout shift to wait for multiple packets
+ * Modify a potentially already running retry timer
+ */
+static inline void rvt_mod_retry_timer_ext(struct rvt_qp *qp, u8 shift)
+{
+ struct ib_qp *ibqp = &qp->ibqp;
+ struct rvt_dev_info *rdi = ib_to_rvt(ibqp->device);
+
+ lockdep_assert_held(&qp->s_lock);
+ qp->s_flags |= RVT_S_TIMER;
+ /* 4.096 usec. * (1 << qp->timeout) */
+ mod_timer(&qp->s_timer, jiffies + rdi->busy_jiffies +
+ (qp->timeout_jiffies << shift));
+}
+
+static inline void rvt_mod_retry_timer(struct rvt_qp *qp)
+{
+ return rvt_mod_retry_timer_ext(qp, 0);
+}
+
+/**
+ * rvt_put_qp_swqe - drop refs held by swqe
+ * @qp: the send qp
+ * @wqe: the send wqe
+ *
+ * This drops any references held by the swqe
+ */
+static inline void rvt_put_qp_swqe(struct rvt_qp *qp, struct rvt_swqe *wqe)
+{
+ rvt_put_swqe(wqe);
+ if (qp->allowed_ops == IB_OPCODE_UD)
+ atomic_dec(&ibah_to_rvtah(wqe->ud_wr.ah)->refcount);
+}
+
extern const int ib_rvt_state_ops[];
struct rvt_dev_info;
#define uobj_get_type(_attrs, _object) \
uapi_get_object((_attrs)->ufile->device->uapi, _object)
-struct ib_uobject *_uobj_get_read(enum uverbs_default_objects type,
- u32 object_id,
- struct uverbs_attr_bundle *attrs);
-
#define uobj_get_read(_type, _id, _attrs) \
- _uobj_get_read(_type, _uobj_check_id(_id), _attrs)
+ rdma_lookup_get_uobject(uobj_get_type(_attrs, _type), (_attrs)->ufile, \
+ _uobj_check_id(_id), UVERBS_LOOKUP_READ, \
+ _attrs)
#define ufd_get_read(_type, _fdnum, _attrs) \
rdma_lookup_get_uobject(uobj_get_type(_attrs, _type), (_attrs)->ufile, \
(_fdnum)*typecheck(s32, _fdnum), \
- UVERBS_LOOKUP_READ)
+ UVERBS_LOOKUP_READ, _attrs)
static inline void *_uobj_get_obj_read(struct ib_uobject *uobj)
{
((struct ib_##_object *)_uobj_get_obj_read( \
uobj_get_read(_type, _id, _attrs)))
-struct ib_uobject *_uobj_get_write(enum uverbs_default_objects type,
- u32 object_id,
- struct uverbs_attr_bundle *attrs);
-
#define uobj_get_write(_type, _id, _attrs) \
- _uobj_get_write(_type, _uobj_check_id(_id), _attrs)
+ rdma_lookup_get_uobject(uobj_get_type(_attrs, _type), (_attrs)->ufile, \
+ _uobj_check_id(_id), UVERBS_LOOKUP_WRITE, \
+ _attrs)
int __uobj_perform_destroy(const struct uverbs_api_object *obj, u32 id,
- const struct uverbs_attr_bundle *attrs);
+ struct uverbs_attr_bundle *attrs);
#define uobj_perform_destroy(_type, _id, _attrs) \
__uobj_perform_destroy(uobj_get_type(_attrs, _type), \
_uobj_check_id(_id), _attrs)
struct ib_uobject *__uobj_get_destroy(const struct uverbs_api_object *obj,
- u32 id,
- const struct uverbs_attr_bundle *attrs);
+ u32 id, struct uverbs_attr_bundle *attrs);
#define uobj_get_destroy(_type, _id, _attrs) \
__uobj_get_destroy(uobj_get_type(_attrs, _type), _uobj_check_id(_id), \
rdma_lookup_put_uobject(uobj, UVERBS_LOOKUP_WRITE);
}
-static inline int __must_check uobj_alloc_commit(struct ib_uobject *uobj)
+static inline int __must_check
+uobj_alloc_commit(struct ib_uobject *uobj, struct uverbs_attr_bundle *attrs)
{
- int ret = rdma_alloc_commit_uobject(uobj);
+ int ret = rdma_alloc_commit_uobject(uobj, attrs);
if (ret)
return ret;
return 0;
}
-static inline void uobj_alloc_abort(struct ib_uobject *uobj)
+static inline void uobj_alloc_abort(struct ib_uobject *uobj,
+ struct uverbs_attr_bundle *attrs)
{
- rdma_alloc_abort_uobject(uobj);
+ rdma_alloc_abort_uobject(uobj, attrs);
}
static inline struct ib_uobject *
__uobj_alloc(const struct uverbs_api_object *obj,
struct uverbs_attr_bundle *attrs, struct ib_device **ib_dev)
{
- struct ib_uobject *uobj = rdma_alloc_begin_uobject(obj, attrs->ufile);
+ struct ib_uobject *uobj =
+ rdma_alloc_begin_uobject(obj, attrs->ufile, attrs);
- if (!IS_ERR(uobj)) {
- *ib_dev = uobj->context->device;
- attrs->context = uobj->context;
- }
+ if (!IS_ERR(uobj))
+ *ib_dev = attrs->context->device;
return uobj;
}
void (*lookup_put)(struct ib_uobject *uobj, enum rdma_lookup_mode mode);
/* This does not consume the kref on uobj */
int __must_check (*destroy_hw)(struct ib_uobject *uobj,
- enum rdma_remove_reason why);
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs);
void (*remove_handle)(struct ib_uobject *uobj);
u8 needs_kfree_rcu;
};
* completely unchanged.
*/
int __must_check (*destroy_object)(struct ib_uobject *uobj,
- enum rdma_remove_reason why);
+ enum rdma_remove_reason why,
+ struct uverbs_attr_bundle *attrs);
};
struct ib_uobject *rdma_lookup_get_uobject(const struct uverbs_api_object *obj,
struct ib_uverbs_file *ufile, s64 id,
- enum rdma_lookup_mode mode);
+ enum rdma_lookup_mode mode,
+ struct uverbs_attr_bundle *attrs);
void rdma_lookup_put_uobject(struct ib_uobject *uobj,
enum rdma_lookup_mode mode);
struct ib_uobject *rdma_alloc_begin_uobject(const struct uverbs_api_object *obj,
- struct ib_uverbs_file *ufile);
-void rdma_alloc_abort_uobject(struct ib_uobject *uobj);
-int __must_check rdma_alloc_commit_uobject(struct ib_uobject *uobj);
+ struct ib_uverbs_file *ufile,
+ struct uverbs_attr_bundle *attrs);
+void rdma_alloc_abort_uobject(struct ib_uobject *uobj,
+ struct uverbs_attr_bundle *attrs);
+int __must_check rdma_alloc_commit_uobject(struct ib_uobject *uobj,
+ struct uverbs_attr_bundle *attrs);
struct uverbs_obj_fd_type {
/*
/* init.c */
-extern struct snd_card *snd_cards[SNDRV_CARDS];
int snd_card_locked(int card);
#if IS_ENABLED(CONFIG_SND_MIXER_OSS)
#define SND_MIXER_OSS_NOTIFY_REGISTER 0
int snd_component_add(struct snd_card *card, const char *component);
int snd_card_file_add(struct snd_card *card, struct file *file);
int snd_card_file_remove(struct snd_card *card, struct file *file);
-#define snd_card_unref(card) put_device(&(card)->card_dev)
+
+struct snd_card *snd_card_ref(int card);
+
+/**
+ * snd_card_unref - Unreference the card object
+ * @card: the card object to unreference
+ *
+ * Call this function for the card object that was obtained via snd_card_ref()
+ * or snd_lookup_minor_data().
+ */
+static inline void snd_card_unref(struct snd_card *card)
+{
+ put_device(&card->card_dev);
+}
#define snd_card_set_dev(card, devptr) ((card)->dev = (devptr))
struct da7219_aad_pdata;
+enum da7219_dai_clks {
+ DA7219_DAI_WCLK_IDX = 0,
+ DA7219_DAI_BCLK_IDX,
+ DA7219_DAI_NUM_CLKS,
+};
+
struct da7219_pdata {
bool wakeup_source;
- const char *dai_clks_name;
+ const char *dai_clk_names[DA7219_DAI_NUM_CLKS];
/* Mic */
enum da7219_micbias_voltage micbias_lvl;
* @num_streams: streams supported
* @idx: HDA link index
* @hlink_list: link list of HDA links
- * @lock: lock for link mgmt
+ * @lock: lock for link and display power mgmt
* @cmd_dma_state: state of cmd DMAs: CORB and RIRB
*/
struct hdac_bus {
/* locks */
spinlock_t reg_lock;
struct mutex cmd_mutex;
+ struct mutex lock;
/* DRM component interface */
struct drm_audio_component *audio_component;
long display_power_status;
- bool display_power_active;
+ unsigned long display_power_active;
/* parameters required for enhanced capabilities */
int num_streams;
int idx;
+ /* link management */
struct list_head hlink_list;
-
- struct mutex lock;
bool cmd_dma_state;
-
};
int snd_hdac_bus_init(struct hdac_bus *bus, struct device *dev,
struct snd_dma_buffer *dmab);
void snd_dma_free_pages(struct snd_dma_buffer *dmab);
-/* basic memory allocation functions */
-void *snd_malloc_pages(size_t size, gfp_t gfp_flags);
-void snd_free_pages(void *ptr, size_t size);
-
#endif /* __SOUND_MEMALLOC_H */
int snd_seq_create_kernel_client(struct snd_card *card, int client_index,
const char *name_fmt, ...);
int snd_seq_delete_kernel_client(int client);
-int snd_seq_kernel_client_enqueue(int client, struct snd_seq_event *ev, int atomic, int hop);
+int snd_seq_kernel_client_enqueue(int client, struct snd_seq_event *ev,
+ struct file *file, bool blocking);
int snd_seq_kernel_client_dispatch(int client, struct snd_seq_event *ev, int atomic, int hop);
int snd_seq_kernel_client_ctl(int client, unsigned int cmd, void *arg);
#include <sound/soc.h>
-#define asoc_simple_card_init_hp(card, sjack, prefix) \
- asoc_simple_card_init_jack(card, sjack, 1, prefix)
-#define asoc_simple_card_init_mic(card, sjack, prefix) \
- asoc_simple_card_init_jack(card, sjack, 0, prefix)
+#define asoc_simple_init_hp(card, sjack, prefix) \
+ asoc_simple_init_jack(card, sjack, 1, prefix)
+#define asoc_simple_init_mic(card, sjack, prefix) \
+ asoc_simple_init_jack(card, sjack, 0, prefix)
struct asoc_simple_dai {
const char *name;
struct clk *clk;
};
-struct asoc_simple_card_data {
+struct asoc_simple_data {
u32 convert_rate;
u32 convert_channels;
};
struct snd_soc_jack_gpio gpio;
};
-int asoc_simple_card_parse_daifmt(struct device *dev,
- struct device_node *node,
- struct device_node *codec,
- char *prefix,
- unsigned int *retfmt);
+struct asoc_simple_priv {
+ struct snd_soc_card snd_card;
+ struct simple_dai_props {
+ struct asoc_simple_dai *cpu_dai;
+ struct asoc_simple_dai *codec_dai;
+ struct snd_soc_dai_link_component codecs; /* single codec */
+ struct snd_soc_dai_link_component platforms;
+ struct asoc_simple_data adata;
+ struct snd_soc_codec_conf *codec_conf;
+ unsigned int mclk_fs;
+ } *dai_props;
+ struct asoc_simple_jack hp_jack;
+ struct asoc_simple_jack mic_jack;
+ struct snd_soc_dai_link *dai_link;
+ struct asoc_simple_dai *dais;
+ struct snd_soc_codec_conf *codec_conf;
+ struct gpio_desc *pa_gpio;
+};
+#define simple_priv_to_card(priv) (&(priv)->snd_card)
+#define simple_priv_to_props(priv, i) ((priv)->dai_props + (i))
+#define simple_priv_to_dev(priv) (simple_priv_to_card(priv)->dev)
+#define simple_priv_to_link(priv, i) (simple_priv_to_card(priv)->dai_link + (i))
+
+struct link_info {
+ int dais; /* number of dai */
+ int link; /* number of link */
+ int conf; /* number of codec_conf */
+ int cpu; /* turn for CPU / Codec */
+};
+
+int asoc_simple_parse_daifmt(struct device *dev,
+ struct device_node *node,
+ struct device_node *codec,
+ char *prefix,
+ unsigned int *retfmt);
__printf(3, 4)
-int asoc_simple_card_set_dailink_name(struct device *dev,
- struct snd_soc_dai_link *dai_link,
- const char *fmt, ...);
-int asoc_simple_card_parse_card_name(struct snd_soc_card *card,
- char *prefix);
-
-#define asoc_simple_card_parse_clk_cpu(dev, node, dai_link, simple_dai) \
- asoc_simple_card_parse_clk(dev, node, dai_link->cpu_of_node, simple_dai, \
+int asoc_simple_set_dailink_name(struct device *dev,
+ struct snd_soc_dai_link *dai_link,
+ const char *fmt, ...);
+int asoc_simple_parse_card_name(struct snd_soc_card *card,
+ char *prefix);
+
+#define asoc_simple_parse_clk_cpu(dev, node, dai_link, simple_dai) \
+ asoc_simple_parse_clk(dev, node, dai_link->cpu_of_node, simple_dai, \
dai_link->cpu_dai_name, NULL)
-#define asoc_simple_card_parse_clk_codec(dev, node, dai_link, simple_dai) \
- asoc_simple_card_parse_clk(dev, node, dai_link->codec_of_node, simple_dai,\
+#define asoc_simple_parse_clk_codec(dev, node, dai_link, simple_dai) \
+ asoc_simple_parse_clk(dev, node, dai_link->codec_of_node, simple_dai,\
dai_link->codec_dai_name, dai_link->codecs)
-int asoc_simple_card_parse_clk(struct device *dev,
- struct device_node *node,
- struct device_node *dai_of_node,
- struct asoc_simple_dai *simple_dai,
- const char *dai_name,
- struct snd_soc_dai_link_component *dlc);
-int asoc_simple_card_clk_enable(struct asoc_simple_dai *dai);
-void asoc_simple_card_clk_disable(struct asoc_simple_dai *dai);
-
-#define asoc_simple_card_parse_cpu(node, dai_link, \
- list_name, cells_name, is_single_link) \
- asoc_simple_card_parse_dai(node, NULL, \
- &dai_link->cpu_of_node, \
- &dai_link->cpu_dai_name, list_name, cells_name, is_single_link)
-#define asoc_simple_card_parse_codec(node, dai_link, list_name, cells_name) \
- asoc_simple_card_parse_dai(node, dai_link->codecs, \
+int asoc_simple_parse_clk(struct device *dev,
+ struct device_node *node,
+ struct device_node *dai_of_node,
+ struct asoc_simple_dai *simple_dai,
+ const char *dai_name,
+ struct snd_soc_dai_link_component *dlc);
+int asoc_simple_startup(struct snd_pcm_substream *substream);
+void asoc_simple_shutdown(struct snd_pcm_substream *substream);
+int asoc_simple_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params);
+int asoc_simple_dai_init(struct snd_soc_pcm_runtime *rtd);
+int asoc_simple_be_hw_params_fixup(struct snd_soc_pcm_runtime *rtd,
+ struct snd_pcm_hw_params *params);
+
+#define asoc_simple_parse_cpu(node, dai_link, is_single_link) \
+ asoc_simple_parse_dai(node, NULL, \
+ &dai_link->cpu_of_node, \
+ &dai_link->cpu_dai_name, is_single_link)
+#define asoc_simple_parse_codec(node, dai_link) \
+ asoc_simple_parse_dai(node, dai_link->codecs, \
&dai_link->codec_of_node, \
- &dai_link->codec_dai_name, \
- list_name, cells_name, NULL)
-#define asoc_simple_card_parse_platform(node, dai_link, list_name, cells_name) \
- asoc_simple_card_parse_dai(node, dai_link->platforms, \
- &dai_link->platform_of_node, \
- NULL, list_name, cells_name, NULL)
-int asoc_simple_card_parse_dai(struct device_node *node,
- struct snd_soc_dai_link_component *dlc,
- struct device_node **endpoint_np,
- const char **dai_name,
- const char *list_name,
- const char *cells_name,
- int *is_single_links);
-
-#define asoc_simple_card_parse_graph_cpu(ep, dai_link) \
- asoc_simple_card_parse_graph_dai(ep, NULL, \
- &dai_link->cpu_of_node, \
- &dai_link->cpu_dai_name)
-#define asoc_simple_card_parse_graph_codec(ep, dai_link) \
- asoc_simple_card_parse_graph_dai(ep, dai_link->codecs, \
- &dai_link->codec_of_node, \
- &dai_link->codec_dai_name)
-int asoc_simple_card_parse_graph_dai(struct device_node *ep,
- struct snd_soc_dai_link_component *dlc,
- struct device_node **endpoint_np,
- const char **dai_name);
-
-#define asoc_simple_card_of_parse_tdm(np, dai) \
+ &dai_link->codec_dai_name, NULL)
+#define asoc_simple_parse_platform(node, dai_link) \
+ asoc_simple_parse_dai(node, dai_link->platforms, \
+ &dai_link->platform_of_node, NULL, NULL)
+
+#define asoc_simple_parse_tdm(np, dai) \
snd_soc_of_parse_tdm_slot(np, &(dai)->tx_slot_mask, \
&(dai)->rx_slot_mask, \
&(dai)->slots, \
&(dai)->slot_width);
-int asoc_simple_card_init_dai(struct snd_soc_dai *dai,
- struct asoc_simple_dai *simple_dai);
-
-void asoc_simple_card_canonicalize_platform(struct snd_soc_dai_link *dai_link);
-void asoc_simple_card_canonicalize_cpu(struct snd_soc_dai_link *dai_link,
+void asoc_simple_canonicalize_platform(struct snd_soc_dai_link *dai_link);
+void asoc_simple_canonicalize_cpu(struct snd_soc_dai_link *dai_link,
int is_single_links);
-int asoc_simple_card_clean_reference(struct snd_soc_card *card);
+int asoc_simple_clean_reference(struct snd_soc_card *card);
-void asoc_simple_card_convert_fixup(struct asoc_simple_card_data *data,
+void asoc_simple_convert_fixup(struct asoc_simple_data *data,
struct snd_pcm_hw_params *params);
-void asoc_simple_card_parse_convert(struct device *dev,
- struct device_node *np, char *prefix,
- struct asoc_simple_card_data *data);
+void asoc_simple_parse_convert(struct device *dev,
+ struct device_node *np, char *prefix,
+ struct asoc_simple_data *data);
-int asoc_simple_card_of_parse_routing(struct snd_soc_card *card,
+int asoc_simple_parse_routing(struct snd_soc_card *card,
char *prefix);
-int asoc_simple_card_of_parse_widgets(struct snd_soc_card *card,
+int asoc_simple_parse_widgets(struct snd_soc_card *card,
char *prefix);
+int asoc_simple_parse_pin_switches(struct snd_soc_card *card,
+ char *prefix);
-int asoc_simple_card_init_jack(struct snd_soc_card *card,
+int asoc_simple_init_jack(struct snd_soc_card *card,
struct asoc_simple_jack *sjack,
int is_hp, char *prefix);
+int asoc_simple_init_priv(struct asoc_simple_priv *priv,
+ struct link_info *li);
+
+#ifdef DEBUG
+inline void asoc_simple_debug_dai(struct asoc_simple_priv *priv,
+ char *name,
+ struct asoc_simple_dai *dai)
+{
+ struct device *dev = simple_priv_to_dev(priv);
+
+ if (dai->name)
+ dev_dbg(dev, "%s dai name = %s\n",
+ name, dai->name);
+ if (dai->sysclk)
+ dev_dbg(dev, "%s sysclk = %d\n",
+ name, dai->sysclk);
+
+ dev_dbg(dev, "%s direction = %s\n",
+ name, dai->clk_direction ? "OUT" : "IN");
+
+ if (dai->slots)
+ dev_dbg(dev, "%s slots = %d\n", name, dai->slots);
+ if (dai->slot_width)
+ dev_dbg(dev, "%s slot width = %d\n", name, dai->slot_width);
+ if (dai->tx_slot_mask)
+ dev_dbg(dev, "%s tx slot mask = %d\n", name, dai->tx_slot_mask);
+ if (dai->rx_slot_mask)
+ dev_dbg(dev, "%s rx slot mask = %d\n", name, dai->rx_slot_mask);
+ if (dai->clk)
+ dev_dbg(dev, "%s clk %luHz\n", name, clk_get_rate(dai->clk));
+}
+
+inline void asoc_simple_debug_info(struct asoc_simple_priv *priv)
+{
+ struct snd_soc_card *card = simple_priv_to_card(priv);
+ struct device *dev = simple_priv_to_dev(priv);
+
+ int i;
+
+ if (card->name)
+ dev_dbg(dev, "Card Name: %s\n", card->name);
+
+ for (i = 0; i < card->num_links; i++) {
+ struct simple_dai_props *props = simple_priv_to_props(priv, i);
+ struct snd_soc_dai_link *link = simple_priv_to_link(priv, i);
+
+ dev_dbg(dev, "DAI%d\n", i);
+
+ asoc_simple_debug_dai(priv, "cpu", props->cpu_dai);
+ asoc_simple_debug_dai(priv, "codec", props->codec_dai);
+
+ if (link->name)
+ dev_dbg(dev, "dai name = %s\n", link->name);
+
+ dev_dbg(dev, "dai format = %04x\n", link->dai_fmt);
+
+ if (props->adata.convert_rate)
+ dev_dbg(dev, "convert_rate = %d\n",
+ props->adata.convert_rate);
+ if (props->adata.convert_channels)
+ dev_dbg(dev, "convert_channels = %d\n",
+ props->adata.convert_channels);
+ if (props->codec_conf && props->codec_conf->name_prefix)
+ dev_dbg(dev, "name prefix = %s\n",
+ props->codec_conf->name_prefix);
+ if (props->mclk_fs)
+ dev_dbg(dev, "mclk-fs = %d\n",
+ props->mclk_fs);
+ }
+}
+#else
+#define asoc_simple_debug_info(priv)
+#endif /* DEBUG */
#endif /* __SIMPLE_CARD_UTILS_H */
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ *
+ * Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_H
+#define __INCLUDE_SOUND_SOF_H
+
+#include <linux/pci.h>
+#include <sound/soc.h>
+#include <sound/soc-acpi.h>
+
+struct snd_sof_dsp_ops;
+
+/*
+ * SOF Platform data.
+ */
+struct snd_sof_pdata {
+ const struct firmware *fw;
+ const char *drv_name;
+ const char *name;
+ const char *platform;
+
+ struct device *dev;
+
+ /*
+ * notification callback used if the hardware initialization
+ * can take time or is handled in a workqueue. This callback
+ * can be used by the caller to e.g. enable runtime_pm
+ * or limit functionality until all low-level inits are
+ * complete.
+ */
+ void (*sof_probe_complete)(struct device *dev);
+
+ /* descriptor */
+ const struct sof_dev_desc *desc;
+
+ /* firmware and topology filenames */
+ const char *fw_filename_prefix;
+ const char *fw_filename;
+ const char *tplg_filename_prefix;
+ const char *tplg_filename;
+
+ /* machine */
+ struct platform_device *pdev_mach;
+ const struct snd_soc_acpi_mach *machine;
+
+ void *hw_pdata;
+};
+
+/*
+ * Descriptor used for setting up SOF platform data. This is used when
+ * ACPI/PCI data is missing or mapped differently.
+ */
+struct sof_dev_desc {
+ /* list of machines using this configuration */
+ struct snd_soc_acpi_mach *machines;
+
+ /* Platform resource indexes in BAR / ACPI resources. */
+ /* Must set to -1 if not used - add new items to end */
+ int resindex_lpe_base;
+ int resindex_pcicfg_base;
+ int resindex_imr_base;
+ int irqindex_host_ipc;
+ int resindex_dma_base;
+
+ /* DMA only valid when resindex_dma_base != -1*/
+ int dma_engine;
+ int dma_size;
+
+ /* IPC timeouts in ms */
+ int ipc_timeout;
+ int boot_timeout;
+
+ /* chip information for dsp */
+ const void *chip_info;
+
+ /* defaults for no codec mode */
+ const char *nocodec_fw_filename;
+ const char *nocodec_tplg_filename;
+
+ /* defaults paths for firmware and topology files */
+ const char *default_fw_path;
+ const char *default_tplg_path;
+
+ const struct snd_sof_dsp_ops *ops;
+ const struct sof_arch_ops *arch_ops;
+};
+
+int sof_nocodec_setup(struct device *dev,
+ struct snd_sof_pdata *sof_pdata,
+ struct snd_soc_acpi_mach *mach,
+ const struct sof_dev_desc *desc,
+ const struct snd_sof_dsp_ops *ops);
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_CONTROL_H__
+#define __INCLUDE_SOUND_SOF_CONTROL_H__
+
+#include <uapi/sound/sof/header.h>
+#include <sound/sof/header.h>
+
+/*
+ * Component Mixers and Controls
+ */
+
+/* channel positions - uses same values as ALSA */
+enum sof_ipc_chmap {
+ SOF_CHMAP_UNKNOWN = 0,
+ SOF_CHMAP_NA, /**< N/A, silent */
+ SOF_CHMAP_MONO, /**< mono stream */
+ SOF_CHMAP_FL, /**< front left */
+ SOF_CHMAP_FR, /**< front right */
+ SOF_CHMAP_RL, /**< rear left */
+ SOF_CHMAP_RR, /**< rear right */
+ SOF_CHMAP_FC, /**< front centre */
+ SOF_CHMAP_LFE, /**< LFE */
+ SOF_CHMAP_SL, /**< side left */
+ SOF_CHMAP_SR, /**< side right */
+ SOF_CHMAP_RC, /**< rear centre */
+ SOF_CHMAP_FLC, /**< front left centre */
+ SOF_CHMAP_FRC, /**< front right centre */
+ SOF_CHMAP_RLC, /**< rear left centre */
+ SOF_CHMAP_RRC, /**< rear right centre */
+ SOF_CHMAP_FLW, /**< front left wide */
+ SOF_CHMAP_FRW, /**< front right wide */
+ SOF_CHMAP_FLH, /**< front left high */
+ SOF_CHMAP_FCH, /**< front centre high */
+ SOF_CHMAP_FRH, /**< front right high */
+ SOF_CHMAP_TC, /**< top centre */
+ SOF_CHMAP_TFL, /**< top front left */
+ SOF_CHMAP_TFR, /**< top front right */
+ SOF_CHMAP_TFC, /**< top front centre */
+ SOF_CHMAP_TRL, /**< top rear left */
+ SOF_CHMAP_TRR, /**< top rear right */
+ SOF_CHMAP_TRC, /**< top rear centre */
+ SOF_CHMAP_TFLC, /**< top front left centre */
+ SOF_CHMAP_TFRC, /**< top front right centre */
+ SOF_CHMAP_TSL, /**< top side left */
+ SOF_CHMAP_TSR, /**< top side right */
+ SOF_CHMAP_LLFE, /**< left LFE */
+ SOF_CHMAP_RLFE, /**< right LFE */
+ SOF_CHMAP_BC, /**< bottom centre */
+ SOF_CHMAP_BLC, /**< bottom left centre */
+ SOF_CHMAP_BRC, /**< bottom right centre */
+ SOF_CHMAP_LAST = SOF_CHMAP_BRC,
+};
+
+/* control data type and direction */
+enum sof_ipc_ctrl_type {
+ /* per channel data - uses struct sof_ipc_ctrl_value_chan */
+ SOF_CTRL_TYPE_VALUE_CHAN_GET = 0,
+ SOF_CTRL_TYPE_VALUE_CHAN_SET,
+ /* component data - uses struct sof_ipc_ctrl_value_comp */
+ SOF_CTRL_TYPE_VALUE_COMP_GET,
+ SOF_CTRL_TYPE_VALUE_COMP_SET,
+ /* bespoke data - uses struct sof_abi_hdr */
+ SOF_CTRL_TYPE_DATA_GET,
+ SOF_CTRL_TYPE_DATA_SET,
+};
+
+/* control command type */
+enum sof_ipc_ctrl_cmd {
+ SOF_CTRL_CMD_VOLUME = 0, /**< maps to ALSA volume style controls */
+ SOF_CTRL_CMD_ENUM, /**< maps to ALSA enum style controls */
+ SOF_CTRL_CMD_SWITCH, /**< maps to ALSA switch style controls */
+ SOF_CTRL_CMD_BINARY, /**< maps to ALSA binary style controls */
+};
+
+/* generic channel mapped value data */
+struct sof_ipc_ctrl_value_chan {
+ uint32_t channel; /**< channel map - enum sof_ipc_chmap */
+ uint32_t value;
+} __packed;
+
+/* generic component mapped value data */
+struct sof_ipc_ctrl_value_comp {
+ uint32_t index; /**< component source/sink/control index in control */
+ union {
+ uint32_t uvalue;
+ int32_t svalue;
+ };
+} __packed;
+
+/* generic control data */
+struct sof_ipc_ctrl_data {
+ struct sof_ipc_reply rhdr;
+ uint32_t comp_id;
+
+ /* control access and data type */
+ uint32_t type; /**< enum sof_ipc_ctrl_type */
+ uint32_t cmd; /**< enum sof_ipc_ctrl_cmd */
+ uint32_t index; /**< control index for comps > 1 control */
+
+ /* control data - can either be appended or DMAed from host */
+ struct sof_ipc_host_buffer buffer;
+ uint32_t num_elems; /**< in array elems or bytes for data type */
+ uint32_t elems_remaining; /**< elems remaining if sent in parts */
+
+ uint32_t msg_index; /**< for large messages sent in parts */
+
+ /* reserved for future use */
+ uint32_t reserved[6];
+
+ /* control data - add new types if needed */
+ union {
+ /* channel values can be used by volume type controls */
+ struct sof_ipc_ctrl_value_chan chanv[0];
+ /* component values used by routing controls like mux, mixer */
+ struct sof_ipc_ctrl_value_comp compv[0];
+ /* data can be used by binary controls */
+ struct sof_abi_hdr data[0];
+ };
+} __packed;
+
+/** Event type */
+enum sof_ipc_ctrl_event_type {
+ SOF_CTRL_EVENT_GENERIC = 0, /**< generic event */
+ SOF_CTRL_EVENT_GENERIC_METADATA, /**< generic event with metadata */
+ SOF_CTRL_EVENT_KD, /**< keyword detection event */
+ SOF_CTRL_EVENT_VAD, /**< voice activity detection event */
+};
+
+/**
+ * Generic notification data.
+ */
+struct sof_ipc_comp_event {
+ struct sof_ipc_reply rhdr;
+ uint16_t src_comp_type; /**< COMP_TYPE_ */
+ uint32_t src_comp_id; /**< source component id */
+ uint32_t event_type; /**< event type - SOF_CTRL_EVENT_* */
+ uint32_t num_elems; /**< in array elems or bytes for data type */
+
+ /* reserved for future use */
+ uint32_t reserved[8];
+
+ /* control data - add new types if needed */
+ union {
+ /* data can be used by binary controls */
+ struct sof_abi_hdr data[0];
+ /* event specific values */
+ uint32_t event_value;
+ };
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_DAI_INTEL_H__
+#define __INCLUDE_SOUND_SOF_DAI_INTEL_H__
+
+#include <sound/sof/header.h>
+
+ /* ssc1: TINTE */
+#define SOF_DAI_INTEL_SSP_QUIRK_TINTE (1 << 0)
+ /* ssc1: PINTE */
+#define SOF_DAI_INTEL_SSP_QUIRK_PINTE (1 << 1)
+ /* ssc2: SMTATF */
+#define SOF_DAI_INTEL_SSP_QUIRK_SMTATF (1 << 2)
+ /* ssc2: MMRATF */
+#define SOF_DAI_INTEL_SSP_QUIRK_MMRATF (1 << 3)
+ /* ssc2: PSPSTWFDFD */
+#define SOF_DAI_INTEL_SSP_QUIRK_PSPSTWFDFD (1 << 4)
+ /* ssc2: PSPSRWFDFD */
+#define SOF_DAI_INTEL_SSP_QUIRK_PSPSRWFDFD (1 << 5)
+/* ssc1: LBM */
+#define SOF_DAI_INTEL_SSP_QUIRK_LBM (1 << 6)
+
+ /* here is the possibility to define others aux macros */
+
+#define SOF_DAI_INTEL_SSP_FRAME_PULSE_WIDTH_MAX 38
+#define SOF_DAI_INTEL_SSP_SLOT_PADDING_MAX 31
+
+/* SSP clocks control settings
+ *
+ * Macros for clks_control field in sof_ipc_dai_ssp_params struct.
+ */
+
+/* mclk 0 disable */
+#define SOF_DAI_INTEL_SSP_MCLK_0_DISABLE BIT(0)
+/* mclk 1 disable */
+#define SOF_DAI_INTEL_SSP_MCLK_1_DISABLE BIT(1)
+/* mclk keep active */
+#define SOF_DAI_INTEL_SSP_CLKCTRL_MCLK_KA BIT(2)
+/* bclk keep active */
+#define SOF_DAI_INTEL_SSP_CLKCTRL_BCLK_KA BIT(3)
+/* fs keep active */
+#define SOF_DAI_INTEL_SSP_CLKCTRL_FS_KA BIT(4)
+/* bclk idle */
+#define SOF_DAI_INTEL_SSP_CLKCTRL_BCLK_IDLE_HIGH BIT(5)
+
+/* SSP Configuration Request - SOF_IPC_DAI_SSP_CONFIG */
+struct sof_ipc_dai_ssp_params {
+ struct sof_ipc_hdr hdr;
+ uint16_t reserved1;
+ uint16_t mclk_id;
+
+ uint32_t mclk_rate; /* mclk frequency in Hz */
+ uint32_t fsync_rate; /* fsync frequency in Hz */
+ uint32_t bclk_rate; /* bclk frequency in Hz */
+
+ /* TDM */
+ uint32_t tdm_slots;
+ uint32_t rx_slots;
+ uint32_t tx_slots;
+
+ /* data */
+ uint32_t sample_valid_bits;
+ uint16_t tdm_slot_width;
+ uint16_t reserved2; /* alignment */
+
+ /* MCLK */
+ uint32_t mclk_direction;
+
+ uint16_t frame_pulse_width;
+ uint16_t tdm_per_slot_padding_flag;
+ uint32_t clks_control;
+ uint32_t quirks;
+} __packed;
+
+/* HDA Configuration Request - SOF_IPC_DAI_HDA_CONFIG */
+struct sof_ipc_dai_hda_params {
+ struct sof_ipc_hdr hdr;
+ uint32_t link_dma_ch;
+} __packed;
+
+/* DMIC Configuration Request - SOF_IPC_DAI_DMIC_CONFIG */
+
+/* This struct is defined per 2ch PDM controller available in the platform.
+ * Normally it is sufficient to set the used microphone specific enables to 1
+ * and keep other parameters as zero. The customizations are:
+ *
+ * 1. If a device mixes different microphones types with different polarity
+ * and/or the absolute polarity matters the PCM signal from a microphone
+ * can be inverted with the controls.
+ *
+ * 2. If the microphones in a stereo pair do not appear in captured stream
+ * in desired order due to board schematics choises they can be swapped with
+ * the clk_edge parameter.
+ *
+ * 3. If PDM bit errors are seen in capture (poor quality) the skew parameter
+ * that delays the sampling time of data by half cycles of DMIC source clock
+ * can be tried for improvement. However there is no guarantee for this to fix
+ * data integrity problems.
+ */
+struct sof_ipc_dai_dmic_pdm_ctrl {
+ struct sof_ipc_hdr hdr;
+ uint16_t id; /**< PDM controller ID */
+
+ uint16_t enable_mic_a; /**< Use A (left) channel mic (0 or 1)*/
+ uint16_t enable_mic_b; /**< Use B (right) channel mic (0 or 1)*/
+
+ uint16_t polarity_mic_a; /**< Optionally invert mic A signal (0 or 1) */
+ uint16_t polarity_mic_b; /**< Optionally invert mic B signal (0 or 1) */
+
+ uint16_t clk_edge; /**< Optionally swap data clock edge (0 or 1) */
+ uint16_t skew; /**< Adjust PDM data sampling vs. clock (0..15) */
+
+ uint16_t reserved[3]; /**< Make sure the total size is 4 bytes aligned */
+} __packed;
+
+/* This struct contains the global settings for all 2ch PDM controllers. The
+ * version number used in configuration data is checked vs. version used by
+ * device driver src/drivers/dmic.c need to match. It is incremented from
+ * initial value 1 if updates done for the to driver would alter the operation
+ * of the microhone.
+ *
+ * Note: The microphone clock (pdmclk_min, pdmclk_max, duty_min, duty_max)
+ * parameters need to be set as defined in microphone data sheet. E.g. clock
+ * range 1.0 - 3.2 MHz is usually supported microphones. Some microphones are
+ * multi-mode capable and there may be denied mic clock frequencies between
+ * the modes. In such case set the clock range limits of the desired mode to
+ * avoid the driver to set clock to an illegal rate.
+ *
+ * The duty cycle could be set to 48-52% if not known. Generally these
+ * parameters can be altered within data sheet specified limits to match
+ * required audio application performance power.
+ *
+ * The microphone clock needs to be usually about 50-80 times the used audio
+ * sample rate. With highest sample rates above 48 kHz this can relaxed
+ * somewhat.
+ *
+ * The parameter wake_up_time describes how long time the microphone needs
+ * for the data line to produce valid output from mic clock start. The driver
+ * will mute the captured audio for the given time. The min_clock_on_time
+ * parameter is used to prevent too short clock bursts to happen. The driver
+ * will keep the clock active after capture stop if this time is not yet
+ * met. The unit for both is microseconds (us). Exceed of 100 ms will be
+ * treated as an error.
+ */
+struct sof_ipc_dai_dmic_params {
+ struct sof_ipc_hdr hdr;
+ uint32_t driver_ipc_version; /**< Version (1..N) */
+
+ uint32_t pdmclk_min; /**< Minimum microphone clock in Hz (100000..N) */
+ uint32_t pdmclk_max; /**< Maximum microphone clock in Hz (min...N) */
+
+ uint32_t fifo_fs; /**< FIFO sample rate in Hz (8000..96000) */
+ uint32_t reserved_1; /**< Reserved */
+ uint16_t fifo_bits; /**< FIFO word length (16 or 32) */
+ uint16_t reserved_2; /**< Reserved */
+
+ uint16_t duty_min; /**< Min. mic clock duty cycle in % (20..80) */
+ uint16_t duty_max; /**< Max. mic clock duty cycle in % (min..80) */
+
+ uint32_t num_pdm_active; /**< Number of active pdm controllers */
+
+ uint32_t wake_up_time; /**< Time from clock start to data (us) */
+ uint32_t min_clock_on_time; /**< Min. time that clk is kept on (us) */
+
+ /* reserved for future use */
+ uint32_t reserved[6];
+
+ /**< variable number of pdm controller config */
+ struct sof_ipc_dai_dmic_pdm_ctrl pdm[0];
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_DAI_H__
+#define __INCLUDE_SOUND_SOF_DAI_H__
+
+#include <sound/sof/header.h>
+#include <sound/sof/dai-intel.h>
+
+/*
+ * DAI Configuration.
+ *
+ * Each different DAI type will have it's own structure and IPC cmd.
+ */
+
+#define SOF_DAI_FMT_I2S 1 /**< I2S mode */
+#define SOF_DAI_FMT_RIGHT_J 2 /**< Right Justified mode */
+#define SOF_DAI_FMT_LEFT_J 3 /**< Left Justified mode */
+#define SOF_DAI_FMT_DSP_A 4 /**< L data MSB after FRM LRC */
+#define SOF_DAI_FMT_DSP_B 5 /**< L data MSB during FRM LRC */
+#define SOF_DAI_FMT_PDM 6 /**< Pulse density modulation */
+
+#define SOF_DAI_FMT_CONT (1 << 4) /**< continuous clock */
+#define SOF_DAI_FMT_GATED (0 << 4) /**< clock is gated */
+
+#define SOF_DAI_FMT_NB_NF (0 << 8) /**< normal bit clock + frame */
+#define SOF_DAI_FMT_NB_IF (2 << 8) /**< normal BCLK + inv FRM */
+#define SOF_DAI_FMT_IB_NF (3 << 8) /**< invert BCLK + nor FRM */
+#define SOF_DAI_FMT_IB_IF (4 << 8) /**< invert BCLK + FRM */
+
+#define SOF_DAI_FMT_CBM_CFM (0 << 12) /**< codec clk & FRM master */
+#define SOF_DAI_FMT_CBS_CFM (2 << 12) /**< codec clk slave & FRM master */
+#define SOF_DAI_FMT_CBM_CFS (3 << 12) /**< codec clk master & frame slave */
+#define SOF_DAI_FMT_CBS_CFS (4 << 12) /**< codec clk & FRM slave */
+
+#define SOF_DAI_FMT_FORMAT_MASK 0x000f
+#define SOF_DAI_FMT_CLOCK_MASK 0x00f0
+#define SOF_DAI_FMT_INV_MASK 0x0f00
+#define SOF_DAI_FMT_MASTER_MASK 0xf000
+
+/** \brief Types of DAI */
+enum sof_ipc_dai_type {
+ SOF_DAI_INTEL_NONE = 0, /**< None */
+ SOF_DAI_INTEL_SSP, /**< Intel SSP */
+ SOF_DAI_INTEL_DMIC, /**< Intel DMIC */
+ SOF_DAI_INTEL_HDA, /**< Intel HD/A */
+};
+
+/* general purpose DAI configuration */
+struct sof_ipc_dai_config {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t type; /**< DAI type - enum sof_ipc_dai_type */
+ uint32_t dai_index; /**< index of this type dai */
+
+ /* physical protocol and clocking */
+ uint16_t format; /**< SOF_DAI_FMT_ */
+ uint16_t reserved16; /**< alignment */
+
+ /* reserved for future use */
+ uint32_t reserved[8];
+
+ /* HW specific data */
+ union {
+ struct sof_ipc_dai_ssp_params ssp;
+ struct sof_ipc_dai_dmic_params dmic;
+ struct sof_ipc_dai_hda_params hda;
+ };
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_HEADER_H__
+#define __INCLUDE_SOUND_SOF_HEADER_H__
+
+#include <uapi/sound/sof/abi.h>
+
+/** \addtogroup sof_uapi uAPI
+ * SOF uAPI specification.
+ * @{
+ */
+
+/*
+ * IPC messages have a prefixed 32 bit identifier made up as follows :-
+ *
+ * 0xGCCCNNNN where
+ * G is global cmd type (4 bits)
+ * C is command type (12 bits)
+ * I is the ID number (16 bits) - monotonic and overflows
+ *
+ * This is sent at the start of the IPM message in the mailbox. Messages should
+ * not be sent in the doorbell (special exceptions for firmware .
+ */
+
+/* Global Message - Generic */
+#define SOF_GLB_TYPE_SHIFT 28
+#define SOF_GLB_TYPE_MASK (0xf << SOF_GLB_TYPE_SHIFT)
+#define SOF_GLB_TYPE(x) ((x) << SOF_GLB_TYPE_SHIFT)
+
+/* Command Message - Generic */
+#define SOF_CMD_TYPE_SHIFT 16
+#define SOF_CMD_TYPE_MASK (0xfff << SOF_CMD_TYPE_SHIFT)
+#define SOF_CMD_TYPE(x) ((x) << SOF_CMD_TYPE_SHIFT)
+
+/* Global Message Types */
+#define SOF_IPC_GLB_REPLY SOF_GLB_TYPE(0x1U)
+#define SOF_IPC_GLB_COMPOUND SOF_GLB_TYPE(0x2U)
+#define SOF_IPC_GLB_TPLG_MSG SOF_GLB_TYPE(0x3U)
+#define SOF_IPC_GLB_PM_MSG SOF_GLB_TYPE(0x4U)
+#define SOF_IPC_GLB_COMP_MSG SOF_GLB_TYPE(0x5U)
+#define SOF_IPC_GLB_STREAM_MSG SOF_GLB_TYPE(0x6U)
+#define SOF_IPC_FW_READY SOF_GLB_TYPE(0x7U)
+#define SOF_IPC_GLB_DAI_MSG SOF_GLB_TYPE(0x8U)
+#define SOF_IPC_GLB_TRACE_MSG SOF_GLB_TYPE(0x9U)
+
+/*
+ * DSP Command Message Types
+ */
+
+/* topology */
+#define SOF_IPC_TPLG_COMP_NEW SOF_CMD_TYPE(0x001)
+#define SOF_IPC_TPLG_COMP_FREE SOF_CMD_TYPE(0x002)
+#define SOF_IPC_TPLG_COMP_CONNECT SOF_CMD_TYPE(0x003)
+#define SOF_IPC_TPLG_PIPE_NEW SOF_CMD_TYPE(0x010)
+#define SOF_IPC_TPLG_PIPE_FREE SOF_CMD_TYPE(0x011)
+#define SOF_IPC_TPLG_PIPE_CONNECT SOF_CMD_TYPE(0x012)
+#define SOF_IPC_TPLG_PIPE_COMPLETE SOF_CMD_TYPE(0x013)
+#define SOF_IPC_TPLG_BUFFER_NEW SOF_CMD_TYPE(0x020)
+#define SOF_IPC_TPLG_BUFFER_FREE SOF_CMD_TYPE(0x021)
+
+/* PM */
+#define SOF_IPC_PM_CTX_SAVE SOF_CMD_TYPE(0x001)
+#define SOF_IPC_PM_CTX_RESTORE SOF_CMD_TYPE(0x002)
+#define SOF_IPC_PM_CTX_SIZE SOF_CMD_TYPE(0x003)
+#define SOF_IPC_PM_CLK_SET SOF_CMD_TYPE(0x004)
+#define SOF_IPC_PM_CLK_GET SOF_CMD_TYPE(0x005)
+#define SOF_IPC_PM_CLK_REQ SOF_CMD_TYPE(0x006)
+#define SOF_IPC_PM_CORE_ENABLE SOF_CMD_TYPE(0x007)
+
+/* component runtime config - multiple different types */
+#define SOF_IPC_COMP_SET_VALUE SOF_CMD_TYPE(0x001)
+#define SOF_IPC_COMP_GET_VALUE SOF_CMD_TYPE(0x002)
+#define SOF_IPC_COMP_SET_DATA SOF_CMD_TYPE(0x003)
+#define SOF_IPC_COMP_GET_DATA SOF_CMD_TYPE(0x004)
+
+/* DAI messages */
+#define SOF_IPC_DAI_CONFIG SOF_CMD_TYPE(0x001)
+#define SOF_IPC_DAI_LOOPBACK SOF_CMD_TYPE(0x002)
+
+/* stream */
+#define SOF_IPC_STREAM_PCM_PARAMS SOF_CMD_TYPE(0x001)
+#define SOF_IPC_STREAM_PCM_PARAMS_REPLY SOF_CMD_TYPE(0x002)
+#define SOF_IPC_STREAM_PCM_FREE SOF_CMD_TYPE(0x003)
+#define SOF_IPC_STREAM_TRIG_START SOF_CMD_TYPE(0x004)
+#define SOF_IPC_STREAM_TRIG_STOP SOF_CMD_TYPE(0x005)
+#define SOF_IPC_STREAM_TRIG_PAUSE SOF_CMD_TYPE(0x006)
+#define SOF_IPC_STREAM_TRIG_RELEASE SOF_CMD_TYPE(0x007)
+#define SOF_IPC_STREAM_TRIG_DRAIN SOF_CMD_TYPE(0x008)
+#define SOF_IPC_STREAM_TRIG_XRUN SOF_CMD_TYPE(0x009)
+#define SOF_IPC_STREAM_POSITION SOF_CMD_TYPE(0x00a)
+#define SOF_IPC_STREAM_VORBIS_PARAMS SOF_CMD_TYPE(0x010)
+#define SOF_IPC_STREAM_VORBIS_FREE SOF_CMD_TYPE(0x011)
+
+/* trace and debug */
+#define SOF_IPC_TRACE_DMA_PARAMS SOF_CMD_TYPE(0x001)
+#define SOF_IPC_TRACE_DMA_POSITION SOF_CMD_TYPE(0x002)
+
+/* Get message component id */
+#define SOF_IPC_MESSAGE_ID(x) ((x) & 0xffff)
+
+/* maximum message size for mailbox Tx/Rx */
+#define SOF_IPC_MSG_MAX_SIZE 384
+
+/*
+ * Structure Header - Header for all IPC structures except command structs.
+ * The size can be greater than the structure size and that means there is
+ * extended bespoke data beyond the end of the structure including variable
+ * arrays.
+ */
+
+struct sof_ipc_hdr {
+ uint32_t size; /**< size of structure */
+} __packed;
+
+/*
+ * Command Header - Header for all IPC commands. Identifies IPC message.
+ * The size can be greater than the structure size and that means there is
+ * extended bespoke data beyond the end of the structure including variable
+ * arrays.
+ */
+
+struct sof_ipc_cmd_hdr {
+ uint32_t size; /**< size of structure */
+ uint32_t cmd; /**< SOF_IPC_GLB_ + cmd */
+} __packed;
+
+/*
+ * Generic reply message. Some commands override this with their own reply
+ * types that must include this at start.
+ */
+struct sof_ipc_reply {
+ struct sof_ipc_cmd_hdr hdr;
+ int32_t error; /**< negative error numbers */
+} __packed;
+
+/*
+ * Compound commands - SOF_IPC_GLB_COMPOUND.
+ *
+ * Compound commands are sent to the DSP as a single IPC operation. The
+ * commands are split into blocks and each block has a header. This header
+ * identifies the command type and the number of commands before the next
+ * header.
+ */
+
+struct sof_ipc_compound_hdr {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t count; /**< count of 0 means end of compound sequence */
+} __packed;
+
+/** @}*/
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_INFO_H__
+#define __INCLUDE_SOUND_SOF_INFO_H__
+
+#include <sound/sof/header.h>
+#include <sound/sof/stream.h>
+
+/*
+ * Firmware boot and version
+ */
+
+#define SOF_IPC_MAX_ELEMS 16
+
+/* extended data types that can be appended onto end of sof_ipc_fw_ready */
+enum sof_ipc_ext_data {
+ SOF_IPC_EXT_DMA_BUFFER = 0,
+ SOF_IPC_EXT_WINDOW,
+};
+
+/* FW version - SOF_IPC_GLB_VERSION */
+struct sof_ipc_fw_version {
+ struct sof_ipc_hdr hdr;
+ uint16_t major;
+ uint16_t minor;
+ uint16_t micro;
+ uint16_t build;
+ uint8_t date[12];
+ uint8_t time[10];
+ uint8_t tag[6];
+ uint32_t abi_version;
+
+ /* reserved for future use */
+ uint32_t reserved[4];
+} __packed;
+
+/* FW ready Message - sent by firmware when boot has completed */
+struct sof_ipc_fw_ready {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t dspbox_offset; /* dsp initiated IPC mailbox */
+ uint32_t hostbox_offset; /* host initiated IPC mailbox */
+ uint32_t dspbox_size;
+ uint32_t hostbox_size;
+ struct sof_ipc_fw_version version;
+
+ /* Miscellaneous debug flags showing build/debug features enabled */
+ union {
+ uint64_t reserved;
+ struct {
+ uint64_t build:1;
+ uint64_t locks:1;
+ uint64_t locks_verbose:1;
+ uint64_t gdb:1;
+ } bits;
+ } debug;
+
+ /* reserved for future use */
+ uint32_t reserved[4];
+} __packed;
+
+/*
+ * Extended Firmware data. All optional, depends on platform/arch.
+ */
+enum sof_ipc_region {
+ SOF_IPC_REGION_DOWNBOX = 0,
+ SOF_IPC_REGION_UPBOX,
+ SOF_IPC_REGION_TRACE,
+ SOF_IPC_REGION_DEBUG,
+ SOF_IPC_REGION_STREAM,
+ SOF_IPC_REGION_REGS,
+ SOF_IPC_REGION_EXCEPTION,
+};
+
+struct sof_ipc_ext_data_hdr {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t type; /**< SOF_IPC_EXT_ */
+} __packed;
+
+struct sof_ipc_dma_buffer_elem {
+ struct sof_ipc_hdr hdr;
+ uint32_t type; /**< SOF_IPC_REGION_ */
+ uint32_t id; /**< platform specific - used to map to host memory */
+ struct sof_ipc_host_buffer buffer;
+} __packed;
+
+/* extended data DMA buffers for IPC, trace and debug */
+struct sof_ipc_dma_buffer_data {
+ struct sof_ipc_ext_data_hdr ext_hdr;
+ uint32_t num_buffers;
+
+ /* host files in buffer[n].buffer */
+ struct sof_ipc_dma_buffer_elem buffer[];
+} __packed;
+
+struct sof_ipc_window_elem {
+ struct sof_ipc_hdr hdr;
+ uint32_t type; /**< SOF_IPC_REGION_ */
+ uint32_t id; /**< platform specific - used to map to host memory */
+ uint32_t flags; /**< R, W, RW, etc - to define */
+ uint32_t size; /**< size of region in bytes */
+ /* offset in window region as windows can be partitioned */
+ uint32_t offset;
+} __packed;
+
+/* extended data memory windows for IPC, trace and debug */
+struct sof_ipc_window {
+ struct sof_ipc_ext_data_hdr ext_hdr;
+ uint32_t num_windows;
+ struct sof_ipc_window_elem window[];
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_PM_H__
+#define __INCLUDE_SOUND_SOF_PM_H__
+
+#include <sound/sof/header.h>
+
+/*
+ * PM
+ */
+
+/* PM context element */
+struct sof_ipc_pm_ctx_elem {
+ struct sof_ipc_hdr hdr;
+ uint32_t type;
+ uint32_t size;
+ uint64_t addr;
+} __packed;
+
+/*
+ * PM context - SOF_IPC_PM_CTX_SAVE, SOF_IPC_PM_CTX_RESTORE,
+ * SOF_IPC_PM_CTX_SIZE
+ */
+struct sof_ipc_pm_ctx {
+ struct sof_ipc_cmd_hdr hdr;
+ struct sof_ipc_host_buffer buffer;
+ uint32_t num_elems;
+ uint32_t size;
+
+ /* reserved for future use */
+ uint32_t reserved[8];
+
+ struct sof_ipc_pm_ctx_elem elems[];
+} __packed;
+
+/* enable or disable cores - SOF_IPC_PM_CORE_ENABLE */
+struct sof_ipc_pm_core_config {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t enable_mask;
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_STREAM_H__
+#define __INCLUDE_SOUND_SOF_STREAM_H__
+
+#include <sound/sof/header.h>
+
+/*
+ * Stream configuration.
+ */
+
+#define SOF_IPC_MAX_CHANNELS 8
+
+/* common sample rates for use in masks */
+#define SOF_RATE_8000 (1 << 0) /**< 8000Hz */
+#define SOF_RATE_11025 (1 << 1) /**< 11025Hz */
+#define SOF_RATE_12000 (1 << 2) /**< 12000Hz */
+#define SOF_RATE_16000 (1 << 3) /**< 16000Hz */
+#define SOF_RATE_22050 (1 << 4) /**< 22050Hz */
+#define SOF_RATE_24000 (1 << 5) /**< 24000Hz */
+#define SOF_RATE_32000 (1 << 6) /**< 32000Hz */
+#define SOF_RATE_44100 (1 << 7) /**< 44100Hz */
+#define SOF_RATE_48000 (1 << 8) /**< 48000Hz */
+#define SOF_RATE_64000 (1 << 9) /**< 64000Hz */
+#define SOF_RATE_88200 (1 << 10) /**< 88200Hz */
+#define SOF_RATE_96000 (1 << 11) /**< 96000Hz */
+#define SOF_RATE_176400 (1 << 12) /**< 176400Hz */
+#define SOF_RATE_192000 (1 << 13) /**< 192000Hz */
+
+/* continuous and non-standard rates for flexibility */
+#define SOF_RATE_CONTINUOUS (1 << 30) /**< range */
+#define SOF_RATE_KNOT (1 << 31) /**< non-continuous */
+
+/* generic PCM flags for runtime settings */
+#define SOF_PCM_FLAG_XRUN_STOP (1 << 0) /**< Stop on any XRUN */
+
+/* stream PCM frame format */
+enum sof_ipc_frame {
+ SOF_IPC_FRAME_S16_LE = 0,
+ SOF_IPC_FRAME_S24_4LE,
+ SOF_IPC_FRAME_S32_LE,
+ SOF_IPC_FRAME_FLOAT,
+ /* other formats here */
+};
+
+/* stream buffer format */
+enum sof_ipc_buffer_format {
+ SOF_IPC_BUFFER_INTERLEAVED,
+ SOF_IPC_BUFFER_NONINTERLEAVED,
+ /* other formats here */
+};
+
+/* stream direction */
+enum sof_ipc_stream_direction {
+ SOF_IPC_STREAM_PLAYBACK = 0,
+ SOF_IPC_STREAM_CAPTURE,
+};
+
+/* stream ring info */
+struct sof_ipc_host_buffer {
+ struct sof_ipc_hdr hdr;
+ uint32_t phy_addr;
+ uint32_t pages;
+ uint32_t size;
+ uint32_t reserved[3];
+} __packed;
+
+struct sof_ipc_stream_params {
+ struct sof_ipc_hdr hdr;
+ struct sof_ipc_host_buffer buffer;
+ uint32_t direction; /**< enum sof_ipc_stream_direction */
+ uint32_t frame_fmt; /**< enum sof_ipc_frame */
+ uint32_t buffer_fmt; /**< enum sof_ipc_buffer_format */
+ uint32_t rate;
+ uint16_t stream_tag;
+ uint16_t channels;
+ uint16_t sample_valid_bytes;
+ uint16_t sample_container_bytes;
+
+ /* for notifying host period has completed - 0 means no period IRQ */
+ uint32_t host_period_bytes;
+
+ uint32_t reserved[2];
+ uint16_t chmap[SOF_IPC_MAX_CHANNELS]; /**< channel map - SOF_CHMAP_ */
+} __packed;
+
+/* PCM params info - SOF_IPC_STREAM_PCM_PARAMS */
+struct sof_ipc_pcm_params {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t comp_id;
+ uint32_t flags; /**< generic PCM flags - SOF_PCM_FLAG_ */
+ uint32_t reserved[2];
+ struct sof_ipc_stream_params params;
+} __packed;
+
+/* PCM params info reply - SOF_IPC_STREAM_PCM_PARAMS_REPLY */
+struct sof_ipc_pcm_params_reply {
+ struct sof_ipc_reply rhdr;
+ uint32_t comp_id;
+ uint32_t posn_offset;
+} __packed;
+
+/* free stream - SOF_IPC_STREAM_PCM_PARAMS */
+struct sof_ipc_stream {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t comp_id;
+} __packed;
+
+/* flags indicating which time stamps are in sync with each other */
+#define SOF_TIME_HOST_SYNC (1 << 0)
+#define SOF_TIME_DAI_SYNC (1 << 1)
+#define SOF_TIME_WALL_SYNC (1 << 2)
+#define SOF_TIME_STAMP_SYNC (1 << 3)
+
+/* flags indicating which time stamps are valid */
+#define SOF_TIME_HOST_VALID (1 << 8)
+#define SOF_TIME_DAI_VALID (1 << 9)
+#define SOF_TIME_WALL_VALID (1 << 10)
+#define SOF_TIME_STAMP_VALID (1 << 11)
+
+/* flags indicating time stamps are 64bit else 3use low 32bit */
+#define SOF_TIME_HOST_64 (1 << 16)
+#define SOF_TIME_DAI_64 (1 << 17)
+#define SOF_TIME_WALL_64 (1 << 18)
+#define SOF_TIME_STAMP_64 (1 << 19)
+
+struct sof_ipc_stream_posn {
+ struct sof_ipc_reply rhdr;
+ uint32_t comp_id; /**< host component ID */
+ uint32_t flags; /**< SOF_TIME_ */
+ uint32_t wallclock_hz; /**< frequency of wallclock in Hz */
+ uint32_t timestamp_ns; /**< resolution of timestamp in ns */
+ uint64_t host_posn; /**< host DMA position in bytes */
+ uint64_t dai_posn; /**< DAI DMA position in bytes */
+ uint64_t comp_posn; /**< comp position in bytes */
+ uint64_t wallclock; /**< audio wall clock */
+ uint64_t timestamp; /**< system time stamp */
+ uint32_t xrun_comp_id; /**< comp ID of XRUN component */
+ int32_t xrun_size; /**< XRUN size in bytes */
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_TOPOLOGY_H__
+#define __INCLUDE_SOUND_SOF_TOPOLOGY_H__
+
+#include <sound/sof/header.h>
+
+/*
+ * Component
+ */
+
+/* types of component */
+enum sof_comp_type {
+ SOF_COMP_NONE = 0,
+ SOF_COMP_HOST,
+ SOF_COMP_DAI,
+ SOF_COMP_SG_HOST, /**< scatter gather variant */
+ SOF_COMP_SG_DAI, /**< scatter gather variant */
+ SOF_COMP_VOLUME,
+ SOF_COMP_MIXER,
+ SOF_COMP_MUX,
+ SOF_COMP_SRC,
+ SOF_COMP_SPLITTER,
+ SOF_COMP_TONE,
+ SOF_COMP_SWITCH,
+ SOF_COMP_BUFFER,
+ SOF_COMP_EQ_IIR,
+ SOF_COMP_EQ_FIR,
+ SOF_COMP_KEYWORD_DETECT,
+ SOF_COMP_KPB, /* A key phrase buffer component */
+ SOF_COMP_SELECTOR, /**< channel selector component */
+ /* keep FILEREAD/FILEWRITE as the last ones */
+ SOF_COMP_FILEREAD = 10000, /**< host test based file IO */
+ SOF_COMP_FILEWRITE = 10001, /**< host test based file IO */
+};
+
+/* XRUN action for component */
+#define SOF_XRUN_STOP 1 /**< stop stream */
+#define SOF_XRUN_UNDER_ZERO 2 /**< send 0s to sink */
+#define SOF_XRUN_OVER_NULL 4 /**< send data to NULL */
+
+/* create new generic component - SOF_IPC_TPLG_COMP_NEW */
+struct sof_ipc_comp {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t id;
+ enum sof_comp_type type;
+ uint32_t pipeline_id;
+
+ /* reserved for future use */
+ uint32_t reserved[2];
+} __packed;
+
+/*
+ * Component Buffers
+ */
+
+/*
+ * SOF memory capabilities, add new ones at the end
+ */
+#define SOF_MEM_CAPS_RAM (1 << 0)
+#define SOF_MEM_CAPS_ROM (1 << 1)
+#define SOF_MEM_CAPS_EXT (1 << 2) /**< external */
+#define SOF_MEM_CAPS_LP (1 << 3) /**< low power */
+#define SOF_MEM_CAPS_HP (1 << 4) /**< high performance */
+#define SOF_MEM_CAPS_DMA (1 << 5) /**< DMA'able */
+#define SOF_MEM_CAPS_CACHE (1 << 6) /**< cacheable */
+#define SOF_MEM_CAPS_EXEC (1 << 7) /**< executable */
+
+/* create new component buffer - SOF_IPC_TPLG_BUFFER_NEW */
+struct sof_ipc_buffer {
+ struct sof_ipc_comp comp;
+ uint32_t size; /**< buffer size in bytes */
+ uint32_t caps; /**< SOF_MEM_CAPS_ */
+} __packed;
+
+/* generic component config data - must always be after struct sof_ipc_comp */
+struct sof_ipc_comp_config {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t periods_sink; /**< 0 means variable */
+ uint32_t periods_source; /**< 0 means variable */
+ uint32_t reserved1; /**< reserved */
+ uint32_t frame_fmt; /**< SOF_IPC_FRAME_ */
+ uint32_t xrun_action;
+
+ /* reserved for future use */
+ uint32_t reserved[2];
+} __packed;
+
+/* generic host component */
+struct sof_ipc_comp_host {
+ struct sof_ipc_comp comp;
+ struct sof_ipc_comp_config config;
+ uint32_t direction; /**< SOF_IPC_STREAM_ */
+ uint32_t no_irq; /**< don't send periodic IRQ to host/DSP */
+ uint32_t dmac_config; /**< DMA engine specific */
+} __packed;
+
+/* generic DAI component */
+struct sof_ipc_comp_dai {
+ struct sof_ipc_comp comp;
+ struct sof_ipc_comp_config config;
+ uint32_t direction; /**< SOF_IPC_STREAM_ */
+ uint32_t dai_index; /**< index of this type dai */
+ uint32_t type; /**< DAI type - SOF_DAI_ */
+ uint32_t reserved; /**< reserved */
+} __packed;
+
+/* generic mixer component */
+struct sof_ipc_comp_mixer {
+ struct sof_ipc_comp comp;
+ struct sof_ipc_comp_config config;
+} __packed;
+
+/* volume ramping types */
+enum sof_volume_ramp {
+ SOF_VOLUME_LINEAR = 0,
+ SOF_VOLUME_LOG,
+ SOF_VOLUME_LINEAR_ZC,
+ SOF_VOLUME_LOG_ZC,
+};
+
+/* generic volume component */
+struct sof_ipc_comp_volume {
+ struct sof_ipc_comp comp;
+ struct sof_ipc_comp_config config;
+ uint32_t channels;
+ uint32_t min_value;
+ uint32_t max_value;
+ uint32_t ramp; /**< SOF_VOLUME_ */
+ uint32_t initial_ramp; /**< ramp space in ms */
+} __packed;
+
+/* generic SRC component */
+struct sof_ipc_comp_src {
+ struct sof_ipc_comp comp;
+ struct sof_ipc_comp_config config;
+ /* either source or sink rate must be non zero */
+ uint32_t source_rate; /**< source rate or 0 for variable */
+ uint32_t sink_rate; /**< sink rate or 0 for variable */
+ uint32_t rate_mask; /**< SOF_RATE_ supported rates */
+} __packed;
+
+/* generic MUX component */
+struct sof_ipc_comp_mux {
+ struct sof_ipc_comp comp;
+ struct sof_ipc_comp_config config;
+} __packed;
+
+/* generic tone generator component */
+struct sof_ipc_comp_tone {
+ struct sof_ipc_comp comp;
+ struct sof_ipc_comp_config config;
+ int32_t sample_rate;
+ int32_t frequency;
+ int32_t amplitude;
+ int32_t freq_mult;
+ int32_t ampl_mult;
+ int32_t length;
+ int32_t period;
+ int32_t repeats;
+ int32_t ramp_step;
+} __packed;
+
+/** \brief Types of processing components */
+enum sof_ipc_process_type {
+ SOF_PROCESS_NONE = 0, /**< None */
+ SOF_PROCESS_EQFIR, /**< Intel FIR */
+ SOF_PROCESS_EQIIR, /**< Intel IIR */
+ SOF_PROCESS_KEYWORD_DETECT, /**< Keyword Detection */
+ SOF_PROCESS_KPB, /**< KeyPhrase Buffer Manager */
+ SOF_PROCESS_CHAN_SELECTOR, /**< Channel Selector */
+};
+
+/* generic "effect", "codec" or proprietary processing component */
+struct sof_ipc_comp_process {
+ struct sof_ipc_comp comp;
+ struct sof_ipc_comp_config config;
+ uint32_t size; /**< size of bespoke data section in bytes */
+ uint32_t type; /**< sof_ipc_process_type */
+
+ /* reserved for future use */
+ uint32_t reserved[7];
+
+ unsigned char data[0];
+} __packed;
+
+/* frees components, buffers and pipelines
+ * SOF_IPC_TPLG_COMP_FREE, SOF_IPC_TPLG_PIPE_FREE, SOF_IPC_TPLG_BUFFER_FREE
+ */
+struct sof_ipc_free {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t id;
+} __packed;
+
+struct sof_ipc_comp_reply {
+ struct sof_ipc_reply rhdr;
+ uint32_t id;
+ uint32_t offset;
+} __packed;
+
+/*
+ * Pipeline
+ */
+
+/** \brief Types of pipeline scheduling time domains */
+enum sof_ipc_pipe_sched_time_domain {
+ SOF_TIME_DOMAIN_DMA = 0, /**< DMA interrupt */
+ SOF_TIME_DOMAIN_TIMER, /**< Timer interrupt */
+};
+
+/* new pipeline - SOF_IPC_TPLG_PIPE_NEW */
+struct sof_ipc_pipe_new {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t comp_id; /**< component id for pipeline */
+ uint32_t pipeline_id; /**< pipeline id */
+ uint32_t sched_id; /**< Scheduling component id */
+ uint32_t core; /**< core we run on */
+ uint32_t period; /**< execution period in us*/
+ uint32_t priority; /**< priority level 0 (low) to 10 (max) */
+ uint32_t period_mips; /**< worst case instruction count per period */
+ uint32_t frames_per_sched;/**< output frames of pipeline, 0 is variable */
+ uint32_t xrun_limit_usecs; /**< report xruns greater than limit */
+ uint32_t time_domain; /**< scheduling time domain */
+} __packed;
+
+/* pipeline construction complete - SOF_IPC_TPLG_PIPE_COMPLETE */
+struct sof_ipc_pipe_ready {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t comp_id;
+} __packed;
+
+struct sof_ipc_pipe_free {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t comp_id;
+} __packed;
+
+/* connect two components in pipeline - SOF_IPC_TPLG_COMP_CONNECT */
+struct sof_ipc_pipe_comp_connect {
+ struct sof_ipc_cmd_hdr hdr;
+ uint32_t source_id;
+ uint32_t sink_id;
+} __packed;
+
+/* external events */
+enum sof_event_types {
+ SOF_EVENT_NONE = 0,
+ SOF_KEYWORD_DETECT_DAPM_EVENT,
+};
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_TRACE_H__
+#define __INCLUDE_SOUND_SOF_TRACE_H__
+
+#include <sound/sof/header.h>
+#include <sound/sof/stream.h>
+
+/*
+ * DMA for Trace
+ */
+
+#define SOF_TRACE_FILENAME_SIZE 32
+
+/* DMA for Trace params info - SOF_IPC_DEBUG_DMA_PARAMS */
+struct sof_ipc_dma_trace_params {
+ struct sof_ipc_cmd_hdr hdr;
+ struct sof_ipc_host_buffer buffer;
+ uint32_t stream_tag;
+} __packed;
+
+/* DMA for Trace params info - SOF_IPC_DEBUG_DMA_PARAMS */
+struct sof_ipc_dma_trace_posn {
+ struct sof_ipc_reply rhdr;
+ uint32_t host_offset; /* Offset of DMA host buffer */
+ uint32_t overflow; /* overflow bytes if any */
+ uint32_t messages; /* total trace messages */
+} __packed;
+
+/*
+ * Commom debug
+ */
+
+/*
+ * SOF panic codes
+ */
+#define SOF_IPC_PANIC_MAGIC 0x0dead000
+#define SOF_IPC_PANIC_MAGIC_MASK 0x0ffff000
+#define SOF_IPC_PANIC_CODE_MASK 0x00000fff
+#define SOF_IPC_PANIC_MEM (SOF_IPC_PANIC_MAGIC | 0x0)
+#define SOF_IPC_PANIC_WORK (SOF_IPC_PANIC_MAGIC | 0x1)
+#define SOF_IPC_PANIC_IPC (SOF_IPC_PANIC_MAGIC | 0x2)
+#define SOF_IPC_PANIC_ARCH (SOF_IPC_PANIC_MAGIC | 0x3)
+#define SOF_IPC_PANIC_PLATFORM (SOF_IPC_PANIC_MAGIC | 0x4)
+#define SOF_IPC_PANIC_TASK (SOF_IPC_PANIC_MAGIC | 0x5)
+#define SOF_IPC_PANIC_EXCEPTION (SOF_IPC_PANIC_MAGIC | 0x6)
+#define SOF_IPC_PANIC_DEADLOCK (SOF_IPC_PANIC_MAGIC | 0x7)
+#define SOF_IPC_PANIC_STACK (SOF_IPC_PANIC_MAGIC | 0x8)
+#define SOF_IPC_PANIC_IDLE (SOF_IPC_PANIC_MAGIC | 0x9)
+#define SOF_IPC_PANIC_WFI (SOF_IPC_PANIC_MAGIC | 0xa)
+#define SOF_IPC_PANIC_ASSERT (SOF_IPC_PANIC_MAGIC | 0xb)
+
+/* panic info include filename and line number */
+struct sof_ipc_panic_info {
+ struct sof_ipc_hdr hdr;
+ uint32_t code; /* SOF_IPC_PANIC_ */
+ char filename[SOF_TRACE_FILENAME_SIZE];
+ uint32_t linenum;
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_SOUND_SOF_XTENSA_H__
+#define __INCLUDE_SOUND_SOF_XTENSA_H__
+
+#include <sound/sof/header.h>
+
+/*
+ * Architecture specific debug
+ */
+
+/* Xtensa Firmware Oops data */
+struct sof_ipc_dsp_oops_xtensa {
+ struct sof_ipc_hdr hdr;
+ uint32_t exccause;
+ uint32_t excvaddr;
+ uint32_t ps;
+ uint32_t epc1;
+ uint32_t epc2;
+ uint32_t epc3;
+ uint32_t epc4;
+ uint32_t epc5;
+ uint32_t epc6;
+ uint32_t epc7;
+ uint32_t eps2;
+ uint32_t eps3;
+ uint32_t eps4;
+ uint32_t eps5;
+ uint32_t eps6;
+ uint32_t eps7;
+ uint32_t depc;
+ uint32_t intenable;
+ uint32_t interrupt;
+ uint32_t sar;
+ uint32_t stack;
+} __packed;
+
+#endif
TP_ARGS(cgrp, path)
);
+DEFINE_EVENT(cgroup, cgroup_freeze,
+
+ TP_PROTO(struct cgroup *cgrp, const char *path),
+
+ TP_ARGS(cgrp, path)
+);
+
+DEFINE_EVENT(cgroup, cgroup_unfreeze,
+
+ TP_PROTO(struct cgroup *cgrp, const char *path),
+
+ TP_ARGS(cgrp, path)
+);
+
DECLARE_EVENT_CLASS(cgroup_migrate,
TP_PROTO(struct cgroup *dst_cgrp, const char *path,
TP_ARGS(dst_cgrp, path, task, threadgroup)
);
+DECLARE_EVENT_CLASS(cgroup_event,
+
+ TP_PROTO(struct cgroup *cgrp, const char *path, int val),
+
+ TP_ARGS(cgrp, path, val),
+
+ TP_STRUCT__entry(
+ __field( int, root )
+ __field( int, id )
+ __field( int, level )
+ __string( path, path )
+ __field( int, val )
+ ),
+
+ TP_fast_assign(
+ __entry->root = cgrp->root->hierarchy_id;
+ __entry->id = cgrp->id;
+ __entry->level = cgrp->level;
+ __assign_str(path, path);
+ __entry->val = val;
+ ),
+
+ TP_printk("root=%d id=%d level=%d path=%s val=%d",
+ __entry->root, __entry->id, __entry->level, __get_str(path),
+ __entry->val)
+);
+
+DEFINE_EVENT(cgroup_event, cgroup_notify_populated,
+
+ TP_PROTO(struct cgroup *cgrp, const char *path, int val),
+
+ TP_ARGS(cgrp, path, val)
+);
+
+DEFINE_EVENT(cgroup_event, cgroup_notify_frozen,
+
+ TP_PROTO(struct cgroup *cgrp, const char *path, int val),
+
+ TP_ARGS(cgrp, path, val)
+);
+
#endif /* _TRACE_CGROUP_H */
/* This part must be outside protection */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+
+/*
+ * Copyright (c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM ib_mad
+
+#if !defined(_TRACE_IB_MAD_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_IB_MAD_H
+
+#include <linux/tracepoint.h>
+#include <rdma/ib_mad.h>
+
+#ifdef CONFIG_TRACEPOINTS
+struct trace_event_raw_ib_mad_send_template;
+static void create_mad_addr_info(struct ib_mad_send_wr_private *mad_send_wr,
+ struct ib_mad_qp_info *qp_info,
+ struct trace_event_raw_ib_mad_send_template *entry);
+#endif
+
+DECLARE_EVENT_CLASS(ib_mad_send_template,
+ TP_PROTO(struct ib_mad_send_wr_private *wr,
+ struct ib_mad_qp_info *qp_info),
+ TP_ARGS(wr, qp_info),
+
+ TP_STRUCT__entry(
+ __field(u8, base_version)
+ __field(u8, mgmt_class)
+ __field(u8, class_version)
+ __field(u8, port_num)
+ __field(u32, qp_num)
+ __field(u8, method)
+ __field(u8, sl)
+ __field(u16, attr_id)
+ __field(u32, attr_mod)
+ __field(u64, wrtid)
+ __field(u64, tid)
+ __field(u16, status)
+ __field(u16, class_specific)
+ __field(u32, length)
+ __field(u32, dlid)
+ __field(u32, rqpn)
+ __field(u32, rqkey)
+ __field(u32, dev_index)
+ __field(void *, agent_priv)
+ __field(unsigned long, timeout)
+ __field(int, retries_left)
+ __field(int, max_retries)
+ __field(int, retry)
+ __field(u16, pkey)
+ ),
+
+ TP_fast_assign(
+ __entry->dev_index = wr->mad_agent_priv->agent.device->index;
+ __entry->port_num = wr->mad_agent_priv->agent.port_num;
+ __entry->qp_num = wr->mad_agent_priv->qp_info->qp->qp_num;
+ __entry->agent_priv = wr->mad_agent_priv;
+ __entry->wrtid = wr->tid;
+ __entry->max_retries = wr->max_retries;
+ __entry->retries_left = wr->retries_left;
+ __entry->retry = wr->retry;
+ __entry->timeout = wr->timeout;
+ __entry->length = wr->send_buf.hdr_len +
+ wr->send_buf.data_len;
+ __entry->base_version =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->base_version;
+ __entry->mgmt_class =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->mgmt_class;
+ __entry->class_version =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->class_version;
+ __entry->method =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->method;
+ __entry->status =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->status;
+ __entry->class_specific =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->class_specific;
+ __entry->tid = ((struct ib_mad_hdr *)wr->send_buf.mad)->tid;
+ __entry->attr_id =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->attr_id;
+ __entry->attr_mod =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->attr_mod;
+ create_mad_addr_info(wr, qp_info, __entry);
+ ),
+
+ TP_printk("%d:%d QP%d agent %p: " \
+ "wrtid 0x%llx; %d/%d retries(%d); timeout %lu length %d : " \
+ "hdr : base_ver 0x%x class 0x%x class_ver 0x%x " \
+ "method 0x%x status 0x%x class_specific 0x%x tid 0x%llx " \
+ "attr_id 0x%x attr_mod 0x%x => dlid 0x%08x sl %d "\
+ "pkey 0x%x rpqn 0x%x rqpkey 0x%x",
+ __entry->dev_index, __entry->port_num, __entry->qp_num,
+ __entry->agent_priv, be64_to_cpu(__entry->wrtid),
+ __entry->retries_left, __entry->max_retries,
+ __entry->retry, __entry->timeout, __entry->length,
+ __entry->base_version, __entry->mgmt_class,
+ __entry->class_version,
+ __entry->method, be16_to_cpu(__entry->status),
+ be16_to_cpu(__entry->class_specific),
+ be64_to_cpu(__entry->tid), be16_to_cpu(__entry->attr_id),
+ be32_to_cpu(__entry->attr_mod),
+ be32_to_cpu(__entry->dlid), __entry->sl, __entry->pkey,
+ __entry->rqpn, __entry->rqkey
+ )
+);
+
+DEFINE_EVENT(ib_mad_send_template, ib_mad_error_handler,
+ TP_PROTO(struct ib_mad_send_wr_private *wr,
+ struct ib_mad_qp_info *qp_info),
+ TP_ARGS(wr, qp_info));
+DEFINE_EVENT(ib_mad_send_template, ib_mad_ib_send_mad,
+ TP_PROTO(struct ib_mad_send_wr_private *wr,
+ struct ib_mad_qp_info *qp_info),
+ TP_ARGS(wr, qp_info));
+DEFINE_EVENT(ib_mad_send_template, ib_mad_send_done_resend,
+ TP_PROTO(struct ib_mad_send_wr_private *wr,
+ struct ib_mad_qp_info *qp_info),
+ TP_ARGS(wr, qp_info));
+
+TRACE_EVENT(ib_mad_send_done_handler,
+ TP_PROTO(struct ib_mad_send_wr_private *wr, struct ib_wc *wc),
+ TP_ARGS(wr, wc),
+
+ TP_STRUCT__entry(
+ __field(u8, port_num)
+ __field(u8, base_version)
+ __field(u8, mgmt_class)
+ __field(u8, class_version)
+ __field(u32, qp_num)
+ __field(u64, wrtid)
+ __field(u16, status)
+ __field(u16, wc_status)
+ __field(u32, length)
+ __field(void *, agent_priv)
+ __field(unsigned long, timeout)
+ __field(u32, dev_index)
+ __field(int, retries_left)
+ __field(int, max_retries)
+ __field(int, retry)
+ __field(u8, method)
+ ),
+
+ TP_fast_assign(
+ __entry->dev_index = wr->mad_agent_priv->agent.device->index;
+ __entry->port_num = wr->mad_agent_priv->agent.port_num;
+ __entry->qp_num = wr->mad_agent_priv->qp_info->qp->qp_num;
+ __entry->agent_priv = wr->mad_agent_priv;
+ __entry->wrtid = wr->tid;
+ __entry->max_retries = wr->max_retries;
+ __entry->retries_left = wr->retries_left;
+ __entry->retry = wr->retry;
+ __entry->timeout = wr->timeout;
+ __entry->base_version =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->base_version;
+ __entry->mgmt_class =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->mgmt_class;
+ __entry->class_version =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->class_version;
+ __entry->method =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->method;
+ __entry->status =
+ ((struct ib_mad_hdr *)wr->send_buf.mad)->status;
+ __entry->wc_status = wc->status;
+ __entry->length = wc->byte_len;
+ ),
+
+ TP_printk("%d:%d QP%d : SEND WC Status %d : agent %p: " \
+ "wrtid 0x%llx %d/%d retries(%d) timeout %lu length %d: " \
+ "hdr : base_ver 0x%x class 0x%x class_ver 0x%x " \
+ "method 0x%x status 0x%x",
+ __entry->dev_index, __entry->port_num, __entry->qp_num,
+ __entry->wc_status,
+ __entry->agent_priv, be64_to_cpu(__entry->wrtid),
+ __entry->retries_left, __entry->max_retries,
+ __entry->retry, __entry->timeout,
+ __entry->length,
+ __entry->base_version, __entry->mgmt_class,
+ __entry->class_version, __entry->method,
+ be16_to_cpu(__entry->status)
+ )
+);
+
+TRACE_EVENT(ib_mad_recv_done_handler,
+ TP_PROTO(struct ib_mad_qp_info *qp_info, struct ib_wc *wc,
+ struct ib_mad_hdr *mad_hdr),
+ TP_ARGS(qp_info, wc, mad_hdr),
+
+ TP_STRUCT__entry(
+ __field(u8, base_version)
+ __field(u8, mgmt_class)
+ __field(u8, class_version)
+ __field(u8, port_num)
+ __field(u32, qp_num)
+ __field(u16, status)
+ __field(u16, class_specific)
+ __field(u32, length)
+ __field(u64, tid)
+ __field(u8, method)
+ __field(u8, sl)
+ __field(u16, attr_id)
+ __field(u32, attr_mod)
+ __field(u16, src_qp)
+ __field(u16, wc_status)
+ __field(u32, slid)
+ __field(u32, dev_index)
+ __field(u16, pkey)
+ ),
+
+ TP_fast_assign(
+ __entry->dev_index = qp_info->port_priv->device->index;
+ __entry->port_num = qp_info->port_priv->port_num;
+ __entry->qp_num = qp_info->qp->qp_num;
+ __entry->length = wc->byte_len;
+ __entry->base_version = mad_hdr->base_version;
+ __entry->mgmt_class = mad_hdr->mgmt_class;
+ __entry->class_version = mad_hdr->class_version;
+ __entry->method = mad_hdr->method;
+ __entry->status = mad_hdr->status;
+ __entry->class_specific = mad_hdr->class_specific;
+ __entry->tid = mad_hdr->tid;
+ __entry->attr_id = mad_hdr->attr_id;
+ __entry->attr_mod = mad_hdr->attr_mod;
+ __entry->slid = wc->slid;
+ __entry->src_qp = wc->src_qp;
+ __entry->sl = wc->sl;
+ ib_query_pkey(qp_info->port_priv->device,
+ qp_info->port_priv->port_num,
+ wc->pkey_index, &__entry->pkey);
+ __entry->wc_status = wc->status;
+ ),
+
+ TP_printk("%d:%d QP%d : RECV WC Status %d : length %d : hdr : " \
+ "base_ver 0x%02x class 0x%02x class_ver 0x%02x " \
+ "method 0x%02x status 0x%04x class_specific 0x%04x " \
+ "tid 0x%016llx attr_id 0x%04x attr_mod 0x%08x " \
+ "slid 0x%08x src QP%d, sl %d pkey 0x%04x",
+ __entry->dev_index, __entry->port_num, __entry->qp_num,
+ __entry->wc_status,
+ __entry->length,
+ __entry->base_version, __entry->mgmt_class,
+ __entry->class_version, __entry->method,
+ be16_to_cpu(__entry->status),
+ be16_to_cpu(__entry->class_specific),
+ be64_to_cpu(__entry->tid), be16_to_cpu(__entry->attr_id),
+ be32_to_cpu(__entry->attr_mod),
+ __entry->slid, __entry->src_qp, __entry->sl, __entry->pkey
+ )
+);
+
+DECLARE_EVENT_CLASS(ib_mad_agent_template,
+ TP_PROTO(struct ib_mad_agent_private *agent),
+ TP_ARGS(agent),
+
+ TP_STRUCT__entry(
+ __field(u32, dev_index)
+ __field(u32, hi_tid)
+ __field(u8, port_num)
+ __field(u8, mgmt_class)
+ __field(u8, mgmt_class_version)
+ ),
+
+ TP_fast_assign(
+ __entry->dev_index = agent->agent.device->index;
+ __entry->port_num = agent->agent.port_num;
+ __entry->hi_tid = agent->agent.hi_tid;
+
+ if (agent->reg_req) {
+ __entry->mgmt_class = agent->reg_req->mgmt_class;
+ __entry->mgmt_class_version =
+ agent->reg_req->mgmt_class_version;
+ } else {
+ __entry->mgmt_class = 0;
+ __entry->mgmt_class_version = 0;
+ }
+ ),
+
+ TP_printk("%d:%d mad agent : hi_tid 0x%08x class 0x%02x class_ver 0x%02x",
+ __entry->dev_index, __entry->port_num,
+ __entry->hi_tid, __entry->mgmt_class,
+ __entry->mgmt_class_version
+ )
+);
+DEFINE_EVENT(ib_mad_agent_template, ib_mad_recv_done_agent,
+ TP_PROTO(struct ib_mad_agent_private *agent),
+ TP_ARGS(agent));
+DEFINE_EVENT(ib_mad_agent_template, ib_mad_send_done_agent,
+ TP_PROTO(struct ib_mad_agent_private *agent),
+ TP_ARGS(agent));
+DEFINE_EVENT(ib_mad_agent_template, ib_mad_create_agent,
+ TP_PROTO(struct ib_mad_agent_private *agent),
+ TP_ARGS(agent));
+DEFINE_EVENT(ib_mad_agent_template, ib_mad_unregister_agent,
+ TP_PROTO(struct ib_mad_agent_private *agent),
+ TP_ARGS(agent));
+
+
+
+DECLARE_EVENT_CLASS(ib_mad_opa_smi_template,
+ TP_PROTO(struct opa_smp *smp),
+ TP_ARGS(smp),
+
+ TP_STRUCT__entry(
+ __field(u64, mkey)
+ __field(u32, dr_slid)
+ __field(u32, dr_dlid)
+ __field(u8, hop_ptr)
+ __field(u8, hop_cnt)
+ __array(u8, initial_path, OPA_SMP_MAX_PATH_HOPS)
+ __array(u8, return_path, OPA_SMP_MAX_PATH_HOPS)
+ ),
+
+ TP_fast_assign(
+ __entry->hop_ptr = smp->hop_ptr;
+ __entry->hop_cnt = smp->hop_cnt;
+ __entry->mkey = smp->mkey;
+ __entry->dr_slid = smp->route.dr.dr_slid;
+ __entry->dr_dlid = smp->route.dr.dr_dlid;
+ memcpy(__entry->initial_path, smp->route.dr.initial_path,
+ OPA_SMP_MAX_PATH_HOPS);
+ memcpy(__entry->return_path, smp->route.dr.return_path,
+ OPA_SMP_MAX_PATH_HOPS);
+ ),
+
+ TP_printk("OPA SMP: hop_ptr %d hop_cnt %d " \
+ "mkey 0x%016llx dr_slid 0x%08x dr_dlid 0x%08x " \
+ "initial_path %*ph return_path %*ph ",
+ __entry->hop_ptr, __entry->hop_cnt,
+ be64_to_cpu(__entry->mkey), be32_to_cpu(__entry->dr_slid),
+ be32_to_cpu(__entry->dr_dlid),
+ OPA_SMP_MAX_PATH_HOPS, __entry->initial_path,
+ OPA_SMP_MAX_PATH_HOPS, __entry->return_path
+ )
+);
+
+DEFINE_EVENT(ib_mad_opa_smi_template, ib_mad_handle_opa_smi,
+ TP_PROTO(struct opa_smp *smp),
+ TP_ARGS(smp));
+DEFINE_EVENT(ib_mad_opa_smi_template, ib_mad_handle_out_opa_smi,
+ TP_PROTO(struct opa_smp *smp),
+ TP_ARGS(smp));
+
+
+DECLARE_EVENT_CLASS(ib_mad_opa_ib_template,
+ TP_PROTO(struct ib_smp *smp),
+ TP_ARGS(smp),
+
+ TP_STRUCT__entry(
+ __field(u64, mkey)
+ __field(u32, dr_slid)
+ __field(u32, dr_dlid)
+ __field(u8, hop_ptr)
+ __field(u8, hop_cnt)
+ __array(u8, initial_path, IB_SMP_MAX_PATH_HOPS)
+ __array(u8, return_path, IB_SMP_MAX_PATH_HOPS)
+ ),
+
+ TP_fast_assign(
+ __entry->hop_ptr = smp->hop_ptr;
+ __entry->hop_cnt = smp->hop_cnt;
+ __entry->mkey = smp->mkey;
+ __entry->dr_slid = smp->dr_slid;
+ __entry->dr_dlid = smp->dr_dlid;
+ memcpy(__entry->initial_path, smp->initial_path,
+ IB_SMP_MAX_PATH_HOPS);
+ memcpy(__entry->return_path, smp->return_path,
+ IB_SMP_MAX_PATH_HOPS);
+ ),
+
+ TP_printk("OPA SMP: hop_ptr %d hop_cnt %d " \
+ "mkey 0x%016llx dr_slid 0x%04x dr_dlid 0x%04x " \
+ "initial_path %*ph return_path %*ph ",
+ __entry->hop_ptr, __entry->hop_cnt,
+ be64_to_cpu(__entry->mkey), be16_to_cpu(__entry->dr_slid),
+ be16_to_cpu(__entry->dr_dlid),
+ IB_SMP_MAX_PATH_HOPS, __entry->initial_path,
+ IB_SMP_MAX_PATH_HOPS, __entry->return_path
+ )
+);
+
+DEFINE_EVENT(ib_mad_opa_ib_template, ib_mad_handle_ib_smi,
+ TP_PROTO(struct ib_smp *smp),
+ TP_ARGS(smp));
+DEFINE_EVENT(ib_mad_opa_ib_template, ib_mad_handle_out_ib_smi,
+ TP_PROTO(struct ib_smp *smp),
+ TP_ARGS(smp));
+
+#endif /* _TRACE_IB_MAD_H */
+
+#include <trace/define_trace.h>
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+
+/*
+ * Copyright (c) 2018 Intel Corporation. All rights reserved.
+ *
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM ib_umad
+
+#if !defined(_TRACE_IB_UMAD_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_IB_UMAD_H
+
+#include <linux/tracepoint.h>
+
+DECLARE_EVENT_CLASS(ib_umad_template,
+ TP_PROTO(struct ib_umad_file *file, struct ib_user_mad_hdr *umad_hdr,
+ struct ib_mad_hdr *mad_hdr),
+ TP_ARGS(file, umad_hdr, mad_hdr),
+
+ TP_STRUCT__entry(
+ __field(u8, port_num)
+ __field(u8, sl)
+ __field(u8, path_bits)
+ __field(u8, grh_present)
+ __field(u32, id)
+ __field(u32, status)
+ __field(u32, timeout_ms)
+ __field(u32, retires)
+ __field(u32, length)
+ __field(u32, qpn)
+ __field(u32, qkey)
+ __field(u8, gid_index)
+ __field(u8, hop_limit)
+ __field(u16, lid)
+ __field(u16, attr_id)
+ __field(u16, pkey_index)
+ __field(u8, base_version)
+ __field(u8, mgmt_class)
+ __field(u8, class_version)
+ __field(u8, method)
+ __field(u32, flow_label)
+ __field(u16, mad_status)
+ __field(u16, class_specific)
+ __field(u32, attr_mod)
+ __field(u64, tid)
+ __array(u8, gid, 16)
+ __field(u32, dev_index)
+ __field(u8, traffic_class)
+ ),
+
+ TP_fast_assign(
+ __entry->dev_index = file->port->ib_dev->index;
+ __entry->port_num = file->port->port_num;
+
+ __entry->id = umad_hdr->id;
+ __entry->status = umad_hdr->status;
+ __entry->timeout_ms = umad_hdr->timeout_ms;
+ __entry->retires = umad_hdr->retries;
+ __entry->length = umad_hdr->length;
+ __entry->qpn = umad_hdr->qpn;
+ __entry->qkey = umad_hdr->qkey;
+ __entry->lid = umad_hdr->lid;
+ __entry->sl = umad_hdr->sl;
+ __entry->path_bits = umad_hdr->path_bits;
+ __entry->grh_present = umad_hdr->grh_present;
+ __entry->gid_index = umad_hdr->gid_index;
+ __entry->hop_limit = umad_hdr->hop_limit;
+ __entry->traffic_class = umad_hdr->traffic_class;
+ memcpy(__entry->gid, umad_hdr->gid, sizeof(umad_hdr->gid));
+ __entry->flow_label = umad_hdr->flow_label;
+ __entry->pkey_index = umad_hdr->pkey_index;
+
+ __entry->base_version = mad_hdr->base_version;
+ __entry->mgmt_class = mad_hdr->mgmt_class;
+ __entry->class_version = mad_hdr->class_version;
+ __entry->method = mad_hdr->method;
+ __entry->mad_status = mad_hdr->status;
+ __entry->class_specific = mad_hdr->class_specific;
+ __entry->tid = mad_hdr->tid;
+ __entry->attr_id = mad_hdr->attr_id;
+ __entry->attr_mod = mad_hdr->attr_mod;
+ ),
+
+ TP_printk("%d:%d umad_hdr: id 0x%08x status 0x%08x ms %u ret %u " \
+ "len %u QP%u qkey 0x%08x lid 0x%04x sl %u path_bits 0x%x " \
+ "grh 0x%x gidi %u hop_lim %u traf_cl %u gid %pI6c " \
+ "flow 0x%08x pkeyi %u MAD: base_ver 0x%x class 0x%x " \
+ "class_ver 0x%x method 0x%x status 0x%04x " \
+ "class_specific 0x%04x tid 0x%016llx attr_id 0x%04x " \
+ "attr_mod 0x%08x ",
+ __entry->dev_index, __entry->port_num,
+ __entry->id, __entry->status, __entry->timeout_ms,
+ __entry->retires, __entry->length, be32_to_cpu(__entry->qpn),
+ be32_to_cpu(__entry->qkey), be16_to_cpu(__entry->lid),
+ __entry->sl, __entry->path_bits, __entry->grh_present,
+ __entry->gid_index, __entry->hop_limit,
+ __entry->traffic_class, &__entry->gid,
+ be32_to_cpu(__entry->flow_label), __entry->pkey_index,
+ __entry->base_version, __entry->mgmt_class,
+ __entry->class_version, __entry->method,
+ be16_to_cpu(__entry->mad_status),
+ be16_to_cpu(__entry->class_specific),
+ be64_to_cpu(__entry->tid), be16_to_cpu(__entry->attr_id),
+ be32_to_cpu(__entry->attr_mod)
+ )
+);
+
+DEFINE_EVENT(ib_umad_template, ib_umad_write,
+ TP_PROTO(struct ib_umad_file *file, struct ib_user_mad_hdr *umad_hdr,
+ struct ib_mad_hdr *mad_hdr),
+ TP_ARGS(file, umad_hdr, mad_hdr));
+
+DEFINE_EVENT(ib_umad_template, ib_umad_read_recv,
+ TP_PROTO(struct ib_umad_file *file, struct ib_user_mad_hdr *umad_hdr,
+ struct ib_mad_hdr *mad_hdr),
+ TP_ARGS(file, umad_hdr, mad_hdr));
+
+DEFINE_EVENT(ib_umad_template, ib_umad_read_send,
+ TP_PROTO(struct ib_umad_file *file, struct ib_user_mad_hdr *umad_hdr,
+ struct ib_mad_hdr *mad_hdr),
+ TP_ARGS(file, umad_hdr, mad_hdr));
+
+#endif /* _TRACE_IB_UMAD_H */
+
+#include <trace/define_trace.h>
)
);
+TRACE_EVENT(xprtrdma_marshal_failed,
+ TP_PROTO(const struct rpc_rqst *rqst,
+ int ret
+ ),
+
+ TP_ARGS(rqst, ret),
+
+ TP_STRUCT__entry(
+ __field(unsigned int, task_id)
+ __field(unsigned int, client_id)
+ __field(u32, xid)
+ __field(int, ret)
+ ),
+
+ TP_fast_assign(
+ __entry->task_id = rqst->rq_task->tk_pid;
+ __entry->client_id = rqst->rq_task->tk_client->cl_clid;
+ __entry->xid = be32_to_cpu(rqst->rq_xid);
+ __entry->ret = ret;
+ ),
+
+ TP_printk("task:%u@%u xid=0x%08x: ret=%d",
+ __entry->task_id, __entry->client_id, __entry->xid,
+ __entry->ret
+ )
+);
+
TRACE_EVENT(xprtrdma_post_send,
TP_PROTO(
const struct rpcrdma_req *req,
TRACE_DEFINE_ENUM(RPC_CALL_MAJORSEEN);
TRACE_DEFINE_ENUM(RPC_TASK_ROOTCREDS);
TRACE_DEFINE_ENUM(RPC_TASK_DYNAMIC);
-TRACE_DEFINE_ENUM(RPC_TASK_KILLED);
TRACE_DEFINE_ENUM(RPC_TASK_SOFT);
TRACE_DEFINE_ENUM(RPC_TASK_SOFTCONN);
TRACE_DEFINE_ENUM(RPC_TASK_SENT);
{ RPC_CALL_MAJORSEEN, "MAJORSEEN" }, \
{ RPC_TASK_ROOTCREDS, "ROOTCREDS" }, \
{ RPC_TASK_DYNAMIC, "DYNAMIC" }, \
- { RPC_TASK_KILLED, "KILLED" }, \
{ RPC_TASK_SOFT, "SOFT" }, \
{ RPC_TASK_SOFTCONN, "SOFTCONN" }, \
{ RPC_TASK_SENT, "SENT" }, \
TRACE_DEFINE_ENUM(RPC_TASK_NEED_XMIT);
TRACE_DEFINE_ENUM(RPC_TASK_NEED_RECV);
TRACE_DEFINE_ENUM(RPC_TASK_MSG_PIN_WAIT);
+TRACE_DEFINE_ENUM(RPC_TASK_SIGNALLED);
#define rpc_show_runstate(flags) \
__print_flags(flags, "|", \
{ (1UL << RPC_TASK_ACTIVE), "ACTIVE" }, \
{ (1UL << RPC_TASK_NEED_XMIT), "NEED_XMIT" }, \
{ (1UL << RPC_TASK_NEED_RECV), "NEED_RECV" }, \
- { (1UL << RPC_TASK_MSG_PIN_WAIT), "MSG_PIN_WAIT" })
+ { (1UL << RPC_TASK_MSG_PIN_WAIT), "MSG_PIN_WAIT" }, \
+ { (1UL << RPC_TASK_SIGNALLED), "SIGNALLED" })
DECLARE_EVENT_CLASS(rpc_task_running,
__entry->client_id = task->tk_client ?
task->tk_client->cl_clid : -1;
__entry->task_id = task->tk_pid;
- __entry->timeout = task->tk_timeout;
+ __entry->timeout = rpc_task_timeout(task);
__entry->runstate = task->tk_runstate;
__entry->status = task->tk_status;
__entry->flags = task->tk_flags;
#define AMDGPU_CTX_QUERY2_FLAGS_VRAMLOST (1<<1)
/* indicate some job from this context once cause gpu hang */
#define AMDGPU_CTX_QUERY2_FLAGS_GUILTY (1<<2)
+/* indicate some errors are detected by RAS */
+#define AMDGPU_CTX_QUERY2_FLAGS_RAS_CE (1<<3)
+#define AMDGPU_CTX_QUERY2_FLAGS_RAS_UE (1<<4)
/* Context priority level */
#define AMDGPU_CTX_PRIORITY_UNSET -2048
#define AMDGPU_CHUNK_ID_SYNCOBJ_OUT 0x05
#define AMDGPU_CHUNK_ID_BO_HANDLES 0x06
#define AMDGPU_CHUNK_ID_SCHEDULED_DEPENDENCIES 0x07
+#define AMDGPU_CHUNK_ID_SYNCOBJ_TIMELINE_WAIT 0x08
+#define AMDGPU_CHUNK_ID_SYNCOBJ_TIMELINE_SIGNAL 0x09
struct drm_amdgpu_cs_chunk {
__u32 chunk_id;
__u32 handle;
};
+struct drm_amdgpu_cs_chunk_syncobj {
+ __u32 handle;
+ __u32 flags;
+ __u64 point;
+};
+
#define AMDGPU_FENCE_TO_HANDLE_GET_SYNCOBJ 0
#define AMDGPU_FENCE_TO_HANDLE_GET_SYNCOBJ_FD 1
#define AMDGPU_FENCE_TO_HANDLE_GET_SYNC_FILE_FD 2
#define AMDGPU_INFO_FW_GFX_RLC_RESTORE_LIST_SRM_MEM 0x11
/* Subquery id: Query DMCU firmware version */
#define AMDGPU_INFO_FW_DMCU 0x12
+ #define AMDGPU_INFO_FW_TA 0x13
/* number of bytes moved for TTM migration */
#define AMDGPU_INFO_NUM_BYTES_MOVED 0x0f
/* the used VRAM size */
/* Number of VRAM page faults on CPU access. */
#define AMDGPU_INFO_NUM_VRAM_CPU_PAGE_FAULTS 0x1E
#define AMDGPU_INFO_VRAM_LOST_COUNTER 0x1F
+/* query ras mask of enabled features*/
+#define AMDGPU_INFO_RAS_ENABLED_FEATURES 0x20
+
+/* RAS MASK: UMC (VRAM) */
+#define AMDGPU_INFO_RAS_ENABLED_UMC (1 << 0)
+/* RAS MASK: SDMA */
+#define AMDGPU_INFO_RAS_ENABLED_SDMA (1 << 1)
+/* RAS MASK: GFX */
+#define AMDGPU_INFO_RAS_ENABLED_GFX (1 << 2)
+/* RAS MASK: MMHUB */
+#define AMDGPU_INFO_RAS_ENABLED_MMHUB (1 << 3)
+/* RAS MASK: ATHUB */
+#define AMDGPU_INFO_RAS_ENABLED_ATHUB (1 << 4)
+/* RAS MASK: PCIE */
+#define AMDGPU_INFO_RAS_ENABLED_PCIE (1 << 5)
+/* RAS MASK: HDP */
+#define AMDGPU_INFO_RAS_ENABLED_HDP (1 << 6)
+/* RAS MASK: XGMI */
+#define AMDGPU_INFO_RAS_ENABLED_XGMI (1 << 7)
+/* RAS MASK: DF */
+#define AMDGPU_INFO_RAS_ENABLED_DF (1 << 8)
+/* RAS MASK: SMN */
+#define AMDGPU_INFO_RAS_ENABLED_SMN (1 << 9)
+/* RAS MASK: SEM */
+#define AMDGPU_INFO_RAS_ENABLED_SEM (1 << 10)
+/* RAS MASK: MP0 */
+#define AMDGPU_INFO_RAS_ENABLED_MP0 (1 << 11)
+/* RAS MASK: MP1 */
+#define AMDGPU_INFO_RAS_ENABLED_MP1 (1 << 12)
+/* RAS MASK: FUSE */
+#define AMDGPU_INFO_RAS_ENABLED_FUSE (1 << 13)
#define AMDGPU_INFO_MMR_SE_INDEX_SHIFT 0
#define AMDGPU_INFO_MMR_SE_INDEX_MASK 0xff
#define DRM_CAP_PAGE_FLIP_TARGET 0x11
#define DRM_CAP_CRTC_IN_VBLANK_EVENT 0x12
#define DRM_CAP_SYNCOBJ 0x13
+#define DRM_CAP_SYNCOBJ_TIMELINE 0x14
/** DRM_IOCTL_GET_CAP ioctl argument type */
struct drm_get_cap {
__u32 pad;
};
+struct drm_syncobj_transfer {
+ __u32 src_handle;
+ __u32 dst_handle;
+ __u64 src_point;
+ __u64 dst_point;
+ __u32 flags;
+ __u32 pad;
+};
+
#define DRM_SYNCOBJ_WAIT_FLAGS_WAIT_ALL (1 << 0)
#define DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT (1 << 1)
+#define DRM_SYNCOBJ_WAIT_FLAGS_WAIT_AVAILABLE (1 << 2) /* wait for time point to become available */
struct drm_syncobj_wait {
__u64 handles;
/* absolute timeout */
__u32 pad;
};
+struct drm_syncobj_timeline_wait {
+ __u64 handles;
+ /* wait on specific timeline point for every handles*/
+ __u64 points;
+ /* absolute timeout */
+ __s64 timeout_nsec;
+ __u32 count_handles;
+ __u32 flags;
+ __u32 first_signaled; /* only valid when not waiting all */
+ __u32 pad;
+};
+
+
struct drm_syncobj_array {
__u64 handles;
__u32 count_handles;
__u32 pad;
};
+struct drm_syncobj_timeline_array {
+ __u64 handles;
+ __u64 points;
+ __u32 count_handles;
+ __u32 pad;
+};
+
+
/* Query current scanout sequence number */
struct drm_crtc_get_sequence {
__u32 crtc_id; /* requested crtc_id */
#define DRM_IOCTL_MODE_GET_LEASE DRM_IOWR(0xC8, struct drm_mode_get_lease)
#define DRM_IOCTL_MODE_REVOKE_LEASE DRM_IOWR(0xC9, struct drm_mode_revoke_lease)
+#define DRM_IOCTL_SYNCOBJ_TIMELINE_WAIT DRM_IOWR(0xCA, struct drm_syncobj_timeline_wait)
+#define DRM_IOCTL_SYNCOBJ_QUERY DRM_IOWR(0xCB, struct drm_syncobj_timeline_array)
+#define DRM_IOCTL_SYNCOBJ_TRANSFER DRM_IOWR(0xCC, struct drm_syncobj_transfer)
+#define DRM_IOCTL_SYNCOBJ_TIMELINE_SIGNAL DRM_IOWR(0xCD, struct drm_syncobj_timeline_array)
+
/**
* Device specific ioctls should only be in their respective headers
* The device specific ioctl range is from 0x40 to 0x9f.
#define DRM_FORMAT_RGBA1010102 fourcc_code('R', 'A', '3', '0') /* [31:0] R:G:B:A 10:10:10:2 little endian */
#define DRM_FORMAT_BGRA1010102 fourcc_code('B', 'A', '3', '0') /* [31:0] B:G:R:A 10:10:10:2 little endian */
+/*
+ * Floating point 64bpp RGB
+ * IEEE 754-2008 binary16 half-precision float
+ * [15:0] sign:exponent:mantissa 1:5:10
+ */
+#define DRM_FORMAT_XRGB16161616F fourcc_code('X', 'R', '4', 'H') /* [63:0] x:R:G:B 16:16:16:16 little endian */
+#define DRM_FORMAT_XBGR16161616F fourcc_code('X', 'B', '4', 'H') /* [63:0] x:B:G:R 16:16:16:16 little endian */
+
+#define DRM_FORMAT_ARGB16161616F fourcc_code('A', 'R', '4', 'H') /* [63:0] A:R:G:B 16:16:16:16 little endian */
+#define DRM_FORMAT_ABGR16161616F fourcc_code('A', 'B', '4', 'H') /* [63:0] A:B:G:R 16:16:16:16 little endian */
+
/* packed YCbCr */
#define DRM_FORMAT_YUYV fourcc_code('Y', 'U', 'Y', 'V') /* [31:0] Cr0:Y1:Cb0:Y0 8:8:8:8 little endian */
#define DRM_FORMAT_YVYU fourcc_code('Y', 'V', 'Y', 'U') /* [31:0] Cb0:Y1:Cr0:Y0 8:8:8:8 little endian */
#define DRM_FORMAT_VYUY fourcc_code('V', 'Y', 'U', 'Y') /* [31:0] Y1:Cb0:Y0:Cr0 8:8:8:8 little endian */
#define DRM_FORMAT_AYUV fourcc_code('A', 'Y', 'U', 'V') /* [31:0] A:Y:Cb:Cr 8:8:8:8 little endian */
-#define DRM_FORMAT_XYUV8888 fourcc_code('X', 'Y', 'U', 'V') /* [31:0] X:Y:Cb:Cr 8:8:8:8 little endian */
+#define DRM_FORMAT_XYUV8888 fourcc_code('X', 'Y', 'U', 'V') /* [31:0] X:Y:Cb:Cr 8:8:8:8 little endian */
+#define DRM_FORMAT_VUY888 fourcc_code('V', 'U', '2', '4') /* [23:0] Cr:Cb:Y 8:8:8 little endian */
+#define DRM_FORMAT_VUY101010 fourcc_code('V', 'U', '3', '0') /* Y followed by U then V, 10:10:10. Non-linear modifier only */
+
+/*
+ * packed Y2xx indicate for each component, xx valid data occupy msb
+ * 16-xx padding occupy lsb
+ */
+#define DRM_FORMAT_Y210 fourcc_code('Y', '2', '1', '0') /* [63:0] Cr0:0:Y1:0:Cb0:0:Y0:0 10:6:10:6:10:6:10:6 little endian per 2 Y pixels */
+#define DRM_FORMAT_Y212 fourcc_code('Y', '2', '1', '2') /* [63:0] Cr0:0:Y1:0:Cb0:0:Y0:0 12:4:12:4:12:4:12:4 little endian per 2 Y pixels */
+#define DRM_FORMAT_Y216 fourcc_code('Y', '2', '1', '6') /* [63:0] Cr0:Y1:Cb0:Y0 16:16:16:16 little endian per 2 Y pixels */
+
+/*
+ * packed Y4xx indicate for each component, xx valid data occupy msb
+ * 16-xx padding occupy lsb except Y410
+ */
+#define DRM_FORMAT_Y410 fourcc_code('Y', '4', '1', '0') /* [31:0] A:Cr:Y:Cb 2:10:10:10 little endian */
+#define DRM_FORMAT_Y412 fourcc_code('Y', '4', '1', '2') /* [63:0] A:0:Cr:0:Y:0:Cb:0 12:4:12:4:12:4:12:4 little endian */
+#define DRM_FORMAT_Y416 fourcc_code('Y', '4', '1', '6') /* [63:0] A:Cr:Y:Cb 16:16:16:16 little endian */
+
+#define DRM_FORMAT_XVYU2101010 fourcc_code('X', 'V', '3', '0') /* [31:0] X:Cr:Y:Cb 2:10:10:10 little endian */
+#define DRM_FORMAT_XVYU12_16161616 fourcc_code('X', 'V', '3', '6') /* [63:0] X:0:Cr:0:Y:0:Cb:0 12:4:12:4:12:4:12:4 little endian */
+#define DRM_FORMAT_XVYU16161616 fourcc_code('X', 'V', '4', '8') /* [63:0] X:Cr:Y:Cb 16:16:16:16 little endian */
/*
* packed YCbCr420 2x2 tiled formats
/* [63:0] X3:X2:Y3:Cr0:Y2:X1:X0:Y1:Cb0:Y0 1:1:10:10:10:1:1:10:10:10 little endian */
#define DRM_FORMAT_X0L2 fourcc_code('X', '0', 'L', '2')
+/*
+ * 1-plane YUV 4:2:0
+ * In these formats, the component ordering is specified (Y, followed by U
+ * then V), but the exact Linear layout is undefined.
+ * These formats can only be used with a non-Linear modifier.
+ */
+#define DRM_FORMAT_YUV420_8BIT fourcc_code('Y', 'U', '0', '8')
+#define DRM_FORMAT_YUV420_10BIT fourcc_code('Y', 'U', '1', '0')
+
/*
* 2 plane RGB + A
* index 0 = RGB plane, same format as the corresponding non _A8 format has
#define DRM_FORMAT_NV24 fourcc_code('N', 'V', '2', '4') /* non-subsampled Cr:Cb plane */
#define DRM_FORMAT_NV42 fourcc_code('N', 'V', '4', '2') /* non-subsampled Cb:Cr plane */
+/*
+ * 2 plane YCbCr MSB aligned
+ * index 0 = Y plane, [15:0] Y:x [10:6] little endian
+ * index 1 = Cr:Cb plane, [31:0] Cr:x:Cb:x [10:6:10:6] little endian
+ */
+#define DRM_FORMAT_P210 fourcc_code('P', '2', '1', '0') /* 2x1 subsampled Cr:Cb plane, 10 bit per channel */
+
/*
* 2 plane YCbCr MSB aligned
* index 0 = Y plane, [15:0] Y:x [10:6] little endian
extern "C" {
#endif
-#define DRM_DISPLAY_INFO_LEN 32
#define DRM_CONNECTOR_NAME_LEN 32
#define DRM_DISPLAY_MODE_LEN 32
#define DRM_PROP_NAME_LEN 32
struct drm_color_lut {
/*
- * Data is U0.16 fixed point format.
+ * Values are mapped linearly to 0.0 - 1.0 range, with 0x0 == 0.0 and
+ * 0xffff == 1.0.
*/
__u16 red;
__u16 green;
#define I915_ERROR_UEVENT "ERROR"
#define I915_RESET_UEVENT "RESET"
+/*
+ * i915_user_extension: Base class for defining a chain of extensions
+ *
+ * Many interfaces need to grow over time. In most cases we can simply
+ * extend the struct and have userspace pass in more data. Another option,
+ * as demonstrated by Vulkan's approach to providing extensions for forward
+ * and backward compatibility, is to use a list of optional structs to
+ * provide those extra details.
+ *
+ * The key advantage to using an extension chain is that it allows us to
+ * redefine the interface more easily than an ever growing struct of
+ * increasing complexity, and for large parts of that interface to be
+ * entirely optional. The downside is more pointer chasing; chasing across
+ * the __user boundary with pointers encapsulated inside u64.
+ */
+struct i915_user_extension {
+ __u64 next_extension;
+ __u32 name;
+ __u32 flags; /* All undefined bits must be zero. */
+ __u32 rsvd[4]; /* Reserved for future use; must be zero. */
+};
+
/*
* MOCS indexes used for GPU surfaces, defining the cacheability of the
* surface data and the coherency for this data wrt. CPU vs. GPU accesses.
I915_ENGINE_CLASS_VIDEO = 2,
I915_ENGINE_CLASS_VIDEO_ENHANCE = 3,
+ /* should be kept compact */
+
I915_ENGINE_CLASS_INVALID = -1
};
+/*
+ * There may be more than one engine fulfilling any role within the system.
+ * Each engine of a class is given a unique instance number and therefore
+ * any engine can be specified by its class:instance tuplet. APIs that allow
+ * access to any engine in the system will use struct i915_engine_class_instance
+ * for this identification.
+ */
+struct i915_engine_class_instance {
+ __u16 engine_class; /* see enum drm_i915_gem_engine_class */
+ __u16 engine_instance;
+};
+
/**
* DOC: perf_events exposed by i915 through /sys/bus/event_sources/drivers/i915
*
#define DRM_I915_PERF_ADD_CONFIG 0x37
#define DRM_I915_PERF_REMOVE_CONFIG 0x38
#define DRM_I915_QUERY 0x39
+/* Must be kept compact -- no holes */
#define DRM_IOCTL_I915_INIT DRM_IOW( DRM_COMMAND_BASE + DRM_I915_INIT, drm_i915_init_t)
#define DRM_IOCTL_I915_FLUSH DRM_IO ( DRM_COMMAND_BASE + DRM_I915_FLUSH)
#define DRM_IOCTL_I915_GET_SPRITE_COLORKEY DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GET_SPRITE_COLORKEY, struct drm_intel_sprite_colorkey)
#define DRM_IOCTL_I915_GEM_WAIT DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT, struct drm_i915_gem_wait)
#define DRM_IOCTL_I915_GEM_CONTEXT_CREATE DRM_IOWR (DRM_COMMAND_BASE + DRM_I915_GEM_CONTEXT_CREATE, struct drm_i915_gem_context_create)
+#define DRM_IOCTL_I915_GEM_CONTEXT_CREATE_EXT DRM_IOWR (DRM_COMMAND_BASE + DRM_I915_GEM_CONTEXT_CREATE, struct drm_i915_gem_context_create_ext)
#define DRM_IOCTL_I915_GEM_CONTEXT_DESTROY DRM_IOW (DRM_COMMAND_BASE + DRM_I915_GEM_CONTEXT_DESTROY, struct drm_i915_gem_context_destroy)
#define DRM_IOCTL_I915_REG_READ DRM_IOWR (DRM_COMMAND_BASE + DRM_I915_REG_READ, struct drm_i915_reg_read)
#define DRM_IOCTL_I915_GET_RESET_STATS DRM_IOWR (DRM_COMMAND_BASE + DRM_I915_GET_RESET_STATS, struct drm_i915_reset_stats)
#define I915_SCHEDULER_CAP_ENABLED (1ul << 0)
#define I915_SCHEDULER_CAP_PRIORITY (1ul << 1)
#define I915_SCHEDULER_CAP_PREEMPTION (1ul << 2)
+#define I915_SCHEDULER_CAP_SEMAPHORES (1ul << 3)
#define I915_PARAM_HUC_STATUS 42
*/
#define I915_PARAM_MMAP_GTT_COHERENT 52
+/* Must be kept compact -- no holes and well documented */
+
typedef struct drm_i915_getparam {
__s32 param;
/*
#define I915_SETPARAM_TEX_LRU_LOG_GRANULARITY 2
#define I915_SETPARAM_ALLOW_BATCHBUFFER 3
#define I915_SETPARAM_NUM_USED_FENCES 4
+/* Must be kept compact -- no holes */
typedef struct drm_i915_setparam {
int param;
* struct drm_i915_gem_exec_fence *fences.
*/
__u64 cliprects_ptr;
-#define I915_EXEC_RING_MASK (7<<0)
+#define I915_EXEC_RING_MASK (0x3f)
#define I915_EXEC_DEFAULT (0<<0)
#define I915_EXEC_RENDER (1<<0)
#define I915_EXEC_BSD (2<<0)
* as busy may become idle before the ioctl is completed.
*
* Furthermore, if the object is busy, which engine is busy is only
- * provided as a guide. There are race conditions which prevent the
- * report of which engines are busy from being always accurate.
- * However, the converse is not true. If the object is idle, the
- * result of the ioctl, that all engines are idle, is accurate.
+ * provided as a guide and only indirectly by reporting its class
+ * (there may be more than one engine in each class). There are race
+ * conditions which prevent the report of which engines are busy from
+ * being always accurate. However, the converse is not true. If the
+ * object is idle, the result of the ioctl, that all engines are idle,
+ * is accurate.
*
* The returned dword is split into two fields to indicate both
- * the engines on which the object is being read, and the
- * engine on which it is currently being written (if any).
+ * the engine classess on which the object is being read, and the
+ * engine class on which it is currently being written (if any).
*
* The low word (bits 0:15) indicate if the object is being written
* to by any engine (there can only be one, as the GEM implicit
* synchronisation rules force writes to be serialised). Only the
- * engine for the last write is reported.
+ * engine class (offset by 1, I915_ENGINE_CLASS_RENDER is reported as
+ * 1 not 0 etc) for the last write is reported.
*
- * The high word (bits 16:31) are a bitmask of which engines are
- * currently reading from the object. Multiple engines may be
+ * The high word (bits 16:31) are a bitmask of which engines classes
+ * are currently reading from the object. Multiple engines may be
* reading from the object simultaneously.
*
- * The value of each engine is the same as specified in the
- * EXECBUFFER2 ioctl, i.e. I915_EXEC_RENDER, I915_EXEC_BSD etc.
- * Note I915_EXEC_DEFAULT is a symbolic value and is mapped to
- * the I915_EXEC_RENDER engine for execution, and so it is never
+ * The value of each engine class is the same as specified in the
+ * I915_CONTEXT_SET_ENGINES parameter and via perf, i.e.
+ * I915_ENGINE_CLASS_RENDER, I915_ENGINE_CLASS_COPY, etc.
* reported as active itself. Some hardware may have parallel
* execution engines, e.g. multiple media engines, which are
- * mapped to the same identifier in the EXECBUFFER2 ioctl and
- * so are not separately reported for busyness.
+ * mapped to the same class identifier and so are not separately
+ * reported for busyness.
*
* Caveat emptor:
* Only the boolean result of this query is reliable; that is whether
};
struct drm_i915_gem_context_create {
- /* output: id of new context*/
- __u32 ctx_id;
- __u32 pad;
-};
-
-struct drm_i915_gem_context_destroy {
- __u32 ctx_id;
+ __u32 ctx_id; /* output: id of new context*/
__u32 pad;
};
-struct drm_i915_reg_read {
- /*
- * Register offset.
- * For 64bit wide registers where the upper 32bits don't immediately
- * follow the lower 32bits, the offset of the lower 32bits must
- * be specified
- */
- __u64 offset;
-#define I915_REG_READ_8B_WA (1ul << 0)
-
- __u64 val; /* Return value */
-};
-/* Known registers:
- *
- * Render engine timestamp - 0x2358 + 64bit - gen7+
- * - Note this register returns an invalid value if using the default
- * single instruction 8byte read, in order to workaround that pass
- * flag I915_REG_READ_8B_WA in offset field.
- *
- */
-
-struct drm_i915_reset_stats {
- __u32 ctx_id;
- __u32 flags;
-
- /* All resets since boot/module reload, for all contexts */
- __u32 reset_count;
-
- /* Number of batches lost when active in GPU, for this context */
- __u32 batch_active;
-
- /* Number of batches lost pending for execution, for this context */
- __u32 batch_pending;
-
- __u32 pad;
-};
-
-struct drm_i915_gem_userptr {
- __u64 user_ptr;
- __u64 user_size;
+struct drm_i915_gem_context_create_ext {
+ __u32 ctx_id; /* output: id of new context*/
__u32 flags;
-#define I915_USERPTR_READ_ONLY 0x1
-#define I915_USERPTR_UNSYNCHRONIZED 0x80000000
- /**
- * Returned handle for the object.
- *
- * Object handles are nonzero.
- */
- __u32 handle;
+#define I915_CONTEXT_CREATE_FLAGS_USE_EXTENSIONS (1u << 0)
+#define I915_CONTEXT_CREATE_FLAGS_UNKNOWN \
+ (-(I915_CONTEXT_CREATE_FLAGS_USE_EXTENSIONS << 1))
+ __u64 extensions;
};
struct drm_i915_gem_context_param {
* drm_i915_gem_context_param_sseu.
*/
#define I915_CONTEXT_PARAM_SSEU 0x7
+
+/*
+ * Not all clients may want to attempt automatic recover of a context after
+ * a hang (for example, some clients may only submit very small incremental
+ * batches relying on known logical state of previous batches which will never
+ * recover correctly and each attempt will hang), and so would prefer that
+ * the context is forever banned instead.
+ *
+ * If set to false (0), after a reset, subsequent (and in flight) rendering
+ * from this context is discarded, and the client will need to create a new
+ * context to use instead.
+ *
+ * If set to true (1), the kernel will automatically attempt to recover the
+ * context by skipping the hanging batch and executing the next batch starting
+ * from the default context state (discarding the incomplete logical context
+ * state lost due to the reset).
+ *
+ * On creation, all new contexts are marked as recoverable.
+ */
+#define I915_CONTEXT_PARAM_RECOVERABLE 0x8
+/* Must be kept compact -- no holes and well documented */
+
__u64 value;
};
/*
* Engine class & instance to be configured or queried.
*/
- __u16 engine_class;
- __u16 engine_instance;
+ struct i915_engine_class_instance engine;
/*
* Unused for now. Must be cleared to zero.
__u32 rsvd;
};
+struct drm_i915_gem_context_create_ext_setparam {
+#define I915_CONTEXT_CREATE_EXT_SETPARAM 0
+ struct i915_user_extension base;
+ struct drm_i915_gem_context_param param;
+};
+
+struct drm_i915_gem_context_destroy {
+ __u32 ctx_id;
+ __u32 pad;
+};
+
+/*
+ * DRM_I915_GEM_VM_CREATE -
+ *
+ * Create a new virtual memory address space (ppGTT) for use within a context
+ * on the same file. Extensions can be provided to configure exactly how the
+ * address space is setup upon creation.
+ *
+ * The id of new VM (bound to the fd) for use with I915_CONTEXT_PARAM_VM is
+ * returned in the outparam @id.
+ *
+ * No flags are defined, with all bits reserved and must be zero.
+ *
+ * An extension chain maybe provided, starting with @extensions, and terminated
+ * by the @next_extension being 0. Currently, no extensions are defined.
+ *
+ * DRM_I915_GEM_VM_DESTROY -
+ *
+ * Destroys a previously created VM id, specified in @id.
+ *
+ * No extensions or flags are allowed currently, and so must be zero.
+ */
+struct drm_i915_gem_vm_control {
+ __u64 extensions;
+ __u32 flags;
+ __u32 vm_id;
+};
+
+struct drm_i915_reg_read {
+ /*
+ * Register offset.
+ * For 64bit wide registers where the upper 32bits don't immediately
+ * follow the lower 32bits, the offset of the lower 32bits must
+ * be specified
+ */
+ __u64 offset;
+#define I915_REG_READ_8B_WA (1ul << 0)
+
+ __u64 val; /* Return value */
+};
+
+/* Known registers:
+ *
+ * Render engine timestamp - 0x2358 + 64bit - gen7+
+ * - Note this register returns an invalid value if using the default
+ * single instruction 8byte read, in order to workaround that pass
+ * flag I915_REG_READ_8B_WA in offset field.
+ *
+ */
+
+struct drm_i915_reset_stats {
+ __u32 ctx_id;
+ __u32 flags;
+
+ /* All resets since boot/module reload, for all contexts */
+ __u32 reset_count;
+
+ /* Number of batches lost when active in GPU, for this context */
+ __u32 batch_active;
+
+ /* Number of batches lost pending for execution, for this context */
+ __u32 batch_pending;
+
+ __u32 pad;
+};
+
+struct drm_i915_gem_userptr {
+ __u64 user_ptr;
+ __u64 user_size;
+ __u32 flags;
+#define I915_USERPTR_READ_ONLY 0x1
+#define I915_USERPTR_UNSYNCHRONIZED 0x80000000
+ /**
+ * Returned handle for the object.
+ *
+ * Object handles are nonzero.
+ */
+ __u32 handle;
+};
+
enum drm_i915_oa_format {
I915_OA_FORMAT_A13 = 1, /* HSW only */
I915_OA_FORMAT_A29, /* HSW only */
struct drm_i915_query_item {
__u64 query_id;
#define DRM_I915_QUERY_TOPOLOGY_INFO 1
+/* Must be kept compact -- no holes and well documented */
/*
* When set to zero by userspace, this is filled with the size of the
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0 WITH Linux-syscall-note) OR MIT */
+/* Copyright 2017-2018 Qiang Yu <yuq825@gmail.com> */
+
+#ifndef __LIMA_DRM_H__
+#define __LIMA_DRM_H__
+
+#include "drm.h"
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+enum drm_lima_param_gpu_id {
+ DRM_LIMA_PARAM_GPU_ID_UNKNOWN,
+ DRM_LIMA_PARAM_GPU_ID_MALI400,
+ DRM_LIMA_PARAM_GPU_ID_MALI450,
+};
+
+enum drm_lima_param {
+ DRM_LIMA_PARAM_GPU_ID,
+ DRM_LIMA_PARAM_NUM_PP,
+ DRM_LIMA_PARAM_GP_VERSION,
+ DRM_LIMA_PARAM_PP_VERSION,
+};
+
+/**
+ * get various information of the GPU
+ */
+struct drm_lima_get_param {
+ __u32 param; /* in, value in enum drm_lima_param */
+ __u32 pad; /* pad, must be zero */
+ __u64 value; /* out, parameter value */
+};
+
+/**
+ * create a buffer for used by GPU
+ */
+struct drm_lima_gem_create {
+ __u32 size; /* in, buffer size */
+ __u32 flags; /* in, currently no flags, must be zero */
+ __u32 handle; /* out, GEM buffer handle */
+ __u32 pad; /* pad, must be zero */
+};
+
+/**
+ * get information of a buffer
+ */
+struct drm_lima_gem_info {
+ __u32 handle; /* in, GEM buffer handle */
+ __u32 va; /* out, virtual address mapped into GPU MMU */
+ __u64 offset; /* out, used to mmap this buffer to CPU */
+};
+
+#define LIMA_SUBMIT_BO_READ 0x01
+#define LIMA_SUBMIT_BO_WRITE 0x02
+
+/* buffer information used by one task */
+struct drm_lima_gem_submit_bo {
+ __u32 handle; /* in, GEM buffer handle */
+ __u32 flags; /* in, buffer read/write by GPU */
+};
+
+#define LIMA_GP_FRAME_REG_NUM 6
+
+/* frame used to setup GP for each task */
+struct drm_lima_gp_frame {
+ __u32 frame[LIMA_GP_FRAME_REG_NUM];
+};
+
+#define LIMA_PP_FRAME_REG_NUM 23
+#define LIMA_PP_WB_REG_NUM 12
+
+/* frame used to setup mali400 GPU PP for each task */
+struct drm_lima_m400_pp_frame {
+ __u32 frame[LIMA_PP_FRAME_REG_NUM];
+ __u32 num_pp;
+ __u32 wb[3 * LIMA_PP_WB_REG_NUM];
+ __u32 plbu_array_address[4];
+ __u32 fragment_stack_address[4];
+};
+
+/* frame used to setup mali450 GPU PP for each task */
+struct drm_lima_m450_pp_frame {
+ __u32 frame[LIMA_PP_FRAME_REG_NUM];
+ __u32 num_pp;
+ __u32 wb[3 * LIMA_PP_WB_REG_NUM];
+ __u32 use_dlbu;
+ __u32 _pad;
+ union {
+ __u32 plbu_array_address[8];
+ __u32 dlbu_regs[4];
+ };
+ __u32 fragment_stack_address[8];
+};
+
+#define LIMA_PIPE_GP 0x00
+#define LIMA_PIPE_PP 0x01
+
+#define LIMA_SUBMIT_FLAG_EXPLICIT_FENCE (1 << 0)
+
+/**
+ * submit a task to GPU
+ *
+ * User can always merge multi sync_file and drm_syncobj
+ * into one drm_syncobj as in_sync[0], but we reserve
+ * in_sync[1] for another task's out_sync to avoid the
+ * export/import/merge pass when explicit sync.
+ */
+struct drm_lima_gem_submit {
+ __u32 ctx; /* in, context handle task is submitted to */
+ __u32 pipe; /* in, which pipe to use, GP/PP */
+ __u32 nr_bos; /* in, array length of bos field */
+ __u32 frame_size; /* in, size of frame field */
+ __u64 bos; /* in, array of drm_lima_gem_submit_bo */
+ __u64 frame; /* in, GP/PP frame */
+ __u32 flags; /* in, submit flags */
+ __u32 out_sync; /* in, drm_syncobj handle used to wait task finish after submission */
+ __u32 in_sync[2]; /* in, drm_syncobj handle used to wait before start this task */
+};
+
+#define LIMA_GEM_WAIT_READ 0x01
+#define LIMA_GEM_WAIT_WRITE 0x02
+
+/**
+ * wait pending GPU task finish of a buffer
+ */
+struct drm_lima_gem_wait {
+ __u32 handle; /* in, GEM buffer handle */
+ __u32 op; /* in, CPU want to read/write this buffer */
+ __s64 timeout_ns; /* in, wait timeout in absulute time */
+};
+
+/**
+ * create a context
+ */
+struct drm_lima_ctx_create {
+ __u32 id; /* out, context handle */
+ __u32 _pad; /* pad, must be zero */
+};
+
+/**
+ * free a context
+ */
+struct drm_lima_ctx_free {
+ __u32 id; /* in, context handle */
+ __u32 _pad; /* pad, must be zero */
+};
+
+#define DRM_LIMA_GET_PARAM 0x00
+#define DRM_LIMA_GEM_CREATE 0x01
+#define DRM_LIMA_GEM_INFO 0x02
+#define DRM_LIMA_GEM_SUBMIT 0x03
+#define DRM_LIMA_GEM_WAIT 0x04
+#define DRM_LIMA_CTX_CREATE 0x05
+#define DRM_LIMA_CTX_FREE 0x06
+
+#define DRM_IOCTL_LIMA_GET_PARAM DRM_IOWR(DRM_COMMAND_BASE + DRM_LIMA_GET_PARAM, struct drm_lima_get_param)
+#define DRM_IOCTL_LIMA_GEM_CREATE DRM_IOWR(DRM_COMMAND_BASE + DRM_LIMA_GEM_CREATE, struct drm_lima_gem_create)
+#define DRM_IOCTL_LIMA_GEM_INFO DRM_IOWR(DRM_COMMAND_BASE + DRM_LIMA_GEM_INFO, struct drm_lima_gem_info)
+#define DRM_IOCTL_LIMA_GEM_SUBMIT DRM_IOW(DRM_COMMAND_BASE + DRM_LIMA_GEM_SUBMIT, struct drm_lima_gem_submit)
+#define DRM_IOCTL_LIMA_GEM_WAIT DRM_IOW(DRM_COMMAND_BASE + DRM_LIMA_GEM_WAIT, struct drm_lima_gem_wait)
+#define DRM_IOCTL_LIMA_CTX_CREATE DRM_IOR(DRM_COMMAND_BASE + DRM_LIMA_CTX_CREATE, struct drm_lima_ctx_create)
+#define DRM_IOCTL_LIMA_CTX_FREE DRM_IOW(DRM_COMMAND_BASE + DRM_LIMA_CTX_FREE, struct drm_lima_ctx_free)
+
+#if defined(__cplusplus)
+}
+#endif
+
+#endif /* __LIMA_DRM_H__ */
#define MSM_PARAM_TIMESTAMP 0x05
#define MSM_PARAM_GMEM_BASE 0x06
#define MSM_PARAM_NR_RINGS 0x07
+#define MSM_PARAM_PP_PGTABLE 0x08 /* => 1 for per-process pagetables, else 0 */
+#define MSM_PARAM_FAULTS 0x09
struct drm_msm_param {
__u32 pipe; /* in, MSM_PIPE_x */
__u32 id; /* out, identifier */
};
+#define MSM_SUBMITQUEUE_PARAM_FAULTS 0
+
+struct drm_msm_submitqueue_query {
+ __u64 data;
+ __u32 id;
+ __u32 param;
+ __u32 len;
+ __u32 pad;
+};
+
#define DRM_MSM_GET_PARAM 0x00
/* placeholder:
#define DRM_MSM_SET_PARAM 0x01
*/
#define DRM_MSM_SUBMITQUEUE_NEW 0x0A
#define DRM_MSM_SUBMITQUEUE_CLOSE 0x0B
+#define DRM_MSM_SUBMITQUEUE_QUERY 0x0C
#define DRM_IOCTL_MSM_GET_PARAM DRM_IOWR(DRM_COMMAND_BASE + DRM_MSM_GET_PARAM, struct drm_msm_param)
#define DRM_IOCTL_MSM_GEM_NEW DRM_IOWR(DRM_COMMAND_BASE + DRM_MSM_GEM_NEW, struct drm_msm_gem_new)
#define DRM_IOCTL_MSM_GEM_MADVISE DRM_IOWR(DRM_COMMAND_BASE + DRM_MSM_GEM_MADVISE, struct drm_msm_gem_madvise)
#define DRM_IOCTL_MSM_SUBMITQUEUE_NEW DRM_IOWR(DRM_COMMAND_BASE + DRM_MSM_SUBMITQUEUE_NEW, struct drm_msm_submitqueue)
#define DRM_IOCTL_MSM_SUBMITQUEUE_CLOSE DRM_IOW (DRM_COMMAND_BASE + DRM_MSM_SUBMITQUEUE_CLOSE, __u32)
+#define DRM_IOCTL_MSM_SUBMITQUEUE_QUERY DRM_IOW (DRM_COMMAND_BASE + DRM_MSM_SUBMITQUEUE_QUERY, struct drm_msm_submitqueue_query)
#if defined(__cplusplus)
}
--- /dev/null
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2014-2018 Broadcom
+ * Copyright © 2019 Collabora ltd.
+ */
+#ifndef _PANFROST_DRM_H_
+#define _PANFROST_DRM_H_
+
+#include "drm.h"
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+#define DRM_PANFROST_SUBMIT 0x00
+#define DRM_PANFROST_WAIT_BO 0x01
+#define DRM_PANFROST_CREATE_BO 0x02
+#define DRM_PANFROST_MMAP_BO 0x03
+#define DRM_PANFROST_GET_PARAM 0x04
+#define DRM_PANFROST_GET_BO_OFFSET 0x05
+
+#define DRM_IOCTL_PANFROST_SUBMIT DRM_IOW(DRM_COMMAND_BASE + DRM_PANFROST_SUBMIT, struct drm_panfrost_submit)
+#define DRM_IOCTL_PANFROST_WAIT_BO DRM_IOW(DRM_COMMAND_BASE + DRM_PANFROST_WAIT_BO, struct drm_panfrost_wait_bo)
+#define DRM_IOCTL_PANFROST_CREATE_BO DRM_IOWR(DRM_COMMAND_BASE + DRM_PANFROST_CREATE_BO, struct drm_panfrost_create_bo)
+#define DRM_IOCTL_PANFROST_MMAP_BO DRM_IOWR(DRM_COMMAND_BASE + DRM_PANFROST_MMAP_BO, struct drm_panfrost_mmap_bo)
+#define DRM_IOCTL_PANFROST_GET_PARAM DRM_IOWR(DRM_COMMAND_BASE + DRM_PANFROST_GET_PARAM, struct drm_panfrost_get_param)
+#define DRM_IOCTL_PANFROST_GET_BO_OFFSET DRM_IOWR(DRM_COMMAND_BASE + DRM_PANFROST_GET_BO_OFFSET, struct drm_panfrost_get_bo_offset)
+
+#define PANFROST_JD_REQ_FS (1 << 0)
+/**
+ * struct drm_panfrost_submit - ioctl argument for submitting commands to the 3D
+ * engine.
+ *
+ * This asks the kernel to have the GPU execute a render command list.
+ */
+struct drm_panfrost_submit {
+
+ /** Address to GPU mapping of job descriptor */
+ __u64 jc;
+
+ /** An optional array of sync objects to wait on before starting this job. */
+ __u64 in_syncs;
+
+ /** Number of sync objects to wait on before starting this job. */
+ __u32 in_sync_count;
+
+ /** An optional sync object to place the completion fence in. */
+ __u32 out_sync;
+
+ /** Pointer to a u32 array of the BOs that are referenced by the job. */
+ __u64 bo_handles;
+
+ /** Number of BO handles passed in (size is that times 4). */
+ __u32 bo_handle_count;
+
+ /** A combination of PANFROST_JD_REQ_* */
+ __u32 requirements;
+};
+
+/**
+ * struct drm_panfrost_wait_bo - ioctl argument for waiting for
+ * completion of the last DRM_PANFROST_SUBMIT on a BO.
+ *
+ * This is useful for cases where multiple processes might be
+ * rendering to a BO and you want to wait for all rendering to be
+ * completed.
+ */
+struct drm_panfrost_wait_bo {
+ __u32 handle;
+ __u32 pad;
+ __s64 timeout_ns; /* absolute */
+};
+
+/**
+ * struct drm_panfrost_create_bo - ioctl argument for creating Panfrost BOs.
+ *
+ * There are currently no values for the flags argument, but it may be
+ * used in a future extension.
+ */
+struct drm_panfrost_create_bo {
+ __u32 size;
+ __u32 flags;
+ /** Returned GEM handle for the BO. */
+ __u32 handle;
+ /* Pad, must be zero-filled. */
+ __u32 pad;
+ /**
+ * Returned offset for the BO in the GPU address space. This offset
+ * is private to the DRM fd and is valid for the lifetime of the GEM
+ * handle.
+ *
+ * This offset value will always be nonzero, since various HW
+ * units treat 0 specially.
+ */
+ __u64 offset;
+};
+
+/**
+ * struct drm_panfrost_mmap_bo - ioctl argument for mapping Panfrost BOs.
+ *
+ * This doesn't actually perform an mmap. Instead, it returns the
+ * offset you need to use in an mmap on the DRM device node. This
+ * means that tools like valgrind end up knowing about the mapped
+ * memory.
+ *
+ * There are currently no values for the flags argument, but it may be
+ * used in a future extension.
+ */
+struct drm_panfrost_mmap_bo {
+ /** Handle for the object being mapped. */
+ __u32 handle;
+ __u32 flags;
+ /** offset into the drm node to use for subsequent mmap call. */
+ __u64 offset;
+};
+
+enum drm_panfrost_param {
+ DRM_PANFROST_PARAM_GPU_PROD_ID,
+};
+
+struct drm_panfrost_get_param {
+ __u32 param;
+ __u32 pad;
+ __u64 value;
+};
+
+/**
+ * Returns the offset for the BO in the GPU address space for this DRM fd.
+ * This is the same value returned by drm_panfrost_create_bo, if that was called
+ * from this DRM fd.
+ */
+struct drm_panfrost_get_bo_offset {
+ __u32 handle;
+ __u32 pad;
+ __u64 offset;
+};
+
+#if defined(__cplusplus)
+}
+#endif
+
+#endif /* _PANFROST_DRM_H_ */
#define KFD_HW_EXCEPTION_GPU_HANG 0
#define KFD_HW_EXCEPTION_ECC 1
+/* For kfd_hsa_memory_exception_data.ErrorType */
+#define KFD_MEM_ERR_NO_RAS 0
+#define KFD_MEM_ERR_SRAM_ECC 1
+#define KFD_MEM_ERR_POISON_CONSUMED 2
+#define KFD_MEM_ERR_GPU_HANG 3
struct kfd_ioctl_create_event_args {
__u64 event_page_offset; /* from KFD */
struct kfd_memory_exception_failure failure;
__u64 va;
__u32 gpu_id;
- __u32 pad;
+ __u32 ErrorType; /* 0 = no RAS error,
+ * 1 = ECC_SRAM,
+ * 2 = Link_SYNFLOOD (poison),
+ * 3 = GPU hang (not attributable to a specific cause),
+ * other values reserved
+ */
};
/* hw exception data */
#define NFS_MOUNT_UNSHARED 0x8000 /* 5 */
#define NFS_MOUNT_FLAGMASK 0xFFFF
-/* The following are for internal use only */
-#define NFS_MOUNT_LOOKUP_CACHE_NONEG 0x10000
-#define NFS_MOUNT_LOOKUP_CACHE_NONE 0x20000
-#define NFS_MOUNT_NORESVPORT 0x40000
-#define NFS_MOUNT_LEGACY_INTERFACE 0x80000
-
-#define NFS_MOUNT_LOCAL_FLOCK 0x100000
-#define NFS_MOUNT_LOCAL_FCNTL 0x200000
-
#endif
#include <linux/types.h>
-#define VIRTIO_GPU_F_VIRGL 0
-#define VIRTIO_GPU_F_EDID 1
+/*
+ * VIRTIO_GPU_CMD_CTX_*
+ * VIRTIO_GPU_CMD_*_3D
+ */
+#define VIRTIO_GPU_F_VIRGL 0
+
+/*
+ * VIRTIO_GPU_CMD_GET_EDID
+ */
+#define VIRTIO_GPU_F_EDID 1
enum virtio_gpu_ctrl_type {
VIRTIO_GPU_UNDEFINED = 0,
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) */
+/*
+ * Copyright 2018-2019 Amazon.com, Inc. or its affiliates. All rights reserved.
+ */
+
+#ifndef EFA_ABI_USER_H
+#define EFA_ABI_USER_H
+
+#include <linux/types.h>
+
+/*
+ * Increment this value if any changes that break userspace ABI
+ * compatibility are made.
+ */
+#define EFA_UVERBS_ABI_VERSION 1
+
+/*
+ * Keep structs aligned to 8 bytes.
+ * Keep reserved fields as arrays of __u8 named reserved_XXX where XXX is the
+ * hex bit offset of the field.
+ */
+
+enum efa_ibv_user_cmds_supp_udata {
+ EFA_USER_CMDS_SUPP_UDATA_QUERY_DEVICE = 1 << 0,
+ EFA_USER_CMDS_SUPP_UDATA_CREATE_AH = 1 << 1,
+};
+
+struct efa_ibv_alloc_ucontext_resp {
+ __u32 comp_mask;
+ __u32 cmds_supp_udata_mask;
+ __u16 sub_cqs_per_cq;
+ __u16 inline_buf_size;
+ __u32 max_llq_size; /* bytes */
+};
+
+struct efa_ibv_alloc_pd_resp {
+ __u32 comp_mask;
+ __u16 pdn;
+ __u8 reserved_30[2];
+};
+
+struct efa_ibv_create_cq {
+ __u32 comp_mask;
+ __u32 cq_entry_size;
+ __u16 num_sub_cqs;
+ __u8 reserved_50[6];
+};
+
+struct efa_ibv_create_cq_resp {
+ __u32 comp_mask;
+ __u8 reserved_20[4];
+ __aligned_u64 q_mmap_key;
+ __aligned_u64 q_mmap_size;
+ __u16 cq_idx;
+ __u8 reserved_d0[6];
+};
+
+enum {
+ EFA_QP_DRIVER_TYPE_SRD = 0,
+};
+
+struct efa_ibv_create_qp {
+ __u32 comp_mask;
+ __u32 rq_ring_size; /* bytes */
+ __u32 sq_ring_size; /* bytes */
+ __u32 driver_qp_type;
+};
+
+struct efa_ibv_create_qp_resp {
+ __u32 comp_mask;
+ /* the offset inside the page of the rq db */
+ __u32 rq_db_offset;
+ /* the offset inside the page of the sq db */
+ __u32 sq_db_offset;
+ /* the offset inside the page of descriptors buffer */
+ __u32 llq_desc_offset;
+ __aligned_u64 rq_mmap_key;
+ __aligned_u64 rq_mmap_size;
+ __aligned_u64 rq_db_mmap_key;
+ __aligned_u64 sq_db_mmap_key;
+ __aligned_u64 llq_desc_mmap_key;
+ __u16 send_sub_cq_idx;
+ __u16 recv_sub_cq_idx;
+ __u8 reserved_1e0[4];
+};
+
+struct efa_ibv_create_ah_resp {
+ __u32 comp_mask;
+ __u16 efa_address_handle;
+ __u8 reserved_30[2];
+};
+
+struct efa_ibv_ex_query_device_resp {
+ __u32 comp_mask;
+ __u32 max_sq_wr;
+ __u32 max_rq_wr;
+ __u16 max_sq_sge;
+ __u16 max_rq_sge;
+};
+
+#endif /* EFA_ABI_USER_H */
MLX5_IB_CREATE_QP_RESP_MASK_TISN = 1UL << 1,
MLX5_IB_CREATE_QP_RESP_MASK_RQN = 1UL << 2,
MLX5_IB_CREATE_QP_RESP_MASK_SQN = 1UL << 3,
+ MLX5_IB_CREATE_QP_RESP_MASK_TIR_ICM_ADDR = 1UL << 4,
};
struct mlx5_ib_create_qp_resp {
__u32 rqn;
__u32 sqn;
__u32 reserved1;
+ __u64 tir_icm_addr;
};
struct mlx5_ib_alloc_mw {
enum mlx5_ib_alloc_dm_attrs {
MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET = (1U << UVERBS_ID_NS_SHIFT),
MLX5_IB_ATTR_ALLOC_DM_RESP_PAGE_INDEX,
+ MLX5_IB_ATTR_ALLOC_DM_REQ_TYPE,
};
enum mlx5_ib_devx_methods {
MLX5_IB_ATTR_FLOW_MATCHER_FLOW_TYPE,
MLX5_IB_ATTR_FLOW_MATCHER_MATCH_CRITERIA,
MLX5_IB_ATTR_FLOW_MATCHER_FLOW_FLAGS,
+ MLX5_IB_ATTR_FLOW_MATCHER_FT_TYPE,
};
enum mlx5_ib_flow_matcher_destroy_attrs {
enum mlx5_ib_uapi_flow_table_type {
MLX5_IB_UAPI_FLOW_TABLE_TYPE_NIC_RX = 0x0,
MLX5_IB_UAPI_FLOW_TABLE_TYPE_NIC_TX = 0x1,
+ MLX5_IB_UAPI_FLOW_TABLE_TYPE_FDB = 0x2,
};
enum mlx5_ib_uapi_flow_action_packet_reformat_type {
__u8 out_data[];
};
+enum mlx5_ib_uapi_dm_type {
+ MLX5_IB_UAPI_DM_TYPE_MEMIC,
+ MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM,
+ MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM,
+};
+
#endif
RDMA_NL_IWPM_NUM_OPS
};
-struct rdma_cm_id_stats {
- __u32 qp_num;
- __u32 bound_dev_if;
- __u32 port_space;
- __s32 pid;
- __u8 cm_state;
- __u8 node_type;
- __u8 port_num;
- __u8 qp_type;
-};
-
enum {
IWPM_NLA_REG_PID_UNSPEC = 0,
IWPM_NLA_REG_PID_SEQ,
RDMA_NLDEV_CMD_PORT_GET, /* can dump */
- /* 6 - 8 are free to use */
+ RDMA_NLDEV_CMD_SYS_GET, /* can dump */
+ RDMA_NLDEV_CMD_SYS_SET,
+
+ /* 8 is free to use */
RDMA_NLDEV_CMD_RES_GET = 9, /* can dump */
*/
RDMA_NLDEV_ATTR_LINK_TYPE, /* string */
+ /*
+ * net namespace mode for rdma subsystem:
+ * either shared or exclusive among multiple net namespaces.
+ */
+ RDMA_NLDEV_SYS_ATTR_NETNS_MODE, /* u8 */
+ /*
+ * Device protocol, e.g. ib, iw, usnic, roce and opa
+ */
+ RDMA_NLDEV_ATTR_DEV_PROTOCOL, /* string */
+
+ /*
+ * File descriptor handle of the net namespace object
+ */
+ RDMA_NLDEV_NET_NS_FD, /* u32 */
+
/*
* Always the end
*/
RDMA_DRIVER_RXE,
RDMA_DRIVER_HFI1,
RDMA_DRIVER_QIB,
+ RDMA_DRIVER_EFA,
};
#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+/**
+ * SOF ABI versioning is based on Semantic Versioning where we have a given
+ * MAJOR.MINOR.PATCH version number. See https://semver.org/
+ *
+ * Rules for incrementing or changing version :-
+ *
+ * 1) Increment MAJOR version if you make incompatible API changes. MINOR and
+ * PATCH should be reset to 0.
+ *
+ * 2) Increment MINOR version if you add backwards compatible features or
+ * changes. PATCH should be reset to 0.
+ *
+ * 3) Increment PATCH version if you add backwards compatible bug fixes.
+ */
+
+#ifndef __INCLUDE_UAPI_SOUND_SOF_ABI_H__
+#define __INCLUDE_UAPI_SOUND_SOF_ABI_H__
+
+/* SOF ABI version major, minor and patch numbers */
+#define SOF_ABI_MAJOR 3
+#define SOF_ABI_MINOR 4
+#define SOF_ABI_PATCH 0
+
+/* SOF ABI version number. Format within 32bit word is MMmmmppp */
+#define SOF_ABI_MAJOR_SHIFT 24
+#define SOF_ABI_MAJOR_MASK 0xff
+#define SOF_ABI_MINOR_SHIFT 12
+#define SOF_ABI_MINOR_MASK 0xfff
+#define SOF_ABI_PATCH_SHIFT 0
+#define SOF_ABI_PATCH_MASK 0xfff
+
+#define SOF_ABI_VER(major, minor, patch) \
+ (((major) << SOF_ABI_MAJOR_SHIFT) | \
+ ((minor) << SOF_ABI_MINOR_SHIFT) | \
+ ((patch) << SOF_ABI_PATCH_SHIFT))
+
+#define SOF_ABI_VERSION_MAJOR(version) \
+ (((version) >> SOF_ABI_MAJOR_SHIFT) & SOF_ABI_MAJOR_MASK)
+#define SOF_ABI_VERSION_MINOR(version) \
+ (((version) >> SOF_ABI_MINOR_SHIFT) & SOF_ABI_MINOR_MASK)
+#define SOF_ABI_VERSION_PATCH(version) \
+ (((version) >> SOF_ABI_PATCH_SHIFT) & SOF_ABI_PATCH_MASK)
+
+#define SOF_ABI_VERSION_INCOMPATIBLE(sof_ver, client_ver) \
+ (SOF_ABI_VERSION_MAJOR((sof_ver)) != \
+ SOF_ABI_VERSION_MAJOR((client_ver)) \
+ )
+
+#define SOF_ABI_VERSION SOF_ABI_VER(SOF_ABI_MAJOR, SOF_ABI_MINOR, SOF_ABI_PATCH)
+
+/* SOF ABI magic number "SOF\0". */
+#define SOF_ABI_MAGIC 0x00464F53
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_UAPI_SOUND_SOF_USER_EQ_H__
+#define __INCLUDE_UAPI_SOUND_SOF_USER_EQ_H__
+
+/* FIR EQ type */
+
+#define SOF_EQ_FIR_IDX_SWITCH 0
+
+#define SOF_EQ_FIR_MAX_SIZE 4096 /* Max size allowed for coef data in bytes */
+
+#define SOF_EQ_FIR_MAX_LENGTH 192 /* Max length for individual filter */
+
+#define SOF_EQ_FIR_MAX_RESPONSES 8 /* A blob can define max 8 FIR EQs */
+
+/*
+ * eq_fir_configuration data structure contains this information
+ * uint32_t size
+ * This is the number of bytes need to store the received EQ
+ * configuration.
+ * uint16_t channels_in_config
+ * This describes the number of channels in this EQ config data. It
+ * can be different from PLATFORM_MAX_CHANNELS.
+ * uint16_t number_of_responses
+ * 0=no responses, 1=one response defined, 2=two responses defined, etc.
+ * int16_t data[]
+ * assign_response[channels_in_config]
+ * 0 = use first response, 1 = use 2nd response, etc.
+ * E.g. {0, 0, 0, 0, 1, 1, 1, 1} would apply to channels 0-3 the
+ * same first defined response and for to channels 4-7 the second.
+ * coef_data[]
+ * Repeated data
+ * { filter_length, output_shift, h[] }
+ * for every EQ response defined where vector h has filter_length
+ * number of coefficients. Coefficients in h[] are in Q1.15 format.
+ * E.g. 16384 (Q1.15) = 0.5. The shifts are number of right shifts.
+ *
+ * NOTE: The channels_in_config must be even to have coef_data aligned to
+ * 32 bit word in RAM. Therefore a mono EQ assign must be duplicated to 2ch
+ * even if it would never used. Similarly a 5ch EQ assign must be increased
+ * to 6ch. EQ init will return an error if this is not met.
+ *
+ * NOTE: The filter_length must be multiple of four. Therefore the filter must
+ * be padded from the end with zeros have this condition met.
+ */
+
+struct sof_eq_fir_config {
+ uint32_t size;
+ uint16_t channels_in_config;
+ uint16_t number_of_responses;
+
+ /* reserved */
+ uint32_t reserved[4];
+
+ int16_t data[];
+} __packed;
+
+struct sof_eq_fir_coef_data {
+ int16_t length; /* Number of FIR taps */
+ int16_t out_shift; /* Amount of right shifts at output */
+
+ /* reserved */
+ uint32_t reserved[4];
+
+ int16_t coef[]; /* FIR coefficients */
+} __packed;
+
+/* In the struct above there's two 16 bit words (length, shift) and four
+ * reserved 32 bit words before the actual FIR coefficients. This information
+ * is used in parsing of the configuration blob.
+ */
+#define SOF_EQ_FIR_COEF_NHEADER \
+ (sizeof(struct sof_eq_fir_coef_data) / sizeof(int16_t))
+
+/* IIR EQ type */
+
+#define SOF_EQ_IIR_IDX_SWITCH 0
+
+#define SOF_EQ_IIR_MAX_SIZE 1024 /* Max size allowed for coef data in bytes */
+
+#define SOF_EQ_IIR_MAX_RESPONSES 8 /* A blob can define max 8 IIR EQs */
+
+/* eq_iir_configuration
+ * uint32_t channels_in_config
+ * This describes the number of channels in this EQ config data. It
+ * can be different from PLATFORM_MAX_CHANNELS.
+ * uint32_t number_of_responses_defined
+ * 0=no responses, 1=one response defined, 2=two responses defined, etc.
+ * int32_t data[]
+ * Data consist of two parts. First is the response assign vector that
+ * has length of channels_in_config. The latter part is coefficient
+ * data.
+ * uint32_t assign_response[channels_in_config]
+ * -1 = not defined, 0 = use first response, 1 = use 2nd, etc.
+ * E.g. {0, 0, 0, 0, -1, -1, -1, -1} would apply to channels 0-3 the
+ * same first defined response and leave channels 4-7 unequalized.
+ * coefficient_data[]
+ * <1st EQ>
+ * uint32_t num_biquads
+ * uint32_t num_biquads_in_series
+ * <1st biquad>
+ * int32_t coef_a2 Q2.30 format
+ * int32_t coef_a1 Q2.30 format
+ * int32_t coef_b2 Q2.30 format
+ * int32_t coef_b1 Q2.30 format
+ * int32_t coef_b0 Q2.30 format
+ * int32_t output_shift number of shifts right, shift left is negative
+ * int32_t output_gain Q2.14 format
+ * <2nd biquad>
+ * ...
+ * <2nd EQ>
+ *
+ * Note: A flat response biquad can be made with a section set to
+ * b0 = 1.0, gain = 1.0, and other parameters set to 0
+ * {0, 0, 0, 0, 1073741824, 0, 16484}
+ */
+
+struct sof_eq_iir_config {
+ uint32_t size;
+ uint32_t channels_in_config;
+ uint32_t number_of_responses;
+
+ /* reserved */
+ uint32_t reserved[4];
+
+ int32_t data[]; /* eq_assign[channels], eq 0, eq 1, ... */
+} __packed;
+
+struct sof_eq_iir_header_df2t {
+ uint32_t num_sections;
+ uint32_t num_sections_in_series;
+
+ /* reserved */
+ uint32_t reserved[4];
+
+ int32_t biquads[]; /* Repeated biquad coefficients */
+} __packed;
+
+struct sof_eq_iir_biquad_df2t {
+ int32_t a2; /* Q2.30 */
+ int32_t a1; /* Q2.30 */
+ int32_t b2; /* Q2.30 */
+ int32_t b1; /* Q2.30 */
+ int32_t b0; /* Q2.30 */
+ int32_t output_shift; /* Number of right shifts */
+ int32_t output_gain; /* Q2.14 */
+} __packed;
+
+/* A full 22th order equalizer with 11 biquads cover octave bands 1-11 in
+ * in the 0 - 20 kHz bandwidth.
+ */
+#define SOF_EQ_IIR_DF2T_BIQUADS_MAX 11
+
+/* The number of int32_t words in sof_eq_iir_header_df2t:
+ * num_sections, num_sections_in_series, reserved[4]
+ */
+#define SOF_EQ_IIR_NHEADER_DF2T \
+ (sizeof(struct sof_eq_iir_header_df2t) / sizeof(int32_t))
+
+/* The number of int32_t words in sof_eq_iir_biquad_df2t:
+ * a2, a1, b2, b1, b0, output_shift, output_gain
+ */
+#define SOF_EQ_IIR_NBIQUAD_DF2T \
+ (sizeof(struct sof_eq_iir_biquad_df2t) / sizeof(int32_t))
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+/*
+ * Firmware file format .
+ */
+
+#ifndef __INCLUDE_UAPI_SOF_FW_H__
+#define __INCLUDE_UAPI_SOF_FW_H__
+
+#define SND_SOF_FW_SIG_SIZE 4
+#define SND_SOF_FW_ABI 1
+#define SND_SOF_FW_SIG "Reef"
+
+/*
+ * Firmware module is made up of 1 . N blocks of different types. The
+ * Block header is used to determine where and how block is to be copied in the
+ * DSP/host memory space.
+ */
+enum snd_sof_fw_blk_type {
+ SOF_FW_BLK_TYPE_INVALID = -1,
+ SOF_FW_BLK_TYPE_START = 0,
+ SOF_FW_BLK_TYPE_RSRVD0 = SOF_FW_BLK_TYPE_START,
+ SOF_FW_BLK_TYPE_IRAM = 1, /* local instruction RAM */
+ SOF_FW_BLK_TYPE_DRAM = 2, /* local data RAM */
+ SOF_FW_BLK_TYPE_SRAM = 3, /* system RAM */
+ SOF_FW_BLK_TYPE_ROM = 4,
+ SOF_FW_BLK_TYPE_IMR = 5,
+ SOF_FW_BLK_TYPE_RSRVD6 = 6,
+ SOF_FW_BLK_TYPE_RSRVD7 = 7,
+ SOF_FW_BLK_TYPE_RSRVD8 = 8,
+ SOF_FW_BLK_TYPE_RSRVD9 = 9,
+ SOF_FW_BLK_TYPE_RSRVD10 = 10,
+ SOF_FW_BLK_TYPE_RSRVD11 = 11,
+ SOF_FW_BLK_TYPE_RSRVD12 = 12,
+ SOF_FW_BLK_TYPE_RSRVD13 = 13,
+ SOF_FW_BLK_TYPE_RSRVD14 = 14,
+ /* use SOF_FW_BLK_TYPE_RSVRDX for new block types */
+ SOF_FW_BLK_TYPE_NUM
+};
+
+struct snd_sof_blk_hdr {
+ enum snd_sof_fw_blk_type type;
+ uint32_t size; /* bytes minus this header */
+ uint32_t offset; /* offset from base */
+} __packed;
+
+/*
+ * Firmware file is made up of 1 .. N different modules types. The module
+ * type is used to determine how to load and parse the module.
+ */
+enum snd_sof_fw_mod_type {
+ SOF_FW_BASE = 0, /* base firmware image */
+ SOF_FW_MODULE = 1, /* firmware module */
+};
+
+struct snd_sof_mod_hdr {
+ enum snd_sof_fw_mod_type type;
+ uint32_t size; /* bytes minus this header */
+ uint32_t num_blocks; /* number of blocks */
+} __packed;
+
+/*
+ * Firmware file header.
+ */
+struct snd_sof_fw_header {
+ unsigned char sig[SND_SOF_FW_SIG_SIZE]; /* "Reef" */
+ uint32_t file_size; /* size of file minus this header */
+ uint32_t num_modules; /* number of modules */
+ uint32_t abi; /* version of header format */
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_UAPI_SOUND_SOF_USER_HEADER_H__
+#define __INCLUDE_UAPI_SOUND_SOF_USER_HEADER_H__
+
+/*
+ * Header for all non IPC ABI data.
+ *
+ * Identifies data type, size and ABI.
+ * Used by any bespoke component data structures or binary blobs.
+ */
+struct sof_abi_hdr {
+ uint32_t magic; /**< 'S', 'O', 'F', '\0' */
+ uint32_t type; /**< component specific type */
+ uint32_t size; /**< size in bytes of data excl. this struct */
+ uint32_t abi; /**< SOF ABI version */
+ uint32_t reserved[4]; /**< reserved for future use */
+ uint32_t data[0]; /**< Component data - opaque to core */
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_UAPI_SOUND_SOF_USER_MANIFEST_H__
+#define __INCLUDE_UAPI_SOUND_SOF_USER_MANIFEST_H__
+
+/* start offset for base FW module */
+#define SOF_MAN_ELF_TEXT_OFFSET 0x2000
+
+/* FW Extended Manifest Header id = $AE1 */
+#define SOF_MAN_EXT_HEADER_MAGIC 0x31454124
+
+/* module type load type */
+#define SOF_MAN_MOD_TYPE_BUILTIN 0
+#define SOF_MAN_MOD_TYPE_MODULE 1
+
+struct sof_man_module_type {
+ uint32_t load_type:4; /* SOF_MAN_MOD_TYPE_ */
+ uint32_t auto_start:1;
+ uint32_t domain_ll:1;
+ uint32_t domain_dp:1;
+ uint32_t rsvd_:25;
+};
+
+/* segment flags.type */
+#define SOF_MAN_SEGMENT_TEXT 0
+#define SOF_MAN_SEGMENT_RODATA 1
+#define SOF_MAN_SEGMENT_DATA 1
+#define SOF_MAN_SEGMENT_BSS 2
+#define SOF_MAN_SEGMENT_EMPTY 15
+
+union sof_man_segment_flags {
+ uint32_t ul;
+ struct {
+ uint32_t contents:1;
+ uint32_t alloc:1;
+ uint32_t load:1;
+ uint32_t readonly:1;
+ uint32_t code:1;
+ uint32_t data:1;
+ uint32_t _rsvd0:2;
+ uint32_t type:4; /* MAN_SEGMENT_ */
+ uint32_t _rsvd1:4;
+ uint32_t length:16; /* of segment in pages */
+ } r;
+} __packed;
+
+/*
+ * Module segment descriptor. Used by ROM - Immutable.
+ */
+struct sof_man_segment_desc {
+ union sof_man_segment_flags flags;
+ uint32_t v_base_addr;
+ uint32_t file_offset;
+} __packed;
+
+/*
+ * The firmware binary can be split into several modules.
+ */
+
+#define SOF_MAN_MOD_ID_LEN 4
+#define SOF_MAN_MOD_NAME_LEN 8
+#define SOF_MAN_MOD_SHA256_LEN 32
+#define SOF_MAN_MOD_ID {'$', 'A', 'M', 'E'}
+
+/*
+ * Each module has an entry in the FW header. Used by ROM - Immutable.
+ */
+struct sof_man_module {
+ uint8_t struct_id[SOF_MAN_MOD_ID_LEN]; /* SOF_MAN_MOD_ID */
+ uint8_t name[SOF_MAN_MOD_NAME_LEN];
+ uint8_t uuid[16];
+ struct sof_man_module_type type;
+ uint8_t hash[SOF_MAN_MOD_SHA256_LEN];
+ uint32_t entry_point;
+ uint16_t cfg_offset;
+ uint16_t cfg_count;
+ uint32_t affinity_mask;
+ uint16_t instance_max_count; /* max number of instances */
+ uint16_t instance_bss_size; /* instance (pages) */
+ struct sof_man_segment_desc segment[3];
+} __packed;
+
+/*
+ * Each module has a configuration in the FW header. Used by ROM - Immutable.
+ */
+struct sof_man_mod_config {
+ uint32_t par[4]; /* module parameters */
+ uint32_t is_pages; /* actual size of instance .bss (pages) */
+ uint32_t cps; /* cycles per second */
+ uint32_t ibs; /* input buffer size (bytes) */
+ uint32_t obs; /* output buffer size (bytes) */
+ uint32_t module_flags; /* flags, reserved for future use */
+ uint32_t cpc; /* cycles per single run */
+ uint32_t obls; /* output block size, reserved for future use */
+} __packed;
+
+/*
+ * FW Manifest Header
+ */
+
+#define SOF_MAN_FW_HDR_FW_NAME_LEN 8
+#define SOF_MAN_FW_HDR_ID {'$', 'A', 'M', '1'}
+#define SOF_MAN_FW_HDR_NAME "ADSPFW"
+#define SOF_MAN_FW_HDR_FLAGS 0x0
+#define SOF_MAN_FW_HDR_FEATURES 0xff
+
+/*
+ * The firmware has a standard header that is checked by the ROM on firmware
+ * loading. preload_page_count is used by DMA code loader and is entire
+ * image size on CNL. i.e. CNL: total size of the binary’s .text and .rodata
+ * Used by ROM - Immutable.
+ */
+struct sof_man_fw_header {
+ uint8_t header_id[4];
+ uint32_t header_len;
+ uint8_t name[SOF_MAN_FW_HDR_FW_NAME_LEN];
+ /* number of pages of preloaded image loaded by driver */
+ uint32_t preload_page_count;
+ uint32_t fw_image_flags;
+ uint32_t feature_mask;
+ uint16_t major_version;
+ uint16_t minor_version;
+ uint16_t hotfix_version;
+ uint16_t build_version;
+ uint32_t num_module_entries;
+ uint32_t hw_buf_base_addr;
+ uint32_t hw_buf_length;
+ /* target address for binary loading as offset in IMR - must be == base offset */
+ uint32_t load_offset;
+} __packed;
+
+/*
+ * Firmware manifest descriptor. This can contain N modules and N module
+ * configs. Used by ROM - Immutable.
+ */
+struct sof_man_fw_desc {
+ struct sof_man_fw_header header;
+
+ /* Warning - hack for module arrays. For some unknown reason the we
+ * have a variable size array of struct man_module followed by a
+ * variable size array of struct mod_config. These should have been
+ * merged into a variable array of a parent structure. We have to hack
+ * around this in many places....
+ *
+ * struct sof_man_module man_module[];
+ * struct sof_man_mod_config mod_config[];
+ */
+
+} __packed;
+
+/*
+ * Component Descriptor. Used by ROM - Immutable.
+ */
+struct sof_man_component_desc {
+ uint32_t reserved[2]; /* all 0 */
+ uint32_t version;
+ uint8_t hash[SOF_MAN_MOD_SHA256_LEN];
+ uint32_t base_offset;
+ uint32_t limit_offset;
+ uint32_t attributes[4];
+} __packed;
+
+/*
+ * Audio DSP extended metadata. Used by ROM - Immutable.
+ */
+struct sof_man_adsp_meta_file_ext {
+ uint32_t ext_type; /* always 17 for ADSP extension */
+ uint32_t ext_len;
+ uint32_t imr_type;
+ uint8_t reserved[16]; /* all 0 */
+ struct sof_man_component_desc comp_desc[1];
+} __packed;
+
+/*
+ * Module Manifest for rimage module metadata. Not used by ROM.
+ */
+struct sof_man_module_manifest {
+ struct sof_man_module module;
+ uint32_t text_size;
+} __packed;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ * Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+ * Keyon Jie <yang.jie@linux.intel.com>
+ */
+
+/*
+ * Topology IDs and tokens.
+ *
+ * ** MUST BE ALIGNED WITH TOPOLOGY CONFIGURATION TOKEN VALUES **
+ */
+
+#ifndef __INCLUDE_UAPI_SOF_TOPOLOGY_H__
+#define __INCLUDE_UAPI_SOF_TOPOLOGY_H__
+
+/*
+ * Kcontrol IDs
+ */
+#define SOF_TPLG_KCTL_VOL_ID 256
+#define SOF_TPLG_KCTL_ENUM_ID 257
+#define SOF_TPLG_KCTL_BYTES_ID 258
+#define SOF_TPLG_KCTL_SWITCH_ID 259
+
+/*
+ * Tokens - must match values in topology configurations
+ */
+
+/* buffers */
+#define SOF_TKN_BUF_SIZE 100
+#define SOF_TKN_BUF_CAPS 101
+
+/* DAI */
+/* Token retired with ABI 3.2, do not use for new capabilities
+ * #define SOF_TKN_DAI_DMAC_CONFIG 153
+ */
+#define SOF_TKN_DAI_TYPE 154
+#define SOF_TKN_DAI_INDEX 155
+#define SOF_TKN_DAI_DIRECTION 156
+
+/* scheduling */
+#define SOF_TKN_SCHED_PERIOD 200
+#define SOF_TKN_SCHED_PRIORITY 201
+#define SOF_TKN_SCHED_MIPS 202
+#define SOF_TKN_SCHED_CORE 203
+#define SOF_TKN_SCHED_FRAMES 204
+#define SOF_TKN_SCHED_TIME_DOMAIN 205
+
+/* volume */
+#define SOF_TKN_VOLUME_RAMP_STEP_TYPE 250
+#define SOF_TKN_VOLUME_RAMP_STEP_MS 251
+
+/* SRC */
+#define SOF_TKN_SRC_RATE_IN 300
+#define SOF_TKN_SRC_RATE_OUT 301
+
+/* PCM */
+#define SOF_TKN_PCM_DMAC_CONFIG 353
+
+/* Generic components */
+#define SOF_TKN_COMP_PERIOD_SINK_COUNT 400
+#define SOF_TKN_COMP_PERIOD_SOURCE_COUNT 401
+#define SOF_TKN_COMP_FORMAT 402
+/* Token retired with ABI 3.2, do not use for new capabilities
+ * #define SOF_TKN_COMP_PRELOAD_COUNT 403
+ */
+
+/* SSP */
+#define SOF_TKN_INTEL_SSP_CLKS_CONTROL 500
+#define SOF_TKN_INTEL_SSP_MCLK_ID 501
+#define SOF_TKN_INTEL_SSP_SAMPLE_BITS 502
+#define SOF_TKN_INTEL_SSP_FRAME_PULSE_WIDTH 503
+#define SOF_TKN_INTEL_SSP_QUIRKS 504
+#define SOF_TKN_INTEL_SSP_TDM_PADDING_PER_SLOT 505
+
+/* DMIC */
+#define SOF_TKN_INTEL_DMIC_DRIVER_VERSION 600
+#define SOF_TKN_INTEL_DMIC_CLK_MIN 601
+#define SOF_TKN_INTEL_DMIC_CLK_MAX 602
+#define SOF_TKN_INTEL_DMIC_DUTY_MIN 603
+#define SOF_TKN_INTEL_DMIC_DUTY_MAX 604
+#define SOF_TKN_INTEL_DMIC_NUM_PDM_ACTIVE 605
+#define SOF_TKN_INTEL_DMIC_SAMPLE_RATE 608
+#define SOF_TKN_INTEL_DMIC_FIFO_WORD_LENGTH 609
+
+/* DMIC PDM */
+#define SOF_TKN_INTEL_DMIC_PDM_CTRL_ID 700
+#define SOF_TKN_INTEL_DMIC_PDM_MIC_A_Enable 701
+#define SOF_TKN_INTEL_DMIC_PDM_MIC_B_Enable 702
+#define SOF_TKN_INTEL_DMIC_PDM_POLARITY_A 703
+#define SOF_TKN_INTEL_DMIC_PDM_POLARITY_B 704
+#define SOF_TKN_INTEL_DMIC_PDM_CLK_EDGE 705
+#define SOF_TKN_INTEL_DMIC_PDM_SKEW 706
+
+/* Tone */
+#define SOF_TKN_TONE_SAMPLE_RATE 800
+
+/* Processing Components */
+#define SOF_TKN_PROCESS_TYPE 900
+
+/* for backward compatibility */
+#define SOF_TKN_EFFECT_TYPE SOF_TKN_PROCESS_TYPE
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+* This file is provided under a dual BSD/GPLv2 license. When using or
+* redistributing this file, you may do so under either license.
+*
+* Copyright(c) 2018 Intel Corporation. All rights reserved.
+*/
+
+#ifndef __INCLUDE_UAPI_SOUND_SOF_USER_TONE_H__
+#define __INCLUDE_UAPI_SOUND_SOF_USER_TONE_H__
+
+#define SOF_TONE_IDX_FREQUENCY 0
+#define SOF_TONE_IDX_AMPLITUDE 1
+#define SOF_TONE_IDX_FREQ_MULT 2
+#define SOF_TONE_IDX_AMPL_MULT 3
+#define SOF_TONE_IDX_LENGTH 4
+#define SOF_TONE_IDX_PERIOD 5
+#define SOF_TONE_IDX_REPEATS 6
+#define SOF_TONE_IDX_LIN_RAMP_STEP 7
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __INCLUDE_UAPI_SOUND_SOF_USER_TRACE_H__
+#define __INCLUDE_UAPI_SOUND_SOF_USER_TRACE_H__
+
+/*
+ * Host system time.
+ *
+ * This property is used by the driver to pass down information about
+ * current system time. It is expressed in us.
+ * FW translates timestamps (in log entries, probe pockets) to this time
+ * domain.
+ *
+ * (cavs: SystemTime).
+ */
+struct system_time {
+ uint32_t val_l; /* Lower dword of current host time value */
+ uint32_t val_u; /* Upper dword of current host time value */
+} __packed;
+
+#define LOG_ENABLE 1 /* Enable logging */
+#define LOG_DISABLE 0 /* Disable logging */
+
+#define LOG_LEVEL_CRITICAL 1 /* (FDK fatal) */
+#define LOG_LEVEL_VERBOSE 2
+
+/*
+ * Layout of a log fifo.
+ */
+struct log_buffer_layout {
+ uint32_t read_ptr; /*read pointer */
+ uint32_t write_ptr; /* write pointer */
+ uint32_t buffer[0]; /* buffer */
+} __packed;
+
+/*
+ * Log buffer status reported by FW.
+ */
+struct log_buffer_status {
+ uint32_t core_id; /* ID of core that logged to other half */
+} __packed;
+
+#define TRACE_ID_LENGTH 12
+
+/*
+ * Log entry header.
+ *
+ * The header is followed by an array of arguments (uint32_t[]).
+ * Number of arguments is specified by the params_num field of log_entry
+ */
+struct log_entry_header {
+ uint32_t id_0 : TRACE_ID_LENGTH; /* e.g. Pipeline ID */
+ uint32_t id_1 : TRACE_ID_LENGTH; /* e.g. Component ID */
+ uint32_t core_id : 8; /* Reporting core's id */
+
+ uint64_t timestamp; /* Timestamp (in dsp ticks) */
+ uint32_t log_entry_address; /* Address of log entry in ELF */
+} __packed;
+
+#endif
# SPDX-License-Identifier: GPL-2.0
-obj-y := cgroup.o rstat.o namespace.o cgroup-v1.o
+obj-y := cgroup.o rstat.o namespace.o cgroup-v1.o freezer.o
-obj-$(CONFIG_CGROUP_FREEZER) += freezer.o
+obj-$(CONFIG_CGROUP_FREEZER) += legacy_freezer.o
obj-$(CONFIG_CGROUP_PIDS) += pids.o
obj-$(CONFIG_CGROUP_RDMA) += rdma.o
obj-$(CONFIG_CPUSETS) += cpuset.o
#define TRACE_CGROUP_PATH(type, cgrp, ...) \
do { \
if (trace_cgroup_##type##_enabled()) { \
- spin_lock(&trace_cgroup_path_lock); \
+ unsigned long flags; \
+ spin_lock_irqsave(&trace_cgroup_path_lock, \
+ flags); \
cgroup_path(cgrp, trace_cgroup_path, \
TRACE_CGROUP_PATH_LEN); \
trace_cgroup_##type(cgrp, trace_cgroup_path, \
##__VA_ARGS__); \
- spin_unlock(&trace_cgroup_path_lock); \
+ spin_unlock_irqrestore(&trace_cgroup_path_lock, \
+ flags); \
} \
} while (0)
int cgroup_show_path(struct seq_file *sf, struct kernfs_node *kf_node,
struct kernfs_root *kf_root);
+int __cgroup_task_count(const struct cgroup *cgrp);
int cgroup_task_count(const struct cgroup *cgrp);
/*
return l;
}
-/**
- * cgroup_task_count - count the number of tasks in a cgroup.
- * @cgrp: the cgroup in question
- */
-int cgroup_task_count(const struct cgroup *cgrp)
-{
- int count = 0;
- struct cgrp_cset_link *link;
-
- spin_lock_irq(&css_set_lock);
- list_for_each_entry(link, &cgrp->cset_links, cset_link)
- count += link->cset->nr_tasks;
- spin_unlock_irq(&css_set_lock);
- return count;
-}
-
/*
* Load a cgroup's pidarray with either procs' tgids or tasks' pids
*/
css_get(&cgrp->self);
}
+/**
+ * __cgroup_task_count - count the number of tasks in a cgroup. The caller
+ * is responsible for taking the css_set_lock.
+ * @cgrp: the cgroup in question
+ */
+int __cgroup_task_count(const struct cgroup *cgrp)
+{
+ int count = 0;
+ struct cgrp_cset_link *link;
+
+ lockdep_assert_held(&css_set_lock);
+
+ list_for_each_entry(link, &cgrp->cset_links, cset_link)
+ count += link->cset->nr_tasks;
+
+ return count;
+}
+
+/**
+ * cgroup_task_count - count the number of tasks in a cgroup.
+ * @cgrp: the cgroup in question
+ */
+int cgroup_task_count(const struct cgroup *cgrp)
+{
+ int count;
+
+ spin_lock_irq(&css_set_lock);
+ count = __cgroup_task_count(cgrp);
+ spin_unlock_irq(&css_set_lock);
+
+ return count;
+}
+
struct cgroup_subsys_state *of_css(struct kernfs_open_file *of)
{
struct cgroup *cgrp = of->kn->parent->priv;
break;
cgroup1_check_for_release(cgrp);
+ TRACE_CGROUP_PATH(notify_populated, cgrp,
+ cgroup_is_populated(cgrp));
cgroup_file_notify(&cgrp->events_file);
child = cgrp;
get_css_set(to_cset);
to_cset->nr_tasks++;
css_set_move_task(task, from_cset, to_cset, true);
- put_css_set_locked(from_cset);
from_cset->nr_tasks--;
+ /*
+ * If the source or destination cgroup is frozen,
+ * the task might require to change its state.
+ */
+ cgroup_freezer_migrate_task(task, from_cset->dfl_cgrp,
+ to_cset->dfl_cgrp);
+ put_css_set_locked(from_cset);
+
}
}
spin_unlock_irq(&css_set_lock);
dst_cset = find_css_set(src_cset, src_cset->mg_dst_cgrp);
if (!dst_cset)
- goto err;
+ return -ENOMEM;
WARN_ON_ONCE(src_cset->mg_dst_cset || dst_cset->mg_dst_cset);
}
return 0;
-err:
- cgroup_migrate_finish(mgctx);
- return -ENOMEM;
}
/**
static int cgroup_events_show(struct seq_file *seq, void *v)
{
- seq_printf(seq, "populated %d\n",
- cgroup_is_populated(seq_css(seq)->cgroup));
+ struct cgroup *cgrp = seq_css(seq)->cgroup;
+
+ seq_printf(seq, "populated %d\n", cgroup_is_populated(cgrp));
+ seq_printf(seq, "frozen %d\n", test_bit(CGRP_FROZEN, &cgrp->flags));
+
return 0;
}
}
#endif
+static int cgroup_freeze_show(struct seq_file *seq, void *v)
+{
+ struct cgroup *cgrp = seq_css(seq)->cgroup;
+
+ seq_printf(seq, "%d\n", cgrp->freezer.freeze);
+
+ return 0;
+}
+
+static ssize_t cgroup_freeze_write(struct kernfs_open_file *of,
+ char *buf, size_t nbytes, loff_t off)
+{
+ struct cgroup *cgrp;
+ ssize_t ret;
+ int freeze;
+
+ ret = kstrtoint(strstrip(buf), 0, &freeze);
+ if (ret)
+ return ret;
+
+ if (freeze < 0 || freeze > 1)
+ return -ERANGE;
+
+ cgrp = cgroup_kn_lock_live(of->kn, false);
+ if (!cgrp)
+ return -ENOENT;
+
+ cgroup_freeze(cgrp, freeze);
+
+ cgroup_kn_unlock(of->kn);
+
+ return nbytes;
+}
+
static int cgroup_file_open(struct kernfs_open_file *of)
{
struct cftype *cft = of->kn->priv;
.name = "cgroup.stat",
.seq_show = cgroup_stat_show,
},
+ {
+ .name = "cgroup.freeze",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = cgroup_freeze_show,
+ .write = cgroup_freeze_write,
+ },
{
.name = "cpu.stat",
.flags = CFTYPE_NOT_ON_ROOT,
if (cgroup_on_dfl(cgrp))
cgroup_rstat_flush(cgrp);
+ spin_lock_irq(&css_set_lock);
for (tcgrp = cgroup_parent(cgrp); tcgrp;
tcgrp = cgroup_parent(tcgrp))
tcgrp->nr_dying_descendants--;
+ spin_unlock_irq(&css_set_lock);
cgroup_idr_remove(&cgrp->root->cgroup_idr, cgrp->id);
cgrp->id = -1;
if (ret)
goto out_psi_free;
+ /*
+ * New cgroup inherits effective freeze counter, and
+ * if the parent has to be frozen, the child has too.
+ */
+ cgrp->freezer.e_freeze = parent->freezer.e_freeze;
+ if (cgrp->freezer.e_freeze)
+ set_bit(CGRP_FROZEN, &cgrp->flags);
+
+ spin_lock_irq(&css_set_lock);
for (tcgrp = cgrp; tcgrp; tcgrp = cgroup_parent(tcgrp)) {
cgrp->ancestor_ids[tcgrp->level] = tcgrp->id;
- if (tcgrp != cgrp)
+ if (tcgrp != cgrp) {
tcgrp->nr_descendants++;
+
+ /*
+ * If the new cgroup is frozen, all ancestor cgroups
+ * get a new frozen descendant, but their state can't
+ * change because of this.
+ */
+ if (cgrp->freezer.e_freeze)
+ tcgrp->freezer.nr_frozen_descendants++;
+ }
}
+ spin_unlock_irq(&css_set_lock);
if (notify_on_release(parent))
set_bit(CGRP_NOTIFY_ON_RELEASE, &cgrp->flags);
if (parent && cgroup_is_threaded(cgrp))
parent->nr_threaded_children--;
+ spin_lock_irq(&css_set_lock);
for (tcgrp = cgroup_parent(cgrp); tcgrp; tcgrp = cgroup_parent(tcgrp)) {
tcgrp->nr_descendants--;
tcgrp->nr_dying_descendants++;
+ /*
+ * If the dying cgroup is frozen, decrease frozen descendants
+ * counters of ancestor cgroups.
+ */
+ if (test_bit(CGRP_FROZEN, &cgrp->flags))
+ tcgrp->freezer.nr_frozen_descendants--;
}
+ spin_unlock_irq(&css_set_lock);
cgroup1_check_for_release(parent);
cset->nr_tasks++;
css_set_move_task(child, NULL, cset, false);
}
+
+ /*
+ * If the cgroup has to be frozen, the new task has too.
+ * Let's set the JOBCTL_TRAP_FREEZE jobctl bit to get
+ * the task into the frozen state.
+ */
+ if (unlikely(cgroup_task_freeze(child))) {
+ spin_lock(&child->sighand->siglock);
+ WARN_ON_ONCE(child->frozen);
+ child->jobctl |= JOBCTL_TRAP_FREEZE;
+ spin_unlock(&child->sighand->siglock);
+
+ /*
+ * Calling cgroup_update_frozen() isn't required here,
+ * because it will be called anyway a bit later
+ * from do_freezer_trap(). So we avoid cgroup's
+ * transient switch from the frozen state and back.
+ */
+ }
+
spin_unlock_irq(&css_set_lock);
}
spin_lock_irq(&css_set_lock);
css_set_move_task(tsk, cset, NULL, false);
cset->nr_tasks--;
+
+ WARN_ON_ONCE(cgroup_task_frozen(tsk));
+ if (unlikely(cgroup_task_freeze(tsk)))
+ cgroup_update_frozen(task_dfl_cgroup(tsk));
+
spin_unlock_irq(&css_set_lock);
} else {
get_css_set(cset);
css = cset->subsys[ss->id];
if (!css)
continue;
- seq_printf(seq, "%2d: %-4s\t- %lx[%d]\n", ss->id, ss->name,
- (unsigned long)css, css->id);
+ seq_printf(seq, "%2d: %-4s\t- %p[%d]\n", ss->id, ss->name,
+ css, css->id);
}
rcu_read_unlock();
spin_unlock_irq(&css_set_lock);
if (css->parent)
snprintf(pbuf, sizeof(pbuf) - 1, " P=%d",
css->parent->id);
- seq_printf(seq, "%2d: %-4s\t- %lx[%d] %d%s\n", ss->id, ss->name,
- (unsigned long)css, css->id,
+ seq_printf(seq, "%2d: %-4s\t- %p[%d] %d%s\n", ss->id, ss->name,
+ css, css->id,
atomic_read(&css->online_cnt), pbuf);
}
-/*
- * cgroup_freezer.c - control group freezer subsystem
- *
- * Copyright IBM Corporation, 2007
- *
- * Author : Cedric Le Goater <clg@fr.ibm.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of version 2.1 of the GNU Lesser General Public License
- * as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it would be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- */
-
-#include <linux/export.h>
-#include <linux/slab.h>
+//SPDX-License-Identifier: GPL-2.0
#include <linux/cgroup.h>
-#include <linux/fs.h>
-#include <linux/uaccess.h>
-#include <linux/freezer.h>
-#include <linux/seq_file.h>
-#include <linux/mutex.h>
-
-/*
- * A cgroup is freezing if any FREEZING flags are set. FREEZING_SELF is
- * set if "FROZEN" is written to freezer.state cgroupfs file, and cleared
- * for "THAWED". FREEZING_PARENT is set if the parent freezer is FREEZING
- * for whatever reason. IOW, a cgroup has FREEZING_PARENT set if one of
- * its ancestors has FREEZING_SELF set.
- */
-enum freezer_state_flags {
- CGROUP_FREEZER_ONLINE = (1 << 0), /* freezer is fully online */
- CGROUP_FREEZING_SELF = (1 << 1), /* this freezer is freezing */
- CGROUP_FREEZING_PARENT = (1 << 2), /* the parent freezer is freezing */
- CGROUP_FROZEN = (1 << 3), /* this and its descendants frozen */
+#include <linux/sched.h>
+#include <linux/sched/task.h>
+#include <linux/sched/signal.h>
- /* mask for all FREEZING flags */
- CGROUP_FREEZING = CGROUP_FREEZING_SELF | CGROUP_FREEZING_PARENT,
-};
+#include "cgroup-internal.h"
-struct freezer {
- struct cgroup_subsys_state css;
- unsigned int state;
-};
+#include <trace/events/cgroup.h>
-static DEFINE_MUTEX(freezer_mutex);
-
-static inline struct freezer *css_freezer(struct cgroup_subsys_state *css)
+/*
+ * Propagate the cgroup frozen state upwards by the cgroup tree.
+ */
+static void cgroup_propagate_frozen(struct cgroup *cgrp, bool frozen)
{
- return css ? container_of(css, struct freezer, css) : NULL;
-}
+ int desc = 1;
-static inline struct freezer *task_freezer(struct task_struct *task)
-{
- return css_freezer(task_css(task, freezer_cgrp_id));
+ /*
+ * If the new state is frozen, some freezing ancestor cgroups may change
+ * their state too, depending on if all their descendants are frozen.
+ *
+ * Otherwise, all ancestor cgroups are forced into the non-frozen state.
+ */
+ while ((cgrp = cgroup_parent(cgrp))) {
+ if (frozen) {
+ cgrp->freezer.nr_frozen_descendants += desc;
+ if (!test_bit(CGRP_FROZEN, &cgrp->flags) &&
+ test_bit(CGRP_FREEZE, &cgrp->flags) &&
+ cgrp->freezer.nr_frozen_descendants ==
+ cgrp->nr_descendants) {
+ set_bit(CGRP_FROZEN, &cgrp->flags);
+ cgroup_file_notify(&cgrp->events_file);
+ TRACE_CGROUP_PATH(notify_frozen, cgrp, 1);
+ desc++;
+ }
+ } else {
+ cgrp->freezer.nr_frozen_descendants -= desc;
+ if (test_bit(CGRP_FROZEN, &cgrp->flags)) {
+ clear_bit(CGRP_FROZEN, &cgrp->flags);
+ cgroup_file_notify(&cgrp->events_file);
+ TRACE_CGROUP_PATH(notify_frozen, cgrp, 0);
+ desc++;
+ }
+ }
+ }
}
-static struct freezer *parent_freezer(struct freezer *freezer)
+/*
+ * Revisit the cgroup frozen state.
+ * Checks if the cgroup is really frozen and perform all state transitions.
+ */
+void cgroup_update_frozen(struct cgroup *cgrp)
{
- return css_freezer(freezer->css.parent);
-}
+ bool frozen;
-bool cgroup_freezing(struct task_struct *task)
-{
- bool ret;
+ lockdep_assert_held(&css_set_lock);
- rcu_read_lock();
- ret = task_freezer(task)->state & CGROUP_FREEZING;
- rcu_read_unlock();
+ /*
+ * If the cgroup has to be frozen (CGRP_FREEZE bit set),
+ * and all tasks are frozen and/or stopped, let's consider
+ * the cgroup frozen. Otherwise it's not frozen.
+ */
+ frozen = test_bit(CGRP_FREEZE, &cgrp->flags) &&
+ cgrp->freezer.nr_frozen_tasks == __cgroup_task_count(cgrp);
- return ret;
-}
+ if (frozen) {
+ /* Already there? */
+ if (test_bit(CGRP_FROZEN, &cgrp->flags))
+ return;
-static const char *freezer_state_strs(unsigned int state)
-{
- if (state & CGROUP_FROZEN)
- return "FROZEN";
- if (state & CGROUP_FREEZING)
- return "FREEZING";
- return "THAWED";
-};
-
-static struct cgroup_subsys_state *
-freezer_css_alloc(struct cgroup_subsys_state *parent_css)
-{
- struct freezer *freezer;
+ set_bit(CGRP_FROZEN, &cgrp->flags);
+ } else {
+ /* Already there? */
+ if (!test_bit(CGRP_FROZEN, &cgrp->flags))
+ return;
- freezer = kzalloc(sizeof(struct freezer), GFP_KERNEL);
- if (!freezer)
- return ERR_PTR(-ENOMEM);
+ clear_bit(CGRP_FROZEN, &cgrp->flags);
+ }
+ cgroup_file_notify(&cgrp->events_file);
+ TRACE_CGROUP_PATH(notify_frozen, cgrp, frozen);
- return &freezer->css;
+ /* Update the state of ancestor cgroups. */
+ cgroup_propagate_frozen(cgrp, frozen);
}
-/**
- * freezer_css_online - commit creation of a freezer css
- * @css: css being created
- *
- * We're committing to creation of @css. Mark it online and inherit
- * parent's freezing state while holding both parent's and our
- * freezer->lock.
+/*
+ * Increment cgroup's nr_frozen_tasks.
*/
-static int freezer_css_online(struct cgroup_subsys_state *css)
+static void cgroup_inc_frozen_cnt(struct cgroup *cgrp)
{
- struct freezer *freezer = css_freezer(css);
- struct freezer *parent = parent_freezer(freezer);
-
- mutex_lock(&freezer_mutex);
-
- freezer->state |= CGROUP_FREEZER_ONLINE;
-
- if (parent && (parent->state & CGROUP_FREEZING)) {
- freezer->state |= CGROUP_FREEZING_PARENT | CGROUP_FROZEN;
- atomic_inc(&system_freezing_cnt);
- }
-
- mutex_unlock(&freezer_mutex);
- return 0;
+ cgrp->freezer.nr_frozen_tasks++;
}
-/**
- * freezer_css_offline - initiate destruction of a freezer css
- * @css: css being destroyed
- *
- * @css is going away. Mark it dead and decrement system_freezing_count if
- * it was holding one.
+/*
+ * Decrement cgroup's nr_frozen_tasks.
*/
-static void freezer_css_offline(struct cgroup_subsys_state *css)
+static void cgroup_dec_frozen_cnt(struct cgroup *cgrp)
{
- struct freezer *freezer = css_freezer(css);
-
- mutex_lock(&freezer_mutex);
-
- if (freezer->state & CGROUP_FREEZING)
- atomic_dec(&system_freezing_cnt);
-
- freezer->state = 0;
-
- mutex_unlock(&freezer_mutex);
+ cgrp->freezer.nr_frozen_tasks--;
+ WARN_ON_ONCE(cgrp->freezer.nr_frozen_tasks < 0);
}
-static void freezer_css_free(struct cgroup_subsys_state *css)
+/*
+ * Enter frozen/stopped state, if not yet there. Update cgroup's counters,
+ * and revisit the state of the cgroup, if necessary.
+ */
+void cgroup_enter_frozen(void)
{
- kfree(css_freezer(css));
+ struct cgroup *cgrp;
+
+ if (current->frozen)
+ return;
+
+ spin_lock_irq(&css_set_lock);
+ current->frozen = true;
+ cgrp = task_dfl_cgroup(current);
+ cgroup_inc_frozen_cnt(cgrp);
+ cgroup_update_frozen(cgrp);
+ spin_unlock_irq(&css_set_lock);
}
/*
- * Tasks can be migrated into a different freezer anytime regardless of its
- * current state. freezer_attach() is responsible for making new tasks
- * conform to the current state.
+ * Conditionally leave frozen/stopped state. Update cgroup's counters,
+ * and revisit the state of the cgroup, if necessary.
*
- * Freezer state changes and task migration are synchronized via
- * @freezer->lock. freezer_attach() makes the new tasks conform to the
- * current state and all following state changes can see the new tasks.
+ * If always_leave is not set, and the cgroup is freezing,
+ * we're racing with the cgroup freezing. In this case, we don't
+ * drop the frozen counter to avoid a transient switch to
+ * the unfrozen state.
*/
-static void freezer_attach(struct cgroup_taskset *tset)
+void cgroup_leave_frozen(bool always_leave)
{
- struct task_struct *task;
- struct cgroup_subsys_state *new_css;
-
- mutex_lock(&freezer_mutex);
-
- /*
- * Make the new tasks conform to the current state of @new_css.
- * For simplicity, when migrating any task to a FROZEN cgroup, we
- * revert it to FREEZING and let update_if_frozen() determine the
- * correct state later.
- *
- * Tasks in @tset are on @new_css but may not conform to its
- * current state before executing the following - !frozen tasks may
- * be visible in a FROZEN cgroup and frozen tasks in a THAWED one.
- */
- cgroup_taskset_for_each(task, new_css, tset) {
- struct freezer *freezer = css_freezer(new_css);
-
- if (!(freezer->state & CGROUP_FREEZING)) {
- __thaw_task(task);
- } else {
- freeze_task(task);
- /* clear FROZEN and propagate upwards */
- while (freezer && (freezer->state & CGROUP_FROZEN)) {
- freezer->state &= ~CGROUP_FROZEN;
- freezer = parent_freezer(freezer);
- }
- }
+ struct cgroup *cgrp;
+
+ spin_lock_irq(&css_set_lock);
+ cgrp = task_dfl_cgroup(current);
+ if (always_leave || !test_bit(CGRP_FREEZE, &cgrp->flags)) {
+ cgroup_dec_frozen_cnt(cgrp);
+ cgroup_update_frozen(cgrp);
+ WARN_ON_ONCE(!current->frozen);
+ current->frozen = false;
+ } else if (!(current->jobctl & JOBCTL_TRAP_FREEZE)) {
+ spin_lock(¤t->sighand->siglock);
+ current->jobctl |= JOBCTL_TRAP_FREEZE;
+ set_thread_flag(TIF_SIGPENDING);
+ spin_unlock(¤t->sighand->siglock);
}
-
- mutex_unlock(&freezer_mutex);
+ spin_unlock_irq(&css_set_lock);
}
-/**
- * freezer_fork - cgroup post fork callback
- * @task: a task which has just been forked
- *
- * @task has just been created and should conform to the current state of
- * the cgroup_freezer it belongs to. This function may race against
- * freezer_attach(). Losing to freezer_attach() means that we don't have
- * to do anything as freezer_attach() will put @task into the appropriate
- * state.
+/*
+ * Freeze or unfreeze the task by setting or clearing the JOBCTL_TRAP_FREEZE
+ * jobctl bit.
*/
-static void freezer_fork(struct task_struct *task)
+static void cgroup_freeze_task(struct task_struct *task, bool freeze)
{
- struct freezer *freezer;
+ unsigned long flags;
- /*
- * The root cgroup is non-freezable, so we can skip locking the
- * freezer. This is safe regardless of race with task migration.
- * If we didn't race or won, skipping is obviously the right thing
- * to do. If we lost and root is the new cgroup, noop is still the
- * right thing to do.
- */
- if (task_css_is_root(task, freezer_cgrp_id))
+ /* If the task is about to die, don't bother with freezing it. */
+ if (!lock_task_sighand(task, &flags))
return;
- mutex_lock(&freezer_mutex);
- rcu_read_lock();
-
- freezer = task_freezer(task);
- if (freezer->state & CGROUP_FREEZING)
- freeze_task(task);
+ if (freeze) {
+ task->jobctl |= JOBCTL_TRAP_FREEZE;
+ signal_wake_up(task, false);
+ } else {
+ task->jobctl &= ~JOBCTL_TRAP_FREEZE;
+ wake_up_process(task);
+ }
- rcu_read_unlock();
- mutex_unlock(&freezer_mutex);
+ unlock_task_sighand(task, &flags);
}
-/**
- * update_if_frozen - update whether a cgroup finished freezing
- * @css: css of interest
- *
- * Once FREEZING is initiated, transition to FROZEN is lazily updated by
- * calling this function. If the current state is FREEZING but not FROZEN,
- * this function checks whether all tasks of this cgroup and the descendant
- * cgroups finished freezing and, if so, sets FROZEN.
- *
- * The caller is responsible for grabbing RCU read lock and calling
- * update_if_frozen() on all descendants prior to invoking this function.
- *
- * Task states and freezer state might disagree while tasks are being
- * migrated into or out of @css, so we can't verify task states against
- * @freezer state here. See freezer_attach() for details.
+/*
+ * Freeze or unfreeze all tasks in the given cgroup.
*/
-static void update_if_frozen(struct cgroup_subsys_state *css)
+static void cgroup_do_freeze(struct cgroup *cgrp, bool freeze)
{
- struct freezer *freezer = css_freezer(css);
- struct cgroup_subsys_state *pos;
struct css_task_iter it;
struct task_struct *task;
- lockdep_assert_held(&freezer_mutex);
-
- if (!(freezer->state & CGROUP_FREEZING) ||
- (freezer->state & CGROUP_FROZEN))
- return;
+ lockdep_assert_held(&cgroup_mutex);
- /* are all (live) children frozen? */
- rcu_read_lock();
- css_for_each_child(pos, css) {
- struct freezer *child = css_freezer(pos);
-
- if ((child->state & CGROUP_FREEZER_ONLINE) &&
- !(child->state & CGROUP_FROZEN)) {
- rcu_read_unlock();
- return;
- }
- }
- rcu_read_unlock();
+ spin_lock_irq(&css_set_lock);
+ if (freeze)
+ set_bit(CGRP_FREEZE, &cgrp->flags);
+ else
+ clear_bit(CGRP_FREEZE, &cgrp->flags);
+ spin_unlock_irq(&css_set_lock);
- /* are all tasks frozen? */
- css_task_iter_start(css, 0, &it);
+ if (freeze)
+ TRACE_CGROUP_PATH(freeze, cgrp);
+ else
+ TRACE_CGROUP_PATH(unfreeze, cgrp);
+ css_task_iter_start(&cgrp->self, 0, &it);
while ((task = css_task_iter_next(&it))) {
- if (freezing(task)) {
- /*
- * freezer_should_skip() indicates that the task
- * should be skipped when determining freezing
- * completion. Consider it frozen in addition to
- * the usual frozen condition.
- */
- if (!frozen(task) && !freezer_should_skip(task))
- goto out_iter_end;
- }
- }
-
- freezer->state |= CGROUP_FROZEN;
-out_iter_end:
- css_task_iter_end(&it);
-}
-
-static int freezer_read(struct seq_file *m, void *v)
-{
- struct cgroup_subsys_state *css = seq_css(m), *pos;
-
- mutex_lock(&freezer_mutex);
- rcu_read_lock();
-
- /* update states bottom-up */
- css_for_each_descendant_post(pos, css) {
- if (!css_tryget_online(pos))
+ /*
+ * Ignore kernel threads here. Freezing cgroups containing
+ * kthreads isn't supported.
+ */
+ if (task->flags & PF_KTHREAD)
continue;
- rcu_read_unlock();
-
- update_if_frozen(pos);
-
- rcu_read_lock();
- css_put(pos);
+ cgroup_freeze_task(task, freeze);
}
-
- rcu_read_unlock();
- mutex_unlock(&freezer_mutex);
-
- seq_puts(m, freezer_state_strs(css_freezer(css)->state));
- seq_putc(m, '\n');
- return 0;
-}
-
-static void freeze_cgroup(struct freezer *freezer)
-{
- struct css_task_iter it;
- struct task_struct *task;
-
- css_task_iter_start(&freezer->css, 0, &it);
- while ((task = css_task_iter_next(&it)))
- freeze_task(task);
css_task_iter_end(&it);
-}
-static void unfreeze_cgroup(struct freezer *freezer)
-{
- struct css_task_iter it;
- struct task_struct *task;
-
- css_task_iter_start(&freezer->css, 0, &it);
- while ((task = css_task_iter_next(&it)))
- __thaw_task(task);
- css_task_iter_end(&it);
+ /*
+ * Cgroup state should be revisited here to cover empty leaf cgroups
+ * and cgroups which descendants are already in the desired state.
+ */
+ spin_lock_irq(&css_set_lock);
+ if (cgrp->nr_descendants == cgrp->freezer.nr_frozen_descendants)
+ cgroup_update_frozen(cgrp);
+ spin_unlock_irq(&css_set_lock);
}
-/**
- * freezer_apply_state - apply state change to a single cgroup_freezer
- * @freezer: freezer to apply state change to
- * @freeze: whether to freeze or unfreeze
- * @state: CGROUP_FREEZING_* flag to set or clear
- *
- * Set or clear @state on @cgroup according to @freeze, and perform
- * freezing or thawing as necessary.
+/*
+ * Adjust the task state (freeze or unfreeze) and revisit the state of
+ * source and destination cgroups.
*/
-static void freezer_apply_state(struct freezer *freezer, bool freeze,
- unsigned int state)
+void cgroup_freezer_migrate_task(struct task_struct *task,
+ struct cgroup *src, struct cgroup *dst)
{
- /* also synchronizes against task migration, see freezer_attach() */
- lockdep_assert_held(&freezer_mutex);
+ lockdep_assert_held(&css_set_lock);
- if (!(freezer->state & CGROUP_FREEZER_ONLINE))
+ /*
+ * Kernel threads are not supposed to be frozen at all.
+ */
+ if (task->flags & PF_KTHREAD)
return;
- if (freeze) {
- if (!(freezer->state & CGROUP_FREEZING))
- atomic_inc(&system_freezing_cnt);
- freezer->state |= state;
- freeze_cgroup(freezer);
- } else {
- bool was_freezing = freezer->state & CGROUP_FREEZING;
-
- freezer->state &= ~state;
-
- if (!(freezer->state & CGROUP_FREEZING)) {
- if (was_freezing)
- atomic_dec(&system_freezing_cnt);
- freezer->state &= ~CGROUP_FROZEN;
- unfreeze_cgroup(freezer);
- }
+ /*
+ * Adjust counters of freezing and frozen tasks.
+ * Note, that if the task is frozen, but the destination cgroup is not
+ * frozen, we bump both counters to keep them balanced.
+ */
+ if (task->frozen) {
+ cgroup_inc_frozen_cnt(dst);
+ cgroup_dec_frozen_cnt(src);
}
-}
-
-/**
- * freezer_change_state - change the freezing state of a cgroup_freezer
- * @freezer: freezer of interest
- * @freeze: whether to freeze or thaw
- *
- * Freeze or thaw @freezer according to @freeze. The operations are
- * recursive - all descendants of @freezer will be affected.
- */
-static void freezer_change_state(struct freezer *freezer, bool freeze)
-{
- struct cgroup_subsys_state *pos;
+ cgroup_update_frozen(dst);
+ cgroup_update_frozen(src);
/*
- * Update all its descendants in pre-order traversal. Each
- * descendant will try to inherit its parent's FREEZING state as
- * CGROUP_FREEZING_PARENT.
+ * Force the task to the desired state.
*/
- mutex_lock(&freezer_mutex);
- rcu_read_lock();
- css_for_each_descendant_pre(pos, &freezer->css) {
- struct freezer *pos_f = css_freezer(pos);
- struct freezer *parent = parent_freezer(pos_f);
-
- if (!css_tryget_online(pos))
- continue;
- rcu_read_unlock();
-
- if (pos_f == freezer)
- freezer_apply_state(pos_f, freeze,
- CGROUP_FREEZING_SELF);
- else
- freezer_apply_state(pos_f,
- parent->state & CGROUP_FREEZING,
- CGROUP_FREEZING_PARENT);
-
- rcu_read_lock();
- css_put(pos);
- }
- rcu_read_unlock();
- mutex_unlock(&freezer_mutex);
+ cgroup_freeze_task(task, test_bit(CGRP_FREEZE, &dst->flags));
}
-static ssize_t freezer_write(struct kernfs_open_file *of,
- char *buf, size_t nbytes, loff_t off)
+void cgroup_freeze(struct cgroup *cgrp, bool freeze)
{
- bool freeze;
+ struct cgroup_subsys_state *css;
+ struct cgroup *dsct;
+ bool applied = false;
- buf = strstrip(buf);
+ lockdep_assert_held(&cgroup_mutex);
- if (strcmp(buf, freezer_state_strs(0)) == 0)
- freeze = false;
- else if (strcmp(buf, freezer_state_strs(CGROUP_FROZEN)) == 0)
- freeze = true;
- else
- return -EINVAL;
+ /*
+ * Nothing changed? Just exit.
+ */
+ if (cgrp->freezer.freeze == freeze)
+ return;
- freezer_change_state(css_freezer(of_css(of)), freeze);
- return nbytes;
-}
+ cgrp->freezer.freeze = freeze;
-static u64 freezer_self_freezing_read(struct cgroup_subsys_state *css,
- struct cftype *cft)
-{
- struct freezer *freezer = css_freezer(css);
+ /*
+ * Propagate changes downwards the cgroup tree.
+ */
+ css_for_each_descendant_pre(css, &cgrp->self) {
+ dsct = css->cgroup;
- return (bool)(freezer->state & CGROUP_FREEZING_SELF);
-}
+ if (cgroup_is_dead(dsct))
+ continue;
-static u64 freezer_parent_freezing_read(struct cgroup_subsys_state *css,
- struct cftype *cft)
-{
- struct freezer *freezer = css_freezer(css);
+ if (freeze) {
+ dsct->freezer.e_freeze++;
+ /*
+ * Already frozen because of ancestor's settings?
+ */
+ if (dsct->freezer.e_freeze > 1)
+ continue;
+ } else {
+ dsct->freezer.e_freeze--;
+ /*
+ * Still frozen because of ancestor's settings?
+ */
+ if (dsct->freezer.e_freeze > 0)
+ continue;
- return (bool)(freezer->state & CGROUP_FREEZING_PARENT);
-}
+ WARN_ON_ONCE(dsct->freezer.e_freeze < 0);
+ }
+
+ /*
+ * Do change actual state: freeze or unfreeze.
+ */
+ cgroup_do_freeze(dsct, freeze);
+ applied = true;
+ }
-static struct cftype files[] = {
- {
- .name = "state",
- .flags = CFTYPE_NOT_ON_ROOT,
- .seq_show = freezer_read,
- .write = freezer_write,
- },
- {
- .name = "self_freezing",
- .flags = CFTYPE_NOT_ON_ROOT,
- .read_u64 = freezer_self_freezing_read,
- },
- {
- .name = "parent_freezing",
- .flags = CFTYPE_NOT_ON_ROOT,
- .read_u64 = freezer_parent_freezing_read,
- },
- { } /* terminate */
-};
-
-struct cgroup_subsys freezer_cgrp_subsys = {
- .css_alloc = freezer_css_alloc,
- .css_online = freezer_css_online,
- .css_offline = freezer_css_offline,
- .css_free = freezer_css_free,
- .attach = freezer_attach,
- .fork = freezer_fork,
- .legacy_cftypes = files,
-};
+ /*
+ * Even if the actual state hasn't changed, let's notify a user.
+ * The state can be enforced by an ancestor cgroup: the cgroup
+ * can already be in the desired state or it can be locked in the
+ * opposite state, so that the transition will never happen.
+ * In both cases it's better to notify a user, that there is
+ * nothing to wait for.
+ */
+ if (!applied) {
+ TRACE_CGROUP_PATH(notify_frozen, cgrp,
+ test_bit(CGRP_FROZEN, &cgrp->flags));
+ cgroup_file_notify(&cgrp->events_file);
+ }
+}
--- /dev/null
+/*
+ * cgroup_freezer.c - control group freezer subsystem
+ *
+ * Copyright IBM Corporation, 2007
+ *
+ * Author : Cedric Le Goater <clg@fr.ibm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of version 2.1 of the GNU Lesser General Public License
+ * as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it would be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+ */
+
+#include <linux/export.h>
+#include <linux/slab.h>
+#include <linux/cgroup.h>
+#include <linux/fs.h>
+#include <linux/uaccess.h>
+#include <linux/freezer.h>
+#include <linux/seq_file.h>
+#include <linux/mutex.h>
+
+/*
+ * A cgroup is freezing if any FREEZING flags are set. FREEZING_SELF is
+ * set if "FROZEN" is written to freezer.state cgroupfs file, and cleared
+ * for "THAWED". FREEZING_PARENT is set if the parent freezer is FREEZING
+ * for whatever reason. IOW, a cgroup has FREEZING_PARENT set if one of
+ * its ancestors has FREEZING_SELF set.
+ */
+enum freezer_state_flags {
+ CGROUP_FREEZER_ONLINE = (1 << 0), /* freezer is fully online */
+ CGROUP_FREEZING_SELF = (1 << 1), /* this freezer is freezing */
+ CGROUP_FREEZING_PARENT = (1 << 2), /* the parent freezer is freezing */
+ CGROUP_FROZEN = (1 << 3), /* this and its descendants frozen */
+
+ /* mask for all FREEZING flags */
+ CGROUP_FREEZING = CGROUP_FREEZING_SELF | CGROUP_FREEZING_PARENT,
+};
+
+struct freezer {
+ struct cgroup_subsys_state css;
+ unsigned int state;
+};
+
+static DEFINE_MUTEX(freezer_mutex);
+
+static inline struct freezer *css_freezer(struct cgroup_subsys_state *css)
+{
+ return css ? container_of(css, struct freezer, css) : NULL;
+}
+
+static inline struct freezer *task_freezer(struct task_struct *task)
+{
+ return css_freezer(task_css(task, freezer_cgrp_id));
+}
+
+static struct freezer *parent_freezer(struct freezer *freezer)
+{
+ return css_freezer(freezer->css.parent);
+}
+
+bool cgroup_freezing(struct task_struct *task)
+{
+ bool ret;
+
+ rcu_read_lock();
+ ret = task_freezer(task)->state & CGROUP_FREEZING;
+ rcu_read_unlock();
+
+ return ret;
+}
+
+static const char *freezer_state_strs(unsigned int state)
+{
+ if (state & CGROUP_FROZEN)
+ return "FROZEN";
+ if (state & CGROUP_FREEZING)
+ return "FREEZING";
+ return "THAWED";
+};
+
+static struct cgroup_subsys_state *
+freezer_css_alloc(struct cgroup_subsys_state *parent_css)
+{
+ struct freezer *freezer;
+
+ freezer = kzalloc(sizeof(struct freezer), GFP_KERNEL);
+ if (!freezer)
+ return ERR_PTR(-ENOMEM);
+
+ return &freezer->css;
+}
+
+/**
+ * freezer_css_online - commit creation of a freezer css
+ * @css: css being created
+ *
+ * We're committing to creation of @css. Mark it online and inherit
+ * parent's freezing state while holding both parent's and our
+ * freezer->lock.
+ */
+static int freezer_css_online(struct cgroup_subsys_state *css)
+{
+ struct freezer *freezer = css_freezer(css);
+ struct freezer *parent = parent_freezer(freezer);
+
+ mutex_lock(&freezer_mutex);
+
+ freezer->state |= CGROUP_FREEZER_ONLINE;
+
+ if (parent && (parent->state & CGROUP_FREEZING)) {
+ freezer->state |= CGROUP_FREEZING_PARENT | CGROUP_FROZEN;
+ atomic_inc(&system_freezing_cnt);
+ }
+
+ mutex_unlock(&freezer_mutex);
+ return 0;
+}
+
+/**
+ * freezer_css_offline - initiate destruction of a freezer css
+ * @css: css being destroyed
+ *
+ * @css is going away. Mark it dead and decrement system_freezing_count if
+ * it was holding one.
+ */
+static void freezer_css_offline(struct cgroup_subsys_state *css)
+{
+ struct freezer *freezer = css_freezer(css);
+
+ mutex_lock(&freezer_mutex);
+
+ if (freezer->state & CGROUP_FREEZING)
+ atomic_dec(&system_freezing_cnt);
+
+ freezer->state = 0;
+
+ mutex_unlock(&freezer_mutex);
+}
+
+static void freezer_css_free(struct cgroup_subsys_state *css)
+{
+ kfree(css_freezer(css));
+}
+
+/*
+ * Tasks can be migrated into a different freezer anytime regardless of its
+ * current state. freezer_attach() is responsible for making new tasks
+ * conform to the current state.
+ *
+ * Freezer state changes and task migration are synchronized via
+ * @freezer->lock. freezer_attach() makes the new tasks conform to the
+ * current state and all following state changes can see the new tasks.
+ */
+static void freezer_attach(struct cgroup_taskset *tset)
+{
+ struct task_struct *task;
+ struct cgroup_subsys_state *new_css;
+
+ mutex_lock(&freezer_mutex);
+
+ /*
+ * Make the new tasks conform to the current state of @new_css.
+ * For simplicity, when migrating any task to a FROZEN cgroup, we
+ * revert it to FREEZING and let update_if_frozen() determine the
+ * correct state later.
+ *
+ * Tasks in @tset are on @new_css but may not conform to its
+ * current state before executing the following - !frozen tasks may
+ * be visible in a FROZEN cgroup and frozen tasks in a THAWED one.
+ */
+ cgroup_taskset_for_each(task, new_css, tset) {
+ struct freezer *freezer = css_freezer(new_css);
+
+ if (!(freezer->state & CGROUP_FREEZING)) {
+ __thaw_task(task);
+ } else {
+ freeze_task(task);
+ /* clear FROZEN and propagate upwards */
+ while (freezer && (freezer->state & CGROUP_FROZEN)) {
+ freezer->state &= ~CGROUP_FROZEN;
+ freezer = parent_freezer(freezer);
+ }
+ }
+ }
+
+ mutex_unlock(&freezer_mutex);
+}
+
+/**
+ * freezer_fork - cgroup post fork callback
+ * @task: a task which has just been forked
+ *
+ * @task has just been created and should conform to the current state of
+ * the cgroup_freezer it belongs to. This function may race against
+ * freezer_attach(). Losing to freezer_attach() means that we don't have
+ * to do anything as freezer_attach() will put @task into the appropriate
+ * state.
+ */
+static void freezer_fork(struct task_struct *task)
+{
+ struct freezer *freezer;
+
+ /*
+ * The root cgroup is non-freezable, so we can skip locking the
+ * freezer. This is safe regardless of race with task migration.
+ * If we didn't race or won, skipping is obviously the right thing
+ * to do. If we lost and root is the new cgroup, noop is still the
+ * right thing to do.
+ */
+ if (task_css_is_root(task, freezer_cgrp_id))
+ return;
+
+ mutex_lock(&freezer_mutex);
+ rcu_read_lock();
+
+ freezer = task_freezer(task);
+ if (freezer->state & CGROUP_FREEZING)
+ freeze_task(task);
+
+ rcu_read_unlock();
+ mutex_unlock(&freezer_mutex);
+}
+
+/**
+ * update_if_frozen - update whether a cgroup finished freezing
+ * @css: css of interest
+ *
+ * Once FREEZING is initiated, transition to FROZEN is lazily updated by
+ * calling this function. If the current state is FREEZING but not FROZEN,
+ * this function checks whether all tasks of this cgroup and the descendant
+ * cgroups finished freezing and, if so, sets FROZEN.
+ *
+ * The caller is responsible for grabbing RCU read lock and calling
+ * update_if_frozen() on all descendants prior to invoking this function.
+ *
+ * Task states and freezer state might disagree while tasks are being
+ * migrated into or out of @css, so we can't verify task states against
+ * @freezer state here. See freezer_attach() for details.
+ */
+static void update_if_frozen(struct cgroup_subsys_state *css)
+{
+ struct freezer *freezer = css_freezer(css);
+ struct cgroup_subsys_state *pos;
+ struct css_task_iter it;
+ struct task_struct *task;
+
+ lockdep_assert_held(&freezer_mutex);
+
+ if (!(freezer->state & CGROUP_FREEZING) ||
+ (freezer->state & CGROUP_FROZEN))
+ return;
+
+ /* are all (live) children frozen? */
+ rcu_read_lock();
+ css_for_each_child(pos, css) {
+ struct freezer *child = css_freezer(pos);
+
+ if ((child->state & CGROUP_FREEZER_ONLINE) &&
+ !(child->state & CGROUP_FROZEN)) {
+ rcu_read_unlock();
+ return;
+ }
+ }
+ rcu_read_unlock();
+
+ /* are all tasks frozen? */
+ css_task_iter_start(css, 0, &it);
+
+ while ((task = css_task_iter_next(&it))) {
+ if (freezing(task)) {
+ /*
+ * freezer_should_skip() indicates that the task
+ * should be skipped when determining freezing
+ * completion. Consider it frozen in addition to
+ * the usual frozen condition.
+ */
+ if (!frozen(task) && !freezer_should_skip(task))
+ goto out_iter_end;
+ }
+ }
+
+ freezer->state |= CGROUP_FROZEN;
+out_iter_end:
+ css_task_iter_end(&it);
+}
+
+static int freezer_read(struct seq_file *m, void *v)
+{
+ struct cgroup_subsys_state *css = seq_css(m), *pos;
+
+ mutex_lock(&freezer_mutex);
+ rcu_read_lock();
+
+ /* update states bottom-up */
+ css_for_each_descendant_post(pos, css) {
+ if (!css_tryget_online(pos))
+ continue;
+ rcu_read_unlock();
+
+ update_if_frozen(pos);
+
+ rcu_read_lock();
+ css_put(pos);
+ }
+
+ rcu_read_unlock();
+ mutex_unlock(&freezer_mutex);
+
+ seq_puts(m, freezer_state_strs(css_freezer(css)->state));
+ seq_putc(m, '\n');
+ return 0;
+}
+
+static void freeze_cgroup(struct freezer *freezer)
+{
+ struct css_task_iter it;
+ struct task_struct *task;
+
+ css_task_iter_start(&freezer->css, 0, &it);
+ while ((task = css_task_iter_next(&it)))
+ freeze_task(task);
+ css_task_iter_end(&it);
+}
+
+static void unfreeze_cgroup(struct freezer *freezer)
+{
+ struct css_task_iter it;
+ struct task_struct *task;
+
+ css_task_iter_start(&freezer->css, 0, &it);
+ while ((task = css_task_iter_next(&it)))
+ __thaw_task(task);
+ css_task_iter_end(&it);
+}
+
+/**
+ * freezer_apply_state - apply state change to a single cgroup_freezer
+ * @freezer: freezer to apply state change to
+ * @freeze: whether to freeze or unfreeze
+ * @state: CGROUP_FREEZING_* flag to set or clear
+ *
+ * Set or clear @state on @cgroup according to @freeze, and perform
+ * freezing or thawing as necessary.
+ */
+static void freezer_apply_state(struct freezer *freezer, bool freeze,
+ unsigned int state)
+{
+ /* also synchronizes against task migration, see freezer_attach() */
+ lockdep_assert_held(&freezer_mutex);
+
+ if (!(freezer->state & CGROUP_FREEZER_ONLINE))
+ return;
+
+ if (freeze) {
+ if (!(freezer->state & CGROUP_FREEZING))
+ atomic_inc(&system_freezing_cnt);
+ freezer->state |= state;
+ freeze_cgroup(freezer);
+ } else {
+ bool was_freezing = freezer->state & CGROUP_FREEZING;
+
+ freezer->state &= ~state;
+
+ if (!(freezer->state & CGROUP_FREEZING)) {
+ if (was_freezing)
+ atomic_dec(&system_freezing_cnt);
+ freezer->state &= ~CGROUP_FROZEN;
+ unfreeze_cgroup(freezer);
+ }
+ }
+}
+
+/**
+ * freezer_change_state - change the freezing state of a cgroup_freezer
+ * @freezer: freezer of interest
+ * @freeze: whether to freeze or thaw
+ *
+ * Freeze or thaw @freezer according to @freeze. The operations are
+ * recursive - all descendants of @freezer will be affected.
+ */
+static void freezer_change_state(struct freezer *freezer, bool freeze)
+{
+ struct cgroup_subsys_state *pos;
+
+ /*
+ * Update all its descendants in pre-order traversal. Each
+ * descendant will try to inherit its parent's FREEZING state as
+ * CGROUP_FREEZING_PARENT.
+ */
+ mutex_lock(&freezer_mutex);
+ rcu_read_lock();
+ css_for_each_descendant_pre(pos, &freezer->css) {
+ struct freezer *pos_f = css_freezer(pos);
+ struct freezer *parent = parent_freezer(pos_f);
+
+ if (!css_tryget_online(pos))
+ continue;
+ rcu_read_unlock();
+
+ if (pos_f == freezer)
+ freezer_apply_state(pos_f, freeze,
+ CGROUP_FREEZING_SELF);
+ else
+ freezer_apply_state(pos_f,
+ parent->state & CGROUP_FREEZING,
+ CGROUP_FREEZING_PARENT);
+
+ rcu_read_lock();
+ css_put(pos);
+ }
+ rcu_read_unlock();
+ mutex_unlock(&freezer_mutex);
+}
+
+static ssize_t freezer_write(struct kernfs_open_file *of,
+ char *buf, size_t nbytes, loff_t off)
+{
+ bool freeze;
+
+ buf = strstrip(buf);
+
+ if (strcmp(buf, freezer_state_strs(0)) == 0)
+ freeze = false;
+ else if (strcmp(buf, freezer_state_strs(CGROUP_FROZEN)) == 0)
+ freeze = true;
+ else
+ return -EINVAL;
+
+ freezer_change_state(css_freezer(of_css(of)), freeze);
+ return nbytes;
+}
+
+static u64 freezer_self_freezing_read(struct cgroup_subsys_state *css,
+ struct cftype *cft)
+{
+ struct freezer *freezer = css_freezer(css);
+
+ return (bool)(freezer->state & CGROUP_FREEZING_SELF);
+}
+
+static u64 freezer_parent_freezing_read(struct cgroup_subsys_state *css,
+ struct cftype *cft)
+{
+ struct freezer *freezer = css_freezer(css);
+
+ return (bool)(freezer->state & CGROUP_FREEZING_PARENT);
+}
+
+static struct cftype files[] = {
+ {
+ .name = "state",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = freezer_read,
+ .write = freezer_write,
+ },
+ {
+ .name = "self_freezing",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .read_u64 = freezer_self_freezing_read,
+ },
+ {
+ .name = "parent_freezing",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .read_u64 = freezer_parent_freezing_read,
+ },
+ { } /* terminate */
+};
+
+struct cgroup_subsys freezer_cgrp_subsys = {
+ .css_alloc = freezer_css_alloc,
+ .css_online = freezer_css_online,
+ .css_offline = freezer_css_offline,
+ .css_free = freezer_css_free,
+ .attach = freezer_attach,
+ .fork = freezer_fork,
+ .legacy_cftypes = files,
+};
config ARCH_HAS_SYNC_DMA_FOR_CPU_ALL
bool
+config ARCH_HAS_DMA_PREP_COHERENT
+ bool
+
config ARCH_HAS_DMA_COHERENT_TO_PFN
bool
config DMA_REMAP
depends on MMU
+ select GENERIC_ALLOCATOR
bool
config DMA_DIRECT_REMAP
size_t size)
{
return swiotlb_force != SWIOTLB_FORCE &&
- (!dev || dma_capable(dev, dma_addr, size));
+ dma_capable(dev, dma_addr, size);
}
dma_addr_t dma_direct_map_page(struct device *dev, struct page *page,
}
EXPORT_SYMBOL_GPL(dma_get_required_mask);
-#ifndef arch_dma_alloc_attrs
-#define arch_dma_alloc_attrs(dev) (true)
-#endif
-
void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle,
gfp_t flag, unsigned long attrs)
{
const struct dma_map_ops *ops = get_dma_ops(dev);
void *cpu_addr;
- WARN_ON_ONCE(dev && !dev->coherent_dma_mask);
+ WARN_ON_ONCE(!dev->coherent_dma_mask);
if (dma_alloc_from_dev_coherent(dev, size, dma_handle, &cpu_addr))
return cpu_addr;
/* let the implementation decide on the zone to allocate from: */
flag &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM);
- if (!arch_dma_alloc_attrs(&dev))
- return NULL;
-
if (dma_is_direct(ops))
cpu_addr = dma_direct_alloc(dev, size, dma_handle, flag, attrs);
else if (ops->alloc)
int killed;
freezer_do_not_count();
+ cgroup_enter_frozen();
killed = wait_for_completion_killable(vfork);
+ cgroup_leave_frozen(false);
freezer_count();
if (killed) {
}
EXPORT_SYMBOL(is_module_sig_enforced);
+void set_module_sig_enforced(void)
+{
+ sig_enforce = true;
+}
+
/* Block module loading/unloading? */
int modules_disabled = 0;
core_param(nomodule, modules_disabled, bint, 0);
#include <linux/compiler.h>
#include <linux/posix-timers.h>
#include <linux/livepatch.h>
+#include <linux/cgroup.h>
#define CREATE_TRACE_POINTS
#include <trace/events/signal.h>
static bool recalc_sigpending_tsk(struct task_struct *t)
{
- if ((t->jobctl & JOBCTL_PENDING_MASK) ||
+ if ((t->jobctl & (JOBCTL_PENDING_MASK | JOBCTL_TRAP_FREEZE)) ||
PENDING(&t->pending, &t->blocked) ||
- PENDING(&t->signal->shared_pending, &t->blocked)) {
+ PENDING(&t->signal->shared_pending, &t->blocked) ||
+ cgroup_task_frozen(t)) {
set_tsk_thread_flag(t, TIF_SIGPENDING);
return true;
}
preempt_disable();
read_unlock(&tasklist_lock);
preempt_enable_no_resched();
+ cgroup_enter_frozen();
freezable_schedule();
} else {
/*
}
/* Now we don't run again until woken by SIGCONT or SIGKILL */
+ cgroup_enter_frozen();
freezable_schedule();
return true;
} else {
}
}
+/**
+ * do_freezer_trap - handle the freezer jobctl trap
+ *
+ * Puts the task into frozen state, if only the task is not about to quit.
+ * In this case it drops JOBCTL_TRAP_FREEZE.
+ *
+ * CONTEXT:
+ * Must be called with @current->sighand->siglock held,
+ * which is always released before returning.
+ */
+static void do_freezer_trap(void)
+ __releases(¤t->sighand->siglock)
+{
+ /*
+ * If there are other trap bits pending except JOBCTL_TRAP_FREEZE,
+ * let's make another loop to give it a chance to be handled.
+ * In any case, we'll return back.
+ */
+ if ((current->jobctl & (JOBCTL_PENDING_MASK | JOBCTL_TRAP_FREEZE)) !=
+ JOBCTL_TRAP_FREEZE) {
+ spin_unlock_irq(¤t->sighand->siglock);
+ return;
+ }
+
+ /*
+ * Now we're sure that there is no pending fatal signal and no
+ * pending traps. Clear TIF_SIGPENDING to not get out of schedule()
+ * immediately (if there is a non-fatal signal pending), and
+ * put the task into sleep.
+ */
+ __set_current_state(TASK_INTERRUPTIBLE);
+ clear_thread_flag(TIF_SIGPENDING);
+ spin_unlock_irq(¤t->sighand->siglock);
+ cgroup_enter_frozen();
+ freezable_schedule();
+}
+
static int ptrace_signal(int signr, kernel_siginfo_t *info)
{
/*
do_signal_stop(0))
goto relock;
- if (unlikely(current->jobctl & JOBCTL_TRAP_MASK)) {
- do_jobctl_trap();
+ if (unlikely(current->jobctl &
+ (JOBCTL_TRAP_MASK | JOBCTL_TRAP_FREEZE))) {
+ if (current->jobctl & JOBCTL_TRAP_MASK) {
+ do_jobctl_trap();
+ spin_unlock_irq(&sighand->siglock);
+ } else if (current->jobctl & JOBCTL_TRAP_FREEZE)
+ do_freezer_trap();
+
+ goto relock;
+ }
+
+ /*
+ * If the task is leaving the frozen state, let's update
+ * cgroup counters and reset the frozen bit.
+ */
+ if (unlikely(cgroup_task_frozen(current))) {
spin_unlock_irq(&sighand->siglock);
+ cgroup_leave_frozen(false);
goto relock;
}
fatal:
spin_unlock_irq(&sighand->siglock);
+ if (unlikely(cgroup_task_frozen(current)))
+ cgroup_leave_frozen(true);
/*
* Anything else is fatal, maybe with a core dump.
*
* PL: wq_pool_mutex protected.
*
- * PR: wq_pool_mutex protected for writes. Sched-RCU protected for reads.
+ * PR: wq_pool_mutex protected for writes. RCU protected for reads.
*
* PW: wq_pool_mutex and wq->mutex protected for writes. Either for reads.
*
* PWR: wq_pool_mutex and wq->mutex protected for writes. Either or
- * sched-RCU for reads.
+ * RCU for reads.
*
* WQ: wq->mutex protected.
*
- * WR: wq->mutex protected for writes. Sched-RCU protected for reads.
+ * WR: wq->mutex protected for writes. RCU protected for reads.
*
* MD: wq_mayday_lock protected.
*/
atomic_t nr_running ____cacheline_aligned_in_smp;
/*
- * Destruction of pool is sched-RCU protected to allow dereferences
+ * Destruction of pool is RCU protected to allow dereferences
* from get_work_pool().
*/
struct rcu_head rcu;
/*
* Release of unbound pwq is punted to system_wq. See put_pwq()
* and pwq_unbound_release_workfn() for details. pool_workqueue
- * itself is also sched-RCU protected so that the first pwq can be
+ * itself is also RCU protected so that the first pwq can be
* determined without grabbing wq->mutex.
*/
struct work_struct unbound_release_work;
char name[WQ_NAME_LEN]; /* I: workqueue name */
/*
- * Destruction of workqueue_struct is sched-RCU protected to allow
- * walking the workqueues list without grabbing wq_pool_mutex.
+ * Destruction of workqueue_struct is RCU protected to allow walking
+ * the workqueues list without grabbing wq_pool_mutex.
* This is used to dump all workqueues from sysrq.
*/
struct rcu_head rcu;
#include <trace/events/workqueue.h>
#define assert_rcu_or_pool_mutex() \
- RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held() && \
+ RCU_LOCKDEP_WARN(!rcu_read_lock_held() && \
!lockdep_is_held(&wq_pool_mutex), \
- "sched RCU or wq_pool_mutex should be held")
+ "RCU or wq_pool_mutex should be held")
#define assert_rcu_or_wq_mutex(wq) \
- RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held() && \
+ RCU_LOCKDEP_WARN(!rcu_read_lock_held() && \
!lockdep_is_held(&wq->mutex), \
- "sched RCU or wq->mutex should be held")
+ "RCU or wq->mutex should be held")
#define assert_rcu_or_wq_mutex_or_pool_mutex(wq) \
- RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held() && \
+ RCU_LOCKDEP_WARN(!rcu_read_lock_held() && \
!lockdep_is_held(&wq->mutex) && \
!lockdep_is_held(&wq_pool_mutex), \
- "sched RCU, wq->mutex or wq_pool_mutex should be held")
+ "RCU, wq->mutex or wq_pool_mutex should be held")
#define for_each_cpu_worker_pool(pool, cpu) \
for ((pool) = &per_cpu(cpu_worker_pools, cpu)[0]; \
* @pool: iteration cursor
* @pi: integer used for iteration
*
- * This must be called either with wq_pool_mutex held or sched RCU read
+ * This must be called either with wq_pool_mutex held or RCU read
* locked. If the pool needs to be used beyond the locking in effect, the
* caller is responsible for guaranteeing that the pool stays online.
*
* @pwq: iteration cursor
* @wq: the target workqueue
*
- * This must be called either with wq->mutex held or sched RCU read locked.
+ * This must be called either with wq->mutex held or RCU read locked.
* If the pwq needs to be used beyond the locking in effect, the caller is
* responsible for guaranteeing that the pwq stays online.
*
* @wq: the target workqueue
* @node: the node ID
*
- * This must be called with any of wq_pool_mutex, wq->mutex or sched RCU
+ * This must be called with any of wq_pool_mutex, wq->mutex or RCU
* read locked.
* If the pwq needs to be used beyond the locking in effect, the caller is
* responsible for guaranteeing that the pwq stays online.
* @work: the work item of interest
*
* Pools are created and destroyed under wq_pool_mutex, and allows read
- * access under sched-RCU read lock. As such, this function should be
- * called under wq_pool_mutex or with preemption disabled.
+ * access under RCU read lock. As such, this function should be
+ * called under wq_pool_mutex or inside of a rcu_read_lock() region.
*
* All fields of the returned pool are accessible as long as the above
* mentioned locking is in effect. If the returned pool needs to be used
/**
* wq_worker_last_func - retrieve worker's last work function
+ * @task: Task to retrieve last work function of.
*
* Determine the last function a worker executed. This is called from
* the scheduler to get a worker's last known identity.
{
if (pwq) {
/*
- * As both pwqs and pools are sched-RCU protected, the
+ * As both pwqs and pools are RCU protected, the
* following lock operations are safe.
*/
spin_lock_irq(&pwq->pool->lock);
if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)))
return 0;
+ rcu_read_lock();
/*
* The queueing is in progress, or it is already queued. Try to
* steal it from ->worklist without clearing WORK_STRUCT_PENDING.
set_work_pool_and_keep_pending(work, pool->id);
spin_unlock(&pool->lock);
+ rcu_read_unlock();
return 1;
}
spin_unlock(&pool->lock);
fail:
+ rcu_read_unlock();
local_irq_restore(*flags);
if (work_is_canceling(work))
return -ENOENT;
if (unlikely(wq->flags & __WQ_DRAINING) &&
WARN_ON_ONCE(!is_chained_work(wq)))
return;
+ rcu_read_lock();
retry:
if (req_cpu == WORK_CPU_UNBOUND)
cpu = wq_select_unbound_cpu(raw_smp_processor_id());
/* pwq determined, queue */
trace_workqueue_queue_work(req_cpu, pwq, work);
- if (WARN_ON(!list_empty(&work->entry))) {
- spin_unlock(&pwq->pool->lock);
- return;
- }
+ if (WARN_ON(!list_empty(&work->entry)))
+ goto out;
pwq->nr_in_flight[pwq->work_color]++;
work_flags = work_color_to_flags(pwq->work_color);
insert_work(pwq, work, worklist, work_flags);
+out:
spin_unlock(&pwq->pool->lock);
+ rcu_read_unlock();
}
/**
might_sleep();
- local_irq_disable();
+ rcu_read_lock();
pool = get_work_pool(work);
if (!pool) {
- local_irq_enable();
+ rcu_read_unlock();
return false;
}
- spin_lock(&pool->lock);
+ spin_lock_irq(&pool->lock);
/* see the comment in try_to_grab_pending() with the same code */
pwq = get_work_pwq(work);
if (pwq) {
lock_map_acquire(&pwq->wq->lockdep_map);
lock_map_release(&pwq->wq->lockdep_map);
}
-
+ rcu_read_unlock();
return true;
already_gone:
spin_unlock_irq(&pool->lock);
+ rcu_read_unlock();
return false;
}
* put_unbound_pool - put a worker_pool
* @pool: worker_pool to put
*
- * Put @pool. If its refcnt reaches zero, it gets destroyed in sched-RCU
+ * Put @pool. If its refcnt reaches zero, it gets destroyed in RCU
* safe manner. get_unbound_pool() calls this function on its failure path
* and this function should be able to release pools which went through,
* successfully or not, init_worker_pool().
del_timer_sync(&pool->idle_timer);
del_timer_sync(&pool->mayday_timer);
- /* sched-RCU protected to allow dereferences from get_work_pool() */
+ /* RCU protected to allow dereferences from get_work_pool() */
call_rcu(&pool->rcu, rcu_free_pool);
}
return 0;
}
+__printf(1, 4)
struct workqueue_struct *alloc_workqueue(const char *fmt,
unsigned int flags,
int max_active, ...)
struct pool_workqueue *pwq;
bool ret;
- rcu_read_lock_sched();
+ rcu_read_lock();
+ preempt_disable();
if (cpu == WORK_CPU_UNBOUND)
cpu = smp_processor_id();
pwq = unbound_pwq_by_node(wq, cpu_to_node(cpu));
ret = !list_empty(&pwq->delayed_works);
- rcu_read_unlock_sched();
+ preempt_enable();
+ rcu_read_unlock();
return ret;
}
if (work_pending(work))
ret |= WORK_BUSY_PENDING;
- local_irq_save(flags);
+ rcu_read_lock();
pool = get_work_pool(work);
if (pool) {
- spin_lock(&pool->lock);
+ spin_lock_irqsave(&pool->lock, flags);
if (find_worker_executing_work(pool, work))
ret |= WORK_BUSY_RUNNING;
- spin_unlock(&pool->lock);
+ spin_unlock_irqrestore(&pool->lock, flags);
}
- local_irq_restore(flags);
+ rcu_read_unlock();
return ret;
}
unsigned long flags;
int pi;
- rcu_read_lock_sched();
+ rcu_read_lock();
pr_info("Showing busy workqueues and worker pools:\n");
touch_nmi_watchdog();
}
- rcu_read_unlock_sched();
+ rcu_read_unlock();
}
/* used to show worker information through /proc/PID/{comm,stat,status} */
* nr_active is monotonically decreasing. It's safe
* to peek without lock.
*/
- rcu_read_lock_sched();
+ rcu_read_lock();
for_each_pwq(pwq, wq) {
WARN_ON_ONCE(pwq->nr_active < 0);
if (pwq->nr_active) {
busy = true;
- rcu_read_unlock_sched();
+ rcu_read_unlock();
goto out_unlock;
}
}
- rcu_read_unlock_sched();
+ rcu_read_unlock();
}
out_unlock:
mutex_unlock(&wq_pool_mutex);
const char *delim = "";
int node, written = 0;
- rcu_read_lock_sched();
+ get_online_cpus();
+ rcu_read_lock();
for_each_node(node) {
written += scnprintf(buf + written, PAGE_SIZE - written,
"%s%d:%d", delim, node,
delim = " ";
}
written += scnprintf(buf + written, PAGE_SIZE - written, "\n");
- rcu_read_unlock_sched();
+ rcu_read_unlock();
+ put_online_cpus();
return written;
}
#include <linux/device.h>
#include <linux/netdevice.h>
+#include <rdma/ib_verbs.h>
+
extern struct _ddebug __start___verbose[];
extern struct _ddebug __stop___verbose[];
#endif
+#if IS_ENABLED(CONFIG_INFINIBAND)
+
+void __dynamic_ibdev_dbg(struct _ddebug *descriptor,
+ const struct ib_device *ibdev, const char *fmt, ...)
+{
+ struct va_format vaf;
+ va_list args;
+
+ va_start(args, fmt);
+
+ vaf.fmt = fmt;
+ vaf.va = &args;
+
+ if (ibdev && ibdev->dev.parent) {
+ char buf[PREFIX_SIZE];
+
+ dev_printk_emit(LOGLEVEL_DEBUG, ibdev->dev.parent,
+ "%s%s %s %s: %pV",
+ dynamic_emit_prefix(descriptor, buf),
+ dev_driver_string(ibdev->dev.parent),
+ dev_name(ibdev->dev.parent),
+ dev_name(&ibdev->dev),
+ &vaf);
+ } else if (ibdev) {
+ printk(KERN_DEBUG "%s: %pV", dev_name(&ibdev->dev), &vaf);
+ } else {
+ printk(KERN_DEBUG "(NULL ib_device): %pV", &vaf);
+ }
+
+ va_end(args);
+}
+EXPORT_SYMBOL(__dynamic_ibdev_dbg);
+
+#endif
+
#define DDEBUG_STRING_SIZE 1024
static __initdata char ddebug_setup_string[DDEBUG_STRING_SIZE];
static int smc_ib_fill_mac(struct smc_ib_device *smcibdev, u8 ibport)
{
const struct ib_gid_attr *attr;
- int rc = 0;
+ int rc;
attr = rdma_get_gid_attr(smcibdev->ibdev, ibport, 0);
if (IS_ERR(attr))
return -ENODEV;
- if (attr->ndev)
- memcpy(smcibdev->mac[ibport - 1], attr->ndev->dev_addr,
- ETH_ALEN);
- else
- rc = -ENODEV;
-
+ rc = rdma_read_gid_l2_fields(attr, NULL, smcibdev->mac[ibport - 1]);
rdma_put_gid_attr(attr);
return rc;
}
unsigned short vlan_id, u8 gid[], u8 *sgid_index)
{
const struct ib_gid_attr *attr;
+ const struct net_device *ndev;
int i;
for (i = 0; i < smcibdev->pattr[ibport - 1].gid_tbl_len; i++) {
if (IS_ERR(attr))
continue;
- if (attr->ndev &&
+ rcu_read_lock();
+ ndev = rdma_read_gid_attr_ndev_rcu(attr);
+ if (!IS_ERR(ndev) &&
((!vlan_id && !is_vlan_dev(attr->ndev)) ||
(vlan_id && is_vlan_dev(attr->ndev) &&
vlan_dev_vlan_id(attr->ndev) == vlan_id)) &&
attr->gid_type == IB_GID_TYPE_ROCE) {
+ rcu_read_unlock();
if (gid)
memcpy(gid, &attr->gid, SMC_GID_SIZE);
if (sgid_index)
rdma_put_gid_attr(attr);
return 0;
}
+ rcu_read_unlock();
rdma_put_gid_attr(attr);
}
return -ENODEV;
struct gss_upcall_msg {
refcount_t count;
kuid_t uid;
+ const char *service_name;
struct rpc_pipe_msg msg;
struct list_head list;
struct gss_auth *auth;
gss_put_ctx(gss_msg->ctx);
rpc_destroy_wait_queue(&gss_msg->rpc_waitqueue);
gss_put_auth(gss_msg->auth);
+ kfree_const(gss_msg->service_name);
kfree(gss_msg);
}
gss_release_msg(gss_msg);
}
-static void gss_encode_v0_msg(struct gss_upcall_msg *gss_msg)
+static void gss_encode_v0_msg(struct gss_upcall_msg *gss_msg,
+ const struct cred *cred)
{
- uid_t uid = from_kuid(&init_user_ns, gss_msg->uid);
+ struct user_namespace *userns = cred->user_ns;
+
+ uid_t uid = from_kuid_munged(userns, gss_msg->uid);
memcpy(gss_msg->databuf, &uid, sizeof(uid));
gss_msg->msg.data = gss_msg->databuf;
gss_msg->msg.len = sizeof(uid);
BUILD_BUG_ON(sizeof(uid) > sizeof(gss_msg->databuf));
}
+static ssize_t
+gss_v0_upcall(struct file *file, struct rpc_pipe_msg *msg,
+ char __user *buf, size_t buflen)
+{
+ struct gss_upcall_msg *gss_msg = container_of(msg,
+ struct gss_upcall_msg,
+ msg);
+ if (msg->copied == 0)
+ gss_encode_v0_msg(gss_msg, file->f_cred);
+ return rpc_pipe_generic_upcall(file, msg, buf, buflen);
+}
+
static int gss_encode_v1_msg(struct gss_upcall_msg *gss_msg,
const char *service_name,
- const char *target_name)
+ const char *target_name,
+ const struct cred *cred)
{
+ struct user_namespace *userns = cred->user_ns;
struct gss_api_mech *mech = gss_msg->auth->mech;
char *p = gss_msg->databuf;
size_t buflen = sizeof(gss_msg->databuf);
int len;
len = scnprintf(p, buflen, "mech=%s uid=%d", mech->gm_name,
- from_kuid(&init_user_ns, gss_msg->uid));
+ from_kuid_munged(userns, gss_msg->uid));
buflen -= len;
p += len;
gss_msg->msg.len = len;
return -ENOMEM;
}
+static ssize_t
+gss_v1_upcall(struct file *file, struct rpc_pipe_msg *msg,
+ char __user *buf, size_t buflen)
+{
+ struct gss_upcall_msg *gss_msg = container_of(msg,
+ struct gss_upcall_msg,
+ msg);
+ int err;
+ if (msg->copied == 0) {
+ err = gss_encode_v1_msg(gss_msg,
+ gss_msg->service_name,
+ gss_msg->auth->target_name,
+ file->f_cred);
+ if (err)
+ return err;
+ }
+ return rpc_pipe_generic_upcall(file, msg, buf, buflen);
+}
+
static struct gss_upcall_msg *
gss_alloc_msg(struct gss_auth *gss_auth,
kuid_t uid, const char *service_name)
refcount_set(&gss_msg->count, 1);
gss_msg->uid = uid;
gss_msg->auth = gss_auth;
- switch (vers) {
- case 0:
- gss_encode_v0_msg(gss_msg);
- break;
- default:
- err = gss_encode_v1_msg(gss_msg, service_name, gss_auth->target_name);
- if (err)
+ kref_get(&gss_auth->kref);
+ if (service_name) {
+ gss_msg->service_name = kstrdup_const(service_name, GFP_NOFS);
+ if (!gss_msg->service_name) {
+ err = -ENOMEM;
goto err_put_pipe_version;
+ }
}
- kref_get(&gss_auth->kref);
return gss_msg;
err_put_pipe_version:
put_pipe_version(gss_auth->net);
/* XXX: warning on the first, under the assumption we
* shouldn't normally hit this case on a refresh. */
warn_gssd();
- task->tk_timeout = 15*HZ;
- rpc_sleep_on(&pipe_version_rpc_waitqueue, task, NULL);
+ rpc_sleep_on_timeout(&pipe_version_rpc_waitqueue,
+ task, NULL, jiffies + (15 * HZ));
err = -EAGAIN;
goto out;
}
if (gss_cred->gc_upcall != NULL)
rpc_sleep_on(&gss_cred->gc_upcall->rpc_waitqueue, task, NULL);
else if (gss_msg->ctx == NULL && gss_msg->msg.errno >= 0) {
- task->tk_timeout = 0;
gss_cred->gc_upcall = gss_msg;
/* gss_upcall_callback will release the reference to gss_upcall_msg */
refcount_inc(&gss_msg->count);
goto err;
}
- uid = make_kuid(&init_user_ns, id);
+ uid = make_kuid(current_user_ns(), id);
if (!uid_valid(uid)) {
err = -EINVAL;
goto err;
};
static const struct rpc_pipe_ops gss_upcall_ops_v0 = {
- .upcall = rpc_pipe_generic_upcall,
+ .upcall = gss_v0_upcall,
.downcall = gss_pipe_downcall,
.destroy_msg = gss_pipe_destroy_msg,
.open_pipe = gss_pipe_open_v0,
};
static const struct rpc_pipe_ops gss_upcall_ops_v1 = {
- .upcall = rpc_pipe_generic_upcall,
+ .upcall = gss_v1_upcall,
.downcall = gss_pipe_downcall,
.destroy_msg = gss_pipe_destroy_msg,
.open_pipe = gss_pipe_open_v1,
__be32 *p, *cred_len, *gidarr_len;
int i;
struct group_info *gi = cred->cr_cred->group_info;
+ struct user_namespace *userns = clnt->cl_cred ?
+ clnt->cl_cred->user_ns : &init_user_ns;
/* Credential */
p = xdr_reserve_space(xdr, 3 * sizeof(*p));
if (!p)
goto marshal_failed;
- *p++ = cpu_to_be32(from_kuid(&init_user_ns, cred->cr_cred->fsuid));
- *p++ = cpu_to_be32(from_kgid(&init_user_ns, cred->cr_cred->fsgid));
+ *p++ = cpu_to_be32(from_kuid_munged(userns, cred->cr_cred->fsuid));
+ *p++ = cpu_to_be32(from_kgid_munged(userns, cred->cr_cred->fsgid));
gidarr_len = p++;
if (gi)
for (i = 0; i < UNX_NGROUPS && i < gi->ngroups; i++)
- *p++ = cpu_to_be32(from_kgid(&init_user_ns,
- gi->gid[i]));
+ *p++ = cpu_to_be32(from_kgid_munged(userns, gi->gid[i]));
*gidarr_len = cpu_to_be32(p - gidarr_len - 1);
*cred_len = cpu_to_be32((p - cred_len - 1) << 2);
p = xdr_reserve_space(xdr, (p - gidarr_len - 1) << 2);
if (err)
goto out_no_clid;
+ clnt->cl_cred = get_cred(args->cred);
clnt->cl_procinfo = version->procs;
clnt->cl_maxproc = version->nrprocs;
clnt->cl_prog = args->prognumber ? : program->number;
out_no_path:
rpc_free_iostats(clnt->cl_metrics);
out_no_stats:
+ put_cred(clnt->cl_cred);
rpc_free_clid(clnt);
out_no_clid:
kfree(clnt);
}
clnt->cl_softrtry = 1;
- if (args->flags & RPC_CLNT_CREATE_HARDRTRY)
+ if (args->flags & (RPC_CLNT_CREATE_HARDRTRY|RPC_CLNT_CREATE_SOFTERR)) {
clnt->cl_softrtry = 0;
+ if (args->flags & RPC_CLNT_CREATE_SOFTERR)
+ clnt->cl_softerr = 1;
+ }
if (args->flags & RPC_CLNT_CREATE_AUTOBIND)
clnt->cl_autobind = 1;
/* Turn off autobind on clones */
new->cl_autobind = 0;
new->cl_softrtry = clnt->cl_softrtry;
+ new->cl_softerr = clnt->cl_softerr;
new->cl_noretranstimeo = clnt->cl_noretranstimeo;
new->cl_discrtry = clnt->cl_discrtry;
new->cl_chatty = clnt->cl_chatty;
new->cl_principal = clnt->cl_principal;
+ new->cl_cred = get_cred(clnt->cl_cred);
return new;
out_err:
.prognumber = clnt->cl_prog,
.version = clnt->cl_vers,
.authflavor = clnt->cl_auth->au_flavor,
+ .cred = clnt->cl_cred,
};
return __rpc_clone_client(&args, clnt);
}
.prognumber = clnt->cl_prog,
.version = clnt->cl_vers,
.authflavor = flavor,
+ .cred = clnt->cl_cred,
};
return __rpc_clone_client(&args, clnt);
}
* Spin lock all_tasks to prevent changes...
*/
spin_lock(&clnt->cl_lock);
- list_for_each_entry(rovr, &clnt->cl_tasks, tk_task) {
- if (!RPC_IS_ACTIVATED(rovr))
- continue;
- if (!(rovr->tk_flags & RPC_TASK_KILLED)) {
- rovr->tk_flags |= RPC_TASK_KILLED;
- rpc_exit(rovr, -EIO);
- }
- }
+ list_for_each_entry(rovr, &clnt->cl_tasks, tk_task)
+ rpc_signal_task(rovr);
spin_unlock(&clnt->cl_lock);
}
EXPORT_SYMBOL_GPL(rpc_killall_tasks);
xprt_put(rcu_dereference_raw(clnt->cl_xprt));
xprt_iter_destroy(&clnt->cl_xpi);
rpciod_down();
+ put_cred(clnt->cl_cred);
rpc_free_clid(clnt);
kfree(clnt);
return parent;
.prognumber = program->number,
.version = vers,
.authflavor = old->cl_auth->au_flavor,
+ .cred = old->cl_cred,
};
struct rpc_clnt *clnt;
int err;
atomic_inc(&clnt->cl_count);
if (clnt->cl_softrtry)
task->tk_flags |= RPC_TASK_SOFT;
+ if (clnt->cl_softerr)
+ task->tk_flags |= RPC_TASK_TIMEOUT;
if (clnt->cl_noretranstimeo)
task->tk_flags |= RPC_TASK_NO_RETRANS_TIMEOUT;
if (atomic_read(&clnt->cl_swapper))
}
EXPORT_SYMBOL_GPL(rpc_force_rebind);
-/*
- * Restart an (async) RPC call from the call_prepare state.
- * Usually called from within the exit handler.
- */
-int
-rpc_restart_call_prepare(struct rpc_task *task)
+static int
+__rpc_restart_call(struct rpc_task *task, void (*action)(struct rpc_task *))
{
- if (RPC_ASSASSINATED(task))
- return 0;
- task->tk_action = call_start;
task->tk_status = 0;
- if (task->tk_ops->rpc_call_prepare != NULL)
- task->tk_action = rpc_prepare_task;
+ task->tk_rpc_status = 0;
+ task->tk_action = action;
return 1;
}
-EXPORT_SYMBOL_GPL(rpc_restart_call_prepare);
/*
* Restart an (async) RPC call. Usually called from within the
int
rpc_restart_call(struct rpc_task *task)
{
- if (RPC_ASSASSINATED(task))
- return 0;
- task->tk_action = call_start;
- task->tk_status = 0;
- return 1;
+ return __rpc_restart_call(task, call_start);
}
EXPORT_SYMBOL_GPL(rpc_restart_call);
+/*
+ * Restart an (async) RPC call from the call_prepare state.
+ * Usually called from within the exit handler.
+ */
+int
+rpc_restart_call_prepare(struct rpc_task *task)
+{
+ if (task->tk_ops->rpc_call_prepare != NULL)
+ return __rpc_restart_call(task, rpc_prepare_task);
+ return rpc_restart_call(task);
+}
+EXPORT_SYMBOL_GPL(rpc_restart_call_prepare);
+
const char
*rpc_proc_name(const struct rpc_task *task)
{
return "no proc";
}
+static void
+__rpc_call_rpcerror(struct rpc_task *task, int tk_status, int rpc_status)
+{
+ task->tk_rpc_status = rpc_status;
+ rpc_exit(task, tk_status);
+}
+
+static void
+rpc_call_rpcerror(struct rpc_task *task, int status)
+{
+ __rpc_call_rpcerror(task, status, status);
+}
+
/*
* 0. Initial state
*
printk(KERN_ERR "%s: status=%d, but no request slot, exiting\n",
__func__, status);
- rpc_exit(task, -EIO);
+ rpc_call_rpcerror(task, -EIO);
return;
}
__func__, status);
break;
}
- rpc_exit(task, status);
+ rpc_call_rpcerror(task, status);
}
/*
}
dprintk("RPC: %5u %s: refresh creds failed with error %d\n",
task->tk_pid, __func__, status);
- rpc_exit(task, status);
+ rpc_call_rpcerror(task, status);
}
/*
if (status == 0)
return;
if (status != -ENOMEM) {
- rpc_exit(task, status);
+ rpc_call_rpcerror(task, status);
return;
}
rpc_delay(task, HZ >> 4);
break;
case -EKEYEXPIRED:
- task->tk_action = call_refresh;
+ if (!task->tk_cred_retry) {
+ rpc_exit(task, task->tk_status);
+ } else {
+ task->tk_action = call_refresh;
+ task->tk_cred_retry--;
+ dprintk("RPC: %5u %s: retry refresh creds\n",
+ task->tk_pid, __func__);
+ }
break;
default:
- rpc_exit(task, task->tk_status);
+ rpc_call_rpcerror(task, task->tk_status);
}
return;
} else {
if (!xprt_prepare_transmit(task))
return;
- task->tk_timeout = xprt->bind_timeout;
xprt->ops->rpcbind(task);
}
task->tk_pid, -task->tk_status);
}
- rpc_exit(task, status);
+ rpc_call_rpcerror(task, status);
return;
retry_timeout:
if (task->tk_status < 0)
return;
if (task->tk_flags & RPC_TASK_NOCONNECT) {
- rpc_exit(task, -ENOTCONN);
+ rpc_call_rpcerror(task, -ENOTCONN);
return;
}
if (!xprt_prepare_transmit(task))
task->tk_action = call_transmit;
return;
}
- rpc_exit(task, status);
+ rpc_call_rpcerror(task, status);
return;
out_retry:
/* Check for timeouts before looping back to call_bind */
if (!task->tk_msg.rpc_proc->p_proc)
trace_xprt_ping(task->tk_xprt,
task->tk_status);
- rpc_exit(task, task->tk_status);
+ rpc_call_rpcerror(task, task->tk_status);
return;
}
/* fall through */
rpc_check_timeout(task);
return;
out_exit:
- rpc_exit(task, status);
+ rpc_call_rpcerror(task, status);
}
static bool
task->tk_timeouts++;
if (RPC_IS_SOFTCONN(task) && !rpc_check_connected(task->tk_rqstp)) {
- rpc_exit(task, -ETIMEDOUT);
+ rpc_call_rpcerror(task, -ETIMEDOUT);
return;
}
if (RPC_IS_SOFT(task)) {
+ /*
+ * Once a "no retrans timeout" soft tasks (a.k.a NFSv4) has
+ * been sent, it should time out only if the transport
+ * connection gets terminally broken.
+ */
+ if ((task->tk_flags & RPC_TASK_NO_RETRANS_TIMEOUT) &&
+ rpc_check_connected(task->tk_rqstp))
+ return;
+
if (clnt->cl_chatty) {
- printk(KERN_NOTICE "%s: server %s not responding, timed out\n",
+ pr_notice_ratelimited(
+ "%s: server %s not responding, timed out\n",
clnt->cl_program->name,
task->tk_xprt->servername);
}
if (task->tk_flags & RPC_TASK_TIMEOUT)
- rpc_exit(task, -ETIMEDOUT);
+ rpc_call_rpcerror(task, -ETIMEDOUT);
else
- rpc_exit(task, -EIO);
+ __rpc_call_rpcerror(task, -EIO, -ETIMEDOUT);
return;
}
if (!(task->tk_flags & RPC_CALL_MAJORSEEN)) {
task->tk_flags |= RPC_CALL_MAJORSEEN;
if (clnt->cl_chatty) {
- printk(KERN_NOTICE "%s: server %s not responding, still trying\n",
- clnt->cl_program->name,
- task->tk_xprt->servername);
+ pr_notice_ratelimited(
+ "%s: server %s not responding, still trying\n",
+ clnt->cl_program->name,
+ task->tk_xprt->servername);
}
}
rpc_force_rebind(clnt);
if (task->tk_flags & RPC_CALL_MAJORSEEN) {
if (clnt->cl_chatty) {
- printk(KERN_NOTICE "%s: server %s OK\n",
+ pr_notice_ratelimited("%s: server %s OK\n",
clnt->cl_program->name,
task->tk_xprt->servername);
}
printk(KERN_INFO "%5u %04x %6d %8p %8p %8ld %8p %sv%u %s a:%ps q:%s\n",
task->tk_pid, task->tk_flags, task->tk_status,
- clnt, task->tk_rqstp, task->tk_timeout, task->tk_ops,
+ clnt, task->tk_rqstp, rpc_task_timeout(task), task->tk_ops,
clnt->cl_program->name, clnt->cl_vers, rpc_proc_name(task),
task->tk_action, rpc_waitq);
}
seq_printf(f, "%5u %04x %6d 0x%x 0x%x %8ld %ps %sv%u %s a:%ps q:%s\n",
task->tk_pid, task->tk_flags, task->tk_status,
- clnt->cl_clid, xid, task->tk_timeout, task->tk_ops,
+ clnt->cl_clid, xid, rpc_task_timeout(task), task->tk_ops,
clnt->cl_program->name, clnt->cl_vers, rpc_proc_name(task),
task->tk_action, rpc_waitq);
return 0;
.program = &rpcb_program,
.version = RPCBVERS_2,
.authflavor = RPC_AUTH_NULL,
+ .cred = current_cred(),
/*
* We turn off the idle timeout to prevent the kernel
* from automatically disconnecting the socket.
.program = &rpcb_program,
.version = RPCBVERS_2,
.authflavor = RPC_AUTH_UNIX,
+ .cred = current_cred(),
.flags = RPC_CLNT_CREATE_NOPING,
};
struct rpc_clnt *clnt, *clnt4;
static struct rpc_clnt *rpcb_create(struct net *net, const char *nodename,
const char *hostname,
struct sockaddr *srvaddr, size_t salen,
- int proto, u32 version)
+ int proto, u32 version,
+ const struct cred *cred)
{
struct rpc_create_args args = {
.net = net,
.program = &rpcb_program,
.version = version,
.authflavor = RPC_AUTH_UNIX,
+ .cred = cred,
.flags = (RPC_CLNT_CREATE_NOPING |
RPC_CLNT_CREATE_NONPRIVPORT),
};
/* Put self on the wait queue to ensure we get notified if
* some other task is already attempting to bind the port */
- rpc_sleep_on(&xprt->binding, task, NULL);
+ rpc_sleep_on_timeout(&xprt->binding, task,
+ NULL, jiffies + xprt->bind_timeout);
if (xprt_test_and_set_binding(xprt)) {
dprintk("RPC: %5u %s: waiting for another binder\n",
rpcb_clnt = rpcb_create(xprt->xprt_net,
clnt->cl_nodename,
xprt->servername, sap, salen,
- xprt->prot, bind_version);
+ xprt->prot, bind_version,
+ clnt->cl_cred);
if (IS_ERR(rpcb_clnt)) {
status = PTR_ERR(rpcb_clnt);
dprintk("RPC: %5u %s: rpcb_create failed, error %ld\n",
struct workqueue_struct *rpciod_workqueue __read_mostly;
struct workqueue_struct *xprtiod_workqueue __read_mostly;
+unsigned long
+rpc_task_timeout(const struct rpc_task *task)
+{
+ unsigned long timeout = READ_ONCE(task->tk_timeout);
+
+ if (timeout != 0) {
+ unsigned long now = jiffies;
+ if (time_before(now, timeout))
+ return timeout - now;
+ }
+ return 0;
+}
+EXPORT_SYMBOL_GPL(rpc_task_timeout);
+
/*
* Disable the timer for a given RPC task. Should be called with
* queue->lock and bh_disabled in order to avoid races within
static void
__rpc_disable_timer(struct rpc_wait_queue *queue, struct rpc_task *task)
{
- if (task->tk_timeout == 0)
+ if (list_empty(&task->u.tk_wait.timer_list))
return;
dprintk("RPC: %5u disabling timer\n", task->tk_pid);
task->tk_timeout = 0;
static void
rpc_set_queue_timer(struct rpc_wait_queue *queue, unsigned long expires)
{
- queue->timer_list.expires = expires;
- mod_timer(&queue->timer_list.timer, expires);
+ timer_reduce(&queue->timer_list.timer, expires);
}
/*
* Set up a timer for the current task.
*/
static void
-__rpc_add_timer(struct rpc_wait_queue *queue, struct rpc_task *task)
+__rpc_add_timer(struct rpc_wait_queue *queue, struct rpc_task *task,
+ unsigned long timeout)
{
- if (!task->tk_timeout)
- return;
-
dprintk("RPC: %5u setting alarm for %u ms\n",
- task->tk_pid, jiffies_to_msecs(task->tk_timeout));
+ task->tk_pid, jiffies_to_msecs(timeout - jiffies));
- task->u.tk_wait.expires = jiffies + task->tk_timeout;
- if (list_empty(&queue->timer_list.list) || time_before(task->u.tk_wait.expires, queue->timer_list.expires))
- rpc_set_queue_timer(queue, task->u.tk_wait.expires);
+ task->tk_timeout = timeout;
+ rpc_set_queue_timer(queue, timeout);
list_add(&task->u.tk_wait.timer_list, &queue->timer_list.list);
}
if (RPC_IS_QUEUED(task))
return;
+ INIT_LIST_HEAD(&task->u.tk_wait.timer_list);
if (RPC_IS_PRIORITY(queue))
__rpc_add_wait_queue_priority(queue, task, queue_priority);
else if (RPC_IS_SWAPPER(task))
queue->maxpriority = nr_queues - 1;
rpc_reset_waitqueue_priority(queue);
queue->qlen = 0;
- timer_setup(&queue->timer_list.timer, __rpc_queue_timer_fn, 0);
+ timer_setup(&queue->timer_list.timer,
+ __rpc_queue_timer_fn,
+ TIMER_DEFERRABLE);
INIT_LIST_HEAD(&queue->timer_list.list);
rpc_assign_waitqueue_name(queue, qname);
}
*/
static void __rpc_sleep_on_priority(struct rpc_wait_queue *q,
struct rpc_task *task,
- rpc_action action,
unsigned char queue_priority)
{
dprintk("RPC: %5u sleep_on(queue \"%s\" time %lu)\n",
__rpc_add_wait_queue(q, task, queue_priority);
- WARN_ON_ONCE(task->tk_callback != NULL);
- task->tk_callback = action;
- __rpc_add_timer(q, task);
}
-void rpc_sleep_on(struct rpc_wait_queue *q, struct rpc_task *task,
- rpc_action action)
+static void __rpc_sleep_on_priority_timeout(struct rpc_wait_queue *q,
+ struct rpc_task *task, unsigned long timeout,
+ unsigned char queue_priority)
+{
+ if (time_is_after_jiffies(timeout)) {
+ __rpc_sleep_on_priority(q, task, queue_priority);
+ __rpc_add_timer(q, task, timeout);
+ } else
+ task->tk_status = -ETIMEDOUT;
+}
+
+static void rpc_set_tk_callback(struct rpc_task *task, rpc_action action)
+{
+ if (action && !WARN_ON_ONCE(task->tk_callback != NULL))
+ task->tk_callback = action;
+}
+
+static bool rpc_sleep_check_activated(struct rpc_task *task)
{
/* We shouldn't ever put an inactive task to sleep */
- WARN_ON_ONCE(!RPC_IS_ACTIVATED(task));
- if (!RPC_IS_ACTIVATED(task)) {
+ if (WARN_ON_ONCE(!RPC_IS_ACTIVATED(task))) {
task->tk_status = -EIO;
rpc_put_task_async(task);
- return;
+ return false;
}
+ return true;
+}
+
+void rpc_sleep_on_timeout(struct rpc_wait_queue *q, struct rpc_task *task,
+ rpc_action action, unsigned long timeout)
+{
+ if (!rpc_sleep_check_activated(task))
+ return;
+
+ rpc_set_tk_callback(task, action);
+
+ /*
+ * Protect the queue operations.
+ */
+ spin_lock_bh(&q->lock);
+ __rpc_sleep_on_priority_timeout(q, task, timeout, task->tk_priority);
+ spin_unlock_bh(&q->lock);
+}
+EXPORT_SYMBOL_GPL(rpc_sleep_on_timeout);
+void rpc_sleep_on(struct rpc_wait_queue *q, struct rpc_task *task,
+ rpc_action action)
+{
+ if (!rpc_sleep_check_activated(task))
+ return;
+
+ rpc_set_tk_callback(task, action);
+
+ WARN_ON_ONCE(task->tk_timeout != 0);
/*
* Protect the queue operations.
*/
spin_lock_bh(&q->lock);
- __rpc_sleep_on_priority(q, task, action, task->tk_priority);
+ __rpc_sleep_on_priority(q, task, task->tk_priority);
spin_unlock_bh(&q->lock);
}
EXPORT_SYMBOL_GPL(rpc_sleep_on);
+void rpc_sleep_on_priority_timeout(struct rpc_wait_queue *q,
+ struct rpc_task *task, unsigned long timeout, int priority)
+{
+ if (!rpc_sleep_check_activated(task))
+ return;
+
+ priority -= RPC_PRIORITY_LOW;
+ /*
+ * Protect the queue operations.
+ */
+ spin_lock_bh(&q->lock);
+ __rpc_sleep_on_priority_timeout(q, task, timeout, priority);
+ spin_unlock_bh(&q->lock);
+}
+EXPORT_SYMBOL_GPL(rpc_sleep_on_priority_timeout);
+
void rpc_sleep_on_priority(struct rpc_wait_queue *q, struct rpc_task *task,
- rpc_action action, int priority)
+ int priority)
{
- /* We shouldn't ever put an inactive task to sleep */
- WARN_ON_ONCE(!RPC_IS_ACTIVATED(task));
- if (!RPC_IS_ACTIVATED(task)) {
- task->tk_status = -EIO;
- rpc_put_task_async(task);
+ if (!rpc_sleep_check_activated(task))
return;
- }
+ WARN_ON_ONCE(task->tk_timeout != 0);
+ priority -= RPC_PRIORITY_LOW;
/*
* Protect the queue operations.
*/
spin_lock_bh(&q->lock);
- __rpc_sleep_on_priority(q, task, action, priority - RPC_PRIORITY_LOW);
+ __rpc_sleep_on_priority(q, task, priority);
spin_unlock_bh(&q->lock);
}
EXPORT_SYMBOL_GPL(rpc_sleep_on_priority);
spin_lock(&queue->lock);
expires = now = jiffies;
list_for_each_entry_safe(task, n, &queue->timer_list.list, u.tk_wait.timer_list) {
- timeo = task->u.tk_wait.expires;
+ timeo = task->tk_timeout;
if (time_after_eq(now, timeo)) {
dprintk("RPC: %5u timeout\n", task->tk_pid);
task->tk_status = -ETIMEDOUT;
*/
void rpc_delay(struct rpc_task *task, unsigned long delay)
{
- task->tk_timeout = delay;
- rpc_sleep_on(&delay_queue, task, __rpc_atrun);
+ rpc_sleep_on_timeout(&delay_queue, task, __rpc_atrun, jiffies + delay);
}
EXPORT_SYMBOL_GPL(rpc_delay);
rpc_reset_task_statistics(struct rpc_task *task)
{
task->tk_timeouts = 0;
- task->tk_flags &= ~(RPC_CALL_MAJORSEEN|RPC_TASK_KILLED|RPC_TASK_SENT);
-
+ task->tk_flags &= ~(RPC_CALL_MAJORSEEN|RPC_TASK_SENT);
rpc_init_task_statistics(task);
}
if (task->tk_ops->rpc_call_done != NULL) {
task->tk_ops->rpc_call_done(task, task->tk_calldata);
if (task->tk_action != NULL) {
- WARN_ON(RPC_ASSASSINATED(task));
/* Always release the RPC slot and buffer memory */
xprt_release(task);
rpc_reset_task_statistics(task);
}
}
+void rpc_signal_task(struct rpc_task *task)
+{
+ struct rpc_wait_queue *queue;
+
+ if (!RPC_IS_ACTIVATED(task))
+ return;
+ set_bit(RPC_TASK_SIGNALLED, &task->tk_runstate);
+ smp_mb__after_atomic();
+ queue = READ_ONCE(task->tk_waitqueue);
+ if (queue)
+ rpc_wake_up_queued_task_set_status(queue, task, -ERESTARTSYS);
+}
+
void rpc_exit(struct rpc_task *task, int status)
{
task->tk_status = status;
*/
if (!RPC_IS_QUEUED(task))
continue;
+
+ /*
+ * Signalled tasks should exit rather than sleep.
+ */
+ if (RPC_SIGNALLED(task))
+ rpc_exit(task, -ERESTARTSYS);
+
/*
* The queue->lock protects against races with
* rpc_make_runnable().
status = out_of_line_wait_on_bit(&task->tk_runstate,
RPC_TASK_QUEUED, rpc_wait_bit_killable,
TASK_KILLABLE);
- if (status == -ERESTARTSYS) {
+ if (status < 0) {
/*
* When a sync task receives a signal, it exits with
* -ERESTARTSYS. In order to catch any callbacks that
* break the loop here, but go around once more.
*/
dprintk("RPC: %5u got signal\n", task->tk_pid);
- task->tk_flags |= RPC_TASK_KILLED;
+ set_bit(RPC_TASK_SIGNALLED, &task->tk_runstate);
rpc_exit(task, -ERESTARTSYS);
}
dprintk("RPC: %5u sync task resuming\n", task->tk_pid);
/* ACL likes to be lazy in allocating pages - ACLs
* are small by default but can get huge. */
if ((xdr->flags & XDRBUF_SPARSE_PAGES) && *ppage == NULL) {
- *ppage = alloc_page(GFP_ATOMIC);
+ *ppage = alloc_page(GFP_NOWAIT | __GFP_NOWARN);
if (unlikely(*ppage == NULL)) {
if (copied == 0)
copied = -ENOMEM;
static DEFINE_SPINLOCK(xprt_list_lock);
static LIST_HEAD(xprt_list);
+static unsigned long xprt_request_timeout(const struct rpc_rqst *req)
+{
+ unsigned long timeout = jiffies + req->rq_timeout;
+
+ if (time_before(timeout, req->rq_majortimeo))
+ return timeout;
+ return req->rq_majortimeo;
+}
+
/**
* xprt_register_transport - register a transport implementation
* @transport: transport to register
out_sleep:
dprintk("RPC: %5u failed to lock transport %p\n",
task->tk_pid, xprt);
- task->tk_timeout = RPC_IS_SOFT(task) ? req->rq_timeout : 0;
task->tk_status = -EAGAIN;
- rpc_sleep_on(&xprt->sending, task, NULL);
+ if (RPC_IS_SOFT(task))
+ rpc_sleep_on_timeout(&xprt->sending, task, NULL,
+ xprt_request_timeout(req));
+ else
+ rpc_sleep_on(&xprt->sending, task, NULL);
return 0;
}
EXPORT_SYMBOL_GPL(xprt_reserve_xprt);
xprt_clear_locked(xprt);
out_sleep:
dprintk("RPC: %5u failed to lock transport %p\n", task->tk_pid, xprt);
- task->tk_timeout = RPC_IS_SOFT(task) ? req->rq_timeout : 0;
task->tk_status = -EAGAIN;
- rpc_sleep_on(&xprt->sending, task, NULL);
+ if (RPC_IS_SOFT(task))
+ rpc_sleep_on_timeout(&xprt->sending, task, NULL,
+ xprt_request_timeout(req));
+ else
+ rpc_sleep_on(&xprt->sending, task, NULL);
return 0;
}
EXPORT_SYMBOL_GPL(xprt_reserve_xprt_cong);
}
EXPORT_SYMBOL_GPL(xprt_write_space);
-/**
- * xprt_set_retrans_timeout_def - set a request's retransmit timeout
- * @task: task whose timeout is to be set
- *
- * Set a request's retransmit timeout based on the transport's
- * default timeout parameters. Used by transports that don't adjust
- * the retransmit timeout based on round-trip time estimation.
- */
-void xprt_set_retrans_timeout_def(struct rpc_task *task)
+static unsigned long xprt_abs_ktime_to_jiffies(ktime_t abstime)
{
- task->tk_timeout = task->tk_rqstp->rq_timeout;
+ s64 delta = ktime_to_ns(ktime_get() - abstime);
+ return likely(delta >= 0) ?
+ jiffies - nsecs_to_jiffies(delta) :
+ jiffies + nsecs_to_jiffies(-delta);
}
-EXPORT_SYMBOL_GPL(xprt_set_retrans_timeout_def);
-/**
- * xprt_set_retrans_timeout_rtt - set a request's retransmit timeout
- * @task: task whose timeout is to be set
- *
- * Set a request's retransmit timeout using the RTT estimator.
- */
-void xprt_set_retrans_timeout_rtt(struct rpc_task *task)
+static unsigned long xprt_calc_majortimeo(struct rpc_rqst *req)
{
- int timer = task->tk_msg.rpc_proc->p_timer;
- struct rpc_clnt *clnt = task->tk_client;
- struct rpc_rtt *rtt = clnt->cl_rtt;
- struct rpc_rqst *req = task->tk_rqstp;
- unsigned long max_timeout = clnt->cl_timeout->to_maxval;
+ const struct rpc_timeout *to = req->rq_task->tk_client->cl_timeout;
+ unsigned long majortimeo = req->rq_timeout;
- task->tk_timeout = rpc_calc_rto(rtt, timer);
- task->tk_timeout <<= rpc_ntimeo(rtt, timer) + req->rq_retries;
- if (task->tk_timeout > max_timeout || task->tk_timeout == 0)
- task->tk_timeout = max_timeout;
+ if (to->to_exponential)
+ majortimeo <<= to->to_retries;
+ else
+ majortimeo += to->to_increment * to->to_retries;
+ if (majortimeo > to->to_maxval || majortimeo == 0)
+ majortimeo = to->to_maxval;
+ return majortimeo;
}
-EXPORT_SYMBOL_GPL(xprt_set_retrans_timeout_rtt);
static void xprt_reset_majortimeo(struct rpc_rqst *req)
{
- const struct rpc_timeout *to = req->rq_task->tk_client->cl_timeout;
+ req->rq_majortimeo += xprt_calc_majortimeo(req);
+}
- req->rq_majortimeo = req->rq_timeout;
- if (to->to_exponential)
- req->rq_majortimeo <<= to->to_retries;
+static void xprt_init_majortimeo(struct rpc_task *task, struct rpc_rqst *req)
+{
+ unsigned long time_init;
+ struct rpc_xprt *xprt = req->rq_xprt;
+
+ if (likely(xprt && xprt_connected(xprt)))
+ time_init = jiffies;
else
- req->rq_majortimeo += to->to_increment * to->to_retries;
- if (req->rq_majortimeo > to->to_maxval || req->rq_majortimeo == 0)
- req->rq_majortimeo = to->to_maxval;
- req->rq_majortimeo += jiffies;
+ time_init = xprt_abs_ktime_to_jiffies(task->tk_start);
+ req->rq_timeout = task->tk_client->cl_timeout->to_initval;
+ req->rq_majortimeo = time_init + xprt_calc_majortimeo(req);
}
/**
xprt->ops->close(xprt);
if (!xprt_connected(xprt)) {
- task->tk_timeout = task->tk_rqstp->rq_timeout;
task->tk_rqstp->rq_connect_cookie = xprt->connect_cookie;
- rpc_sleep_on(&xprt->pending, task, NULL);
+ rpc_sleep_on_timeout(&xprt->pending, task, NULL,
+ xprt_request_timeout(task->tk_rqstp));
if (test_bit(XPRT_CLOSING, &xprt->state))
return;
* @req: Request to pin
*
* Caller must ensure this is atomic with the call to xprt_lookup_rqst()
- * so should be holding the xprt receive lock.
+ * so should be holding xprt->queue_lock.
*/
void xprt_pin_rqst(struct rpc_rqst *req)
{
* xprt_unpin_rqst - Unpin a request on the transport receive list
* @req: Request to pin
*
- * Caller should be holding the xprt receive lock.
+ * Caller should be holding xprt->queue_lock.
*/
void xprt_unpin_rqst(struct rpc_rqst *req)
{
set_bit(RPC_TASK_NEED_RECV, &task->tk_runstate);
spin_unlock(&xprt->queue_lock);
- xprt_reset_majortimeo(req);
/* Turn off autodisconnect */
del_singleshot_timer_sync(&xprt->timer);
}
task->tk_status = 0;
}
+/**
+ * xprt_wait_for_reply_request_def - wait for reply
+ * @task: pointer to rpc_task
+ *
+ * Set a request's retransmit timeout based on the transport's
+ * default timeout parameters. Used by transports that don't adjust
+ * the retransmit timeout based on round-trip time estimation,
+ * and put the task to sleep on the pending queue.
+ */
+void xprt_wait_for_reply_request_def(struct rpc_task *task)
+{
+ struct rpc_rqst *req = task->tk_rqstp;
+
+ rpc_sleep_on_timeout(&req->rq_xprt->pending, task, xprt_timer,
+ xprt_request_timeout(req));
+}
+EXPORT_SYMBOL_GPL(xprt_wait_for_reply_request_def);
+
+/**
+ * xprt_wait_for_reply_request_rtt - wait for reply using RTT estimator
+ * @task: pointer to rpc_task
+ *
+ * Set a request's retransmit timeout using the RTT estimator,
+ * and put the task to sleep on the pending queue.
+ */
+void xprt_wait_for_reply_request_rtt(struct rpc_task *task)
+{
+ int timer = task->tk_msg.rpc_proc->p_timer;
+ struct rpc_clnt *clnt = task->tk_client;
+ struct rpc_rtt *rtt = clnt->cl_rtt;
+ struct rpc_rqst *req = task->tk_rqstp;
+ unsigned long max_timeout = clnt->cl_timeout->to_maxval;
+ unsigned long timeout;
+
+ timeout = rpc_calc_rto(rtt, timer);
+ timeout <<= rpc_ntimeo(rtt, timer) + req->rq_retries;
+ if (timeout > max_timeout || timeout == 0)
+ timeout = max_timeout;
+ rpc_sleep_on_timeout(&req->rq_xprt->pending, task, xprt_timer,
+ jiffies + timeout);
+}
+EXPORT_SYMBOL_GPL(xprt_wait_for_reply_request_rtt);
+
/**
* xprt_request_wait_receive - wait for the reply to an RPC request
* @task: RPC task about to send a request
*/
spin_lock(&xprt->queue_lock);
if (test_bit(RPC_TASK_NEED_RECV, &task->tk_runstate)) {
- xprt->ops->set_retrans_timeout(task);
- rpc_sleep_on(&xprt->pending, task, xprt_timer);
+ xprt->ops->wait_for_reply_request(task);
/*
* Send an extra queue wakeup call if the
* connection was dropped in case the call to
if (status < 0)
goto out_dequeue;
}
+ if (RPC_SIGNALLED(task)) {
+ status = -ERESTARTSYS;
+ goto out_dequeue;
+ }
}
/*
struct rpc_xprt *xprt = task->tk_xprt;
struct rpc_rqst *req = task->tk_rqstp;
- req->rq_timeout = task->tk_client->cl_timeout->to_initval;
req->rq_task = task;
req->rq_xprt = xprt;
req->rq_buffer = NULL;
req->rq_snd_buf.bvec = NULL;
req->rq_rcv_buf.bvec = NULL;
req->rq_release_snd_buf = NULL;
- xprt_reset_majortimeo(req);
+ xprt_init_majortimeo(task, req);
dprintk("RPC: %5u reserved req %p xid %08x\n", task->tk_pid,
req, ntohl(req->rq_xid));
}
if (task->tk_rqstp != NULL)
return;
- task->tk_timeout = 0;
task->tk_status = -EAGAIN;
if (!xprt_throttle_congested(xprt, task))
xprt_do_reserve(xprt, task);
if (task->tk_rqstp != NULL)
return;
- task->tk_timeout = 0;
task->tk_status = -EAGAIN;
xprt_do_reserve(xprt, task);
}
xprt->idle_timeout = 0;
INIT_WORK(&xprt->task_cleanup, xprt_autoclose);
if (xprt_has_timer(xprt))
- timer_setup(&xprt->timer, xprt_init_autodisconnect, 0);
+ timer_setup(&xprt->timer,
+ xprt_init_autodisconnect,
+ TIMER_DEFERRABLE);
else
timer_setup(&xprt->timer, NULL, 0);
#undef RPCRDMA_BACKCHANNEL_DEBUG
-static int rpcrdma_bc_setup_reqs(struct rpcrdma_xprt *r_xprt,
- unsigned int count)
-{
- struct rpc_xprt *xprt = &r_xprt->rx_xprt;
- struct rpcrdma_req *req;
- struct rpc_rqst *rqst;
- unsigned int i;
-
- for (i = 0; i < (count << 1); i++) {
- struct rpcrdma_regbuf *rb;
- size_t size;
-
- req = rpcrdma_create_req(r_xprt);
- if (IS_ERR(req))
- return PTR_ERR(req);
- rqst = &req->rl_slot;
-
- rqst->rq_xprt = xprt;
- INIT_LIST_HEAD(&rqst->rq_bc_list);
- __set_bit(RPC_BC_PA_IN_USE, &rqst->rq_bc_pa_state);
- spin_lock(&xprt->bc_pa_lock);
- list_add(&rqst->rq_bc_pa_list, &xprt->bc_pa_list);
- spin_unlock(&xprt->bc_pa_lock);
-
- size = r_xprt->rx_data.inline_rsize;
- rb = rpcrdma_alloc_regbuf(size, DMA_TO_DEVICE, GFP_KERNEL);
- if (IS_ERR(rb))
- goto out_fail;
- req->rl_sendbuf = rb;
- xdr_buf_init(&rqst->rq_snd_buf, rb->rg_base,
- min_t(size_t, size, PAGE_SIZE));
- }
- return 0;
-
-out_fail:
- rpcrdma_req_destroy(req);
- return -ENOMEM;
-}
-
/**
* xprt_rdma_bc_setup - Pre-allocate resources for handling backchannel requests
* @xprt: transport associated with these backchannel resources
int xprt_rdma_bc_setup(struct rpc_xprt *xprt, unsigned int reqs)
{
struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt);
- int rc;
- /* The backchannel reply path returns each rpc_rqst to the
- * bc_pa_list _after_ the reply is sent. If the server is
- * faster than the client, it can send another backward
- * direction request before the rpc_rqst is returned to the
- * list. The client rejects the request in this case.
- *
- * Twice as many rpc_rqsts are prepared to ensure there is
- * always an rpc_rqst available as soon as a reply is sent.
- */
- if (reqs > RPCRDMA_BACKWARD_WRS >> 1)
- goto out_err;
-
- rc = rpcrdma_bc_setup_reqs(r_xprt, reqs);
- if (rc)
- goto out_free;
-
- r_xprt->rx_buf.rb_bc_srv_max_requests = reqs;
+ r_xprt->rx_buf.rb_bc_srv_max_requests = RPCRDMA_BACKWARD_WRS >> 1;
trace_xprtrdma_cb_setup(r_xprt, reqs);
return 0;
-
-out_free:
- xprt_rdma_bc_destroy(xprt, reqs);
-
-out_err:
- pr_err("RPC: %s: setup backchannel transport failed\n", __func__);
- return -ENOMEM;
}
/**
size_t xprt_rdma_bc_maxpayload(struct rpc_xprt *xprt)
{
struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt);
- struct rpcrdma_create_data_internal *cdata = &r_xprt->rx_data;
+ struct rpcrdma_ep *ep = &r_xprt->rx_ep;
size_t maxmsg;
- maxmsg = min_t(unsigned int, cdata->inline_rsize, cdata->inline_wsize);
+ maxmsg = min_t(unsigned int, ep->rep_inline_send, ep->rep_inline_recv);
maxmsg = min_t(unsigned int, maxmsg, PAGE_SIZE);
return maxmsg - RPCRDMA_HDRLEN_MIN;
}
rpcrdma_set_xdrlen(&req->rl_hdrbuf, 0);
xdr_init_encode(&req->rl_stream, &req->rl_hdrbuf,
- req->rl_rdmabuf->rg_base, rqst);
+ rdmab_data(req->rl_rdmabuf), rqst);
p = xdr_reserve_space(&req->rl_stream, 28);
if (unlikely(!p))
spin_unlock(&xprt->bc_pa_lock);
}
+static struct rpc_rqst *rpcrdma_bc_rqst_get(struct rpcrdma_xprt *r_xprt)
+{
+ struct rpc_xprt *xprt = &r_xprt->rx_xprt;
+ struct rpcrdma_req *req;
+ struct rpc_rqst *rqst;
+ size_t size;
+
+ spin_lock(&xprt->bc_pa_lock);
+ rqst = list_first_entry_or_null(&xprt->bc_pa_list, struct rpc_rqst,
+ rq_bc_pa_list);
+ if (!rqst)
+ goto create_req;
+ list_del(&rqst->rq_bc_pa_list);
+ spin_unlock(&xprt->bc_pa_lock);
+ return rqst;
+
+create_req:
+ spin_unlock(&xprt->bc_pa_lock);
+
+ /* Set a limit to prevent a remote from overrunning our resources.
+ */
+ if (xprt->bc_alloc_count >= RPCRDMA_BACKWARD_WRS)
+ return NULL;
+
+ size = min_t(size_t, r_xprt->rx_ep.rep_inline_recv, PAGE_SIZE);
+ req = rpcrdma_req_create(r_xprt, size, GFP_KERNEL);
+ if (!req)
+ return NULL;
+
+ xprt->bc_alloc_count++;
+ rqst = &req->rl_slot;
+ rqst->rq_xprt = xprt;
+ __set_bit(RPC_BC_PA_IN_USE, &rqst->rq_bc_pa_state);
+ xdr_buf_init(&rqst->rq_snd_buf, rdmab_data(req->rl_sendbuf), size);
+ return rqst;
+}
+
/**
* rpcrdma_bc_receive_call - Handle a backward direction call
* @r_xprt: transport receiving the call
pr_info("RPC: %s: %*ph\n", __func__, size, p);
#endif
- /* Grab a free bc rqst */
- spin_lock(&xprt->bc_pa_lock);
- if (list_empty(&xprt->bc_pa_list)) {
- spin_unlock(&xprt->bc_pa_lock);
+ rqst = rpcrdma_bc_rqst_get(r_xprt);
+ if (!rqst)
goto out_overflow;
- }
- rqst = list_first_entry(&xprt->bc_pa_list,
- struct rpc_rqst, rq_bc_pa_list);
- list_del(&rqst->rq_bc_pa_list);
- spin_unlock(&xprt->bc_pa_lock);
- /* Prepare rqst */
rqst->rq_reply_bytes_recvd = 0;
rqst->rq_xid = *p;
/**
* frwr_is_supported - Check if device supports FRWR
- * @ia: interface adapter to check
+ * @device: interface adapter to check
*
* Returns true if device supports FRWR, otherwise false
*/
-bool frwr_is_supported(struct rpcrdma_ia *ia)
+bool frwr_is_supported(struct ib_device *device)
{
- struct ib_device_attr *attrs = &ia->ri_device->attrs;
+ struct ib_device_attr *attrs = &device->attrs;
if (!(attrs->device_cap_flags & IB_DEVICE_MEM_MGT_EXTENSIONS))
goto out_not_supported;
out_not_supported:
pr_info("rpcrdma: 'frwr' mode is not supported by device %s\n",
- ia->ri_device->name);
+ device->name);
return false;
}
if (mr->mr_dir != DMA_NONE) {
trace_xprtrdma_mr_unmap(mr);
- ib_dma_unmap_sg(r_xprt->rx_ia.ri_device,
+ ib_dma_unmap_sg(r_xprt->rx_ia.ri_id->device,
mr->mr_sg, mr->mr_nents, mr->mr_dir);
mr->mr_dir = DMA_NONE;
}
* frwr_open - Prepare an endpoint for use with FRWR
* @ia: interface adapter this endpoint will use
* @ep: endpoint to prepare
- * @cdata: transport parameters
*
* On success, sets:
* ep->rep_attr.cap.max_send_wr
* ep->rep_attr.cap.max_recv_wr
- * cdata->max_requests
+ * ep->rep_max_requests
* ia->ri_max_segs
*
* And these FRWR-related fields:
*
* On failure, a negative errno is returned.
*/
-int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep,
- struct rpcrdma_create_data_internal *cdata)
+int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep)
{
- struct ib_device_attr *attrs = &ia->ri_device->attrs;
+ struct ib_device_attr *attrs = &ia->ri_id->device->attrs;
int max_qp_wr, depth, delta;
ia->ri_mrtype = IB_MR_TYPE_MEM_REG;
} while (delta > 0);
}
- max_qp_wr = ia->ri_device->attrs.max_qp_wr;
+ max_qp_wr = ia->ri_id->device->attrs.max_qp_wr;
max_qp_wr -= RPCRDMA_BACKWARD_WRS;
max_qp_wr -= 1;
if (max_qp_wr < RPCRDMA_MIN_SLOT_TABLE)
return -ENOMEM;
- if (cdata->max_requests > max_qp_wr)
- cdata->max_requests = max_qp_wr;
- ep->rep_attr.cap.max_send_wr = cdata->max_requests * depth;
+ if (ep->rep_max_requests > max_qp_wr)
+ ep->rep_max_requests = max_qp_wr;
+ ep->rep_attr.cap.max_send_wr = ep->rep_max_requests * depth;
if (ep->rep_attr.cap.max_send_wr > max_qp_wr) {
- cdata->max_requests = max_qp_wr / depth;
- if (!cdata->max_requests)
+ ep->rep_max_requests = max_qp_wr / depth;
+ if (!ep->rep_max_requests)
return -EINVAL;
- ep->rep_attr.cap.max_send_wr = cdata->max_requests *
- depth;
+ ep->rep_attr.cap.max_send_wr = ep->rep_max_requests * depth;
}
ep->rep_attr.cap.max_send_wr += RPCRDMA_BACKWARD_WRS;
ep->rep_attr.cap.max_send_wr += 1; /* for ib_drain_sq */
- ep->rep_attr.cap.max_recv_wr = cdata->max_requests;
+ ep->rep_attr.cap.max_recv_wr = ep->rep_max_requests;
ep->rep_attr.cap.max_recv_wr += RPCRDMA_BACKWARD_WRS;
ep->rep_attr.cap.max_recv_wr += 1; /* for ib_drain_rq */
(ia->ri_max_segs - 2) * ia->ri_max_frwr_depth);
}
-static void
-__frwr_sendcompletion_flush(struct ib_wc *wc, const char *wr)
-{
- if (wc->status != IB_WC_WR_FLUSH_ERR)
- pr_err("rpcrdma: %s: %s (%u/0x%x)\n",
- wr, ib_wc_status_msg(wc->status),
- wc->status, wc->vendor_err);
-}
-
/**
* frwr_wc_fastreg - Invoked by RDMA provider for a flushed FastReg WC
* @cq: completion queue (ignored)
container_of(cqe, struct rpcrdma_frwr, fr_cqe);
/* WARNING: Only wr_cqe and status are reliable at this point */
- if (wc->status != IB_WC_SUCCESS) {
+ if (wc->status != IB_WC_SUCCESS)
frwr->fr_state = FRWR_FLUSHED_FR;
- __frwr_sendcompletion_flush(wc, "fastreg");
- }
trace_xprtrdma_wc_fastreg(wc, frwr);
}
fr_cqe);
/* WARNING: Only wr_cqe and status are reliable at this point */
- if (wc->status != IB_WC_SUCCESS) {
+ if (wc->status != IB_WC_SUCCESS)
frwr->fr_state = FRWR_FLUSHED_LI;
- __frwr_sendcompletion_flush(wc, "localinv");
- }
trace_xprtrdma_wc_li(wc, frwr);
}
fr_cqe);
/* WARNING: Only wr_cqe and status are reliable at this point */
- if (wc->status != IB_WC_SUCCESS) {
+ if (wc->status != IB_WC_SUCCESS)
frwr->fr_state = FRWR_FLUSHED_LI;
- __frwr_sendcompletion_flush(wc, "localinv");
- }
- complete(&frwr->fr_linv_done);
trace_xprtrdma_wc_li_wake(wc, frwr);
+ complete(&frwr->fr_linv_done);
}
/**
}
mr->mr_dir = rpcrdma_data_dir(writing);
- mr->mr_nents = ib_dma_map_sg(ia->ri_device, mr->mr_sg, i, mr->mr_dir);
+ mr->mr_nents =
+ ib_dma_map_sg(ia->ri_id->device, mr->mr_sg, i, mr->mr_dir);
if (!mr->mr_nents)
goto out_dmamap_err;
return seg;
out_dmamap_err:
- frwr->fr_state = FRWR_IS_INVALID;
+ mr->mr_dir = DMA_NONE;
trace_xprtrdma_frwr_sgerr(mr, i);
rpcrdma_mr_put(mr);
return ERR_PTR(-EIO);
return size;
}
+/**
+ * rpcrdma_set_max_header_sizes - Initialize inline payload sizes
+ * @r_xprt: transport instance to initialize
+ *
+ * The max_inline fields contain the maximum size of an RPC message
+ * so the marshaling code doesn't have to repeat this calculation
+ * for every RPC.
+ */
void rpcrdma_set_max_header_sizes(struct rpcrdma_xprt *r_xprt)
{
- struct rpcrdma_create_data_internal *cdata = &r_xprt->rx_data;
- struct rpcrdma_ia *ia = &r_xprt->rx_ia;
- unsigned int maxsegs = ia->ri_max_segs;
-
- ia->ri_max_inline_write = cdata->inline_wsize -
- rpcrdma_max_call_header_size(maxsegs);
- ia->ri_max_inline_read = cdata->inline_rsize -
- rpcrdma_max_reply_header_size(maxsegs);
+ unsigned int maxsegs = r_xprt->rx_ia.ri_max_segs;
+ struct rpcrdma_ep *ep = &r_xprt->rx_ep;
+
+ ep->rep_max_inline_send =
+ ep->rep_inline_send - rpcrdma_max_call_header_size(maxsegs);
+ ep->rep_max_inline_recv =
+ ep->rep_inline_recv - rpcrdma_max_reply_header_size(maxsegs);
}
/* The client can send a request inline as long as the RPCRDMA header
struct xdr_buf *xdr = &rqst->rq_snd_buf;
unsigned int count, remaining, offset;
- if (xdr->len > r_xprt->rx_ia.ri_max_inline_write)
+ if (xdr->len > r_xprt->rx_ep.rep_max_inline_send)
return false;
if (xdr->page_len) {
static bool rpcrdma_results_inline(struct rpcrdma_xprt *r_xprt,
struct rpc_rqst *rqst)
{
- struct rpcrdma_ia *ia = &r_xprt->rx_ia;
-
- return rqst->rq_rcv_buf.buflen <= ia->ri_max_inline_read;
+ return rqst->rq_rcv_buf.buflen <= r_xprt->rx_ep.rep_max_inline_recv;
}
/* The client is required to provide a Reply chunk if the maximum
const struct rpc_rqst *rqst)
{
const struct xdr_buf *buf = &rqst->rq_rcv_buf;
- const struct rpcrdma_ia *ia = &r_xprt->rx_ia;
- return buf->head[0].iov_len + buf->tail[0].iov_len <
- ia->ri_max_inline_read;
+ return (buf->head[0].iov_len + buf->tail[0].iov_len) <
+ r_xprt->rx_ep.rep_max_inline_recv;
}
/* Split @vec on page boundaries into SGEs. FMR registers pages, not
*/
if (unlikely(xdrbuf->flags & XDRBUF_SPARSE_PAGES)) {
if (!*ppages)
- *ppages = alloc_page(GFP_ATOMIC);
+ *ppages = alloc_page(GFP_NOWAIT | __GFP_NOWARN);
if (!*ppages)
return -ENOBUFS;
}
}
/**
- * rpcrdma_unmap_sendctx - DMA-unmap Send buffers
+ * rpcrdma_sendctx_unmap - DMA-unmap Send buffer
* @sc: sendctx containing SGEs to unmap
*
*/
-void
-rpcrdma_unmap_sendctx(struct rpcrdma_sendctx *sc)
+void rpcrdma_sendctx_unmap(struct rpcrdma_sendctx *sc)
{
- struct rpcrdma_ia *ia = &sc->sc_xprt->rx_ia;
struct ib_sge *sge;
- unsigned int count;
/* The first two SGEs contain the transport header and
* the inline buffer. These are always left mapped so
* they can be cheaply re-used.
*/
- sge = &sc->sc_sges[2];
- for (count = sc->sc_unmap_count; count; ++sge, --count)
- ib_dma_unmap_page(ia->ri_device,
- sge->addr, sge->length, DMA_TO_DEVICE);
+ for (sge = &sc->sc_sges[2]; sc->sc_unmap_count;
+ ++sge, --sc->sc_unmap_count)
+ ib_dma_unmap_page(sc->sc_device, sge->addr, sge->length,
+ DMA_TO_DEVICE);
- if (test_and_clear_bit(RPCRDMA_REQ_F_TX_RESOURCES, &sc->sc_req->rl_flags)) {
- smp_mb__after_atomic();
+ if (test_and_clear_bit(RPCRDMA_REQ_F_TX_RESOURCES,
+ &sc->sc_req->rl_flags))
wake_up_bit(&sc->sc_req->rl_flags, RPCRDMA_REQ_F_TX_RESOURCES);
- }
}
/* Prepare an SGE for the RPC-over-RDMA transport header.
*/
-static bool
-rpcrdma_prepare_hdr_sge(struct rpcrdma_ia *ia, struct rpcrdma_req *req,
- u32 len)
+static bool rpcrdma_prepare_hdr_sge(struct rpcrdma_xprt *r_xprt,
+ struct rpcrdma_req *req, u32 len)
{
struct rpcrdma_sendctx *sc = req->rl_sendctx;
struct rpcrdma_regbuf *rb = req->rl_rdmabuf;
struct ib_sge *sge = sc->sc_sges;
- if (!rpcrdma_dma_map_regbuf(ia, rb))
+ if (!rpcrdma_regbuf_dma_map(r_xprt, rb))
goto out_regbuf;
sge->addr = rdmab_addr(rb);
sge->length = len;
sge->lkey = rdmab_lkey(rb);
- ib_dma_sync_single_for_device(rdmab_device(rb), sge->addr,
- sge->length, DMA_TO_DEVICE);
+ ib_dma_sync_single_for_device(rdmab_device(rb), sge->addr, sge->length,
+ DMA_TO_DEVICE);
sc->sc_wr.num_sge++;
return true;
/* Prepare the Send SGEs. The head and tail iovec, and each entry
* in the page list, gets its own SGE.
*/
-static bool
-rpcrdma_prepare_msg_sges(struct rpcrdma_ia *ia, struct rpcrdma_req *req,
- struct xdr_buf *xdr, enum rpcrdma_chunktype rtype)
+static bool rpcrdma_prepare_msg_sges(struct rpcrdma_xprt *r_xprt,
+ struct rpcrdma_req *req,
+ struct xdr_buf *xdr,
+ enum rpcrdma_chunktype rtype)
{
struct rpcrdma_sendctx *sc = req->rl_sendctx;
unsigned int sge_no, page_base, len, remaining;
struct rpcrdma_regbuf *rb = req->rl_sendbuf;
- struct ib_device *device = ia->ri_device;
struct ib_sge *sge = sc->sc_sges;
- u32 lkey = ia->ri_pd->local_dma_lkey;
struct page *page, **ppages;
/* The head iovec is straightforward, as it is already
* DMA-mapped. Sync the content that has changed.
*/
- if (!rpcrdma_dma_map_regbuf(ia, rb))
+ if (!rpcrdma_regbuf_dma_map(r_xprt, rb))
goto out_regbuf;
+ sc->sc_device = rdmab_device(rb);
sge_no = 1;
sge[sge_no].addr = rdmab_addr(rb);
sge[sge_no].length = xdr->head[0].iov_len;
goto out_mapping_overflow;
len = min_t(u32, PAGE_SIZE - page_base, remaining);
- sge[sge_no].addr = ib_dma_map_page(device, *ppages,
- page_base, len,
- DMA_TO_DEVICE);
- if (ib_dma_mapping_error(device, sge[sge_no].addr))
+ sge[sge_no].addr =
+ ib_dma_map_page(rdmab_device(rb), *ppages,
+ page_base, len, DMA_TO_DEVICE);
+ if (ib_dma_mapping_error(rdmab_device(rb),
+ sge[sge_no].addr))
goto out_mapping_err;
sge[sge_no].length = len;
- sge[sge_no].lkey = lkey;
+ sge[sge_no].lkey = rdmab_lkey(rb);
sc->sc_unmap_count++;
ppages++;
map_tail:
sge_no++;
- sge[sge_no].addr = ib_dma_map_page(device, page,
- page_base, len,
- DMA_TO_DEVICE);
- if (ib_dma_mapping_error(device, sge[sge_no].addr))
+ sge[sge_no].addr =
+ ib_dma_map_page(rdmab_device(rb), page, page_base, len,
+ DMA_TO_DEVICE);
+ if (ib_dma_mapping_error(rdmab_device(rb), sge[sge_no].addr))
goto out_mapping_err;
sge[sge_no].length = len;
- sge[sge_no].lkey = lkey;
+ sge[sge_no].lkey = rdmab_lkey(rb);
sc->sc_unmap_count++;
}
return false;
out_mapping_overflow:
- rpcrdma_unmap_sendctx(sc);
+ rpcrdma_sendctx_unmap(sc);
pr_err("rpcrdma: too many Send SGEs (%u)\n", sge_no);
return false;
out_mapping_err:
- rpcrdma_unmap_sendctx(sc);
+ rpcrdma_sendctx_unmap(sc);
trace_xprtrdma_dma_maperr(sge[sge_no].addr);
return false;
}
struct rpcrdma_req *req, u32 hdrlen,
struct xdr_buf *xdr, enum rpcrdma_chunktype rtype)
{
- req->rl_sendctx = rpcrdma_sendctx_get_locked(&r_xprt->rx_buf);
+ req->rl_sendctx = rpcrdma_sendctx_get_locked(r_xprt);
if (!req->rl_sendctx)
return -EAGAIN;
req->rl_sendctx->sc_wr.num_sge = 0;
req->rl_sendctx->sc_req = req;
__clear_bit(RPCRDMA_REQ_F_TX_RESOURCES, &req->rl_flags);
- if (!rpcrdma_prepare_hdr_sge(&r_xprt->rx_ia, req, hdrlen))
+ if (!rpcrdma_prepare_hdr_sge(r_xprt, req, hdrlen))
return -EIO;
if (rtype != rpcrdma_areadch)
- if (!rpcrdma_prepare_msg_sges(&r_xprt->rx_ia, req, xdr, rtype))
+ if (!rpcrdma_prepare_msg_sges(r_xprt, req, xdr, rtype))
return -EIO;
return 0;
int ret;
rpcrdma_set_xdrlen(&req->rl_hdrbuf, 0);
- xdr_init_encode(xdr, &req->rl_hdrbuf,
- req->rl_rdmabuf->rg_base, rqst);
+ xdr_init_encode(xdr, &req->rl_hdrbuf, rdmab_data(req->rl_rdmabuf),
+ rqst);
/* Fixed header fields */
ret = -EMSGSIZE;
return 0;
out_err:
+ trace_xprtrdma_marshal_failed(rqst, ret);
switch (ret) {
case -EAGAIN:
xprt_wait_for_buffer_space(rqst->rq_xprt);
.buf_alloc = xprt_rdma_bc_allocate,
.buf_free = xprt_rdma_bc_free,
.send_request = xprt_rdma_bc_send_request,
- .set_retrans_timeout = xprt_set_retrans_timeout_def,
+ .wait_for_reply_request = xprt_wait_for_reply_request_def,
.close = xprt_rdma_bc_close,
.destroy = xprt_rdma_bc_put,
.print_stats = xprt_rdma_print_stats
* tunables
*/
-static unsigned int xprt_rdma_slot_table_entries = RPCRDMA_DEF_SLOT_TABLE;
+unsigned int xprt_rdma_slot_table_entries = RPCRDMA_DEF_SLOT_TABLE;
unsigned int xprt_rdma_max_inline_read = RPCRDMA_DEF_INLINE;
-static unsigned int xprt_rdma_max_inline_write = RPCRDMA_DEF_INLINE;
+unsigned int xprt_rdma_max_inline_write = RPCRDMA_DEF_INLINE;
unsigned int xprt_rdma_memreg_strategy = RPCRDMA_FRWR;
int xprt_rdma_pad_optimize;
cancel_delayed_work_sync(&r_xprt->rx_connect_worker);
- rpcrdma_ep_destroy(&r_xprt->rx_ep, &r_xprt->rx_ia);
+ rpcrdma_ep_destroy(r_xprt);
rpcrdma_buffer_destroy(&r_xprt->rx_buf);
rpcrdma_ia_close(&r_xprt->rx_ia);
static struct rpc_xprt *
xprt_setup_rdma(struct xprt_create *args)
{
- struct rpcrdma_create_data_internal cdata;
struct rpc_xprt *xprt;
struct rpcrdma_xprt *new_xprt;
- struct rpcrdma_ep *new_ep;
struct sockaddr *sap;
int rc;
xprt_set_bound(xprt);
xprt_rdma_format_addresses(xprt, sap);
- cdata.max_requests = xprt_rdma_slot_table_entries;
-
- cdata.rsize = RPCRDMA_MAX_SEGS * PAGE_SIZE; /* RDMA write max */
- cdata.wsize = RPCRDMA_MAX_SEGS * PAGE_SIZE; /* RDMA read max */
-
- cdata.inline_wsize = xprt_rdma_max_inline_write;
- if (cdata.inline_wsize > cdata.wsize)
- cdata.inline_wsize = cdata.wsize;
-
- cdata.inline_rsize = xprt_rdma_max_inline_read;
- if (cdata.inline_rsize > cdata.rsize)
- cdata.inline_rsize = cdata.rsize;
-
- /*
- * Create new transport instance, which includes initialized
- * o ia
- * o endpoint
- * o buffers
- */
-
new_xprt = rpcx_to_rdmax(xprt);
-
rc = rpcrdma_ia_open(new_xprt);
if (rc)
goto out1;
- /*
- * initialize and create ep
- */
- new_xprt->rx_data = cdata;
- new_ep = &new_xprt->rx_ep;
-
- rc = rpcrdma_ep_create(&new_xprt->rx_ep,
- &new_xprt->rx_ia, &new_xprt->rx_data);
+ rc = rpcrdma_ep_create(new_xprt);
if (rc)
goto out2;
rpcrdma_buffer_destroy(&new_xprt->rx_buf);
rc = -ENODEV;
out3:
- rpcrdma_ep_destroy(new_ep, &new_xprt->rx_ia);
+ rpcrdma_ep_destroy(new_xprt);
out2:
rpcrdma_ia_close(&new_xprt->rx_ia);
out1:
rpc_wake_up_next(&xprt->backlog);
}
-static bool
-rpcrdma_get_sendbuf(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req,
- size_t size, gfp_t flags)
+static bool rpcrdma_check_regbuf(struct rpcrdma_xprt *r_xprt,
+ struct rpcrdma_regbuf *rb, size_t size,
+ gfp_t flags)
{
- struct rpcrdma_regbuf *rb;
-
- if (req->rl_sendbuf && rdmab_length(req->rl_sendbuf) >= size)
- return true;
-
- rb = rpcrdma_alloc_regbuf(size, DMA_TO_DEVICE, flags);
- if (IS_ERR(rb))
- return false;
-
- rpcrdma_free_regbuf(req->rl_sendbuf);
- r_xprt->rx_stats.hardway_register_count += size;
- req->rl_sendbuf = rb;
- return true;
-}
-
-/* The rq_rcv_buf is used only if a Reply chunk is necessary.
- * The decision to use a Reply chunk is made later in
- * rpcrdma_marshal_req. This buffer is registered at that time.
- *
- * Otherwise, the associated RPC Reply arrives in a separate
- * Receive buffer, arbitrarily chosen by the HCA. The buffer
- * allocated here for the RPC Reply is not utilized in that
- * case. See rpcrdma_inline_fixup.
- *
- * A regbuf is used here to remember the buffer size.
- */
-static bool
-rpcrdma_get_recvbuf(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req,
- size_t size, gfp_t flags)
-{
- struct rpcrdma_regbuf *rb;
-
- if (req->rl_recvbuf && rdmab_length(req->rl_recvbuf) >= size)
- return true;
-
- rb = rpcrdma_alloc_regbuf(size, DMA_NONE, flags);
- if (IS_ERR(rb))
- return false;
-
- rpcrdma_free_regbuf(req->rl_recvbuf);
- r_xprt->rx_stats.hardway_register_count += size;
- req->rl_recvbuf = rb;
+ if (unlikely(rdmab_length(rb) < size)) {
+ if (!rpcrdma_regbuf_realloc(rb, size, flags))
+ return false;
+ r_xprt->rx_stats.hardway_register_count += size;
+ }
return true;
}
if (RPC_IS_SWAPPER(task))
flags = __GFP_MEMALLOC | GFP_NOWAIT | __GFP_NOWARN;
- if (!rpcrdma_get_sendbuf(r_xprt, req, rqst->rq_callsize, flags))
+ if (!rpcrdma_check_regbuf(r_xprt, req->rl_sendbuf, rqst->rq_callsize,
+ flags))
goto out_fail;
- if (!rpcrdma_get_recvbuf(r_xprt, req, rqst->rq_rcvsize, flags))
+ if (!rpcrdma_check_regbuf(r_xprt, req->rl_recvbuf, rqst->rq_rcvsize,
+ flags))
goto out_fail;
- rqst->rq_buffer = req->rl_sendbuf->rg_base;
- rqst->rq_rbuffer = req->rl_recvbuf->rg_base;
+ rqst->rq_buffer = rdmab_data(req->rl_sendbuf);
+ rqst->rq_rbuffer = rdmab_data(req->rl_recvbuf);
trace_xprtrdma_op_allocate(task, req);
return 0;
.alloc_slot = xprt_rdma_alloc_slot,
.free_slot = xprt_rdma_free_slot,
.release_request = xprt_release_rqst_cong, /* ditto */
- .set_retrans_timeout = xprt_set_retrans_timeout_def, /* ditto */
+ .wait_for_reply_request = xprt_wait_for_reply_request_def, /* ditto */
.timer = xprt_rdma_timer,
.rpcbind = rpcb_getport_async, /* sunrpc/rpcb_clnt.c */
.set_port = xprt_rdma_set_port,
static void rpcrdma_sendctx_put_locked(struct rpcrdma_sendctx *sc);
static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt);
static void rpcrdma_mrs_destroy(struct rpcrdma_buffer *buf);
-static int rpcrdma_create_rep(struct rpcrdma_xprt *r_xprt, bool temp);
-static void rpcrdma_dma_unmap_regbuf(struct rpcrdma_regbuf *rb);
+static struct rpcrdma_regbuf *
+rpcrdma_regbuf_alloc(size_t size, enum dma_data_direction direction,
+ gfp_t flags);
+static void rpcrdma_regbuf_dma_unmap(struct rpcrdma_regbuf *rb);
+static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb);
static void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, bool temp);
-/* Wait for outstanding transport work to finish.
+/* Wait for outstanding transport work to finish. ib_drain_qp
+ * handles the drains in the wrong order for us, so open code
+ * them here.
*/
static void rpcrdma_xprt_drain(struct rpcrdma_xprt *r_xprt)
{
/* WARNING: Only wr_cqe and status are reliable at this point */
trace_xprtrdma_wc_send(sc, wc);
- if (wc->status != IB_WC_SUCCESS && wc->status != IB_WC_WR_FLUSH_ERR)
- pr_err("rpcrdma: Send: %s (%u/0x%x)\n",
- ib_wc_status_msg(wc->status),
- wc->status, wc->vendor_err);
-
rpcrdma_sendctx_put_locked(sc);
}
return;
out_flushed:
- if (wc->status != IB_WC_WR_FLUSH_ERR)
- pr_err("rpcrdma: Recv: %s (%u/0x%x)\n",
- ib_wc_status_msg(wc->status),
- wc->status, wc->vendor_err);
rpcrdma_recv_buffer_put(rep);
}
rpcrdma_update_connect_private(struct rpcrdma_xprt *r_xprt,
struct rdma_conn_param *param)
{
- struct rpcrdma_create_data_internal *cdata = &r_xprt->rx_data;
const struct rpcrdma_connect_private *pmsg = param->private_data;
unsigned int rsize, wsize;
wsize = rpcrdma_decode_buffer_size(pmsg->cp_recv_size);
}
- if (rsize < cdata->inline_rsize)
- cdata->inline_rsize = rsize;
- if (wsize < cdata->inline_wsize)
- cdata->inline_wsize = wsize;
- dprintk("RPC: %s: max send %u, max recv %u\n",
- __func__, cdata->inline_wsize, cdata->inline_rsize);
+ if (rsize < r_xprt->rx_ep.rep_inline_recv)
+ r_xprt->rx_ep.rep_inline_recv = rsize;
+ if (wsize < r_xprt->rx_ep.rep_inline_send)
+ r_xprt->rx_ep.rep_inline_send = wsize;
+ dprintk("RPC: %s: max send %u, max recv %u\n", __func__,
+ r_xprt->rx_ep.rep_inline_send,
+ r_xprt->rx_ep.rep_inline_recv);
rpcrdma_set_max_header_sizes(r_xprt);
}
case RDMA_CM_EVENT_DEVICE_REMOVAL:
#if IS_ENABLED(CONFIG_SUNRPC_DEBUG)
pr_info("rpcrdma: removing device %s for %s:%s\n",
- ia->ri_device->name,
+ ia->ri_id->device->name,
rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt));
#endif
set_bit(RPCRDMA_IAF_REMOVING, &ia->ri_flags);
wait_for_completion(&ia->ri_remove_done);
ia->ri_id = NULL;
- ia->ri_device = NULL;
/* Return 1 to ensure the core destroys the id. */
return 1;
case RDMA_CM_EVENT_ESTABLISHED:
dprintk("RPC: %s: %s:%s on %s/frwr: %s\n", __func__,
rpcrdma_addrstr(r_xprt), rpcrdma_portstr(r_xprt),
- ia->ri_device->name, rdma_event_msg(event->event));
+ ia->ri_id->device->name, rdma_event_msg(event->event));
return 0;
}
rc = PTR_ERR(ia->ri_id);
goto out_err;
}
- ia->ri_device = ia->ri_id->device;
- ia->ri_pd = ib_alloc_pd(ia->ri_device, 0);
+ ia->ri_pd = ib_alloc_pd(ia->ri_id->device, 0);
if (IS_ERR(ia->ri_pd)) {
rc = PTR_ERR(ia->ri_pd);
pr_err("rpcrdma: ib_alloc_pd() returned %d\n", rc);
switch (xprt_rdma_memreg_strategy) {
case RPCRDMA_FRWR:
- if (frwr_is_supported(ia))
+ if (frwr_is_supported(ia->ri_id->device))
break;
/*FALLTHROUGH*/
default:
pr_err("rpcrdma: Device %s does not support memreg mode %d\n",
- ia->ri_device->name, xprt_rdma_memreg_strategy);
+ ia->ri_id->device->name, xprt_rdma_memreg_strategy);
rc = -EINVAL;
goto out_err;
}
* mappings and MRs are gone.
*/
list_for_each_entry(rep, &buf->rb_recv_bufs, rr_list)
- rpcrdma_dma_unmap_regbuf(rep->rr_rdmabuf);
+ rpcrdma_regbuf_dma_unmap(rep->rr_rdmabuf);
list_for_each_entry(req, &buf->rb_allreqs, rl_all) {
- rpcrdma_dma_unmap_regbuf(req->rl_rdmabuf);
- rpcrdma_dma_unmap_regbuf(req->rl_sendbuf);
- rpcrdma_dma_unmap_regbuf(req->rl_recvbuf);
+ rpcrdma_regbuf_dma_unmap(req->rl_rdmabuf);
+ rpcrdma_regbuf_dma_unmap(req->rl_sendbuf);
+ rpcrdma_regbuf_dma_unmap(req->rl_recvbuf);
}
rpcrdma_mrs_destroy(buf);
ib_dealloc_pd(ia->ri_pd);
rdma_destroy_id(ia->ri_id);
}
ia->ri_id = NULL;
- ia->ri_device = NULL;
/* If the pd is still busy, xprtrdma missed freeing a resource */
if (ia->ri_pd && !IS_ERR(ia->ri_pd))
ia->ri_pd = NULL;
}
-/*
- * Create unconnected endpoint.
+/**
+ * rpcrdma_ep_create - Create unconnected endpoint
+ * @r_xprt: transport to instantiate
+ *
+ * Returns zero on success, or a negative errno.
*/
-int
-rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia,
- struct rpcrdma_create_data_internal *cdata)
+int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt)
{
+ struct rpcrdma_ep *ep = &r_xprt->rx_ep;
+ struct rpcrdma_ia *ia = &r_xprt->rx_ia;
struct rpcrdma_connect_private *pmsg = &ep->rep_cm_private;
struct ib_cq *sendcq, *recvcq;
unsigned int max_sge;
int rc;
- max_sge = min_t(unsigned int, ia->ri_device->attrs.max_send_sge,
+ ep->rep_max_requests = xprt_rdma_slot_table_entries;
+ ep->rep_inline_send = xprt_rdma_max_inline_write;
+ ep->rep_inline_recv = xprt_rdma_max_inline_read;
+
+ max_sge = min_t(unsigned int, ia->ri_id->device->attrs.max_send_sge,
RPCRDMA_MAX_SEND_SGES);
if (max_sge < RPCRDMA_MIN_SEND_SGES) {
pr_warn("rpcrdma: HCA provides only %d send SGEs\n", max_sge);
}
ia->ri_max_send_sges = max_sge;
- rc = frwr_open(ia, ep, cdata);
+ rc = frwr_open(ia, ep);
if (rc)
return rc;
ep->rep_attr.cap.max_send_sge,
ep->rep_attr.cap.max_recv_sge);
- /* set trigger for requesting send completion */
- ep->rep_send_batch = min_t(unsigned int, RPCRDMA_MAX_SEND_BATCH,
- cdata->max_requests >> 2);
+ ep->rep_send_batch = ep->rep_max_requests >> 3;
ep->rep_send_count = ep->rep_send_batch;
init_waitqueue_head(&ep->rep_connect_wait);
ep->rep_receive_count = 0;
- sendcq = ib_alloc_cq(ia->ri_device, NULL,
+ sendcq = ib_alloc_cq(ia->ri_id->device, NULL,
ep->rep_attr.cap.max_send_wr + 1,
- ia->ri_device->num_comp_vectors > 1 ? 1 : 0,
+ ia->ri_id->device->num_comp_vectors > 1 ? 1 : 0,
IB_POLL_WORKQUEUE);
if (IS_ERR(sendcq)) {
rc = PTR_ERR(sendcq);
goto out1;
}
- recvcq = ib_alloc_cq(ia->ri_device, NULL,
+ recvcq = ib_alloc_cq(ia->ri_id->device, NULL,
ep->rep_attr.cap.max_recv_wr + 1,
0, IB_POLL_WORKQUEUE);
if (IS_ERR(recvcq)) {
pmsg->cp_magic = rpcrdma_cmp_magic;
pmsg->cp_version = RPCRDMA_CMP_VERSION;
pmsg->cp_flags |= RPCRDMA_CMP_F_SND_W_INV_OK;
- pmsg->cp_send_size = rpcrdma_encode_buffer_size(cdata->inline_wsize);
- pmsg->cp_recv_size = rpcrdma_encode_buffer_size(cdata->inline_rsize);
+ pmsg->cp_send_size = rpcrdma_encode_buffer_size(ep->rep_inline_send);
+ pmsg->cp_recv_size = rpcrdma_encode_buffer_size(ep->rep_inline_recv);
ep->rep_remote_cma.private_data = pmsg;
ep->rep_remote_cma.private_data_len = sizeof(*pmsg);
/* Client offers RDMA Read but does not initiate */
ep->rep_remote_cma.initiator_depth = 0;
ep->rep_remote_cma.responder_resources =
- min_t(int, U8_MAX, ia->ri_device->attrs.max_qp_rd_atom);
+ min_t(int, U8_MAX, ia->ri_id->device->attrs.max_qp_rd_atom);
/* Limit transport retries so client can detect server
* GID changes quickly. RPC layer handles re-establishing
return rc;
}
-/*
- * rpcrdma_ep_destroy
+/**
+ * rpcrdma_ep_destroy - Disconnect and destroy endpoint.
+ * @r_xprt: transport instance to shut down
*
- * Disconnect and destroy endpoint. After this, the only
- * valid operations on the ep are to free it (if dynamically
- * allocated) or re-create it.
*/
-void
-rpcrdma_ep_destroy(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia)
+void rpcrdma_ep_destroy(struct rpcrdma_xprt *r_xprt)
{
+ struct rpcrdma_ep *ep = &r_xprt->rx_ep;
+ struct rpcrdma_ia *ia = &r_xprt->rx_ia;
+
if (ia->ri_id && ia->ri_id->qp) {
rpcrdma_ep_disconnect(ep, ia);
rdma_destroy_qp(ia->ri_id);
goto out1;
rc = -ENOMEM;
- err = rpcrdma_ep_create(ep, ia, &r_xprt->rx_data);
+ err = rpcrdma_ep_create(r_xprt);
if (err) {
pr_err("rpcrdma: rpcrdma_ep_create returned %d\n", err);
goto out2;
return 0;
out3:
- rpcrdma_ep_destroy(ep, ia);
+ rpcrdma_ep_destroy(r_xprt);
out2:
rpcrdma_ia_close(ia);
out1:
*/
old = id;
rc = -ENETUNREACH;
- if (ia->ri_device != id->device) {
+ if (ia->ri_id->device != id->device) {
pr_err("rpcrdma: can't reconnect on different device!\n");
goto out_destroy;
}
*/
/* rpcrdma_sendctxs_destroy() assumes caller has already quiesced
- * queue activity, and ib_drain_qp has flushed all remaining Send
- * requests.
+ * queue activity, and rpcrdma_xprt_drain has flushed all remaining
+ * Send requests.
*/
static void rpcrdma_sendctxs_destroy(struct rpcrdma_buffer *buf)
{
/**
* rpcrdma_sendctx_get_locked - Acquire a send context
- * @buf: transport buffers from which to acquire an unused context
+ * @r_xprt: controlling transport instance
*
* Returns pointer to a free send completion context; or NULL if
* the queue is empty.
*
* Usage: Called to acquire an SGE array before preparing a Send WR.
*
- * The caller serializes calls to this function (per rpcrdma_buffer),
- * and provides an effective memory barrier that flushes the new value
+ * The caller serializes calls to this function (per transport), and
+ * provides an effective memory barrier that flushes the new value
* of rb_sc_head.
*/
-struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_buffer *buf)
+struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt)
{
- struct rpcrdma_xprt *r_xprt;
+ struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
struct rpcrdma_sendctx *sc;
unsigned long next_head;
* backing up. Cause the caller to pause and try again.
*/
set_bit(RPCRDMA_BUF_F_EMPTY_SCQ, &buf->rb_flags);
- r_xprt = container_of(buf, struct rpcrdma_xprt, rx_buf);
r_xprt->rx_stats.empty_sendctx_q++;
return NULL;
}
* Usage: Called from Send completion to return a sendctxt
* to the queue.
*
- * The caller serializes calls to this function (per rpcrdma_buffer).
+ * The caller serializes calls to this function (per transport).
*/
static void
rpcrdma_sendctx_put_locked(struct rpcrdma_sendctx *sc)
struct rpcrdma_buffer *buf = &sc->sc_xprt->rx_buf;
unsigned long next_tail;
- /* Unmap SGEs of previously completed by unsignaled
+ /* Unmap SGEs of previously completed but unsignaled
* Sends by walking up the queue until @sc is found.
*/
next_tail = buf->rb_sc_tail;
next_tail = rpcrdma_sendctx_next(buf, next_tail);
/* ORDER: item must be accessed _before_ tail is updated */
- rpcrdma_unmap_sendctx(buf->rb_sc_ctxs[next_tail]);
+ rpcrdma_sendctx_unmap(buf->rb_sc_ctxs[next_tail]);
} while (buf->rb_sc_ctxs[next_tail] != sc);
rpcrdma_mrs_create(r_xprt);
}
-struct rpcrdma_req *
-rpcrdma_create_req(struct rpcrdma_xprt *r_xprt)
+/**
+ * rpcrdma_req_create - Allocate an rpcrdma_req object
+ * @r_xprt: controlling r_xprt
+ * @size: initial size, in bytes, of send and receive buffers
+ * @flags: GFP flags passed to memory allocators
+ *
+ * Returns an allocated and fully initialized rpcrdma_req or NULL.
+ */
+struct rpcrdma_req *rpcrdma_req_create(struct rpcrdma_xprt *r_xprt, size_t size,
+ gfp_t flags)
{
struct rpcrdma_buffer *buffer = &r_xprt->rx_buf;
struct rpcrdma_regbuf *rb;
struct rpcrdma_req *req;
- req = kzalloc(sizeof(*req), GFP_KERNEL);
+ req = kzalloc(sizeof(*req), flags);
if (req == NULL)
- return ERR_PTR(-ENOMEM);
+ goto out1;
- rb = rpcrdma_alloc_regbuf(RPCRDMA_HDRBUF_SIZE,
- DMA_TO_DEVICE, GFP_KERNEL);
- if (IS_ERR(rb)) {
- kfree(req);
- return ERR_PTR(-ENOMEM);
- }
+ rb = rpcrdma_regbuf_alloc(RPCRDMA_HDRBUF_SIZE, DMA_TO_DEVICE, flags);
+ if (!rb)
+ goto out2;
req->rl_rdmabuf = rb;
- xdr_buf_init(&req->rl_hdrbuf, rb->rg_base, rdmab_length(rb));
+ xdr_buf_init(&req->rl_hdrbuf, rdmab_data(rb), rdmab_length(rb));
+
+ req->rl_sendbuf = rpcrdma_regbuf_alloc(size, DMA_TO_DEVICE, flags);
+ if (!req->rl_sendbuf)
+ goto out3;
+
+ req->rl_recvbuf = rpcrdma_regbuf_alloc(size, DMA_NONE, flags);
+ if (!req->rl_recvbuf)
+ goto out4;
+
req->rl_buffer = buffer;
INIT_LIST_HEAD(&req->rl_registered);
-
spin_lock(&buffer->rb_lock);
list_add(&req->rl_all, &buffer->rb_allreqs);
spin_unlock(&buffer->rb_lock);
return req;
+
+out4:
+ kfree(req->rl_sendbuf);
+out3:
+ kfree(req->rl_rdmabuf);
+out2:
+ kfree(req);
+out1:
+ return NULL;
}
-static int
-rpcrdma_create_rep(struct rpcrdma_xprt *r_xprt, bool temp)
+static bool rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt, bool temp)
{
- struct rpcrdma_create_data_internal *cdata = &r_xprt->rx_data;
struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
struct rpcrdma_rep *rep;
- int rc;
- rc = -ENOMEM;
rep = kzalloc(sizeof(*rep), GFP_KERNEL);
if (rep == NULL)
goto out;
- rep->rr_rdmabuf = rpcrdma_alloc_regbuf(cdata->inline_rsize,
+ rep->rr_rdmabuf = rpcrdma_regbuf_alloc(r_xprt->rx_ep.rep_inline_recv,
DMA_FROM_DEVICE, GFP_KERNEL);
- if (IS_ERR(rep->rr_rdmabuf)) {
- rc = PTR_ERR(rep->rr_rdmabuf);
+ if (!rep->rr_rdmabuf)
goto out_free;
- }
- xdr_buf_init(&rep->rr_hdrbuf, rep->rr_rdmabuf->rg_base,
+ xdr_buf_init(&rep->rr_hdrbuf, rdmab_data(rep->rr_rdmabuf),
rdmab_length(rep->rr_rdmabuf));
rep->rr_cqe.done = rpcrdma_wc_receive;
spin_lock(&buf->rb_lock);
list_add(&rep->rr_list, &buf->rb_recv_bufs);
spin_unlock(&buf->rb_lock);
- return 0;
+ return true;
out_free:
kfree(rep);
out:
- return rc;
+ return false;
}
-int
-rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt)
+/**
+ * rpcrdma_buffer_create - Create initial set of req/rep objects
+ * @r_xprt: transport instance to (re)initialize
+ *
+ * Returns zero on success, otherwise a negative errno.
+ */
+int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt)
{
struct rpcrdma_buffer *buf = &r_xprt->rx_buf;
int i, rc;
buf->rb_flags = 0;
- buf->rb_max_requests = r_xprt->rx_data.max_requests;
+ buf->rb_max_requests = r_xprt->rx_ep.rep_max_requests;
buf->rb_bc_srv_max_requests = 0;
spin_lock_init(&buf->rb_mrlock);
spin_lock_init(&buf->rb_lock);
INIT_LIST_HEAD(&buf->rb_send_bufs);
INIT_LIST_HEAD(&buf->rb_allreqs);
+
+ rc = -ENOMEM;
for (i = 0; i < buf->rb_max_requests; i++) {
struct rpcrdma_req *req;
- req = rpcrdma_create_req(r_xprt);
- if (IS_ERR(req)) {
- dprintk("RPC: %s: request buffer %d alloc"
- " failed\n", __func__, i);
- rc = PTR_ERR(req);
+ req = rpcrdma_req_create(r_xprt, RPCRDMA_V1_DEF_INLINE_SIZE,
+ GFP_KERNEL);
+ if (!req)
goto out;
- }
list_add(&req->rl_list, &buf->rb_send_bufs);
}
return rc;
}
-static void
-rpcrdma_destroy_rep(struct rpcrdma_rep *rep)
+static void rpcrdma_rep_destroy(struct rpcrdma_rep *rep)
{
- rpcrdma_free_regbuf(rep->rr_rdmabuf);
+ rpcrdma_regbuf_free(rep->rr_rdmabuf);
kfree(rep);
}
{
list_del(&req->rl_all);
- rpcrdma_free_regbuf(req->rl_recvbuf);
- rpcrdma_free_regbuf(req->rl_sendbuf);
- rpcrdma_free_regbuf(req->rl_rdmabuf);
+ rpcrdma_regbuf_free(req->rl_recvbuf);
+ rpcrdma_regbuf_free(req->rl_sendbuf);
+ rpcrdma_regbuf_free(req->rl_rdmabuf);
kfree(req);
}
* rpcrdma_buffer_destroy - Release all hw resources
* @buf: root control block for resources
*
- * ORDERING: relies on a prior ib_drain_qp :
+ * ORDERING: relies on a prior rpcrdma_xprt_drain :
* - No more Send or Receive completions can occur
* - All MRs, reps, and reqs are returned to their free lists
*/
rep = list_first_entry(&buf->rb_recv_bufs,
struct rpcrdma_rep, rr_list);
list_del(&rep->rr_list);
- rpcrdma_destroy_rep(rep);
+ rpcrdma_rep_destroy(rep);
}
while (!list_empty(&buf->rb_send_bufs)) {
if (mr->mr_dir != DMA_NONE) {
trace_xprtrdma_mr_unmap(mr);
- ib_dma_unmap_sg(r_xprt->rx_ia.ri_device,
+ ib_dma_unmap_sg(r_xprt->rx_ia.ri_id->device,
mr->mr_sg, mr->mr_nents, mr->mr_dir);
mr->mr_dir = DMA_NONE;
}
}
spin_unlock(&buffers->rb_lock);
if (rep)
- rpcrdma_destroy_rep(rep);
+ rpcrdma_rep_destroy(rep);
}
/*
list_add(&rep->rr_list, &buffers->rb_recv_bufs);
spin_unlock(&buffers->rb_lock);
} else {
- rpcrdma_destroy_rep(rep);
+ rpcrdma_rep_destroy(rep);
}
}
-/**
- * rpcrdma_alloc_regbuf - allocate and DMA-map memory for SEND/RECV buffers
- * @size: size of buffer to be allocated, in bytes
- * @direction: direction of data movement
- * @flags: GFP flags
- *
- * Returns an ERR_PTR, or a pointer to a regbuf, a buffer that
- * can be persistently DMA-mapped for I/O.
+/* Returns a pointer to a rpcrdma_regbuf object, or NULL.
*
* xprtrdma uses a regbuf for posting an outgoing RDMA SEND, or for
* receiving the payload of RDMA RECV operations. During Long Calls
* or Replies they may be registered externally via frwr_map.
*/
-struct rpcrdma_regbuf *
-rpcrdma_alloc_regbuf(size_t size, enum dma_data_direction direction,
+static struct rpcrdma_regbuf *
+rpcrdma_regbuf_alloc(size_t size, enum dma_data_direction direction,
gfp_t flags)
{
struct rpcrdma_regbuf *rb;
- rb = kmalloc(sizeof(*rb) + size, flags);
- if (rb == NULL)
- return ERR_PTR(-ENOMEM);
+ rb = kmalloc(sizeof(*rb), flags);
+ if (!rb)
+ return NULL;
+ rb->rg_data = kmalloc(size, flags);
+ if (!rb->rg_data) {
+ kfree(rb);
+ return NULL;
+ }
rb->rg_device = NULL;
rb->rg_direction = direction;
rb->rg_iov.length = size;
-
return rb;
}
/**
- * __rpcrdma_map_regbuf - DMA-map a regbuf
- * @ia: controlling rpcrdma_ia
+ * rpcrdma_regbuf_realloc - re-allocate a SEND/RECV buffer
+ * @rb: regbuf to reallocate
+ * @size: size of buffer to be allocated, in bytes
+ * @flags: GFP flags
+ *
+ * Returns true if reallocation was successful. If false is
+ * returned, @rb is left untouched.
+ */
+bool rpcrdma_regbuf_realloc(struct rpcrdma_regbuf *rb, size_t size, gfp_t flags)
+{
+ void *buf;
+
+ buf = kmalloc(size, flags);
+ if (!buf)
+ return false;
+
+ rpcrdma_regbuf_dma_unmap(rb);
+ kfree(rb->rg_data);
+
+ rb->rg_data = buf;
+ rb->rg_iov.length = size;
+ return true;
+}
+
+/**
+ * __rpcrdma_regbuf_dma_map - DMA-map a regbuf
+ * @r_xprt: controlling transport instance
* @rb: regbuf to be mapped
+ *
+ * Returns true if the buffer is now DMA mapped to @r_xprt's device
*/
-bool
-__rpcrdma_dma_map_regbuf(struct rpcrdma_ia *ia, struct rpcrdma_regbuf *rb)
+bool __rpcrdma_regbuf_dma_map(struct rpcrdma_xprt *r_xprt,
+ struct rpcrdma_regbuf *rb)
{
- struct ib_device *device = ia->ri_device;
+ struct ib_device *device = r_xprt->rx_ia.ri_id->device;
if (rb->rg_direction == DMA_NONE)
return false;
- rb->rg_iov.addr = ib_dma_map_single(device,
- (void *)rb->rg_base,
- rdmab_length(rb),
- rb->rg_direction);
+ rb->rg_iov.addr = ib_dma_map_single(device, rdmab_data(rb),
+ rdmab_length(rb), rb->rg_direction);
if (ib_dma_mapping_error(device, rdmab_addr(rb))) {
trace_xprtrdma_dma_maperr(rdmab_addr(rb));
return false;
}
rb->rg_device = device;
- rb->rg_iov.lkey = ia->ri_pd->local_dma_lkey;
+ rb->rg_iov.lkey = r_xprt->rx_ia.ri_pd->local_dma_lkey;
return true;
}
-static void
-rpcrdma_dma_unmap_regbuf(struct rpcrdma_regbuf *rb)
+static void rpcrdma_regbuf_dma_unmap(struct rpcrdma_regbuf *rb)
{
if (!rb)
return;
if (!rpcrdma_regbuf_is_mapped(rb))
return;
- ib_dma_unmap_single(rb->rg_device, rdmab_addr(rb),
- rdmab_length(rb), rb->rg_direction);
+ ib_dma_unmap_single(rb->rg_device, rdmab_addr(rb), rdmab_length(rb),
+ rb->rg_direction);
rb->rg_device = NULL;
}
-/**
- * rpcrdma_free_regbuf - deregister and free registered buffer
- * @rb: regbuf to be deregistered and freed
- */
-void
-rpcrdma_free_regbuf(struct rpcrdma_regbuf *rb)
+static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb)
{
- rpcrdma_dma_unmap_regbuf(rb);
+ rpcrdma_regbuf_dma_unmap(rb);
+ if (rb)
+ kfree(rb->rg_data);
kfree(rb);
}
list_del(&rep->rr_list);
spin_unlock(&buf->rb_lock);
if (!rep) {
- if (rpcrdma_create_rep(r_xprt, temp))
+ if (!rpcrdma_rep_create(r_xprt, temp))
break;
continue;
}
rb = rep->rr_rdmabuf;
- if (!rpcrdma_regbuf_is_mapped(rb)) {
- if (!__rpcrdma_dma_map_regbuf(&r_xprt->rx_ia, rb)) {
- rpcrdma_recv_buffer_put(rep);
- break;
- }
+ if (!rpcrdma_regbuf_dma_map(r_xprt, rb)) {
+ rpcrdma_recv_buffer_put(rep);
+ break;
}
trace_xprtrdma_post_recv(rep->rr_recv_wr.wr_cqe);
* Interface Adapter -- one per transport instance
*/
struct rpcrdma_ia {
- struct ib_device *ri_device;
struct rdma_cm_id *ri_id;
struct ib_pd *ri_pd;
- struct completion ri_done;
- struct completion ri_remove_done;
int ri_async_rc;
unsigned int ri_max_segs;
unsigned int ri_max_frwr_depth;
- unsigned int ri_max_inline_write;
- unsigned int ri_max_inline_read;
unsigned int ri_max_send_sges;
bool ri_implicit_roundup;
enum ib_mr_type ri_mrtype;
unsigned long ri_flags;
+ struct completion ri_done;
+ struct completion ri_remove_done;
};
enum {
struct rpcrdma_ep {
unsigned int rep_send_count;
unsigned int rep_send_batch;
+ unsigned int rep_max_inline_send;
+ unsigned int rep_max_inline_recv;
int rep_connected;
struct ib_qp_init_attr rep_attr;
wait_queue_head_t rep_connect_wait;
struct rpcrdma_connect_private rep_cm_private;
struct rdma_conn_param rep_remote_cma;
+ unsigned int rep_max_requests; /* set by /proc */
+ unsigned int rep_inline_send; /* negotiated */
+ unsigned int rep_inline_recv; /* negotiated */
int rep_receive_count;
};
/* Pre-allocate extra Work Requests for handling backward receives
* and sends. This is a fixed value because the Work Queues are
- * allocated when the forward channel is set up.
+ * allocated when the forward channel is set up, long before the
+ * backchannel is provisioned. This value is two times
+ * NFS4_DEF_CB_SLOT_TABLE_SIZE.
*/
#if defined(CONFIG_SUNRPC_BACKCHANNEL)
-#define RPCRDMA_BACKWARD_WRS (8)
+#define RPCRDMA_BACKWARD_WRS (32)
#else
-#define RPCRDMA_BACKWARD_WRS (0)
+#define RPCRDMA_BACKWARD_WRS (0)
#endif
/* Registered buffer -- registered kmalloc'd memory for RDMA SEND/RECV
struct ib_sge rg_iov;
struct ib_device *rg_device;
enum dma_data_direction rg_direction;
- __be32 rg_base[0] __attribute__ ((aligned(256)));
+ void *rg_data;
};
-static inline u64
-rdmab_addr(struct rpcrdma_regbuf *rb)
+static inline u64 rdmab_addr(struct rpcrdma_regbuf *rb)
{
return rb->rg_iov.addr;
}
-static inline u32
-rdmab_length(struct rpcrdma_regbuf *rb)
+static inline u32 rdmab_length(struct rpcrdma_regbuf *rb)
{
return rb->rg_iov.length;
}
-static inline u32
-rdmab_lkey(struct rpcrdma_regbuf *rb)
+static inline u32 rdmab_lkey(struct rpcrdma_regbuf *rb)
{
return rb->rg_iov.lkey;
}
-static inline struct ib_device *
-rdmab_device(struct rpcrdma_regbuf *rb)
+static inline struct ib_device *rdmab_device(struct rpcrdma_regbuf *rb)
{
return rb->rg_device;
}
+static inline void *rdmab_data(const struct rpcrdma_regbuf *rb)
+{
+ return rb->rg_data;
+}
+
#define RPCRDMA_DEF_GFP (GFP_NOIO | __GFP_NOWARN)
/* To ensure a transport can always make forward progress,
struct rpcrdma_sendctx {
struct ib_send_wr sc_wr;
struct ib_cqe sc_cqe;
+ struct ib_device *sc_device;
struct rpcrdma_xprt *sc_xprt;
struct rpcrdma_req *sc_req;
unsigned int sc_unmap_count;
struct ib_sge sc_sges[];
};
-/* Limit the number of SGEs that can be unmapped during one
- * Send completion. This caps the amount of work a single
- * completion can do before returning to the provider.
- *
- * Setting this to zero disables Send completion batching.
- */
-enum {
- RPCRDMA_MAX_SEND_BATCH = 7,
-};
-
/*
* struct rpcrdma_mr - external memory region metadata
*
* An external memory region is any buffer or page that is registered
* on the fly (ie, not pre-registered).
- *
- * Each rpcrdma_buffer has a list of free MWs anchored in rb_mrs. During
- * call_allocate, rpcrdma_buffer_get() assigns one to each segment in
- * an rpcrdma_req. Then rpcrdma_register_external() grabs these to keep
- * track of registration metadata while each RPC is pending.
- * rpcrdma_deregister_external() uses this metadata to unmap and
- * release these resources when an RPC is complete.
*/
enum rpcrdma_frwr_state {
FRWR_IS_INVALID, /* ready to be used */
RPCRDMA_BUF_F_EMPTY_SCQ = 0,
};
-/*
- * Internal structure for transport instance creation. This
- * exists primarily for modularity.
- *
- * This data should be set with mount options
- */
-struct rpcrdma_create_data_internal {
- unsigned int max_requests; /* max requests (slots) in flight */
- unsigned int rsize; /* mount rsize - max read hdr+data */
- unsigned int wsize; /* mount wsize - max write hdr+data */
- unsigned int inline_rsize; /* max non-rdma read data payload */
- unsigned int inline_wsize; /* max non-rdma write data payload */
-};
-
/*
* Statistics for RPCRDMA
*/
struct rpcrdma_ia rx_ia;
struct rpcrdma_ep rx_ep;
struct rpcrdma_buffer rx_buf;
- struct rpcrdma_create_data_internal rx_data;
struct delayed_work rx_connect_worker;
struct rpcrdma_stats rx_stats;
};
#define rpcx_to_rdmax(x) container_of(x, struct rpcrdma_xprt, rx_xprt)
-#define rpcx_to_rdmad(x) (rpcx_to_rdmax(x)->rx_data)
static inline const char *
rpcrdma_addrstr(const struct rpcrdma_xprt *r_xprt)
/*
* Endpoint calls - xprtrdma/verbs.c
*/
-int rpcrdma_ep_create(struct rpcrdma_ep *, struct rpcrdma_ia *,
- struct rpcrdma_create_data_internal *);
-void rpcrdma_ep_destroy(struct rpcrdma_ep *, struct rpcrdma_ia *);
+int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt);
+void rpcrdma_ep_destroy(struct rpcrdma_xprt *r_xprt);
int rpcrdma_ep_connect(struct rpcrdma_ep *, struct rpcrdma_ia *);
void rpcrdma_ep_disconnect(struct rpcrdma_ep *, struct rpcrdma_ia *);
/*
* Buffer calls - xprtrdma/verbs.c
*/
-struct rpcrdma_req *rpcrdma_create_req(struct rpcrdma_xprt *);
+struct rpcrdma_req *rpcrdma_req_create(struct rpcrdma_xprt *r_xprt, size_t size,
+ gfp_t flags);
void rpcrdma_req_destroy(struct rpcrdma_req *req);
int rpcrdma_buffer_create(struct rpcrdma_xprt *);
void rpcrdma_buffer_destroy(struct rpcrdma_buffer *);
-struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_buffer *buf);
+struct rpcrdma_sendctx *rpcrdma_sendctx_get_locked(struct rpcrdma_xprt *r_xprt);
struct rpcrdma_mr *rpcrdma_mr_get(struct rpcrdma_xprt *r_xprt);
void rpcrdma_mr_put(struct rpcrdma_mr *mr);
void rpcrdma_buffer_put(struct rpcrdma_req *);
void rpcrdma_recv_buffer_put(struct rpcrdma_rep *);
-struct rpcrdma_regbuf *rpcrdma_alloc_regbuf(size_t, enum dma_data_direction,
- gfp_t);
-bool __rpcrdma_dma_map_regbuf(struct rpcrdma_ia *, struct rpcrdma_regbuf *);
-void rpcrdma_free_regbuf(struct rpcrdma_regbuf *);
+bool rpcrdma_regbuf_realloc(struct rpcrdma_regbuf *rb, size_t size,
+ gfp_t flags);
+bool __rpcrdma_regbuf_dma_map(struct rpcrdma_xprt *r_xprt,
+ struct rpcrdma_regbuf *rb);
-static inline bool
-rpcrdma_regbuf_is_mapped(struct rpcrdma_regbuf *rb)
+/**
+ * rpcrdma_regbuf_is_mapped - check if buffer is DMA mapped
+ *
+ * Returns true if the buffer is now mapped to rb->rg_device.
+ */
+static inline bool rpcrdma_regbuf_is_mapped(struct rpcrdma_regbuf *rb)
{
return rb->rg_device != NULL;
}
-static inline bool
-rpcrdma_dma_map_regbuf(struct rpcrdma_ia *ia, struct rpcrdma_regbuf *rb)
+/**
+ * rpcrdma_regbuf_dma_map - DMA-map a regbuf
+ * @r_xprt: controlling transport instance
+ * @rb: regbuf to be mapped
+ *
+ * Returns true if the buffer is currently DMA mapped.
+ */
+static inline bool rpcrdma_regbuf_dma_map(struct rpcrdma_xprt *r_xprt,
+ struct rpcrdma_regbuf *rb)
{
if (likely(rpcrdma_regbuf_is_mapped(rb)))
return true;
- return __rpcrdma_dma_map_regbuf(ia, rb);
+ return __rpcrdma_regbuf_dma_map(r_xprt, rb);
}
/*
/* Memory registration calls xprtrdma/frwr_ops.c
*/
-bool frwr_is_supported(struct rpcrdma_ia *);
-int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep,
- struct rpcrdma_create_data_internal *cdata);
+bool frwr_is_supported(struct ib_device *device);
+int frwr_open(struct rpcrdma_ia *ia, struct rpcrdma_ep *ep);
int frwr_init_mr(struct rpcrdma_ia *ia, struct rpcrdma_mr *mr);
void frwr_release_mr(struct rpcrdma_mr *mr);
size_t frwr_maxpages(struct rpcrdma_xprt *r_xprt);
struct rpcrdma_req *req, u32 hdrlen,
struct xdr_buf *xdr,
enum rpcrdma_chunktype rtype);
-void rpcrdma_unmap_sendctx(struct rpcrdma_sendctx *sc);
+void rpcrdma_sendctx_unmap(struct rpcrdma_sendctx *sc);
int rpcrdma_marshal_req(struct rpcrdma_xprt *r_xprt, struct rpc_rqst *rqst);
void rpcrdma_set_max_header_sizes(struct rpcrdma_xprt *);
void rpcrdma_complete_rqst(struct rpcrdma_rep *rep);
/* RPC/RDMA module init - xprtrdma/transport.c
*/
+extern unsigned int xprt_rdma_slot_table_entries;
extern unsigned int xprt_rdma_max_inline_read;
+extern unsigned int xprt_rdma_max_inline_write;
void xprt_rdma_format_addresses(struct rpc_xprt *xprt, struct sockaddr *sap);
void xprt_rdma_free_addresses(struct rpc_xprt *xprt);
void xprt_rdma_close(struct rpc_xprt *xprt);
* we'll need to figure out how to pass a namespace to
* connect.
*/
+ task->tk_rpc_status = -ENOTCONN;
rpc_exit(task, -ENOTCONN);
return;
}
.buf_free = rpc_free,
.prepare_request = xs_stream_prepare_request,
.send_request = xs_local_send_request,
- .set_retrans_timeout = xprt_set_retrans_timeout_def,
+ .wait_for_reply_request = xprt_wait_for_reply_request_def,
.close = xs_close,
.destroy = xs_destroy,
.print_stats = xs_local_print_stats,
.buf_alloc = rpc_malloc,
.buf_free = rpc_free,
.send_request = xs_udp_send_request,
- .set_retrans_timeout = xprt_set_retrans_timeout_rtt,
+ .wait_for_reply_request = xprt_wait_for_reply_request_rtt,
.timer = xs_udp_timer,
.release_request = xprt_release_rqst_cong,
.close = xs_close,
.buf_free = rpc_free,
.prepare_request = xs_stream_prepare_request,
.send_request = xs_tcp_send_request,
- .set_retrans_timeout = xprt_set_retrans_timeout_def,
+ .wait_for_reply_request = xprt_wait_for_reply_request_def,
.close = xs_tcp_shutdown,
.destroy = xs_destroy,
.set_connect_timeout = xs_tcp_set_connect_timeout,
.buf_alloc = bc_malloc,
.buf_free = bc_free,
.send_request = bc_send_request,
- .set_retrans_timeout = xprt_set_retrans_timeout_def,
+ .wait_for_reply_request = xprt_wait_for_reply_request_def,
.close = bc_close,
.destroy = bc_destroy,
.print_stats = xs_tcp_print_stats,
hostprogs-y += xdp_fwd
hostprogs-y += task_fd_query
hostprogs-y += xdp_sample_pkts
+hostprogs-y += ibumad
hostprogs-y += hbm
# Libbpf dependencies
xdp_fwd-objs := xdp_fwd_user.o
task_fd_query-objs := bpf_load.o task_fd_query_user.o $(TRACE_HELPERS)
xdp_sample_pkts-objs := xdp_sample_pkts_user.o $(TRACE_HELPERS)
+ibumad-objs := bpf_load.o ibumad_user.o $(TRACE_HELPERS)
hbm-objs := bpf_load.o hbm.o $(CGROUP_HELPERS)
# Tell kbuild to always build the programs
always += xdp_fwd_kern.o
always += task_fd_query_kern.o
always += xdp_sample_pkts_kern.o
+always += ibumad_kern.o
always += hbm_out_kern.o
KBUILD_HOSTCFLAGS += -I$(objtree)/usr/include
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+
+/**
+ * ibumad BPF sample kernel side
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * Copyright(c) 2018 Ira Weiny, Intel Corporation
+ */
+
+#define KBUILD_MODNAME "ibumad_count_pkts_by_class"
+#include <uapi/linux/bpf.h>
+
+#include "bpf_helpers.h"
+
+
+struct bpf_map_def SEC("maps") read_count = {
+ .type = BPF_MAP_TYPE_ARRAY,
+ .key_size = sizeof(u32), /* class; u32 required */
+ .value_size = sizeof(u64), /* count of mads read */
+ .max_entries = 256, /* Room for all Classes */
+};
+
+struct bpf_map_def SEC("maps") write_count = {
+ .type = BPF_MAP_TYPE_ARRAY,
+ .key_size = sizeof(u32), /* class; u32 required */
+ .value_size = sizeof(u64), /* count of mads written */
+ .max_entries = 256, /* Room for all Classes */
+};
+
+#undef DEBUG
+#ifdef DEBUG
+#define bpf_debug(fmt, ...) \
+({ \
+ char ____fmt[] = fmt; \
+ bpf_trace_printk(____fmt, sizeof(____fmt), \
+ ##__VA_ARGS__); \
+})
+#else
+#define bpf_debug(fmt, ...)
+#endif
+
+/* Taken from the current format defined in
+ * include/trace/events/ib_umad.h
+ * and
+ * /sys/kernel/debug/tracing/events/ib_umad/ib_umad_read/format
+ * /sys/kernel/debug/tracing/events/ib_umad/ib_umad_write/format
+ */
+struct ib_umad_rw_args {
+ u64 pad;
+ u8 port_num;
+ u8 sl;
+ u8 path_bits;
+ u8 grh_present;
+ u32 id;
+ u32 status;
+ u32 timeout_ms;
+ u32 retires;
+ u32 length;
+ u32 qpn;
+ u32 qkey;
+ u8 gid_index;
+ u8 hop_limit;
+ u16 lid;
+ u16 attr_id;
+ u16 pkey_index;
+ u8 base_version;
+ u8 mgmt_class;
+ u8 class_version;
+ u8 method;
+ u32 flow_label;
+ u16 mad_status;
+ u16 class_specific;
+ u32 attr_mod;
+ u64 tid;
+ u8 gid[16];
+ u32 dev_index;
+ u8 traffic_class;
+};
+
+SEC("tracepoint/ib_umad/ib_umad_read_recv")
+int on_ib_umad_read_recv(struct ib_umad_rw_args *ctx)
+{
+ u64 zero = 0, *val;
+ u8 class = ctx->mgmt_class;
+
+ bpf_debug("ib_umad read recv : class 0x%x\n", class);
+
+ val = bpf_map_lookup_elem(&read_count, &class);
+ if (!val) {
+ bpf_map_update_elem(&read_count, &class, &zero, BPF_NOEXIST);
+ val = bpf_map_lookup_elem(&read_count, &class);
+ if (!val)
+ return 0;
+ }
+
+ (*val) += 1;
+
+ return 0;
+}
+SEC("tracepoint/ib_umad/ib_umad_read_send")
+int on_ib_umad_read_send(struct ib_umad_rw_args *ctx)
+{
+ u64 zero = 0, *val;
+ u8 class = ctx->mgmt_class;
+
+ bpf_debug("ib_umad read send : class 0x%x\n", class);
+
+ val = bpf_map_lookup_elem(&read_count, &class);
+ if (!val) {
+ bpf_map_update_elem(&read_count, &class, &zero, BPF_NOEXIST);
+ val = bpf_map_lookup_elem(&read_count, &class);
+ if (!val)
+ return 0;
+ }
+
+ (*val) += 1;
+
+ return 0;
+}
+SEC("tracepoint/ib_umad/ib_umad_write")
+int on_ib_umad_write(struct ib_umad_rw_args *ctx)
+{
+ u64 zero = 0, *val;
+ u8 class = ctx->mgmt_class;
+
+ bpf_debug("ib_umad write : class 0x%x\n", class);
+
+ val = bpf_map_lookup_elem(&write_count, &class);
+ if (!val) {
+ bpf_map_update_elem(&write_count, &class, &zero, BPF_NOEXIST);
+ val = bpf_map_lookup_elem(&write_count, &class);
+ if (!val)
+ return 0;
+ }
+
+ (*val) += 1;
+
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+
+/**
+ * ibumad BPF sample user side
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * Copyright(c) 2018 Ira Weiny, Intel Corporation
+ */
+
+#include <linux/bpf.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/types.h>
+#include <limits.h>
+
+#include <sys/resource.h>
+#include <getopt.h>
+#include <net/if.h>
+
+#include "bpf_load.h"
+#include "bpf_util.h"
+#include "bpf/libbpf.h"
+
+static void dump_counts(int fd)
+{
+ __u32 key;
+ __u64 value;
+
+ for (key = 0; key < 256; key++) {
+ if (bpf_map_lookup_elem(fd, &key, &value)) {
+ printf("failed to read key %u\n", key);
+ continue;
+ }
+ if (value)
+ printf("0x%02x : %llu\n", key, value);
+ }
+}
+
+static void dump_all_counts(void)
+{
+ printf("Read 'Class : count'\n");
+ dump_counts(map_fd[0]);
+ printf("Write 'Class : count'\n");
+ dump_counts(map_fd[1]);
+}
+
+static void dump_exit(int sig)
+{
+ dump_all_counts();
+ exit(0);
+}
+
+static const struct option long_options[] = {
+ {"help", no_argument, NULL, 'h'},
+ {"delay", required_argument, NULL, 'd'},
+};
+
+static void usage(char *cmd)
+{
+ printf("eBPF test program to count packets from various IP addresses\n"
+ "Usage: %s <options>\n"
+ " --help, -h this menu\n"
+ " --delay, -d <delay> wait <delay> sec between prints [1 - 1000000]\n"
+ , cmd
+ );
+}
+
+int main(int argc, char **argv)
+{
+ unsigned long delay = 5;
+ int longindex = 0;
+ int opt;
+ char bpf_file[256];
+
+ /* Create the eBPF kernel code path name.
+ * This follows the pattern of all of the other bpf samples
+ */
+ snprintf(bpf_file, sizeof(bpf_file), "%s_kern.o", argv[0]);
+
+ /* Do one final dump when exiting */
+ signal(SIGINT, dump_exit);
+ signal(SIGTERM, dump_exit);
+
+ while ((opt = getopt_long(argc, argv, "hd:rSw",
+ long_options, &longindex)) != -1) {
+ switch (opt) {
+ case 'd':
+ delay = strtoul(optarg, NULL, 0);
+ if (delay == ULONG_MAX || delay < 0 ||
+ delay > 1000000) {
+ fprintf(stderr, "ERROR: invalid delay : %s\n",
+ optarg);
+ usage(argv[0]);
+ return 1;
+ }
+ break;
+ default:
+ case 'h':
+ usage(argv[0]);
+ return 1;
+ }
+ }
+
+ if (load_bpf_file(bpf_file)) {
+ fprintf(stderr, "ERROR: failed to load eBPF from file : %s\n",
+ bpf_file);
+ return 1;
+ }
+
+ while (1) {
+ sleep(delay);
+ dump_all_counts();
+ }
+
+ return 0;
+}
#define SMACK_HASH_SLOTS 16
extern struct hlist_head smack_known_hash[SMACK_HASH_SLOTS];
+extern struct kmem_cache *smack_rule_cache;
static inline struct task_smack *smack_cred(const struct cred *cred)
{
static LIST_HEAD(smk_ipv6_port_list);
#endif
static struct kmem_cache *smack_inode_cache;
+struct kmem_cache *smack_rule_cache;
int smack_enabled;
#define A(s) {"smack"#s, sizeof("smack"#s) - 1, Opt_##s}
int rc = 0;
list_for_each_entry_rcu(orp, ohead, list) {
- nrp = kzalloc(sizeof(struct smack_rule), gfp);
+ nrp = kmem_cache_zalloc(smack_rule_cache, gfp);
if (nrp == NULL) {
rc = -ENOMEM;
break;
list_for_each_safe(l, n, &tsp->smk_rules) {
rp = list_entry(l, struct smack_rule, list);
list_del(&rp->list);
- kfree(rp);
+ kmem_cache_free(smack_rule_cache, rp);
}
}
*
* Records the label bound to a port.
*
- * Returns 0
+ * Returns 0 on success, and error code otherwise
*/
static int smack_socket_bind(struct socket *sock, struct sockaddr *address,
int addrlen)
{
- if (sock->sk != NULL && sock->sk->sk_family == PF_INET6)
+ if (sock->sk != NULL && sock->sk->sk_family == PF_INET6) {
+ if (addrlen < SIN6_LEN_RFC2133 ||
+ address->sa_family != AF_INET6)
+ return -EINVAL;
smk_ipv6_port_label(sock, address);
+ }
return 0;
}
#endif /* SMACK_IPV6_PORT_LABELING */
switch (sock->sk->sk_family) {
case PF_INET:
- if (addrlen < sizeof(struct sockaddr_in))
+ if (addrlen < sizeof(struct sockaddr_in) ||
+ sap->sa_family != AF_INET)
return -EINVAL;
rc = smack_netlabel_send(sock->sk, (struct sockaddr_in *)sap);
break;
case PF_INET6:
- if (addrlen < sizeof(struct sockaddr_in6))
+ if (addrlen < SIN6_LEN_RFC2133 || sap->sa_family != AF_INET6)
return -EINVAL;
#ifdef SMACK_IPV6_SECMARK_LABELING
rsp = smack_ipv6host_label(sip);
switch (sock->sk->sk_family) {
case AF_INET:
+ if (msg->msg_namelen < sizeof(struct sockaddr_in) ||
+ sip->sin_family != AF_INET)
+ return -EINVAL;
rc = smack_netlabel_send(sock->sk, sip);
break;
+#if IS_ENABLED(CONFIG_IPV6)
case AF_INET6:
+ if (msg->msg_namelen < SIN6_LEN_RFC2133 ||
+ sap->sin6_family != AF_INET6)
+ return -EINVAL;
#ifdef SMACK_IPV6_SECMARK_LABELING
rsp = smack_ipv6host_label(sap);
if (rsp != NULL)
#ifdef SMACK_IPV6_PORT_LABELING
rc = smk_ipv6_port_check(sock->sk, sap, SMK_SENDING);
#endif
+#endif /* IS_ENABLED(CONFIG_IPV6) */
break;
}
return rc;
#ifdef SMACK_IPV6_SECMARK_LABELING
if (skb && skb->secmark != 0)
skp = smack_from_secid(skb->secmark);
+ else if (smk_ipv6_localhost(&sadd))
+ break;
else
skp = smack_ipv6host_label(&sadd);
if (skp == NULL)
if (!smack_inode_cache)
return -ENOMEM;
+ smack_rule_cache = KMEM_CACHE(smack_rule, 0);
+ if (!smack_rule_cache) {
+ kmem_cache_destroy(smack_inode_cache);
+ return -ENOMEM;
+ }
+
/*
* Set the security state for the initial task.
*/
/*
* List locks
*/
-static DEFINE_MUTEX(smack_master_list_lock);
static DEFINE_MUTEX(smack_cipso_lock);
static DEFINE_MUTEX(smack_ambient_lock);
static DEFINE_MUTEX(smk_net4addr_lock);
/*
* Rule lists are maintained for each label.
- * This master list is just for reading /smack/load and /smack/load2.
*/
-struct smack_master_list {
- struct list_head list;
- struct smack_rule *smk_rule;
-};
-
-static LIST_HEAD(smack_rule_list);
-
struct smack_parsed_rule {
struct smack_known *smk_subject;
struct smack_known *smk_object;
* @srp: the rule to add or replace
* @rule_list: the list of rules
* @rule_lock: the rule list lock
- * @global: if non-zero, indicates a global rule
*
* Looks through the current subject/object/access list for
* the subject/object pair and replaces the access that was
*/
static int smk_set_access(struct smack_parsed_rule *srp,
struct list_head *rule_list,
- struct mutex *rule_lock, int global)
+ struct mutex *rule_lock)
{
struct smack_rule *sp;
- struct smack_master_list *smlp;
int found = 0;
int rc = 0;
}
if (found == 0) {
- sp = kzalloc(sizeof(*sp), GFP_KERNEL);
+ sp = kmem_cache_zalloc(smack_rule_cache, GFP_KERNEL);
if (sp == NULL) {
rc = -ENOMEM;
goto out;
sp->smk_access = srp->smk_access1 & ~srp->smk_access2;
list_add_rcu(&sp->list, rule_list);
- /*
- * If this is a global as opposed to self and a new rule
- * it needs to get added for reporting.
- */
- if (global) {
- mutex_unlock(rule_lock);
- smlp = kzalloc(sizeof(*smlp), GFP_KERNEL);
- if (smlp != NULL) {
- smlp->smk_rule = sp;
- mutex_lock(&smack_master_list_lock);
- list_add_rcu(&smlp->list, &smack_rule_list);
- mutex_unlock(&smack_master_list_lock);
- } else
- rc = -ENOMEM;
- return rc;
- }
}
out:
if (rule_list == NULL)
rc = smk_set_access(&rule, &rule.smk_subject->smk_rules,
- &rule.smk_subject->smk_rules_lock, 1);
+ &rule.smk_subject->smk_rules_lock);
else
- rc = smk_set_access(&rule, rule_list, rule_lock, 0);
+ rc = smk_set_access(&rule, rule_list, rule_lock);
if (rc)
goto out;
static void *load2_seq_start(struct seq_file *s, loff_t *pos)
{
- return smk_seq_start(s, pos, &smack_rule_list);
+ return smk_seq_start(s, pos, &smack_known_list);
}
static void *load2_seq_next(struct seq_file *s, void *v, loff_t *pos)
{
- return smk_seq_next(s, v, pos, &smack_rule_list);
+ return smk_seq_next(s, v, pos, &smack_known_list);
}
static int load_seq_show(struct seq_file *s, void *v)
{
struct list_head *list = v;
- struct smack_master_list *smlp =
- list_entry_rcu(list, struct smack_master_list, list);
+ struct smack_rule *srp;
+ struct smack_known *skp =
+ list_entry_rcu(list, struct smack_known, list);
- smk_rule_show(s, smlp->smk_rule, SMK_LABELLEN);
+ list_for_each_entry_rcu(srp, &skp->smk_rules, list)
+ smk_rule_show(s, srp, SMK_LABELLEN);
return 0;
}
static int load2_seq_show(struct seq_file *s, void *v)
{
struct list_head *list = v;
- struct smack_master_list *smlp =
- list_entry_rcu(list, struct smack_master_list, list);
+ struct smack_rule *srp;
+ struct smack_known *skp =
+ list_entry_rcu(list, struct smack_known, list);
- smk_rule_show(s, smlp->smk_rule, SMK_LONGLABEL);
+ list_for_each_entry_rcu(srp, &skp->smk_rules, list)
+ smk_rule_show(s, srp, SMK_LONGLABEL);
return 0;
}
/* locked for registering/using */
static DECLARE_BITMAP(snd_cards_lock, SNDRV_CARDS);
-struct snd_card *snd_cards[SNDRV_CARDS];
-EXPORT_SYMBOL(snd_cards);
+static struct snd_card *snd_cards[SNDRV_CARDS];
static DEFINE_MUTEX(snd_card_mutex);
}
EXPORT_SYMBOL(snd_card_new);
+/**
+ * snd_card_ref - Get the card object from the index
+ * @idx: the card index
+ *
+ * Returns a card object corresponding to the given index or NULL if not found.
+ * Release the object via snd_card_unref().
+ */
+struct snd_card *snd_card_ref(int idx)
+{
+ struct snd_card *card;
+
+ mutex_lock(&snd_card_mutex);
+ card = snd_cards[idx];
+ if (card)
+ get_device(&card->card_dev);
+ mutex_unlock(&snd_card_mutex);
+ return card;
+}
+EXPORT_SYMBOL_GPL(snd_card_ref);
+
/* return non-zero if a card is already locked */
int snd_card_locked(int card)
{
#endif
#include <sound/memalloc.h>
-/*
- *
- * Generic memory allocators
- *
- */
-
-/**
- * snd_malloc_pages - allocate pages with the given size
- * @size: the size to allocate in bytes
- * @gfp_flags: the allocation conditions, GFP_XXX
- *
- * Allocates the physically contiguous pages with the given size.
- *
- * Return: The pointer of the buffer, or %NULL if no enough memory.
- */
-void *snd_malloc_pages(size_t size, gfp_t gfp_flags)
-{
- int pg;
-
- if (WARN_ON(!size))
- return NULL;
- if (WARN_ON(!gfp_flags))
- return NULL;
- gfp_flags |= __GFP_COMP; /* compound page lets parts be mapped */
- pg = get_order(size);
- return (void *) __get_free_pages(gfp_flags, pg);
-}
-EXPORT_SYMBOL(snd_malloc_pages);
-
-/**
- * snd_free_pages - release the pages
- * @ptr: the buffer pointer to release
- * @size: the allocated buffer size
- *
- * Releases the buffer allocated via snd_malloc_pages().
- */
-void snd_free_pages(void *ptr, size_t size)
-{
- int pg;
-
- if (ptr == NULL)
- return;
- pg = get_order(size);
- free_pages((unsigned long) ptr, pg);
-}
-EXPORT_SYMBOL(snd_free_pages);
-
/*
*
* Bus-specific memory allocators
dmab->bytes = 0;
switch (type) {
case SNDRV_DMA_TYPE_CONTINUOUS:
- dmab->area = snd_malloc_pages(size,
- (__force gfp_t)(unsigned long)device);
+ dmab->area = alloc_pages_exact(size,
+ (__force gfp_t)(unsigned long)device);
dmab->addr = 0;
break;
#ifdef CONFIG_HAS_DMA
{
switch (dmab->dev.type) {
case SNDRV_DMA_TYPE_CONTINUOUS:
- snd_free_pages(dmab->area, dmab->bytes);
+ free_pages_exact(dmab->area, dmab->bytes);
break;
#ifdef CONFIG_HAS_DMA
#ifdef CONFIG_GENERIC_ALLOCATOR
static int __init alsa_mixer_oss_init(void)
{
+ struct snd_card *card;
int idx;
snd_mixer_oss_notify_callback = snd_mixer_oss_notify_handler;
for (idx = 0; idx < SNDRV_CARDS; idx++) {
- if (snd_cards[idx])
- snd_mixer_oss_notify_handler(snd_cards[idx], SND_MIXER_OSS_NOTIFY_REGISTER);
+ card = snd_card_ref(idx);
+ if (card) {
+ snd_mixer_oss_notify_handler(card, SND_MIXER_OSS_NOTIFY_REGISTER);
+ snd_card_unref(card);
+ }
}
return 0;
}
static void __exit alsa_mixer_oss_exit(void)
{
+ struct snd_card *card;
int idx;
snd_mixer_oss_notify_callback = NULL;
for (idx = 0; idx < SNDRV_CARDS; idx++) {
- if (snd_cards[idx])
- snd_mixer_oss_notify_handler(snd_cards[idx], SND_MIXER_OSS_NOTIFY_FREE);
+ card = snd_card_ref(idx);
+ if (card) {
+ snd_mixer_oss_notify_handler(card, SND_MIXER_OSS_NOTIFY_FREE);
+ snd_card_unref(card);
+ }
}
}
return -ENOMEM;
size = PAGE_ALIGN(sizeof(struct snd_pcm_mmap_status));
- runtime->status = snd_malloc_pages(size, GFP_KERNEL);
+ runtime->status = alloc_pages_exact(size, GFP_KERNEL);
if (runtime->status == NULL) {
kfree(runtime);
return -ENOMEM;
}
- memset((void*)runtime->status, 0, size);
+ memset(runtime->status, 0, size);
size = PAGE_ALIGN(sizeof(struct snd_pcm_mmap_control));
- runtime->control = snd_malloc_pages(size, GFP_KERNEL);
+ runtime->control = alloc_pages_exact(size, GFP_KERNEL);
if (runtime->control == NULL) {
- snd_free_pages((void*)runtime->status,
+ free_pages_exact(runtime->status,
PAGE_ALIGN(sizeof(struct snd_pcm_mmap_status)));
kfree(runtime);
return -ENOMEM;
}
- memset((void*)runtime->control, 0, size);
+ memset(runtime->control, 0, size);
init_waitqueue_head(&runtime->sleep);
init_waitqueue_head(&runtime->tsleep);
runtime = substream->runtime;
if (runtime->private_free != NULL)
runtime->private_free(runtime);
- snd_free_pages((void*)runtime->status,
+ free_pages_exact(runtime->status,
PAGE_ALIGN(sizeof(struct snd_pcm_mmap_status)));
- snd_free_pages((void*)runtime->control,
+ free_pages_exact(runtime->control,
PAGE_ALIGN(sizeof(struct snd_pcm_mmap_control)));
kfree(runtime->hw_constraints.rules);
/* Avoid concurrent access to runtime via PCM timer interface */
#include <sound/rawmidi.h>
#include <sound/seq_kernel.h>
#include <sound/info.h>
+#include "../seq_clientmgr.h"
/* max. applications */
#define SNDRV_SEQ_OSS_MAX_CLIENTS 16
return snd_seq_kernel_client_dispatch(dp->cseq, ev, atomic, hop);
}
-/* ioctl */
+/* ioctl for writeq */
static inline int
snd_seq_oss_control(struct seq_oss_devinfo *dp, unsigned int type, void *arg)
{
- return snd_seq_kernel_client_ctl(dp->cseq, type, arg);
+ int err;
+
+ snd_seq_client_ioctl_lock(dp->cseq);
+ err = snd_seq_kernel_client_ctl(dp->cseq, type, arg);
+ snd_seq_client_ioctl_unlock(dp->cseq);
+ return err;
}
/* fill the addresses in header */
return 0; /* invalid event - no need to insert queue */
event.time.tick = snd_seq_oss_timer_cur_tick(dp->timer);
- if (dp->timer->realtime || !dp->timer->running) {
+ if (dp->timer->realtime || !dp->timer->running)
snd_seq_oss_dispatch(dp, &event, 0, 0);
- } else {
- if (is_nonblock_mode(dp->file_mode))
- rc = snd_seq_kernel_client_enqueue(dp->cseq, &event, 0, 0);
- else
- rc = snd_seq_kernel_client_enqueue_blocking(dp->cseq, &event, opt, 0, 0);
- }
+ else
+ rc = snd_seq_kernel_client_enqueue(dp->cseq, &event, opt,
+ !is_nonblock_mode(dp->file_mode));
return rc;
}
rec->t.code = SEQ_SYNCTIMER;
rec->t.time = time;
q->sync_event_put = 1;
- snd_seq_kernel_client_enqueue_blocking(dp->cseq, &ev, NULL, 0, 0);
+ snd_seq_kernel_client_enqueue(dp->cseq, &ev, NULL, true);
}
wait_event_interruptible_timeout(q->sync_sleep, ! q->sync_event_put, HZ);
return client;
}
+/* Take refcount and perform ioctl_mutex lock on the given client;
+ * used only for OSS sequencer
+ * Unlock via snd_seq_client_ioctl_unlock() below
+ */
+bool snd_seq_client_ioctl_lock(int clientid)
+{
+ struct snd_seq_client *client;
+
+ client = snd_seq_client_use_ptr(clientid);
+ if (!client)
+ return false;
+ mutex_lock(&client->ioctl_mutex);
+ /* The client isn't unrefed here; see snd_seq_client_ioctl_unlock() */
+ return true;
+}
+EXPORT_SYMBOL_GPL(snd_seq_client_ioctl_lock);
+
+/* Unlock and unref the given client; for OSS sequencer use only */
+void snd_seq_client_ioctl_unlock(int clientid)
+{
+ struct snd_seq_client *client;
+
+ client = snd_seq_client_use_ptr(clientid);
+ if (WARN_ON(!client))
+ return;
+ mutex_unlock(&client->ioctl_mutex);
+ /* The doubly unrefs below are intentional; the first one releases the
+ * leftover from snd_seq_client_ioctl_lock() above, and the second one
+ * is for releasing snd_seq_client_use_ptr() in this function
+ */
+ snd_seq_client_unlock(client);
+ snd_seq_client_unlock(client);
+}
+EXPORT_SYMBOL_GPL(snd_seq_client_ioctl_unlock);
+
static void usage_alloc(struct snd_seq_usage *res, int num)
{
res->cur += num;
static struct snd_seq_client *seq_create_client1(int client_index, int poolsize)
{
- unsigned long flags;
int c;
struct snd_seq_client *client;
mutex_init(&client->ioctl_mutex);
/* find free slot in the client table */
- spin_lock_irqsave(&clients_lock, flags);
+ spin_lock_irq(&clients_lock);
if (client_index < 0) {
for (c = SNDRV_SEQ_DYNAMIC_CLIENTS_BEGIN;
c < SNDRV_SEQ_MAX_CLIENTS;
if (clienttab[c] || clienttablock[c])
continue;
clienttab[client->number = c] = client;
- spin_unlock_irqrestore(&clients_lock, flags);
+ spin_unlock_irq(&clients_lock);
return client;
}
} else {
if (clienttab[client_index] == NULL && !clienttablock[client_index]) {
clienttab[client->number = client_index] = client;
- spin_unlock_irqrestore(&clients_lock, flags);
+ spin_unlock_irq(&clients_lock);
return client;
}
}
- spin_unlock_irqrestore(&clients_lock, flags);
+ spin_unlock_irq(&clients_lock);
snd_seq_pool_delete(&client->pool);
kfree(client);
return NULL; /* no free slot found or busy, return failure code */
static int seq_free_client1(struct snd_seq_client *client)
{
- unsigned long flags;
-
if (!client)
return 0;
- spin_lock_irqsave(&clients_lock, flags);
+ spin_lock_irq(&clients_lock);
clienttablock[client->number] = 1;
clienttab[client->number] = NULL;
- spin_unlock_irqrestore(&clients_lock, flags);
+ spin_unlock_irq(&clients_lock);
snd_seq_delete_all_ports(client);
snd_seq_queue_client_leave(client->number);
snd_use_lock_sync(&client->use_lock);
snd_seq_queue_client_termination(client->number);
if (client->pool)
snd_seq_pool_delete(&client->pool);
- spin_lock_irqsave(&clients_lock, flags);
+ spin_lock_irq(&clients_lock);
clienttablock[client->number] = 0;
- spin_unlock_irqrestore(&clients_lock, flags);
+ spin_unlock_irq(&clients_lock);
return 0;
}
int result;
struct snd_seq_client *sender = NULL;
struct snd_seq_client_port *sport = NULL;
- struct snd_seq_subscribers *p;
result = -EINVAL;
if ((sender = snd_seq_client_use_ptr(subs->sender.client)) == NULL)
goto __end;
if ((sport = snd_seq_port_use_ptr(sender, subs->sender.port)) == NULL)
goto __end;
- p = snd_seq_port_get_subscription(&sport->c_src, &subs->dest);
- if (p) {
- result = 0;
- *subs = p->info;
- } else
- result = -ENOENT;
-
+ result = snd_seq_port_get_subscription(&sport->c_src, &subs->dest,
+ subs);
__end:
if (sport)
snd_seq_port_unlock(sport);
}
EXPORT_SYMBOL(snd_seq_delete_kernel_client);
-/* skeleton to enqueue event, called from snd_seq_kernel_client_enqueue
- * and snd_seq_kernel_client_enqueue_blocking
+/*
+ * exported, called by kernel clients to enqueue events (w/o blocking)
+ *
+ * RETURN VALUE: zero if succeed, negative if error
*/
-static int kernel_client_enqueue(int client, struct snd_seq_event *ev,
- struct file *file, int blocking,
- int atomic, int hop)
+int snd_seq_kernel_client_enqueue(int client, struct snd_seq_event *ev,
+ struct file *file, bool blocking)
{
struct snd_seq_client *cptr;
int result;
if (cptr == NULL)
return -EINVAL;
- if (! cptr->accept_output)
+ if (!cptr->accept_output) {
result = -EPERM;
- else /* send it */
+ } else { /* send it */
+ mutex_lock(&cptr->ioctl_mutex);
result = snd_seq_client_enqueue_event(cptr, ev, file, blocking,
- atomic, hop, NULL);
+ false, 0,
+ &cptr->ioctl_mutex);
+ mutex_unlock(&cptr->ioctl_mutex);
+ }
snd_seq_client_unlock(cptr);
return result;
}
-
-/*
- * exported, called by kernel clients to enqueue events (w/o blocking)
- *
- * RETURN VALUE: zero if succeed, negative if error
- */
-int snd_seq_kernel_client_enqueue(int client, struct snd_seq_event * ev,
- int atomic, int hop)
-{
- return kernel_client_enqueue(client, ev, NULL, 0, atomic, hop);
-}
EXPORT_SYMBOL(snd_seq_kernel_client_enqueue);
-/*
- * exported, called by kernel clients to enqueue events (with blocking)
- *
- * RETURN VALUE: zero if succeed, negative if error
- */
-int snd_seq_kernel_client_enqueue_blocking(int client, struct snd_seq_event * ev,
- struct file *file,
- int atomic, int hop)
-{
- return kernel_client_enqueue(client, ev, file, 1, atomic, hop);
-}
-EXPORT_SYMBOL(snd_seq_kernel_client_enqueue_blocking);
-
/*
* exported, called by kernel clients to dispatch events directly to other
* clients, bypassing the queues. Event time-stamp will be updated.
/* dispatch event to client(s) */
int snd_seq_dispatch_event(struct snd_seq_event_cell *cell, int atomic, int hop);
-/* exported to other modules */
-int snd_seq_kernel_client_enqueue(int client, struct snd_seq_event *ev, int atomic, int hop);
-int snd_seq_kernel_client_enqueue_blocking(int client, struct snd_seq_event * ev,
- struct file *file, int atomic, int hop);
int snd_seq_kernel_client_write_poll(int clientid, struct file *file, poll_table *wait);
int snd_seq_client_notify_subscription(int client, int port,
struct snd_seq_port_subscribe *info, int evtype);
+/* only for OSS sequencer */
+bool snd_seq_client_ioctl_lock(int clientid);
+void snd_seq_client_ioctl_unlock(int clientid);
+
extern int seq_client_load[15];
#endif
void snd_seq_fifo_clear(struct snd_seq_fifo *f)
{
struct snd_seq_event_cell *cell;
- unsigned long flags;
/* clear overflow flag */
atomic_set(&f->overflow, 0);
snd_use_lock_sync(&f->use_lock);
- spin_lock_irqsave(&f->lock, flags);
+ spin_lock_irq(&f->lock);
/* drain the fifo */
while ((cell = fifo_cell_out(f)) != NULL) {
snd_seq_cell_free(cell);
}
- spin_unlock_irqrestore(&f->lock, flags);
+ spin_unlock_irq(&f->lock);
}
}
set_current_state(TASK_INTERRUPTIBLE);
add_wait_queue(&f->input_sleep, &wait);
- spin_unlock_irq(&f->lock);
+ spin_unlock_irqrestore(&f->lock, flags);
schedule();
- spin_lock_irq(&f->lock);
+ spin_lock_irqsave(&f->lock, flags);
remove_wait_queue(&f->input_sleep, &wait);
if (signal_pending(current)) {
spin_unlock_irqrestore(&f->lock, flags);
/* change the size of pool; all old events are removed */
int snd_seq_fifo_resize(struct snd_seq_fifo *f, int poolsize)
{
- unsigned long flags;
struct snd_seq_pool *newpool, *oldpool;
struct snd_seq_event_cell *cell, *next, *oldhead;
return -ENOMEM;
}
- spin_lock_irqsave(&f->lock, flags);
+ spin_lock_irq(&f->lock);
/* remember old pool */
oldpool = f->pool;
oldhead = f->head;
f->tail = NULL;
f->cells = 0;
/* NOTE: overflow flag is not cleared */
- spin_unlock_irqrestore(&f->lock, flags);
+ spin_unlock_irq(&f->lock);
/* close the old pool and wait until all users are gone */
snd_seq_pool_mark_closing(oldpool);
#include <linux/export.h>
#include <linux/slab.h>
#include <linux/sched/signal.h>
-#include <linux/vmalloc.h>
+#include <linux/mm.h>
#include <sound/core.h>
#include <sound/seq_kernel.h>
set_current_state(TASK_INTERRUPTIBLE);
add_wait_queue(&pool->output_sleep, &wait);
- spin_unlock_irq(&pool->lock);
+ spin_unlock_irqrestore(&pool->lock, flags);
if (mutexp)
mutex_unlock(mutexp);
schedule();
if (mutexp)
mutex_lock(mutexp);
- spin_lock_irq(&pool->lock);
+ spin_lock_irqsave(&pool->lock, flags);
remove_wait_queue(&pool->output_sleep, &wait);
/* interrupted? */
if (signal_pending(current)) {
{
int cell;
struct snd_seq_event_cell *cellptr;
- unsigned long flags;
if (snd_BUG_ON(!pool))
return -EINVAL;
- cellptr = vmalloc(array_size(sizeof(struct snd_seq_event_cell),
- pool->size));
+ cellptr = kvmalloc_array(sizeof(struct snd_seq_event_cell), pool->size,
+ GFP_KERNEL);
if (!cellptr)
return -ENOMEM;
/* add new cells to the free cell list */
- spin_lock_irqsave(&pool->lock, flags);
+ spin_lock_irq(&pool->lock);
if (pool->ptr) {
- spin_unlock_irqrestore(&pool->lock, flags);
- vfree(cellptr);
+ spin_unlock_irq(&pool->lock);
+ kvfree(cellptr);
return 0;
}
/* init statistics */
pool->max_used = 0;
pool->total_elements = pool->size;
- spin_unlock_irqrestore(&pool->lock, flags);
+ spin_unlock_irq(&pool->lock);
return 0;
}
/* remove events */
int snd_seq_pool_done(struct snd_seq_pool *pool)
{
- unsigned long flags;
struct snd_seq_event_cell *ptr;
if (snd_BUG_ON(!pool))
schedule_timeout_uninterruptible(1);
/* release all resources */
- spin_lock_irqsave(&pool->lock, flags);
+ spin_lock_irq(&pool->lock);
ptr = pool->ptr;
pool->ptr = NULL;
pool->free = NULL;
pool->total_elements = 0;
- spin_unlock_irqrestore(&pool->lock, flags);
+ spin_unlock_irq(&pool->lock);
- vfree(ptr);
+ kvfree(ptr);
- spin_lock_irqsave(&pool->lock, flags);
+ spin_lock_irq(&pool->lock);
pool->closing = 0;
- spin_unlock_irqrestore(&pool->lock, flags);
+ spin_unlock_irq(&pool->lock);
return 0;
}
struct snd_seq_client_port *snd_seq_create_port(struct snd_seq_client *client,
int port)
{
- unsigned long flags;
struct snd_seq_client_port *new_port, *p;
int num = -1;
num = port >= 0 ? port : 0;
mutex_lock(&client->ports_mutex);
- write_lock_irqsave(&client->ports_lock, flags);
+ write_lock_irq(&client->ports_lock);
list_for_each_entry(p, &client->ports_list_head, list) {
if (p->addr.port > num)
break;
client->num_ports++;
new_port->addr.port = num; /* store the port number in the port */
sprintf(new_port->name, "port-%d", num);
- write_unlock_irqrestore(&client->ports_lock, flags);
+ write_unlock_irq(&client->ports_lock);
mutex_unlock(&client->ports_mutex);
return new_port;
/* delete a port with the given port id */
int snd_seq_delete_port(struct snd_seq_client *client, int port)
{
- unsigned long flags;
struct snd_seq_client_port *found = NULL, *p;
mutex_lock(&client->ports_mutex);
- write_lock_irqsave(&client->ports_lock, flags);
+ write_lock_irq(&client->ports_lock);
list_for_each_entry(p, &client->ports_list_head, list) {
if (p->addr.port == port) {
/* ok found. delete from the list at first */
break;
}
}
- write_unlock_irqrestore(&client->ports_lock, flags);
+ write_unlock_irq(&client->ports_lock);
mutex_unlock(&client->ports_mutex);
if (found)
return port_delete(client, found);
/* delete the all ports belonging to the given client */
int snd_seq_delete_all_ports(struct snd_seq_client *client)
{
- unsigned long flags;
struct list_head deleted_list;
struct snd_seq_client_port *port, *tmp;
* clear the port list in the client data.
*/
mutex_lock(&client->ports_mutex);
- write_lock_irqsave(&client->ports_lock, flags);
+ write_lock_irq(&client->ports_lock);
if (! list_empty(&client->ports_list_head)) {
list_add(&deleted_list, &client->ports_list_head);
list_del_init(&client->ports_list_head);
INIT_LIST_HEAD(&deleted_list);
}
client->num_ports = 0;
- write_unlock_irqrestore(&client->ports_lock, flags);
+ write_unlock_irq(&client->ports_lock);
/* remove each port in deleted_list */
list_for_each_entry_safe(port, tmp, &deleted_list, list) {
list_del_init(list);
grp->exclusive = 0;
write_unlock_irq(&grp->list_lock);
- up_write(&grp->list_mutex);
if (!empty)
unsubscribe_port(client, port, grp, &subs->info, ack);
+ up_write(&grp->list_mutex);
}
/* connect two ports */
/* get matched subscriber */
-struct snd_seq_subscribers *snd_seq_port_get_subscription(struct snd_seq_port_subs_info *src_grp,
- struct snd_seq_addr *dest_addr)
+int snd_seq_port_get_subscription(struct snd_seq_port_subs_info *src_grp,
+ struct snd_seq_addr *dest_addr,
+ struct snd_seq_port_subscribe *subs)
{
- struct snd_seq_subscribers *s, *found = NULL;
+ struct snd_seq_subscribers *s;
+ int err = -ENOENT;
down_read(&src_grp->list_mutex);
list_for_each_entry(s, &src_grp->list_head, src_list) {
if (addr_match(dest_addr, &s->info.dest)) {
- found = s;
+ *subs = s->info;
+ err = 0;
break;
}
}
up_read(&src_grp->list_mutex);
- return found;
+ return err;
}
/*
struct snd_seq_port_subscribe *info);
/* get matched subscriber */
-struct snd_seq_subscribers *snd_seq_port_get_subscription(struct snd_seq_port_subs_info *src_grp,
- struct snd_seq_addr *dest_addr);
+int snd_seq_port_get_subscription(struct snd_seq_port_subs_info *src_grp,
+ struct snd_seq_addr *dest_addr,
+ struct snd_seq_port_subscribe *subs);
#endif
if (dev == SNDRV_MINOR_CONTROL) {
/* /dev/aloadC? */
int card = SNDRV_MINOR_CARD(minor);
- if (snd_cards[card] == NULL)
+ struct snd_card *ref = snd_card_ref(card);
+ if (!ref)
snd_request_card(card);
+ else
+ snd_card_unref(ref);
} else if (dev == SNDRV_MINOR_GLOBAL) {
/* /dev/aloadSEQ */
snd_request_other(minor);
/* internal flags */
#define SNDRV_TIMER_IFLG_PAUSED 0x00010000
+#define SNDRV_TIMER_IFLG_DEAD 0x00020000
#if IS_ENABLED(CONFIG_SND_HRTIMER)
#define DEFAULT_TIMER_LIMIT 4
struct snd_timer_instance *timeri = NULL;
int err;
+ mutex_lock(®ister_mutex);
if (tid->dev_class == SNDRV_TIMER_CLASS_SLAVE) {
/* open a slave instance */
if (tid->dev_sclass <= SNDRV_TIMER_SCLASS_NONE ||
tid->dev_sclass > SNDRV_TIMER_SCLASS_OSS_SEQUENCER) {
pr_debug("ALSA: timer: invalid slave class %i\n",
tid->dev_sclass);
- return -EINVAL;
+ err = -EINVAL;
+ goto unlock;
}
- mutex_lock(®ister_mutex);
timeri = snd_timer_instance_new(owner, NULL);
if (!timeri) {
- mutex_unlock(®ister_mutex);
- return -ENOMEM;
+ err = -ENOMEM;
+ goto unlock;
}
timeri->slave_class = tid->dev_sclass;
timeri->slave_id = tid->device;
snd_timer_close_locked(timeri);
timeri = NULL;
}
- mutex_unlock(®ister_mutex);
- *ti = timeri;
- return err;
+ goto unlock;
}
/* open a master instance */
- mutex_lock(®ister_mutex);
timer = snd_timer_find(tid);
#ifdef CONFIG_MODULES
if (!timer) {
}
#endif
if (!timer) {
- mutex_unlock(®ister_mutex);
- return -ENODEV;
+ err = -ENODEV;
+ goto unlock;
}
if (!list_empty(&timer->open_list_head)) {
timeri = list_entry(timer->open_list_head.next,
struct snd_timer_instance, open_list);
if (timeri->flags & SNDRV_TIMER_IFLG_EXCLUSIVE) {
- mutex_unlock(®ister_mutex);
- return -EBUSY;
+ err = -EBUSY;
+ timeri = NULL;
+ goto unlock;
}
}
if (timer->num_instances >= timer->max_instances) {
- mutex_unlock(®ister_mutex);
- return -EBUSY;
+ err = -EBUSY;
+ goto unlock;
}
timeri = snd_timer_instance_new(owner, timer);
if (!timeri) {
- mutex_unlock(®ister_mutex);
- return -ENOMEM;
+ err = -ENOMEM;
+ goto unlock;
}
/* take a card refcount for safe disconnection */
if (timer->card)
timeri->slave_id = slave_id;
if (list_empty(&timer->open_list_head) && timer->hw.open) {
- int err = timer->hw.open(timer);
+ err = timer->hw.open(timer);
if (err) {
kfree(timeri->owner);
kfree(timeri);
+ timeri = NULL;
if (timer->card)
put_device(&timer->card->card_dev);
module_put(timer->module);
- mutex_unlock(®ister_mutex);
- return err;
+ goto unlock;
}
}
snd_timer_close_locked(timeri);
timeri = NULL;
}
+
+ unlock:
mutex_unlock(®ister_mutex);
*ti = timeri;
return err;
*/
static int snd_timer_close_locked(struct snd_timer_instance *timeri)
{
- struct snd_timer *timer = NULL;
+ struct snd_timer *timer = timeri->timer;
struct snd_timer_instance *slave, *tmp;
+ if (timer) {
+ spin_lock_irq(&timer->lock);
+ timeri->flags |= SNDRV_TIMER_IFLG_DEAD;
+ spin_unlock_irq(&timer->lock);
+ }
+
list_del(&timeri->open_list);
/* force to stop the timer */
snd_timer_stop(timeri);
- timer = timeri->timer;
if (timer) {
timer->num_instances--;
/* wait, until the active callback is finished */
return -EINVAL;
spin_lock_irqsave(&timer->lock, flags);
+ if (timeri->flags & SNDRV_TIMER_IFLG_DEAD) {
+ result = -EINVAL;
+ goto unlock;
+ }
if (timer->card && timer->card->shutdown) {
result = -ENODEV;
goto unlock;
bool start)
{
unsigned long flags;
+ int err;
spin_lock_irqsave(&slave_active_lock, flags);
+ if (timeri->flags & SNDRV_TIMER_IFLG_DEAD) {
+ err = -EINVAL;
+ goto unlock;
+ }
if (timeri->flags & SNDRV_TIMER_IFLG_RUNNING) {
- spin_unlock_irqrestore(&slave_active_lock, flags);
- return -EBUSY;
+ err = -EBUSY;
+ goto unlock;
}
timeri->flags |= SNDRV_TIMER_IFLG_RUNNING;
if (timeri->master && timeri->timer) {
SNDRV_TIMER_EVENT_CONTINUE);
spin_unlock(&timeri->timer->lock);
}
+ err = 1; /* delayed start */
+ unlock:
spin_unlock_irqrestore(&slave_active_lock, flags);
- return 1; /* delayed start */
+ return err;
}
/* stop/pause a master timer */
timer->sticks = ticks;
}
+/* call callbacks in timer ack list */
+static void snd_timer_process_callbacks(struct snd_timer *timer,
+ struct list_head *head)
+{
+ struct snd_timer_instance *ti;
+ unsigned long resolution, ticks;
+
+ while (!list_empty(head)) {
+ ti = list_first_entry(head, struct snd_timer_instance,
+ ack_list);
+
+ /* remove from ack_list and make empty */
+ list_del_init(&ti->ack_list);
+
+ if (!(ti->flags & SNDRV_TIMER_IFLG_DEAD)) {
+ ticks = ti->pticks;
+ ti->pticks = 0;
+ resolution = ti->resolution;
+ ti->flags |= SNDRV_TIMER_IFLG_CALLBACK;
+ spin_unlock(&timer->lock);
+ if (ti->callback)
+ ti->callback(ti, resolution, ticks);
+ spin_lock(&timer->lock);
+ ti->flags &= ~SNDRV_TIMER_IFLG_CALLBACK;
+ }
+ }
+}
+
+/* clear pending instances from ack list */
+static void snd_timer_clear_callbacks(struct snd_timer *timer,
+ struct list_head *head)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&timer->lock, flags);
+ while (!list_empty(head))
+ list_del_init(head->next);
+ spin_unlock_irqrestore(&timer->lock, flags);
+}
+
/*
* timer tasklet
*
static void snd_timer_tasklet(unsigned long arg)
{
struct snd_timer *timer = (struct snd_timer *) arg;
- struct snd_timer_instance *ti;
- struct list_head *p;
- unsigned long resolution, ticks;
unsigned long flags;
- if (timer->card && timer->card->shutdown)
+ if (timer->card && timer->card->shutdown) {
+ snd_timer_clear_callbacks(timer, &timer->sack_list_head);
return;
+ }
spin_lock_irqsave(&timer->lock, flags);
- /* now process all callbacks */
- while (!list_empty(&timer->sack_list_head)) {
- p = timer->sack_list_head.next; /* get first item */
- ti = list_entry(p, struct snd_timer_instance, ack_list);
-
- /* remove from ack_list and make empty */
- list_del_init(p);
-
- ticks = ti->pticks;
- ti->pticks = 0;
- resolution = ti->resolution;
-
- ti->flags |= SNDRV_TIMER_IFLG_CALLBACK;
- spin_unlock(&timer->lock);
- if (ti->callback)
- ti->callback(ti, resolution, ticks);
- spin_lock(&timer->lock);
- ti->flags &= ~SNDRV_TIMER_IFLG_CALLBACK;
- }
+ snd_timer_process_callbacks(timer, &timer->sack_list_head);
spin_unlock_irqrestore(&timer->lock, flags);
}
void snd_timer_interrupt(struct snd_timer * timer, unsigned long ticks_left)
{
struct snd_timer_instance *ti, *ts, *tmp;
- unsigned long resolution, ticks;
- struct list_head *p, *ack_list_head;
+ unsigned long resolution;
+ struct list_head *ack_list_head;
unsigned long flags;
int use_tasklet = 0;
if (timer == NULL)
return;
- if (timer->card && timer->card->shutdown)
+ if (timer->card && timer->card->shutdown) {
+ snd_timer_clear_callbacks(timer, &timer->ack_list_head);
return;
+ }
spin_lock_irqsave(&timer->lock, flags);
*/
list_for_each_entry_safe(ti, tmp, &timer->active_list_head,
active_list) {
+ if (ti->flags & SNDRV_TIMER_IFLG_DEAD)
+ continue;
if (!(ti->flags & SNDRV_TIMER_IFLG_RUNNING))
continue;
ti->pticks += ticks_left;
}
/* now process all fast callbacks */
- while (!list_empty(&timer->ack_list_head)) {
- p = timer->ack_list_head.next; /* get first item */
- ti = list_entry(p, struct snd_timer_instance, ack_list);
-
- /* remove from ack_list and make empty */
- list_del_init(p);
-
- ticks = ti->pticks;
- ti->pticks = 0;
-
- ti->flags |= SNDRV_TIMER_IFLG_CALLBACK;
- spin_unlock(&timer->lock);
- if (ti->callback)
- ti->callback(ti, resolution, ticks);
- spin_lock(&timer->lock);
- ti->flags &= ~SNDRV_TIMER_IFLG_CALLBACK;
- }
+ snd_timer_process_callbacks(timer, &timer->ack_list_head);
/* do we have any slow callbacks? */
use_tasklet = !list_empty(&timer->sack_list_head);
snd_timer_stop(tu->timeri);
tu->timeri->lost = 0;
tu->last_resolution = 0;
- return (err = snd_timer_start(tu->timeri, tu->ticks)) < 0 ? err : 0;
+ err = snd_timer_start(tu->timeri, tu->ticks);
+ if (err < 0)
+ return err;
+ return 0;
}
static int snd_timer_user_stop(struct file *file)
tu = file->private_data;
if (!tu->timeri)
return -EBADFD;
- return (err = snd_timer_stop(tu->timeri)) < 0 ? err : 0;
+ err = snd_timer_stop(tu->timeri);
+ if (err < 0)
+ return err;
+ return 0;
}
static int snd_timer_user_continue(struct file *file)
if (!(tu->timeri->flags & SNDRV_TIMER_IFLG_PAUSED))
return snd_timer_user_start(file);
tu->timeri->lost = 0;
- return (err = snd_timer_continue(tu->timeri)) < 0 ? err : 0;
+ err = snd_timer_continue(tu->timeri);
+ if (err < 0)
+ return err;
+ return 0;
}
static int snd_timer_user_pause(struct file *file)
tu = file->private_data;
if (!tu->timeri)
return -EBADFD;
- return (err = snd_timer_pause(tu->timeri)) < 0 ? err : 0;
+ err = snd_timer_pause(tu->timeri);
+ if (err < 0)
+ return err;
+ return 0;
}
enum {
loopback_timer_stop_sync(dpcm);
- salign = (snd_pcm_format_width(runtime->format) *
+ salign = (snd_pcm_format_physical_width(runtime->format) *
runtime->channels) / 8;
bps = salign * runtime->rate;
if (bps <= 0 || salign <= 0)
SNDRV_PCM_INFO_MMAP_VALID | SNDRV_PCM_INFO_PAUSE |
SNDRV_PCM_INFO_RESUME),
.formats = (SNDRV_PCM_FMTBIT_S16_LE | SNDRV_PCM_FMTBIT_S16_BE |
+ SNDRV_PCM_FMTBIT_S24_LE | SNDRV_PCM_FMTBIT_S24_BE |
+ SNDRV_PCM_FMTBIT_S24_3LE | SNDRV_PCM_FMTBIT_S24_3BE |
SNDRV_PCM_FMTBIT_S32_LE | SNDRV_PCM_FMTBIT_S32_BE |
SNDRV_PCM_FMTBIT_FLOAT_LE | SNDRV_PCM_FMTBIT_FLOAT_BE),
.rates = SNDRV_PCM_RATE_CONTINUOUS | SNDRV_PCM_RATE_8000_192000,
#define INTERRUPT_INTERVAL 16
#define QUEUE_LENGTH 48
-#define IN_PACKET_HEADER_SIZE 4
+#define IR_HEADER_SIZE 8 // For header and timestamp.
#define OUT_PACKET_HEADER_SIZE 0
+#define HEADER_TSTAMP_MASK 0x0000ffff
static void pcm_period_tasklet(unsigned long data);
static inline int queue_in_packet(struct amdtp_stream *s)
{
- return queue_packet(s, IN_PACKET_HEADER_SIZE, s->max_payload_length);
+ return queue_packet(s, IR_HEADER_SIZE, s->max_payload_length);
}
static int handle_out_packet(struct amdtp_stream *s,
return cycle;
}
-static inline u32 decrement_cycle_count(u32 cycle, unsigned int subtrahend)
-{
- if (cycle < subtrahend)
- cycle += 8 * CYCLES_PER_SECOND;
- return cycle - subtrahend;
-}
-
static void out_stream_callback(struct fw_iso_context *context, u32 tstamp,
size_t header_length, void *header,
void *private_data)
struct amdtp_stream *s = private_data;
unsigned int i, packets;
unsigned int payload_length, max_payload_length;
- __be32 *headers = header;
- u32 cycle;
+ __be32 *ctx_header = header;
if (s->packet_index < 0)
return;
/* The number of packets in buffer */
- packets = header_length / IN_PACKET_HEADER_SIZE;
-
- cycle = compute_cycle_count(tstamp);
-
- /* Align to actual cycle count for the last packet. */
- cycle = decrement_cycle_count(cycle, packets);
+ packets = header_length / IR_HEADER_SIZE;
/* For buffer-over-run prevention. */
max_payload_length = s->max_payload_length;
for (i = 0; i < packets; i++) {
- cycle = increment_cycle_count(cycle, 1);
+ u32 iso_header = be32_to_cpu(ctx_header[0]);
+ unsigned int cycle;
+
+ tstamp = be32_to_cpu(ctx_header[1]) & HEADER_TSTAMP_MASK;
+ cycle = compute_cycle_count(tstamp);
/* The number of bytes in this packet */
- payload_length =
- (be32_to_cpu(headers[i]) >> ISO_DATA_LENGTH_SHIFT);
+ payload_length = iso_header >> ISO_DATA_LENGTH_SHIFT;
if (payload_length > max_payload_length) {
dev_err(&s->unit->device,
"Detect jumbo payload: %04x %04x\n",
if (s->handle_packet(s, payload_length, cycle, i) < 0)
break;
+
+ ctx_header += IR_HEADER_SIZE / sizeof(__be32);
}
/* Queueing error or detecting invalid payload. */
void *header, void *private_data)
{
struct amdtp_stream *s = private_data;
+ __be32 *ctx_header = header;
u32 cycle;
unsigned int packets;
s->callbacked = true;
wake_up(&s->callback_wait);
- cycle = compute_cycle_count(tstamp);
-
if (s->direction == AMDTP_IN_STREAM) {
- packets = header_length / IN_PACKET_HEADER_SIZE;
- cycle = decrement_cycle_count(cycle, packets);
+ tstamp = be32_to_cpu(ctx_header[1]) & HEADER_TSTAMP_MASK;
+ cycle = compute_cycle_count(tstamp);
+
context->callback.sc = in_stream_callback;
if (s->flags & CIP_NO_HEADER)
s->handle_packet = handle_in_packet_without_header;
s->handle_packet = handle_in_packet;
} else {
packets = header_length / 4;
+ cycle = compute_cycle_count(tstamp);
cycle = increment_cycle_count(cycle, QUEUE_LENGTH - packets);
context->callback.sc = out_stream_callback;
if (s->flags & CIP_NO_HEADER)
if (s->direction == AMDTP_IN_STREAM) {
dir = DMA_FROM_DEVICE;
type = FW_ISO_CONTEXT_RECEIVE;
- header_size = IN_PACKET_HEADER_SIZE;
+ header_size = IR_HEADER_SIZE;
} else {
dir = DMA_TO_DEVICE;
type = FW_ISO_CONTEXT_TRANSMIT;
CIP_HEADER_WITHOUT_EOH;
fmt = CIP_FMT_MOTU_TX_V3;
}
+
+ if (protocol == &snd_motu_protocol_v2) {
+ // 8pre has some quirks.
+ flags |= CIP_WRONG_DBS |
+ CIP_SKIP_DBC_ZERO_CHECK;
+ }
} else {
process_data_blocks = process_rx_data_blocks;
flags |= CIP_DBC_IS_END_EVENT;
#define V2_CLOCK_SRC_SHIFT 0
#define V2_CLOCK_TRAVELER_FETCH_DISABLE 0x04000000
#define V2_CLOCK_TRAVELER_FETCH_ENABLE 0x03000000
+#define V2_CLOCK_8PRE_FETCH_DISABLE 0x02000000
+#define V2_CLOCK_8PRE_FETCH_ENABLE 0x00000000
#define V2_IN_OUT_CONF_OFFSET 0x0c04
#define V2_OPT_OUT_IFACE_MASK 0x00000c00
u32 data;
int err = 0;
- if (motu->spec == &snd_motu_spec_traveler) {
+ if (motu->spec == &snd_motu_spec_traveler ||
+ motu->spec == &snd_motu_spec_8pre) {
err = snd_motu_transaction_read(motu, V2_CLOCK_STATUS_OFFSET,
®, sizeof(reg));
if (err < 0)
return err;
data = be32_to_cpu(reg);
- data &= ~(V2_CLOCK_TRAVELER_FETCH_DISABLE |
- V2_CLOCK_TRAVELER_FETCH_ENABLE);
-
- if (enable)
- data |= V2_CLOCK_TRAVELER_FETCH_ENABLE;
- else
- data |= V2_CLOCK_TRAVELER_FETCH_DISABLE;
+ if (motu->spec == &snd_motu_spec_traveler) {
+ data &= ~(V2_CLOCK_TRAVELER_FETCH_DISABLE |
+ V2_CLOCK_TRAVELER_FETCH_ENABLE);
+
+ if (enable)
+ data |= V2_CLOCK_TRAVELER_FETCH_ENABLE;
+ else
+ data |= V2_CLOCK_TRAVELER_FETCH_DISABLE;
+ } else if (motu->spec == &snd_motu_spec_8pre) {
+ data &= ~(V2_CLOCK_8PRE_FETCH_DISABLE |
+ V2_CLOCK_8PRE_FETCH_ENABLE);
+
+ if (enable)
+ data |= V2_CLOCK_8PRE_FETCH_DISABLE;
+ else
+ data |= V2_CLOCK_8PRE_FETCH_ENABLE;
+ }
reg = cpu_to_be32(data);
err = snd_motu_transaction_write(motu, V2_CLOCK_STATUS_OFFSET,
* interfaces.
*/
data = (data & mask) >> shift;
- if ((flags & SND_MOTU_SPEC_HAS_OPT_IFACE_A) &&
- data == V2_OPT_IFACE_MODE_ADAT) {
- pcm_chunks[0] += 8;
- pcm_chunks[1] += 4;
+ if (data == V2_OPT_IFACE_MODE_ADAT) {
+ if (flags & SND_MOTU_SPEC_HAS_OPT_IFACE_A) {
+ pcm_chunks[0] += 8;
+ pcm_chunks[1] += 4;
+ }
+ // 8pre has two sets of optical interface and doesn't reduce
+ // chunks for ADAT signals.
+ if (flags & SND_MOTU_SPEC_HAS_OPT_IFACE_B) {
+ pcm_chunks[1] += 4;
+ }
}
/* At mode x4, no data chunks are supported in this part. */
.analog_out_ports = 8,
};
+const struct snd_motu_spec snd_motu_spec_8pre = {
+ .name = "8pre",
+ .protocol = &snd_motu_protocol_v2,
+ // In tx, use coax chunks for mix-return 1/2. In rx, use coax chunks for
+ // dummy 1/2.
+ .flags = SND_MOTU_SPEC_SUPPORT_CLOCK_X2 |
+ SND_MOTU_SPEC_HAS_OPT_IFACE_A |
+ SND_MOTU_SPEC_HAS_OPT_IFACE_B |
+ SND_MOTU_SPEC_RX_MIDI_2ND_Q |
+ SND_MOTU_SPEC_TX_MIDI_2ND_Q,
+ .analog_in_ports = 8,
+ .analog_out_ports = 2,
+};
+
static const struct snd_motu_spec motu_828mk3 = {
.name = "828mk3",
.protocol = &snd_motu_protocol_v3,
static const struct ieee1394_device_id motu_id_table[] = {
SND_MOTU_DEV_ENTRY(0x000003, &motu_828mk2),
SND_MOTU_DEV_ENTRY(0x000009, &snd_motu_spec_traveler),
+ SND_MOTU_DEV_ENTRY(0x00000f, &snd_motu_spec_8pre),
SND_MOTU_DEV_ENTRY(0x000015, &motu_828mk3), /* FireWire only. */
SND_MOTU_DEV_ENTRY(0x000035, &motu_828mk3), /* Hybrid. */
SND_MOTU_DEV_ENTRY(0x000033, &motu_audio_express),
extern const struct snd_motu_protocol snd_motu_protocol_v3;
extern const struct snd_motu_spec snd_motu_spec_traveler;
+extern const struct snd_motu_spec snd_motu_spec_8pre;
int amdtp_motu_init(struct amdtp_stream *s, struct fw_unit *unit,
enum amdtp_stream_direction dir,
return ret;
bus->ext_ops = ext_ops;
- INIT_LIST_HEAD(&bus->hlink_list);
bus->idx = idx++;
-
bus->cmd_dma_state = true;
return 0;
spin_lock_init(&bus->reg_lock);
mutex_init(&bus->cmd_mutex);
mutex_init(&bus->lock);
+ INIT_LIST_HEAD(&bus->hlink_list);
bus->irq = -1;
return 0;
}
if (bus->display_power_status) {
if (!bus->display_power_active) {
+ unsigned long cookie = -1;
+
if (acomp->ops->get_power)
- acomp->ops->get_power(acomp->dev);
+ cookie = acomp->ops->get_power(acomp->dev);
+
snd_hdac_set_codec_wakeup(bus, true);
snd_hdac_set_codec_wakeup(bus, false);
- bus->display_power_active = true;
+ bus->display_power_active = cookie;
}
} else {
if (bus->display_power_active) {
+ unsigned long cookie = bus->display_power_active;
+
if (acomp->ops->put_power)
- acomp->ops->put_power(acomp->dev);
- bus->display_power_active = false;
+ acomp->ops->put_power(acomp->dev, cookie);
+
+ bus->display_power_active = 0;
}
}
unlock:
return 0;
if (WARN_ON(bus->display_power_active) && acomp->ops)
- acomp->ops->put_power(acomp->dev);
+ acomp->ops->put_power(acomp->dev, bus->display_power_active);
- bus->display_power_active = false;
+ bus->display_power_active = 0;
bus->display_power_status = 0;
component_master_del(dev, &hdac_component_master_ops);
used = 0;
for (block = alloc->first, i = 0; block; block = block->next, i++) {
used += block->size;
- snd_iprintf(buffer, "Block %i at 0x%lx onboard 0x%x size %i (0x%x):\n", i, (long) block, block->ptr, block->size, block->size);
+ snd_iprintf(buffer, "Block %i onboard 0x%x size %i (0x%x):\n", i, block->ptr, block->size, block->size);
if (block->share ||
block->share_id[0] || block->share_id[1] ||
block->share_id[2] || block->share_id[3])
static int __init alsa_sound_last_init(void)
{
+ struct snd_card *card;
int idx, ok = 0;
printk(KERN_INFO "ALSA device list:\n");
- for (idx = 0; idx < SNDRV_CARDS; idx++)
- if (snd_cards[idx] != NULL) {
- printk(KERN_INFO " #%i: %s\n", idx, snd_cards[idx]->longname);
+ for (idx = 0; idx < SNDRV_CARDS; idx++) {
+ card = snd_card_ref(idx);
+ if (card) {
+ printk(KERN_INFO " #%i: %s\n", idx, card->longname);
+ snd_card_unref(card);
ok++;
}
+ }
if (ok == 0)
printk(KERN_INFO " No soundcards found.\n");
return 0;
c->name, pci->vendor, pci->device,
emu->serial);
- if (!*card->id && c->id) {
- int i, n = 0;
+ if (!*card->id && c->id)
strlcpy(card->id, c->id, sizeof(card->id));
- for (;;) {
- for (i = 0; i < snd_ecards_limit; i++) {
- if (snd_cards[i] && !strcmp(snd_cards[i]->id, card->id))
- break;
- }
- if (i >= snd_ecards_limit)
- break;
- n++;
- if (n >= SNDRV_CARDS)
- break;
- snprintf(card->id, sizeof(card->id), "%s_%d", c->id, n);
- }
- }
is_audigy = emu->audigy = c->emu10k2_chip;
struct hda_codec *codec = device->device_data;
codec->in_freeing = 1;
- snd_hdac_device_unregister(&codec->core);
+ /*
+ * snd_hda_codec_device_new() is used by legacy HDA and ASoC driver.
+ * We can't unregister ASoC device since it will be unregistered in
+ * snd_hdac_ext_bus_device_remove().
+ */
+ if (codec->core.type == HDA_DEV_LEGACY)
+ snd_hdac_device_unregister(&codec->core);
codec_display_power(codec, false);
put_device(hda_codec_dev(codec));
return 0;
chip->msi = 0;
}
- if (azx_acquire_irq(chip, 0) < 0)
- return -EBUSY;
-
pci_set_master(pci);
synchronize_irq(bus->irq);
return -ENODEV;
}
+ if (azx_acquire_irq(chip, 0) < 0)
+ return -EBUSY;
+
strcpy(card->driver, "HDA-Intel");
strlcpy(card->shortname, driver_short_names[chip->driver_type],
sizeof(card->shortname));
/* Cannonlake */
{ PCI_DEVICE(0x8086, 0x9dc8),
.driver_data = AZX_DRIVER_SKL | AZX_DCAPS_INTEL_SKYLAKE},
+ /* CometLake-LP */
+ { PCI_DEVICE(0x8086, 0x02C8),
+ .driver_data = AZX_DRIVER_SKL | AZX_DCAPS_INTEL_SKYLAKE},
+ /* CometLake-H */
+ { PCI_DEVICE(0x8086, 0x06C8),
+ .driver_data = AZX_DRIVER_SKL | AZX_DCAPS_INTEL_SKYLAKE},
/* Icelake */
{ PCI_DEVICE(0x8086, 0x34c8),
.driver_data = AZX_DRIVER_SKL | AZX_DCAPS_INTEL_SKYLAKE},
ret = !repoll || !eld->monitor_present || eld->eld_valid;
jack = snd_hda_jack_tbl_get(codec, pin_nid);
- if (jack)
+ if (jack) {
jack->block_report = !ret;
-
+ jack->pin_sense = (eld->monitor_present && eld->eld_valid) ?
+ AC_PINSENSE_PRESENCE : 0;
+ }
mutex_unlock(&per_pin->lock);
return ret;
}
container_of(to_delayed_work(work), struct hdmi_spec_per_pin, work);
struct hda_codec *codec = per_pin->codec;
struct hdmi_spec *spec = codec->spec;
+ struct hda_jack_tbl *jack;
+
+ jack = snd_hda_jack_tbl_get(codec, per_pin->pin_nid);
+ if (jack)
+ jack->jack_dirty = 1;
if (per_pin->repoll_count++ > 6)
per_pin->repoll_count = 0;
unsigned int no_depop_delay:1;
unsigned int done_hp_init:1;
unsigned int no_shutup_pins:1;
+ unsigned int ultra_low_power:1;
/* for PLL fix */
hda_nid_t pll_nid;
if (spec->init_hook)
spec->init_hook(codec);
+ snd_hda_gen_init(codec);
alc_fix_pll(codec);
alc_auto_init_amp(codec, spec->init_amp);
- snd_hda_gen_init(codec);
-
snd_hda_apply_fixup(codec, HDA_FIXUP_ACT_INIT);
return 0;
bool hp_pin_sense;
if (!hp_pin)
- return;
+ hp_pin = 0x21;
msleep(30);
msleep(2);
alc_update_coefex_idx(codec, 0x57, 0x04, 0x0007, 0x1); /* Low power */
+ if (spec->ultra_low_power) {
+ alc_update_coef_idx(codec, 0x03, 1<<1, 1<<1);
+ alc_update_coef_idx(codec, 0x08, 3<<2, 3<<2);
+ alc_update_coef_idx(codec, 0x08, 7<<4, 0);
+ alc_update_coef_idx(codec, 0x3b, 1<<15, 0);
+ alc_update_coef_idx(codec, 0x0e, 7<<6, 7<<6);
+ msleep(30);
+ }
snd_hda_codec_write(codec, hp_pin, 0,
AC_VERB_SET_AMP_GAIN_MUTE, AMP_OUT_MUTE);
- if (hp_pin_sense)
+ if (hp_pin_sense || spec->ultra_low_power)
msleep(85);
snd_hda_codec_write(codec, hp_pin, 0,
AC_VERB_SET_PIN_WIDGET_CONTROL, PIN_OUT);
- if (hp_pin_sense)
+ if (hp_pin_sense || spec->ultra_low_power)
msleep(100);
alc_update_coef_idx(codec, 0x46, 3 << 12, 0);
hda_nid_t hp_pin = alc_get_hp_pin(spec);
bool hp_pin_sense;
- if (!hp_pin) {
- alc269_shutup(codec);
- return;
- }
+ if (!hp_pin)
+ hp_pin = 0x21;
hp_pin_sense = snd_hda_jack_detect(codec, hp_pin);
snd_hda_codec_write(codec, hp_pin, 0,
AC_VERB_SET_AMP_GAIN_MUTE, AMP_OUT_MUTE);
- if (hp_pin_sense)
+ if (hp_pin_sense || spec->ultra_low_power)
msleep(85);
/* 3k pull low control for Headset jack. */
snd_hda_codec_write(codec, hp_pin, 0,
AC_VERB_SET_PIN_WIDGET_CONTROL, 0x0);
- if (hp_pin_sense)
+ if (hp_pin_sense || spec->ultra_low_power)
msleep(100);
alc_auto_setup_eapd(codec, false);
alc_shutup_pins(codec);
+ if (spec->ultra_low_power) {
+ msleep(50);
+ alc_update_coef_idx(codec, 0x03, 1<<1, 0);
+ alc_update_coef_idx(codec, 0x08, 7<<4, 7<<4);
+ alc_update_coef_idx(codec, 0x08, 3<<2, 0);
+ alc_update_coef_idx(codec, 0x3b, 1<<15, 1<<15);
+ alc_update_coef_idx(codec, 0x0e, 7<<6, 0);
+ msleep(30);
+ }
}
static void alc225_init(struct hda_codec *codec)
bool hp1_pin_sense, hp2_pin_sense;
if (!hp_pin)
- return;
-
+ hp_pin = 0x21;
msleep(30);
hp1_pin_sense = snd_hda_jack_detect(codec, hp_pin);
msleep(2);
alc_update_coefex_idx(codec, 0x57, 0x04, 0x0007, 0x1); /* Low power */
+ if (spec->ultra_low_power) {
+ alc_update_coef_idx(codec, 0x08, 0x0f << 2, 3<<2);
+ alc_update_coef_idx(codec, 0x0e, 7<<6, 7<<6);
+ alc_update_coef_idx(codec, 0x33, 1<<11, 0);
+ msleep(30);
+ }
- if (hp1_pin_sense)
+ if (hp1_pin_sense || spec->ultra_low_power)
snd_hda_codec_write(codec, hp_pin, 0,
AC_VERB_SET_AMP_GAIN_MUTE, AMP_OUT_MUTE);
if (hp2_pin_sense)
snd_hda_codec_write(codec, 0x16, 0,
AC_VERB_SET_AMP_GAIN_MUTE, AMP_OUT_MUTE);
- if (hp1_pin_sense || hp2_pin_sense)
+ if (hp1_pin_sense || hp2_pin_sense || spec->ultra_low_power)
msleep(85);
- if (hp1_pin_sense)
+ if (hp1_pin_sense || spec->ultra_low_power)
snd_hda_codec_write(codec, hp_pin, 0,
AC_VERB_SET_PIN_WIDGET_CONTROL, PIN_OUT);
if (hp2_pin_sense)
snd_hda_codec_write(codec, 0x16, 0,
AC_VERB_SET_PIN_WIDGET_CONTROL, PIN_OUT);
- if (hp1_pin_sense || hp2_pin_sense)
+ if (hp1_pin_sense || hp2_pin_sense || spec->ultra_low_power)
msleep(100);
alc_update_coef_idx(codec, 0x4a, 3 << 10, 0);
hda_nid_t hp_pin = alc_get_hp_pin(spec);
bool hp1_pin_sense, hp2_pin_sense;
- if (!hp_pin) {
- alc269_shutup(codec);
- return;
- }
-
+ if (!hp_pin)
+ hp_pin = 0x21;
/* 3k pull low control for Headset jack. */
alc_update_coef_idx(codec, 0x4a, 0, 3 << 10);
if (hp1_pin_sense || hp2_pin_sense)
msleep(2);
- if (hp1_pin_sense)
+ if (hp1_pin_sense || spec->ultra_low_power)
snd_hda_codec_write(codec, hp_pin, 0,
AC_VERB_SET_AMP_GAIN_MUTE, AMP_OUT_MUTE);
if (hp2_pin_sense)
snd_hda_codec_write(codec, 0x16, 0,
AC_VERB_SET_AMP_GAIN_MUTE, AMP_OUT_MUTE);
- if (hp1_pin_sense || hp2_pin_sense)
+ if (hp1_pin_sense || hp2_pin_sense || spec->ultra_low_power)
msleep(85);
- if (hp1_pin_sense)
+ if (hp1_pin_sense || spec->ultra_low_power)
snd_hda_codec_write(codec, hp_pin, 0,
AC_VERB_SET_PIN_WIDGET_CONTROL, 0x0);
if (hp2_pin_sense)
snd_hda_codec_write(codec, 0x16, 0,
AC_VERB_SET_PIN_WIDGET_CONTROL, 0x0);
- if (hp1_pin_sense || hp2_pin_sense)
+ if (hp1_pin_sense || hp2_pin_sense || spec->ultra_low_power)
msleep(100);
alc_auto_setup_eapd(codec, false);
alc_shutup_pins(codec);
+ if (spec->ultra_low_power) {
+ msleep(50);
+ alc_update_coef_idx(codec, 0x08, 0x0f << 2, 0x0c << 2);
+ alc_update_coef_idx(codec, 0x0e, 7<<6, 0);
+ alc_update_coef_idx(codec, 0x33, 1<<11, 1<<11);
+ alc_update_coef_idx(codec, 0x4a, 3<<4, 2<<4);
+ msleep(30);
+ }
}
static void alc_default_init(struct hda_codec *codec)
static void alc295_fixup_chromebook(struct hda_codec *codec,
const struct hda_fixup *fix, int action)
{
+ struct alc_spec *spec = codec->spec;
+
switch (action) {
+ case HDA_FIXUP_ACT_PRE_PROBE:
+ spec->ultra_low_power = true;
+ break;
case HDA_FIXUP_ACT_INIT:
switch (codec->core.vendor_id) {
case 0x10ec0295:
SND_PCI_QUIRK(0x1462, 0xb120, "MSI Cubi MS-B120", ALC283_FIXUP_HEADSET_MIC),
SND_PCI_QUIRK(0x1462, 0xb171, "Cubi N 8GL (MS-B171)", ALC283_FIXUP_HEADSET_MIC),
SND_PCI_QUIRK(0x1558, 0x1325, "System76 Darter Pro (darp5)", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x8550, "System76 Gazelle (gaze14)", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x8560, "System76 Gazelle (gaze14)", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x17aa, 0x1036, "Lenovo P520", ALC233_FIXUP_LENOVO_MULTI_CODECS),
SND_PCI_QUIRK(0x17aa, 0x20f2, "Thinkpad SL410/510", ALC269_FIXUP_SKU_IGNORE),
SND_PCI_QUIRK(0x17aa, 0x215e, "Thinkpad L512", ALC269_FIXUP_SKU_IGNORE),
int fill_stages, dma_ch, stage;
enum snd_ps3_ch ch;
uint32_t ch0_kick_event = 0; /* initialize to mute gcc */
- void *start_vaddr;
unsigned long irqsave;
int silent = 0;
fill_stages = 4;
spin_lock_irqsave(&card->dma_lock, irqsave);
for (ch = 0; ch < 2; ch++) {
- start_vaddr = card->dma_next_transfer_vaddr[0];
for (stage = 0; stage < fill_stages; stage++) {
dma_ch = stage * 2 + ch;
if (silent)
{
struct snd_pcm_runtime *runtime = substream->runtime;
struct snd_ps3_card_info *card = snd_pcm_substream_chip(substream);
- int pcm_index;
- pcm_index = substream->pcm->device;
/* to retrieve substream/runtime in interrupt handler */
card->substream = substream;
{
struct snd_card_aica *dreamcastcard = from_timer(dreamcastcard,
t, timer);
- struct snd_pcm_substream *substream = dreamcastcard->timer_substream;
+ struct snd_pcm_substream *substream = dreamcastcard->substream;
/*timer function - so cannot sleep */
int play_period;
struct snd_pcm_runtime *runtime;
dreamcastcard = substream->pcm->private_data;
/*get the queue to do the work */
schedule_work(&(dreamcastcard->spu_dma_work));
- /* Timer may already be running */
- if (unlikely(dreamcastcard->timer_substream)) {
- mod_timer(&dreamcastcard->timer, jiffies + 4);
- return;
- }
- timer_setup(&dreamcastcard->timer, aica_period_elapsed, 0);
- dreamcastcard->timer_substream = substream;
mod_timer(&dreamcastcard->timer, jiffies + 4);
}
{
struct snd_card_aica *dreamcastcard = substream->pcm->private_data;
flush_work(&(dreamcastcard->spu_dma_work));
- if (dreamcastcard->timer_substream)
- del_timer(&dreamcastcard->timer);
+ del_timer(&dreamcastcard->timer);
+ dreamcastcard->substream = NULL;
kfree(dreamcastcard->channel);
spu_disable();
return 0;
"Yamaha AICA Super Intelligent Sound Processor for SEGA Dreamcast");
/* Prepare to use the queue */
INIT_WORK(&(dreamcastcard->spu_dma_work), run_spu_dma);
+ timer_setup(&dreamcastcard->timer, aica_period_elapsed, 0);
/* Load the PCM 'chip' */
err = snd_aicapcmchip(dreamcastcard, 0);
if (unlikely(err < 0))
source "sound/soc/samsung/Kconfig"
source "sound/soc/sh/Kconfig"
source "sound/soc/sirf/Kconfig"
+source "sound/soc/sof/Kconfig"
source "sound/soc/spear/Kconfig"
source "sound/soc/sprd/Kconfig"
source "sound/soc/sti/Kconfig"
obj-$(CONFIG_SND_SOC) += samsung/
obj-$(CONFIG_SND_SOC) += sh/
obj-$(CONFIG_SND_SOC) += sirf/
+obj-$(CONFIG_SND_SOC) += sof/
obj-$(CONFIG_SND_SOC) += spear/
obj-$(CONFIG_SND_SOC) += sprd/
obj-$(CONFIG_SND_SOC) += sti/
struct clk *clk;
struct clk *clk_ref;
+ bool has_capture;
+ bool has_playback;
+
struct snd_soc_dai_driver dai_driver;
struct snd_dmaengine_dai_dma_data capture_dma_data;
{
struct axi_i2s *i2s = snd_soc_dai_get_drvdata(dai);
- snd_soc_dai_init_dma_data(dai, &i2s->playback_dma_data,
- &i2s->capture_dma_data);
+ snd_soc_dai_init_dma_data(
+ dai,
+ i2s->has_playback ? &i2s->playback_dma_data : NULL,
+ i2s->has_capture ? &i2s->capture_dma_data : NULL);
return 0;
}
static struct snd_soc_dai_driver axi_i2s_dai = {
.probe = axi_i2s_dai_probe,
- .playback = {
- .channels_min = 2,
- .channels_max = 2,
- .rates = SNDRV_PCM_RATE_KNOT,
- .formats = SNDRV_PCM_FMTBIT_S32_LE | SNDRV_PCM_FMTBIT_U32_LE,
- },
- .capture = {
- .channels_min = 2,
- .channels_max = 2,
- .rates = SNDRV_PCM_RATE_KNOT,
- .formats = SNDRV_PCM_FMTBIT_S32_LE | SNDRV_PCM_FMTBIT_U32_LE,
- },
.ops = &axi_i2s_dai_ops,
.symmetric_rates = 1,
};
.max_register = AXI_I2S_REG_STATUS,
};
+static void axi_i2s_parse_of(struct axi_i2s *i2s, const struct device_node *np)
+{
+ struct property *dma_names;
+ const char *dma_name;
+
+ of_property_for_each_string(np, "dma-names", dma_names, dma_name) {
+ if (strcmp(dma_name, "rx") == 0)
+ i2s->has_capture = true;
+ if (strcmp(dma_name, "tx") == 0)
+ i2s->has_playback = true;
+ }
+}
+
static int axi_i2s_probe(struct platform_device *pdev)
{
struct resource *res;
platform_set_drvdata(pdev, i2s);
+ axi_i2s_parse_of(i2s, pdev->dev.of_node);
+
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
base = devm_ioremap_resource(&pdev->dev, res);
if (IS_ERR(base))
if (ret)
return ret;
- i2s->playback_dma_data.addr = res->start + AXI_I2S_REG_TX_FIFO;
- i2s->playback_dma_data.addr_width = 4;
- i2s->playback_dma_data.maxburst = 1;
+ if (i2s->has_playback) {
+ axi_i2s_dai.playback.channels_min = 2;
+ axi_i2s_dai.playback.channels_max = 2;
+ axi_i2s_dai.playback.rates = SNDRV_PCM_RATE_KNOT;
+ axi_i2s_dai.playback.formats =
+ SNDRV_PCM_FMTBIT_S32_LE | SNDRV_PCM_FMTBIT_U32_LE;
+
+ i2s->playback_dma_data.addr = res->start + AXI_I2S_REG_TX_FIFO;
+ i2s->playback_dma_data.addr_width = 4;
+ i2s->playback_dma_data.maxburst = 1;
+ }
+
+ if (i2s->has_capture) {
+ axi_i2s_dai.capture.channels_min = 2;
+ axi_i2s_dai.capture.channels_max = 2;
+ axi_i2s_dai.capture.rates = SNDRV_PCM_RATE_KNOT;
+ axi_i2s_dai.capture.formats =
+ SNDRV_PCM_FMTBIT_S32_LE | SNDRV_PCM_FMTBIT_U32_LE;
- i2s->capture_dma_data.addr = res->start + AXI_I2S_REG_RX_FIFO;
- i2s->capture_dma_data.addr_width = 4;
- i2s->capture_dma_data.maxburst = 1;
+ i2s->capture_dma_data.addr = res->start + AXI_I2S_REG_RX_FIFO;
+ i2s->capture_dma_data.addr_width = 4;
+ i2s->capture_dma_data.maxburst = 1;
+ }
i2s->ratnum.num = clk_get_rate(i2s->clk_ref) / 2 / AXI_I2S_BITS_PER_FRAME;
i2s->ratnum.den_step = 1;
if (ret)
goto err_clk_disable;
+ dev_info(&pdev->dev, "probed, capture %s, playback %s\n",
+ i2s->has_capture ? "enabled" : "disabled",
+ i2s->has_playback ? "enabled" : "disabled");
+
return 0;
err_clk_disable:
#define DUAL_CHANNEL 2
static struct snd_soc_jack cz_jack;
-static struct clk *da7219_dai_clk;
-extern int bt_uart_enable;
+static struct clk *da7219_dai_wclk;
+static struct clk *da7219_dai_bclk;
+extern bool bt_uart_enable;
static int cz_da7219_init(struct snd_soc_pcm_runtime *rtd)
{
return ret;
}
- da7219_dai_clk = clk_get(component->dev, "da7219-dai-clks");
+ da7219_dai_wclk = clk_get(component->dev, "da7219-dai-wclk");
+ da7219_dai_bclk = clk_get(component->dev, "da7219-dai-bclk");
ret = snd_soc_card_jack_new(card, "Headset Jack",
SND_JACK_HEADSET | SND_JACK_LINEOUT |
return 0;
}
-static int da7219_clk_enable(struct snd_pcm_substream *substream)
+static int da7219_clk_enable(struct snd_pcm_substream *substream,
+ int wclk_rate, int bclk_rate)
{
int ret = 0;
struct snd_soc_pcm_runtime *rtd = substream->private_data;
- ret = clk_prepare_enable(da7219_dai_clk);
+ clk_set_rate(da7219_dai_wclk, wclk_rate);
+ clk_set_rate(da7219_dai_bclk, bclk_rate);
+ ret = clk_prepare_enable(da7219_dai_bclk);
if (ret < 0) {
dev_err(rtd->dev, "can't enable master clock %d\n", ret);
return ret;
static void da7219_clk_disable(void)
{
- clk_disable_unprepare(da7219_dai_clk);
+ clk_disable_unprepare(da7219_dai_bclk);
}
static const unsigned int channels[] = {
&constraints_rates);
machine->play_i2s_instance = I2S_SP_INSTANCE;
- return da7219_clk_enable(substream);
+ return 0;
}
static int cz_da7219_cap_startup(struct snd_pcm_substream *substream)
machine->cap_i2s_instance = I2S_SP_INSTANCE;
machine->capture_channel = CAP_CHANNEL1;
- return da7219_clk_enable(substream);
-}
-
-static void cz_da7219_shutdown(struct snd_pcm_substream *substream)
-{
- da7219_clk_disable();
+ return 0;
}
static int cz_max_startup(struct snd_pcm_substream *substream)
&constraints_rates);
machine->play_i2s_instance = I2S_BT_INSTANCE;
- return da7219_clk_enable(substream);
-}
-
-static void cz_max_shutdown(struct snd_pcm_substream *substream)
-{
- da7219_clk_disable();
+ return 0;
}
static int cz_dmic0_startup(struct snd_pcm_substream *substream)
&constraints_rates);
machine->cap_i2s_instance = I2S_BT_INSTANCE;
- return da7219_clk_enable(substream);
+ return 0;
}
static int cz_dmic1_startup(struct snd_pcm_substream *substream)
machine->cap_i2s_instance = I2S_SP_INSTANCE;
machine->capture_channel = CAP_CHANNEL0;
- return da7219_clk_enable(substream);
+ return 0;
}
-static void cz_dmic_shutdown(struct snd_pcm_substream *substream)
+static int cz_da7219_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
+{
+ int wclk, bclk;
+
+ wclk = params_rate(params);
+ bclk = wclk * params_channels(params) *
+ snd_pcm_format_width(params_format(params));
+ /* ADAU7002 spec: "The ADAU7002 requires a BCLK rate
+ * that is minimum of 64x the LRCLK sample rate."
+ * DA7219 is the only clk source so for all codecs
+ * we have to limit bclk to 64X lrclk.
+ */
+ if (bclk < (wclk * 64))
+ bclk = wclk * 64;
+ return da7219_clk_enable(substream, wclk, bclk);
+}
+
+static void cz_da7219_shutdown(struct snd_pcm_substream *substream)
{
da7219_clk_disable();
}
static const struct snd_soc_ops cz_da7219_play_ops = {
.startup = cz_da7219_play_startup,
.shutdown = cz_da7219_shutdown,
+ .hw_params = cz_da7219_params,
};
static const struct snd_soc_ops cz_da7219_cap_ops = {
.startup = cz_da7219_cap_startup,
.shutdown = cz_da7219_shutdown,
+ .hw_params = cz_da7219_params,
};
static const struct snd_soc_ops cz_max_play_ops = {
.startup = cz_max_startup,
- .shutdown = cz_max_shutdown,
+ .shutdown = cz_da7219_shutdown,
+ .hw_params = cz_da7219_params,
};
static const struct snd_soc_ops cz_dmic0_cap_ops = {
.startup = cz_dmic0_startup,
- .shutdown = cz_dmic_shutdown,
+ .shutdown = cz_da7219_shutdown,
+ .hw_params = cz_da7219_params,
};
static const struct snd_soc_ops cz_dmic1_cap_ops = {
.startup = cz_dmic1_startup,
- .shutdown = cz_dmic_shutdown,
+ .shutdown = cz_da7219_shutdown,
+ .hw_params = cz_da7219_params,
};
static struct snd_soc_dai_link cz_dai_7219_98357[] = {
return ret;
}
-struct snd_soc_dai_ops acp3x_dai_i2s_ops = {
+static struct snd_soc_dai_ops acp3x_dai_i2s_ops = {
.hw_params = acp3x_dai_i2s_hwparams,
.trigger = acp3x_dai_i2s_trigger,
.set_fmt = acp3x_dai_i2s_set_fmt,
using I2C over SDA (MPU Data Input) and SCL (MPU Clock Input) pins.
Both playback and capture are supported.
+config SND_MCHP_SOC_I2S_MCC
+ tristate "Microchip ASoC driver for boards using I2S MCC"
+ depends on OF && (ARCH_AT91 || COMPILE_TEST)
+ select SND_SOC_GENERIC_DMAENGINE_PCM
+ select REGMAP_MMIO
+ help
+ Say Y or M if you want to add support for I2S Multi-Channel ASoC
+ driver on the following Microchip platforms:
+ - sam9x60
+
+ The I2SMCC complies with the Inter-IC Sound (I2S) bus specification
+ and supports a Time Division Multiplexed (TDM) interface with
+ external multi-channel audio codecs.
+
endif
snd-soc-atmel-pcm-dma-objs := atmel-pcm-dma.o
snd-soc-atmel_ssc_dai-objs := atmel_ssc_dai.o
snd-soc-atmel-i2s-objs := atmel-i2s.o
+snd-soc-mchp-i2s-mcc-objs := mchp-i2s-mcc.o
obj-$(CONFIG_SND_ATMEL_SOC_PDC) += snd-soc-atmel-pcm-pdc.o
obj-$(CONFIG_SND_ATMEL_SOC_DMA) += snd-soc-atmel-pcm-dma.o
obj-$(CONFIG_SND_ATMEL_SOC_SSC) += snd-soc-atmel_ssc_dai.o
obj-$(CONFIG_SND_ATMEL_SOC_I2S) += snd-soc-atmel-i2s.o
+obj-$(CONFIG_SND_MCHP_SOC_I2S_MCC) += snd-soc-mchp-i2s-mcc.o
# AT91 Machine Support
snd-soc-sam9g20-wm8731-objs := sam9g20_wm8731.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Driver for Microchip I2S Multi-channel controller
+//
+// Copyright (C) 2018 Microchip Technology Inc. and its subsidiaries
+//
+// Author: Codrin Ciubotariu <codrin.ciubotariu@microchip.com>
+
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/slab.h>
+
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/clk.h>
+#include <linux/mfd/syscon.h>
+#include <linux/lcm.h>
+
+#include <sound/core.h>
+#include <sound/pcm.h>
+#include <sound/pcm_params.h>
+#include <sound/initval.h>
+#include <sound/soc.h>
+#include <sound/dmaengine_pcm.h>
+
+/*
+ * ---- I2S Controller Register map ----
+ */
+#define MCHP_I2SMCC_CR 0x0000 /* Control Register */
+#define MCHP_I2SMCC_MRA 0x0004 /* Mode Register A */
+#define MCHP_I2SMCC_MRB 0x0008 /* Mode Register B */
+#define MCHP_I2SMCC_SR 0x000C /* Status Register */
+#define MCHP_I2SMCC_IERA 0x0010 /* Interrupt Enable Register A */
+#define MCHP_I2SMCC_IDRA 0x0014 /* Interrupt Disable Register A */
+#define MCHP_I2SMCC_IMRA 0x0018 /* Interrupt Mask Register A */
+#define MCHP_I2SMCC_ISRA 0X001C /* Interrupt Status Register A */
+
+#define MCHP_I2SMCC_IERB 0x0020 /* Interrupt Enable Register B */
+#define MCHP_I2SMCC_IDRB 0x0024 /* Interrupt Disable Register B */
+#define MCHP_I2SMCC_IMRB 0x0028 /* Interrupt Mask Register B */
+#define MCHP_I2SMCC_ISRB 0X002C /* Interrupt Status Register B */
+
+#define MCHP_I2SMCC_RHR 0x0030 /* Receiver Holding Register */
+#define MCHP_I2SMCC_THR 0x0034 /* Transmitter Holding Register */
+
+#define MCHP_I2SMCC_RHL0R 0x0040 /* Receiver Holding Left 0 Register */
+#define MCHP_I2SMCC_RHR0R 0x0044 /* Receiver Holding Right 0 Register */
+
+#define MCHP_I2SMCC_RHL1R 0x0048 /* Receiver Holding Left 1 Register */
+#define MCHP_I2SMCC_RHR1R 0x004C /* Receiver Holding Right 1 Register */
+
+#define MCHP_I2SMCC_RHL2R 0x0050 /* Receiver Holding Left 2 Register */
+#define MCHP_I2SMCC_RHR2R 0x0054 /* Receiver Holding Right 2 Register */
+
+#define MCHP_I2SMCC_RHL3R 0x0058 /* Receiver Holding Left 3 Register */
+#define MCHP_I2SMCC_RHR3R 0x005C /* Receiver Holding Right 3 Register */
+
+#define MCHP_I2SMCC_THL0R 0x0060 /* Transmitter Holding Left 0 Register */
+#define MCHP_I2SMCC_THR0R 0x0064 /* Transmitter Holding Right 0 Register */
+
+#define MCHP_I2SMCC_THL1R 0x0068 /* Transmitter Holding Left 1 Register */
+#define MCHP_I2SMCC_THR1R 0x006C /* Transmitter Holding Right 1 Register */
+
+#define MCHP_I2SMCC_THL2R 0x0070 /* Transmitter Holding Left 2 Register */
+#define MCHP_I2SMCC_THR2R 0x0074 /* Transmitter Holding Right 2 Register */
+
+#define MCHP_I2SMCC_THL3R 0x0078 /* Transmitter Holding Left 3 Register */
+#define MCHP_I2SMCC_THR3R 0x007C /* Transmitter Holding Right 3 Register */
+
+#define MCHP_I2SMCC_VERSION 0x00FC /* Version Register */
+
+/*
+ * ---- Control Register (Write-only) ----
+ */
+#define MCHP_I2SMCC_CR_RXEN BIT(0) /* Receiver Enable */
+#define MCHP_I2SMCC_CR_RXDIS BIT(1) /* Receiver Disable */
+#define MCHP_I2SMCC_CR_CKEN BIT(2) /* Clock Enable */
+#define MCHP_I2SMCC_CR_CKDIS BIT(3) /* Clock Disable */
+#define MCHP_I2SMCC_CR_TXEN BIT(4) /* Transmitter Enable */
+#define MCHP_I2SMCC_CR_TXDIS BIT(5) /* Transmitter Disable */
+#define MCHP_I2SMCC_CR_SWRST BIT(7) /* Software Reset */
+
+/*
+ * ---- Mode Register A (Read/Write) ----
+ */
+#define MCHP_I2SMCC_MRA_MODE_MASK GENMASK(0, 0)
+#define MCHP_I2SMCC_MRA_MODE_SLAVE (0 << 0)
+#define MCHP_I2SMCC_MRA_MODE_MASTER (1 << 0)
+
+#define MCHP_I2SMCC_MRA_DATALENGTH_MASK GENMASK(3, 1)
+#define MCHP_I2SMCC_MRA_DATALENGTH_32_BITS (0 << 1)
+#define MCHP_I2SMCC_MRA_DATALENGTH_24_BITS (1 << 1)
+#define MCHP_I2SMCC_MRA_DATALENGTH_20_BITS (2 << 1)
+#define MCHP_I2SMCC_MRA_DATALENGTH_18_BITS (3 << 1)
+#define MCHP_I2SMCC_MRA_DATALENGTH_16_BITS (4 << 1)
+#define MCHP_I2SMCC_MRA_DATALENGTH_16_BITS_COMPACT (5 << 1)
+#define MCHP_I2SMCC_MRA_DATALENGTH_8_BITS (6 << 1)
+#define MCHP_I2SMCC_MRA_DATALENGTH_8_BITS_COMPACT (7 << 1)
+
+#define MCHP_I2SMCC_MRA_WIRECFG_MASK GENMASK(5, 4)
+#define MCHP_I2SMCC_MRA_WIRECFG_I2S_1_TDM_0 (0 << 4)
+#define MCHP_I2SMCC_MRA_WIRECFG_I2S_2_TDM_1 (1 << 4)
+#define MCHP_I2SMCC_MRA_WIRECFG_I2S_4_TDM_2 (2 << 4)
+#define MCHP_I2SMCC_MRA_WIRECFG_TDM_3 (3 << 4)
+
+#define MCHP_I2SMCC_MRA_FORMAT_MASK GENMASK(7, 6)
+#define MCHP_I2SMCC_MRA_FORMAT_I2S (0 << 6)
+#define MCHP_I2SMCC_MRA_FORMAT_LJ (1 << 6) /* Left Justified */
+#define MCHP_I2SMCC_MRA_FORMAT_TDM (2 << 6)
+#define MCHP_I2SMCC_MRA_FORMAT_TDMLJ (3 << 6)
+
+/* Transmitter uses one DMA channel ... */
+/* Left audio samples duplicated to right audio channel */
+#define MCHP_I2SMCC_MRA_RXMONO BIT(8)
+
+/* I2SDO output of I2SC is internally connected to I2SDI input */
+#define MCHP_I2SMCC_MRA_RXLOOP BIT(9)
+
+/* Receiver uses one DMA channel ... */
+/* Left audio samples duplicated to right audio channel */
+#define MCHP_I2SMCC_MRA_TXMONO BIT(10)
+
+/* x sample transmitted when underrun */
+#define MCHP_I2SMCC_MRA_TXSAME_ZERO (0 << 11) /* Zero sample */
+#define MCHP_I2SMCC_MRA_TXSAME_PREVIOUS (1 << 11) /* Previous sample */
+
+/* select between peripheral clock and generated clock */
+#define MCHP_I2SMCC_MRA_SRCCLK_PCLK (0 << 12)
+#define MCHP_I2SMCC_MRA_SRCCLK_GCLK (1 << 12)
+
+/* Number of TDM Channels - 1 */
+#define MCHP_I2SMCC_MRA_NBCHAN_MASK GENMASK(15, 13)
+#define MCHP_I2SMCC_MRA_NBCHAN(ch) \
+ ((((ch) - 1) << 13) & MCHP_I2SMCC_MRA_NBCHAN_MASK)
+
+/* Selected Clock to I2SMCC Master Clock ratio */
+#define MCHP_I2SMCC_MRA_IMCKDIV_MASK GENMASK(21, 16)
+#define MCHP_I2SMCC_MRA_IMCKDIV(div) \
+ (((div) << 16) & MCHP_I2SMCC_MRA_IMCKDIV_MASK)
+
+/* TDM Frame Synchronization */
+#define MCHP_I2SMCC_MRA_TDMFS_MASK GENMASK(23, 22)
+#define MCHP_I2SMCC_MRA_TDMFS_SLOT (0 << 22)
+#define MCHP_I2SMCC_MRA_TDMFS_HALF (1 << 22)
+#define MCHP_I2SMCC_MRA_TDMFS_BIT (2 << 22)
+
+/* Selected Clock to I2SMC Serial Clock ratio */
+#define MCHP_I2SMCC_MRA_ISCKDIV_MASK GENMASK(29, 24)
+#define MCHP_I2SMCC_MRA_ISCKDIV(div) \
+ (((div) << 24) & MCHP_I2SMCC_MRA_ISCKDIV_MASK)
+
+/* Master Clock mode */
+#define MCHP_I2SMCC_MRA_IMCKMODE_MASK GENMASK(30, 30)
+/* 0: No master clock generated*/
+#define MCHP_I2SMCC_MRA_IMCKMODE_NONE (0 << 30)
+/* 1: master clock generated (internally generated clock drives I2SMCK pin) */
+#define MCHP_I2SMCC_MRA_IMCKMODE_GEN (1 << 30)
+
+/* Slot Width */
+/* 0: slot is 32 bits wide for DATALENGTH = 18/20/24 bits. */
+/* 1: slot is 24 bits wide for DATALENGTH = 18/20/24 bits. */
+#define MCHP_I2SMCC_MRA_IWS BIT(31)
+
+/*
+ * ---- Mode Register B (Read/Write) ----
+ */
+/* all enabled I2S left channels are filled first, then I2S right channels */
+#define MCHP_I2SMCC_MRB_CRAMODE_LEFT_FIRST (0 << 0)
+/*
+ * an enabled I2S left channel is filled, then the corresponding right
+ * channel, until all channels are filled
+ */
+#define MCHP_I2SMCC_MRB_CRAMODE_REGULAR (1 << 0)
+
+#define MCHP_I2SMCC_MRB_FIFOEN BIT(1)
+
+#define MCHP_I2SMCC_MRB_DMACHUNK_MASK GENMASK(9, 8)
+#define MCHP_I2SMCC_MRB_DMACHUNK(no_words) \
+ (((fls(no_words) - 1) << 8) & MCHP_I2SMCC_MRB_DMACHUNK_MASK)
+
+#define MCHP_I2SMCC_MRB_CLKSEL_MASK GENMASK(16, 16)
+#define MCHP_I2SMCC_MRB_CLKSEL_EXT (0 << 16)
+#define MCHP_I2SMCC_MRB_CLKSEL_INT (1 << 16)
+
+/*
+ * ---- Status Registers (Read-only) ----
+ */
+#define MCHP_I2SMCC_SR_RXEN BIT(0) /* Receiver Enabled */
+#define MCHP_I2SMCC_SR_TXEN BIT(4) /* Transmitter Enabled */
+
+/*
+ * ---- Interrupt Enable/Disable/Mask/Status Registers A ----
+ */
+#define MCHP_I2SMCC_INT_TXRDY_MASK(ch) GENMASK((ch) - 1, 0)
+#define MCHP_I2SMCC_INT_TXRDYCH(ch) BIT(ch)
+#define MCHP_I2SMCC_INT_TXUNF_MASK(ch) GENMASK((ch) + 7, 8)
+#define MCHP_I2SMCC_INT_TXUNFCH(ch) BIT((ch) + 8)
+#define MCHP_I2SMCC_INT_RXRDY_MASK(ch) GENMASK((ch) + 15, 16)
+#define MCHP_I2SMCC_INT_RXRDYCH(ch) BIT((ch) + 16)
+#define MCHP_I2SMCC_INT_RXOVF_MASK(ch) GENMASK((ch) + 23, 24)
+#define MCHP_I2SMCC_INT_RXOVFCH(ch) BIT((ch) + 24)
+
+/*
+ * ---- Interrupt Enable/Disable/Mask/Status Registers B ----
+ */
+#define MCHP_I2SMCC_INT_WERR BIT(0)
+#define MCHP_I2SMCC_INT_TXFFRDY BIT(8)
+#define MCHP_I2SMCC_INT_TXFFEMP BIT(9)
+#define MCHP_I2SMCC_INT_RXFFRDY BIT(12)
+#define MCHP_I2SMCC_INT_RXFFFUL BIT(13)
+
+/*
+ * ---- Version Register (Read-only) ----
+ */
+#define MCHP_I2SMCC_VERSION_MASK GENMASK(11, 0)
+
+#define MCHP_I2SMCC_MAX_CHANNELS 8
+#define MCHP_I2MCC_TDM_SLOT_WIDTH 32
+
+static const struct regmap_config mchp_i2s_mcc_regmap_config = {
+ .reg_bits = 32,
+ .reg_stride = 4,
+ .val_bits = 32,
+ .max_register = MCHP_I2SMCC_VERSION,
+};
+
+struct mchp_i2s_mcc_dev {
+ struct wait_queue_head wq_txrdy;
+ struct wait_queue_head wq_rxrdy;
+ struct device *dev;
+ struct regmap *regmap;
+ struct clk *pclk;
+ struct clk *gclk;
+ struct snd_dmaengine_dai_dma_data playback;
+ struct snd_dmaengine_dai_dma_data capture;
+ unsigned int fmt;
+ unsigned int sysclk;
+ unsigned int frame_length;
+ int tdm_slots;
+ int channels;
+ int gclk_use:1;
+ int gclk_running:1;
+ int tx_rdy:1;
+ int rx_rdy:1;
+};
+
+static irqreturn_t mchp_i2s_mcc_interrupt(int irq, void *dev_id)
+{
+ struct mchp_i2s_mcc_dev *dev = dev_id;
+ u32 sra, imra, srb, imrb, pendinga, pendingb, idra = 0;
+ irqreturn_t ret = IRQ_NONE;
+
+ regmap_read(dev->regmap, MCHP_I2SMCC_IMRA, &imra);
+ regmap_read(dev->regmap, MCHP_I2SMCC_ISRA, &sra);
+ pendinga = imra & sra;
+
+ regmap_read(dev->regmap, MCHP_I2SMCC_IMRB, &imrb);
+ regmap_read(dev->regmap, MCHP_I2SMCC_ISRB, &srb);
+ pendingb = imrb & srb;
+
+ if (!pendinga && !pendingb)
+ return IRQ_NONE;
+
+ /*
+ * Tx/Rx ready interrupts are enabled when stopping only, to assure
+ * availability and to disable clocks if necessary
+ */
+ idra |= pendinga & (MCHP_I2SMCC_INT_TXRDY_MASK(dev->channels) |
+ MCHP_I2SMCC_INT_RXRDY_MASK(dev->channels));
+ if (idra)
+ ret = IRQ_HANDLED;
+
+ if ((imra & MCHP_I2SMCC_INT_TXRDY_MASK(dev->channels)) &&
+ (imra & MCHP_I2SMCC_INT_TXRDY_MASK(dev->channels)) ==
+ (idra & MCHP_I2SMCC_INT_TXRDY_MASK(dev->channels))) {
+ dev->tx_rdy = 1;
+ wake_up_interruptible(&dev->wq_txrdy);
+ }
+ if ((imra & MCHP_I2SMCC_INT_RXRDY_MASK(dev->channels)) &&
+ (imra & MCHP_I2SMCC_INT_RXRDY_MASK(dev->channels)) ==
+ (idra & MCHP_I2SMCC_INT_RXRDY_MASK(dev->channels))) {
+ dev->rx_rdy = 1;
+ wake_up_interruptible(&dev->wq_rxrdy);
+ }
+ regmap_write(dev->regmap, MCHP_I2SMCC_IDRA, idra);
+
+ return ret;
+}
+
+static int mchp_i2s_mcc_set_sysclk(struct snd_soc_dai *dai,
+ int clk_id, unsigned int freq, int dir)
+{
+ struct mchp_i2s_mcc_dev *dev = snd_soc_dai_get_drvdata(dai);
+
+ dev_dbg(dev->dev, "%s() clk_id=%d freq=%u dir=%d\n",
+ __func__, clk_id, freq, dir);
+
+ /* We do not need SYSCLK */
+ if (dir == SND_SOC_CLOCK_IN)
+ return 0;
+
+ dev->sysclk = freq;
+
+ return 0;
+}
+
+static int mchp_i2s_mcc_set_bclk_ratio(struct snd_soc_dai *dai,
+ unsigned int ratio)
+{
+ struct mchp_i2s_mcc_dev *dev = snd_soc_dai_get_drvdata(dai);
+
+ dev_dbg(dev->dev, "%s() ratio=%u\n", __func__, ratio);
+
+ dev->frame_length = ratio;
+
+ return 0;
+}
+
+static int mchp_i2s_mcc_set_dai_fmt(struct snd_soc_dai *dai, unsigned int fmt)
+{
+ struct mchp_i2s_mcc_dev *dev = snd_soc_dai_get_drvdata(dai);
+
+ dev_dbg(dev->dev, "%s() fmt=%#x\n", __func__, fmt);
+
+ /* We don't support any kind of clock inversion */
+ if ((fmt & SND_SOC_DAIFMT_INV_MASK) != SND_SOC_DAIFMT_NB_NF)
+ return -EINVAL;
+
+ /* We can't generate only FSYNC */
+ if ((fmt & SND_SOC_DAIFMT_MASTER_MASK) == SND_SOC_DAIFMT_CBM_CFS)
+ return -EINVAL;
+
+ /* We can only reconfigure the IP when it's stopped */
+ if (fmt & SND_SOC_DAIFMT_CONT)
+ return -EINVAL;
+
+ dev->fmt = fmt;
+
+ return 0;
+}
+
+static int mchp_i2s_mcc_set_dai_tdm_slot(struct snd_soc_dai *dai,
+ unsigned int tx_mask,
+ unsigned int rx_mask,
+ int slots, int slot_width)
+{
+ struct mchp_i2s_mcc_dev *dev = snd_soc_dai_get_drvdata(dai);
+
+ dev_dbg(dev->dev,
+ "%s() tx_mask=0x%08x rx_mask=0x%08x slots=%d width=%d\n",
+ __func__, tx_mask, rx_mask, slots, slot_width);
+
+ if (slots < 0 || slots > MCHP_I2SMCC_MAX_CHANNELS ||
+ slot_width != MCHP_I2MCC_TDM_SLOT_WIDTH)
+ return -EINVAL;
+
+ if (slots) {
+ /* We do not support daisy chain */
+ if (rx_mask != GENMASK(slots - 1, 0) ||
+ rx_mask != tx_mask)
+ return -EINVAL;
+ }
+
+ dev->tdm_slots = slots;
+ dev->frame_length = slots * MCHP_I2MCC_TDM_SLOT_WIDTH;
+
+ return 0;
+}
+
+static int mchp_i2s_mcc_clk_get_rate_diff(struct clk *clk,
+ unsigned long rate,
+ struct clk **best_clk,
+ unsigned long *best_rate,
+ unsigned long *best_diff_rate)
+{
+ long round_rate;
+ unsigned int diff_rate;
+
+ round_rate = clk_round_rate(clk, rate);
+ if (round_rate < 0)
+ return (int)round_rate;
+
+ diff_rate = abs(rate - round_rate);
+ if (diff_rate < *best_diff_rate) {
+ *best_clk = clk;
+ *best_diff_rate = diff_rate;
+ *best_rate = rate;
+ }
+
+ return 0;
+}
+
+static int mchp_i2s_mcc_config_divs(struct mchp_i2s_mcc_dev *dev,
+ unsigned int bclk, unsigned int *mra)
+{
+ unsigned long clk_rate;
+ unsigned long lcm_rate;
+ unsigned long best_rate = 0;
+ unsigned long best_diff_rate = ~0;
+ unsigned int sysclk;
+ struct clk *best_clk = NULL;
+ int ret;
+
+ /* For code simplification */
+ if (!dev->sysclk)
+ sysclk = bclk;
+ else
+ sysclk = dev->sysclk;
+
+ /*
+ * MCLK is Selected CLK / (2 * IMCKDIV),
+ * BCLK is Selected CLK / (2 * ISCKDIV);
+ * if IMCKDIV or ISCKDIV are 0, MCLK or BCLK = Selected CLK
+ */
+ lcm_rate = lcm(sysclk, bclk);
+ if ((lcm_rate / sysclk % 2 == 1 && lcm_rate / sysclk > 2) ||
+ (lcm_rate / bclk % 2 == 1 && lcm_rate / bclk > 2))
+ lcm_rate *= 2;
+
+ for (clk_rate = lcm_rate;
+ (clk_rate == sysclk || clk_rate / (sysclk * 2) <= GENMASK(5, 0)) &&
+ (clk_rate == bclk || clk_rate / (bclk * 2) <= GENMASK(5, 0));
+ clk_rate += lcm_rate) {
+ ret = mchp_i2s_mcc_clk_get_rate_diff(dev->gclk, clk_rate,
+ &best_clk, &best_rate,
+ &best_diff_rate);
+ if (ret) {
+ dev_err(dev->dev, "gclk error for rate %lu: %d",
+ clk_rate, ret);
+ } else {
+ if (!best_diff_rate) {
+ dev_dbg(dev->dev, "found perfect rate on gclk: %lu\n",
+ clk_rate);
+ break;
+ }
+ }
+
+ ret = mchp_i2s_mcc_clk_get_rate_diff(dev->pclk, clk_rate,
+ &best_clk, &best_rate,
+ &best_diff_rate);
+ if (ret) {
+ dev_err(dev->dev, "pclk error for rate %lu: %d",
+ clk_rate, ret);
+ } else {
+ if (!best_diff_rate) {
+ dev_dbg(dev->dev, "found perfect rate on pclk: %lu\n",
+ clk_rate);
+ break;
+ }
+ }
+ }
+
+ /* check if clocks returned only errors */
+ if (!best_clk) {
+ dev_err(dev->dev, "unable to change rate to clocks\n");
+ return -EINVAL;
+ }
+
+ dev_dbg(dev->dev, "source CLK is %s with rate %lu, diff %lu\n",
+ best_clk == dev->pclk ? "pclk" : "gclk",
+ best_rate, best_diff_rate);
+
+ /* set the rate */
+ ret = clk_set_rate(best_clk, best_rate);
+ if (ret) {
+ dev_err(dev->dev, "unable to set rate %lu to %s: %d\n",
+ best_rate, best_clk == dev->pclk ? "PCLK" : "GCLK",
+ ret);
+ return ret;
+ }
+
+ /* Configure divisors */
+ if (dev->sysclk)
+ *mra |= MCHP_I2SMCC_MRA_IMCKDIV(best_rate / (2 * sysclk));
+ *mra |= MCHP_I2SMCC_MRA_ISCKDIV(best_rate / (2 * bclk));
+
+ if (best_clk == dev->gclk) {
+ *mra |= MCHP_I2SMCC_MRA_SRCCLK_GCLK;
+ ret = clk_prepare(dev->gclk);
+ if (ret < 0)
+ dev_err(dev->dev, "unable to prepare GCLK: %d\n", ret);
+ else
+ dev->gclk_use = 1;
+ } else {
+ *mra |= MCHP_I2SMCC_MRA_SRCCLK_PCLK;
+ dev->gclk_use = 0;
+ }
+
+ return 0;
+}
+
+static int mchp_i2s_mcc_is_running(struct mchp_i2s_mcc_dev *dev)
+{
+ u32 sr;
+
+ regmap_read(dev->regmap, MCHP_I2SMCC_SR, &sr);
+ return !!(sr & (MCHP_I2SMCC_SR_TXEN | MCHP_I2SMCC_SR_RXEN));
+}
+
+static int mchp_i2s_mcc_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params,
+ struct snd_soc_dai *dai)
+{
+ struct mchp_i2s_mcc_dev *dev = snd_soc_dai_get_drvdata(dai);
+ u32 mra = 0;
+ u32 mrb = 0;
+ unsigned int channels = params_channels(params);
+ unsigned int frame_length = dev->frame_length;
+ unsigned int bclk_rate;
+ int set_divs = 0;
+ int ret;
+ bool is_playback = (substream->stream == SNDRV_PCM_STREAM_PLAYBACK);
+
+ dev_dbg(dev->dev, "%s() rate=%u format=%#x width=%u channels=%u\n",
+ __func__, params_rate(params), params_format(params),
+ params_width(params), params_channels(params));
+
+ switch (dev->fmt & SND_SOC_DAIFMT_FORMAT_MASK) {
+ case SND_SOC_DAIFMT_I2S:
+ if (dev->tdm_slots) {
+ dev_err(dev->dev, "I2S with TDM is not supported\n");
+ return -EINVAL;
+ }
+ mra |= MCHP_I2SMCC_MRA_FORMAT_I2S;
+ break;
+ case SND_SOC_DAIFMT_LEFT_J:
+ if (dev->tdm_slots) {
+ dev_err(dev->dev, "Left-Justified with TDM is not supported\n");
+ return -EINVAL;
+ }
+ mra |= MCHP_I2SMCC_MRA_FORMAT_LJ;
+ break;
+ case SND_SOC_DAIFMT_DSP_A:
+ mra |= MCHP_I2SMCC_MRA_FORMAT_TDM;
+ break;
+ default:
+ dev_err(dev->dev, "unsupported bus format\n");
+ return -EINVAL;
+ }
+
+ switch (dev->fmt & SND_SOC_DAIFMT_MASTER_MASK) {
+ case SND_SOC_DAIFMT_CBS_CFS:
+ /* cpu is BCLK and LRC master */
+ mra |= MCHP_I2SMCC_MRA_MODE_MASTER;
+ if (dev->sysclk)
+ mra |= MCHP_I2SMCC_MRA_IMCKMODE_GEN;
+ set_divs = 1;
+ break;
+ case SND_SOC_DAIFMT_CBS_CFM:
+ /* cpu is BCLK master */
+ mrb |= MCHP_I2SMCC_MRB_CLKSEL_INT;
+ set_divs = 1;
+ /* fall through */
+ case SND_SOC_DAIFMT_CBM_CFM:
+ /* cpu is slave */
+ mra |= MCHP_I2SMCC_MRA_MODE_SLAVE;
+ if (dev->sysclk)
+ dev_warn(dev->dev, "Unable to generate MCLK in Slave mode\n");
+ break;
+ default:
+ dev_err(dev->dev, "unsupported master/slave mode\n");
+ return -EINVAL;
+ }
+
+ if (dev->fmt & (SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_LEFT_J)) {
+ switch (channels) {
+ case 1:
+ if (is_playback)
+ mra |= MCHP_I2SMCC_MRA_TXMONO;
+ else
+ mra |= MCHP_I2SMCC_MRA_RXMONO;
+ break;
+ case 2:
+ break;
+ default:
+ dev_err(dev->dev, "unsupported number of audio channels\n");
+ return -EINVAL;
+ }
+
+ if (!frame_length)
+ frame_length = 2 * params_physical_width(params);
+ } else if (dev->fmt & SND_SOC_DAIFMT_DSP_A) {
+ if (dev->tdm_slots) {
+ if (channels % 2 && channels * 2 <= dev->tdm_slots) {
+ /*
+ * Duplicate data for even-numbered channels
+ * to odd-numbered channels
+ */
+ if (is_playback)
+ mra |= MCHP_I2SMCC_MRA_TXMONO;
+ else
+ mra |= MCHP_I2SMCC_MRA_RXMONO;
+ }
+ channels = dev->tdm_slots;
+ }
+
+ mra |= MCHP_I2SMCC_MRA_NBCHAN(channels);
+ if (!frame_length)
+ frame_length = channels * MCHP_I2MCC_TDM_SLOT_WIDTH;
+ }
+
+ /*
+ * We must have the same burst size configured
+ * in the DMA transfer and in out IP
+ */
+ mrb |= MCHP_I2SMCC_MRB_DMACHUNK(channels);
+ if (is_playback)
+ dev->playback.maxburst = 1 << (fls(channels) - 1);
+ else
+ dev->capture.maxburst = 1 << (fls(channels) - 1);
+
+ switch (params_format(params)) {
+ case SNDRV_PCM_FORMAT_S8:
+ mra |= MCHP_I2SMCC_MRA_DATALENGTH_8_BITS;
+ break;
+ case SNDRV_PCM_FORMAT_S16_LE:
+ mra |= MCHP_I2SMCC_MRA_DATALENGTH_16_BITS;
+ break;
+ case SNDRV_PCM_FORMAT_S18_3LE:
+ mra |= MCHP_I2SMCC_MRA_DATALENGTH_18_BITS |
+ MCHP_I2SMCC_MRA_IWS;
+ break;
+ case SNDRV_PCM_FORMAT_S20_3LE:
+ mra |= MCHP_I2SMCC_MRA_DATALENGTH_20_BITS |
+ MCHP_I2SMCC_MRA_IWS;
+ break;
+ case SNDRV_PCM_FORMAT_S24_3LE:
+ mra |= MCHP_I2SMCC_MRA_DATALENGTH_24_BITS |
+ MCHP_I2SMCC_MRA_IWS;
+ break;
+ case SNDRV_PCM_FORMAT_S24_LE:
+ mra |= MCHP_I2SMCC_MRA_DATALENGTH_24_BITS;
+ break;
+ case SNDRV_PCM_FORMAT_S32_LE:
+ mra |= MCHP_I2SMCC_MRA_DATALENGTH_32_BITS;
+ break;
+ default:
+ dev_err(dev->dev, "unsupported size/endianness for audio samples\n");
+ return -EINVAL;
+ }
+
+ /*
+ * If we are already running, the wanted setup must be
+ * the same with the one that's currently ongoing
+ */
+ if (mchp_i2s_mcc_is_running(dev)) {
+ u32 mra_cur;
+ u32 mrb_cur;
+
+ regmap_read(dev->regmap, MCHP_I2SMCC_MRA, &mra_cur);
+ regmap_read(dev->regmap, MCHP_I2SMCC_MRB, &mrb_cur);
+ if (mra != mra_cur || mrb != mrb_cur)
+ return -EINVAL;
+
+ return 0;
+ }
+
+ /* Save the number of channels to know what interrupts to enable */
+ dev->channels = channels;
+
+ if (set_divs) {
+ bclk_rate = frame_length * params_rate(params);
+ ret = mchp_i2s_mcc_config_divs(dev, bclk_rate, &mra);
+ if (ret) {
+ dev_err(dev->dev, "unable to configure the divisors: %d\n",
+ ret);
+ return ret;
+ }
+ }
+
+ ret = regmap_write(dev->regmap, MCHP_I2SMCC_MRA, mra);
+ if (ret < 0)
+ return ret;
+ return regmap_write(dev->regmap, MCHP_I2SMCC_MRB, mrb);
+}
+
+static int mchp_i2s_mcc_hw_free(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct mchp_i2s_mcc_dev *dev = snd_soc_dai_get_drvdata(dai);
+ bool is_playback = (substream->stream == SNDRV_PCM_STREAM_PLAYBACK);
+ long err;
+
+ if (is_playback) {
+ err = wait_event_interruptible_timeout(dev->wq_txrdy,
+ dev->tx_rdy,
+ msecs_to_jiffies(500));
+ } else {
+ err = wait_event_interruptible_timeout(dev->wq_rxrdy,
+ dev->rx_rdy,
+ msecs_to_jiffies(500));
+ }
+
+ if (err == 0) {
+ u32 idra;
+
+ dev_warn_once(dev->dev, "Timeout waiting for %s\n",
+ is_playback ? "Tx ready" : "Rx ready");
+ if (is_playback)
+ idra = MCHP_I2SMCC_INT_TXRDY_MASK(dev->channels);
+ else
+ idra = MCHP_I2SMCC_INT_RXRDY_MASK(dev->channels);
+ regmap_write(dev->regmap, MCHP_I2SMCC_IDRA, idra);
+ }
+
+ if (!mchp_i2s_mcc_is_running(dev)) {
+ regmap_write(dev->regmap, MCHP_I2SMCC_CR, MCHP_I2SMCC_CR_CKDIS);
+
+ if (dev->gclk_running) {
+ clk_disable_unprepare(dev->gclk);
+ dev->gclk_running = 0;
+ }
+ }
+
+ return 0;
+}
+
+static int mchp_i2s_mcc_trigger(struct snd_pcm_substream *substream, int cmd,
+ struct snd_soc_dai *dai)
+{
+ struct mchp_i2s_mcc_dev *dev = snd_soc_dai_get_drvdata(dai);
+ bool is_playback = (substream->stream == SNDRV_PCM_STREAM_PLAYBACK);
+ u32 cr = 0;
+ u32 iera = 0;
+ u32 sr;
+ int err;
+
+ switch (cmd) {
+ case SNDRV_PCM_TRIGGER_START:
+ case SNDRV_PCM_TRIGGER_RESUME:
+ case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
+ if (is_playback)
+ cr = MCHP_I2SMCC_CR_TXEN | MCHP_I2SMCC_CR_CKEN;
+ else
+ cr = MCHP_I2SMCC_CR_RXEN | MCHP_I2SMCC_CR_CKEN;
+ break;
+ case SNDRV_PCM_TRIGGER_STOP:
+ case SNDRV_PCM_TRIGGER_SUSPEND:
+ case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
+ regmap_read(dev->regmap, MCHP_I2SMCC_SR, &sr);
+ if (is_playback && (sr & MCHP_I2SMCC_SR_TXEN)) {
+ cr = MCHP_I2SMCC_CR_TXDIS;
+ dev->tx_rdy = 0;
+ /*
+ * Enable Tx Ready interrupts on all channels
+ * to assure all data is sent
+ */
+ iera = MCHP_I2SMCC_INT_TXRDY_MASK(dev->channels);
+ } else if (!is_playback && (sr & MCHP_I2SMCC_SR_RXEN)) {
+ cr = MCHP_I2SMCC_CR_RXDIS;
+ dev->rx_rdy = 0;
+ /*
+ * Enable Rx Ready interrupts on all channels
+ * to assure all data is received
+ */
+ iera = MCHP_I2SMCC_INT_RXRDY_MASK(dev->channels);
+ }
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if ((cr & MCHP_I2SMCC_CR_CKEN) && dev->gclk_use &&
+ !dev->gclk_running) {
+ err = clk_enable(dev->gclk);
+ if (err) {
+ dev_err_once(dev->dev, "failed to enable GCLK: %d\n",
+ err);
+ } else {
+ dev->gclk_running = 1;
+ }
+ }
+
+ regmap_write(dev->regmap, MCHP_I2SMCC_IERA, iera);
+ regmap_write(dev->regmap, MCHP_I2SMCC_CR, cr);
+
+ return 0;
+}
+
+static int mchp_i2s_mcc_startup(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct mchp_i2s_mcc_dev *dev = snd_soc_dai_get_drvdata(dai);
+
+ /* Software reset the IP if it's not running */
+ if (!mchp_i2s_mcc_is_running(dev)) {
+ return regmap_write(dev->regmap, MCHP_I2SMCC_CR,
+ MCHP_I2SMCC_CR_SWRST);
+ }
+
+ return 0;
+}
+
+static const struct snd_soc_dai_ops mchp_i2s_mcc_dai_ops = {
+ .set_sysclk = mchp_i2s_mcc_set_sysclk,
+ .set_bclk_ratio = mchp_i2s_mcc_set_bclk_ratio,
+ .startup = mchp_i2s_mcc_startup,
+ .trigger = mchp_i2s_mcc_trigger,
+ .hw_params = mchp_i2s_mcc_hw_params,
+ .hw_free = mchp_i2s_mcc_hw_free,
+ .set_fmt = mchp_i2s_mcc_set_dai_fmt,
+ .set_tdm_slot = mchp_i2s_mcc_set_dai_tdm_slot,
+};
+
+static int mchp_i2s_mcc_dai_probe(struct snd_soc_dai *dai)
+{
+ struct mchp_i2s_mcc_dev *dev = snd_soc_dai_get_drvdata(dai);
+
+ init_waitqueue_head(&dev->wq_txrdy);
+ init_waitqueue_head(&dev->wq_rxrdy);
+
+ snd_soc_dai_init_dma_data(dai, &dev->playback, &dev->capture);
+
+ return 0;
+}
+
+#define MCHP_I2SMCC_RATES SNDRV_PCM_RATE_8000_192000
+
+#define MCHP_I2SMCC_FORMATS (SNDRV_PCM_FMTBIT_S8 | \
+ SNDRV_PCM_FMTBIT_S16_LE | \
+ SNDRV_PCM_FMTBIT_S18_3LE | \
+ SNDRV_PCM_FMTBIT_S20_3LE | \
+ SNDRV_PCM_FMTBIT_S24_3LE | \
+ SNDRV_PCM_FMTBIT_S24_LE | \
+ SNDRV_PCM_FMTBIT_S32_LE)
+
+static struct snd_soc_dai_driver mchp_i2s_mcc_dai = {
+ .probe = mchp_i2s_mcc_dai_probe,
+ .playback = {
+ .stream_name = "I2SMCC-Playback",
+ .channels_min = 1,
+ .channels_max = 8,
+ .rates = MCHP_I2SMCC_RATES,
+ .formats = MCHP_I2SMCC_FORMATS,
+ },
+ .capture = {
+ .stream_name = "I2SMCC-Capture",
+ .channels_min = 1,
+ .channels_max = 8,
+ .rates = MCHP_I2SMCC_RATES,
+ .formats = MCHP_I2SMCC_FORMATS,
+ },
+ .ops = &mchp_i2s_mcc_dai_ops,
+ .symmetric_rates = 1,
+ .symmetric_samplebits = 1,
+ .symmetric_channels = 1,
+};
+
+static const struct snd_soc_component_driver mchp_i2s_mcc_component = {
+ .name = "mchp-i2s-mcc",
+};
+
+#ifdef CONFIG_OF
+static const struct of_device_id mchp_i2s_mcc_dt_ids[] = {
+ {
+ .compatible = "microchip,sam9x60-i2smcc",
+ },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, mchp_i2s_mcc_dt_ids);
+#endif
+
+static int mchp_i2s_mcc_probe(struct platform_device *pdev)
+{
+ struct mchp_i2s_mcc_dev *dev;
+ struct resource *mem;
+ struct regmap *regmap;
+ void __iomem *base;
+ u32 version;
+ int irq;
+ int err;
+
+ dev = devm_kzalloc(&pdev->dev, sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return -ENOMEM;
+
+ mem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ base = devm_ioremap_resource(&pdev->dev, mem);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ regmap = devm_regmap_init_mmio(&pdev->dev, base,
+ &mchp_i2s_mcc_regmap_config);
+ if (IS_ERR(regmap))
+ return PTR_ERR(regmap);
+
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0)
+ return irq;
+
+ err = devm_request_irq(&pdev->dev, irq, mchp_i2s_mcc_interrupt, 0,
+ dev_name(&pdev->dev), dev);
+ if (err)
+ return err;
+
+ dev->pclk = devm_clk_get(&pdev->dev, "pclk");
+ if (IS_ERR(dev->pclk)) {
+ err = PTR_ERR(dev->pclk);
+ dev_err(&pdev->dev,
+ "failed to get the peripheral clock: %d\n", err);
+ return err;
+ }
+
+ /* Get the optional generated clock */
+ dev->gclk = devm_clk_get(&pdev->dev, "gclk");
+ if (IS_ERR(dev->gclk)) {
+ if (PTR_ERR(dev->gclk) == -EPROBE_DEFER)
+ return -EPROBE_DEFER;
+ dev_warn(&pdev->dev,
+ "generated clock not found: %d\n", err);
+ dev->gclk = NULL;
+ }
+
+ dev->dev = &pdev->dev;
+ dev->regmap = regmap;
+ platform_set_drvdata(pdev, dev);
+
+ err = clk_prepare_enable(dev->pclk);
+ if (err) {
+ dev_err(&pdev->dev,
+ "failed to enable the peripheral clock: %d\n", err);
+ return err;
+ }
+
+ err = devm_snd_soc_register_component(&pdev->dev,
+ &mchp_i2s_mcc_component,
+ &mchp_i2s_mcc_dai, 1);
+ if (err) {
+ dev_err(&pdev->dev, "failed to register DAI: %d\n", err);
+ clk_disable_unprepare(dev->pclk);
+ return err;
+ }
+
+ dev->playback.addr = (dma_addr_t)mem->start + MCHP_I2SMCC_THR;
+ dev->capture.addr = (dma_addr_t)mem->start + MCHP_I2SMCC_RHR;
+
+ err = devm_snd_dmaengine_pcm_register(&pdev->dev, NULL, 0);
+ if (err) {
+ dev_err(&pdev->dev, "failed to register PCM: %d\n", err);
+ clk_disable_unprepare(dev->pclk);
+ return err;
+ }
+
+ /* Get IP version. */
+ regmap_read(dev->regmap, MCHP_I2SMCC_VERSION, &version);
+ dev_info(&pdev->dev, "hw version: %#lx\n",
+ version & MCHP_I2SMCC_VERSION_MASK);
+
+ return 0;
+}
+
+static int mchp_i2s_mcc_remove(struct platform_device *pdev)
+{
+ struct mchp_i2s_mcc_dev *dev = platform_get_drvdata(pdev);
+
+ clk_disable_unprepare(dev->pclk);
+
+ return 0;
+}
+
+static struct platform_driver mchp_i2s_mcc_driver = {
+ .driver = {
+ .name = "mchp_i2s_mcc",
+ .of_match_table = of_match_ptr(mchp_i2s_mcc_dt_ids),
+ },
+ .probe = mchp_i2s_mcc_probe,
+ .remove = mchp_i2s_mcc_remove,
+};
+module_platform_driver(mchp_i2s_mcc_driver);
+
+MODULE_DESCRIPTION("Microchip I2S Multi-Channel Controller driver");
+MODULE_AUTHOR("Codrin Ciubotariu <codrin.ciubotariu@microchip.com>");
+MODULE_LICENSE("GPL v2");
return snd_soc_dapm_put_enum_double(kctrl, ucontrol);
}
-int tse850_get_mix(struct snd_kcontrol *kctrl,
- struct snd_ctl_elem_value *ucontrol)
+static int tse850_get_mix(struct snd_kcontrol *kctrl,
+ struct snd_ctl_elem_value *ucontrol)
{
struct snd_soc_dapm_context *dapm = snd_soc_dapm_kcontrol_dapm(kctrl);
struct snd_soc_card *card = dapm->card;
return 0;
}
-int tse850_put_mix(struct snd_kcontrol *kctrl,
- struct snd_ctl_elem_value *ucontrol)
+static int tse850_put_mix(struct snd_kcontrol *kctrl,
+ struct snd_ctl_elem_value *ucontrol)
{
struct snd_soc_dapm_context *dapm = snd_soc_dapm_kcontrol_dapm(kctrl);
struct snd_soc_card *card = dapm->card;
return 1;
}
-int tse850_get_ana(struct snd_kcontrol *kctrl,
- struct snd_ctl_elem_value *ucontrol)
+static int tse850_get_ana(struct snd_kcontrol *kctrl,
+ struct snd_ctl_elem_value *ucontrol)
{
struct snd_soc_dapm_context *dapm = snd_soc_dapm_kcontrol_dapm(kctrl);
struct snd_soc_card *card = dapm->card;
return 0;
}
-int tse850_put_ana(struct snd_kcontrol *kctrl,
- struct snd_ctl_elem_value *ucontrol)
+static int tse850_put_ana(struct snd_kcontrol *kctrl,
+ struct snd_ctl_elem_value *ucontrol)
{
struct snd_soc_dapm_context *dapm = snd_soc_dapm_kcontrol_dapm(kctrl);
struct snd_soc_card *card = dapm->card;
select SND_SOC_JZ4725B_CODEC
select SND_SOC_LM4857 if I2C
select SND_SOC_LM49453 if I2C
+ select SND_SOC_LOCHNAGAR_SC if MFD_LOCHNAGAR
select SND_SOC_MAX98088 if I2C
select SND_SOC_MAX98090 if I2C
select SND_SOC_MAX98095 if I2C
select SND_SOC_TLV320AIC23_SPI if SPI_MASTER
select SND_SOC_TLV320AIC26 if SPI_MASTER
select SND_SOC_TLV320AIC31XX if I2C
- select SND_SOC_TLV320AIC32X4_I2C if I2C
- select SND_SOC_TLV320AIC32X4_SPI if SPI_MASTER
+ select SND_SOC_TLV320AIC32X4_I2C if I2C && COMMON_CLK
+ select SND_SOC_TLV320AIC32X4_SPI if SPI_MASTER && COMMON_CLK
select SND_SOC_TLV320AIC3X if I2C
select SND_SOC_TPA6130A2 if I2C
select SND_SOC_TLV320DAC33 if I2C
config SND_SOC_LM49453
tristate
+config SND_SOC_LOCHNAGAR_SC
+ tristate "Lochnagar Sound Card"
+ depends on MFD_LOCHNAGAR
+ help
+ This driver support the sound card functionality of the Cirrus
+ Logic Lochnagar audio development board.
+
config SND_SOC_MAX98088
tristate "Maxim MAX98088/9 Low-Power, Stereo Audio Codec"
depends on I2C
config SND_SOC_TLV320AIC32X4
tristate
+ depends on COMMON_CLK
config SND_SOC_TLV320AIC32X4_I2C
tristate "Texas Instruments TLV320AIC32x4 audio CODECs - I2C"
depends on I2C
+ depends on COMMON_CLK
select SND_SOC_TLV320AIC32X4
config SND_SOC_TLV320AIC32X4_SPI
tristate "Texas Instruments TLV320AIC32x4 audio CODECs - SPI"
depends on SPI_MASTER
+ depends on COMMON_CLK
select SND_SOC_TLV320AIC32X4
config SND_SOC_TLV320AIC3X
snd-soc-l3-objs := l3.o
snd-soc-lm4857-objs := lm4857.o
snd-soc-lm49453-objs := lm49453.o
+snd-soc-lochnagar-sc-objs := lochnagar-sc.o
snd-soc-max9759-objs := max9759.o
snd-soc-max9768-objs := max9768.o
snd-soc-max98088-objs := max98088.o
snd-soc-tlv320aic23-spi-objs := tlv320aic23-spi.o
snd-soc-tlv320aic26-objs := tlv320aic26.o
snd-soc-tlv320aic31xx-objs := tlv320aic31xx.o
-snd-soc-tlv320aic32x4-objs := tlv320aic32x4.o
+snd-soc-tlv320aic32x4-objs := tlv320aic32x4.o tlv320aic32x4-clk.o
snd-soc-tlv320aic32x4-i2c-objs := tlv320aic32x4-i2c.o
snd-soc-tlv320aic32x4-spi-objs := tlv320aic32x4-spi.o
snd-soc-tlv320aic3x-objs := tlv320aic3x.o
obj-$(CONFIG_SND_SOC_L3) += snd-soc-l3.o
obj-$(CONFIG_SND_SOC_LM4857) += snd-soc-lm4857.o
obj-$(CONFIG_SND_SOC_LM49453) += snd-soc-lm49453.o
+obj-$(CONFIG_SND_SOC_LOCHNAGAR_SC) += snd-soc-lochnagar-sc.o
obj-$(CONFIG_SND_SOC_MAX9759) += snd-soc-max9759.o
obj-$(CONFIG_SND_SOC_MAX9768) += snd-soc-max9768.o
obj-$(CONFIG_SND_SOC_MAX98088) += snd-soc-max98088.o
struct regmap_config config;
config = cs42l51_regmap;
- config.val_bits = 8;
- config.reg_bits = 8;
return cs42l51_probe(&i2c->dev, devm_regmap_init_i2c(i2c, &config));
}
+static int cs42l51_i2c_remove(struct i2c_client *i2c)
+{
+ return cs42l51_remove(&i2c->dev);
+}
+
+static const struct dev_pm_ops cs42l51_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(cs42l51_suspend, cs42l51_resume)
+};
+
static struct i2c_driver cs42l51_i2c_driver = {
.driver = {
.name = "cs42l51",
.of_match_table = cs42l51_of_match,
+ .pm = &cs42l51_pm_ops,
},
.probe = cs42l51_i2c_probe,
+ .remove = cs42l51_i2c_remove,
.id_table = cs42l51_i2c_id,
};
#include <sound/initval.h>
#include <sound/pcm_params.h>
#include <sound/pcm.h>
+#include <linux/gpio/consumer.h>
#include <linux/regmap.h>
+#include <linux/regulator/consumer.h>
#include "cs42l51.h"
MODE_MASTER,
};
+static const char * const cs42l51_supply_names[] = {
+ "VL",
+ "VD",
+ "VA",
+ "VAHP",
+};
+
struct cs42l51_private {
unsigned int mclk;
struct clk *mclk_handle;
unsigned int audio_mode; /* The mode (I2S or left-justified) */
enum master_slave_mode func;
+ struct regulator_bulk_data supplies[ARRAY_SIZE(cs42l51_supply_names)];
+ struct gpio_desc *reset_gpio;
+ struct regmap *regmap;
};
#define CS42L51_FORMATS ( \
static const DECLARE_TLV_DB_SCALE(aout_tlv, -10200, 50, 0);
static const DECLARE_TLV_DB_SCALE(boost_tlv, 1600, 1600, 0);
+static const DECLARE_TLV_DB_SCALE(adc_boost_tlv, 2000, 2000, 0);
static const char *chan_mix[] = {
"L R",
"L+R",
SOC_SINGLE("Zero Cross Switch", CS42L51_DAC_CTL, 0, 0, 0),
SOC_DOUBLE_TLV("Mic Boost Volume",
CS42L51_MIC_CTL, 0, 1, 1, 0, boost_tlv),
+ SOC_DOUBLE_TLV("ADC Boost Volume",
+ CS42L51_MIC_CTL, 5, 6, 1, 0, adc_boost_tlv),
SOC_SINGLE_TLV("Bass Volume", CS42L51_TONE_CTL, 0, 0xf, 1, tone_tlv),
SOC_SINGLE_TLV("Treble Volume", CS42L51_TONE_CTL, 4, 0xf, 1, tone_tlv),
SOC_ENUM_EXT("PCM channel mixer",
SOC_DAPM_ENUM("Route", cs42l51_adcr_mux_enum);
static const struct snd_soc_dapm_widget cs42l51_dapm_widgets[] = {
- SND_SOC_DAPM_MICBIAS("Mic Bias", CS42L51_MIC_POWER_CTL, 1, 1),
+ SND_SOC_DAPM_SUPPLY("Mic Bias", CS42L51_MIC_POWER_CTL, 1, 1, NULL,
+ SND_SOC_DAPM_PRE_PMU | SND_SOC_DAPM_POST_PMD),
SND_SOC_DAPM_PGA_E("Left PGA", CS42L51_POWER_CTL1, 3, 1, NULL, 0,
cs42l51_pdn_event, SND_SOC_DAPM_PRE_POST_PMD),
SND_SOC_DAPM_PGA_E("Right PGA", CS42L51_POWER_CTL1, 4, 1, NULL, 0,
{ 256, CS42L51_DSM_MODE, 1 }, { 384, CS42L51_DSM_MODE, 1 },
};
+/*
+ * Master mode mclk/fs ratios.
+ * Recommended configurations are SSM for 4-50khz and DSM for 50-100kHz ranges
+ * The table below provides support of following ratios:
+ * 128: SSM (%128) with div2 disabled
+ * 256: SSM (%128) with div2 enabled
+ * In both cases, if sampling rate is above 50kHz, SSM is overridden
+ * with DSM (%128) configuration
+ */
+static struct cs42l51_ratios master_ratios[] = {
+ { 128, CS42L51_SSM_MODE, 0 }, { 256, CS42L51_SSM_MODE, 1 },
+};
+
static int cs42l51_set_dai_sysclk(struct snd_soc_dai *codec_dai,
int clk_id, unsigned int freq, int dir)
{
unsigned int ratio;
struct cs42l51_ratios *ratios = NULL;
int nr_ratios = 0;
- int intf_ctl, power_ctl, fmt;
+ int intf_ctl, power_ctl, fmt, mode;
switch (cs42l51->func) {
case MODE_MASTER:
- return -EINVAL;
+ ratios = master_ratios;
+ nr_ratios = ARRAY_SIZE(master_ratios);
+ break;
case MODE_SLAVE:
ratios = slave_ratios;
nr_ratios = ARRAY_SIZE(slave_ratios);
switch (cs42l51->func) {
case MODE_MASTER:
intf_ctl |= CS42L51_INTF_CTL_MASTER;
- power_ctl |= CS42L51_MIC_POWER_CTL_SPEED(ratios[i].speed_mode);
+ mode = ratios[i].speed_mode;
+ /* Force DSM mode if sampling rate is above 50kHz */
+ if (rate > 50000)
+ mode = CS42L51_DSM_MODE;
+ power_ctl |= CS42L51_MIC_POWER_CTL_SPEED(mode);
+ /*
+ * Auto detect mode is not applicable for master mode and has to
+ * be disabled. Otherwise SPEED[1:0] bits will be ignored.
+ */
+ power_ctl &= ~CS42L51_MIC_POWER_CTL_AUTO;
break;
case MODE_SLAVE:
power_ctl |= CS42L51_MIC_POWER_CTL_SPEED(ratios[i].speed_mode);
return snd_soc_component_write(component, CS42L51_DAC_OUT_CTL, reg);
}
+static int cs42l51_of_xlate_dai_id(struct snd_soc_component *component,
+ struct device_node *endpoint)
+{
+ /* return dai id 0, whatever the endpoint index */
+ return 0;
+}
+
static const struct snd_soc_dai_ops cs42l51_dai_ops = {
.hw_params = cs42l51_hw_params,
.set_sysclk = cs42l51_set_dai_sysclk,
.num_dapm_widgets = ARRAY_SIZE(cs42l51_dapm_widgets),
.dapm_routes = cs42l51_routes,
.num_dapm_routes = ARRAY_SIZE(cs42l51_routes),
+ .of_xlate_dai_id = cs42l51_of_xlate_dai_id,
.idle_bias_on = 1,
.use_pmdown_time = 1,
.endianness = 1,
.non_legacy_dai_naming = 1,
};
+static bool cs42l51_writeable_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case CS42L51_POWER_CTL1:
+ case CS42L51_MIC_POWER_CTL:
+ case CS42L51_INTF_CTL:
+ case CS42L51_MIC_CTL:
+ case CS42L51_ADC_CTL:
+ case CS42L51_ADC_INPUT:
+ case CS42L51_DAC_OUT_CTL:
+ case CS42L51_DAC_CTL:
+ case CS42L51_ALC_PGA_CTL:
+ case CS42L51_ALC_PGB_CTL:
+ case CS42L51_ADCA_ATT:
+ case CS42L51_ADCB_ATT:
+ case CS42L51_ADCA_VOL:
+ case CS42L51_ADCB_VOL:
+ case CS42L51_PCMA_VOL:
+ case CS42L51_PCMB_VOL:
+ case CS42L51_BEEP_FREQ:
+ case CS42L51_BEEP_VOL:
+ case CS42L51_BEEP_CONF:
+ case CS42L51_TONE_CTL:
+ case CS42L51_AOUTA_VOL:
+ case CS42L51_AOUTB_VOL:
+ case CS42L51_PCM_MIXER:
+ case CS42L51_LIMIT_THRES_DIS:
+ case CS42L51_LIMIT_REL:
+ case CS42L51_LIMIT_ATT:
+ case CS42L51_ALC_EN:
+ case CS42L51_ALC_REL:
+ case CS42L51_ALC_THRES:
+ case CS42L51_NOISE_CONF:
+ case CS42L51_CHARGE_FREQ:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static bool cs42l51_volatile_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case CS42L51_STATUS:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static bool cs42l51_readable_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case CS42L51_CHIP_REV_ID:
+ case CS42L51_POWER_CTL1:
+ case CS42L51_MIC_POWER_CTL:
+ case CS42L51_INTF_CTL:
+ case CS42L51_MIC_CTL:
+ case CS42L51_ADC_CTL:
+ case CS42L51_ADC_INPUT:
+ case CS42L51_DAC_OUT_CTL:
+ case CS42L51_DAC_CTL:
+ case CS42L51_ALC_PGA_CTL:
+ case CS42L51_ALC_PGB_CTL:
+ case CS42L51_ADCA_ATT:
+ case CS42L51_ADCB_ATT:
+ case CS42L51_ADCA_VOL:
+ case CS42L51_ADCB_VOL:
+ case CS42L51_PCMA_VOL:
+ case CS42L51_PCMB_VOL:
+ case CS42L51_BEEP_FREQ:
+ case CS42L51_BEEP_VOL:
+ case CS42L51_BEEP_CONF:
+ case CS42L51_TONE_CTL:
+ case CS42L51_AOUTA_VOL:
+ case CS42L51_AOUTB_VOL:
+ case CS42L51_PCM_MIXER:
+ case CS42L51_LIMIT_THRES_DIS:
+ case CS42L51_LIMIT_REL:
+ case CS42L51_LIMIT_ATT:
+ case CS42L51_ALC_EN:
+ case CS42L51_ALC_REL:
+ case CS42L51_ALC_THRES:
+ case CS42L51_NOISE_CONF:
+ case CS42L51_STATUS:
+ case CS42L51_CHARGE_FREQ:
+ return true;
+ default:
+ return false;
+ }
+}
+
const struct regmap_config cs42l51_regmap = {
+ .reg_bits = 8,
+ .reg_stride = 1,
+ .val_bits = 8,
+ .use_single_write = true,
+ .readable_reg = cs42l51_readable_reg,
+ .volatile_reg = cs42l51_volatile_reg,
+ .writeable_reg = cs42l51_writeable_reg,
.max_register = CS42L51_CHARGE_FREQ,
.cache_type = REGCACHE_RBTREE,
};
{
struct cs42l51_private *cs42l51;
unsigned int val;
- int ret;
+ int ret, i;
if (IS_ERR(regmap))
return PTR_ERR(regmap);
return -ENOMEM;
dev_set_drvdata(dev, cs42l51);
+ cs42l51->regmap = regmap;
cs42l51->mclk_handle = devm_clk_get(dev, "MCLK");
if (IS_ERR(cs42l51->mclk_handle)) {
cs42l51->mclk_handle = NULL;
}
+ for (i = 0; i < ARRAY_SIZE(cs42l51->supplies); i++)
+ cs42l51->supplies[i].supply = cs42l51_supply_names[i];
+
+ ret = devm_regulator_bulk_get(dev, ARRAY_SIZE(cs42l51->supplies),
+ cs42l51->supplies);
+ if (ret != 0) {
+ dev_err(dev, "Failed to request supplies: %d\n", ret);
+ return ret;
+ }
+
+ ret = regulator_bulk_enable(ARRAY_SIZE(cs42l51->supplies),
+ cs42l51->supplies);
+ if (ret != 0) {
+ dev_err(dev, "Failed to enable supplies: %d\n", ret);
+ return ret;
+ }
+
+ cs42l51->reset_gpio = devm_gpiod_get_optional(dev, "reset",
+ GPIOD_OUT_LOW);
+ if (IS_ERR(cs42l51->reset_gpio))
+ return PTR_ERR(cs42l51->reset_gpio);
+
+ if (cs42l51->reset_gpio) {
+ dev_dbg(dev, "Release reset gpio\n");
+ gpiod_set_value_cansleep(cs42l51->reset_gpio, 0);
+ mdelay(2);
+ }
+
/* Verify that we have a CS42L51 */
ret = regmap_read(regmap, CS42L51_CHIP_REV_ID, &val);
if (ret < 0) {
ret = devm_snd_soc_register_component(dev,
&soc_component_device_cs42l51, &cs42l51_dai, 1);
+ if (ret < 0)
+ goto error;
+
+ return 0;
+
error:
+ regulator_bulk_disable(ARRAY_SIZE(cs42l51->supplies),
+ cs42l51->supplies);
return ret;
}
EXPORT_SYMBOL_GPL(cs42l51_probe);
+int cs42l51_remove(struct device *dev)
+{
+ struct cs42l51_private *cs42l51 = dev_get_drvdata(dev);
+
+ gpiod_set_value_cansleep(cs42l51->reset_gpio, 1);
+
+ return regulator_bulk_disable(ARRAY_SIZE(cs42l51->supplies),
+ cs42l51->supplies);
+}
+EXPORT_SYMBOL_GPL(cs42l51_remove);
+
+int __maybe_unused cs42l51_suspend(struct device *dev)
+{
+ struct cs42l51_private *cs42l51 = dev_get_drvdata(dev);
+
+ regcache_cache_only(cs42l51->regmap, true);
+ regcache_mark_dirty(cs42l51->regmap);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cs42l51_suspend);
+
+int __maybe_unused cs42l51_resume(struct device *dev)
+{
+ struct cs42l51_private *cs42l51 = dev_get_drvdata(dev);
+
+ regcache_cache_only(cs42l51->regmap, false);
+
+ return regcache_sync(cs42l51->regmap);
+}
+EXPORT_SYMBOL_GPL(cs42l51_resume);
+
const struct of_device_id cs42l51_of_match[] = {
{ .compatible = "cirrus,cs42l51", },
{ }
extern const struct regmap_config cs42l51_regmap;
int cs42l51_probe(struct device *dev, struct regmap *regmap);
+int cs42l51_remove(struct device *dev);
+int __maybe_unused cs42l51_suspend(struct device *dev);
+int __maybe_unused cs42l51_resume(struct device *dev);
extern const struct of_device_id cs42l51_of_match[];
#define CS42L51_CHIP_ID 0x1B
return ret;
cs43130->wq = create_singlethread_workqueue("cs43130_hp");
+ if (!cs43130->wq)
+ return -ENOMEM;
INIT_WORK(&cs43130->work, cs43130_imp_meas);
}
v = (v & ARIZONA_SYSCLK_FREQ_MASK) >> ARIZONA_SYSCLK_FREQ_SHIFT;
- return wm_adsp2_early_event(w, kcontrol, event, v);
+ wm_adsp2_set_dspclk(w, v);
+
+ return wm_adsp_early_event(w, kcontrol, event);
}
static DECLARE_TLV_DB_SCALE(eq_tlv, -1200, 100, 0);
/* By default only 64 BCLK per WCLK is supported */
dai_clk_mode |= DA7213_DAI_BCLKS_PER_WCLK_64;
- snd_soc_component_write(component, DA7213_DAI_CLK_MODE, dai_clk_mode);
+ snd_soc_component_update_bits(component, DA7213_DAI_CLK_MODE,
+ DA7213_DAI_BCLKS_PER_WCLK_MASK |
+ DA7213_DAI_CLK_POL_MASK | DA7213_DAI_WCLK_POL_MASK,
+ dai_clk_mode);
snd_soc_component_update_bits(component, DA7213_DAI_CTRL, DA7213_DAI_FORMAT_MASK,
dai_ctrl);
snd_soc_component_write(component, DA7213_DAI_OFFSET, dai_offset);
#define DA7213_DAI_BCLKS_PER_WCLK_256 (0x3 << 0)
#define DA7213_DAI_BCLKS_PER_WCLK_MASK (0x3 << 0)
#define DA7213_DAI_CLK_POL_INV (0x1 << 2)
+#define DA7213_DAI_CLK_POL_MASK (0x1 << 2)
#define DA7213_DAI_WCLK_POL_INV (0x1 << 3)
+#define DA7213_DAI_WCLK_POL_MASK (0x1 << 3)
#define DA7213_DAI_CLK_EN_MASK (0x1 << 7)
/* DA7213_DAI_CTRL = 0x29 */
{
struct snd_soc_component *component = snd_soc_dapm_to_component(w->dapm);
struct da7219_priv *da7219 = snd_soc_component_get_drvdata(component);
+ struct clk *bclk = da7219->dai_clks[DA7219_DAI_BCLK_IDX];
u8 pll_ctrl, pll_status;
int i = 0, ret;
bool srm_lock = false;
case SND_SOC_DAPM_PRE_PMU:
if (da7219->master) {
/* Enable DAI clks for master mode */
- if (da7219->dai_clks) {
- ret = clk_prepare_enable(da7219->dai_clks);
+ if (bclk) {
+ ret = clk_prepare_enable(bclk);
if (ret) {
dev_err(component->dev,
- "Failed to enable dai_clks\n");
+ "Failed to enable DAI clks\n");
return ret;
}
} else {
/* Disable DAI clks if in master mode */
if (da7219->master) {
- if (da7219->dai_clks)
- clk_disable_unprepare(da7219->dai_clks);
+ if (bclk)
+ clk_disable_unprepare(bclk);
else
snd_soc_component_update_bits(component,
DA7219_DAI_CLK_MODE,
return 0;
}
+static int da7219_set_bclks_per_wclk(struct snd_soc_component *component,
+ unsigned long factor)
+{
+ u8 bclks_per_wclk;
+
+ switch (factor) {
+ case 32:
+ bclks_per_wclk = DA7219_DAI_BCLKS_PER_WCLK_32;
+ break;
+ case 64:
+ bclks_per_wclk = DA7219_DAI_BCLKS_PER_WCLK_64;
+ break;
+ case 128:
+ bclks_per_wclk = DA7219_DAI_BCLKS_PER_WCLK_128;
+ break;
+ case 256:
+ bclks_per_wclk = DA7219_DAI_BCLKS_PER_WCLK_256;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ snd_soc_component_update_bits(component, DA7219_DAI_CLK_MODE,
+ DA7219_DAI_BCLKS_PER_WCLK_MASK,
+ bclks_per_wclk);
+
+ return 0;
+}
+
static int da7219_set_dai_tdm_slot(struct snd_soc_dai *dai,
unsigned int tx_mask, unsigned int rx_mask,
int slots, int slot_width)
{
struct snd_soc_component *component = dai->component;
struct da7219_priv *da7219 = snd_soc_component_get_drvdata(component);
+ struct clk *wclk = da7219->dai_clks[DA7219_DAI_WCLK_IDX];
+ struct clk *bclk = da7219->dai_clks[DA7219_DAI_BCLK_IDX];
unsigned int ch_mask;
- u8 dai_bclks_per_wclk, slot_offset;
+ unsigned long sr, bclk_rate;
+ u8 slot_offset;
u16 offset;
__le16 dai_offset;
u32 frame_size;
+ int ret;
/* No channels enabled so disable TDM */
if (!tx_mask) {
*/
if (da7219->master) {
frame_size = slots * slot_width;
- switch (frame_size) {
- case 32:
- dai_bclks_per_wclk = DA7219_DAI_BCLKS_PER_WCLK_32;
- break;
- case 64:
- dai_bclks_per_wclk = DA7219_DAI_BCLKS_PER_WCLK_64;
- break;
- case 128:
- dai_bclks_per_wclk = DA7219_DAI_BCLKS_PER_WCLK_128;
- break;
- case 256:
- dai_bclks_per_wclk = DA7219_DAI_BCLKS_PER_WCLK_256;
- break;
- default:
- dev_err(component->dev, "Invalid frame size %d\n",
- frame_size);
- return -EINVAL;
- }
- snd_soc_component_update_bits(component, DA7219_DAI_CLK_MODE,
- DA7219_DAI_BCLKS_PER_WCLK_MASK,
- dai_bclks_per_wclk);
+ if (bclk) {
+ sr = clk_get_rate(wclk);
+ bclk_rate = sr * frame_size;
+ ret = clk_set_rate(bclk, bclk_rate);
+ if (ret) {
+ dev_err(component->dev,
+ "Failed to set TDM BCLK rate %lu: %d\n",
+ bclk_rate, ret);
+ return ret;
+ }
+ } else {
+ ret = da7219_set_bclks_per_wclk(component, frame_size);
+ if (ret) {
+ dev_err(component->dev,
+ "Failed to set TDM BCLKs per WCLK %d: %d\n",
+ frame_size, ret);
+ return ret;
+ }
+ }
}
dai_offset = cpu_to_le16(offset);
return 0;
}
-static int da7219_hw_params(struct snd_pcm_substream *substream,
- struct snd_pcm_hw_params *params,
- struct snd_soc_dai *dai)
+static int da7219_set_sr(struct snd_soc_component *component,
+ unsigned long rate)
{
- struct snd_soc_component *component = dai->component;
- struct da7219_priv *da7219 = snd_soc_component_get_drvdata(component);
- u8 dai_ctrl = 0, dai_bclks_per_wclk = 0, fs;
- unsigned int channels;
- int word_len = params_width(params);
- int frame_size;
+ u8 fs;
- switch (word_len) {
- case 16:
- dai_ctrl |= DA7219_DAI_WORD_LENGTH_S16_LE;
- break;
- case 20:
- dai_ctrl |= DA7219_DAI_WORD_LENGTH_S20_LE;
- break;
- case 24:
- dai_ctrl |= DA7219_DAI_WORD_LENGTH_S24_LE;
- break;
- case 32:
- dai_ctrl |= DA7219_DAI_WORD_LENGTH_S32_LE;
- break;
- default:
- return -EINVAL;
- }
-
- channels = params_channels(params);
- if ((channels < 1) || (channels > DA7219_DAI_CH_NUM_MAX)) {
- dev_err(component->dev,
- "Invalid number of channels, only 1 to %d supported\n",
- DA7219_DAI_CH_NUM_MAX);
- return -EINVAL;
- }
- dai_ctrl |= channels << DA7219_DAI_CH_NUM_SHIFT;
-
- switch (params_rate(params)) {
+ switch (rate) {
case 8000:
fs = DA7219_SR_8000;
break;
return -EINVAL;
}
+ snd_soc_component_write(component, DA7219_SR, fs);
+
+ return 0;
+}
+
+static int da7219_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params,
+ struct snd_soc_dai *dai)
+{
+ struct snd_soc_component *component = dai->component;
+ struct da7219_priv *da7219 = snd_soc_component_get_drvdata(component);
+ struct clk *wclk = da7219->dai_clks[DA7219_DAI_WCLK_IDX];
+ struct clk *bclk = da7219->dai_clks[DA7219_DAI_BCLK_IDX];
+ u8 dai_ctrl = 0;
+ unsigned int channels;
+ unsigned long sr, bclk_rate;
+ int word_len = params_width(params);
+ int frame_size, ret;
+
+ switch (word_len) {
+ case 16:
+ dai_ctrl |= DA7219_DAI_WORD_LENGTH_S16_LE;
+ break;
+ case 20:
+ dai_ctrl |= DA7219_DAI_WORD_LENGTH_S20_LE;
+ break;
+ case 24:
+ dai_ctrl |= DA7219_DAI_WORD_LENGTH_S24_LE;
+ break;
+ case 32:
+ dai_ctrl |= DA7219_DAI_WORD_LENGTH_S32_LE;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ channels = params_channels(params);
+ if ((channels < 1) || (channels > DA7219_DAI_CH_NUM_MAX)) {
+ dev_err(component->dev,
+ "Invalid number of channels, only 1 to %d supported\n",
+ DA7219_DAI_CH_NUM_MAX);
+ return -EINVAL;
+ }
+ dai_ctrl |= channels << DA7219_DAI_CH_NUM_SHIFT;
+
+ sr = params_rate(params);
+ if (da7219->master && wclk) {
+ ret = clk_set_rate(wclk, sr);
+ if (ret) {
+ dev_err(component->dev,
+ "Failed to set WCLK SR %lu: %d\n", sr, ret);
+ return ret;
+ }
+ } else {
+ ret = da7219_set_sr(component, sr);
+ if (ret) {
+ dev_err(component->dev,
+ "Failed to set SR %lu: %d\n", sr, ret);
+ return ret;
+ }
+ }
+
/*
* If we're master, then we have a limited set of BCLK rates we
* support. For slave mode this isn't the case and the codec can detect
* the BCLK rate automatically.
*/
if (da7219->master && !da7219->tdm_en) {
- frame_size = word_len * 2;
- if (frame_size <= 32)
- dai_bclks_per_wclk = DA7219_DAI_BCLKS_PER_WCLK_32;
+ if ((word_len * DA7219_DAI_CH_NUM_MAX) <= 32)
+ frame_size = 32;
else
- dai_bclks_per_wclk = DA7219_DAI_BCLKS_PER_WCLK_64;
+ frame_size = 64;
+
+ if (bclk) {
+ bclk_rate = frame_size * sr;
+ /*
+ * Rounding the rate here avoids failure trying to set a
+ * new rate on an already enabled bclk. In that
+ * instance this will just set the same rate as is
+ * currently in use, and so should continue without
+ * problem, as long as the BCLK rate is suitable for the
+ * desired frame size.
+ */
+ bclk_rate = clk_round_rate(bclk, bclk_rate);
+ if ((bclk_rate / sr) < frame_size) {
+ dev_err(component->dev,
+ "BCLK rate mismatch against frame size");
+ return -EINVAL;
+ }
- snd_soc_component_update_bits(component, DA7219_DAI_CLK_MODE,
- DA7219_DAI_BCLKS_PER_WCLK_MASK,
- dai_bclks_per_wclk);
+ ret = clk_set_rate(bclk, bclk_rate);
+ if (ret) {
+ dev_err(component->dev,
+ "Failed to set BCLK rate %lu: %d\n",
+ bclk_rate, ret);
+ return ret;
+ }
+ } else {
+ ret = da7219_set_bclks_per_wclk(component, frame_size);
+ if (ret) {
+ dev_err(component->dev,
+ "Failed to set BCLKs per WCLK %d: %d\n",
+ frame_size, ret);
+ return ret;
+ }
+ }
}
snd_soc_component_update_bits(component, DA7219_DAI_CTRL,
DA7219_DAI_WORD_LENGTH_MASK |
DA7219_DAI_CH_NUM_MASK,
dai_ctrl);
- snd_soc_component_write(component, DA7219_SR, fs);
return 0;
}
#define DA7219_FORMATS (SNDRV_PCM_FMTBIT_S16_LE | SNDRV_PCM_FMTBIT_S20_3LE |\
SNDRV_PCM_FMTBIT_S24_LE | SNDRV_PCM_FMTBIT_S32_LE)
+#define DA7219_RATES (SNDRV_PCM_RATE_8000 | SNDRV_PCM_RATE_11025 |\
+ SNDRV_PCM_RATE_16000 | SNDRV_PCM_RATE_22050 |\
+ SNDRV_PCM_RATE_32000 | SNDRV_PCM_RATE_44100 |\
+ SNDRV_PCM_RATE_48000 | SNDRV_PCM_RATE_88200 |\
+ SNDRV_PCM_RATE_96000)
+
static struct snd_soc_dai_driver da7219_dai = {
.name = "da7219-hifi",
.playback = {
.stream_name = "Playback",
.channels_min = 1,
.channels_max = DA7219_DAI_CH_NUM_MAX,
- .rates = SNDRV_PCM_RATE_8000_96000,
+ .rates = DA7219_RATES,
.formats = DA7219_FORMATS,
},
.capture = {
.stream_name = "Capture",
.channels_min = 1,
.channels_max = DA7219_DAI_CH_NUM_MAX,
- .rates = SNDRV_PCM_RATE_8000_96000,
+ .rates = DA7219_RATES,
.formats = DA7219_FORMATS,
},
.ops = &da7219_dai_ops,
pdata->wakeup_source = device_property_read_bool(dev, "wakeup-source");
- pdata->dai_clks_name = "da7219-dai-clks";
- if (device_property_read_string(dev, "clock-output-names",
- &pdata->dai_clks_name))
- dev_warn(dev, "Using default clk name: %s\n",
- pdata->dai_clks_name);
+ pdata->dai_clk_names[DA7219_DAI_WCLK_IDX] = "da7219-dai-wclk";
+ pdata->dai_clk_names[DA7219_DAI_BCLK_IDX] = "da7219-dai-bclk";
+ if (device_property_read_string_array(dev, "clock-output-names",
+ pdata->dai_clk_names,
+ DA7219_DAI_NUM_CLKS) < 0)
+ dev_warn(dev, "Using default DAI clk names: %s, %s\n",
+ pdata->dai_clk_names[DA7219_DAI_WCLK_IDX],
+ pdata->dai_clk_names[DA7219_DAI_BCLK_IDX]);
if (device_property_read_u32(dev, "dlg,micbias-lvl", &of_val32) >= 0)
pdata->micbias_lvl = da7219_fw_micbias_lvl(dev, of_val32);
}
#ifdef CONFIG_COMMON_CLK
-static int da7219_dai_clks_prepare(struct clk_hw *hw)
+static int da7219_wclk_prepare(struct clk_hw *hw)
{
struct da7219_priv *da7219 =
- container_of(hw, struct da7219_priv, dai_clks_hw);
+ container_of(hw, struct da7219_priv,
+ dai_clks_hw[DA7219_DAI_WCLK_IDX]);
struct snd_soc_component *component = da7219->component;
+ if (!da7219->master)
+ return -EINVAL;
+
snd_soc_component_update_bits(component, DA7219_DAI_CLK_MODE,
DA7219_DAI_CLK_EN_MASK,
DA7219_DAI_CLK_EN_MASK);
return 0;
}
-static void da7219_dai_clks_unprepare(struct clk_hw *hw)
+static void da7219_wclk_unprepare(struct clk_hw *hw)
{
struct da7219_priv *da7219 =
- container_of(hw, struct da7219_priv, dai_clks_hw);
+ container_of(hw, struct da7219_priv,
+ dai_clks_hw[DA7219_DAI_WCLK_IDX]);
struct snd_soc_component *component = da7219->component;
+ if (!da7219->master)
+ return;
+
snd_soc_component_update_bits(component, DA7219_DAI_CLK_MODE,
DA7219_DAI_CLK_EN_MASK, 0);
}
-static int da7219_dai_clks_is_prepared(struct clk_hw *hw)
+static int da7219_wclk_is_prepared(struct clk_hw *hw)
{
struct da7219_priv *da7219 =
- container_of(hw, struct da7219_priv, dai_clks_hw);
+ container_of(hw, struct da7219_priv,
+ dai_clks_hw[DA7219_DAI_WCLK_IDX]);
struct snd_soc_component *component = da7219->component;
u8 clk_reg;
+ if (!da7219->master)
+ return -EINVAL;
+
clk_reg = snd_soc_component_read32(component, DA7219_DAI_CLK_MODE);
return !!(clk_reg & DA7219_DAI_CLK_EN_MASK);
}
-static unsigned long da7219_dai_clks_recalc_rate(struct clk_hw *hw,
- unsigned long parent_rate)
+static unsigned long da7219_wclk_recalc_rate(struct clk_hw *hw,
+ unsigned long parent_rate)
{
struct da7219_priv *da7219 =
- container_of(hw, struct da7219_priv, dai_clks_hw);
+ container_of(hw, struct da7219_priv,
+ dai_clks_hw[DA7219_DAI_WCLK_IDX]);
struct snd_soc_component *component = da7219->component;
u8 fs = snd_soc_component_read32(component, DA7219_SR);
}
}
-static const struct clk_ops da7219_dai_clks_ops = {
- .prepare = da7219_dai_clks_prepare,
- .unprepare = da7219_dai_clks_unprepare,
- .is_prepared = da7219_dai_clks_is_prepared,
- .recalc_rate = da7219_dai_clks_recalc_rate,
+static long da7219_wclk_round_rate(struct clk_hw *hw, unsigned long rate,
+ unsigned long *parent_rate)
+{
+ struct da7219_priv *da7219 =
+ container_of(hw, struct da7219_priv,
+ dai_clks_hw[DA7219_DAI_WCLK_IDX]);
+
+ if (!da7219->master)
+ return -EINVAL;
+
+ if (rate < 11025)
+ return 8000;
+ else if (rate < 12000)
+ return 11025;
+ else if (rate < 16000)
+ return 12000;
+ else if (rate < 22050)
+ return 16000;
+ else if (rate < 24000)
+ return 22050;
+ else if (rate < 32000)
+ return 24000;
+ else if (rate < 44100)
+ return 32000;
+ else if (rate < 48000)
+ return 44100;
+ else if (rate < 88200)
+ return 48000;
+ else if (rate < 96000)
+ return 88200;
+ else
+ return 96000;
+}
+
+static int da7219_wclk_set_rate(struct clk_hw *hw, unsigned long rate,
+ unsigned long parent_rate)
+{
+ struct da7219_priv *da7219 =
+ container_of(hw, struct da7219_priv,
+ dai_clks_hw[DA7219_DAI_WCLK_IDX]);
+ struct snd_soc_component *component = da7219->component;
+
+ if (!da7219->master)
+ return -EINVAL;
+
+ return da7219_set_sr(component, rate);
+}
+
+static unsigned long da7219_bclk_recalc_rate(struct clk_hw *hw,
+ unsigned long parent_rate)
+{
+ struct da7219_priv *da7219 =
+ container_of(hw, struct da7219_priv,
+ dai_clks_hw[DA7219_DAI_BCLK_IDX]);
+ struct snd_soc_component *component = da7219->component;
+ u8 bclks_per_wclk = snd_soc_component_read32(component,
+ DA7219_DAI_CLK_MODE);
+
+ switch (bclks_per_wclk & DA7219_DAI_BCLKS_PER_WCLK_MASK) {
+ case DA7219_DAI_BCLKS_PER_WCLK_32:
+ return parent_rate * 32;
+ case DA7219_DAI_BCLKS_PER_WCLK_64:
+ return parent_rate * 64;
+ case DA7219_DAI_BCLKS_PER_WCLK_128:
+ return parent_rate * 128;
+ case DA7219_DAI_BCLKS_PER_WCLK_256:
+ return parent_rate * 256;
+ default:
+ return 0;
+ }
+}
+
+static unsigned long da7219_bclk_get_factor(unsigned long rate,
+ unsigned long parent_rate)
+{
+ unsigned long factor;
+
+ factor = rate / parent_rate;
+ if (factor < 64)
+ return 32;
+ else if (factor < 128)
+ return 64;
+ else if (factor < 256)
+ return 128;
+ else
+ return 256;
+}
+
+static long da7219_bclk_round_rate(struct clk_hw *hw, unsigned long rate,
+ unsigned long *parent_rate)
+{
+ struct da7219_priv *da7219 =
+ container_of(hw, struct da7219_priv,
+ dai_clks_hw[DA7219_DAI_BCLK_IDX]);
+ unsigned long factor;
+
+ if (!*parent_rate || !da7219->master)
+ return -EINVAL;
+
+ /*
+ * We don't allow changing the parent rate as some BCLK rates can be
+ * derived from multiple parent WCLK rates (BCLK rates are set as a
+ * multiplier of WCLK in HW). We just do some rounding down based on the
+ * parent WCLK rate set and find the appropriate multiplier of BCLK to
+ * get the rounded down BCLK value.
+ */
+ factor = da7219_bclk_get_factor(rate, *parent_rate);
+
+ return *parent_rate * factor;
+}
+
+static int da7219_bclk_set_rate(struct clk_hw *hw, unsigned long rate,
+ unsigned long parent_rate)
+{
+ struct da7219_priv *da7219 =
+ container_of(hw, struct da7219_priv,
+ dai_clks_hw[DA7219_DAI_BCLK_IDX]);
+ struct snd_soc_component *component = da7219->component;
+ unsigned long factor;
+
+ if (!da7219->master)
+ return -EINVAL;
+
+ factor = da7219_bclk_get_factor(rate, parent_rate);
+
+ return da7219_set_bclks_per_wclk(component, factor);
+}
+
+static const struct clk_ops da7219_dai_clk_ops[DA7219_DAI_NUM_CLKS] = {
+ [DA7219_DAI_WCLK_IDX] = {
+ .prepare = da7219_wclk_prepare,
+ .unprepare = da7219_wclk_unprepare,
+ .is_prepared = da7219_wclk_is_prepared,
+ .recalc_rate = da7219_wclk_recalc_rate,
+ .round_rate = da7219_wclk_round_rate,
+ .set_rate = da7219_wclk_set_rate,
+ },
+ [DA7219_DAI_BCLK_IDX] = {
+ .recalc_rate = da7219_bclk_recalc_rate,
+ .round_rate = da7219_bclk_round_rate,
+ .set_rate = da7219_bclk_set_rate,
+ },
};
static int da7219_register_dai_clks(struct snd_soc_component *component)
struct device *dev = component->dev;
struct da7219_priv *da7219 = snd_soc_component_get_drvdata(component);
struct da7219_pdata *pdata = da7219->pdata;
- struct clk_init_data init = {};
- struct clk *dai_clks;
- struct clk_lookup *dai_clks_lookup;
const char *parent_name;
+ int i, ret;
- if (da7219->mclk) {
- parent_name = __clk_get_name(da7219->mclk);
- init.parent_names = &parent_name;
- init.num_parents = 1;
- } else {
- init.parent_names = NULL;
- init.num_parents = 0;
- }
+ for (i = 0; i < DA7219_DAI_NUM_CLKS; ++i) {
+ struct clk_init_data init = {};
+ struct clk *dai_clk;
+ struct clk_lookup *dai_clk_lookup;
+ struct clk_hw *dai_clk_hw = &da7219->dai_clks_hw[i];
- init.name = pdata->dai_clks_name;
- init.ops = &da7219_dai_clks_ops;
- init.flags = CLK_GET_RATE_NOCACHE;
- da7219->dai_clks_hw.init = &init;
+ switch (i) {
+ case DA7219_DAI_WCLK_IDX:
+ /*
+ * If we can, make MCLK the parent of WCLK to ensure
+ * it's enabled as required.
+ */
+ if (da7219->mclk) {
+ parent_name = __clk_get_name(da7219->mclk);
+ init.parent_names = &parent_name;
+ init.num_parents = 1;
+ } else {
+ init.parent_names = NULL;
+ init.num_parents = 0;
+ }
+ break;
+ case DA7219_DAI_BCLK_IDX:
+ /* Make WCLK the parent of BCLK */
+ parent_name = __clk_get_name(da7219->dai_clks[DA7219_DAI_WCLK_IDX]);
+ init.parent_names = &parent_name;
+ init.num_parents = 1;
+ break;
+ default:
+ dev_err(dev, "Invalid clock index\n");
+ ret = -EINVAL;
+ goto err;
+ }
- dai_clks = devm_clk_register(dev, &da7219->dai_clks_hw);
- if (IS_ERR(dai_clks)) {
- dev_warn(dev, "Failed to register DAI clocks: %ld\n",
- PTR_ERR(dai_clks));
- return PTR_ERR(dai_clks);
- }
- da7219->dai_clks = dai_clks;
+ init.name = pdata->dai_clk_names[i];
+ init.ops = &da7219_dai_clk_ops[i];
+ init.flags = CLK_GET_RATE_NOCACHE | CLK_SET_RATE_GATE;
+ dai_clk_hw->init = &init;
+
+ dai_clk = devm_clk_register(dev, dai_clk_hw);
+ if (IS_ERR(dai_clk)) {
+ dev_warn(dev, "Failed to register %s: %ld\n",
+ init.name, PTR_ERR(dai_clk));
+ ret = PTR_ERR(dai_clk);
+ goto err;
+ }
+ da7219->dai_clks[i] = dai_clk;
- /* If we're using DT, then register as provider accordingly */
- if (dev->of_node) {
- devm_of_clk_add_hw_provider(dev, of_clk_hw_simple_get,
- &da7219->dai_clks_hw);
- } else {
- dai_clks_lookup = clkdev_create(dai_clks, pdata->dai_clks_name,
- "%s", dev_name(dev));
- if (!dai_clks_lookup)
- return -ENOMEM;
- else
- da7219->dai_clks_lookup = dai_clks_lookup;
+ /* If we're using DT, then register as provider accordingly */
+ if (dev->of_node) {
+ devm_of_clk_add_hw_provider(dev, of_clk_hw_simple_get,
+ dai_clk_hw);
+ } else {
+ dai_clk_lookup = clkdev_create(dai_clk, init.name,
+ "%s", dev_name(dev));
+ if (!dai_clk_lookup) {
+ ret = -ENOMEM;
+ goto err;
+ } else {
+ da7219->dai_clks_lookup[i] = dai_clk_lookup;
+ }
+ }
}
return 0;
+
+err:
+ do {
+ if (da7219->dai_clks_lookup[i])
+ clkdev_drop(da7219->dai_clks_lookup[i]);
+ } while (i-- > 0);
+
+ return ret;
}
#else
static inline int da7219_register_dai_clks(struct snd_soc_component *component)
static void da7219_remove(struct snd_soc_component *component)
{
struct da7219_priv *da7219 = snd_soc_component_get_drvdata(component);
+#ifdef CONFIG_COMMON_CLK
+ int i;
+#endif
da7219_aad_exit(component);
#ifdef CONFIG_COMMON_CLK
- if (da7219->dai_clks_lookup)
- clkdev_drop(da7219->dai_clks_lookup);
+ for (i = DA7219_DAI_NUM_CLKS - 1; i >= 0; --i) {
+ if (da7219->dai_clks_lookup[i])
+ clkdev_drop(da7219->dai_clks_lookup[i]);
+ }
#endif
/* Supplies */
struct mutex pll_lock;
#ifdef CONFIG_COMMON_CLK
- struct clk_hw dai_clks_hw;
+ struct clk_hw dai_clks_hw[DA7219_DAI_NUM_CLKS];
#endif
- struct clk_lookup *dai_clks_lookup;
- struct clk *dai_clks;
+ struct clk_lookup *dai_clks_lookup[DA7219_DAI_NUM_CLKS];
+ struct clk *dai_clks[DA7219_DAI_NUM_CLKS];
struct clk *mclk;
unsigned int mclk_rate;
unsigned int sysclk;
unsigned int allowed_rates[NR_SUPPORTED_MCLK_LRCK_RATIOS];
struct snd_pcm_hw_constraint_list sysclk_constraints;
+ bool jd_inverted;
};
/*
if (!es8316->jack)
goto out;
+ if (es8316->jd_inverted)
+ flags ^= ES8316_GPIO_FLAG_HP_NOT_INSERTED;
+
dev_dbg(comp->dev, "gpio flags %#04x\n", flags);
if (flags & ES8316_GPIO_FLAG_HP_NOT_INSERTED) {
/* Jack removed, or spurious IRQ? */
/* Jack inserted, determine type */
es8316_enable_micbias_for_mic_gnd_short_detect(comp);
regmap_read(es8316->regmap, ES8316_GPIO_FLAG, &flags);
+ if (es8316->jd_inverted)
+ flags ^= ES8316_GPIO_FLAG_HP_NOT_INSERTED;
dev_dbg(comp->dev, "gpio flags %#04x\n", flags);
if (flags & ES8316_GPIO_FLAG_HP_NOT_INSERTED) {
/* Jack unplugged underneath us */
{
struct es8316_priv *es8316 = snd_soc_component_get_drvdata(component);
+ /*
+ * Init es8316->jd_inverted here and not in the probe, as we cannot
+ * guarantee that the bytchr-es8316 driver, which might set this
+ * property, will probe before us.
+ */
+ es8316->jd_inverted = device_property_read_bool(component->dev,
+ "everest,jack-detect-inverted");
+
mutex_lock(&es8316->lock);
es8316->jack = jack;
dev_err(&hdev->dev, "failed to create hda codec %d\n", ret);
goto error_no_pm;
}
+ /*
+ * Overwrite type to HDA_DEV_ASOC since it is a ASoC driver
+ * hda_codec.c will check this flag to determine if unregister
+ * device is needed.
+ */
+ hdev->type = HDA_DEV_ASOC;
/*
* snd_hda_codec_device_new decrements the usage count so call get pm
struct snd_pcm_hw_params *hparams, struct snd_soc_dai *dai)
{
struct hdac_hdmi_priv *hdmi = snd_soc_dai_get_drvdata(dai);
- struct hdac_device *hdev = hdmi->hdev;
struct hdac_hdmi_dai_port_map *dai_map;
- struct hdac_hdmi_port *port;
struct hdac_hdmi_pcm *pcm;
int format;
dai_map = &hdmi->dai_map[dai->id];
- port = dai_map->port;
-
- if (!port)
- return -ENODEV;
-
- if ((!port->eld.monitor_present) || (!port->eld.eld_valid)) {
- dev_err(&hdev->dev,
- "device is not configured for this pin:port%d:%d\n",
- port->pin->nid, port->id);
- return -ENODEV;
- }
format = snd_hdac_calc_stream_format(params_rate(hparams),
params_channels(hparams), params_format(hparams),
/* Imp: Store the card pointer in hda_codec */
hdmi->card = dapm->card->snd_card;
+ /*
+ * Setup a device_link between card device and HDMI codec device.
+ * The card device is the consumer and the HDMI codec device is
+ * the supplier. With this setting, we can make sure that the audio
+ * domain in display power will be always turned on before operating
+ * on the HDMI audio codec registers.
+ * Let's use the flag DL_FLAG_AUTOREMOVE_CONSUMER. This can make
+ * sure the device link is freed when the machine driver is removed.
+ */
+ device_link_add(component->card->dev, &hdev->dev, DL_FLAG_RPM_ACTIVE |
+ DL_FLAG_AUTOREMOVE_CONSUMER);
/*
* hdac_device core already sets the state to active and calls
* get_noresume. So enable runtime and set the device to suspend.
if (!ret) {
ret = snd_pcm_hw_constraint_eld(substream->runtime,
hcp->eld);
- if (ret)
+ if (ret) {
+ mutex_lock(&hcp->current_stream_lock);
+ hcp->current_stream = NULL;
+ mutex_unlock(&hcp->current_stream_lock);
return ret;
+ }
}
/* Select chmap supported */
hdmi_codec_eld_chmap(hcp);
return ret;
}
- ret = hdmi_codec_new_stream(substream, dai);
- if (ret)
- return ret;
-
hdmi_audio_infoframe_init(&hp.cea);
hp.cea.channels = params_channels(params);
hp.cea.coding_type = HDMI_AUDIO_CODING_TYPE_STREAM;
dev_dbg(dev, "%s()\n", __func__);
if (!hcd) {
- dev_err(dev, "%s: No plalform data\n", __func__);
+ dev_err(dev, "%s: No platform data\n", __func__);
return -EINVAL;
}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// Lochnagar sound card driver
+//
+// Copyright (c) 2017-2019 Cirrus Logic, Inc. and
+// Cirrus Logic International Semiconductor Ltd.
+//
+// Author: Charles Keepax <ckeepax@opensource.cirrus.com>
+// Piotr Stankiewicz <piotrs@opensource.cirrus.com>
+
+#include <linux/clk.h>
+#include <linux/module.h>
+#include <sound/soc.h>
+
+#include <linux/mfd/lochnagar.h>
+#include <linux/mfd/lochnagar1_regs.h>
+#include <linux/mfd/lochnagar2_regs.h>
+
+struct lochnagar_sc_priv {
+ struct clk *mclk;
+};
+
+static const struct snd_soc_dapm_widget lochnagar_sc_widgets[] = {
+ SND_SOC_DAPM_LINE("Line Jack", NULL),
+ SND_SOC_DAPM_LINE("USB Audio", NULL),
+};
+
+static const struct snd_soc_dapm_route lochnagar_sc_routes[] = {
+ { "Line Jack", NULL, "AIF1 Playback" },
+ { "AIF1 Capture", NULL, "Line Jack" },
+
+ { "USB Audio", NULL, "USB1 Playback" },
+ { "USB Audio", NULL, "USB2 Playback" },
+ { "USB1 Capture", NULL, "USB Audio" },
+ { "USB2 Capture", NULL, "USB Audio" },
+};
+
+static const unsigned int lochnagar_sc_chan_vals[] = {
+ 4, 8,
+};
+
+static const struct snd_pcm_hw_constraint_list lochnagar_sc_chan_constraint = {
+ .count = ARRAY_SIZE(lochnagar_sc_chan_vals),
+ .list = lochnagar_sc_chan_vals,
+};
+
+static const unsigned int lochnagar_sc_rate_vals[] = {
+ 8000, 16000, 24000, 32000, 48000, 96000, 192000,
+ 22050, 44100, 88200, 176400,
+};
+
+static const struct snd_pcm_hw_constraint_list lochnagar_sc_rate_constraint = {
+ .count = ARRAY_SIZE(lochnagar_sc_rate_vals),
+ .list = lochnagar_sc_rate_vals,
+};
+
+static int lochnagar_sc_hw_rule_rate(struct snd_pcm_hw_params *params,
+ struct snd_pcm_hw_rule *rule)
+{
+ struct snd_interval range = {
+ .min = 8000,
+ .max = 24576000 / hw_param_interval(params, rule->deps[0])->max,
+ };
+
+ return snd_interval_refine(hw_param_interval(params, rule->var),
+ &range);
+}
+
+static int lochnagar_sc_startup(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct snd_soc_component *comp = dai->component;
+ struct lochnagar_sc_priv *priv = snd_soc_component_get_drvdata(comp);
+ int ret;
+
+ ret = snd_pcm_hw_constraint_list(substream->runtime, 0,
+ SNDRV_PCM_HW_PARAM_RATE,
+ &lochnagar_sc_rate_constraint);
+ if (ret)
+ return ret;
+
+ return snd_pcm_hw_rule_add(substream->runtime, 0,
+ SNDRV_PCM_HW_PARAM_RATE,
+ lochnagar_sc_hw_rule_rate, priv,
+ SNDRV_PCM_HW_PARAM_FRAME_BITS, -1);
+}
+
+static int lochnagar_sc_line_startup(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct snd_soc_component *comp = dai->component;
+ struct lochnagar_sc_priv *priv = snd_soc_component_get_drvdata(comp);
+ int ret;
+
+ ret = clk_prepare_enable(priv->mclk);
+ if (ret < 0) {
+ dev_err(dai->dev, "Failed to enable MCLK: %d\n", ret);
+ return ret;
+ }
+
+ ret = lochnagar_sc_startup(substream, dai);
+ if (ret)
+ return ret;
+
+ return snd_pcm_hw_constraint_list(substream->runtime, 0,
+ SNDRV_PCM_HW_PARAM_CHANNELS,
+ &lochnagar_sc_chan_constraint);
+}
+
+static void lochnagar_sc_line_shutdown(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct snd_soc_component *comp = dai->component;
+ struct lochnagar_sc_priv *priv = snd_soc_component_get_drvdata(comp);
+
+ clk_disable_unprepare(priv->mclk);
+}
+
+static int lochnagar_sc_check_fmt(struct snd_soc_dai *dai, unsigned int fmt,
+ unsigned int tar)
+{
+ tar |= SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF;
+
+ if ((fmt & ~SND_SOC_DAIFMT_CLOCK_MASK) != tar)
+ return -EINVAL;
+
+ return 0;
+}
+
+static int lochnagar_sc_set_line_fmt(struct snd_soc_dai *dai, unsigned int fmt)
+{
+ return lochnagar_sc_check_fmt(dai, fmt, SND_SOC_DAIFMT_CBS_CFS);
+}
+
+static int lochnagar_sc_set_usb_fmt(struct snd_soc_dai *dai, unsigned int fmt)
+{
+ return lochnagar_sc_check_fmt(dai, fmt, SND_SOC_DAIFMT_CBM_CFM);
+}
+
+static const struct snd_soc_dai_ops lochnagar_sc_line_ops = {
+ .startup = lochnagar_sc_line_startup,
+ .shutdown = lochnagar_sc_line_shutdown,
+ .set_fmt = lochnagar_sc_set_line_fmt,
+};
+
+static const struct snd_soc_dai_ops lochnagar_sc_usb_ops = {
+ .startup = lochnagar_sc_startup,
+ .set_fmt = lochnagar_sc_set_usb_fmt,
+};
+
+static struct snd_soc_dai_driver lochnagar_sc_dai[] = {
+ {
+ .name = "lochnagar-line",
+ .playback = {
+ .stream_name = "AIF1 Playback",
+ .channels_min = 4,
+ .channels_max = 8,
+ .rates = SNDRV_PCM_RATE_KNOT,
+ .formats = SNDRV_PCM_FMTBIT_S32_LE,
+ },
+ .capture = {
+ .stream_name = "AIF1 Capture",
+ .channels_min = 4,
+ .channels_max = 8,
+ .rates = SNDRV_PCM_RATE_KNOT,
+ .formats = SNDRV_PCM_FMTBIT_S32_LE,
+ },
+ .ops = &lochnagar_sc_line_ops,
+ .symmetric_rates = true,
+ .symmetric_samplebits = true,
+ },
+ {
+ .name = "lochnagar-usb1",
+ .playback = {
+ .stream_name = "USB1 Playback",
+ .channels_min = 1,
+ .channels_max = 8,
+ .rates = SNDRV_PCM_RATE_KNOT,
+ .formats = SNDRV_PCM_FMTBIT_S32_LE,
+ },
+ .capture = {
+ .stream_name = "USB1 Capture",
+ .channels_min = 1,
+ .channels_max = 8,
+ .rates = SNDRV_PCM_RATE_KNOT,
+ .formats = SNDRV_PCM_FMTBIT_S32_LE,
+ },
+ .ops = &lochnagar_sc_usb_ops,
+ .symmetric_rates = true,
+ .symmetric_samplebits = true,
+ },
+ {
+ .name = "lochnagar-usb2",
+ .playback = {
+ .stream_name = "USB2 Playback",
+ .channels_min = 1,
+ .channels_max = 8,
+ .rates = SNDRV_PCM_RATE_KNOT,
+ .formats = SNDRV_PCM_FMTBIT_S32_LE,
+ },
+ .capture = {
+ .stream_name = "USB2 Capture",
+ .channels_min = 1,
+ .channels_max = 8,
+ .rates = SNDRV_PCM_RATE_KNOT,
+ .formats = SNDRV_PCM_FMTBIT_S32_LE,
+ },
+ .ops = &lochnagar_sc_usb_ops,
+ .symmetric_rates = true,
+ .symmetric_samplebits = true,
+ },
+};
+
+static const struct snd_soc_component_driver lochnagar_sc_driver = {
+ .non_legacy_dai_naming = 1,
+
+ .dapm_widgets = lochnagar_sc_widgets,
+ .num_dapm_widgets = ARRAY_SIZE(lochnagar_sc_widgets),
+ .dapm_routes = lochnagar_sc_routes,
+ .num_dapm_routes = ARRAY_SIZE(lochnagar_sc_routes),
+};
+
+static int lochnagar_sc_probe(struct platform_device *pdev)
+{
+ struct lochnagar_sc_priv *priv;
+ int ret;
+
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->mclk = devm_clk_get(&pdev->dev, "mclk");
+ if (IS_ERR(priv->mclk)) {
+ ret = PTR_ERR(priv->mclk);
+ dev_err(&pdev->dev, "Failed to get MCLK: %d\n", ret);
+ return ret;
+ }
+
+ platform_set_drvdata(pdev, priv);
+
+ return devm_snd_soc_register_component(&pdev->dev,
+ &lochnagar_sc_driver,
+ lochnagar_sc_dai,
+ ARRAY_SIZE(lochnagar_sc_dai));
+}
+
+static const struct of_device_id lochnagar_of_match[] = {
+ { .compatible = "cirrus,lochnagar2-soundcard" },
+ {}
+};
+MODULE_DEVICE_TABLE(of, lochnagar_of_match);
+
+static struct platform_driver lochnagar_sc_codec_driver = {
+ .driver = {
+ .name = "lochnagar-soundcard",
+ .of_match_table = of_match_ptr(lochnagar_of_match),
+ },
+
+ .probe = lochnagar_sc_probe,
+};
+module_platform_driver(lochnagar_sc_codec_driver);
+
+MODULE_DESCRIPTION("ASoC Lochnagar Sound Card Driver");
+MODULE_AUTHOR("Piotr Stankiewicz <piotrs@opensource.cirrus.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:lochnagar-soundcard");
&max98090_right_rcv_mixer_controls[0],
ARRAY_SIZE(max98090_right_rcv_mixer_controls)),
- SND_SOC_DAPM_MUX("LINMOD Mux", M98090_REG_LOUTR_MIXER,
- M98090_LINMOD_SHIFT, 0, &max98090_linmod_mux),
+ SND_SOC_DAPM_MUX("LINMOD Mux", SND_SOC_NOPM, 0, 0,
+ &max98090_linmod_mux),
- SND_SOC_DAPM_MUX("MIXHPLSEL Mux", M98090_REG_HP_CONTROL,
- M98090_MIXHPLSEL_SHIFT, 0, &max98090_mixhplsel_mux),
+ SND_SOC_DAPM_MUX("MIXHPLSEL Mux", SND_SOC_NOPM, 0, 0,
+ &max98090_mixhplsel_mux),
- SND_SOC_DAPM_MUX("MIXHPRSEL Mux", M98090_REG_HP_CONTROL,
- M98090_MIXHPRSEL_SHIFT, 0, &max98090_mixhprsel_mux),
+ SND_SOC_DAPM_MUX("MIXHPRSEL Mux", SND_SOC_NOPM, 0, 0,
+ &max98090_mixhprsel_mux),
SND_SOC_DAPM_PGA("HP Left Out", M98090_REG_OUTPUT_ENABLE,
M98090_HPLEN_SHIFT, 0, NULL, 0),
SNDRV_PCM_FMTBIT_S32,
.rates = SNDRV_PCM_RATE_8000 |
SNDRV_PCM_RATE_16000 |
+ SNDRV_PCM_RATE_32000 |
+ SNDRV_PCM_RATE_44100 |
SNDRV_PCM_RATE_48000 |
+ SNDRV_PCM_RATE_88200 |
SNDRV_PCM_RATE_96000,
.rate_min = 8000,
.rate_max = 96000,
#define DL_GAIN_N_40DB_REG (DL_GAIN_N_40DB << 7 | DL_GAIN_N_40DB)
#define DL_GAIN_REG_MASK 0x0f9f
-static void lo_store_gain(struct mt6358_priv *priv)
-{
- unsigned int reg;
- unsigned int gain_l, gain_r;
-
- regmap_read(priv->regmap, MT6358_ZCD_CON1, ®);
- gain_l = (reg >> RG_AUDLOLGAIN_SFT) & RG_AUDLOLGAIN_MASK;
- gain_r = (reg >> RG_AUDLORGAIN_SFT) & RG_AUDLORGAIN_MASK;
-
- priv->ana_gain[AUDIO_ANALOG_VOLUME_LINEOUTL] = gain_l;
- priv->ana_gain[AUDIO_ANALOG_VOLUME_LINEOUTR] = gain_r;
-}
-
-static void hp_store_gain(struct mt6358_priv *priv)
-{
- unsigned int reg;
- unsigned int gain_l, gain_r;
-
- regmap_read(priv->regmap, MT6358_ZCD_CON2, ®);
- gain_l = (reg >> RG_AUDHPLGAIN_SFT) & RG_AUDHPLGAIN_MASK;
- gain_r = (reg >> RG_AUDHPRGAIN_SFT) & RG_AUDHPRGAIN_MASK;
-
- priv->ana_gain[AUDIO_ANALOG_VOLUME_HPOUTL] = gain_l;
- priv->ana_gain[AUDIO_ANALOG_VOLUME_HPOUTR] = gain_r;
-}
-
static void hp_zcd_disable(struct mt6358_priv *priv)
{
regmap_write(priv->regmap, MT6358_ZCD_CON0, 0x0000);
reg_idx == DL_GAIN_N_40DB;
}
-static void headset_volume_ramp(struct mt6358_priv *priv,
- int from, int to)
+static void headset_volume_ramp(struct mt6358_priv *priv, int from, int to)
{
- int offset = 0, count = 1, reg_idx;
+ int offset = 0, count = 0, reg_idx;
if (!is_valid_hp_pga_idx(from) || !is_valid_hp_pga_idx(to))
dev_warn(priv->dev, "%s(), volume index is not valid, from %d, to %d\n",
else
offset = from - to;
- while (offset > 0) {
+ while (offset >= 0) {
if (to > from)
reg_idx = from + count;
else
}
}
+static int mt6358_put_volsw(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct snd_soc_component *component =
+ snd_soc_kcontrol_component(kcontrol);
+ struct mt6358_priv *priv = snd_soc_component_get_drvdata(component);
+ struct soc_mixer_control *mc =
+ (struct soc_mixer_control *)kcontrol->private_value;
+ unsigned int reg;
+ int ret;
+
+ ret = snd_soc_put_volsw(kcontrol, ucontrol);
+ if (ret < 0)
+ return ret;
+
+ switch (mc->reg) {
+ case MT6358_ZCD_CON2:
+ regmap_read(priv->regmap, MT6358_ZCD_CON2, ®);
+ priv->ana_gain[AUDIO_ANALOG_VOLUME_HPOUTL] =
+ (reg >> RG_AUDHPLGAIN_SFT) & RG_AUDHPLGAIN_MASK;
+ priv->ana_gain[AUDIO_ANALOG_VOLUME_HPOUTR] =
+ (reg >> RG_AUDHPRGAIN_SFT) & RG_AUDHPRGAIN_MASK;
+ break;
+ case MT6358_ZCD_CON1:
+ regmap_read(priv->regmap, MT6358_ZCD_CON1, ®);
+ priv->ana_gain[AUDIO_ANALOG_VOLUME_LINEOUTL] =
+ (reg >> RG_AUDLOLGAIN_SFT) & RG_AUDLOLGAIN_MASK;
+ priv->ana_gain[AUDIO_ANALOG_VOLUME_LINEOUTR] =
+ (reg >> RG_AUDLORGAIN_SFT) & RG_AUDLORGAIN_MASK;
+ break;
+ case MT6358_ZCD_CON3:
+ regmap_read(priv->regmap, MT6358_ZCD_CON3, ®);
+ priv->ana_gain[AUDIO_ANALOG_VOLUME_HSOUTL] =
+ (reg >> RG_AUDHSGAIN_SFT) & RG_AUDHSGAIN_MASK;
+ priv->ana_gain[AUDIO_ANALOG_VOLUME_HSOUTR] =
+ (reg >> RG_AUDHSGAIN_SFT) & RG_AUDHSGAIN_MASK;
+ break;
+ case MT6358_AUDENC_ANA_CON0:
+ case MT6358_AUDENC_ANA_CON1:
+ regmap_read(priv->regmap, MT6358_AUDENC_ANA_CON0, ®);
+ priv->ana_gain[AUDIO_ANALOG_VOLUME_MICAMP1] =
+ (reg >> RG_AUDPREAMPLGAIN_SFT) & RG_AUDPREAMPLGAIN_MASK;
+ regmap_read(priv->regmap, MT6358_AUDENC_ANA_CON1, ®);
+ priv->ana_gain[AUDIO_ANALOG_VOLUME_MICAMP2] =
+ (reg >> RG_AUDPREAMPRGAIN_SFT) & RG_AUDPREAMPRGAIN_MASK;
+ break;
+ }
+
+ return ret;
+}
+
static const DECLARE_TLV_DB_SCALE(playback_tlv, -1000, 100, 0);
static const DECLARE_TLV_DB_SCALE(pga_tlv, 0, 600, 0);
static const struct snd_kcontrol_new mt6358_snd_controls[] = {
/* dl pga gain */
- SOC_DOUBLE_TLV("Headphone Volume",
- MT6358_ZCD_CON2, 0, 7, 0x12, 1,
- playback_tlv),
- SOC_DOUBLE_TLV("Lineout Volume",
- MT6358_ZCD_CON1, 0, 7, 0x12, 1,
- playback_tlv),
- SOC_SINGLE_TLV("Handset Volume",
- MT6358_ZCD_CON3, 0, 0x12, 1,
- playback_tlv),
+ SOC_DOUBLE_EXT_TLV("Headphone Volume",
+ MT6358_ZCD_CON2, 0, 7, 0x12, 1,
+ snd_soc_get_volsw, mt6358_put_volsw, playback_tlv),
+ SOC_DOUBLE_EXT_TLV("Lineout Volume",
+ MT6358_ZCD_CON1, 0, 7, 0x12, 1,
+ snd_soc_get_volsw, mt6358_put_volsw, playback_tlv),
+ SOC_SINGLE_EXT_TLV("Handset Volume",
+ MT6358_ZCD_CON3, 0, 0x12, 1,
+ snd_soc_get_volsw, mt6358_put_volsw, playback_tlv),
/* ul pga gain */
- SOC_DOUBLE_R_TLV("PGA Volume",
- MT6358_AUDENC_ANA_CON0, MT6358_AUDENC_ANA_CON1,
- 8, 4, 0,
- pga_tlv),
+ SOC_DOUBLE_R_EXT_TLV("PGA Volume",
+ MT6358_AUDENC_ANA_CON0, MT6358_AUDENC_ANA_CON1,
+ 8, 4, 0,
+ snd_soc_get_volsw, mt6358_put_volsw, pga_tlv),
};
/* MUX */
/* Reduce ESD resistance of AU_REFN */
regmap_write(priv->regmap, MT6358_AUDDEC_ANA_CON2, 0x4000);
- /* save target gain to restore after hardware open complete */
- hp_store_gain(priv);
/* Set HPR/HPL gain as minimum (~ -40dB) */
regmap_write(priv->regmap, MT6358_ZCD_CON2, DL_GAIN_N_40DB_REG);
/* Reduce ESD resistance of AU_REFN */
regmap_write(priv->regmap, MT6358_AUDDEC_ANA_CON2, 0x4000);
- /* save target gain to restore after hardware open complete */
- hp_store_gain(priv);
/* Set HPR/HPL gain to -10dB */
regmap_write(priv->regmap, MT6358_ZCD_CON2, DL_GAIN_N_10DB_REG);
hp_main_output_ramp(priv, true);
/* Set LO gain as minimum (~ -40dB) */
- lo_store_gain(priv);
regmap_write(priv->regmap, MT6358_ZCD_CON1, DL_GAIN_N_40DB_REG);
/* apply volume setting */
headset_volume_ramp(priv,
regmap_write(priv->regmap, MT6358_AUDENC_ANA_CON9, 0x0000);
}
+static void mt6358_restore_pga(struct mt6358_priv *priv)
+{
+ unsigned int gain_l, gain_r;
+
+ gain_l = priv->ana_gain[AUDIO_ANALOG_VOLUME_MICAMP1];
+ gain_r = priv->ana_gain[AUDIO_ANALOG_VOLUME_MICAMP2];
+
+ regmap_update_bits(priv->regmap, MT6358_AUDENC_ANA_CON0,
+ RG_AUDPREAMPLGAIN_MASK_SFT,
+ gain_l << RG_AUDPREAMPLGAIN_SFT);
+ regmap_update_bits(priv->regmap, MT6358_AUDENC_ANA_CON1,
+ RG_AUDPREAMPRGAIN_MASK_SFT,
+ gain_r << RG_AUDPREAMPRGAIN_SFT);
+}
+
static int mt_mic_type_event(struct snd_soc_dapm_widget *w,
struct snd_kcontrol *kcontrol,
int event)
mt6358_amic_enable(priv);
break;
}
+ mt6358_restore_pga(priv);
break;
case SND_SOC_DAPM_POST_PMD:
return 0;
}
-static int nau88l0_calc_pll(unsigned int pll_in,
+static int nau8810_calc_pll(unsigned int pll_in,
unsigned int fs, struct nau8810_pll *pll_param)
{
u64 f2, f2_max, pll_ratio;
f2_max = 0;
scal_sel = ARRAY_SIZE(nau8810_mclk_scaler);
for (i = 0; i < ARRAY_SIZE(nau8810_mclk_scaler); i++) {
- f2 = 256 * fs * 4 * nau8810_mclk_scaler[i] / 10;
+ f2 = 256ULL * fs * 4 * nau8810_mclk_scaler[i];
+ f2 = div_u64(f2, 10);
if (f2 > NAU_PLL_FREQ_MIN && f2 < NAU_PLL_FREQ_MAX &&
f2_max < f2) {
f2_max = f2;
int ret, fs;
fs = freq_out / 256;
- ret = nau88l0_calc_pll(freq_in, fs, pll_param);
+ ret = nau8810_calc_pll(freq_in, fs, pll_param);
if (ret < 0) {
dev_err(nau8810->dev, "Unsupported input clock %d\n", freq_in);
return ret;
struct snd_soc_component *component = dai->component;
struct nau8810 *nau8810 = snd_soc_component_get_drvdata(component);
int val_len = 0, val_rate = 0, ret = 0;
+ unsigned int ctrl_val, bclk_fs, bclk_div;
+
+ /* Select BCLK configuration if the codec as master. */
+ regmap_read(nau8810->regmap, NAU8810_REG_CLOCK, &ctrl_val);
+ if (ctrl_val & NAU8810_CLKIO_MASTER) {
+ /* get the bclk and fs ratio */
+ bclk_fs = snd_soc_params_to_bclk(params) / params_rate(params);
+ if (bclk_fs <= 32)
+ bclk_div = NAU8810_BCLKDIV_8;
+ else if (bclk_fs <= 64)
+ bclk_div = NAU8810_BCLKDIV_4;
+ else if (bclk_fs <= 128)
+ bclk_div = NAU8810_BCLKDIV_2;
+ else
+ return -EINVAL;
+ regmap_update_bits(nau8810->regmap, NAU8810_REG_CLOCK,
+ NAU8810_BCLKSEL_MASK, bclk_div);
+ }
switch (params_width(params)) {
case 16:
if (chan > 2) {
switch (fmt) {
case PCM3168A_FMT_I2S:
+ case PCM3168A_FMT_DSP_A:
fmt = PCM3168A_FMT_I2S_TDM;
break;
case PCM3168A_FMT_LEFT_J:
+ case PCM3168A_FMT_DSP_B:
fmt = PCM3168A_FMT_LEFT_J_TDM;
break;
default:
- dev_err(component->dev, "TDM is supported under I2S/Left_J only\n");
+ dev_err(component->dev,
+ "TDM is supported under DSP/I2S/Left_J only\n");
return -EINVAL;
}
}
break;
case PCM3168A_FMT_LEFT_J:
case PCM3168A_FMT_I2S:
+ case PCM3168A_FMT_DSP_A:
+ case PCM3168A_FMT_DSP_B:
sample_min = 24;
channel_max = channel_maxs[tx];
break;
RT5645_HWEQ_NUM, sizeof(struct rt5645_eq_param_s),
GFP_KERNEL);
+ if (!rt5645->eq_param)
+ return -ENOMEM;
+
return 0;
}
.jd_mode = 3,
};
+static const struct rt5645_platform_data lattepanda_board_platform_data = {
+ .jd_mode = 2,
+ .inv_jd1_1 = true
+};
+
static const struct dmi_system_id dmi_platform_data[] = {
{
.ident = "Chrome Buddy",
},
.driver_data = (void *)&intel_braswell_platform_data,
},
+ {
+ .ident = "LattePanda board",
+ .matches = {
+ DMI_EXACT_MATCH(DMI_BOARD_VENDOR, "AMI Corporation"),
+ DMI_EXACT_MATCH(DMI_BOARD_NAME, "Cherry Trail CR"),
+ DMI_EXACT_MATCH(DMI_BOARD_VERSION, "Default string"),
+ },
+ .driver_data = (void *)&lattepanda_board_platform_data,
+ },
{ }
};
break;
}
- return val == 0;
+ if (rt5651->jd_active_high)
+ return val != 0;
+ else
+ return val == 0;
}
/* Jack detect and button-press timings */
case RT5651_JD1_1:
snd_soc_component_update_bits(component, RT5651_JD_CTRL2,
RT5651_JD_TRG_SEL_MASK, RT5651_JD_TRG_SEL_JD1_1);
- snd_soc_component_update_bits(component, RT5651_IRQ_CTRL1,
- RT5651_JD1_1_IRQ_EN, RT5651_JD1_1_IRQ_EN);
+ /* active-low is normal, set inv flag for active-high */
+ if (rt5651->jd_active_high)
+ snd_soc_component_update_bits(component,
+ RT5651_IRQ_CTRL1,
+ RT5651_JD1_1_IRQ_EN | RT5651_JD1_1_INV,
+ RT5651_JD1_1_IRQ_EN | RT5651_JD1_1_INV);
+ else
+ snd_soc_component_update_bits(component,
+ RT5651_IRQ_CTRL1,
+ RT5651_JD1_1_IRQ_EN | RT5651_JD1_1_INV,
+ RT5651_JD1_1_IRQ_EN);
break;
case RT5651_JD1_2:
snd_soc_component_update_bits(component, RT5651_JD_CTRL2,
RT5651_JD_TRG_SEL_MASK, RT5651_JD_TRG_SEL_JD1_2);
- snd_soc_component_update_bits(component, RT5651_IRQ_CTRL1,
- RT5651_JD1_2_IRQ_EN, RT5651_JD1_2_IRQ_EN);
+ /* active-low is normal, set inv flag for active-high */
+ if (rt5651->jd_active_high)
+ snd_soc_component_update_bits(component,
+ RT5651_IRQ_CTRL1,
+ RT5651_JD1_2_IRQ_EN | RT5651_JD1_2_INV,
+ RT5651_JD1_2_IRQ_EN | RT5651_JD1_2_INV);
+ else
+ snd_soc_component_update_bits(component,
+ RT5651_IRQ_CTRL1,
+ RT5651_JD1_2_IRQ_EN | RT5651_JD1_2_INV,
+ RT5651_JD1_2_IRQ_EN);
break;
case RT5651_JD2:
snd_soc_component_update_bits(component, RT5651_JD_CTRL2,
RT5651_JD_TRG_SEL_MASK, RT5651_JD_TRG_SEL_JD2);
- snd_soc_component_update_bits(component, RT5651_IRQ_CTRL1,
- RT5651_JD2_IRQ_EN, RT5651_JD2_IRQ_EN);
+ /* active-low is normal, set inv flag for active-high */
+ if (rt5651->jd_active_high)
+ snd_soc_component_update_bits(component,
+ RT5651_IRQ_CTRL1,
+ RT5651_JD2_IRQ_EN | RT5651_JD2_INV,
+ RT5651_JD2_IRQ_EN | RT5651_JD2_INV);
+ else
+ snd_soc_component_update_bits(component,
+ RT5651_IRQ_CTRL1,
+ RT5651_JD2_IRQ_EN | RT5651_JD2_INV,
+ RT5651_JD2_IRQ_EN);
break;
default:
dev_err(component->dev, "Currently only JD1_1 / JD1_2 / JD2 are supported\n");
"realtek,jack-detect-source", &val) == 0)
rt5651->jd_src = val;
+ if (device_property_read_bool(component->dev, "realtek,jack-detect-not-inverted"))
+ rt5651->jd_active_high = true;
+
/*
* Testing on various boards has shown that good defaults for the OVCD
* threshold and scale-factor are 2000µA and 0.75. For an effective
int release_count;
int poll_count;
unsigned int jd_src;
+ bool jd_active_high;
unsigned int ovcd_th;
unsigned int ovcd_sf;
#include <linux/sysfs.h>
#include <linux/clk.h>
#include <linux/firmware.h>
+#include <linux/acpi.h>
#include "rt5677-spi.h"
* RT5677_SPI_READ/WRITE_32: Transfer 4 bytes
* RT5677_SPI_READ/WRITE_BURST: Transfer any multiples of 8 bytes
*
- * For example, reading 260 bytes at 0x60030002 uses the following commands:
- * 0x60030002 RT5677_SPI_READ_16 2 bytes
+ * Note:
+ * 16 Bit writes and reads are restricted to the address range
+ * 0x18020000 ~ 0x18021000
+ *
+ * For example, reading 256 bytes at 0x60030004 uses the following commands:
* 0x60030004 RT5677_SPI_READ_32 4 bytes
* 0x60030008 RT5677_SPI_READ_BURST 240 bytes
* 0x600300F8 RT5677_SPI_READ_BURST 8 bytes
* 0x60030100 RT5677_SPI_READ_32 4 bytes
- * 0x60030104 RT5677_SPI_READ_16 2 bytes
*
* Input:
* @read: true for read commands; false for write commands
{
u8 cmd;
- if (align == 2 || align == 6 || remain == 2) {
- cmd = RT5677_SPI_READ_16;
- *len = 2;
- } else if (align == 4 || remain <= 6) {
+ if (align == 4 || remain <= 4) {
cmd = RT5677_SPI_READ_32;
*len = 4;
} else {
cmd = RT5677_SPI_READ_BURST;
- *len = min_t(u32, remain & ~7, RT5677_SPI_BURST_LEN);
+ *len = (((remain - 1) >> 3) + 1) << 3;
+ *len = min_t(u32, *len, RT5677_SPI_BURST_LEN);
}
return read ? cmd : cmd + 1;
}
}
}
-/* Read DSP address space using SPI. addr and len have to be 2-byte aligned. */
+/* Read DSP address space using SPI. addr and len have to be 4-byte aligned. */
int rt5677_spi_read(u32 addr, void *rxbuf, size_t len)
{
u32 offset;
if (!g_spi)
return -ENODEV;
- if ((addr & 1) || (len & 1)) {
+ if ((addr & 3) || (len & 3)) {
dev_err(&g_spi->dev, "Bad read align 0x%x(%zu)\n", addr, len);
return -EACCES;
}
}
EXPORT_SYMBOL_GPL(rt5677_spi_read);
-/* Write DSP address space using SPI. addr has to be 2-byte aligned.
- * If len is not 2-byte aligned, an extra byte of zero is written at the end
+/* Write DSP address space using SPI. addr has to be 4-byte aligned.
+ * If len is not 4-byte aligned, then extra zeros are written at the end
* as padding.
*/
int rt5677_spi_write(u32 addr, const void *txbuf, size_t len)
{
- u32 offset, len_with_pad = len;
+ u32 offset;
int status = 0;
struct spi_transfer t;
struct spi_message m;
if (!g_spi)
return -ENODEV;
- if (addr & 1) {
+ if (addr & 3) {
dev_err(&g_spi->dev, "Bad write align 0x%x(%zu)\n", addr, len);
return -EACCES;
}
- if (len & 1)
- len_with_pad = len + 1;
-
memset(&t, 0, sizeof(t));
t.tx_buf = buf;
t.speed_hz = RT5677_SPI_FREQ;
spi_message_init_with_transfers(&m, &t, 1);
- for (offset = 0; offset < len_with_pad;) {
+ for (offset = 0; offset < len;) {
spi_cmd = rt5677_spi_select_cmd(false, (addr + offset) & 7,
- len_with_pad - offset, &t.len);
+ len - offset, &t.len);
/* Construct SPI message header */
buf[0] = spi_cmd;
return 0;
}
+static const struct acpi_device_id rt5677_spi_acpi_id[] = {
+ { "RT5677AA", 0 },
+ { }
+};
+MODULE_DEVICE_TABLE(acpi, rt5677_spi_acpi_id);
+
static struct spi_driver rt5677_spi_driver = {
.driver = {
.name = "rt5677",
+ .acpi_match_table = ACPI_PTR(rt5677_spi_acpi_id),
},
.probe = rt5677_spi_probe,
};
rt5682_reset(rt5682->regmap);
+ mutex_init(&rt5682->calibrate_mutex);
rt5682_calibrate(rt5682);
ret = regmap_multi_reg_write(rt5682->regmap, patch_list,
INIT_DELAYED_WORK(&rt5682->jd_check_work,
rt5682_jd_check_handler);
- mutex_init(&rt5682->calibrate_mutex);
if (i2c->irq) {
ret = devm_request_threaded_irq(&i2c->dev, i2c->irq, NULL,
return -ENOMEM;
platform_set_drvdata(pdev, priv);
- priv->gpiod_enable = devm_gpiod_get(dev, "enable", GPIOD_OUT_LOW);
+ priv->gpiod_enable = devm_gpiod_get_optional(dev, "enable",
+ GPIOD_OUT_LOW);
if (IS_ERR(priv->gpiod_enable)) {
err = PTR_ERR(priv->gpiod_enable);
if (err != -EPROBE_DEFER)
struct sirf_audio_codec *sirf_audio_codec;
void __iomem *base;
struct resource *mem_res;
- const struct of_device_id *match;
-
- match = of_match_node(sirf_audio_codec_of_match, pdev->dev.of_node);
sirf_audio_codec = devm_kzalloc(&pdev->dev,
sizeof(struct sirf_audio_codec), GFP_KERNEL);
#include <linux/of_gpio.h>
#include <linux/slab.h>
#include <sound/core.h>
+#include <sound/jack.h>
#include <sound/pcm.h>
#include <sound/pcm_params.h>
#include <sound/soc.h>
case AIC31XX_INTRADCFLAG: /* Sticky interrupt flags */
case AIC31XX_INTRDACFLAG2:
case AIC31XX_INTRADCFLAG2:
+ case AIC31XX_HSDETECT:
return true;
}
return false;
struct aic31xx_pdata pdata;
struct regulator_bulk_data supplies[AIC31XX_NUM_SUPPLIES];
struct aic31xx_disable_nb disable_nb[AIC31XX_NUM_SUPPLIES];
+ struct snd_soc_jack *jack;
unsigned int sysclk;
u8 p_div;
int rate_div_line;
return 0;
}
+static int aic31xx_set_jack(struct snd_soc_component *component,
+ struct snd_soc_jack *jack, void *data)
+{
+ struct aic31xx_priv *aic31xx = snd_soc_component_get_drvdata(component);
+
+ aic31xx->jack = jack;
+
+ /* Enable/Disable jack detection */
+ regmap_write(aic31xx->regmap, AIC31XX_HSDETECT,
+ jack ? AIC31XX_HSD_ENABLE : 0);
+
+ return 0;
+}
+
static int aic31xx_codec_probe(struct snd_soc_component *component)
{
struct aic31xx_priv *aic31xx = snd_soc_component_get_drvdata(component);
static const struct snd_soc_component_driver soc_codec_driver_aic31xx = {
.probe = aic31xx_codec_probe,
+ .set_jack = aic31xx_set_jack,
.set_bias_level = aic31xx_set_bias_level,
.controls = common31xx_snd_controls,
.num_controls = ARRAY_SIZE(common31xx_snd_controls),
dev_err(dev, "Short circuit on Left output is detected\n");
if (value & AIC31XX_HPRSCDETECT)
dev_err(dev, "Short circuit on Right output is detected\n");
+ if (value & (AIC31XX_HSPLUG | AIC31XX_BUTTONPRESS)) {
+ unsigned int val;
+ int status = 0;
+
+ ret = regmap_read(aic31xx->regmap, AIC31XX_INTRDACFLAG2,
+ &val);
+ if (ret) {
+ dev_err(dev, "Failed to read interrupt mask: %d\n",
+ ret);
+ goto exit;
+ }
+
+ if (val & AIC31XX_BUTTONPRESS)
+ status |= SND_JACK_BTN_0;
+
+ ret = regmap_read(aic31xx->regmap, AIC31XX_HSDETECT, &val);
+ if (ret) {
+ dev_err(dev, "Failed to read headset type: %d\n", ret);
+ goto exit;
+ }
+
+ switch ((val & AIC31XX_HSD_TYPE_MASK) >>
+ AIC31XX_HSD_TYPE_SHIFT) {
+ case AIC31XX_HSD_HP:
+ status |= SND_JACK_HEADPHONE;
+ break;
+ case AIC31XX_HSD_HS:
+ status |= SND_JACK_HEADSET;
+ break;
+ default:
+ break;
+ }
+
+ if (aic31xx->jack)
+ snd_soc_jack_report(aic31xx->jack, status,
+ AIC31XX_JACK_MASK);
+ }
if (value & ~(AIC31XX_HPLSCDETECT |
- AIC31XX_HPRSCDETECT))
+ AIC31XX_HPRSCDETECT |
+ AIC31XX_HSPLUG |
+ AIC31XX_BUTTONPRESS))
dev_err(dev, "Unknown DAC interrupt flags: 0x%08x\n", value);
read_overflow:
AIC31XX_GPIO1_FUNC_SHIFT);
regmap_write(aic31xx->regmap, AIC31XX_INT1CTRL,
+ AIC31XX_HSPLUGDET |
+ AIC31XX_BUTTONPRESSDET |
AIC31XX_SC |
AIC31XX_ENGINE);
#define AIC31XX_MINIDSP_BIT BIT(2)
#define DAC31XX_BIT BIT(3)
+#define AIC31XX_JACK_MASK (SND_JACK_HEADPHONE | \
+ SND_JACK_HEADSET | \
+ SND_JACK_BTN_0)
+
enum aic31xx_type {
AIC3100 = 0,
AIC3110 = AIC31XX_STEREO_CLASS_D_BIT,
/* AIC31XX_DACMUTE */
#define AIC31XX_DACMUTE_MASK GENMASK(3, 2)
+/* AIC31XX_HSDETECT */
+#define AIC31XX_HSD_ENABLE BIT(7)
+#define AIC31XX_HSD_TYPE_MASK GENMASK(6, 5)
+#define AIC31XX_HSD_TYPE_SHIFT 5
+#define AIC31XX_HSD_NONE 0x00
+#define AIC31XX_HSD_HP 0x01
+#define AIC31XX_HSD_HS 0x03
+
/* AIC31XX_MICBIAS */
#define AIC31XX_MICBIAS_MASK GENMASK(1, 0)
#define AIC31XX_MICBIAS_SHIFT 0
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * Clock Tree for the Texas Instruments TLV320AIC32x4
+ *
+ * Copyright 2019 Annaliese McDermond
+ *
+ * Author: Annaliese McDermond <nh6z@nh6z.net>
+ */
+
+#include <linux/clk-provider.h>
+#include <linux/clkdev.h>
+#include <linux/regmap.h>
+#include <linux/device.h>
+
+#include "tlv320aic32x4.h"
+
+#define to_clk_aic32x4(_hw) container_of(_hw, struct clk_aic32x4, hw)
+struct clk_aic32x4 {
+ struct clk_hw hw;
+ struct device *dev;
+ struct regmap *regmap;
+ unsigned int reg;
+};
+
+/*
+ * struct clk_aic32x4_pll_muldiv - Multiplier/divider settings
+ * @p: Divider
+ * @r: first multiplier
+ * @j: integer part of second multiplier
+ * @d: decimal part of second multiplier
+ */
+struct clk_aic32x4_pll_muldiv {
+ u8 p;
+ u16 r;
+ u8 j;
+ u16 d;
+};
+
+struct aic32x4_clkdesc {
+ const char *name;
+ const char * const *parent_names;
+ unsigned int num_parents;
+ const struct clk_ops *ops;
+ unsigned int reg;
+};
+
+static int clk_aic32x4_pll_prepare(struct clk_hw *hw)
+{
+ struct clk_aic32x4 *pll = to_clk_aic32x4(hw);
+
+ return regmap_update_bits(pll->regmap, AIC32X4_PLLPR,
+ AIC32X4_PLLEN, AIC32X4_PLLEN);
+}
+
+static void clk_aic32x4_pll_unprepare(struct clk_hw *hw)
+{
+ struct clk_aic32x4 *pll = to_clk_aic32x4(hw);
+
+ regmap_update_bits(pll->regmap, AIC32X4_PLLPR,
+ AIC32X4_PLLEN, 0);
+}
+
+static int clk_aic32x4_pll_is_prepared(struct clk_hw *hw)
+{
+ struct clk_aic32x4 *pll = to_clk_aic32x4(hw);
+
+ unsigned int val;
+ int ret;
+
+ ret = regmap_read(pll->regmap, AIC32X4_PLLPR, &val);
+ if (ret < 0)
+ return ret;
+
+ return !!(val & AIC32X4_PLLEN);
+}
+
+static int clk_aic32x4_pll_get_muldiv(struct clk_aic32x4 *pll,
+ struct clk_aic32x4_pll_muldiv *settings)
+{
+ /* Change to use regmap_bulk_read? */
+ unsigned int val;
+ int ret;
+
+ ret = regmap_read(pll->regmap, AIC32X4_PLLPR, &val);
+ if (ret < 0)
+ return ret;
+ settings->r = val & AIC32X4_PLL_R_MASK;
+ settings->p = (val & AIC32X4_PLL_P_MASK) >> AIC32X4_PLL_P_SHIFT;
+
+ ret = regmap_read(pll->regmap, AIC32X4_PLLJ, &val);
+ if (ret < 0)
+ return ret;
+ settings->j = val;
+
+ ret = regmap_read(pll->regmap, AIC32X4_PLLDMSB, &val);
+ if (ret < 0)
+ return ret;
+ settings->d = val << 8;
+
+ ret = regmap_read(pll->regmap, AIC32X4_PLLDLSB, &val);
+ if (ret < 0)
+ return ret;
+ settings->d |= val;
+
+ return 0;
+}
+
+static int clk_aic32x4_pll_set_muldiv(struct clk_aic32x4 *pll,
+ struct clk_aic32x4_pll_muldiv *settings)
+{
+ int ret;
+ /* Change to use regmap_bulk_write for some if not all? */
+
+ ret = regmap_update_bits(pll->regmap, AIC32X4_PLLPR,
+ AIC32X4_PLL_R_MASK, settings->r);
+ if (ret < 0)
+ return ret;
+
+ ret = regmap_update_bits(pll->regmap, AIC32X4_PLLPR,
+ AIC32X4_PLL_P_MASK,
+ settings->p << AIC32X4_PLL_P_SHIFT);
+ if (ret < 0)
+ return ret;
+
+ ret = regmap_write(pll->regmap, AIC32X4_PLLJ, settings->j);
+ if (ret < 0)
+ return ret;
+
+ ret = regmap_write(pll->regmap, AIC32X4_PLLDMSB, (settings->d >> 8));
+ if (ret < 0)
+ return ret;
+ ret = regmap_write(pll->regmap, AIC32X4_PLLDLSB, (settings->d & 0xff));
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static unsigned long clk_aic32x4_pll_calc_rate(
+ struct clk_aic32x4_pll_muldiv *settings,
+ unsigned long parent_rate)
+{
+ u64 rate;
+ /*
+ * We scale j by 10000 to account for the decimal part of P and divide
+ * it back out later.
+ */
+ rate = (u64) parent_rate * settings->r *
+ ((settings->j * 10000) + settings->d);
+
+ return (unsigned long) DIV_ROUND_UP_ULL(rate, settings->p * 10000);
+}
+
+static int clk_aic32x4_pll_calc_muldiv(struct clk_aic32x4_pll_muldiv *settings,
+ unsigned long rate, unsigned long parent_rate)
+{
+ u64 multiplier;
+
+ settings->p = parent_rate / AIC32X4_MAX_PLL_CLKIN + 1;
+ if (settings->p > 8)
+ return -1;
+
+ /*
+ * We scale this figure by 10000 so that we can get the decimal part
+ * of the multiplier. This is because we can't do floating point
+ * math in the kernel.
+ */
+ multiplier = (u64) rate * settings->p * 10000;
+ do_div(multiplier, parent_rate);
+
+ /*
+ * J can't be over 64, so R can scale this.
+ * R can't be greater than 4.
+ */
+ settings->r = ((u32) multiplier / 640000) + 1;
+ if (settings->r > 4)
+ return -1;
+ do_div(multiplier, settings->r);
+
+ /*
+ * J can't be < 1.
+ */
+ if (multiplier < 10000)
+ return -1;
+
+ /* Figure out the integer part, J, and the fractional part, D. */
+ settings->j = (u32) multiplier / 10000;
+ settings->d = (u32) multiplier % 10000;
+
+ return 0;
+}
+
+static unsigned long clk_aic32x4_pll_recalc_rate(struct clk_hw *hw,
+ unsigned long parent_rate)
+{
+ struct clk_aic32x4 *pll = to_clk_aic32x4(hw);
+ struct clk_aic32x4_pll_muldiv settings;
+ int ret;
+
+ ret = clk_aic32x4_pll_get_muldiv(pll, &settings);
+ if (ret < 0)
+ return 0;
+
+ return clk_aic32x4_pll_calc_rate(&settings, parent_rate);
+}
+
+static long clk_aic32x4_pll_round_rate(struct clk_hw *hw,
+ unsigned long rate,
+ unsigned long *parent_rate)
+{
+ struct clk_aic32x4_pll_muldiv settings;
+ int ret;
+
+ ret = clk_aic32x4_pll_calc_muldiv(&settings, rate, *parent_rate);
+ if (ret < 0)
+ return 0;
+
+ return clk_aic32x4_pll_calc_rate(&settings, *parent_rate);
+}
+
+static int clk_aic32x4_pll_set_rate(struct clk_hw *hw,
+ unsigned long rate,
+ unsigned long parent_rate)
+{
+ struct clk_aic32x4 *pll = to_clk_aic32x4(hw);
+ struct clk_aic32x4_pll_muldiv settings;
+ int ret;
+
+ ret = clk_aic32x4_pll_calc_muldiv(&settings, rate, parent_rate);
+ if (ret < 0)
+ return -EINVAL;
+
+ return clk_aic32x4_pll_set_muldiv(pll, &settings);
+}
+
+static int clk_aic32x4_pll_set_parent(struct clk_hw *hw, u8 index)
+{
+ struct clk_aic32x4 *pll = to_clk_aic32x4(hw);
+
+ return regmap_update_bits(pll->regmap,
+ AIC32X4_CLKMUX,
+ AIC32X4_PLL_CLKIN_MASK,
+ index << AIC32X4_PLL_CLKIN_SHIFT);
+}
+
+static u8 clk_aic32x4_pll_get_parent(struct clk_hw *hw)
+{
+ struct clk_aic32x4 *pll = to_clk_aic32x4(hw);
+ unsigned int val;
+
+ regmap_read(pll->regmap, AIC32X4_PLLPR, &val);
+
+ return (val & AIC32X4_PLL_CLKIN_MASK) >> AIC32X4_PLL_CLKIN_SHIFT;
+}
+
+
+static const struct clk_ops aic32x4_pll_ops = {
+ .prepare = clk_aic32x4_pll_prepare,
+ .unprepare = clk_aic32x4_pll_unprepare,
+ .is_prepared = clk_aic32x4_pll_is_prepared,
+ .recalc_rate = clk_aic32x4_pll_recalc_rate,
+ .round_rate = clk_aic32x4_pll_round_rate,
+ .set_rate = clk_aic32x4_pll_set_rate,
+ .set_parent = clk_aic32x4_pll_set_parent,
+ .get_parent = clk_aic32x4_pll_get_parent,
+};
+
+static int clk_aic32x4_codec_clkin_set_parent(struct clk_hw *hw, u8 index)
+{
+ struct clk_aic32x4 *mux = to_clk_aic32x4(hw);
+
+ return regmap_update_bits(mux->regmap,
+ AIC32X4_CLKMUX,
+ AIC32X4_CODEC_CLKIN_MASK, index << AIC32X4_CODEC_CLKIN_SHIFT);
+}
+
+static u8 clk_aic32x4_codec_clkin_get_parent(struct clk_hw *hw)
+{
+ struct clk_aic32x4 *mux = to_clk_aic32x4(hw);
+ unsigned int val;
+
+ regmap_read(mux->regmap, AIC32X4_CLKMUX, &val);
+
+ return (val & AIC32X4_CODEC_CLKIN_MASK) >> AIC32X4_CODEC_CLKIN_SHIFT;
+}
+
+static const struct clk_ops aic32x4_codec_clkin_ops = {
+ .set_parent = clk_aic32x4_codec_clkin_set_parent,
+ .get_parent = clk_aic32x4_codec_clkin_get_parent,
+};
+
+static int clk_aic32x4_div_prepare(struct clk_hw *hw)
+{
+ struct clk_aic32x4 *div = to_clk_aic32x4(hw);
+
+ return regmap_update_bits(div->regmap, div->reg,
+ AIC32X4_DIVEN, AIC32X4_DIVEN);
+}
+
+static void clk_aic32x4_div_unprepare(struct clk_hw *hw)
+{
+ struct clk_aic32x4 *div = to_clk_aic32x4(hw);
+
+ regmap_update_bits(div->regmap, div->reg,
+ AIC32X4_DIVEN, 0);
+}
+
+static int clk_aic32x4_div_set_rate(struct clk_hw *hw, unsigned long rate,
+ unsigned long parent_rate)
+{
+ struct clk_aic32x4 *div = to_clk_aic32x4(hw);
+ u8 divisor;
+
+ divisor = DIV_ROUND_UP(parent_rate, rate);
+ if (divisor > 128)
+ return -EINVAL;
+
+ return regmap_update_bits(div->regmap, div->reg,
+ AIC32X4_DIV_MASK, divisor);
+}
+
+static long clk_aic32x4_div_round_rate(struct clk_hw *hw, unsigned long rate,
+ unsigned long *parent_rate)
+{
+ unsigned long divisor;
+
+ divisor = DIV_ROUND_UP(*parent_rate, rate);
+ if (divisor > 128)
+ return -EINVAL;
+
+ return DIV_ROUND_UP(*parent_rate, divisor);
+}
+
+static unsigned long clk_aic32x4_div_recalc_rate(struct clk_hw *hw,
+ unsigned long parent_rate)
+{
+ struct clk_aic32x4 *div = to_clk_aic32x4(hw);
+
+ unsigned int val;
+
+ regmap_read(div->regmap, div->reg, &val);
+
+ return DIV_ROUND_UP(parent_rate, val & AIC32X4_DIV_MASK);
+}
+
+static const struct clk_ops aic32x4_div_ops = {
+ .prepare = clk_aic32x4_div_prepare,
+ .unprepare = clk_aic32x4_div_unprepare,
+ .set_rate = clk_aic32x4_div_set_rate,
+ .round_rate = clk_aic32x4_div_round_rate,
+ .recalc_rate = clk_aic32x4_div_recalc_rate,
+};
+
+static int clk_aic32x4_bdiv_set_parent(struct clk_hw *hw, u8 index)
+{
+ struct clk_aic32x4 *mux = to_clk_aic32x4(hw);
+
+ return regmap_update_bits(mux->regmap, AIC32X4_IFACE3,
+ AIC32X4_BDIVCLK_MASK, index);
+}
+
+static u8 clk_aic32x4_bdiv_get_parent(struct clk_hw *hw)
+{
+ struct clk_aic32x4 *mux = to_clk_aic32x4(hw);
+ unsigned int val;
+
+ regmap_read(mux->regmap, AIC32X4_IFACE3, &val);
+
+ return val & AIC32X4_BDIVCLK_MASK;
+}
+
+static const struct clk_ops aic32x4_bdiv_ops = {
+ .prepare = clk_aic32x4_div_prepare,
+ .unprepare = clk_aic32x4_div_unprepare,
+ .set_parent = clk_aic32x4_bdiv_set_parent,
+ .get_parent = clk_aic32x4_bdiv_get_parent,
+ .set_rate = clk_aic32x4_div_set_rate,
+ .round_rate = clk_aic32x4_div_round_rate,
+ .recalc_rate = clk_aic32x4_div_recalc_rate,
+};
+
+static struct aic32x4_clkdesc aic32x4_clkdesc_array[] = {
+ {
+ .name = "pll",
+ .parent_names =
+ (const char* []) { "mclk", "bclk", "gpio", "din" },
+ .num_parents = 4,
+ .ops = &aic32x4_pll_ops,
+ .reg = 0,
+ },
+ {
+ .name = "codec_clkin",
+ .parent_names =
+ (const char *[]) { "mclk", "bclk", "gpio", "pll" },
+ .num_parents = 4,
+ .ops = &aic32x4_codec_clkin_ops,
+ .reg = 0,
+ },
+ {
+ .name = "ndac",
+ .parent_names = (const char * []) { "codec_clkin" },
+ .num_parents = 1,
+ .ops = &aic32x4_div_ops,
+ .reg = AIC32X4_NDAC,
+ },
+ {
+ .name = "mdac",
+ .parent_names = (const char * []) { "ndac" },
+ .num_parents = 1,
+ .ops = &aic32x4_div_ops,
+ .reg = AIC32X4_MDAC,
+ },
+ {
+ .name = "nadc",
+ .parent_names = (const char * []) { "codec_clkin" },
+ .num_parents = 1,
+ .ops = &aic32x4_div_ops,
+ .reg = AIC32X4_NADC,
+ },
+ {
+ .name = "madc",
+ .parent_names = (const char * []) { "nadc" },
+ .num_parents = 1,
+ .ops = &aic32x4_div_ops,
+ .reg = AIC32X4_MADC,
+ },
+ {
+ .name = "bdiv",
+ .parent_names =
+ (const char *[]) { "ndac", "mdac", "nadc", "madc" },
+ .num_parents = 4,
+ .ops = &aic32x4_bdiv_ops,
+ .reg = AIC32X4_BCLKN,
+ },
+};
+
+static struct clk *aic32x4_register_clk(struct device *dev,
+ struct aic32x4_clkdesc *desc)
+{
+ struct clk_init_data init;
+ struct clk_aic32x4 *priv;
+ const char *devname = dev_name(dev);
+
+ init.ops = desc->ops;
+ init.name = desc->name;
+ init.parent_names = desc->parent_names;
+ init.num_parents = desc->num_parents;
+ init.flags = 0;
+
+ priv = devm_kzalloc(dev, sizeof(struct clk_aic32x4), GFP_KERNEL);
+ if (priv == NULL)
+ return (struct clk *) -ENOMEM;
+
+ priv->dev = dev;
+ priv->hw.init = &init;
+ priv->regmap = dev_get_regmap(dev, NULL);
+ priv->reg = desc->reg;
+
+ clk_hw_register_clkdev(&priv->hw, desc->name, devname);
+ return devm_clk_register(dev, &priv->hw);
+}
+
+int aic32x4_register_clocks(struct device *dev, const char *mclk_name)
+{
+ int i;
+
+ /*
+ * These lines are here to preserve the current functionality of
+ * the driver with regard to the DT. These should eventually be set
+ * by DT nodes so that the connections can be set up in configuration
+ * rather than code.
+ */
+ aic32x4_clkdesc_array[0].parent_names =
+ (const char* []) { mclk_name, "bclk", "gpio", "din" };
+ aic32x4_clkdesc_array[1].parent_names =
+ (const char *[]) { mclk_name, "bclk", "gpio", "pll" };
+
+ for (i = 0; i < ARRAY_SIZE(aic32x4_clkdesc_array); ++i)
+ aic32x4_register_clk(dev, &aic32x4_clkdesc_array[i]);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(aic32x4_register_clocks);
-/*
- * linux/sound/soc/codecs/tlv320aic32x4-i2c.c
+/* SPDX-License-Identifier: GPL-2.0
*
- * Copyright 2011 NW Digital Radio
+ * Copyright 2011-2019 NW Digital Radio
*
* Author: Annaliese McDermond <nh6z@nh6z.net>
*
* Based on sound/soc/codecs/wm8974 and TI driver for kernel 2.6.27.
*
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
*/
#include <linux/i2c.h>
-/*
- * linux/sound/soc/codecs/tlv320aic32x4-spi.c
+/* SPDX-License-Identifier: GPL-2.0
*
- * Copyright 2011 NW Digital Radio
+ * Copyright 2011-2019 NW Digital Radio
*
* Author: Annaliese McDermond <nh6z@nh6z.net>
*
* Based on sound/soc/codecs/wm8974 and TI driver for kernel 2.6.27.
*
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
*/
#include <linux/spi/spi.h>
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
#include <linux/cdev.h>
#include <linux/slab.h>
#include <linux/clk.h>
+#include <linux/of_clk.h>
#include <linux/regulator/consumer.h>
#include <sound/tlv320aic32x4.h>
#include "tlv320aic32x4.h"
-struct aic32x4_rate_divs {
- u32 mclk;
- u32 rate;
- u8 p_val;
- u8 pll_j;
- u16 pll_d;
- u16 dosr;
- u8 ndac;
- u8 mdac;
- u8 aosr;
- u8 nadc;
- u8 madc;
- u8 blck_N;
-};
-
struct aic32x4_priv {
struct regmap *regmap;
- u32 sysclk;
u32 power_cfg;
u32 micpga_routing;
bool swapdacs;
int rstn_gpio;
- struct clk *mclk;
+ const char *mclk_name;
struct regulator *supply_ldo;
struct regulator *supply_iov;
/* -12dB min, 0.5dB steps */
static DECLARE_TLV_DB_SCALE(tlv_adc_vol, -1200, 50, 0);
+static const char * const lo_cm_text[] = {
+ "Full Chip", "1.65V",
+};
+
+static SOC_ENUM_SINGLE_DECL(lo_cm_enum, AIC32X4_CMMODE, 3, lo_cm_text);
+
+static const char * const ptm_text[] = {
+ "P3", "P2", "P1",
+};
+
+static SOC_ENUM_SINGLE_DECL(l_ptm_enum, AIC32X4_LPLAYBACK, 2, ptm_text);
+static SOC_ENUM_SINGLE_DECL(r_ptm_enum, AIC32X4_RPLAYBACK, 2, ptm_text);
+
static const struct snd_kcontrol_new aic32x4_snd_controls[] = {
SOC_DOUBLE_R_S_TLV("PCM Playback Volume", AIC32X4_LDACVOL,
AIC32X4_RDACVOL, 0, -0x7f, 0x30, 7, 0, tlv_pcm),
+ SOC_ENUM("DAC Left Playback PowerTune Switch", l_ptm_enum),
+ SOC_ENUM("DAC Right Playback PowerTune Switch", r_ptm_enum),
SOC_DOUBLE_R_S_TLV("HP Driver Gain Volume", AIC32X4_HPLGAIN,
AIC32X4_HPRGAIN, 0, -0x6, 0x1d, 5, 0,
tlv_driver_gain),
AIC32X4_HPRGAIN, 6, 0x01, 1),
SOC_DOUBLE_R("LO DAC Playback Switch", AIC32X4_LOLGAIN,
AIC32X4_LORGAIN, 6, 0x01, 1),
+ SOC_ENUM("LO Playback Common Mode Switch", lo_cm_enum),
SOC_DOUBLE_R("Mic PGA Switch", AIC32X4_LMICPGAVOL,
AIC32X4_RMICPGAVOL, 7, 0x01, 1),
0, 0x0F, 0),
};
-static const struct aic32x4_rate_divs aic32x4_divs[] = {
- /* 8k rate */
- {12000000, 8000, 1, 7, 6800, 768, 5, 3, 128, 5, 18, 24},
- {24000000, 8000, 2, 7, 6800, 768, 15, 1, 64, 45, 4, 24},
- {25000000, 8000, 2, 7, 3728, 768, 15, 1, 64, 45, 4, 24},
- /* 11.025k rate */
- {12000000, 11025, 1, 7, 5264, 512, 8, 2, 128, 8, 8, 16},
- {24000000, 11025, 2, 7, 5264, 512, 16, 1, 64, 32, 4, 16},
- /* 16k rate */
- {12000000, 16000, 1, 7, 6800, 384, 5, 3, 128, 5, 9, 12},
- {24000000, 16000, 2, 7, 6800, 384, 15, 1, 64, 18, 5, 12},
- {25000000, 16000, 2, 7, 3728, 384, 15, 1, 64, 18, 5, 12},
- /* 22.05k rate */
- {12000000, 22050, 1, 7, 5264, 256, 4, 4, 128, 4, 8, 8},
- {24000000, 22050, 2, 7, 5264, 256, 16, 1, 64, 16, 4, 8},
- {25000000, 22050, 2, 7, 2253, 256, 16, 1, 64, 16, 4, 8},
- /* 32k rate */
- {12000000, 32000, 1, 7, 1680, 192, 2, 7, 64, 2, 21, 6},
- {24000000, 32000, 2, 7, 1680, 192, 7, 2, 64, 7, 6, 6},
- /* 44.1k rate */
- {12000000, 44100, 1, 7, 5264, 128, 2, 8, 128, 2, 8, 4},
- {24000000, 44100, 2, 7, 5264, 128, 8, 2, 64, 8, 4, 4},
- {25000000, 44100, 2, 7, 2253, 128, 8, 2, 64, 8, 4, 4},
- /* 48k rate */
- {12000000, 48000, 1, 8, 1920, 128, 2, 8, 128, 2, 8, 4},
- {24000000, 48000, 2, 8, 1920, 128, 8, 2, 64, 8, 4, 4},
- {25000000, 48000, 2, 7, 8643, 128, 8, 2, 64, 8, 4, 4},
-
- /* 96k rate */
- {25000000, 96000, 2, 7, 8643, 64, 4, 4, 64, 4, 4, 1},
-};
-
static const struct snd_kcontrol_new hpl_output_mixer_controls[] = {
SOC_DAPM_SINGLE("L_DAC Switch", AIC32X4_HPLROUTE, 3, 1, 0),
SOC_DAPM_SINGLE("IN1_L Switch", AIC32X4_HPLROUTE, 2, 1, 0),
SOC_DAPM_ENUM("IN3_R L- Switch", in3r_lpga_n_enum),
};
-/* Right mixer pins */
+/* Right mixer pins */
static SOC_ENUM_SINGLE_DECL(in1r_rpga_p_enum, AIC32X4_RMICPGAPIN, 6, resistor_text);
static SOC_ENUM_SINGLE_DECL(in2r_rpga_p_enum, AIC32X4_RMICPGAPIN, 4, resistor_text);
static SOC_ENUM_SINGLE_DECL(in3r_rpga_p_enum, AIC32X4_RMICPGAPIN, 2, resistor_text);
static const struct regmap_range_cfg aic32x4_regmap_pages[] = {
{
.selector_reg = 0,
- .selector_mask = 0xff,
+ .selector_mask = 0xff,
.window_start = 0,
.window_len = 128,
.range_min = 0,
};
EXPORT_SYMBOL(aic32x4_regmap_config);
-static inline int aic32x4_get_divs(int mclk, int rate)
-{
- int i;
-
- for (i = 0; i < ARRAY_SIZE(aic32x4_divs); i++) {
- if ((aic32x4_divs[i].rate == rate)
- && (aic32x4_divs[i].mclk == mclk)) {
- return i;
- }
- }
- printk(KERN_ERR "aic32x4: master clock and sample rate is not supported\n");
- return -EINVAL;
-}
-
static int aic32x4_set_dai_sysclk(struct snd_soc_dai *codec_dai,
int clk_id, unsigned int freq, int dir)
{
struct snd_soc_component *component = codec_dai->component;
- struct aic32x4_priv *aic32x4 = snd_soc_component_get_drvdata(component);
+ struct clk *mclk;
+ struct clk *pll;
- switch (freq) {
- case 12000000:
- case 24000000:
- case 25000000:
- aic32x4->sysclk = freq;
- return 0;
- }
- printk(KERN_ERR "aic32x4: invalid frequency to set DAI system clock\n");
- return -EINVAL;
+ pll = devm_clk_get(component->dev, "pll");
+ mclk = clk_get_parent(pll);
+
+ return clk_set_rate(mclk, freq);
}
static int aic32x4_set_dai_fmt(struct snd_soc_dai *codec_dai, unsigned int fmt)
}
snd_soc_component_update_bits(component, AIC32X4_IFACE1,
- AIC32X4_IFACE1_DATATYPE_MASK |
- AIC32X4_IFACE1_MASTER_MASK, iface_reg_1);
+ AIC32X4_IFACE1_DATATYPE_MASK |
+ AIC32X4_IFACE1_MASTER_MASK, iface_reg_1);
snd_soc_component_update_bits(component, AIC32X4_IFACE2,
- AIC32X4_DATA_OFFSET_MASK, iface_reg_2);
+ AIC32X4_DATA_OFFSET_MASK, iface_reg_2);
snd_soc_component_update_bits(component, AIC32X4_IFACE3,
- AIC32X4_BCLKINV_MASK, iface_reg_3);
+ AIC32X4_BCLKINV_MASK, iface_reg_3);
return 0;
}
-static int aic32x4_hw_params(struct snd_pcm_substream *substream,
- struct snd_pcm_hw_params *params,
- struct snd_soc_dai *dai)
+static int aic32x4_set_aosr(struct snd_soc_component *component, u8 aosr)
{
- struct snd_soc_component *component = dai->component;
- struct aic32x4_priv *aic32x4 = snd_soc_component_get_drvdata(component);
- u8 iface1_reg = 0;
- u8 dacsetup_reg = 0;
- int i;
-
- i = aic32x4_get_divs(aic32x4->sysclk, params_rate(params));
- if (i < 0) {
- printk(KERN_ERR "aic32x4: sampling rate not supported\n");
- return i;
- }
+ return snd_soc_component_write(component, AIC32X4_AOSR, aosr);
+}
- /* MCLK as PLL_CLKIN */
- snd_soc_component_update_bits(component, AIC32X4_CLKMUX, AIC32X4_PLL_CLKIN_MASK,
- AIC32X4_PLL_CLKIN_MCLK << AIC32X4_PLL_CLKIN_SHIFT);
- /* PLL as CODEC_CLKIN */
- snd_soc_component_update_bits(component, AIC32X4_CLKMUX, AIC32X4_CODEC_CLKIN_MASK,
- AIC32X4_CODEC_CLKIN_PLL << AIC32X4_CODEC_CLKIN_SHIFT);
- /* DAC_MOD_CLK as BDIV_CLKIN */
- snd_soc_component_update_bits(component, AIC32X4_IFACE3, AIC32X4_BDIVCLK_MASK,
- AIC32X4_DACMOD2BCLK << AIC32X4_BDIVCLK_SHIFT);
+static int aic32x4_set_dosr(struct snd_soc_component *component, u16 dosr)
+{
+ snd_soc_component_write(component, AIC32X4_DOSRMSB, dosr >> 8);
+ snd_soc_component_write(component, AIC32X4_DOSRLSB,
+ (dosr & 0xff));
- /* We will fix R value to 1 and will make P & J=K.D as variable */
- snd_soc_component_update_bits(component, AIC32X4_PLLPR, AIC32X4_PLL_R_MASK, 0x01);
+ return 0;
+}
- /* PLL P value */
- snd_soc_component_update_bits(component, AIC32X4_PLLPR, AIC32X4_PLL_P_MASK,
- aic32x4_divs[i].p_val << AIC32X4_PLL_P_SHIFT);
+static int aic32x4_set_processing_blocks(struct snd_soc_component *component,
+ u8 r_block, u8 p_block)
+{
+ if (r_block > 18 || p_block > 25)
+ return -EINVAL;
- /* PLL J value */
- snd_soc_component_write(component, AIC32X4_PLLJ, aic32x4_divs[i].pll_j);
+ snd_soc_component_write(component, AIC32X4_ADCSPB, r_block);
+ snd_soc_component_write(component, AIC32X4_DACSPB, p_block);
- /* PLL D value */
- snd_soc_component_write(component, AIC32X4_PLLDMSB, (aic32x4_divs[i].pll_d >> 8));
- snd_soc_component_write(component, AIC32X4_PLLDLSB, (aic32x4_divs[i].pll_d & 0xff));
+ return 0;
+}
- /* NDAC divider value */
- snd_soc_component_update_bits(component, AIC32X4_NDAC,
- AIC32X4_NDAC_MASK, aic32x4_divs[i].ndac);
+static int aic32x4_setup_clocks(struct snd_soc_component *component,
+ unsigned int sample_rate)
+{
+ u8 aosr;
+ u16 dosr;
+ u8 adc_resource_class, dac_resource_class;
+ u8 madc, nadc, mdac, ndac, max_nadc, min_mdac, max_ndac;
+ u8 dosr_increment;
+ u16 max_dosr, min_dosr;
+ unsigned long adc_clock_rate, dac_clock_rate;
+ int ret;
- /* MDAC divider value */
- snd_soc_component_update_bits(component, AIC32X4_MDAC,
- AIC32X4_MDAC_MASK, aic32x4_divs[i].mdac);
+ struct clk_bulk_data clocks[] = {
+ { .id = "pll" },
+ { .id = "nadc" },
+ { .id = "madc" },
+ { .id = "ndac" },
+ { .id = "mdac" },
+ { .id = "bdiv" },
+ };
+ ret = devm_clk_bulk_get(component->dev, ARRAY_SIZE(clocks), clocks);
+ if (ret)
+ return ret;
- /* DOSR MSB & LSB values */
- snd_soc_component_write(component, AIC32X4_DOSRMSB, aic32x4_divs[i].dosr >> 8);
- snd_soc_component_write(component, AIC32X4_DOSRLSB, (aic32x4_divs[i].dosr & 0xff));
+ if (sample_rate <= 48000) {
+ aosr = 128;
+ adc_resource_class = 6;
+ dac_resource_class = 8;
+ dosr_increment = 8;
+ aic32x4_set_processing_blocks(component, 1, 1);
+ } else if (sample_rate <= 96000) {
+ aosr = 64;
+ adc_resource_class = 6;
+ dac_resource_class = 8;
+ dosr_increment = 4;
+ aic32x4_set_processing_blocks(component, 1, 9);
+ } else if (sample_rate == 192000) {
+ aosr = 32;
+ adc_resource_class = 3;
+ dac_resource_class = 4;
+ dosr_increment = 2;
+ aic32x4_set_processing_blocks(component, 13, 19);
+ } else {
+ dev_err(component->dev, "Sampling rate not supported\n");
+ return -EINVAL;
+ }
- /* NADC divider value */
- snd_soc_component_update_bits(component, AIC32X4_NADC,
- AIC32X4_NADC_MASK, aic32x4_divs[i].nadc);
+ madc = DIV_ROUND_UP((32 * adc_resource_class), aosr);
+ max_dosr = (AIC32X4_MAX_DOSR_FREQ / sample_rate / dosr_increment) *
+ dosr_increment;
+ min_dosr = (AIC32X4_MIN_DOSR_FREQ / sample_rate / dosr_increment) *
+ dosr_increment;
+ max_nadc = AIC32X4_MAX_CODEC_CLKIN_FREQ / (madc * aosr * sample_rate);
+
+ for (nadc = max_nadc; nadc > 0; --nadc) {
+ adc_clock_rate = nadc * madc * aosr * sample_rate;
+ for (dosr = max_dosr; dosr >= min_dosr;
+ dosr -= dosr_increment) {
+ min_mdac = DIV_ROUND_UP((32 * dac_resource_class), dosr);
+ max_ndac = AIC32X4_MAX_CODEC_CLKIN_FREQ /
+ (min_mdac * dosr * sample_rate);
+ for (mdac = min_mdac; mdac <= 128; ++mdac) {
+ for (ndac = max_ndac; ndac > 0; --ndac) {
+ dac_clock_rate = ndac * mdac * dosr *
+ sample_rate;
+ if (dac_clock_rate == adc_clock_rate) {
+ if (clk_round_rate(clocks[0].clk, dac_clock_rate) == 0)
+ continue;
+
+ clk_set_rate(clocks[0].clk,
+ dac_clock_rate);
+
+ clk_set_rate(clocks[1].clk,
+ sample_rate * aosr *
+ madc);
+ clk_set_rate(clocks[2].clk,
+ sample_rate * aosr);
+ aic32x4_set_aosr(component,
+ aosr);
+
+ clk_set_rate(clocks[3].clk,
+ sample_rate * dosr *
+ mdac);
+ clk_set_rate(clocks[4].clk,
+ sample_rate * dosr);
+ aic32x4_set_dosr(component,
+ dosr);
+
+ clk_set_rate(clocks[5].clk,
+ sample_rate * 32);
+ return 0;
+ }
+ }
+ }
+ }
+ }
- /* MADC divider value */
- snd_soc_component_update_bits(component, AIC32X4_MADC,
- AIC32X4_MADC_MASK, aic32x4_divs[i].madc);
+ dev_err(component->dev,
+ "Could not set clocks to support sample rate.\n");
+ return -EINVAL;
+}
- /* AOSR value */
- snd_soc_component_write(component, AIC32X4_AOSR, aic32x4_divs[i].aosr);
+static int aic32x4_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params,
+ struct snd_soc_dai *dai)
+{
+ struct snd_soc_component *component = dai->component;
+ struct aic32x4_priv *aic32x4 = snd_soc_component_get_drvdata(component);
+ u8 iface1_reg = 0;
+ u8 dacsetup_reg = 0;
- /* BCLK N divider */
- snd_soc_component_update_bits(component, AIC32X4_BCLKN,
- AIC32X4_BCLK_MASK, aic32x4_divs[i].blck_N);
+ aic32x4_setup_clocks(component, params_rate(params));
switch (params_width(params)) {
case 16:
iface1_reg |= (AIC32X4_WORD_LEN_16BITS <<
- AIC32X4_IFACE1_DATALEN_SHIFT);
+ AIC32X4_IFACE1_DATALEN_SHIFT);
break;
case 20:
iface1_reg |= (AIC32X4_WORD_LEN_20BITS <<
- AIC32X4_IFACE1_DATALEN_SHIFT);
+ AIC32X4_IFACE1_DATALEN_SHIFT);
break;
case 24:
iface1_reg |= (AIC32X4_WORD_LEN_24BITS <<
- AIC32X4_IFACE1_DATALEN_SHIFT);
+ AIC32X4_IFACE1_DATALEN_SHIFT);
break;
case 32:
iface1_reg |= (AIC32X4_WORD_LEN_32BITS <<
- AIC32X4_IFACE1_DATALEN_SHIFT);
+ AIC32X4_IFACE1_DATALEN_SHIFT);
break;
}
snd_soc_component_update_bits(component, AIC32X4_IFACE1,
- AIC32X4_IFACE1_DATALEN_MASK, iface1_reg);
+ AIC32X4_IFACE1_DATALEN_MASK, iface1_reg);
if (params_channels(params) == 1) {
dacsetup_reg = AIC32X4_RDAC2LCHN | AIC32X4_LDAC2LCHN;
dacsetup_reg = AIC32X4_LDAC2LCHN | AIC32X4_RDAC2RCHN;
}
snd_soc_component_update_bits(component, AIC32X4_DACSETUP,
- AIC32X4_DAC_CHAN_MASK, dacsetup_reg);
+ AIC32X4_DAC_CHAN_MASK, dacsetup_reg);
return 0;
}
struct snd_soc_component *component = dai->component;
snd_soc_component_update_bits(component, AIC32X4_DACMUTE,
- AIC32X4_MUTEON, mute ? AIC32X4_MUTEON : 0);
+ AIC32X4_MUTEON, mute ? AIC32X4_MUTEON : 0);
return 0;
}
static int aic32x4_set_bias_level(struct snd_soc_component *component,
enum snd_soc_bias_level level)
{
- struct aic32x4_priv *aic32x4 = snd_soc_component_get_drvdata(component);
int ret;
+ struct clk_bulk_data clocks[] = {
+ { .id = "madc" },
+ { .id = "mdac" },
+ { .id = "bdiv" },
+ };
+
+ ret = devm_clk_bulk_get(component->dev, ARRAY_SIZE(clocks), clocks);
+ if (ret)
+ return ret;
+
switch (level) {
case SND_SOC_BIAS_ON:
- /* Switch on master clock */
- ret = clk_prepare_enable(aic32x4->mclk);
+ ret = clk_bulk_prepare_enable(ARRAY_SIZE(clocks), clocks);
if (ret) {
- dev_err(component->dev, "Failed to enable master clock\n");
+ dev_err(component->dev, "Failed to enable clocks\n");
return ret;
}
-
- /* Switch on PLL */
- snd_soc_component_update_bits(component, AIC32X4_PLLPR,
- AIC32X4_PLLEN, AIC32X4_PLLEN);
-
- /* Switch on NDAC Divider */
- snd_soc_component_update_bits(component, AIC32X4_NDAC,
- AIC32X4_NDACEN, AIC32X4_NDACEN);
-
- /* Switch on MDAC Divider */
- snd_soc_component_update_bits(component, AIC32X4_MDAC,
- AIC32X4_MDACEN, AIC32X4_MDACEN);
-
- /* Switch on NADC Divider */
- snd_soc_component_update_bits(component, AIC32X4_NADC,
- AIC32X4_NADCEN, AIC32X4_NADCEN);
-
- /* Switch on MADC Divider */
- snd_soc_component_update_bits(component, AIC32X4_MADC,
- AIC32X4_MADCEN, AIC32X4_MADCEN);
-
- /* Switch on BCLK_N Divider */
- snd_soc_component_update_bits(component, AIC32X4_BCLKN,
- AIC32X4_BCLKEN, AIC32X4_BCLKEN);
break;
case SND_SOC_BIAS_PREPARE:
break;
if (snd_soc_component_get_bias_level(component) == SND_SOC_BIAS_OFF)
break;
- /* Switch off BCLK_N Divider */
- snd_soc_component_update_bits(component, AIC32X4_BCLKN,
- AIC32X4_BCLKEN, 0);
-
- /* Switch off MADC Divider */
- snd_soc_component_update_bits(component, AIC32X4_MADC,
- AIC32X4_MADCEN, 0);
-
- /* Switch off NADC Divider */
- snd_soc_component_update_bits(component, AIC32X4_NADC,
- AIC32X4_NADCEN, 0);
-
- /* Switch off MDAC Divider */
- snd_soc_component_update_bits(component, AIC32X4_MDAC,
- AIC32X4_MDACEN, 0);
-
- /* Switch off NDAC Divider */
- snd_soc_component_update_bits(component, AIC32X4_NDAC,
- AIC32X4_NDACEN, 0);
-
- /* Switch off PLL */
- snd_soc_component_update_bits(component, AIC32X4_PLLPR,
- AIC32X4_PLLEN, 0);
-
- /* Switch off master clock */
- clk_disable_unprepare(aic32x4->mclk);
+ clk_bulk_disable_unprepare(ARRAY_SIZE(clocks), clocks);
break;
case SND_SOC_BIAS_OFF:
break;
return 0;
}
-#define AIC32X4_RATES SNDRV_PCM_RATE_8000_96000
-#define AIC32X4_FORMATS (SNDRV_PCM_FMTBIT_S16_LE | SNDRV_PCM_FMTBIT_S20_3LE \
+#define AIC32X4_RATES SNDRV_PCM_RATE_8000_192000
+#define AIC32X4_FORMATS (SNDRV_PCM_FMTBIT_S16_LE | SNDRV_PCM_FMTBIT_S20_3LE \
| SNDRV_PCM_FMTBIT_S24_3LE | SNDRV_PCM_FMTBIT_S32_LE)
static const struct snd_soc_dai_ops aic32x4_ops = {
static struct snd_soc_dai_driver aic32x4_dai = {
.name = "tlv320aic32x4-hifi",
.playback = {
- .stream_name = "Playback",
- .channels_min = 1,
- .channels_max = 2,
- .rates = AIC32X4_RATES,
- .formats = AIC32X4_FORMATS,},
+ .stream_name = "Playback",
+ .channels_min = 1,
+ .channels_max = 2,
+ .rates = AIC32X4_RATES,
+ .formats = AIC32X4_FORMATS,},
.capture = {
- .stream_name = "Capture",
- .channels_min = 1,
- .channels_max = 2,
- .rates = AIC32X4_RATES,
- .formats = AIC32X4_FORMATS,},
+ .stream_name = "Capture",
+ .channels_min = 1,
+ .channels_max = 2,
+ .rates = AIC32X4_RATES,
+ .formats = AIC32X4_FORMATS,},
.ops = &aic32x4_ops,
.symmetric_rates = 1,
};
/* MFP1 */
if (aic32x4->setup->gpio_func[0] != AIC32X4_MFPX_DEFAULT_VALUE) {
snd_soc_component_write(component, AIC32X4_DINCTL,
- aic32x4->setup->gpio_func[0]);
+ aic32x4->setup->gpio_func[0]);
snd_soc_add_component_controls(component, aic32x4_mfp1,
ARRAY_SIZE(aic32x4_mfp1));
}
/* MFP2 */
if (aic32x4->setup->gpio_func[1] != AIC32X4_MFPX_DEFAULT_VALUE) {
snd_soc_component_write(component, AIC32X4_DOUTCTL,
- aic32x4->setup->gpio_func[1]);
+ aic32x4->setup->gpio_func[1]);
snd_soc_add_component_controls(component, aic32x4_mfp2,
ARRAY_SIZE(aic32x4_mfp2));
}
/* MFP3 */
if (aic32x4->setup->gpio_func[2] != AIC32X4_MFPX_DEFAULT_VALUE) {
snd_soc_component_write(component, AIC32X4_SCLKCTL,
- aic32x4->setup->gpio_func[2]);
+ aic32x4->setup->gpio_func[2]);
snd_soc_add_component_controls(component, aic32x4_mfp3,
ARRAY_SIZE(aic32x4_mfp3));
}
/* MFP4 */
if (aic32x4->setup->gpio_func[3] != AIC32X4_MFPX_DEFAULT_VALUE) {
snd_soc_component_write(component, AIC32X4_MISOCTL,
- aic32x4->setup->gpio_func[3]);
+ aic32x4->setup->gpio_func[3]);
snd_soc_add_component_controls(component, aic32x4_mfp4,
ARRAY_SIZE(aic32x4_mfp4));
}
/* MFP5 */
if (aic32x4->setup->gpio_func[4] != AIC32X4_MFPX_DEFAULT_VALUE) {
snd_soc_component_write(component, AIC32X4_GPIOCTL,
- aic32x4->setup->gpio_func[4]);
+ aic32x4->setup->gpio_func[4]);
snd_soc_add_component_controls(component, aic32x4_mfp5,
ARRAY_SIZE(aic32x4_mfp5));
}
{
struct aic32x4_priv *aic32x4 = snd_soc_component_get_drvdata(component);
u32 tmp_reg;
+ int ret;
+
+ struct clk_bulk_data clocks[] = {
+ { .id = "codec_clkin" },
+ { .id = "pll" },
+ { .id = "bdiv" },
+ { .id = "mdac" },
+ };
+
+ ret = devm_clk_bulk_get(component->dev, ARRAY_SIZE(clocks), clocks);
+ if (ret)
+ return ret;
if (gpio_is_valid(aic32x4->rstn_gpio)) {
ndelay(10);
if (aic32x4->setup)
aic32x4_setup_gpios(component);
+ clk_set_parent(clocks[0].clk, clocks[1].clk);
+ clk_set_parent(clocks[2].clk, clocks[3].clk);
+
/* Power platform configuration */
if (aic32x4->power_cfg & AIC32X4_PWR_MICBIAS_2075_LDOIN) {
- snd_soc_component_write(component, AIC32X4_MICBIAS, AIC32X4_MICBIAS_LDOIN |
- AIC32X4_MICBIAS_2075V);
+ snd_soc_component_write(component, AIC32X4_MICBIAS,
+ AIC32X4_MICBIAS_LDOIN | AIC32X4_MICBIAS_2075V);
}
if (aic32x4->power_cfg & AIC32X4_PWR_AVDD_DVDD_WEAK_DISABLE)
snd_soc_component_write(component, AIC32X4_PWRCFG, AIC32X4_AVDDWEAKDISABLE);
struct device_node *np)
{
struct aic32x4_setup_data *aic32x4_setup;
+ int ret;
aic32x4_setup = devm_kzalloc(aic32x4->dev, sizeof(*aic32x4_setup),
GFP_KERNEL);
if (!aic32x4_setup)
return -ENOMEM;
+ ret = of_property_match_string(np, "clock-names", "mclk");
+ if (ret < 0)
+ return -EINVAL;
+ aic32x4->mclk_name = of_clk_get_parent_name(np, ret);
+
aic32x4->swapdacs = false;
aic32x4->micpga_routing = 0;
aic32x4->rstn_gpio = of_get_named_gpio(np, "reset-gpios", 0);
return PTR_ERR(regmap);
aic32x4 = devm_kzalloc(dev, sizeof(struct aic32x4_priv),
- GFP_KERNEL);
+ GFP_KERNEL);
if (aic32x4 == NULL)
return -ENOMEM;
aic32x4->swapdacs = pdata->swapdacs;
aic32x4->micpga_routing = pdata->micpga_routing;
aic32x4->rstn_gpio = pdata->rstn_gpio;
+ aic32x4->mclk_name = "mclk";
} else if (np) {
ret = aic32x4_parse_dt(aic32x4, np);
if (ret) {
aic32x4->swapdacs = false;
aic32x4->micpga_routing = 0;
aic32x4->rstn_gpio = -1;
+ aic32x4->mclk_name = "mclk";
}
- aic32x4->mclk = devm_clk_get(dev, "mclk");
- if (IS_ERR(aic32x4->mclk)) {
- dev_err(dev, "Failed getting the mclk. The current implementation does not support the usage of this codec without mclk\n");
- return PTR_ERR(aic32x4->mclk);
- }
+ ret = aic32x4_register_clocks(dev, aic32x4->mclk_name);
+ if (ret)
+ return ret;
if (gpio_is_valid(aic32x4->rstn_gpio)) {
ret = devm_gpio_request_one(dev, aic32x4->rstn_gpio,
extern const struct regmap_config aic32x4_regmap_config;
int aic32x4_probe(struct device *dev, struct regmap *regmap);
int aic32x4_remove(struct device *dev);
+int aic32x4_register_clocks(struct device *dev, const char *mclk_name);
/* tlv320aic32x4 register space (in decimal to match datasheet) */
#define AIC32X4_PWRCFG AIC32X4_REG(1, 1)
#define AIC32X4_LDOCTL AIC32X4_REG(1, 2)
+#define AIC32X4_LPLAYBACK AIC32X4_REG(1, 3)
+#define AIC32X4_RPLAYBACK AIC32X4_REG(1, 4)
#define AIC32X4_OUTPWRCTL AIC32X4_REG(1, 9)
#define AIC32X4_CMMODE AIC32X4_REG(1, 10)
#define AIC32X4_HPLROUTE AIC32X4_REG(1, 12)
#define AIC32X4_RMICPGANIN_IN1L_10K 0x10
#define AIC32X4_RMICPGANIN_CM1R_10K 0x40
+/* Common mask and enable for all of the dividers */
+#define AIC32X4_DIVEN BIT(7)
+#define AIC32X4_DIV_MASK GENMASK(6, 0)
+
+/* Clock Limits */
+#define AIC32X4_MAX_DOSR_FREQ 6200000
+#define AIC32X4_MIN_DOSR_FREQ 2800000
+#define AIC32X4_MAX_CODEC_CLKIN_FREQ 110000000
+#define AIC32X4_MAX_PLL_CLKIN 20000000
+
#endif /* _TLV320AIC32X4_H */
wcd->slim = sdev;
wcd->slim_ifc_dev = of_slim_get_device(sdev->ctrl, ifc_dev_np);
+ of_node_put(ifc_dev_np);
if (!wcd->slim_ifc_dev) {
dev_err(dev, "Unable to get SLIM Interface device\n");
return -EINVAL;
return ret;
}
}
+
+ wm_adsp2_set_dspclk(w, v);
break;
case SND_SOC_DAPM_POST_PMD:
break;
}
- return wm_adsp2_early_event(w, kcontrol, event, v);
+ return wm_adsp_early_event(w, kcontrol, event);
}
static int wm5102_out_comp_coeff_get(struct snd_kcontrol *kcontrol,
v = (v & ARIZONA_SYSCLK_FREQ_MASK) >> ARIZONA_SYSCLK_FREQ_SHIFT;
- return wm_adsp2_early_event(w, kcontrol, event, v);
+ wm_adsp2_set_dspclk(w, v);
+
+ return wm_adsp_early_event(w, kcontrol, event);
}
static const struct reg_sequence wm5110_no_dre_left_enable[] = {
*/
#define WM_ADSP_FW_EVENT_SHUTDOWN 0x000001
+/*
+ * HALO system info
+ */
+#define HALO_AHBM_WINDOW_DEBUG_0 0x02040
+#define HALO_AHBM_WINDOW_DEBUG_1 0x02044
+
+/*
+ * HALO core
+ */
+#define HALO_SCRATCH1 0x005c0
+#define HALO_SCRATCH2 0x005c8
+#define HALO_SCRATCH3 0x005d0
+#define HALO_SCRATCH4 0x005d8
+#define HALO_CCM_CORE_CONTROL 0x41000
+#define HALO_CORE_SOFT_RESET 0x00010
+#define HALO_WDT_CONTROL 0x47000
+
+/*
+ * HALO MPU banks
+ */
+#define HALO_MPU_XMEM_ACCESS_0 0x43000
+#define HALO_MPU_YMEM_ACCESS_0 0x43004
+#define HALO_MPU_WINDOW_ACCESS_0 0x43008
+#define HALO_MPU_XREG_ACCESS_0 0x4300C
+#define HALO_MPU_YREG_ACCESS_0 0x43014
+#define HALO_MPU_XMEM_ACCESS_1 0x43018
+#define HALO_MPU_YMEM_ACCESS_1 0x4301C
+#define HALO_MPU_WINDOW_ACCESS_1 0x43020
+#define HALO_MPU_XREG_ACCESS_1 0x43024
+#define HALO_MPU_YREG_ACCESS_1 0x4302C
+#define HALO_MPU_XMEM_ACCESS_2 0x43030
+#define HALO_MPU_YMEM_ACCESS_2 0x43034
+#define HALO_MPU_WINDOW_ACCESS_2 0x43038
+#define HALO_MPU_XREG_ACCESS_2 0x4303C
+#define HALO_MPU_YREG_ACCESS_2 0x43044
+#define HALO_MPU_XMEM_ACCESS_3 0x43048
+#define HALO_MPU_YMEM_ACCESS_3 0x4304C
+#define HALO_MPU_WINDOW_ACCESS_3 0x43050
+#define HALO_MPU_XREG_ACCESS_3 0x43054
+#define HALO_MPU_YREG_ACCESS_3 0x4305C
+#define HALO_MPU_XM_VIO_ADDR 0x43100
+#define HALO_MPU_XM_VIO_STATUS 0x43104
+#define HALO_MPU_YM_VIO_ADDR 0x43108
+#define HALO_MPU_YM_VIO_STATUS 0x4310C
+#define HALO_MPU_PM_VIO_ADDR 0x43110
+#define HALO_MPU_PM_VIO_STATUS 0x43114
+#define HALO_MPU_LOCK_CONFIG 0x43140
+
+/*
+ * HALO_AHBM_WINDOW_DEBUG_1
+ */
+#define HALO_AHBM_CORE_ERR_ADDR_MASK 0x0fffff00
+#define HALO_AHBM_CORE_ERR_ADDR_SHIFT 8
+#define HALO_AHBM_FLAGS_ERR_MASK 0x000000ff
+
+/*
+ * HALO_CCM_CORE_CONTROL
+ */
+#define HALO_CORE_EN 0x00000001
+
+/*
+ * HALO_CORE_SOFT_RESET
+ */
+#define HALO_CORE_SOFT_RESET_MASK 0x00000001
+
+/*
+ * HALO_WDT_CONTROL
+ */
+#define HALO_WDT_EN_MASK 0x00000001
+
+/*
+ * HALO_MPU_?M_VIO_STATUS
+ */
+#define HALO_MPU_VIO_STS_MASK 0x007e0000
+#define HALO_MPU_VIO_STS_SHIFT 17
+#define HALO_MPU_VIO_ERR_WR_MASK 0x00008000
+#define HALO_MPU_VIO_ERR_SRC_MASK 0x00007fff
+#define HALO_MPU_VIO_ERR_SRC_SHIFT 0
+
+static struct wm_adsp_ops wm_adsp1_ops;
+static struct wm_adsp_ops wm_adsp2_ops[];
+static struct wm_adsp_ops wm_halo_ops;
+
struct wm_adsp_buf {
struct list_head list;
void *buf;
__be32 build_job_number;
};
+struct wm_halo_system_config_xm_hdr {
+ __be32 halo_heartbeat;
+ __be32 build_job_name[3];
+ __be32 build_job_number;
+};
+
struct wm_adsp_alg_xm_struct {
__be32 magic;
__be32 smoothing;
switch (type) {
case WMFW_ADSP1_PM:
return "PM";
+ case WMFW_HALO_PM_PACKED:
+ return "PM_PACKED";
case WMFW_ADSP1_DM:
return "DM";
case WMFW_ADSP2_XM:
return "XM";
+ case WMFW_HALO_XM_PACKED:
+ return "XM_PACKED";
case WMFW_ADSP2_YM:
return "YM";
+ case WMFW_HALO_YM_PACKED:
+ return "YM_PACKED";
case WMFW_ADSP1_ZM:
return "ZM";
default:
static unsigned int wm_adsp_region_to_reg(struct wm_adsp_region const *mem,
unsigned int offset)
{
- if (WARN_ON(!mem))
- return offset;
switch (mem->type) {
case WMFW_ADSP1_PM:
return mem->base + (offset * 3);
case WMFW_ADSP1_DM:
- return mem->base + (offset * 2);
case WMFW_ADSP2_XM:
- return mem->base + (offset * 2);
case WMFW_ADSP2_YM:
- return mem->base + (offset * 2);
case WMFW_ADSP1_ZM:
return mem->base + (offset * 2);
default:
}
}
-static void wm_adsp2_show_fw_status(struct wm_adsp *dsp)
+static unsigned int wm_halo_region_to_reg(struct wm_adsp_region const *mem,
+ unsigned int offset)
+{
+ switch (mem->type) {
+ case WMFW_ADSP2_XM:
+ case WMFW_ADSP2_YM:
+ return mem->base + (offset * 4);
+ case WMFW_HALO_XM_PACKED:
+ case WMFW_HALO_YM_PACKED:
+ return (mem->base + (offset * 3)) & ~0x3;
+ case WMFW_HALO_PM_PACKED:
+ return mem->base + (offset * 5);
+ default:
+ WARN(1, "Unknown memory region type");
+ return offset;
+ }
+}
+
+static void wm_adsp_read_fw_status(struct wm_adsp *dsp,
+ int noffs, unsigned int *offs)
{
- unsigned int scratch[4];
- unsigned int addr = dsp->base + ADSP2_SCRATCH0;
unsigned int i;
int ret;
- for (i = 0; i < ARRAY_SIZE(scratch); ++i) {
- ret = regmap_read(dsp->regmap, addr + i, &scratch[i]);
+ for (i = 0; i < noffs; ++i) {
+ ret = regmap_read(dsp->regmap, dsp->base + offs[i], &offs[i]);
if (ret) {
adsp_err(dsp, "Failed to read SCRATCH%u: %d\n", i, ret);
return;
}
}
+}
+
+static void wm_adsp2_show_fw_status(struct wm_adsp *dsp)
+{
+ unsigned int offs[] = {
+ ADSP2_SCRATCH0, ADSP2_SCRATCH1, ADSP2_SCRATCH2, ADSP2_SCRATCH3,
+ };
+
+ wm_adsp_read_fw_status(dsp, ARRAY_SIZE(offs), offs);
adsp_dbg(dsp, "FW SCRATCH 0:0x%x 1:0x%x 2:0x%x 3:0x%x\n",
- scratch[0], scratch[1], scratch[2], scratch[3]);
+ offs[0], offs[1], offs[2], offs[3]);
}
static void wm_adsp2v2_show_fw_status(struct wm_adsp *dsp)
{
- unsigned int scratch[2];
- int ret;
+ unsigned int offs[] = { ADSP2V2_SCRATCH0_1, ADSP2V2_SCRATCH2_3 };
- ret = regmap_read(dsp->regmap, dsp->base + ADSP2V2_SCRATCH0_1,
- &scratch[0]);
- if (ret) {
- adsp_err(dsp, "Failed to read SCRATCH0_1: %d\n", ret);
- return;
- }
+ wm_adsp_read_fw_status(dsp, ARRAY_SIZE(offs), offs);
- ret = regmap_read(dsp->regmap, dsp->base + ADSP2V2_SCRATCH2_3,
- &scratch[1]);
- if (ret) {
- adsp_err(dsp, "Failed to read SCRATCH2_3: %d\n", ret);
- return;
- }
+ adsp_dbg(dsp, "FW SCRATCH 0:0x%x 1:0x%x 2:0x%x 3:0x%x\n",
+ offs[0] & 0xFFFF, offs[0] >> 16,
+ offs[1] & 0xFFFF, offs[1] >> 16);
+}
+
+static void wm_halo_show_fw_status(struct wm_adsp *dsp)
+{
+ unsigned int offs[] = {
+ HALO_SCRATCH1, HALO_SCRATCH2, HALO_SCRATCH3, HALO_SCRATCH4,
+ };
+
+ wm_adsp_read_fw_status(dsp, ARRAY_SIZE(offs), offs);
adsp_dbg(dsp, "FW SCRATCH 0:0x%x 1:0x%x 2:0x%x 3:0x%x\n",
- scratch[0] & 0xFFFF,
- scratch[0] >> 16,
- scratch[1] & 0xFFFF,
- scratch[1] >> 16);
+ offs[0], offs[1], offs[2], offs[3]);
}
static inline struct wm_coeff_ctl *bytes_ext_to_ctl(struct soc_bytes_ext *ext)
return -EINVAL;
}
- *reg = wm_adsp_region_to_reg(mem, ctl->alg_region.base + ctl->offset);
+ *reg = dsp->ops->region_to_reg(mem, ctl->alg_region.base + ctl->offset);
return 0;
}
case 1:
snprintf(name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN, "%s %s %x",
dsp->name, region_name, alg_region->alg);
+ subname = NULL; /* don't append subname */
break;
- default:
+ case 2:
ret = snprintf(name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN,
"%s%c %.12s %x", dsp->name, *region_name,
wm_adsp_fw_text[dsp->fw], alg_region->alg);
+ break;
+ default:
+ ret = snprintf(name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN,
+ "%s %.12s %x", dsp->name,
+ wm_adsp_fw_text[dsp->fw], alg_region->alg);
+ break;
+ }
- /* Truncate the subname from the start if it is too long */
- if (subname) {
- int avail = SNDRV_CTL_ELEM_ID_NAME_MAXLEN - ret - 2;
- int skip = 0;
+ if (subname) {
+ int avail = SNDRV_CTL_ELEM_ID_NAME_MAXLEN - ret - 2;
+ int skip = 0;
- if (dsp->component->name_prefix)
- avail -= strlen(dsp->component->name_prefix) + 1;
+ if (dsp->component->name_prefix)
+ avail -= strlen(dsp->component->name_prefix) + 1;
- if (subname_len > avail)
- skip = subname_len - avail;
+ /* Truncate the subname from the start if it is too long */
+ if (subname_len > avail)
+ skip = subname_len - avail;
- snprintf(name + ret,
- SNDRV_CTL_ELEM_ID_NAME_MAXLEN - ret, " %.*s",
- subname_len - skip, subname + skip);
- }
- break;
+ snprintf(name + ret, SNDRV_CTL_ELEM_ID_NAME_MAXLEN - ret,
+ " %.*s", subname_len - skip, subname + skip);
}
list_for_each_entry(ctl, &dsp->ctl_list, list) {
return 0;
}
+static unsigned int wm_adsp1_parse_sizes(struct wm_adsp *dsp,
+ const char * const file,
+ unsigned int pos,
+ const struct firmware *firmware)
+{
+ const struct wmfw_adsp1_sizes *adsp1_sizes;
+
+ adsp1_sizes = (void *)&firmware->data[pos];
+
+ adsp_dbg(dsp, "%s: %d DM, %d PM, %d ZM\n", file,
+ le32_to_cpu(adsp1_sizes->dm), le32_to_cpu(adsp1_sizes->pm),
+ le32_to_cpu(adsp1_sizes->zm));
+
+ return pos + sizeof(*adsp1_sizes);
+}
+
+static unsigned int wm_adsp2_parse_sizes(struct wm_adsp *dsp,
+ const char * const file,
+ unsigned int pos,
+ const struct firmware *firmware)
+{
+ const struct wmfw_adsp2_sizes *adsp2_sizes;
+
+ adsp2_sizes = (void *)&firmware->data[pos];
+
+ adsp_dbg(dsp, "%s: %d XM, %d YM %d PM, %d ZM\n", file,
+ le32_to_cpu(adsp2_sizes->xm), le32_to_cpu(adsp2_sizes->ym),
+ le32_to_cpu(adsp2_sizes->pm), le32_to_cpu(adsp2_sizes->zm));
+
+ return pos + sizeof(*adsp2_sizes);
+}
+
+static bool wm_adsp_validate_version(struct wm_adsp *dsp, unsigned int version)
+{
+ switch (version) {
+ case 0:
+ adsp_warn(dsp, "Deprecated file format %d\n", version);
+ return true;
+ case 1:
+ case 2:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static bool wm_halo_validate_version(struct wm_adsp *dsp, unsigned int version)
+{
+ switch (version) {
+ case 3:
+ return true;
+ default:
+ return false;
+ }
+}
+
static int wm_adsp_load(struct wm_adsp *dsp)
{
LIST_HEAD(buf_list);
unsigned int pos = 0;
const struct wmfw_header *header;
const struct wmfw_adsp1_sizes *adsp1_sizes;
- const struct wmfw_adsp2_sizes *adsp2_sizes;
const struct wmfw_footer *footer;
const struct wmfw_region *region;
const struct wm_adsp_region *mem;
struct wm_adsp_buf *buf;
unsigned int reg;
int regions = 0;
- int ret, offset, type, sizes;
+ int ret, offset, type;
file = kzalloc(PAGE_SIZE, GFP_KERNEL);
if (file == NULL)
goto out_fw;
}
- switch (header->ver) {
- case 0:
- adsp_warn(dsp, "%s: Depreciated file format %d\n",
- file, header->ver);
- break;
- case 1:
- case 2:
- break;
- default:
+ if (!dsp->ops->validate_version(dsp, header->ver)) {
adsp_err(dsp, "%s: unknown file format %d\n",
file, header->ver);
goto out_fw;
goto out_fw;
}
- switch (dsp->type) {
- case WMFW_ADSP1:
- pos = sizeof(*header) + sizeof(*adsp1_sizes) + sizeof(*footer);
- adsp1_sizes = (void *)&(header[1]);
- footer = (void *)&(adsp1_sizes[1]);
- sizes = sizeof(*adsp1_sizes);
+ pos = sizeof(*header);
+ pos = dsp->ops->parse_sizes(dsp, file, pos, firmware);
- adsp_dbg(dsp, "%s: %d DM, %d PM, %d ZM\n",
- file, le32_to_cpu(adsp1_sizes->dm),
- le32_to_cpu(adsp1_sizes->pm),
- le32_to_cpu(adsp1_sizes->zm));
- break;
-
- case WMFW_ADSP2:
- pos = sizeof(*header) + sizeof(*adsp2_sizes) + sizeof(*footer);
- adsp2_sizes = (void *)&(header[1]);
- footer = (void *)&(adsp2_sizes[1]);
- sizes = sizeof(*adsp2_sizes);
-
- adsp_dbg(dsp, "%s: %d XM, %d YM %d PM, %d ZM\n",
- file, le32_to_cpu(adsp2_sizes->xm),
- le32_to_cpu(adsp2_sizes->ym),
- le32_to_cpu(adsp2_sizes->pm),
- le32_to_cpu(adsp2_sizes->zm));
- break;
-
- default:
- WARN(1, "Unknown DSP type");
- goto out_fw;
- }
+ footer = (void *)&firmware->data[pos];
+ pos += sizeof(*footer);
- if (le32_to_cpu(header->len) != sizeof(*header) +
- sizes + sizeof(*footer)) {
+ if (le32_to_cpu(header->len) != pos) {
adsp_err(dsp, "%s: unexpected header length %d\n",
file, le32_to_cpu(header->len));
goto out_fw;
text = NULL;
offset = le32_to_cpu(region->offset) & 0xffffff;
type = be32_to_cpu(region->type) & 0xff;
- mem = wm_adsp_find_region(dsp, type);
switch (type) {
case WMFW_NAME_TEXT:
case WMFW_ADSP2_XM:
case WMFW_ADSP2_YM:
case WMFW_ADSP1_ZM:
+ case WMFW_HALO_PM_PACKED:
+ case WMFW_HALO_XM_PACKED:
+ case WMFW_HALO_YM_PACKED:
+ mem = wm_adsp_find_region(dsp, type);
+ if (!mem) {
+ adsp_err(dsp, "No region of type: %x\n", type);
+ goto out_fw;
+ }
+
region_name = wm_adsp_mem_region_name(type);
- reg = wm_adsp_region_to_reg(mem, offset);
+ reg = dsp->ops->region_to_reg(mem, offset);
break;
default:
adsp_warn(dsp,
}
/* Read the terminator first to validate the length */
- reg = wm_adsp_region_to_reg(mem, pos + len);
+ reg = dsp->ops->region_to_reg(mem, pos + len);
ret = regmap_raw_read(dsp->regmap, reg, &val, sizeof(val));
if (ret != 0) {
if (!alg)
return ERR_PTR(-ENOMEM);
- reg = wm_adsp_region_to_reg(mem, pos);
+ reg = dsp->ops->region_to_reg(mem, pos);
ret = regmap_raw_read(dsp->regmap, reg, alg, len);
if (ret != 0) {
}
}
+static void wmfw_parse_id_header(struct wm_adsp *dsp,
+ struct wmfw_id_hdr *fw, int nalgs)
+{
+ dsp->fw_id = be32_to_cpu(fw->id);
+ dsp->fw_id_version = be32_to_cpu(fw->ver);
+
+ adsp_info(dsp, "Firmware: %x v%d.%d.%d, %d algorithms\n",
+ dsp->fw_id, (dsp->fw_id_version & 0xff0000) >> 16,
+ (dsp->fw_id_version & 0xff00) >> 8, dsp->fw_id_version & 0xff,
+ nalgs);
+}
+
+static void wmfw_v3_parse_id_header(struct wm_adsp *dsp,
+ struct wmfw_v3_id_hdr *fw, int nalgs)
+{
+ dsp->fw_id = be32_to_cpu(fw->id);
+ dsp->fw_id_version = be32_to_cpu(fw->ver);
+ dsp->fw_vendor_id = be32_to_cpu(fw->vendor_id);
+
+ adsp_info(dsp, "Firmware: %x vendor: 0x%x v%d.%d.%d, %d algorithms\n",
+ dsp->fw_id, dsp->fw_vendor_id,
+ (dsp->fw_id_version & 0xff0000) >> 16,
+ (dsp->fw_id_version & 0xff00) >> 8, dsp->fw_id_version & 0xff,
+ nalgs);
+}
+
+static int wm_adsp_create_regions(struct wm_adsp *dsp, __be32 id, int nregions,
+ int *type, __be32 *base)
+{
+ struct wm_adsp_alg_region *alg_region;
+ int i;
+
+ for (i = 0; i < nregions; i++) {
+ alg_region = wm_adsp_create_region(dsp, type[i], id, base[i]);
+ if (IS_ERR(alg_region))
+ return PTR_ERR(alg_region);
+ }
+
+ return 0;
+}
+
static int wm_adsp1_setup_algs(struct wm_adsp *dsp)
{
struct wmfw_adsp1_id_hdr adsp1_id;
}
n_algs = be32_to_cpu(adsp1_id.n_algs);
- dsp->fw_id = be32_to_cpu(adsp1_id.fw.id);
- adsp_info(dsp, "Firmware: %x v%d.%d.%d, %zu algorithms\n",
- dsp->fw_id,
- (be32_to_cpu(adsp1_id.fw.ver) & 0xff0000) >> 16,
- (be32_to_cpu(adsp1_id.fw.ver) & 0xff00) >> 8,
- be32_to_cpu(adsp1_id.fw.ver) & 0xff,
- n_algs);
+
+ wmfw_parse_id_header(dsp, &adsp1_id.fw, n_algs);
alg_region = wm_adsp_create_region(dsp, WMFW_ADSP1_ZM,
adsp1_id.fw.id, adsp1_id.zm);
}
n_algs = be32_to_cpu(adsp2_id.n_algs);
- dsp->fw_id = be32_to_cpu(adsp2_id.fw.id);
- dsp->fw_id_version = be32_to_cpu(adsp2_id.fw.ver);
- adsp_info(dsp, "Firmware: %x v%d.%d.%d, %zu algorithms\n",
- dsp->fw_id,
- (dsp->fw_id_version & 0xff0000) >> 16,
- (dsp->fw_id_version & 0xff00) >> 8,
- dsp->fw_id_version & 0xff,
- n_algs);
+
+ wmfw_parse_id_header(dsp, &adsp2_id.fw, n_algs);
alg_region = wm_adsp_create_region(dsp, WMFW_ADSP2_XM,
adsp2_id.fw.id, adsp2_id.xm);
return ret;
}
+static int wm_halo_create_regions(struct wm_adsp *dsp, __be32 id,
+ __be32 xm_base, __be32 ym_base)
+{
+ int types[] = {
+ WMFW_ADSP2_XM, WMFW_HALO_XM_PACKED,
+ WMFW_ADSP2_YM, WMFW_HALO_YM_PACKED
+ };
+ __be32 bases[] = { xm_base, xm_base, ym_base, ym_base };
+
+ return wm_adsp_create_regions(dsp, id, ARRAY_SIZE(types), types, bases);
+}
+
+static int wm_halo_setup_algs(struct wm_adsp *dsp)
+{
+ struct wmfw_halo_id_hdr halo_id;
+ struct wmfw_halo_alg_hdr *halo_alg;
+ const struct wm_adsp_region *mem;
+ unsigned int pos, len;
+ size_t n_algs;
+ int i, ret;
+
+ mem = wm_adsp_find_region(dsp, WMFW_ADSP2_XM);
+ if (WARN_ON(!mem))
+ return -EINVAL;
+
+ ret = regmap_raw_read(dsp->regmap, mem->base, &halo_id,
+ sizeof(halo_id));
+ if (ret != 0) {
+ adsp_err(dsp, "Failed to read algorithm info: %d\n",
+ ret);
+ return ret;
+ }
+
+ n_algs = be32_to_cpu(halo_id.n_algs);
+
+ wmfw_v3_parse_id_header(dsp, &halo_id.fw, n_algs);
+
+ ret = wm_halo_create_regions(dsp, halo_id.fw.id,
+ halo_id.xm_base, halo_id.ym_base);
+ if (ret)
+ return ret;
+
+ /* Calculate offset and length in DSP words */
+ pos = sizeof(halo_id) / sizeof(u32);
+ len = (sizeof(*halo_alg) * n_algs) / sizeof(u32);
+
+ halo_alg = wm_adsp_read_algs(dsp, n_algs, mem, pos, len);
+ if (IS_ERR(halo_alg))
+ return PTR_ERR(halo_alg);
+
+ for (i = 0; i < n_algs; i++) {
+ adsp_info(dsp,
+ "%d: ID %x v%d.%d.%d XM@%x YM@%x\n",
+ i, be32_to_cpu(halo_alg[i].alg.id),
+ (be32_to_cpu(halo_alg[i].alg.ver) & 0xff0000) >> 16,
+ (be32_to_cpu(halo_alg[i].alg.ver) & 0xff00) >> 8,
+ be32_to_cpu(halo_alg[i].alg.ver) & 0xff,
+ be32_to_cpu(halo_alg[i].xm_base),
+ be32_to_cpu(halo_alg[i].ym_base));
+
+ ret = wm_halo_create_regions(dsp, halo_alg[i].alg.id,
+ halo_alg[i].xm_base,
+ halo_alg[i].ym_base);
+ if (ret)
+ goto out;
+ }
+
+out:
+ kfree(halo_alg);
+ return ret;
+}
+
static int wm_adsp_load_coeff(struct wm_adsp *dsp)
{
LIST_HEAD(buf_list);
adsp_err(dsp, "No ZM\n");
break;
}
- reg = wm_adsp_region_to_reg(mem, 0);
+ reg = dsp->ops->region_to_reg(mem, 0);
} else {
region_name = "register";
case WMFW_ADSP1_ZM:
case WMFW_ADSP2_XM:
case WMFW_ADSP2_YM:
+ case WMFW_HALO_XM_PACKED:
+ case WMFW_HALO_YM_PACKED:
+ case WMFW_HALO_PM_PACKED:
adsp_dbg(dsp, "%s.%d: %d bytes in %x for %x\n",
file, blocks, le32_to_cpu(blk->len),
type, le32_to_cpu(blk->id));
le32_to_cpu(blk->id));
if (alg_region) {
reg = alg_region->base;
- reg = wm_adsp_region_to_reg(mem, reg);
+ reg = dsp->ops->region_to_reg(mem, reg);
reg += offset;
} else {
adsp_err(dsp, "No %x for algorithm %x\n",
int wm_adsp1_init(struct wm_adsp *dsp)
{
+ dsp->ops = &wm_adsp1_ops;
+
return wm_adsp_common_init(dsp);
}
EXPORT_SYMBOL_GPL(wm_adsp1_init);
}
EXPORT_SYMBOL_GPL(wm_adsp1_event);
-static int wm_adsp2_ena(struct wm_adsp *dsp)
+static int wm_adsp2v2_enable_core(struct wm_adsp *dsp)
{
unsigned int val;
int ret, count;
- switch (dsp->rev) {
- case 0:
- ret = regmap_update_bits_async(dsp->regmap,
- dsp->base + ADSP2_CONTROL,
- ADSP2_SYS_ENA, ADSP2_SYS_ENA);
- if (ret != 0)
- return ret;
- break;
- default:
- break;
- }
-
/* Wait for the RAM to start, should be near instantaneous */
for (count = 0; count < 10; ++count) {
ret = regmap_read(dsp->regmap, dsp->base + ADSP2_STATUS1, &val);
return 0;
}
-static void wm_adsp2_boot_work(struct work_struct *work)
+static int wm_adsp2_enable_core(struct wm_adsp *dsp)
+{
+ int ret;
+
+ ret = regmap_update_bits_async(dsp->regmap, dsp->base + ADSP2_CONTROL,
+ ADSP2_SYS_ENA, ADSP2_SYS_ENA);
+ if (ret != 0)
+ return ret;
+
+ return wm_adsp2v2_enable_core(dsp);
+}
+
+static int wm_adsp2_lock(struct wm_adsp *dsp, unsigned int lock_regions)
+{
+ struct regmap *regmap = dsp->regmap;
+ unsigned int code0, code1, lock_reg;
+
+ if (!(lock_regions & WM_ADSP2_REGION_ALL))
+ return 0;
+
+ lock_regions &= WM_ADSP2_REGION_ALL;
+ lock_reg = dsp->base + ADSP2_LOCK_REGION_1_LOCK_REGION_0;
+
+ while (lock_regions) {
+ code0 = code1 = 0;
+ if (lock_regions & BIT(0)) {
+ code0 = ADSP2_LOCK_CODE_0;
+ code1 = ADSP2_LOCK_CODE_1;
+ }
+ if (lock_regions & BIT(1)) {
+ code0 |= ADSP2_LOCK_CODE_0 << ADSP2_LOCK_REGION_SHIFT;
+ code1 |= ADSP2_LOCK_CODE_1 << ADSP2_LOCK_REGION_SHIFT;
+ }
+ regmap_write(regmap, lock_reg, code0);
+ regmap_write(regmap, lock_reg, code1);
+ lock_regions >>= 2;
+ lock_reg += 2;
+ }
+
+ return 0;
+}
+
+static int wm_adsp2_enable_memory(struct wm_adsp *dsp)
+{
+ return regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
+ ADSP2_MEM_ENA, ADSP2_MEM_ENA);
+}
+
+static void wm_adsp2_disable_memory(struct wm_adsp *dsp)
+{
+ regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
+ ADSP2_MEM_ENA, 0);
+}
+
+static void wm_adsp2_disable_core(struct wm_adsp *dsp)
+{
+ regmap_write(dsp->regmap, dsp->base + ADSP2_RDMA_CONFIG_1, 0);
+ regmap_write(dsp->regmap, dsp->base + ADSP2_WDMA_CONFIG_1, 0);
+ regmap_write(dsp->regmap, dsp->base + ADSP2_WDMA_CONFIG_2, 0);
+
+ regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
+ ADSP2_SYS_ENA, 0);
+}
+
+static void wm_adsp2v2_disable_core(struct wm_adsp *dsp)
+{
+ regmap_write(dsp->regmap, dsp->base + ADSP2_RDMA_CONFIG_1, 0);
+ regmap_write(dsp->regmap, dsp->base + ADSP2_WDMA_CONFIG_1, 0);
+ regmap_write(dsp->regmap, dsp->base + ADSP2V2_WDMA_CONFIG_2, 0);
+}
+
+static void wm_adsp_boot_work(struct work_struct *work)
{
struct wm_adsp *dsp = container_of(work,
struct wm_adsp,
mutex_lock(&dsp->pwr_lock);
- ret = regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
- ADSP2_MEM_ENA, ADSP2_MEM_ENA);
- if (ret != 0)
- goto err_mutex;
+ if (dsp->ops->enable_memory) {
+ ret = dsp->ops->enable_memory(dsp);
+ if (ret != 0)
+ goto err_mutex;
+ }
- ret = wm_adsp2_ena(dsp);
- if (ret != 0)
- goto err_mem;
+ if (dsp->ops->enable_core) {
+ ret = dsp->ops->enable_core(dsp);
+ if (ret != 0)
+ goto err_mem;
+ }
ret = wm_adsp_load(dsp);
if (ret != 0)
goto err_ena;
- ret = wm_adsp2_setup_algs(dsp);
+ ret = dsp->ops->setup_algs(dsp);
if (ret != 0)
goto err_ena;
if (ret != 0)
goto err_ena;
- switch (dsp->rev) {
- case 0:
- /* Turn DSP back off until we are ready to run */
- ret = regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
- ADSP2_SYS_ENA, 0);
- if (ret != 0)
- goto err_ena;
- break;
- default:
- break;
- }
+ if (dsp->ops->disable_core)
+ dsp->ops->disable_core(dsp);
dsp->booted = true;
return;
err_ena:
- regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
- ADSP2_SYS_ENA | ADSP2_CORE_ENA | ADSP2_START, 0);
+ if (dsp->ops->disable_core)
+ dsp->ops->disable_core(dsp);
err_mem:
- regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
- ADSP2_MEM_ENA, 0);
+ if (dsp->ops->disable_memory)
+ dsp->ops->disable_memory(dsp);
err_mutex:
mutex_unlock(&dsp->pwr_lock);
}
-static void wm_adsp2_set_dspclk(struct wm_adsp *dsp, unsigned int freq)
+static int wm_halo_configure_mpu(struct wm_adsp *dsp, unsigned int lock_regions)
+{
+ struct reg_sequence config[] = {
+ { dsp->base + HALO_MPU_LOCK_CONFIG, 0x5555 },
+ { dsp->base + HALO_MPU_LOCK_CONFIG, 0xAAAA },
+ { dsp->base + HALO_MPU_XMEM_ACCESS_0, 0xFFFFFFFF },
+ { dsp->base + HALO_MPU_YMEM_ACCESS_0, 0xFFFFFFFF },
+ { dsp->base + HALO_MPU_WINDOW_ACCESS_0, lock_regions },
+ { dsp->base + HALO_MPU_XREG_ACCESS_0, lock_regions },
+ { dsp->base + HALO_MPU_YREG_ACCESS_0, lock_regions },
+ { dsp->base + HALO_MPU_XMEM_ACCESS_1, 0xFFFFFFFF },
+ { dsp->base + HALO_MPU_YMEM_ACCESS_1, 0xFFFFFFFF },
+ { dsp->base + HALO_MPU_WINDOW_ACCESS_1, lock_regions },
+ { dsp->base + HALO_MPU_XREG_ACCESS_1, lock_regions },
+ { dsp->base + HALO_MPU_YREG_ACCESS_1, lock_regions },
+ { dsp->base + HALO_MPU_XMEM_ACCESS_2, 0xFFFFFFFF },
+ { dsp->base + HALO_MPU_YMEM_ACCESS_2, 0xFFFFFFFF },
+ { dsp->base + HALO_MPU_WINDOW_ACCESS_2, lock_regions },
+ { dsp->base + HALO_MPU_XREG_ACCESS_2, lock_regions },
+ { dsp->base + HALO_MPU_YREG_ACCESS_2, lock_regions },
+ { dsp->base + HALO_MPU_XMEM_ACCESS_3, 0xFFFFFFFF },
+ { dsp->base + HALO_MPU_YMEM_ACCESS_3, 0xFFFFFFFF },
+ { dsp->base + HALO_MPU_WINDOW_ACCESS_3, lock_regions },
+ { dsp->base + HALO_MPU_XREG_ACCESS_3, lock_regions },
+ { dsp->base + HALO_MPU_YREG_ACCESS_3, lock_regions },
+ { dsp->base + HALO_MPU_LOCK_CONFIG, 0 },
+ };
+
+ return regmap_multi_reg_write(dsp->regmap, config, ARRAY_SIZE(config));
+}
+
+int wm_adsp2_set_dspclk(struct snd_soc_dapm_widget *w, unsigned int freq)
{
+ struct snd_soc_component *component = snd_soc_dapm_to_component(w->dapm);
+ struct wm_adsp *dsps = snd_soc_component_get_drvdata(component);
+ struct wm_adsp *dsp = &dsps[w->shift];
int ret;
- switch (dsp->rev) {
- case 0:
- ret = regmap_update_bits_async(dsp->regmap,
- dsp->base + ADSP2_CLOCKING,
- ADSP2_CLK_SEL_MASK,
- freq << ADSP2_CLK_SEL_SHIFT);
- if (ret) {
- adsp_err(dsp, "Failed to set clock rate: %d\n", ret);
- return;
- }
- break;
- default:
- /* clock is handled by parent codec driver */
- break;
- }
+ ret = regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CLOCKING,
+ ADSP2_CLK_SEL_MASK,
+ freq << ADSP2_CLK_SEL_SHIFT);
+ if (ret)
+ adsp_err(dsp, "Failed to set clock rate: %d\n", ret);
+
+ return ret;
}
+EXPORT_SYMBOL_GPL(wm_adsp2_set_dspclk);
int wm_adsp2_preloader_get(struct snd_kcontrol *kcontrol,
struct snd_ctl_elem_value *ucontrol)
static void wm_adsp_stop_watchdog(struct wm_adsp *dsp)
{
- switch (dsp->rev) {
- case 0:
- case 1:
- return;
- default:
- regmap_update_bits(dsp->regmap, dsp->base + ADSP2_WATCHDOG,
- ADSP2_WDT_ENA_MASK, 0);
- }
+ regmap_update_bits(dsp->regmap, dsp->base + ADSP2_WATCHDOG,
+ ADSP2_WDT_ENA_MASK, 0);
}
-int wm_adsp2_early_event(struct snd_soc_dapm_widget *w,
- struct snd_kcontrol *kcontrol, int event,
- unsigned int freq)
+static void wm_halo_stop_watchdog(struct wm_adsp *dsp)
+{
+ regmap_update_bits(dsp->regmap, dsp->base + HALO_WDT_CONTROL,
+ HALO_WDT_EN_MASK, 0);
+}
+
+int wm_adsp_early_event(struct snd_soc_dapm_widget *w,
+ struct snd_kcontrol *kcontrol, int event)
{
struct snd_soc_component *component = snd_soc_dapm_to_component(w->dapm);
struct wm_adsp *dsps = snd_soc_component_get_drvdata(component);
switch (event) {
case SND_SOC_DAPM_PRE_PMU:
- wm_adsp2_set_dspclk(dsp, freq);
queue_work(system_unbound_wq, &dsp->boot_work);
break;
case SND_SOC_DAPM_PRE_PMD:
dsp->booted = false;
- regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
- ADSP2_MEM_ENA, 0);
+ if (dsp->ops->disable_memory)
+ dsp->ops->disable_memory(dsp);
list_for_each_entry(ctl, &dsp->ctl_list, list)
ctl->enabled = 0;
return 0;
}
-EXPORT_SYMBOL_GPL(wm_adsp2_early_event);
+EXPORT_SYMBOL_GPL(wm_adsp_early_event);
+
+static int wm_adsp2_start_core(struct wm_adsp *dsp)
+{
+ return regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
+ ADSP2_CORE_ENA | ADSP2_START,
+ ADSP2_CORE_ENA | ADSP2_START);
+}
-int wm_adsp2_event(struct snd_soc_dapm_widget *w,
- struct snd_kcontrol *kcontrol, int event)
+static void wm_adsp2_stop_core(struct wm_adsp *dsp)
+{
+ regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
+ ADSP2_CORE_ENA | ADSP2_START, 0);
+}
+
+int wm_adsp_event(struct snd_soc_dapm_widget *w,
+ struct snd_kcontrol *kcontrol, int event)
{
struct snd_soc_component *component = snd_soc_dapm_to_component(w->dapm);
struct wm_adsp *dsps = snd_soc_component_get_drvdata(component);
goto err;
}
- ret = wm_adsp2_ena(dsp);
- if (ret != 0)
- goto err;
+ if (dsp->ops->enable_core) {
+ ret = dsp->ops->enable_core(dsp);
+ if (ret != 0)
+ goto err;
+ }
/* Sync set controls */
ret = wm_coeff_sync_controls(dsp);
if (ret != 0)
goto err;
- wm_adsp2_lock(dsp, dsp->lock_regions);
+ if (dsp->ops->lock_memory) {
+ ret = dsp->ops->lock_memory(dsp, dsp->lock_regions);
+ if (ret != 0) {
+ adsp_err(dsp, "Error configuring MPU: %d\n",
+ ret);
+ goto err;
+ }
+ }
- ret = regmap_update_bits(dsp->regmap,
- dsp->base + ADSP2_CONTROL,
- ADSP2_CORE_ENA | ADSP2_START,
- ADSP2_CORE_ENA | ADSP2_START);
- if (ret != 0)
- goto err;
+ if (dsp->ops->start_core) {
+ ret = dsp->ops->start_core(dsp);
+ if (ret != 0)
+ goto err;
+ }
if (wm_adsp_fw[dsp->fw].num_caps != 0) {
ret = wm_adsp_buffer_init(dsp);
dsp->running = true;
mutex_unlock(&dsp->pwr_lock);
-
break;
case SND_SOC_DAPM_PRE_PMD:
/* Tell the firmware to cleanup */
wm_adsp_signal_event_controls(dsp, WM_ADSP_FW_EVENT_SHUTDOWN);
- wm_adsp_stop_watchdog(dsp);
+ if (dsp->ops->stop_watchdog)
+ dsp->ops->stop_watchdog(dsp);
/* Log firmware state, it can be useful for analysis */
- switch (dsp->rev) {
- case 0:
- wm_adsp2_show_fw_status(dsp);
- break;
- default:
- wm_adsp2v2_show_fw_status(dsp);
- break;
- }
+ if (dsp->ops->show_fw_status)
+ dsp->ops->show_fw_status(dsp);
mutex_lock(&dsp->pwr_lock);
dsp->running = false;
- regmap_update_bits(dsp->regmap,
- dsp->base + ADSP2_CONTROL,
- ADSP2_CORE_ENA | ADSP2_START, 0);
-
- /* Make sure DMAs are quiesced */
- switch (dsp->rev) {
- case 0:
- regmap_write(dsp->regmap,
- dsp->base + ADSP2_RDMA_CONFIG_1, 0);
- regmap_write(dsp->regmap,
- dsp->base + ADSP2_WDMA_CONFIG_1, 0);
- regmap_write(dsp->regmap,
- dsp->base + ADSP2_WDMA_CONFIG_2, 0);
-
- regmap_update_bits(dsp->regmap,
- dsp->base + ADSP2_CONTROL,
- ADSP2_SYS_ENA, 0);
- break;
- default:
- regmap_write(dsp->regmap,
- dsp->base + ADSP2_RDMA_CONFIG_1, 0);
- regmap_write(dsp->regmap,
- dsp->base + ADSP2_WDMA_CONFIG_1, 0);
- regmap_write(dsp->regmap,
- dsp->base + ADSP2V2_WDMA_CONFIG_2, 0);
- break;
- }
+ if (dsp->ops->stop_core)
+ dsp->ops->stop_core(dsp);
+ if (dsp->ops->disable_core)
+ dsp->ops->disable_core(dsp);
if (wm_adsp_fw[dsp->fw].num_caps != 0)
wm_adsp_buffer_free(dsp);
return 0;
err:
- regmap_update_bits(dsp->regmap, dsp->base + ADSP2_CONTROL,
- ADSP2_SYS_ENA | ADSP2_CORE_ENA | ADSP2_START, 0);
+ if (dsp->ops->stop_core)
+ dsp->ops->stop_core(dsp);
+ if (dsp->ops->disable_core)
+ dsp->ops->disable_core(dsp);
mutex_unlock(&dsp->pwr_lock);
return ret;
}
-EXPORT_SYMBOL_GPL(wm_adsp2_event);
+EXPORT_SYMBOL_GPL(wm_adsp_event);
+
+static int wm_halo_start_core(struct wm_adsp *dsp)
+{
+ return regmap_update_bits(dsp->regmap,
+ dsp->base + HALO_CCM_CORE_CONTROL,
+ HALO_CORE_EN, HALO_CORE_EN);
+}
+
+static void wm_halo_stop_core(struct wm_adsp *dsp)
+{
+ regmap_update_bits(dsp->regmap, dsp->base + HALO_CCM_CORE_CONTROL,
+ HALO_CORE_EN, 0);
+
+ /* reset halo core with CORE_SOFT_RESET */
+ regmap_update_bits(dsp->regmap, dsp->base + HALO_CORE_SOFT_RESET,
+ HALO_CORE_SOFT_RESET_MASK, 1);
+}
int wm_adsp2_component_probe(struct wm_adsp *dsp, struct snd_soc_component *component)
{
"Failed to clear memory retention: %d\n", ret);
return ret;
}
+
+ dsp->ops = &wm_adsp2_ops[0];
+ break;
+ case 1:
+ dsp->ops = &wm_adsp2_ops[1];
break;
default:
+ dsp->ops = &wm_adsp2_ops[2];
break;
}
- INIT_WORK(&dsp->boot_work, wm_adsp2_boot_work);
+ INIT_WORK(&dsp->boot_work, wm_adsp_boot_work);
return 0;
}
EXPORT_SYMBOL_GPL(wm_adsp2_init);
+int wm_halo_init(struct wm_adsp *dsp)
+{
+ int ret;
+
+ ret = wm_adsp_common_init(dsp);
+ if (ret)
+ return ret;
+
+ dsp->ops = &wm_halo_ops;
+
+ INIT_WORK(&dsp->boot_work, wm_adsp_boot_work);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(wm_halo_init);
+
void wm_adsp2_remove(struct wm_adsp *dsp)
{
struct wm_coeff_ctl *ctl;
return -EINVAL;
compr->buf = buf;
- compr->buf->compr = compr;
+ buf->compr = compr;
return 0;
}
if (!mem)
return -EINVAL;
- reg = wm_adsp_region_to_reg(mem, mem_addr);
+ reg = dsp->ops->region_to_reg(mem, mem_addr);
ret = regmap_raw_read(dsp->regmap, reg, data,
sizeof(*data) * num_words);
if (!mem)
return -EINVAL;
- reg = wm_adsp_region_to_reg(mem, mem_addr);
+ reg = dsp->ops->region_to_reg(mem, mem_addr);
data = cpu_to_be32(data & 0x00ffffffu);
return -ENOMEM;
alg_region = wm_adsp_find_alg_region(dsp, WMFW_ADSP2_XM, dsp->fw_id);
- xmalg = sizeof(struct wm_adsp_system_config_xm_hdr) / sizeof(__be32);
+ xmalg = dsp->ops->sys_config_size / sizeof(__be32);
addr = alg_region->base + xmalg + ALG_XM_FIELD(magic);
ret = wm_adsp_read_data_word(dsp, WMFW_ADSP2_XM, addr, &magic);
struct wm_adsp_compr_buf *buf, *tmp;
list_for_each_entry_safe(buf, tmp, &dsp->buffer_list, list) {
- if (buf->compr)
- wm_adsp_compr_detach(buf->compr);
+ wm_adsp_compr_detach(buf->compr);
kfree(buf->name);
kfree(buf->regions);
buf = compr->buf;
- if (!compr->buf || compr->buf->error) {
+ if (dsp->fatal_error || !buf || buf->error) {
snd_compr_stop_error(stream, SNDRV_PCM_STATE_XRUN);
ret = -EIO;
goto out;
if (buf->avail < wm_adsp_compr_frag_words(compr)) {
ret = wm_adsp_buffer_get_error(buf);
if (ret < 0) {
- if (compr->buf->error)
+ if (buf->error)
snd_compr_stop_error(stream,
SNDRV_PCM_STATE_XRUN);
goto out;
static int wm_adsp_compr_read(struct wm_adsp_compr *compr,
char __user *buf, size_t count)
{
+ struct wm_adsp *dsp = compr->dsp;
int ntotal = 0;
int nwords, nbytes;
compr_dbg(compr, "Requested read of %zu bytes\n", count);
- if (!compr->buf || compr->buf->error) {
+ if (dsp->fatal_error || !compr->buf || compr->buf->error) {
snd_compr_stop_error(compr->stream, SNDRV_PCM_STATE_XRUN);
return -EIO;
}
}
EXPORT_SYMBOL_GPL(wm_adsp_compr_copy);
-int wm_adsp2_lock(struct wm_adsp *dsp, unsigned int lock_regions)
-{
- struct regmap *regmap = dsp->regmap;
- unsigned int code0, code1, lock_reg;
-
- if (!(lock_regions & WM_ADSP2_REGION_ALL))
- return 0;
-
- lock_regions &= WM_ADSP2_REGION_ALL;
- lock_reg = dsp->base + ADSP2_LOCK_REGION_1_LOCK_REGION_0;
-
- while (lock_regions) {
- code0 = code1 = 0;
- if (lock_regions & BIT(0)) {
- code0 = ADSP2_LOCK_CODE_0;
- code1 = ADSP2_LOCK_CODE_1;
- }
- if (lock_regions & BIT(1)) {
- code0 |= ADSP2_LOCK_CODE_0 << ADSP2_LOCK_REGION_SHIFT;
- code1 |= ADSP2_LOCK_CODE_1 << ADSP2_LOCK_REGION_SHIFT;
- }
- regmap_write(regmap, lock_reg, code0);
- regmap_write(regmap, lock_reg, code1);
- lock_regions >>= 2;
- lock_reg += 2;
- }
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(wm_adsp2_lock);
-
static void wm_adsp_fatal_error(struct wm_adsp *dsp)
{
struct wm_adsp_compr *compr;
dsp->fatal_error = true;
list_for_each_entry(compr, &dsp->compr_list, list) {
- if (compr->stream) {
- snd_compr_stop_error(compr->stream,
- SNDRV_PCM_STATE_XRUN);
+ if (compr->stream)
snd_compr_fragment_elapsed(compr->stream);
- }
}
}
if (val & ADSP2_WDT_TIMEOUT_STS_MASK) {
adsp_err(dsp, "watchdog timeout error\n");
- wm_adsp_stop_watchdog(dsp);
+ dsp->ops->stop_watchdog(dsp);
wm_adsp_fatal_error(dsp);
}
}
EXPORT_SYMBOL_GPL(wm_adsp2_bus_error);
+irqreturn_t wm_halo_bus_error(struct wm_adsp *dsp)
+{
+ struct regmap *regmap = dsp->regmap;
+ unsigned int fault[6];
+ struct reg_sequence clear[] = {
+ { dsp->base + HALO_MPU_XM_VIO_STATUS, 0x0 },
+ { dsp->base + HALO_MPU_YM_VIO_STATUS, 0x0 },
+ { dsp->base + HALO_MPU_PM_VIO_STATUS, 0x0 },
+ };
+ int ret;
+
+ mutex_lock(&dsp->pwr_lock);
+
+ ret = regmap_read(regmap, dsp->base_sysinfo + HALO_AHBM_WINDOW_DEBUG_1,
+ fault);
+ if (ret) {
+ adsp_warn(dsp, "Failed to read AHB DEBUG_1: %d\n", ret);
+ goto exit_unlock;
+ }
+
+ adsp_warn(dsp, "AHB: STATUS: 0x%x ADDR: 0x%x\n",
+ *fault & HALO_AHBM_FLAGS_ERR_MASK,
+ (*fault & HALO_AHBM_CORE_ERR_ADDR_MASK) >>
+ HALO_AHBM_CORE_ERR_ADDR_SHIFT);
+
+ ret = regmap_read(regmap, dsp->base_sysinfo + HALO_AHBM_WINDOW_DEBUG_0,
+ fault);
+ if (ret) {
+ adsp_warn(dsp, "Failed to read AHB DEBUG_0: %d\n", ret);
+ goto exit_unlock;
+ }
+
+ adsp_warn(dsp, "AHB: SYS_ADDR: 0x%x\n", *fault);
+
+ ret = regmap_bulk_read(regmap, dsp->base + HALO_MPU_XM_VIO_ADDR,
+ fault, ARRAY_SIZE(fault));
+ if (ret) {
+ adsp_warn(dsp, "Failed to read MPU fault info: %d\n", ret);
+ goto exit_unlock;
+ }
+
+ adsp_warn(dsp, "XM: STATUS:0x%x ADDR:0x%x\n", fault[1], fault[0]);
+ adsp_warn(dsp, "YM: STATUS:0x%x ADDR:0x%x\n", fault[3], fault[2]);
+ adsp_warn(dsp, "PM: STATUS:0x%x ADDR:0x%x\n", fault[5], fault[4]);
+
+ ret = regmap_multi_reg_write(dsp->regmap, clear, ARRAY_SIZE(clear));
+ if (ret)
+ adsp_warn(dsp, "Failed to clear MPU status: %d\n", ret);
+
+exit_unlock:
+ mutex_unlock(&dsp->pwr_lock);
+
+ return IRQ_HANDLED;
+}
+EXPORT_SYMBOL_GPL(wm_halo_bus_error);
+
+irqreturn_t wm_halo_wdt_expire(int irq, void *data)
+{
+ struct wm_adsp *dsp = data;
+
+ mutex_lock(&dsp->pwr_lock);
+
+ adsp_warn(dsp, "WDT Expiry Fault\n");
+ dsp->ops->stop_watchdog(dsp);
+ wm_adsp_fatal_error(dsp);
+
+ mutex_unlock(&dsp->pwr_lock);
+
+ return IRQ_HANDLED;
+}
+EXPORT_SYMBOL_GPL(wm_halo_wdt_expire);
+
+static struct wm_adsp_ops wm_adsp1_ops = {
+ .validate_version = wm_adsp_validate_version,
+ .parse_sizes = wm_adsp1_parse_sizes,
+ .region_to_reg = wm_adsp_region_to_reg,
+};
+
+static struct wm_adsp_ops wm_adsp2_ops[] = {
+ {
+ .sys_config_size = sizeof(struct wm_adsp_system_config_xm_hdr),
+ .parse_sizes = wm_adsp2_parse_sizes,
+ .validate_version = wm_adsp_validate_version,
+ .setup_algs = wm_adsp2_setup_algs,
+ .region_to_reg = wm_adsp_region_to_reg,
+
+ .show_fw_status = wm_adsp2_show_fw_status,
+
+ .enable_memory = wm_adsp2_enable_memory,
+ .disable_memory = wm_adsp2_disable_memory,
+
+ .enable_core = wm_adsp2_enable_core,
+ .disable_core = wm_adsp2_disable_core,
+
+ .start_core = wm_adsp2_start_core,
+ .stop_core = wm_adsp2_stop_core,
+
+ },
+ {
+ .sys_config_size = sizeof(struct wm_adsp_system_config_xm_hdr),
+ .parse_sizes = wm_adsp2_parse_sizes,
+ .validate_version = wm_adsp_validate_version,
+ .setup_algs = wm_adsp2_setup_algs,
+ .region_to_reg = wm_adsp_region_to_reg,
+
+ .show_fw_status = wm_adsp2v2_show_fw_status,
+
+ .enable_memory = wm_adsp2_enable_memory,
+ .disable_memory = wm_adsp2_disable_memory,
+ .lock_memory = wm_adsp2_lock,
+
+ .enable_core = wm_adsp2v2_enable_core,
+ .disable_core = wm_adsp2v2_disable_core,
+
+ .start_core = wm_adsp2_start_core,
+ .stop_core = wm_adsp2_stop_core,
+ },
+ {
+ .sys_config_size = sizeof(struct wm_adsp_system_config_xm_hdr),
+ .parse_sizes = wm_adsp2_parse_sizes,
+ .validate_version = wm_adsp_validate_version,
+ .setup_algs = wm_adsp2_setup_algs,
+ .region_to_reg = wm_adsp_region_to_reg,
+
+ .show_fw_status = wm_adsp2v2_show_fw_status,
+ .stop_watchdog = wm_adsp_stop_watchdog,
+
+ .enable_memory = wm_adsp2_enable_memory,
+ .disable_memory = wm_adsp2_disable_memory,
+ .lock_memory = wm_adsp2_lock,
+
+ .enable_core = wm_adsp2v2_enable_core,
+ .disable_core = wm_adsp2v2_disable_core,
+
+ .start_core = wm_adsp2_start_core,
+ .stop_core = wm_adsp2_stop_core,
+ },
+};
+
+static struct wm_adsp_ops wm_halo_ops = {
+ .sys_config_size = sizeof(struct wm_halo_system_config_xm_hdr),
+ .parse_sizes = wm_adsp2_parse_sizes,
+ .validate_version = wm_halo_validate_version,
+ .setup_algs = wm_halo_setup_algs,
+ .region_to_reg = wm_halo_region_to_reg,
+
+ .show_fw_status = wm_halo_show_fw_status,
+ .stop_watchdog = wm_halo_stop_watchdog,
+
+ .lock_memory = wm_halo_configure_mpu,
+
+ .start_core = wm_halo_start_core,
+ .stop_core = wm_halo_stop_core,
+};
+
MODULE_LICENSE("GPL v2");
struct wm_adsp_compr;
struct wm_adsp_compr_buf;
+struct wm_adsp_ops;
struct wm_adsp {
const char *part;
struct regmap *regmap;
struct snd_soc_component *component;
+ struct wm_adsp_ops *ops;
+
unsigned int base;
+ unsigned int base_sysinfo;
unsigned int sysclk_reg;
unsigned int sysclk_mask;
unsigned int sysclk_shift;
unsigned int fw_id;
unsigned int fw_id_version;
+ unsigned int fw_vendor_id;
const struct wm_adsp_region *mem;
int num_mems;
};
+struct wm_adsp_ops {
+ unsigned int sys_config_size;
+
+ bool (*validate_version)(struct wm_adsp *dsp, unsigned int version);
+ unsigned int (*parse_sizes)(struct wm_adsp *dsp,
+ const char * const file,
+ unsigned int pos,
+ const struct firmware *firmware);
+ int (*setup_algs)(struct wm_adsp *dsp);
+ unsigned int (*region_to_reg)(struct wm_adsp_region const *mem,
+ unsigned int offset);
+
+ void (*show_fw_status)(struct wm_adsp *dsp);
+ void (*stop_watchdog)(struct wm_adsp *dsp);
+
+ int (*enable_memory)(struct wm_adsp *dsp);
+ void (*disable_memory)(struct wm_adsp *dsp);
+ int (*lock_memory)(struct wm_adsp *dsp, unsigned int lock_regions);
+
+ int (*enable_core)(struct wm_adsp *dsp);
+ void (*disable_core)(struct wm_adsp *dsp);
+
+ int (*start_core)(struct wm_adsp *dsp);
+ void (*stop_core)(struct wm_adsp *dsp);
+};
+
#define WM_ADSP1(wname, num) \
SND_SOC_DAPM_PGA_E(wname, SND_SOC_NOPM, num, 0, NULL, 0, \
wm_adsp1_event, SND_SOC_DAPM_POST_PMU | SND_SOC_DAPM_PRE_PMD)
.event_flags = SND_SOC_DAPM_PRE_PMU | SND_SOC_DAPM_PRE_PMD, \
.subseq = 100, /* Ensure we run after SYSCLK supply widget */ }, \
{ .id = snd_soc_dapm_out_drv, .name = wname, \
- .reg = SND_SOC_NOPM, .shift = num, .event = wm_adsp2_event, \
+ .reg = SND_SOC_NOPM, .shift = num, .event = wm_adsp_event, \
.event_flags = SND_SOC_DAPM_POST_PMU | SND_SOC_DAPM_PRE_PMD }
#define WM_ADSP_FW_CONTROL(dspname, num) \
void wm_adsp2_remove(struct wm_adsp *dsp);
int wm_adsp2_component_probe(struct wm_adsp *dsp, struct snd_soc_component *component);
int wm_adsp2_component_remove(struct wm_adsp *dsp, struct snd_soc_component *component);
+int wm_halo_init(struct wm_adsp *dsp);
+
int wm_adsp1_event(struct snd_soc_dapm_widget *w,
struct snd_kcontrol *kcontrol, int event);
-int wm_adsp2_early_event(struct snd_soc_dapm_widget *w,
- struct snd_kcontrol *kcontrol, int event,
- unsigned int freq);
-int wm_adsp2_lock(struct wm_adsp *adsp, unsigned int regions);
+int wm_adsp_early_event(struct snd_soc_dapm_widget *w,
+ struct snd_kcontrol *kcontrol, int event);
+
irqreturn_t wm_adsp2_bus_error(struct wm_adsp *adsp);
+irqreturn_t wm_halo_bus_error(struct wm_adsp *dsp);
+irqreturn_t wm_halo_wdt_expire(int irq, void *data);
-int wm_adsp2_event(struct snd_soc_dapm_widget *w,
- struct snd_kcontrol *kcontrol, int event);
+int wm_adsp_event(struct snd_soc_dapm_widget *w,
+ struct snd_kcontrol *kcontrol, int event);
+
+int wm_adsp2_set_dspclk(struct snd_soc_dapm_widget *w, unsigned int freq);
int wm_adsp2_preloader_get(struct snd_kcontrol *kcontrol,
struct snd_ctl_elem_value *ucontrol);
__be32 ver;
} __packed;
+struct wmfw_v3_id_hdr {
+ __be32 core_id;
+ __be32 block_rev;
+ __be32 vendor_id;
+ __be32 id;
+ __be32 ver;
+} __packed;
+
struct wmfw_adsp1_id_hdr {
struct wmfw_id_hdr fw;
__be32 zm;
__be32 n_algs;
} __packed;
+struct wmfw_halo_id_hdr {
+ struct wmfw_v3_id_hdr fw;
+ __be32 xm_base;
+ __be32 xm_size;
+ __be32 ym_base;
+ __be32 ym_size;
+ __be32 n_algs;
+} __packed;
+
struct wmfw_alg_hdr {
__be32 id;
__be32 ver;
__be32 ym;
} __packed;
+struct wmfw_halo_alg_hdr {
+ struct wmfw_alg_hdr alg;
+ __be32 xm_base;
+ __be32 xm_size;
+ __be32 ym_base;
+ __be32 ym_size;
+} __packed;
+
struct wmfw_adsp_alg_data {
__le32 id;
u8 name[WMFW_MAX_ALG_NAME];
#define WMFW_ADSP1 1
#define WMFW_ADSP2 2
+#define WMFW_HALO 4
#define WMFW_ABSOLUTE 0xf0
#define WMFW_ALGORITHM_DATA 0xf2
#define WMFW_ADSP2_XM 5
#define WMFW_ADSP2_YM 6
+#define WMFW_HALO_PM_PACKED 0x10
+#define WMFW_HALO_XM_PACKED 0x11
+#define WMFW_HALO_YM_PACKED 0x12
+
#endif
This option is only useful for out-of-tree drivers since
in-tree drivers select it automatically.
+config SND_SOC_FSL_AUDMIX
+ tristate "Audio Mixer (AUDMIX) module support"
+ select REGMAP_MMIO
+ help
+ Say Y if you want to add Audio Mixer (AUDMIX)
+ support for the NXP iMX CPUs.
+
config SND_SOC_FSL_SSI
tristate "Synchronous Serial Interface module (SSI) support"
select SND_SOC_IMX_PCM_DMA if SND_IMX_SOC != n
endif # SND_POWERPC_SOC
+config SND_SOC_IMX_PCM_FIQ
+ tristate
+ default y if SND_SOC_IMX_SSI=y && (SND_SOC_FSL_SSI=m || SND_SOC_FSL_SPDIF=m) && (MXC_TZIC || MXC_AVIC)
+ select FIQ
+
if SND_IMX_SOC
config SND_SOC_IMX_SSI
tristate
select SND_SOC_FSL_UTILS
-config SND_SOC_IMX_PCM_FIQ
- tristate
- select FIQ
-
comment "SoC Audio support for Freescale i.MX boards:"
config SND_MXC_SOC_WM1133_EV1
CS4271, CS4272 and SGTL5000.
Say Y if you want to add support for Freescale Generic ASoC Sound Card.
+config SND_SOC_IMX_AUDMIX
+ tristate "SoC Audio support for i.MX boards with AUDMIX"
+ select SND_SOC_FSL_AUDMIX
+ select SND_SOC_FSL_SAI
+ help
+ SoC Audio support for i.MX boards with Audio Mixer
+ Say Y if you want to add support for SoC audio on an i.MX board with
+ an Audio Mixer.
+
endif # SND_IMX_SOC
endmenu
obj-$(CONFIG_SND_SOC_P1022_RDK) += snd-soc-p1022-rdk.o
# Freescale SSI/DMA/SAI/SPDIF Support
+snd-soc-fsl-audmix-objs := fsl_audmix.o
snd-soc-fsl-asoc-card-objs := fsl-asoc-card.o
snd-soc-fsl-asrc-objs := fsl_asrc.o fsl_asrc_dma.o
snd-soc-fsl-sai-objs := fsl_sai.o
snd-soc-fsl-micfil-objs := fsl_micfil.o
snd-soc-fsl-utils-objs := fsl_utils.o
snd-soc-fsl-dma-objs := fsl_dma.o
+
+obj-$(CONFIG_SND_SOC_FSL_AUDMIX) += snd-soc-fsl-audmix.o
obj-$(CONFIG_SND_SOC_FSL_ASOC_CARD) += snd-soc-fsl-asoc-card.o
obj-$(CONFIG_SND_SOC_FSL_ASRC) += snd-soc-fsl-asrc.o
obj-$(CONFIG_SND_SOC_FSL_SAI) += snd-soc-fsl-sai.o
snd-soc-imx-sgtl5000-objs := imx-sgtl5000.o
snd-soc-imx-spdif-objs := imx-spdif.o
snd-soc-imx-mc13783-objs := imx-mc13783.o
+snd-soc-imx-audmix-objs := imx-audmix.o
obj-$(CONFIG_SND_SOC_EUKREA_TLV320) += snd-soc-eukrea-tlv320.o
obj-$(CONFIG_SND_SOC_PHYCORE_AC97) += snd-soc-phycore-ac97.o
obj-$(CONFIG_SND_SOC_IMX_SGTL5000) += snd-soc-imx-sgtl5000.o
obj-$(CONFIG_SND_SOC_IMX_SPDIF) += snd-soc-imx-spdif.o
obj-$(CONFIG_SND_SOC_IMX_MC13783) += snd-soc-imx-mc13783.o
+obj-$(CONFIG_SND_SOC_IMX_AUDMIX) += snd-soc-imx-audmix.o
-/*
- * eukrea-tlv320.c -- SoC audio for eukrea_cpuimxXX in I2S mode
- *
- * Copyright 2010 Eric Bénard, Eukréa Electromatique <eric@eukrea.com>
- *
- * based on sound/soc/s3c24xx/s3c24xx_simtec_tlv320aic23.c
- * which is Copyright 2009 Simtec Electronics
- * and on sound/soc/imx/phycore-ac97.c which is
- * Copyright 2009 Sascha Hauer, Pengutronix <s.hauer@pengutronix.de>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// eukrea-tlv320.c -- SoC audio for eukrea_cpuimxXX in I2S mode
+//
+// Copyright 2010 Eric Bénard, Eukréa Electromatique <eric@eukrea.com>
+//
+// based on sound/soc/s3c24xx/s3c24xx_simtec_tlv320aic23.c
+// which is Copyright 2009 Simtec Electronics
+// and on sound/soc/imx/phycore-ac97.c which is
+// Copyright 2009 Sascha Hauer, Pengutronix <s.hauer@pengutronix.de>
#include <linux/errno.h>
#include <linux/module.h>
if (ret) {
dev_err(&pdev->dev,
"fsl,mux-int-port node missing or invalid.\n");
- return ret;
+ goto err;
}
ret = of_property_read_u32(np, "fsl,mux-ext-port", &ext_port);
if (ret) {
dev_err(&pdev->dev,
"fsl,mux-ext-port node missing or invalid.\n");
- return ret;
+ goto err;
}
/*
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NXP AUDMIX ALSA SoC Digital Audio Interface (DAI) driver
+ *
+ * Copyright 2017 NXP
+ */
+
+#include <linux/clk.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/pm_runtime.h>
+#include <sound/soc.h>
+#include <sound/pcm_params.h>
+
+#include "fsl_audmix.h"
+
+#define SOC_ENUM_SINGLE_S(xreg, xshift, xtexts) \
+ SOC_ENUM_SINGLE(xreg, xshift, ARRAY_SIZE(xtexts), xtexts)
+
+static const char
+ *tdm_sel[] = { "TDM1", "TDM2", },
+ *mode_sel[] = { "Disabled", "TDM1", "TDM2", "Mixed", },
+ *width_sel[] = { "16b", "18b", "20b", "24b", "32b", },
+ *endis_sel[] = { "Disabled", "Enabled", },
+ *updn_sel[] = { "Downward", "Upward", },
+ *mask_sel[] = { "Unmask", "Mask", };
+
+static const struct soc_enum fsl_audmix_enum[] = {
+/* FSL_AUDMIX_CTR enums */
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_CTR, FSL_AUDMIX_CTR_MIXCLK_SHIFT, tdm_sel),
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_CTR, FSL_AUDMIX_CTR_OUTSRC_SHIFT, mode_sel),
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_CTR, FSL_AUDMIX_CTR_OUTWIDTH_SHIFT, width_sel),
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_CTR, FSL_AUDMIX_CTR_MASKRTDF_SHIFT, mask_sel),
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_CTR, FSL_AUDMIX_CTR_MASKCKDF_SHIFT, mask_sel),
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_CTR, FSL_AUDMIX_CTR_SYNCMODE_SHIFT, endis_sel),
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_CTR, FSL_AUDMIX_CTR_SYNCSRC_SHIFT, tdm_sel),
+/* FSL_AUDMIX_ATCR0 enums */
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_ATCR0, 0, endis_sel),
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_ATCR0, 1, updn_sel),
+/* FSL_AUDMIX_ATCR1 enums */
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_ATCR1, 0, endis_sel),
+SOC_ENUM_SINGLE_S(FSL_AUDMIX_ATCR1, 1, updn_sel),
+};
+
+struct fsl_audmix_state {
+ u8 tdms;
+ u8 clk;
+ char msg[64];
+};
+
+static const struct fsl_audmix_state prms[4][4] = {{
+ /* DIS->DIS, do nothing */
+ { .tdms = 0, .clk = 0, .msg = "" },
+ /* DIS->TDM1*/
+ { .tdms = 1, .clk = 1, .msg = "DIS->TDM1: TDM1 not started!\n" },
+ /* DIS->TDM2*/
+ { .tdms = 2, .clk = 2, .msg = "DIS->TDM2: TDM2 not started!\n" },
+ /* DIS->MIX */
+ { .tdms = 3, .clk = 0, .msg = "DIS->MIX: Please start both TDMs!\n" }
+}, { /* TDM1->DIS */
+ { .tdms = 1, .clk = 0, .msg = "TDM1->DIS: TDM1 not started!\n" },
+ /* TDM1->TDM1, do nothing */
+ { .tdms = 0, .clk = 0, .msg = "" },
+ /* TDM1->TDM2 */
+ { .tdms = 3, .clk = 2, .msg = "TDM1->TDM2: Please start both TDMs!\n" },
+ /* TDM1->MIX */
+ { .tdms = 3, .clk = 0, .msg = "TDM1->MIX: Please start both TDMs!\n" }
+}, { /* TDM2->DIS */
+ { .tdms = 2, .clk = 0, .msg = "TDM2->DIS: TDM2 not started!\n" },
+ /* TDM2->TDM1 */
+ { .tdms = 3, .clk = 1, .msg = "TDM2->TDM1: Please start both TDMs!\n" },
+ /* TDM2->TDM2, do nothing */
+ { .tdms = 0, .clk = 0, .msg = "" },
+ /* TDM2->MIX */
+ { .tdms = 3, .clk = 0, .msg = "TDM2->MIX: Please start both TDMs!\n" }
+}, { /* MIX->DIS */
+ { .tdms = 3, .clk = 0, .msg = "MIX->DIS: Please start both TDMs!\n" },
+ /* MIX->TDM1 */
+ { .tdms = 3, .clk = 1, .msg = "MIX->TDM1: Please start both TDMs!\n" },
+ /* MIX->TDM2 */
+ { .tdms = 3, .clk = 2, .msg = "MIX->TDM2: Please start both TDMs!\n" },
+ /* MIX->MIX, do nothing */
+ { .tdms = 0, .clk = 0, .msg = "" }
+}, };
+
+static int fsl_audmix_state_trans(struct snd_soc_component *comp,
+ unsigned int *mask, unsigned int *ctr,
+ const struct fsl_audmix_state prm)
+{
+ struct fsl_audmix *priv = snd_soc_component_get_drvdata(comp);
+ /* Enforce all required TDMs are started */
+ if ((priv->tdms & prm.tdms) != prm.tdms) {
+ dev_dbg(comp->dev, "%s", prm.msg);
+ return -EINVAL;
+ }
+
+ switch (prm.clk) {
+ case 1:
+ case 2:
+ /* Set mix clock */
+ (*mask) |= FSL_AUDMIX_CTR_MIXCLK_MASK;
+ (*ctr) |= FSL_AUDMIX_CTR_MIXCLK(prm.clk - 1);
+ break;
+ default:
+ break;
+ }
+
+ return 0;
+}
+
+static int fsl_audmix_put_mix_clk_src(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct snd_soc_component *comp = snd_kcontrol_chip(kcontrol);
+ struct fsl_audmix *priv = snd_soc_component_get_drvdata(comp);
+ struct soc_enum *e = (struct soc_enum *)kcontrol->private_value;
+ unsigned int *item = ucontrol->value.enumerated.item;
+ unsigned int reg_val, val, mix_clk;
+ int ret = 0;
+
+ /* Get current state */
+ ret = snd_soc_component_read(comp, FSL_AUDMIX_CTR, ®_val);
+ if (ret)
+ return ret;
+
+ mix_clk = ((reg_val & FSL_AUDMIX_CTR_MIXCLK_MASK)
+ >> FSL_AUDMIX_CTR_MIXCLK_SHIFT);
+ val = snd_soc_enum_item_to_val(e, item[0]);
+
+ dev_dbg(comp->dev, "TDMs=x%08x, val=x%08x\n", priv->tdms, val);
+
+ /**
+ * Ensure the current selected mixer clock is available
+ * for configuration propagation
+ */
+ if (!(priv->tdms & BIT(mix_clk))) {
+ dev_err(comp->dev,
+ "Started TDM%d needed for config propagation!\n",
+ mix_clk + 1);
+ return -EINVAL;
+ }
+
+ if (!(priv->tdms & BIT(val))) {
+ dev_err(comp->dev,
+ "The selected clock source has no TDM%d enabled!\n",
+ val + 1);
+ return -EINVAL;
+ }
+
+ return snd_soc_put_enum_double(kcontrol, ucontrol);
+}
+
+static int fsl_audmix_put_out_src(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct snd_soc_component *comp = snd_kcontrol_chip(kcontrol);
+ struct fsl_audmix *priv = snd_soc_component_get_drvdata(comp);
+ struct soc_enum *e = (struct soc_enum *)kcontrol->private_value;
+ unsigned int *item = ucontrol->value.enumerated.item;
+ u32 out_src, mix_clk;
+ unsigned int reg_val, val, mask = 0, ctr = 0;
+ int ret = 0;
+
+ /* Get current state */
+ ret = snd_soc_component_read(comp, FSL_AUDMIX_CTR, ®_val);
+ if (ret)
+ return ret;
+
+ /* "From" state */
+ out_src = ((reg_val & FSL_AUDMIX_CTR_OUTSRC_MASK)
+ >> FSL_AUDMIX_CTR_OUTSRC_SHIFT);
+ mix_clk = ((reg_val & FSL_AUDMIX_CTR_MIXCLK_MASK)
+ >> FSL_AUDMIX_CTR_MIXCLK_SHIFT);
+
+ /* "To" state */
+ val = snd_soc_enum_item_to_val(e, item[0]);
+
+ dev_dbg(comp->dev, "TDMs=x%08x, val=x%08x\n", priv->tdms, val);
+
+ /* Check if state is changing ... */
+ if (out_src == val)
+ return 0;
+ /**
+ * Ensure the current selected mixer clock is available
+ * for configuration propagation
+ */
+ if (!(priv->tdms & BIT(mix_clk))) {
+ dev_err(comp->dev,
+ "Started TDM%d needed for config propagation!\n",
+ mix_clk + 1);
+ return -EINVAL;
+ }
+
+ /* Check state transition constraints */
+ ret = fsl_audmix_state_trans(comp, &mask, &ctr, prms[out_src][val]);
+ if (ret)
+ return ret;
+
+ /* Complete transition to new state */
+ mask |= FSL_AUDMIX_CTR_OUTSRC_MASK;
+ ctr |= FSL_AUDMIX_CTR_OUTSRC(val);
+
+ return snd_soc_component_update_bits(comp, FSL_AUDMIX_CTR, mask, ctr);
+}
+
+static const struct snd_kcontrol_new fsl_audmix_snd_controls[] = {
+ /* FSL_AUDMIX_CTR controls */
+ SOC_ENUM_EXT("Mixing Clock Source", fsl_audmix_enum[0],
+ snd_soc_get_enum_double, fsl_audmix_put_mix_clk_src),
+ SOC_ENUM_EXT("Output Source", fsl_audmix_enum[1],
+ snd_soc_get_enum_double, fsl_audmix_put_out_src),
+ SOC_ENUM("Output Width", fsl_audmix_enum[2]),
+ SOC_ENUM("Frame Rate Diff Error", fsl_audmix_enum[3]),
+ SOC_ENUM("Clock Freq Diff Error", fsl_audmix_enum[4]),
+ SOC_ENUM("Sync Mode Config", fsl_audmix_enum[5]),
+ SOC_ENUM("Sync Mode Clk Source", fsl_audmix_enum[6]),
+ /* TDM1 Attenuation controls */
+ SOC_ENUM("TDM1 Attenuation", fsl_audmix_enum[7]),
+ SOC_ENUM("TDM1 Attenuation Direction", fsl_audmix_enum[8]),
+ SOC_SINGLE("TDM1 Attenuation Step Divider", FSL_AUDMIX_ATCR0,
+ 2, 0x00fff, 0),
+ SOC_SINGLE("TDM1 Attenuation Initial Value", FSL_AUDMIX_ATIVAL0,
+ 0, 0x3ffff, 0),
+ SOC_SINGLE("TDM1 Attenuation Step Up Factor", FSL_AUDMIX_ATSTPUP0,
+ 0, 0x3ffff, 0),
+ SOC_SINGLE("TDM1 Attenuation Step Down Factor", FSL_AUDMIX_ATSTPDN0,
+ 0, 0x3ffff, 0),
+ SOC_SINGLE("TDM1 Attenuation Step Target", FSL_AUDMIX_ATSTPTGT0,
+ 0, 0x3ffff, 0),
+ /* TDM2 Attenuation controls */
+ SOC_ENUM("TDM2 Attenuation", fsl_audmix_enum[9]),
+ SOC_ENUM("TDM2 Attenuation Direction", fsl_audmix_enum[10]),
+ SOC_SINGLE("TDM2 Attenuation Step Divider", FSL_AUDMIX_ATCR1,
+ 2, 0x00fff, 0),
+ SOC_SINGLE("TDM2 Attenuation Initial Value", FSL_AUDMIX_ATIVAL1,
+ 0, 0x3ffff, 0),
+ SOC_SINGLE("TDM2 Attenuation Step Up Factor", FSL_AUDMIX_ATSTPUP1,
+ 0, 0x3ffff, 0),
+ SOC_SINGLE("TDM2 Attenuation Step Down Factor", FSL_AUDMIX_ATSTPDN1,
+ 0, 0x3ffff, 0),
+ SOC_SINGLE("TDM2 Attenuation Step Target", FSL_AUDMIX_ATSTPTGT1,
+ 0, 0x3ffff, 0),
+};
+
+static int fsl_audmix_dai_set_fmt(struct snd_soc_dai *dai, unsigned int fmt)
+{
+ struct snd_soc_component *comp = dai->component;
+ u32 mask = 0, ctr = 0;
+
+ /* AUDMIX is working in DSP_A format only */
+ switch (fmt & SND_SOC_DAIFMT_FORMAT_MASK) {
+ case SND_SOC_DAIFMT_DSP_A:
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ /* For playback the AUDMIX is slave, and for record is master */
+ switch (fmt & SND_SOC_DAIFMT_MASTER_MASK) {
+ case SND_SOC_DAIFMT_CBM_CFM:
+ case SND_SOC_DAIFMT_CBS_CFS:
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ switch (fmt & SND_SOC_DAIFMT_INV_MASK) {
+ case SND_SOC_DAIFMT_IB_NF:
+ /* Output data will be written on positive edge of the clock */
+ ctr |= FSL_AUDMIX_CTR_OUTCKPOL(0);
+ break;
+ case SND_SOC_DAIFMT_NB_NF:
+ /* Output data will be written on negative edge of the clock */
+ ctr |= FSL_AUDMIX_CTR_OUTCKPOL(1);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ mask |= FSL_AUDMIX_CTR_OUTCKPOL_MASK;
+
+ return snd_soc_component_update_bits(comp, FSL_AUDMIX_CTR, mask, ctr);
+}
+
+static int fsl_audmix_dai_trigger(struct snd_pcm_substream *substream, int cmd,
+ struct snd_soc_dai *dai)
+{
+ struct fsl_audmix *priv = snd_soc_dai_get_drvdata(dai);
+
+ /* Capture stream shall not be handled */
+ if (substream->stream == SNDRV_PCM_STREAM_CAPTURE)
+ return 0;
+
+ switch (cmd) {
+ case SNDRV_PCM_TRIGGER_START:
+ case SNDRV_PCM_TRIGGER_RESUME:
+ case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
+ priv->tdms |= BIT(dai->driver->id);
+ break;
+ case SNDRV_PCM_TRIGGER_STOP:
+ case SNDRV_PCM_TRIGGER_SUSPEND:
+ case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
+ priv->tdms &= ~BIT(dai->driver->id);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static const struct snd_soc_dai_ops fsl_audmix_dai_ops = {
+ .set_fmt = fsl_audmix_dai_set_fmt,
+ .trigger = fsl_audmix_dai_trigger,
+};
+
+static struct snd_soc_dai_driver fsl_audmix_dai[] = {
+ {
+ .id = 0,
+ .name = "audmix-0",
+ .playback = {
+ .stream_name = "AUDMIX-Playback-0",
+ .channels_min = 8,
+ .channels_max = 8,
+ .rate_min = 8000,
+ .rate_max = 96000,
+ .rates = SNDRV_PCM_RATE_8000_96000,
+ .formats = FSL_AUDMIX_FORMATS,
+ },
+ .capture = {
+ .stream_name = "AUDMIX-Capture-0",
+ .channels_min = 8,
+ .channels_max = 8,
+ .rate_min = 8000,
+ .rate_max = 96000,
+ .rates = SNDRV_PCM_RATE_8000_96000,
+ .formats = FSL_AUDMIX_FORMATS,
+ },
+ .ops = &fsl_audmix_dai_ops,
+ },
+ {
+ .id = 1,
+ .name = "audmix-1",
+ .playback = {
+ .stream_name = "AUDMIX-Playback-1",
+ .channels_min = 8,
+ .channels_max = 8,
+ .rate_min = 8000,
+ .rate_max = 96000,
+ .rates = SNDRV_PCM_RATE_8000_96000,
+ .formats = FSL_AUDMIX_FORMATS,
+ },
+ .capture = {
+ .stream_name = "AUDMIX-Capture-1",
+ .channels_min = 8,
+ .channels_max = 8,
+ .rate_min = 8000,
+ .rate_max = 96000,
+ .rates = SNDRV_PCM_RATE_8000_96000,
+ .formats = FSL_AUDMIX_FORMATS,
+ },
+ .ops = &fsl_audmix_dai_ops,
+ },
+};
+
+static const struct snd_soc_component_driver fsl_audmix_component = {
+ .name = "fsl-audmix-dai",
+ .controls = fsl_audmix_snd_controls,
+ .num_controls = ARRAY_SIZE(fsl_audmix_snd_controls),
+};
+
+static bool fsl_audmix_readable_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case FSL_AUDMIX_CTR:
+ case FSL_AUDMIX_STR:
+ case FSL_AUDMIX_ATCR0:
+ case FSL_AUDMIX_ATIVAL0:
+ case FSL_AUDMIX_ATSTPUP0:
+ case FSL_AUDMIX_ATSTPDN0:
+ case FSL_AUDMIX_ATSTPTGT0:
+ case FSL_AUDMIX_ATTNVAL0:
+ case FSL_AUDMIX_ATSTP0:
+ case FSL_AUDMIX_ATCR1:
+ case FSL_AUDMIX_ATIVAL1:
+ case FSL_AUDMIX_ATSTPUP1:
+ case FSL_AUDMIX_ATSTPDN1:
+ case FSL_AUDMIX_ATSTPTGT1:
+ case FSL_AUDMIX_ATTNVAL1:
+ case FSL_AUDMIX_ATSTP1:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static bool fsl_audmix_writeable_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case FSL_AUDMIX_CTR:
+ case FSL_AUDMIX_ATCR0:
+ case FSL_AUDMIX_ATIVAL0:
+ case FSL_AUDMIX_ATSTPUP0:
+ case FSL_AUDMIX_ATSTPDN0:
+ case FSL_AUDMIX_ATSTPTGT0:
+ case FSL_AUDMIX_ATCR1:
+ case FSL_AUDMIX_ATIVAL1:
+ case FSL_AUDMIX_ATSTPUP1:
+ case FSL_AUDMIX_ATSTPDN1:
+ case FSL_AUDMIX_ATSTPTGT1:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static const struct reg_default fsl_audmix_reg[] = {
+ { FSL_AUDMIX_CTR, 0x00060 },
+ { FSL_AUDMIX_STR, 0x00003 },
+ { FSL_AUDMIX_ATCR0, 0x00000 },
+ { FSL_AUDMIX_ATIVAL0, 0x3FFFF },
+ { FSL_AUDMIX_ATSTPUP0, 0x2AAAA },
+ { FSL_AUDMIX_ATSTPDN0, 0x30000 },
+ { FSL_AUDMIX_ATSTPTGT0, 0x00010 },
+ { FSL_AUDMIX_ATTNVAL0, 0x00000 },
+ { FSL_AUDMIX_ATSTP0, 0x00000 },
+ { FSL_AUDMIX_ATCR1, 0x00000 },
+ { FSL_AUDMIX_ATIVAL1, 0x3FFFF },
+ { FSL_AUDMIX_ATSTPUP1, 0x2AAAA },
+ { FSL_AUDMIX_ATSTPDN1, 0x30000 },
+ { FSL_AUDMIX_ATSTPTGT1, 0x00010 },
+ { FSL_AUDMIX_ATTNVAL1, 0x00000 },
+ { FSL_AUDMIX_ATSTP1, 0x00000 },
+};
+
+static const struct regmap_config fsl_audmix_regmap_config = {
+ .reg_bits = 32,
+ .reg_stride = 4,
+ .val_bits = 32,
+ .max_register = FSL_AUDMIX_ATSTP1,
+ .reg_defaults = fsl_audmix_reg,
+ .num_reg_defaults = ARRAY_SIZE(fsl_audmix_reg),
+ .readable_reg = fsl_audmix_readable_reg,
+ .writeable_reg = fsl_audmix_writeable_reg,
+ .cache_type = REGCACHE_FLAT,
+};
+
+static const struct of_device_id fsl_audmix_ids[] = {
+ {
+ .compatible = "fsl,imx8qm-audmix",
+ .data = "imx-audmix",
+ },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, fsl_audmix_ids);
+
+static int fsl_audmix_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct fsl_audmix *priv;
+ struct resource *res;
+ const char *mdrv;
+ const struct of_device_id *of_id;
+ void __iomem *regs;
+ int ret;
+
+ of_id = of_match_device(fsl_audmix_ids, dev);
+ if (!of_id || !of_id->data)
+ return -EINVAL;
+
+ mdrv = of_id->data;
+
+ priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ /* Get the addresses */
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ regs = devm_ioremap_resource(dev, res);
+ if (IS_ERR(regs))
+ return PTR_ERR(regs);
+
+ priv->regmap = devm_regmap_init_mmio_clk(dev, "ipg", regs,
+ &fsl_audmix_regmap_config);
+ if (IS_ERR(priv->regmap)) {
+ dev_err(dev, "failed to init regmap\n");
+ return PTR_ERR(priv->regmap);
+ }
+
+ priv->ipg_clk = devm_clk_get(dev, "ipg");
+ if (IS_ERR(priv->ipg_clk)) {
+ dev_err(dev, "failed to get ipg clock\n");
+ return PTR_ERR(priv->ipg_clk);
+ }
+
+ platform_set_drvdata(pdev, priv);
+ pm_runtime_enable(dev);
+
+ ret = devm_snd_soc_register_component(dev, &fsl_audmix_component,
+ fsl_audmix_dai,
+ ARRAY_SIZE(fsl_audmix_dai));
+ if (ret) {
+ dev_err(dev, "failed to register ASoC DAI\n");
+ return ret;
+ }
+
+ priv->pdev = platform_device_register_data(dev, mdrv, 0, NULL, 0);
+ if (IS_ERR(priv->pdev)) {
+ ret = PTR_ERR(priv->pdev);
+ dev_err(dev, "failed to register platform %s: %d\n", mdrv, ret);
+ }
+
+ return ret;
+}
+
+static int fsl_audmix_remove(struct platform_device *pdev)
+{
+ struct fsl_audmix *priv = dev_get_drvdata(&pdev->dev);
+
+ if (priv->pdev)
+ platform_device_unregister(priv->pdev);
+
+ return 0;
+}
+
+#ifdef CONFIG_PM
+static int fsl_audmix_runtime_resume(struct device *dev)
+{
+ struct fsl_audmix *priv = dev_get_drvdata(dev);
+ int ret;
+
+ ret = clk_prepare_enable(priv->ipg_clk);
+ if (ret) {
+ dev_err(dev, "Failed to enable IPG clock: %d\n", ret);
+ return ret;
+ }
+
+ regcache_cache_only(priv->regmap, false);
+ regcache_mark_dirty(priv->regmap);
+
+ return regcache_sync(priv->regmap);
+}
+
+static int fsl_audmix_runtime_suspend(struct device *dev)
+{
+ struct fsl_audmix *priv = dev_get_drvdata(dev);
+
+ regcache_cache_only(priv->regmap, true);
+
+ clk_disable_unprepare(priv->ipg_clk);
+
+ return 0;
+}
+#endif /* CONFIG_PM */
+
+static const struct dev_pm_ops fsl_audmix_pm = {
+ SET_RUNTIME_PM_OPS(fsl_audmix_runtime_suspend,
+ fsl_audmix_runtime_resume,
+ NULL)
+ SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
+ pm_runtime_force_resume)
+};
+
+static struct platform_driver fsl_audmix_driver = {
+ .probe = fsl_audmix_probe,
+ .remove = fsl_audmix_remove,
+ .driver = {
+ .name = "fsl-audmix",
+ .of_match_table = fsl_audmix_ids,
+ .pm = &fsl_audmix_pm,
+ },
+};
+module_platform_driver(fsl_audmix_driver);
+
+MODULE_DESCRIPTION("NXP AUDMIX ASoC DAI driver");
+MODULE_AUTHOR("Viorel Suman <viorel.suman@nxp.com>");
+MODULE_ALIAS("platform:fsl-audmix");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * NXP AUDMIX ALSA SoC Digital Audio Interface (DAI) driver
+ *
+ * Copyright 2017 NXP
+ */
+
+#ifndef __FSL_AUDMIX_H
+#define __FSL_AUDMIX_H
+
+#define FSL_AUDMIX_FORMATS (SNDRV_PCM_FMTBIT_S16_LE |\
+ SNDRV_PCM_FMTBIT_S24_LE |\
+ SNDRV_PCM_FMTBIT_S32_LE)
+/* AUDMIX Registers */
+#define FSL_AUDMIX_CTR 0x200 /* Control */
+#define FSL_AUDMIX_STR 0x204 /* Status */
+
+#define FSL_AUDMIX_ATCR0 0x208 /* Attenuation Control */
+#define FSL_AUDMIX_ATIVAL0 0x20c /* Attenuation Initial Value */
+#define FSL_AUDMIX_ATSTPUP0 0x210 /* Attenuation step up factor */
+#define FSL_AUDMIX_ATSTPDN0 0x214 /* Attenuation step down factor */
+#define FSL_AUDMIX_ATSTPTGT0 0x218 /* Attenuation step target */
+#define FSL_AUDMIX_ATTNVAL0 0x21c /* Attenuation Value */
+#define FSL_AUDMIX_ATSTP0 0x220 /* Attenuation step number */
+
+#define FSL_AUDMIX_ATCR1 0x228 /* Attenuation Control */
+#define FSL_AUDMIX_ATIVAL1 0x22c /* Attenuation Initial Value */
+#define FSL_AUDMIX_ATSTPUP1 0x230 /* Attenuation step up factor */
+#define FSL_AUDMIX_ATSTPDN1 0x234 /* Attenuation step down factor */
+#define FSL_AUDMIX_ATSTPTGT1 0x238 /* Attenuation step target */
+#define FSL_AUDMIX_ATTNVAL1 0x23c /* Attenuation Value */
+#define FSL_AUDMIX_ATSTP1 0x240 /* Attenuation step number */
+
+/* AUDMIX Control Register */
+#define FSL_AUDMIX_CTR_MIXCLK_SHIFT 0
+#define FSL_AUDMIX_CTR_MIXCLK_MASK BIT(FSL_AUDMIX_CTR_MIXCLK_SHIFT)
+#define FSL_AUDMIX_CTR_MIXCLK(i) ((i) << FSL_AUDMIX_CTR_MIXCLK_SHIFT)
+#define FSL_AUDMIX_CTR_OUTSRC_SHIFT 1
+#define FSL_AUDMIX_CTR_OUTSRC_MASK (0x3 << FSL_AUDMIX_CTR_OUTSRC_SHIFT)
+#define FSL_AUDMIX_CTR_OUTSRC(i) (((i) << FSL_AUDMIX_CTR_OUTSRC_SHIFT)\
+ & FSL_AUDMIX_CTR_OUTSRC_MASK)
+#define FSL_AUDMIX_CTR_OUTWIDTH_SHIFT 3
+#define FSL_AUDMIX_CTR_OUTWIDTH_MASK (0x7 << FSL_AUDMIX_CTR_OUTWIDTH_SHIFT)
+#define FSL_AUDMIX_CTR_OUTWIDTH(i) (((i) << FSL_AUDMIX_CTR_OUTWIDTH_SHIFT)\
+ & FSL_AUDMIX_CTR_OUTWIDTH_MASK)
+#define FSL_AUDMIX_CTR_OUTCKPOL_SHIFT 6
+#define FSL_AUDMIX_CTR_OUTCKPOL_MASK BIT(FSL_AUDMIX_CTR_OUTCKPOL_SHIFT)
+#define FSL_AUDMIX_CTR_OUTCKPOL(i) ((i) << FSL_AUDMIX_CTR_OUTCKPOL_SHIFT)
+#define FSL_AUDMIX_CTR_MASKRTDF_SHIFT 7
+#define FSL_AUDMIX_CTR_MASKRTDF_MASK BIT(FSL_AUDMIX_CTR_MASKRTDF_SHIFT)
+#define FSL_AUDMIX_CTR_MASKRTDF(i) ((i) << FSL_AUDMIX_CTR_MASKRTDF_SHIFT)
+#define FSL_AUDMIX_CTR_MASKCKDF_SHIFT 8
+#define FSL_AUDMIX_CTR_MASKCKDF_MASK BIT(FSL_AUDMIX_CTR_MASKCKDF_SHIFT)
+#define FSL_AUDMIX_CTR_MASKCKDF(i) ((i) << FSL_AUDMIX_CTR_MASKCKDF_SHIFT)
+#define FSL_AUDMIX_CTR_SYNCMODE_SHIFT 9
+#define FSL_AUDMIX_CTR_SYNCMODE_MASK BIT(FSL_AUDMIX_CTR_SYNCMODE_SHIFT)
+#define FSL_AUDMIX_CTR_SYNCMODE(i) ((i) << FSL_AUDMIX_CTR_SYNCMODE_SHIFT)
+#define FSL_AUDMIX_CTR_SYNCSRC_SHIFT 10
+#define FSL_AUDMIX_CTR_SYNCSRC_MASK BIT(FSL_AUDMIX_CTR_SYNCSRC_SHIFT)
+#define FSL_AUDMIX_CTR_SYNCSRC(i) ((i) << FSL_AUDMIX_CTR_SYNCSRC_SHIFT)
+
+/* AUDMIX Status Register */
+#define FSL_AUDMIX_STR_RATEDIFF BIT(0)
+#define FSL_AUDMIX_STR_CLKDIFF BIT(1)
+#define FSL_AUDMIX_STR_MIXSTAT_SHIFT 2
+#define FSL_AUDMIX_STR_MIXSTAT_MASK (0x3 << FSL_AUDMIX_STR_MIXSTAT_SHIFT)
+#define FSL_AUDMIX_STR_MIXSTAT(i) (((i) & FSL_AUDMIX_STR_MIXSTAT_MASK) \
+ >> FSL_AUDMIX_STR_MIXSTAT_SHIFT)
+/* AUDMIX Attenuation Control Register */
+#define FSL_AUDMIX_ATCR_AT_EN BIT(0)
+#define FSL_AUDMIX_ATCR_AT_UPDN BIT(1)
+#define FSL_AUDMIX_ATCR_ATSTPDIF_SHIFT 2
+#define FSL_AUDMIX_ATCR_ATSTPDFI_MASK \
+ (0xfff << FSL_AUDMIX_ATCR_ATSTPDIF_SHIFT)
+
+/* AUDMIX Attenuation Initial Value Register */
+#define FSL_AUDMIX_ATIVAL_ATINVAL_MASK 0x3FFFF
+
+/* AUDMIX Attenuation Step Up Factor Register */
+#define FSL_AUDMIX_ATSTPUP_ATSTEPUP_MASK 0x3FFFF
+
+/* AUDMIX Attenuation Step Down Factor Register */
+#define FSL_AUDMIX_ATSTPDN_ATSTEPDN_MASK 0x3FFFF
+
+/* AUDMIX Attenuation Step Target Register */
+#define FSL_AUDMIX_ATSTPTGT_ATSTPTG_MASK 0x3FFFF
+
+/* AUDMIX Attenuation Value Register */
+#define FSL_AUDMIX_ATTNVAL_ATCURVAL_MASK 0x3FFFF
+
+/* AUDMIX Attenuation Step Number Register */
+#define FSL_AUDMIX_ATSTP_STPCTR_MASK 0x3FFFF
+
+#define FSL_AUDMIX_MAX_DAIS 2
+struct fsl_audmix {
+ struct platform_device *pdev;
+ struct regmap *regmap;
+ struct clk *ipg_clk;
+ u8 tdms;
+};
+
+#endif /* __FSL_AUDMIX_H */
-/*
- * Freescale DMA ALSA SoC PCM driver
- *
- * Author: Timur Tabi <timur@freescale.com>
- *
- * Copyright 2007-2010 Freescale Semiconductor, Inc.
- *
- * This file is licensed under the terms of the GNU General Public License
- * version 2. This program is licensed "as is" without any warranty of any
- * kind, whether express or implied.
- *
- * This driver implements ASoC support for the Elo DMA controller, which is
- * the DMA controller on Freescale 83xx, 85xx, and 86xx SOCs. In ALSA terms,
- * the PCM driver is what handles the DMA buffer.
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// Freescale DMA ALSA SoC PCM driver
+//
+// Author: Timur Tabi <timur@freescale.com>
+//
+// Copyright 2007-2010 Freescale Semiconductor, Inc.
+//
+// This driver implements ASoC support for the Elo DMA controller, which is
+// the DMA controller on Freescale 83xx, 85xx, and 86xx SOCs. In ALSA terms,
+// the PCM driver is what handles the DMA buffer.
#include <linux/module.h>
#include <linux/init.h>
+/* SPDX-License-Identifier: GPL-2.0 */
/*
* mpc8610-pcm.h - ALSA PCM interface for the Freescale MPC8610 SoC
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#ifndef _MPC8610_PCM_H
{
struct fsl_esai *esai_priv = snd_soc_dai_get_drvdata(dai);
struct clk *clksrc = esai_priv->extalclk;
- bool tx = clk_id <= ESAI_HCKT_EXTAL;
+ bool tx = (clk_id <= ESAI_HCKT_EXTAL || esai_priv->synchronous);
bool in = dir == SND_SOC_CLOCK_IN;
u32 ratio, ecr = 0;
unsigned long clk_rate;
break;
case ESAI_HCKT_EXTAL:
ecr |= ESAI_ECR_ETI;
- /* fall through */
+ break;
case ESAI_HCKR_EXTAL:
- ecr |= ESAI_ECR_ERI;
+ ecr |= esai_priv->synchronous ? ESAI_ECR_ETI : ESAI_ECR_ERI;
break;
default:
return -EINVAL;
bclk = params_rate(params) * slot_width * esai_priv->slots;
- ret = fsl_esai_set_bclk(dai, tx, bclk);
+ ret = fsl_esai_set_bclk(dai, esai_priv->synchronous || tx, bclk);
if (ret)
return ret;
+ mask = ESAI_xCR_xSWS_MASK;
+ val = ESAI_xCR_xSWS(slot_width, width);
+
+ regmap_update_bits(esai_priv->regmap, REG_ESAI_xCR(tx), mask, val);
+ /* Recording in synchronous mode needs to set TCR also */
+ if (!tx && esai_priv->synchronous)
+ regmap_update_bits(esai_priv->regmap, REG_ESAI_TCR, mask, val);
+
/* Use Normal mode to support monaural audio */
regmap_update_bits(esai_priv->regmap, REG_ESAI_xCR(tx),
ESAI_xCR_xMOD_MASK, params_channels(params) > 1 ?
regmap_update_bits(esai_priv->regmap, REG_ESAI_xFCR(tx), mask, val);
- mask = ESAI_xCR_xSWS_MASK | (tx ? ESAI_xCR_PADC : 0);
- val = ESAI_xCR_xSWS(slot_width, width) | (tx ? ESAI_xCR_PADC : 0);
-
- regmap_update_bits(esai_priv->regmap, REG_ESAI_xCR(tx), mask, val);
+ if (tx)
+ regmap_update_bits(esai_priv->regmap, REG_ESAI_TCR,
+ ESAI_xCR_PADC, ESAI_xCR_PADC);
/* Remove ESAI personal reset by configuring ESAI_PCRC and ESAI_PRRC */
regmap_update_bits(esai_priv->regmap, REG_ESAI_PRRC,
{
u32 ctrl2_reg;
long mclk_rate;
- int osr;
int clk_div;
regmap_read(micfil->regmap, REG_MICFIL_CTRL2, &ctrl2_reg);
- osr = 16 - ((ctrl2_reg & MICFIL_CTRL2_CICOSR_MASK)
- >> MICFIL_CTRL2_CICOSR_SHIFT);
mclk_rate = clk_get_rate(micfil->mclk);
#include <linux/dmaengine.h>
#include <linux/module.h>
#include <linux/of_address.h>
+#include <linux/pm_runtime.h>
#include <linux/regmap.h>
#include <linux/slab.h>
#include <linux/time.h>
case SND_SOC_DAIFMT_CBS_CFS:
val_cr2 |= FSL_SAI_CR2_BCD_MSTR;
val_cr4 |= FSL_SAI_CR4_FSD_MSTR;
+ sai->is_slave_mode = false;
break;
case SND_SOC_DAIFMT_CBM_CFM:
sai->is_slave_mode = true;
break;
case SND_SOC_DAIFMT_CBS_CFM:
val_cr2 |= FSL_SAI_CR2_BCD_MSTR;
+ sai->is_slave_mode = false;
break;
case SND_SOC_DAIFMT_CBM_CFS:
val_cr4 |= FSL_SAI_CR4_FSD_MSTR;
platform_set_drvdata(pdev, sai);
+ pm_runtime_enable(&pdev->dev);
+
ret = devm_snd_soc_register_component(&pdev->dev, &fsl_component,
&fsl_sai_dai, 1);
if (ret)
return devm_snd_dmaengine_pcm_register(&pdev->dev, NULL, 0);
}
+static int fsl_sai_remove(struct platform_device *pdev)
+{
+ pm_runtime_disable(&pdev->dev);
+
+ return 0;
+}
+
static const struct of_device_id fsl_sai_ids[] = {
{ .compatible = "fsl,vf610-sai", },
{ .compatible = "fsl,imx6sx-sai", },
};
MODULE_DEVICE_TABLE(of, fsl_sai_ids);
-#ifdef CONFIG_PM_SLEEP
-static int fsl_sai_suspend(struct device *dev)
+#ifdef CONFIG_PM
+static int fsl_sai_runtime_suspend(struct device *dev)
{
struct fsl_sai *sai = dev_get_drvdata(dev);
return 0;
}
-static int fsl_sai_resume(struct device *dev)
+static int fsl_sai_runtime_resume(struct device *dev)
{
struct fsl_sai *sai = dev_get_drvdata(dev);
regmap_write(sai->regmap, FSL_SAI_RCSR, 0);
return regcache_sync(sai->regmap);
}
-#endif /* CONFIG_PM_SLEEP */
+#endif /* CONFIG_PM */
static const struct dev_pm_ops fsl_sai_pm_ops = {
- SET_SYSTEM_SLEEP_PM_OPS(fsl_sai_suspend, fsl_sai_resume)
+ SET_RUNTIME_PM_OPS(fsl_sai_runtime_suspend,
+ fsl_sai_runtime_resume, NULL)
+ SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
+ pm_runtime_force_resume)
};
static struct platform_driver fsl_sai_driver = {
.probe = fsl_sai_probe,
+ .remove = fsl_sai_remove,
.driver = {
.name = "fsl-sai",
.pm = &fsl_sai_pm_ops,
iprop = of_get_property(dma_np, "cell-index", NULL);
if (!iprop) {
of_node_put(dma_np);
+ of_node_put(dma_channel_np);
return -EINVAL;
}
*dma_id = be32_to_cpup(iprop);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2017 NXP
+ *
+ * The code contained herein is licensed under the GNU General Public
+ * License. You may obtain a copy of the GNU General Public License
+ * Version 2 or later at the following locations:
+ *
+ * http://www.opensource.org/licenses/gpl-license.html
+ * http://www.gnu.org/copyleft/gpl.html
+ */
+
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/clk.h>
+#include <sound/soc.h>
+#include <sound/soc-dapm.h>
+#include <linux/pm_runtime.h>
+#include "fsl_sai.h"
+#include "fsl_audmix.h"
+
+struct imx_audmix {
+ struct platform_device *pdev;
+ struct snd_soc_card card;
+ struct platform_device *audmix_pdev;
+ struct platform_device *out_pdev;
+ struct clk *cpu_mclk;
+ int num_dai;
+ struct snd_soc_dai_link *dai;
+ int num_dai_conf;
+ struct snd_soc_codec_conf *dai_conf;
+ int num_dapm_routes;
+ struct snd_soc_dapm_route *dapm_routes;
+};
+
+static const u32 imx_audmix_rates[] = {
+ 8000, 12000, 16000, 24000, 32000, 48000, 64000, 96000,
+};
+
+static const struct snd_pcm_hw_constraint_list imx_audmix_rate_constraints = {
+ .count = ARRAY_SIZE(imx_audmix_rates),
+ .list = imx_audmix_rates,
+};
+
+static int imx_audmix_fe_startup(struct snd_pcm_substream *substream)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct imx_audmix *priv = snd_soc_card_get_drvdata(rtd->card);
+ struct snd_pcm_runtime *runtime = substream->runtime;
+ struct device *dev = rtd->card->dev;
+ unsigned long clk_rate = clk_get_rate(priv->cpu_mclk);
+ int ret;
+
+ if (clk_rate % 24576000 == 0) {
+ ret = snd_pcm_hw_constraint_list(runtime, 0,
+ SNDRV_PCM_HW_PARAM_RATE,
+ &imx_audmix_rate_constraints);
+ if (ret < 0)
+ return ret;
+ } else {
+ dev_warn(dev, "mclk may be not supported %lu\n", clk_rate);
+ }
+
+ ret = snd_pcm_hw_constraint_minmax(runtime, SNDRV_PCM_HW_PARAM_CHANNELS,
+ 1, 8);
+ if (ret < 0)
+ return ret;
+
+ return snd_pcm_hw_constraint_mask64(runtime, SNDRV_PCM_HW_PARAM_FORMAT,
+ FSL_AUDMIX_FORMATS);
+}
+
+static int imx_audmix_fe_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct device *dev = rtd->card->dev;
+ bool tx = substream->stream == SNDRV_PCM_STREAM_PLAYBACK;
+ unsigned int fmt = SND_SOC_DAIFMT_DSP_A | SND_SOC_DAIFMT_NB_NF;
+ u32 channels = params_channels(params);
+ int ret, dir;
+
+ /* For playback the AUDMIX is slave, and for record is master */
+ fmt |= tx ? SND_SOC_DAIFMT_CBS_CFS : SND_SOC_DAIFMT_CBM_CFM;
+ dir = tx ? SND_SOC_CLOCK_OUT : SND_SOC_CLOCK_IN;
+
+ /* set DAI configuration */
+ ret = snd_soc_dai_set_fmt(rtd->cpu_dai, fmt);
+ if (ret) {
+ dev_err(dev, "failed to set cpu dai fmt: %d\n", ret);
+ return ret;
+ }
+
+ ret = snd_soc_dai_set_sysclk(rtd->cpu_dai, FSL_SAI_CLK_MAST1, 0, dir);
+ if (ret) {
+ dev_err(dev, "failed to set cpu sysclk: %d\n", ret);
+ return ret;
+ }
+
+ /*
+ * Per datasheet, AUDMIX expects 8 slots and 32 bits
+ * for every slot in TDM mode.
+ */
+ ret = snd_soc_dai_set_tdm_slot(rtd->cpu_dai, BIT(channels) - 1,
+ BIT(channels) - 1, 8, 32);
+ if (ret)
+ dev_err(dev, "failed to set cpu dai tdm slot: %d\n", ret);
+
+ return ret;
+}
+
+static int imx_audmix_be_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct device *dev = rtd->card->dev;
+ bool tx = substream->stream == SNDRV_PCM_STREAM_PLAYBACK;
+ unsigned int fmt = SND_SOC_DAIFMT_DSP_A | SND_SOC_DAIFMT_NB_NF;
+ int ret;
+
+ if (!tx)
+ return 0;
+
+ /* For playback the AUDMIX is slave */
+ fmt |= SND_SOC_DAIFMT_CBM_CFM;
+
+ /* set AUDMIX DAI configuration */
+ ret = snd_soc_dai_set_fmt(rtd->cpu_dai, fmt);
+ if (ret)
+ dev_err(dev, "failed to set AUDMIX DAI fmt: %d\n", ret);
+
+ return ret;
+}
+
+static struct snd_soc_ops imx_audmix_fe_ops = {
+ .startup = imx_audmix_fe_startup,
+ .hw_params = imx_audmix_fe_hw_params,
+};
+
+static struct snd_soc_ops imx_audmix_be_ops = {
+ .hw_params = imx_audmix_be_hw_params,
+};
+
+static int imx_audmix_probe(struct platform_device *pdev)
+{
+ struct device_node *np = pdev->dev.of_node;
+ struct device_node *audmix_np = NULL, *out_cpu_np = NULL;
+ struct platform_device *audmix_pdev = NULL;
+ struct platform_device *cpu_pdev;
+ struct of_phandle_args args;
+ struct imx_audmix *priv;
+ int i, num_dai, ret;
+ const char *fe_name_pref = "HiFi-AUDMIX-FE-";
+ char *be_name, *be_pb, *be_cp, *dai_name, *capture_dai_name;
+
+ if (pdev->dev.parent) {
+ audmix_np = pdev->dev.parent->of_node;
+ } else {
+ dev_err(&pdev->dev, "Missing parent device.\n");
+ return -EINVAL;
+ }
+
+ if (!audmix_np) {
+ dev_err(&pdev->dev, "Missing DT node for parent device.\n");
+ return -EINVAL;
+ }
+
+ audmix_pdev = of_find_device_by_node(audmix_np);
+ if (!audmix_pdev) {
+ dev_err(&pdev->dev, "Missing AUDMIX platform device for %s\n",
+ np->full_name);
+ return -EINVAL;
+ }
+ put_device(&audmix_pdev->dev);
+
+ num_dai = of_count_phandle_with_args(audmix_np, "dais", NULL);
+ if (num_dai != FSL_AUDMIX_MAX_DAIS) {
+ dev_err(&pdev->dev, "Need 2 dais to be provided for %s\n",
+ audmix_np->full_name);
+ return -EINVAL;
+ }
+
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->num_dai = 2 * num_dai;
+ priv->dai = devm_kzalloc(&pdev->dev, priv->num_dai *
+ sizeof(struct snd_soc_dai_link), GFP_KERNEL);
+ if (!priv->dai)
+ return -ENOMEM;
+
+ priv->num_dai_conf = num_dai;
+ priv->dai_conf = devm_kzalloc(&pdev->dev, priv->num_dai_conf *
+ sizeof(struct snd_soc_codec_conf),
+ GFP_KERNEL);
+ if (!priv->dai_conf)
+ return -ENOMEM;
+
+ priv->num_dapm_routes = 3 * num_dai;
+ priv->dapm_routes = devm_kzalloc(&pdev->dev, priv->num_dapm_routes *
+ sizeof(struct snd_soc_dapm_route),
+ GFP_KERNEL);
+ if (!priv->dapm_routes)
+ return -ENOMEM;
+
+ for (i = 0; i < num_dai; i++) {
+ ret = of_parse_phandle_with_args(audmix_np, "dais", NULL, i,
+ &args);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "of_parse_phandle_with_args failed\n");
+ return ret;
+ }
+
+ cpu_pdev = of_find_device_by_node(args.np);
+ if (!cpu_pdev) {
+ dev_err(&pdev->dev, "failed to find SAI platform device\n");
+ return -EINVAL;
+ }
+ put_device(&cpu_pdev->dev);
+
+ dai_name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "%s%s",
+ fe_name_pref, args.np->full_name + 1);
+
+ dev_info(pdev->dev.parent, "DAI FE name:%s\n", dai_name);
+
+ if (i == 0) {
+ out_cpu_np = args.np;
+ capture_dai_name =
+ devm_kasprintf(&pdev->dev, GFP_KERNEL, "%s %s",
+ dai_name, "CPU-Capture");
+ }
+
+ priv->dai[i].name = dai_name;
+ priv->dai[i].stream_name = "HiFi-AUDMIX-FE";
+ priv->dai[i].codec_dai_name = "snd-soc-dummy-dai";
+ priv->dai[i].codec_name = "snd-soc-dummy";
+ priv->dai[i].cpu_of_node = args.np;
+ priv->dai[i].cpu_dai_name = dev_name(&cpu_pdev->dev);
+ priv->dai[i].platform_of_node = args.np;
+ priv->dai[i].dynamic = 1;
+ priv->dai[i].dpcm_playback = 1;
+ priv->dai[i].dpcm_capture = (i == 0 ? 1 : 0);
+ priv->dai[i].ignore_pmdown_time = 1;
+ priv->dai[i].ops = &imx_audmix_fe_ops;
+
+ /* Add AUDMIX Backend */
+ be_name = devm_kasprintf(&pdev->dev, GFP_KERNEL,
+ "audmix-%d", i);
+ be_pb = devm_kasprintf(&pdev->dev, GFP_KERNEL,
+ "AUDMIX-Playback-%d", i);
+ be_cp = devm_kasprintf(&pdev->dev, GFP_KERNEL,
+ "AUDMIX-Capture-%d", i);
+
+ priv->dai[num_dai + i].name = be_name;
+ priv->dai[num_dai + i].codec_dai_name = "snd-soc-dummy-dai";
+ priv->dai[num_dai + i].codec_name = "snd-soc-dummy";
+ priv->dai[num_dai + i].cpu_of_node = audmix_np;
+ priv->dai[num_dai + i].cpu_dai_name = be_name;
+ priv->dai[num_dai + i].platform_name = "snd-soc-dummy";
+ priv->dai[num_dai + i].no_pcm = 1;
+ priv->dai[num_dai + i].dpcm_playback = 1;
+ priv->dai[num_dai + i].dpcm_capture = 1;
+ priv->dai[num_dai + i].ignore_pmdown_time = 1;
+ priv->dai[num_dai + i].ops = &imx_audmix_be_ops;
+
+ priv->dai_conf[i].of_node = args.np;
+ priv->dai_conf[i].name_prefix = dai_name;
+
+ priv->dapm_routes[i].source =
+ devm_kasprintf(&pdev->dev, GFP_KERNEL, "%s %s",
+ dai_name, "CPU-Playback");
+ priv->dapm_routes[i].sink = be_pb;
+ priv->dapm_routes[num_dai + i].source = be_pb;
+ priv->dapm_routes[num_dai + i].sink = be_cp;
+ priv->dapm_routes[2 * num_dai + i].source = be_cp;
+ priv->dapm_routes[2 * num_dai + i].sink = capture_dai_name;
+ }
+
+ cpu_pdev = of_find_device_by_node(out_cpu_np);
+ if (!cpu_pdev) {
+ dev_err(&pdev->dev, "failed to find SAI platform device\n");
+ return -EINVAL;
+ }
+ put_device(&cpu_pdev->dev);
+
+ priv->cpu_mclk = devm_clk_get(&cpu_pdev->dev, "mclk1");
+ if (IS_ERR(priv->cpu_mclk)) {
+ ret = PTR_ERR(priv->cpu_mclk);
+ dev_err(&cpu_pdev->dev, "failed to get DAI mclk1: %d\n", ret);
+ return -EINVAL;
+ }
+
+ priv->audmix_pdev = audmix_pdev;
+ priv->out_pdev = cpu_pdev;
+
+ priv->card.dai_link = priv->dai;
+ priv->card.num_links = priv->num_dai;
+ priv->card.codec_conf = priv->dai_conf;
+ priv->card.num_configs = priv->num_dai_conf;
+ priv->card.dapm_routes = priv->dapm_routes;
+ priv->card.num_dapm_routes = priv->num_dapm_routes;
+ priv->card.dev = pdev->dev.parent;
+ priv->card.owner = THIS_MODULE;
+ priv->card.name = "imx-audmix";
+
+ platform_set_drvdata(pdev, &priv->card);
+ snd_soc_card_set_drvdata(&priv->card, priv);
+
+ ret = devm_snd_soc_register_card(pdev->dev.parent, &priv->card);
+ if (ret) {
+ dev_err(&pdev->dev, "snd_soc_register_card failed\n");
+ return ret;
+ }
+
+ return ret;
+}
+
+static struct platform_driver imx_audmix_driver = {
+ .probe = imx_audmix_probe,
+ .driver = {
+ .name = "imx-audmix",
+ .pm = &snd_soc_pm_ops,
+ },
+};
+module_platform_driver(imx_audmix_driver);
+
+MODULE_DESCRIPTION("NXP AUDMIX ASoC machine driver");
+MODULE_AUTHOR("Viorel Suman <viorel.suman@nxp.com>");
+MODULE_ALIAS("platform:imx-audmix");
+MODULE_LICENSE("GPL v2");
-/*
- * Copyright 2012 Freescale Semiconductor, Inc.
- * Copyright 2012 Linaro Ltd.
- * Copyright 2009 Pengutronix, Sascha Hauer <s.hauer@pengutronix.de>
- *
- * Initial development of this code was funded by
- * Phytec Messtechnik GmbH, http://www.phytec.de
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Copyright 2012 Freescale Semiconductor, Inc.
+// Copyright 2012 Linaro Ltd.
+// Copyright 2009 Pengutronix, Sascha Hauer <s.hauer@pengutronix.de>
+//
+// Initial development of this code was funded by
+// Phytec Messtechnik GmbH, http://www.phytec.de
#include <linux/clk.h>
#include <linux/debugfs.h>
-/*
- * Copyright 2012 Freescale Semiconductor, Inc.
- * Copyright 2012 Linaro Ltd.
- *
- * The code contained herein is licensed under the GNU General Public
- * License. You may obtain a copy of the GNU General Public License
- * Version 2 or later at the following locations:
- *
- * http://www.opensource.org/licenses/gpl-license.html
- * http://www.gnu.org/copyleft/gpl.html
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Copyright 2012 Freescale Semiconductor, Inc.
+// Copyright 2012 Linaro Ltd.
#include <linux/gpio.h>
#include <linux/module.h>
-/*
- * imx-mc13783.c -- SoC audio for imx based boards with mc13783 codec
- *
- * Copyright 2012 Philippe Retornaz, <philippe.retornaz@epfl.ch>
- *
- * Heavly based on phycore-mc13783:
- * Copyright 2009 Sascha Hauer, Pengutronix <s.hauer@pengutronix.de>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// imx-mc13783.c -- SoC audio for imx based boards with mc13783 codec
+//
+// Copyright 2012 Philippe Retornaz, <philippe.retornaz@epfl.ch>
+//
+// Heavly based on phycore-mc13783:
+// Copyright 2009 Sascha Hauer, Pengutronix <s.hauer@pengutronix.de>
#include <linux/module.h>
#include <linux/moduleparam.h>
-/*
- * imx-pcm-fiq.c -- ALSA Soc Audio Layer
- *
- * Copyright 2009 Sascha Hauer <s.hauer@pengutronix.de>
- *
- * This code is based on code copyrighted by Freescale,
- * Liam Girdwood, Javier Martin and probably others.
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+// imx-pcm-fiq.c -- ALSA Soc Audio Layer
+//
+// Copyright 2009 Sascha Hauer <s.hauer@pengutronix.de>
+//
+// This code is based on code copyrighted by Freescale,
+// Liam Girdwood, Javier Martin and probably others.
+
#include <linux/clk.h>
#include <linux/delay.h>
#include <linux/device.h>
+/* SPDX-License-Identifier: GPL-2.0+ */
/*
* Copyright 2009 Sascha Hauer <s.hauer@pengutronix.de>
*
* This code is based on code copyrighted by Freescale,
* Liam Girdwood, Javier Martin and probably others.
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
*/
#ifndef _IMX_PCM_H
-/*
- * Copyright (C) 2013 Freescale Semiconductor, Inc.
- *
- * The code contained herein is licensed under the GNU General Public
- * License. You may obtain a copy of the GNU General Public License
- * Version 2 or later at the following locations:
- *
- * http://www.opensource.org/licenses/gpl-license.html
- * http://www.gnu.org/copyleft/gpl.html
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Copyright (C) 2013 Freescale Semiconductor, Inc.
#include <linux/module.h>
#include <linux/of_platform.h>
-/*
- * imx-ssi.c -- ALSA Soc Audio Layer
- *
- * Copyright 2009 Sascha Hauer <s.hauer@pengutronix.de>
- *
- * This code is based on code copyrighted by Freescale,
- * Liam Girdwood, Javier Martin and probably others.
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- *
- * The i.MX SSI core has some nasty limitations in AC97 mode. While most
- * sane processor vendors have a FIFO per AC97 slot, the i.MX has only
- * one FIFO which combines all valid receive slots. We cannot even select
- * which slots we want to receive. The WM9712 with which this driver
- * was developed with always sends GPIO status data in slot 12 which
- * we receive in our (PCM-) data stream. The only chance we have is to
- * manually skip this data in the FIQ handler. With sampling rates different
- * from 48000Hz not every frame has valid receive data, so the ratio
- * between pcm data and GPIO status data changes. Our FIQ handler is not
- * able to handle this, hence this driver only works with 48000Hz sampling
- * rate.
- * Reading and writing AC97 registers is another challenge. The core
- * provides us status bits when the read register is updated with *another*
- * value. When we read the same register two times (and the register still
- * contains the same value) these status bits are not set. We work
- * around this by not polling these bits but only wait a fixed delay.
- *
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// imx-ssi.c -- ALSA Soc Audio Layer
+//
+// Copyright 2009 Sascha Hauer <s.hauer@pengutronix.de>
+//
+// This code is based on code copyrighted by Freescale,
+// Liam Girdwood, Javier Martin and probably others.
+//
+// The i.MX SSI core has some nasty limitations in AC97 mode. While most
+// sane processor vendors have a FIFO per AC97 slot, the i.MX has only
+// one FIFO which combines all valid receive slots. We cannot even select
+// which slots we want to receive. The WM9712 with which this driver
+// was developed with always sends GPIO status data in slot 12 which
+// we receive in our (PCM-) data stream. The only chance we have is to
+// manually skip this data in the FIQ handler. With sampling rates different
+// from 48000Hz not every frame has valid receive data, so the ratio
+// between pcm data and GPIO status data changes. Our FIQ handler is not
+// able to handle this, hence this driver only works with 48000Hz sampling
+// rate.
+// Reading and writing AC97 registers is another challenge. The core
+// provides us status bits when the read register is updated with *another*
+// value. When we read the same register two times (and the register still
+// contains the same value) these status bits are not set. We work
+// around this by not polling these bits but only wait a fixed delay.
#include <linux/clk.h>
#include <linux/delay.h>
-/*
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
+/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _IMX_SSI_H
#define _IMX_SSI_H
-/*
- * Freescale MPC5200 PSC DMA
- * ALSA SoC Platform driver
- *
- * Copyright (C) 2008 Secret Lab Technologies Ltd.
- * Copyright (C) 2009 Jon Smirl, Digispeaker
- */
+// SPDX-License-Identifier: GPL-2.0-only
+//
+// Freescale MPC5200 PSC DMA
+// ALSA SoC Platform driver
+//
+// Copyright (C) 2008 Secret Lab Technologies Ltd.
+// Copyright (C) 2009 Jon Smirl, Digispeaker
#include <linux/module.h>
#include <linux/of_device.h>
-/*
- * linux/sound/mpc5200-ac97.c -- AC97 support for the Freescale MPC52xx chip.
- *
- * Copyright (C) 2009 Jon Smirl, Digispeaker
- * Author: Jon Smirl <jonsmirl@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// linux/sound/mpc5200-ac97.c -- AC97 support for the Freescale MPC52xx chip.
+//
+// Copyright (C) 2009 Jon Smirl, Digispeaker
+// Author: Jon Smirl <jonsmirl@gmail.com>
#include <linux/module.h>
#include <linux/of_device.h>
-/*
- * Freescale MPC5200 PSC in I2S mode
- * ALSA SoC Digital Audio Interface (DAI) driver
- *
- * Copyright (C) 2008 Secret Lab Technologies Ltd.
- * Copyright (C) 2009 Jon Smirl, Digispeaker
- */
+// SPDX-License-Identifier: GPL-2.0-only
+//
+// Freescale MPC5200 PSC in I2S mode
+// ALSA SoC Digital Audio Interface (DAI) driver
+//
+// Copyright (C) 2008 Secret Lab Technologies Ltd.
+// Copyright (C) 2009 Jon Smirl, Digispeaker
#include <linux/module.h>
#include <linux/of_device.h>
-/**
- * Freescale MPC8610HPCD ALSA SoC Machine driver
- *
- * Author: Timur Tabi <timur@freescale.com>
- *
- * Copyright 2007-2010 Freescale Semiconductor, Inc.
- *
- * This file is licensed under the terms of the GNU General Public License
- * version 2. This program is licensed "as is" without any warranty of any
- * kind, whether express or implied.
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// Freescale MPC8610HPCD ALSA SoC Machine driver
+//
+// Author: Timur Tabi <timur@freescale.com>
+//
+// Copyright 2007-2010 Freescale Semiconductor, Inc.
#include <linux/module.h>
#include <linux/interrupt.h>
-/*
- * mx27vis-aic32x4.c
- *
- * Copyright 2011 Vista Silicon S.L.
- *
- * Author: Javier Martin <javier.martin@vista-silicon.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// mx27vis-aic32x4.c
+//
+// Copyright 2011 Vista Silicon S.L.
+//
+// Author: Javier Martin <javier.martin@vista-silicon.com>
#include <linux/module.h>
#include <linux/moduleparam.h>
-/**
- * Freescale P1022DS ALSA SoC Machine driver
- *
- * Author: Timur Tabi <timur@freescale.com>
- *
- * Copyright 2010 Freescale Semiconductor, Inc.
- *
- * This file is licensed under the terms of the GNU General Public License
- * version 2. This program is licensed "as is" without any warranty of any
- * kind, whether express or implied.
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// Freescale P1022DS ALSA SoC Machine driver
+//
+// Author: Timur Tabi <timur@freescale.com>
+//
+// Copyright 2010 Freescale Semiconductor, Inc.
#include <linux/module.h>
#include <linux/fsl/guts.h>
-/**
- * Freescale P1022RDK ALSA SoC Machine driver
- *
- * Author: Timur Tabi <timur@freescale.com>
- *
- * Copyright 2012 Freescale Semiconductor, Inc.
- *
- * This file is licensed under the terms of the GNU General Public License
- * version 2. This program is licensed "as is" without any warranty of any
- * kind, whether express or implied.
- *
- * Note: in order for audio to work correctly, the output controls need
- * to be enabled, because they control the clock. So for playback, for
- * example:
- *
- * amixer sset 'Left Output Mixer PCM' on
- * amixer sset 'Right Output Mixer PCM' on
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// Freescale P1022RDK ALSA SoC Machine driver
+//
+// Author: Timur Tabi <timur@freescale.com>
+//
+// Copyright 2012 Freescale Semiconductor, Inc.
+//
+// Note: in order for audio to work correctly, the output controls need
+// to be enabled, because they control the clock. So for playback, for
+// example:
+//
+// amixer sset 'Left Output Mixer PCM' on
+// amixer sset 'Right Output Mixer PCM' on
#include <linux/module.h>
#include <linux/fsl/guts.h>
-/*
- * Phytec pcm030 driver for the PSC of the Freescale MPC52xx
- * configured as AC97 interface
- *
- * Copyright 2008 Jon Smirl, Digispeaker
- * Author: Jon Smirl <jonsmirl@gmail.com>
- *
- * This file is licensed under the terms of the GNU General Public License
- * version 2. This program is licensed "as is" without any warranty of any
- * kind, whether express or implied.
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// Phytec pcm030 driver for the PSC of the Freescale MPC52xx
+// configured as AC97 interface
+//
+// Copyright 2008 Jon Smirl, Digispeaker
+// Author: Jon Smirl <jonsmirl@gmail.com>
#include <linux/init.h>
#include <linux/module.h>
-/*
- * phycore-ac97.c -- SoC audio for imx_phycore in AC97 mode
- *
- * Copyright 2009 Sascha Hauer, Pengutronix <s.hauer@pengutronix.de>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// phycore-ac97.c -- SoC audio for imx_phycore in AC97 mode
+//
+// Copyright 2009 Sascha Hauer, Pengutronix <s.hauer@pengutronix.de>
#include <linux/module.h>
#include <linux/moduleparam.h>
-/*
- * wm1133-ev1.c - Audio for WM1133-EV1 on i.MX31ADS
- *
- * Copyright (c) 2010 Wolfson Microelectronics plc
- * Author: Mark Brown <broonie@opensource.wolfsonmicro.com>
- *
- * Based on an earlier driver for the same hardware by Liam Girdwood.
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// wm1133-ev1.c - Audio for WM1133-EV1 on i.MX31ADS
+//
+// Copyright (c) 2010 Wolfson Microelectronics plc
+// Author: Mark Brown <broonie@opensource.wolfsonmicro.com>
+//
+// Based on an earlier driver for the same hardware by Liam Girdwood.
#include <linux/platform_device.h>
#include <linux/clk.h>
#define DPCM_SELECTABLE 1
-struct graph_priv {
- struct snd_soc_card snd_card;
- struct graph_dai_props {
- struct asoc_simple_dai *cpu_dai;
- struct asoc_simple_dai *codec_dai;
- struct snd_soc_dai_link_component codecs; /* single codec */
- struct snd_soc_dai_link_component platforms;
- struct asoc_simple_card_data adata;
- struct snd_soc_codec_conf *codec_conf;
- unsigned int mclk_fs;
- } *dai_props;
- struct asoc_simple_jack hp_jack;
- struct asoc_simple_jack mic_jack;
- struct snd_soc_dai_link *dai_link;
- struct asoc_simple_dai *dais;
- struct snd_soc_codec_conf *codec_conf;
- struct gpio_desc *pa_gpio;
-};
-
-struct link_info {
- int dais; /* number of dai */
- int link; /* number of link */
- int conf; /* number of codec_conf */
- int cpu; /* turn for CPU / Codec */
-};
-
-#define graph_priv_to_card(priv) (&(priv)->snd_card)
-#define graph_priv_to_props(priv, i) ((priv)->dai_props + (i))
-#define graph_priv_to_dev(priv) (graph_priv_to_card(priv)->dev)
-#define graph_priv_to_link(priv, i) (graph_priv_to_card(priv)->dai_link + (i))
-
#define PREFIX "audio-graph-card,"
static int graph_outdrv_event(struct snd_soc_dapm_widget *w,
int event)
{
struct snd_soc_dapm_context *dapm = w->dapm;
- struct graph_priv *priv = snd_soc_card_get_drvdata(dapm->card);
+ struct asoc_simple_priv *priv = snd_soc_card_get_drvdata(dapm->card);
switch (event) {
case SND_SOC_DAPM_POST_PMU:
SND_SOC_DAPM_POST_PMU | SND_SOC_DAPM_PRE_PMD),
};
-static int graph_startup(struct snd_pcm_substream *substream)
+static const struct snd_soc_ops graph_ops = {
+ .startup = asoc_simple_startup,
+ .shutdown = asoc_simple_shutdown,
+ .hw_params = asoc_simple_hw_params,
+};
+
+static int graph_get_dai_id(struct device_node *ep)
{
- struct snd_soc_pcm_runtime *rtd = substream->private_data;
- struct graph_priv *priv = snd_soc_card_get_drvdata(rtd->card);
- struct graph_dai_props *dai_props = graph_priv_to_props(priv, rtd->num);
+ struct device_node *node;
+ struct device_node *endpoint;
+ struct of_endpoint info;
+ int i, id;
int ret;
- ret = asoc_simple_card_clk_enable(dai_props->cpu_dai);
- if (ret)
+ /* use driver specified DAI ID if exist */
+ ret = snd_soc_get_dai_id(ep);
+ if (ret != -ENOTSUPP)
return ret;
- ret = asoc_simple_card_clk_enable(dai_props->codec_dai);
- if (ret)
- asoc_simple_card_clk_disable(dai_props->cpu_dai);
+ /* use endpoint/port reg if exist */
+ ret = of_graph_parse_endpoint(ep, &info);
+ if (ret == 0) {
+ /*
+ * Because it will count port/endpoint if it doesn't have "reg".
+ * But, we can't judge whether it has "no reg", or "reg = <0>"
+ * only of_graph_parse_endpoint().
+ * We need to check "reg" property
+ */
+ if (of_get_property(ep, "reg", NULL))
+ return info.id;
- return ret;
-}
+ node = of_get_parent(ep);
+ of_node_put(node);
+ if (of_get_property(node, "reg", NULL))
+ return info.port;
+ }
+ node = of_graph_get_port_parent(ep);
-static void graph_shutdown(struct snd_pcm_substream *substream)
-{
- struct snd_soc_pcm_runtime *rtd = substream->private_data;
- struct graph_priv *priv = snd_soc_card_get_drvdata(rtd->card);
- struct graph_dai_props *dai_props = graph_priv_to_props(priv, rtd->num);
+ /*
+ * Non HDMI sound case, counting port/endpoint on its DT
+ * is enough. Let's count it.
+ */
+ i = 0;
+ id = -1;
+ for_each_endpoint_of_node(node, endpoint) {
+ if (endpoint == ep)
+ id = i;
+ i++;
+ }
- asoc_simple_card_clk_disable(dai_props->cpu_dai);
+ of_node_put(node);
+
+ if (id < 0)
+ return -ENODEV;
- asoc_simple_card_clk_disable(dai_props->codec_dai);
+ return id;
}
-static int graph_hw_params(struct snd_pcm_substream *substream,
- struct snd_pcm_hw_params *params)
+static int asoc_simple_parse_dai(struct device_node *ep,
+ struct snd_soc_dai_link_component *dlc,
+ struct device_node **dai_of_node,
+ const char **dai_name,
+ int *is_single_link)
{
- struct snd_soc_pcm_runtime *rtd = substream->private_data;
- struct snd_soc_dai *codec_dai = rtd->codec_dai;
- struct snd_soc_dai *cpu_dai = rtd->cpu_dai;
- struct graph_priv *priv = snd_soc_card_get_drvdata(rtd->card);
- struct graph_dai_props *dai_props = graph_priv_to_props(priv, rtd->num);
- unsigned int mclk, mclk_fs = 0;
- int ret = 0;
-
- if (dai_props->mclk_fs)
- mclk_fs = dai_props->mclk_fs;
-
- if (mclk_fs) {
- mclk = params_rate(params) * mclk_fs;
- ret = snd_soc_dai_set_sysclk(codec_dai, 0, mclk,
- SND_SOC_CLOCK_IN);
- if (ret && ret != -ENOTSUPP)
- goto err;
-
- ret = snd_soc_dai_set_sysclk(cpu_dai, 0, mclk,
- SND_SOC_CLOCK_OUT);
- if (ret && ret != -ENOTSUPP)
- goto err;
+ struct device_node *node;
+ struct of_phandle_args args;
+ int ret;
+
+ /*
+ * Use snd_soc_dai_link_component instead of legacy style.
+ * It is only for codec, but cpu will be supported in the future.
+ * see
+ * soc-core.c :: snd_soc_init_multicodec()
+ */
+ if (dlc) {
+ dai_name = &dlc->dai_name;
+ dai_of_node = &dlc->of_node;
}
- return 0;
-err:
- return ret;
-}
-static const struct snd_soc_ops graph_ops = {
- .startup = graph_startup,
- .shutdown = graph_shutdown,
- .hw_params = graph_hw_params,
-};
+ if (!ep)
+ return 0;
+ if (!dai_name)
+ return 0;
-static int graph_dai_init(struct snd_soc_pcm_runtime *rtd)
-{
- struct graph_priv *priv = snd_soc_card_get_drvdata(rtd->card);
- struct graph_dai_props *dai_props = graph_priv_to_props(priv, rtd->num);
- int ret = 0;
+ node = of_graph_get_port_parent(ep);
- ret = asoc_simple_card_init_dai(rtd->codec_dai,
- dai_props->codec_dai);
- if (ret < 0)
- return ret;
+ /* Get dai->name */
+ args.np = node;
+ args.args[0] = graph_get_dai_id(ep);
+ args.args_count = (of_graph_get_endpoint_count(node) > 1);
- ret = asoc_simple_card_init_dai(rtd->cpu_dai,
- dai_props->cpu_dai);
+ ret = snd_soc_get_dai_name(&args, dai_name);
if (ret < 0)
return ret;
- return 0;
-}
-
-static int graph_be_hw_params_fixup(struct snd_soc_pcm_runtime *rtd,
- struct snd_pcm_hw_params *params)
-{
- struct graph_priv *priv = snd_soc_card_get_drvdata(rtd->card);
- struct graph_dai_props *dai_props = graph_priv_to_props(priv, rtd->num);
+ *dai_of_node = node;
- asoc_simple_card_convert_fixup(&dai_props->adata, params);
+ if (is_single_link)
+ *is_single_link = of_graph_get_endpoint_count(node) == 1;
return 0;
}
-static void graph_get_conversion(struct device *dev,
- struct device_node *ep,
- struct asoc_simple_card_data *adata)
+static void graph_parse_convert(struct device *dev,
+ struct device_node *ep,
+ struct asoc_simple_data *adata)
{
struct device_node *top = dev->of_node;
struct device_node *port = of_get_parent(ep);
struct device_node *ports = of_get_parent(port);
struct device_node *node = of_graph_get_port_parent(ep);
- asoc_simple_card_parse_convert(dev, top, NULL, adata);
- asoc_simple_card_parse_convert(dev, node, PREFIX, adata);
- asoc_simple_card_parse_convert(dev, ports, NULL, adata);
- asoc_simple_card_parse_convert(dev, port, NULL, adata);
- asoc_simple_card_parse_convert(dev, ep, NULL, adata);
+ asoc_simple_parse_convert(dev, top, NULL, adata);
+ asoc_simple_parse_convert(dev, node, PREFIX, adata);
+ asoc_simple_parse_convert(dev, ports, NULL, adata);
+ asoc_simple_parse_convert(dev, port, NULL, adata);
+ asoc_simple_parse_convert(dev, ep, NULL, adata);
+
+ of_node_put(port);
+ of_node_put(ports);
+ of_node_put(node);
}
-static int graph_dai_link_of_dpcm(struct graph_priv *priv,
+static void graph_parse_mclk_fs(struct device_node *top,
+ struct device_node *ep,
+ struct simple_dai_props *props)
+{
+ struct device_node *port = of_get_parent(ep);
+ struct device_node *ports = of_get_parent(port);
+ struct device_node *node = of_graph_get_port_parent(ep);
+
+ of_property_read_u32(top, "mclk-fs", &props->mclk_fs);
+ of_property_read_u32(ports, "mclk-fs", &props->mclk_fs);
+ of_property_read_u32(port, "mclk-fs", &props->mclk_fs);
+ of_property_read_u32(ep, "mclk-fs", &props->mclk_fs);
+
+ of_node_put(port);
+ of_node_put(ports);
+ of_node_put(node);
+}
+
+static int graph_dai_link_of_dpcm(struct asoc_simple_priv *priv,
struct device_node *cpu_ep,
struct device_node *codec_ep,
struct link_info *li,
int dup_codec)
{
- struct device *dev = graph_priv_to_dev(priv);
- struct snd_soc_dai_link *dai_link = graph_priv_to_link(priv, li->link);
- struct graph_dai_props *dai_props = graph_priv_to_props(priv, li->link);
+ struct device *dev = simple_priv_to_dev(priv);
+ struct snd_soc_dai_link *dai_link = simple_priv_to_link(priv, li->link);
+ struct simple_dai_props *dai_props = simple_priv_to_props(priv, li->link);
struct device_node *top = dev->of_node;
struct device_node *ep = li->cpu ? cpu_ep : codec_ep;
struct device_node *port;
dev_dbg(dev, "link_of DPCM (%pOF)\n", ep);
- of_property_read_u32(top, "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(ports, "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(port, "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(ep, "mclk-fs", &dai_props->mclk_fs);
-
- graph_get_conversion(dev, ep, &dai_props->adata);
-
of_node_put(ports);
of_node_put(port);
of_node_put(node);
if (li->cpu) {
+ int is_single_links = 0;
/* BE is dummy */
codecs->of_node = NULL;
dai =
dai_props->cpu_dai = &priv->dais[li->dais++];
- ret = asoc_simple_card_parse_graph_cpu(ep, dai_link);
+ ret = asoc_simple_parse_cpu(ep, dai_link, &is_single_links);
if (ret)
return ret;
- ret = asoc_simple_card_parse_clk_cpu(dev, ep, dai_link, dai);
+ ret = asoc_simple_parse_clk_cpu(dev, ep, dai_link, dai);
if (ret < 0)
return ret;
- ret = asoc_simple_card_set_dailink_name(dev, dai_link,
- "fe.%s",
- dai_link->cpu_dai_name);
+ ret = asoc_simple_set_dailink_name(dev, dai_link,
+ "fe.%s",
+ dai_link->cpu_dai_name);
if (ret < 0)
return ret;
/* card->num_links includes Codec */
- asoc_simple_card_canonicalize_cpu(dai_link,
- of_graph_get_endpoint_count(dai_link->cpu_of_node) == 1);
+ asoc_simple_canonicalize_cpu(dai_link, is_single_links);
} else {
struct snd_soc_codec_conf *cconf;
/* BE settings */
dai_link->no_pcm = 1;
- dai_link->be_hw_params_fixup = graph_be_hw_params_fixup;
+ dai_link->be_hw_params_fixup = asoc_simple_be_hw_params_fixup;
dai =
dai_props->codec_dai = &priv->dais[li->dais++];
cconf =
dai_props->codec_conf = &priv->codec_conf[li->conf++];
- ret = asoc_simple_card_parse_graph_codec(ep, dai_link);
+ ret = asoc_simple_parse_codec(ep, dai_link);
if (ret < 0)
return ret;
- ret = asoc_simple_card_parse_clk_codec(dev, ep, dai_link, dai);
+ ret = asoc_simple_parse_clk_codec(dev, ep, dai_link, dai);
if (ret < 0)
return ret;
- ret = asoc_simple_card_set_dailink_name(dev, dai_link,
- "be.%s",
- codecs->dai_name);
+ ret = asoc_simple_set_dailink_name(dev, dai_link,
+ "be.%s",
+ codecs->dai_name);
if (ret < 0)
return ret;
"prefix");
}
- asoc_simple_card_canonicalize_platform(dai_link);
+ graph_parse_convert(dev, ep, &dai_props->adata);
+ graph_parse_mclk_fs(top, ep, dai_props);
- ret = asoc_simple_card_of_parse_tdm(ep, dai);
+ asoc_simple_canonicalize_platform(dai_link);
+
+ ret = asoc_simple_parse_tdm(ep, dai);
if (ret)
return ret;
- ret = asoc_simple_card_parse_daifmt(dev, cpu_ep, codec_ep,
- NULL, &dai_link->dai_fmt);
+ ret = asoc_simple_parse_daifmt(dev, cpu_ep, codec_ep,
+ NULL, &dai_link->dai_fmt);
if (ret < 0)
return ret;
dai_link->dpcm_playback = 1;
dai_link->dpcm_capture = 1;
dai_link->ops = &graph_ops;
- dai_link->init = graph_dai_init;
+ dai_link->init = asoc_simple_dai_init;
return 0;
}
-static int graph_dai_link_of(struct graph_priv *priv,
+static int graph_dai_link_of(struct asoc_simple_priv *priv,
struct device_node *cpu_ep,
struct device_node *codec_ep,
struct link_info *li)
{
- struct device *dev = graph_priv_to_dev(priv);
- struct snd_soc_dai_link *dai_link = graph_priv_to_link(priv, li->link);
- struct graph_dai_props *dai_props = graph_priv_to_props(priv, li->link);
+ struct device *dev = simple_priv_to_dev(priv);
+ struct snd_soc_dai_link *dai_link = simple_priv_to_link(priv, li->link);
+ struct simple_dai_props *dai_props = simple_priv_to_props(priv, li->link);
struct device_node *top = dev->of_node;
- struct device_node *cpu_port;
- struct device_node *cpu_ports;
- struct device_node *codec_port;
- struct device_node *codec_ports;
struct asoc_simple_dai *cpu_dai;
struct asoc_simple_dai *codec_dai;
- int ret;
+ int ret, single_cpu;
/* Do it only CPU turn */
if (!li->cpu)
return 0;
- cpu_port = of_get_parent(cpu_ep);
- cpu_ports = of_get_parent(cpu_port);
- codec_port = of_get_parent(codec_ep);
- codec_ports = of_get_parent(codec_port);
-
dev_dbg(dev, "link_of (%pOF)\n", cpu_ep);
li->link++;
dai_props->codec_dai = &priv->dais[li->dais++];
/* Factor to mclk, used in hw_params() */
- of_property_read_u32(top, "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(cpu_ports, "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(codec_ports, "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(cpu_port, "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(codec_port, "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(cpu_ep, "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(codec_ep, "mclk-fs", &dai_props->mclk_fs);
- of_node_put(cpu_port);
- of_node_put(cpu_ports);
- of_node_put(codec_port);
- of_node_put(codec_ports);
-
- ret = asoc_simple_card_parse_daifmt(dev, cpu_ep, codec_ep,
- NULL, &dai_link->dai_fmt);
+ graph_parse_mclk_fs(top, cpu_ep, dai_props);
+ graph_parse_mclk_fs(top, codec_ep, dai_props);
+
+ ret = asoc_simple_parse_daifmt(dev, cpu_ep, codec_ep,
+ NULL, &dai_link->dai_fmt);
if (ret < 0)
return ret;
- ret = asoc_simple_card_parse_graph_cpu(cpu_ep, dai_link);
+ ret = asoc_simple_parse_cpu(cpu_ep, dai_link, &single_cpu);
if (ret < 0)
return ret;
- ret = asoc_simple_card_parse_graph_codec(codec_ep, dai_link);
+ ret = asoc_simple_parse_codec(codec_ep, dai_link);
if (ret < 0)
return ret;
- ret = asoc_simple_card_of_parse_tdm(cpu_ep, cpu_dai);
+ ret = asoc_simple_parse_tdm(cpu_ep, cpu_dai);
if (ret < 0)
return ret;
- ret = asoc_simple_card_of_parse_tdm(codec_ep, codec_dai);
+ ret = asoc_simple_parse_tdm(codec_ep, codec_dai);
if (ret < 0)
return ret;
- ret = asoc_simple_card_parse_clk_cpu(dev, cpu_ep, dai_link, cpu_dai);
+ ret = asoc_simple_parse_clk_cpu(dev, cpu_ep, dai_link, cpu_dai);
if (ret < 0)
return ret;
- ret = asoc_simple_card_parse_clk_codec(dev, codec_ep, dai_link, codec_dai);
+ ret = asoc_simple_parse_clk_codec(dev, codec_ep, dai_link, codec_dai);
if (ret < 0)
return ret;
- ret = asoc_simple_card_set_dailink_name(dev, dai_link,
- "%s-%s",
- dai_link->cpu_dai_name,
- dai_link->codecs->dai_name);
+ ret = asoc_simple_set_dailink_name(dev, dai_link,
+ "%s-%s",
+ dai_link->cpu_dai_name,
+ dai_link->codecs->dai_name);
if (ret < 0)
return ret;
dai_link->ops = &graph_ops;
- dai_link->init = graph_dai_init;
+ dai_link->init = asoc_simple_dai_init;
- asoc_simple_card_canonicalize_platform(dai_link);
- asoc_simple_card_canonicalize_cpu(dai_link,
- of_graph_get_endpoint_count(dai_link->cpu_of_node) == 1);
+ asoc_simple_canonicalize_cpu(dai_link, single_cpu);
+ asoc_simple_canonicalize_platform(dai_link);
return 0;
}
-static int graph_for_each_link(struct graph_priv *priv,
+static int graph_for_each_link(struct asoc_simple_priv *priv,
struct link_info *li,
- int (*func_noml)(struct graph_priv *priv,
+ int (*func_noml)(struct asoc_simple_priv *priv,
struct device_node *cpu_ep,
struct device_node *codec_ep,
struct link_info *li),
- int (*func_dpcm)(struct graph_priv *priv,
+ int (*func_dpcm)(struct asoc_simple_priv *priv,
struct device_node *cpu_ep,
struct device_node *codec_ep,
struct link_info *li, int dup_codec))
{
struct of_phandle_iterator it;
- struct device *dev = graph_priv_to_dev(priv);
+ struct device *dev = simple_priv_to_dev(priv);
struct device_node *node = dev->of_node;
struct device_node *cpu_port;
struct device_node *cpu_ep;
struct device_node *codec_ep;
struct device_node *codec_port;
struct device_node *codec_port_old = NULL;
- struct asoc_simple_card_data adata;
+ struct asoc_simple_data adata;
uintptr_t dpcm_selectable = (uintptr_t)of_device_get_match_data(dev);
int rc, ret;
/* get convert-xxx property */
memset(&adata, 0, sizeof(adata));
- graph_get_conversion(dev, codec_ep, &adata);
- graph_get_conversion(dev, cpu_ep, &adata);
+ graph_parse_convert(dev, codec_ep, &adata);
+ graph_parse_convert(dev, cpu_ep, &adata);
/*
* It is DPCM
return 0;
}
-static int graph_parse_of(struct graph_priv *priv)
+static int graph_parse_of(struct asoc_simple_priv *priv)
{
- struct snd_soc_card *card = graph_priv_to_card(priv);
+ struct snd_soc_card *card = simple_priv_to_card(priv);
struct link_info li;
int ret;
- ret = asoc_simple_card_of_parse_widgets(card, NULL);
+ ret = asoc_simple_parse_widgets(card, NULL);
if (ret < 0)
return ret;
- ret = asoc_simple_card_of_parse_routing(card, NULL);
+ ret = asoc_simple_parse_routing(card, NULL);
if (ret < 0)
return ret;
return ret;
}
- return asoc_simple_card_parse_card_name(card, NULL);
+ return asoc_simple_parse_card_name(card, NULL);
}
-static int graph_count_noml(struct graph_priv *priv,
+static int graph_count_noml(struct asoc_simple_priv *priv,
struct device_node *cpu_ep,
struct device_node *codec_ep,
struct link_info *li)
{
- struct device *dev = graph_priv_to_dev(priv);
+ struct device *dev = simple_priv_to_dev(priv);
li->link += 1; /* 1xCPU-Codec */
li->dais += 2; /* 1xCPU + 1xCodec */
return 0;
}
-static int graph_count_dpcm(struct graph_priv *priv,
+static int graph_count_dpcm(struct asoc_simple_priv *priv,
struct device_node *cpu_ep,
struct device_node *codec_ep,
struct link_info *li,
int dup_codec)
{
- struct device *dev = graph_priv_to_dev(priv);
+ struct device *dev = simple_priv_to_dev(priv);
li->link++; /* 1xCPU-dummy */
li->dais++; /* 1xCPU */
return 0;
}
-static void graph_get_dais_count(struct graph_priv *priv,
+static void graph_get_dais_count(struct asoc_simple_priv *priv,
struct link_info *li)
{
- struct device *dev = graph_priv_to_dev(priv);
+ struct device *dev = simple_priv_to_dev(priv);
/*
* link_num : number of links.
static int graph_card_probe(struct snd_soc_card *card)
{
- struct graph_priv *priv = snd_soc_card_get_drvdata(card);
+ struct asoc_simple_priv *priv = snd_soc_card_get_drvdata(card);
int ret;
- ret = asoc_simple_card_init_hp(card, &priv->hp_jack, NULL);
+ ret = asoc_simple_init_hp(card, &priv->hp_jack, NULL);
if (ret < 0)
return ret;
- ret = asoc_simple_card_init_mic(card, &priv->mic_jack, NULL);
+ ret = asoc_simple_init_mic(card, &priv->mic_jack, NULL);
if (ret < 0)
return ret;
static int graph_probe(struct platform_device *pdev)
{
- struct graph_priv *priv;
- struct snd_soc_dai_link *dai_link;
- struct graph_dai_props *dai_props;
- struct asoc_simple_dai *dais;
+ struct asoc_simple_priv *priv;
struct device *dev = &pdev->dev;
struct snd_soc_card *card;
- struct snd_soc_codec_conf *cconf;
struct link_info li;
- int ret, i;
+ int ret;
/* Allocate the private data and the DAI link array */
priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
if (!priv)
return -ENOMEM;
- card = graph_priv_to_card(priv);
+ card = simple_priv_to_card(priv);
card->owner = THIS_MODULE;
card->dev = dev;
card->dapm_widgets = graph_dapm_widgets;
if (!li.link || !li.dais)
return -EINVAL;
- dai_props = devm_kcalloc(dev, li.link, sizeof(*dai_props), GFP_KERNEL);
- dai_link = devm_kcalloc(dev, li.link, sizeof(*dai_link), GFP_KERNEL);
- dais = devm_kcalloc(dev, li.dais, sizeof(*dais), GFP_KERNEL);
- cconf = devm_kcalloc(dev, li.conf, sizeof(*cconf), GFP_KERNEL);
- if (!dai_props || !dai_link || !dais)
- return -ENOMEM;
-
- /*
- * Use snd_soc_dai_link_component instead of legacy style
- * It is codec only. but cpu/platform will be supported in the future.
- * see
- * soc-core.c :: snd_soc_init_multicodec()
- */
- for (i = 0; i < li.link; i++) {
- dai_link[i].codecs = &dai_props[i].codecs;
- dai_link[i].num_codecs = 1;
- dai_link[i].platforms = &dai_props[i].platforms;
- dai_link[i].num_platforms = 1;
- }
+ ret = asoc_simple_init_priv(priv, &li);
+ if (ret < 0)
+ return ret;
priv->pa_gpio = devm_gpiod_get_optional(dev, "pa", GPIOD_OUT_LOW);
if (IS_ERR(priv->pa_gpio)) {
return ret;
}
- priv->dai_props = dai_props;
- priv->dai_link = dai_link;
- priv->dais = dais;
- priv->codec_conf = cconf;
-
- card->dai_link = dai_link;
- card->num_links = li.link;
- card->codec_conf = cconf;
- card->num_configs = li.conf;
-
ret = graph_parse_of(priv);
if (ret < 0) {
if (ret != -EPROBE_DEFER)
snd_soc_card_set_drvdata(card, priv);
+ asoc_simple_debug_info(priv);
+
ret = devm_snd_soc_register_card(dev, card);
if (ret < 0)
goto err;
return 0;
err:
- asoc_simple_card_clean_reference(card);
+ asoc_simple_clean_reference(card);
return ret;
}
{
struct snd_soc_card *card = platform_get_drvdata(pdev);
- return asoc_simple_card_clean_reference(card);
+ return asoc_simple_clean_reference(card);
}
static const struct of_device_id graph_of_match[] = {
#include <sound/jack.h>
#include <sound/simple_card_utils.h>
-void asoc_simple_card_convert_fixup(struct asoc_simple_card_data *data,
- struct snd_pcm_hw_params *params)
+void asoc_simple_convert_fixup(struct asoc_simple_data *data,
+ struct snd_pcm_hw_params *params)
{
struct snd_interval *rate = hw_param_interval(params,
SNDRV_PCM_HW_PARAM_RATE);
channels->min =
channels->max = data->convert_channels;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_convert_fixup);
+EXPORT_SYMBOL_GPL(asoc_simple_convert_fixup);
-void asoc_simple_card_parse_convert(struct device *dev,
- struct device_node *np,
- char *prefix,
- struct asoc_simple_card_data *data)
+void asoc_simple_parse_convert(struct device *dev,
+ struct device_node *np,
+ char *prefix,
+ struct asoc_simple_data *data)
{
char prop[128];
/* channels transfer */
snprintf(prop, sizeof(prop), "%s%s", prefix, "convert-channels");
of_property_read_u32(np, prop, &data->convert_channels);
-
- dev_dbg(dev, "convert_rate %d\n", data->convert_rate);
- dev_dbg(dev, "convert_channels %d\n", data->convert_channels);
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_parse_convert);
+EXPORT_SYMBOL_GPL(asoc_simple_parse_convert);
-int asoc_simple_card_parse_daifmt(struct device *dev,
- struct device_node *node,
- struct device_node *codec,
- char *prefix,
- unsigned int *retfmt)
+int asoc_simple_parse_daifmt(struct device *dev,
+ struct device_node *node,
+ struct device_node *codec,
+ char *prefix,
+ unsigned int *retfmt)
{
struct device_node *bitclkmaster = NULL;
struct device_node *framemaster = NULL;
*retfmt = daifmt;
- dev_dbg(dev, "format : %04x\n", daifmt);
-
return 0;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_parse_daifmt);
+EXPORT_SYMBOL_GPL(asoc_simple_parse_daifmt);
-int asoc_simple_card_set_dailink_name(struct device *dev,
- struct snd_soc_dai_link *dai_link,
- const char *fmt, ...)
+int asoc_simple_set_dailink_name(struct device *dev,
+ struct snd_soc_dai_link *dai_link,
+ const char *fmt, ...)
{
va_list ap;
char *name = NULL;
dai_link->name = name;
dai_link->stream_name = name;
-
- dev_dbg(dev, "name : %s\n", name);
}
return ret;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_set_dailink_name);
+EXPORT_SYMBOL_GPL(asoc_simple_set_dailink_name);
-int asoc_simple_card_parse_card_name(struct snd_soc_card *card,
- char *prefix)
+int asoc_simple_parse_card_name(struct snd_soc_card *card,
+ char *prefix)
{
int ret;
if (!card->name && card->dai_link)
card->name = card->dai_link->name;
- dev_dbg(card->dev, "Card Name: %s\n", card->name ? card->name : "");
-
return 0;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_parse_card_name);
+EXPORT_SYMBOL_GPL(asoc_simple_parse_card_name);
-int asoc_simple_card_clk_enable(struct asoc_simple_dai *dai)
+static int asoc_simple_clk_enable(struct asoc_simple_dai *dai)
{
if (dai)
return clk_prepare_enable(dai->clk);
return 0;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_clk_enable);
-void asoc_simple_card_clk_disable(struct asoc_simple_dai *dai)
+static void asoc_simple_clk_disable(struct asoc_simple_dai *dai)
{
if (dai)
clk_disable_unprepare(dai->clk);
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_clk_disable);
-
-int asoc_simple_card_parse_clk(struct device *dev,
- struct device_node *node,
- struct device_node *dai_of_node,
- struct asoc_simple_dai *simple_dai,
- const char *dai_name,
- struct snd_soc_dai_link_component *dlc)
+
+int asoc_simple_parse_clk(struct device *dev,
+ struct device_node *node,
+ struct device_node *dai_of_node,
+ struct asoc_simple_dai *simple_dai,
+ const char *dai_name,
+ struct snd_soc_dai_link_component *dlc)
{
struct clk *clk;
u32 val;
* see
* soc-core.c :: snd_soc_init_multicodec()
*/
- if (dlc) {
+ if (dlc)
dai_of_node = dlc->of_node;
- dai_name = dlc->dai_name;
- }
/*
* Parse dai->sysclk come from "clocks = <&xxx>"
if (of_property_read_bool(node, "system-clock-direction-out"))
simple_dai->clk_direction = SND_SOC_CLOCK_OUT;
- dev_dbg(dev, "%s : sysclk = %d, direction %d\n", dai_name,
- simple_dai->sysclk, simple_dai->clk_direction);
-
return 0;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_parse_clk);
-
-int asoc_simple_card_parse_dai(struct device_node *node,
- struct snd_soc_dai_link_component *dlc,
- struct device_node **dai_of_node,
- const char **dai_name,
- const char *list_name,
- const char *cells_name,
- int *is_single_link)
+EXPORT_SYMBOL_GPL(asoc_simple_parse_clk);
+
+int asoc_simple_startup(struct snd_pcm_substream *substream)
{
- struct of_phandle_args args;
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct asoc_simple_priv *priv = snd_soc_card_get_drvdata(rtd->card);
+ struct simple_dai_props *dai_props = simple_priv_to_props(priv, rtd->num);
int ret;
- if (!node)
- return 0;
-
- /*
- * Use snd_soc_dai_link_component instead of legacy style.
- * It is only for codec, but cpu will be supported in the future.
- * see
- * soc-core.c :: snd_soc_init_multicodec()
- */
- if (dlc) {
- dai_name = &dlc->dai_name;
- dai_of_node = &dlc->of_node;
- }
-
- /*
- * Get node via "sound-dai = <&phandle port>"
- * it will be used as xxx_of_node on soc_bind_dai_link()
- */
- ret = of_parse_phandle_with_args(node, list_name, cells_name, 0, &args);
+ ret = asoc_simple_clk_enable(dai_props->cpu_dai);
if (ret)
return ret;
- /* Get dai->name */
- if (dai_name) {
- ret = snd_soc_of_get_dai_name(node, dai_name);
- if (ret < 0)
- return ret;
- }
-
- *dai_of_node = args.np;
-
- if (is_single_link)
- *is_single_link = !args.args_count;
+ ret = asoc_simple_clk_enable(dai_props->codec_dai);
+ if (ret)
+ asoc_simple_clk_disable(dai_props->cpu_dai);
- return 0;
+ return ret;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_parse_dai);
+EXPORT_SYMBOL_GPL(asoc_simple_startup);
-static int asoc_simple_card_get_dai_id(struct device_node *ep)
+void asoc_simple_shutdown(struct snd_pcm_substream *substream)
{
- struct device_node *node;
- struct device_node *endpoint;
- struct of_endpoint info;
- int i, id;
- int ret;
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct asoc_simple_priv *priv = snd_soc_card_get_drvdata(rtd->card);
+ struct simple_dai_props *dai_props =
+ simple_priv_to_props(priv, rtd->num);
- /* use driver specified DAI ID if exist */
- ret = snd_soc_get_dai_id(ep);
- if (ret != -ENOTSUPP)
- return ret;
+ asoc_simple_clk_disable(dai_props->cpu_dai);
- /* use endpoint/port reg if exist */
- ret = of_graph_parse_endpoint(ep, &info);
- if (ret == 0) {
- /*
- * Because it will count port/endpoint if it doesn't have "reg".
- * But, we can't judge whether it has "no reg", or "reg = <0>"
- * only of_graph_parse_endpoint().
- * We need to check "reg" property
- */
- if (of_get_property(ep, "reg", NULL))
- return info.id;
-
- node = of_get_parent(ep);
- of_node_put(node);
- if (of_get_property(node, "reg", NULL))
- return info.port;
- }
- node = of_graph_get_port_parent(ep);
+ asoc_simple_clk_disable(dai_props->codec_dai);
+}
+EXPORT_SYMBOL_GPL(asoc_simple_shutdown);
- /*
- * Non HDMI sound case, counting port/endpoint on its DT
- * is enough. Let's count it.
- */
- i = 0;
- id = -1;
- for_each_endpoint_of_node(node, endpoint) {
- if (endpoint == ep)
- id = i;
- i++;
- }
+static int asoc_simple_set_clk_rate(struct asoc_simple_dai *simple_dai,
+ unsigned long rate)
+{
+ if (!simple_dai)
+ return 0;
- of_node_put(node);
+ if (!simple_dai->clk)
+ return 0;
- if (id < 0)
- return -ENODEV;
+ if (clk_get_rate(simple_dai->clk) == rate)
+ return 0;
- return id;
+ return clk_set_rate(simple_dai->clk, rate);
}
-int asoc_simple_card_parse_graph_dai(struct device_node *ep,
- struct snd_soc_dai_link_component *dlc,
- struct device_node **dai_of_node,
- const char **dai_name)
+int asoc_simple_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
{
- struct device_node *node;
- struct of_phandle_args args;
- int ret;
-
- /*
- * Use snd_soc_dai_link_component instead of legacy style.
- * It is only for codec, but cpu will be supported in the future.
- * see
- * soc-core.c :: snd_soc_init_multicodec()
- */
- if (dlc) {
- dai_name = &dlc->dai_name;
- dai_of_node = &dlc->of_node;
- }
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_dai *codec_dai = rtd->codec_dai;
+ struct snd_soc_dai *cpu_dai = rtd->cpu_dai;
+ struct asoc_simple_priv *priv = snd_soc_card_get_drvdata(rtd->card);
+ struct simple_dai_props *dai_props =
+ simple_priv_to_props(priv, rtd->num);
+ unsigned int mclk, mclk_fs = 0;
+ int ret = 0;
+
+ if (dai_props->mclk_fs)
+ mclk_fs = dai_props->mclk_fs;
+
+ if (mclk_fs) {
+ mclk = params_rate(params) * mclk_fs;
+
+ ret = asoc_simple_set_clk_rate(dai_props->codec_dai, mclk);
+ if (ret < 0)
+ return ret;
- if (!ep)
- return 0;
- if (!dai_name)
- return 0;
+ ret = asoc_simple_set_clk_rate(dai_props->cpu_dai, mclk);
+ if (ret < 0)
+ return ret;
- node = of_graph_get_port_parent(ep);
+ ret = snd_soc_dai_set_sysclk(codec_dai, 0, mclk,
+ SND_SOC_CLOCK_IN);
+ if (ret && ret != -ENOTSUPP)
+ goto err;
- /* Get dai->name */
- args.np = node;
- args.args[0] = asoc_simple_card_get_dai_id(ep);
- args.args_count = (of_graph_get_endpoint_count(node) > 1);
+ ret = snd_soc_dai_set_sysclk(cpu_dai, 0, mclk,
+ SND_SOC_CLOCK_OUT);
+ if (ret && ret != -ENOTSUPP)
+ goto err;
+ }
+ return 0;
+err:
+ return ret;
+}
+EXPORT_SYMBOL_GPL(asoc_simple_hw_params);
- ret = snd_soc_get_dai_name(&args, dai_name);
- if (ret < 0)
- return ret;
+int asoc_simple_be_hw_params_fixup(struct snd_soc_pcm_runtime *rtd,
+ struct snd_pcm_hw_params *params)
+{
+ struct asoc_simple_priv *priv = snd_soc_card_get_drvdata(rtd->card);
+ struct simple_dai_props *dai_props = simple_priv_to_props(priv, rtd->num);
- *dai_of_node = node;
+ asoc_simple_convert_fixup(&dai_props->adata, params);
return 0;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_parse_graph_dai);
+EXPORT_SYMBOL_GPL(asoc_simple_be_hw_params_fixup);
-int asoc_simple_card_init_dai(struct snd_soc_dai *dai,
- struct asoc_simple_dai *simple_dai)
+static int asoc_simple_init_dai(struct snd_soc_dai *dai,
+ struct asoc_simple_dai *simple_dai)
{
int ret;
return 0;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_init_dai);
-void asoc_simple_card_canonicalize_platform(struct snd_soc_dai_link *dai_link)
+int asoc_simple_dai_init(struct snd_soc_pcm_runtime *rtd)
+{
+ struct asoc_simple_priv *priv = snd_soc_card_get_drvdata(rtd->card);
+ struct simple_dai_props *dai_props = simple_priv_to_props(priv, rtd->num);
+ int ret;
+
+ ret = asoc_simple_init_dai(rtd->codec_dai,
+ dai_props->codec_dai);
+ if (ret < 0)
+ return ret;
+
+ ret = asoc_simple_init_dai(rtd->cpu_dai,
+ dai_props->cpu_dai);
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(asoc_simple_dai_init);
+
+void asoc_simple_canonicalize_platform(struct snd_soc_dai_link *dai_link)
{
/* Assumes platform == cpu */
if (!dai_link->platforms->of_node)
dai_link->platforms->of_node = dai_link->cpu_of_node;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_canonicalize_platform);
+EXPORT_SYMBOL_GPL(asoc_simple_canonicalize_platform);
-void asoc_simple_card_canonicalize_cpu(struct snd_soc_dai_link *dai_link,
- int is_single_links)
+void asoc_simple_canonicalize_cpu(struct snd_soc_dai_link *dai_link,
+ int is_single_links)
{
/*
* In soc_bind_dai_link() will check cpu name after
if (is_single_links)
dai_link->cpu_dai_name = NULL;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_canonicalize_cpu);
+EXPORT_SYMBOL_GPL(asoc_simple_canonicalize_cpu);
-int asoc_simple_card_clean_reference(struct snd_soc_card *card)
+int asoc_simple_clean_reference(struct snd_soc_card *card)
{
struct snd_soc_dai_link *dai_link;
int i;
}
return 0;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_clean_reference);
+EXPORT_SYMBOL_GPL(asoc_simple_clean_reference);
-int asoc_simple_card_of_parse_routing(struct snd_soc_card *card,
- char *prefix)
+int asoc_simple_parse_routing(struct snd_soc_card *card,
+ char *prefix)
{
struct device_node *node = card->dev->of_node;
char prop[128];
return snd_soc_of_parse_audio_routing(card, prop);
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_of_parse_routing);
+EXPORT_SYMBOL_GPL(asoc_simple_parse_routing);
-int asoc_simple_card_of_parse_widgets(struct snd_soc_card *card,
- char *prefix)
+int asoc_simple_parse_widgets(struct snd_soc_card *card,
+ char *prefix)
{
struct device_node *node = card->dev->of_node;
char prop[128];
/* no widgets is not error */
return 0;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_of_parse_widgets);
+EXPORT_SYMBOL_GPL(asoc_simple_parse_widgets);
+
+int asoc_simple_parse_pin_switches(struct snd_soc_card *card,
+ char *prefix)
+{
+ const unsigned int nb_controls_max = 16;
+ const char **strings, *control_name;
+ struct snd_kcontrol_new *controls;
+ struct device *dev = card->dev;
+ unsigned int i, nb_controls;
+ char prop[128];
+ int ret;
+
+ if (!prefix)
+ prefix = "";
+
+ snprintf(prop, sizeof(prop), "%s%s", prefix, "pin-switches");
+
+ if (!of_property_read_bool(dev->of_node, prop))
+ return 0;
+
+ strings = devm_kcalloc(dev, nb_controls_max,
+ sizeof(*strings), GFP_KERNEL);
+ if (!strings)
+ return -ENOMEM;
+
+ ret = of_property_read_string_array(dev->of_node, prop,
+ strings, nb_controls_max);
+ if (ret < 0)
+ return ret;
+
+ nb_controls = (unsigned int)ret;
+
+ controls = devm_kcalloc(dev, nb_controls,
+ sizeof(*controls), GFP_KERNEL);
+ if (!controls)
+ return -ENOMEM;
+
+ for (i = 0; i < nb_controls; i++) {
+ control_name = devm_kasprintf(dev, GFP_KERNEL,
+ "%s Switch", strings[i]);
+ if (!control_name)
+ return -ENOMEM;
+
+ controls[i].iface = SNDRV_CTL_ELEM_IFACE_MIXER;
+ controls[i].name = control_name;
+ controls[i].info = snd_soc_dapm_info_pin_switch;
+ controls[i].get = snd_soc_dapm_get_pin_switch;
+ controls[i].put = snd_soc_dapm_put_pin_switch;
+ controls[i].private_value = (unsigned long)strings[i];
+ }
+
+ card->controls = controls;
+ card->num_controls = nb_controls;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(asoc_simple_parse_pin_switches);
-int asoc_simple_card_init_jack(struct snd_soc_card *card,
- struct asoc_simple_jack *sjack,
- int is_hp, char *prefix)
+int asoc_simple_init_jack(struct snd_soc_card *card,
+ struct asoc_simple_jack *sjack,
+ int is_hp, char *prefix)
{
struct device *dev = card->dev;
enum of_gpio_flags flags;
return 0;
}
-EXPORT_SYMBOL_GPL(asoc_simple_card_init_jack);
+EXPORT_SYMBOL_GPL(asoc_simple_init_jack);
+
+int asoc_simple_init_priv(struct asoc_simple_priv *priv,
+ struct link_info *li)
+{
+ struct snd_soc_card *card = simple_priv_to_card(priv);
+ struct device *dev = simple_priv_to_dev(priv);
+ struct snd_soc_dai_link *dai_link;
+ struct simple_dai_props *dai_props;
+ struct asoc_simple_dai *dais;
+ struct snd_soc_codec_conf *cconf = NULL;
+ int i;
+
+ dai_props = devm_kcalloc(dev, li->link, sizeof(*dai_props), GFP_KERNEL);
+ dai_link = devm_kcalloc(dev, li->link, sizeof(*dai_link), GFP_KERNEL);
+ dais = devm_kcalloc(dev, li->dais, sizeof(*dais), GFP_KERNEL);
+ if (!dai_props || !dai_link || !dais)
+ return -ENOMEM;
+
+ if (li->conf) {
+ cconf = devm_kcalloc(dev, li->conf, sizeof(*cconf), GFP_KERNEL);
+ if (!cconf)
+ return -ENOMEM;
+ }
+
+ /*
+ * Use snd_soc_dai_link_component instead of legacy style
+ * It is codec only. but cpu/platform will be supported in the future.
+ * see
+ * soc-core.c :: snd_soc_init_multicodec()
+ *
+ * "platform" might be removed
+ * see
+ * simple-card-utils.c :: asoc_simple_canonicalize_platform()
+ */
+ for (i = 0; i < li->link; i++) {
+ dai_link[i].codecs = &dai_props[i].codecs;
+ dai_link[i].num_codecs = 1;
+ dai_link[i].platforms = &dai_props[i].platforms;
+ dai_link[i].num_platforms = 1;
+ }
+
+ priv->dai_props = dai_props;
+ priv->dai_link = dai_link;
+ priv->dais = dais;
+ priv->codec_conf = cconf;
+
+ card->dai_link = priv->dai_link;
+ card->num_links = li->link;
+ card->codec_conf = cconf;
+ card->num_configs = li->conf;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(asoc_simple_init_priv);
/* Module information */
MODULE_AUTHOR("Kuninori Morimoto <kuninori.morimoto.gx@renesas.com>");
#define DPCM_SELECTABLE 1
-struct simple_priv {
- struct snd_soc_card snd_card;
- struct simple_dai_props {
- struct asoc_simple_dai *cpu_dai;
- struct asoc_simple_dai *codec_dai;
- struct snd_soc_dai_link_component codecs; /* single codec */
- struct snd_soc_dai_link_component platforms;
- struct asoc_simple_card_data adata;
- struct snd_soc_codec_conf *codec_conf;
- unsigned int mclk_fs;
- } *dai_props;
- struct asoc_simple_jack hp_jack;
- struct asoc_simple_jack mic_jack;
- struct snd_soc_dai_link *dai_link;
- struct asoc_simple_dai *dais;
- struct snd_soc_codec_conf *codec_conf;
-};
-
-struct link_info {
- int dais; /* number of dai */
- int link; /* number of link */
- int conf; /* number of codec_conf */
- int cpu; /* turn for CPU / Codec */
-};
-
-#define simple_priv_to_card(priv) (&(priv)->snd_card)
-#define simple_priv_to_props(priv, i) ((priv)->dai_props + (i))
-#define simple_priv_to_dev(priv) (simple_priv_to_card(priv)->dev)
-#define simple_priv_to_link(priv, i) (simple_priv_to_card(priv)->dai_link + (i))
-
#define DAI "sound-dai"
#define CELL "#sound-dai-cells"
#define PREFIX "simple-audio-card,"
-static int simple_startup(struct snd_pcm_substream *substream)
-{
- struct snd_soc_pcm_runtime *rtd = substream->private_data;
- struct simple_priv *priv = snd_soc_card_get_drvdata(rtd->card);
- struct simple_dai_props *dai_props =
- simple_priv_to_props(priv, rtd->num);
- int ret;
-
- ret = asoc_simple_card_clk_enable(dai_props->cpu_dai);
- if (ret)
- return ret;
-
- ret = asoc_simple_card_clk_enable(dai_props->codec_dai);
- if (ret)
- asoc_simple_card_clk_disable(dai_props->cpu_dai);
-
- return ret;
-}
-
-static void simple_shutdown(struct snd_pcm_substream *substream)
-{
- struct snd_soc_pcm_runtime *rtd = substream->private_data;
- struct simple_priv *priv = snd_soc_card_get_drvdata(rtd->card);
- struct simple_dai_props *dai_props =
- simple_priv_to_props(priv, rtd->num);
-
- asoc_simple_card_clk_disable(dai_props->cpu_dai);
-
- asoc_simple_card_clk_disable(dai_props->codec_dai);
-}
+static const struct snd_soc_ops simple_ops = {
+ .startup = asoc_simple_startup,
+ .shutdown = asoc_simple_shutdown,
+ .hw_params = asoc_simple_hw_params,
+};
-static int simple_set_clk_rate(struct asoc_simple_dai *simple_dai,
- unsigned long rate)
+static int asoc_simple_parse_dai(struct device_node *node,
+ struct snd_soc_dai_link_component *dlc,
+ struct device_node **dai_of_node,
+ const char **dai_name,
+ int *is_single_link)
{
- if (!simple_dai)
- return 0;
-
- if (!simple_dai->clk)
- return 0;
+ struct of_phandle_args args;
+ int ret;
- if (clk_get_rate(simple_dai->clk) == rate)
+ if (!node)
return 0;
- return clk_set_rate(simple_dai->clk, rate);
-}
-
-static int simple_hw_params(struct snd_pcm_substream *substream,
- struct snd_pcm_hw_params *params)
-{
- struct snd_soc_pcm_runtime *rtd = substream->private_data;
- struct snd_soc_dai *codec_dai = rtd->codec_dai;
- struct snd_soc_dai *cpu_dai = rtd->cpu_dai;
- struct simple_priv *priv = snd_soc_card_get_drvdata(rtd->card);
- struct simple_dai_props *dai_props =
- simple_priv_to_props(priv, rtd->num);
- unsigned int mclk, mclk_fs = 0;
- int ret = 0;
-
- if (dai_props->mclk_fs)
- mclk_fs = dai_props->mclk_fs;
-
- if (mclk_fs) {
- mclk = params_rate(params) * mclk_fs;
+ /*
+ * Use snd_soc_dai_link_component instead of legacy style.
+ * It is only for codec, but cpu will be supported in the future.
+ * see
+ * soc-core.c :: snd_soc_init_multicodec()
+ */
+ if (dlc) {
+ dai_name = &dlc->dai_name;
+ dai_of_node = &dlc->of_node;
+ }
- ret = simple_set_clk_rate(dai_props->codec_dai, mclk);
- if (ret < 0)
- return ret;
+ /*
+ * Get node via "sound-dai = <&phandle port>"
+ * it will be used as xxx_of_node on soc_bind_dai_link()
+ */
+ ret = of_parse_phandle_with_args(node, DAI, CELL, 0, &args);
+ if (ret)
+ return ret;
- ret = simple_set_clk_rate(dai_props->cpu_dai, mclk);
+ /* Get dai->name */
+ if (dai_name) {
+ ret = snd_soc_of_get_dai_name(node, dai_name);
if (ret < 0)
return ret;
-
- ret = snd_soc_dai_set_sysclk(codec_dai, 0, mclk,
- SND_SOC_CLOCK_IN);
- if (ret && ret != -ENOTSUPP)
- goto err;
-
- ret = snd_soc_dai_set_sysclk(cpu_dai, 0, mclk,
- SND_SOC_CLOCK_OUT);
- if (ret && ret != -ENOTSUPP)
- goto err;
}
- return 0;
-err:
- return ret;
-}
-
-static const struct snd_soc_ops simple_ops = {
- .startup = simple_startup,
- .shutdown = simple_shutdown,
- .hw_params = simple_hw_params,
-};
-static int simple_dai_init(struct snd_soc_pcm_runtime *rtd)
-{
- struct simple_priv *priv = snd_soc_card_get_drvdata(rtd->card);
- struct simple_dai_props *dai_props = simple_priv_to_props(priv, rtd->num);
- int ret;
+ *dai_of_node = args.np;
- ret = asoc_simple_card_init_dai(rtd->codec_dai,
- dai_props->codec_dai);
- if (ret < 0)
- return ret;
-
- ret = asoc_simple_card_init_dai(rtd->cpu_dai,
- dai_props->cpu_dai);
- if (ret < 0)
- return ret;
+ if (is_single_link)
+ *is_single_link = !args.args_count;
return 0;
}
-static int simple_be_hw_params_fixup(struct snd_soc_pcm_runtime *rtd,
- struct snd_pcm_hw_params *params)
+static void simple_parse_convert(struct device *dev,
+ struct device_node *np,
+ struct asoc_simple_data *adata)
{
- struct simple_priv *priv = snd_soc_card_get_drvdata(rtd->card);
- struct simple_dai_props *dai_props = simple_priv_to_props(priv, rtd->num);
+ struct device_node *top = dev->of_node;
+ struct device_node *node = of_get_parent(np);
- asoc_simple_card_convert_fixup(&dai_props->adata, params);
+ asoc_simple_parse_convert(dev, top, PREFIX, adata);
+ asoc_simple_parse_convert(dev, node, PREFIX, adata);
+ asoc_simple_parse_convert(dev, node, NULL, adata);
+ asoc_simple_parse_convert(dev, np, NULL, adata);
- return 0;
+ of_node_put(node);
}
-static void simple_get_conversion(struct device *dev,
- struct device_node *np,
- struct asoc_simple_card_data *adata)
+static void simple_parse_mclk_fs(struct device_node *top,
+ struct device_node *cpu,
+ struct device_node *codec,
+ struct simple_dai_props *props,
+ char *prefix)
{
- struct device_node *top = dev->of_node;
- struct device_node *node = of_get_parent(np);
+ struct device_node *node = of_get_parent(cpu);
+ char prop[128];
- asoc_simple_card_parse_convert(dev, top, PREFIX, adata);
- asoc_simple_card_parse_convert(dev, node, PREFIX, adata);
- asoc_simple_card_parse_convert(dev, node, NULL, adata);
- asoc_simple_card_parse_convert(dev, np, NULL, adata);
+ snprintf(prop, sizeof(prop), "%smclk-fs", PREFIX);
+ of_property_read_u32(top, prop, &props->mclk_fs);
+
+ snprintf(prop, sizeof(prop), "%smclk-fs", prefix);
+ of_property_read_u32(node, prop, &props->mclk_fs);
+ of_property_read_u32(cpu, prop, &props->mclk_fs);
+ of_property_read_u32(codec, prop, &props->mclk_fs);
of_node_put(node);
}
-static int simple_dai_link_of_dpcm(struct simple_priv *priv,
+static int simple_dai_link_of_dpcm(struct asoc_simple_priv *priv,
struct device_node *np,
struct device_node *codec,
struct link_info *li,
struct snd_soc_dai_link_component *codecs = dai_link->codecs;
struct device_node *top = dev->of_node;
struct device_node *node = of_get_parent(np);
- char prop[128];
char *prefix = "";
int ret;
dai =
dai_props->cpu_dai = &priv->dais[li->dais++];
- ret = asoc_simple_card_parse_cpu(np, dai_link, DAI, CELL,
- &is_single_links);
+ ret = asoc_simple_parse_cpu(np, dai_link, &is_single_links);
if (ret)
return ret;
- ret = asoc_simple_card_parse_clk_cpu(dev, np, dai_link, dai);
+ ret = asoc_simple_parse_clk_cpu(dev, np, dai_link, dai);
if (ret < 0)
return ret;
- ret = asoc_simple_card_set_dailink_name(dev, dai_link,
- "fe.%s",
- dai_link->cpu_dai_name);
+ ret = asoc_simple_set_dailink_name(dev, dai_link,
+ "fe.%s",
+ dai_link->cpu_dai_name);
if (ret < 0)
return ret;
- asoc_simple_card_canonicalize_cpu(dai_link, is_single_links);
+ asoc_simple_canonicalize_cpu(dai_link, is_single_links);
} else {
struct snd_soc_codec_conf *cconf;
/* BE settings */
dai_link->no_pcm = 1;
- dai_link->be_hw_params_fixup = simple_be_hw_params_fixup;
+ dai_link->be_hw_params_fixup = asoc_simple_be_hw_params_fixup;
dai =
dai_props->codec_dai = &priv->dais[li->dais++];
cconf =
dai_props->codec_conf = &priv->codec_conf[li->conf++];
- ret = asoc_simple_card_parse_codec(np, dai_link, DAI, CELL);
+ ret = asoc_simple_parse_codec(np, dai_link);
if (ret < 0)
return ret;
- ret = asoc_simple_card_parse_clk_codec(dev, np, dai_link, dai);
+ ret = asoc_simple_parse_clk_codec(dev, np, dai_link, dai);
if (ret < 0)
return ret;
- ret = asoc_simple_card_set_dailink_name(dev, dai_link,
- "be.%s",
- codecs->dai_name);
+ ret = asoc_simple_set_dailink_name(dev, dai_link,
+ "be.%s",
+ codecs->dai_name);
if (ret < 0)
return ret;
"prefix");
}
- simple_get_conversion(dev, np, &dai_props->adata);
+ simple_parse_convert(dev, np, &dai_props->adata);
+ simple_parse_mclk_fs(top, np, codec, dai_props, prefix);
- asoc_simple_card_canonicalize_platform(dai_link);
+ asoc_simple_canonicalize_platform(dai_link);
- ret = asoc_simple_card_of_parse_tdm(np, dai);
+ ret = asoc_simple_parse_tdm(np, dai);
if (ret)
return ret;
- snprintf(prop, sizeof(prop), "%smclk-fs", prefix);
- of_property_read_u32(top, PREFIX "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(node, prop, &dai_props->mclk_fs);
- of_property_read_u32(np, prop, &dai_props->mclk_fs);
-
- ret = asoc_simple_card_parse_daifmt(dev, node, codec,
- prefix, &dai_link->dai_fmt);
+ ret = asoc_simple_parse_daifmt(dev, node, codec,
+ prefix, &dai_link->dai_fmt);
if (ret < 0)
return ret;
dai_link->dpcm_playback = 1;
dai_link->dpcm_capture = 1;
dai_link->ops = &simple_ops;
- dai_link->init = simple_dai_init;
+ dai_link->init = asoc_simple_dai_init;
return 0;
}
-static int simple_dai_link_of(struct simple_priv *priv,
+static int simple_dai_link_of(struct asoc_simple_priv *priv,
struct device_node *np,
struct device_node *codec,
struct link_info *li,
codec_dai =
dai_props->codec_dai = &priv->dais[li->dais++];
- ret = asoc_simple_card_parse_daifmt(dev, node, codec,
- prefix, &dai_link->dai_fmt);
+ ret = asoc_simple_parse_daifmt(dev, node, codec,
+ prefix, &dai_link->dai_fmt);
if (ret < 0)
goto dai_link_of_err;
- snprintf(prop, sizeof(prop), "%smclk-fs", prefix);
- of_property_read_u32(top, PREFIX "mclk-fs", &dai_props->mclk_fs);
- of_property_read_u32(node, prop, &dai_props->mclk_fs);
- of_property_read_u32(cpu, prop, &dai_props->mclk_fs);
- of_property_read_u32(codec, prop, &dai_props->mclk_fs);
+ simple_parse_mclk_fs(top, cpu, codec, dai_props, prefix);
- ret = asoc_simple_card_parse_cpu(cpu, dai_link,
- DAI, CELL, &single_cpu);
+ ret = asoc_simple_parse_cpu(cpu, dai_link, &single_cpu);
if (ret < 0)
goto dai_link_of_err;
- ret = asoc_simple_card_parse_codec(codec, dai_link, DAI, CELL);
+ ret = asoc_simple_parse_codec(codec, dai_link);
if (ret < 0)
goto dai_link_of_err;
- ret = asoc_simple_card_parse_platform(plat, dai_link, DAI, CELL);
+ ret = asoc_simple_parse_platform(plat, dai_link);
if (ret < 0)
goto dai_link_of_err;
- ret = asoc_simple_card_of_parse_tdm(cpu, cpu_dai);
+ ret = asoc_simple_parse_tdm(cpu, cpu_dai);
if (ret < 0)
goto dai_link_of_err;
- ret = asoc_simple_card_of_parse_tdm(codec, codec_dai);
+ ret = asoc_simple_parse_tdm(codec, codec_dai);
if (ret < 0)
goto dai_link_of_err;
- ret = asoc_simple_card_parse_clk_cpu(dev, cpu, dai_link, cpu_dai);
+ ret = asoc_simple_parse_clk_cpu(dev, cpu, dai_link, cpu_dai);
if (ret < 0)
goto dai_link_of_err;
- ret = asoc_simple_card_parse_clk_codec(dev, codec, dai_link, codec_dai);
+ ret = asoc_simple_parse_clk_codec(dev, codec, dai_link, codec_dai);
if (ret < 0)
goto dai_link_of_err;
- ret = asoc_simple_card_set_dailink_name(dev, dai_link,
- "%s-%s",
- dai_link->cpu_dai_name,
- dai_link->codecs->dai_name);
+ ret = asoc_simple_set_dailink_name(dev, dai_link,
+ "%s-%s",
+ dai_link->cpu_dai_name,
+ dai_link->codecs->dai_name);
if (ret < 0)
goto dai_link_of_err;
dai_link->ops = &simple_ops;
- dai_link->init = simple_dai_init;
+ dai_link->init = asoc_simple_dai_init;
- asoc_simple_card_canonicalize_cpu(dai_link, single_cpu);
- asoc_simple_card_canonicalize_platform(dai_link);
+ asoc_simple_canonicalize_cpu(dai_link, single_cpu);
+ asoc_simple_canonicalize_platform(dai_link);
dai_link_of_err:
of_node_put(plat);
return ret;
}
-static int simple_for_each_link(struct simple_priv *priv,
+static int simple_for_each_link(struct asoc_simple_priv *priv,
struct link_info *li,
- int (*func_noml)(struct simple_priv *priv,
+ int (*func_noml)(struct asoc_simple_priv *priv,
struct device_node *np,
struct device_node *codec,
struct link_info *li, bool is_top),
- int (*func_dpcm)(struct simple_priv *priv,
+ int (*func_dpcm)(struct asoc_simple_priv *priv,
struct device_node *np,
struct device_node *codec,
struct link_info *li, bool is_top))
/* loop for all dai-link */
do {
- struct asoc_simple_card_data adata;
+ struct asoc_simple_data adata;
struct device_node *codec;
struct device_node *np;
int num = of_get_child_count(node);
/* get convert-xxx property */
memset(&adata, 0, sizeof(adata));
for_each_child_of_node(node, np)
- simple_get_conversion(dev, np, &adata);
+ simple_parse_convert(dev, np, &adata);
/* loop for all CPU/Codec node */
for_each_child_of_node(node, np) {
}
static int simple_parse_aux_devs(struct device_node *node,
- struct simple_priv *priv)
+ struct asoc_simple_priv *priv)
{
struct device *dev = simple_priv_to_dev(priv);
struct device_node *aux_node;
return 0;
}
-static int simple_parse_of(struct simple_priv *priv)
+static int simple_parse_of(struct asoc_simple_priv *priv)
{
struct device *dev = simple_priv_to_dev(priv);
struct device_node *top = dev->of_node;
if (!top)
return -EINVAL;
- ret = asoc_simple_card_of_parse_widgets(card, PREFIX);
+ ret = asoc_simple_parse_widgets(card, PREFIX);
if (ret < 0)
return ret;
- ret = asoc_simple_card_of_parse_routing(card, PREFIX);
+ ret = asoc_simple_parse_routing(card, PREFIX);
+ if (ret < 0)
+ return ret;
+
+ ret = asoc_simple_parse_pin_switches(card, PREFIX);
if (ret < 0)
return ret;
return ret;
}
- ret = asoc_simple_card_parse_card_name(card, PREFIX);
+ ret = asoc_simple_parse_card_name(card, PREFIX);
if (ret < 0)
return ret;
return ret;
}
-static int simple_count_noml(struct simple_priv *priv,
+static int simple_count_noml(struct asoc_simple_priv *priv,
struct device_node *np,
struct device_node *codec,
struct link_info *li, bool is_top)
return 0;
}
-static int simple_count_dpcm(struct simple_priv *priv,
+static int simple_count_dpcm(struct asoc_simple_priv *priv,
struct device_node *np,
struct device_node *codec,
struct link_info *li, bool is_top)
return 0;
}
-static void simple_get_dais_count(struct simple_priv *priv,
+static void simple_get_dais_count(struct asoc_simple_priv *priv,
struct link_info *li)
{
struct device *dev = simple_priv_to_dev(priv);
static int simple_soc_probe(struct snd_soc_card *card)
{
- struct simple_priv *priv = snd_soc_card_get_drvdata(card);
+ struct asoc_simple_priv *priv = snd_soc_card_get_drvdata(card);
int ret;
- ret = asoc_simple_card_init_hp(card, &priv->hp_jack, PREFIX);
+ ret = asoc_simple_init_hp(card, &priv->hp_jack, PREFIX);
if (ret < 0)
return ret;
- ret = asoc_simple_card_init_mic(card, &priv->mic_jack, PREFIX);
+ ret = asoc_simple_init_mic(card, &priv->mic_jack, PREFIX);
if (ret < 0)
return ret;
static int simple_probe(struct platform_device *pdev)
{
- struct simple_priv *priv;
- struct snd_soc_dai_link *dai_link;
- struct simple_dai_props *dai_props;
- struct asoc_simple_dai *dais;
+ struct asoc_simple_priv *priv;
struct device *dev = &pdev->dev;
struct device_node *np = dev->of_node;
struct snd_soc_card *card;
- struct snd_soc_codec_conf *cconf;
struct link_info li;
- int ret, i;
+ int ret;
/* Allocate the private data and the DAI link array */
priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
if (!li.link || !li.dais)
return -EINVAL;
- dai_props = devm_kcalloc(dev, li.link, sizeof(*dai_props), GFP_KERNEL);
- dai_link = devm_kcalloc(dev, li.link, sizeof(*dai_link), GFP_KERNEL);
- dais = devm_kcalloc(dev, li.dais, sizeof(*dais), GFP_KERNEL);
- cconf = devm_kcalloc(dev, li.conf, sizeof(*cconf), GFP_KERNEL);
- if (!dai_props || !dai_link || !dais)
- return -ENOMEM;
-
- /*
- * Use snd_soc_dai_link_component instead of legacy style
- * It is codec only. but cpu/platform will be supported in the future.
- * see
- * soc-core.c :: snd_soc_init_multicodec()
- */
- for (i = 0; i < li.link; i++) {
- dai_link[i].codecs = &dai_props[i].codecs;
- dai_link[i].num_codecs = 1;
- dai_link[i].platforms = &dai_props[i].platforms;
- dai_link[i].num_platforms = 1;
- }
-
- priv->dai_props = dai_props;
- priv->dai_link = dai_link;
- priv->dais = dais;
- priv->codec_conf = cconf;
-
- card->dai_link = priv->dai_link;
- card->num_links = li.link;
- card->codec_conf = cconf;
- card->num_configs = li.conf;
+ ret = asoc_simple_init_priv(priv, &li);
+ if (ret < 0)
+ return ret;
if (np && of_device_is_available(np)) {
struct asoc_simple_card_info *cinfo;
struct snd_soc_dai_link_component *codecs;
struct snd_soc_dai_link_component *platform;
+ struct snd_soc_dai_link *dai_link = priv->dai_link;
+ struct simple_dai_props *dai_props = priv->dai_props;
+
int dai_idx = 0;
cinfo = dev->platform_data;
dai_link->stream_name = cinfo->name;
dai_link->cpu_dai_name = cinfo->cpu_dai.name;
dai_link->dai_fmt = cinfo->daifmt;
- dai_link->init = simple_dai_init;
- memcpy(priv->dai_props->cpu_dai, &cinfo->cpu_dai,
- sizeof(*priv->dai_props->cpu_dai));
- memcpy(priv->dai_props->codec_dai, &cinfo->codec_dai,
- sizeof(*priv->dai_props->codec_dai));
+ dai_link->init = asoc_simple_dai_init;
+ memcpy(dai_props->cpu_dai, &cinfo->cpu_dai,
+ sizeof(*dai_props->cpu_dai));
+ memcpy(dai_props->codec_dai, &cinfo->codec_dai,
+ sizeof(*dai_props->codec_dai));
}
snd_soc_card_set_drvdata(card, priv);
+ asoc_simple_debug_info(priv);
+
ret = devm_snd_soc_register_card(dev, card);
if (ret < 0)
goto err;
return 0;
err:
- asoc_simple_card_clean_reference(card);
+ asoc_simple_clean_reference(card);
return ret;
}
{
struct snd_soc_card *card = platform_get_drvdata(pdev);
- return asoc_simple_card_clean_reference(card);
+ return asoc_simple_clean_reference(card);
}
static const struct of_device_id simple_of_match[] = {
endif ## SND_SOC_INTEL_SKYLAKE_FAMILY
+endif ## SND_SOC_INTEL_SST_TOPLEVEL
+
+if SND_SOC_INTEL_SST_TOPLEVEL || SND_SOC_SOF_INTEL_TOPLEVEL
+
config SND_SOC_ACPI_INTEL_MATCH
tristate
select SND_SOC_ACPI if ACPI
# this option controls the compilation of ACPI matching tables and
# helpers and is not meant to be selected by the user.
-endif ## SND_SOC_INTEL_SST_TOPLEVEL
+endif ## SND_SOC_INTEL_SST_TOPLEVEL || SND_SOC_SOF_INTEL_TOPLEVEL
+
# ASoC codec drivers
source "sound/soc/intel/boards/Kconfig"
menuconfig SND_SOC_INTEL_MACH
bool "Intel Machine drivers"
- depends on SND_SOC_INTEL_SST_TOPLEVEL
+ depends on SND_SOC_INTEL_SST_TOPLEVEL || SND_SOC_SOF_INTEL_TOPLEVEL
help
Intel ASoC Machine Drivers. If you have a Intel machine that
has an audio controller with a DSP and I2S or DMIC port, then
config SND_SOC_INTEL_HASWELL_MACH
tristate "Haswell Lynxpoint"
- depends on X86_INTEL_LPSS && I2C && I2C_DESIGNWARE_PLATFORM
+ depends on I2C
+ depends on I2C_DESIGNWARE_PLATFORM || COMPILE_TEST
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_RT5640
help
This adds support for the Lynxpoint Audio DSP on Intel(R) Haswell
Say Y or m if you have such a device.
If unsure select "N".
+endif ## SND_SOC_INTEL_HASWELL
+
+if SND_SOC_INTEL_HASWELL || SND_SOC_SOF_BROADWELL
+
config SND_SOC_INTEL_BDW_RT5677_MACH
tristate "Broadwell with RT5677 codec"
- depends on X86_INTEL_LPSS && I2C && I2C_DESIGNWARE_PLATFORM && GPIOLIB
+ depends on I2C
+ depends on I2C_DESIGNWARE_PLATFORM || COMPILE_TEST
+ depends on GPIOLIB || COMPILE_TEST
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_RT5677
help
This adds support for Intel Broadwell platform based boards with
config SND_SOC_INTEL_BROADWELL_MACH
tristate "Broadwell Wildcatpoint"
- depends on X86_INTEL_LPSS && I2C && I2C_DESIGNWARE_PLATFORM
+ depends on I2C
+ depends on I2C_DESIGNWARE_PLATFORM || COMPILE_TEST
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_RT286
help
This adds support for the Wilcatpoint Audio DSP on Intel(R) Broadwell
Ultrabook platforms.
Say Y or m if you have such a device. This is a recommended option.
If unsure select "N".
-endif ## SND_SOC_INTEL_HASWELL
+endif ## SND_SOC_INTEL_HASWELL || SND_SOC_SOF_BROADWELL
if SND_SOC_INTEL_BAYTRAIL
config SND_SOC_INTEL_BYT_MAX98090_MACH
tristate "Baytrail with MAX98090 codec"
- depends on X86_INTEL_LPSS && I2C
+ depends on I2C
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_MAX98090
help
This adds audio driver for Intel Baytrail platform based boards
config SND_SOC_INTEL_BYT_RT5640_MACH
tristate "Baytrail with RT5640 codec"
- depends on X86_INTEL_LPSS && I2C
+ depends on I2C
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_RT5640
help
This adds audio driver for Intel Baytrail platform based boards
endif ## SND_SOC_INTEL_BAYTRAIL
-if SND_SST_ATOM_HIFI2_PLATFORM
+if SND_SST_ATOM_HIFI2_PLATFORM || SND_SOC_SOF_BAYTRAIL
config SND_SOC_INTEL_BYTCR_RT5640_MACH
tristate "Baytrail and Baytrail-CR with RT5640 codec"
- depends on X86_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_ACPI
select SND_SOC_RT5640
help
config SND_SOC_INTEL_BYTCR_RT5651_MACH
tristate "Baytrail and Baytrail-CR with RT5651 codec"
- depends on X86_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_ACPI
select SND_SOC_RT5651
help
config SND_SOC_INTEL_CHT_BSW_RT5672_MACH
tristate "Cherrytrail & Braswell with RT5672 codec"
- depends on X86_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_ACPI
select SND_SOC_RT5670
help
config SND_SOC_INTEL_CHT_BSW_RT5645_MACH
tristate "Cherrytrail & Braswell with RT5645/5650 codec"
- depends on X86_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_ACPI
select SND_SOC_RT5645
help
config SND_SOC_INTEL_CHT_BSW_MAX98090_TI_MACH
tristate "Cherrytrail & Braswell with MAX98090 & TI codec"
- depends on X86_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_MAX98090
select SND_SOC_TS3A227E
help
config SND_SOC_INTEL_CHT_BSW_NAU8824_MACH
tristate "Cherrytrail & Braswell with NAU88L24 codec"
- depends on X86_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_ACPI
select SND_SOC_NAU8824
help
config SND_SOC_INTEL_BYT_CHT_DA7213_MACH
tristate "Baytrail & Cherrytrail with DA7212/7213 codec"
- depends on X86_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_ACPI
select SND_SOC_DA7213
help
config SND_SOC_INTEL_BYT_CHT_ES8316_MACH
tristate "Baytrail & Cherrytrail with ES8316 codec"
- depends on X86_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on X86_INTEL_LPSS || COMPILE_TEST
select SND_SOC_ACPI
select SND_SOC_ES8316
help
Say Y or m if you have such a device. This is a recommended option.
If unsure select "N".
+endif ## SND_SST_ATOM_HIFI2_PLATFORM || SND_SOC_SOF_BAYTRAIL
+
+if SND_SST_ATOM_HIFI2_PLATFORM
+
config SND_SOC_INTEL_BYT_CHT_NOCODEC_MACH
tristate "Baytrail & Cherrytrail platform with no codec (MinnowBoard MAX, Up)"
- depends on X86_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on X86_INTEL_LPSS || COMPILE_TEST
help
This adds support for ASoC machine driver for the MinnowBoard Max or
Up boards and provides access to I2S signals on the Low-Speed
config SND_SOC_INTEL_SKL_RT286_MACH
tristate "SKL with RT286 I2S mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
select SND_SOC_RT286
select SND_SOC_DMIC
select SND_SOC_HDAC_HDMI
config SND_SOC_INTEL_SKL_NAU88L25_SSM4567_MACH
tristate "SKL with NAU88L25 and SSM4567 in I2S Mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
select SND_SOC_NAU8825
select SND_SOC_SSM4567
select SND_SOC_DMIC
config SND_SOC_INTEL_SKL_NAU88L25_MAX98357A_MACH
tristate "SKL with NAU88L25 and MAX98357A in I2S Mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
select SND_SOC_NAU8825
select SND_SOC_MAX98357A
select SND_SOC_DMIC
config SND_SOC_INTEL_BXT_DA7219_MAX98357A_MACH
tristate "Broxton with DA7219 and MAX98357A in I2S Mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
select SND_SOC_DA7219
select SND_SOC_MAX98357A
select SND_SOC_DMIC
config SND_SOC_INTEL_BXT_RT298_MACH
tristate "Broxton with RT298 I2S mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
select SND_SOC_RT298
select SND_SOC_DMIC
select SND_SOC_HDAC_HDMI
config SND_SOC_INTEL_KBL_RT5663_MAX98927_MACH
tristate "KBL with RT5663 and MAX98927 in I2S Mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
select SND_SOC_RT5663
select SND_SOC_MAX98927
select SND_SOC_DMIC
config SND_SOC_INTEL_KBL_RT5663_RT5514_MAX98927_MACH
tristate "KBL with RT5663, RT5514 and MAX98927 in I2S Mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
depends on SPI
select SND_SOC_RT5663
select SND_SOC_RT5514
config SND_SOC_INTEL_KBL_DA7219_MAX98357A_MACH
tristate "KBL with DA7219 and MAX98357A in I2S Mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
select SND_SOC_DA7219
select SND_SOC_MAX98357A
select SND_SOC_DMIC
config SND_SOC_INTEL_KBL_DA7219_MAX98927_MACH
tristate "KBL with DA7219 and MAX98927 in I2S Mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
select SND_SOC_DA7219
select SND_SOC_MAX98927
select SND_SOC_MAX98373
config SND_SOC_INTEL_KBL_RT5660_MACH
tristate "KBL with RT5660 in I2S Mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
select SND_SOC_RT5660
select SND_SOC_HDAC_HDMI
help
endif ## SND_SOC_INTEL_KBL
-if SND_SOC_INTEL_GLK
+if SND_SOC_INTEL_GLK || (SND_SOC_SOF_GEMINILAKE && SND_SOC_SOF_HDA_LINK)
config SND_SOC_INTEL_GLK_RT5682_MAX98357A_MACH
tristate "GLK with RT5682 and MAX98357A in I2S Mode"
- depends on MFD_INTEL_LPSS && I2C && ACPI
+ depends on I2C && ACPI
+ depends on MFD_INTEL_LPSS || COMPILE_TEST
select SND_SOC_RT5682
select SND_SOC_MAX98357A
select SND_SOC_DMIC
Say Y if you have such a device.
If unsure select "N".
-endif ## SND_SOC_INTEL_GLK
+endif ## SND_SOC_INTEL_GLK || (SND_SOC_SOF_GEMINILAKE && SND_SOC_SOF_HDA_LINK)
-if SND_SOC_INTEL_SKYLAKE_HDAUDIO_CODEC
+if SND_SOC_INTEL_SKYLAKE_HDAUDIO_CODEC || SND_SOC_SOF_HDA_AUDIO_CODEC
config SND_SOC_INTEL_SKL_HDA_DSP_GENERIC_MACH
tristate "SKL/KBL/BXT/APL with HDA Codecs"
Say Y or m if you have such a device. This is a recommended option.
If unsure select "N".
-endif ## SND_SOC_INTEL_SKYLAKE_HDAUDIO_CODEC
+endif ## SND_SOC_INTEL_SKYLAKE_HDAUDIO_CODEC || SND_SOC_SOF_HDA_AUDIO_CODEC
+
+if SND_SOC_SOF_HDA_COMMON || SND_SOC_SOF_BAYTRAIL
+config SND_SOC_INTEL_SOF_RT5682_MACH
+ tristate "SOF with rt5682 codec in I2S Mode"
+ depends on I2C && ACPI
+ depends on (SND_SOC_SOF_HDA_COMMON && MFD_INTEL_LPSS) ||\
+ (SND_SOC_SOF_BAYTRAIL && X86_INTEL_LPSS)
+ select SND_SOC_RT5682
+ select SND_SOC_DMIC
+ select SND_SOC_HDAC_HDMI if SND_SOC_SOF_HDA_COMMON
+ help
+ This adds support for ASoC machine driver for SOF platforms
+ with rt5682 codec.
+ Say Y if you have such a device.
+ If unsure select "N".
+endif ## SND_SOC_SOF_HDA_COMMON || SND_SOC_SOF_BAYTRAIL
endif ## SND_SOC_INTEL_MACH
snd-soc-sst-byt-cht-da7213-objs := bytcht_da7213.o
snd-soc-sst-byt-cht-es8316-objs := bytcht_es8316.o
snd-soc-sst-byt-cht-nocodec-objs := bytcht_nocodec.o
+snd-soc-sof_rt5682-objs := sof_rt5682.o
snd-soc-kbl_da7219_max98357a-objs := kbl_da7219_max98357a.o
snd-soc-kbl_da7219_max98927-objs := kbl_da7219_max98927.o
snd-soc-kbl_rt5663_max98927-objs := kbl_rt5663_max98927.o
snd-skl_nau88l25_max98357a-objs := skl_nau88l25_max98357a.o
snd-soc-skl_nau88l25_ssm4567-objs := skl_nau88l25_ssm4567.o
+obj-$(CONFIG_SND_SOC_INTEL_SOF_RT5682_MACH) += snd-soc-sof_rt5682.o
obj-$(CONFIG_SND_SOC_INTEL_HASWELL_MACH) += snd-soc-sst-haswell.o
obj-$(CONFIG_SND_SOC_INTEL_BYT_RT5640_MACH) += snd-soc-sst-byt-rt5640-mach.o
obj-$(CONFIG_SND_SOC_INTEL_BYT_MAX98090_MACH) += snd-soc-sst-byt-max98090-mach.o
.hw_params = bdw_rt5677_hw_params,
};
+#if !IS_ENABLED(CONFIG_SND_SOC_SOF_BROADWELL)
static int bdw_rt5677_rtd_init(struct snd_soc_pcm_runtime *rtd)
{
struct snd_soc_component *component = snd_soc_rtdcom_lookup(rtd, DRV_NAME);
return 0;
}
+#endif
static int bdw_rt5677_init(struct snd_soc_pcm_runtime *rtd)
{
.dynamic = 1,
.codec_name = "snd-soc-dummy",
.codec_dai_name = "snd-soc-dummy-dai",
+#if !IS_ENABLED(CONFIG_SND_SOC_SOF_BROADWELL)
.init = bdw_rt5677_rtd_init,
+#endif
.trigger = {
SND_SOC_DPCM_TRIGGER_POST,
SND_SOC_DPCM_TRIGGER_POST
.hw_params = broadwell_rt286_hw_params,
};
+#if !IS_ENABLED(CONFIG_SND_SOC_SOF_BROADWELL)
static int broadwell_rtd_init(struct snd_soc_pcm_runtime *rtd)
{
struct snd_soc_component *component = snd_soc_rtdcom_lookup(rtd, DRV_NAME);
return 0;
}
+#endif
/* broadwell digital audio interface glue - connects codec <--> CPU */
static struct snd_soc_dai_link broadwell_rt286_dais[] = {
.dynamic = 1,
.codec_name = "snd-soc-dummy",
.codec_dai_name = "snd-soc-dummy-dai",
+#if !IS_ENABLED(CONFIG_SND_SOC_SOF_BROADWELL)
.init = broadwell_rtd_init,
+#endif
.trigger = {SND_SOC_DPCM_TRIGGER_POST, SND_SOC_DPCM_TRIGGER_POST},
.dpcm_playback = 1,
.dpcm_capture = 1,
#include <linux/acpi.h>
#include <linux/clk.h>
#include <linux/device.h>
+#include <linux/dmi.h>
#include <linux/gpio/consumer.h>
#include <linux/i2c.h>
#include <linux/init.h>
#include "../atom/sst-atom-controls.h"
#include "../common/sst-dsp.h"
+/* jd-inv + terminating entry */
+#define MAX_NO_PROPS 2
+
struct byt_cht_es8316_private {
struct clk *mclk;
struct snd_soc_jack jack;
#define BYT_CHT_ES8316_MAP(quirk) ((quirk) & GENMASK(3, 0))
#define BYT_CHT_ES8316_SSP0 BIT(16)
#define BYT_CHT_ES8316_MONO_SPEAKER BIT(17)
+#define BYT_CHT_ES8316_JD_INVERTED BIT(18)
-static int quirk;
+static unsigned long quirk;
static int quirk_override = -1;
module_param_named(quirk, quirk_override, int, 0444);
dev_info(dev, "quirk SSP0 enabled");
if (quirk & BYT_CHT_ES8316_MONO_SPEAKER)
dev_info(dev, "quirk MONO_SPEAKER enabled\n");
+ if (quirk & BYT_CHT_ES8316_JD_INVERTED)
+ dev_info(dev, "quirk JD_INVERTED enabled\n");
}
static int byt_cht_es8316_speaker_power_event(struct snd_soc_dapm_widget *w,
{ },
};
+/* Please keep this list alphabetically sorted */
+static const struct dmi_system_id byt_cht_es8316_quirk_table[] = {
+ { /* Teclast X98 Plus II */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "TECLAST"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "X98 Plus II"),
+ },
+ .driver_data = (void *)(BYT_CHT_ES8316_INTMIC_IN1_MAP
+ | BYT_CHT_ES8316_JD_INVERTED),
+ },
+ {}
+};
+
static int snd_byt_cht_es8316_mc_probe(struct platform_device *pdev)
{
static const char * const mic_name[] = { "in1", "in2" };
+ struct property_entry props[MAX_NO_PROPS] = {};
struct byt_cht_es8316_private *priv;
+ const struct dmi_system_id *dmi_id;
struct device *dev = &pdev->dev;
struct snd_soc_acpi_mach *mach;
const char *platform_name;
struct acpi_device *adev;
struct device *codec_dev;
+ unsigned int cnt = 0;
int dai_index = 0;
int i;
int ret = 0;
return ret;
/* Check for BYTCR or other platform and setup quirks */
- if (x86_match_cpu(baytrail_cpu_ids) &&
+ dmi_id = dmi_first_match(byt_cht_es8316_quirk_table);
+ if (dmi_id) {
+ quirk = (unsigned long)dmi_id->driver_data;
+ } else if (x86_match_cpu(baytrail_cpu_ids) &&
mach->mach_params.acpi_ipc_irq_index == 0) {
/* On BYTCR default to SSP0, internal-mic-in2-map, mono-spk */
quirk = BYT_CHT_ES8316_SSP0 | BYT_CHT_ES8316_INTMIC_IN2_MAP |
BYT_CHT_ES8316_MONO_SPEAKER;
}
if (quirk_override != -1) {
- dev_info(dev, "Overriding quirk 0x%x => 0x%x\n", quirk,
+ dev_info(dev, "Overriding quirk 0x%x => 0x%x\n",
+ (unsigned int)quirk,
quirk_override);
quirk = quirk_override;
}
if (!codec_dev)
return -EPROBE_DEFER;
+ if (quirk & BYT_CHT_ES8316_JD_INVERTED)
+ props[cnt++] = PROPERTY_ENTRY_BOOL("everest,jack-detect-inverted");
+
+ if (cnt) {
+ ret = device_add_properties(codec_dev, props);
+ if (ret)
+ return ret;
+ }
+
devm_acpi_dev_add_driver_gpios(codec_dev, byt_cht_es8316_gpios);
priv->speaker_en_gpio =
gpiod_get_index(codec_dev, "speaker-enable", 0,
static bool is_bytcr;
static unsigned long byt_rt5640_quirk = BYT_RT5640_MCLK_EN;
-static unsigned int quirk_override;
-module_param_named(quirk, quirk_override, uint, 0444);
+static int quirk_override = -1;
+module_param_named(quirk, quirk_override, int, 0444);
MODULE_PARM_DESC(quirk, "Board-specific quirk override");
static void log_quirks(struct device *dev)
dmi_id = dmi_first_match(byt_rt5640_quirk_table);
if (dmi_id)
byt_rt5640_quirk = (unsigned long)dmi_id->driver_data;
- if (quirk_override) {
+ if (quirk_override != -1) {
dev_info(&pdev->dev, "Overriding quirk 0x%x => 0x%x\n",
(unsigned int)byt_rt5640_quirk, quirk_override);
byt_rt5640_quirk = quirk_override;
#define BYT_RT5651_SSP0_AIF2 BIT(21)
#define BYT_RT5651_HP_LR_SWAPPED BIT(22)
#define BYT_RT5651_MONO_SPEAKER BIT(23)
+#define BYT_RT5651_JD_NOT_INV BIT(24)
#define BYT_RT5651_DEFAULT_QUIRKS (BYT_RT5651_MCLK_EN | \
BYT_RT5651_JD1_1 | \
BYT_RT5651_OVCD_TH_2000UA | \
BYT_RT5651_OVCD_SF_0P75)
-/* jack-detect-source + dmic-en + ovcd-th + -sf + terminating empty entry */
-#define MAX_NO_PROPS 5
+/* jack-detect-source + inv + dmic-en + ovcd-th + -sf + terminating entry */
+#define MAX_NO_PROPS 6
struct byt_rt5651_private {
struct clk *mclk;
static unsigned long byt_rt5651_quirk = BYT_RT5651_DEFAULT_QUIRKS |
BYT_RT5651_IN2_MAP;
-static unsigned int quirk_override;
-module_param_named(quirk, quirk_override, uint, 0444);
+static int quirk_override = -1;
+module_param_named(quirk, quirk_override, int, 0444);
MODULE_PARM_DESC(quirk, "Board-specific quirk override");
static void log_quirks(struct device *dev)
dev_info(dev, "quirk SSP0_AIF2 enabled\n");
if (byt_rt5651_quirk & BYT_RT5651_MONO_SPEAKER)
dev_info(dev, "quirk MONO_SPEAKER enabled\n");
+ if (byt_rt5651_quirk & BYT_RT5651_JD_NOT_INV)
+ dev_info(dev, "quirk JD_NOT_INV enabled\n");
}
#define BYT_CODEC_DAI1 "rt5651-aif1"
BYT_RT5651_HP_LR_SWAPPED |
BYT_RT5651_MONO_SPEAKER),
},
+ {
+ /* Complet Electro Serv MY8307 */
+ .callback = byt_rt5651_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Complet Electro Serv"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "MY8307"),
+ },
+ .driver_data = (void *)(BYT_RT5651_DEFAULT_QUIRKS |
+ BYT_RT5651_IN2_MAP |
+ BYT_RT5651_MONO_SPEAKER |
+ BYT_RT5651_JD_NOT_INV),
+ },
{
/* I.T.Works TW701, Ployer Momo7w and Trekstor ST70416-6
* (these all use the same mainboard) */
if (byt_rt5651_quirk & BYT_RT5651_DMIC_EN)
props[cnt++] = PROPERTY_ENTRY_BOOL("realtek,dmic-en");
+ if (byt_rt5651_quirk & BYT_RT5651_JD_NOT_INV)
+ props[cnt++] = PROPERTY_ENTRY_BOOL("realtek,jack-detect-not-inverted");
+
return device_add_properties(i2c_dev, props);
}
/* check quirks before creating card */
dmi_check_system(byt_rt5651_quirk_table);
- if (quirk_override) {
+ if (quirk_override != -1) {
dev_info(&pdev->dev, "Overriding quirk 0x%x => 0x%x\n",
(unsigned int)byt_rt5651_quirk, quirk_override);
byt_rt5651_quirk = quirk_override;
jack = &ctx->kabylake_headset;
- snd_jack_set_key(jack->jack, SND_JACK_BTN_0, KEY_MEDIA);
+ snd_jack_set_key(jack->jack, SND_JACK_BTN_0, KEY_PLAYPAUSE);
snd_jack_set_key(jack->jack, SND_JACK_BTN_1, KEY_VOLUMEUP);
snd_jack_set_key(jack->jack, SND_JACK_BTN_2, KEY_VOLUMEDOWN);
snd_jack_set_key(jack->jack, SND_JACK_BTN_3, KEY_VOICECOMMAND);
enum {
KBL_DPCM_AUDIO_PB = 0,
- KBL_DPCM_AUDIO_CP,
KBL_DPCM_AUDIO_ECHO_REF_CP,
KBL_DPCM_AUDIO_REF_CP,
KBL_DPCM_AUDIO_DMIC_CP,
KBL_DPCM_AUDIO_HDMI2_PB,
KBL_DPCM_AUDIO_HDMI3_PB,
KBL_DPCM_AUDIO_HS_PB,
+ KBL_DPCM_AUDIO_CP,
};
static int platform_clock_control(struct snd_soc_dapm_widget *w,
da7219_aad_jack_det(component, &ctx->kabylake_headset);
+ return 0;
+}
+
+static int kabylake_dmic_init(struct snd_soc_pcm_runtime *rtd)
+{
+ int ret;
ret = snd_soc_dapm_ignore_suspend(&rtd->card->dapm, "SoC DMIC");
if (ret)
dev_err(rtd->dev, "SoC DMIC - Ignore suspend failed %d\n", ret);
.dpcm_playback = 1,
.ops = &kabylake_da7219_fe_ops,
},
- [KBL_DPCM_AUDIO_CP] = {
- .name = "Kbl Audio Capture Port",
- .stream_name = "Audio Record",
- .cpu_dai_name = "System Pin",
- .platform_name = "0000:00:1f.3",
- .dynamic = 1,
- .codec_name = "snd-soc-dummy",
- .codec_dai_name = "snd-soc-dummy-dai",
- .nonatomic = 1,
- .trigger = {
- SND_SOC_DPCM_TRIGGER_POST, SND_SOC_DPCM_TRIGGER_POST},
- .dpcm_capture = 1,
- .ops = &kabylake_da7219_fe_ops,
- },
[KBL_DPCM_AUDIO_ECHO_REF_CP] = {
.name = "Kbl Audio Echo Reference cap",
.stream_name = "Echoreference Capture",
.ops = &kabylake_da7219_fe_ops,
},
+ [KBL_DPCM_AUDIO_CP] = {
+ .name = "Kbl Audio Capture Port",
+ .stream_name = "Audio Record",
+ .cpu_dai_name = "System Pin",
+ .platform_name = "0000:00:1f.3",
+ .dynamic = 1,
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .nonatomic = 1,
+ .trigger = {
+ SND_SOC_DPCM_TRIGGER_POST, SND_SOC_DPCM_TRIGGER_POST},
+ .dpcm_capture = 1,
+ .ops = &kabylake_da7219_fe_ops,
+ },
/* Back End DAI links */
{
.cpu_dai_name = "DMIC01 Pin",
.codec_name = "dmic-codec",
.codec_dai_name = "dmic-hifi",
+ .init = kabylake_dmic_init,
.platform_name = "0000:00:1f.3",
.be_hw_params_fixup = kabylake_dmic_fixup,
.ignore_suspend = 1,
.dpcm_playback = 1,
.ops = &kabylake_da7219_fe_ops,
},
- [KBL_DPCM_AUDIO_CP] = {
- .name = "Kbl Audio Capture Port",
- .stream_name = "Audio Record",
- .cpu_dai_name = "System Pin",
- .platform_name = "0000:00:1f.3",
- .dynamic = 1,
- .codec_name = "snd-soc-dummy",
- .codec_dai_name = "snd-soc-dummy-dai",
- .nonatomic = 1,
- .trigger = {
- SND_SOC_DPCM_TRIGGER_POST, SND_SOC_DPCM_TRIGGER_POST},
- .dpcm_capture = 1,
- .ops = &kabylake_da7219_fe_ops,
- },
[KBL_DPCM_AUDIO_ECHO_REF_CP] = {
.name = "Kbl Audio Echo Reference cap",
.stream_name = "Echoreference Capture",
.cpu_dai_name = "DMIC01 Pin",
.codec_name = "dmic-codec",
.codec_dai_name = "dmic-hifi",
+ .init = kabylake_dmic_init,
.platform_name = "0000:00:1f.3",
.be_hw_params_fixup = kabylake_dmic_fixup,
.ignore_suspend = 1,
.platform_name = "0000:00:1f.3",
.dpcm_playback = 1,
.dpcm_capture = 1,
- .init = NULL,
.no_pcm = 1,
},
{
.platform_name = "0000:00:1f.3",
.dpcm_playback = 1,
.dpcm_capture = 1,
- .init = NULL,
+ .no_pcm = 1,
+ },
+ {
+ .name = "dmic01",
+ .id = 6,
+ .cpu_dai_name = "DMIC01 Pin",
+ .codec_name = "dmic-codec",
+ .codec_dai_name = "dmic-hifi",
+ .platform_name = "0000:00:1f.3",
+ .dpcm_capture = 1,
+ .no_pcm = 1,
+ },
+ {
+ .name = "dmic16k",
+ .id = 7,
+ .cpu_dai_name = "DMIC16k Pin",
+ .codec_name = "dmic-codec",
+ .codec_dai_name = "dmic-hifi",
+ .platform_name = "0000:00:1f.3",
+ .dpcm_capture = 1,
.no_pcm = 1,
},
};
#include <sound/core.h>
#include <sound/jack.h>
-#define HDA_DSP_MAX_BE_DAI_LINKS 5
+#define HDA_DSP_MAX_BE_DAI_LINKS 7
struct skl_hda_hdmi_pcm {
struct list_head head;
};
#define IDISP_DAI_COUNT 3
+#define HDAC_DAI_COUNT 2
+#define DMIC_DAI_COUNT 2
+
/* there are two routes per iDisp output */
#define IDISP_ROUTE_COUNT (IDISP_DAI_COUNT * 2)
#define IDISP_CODEC_MASK 0x4
codec_count = hweight_long(codec_mask);
if (codec_count == 1 && codec_mask & IDISP_CODEC_MASK) {
- num_links = IDISP_DAI_COUNT;
+ num_links = IDISP_DAI_COUNT + DMIC_DAI_COUNT;
num_route = IDISP_ROUTE_COUNT;
+
+ /*
+ * rearrange the dai link array and make the
+ * dmic dai links follow idsp dai links for only
+ * num_links of dai links need to be registered
+ * to ASoC.
+ */
+ for (i = 0; i < DMIC_DAI_COUNT; i++) {
+ skl_hda_be_dai_links[IDISP_DAI_COUNT + i] =
+ skl_hda_be_dai_links[IDISP_DAI_COUNT +
+ HDAC_DAI_COUNT + i];
+ }
} else if (codec_count == 2 && codec_mask & IDISP_CODEC_MASK) {
num_links = ARRAY_SIZE(skl_hda_be_dai_links);
- num_route = ARRAY_SIZE(skl_hda_map),
+ num_route = ARRAY_SIZE(skl_hda_map);
card->dapm_widgets = skl_hda_widgets;
card->num_dapm_widgets = ARRAY_SIZE(skl_hda_widgets);
} else {
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+// Copyright(c) 2019 Intel Corporation.
+
+/*
+ * Intel SOF Machine Driver with Realtek rt5682 Codec
+ * and speaker codec MAX98357A
+ */
+#include <linux/i2c.h>
+#include <linux/input.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/dmi.h>
+#include <asm/cpu_device_id.h>
+#include <asm/intel-family.h>
+#include <sound/core.h>
+#include <sound/jack.h>
+#include <sound/pcm.h>
+#include <sound/pcm_params.h>
+#include <sound/soc.h>
+#include <sound/rt5682.h>
+#include <sound/soc-acpi.h>
+#include "../../codecs/rt5682.h"
+#include "../../codecs/hdac_hdmi.h"
+
+#define NAME_SIZE 32
+
+#define SOF_RT5682_SSP_CODEC(quirk) ((quirk) & GENMASK(2, 0))
+#define SOF_RT5682_SSP_CODEC_MASK (GENMASK(2, 0))
+#define SOF_RT5682_MCLK_EN BIT(3)
+#define SOF_RT5682_MCLK_24MHZ BIT(4)
+#define SOF_SPEAKER_AMP_PRESENT BIT(5)
+#define SOF_RT5682_SSP_AMP(quirk) ((quirk) & GENMASK(8, 6))
+#define SOF_RT5682_SSP_AMP_MASK (GENMASK(8, 6))
+#define SOF_RT5682_SSP_AMP_SHIFT 6
+
+/* Default: MCLK on, MCLK 19.2M, SSP0 */
+static unsigned long sof_rt5682_quirk = SOF_RT5682_MCLK_EN |
+ SOF_RT5682_SSP_CODEC(0);
+
+static int is_legacy_cpu;
+
+static struct snd_soc_jack sof_hdmi[3];
+
+struct sof_hdmi_pcm {
+ struct list_head head;
+ struct snd_soc_dai *codec_dai;
+ int device;
+};
+
+struct sof_card_private {
+ struct snd_soc_jack sof_headset;
+ struct list_head hdmi_pcm_list;
+};
+
+static int sof_rt5682_quirk_cb(const struct dmi_system_id *id)
+{
+ sof_rt5682_quirk = (unsigned long)id->driver_data;
+ return 1;
+}
+
+static const struct dmi_system_id sof_rt5682_quirk_table[] = {
+ {
+ .callback = sof_rt5682_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Intel Corporation"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "WhiskeyLake Client"),
+ },
+ .driver_data = (void *)(SOF_RT5682_MCLK_EN |
+ SOF_RT5682_MCLK_24MHZ |
+ SOF_RT5682_SSP_CODEC(1)),
+ },
+ {
+ .callback = sof_rt5682_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Google"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "Hatch"),
+ },
+ .driver_data = (void *)(SOF_RT5682_MCLK_EN |
+ SOF_RT5682_MCLK_24MHZ |
+ SOF_RT5682_SSP_CODEC(0) |
+ SOF_SPEAKER_AMP_PRESENT |
+ SOF_RT5682_SSP_AMP(1)),
+ },
+ {
+ .callback = sof_rt5682_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Intel Corporation"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "Ice Lake Client"),
+ },
+ .driver_data = (void *)(SOF_RT5682_MCLK_EN |
+ SOF_RT5682_SSP_CODEC(0)),
+ },
+ {}
+};
+
+static int sof_hdmi_init(struct snd_soc_pcm_runtime *rtd)
+{
+ struct sof_card_private *ctx = snd_soc_card_get_drvdata(rtd->card);
+ struct snd_soc_dai *dai = rtd->codec_dai;
+ struct sof_hdmi_pcm *pcm;
+
+ pcm = devm_kzalloc(rtd->card->dev, sizeof(*pcm), GFP_KERNEL);
+ if (!pcm)
+ return -ENOMEM;
+
+ /* dai_link id is 1:1 mapped to the PCM device */
+ pcm->device = rtd->dai_link->id;
+ pcm->codec_dai = dai;
+
+ list_add_tail(&pcm->head, &ctx->hdmi_pcm_list);
+
+ return 0;
+}
+
+static int sof_rt5682_codec_init(struct snd_soc_pcm_runtime *rtd)
+{
+ struct sof_card_private *ctx = snd_soc_card_get_drvdata(rtd->card);
+ struct snd_soc_component *component = rtd->codec_dai->component;
+ struct snd_soc_jack *jack;
+ int ret;
+
+ /* need to enable ASRC function for 24MHz mclk rate */
+ if ((sof_rt5682_quirk & SOF_RT5682_MCLK_EN) &&
+ (sof_rt5682_quirk & SOF_RT5682_MCLK_24MHZ)) {
+ rt5682_sel_asrc_clk_src(component, RT5682_DA_STEREO1_FILTER |
+ RT5682_AD_STEREO1_FILTER,
+ RT5682_CLK_SEL_I2S1_ASRC);
+ }
+
+ /*
+ * Headset buttons map to the google Reference headset.
+ * These can be configured by userspace.
+ */
+ ret = snd_soc_card_jack_new(rtd->card, "Headset Jack",
+ SND_JACK_HEADSET | SND_JACK_BTN_0 |
+ SND_JACK_BTN_1 | SND_JACK_BTN_2 |
+ SND_JACK_BTN_3,
+ &ctx->sof_headset, NULL, 0);
+ if (ret) {
+ dev_err(rtd->dev, "Headset Jack creation failed: %d\n", ret);
+ return ret;
+ }
+
+ jack = &ctx->sof_headset;
+
+ snd_jack_set_key(jack->jack, SND_JACK_BTN_0, KEY_PLAYPAUSE);
+ snd_jack_set_key(jack->jack, SND_JACK_BTN_1, KEY_VOLUMEUP);
+ snd_jack_set_key(jack->jack, SND_JACK_BTN_2, KEY_VOLUMEDOWN);
+ snd_jack_set_key(jack->jack, SND_JACK_BTN_3, KEY_VOICECOMMAND);
+ ret = snd_soc_component_set_jack(component, jack, NULL);
+
+ if (ret) {
+ dev_err(rtd->dev, "Headset Jack call-back failed: %d\n", ret);
+ return ret;
+ }
+
+ return ret;
+};
+
+static int sof_rt5682_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_dai *codec_dai = rtd->codec_dai;
+ int clk_id, clk_freq, pll_out, ret;
+
+ if (sof_rt5682_quirk & SOF_RT5682_MCLK_EN) {
+ clk_id = RT5682_PLL1_S_MCLK;
+ if (sof_rt5682_quirk & SOF_RT5682_MCLK_24MHZ)
+ clk_freq = 24000000;
+ else
+ clk_freq = 19200000;
+ } else {
+ clk_id = RT5682_PLL1_S_BCLK1;
+ clk_freq = params_rate(params) * 50;
+ }
+
+ pll_out = params_rate(params) * 512;
+
+ ret = snd_soc_dai_set_pll(codec_dai, 0, clk_id, clk_freq, pll_out);
+ if (ret < 0)
+ dev_err(rtd->dev, "snd_soc_dai_set_pll err = %d\n", ret);
+
+ /* Configure sysclk for codec */
+ ret = snd_soc_dai_set_sysclk(codec_dai, RT5682_SCLK_S_PLL1,
+ pll_out, SND_SOC_CLOCK_IN);
+ if (ret < 0)
+ dev_err(rtd->dev, "snd_soc_dai_set_sysclk err = %d\n", ret);
+
+ /*
+ * slot_width should equal or large than data length, set them
+ * be the same
+ */
+ ret = snd_soc_dai_set_tdm_slot(codec_dai, 0x0, 0x0, 2,
+ params_width(params));
+ if (ret < 0) {
+ dev_err(rtd->dev, "set TDM slot err:%d\n", ret);
+ return ret;
+ }
+
+ return ret;
+}
+
+static struct snd_soc_ops sof_rt5682_ops = {
+ .hw_params = sof_rt5682_hw_params,
+};
+
+static struct snd_soc_dai_link_component platform_component[] = {
+ {
+ /* name might be overridden during probe */
+ .name = "0000:00:1f.3"
+ }
+};
+
+static int sof_card_late_probe(struct snd_soc_card *card)
+{
+ struct sof_card_private *ctx = snd_soc_card_get_drvdata(card);
+ struct snd_soc_component *component = NULL;
+ char jack_name[NAME_SIZE];
+ struct sof_hdmi_pcm *pcm;
+ int err = 0;
+ int i = 0;
+
+ /* HDMI is not supported by SOF on Baytrail/CherryTrail */
+ if (is_legacy_cpu)
+ return 0;
+
+ list_for_each_entry(pcm, &ctx->hdmi_pcm_list, head) {
+ component = pcm->codec_dai->component;
+ snprintf(jack_name, sizeof(jack_name),
+ "HDMI/DP, pcm=%d Jack", pcm->device);
+ err = snd_soc_card_jack_new(card, jack_name,
+ SND_JACK_AVOUT, &sof_hdmi[i],
+ NULL, 0);
+
+ if (err)
+ return err;
+
+ err = hdac_hdmi_jack_init(pcm->codec_dai, pcm->device,
+ &sof_hdmi[i]);
+ if (err < 0)
+ return err;
+
+ i++;
+ }
+ if (!component)
+ return -EINVAL;
+
+ return hdac_hdmi_jack_port_init(component, &card->dapm);
+}
+
+static const struct snd_kcontrol_new sof_controls[] = {
+ SOC_DAPM_PIN_SWITCH("Headphone Jack"),
+ SOC_DAPM_PIN_SWITCH("Headset Mic"),
+ SOC_DAPM_PIN_SWITCH("Spk"),
+};
+
+static const struct snd_soc_dapm_widget sof_widgets[] = {
+ SND_SOC_DAPM_HP("Headphone Jack", NULL),
+ SND_SOC_DAPM_MIC("Headset Mic", NULL),
+ SND_SOC_DAPM_SPK("Spk", NULL),
+};
+
+static const struct snd_soc_dapm_route sof_map[] = {
+ /* HP jack connectors - unknown if we have jack detection */
+ { "Headphone Jack", NULL, "HPOL" },
+ { "Headphone Jack", NULL, "HPOR" },
+
+ /* other jacks */
+ { "IN1P", NULL, "Headset Mic" },
+
+};
+
+static const struct snd_soc_dapm_route speaker_map[] = {
+ /* speaker */
+ { "Spk", NULL, "Speaker" },
+};
+
+static int speaker_codec_init(struct snd_soc_pcm_runtime *rtd)
+{
+ struct snd_soc_card *card = rtd->card;
+ int ret;
+
+ ret = snd_soc_dapm_add_routes(&card->dapm, speaker_map,
+ ARRAY_SIZE(speaker_map));
+
+ if (ret)
+ dev_err(rtd->dev, "Speaker map addition failed: %d\n", ret);
+ return ret;
+}
+
+/* sof audio machine driver for rt5682 codec */
+static struct snd_soc_card sof_audio_card_rt5682 = {
+ .name = "sof_rt5682",
+ .owner = THIS_MODULE,
+ .controls = sof_controls,
+ .num_controls = ARRAY_SIZE(sof_controls),
+ .dapm_widgets = sof_widgets,
+ .num_dapm_widgets = ARRAY_SIZE(sof_widgets),
+ .dapm_routes = sof_map,
+ .num_dapm_routes = ARRAY_SIZE(sof_map),
+ .fully_routed = true,
+ .late_probe = sof_card_late_probe,
+};
+
+static const struct x86_cpu_id legacy_cpi_ids[] = {
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_SILVERMONT }, /* Baytrail */
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_AIRMONT }, /* Cherrytrail */
+ {}
+};
+
+static struct snd_soc_dai_link_component rt5682_component[] = {
+ {
+ .name = "i2c-10EC5682:00",
+ .dai_name = "rt5682-aif1",
+ }
+};
+
+static struct snd_soc_dai_link_component dmic_component[] = {
+ {
+ .name = "dmic-codec",
+ .dai_name = "dmic-hifi",
+ }
+};
+
+static struct snd_soc_dai_link_component max98357a_component[] = {
+ {
+ .name = "MX98357A:00",
+ .dai_name = "HiFi",
+ }
+};
+
+static struct snd_soc_dai_link *sof_card_dai_links_create(struct device *dev,
+ int ssp_codec,
+ int ssp_amp,
+ int dmic_num,
+ int hdmi_num)
+{
+ struct snd_soc_dai_link_component *idisp_components;
+ struct snd_soc_dai_link *links;
+ int i, id = 0;
+
+ links = devm_kzalloc(dev, sizeof(struct snd_soc_dai_link) *
+ sof_audio_card_rt5682.num_links, GFP_KERNEL);
+ if (!links)
+ goto devm_err;
+
+ /* codec SSP */
+ links[id].name = devm_kasprintf(dev, GFP_KERNEL,
+ "SSP%d-Codec", ssp_codec);
+ if (!links[id].name)
+ goto devm_err;
+
+ links[id].id = id;
+ links[id].codecs = rt5682_component;
+ links[id].num_codecs = ARRAY_SIZE(rt5682_component);
+ links[id].platforms = platform_component;
+ links[id].num_platforms = ARRAY_SIZE(platform_component);
+ links[id].init = sof_rt5682_codec_init;
+ links[id].ops = &sof_rt5682_ops;
+ links[id].nonatomic = true;
+ links[id].dpcm_playback = 1;
+ links[id].dpcm_capture = 1;
+ links[id].no_pcm = 1;
+ if (is_legacy_cpu) {
+ links[id].cpu_dai_name = devm_kasprintf(dev, GFP_KERNEL,
+ "ssp%d-port",
+ ssp_codec);
+ if (!links[id].cpu_dai_name)
+ goto devm_err;
+ } else {
+ /*
+ * Currently, On SKL+ platforms MCLK will be turned off in sof
+ * runtime suspended, and it will go into runtime suspended
+ * right after playback is stop. However, rt5682 will output
+ * static noise if sysclk turns off during playback. Set
+ * ignore_pmdown_time to power down rt5682 immediately and
+ * avoid the noise.
+ * It can be removed once we can control MCLK by driver.
+ */
+ links[id].ignore_pmdown_time = 1;
+ links[id].cpu_dai_name = devm_kasprintf(dev, GFP_KERNEL,
+ "SSP%d Pin",
+ ssp_codec);
+ if (!links[id].cpu_dai_name)
+ goto devm_err;
+ }
+ id++;
+
+ /* dmic */
+ for (i = 1; i <= dmic_num; i++) {
+ links[id].name = devm_kasprintf(dev, GFP_KERNEL,
+ "dmic%02d", i);
+ if (!links[id].name)
+ goto devm_err;
+
+ links[id].id = id;
+ links[id].cpu_dai_name = devm_kasprintf(dev, GFP_KERNEL,
+ "DMIC%02d Pin", i);
+ if (!links[id].cpu_dai_name)
+ goto devm_err;
+
+ links[id].codecs = dmic_component;
+ links[id].num_codecs = ARRAY_SIZE(dmic_component);
+ links[id].platforms = platform_component;
+ links[id].num_platforms = ARRAY_SIZE(platform_component);
+ links[id].ignore_suspend = 1;
+ links[id].dpcm_capture = 1;
+ links[id].no_pcm = 1;
+ id++;
+ }
+
+ /* HDMI */
+ if (hdmi_num > 0) {
+ idisp_components = devm_kzalloc(dev,
+ sizeof(struct snd_soc_dai_link_component) *
+ hdmi_num, GFP_KERNEL);
+ if (!idisp_components)
+ goto devm_err;
+ }
+ for (i = 1; i <= hdmi_num; i++) {
+ links[id].name = devm_kasprintf(dev, GFP_KERNEL,
+ "iDisp%d", i);
+ if (!links[id].name)
+ goto devm_err;
+
+ links[id].id = id;
+ links[id].cpu_dai_name = devm_kasprintf(dev, GFP_KERNEL,
+ "iDisp%d Pin", i);
+ if (!links[id].cpu_dai_name)
+ goto devm_err;
+
+ idisp_components[i - 1].name = "ehdaudio0D2";
+ idisp_components[i - 1].dai_name = devm_kasprintf(dev,
+ GFP_KERNEL,
+ "intel-hdmi-hifi%d",
+ i);
+ if (!idisp_components[i - 1].dai_name)
+ goto devm_err;
+
+ links[id].codecs = &idisp_components[i - 1];
+ links[id].num_codecs = 1;
+ links[id].platforms = platform_component;
+ links[id].num_platforms = ARRAY_SIZE(platform_component);
+ links[id].init = sof_hdmi_init;
+ links[id].dpcm_playback = 1;
+ links[id].no_pcm = 1;
+ id++;
+ }
+
+ /* speaker amp */
+ if (sof_rt5682_quirk & SOF_SPEAKER_AMP_PRESENT) {
+ links[id].name = devm_kasprintf(dev, GFP_KERNEL,
+ "SSP%d-Codec", ssp_amp);
+ if (!links[id].name)
+ goto devm_err;
+
+ links[id].id = id;
+ links[id].codecs = max98357a_component;
+ links[id].num_codecs = ARRAY_SIZE(max98357a_component);
+ links[id].platforms = platform_component;
+ links[id].num_platforms = ARRAY_SIZE(platform_component);
+ links[id].init = speaker_codec_init,
+ links[id].nonatomic = true;
+ links[id].dpcm_playback = 1;
+ links[id].no_pcm = 1;
+ if (is_legacy_cpu) {
+ links[id].cpu_dai_name = devm_kasprintf(dev, GFP_KERNEL,
+ "ssp%d-port",
+ ssp_amp);
+ if (!links[id].cpu_dai_name)
+ goto devm_err;
+
+ } else {
+ links[id].cpu_dai_name = devm_kasprintf(dev, GFP_KERNEL,
+ "SSP%d Pin",
+ ssp_amp);
+ if (!links[id].cpu_dai_name)
+ goto devm_err;
+ }
+ }
+
+ return links;
+devm_err:
+ return NULL;
+}
+
+static int sof_audio_probe(struct platform_device *pdev)
+{
+ struct snd_soc_dai_link *dai_links;
+ struct snd_soc_acpi_mach *mach;
+ struct sof_card_private *ctx;
+ int dmic_num, hdmi_num;
+ int ret, ssp_amp, ssp_codec;
+
+ ctx = devm_kzalloc(&pdev->dev, sizeof(*ctx), GFP_ATOMIC);
+ if (!ctx)
+ return -ENOMEM;
+
+ if (x86_match_cpu(legacy_cpi_ids)) {
+ is_legacy_cpu = 1;
+ dmic_num = 0;
+ hdmi_num = 0;
+ /* default quirk for legacy cpu */
+ sof_rt5682_quirk = SOF_RT5682_SSP_CODEC(2);
+ } else {
+ dmic_num = 1;
+ hdmi_num = 3;
+ }
+
+ dmi_check_system(sof_rt5682_quirk_table);
+
+ dev_dbg(&pdev->dev, "sof_rt5682_quirk = %lx\n", sof_rt5682_quirk);
+
+ ssp_amp = (sof_rt5682_quirk & SOF_RT5682_SSP_AMP_MASK) >>
+ SOF_RT5682_SSP_AMP_SHIFT;
+
+ ssp_codec = sof_rt5682_quirk & SOF_RT5682_SSP_CODEC_MASK;
+
+ /* compute number of dai links */
+ sof_audio_card_rt5682.num_links = 1 + dmic_num + hdmi_num;
+ if (sof_rt5682_quirk & SOF_SPEAKER_AMP_PRESENT)
+ sof_audio_card_rt5682.num_links++;
+
+ dai_links = sof_card_dai_links_create(&pdev->dev, ssp_codec, ssp_amp,
+ dmic_num, hdmi_num);
+ if (!dai_links)
+ return -ENOMEM;
+
+ sof_audio_card_rt5682.dai_link = dai_links;
+
+ INIT_LIST_HEAD(&ctx->hdmi_pcm_list);
+
+ sof_audio_card_rt5682.dev = &pdev->dev;
+ mach = (&pdev->dev)->platform_data;
+
+ /* set platform name for each dailink */
+ ret = snd_soc_fixup_dai_links_platform_name(&sof_audio_card_rt5682,
+ mach->mach_params.platform);
+ if (ret)
+ return ret;
+
+ snd_soc_card_set_drvdata(&sof_audio_card_rt5682, ctx);
+
+ return devm_snd_soc_register_card(&pdev->dev,
+ &sof_audio_card_rt5682);
+}
+
+static struct platform_driver sof_audio = {
+ .probe = sof_audio_probe,
+ .driver = {
+ .name = "sof_rt5682",
+ .pm = &snd_soc_pm_ops,
+ },
+};
+module_platform_driver(sof_audio)
+
+/* Module information */
+MODULE_DESCRIPTION("SOF Audio Machine driver");
+MODULE_AUTHOR("Bard Liao <bard.liao@intel.com>");
+MODULE_AUTHOR("Sathya Prakash M R <sathya.prakash.m.r@intel.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:sof_rt5682");
.sof_fw_filename = "sof-byt.ri",
.sof_tplg_filename = "sof-byt-es8316.tplg",
},
+ {
+ .id = "10EC5682",
+ .drv_name = "sof_rt5682",
+ .sof_fw_filename = "sof-byt.ri",
+ .sof_tplg_filename = "sof-byt-rt5682.tplg",
+ },
/* some Baytrail platforms rely on RT5645, use CHT machine driver */
{
.id = "10EC5645",
.sof_fw_filename = "sof-cht.ri",
.sof_tplg_filename = "sof-cht-rt5640.tplg",
},
+ {
+ .id = "10EC5682",
+ .drv_name = "sof_rt5682",
+ .sof_fw_filename = "sof-cht.ri",
+ .sof_tplg_filename = "sof-cht-rt5682.tplg",
+ },
/* some CHT-T platforms rely on RT5651, use Baytrail machine driver */
{
.id = "10EC5651",
.use_tplg_pcm = true,
};
+static struct snd_soc_acpi_codecs cml_codecs = {
+ .num_codecs = 1,
+ .codecs = {"10EC5682"}
+};
+
struct snd_soc_acpi_mach snd_soc_acpi_intel_cnl_machines[] = {
{
.id = "INT34C2",
.sof_fw_filename = "sof-cnl.ri",
.sof_tplg_filename = "sof-cnl-rt274.tplg",
},
+ {
+ .id = "10EC5682",
+ .drv_name = "sof_rt5682",
+ .sof_fw_filename = "sof-cnl.ri",
+ .sof_tplg_filename = "sof-cml-rt5682.tplg",
+ },
+ {
+ .id = "MX98357A",
+ .drv_name = "sof_rt5682",
+ .quirk_data = &cml_codecs,
+ .sof_fw_filename = "sof-cnl.ri",
+ .sof_tplg_filename = "sof-cml-rt5682-max98357a.tplg",
+ },
+
{},
};
EXPORT_SYMBOL_GPL(snd_soc_acpi_intel_cnl_machines);
.sof_fw_filename = "sof-glk.ri",
.sof_tplg_filename = "sof-glk-da7219.tplg",
},
+ {
+ .id = "10EC5682",
+ .drv_name = "glk_rt5682_max98357a",
+ .fw_filename = "intel/dsp_fw_glk.bin",
+ .machine_quirk = snd_soc_acpi_codec_list,
+ .quirk_data = &glk_codecs,
+ .sof_fw_filename = "sof-glk.ri",
+ .sof_tplg_filename = "sof-glk-rt5682.tplg",
+ },
{},
};
EXPORT_SYMBOL_GPL(snd_soc_acpi_intel_glk_machines);
.sof_fw_filename = "sof-icl.ri",
.sof_tplg_filename = "sof-icl-rt274.tplg",
},
+ {
+ .id = "10EC5682",
+ .drv_name = "sof_rt5682",
+ .sof_fw_filename = "sof-icl.ri",
+ .sof_tplg_filename = "sof-icl-rt5682.tplg",
+ },
{},
};
EXPORT_SYMBOL_GPL(snd_soc_acpi_intel_icl_machines);
goto irq_err;
err = sst_dma_new(sst);
- if (err)
- dev_warn(dev, "sst_dma_new failed %d\n", err);
+ if (err) {
+ dev_err(dev, "sst_dma_new failed %d\n", err);
+ goto dma_err;
+ }
return sst;
+dma_err:
+ free_irq(sst->irq, sst);
irq_err:
if (sst->ops->free)
sst->ops->free(sst);
return (msg & IPC_STR_TYPE_MASK) >> IPC_STR_TYPE_SHIFT;
}
-static inline u32 msg_get_stage_type(u32 msg)
-{
- return (msg & IPC_STG_TYPE_MASK) >> IPC_STG_TYPE_SHIFT;
-}
-
static inline u32 msg_get_stream_id(u32 msg)
{
return (msg & IPC_STR_ID_MASK) >> IPC_STR_ID_SHIFT;
static int hsw_stream_message(struct sst_hsw *hsw, u32 header)
{
- u32 stream_msg, stream_id, stage_type;
+ u32 stream_msg, stream_id;
struct sst_hsw_stream *stream;
int handled = 0;
stream_msg = msg_get_stream_type(header);
stream_id = msg_get_stream_id(header);
- stage_type = msg_get_stage_type(header);
stream = get_stream_by_id(hsw, stream_id);
if (stream == NULL)
config SND_JZ4740_SOC
tristate "SoC Audio for Ingenic JZ4740 SoC"
- depends on MACH_JZ4740 || COMPILE_TEST
+ depends on MIPS || COMPILE_TEST
select SND_SOC_GENERIC_DMAENGINE_PCM
help
Say Y or M if you want to add support for codecs attached to
Select Y if you have such device.
If unsure select "N".
+config SND_SOC_MT8183_MT6358_TS3A227E_MAX98357A
+ tristate "ASoC Audio driver for MT8183 with MT6358 TS3A227E MAX98357A codec"
+ depends on I2C
+ depends on SND_SOC_MT8183
+ select SND_SOC_MT6358
+ select SND_SOC_MAX98357A
+ select SND_SOC_BT_SCO
+ select SND_SOC_TS3A227E
+ help
+ This adds ASoC driver for Mediatek MT8183 boards
+ with the MT6358 TS3A227E MAX98357A audio codec.
+ Select Y if you have such device.
+ If unsure select "N".
+
+config SND_SOC_MT8183_DA7219_MAX98357A
+ tristate "ASoC Audio driver for MT8183 with DA7219 MAX98357A codec"
+ depends on SND_SOC_MT8183
+ select SND_SOC_MT6358
+ select SND_SOC_MAX98357A
+ select SND_SOC_DA7219
+ select SND_SOC_BT_SCO
+ help
+ This adds ASoC driver for Mediatek MT8183 boards
+ with the DA7219 MAX98357A audio codec.
+ Select Y if you have such device.
+ If unsure select "N".
+
config SND_SOC_MTK_BTCVSD
tristate "ALSA BT SCO CVSD/MSBC Driver"
help
static int mtk_regmap_update_bits(struct regmap *map, int reg,
unsigned int mask,
- unsigned int val)
+ unsigned int val, int shift)
{
- if (reg < 0)
+ if (reg < 0 || WARN_ON_ONCE(shift < 0))
return 0;
- return regmap_update_bits(map, reg, mask, val);
+ return regmap_update_bits(map, reg, mask << shift, val << shift);
}
static int mtk_regmap_write(struct regmap *map, int reg, unsigned int val)
SNDRV_PCM_HW_PARAM_BUFFER_BYTES, 16);
/* enable agent */
mtk_regmap_update_bits(afe->regmap, memif->data->agent_disable_reg,
- 1 << memif->data->agent_disable_shift,
- 0 << memif->data->agent_disable_shift);
+ 1, 0, memif->data->agent_disable_shift);
snd_soc_set_runtime_hwparams(substream, mtk_afe_hardware);
irq_id = memif->irq_usage;
mtk_regmap_update_bits(afe->regmap, memif->data->agent_disable_reg,
- 1 << memif->data->agent_disable_shift,
- 1 << memif->data->agent_disable_shift);
+ 1, 1, memif->data->agent_disable_shift);
if (!memif->const_irq) {
mtk_dynamic_irq_release(afe, irq_id);
/* set MSB to 33-bit */
mtk_regmap_update_bits(afe->regmap, memif->data->msb_reg,
- 1 << memif->data->msb_shift,
- msb_at_bit33 << memif->data->msb_shift);
+ 1, msb_at_bit33, memif->data->msb_shift);
/* set channel */
if (memif->data->mono_shift >= 0) {
unsigned int mono = (params_channels(params) == 1) ? 1 : 0;
mtk_regmap_update_bits(afe->regmap, memif->data->mono_reg,
- 1 << memif->data->mono_shift,
- mono << memif->data->mono_shift);
+ 1, mono, memif->data->mono_shift);
}
/* set rate */
return -EINVAL;
mtk_regmap_update_bits(afe->regmap, memif->data->fs_reg,
- memif->data->fs_maskbit << memif->data->fs_shift,
- fs << memif->data->fs_shift);
+ memif->data->fs_maskbit, fs,
+ memif->data->fs_shift);
return 0;
}
switch (cmd) {
case SNDRV_PCM_TRIGGER_START:
case SNDRV_PCM_TRIGGER_RESUME:
- if (memif->data->enable_shift >= 0)
- mtk_regmap_update_bits(afe->regmap,
- memif->data->enable_reg,
- 1 << memif->data->enable_shift,
- 1 << memif->data->enable_shift);
+ mtk_regmap_update_bits(afe->regmap,
+ memif->data->enable_reg,
+ 1, 1, memif->data->enable_shift);
/* set irq counter */
mtk_regmap_update_bits(afe->regmap, irq_data->irq_cnt_reg,
- irq_data->irq_cnt_maskbit
- << irq_data->irq_cnt_shift,
- counter << irq_data->irq_cnt_shift);
+ irq_data->irq_cnt_maskbit, counter,
+ irq_data->irq_cnt_shift);
/* set irq fs */
fs = afe->irq_fs(substream, runtime->rate);
return -EINVAL;
mtk_regmap_update_bits(afe->regmap, irq_data->irq_fs_reg,
- irq_data->irq_fs_maskbit
- << irq_data->irq_fs_shift,
- fs << irq_data->irq_fs_shift);
+ irq_data->irq_fs_maskbit, fs,
+ irq_data->irq_fs_shift);
/* enable interrupt */
mtk_regmap_update_bits(afe->regmap, irq_data->irq_en_reg,
- 1 << irq_data->irq_en_shift,
- 1 << irq_data->irq_en_shift);
+ 1, 1, irq_data->irq_en_shift);
return 0;
case SNDRV_PCM_TRIGGER_STOP:
case SNDRV_PCM_TRIGGER_SUSPEND:
mtk_regmap_update_bits(afe->regmap, memif->data->enable_reg,
- 1 << memif->data->enable_shift, 0);
+ 1, 0, memif->data->enable_shift);
/* disable interrupt */
mtk_regmap_update_bits(afe->regmap, irq_data->irq_en_reg,
- 1 << irq_data->irq_en_shift,
- 0 << irq_data->irq_en_shift);
+ 1, 0, irq_data->irq_en_shift);
/* and clear pending IRQ */
mtk_regmap_write(afe->regmap, irq_data->irq_clr_reg,
1 << irq_data->irq_clr_shift);
}
mtk_regmap_update_bits(afe->regmap, memif->data->hd_reg,
- 1 << memif->data->hd_shift,
- hd_audio << memif->data->hd_shift);
+ 1, hd_audio, memif->data->hd_shift);
return 0;
}
static void mtk_btcvsd_snd_irq_enable(struct mtk_btcvsd_snd *bt)
{
regmap_update_bits(bt->infra, bt->infra_misc_offset,
- bt->conn_bt_cvsd_mask, bt->conn_bt_cvsd_mask);
+ bt->conn_bt_cvsd_mask, 0);
}
static void mtk_btcvsd_snd_irq_disable(struct mtk_btcvsd_snd *bt)
{
regmap_update_bits(bt->infra, bt->infra_misc_offset,
- bt->conn_bt_cvsd_mask, 0);
+ bt->conn_bt_cvsd_mask, bt->conn_bt_cvsd_mask);
}
static void mtk_btcvsd_snd_set_state(struct mtk_btcvsd_snd *bt,
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 6,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "DL2",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 7,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "DL3",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 8,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "DL4",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 9,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "DL5",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 10,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "DLM",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 12,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "UL1",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 0,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "UL2",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 1,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "UL3",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 2,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "UL4",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 3,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "UL5",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 4,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "DLBT",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 13,
.msb_reg = -1,
- .msb_shift = -1,
},
{
.name = "ULBT",
.agent_disable_reg = AUDIO_TOP_CON5,
.agent_disable_shift = 16,
.msb_reg = -1,
- .msb_shift = -1,
},
};
.hd_reg = AFE_MEMIF_HD_MODE,
.hd_shift = DL1_HD_SFT,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
.msb_reg = -1,
- .msb_shift = -1,
},
[MT6797_MEMIF_DL2] = {
.name = "DL2",
.hd_reg = AFE_MEMIF_HD_MODE,
.hd_shift = DL2_HD_SFT,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
.msb_reg = -1,
- .msb_shift = -1,
},
[MT6797_MEMIF_DL3] = {
.name = "DL3",
.hd_reg = AFE_MEMIF_HD_MODE,
.hd_shift = DL3_HD_SFT,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
.msb_reg = -1,
- .msb_shift = -1,
},
[MT6797_MEMIF_VUL] = {
.name = "VUL",
.hd_reg = AFE_MEMIF_HD_MODE,
.hd_shift = VUL_HD_SFT,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
.msb_reg = -1,
- .msb_shift = -1,
},
[MT6797_MEMIF_AWB] = {
.name = "AWB",
.hd_reg = AFE_MEMIF_HD_MODE,
.hd_shift = AWB_HD_SFT,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
.msb_reg = -1,
- .msb_shift = -1,
},
[MT6797_MEMIF_VUL12] = {
.name = "VUL12",
.hd_reg = AFE_MEMIF_HD_MODE,
.hd_shift = VUL_DATA2_HD_SFT,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
.msb_reg = -1,
- .msb_shift = -1,
},
[MT6797_MEMIF_DAI] = {
.name = "DAI",
.hd_reg = AFE_MEMIF_HD_MODE,
.hd_shift = DAI_HD_SFT,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
.msb_reg = -1,
- .msb_shift = -1,
},
[MT6797_MEMIF_MOD_DAI] = {
.name = "MOD_DAI",
.hd_reg = AFE_MEMIF_HD_MODE,
.hd_shift = MOD_DAI_HD_SFT,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
.msb_reg = -1,
- .msb_shift = -1,
},
};
.mono_reg = AFE_DAC_CON1,
.mono_shift = 21,
.hd_reg = -1,
- .hd_shift = -1,
.enable_reg = AFE_DAC_CON0,
.enable_shift = 1,
.msb_reg = AFE_MEMIF_MSB,
.msb_shift = 0,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
}, {
.name = "DL2",
.id = MT8173_AFE_MEMIF_DL2,
.mono_reg = AFE_DAC_CON1,
.mono_shift = 22,
.hd_reg = -1,
- .hd_shift = -1,
.enable_reg = AFE_DAC_CON0,
.enable_shift = 2,
.msb_reg = AFE_MEMIF_MSB,
.msb_shift = 1,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
}, {
.name = "VUL",
.id = MT8173_AFE_MEMIF_VUL,
.mono_reg = AFE_DAC_CON1,
.mono_shift = 27,
.hd_reg = -1,
- .hd_shift = -1,
.enable_reg = AFE_DAC_CON0,
.enable_shift = 3,
.msb_reg = AFE_MEMIF_MSB,
.msb_shift = 6,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
}, {
.name = "DAI",
.id = MT8173_AFE_MEMIF_DAI,
.mono_reg = -1,
.mono_shift = -1,
.hd_reg = -1,
- .hd_shift = -1,
.enable_reg = AFE_DAC_CON0,
.enable_shift = 4,
.msb_reg = AFE_MEMIF_MSB,
.msb_shift = 5,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
}, {
.name = "AWB",
.id = MT8173_AFE_MEMIF_AWB,
.mono_reg = AFE_DAC_CON1,
.mono_shift = 24,
.hd_reg = -1,
- .hd_shift = -1,
.enable_reg = AFE_DAC_CON0,
.enable_shift = 6,
.msb_reg = AFE_MEMIF_MSB,
.msb_shift = 3,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
}, {
.name = "MOD_DAI",
.id = MT8173_AFE_MEMIF_MOD_DAI,
.mono_reg = AFE_DAC_CON1,
.mono_shift = 30,
.hd_reg = -1,
- .hd_shift = -1,
.enable_reg = AFE_DAC_CON0,
.enable_shift = 7,
.msb_reg = AFE_MEMIF_MSB,
.msb_shift = 4,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
}, {
.name = "HDMI",
.id = MT8173_AFE_MEMIF_HDMI,
.mono_reg = -1,
.mono_shift = -1,
.hd_reg = -1,
- .hd_shift = -1,
.enable_reg = -1,
- .enable_shift = -1,
.msb_reg = AFE_MEMIF_MSB,
.msb_shift = 8,
.agent_disable_reg = -1,
- .agent_disable_shift = -1,
},
};
.irq_en_reg = AFE_IRQ_MCU_CON,
.irq_en_shift = 12,
.irq_fs_reg = -1,
- .irq_fs_shift = -1,
.irq_fs_maskbit = -1,
.irq_clr_reg = AFE_IRQ_CLR,
.irq_clr_shift = 4,
mt8183-dai-adda.o
obj-$(CONFIG_SND_SOC_MT8183) += snd-soc-mt8183-afe.o
+obj-$(CONFIG_SND_SOC_MT8183_MT6358_TS3A227E_MAX98357A) += mt8183-mt6358-ts3a227-max98357.o
+obj-$(CONFIG_SND_SOC_MT8183_DA7219_MAX98357A) += mt8183-da7219-max98357.o
static const struct snd_kcontrol_new memif_ul1_ch1_mix[] = {
SOC_DAPM_SINGLE_AUTODISABLE("ADDA_UL_CH1", AFE_CONN21,
I_ADDA_UL_CH1, 1, 0),
+ SOC_DAPM_SINGLE_AUTODISABLE("I2S0_CH1", AFE_CONN21,
+ I_I2S0_CH1, 1, 0),
};
static const struct snd_kcontrol_new memif_ul1_ch2_mix[] = {
SOC_DAPM_SINGLE_AUTODISABLE("ADDA_UL_CH2", AFE_CONN22,
I_ADDA_UL_CH2, 1, 0),
+ SOC_DAPM_SINGLE_AUTODISABLE("I2S0_CH2", AFE_CONN21,
+ I_I2S0_CH2, 1, 0),
};
static const struct snd_kcontrol_new memif_ul2_ch1_mix[] = {
I_DL2_CH1, 1, 0),
SOC_DAPM_SINGLE_AUTODISABLE("DL3_CH1", AFE_CONN5,
I_DL3_CH1, 1, 0),
+ SOC_DAPM_SINGLE_AUTODISABLE("I2S2_CH1", AFE_CONN5,
+ I_I2S2_CH1, 1, 0),
};
static const struct snd_kcontrol_new memif_ul2_ch2_mix[] = {
I_DL2_CH2, 1, 0),
SOC_DAPM_SINGLE_AUTODISABLE("DL3_CH2", AFE_CONN6,
I_DL3_CH2, 1, 0),
+ SOC_DAPM_SINGLE_AUTODISABLE("I2S2_CH2", AFE_CONN6,
+ I_I2S2_CH2, 1, 0),
};
static const struct snd_kcontrol_new memif_ul3_ch1_mix[] = {
SOC_DAPM_SINGLE_AUTODISABLE("ADDA_UL_CH1", AFE_CONN32,
I_ADDA_UL_CH1, 1, 0),
+ SOC_DAPM_SINGLE_AUTODISABLE("I2S2_CH1", AFE_CONN32,
+ I_I2S2_CH1, 1, 0),
};
static const struct snd_kcontrol_new memif_ul3_ch2_mix[] = {
SOC_DAPM_SINGLE_AUTODISABLE("ADDA_UL_CH2", AFE_CONN33,
I_ADDA_UL_CH2, 1, 0),
+ SOC_DAPM_SINGLE_AUTODISABLE("I2S2_CH2", AFE_CONN33,
+ I_I2S2_CH2, 1, 0),
};
static const struct snd_kcontrol_new memif_ul4_ch1_mix[] = {
{"UL1", NULL, "UL1_CH2"},
{"UL1_CH1", "ADDA_UL_CH1", "ADDA Capture"},
{"UL1_CH2", "ADDA_UL_CH2", "ADDA Capture"},
+ {"UL1_CH1", "I2S0_CH1", "I2S0"},
+ {"UL1_CH2", "I2S0_CH2", "I2S0"},
{"UL2", NULL, "UL2_CH1"},
{"UL2", NULL, "UL2_CH2"},
{"UL2_CH1", "ADDA_UL_CH1", "ADDA Capture"},
{"UL2_CH2", "ADDA_UL_CH2", "ADDA Capture"},
+ {"UL2_CH1", "I2S2_CH1", "I2S2"},
+ {"UL2_CH2", "I2S2_CH2", "I2S2"},
{"UL3", NULL, "UL3_CH1"},
{"UL3", NULL, "UL3_CH2"},
{"UL3_CH1", "ADDA_UL_CH1", "ADDA Capture"},
{"UL3_CH2", "ADDA_UL_CH2", "ADDA Capture"},
+ {"UL3_CH1", "I2S2_CH1", "I2S2"},
+ {"UL3_CH2", "I2S2_CH2", "I2S2"},
{"UL4", NULL, "UL4_CH1"},
{"UL4", NULL, "UL4_CH2"},
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// mt8183-da7219-max98357.c
+// -- MT8183-DA7219-MAX98357 ALSA SoC machine driver
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Shunli Wang <shunli.wang@mediatek.com>
+
+#include <linux/module.h>
+#include <sound/pcm_params.h>
+#include <sound/soc.h>
+#include <sound/jack.h>
+#include <linux/pinctrl/consumer.h>
+
+#include "mt8183-afe-common.h"
+#include "../../codecs/da7219-aad.h"
+#include "../../codecs/da7219.h"
+
+static struct snd_soc_jack headset_jack;
+
+/* Headset jack detection DAPM pins */
+static struct snd_soc_jack_pin headset_jack_pins[] = {
+ {
+ .pin = "Headphone",
+ .mask = SND_JACK_HEADPHONE,
+ },
+ {
+ .pin = "Headset Mic",
+ .mask = SND_JACK_MICROPHONE,
+ },
+};
+
+static struct snd_soc_dai_link_component
+mt8183_da7219_max98357_external_codecs[] = {
+ {
+ .name = "max98357a",
+ .dai_name = "HiFi",
+ },
+ {
+ .name = "da7219.5-001a",
+ .dai_name = "da7219-hifi",
+ },
+};
+
+static int mt8183_mt6358_i2s_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ unsigned int rate = params_rate(params);
+ unsigned int mclk_fs_ratio = 128;
+ unsigned int mclk_fs = rate * mclk_fs_ratio;
+
+ return snd_soc_dai_set_sysclk(rtd->cpu_dai,
+ 0, mclk_fs, SND_SOC_CLOCK_OUT);
+}
+
+static const struct snd_soc_ops mt8183_mt6358_i2s_ops = {
+ .hw_params = mt8183_mt6358_i2s_hw_params,
+};
+
+static int mt8183_da7219_i2s_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ unsigned int rate = params_rate(params);
+ unsigned int mclk_fs_ratio = 256;
+ unsigned int mclk_fs = rate * mclk_fs_ratio;
+ unsigned int freq;
+ int ret = 0, j;
+
+ ret = snd_soc_dai_set_sysclk(rtd->cpu_dai, 0,
+ mclk_fs, SND_SOC_CLOCK_OUT);
+ if (ret < 0)
+ dev_err(rtd->dev, "failed to set cpu dai sysclk\n");
+
+ for (j = 0; j < rtd->num_codecs; j++) {
+ struct snd_soc_dai *codec_dai = rtd->codec_dais[j];
+
+ if (!strcmp(codec_dai->component->name, "da7219.5-001a")) {
+ ret = snd_soc_dai_set_sysclk(codec_dai,
+ DA7219_CLKSRC_MCLK,
+ mclk_fs,
+ SND_SOC_CLOCK_IN);
+ if (ret < 0)
+ dev_err(rtd->dev, "failed to set sysclk\n");
+
+ if ((rate % 8000) == 0)
+ freq = DA7219_PLL_FREQ_OUT_98304;
+ else
+ freq = DA7219_PLL_FREQ_OUT_90316;
+
+ ret = snd_soc_dai_set_pll(codec_dai, 0,
+ DA7219_SYSCLK_PLL_SRM,
+ 0, freq);
+ if (ret)
+ dev_err(rtd->dev, "failed to start PLL: %d\n",
+ ret);
+ }
+ }
+
+ return ret;
+}
+
+static int mt8183_da7219_hw_free(struct snd_pcm_substream *substream)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ int ret = 0, j;
+
+ for (j = 0; j < rtd->num_codecs; j++) {
+ struct snd_soc_dai *codec_dai = rtd->codec_dais[j];
+
+ if (!strcmp(codec_dai->component->name, "da7219.5-001a")) {
+ ret = snd_soc_dai_set_pll(codec_dai,
+ 0, DA7219_SYSCLK_MCLK, 0, 0);
+ if (ret < 0) {
+ dev_err(rtd->dev, "failed to stop PLL: %d\n",
+ ret);
+ break;
+ }
+ }
+ }
+
+ return ret;
+}
+
+static const struct snd_soc_ops mt8183_da7219_i2s_ops = {
+ .hw_params = mt8183_da7219_i2s_hw_params,
+ .hw_free = mt8183_da7219_hw_free,
+};
+
+static int mt8183_i2s_hw_params_fixup(struct snd_soc_pcm_runtime *rtd,
+ struct snd_pcm_hw_params *params)
+{
+ /* fix BE i2s format to 32bit, clean param mask first */
+ snd_mask_reset_range(hw_param_mask(params, SNDRV_PCM_HW_PARAM_FORMAT),
+ 0, SNDRV_PCM_FORMAT_LAST);
+
+ params_set_format(params, SNDRV_PCM_FORMAT_S32_LE);
+
+ return 0;
+}
+
+static const struct snd_soc_dapm_widget
+mt8183_da7219_max98357_dapm_widgets[] = {
+ SND_SOC_DAPM_OUTPUT("IT6505_8CH"),
+};
+
+static const struct snd_soc_dapm_route mt8183_da7219_max98357_dapm_routes[] = {
+ {"IT6505_8CH", NULL, "TDM"},
+};
+
+static struct snd_soc_dai_link mt8183_da7219_max98357_dai_links[] = {
+ /* FE */
+ {
+ .name = "Playback_1",
+ .stream_name = "Playback_1",
+ .cpu_dai_name = "DL1",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_playback = 1,
+ },
+ {
+ .name = "Playback_2",
+ .stream_name = "Playback_2",
+ .cpu_dai_name = "DL2",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_playback = 1,
+ },
+ {
+ .name = "Playback_3",
+ .stream_name = "Playback_3",
+ .cpu_dai_name = "DL3",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_playback = 1,
+ },
+ {
+ .name = "Capture_1",
+ .stream_name = "Capture_1",
+ .cpu_dai_name = "UL1",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_capture = 1,
+ },
+ {
+ .name = "Capture_2",
+ .stream_name = "Capture_2",
+ .cpu_dai_name = "UL2",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_capture = 1,
+ },
+ {
+ .name = "Capture_3",
+ .stream_name = "Capture_3",
+ .cpu_dai_name = "UL3",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_capture = 1,
+ },
+ {
+ .name = "Capture_Mono_1",
+ .stream_name = "Capture_Mono_1",
+ .cpu_dai_name = "UL_MONO_1",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_capture = 1,
+ },
+ {
+ .name = "Playback_HDMI",
+ .stream_name = "Playback_HDMI",
+ .cpu_dai_name = "HDMI",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_playback = 1,
+ },
+ /* BE */
+ {
+ .name = "Primary Codec",
+ .cpu_dai_name = "ADDA",
+ .codec_dai_name = "mt6358-snd-codec-aif1",
+ .codec_name = "mt6358-sound",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .dpcm_capture = 1,
+ .ignore_suspend = 1,
+ },
+ {
+ .name = "PCM 1",
+ .cpu_dai_name = "PCM 1",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .dpcm_capture = 1,
+ .ignore_suspend = 1,
+ },
+ {
+ .name = "PCM 2",
+ .cpu_dai_name = "PCM 2",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .dpcm_capture = 1,
+ .ignore_suspend = 1,
+ },
+ {
+ .name = "I2S0",
+ .cpu_dai_name = "I2S0",
+ .codec_dai_name = "bt-sco-pcm",
+ .codec_name = "bt-sco",
+ .no_pcm = 1,
+ .dpcm_capture = 1,
+ .ignore_suspend = 1,
+ .be_hw_params_fixup = mt8183_i2s_hw_params_fixup,
+ .ops = &mt8183_mt6358_i2s_ops,
+ },
+ {
+ .name = "I2S1",
+ .cpu_dai_name = "I2S1",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .codec_name = "snd-soc-dummy",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .ignore_suspend = 1,
+ .be_hw_params_fixup = mt8183_i2s_hw_params_fixup,
+ .ops = &mt8183_mt6358_i2s_ops,
+ },
+ {
+ .name = "I2S2",
+ .cpu_dai_name = "I2S2",
+ .codec_dai_name = "da7219-hifi",
+ .codec_name = "da7219.5-001a",
+ .no_pcm = 1,
+ .dpcm_capture = 1,
+ .ignore_suspend = 1,
+ .be_hw_params_fixup = mt8183_i2s_hw_params_fixup,
+ .ops = &mt8183_da7219_i2s_ops,
+ },
+ {
+ .name = "I2S3",
+ .cpu_dai_name = "I2S3",
+ .codecs = mt8183_da7219_max98357_external_codecs,
+ .num_codecs =
+ ARRAY_SIZE(mt8183_da7219_max98357_external_codecs),
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .ignore_suspend = 1,
+ .be_hw_params_fixup = mt8183_i2s_hw_params_fixup,
+ .ops = &mt8183_da7219_i2s_ops,
+ },
+ {
+ .name = "I2S5",
+ .cpu_dai_name = "I2S5",
+ .codec_dai_name = "bt-sco-pcm",
+ .codec_name = "bt-sco",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .ignore_suspend = 1,
+ .be_hw_params_fixup = mt8183_i2s_hw_params_fixup,
+ .ops = &mt8183_mt6358_i2s_ops,
+ },
+ {
+ .name = "TDM",
+ .cpu_dai_name = "TDM",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .ignore_suspend = 1,
+ },
+};
+
+static int
+mt8183_da7219_max98357_headset_init(struct snd_soc_component *component);
+
+static struct snd_soc_aux_dev mt8183_da7219_max98357_headset_dev = {
+ .name = "Headset Chip",
+ .init = mt8183_da7219_max98357_headset_init,
+};
+
+static struct snd_soc_codec_conf mt6358_codec_conf[] = {
+ {
+ .dev_name = "mt6358-sound",
+ .name_prefix = "Mt6358",
+ },
+};
+
+static struct snd_soc_card mt8183_da7219_max98357_card = {
+ .name = "mt8183_da7219_max98357",
+ .owner = THIS_MODULE,
+ .dai_link = mt8183_da7219_max98357_dai_links,
+ .num_links = ARRAY_SIZE(mt8183_da7219_max98357_dai_links),
+ .aux_dev = &mt8183_da7219_max98357_headset_dev,
+ .num_aux_devs = 1,
+ .codec_conf = mt6358_codec_conf,
+ .num_configs = ARRAY_SIZE(mt6358_codec_conf),
+};
+
+static int
+mt8183_da7219_max98357_headset_init(struct snd_soc_component *component)
+{
+ int ret;
+
+ /* Enable Headset and 4 Buttons Jack detection */
+ ret = snd_soc_card_jack_new(&mt8183_da7219_max98357_card,
+ "Headset Jack",
+ SND_JACK_HEADSET |
+ SND_JACK_BTN_0 | SND_JACK_BTN_1 |
+ SND_JACK_BTN_2 | SND_JACK_BTN_3,
+ &headset_jack,
+ headset_jack_pins,
+ ARRAY_SIZE(headset_jack_pins));
+ if (ret)
+ return ret;
+
+ da7219_aad_jack_det(component, &headset_jack);
+
+ return ret;
+}
+
+static int mt8183_da7219_max98357_dev_probe(struct platform_device *pdev)
+{
+ struct snd_soc_card *card = &mt8183_da7219_max98357_card;
+ struct device_node *platform_node;
+ struct snd_soc_dai_link *dai_link;
+ struct pinctrl *default_pins;
+ int ret, i;
+
+ card->dev = &pdev->dev;
+
+ platform_node = of_parse_phandle(pdev->dev.of_node,
+ "mediatek,platform", 0);
+ if (!platform_node) {
+ dev_err(&pdev->dev, "Property 'platform' missing or invalid\n");
+ return -EINVAL;
+ }
+
+ for_each_card_prelinks(card, i, dai_link) {
+ /* In the alsa soc-core, the "platform" will be
+ * allocated by devm_kzalloc if null.
+ * There is a special case that registerring
+ * sound card is failed at the first time, but
+ * the "platform" will not null when probe is trying
+ * again. It's not expected normally.
+ */
+ dai_link->platforms = NULL;
+
+ if (dai_link->platform_name)
+ continue;
+ dai_link->platform_of_node = platform_node;
+ }
+
+ mt8183_da7219_max98357_headset_dev.codec_of_node =
+ of_parse_phandle(pdev->dev.of_node,
+ "mediatek,headset-codec", 0);
+ if (!mt8183_da7219_max98357_headset_dev.codec_of_node) {
+ dev_err(&pdev->dev,
+ "Property 'mediatek,headset-codec' missing/invalid\n");
+ return -EINVAL;
+ }
+
+ ret = devm_snd_soc_register_card(&pdev->dev, card);
+ if (ret) {
+ dev_err(&pdev->dev, "%s snd_soc_register_card fail %d\n",
+ __func__, ret);
+ return ret;
+ }
+
+ default_pins =
+ devm_pinctrl_get_select(&pdev->dev, PINCTRL_STATE_DEFAULT);
+ if (IS_ERR(default_pins)) {
+ dev_err(&pdev->dev, "%s set pins failed\n",
+ __func__);
+ return PTR_ERR(default_pins);
+ }
+
+ return ret;
+}
+
+#ifdef CONFIG_OF
+static const struct of_device_id mt8183_da7219_max98357_dt_match[] = {
+ {.compatible = "mediatek,mt8183_da7219_max98357",},
+ {}
+};
+#endif
+
+static struct platform_driver mt8183_da7219_max98357_driver = {
+ .driver = {
+ .name = "mt8183_da7219_max98357",
+#ifdef CONFIG_OF
+ .of_match_table = mt8183_da7219_max98357_dt_match,
+#endif
+ },
+ .probe = mt8183_da7219_max98357_dev_probe,
+};
+
+module_platform_driver(mt8183_da7219_max98357_driver);
+
+/* Module information */
+MODULE_DESCRIPTION("MT8183-DA7219-MAX98357 ALSA SoC machine driver");
+MODULE_AUTHOR("Shunli Wang <shunli.wang@mediatek.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("mt8183_da7219_max98357 soc card");
+
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+//
+// mt8183-mt6358.c --
+// MT8183-MT6358-TS3A227-MAX98357 ALSA SoC machine driver
+//
+// Copyright (c) 2018 MediaTek Inc.
+// Author: Shunli Wang <shunli.wang@mediatek.com>
+
+#include <linux/module.h>
+#include <sound/pcm_params.h>
+#include <sound/soc.h>
+#include <sound/jack.h>
+#include <linux/pinctrl/consumer.h>
+
+#include "mt8183-afe-common.h"
+#include "../../codecs/ts3a227e.h"
+
+static struct snd_soc_jack headset_jack;
+
+/* Headset jack detection DAPM pins */
+static struct snd_soc_jack_pin headset_jack_pins[] = {
+ {
+ .pin = "Headphone",
+ .mask = SND_JACK_HEADPHONE,
+ },
+ {
+ .pin = "Headset Mic",
+ .mask = SND_JACK_MICROPHONE,
+ },
+
+};
+
+static int mt8183_mt6358_i2s_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ unsigned int rate = params_rate(params);
+ unsigned int mclk_fs_ratio = 128;
+ unsigned int mclk_fs = rate * mclk_fs_ratio;
+
+ return snd_soc_dai_set_sysclk(rtd->cpu_dai,
+ 0, mclk_fs, SND_SOC_CLOCK_OUT);
+}
+
+static const struct snd_soc_ops mt8183_mt6358_i2s_ops = {
+ .hw_params = mt8183_mt6358_i2s_hw_params,
+};
+
+static int mt8183_i2s_hw_params_fixup(struct snd_soc_pcm_runtime *rtd,
+ struct snd_pcm_hw_params *params)
+{
+ dev_dbg(rtd->dev, "%s(), fix format to 32bit\n", __func__);
+
+ /* fix BE i2s format to 32bit, clean param mask first */
+ snd_mask_reset_range(hw_param_mask(params, SNDRV_PCM_HW_PARAM_FORMAT),
+ 0, SNDRV_PCM_FORMAT_LAST);
+
+ params_set_format(params, SNDRV_PCM_FORMAT_S32_LE);
+ return 0;
+}
+
+static const struct snd_soc_dapm_widget
+mt8183_mt6358_ts3a227_max98357_dapm_widgets[] = {
+ SND_SOC_DAPM_OUTPUT("IT6505_8CH"),
+};
+
+static const struct snd_soc_dapm_route
+mt8183_mt6358_ts3a227_max98357_dapm_routes[] = {
+ {"IT6505_8CH", NULL, "TDM"},
+};
+
+static int
+mt8183_mt6358_ts3a227_max98357_bt_sco_startup(
+ struct snd_pcm_substream *substream)
+{
+ static const unsigned int rates[] = {
+ 8000, 16000
+ };
+ static const struct snd_pcm_hw_constraint_list constraints_rates = {
+ .count = ARRAY_SIZE(rates),
+ .list = rates,
+ .mask = 0,
+ };
+ static const unsigned int channels[] = {
+ 1,
+ };
+ static const struct snd_pcm_hw_constraint_list constraints_channels = {
+ .count = ARRAY_SIZE(channels),
+ .list = channels,
+ .mask = 0,
+ };
+
+ struct snd_pcm_runtime *runtime = substream->runtime;
+
+ snd_pcm_hw_constraint_list(runtime, 0,
+ SNDRV_PCM_HW_PARAM_RATE, &constraints_rates);
+ runtime->hw.channels_max = 1;
+ snd_pcm_hw_constraint_list(runtime, 0,
+ SNDRV_PCM_HW_PARAM_CHANNELS,
+ &constraints_channels);
+
+ runtime->hw.formats = SNDRV_PCM_FMTBIT_S16_LE;
+ snd_pcm_hw_constraint_msbits(runtime, 0, 16, 16);
+
+ return 0;
+}
+
+static const struct snd_soc_ops mt8183_mt6358_ts3a227_max98357_bt_sco_ops = {
+ .startup = mt8183_mt6358_ts3a227_max98357_bt_sco_startup,
+};
+
+static struct snd_soc_dai_link
+mt8183_mt6358_ts3a227_max98357_dai_links[] = {
+ /* FE */
+ {
+ .name = "Playback_1",
+ .stream_name = "Playback_1",
+ .cpu_dai_name = "DL1",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_playback = 1,
+ },
+ {
+ .name = "Playback_2",
+ .stream_name = "Playback_2",
+ .cpu_dai_name = "DL2",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_playback = 1,
+ .ops = &mt8183_mt6358_ts3a227_max98357_bt_sco_ops,
+ },
+ {
+ .name = "Playback_3",
+ .stream_name = "Playback_3",
+ .cpu_dai_name = "DL3",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_playback = 1,
+ },
+ {
+ .name = "Capture_1",
+ .stream_name = "Capture_1",
+ .cpu_dai_name = "UL1",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_capture = 1,
+ .ops = &mt8183_mt6358_ts3a227_max98357_bt_sco_ops,
+ },
+ {
+ .name = "Capture_2",
+ .stream_name = "Capture_2",
+ .cpu_dai_name = "UL2",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_capture = 1,
+ },
+ {
+ .name = "Capture_3",
+ .stream_name = "Capture_3",
+ .cpu_dai_name = "UL3",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_capture = 1,
+ },
+ {
+ .name = "Capture_Mono_1",
+ .stream_name = "Capture_Mono_1",
+ .cpu_dai_name = "UL_MONO_1",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_capture = 1,
+ },
+ {
+ .name = "Playback_HDMI",
+ .stream_name = "Playback_HDMI",
+ .cpu_dai_name = "HDMI",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .trigger = {SND_SOC_DPCM_TRIGGER_PRE,
+ SND_SOC_DPCM_TRIGGER_PRE},
+ .dynamic = 1,
+ .dpcm_playback = 1,
+ },
+ /* BE */
+ {
+ .name = "Primary Codec",
+ .cpu_dai_name = "ADDA",
+ .codec_dai_name = "mt6358-snd-codec-aif1",
+ .codec_name = "mt6358-sound",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .dpcm_capture = 1,
+ .ignore_suspend = 1,
+ },
+ {
+ .name = "PCM 1",
+ .cpu_dai_name = "PCM 1",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .dpcm_capture = 1,
+ .ignore_suspend = 1,
+ },
+ {
+ .name = "PCM 2",
+ .cpu_dai_name = "PCM 2",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .dpcm_capture = 1,
+ .ignore_suspend = 1,
+ },
+ {
+ .name = "I2S0",
+ .cpu_dai_name = "I2S0",
+ .codec_dai_name = "bt-sco-pcm",
+ .codec_name = "bt-sco",
+ .no_pcm = 1,
+ .dpcm_capture = 1,
+ .ignore_suspend = 1,
+ .be_hw_params_fixup = mt8183_i2s_hw_params_fixup,
+ .ops = &mt8183_mt6358_i2s_ops,
+ },
+ {
+ .name = "I2S1",
+ .cpu_dai_name = "I2S1",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .codec_name = "snd-soc-dummy",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .ignore_suspend = 1,
+ .be_hw_params_fixup = mt8183_i2s_hw_params_fixup,
+ .ops = &mt8183_mt6358_i2s_ops,
+ },
+ {
+ .name = "I2S2",
+ .cpu_dai_name = "I2S2",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .codec_name = "snd-soc-dummy",
+ .no_pcm = 1,
+ .dpcm_capture = 1,
+ .ignore_suspend = 1,
+ .be_hw_params_fixup = mt8183_i2s_hw_params_fixup,
+ .ops = &mt8183_mt6358_i2s_ops,
+ },
+ {
+ .name = "I2S3",
+ .cpu_dai_name = "I2S3",
+ .codec_dai_name = "HiFi",
+ .codec_name = "max98357a",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .ignore_suspend = 1,
+ .be_hw_params_fixup = mt8183_i2s_hw_params_fixup,
+ .ops = &mt8183_mt6358_i2s_ops,
+ },
+ {
+ .name = "I2S5",
+ .cpu_dai_name = "I2S5",
+ .codec_dai_name = "bt-sco-pcm",
+ .codec_name = "bt-sco",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .ignore_suspend = 1,
+ .be_hw_params_fixup = mt8183_i2s_hw_params_fixup,
+ .ops = &mt8183_mt6358_i2s_ops,
+ },
+ {
+ .name = "TDM",
+ .cpu_dai_name = "TDM",
+ .codec_name = "snd-soc-dummy",
+ .codec_dai_name = "snd-soc-dummy-dai",
+ .no_pcm = 1,
+ .dpcm_playback = 1,
+ .ignore_suspend = 1,
+ },
+};
+
+static int
+mt8183_mt6358_ts3a227_max98357_headset_init(struct snd_soc_component *cpnt);
+
+static struct snd_soc_aux_dev mt8183_mt6358_ts3a227_max98357_headset_dev = {
+ .name = "Headset Chip",
+ .init = mt8183_mt6358_ts3a227_max98357_headset_init,
+};
+
+static struct snd_soc_card mt8183_mt6358_ts3a227_max98357_card = {
+ .name = "mt8183_mt6358_ts3a227_max98357",
+ .owner = THIS_MODULE,
+ .dai_link = mt8183_mt6358_ts3a227_max98357_dai_links,
+ .num_links = ARRAY_SIZE(mt8183_mt6358_ts3a227_max98357_dai_links),
+ .aux_dev = &mt8183_mt6358_ts3a227_max98357_headset_dev,
+ .num_aux_devs = 1,
+};
+
+static int
+mt8183_mt6358_ts3a227_max98357_headset_init(struct snd_soc_component *component)
+{
+ int ret;
+
+ /* Enable Headset and 4 Buttons Jack detection */
+ ret = snd_soc_card_jack_new(&mt8183_mt6358_ts3a227_max98357_card,
+ "Headset Jack",
+ SND_JACK_HEADSET |
+ SND_JACK_BTN_0 | SND_JACK_BTN_1 |
+ SND_JACK_BTN_2 | SND_JACK_BTN_3,
+ &headset_jack,
+ headset_jack_pins,
+ ARRAY_SIZE(headset_jack_pins));
+ if (ret)
+ return ret;
+
+ ret = ts3a227e_enable_jack_detect(component, &headset_jack);
+
+ return ret;
+}
+
+static int
+mt8183_mt6358_ts3a227_max98357_dev_probe(struct platform_device *pdev)
+{
+ struct snd_soc_card *card = &mt8183_mt6358_ts3a227_max98357_card;
+ struct device_node *platform_node;
+ struct snd_soc_dai_link *dai_link;
+ struct pinctrl *default_pins;
+ int ret, i;
+
+ card->dev = &pdev->dev;
+
+ platform_node = of_parse_phandle(pdev->dev.of_node,
+ "mediatek,platform", 0);
+ if (!platform_node) {
+ dev_err(&pdev->dev, "Property 'platform' missing or invalid\n");
+ return -EINVAL;
+ }
+
+ for_each_card_prelinks(card, i, dai_link) {
+ /* In the alsa soc-core, the "platform" will be
+ * allocated by devm_kzalloc if null.
+ * There is a special case that registerring
+ * sound card is failed at the first time, but
+ * the "platform" will not null when probe is trying
+ * again. It's not expected normally.
+ */
+ dai_link->platforms = NULL;
+
+ if (dai_link->platform_name)
+ continue;
+ dai_link->platform_of_node = platform_node;
+ }
+
+ mt8183_mt6358_ts3a227_max98357_headset_dev.codec_of_node =
+ of_parse_phandle(pdev->dev.of_node,
+ "mediatek,headset-codec", 0);
+ if (!mt8183_mt6358_ts3a227_max98357_headset_dev.codec_of_node) {
+ dev_err(&pdev->dev,
+ "Property 'mediatek,headset-codec' missing/invalid\n");
+ return -EINVAL;
+ }
+
+ ret = devm_snd_soc_register_card(&pdev->dev, card);
+ if (ret)
+ dev_err(&pdev->dev, "%s snd_soc_register_card fail %d\n",
+ __func__, ret);
+
+ default_pins =
+ devm_pinctrl_get_select(&pdev->dev, PINCTRL_STATE_DEFAULT);
+ if (IS_ERR(default_pins)) {
+ dev_err(&pdev->dev, "%s set pins failed\n",
+ __func__);
+ return PTR_ERR(default_pins);
+ }
+
+ return ret;
+}
+
+#ifdef CONFIG_OF
+static const struct of_device_id mt8183_mt6358_ts3a227_max98357_dt_match[] = {
+ {.compatible = "mediatek,mt8183_mt6358_ts3a227_max98357",},
+ {}
+};
+#endif
+
+static struct platform_driver mt8183_mt6358_ts3a227_max98357_driver = {
+ .driver = {
+ .name = "mt8183_mt6358_ts3a227_max98357",
+#ifdef CONFIG_OF
+ .of_match_table = mt8183_mt6358_ts3a227_max98357_dt_match,
+#endif
+ },
+ .probe = mt8183_mt6358_ts3a227_max98357_dev_probe,
+};
+
+module_platform_driver(mt8183_mt6358_ts3a227_max98357_driver);
+
+/* Module information */
+MODULE_DESCRIPTION("MT8183-MT6358-TS3A227-MAX98357 ALSA SoC machine driver");
+MODULE_AUTHOR("Shunli Wang <shunli.wang@mediatek.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("mt8183_mt6358_ts3a227_max98357 soc card");
+
* This file implements the platform operations common to the playback and
* capture frontend DAI. The logic behind this two types of fifo is very
* similar but some difference exist.
- * These differences the respective DAI drivers
+ * These differences are handled in the respective DAI drivers
*/
static struct snd_pcm_hardware axg_fifo_hw = {
return 0;
}
+static int g12a_fifo_pcm_hw_params(struct snd_pcm_substream *ss,
+ struct snd_pcm_hw_params *params)
+{
+ struct axg_fifo *fifo = axg_fifo_data(ss);
+ struct snd_pcm_runtime *runtime = ss->runtime;
+ int ret;
+
+ ret = axg_fifo_pcm_hw_params(ss, params);
+ if (ret)
+ return ret;
+
+ /* Set the initial memory address of the DMA */
+ regmap_write(fifo->map, FIFO_INIT_ADDR, runtime->dma_addr);
+
+ return 0;
+}
+
static int axg_fifo_pcm_hw_free(struct snd_pcm_substream *ss)
{
struct axg_fifo *fifo = axg_fifo_data(ss);
};
EXPORT_SYMBOL_GPL(axg_fifo_pcm_ops);
+const struct snd_pcm_ops g12a_fifo_pcm_ops = {
+ .open = axg_fifo_pcm_open,
+ .close = axg_fifo_pcm_close,
+ .ioctl = snd_pcm_lib_ioctl,
+ .hw_params = g12a_fifo_pcm_hw_params,
+ .hw_free = axg_fifo_pcm_hw_free,
+ .pointer = axg_fifo_pcm_pointer,
+ .trigger = axg_fifo_pcm_trigger,
+};
+EXPORT_SYMBOL_GPL(g12a_fifo_pcm_ops);
+
int axg_fifo_pcm_new(struct snd_soc_pcm_runtime *rtd, unsigned int type)
{
struct snd_card *card = rtd->card->snd_card;
.reg_bits = 32,
.val_bits = 32,
.reg_stride = 4,
- .max_register = FIFO_STATUS2,
+ .max_register = FIFO_INIT_ADDR,
};
int axg_fifo_probe(struct platform_device *pdev)
}
EXPORT_SYMBOL_GPL(axg_fifo_probe);
-MODULE_DESCRIPTION("Amlogic AXG fifo driver");
+MODULE_DESCRIPTION("Amlogic AXG/G12A fifo driver");
MODULE_AUTHOR("Jerome Brunet <jbrunet@baylibre.com>");
MODULE_LICENSE("GPL v2");
#define FIFO_STATUS1 0x14
#define STATUS1_INT_STS(x) ((x) << 0)
#define FIFO_STATUS2 0x18
+#define FIFO_INIT_ADDR 0x24
struct axg_fifo {
struct regmap *map;
};
extern const struct snd_pcm_ops axg_fifo_pcm_ops;
+extern const struct snd_pcm_ops g12a_fifo_pcm_ops;
int axg_fifo_pcm_new(struct snd_soc_pcm_runtime *rtd, unsigned int type);
int axg_fifo_probe(struct platform_device *pdev);
// Copyright (c) 2018 BayLibre, SAS.
// Author: Jerome Brunet <jbrunet@baylibre.com>
-/* This driver implements the frontend playback DAI of AXG based SoCs */
+/*
+ * This driver implements the frontend playback DAI of AXG and G12A based SoCs
+ */
#include <linux/clk.h>
#include <linux/regmap.h>
#include "axg-fifo.h"
-#define CTRL0_FRDDR_PP_MODE BIT(30)
+#define CTRL0_FRDDR_PP_MODE BIT(30)
+#define CTRL0_SEL1_EN_SHIFT 3
+#define CTRL0_SEL2_SHIFT 4
+#define CTRL0_SEL2_EN_SHIFT 7
+#define CTRL0_SEL3_SHIFT 8
+#define CTRL0_SEL3_EN_SHIFT 11
+#define CTRL1_FRDDR_FORCE_FINISH BIT(12)
+
+static int g12a_frddr_dai_prepare(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct axg_fifo *fifo = snd_soc_dai_get_drvdata(dai);
+
+ /* Reset the read pointer to the FIFO_INIT_ADDR */
+ regmap_update_bits(fifo->map, FIFO_CTRL1,
+ CTRL1_FRDDR_FORCE_FINISH, 0);
+ regmap_update_bits(fifo->map, FIFO_CTRL1,
+ CTRL1_FRDDR_FORCE_FINISH, CTRL1_FRDDR_FORCE_FINISH);
+ regmap_update_bits(fifo->map, FIFO_CTRL1,
+ CTRL1_FRDDR_FORCE_FINISH, 0);
+
+ return 0;
+}
static int axg_frddr_dai_startup(struct snd_pcm_substream *substream,
struct snd_soc_dai *dai)
.dai_drv = &axg_frddr_dai_drv
};
+static const struct snd_soc_dai_ops g12a_frddr_ops = {
+ .prepare = g12a_frddr_dai_prepare,
+ .startup = axg_frddr_dai_startup,
+ .shutdown = axg_frddr_dai_shutdown,
+};
+
+static struct snd_soc_dai_driver g12a_frddr_dai_drv = {
+ .name = "FRDDR",
+ .playback = {
+ .stream_name = "Playback",
+ .channels_min = 1,
+ .channels_max = AXG_FIFO_CH_MAX,
+ .rates = AXG_FIFO_RATES,
+ .formats = AXG_FIFO_FORMATS,
+ },
+ .ops = &g12a_frddr_ops,
+ .pcm_new = axg_frddr_pcm_new,
+};
+
+static const char * const g12a_frddr_sel_texts[] = {
+ "OUT 0", "OUT 1", "OUT 2", "OUT 3", "OUT 4",
+};
+
+static SOC_ENUM_SINGLE_DECL(g12a_frddr_sel1_enum, FIFO_CTRL0, CTRL0_SEL_SHIFT,
+ g12a_frddr_sel_texts);
+static SOC_ENUM_SINGLE_DECL(g12a_frddr_sel2_enum, FIFO_CTRL0, CTRL0_SEL2_SHIFT,
+ g12a_frddr_sel_texts);
+static SOC_ENUM_SINGLE_DECL(g12a_frddr_sel3_enum, FIFO_CTRL0, CTRL0_SEL3_SHIFT,
+ g12a_frddr_sel_texts);
+
+static const struct snd_kcontrol_new g12a_frddr_out1_demux =
+ SOC_DAPM_ENUM("Output Src 1", g12a_frddr_sel1_enum);
+static const struct snd_kcontrol_new g12a_frddr_out2_demux =
+ SOC_DAPM_ENUM("Output Src 2", g12a_frddr_sel2_enum);
+static const struct snd_kcontrol_new g12a_frddr_out3_demux =
+ SOC_DAPM_ENUM("Output Src 3", g12a_frddr_sel3_enum);
+
+static const struct snd_kcontrol_new g12a_frddr_out1_enable =
+ SOC_DAPM_SINGLE_AUTODISABLE("Switch", FIFO_CTRL0,
+ CTRL0_SEL1_EN_SHIFT, 1, 0);
+static const struct snd_kcontrol_new g12a_frddr_out2_enable =
+ SOC_DAPM_SINGLE_AUTODISABLE("Switch", FIFO_CTRL0,
+ CTRL0_SEL2_EN_SHIFT, 1, 0);
+static const struct snd_kcontrol_new g12a_frddr_out3_enable =
+ SOC_DAPM_SINGLE_AUTODISABLE("Switch", FIFO_CTRL0,
+ CTRL0_SEL3_EN_SHIFT, 1, 0);
+
+static const struct snd_soc_dapm_widget g12a_frddr_dapm_widgets[] = {
+ SND_SOC_DAPM_AIF_OUT("SRC 1", NULL, 0, SND_SOC_NOPM, 0, 0),
+ SND_SOC_DAPM_AIF_OUT("SRC 2", NULL, 0, SND_SOC_NOPM, 0, 0),
+ SND_SOC_DAPM_AIF_OUT("SRC 3", NULL, 0, SND_SOC_NOPM, 0, 0),
+ SND_SOC_DAPM_SWITCH("SRC 1 EN", SND_SOC_NOPM, 0, 0,
+ &g12a_frddr_out1_enable),
+ SND_SOC_DAPM_SWITCH("SRC 2 EN", SND_SOC_NOPM, 0, 0,
+ &g12a_frddr_out2_enable),
+ SND_SOC_DAPM_SWITCH("SRC 3 EN", SND_SOC_NOPM, 0, 0,
+ &g12a_frddr_out3_enable),
+ SND_SOC_DAPM_DEMUX("SINK 1 SEL", SND_SOC_NOPM, 0, 0,
+ &g12a_frddr_out1_demux),
+ SND_SOC_DAPM_DEMUX("SINK 2 SEL", SND_SOC_NOPM, 0, 0,
+ &g12a_frddr_out2_demux),
+ SND_SOC_DAPM_DEMUX("SINK 3 SEL", SND_SOC_NOPM, 0, 0,
+ &g12a_frddr_out3_demux),
+ SND_SOC_DAPM_AIF_OUT("OUT 0", NULL, 0, SND_SOC_NOPM, 0, 0),
+ SND_SOC_DAPM_AIF_OUT("OUT 1", NULL, 0, SND_SOC_NOPM, 0, 0),
+ SND_SOC_DAPM_AIF_OUT("OUT 2", NULL, 0, SND_SOC_NOPM, 0, 0),
+ SND_SOC_DAPM_AIF_OUT("OUT 3", NULL, 0, SND_SOC_NOPM, 0, 0),
+ SND_SOC_DAPM_AIF_OUT("OUT 4", NULL, 0, SND_SOC_NOPM, 0, 0),
+};
+
+static const struct snd_soc_dapm_route g12a_frddr_dapm_routes[] = {
+ { "SRC 1", NULL, "Playback" },
+ { "SRC 2", NULL, "Playback" },
+ { "SRC 3", NULL, "Playback" },
+ { "SRC 1 EN", "Switch", "SRC 1" },
+ { "SRC 2 EN", "Switch", "SRC 2" },
+ { "SRC 3 EN", "Switch", "SRC 3" },
+ { "SINK 1 SEL", NULL, "SRC 1 EN" },
+ { "SINK 2 SEL", NULL, "SRC 2 EN" },
+ { "SINK 3 SEL", NULL, "SRC 3 EN" },
+ { "OUT 0", "OUT 0", "SINK 1 SEL" },
+ { "OUT 1", "OUT 1", "SINK 1 SEL" },
+ { "OUT 2", "OUT 2", "SINK 1 SEL" },
+ { "OUT 3", "OUT 3", "SINK 1 SEL" },
+ { "OUT 4", "OUT 4", "SINK 1 SEL" },
+ { "OUT 0", "OUT 0", "SINK 2 SEL" },
+ { "OUT 1", "OUT 1", "SINK 2 SEL" },
+ { "OUT 2", "OUT 2", "SINK 2 SEL" },
+ { "OUT 3", "OUT 3", "SINK 2 SEL" },
+ { "OUT 4", "OUT 4", "SINK 2 SEL" },
+ { "OUT 0", "OUT 0", "SINK 3 SEL" },
+ { "OUT 1", "OUT 1", "SINK 3 SEL" },
+ { "OUT 2", "OUT 2", "SINK 3 SEL" },
+ { "OUT 3", "OUT 3", "SINK 3 SEL" },
+ { "OUT 4", "OUT 4", "SINK 3 SEL" },
+};
+
+static const struct snd_soc_component_driver g12a_frddr_component_drv = {
+ .dapm_widgets = g12a_frddr_dapm_widgets,
+ .num_dapm_widgets = ARRAY_SIZE(g12a_frddr_dapm_widgets),
+ .dapm_routes = g12a_frddr_dapm_routes,
+ .num_dapm_routes = ARRAY_SIZE(g12a_frddr_dapm_routes),
+ .ops = &g12a_fifo_pcm_ops
+};
+
+static const struct axg_fifo_match_data g12a_frddr_match_data = {
+ .component_drv = &g12a_frddr_component_drv,
+ .dai_drv = &g12a_frddr_dai_drv
+};
+
static const struct of_device_id axg_frddr_of_match[] = {
{
.compatible = "amlogic,axg-frddr",
.data = &axg_frddr_match_data,
+ }, {
+ .compatible = "amlogic,g12a-frddr",
+ .data = &g12a_frddr_match_data,
}, {}
};
MODULE_DEVICE_TABLE(of, axg_frddr_of_match);
};
module_platform_driver(axg_frddr_pdrv);
-MODULE_DESCRIPTION("Amlogic AXG playback fifo driver");
+MODULE_DESCRIPTION("Amlogic AXG/G12A playback fifo driver");
MODULE_AUTHOR("Jerome Brunet <jbrunet@baylibre.com>");
MODULE_LICENSE("GPL v2");
static int axg_tdm_formatter_enable(struct axg_tdm_formatter *formatter)
{
struct axg_tdm_stream *ts = formatter->stream;
- bool invert = formatter->drv->invert_sclk;
+ bool invert = formatter->drv->quirks->invert_sclk;
int ret;
/* Do nothing if the formatter is already enabled */
return ret;
/* Setup the stream parameter in the formatter */
- ret = formatter->drv->ops->prepare(formatter->map, formatter->stream);
+ ret = formatter->drv->ops->prepare(formatter->map,
+ formatter->drv->quirks,
+ formatter->stream);
if (ret)
return ret;
struct snd_soc_dapm_widget;
struct snd_kcontrol;
+struct axg_tdm_formatter_hw {
+ unsigned int skew_offset;
+ bool invert_sclk;
+};
+
struct axg_tdm_formatter_ops {
struct axg_tdm_stream *(*get_stream)(struct snd_soc_dapm_widget *w);
void (*enable)(struct regmap *map);
void (*disable)(struct regmap *map);
- int (*prepare)(struct regmap *map, struct axg_tdm_stream *ts);
+ int (*prepare)(struct regmap *map,
+ const struct axg_tdm_formatter_hw *quirks,
+ struct axg_tdm_stream *ts);
};
struct axg_tdm_formatter_driver {
const struct snd_soc_component_driver *component_drv;
const struct regmap_config *regmap_cfg;
const struct axg_tdm_formatter_ops *ops;
- bool invert_sclk;
+ const struct axg_tdm_formatter_hw *quirks;
};
int axg_tdm_formatter_set_channel_masks(struct regmap *map,
regmap_update_bits(map, TDMIN_CTRL, TDMIN_CTRL_ENABLE, 0);
}
-static int axg_tdmin_prepare(struct regmap *map, struct axg_tdm_stream *ts)
+static int axg_tdmin_prepare(struct regmap *map,
+ const struct axg_tdm_formatter_hw *quirks,
+ struct axg_tdm_stream *ts)
{
- unsigned int val = 0;
+ unsigned int val, skew = quirks->skew_offset;
/* Set stream skew */
switch (ts->iface->fmt & SND_SOC_DAIFMT_FORMAT_MASK) {
case SND_SOC_DAIFMT_I2S:
case SND_SOC_DAIFMT_DSP_A:
- val |= TDMIN_CTRL_IN_BIT_SKEW(3);
+ skew += 1;
break;
case SND_SOC_DAIFMT_LEFT_J:
case SND_SOC_DAIFMT_RIGHT_J:
case SND_SOC_DAIFMT_DSP_B:
- val = TDMIN_CTRL_IN_BIT_SKEW(2);
break;
default:
return -EINVAL;
}
+ val = TDMIN_CTRL_IN_BIT_SKEW(skew);
+
/* Set stream format mode */
switch (ts->iface->fmt & SND_SOC_DAIFMT_FORMAT_MASK) {
case SND_SOC_DAIFMT_I2S:
.component_drv = &axg_tdmin_component_drv,
.regmap_cfg = &axg_tdmin_regmap_cfg,
.ops = &axg_tdmin_ops,
- .invert_sclk = false,
+ .quirks = &(const struct axg_tdm_formatter_hw) {
+ .invert_sclk = false,
+ .skew_offset = 2,
+ },
};
static const struct of_device_id axg_tdmin_of_match[] = {
regmap_update_bits(map, TDMOUT_CTRL0, TDMOUT_CTRL0_ENABLE, 0);
}
-static int axg_tdmout_prepare(struct regmap *map, struct axg_tdm_stream *ts)
+static int axg_tdmout_prepare(struct regmap *map,
+ const struct axg_tdm_formatter_hw *quirks,
+ struct axg_tdm_stream *ts)
{
- unsigned int val = 0;
+ unsigned int val, skew = quirks->skew_offset;
/* Set the stream skew */
switch (ts->iface->fmt & SND_SOC_DAIFMT_FORMAT_MASK) {
case SND_SOC_DAIFMT_I2S:
case SND_SOC_DAIFMT_DSP_A:
- val |= TDMOUT_CTRL0_INIT_BITNUM(1);
break;
case SND_SOC_DAIFMT_LEFT_J:
case SND_SOC_DAIFMT_RIGHT_J:
case SND_SOC_DAIFMT_DSP_B:
- val |= TDMOUT_CTRL0_INIT_BITNUM(2);
+ skew += 1;
break;
default:
return -EINVAL;
}
+ val = TDMOUT_CTRL0_INIT_BITNUM(skew);
+
/* Set the slot width */
val |= TDMOUT_CTRL0_BITNUM(ts->iface->slot_width - 1);
.component_drv = &axg_tdmout_component_drv,
.regmap_cfg = &axg_tdmout_regmap_cfg,
.ops = &axg_tdmout_ops,
- .invert_sclk = true,
+ .quirks = &(const struct axg_tdm_formatter_hw) {
+ .invert_sclk = true,
+ .skew_offset = 1,
+ },
+};
+
+static const struct axg_tdm_formatter_driver g12a_tdmout_drv = {
+ .component_drv = &axg_tdmout_component_drv,
+ .regmap_cfg = &axg_tdmout_regmap_cfg,
+ .ops = &axg_tdmout_ops,
+ .quirks = &(const struct axg_tdm_formatter_hw) {
+ .invert_sclk = true,
+ .skew_offset = 2,
+ },
};
static const struct of_device_id axg_tdmout_of_match[] = {
{
.compatible = "amlogic,axg-tdmout",
.data = &axg_tdmout_drv,
+ }, {
+ .compatible = "amlogic,g12a-tdmout",
+ .data = &g12a_tdmout_drv,
}, {}
};
MODULE_DEVICE_TABLE(of, axg_tdmout_of_match);
#define CTRL0_TODDR_MSB_POS(x) ((x) << 8)
#define CTRL0_TODDR_LSB_POS_MASK GENMASK(7, 3)
#define CTRL0_TODDR_LSB_POS(x) ((x) << 3)
+#define CTRL1_TODDR_FORCE_FINISH BIT(25)
#define TODDR_MSB_POS 31
return axg_fifo_pcm_new(rtd, SNDRV_PCM_STREAM_CAPTURE);
}
+static int g12a_toddr_dai_prepare(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct axg_fifo *fifo = snd_soc_dai_get_drvdata(dai);
+
+ /* Reset the write pointer to the FIFO_INIT_ADDR */
+ regmap_update_bits(fifo->map, FIFO_CTRL1,
+ CTRL1_TODDR_FORCE_FINISH, 0);
+ regmap_update_bits(fifo->map, FIFO_CTRL1,
+ CTRL1_TODDR_FORCE_FINISH, CTRL1_TODDR_FORCE_FINISH);
+ regmap_update_bits(fifo->map, FIFO_CTRL1,
+ CTRL1_TODDR_FORCE_FINISH, 0);
+
+ return 0;
+}
+
static int axg_toddr_dai_hw_params(struct snd_pcm_substream *substream,
struct snd_pcm_hw_params *params,
struct snd_soc_dai *dai)
.dai_drv = &axg_toddr_dai_drv
};
+static const struct snd_soc_dai_ops g12a_toddr_ops = {
+ .prepare = g12a_toddr_dai_prepare,
+ .hw_params = axg_toddr_dai_hw_params,
+ .startup = axg_toddr_dai_startup,
+ .shutdown = axg_toddr_dai_shutdown,
+};
+
+static struct snd_soc_dai_driver g12a_toddr_dai_drv = {
+ .name = "TODDR",
+ .capture = {
+ .stream_name = "Capture",
+ .channels_min = 1,
+ .channels_max = AXG_FIFO_CH_MAX,
+ .rates = AXG_FIFO_RATES,
+ .formats = AXG_FIFO_FORMATS,
+ },
+ .ops = &g12a_toddr_ops,
+ .pcm_new = axg_toddr_pcm_new,
+};
+
+static const struct snd_soc_component_driver g12a_toddr_component_drv = {
+ .dapm_widgets = axg_toddr_dapm_widgets,
+ .num_dapm_widgets = ARRAY_SIZE(axg_toddr_dapm_widgets),
+ .dapm_routes = axg_toddr_dapm_routes,
+ .num_dapm_routes = ARRAY_SIZE(axg_toddr_dapm_routes),
+ .ops = &g12a_fifo_pcm_ops
+};
+
+static const struct axg_fifo_match_data g12a_toddr_match_data = {
+ .component_drv = &g12a_toddr_component_drv,
+ .dai_drv = &g12a_toddr_dai_drv
+};
+
static const struct of_device_id axg_toddr_of_match[] = {
{
.compatible = "amlogic,axg-toddr",
.data = &axg_toddr_match_data,
+ }, {
+ .compatible = "amlogic,g12a-toddr",
+ .data = &g12a_toddr_match_data,
}, {}
};
MODULE_DEVICE_TABLE(of, axg_toddr_of_match);
config SND_SOC_SDM845
tristate "SoC Machine driver for SDM845 boards"
- depends on QCOM_APR && MFD_CROS_EC
+ depends on QCOM_APR && MFD_CROS_EC && I2C
select SND_SOC_QDSP6
select SND_SOC_QCOM_COMMON
select SND_SOC_RT5663
#include <linux/module.h>
#include <linux/clk.h>
#include <linux/of.h>
+#include <linux/of_device.h>
#include <linux/pm_runtime.h>
+#include <linux/rational.h>
#include <linux/regmap.h>
+#include <linux/reset.h>
#include <sound/dmaengine_pcm.h>
#include <sound/pcm_params.h>
#include "rockchip_pdm.h"
#define PDM_DMA_BURST_SIZE (8) /* size * width: 8*4 = 32 bytes */
+#define PDM_SIGNOFF_CLK_RATE (100000000)
+
+enum rk_pdm_version {
+ RK_PDM_RK3229,
+ RK_PDM_RK3308,
+};
struct rk_pdm_dev {
struct device *dev;
struct clk *hclk;
struct regmap *regmap;
struct snd_dmaengine_dai_dma_data capture_dma_data;
+ struct reset_control *reset;
+ enum rk_pdm_version version;
};
struct rk_pdm_clkref {
unsigned int sr;
unsigned int clk;
+ unsigned int clk_out;
+};
+
+struct rk_pdm_ds_ratio {
+ unsigned int ratio;
+ unsigned int sr;
};
static struct rk_pdm_clkref clkref[] = {
- { 8000, 40960000 },
- { 11025, 56448000 },
- { 12000, 61440000 },
+ { 8000, 40960000, 2048000 },
+ { 11025, 56448000, 2822400 },
+ { 12000, 61440000, 3072000 },
+ { 8000, 98304000, 2048000 },
+ { 12000, 98304000, 3072000 },
+};
+
+static struct rk_pdm_ds_ratio ds_ratio[] = {
+ { 0, 192000 },
+ { 0, 176400 },
+ { 0, 128000 },
+ { 1, 96000 },
+ { 1, 88200 },
+ { 1, 64000 },
+ { 2, 48000 },
+ { 2, 44100 },
+ { 2, 32000 },
+ { 3, 24000 },
+ { 3, 22050 },
+ { 3, 16000 },
+ { 4, 12000 },
+ { 4, 11025 },
+ { 4, 8000 },
};
-static unsigned int get_pdm_clk(unsigned int sr)
+static unsigned int get_pdm_clk(struct rk_pdm_dev *pdm, unsigned int sr,
+ unsigned int *clk_src, unsigned int *clk_out)
{
- unsigned int i, count, clk, div;
+ unsigned int i, count, clk, div, rate;
clk = 0;
if (!sr)
continue;
div = sr / clkref[i].sr;
if ((div & (div - 1)) == 0) {
+ *clk_out = clkref[i].clk_out;
+ rate = clk_round_rate(pdm->clk, clkref[i].clk);
+ if (rate != clkref[i].clk)
+ continue;
clk = clkref[i].clk;
+ *clk_src = clkref[i].clk;
break;
}
}
+ if (!clk) {
+ clk = clk_round_rate(pdm->clk, PDM_SIGNOFF_CLK_RATE);
+ *clk_src = clk;
+ }
return clk;
}
+static unsigned int get_pdm_ds_ratio(unsigned int sr)
+{
+ unsigned int i, count, ratio;
+
+ ratio = 0;
+ if (!sr)
+ return ratio;
+
+ count = ARRAY_SIZE(ds_ratio);
+ for (i = 0; i < count; i++) {
+ if (sr == ds_ratio[i].sr)
+ ratio = ds_ratio[i].ratio;
+ }
+ return ratio;
+}
+
static inline struct rk_pdm_dev *to_info(struct snd_soc_dai *dai)
{
return snd_soc_dai_get_drvdata(dai);
struct rk_pdm_dev *pdm = to_info(dai);
unsigned int val = 0;
unsigned int clk_rate, clk_div, samplerate;
+ unsigned int clk_src, clk_out = 0;
+ unsigned long m, n;
+ bool change;
int ret;
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ return 0;
+
samplerate = params_rate(params);
- clk_rate = get_pdm_clk(samplerate);
+ clk_rate = get_pdm_clk(pdm, samplerate, &clk_src, &clk_out);
if (!clk_rate)
return -EINVAL;
- ret = clk_set_rate(pdm->clk, clk_rate);
+ ret = clk_set_rate(pdm->clk, clk_src);
if (ret)
return -EINVAL;
- clk_div = DIV_ROUND_CLOSEST(clk_rate, samplerate);
-
- switch (clk_div) {
- case 320:
- val = PDM_CLK_320FS;
- break;
- case 640:
- val = PDM_CLK_640FS;
- break;
- case 1280:
- val = PDM_CLK_1280FS;
- break;
- case 2560:
- val = PDM_CLK_2560FS;
- break;
- case 5120:
- val = PDM_CLK_5120FS;
- break;
- default:
- dev_err(pdm->dev, "unsupported div: %d\n", clk_div);
- return -EINVAL;
+ if (pdm->version == RK_PDM_RK3308) {
+ rational_best_approximation(clk_out, clk_src,
+ GENMASK(16 - 1, 0),
+ GENMASK(16 - 1, 0),
+ &m, &n);
+
+ val = (m << PDM_FD_NUMERATOR_SFT) |
+ (n << PDM_FD_DENOMINATOR_SFT);
+ regmap_update_bits_check(pdm->regmap, PDM_CTRL1,
+ PDM_FD_NUMERATOR_MSK |
+ PDM_FD_DENOMINATOR_MSK,
+ val, &change);
+ if (change) {
+ reset_control_assert(pdm->reset);
+ reset_control_deassert(pdm->reset);
+ rockchip_pdm_rxctrl(pdm, 0);
+ }
+ clk_div = n / m;
+ if (clk_div >= 40)
+ val = PDM_CLK_FD_RATIO_40;
+ else if (clk_div <= 35)
+ val = PDM_CLK_FD_RATIO_35;
+ else
+ return -EINVAL;
+ regmap_update_bits(pdm->regmap, PDM_CLK_CTRL,
+ PDM_CLK_FD_RATIO_MSK,
+ val);
}
-
+ val = get_pdm_ds_ratio(samplerate);
regmap_update_bits(pdm->regmap, PDM_CLK_CTRL, PDM_DS_RATIO_MSK, val);
regmap_update_bits(pdm->regmap, PDM_HPF_CTRL,
PDM_HPF_CF_MSK, PDM_HPF_60HZ);
regmap_update_bits(pdm->regmap, PDM_HPF_CTRL,
PDM_HPF_LE | PDM_HPF_RE, PDM_HPF_LE | PDM_HPF_RE);
regmap_update_bits(pdm->regmap, PDM_CLK_CTRL, PDM_CLK_EN, PDM_CLK_EN);
+ if (pdm->version != RK_PDM_RK3229)
+ regmap_update_bits(pdm->regmap, PDM_CTRL0,
+ PDM_MODE_MSK, PDM_MODE_LJ);
val = 0;
switch (params_format(params)) {
return -EINVAL;
}
- if (substream->stream == SNDRV_PCM_STREAM_CAPTURE) {
- regmap_update_bits(pdm->regmap, PDM_CTRL0,
- PDM_PATH_MSK | PDM_VDW_MSK,
- val);
- regmap_update_bits(pdm->regmap, PDM_DMA_CTRL, PDM_DMA_RDL_MSK,
- PDM_DMA_RDL(16));
- regmap_update_bits(pdm->regmap, PDM_SYSCONFIG,
- PDM_RX_MASK | PDM_RX_CLR_MASK,
- PDM_RX_STOP | PDM_RX_CLR_WR);
- }
+ regmap_update_bits(pdm->regmap, PDM_CTRL0,
+ PDM_PATH_MSK | PDM_VDW_MSK,
+ val);
+ /* all channels share the single FIFO */
+ regmap_update_bits(pdm->regmap, PDM_DMA_CTRL, PDM_DMA_RDL_MSK,
+ PDM_DMA_RDL(8 * params_channels(params)));
return 0;
}
case PDM_INT_CLR:
case PDM_INT_ST:
case PDM_DATA_VALID:
+ case PDM_RXFIFO_DATA:
case PDM_VERSION:
return true;
default:
{
switch (reg) {
case PDM_SYSCONFIG:
+ case PDM_FIFO_CTRL:
case PDM_INT_CLR:
case PDM_INT_ST:
+ case PDM_RXFIFO_DATA:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static bool rockchip_pdm_precious_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case PDM_RXFIFO_DATA:
return true;
default:
return false;
}
}
+static const struct reg_default rockchip_pdm_reg_defaults[] = {
+ {0x04, 0x78000017},
+ {0x08, 0x0bb8ea60},
+ {0x18, 0x0000001f},
+};
+
static const struct regmap_config rockchip_pdm_regmap_config = {
.reg_bits = 32,
.reg_stride = 4,
.val_bits = 32,
.max_register = PDM_VERSION,
+ .reg_defaults = rockchip_pdm_reg_defaults,
+ .num_reg_defaults = ARRAY_SIZE(rockchip_pdm_reg_defaults),
.writeable_reg = rockchip_pdm_wr_reg,
.readable_reg = rockchip_pdm_rd_reg,
.volatile_reg = rockchip_pdm_volatile_reg,
+ .precious_reg = rockchip_pdm_precious_reg,
.cache_type = REGCACHE_FLAT,
};
+static const struct of_device_id rockchip_pdm_match[] = {
+ { .compatible = "rockchip,pdm",
+ .data = (void *)RK_PDM_RK3229 },
+ { .compatible = "rockchip,px30-pdm",
+ .data = (void *)RK_PDM_RK3308 },
+ { .compatible = "rockchip,rk1808-pdm",
+ .data = (void *)RK_PDM_RK3308 },
+ { .compatible = "rockchip,rk3308-pdm",
+ .data = (void *)RK_PDM_RK3308 },
+ {},
+};
+MODULE_DEVICE_TABLE(of, rockchip_pdm_match);
+
static int rockchip_pdm_probe(struct platform_device *pdev)
{
+ const struct of_device_id *match;
struct rk_pdm_dev *pdm;
struct resource *res;
void __iomem *regs;
if (!pdm)
return -ENOMEM;
+ match = of_match_device(rockchip_pdm_match, &pdev->dev);
+ if (match)
+ pdm->version = (enum rk_pdm_version)match->data;
+
+ if (pdm->version == RK_PDM_RK3308) {
+ pdm->reset = devm_reset_control_get(&pdev->dev, "pdm-m");
+ if (IS_ERR(pdm->reset))
+ return PTR_ERR(pdm->reset);
+ }
+
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
regs = devm_ioremap_resource(&pdev->dev, res);
if (IS_ERR(regs))
goto err_suspend;
}
+ rockchip_pdm_rxctrl(pdm, 0);
ret = devm_snd_dmaengine_pcm_register(&pdev->dev, NULL, 0);
if (ret) {
dev_err(&pdev->dev, "could not register pcm: %d\n", ret);
SET_SYSTEM_SLEEP_PM_OPS(rockchip_pdm_suspend, rockchip_pdm_resume)
};
-static const struct of_device_id rockchip_pdm_match[] = {
- { .compatible = "rockchip,pdm", },
- {},
-};
-MODULE_DEVICE_TABLE(of, rockchip_pdm_match);
-
static struct platform_driver rockchip_pdm_driver = {
.probe = rockchip_pdm_probe,
.remove = rockchip_pdm_remove,
/* PDM CTRL0 */
#define PDM_PATH_MSK (0xf << 27)
+#define PDM_MODE_MSK BIT(31)
+#define PDM_MODE_RJ 0
+#define PDM_MODE_LJ BIT(31)
#define PDM_PATH3_EN BIT(30)
#define PDM_PATH2_EN BIT(29)
#define PDM_PATH1_EN BIT(28)
#define PDM_VDW_MSK (0x1f << 0)
#define PDM_VDW(X) ((X - 1) << 0)
+/* PDM CTRL1 */
+#define PDM_FD_NUMERATOR_SFT 16
+#define PDM_FD_NUMERATOR_MSK GENMASK(31, 16)
+#define PDM_FD_DENOMINATOR_SFT 0
+#define PDM_FD_DENOMINATOR_MSK GENMASK(15, 0)
+
/* PDM CLK CTRL */
+#define PDM_CLK_FD_RATIO_MSK BIT(6)
+#define PDM_CLK_FD_RATIO_40 (0X0 << 6)
+#define PDM_CLK_FD_RATIO_35 BIT(6)
#define PDM_CLK_MSK BIT(5)
#define PDM_CLK_EN BIT(5)
#define PDM_CLK_DIS (0x0 << 5)
-/*
- * arndale_rt5631.c
- *
- * Copyright (c) 2014, Insignal Co., Ltd.
- *
- * Author: Claude <claude@insginal.co.kr>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Copyright (c) 2014, Insignal Co., Ltd.
+//
+// Author: Claude <claude@insginal.co.kr>
#include <linux/module.h>
#include <linux/platform_device.h>
-/*
- * Bells audio support
- *
- * Copyright 2012 Wolfson Microelectronics
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Bells audio support
+//
+// Copyright 2012 Wolfson Microelectronics
#include <sound/soc.h>
#include <sound/soc-dapm.h>
+/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- * ALSA PCM interface for the Samsung SoC
+ * ALSA PCM interface for the Samsung SoC
*/
#ifndef _SAMSUNG_DMA_H
-/*
- * dmaengine.c - Samsung dmaengine wrapper
- *
- * Author: Mark Brown <broonie@linaro.org>
- * Copyright 2013 Linaro
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// dmaengine.c - Samsung dmaengine wrapper
+//
+// Author: Mark Brown <broonie@linaro.org>
+// Copyright 2013 Linaro
#include <linux/module.h>
#include <sound/core.h>
-/*
- * h1940-uda1380.c -- ALSA Soc Audio Layer
- *
- * Copyright (c) 2010 Arnaud Patard <arnaud.patard@rtp-net.org>
- * Copyright (c) 2010 Vasily Khoruzhick <anarsoul@gmail.com>
- *
- * Based on version from Arnaud Patard <arnaud.patard@rtp-net.org>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// h1940_uda1380.c - ALSA SoC Audio Layer
+//
+// Copyright (c) 2010 Arnaud Patard <arnaud.patard@rtp-net.org>
+// Copyright (c) 2010 Vasily Khoruzhick <anarsoul@gmail.com>
+//
+// Based on version from Arnaud Patard <arnaud.patard@rtp-net.org>
#include <linux/types.h>
#include <linux/gpio.h>
+/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * linux/sound/soc/samsung/i2s-regs.h
- *
* Copyright (c) 2011 Samsung Electronics Co., Ltd.
* http://www.samsung.com
*
* Samsung I2S driver's register header
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
*/
#ifndef __SND_SOC_SAMSUNG_I2S_REGS_H
struct platform_device *pdev;
struct platform_device *pdev_sec;
- /* Memory mapped SFR region */
- void __iomem *addr;
-
- /* Spinlock protecting access to the device's registers */
- spinlock_t lock;
-
/* Lock for cross interface checks */
spinlock_t pcm_lock;
/* The clock provider's data */
struct clk *clk_table[3];
struct clk_onecell_data clk_data;
+
+ /* Spinlock protecting member fields below */
+ spinlock_t lock;
+
+ /* Memory mapped SFR region */
+ void __iomem *addr;
+
+ /* A flag indicating the I2S slave mode operation */
+ bool slave_mode;
};
/* Returns true if this is the 'overlay' stereo DAI */
return i2s->drv->id == SAMSUNG_I2S_ID_SECONDARY;
}
-/* If operating in SoC-Slave mode */
-static inline bool is_slave(struct i2s_dai *i2s)
-{
- struct samsung_i2s_priv *priv = i2s->priv;
-
- u32 mod = readl(priv->addr + I2SMOD);
- return (mod & (1 << priv->variant_regs->mss_off)) ? true : false;
-}
-
/* If this interface of the controller is transmitting data */
static inline bool tx_active(struct i2s_dai *i2s)
{
mod &= ~(sdf_mask | lrp_rlow | mod_slave);
mod |= tmp;
writel(mod, priv->addr + I2SMOD);
+ priv->slave_mode = (mod & mod_slave);
spin_unlock_irqrestore(&priv->lock, flags);
pm_runtime_put(dai->dev);
set_rfs(i2s, rfs);
/* Don't bother with PSR in Slave mode */
- if (is_slave(i2s))
+ if (priv->slave_mode)
return 0;
if (!(priv->quirks & QUIRK_NO_MUXPSR)) {
-/* sound/soc/samsung/i2s.h
- *
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
* ALSA SoC Audio Layer - Samsung I2S Controller driver
*
* Copyright (c) 2010 Samsung Electronics Co. Ltd.
* Jaswinder Singh <jassisinghbrar@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#ifndef __SND_SOC_SAMSUNG_I2S_H
-/*
- * sound/soc/samsung/idma.c
- *
- * Copyright (c) 2011 Samsung Electronics Co., Ltd.
- * http://www.samsung.com
- *
- * I2S0's Internal DMA driver
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// idma.c - I2S0 internal DMA driver
+//
+// Copyright (c) 2011 Samsung Electronics Co., Ltd.
+// http://www.samsung.com
+
#include <linux/interrupt.h>
#include <linux/platform_device.h>
#include <linux/dma-mapping.h>
+/* SPDX-License-Identifier: GPL-2.0 */
/*
- * sound/soc/samsung/idma.h
- *
* Copyright (c) 2011 Samsung Electronics Co., Ltd
* http://www.samsung.com
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
*/
#ifndef __SND_SOC_SAMSUNG_IDMA_H_
-/* sound/soc/samsung/jive_wm8750.c
- *
- * Copyright 2007,2008 Simtec Electronics
- *
- * Based on sound/soc/pxa/spitz.c
- * Copyright 2005 Wolfson Microelectronics PLC.
- * Copyright 2005 Openedhand Ltd.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright 2007,2008 Simtec Electronics
+//
+// Based on sound/soc/pxa/spitz.c
+// Copyright 2005 Wolfson Microelectronics PLC.
+// Copyright 2005 Openedhand Ltd.
#include <linux/module.h>
#include <sound/soc.h>
-/*
- * Littlemill audio support
- *
- * Copyright 2011 Wolfson Microelectronics
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Littlemill audio support
+//
+// Copyright 2011 Wolfson Microelectronics
#include <sound/soc.h>
#include <sound/soc-dapm.h>
-/*
- * Lowland audio support
- *
- * Copyright 2011 Wolfson Microelectronics
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Lowland audio support
+//
+// Copyright 2011 Wolfson Microelectronics
#include <sound/soc.h>
#include <sound/soc-dapm.h>
-/*
- * neo1973_wm8753.c -- SoC audio for Openmoko Neo1973 and Freerunner devices
- *
- * Copyright 2007 Openmoko Inc
- * Author: Graeme Gregory <graeme@openmoko.org>
- * Copyright 2007 Wolfson Microelectronics PLC.
- * Author: Graeme Gregory
- * graeme.gregory@wolfsonmicro.com or linux@wolfsonmicro.com
- * Copyright 2009 Wolfson Microelectronics
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// neo1973_wm8753.c - SoC audio for Openmoko Neo1973 and Freerunner devices
+//
+// Copyright 2007 Openmoko Inc
+// Author: Graeme Gregory <graeme@openmoko.org>
+// Copyright 2007 Wolfson Microelectronics PLC.
+// Author: Graeme Gregory
+// graeme.gregory@wolfsonmicro.com or linux@wolfsonmicro.com
+// Copyright 2009 Wolfson Microelectronics
#include <linux/module.h>
#include <linux/platform_device.h>
-/*
- * Copyright (C) 2017 Samsung Electronics Co., Ltd.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (C) 2017 Samsung Electronics Co., Ltd.
#include <linux/clk.h>
#include <linux/clk-provider.h>
-/* sound/soc/samsung/pcm.c
- *
- * ALSA SoC Audio Layer - S3C PCM-Controller driver
- *
- * Copyright (c) 2009 Samsung Electronics Co. Ltd
- * Author: Jaswinder Singh <jassisinghbrar@gmail.com>
- * based upon I2S drivers by Ben Dooks.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// ALSA SoC Audio Layer - S3C PCM-Controller driver
+//
+// Copyright (c) 2009 Samsung Electronics Co. Ltd
+// Author: Jaswinder Singh <jassisinghbrar@gmail.com>
+// based upon I2S drivers by Ben Dooks.
#include <linux/clk.h>
#include <linux/io.h>
-/* sound/soc/samsung/pcm.h
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- */
+/* SPDX-License-Identifier: GPL-2.0 */
#ifndef __S3C_PCM_H
#define __S3C_PCM_H __FILE__
-/* linux/include/asm-arm/plat-s3c24xx/regs-s3c2412-iis.h
- *
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
* Copyright 2007 Simtec Electronics <linux@simtec.co.uk>
* http://armlinux.simtec.co.uk/
*
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
* S3C2412 IIS register definition
-*/
+ */
#ifndef __ASM_ARCH_REGS_S3C2412_IIS_H
#define __ASM_ARCH_REGS_S3C2412_IIS_H
+/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (c) 2003 Simtec Electronics <linux@simtec.co.uk>
* http://www.simtec.co.uk/products/SWLINUX/
*
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
* S3C2410 IIS register definition
-*/
+ */
#ifndef __SAMSUNG_REGS_IIS_H__
#define __SAMSUNG_REGS_IIS_H__
-/*
- * rx1950.c -- ALSA Soc Audio Layer
- *
- * Copyright (c) 2010 Vasily Khoruzhick <anarsoul@gmail.com>
- *
- * Based on smdk2440.c and magician.c
- *
- * Authors: Graeme Gregory graeme.gregory@wolfsonmicro.com
- * Philipp Zabel <philipp.zabel@gmail.com>
- * Denis Grigoriev <dgreenday@gmail.com>
- * Vasily Khoruzhick <anarsoul@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// rx1950.c - ALSA SoC Audio Layer
+//
+// Copyright (c) 2010 Vasily Khoruzhick <anarsoul@gmail.com>
+//
+// Based on smdk2440.c and magician.c
+//
+// Authors: Graeme Gregory graeme.gregory@wolfsonmicro.com
+// Philipp Zabel <philipp.zabel@gmail.com>
+// Denis Grigoriev <dgreenday@gmail.com>
+// Vasily Khoruzhick <anarsoul@gmail.com>
#include <linux/types.h>
#include <linux/gpio.h>
-/* ALSA Soc Audio Layer - I2S core for newer Samsung SoCs.
- *
- * Copyright (c) 2006 Wolfson Microelectronics PLC.
- * Graeme Gregory graeme.gregory@wolfsonmicro.com
- * linux@wolfsonmicro.com
- *
- * Copyright (c) 2008, 2007, 2004-2005 Simtec Electronics
- * http://armlinux.simtec.co.uk/
- * Ben Dooks <ben@simtec.co.uk>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// ALSA Soc Audio Layer - I2S core for newer Samsung SoCs.
+//
+// Copyright (c) 2006 Wolfson Microelectronics PLC.
+// Graeme Gregory graeme.gregory@wolfsonmicro.com
+// linux@wolfsonmicro.com
+//
+// Copyright (c) 2008, 2007, 2004-2005 Simtec Electronics
+// http://armlinux.simtec.co.uk/
+// Ben Dooks <ben@simtec.co.uk>
#include <linux/module.h>
#include <linux/delay.h>
-/* sound/soc/samsung/s3c-i2s-v2.h
- *
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
* ALSA Soc Audio Layer - S3C_I2SV2 I2S driver
*
* Copyright (c) 2007 Simtec Electronics
* http://armlinux.simtec.co.uk/
* Ben Dooks <ben@simtec.co.uk>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
-*/
+ */
/* This code is the core support for the I2S block found in a number of
* Samsung SoC devices which is unofficially named I2S-V2. Currently the
-/* sound/soc/samsung/s3c2412-i2s.c
- *
- * ALSA Soc Audio Layer - S3C2412 I2S driver
- *
- * Copyright (c) 2006 Wolfson Microelectronics PLC.
- * Graeme Gregory graeme.gregory@wolfsonmicro.com
- * linux@wolfsonmicro.com
- *
- * Copyright (c) 2007, 2004-2005 Simtec Electronics
- * http://armlinux.simtec.co.uk/
- * Ben Dooks <ben@simtec.co.uk>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// ALSA Soc Audio Layer - S3C2412 I2S driver
+//
+// Copyright (c) 2006 Wolfson Microelectronics PLC.
+// Graeme Gregory graeme.gregory@wolfsonmicro.com
+// linux@wolfsonmicro.com
+//
+// Copyright (c) 2007, 2004-2005 Simtec Electronics
+// http://armlinux.simtec.co.uk/
+// Ben Dooks <ben@simtec.co.uk>
#include <linux/delay.h>
#include <linux/gpio.h>
-/* sound/soc/samsung/s3c2412-i2s.c
- *
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
* ALSA Soc Audio Layer - S3C2412 I2S driver
*
* Copyright (c) 2007 Simtec Electronics
* http://armlinux.simtec.co.uk/
* Ben Dooks <ben@simtec.co.uk>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
-*/
+ */
#ifndef __SND_SOC_S3C24XX_S3C2412_I2S_H
#define __SND_SOC_S3C24XX_S3C2412_I2S_H __FILE__
-/*
- * s3c24xx-i2s.c -- ALSA Soc Audio Layer
- *
- * (c) 2006 Wolfson Microelectronics PLC.
- * Graeme Gregory graeme.gregory@wolfsonmicro.com or linux@wolfsonmicro.com
- *
- * Copyright 2004-2005 Simtec Electronics
- * http://armlinux.simtec.co.uk/
- * Ben Dooks <ben@simtec.co.uk>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// s3c24xx-i2s.c -- ALSA Soc Audio Layer
+//
+// (c) 2006 Wolfson Microelectronics PLC.
+// Graeme Gregory graeme.gregory@wolfsonmicro.com or linux@wolfsonmicro.com
+//
+// Copyright 2004-2005 Simtec Electronics
+// http://armlinux.simtec.co.uk/
+// Ben Dooks <ben@simtec.co.uk>
#include <linux/delay.h>
#include <linux/clk.h>
+/* SPDX-License-Identifier: GPL-2.0+ */
/*
* s3c24xx-i2s.c -- ALSA Soc Audio Layer
*
* Author: Graeme Gregory
* graeme.gregory@wolfsonmicro.com or linux@wolfsonmicro.com
*
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
* Revision history
* 10th Nov 2006 Initial version.
*/
-/* sound/soc/samsung/s3c24xx_simtec.c
- *
- * Copyright 2009 Simtec Electronics
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright 2009 Simtec Electronics
#include <linux/gpio.h>
#include <linux/clk.h>
-/* sound/soc/samsung/s3c24xx_simtec.h
- *
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
* Copyright 2009 Simtec Electronics
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+ */
extern void simtec_audio_init(struct snd_soc_pcm_runtime *rtd);
-/* sound/soc/samsung/s3c24xx_simtec_hermes.c
- *
- * Copyright 2009 Simtec Electronics
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright 2009 Simtec Electronics
#include <linux/module.h>
#include <sound/soc.h>
-/* sound/soc/samsung/s3c24xx_simtec_tlv320aic23.c
- *
- * Copyright 2009 Simtec Electronics
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
-*/
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright 2009 Simtec Electronics
#include <linux/module.h>
#include <sound/soc.h>
-/*
- * Modifications by Christian Pellegrin <chripell@evolware.org>
- *
- * s3c24xx_uda134x.c -- S3C24XX_UDA134X ALSA SoC Audio board driver
- *
- * Copyright 2007 Dension Audio Systems Ltd.
- * Author: Zoltan Devai
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// Modifications by Christian Pellegrin <chripell@evolware.org>
+//
+// s3c24xx_uda134x.c - S3C24XX_UDA134X ALSA SoC Audio board driver
+//
+// Copyright 2007 Dension Audio Systems Ltd.
+// Author: Zoltan Devai
#include <linux/clk.h>
#include <linux/gpio.h>
-/* sound/soc/samsung/smartq_wm8987.c
- *
- * Copyright 2010 Maurus Cuelenaere <mcuelenaere@gmail.com>
- *
- * Based on smdk6410_wm8987.c
- * Copyright 2007 Wolfson Microelectronics PLC. - linux@wolfsonmicro.com
- * Graeme Gregory - graeme.gregory@wolfsonmicro.com
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Copyright 2010 Maurus Cuelenaere <mcuelenaere@gmail.com>
+//
+// Based on smdk6410_wm8987.c
+// Copyright 2007 Wolfson Microelectronics PLC. - linux@wolfsonmicro.com
+// Graeme Gregory - graeme.gregory@wolfsonmicro.com
#include <linux/gpio/consumer.h>
#include <linux/module.h>
-/*
- * smdk_spdif.c -- S/PDIF audio for SMDK
- *
- * Copyright 2010 Samsung Electronics Co. Ltd.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation; either version 2 of the
- * License, or (at your option) any later version.
- *
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// smdk_spdif.c - S/PDIF audio for SMDK
+//
+// Copyright (C) 2010 Samsung Electronics Co., Ltd.
#include <linux/clk.h>
#include <linux/module.h>
-/*
- * smdk_wm8580.c
- *
- * Copyright (c) 2009 Samsung Electronics Co. Ltd
- * Author: Jaswinder Singh <jassisinghbrar@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Copyright (c) 2009 Samsung Electronics Co. Ltd
+// Author: Jaswinder Singh <jassisinghbrar@gmail.com>
#include <linux/module.h>
#include <sound/soc.h>
-/*
- * smdk_wm8994.c
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
#include "../codecs/wm8994.h"
#include <sound/pcm_params.h>
-/*
- * sound/soc/samsung/smdk_wm8994pcm.c
- *
- * Copyright (c) 2011 Samsung Electronics Co., Ltd
- * http://www.samsung.com
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Copyright (c) 2011 Samsung Electronics Co., Ltd
+// http://www.samsung.com
+
#include <linux/module.h>
#include <sound/soc.h>
#include <sound/pcm.h>
-/*
- * ASoC machine driver for Snow boards
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// ASoC machine driver for Snow boards
#include <linux/clk.h>
#include <linux/module.h>
-/* sound/soc/samsung/spdif.c
- *
- * ALSA SoC Audio Layer - Samsung S/PDIF Controller driver
- *
- * Copyright (c) 2010 Samsung Electronics Co. Ltd
- * http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
+// SPDX-License-Identifier: GPL-2.0
+//
+// ALSA SoC Audio Layer - Samsung S/PDIF Controller driver
+//
+// Copyright (c) 2010 Samsung Electronics Co. Ltd
+// http://www.samsung.com/
#include <linux/clk.h>
#include <linux/io.h>
-/* sound/soc/samsung/spdif.h
- *
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
* ALSA SoC Audio Layer - Samsung S/PDIF Controller driver
*
* Copyright (c) 2010 Samsung Electronics Co. Ltd
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#ifndef __SND_SOC_SAMSUNG_SPDIF_H
-/*
- * Speyside audio support
- *
- * Copyright 2011 Wolfson Microelectronics
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Speyside audio support
+//
+// Copyright 2011 Wolfson Microelectronics
#include <sound/soc.h>
#include <sound/soc-dapm.h>
-/*
- * Copyright (C) 2015 - 2016 Samsung Electronics Co., Ltd.
- *
- * Authors: Inha Song <ideal.song@samsung.com>
- * Sylwester Nawrocki <s.nawrocki@samsung.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Copyright (C) 2015 - 2016 Samsung Electronics Co., Ltd.
+//
+// Authors: Inha Song <ideal.song@samsung.com>
+// Sylwester Nawrocki <s.nawrocki@samsung.com>
#include <linux/clk.h>
#include <linux/gpio.h>
-/*
- * Tobermory audio support
- *
- * Copyright 2011 Wolfson Microelectronics
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- */
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Tobermory audio support
+//
+// Copyright 2011 Wolfson Microelectronics
#include <sound/soc.h>
#include <sound/soc-dapm.h>
return chan;
}
+int rsnd_channel_normalization(int chan)
+{
+ if ((chan > 8) || (chan < 0))
+ return 0;
+
+ /* TDM Extend Mode needs 8ch */
+ if (chan == 6)
+ chan = 8;
+
+ return chan;
+}
+
int rsnd_runtime_channel_for_ssi_with_params(struct rsnd_dai_stream *io,
struct snd_pcm_hw_params *params)
{
if (rsnd_runtime_is_multi_ssi(io))
chan /= rsnd_rdai_ssi_lane_get(rdai);
- /* TDM Extend Mode needs 8ch */
- if (chan == 6)
- chan = 8;
-
- return chan;
+ return rsnd_channel_normalization(chan);
}
int rsnd_runtime_is_multi_ssi(struct rsnd_dai_stream *io)
struct device_node *playback,
struct device_node *capture);
+int rsnd_channel_normalization(int chan);
#define rsnd_runtime_channel_original(io) \
rsnd_runtime_channel_original_with_params(io, NULL)
int rsnd_runtime_channel_original_with_params(struct rsnd_dai_stream *io,
if (rsnd_runtime_is_tdm_split(io))
chan = rsnd_io_converted_chan(io);
+ chan = rsnd_channel_normalization(chan);
+
main_rate = rsnd_ssi_clk_query(rdai, rate, chan, &idx);
if (!main_rate) {
dev_err(dev, "unsupported clock rate\n");
continue;
/* for this machine ? */
+ if (!strcmp(component->driver->ignore_machine,
+ card->dev->driver->name))
+ goto match;
if (strcmp(component->driver->ignore_machine,
- card->dev->driver->name))
+ dev_name(card->dev)))
continue;
-
+match:
/* machine matches, so override the rtd data */
for_each_card_prelinks(card, i, dai_link) {
static void snd_soc_unbind_card(struct snd_soc_card *card, bool unregister)
{
+ struct snd_soc_pcm_runtime *rtd;
+ int order;
+
if (card->instantiated) {
card->instantiated = false;
snd_soc_dapm_shutdown(card);
snd_soc_flush_all_delayed_work(card);
+
+ /* remove all components used by DAI links on this card */
+ for_each_comp_order(order) {
+ for_each_card_rtds(card, rtd) {
+ soc_remove_link_components(card, rtd, order);
+ }
+ }
+
soc_cleanup_card_resources(card);
if (!unregister)
list_add(&card->list, &unbind_card_list);
case snd_soc_dapm_switch:
case snd_soc_dapm_mixer:
case snd_soc_dapm_pga:
+ case snd_soc_dapm_effect:
case snd_soc_dapm_out_drv:
wname_in_long_name = true;
kcname_in_long_name = true;
case snd_soc_dapm_dac:
case snd_soc_dapm_adc:
case snd_soc_dapm_pga:
+ case snd_soc_dapm_effect:
case snd_soc_dapm_out_drv:
case snd_soc_dapm_mixer:
case snd_soc_dapm_mixer_named_ctl:
dapm_new_mux(w);
break;
case snd_soc_dapm_pga:
+ case snd_soc_dapm_effect:
case snd_soc_dapm_out_drv:
dapm_new_pga(w);
break;
struct snd_soc_dapm_widget template;
struct snd_soc_dapm_widget *w;
const char **w_param_text;
- unsigned long private_value;
+ unsigned long private_value = 0;
char *link_name;
int ret;
else
codec_stream = &dai->driver->capture;
- /* If the codec specifies any rate at all, it supports the stream. */
- return codec_stream->rates;
+ /* If the codec specifies any channels at all, it supports the stream */
+ return codec_stream->channels_min;
}
/**
codec_err:
for_each_rtd_codec_dai_rollback(rtd, i, codec_dai) {
+ if (!snd_soc_dai_stream_valid(codec_dai, substream->stream))
+ continue;
+
if (codec_dai->driver->ops->hw_free)
codec_dai->driver->ops->hw_free(substream, codec_dai);
codec_dai->rate = 0;
/* now free hw params for the DAIs */
for_each_rtd_codec_dai(rtd, i, codec_dai) {
+ if (!snd_soc_dai_stream_valid(codec_dai, substream->stream))
+ continue;
+
if (codec_dai->driver->ops->hw_free)
codec_dai->driver->ops->hw_free(substream, codec_dai);
}
}
}
+ /* copy the fixed-up hw params for BE dai */
+ memcpy(&be->dpcm[stream].hw_params, &dpcm->hw_params,
+ sizeof(struct snd_pcm_hw_params));
+
/* only allow hw_params() if no connected FEs are running */
if (!snd_soc_dpcm_can_be_params(fe, be, stream))
continue;
#include <sound/soc-topology.h>
#include <sound/tlv.h>
+#define SOC_TPLG_MAGIC_BIG_ENDIAN 0x436F5341 /* ASoC in reverse */
+
/*
* We make several passes over the data (since it wont necessarily be ordered)
* and process objects in the following order. This guarantees the component
int i;
for (i = 0; i < SND_SOC_TPLG_MAX_CHAN; i++) {
- if (chan[i].id == map)
- return chan[i].reg;
+ if (le32_to_cpu(chan[i].id) == map)
+ return le32_to_cpu(chan[i].reg);
}
return -EINVAL;
int i;
for (i = 0; i < SND_SOC_TPLG_MAX_CHAN; i++) {
- if (chan[i].id == map)
- return chan[i].shift;
+ if (le32_to_cpu(chan[i].id) == map)
+ return le32_to_cpu(chan[i].shift);
}
return -EINVAL;
if (dai->driver == dai_drv)
dai->driver = NULL;
+ kfree(dai_drv->playback.stream_name);
+ kfree(dai_drv->capture.stream_name);
kfree(dai_drv->name);
list_del(&dobj->list);
kfree(dai_drv);
const struct snd_soc_tplg_bytes_ext_ops *ext_ops;
int num_ops, i;
- if (hdr->ops.info == SND_SOC_TPLG_CTL_BYTES
+ if (le32_to_cpu(hdr->ops.info) == SND_SOC_TPLG_CTL_BYTES
&& k->iface & SNDRV_CTL_ELEM_IFACE_MIXER
&& k->access & SNDRV_CTL_ELEM_ACCESS_TLV_READWRITE
&& k->access & SNDRV_CTL_ELEM_ACCESS_TLV_CALLBACK) {
p[0] = SNDRV_CTL_TLVT_DB_SCALE;
p[1] = item_len;
- p[2] = scale->min;
- p[3] = (scale->step & TLV_DB_SCALE_MASK)
- | (scale->mute ? TLV_DB_SCALE_MUTE : 0);
+ p[2] = le32_to_cpu(scale->min);
+ p[3] = (le32_to_cpu(scale->step) & TLV_DB_SCALE_MASK)
+ | (le32_to_cpu(scale->mute) ? TLV_DB_SCALE_MUTE : 0);
kc->tlv.p = (void *)p;
return 0;
struct snd_kcontrol_new *kc, struct snd_soc_tplg_ctl_hdr *tc)
{
struct snd_soc_tplg_ctl_tlv *tplg_tlv;
+ u32 access = le32_to_cpu(tc->access);
- if (!(tc->access & SNDRV_CTL_ELEM_ACCESS_TLV_READWRITE))
+ if (!(access & SNDRV_CTL_ELEM_ACCESS_TLV_READWRITE))
return 0;
- if (!(tc->access & SNDRV_CTL_ELEM_ACCESS_TLV_CALLBACK)) {
+ if (!(access & SNDRV_CTL_ELEM_ACCESS_TLV_CALLBACK)) {
tplg_tlv = &tc->tlv;
- switch (tplg_tlv->type) {
+ switch (le32_to_cpu(tplg_tlv->type)) {
case SNDRV_CTL_TLVT_DB_SCALE:
return soc_tplg_create_tlv_db_scale(tplg, kc,
&tplg_tlv->scale);
return -ENOMEM;
tplg->pos += (sizeof(struct snd_soc_tplg_bytes_control) +
- be->priv.size);
+ le32_to_cpu(be->priv.size));
dev_dbg(tplg->dev,
"ASoC: adding bytes kcontrol %s with access 0x%x\n",
kc.name = be->hdr.name;
kc.private_value = (long)sbe;
kc.iface = SNDRV_CTL_ELEM_IFACE_MIXER;
- kc.access = be->hdr.access;
+ kc.access = le32_to_cpu(be->hdr.access);
- sbe->max = be->max;
+ sbe->max = le32_to_cpu(be->max);
sbe->dobj.type = SND_SOC_DOBJ_BYTES;
sbe->dobj.ops = tplg->ops;
INIT_LIST_HEAD(&sbe->dobj.list);
if (sm == NULL)
return -ENOMEM;
tplg->pos += (sizeof(struct snd_soc_tplg_mixer_control) +
- mc->priv.size);
+ le32_to_cpu(mc->priv.size));
dev_dbg(tplg->dev,
"ASoC: adding mixer kcontrol %s with access 0x%x\n",
kc.name = mc->hdr.name;
kc.private_value = (long)sm;
kc.iface = SNDRV_CTL_ELEM_IFACE_MIXER;
- kc.access = mc->hdr.access;
+ kc.access = le32_to_cpu(mc->hdr.access);
/* we only support FL/FR channel mapping atm */
sm->reg = tplc_chan_get_reg(tplg, mc->channel,
sm->rshift = tplc_chan_get_shift(tplg, mc->channel,
SNDRV_CHMAP_FR);
- sm->max = mc->max;
- sm->min = mc->min;
- sm->invert = mc->invert;
- sm->platform_max = mc->platform_max;
+ sm->max = le32_to_cpu(mc->max);
+ sm->min = le32_to_cpu(mc->min);
+ sm->invert = le32_to_cpu(mc->invert);
+ sm->platform_max = le32_to_cpu(mc->platform_max);
sm->dobj.index = tplg->index;
sm->dobj.ops = tplg->ops;
sm->dobj.type = SND_SOC_DOBJ_MIXER;
continue;
}
+ /* create any TLV data */
+ soc_tplg_create_tlv(tplg, &kc, &mc->hdr);
+
/* pass control to driver for optional further init */
err = soc_tplg_init_kcontrol(tplg, &kc,
(struct snd_soc_tplg_ctl_hdr *) mc);
if (err < 0) {
dev_err(tplg->dev, "ASoC: failed to init %s\n",
mc->hdr.name);
+ soc_tplg_free_tlv(tplg, &kc);
kfree(sm);
continue;
}
- /* create any TLV data */
- soc_tplg_create_tlv(tplg, &kc, &mc->hdr);
-
/* register control here */
err = soc_tplg_add_kcontrol(tplg, &kc,
&sm->dobj.control.kcontrol);
int i, ret;
se->dobj.control.dtexts =
- kcalloc(ec->items, sizeof(char *), GFP_KERNEL);
+ kcalloc(le32_to_cpu(ec->items), sizeof(char *), GFP_KERNEL);
if (se->dobj.control.dtexts == NULL)
return -ENOMEM;
static int soc_tplg_denum_create_values(struct soc_enum *se,
struct snd_soc_tplg_enum_control *ec)
{
- if (ec->items > sizeof(*ec->values))
+ int i;
+
+ if (le32_to_cpu(ec->items) > sizeof(*ec->values))
return -EINVAL;
- se->dobj.control.dvalues = kmemdup(ec->values,
- ec->items * sizeof(u32),
+ se->dobj.control.dvalues = kzalloc(le32_to_cpu(ec->items) *
+ sizeof(u32),
GFP_KERNEL);
if (!se->dobj.control.dvalues)
return -ENOMEM;
+ /* convert from little-endian */
+ for (i = 0; i < le32_to_cpu(ec->items); i++) {
+ se->dobj.control.dvalues[i] = le32_to_cpu(ec->values[i]);
+ }
+
return 0;
}
for (i = 0; i < count; i++) {
ec = (struct snd_soc_tplg_enum_control *)tplg->pos;
- tplg->pos += (sizeof(struct snd_soc_tplg_enum_control) +
- ec->priv.size);
/* validate kcontrol */
if (strnlen(ec->hdr.name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN) ==
if (se == NULL)
return -ENOMEM;
+ tplg->pos += (sizeof(struct snd_soc_tplg_enum_control) +
+ le32_to_cpu(ec->priv.size));
+
dev_dbg(tplg->dev, "ASoC: adding enum kcontrol %s size %d\n",
ec->hdr.name, ec->items);
kc.name = ec->hdr.name;
kc.private_value = (long)se;
kc.iface = SNDRV_CTL_ELEM_IFACE_MIXER;
- kc.access = ec->hdr.access;
+ kc.access = le32_to_cpu(ec->hdr.access);
se->reg = tplc_chan_get_reg(tplg, ec->channel, SNDRV_CHMAP_FL);
se->shift_l = tplc_chan_get_shift(tplg, ec->channel,
se->shift_r = tplc_chan_get_shift(tplg, ec->channel,
SNDRV_CHMAP_FL);
- se->items = ec->items;
- se->mask = ec->mask;
+ se->items = le32_to_cpu(ec->items);
+ se->mask = le32_to_cpu(ec->mask);
se->dobj.index = tplg->index;
se->dobj.type = SND_SOC_DOBJ_ENUM;
se->dobj.ops = tplg->ops;
INIT_LIST_HEAD(&se->dobj.list);
- switch (ec->hdr.ops.info) {
+ switch (le32_to_cpu(ec->hdr.ops.info)) {
case SND_SOC_TPLG_DAPM_CTL_ENUM_VALUE:
case SND_SOC_TPLG_CTL_ENUM_VALUE:
err = soc_tplg_denum_create_values(se, ec);
int i;
if (tplg->pass != SOC_TPLG_PASS_MIXER) {
- tplg->pos += hdr->size + hdr->payload_size;
+ tplg->pos += le32_to_cpu(hdr->size) +
+ le32_to_cpu(hdr->payload_size);
return 0;
}
dev_dbg(tplg->dev, "ASoC: adding %d kcontrols at 0x%lx\n", hdr->count,
soc_tplg_get_offset(tplg));
- for (i = 0; i < hdr->count; i++) {
+ for (i = 0; i < le32_to_cpu(hdr->count); i++) {
control_hdr = (struct snd_soc_tplg_ctl_hdr *)tplg->pos;
- if (control_hdr->size != sizeof(*control_hdr)) {
+ if (le32_to_cpu(control_hdr->size) != sizeof(*control_hdr)) {
dev_err(tplg->dev, "ASoC: invalid control size\n");
return -EINVAL;
}
- switch (control_hdr->ops.info) {
+ switch (le32_to_cpu(control_hdr->ops.info)) {
case SND_SOC_TPLG_CTL_VOLSW:
case SND_SOC_TPLG_CTL_STROBE:
case SND_SOC_TPLG_CTL_VOLSW_SX:
case SND_SOC_TPLG_CTL_RANGE:
case SND_SOC_TPLG_DAPM_CTL_VOLSW:
case SND_SOC_TPLG_DAPM_CTL_PIN:
- soc_tplg_dmixer_create(tplg, 1, hdr->payload_size);
+ soc_tplg_dmixer_create(tplg, 1,
+ le32_to_cpu(hdr->payload_size));
break;
case SND_SOC_TPLG_CTL_ENUM:
case SND_SOC_TPLG_CTL_ENUM_VALUE:
case SND_SOC_TPLG_DAPM_CTL_ENUM_DOUBLE:
case SND_SOC_TPLG_DAPM_CTL_ENUM_VIRT:
case SND_SOC_TPLG_DAPM_CTL_ENUM_VALUE:
- soc_tplg_denum_create(tplg, 1, hdr->payload_size);
+ soc_tplg_denum_create(tplg, 1,
+ le32_to_cpu(hdr->payload_size));
break;
case SND_SOC_TPLG_CTL_BYTES:
- soc_tplg_dbytes_create(tplg, 1, hdr->payload_size);
+ soc_tplg_dbytes_create(tplg, 1,
+ le32_to_cpu(hdr->payload_size));
break;
default:
soc_bind_err(tplg, control_hdr, i);
struct snd_soc_dapm_context *dapm = &tplg->comp->dapm;
struct snd_soc_tplg_dapm_graph_elem *elem;
struct snd_soc_dapm_route **routes;
- int count = hdr->count, i, j;
+ int count, i, j;
int ret = 0;
+ count = le32_to_cpu(hdr->count);
+
if (tplg->pass != SOC_TPLG_PASS_GRAPH) {
- tplg->pos += hdr->size + hdr->payload_size;
+ tplg->pos +=
+ le32_to_cpu(hdr->size) +
+ le32_to_cpu(hdr->payload_size);
+
return 0;
}
if (soc_tplg_check_elem_count(tplg,
sizeof(struct snd_soc_tplg_dapm_graph_elem),
- count, hdr->payload_size, "graph")) {
+ count, le32_to_cpu(hdr->payload_size), "graph")) {
dev_err(tplg->dev, "ASoC: invalid count %d for DAPM routes\n",
count);
if (sm == NULL)
goto err;
- tplg->pos += (sizeof(struct snd_soc_tplg_mixer_control) +
- mc->priv.size);
-
/* validate kcontrol */
if (strnlen(mc->hdr.name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN) ==
SNDRV_CTL_ELEM_ID_NAME_MAXLEN)
goto err_str;
+ tplg->pos += (sizeof(struct snd_soc_tplg_mixer_control) +
+ le32_to_cpu(mc->priv.size));
+
dev_dbg(tplg->dev, " adding DAPM widget mixer control %s at %d\n",
mc->hdr.name, i);
continue;
}
+ /* create any TLV data */
+ soc_tplg_create_tlv(tplg, &kc[i], &mc->hdr);
+
/* pass control to driver for optional further init */
err = soc_tplg_init_kcontrol(tplg, &kc[i],
(struct snd_soc_tplg_ctl_hdr *)mc);
if (err < 0) {
dev_err(tplg->dev, "ASoC: failed to init %s\n",
mc->hdr.name);
+ soc_tplg_free_tlv(tplg, &kc[i]);
kfree(sm);
continue;
}
-
- /* create any TLV data */
- soc_tplg_create_tlv(tplg, &kc[i], &mc->hdr);
}
return kc;
if (se == NULL)
goto err;
+ tplg->pos += (sizeof(struct snd_soc_tplg_enum_control) +
+ ec->priv.size);
+
dev_dbg(tplg->dev, " adding DAPM widget enum control %s\n",
ec->hdr.name);
se->mask = ec->mask;
se->dobj.index = tplg->index;
- switch (ec->hdr.ops.info) {
+ switch (le32_to_cpu(ec->hdr.ops.info)) {
case SND_SOC_TPLG_CTL_ENUM_VALUE:
case SND_SOC_TPLG_DAPM_CTL_ENUM_VALUE:
err = soc_tplg_denum_create_values(se, ec);
ec->hdr.name);
goto err_se;
}
-
- tplg->pos += (sizeof(struct snd_soc_tplg_enum_control) +
- ec->priv.size);
}
return kc;
goto err;
tplg->pos += (sizeof(struct snd_soc_tplg_bytes_control) +
- be->priv.size);
+ le32_to_cpu(be->priv.size));
dev_dbg(tplg->dev,
"ASoC: adding bytes kcontrol %s with access 0x%x\n",
memset(&template, 0, sizeof(template));
/* map user to kernel widget ID */
- template.id = get_widget_id(w->id);
+ template.id = get_widget_id(le32_to_cpu(w->id));
if (template.id < 0)
return template.id;
ret = -ENOMEM;
goto err;
}
- template.reg = w->reg;
- template.shift = w->shift;
- template.mask = w->mask;
- template.subseq = w->subseq;
+ template.reg = le32_to_cpu(w->reg);
+ template.shift = le32_to_cpu(w->shift);
+ template.mask = le32_to_cpu(w->mask);
+ template.subseq = le32_to_cpu(w->subseq);
template.on_val = w->invert ? 0 : 1;
template.off_val = w->invert ? 1 : 0;
- template.ignore_suspend = w->ignore_suspend;
- template.event_flags = w->event_flags;
+ template.ignore_suspend = le32_to_cpu(w->ignore_suspend);
+ template.event_flags = le16_to_cpu(w->event_flags);
template.dobj.index = tplg->index;
tplg->pos +=
- (sizeof(struct snd_soc_tplg_dapm_widget) + w->priv.size);
+ (sizeof(struct snd_soc_tplg_dapm_widget) +
+ le32_to_cpu(w->priv.size));
+
if (w->num_kcontrols == 0) {
kcontrol_type = 0;
template.num_kcontrols = 0;
dev_dbg(tplg->dev, "ASoC: template %s has %d controls of type %x\n",
w->name, w->num_kcontrols, control_hdr->type);
- switch (control_hdr->ops.info) {
+ switch (le32_to_cpu(control_hdr->ops.info)) {
case SND_SOC_TPLG_CTL_VOLSW:
case SND_SOC_TPLG_CTL_STROBE:
case SND_SOC_TPLG_CTL_VOLSW_SX:
case SND_SOC_TPLG_CTL_RANGE:
case SND_SOC_TPLG_DAPM_CTL_VOLSW:
kcontrol_type = SND_SOC_TPLG_TYPE_MIXER; /* volume mixer */
- template.num_kcontrols = w->num_kcontrols;
+ template.num_kcontrols = le32_to_cpu(w->num_kcontrols);
template.kcontrol_news =
soc_tplg_dapm_widget_dmixer_create(tplg,
template.num_kcontrols);
case SND_SOC_TPLG_DAPM_CTL_ENUM_VIRT:
case SND_SOC_TPLG_DAPM_CTL_ENUM_VALUE:
kcontrol_type = SND_SOC_TPLG_TYPE_ENUM; /* enumerated mixer */
- template.num_kcontrols = w->num_kcontrols;
+ template.num_kcontrols = le32_to_cpu(w->num_kcontrols);
template.kcontrol_news =
soc_tplg_dapm_widget_denum_create(tplg,
template.num_kcontrols);
break;
case SND_SOC_TPLG_CTL_BYTES:
kcontrol_type = SND_SOC_TPLG_TYPE_BYTES; /* bytes control */
- template.num_kcontrols = w->num_kcontrols;
+ template.num_kcontrols = le32_to_cpu(w->num_kcontrols);
template.kcontrol_news =
soc_tplg_dapm_widget_dbytes_create(tplg,
template.num_kcontrols);
default:
dev_err(tplg->dev, "ASoC: invalid widget control type %d:%d:%d\n",
control_hdr->ops.get, control_hdr->ops.put,
- control_hdr->ops.info);
+ le32_to_cpu(control_hdr->ops.info));
ret = -EINVAL;
goto hdr_err;
}
struct snd_soc_tplg_hdr *hdr)
{
struct snd_soc_tplg_dapm_widget *widget;
- int ret, count = hdr->count, i;
+ int ret, count, i;
+
+ count = le32_to_cpu(hdr->count);
if (tplg->pass != SOC_TPLG_PASS_WIDGET)
return 0;
for (i = 0; i < count; i++) {
widget = (struct snd_soc_tplg_dapm_widget *) tplg->pos;
- if (widget->size != sizeof(*widget)) {
+ if (le32_to_cpu(widget->size) != sizeof(*widget)) {
dev_err(tplg->dev, "ASoC: invalid widget size\n");
return -EINVAL;
}
struct snd_soc_tplg_stream_caps *caps)
{
stream->stream_name = kstrdup(caps->name, GFP_KERNEL);
- stream->channels_min = caps->channels_min;
- stream->channels_max = caps->channels_max;
- stream->rates = caps->rates;
- stream->rate_min = caps->rate_min;
- stream->rate_max = caps->rate_max;
- stream->formats = caps->formats;
- stream->sig_bits = caps->sig_bits;
+ stream->channels_min = le32_to_cpu(caps->channels_min);
+ stream->channels_max = le32_to_cpu(caps->channels_max);
+ stream->rates = le32_to_cpu(caps->rates);
+ stream->rate_min = le32_to_cpu(caps->rate_min);
+ stream->rate_max = le32_to_cpu(caps->rate_max);
+ stream->formats = le64_to_cpu(caps->formats);
+ stream->sig_bits = le32_to_cpu(caps->sig_bits);
}
static void set_dai_flags(struct snd_soc_dai_driver *dai_drv,
if (strlen(pcm->dai_name))
dai_drv->name = kstrdup(pcm->dai_name, GFP_KERNEL);
- dai_drv->id = pcm->dai_id;
+ dai_drv->id = le32_to_cpu(pcm->dai_id);
if (pcm->playback) {
stream = &dai_drv->playback;
ret = soc_tplg_dai_load(tplg, dai_drv, pcm, NULL);
if (ret < 0) {
dev_err(tplg->comp->dev, "ASoC: DAI loading failed\n");
+ kfree(dai_drv->playback.stream_name);
+ kfree(dai_drv->capture.stream_name);
+ kfree(dai_drv->name);
kfree(dai_drv);
return ret;
}
link->name = kstrdup(pcm->pcm_name, GFP_KERNEL);
link->stream_name = kstrdup(pcm->pcm_name, GFP_KERNEL);
}
- link->id = pcm->pcm_id;
+ link->id = le32_to_cpu(pcm->pcm_id);
if (strlen(pcm->dai_name))
link->cpu_dai_name = kstrdup(pcm->dai_name, GFP_KERNEL);
/* enable DPCM */
link->dynamic = 1;
- link->dpcm_playback = pcm->playback;
- link->dpcm_capture = pcm->capture;
+ link->dpcm_playback = le32_to_cpu(pcm->playback);
+ link->dpcm_capture = le32_to_cpu(pcm->capture);
if (pcm->flag_mask)
- set_link_flags(link, pcm->flag_mask, pcm->flags);
+ set_link_flags(link,
+ le32_to_cpu(pcm->flag_mask),
+ le32_to_cpu(pcm->flags));
/* pass control to component driver for optional further init */
ret = soc_tplg_dai_link_load(tplg, link, NULL);
if (ret < 0) {
dev_err(tplg->comp->dev, "ASoC: FE link loading failed\n");
+ kfree(link->name);
+ kfree(link->stream_name);
+ kfree(link->cpu_dai_name);
kfree(link);
return ret;
}
static void stream_caps_new_ver(struct snd_soc_tplg_stream_caps *dest,
struct snd_soc_tplg_stream_caps_v4 *src)
{
- dest->size = sizeof(*dest);
+ dest->size = cpu_to_le32(sizeof(*dest));
memcpy(dest->name, src->name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN);
dest->formats = src->formats;
dest->rates = src->rates;
*pcm = NULL;
- if (src->size != sizeof(*src_v4)) {
+ if (le32_to_cpu(src->size) != sizeof(*src_v4)) {
dev_err(tplg->dev, "ASoC: invalid PCM size\n");
return -EINVAL;
}
if (!dest)
return -ENOMEM;
- dest->size = sizeof(*dest); /* size of latest abi version */
+ dest->size = cpu_to_le32(sizeof(*dest)); /* size of latest abi version */
memcpy(dest->pcm_name, src_v4->pcm_name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN);
memcpy(dest->dai_name, src_v4->dai_name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN);
dest->pcm_id = src_v4->pcm_id;
dest->capture = src_v4->capture;
dest->compress = src_v4->compress;
dest->num_streams = src_v4->num_streams;
- for (i = 0; i < dest->num_streams; i++)
+ for (i = 0; i < le32_to_cpu(dest->num_streams); i++)
memcpy(&dest->stream[i], &src_v4->stream[i],
sizeof(struct snd_soc_tplg_stream));
struct snd_soc_tplg_hdr *hdr)
{
struct snd_soc_tplg_pcm *pcm, *_pcm;
- int count = hdr->count;
+ int count;
+ int size;
int i;
bool abi_match;
+ count = le32_to_cpu(hdr->count);
+
if (tplg->pass != SOC_TPLG_PASS_PCM_DAI)
return 0;
/* check the element size and count */
pcm = (struct snd_soc_tplg_pcm *)tplg->pos;
- if (pcm->size > sizeof(struct snd_soc_tplg_pcm)
- || pcm->size < sizeof(struct snd_soc_tplg_pcm_v4)) {
+ size = le32_to_cpu(pcm->size);
+ if (size > sizeof(struct snd_soc_tplg_pcm)
+ || size < sizeof(struct snd_soc_tplg_pcm_v4)) {
dev_err(tplg->dev, "ASoC: invalid size %d for PCM elems\n",
- pcm->size);
+ size);
return -EINVAL;
}
if (soc_tplg_check_elem_count(tplg,
- pcm->size, count,
- hdr->payload_size, "PCM DAI")) {
+ size, count,
+ le32_to_cpu(hdr->payload_size),
+ "PCM DAI")) {
dev_err(tplg->dev, "ASoC: invalid count %d for PCM DAI elems\n",
count);
return -EINVAL;
for (i = 0; i < count; i++) {
pcm = (struct snd_soc_tplg_pcm *)tplg->pos;
+ size = le32_to_cpu(pcm->size);
/* check ABI version by size, create a new version of pcm
* if abi not match.
*/
- if (pcm->size == sizeof(*pcm)) {
+ if (size == sizeof(*pcm)) {
abi_match = true;
_pcm = pcm;
} else {
/* offset by version-specific struct size and
* real priv data size
*/
- tplg->pos += pcm->size + _pcm->priv.size;
+ tplg->pos += size + le32_to_cpu(_pcm->priv.size);
if (!abi_match)
kfree(_pcm); /* free the duplicated one */
unsigned char invert_bclk, invert_fsync;
int i;
- for (i = 0; i < cfg->num_hw_configs; i++) {
+ for (i = 0; i < le32_to_cpu(cfg->num_hw_configs); i++) {
hw_config = &cfg->hw_config[i];
if (hw_config->id != cfg->default_hw_config_id)
continue;
- link->dai_fmt = hw_config->fmt & SND_SOC_DAIFMT_FORMAT_MASK;
+ link->dai_fmt = le32_to_cpu(hw_config->fmt) &
+ SND_SOC_DAIFMT_FORMAT_MASK;
/* clock gating */
switch (hw_config->clock_gated) {
*link = NULL;
- if (src->size != sizeof(struct snd_soc_tplg_link_config_v4)) {
+ if (le32_to_cpu(src->size) !=
+ sizeof(struct snd_soc_tplg_link_config_v4)) {
dev_err(tplg->dev, "ASoC: invalid physical link config size\n");
return -EINVAL;
}
if (!dest)
return -ENOMEM;
- dest->size = sizeof(*dest);
+ dest->size = cpu_to_le32(sizeof(*dest));
dest->id = src_v4->id;
dest->num_streams = src_v4->num_streams;
- for (i = 0; i < dest->num_streams; i++)
+ for (i = 0; i < le32_to_cpu(dest->num_streams); i++)
memcpy(&dest->stream[i], &src_v4->stream[i],
sizeof(struct snd_soc_tplg_stream));
else
stream_name = NULL;
- link = snd_soc_find_dai_link(tplg->comp->card, cfg->id,
+ link = snd_soc_find_dai_link(tplg->comp->card, le32_to_cpu(cfg->id),
name, stream_name);
if (!link) {
dev_err(tplg->dev, "ASoC: physical link %s (id %d) not exist\n",
/* flags */
if (cfg->flag_mask)
- set_link_flags(link, cfg->flag_mask, cfg->flags);
+ set_link_flags(link,
+ le32_to_cpu(cfg->flag_mask),
+ le32_to_cpu(cfg->flags));
/* pass control to component driver for optional further init */
ret = soc_tplg_dai_link_load(tplg, link, cfg);
struct snd_soc_tplg_hdr *hdr)
{
struct snd_soc_tplg_link_config *link, *_link;
- int count = hdr->count;
+ int count;
+ int size;
int i, ret;
bool abi_match;
+ count = le32_to_cpu(hdr->count);
+
if (tplg->pass != SOC_TPLG_PASS_LINK) {
- tplg->pos += hdr->size + hdr->payload_size;
+ tplg->pos += le32_to_cpu(hdr->size) +
+ le32_to_cpu(hdr->payload_size);
return 0;
};
/* check the element size and count */
link = (struct snd_soc_tplg_link_config *)tplg->pos;
- if (link->size > sizeof(struct snd_soc_tplg_link_config)
- || link->size < sizeof(struct snd_soc_tplg_link_config_v4)) {
+ size = le32_to_cpu(link->size);
+ if (size > sizeof(struct snd_soc_tplg_link_config)
+ || size < sizeof(struct snd_soc_tplg_link_config_v4)) {
dev_err(tplg->dev, "ASoC: invalid size %d for physical link elems\n",
- link->size);
+ size);
return -EINVAL;
}
if (soc_tplg_check_elem_count(tplg,
- link->size, count,
- hdr->payload_size, "physical link config")) {
+ size, count,
+ le32_to_cpu(hdr->payload_size),
+ "physical link config")) {
dev_err(tplg->dev, "ASoC: invalid count %d for physical link elems\n",
count);
return -EINVAL;
/* config physical DAI links */
for (i = 0; i < count; i++) {
link = (struct snd_soc_tplg_link_config *)tplg->pos;
- if (link->size == sizeof(*link)) {
+ size = le32_to_cpu(link->size);
+ if (size == sizeof(*link)) {
abi_match = true;
_link = link;
} else {
/* offset by version-specific struct size and
* real priv data size
*/
- tplg->pos += link->size + _link->priv.size;
+ tplg->pos += size + le32_to_cpu(_link->priv.size);
if (!abi_match)
kfree(_link); /* free the duplicated one */
static int soc_tplg_dai_config(struct soc_tplg *tplg,
struct snd_soc_tplg_dai *d)
{
- struct snd_soc_dai_link_component dai_component = {0};
+ struct snd_soc_dai_link_component dai_component;
struct snd_soc_dai *dai;
struct snd_soc_dai_driver *dai_drv;
struct snd_soc_pcm_stream *stream;
struct snd_soc_tplg_stream_caps *caps;
int ret;
+ memset(&dai_component, 0, sizeof(dai_component));
+
dai_component.dai_name = d->dai_name;
dai = snd_soc_find_dai(&dai_component);
if (!dai) {
return -EINVAL;
}
- if (d->dai_id != dai->id) {
+ if (le32_to_cpu(d->dai_id) != dai->id) {
dev_err(tplg->dev, "ASoC: physical DAI %s id mismatch\n",
d->dai_name);
return -EINVAL;
}
if (d->flag_mask)
- set_dai_flags(dai_drv, d->flag_mask, d->flags);
+ set_dai_flags(dai_drv,
+ le32_to_cpu(d->flag_mask),
+ le32_to_cpu(d->flags));
/* pass control to component driver for optional further init */
ret = soc_tplg_dai_load(tplg, dai_drv, NULL, dai);
struct snd_soc_tplg_hdr *hdr)
{
struct snd_soc_tplg_dai *dai;
- int count = hdr->count;
+ int count;
int i;
+ count = le32_to_cpu(hdr->count);
+
if (tplg->pass != SOC_TPLG_PASS_BE_DAI)
return 0;
/* config the existing BE DAIs */
for (i = 0; i < count; i++) {
dai = (struct snd_soc_tplg_dai *)tplg->pos;
- if (dai->size != sizeof(*dai)) {
+ if (le32_to_cpu(dai->size) != sizeof(*dai)) {
dev_err(tplg->dev, "ASoC: invalid physical DAI size\n");
return -EINVAL;
}
soc_tplg_dai_config(tplg, dai);
- tplg->pos += (sizeof(*dai) + dai->priv.size);
+ tplg->pos += (sizeof(*dai) + le32_to_cpu(dai->priv.size));
}
dev_dbg(tplg->dev, "ASoC: Configure %d BE DAIs\n", count);
{
struct snd_soc_tplg_manifest *dest;
struct snd_soc_tplg_manifest_v4 *src_v4;
+ int size;
*manifest = NULL;
- if (src->size != sizeof(*src_v4)) {
+ size = le32_to_cpu(src->size);
+ if (size != sizeof(*src_v4)) {
dev_warn(tplg->dev, "ASoC: invalid manifest size %d\n",
- src->size);
- if (src->size)
+ size);
+ if (size)
return -EINVAL;
- src->size = sizeof(*src_v4);
+ src->size = cpu_to_le32(sizeof(*src_v4));
}
dev_warn(tplg->dev, "ASoC: old version of manifest\n");
src_v4 = (struct snd_soc_tplg_manifest_v4 *)src;
- dest = kzalloc(sizeof(*dest) + src_v4->priv.size, GFP_KERNEL);
+ dest = kzalloc(sizeof(*dest) + le32_to_cpu(src_v4->priv.size),
+ GFP_KERNEL);
if (!dest)
return -ENOMEM;
- dest->size = sizeof(*dest); /* size of latest abi version */
+ dest->size = cpu_to_le32(sizeof(*dest)); /* size of latest abi version */
dest->control_elems = src_v4->control_elems;
dest->widget_elems = src_v4->widget_elems;
dest->graph_elems = src_v4->graph_elems;
dest->priv.size = src_v4->priv.size;
if (dest->priv.size)
memcpy(dest->priv.data, src_v4->priv.data,
- src_v4->priv.size);
+ le32_to_cpu(src_v4->priv.size));
*manifest = dest;
return 0;
manifest = (struct snd_soc_tplg_manifest *)tplg->pos;
/* check ABI version by size, create a new manifest if abi not match */
- if (manifest->size == sizeof(*manifest)) {
+ if (le32_to_cpu(manifest->size) == sizeof(*manifest)) {
abi_match = true;
_manifest = manifest;
} else {
if (soc_tplg_get_hdr_offset(tplg) >= tplg->fw->size)
return 0;
- if (hdr->size != sizeof(*hdr)) {
+ if (le32_to_cpu(hdr->size) != sizeof(*hdr)) {
dev_err(tplg->dev,
"ASoC: invalid header size for type %d at offset 0x%lx size 0x%zx.\n",
- hdr->type, soc_tplg_get_hdr_offset(tplg),
+ le32_to_cpu(hdr->type), soc_tplg_get_hdr_offset(tplg),
tplg->fw->size);
return -EINVAL;
}
/* big endian firmware objects not supported atm */
- if (hdr->magic == cpu_to_be32(SND_SOC_TPLG_MAGIC)) {
+ if (hdr->magic == SOC_TPLG_MAGIC_BIG_ENDIAN) {
dev_err(tplg->dev,
"ASoC: pass %d big endian not supported header got %x at offset 0x%lx size 0x%zx.\n",
tplg->pass, hdr->magic,
return -EINVAL;
}
- if (hdr->magic != SND_SOC_TPLG_MAGIC) {
+ if (le32_to_cpu(hdr->magic) != SND_SOC_TPLG_MAGIC) {
dev_err(tplg->dev,
"ASoC: pass %d does not have a valid header got %x at offset 0x%lx size 0x%zx.\n",
tplg->pass, hdr->magic,
}
/* Support ABI from version 4 */
- if (hdr->abi > SND_SOC_TPLG_ABI_VERSION
- || hdr->abi < SND_SOC_TPLG_ABI_VERSION_MIN) {
+ if (le32_to_cpu(hdr->abi) > SND_SOC_TPLG_ABI_VERSION ||
+ le32_to_cpu(hdr->abi) < SND_SOC_TPLG_ABI_VERSION_MIN) {
dev_err(tplg->dev,
"ASoC: pass %d invalid ABI version got 0x%x need 0x%x at offset 0x%lx size 0x%zx.\n",
tplg->pass, hdr->abi,
return -EINVAL;
}
- if (tplg->pass == hdr->type)
+ if (tplg->pass == le32_to_cpu(hdr->type))
dev_dbg(tplg->dev,
"ASoC: Got 0x%x bytes of type %d version %d vendor %d at pass %d\n",
hdr->payload_size, hdr->type, hdr->version,
tplg->pos = tplg->hdr_pos + sizeof(struct snd_soc_tplg_hdr);
/* check for matching ID */
- if (hdr->index != tplg->req_index &&
+ if (le32_to_cpu(hdr->index) != tplg->req_index &&
tplg->req_index != SND_SOC_TPLG_INDEX_ALL)
return 0;
- tplg->index = hdr->index;
+ tplg->index = le32_to_cpu(hdr->index);
- switch (hdr->type) {
+ switch (le32_to_cpu(hdr->type)) {
case SND_SOC_TPLG_TYPE_MIXER:
case SND_SOC_TPLG_TYPE_ENUM:
case SND_SOC_TPLG_TYPE_BYTES:
return ret;
/* goto next header */
- tplg->hdr_pos += hdr->payload_size +
+ tplg->hdr_pos += le32_to_cpu(hdr->payload_size) +
sizeof(struct snd_soc_tplg_hdr);
hdr = (struct snd_soc_tplg_hdr *)tplg->hdr_pos;
}
--- /dev/null
+config SND_SOC_SOF_TOPLEVEL
+ bool "Sound Open Firmware Support"
+ help
+ This adds support for Sound Open Firmware (SOF). SOF is a free and
+ generic open source audio DSP firmware for multiple devices.
+ Say Y if you have such a device that is supported by SOF.
+ If unsure select "N".
+
+if SND_SOC_SOF_TOPLEVEL
+
+config SND_SOC_SOF_PCI
+ tristate "SOF PCI enumeration support"
+ depends on PCI
+ select SND_SOC_SOF
+ select SND_SOC_ACPI if ACPI
+ select SND_SOC_SOF_OPTIONS
+ select SND_SOC_SOF_INTEL_PCI if SND_SOC_SOF_INTEL_TOPLEVEL
+ help
+ This adds support for PCI enumeration. This option is
+ required to enable Intel Skylake+ devices
+ Say Y if you need this option
+ If unsure select "N".
+
+config SND_SOC_SOF_ACPI
+ tristate "SOF ACPI enumeration support"
+ depends on ACPI || COMPILE_TEST
+ select SND_SOC_SOF
+ select SND_SOC_ACPI if ACPI
+ select SND_SOC_SOF_OPTIONS
+ select SND_SOC_SOF_INTEL_ACPI if SND_SOC_SOF_INTEL_TOPLEVEL
+ select IOSF_MBI if X86 && PCI
+ help
+ This adds support for ACPI enumeration. This option is required
+ to enable Intel Haswell/Broadwell/Baytrail/Cherrytrail devices
+ Say Y if you need this option
+ If unsure select "N".
+
+config SND_SOC_SOF_OPTIONS
+ tristate
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+if SND_SOC_SOF_OPTIONS
+
+config SND_SOC_SOF_NOCODEC
+ tristate "SOF nocodec mode Support"
+ help
+ This adds support for a dummy/nocodec machine driver fallback
+ option if no known codec is detected. This is typically only
+ enabled for developers or devices where the sound card is
+ controlled externally
+ Say Y if you need this nocodec fallback option
+ If unsure select "N".
+
+config SND_SOC_SOF_STRICT_ABI_CHECKS
+ bool "SOF strict ABI checks"
+ help
+ This option enables strict ABI checks for firmware and topology
+ files.
+ When these files are more recent than the kernel, the kernel
+ will handle the functionality it supports and may report errors
+ during topology creation or run-time usage if new functionality
+ is invoked.
+ This option will stop topology creation and firmware load upfront.
+ It is intended for SOF CI/releases and not for users or distros.
+ Say Y if you want strict ABI checks for an SOF release
+ If you are not involved in SOF releases and CI development
+ select "N".
+
+config SND_SOC_SOF_DEBUG
+ bool "SOF debugging features"
+ help
+ This option can be used to enable or disable individual SOF firmware
+ and driver debugging options.
+ Say Y if you are debugging SOF FW or drivers.
+ If unsure select "N".
+
+if SND_SOC_SOF_DEBUG
+
+config SND_SOC_SOF_FORCE_NOCODEC_MODE
+ bool "SOF force nocodec Mode"
+ depends on SND_SOC_SOF_NOCODEC
+ help
+ This forces SOF to use dummy/nocodec as machine driver, even
+ though there is a codec detected on the real platform. This is
+ typically only enabled for developers for debug purposes, before
+ codec/machine driver is ready, or to exclude the impact of those
+ drivers
+ Say Y if you need this force nocodec mode option
+ If unsure select "N".
+
+config SND_SOC_SOF_DEBUG_XRUN_STOP
+ bool "SOF stop on XRUN"
+ help
+ This option forces PCMs to stop on any XRUN event. This is useful to
+ preserve any trace data ond pipeline status prior to the XRUN.
+ Say Y if you are debugging SOF FW pipeline XRUNs.
+ If unsure select "N".
+
+config SND_SOC_SOF_DEBUG_VERBOSE_IPC
+ bool "SOF verbose IPC logs"
+ help
+ This option enables more verbose IPC logs, with command types in
+ human-readable form instead of just 32-bit hex dumps. This is useful
+ if you are trying to debug IPC with the DSP firmware.
+ If unsure select "N".
+
+config SND_SOC_SOF_DEBUG_FORCE_IPC_POSITION
+ bool "SOF force to use IPC for position update on SKL+"
+ help
+ This option force to handle stream position update IPCs and run pcm
+ elapse to inform ALSA about that, on platforms (e.g. Intel SKL+) that
+ with other approach (e.g. HDAC DPIB/posbuf) to elapse PCM.
+ On platforms (e.g. Intel SKL-) where position update IPC is the only
+ one choice, this setting won't impact anything.
+ if you are trying to debug pointer update with position IPCs or where
+ DPIB/posbuf is not ready, select "Y".
+ If unsure select "N".
+
+config SND_SOC_SOF_DEBUG_ENABLE_DEBUGFS_CACHE
+ bool "SOF enable debugfs caching"
+ help
+ This option enables caching of debugfs
+ memory -> DSP resource (memory, register, etc)
+ before the audio DSP is suspended. This will increase the suspend
+ latency and therefore should be used for debug purposes only.
+ Say Y if you want to enable caching the memory windows.
+ If unsure, select "N".
+
+endif ## SND_SOC_SOF_DEBUG
+
+endif ## SND_SOC_SOF_OPTIONS
+
+config SND_SOC_SOF
+ tristate
+ select SND_SOC_TOPOLOGY
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+ The selection is made at the top level and does not exactly follow
+ module dependencies but since the module or built-in type is decided
+ at the top level it doesn't matter.
+
+config SND_SOC_SOF_PROBE_WORK_QUEUE
+ bool
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+ When selected, the probe is handled in two steps, for example to
+ avoid lockdeps if request_module is used in the probe.
+
+source "sound/soc/sof/intel/Kconfig"
+source "sound/soc/sof/xtensa/Kconfig"
+
+endif
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+
+snd-sof-objs := core.o ops.o loader.o ipc.o pcm.o pm.o debug.o topology.o\
+ control.o trace.o utils.o
+
+snd-sof-pci-objs := sof-pci-dev.o
+snd-sof-acpi-objs := sof-acpi-dev.o
+snd-sof-nocodec-objs := nocodec.o
+
+obj-$(CONFIG_SND_SOC_SOF) += snd-sof.o
+obj-$(CONFIG_SND_SOC_SOF_NOCODEC) += snd-sof-nocodec.o
+
+
+obj-$(CONFIG_SND_SOC_SOF_ACPI) += sof-acpi-dev.o
+obj-$(CONFIG_SND_SOC_SOF_PCI) += sof-pci-dev.o
+
+obj-$(CONFIG_SND_SOC_SOF_INTEL_TOPLEVEL) += intel/
+obj-$(CONFIG_SND_SOC_SOF_XTENSA) += xtensa/
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+/* Mixer Controls */
+
+#include <linux/pm_runtime.h>
+#include "sof-priv.h"
+
+static inline u32 mixer_to_ipc(unsigned int value, u32 *volume_map, int size)
+{
+ if (value >= size)
+ return volume_map[size - 1];
+
+ return volume_map[value];
+}
+
+static inline u32 ipc_to_mixer(u32 value, u32 *volume_map, int size)
+{
+ int i;
+
+ for (i = 0; i < size; i++) {
+ if (volume_map[i] >= value)
+ return i;
+ }
+
+ return i - 1;
+}
+
+int snd_sof_volume_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct soc_mixer_control *sm =
+ (struct soc_mixer_control *)kcontrol->private_value;
+ struct snd_sof_control *scontrol = sm->dobj.private;
+ struct snd_sof_dev *sdev = scontrol->sdev;
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ unsigned int i, channels = scontrol->num_channels;
+ int err, ret;
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err_ratelimited(sdev->dev,
+ "error: volume get failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /* get all the mixer data from DSP */
+ snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ SOF_IPC_COMP_GET_VALUE,
+ SOF_CTRL_TYPE_VALUE_CHAN_GET,
+ SOF_CTRL_CMD_VOLUME,
+ false);
+
+ /* read back each channel */
+ for (i = 0; i < channels; i++)
+ ucontrol->value.integer.value[i] =
+ ipc_to_mixer(cdata->chanv[i].value,
+ scontrol->volume_table, sm->max + 1);
+
+ pm_runtime_mark_last_busy(sdev->dev);
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err_ratelimited(sdev->dev,
+ "error: volume get failed to idle %d\n",
+ err);
+ return 0;
+}
+
+int snd_sof_volume_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct soc_mixer_control *sm =
+ (struct soc_mixer_control *)kcontrol->private_value;
+ struct snd_sof_control *scontrol = sm->dobj.private;
+ struct snd_sof_dev *sdev = scontrol->sdev;
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ unsigned int i, channels = scontrol->num_channels;
+ int ret, err;
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err_ratelimited(sdev->dev,
+ "error: volume put failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /* update each channel */
+ for (i = 0; i < channels; i++) {
+ cdata->chanv[i].value =
+ mixer_to_ipc(ucontrol->value.integer.value[i],
+ scontrol->volume_table, sm->max + 1);
+ cdata->chanv[i].channel = i;
+ }
+
+ /* notify DSP of mixer updates */
+ snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ SOF_IPC_COMP_SET_VALUE,
+ SOF_CTRL_TYPE_VALUE_CHAN_GET,
+ SOF_CTRL_CMD_VOLUME,
+ true);
+
+ pm_runtime_mark_last_busy(sdev->dev);
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err_ratelimited(sdev->dev,
+ "error: volume put failed to idle %d\n",
+ err);
+ return 0;
+}
+
+int snd_sof_switch_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct soc_mixer_control *sm =
+ (struct soc_mixer_control *)kcontrol->private_value;
+ struct snd_sof_control *scontrol = sm->dobj.private;
+ struct snd_sof_dev *sdev = scontrol->sdev;
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ unsigned int i, channels = scontrol->num_channels;
+ int err, ret;
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err_ratelimited(sdev->dev,
+ "error: switch get failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /* get all the mixer data from DSP */
+ snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ SOF_IPC_COMP_GET_VALUE,
+ SOF_CTRL_TYPE_VALUE_CHAN_GET,
+ SOF_CTRL_CMD_SWITCH,
+ false);
+
+ /* read back each channel */
+ for (i = 0; i < channels; i++)
+ ucontrol->value.integer.value[i] = cdata->chanv[i].value;
+
+ pm_runtime_mark_last_busy(sdev->dev);
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err_ratelimited(sdev->dev,
+ "error: switch get failed to idle %d\n",
+ err);
+ return 0;
+}
+
+int snd_sof_switch_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct soc_mixer_control *sm =
+ (struct soc_mixer_control *)kcontrol->private_value;
+ struct snd_sof_control *scontrol = sm->dobj.private;
+ struct snd_sof_dev *sdev = scontrol->sdev;
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ unsigned int i, channels = scontrol->num_channels;
+ int ret, err;
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err_ratelimited(sdev->dev,
+ "error: switch put failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /* update each channel */
+ for (i = 0; i < channels; i++) {
+ cdata->chanv[i].value = ucontrol->value.integer.value[i];
+ cdata->chanv[i].channel = i;
+ }
+
+ /* notify DSP of mixer updates */
+ snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ SOF_IPC_COMP_SET_VALUE,
+ SOF_CTRL_TYPE_VALUE_CHAN_GET,
+ SOF_CTRL_CMD_SWITCH,
+ true);
+
+ pm_runtime_mark_last_busy(sdev->dev);
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err_ratelimited(sdev->dev,
+ "error: switch put failed to idle %d\n",
+ err);
+ return 0;
+}
+
+int snd_sof_enum_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct soc_enum *se =
+ (struct soc_enum *)kcontrol->private_value;
+ struct snd_sof_control *scontrol = se->dobj.private;
+ struct snd_sof_dev *sdev = scontrol->sdev;
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ unsigned int i, channels = scontrol->num_channels;
+ int err, ret;
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err_ratelimited(sdev->dev,
+ "error: enum get failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /* get all the enum data from DSP */
+ snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ SOF_IPC_COMP_GET_VALUE,
+ SOF_CTRL_TYPE_VALUE_CHAN_GET,
+ SOF_CTRL_CMD_ENUM,
+ false);
+
+ /* read back each channel */
+ for (i = 0; i < channels; i++)
+ ucontrol->value.enumerated.item[i] = cdata->chanv[i].value;
+
+ pm_runtime_mark_last_busy(sdev->dev);
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err_ratelimited(sdev->dev,
+ "error: enum get failed to idle %d\n",
+ err);
+ return 0;
+}
+
+int snd_sof_enum_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct soc_enum *se =
+ (struct soc_enum *)kcontrol->private_value;
+ struct snd_sof_control *scontrol = se->dobj.private;
+ struct snd_sof_dev *sdev = scontrol->sdev;
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ unsigned int i, channels = scontrol->num_channels;
+ int ret, err;
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err_ratelimited(sdev->dev,
+ "error: enum put failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /* update each channel */
+ for (i = 0; i < channels; i++) {
+ cdata->chanv[i].value = ucontrol->value.enumerated.item[i];
+ cdata->chanv[i].channel = i;
+ }
+
+ /* notify DSP of enum updates */
+ snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ SOF_IPC_COMP_SET_VALUE,
+ SOF_CTRL_TYPE_VALUE_CHAN_GET,
+ SOF_CTRL_CMD_ENUM,
+ true);
+
+ pm_runtime_mark_last_busy(sdev->dev);
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err_ratelimited(sdev->dev,
+ "error: enum put failed to idle %d\n",
+ err);
+ return 0;
+}
+
+int snd_sof_bytes_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct soc_bytes_ext *be =
+ (struct soc_bytes_ext *)kcontrol->private_value;
+ struct snd_sof_control *scontrol = be->dobj.private;
+ struct snd_sof_dev *sdev = scontrol->sdev;
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ struct sof_abi_hdr *data = cdata->data;
+ size_t size;
+ int ret, err;
+
+ if (be->max > sizeof(ucontrol->value.bytes.data)) {
+ dev_err_ratelimited(sdev->dev,
+ "error: data max %d exceeds ucontrol data array size\n",
+ be->max);
+ return -EINVAL;
+ }
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err_ratelimited(sdev->dev,
+ "error: bytes get failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /* get all the binary data from DSP */
+ snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ SOF_IPC_COMP_GET_DATA,
+ SOF_CTRL_TYPE_DATA_GET,
+ scontrol->cmd,
+ false);
+
+ size = data->size + sizeof(*data);
+ if (size > be->max) {
+ dev_err_ratelimited(sdev->dev,
+ "error: DSP sent %zu bytes max is %d\n",
+ size, be->max);
+ ret = -EINVAL;
+ goto out;
+ }
+
+ /* copy back to kcontrol */
+ memcpy(ucontrol->value.bytes.data, data, size);
+
+out:
+ pm_runtime_mark_last_busy(sdev->dev);
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err_ratelimited(sdev->dev,
+ "error: bytes get failed to idle %d\n",
+ err);
+ return ret;
+}
+
+int snd_sof_bytes_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct soc_bytes_ext *be =
+ (struct soc_bytes_ext *)kcontrol->private_value;
+ struct snd_sof_control *scontrol = be->dobj.private;
+ struct snd_sof_dev *sdev = scontrol->sdev;
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ struct sof_abi_hdr *data = cdata->data;
+ int ret, err;
+
+ if (be->max > sizeof(ucontrol->value.bytes.data)) {
+ dev_err_ratelimited(sdev->dev,
+ "error: data max %d exceeds ucontrol data array size\n",
+ be->max);
+ return -EINVAL;
+ }
+
+ if (data->size > be->max) {
+ dev_err_ratelimited(sdev->dev,
+ "error: size too big %d bytes max is %d\n",
+ data->size, be->max);
+ return -EINVAL;
+ }
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err_ratelimited(sdev->dev,
+ "error: bytes put failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /* copy from kcontrol */
+ memcpy(data, ucontrol->value.bytes.data, data->size);
+
+ /* notify DSP of byte control updates */
+ snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ SOF_IPC_COMP_SET_DATA,
+ SOF_CTRL_TYPE_DATA_SET,
+ scontrol->cmd,
+ true);
+
+ pm_runtime_mark_last_busy(sdev->dev);
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err_ratelimited(sdev->dev,
+ "error: bytes put failed to idle %d\n",
+ err);
+ return ret;
+}
+
+int snd_sof_bytes_ext_put(struct snd_kcontrol *kcontrol,
+ const unsigned int __user *binary_data,
+ unsigned int size)
+{
+ struct soc_bytes_ext *be =
+ (struct soc_bytes_ext *)kcontrol->private_value;
+ struct snd_sof_control *scontrol = be->dobj.private;
+ struct snd_sof_dev *sdev = scontrol->sdev;
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ struct snd_ctl_tlv header;
+ const struct snd_ctl_tlv __user *tlvd =
+ (const struct snd_ctl_tlv __user *)binary_data;
+ int ret;
+ int err;
+
+ /*
+ * The beginning of bytes data contains a header from where
+ * the length (as bytes) is needed to know the correct copy
+ * length of data from tlvd->tlv.
+ */
+ if (copy_from_user(&header, tlvd, sizeof(const struct snd_ctl_tlv)))
+ return -EFAULT;
+
+ /* be->max is coming from topology */
+ if (header.length > be->max) {
+ dev_err_ratelimited(sdev->dev, "error: Bytes data size %d exceeds max %d.\n",
+ header.length, be->max);
+ return -EINVAL;
+ }
+
+ /* Check that header id matches the command */
+ if (header.numid != scontrol->cmd) {
+ dev_err_ratelimited(sdev->dev,
+ "error: incorrect numid %d\n",
+ header.numid);
+ return -EINVAL;
+ }
+
+ if (copy_from_user(cdata->data, tlvd->tlv, header.length))
+ return -EFAULT;
+
+ if (cdata->data->magic != SOF_ABI_MAGIC) {
+ dev_err_ratelimited(sdev->dev,
+ "error: Wrong ABI magic 0x%08x.\n",
+ cdata->data->magic);
+ return -EINVAL;
+ }
+
+ if (SOF_ABI_VERSION_INCOMPATIBLE(SOF_ABI_VERSION, cdata->data->abi)) {
+ dev_err_ratelimited(sdev->dev, "error: Incompatible ABI version 0x%08x.\n",
+ cdata->data->abi);
+ return -EINVAL;
+ }
+
+ if (cdata->data->size + sizeof(const struct sof_abi_hdr) > be->max) {
+ dev_err_ratelimited(sdev->dev, "error: Mismatch in ABI data size (truncated?).\n");
+ return -EINVAL;
+ }
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err_ratelimited(sdev->dev,
+ "error: bytes_ext put failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /* notify DSP of byte control updates */
+ snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ SOF_IPC_COMP_SET_DATA,
+ SOF_CTRL_TYPE_DATA_SET,
+ scontrol->cmd,
+ true);
+
+ pm_runtime_mark_last_busy(sdev->dev);
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err_ratelimited(sdev->dev,
+ "error: bytes_ext put failed to idle %d\n",
+ err);
+
+ return ret;
+}
+
+int snd_sof_bytes_ext_get(struct snd_kcontrol *kcontrol,
+ unsigned int __user *binary_data,
+ unsigned int size)
+{
+ struct soc_bytes_ext *be =
+ (struct soc_bytes_ext *)kcontrol->private_value;
+ struct snd_sof_control *scontrol = be->dobj.private;
+ struct snd_sof_dev *sdev = scontrol->sdev;
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ struct snd_ctl_tlv header;
+ struct snd_ctl_tlv __user *tlvd =
+ (struct snd_ctl_tlv __user *)binary_data;
+ int data_size;
+ int err;
+ int ret;
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err_ratelimited(sdev->dev,
+ "error: bytes_ext get failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /*
+ * Decrement the limit by ext bytes header size to
+ * ensure the user space buffer is not exceeded.
+ */
+ size -= sizeof(const struct snd_ctl_tlv);
+
+ /* set the ABI header values */
+ cdata->data->magic = SOF_ABI_MAGIC;
+ cdata->data->abi = SOF_ABI_VERSION;
+
+ /* get all the component data from DSP */
+ ret = snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ SOF_IPC_COMP_GET_DATA,
+ SOF_CTRL_TYPE_DATA_GET,
+ scontrol->cmd,
+ false);
+
+ /* Prevent read of other kernel data or possibly corrupt response */
+ data_size = cdata->data->size + sizeof(const struct sof_abi_hdr);
+
+ /* check data size doesn't exceed max coming from topology */
+ if (data_size > be->max) {
+ dev_err_ratelimited(sdev->dev, "error: user data size %d exceeds max size %d.\n",
+ data_size, be->max);
+ ret = -EINVAL;
+ goto out;
+ }
+
+ header.numid = scontrol->cmd;
+ header.length = data_size;
+ if (copy_to_user(tlvd, &header, sizeof(const struct snd_ctl_tlv))) {
+ ret = -EFAULT;
+ goto out;
+ }
+
+ if (copy_to_user(tlvd->tlv, cdata->data, data_size))
+ ret = -EFAULT;
+
+out:
+ pm_runtime_mark_last_busy(sdev->dev);
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err_ratelimited(sdev->dev,
+ "error: bytes_ext get failed to idle %d\n",
+ err);
+ return ret;
+}
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+#include <linux/firmware.h>
+#include <linux/module.h>
+#include <asm/unaligned.h>
+#include <sound/soc.h>
+#include <sound/sof.h>
+#include "sof-priv.h"
+#include "ops.h"
+
+/* SOF defaults if not provided by the platform in ms */
+#define TIMEOUT_DEFAULT_IPC_MS 5
+#define TIMEOUT_DEFAULT_BOOT_MS 100
+
+/*
+ * Generic object lookup APIs.
+ */
+
+struct snd_sof_pcm *snd_sof_find_spcm_name(struct snd_sof_dev *sdev,
+ const char *name)
+{
+ struct snd_sof_pcm *spcm;
+
+ list_for_each_entry(spcm, &sdev->pcm_list, list) {
+ /* match with PCM dai name */
+ if (strcmp(spcm->pcm.dai_name, name) == 0)
+ return spcm;
+
+ /* match with playback caps name if set */
+ if (*spcm->pcm.caps[0].name &&
+ !strcmp(spcm->pcm.caps[0].name, name))
+ return spcm;
+
+ /* match with capture caps name if set */
+ if (*spcm->pcm.caps[1].name &&
+ !strcmp(spcm->pcm.caps[1].name, name))
+ return spcm;
+ }
+
+ return NULL;
+}
+
+struct snd_sof_pcm *snd_sof_find_spcm_comp(struct snd_sof_dev *sdev,
+ unsigned int comp_id,
+ int *direction)
+{
+ struct snd_sof_pcm *spcm;
+
+ list_for_each_entry(spcm, &sdev->pcm_list, list) {
+ if (spcm->stream[SNDRV_PCM_STREAM_PLAYBACK].comp_id == comp_id) {
+ *direction = SNDRV_PCM_STREAM_PLAYBACK;
+ return spcm;
+ }
+ if (spcm->stream[SNDRV_PCM_STREAM_CAPTURE].comp_id == comp_id) {
+ *direction = SNDRV_PCM_STREAM_CAPTURE;
+ return spcm;
+ }
+ }
+
+ return NULL;
+}
+
+struct snd_sof_pcm *snd_sof_find_spcm_pcm_id(struct snd_sof_dev *sdev,
+ unsigned int pcm_id)
+{
+ struct snd_sof_pcm *spcm;
+
+ list_for_each_entry(spcm, &sdev->pcm_list, list) {
+ if (le32_to_cpu(spcm->pcm.pcm_id) == pcm_id)
+ return spcm;
+ }
+
+ return NULL;
+}
+
+struct snd_sof_widget *snd_sof_find_swidget(struct snd_sof_dev *sdev,
+ const char *name)
+{
+ struct snd_sof_widget *swidget;
+
+ list_for_each_entry(swidget, &sdev->widget_list, list) {
+ if (strcmp(name, swidget->widget->name) == 0)
+ return swidget;
+ }
+
+ return NULL;
+}
+
+/* find widget by stream name and direction */
+struct snd_sof_widget *snd_sof_find_swidget_sname(struct snd_sof_dev *sdev,
+ const char *pcm_name, int dir)
+{
+ struct snd_sof_widget *swidget;
+ enum snd_soc_dapm_type type;
+
+ if (dir == SNDRV_PCM_STREAM_PLAYBACK)
+ type = snd_soc_dapm_aif_in;
+ else
+ type = snd_soc_dapm_aif_out;
+
+ list_for_each_entry(swidget, &sdev->widget_list, list) {
+ if (!strcmp(pcm_name, swidget->widget->sname) && swidget->id == type)
+ return swidget;
+ }
+
+ return NULL;
+}
+
+struct snd_sof_dai *snd_sof_find_dai(struct snd_sof_dev *sdev,
+ const char *name)
+{
+ struct snd_sof_dai *dai;
+
+ list_for_each_entry(dai, &sdev->dai_list, list) {
+ if (dai->name && (strcmp(name, dai->name) == 0))
+ return dai;
+ }
+
+ return NULL;
+}
+
+/*
+ * FW Panic/fault handling.
+ */
+
+struct sof_panic_msg {
+ u32 id;
+ const char *msg;
+};
+
+/* standard FW panic types */
+static const struct sof_panic_msg panic_msg[] = {
+ {SOF_IPC_PANIC_MEM, "out of memory"},
+ {SOF_IPC_PANIC_WORK, "work subsystem init failed"},
+ {SOF_IPC_PANIC_IPC, "IPC subsystem init failed"},
+ {SOF_IPC_PANIC_ARCH, "arch init failed"},
+ {SOF_IPC_PANIC_PLATFORM, "platform init failed"},
+ {SOF_IPC_PANIC_TASK, "scheduler init failed"},
+ {SOF_IPC_PANIC_EXCEPTION, "runtime exception"},
+ {SOF_IPC_PANIC_DEADLOCK, "deadlock"},
+ {SOF_IPC_PANIC_STACK, "stack overflow"},
+ {SOF_IPC_PANIC_IDLE, "can't enter idle"},
+ {SOF_IPC_PANIC_WFI, "invalid wait state"},
+ {SOF_IPC_PANIC_ASSERT, "assertion failed"},
+};
+
+/*
+ * helper to be called from .dbg_dump callbacks. No error code is
+ * provided, it's left as an exercise for the caller of .dbg_dump
+ * (typically IPC or loader)
+ */
+void snd_sof_get_status(struct snd_sof_dev *sdev, u32 panic_code,
+ u32 tracep_code, void *oops,
+ struct sof_ipc_panic_info *panic_info,
+ void *stack, size_t stack_words)
+{
+ u32 code;
+ int i;
+
+ /* is firmware dead ? */
+ if ((panic_code & SOF_IPC_PANIC_MAGIC_MASK) != SOF_IPC_PANIC_MAGIC) {
+ dev_err(sdev->dev, "error: unexpected fault 0x%8.8x trace 0x%8.8x\n",
+ panic_code, tracep_code);
+ return; /* no fault ? */
+ }
+
+ code = panic_code & (SOF_IPC_PANIC_MAGIC_MASK | SOF_IPC_PANIC_CODE_MASK);
+
+ for (i = 0; i < ARRAY_SIZE(panic_msg); i++) {
+ if (panic_msg[i].id == code) {
+ dev_err(sdev->dev, "error: %s\n", panic_msg[i].msg);
+ dev_err(sdev->dev, "error: trace point %8.8x\n",
+ tracep_code);
+ goto out;
+ }
+ }
+
+ /* unknown error */
+ dev_err(sdev->dev, "error: unknown reason %8.8x\n", panic_code);
+ dev_err(sdev->dev, "error: trace point %8.8x\n", tracep_code);
+
+out:
+ dev_err(sdev->dev, "error: panic at %s:%d\n",
+ panic_info->filename, panic_info->linenum);
+ sof_oops(sdev, oops);
+ sof_stack(sdev, oops, stack, stack_words);
+}
+EXPORT_SYMBOL(snd_sof_get_status);
+
+/*
+ * Generic buffer page table creation.
+ * Take the each physical page address and drop the least significant unused
+ * bits from each (based on PAGE_SIZE). Then pack valid page address bits
+ * into compressed page table.
+ */
+
+int snd_sof_create_page_table(struct snd_sof_dev *sdev,
+ struct snd_dma_buffer *dmab,
+ unsigned char *page_table, size_t size)
+{
+ int i, pages;
+
+ pages = snd_sgbuf_aligned_pages(size);
+
+ dev_dbg(sdev->dev, "generating page table for %p size 0x%zx pages %d\n",
+ dmab->area, size, pages);
+
+ for (i = 0; i < pages; i++) {
+ /*
+ * The number of valid address bits for each page is 20.
+ * idx determines the byte position within page_table
+ * where the current page's address is stored
+ * in the compressed page_table.
+ * This can be calculated by multiplying the page number by 2.5.
+ */
+ u32 idx = (5 * i) >> 1;
+ u32 pfn = snd_sgbuf_get_addr(dmab, i * PAGE_SIZE) >> PAGE_SHIFT;
+ u8 *pg_table;
+
+ dev_vdbg(sdev->dev, "pfn i %i idx %d pfn %x\n", i, idx, pfn);
+
+ pg_table = (u8 *)(page_table + idx);
+
+ /*
+ * pagetable compression:
+ * byte 0 byte 1 byte 2 byte 3 byte 4 byte 5
+ * ___________pfn 0__________ __________pfn 1___________ _pfn 2...
+ * .... .... .... .... .... .... .... .... .... .... ....
+ * It is created by:
+ * 1. set current location to 0, PFN index i to 0
+ * 2. put pfn[i] at current location in Little Endian byte order
+ * 3. calculate an intermediate value as
+ * x = (pfn[i+1] << 4) | (pfn[i] & 0xf)
+ * 4. put x at offset (current location + 2) in LE byte order
+ * 5. increment current location by 5 bytes, increment i by 2
+ * 6. continue to (2)
+ */
+ if (i & 1)
+ put_unaligned_le32((pg_table[0] & 0xf) | pfn << 4,
+ pg_table);
+ else
+ put_unaligned_le32(pfn, pg_table);
+ }
+
+ return pages;
+}
+
+/*
+ * SOF Driver enumeration.
+ */
+static int sof_machine_check(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_pdata *plat_data = sdev->pdata;
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_NOCODEC)
+ struct snd_soc_acpi_mach *machine;
+ int ret;
+#endif
+
+ if (plat_data->machine)
+ return 0;
+
+#if !IS_ENABLED(CONFIG_SND_SOC_SOF_NOCODEC)
+ dev_err(sdev->dev, "error: no matching ASoC machine driver found - aborting probe\n");
+ return -ENODEV;
+#else
+ /* fallback to nocodec mode */
+ dev_warn(sdev->dev, "No ASoC machine driver found - using nocodec\n");
+ machine = devm_kzalloc(sdev->dev, sizeof(*machine), GFP_KERNEL);
+ if (!machine)
+ return -ENOMEM;
+
+ ret = sof_nocodec_setup(sdev->dev, plat_data, machine,
+ plat_data->desc, plat_data->desc->ops);
+ if (ret < 0)
+ return ret;
+
+ plat_data->machine = machine;
+
+ return 0;
+#endif
+}
+
+static int sof_probe_continue(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_pdata *plat_data = sdev->pdata;
+ const char *drv_name;
+ const void *mach;
+ int size;
+ int ret;
+
+ /* probe the DSP hardware */
+ ret = snd_sof_probe(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to probe DSP %d\n", ret);
+ return ret;
+ }
+
+ /* check machine info */
+ ret = sof_machine_check(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to get machine info %d\n",
+ ret);
+ goto dbg_err;
+ }
+
+ /* set up platform component driver */
+ snd_sof_new_platform_drv(sdev);
+
+ /* register any debug/trace capabilities */
+ ret = snd_sof_dbg_init(sdev);
+ if (ret < 0) {
+ /*
+ * debugfs issues are suppressed in snd_sof_dbg_init() since
+ * we cannot rely on debugfs
+ * here we trap errors due to memory allocation only.
+ */
+ dev_err(sdev->dev, "error: failed to init DSP trace/debug %d\n",
+ ret);
+ goto dbg_err;
+ }
+
+ /* init the IPC */
+ sdev->ipc = snd_sof_ipc_init(sdev);
+ if (!sdev->ipc) {
+ dev_err(sdev->dev, "error: failed to init DSP IPC %d\n", ret);
+ goto ipc_err;
+ }
+
+ /* load the firmware */
+ ret = snd_sof_load_firmware(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to load DSP firmware %d\n",
+ ret);
+ goto fw_load_err;
+ }
+
+ /* boot the firmware */
+ ret = snd_sof_run_firmware(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to boot DSP firmware %d\n",
+ ret);
+ goto fw_run_err;
+ }
+
+ /* init DMA trace */
+ ret = snd_sof_init_trace(sdev);
+ if (ret < 0) {
+ /* non fatal */
+ dev_warn(sdev->dev,
+ "warning: failed to initialize trace %d\n", ret);
+ }
+
+ /* hereafter all FW boot flows are for PM reasons */
+ sdev->first_boot = false;
+
+ /* now register audio DSP platform driver and dai */
+ ret = devm_snd_soc_register_component(sdev->dev, &sdev->plat_drv,
+ sof_ops(sdev)->drv,
+ sof_ops(sdev)->num_drv);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to register DSP DAI driver %d\n", ret);
+ goto fw_run_err;
+ }
+
+ drv_name = plat_data->machine->drv_name;
+ mach = (const void *)plat_data->machine;
+ size = sizeof(*plat_data->machine);
+
+ /* register machine driver, pass machine info as pdata */
+ plat_data->pdev_mach =
+ platform_device_register_data(sdev->dev, drv_name,
+ PLATFORM_DEVID_NONE, mach, size);
+
+ if (IS_ERR(plat_data->pdev_mach)) {
+ ret = PTR_ERR(plat_data->pdev_mach);
+ goto comp_err;
+ }
+
+ dev_dbg(sdev->dev, "created machine %s\n",
+ dev_name(&plat_data->pdev_mach->dev));
+
+ if (plat_data->sof_probe_complete)
+ plat_data->sof_probe_complete(sdev->dev);
+
+ return 0;
+
+comp_err:
+ snd_soc_unregister_component(sdev->dev);
+fw_run_err:
+ snd_sof_fw_unload(sdev);
+fw_load_err:
+ snd_sof_ipc_free(sdev);
+ipc_err:
+ snd_sof_free_debug(sdev);
+dbg_err:
+ snd_sof_remove(sdev);
+
+ return ret;
+}
+
+static void sof_probe_work(struct work_struct *work)
+{
+ struct snd_sof_dev *sdev =
+ container_of(work, struct snd_sof_dev, probe_work);
+ int ret;
+
+ ret = sof_probe_continue(sdev);
+ if (ret < 0) {
+ /* errors cannot be propagated, log */
+ dev_err(sdev->dev, "error: %s failed err: %d\n", __func__, ret);
+ }
+}
+
+int snd_sof_device_probe(struct device *dev, struct snd_sof_pdata *plat_data)
+{
+ struct snd_sof_dev *sdev;
+
+ sdev = devm_kzalloc(dev, sizeof(*sdev), GFP_KERNEL);
+ if (!sdev)
+ return -ENOMEM;
+
+ /* initialize sof device */
+ sdev->dev = dev;
+
+ sdev->pdata = plat_data;
+ sdev->first_boot = true;
+ dev_set_drvdata(dev, sdev);
+
+ /* check all mandatory ops */
+ if (!sof_ops(sdev) || !sof_ops(sdev)->probe || !sof_ops(sdev)->run ||
+ !sof_ops(sdev)->block_read || !sof_ops(sdev)->block_write ||
+ !sof_ops(sdev)->send_msg || !sof_ops(sdev)->load_firmware ||
+ !sof_ops(sdev)->ipc_msg_data || !sof_ops(sdev)->ipc_pcm_params)
+ return -EINVAL;
+
+ INIT_LIST_HEAD(&sdev->pcm_list);
+ INIT_LIST_HEAD(&sdev->kcontrol_list);
+ INIT_LIST_HEAD(&sdev->widget_list);
+ INIT_LIST_HEAD(&sdev->dai_list);
+ INIT_LIST_HEAD(&sdev->route_list);
+ spin_lock_init(&sdev->ipc_lock);
+ spin_lock_init(&sdev->hw_lock);
+
+ if (IS_ENABLED(CONFIG_SND_SOC_SOF_PROBE_WORK_QUEUE))
+ INIT_WORK(&sdev->probe_work, sof_probe_work);
+
+ /* set default timeouts if none provided */
+ if (plat_data->desc->ipc_timeout == 0)
+ sdev->ipc_timeout = TIMEOUT_DEFAULT_IPC_MS;
+ else
+ sdev->ipc_timeout = plat_data->desc->ipc_timeout;
+ if (plat_data->desc->boot_timeout == 0)
+ sdev->boot_timeout = TIMEOUT_DEFAULT_BOOT_MS;
+ else
+ sdev->boot_timeout = plat_data->desc->boot_timeout;
+
+ if (IS_ENABLED(CONFIG_SND_SOC_SOF_PROBE_WORK_QUEUE)) {
+ schedule_work(&sdev->probe_work);
+ return 0;
+ }
+
+ return sof_probe_continue(sdev);
+}
+EXPORT_SYMBOL(snd_sof_device_probe);
+
+int snd_sof_device_remove(struct device *dev)
+{
+ struct snd_sof_dev *sdev = dev_get_drvdata(dev);
+ struct snd_sof_pdata *pdata = sdev->pdata;
+
+ if (IS_ENABLED(CONFIG_SND_SOC_SOF_PROBE_WORK_QUEUE))
+ cancel_work_sync(&sdev->probe_work);
+
+ snd_sof_fw_unload(sdev);
+ snd_sof_ipc_free(sdev);
+ snd_sof_free_debug(sdev);
+ snd_sof_free_trace(sdev);
+ snd_sof_remove(sdev);
+
+ /*
+ * Unregister machine driver. This will unbind the snd_card which
+ * will remove the component driver and unload the topology
+ * before freeing the snd_card.
+ */
+ if (!IS_ERR_OR_NULL(pdata->pdev_mach))
+ platform_device_unregister(pdata->pdev_mach);
+
+ /* release firmware */
+ release_firmware(pdata->fw);
+ pdata->fw = NULL;
+
+ return 0;
+}
+EXPORT_SYMBOL(snd_sof_device_remove);
+
+MODULE_AUTHOR("Liam Girdwood");
+MODULE_DESCRIPTION("Sound Open Firmware (SOF) Core");
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_ALIAS("platform:sof-audio");
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+// Generic debug routines used to export DSP MMIO and memories to userspace
+// for firmware debugging.
+//
+
+#include <linux/debugfs.h>
+#include <linux/io.h>
+#include <linux/pm_runtime.h>
+#include "sof-priv.h"
+#include "ops.h"
+
+static ssize_t sof_dfsentry_read(struct file *file, char __user *buffer,
+ size_t count, loff_t *ppos)
+{
+ struct snd_sof_dfsentry *dfse = file->private_data;
+ struct snd_sof_dev *sdev = dfse->sdev;
+ loff_t pos = *ppos;
+ size_t size_ret;
+ int skip = 0;
+ int size;
+ u8 *buf;
+
+ size = dfse->size;
+
+ /* validate position & count */
+ if (pos < 0)
+ return -EINVAL;
+ if (pos >= size || !count)
+ return 0;
+ /* find the minimum. min() is not used since it adds sparse warnings */
+ if (count > size - pos)
+ count = size - pos;
+
+ /* align io read start to u32 multiple */
+ pos = ALIGN_DOWN(pos, 4);
+
+ /* intermediate buffer size must be u32 multiple */
+ size = ALIGN(count, 4);
+
+ /* if start position is unaligned, read extra u32 */
+ if (unlikely(pos != *ppos)) {
+ skip = *ppos - pos;
+ if (pos + size + 4 < dfse->size)
+ size += 4;
+ }
+
+ buf = kzalloc(size, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ if (dfse->type == SOF_DFSENTRY_TYPE_IOMEM) {
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_DEBUG_ENABLE_DEBUGFS_CACHE)
+ /*
+ * If the DSP is active: copy from IO.
+ * If the DSP is suspended:
+ * - Copy from IO if the memory is always accessible.
+ * - Otherwise, copy from cached buffer.
+ */
+ if (pm_runtime_active(sdev->dev) ||
+ dfse->access_type == SOF_DEBUGFS_ACCESS_ALWAYS) {
+ memcpy_fromio(buf, dfse->io_mem + pos, size);
+ } else {
+ dev_info(sdev->dev,
+ "Copying cached debugfs data\n");
+ memcpy(buf, dfse->cache_buf + pos, size);
+ }
+#else
+ /* if the DSP is in D3 */
+ if (!pm_runtime_active(sdev->dev) &&
+ dfse->access_type == SOF_DEBUGFS_ACCESS_D0_ONLY) {
+ dev_err(sdev->dev,
+ "error: debugfs entry %s cannot be read in DSP D3\n",
+ dfse->dfsentry->d_name.name);
+ kfree(buf);
+ return -EINVAL;
+ }
+
+ memcpy_fromio(buf, dfse->io_mem + pos, size);
+#endif
+ } else {
+ memcpy(buf, ((u8 *)(dfse->buf) + pos), size);
+ }
+
+ /* copy to userspace */
+ size_ret = copy_to_user(buffer, buf + skip, count);
+
+ kfree(buf);
+
+ /* update count & position if copy succeeded */
+ if (size_ret)
+ return -EFAULT;
+
+ *ppos = pos + count;
+
+ return count;
+}
+
+static const struct file_operations sof_dfs_fops = {
+ .open = simple_open,
+ .read = sof_dfsentry_read,
+ .llseek = default_llseek,
+};
+
+/* create FS entry for debug files that can expose DSP memories, registers */
+int snd_sof_debugfs_io_item(struct snd_sof_dev *sdev,
+ void __iomem *base, size_t size,
+ const char *name,
+ enum sof_debugfs_access_type access_type)
+{
+ struct snd_sof_dfsentry *dfse;
+
+ if (!sdev)
+ return -EINVAL;
+
+ dfse = devm_kzalloc(sdev->dev, sizeof(*dfse), GFP_KERNEL);
+ if (!dfse)
+ return -ENOMEM;
+
+ dfse->type = SOF_DFSENTRY_TYPE_IOMEM;
+ dfse->io_mem = base;
+ dfse->size = size;
+ dfse->sdev = sdev;
+ dfse->access_type = access_type;
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_DEBUG_ENABLE_DEBUGFS_CACHE)
+ /*
+ * allocate cache buffer that will be used to save the mem window
+ * contents prior to suspend
+ */
+ if (access_type == SOF_DEBUGFS_ACCESS_D0_ONLY) {
+ dfse->cache_buf = devm_kzalloc(sdev->dev, size, GFP_KERNEL);
+ if (!dfse->cache_buf)
+ return -ENOMEM;
+ }
+#endif
+
+ dfse->dfsentry = debugfs_create_file(name, 0444, sdev->debugfs_root,
+ dfse, &sof_dfs_fops);
+ if (!dfse->dfsentry) {
+ /* can't rely on debugfs, only log error and keep going */
+ dev_err(sdev->dev, "error: cannot create debugfs entry %s\n",
+ name);
+ } else {
+ /* add to dfsentry list */
+ list_add(&dfse->list, &sdev->dfsentry_list);
+
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(snd_sof_debugfs_io_item);
+
+/* create FS entry for debug files to expose kernel memory */
+int snd_sof_debugfs_buf_item(struct snd_sof_dev *sdev,
+ void *base, size_t size,
+ const char *name)
+{
+ struct snd_sof_dfsentry *dfse;
+
+ if (!sdev)
+ return -EINVAL;
+
+ dfse = devm_kzalloc(sdev->dev, sizeof(*dfse), GFP_KERNEL);
+ if (!dfse)
+ return -ENOMEM;
+
+ dfse->type = SOF_DFSENTRY_TYPE_BUF;
+ dfse->buf = base;
+ dfse->size = size;
+ dfse->sdev = sdev;
+
+ dfse->dfsentry = debugfs_create_file(name, 0444, sdev->debugfs_root,
+ dfse, &sof_dfs_fops);
+ if (!dfse->dfsentry) {
+ /* can't rely on debugfs, only log error and keep going */
+ dev_err(sdev->dev, "error: cannot create debugfs entry %s\n",
+ name);
+ } else {
+ /* add to dfsentry list */
+ list_add(&dfse->list, &sdev->dfsentry_list);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(snd_sof_debugfs_buf_item);
+
+int snd_sof_dbg_init(struct snd_sof_dev *sdev)
+{
+ const struct snd_sof_dsp_ops *ops = sof_ops(sdev);
+ const struct snd_sof_debugfs_map *map;
+ int i;
+ int err;
+
+ /* use "sof" as top level debugFS dir */
+ sdev->debugfs_root = debugfs_create_dir("sof", NULL);
+ if (IS_ERR_OR_NULL(sdev->debugfs_root)) {
+ dev_err(sdev->dev, "error: failed to create debugfs directory\n");
+ return 0;
+ }
+
+ /* init dfsentry list */
+ INIT_LIST_HEAD(&sdev->dfsentry_list);
+
+ /* create debugFS files for platform specific MMIO/DSP memories */
+ for (i = 0; i < ops->debug_map_count; i++) {
+ map = &ops->debug_map[i];
+
+ err = snd_sof_debugfs_io_item(sdev, sdev->bar[map->bar] +
+ map->offset, map->size,
+ map->name, map->access_type);
+ /* errors are only due to memory allocation, not debugfs */
+ if (err < 0)
+ return err;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(snd_sof_dbg_init);
+
+void snd_sof_free_debug(struct snd_sof_dev *sdev)
+{
+ debugfs_remove_recursive(sdev->debugfs_root);
+}
+EXPORT_SYMBOL_GPL(snd_sof_free_debug);
--- /dev/null
+config SND_SOC_SOF_INTEL_TOPLEVEL
+ bool "SOF support for Intel audio DSPs"
+ depends on X86 || COMPILE_TEST
+ help
+ This adds support for Sound Open Firmware for Intel(R) platforms.
+ Say Y if you have such a device.
+ If unsure select "N".
+
+if SND_SOC_SOF_INTEL_TOPLEVEL
+
+config SND_SOC_SOF_INTEL_ACPI
+ tristate
+ select SND_SOC_SOF_BAYTRAIL if SND_SOC_SOF_BAYTRAIL_SUPPORT
+ select SND_SOC_SOF_BROADWELL if SND_SOC_SOF_BROADWELL_SUPPORT
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_INTEL_PCI
+ tristate
+ select SND_SOC_SOF_MERRIFIELD if SND_SOC_SOF_MERRIFIELD_SUPPORT
+ select SND_SOC_SOF_APOLLOLAKE if SND_SOC_SOF_APOLLOLAKE_SUPPORT
+ select SND_SOC_SOF_GEMINILAKE if SND_SOC_SOF_GEMINILAKE_SUPPORT
+ select SND_SOC_SOF_CANNONLAKE if SND_SOC_SOF_CANNONLAKE_SUPPORT
+ select SND_SOC_SOF_COFFEELAKE if SND_SOC_SOF_COFFEELAKE_SUPPORT
+ select SND_SOC_SOF_ICELAKE if SND_SOC_SOF_ICELAKE_SUPPORT
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_INTEL_HIFI_EP_IPC
+ tristate
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_INTEL_ATOM_HIFI_EP
+ tristate
+ select SND_SOC_SOF_INTEL_COMMON
+ select SND_SOC_SOF_INTEL_HIFI_EP_IPC
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_INTEL_COMMON
+ tristate
+ select SND_SOC_ACPI_INTEL_MATCH
+ select SND_SOC_SOF_XTENSA
+ select SND_SOC_INTEL_MACH
+ select SND_SOC_ACPI if ACPI
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+if SND_SOC_SOF_INTEL_ACPI
+
+config SND_SOC_SOF_BAYTRAIL_SUPPORT
+ bool "SOF support for Baytrail, Braswell and Cherrytrail"
+ help
+ This adds support for Sound Open Firmware for Intel(R) platforms
+ using the Baytrail, Braswell or Cherrytrail processors.
+ Say Y if you have such a device.
+ If unsure select "N".
+
+config SND_SOC_SOF_BAYTRAIL
+ tristate
+ select SND_SOC_SOF_INTEL_ATOM_HIFI_EP
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_BROADWELL_SUPPORT
+ bool "SOF support for Broadwell"
+ help
+ This adds support for Sound Open Firmware for Intel(R) platforms
+ using the Broadwell processors.
+ Say Y if you have such a device.
+ If unsure select "N".
+
+config SND_SOC_SOF_BROADWELL
+ tristate
+ select SND_SOC_SOF_INTEL_COMMON
+ select SND_SOC_SOF_INTEL_HIFI_EP_IPC
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+endif ## SND_SOC_SOF_INTEL_ACPI
+
+if SND_SOC_SOF_INTEL_PCI
+
+config SND_SOC_SOF_MERRIFIELD_SUPPORT
+ bool "SOF support for Tangier/Merrifield"
+ help
+ This adds support for Sound Open Firmware for Intel(R) platforms
+ using the Tangier/Merrifield processors.
+ Say Y if you have such a device.
+ If unsure select "N".
+
+config SND_SOC_SOF_MERRIFIELD
+ tristate
+ select SND_SOC_SOF_INTEL_ATOM_HIFI_EP
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_APOLLOLAKE_SUPPORT
+ bool "SOF support for Apollolake"
+ help
+ This adds support for Sound Open Firmware for Intel(R) platforms
+ using the Apollolake processors.
+ Say Y if you have such a device.
+ If unsure select "N".
+
+config SND_SOC_SOF_APOLLOLAKE
+ tristate
+ select SND_SOC_SOF_HDA_COMMON
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_GEMINILAKE_SUPPORT
+ bool "SOF support for GeminiLake"
+ help
+ This adds support for Sound Open Firmware for Intel(R) platforms
+ using the Geminilake processors.
+ Say Y if you have such a device.
+ If unsure select "N".
+
+config SND_SOC_SOF_GEMINILAKE
+ tristate
+ select SND_SOC_SOF_HDA_COMMON
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_CANNONLAKE_SUPPORT
+ bool "SOF support for Cannonlake"
+ help
+ This adds support for Sound Open Firmware for Intel(R) platforms
+ using the Cannonlake processors.
+ Say Y if you have such a device.
+ If unsure select "N".
+
+config SND_SOC_SOF_CANNONLAKE
+ tristate
+ select SND_SOC_SOF_HDA_COMMON
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_COFFEELAKE_SUPPORT
+ bool "SOF support for CoffeeLake"
+ help
+ This adds support for Sound Open Firmware for Intel(R) platforms
+ using the Coffeelake processors.
+ Say Y if you have such a device.
+ If unsure select "N".
+
+config SND_SOC_SOF_COFFEELAKE
+ tristate
+ select SND_SOC_SOF_HDA_COMMON
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_ICELAKE_SUPPORT
+ bool "SOF support for Icelake"
+ help
+ This adds support for Sound Open Firmware for Intel(R) platforms
+ using the Icelake processors.
+ Say Y if you have such a device.
+ If unsure select "N".
+
+config SND_SOC_SOF_ICELAKE
+ tristate
+ select SND_SOC_SOF_HDA_COMMON
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_HDA_COMMON
+ tristate
+ select SND_SOC_SOF_INTEL_COMMON
+ select SND_SOC_SOF_HDA_LINK_BASELINE
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+if SND_SOC_SOF_HDA_COMMON
+
+config SND_SOC_SOF_HDA_LINK
+ bool "SOF support for HDA Links(HDA/HDMI)"
+ depends on SND_SOC_SOF_NOCODEC=n
+ select SND_SOC_SOF_PROBE_WORK_QUEUE
+ help
+ This adds support for HDA links(HDA/HDMI) with Sound Open Firmware
+ for Intel(R) platforms.
+ Say Y if you want to enable HDA links with SOF.
+ If unsure select "N".
+
+config SND_SOC_SOF_HDA_AUDIO_CODEC
+ bool "SOF support for HDAudio codecs"
+ depends on SND_SOC_SOF_HDA_LINK
+ help
+ This adds support for HDAudio codecs with Sound Open Firmware
+ for Intel(R) platforms.
+ Say Y if you want to enable HDAudio codecs with SOF.
+ If unsure select "N".
+
+endif ## SND_SOC_SOF_HDA_COMMON
+
+config SND_SOC_SOF_HDA_LINK_BASELINE
+ tristate
+ select SND_SOC_SOF_HDA if SND_SOC_SOF_HDA_LINK
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+config SND_SOC_SOF_HDA
+ tristate
+ select SND_HDA_EXT_CORE if SND_SOC_SOF_HDA_LINK
+ select SND_SOC_HDAC_HDA if SND_SOC_SOF_HDA_AUDIO_CODEC
+ help
+ This option is not user-selectable but automagically handled by
+ 'select' statements at a higher level
+
+endif ## SND_SOC_SOF_INTEL_PCI
+
+endif ## SND_SOC_SOF_INTEL_TOPLEVEL
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+
+snd-sof-intel-byt-objs := byt.o
+snd-sof-intel-bdw-objs := bdw.o
+
+snd-sof-intel-ipc-objs := intel-ipc.o
+
+snd-sof-intel-hda-common-objs := hda.o hda-loader.o hda-stream.o hda-trace.o \
+ hda-dsp.o hda-ipc.o hda-ctrl.o hda-pcm.o \
+ hda-dai.o hda-bus.o \
+ apl.o cnl.o
+
+snd-sof-intel-hda-objs := hda-codec.o
+
+obj-$(CONFIG_SND_SOC_SOF_INTEL_ATOM_HIFI_EP) += snd-sof-intel-byt.o
+obj-$(CONFIG_SND_SOC_SOF_BROADWELL) += snd-sof-intel-bdw.o
+obj-$(CONFIG_SND_SOC_SOF_INTEL_HIFI_EP_IPC) += snd-sof-intel-ipc.o
+obj-$(CONFIG_SND_SOC_SOF_HDA_COMMON) += snd-sof-intel-hda-common.o
+obj-$(CONFIG_SND_SOC_SOF_HDA) += snd-sof-intel-hda.o
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+// Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
+// Rander Wang <rander.wang@intel.com>
+// Keyon Jie <yang.jie@linux.intel.com>
+//
+
+/*
+ * Hardware interface for audio DSP on Apollolake and GeminiLake
+ */
+
+#include "../sof-priv.h"
+#include "hda.h"
+
+static const struct snd_sof_debugfs_map apl_dsp_debugfs[] = {
+ {"hda", HDA_DSP_HDA_BAR, 0, 0x4000, SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"pp", HDA_DSP_PP_BAR, 0, 0x1000, SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"dsp", HDA_DSP_BAR, 0, 0x10000, SOF_DEBUGFS_ACCESS_ALWAYS},
+};
+
+/* apollolake ops */
+const struct snd_sof_dsp_ops sof_apl_ops = {
+ /* probe and remove */
+ .probe = hda_dsp_probe,
+ .remove = hda_dsp_remove,
+
+ /* Register IO */
+ .write = sof_io_write,
+ .read = sof_io_read,
+ .write64 = sof_io_write64,
+ .read64 = sof_io_read64,
+
+ /* Block IO */
+ .block_read = sof_block_read,
+ .block_write = sof_block_write,
+
+ /* doorbell */
+ .irq_handler = hda_dsp_ipc_irq_handler,
+ .irq_thread = hda_dsp_ipc_irq_thread,
+
+ /* ipc */
+ .send_msg = hda_dsp_ipc_send_msg,
+ .fw_ready = hda_dsp_ipc_fw_ready,
+
+ .ipc_msg_data = hda_ipc_msg_data,
+ .ipc_pcm_params = hda_ipc_pcm_params,
+
+ /* debug */
+ .debug_map = apl_dsp_debugfs,
+ .debug_map_count = ARRAY_SIZE(apl_dsp_debugfs),
+ .dbg_dump = hda_dsp_dump,
+ .ipc_dump = hda_ipc_dump,
+
+ /* stream callbacks */
+ .pcm_open = hda_dsp_pcm_open,
+ .pcm_close = hda_dsp_pcm_close,
+ .pcm_hw_params = hda_dsp_pcm_hw_params,
+ .pcm_trigger = hda_dsp_pcm_trigger,
+ .pcm_pointer = hda_dsp_pcm_pointer,
+
+ /* firmware loading */
+ .load_firmware = snd_sof_load_firmware_raw,
+
+ /* firmware run */
+ .run = hda_dsp_cl_boot_firmware,
+
+ /* pre/post fw run */
+ .pre_fw_run = hda_dsp_pre_fw_run,
+ .post_fw_run = hda_dsp_post_fw_run,
+
+ /* dsp core power up/down */
+ .core_power_up = hda_dsp_enable_core,
+ .core_power_down = hda_dsp_core_reset_power_down,
+
+ /* trace callback */
+ .trace_init = hda_dsp_trace_init,
+ .trace_release = hda_dsp_trace_release,
+ .trace_trigger = hda_dsp_trace_trigger,
+
+ /* DAI drivers */
+ .drv = skl_dai,
+ .num_drv = SOF_SKL_NUM_DAIS,
+
+ /* PM */
+ .suspend = hda_dsp_suspend,
+ .resume = hda_dsp_resume,
+ .runtime_suspend = hda_dsp_runtime_suspend,
+ .runtime_resume = hda_dsp_runtime_resume,
+ .set_hw_params_upon_resume = hda_dsp_set_hw_params_upon_resume,
+};
+EXPORT_SYMBOL(sof_apl_ops);
+
+const struct sof_intel_dsp_desc apl_chip_info = {
+ /* Apollolake */
+ .cores_num = 2,
+ .init_core_mask = 1,
+ .cores_mask = HDA_DSP_CORE_MASK(0) | HDA_DSP_CORE_MASK(1),
+ .ipc_req = HDA_DSP_REG_HIPCI,
+ .ipc_req_mask = HDA_DSP_REG_HIPCI_BUSY,
+ .ipc_ack = HDA_DSP_REG_HIPCIE,
+ .ipc_ack_mask = HDA_DSP_REG_HIPCIE_DONE,
+ .ipc_ctl = HDA_DSP_REG_HIPCCTL,
+ .rom_init_timeout = 150,
+ .ssp_count = APL_SSP_COUNT,
+ .ssp_base_offset = APL_SSP_BASE_OFFSET,
+};
+EXPORT_SYMBOL(apl_chip_info);
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+/*
+ * Hardware interface for audio DSP on Broadwell
+ */
+
+#include <linux/module.h>
+#include <sound/sof.h>
+#include <sound/sof/xtensa.h>
+#include "../ops.h"
+#include "shim.h"
+
+/* BARs */
+#define BDW_DSP_BAR 0
+#define BDW_PCI_BAR 1
+
+/*
+ * Debug
+ */
+
+/* DSP memories for BDW */
+#define IRAM_OFFSET 0xA0000
+#define BDW_IRAM_SIZE (10 * 32 * 1024)
+#define DRAM_OFFSET 0x00000
+#define BDW_DRAM_SIZE (20 * 32 * 1024)
+#define SHIM_OFFSET 0xFB000
+#define SHIM_SIZE 0x100
+#define MBOX_OFFSET 0x9E000
+#define MBOX_SIZE 0x1000
+#define MBOX_DUMP_SIZE 0x30
+#define EXCEPT_OFFSET 0x800
+
+/* DSP peripherals */
+#define DMAC0_OFFSET 0xFE000
+#define DMAC1_OFFSET 0xFF000
+#define DMAC_SIZE 0x420
+#define SSP0_OFFSET 0xFC000
+#define SSP1_OFFSET 0xFD000
+#define SSP_SIZE 0x100
+
+#define BDW_STACK_DUMP_SIZE 32
+
+#define BDW_PANIC_OFFSET(x) ((x) & 0xFFFF)
+
+static const struct snd_sof_debugfs_map bdw_debugfs[] = {
+ {"dmac0", BDW_DSP_BAR, DMAC0_OFFSET, DMAC_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"dmac1", BDW_DSP_BAR, DMAC1_OFFSET, DMAC_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp0", BDW_DSP_BAR, SSP0_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp1", BDW_DSP_BAR, SSP1_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"iram", BDW_DSP_BAR, IRAM_OFFSET, BDW_IRAM_SIZE,
+ SOF_DEBUGFS_ACCESS_D0_ONLY},
+ {"dram", BDW_DSP_BAR, DRAM_OFFSET, BDW_DRAM_SIZE,
+ SOF_DEBUGFS_ACCESS_D0_ONLY},
+ {"shim", BDW_DSP_BAR, SHIM_OFFSET, SHIM_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+};
+
+static void bdw_host_done(struct snd_sof_dev *sdev);
+static void bdw_dsp_done(struct snd_sof_dev *sdev);
+static void bdw_get_reply(struct snd_sof_dev *sdev);
+
+/*
+ * DSP Control.
+ */
+
+static int bdw_run(struct snd_sof_dev *sdev)
+{
+ /* set opportunistic mode on engine 0,1 for all channels */
+ snd_sof_dsp_update_bits(sdev, BDW_DSP_BAR, SHIM_HMDC,
+ SHIM_HMDC_HDDA_E0_ALLCH |
+ SHIM_HMDC_HDDA_E1_ALLCH, 0);
+
+ /* set DSP to RUN */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR, SHIM_CSR,
+ SHIM_CSR_STALL, 0x0);
+
+ /* return init core mask */
+ return 1;
+}
+
+static int bdw_reset(struct snd_sof_dev *sdev)
+{
+ /* put DSP into reset and stall */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR, SHIM_CSR,
+ SHIM_CSR_RST | SHIM_CSR_STALL,
+ SHIM_CSR_RST | SHIM_CSR_STALL);
+
+ /* keep in reset for 10ms */
+ mdelay(10);
+
+ /* take DSP out of reset and keep stalled for FW loading */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR, SHIM_CSR,
+ SHIM_CSR_RST | SHIM_CSR_STALL,
+ SHIM_CSR_STALL);
+
+ return 0;
+}
+
+static int bdw_set_dsp_D0(struct snd_sof_dev *sdev)
+{
+ int tries = 10;
+ u32 reg;
+
+ /* Disable core clock gating (VDRTCTL2.DCLCGE = 0) */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_PCI_BAR, PCI_VDRTCTL2,
+ PCI_VDRTCL2_DCLCGE |
+ PCI_VDRTCL2_DTCGE, 0);
+
+ /* Disable D3PG (VDRTCTL0.D3PGD = 1) */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_PCI_BAR, PCI_VDRTCTL0,
+ PCI_VDRTCL0_D3PGD, PCI_VDRTCL0_D3PGD);
+
+ /* Set D0 state */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_PCI_BAR, PCI_PMCS,
+ PCI_PMCS_PS_MASK, 0);
+
+ /* check that ADSP shim is enabled */
+ while (tries--) {
+ reg = readl(sdev->bar[BDW_PCI_BAR] + PCI_PMCS)
+ & PCI_PMCS_PS_MASK;
+ if (reg == 0)
+ goto finish;
+
+ msleep(20);
+ }
+
+ return -ENODEV;
+
+finish:
+ /*
+ * select SSP1 19.2MHz base clock, SSP clock 0,
+ * turn off Low Power Clock
+ */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR, SHIM_CSR,
+ SHIM_CSR_S1IOCS | SHIM_CSR_SBCS1 |
+ SHIM_CSR_LPCS, 0x0);
+
+ /* stall DSP core, set clk to 192/96Mhz */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR,
+ SHIM_CSR, SHIM_CSR_STALL |
+ SHIM_CSR_DCS_MASK,
+ SHIM_CSR_STALL |
+ SHIM_CSR_DCS(4));
+
+ /* Set 24MHz MCLK, prevent local clock gating, enable SSP0 clock */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR, SHIM_CLKCTL,
+ SHIM_CLKCTL_MASK |
+ SHIM_CLKCTL_DCPLCG |
+ SHIM_CLKCTL_SCOE0,
+ SHIM_CLKCTL_MASK |
+ SHIM_CLKCTL_DCPLCG |
+ SHIM_CLKCTL_SCOE0);
+
+ /* Stall and reset core, set CSR */
+ bdw_reset(sdev);
+
+ /* Enable core clock gating (VDRTCTL2.DCLCGE = 1), delay 50 us */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_PCI_BAR, PCI_VDRTCTL2,
+ PCI_VDRTCL2_DCLCGE |
+ PCI_VDRTCL2_DTCGE,
+ PCI_VDRTCL2_DCLCGE |
+ PCI_VDRTCL2_DTCGE);
+
+ usleep_range(50, 55);
+
+ /* switch on audio PLL */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_PCI_BAR, PCI_VDRTCTL2,
+ PCI_VDRTCL2_APLLSE_MASK, 0);
+
+ /*
+ * set default power gating control, enable power gating control for
+ * all blocks. that is, can't be accessed, please enable each block
+ * before accessing.
+ */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_PCI_BAR, PCI_VDRTCTL0,
+ 0xfffffffC, 0x0);
+
+ /* disable DMA finish function for SSP0 & SSP1 */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR, SHIM_CSR2,
+ SHIM_CSR2_SDFD_SSP1,
+ SHIM_CSR2_SDFD_SSP1);
+
+ /* set on-demond mode on engine 0,1 for all channels */
+ snd_sof_dsp_update_bits(sdev, BDW_DSP_BAR, SHIM_HMDC,
+ SHIM_HMDC_HDDA_E0_ALLCH |
+ SHIM_HMDC_HDDA_E1_ALLCH,
+ SHIM_HMDC_HDDA_E0_ALLCH |
+ SHIM_HMDC_HDDA_E1_ALLCH);
+
+ /* Enable Interrupt from both sides */
+ snd_sof_dsp_update_bits(sdev, BDW_DSP_BAR, SHIM_IMRX,
+ (SHIM_IMRX_BUSY | SHIM_IMRX_DONE), 0x0);
+ snd_sof_dsp_update_bits(sdev, BDW_DSP_BAR, SHIM_IMRD,
+ (SHIM_IMRD_DONE | SHIM_IMRD_BUSY |
+ SHIM_IMRD_SSP0 | SHIM_IMRD_DMAC), 0x0);
+
+ /* clear IPC registers */
+ snd_sof_dsp_write(sdev, BDW_DSP_BAR, SHIM_IPCX, 0x0);
+ snd_sof_dsp_write(sdev, BDW_DSP_BAR, SHIM_IPCD, 0x0);
+ snd_sof_dsp_write(sdev, BDW_DSP_BAR, 0x80, 0x6);
+ snd_sof_dsp_write(sdev, BDW_DSP_BAR, 0xe0, 0x300a);
+
+ return 0;
+}
+
+static void bdw_get_registers(struct snd_sof_dev *sdev,
+ struct sof_ipc_dsp_oops_xtensa *xoops,
+ struct sof_ipc_panic_info *panic_info,
+ u32 *stack, size_t stack_words)
+{
+ /* first read regsisters */
+ sof_mailbox_read(sdev, sdev->dsp_oops_offset, xoops, sizeof(*xoops));
+
+ /* then get panic info */
+ sof_mailbox_read(sdev, sdev->dsp_oops_offset + sizeof(*xoops),
+ panic_info, sizeof(*panic_info));
+
+ /* then get the stack */
+ sof_mailbox_read(sdev, sdev->dsp_oops_offset + sizeof(*xoops) +
+ sizeof(*panic_info), stack,
+ stack_words * sizeof(u32));
+}
+
+static void bdw_dump(struct snd_sof_dev *sdev, u32 flags)
+{
+ struct sof_ipc_dsp_oops_xtensa xoops;
+ struct sof_ipc_panic_info panic_info;
+ u32 stack[BDW_STACK_DUMP_SIZE];
+ u32 status, panic;
+
+ /* now try generic SOF status messages */
+ status = snd_sof_dsp_read(sdev, BDW_DSP_BAR, SHIM_IPCD);
+ panic = snd_sof_dsp_read(sdev, BDW_DSP_BAR, SHIM_IPCX);
+ bdw_get_registers(sdev, &xoops, &panic_info, stack,
+ BDW_STACK_DUMP_SIZE);
+ snd_sof_get_status(sdev, status, panic, &xoops, &panic_info, stack,
+ BDW_STACK_DUMP_SIZE);
+}
+
+/*
+ * IPC Doorbell IRQ handler and thread.
+ */
+
+static irqreturn_t bdw_irq_handler(int irq, void *context)
+{
+ struct snd_sof_dev *sdev = context;
+ u32 isr;
+ int ret = IRQ_NONE;
+
+ /* Interrupt arrived, check src */
+ isr = snd_sof_dsp_read(sdev, BDW_DSP_BAR, SHIM_ISRX);
+ if (isr & (SHIM_ISRX_DONE | SHIM_ISRX_BUSY))
+ ret = IRQ_WAKE_THREAD;
+
+ return ret;
+}
+
+static irqreturn_t bdw_irq_thread(int irq, void *context)
+{
+ struct snd_sof_dev *sdev = context;
+ u32 ipcx, ipcd, imrx;
+
+ imrx = snd_sof_dsp_read64(sdev, BDW_DSP_BAR, SHIM_IMRX);
+ ipcx = snd_sof_dsp_read(sdev, BDW_DSP_BAR, SHIM_IPCX);
+
+ /* reply message from DSP */
+ if (ipcx & SHIM_IPCX_DONE &&
+ !(imrx & SHIM_IMRX_DONE)) {
+ /* Mask Done interrupt before return */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR,
+ SHIM_IMRX, SHIM_IMRX_DONE,
+ SHIM_IMRX_DONE);
+
+ /*
+ * handle immediate reply from DSP core. If the msg is
+ * found, set done bit in cmd_done which is called at the
+ * end of message processing function, else set it here
+ * because the done bit can't be set in cmd_done function
+ * which is triggered by msg
+ */
+ bdw_get_reply(sdev);
+ snd_sof_ipc_reply(sdev, ipcx);
+
+ bdw_dsp_done(sdev);
+ }
+
+ ipcd = snd_sof_dsp_read(sdev, BDW_DSP_BAR, SHIM_IPCD);
+
+ /* new message from DSP */
+ if (ipcd & SHIM_IPCD_BUSY &&
+ !(imrx & SHIM_IMRX_BUSY)) {
+ /* Mask Busy interrupt before return */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR,
+ SHIM_IMRX, SHIM_IMRX_BUSY,
+ SHIM_IMRX_BUSY);
+
+ /* Handle messages from DSP Core */
+ if ((ipcd & SOF_IPC_PANIC_MAGIC_MASK) == SOF_IPC_PANIC_MAGIC) {
+ snd_sof_dsp_panic(sdev, BDW_PANIC_OFFSET(ipcx) +
+ MBOX_OFFSET);
+ } else {
+ snd_sof_ipc_msgs_rx(sdev);
+ }
+
+ bdw_host_done(sdev);
+ }
+
+ return IRQ_HANDLED;
+}
+
+/*
+ * IPC Firmware ready.
+ */
+static void bdw_get_windows(struct snd_sof_dev *sdev)
+{
+ struct sof_ipc_window_elem *elem;
+ u32 outbox_offset = 0;
+ u32 stream_offset = 0;
+ u32 inbox_offset = 0;
+ u32 outbox_size = 0;
+ u32 stream_size = 0;
+ u32 inbox_size = 0;
+ int i;
+
+ if (!sdev->info_window) {
+ dev_err(sdev->dev, "error: have no window info\n");
+ return;
+ }
+
+ for (i = 0; i < sdev->info_window->num_windows; i++) {
+ elem = &sdev->info_window->window[i];
+
+ switch (elem->type) {
+ case SOF_IPC_REGION_UPBOX:
+ inbox_offset = elem->offset + MBOX_OFFSET;
+ inbox_size = elem->size;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BDW_DSP_BAR] +
+ inbox_offset,
+ elem->size, "inbox",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_DOWNBOX:
+ outbox_offset = elem->offset + MBOX_OFFSET;
+ outbox_size = elem->size;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BDW_DSP_BAR] +
+ outbox_offset,
+ elem->size, "outbox",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_TRACE:
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BDW_DSP_BAR] +
+ elem->offset +
+ MBOX_OFFSET,
+ elem->size, "etrace",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_DEBUG:
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BDW_DSP_BAR] +
+ elem->offset +
+ MBOX_OFFSET,
+ elem->size, "debug",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_STREAM:
+ stream_offset = elem->offset + MBOX_OFFSET;
+ stream_size = elem->size;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BDW_DSP_BAR] +
+ stream_offset,
+ elem->size, "stream",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_REGS:
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BDW_DSP_BAR] +
+ elem->offset +
+ MBOX_OFFSET,
+ elem->size, "regs",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_EXCEPTION:
+ sdev->dsp_oops_offset = elem->offset + MBOX_OFFSET;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BDW_DSP_BAR] +
+ elem->offset +
+ MBOX_OFFSET,
+ elem->size, "exception",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ default:
+ dev_err(sdev->dev, "error: get illegal window info\n");
+ return;
+ }
+ }
+
+ if (outbox_size == 0 || inbox_size == 0) {
+ dev_err(sdev->dev, "error: get illegal mailbox window\n");
+ return;
+ }
+
+ snd_sof_dsp_mailbox_init(sdev, inbox_offset, inbox_size,
+ outbox_offset, outbox_size);
+ sdev->stream_box.offset = stream_offset;
+ sdev->stream_box.size = stream_size;
+
+ dev_dbg(sdev->dev, " mailbox upstream 0x%x - size 0x%x\n",
+ inbox_offset, inbox_size);
+ dev_dbg(sdev->dev, " mailbox downstream 0x%x - size 0x%x\n",
+ outbox_offset, outbox_size);
+ dev_dbg(sdev->dev, " stream region 0x%x - size 0x%x\n",
+ stream_offset, stream_size);
+}
+
+/* check for ABI compatibility and create memory windows on first boot */
+static int bdw_fw_ready(struct snd_sof_dev *sdev, u32 msg_id)
+{
+ struct sof_ipc_fw_ready *fw_ready = &sdev->fw_ready;
+ u32 offset;
+ int ret;
+
+ /* mailbox must be on 4k boundary */
+ offset = MBOX_OFFSET;
+
+ dev_dbg(sdev->dev, "ipc: DSP is ready 0x%8.8x offset %d\n",
+ msg_id, offset);
+
+ /* no need to re-check version/ABI for subsequent boots */
+ if (!sdev->first_boot)
+ return 0;
+
+ /* copy data from the DSP FW ready offset */
+ sof_block_read(sdev, sdev->mmio_bar, offset, fw_ready,
+ sizeof(*fw_ready));
+
+ snd_sof_dsp_mailbox_init(sdev, fw_ready->dspbox_offset,
+ fw_ready->dspbox_size,
+ fw_ready->hostbox_offset,
+ fw_ready->hostbox_size);
+
+ /* make sure ABI version is compatible */
+ ret = snd_sof_ipc_valid(sdev);
+ if (ret < 0)
+ return ret;
+
+ /* now check for extended data */
+ snd_sof_fw_parse_ext_data(sdev, sdev->mmio_bar, MBOX_OFFSET +
+ sizeof(struct sof_ipc_fw_ready));
+
+ bdw_get_windows(sdev);
+
+ return 0;
+}
+
+/*
+ * IPC Mailbox IO
+ */
+
+static int bdw_send_msg(struct snd_sof_dev *sdev, struct snd_sof_ipc_msg *msg)
+{
+ /* send the message */
+ sof_mailbox_write(sdev, sdev->host_box.offset, msg->msg_data,
+ msg->msg_size);
+ snd_sof_dsp_write(sdev, BDW_DSP_BAR, SHIM_IPCX, SHIM_IPCX_BUSY);
+
+ return 0;
+}
+
+static void bdw_get_reply(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_ipc_msg *msg = sdev->msg;
+ struct sof_ipc_reply reply;
+ unsigned long flags;
+ int ret = 0;
+
+ /*
+ * Sometimes, there is unexpected reply ipc arriving. The reply
+ * ipc belongs to none of the ipcs sent from driver.
+ * In this case, the driver must ignore the ipc.
+ */
+ if (!msg) {
+ dev_warn(sdev->dev, "unexpected ipc interrupt raised!\n");
+ return;
+ }
+
+ /* get reply */
+ sof_mailbox_read(sdev, sdev->host_box.offset, &reply, sizeof(reply));
+
+ spin_lock_irqsave(&sdev->ipc_lock, flags);
+
+ if (reply.error < 0) {
+ memcpy(msg->reply_data, &reply, sizeof(reply));
+ ret = reply.error;
+ } else {
+ /* reply correct size ? */
+ if (reply.hdr.size != msg->reply_size) {
+ dev_err(sdev->dev, "error: reply expected %zu got %u bytes\n",
+ msg->reply_size, reply.hdr.size);
+ ret = -EINVAL;
+ }
+
+ /* read the message */
+ if (msg->reply_size > 0)
+ sof_mailbox_read(sdev, sdev->host_box.offset,
+ msg->reply_data, msg->reply_size);
+ }
+
+ msg->reply_error = ret;
+
+ spin_unlock_irqrestore(&sdev->ipc_lock, flags);
+}
+
+static void bdw_host_done(struct snd_sof_dev *sdev)
+{
+ /* clear BUSY bit and set DONE bit - accept new messages */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR, SHIM_IPCD,
+ SHIM_IPCD_BUSY | SHIM_IPCD_DONE,
+ SHIM_IPCD_DONE);
+
+ /* unmask busy interrupt */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR, SHIM_IMRX,
+ SHIM_IMRX_BUSY, 0);
+}
+
+static void bdw_dsp_done(struct snd_sof_dev *sdev)
+{
+ /* clear DONE bit - tell DSP we have completed */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR, SHIM_IPCX,
+ SHIM_IPCX_DONE, 0);
+
+ /* unmask Done interrupt */
+ snd_sof_dsp_update_bits_unlocked(sdev, BDW_DSP_BAR, SHIM_IMRX,
+ SHIM_IMRX_DONE, 0);
+}
+
+/*
+ * Probe and remove.
+ */
+static int bdw_probe(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_pdata *pdata = sdev->pdata;
+ const struct sof_dev_desc *desc = pdata->desc;
+ struct platform_device *pdev =
+ container_of(sdev->dev, struct platform_device, dev);
+ struct resource *mmio;
+ u32 base, size;
+ int ret;
+
+ /* LPE base */
+ mmio = platform_get_resource(pdev, IORESOURCE_MEM,
+ desc->resindex_lpe_base);
+ if (mmio) {
+ base = mmio->start;
+ size = resource_size(mmio);
+ } else {
+ dev_err(sdev->dev, "error: failed to get LPE base at idx %d\n",
+ desc->resindex_lpe_base);
+ return -EINVAL;
+ }
+
+ dev_dbg(sdev->dev, "LPE PHY base at 0x%x size 0x%x", base, size);
+ sdev->bar[BDW_DSP_BAR] = devm_ioremap(sdev->dev, base, size);
+ if (!sdev->bar[BDW_DSP_BAR]) {
+ dev_err(sdev->dev,
+ "error: failed to ioremap LPE base 0x%x size 0x%x\n",
+ base, size);
+ return -ENODEV;
+ }
+ dev_dbg(sdev->dev, "LPE VADDR %p\n", sdev->bar[BDW_DSP_BAR]);
+
+ /* TODO: add offsets */
+ sdev->mmio_bar = BDW_DSP_BAR;
+ sdev->mailbox_bar = BDW_DSP_BAR;
+
+ /* PCI base */
+ mmio = platform_get_resource(pdev, IORESOURCE_MEM,
+ desc->resindex_pcicfg_base);
+ if (mmio) {
+ base = mmio->start;
+ size = resource_size(mmio);
+ } else {
+ dev_err(sdev->dev, "error: failed to get PCI base at idx %d\n",
+ desc->resindex_pcicfg_base);
+ return -ENODEV;
+ }
+
+ dev_dbg(sdev->dev, "PCI base at 0x%x size 0x%x", base, size);
+ sdev->bar[BDW_PCI_BAR] = devm_ioremap(sdev->dev, base, size);
+ if (!sdev->bar[BDW_PCI_BAR]) {
+ dev_err(sdev->dev,
+ "error: failed to ioremap PCI base 0x%x size 0x%x\n",
+ base, size);
+ return -ENODEV;
+ }
+ dev_dbg(sdev->dev, "PCI VADDR %p\n", sdev->bar[BDW_PCI_BAR]);
+
+ /* register our IRQ */
+ sdev->ipc_irq = platform_get_irq(pdev, desc->irqindex_host_ipc);
+ if (sdev->ipc_irq < 0) {
+ dev_err(sdev->dev, "error: failed to get IRQ at index %d\n",
+ desc->irqindex_host_ipc);
+ return sdev->ipc_irq;
+ }
+
+ dev_dbg(sdev->dev, "using IRQ %d\n", sdev->ipc_irq);
+ ret = devm_request_threaded_irq(sdev->dev, sdev->ipc_irq,
+ bdw_irq_handler, bdw_irq_thread,
+ IRQF_SHARED, "AudioDSP", sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to register IRQ %d\n",
+ sdev->ipc_irq);
+ return ret;
+ }
+
+ /* enable the DSP SHIM */
+ ret = bdw_set_dsp_D0(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to set DSP D0\n");
+ return ret;
+ }
+
+ /* DSP DMA can only access low 31 bits of host memory */
+ ret = dma_coerce_mask_and_coherent(sdev->dev, DMA_BIT_MASK(31));
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to set DMA mask %d\n", ret);
+ return ret;
+ }
+
+ /* set default mailbox */
+ snd_sof_dsp_mailbox_init(sdev, MBOX_OFFSET, MBOX_SIZE, 0, 0);
+
+ return ret;
+}
+
+/* Broadwell DAIs */
+static struct snd_soc_dai_driver bdw_dai[] = {
+{
+ .name = "ssp0-port",
+},
+{
+ .name = "ssp1-port",
+},
+};
+
+/* broadwell ops */
+const struct snd_sof_dsp_ops sof_bdw_ops = {
+ /*Device init */
+ .probe = bdw_probe,
+
+ /* DSP Core Control */
+ .run = bdw_run,
+ .reset = bdw_reset,
+
+ /* Register IO */
+ .write = sof_io_write,
+ .read = sof_io_read,
+ .write64 = sof_io_write64,
+ .read64 = sof_io_read64,
+
+ /* Block IO */
+ .block_read = sof_block_read,
+ .block_write = sof_block_write,
+
+ /* ipc */
+ .send_msg = bdw_send_msg,
+ .fw_ready = bdw_fw_ready,
+
+ .ipc_msg_data = intel_ipc_msg_data,
+ .ipc_pcm_params = intel_ipc_pcm_params,
+
+ /* debug */
+ .debug_map = bdw_debugfs,
+ .debug_map_count = ARRAY_SIZE(bdw_debugfs),
+ .dbg_dump = bdw_dump,
+
+ /* stream callbacks */
+ .pcm_open = intel_pcm_open,
+ .pcm_close = intel_pcm_close,
+
+ /* Module loading */
+ .load_module = snd_sof_parse_module_memcpy,
+
+ /*Firmware loading */
+ .load_firmware = snd_sof_load_firmware_memcpy,
+
+ /* DAI drivers */
+ .drv = bdw_dai,
+ .num_drv = ARRAY_SIZE(bdw_dai)
+};
+EXPORT_SYMBOL(sof_bdw_ops);
+
+const struct sof_intel_dsp_desc bdw_chip_info = {
+ .cores_num = 1,
+ .cores_mask = 1,
+};
+EXPORT_SYMBOL(bdw_chip_info);
+
+MODULE_LICENSE("Dual BSD/GPL");
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+/*
+ * Hardware interface for audio DSP on Baytrail, Braswell and Cherrytrail.
+ */
+
+#include <linux/module.h>
+#include <sound/sof.h>
+#include <sound/sof/xtensa.h>
+#include "../ops.h"
+#include "shim.h"
+
+/* DSP memories */
+#define IRAM_OFFSET 0x0C0000
+#define IRAM_SIZE (80 * 1024)
+#define DRAM_OFFSET 0x100000
+#define DRAM_SIZE (160 * 1024)
+#define SHIM_OFFSET 0x140000
+#define SHIM_SIZE 0x100
+#define MBOX_OFFSET 0x144000
+#define MBOX_SIZE 0x1000
+#define EXCEPT_OFFSET 0x800
+
+/* DSP peripherals */
+#define DMAC0_OFFSET 0x098000
+#define DMAC1_OFFSET 0x09c000
+#define DMAC2_OFFSET 0x094000
+#define DMAC_SIZE 0x420
+#define SSP0_OFFSET 0x0a0000
+#define SSP1_OFFSET 0x0a1000
+#define SSP2_OFFSET 0x0a2000
+#define SSP3_OFFSET 0x0a4000
+#define SSP4_OFFSET 0x0a5000
+#define SSP5_OFFSET 0x0a6000
+#define SSP_SIZE 0x100
+
+#define BYT_STACK_DUMP_SIZE 32
+
+#define BYT_PCI_BAR_SIZE 0x200000
+
+#define BYT_PANIC_OFFSET(x) (((x) & GENMASK_ULL(47, 32)) >> 32)
+
+/*
+ * Debug
+ */
+
+#define MBOX_DUMP_SIZE 0x30
+
+/* BARs */
+#define BYT_DSP_BAR 0
+#define BYT_PCI_BAR 1
+#define BYT_IMR_BAR 2
+
+static const struct snd_sof_debugfs_map byt_debugfs[] = {
+ {"dmac0", BYT_DSP_BAR, DMAC0_OFFSET, DMAC_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"dmac1", BYT_DSP_BAR, DMAC1_OFFSET, DMAC_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp0", BYT_DSP_BAR, SSP0_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp1", BYT_DSP_BAR, SSP1_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp2", BYT_DSP_BAR, SSP2_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"iram", BYT_DSP_BAR, IRAM_OFFSET, IRAM_SIZE,
+ SOF_DEBUGFS_ACCESS_D0_ONLY},
+ {"dram", BYT_DSP_BAR, DRAM_OFFSET, DRAM_SIZE,
+ SOF_DEBUGFS_ACCESS_D0_ONLY},
+ {"shim", BYT_DSP_BAR, SHIM_OFFSET, SHIM_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+};
+
+static const struct snd_sof_debugfs_map cht_debugfs[] = {
+ {"dmac0", BYT_DSP_BAR, DMAC0_OFFSET, DMAC_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"dmac1", BYT_DSP_BAR, DMAC1_OFFSET, DMAC_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"dmac2", BYT_DSP_BAR, DMAC2_OFFSET, DMAC_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp0", BYT_DSP_BAR, SSP0_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp1", BYT_DSP_BAR, SSP1_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp2", BYT_DSP_BAR, SSP2_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp3", BYT_DSP_BAR, SSP3_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp4", BYT_DSP_BAR, SSP4_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"ssp5", BYT_DSP_BAR, SSP5_OFFSET, SSP_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"iram", BYT_DSP_BAR, IRAM_OFFSET, IRAM_SIZE,
+ SOF_DEBUGFS_ACCESS_D0_ONLY},
+ {"dram", BYT_DSP_BAR, DRAM_OFFSET, DRAM_SIZE,
+ SOF_DEBUGFS_ACCESS_D0_ONLY},
+ {"shim", BYT_DSP_BAR, SHIM_OFFSET, SHIM_SIZE,
+ SOF_DEBUGFS_ACCESS_ALWAYS},
+};
+
+static void byt_host_done(struct snd_sof_dev *sdev);
+static void byt_dsp_done(struct snd_sof_dev *sdev);
+static void byt_get_reply(struct snd_sof_dev *sdev);
+
+/*
+ * IPC Firmware ready.
+ */
+static void byt_get_windows(struct snd_sof_dev *sdev)
+{
+ struct sof_ipc_window_elem *elem;
+ u32 outbox_offset = 0;
+ u32 stream_offset = 0;
+ u32 inbox_offset = 0;
+ u32 outbox_size = 0;
+ u32 stream_size = 0;
+ u32 inbox_size = 0;
+ int i;
+
+ if (!sdev->info_window) {
+ dev_err(sdev->dev, "error: have no window info\n");
+ return;
+ }
+
+ for (i = 0; i < sdev->info_window->num_windows; i++) {
+ elem = &sdev->info_window->window[i];
+
+ switch (elem->type) {
+ case SOF_IPC_REGION_UPBOX:
+ inbox_offset = elem->offset + MBOX_OFFSET;
+ inbox_size = elem->size;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BYT_DSP_BAR] +
+ inbox_offset,
+ elem->size, "inbox",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_DOWNBOX:
+ outbox_offset = elem->offset + MBOX_OFFSET;
+ outbox_size = elem->size;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BYT_DSP_BAR] +
+ outbox_offset,
+ elem->size, "outbox",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_TRACE:
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BYT_DSP_BAR] +
+ elem->offset +
+ MBOX_OFFSET,
+ elem->size, "etrace",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_DEBUG:
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BYT_DSP_BAR] +
+ elem->offset +
+ MBOX_OFFSET,
+ elem->size, "debug",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_STREAM:
+ stream_offset = elem->offset + MBOX_OFFSET;
+ stream_size = elem->size;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BYT_DSP_BAR] +
+ stream_offset,
+ elem->size, "stream",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_REGS:
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BYT_DSP_BAR] +
+ elem->offset +
+ MBOX_OFFSET,
+ elem->size, "regs",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_EXCEPTION:
+ sdev->dsp_oops_offset = elem->offset + MBOX_OFFSET;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[BYT_DSP_BAR] +
+ elem->offset +
+ MBOX_OFFSET,
+ elem->size, "exception",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ default:
+ dev_err(sdev->dev, "error: get illegal window info\n");
+ return;
+ }
+ }
+
+ if (outbox_size == 0 || inbox_size == 0) {
+ dev_err(sdev->dev, "error: get illegal mailbox window\n");
+ return;
+ }
+
+ snd_sof_dsp_mailbox_init(sdev, inbox_offset, inbox_size,
+ outbox_offset, outbox_size);
+ sdev->stream_box.offset = stream_offset;
+ sdev->stream_box.size = stream_size;
+
+ dev_dbg(sdev->dev, " mailbox upstream 0x%x - size 0x%x\n",
+ inbox_offset, inbox_size);
+ dev_dbg(sdev->dev, " mailbox downstream 0x%x - size 0x%x\n",
+ outbox_offset, outbox_size);
+ dev_dbg(sdev->dev, " stream region 0x%x - size 0x%x\n",
+ stream_offset, stream_size);
+}
+
+/* check for ABI compatibility and create memory windows on first boot */
+static int byt_fw_ready(struct snd_sof_dev *sdev, u32 msg_id)
+{
+ struct sof_ipc_fw_ready *fw_ready = &sdev->fw_ready;
+ u32 offset;
+ int ret;
+
+ /* mailbox must be on 4k boundary */
+ offset = MBOX_OFFSET;
+
+ dev_dbg(sdev->dev, "ipc: DSP is ready 0x%8.8x offset 0x%x\n",
+ msg_id, offset);
+
+ /* no need to re-check version/ABI for subsequent boots */
+ if (!sdev->first_boot)
+ return 0;
+
+ /* copy data from the DSP FW ready offset */
+ sof_block_read(sdev, sdev->mmio_bar, offset, fw_ready,
+ sizeof(*fw_ready));
+
+ snd_sof_dsp_mailbox_init(sdev, fw_ready->dspbox_offset,
+ fw_ready->dspbox_size,
+ fw_ready->hostbox_offset,
+ fw_ready->hostbox_size);
+
+ /* make sure ABI version is compatible */
+ ret = snd_sof_ipc_valid(sdev);
+ if (ret < 0)
+ return ret;
+
+ /* now check for extended data */
+ snd_sof_fw_parse_ext_data(sdev, sdev->mmio_bar, MBOX_OFFSET +
+ sizeof(struct sof_ipc_fw_ready));
+
+ byt_get_windows(sdev);
+
+ return 0;
+}
+
+/*
+ * Debug
+ */
+
+static void byt_get_registers(struct snd_sof_dev *sdev,
+ struct sof_ipc_dsp_oops_xtensa *xoops,
+ struct sof_ipc_panic_info *panic_info,
+ u32 *stack, size_t stack_words)
+{
+ /* first read regsisters */
+ sof_mailbox_read(sdev, sdev->dsp_oops_offset, xoops, sizeof(*xoops));
+
+ /* then get panic info */
+ sof_mailbox_read(sdev, sdev->dsp_oops_offset + sizeof(*xoops),
+ panic_info, sizeof(*panic_info));
+
+ /* then get the stack */
+ sof_mailbox_read(sdev, sdev->dsp_oops_offset + sizeof(*xoops) +
+ sizeof(*panic_info), stack,
+ stack_words * sizeof(u32));
+}
+
+static void byt_dump(struct snd_sof_dev *sdev, u32 flags)
+{
+ struct sof_ipc_dsp_oops_xtensa xoops;
+ struct sof_ipc_panic_info panic_info;
+ u32 stack[BYT_STACK_DUMP_SIZE];
+ u32 status, panic;
+
+ /* now try generic SOF status messages */
+ status = snd_sof_dsp_read(sdev, BYT_DSP_BAR, SHIM_IPCD);
+ panic = snd_sof_dsp_read(sdev, BYT_DSP_BAR, SHIM_IPCX);
+ byt_get_registers(sdev, &xoops, &panic_info, stack,
+ BYT_STACK_DUMP_SIZE);
+ snd_sof_get_status(sdev, status, panic, &xoops, &panic_info, stack,
+ BYT_STACK_DUMP_SIZE);
+}
+
+/*
+ * IPC Doorbell IRQ handler and thread.
+ */
+
+static irqreturn_t byt_irq_handler(int irq, void *context)
+{
+ struct snd_sof_dev *sdev = context;
+ u64 isr;
+ int ret = IRQ_NONE;
+
+ /* Interrupt arrived, check src */
+ isr = snd_sof_dsp_read64(sdev, BYT_DSP_BAR, SHIM_ISRX);
+ if (isr & (SHIM_ISRX_DONE | SHIM_ISRX_BUSY))
+ ret = IRQ_WAKE_THREAD;
+
+ return ret;
+}
+
+static irqreturn_t byt_irq_thread(int irq, void *context)
+{
+ struct snd_sof_dev *sdev = context;
+ u64 ipcx, ipcd;
+ u64 imrx;
+
+ imrx = snd_sof_dsp_read64(sdev, BYT_DSP_BAR, SHIM_IMRX);
+ ipcx = snd_sof_dsp_read64(sdev, BYT_DSP_BAR, SHIM_IPCX);
+
+ /* reply message from DSP */
+ if (ipcx & SHIM_BYT_IPCX_DONE &&
+ !(imrx & SHIM_IMRX_DONE)) {
+ /* Mask Done interrupt before first */
+ snd_sof_dsp_update_bits64_unlocked(sdev, BYT_DSP_BAR,
+ SHIM_IMRX,
+ SHIM_IMRX_DONE,
+ SHIM_IMRX_DONE);
+ /*
+ * handle immediate reply from DSP core. If the msg is
+ * found, set done bit in cmd_done which is called at the
+ * end of message processing function, else set it here
+ * because the done bit can't be set in cmd_done function
+ * which is triggered by msg
+ */
+ byt_get_reply(sdev);
+ snd_sof_ipc_reply(sdev, ipcx);
+
+ byt_dsp_done(sdev);
+ }
+
+ /* new message from DSP */
+ ipcd = snd_sof_dsp_read64(sdev, BYT_DSP_BAR, SHIM_IPCD);
+ if (ipcd & SHIM_BYT_IPCD_BUSY &&
+ !(imrx & SHIM_IMRX_BUSY)) {
+ /* Mask Busy interrupt before return */
+ snd_sof_dsp_update_bits64_unlocked(sdev, BYT_DSP_BAR,
+ SHIM_IMRX,
+ SHIM_IMRX_BUSY,
+ SHIM_IMRX_BUSY);
+
+ /* Handle messages from DSP Core */
+ if ((ipcd & SOF_IPC_PANIC_MAGIC_MASK) == SOF_IPC_PANIC_MAGIC) {
+ snd_sof_dsp_panic(sdev, BYT_PANIC_OFFSET(ipcd) +
+ MBOX_OFFSET);
+ } else {
+ snd_sof_ipc_msgs_rx(sdev);
+ }
+
+ byt_host_done(sdev);
+ }
+
+ return IRQ_HANDLED;
+}
+
+static int byt_send_msg(struct snd_sof_dev *sdev, struct snd_sof_ipc_msg *msg)
+{
+ u64 cmd = msg->header;
+
+ /* send the message */
+ sof_mailbox_write(sdev, sdev->host_box.offset, msg->msg_data,
+ msg->msg_size);
+ snd_sof_dsp_write64(sdev, BYT_DSP_BAR, SHIM_IPCX,
+ cmd | SHIM_BYT_IPCX_BUSY);
+
+ return 0;
+}
+
+static void byt_get_reply(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_ipc_msg *msg = sdev->msg;
+ struct sof_ipc_reply reply;
+ unsigned long flags;
+ int ret = 0;
+
+ /*
+ * Sometimes, there is unexpected reply ipc arriving. The reply
+ * ipc belongs to none of the ipcs sent from driver.
+ * In this case, the driver must ignore the ipc.
+ */
+ if (!msg) {
+ dev_warn(sdev->dev, "unexpected ipc interrupt raised!\n");
+ return;
+ }
+
+ /* get reply */
+ sof_mailbox_read(sdev, sdev->host_box.offset, &reply, sizeof(reply));
+
+ spin_lock_irqsave(&sdev->ipc_lock, flags);
+
+ if (reply.error < 0) {
+ memcpy(msg->reply_data, &reply, sizeof(reply));
+ ret = reply.error;
+ } else {
+ /* reply correct size ? */
+ if (reply.hdr.size != msg->reply_size) {
+ dev_err(sdev->dev, "error: reply expected %zu got %u bytes\n",
+ msg->reply_size, reply.hdr.size);
+ ret = -EINVAL;
+ }
+
+ /* read the message */
+ if (msg->reply_size > 0)
+ sof_mailbox_read(sdev, sdev->host_box.offset,
+ msg->reply_data, msg->reply_size);
+ }
+
+ msg->reply_error = ret;
+
+ spin_unlock_irqrestore(&sdev->ipc_lock, flags);
+}
+
+static void byt_host_done(struct snd_sof_dev *sdev)
+{
+ /* clear BUSY bit and set DONE bit - accept new messages */
+ snd_sof_dsp_update_bits64_unlocked(sdev, BYT_DSP_BAR, SHIM_IPCD,
+ SHIM_BYT_IPCD_BUSY |
+ SHIM_BYT_IPCD_DONE,
+ SHIM_BYT_IPCD_DONE);
+
+ /* unmask busy interrupt */
+ snd_sof_dsp_update_bits64_unlocked(sdev, BYT_DSP_BAR, SHIM_IMRX,
+ SHIM_IMRX_BUSY, 0);
+}
+
+static void byt_dsp_done(struct snd_sof_dev *sdev)
+{
+ /* clear DONE bit - tell DSP we have completed */
+ snd_sof_dsp_update_bits64_unlocked(sdev, BYT_DSP_BAR, SHIM_IPCX,
+ SHIM_BYT_IPCX_DONE, 0);
+
+ /* unmask Done interrupt */
+ snd_sof_dsp_update_bits64_unlocked(sdev, BYT_DSP_BAR, SHIM_IMRX,
+ SHIM_IMRX_DONE, 0);
+}
+
+/*
+ * DSP control.
+ */
+
+static int byt_run(struct snd_sof_dev *sdev)
+{
+ int tries = 10;
+
+ /* release stall and wait to unstall */
+ snd_sof_dsp_update_bits64(sdev, BYT_DSP_BAR, SHIM_CSR,
+ SHIM_BYT_CSR_STALL, 0x0);
+ while (tries--) {
+ if (!(snd_sof_dsp_read64(sdev, BYT_DSP_BAR, SHIM_CSR) &
+ SHIM_BYT_CSR_PWAITMODE))
+ break;
+ msleep(100);
+ }
+ if (tries < 0) {
+ dev_err(sdev->dev, "error: unable to run DSP firmware\n");
+ byt_dump(sdev, SOF_DBG_REGS | SOF_DBG_MBOX);
+ return -ENODEV;
+ }
+
+ /* return init core mask */
+ return 1;
+}
+
+static int byt_reset(struct snd_sof_dev *sdev)
+{
+ /* put DSP into reset, set reset vector and stall */
+ snd_sof_dsp_update_bits64(sdev, BYT_DSP_BAR, SHIM_CSR,
+ SHIM_BYT_CSR_RST | SHIM_BYT_CSR_VECTOR_SEL |
+ SHIM_BYT_CSR_STALL,
+ SHIM_BYT_CSR_RST | SHIM_BYT_CSR_VECTOR_SEL |
+ SHIM_BYT_CSR_STALL);
+
+ usleep_range(10, 15);
+
+ /* take DSP out of reset and keep stalled for FW loading */
+ snd_sof_dsp_update_bits64(sdev, BYT_DSP_BAR, SHIM_CSR,
+ SHIM_BYT_CSR_RST, 0);
+
+ return 0;
+}
+
+/* Baytrail DAIs */
+static struct snd_soc_dai_driver byt_dai[] = {
+{
+ .name = "ssp0-port",
+},
+{
+ .name = "ssp1-port",
+},
+{
+ .name = "ssp2-port",
+},
+{
+ .name = "ssp3-port",
+},
+{
+ .name = "ssp4-port",
+},
+{
+ .name = "ssp5-port",
+},
+};
+
+/*
+ * Probe and remove.
+ */
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_MERRIFIELD)
+
+static int tangier_pci_probe(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_pdata *pdata = sdev->pdata;
+ const struct sof_dev_desc *desc = pdata->desc;
+ struct pci_dev *pci = to_pci_dev(sdev->dev);
+ u32 base, size;
+ int ret;
+
+ /* DSP DMA can only access low 31 bits of host memory */
+ ret = dma_coerce_mask_and_coherent(&pci->dev, DMA_BIT_MASK(31));
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to set DMA mask %d\n", ret);
+ return ret;
+ }
+
+ /* LPE base */
+ base = pci_resource_start(pci, desc->resindex_lpe_base) - IRAM_OFFSET;
+ size = BYT_PCI_BAR_SIZE;
+
+ dev_dbg(sdev->dev, "LPE PHY base at 0x%x size 0x%x", base, size);
+ sdev->bar[BYT_DSP_BAR] = devm_ioremap(sdev->dev, base, size);
+ if (!sdev->bar[BYT_DSP_BAR]) {
+ dev_err(sdev->dev, "error: failed to ioremap LPE base 0x%x size 0x%x\n",
+ base, size);
+ return -ENODEV;
+ }
+ dev_dbg(sdev->dev, "LPE VADDR %p\n", sdev->bar[BYT_DSP_BAR]);
+
+ /* IMR base - optional */
+ if (desc->resindex_imr_base == -1)
+ goto irq;
+
+ base = pci_resource_start(pci, desc->resindex_imr_base);
+ size = pci_resource_len(pci, desc->resindex_imr_base);
+
+ /* some BIOSes don't map IMR */
+ if (base == 0x55aa55aa || base == 0x0) {
+ dev_info(sdev->dev, "IMR not set by BIOS. Ignoring\n");
+ goto irq;
+ }
+
+ dev_dbg(sdev->dev, "IMR base at 0x%x size 0x%x", base, size);
+ sdev->bar[BYT_IMR_BAR] = devm_ioremap(sdev->dev, base, size);
+ if (!sdev->bar[BYT_IMR_BAR]) {
+ dev_err(sdev->dev, "error: failed to ioremap IMR base 0x%x size 0x%x\n",
+ base, size);
+ return -ENODEV;
+ }
+ dev_dbg(sdev->dev, "IMR VADDR %p\n", sdev->bar[BYT_IMR_BAR]);
+
+irq:
+ /* register our IRQ */
+ sdev->ipc_irq = pci->irq;
+ dev_dbg(sdev->dev, "using IRQ %d\n", sdev->ipc_irq);
+ ret = devm_request_threaded_irq(sdev->dev, sdev->ipc_irq,
+ byt_irq_handler, byt_irq_thread,
+ 0, "AudioDSP", sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to register IRQ %d\n",
+ sdev->ipc_irq);
+ return ret;
+ }
+
+ /* enable Interrupt from both sides */
+ snd_sof_dsp_update_bits64(sdev, BYT_DSP_BAR, SHIM_IMRX, 0x3, 0x0);
+ snd_sof_dsp_update_bits64(sdev, BYT_DSP_BAR, SHIM_IMRD, 0x3, 0x0);
+
+ /* set default mailbox offset for FW ready message */
+ sdev->dsp_box.offset = MBOX_OFFSET;
+
+ return ret;
+}
+
+const struct snd_sof_dsp_ops sof_tng_ops = {
+ /* device init */
+ .probe = tangier_pci_probe,
+
+ /* DSP core boot / reset */
+ .run = byt_run,
+ .reset = byt_reset,
+
+ /* Register IO */
+ .write = sof_io_write,
+ .read = sof_io_read,
+ .write64 = sof_io_write64,
+ .read64 = sof_io_read64,
+
+ /* Block IO */
+ .block_read = sof_block_read,
+ .block_write = sof_block_write,
+
+ /* doorbell */
+ .irq_handler = byt_irq_handler,
+ .irq_thread = byt_irq_thread,
+
+ /* ipc */
+ .send_msg = byt_send_msg,
+ .fw_ready = byt_fw_ready,
+
+ .ipc_msg_data = intel_ipc_msg_data,
+ .ipc_pcm_params = intel_ipc_pcm_params,
+
+ /* debug */
+ .debug_map = byt_debugfs,
+ .debug_map_count = ARRAY_SIZE(byt_debugfs),
+ .dbg_dump = byt_dump,
+
+ /* stream callbacks */
+ .pcm_open = intel_pcm_open,
+ .pcm_close = intel_pcm_close,
+
+ /* module loading */
+ .load_module = snd_sof_parse_module_memcpy,
+
+ /*Firmware loading */
+ .load_firmware = snd_sof_load_firmware_memcpy,
+
+ /* DAI drivers */
+ .drv = byt_dai,
+ .num_drv = 3, /* we have only 3 SSPs on byt*/
+};
+EXPORT_SYMBOL(sof_tng_ops);
+
+const struct sof_intel_dsp_desc tng_chip_info = {
+ .cores_num = 1,
+ .cores_mask = 1,
+};
+EXPORT_SYMBOL(tng_chip_info);
+
+#endif /* CONFIG_SND_SOC_SOF_MERRIFIELD */
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_BAYTRAIL)
+
+static int byt_acpi_probe(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_pdata *pdata = sdev->pdata;
+ const struct sof_dev_desc *desc = pdata->desc;
+ struct platform_device *pdev =
+ container_of(sdev->dev, struct platform_device, dev);
+ struct resource *mmio;
+ u32 base, size;
+ int ret;
+
+ /* DSP DMA can only access low 31 bits of host memory */
+ ret = dma_coerce_mask_and_coherent(sdev->dev, DMA_BIT_MASK(31));
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to set DMA mask %d\n", ret);
+ return ret;
+ }
+
+ /* LPE base */
+ mmio = platform_get_resource(pdev, IORESOURCE_MEM,
+ desc->resindex_lpe_base);
+ if (mmio) {
+ base = mmio->start;
+ size = resource_size(mmio);
+ } else {
+ dev_err(sdev->dev, "error: failed to get LPE base at idx %d\n",
+ desc->resindex_lpe_base);
+ return -EINVAL;
+ }
+
+ dev_dbg(sdev->dev, "LPE PHY base at 0x%x size 0x%x", base, size);
+ sdev->bar[BYT_DSP_BAR] = devm_ioremap(sdev->dev, base, size);
+ if (!sdev->bar[BYT_DSP_BAR]) {
+ dev_err(sdev->dev, "error: failed to ioremap LPE base 0x%x size 0x%x\n",
+ base, size);
+ return -ENODEV;
+ }
+ dev_dbg(sdev->dev, "LPE VADDR %p\n", sdev->bar[BYT_DSP_BAR]);
+
+ /* TODO: add offsets */
+ sdev->mmio_bar = BYT_DSP_BAR;
+ sdev->mailbox_bar = BYT_DSP_BAR;
+
+ /* IMR base - optional */
+ if (desc->resindex_imr_base == -1)
+ goto irq;
+
+ mmio = platform_get_resource(pdev, IORESOURCE_MEM,
+ desc->resindex_imr_base);
+ if (mmio) {
+ base = mmio->start;
+ size = resource_size(mmio);
+ } else {
+ dev_err(sdev->dev, "error: failed to get IMR base at idx %d\n",
+ desc->resindex_imr_base);
+ return -ENODEV;
+ }
+
+ /* some BIOSes don't map IMR */
+ if (base == 0x55aa55aa || base == 0x0) {
+ dev_info(sdev->dev, "IMR not set by BIOS. Ignoring\n");
+ goto irq;
+ }
+
+ dev_dbg(sdev->dev, "IMR base at 0x%x size 0x%x", base, size);
+ sdev->bar[BYT_IMR_BAR] = devm_ioremap(sdev->dev, base, size);
+ if (!sdev->bar[BYT_IMR_BAR]) {
+ dev_err(sdev->dev, "error: failed to ioremap IMR base 0x%x size 0x%x\n",
+ base, size);
+ return -ENODEV;
+ }
+ dev_dbg(sdev->dev, "IMR VADDR %p\n", sdev->bar[BYT_IMR_BAR]);
+
+irq:
+ /* register our IRQ */
+ sdev->ipc_irq = platform_get_irq(pdev, desc->irqindex_host_ipc);
+ if (sdev->ipc_irq < 0) {
+ dev_err(sdev->dev, "error: failed to get IRQ at index %d\n",
+ desc->irqindex_host_ipc);
+ return sdev->ipc_irq;
+ }
+
+ dev_dbg(sdev->dev, "using IRQ %d\n", sdev->ipc_irq);
+ ret = devm_request_threaded_irq(sdev->dev, sdev->ipc_irq,
+ byt_irq_handler, byt_irq_thread,
+ IRQF_SHARED, "AudioDSP", sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to register IRQ %d\n",
+ sdev->ipc_irq);
+ return ret;
+ }
+
+ /* enable Interrupt from both sides */
+ snd_sof_dsp_update_bits64(sdev, BYT_DSP_BAR, SHIM_IMRX, 0x3, 0x0);
+ snd_sof_dsp_update_bits64(sdev, BYT_DSP_BAR, SHIM_IMRD, 0x3, 0x0);
+
+ /* set default mailbox offset for FW ready message */
+ sdev->dsp_box.offset = MBOX_OFFSET;
+
+ return ret;
+}
+
+/* baytrail ops */
+const struct snd_sof_dsp_ops sof_byt_ops = {
+ /* device init */
+ .probe = byt_acpi_probe,
+
+ /* DSP core boot / reset */
+ .run = byt_run,
+ .reset = byt_reset,
+
+ /* Register IO */
+ .write = sof_io_write,
+ .read = sof_io_read,
+ .write64 = sof_io_write64,
+ .read64 = sof_io_read64,
+
+ /* Block IO */
+ .block_read = sof_block_read,
+ .block_write = sof_block_write,
+
+ /* doorbell */
+ .irq_handler = byt_irq_handler,
+ .irq_thread = byt_irq_thread,
+
+ /* ipc */
+ .send_msg = byt_send_msg,
+ .fw_ready = byt_fw_ready,
+
+ .ipc_msg_data = intel_ipc_msg_data,
+ .ipc_pcm_params = intel_ipc_pcm_params,
+
+ /* debug */
+ .debug_map = byt_debugfs,
+ .debug_map_count = ARRAY_SIZE(byt_debugfs),
+ .dbg_dump = byt_dump,
+
+ /* stream callbacks */
+ .pcm_open = intel_pcm_open,
+ .pcm_close = intel_pcm_close,
+
+ /* module loading */
+ .load_module = snd_sof_parse_module_memcpy,
+
+ /*Firmware loading */
+ .load_firmware = snd_sof_load_firmware_memcpy,
+
+ /* DAI drivers */
+ .drv = byt_dai,
+ .num_drv = 3, /* we have only 3 SSPs on byt*/
+};
+EXPORT_SYMBOL(sof_byt_ops);
+
+const struct sof_intel_dsp_desc byt_chip_info = {
+ .cores_num = 1,
+ .cores_mask = 1,
+};
+EXPORT_SYMBOL(byt_chip_info);
+
+/* cherrytrail and braswell ops */
+const struct snd_sof_dsp_ops sof_cht_ops = {
+ /* device init */
+ .probe = byt_acpi_probe,
+
+ /* DSP core boot / reset */
+ .run = byt_run,
+ .reset = byt_reset,
+
+ /* Register IO */
+ .write = sof_io_write,
+ .read = sof_io_read,
+ .write64 = sof_io_write64,
+ .read64 = sof_io_read64,
+
+ /* Block IO */
+ .block_read = sof_block_read,
+ .block_write = sof_block_write,
+
+ /* doorbell */
+ .irq_handler = byt_irq_handler,
+ .irq_thread = byt_irq_thread,
+
+ /* ipc */
+ .send_msg = byt_send_msg,
+ .fw_ready = byt_fw_ready,
+
+ .ipc_msg_data = intel_ipc_msg_data,
+ .ipc_pcm_params = intel_ipc_pcm_params,
+
+ /* debug */
+ .debug_map = cht_debugfs,
+ .debug_map_count = ARRAY_SIZE(cht_debugfs),
+ .dbg_dump = byt_dump,
+
+ /* stream callbacks */
+ .pcm_open = intel_pcm_open,
+ .pcm_close = intel_pcm_close,
+
+ /* module loading */
+ .load_module = snd_sof_parse_module_memcpy,
+
+ /*Firmware loading */
+ .load_firmware = snd_sof_load_firmware_memcpy,
+
+ /* DAI drivers */
+ .drv = byt_dai,
+ /* all 6 SSPs may be available for cherrytrail */
+ .num_drv = ARRAY_SIZE(byt_dai),
+};
+EXPORT_SYMBOL(sof_cht_ops);
+
+const struct sof_intel_dsp_desc cht_chip_info = {
+ .cores_num = 1,
+ .cores_mask = 1,
+};
+EXPORT_SYMBOL(cht_chip_info);
+
+#endif /* CONFIG_SND_SOC_SOF_BAYTRAIL */
+
+MODULE_LICENSE("Dual BSD/GPL");
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+// Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
+// Rander Wang <rander.wang@intel.com>
+// Keyon Jie <yang.jie@linux.intel.com>
+//
+
+/*
+ * Hardware interface for audio DSP on Cannonlake.
+ */
+
+#include "../ops.h"
+#include "hda.h"
+
+static const struct snd_sof_debugfs_map cnl_dsp_debugfs[] = {
+ {"hda", HDA_DSP_HDA_BAR, 0, 0x4000, SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"pp", HDA_DSP_PP_BAR, 0, 0x1000, SOF_DEBUGFS_ACCESS_ALWAYS},
+ {"dsp", HDA_DSP_BAR, 0, 0x10000, SOF_DEBUGFS_ACCESS_ALWAYS},
+};
+
+static void cnl_ipc_host_done(struct snd_sof_dev *sdev);
+static void cnl_ipc_dsp_done(struct snd_sof_dev *sdev);
+
+static irqreturn_t cnl_ipc_irq_thread(int irq, void *context)
+{
+ struct snd_sof_dev *sdev = context;
+ u32 hipci;
+ u32 hipcctl;
+ u32 hipcida;
+ u32 hipctdr;
+ u32 hipctdd;
+ u32 msg;
+ u32 msg_ext;
+ irqreturn_t ret = IRQ_NONE;
+
+ hipcida = snd_sof_dsp_read(sdev, HDA_DSP_BAR, CNL_DSP_REG_HIPCIDA);
+ hipcctl = snd_sof_dsp_read(sdev, HDA_DSP_BAR, CNL_DSP_REG_HIPCCTL);
+ hipctdr = snd_sof_dsp_read(sdev, HDA_DSP_BAR, CNL_DSP_REG_HIPCTDR);
+
+ /* reenable IPC interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR, HDA_DSP_REG_ADSPIC,
+ HDA_DSP_ADSPIC_IPC, HDA_DSP_ADSPIC_IPC);
+
+ /* reply message from DSP */
+ if (hipcida & CNL_DSP_REG_HIPCIDA_DONE &&
+ hipcctl & CNL_DSP_REG_HIPCCTL_DONE) {
+ hipci = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ CNL_DSP_REG_HIPCIDR);
+ msg_ext = hipci & CNL_DSP_REG_HIPCIDR_MSG_MASK;
+ msg = hipcida & CNL_DSP_REG_HIPCIDA_MSG_MASK;
+
+ dev_vdbg(sdev->dev,
+ "ipc: firmware response, msg:0x%x, msg_ext:0x%x\n",
+ msg, msg_ext);
+
+ /* mask Done interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR,
+ CNL_DSP_REG_HIPCCTL,
+ CNL_DSP_REG_HIPCCTL_DONE, 0);
+
+ /* handle immediate reply from DSP core */
+ hda_dsp_ipc_get_reply(sdev);
+ snd_sof_ipc_reply(sdev, msg);
+
+ if (sdev->code_loading) {
+ sdev->code_loading = 0;
+ wake_up(&sdev->waitq);
+ }
+
+ cnl_ipc_dsp_done(sdev);
+
+ ret = IRQ_HANDLED;
+ }
+
+ /* new message from DSP */
+ if (hipctdr & CNL_DSP_REG_HIPCTDR_BUSY) {
+ hipctdd = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ CNL_DSP_REG_HIPCTDD);
+ msg = hipctdr & CNL_DSP_REG_HIPCTDR_MSG_MASK;
+ msg_ext = hipctdd & CNL_DSP_REG_HIPCTDD_MSG_MASK;
+
+ dev_vdbg(sdev->dev,
+ "ipc: firmware initiated, msg:0x%x, msg_ext:0x%x\n",
+ msg, msg_ext);
+
+ /* handle messages from DSP */
+ if ((hipctdr & SOF_IPC_PANIC_MAGIC_MASK) ==
+ SOF_IPC_PANIC_MAGIC) {
+ snd_sof_dsp_panic(sdev, HDA_DSP_PANIC_OFFSET(msg_ext));
+ } else {
+ snd_sof_ipc_msgs_rx(sdev);
+ }
+
+ /*
+ * clear busy interrupt to tell dsp controller this
+ * interrupt has been accepted, not trigger it again
+ */
+ snd_sof_dsp_update_bits_forced(sdev, HDA_DSP_BAR,
+ CNL_DSP_REG_HIPCTDR,
+ CNL_DSP_REG_HIPCTDR_BUSY,
+ CNL_DSP_REG_HIPCTDR_BUSY);
+
+ cnl_ipc_host_done(sdev);
+
+ ret = IRQ_HANDLED;
+ }
+
+ return ret;
+}
+
+static void cnl_ipc_host_done(struct snd_sof_dev *sdev)
+{
+ /*
+ * set done bit to ack dsp the msg has been
+ * processed and send reply msg to dsp
+ */
+ snd_sof_dsp_update_bits_forced(sdev, HDA_DSP_BAR,
+ CNL_DSP_REG_HIPCTDA,
+ CNL_DSP_REG_HIPCTDA_DONE,
+ CNL_DSP_REG_HIPCTDA_DONE);
+}
+
+static void cnl_ipc_dsp_done(struct snd_sof_dev *sdev)
+{
+ /*
+ * set DONE bit - tell DSP we have received the reply msg
+ * from DSP, and processed it, don't send more reply to host
+ */
+ snd_sof_dsp_update_bits_forced(sdev, HDA_DSP_BAR,
+ CNL_DSP_REG_HIPCIDA,
+ CNL_DSP_REG_HIPCIDA_DONE,
+ CNL_DSP_REG_HIPCIDA_DONE);
+
+ /* unmask Done interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR,
+ CNL_DSP_REG_HIPCCTL,
+ CNL_DSP_REG_HIPCCTL_DONE,
+ CNL_DSP_REG_HIPCCTL_DONE);
+}
+
+static int cnl_ipc_send_msg(struct snd_sof_dev *sdev,
+ struct snd_sof_ipc_msg *msg)
+{
+ u32 cmd = msg->header;
+
+ /* send the message */
+ sof_mailbox_write(sdev, sdev->host_box.offset, msg->msg_data,
+ msg->msg_size);
+ snd_sof_dsp_write(sdev, HDA_DSP_BAR, CNL_DSP_REG_HIPCIDR,
+ cmd | CNL_DSP_REG_HIPCIDR_BUSY);
+
+ return 0;
+}
+
+static void cnl_ipc_dump(struct snd_sof_dev *sdev)
+{
+ u32 hipcctl;
+ u32 hipcida;
+ u32 hipctdr;
+
+ /* read IPC status */
+ hipcida = snd_sof_dsp_read(sdev, HDA_DSP_BAR, CNL_DSP_REG_HIPCIDA);
+ hipcctl = snd_sof_dsp_read(sdev, HDA_DSP_BAR, CNL_DSP_REG_HIPCCTL);
+ hipctdr = snd_sof_dsp_read(sdev, HDA_DSP_BAR, CNL_DSP_REG_HIPCTDR);
+
+ /* dump the IPC regs */
+ /* TODO: parse the raw msg */
+ dev_err(sdev->dev,
+ "error: host status 0x%8.8x dsp status 0x%8.8x mask 0x%8.8x\n",
+ hipcida, hipctdr, hipcctl);
+}
+
+/* cannonlake ops */
+const struct snd_sof_dsp_ops sof_cnl_ops = {
+ /* probe and remove */
+ .probe = hda_dsp_probe,
+ .remove = hda_dsp_remove,
+
+ /* Register IO */
+ .write = sof_io_write,
+ .read = sof_io_read,
+ .write64 = sof_io_write64,
+ .read64 = sof_io_read64,
+
+ /* Block IO */
+ .block_read = sof_block_read,
+ .block_write = sof_block_write,
+
+ /* doorbell */
+ .irq_handler = hda_dsp_ipc_irq_handler,
+ .irq_thread = cnl_ipc_irq_thread,
+
+ /* ipc */
+ .send_msg = cnl_ipc_send_msg,
+ .fw_ready = hda_dsp_ipc_fw_ready,
+
+ .ipc_msg_data = hda_ipc_msg_data,
+ .ipc_pcm_params = hda_ipc_pcm_params,
+
+ /* debug */
+ .debug_map = cnl_dsp_debugfs,
+ .debug_map_count = ARRAY_SIZE(cnl_dsp_debugfs),
+ .dbg_dump = hda_dsp_dump,
+ .ipc_dump = cnl_ipc_dump,
+
+ /* stream callbacks */
+ .pcm_open = hda_dsp_pcm_open,
+ .pcm_close = hda_dsp_pcm_close,
+ .pcm_hw_params = hda_dsp_pcm_hw_params,
+ .pcm_trigger = hda_dsp_pcm_trigger,
+ .pcm_pointer = hda_dsp_pcm_pointer,
+
+ /* firmware loading */
+ .load_firmware = snd_sof_load_firmware_raw,
+
+ /* pre/post fw run */
+ .pre_fw_run = hda_dsp_pre_fw_run,
+ .post_fw_run = hda_dsp_post_fw_run,
+
+ /* dsp core power up/down */
+ .core_power_up = hda_dsp_enable_core,
+ .core_power_down = hda_dsp_core_reset_power_down,
+
+ /* firmware run */
+ .run = hda_dsp_cl_boot_firmware,
+
+ /* trace callback */
+ .trace_init = hda_dsp_trace_init,
+ .trace_release = hda_dsp_trace_release,
+ .trace_trigger = hda_dsp_trace_trigger,
+
+ /* DAI drivers */
+ .drv = skl_dai,
+ .num_drv = SOF_SKL_NUM_DAIS,
+
+ /* PM */
+ .suspend = hda_dsp_suspend,
+ .resume = hda_dsp_resume,
+ .runtime_suspend = hda_dsp_runtime_suspend,
+ .runtime_resume = hda_dsp_runtime_resume,
+ .set_hw_params_upon_resume = hda_dsp_set_hw_params_upon_resume,
+};
+EXPORT_SYMBOL(sof_cnl_ops);
+
+const struct sof_intel_dsp_desc cnl_chip_info = {
+ /* Cannonlake */
+ .cores_num = 4,
+ .init_core_mask = 1,
+ .cores_mask = HDA_DSP_CORE_MASK(0) |
+ HDA_DSP_CORE_MASK(1) |
+ HDA_DSP_CORE_MASK(2) |
+ HDA_DSP_CORE_MASK(3),
+ .ipc_req = CNL_DSP_REG_HIPCIDR,
+ .ipc_req_mask = CNL_DSP_REG_HIPCIDR_BUSY,
+ .ipc_ack = CNL_DSP_REG_HIPCIDA,
+ .ipc_ack_mask = CNL_DSP_REG_HIPCIDA_DONE,
+ .ipc_ctl = CNL_DSP_REG_HIPCCTL,
+ .rom_init_timeout = 300,
+ .ssp_count = CNL_SSP_COUNT,
+ .ssp_base_offset = CNL_SSP_BASE_OFFSET,
+};
+EXPORT_SYMBOL(cnl_chip_info);
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Keyon Jie <yang.jie@linux.intel.com>
+
+#include <linux/io.h>
+#include <sound/hdaudio.h>
+#include "../sof-priv.h"
+#include "hda.h"
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+
+static const struct hdac_bus_ops bus_ops = {
+ .command = snd_hdac_bus_send_cmd,
+ .get_response = snd_hdac_bus_get_response,
+};
+
+#endif
+
+static void sof_hda_writel(u32 value, u32 __iomem *addr)
+{
+ writel(value, addr);
+}
+
+static u32 sof_hda_readl(u32 __iomem *addr)
+{
+ return readl(addr);
+}
+
+static void sof_hda_writew(u16 value, u16 __iomem *addr)
+{
+ writew(value, addr);
+}
+
+static u16 sof_hda_readw(u16 __iomem *addr)
+{
+ return readw(addr);
+}
+
+static void sof_hda_writeb(u8 value, u8 __iomem *addr)
+{
+ writeb(value, addr);
+}
+
+static u8 sof_hda_readb(u8 __iomem *addr)
+{
+ return readb(addr);
+}
+
+static int sof_hda_dma_alloc_pages(struct hdac_bus *bus, int type,
+ size_t size, struct snd_dma_buffer *buf)
+{
+ return snd_dma_alloc_pages(type, bus->dev, size, buf);
+}
+
+static void sof_hda_dma_free_pages(struct hdac_bus *bus,
+ struct snd_dma_buffer *buf)
+{
+ snd_dma_free_pages(buf);
+}
+
+static const struct hdac_io_ops io_ops = {
+ .reg_writel = sof_hda_writel,
+ .reg_readl = sof_hda_readl,
+ .reg_writew = sof_hda_writew,
+ .reg_readw = sof_hda_readw,
+ .reg_writeb = sof_hda_writeb,
+ .reg_readb = sof_hda_readb,
+ .dma_alloc_pages = sof_hda_dma_alloc_pages,
+ .dma_free_pages = sof_hda_dma_free_pages,
+};
+
+/*
+ * This can be used for both with/without hda link support.
+ */
+void sof_hda_bus_init(struct hdac_bus *bus, struct device *dev,
+ const struct hdac_ext_bus_ops *ext_ops)
+{
+ memset(bus, 0, sizeof(*bus));
+ bus->dev = dev;
+
+ bus->io_ops = &io_ops;
+ INIT_LIST_HEAD(&bus->stream_list);
+
+ bus->irq = -1;
+ bus->ext_ops = ext_ops;
+
+ /*
+ * There is only one HDA bus atm. keep the index as 0.
+ * Need to fix when there are more than one HDA bus.
+ */
+ bus->idx = 0;
+
+ spin_lock_init(&bus->reg_lock);
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ INIT_LIST_HEAD(&bus->codec_list);
+ INIT_LIST_HEAD(&bus->hlink_list);
+
+ mutex_init(&bus->cmd_mutex);
+ mutex_init(&bus->lock);
+ bus->ops = &bus_ops;
+ INIT_WORK(&bus->unsol_work, snd_hdac_bus_process_unsol_events);
+ bus->cmd_dma_state = true;
+#endif
+
+}
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Keyon Jie <yang.jie@linux.intel.com>
+//
+
+#include <linux/module.h>
+#include <sound/hdaudio_ext.h>
+#include <sound/hda_codec.h>
+#include <sound/hda_i915.h>
+#include <sound/sof.h>
+#include "../ops.h"
+#include "hda.h"
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA_AUDIO_CODEC)
+#include "../../codecs/hdac_hda.h"
+#endif /* CONFIG_SND_SOC_SOF_HDA_AUDIO_CODEC */
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA_AUDIO_CODEC)
+#define IDISP_VID_INTEL 0x80860000
+
+/* load the legacy HDA codec driver */
+#ifdef MODULE
+static void hda_codec_load_module(struct hda_codec *codec)
+{
+ char alias[MODULE_NAME_LEN];
+ const char *module = alias;
+
+ snd_hdac_codec_modalias(&codec->core, alias, sizeof(alias));
+ dev_dbg(&codec->core.dev, "loading codec module: %s\n", module);
+ request_module(module);
+}
+#else
+static void hda_codec_load_module(struct hda_codec *codec) {}
+#endif
+
+#endif /* CONFIG_SND_SOC_SOF_HDA_AUDIO_CODEC */
+
+/* probe individual codec */
+static int hda_codec_probe(struct snd_sof_dev *sdev, int address)
+{
+ struct hda_bus *hbus = sof_to_hbus(sdev);
+ struct hdac_device *hdev;
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA_AUDIO_CODEC)
+ struct hdac_hda_priv *hda_priv;
+#endif
+ u32 hda_cmd = (address << 28) | (AC_NODE_ROOT << 20) |
+ (AC_VERB_PARAMETERS << 8) | AC_PAR_VENDOR_ID;
+ u32 resp = -1;
+ int ret;
+
+ mutex_lock(&hbus->core.cmd_mutex);
+ snd_hdac_bus_send_cmd(&hbus->core, hda_cmd);
+ snd_hdac_bus_get_response(&hbus->core, address, &resp);
+ mutex_unlock(&hbus->core.cmd_mutex);
+ if (resp == -1)
+ return -EIO;
+ dev_dbg(sdev->dev, "HDA codec #%d probed OK: response: %x\n",
+ address, resp);
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA_AUDIO_CODEC)
+ /* snd_hdac_ext_bus_device_exit will use kfree to free hdev */
+ hda_priv = kzalloc(sizeof(*hda_priv), GFP_KERNEL);
+ if (!hda_priv)
+ return -ENOMEM;
+
+ hda_priv->codec.bus = hbus;
+ hdev = &hda_priv->codec.core;
+
+ ret = snd_hdac_ext_bus_device_init(&hbus->core, address, hdev);
+ if (ret < 0)
+ return ret;
+
+ /* use legacy bus only for HDA codecs, idisp uses ext bus */
+ if ((resp & 0xFFFF0000) != IDISP_VID_INTEL) {
+ hdev->type = HDA_DEV_LEGACY;
+ hda_codec_load_module(&hda_priv->codec);
+ }
+
+ return 0;
+#else
+ /* snd_hdac_ext_bus_device_exit will use kfree to free hdev */
+ hdev = kzalloc(sizeof(*hdev), GFP_KERNEL);
+ if (!hdev)
+ return -ENOMEM;
+
+ ret = snd_hdac_ext_bus_device_init(&hbus->core, address, hdev);
+
+ return ret;
+#endif
+}
+
+/* Codec initialization */
+int hda_codec_probe_bus(struct snd_sof_dev *sdev)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ int i, ret;
+
+ /* probe codecs in avail slots */
+ for (i = 0; i < HDA_MAX_CODECS; i++) {
+
+ if (!(bus->codec_mask & (1 << i)))
+ continue;
+
+ ret = hda_codec_probe(sdev, i);
+ if (ret < 0) {
+ dev_err(bus->dev, "error: codec #%d probe error, ret: %d\n",
+ i, ret);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(hda_codec_probe_bus);
+
+#if IS_ENABLED(CONFIG_SND_SOC_HDAC_HDMI)
+
+void hda_codec_i915_get(struct snd_sof_dev *sdev)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+
+ dev_dbg(bus->dev, "Turning i915 HDAC power on\n");
+ snd_hdac_display_power(bus, HDA_CODEC_IDX_CONTROLLER, true);
+}
+EXPORT_SYMBOL(hda_codec_i915_get);
+
+void hda_codec_i915_put(struct snd_sof_dev *sdev)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+
+ dev_dbg(bus->dev, "Turning i915 HDAC power off\n");
+ snd_hdac_display_power(bus, HDA_CODEC_IDX_CONTROLLER, false);
+}
+EXPORT_SYMBOL(hda_codec_i915_put);
+
+int hda_codec_i915_init(struct snd_sof_dev *sdev)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ int ret;
+
+ /* i915 exposes a HDA codec for HDMI audio */
+ ret = snd_hdac_i915_init(bus);
+ if (ret < 0)
+ return ret;
+
+ hda_codec_i915_get(sdev);
+
+ return 0;
+}
+EXPORT_SYMBOL(hda_codec_i915_init);
+
+int hda_codec_i915_exit(struct snd_sof_dev *sdev)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ int ret;
+
+ hda_codec_i915_put(sdev);
+
+ ret = snd_hdac_i915_exit(bus);
+
+ return ret;
+}
+EXPORT_SYMBOL(hda_codec_i915_exit);
+
+#endif /* CONFIG_SND_SOC_HDAC_HDMI */
+
+MODULE_LICENSE("Dual BSD/GPL");
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+// Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
+// Rander Wang <rander.wang@intel.com>
+// Keyon Jie <yang.jie@linux.intel.com>
+//
+
+/*
+ * Hardware interface for generic Intel audio DSP HDA IP
+ */
+
+#include <sound/hdaudio_ext.h>
+#include <sound/hda_register.h>
+#include "../ops.h"
+#include "hda.h"
+
+/*
+ * HDA Operations.
+ */
+
+int hda_dsp_ctrl_link_reset(struct snd_sof_dev *sdev, bool reset)
+{
+ unsigned long timeout;
+ u32 gctl = 0;
+ u32 val;
+
+ /* 0 to enter reset and 1 to exit reset */
+ val = reset ? 0 : SOF_HDA_GCTL_RESET;
+
+ /* enter/exit HDA controller reset */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, SOF_HDA_GCTL,
+ SOF_HDA_GCTL_RESET, val);
+
+ /* wait to enter/exit reset */
+ timeout = jiffies + msecs_to_jiffies(HDA_DSP_CTRL_RESET_TIMEOUT);
+ while (time_before(jiffies, timeout)) {
+ gctl = snd_sof_dsp_read(sdev, HDA_DSP_HDA_BAR, SOF_HDA_GCTL);
+ if ((gctl & SOF_HDA_GCTL_RESET) == val)
+ return 0;
+ usleep_range(500, 1000);
+ }
+
+ /* enter/exit reset failed */
+ dev_err(sdev->dev, "error: failed to %s HDA controller gctl 0x%x\n",
+ reset ? "reset" : "ready", gctl);
+ return -EIO;
+}
+
+int hda_dsp_ctrl_get_caps(struct snd_sof_dev *sdev)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ u32 cap, offset, feature;
+ int count = 0;
+
+ offset = snd_sof_dsp_read(sdev, HDA_DSP_HDA_BAR, SOF_HDA_LLCH);
+
+ do {
+ cap = snd_sof_dsp_read(sdev, HDA_DSP_HDA_BAR, offset);
+
+ dev_dbg(sdev->dev, "checking for capabilities at offset 0x%x\n",
+ offset & SOF_HDA_CAP_NEXT_MASK);
+
+ feature = (cap & SOF_HDA_CAP_ID_MASK) >> SOF_HDA_CAP_ID_OFF;
+
+ switch (feature) {
+ case SOF_HDA_PP_CAP_ID:
+ dev_dbg(sdev->dev, "found DSP capability at 0x%x\n",
+ offset);
+ bus->ppcap = bus->remap_addr + offset;
+ sdev->bar[HDA_DSP_PP_BAR] = bus->ppcap;
+ break;
+ case SOF_HDA_SPIB_CAP_ID:
+ dev_dbg(sdev->dev, "found SPIB capability at 0x%x\n",
+ offset);
+ bus->spbcap = bus->remap_addr + offset;
+ sdev->bar[HDA_DSP_SPIB_BAR] = bus->spbcap;
+ break;
+ case SOF_HDA_DRSM_CAP_ID:
+ dev_dbg(sdev->dev, "found DRSM capability at 0x%x\n",
+ offset);
+ bus->drsmcap = bus->remap_addr + offset;
+ sdev->bar[HDA_DSP_DRSM_BAR] = bus->drsmcap;
+ break;
+ case SOF_HDA_GTS_CAP_ID:
+ dev_dbg(sdev->dev, "found GTS capability at 0x%x\n",
+ offset);
+ bus->gtscap = bus->remap_addr + offset;
+ break;
+ case SOF_HDA_ML_CAP_ID:
+ dev_dbg(sdev->dev, "found ML capability at 0x%x\n",
+ offset);
+ bus->mlcap = bus->remap_addr + offset;
+ break;
+ default:
+ dev_vdbg(sdev->dev, "found capability %d at 0x%x\n",
+ feature, offset);
+ break;
+ }
+
+ offset = cap & SOF_HDA_CAP_NEXT_MASK;
+ } while (count++ <= SOF_HDA_MAX_CAPS && offset);
+
+ return 0;
+}
+
+void hda_dsp_ctrl_ppcap_enable(struct snd_sof_dev *sdev, bool enable)
+{
+ u32 val = enable ? SOF_HDA_PPCTL_GPROCEN : 0;
+
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_PP_BAR, SOF_HDA_REG_PP_PPCTL,
+ SOF_HDA_PPCTL_GPROCEN, val);
+}
+
+void hda_dsp_ctrl_ppcap_int_enable(struct snd_sof_dev *sdev, bool enable)
+{
+ u32 val = enable ? SOF_HDA_PPCTL_PIE : 0;
+
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_PP_BAR, SOF_HDA_REG_PP_PPCTL,
+ SOF_HDA_PPCTL_PIE, val);
+}
+
+void hda_dsp_ctrl_misc_clock_gating(struct snd_sof_dev *sdev, bool enable)
+{
+ u32 val = enable ? PCI_CGCTL_MISCBDCGE_MASK : 0;
+
+ snd_sof_pci_update_bits(sdev, PCI_CGCTL, PCI_CGCTL_MISCBDCGE_MASK, val);
+}
+
+/*
+ * enable/disable audio dsp clock gating and power gating bits.
+ * This allows the HW to opportunistically power and clock gate
+ * the audio dsp when it is idle
+ */
+int hda_dsp_ctrl_clock_power_gating(struct snd_sof_dev *sdev, bool enable)
+{
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ struct hdac_bus *bus = sof_to_bus(sdev);
+#endif
+ u32 val;
+
+ /* enable/disable audio dsp clock gating */
+ val = enable ? PCI_CGCTL_ADSPDCGE : 0;
+ snd_sof_pci_update_bits(sdev, PCI_CGCTL, PCI_CGCTL_ADSPDCGE, val);
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ /* enable/disable L1 support */
+ val = enable ? SOF_HDA_VS_EM2_L1SEN : 0;
+ snd_hdac_chip_updatel(bus, VS_EM2, SOF_HDA_VS_EM2_L1SEN, val);
+#endif
+
+ /* enable/disable audio dsp power gating */
+ val = enable ? 0 : PCI_PGCTL_ADSPPGD;
+ snd_sof_pci_update_bits(sdev, PCI_PGCTL, PCI_PGCTL_ADSPPGD, val);
+
+ return 0;
+}
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+/*
+ * While performing reset, controller may not come back properly and causing
+ * issues, so recommendation is to set CGCTL.MISCBDCGE to 0 then do reset
+ * (init chip) and then again set CGCTL.MISCBDCGE to 1
+ */
+int hda_dsp_ctrl_init_chip(struct snd_sof_dev *sdev, bool full_reset)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ int ret;
+
+ hda_dsp_ctrl_misc_clock_gating(sdev, false);
+ ret = snd_hdac_bus_init_chip(bus, full_reset);
+ hda_dsp_ctrl_misc_clock_gating(sdev, true);
+
+ return ret;
+}
+#endif
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Keyon Jie <yang.jie@linux.intel.com>
+//
+
+#include <sound/pcm_params.h>
+#include <sound/hdaudio_ext.h>
+#include "../sof-priv.h"
+#include "hda.h"
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+
+struct hda_pipe_params {
+ u8 host_dma_id;
+ u8 link_dma_id;
+ u32 ch;
+ u32 s_freq;
+ u32 s_fmt;
+ u8 linktype;
+ snd_pcm_format_t format;
+ int link_index;
+ int stream;
+ unsigned int host_bps;
+ unsigned int link_bps;
+};
+
+/*
+ * Unlike GP dma, there is a set of stream registers in hda controller
+ * to control the link dma channels. Each register controls one link
+ * dma channel and the relation is fixed. To make sure FW uses correct
+ * link dma channels, host allocates stream registers and sends the
+ * corresponding link dma channels to FW to allocate link dma channel
+ *
+ * FIXME: this API is abused in the sense that tx_num and rx_num are
+ * passed as arguments, not returned. We need to find a better way to
+ * retrieve the stream tag allocated for the link DMA
+ */
+static int hda_link_dma_get_channels(struct snd_soc_dai *dai,
+ unsigned int *tx_num,
+ unsigned int *tx_slot,
+ unsigned int *rx_num,
+ unsigned int *rx_slot)
+{
+ struct hdac_bus *bus;
+ struct hdac_ext_stream *stream;
+ struct snd_pcm_substream substream;
+ struct snd_sof_dev *sdev =
+ snd_soc_component_get_drvdata(dai->component);
+
+ bus = sof_to_bus(sdev);
+
+ memset(&substream, 0, sizeof(substream));
+ if (*tx_num == 1) {
+ substream.stream = SNDRV_PCM_STREAM_PLAYBACK;
+ stream = snd_hdac_ext_stream_assign(bus, &substream,
+ HDAC_EXT_STREAM_TYPE_LINK);
+ if (!stream) {
+ dev_err(bus->dev, "error: failed to find a free hda ext stream for playback");
+ return -EBUSY;
+ }
+
+ snd_soc_dai_set_dma_data(dai, &substream, stream);
+ *tx_slot = hdac_stream(stream)->stream_tag - 1;
+
+ dev_dbg(bus->dev, "link dma channel %d for playback", *tx_slot);
+ }
+
+ if (*rx_num == 1) {
+ substream.stream = SNDRV_PCM_STREAM_CAPTURE;
+ stream = snd_hdac_ext_stream_assign(bus, &substream,
+ HDAC_EXT_STREAM_TYPE_LINK);
+ if (!stream) {
+ dev_err(bus->dev, "error: failed to find a free hda ext stream for capture");
+ return -EBUSY;
+ }
+
+ snd_soc_dai_set_dma_data(dai, &substream, stream);
+ *rx_slot = hdac_stream(stream)->stream_tag - 1;
+
+ dev_dbg(bus->dev, "link dma channel %d for capture", *rx_slot);
+ }
+
+ return 0;
+}
+
+static int hda_link_dma_params(struct hdac_ext_stream *stream,
+ struct hda_pipe_params *params)
+{
+ struct hdac_stream *hstream = &stream->hstream;
+ unsigned char stream_tag = hstream->stream_tag;
+ struct hdac_bus *bus = hstream->bus;
+ struct hdac_ext_link *link;
+ unsigned int format_val;
+
+ snd_hdac_ext_stream_decouple(bus, stream, true);
+ snd_hdac_ext_link_stream_reset(stream);
+
+ format_val = snd_hdac_calc_stream_format(params->s_freq, params->ch,
+ params->format,
+ params->link_bps, 0);
+
+ dev_dbg(bus->dev, "format_val=%d, rate=%d, ch=%d, format=%d\n",
+ format_val, params->s_freq, params->ch, params->format);
+
+ snd_hdac_ext_link_stream_setup(stream, format_val);
+
+ if (stream->hstream.direction == SNDRV_PCM_STREAM_PLAYBACK) {
+ list_for_each_entry(link, &bus->hlink_list, list) {
+ if (link->index == params->link_index)
+ snd_hdac_ext_link_set_stream_id(link,
+ stream_tag);
+ }
+ }
+
+ stream->link_prepared = 1;
+
+ return 0;
+}
+
+static int hda_link_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params,
+ struct snd_soc_dai *dai)
+{
+ struct hdac_stream *hstream = substream->runtime->private_data;
+ struct hdac_bus *bus = hstream->bus;
+ struct hdac_ext_stream *link_dev;
+ struct snd_soc_pcm_runtime *rtd = snd_pcm_substream_chip(substream);
+ struct snd_soc_dai *codec_dai = rtd->codec_dai;
+ struct sof_intel_hda_stream *hda_stream;
+ struct hda_pipe_params p_params = {0};
+ struct hdac_ext_link *link;
+ int stream_tag;
+
+ link_dev = snd_soc_dai_get_dma_data(dai, substream);
+
+ hda_stream = container_of(link_dev, struct sof_intel_hda_stream,
+ hda_stream);
+ hda_stream->hw_params_upon_resume = 0;
+
+ link = snd_hdac_ext_bus_get_link(bus, codec_dai->component->name);
+ if (!link)
+ return -EINVAL;
+
+ stream_tag = hdac_stream(link_dev)->stream_tag;
+
+ /* set the stream tag in the codec dai dma params */
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ snd_soc_dai_set_tdm_slot(codec_dai, stream_tag, 0, 0, 0);
+ else
+ snd_soc_dai_set_tdm_slot(codec_dai, 0, stream_tag, 0, 0);
+
+ p_params.s_fmt = snd_pcm_format_width(params_format(params));
+ p_params.ch = params_channels(params);
+ p_params.s_freq = params_rate(params);
+ p_params.stream = substream->stream;
+ p_params.link_dma_id = stream_tag - 1;
+ p_params.link_index = link->index;
+ p_params.format = params_format(params);
+
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ p_params.link_bps = codec_dai->driver->playback.sig_bits;
+ else
+ p_params.link_bps = codec_dai->driver->capture.sig_bits;
+
+ return hda_link_dma_params(link_dev, &p_params);
+}
+
+static int hda_link_pcm_prepare(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ struct hdac_ext_stream *link_dev =
+ snd_soc_dai_get_dma_data(dai, substream);
+ struct sof_intel_hda_stream *hda_stream;
+ struct snd_sof_dev *sdev =
+ snd_soc_component_get_drvdata(dai->component);
+ struct snd_soc_pcm_runtime *rtd = snd_pcm_substream_chip(substream);
+ int stream = substream->stream;
+
+ hda_stream = container_of(link_dev, struct sof_intel_hda_stream,
+ hda_stream);
+
+ /* setup hw_params again only if resuming from system suspend */
+ if (!hda_stream->hw_params_upon_resume)
+ return 0;
+
+ dev_dbg(sdev->dev, "hda: prepare stream dir %d\n", substream->stream);
+
+ return hda_link_hw_params(substream, &rtd->dpcm[stream].hw_params,
+ dai);
+}
+
+static int hda_link_pcm_trigger(struct snd_pcm_substream *substream,
+ int cmd, struct snd_soc_dai *dai)
+{
+ struct hdac_ext_stream *link_dev =
+ snd_soc_dai_get_dma_data(dai, substream);
+ int ret;
+
+ dev_dbg(dai->dev, "In %s cmd=%d\n", __func__, cmd);
+ switch (cmd) {
+ case SNDRV_PCM_TRIGGER_RESUME:
+ /* set up hw_params */
+ ret = hda_link_pcm_prepare(substream, dai);
+ if (ret < 0) {
+ dev_err(dai->dev,
+ "error: setting up hw_params during resume\n");
+ return ret;
+ }
+
+ /* fallthrough */
+ case SNDRV_PCM_TRIGGER_START:
+ case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
+ snd_hdac_ext_link_stream_start(link_dev);
+ break;
+ case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
+ case SNDRV_PCM_TRIGGER_SUSPEND:
+ case SNDRV_PCM_TRIGGER_STOP:
+ snd_hdac_ext_link_stream_clear(link_dev);
+ break;
+ default:
+ return -EINVAL;
+ }
+ return 0;
+}
+
+/*
+ * FIXME: This API is also abused since it's used for two purposes.
+ * when the substream argument is NULL this function is used for cleanups
+ * that aren't necessarily required, and called explicitly by handling
+ * ASoC core structures, which is not recommended.
+ * This part will be reworked in follow-up patches.
+ */
+static int hda_link_hw_free(struct snd_pcm_substream *substream,
+ struct snd_soc_dai *dai)
+{
+ const char *name;
+ unsigned int stream_tag;
+ struct hdac_bus *bus;
+ struct hdac_ext_link *link;
+ struct hdac_stream *hstream;
+ struct hdac_ext_stream *stream;
+ struct snd_soc_pcm_runtime *rtd;
+ struct hdac_ext_stream *link_dev;
+ struct snd_pcm_substream pcm_substream;
+
+ memset(&pcm_substream, 0, sizeof(pcm_substream));
+ if (substream) {
+ hstream = substream->runtime->private_data;
+ bus = hstream->bus;
+ rtd = snd_pcm_substream_chip(substream);
+ link_dev = snd_soc_dai_get_dma_data(dai, substream);
+ snd_hdac_ext_stream_decouple(bus, link_dev, false);
+ name = rtd->codec_dai->component->name;
+ link = snd_hdac_ext_bus_get_link(bus, name);
+ if (!link)
+ return -EINVAL;
+
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
+ stream_tag = hdac_stream(link_dev)->stream_tag;
+ snd_hdac_ext_link_clear_stream_id(link, stream_tag);
+ }
+
+ link_dev->link_prepared = 0;
+ } else {
+ /* release all hda streams when dai link is unloaded */
+ pcm_substream.stream = SNDRV_PCM_STREAM_PLAYBACK;
+ stream = snd_soc_dai_get_dma_data(dai, &pcm_substream);
+ if (stream) {
+ snd_soc_dai_set_dma_data(dai, &pcm_substream, NULL);
+ snd_hdac_ext_stream_release(stream,
+ HDAC_EXT_STREAM_TYPE_LINK);
+ }
+
+ pcm_substream.stream = SNDRV_PCM_STREAM_CAPTURE;
+ stream = snd_soc_dai_get_dma_data(dai, &pcm_substream);
+ if (stream) {
+ snd_soc_dai_set_dma_data(dai, &pcm_substream, NULL);
+ snd_hdac_ext_stream_release(stream,
+ HDAC_EXT_STREAM_TYPE_LINK);
+ }
+ }
+
+ return 0;
+}
+
+static const struct snd_soc_dai_ops hda_link_dai_ops = {
+ .hw_params = hda_link_hw_params,
+ .hw_free = hda_link_hw_free,
+ .trigger = hda_link_pcm_trigger,
+ .prepare = hda_link_pcm_prepare,
+ .get_channel_map = hda_link_dma_get_channels,
+};
+#endif
+
+/*
+ * common dai driver for skl+ platforms.
+ * some products who use this DAI array only physically have a subset of
+ * the DAIs, but no harm is done here by adding the whole set.
+ */
+struct snd_soc_dai_driver skl_dai[] = {
+{
+ .name = "SSP0 Pin",
+},
+{
+ .name = "SSP1 Pin",
+},
+{
+ .name = "SSP2 Pin",
+},
+{
+ .name = "SSP3 Pin",
+},
+{
+ .name = "SSP4 Pin",
+},
+{
+ .name = "SSP5 Pin",
+},
+{
+ .name = "DMIC01 Pin",
+},
+{
+ .name = "DMIC16k Pin",
+},
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+{
+ .name = "iDisp1 Pin",
+ .ops = &hda_link_dai_ops,
+},
+{
+ .name = "iDisp2 Pin",
+ .ops = &hda_link_dai_ops,
+},
+{
+ .name = "iDisp3 Pin",
+ .ops = &hda_link_dai_ops,
+},
+{
+ .name = "Analog CPU DAI",
+ .ops = &hda_link_dai_ops,
+},
+{
+ .name = "Digital CPU DAI",
+ .ops = &hda_link_dai_ops,
+},
+{
+ .name = "Alt Analog CPU DAI",
+ .ops = &hda_link_dai_ops,
+},
+#endif
+};
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+// Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
+// Rander Wang <rander.wang@intel.com>
+// Keyon Jie <yang.jie@linux.intel.com>
+//
+
+/*
+ * Hardware interface for generic Intel audio DSP HDA IP
+ */
+
+#include <sound/hdaudio_ext.h>
+#include <sound/hda_register.h>
+#include "../ops.h"
+#include "hda.h"
+
+/*
+ * DSP Core control.
+ */
+
+int hda_dsp_core_reset_enter(struct snd_sof_dev *sdev, unsigned int core_mask)
+{
+ u32 adspcs;
+ u32 reset;
+ int ret;
+
+ /* set reset bits for cores */
+ reset = HDA_DSP_ADSPCS_CRST_MASK(core_mask);
+ snd_sof_dsp_update_bits_unlocked(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS,
+ reset, reset),
+
+ /* poll with timeout to check if operation successful */
+ ret = snd_sof_dsp_read_poll_timeout(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS, adspcs,
+ ((adspcs & reset) == reset),
+ HDA_DSP_REG_POLL_INTERVAL_US,
+ HDA_DSP_RESET_TIMEOUT_US);
+
+ /* has core entered reset ? */
+ adspcs = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS);
+ if ((adspcs & HDA_DSP_ADSPCS_CRST_MASK(core_mask)) !=
+ HDA_DSP_ADSPCS_CRST_MASK(core_mask)) {
+ dev_err(sdev->dev,
+ "error: reset enter failed: core_mask %x adspcs 0x%x\n",
+ core_mask, adspcs);
+ ret = -EIO;
+ }
+
+ return ret;
+}
+
+int hda_dsp_core_reset_leave(struct snd_sof_dev *sdev, unsigned int core_mask)
+{
+ unsigned int crst;
+ u32 adspcs;
+ int ret;
+
+ /* clear reset bits for cores */
+ snd_sof_dsp_update_bits_unlocked(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS,
+ HDA_DSP_ADSPCS_CRST_MASK(core_mask),
+ 0);
+
+ /* poll with timeout to check if operation successful */
+ crst = HDA_DSP_ADSPCS_CRST_MASK(core_mask);
+ ret = snd_sof_dsp_read_poll_timeout(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS, adspcs,
+ !(adspcs & crst),
+ HDA_DSP_REG_POLL_INTERVAL_US,
+ HDA_DSP_RESET_TIMEOUT_US);
+
+ /* has core left reset ? */
+ adspcs = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS);
+ if ((adspcs & HDA_DSP_ADSPCS_CRST_MASK(core_mask)) != 0) {
+ dev_err(sdev->dev,
+ "error: reset leave failed: core_mask %x adspcs 0x%x\n",
+ core_mask, adspcs);
+ ret = -EIO;
+ }
+
+ return ret;
+}
+
+int hda_dsp_core_stall_reset(struct snd_sof_dev *sdev, unsigned int core_mask)
+{
+ /* stall core */
+ snd_sof_dsp_update_bits_unlocked(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS,
+ HDA_DSP_ADSPCS_CSTALL_MASK(core_mask),
+ HDA_DSP_ADSPCS_CSTALL_MASK(core_mask));
+
+ /* set reset state */
+ return hda_dsp_core_reset_enter(sdev, core_mask);
+}
+
+int hda_dsp_core_run(struct snd_sof_dev *sdev, unsigned int core_mask)
+{
+ int ret;
+
+ /* leave reset state */
+ ret = hda_dsp_core_reset_leave(sdev, core_mask);
+ if (ret < 0)
+ return ret;
+
+ /* run core */
+ dev_dbg(sdev->dev, "unstall/run core: core_mask = %x\n", core_mask);
+ snd_sof_dsp_update_bits_unlocked(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS,
+ HDA_DSP_ADSPCS_CSTALL_MASK(core_mask),
+ 0);
+
+ /* is core now running ? */
+ if (!hda_dsp_core_is_enabled(sdev, core_mask)) {
+ hda_dsp_core_stall_reset(sdev, core_mask);
+ dev_err(sdev->dev, "error: DSP start core failed: core_mask %x\n",
+ core_mask);
+ ret = -EIO;
+ }
+
+ return ret;
+}
+
+/*
+ * Power Management.
+ */
+
+int hda_dsp_core_power_up(struct snd_sof_dev *sdev, unsigned int core_mask)
+{
+ unsigned int cpa;
+ u32 adspcs;
+ int ret;
+
+ /* update bits */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR, HDA_DSP_REG_ADSPCS,
+ HDA_DSP_ADSPCS_SPA_MASK(core_mask),
+ HDA_DSP_ADSPCS_SPA_MASK(core_mask));
+
+ /* poll with timeout to check if operation successful */
+ cpa = HDA_DSP_ADSPCS_CPA_MASK(core_mask);
+ ret = snd_sof_dsp_read_poll_timeout(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS, adspcs,
+ (adspcs & cpa) == cpa,
+ HDA_DSP_REG_POLL_INTERVAL_US,
+ HDA_DSP_RESET_TIMEOUT_US);
+ if (ret < 0)
+ dev_err(sdev->dev, "error: timeout on core powerup\n");
+
+ /* did core power up ? */
+ adspcs = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS);
+ if ((adspcs & HDA_DSP_ADSPCS_CPA_MASK(core_mask)) !=
+ HDA_DSP_ADSPCS_CPA_MASK(core_mask)) {
+ dev_err(sdev->dev,
+ "error: power up core failed core_mask %xadspcs 0x%x\n",
+ core_mask, adspcs);
+ ret = -EIO;
+ }
+
+ return ret;
+}
+
+int hda_dsp_core_power_down(struct snd_sof_dev *sdev, unsigned int core_mask)
+{
+ u32 adspcs;
+
+ /* update bits */
+ snd_sof_dsp_update_bits_unlocked(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS,
+ HDA_DSP_ADSPCS_SPA_MASK(core_mask), 0);
+
+ return snd_sof_dsp_read_poll_timeout(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPCS, adspcs,
+ !(adspcs & HDA_DSP_ADSPCS_SPA_MASK(core_mask)),
+ HDA_DSP_REG_POLL_INTERVAL_US,
+ HDA_DSP_PD_TIMEOUT * USEC_PER_MSEC);
+}
+
+bool hda_dsp_core_is_enabled(struct snd_sof_dev *sdev,
+ unsigned int core_mask)
+{
+ int val;
+ bool is_enable;
+
+ val = snd_sof_dsp_read(sdev, HDA_DSP_BAR, HDA_DSP_REG_ADSPCS);
+
+ is_enable = ((val & HDA_DSP_ADSPCS_CPA_MASK(core_mask)) &&
+ (val & HDA_DSP_ADSPCS_SPA_MASK(core_mask)) &&
+ !(val & HDA_DSP_ADSPCS_CRST_MASK(core_mask)) &&
+ !(val & HDA_DSP_ADSPCS_CSTALL_MASK(core_mask)));
+
+ dev_dbg(sdev->dev, "DSP core(s) enabled? %d : core_mask %x\n",
+ is_enable, core_mask);
+
+ return is_enable;
+}
+
+int hda_dsp_enable_core(struct snd_sof_dev *sdev, unsigned int core_mask)
+{
+ int ret;
+
+ /* return if core is already enabled */
+ if (hda_dsp_core_is_enabled(sdev, core_mask))
+ return 0;
+
+ /* power up */
+ ret = hda_dsp_core_power_up(sdev, core_mask);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: dsp core power up failed: core_mask %x\n",
+ core_mask);
+ return ret;
+ }
+
+ return hda_dsp_core_run(sdev, core_mask);
+}
+
+int hda_dsp_core_reset_power_down(struct snd_sof_dev *sdev,
+ unsigned int core_mask)
+{
+ int ret;
+
+ /* place core in reset prior to power down */
+ ret = hda_dsp_core_stall_reset(sdev, core_mask);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: dsp core reset failed: core_mask %x\n",
+ core_mask);
+ return ret;
+ }
+
+ /* power down core */
+ ret = hda_dsp_core_power_down(sdev, core_mask);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: dsp core power down fail mask %x: %d\n",
+ core_mask, ret);
+ return ret;
+ }
+
+ /* make sure we are in OFF state */
+ if (hda_dsp_core_is_enabled(sdev, core_mask)) {
+ dev_err(sdev->dev, "error: dsp core disable fail mask %x: %d\n",
+ core_mask, ret);
+ ret = -EIO;
+ }
+
+ return ret;
+}
+
+void hda_dsp_ipc_int_enable(struct snd_sof_dev *sdev)
+{
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ const struct sof_intel_dsp_desc *chip = hda->desc;
+
+ /* enable IPC DONE and BUSY interrupts */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR, chip->ipc_ctl,
+ HDA_DSP_REG_HIPCCTL_DONE | HDA_DSP_REG_HIPCCTL_BUSY,
+ HDA_DSP_REG_HIPCCTL_DONE | HDA_DSP_REG_HIPCCTL_BUSY);
+
+ /* enable IPC interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR, HDA_DSP_REG_ADSPIC,
+ HDA_DSP_ADSPIC_IPC, HDA_DSP_ADSPIC_IPC);
+}
+
+void hda_dsp_ipc_int_disable(struct snd_sof_dev *sdev)
+{
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ const struct sof_intel_dsp_desc *chip = hda->desc;
+
+ /* disable IPC interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR, HDA_DSP_REG_ADSPIC,
+ HDA_DSP_ADSPIC_IPC, 0);
+
+ /* disable IPC BUSY and DONE interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR, chip->ipc_ctl,
+ HDA_DSP_REG_HIPCCTL_BUSY | HDA_DSP_REG_HIPCCTL_DONE, 0);
+}
+
+static int hda_suspend(struct snd_sof_dev *sdev, int state)
+{
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ const struct sof_intel_dsp_desc *chip = hda->desc;
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ struct hdac_bus *bus = sof_to_bus(sdev);
+#endif
+ int ret;
+
+ /* disable IPC interrupts */
+ hda_dsp_ipc_int_disable(sdev);
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ /* power down all hda link */
+ snd_hdac_ext_bus_link_power_down_all(bus);
+#endif
+
+ /* power down DSP */
+ ret = hda_dsp_core_reset_power_down(sdev, chip->cores_mask);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to power down core during suspend\n");
+ return ret;
+ }
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ /* disable ppcap interrupt */
+ snd_hdac_ext_bus_ppcap_int_enable(bus, false);
+ snd_hdac_ext_bus_ppcap_enable(bus, false);
+
+ /* disable hda bus irq and i/o */
+ snd_hdac_bus_stop_chip(bus);
+#else
+ /* disable ppcap interrupt */
+ hda_dsp_ctrl_ppcap_enable(sdev, false);
+ hda_dsp_ctrl_ppcap_int_enable(sdev, false);
+
+ /* disable hda bus irq */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, SOF_HDA_INTCTL,
+ SOF_HDA_INT_CTRL_EN | SOF_HDA_INT_GLOBAL_EN,
+ 0);
+#endif
+
+ /* disable LP retention mode */
+ snd_sof_pci_update_bits(sdev, PCI_PGCTL,
+ PCI_PGCTL_LSRMD_MASK, PCI_PGCTL_LSRMD_MASK);
+
+ /* reset controller */
+ ret = hda_dsp_ctrl_link_reset(sdev, true);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to reset controller during suspend\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int hda_resume(struct snd_sof_dev *sdev)
+{
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct hdac_ext_link *hlink = NULL;
+#endif
+ int ret;
+
+ /*
+ * clear TCSEL to clear playback on some HD Audio
+ * codecs. PCI TCSEL is defined in the Intel manuals.
+ */
+ snd_sof_pci_update_bits(sdev, PCI_TCSEL, 0x07, 0);
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ /* reset and start hda controller */
+ ret = hda_dsp_ctrl_init_chip(sdev, true);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to start controller after resume\n");
+ return ret;
+ }
+
+ hda_dsp_ctrl_misc_clock_gating(sdev, false);
+
+ /* Reset stream-to-link mapping */
+ list_for_each_entry(hlink, &bus->hlink_list, list)
+ bus->io_ops->reg_writel(0, hlink->ml_addr + AZX_REG_ML_LOSIDV);
+
+ hda_dsp_ctrl_misc_clock_gating(sdev, true);
+
+ /* enable ppcap interrupt */
+ snd_hdac_ext_bus_ppcap_enable(bus, true);
+ snd_hdac_ext_bus_ppcap_int_enable(bus, true);
+#else
+
+ hda_dsp_ctrl_misc_clock_gating(sdev, false);
+
+ /* reset controller */
+ ret = hda_dsp_ctrl_link_reset(sdev, true);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to reset controller during resume\n");
+ return ret;
+ }
+
+ /* take controller out of reset */
+ ret = hda_dsp_ctrl_link_reset(sdev, false);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to ready controller during resume\n");
+ return ret;
+ }
+
+ /* enable hda bus irq */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, SOF_HDA_INTCTL,
+ SOF_HDA_INT_CTRL_EN | SOF_HDA_INT_GLOBAL_EN,
+ SOF_HDA_INT_CTRL_EN | SOF_HDA_INT_GLOBAL_EN);
+
+ hda_dsp_ctrl_misc_clock_gating(sdev, true);
+
+ /* enable ppcap interrupt */
+ hda_dsp_ctrl_ppcap_enable(sdev, true);
+ hda_dsp_ctrl_ppcap_int_enable(sdev, true);
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ /* turn off the links that were off before suspend */
+ list_for_each_entry(hlink, &bus->hlink_list, list) {
+ if (!hlink->ref_count)
+ snd_hdac_ext_bus_link_power_down(hlink);
+ }
+
+ /* check dma status and clean up CORB/RIRB buffers */
+ if (!bus->cmd_dma_state)
+ snd_hdac_bus_stop_cmd_io(bus);
+#endif
+
+ return 0;
+}
+
+int hda_dsp_resume(struct snd_sof_dev *sdev)
+{
+ /* init hda controller. DSP cores will be powered up during fw boot */
+ return hda_resume(sdev);
+}
+
+int hda_dsp_runtime_resume(struct snd_sof_dev *sdev)
+{
+ /* init hda controller. DSP cores will be powered up during fw boot */
+ return hda_resume(sdev);
+}
+
+int hda_dsp_runtime_suspend(struct snd_sof_dev *sdev, int state)
+{
+ /* stop hda controller and power dsp off */
+ return hda_suspend(sdev, state);
+}
+
+int hda_dsp_suspend(struct snd_sof_dev *sdev, int state)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ int ret;
+
+ /* stop hda controller and power dsp off */
+ ret = hda_suspend(sdev, state);
+ if (ret < 0) {
+ dev_err(bus->dev, "error: suspending dsp\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+void hda_dsp_set_hw_params_upon_resume(struct snd_sof_dev *sdev)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct sof_intel_hda_stream *hda_stream;
+ struct hdac_ext_stream *stream;
+ struct hdac_stream *s;
+
+ /* set internal flag for BE */
+ list_for_each_entry(s, &bus->stream_list, list) {
+ stream = stream_to_hdac_ext_stream(s);
+ hda_stream = container_of(stream, struct sof_intel_hda_stream,
+ hda_stream);
+ hda_stream->hw_params_upon_resume = 1;
+ }
+}
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+// Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
+// Rander Wang <rander.wang@intel.com>
+// Keyon Jie <yang.jie@linux.intel.com>
+//
+
+/*
+ * Hardware interface for generic Intel audio DSP HDA IP
+ */
+
+#include "../ops.h"
+#include "hda.h"
+
+static void hda_dsp_ipc_host_done(struct snd_sof_dev *sdev)
+{
+ /*
+ * tell DSP cmd is done - clear busy
+ * interrupt and send reply msg to dsp
+ */
+ snd_sof_dsp_update_bits_forced(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_HIPCT,
+ HDA_DSP_REG_HIPCT_BUSY,
+ HDA_DSP_REG_HIPCT_BUSY);
+
+ /* unmask BUSY interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_HIPCCTL,
+ HDA_DSP_REG_HIPCCTL_BUSY,
+ HDA_DSP_REG_HIPCCTL_BUSY);
+}
+
+static void hda_dsp_ipc_dsp_done(struct snd_sof_dev *sdev)
+{
+ /*
+ * set DONE bit - tell DSP we have received the reply msg
+ * from DSP, and processed it, don't send more reply to host
+ */
+ snd_sof_dsp_update_bits_forced(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_HIPCIE,
+ HDA_DSP_REG_HIPCIE_DONE,
+ HDA_DSP_REG_HIPCIE_DONE);
+
+ /* unmask Done interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_HIPCCTL,
+ HDA_DSP_REG_HIPCCTL_DONE,
+ HDA_DSP_REG_HIPCCTL_DONE);
+}
+
+int hda_dsp_ipc_send_msg(struct snd_sof_dev *sdev, struct snd_sof_ipc_msg *msg)
+{
+ u32 cmd = msg->header;
+
+ /* send IPC message to DSP */
+ sof_mailbox_write(sdev, sdev->host_box.offset, msg->msg_data,
+ msg->msg_size);
+ snd_sof_dsp_write(sdev, HDA_DSP_BAR, HDA_DSP_REG_HIPCI,
+ cmd | HDA_DSP_REG_HIPCI_BUSY);
+
+ return 0;
+}
+
+void hda_dsp_ipc_get_reply(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_ipc_msg *msg = sdev->msg;
+ struct sof_ipc_reply reply;
+ struct sof_ipc_cmd_hdr *hdr;
+ unsigned long flags;
+ int ret = 0;
+
+ /*
+ * Sometimes, there is unexpected reply ipc arriving. The reply
+ * ipc belongs to none of the ipcs sent from driver.
+ * In this case, the driver must ignore the ipc.
+ */
+ if (!msg) {
+ dev_warn(sdev->dev, "unexpected ipc interrupt raised!\n");
+ return;
+ }
+ spin_lock_irqsave(&sdev->ipc_lock, flags);
+
+ hdr = msg->msg_data;
+ if (hdr->cmd == (SOF_IPC_GLB_PM_MSG | SOF_IPC_PM_CTX_SAVE)) {
+ /*
+ * memory windows are powered off before sending IPC reply,
+ * so we can't read the mailbox for CTX_SAVE reply.
+ */
+ reply.error = 0;
+ reply.hdr.cmd = SOF_IPC_GLB_REPLY;
+ reply.hdr.size = sizeof(reply);
+ memcpy(msg->reply_data, &reply, sizeof(reply));
+ goto out;
+ }
+
+ /* get IPC reply from DSP in the mailbox */
+ sof_mailbox_read(sdev, sdev->host_box.offset, &reply,
+ sizeof(reply));
+
+ if (reply.error < 0) {
+ memcpy(msg->reply_data, &reply, sizeof(reply));
+ ret = reply.error;
+ } else {
+ /* reply correct size ? */
+ if (reply.hdr.size != msg->reply_size) {
+ dev_err(sdev->dev, "error: reply expected %zu got %u bytes\n",
+ msg->reply_size, reply.hdr.size);
+ ret = -EINVAL;
+ }
+
+ /* read the message */
+ if (msg->reply_size > 0)
+ sof_mailbox_read(sdev, sdev->host_box.offset,
+ msg->reply_data, msg->reply_size);
+ }
+
+out:
+ msg->reply_error = ret;
+
+ spin_unlock_irqrestore(&sdev->ipc_lock, flags);
+}
+
+static bool hda_dsp_ipc_is_sof(uint32_t msg)
+{
+ return (msg & (HDA_DSP_IPC_PURGE_FW | 0xf << 9)) != msg ||
+ (msg & HDA_DSP_IPC_PURGE_FW) != HDA_DSP_IPC_PURGE_FW;
+}
+
+/* IPC handler thread */
+irqreturn_t hda_dsp_ipc_irq_thread(int irq, void *context)
+{
+ struct snd_sof_dev *sdev = context;
+ irqreturn_t ret = IRQ_NONE;
+ u32 hipci;
+ u32 hipcie;
+ u32 hipct;
+ u32 hipcte;
+ u32 hipcctl;
+ u32 msg;
+ u32 msg_ext;
+
+ /* read IPC status */
+ hipcie = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_HIPCIE);
+ hipct = snd_sof_dsp_read(sdev, HDA_DSP_BAR, HDA_DSP_REG_HIPCT);
+ hipcctl = snd_sof_dsp_read(sdev, HDA_DSP_BAR, HDA_DSP_REG_HIPCCTL);
+
+ /* reenable IPC interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR, HDA_DSP_REG_ADSPIC,
+ HDA_DSP_ADSPIC_IPC, HDA_DSP_ADSPIC_IPC);
+
+ /* is this a reply message from the DSP */
+ if (hipcie & HDA_DSP_REG_HIPCIE_DONE &&
+ hipcctl & HDA_DSP_REG_HIPCCTL_DONE) {
+ hipci = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_HIPCI);
+ msg = hipci & HDA_DSP_REG_HIPCI_MSG_MASK;
+ msg_ext = hipcie & HDA_DSP_REG_HIPCIE_MSG_MASK;
+
+ dev_vdbg(sdev->dev,
+ "ipc: firmware response, msg:0x%x, msg_ext:0x%x\n",
+ msg, msg_ext);
+
+ /* mask Done interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_HIPCCTL,
+ HDA_DSP_REG_HIPCCTL_DONE, 0);
+
+ /* handle immediate reply from DSP core - ignore ROM messages */
+ if (hda_dsp_ipc_is_sof(msg)) {
+ hda_dsp_ipc_get_reply(sdev);
+ snd_sof_ipc_reply(sdev, msg);
+ }
+
+ /* wake up sleeper if we are loading code */
+ if (sdev->code_loading) {
+ sdev->code_loading = 0;
+ wake_up(&sdev->waitq);
+ }
+
+ /* set the done bit */
+ hda_dsp_ipc_dsp_done(sdev);
+
+ ret = IRQ_HANDLED;
+ }
+
+ /* is this a new message from DSP */
+ if (hipct & HDA_DSP_REG_HIPCT_BUSY &&
+ hipcctl & HDA_DSP_REG_HIPCCTL_BUSY) {
+
+ hipcte = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_HIPCTE);
+ msg = hipct & HDA_DSP_REG_HIPCT_MSG_MASK;
+ msg_ext = hipcte & HDA_DSP_REG_HIPCTE_MSG_MASK;
+
+ dev_vdbg(sdev->dev,
+ "ipc: firmware initiated, msg:0x%x, msg_ext:0x%x\n",
+ msg, msg_ext);
+
+ /* mask BUSY interrupt */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_HIPCCTL,
+ HDA_DSP_REG_HIPCCTL_BUSY, 0);
+
+ /* handle messages from DSP */
+ if ((hipct & SOF_IPC_PANIC_MAGIC_MASK) == SOF_IPC_PANIC_MAGIC) {
+ /* this is a PANIC message !! */
+ snd_sof_dsp_panic(sdev, HDA_DSP_PANIC_OFFSET(msg_ext));
+ } else {
+ /* normal message - process normally */
+ snd_sof_ipc_msgs_rx(sdev);
+ }
+
+ hda_dsp_ipc_host_done(sdev);
+
+ ret = IRQ_HANDLED;
+ }
+
+ return ret;
+}
+
+/* is this IRQ for ADSP ? - we only care about IPC here */
+irqreturn_t hda_dsp_ipc_irq_handler(int irq, void *context)
+{
+ struct snd_sof_dev *sdev = context;
+ int ret = IRQ_NONE;
+ u32 irq_status;
+
+ spin_lock(&sdev->hw_lock);
+
+ /* store status */
+ irq_status = snd_sof_dsp_read(sdev, HDA_DSP_BAR, HDA_DSP_REG_ADSPIS);
+ dev_vdbg(sdev->dev, "irq handler: irq_status:0x%x\n", irq_status);
+
+ /* invalid message ? */
+ if (irq_status == 0xffffffff)
+ goto out;
+
+ /* IPC message ? */
+ if (irq_status & HDA_DSP_ADSPIS_IPC) {
+ /* disable IPC interrupt */
+ snd_sof_dsp_update_bits_unlocked(sdev, HDA_DSP_BAR,
+ HDA_DSP_REG_ADSPIC,
+ HDA_DSP_ADSPIC_IPC, 0);
+ ret = IRQ_WAKE_THREAD;
+ }
+
+out:
+ spin_unlock(&sdev->hw_lock);
+ return ret;
+}
+
+/* IPC Firmware ready */
+
+static void ipc_get_windows(struct snd_sof_dev *sdev)
+{
+ struct sof_ipc_window_elem *elem;
+ u32 outbox_offset = 0;
+ u32 stream_offset = 0;
+ u32 inbox_offset = 0;
+ u32 outbox_size = 0;
+ u32 stream_size = 0;
+ u32 inbox_size = 0;
+ int i;
+
+ if (!sdev->info_window) {
+ dev_err(sdev->dev, "error: have no window info\n");
+ return;
+ }
+
+ for (i = 0; i < sdev->info_window->num_windows; i++) {
+ elem = &sdev->info_window->window[i];
+
+ switch (elem->type) {
+ case SOF_IPC_REGION_UPBOX:
+ inbox_offset =
+ elem->offset + SRAM_WINDOW_OFFSET(elem->id);
+ inbox_size = elem->size;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[HDA_DSP_BAR] +
+ inbox_offset,
+ elem->size, "inbox",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_DOWNBOX:
+ outbox_offset =
+ elem->offset + SRAM_WINDOW_OFFSET(elem->id);
+ outbox_size = elem->size;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[HDA_DSP_BAR] +
+ outbox_offset,
+ elem->size, "outbox",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_TRACE:
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[HDA_DSP_BAR] +
+ elem->offset +
+ SRAM_WINDOW_OFFSET
+ (elem->id),
+ elem->size, "etrace",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_DEBUG:
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[HDA_DSP_BAR] +
+ elem->offset +
+ SRAM_WINDOW_OFFSET
+ (elem->id),
+ elem->size, "debug",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_STREAM:
+ stream_offset =
+ elem->offset + SRAM_WINDOW_OFFSET(elem->id);
+ stream_size = elem->size;
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[HDA_DSP_BAR] +
+ elem->offset +
+ SRAM_WINDOW_OFFSET
+ (elem->id),
+ elem->size, "stream",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_REGS:
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[HDA_DSP_BAR] +
+ elem->offset +
+ SRAM_WINDOW_OFFSET
+ (elem->id),
+ elem->size, "regs",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ case SOF_IPC_REGION_EXCEPTION:
+ sdev->dsp_oops_offset = elem->offset +
+ SRAM_WINDOW_OFFSET(elem->id);
+ snd_sof_debugfs_io_item(sdev,
+ sdev->bar[HDA_DSP_BAR] +
+ elem->offset +
+ SRAM_WINDOW_OFFSET
+ (elem->id),
+ elem->size, "exception",
+ SOF_DEBUGFS_ACCESS_D0_ONLY);
+ break;
+ default:
+ dev_err(sdev->dev, "error: get illegal window info\n");
+ return;
+ }
+ }
+
+ if (outbox_size == 0 || inbox_size == 0) {
+ dev_err(sdev->dev, "error: get illegal mailbox window\n");
+ return;
+ }
+
+ snd_sof_dsp_mailbox_init(sdev, inbox_offset, inbox_size,
+ outbox_offset, outbox_size);
+ sdev->stream_box.offset = stream_offset;
+ sdev->stream_box.size = stream_size;
+
+ dev_dbg(sdev->dev, " mailbox upstream 0x%x - size 0x%x\n",
+ inbox_offset, inbox_size);
+ dev_dbg(sdev->dev, " mailbox downstream 0x%x - size 0x%x\n",
+ outbox_offset, outbox_size);
+ dev_dbg(sdev->dev, " stream region 0x%x - size 0x%x\n",
+ stream_offset, stream_size);
+}
+
+/* check for ABI compatibility and create memory windows on first boot */
+int hda_dsp_ipc_fw_ready(struct snd_sof_dev *sdev, u32 msg_id)
+{
+ struct sof_ipc_fw_ready *fw_ready = &sdev->fw_ready;
+ u32 offset;
+ int ret;
+
+ /* mailbox must be on 4k boundary */
+ offset = HDA_DSP_MBOX_UPLINK_OFFSET;
+
+ dev_dbg(sdev->dev, "ipc: DSP is ready 0x%8.8x offset 0x%x\n",
+ msg_id, offset);
+
+ /* no need to re-check version/ABI for subsequent boots */
+ if (!sdev->first_boot)
+ return 0;
+
+ /* copy data from the DSP FW ready offset */
+ sof_block_read(sdev, sdev->mmio_bar, offset, fw_ready,
+ sizeof(*fw_ready));
+
+ /* make sure ABI version is compatible */
+ ret = snd_sof_ipc_valid(sdev);
+ if (ret < 0)
+ return ret;
+
+ /* now check for extended data */
+ snd_sof_fw_parse_ext_data(sdev, sdev->mmio_bar,
+ HDA_DSP_MBOX_UPLINK_OFFSET +
+ sizeof(struct sof_ipc_fw_ready));
+
+ ipc_get_windows(sdev);
+
+ return 0;
+}
+
+void hda_ipc_msg_data(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ void *p, size_t sz)
+{
+ if (!substream || !sdev->stream_box.size) {
+ sof_mailbox_read(sdev, sdev->dsp_box.offset, p, sz);
+ } else {
+ struct hdac_stream *hstream = substream->runtime->private_data;
+ struct sof_intel_hda_stream *hda_stream;
+
+ hda_stream = container_of(hstream,
+ struct sof_intel_hda_stream,
+ hda_stream.hstream);
+
+ /* The stream might already be closed */
+ if (hstream)
+ sof_mailbox_read(sdev, hda_stream->stream.posn_offset,
+ p, sz);
+ }
+}
+
+int hda_ipc_pcm_params(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ const struct sof_ipc_pcm_params_reply *reply)
+{
+ struct hdac_stream *hstream = substream->runtime->private_data;
+ struct sof_intel_hda_stream *hda_stream;
+ /* validate offset */
+ size_t posn_offset = reply->posn_offset;
+
+ hda_stream = container_of(hstream, struct sof_intel_hda_stream,
+ hda_stream.hstream);
+
+ /* check for unaligned offset or overflow */
+ if (posn_offset > sdev->stream_box.size ||
+ posn_offset % sizeof(struct sof_ipc_stream_posn) != 0)
+ return -EINVAL;
+
+ hda_stream->stream.posn_offset = sdev->stream_box.offset + posn_offset;
+
+ dev_dbg(sdev->dev, "pcm: stream dir %d, posn mailbox offset is %zu",
+ substream->stream, hda_stream->stream.posn_offset);
+
+ return 0;
+}
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+// Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
+// Rander Wang <rander.wang@intel.com>
+// Keyon Jie <yang.jie@linux.intel.com>
+//
+
+/*
+ * Hardware interface for HDA DSP code loader
+ */
+
+#include <linux/firmware.h>
+#include <sound/hdaudio_ext.h>
+#include <sound/sof.h>
+#include "../ops.h"
+#include "hda.h"
+
+#define HDA_FW_BOOT_ATTEMPTS 3
+
+static int cl_stream_prepare(struct snd_sof_dev *sdev, unsigned int format,
+ unsigned int size, struct snd_dma_buffer *dmab,
+ int direction)
+{
+ struct hdac_ext_stream *dsp_stream;
+ struct hdac_stream *hstream;
+ struct pci_dev *pci = to_pci_dev(sdev->dev);
+ int ret;
+
+ if (direction != SNDRV_PCM_STREAM_PLAYBACK) {
+ dev_err(sdev->dev, "error: code loading DMA is playback only\n");
+ return -EINVAL;
+ }
+
+ dsp_stream = hda_dsp_stream_get(sdev, direction);
+
+ if (!dsp_stream) {
+ dev_err(sdev->dev, "error: no stream available\n");
+ return -ENODEV;
+ }
+ hstream = &dsp_stream->hstream;
+
+ /* allocate DMA buffer */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV_SG, &pci->dev, size, dmab);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: memory alloc failed: %x\n", ret);
+ goto error;
+ }
+
+ hstream->period_bytes = 0;/* initialize period_bytes */
+ hstream->format_val = format;
+ hstream->bufsize = size;
+
+ ret = hda_dsp_stream_hw_params(sdev, dsp_stream, dmab, NULL);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: hdac prepare failed: %x\n", ret);
+ goto error;
+ }
+
+ hda_dsp_stream_spib_config(sdev, dsp_stream, HDA_DSP_SPIB_ENABLE, size);
+
+ return hstream->stream_tag;
+
+error:
+ hda_dsp_stream_put(sdev, direction, hstream->stream_tag);
+ snd_dma_free_pages(dmab);
+ return ret;
+}
+
+/*
+ * first boot sequence has some extra steps. core 0 waits for power
+ * status on core 1, so power up core 1 also momentarily, keep it in
+ * reset/stall and then turn it off
+ */
+static int cl_dsp_init(struct snd_sof_dev *sdev, const void *fwdata,
+ u32 fwsize, int stream_tag)
+{
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ const struct sof_intel_dsp_desc *chip = hda->desc;
+ unsigned int status;
+ int ret;
+ int i;
+
+ /* step 1: power up corex */
+ ret = hda_dsp_core_power_up(sdev, chip->cores_mask);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: dsp core 0/1 power up failed\n");
+ goto err;
+ }
+
+ /* DSP is powered up, set all SSPs to slave mode */
+ for (i = 0; i < chip->ssp_count; i++) {
+ snd_sof_dsp_update_bits_unlocked(sdev, HDA_DSP_BAR,
+ chip->ssp_base_offset
+ + i * SSP_DEV_MEM_SIZE
+ + SSP_SSC1_OFFSET,
+ SSP_SET_SLAVE,
+ SSP_SET_SLAVE);
+ }
+
+ /* step 2: purge FW request */
+ snd_sof_dsp_write(sdev, HDA_DSP_BAR, chip->ipc_req,
+ chip->ipc_req_mask | (HDA_DSP_IPC_PURGE_FW |
+ ((stream_tag - 1) << 9)));
+
+ /* step 3: unset core 0 reset state & unstall/run core 0 */
+ ret = hda_dsp_core_run(sdev, HDA_DSP_CORE_MASK(0));
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: dsp core start failed %d\n", ret);
+ ret = -EIO;
+ goto err;
+ }
+
+ /* step 4: wait for IPC DONE bit from ROM */
+ ret = snd_sof_dsp_read_poll_timeout(sdev, HDA_DSP_BAR,
+ chip->ipc_ack, status,
+ ((status & chip->ipc_ack_mask)
+ == chip->ipc_ack_mask),
+ HDA_DSP_REG_POLL_INTERVAL_US,
+ HDA_DSP_INIT_TIMEOUT_US);
+
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: waiting for HIPCIE done\n");
+ goto err;
+ }
+
+ /* step 5: power down corex */
+ ret = hda_dsp_core_power_down(sdev,
+ chip->cores_mask & ~(HDA_DSP_CORE_MASK(0)));
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: dsp core x power down failed\n");
+ goto err;
+ }
+
+ /* step 6: enable IPC interrupts */
+ hda_dsp_ipc_int_enable(sdev);
+
+ /* step 7: wait for ROM init */
+ ret = snd_sof_dsp_read_poll_timeout(sdev, HDA_DSP_BAR,
+ HDA_DSP_SRAM_REG_ROM_STATUS, status,
+ ((status & HDA_DSP_ROM_STS_MASK)
+ == HDA_DSP_ROM_INIT),
+ HDA_DSP_REG_POLL_INTERVAL_US,
+ chip->rom_init_timeout *
+ USEC_PER_MSEC);
+ if (!ret)
+ return 0;
+
+err:
+ hda_dsp_dump(sdev, SOF_DBG_REGS | SOF_DBG_PCI | SOF_DBG_MBOX);
+ hda_dsp_core_reset_power_down(sdev, chip->cores_mask);
+
+ return ret;
+}
+
+static int cl_trigger(struct snd_sof_dev *sdev,
+ struct hdac_ext_stream *stream, int cmd)
+{
+ struct hdac_stream *hstream = &stream->hstream;
+ int sd_offset = SOF_STREAM_SD_OFFSET(hstream);
+
+ /* code loader is special case that reuses stream ops */
+ switch (cmd) {
+ case SNDRV_PCM_TRIGGER_START:
+ wait_event_timeout(sdev->waitq, !sdev->code_loading,
+ HDA_DSP_CL_TRIGGER_TIMEOUT);
+
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, SOF_HDA_INTCTL,
+ 1 << hstream->index,
+ 1 << hstream->index);
+
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR,
+ sd_offset,
+ SOF_HDA_SD_CTL_DMA_START |
+ SOF_HDA_CL_DMA_SD_INT_MASK,
+ SOF_HDA_SD_CTL_DMA_START |
+ SOF_HDA_CL_DMA_SD_INT_MASK);
+
+ hstream->running = true;
+ return 0;
+ default:
+ return hda_dsp_stream_trigger(sdev, stream, cmd);
+ }
+}
+
+static struct hdac_ext_stream *get_stream_with_tag(struct snd_sof_dev *sdev,
+ int tag)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct hdac_stream *s;
+
+ /* get stream with tag */
+ list_for_each_entry(s, &bus->stream_list, list) {
+ if (s->direction == SNDRV_PCM_STREAM_PLAYBACK &&
+ s->stream_tag == tag) {
+ return stream_to_hdac_ext_stream(s);
+ }
+ }
+
+ return NULL;
+}
+
+static int cl_cleanup(struct snd_sof_dev *sdev, struct snd_dma_buffer *dmab,
+ struct hdac_ext_stream *stream)
+{
+ struct hdac_stream *hstream = &stream->hstream;
+ int sd_offset = SOF_STREAM_SD_OFFSET(hstream);
+ int ret;
+
+ ret = hda_dsp_stream_spib_config(sdev, stream, HDA_DSP_SPIB_DISABLE, 0);
+
+ hda_dsp_stream_put(sdev, SNDRV_PCM_STREAM_PLAYBACK,
+ hstream->stream_tag);
+ hstream->running = 0;
+ hstream->substream = NULL;
+
+ /* reset BDL address */
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR,
+ sd_offset + SOF_HDA_ADSP_REG_CL_SD_BDLPL, 0);
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR,
+ sd_offset + SOF_HDA_ADSP_REG_CL_SD_BDLPU, 0);
+
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR, sd_offset, 0);
+ snd_dma_free_pages(dmab);
+ dmab->area = NULL;
+ hstream->bufsize = 0;
+ hstream->format_val = 0;
+
+ return ret;
+}
+
+static int cl_copy_fw(struct snd_sof_dev *sdev, struct hdac_ext_stream *stream)
+{
+ unsigned int reg;
+ int ret, status;
+
+ ret = cl_trigger(sdev, stream, SNDRV_PCM_TRIGGER_START);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: DMA trigger start failed\n");
+ return ret;
+ }
+
+ status = snd_sof_dsp_read_poll_timeout(sdev, HDA_DSP_BAR,
+ HDA_DSP_SRAM_REG_ROM_STATUS, reg,
+ ((reg & HDA_DSP_ROM_STS_MASK)
+ == HDA_DSP_ROM_FW_ENTERED),
+ HDA_DSP_REG_POLL_INTERVAL_US,
+ HDA_DSP_BASEFW_TIMEOUT_US);
+
+ ret = cl_trigger(sdev, stream, SNDRV_PCM_TRIGGER_STOP);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: DMA trigger stop failed\n");
+ return ret;
+ }
+
+ return status;
+}
+
+int hda_dsp_cl_boot_firmware(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_pdata *plat_data = sdev->pdata;
+ const struct sof_dev_desc *desc = plat_data->desc;
+ const struct sof_intel_dsp_desc *chip_info;
+ struct hdac_ext_stream *stream;
+ struct firmware stripped_firmware;
+ int ret, ret1, tag, i;
+
+ chip_info = desc->chip_info;
+
+ stripped_firmware.data = plat_data->fw->data;
+ stripped_firmware.size = plat_data->fw->size;
+
+ /* init for booting wait */
+ init_waitqueue_head(&sdev->boot_wait);
+ sdev->boot_complete = false;
+
+ /* prepare DMA for code loader stream */
+ tag = cl_stream_prepare(sdev, 0x40, stripped_firmware.size,
+ &sdev->dmab, SNDRV_PCM_STREAM_PLAYBACK);
+
+ if (tag < 0) {
+ dev_err(sdev->dev, "error: dma prepare for fw loading err: %x\n",
+ tag);
+ return tag;
+ }
+
+ /* get stream with tag */
+ stream = get_stream_with_tag(sdev, tag);
+ if (!stream) {
+ dev_err(sdev->dev,
+ "error: could not get stream with stream tag %d\n",
+ tag);
+ ret = -ENODEV;
+ goto err;
+ }
+
+ memcpy(sdev->dmab.area, stripped_firmware.data,
+ stripped_firmware.size);
+
+ /* try ROM init a few times before giving up */
+ for (i = 0; i < HDA_FW_BOOT_ATTEMPTS; i++) {
+ ret = cl_dsp_init(sdev, stripped_firmware.data,
+ stripped_firmware.size, tag);
+
+ /* don't retry anymore if successful */
+ if (!ret)
+ break;
+
+ dev_err(sdev->dev, "error: Error code=0x%x: FW status=0x%x\n",
+ snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_DSP_SRAM_REG_ROM_ERROR),
+ snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_DSP_SRAM_REG_ROM_STATUS));
+ dev_err(sdev->dev, "error: iteration %d of Core En/ROM load failed: %d\n",
+ i, ret);
+ }
+
+ if (i == HDA_FW_BOOT_ATTEMPTS) {
+ dev_err(sdev->dev, "error: dsp init failed after %d attempts with err: %d\n",
+ i, ret);
+ goto cleanup;
+ }
+
+ /*
+ * at this point DSP ROM has been initialized and
+ * should be ready for code loading and firmware boot
+ */
+ ret = cl_copy_fw(sdev, stream);
+ if (!ret)
+ dev_dbg(sdev->dev, "Firmware download successful, booting...\n");
+ else
+ dev_err(sdev->dev, "error: load fw failed ret: %d\n", ret);
+
+cleanup:
+ /*
+ * Perform codeloader stream cleanup.
+ * This should be done even if firmware loading fails.
+ */
+ ret1 = cl_cleanup(sdev, &sdev->dmab, stream);
+ if (ret1 < 0) {
+ dev_err(sdev->dev, "error: Code loader DSP cleanup failed\n");
+
+ /* set return value to indicate cleanup failure */
+ ret = ret1;
+ }
+
+ /*
+ * return master core id if both fw copy
+ * and stream clean up are successful
+ */
+ if (!ret)
+ return chip_info->init_core_mask;
+
+ /* dump dsp registers and disable DSP upon error */
+err:
+ hda_dsp_dump(sdev, SOF_DBG_REGS | SOF_DBG_PCI | SOF_DBG_MBOX);
+
+ /* disable DSP */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_PP_BAR,
+ SOF_HDA_REG_PP_PPCTL,
+ SOF_HDA_PPCTL_GPROCEN, 0);
+ return ret;
+}
+
+/* pre fw run operations */
+int hda_dsp_pre_fw_run(struct snd_sof_dev *sdev)
+{
+ /* disable clock gating and power gating */
+ return hda_dsp_ctrl_clock_power_gating(sdev, false);
+}
+
+/* post fw run operations */
+int hda_dsp_post_fw_run(struct snd_sof_dev *sdev)
+{
+ /* re-enable clock gating and power gating */
+ return hda_dsp_ctrl_clock_power_gating(sdev, true);
+}
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+// Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
+// Rander Wang <rander.wang@intel.com>
+// Keyon Jie <yang.jie@linux.intel.com>
+//
+
+/*
+ * Hardware interface for generic Intel audio DSP HDA IP
+ */
+
+#include <sound/hda_register.h>
+#include <sound/pcm_params.h>
+#include "../ops.h"
+#include "hda.h"
+
+#define SDnFMT_BASE(x) ((x) << 14)
+#define SDnFMT_MULT(x) (((x) - 1) << 11)
+#define SDnFMT_DIV(x) (((x) - 1) << 8)
+#define SDnFMT_BITS(x) ((x) << 4)
+#define SDnFMT_CHAN(x) ((x) << 0)
+
+static inline u32 get_mult_div(struct snd_sof_dev *sdev, int rate)
+{
+ switch (rate) {
+ case 8000:
+ return SDnFMT_DIV(6);
+ case 9600:
+ return SDnFMT_DIV(5);
+ case 11025:
+ return SDnFMT_BASE(1) | SDnFMT_DIV(4);
+ case 16000:
+ return SDnFMT_DIV(3);
+ case 22050:
+ return SDnFMT_BASE(1) | SDnFMT_DIV(2);
+ case 32000:
+ return SDnFMT_DIV(3) | SDnFMT_MULT(2);
+ case 44100:
+ return SDnFMT_BASE(1);
+ case 48000:
+ return 0;
+ case 88200:
+ return SDnFMT_BASE(1) | SDnFMT_MULT(2);
+ case 96000:
+ return SDnFMT_MULT(2);
+ case 176400:
+ return SDnFMT_BASE(1) | SDnFMT_MULT(4);
+ case 192000:
+ return SDnFMT_MULT(4);
+ default:
+ dev_warn(sdev->dev, "can't find div rate %d using 48kHz\n",
+ rate);
+ return 0; /* use 48KHz if not found */
+ }
+};
+
+static inline u32 get_bits(struct snd_sof_dev *sdev, int sample_bits)
+{
+ switch (sample_bits) {
+ case 8:
+ return SDnFMT_BITS(0);
+ case 16:
+ return SDnFMT_BITS(1);
+ case 20:
+ return SDnFMT_BITS(2);
+ case 24:
+ return SDnFMT_BITS(3);
+ case 32:
+ return SDnFMT_BITS(4);
+ default:
+ dev_warn(sdev->dev, "can't find %d bits using 16bit\n",
+ sample_bits);
+ return SDnFMT_BITS(1); /* use 16bits format if not found */
+ }
+};
+
+int hda_dsp_pcm_hw_params(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params,
+ struct sof_ipc_stream_params *ipc_params)
+{
+ struct hdac_stream *hstream = substream->runtime->private_data;
+ struct hdac_ext_stream *stream = stream_to_hdac_ext_stream(hstream);
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ struct snd_dma_buffer *dmab;
+ int ret;
+ u32 size, rate, bits;
+
+ size = params_buffer_bytes(params);
+ rate = get_mult_div(sdev, params_rate(params));
+ bits = get_bits(sdev, params_width(params));
+
+ hstream->substream = substream;
+
+ dmab = substream->runtime->dma_buffer_p;
+
+ hstream->format_val = rate | bits | (params_channels(params) - 1);
+ hstream->bufsize = size;
+ hstream->period_bytes = params_period_bytes(params);
+ hstream->no_period_wakeup =
+ (params->info & SNDRV_PCM_INFO_NO_PERIOD_WAKEUP) &&
+ (params->flags & SNDRV_PCM_HW_PARAMS_NO_PERIOD_WAKEUP);
+
+ ret = hda_dsp_stream_hw_params(sdev, stream, dmab, params);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: hdac prepare failed: %x\n", ret);
+ return ret;
+ }
+
+ /* disable SPIB, to enable buffer wrap for stream */
+ hda_dsp_stream_spib_config(sdev, stream, HDA_DSP_SPIB_DISABLE, 0);
+
+ /* set host_period_bytes to 0 if no IPC position */
+ if (hda && hda->no_ipc_position)
+ ipc_params->host_period_bytes = 0;
+
+ ipc_params->stream_tag = hstream->stream_tag;
+
+ return 0;
+}
+
+int hda_dsp_pcm_trigger(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream, int cmd)
+{
+ struct hdac_stream *hstream = substream->runtime->private_data;
+ struct hdac_ext_stream *stream = stream_to_hdac_ext_stream(hstream);
+
+ return hda_dsp_stream_trigger(sdev, stream, cmd);
+}
+
+snd_pcm_uframes_t hda_dsp_pcm_pointer(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct hdac_stream *hstream = substream->runtime->private_data;
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ struct snd_sof_pcm *spcm;
+ snd_pcm_uframes_t pos;
+
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm) {
+ dev_warn_ratelimited(sdev->dev, "warn: can't find PCM with DAI ID %d\n",
+ rtd->dai_link->id);
+ return 0;
+ }
+
+ if (hda && !hda->no_ipc_position) {
+ /* read position from IPC position */
+ pos = spcm->stream[substream->stream].posn.host_posn;
+ goto found;
+ }
+
+ /*
+ * DPIB/posbuf position mode:
+ * For Playback, Use DPIB register from HDA space which
+ * reflects the actual data transferred.
+ * For Capture, Use the position buffer for pointer, as DPIB
+ * is not accurate enough, its update may be completed
+ * earlier than the data written to DDR.
+ */
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
+ pos = snd_sof_dsp_read(sdev, HDA_DSP_HDA_BAR,
+ AZX_REG_VS_SDXDPIB_XBASE +
+ (AZX_REG_VS_SDXDPIB_XINTERVAL *
+ hstream->index));
+ } else {
+ /*
+ * For capture stream, we need more workaround to fix the
+ * position incorrect issue:
+ *
+ * 1. Wait at least 20us before reading position buffer after
+ * the interrupt generated(IOC), to make sure position update
+ * happens on frame boundary i.e. 20.833uSec for 48KHz.
+ * 2. Perform a dummy Read to DPIB register to flush DMA
+ * position value.
+ * 3. Read the DMA Position from posbuf. Now the readback
+ * value should be >= period boundary.
+ */
+ usleep_range(20, 21);
+ snd_sof_dsp_read(sdev, HDA_DSP_HDA_BAR,
+ AZX_REG_VS_SDXDPIB_XBASE +
+ (AZX_REG_VS_SDXDPIB_XINTERVAL *
+ hstream->index));
+ pos = snd_hdac_stream_get_pos_posbuf(hstream);
+ }
+
+ if (pos >= hstream->bufsize)
+ pos = 0;
+
+found:
+ pos = bytes_to_frames(substream->runtime, pos);
+
+ dev_vdbg(sdev->dev, "PCM: stream %d dir %d position %lu\n",
+ hstream->index, substream->stream, pos);
+ return pos;
+}
+
+int hda_dsp_pcm_open(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream)
+{
+ struct hdac_ext_stream *dsp_stream;
+ int direction = substream->stream;
+
+ dsp_stream = hda_dsp_stream_get(sdev, direction);
+
+ if (!dsp_stream) {
+ dev_err(sdev->dev, "error: no stream available\n");
+ return -ENODEV;
+ }
+
+ /* binding pcm substream to hda stream */
+ substream->runtime->private_data = &dsp_stream->hstream;
+ return 0;
+}
+
+int hda_dsp_pcm_close(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream)
+{
+ struct hdac_stream *hstream = substream->runtime->private_data;
+ int direction = substream->stream;
+ int ret;
+
+ ret = hda_dsp_stream_put(sdev, direction, hstream->stream_tag);
+
+ if (ret) {
+ dev_dbg(sdev->dev, "stream %s not opened!\n", substream->name);
+ return -ENODEV;
+ }
+
+ /* unbinding pcm substream to hda stream */
+ substream->runtime->private_data = NULL;
+ return 0;
+}
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+// Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
+// Rander Wang <rander.wang@intel.com>
+// Keyon Jie <yang.jie@linux.intel.com>
+//
+
+/*
+ * Hardware interface for generic Intel audio DSP HDA IP
+ */
+
+#include <linux/pm_runtime.h>
+#include <sound/hdaudio_ext.h>
+#include <sound/hda_register.h>
+#include <sound/sof.h>
+#include "../ops.h"
+#include "hda.h"
+
+/*
+ * set up one of BDL entries for a stream
+ */
+static int hda_setup_bdle(struct snd_sof_dev *sdev,
+ struct snd_dma_buffer *dmab,
+ struct hdac_stream *stream,
+ struct sof_intel_dsp_bdl **bdlp,
+ int offset, int size, int ioc)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct sof_intel_dsp_bdl *bdl = *bdlp;
+
+ while (size > 0) {
+ dma_addr_t addr;
+ int chunk;
+
+ if (stream->frags >= HDA_DSP_MAX_BDL_ENTRIES) {
+ dev_err(sdev->dev, "error: stream frags exceeded\n");
+ return -EINVAL;
+ }
+
+ addr = snd_sgbuf_get_addr(dmab, offset);
+ /* program BDL addr */
+ bdl->addr_l = cpu_to_le32(lower_32_bits(addr));
+ bdl->addr_h = cpu_to_le32(upper_32_bits(addr));
+ /* program BDL size */
+ chunk = snd_sgbuf_get_chunk_size(dmab, offset, size);
+ /* one BDLE should not cross 4K boundary */
+ if (bus->align_bdle_4k) {
+ u32 remain = 0x1000 - (offset & 0xfff);
+
+ if (chunk > remain)
+ chunk = remain;
+ }
+ bdl->size = cpu_to_le32(chunk);
+ /* only program IOC when the whole segment is processed */
+ size -= chunk;
+ bdl->ioc = (size || !ioc) ? 0 : cpu_to_le32(0x01);
+ bdl++;
+ stream->frags++;
+ offset += chunk;
+
+ dev_vdbg(sdev->dev, "bdl, frags:%d, chunk size:0x%x;\n",
+ stream->frags, chunk);
+ }
+
+ *bdlp = bdl;
+ return offset;
+}
+
+/*
+ * set up Buffer Descriptor List (BDL) for host memory transfer
+ * BDL describes the location of the individual buffers and is little endian.
+ */
+int hda_dsp_stream_setup_bdl(struct snd_sof_dev *sdev,
+ struct snd_dma_buffer *dmab,
+ struct hdac_stream *stream)
+{
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ struct sof_intel_dsp_bdl *bdl;
+ int i, offset, period_bytes, periods;
+ int remain, ioc;
+
+ period_bytes = stream->period_bytes;
+ dev_dbg(sdev->dev, "period_bytes:0x%x\n", period_bytes);
+ if (!period_bytes)
+ period_bytes = stream->bufsize;
+
+ periods = stream->bufsize / period_bytes;
+
+ dev_dbg(sdev->dev, "periods:%d\n", periods);
+
+ remain = stream->bufsize % period_bytes;
+ if (remain)
+ periods++;
+
+ /* program the initial BDL entries */
+ bdl = (struct sof_intel_dsp_bdl *)stream->bdl.area;
+ offset = 0;
+ stream->frags = 0;
+
+ /*
+ * set IOC if don't use position IPC
+ * and period_wakeup needed.
+ */
+ ioc = hda->no_ipc_position ?
+ !stream->no_period_wakeup : 0;
+
+ for (i = 0; i < periods; i++) {
+ if (i == (periods - 1) && remain)
+ /* set the last small entry */
+ offset = hda_setup_bdle(sdev, dmab,
+ stream, &bdl, offset,
+ remain, 0);
+ else
+ offset = hda_setup_bdle(sdev, dmab,
+ stream, &bdl, offset,
+ period_bytes, ioc);
+ }
+
+ return offset;
+}
+
+int hda_dsp_stream_spib_config(struct snd_sof_dev *sdev,
+ struct hdac_ext_stream *stream,
+ int enable, u32 size)
+{
+ struct hdac_stream *hstream = &stream->hstream;
+ u32 mask;
+
+ if (!sdev->bar[HDA_DSP_SPIB_BAR]) {
+ dev_err(sdev->dev, "error: address of spib capability is NULL\n");
+ return -EINVAL;
+ }
+
+ mask = (1 << hstream->index);
+
+ /* enable/disable SPIB for the stream */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_SPIB_BAR,
+ SOF_HDA_ADSP_REG_CL_SPBFIFO_SPBFCCTL, mask,
+ enable << hstream->index);
+
+ /* set the SPIB value */
+ sof_io_write(sdev, stream->spib_addr, size);
+
+ return 0;
+}
+
+/* get next unused stream */
+struct hdac_ext_stream *
+hda_dsp_stream_get(struct snd_sof_dev *sdev, int direction)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct hdac_ext_stream *stream = NULL;
+ struct hdac_stream *s;
+
+ spin_lock_irq(&bus->reg_lock);
+
+ /* get an unused stream */
+ list_for_each_entry(s, &bus->stream_list, list) {
+ if (s->direction == direction && !s->opened) {
+ s->opened = true;
+ stream = stream_to_hdac_ext_stream(s);
+ break;
+ }
+ }
+
+ spin_unlock_irq(&bus->reg_lock);
+
+ /* stream found ? */
+ if (!stream)
+ dev_err(sdev->dev, "error: no free %s streams\n",
+ direction == SNDRV_PCM_STREAM_PLAYBACK ?
+ "playback" : "capture");
+
+ return stream;
+}
+
+/* free a stream */
+int hda_dsp_stream_put(struct snd_sof_dev *sdev, int direction, int stream_tag)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct hdac_stream *s;
+
+ spin_lock_irq(&bus->reg_lock);
+
+ /* find used stream */
+ list_for_each_entry(s, &bus->stream_list, list) {
+ if (s->direction == direction &&
+ s->opened && s->stream_tag == stream_tag) {
+ s->opened = false;
+ spin_unlock_irq(&bus->reg_lock);
+ return 0;
+ }
+ }
+
+ spin_unlock_irq(&bus->reg_lock);
+
+ dev_dbg(sdev->dev, "stream_tag %d not opened!\n", stream_tag);
+ return -ENODEV;
+}
+
+int hda_dsp_stream_trigger(struct snd_sof_dev *sdev,
+ struct hdac_ext_stream *stream, int cmd)
+{
+ struct hdac_stream *hstream = &stream->hstream;
+ int sd_offset = SOF_STREAM_SD_OFFSET(hstream);
+
+ /* cmd must be for audio stream */
+ switch (cmd) {
+ case SNDRV_PCM_TRIGGER_RESUME:
+ case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
+ case SNDRV_PCM_TRIGGER_START:
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, SOF_HDA_INTCTL,
+ 1 << hstream->index,
+ 1 << hstream->index);
+
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR,
+ sd_offset,
+ SOF_HDA_SD_CTL_DMA_START |
+ SOF_HDA_CL_DMA_SD_INT_MASK,
+ SOF_HDA_SD_CTL_DMA_START |
+ SOF_HDA_CL_DMA_SD_INT_MASK);
+
+ hstream->running = true;
+ break;
+ case SNDRV_PCM_TRIGGER_SUSPEND:
+ case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
+ case SNDRV_PCM_TRIGGER_STOP:
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR,
+ sd_offset,
+ SOF_HDA_SD_CTL_DMA_START |
+ SOF_HDA_CL_DMA_SD_INT_MASK, 0x0);
+
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR, sd_offset +
+ SOF_HDA_ADSP_REG_CL_SD_STS,
+ SOF_HDA_CL_DMA_SD_INT_MASK);
+
+ hstream->running = false;
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, SOF_HDA_INTCTL,
+ 1 << hstream->index, 0x0);
+ break;
+ default:
+ dev_err(sdev->dev, "error: unknown command: %d\n", cmd);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+/*
+ * prepare for common hdac registers settings, for both code loader
+ * and normal stream.
+ */
+int hda_dsp_stream_hw_params(struct snd_sof_dev *sdev,
+ struct hdac_ext_stream *stream,
+ struct snd_dma_buffer *dmab,
+ struct snd_pcm_hw_params *params)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct hdac_stream *hstream = &stream->hstream;
+ int sd_offset = SOF_STREAM_SD_OFFSET(hstream);
+ int ret, timeout = HDA_DSP_STREAM_RESET_TIMEOUT;
+ u32 val, mask;
+
+ if (!stream) {
+ dev_err(sdev->dev, "error: no stream available\n");
+ return -ENODEV;
+ }
+
+ /* decouple host and link DMA */
+ mask = 0x1 << hstream->index;
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_PP_BAR, SOF_HDA_REG_PP_PPCTL,
+ mask, mask);
+
+ if (!dmab) {
+ dev_err(sdev->dev, "error: no dma buffer allocated!\n");
+ return -ENODEV;
+ }
+
+ /* clear stream status */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, sd_offset,
+ SOF_HDA_CL_DMA_SD_INT_MASK |
+ SOF_HDA_SD_CTL_DMA_START, 0);
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR,
+ sd_offset + SOF_HDA_ADSP_REG_CL_SD_STS,
+ SOF_HDA_CL_DMA_SD_INT_MASK,
+ SOF_HDA_CL_DMA_SD_INT_MASK);
+
+ /* stream reset */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, sd_offset, 0x1,
+ 0x1);
+ udelay(3);
+ do {
+ val = snd_sof_dsp_read(sdev, HDA_DSP_HDA_BAR,
+ sd_offset);
+ if (val & 0x1)
+ break;
+ } while (--timeout);
+ if (timeout == 0) {
+ dev_err(sdev->dev, "error: stream reset failed\n");
+ return -ETIMEDOUT;
+ }
+
+ timeout = HDA_DSP_STREAM_RESET_TIMEOUT;
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, sd_offset, 0x1,
+ 0x0);
+
+ /* wait for hardware to report that stream is out of reset */
+ udelay(3);
+ do {
+ val = snd_sof_dsp_read(sdev, HDA_DSP_HDA_BAR,
+ sd_offset);
+ if ((val & 0x1) == 0)
+ break;
+ } while (--timeout);
+ if (timeout == 0) {
+ dev_err(sdev->dev, "error: timeout waiting for stream reset\n");
+ return -ETIMEDOUT;
+ }
+
+ if (hstream->posbuf)
+ *hstream->posbuf = 0;
+
+ /* reset BDL address */
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR,
+ sd_offset + SOF_HDA_ADSP_REG_CL_SD_BDLPL,
+ 0x0);
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR,
+ sd_offset + SOF_HDA_ADSP_REG_CL_SD_BDLPU,
+ 0x0);
+
+ /* clear stream status */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, sd_offset,
+ SOF_HDA_CL_DMA_SD_INT_MASK |
+ SOF_HDA_SD_CTL_DMA_START, 0);
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR,
+ sd_offset + SOF_HDA_ADSP_REG_CL_SD_STS,
+ SOF_HDA_CL_DMA_SD_INT_MASK,
+ SOF_HDA_CL_DMA_SD_INT_MASK);
+
+ hstream->frags = 0;
+
+ ret = hda_dsp_stream_setup_bdl(sdev, dmab, hstream);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: set up of BDL failed\n");
+ return ret;
+ }
+
+ /* program stream tag to set up stream descriptor for DMA */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, sd_offset,
+ SOF_HDA_CL_SD_CTL_STREAM_TAG_MASK,
+ hstream->stream_tag <<
+ SOF_HDA_CL_SD_CTL_STREAM_TAG_SHIFT);
+
+ /* program cyclic buffer length */
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR,
+ sd_offset + SOF_HDA_ADSP_REG_CL_SD_CBL,
+ hstream->bufsize);
+
+ /*
+ * Recommended hardware programming sequence for HDAudio DMA format
+ *
+ * 1. Put DMA into coupled mode by clearing PPCTL.PROCEN bit
+ * for corresponding stream index before the time of writing
+ * format to SDxFMT register.
+ * 2. Write SDxFMT
+ * 3. Set PPCTL.PROCEN bit for corresponding stream index to
+ * enable decoupled mode
+ */
+
+ /* couple host and link DMA, disable DSP features */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_PP_BAR, SOF_HDA_REG_PP_PPCTL,
+ mask, 0);
+
+ /* program stream format */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR,
+ sd_offset +
+ SOF_HDA_ADSP_REG_CL_SD_FORMAT,
+ 0xffff, hstream->format_val);
+
+ /* decouple host and link DMA, enable DSP features */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_PP_BAR, SOF_HDA_REG_PP_PPCTL,
+ mask, mask);
+
+ /* program last valid index */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR,
+ sd_offset + SOF_HDA_ADSP_REG_CL_SD_LVI,
+ 0xffff, (hstream->frags - 1));
+
+ /* program BDL address */
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR,
+ sd_offset + SOF_HDA_ADSP_REG_CL_SD_BDLPL,
+ (u32)hstream->bdl.addr);
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR,
+ sd_offset + SOF_HDA_ADSP_REG_CL_SD_BDLPU,
+ upper_32_bits(hstream->bdl.addr));
+
+ /* enable position buffer */
+ if (!(snd_sof_dsp_read(sdev, HDA_DSP_HDA_BAR, SOF_HDA_ADSP_DPLBASE)
+ & SOF_HDA_ADSP_DPLBASE_ENABLE)) {
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR, SOF_HDA_ADSP_DPUBASE,
+ upper_32_bits(bus->posbuf.addr));
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR, SOF_HDA_ADSP_DPLBASE,
+ (u32)bus->posbuf.addr |
+ SOF_HDA_ADSP_DPLBASE_ENABLE);
+ }
+
+ /* set interrupt enable bits */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, sd_offset,
+ SOF_HDA_CL_DMA_SD_INT_MASK,
+ SOF_HDA_CL_DMA_SD_INT_MASK);
+
+ /* read FIFO size */
+ if (hstream->direction == SNDRV_PCM_STREAM_PLAYBACK) {
+ hstream->fifo_size =
+ snd_sof_dsp_read(sdev, HDA_DSP_HDA_BAR,
+ sd_offset +
+ SOF_HDA_ADSP_REG_CL_SD_FIFOSIZE);
+ hstream->fifo_size &= 0xffff;
+ hstream->fifo_size += 1;
+ } else {
+ hstream->fifo_size = 0;
+ }
+
+ return ret;
+}
+
+irqreturn_t hda_dsp_stream_interrupt(int irq, void *context)
+{
+ struct hdac_bus *bus = context;
+ struct sof_intel_hda_dev *sof_hda = bus_to_sof_hda(bus);
+ u32 stream_mask;
+ u32 status;
+
+ if (!pm_runtime_active(bus->dev))
+ return IRQ_NONE;
+
+ spin_lock(&bus->reg_lock);
+
+ status = snd_hdac_chip_readl(bus, INTSTS);
+ stream_mask = GENMASK(sof_hda->stream_max - 1, 0) | AZX_INT_CTRL_EN;
+
+ /* Not stream interrupt or register inaccessible, ignore it.*/
+ if (!(status & stream_mask) || status == 0xffffffff) {
+ spin_unlock(&bus->reg_lock);
+ return IRQ_NONE;
+ }
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ /* clear rirb int */
+ status = snd_hdac_chip_readb(bus, RIRBSTS);
+ if (status & RIRB_INT_MASK) {
+ if (status & RIRB_INT_RESPONSE)
+ snd_hdac_bus_update_rirb(bus);
+ snd_hdac_chip_writeb(bus, RIRBSTS, RIRB_INT_MASK);
+ }
+#endif
+
+ spin_unlock(&bus->reg_lock);
+
+ return snd_hdac_chip_readl(bus, INTSTS) ? IRQ_WAKE_THREAD : IRQ_HANDLED;
+}
+
+irqreturn_t hda_dsp_stream_threaded_handler(int irq, void *context)
+{
+ struct hdac_bus *bus = context;
+ struct sof_intel_hda_dev *sof_hda = bus_to_sof_hda(bus);
+ u32 status = snd_hdac_chip_readl(bus, INTSTS);
+ struct hdac_stream *s;
+ u32 sd_status;
+
+ /* check streams */
+ list_for_each_entry(s, &bus->stream_list, list) {
+ if (status & (1 << s->index) && s->opened) {
+ sd_status = snd_hdac_stream_readb(s, SD_STS);
+
+ dev_vdbg(bus->dev, "stream %d status 0x%x\n",
+ s->index, sd_status);
+
+ snd_hdac_stream_writeb(s, SD_STS, SD_INT_MASK);
+
+ if (!s->substream ||
+ !s->running ||
+ (sd_status & SOF_HDA_CL_DMA_SD_INT_COMPLETE) == 0)
+ continue;
+
+ /* Inform ALSA only in case not do that with IPC */
+ if (sof_hda->no_ipc_position)
+ snd_sof_pcm_period_elapsed(s->substream);
+
+ }
+ }
+
+ return IRQ_HANDLED;
+}
+
+int hda_dsp_stream_init(struct snd_sof_dev *sdev)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct hdac_ext_stream *stream;
+ struct hdac_stream *hstream;
+ struct pci_dev *pci = to_pci_dev(sdev->dev);
+ struct sof_intel_hda_dev *sof_hda = bus_to_sof_hda(bus);
+ int sd_offset;
+ int i, num_playback, num_capture, num_total, ret;
+ u32 gcap;
+
+ gcap = snd_sof_dsp_read(sdev, HDA_DSP_HDA_BAR, SOF_HDA_GCAP);
+ dev_dbg(sdev->dev, "hda global caps = 0x%x\n", gcap);
+
+ /* get stream count from GCAP */
+ num_capture = (gcap >> 8) & 0x0f;
+ num_playback = (gcap >> 12) & 0x0f;
+ num_total = num_playback + num_capture;
+
+ dev_dbg(sdev->dev, "detected %d playback and %d capture streams\n",
+ num_playback, num_capture);
+
+ if (num_playback >= SOF_HDA_PLAYBACK_STREAMS) {
+ dev_err(sdev->dev, "error: too many playback streams %d\n",
+ num_playback);
+ return -EINVAL;
+ }
+
+ if (num_capture >= SOF_HDA_CAPTURE_STREAMS) {
+ dev_err(sdev->dev, "error: too many capture streams %d\n",
+ num_playback);
+ return -EINVAL;
+ }
+
+ /*
+ * mem alloc for the position buffer
+ * TODO: check position buffer update
+ */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV, &pci->dev,
+ SOF_HDA_DPIB_ENTRY_SIZE * num_total,
+ &bus->posbuf);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: posbuffer dma alloc failed\n");
+ return -ENOMEM;
+ }
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ /* mem alloc for the CORB/RIRB ringbuffers */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV, &pci->dev,
+ PAGE_SIZE, &bus->rb);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: RB alloc failed\n");
+ return -ENOMEM;
+ }
+#endif
+
+ /* create capture streams */
+ for (i = 0; i < num_capture; i++) {
+ struct sof_intel_hda_stream *hda_stream;
+
+ hda_stream = devm_kzalloc(sdev->dev, sizeof(*hda_stream),
+ GFP_KERNEL);
+ if (!hda_stream)
+ return -ENOMEM;
+
+ stream = &hda_stream->hda_stream;
+
+ stream->pphc_addr = sdev->bar[HDA_DSP_PP_BAR] +
+ SOF_HDA_PPHC_BASE + SOF_HDA_PPHC_INTERVAL * i;
+
+ stream->pplc_addr = sdev->bar[HDA_DSP_PP_BAR] +
+ SOF_HDA_PPLC_BASE + SOF_HDA_PPLC_MULTI * num_total +
+ SOF_HDA_PPLC_INTERVAL * i;
+
+ /* do we support SPIB */
+ if (sdev->bar[HDA_DSP_SPIB_BAR]) {
+ stream->spib_addr = sdev->bar[HDA_DSP_SPIB_BAR] +
+ SOF_HDA_SPIB_BASE + SOF_HDA_SPIB_INTERVAL * i +
+ SOF_HDA_SPIB_SPIB;
+
+ stream->fifo_addr = sdev->bar[HDA_DSP_SPIB_BAR] +
+ SOF_HDA_SPIB_BASE + SOF_HDA_SPIB_INTERVAL * i +
+ SOF_HDA_SPIB_MAXFIFO;
+ }
+
+ hstream = &stream->hstream;
+ hstream->bus = bus;
+ hstream->sd_int_sta_mask = 1 << i;
+ hstream->index = i;
+ sd_offset = SOF_STREAM_SD_OFFSET(hstream);
+ hstream->sd_addr = sdev->bar[HDA_DSP_HDA_BAR] + sd_offset;
+ hstream->stream_tag = i + 1;
+ hstream->opened = false;
+ hstream->running = false;
+ hstream->direction = SNDRV_PCM_STREAM_CAPTURE;
+
+ /* memory alloc for stream BDL */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV, &pci->dev,
+ HDA_DSP_BDL_SIZE, &hstream->bdl);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: stream bdl dma alloc failed\n");
+ return -ENOMEM;
+ }
+ hstream->posbuf = (__le32 *)(bus->posbuf.area +
+ (hstream->index) * 8);
+
+ list_add_tail(&hstream->list, &bus->stream_list);
+ }
+
+ /* create playback streams */
+ for (i = num_capture; i < num_total; i++) {
+ struct sof_intel_hda_stream *hda_stream;
+
+ hda_stream = devm_kzalloc(sdev->dev, sizeof(*hda_stream),
+ GFP_KERNEL);
+ if (!hda_stream)
+ return -ENOMEM;
+
+ stream = &hda_stream->hda_stream;
+
+ /* we always have DSP support */
+ stream->pphc_addr = sdev->bar[HDA_DSP_PP_BAR] +
+ SOF_HDA_PPHC_BASE + SOF_HDA_PPHC_INTERVAL * i;
+
+ stream->pplc_addr = sdev->bar[HDA_DSP_PP_BAR] +
+ SOF_HDA_PPLC_BASE + SOF_HDA_PPLC_MULTI * num_total +
+ SOF_HDA_PPLC_INTERVAL * i;
+
+ /* do we support SPIB */
+ if (sdev->bar[HDA_DSP_SPIB_BAR]) {
+ stream->spib_addr = sdev->bar[HDA_DSP_SPIB_BAR] +
+ SOF_HDA_SPIB_BASE + SOF_HDA_SPIB_INTERVAL * i +
+ SOF_HDA_SPIB_SPIB;
+
+ stream->fifo_addr = sdev->bar[HDA_DSP_SPIB_BAR] +
+ SOF_HDA_SPIB_BASE + SOF_HDA_SPIB_INTERVAL * i +
+ SOF_HDA_SPIB_MAXFIFO;
+ }
+
+ hstream = &stream->hstream;
+ hstream->bus = bus;
+ hstream->sd_int_sta_mask = 1 << i;
+ hstream->index = i;
+ sd_offset = SOF_STREAM_SD_OFFSET(hstream);
+ hstream->sd_addr = sdev->bar[HDA_DSP_HDA_BAR] + sd_offset;
+ hstream->stream_tag = i - num_capture + 1;
+ hstream->opened = false;
+ hstream->running = false;
+ hstream->direction = SNDRV_PCM_STREAM_PLAYBACK;
+
+ /* mem alloc for stream BDL */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV, &pci->dev,
+ HDA_DSP_BDL_SIZE, &hstream->bdl);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: stream bdl dma alloc failed\n");
+ return -ENOMEM;
+ }
+
+ hstream->posbuf = (__le32 *)(bus->posbuf.area +
+ (hstream->index) * 8);
+
+ list_add_tail(&hstream->list, &bus->stream_list);
+ }
+
+ /* store total stream count (playback + capture) from GCAP */
+ sof_hda->stream_max = num_total;
+
+ return 0;
+}
+
+void hda_dsp_stream_free(struct snd_sof_dev *sdev)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct hdac_stream *s, *_s;
+ struct hdac_ext_stream *stream;
+ struct sof_intel_hda_stream *hda_stream;
+
+ /* free position buffer */
+ if (bus->posbuf.area)
+ snd_dma_free_pages(&bus->posbuf);
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ /* free position buffer */
+ if (bus->rb.area)
+ snd_dma_free_pages(&bus->rb);
+#endif
+
+ list_for_each_entry_safe(s, _s, &bus->stream_list, list) {
+ /* TODO: decouple */
+
+ /* free bdl buffer */
+ if (s->bdl.area)
+ snd_dma_free_pages(&s->bdl);
+ list_del(&s->list);
+ stream = stream_to_hdac_ext_stream(s);
+ hda_stream = container_of(stream, struct sof_intel_hda_stream,
+ hda_stream);
+ devm_kfree(sdev->dev, hda_stream);
+ }
+}
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+// Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
+// Rander Wang <rander.wang@intel.com>
+// Keyon Jie <yang.jie@linux.intel.com>
+//
+
+/*
+ * Hardware interface for generic Intel audio DSP HDA IP
+ */
+
+#include <sound/hdaudio_ext.h>
+#include "../ops.h"
+#include "hda.h"
+
+static int hda_dsp_trace_prepare(struct snd_sof_dev *sdev)
+{
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ struct hdac_ext_stream *stream = hda->dtrace_stream;
+ struct hdac_stream *hstream = &stream->hstream;
+ struct snd_dma_buffer *dmab = &sdev->dmatb;
+ int ret;
+
+ hstream->period_bytes = 0;/* initialize period_bytes */
+ hstream->bufsize = sdev->dmatb.bytes;
+
+ ret = hda_dsp_stream_hw_params(sdev, stream, dmab, NULL);
+ if (ret < 0)
+ dev_err(sdev->dev, "error: hdac prepare failed: %x\n", ret);
+
+ return ret;
+}
+
+int hda_dsp_trace_init(struct snd_sof_dev *sdev, u32 *stream_tag)
+{
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ int ret;
+
+ hda->dtrace_stream = hda_dsp_stream_get(sdev,
+ SNDRV_PCM_STREAM_CAPTURE);
+
+ if (!hda->dtrace_stream) {
+ dev_err(sdev->dev,
+ "error: no available capture stream for DMA trace\n");
+ return -ENODEV;
+ }
+
+ *stream_tag = hda->dtrace_stream->hstream.stream_tag;
+
+ /*
+ * initialize capture stream, set BDL address and return corresponding
+ * stream tag which will be sent to the firmware by IPC message.
+ */
+ ret = hda_dsp_trace_prepare(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: hdac trace init failed: %x\n", ret);
+ hda_dsp_stream_put(sdev, SNDRV_PCM_STREAM_CAPTURE, *stream_tag);
+ hda->dtrace_stream = NULL;
+ *stream_tag = 0;
+ }
+
+ return ret;
+}
+
+int hda_dsp_trace_release(struct snd_sof_dev *sdev)
+{
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ struct hdac_stream *hstream;
+
+ if (hda->dtrace_stream) {
+ hstream = &hda->dtrace_stream->hstream;
+ hda_dsp_stream_put(sdev,
+ SNDRV_PCM_STREAM_CAPTURE,
+ hstream->stream_tag);
+ hda->dtrace_stream = NULL;
+ return 0;
+ }
+
+ dev_dbg(sdev->dev, "DMA trace stream is not opened!\n");
+ return -ENODEV;
+}
+
+int hda_dsp_trace_trigger(struct snd_sof_dev *sdev, int cmd)
+{
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+
+ return hda_dsp_stream_trigger(sdev, hda->dtrace_stream, cmd);
+}
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Authors: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+// Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
+// Rander Wang <rander.wang@intel.com>
+// Keyon Jie <yang.jie@linux.intel.com>
+//
+
+/*
+ * Hardware interface for generic Intel audio DSP HDA IP
+ */
+
+#include <linux/module.h>
+#include <sound/hdaudio_ext.h>
+#include <sound/sof.h>
+#include <sound/sof/xtensa.h>
+#include "../ops.h"
+#include "hda.h"
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA_AUDIO_CODEC)
+#include "../../codecs/hdac_hda.h"
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+#include <sound/soc-acpi-intel-match.h>
+#endif
+
+/* platform specific devices */
+#include "shim.h"
+
+/*
+ * Debug
+ */
+
+struct hda_dsp_msg_code {
+ u32 code;
+ const char *msg;
+};
+
+static const struct hda_dsp_msg_code hda_dsp_rom_msg[] = {
+ {HDA_DSP_ROM_FW_MANIFEST_LOADED, "status: manifest loaded"},
+ {HDA_DSP_ROM_FW_FW_LOADED, "status: fw loaded"},
+ {HDA_DSP_ROM_FW_ENTERED, "status: fw entered"},
+ {HDA_DSP_ROM_CSE_ERROR, "error: cse error"},
+ {HDA_DSP_ROM_CSE_WRONG_RESPONSE, "error: cse wrong response"},
+ {HDA_DSP_ROM_IMR_TO_SMALL, "error: IMR too small"},
+ {HDA_DSP_ROM_BASE_FW_NOT_FOUND, "error: base fw not found"},
+ {HDA_DSP_ROM_CSE_VALIDATION_FAILED, "error: signature verification failed"},
+ {HDA_DSP_ROM_IPC_FATAL_ERROR, "error: ipc fatal error"},
+ {HDA_DSP_ROM_L2_CACHE_ERROR, "error: L2 cache error"},
+ {HDA_DSP_ROM_LOAD_OFFSET_TO_SMALL, "error: load offset too small"},
+ {HDA_DSP_ROM_API_PTR_INVALID, "error: API ptr invalid"},
+ {HDA_DSP_ROM_BASEFW_INCOMPAT, "error: base fw incompatible"},
+ {HDA_DSP_ROM_UNHANDLED_INTERRUPT, "error: unhandled interrupt"},
+ {HDA_DSP_ROM_MEMORY_HOLE_ECC, "error: ECC memory hole"},
+ {HDA_DSP_ROM_KERNEL_EXCEPTION, "error: kernel exception"},
+ {HDA_DSP_ROM_USER_EXCEPTION, "error: user exception"},
+ {HDA_DSP_ROM_UNEXPECTED_RESET, "error: unexpected reset"},
+ {HDA_DSP_ROM_NULL_FW_ENTRY, "error: null FW entry point"},
+};
+
+static void hda_dsp_get_status_skl(struct snd_sof_dev *sdev)
+{
+ u32 status;
+ int i;
+
+ status = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_ADSP_FW_STATUS_SKL);
+
+ for (i = 0; i < ARRAY_SIZE(hda_dsp_rom_msg); i++) {
+ if (status == hda_dsp_rom_msg[i].code) {
+ dev_err(sdev->dev, "%s - code %8.8x\n",
+ hda_dsp_rom_msg[i].msg, status);
+ return;
+ }
+ }
+
+ /* not for us, must be generic sof message */
+ dev_dbg(sdev->dev, "unknown ROM status value %8.8x\n", status);
+}
+
+static void hda_dsp_get_status(struct snd_sof_dev *sdev)
+{
+ u32 status;
+ int i;
+
+ status = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_DSP_SRAM_REG_ROM_STATUS);
+
+ for (i = 0; i < ARRAY_SIZE(hda_dsp_rom_msg); i++) {
+ if (status == hda_dsp_rom_msg[i].code) {
+ dev_err(sdev->dev, "%s - code %8.8x\n",
+ hda_dsp_rom_msg[i].msg, status);
+ return;
+ }
+ }
+
+ /* not for us, must be generic sof message */
+ dev_dbg(sdev->dev, "unknown ROM status value %8.8x\n", status);
+}
+
+static void hda_dsp_get_registers(struct snd_sof_dev *sdev,
+ struct sof_ipc_dsp_oops_xtensa *xoops,
+ struct sof_ipc_panic_info *panic_info,
+ u32 *stack, size_t stack_words)
+{
+ /* first read registers */
+ sof_block_read(sdev, sdev->mmio_bar, sdev->dsp_oops_offset, xoops,
+ sizeof(*xoops));
+
+ /* then get panic info */
+ sof_block_read(sdev, sdev->mmio_bar, sdev->dsp_oops_offset +
+ sizeof(*xoops), panic_info, sizeof(*panic_info));
+
+ /* then get the stack */
+ sof_block_read(sdev, sdev->mmio_bar, sdev->dsp_oops_offset +
+ sizeof(*xoops) + sizeof(*panic_info), stack,
+ stack_words * sizeof(u32));
+}
+
+void hda_dsp_dump_skl(struct snd_sof_dev *sdev, u32 flags)
+{
+ struct sof_ipc_dsp_oops_xtensa xoops;
+ struct sof_ipc_panic_info panic_info;
+ u32 stack[HDA_DSP_STACK_DUMP_SIZE];
+ u32 status, panic;
+
+ /* try APL specific status message types first */
+ hda_dsp_get_status_skl(sdev);
+
+ /* now try generic SOF status messages */
+ status = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_ADSP_ERROR_CODE_SKL);
+
+ /*TODO: Check: there is no define in spec, but it is used in the code*/
+ panic = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_ADSP_ERROR_CODE_SKL + 0x4);
+
+ if (sdev->boot_complete) {
+ hda_dsp_get_registers(sdev, &xoops, &panic_info, stack,
+ HDA_DSP_STACK_DUMP_SIZE);
+ snd_sof_get_status(sdev, status, panic, &xoops, &panic_info,
+ stack, HDA_DSP_STACK_DUMP_SIZE);
+ } else {
+ dev_err(sdev->dev, "error: status = 0x%8.8x panic = 0x%8.8x\n",
+ status, panic);
+ hda_dsp_get_status_skl(sdev);
+ }
+}
+
+void hda_dsp_dump(struct snd_sof_dev *sdev, u32 flags)
+{
+ struct sof_ipc_dsp_oops_xtensa xoops;
+ struct sof_ipc_panic_info panic_info;
+ u32 stack[HDA_DSP_STACK_DUMP_SIZE];
+ u32 status, panic;
+
+ /* try APL specific status message types first */
+ hda_dsp_get_status(sdev);
+
+ /* now try generic SOF status messages */
+ status = snd_sof_dsp_read(sdev, HDA_DSP_BAR,
+ HDA_DSP_SRAM_REG_FW_STATUS);
+ panic = snd_sof_dsp_read(sdev, HDA_DSP_BAR, HDA_DSP_SRAM_REG_FW_TRACEP);
+
+ if (sdev->boot_complete) {
+ hda_dsp_get_registers(sdev, &xoops, &panic_info, stack,
+ HDA_DSP_STACK_DUMP_SIZE);
+ snd_sof_get_status(sdev, status, panic, &xoops, &panic_info,
+ stack, HDA_DSP_STACK_DUMP_SIZE);
+ } else {
+ dev_err(sdev->dev, "error: status = 0x%8.8x panic = 0x%8.8x\n",
+ status, panic);
+ hda_dsp_get_status(sdev);
+ }
+}
+
+void hda_ipc_dump(struct snd_sof_dev *sdev)
+{
+ u32 hipcie;
+ u32 hipct;
+ u32 hipcctl;
+
+ /* read IPC status */
+ hipcie = snd_sof_dsp_read(sdev, HDA_DSP_BAR, HDA_DSP_REG_HIPCIE);
+ hipct = snd_sof_dsp_read(sdev, HDA_DSP_BAR, HDA_DSP_REG_HIPCT);
+ hipcctl = snd_sof_dsp_read(sdev, HDA_DSP_BAR, HDA_DSP_REG_HIPCCTL);
+
+ /* dump the IPC regs */
+ /* TODO: parse the raw msg */
+ dev_err(sdev->dev,
+ "error: host status 0x%8.8x dsp status 0x%8.8x mask 0x%8.8x\n",
+ hipcie, hipct, hipcctl);
+}
+
+static int hda_init(struct snd_sof_dev *sdev)
+{
+ struct hda_bus *hbus;
+ struct hdac_bus *bus;
+ struct hdac_ext_bus_ops *ext_ops = NULL;
+ struct pci_dev *pci = to_pci_dev(sdev->dev);
+ int ret;
+
+ hbus = sof_to_hbus(sdev);
+ bus = sof_to_bus(sdev);
+
+ /* HDA bus init */
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA_AUDIO_CODEC)
+ ext_ops = snd_soc_hdac_hda_get_ops();
+#endif
+ sof_hda_bus_init(bus, &pci->dev, ext_ops);
+ bus->use_posbuf = 1;
+ bus->bdl_pos_adj = 0;
+
+ mutex_init(&hbus->prepare_mutex);
+ hbus->pci = pci;
+ hbus->mixer_assigned = -1;
+ hbus->modelname = "sofbus";
+
+ /* initialise hdac bus */
+ bus->addr = pci_resource_start(pci, 0);
+ bus->remap_addr = pci_ioremap_bar(pci, 0);
+ if (!bus->remap_addr) {
+ dev_err(bus->dev, "error: ioremap error\n");
+ return -ENXIO;
+ }
+
+ /* HDA base */
+ sdev->bar[HDA_DSP_HDA_BAR] = bus->remap_addr;
+
+ /* get controller capabilities */
+ ret = hda_dsp_ctrl_get_caps(sdev);
+ if (ret < 0)
+ dev_err(sdev->dev, "error: get caps error\n");
+
+ return ret;
+}
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+
+static const char *fixup_tplg_name(struct snd_sof_dev *sdev,
+ const char *sof_tplg_filename)
+{
+ const char *tplg_filename = NULL;
+ char *filename;
+ char *split_ext;
+
+ filename = devm_kstrdup(sdev->dev, sof_tplg_filename, GFP_KERNEL);
+ if (!filename)
+ return NULL;
+
+ /* this assumes a .tplg extension */
+ split_ext = strsep(&filename, ".");
+ if (split_ext) {
+ tplg_filename = devm_kasprintf(sdev->dev, GFP_KERNEL,
+ "%s-idisp.tplg", split_ext);
+ if (!tplg_filename)
+ return NULL;
+ }
+ return tplg_filename;
+}
+
+static int hda_init_caps(struct snd_sof_dev *sdev)
+{
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct hdac_ext_link *hlink;
+ struct snd_soc_acpi_mach_params *mach_params;
+ struct snd_soc_acpi_mach *hda_mach;
+ struct snd_sof_pdata *pdata = sdev->pdata;
+ struct snd_soc_acpi_mach *mach;
+ const char *tplg_filename;
+ int codec_num = 0;
+ int ret = 0;
+ int i;
+
+ device_disable_async_suspend(bus->dev);
+
+ /* check if dsp is there */
+ if (bus->ppcap)
+ dev_dbg(sdev->dev, "PP capability, will probe DSP later.\n");
+
+ if (bus->mlcap)
+ snd_hdac_ext_bus_get_ml_capabilities(bus);
+
+ /* init i915 and HDMI codecs */
+ ret = hda_codec_i915_init(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: no HDMI audio devices found\n");
+ return ret;
+ }
+
+ ret = hda_dsp_ctrl_init_chip(sdev, true);
+ if (ret < 0) {
+ dev_err(bus->dev, "error: init chip failed with ret: %d\n", ret);
+ goto out;
+ }
+
+ /* codec detection */
+ if (!bus->codec_mask) {
+ dev_info(bus->dev, "no hda codecs found!\n");
+ } else {
+ dev_info(bus->dev, "hda codecs found, mask %lx\n",
+ bus->codec_mask);
+
+ for (i = 0; i < HDA_MAX_CODECS; i++) {
+ if (bus->codec_mask & (1 << i))
+ codec_num++;
+ }
+
+ /*
+ * If no machine driver is found, then:
+ *
+ * hda machine driver is used if :
+ * 1. there is one HDMI codec and one external HDAudio codec
+ * 2. only HDMI codec
+ */
+ if (!pdata->machine && codec_num <= 2 &&
+ HDA_IDISP_CODEC(bus->codec_mask)) {
+ hda_mach = snd_soc_acpi_intel_hda_machines;
+ pdata->machine = hda_mach;
+
+ /* topology: use the info from hda_machines */
+ pdata->tplg_filename =
+ hda_mach->sof_tplg_filename;
+
+ /* firmware: pick the first in machine list */
+ mach = pdata->desc->machines;
+ pdata->fw_filename = mach->sof_fw_filename;
+
+ dev_info(bus->dev, "using HDA machine driver %s now\n",
+ hda_mach->drv_name);
+
+ /* fixup topology file for HDMI only platforms */
+ if (codec_num == 1) {
+ /* use local variable for readability */
+ tplg_filename = pdata->tplg_filename;
+ tplg_filename = fixup_tplg_name(sdev, tplg_filename);
+ if (!tplg_filename)
+ goto out;
+ pdata->tplg_filename = tplg_filename;
+ }
+ }
+ }
+
+ /* used by hda machine driver to create dai links */
+ if (pdata->machine) {
+ mach_params = (struct snd_soc_acpi_mach_params *)
+ &pdata->machine->mach_params;
+ mach_params->codec_mask = bus->codec_mask;
+ mach_params->platform = dev_name(sdev->dev);
+ }
+
+ /* create codec instances */
+ hda_codec_probe_bus(sdev);
+
+ hda_codec_i915_put(sdev);
+
+ /*
+ * we are done probing so decrement link counts
+ */
+ list_for_each_entry(hlink, &bus->hlink_list, list)
+ snd_hdac_ext_bus_link_put(bus, hlink);
+
+ return 0;
+
+out:
+ hda_codec_i915_exit(sdev);
+ return ret;
+}
+
+#else
+
+static int hda_init_caps(struct snd_sof_dev *sdev)
+{
+ /*
+ * set CGCTL.MISCBDCGE to 0 during reset and set back to 1
+ * when reset finished.
+ * TODO: maybe no need for init_caps?
+ */
+ hda_dsp_ctrl_misc_clock_gating(sdev, 0);
+
+ /* clear WAKESTS */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, SOF_HDA_WAKESTS,
+ SOF_HDA_WAKESTS_INT_MASK,
+ SOF_HDA_WAKESTS_INT_MASK);
+
+ return 0;
+}
+
+#endif
+
+static const struct sof_intel_dsp_desc
+ *get_chip_info(struct snd_sof_pdata *pdata)
+{
+ const struct sof_dev_desc *desc = pdata->desc;
+ const struct sof_intel_dsp_desc *chip_info;
+
+ chip_info = desc->chip_info;
+
+ return chip_info;
+}
+
+int hda_dsp_probe(struct snd_sof_dev *sdev)
+{
+ struct pci_dev *pci = to_pci_dev(sdev->dev);
+ struct sof_intel_hda_dev *hdev;
+ struct hdac_bus *bus;
+ struct hdac_stream *stream;
+ const struct sof_intel_dsp_desc *chip;
+ int sd_offset, ret = 0;
+
+ /*
+ * detect DSP by checking class/subclass/prog-id information
+ * class=04 subclass 03 prog-if 00: no DSP, legacy driver is required
+ * class=04 subclass 01 prog-if 00: DSP is present
+ * (and may be required e.g. for DMIC or SSP support)
+ * class=04 subclass 03 prog-if 80: either of DSP or legacy mode works
+ */
+ if (pci->class == 0x040300) {
+ dev_err(sdev->dev, "error: the DSP is not enabled on this platform, aborting probe\n");
+ return -ENODEV;
+ } else if (pci->class != 0x040100 && pci->class != 0x040380) {
+ dev_err(sdev->dev, "error: unknown PCI class/subclass/prog-if 0x%06x found, aborting probe\n", pci->class);
+ return -ENODEV;
+ }
+ dev_info(sdev->dev, "DSP detected with PCI class/subclass/prog-if 0x%06x\n", pci->class);
+
+ chip = get_chip_info(sdev->pdata);
+ if (!chip) {
+ dev_err(sdev->dev, "error: no such device supported, chip id:%x\n",
+ pci->device);
+ ret = -EIO;
+ goto err;
+ }
+
+ hdev = devm_kzalloc(sdev->dev, sizeof(*hdev), GFP_KERNEL);
+ if (!hdev)
+ return -ENOMEM;
+ sdev->pdata->hw_pdata = hdev;
+ hdev->desc = chip;
+
+ hdev->dmic_dev = platform_device_register_data(sdev->dev, "dmic-codec",
+ PLATFORM_DEVID_NONE,
+ NULL, 0);
+ if (IS_ERR(hdev->dmic_dev)) {
+ dev_err(sdev->dev, "error: failed to create DMIC device\n");
+ return PTR_ERR(hdev->dmic_dev);
+ }
+
+ /*
+ * use position update IPC if either it is forced
+ * or we don't have other choice
+ */
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_DEBUG_FORCE_IPC_POSITION)
+ hdev->no_ipc_position = 0;
+#else
+ hdev->no_ipc_position = sof_ops(sdev)->pcm_pointer ? 1 : 0;
+#endif
+
+ /* set up HDA base */
+ bus = sof_to_bus(sdev);
+ ret = hda_init(sdev);
+ if (ret < 0)
+ goto hdac_bus_unmap;
+
+ /* DSP base */
+ sdev->bar[HDA_DSP_BAR] = pci_ioremap_bar(pci, HDA_DSP_BAR);
+ if (!sdev->bar[HDA_DSP_BAR]) {
+ dev_err(sdev->dev, "error: ioremap error\n");
+ ret = -ENXIO;
+ goto hdac_bus_unmap;
+ }
+
+ sdev->mmio_bar = HDA_DSP_BAR;
+ sdev->mailbox_bar = HDA_DSP_BAR;
+
+ /* allow 64bit DMA address if supported by H/W */
+ if (!dma_set_mask(&pci->dev, DMA_BIT_MASK(64))) {
+ dev_dbg(sdev->dev, "DMA mask is 64 bit\n");
+ dma_set_coherent_mask(&pci->dev, DMA_BIT_MASK(64));
+ } else {
+ dev_dbg(sdev->dev, "DMA mask is 32 bit\n");
+ dma_set_mask(&pci->dev, DMA_BIT_MASK(32));
+ dma_set_coherent_mask(&pci->dev, DMA_BIT_MASK(32));
+ }
+
+ /* init streams */
+ ret = hda_dsp_stream_init(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to init streams\n");
+ /*
+ * not all errors are due to memory issues, but trying
+ * to free everything does not harm
+ */
+ goto free_streams;
+ }
+
+ /*
+ * register our IRQ
+ * let's try to enable msi firstly
+ * if it fails, use legacy interrupt mode
+ * TODO: support interrupt mode selection with kernel parameter
+ * support msi multiple vectors
+ */
+ ret = pci_alloc_irq_vectors(pci, 1, 1, PCI_IRQ_MSI);
+ if (ret < 0) {
+ dev_info(sdev->dev, "use legacy interrupt mode\n");
+ /*
+ * in IO-APIC mode, hda->irq and ipc_irq are using the same
+ * irq number of pci->irq
+ */
+ hdev->irq = pci->irq;
+ sdev->ipc_irq = pci->irq;
+ sdev->msi_enabled = 0;
+ } else {
+ dev_info(sdev->dev, "use msi interrupt mode\n");
+ hdev->irq = pci_irq_vector(pci, 0);
+ /* ipc irq number is the same of hda irq */
+ sdev->ipc_irq = hdev->irq;
+ sdev->msi_enabled = 1;
+ }
+
+ dev_dbg(sdev->dev, "using HDA IRQ %d\n", hdev->irq);
+ ret = request_threaded_irq(hdev->irq, hda_dsp_stream_interrupt,
+ hda_dsp_stream_threaded_handler,
+ IRQF_SHARED, "AudioHDA", bus);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to register HDA IRQ %d\n",
+ hdev->irq);
+ goto free_irq_vector;
+ }
+
+ dev_dbg(sdev->dev, "using IPC IRQ %d\n", sdev->ipc_irq);
+ ret = request_threaded_irq(sdev->ipc_irq, hda_dsp_ipc_irq_handler,
+ sof_ops(sdev)->irq_thread, IRQF_SHARED,
+ "AudioDSP", sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to register IPC IRQ %d\n",
+ sdev->ipc_irq);
+ goto free_hda_irq;
+ }
+
+ pci_set_master(pci);
+ synchronize_irq(pci->irq);
+
+ /*
+ * clear TCSEL to clear playback on some HD Audio
+ * codecs. PCI TCSEL is defined in the Intel manuals.
+ */
+ snd_sof_pci_update_bits(sdev, PCI_TCSEL, 0x07, 0);
+
+ /* init HDA capabilities */
+ ret = hda_init_caps(sdev);
+ if (ret < 0)
+ goto free_ipc_irq;
+
+ /* reset HDA controller */
+ ret = hda_dsp_ctrl_link_reset(sdev, true);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to reset HDA controller\n");
+ goto free_ipc_irq;
+ }
+
+ /* exit HDA controller reset */
+ ret = hda_dsp_ctrl_link_reset(sdev, false);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to exit HDA controller reset\n");
+ goto free_ipc_irq;
+ }
+
+ /* clear stream status */
+ list_for_each_entry(stream, &bus->stream_list, list) {
+ sd_offset = SOF_STREAM_SD_OFFSET(stream);
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR,
+ sd_offset +
+ SOF_HDA_ADSP_REG_CL_SD_STS,
+ SOF_HDA_CL_DMA_SD_INT_MASK,
+ SOF_HDA_CL_DMA_SD_INT_MASK);
+ }
+
+ /* clear WAKESTS */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, SOF_HDA_WAKESTS,
+ SOF_HDA_WAKESTS_INT_MASK,
+ SOF_HDA_WAKESTS_INT_MASK);
+
+ /* clear interrupt status register */
+ snd_sof_dsp_write(sdev, HDA_DSP_HDA_BAR, SOF_HDA_INTSTS,
+ SOF_HDA_INT_CTRL_EN | SOF_HDA_INT_ALL_STREAM);
+
+ /* enable CIE and GIE interrupts */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, SOF_HDA_INTCTL,
+ SOF_HDA_INT_CTRL_EN | SOF_HDA_INT_GLOBAL_EN,
+ SOF_HDA_INT_CTRL_EN | SOF_HDA_INT_GLOBAL_EN);
+
+ /* re-enable CGCTL.MISCBDCGE after reset */
+ hda_dsp_ctrl_misc_clock_gating(sdev, true);
+
+ device_disable_async_suspend(&pci->dev);
+
+ /* enable DSP features */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_PP_BAR, SOF_HDA_REG_PP_PPCTL,
+ SOF_HDA_PPCTL_GPROCEN, SOF_HDA_PPCTL_GPROCEN);
+
+ /* enable DSP IRQ */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_PP_BAR, SOF_HDA_REG_PP_PPCTL,
+ SOF_HDA_PPCTL_PIE, SOF_HDA_PPCTL_PIE);
+
+ /* initialize waitq for code loading */
+ init_waitqueue_head(&sdev->waitq);
+
+ /* set default mailbox offset for FW ready message */
+ sdev->dsp_box.offset = HDA_DSP_MBOX_UPLINK_OFFSET;
+
+ return 0;
+
+free_ipc_irq:
+ free_irq(sdev->ipc_irq, sdev);
+free_hda_irq:
+ free_irq(hdev->irq, bus);
+free_irq_vector:
+ if (sdev->msi_enabled)
+ pci_free_irq_vectors(pci);
+free_streams:
+ hda_dsp_stream_free(sdev);
+/* dsp_unmap: not currently used */
+ iounmap(sdev->bar[HDA_DSP_BAR]);
+hdac_bus_unmap:
+ iounmap(bus->remap_addr);
+err:
+ return ret;
+}
+
+int hda_dsp_remove(struct snd_sof_dev *sdev)
+{
+ struct sof_intel_hda_dev *hda = sdev->pdata->hw_pdata;
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ struct pci_dev *pci = to_pci_dev(sdev->dev);
+ const struct sof_intel_dsp_desc *chip = hda->desc;
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ /* codec removal, invoke bus_device_remove */
+ snd_hdac_ext_bus_device_remove(bus);
+#endif
+
+ if (!IS_ERR_OR_NULL(hda->dmic_dev))
+ platform_device_unregister(hda->dmic_dev);
+
+ /* disable DSP IRQ */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_PP_BAR, SOF_HDA_REG_PP_PPCTL,
+ SOF_HDA_PPCTL_PIE, 0);
+
+ /* disable CIE and GIE interrupts */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR, SOF_HDA_INTCTL,
+ SOF_HDA_INT_CTRL_EN | SOF_HDA_INT_GLOBAL_EN, 0);
+
+ /* disable cores */
+ if (chip)
+ hda_dsp_core_reset_power_down(sdev, chip->cores_mask);
+
+ /* disable DSP */
+ snd_sof_dsp_update_bits(sdev, HDA_DSP_PP_BAR, SOF_HDA_REG_PP_PPCTL,
+ SOF_HDA_PPCTL_GPROCEN, 0);
+
+ free_irq(sdev->ipc_irq, sdev);
+ free_irq(hda->irq, bus);
+ if (sdev->msi_enabled)
+ pci_free_irq_vectors(pci);
+
+ hda_dsp_stream_free(sdev);
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ snd_hdac_link_free_all(bus);
+#endif
+
+ iounmap(sdev->bar[HDA_DSP_BAR]);
+ iounmap(bus->remap_addr);
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ snd_hdac_ext_bus_exit(bus);
+#endif
+ hda_codec_i915_exit(sdev);
+
+ return 0;
+}
+
+MODULE_LICENSE("Dual BSD/GPL");
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ *
+ * Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+ */
+
+#ifndef __SOF_INTEL_HDA_H
+#define __SOF_INTEL_HDA_H
+
+#include <sound/hda_codec.h>
+#include <sound/hdaudio_ext.h>
+#include "shim.h"
+
+/* PCI registers */
+#define PCI_TCSEL 0x44
+#define PCI_PGCTL PCI_TCSEL
+#define PCI_CGCTL 0x48
+
+/* PCI_PGCTL bits */
+#define PCI_PGCTL_ADSPPGD BIT(2)
+#define PCI_PGCTL_LSRMD_MASK BIT(4)
+
+/* PCI_CGCTL bits */
+#define PCI_CGCTL_MISCBDCGE_MASK BIT(6)
+#define PCI_CGCTL_ADSPDCGE BIT(1)
+
+/* Legacy HDA registers and bits used - widths are variable */
+#define SOF_HDA_GCAP 0x0
+#define SOF_HDA_GCTL 0x8
+/* accept unsol. response enable */
+#define SOF_HDA_GCTL_UNSOL BIT(8)
+#define SOF_HDA_LLCH 0x14
+#define SOF_HDA_INTCTL 0x20
+#define SOF_HDA_INTSTS 0x24
+#define SOF_HDA_WAKESTS 0x0E
+#define SOF_HDA_WAKESTS_INT_MASK ((1 << 8) - 1)
+#define SOF_HDA_RIRBSTS 0x5d
+#define SOF_HDA_VS_EM2_L1SEN BIT(13)
+
+/* SOF_HDA_GCTL register bist */
+#define SOF_HDA_GCTL_RESET BIT(0)
+
+/* SOF_HDA_INCTL and SOF_HDA_INTSTS regs */
+#define SOF_HDA_INT_GLOBAL_EN BIT(31)
+#define SOF_HDA_INT_CTRL_EN BIT(30)
+#define SOF_HDA_INT_ALL_STREAM 0xff
+
+#define SOF_HDA_MAX_CAPS 10
+#define SOF_HDA_CAP_ID_OFF 16
+#define SOF_HDA_CAP_ID_MASK GENMASK(SOF_HDA_CAP_ID_OFF + 11,\
+ SOF_HDA_CAP_ID_OFF)
+#define SOF_HDA_CAP_NEXT_MASK 0xFFFF
+
+#define SOF_HDA_GTS_CAP_ID 0x1
+#define SOF_HDA_ML_CAP_ID 0x2
+
+#define SOF_HDA_PP_CAP_ID 0x3
+#define SOF_HDA_REG_PP_PPCH 0x10
+#define SOF_HDA_REG_PP_PPCTL 0x04
+#define SOF_HDA_PPCTL_PIE BIT(31)
+#define SOF_HDA_PPCTL_GPROCEN BIT(30)
+
+/* DPIB entry size: 8 Bytes = 2 DWords */
+#define SOF_HDA_DPIB_ENTRY_SIZE 0x8
+
+#define SOF_HDA_SPIB_CAP_ID 0x4
+#define SOF_HDA_DRSM_CAP_ID 0x5
+
+#define SOF_HDA_SPIB_BASE 0x08
+#define SOF_HDA_SPIB_INTERVAL 0x08
+#define SOF_HDA_SPIB_SPIB 0x00
+#define SOF_HDA_SPIB_MAXFIFO 0x04
+
+#define SOF_HDA_PPHC_BASE 0x10
+#define SOF_HDA_PPHC_INTERVAL 0x10
+
+#define SOF_HDA_PPLC_BASE 0x10
+#define SOF_HDA_PPLC_MULTI 0x10
+#define SOF_HDA_PPLC_INTERVAL 0x10
+
+#define SOF_HDA_DRSM_BASE 0x08
+#define SOF_HDA_DRSM_INTERVAL 0x08
+
+/* Descriptor error interrupt */
+#define SOF_HDA_CL_DMA_SD_INT_DESC_ERR 0x10
+
+/* FIFO error interrupt */
+#define SOF_HDA_CL_DMA_SD_INT_FIFO_ERR 0x08
+
+/* Buffer completion interrupt */
+#define SOF_HDA_CL_DMA_SD_INT_COMPLETE 0x04
+
+#define SOF_HDA_CL_DMA_SD_INT_MASK \
+ (SOF_HDA_CL_DMA_SD_INT_DESC_ERR | \
+ SOF_HDA_CL_DMA_SD_INT_FIFO_ERR | \
+ SOF_HDA_CL_DMA_SD_INT_COMPLETE)
+#define SOF_HDA_SD_CTL_DMA_START 0x02 /* Stream DMA start bit */
+
+/* Intel HD Audio Code Loader DMA Registers */
+#define SOF_HDA_ADSP_LOADER_BASE 0x80
+#define SOF_HDA_ADSP_DPLBASE 0x70
+#define SOF_HDA_ADSP_DPUBASE 0x74
+#define SOF_HDA_ADSP_DPLBASE_ENABLE 0x01
+
+/* Stream Registers */
+#define SOF_HDA_ADSP_REG_CL_SD_CTL 0x00
+#define SOF_HDA_ADSP_REG_CL_SD_STS 0x03
+#define SOF_HDA_ADSP_REG_CL_SD_LPIB 0x04
+#define SOF_HDA_ADSP_REG_CL_SD_CBL 0x08
+#define SOF_HDA_ADSP_REG_CL_SD_LVI 0x0C
+#define SOF_HDA_ADSP_REG_CL_SD_FIFOW 0x0E
+#define SOF_HDA_ADSP_REG_CL_SD_FIFOSIZE 0x10
+#define SOF_HDA_ADSP_REG_CL_SD_FORMAT 0x12
+#define SOF_HDA_ADSP_REG_CL_SD_FIFOL 0x14
+#define SOF_HDA_ADSP_REG_CL_SD_BDLPL 0x18
+#define SOF_HDA_ADSP_REG_CL_SD_BDLPU 0x1C
+#define SOF_HDA_ADSP_SD_ENTRY_SIZE 0x20
+
+/* CL: Software Position Based FIFO Capability Registers */
+#define SOF_DSP_REG_CL_SPBFIFO \
+ (SOF_HDA_ADSP_LOADER_BASE + 0x20)
+#define SOF_HDA_ADSP_REG_CL_SPBFIFO_SPBFCH 0x0
+#define SOF_HDA_ADSP_REG_CL_SPBFIFO_SPBFCCTL 0x4
+#define SOF_HDA_ADSP_REG_CL_SPBFIFO_SPIB 0x8
+#define SOF_HDA_ADSP_REG_CL_SPBFIFO_MAXFIFOS 0xc
+
+/* Stream Number */
+#define SOF_HDA_CL_SD_CTL_STREAM_TAG_SHIFT 20
+#define SOF_HDA_CL_SD_CTL_STREAM_TAG_MASK \
+ GENMASK(SOF_HDA_CL_SD_CTL_STREAM_TAG_SHIFT + 3,\
+ SOF_HDA_CL_SD_CTL_STREAM_TAG_SHIFT)
+
+#define HDA_DSP_HDA_BAR 0
+#define HDA_DSP_PP_BAR 1
+#define HDA_DSP_SPIB_BAR 2
+#define HDA_DSP_DRSM_BAR 3
+#define HDA_DSP_BAR 4
+
+#define SRAM_WINDOW_OFFSET(x) (0x80000 + (x) * 0x20000)
+
+#define HDA_DSP_MBOX_OFFSET SRAM_WINDOW_OFFSET(0)
+
+#define HDA_DSP_PANIC_OFFSET(x) \
+ (((x) & 0xFFFFFF) + HDA_DSP_MBOX_OFFSET)
+
+/* SRAM window 0 FW "registers" */
+#define HDA_DSP_SRAM_REG_ROM_STATUS (HDA_DSP_MBOX_OFFSET + 0x0)
+#define HDA_DSP_SRAM_REG_ROM_ERROR (HDA_DSP_MBOX_OFFSET + 0x4)
+/* FW and ROM share offset 4 */
+#define HDA_DSP_SRAM_REG_FW_STATUS (HDA_DSP_MBOX_OFFSET + 0x4)
+#define HDA_DSP_SRAM_REG_FW_TRACEP (HDA_DSP_MBOX_OFFSET + 0x8)
+#define HDA_DSP_SRAM_REG_FW_END (HDA_DSP_MBOX_OFFSET + 0xc)
+
+#define HDA_DSP_MBOX_UPLINK_OFFSET 0x81000
+
+#define HDA_DSP_STREAM_RESET_TIMEOUT 300
+#define HDA_DSP_CL_TRIGGER_TIMEOUT 300
+
+#define HDA_DSP_SPIB_ENABLE 1
+#define HDA_DSP_SPIB_DISABLE 0
+
+#define SOF_HDA_MAX_BUFFER_SIZE (32 * PAGE_SIZE)
+
+#define HDA_DSP_STACK_DUMP_SIZE 32
+
+/* ROM status/error values */
+#define HDA_DSP_ROM_STS_MASK 0xf
+#define HDA_DSP_ROM_INIT 0x1
+#define HDA_DSP_ROM_FW_MANIFEST_LOADED 0x3
+#define HDA_DSP_ROM_FW_FW_LOADED 0x4
+#define HDA_DSP_ROM_FW_ENTERED 0x5
+#define HDA_DSP_ROM_RFW_START 0xf
+#define HDA_DSP_ROM_CSE_ERROR 40
+#define HDA_DSP_ROM_CSE_WRONG_RESPONSE 41
+#define HDA_DSP_ROM_IMR_TO_SMALL 42
+#define HDA_DSP_ROM_BASE_FW_NOT_FOUND 43
+#define HDA_DSP_ROM_CSE_VALIDATION_FAILED 44
+#define HDA_DSP_ROM_IPC_FATAL_ERROR 45
+#define HDA_DSP_ROM_L2_CACHE_ERROR 46
+#define HDA_DSP_ROM_LOAD_OFFSET_TO_SMALL 47
+#define HDA_DSP_ROM_API_PTR_INVALID 50
+#define HDA_DSP_ROM_BASEFW_INCOMPAT 51
+#define HDA_DSP_ROM_UNHANDLED_INTERRUPT 0xBEE00000
+#define HDA_DSP_ROM_MEMORY_HOLE_ECC 0xECC00000
+#define HDA_DSP_ROM_KERNEL_EXCEPTION 0xCAFE0000
+#define HDA_DSP_ROM_USER_EXCEPTION 0xBEEF0000
+#define HDA_DSP_ROM_UNEXPECTED_RESET 0xDECAF000
+#define HDA_DSP_ROM_NULL_FW_ENTRY 0x4c4c4e55
+#define HDA_DSP_IPC_PURGE_FW 0x01004000
+
+/* various timeout values */
+#define HDA_DSP_PU_TIMEOUT 50
+#define HDA_DSP_PD_TIMEOUT 50
+#define HDA_DSP_RESET_TIMEOUT_US 50000
+#define HDA_DSP_BASEFW_TIMEOUT_US 3000000
+#define HDA_DSP_INIT_TIMEOUT_US 500000
+#define HDA_DSP_CTRL_RESET_TIMEOUT 100
+#define HDA_DSP_WAIT_TIMEOUT 500 /* 500 msec */
+#define HDA_DSP_REG_POLL_INTERVAL_US 500 /* 0.5 msec */
+
+#define HDA_DSP_ADSPIC_IPC 1
+#define HDA_DSP_ADSPIS_IPC 1
+
+/* Intel HD Audio General DSP Registers */
+#define HDA_DSP_GEN_BASE 0x0
+#define HDA_DSP_REG_ADSPCS (HDA_DSP_GEN_BASE + 0x04)
+#define HDA_DSP_REG_ADSPIC (HDA_DSP_GEN_BASE + 0x08)
+#define HDA_DSP_REG_ADSPIS (HDA_DSP_GEN_BASE + 0x0C)
+#define HDA_DSP_REG_ADSPIC2 (HDA_DSP_GEN_BASE + 0x10)
+#define HDA_DSP_REG_ADSPIS2 (HDA_DSP_GEN_BASE + 0x14)
+
+/* Intel HD Audio Inter-Processor Communication Registers */
+#define HDA_DSP_IPC_BASE 0x40
+#define HDA_DSP_REG_HIPCT (HDA_DSP_IPC_BASE + 0x00)
+#define HDA_DSP_REG_HIPCTE (HDA_DSP_IPC_BASE + 0x04)
+#define HDA_DSP_REG_HIPCI (HDA_DSP_IPC_BASE + 0x08)
+#define HDA_DSP_REG_HIPCIE (HDA_DSP_IPC_BASE + 0x0C)
+#define HDA_DSP_REG_HIPCCTL (HDA_DSP_IPC_BASE + 0x10)
+
+/* HIPCI */
+#define HDA_DSP_REG_HIPCI_BUSY BIT(31)
+#define HDA_DSP_REG_HIPCI_MSG_MASK 0x7FFFFFFF
+
+/* HIPCIE */
+#define HDA_DSP_REG_HIPCIE_DONE BIT(30)
+#define HDA_DSP_REG_HIPCIE_MSG_MASK 0x3FFFFFFF
+
+/* HIPCCTL */
+#define HDA_DSP_REG_HIPCCTL_DONE BIT(1)
+#define HDA_DSP_REG_HIPCCTL_BUSY BIT(0)
+
+/* HIPCT */
+#define HDA_DSP_REG_HIPCT_BUSY BIT(31)
+#define HDA_DSP_REG_HIPCT_MSG_MASK 0x7FFFFFFF
+
+/* HIPCTE */
+#define HDA_DSP_REG_HIPCTE_MSG_MASK 0x3FFFFFFF
+
+#define HDA_DSP_ADSPIC_CL_DMA 0x2
+#define HDA_DSP_ADSPIS_CL_DMA 0x2
+
+/* Delay before scheduling D0i3 entry */
+#define BXT_D0I3_DELAY 5000
+
+#define FW_CL_STREAM_NUMBER 0x1
+
+/* ADSPCS - Audio DSP Control & Status */
+
+/*
+ * Core Reset - asserted high
+ * CRST Mask for a given core mask pattern, cm
+ */
+#define HDA_DSP_ADSPCS_CRST_SHIFT 0
+#define HDA_DSP_ADSPCS_CRST_MASK(cm) ((cm) << HDA_DSP_ADSPCS_CRST_SHIFT)
+
+/*
+ * Core run/stall - when set to '1' core is stalled
+ * CSTALL Mask for a given core mask pattern, cm
+ */
+#define HDA_DSP_ADSPCS_CSTALL_SHIFT 8
+#define HDA_DSP_ADSPCS_CSTALL_MASK(cm) ((cm) << HDA_DSP_ADSPCS_CSTALL_SHIFT)
+
+/*
+ * Set Power Active - when set to '1' turn cores on
+ * SPA Mask for a given core mask pattern, cm
+ */
+#define HDA_DSP_ADSPCS_SPA_SHIFT 16
+#define HDA_DSP_ADSPCS_SPA_MASK(cm) ((cm) << HDA_DSP_ADSPCS_SPA_SHIFT)
+
+/*
+ * Current Power Active - power status of cores, set by hardware
+ * CPA Mask for a given core mask pattern, cm
+ */
+#define HDA_DSP_ADSPCS_CPA_SHIFT 24
+#define HDA_DSP_ADSPCS_CPA_MASK(cm) ((cm) << HDA_DSP_ADSPCS_CPA_SHIFT)
+
+/* Mask for a given core index, c = 0.. number of supported cores - 1 */
+#define HDA_DSP_CORE_MASK(c) BIT(c)
+
+/*
+ * Mask for a given number of cores
+ * nc = number of supported cores
+ */
+#define SOF_DSP_CORES_MASK(nc) GENMASK(((nc) - 1), 0)
+
+/* Intel HD Audio Inter-Processor Communication Registers for Cannonlake*/
+#define CNL_DSP_IPC_BASE 0xc0
+#define CNL_DSP_REG_HIPCTDR (CNL_DSP_IPC_BASE + 0x00)
+#define CNL_DSP_REG_HIPCTDA (CNL_DSP_IPC_BASE + 0x04)
+#define CNL_DSP_REG_HIPCTDD (CNL_DSP_IPC_BASE + 0x08)
+#define CNL_DSP_REG_HIPCIDR (CNL_DSP_IPC_BASE + 0x10)
+#define CNL_DSP_REG_HIPCIDA (CNL_DSP_IPC_BASE + 0x14)
+#define CNL_DSP_REG_HIPCCTL (CNL_DSP_IPC_BASE + 0x28)
+
+/* HIPCI */
+#define CNL_DSP_REG_HIPCIDR_BUSY BIT(31)
+#define CNL_DSP_REG_HIPCIDR_MSG_MASK 0x7FFFFFFF
+
+/* HIPCIE */
+#define CNL_DSP_REG_HIPCIDA_DONE BIT(31)
+#define CNL_DSP_REG_HIPCIDA_MSG_MASK 0x7FFFFFFF
+
+/* HIPCCTL */
+#define CNL_DSP_REG_HIPCCTL_DONE BIT(1)
+#define CNL_DSP_REG_HIPCCTL_BUSY BIT(0)
+
+/* HIPCT */
+#define CNL_DSP_REG_HIPCTDR_BUSY BIT(31)
+#define CNL_DSP_REG_HIPCTDR_MSG_MASK 0x7FFFFFFF
+
+/* HIPCTDA */
+#define CNL_DSP_REG_HIPCTDA_DONE BIT(31)
+#define CNL_DSP_REG_HIPCTDA_MSG_MASK 0x7FFFFFFF
+
+/* HIPCTDD */
+#define CNL_DSP_REG_HIPCTDD_MSG_MASK 0x7FFFFFFF
+
+/* BDL */
+#define HDA_DSP_BDL_SIZE 4096
+#define HDA_DSP_MAX_BDL_ENTRIES \
+ (HDA_DSP_BDL_SIZE / sizeof(struct sof_intel_dsp_bdl))
+
+/* Number of DAIs */
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+#define SOF_SKL_NUM_DAIS 14
+#else
+#define SOF_SKL_NUM_DAIS 8
+#endif
+
+/* Intel HD Audio SRAM Window 0*/
+#define HDA_ADSP_SRAM0_BASE_SKL 0x8000
+
+/* Firmware status window */
+#define HDA_ADSP_FW_STATUS_SKL HDA_ADSP_SRAM0_BASE_SKL
+#define HDA_ADSP_ERROR_CODE_SKL (HDA_ADSP_FW_STATUS_SKL + 0x4)
+
+/* Host Device Memory Space */
+#define APL_SSP_BASE_OFFSET 0x2000
+#define CNL_SSP_BASE_OFFSET 0x10000
+
+/* Host Device Memory Size of a Single SSP */
+#define SSP_DEV_MEM_SIZE 0x1000
+
+/* SSP Count of the Platform */
+#define APL_SSP_COUNT 6
+#define CNL_SSP_COUNT 3
+
+/* SSP Registers */
+#define SSP_SSC1_OFFSET 0x4
+#define SSP_SET_SCLK_SLAVE BIT(25)
+#define SSP_SET_SFRM_SLAVE BIT(24)
+#define SSP_SET_SLAVE (SSP_SET_SCLK_SLAVE | SSP_SET_SFRM_SLAVE)
+
+#define HDA_IDISP_CODEC(x) ((x) & BIT(2))
+
+struct sof_intel_dsp_bdl {
+ __le32 addr_l;
+ __le32 addr_h;
+ __le32 size;
+ __le32 ioc;
+} __attribute((packed));
+
+#define SOF_HDA_PLAYBACK_STREAMS 16
+#define SOF_HDA_CAPTURE_STREAMS 16
+#define SOF_HDA_PLAYBACK 0
+#define SOF_HDA_CAPTURE 1
+
+/* represents DSP HDA controller frontend - i.e. host facing control */
+struct sof_intel_hda_dev {
+
+ struct hda_bus hbus;
+
+ /* hw config */
+ const struct sof_intel_dsp_desc *desc;
+
+ /* trace */
+ struct hdac_ext_stream *dtrace_stream;
+
+ /* if position update IPC needed */
+ u32 no_ipc_position;
+
+ /* the maximum number of streams (playback + capture) supported */
+ u32 stream_max;
+
+ int irq;
+
+ /* DMIC device */
+ struct platform_device *dmic_dev;
+};
+
+static inline struct hdac_bus *sof_to_bus(struct snd_sof_dev *s)
+{
+ struct sof_intel_hda_dev *hda = s->pdata->hw_pdata;
+
+ return &hda->hbus.core;
+}
+
+static inline struct hda_bus *sof_to_hbus(struct snd_sof_dev *s)
+{
+ struct sof_intel_hda_dev *hda = s->pdata->hw_pdata;
+
+ return &hda->hbus;
+}
+
+struct sof_intel_hda_stream {
+ struct hdac_ext_stream hda_stream;
+ struct sof_intel_stream stream;
+ int hw_params_upon_resume; /* set up hw_params upon resume */
+};
+
+#define bus_to_sof_hda(bus) \
+ container_of(bus, struct sof_intel_hda_dev, hbus.core)
+
+#define SOF_STREAM_SD_OFFSET(s) \
+ (SOF_HDA_ADSP_SD_ENTRY_SIZE * ((s)->index) \
+ + SOF_HDA_ADSP_LOADER_BASE)
+
+/*
+ * DSP Core services.
+ */
+int hda_dsp_probe(struct snd_sof_dev *sdev);
+int hda_dsp_remove(struct snd_sof_dev *sdev);
+int hda_dsp_core_reset_enter(struct snd_sof_dev *sdev,
+ unsigned int core_mask);
+int hda_dsp_core_reset_leave(struct snd_sof_dev *sdev,
+ unsigned int core_mask);
+int hda_dsp_core_stall_reset(struct snd_sof_dev *sdev, unsigned int core_mask);
+int hda_dsp_core_run(struct snd_sof_dev *sdev, unsigned int core_mask);
+int hda_dsp_core_power_up(struct snd_sof_dev *sdev, unsigned int core_mask);
+int hda_dsp_enable_core(struct snd_sof_dev *sdev, unsigned int core_mask);
+int hda_dsp_core_power_down(struct snd_sof_dev *sdev, unsigned int core_mask);
+bool hda_dsp_core_is_enabled(struct snd_sof_dev *sdev,
+ unsigned int core_mask);
+int hda_dsp_core_reset_power_down(struct snd_sof_dev *sdev,
+ unsigned int core_mask);
+void hda_dsp_ipc_int_enable(struct snd_sof_dev *sdev);
+void hda_dsp_ipc_int_disable(struct snd_sof_dev *sdev);
+
+int hda_dsp_suspend(struct snd_sof_dev *sdev, int state);
+int hda_dsp_resume(struct snd_sof_dev *sdev);
+int hda_dsp_runtime_suspend(struct snd_sof_dev *sdev, int state);
+int hda_dsp_runtime_resume(struct snd_sof_dev *sdev);
+void hda_dsp_set_hw_params_upon_resume(struct snd_sof_dev *sdev);
+void hda_dsp_dump_skl(struct snd_sof_dev *sdev, u32 flags);
+void hda_dsp_dump(struct snd_sof_dev *sdev, u32 flags);
+void hda_ipc_dump(struct snd_sof_dev *sdev);
+
+/*
+ * DSP PCM Operations.
+ */
+int hda_dsp_pcm_open(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream);
+int hda_dsp_pcm_close(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream);
+int hda_dsp_pcm_hw_params(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params,
+ struct sof_ipc_stream_params *ipc_params);
+int hda_dsp_pcm_trigger(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream, int cmd);
+snd_pcm_uframes_t hda_dsp_pcm_pointer(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream);
+
+/*
+ * DSP Stream Operations.
+ */
+
+int hda_dsp_stream_init(struct snd_sof_dev *sdev);
+void hda_dsp_stream_free(struct snd_sof_dev *sdev);
+int hda_dsp_stream_hw_params(struct snd_sof_dev *sdev,
+ struct hdac_ext_stream *stream,
+ struct snd_dma_buffer *dmab,
+ struct snd_pcm_hw_params *params);
+int hda_dsp_stream_trigger(struct snd_sof_dev *sdev,
+ struct hdac_ext_stream *stream, int cmd);
+irqreturn_t hda_dsp_stream_interrupt(int irq, void *context);
+irqreturn_t hda_dsp_stream_threaded_handler(int irq, void *context);
+int hda_dsp_stream_setup_bdl(struct snd_sof_dev *sdev,
+ struct snd_dma_buffer *dmab,
+ struct hdac_stream *stream);
+
+struct hdac_ext_stream *
+ hda_dsp_stream_get(struct snd_sof_dev *sdev, int direction);
+int hda_dsp_stream_put(struct snd_sof_dev *sdev, int direction, int stream_tag);
+int hda_dsp_stream_spib_config(struct snd_sof_dev *sdev,
+ struct hdac_ext_stream *stream,
+ int enable, u32 size);
+
+void hda_ipc_msg_data(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ void *p, size_t sz);
+int hda_ipc_pcm_params(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ const struct sof_ipc_pcm_params_reply *reply);
+
+/*
+ * DSP IPC Operations.
+ */
+int hda_dsp_ipc_send_msg(struct snd_sof_dev *sdev,
+ struct snd_sof_ipc_msg *msg);
+void hda_dsp_ipc_get_reply(struct snd_sof_dev *sdev);
+int hda_dsp_ipc_fw_ready(struct snd_sof_dev *sdev, u32 msg_id);
+irqreturn_t hda_dsp_ipc_irq_handler(int irq, void *context);
+irqreturn_t hda_dsp_ipc_irq_thread(int irq, void *context);
+int hda_dsp_ipc_cmd_done(struct snd_sof_dev *sdev, int dir);
+
+/*
+ * DSP Code loader.
+ */
+int hda_dsp_cl_boot_firmware(struct snd_sof_dev *sdev);
+int hda_dsp_cl_boot_firmware_skl(struct snd_sof_dev *sdev);
+
+/* pre and post fw run ops */
+int hda_dsp_pre_fw_run(struct snd_sof_dev *sdev);
+int hda_dsp_post_fw_run(struct snd_sof_dev *sdev);
+
+/*
+ * HDA Controller Operations.
+ */
+int hda_dsp_ctrl_get_caps(struct snd_sof_dev *sdev);
+void hda_dsp_ctrl_ppcap_enable(struct snd_sof_dev *sdev, bool enable);
+void hda_dsp_ctrl_ppcap_int_enable(struct snd_sof_dev *sdev, bool enable);
+int hda_dsp_ctrl_link_reset(struct snd_sof_dev *sdev, bool reset);
+void hda_dsp_ctrl_misc_clock_gating(struct snd_sof_dev *sdev, bool enable);
+int hda_dsp_ctrl_clock_power_gating(struct snd_sof_dev *sdev, bool enable);
+int hda_dsp_ctrl_init_chip(struct snd_sof_dev *sdev, bool full_reset);
+
+/*
+ * HDA bus operations.
+ */
+void sof_hda_bus_init(struct hdac_bus *bus, struct device *dev,
+ const struct hdac_ext_bus_ops *ext_ops);
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+/*
+ * HDA Codec operations.
+ */
+int hda_codec_probe_bus(struct snd_sof_dev *sdev);
+
+#endif /* CONFIG_SND_SOC_SOF_HDA */
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA) && IS_ENABLED(CONFIG_SND_SOC_HDAC_HDMI)
+
+void hda_codec_i915_get(struct snd_sof_dev *sdev);
+void hda_codec_i915_put(struct snd_sof_dev *sdev);
+int hda_codec_i915_init(struct snd_sof_dev *sdev);
+int hda_codec_i915_exit(struct snd_sof_dev *sdev);
+
+#else
+
+static inline void hda_codec_i915_get(struct snd_sof_dev *sdev) { }
+static inline void hda_codec_i915_put(struct snd_sof_dev *sdev) { }
+static inline int hda_codec_i915_init(struct snd_sof_dev *sdev) { return 0; }
+static inline int hda_codec_i915_exit(struct snd_sof_dev *sdev) { return 0; }
+
+#endif /* CONFIG_SND_SOC_SOF_HDA && CONFIG_SND_SOC_HDAC_HDMI */
+
+/*
+ * Trace Control.
+ */
+int hda_dsp_trace_init(struct snd_sof_dev *sdev, u32 *stream_tag);
+int hda_dsp_trace_release(struct snd_sof_dev *sdev);
+int hda_dsp_trace_trigger(struct snd_sof_dev *sdev, int cmd);
+
+/* common dai driver */
+extern struct snd_soc_dai_driver skl_dai[];
+
+/*
+ * Platform Specific HW abstraction Ops.
+ */
+extern const struct snd_sof_dsp_ops sof_apl_ops;
+extern const struct snd_sof_dsp_ops sof_cnl_ops;
+extern const struct snd_sof_dsp_ops sof_skl_ops;
+
+extern const struct sof_intel_dsp_desc apl_chip_info;
+extern const struct sof_intel_dsp_desc cnl_chip_info;
+extern const struct sof_intel_dsp_desc skl_chip_info;
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2019 Intel Corporation. All rights reserved.
+//
+// Authors: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
+
+/* Intel-specific SOF IPC code */
+
+#include <linux/device.h>
+#include <linux/export.h>
+#include <linux/module.h>
+#include <linux/types.h>
+
+#include <sound/pcm.h>
+#include <sound/sof/stream.h>
+
+#include "../ops.h"
+#include "../sof-priv.h"
+
+struct intel_stream {
+ size_t posn_offset;
+};
+
+/* Mailbox-based Intel IPC implementation */
+void intel_ipc_msg_data(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ void *p, size_t sz)
+{
+ if (!substream || !sdev->stream_box.size) {
+ sof_mailbox_read(sdev, sdev->dsp_box.offset, p, sz);
+ } else {
+ struct intel_stream *stream = substream->runtime->private_data;
+
+ /* The stream might already be closed */
+ if (stream)
+ sof_mailbox_read(sdev, stream->posn_offset, p, sz);
+ }
+}
+EXPORT_SYMBOL(intel_ipc_msg_data);
+
+int intel_ipc_pcm_params(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ const struct sof_ipc_pcm_params_reply *reply)
+{
+ struct intel_stream *stream = substream->runtime->private_data;
+ size_t posn_offset = reply->posn_offset;
+
+ /* check if offset is overflow or it is not aligned */
+ if (posn_offset > sdev->stream_box.size ||
+ posn_offset % sizeof(struct sof_ipc_stream_posn) != 0)
+ return -EINVAL;
+
+ stream->posn_offset = sdev->stream_box.offset + posn_offset;
+
+ dev_dbg(sdev->dev, "pcm: stream dir %d, posn mailbox offset is %zu",
+ substream->stream, stream->posn_offset);
+
+ return 0;
+}
+EXPORT_SYMBOL(intel_ipc_pcm_params);
+
+int intel_pcm_open(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream)
+{
+ struct intel_stream *stream = kmalloc(sizeof(*stream), GFP_KERNEL);
+
+ if (!stream)
+ return -ENOMEM;
+
+ /* binding pcm substream to hda stream */
+ substream->runtime->private_data = stream;
+
+ return 0;
+}
+EXPORT_SYMBOL(intel_pcm_open);
+
+int intel_pcm_close(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream)
+{
+ struct intel_stream *stream = substream->runtime->private_data;
+
+ substream->runtime->private_data = NULL;
+ kfree(stream);
+
+ return 0;
+}
+EXPORT_SYMBOL(intel_pcm_close);
+
+MODULE_LICENSE("Dual BSD/GPL");
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ *
+ * Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+ */
+
+#ifndef __SOF_INTEL_SHIM_H
+#define __SOF_INTEL_SHIM_H
+
+/*
+ * SHIM registers for BYT, BSW, CHT, HSW, BDW
+ */
+
+#define SHIM_CSR (SHIM_OFFSET + 0x00)
+#define SHIM_PISR (SHIM_OFFSET + 0x08)
+#define SHIM_PIMR (SHIM_OFFSET + 0x10)
+#define SHIM_ISRX (SHIM_OFFSET + 0x18)
+#define SHIM_ISRD (SHIM_OFFSET + 0x20)
+#define SHIM_IMRX (SHIM_OFFSET + 0x28)
+#define SHIM_IMRD (SHIM_OFFSET + 0x30)
+#define SHIM_IPCX (SHIM_OFFSET + 0x38)
+#define SHIM_IPCD (SHIM_OFFSET + 0x40)
+#define SHIM_ISRSC (SHIM_OFFSET + 0x48)
+#define SHIM_ISRLPESC (SHIM_OFFSET + 0x50)
+#define SHIM_IMRSC (SHIM_OFFSET + 0x58)
+#define SHIM_IMRLPESC (SHIM_OFFSET + 0x60)
+#define SHIM_IPCSC (SHIM_OFFSET + 0x68)
+#define SHIM_IPCLPESC (SHIM_OFFSET + 0x70)
+#define SHIM_CLKCTL (SHIM_OFFSET + 0x78)
+#define SHIM_CSR2 (SHIM_OFFSET + 0x80)
+#define SHIM_LTRC (SHIM_OFFSET + 0xE0)
+#define SHIM_HMDC (SHIM_OFFSET + 0xE8)
+
+#define SHIM_PWMCTRL 0x1000
+
+/*
+ * SST SHIM register bits for BYT, BSW, CHT HSW, BDW
+ * Register bit naming and functionaility can differ between devices.
+ */
+
+/* CSR / CS */
+#define SHIM_CSR_RST BIT(1)
+#define SHIM_CSR_SBCS0 BIT(2)
+#define SHIM_CSR_SBCS1 BIT(3)
+#define SHIM_CSR_DCS(x) ((x) << 4)
+#define SHIM_CSR_DCS_MASK (0x7 << 4)
+#define SHIM_CSR_STALL BIT(10)
+#define SHIM_CSR_S0IOCS BIT(21)
+#define SHIM_CSR_S1IOCS BIT(23)
+#define SHIM_CSR_LPCS BIT(31)
+#define SHIM_CSR_24MHZ_LPCS \
+ (SHIM_CSR_SBCS0 | SHIM_CSR_SBCS1 | SHIM_CSR_LPCS)
+#define SHIM_CSR_24MHZ_NO_LPCS (SHIM_CSR_SBCS0 | SHIM_CSR_SBCS1)
+#define SHIM_BYT_CSR_RST BIT(0)
+#define SHIM_BYT_CSR_VECTOR_SEL BIT(1)
+#define SHIM_BYT_CSR_STALL BIT(2)
+#define SHIM_BYT_CSR_PWAITMODE BIT(3)
+
+/* ISRX / ISC */
+#define SHIM_ISRX_BUSY BIT(1)
+#define SHIM_ISRX_DONE BIT(0)
+#define SHIM_BYT_ISRX_REQUEST BIT(1)
+
+/* ISRD / ISD */
+#define SHIM_ISRD_BUSY BIT(1)
+#define SHIM_ISRD_DONE BIT(0)
+
+/* IMRX / IMC */
+#define SHIM_IMRX_BUSY BIT(1)
+#define SHIM_IMRX_DONE BIT(0)
+#define SHIM_BYT_IMRX_REQUEST BIT(1)
+
+/* IMRD / IMD */
+#define SHIM_IMRD_DONE BIT(0)
+#define SHIM_IMRD_BUSY BIT(1)
+#define SHIM_IMRD_SSP0 BIT(16)
+#define SHIM_IMRD_DMAC0 BIT(21)
+#define SHIM_IMRD_DMAC1 BIT(22)
+#define SHIM_IMRD_DMAC (SHIM_IMRD_DMAC0 | SHIM_IMRD_DMAC1)
+
+/* IPCX / IPCC */
+#define SHIM_IPCX_DONE BIT(30)
+#define SHIM_IPCX_BUSY BIT(31)
+#define SHIM_BYT_IPCX_DONE BIT_ULL(62)
+#define SHIM_BYT_IPCX_BUSY BIT_ULL(63)
+
+/* IPCD */
+#define SHIM_IPCD_DONE BIT(30)
+#define SHIM_IPCD_BUSY BIT(31)
+#define SHIM_BYT_IPCD_DONE BIT_ULL(62)
+#define SHIM_BYT_IPCD_BUSY BIT_ULL(63)
+
+/* CLKCTL */
+#define SHIM_CLKCTL_SMOS(x) ((x) << 24)
+#define SHIM_CLKCTL_MASK (3 << 24)
+#define SHIM_CLKCTL_DCPLCG BIT(18)
+#define SHIM_CLKCTL_SCOE1 BIT(17)
+#define SHIM_CLKCTL_SCOE0 BIT(16)
+
+/* CSR2 / CS2 */
+#define SHIM_CSR2_SDFD_SSP0 BIT(1)
+#define SHIM_CSR2_SDFD_SSP1 BIT(2)
+
+/* LTRC */
+#define SHIM_LTRC_VAL(x) ((x) << 0)
+
+/* HMDC */
+#define SHIM_HMDC_HDDA0(x) ((x) << 0)
+#define SHIM_HMDC_HDDA1(x) ((x) << 7)
+#define SHIM_HMDC_HDDA_E0_CH0 1
+#define SHIM_HMDC_HDDA_E0_CH1 2
+#define SHIM_HMDC_HDDA_E0_CH2 4
+#define SHIM_HMDC_HDDA_E0_CH3 8
+#define SHIM_HMDC_HDDA_E1_CH0 SHIM_HMDC_HDDA1(SHIM_HMDC_HDDA_E0_CH0)
+#define SHIM_HMDC_HDDA_E1_CH1 SHIM_HMDC_HDDA1(SHIM_HMDC_HDDA_E0_CH1)
+#define SHIM_HMDC_HDDA_E1_CH2 SHIM_HMDC_HDDA1(SHIM_HMDC_HDDA_E0_CH2)
+#define SHIM_HMDC_HDDA_E1_CH3 SHIM_HMDC_HDDA1(SHIM_HMDC_HDDA_E0_CH3)
+#define SHIM_HMDC_HDDA_E0_ALLCH \
+ (SHIM_HMDC_HDDA_E0_CH0 | SHIM_HMDC_HDDA_E0_CH1 | \
+ SHIM_HMDC_HDDA_E0_CH2 | SHIM_HMDC_HDDA_E0_CH3)
+#define SHIM_HMDC_HDDA_E1_ALLCH \
+ (SHIM_HMDC_HDDA_E1_CH0 | SHIM_HMDC_HDDA_E1_CH1 | \
+ SHIM_HMDC_HDDA_E1_CH2 | SHIM_HMDC_HDDA_E1_CH3)
+
+/* Audio DSP PCI registers */
+#define PCI_VDRTCTL0 0xa0
+#define PCI_VDRTCTL1 0xa4
+#define PCI_VDRTCTL2 0xa8
+#define PCI_VDRTCTL3 0xaC
+
+/* VDRTCTL0 */
+#define PCI_VDRTCL0_D3PGD BIT(0)
+#define PCI_VDRTCL0_D3SRAMPGD BIT(1)
+#define PCI_VDRTCL0_DSRAMPGE_SHIFT 12
+#define PCI_VDRTCL0_DSRAMPGE_MASK GENMASK(PCI_VDRTCL0_DSRAMPGE_SHIFT + 19,\
+ PCI_VDRTCL0_DSRAMPGE_SHIFT)
+#define PCI_VDRTCL0_ISRAMPGE_SHIFT 2
+#define PCI_VDRTCL0_ISRAMPGE_MASK GENMASK(PCI_VDRTCL0_ISRAMPGE_SHIFT + 9,\
+ PCI_VDRTCL0_ISRAMPGE_SHIFT)
+
+/* VDRTCTL2 */
+#define PCI_VDRTCL2_DCLCGE BIT(1)
+#define PCI_VDRTCL2_DTCGE BIT(10)
+#define PCI_VDRTCL2_APLLSE_MASK BIT(31)
+
+/* PMCS */
+#define PCI_PMCS 0x84
+#define PCI_PMCS_PS_MASK 0x3
+
+/* DSP hardware descriptor */
+struct sof_intel_dsp_desc {
+ int cores_num;
+ int cores_mask;
+ int init_core_mask; /* cores available after fw boot */
+ int ipc_req;
+ int ipc_req_mask;
+ int ipc_ack;
+ int ipc_ack_mask;
+ int ipc_ctl;
+ int rom_init_timeout;
+ int ssp_count; /* ssp count of the platform */
+ int ssp_base_offset; /* base address of the SSPs */
+};
+
+extern const struct snd_sof_dsp_ops sof_tng_ops;
+extern const struct snd_sof_dsp_ops sof_byt_ops;
+extern const struct snd_sof_dsp_ops sof_cht_ops;
+extern const struct snd_sof_dsp_ops sof_hsw_ops;
+extern const struct snd_sof_dsp_ops sof_bdw_ops;
+
+extern const struct sof_intel_dsp_desc byt_chip_info;
+extern const struct sof_intel_dsp_desc cht_chip_info;
+extern const struct sof_intel_dsp_desc bdw_chip_info;
+extern const struct sof_intel_dsp_desc hsw_chip_info;
+extern const struct sof_intel_dsp_desc tng_chip_info;
+
+struct sof_intel_stream {
+ size_t posn_offset;
+};
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+// Generic IPC layer that can work over MMIO and SPI/I2C. PHY layer provided
+// by platform driver code.
+//
+
+#include <linux/mutex.h>
+#include <linux/types.h>
+
+#include "sof-priv.h"
+#include "ops.h"
+
+/*
+ * IPC message default size and timeout (ms).
+ * TODO: allow platforms to set size and timeout.
+ */
+#define IPC_TIMEOUT_MS 300
+
+static void ipc_trace_message(struct snd_sof_dev *sdev, u32 msg_id);
+static void ipc_stream_message(struct snd_sof_dev *sdev, u32 msg_cmd);
+
+/*
+ * IPC message Tx/Rx message handling.
+ */
+
+/* SOF generic IPC data */
+struct snd_sof_ipc {
+ struct snd_sof_dev *sdev;
+
+ /* protects messages and the disable flag */
+ struct mutex tx_mutex;
+ /* disables further sending of ipc's */
+ bool disable_ipc_tx;
+
+ struct snd_sof_ipc_msg msg;
+};
+
+struct sof_ipc_ctrl_data_params {
+ size_t msg_bytes;
+ size_t hdr_bytes;
+ size_t pl_size;
+ size_t elems;
+ u32 num_msg;
+ u8 *src;
+ u8 *dst;
+};
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_DEBUG_VERBOSE_IPC)
+static void ipc_log_header(struct device *dev, u8 *text, u32 cmd)
+{
+ u8 *str;
+ u8 *str2 = NULL;
+ u32 glb;
+ u32 type;
+
+ glb = cmd & SOF_GLB_TYPE_MASK;
+ type = cmd & SOF_CMD_TYPE_MASK;
+
+ switch (glb) {
+ case SOF_IPC_GLB_REPLY:
+ str = "GLB_REPLY"; break;
+ case SOF_IPC_GLB_COMPOUND:
+ str = "GLB_COMPOUND"; break;
+ case SOF_IPC_GLB_TPLG_MSG:
+ str = "GLB_TPLG_MSG";
+ switch (type) {
+ case SOF_IPC_TPLG_COMP_NEW:
+ str2 = "COMP_NEW"; break;
+ case SOF_IPC_TPLG_COMP_FREE:
+ str2 = "COMP_FREE"; break;
+ case SOF_IPC_TPLG_COMP_CONNECT:
+ str2 = "COMP_CONNECT"; break;
+ case SOF_IPC_TPLG_PIPE_NEW:
+ str2 = "PIPE_NEW"; break;
+ case SOF_IPC_TPLG_PIPE_FREE:
+ str2 = "PIPE_FREE"; break;
+ case SOF_IPC_TPLG_PIPE_CONNECT:
+ str2 = "PIPE_CONNECT"; break;
+ case SOF_IPC_TPLG_PIPE_COMPLETE:
+ str2 = "PIPE_COMPLETE"; break;
+ case SOF_IPC_TPLG_BUFFER_NEW:
+ str2 = "BUFFER_NEW"; break;
+ case SOF_IPC_TPLG_BUFFER_FREE:
+ str2 = "BUFFER_FREE"; break;
+ default:
+ str2 = "unknown type"; break;
+ }
+ break;
+ case SOF_IPC_GLB_PM_MSG:
+ str = "GLB_PM_MSG";
+ switch (type) {
+ case SOF_IPC_PM_CTX_SAVE:
+ str2 = "CTX_SAVE"; break;
+ case SOF_IPC_PM_CTX_RESTORE:
+ str2 = "CTX_RESTORE"; break;
+ case SOF_IPC_PM_CTX_SIZE:
+ str2 = "CTX_SIZE"; break;
+ case SOF_IPC_PM_CLK_SET:
+ str2 = "CLK_SET"; break;
+ case SOF_IPC_PM_CLK_GET:
+ str2 = "CLK_GET"; break;
+ case SOF_IPC_PM_CLK_REQ:
+ str2 = "CLK_REQ"; break;
+ case SOF_IPC_PM_CORE_ENABLE:
+ str2 = "CORE_ENABLE"; break;
+ default:
+ str2 = "unknown type"; break;
+ }
+ break;
+ case SOF_IPC_GLB_COMP_MSG:
+ str = "GLB_COMP_MSG: SET_VALUE";
+ switch (type) {
+ case SOF_IPC_COMP_SET_VALUE:
+ str2 = "SET_VALUE"; break;
+ case SOF_IPC_COMP_GET_VALUE:
+ str2 = "GET_VALUE"; break;
+ case SOF_IPC_COMP_SET_DATA:
+ str2 = "SET_DATA"; break;
+ case SOF_IPC_COMP_GET_DATA:
+ str2 = "GET_DATA"; break;
+ default:
+ str2 = "unknown type"; break;
+ }
+ break;
+ case SOF_IPC_GLB_STREAM_MSG:
+ str = "GLB_STREAM_MSG";
+ switch (type) {
+ case SOF_IPC_STREAM_PCM_PARAMS:
+ str2 = "PCM_PARAMS"; break;
+ case SOF_IPC_STREAM_PCM_PARAMS_REPLY:
+ str2 = "PCM_REPLY"; break;
+ case SOF_IPC_STREAM_PCM_FREE:
+ str2 = "PCM_FREE"; break;
+ case SOF_IPC_STREAM_TRIG_START:
+ str2 = "TRIG_START"; break;
+ case SOF_IPC_STREAM_TRIG_STOP:
+ str2 = "TRIG_STOP"; break;
+ case SOF_IPC_STREAM_TRIG_PAUSE:
+ str2 = "TRIG_PAUSE"; break;
+ case SOF_IPC_STREAM_TRIG_RELEASE:
+ str2 = "TRIG_RELEASE"; break;
+ case SOF_IPC_STREAM_TRIG_DRAIN:
+ str2 = "TRIG_DRAIN"; break;
+ case SOF_IPC_STREAM_TRIG_XRUN:
+ str2 = "TRIG_XRUN"; break;
+ case SOF_IPC_STREAM_POSITION:
+ str2 = "POSITION"; break;
+ case SOF_IPC_STREAM_VORBIS_PARAMS:
+ str2 = "VORBIS_PARAMS"; break;
+ case SOF_IPC_STREAM_VORBIS_FREE:
+ str2 = "VORBIS_FREE"; break;
+ default:
+ str2 = "unknown type"; break;
+ }
+ break;
+ case SOF_IPC_FW_READY:
+ str = "FW_READY"; break;
+ case SOF_IPC_GLB_DAI_MSG:
+ str = "GLB_DAI_MSG";
+ switch (type) {
+ case SOF_IPC_DAI_CONFIG:
+ str2 = "CONFIG"; break;
+ case SOF_IPC_DAI_LOOPBACK:
+ str2 = "LOOPBACK"; break;
+ default:
+ str2 = "unknown type"; break;
+ }
+ break;
+ case SOF_IPC_GLB_TRACE_MSG:
+ str = "GLB_TRACE_MSG"; break;
+ default:
+ str = "unknown GLB command"; break;
+ }
+
+ if (str2)
+ dev_dbg(dev, "%s: 0x%x: %s: %s\n", text, cmd, str, str2);
+ else
+ dev_dbg(dev, "%s: 0x%x: %s\n", text, cmd, str);
+}
+#else
+static inline void ipc_log_header(struct device *dev, u8 *text, u32 cmd)
+{
+ dev_dbg(dev, "%s: 0x%x\n", text, cmd);
+}
+#endif
+
+/* wait for IPC message reply */
+static int tx_wait_done(struct snd_sof_ipc *ipc, struct snd_sof_ipc_msg *msg,
+ void *reply_data)
+{
+ struct snd_sof_dev *sdev = ipc->sdev;
+ struct sof_ipc_cmd_hdr *hdr = msg->msg_data;
+ int ret;
+
+ /* wait for DSP IPC completion */
+ ret = wait_event_timeout(msg->waitq, msg->ipc_complete,
+ msecs_to_jiffies(IPC_TIMEOUT_MS));
+
+ if (ret == 0) {
+ dev_err(sdev->dev, "error: ipc timed out for 0x%x size %d\n",
+ hdr->cmd, hdr->size);
+ snd_sof_dsp_dbg_dump(ipc->sdev, SOF_DBG_REGS | SOF_DBG_MBOX);
+ snd_sof_ipc_dump(ipc->sdev);
+ snd_sof_trace_notify_for_error(ipc->sdev);
+ ret = -ETIMEDOUT;
+ } else {
+ /* copy the data returned from DSP */
+ ret = msg->reply_error;
+ if (msg->reply_size)
+ memcpy(reply_data, msg->reply_data, msg->reply_size);
+ if (ret < 0)
+ dev_err(sdev->dev, "error: ipc error for 0x%x size %zu\n",
+ hdr->cmd, msg->reply_size);
+ else
+ ipc_log_header(sdev->dev, "ipc tx succeeded", hdr->cmd);
+ }
+
+ return ret;
+}
+
+/* send IPC message from host to DSP */
+static int sof_ipc_tx_message_unlocked(struct snd_sof_ipc *ipc, u32 header,
+ void *msg_data, size_t msg_bytes,
+ void *reply_data, size_t reply_bytes)
+{
+ struct snd_sof_dev *sdev = ipc->sdev;
+ struct snd_sof_ipc_msg *msg;
+ int ret;
+
+ if (ipc->disable_ipc_tx)
+ return -ENODEV;
+
+ /*
+ * The spin-lock is also still needed to protect message objects against
+ * other atomic contexts.
+ */
+ spin_lock_irq(&sdev->ipc_lock);
+
+ /* initialise the message */
+ msg = &ipc->msg;
+
+ msg->header = header;
+ msg->msg_size = msg_bytes;
+ msg->reply_size = reply_bytes;
+ msg->reply_error = 0;
+
+ /* attach any data */
+ if (msg_bytes)
+ memcpy(msg->msg_data, msg_data, msg_bytes);
+
+ sdev->msg = msg;
+
+ ret = snd_sof_dsp_send_msg(sdev, msg);
+ /* Next reply that we receive will be related to this message */
+ if (!ret)
+ msg->ipc_complete = false;
+
+ spin_unlock_irq(&sdev->ipc_lock);
+
+ if (ret < 0) {
+ /* So far IPC TX never fails, consider making the above void */
+ dev_err_ratelimited(sdev->dev,
+ "error: ipc tx failed with error %d\n",
+ ret);
+ return ret;
+ }
+
+ ipc_log_header(sdev->dev, "ipc tx", msg->header);
+
+ /* now wait for completion */
+ if (!ret)
+ ret = tx_wait_done(ipc, msg, reply_data);
+
+ return ret;
+}
+
+/* send IPC message from host to DSP */
+int sof_ipc_tx_message(struct snd_sof_ipc *ipc, u32 header,
+ void *msg_data, size_t msg_bytes, void *reply_data,
+ size_t reply_bytes)
+{
+ int ret;
+
+ if (msg_bytes > SOF_IPC_MSG_MAX_SIZE ||
+ reply_bytes > SOF_IPC_MSG_MAX_SIZE)
+ return -ENOBUFS;
+
+ /* Serialise IPC TX */
+ mutex_lock(&ipc->tx_mutex);
+
+ ret = sof_ipc_tx_message_unlocked(ipc, header, msg_data, msg_bytes,
+ reply_data, reply_bytes);
+
+ mutex_unlock(&ipc->tx_mutex);
+
+ return ret;
+}
+EXPORT_SYMBOL(sof_ipc_tx_message);
+
+/* handle reply message from DSP */
+int snd_sof_ipc_reply(struct snd_sof_dev *sdev, u32 msg_id)
+{
+ struct snd_sof_ipc_msg *msg = &sdev->ipc->msg;
+ unsigned long flags;
+
+ /*
+ * Protect against a theoretical race with sof_ipc_tx_message(): if the
+ * DSP is fast enough to receive an IPC message, reply to it, and the
+ * host interrupt processing calls this function on a different core
+ * from the one, where the sending is taking place, the message might
+ * not yet be marked as expecting a reply.
+ */
+ spin_lock_irqsave(&sdev->ipc_lock, flags);
+
+ if (msg->ipc_complete) {
+ spin_unlock_irqrestore(&sdev->ipc_lock, flags);
+ dev_err(sdev->dev, "error: no reply expected, received 0x%x",
+ msg_id);
+ return -EINVAL;
+ }
+
+ /* wake up and return the error if we have waiters on this message ? */
+ msg->ipc_complete = true;
+ wake_up(&msg->waitq);
+
+ spin_unlock_irqrestore(&sdev->ipc_lock, flags);
+
+ return 0;
+}
+EXPORT_SYMBOL(snd_sof_ipc_reply);
+
+/* DSP firmware has sent host a message */
+void snd_sof_ipc_msgs_rx(struct snd_sof_dev *sdev)
+{
+ struct sof_ipc_cmd_hdr hdr;
+ u32 cmd, type;
+ int err = 0;
+
+ /* read back header */
+ snd_sof_ipc_msg_data(sdev, NULL, &hdr, sizeof(hdr));
+ ipc_log_header(sdev->dev, "ipc rx", hdr.cmd);
+
+ cmd = hdr.cmd & SOF_GLB_TYPE_MASK;
+ type = hdr.cmd & SOF_CMD_TYPE_MASK;
+
+ /* check message type */
+ switch (cmd) {
+ case SOF_IPC_GLB_REPLY:
+ dev_err(sdev->dev, "error: ipc reply unknown\n");
+ break;
+ case SOF_IPC_FW_READY:
+ /* check for FW boot completion */
+ if (!sdev->boot_complete) {
+ err = sof_ops(sdev)->fw_ready(sdev, cmd);
+ if (err < 0) {
+ /*
+ * this indicates a mismatch in ABI
+ * between the driver and fw
+ */
+ dev_err(sdev->dev, "error: ABI mismatch %d\n",
+ err);
+ } else {
+ /* firmware boot completed OK */
+ sdev->boot_complete = true;
+ }
+
+ /* wake up firmware loader */
+ wake_up(&sdev->boot_wait);
+ }
+ break;
+ case SOF_IPC_GLB_COMPOUND:
+ case SOF_IPC_GLB_TPLG_MSG:
+ case SOF_IPC_GLB_PM_MSG:
+ case SOF_IPC_GLB_COMP_MSG:
+ break;
+ case SOF_IPC_GLB_STREAM_MSG:
+ /* need to pass msg id into the function */
+ ipc_stream_message(sdev, hdr.cmd);
+ break;
+ case SOF_IPC_GLB_TRACE_MSG:
+ ipc_trace_message(sdev, type);
+ break;
+ default:
+ dev_err(sdev->dev, "error: unknown DSP message 0x%x\n", cmd);
+ break;
+ }
+
+ ipc_log_header(sdev->dev, "ipc rx done", hdr.cmd);
+}
+EXPORT_SYMBOL(snd_sof_ipc_msgs_rx);
+
+/*
+ * IPC trace mechanism.
+ */
+
+static void ipc_trace_message(struct snd_sof_dev *sdev, u32 msg_id)
+{
+ struct sof_ipc_dma_trace_posn posn;
+
+ switch (msg_id) {
+ case SOF_IPC_TRACE_DMA_POSITION:
+ /* read back full message */
+ snd_sof_ipc_msg_data(sdev, NULL, &posn, sizeof(posn));
+ snd_sof_trace_update_pos(sdev, &posn);
+ break;
+ default:
+ dev_err(sdev->dev, "error: unhandled trace message %x\n",
+ msg_id);
+ break;
+ }
+}
+
+/*
+ * IPC stream position.
+ */
+
+static void ipc_period_elapsed(struct snd_sof_dev *sdev, u32 msg_id)
+{
+ struct snd_sof_pcm_stream *stream;
+ struct sof_ipc_stream_posn posn;
+ struct snd_sof_pcm *spcm;
+ int direction;
+
+ spcm = snd_sof_find_spcm_comp(sdev, msg_id, &direction);
+ if (!spcm) {
+ dev_err(sdev->dev,
+ "error: period elapsed for unknown stream, msg_id %d\n",
+ msg_id);
+ return;
+ }
+
+ stream = &spcm->stream[direction];
+ snd_sof_ipc_msg_data(sdev, stream->substream, &posn, sizeof(posn));
+
+ dev_dbg(sdev->dev, "posn : host 0x%llx dai 0x%llx wall 0x%llx\n",
+ posn.host_posn, posn.dai_posn, posn.wallclock);
+
+ memcpy(&stream->posn, &posn, sizeof(posn));
+
+ /* only inform ALSA for period_wakeup mode */
+ if (!stream->substream->runtime->no_period_wakeup)
+ snd_sof_pcm_period_elapsed(stream->substream);
+}
+
+/* DSP notifies host of an XRUN within FW */
+static void ipc_xrun(struct snd_sof_dev *sdev, u32 msg_id)
+{
+ struct snd_sof_pcm_stream *stream;
+ struct sof_ipc_stream_posn posn;
+ struct snd_sof_pcm *spcm;
+ int direction;
+
+ spcm = snd_sof_find_spcm_comp(sdev, msg_id, &direction);
+ if (!spcm) {
+ dev_err(sdev->dev, "error: XRUN for unknown stream, msg_id %d\n",
+ msg_id);
+ return;
+ }
+
+ stream = &spcm->stream[direction];
+ snd_sof_ipc_msg_data(sdev, stream->substream, &posn, sizeof(posn));
+
+ dev_dbg(sdev->dev, "posn XRUN: host %llx comp %d size %d\n",
+ posn.host_posn, posn.xrun_comp_id, posn.xrun_size);
+
+#if defined(CONFIG_SND_SOC_SOF_DEBUG_XRUN_STOP)
+ /* stop PCM on XRUN - used for pipeline debug */
+ memcpy(&stream->posn, &posn, sizeof(posn));
+ snd_pcm_stop_xrun(stream->substream);
+#endif
+}
+
+/* stream notifications from DSP FW */
+static void ipc_stream_message(struct snd_sof_dev *sdev, u32 msg_cmd)
+{
+ /* get msg cmd type and msd id */
+ u32 msg_type = msg_cmd & SOF_CMD_TYPE_MASK;
+ u32 msg_id = SOF_IPC_MESSAGE_ID(msg_cmd);
+
+ switch (msg_type) {
+ case SOF_IPC_STREAM_POSITION:
+ ipc_period_elapsed(sdev, msg_id);
+ break;
+ case SOF_IPC_STREAM_TRIG_XRUN:
+ ipc_xrun(sdev, msg_id);
+ break;
+ default:
+ dev_err(sdev->dev, "error: unhandled stream message %x\n",
+ msg_id);
+ break;
+ }
+}
+
+/* get stream position IPC - use faster MMIO method if available on platform */
+int snd_sof_ipc_stream_posn(struct snd_sof_dev *sdev,
+ struct snd_sof_pcm *spcm, int direction,
+ struct sof_ipc_stream_posn *posn)
+{
+ struct sof_ipc_stream stream;
+ int err;
+
+ /* read position via slower IPC */
+ stream.hdr.size = sizeof(stream);
+ stream.hdr.cmd = SOF_IPC_GLB_STREAM_MSG | SOF_IPC_STREAM_POSITION;
+ stream.comp_id = spcm->stream[direction].comp_id;
+
+ /* send IPC to the DSP */
+ err = sof_ipc_tx_message(sdev->ipc,
+ stream.hdr.cmd, &stream, sizeof(stream), &posn,
+ sizeof(*posn));
+ if (err < 0) {
+ dev_err(sdev->dev, "error: failed to get stream %d position\n",
+ stream.comp_id);
+ return err;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(snd_sof_ipc_stream_posn);
+
+static int sof_get_ctrl_copy_params(enum sof_ipc_ctrl_type ctrl_type,
+ struct sof_ipc_ctrl_data *src,
+ struct sof_ipc_ctrl_data *dst,
+ struct sof_ipc_ctrl_data_params *sparams)
+{
+ switch (ctrl_type) {
+ case SOF_CTRL_TYPE_VALUE_CHAN_GET:
+ case SOF_CTRL_TYPE_VALUE_CHAN_SET:
+ sparams->src = (u8 *)src->chanv;
+ sparams->dst = (u8 *)dst->chanv;
+ break;
+ case SOF_CTRL_TYPE_VALUE_COMP_GET:
+ case SOF_CTRL_TYPE_VALUE_COMP_SET:
+ sparams->src = (u8 *)src->compv;
+ sparams->dst = (u8 *)dst->compv;
+ break;
+ case SOF_CTRL_TYPE_DATA_GET:
+ case SOF_CTRL_TYPE_DATA_SET:
+ sparams->src = (u8 *)src->data->data;
+ sparams->dst = (u8 *)dst->data->data;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ /* calculate payload size and number of messages */
+ sparams->pl_size = SOF_IPC_MSG_MAX_SIZE - sparams->hdr_bytes;
+ sparams->num_msg = DIV_ROUND_UP(sparams->msg_bytes, sparams->pl_size);
+
+ return 0;
+}
+
+static int sof_set_get_large_ctrl_data(struct snd_sof_dev *sdev,
+ struct sof_ipc_ctrl_data *cdata,
+ struct sof_ipc_ctrl_data_params *sparams,
+ bool send)
+{
+ struct sof_ipc_ctrl_data *partdata;
+ size_t send_bytes;
+ size_t offset = 0;
+ size_t msg_bytes;
+ size_t pl_size;
+ int err;
+ int i;
+
+ /* allocate max ipc size because we have at least one */
+ partdata = kzalloc(SOF_IPC_MSG_MAX_SIZE, GFP_KERNEL);
+ if (!partdata)
+ return -ENOMEM;
+
+ if (send)
+ err = sof_get_ctrl_copy_params(cdata->type, cdata, partdata,
+ sparams);
+ else
+ err = sof_get_ctrl_copy_params(cdata->type, partdata, cdata,
+ sparams);
+ if (err < 0)
+ return err;
+
+ msg_bytes = sparams->msg_bytes;
+ pl_size = sparams->pl_size;
+
+ /* copy the header data */
+ memcpy(partdata, cdata, sparams->hdr_bytes);
+
+ /* Serialise IPC TX */
+ mutex_lock(&sdev->ipc->tx_mutex);
+
+ /* copy the payload data in a loop */
+ for (i = 0; i < sparams->num_msg; i++) {
+ send_bytes = min(msg_bytes, pl_size);
+ partdata->num_elems = send_bytes;
+ partdata->rhdr.hdr.size = sparams->hdr_bytes + send_bytes;
+ partdata->msg_index = i;
+ msg_bytes -= send_bytes;
+ partdata->elems_remaining = msg_bytes;
+
+ if (send)
+ memcpy(sparams->dst, sparams->src + offset, send_bytes);
+
+ err = sof_ipc_tx_message_unlocked(sdev->ipc,
+ partdata->rhdr.hdr.cmd,
+ partdata,
+ partdata->rhdr.hdr.size,
+ partdata,
+ partdata->rhdr.hdr.size);
+ if (err < 0)
+ break;
+
+ if (!send)
+ memcpy(sparams->dst + offset, sparams->src, send_bytes);
+
+ offset += pl_size;
+ }
+
+ mutex_unlock(&sdev->ipc->tx_mutex);
+
+ kfree(partdata);
+ return err;
+}
+
+/*
+ * IPC get()/set() for kcontrols.
+ */
+int snd_sof_ipc_set_get_comp_data(struct snd_sof_ipc *ipc,
+ struct snd_sof_control *scontrol,
+ u32 ipc_cmd,
+ enum sof_ipc_ctrl_type ctrl_type,
+ enum sof_ipc_ctrl_cmd ctrl_cmd,
+ bool send)
+{
+ struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
+ struct snd_sof_dev *sdev = ipc->sdev;
+ struct sof_ipc_fw_ready *ready = &sdev->fw_ready;
+ struct sof_ipc_fw_version *v = &ready->version;
+ struct sof_ipc_ctrl_data_params sparams;
+ size_t send_bytes;
+ int err;
+
+ /* read or write firmware volume */
+ if (scontrol->readback_offset != 0) {
+ /* write/read value header via mmaped region */
+ send_bytes = sizeof(struct sof_ipc_ctrl_value_chan) *
+ cdata->num_elems;
+ if (send)
+ snd_sof_dsp_block_write(sdev, sdev->mmio_bar,
+ scontrol->readback_offset,
+ cdata->chanv, send_bytes);
+
+ else
+ snd_sof_dsp_block_read(sdev, sdev->mmio_bar,
+ scontrol->readback_offset,
+ cdata->chanv, send_bytes);
+ return 0;
+ }
+
+ cdata->rhdr.hdr.cmd = SOF_IPC_GLB_COMP_MSG | ipc_cmd;
+ cdata->cmd = ctrl_cmd;
+ cdata->type = ctrl_type;
+ cdata->comp_id = scontrol->comp_id;
+ cdata->msg_index = 0;
+
+ /* calculate header and data size */
+ switch (cdata->type) {
+ case SOF_CTRL_TYPE_VALUE_CHAN_GET:
+ case SOF_CTRL_TYPE_VALUE_CHAN_SET:
+ sparams.msg_bytes = scontrol->num_channels *
+ sizeof(struct sof_ipc_ctrl_value_chan);
+ sparams.hdr_bytes = sizeof(struct sof_ipc_ctrl_data);
+ sparams.elems = scontrol->num_channels;
+ break;
+ case SOF_CTRL_TYPE_VALUE_COMP_GET:
+ case SOF_CTRL_TYPE_VALUE_COMP_SET:
+ sparams.msg_bytes = scontrol->num_channels *
+ sizeof(struct sof_ipc_ctrl_value_comp);
+ sparams.hdr_bytes = sizeof(struct sof_ipc_ctrl_data);
+ sparams.elems = scontrol->num_channels;
+ break;
+ case SOF_CTRL_TYPE_DATA_GET:
+ case SOF_CTRL_TYPE_DATA_SET:
+ sparams.msg_bytes = cdata->data->size;
+ sparams.hdr_bytes = sizeof(struct sof_ipc_ctrl_data) +
+ sizeof(struct sof_abi_hdr);
+ sparams.elems = cdata->data->size;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ cdata->rhdr.hdr.size = sparams.msg_bytes + sparams.hdr_bytes;
+ cdata->num_elems = sparams.elems;
+ cdata->elems_remaining = 0;
+
+ /* send normal size ipc in one part */
+ if (cdata->rhdr.hdr.size <= SOF_IPC_MSG_MAX_SIZE) {
+ err = sof_ipc_tx_message(sdev->ipc, cdata->rhdr.hdr.cmd, cdata,
+ cdata->rhdr.hdr.size, cdata,
+ cdata->rhdr.hdr.size);
+
+ if (err < 0)
+ dev_err(sdev->dev, "error: set/get ctrl ipc comp %d\n",
+ cdata->comp_id);
+
+ return err;
+ }
+
+ /* data is bigger than max ipc size, chop into smaller pieces */
+ dev_dbg(sdev->dev, "large ipc size %u, control size %u\n",
+ cdata->rhdr.hdr.size, scontrol->size);
+
+ /* large messages is only supported from ABI 3.3.0 onwards */
+ if (v->abi_version < SOF_ABI_VER(3, 3, 0)) {
+ dev_err(sdev->dev, "error: incompatible FW ABI version\n");
+ return -EINVAL;
+ }
+
+ err = sof_set_get_large_ctrl_data(sdev, cdata, &sparams, send);
+
+ if (err < 0)
+ dev_err(sdev->dev, "error: set/get large ctrl ipc comp %d\n",
+ cdata->comp_id);
+
+ return err;
+}
+EXPORT_SYMBOL(snd_sof_ipc_set_get_comp_data);
+
+/*
+ * IPC layer enumeration.
+ */
+
+int snd_sof_dsp_mailbox_init(struct snd_sof_dev *sdev, u32 dspbox,
+ size_t dspbox_size, u32 hostbox,
+ size_t hostbox_size)
+{
+ sdev->dsp_box.offset = dspbox;
+ sdev->dsp_box.size = dspbox_size;
+ sdev->host_box.offset = hostbox;
+ sdev->host_box.size = hostbox_size;
+ return 0;
+}
+EXPORT_SYMBOL(snd_sof_dsp_mailbox_init);
+
+int snd_sof_ipc_valid(struct snd_sof_dev *sdev)
+{
+ struct sof_ipc_fw_ready *ready = &sdev->fw_ready;
+ struct sof_ipc_fw_version *v = &ready->version;
+
+ dev_info(sdev->dev,
+ "Firmware info: version %d:%d:%d-%s\n", v->major, v->minor,
+ v->micro, v->tag);
+ dev_info(sdev->dev,
+ "Firmware: ABI %d:%d:%d Kernel ABI %d:%d:%d\n",
+ SOF_ABI_VERSION_MAJOR(v->abi_version),
+ SOF_ABI_VERSION_MINOR(v->abi_version),
+ SOF_ABI_VERSION_PATCH(v->abi_version),
+ SOF_ABI_MAJOR, SOF_ABI_MINOR, SOF_ABI_PATCH);
+
+ if (SOF_ABI_VERSION_INCOMPATIBLE(SOF_ABI_VERSION, v->abi_version)) {
+ dev_err(sdev->dev, "error: incompatible FW ABI version\n");
+ return -EINVAL;
+ }
+
+ if (v->abi_version > SOF_ABI_VERSION) {
+ if (!IS_ENABLED(CONFIG_SND_SOC_SOF_STRICT_ABI_CHECKS)) {
+ dev_warn(sdev->dev, "warn: FW ABI is more recent than kernel\n");
+ } else {
+ dev_err(sdev->dev, "error: FW ABI is more recent than kernel\n");
+ return -EINVAL;
+ }
+ }
+
+ if (ready->debug.bits.build) {
+ dev_info(sdev->dev,
+ "Firmware debug build %d on %s-%s - options:\n"
+ " GDB: %s\n"
+ " lock debug: %s\n"
+ " lock vdebug: %s\n",
+ v->build, v->date, v->time,
+ ready->debug.bits.gdb ? "enabled" : "disabled",
+ ready->debug.bits.locks ? "enabled" : "disabled",
+ ready->debug.bits.locks_verbose ? "enabled" : "disabled");
+ }
+
+ /* copy the fw_version into debugfs at first boot */
+ memcpy(&sdev->fw_version, v, sizeof(*v));
+
+ return 0;
+}
+EXPORT_SYMBOL(snd_sof_ipc_valid);
+
+struct snd_sof_ipc *snd_sof_ipc_init(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_ipc *ipc;
+ struct snd_sof_ipc_msg *msg;
+
+ /* check if mandatory ops required for ipc are defined */
+ if (!sof_ops(sdev)->fw_ready) {
+ dev_err(sdev->dev, "error: ipc mandatory ops not defined\n");
+ return NULL;
+ }
+
+ ipc = devm_kzalloc(sdev->dev, sizeof(*ipc), GFP_KERNEL);
+ if (!ipc)
+ return NULL;
+
+ mutex_init(&ipc->tx_mutex);
+ ipc->sdev = sdev;
+ msg = &ipc->msg;
+
+ /* indicate that we aren't sending a message ATM */
+ msg->ipc_complete = true;
+
+ /* pre-allocate message data */
+ msg->msg_data = devm_kzalloc(sdev->dev, SOF_IPC_MSG_MAX_SIZE,
+ GFP_KERNEL);
+ if (!msg->msg_data)
+ return NULL;
+
+ msg->reply_data = devm_kzalloc(sdev->dev, SOF_IPC_MSG_MAX_SIZE,
+ GFP_KERNEL);
+ if (!msg->reply_data)
+ return NULL;
+
+ init_waitqueue_head(&msg->waitq);
+
+ return ipc;
+}
+EXPORT_SYMBOL(snd_sof_ipc_init);
+
+void snd_sof_ipc_free(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_ipc *ipc = sdev->ipc;
+
+ /* disable sending of ipc's */
+ mutex_lock(&ipc->tx_mutex);
+ ipc->disable_ipc_tx = true;
+ mutex_unlock(&ipc->tx_mutex);
+}
+EXPORT_SYMBOL(snd_sof_ipc_free);
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+// Generic firmware loader.
+//
+
+#include <linux/firmware.h>
+#include <sound/sof.h>
+#include "ops.h"
+
+static int get_ext_windows(struct snd_sof_dev *sdev,
+ struct sof_ipc_ext_data_hdr *ext_hdr)
+{
+ struct sof_ipc_window *w =
+ container_of(ext_hdr, struct sof_ipc_window, ext_hdr);
+ size_t size;
+
+ if (w->num_windows == 0 || w->num_windows > SOF_IPC_MAX_ELEMS)
+ return -EINVAL;
+
+ size = sizeof(*w) + sizeof(struct sof_ipc_window_elem) * w->num_windows;
+
+ /* keep a local copy of the data */
+ sdev->info_window = kmemdup(w, size, GFP_KERNEL);
+ if (!sdev->info_window)
+ return -ENOMEM;
+
+ return 0;
+}
+
+/* parse the extended FW boot data structures from FW boot message */
+int snd_sof_fw_parse_ext_data(struct snd_sof_dev *sdev, u32 bar, u32 offset)
+{
+ struct sof_ipc_ext_data_hdr *ext_hdr;
+ void *ext_data;
+ int ret = 0;
+
+ ext_data = kzalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!ext_data)
+ return -ENOMEM;
+
+ /* get first header */
+ snd_sof_dsp_block_read(sdev, bar, offset, ext_data,
+ sizeof(*ext_hdr));
+ ext_hdr = ext_data;
+
+ while (ext_hdr->hdr.cmd == SOF_IPC_FW_READY) {
+ /* read in ext structure */
+ offset += sizeof(*ext_hdr);
+ snd_sof_dsp_block_read(sdev, bar, offset,
+ (void *)((u8 *)ext_data + sizeof(*ext_hdr)),
+ ext_hdr->hdr.size - sizeof(*ext_hdr));
+
+ dev_dbg(sdev->dev, "found ext header type %d size 0x%x\n",
+ ext_hdr->type, ext_hdr->hdr.size);
+
+ /* process structure data */
+ switch (ext_hdr->type) {
+ case SOF_IPC_EXT_DMA_BUFFER:
+ break;
+ case SOF_IPC_EXT_WINDOW:
+ ret = get_ext_windows(sdev, ext_hdr);
+ break;
+ default:
+ break;
+ }
+
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to parse ext data type %d\n",
+ ext_hdr->type);
+ break;
+ }
+
+ /* move to next header */
+ offset += ext_hdr->hdr.size;
+ snd_sof_dsp_block_read(sdev, bar, offset, ext_data,
+ sizeof(*ext_hdr));
+ ext_hdr = ext_data;
+ }
+
+ kfree(ext_data);
+ return ret;
+}
+EXPORT_SYMBOL(snd_sof_fw_parse_ext_data);
+
+/* generic module parser for mmaped DSPs */
+int snd_sof_parse_module_memcpy(struct snd_sof_dev *sdev,
+ struct snd_sof_mod_hdr *module)
+{
+ struct snd_sof_blk_hdr *block;
+ int count;
+ u32 offset;
+ size_t remaining;
+
+ dev_dbg(sdev->dev, "new module size 0x%x blocks 0x%x type 0x%x\n",
+ module->size, module->num_blocks, module->type);
+
+ block = (struct snd_sof_blk_hdr *)((u8 *)module + sizeof(*module));
+
+ /* module->size doesn't include header size */
+ remaining = module->size;
+ for (count = 0; count < module->num_blocks; count++) {
+ /* check for wrap */
+ if (remaining < sizeof(*block)) {
+ dev_err(sdev->dev, "error: not enough data remaining\n");
+ return -EINVAL;
+ }
+
+ /* minus header size of block */
+ remaining -= sizeof(*block);
+
+ if (block->size == 0) {
+ dev_warn(sdev->dev,
+ "warning: block %d size zero\n", count);
+ dev_warn(sdev->dev, " type 0x%x offset 0x%x\n",
+ block->type, block->offset);
+ continue;
+ }
+
+ switch (block->type) {
+ case SOF_FW_BLK_TYPE_RSRVD0:
+ case SOF_FW_BLK_TYPE_SRAM...SOF_FW_BLK_TYPE_RSRVD14:
+ continue; /* not handled atm */
+ case SOF_FW_BLK_TYPE_IRAM:
+ case SOF_FW_BLK_TYPE_DRAM:
+ offset = block->offset;
+ break;
+ default:
+ dev_err(sdev->dev, "error: bad type 0x%x for block 0x%x\n",
+ block->type, count);
+ return -EINVAL;
+ }
+
+ dev_dbg(sdev->dev,
+ "block %d type 0x%x size 0x%x ==> offset 0x%x\n",
+ count, block->type, block->size, offset);
+
+ /* checking block->size to avoid unaligned access */
+ if (block->size % sizeof(u32)) {
+ dev_err(sdev->dev, "error: invalid block size 0x%x\n",
+ block->size);
+ return -EINVAL;
+ }
+ snd_sof_dsp_block_write(sdev, sdev->mmio_bar, offset,
+ block + 1, block->size);
+
+ if (remaining < block->size) {
+ dev_err(sdev->dev, "error: not enough data remaining\n");
+ return -EINVAL;
+ }
+
+ /* minus body size of block */
+ remaining -= block->size;
+ /* next block */
+ block = (struct snd_sof_blk_hdr *)((u8 *)block + sizeof(*block)
+ + block->size);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(snd_sof_parse_module_memcpy);
+
+static int check_header(struct snd_sof_dev *sdev, const struct firmware *fw)
+{
+ struct snd_sof_fw_header *header;
+
+ /* Read the header information from the data pointer */
+ header = (struct snd_sof_fw_header *)fw->data;
+
+ /* verify FW sig */
+ if (strncmp(header->sig, SND_SOF_FW_SIG, SND_SOF_FW_SIG_SIZE) != 0) {
+ dev_err(sdev->dev, "error: invalid firmware signature\n");
+ return -EINVAL;
+ }
+
+ /* check size is valid */
+ if (fw->size != header->file_size + sizeof(*header)) {
+ dev_err(sdev->dev, "error: invalid filesize mismatch got 0x%zx expected 0x%zx\n",
+ fw->size, header->file_size + sizeof(*header));
+ return -EINVAL;
+ }
+
+ dev_dbg(sdev->dev, "header size=0x%x modules=0x%x abi=0x%x size=%zu\n",
+ header->file_size, header->num_modules,
+ header->abi, sizeof(*header));
+
+ return 0;
+}
+
+static int load_modules(struct snd_sof_dev *sdev, const struct firmware *fw)
+{
+ struct snd_sof_fw_header *header;
+ struct snd_sof_mod_hdr *module;
+ int (*load_module)(struct snd_sof_dev *sof_dev,
+ struct snd_sof_mod_hdr *hdr);
+ int ret, count;
+ size_t remaining;
+
+ header = (struct snd_sof_fw_header *)fw->data;
+ load_module = sof_ops(sdev)->load_module;
+ if (!load_module)
+ return -EINVAL;
+
+ /* parse each module */
+ module = (struct snd_sof_mod_hdr *)((u8 *)(fw->data) + sizeof(*header));
+ remaining = fw->size - sizeof(*header);
+ /* check for wrap */
+ if (remaining > fw->size) {
+ dev_err(sdev->dev, "error: fw size smaller than header size\n");
+ return -EINVAL;
+ }
+
+ for (count = 0; count < header->num_modules; count++) {
+ /* check for wrap */
+ if (remaining < sizeof(*module)) {
+ dev_err(sdev->dev, "error: not enough data remaining\n");
+ return -EINVAL;
+ }
+
+ /* minus header size of module */
+ remaining -= sizeof(*module);
+
+ /* module */
+ ret = load_module(sdev, module);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: invalid module %d\n", count);
+ return ret;
+ }
+
+ if (remaining < module->size) {
+ dev_err(sdev->dev, "error: not enough data remaining\n");
+ return -EINVAL;
+ }
+
+ /* minus body size of module */
+ remaining -= module->size;
+ module = (struct snd_sof_mod_hdr *)((u8 *)module
+ + sizeof(*module) + module->size);
+ }
+
+ return 0;
+}
+
+int snd_sof_load_firmware_raw(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_pdata *plat_data = sdev->pdata;
+ const char *fw_filename;
+ int ret;
+
+ /* set code loading condition to true */
+ sdev->code_loading = 1;
+
+ /* Don't request firmware again if firmware is already requested */
+ if (plat_data->fw)
+ return 0;
+
+ fw_filename = kasprintf(GFP_KERNEL, "%s/%s",
+ plat_data->fw_filename_prefix,
+ plat_data->fw_filename);
+ if (!fw_filename)
+ return -ENOMEM;
+
+ ret = request_firmware(&plat_data->fw, fw_filename, sdev->dev);
+
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: request firmware %s failed err: %d\n",
+ fw_filename, ret);
+ }
+
+ kfree(fw_filename);
+
+ return ret;
+}
+EXPORT_SYMBOL(snd_sof_load_firmware_raw);
+
+int snd_sof_load_firmware_memcpy(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_pdata *plat_data = sdev->pdata;
+ int ret;
+
+ ret = snd_sof_load_firmware_raw(sdev);
+ if (ret < 0)
+ return ret;
+
+ /* make sure the FW header and file is valid */
+ ret = check_header(sdev, plat_data->fw);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: invalid FW header\n");
+ goto error;
+ }
+
+ /* prepare the DSP for FW loading */
+ ret = snd_sof_dsp_reset(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to reset DSP\n");
+ goto error;
+ }
+
+ /* parse and load firmware modules to DSP */
+ ret = load_modules(sdev, plat_data->fw);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: invalid FW modules\n");
+ goto error;
+ }
+
+ return 0;
+
+error:
+ release_firmware(plat_data->fw);
+ plat_data->fw = NULL;
+ return ret;
+
+}
+EXPORT_SYMBOL(snd_sof_load_firmware_memcpy);
+
+int snd_sof_load_firmware(struct snd_sof_dev *sdev)
+{
+ dev_dbg(sdev->dev, "loading firmware\n");
+
+ if (sof_ops(sdev)->load_firmware)
+ return sof_ops(sdev)->load_firmware(sdev);
+ return 0;
+}
+EXPORT_SYMBOL(snd_sof_load_firmware);
+
+int snd_sof_run_firmware(struct snd_sof_dev *sdev)
+{
+ int ret;
+ int init_core_mask;
+
+ init_waitqueue_head(&sdev->boot_wait);
+ sdev->boot_complete = false;
+
+ /* create fw_version debugfs to store boot version info */
+ if (sdev->first_boot) {
+ ret = snd_sof_debugfs_buf_item(sdev, &sdev->fw_version,
+ sizeof(sdev->fw_version),
+ "fw_version");
+ /* errors are only due to memory allocation, not debugfs */
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: snd_sof_debugfs_buf_item failed\n");
+ return ret;
+ }
+ }
+
+ /* perform pre fw run operations */
+ ret = snd_sof_dsp_pre_fw_run(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed pre fw run op\n");
+ return ret;
+ }
+
+ dev_dbg(sdev->dev, "booting DSP firmware\n");
+
+ /* boot the firmware on the DSP */
+ ret = snd_sof_dsp_run(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to reset DSP\n");
+ return ret;
+ }
+
+ init_core_mask = ret;
+
+ /* now wait for the DSP to boot */
+ ret = wait_event_timeout(sdev->boot_wait, sdev->boot_complete,
+ msecs_to_jiffies(sdev->boot_timeout));
+ if (ret == 0) {
+ dev_err(sdev->dev, "error: firmware boot failure\n");
+ snd_sof_dsp_dbg_dump(sdev, SOF_DBG_REGS | SOF_DBG_MBOX |
+ SOF_DBG_TEXT | SOF_DBG_PCI);
+ return -EIO;
+ }
+
+ dev_info(sdev->dev, "firmware boot complete\n");
+
+ /* perform post fw run operations */
+ ret = snd_sof_dsp_post_fw_run(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed post fw run op\n");
+ return ret;
+ }
+
+ /* fw boot is complete. Update the active cores mask */
+ sdev->enabled_cores_mask = init_core_mask;
+
+ return 0;
+}
+EXPORT_SYMBOL(snd_sof_run_firmware);
+
+void snd_sof_fw_unload(struct snd_sof_dev *sdev)
+{
+ /* TODO: support module unloading at runtime */
+}
+EXPORT_SYMBOL(snd_sof_fw_unload);
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+#include <linux/module.h>
+#include <sound/sof.h>
+#include "sof-priv.h"
+
+static struct snd_soc_card sof_nocodec_card = {
+ .name = "nocodec", /* the sof- prefix is added by the core */
+};
+
+static int sof_nocodec_bes_setup(struct device *dev,
+ const struct snd_sof_dsp_ops *ops,
+ struct snd_soc_dai_link *links,
+ int link_num, struct snd_soc_card *card)
+{
+ int i;
+
+ if (!ops || !links || !card)
+ return -EINVAL;
+
+ /* set up BE dai_links */
+ for (i = 0; i < link_num; i++) {
+ links[i].name = devm_kasprintf(dev, GFP_KERNEL,
+ "NoCodec-%d", i);
+ if (!links[i].name)
+ return -ENOMEM;
+
+ links[i].id = i;
+ links[i].no_pcm = 1;
+ links[i].cpu_dai_name = ops->drv[i].name;
+ links[i].platform_name = dev_name(dev);
+ links[i].codec_dai_name = "snd-soc-dummy-dai";
+ links[i].codec_name = "snd-soc-dummy";
+ links[i].dpcm_playback = 1;
+ links[i].dpcm_capture = 1;
+ }
+
+ card->dai_link = links;
+ card->num_links = link_num;
+
+ return 0;
+}
+
+int sof_nocodec_setup(struct device *dev,
+ struct snd_sof_pdata *sof_pdata,
+ struct snd_soc_acpi_mach *mach,
+ const struct sof_dev_desc *desc,
+ const struct snd_sof_dsp_ops *ops)
+{
+ struct snd_soc_dai_link *links;
+ int ret;
+
+ if (!mach)
+ return -EINVAL;
+
+ sof_pdata->drv_name = "sof-nocodec";
+
+ mach->drv_name = "sof-nocodec";
+ sof_pdata->fw_filename = desc->nocodec_fw_filename;
+ sof_pdata->tplg_filename = desc->nocodec_tplg_filename;
+
+ /* create dummy BE dai_links */
+ links = devm_kzalloc(dev, sizeof(struct snd_soc_dai_link) *
+ ops->num_drv, GFP_KERNEL);
+ if (!links)
+ return -ENOMEM;
+
+ ret = sof_nocodec_bes_setup(dev, ops, links, ops->num_drv,
+ &sof_nocodec_card);
+ return ret;
+}
+EXPORT_SYMBOL(sof_nocodec_setup);
+
+static int sof_nocodec_probe(struct platform_device *pdev)
+{
+ struct snd_soc_card *card = &sof_nocodec_card;
+
+ card->dev = &pdev->dev;
+
+ return devm_snd_soc_register_card(&pdev->dev, card);
+}
+
+static int sof_nocodec_remove(struct platform_device *pdev)
+{
+ return 0;
+}
+
+static struct platform_driver sof_nocodec_audio = {
+ .probe = sof_nocodec_probe,
+ .remove = sof_nocodec_remove,
+ .driver = {
+ .name = "sof-nocodec",
+ .pm = &snd_soc_pm_ops,
+ },
+};
+module_platform_driver(sof_nocodec_audio)
+
+MODULE_DESCRIPTION("ASoC sof nocodec");
+MODULE_AUTHOR("Liam Girdwood");
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_ALIAS("platform:sof-nocodec");
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+#include <linux/pci.h>
+#include "ops.h"
+
+static
+bool snd_sof_pci_update_bits_unlocked(struct snd_sof_dev *sdev, u32 offset,
+ u32 mask, u32 value)
+{
+ struct pci_dev *pci = to_pci_dev(sdev->dev);
+ unsigned int old, new;
+ u32 ret = 0;
+
+ pci_read_config_dword(pci, offset, &ret);
+ old = ret;
+ dev_dbg(sdev->dev, "Debug PCIR: %8.8x at %8.8x\n", old & mask, offset);
+
+ new = (old & ~mask) | (value & mask);
+
+ if (old == new)
+ return false;
+
+ pci_write_config_dword(pci, offset, new);
+ dev_dbg(sdev->dev, "Debug PCIW: %8.8x at %8.8x\n", value,
+ offset);
+
+ return true;
+}
+
+bool snd_sof_pci_update_bits(struct snd_sof_dev *sdev, u32 offset,
+ u32 mask, u32 value)
+{
+ unsigned long flags;
+ bool change;
+
+ spin_lock_irqsave(&sdev->hw_lock, flags);
+ change = snd_sof_pci_update_bits_unlocked(sdev, offset, mask, value);
+ spin_unlock_irqrestore(&sdev->hw_lock, flags);
+ return change;
+}
+EXPORT_SYMBOL(snd_sof_pci_update_bits);
+
+bool snd_sof_dsp_update_bits_unlocked(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, u32 mask, u32 value)
+{
+ unsigned int old, new;
+ u32 ret;
+
+ ret = snd_sof_dsp_read(sdev, bar, offset);
+
+ old = ret;
+ new = (old & ~mask) | (value & mask);
+
+ if (old == new)
+ return false;
+
+ snd_sof_dsp_write(sdev, bar, offset, new);
+
+ return true;
+}
+EXPORT_SYMBOL(snd_sof_dsp_update_bits_unlocked);
+
+bool snd_sof_dsp_update_bits64_unlocked(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, u64 mask, u64 value)
+{
+ u64 old, new;
+
+ old = snd_sof_dsp_read64(sdev, bar, offset);
+
+ new = (old & ~mask) | (value & mask);
+
+ if (old == new)
+ return false;
+
+ snd_sof_dsp_write64(sdev, bar, offset, new);
+
+ return true;
+}
+EXPORT_SYMBOL(snd_sof_dsp_update_bits64_unlocked);
+
+/* This is for registers bits with attribute RWC */
+bool snd_sof_dsp_update_bits(struct snd_sof_dev *sdev, u32 bar, u32 offset,
+ u32 mask, u32 value)
+{
+ unsigned long flags;
+ bool change;
+
+ spin_lock_irqsave(&sdev->hw_lock, flags);
+ change = snd_sof_dsp_update_bits_unlocked(sdev, bar, offset, mask,
+ value);
+ spin_unlock_irqrestore(&sdev->hw_lock, flags);
+ return change;
+}
+EXPORT_SYMBOL(snd_sof_dsp_update_bits);
+
+bool snd_sof_dsp_update_bits64(struct snd_sof_dev *sdev, u32 bar, u32 offset,
+ u64 mask, u64 value)
+{
+ unsigned long flags;
+ bool change;
+
+ spin_lock_irqsave(&sdev->hw_lock, flags);
+ change = snd_sof_dsp_update_bits64_unlocked(sdev, bar, offset, mask,
+ value);
+ spin_unlock_irqrestore(&sdev->hw_lock, flags);
+ return change;
+}
+EXPORT_SYMBOL(snd_sof_dsp_update_bits64);
+
+static
+void snd_sof_dsp_update_bits_forced_unlocked(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, u32 mask, u32 value)
+{
+ unsigned int old, new;
+ u32 ret;
+
+ ret = snd_sof_dsp_read(sdev, bar, offset);
+
+ old = ret;
+ new = (old & ~mask) | (value & mask);
+
+ snd_sof_dsp_write(sdev, bar, offset, new);
+}
+
+/* This is for registers bits with attribute RWC */
+void snd_sof_dsp_update_bits_forced(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, u32 mask, u32 value)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&sdev->hw_lock, flags);
+ snd_sof_dsp_update_bits_forced_unlocked(sdev, bar, offset, mask, value);
+ spin_unlock_irqrestore(&sdev->hw_lock, flags);
+}
+EXPORT_SYMBOL(snd_sof_dsp_update_bits_forced);
+
+void snd_sof_dsp_panic(struct snd_sof_dev *sdev, u32 offset)
+{
+ dev_err(sdev->dev, "error : DSP panic!\n");
+
+ /*
+ * check if DSP is not ready and did not set the dsp_oops_offset.
+ * if the dsp_oops_offset is not set, set it from the panic message.
+ * Also add a check to memory window setting with panic message.
+ */
+ if (!sdev->dsp_oops_offset)
+ sdev->dsp_oops_offset = offset;
+ else
+ dev_dbg(sdev->dev, "panic: dsp_oops_offset %zu offset %d\n",
+ sdev->dsp_oops_offset, offset);
+
+ snd_sof_dsp_dbg_dump(sdev, SOF_DBG_REGS | SOF_DBG_MBOX);
+ snd_sof_trace_notify_for_error(sdev);
+}
+EXPORT_SYMBOL(snd_sof_dsp_panic);
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ *
+ * Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+ */
+
+#ifndef __SOUND_SOC_SOF_IO_H
+#define __SOUND_SOC_SOF_IO_H
+
+#include <linux/device.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <sound/pcm.h>
+#include "sof-priv.h"
+
+#define sof_ops(sdev) \
+ ((sdev)->pdata->desc->ops)
+
+/* Mandatory operations are verified during probing */
+
+/* init */
+static inline int snd_sof_probe(struct snd_sof_dev *sdev)
+{
+ return sof_ops(sdev)->probe(sdev);
+}
+
+static inline int snd_sof_remove(struct snd_sof_dev *sdev)
+{
+ if (sof_ops(sdev)->remove)
+ return sof_ops(sdev)->remove(sdev);
+
+ return 0;
+}
+
+/* control */
+
+/*
+ * snd_sof_dsp_run returns the core mask of the cores that are available
+ * after successful fw boot
+ */
+static inline int snd_sof_dsp_run(struct snd_sof_dev *sdev)
+{
+ return sof_ops(sdev)->run(sdev);
+}
+
+static inline int snd_sof_dsp_stall(struct snd_sof_dev *sdev)
+{
+ if (sof_ops(sdev)->stall)
+ return sof_ops(sdev)->stall(sdev);
+
+ return 0;
+}
+
+static inline int snd_sof_dsp_reset(struct snd_sof_dev *sdev)
+{
+ if (sof_ops(sdev)->reset)
+ return sof_ops(sdev)->reset(sdev);
+
+ return 0;
+}
+
+/* dsp core power up/power down */
+static inline int snd_sof_dsp_core_power_up(struct snd_sof_dev *sdev,
+ unsigned int core_mask)
+{
+ if (sof_ops(sdev)->core_power_up)
+ return sof_ops(sdev)->core_power_up(sdev, core_mask);
+
+ return 0;
+}
+
+static inline int snd_sof_dsp_core_power_down(struct snd_sof_dev *sdev,
+ unsigned int core_mask)
+{
+ if (sof_ops(sdev)->core_power_down)
+ return sof_ops(sdev)->core_power_down(sdev, core_mask);
+
+ return 0;
+}
+
+/* pre/post fw load */
+static inline int snd_sof_dsp_pre_fw_run(struct snd_sof_dev *sdev)
+{
+ if (sof_ops(sdev)->pre_fw_run)
+ return sof_ops(sdev)->pre_fw_run(sdev);
+
+ return 0;
+}
+
+static inline int snd_sof_dsp_post_fw_run(struct snd_sof_dev *sdev)
+{
+ if (sof_ops(sdev)->post_fw_run)
+ return sof_ops(sdev)->post_fw_run(sdev);
+
+ return 0;
+}
+
+/* power management */
+static inline int snd_sof_dsp_resume(struct snd_sof_dev *sdev)
+{
+ if (sof_ops(sdev)->resume)
+ return sof_ops(sdev)->resume(sdev);
+
+ return 0;
+}
+
+static inline int snd_sof_dsp_suspend(struct snd_sof_dev *sdev, int state)
+{
+ if (sof_ops(sdev)->suspend)
+ return sof_ops(sdev)->suspend(sdev, state);
+
+ return 0;
+}
+
+static inline int snd_sof_dsp_runtime_resume(struct snd_sof_dev *sdev)
+{
+ if (sof_ops(sdev)->runtime_resume)
+ return sof_ops(sdev)->runtime_resume(sdev);
+
+ return 0;
+}
+
+static inline int snd_sof_dsp_runtime_suspend(struct snd_sof_dev *sdev,
+ int state)
+{
+ if (sof_ops(sdev)->runtime_suspend)
+ return sof_ops(sdev)->runtime_suspend(sdev, state);
+
+ return 0;
+}
+
+static inline void snd_sof_dsp_hw_params_upon_resume(struct snd_sof_dev *sdev)
+{
+ if (sof_ops(sdev)->set_hw_params_upon_resume)
+ sof_ops(sdev)->set_hw_params_upon_resume(sdev);
+}
+
+static inline int snd_sof_dsp_set_clk(struct snd_sof_dev *sdev, u32 freq)
+{
+ if (sof_ops(sdev)->set_clk)
+ return sof_ops(sdev)->set_clk(sdev, freq);
+
+ return 0;
+}
+
+/* debug */
+static inline void snd_sof_dsp_dbg_dump(struct snd_sof_dev *sdev, u32 flags)
+{
+ if (sof_ops(sdev)->dbg_dump)
+ return sof_ops(sdev)->dbg_dump(sdev, flags);
+}
+
+static inline void snd_sof_ipc_dump(struct snd_sof_dev *sdev)
+{
+ if (sof_ops(sdev)->ipc_dump)
+ return sof_ops(sdev)->ipc_dump(sdev);
+}
+
+/* register IO */
+static inline void snd_sof_dsp_write(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, u32 value)
+{
+ if (sof_ops(sdev)->write) {
+ sof_ops(sdev)->write(sdev, sdev->bar[bar] + offset, value);
+ return;
+ }
+
+ dev_err_ratelimited(sdev->dev, "error: %s not defined\n", __func__);
+}
+
+static inline void snd_sof_dsp_write64(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, u64 value)
+{
+ if (sof_ops(sdev)->write64) {
+ sof_ops(sdev)->write64(sdev, sdev->bar[bar] + offset, value);
+ return;
+ }
+
+ dev_err_ratelimited(sdev->dev, "error: %s not defined\n", __func__);
+}
+
+static inline u32 snd_sof_dsp_read(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset)
+{
+ if (sof_ops(sdev)->read)
+ return sof_ops(sdev)->read(sdev, sdev->bar[bar] + offset);
+
+ dev_err(sdev->dev, "error: %s not defined\n", __func__);
+ return -ENOTSUPP;
+}
+
+static inline u64 snd_sof_dsp_read64(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset)
+{
+ if (sof_ops(sdev)->read64)
+ return sof_ops(sdev)->read64(sdev, sdev->bar[bar] + offset);
+
+ dev_err(sdev->dev, "error: %s not defined\n", __func__);
+ return -ENOTSUPP;
+}
+
+/* block IO */
+static inline void snd_sof_dsp_block_read(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, void *dest, size_t bytes)
+{
+ sof_ops(sdev)->block_read(sdev, bar, offset, dest, bytes);
+}
+
+static inline void snd_sof_dsp_block_write(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, void *src, size_t bytes)
+{
+ sof_ops(sdev)->block_write(sdev, bar, offset, src, bytes);
+}
+
+/* ipc */
+static inline int snd_sof_dsp_send_msg(struct snd_sof_dev *sdev,
+ struct snd_sof_ipc_msg *msg)
+{
+ return sof_ops(sdev)->send_msg(sdev, msg);
+}
+
+/* host DMA trace */
+static inline int snd_sof_dma_trace_init(struct snd_sof_dev *sdev,
+ u32 *stream_tag)
+{
+ if (sof_ops(sdev)->trace_init)
+ return sof_ops(sdev)->trace_init(sdev, stream_tag);
+
+ return 0;
+}
+
+static inline int snd_sof_dma_trace_release(struct snd_sof_dev *sdev)
+{
+ if (sof_ops(sdev)->trace_release)
+ return sof_ops(sdev)->trace_release(sdev);
+
+ return 0;
+}
+
+static inline int snd_sof_dma_trace_trigger(struct snd_sof_dev *sdev, int cmd)
+{
+ if (sof_ops(sdev)->trace_trigger)
+ return sof_ops(sdev)->trace_trigger(sdev, cmd);
+
+ return 0;
+}
+
+/* host PCM ops */
+static inline int
+snd_sof_pcm_platform_open(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream)
+{
+ if (sof_ops(sdev) && sof_ops(sdev)->pcm_open)
+ return sof_ops(sdev)->pcm_open(sdev, substream);
+
+ return 0;
+}
+
+/* disconnect pcm substream to a host stream */
+static inline int
+snd_sof_pcm_platform_close(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream)
+{
+ if (sof_ops(sdev) && sof_ops(sdev)->pcm_close)
+ return sof_ops(sdev)->pcm_close(sdev, substream);
+
+ return 0;
+}
+
+/* host stream hw params */
+static inline int
+snd_sof_pcm_platform_hw_params(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params,
+ struct sof_ipc_stream_params *ipc_params)
+{
+ if (sof_ops(sdev) && sof_ops(sdev)->pcm_hw_params)
+ return sof_ops(sdev)->pcm_hw_params(sdev, substream,
+ params, ipc_params);
+
+ return 0;
+}
+
+/* host stream trigger */
+static inline int
+snd_sof_pcm_platform_trigger(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream, int cmd)
+{
+ if (sof_ops(sdev) && sof_ops(sdev)->pcm_trigger)
+ return sof_ops(sdev)->pcm_trigger(sdev, substream, cmd);
+
+ return 0;
+}
+
+/* host DSP message data */
+static inline void snd_sof_ipc_msg_data(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ void *p, size_t sz)
+{
+ sof_ops(sdev)->ipc_msg_data(sdev, substream, p, sz);
+}
+
+/* host configure DSP HW parameters */
+static inline int
+snd_sof_ipc_pcm_params(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ const struct sof_ipc_pcm_params_reply *reply)
+{
+ return sof_ops(sdev)->ipc_pcm_params(sdev, substream, reply);
+}
+
+/* host stream pointer */
+static inline snd_pcm_uframes_t
+snd_sof_pcm_platform_pointer(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream)
+{
+ if (sof_ops(sdev) && sof_ops(sdev)->pcm_pointer)
+ return sof_ops(sdev)->pcm_pointer(sdev, substream);
+
+ return 0;
+}
+
+static inline const struct snd_sof_dsp_ops
+*sof_get_ops(const struct sof_dev_desc *d,
+ const struct sof_ops_table mach_ops[], int asize)
+{
+ int i;
+
+ for (i = 0; i < asize; i++) {
+ if (d == mach_ops[i].desc)
+ return mach_ops[i].ops;
+ }
+
+ /* not found */
+ return NULL;
+}
+
+/**
+ * snd_sof_dsp_register_poll_timeout - Periodically poll an address
+ * until a condition is met or a timeout occurs
+ * @op: accessor function (takes @addr as its only argument)
+ * @addr: Address to poll
+ * @val: Variable to read the value into
+ * @cond: Break condition (usually involving @val)
+ * @sleep_us: Maximum time to sleep between reads in us (0
+ * tight-loops). Should be less than ~20ms since usleep_range
+ * is used (see Documentation/timers/timers-howto.txt).
+ * @timeout_us: Timeout in us, 0 means never timeout
+ *
+ * Returns 0 on success and -ETIMEDOUT upon a timeout. In either
+ * case, the last read value at @addr is stored in @val. Must not
+ * be called from atomic context if sleep_us or timeout_us are used.
+ *
+ * This is modelled after the readx_poll_timeout macros in linux/iopoll.h.
+ */
+#define snd_sof_dsp_read_poll_timeout(sdev, bar, offset, val, cond, sleep_us, timeout_us) \
+({ \
+ u64 __timeout_us = (timeout_us); \
+ unsigned long __sleep_us = (sleep_us); \
+ ktime_t __timeout = ktime_add_us(ktime_get(), __timeout_us); \
+ might_sleep_if((__sleep_us) != 0); \
+ for (;;) { \
+ (val) = snd_sof_dsp_read(sdev, bar, offset); \
+ if (cond) { \
+ dev_dbg(sdev->dev, \
+ "FW Poll Status: reg=%#x successful\n", (val)); \
+ break; \
+ } \
+ if (__timeout_us && \
+ ktime_compare(ktime_get(), __timeout) > 0) { \
+ (val) = snd_sof_dsp_read(sdev, bar, offset); \
+ dev_dbg(sdev->dev, \
+ "FW Poll Status: reg=%#x timedout\n", (val)); \
+ break; \
+ } \
+ if (__sleep_us) \
+ usleep_range((__sleep_us >> 2) + 1, __sleep_us); \
+ } \
+ (cond) ? 0 : -ETIMEDOUT; \
+})
+
+/* This is for registers bits with attribute RWC */
+bool snd_sof_pci_update_bits(struct snd_sof_dev *sdev, u32 offset,
+ u32 mask, u32 value);
+
+bool snd_sof_dsp_update_bits_unlocked(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, u32 mask, u32 value);
+
+bool snd_sof_dsp_update_bits64_unlocked(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, u64 mask, u64 value);
+
+bool snd_sof_dsp_update_bits(struct snd_sof_dev *sdev, u32 bar, u32 offset,
+ u32 mask, u32 value);
+
+bool snd_sof_dsp_update_bits64(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, u64 mask, u64 value);
+
+void snd_sof_dsp_update_bits_forced(struct snd_sof_dev *sdev, u32 bar,
+ u32 offset, u32 mask, u32 value);
+
+int snd_sof_dsp_register_poll(struct snd_sof_dev *sdev, u32 bar, u32 offset,
+ u32 mask, u32 target, u32 timeout_ms,
+ u32 interval_us);
+
+void snd_sof_dsp_panic(struct snd_sof_dev *sdev, u32 offset);
+#endif
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+// PCM Layer, interface between ALSA and IPC.
+//
+
+#include <linux/pm_runtime.h>
+#include <sound/pcm_params.h>
+#include <sound/sof.h>
+#include "sof-priv.h"
+#include "ops.h"
+
+#define DRV_NAME "sof-audio-component"
+
+/* Create DMA buffer page table for DSP */
+static int create_page_table(struct snd_pcm_substream *substream,
+ unsigned char *dma_area, size_t size)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pcm *spcm;
+ struct snd_dma_buffer *dmab = snd_pcm_get_dma_buf(substream);
+ int stream = substream->stream;
+
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm)
+ return -EINVAL;
+
+ return snd_sof_create_page_table(sdev, dmab,
+ spcm->stream[stream].page_table.area, size);
+}
+
+static int sof_pcm_dsp_params(struct snd_sof_pcm *spcm, struct snd_pcm_substream *substream,
+ const struct sof_ipc_pcm_params_reply *reply)
+{
+ struct snd_sof_dev *sdev = spcm->sdev;
+ /* validate offset */
+ int ret = snd_sof_ipc_pcm_params(sdev, substream, reply);
+
+ if (ret < 0)
+ dev_err(sdev->dev, "error: got wrong reply for PCM %d\n",
+ spcm->pcm.pcm_id);
+
+ return ret;
+}
+
+/*
+ * sof pcm period elapse work
+ */
+static void sof_pcm_period_elapsed_work(struct work_struct *work)
+{
+ struct snd_sof_pcm_stream *sps =
+ container_of(work, struct snd_sof_pcm_stream,
+ period_elapsed_work);
+
+ snd_pcm_period_elapsed(sps->substream);
+}
+
+/*
+ * sof pcm period elapse, this could be called at irq thread context.
+ */
+void snd_sof_pcm_period_elapsed(struct snd_pcm_substream *substream)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pcm *spcm;
+
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm) {
+ dev_err(sdev->dev,
+ "error: period elapsed for unknown stream!\n");
+ return;
+ }
+
+ /*
+ * snd_pcm_period_elapsed() can be called in interrupt context
+ * before IRQ_HANDLED is returned. Inside snd_pcm_period_elapsed(),
+ * when the PCM is done draining or xrun happened, a STOP IPC will
+ * then be sent and this IPC will hit IPC timeout.
+ * To avoid sending IPC before the previous IPC is handled, we
+ * schedule delayed work here to call the snd_pcm_period_elapsed().
+ */
+ schedule_work(&spcm->stream[substream->stream].period_elapsed_work);
+}
+EXPORT_SYMBOL(snd_sof_pcm_period_elapsed);
+
+/* this may get called several times by oss emulation */
+static int sof_pcm_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_pcm_runtime *runtime = substream->runtime;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pcm *spcm;
+ struct sof_ipc_pcm_params pcm;
+ struct sof_ipc_pcm_params_reply ipc_params_reply;
+ int ret;
+
+ /* nothing to do for BE */
+ if (rtd->dai_link->no_pcm)
+ return 0;
+
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm)
+ return -EINVAL;
+
+ dev_dbg(sdev->dev, "pcm: hw params stream %d dir %d\n",
+ spcm->pcm.pcm_id, substream->stream);
+
+ memset(&pcm, 0, sizeof(pcm));
+
+ /* allocate audio buffer pages */
+ ret = snd_pcm_lib_malloc_pages(substream, params_buffer_bytes(params));
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: could not allocate %d bytes for PCM %d\n",
+ params_buffer_bytes(params), spcm->pcm.pcm_id);
+ return ret;
+ }
+ if (ret) {
+ /*
+ * ret == 1 means the buffer is changed
+ * create compressed page table for audio firmware
+ * ret == 0 means the buffer is not changed
+ * so no need to regenerate the page table
+ */
+ ret = create_page_table(substream, runtime->dma_area,
+ runtime->dma_bytes);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* number of pages should be rounded up */
+ pcm.params.buffer.pages = PFN_UP(runtime->dma_bytes);
+
+ /* set IPC PCM parameters */
+ pcm.hdr.size = sizeof(pcm);
+ pcm.hdr.cmd = SOF_IPC_GLB_STREAM_MSG | SOF_IPC_STREAM_PCM_PARAMS;
+ pcm.comp_id = spcm->stream[substream->stream].comp_id;
+ pcm.params.hdr.size = sizeof(pcm.params);
+ pcm.params.buffer.phy_addr =
+ spcm->stream[substream->stream].page_table.addr;
+ pcm.params.buffer.size = runtime->dma_bytes;
+ pcm.params.direction = substream->stream;
+ pcm.params.sample_valid_bytes = params_width(params) >> 3;
+ pcm.params.buffer_fmt = SOF_IPC_BUFFER_INTERLEAVED;
+ pcm.params.rate = params_rate(params);
+ pcm.params.channels = params_channels(params);
+ pcm.params.host_period_bytes = params_period_bytes(params);
+
+ /* container size */
+ ret = snd_pcm_format_physical_width(params_format(params));
+ if (ret < 0)
+ return ret;
+ pcm.params.sample_container_bytes = ret >> 3;
+
+ /* format */
+ switch (params_format(params)) {
+ case SNDRV_PCM_FORMAT_S16:
+ pcm.params.frame_fmt = SOF_IPC_FRAME_S16_LE;
+ break;
+ case SNDRV_PCM_FORMAT_S24:
+ pcm.params.frame_fmt = SOF_IPC_FRAME_S24_4LE;
+ break;
+ case SNDRV_PCM_FORMAT_S32:
+ pcm.params.frame_fmt = SOF_IPC_FRAME_S32_LE;
+ break;
+ case SNDRV_PCM_FORMAT_FLOAT:
+ pcm.params.frame_fmt = SOF_IPC_FRAME_FLOAT;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ /* firmware already configured host stream */
+ ret = snd_sof_pcm_platform_hw_params(sdev,
+ substream,
+ params,
+ &pcm.params);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: platform hw params failed\n");
+ return ret;
+ }
+
+ dev_dbg(sdev->dev, "stream_tag %d", pcm.params.stream_tag);
+
+ /* send IPC to the DSP */
+ ret = sof_ipc_tx_message(sdev->ipc, pcm.hdr.cmd, &pcm, sizeof(pcm),
+ &ipc_params_reply, sizeof(ipc_params_reply));
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: hw params ipc failed for stream %d\n",
+ pcm.params.stream_tag);
+ return ret;
+ }
+
+ ret = sof_pcm_dsp_params(spcm, substream, &ipc_params_reply);
+ if (ret < 0)
+ return ret;
+
+ /* save pcm hw_params */
+ memcpy(&spcm->params[substream->stream], params, sizeof(*params));
+
+ INIT_WORK(&spcm->stream[substream->stream].period_elapsed_work,
+ sof_pcm_period_elapsed_work);
+
+ return ret;
+}
+
+static int sof_pcm_hw_free(struct snd_pcm_substream *substream)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pcm *spcm;
+ struct sof_ipc_stream stream;
+ struct sof_ipc_reply reply;
+ int ret;
+
+ /* nothing to do for BE */
+ if (rtd->dai_link->no_pcm)
+ return 0;
+
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm)
+ return -EINVAL;
+
+ dev_dbg(sdev->dev, "pcm: free stream %d dir %d\n", spcm->pcm.pcm_id,
+ substream->stream);
+
+ stream.hdr.size = sizeof(stream);
+ stream.hdr.cmd = SOF_IPC_GLB_STREAM_MSG | SOF_IPC_STREAM_PCM_FREE;
+ stream.comp_id = spcm->stream[substream->stream].comp_id;
+
+ /* send IPC to the DSP */
+ ret = sof_ipc_tx_message(sdev->ipc, stream.hdr.cmd, &stream,
+ sizeof(stream), &reply, sizeof(reply));
+
+ snd_pcm_lib_free_pages(substream);
+
+ cancel_work_sync(&spcm->stream[substream->stream].period_elapsed_work);
+
+ return ret;
+}
+
+static int sof_pcm_prepare(struct snd_pcm_substream *substream)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pcm *spcm;
+ int ret;
+
+ /* nothing to do for BE */
+ if (rtd->dai_link->no_pcm)
+ return 0;
+
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm)
+ return -EINVAL;
+
+ /*
+ * check if hw_params needs to be set-up again.
+ * This is only needed when resuming from system sleep.
+ */
+ if (!spcm->hw_params_upon_resume[substream->stream])
+ return 0;
+
+ dev_dbg(sdev->dev, "pcm: prepare stream %d dir %d\n", spcm->pcm.pcm_id,
+ substream->stream);
+
+ /* set hw_params */
+ ret = sof_pcm_hw_params(substream, &spcm->params[substream->stream]);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: set pcm hw_params after resume\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+/*
+ * FE dai link trigger actions are always executed in non-atomic context because
+ * they involve IPC's.
+ */
+static int sof_pcm_trigger(struct snd_pcm_substream *substream, int cmd)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pcm *spcm;
+ struct sof_ipc_stream stream;
+ struct sof_ipc_reply reply;
+ int ret;
+
+ /* nothing to do for BE */
+ if (rtd->dai_link->no_pcm)
+ return 0;
+
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm)
+ return -EINVAL;
+
+ dev_dbg(sdev->dev, "pcm: trigger stream %d dir %d cmd %d\n",
+ spcm->pcm.pcm_id, substream->stream, cmd);
+
+ stream.hdr.size = sizeof(stream);
+ stream.hdr.cmd = SOF_IPC_GLB_STREAM_MSG;
+ stream.comp_id = spcm->stream[substream->stream].comp_id;
+
+ switch (cmd) {
+ case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
+ stream.hdr.cmd |= SOF_IPC_STREAM_TRIG_PAUSE;
+ break;
+ case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
+ stream.hdr.cmd |= SOF_IPC_STREAM_TRIG_RELEASE;
+ break;
+ case SNDRV_PCM_TRIGGER_RESUME:
+ /* set up hw_params */
+ ret = sof_pcm_prepare(substream);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to set up hw_params upon resume\n");
+ return ret;
+ }
+
+ /* fallthrough */
+ case SNDRV_PCM_TRIGGER_START:
+ stream.hdr.cmd |= SOF_IPC_STREAM_TRIG_START;
+ break;
+ case SNDRV_PCM_TRIGGER_SUSPEND:
+ case SNDRV_PCM_TRIGGER_STOP:
+ stream.hdr.cmd |= SOF_IPC_STREAM_TRIG_STOP;
+ break;
+ default:
+ dev_err(sdev->dev, "error: unhandled trigger cmd %d\n", cmd);
+ return -EINVAL;
+ }
+
+ snd_sof_pcm_platform_trigger(sdev, substream, cmd);
+
+ /* send IPC to the DSP */
+ ret = sof_ipc_tx_message(sdev->ipc, stream.hdr.cmd, &stream,
+ sizeof(stream), &reply, sizeof(reply));
+
+ if (ret < 0 || cmd != SNDRV_PCM_TRIGGER_SUSPEND)
+ return ret;
+
+ /*
+ * The hw_free op is usually called when the pcm stream is closed.
+ * Since the stream is not closed during suspend, the DSP needs to be
+ * notified explicitly to free pcm to prevent errors upon resume.
+ */
+ stream.hdr.size = sizeof(stream);
+ stream.hdr.cmd = SOF_IPC_GLB_STREAM_MSG | SOF_IPC_STREAM_PCM_FREE;
+ stream.comp_id = spcm->stream[substream->stream].comp_id;
+
+ /* send IPC to the DSP */
+ return sof_ipc_tx_message(sdev->ipc, stream.hdr.cmd, &stream,
+ sizeof(stream), &reply, sizeof(reply));
+}
+
+static snd_pcm_uframes_t sof_pcm_pointer(struct snd_pcm_substream *substream)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pcm *spcm;
+ snd_pcm_uframes_t host, dai;
+
+ /* nothing to do for BE */
+ if (rtd->dai_link->no_pcm)
+ return 0;
+
+ /* use dsp ops pointer callback directly if set */
+ if (sof_ops(sdev)->pcm_pointer)
+ return sof_ops(sdev)->pcm_pointer(sdev, substream);
+
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm)
+ return -EINVAL;
+
+ /* read position from DSP */
+ host = bytes_to_frames(substream->runtime,
+ spcm->stream[substream->stream].posn.host_posn);
+ dai = bytes_to_frames(substream->runtime,
+ spcm->stream[substream->stream].posn.dai_posn);
+
+ dev_dbg(sdev->dev, "PCM: stream %d dir %d DMA position %lu DAI position %lu\n",
+ spcm->pcm.pcm_id, substream->stream, host, dai);
+
+ return host;
+}
+
+static int sof_pcm_open(struct snd_pcm_substream *substream)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_pcm_runtime *runtime = substream->runtime;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pcm *spcm;
+ struct snd_soc_tplg_stream_caps *caps;
+ int ret;
+ int err;
+
+ /* nothing to do for BE */
+ if (rtd->dai_link->no_pcm)
+ return 0;
+
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm)
+ return -EINVAL;
+
+ dev_dbg(sdev->dev, "pcm: open stream %d dir %d\n", spcm->pcm.pcm_id,
+ substream->stream);
+
+ /* clear hw_params_upon_resume flag */
+ spcm->hw_params_upon_resume[substream->stream] = 0;
+
+ caps = &spcm->pcm.caps[substream->stream];
+
+ ret = pm_runtime_get_sync(sdev->dev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: pcm open failed to resume %d\n",
+ ret);
+ pm_runtime_put_noidle(sdev->dev);
+ return ret;
+ }
+
+ /* set any runtime constraints based on topology */
+ snd_pcm_hw_constraint_step(substream->runtime, 0,
+ SNDRV_PCM_HW_PARAM_BUFFER_BYTES,
+ le32_to_cpu(caps->period_size_min));
+ snd_pcm_hw_constraint_step(substream->runtime, 0,
+ SNDRV_PCM_HW_PARAM_PERIOD_BYTES,
+ le32_to_cpu(caps->period_size_min));
+
+ /* set runtime config */
+ runtime->hw.info = SNDRV_PCM_INFO_MMAP |
+ SNDRV_PCM_INFO_MMAP_VALID |
+ SNDRV_PCM_INFO_INTERLEAVED |
+ SNDRV_PCM_INFO_PAUSE |
+ SNDRV_PCM_INFO_NO_PERIOD_WAKEUP;
+ runtime->hw.formats = le64_to_cpu(caps->formats);
+ runtime->hw.period_bytes_min = le32_to_cpu(caps->period_size_min);
+ runtime->hw.period_bytes_max = le32_to_cpu(caps->period_size_max);
+ runtime->hw.periods_min = le32_to_cpu(caps->periods_min);
+ runtime->hw.periods_max = le32_to_cpu(caps->periods_max);
+
+ /*
+ * caps->buffer_size_min is not used since the
+ * snd_pcm_hardware structure only defines buffer_bytes_max
+ */
+ runtime->hw.buffer_bytes_max = le32_to_cpu(caps->buffer_size_max);
+
+ dev_dbg(sdev->dev, "period min %zd max %zd bytes\n",
+ runtime->hw.period_bytes_min,
+ runtime->hw.period_bytes_max);
+ dev_dbg(sdev->dev, "period count %d max %d\n",
+ runtime->hw.periods_min,
+ runtime->hw.periods_max);
+ dev_dbg(sdev->dev, "buffer max %zd bytes\n",
+ runtime->hw.buffer_bytes_max);
+
+ /* set wait time - TODO: come from topology */
+ substream->wait_time = 500;
+
+ spcm->stream[substream->stream].posn.host_posn = 0;
+ spcm->stream[substream->stream].posn.dai_posn = 0;
+ spcm->stream[substream->stream].substream = substream;
+
+ ret = snd_sof_pcm_platform_open(sdev, substream);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: pcm open failed %d\n",
+ ret);
+
+ pm_runtime_mark_last_busy(sdev->dev);
+
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err(sdev->dev, "error: pcm close failed to idle %d\n",
+ err);
+ }
+
+ return ret;
+}
+
+static int sof_pcm_close(struct snd_pcm_substream *substream)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pcm *spcm;
+ int err;
+
+ /* nothing to do for BE */
+ if (rtd->dai_link->no_pcm)
+ return 0;
+
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm)
+ return -EINVAL;
+
+ dev_dbg(sdev->dev, "pcm: close stream %d dir %d\n", spcm->pcm.pcm_id,
+ substream->stream);
+
+ err = snd_sof_pcm_platform_close(sdev, substream);
+ if (err < 0) {
+ dev_err(sdev->dev, "error: pcm close failed %d\n",
+ err);
+ /*
+ * keep going, no point in preventing the close
+ * from happening
+ */
+ }
+
+ pm_runtime_mark_last_busy(sdev->dev);
+
+ err = pm_runtime_put_autosuspend(sdev->dev);
+ if (err < 0)
+ dev_err(sdev->dev, "error: pcm close failed to idle %d\n",
+ err);
+
+ return 0;
+}
+
+static struct snd_pcm_ops sof_pcm_ops = {
+ .open = sof_pcm_open,
+ .close = sof_pcm_close,
+ .ioctl = snd_pcm_lib_ioctl,
+ .hw_params = sof_pcm_hw_params,
+ .prepare = sof_pcm_prepare,
+ .hw_free = sof_pcm_hw_free,
+ .trigger = sof_pcm_trigger,
+ .pointer = sof_pcm_pointer,
+ .page = snd_pcm_sgbuf_ops_page,
+};
+
+/*
+ * Pre-allocate playback/capture audio buffer pages.
+ * no need to explicitly release memory preallocated by sof_pcm_new in pcm_free
+ * snd_pcm_lib_preallocate_free_for_all() is called by the core.
+ */
+static int sof_pcm_new(struct snd_soc_pcm_runtime *rtd)
+{
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pcm *spcm;
+ struct snd_pcm *pcm = rtd->pcm;
+ struct snd_soc_tplg_stream_caps *caps;
+ int stream = SNDRV_PCM_STREAM_PLAYBACK;
+
+ /* find SOF PCM for this RTD */
+ spcm = snd_sof_find_spcm_dai(sdev, rtd);
+ if (!spcm) {
+ dev_warn(sdev->dev, "warn: can't find PCM with DAI ID %d\n",
+ rtd->dai_link->id);
+ return 0;
+ }
+
+ dev_dbg(sdev->dev, "creating new PCM %s\n", spcm->pcm.pcm_name);
+
+ /* do we need to pre-allocate playback audio buffer pages */
+ if (!spcm->pcm.playback)
+ goto capture;
+
+ caps = &spcm->pcm.caps[stream];
+
+ /* pre-allocate playback audio buffer pages */
+ dev_dbg(sdev->dev, "spcm: allocate %s playback DMA buffer size 0x%x max 0x%x\n",
+ caps->name, caps->buffer_size_min, caps->buffer_size_max);
+
+ snd_pcm_lib_preallocate_pages(pcm->streams[stream].substream,
+ SNDRV_DMA_TYPE_DEV_SG, sdev->dev,
+ le32_to_cpu(caps->buffer_size_min),
+ le32_to_cpu(caps->buffer_size_max));
+capture:
+ stream = SNDRV_PCM_STREAM_CAPTURE;
+
+ /* do we need to pre-allocate capture audio buffer pages */
+ if (!spcm->pcm.capture)
+ return 0;
+
+ caps = &spcm->pcm.caps[stream];
+
+ /* pre-allocate capture audio buffer pages */
+ dev_dbg(sdev->dev, "spcm: allocate %s capture DMA buffer size 0x%x max 0x%x\n",
+ caps->name, caps->buffer_size_min, caps->buffer_size_max);
+
+ snd_pcm_lib_preallocate_pages(pcm->streams[stream].substream,
+ SNDRV_DMA_TYPE_DEV_SG, sdev->dev,
+ le32_to_cpu(caps->buffer_size_min),
+ le32_to_cpu(caps->buffer_size_max));
+
+ return 0;
+}
+
+/* fixup the BE DAI link to match any values from topology */
+static int sof_pcm_dai_link_fixup(struct snd_soc_pcm_runtime *rtd,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_interval *rate = hw_param_interval(params,
+ SNDRV_PCM_HW_PARAM_RATE);
+ struct snd_interval *channels = hw_param_interval(params,
+ SNDRV_PCM_HW_PARAM_CHANNELS);
+ struct snd_mask *fmt = hw_param_mask(params, SNDRV_PCM_HW_PARAM_FORMAT);
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_dai *dai =
+ snd_sof_find_dai(sdev, (char *)rtd->dai_link->name);
+
+ /* no topology exists for this BE, try a common configuration */
+ if (!dai) {
+ dev_warn(sdev->dev, "warning: no topology found for BE DAI %s config\n",
+ rtd->dai_link->name);
+
+ /* set 48k, stereo, 16bits by default */
+ rate->min = 48000;
+ rate->max = 48000;
+
+ channels->min = 2;
+ channels->max = 2;
+
+ snd_mask_none(fmt);
+ snd_mask_set_format(fmt, SNDRV_PCM_FORMAT_S16_LE);
+
+ return 0;
+ }
+
+ /* read format from topology */
+ snd_mask_none(fmt);
+
+ switch (dai->comp_dai.config.frame_fmt) {
+ case SOF_IPC_FRAME_S16_LE:
+ snd_mask_set_format(fmt, SNDRV_PCM_FORMAT_S16_LE);
+ break;
+ case SOF_IPC_FRAME_S24_4LE:
+ snd_mask_set_format(fmt, SNDRV_PCM_FORMAT_S24_LE);
+ break;
+ case SOF_IPC_FRAME_S32_LE:
+ snd_mask_set_format(fmt, SNDRV_PCM_FORMAT_S32_LE);
+ break;
+ default:
+ dev_err(sdev->dev, "error: No available DAI format!\n");
+ return -EINVAL;
+ }
+
+ /* read rate and channels from topology */
+ switch (dai->dai_config->type) {
+ case SOF_DAI_INTEL_SSP:
+ rate->min = dai->dai_config->ssp.fsync_rate;
+ rate->max = dai->dai_config->ssp.fsync_rate;
+ channels->min = dai->dai_config->ssp.tdm_slots;
+ channels->max = dai->dai_config->ssp.tdm_slots;
+
+ dev_dbg(sdev->dev,
+ "rate_min: %d rate_max: %d\n", rate->min, rate->max);
+ dev_dbg(sdev->dev,
+ "channels_min: %d channels_max: %d\n",
+ channels->min, channels->max);
+
+ break;
+ case SOF_DAI_INTEL_DMIC:
+ /* DMIC only supports 16 or 32 bit formats */
+ if (dai->comp_dai.config.frame_fmt == SOF_IPC_FRAME_S24_4LE) {
+ dev_err(sdev->dev,
+ "error: invalid fmt %d for DAI type %d\n",
+ dai->comp_dai.config.frame_fmt,
+ dai->dai_config->type);
+ }
+ break;
+ case SOF_DAI_INTEL_HDA:
+ /* do nothing for HDA dai_link */
+ break;
+ default:
+ dev_err(sdev->dev, "error: invalid DAI type %d\n",
+ dai->dai_config->type);
+ break;
+ }
+
+ return 0;
+}
+
+static int sof_pcm_probe(struct snd_soc_component *component)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(component);
+ struct snd_sof_pdata *plat_data = sdev->pdata;
+ const char *tplg_filename;
+ int ret;
+
+ /* load the default topology */
+ sdev->component = component;
+
+ tplg_filename = devm_kasprintf(sdev->dev, GFP_KERNEL,
+ "%s/%s",
+ plat_data->tplg_filename_prefix,
+ plat_data->tplg_filename);
+ if (!tplg_filename)
+ return -ENOMEM;
+
+ ret = snd_sof_load_topology(sdev, tplg_filename);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to load DSP topology %d\n",
+ ret);
+ return ret;
+ }
+
+ /*
+ * Some platforms in SOF, ex: BYT, may not have their platform PM
+ * callbacks set. Increment the usage count so as to
+ * prevent the device from entering runtime suspend.
+ */
+ if (!sof_ops(sdev)->runtime_suspend || !sof_ops(sdev)->runtime_resume)
+ pm_runtime_get_noresume(sdev->dev);
+
+ return ret;
+}
+
+static void sof_pcm_remove(struct snd_soc_component *component)
+{
+ /* remove topology */
+ snd_soc_tplg_component_remove(component, SND_SOC_TPLG_INDEX_ALL);
+}
+
+void snd_sof_new_platform_drv(struct snd_sof_dev *sdev)
+{
+ struct snd_soc_component_driver *pd = &sdev->plat_drv;
+ struct snd_sof_pdata *plat_data = sdev->pdata;
+ const char *drv_name;
+
+ drv_name = plat_data->machine->drv_name;
+
+ pd->name = "sof-audio-component";
+ pd->probe = sof_pcm_probe;
+ pd->remove = sof_pcm_remove;
+ pd->ops = &sof_pcm_ops;
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_COMPRESS)
+ pd->compr_ops = &sof_compressed_ops;
+#endif
+ pd->pcm_new = sof_pcm_new;
+ pd->ignore_machine = drv_name;
+ pd->be_hw_params_fixup = sof_pcm_dai_link_fixup;
+ pd->be_pcm_base = SOF_BE_PCM_BASE;
+ pd->use_dai_pcm_id = true;
+ pd->topology_name_prefix = "sof";
+
+ /* increment module refcount when a pcm is opened */
+ pd->module_get_upon_open = 1;
+}
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+#include "ops.h"
+#include "sof-priv.h"
+
+static int sof_restore_kcontrols(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_control *scontrol;
+ int ipc_cmd, ctrl_type;
+ int ret = 0;
+
+ /* restore kcontrol values */
+ list_for_each_entry(scontrol, &sdev->kcontrol_list, list) {
+ /* reset readback offset for scontrol after resuming */
+ scontrol->readback_offset = 0;
+
+ /* notify DSP of kcontrol values */
+ switch (scontrol->cmd) {
+ case SOF_CTRL_CMD_VOLUME:
+ case SOF_CTRL_CMD_ENUM:
+ case SOF_CTRL_CMD_SWITCH:
+ ipc_cmd = SOF_IPC_COMP_SET_VALUE;
+ ctrl_type = SOF_CTRL_TYPE_VALUE_CHAN_SET;
+ ret = snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ ipc_cmd, ctrl_type,
+ scontrol->cmd,
+ true);
+ break;
+ case SOF_CTRL_CMD_BINARY:
+ ipc_cmd = SOF_IPC_COMP_SET_DATA;
+ ctrl_type = SOF_CTRL_TYPE_DATA_SET;
+ ret = snd_sof_ipc_set_get_comp_data(sdev->ipc, scontrol,
+ ipc_cmd, ctrl_type,
+ scontrol->cmd,
+ true);
+ break;
+
+ default:
+ break;
+ }
+
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed kcontrol value set for widget: %d\n",
+ scontrol->comp_id);
+
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int sof_restore_pipelines(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_widget *swidget;
+ struct snd_sof_route *sroute;
+ struct sof_ipc_pipe_new *pipeline;
+ struct snd_sof_dai *dai;
+ struct sof_ipc_comp_dai *comp_dai;
+ struct sof_ipc_cmd_hdr *hdr;
+ int ret;
+
+ /* restore pipeline components */
+ list_for_each_entry_reverse(swidget, &sdev->widget_list, list) {
+ struct sof_ipc_comp_reply r;
+
+ /* skip if there is no private data */
+ if (!swidget->private)
+ continue;
+
+ switch (swidget->id) {
+ case snd_soc_dapm_dai_in:
+ case snd_soc_dapm_dai_out:
+ dai = swidget->private;
+ comp_dai = &dai->comp_dai;
+ ret = sof_ipc_tx_message(sdev->ipc,
+ comp_dai->comp.hdr.cmd,
+ comp_dai, sizeof(*comp_dai),
+ &r, sizeof(r));
+ break;
+ case snd_soc_dapm_scheduler:
+
+ /*
+ * During suspend, all DSP cores are powered off.
+ * Therefore upon resume, create the pipeline comp
+ * and power up the core that the pipeline is
+ * scheduled on.
+ */
+ pipeline = swidget->private;
+ ret = sof_load_pipeline_ipc(sdev, pipeline, &r);
+ break;
+ default:
+ hdr = swidget->private;
+ ret = sof_ipc_tx_message(sdev->ipc, hdr->cmd,
+ swidget->private, hdr->size,
+ &r, sizeof(r));
+ break;
+ }
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to load widget type %d with ID: %d\n",
+ swidget->widget->id, swidget->comp_id);
+
+ return ret;
+ }
+ }
+
+ /* restore pipeline connections */
+ list_for_each_entry_reverse(sroute, &sdev->route_list, list) {
+ struct sof_ipc_pipe_comp_connect *connect;
+ struct sof_ipc_reply reply;
+
+ /* skip if there's no private data */
+ if (!sroute->private)
+ continue;
+
+ connect = sroute->private;
+
+ /* send ipc */
+ ret = sof_ipc_tx_message(sdev->ipc,
+ connect->hdr.cmd,
+ connect, sizeof(*connect),
+ &reply, sizeof(reply));
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to load route sink %s control %s source %s\n",
+ sroute->route->sink,
+ sroute->route->control ? sroute->route->control
+ : "none",
+ sroute->route->source);
+
+ return ret;
+ }
+ }
+
+ /* restore dai links */
+ list_for_each_entry_reverse(dai, &sdev->dai_list, list) {
+ struct sof_ipc_reply reply;
+ struct sof_ipc_dai_config *config = dai->dai_config;
+
+ if (!config) {
+ dev_err(sdev->dev, "error: no config for DAI %s\n",
+ dai->name);
+ continue;
+ }
+
+ ret = sof_ipc_tx_message(sdev->ipc,
+ config->hdr.cmd, config,
+ config->hdr.size,
+ &reply, sizeof(reply));
+
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to set dai config for %s\n",
+ dai->name);
+
+ return ret;
+ }
+ }
+
+ /* complete pipeline */
+ list_for_each_entry(swidget, &sdev->widget_list, list) {
+ switch (swidget->id) {
+ case snd_soc_dapm_scheduler:
+ swidget->complete =
+ snd_sof_complete_pipeline(sdev, swidget);
+ break;
+ default:
+ break;
+ }
+ }
+
+ /* restore pipeline kcontrols */
+ ret = sof_restore_kcontrols(sdev);
+ if (ret < 0)
+ dev_err(sdev->dev,
+ "error: restoring kcontrols after resume\n");
+
+ return ret;
+}
+
+static int sof_send_pm_ipc(struct snd_sof_dev *sdev, int cmd)
+{
+ struct sof_ipc_pm_ctx pm_ctx;
+ struct sof_ipc_reply reply;
+
+ memset(&pm_ctx, 0, sizeof(pm_ctx));
+
+ /* configure ctx save ipc message */
+ pm_ctx.hdr.size = sizeof(pm_ctx);
+ pm_ctx.hdr.cmd = SOF_IPC_GLB_PM_MSG | cmd;
+
+ /* send ctx save ipc to dsp */
+ return sof_ipc_tx_message(sdev->ipc, pm_ctx.hdr.cmd, &pm_ctx,
+ sizeof(pm_ctx), &reply, sizeof(reply));
+}
+
+static void sof_set_hw_params_upon_resume(struct snd_sof_dev *sdev)
+{
+ struct snd_pcm_substream *substream;
+ struct snd_sof_pcm *spcm;
+ snd_pcm_state_t state;
+ int dir;
+
+ /*
+ * SOF requires hw_params to be set-up internally upon resume.
+ * So, set the flag to indicate this for those streams that
+ * have been suspended.
+ */
+ list_for_each_entry(spcm, &sdev->pcm_list, list) {
+ for (dir = 0; dir <= SNDRV_PCM_STREAM_CAPTURE; dir++) {
+ substream = spcm->stream[dir].substream;
+ if (!substream || !substream->runtime)
+ continue;
+
+ state = substream->runtime->status->state;
+ if (state == SNDRV_PCM_STATE_SUSPENDED)
+ spcm->hw_params_upon_resume[dir] = 1;
+ }
+ }
+
+ /* set internal flag for BE */
+ snd_sof_dsp_hw_params_upon_resume(sdev);
+}
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_DEBUG_ENABLE_DEBUGFS_CACHE)
+static void sof_cache_debugfs(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_dfsentry *dfse;
+
+ list_for_each_entry(dfse, &sdev->dfsentry_list, list) {
+
+ /* nothing to do if debugfs buffer is not IO mem */
+ if (dfse->type == SOF_DFSENTRY_TYPE_BUF)
+ continue;
+
+ /* cache memory that is only accessible in D0 */
+ if (dfse->access_type == SOF_DEBUGFS_ACCESS_D0_ONLY)
+ memcpy_fromio(dfse->cache_buf, dfse->io_mem,
+ dfse->size);
+ }
+}
+#endif
+
+static int sof_resume(struct device *dev, bool runtime_resume)
+{
+ struct snd_sof_dev *sdev = dev_get_drvdata(dev);
+ int ret;
+
+ /* do nothing if dsp resume callbacks are not set */
+ if (!sof_ops(sdev)->resume || !sof_ops(sdev)->runtime_resume)
+ return 0;
+
+ /*
+ * if the runtime_resume flag is set, call the runtime_resume routine
+ * or else call the system resume routine
+ */
+ if (runtime_resume)
+ ret = snd_sof_dsp_runtime_resume(sdev);
+ else
+ ret = snd_sof_dsp_resume(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to power up DSP after resume\n");
+ return ret;
+ }
+
+ /* load the firmware */
+ ret = snd_sof_load_firmware(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to load DSP firmware after resume %d\n",
+ ret);
+ return ret;
+ }
+
+ /* boot the firmware */
+ ret = snd_sof_run_firmware(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to boot DSP firmware after resume %d\n",
+ ret);
+ return ret;
+ }
+
+ /* resume DMA trace, only need send ipc */
+ ret = snd_sof_init_trace_ipc(sdev);
+ if (ret < 0) {
+ /* non fatal */
+ dev_warn(sdev->dev,
+ "warning: failed to init trace after resume %d\n",
+ ret);
+ }
+
+ /* restore pipelines */
+ ret = sof_restore_pipelines(sdev);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to restore pipeline after resume %d\n",
+ ret);
+ return ret;
+ }
+
+ /* notify DSP of system resume */
+ ret = sof_send_pm_ipc(sdev, SOF_IPC_PM_CTX_RESTORE);
+ if (ret < 0)
+ dev_err(sdev->dev,
+ "error: ctx_restore ipc error during resume %d\n",
+ ret);
+
+ return ret;
+}
+
+static int sof_suspend(struct device *dev, bool runtime_suspend)
+{
+ struct snd_sof_dev *sdev = dev_get_drvdata(dev);
+ int ret;
+
+ /* do nothing if dsp suspend callback is not set */
+ if (!sof_ops(sdev)->suspend)
+ return 0;
+
+ /* release trace */
+ snd_sof_release_trace(sdev);
+
+ /* set restore_stream for all streams during system suspend */
+ if (!runtime_suspend)
+ sof_set_hw_params_upon_resume(sdev);
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_DEBUG_ENABLE_DEBUGFS_CACHE)
+ /* cache debugfs contents during runtime suspend */
+ if (runtime_suspend)
+ sof_cache_debugfs(sdev);
+#endif
+ /* notify DSP of upcoming power down */
+ ret = sof_send_pm_ipc(sdev, SOF_IPC_PM_CTX_SAVE);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: ctx_save ipc error during suspend %d\n",
+ ret);
+ return ret;
+ }
+
+ /* power down all DSP cores */
+ if (runtime_suspend)
+ ret = snd_sof_dsp_runtime_suspend(sdev, 0);
+ else
+ ret = snd_sof_dsp_suspend(sdev, 0);
+ if (ret < 0)
+ dev_err(sdev->dev,
+ "error: failed to power down DSP during suspend %d\n",
+ ret);
+
+ return ret;
+}
+
+int snd_sof_runtime_suspend(struct device *dev)
+{
+ return sof_suspend(dev, true);
+}
+EXPORT_SYMBOL(snd_sof_runtime_suspend);
+
+int snd_sof_runtime_resume(struct device *dev)
+{
+ return sof_resume(dev, true);
+}
+EXPORT_SYMBOL(snd_sof_runtime_resume);
+
+int snd_sof_resume(struct device *dev)
+{
+ return sof_resume(dev, false);
+}
+EXPORT_SYMBOL(snd_sof_resume);
+
+int snd_sof_suspend(struct device *dev)
+{
+ return sof_suspend(dev, false);
+}
+EXPORT_SYMBOL(snd_sof_suspend);
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+#include <linux/acpi.h>
+#include <linux/firmware.h>
+#include <linux/module.h>
+#include <linux/pm_runtime.h>
+#include <sound/soc-acpi.h>
+#include <sound/soc-acpi-intel-match.h>
+#include <sound/sof.h>
+#ifdef CONFIG_X86
+#include <asm/iosf_mbi.h>
+#endif
+
+#include "ops.h"
+
+/* platform specific devices */
+#include "intel/shim.h"
+
+static char *fw_path;
+module_param(fw_path, charp, 0444);
+MODULE_PARM_DESC(fw_path, "alternate path for SOF firmware.");
+
+static char *tplg_path;
+module_param(tplg_path, charp, 0444);
+MODULE_PARM_DESC(tplg_path, "alternate path for SOF topology.");
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HASWELL)
+static const struct sof_dev_desc sof_acpi_haswell_desc = {
+ .machines = snd_soc_acpi_intel_haswell_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = 1,
+ .resindex_imr_base = -1,
+ .irqindex_host_ipc = 0,
+ .chip_info = &hsw_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-hsw.ri",
+ .nocodec_tplg_filename = "sof-hsw-nocodec.tplg",
+ .ops = &sof_hsw_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_BROADWELL)
+static const struct sof_dev_desc sof_acpi_broadwell_desc = {
+ .machines = snd_soc_acpi_intel_broadwell_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = 1,
+ .resindex_imr_base = -1,
+ .irqindex_host_ipc = 0,
+ .chip_info = &bdw_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-bdw.ri",
+ .nocodec_tplg_filename = "sof-bdw-nocodec.tplg",
+ .ops = &sof_bdw_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_BAYTRAIL)
+
+/* BYTCR uses different IRQ index */
+static const struct sof_dev_desc sof_acpi_baytrailcr_desc = {
+ .machines = snd_soc_acpi_intel_baytrail_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = 1,
+ .resindex_imr_base = 2,
+ .irqindex_host_ipc = 0,
+ .chip_info = &byt_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-byt.ri",
+ .nocodec_tplg_filename = "sof-byt-nocodec.tplg",
+ .ops = &sof_byt_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+
+static const struct sof_dev_desc sof_acpi_baytrail_desc = {
+ .machines = snd_soc_acpi_intel_baytrail_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = 1,
+ .resindex_imr_base = 2,
+ .irqindex_host_ipc = 5,
+ .chip_info = &byt_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-byt.ri",
+ .nocodec_tplg_filename = "sof-byt-nocodec.tplg",
+ .ops = &sof_byt_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+
+#ifdef CONFIG_X86 /* TODO: move this to common helper */
+
+static bool is_byt_cr(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ int status;
+
+ if (iosf_mbi_available()) {
+ u32 bios_status;
+ status = iosf_mbi_read(BT_MBI_UNIT_PMC, /* 0x04 PUNIT */
+ MBI_REG_READ, /* 0x10 */
+ 0x006, /* BIOS_CONFIG */
+ &bios_status);
+
+ if (status) {
+ dev_err(dev, "could not read PUNIT BIOS_CONFIG\n");
+ } else {
+ /* bits 26:27 mirror PMIC options */
+ bios_status = (bios_status >> 26) & 3;
+
+ if (bios_status == 1 || bios_status == 3) {
+ dev_info(dev, "Detected Baytrail-CR platform\n");
+ return true;
+ }
+
+ dev_info(dev, "BYT-CR not detected\n");
+ }
+ } else {
+ dev_info(dev, "IOSF_MBI not available, no BYT-CR detection\n");
+ }
+
+ if (platform_get_resource(pdev, IORESOURCE_IRQ, 5) == NULL) {
+ /*
+ * Some devices detected as BYT-T have only a single IRQ listed,
+ * causing platform_get_irq with index 5 to return -ENXIO.
+ * The correct IRQ in this case is at index 0, as on BYT-CR.
+ */
+ dev_info(dev, "Falling back to Baytrail-CR platform\n");
+ return true;
+ }
+
+ return false;
+}
+#else
+static int is_byt_cr(struct platform_device *pdev)
+{
+ return 0;
+}
+#endif
+
+static const struct sof_dev_desc sof_acpi_cherrytrail_desc = {
+ .machines = snd_soc_acpi_intel_cherrytrail_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = 1,
+ .resindex_imr_base = 2,
+ .irqindex_host_ipc = 5,
+ .chip_info = &cht_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-cht.ri",
+ .nocodec_tplg_filename = "sof-cht-nocodec.tplg",
+ .ops = &sof_cht_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+
+#endif
+
+static const struct dev_pm_ops sof_acpi_pm = {
+ SET_SYSTEM_SLEEP_PM_OPS(snd_sof_suspend, snd_sof_resume)
+ SET_RUNTIME_PM_OPS(snd_sof_runtime_suspend, snd_sof_runtime_resume,
+ NULL)
+};
+
+static void sof_acpi_probe_complete(struct device *dev)
+{
+ /* allow runtime_pm */
+ pm_runtime_set_autosuspend_delay(dev, SND_SOF_SUSPEND_DELAY_MS);
+ pm_runtime_use_autosuspend(dev);
+ pm_runtime_enable(dev);
+}
+
+static int sof_acpi_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ const struct sof_dev_desc *desc;
+ struct snd_soc_acpi_mach *mach;
+ struct snd_sof_pdata *sof_pdata;
+ const struct snd_sof_dsp_ops *ops;
+ int ret;
+
+ dev_dbg(&pdev->dev, "ACPI DSP detected");
+
+ sof_pdata = devm_kzalloc(dev, sizeof(*sof_pdata), GFP_KERNEL);
+ if (!sof_pdata)
+ return -ENOMEM;
+
+ desc = device_get_match_data(dev);
+ if (!desc)
+ return -ENODEV;
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_BAYTRAIL)
+ if (desc == &sof_acpi_baytrail_desc && is_byt_cr(pdev))
+ desc = &sof_acpi_baytrailcr_desc;
+#endif
+
+ /* get ops for platform */
+ ops = desc->ops;
+ if (!ops) {
+ dev_err(dev, "error: no matching ACPI descriptor ops\n");
+ return -ENODEV;
+ }
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_FORCE_NOCODEC_MODE)
+ /* force nocodec mode */
+ dev_warn(dev, "Force to use nocodec mode\n");
+ mach = devm_kzalloc(dev, sizeof(*mach), GFP_KERNEL);
+ if (!mach)
+ return -ENOMEM;
+ ret = sof_nocodec_setup(dev, sof_pdata, mach, desc, ops);
+ if (ret < 0)
+ return ret;
+#else
+ /* find machine */
+ mach = snd_soc_acpi_find_machine(desc->machines);
+ if (!mach) {
+ dev_warn(dev, "warning: No matching ASoC machine driver found\n");
+ } else {
+ sof_pdata->fw_filename = mach->sof_fw_filename;
+ sof_pdata->tplg_filename = mach->sof_tplg_filename;
+ }
+#endif
+
+ if (mach) {
+ mach->mach_params.platform = dev_name(dev);
+ mach->mach_params.acpi_ipc_irq_index = desc->irqindex_host_ipc;
+ }
+
+ sof_pdata->machine = mach;
+ sof_pdata->desc = desc;
+ sof_pdata->dev = &pdev->dev;
+ sof_pdata->platform = dev_name(dev);
+
+ /* alternate fw and tplg filenames ? */
+ if (fw_path)
+ sof_pdata->fw_filename_prefix = fw_path;
+ else
+ sof_pdata->fw_filename_prefix =
+ sof_pdata->desc->default_fw_path;
+
+ if (tplg_path)
+ sof_pdata->tplg_filename_prefix = tplg_path;
+ else
+ sof_pdata->tplg_filename_prefix =
+ sof_pdata->desc->default_tplg_path;
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_PROBE_WORK_QUEUE)
+ /* set callback to enable runtime_pm */
+ sof_pdata->sof_probe_complete = sof_acpi_probe_complete;
+#endif
+ /* call sof helper for DSP hardware probe */
+ ret = snd_sof_device_probe(dev, sof_pdata);
+ if (ret) {
+ dev_err(dev, "error: failed to probe DSP hardware!\n");
+ return ret;
+ }
+
+#if !IS_ENABLED(CONFIG_SND_SOC_SOF_PROBE_WORK_QUEUE)
+ sof_acpi_probe_complete(dev);
+#endif
+
+ return ret;
+}
+
+static int sof_acpi_remove(struct platform_device *pdev)
+{
+ pm_runtime_disable(&pdev->dev);
+
+ /* call sof helper for DSP hardware remove */
+ snd_sof_device_remove(&pdev->dev);
+
+ return 0;
+}
+
+static const struct acpi_device_id sof_acpi_match[] = {
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HASWELL)
+ { "INT33C8", (unsigned long)&sof_acpi_haswell_desc },
+#endif
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_BROADWELL)
+ { "INT3438", (unsigned long)&sof_acpi_broadwell_desc },
+#endif
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_BAYTRAIL)
+ { "80860F28", (unsigned long)&sof_acpi_baytrail_desc },
+ { "808622A8", (unsigned long)&sof_acpi_cherrytrail_desc },
+#endif
+ { }
+};
+MODULE_DEVICE_TABLE(acpi, sof_acpi_match);
+
+/* acpi_driver definition */
+static struct platform_driver snd_sof_acpi_driver = {
+ .probe = sof_acpi_probe,
+ .remove = sof_acpi_remove,
+ .driver = {
+ .name = "sof-audio-acpi",
+ .pm = &sof_acpi_pm,
+ .acpi_match_table = ACPI_PTR(sof_acpi_match),
+ },
+};
+module_platform_driver(snd_sof_acpi_driver);
+
+MODULE_LICENSE("Dual BSD/GPL");
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+#include <linux/firmware.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/pm_runtime.h>
+#include <sound/soc-acpi.h>
+#include <sound/soc-acpi-intel-match.h>
+#include <sound/sof.h>
+#include "ops.h"
+
+/* platform specific devices */
+#include "intel/shim.h"
+#include "intel/hda.h"
+
+static char *fw_path;
+module_param(fw_path, charp, 0444);
+MODULE_PARM_DESC(fw_path, "alternate path for SOF firmware.");
+
+static char *tplg_path;
+module_param(tplg_path, charp, 0444);
+MODULE_PARM_DESC(tplg_path, "alternate path for SOF topology.");
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_APOLLOLAKE)
+static const struct sof_dev_desc bxt_desc = {
+ .machines = snd_soc_acpi_intel_bxt_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = -1,
+ .resindex_imr_base = -1,
+ .irqindex_host_ipc = -1,
+ .resindex_dma_base = -1,
+ .chip_info = &apl_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-apl.ri",
+ .nocodec_tplg_filename = "sof-apl-nocodec.tplg",
+ .ops = &sof_apl_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_GEMINILAKE)
+static const struct sof_dev_desc glk_desc = {
+ .machines = snd_soc_acpi_intel_glk_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = -1,
+ .resindex_imr_base = -1,
+ .irqindex_host_ipc = -1,
+ .resindex_dma_base = -1,
+ .chip_info = &apl_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-glk.ri",
+ .nocodec_tplg_filename = "sof-glk-nocodec.tplg",
+ .ops = &sof_apl_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_MERRIFIELD)
+static struct snd_soc_acpi_mach sof_tng_machines[] = {
+ {
+ .id = "INT343A",
+ .drv_name = "edison",
+ .sof_fw_filename = "sof-byt.ri",
+ .sof_tplg_filename = "sof-byt.tplg",
+ },
+ {}
+};
+
+static const struct sof_dev_desc tng_desc = {
+ .machines = sof_tng_machines,
+ .resindex_lpe_base = 3, /* IRAM, but subtract IRAM offset */
+ .resindex_pcicfg_base = -1,
+ .resindex_imr_base = 0,
+ .irqindex_host_ipc = -1,
+ .resindex_dma_base = -1,
+ .chip_info = &tng_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-byt.ri",
+ .nocodec_tplg_filename = "sof-byt.tplg",
+ .ops = &sof_tng_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_CANNONLAKE)
+static const struct sof_dev_desc cnl_desc = {
+ .machines = snd_soc_acpi_intel_cnl_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = -1,
+ .resindex_imr_base = -1,
+ .irqindex_host_ipc = -1,
+ .resindex_dma_base = -1,
+ .chip_info = &cnl_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-cnl.ri",
+ .nocodec_tplg_filename = "sof-cnl-nocodec.tplg",
+ .ops = &sof_cnl_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_COFFEELAKE)
+static const struct sof_dev_desc cfl_desc = {
+ .machines = snd_soc_acpi_intel_cnl_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = -1,
+ .resindex_imr_base = -1,
+ .irqindex_host_ipc = -1,
+ .resindex_dma_base = -1,
+ .chip_info = &cnl_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-cnl.ri",
+ .nocodec_tplg_filename = "sof-cnl-nocodec.tplg",
+ .ops = &sof_cnl_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_ICELAKE)
+static const struct sof_dev_desc icl_desc = {
+ .machines = snd_soc_acpi_intel_icl_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = -1,
+ .resindex_imr_base = -1,
+ .irqindex_host_ipc = -1,
+ .resindex_dma_base = -1,
+ .chip_info = &cnl_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-icl.ri",
+ .nocodec_tplg_filename = "sof-icl-nocodec.tplg",
+ .ops = &sof_cnl_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_SKYLAKE)
+static const struct sof_dev_desc skl_desc = {
+ .machines = snd_soc_acpi_intel_skl_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = -1,
+ .resindex_imr_base = -1,
+ .irqindex_host_ipc = -1,
+ .resindex_dma_base = -1,
+ .chip_info = &skl_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-skl.ri",
+ .nocodec_tplg_filename = "sof-skl-nocodec.tplg",
+ .ops = &sof_skl_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+#endif
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_KABYLAKE)
+static const struct sof_dev_desc kbl_desc = {
+ .machines = snd_soc_acpi_intel_kbl_machines,
+ .resindex_lpe_base = 0,
+ .resindex_pcicfg_base = -1,
+ .resindex_imr_base = -1,
+ .irqindex_host_ipc = -1,
+ .resindex_dma_base = -1,
+ .chip_info = &skl_chip_info,
+ .default_fw_path = "intel/sof",
+ .default_tplg_path = "intel/sof-tplg",
+ .nocodec_fw_filename = "sof-kbl.ri",
+ .nocodec_tplg_filename = "sof-kbl-nocodec.tplg",
+ .ops = &sof_skl_ops,
+ .arch_ops = &sof_xtensa_arch_ops
+};
+#endif
+
+static const struct dev_pm_ops sof_pci_pm = {
+ SET_SYSTEM_SLEEP_PM_OPS(snd_sof_suspend, snd_sof_resume)
+ SET_RUNTIME_PM_OPS(snd_sof_runtime_suspend, snd_sof_runtime_resume,
+ NULL)
+};
+
+static void sof_pci_probe_complete(struct device *dev)
+{
+ dev_dbg(dev, "Completing SOF PCI probe");
+
+ /* allow runtime_pm */
+ pm_runtime_set_autosuspend_delay(dev, SND_SOF_SUSPEND_DELAY_MS);
+ pm_runtime_use_autosuspend(dev);
+
+ /*
+ * runtime pm for pci device is "forbidden" by default.
+ * so call pm_runtime_allow() to enable it.
+ */
+ pm_runtime_allow(dev);
+
+ /* follow recommendation in pci-driver.c to decrement usage counter */
+ pm_runtime_put_noidle(dev);
+}
+
+static int sof_pci_probe(struct pci_dev *pci,
+ const struct pci_device_id *pci_id)
+{
+ struct device *dev = &pci->dev;
+ const struct sof_dev_desc *desc =
+ (const struct sof_dev_desc *)pci_id->driver_data;
+ struct snd_soc_acpi_mach *mach;
+ struct snd_sof_pdata *sof_pdata;
+ const struct snd_sof_dsp_ops *ops;
+ int ret;
+
+ dev_dbg(&pci->dev, "PCI DSP detected");
+
+ /* get ops for platform */
+ ops = desc->ops;
+ if (!ops) {
+ dev_err(dev, "error: no matching PCI descriptor ops\n");
+ return -ENODEV;
+ }
+
+ sof_pdata = devm_kzalloc(dev, sizeof(*sof_pdata), GFP_KERNEL);
+ if (!sof_pdata)
+ return -ENOMEM;
+
+ ret = pcim_enable_device(pci);
+ if (ret < 0)
+ return ret;
+
+ ret = pci_request_regions(pci, "Audio DSP");
+ if (ret < 0)
+ return ret;
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_FORCE_NOCODEC_MODE)
+ /* force nocodec mode */
+ dev_warn(dev, "Force to use nocodec mode\n");
+ mach = devm_kzalloc(dev, sizeof(*mach), GFP_KERNEL);
+ if (!mach) {
+ ret = -ENOMEM;
+ goto release_regions;
+ }
+ ret = sof_nocodec_setup(dev, sof_pdata, mach, desc, ops);
+ if (ret < 0)
+ goto release_regions;
+
+#else
+ /* find machine */
+ mach = snd_soc_acpi_find_machine(desc->machines);
+ if (!mach) {
+ dev_warn(dev, "warning: No matching ASoC machine driver found\n");
+ } else {
+ mach->mach_params.platform = dev_name(dev);
+ sof_pdata->fw_filename = mach->sof_fw_filename;
+ sof_pdata->tplg_filename = mach->sof_tplg_filename;
+ }
+#endif /* CONFIG_SND_SOC_SOF_FORCE_NOCODEC_MODE */
+
+ sof_pdata->name = pci_name(pci);
+ sof_pdata->machine = mach;
+ sof_pdata->desc = (struct sof_dev_desc *)pci_id->driver_data;
+ sof_pdata->dev = dev;
+ sof_pdata->platform = dev_name(dev);
+
+ /* alternate fw and tplg filenames ? */
+ if (fw_path)
+ sof_pdata->fw_filename_prefix = fw_path;
+ else
+ sof_pdata->fw_filename_prefix =
+ sof_pdata->desc->default_fw_path;
+
+ if (tplg_path)
+ sof_pdata->tplg_filename_prefix = tplg_path;
+ else
+ sof_pdata->tplg_filename_prefix =
+ sof_pdata->desc->default_tplg_path;
+
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_PROBE_WORK_QUEUE)
+ /* set callback to enable runtime_pm */
+ sof_pdata->sof_probe_complete = sof_pci_probe_complete;
+#endif
+ /* call sof helper for DSP hardware probe */
+ ret = snd_sof_device_probe(dev, sof_pdata);
+ if (ret) {
+ dev_err(dev, "error: failed to probe DSP hardware!\n");
+ goto release_regions;
+ }
+
+#if !IS_ENABLED(CONFIG_SND_SOC_SOF_PROBE_WORK_QUEUE)
+ sof_pci_probe_complete(dev);
+#endif
+
+ return ret;
+
+release_regions:
+ pci_release_regions(pci);
+
+ return ret;
+}
+
+static void sof_pci_remove(struct pci_dev *pci)
+{
+ /* call sof helper for DSP hardware remove */
+ snd_sof_device_remove(&pci->dev);
+
+ /* follow recommendation in pci-driver.c to increment usage counter */
+ pm_runtime_get_noresume(&pci->dev);
+
+ /* release pci regions and disable device */
+ pci_release_regions(pci);
+}
+
+/* PCI IDs */
+static const struct pci_device_id sof_pci_ids[] = {
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_MERRIFIELD)
+ { PCI_DEVICE(0x8086, 0x119a),
+ .driver_data = (unsigned long)&tng_desc},
+#endif
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_APOLLOLAKE)
+ /* BXT-P & Apollolake */
+ { PCI_DEVICE(0x8086, 0x5a98),
+ .driver_data = (unsigned long)&bxt_desc},
+ { PCI_DEVICE(0x8086, 0x1a98),
+ .driver_data = (unsigned long)&bxt_desc},
+#endif
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_GEMINILAKE)
+ { PCI_DEVICE(0x8086, 0x3198),
+ .driver_data = (unsigned long)&glk_desc},
+#endif
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_CANNONLAKE)
+ { PCI_DEVICE(0x8086, 0x9dc8),
+ .driver_data = (unsigned long)&cnl_desc},
+#endif
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_COFFEELAKE)
+ { PCI_DEVICE(0x8086, 0xa348),
+ .driver_data = (unsigned long)&cfl_desc},
+#endif
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_KABYLAKE)
+ { PCI_DEVICE(0x8086, 0x9d71),
+ .driver_data = (unsigned long)&kbl_desc},
+#endif
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_SKYLAKE)
+ { PCI_DEVICE(0x8086, 0x9d70),
+ .driver_data = (unsigned long)&skl_desc},
+#endif
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_ICELAKE)
+ { PCI_DEVICE(0x8086, 0x34C8),
+ .driver_data = (unsigned long)&icl_desc},
+#endif
+ { 0, }
+};
+MODULE_DEVICE_TABLE(pci, sof_pci_ids);
+
+/* pci_driver definition */
+static struct pci_driver snd_sof_pci_driver = {
+ .name = "sof-audio-pci",
+ .id_table = sof_pci_ids,
+ .probe = sof_pci_probe,
+ .remove = sof_pci_remove,
+ .driver = {
+ .pm = &sof_pci_pm,
+ },
+};
+module_pci_driver(snd_sof_pci_driver);
+
+MODULE_LICENSE("Dual BSD/GPL");
--- /dev/null
+/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2018 Intel Corporation. All rights reserved.
+ *
+ * Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+ */
+
+#ifndef __SOUND_SOC_SOF_PRIV_H
+#define __SOUND_SOC_SOF_PRIV_H
+
+#include <linux/device.h>
+
+#include <sound/hdaudio.h>
+#include <sound/soc.h>
+
+#include <sound/sof.h>
+#include <sound/sof/stream.h> /* needs to be included before control.h */
+#include <sound/sof/control.h>
+#include <sound/sof/dai.h>
+#include <sound/sof/info.h>
+#include <sound/sof/pm.h>
+#include <sound/sof/topology.h>
+#include <sound/sof/trace.h>
+
+#include <uapi/sound/sof/fw.h>
+
+/* debug flags */
+#define SOF_DBG_REGS BIT(1)
+#define SOF_DBG_MBOX BIT(2)
+#define SOF_DBG_TEXT BIT(3)
+#define SOF_DBG_PCI BIT(4)
+
+/* max BARs mmaped devices can use */
+#define SND_SOF_BARS 8
+
+/* time in ms for runtime suspend delay */
+#define SND_SOF_SUSPEND_DELAY_MS 2000
+
+/* DMA buffer size for trace */
+#define DMA_BUF_SIZE_FOR_TRACE (PAGE_SIZE * 16)
+
+/* max number of FE PCMs before BEs */
+#define SOF_BE_PCM_BASE 16
+
+#define SOF_IPC_DSP_REPLY 0
+#define SOF_IPC_HOST_REPLY 1
+
+/* convenience constructor for DAI driver streams */
+#define SOF_DAI_STREAM(sname, scmin, scmax, srates, sfmt) \
+ {.stream_name = sname, .channels_min = scmin, .channels_max = scmax, \
+ .rates = srates, .formats = sfmt}
+
+#define SOF_FORMATS (SNDRV_PCM_FMTBIT_S16_LE | SNDRV_PCM_FMTBIT_S24_LE | \
+ SNDRV_PCM_FMTBIT_S32_LE | SNDRV_PCM_FMTBIT_FLOAT)
+
+struct snd_sof_dev;
+struct snd_sof_ipc_msg;
+struct snd_sof_ipc;
+struct snd_sof_debugfs_map;
+struct snd_soc_tplg_ops;
+struct snd_soc_component;
+struct snd_sof_pdata;
+
+/*
+ * SOF DSP HW abstraction operations.
+ * Used to abstract DSP HW architecture and any IO busses between host CPU
+ * and DSP device(s).
+ */
+struct snd_sof_dsp_ops {
+
+ /* probe and remove */
+ int (*probe)(struct snd_sof_dev *sof_dev); /* mandatory */
+ int (*remove)(struct snd_sof_dev *sof_dev); /* optional */
+
+ /* DSP core boot / reset */
+ int (*run)(struct snd_sof_dev *sof_dev); /* mandatory */
+ int (*stall)(struct snd_sof_dev *sof_dev); /* optional */
+ int (*reset)(struct snd_sof_dev *sof_dev); /* optional */
+ int (*core_power_up)(struct snd_sof_dev *sof_dev,
+ unsigned int core_mask); /* optional */
+ int (*core_power_down)(struct snd_sof_dev *sof_dev,
+ unsigned int core_mask); /* optional */
+
+ /*
+ * Register IO: only used by respective drivers themselves,
+ * TODO: consider removing these operations and calling respective
+ * implementations directly
+ */
+ void (*write)(struct snd_sof_dev *sof_dev, void __iomem *addr,
+ u32 value); /* optional */
+ u32 (*read)(struct snd_sof_dev *sof_dev,
+ void __iomem *addr); /* optional */
+ void (*write64)(struct snd_sof_dev *sof_dev, void __iomem *addr,
+ u64 value); /* optional */
+ u64 (*read64)(struct snd_sof_dev *sof_dev,
+ void __iomem *addr); /* optional */
+
+ /* memcpy IO */
+ void (*block_read)(struct snd_sof_dev *sof_dev, u32 bar,
+ u32 offset, void *dest,
+ size_t size); /* mandatory */
+ void (*block_write)(struct snd_sof_dev *sof_dev, u32 bar,
+ u32 offset, void *src,
+ size_t size); /* mandatory */
+
+ /* doorbell */
+ irqreturn_t (*irq_handler)(int irq, void *context); /* optional */
+ irqreturn_t (*irq_thread)(int irq, void *context); /* optional */
+
+ /* ipc */
+ int (*send_msg)(struct snd_sof_dev *sof_dev,
+ struct snd_sof_ipc_msg *msg); /* mandatory */
+
+ /* FW loading */
+ int (*load_firmware)(struct snd_sof_dev *sof_dev); /* mandatory */
+ int (*load_module)(struct snd_sof_dev *sof_dev,
+ struct snd_sof_mod_hdr *hdr); /* optional */
+ /*
+ * FW ready checks for ABI compatibility and creates
+ * memory windows at first boot
+ */
+ int (*fw_ready)(struct snd_sof_dev *sdev, u32 msg_id); /* optional */
+
+ /* connect pcm substream to a host stream */
+ int (*pcm_open)(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream); /* optional */
+ /* disconnect pcm substream to a host stream */
+ int (*pcm_close)(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream); /* optional */
+
+ /* host stream hw params */
+ int (*pcm_hw_params)(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params,
+ struct sof_ipc_stream_params *ipc_params); /* optional */
+
+ /* host stream trigger */
+ int (*pcm_trigger)(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ int cmd); /* optional */
+
+ /* host stream pointer */
+ snd_pcm_uframes_t (*pcm_pointer)(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream); /* optional */
+
+ /* host read DSP stream data */
+ void (*ipc_msg_data)(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ void *p, size_t sz); /* mandatory */
+
+ /* host configure DSP HW parameters */
+ int (*ipc_pcm_params)(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ const struct sof_ipc_pcm_params_reply *reply); /* mandatory */
+
+ /* pre/post firmware run */
+ int (*pre_fw_run)(struct snd_sof_dev *sof_dev); /* optional */
+ int (*post_fw_run)(struct snd_sof_dev *sof_dev); /* optional */
+
+ /* DSP PM */
+ int (*suspend)(struct snd_sof_dev *sof_dev, int state); /* optional */
+ int (*resume)(struct snd_sof_dev *sof_dev); /* optional */
+ int (*runtime_suspend)(struct snd_sof_dev *sof_dev,
+ int state); /* optional */
+ int (*runtime_resume)(struct snd_sof_dev *sof_dev); /* optional */
+ void (*set_hw_params_upon_resume)(struct snd_sof_dev *sdev); /* optional */
+
+ /* DSP clocking */
+ int (*set_clk)(struct snd_sof_dev *sof_dev, u32 freq); /* optional */
+
+ /* debug */
+ const struct snd_sof_debugfs_map *debug_map; /* optional */
+ int debug_map_count; /* optional */
+ void (*dbg_dump)(struct snd_sof_dev *sof_dev,
+ u32 flags); /* optional */
+ void (*ipc_dump)(struct snd_sof_dev *sof_dev); /* optional */
+
+ /* host DMA trace initialization */
+ int (*trace_init)(struct snd_sof_dev *sdev,
+ u32 *stream_tag); /* optional */
+ int (*trace_release)(struct snd_sof_dev *sdev); /* optional */
+ int (*trace_trigger)(struct snd_sof_dev *sdev,
+ int cmd); /* optional */
+
+ /* DAI ops */
+ struct snd_soc_dai_driver *drv;
+ int num_drv;
+};
+
+/* DSP architecture specific callbacks for oops and stack dumps */
+struct sof_arch_ops {
+ void (*dsp_oops)(struct snd_sof_dev *sdev, void *oops);
+ void (*dsp_stack)(struct snd_sof_dev *sdev, void *oops,
+ u32 *stack, u32 stack_words);
+};
+
+#define sof_arch_ops(sdev) ((sdev)->pdata->desc->arch_ops)
+
+/* DSP device HW descriptor mapping between bus ID and ops */
+struct sof_ops_table {
+ const struct sof_dev_desc *desc;
+ const struct snd_sof_dsp_ops *ops;
+};
+
+enum sof_dfsentry_type {
+ SOF_DFSENTRY_TYPE_IOMEM = 0,
+ SOF_DFSENTRY_TYPE_BUF,
+};
+
+enum sof_debugfs_access_type {
+ SOF_DEBUGFS_ACCESS_ALWAYS = 0,
+ SOF_DEBUGFS_ACCESS_D0_ONLY,
+};
+
+/* FS entry for debug files that can expose DSP memories, registers */
+struct snd_sof_dfsentry {
+ struct dentry *dfsentry;
+ size_t size;
+ enum sof_dfsentry_type type;
+ /*
+ * access_type specifies if the
+ * memory -> DSP resource (memory, register etc) is always accessible
+ * or if it is accessible only when the DSP is in D0.
+ */
+ enum sof_debugfs_access_type access_type;
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_DEBUG_ENABLE_DEBUGFS_CACHE)
+ char *cache_buf; /* buffer to cache the contents of debugfs memory */
+#endif
+ struct snd_sof_dev *sdev;
+ struct list_head list; /* list in sdev dfsentry list */
+ union {
+ void __iomem *io_mem;
+ void *buf;
+ };
+};
+
+/* Debug mapping for any DSP memory or registers that can used for debug */
+struct snd_sof_debugfs_map {
+ const char *name;
+ u32 bar;
+ u32 offset;
+ u32 size;
+ /*
+ * access_type specifies if the memory is always accessible
+ * or if it is accessible only when the DSP is in D0.
+ */
+ enum sof_debugfs_access_type access_type;
+};
+
+/* mailbox descriptor, used for host <-> DSP IPC */
+struct snd_sof_mailbox {
+ u32 offset;
+ size_t size;
+};
+
+/* IPC message descriptor for host <-> DSP IO */
+struct snd_sof_ipc_msg {
+ /* message data */
+ u32 header;
+ void *msg_data;
+ void *reply_data;
+ size_t msg_size;
+ size_t reply_size;
+ int reply_error;
+
+ wait_queue_head_t waitq;
+ bool ipc_complete;
+};
+
+/* PCM stream, mapped to FW component */
+struct snd_sof_pcm_stream {
+ u32 comp_id;
+ struct snd_dma_buffer page_table;
+ struct sof_ipc_stream_posn posn;
+ struct snd_pcm_substream *substream;
+ struct work_struct period_elapsed_work;
+};
+
+/* ALSA SOF PCM device */
+struct snd_sof_pcm {
+ struct snd_sof_dev *sdev;
+ struct snd_soc_tplg_pcm pcm;
+ struct snd_sof_pcm_stream stream[2];
+ struct list_head list; /* list in sdev pcm list */
+ struct snd_pcm_hw_params params[2];
+ int hw_params_upon_resume[2]; /* set up hw_params upon resume */
+};
+
+/* ALSA SOF Kcontrol device */
+struct snd_sof_control {
+ struct snd_sof_dev *sdev;
+ int comp_id;
+ int num_channels;
+ u32 readback_offset; /* offset to mmaped data if used */
+ struct sof_ipc_ctrl_data *control_data;
+ u32 size; /* cdata size */
+ enum sof_ipc_ctrl_cmd cmd;
+ u32 *volume_table; /* volume table computed from tlv data*/
+
+ struct list_head list; /* list in sdev control list */
+};
+
+/* ASoC SOF DAPM widget */
+struct snd_sof_widget {
+ struct snd_sof_dev *sdev;
+ int comp_id;
+ int pipeline_id;
+ int complete;
+ int id;
+
+ struct snd_soc_dapm_widget *widget;
+ struct list_head list; /* list in sdev widget list */
+
+ void *private; /* core does not touch this */
+};
+
+/* ASoC SOF DAPM route */
+struct snd_sof_route {
+ struct snd_sof_dev *sdev;
+
+ struct snd_soc_dapm_route *route;
+ struct list_head list; /* list in sdev route list */
+
+ void *private;
+};
+
+/* ASoC DAI device */
+struct snd_sof_dai {
+ struct snd_sof_dev *sdev;
+ const char *name;
+
+ struct sof_ipc_comp_dai comp_dai;
+ struct sof_ipc_dai_config *dai_config;
+ struct list_head list; /* list in sdev dai list */
+};
+
+/*
+ * SOF Device Level.
+ */
+struct snd_sof_dev {
+ struct device *dev;
+ spinlock_t ipc_lock; /* lock for IPC users */
+ spinlock_t hw_lock; /* lock for HW IO access */
+
+ /*
+ * ASoC components. plat_drv fields are set dynamically so
+ * can't use const
+ */
+ struct snd_soc_component_driver plat_drv;
+
+ /* DSP firmware boot */
+ wait_queue_head_t boot_wait;
+ u32 boot_complete;
+ u32 first_boot;
+
+ /* work queue in case the probe is implemented in two steps */
+ struct work_struct probe_work;
+
+ /* DSP HW differentiation */
+ struct snd_sof_pdata *pdata;
+
+ /* IPC */
+ struct snd_sof_ipc *ipc;
+ struct snd_sof_mailbox dsp_box; /* DSP initiated IPC */
+ struct snd_sof_mailbox host_box; /* Host initiated IPC */
+ struct snd_sof_mailbox stream_box; /* Stream position update */
+ struct snd_sof_ipc_msg *msg;
+ int ipc_irq;
+ u32 next_comp_id; /* monotonic - reset during S3 */
+
+ /* memory bases for mmaped DSPs - set by dsp_init() */
+ void __iomem *bar[SND_SOF_BARS]; /* DSP base address */
+ int mmio_bar;
+ int mailbox_bar;
+ size_t dsp_oops_offset;
+
+ /* debug */
+ struct dentry *debugfs_root;
+ struct list_head dfsentry_list;
+
+ /* firmware loader */
+ struct snd_dma_buffer dmab;
+ struct snd_dma_buffer dmab_bdl;
+ struct sof_ipc_fw_ready fw_ready;
+ struct sof_ipc_fw_version fw_version;
+
+ /* topology */
+ struct snd_soc_tplg_ops *tplg_ops;
+ struct list_head pcm_list;
+ struct list_head kcontrol_list;
+ struct list_head widget_list;
+ struct list_head dai_list;
+ struct list_head route_list;
+ struct snd_soc_component *component;
+ u32 enabled_cores_mask; /* keep track of enabled cores */
+
+ /* FW configuration */
+ struct sof_ipc_dma_buffer_data *info_buffer;
+ struct sof_ipc_window *info_window;
+
+ /* IPC timeouts in ms */
+ int ipc_timeout;
+ int boot_timeout;
+
+ /* Wait queue for code loading */
+ wait_queue_head_t waitq;
+ int code_loading;
+
+ /* DMA for Trace */
+ struct snd_dma_buffer dmatb;
+ struct snd_dma_buffer dmatp;
+ int dma_trace_pages;
+ wait_queue_head_t trace_sleep;
+ u32 host_offset;
+ u32 dtrace_is_enabled;
+ u32 dtrace_error;
+ u32 msi_enabled;
+
+ void *private; /* core does not touch this */
+};
+
+/*
+ * Device Level.
+ */
+
+int snd_sof_device_probe(struct device *dev, struct snd_sof_pdata *plat_data);
+int snd_sof_device_remove(struct device *dev);
+
+int snd_sof_runtime_suspend(struct device *dev);
+int snd_sof_runtime_resume(struct device *dev);
+int snd_sof_resume(struct device *dev);
+int snd_sof_suspend(struct device *dev);
+
+void snd_sof_new_platform_drv(struct snd_sof_dev *sdev);
+
+int snd_sof_create_page_table(struct snd_sof_dev *sdev,
+ struct snd_dma_buffer *dmab,
+ unsigned char *page_table, size_t size);
+
+/*
+ * Firmware loading.
+ */
+int snd_sof_load_firmware(struct snd_sof_dev *sdev);
+int snd_sof_load_firmware_raw(struct snd_sof_dev *sdev);
+int snd_sof_load_firmware_memcpy(struct snd_sof_dev *sdev);
+int snd_sof_run_firmware(struct snd_sof_dev *sdev);
+int snd_sof_parse_module_memcpy(struct snd_sof_dev *sdev,
+ struct snd_sof_mod_hdr *module);
+void snd_sof_fw_unload(struct snd_sof_dev *sdev);
+int snd_sof_fw_parse_ext_data(struct snd_sof_dev *sdev, u32 bar, u32 offset);
+
+/*
+ * IPC low level APIs.
+ */
+struct snd_sof_ipc *snd_sof_ipc_init(struct snd_sof_dev *sdev);
+void snd_sof_ipc_free(struct snd_sof_dev *sdev);
+int snd_sof_ipc_reply(struct snd_sof_dev *sdev, u32 msg_id);
+void snd_sof_ipc_msgs_rx(struct snd_sof_dev *sdev);
+int snd_sof_ipc_stream_pcm_params(struct snd_sof_dev *sdev,
+ struct sof_ipc_pcm_params *params);
+int snd_sof_dsp_mailbox_init(struct snd_sof_dev *sdev, u32 dspbox,
+ size_t dspbox_size, u32 hostbox,
+ size_t hostbox_size);
+int snd_sof_ipc_valid(struct snd_sof_dev *sdev);
+int sof_ipc_tx_message(struct snd_sof_ipc *ipc, u32 header,
+ void *msg_data, size_t msg_bytes, void *reply_data,
+ size_t reply_bytes);
+struct snd_sof_widget *snd_sof_find_swidget(struct snd_sof_dev *sdev,
+ const char *name);
+struct snd_sof_widget *snd_sof_find_swidget_sname(struct snd_sof_dev *sdev,
+ const char *pcm_name,
+ int dir);
+struct snd_sof_dai *snd_sof_find_dai(struct snd_sof_dev *sdev,
+ const char *name);
+
+static inline
+struct snd_sof_pcm *snd_sof_find_spcm_dai(struct snd_sof_dev *sdev,
+ struct snd_soc_pcm_runtime *rtd)
+{
+ struct snd_sof_pcm *spcm = NULL;
+
+ list_for_each_entry(spcm, &sdev->pcm_list, list) {
+ if (le32_to_cpu(spcm->pcm.dai_id) == rtd->dai_link->id)
+ return spcm;
+ }
+
+ return NULL;
+}
+
+struct snd_sof_pcm *snd_sof_find_spcm_name(struct snd_sof_dev *sdev,
+ const char *name);
+struct snd_sof_pcm *snd_sof_find_spcm_comp(struct snd_sof_dev *sdev,
+ unsigned int comp_id,
+ int *direction);
+struct snd_sof_pcm *snd_sof_find_spcm_pcm_id(struct snd_sof_dev *sdev,
+ unsigned int pcm_id);
+void snd_sof_pcm_period_elapsed(struct snd_pcm_substream *substream);
+
+/*
+ * Stream IPC
+ */
+int snd_sof_ipc_stream_posn(struct snd_sof_dev *sdev,
+ struct snd_sof_pcm *spcm, int direction,
+ struct sof_ipc_stream_posn *posn);
+
+/*
+ * Mixer IPC
+ */
+int snd_sof_ipc_set_get_comp_data(struct snd_sof_ipc *ipc,
+ struct snd_sof_control *scontrol, u32 ipc_cmd,
+ enum sof_ipc_ctrl_type ctrl_type,
+ enum sof_ipc_ctrl_cmd ctrl_cmd,
+ bool send);
+
+/*
+ * Topology.
+ * There is no snd_sof_free_topology since topology components will
+ * be freed by snd_soc_unregister_component,
+ */
+int snd_sof_init_topology(struct snd_sof_dev *sdev,
+ struct snd_soc_tplg_ops *ops);
+int snd_sof_load_topology(struct snd_sof_dev *sdev, const char *file);
+int snd_sof_complete_pipeline(struct snd_sof_dev *sdev,
+ struct snd_sof_widget *swidget);
+
+int sof_load_pipeline_ipc(struct snd_sof_dev *sdev,
+ struct sof_ipc_pipe_new *pipeline,
+ struct sof_ipc_comp_reply *r);
+
+/*
+ * Trace/debug
+ */
+int snd_sof_init_trace(struct snd_sof_dev *sdev);
+void snd_sof_release_trace(struct snd_sof_dev *sdev);
+void snd_sof_free_trace(struct snd_sof_dev *sdev);
+int snd_sof_dbg_init(struct snd_sof_dev *sdev);
+void snd_sof_free_debug(struct snd_sof_dev *sdev);
+int snd_sof_debugfs_io_item(struct snd_sof_dev *sdev,
+ void __iomem *base, size_t size,
+ const char *name,
+ enum sof_debugfs_access_type access_type);
+int snd_sof_debugfs_buf_item(struct snd_sof_dev *sdev,
+ void *base, size_t size,
+ const char *name);
+int snd_sof_trace_update_pos(struct snd_sof_dev *sdev,
+ struct sof_ipc_dma_trace_posn *posn);
+void snd_sof_trace_notify_for_error(struct snd_sof_dev *sdev);
+void snd_sof_get_status(struct snd_sof_dev *sdev, u32 panic_code,
+ u32 tracep_code, void *oops,
+ struct sof_ipc_panic_info *panic_info,
+ void *stack, size_t stack_words);
+int snd_sof_init_trace_ipc(struct snd_sof_dev *sdev);
+
+/*
+ * Platform specific ops.
+ */
+extern struct snd_compr_ops sof_compressed_ops;
+
+/*
+ * Kcontrols.
+ */
+
+int snd_sof_volume_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol);
+int snd_sof_volume_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol);
+int snd_sof_switch_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol);
+int snd_sof_switch_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol);
+int snd_sof_enum_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol);
+int snd_sof_enum_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol);
+int snd_sof_bytes_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol);
+int snd_sof_bytes_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol);
+int snd_sof_bytes_ext_put(struct snd_kcontrol *kcontrol,
+ const unsigned int __user *binary_data,
+ unsigned int size);
+int snd_sof_bytes_ext_get(struct snd_kcontrol *kcontrol,
+ unsigned int __user *binary_data,
+ unsigned int size);
+
+/*
+ * DSP Architectures.
+ */
+static inline void sof_stack(struct snd_sof_dev *sdev, void *oops, u32 *stack,
+ u32 stack_words)
+{
+ if (sof_arch_ops(sdev)->dsp_stack)
+ sof_arch_ops(sdev)->dsp_stack(sdev, oops, stack, stack_words);
+}
+
+static inline void sof_oops(struct snd_sof_dev *sdev, void *oops)
+{
+ if (sof_arch_ops(sdev)->dsp_oops)
+ sof_arch_ops(sdev)->dsp_oops(sdev, oops);
+}
+
+extern const struct sof_arch_ops sof_xtensa_arch_ops;
+
+/*
+ * Utilities
+ */
+void sof_io_write(struct snd_sof_dev *sdev, void __iomem *addr, u32 value);
+void sof_io_write64(struct snd_sof_dev *sdev, void __iomem *addr, u64 value);
+u32 sof_io_read(struct snd_sof_dev *sdev, void __iomem *addr);
+u64 sof_io_read64(struct snd_sof_dev *sdev, void __iomem *addr);
+void sof_mailbox_write(struct snd_sof_dev *sdev, u32 offset,
+ void *message, size_t bytes);
+void sof_mailbox_read(struct snd_sof_dev *sdev, u32 offset,
+ void *message, size_t bytes);
+void sof_block_write(struct snd_sof_dev *sdev, u32 bar, u32 offset, void *src,
+ size_t size);
+void sof_block_read(struct snd_sof_dev *sdev, u32 bar, u32 offset, void *dest,
+ size_t size);
+
+void intel_ipc_msg_data(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ void *p, size_t sz);
+int intel_ipc_pcm_params(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ const struct sof_ipc_pcm_params_reply *reply);
+
+int intel_pcm_open(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream);
+int intel_pcm_close(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+#include <linux/firmware.h>
+#include <sound/tlv.h>
+#include <sound/pcm_params.h>
+#include <uapi/sound/sof/tokens.h>
+#include "sof-priv.h"
+#include "ops.h"
+
+#define COMP_ID_UNASSIGNED 0xffffffff
+/*
+ * Constants used in the computation of linear volume gain
+ * from dB gain 20th root of 10 in Q1.16 fixed-point notation
+ */
+#define VOL_TWENTIETH_ROOT_OF_TEN 73533
+/* 40th root of 10 in Q1.16 fixed-point notation*/
+#define VOL_FORTIETH_ROOT_OF_TEN 69419
+/*
+ * Volume fractional word length define to 16 sets
+ * the volume linear gain value to use Qx.16 format
+ */
+#define VOLUME_FWL 16
+/* 0.5 dB step value in topology TLV */
+#define VOL_HALF_DB_STEP 50
+/* Full volume for default values */
+#define VOL_ZERO_DB BIT(VOLUME_FWL)
+
+/* TLV data items */
+#define TLV_ITEMS 3
+#define TLV_MIN 0
+#define TLV_STEP 1
+#define TLV_MUTE 2
+
+/* size of tplg abi in byte */
+#define SOF_TPLG_ABI_SIZE 3
+
+/* send pcm params ipc */
+static int ipc_pcm_params(struct snd_sof_widget *swidget, int dir)
+{
+ struct sof_ipc_pcm_params_reply ipc_params_reply;
+ struct snd_sof_dev *sdev = swidget->sdev;
+ struct sof_ipc_pcm_params pcm;
+ struct snd_pcm_hw_params *params;
+ struct snd_sof_pcm *spcm;
+ int ret = 0;
+
+ memset(&pcm, 0, sizeof(pcm));
+
+ /* get runtime PCM params using widget's stream name */
+ spcm = snd_sof_find_spcm_name(sdev, swidget->widget->sname);
+ if (!spcm) {
+ dev_err(sdev->dev, "error: cannot find PCM for %s\n",
+ swidget->widget->name);
+ return -EINVAL;
+ }
+
+ params = &spcm->params[dir];
+
+ /* set IPC PCM params */
+ pcm.hdr.size = sizeof(pcm);
+ pcm.hdr.cmd = SOF_IPC_GLB_STREAM_MSG | SOF_IPC_STREAM_PCM_PARAMS;
+ pcm.comp_id = swidget->comp_id;
+ pcm.params.hdr.size = sizeof(pcm.params);
+ pcm.params.direction = dir;
+ pcm.params.sample_valid_bytes = params_width(params) >> 3;
+ pcm.params.buffer_fmt = SOF_IPC_BUFFER_INTERLEAVED;
+ pcm.params.rate = params_rate(params);
+ pcm.params.channels = params_channels(params);
+ pcm.params.host_period_bytes = params_period_bytes(params);
+
+ /* set format */
+ switch (params_format(params)) {
+ case SNDRV_PCM_FORMAT_S16:
+ pcm.params.frame_fmt = SOF_IPC_FRAME_S16_LE;
+ break;
+ case SNDRV_PCM_FORMAT_S24:
+ pcm.params.frame_fmt = SOF_IPC_FRAME_S24_4LE;
+ break;
+ case SNDRV_PCM_FORMAT_S32:
+ pcm.params.frame_fmt = SOF_IPC_FRAME_S32_LE;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ /* send IPC to the DSP */
+ ret = sof_ipc_tx_message(sdev->ipc, pcm.hdr.cmd, &pcm, sizeof(pcm),
+ &ipc_params_reply, sizeof(ipc_params_reply));
+ if (ret < 0)
+ dev_err(sdev->dev, "error: pcm params failed for %s\n",
+ swidget->widget->name);
+
+ return ret;
+}
+
+ /* send stream trigger ipc */
+static int ipc_trigger(struct snd_sof_widget *swidget, int cmd)
+{
+ struct snd_sof_dev *sdev = swidget->sdev;
+ struct sof_ipc_stream stream;
+ struct sof_ipc_reply reply;
+ int ret = 0;
+
+ /* set IPC stream params */
+ stream.hdr.size = sizeof(stream);
+ stream.hdr.cmd = SOF_IPC_GLB_STREAM_MSG | cmd;
+ stream.comp_id = swidget->comp_id;
+
+ /* send IPC to the DSP */
+ ret = sof_ipc_tx_message(sdev->ipc, stream.hdr.cmd, &stream,
+ sizeof(stream), &reply, sizeof(reply));
+ if (ret < 0)
+ dev_err(sdev->dev, "error: failed to trigger %s\n",
+ swidget->widget->name);
+
+ return ret;
+}
+
+static int sof_keyword_dapm_event(struct snd_soc_dapm_widget *w,
+ struct snd_kcontrol *k, int event)
+{
+ struct snd_sof_widget *swidget = w->dobj.private;
+ struct snd_sof_dev *sdev;
+ int ret = 0;
+
+ if (!swidget)
+ return 0;
+
+ sdev = swidget->sdev;
+
+ dev_dbg(sdev->dev, "received event %d for widget %s\n",
+ event, w->name);
+
+ /* process events */
+ switch (event) {
+ case SND_SOC_DAPM_PRE_PMU:
+ /* set pcm params */
+ ret = ipc_pcm_params(swidget, SOF_IPC_STREAM_CAPTURE);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to set pcm params for widget %s\n",
+ swidget->widget->name);
+ break;
+ }
+
+ /* start trigger */
+ ret = ipc_trigger(swidget, SOF_IPC_STREAM_TRIG_START);
+ if (ret < 0)
+ dev_err(sdev->dev,
+ "error: failed to trigger widget %s\n",
+ swidget->widget->name);
+ break;
+ case SND_SOC_DAPM_POST_PMD:
+ /* stop trigger */
+ ret = ipc_trigger(swidget, SOF_IPC_STREAM_TRIG_STOP);
+ if (ret < 0)
+ dev_err(sdev->dev,
+ "error: failed to trigger widget %s\n",
+ swidget->widget->name);
+
+ /* pcm free */
+ ret = ipc_trigger(swidget, SOF_IPC_STREAM_PCM_FREE);
+ if (ret < 0)
+ dev_err(sdev->dev,
+ "error: failed to trigger widget %s\n",
+ swidget->widget->name);
+ break;
+ default:
+ break;
+ }
+
+ return ret;
+}
+
+/* event handlers for keyword detect component */
+static const struct snd_soc_tplg_widget_events sof_kwd_events[] = {
+ {SOF_KEYWORD_DETECT_DAPM_EVENT, sof_keyword_dapm_event},
+};
+
+static inline int get_tlv_data(const int *p, int tlv[TLV_ITEMS])
+{
+ /* we only support dB scale TLV type at the moment */
+ if ((int)p[SNDRV_CTL_TLVO_TYPE] != SNDRV_CTL_TLVT_DB_SCALE)
+ return -EINVAL;
+
+ /* min value in topology tlv data is multiplied by 100 */
+ tlv[TLV_MIN] = (int)p[SNDRV_CTL_TLVO_DB_SCALE_MIN] / 100;
+
+ /* volume steps */
+ tlv[TLV_STEP] = (int)(p[SNDRV_CTL_TLVO_DB_SCALE_MUTE_AND_STEP] &
+ TLV_DB_SCALE_MASK);
+
+ /* mute ON/OFF */
+ if ((p[SNDRV_CTL_TLVO_DB_SCALE_MUTE_AND_STEP] &
+ TLV_DB_SCALE_MUTE) == 0)
+ tlv[TLV_MUTE] = 0;
+ else
+ tlv[TLV_MUTE] = 1;
+
+ return 0;
+}
+
+/*
+ * Function to truncate an unsigned 64-bit number
+ * by x bits and return 32-bit unsigned number. This
+ * function also takes care of rounding while truncating
+ */
+static inline u32 vol_shift_64(u64 i, u32 x)
+{
+ /* do not truncate more than 32 bits */
+ if (x > 32)
+ x = 32;
+
+ if (x == 0)
+ return (u32)i;
+
+ return (u32)(((i >> (x - 1)) + 1) >> 1);
+}
+
+/*
+ * Function to compute a ^ exp where,
+ * a is a fractional number represented by a fixed-point
+ * integer with a fractional world length of "fwl"
+ * exp is an integer
+ * fwl is the fractional word length
+ * Return value is a fractional number represented by a
+ * fixed-point integer with a fractional word length of "fwl"
+ */
+static u32 vol_pow32(u32 a, int exp, u32 fwl)
+{
+ int i, iter;
+ u32 power = 1 << fwl;
+ u64 numerator;
+
+ /* if exponent is 0, return 1 */
+ if (exp == 0)
+ return power;
+
+ /* determine the number of iterations based on the exponent */
+ if (exp < 0)
+ iter = exp * -1;
+ else
+ iter = exp;
+
+ /* mutiply a "iter" times to compute power */
+ for (i = 0; i < iter; i++) {
+ /*
+ * Product of 2 Qx.fwl fixed-point numbers yields a Q2*x.2*fwl
+ * Truncate product back to fwl fractional bits with rounding
+ */
+ power = vol_shift_64((u64)power * a, fwl);
+ }
+
+ if (exp > 0) {
+ /* if exp is positive, return the result */
+ return power;
+ }
+
+ /* if exp is negative, return the multiplicative inverse */
+ numerator = (u64)1 << (fwl << 1);
+ do_div(numerator, power);
+
+ return (u32)numerator;
+}
+
+/*
+ * Function to calculate volume gain from TLV data.
+ * This function can only handle gain steps that are multiples of 0.5 dB
+ */
+static u32 vol_compute_gain(u32 value, int *tlv)
+{
+ int dB_gain;
+ u32 linear_gain;
+ int f_step;
+
+ /* mute volume */
+ if (value == 0 && tlv[TLV_MUTE])
+ return 0;
+
+ /*
+ * compute dB gain from tlv. tlv_step
+ * in topology is multiplied by 100
+ */
+ dB_gain = tlv[TLV_MIN] + (value * tlv[TLV_STEP]) / 100;
+
+ /*
+ * compute linear gain represented by fixed-point
+ * int with VOLUME_FWL fractional bits
+ */
+ linear_gain = vol_pow32(VOL_TWENTIETH_ROOT_OF_TEN, dB_gain, VOLUME_FWL);
+
+ /* extract the fractional part of volume step */
+ f_step = tlv[TLV_STEP] - (tlv[TLV_STEP] / 100);
+
+ /* if volume step is an odd multiple of 0.5 dB */
+ if (f_step == VOL_HALF_DB_STEP && (value & 1))
+ linear_gain = vol_shift_64((u64)linear_gain *
+ VOL_FORTIETH_ROOT_OF_TEN,
+ VOLUME_FWL);
+
+ return linear_gain;
+}
+
+/*
+ * Set up volume table for kcontrols from tlv data
+ * "size" specifies the number of entries in the table
+ */
+static int set_up_volume_table(struct snd_sof_control *scontrol,
+ int tlv[TLV_ITEMS], int size)
+{
+ int j;
+
+ /* init the volume table */
+ scontrol->volume_table = kcalloc(size, sizeof(u32), GFP_KERNEL);
+ if (!scontrol->volume_table)
+ return -ENOMEM;
+
+ /* populate the volume table */
+ for (j = 0; j < size ; j++)
+ scontrol->volume_table[j] = vol_compute_gain(j, tlv);
+
+ return 0;
+}
+
+struct sof_dai_types {
+ const char *name;
+ enum sof_ipc_dai_type type;
+};
+
+static const struct sof_dai_types sof_dais[] = {
+ {"SSP", SOF_DAI_INTEL_SSP},
+ {"HDA", SOF_DAI_INTEL_HDA},
+ {"DMIC", SOF_DAI_INTEL_DMIC},
+};
+
+static enum sof_ipc_dai_type find_dai(const char *name)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(sof_dais); i++) {
+ if (strcmp(name, sof_dais[i].name) == 0)
+ return sof_dais[i].type;
+ }
+
+ return SOF_DAI_INTEL_NONE;
+}
+
+/*
+ * Supported Frame format types and lookup, add new ones to end of list.
+ */
+
+struct sof_frame_types {
+ const char *name;
+ enum sof_ipc_frame frame;
+};
+
+static const struct sof_frame_types sof_frames[] = {
+ {"s16le", SOF_IPC_FRAME_S16_LE},
+ {"s24le", SOF_IPC_FRAME_S24_4LE},
+ {"s32le", SOF_IPC_FRAME_S32_LE},
+ {"float", SOF_IPC_FRAME_FLOAT},
+};
+
+static enum sof_ipc_frame find_format(const char *name)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(sof_frames); i++) {
+ if (strcmp(name, sof_frames[i].name) == 0)
+ return sof_frames[i].frame;
+ }
+
+ /* use s32le if nothing is specified */
+ return SOF_IPC_FRAME_S32_LE;
+}
+
+struct sof_process_types {
+ const char *name;
+ enum sof_ipc_process_type type;
+ enum sof_comp_type comp_type;
+};
+
+static const struct sof_process_types sof_process[] = {
+ {"EQFIR", SOF_PROCESS_EQFIR, SOF_COMP_EQ_FIR},
+ {"EQIIR", SOF_PROCESS_EQIIR, SOF_COMP_EQ_IIR},
+ {"KEYWORD_DETECT", SOF_PROCESS_KEYWORD_DETECT, SOF_COMP_KEYWORD_DETECT},
+ {"KPB", SOF_PROCESS_KPB, SOF_COMP_KPB},
+ {"CHAN_SELECTOR", SOF_PROCESS_CHAN_SELECTOR, SOF_COMP_SELECTOR},
+};
+
+static enum sof_ipc_process_type find_process(const char *name)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(sof_process); i++) {
+ if (strcmp(name, sof_process[i].name) == 0)
+ return sof_process[i].type;
+ }
+
+ return SOF_PROCESS_NONE;
+}
+
+static enum sof_comp_type find_process_comp_type(enum sof_ipc_process_type type)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(sof_process); i++) {
+ if (sof_process[i].type == type)
+ return sof_process[i].comp_type;
+ }
+
+ return SOF_COMP_NONE;
+}
+
+/*
+ * Standard Kcontrols.
+ */
+
+static int sof_control_load_volume(struct snd_soc_component *scomp,
+ struct snd_sof_control *scontrol,
+ struct snd_kcontrol_new *kc,
+ struct snd_soc_tplg_ctl_hdr *hdr)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_mixer_control *mc =
+ container_of(hdr, struct snd_soc_tplg_mixer_control, hdr);
+ struct sof_ipc_ctrl_data *cdata;
+ int tlv[TLV_ITEMS];
+ unsigned int i;
+ int ret;
+
+ /* validate topology data */
+ if (le32_to_cpu(mc->num_channels) > SND_SOC_TPLG_MAX_CHAN)
+ return -EINVAL;
+
+ /* init the volume get/put data */
+ scontrol->size = sizeof(struct sof_ipc_ctrl_data) +
+ sizeof(struct sof_ipc_ctrl_value_chan) *
+ le32_to_cpu(mc->num_channels);
+ scontrol->control_data = kzalloc(scontrol->size, GFP_KERNEL);
+ if (!scontrol->control_data)
+ return -ENOMEM;
+
+ scontrol->comp_id = sdev->next_comp_id;
+ scontrol->num_channels = le32_to_cpu(mc->num_channels);
+
+ /* set cmd for mixer control */
+ if (le32_to_cpu(mc->max) == 1) {
+ scontrol->cmd = SOF_CTRL_CMD_SWITCH;
+ goto out;
+ }
+
+ scontrol->cmd = SOF_CTRL_CMD_VOLUME;
+
+ /* extract tlv data */
+ if (get_tlv_data(kc->tlv.p, tlv) < 0) {
+ dev_err(sdev->dev, "error: invalid TLV data\n");
+ return -EINVAL;
+ }
+
+ /* set up volume table */
+ ret = set_up_volume_table(scontrol, tlv, le32_to_cpu(mc->max) + 1);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: setting up volume table\n");
+ return ret;
+ }
+
+ /* set default volume values to 0dB in control */
+ cdata = scontrol->control_data;
+ for (i = 0; i < scontrol->num_channels; i++) {
+ cdata->chanv[i].channel = i;
+ cdata->chanv[i].value = VOL_ZERO_DB;
+ }
+
+out:
+ dev_dbg(sdev->dev, "tplg: load kcontrol index %d chans %d\n",
+ scontrol->comp_id, scontrol->num_channels);
+
+ return 0;
+}
+
+static int sof_control_load_enum(struct snd_soc_component *scomp,
+ struct snd_sof_control *scontrol,
+ struct snd_kcontrol_new *kc,
+ struct snd_soc_tplg_ctl_hdr *hdr)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_enum_control *ec =
+ container_of(hdr, struct snd_soc_tplg_enum_control, hdr);
+
+ /* validate topology data */
+ if (le32_to_cpu(ec->num_channels) > SND_SOC_TPLG_MAX_CHAN)
+ return -EINVAL;
+
+ /* init the enum get/put data */
+ scontrol->size = sizeof(struct sof_ipc_ctrl_data) +
+ sizeof(struct sof_ipc_ctrl_value_chan) *
+ le32_to_cpu(ec->num_channels);
+ scontrol->control_data = kzalloc(scontrol->size, GFP_KERNEL);
+ if (!scontrol->control_data)
+ return -ENOMEM;
+
+ scontrol->comp_id = sdev->next_comp_id;
+ scontrol->num_channels = le32_to_cpu(ec->num_channels);
+
+ scontrol->cmd = SOF_CTRL_CMD_ENUM;
+
+ dev_dbg(sdev->dev, "tplg: load kcontrol index %d chans %d comp_id %d\n",
+ scontrol->comp_id, scontrol->num_channels, scontrol->comp_id);
+
+ return 0;
+}
+
+static int sof_control_load_bytes(struct snd_soc_component *scomp,
+ struct snd_sof_control *scontrol,
+ struct snd_kcontrol_new *kc,
+ struct snd_soc_tplg_ctl_hdr *hdr)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct sof_ipc_ctrl_data *cdata;
+ struct snd_soc_tplg_bytes_control *control =
+ container_of(hdr, struct snd_soc_tplg_bytes_control, hdr);
+ struct soc_bytes_ext *sbe = (struct soc_bytes_ext *)kc->private_value;
+ int max_size = sbe->max;
+
+ if (le32_to_cpu(control->priv.size) > max_size) {
+ dev_err(sdev->dev, "err: bytes data size %d exceeds max %d.\n",
+ control->priv.size, max_size);
+ return -EINVAL;
+ }
+
+ /* init the get/put bytes data */
+ scontrol->size = sizeof(struct sof_ipc_ctrl_data) +
+ le32_to_cpu(control->priv.size);
+ scontrol->control_data = kzalloc(max_size, GFP_KERNEL);
+ cdata = scontrol->control_data;
+ if (!scontrol->control_data)
+ return -ENOMEM;
+
+ scontrol->comp_id = sdev->next_comp_id;
+ scontrol->cmd = SOF_CTRL_CMD_BINARY;
+
+ dev_dbg(sdev->dev, "tplg: load kcontrol index %d chans %d\n",
+ scontrol->comp_id, scontrol->num_channels);
+
+ if (le32_to_cpu(control->priv.size) > 0) {
+ memcpy(cdata->data, control->priv.data,
+ le32_to_cpu(control->priv.size));
+
+ if (cdata->data->magic != SOF_ABI_MAGIC) {
+ dev_err(sdev->dev, "error: Wrong ABI magic 0x%08x.\n",
+ cdata->data->magic);
+ return -EINVAL;
+ }
+ if (SOF_ABI_VERSION_INCOMPATIBLE(SOF_ABI_VERSION,
+ cdata->data->abi)) {
+ dev_err(sdev->dev,
+ "error: Incompatible ABI version 0x%08x.\n",
+ cdata->data->abi);
+ return -EINVAL;
+ }
+ if (cdata->data->size + sizeof(const struct sof_abi_hdr) !=
+ le32_to_cpu(control->priv.size)) {
+ dev_err(sdev->dev,
+ "error: Conflict in bytes vs. priv size.\n");
+ return -EINVAL;
+ }
+ }
+ return 0;
+}
+
+/*
+ * Topology Token Parsing.
+ * New tokens should be added to headers and parsing tables below.
+ */
+
+struct sof_topology_token {
+ u32 token;
+ u32 type;
+ int (*get_token)(void *elem, void *object, u32 offset, u32 size);
+ u32 offset;
+ u32 size;
+};
+
+static int get_token_u32(void *elem, void *object, u32 offset, u32 size)
+{
+ struct snd_soc_tplg_vendor_value_elem *velem = elem;
+ u32 *val = (u32 *)((u8 *)object + offset);
+
+ *val = le32_to_cpu(velem->value);
+ return 0;
+}
+
+static int get_token_u16(void *elem, void *object, u32 offset, u32 size)
+{
+ struct snd_soc_tplg_vendor_value_elem *velem = elem;
+ u16 *val = (u16 *)((u8 *)object + offset);
+
+ *val = (u16)le32_to_cpu(velem->value);
+ return 0;
+}
+
+static int get_token_comp_format(void *elem, void *object, u32 offset, u32 size)
+{
+ struct snd_soc_tplg_vendor_string_elem *velem = elem;
+ u32 *val = (u32 *)((u8 *)object + offset);
+
+ *val = find_format(velem->string);
+ return 0;
+}
+
+static int get_token_dai_type(void *elem, void *object, u32 offset, u32 size)
+{
+ struct snd_soc_tplg_vendor_string_elem *velem = elem;
+ u32 *val = (u32 *)((u8 *)object + offset);
+
+ *val = find_dai(velem->string);
+ return 0;
+}
+
+static int get_token_process_type(void *elem, void *object, u32 offset,
+ u32 size)
+{
+ struct snd_soc_tplg_vendor_string_elem *velem = elem;
+ u32 *val = (u32 *)((u8 *)object + offset);
+
+ *val = find_process(velem->string);
+ return 0;
+}
+
+/* Buffers */
+static const struct sof_topology_token buffer_tokens[] = {
+ {SOF_TKN_BUF_SIZE, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_buffer, size), 0},
+ {SOF_TKN_BUF_CAPS, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_buffer, caps), 0},
+};
+
+/* DAI */
+static const struct sof_topology_token dai_tokens[] = {
+ {SOF_TKN_DAI_TYPE, SND_SOC_TPLG_TUPLE_TYPE_STRING, get_token_dai_type,
+ offsetof(struct sof_ipc_comp_dai, type), 0},
+ {SOF_TKN_DAI_INDEX, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_comp_dai, dai_index), 0},
+ {SOF_TKN_DAI_DIRECTION, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_comp_dai, direction), 0},
+};
+
+/* BE DAI link */
+static const struct sof_topology_token dai_link_tokens[] = {
+ {SOF_TKN_DAI_TYPE, SND_SOC_TPLG_TUPLE_TYPE_STRING, get_token_dai_type,
+ offsetof(struct sof_ipc_dai_config, type), 0},
+ {SOF_TKN_DAI_INDEX, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_dai_config, dai_index), 0},
+};
+
+/* scheduling */
+static const struct sof_topology_token sched_tokens[] = {
+ {SOF_TKN_SCHED_PERIOD, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_pipe_new, period), 0},
+ {SOF_TKN_SCHED_PRIORITY, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_pipe_new, priority), 0},
+ {SOF_TKN_SCHED_MIPS, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_pipe_new, period_mips), 0},
+ {SOF_TKN_SCHED_CORE, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_pipe_new, core), 0},
+ {SOF_TKN_SCHED_FRAMES, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_pipe_new, frames_per_sched), 0},
+ {SOF_TKN_SCHED_TIME_DOMAIN, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_pipe_new, time_domain), 0},
+};
+
+/* volume */
+static const struct sof_topology_token volume_tokens[] = {
+ {SOF_TKN_VOLUME_RAMP_STEP_TYPE, SND_SOC_TPLG_TUPLE_TYPE_WORD,
+ get_token_u32, offsetof(struct sof_ipc_comp_volume, ramp), 0},
+ {SOF_TKN_VOLUME_RAMP_STEP_MS,
+ SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_comp_volume, initial_ramp), 0},
+};
+
+/* SRC */
+static const struct sof_topology_token src_tokens[] = {
+ {SOF_TKN_SRC_RATE_IN, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_comp_src, source_rate), 0},
+ {SOF_TKN_SRC_RATE_OUT, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_comp_src, sink_rate), 0},
+};
+
+/* Tone */
+static const struct sof_topology_token tone_tokens[] = {
+};
+
+/* EFFECT */
+static const struct sof_topology_token process_tokens[] = {
+ {SOF_TKN_PROCESS_TYPE, SND_SOC_TPLG_TUPLE_TYPE_STRING,
+ get_token_process_type,
+ offsetof(struct sof_ipc_comp_process, type), 0},
+};
+
+/* PCM */
+static const struct sof_topology_token pcm_tokens[] = {
+ {SOF_TKN_PCM_DMAC_CONFIG, SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_comp_host, dmac_config), 0},
+};
+
+/* Generic components */
+static const struct sof_topology_token comp_tokens[] = {
+ {SOF_TKN_COMP_PERIOD_SINK_COUNT,
+ SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_comp_config, periods_sink), 0},
+ {SOF_TKN_COMP_PERIOD_SOURCE_COUNT,
+ SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_comp_config, periods_source), 0},
+ {SOF_TKN_COMP_FORMAT,
+ SND_SOC_TPLG_TUPLE_TYPE_STRING, get_token_comp_format,
+ offsetof(struct sof_ipc_comp_config, frame_fmt), 0},
+};
+
+/* SSP */
+static const struct sof_topology_token ssp_tokens[] = {
+ {SOF_TKN_INTEL_SSP_CLKS_CONTROL,
+ SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_dai_ssp_params, clks_control), 0},
+ {SOF_TKN_INTEL_SSP_MCLK_ID,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_ssp_params, mclk_id), 0},
+ {SOF_TKN_INTEL_SSP_SAMPLE_BITS, SND_SOC_TPLG_TUPLE_TYPE_WORD,
+ get_token_u32,
+ offsetof(struct sof_ipc_dai_ssp_params, sample_valid_bits), 0},
+ {SOF_TKN_INTEL_SSP_FRAME_PULSE_WIDTH, SND_SOC_TPLG_TUPLE_TYPE_SHORT,
+ get_token_u16,
+ offsetof(struct sof_ipc_dai_ssp_params, frame_pulse_width), 0},
+ {SOF_TKN_INTEL_SSP_QUIRKS, SND_SOC_TPLG_TUPLE_TYPE_WORD,
+ get_token_u32,
+ offsetof(struct sof_ipc_dai_ssp_params, quirks), 0},
+ {SOF_TKN_INTEL_SSP_TDM_PADDING_PER_SLOT, SND_SOC_TPLG_TUPLE_TYPE_BOOL,
+ get_token_u16,
+ offsetof(struct sof_ipc_dai_ssp_params,
+ tdm_per_slot_padding_flag), 0},
+
+};
+
+/* DMIC */
+static const struct sof_topology_token dmic_tokens[] = {
+ {SOF_TKN_INTEL_DMIC_DRIVER_VERSION,
+ SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_dai_dmic_params, driver_ipc_version),
+ 0},
+ {SOF_TKN_INTEL_DMIC_CLK_MIN,
+ SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_dai_dmic_params, pdmclk_min), 0},
+ {SOF_TKN_INTEL_DMIC_CLK_MAX,
+ SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_dai_dmic_params, pdmclk_max), 0},
+ {SOF_TKN_INTEL_DMIC_SAMPLE_RATE,
+ SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_dai_dmic_params, fifo_fs), 0},
+ {SOF_TKN_INTEL_DMIC_DUTY_MIN,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_dmic_params, duty_min), 0},
+ {SOF_TKN_INTEL_DMIC_DUTY_MAX,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_dmic_params, duty_max), 0},
+ {SOF_TKN_INTEL_DMIC_NUM_PDM_ACTIVE,
+ SND_SOC_TPLG_TUPLE_TYPE_WORD, get_token_u32,
+ offsetof(struct sof_ipc_dai_dmic_params,
+ num_pdm_active), 0},
+ {SOF_TKN_INTEL_DMIC_FIFO_WORD_LENGTH,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_dmic_params, fifo_bits), 0},
+};
+
+/*
+ * DMIC PDM Tokens
+ * SOF_TKN_INTEL_DMIC_PDM_CTRL_ID should be the first token
+ * as it increments the index while parsing the array of pdm tokens
+ * and determines the correct offset
+ */
+static const struct sof_topology_token dmic_pdm_tokens[] = {
+ {SOF_TKN_INTEL_DMIC_PDM_CTRL_ID,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_dmic_pdm_ctrl, id),
+ 0},
+ {SOF_TKN_INTEL_DMIC_PDM_MIC_A_Enable,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_dmic_pdm_ctrl, enable_mic_a),
+ 0},
+ {SOF_TKN_INTEL_DMIC_PDM_MIC_B_Enable,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_dmic_pdm_ctrl, enable_mic_b),
+ 0},
+ {SOF_TKN_INTEL_DMIC_PDM_POLARITY_A,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_dmic_pdm_ctrl, polarity_mic_a),
+ 0},
+ {SOF_TKN_INTEL_DMIC_PDM_POLARITY_B,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_dmic_pdm_ctrl, polarity_mic_b),
+ 0},
+ {SOF_TKN_INTEL_DMIC_PDM_CLK_EDGE,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_dmic_pdm_ctrl, clk_edge),
+ 0},
+ {SOF_TKN_INTEL_DMIC_PDM_SKEW,
+ SND_SOC_TPLG_TUPLE_TYPE_SHORT, get_token_u16,
+ offsetof(struct sof_ipc_dai_dmic_pdm_ctrl, skew),
+ 0},
+};
+
+/* HDA */
+static const struct sof_topology_token hda_tokens[] = {
+};
+
+static void sof_parse_uuid_tokens(struct snd_soc_component *scomp,
+ void *object,
+ const struct sof_topology_token *tokens,
+ int count,
+ struct snd_soc_tplg_vendor_array *array)
+{
+ struct snd_soc_tplg_vendor_uuid_elem *elem;
+ int i, j;
+
+ /* parse element by element */
+ for (i = 0; i < le32_to_cpu(array->num_elems); i++) {
+ elem = &array->uuid[i];
+
+ /* search for token */
+ for (j = 0; j < count; j++) {
+ /* match token type */
+ if (tokens[j].type != SND_SOC_TPLG_TUPLE_TYPE_UUID)
+ continue;
+
+ /* match token id */
+ if (tokens[j].token != le32_to_cpu(elem->token))
+ continue;
+
+ /* matched - now load token */
+ tokens[j].get_token(elem, object, tokens[j].offset,
+ tokens[j].size);
+ }
+ }
+}
+
+static void sof_parse_string_tokens(struct snd_soc_component *scomp,
+ void *object,
+ const struct sof_topology_token *tokens,
+ int count,
+ struct snd_soc_tplg_vendor_array *array)
+{
+ struct snd_soc_tplg_vendor_string_elem *elem;
+ int i, j;
+
+ /* parse element by element */
+ for (i = 0; i < le32_to_cpu(array->num_elems); i++) {
+ elem = &array->string[i];
+
+ /* search for token */
+ for (j = 0; j < count; j++) {
+ /* match token type */
+ if (tokens[j].type != SND_SOC_TPLG_TUPLE_TYPE_STRING)
+ continue;
+
+ /* match token id */
+ if (tokens[j].token != le32_to_cpu(elem->token))
+ continue;
+
+ /* matched - now load token */
+ tokens[j].get_token(elem, object, tokens[j].offset,
+ tokens[j].size);
+ }
+ }
+}
+
+static void sof_parse_word_tokens(struct snd_soc_component *scomp,
+ void *object,
+ const struct sof_topology_token *tokens,
+ int count,
+ struct snd_soc_tplg_vendor_array *array)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_vendor_value_elem *elem;
+ size_t size = sizeof(struct sof_ipc_dai_dmic_pdm_ctrl);
+ int i, j;
+ u32 offset;
+ u32 *index = NULL;
+
+ /* parse element by element */
+ for (i = 0; i < le32_to_cpu(array->num_elems); i++) {
+ elem = &array->value[i];
+
+ /* search for token */
+ for (j = 0; j < count; j++) {
+ /* match token type */
+ if (!(tokens[j].type == SND_SOC_TPLG_TUPLE_TYPE_WORD ||
+ tokens[j].type == SND_SOC_TPLG_TUPLE_TYPE_SHORT))
+ continue;
+
+ /* match token id */
+ if (tokens[j].token != le32_to_cpu(elem->token))
+ continue;
+
+ /* pdm config array index */
+ if (sdev->private)
+ index = sdev->private;
+
+ /* matched - determine offset */
+ switch (tokens[j].token) {
+ case SOF_TKN_INTEL_DMIC_PDM_CTRL_ID:
+
+ /* inc number of pdm array index */
+ if (index)
+ (*index)++;
+ /* fallthrough */
+ case SOF_TKN_INTEL_DMIC_PDM_MIC_A_Enable:
+ case SOF_TKN_INTEL_DMIC_PDM_MIC_B_Enable:
+ case SOF_TKN_INTEL_DMIC_PDM_POLARITY_A:
+ case SOF_TKN_INTEL_DMIC_PDM_POLARITY_B:
+ case SOF_TKN_INTEL_DMIC_PDM_CLK_EDGE:
+ case SOF_TKN_INTEL_DMIC_PDM_SKEW:
+
+ /* check if array index is valid */
+ if (!index || *index == 0) {
+ dev_err(sdev->dev,
+ "error: invalid array offset\n");
+ continue;
+ } else {
+ /* offset within the pdm config array */
+ offset = size * (*index - 1);
+ }
+ break;
+ default:
+ offset = 0;
+ break;
+ }
+
+ /* load token */
+ tokens[j].get_token(elem, object,
+ offset + tokens[j].offset,
+ tokens[j].size);
+ }
+ }
+}
+
+static int sof_parse_tokens(struct snd_soc_component *scomp,
+ void *object,
+ const struct sof_topology_token *tokens,
+ int count,
+ struct snd_soc_tplg_vendor_array *array,
+ int priv_size)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ int asize;
+
+ while (priv_size > 0) {
+ asize = le32_to_cpu(array->size);
+
+ /* validate asize */
+ if (asize < 0) { /* FIXME: A zero-size array makes no sense */
+ dev_err(sdev->dev, "error: invalid array size 0x%x\n",
+ asize);
+ return -EINVAL;
+ }
+
+ /* make sure there is enough data before parsing */
+ priv_size -= asize;
+ if (priv_size < 0) {
+ dev_err(sdev->dev, "error: invalid array size 0x%x\n",
+ asize);
+ return -EINVAL;
+ }
+
+ /* call correct parser depending on type */
+ switch (le32_to_cpu(array->type)) {
+ case SND_SOC_TPLG_TUPLE_TYPE_UUID:
+ sof_parse_uuid_tokens(scomp, object, tokens, count,
+ array);
+ break;
+ case SND_SOC_TPLG_TUPLE_TYPE_STRING:
+ sof_parse_string_tokens(scomp, object, tokens, count,
+ array);
+ break;
+ case SND_SOC_TPLG_TUPLE_TYPE_BOOL:
+ case SND_SOC_TPLG_TUPLE_TYPE_BYTE:
+ case SND_SOC_TPLG_TUPLE_TYPE_WORD:
+ case SND_SOC_TPLG_TUPLE_TYPE_SHORT:
+ sof_parse_word_tokens(scomp, object, tokens, count,
+ array);
+ break;
+ default:
+ dev_err(sdev->dev, "error: unknown token type %d\n",
+ array->type);
+ return -EINVAL;
+ }
+
+ /* next array */
+ array = (struct snd_soc_tplg_vendor_array *)((u8 *)array
+ + asize);
+ }
+ return 0;
+}
+
+static void sof_dbg_comp_config(struct snd_soc_component *scomp,
+ struct sof_ipc_comp_config *config)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+
+ dev_dbg(sdev->dev, " config: periods snk %d src %d fmt %d\n",
+ config->periods_sink, config->periods_source,
+ config->frame_fmt);
+}
+
+/* external kcontrol init - used for any driver specific init */
+static int sof_control_load(struct snd_soc_component *scomp, int index,
+ struct snd_kcontrol_new *kc,
+ struct snd_soc_tplg_ctl_hdr *hdr)
+{
+ struct soc_mixer_control *sm;
+ struct soc_bytes_ext *sbe;
+ struct soc_enum *se;
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_dobj *dobj;
+ struct snd_sof_control *scontrol;
+ int ret = -EINVAL;
+
+ dev_dbg(sdev->dev, "tplg: load control type %d name : %s\n",
+ hdr->type, hdr->name);
+
+ scontrol = kzalloc(sizeof(*scontrol), GFP_KERNEL);
+ if (!scontrol)
+ return -ENOMEM;
+
+ scontrol->sdev = sdev;
+
+ switch (le32_to_cpu(hdr->ops.info)) {
+ case SND_SOC_TPLG_CTL_VOLSW:
+ case SND_SOC_TPLG_CTL_VOLSW_SX:
+ case SND_SOC_TPLG_CTL_VOLSW_XR_SX:
+ sm = (struct soc_mixer_control *)kc->private_value;
+ dobj = &sm->dobj;
+ ret = sof_control_load_volume(scomp, scontrol, kc, hdr);
+ break;
+ case SND_SOC_TPLG_CTL_BYTES:
+ sbe = (struct soc_bytes_ext *)kc->private_value;
+ dobj = &sbe->dobj;
+ ret = sof_control_load_bytes(scomp, scontrol, kc, hdr);
+ break;
+ case SND_SOC_TPLG_CTL_ENUM:
+ case SND_SOC_TPLG_CTL_ENUM_VALUE:
+ se = (struct soc_enum *)kc->private_value;
+ dobj = &se->dobj;
+ ret = sof_control_load_enum(scomp, scontrol, kc, hdr);
+ break;
+ case SND_SOC_TPLG_CTL_RANGE:
+ case SND_SOC_TPLG_CTL_STROBE:
+ case SND_SOC_TPLG_DAPM_CTL_VOLSW:
+ case SND_SOC_TPLG_DAPM_CTL_ENUM_DOUBLE:
+ case SND_SOC_TPLG_DAPM_CTL_ENUM_VIRT:
+ case SND_SOC_TPLG_DAPM_CTL_ENUM_VALUE:
+ case SND_SOC_TPLG_DAPM_CTL_PIN:
+ default:
+ dev_warn(sdev->dev, "control type not supported %d:%d:%d\n",
+ hdr->ops.get, hdr->ops.put, hdr->ops.info);
+ kfree(scontrol);
+ return 0;
+ }
+
+ dobj->private = scontrol;
+ list_add(&scontrol->list, &sdev->kcontrol_list);
+ return ret;
+}
+
+static int sof_control_unload(struct snd_soc_component *scomp,
+ struct snd_soc_dobj *dobj)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct sof_ipc_free fcomp;
+ struct snd_sof_control *scontrol = dobj->private;
+
+ dev_dbg(sdev->dev, "tplg: unload control name : %s\n", scomp->name);
+
+ fcomp.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_COMP_FREE;
+ fcomp.hdr.size = sizeof(fcomp);
+ fcomp.id = scontrol->comp_id;
+
+ kfree(scontrol->control_data);
+ list_del(&scontrol->list);
+ kfree(scontrol);
+ /* send IPC to the DSP */
+ return sof_ipc_tx_message(sdev->ipc,
+ fcomp.hdr.cmd, &fcomp, sizeof(fcomp),
+ NULL, 0);
+}
+
+/*
+ * DAI Topology
+ */
+
+static int sof_connect_dai_widget(struct snd_soc_component *scomp,
+ struct snd_soc_dapm_widget *w,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct snd_sof_dai *dai)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_card *card = scomp->card;
+ struct snd_soc_pcm_runtime *rtd;
+
+ list_for_each_entry(rtd, &card->rtd_list, list) {
+ dev_vdbg(sdev->dev, "tplg: check widget: %s stream: %s dai stream: %s\n",
+ w->name, w->sname, rtd->dai_link->stream_name);
+
+ if (!w->sname || !rtd->dai_link->stream_name)
+ continue;
+
+ /* does stream match DAI link ? */
+ if (strcmp(w->sname, rtd->dai_link->stream_name))
+ continue;
+
+ switch (w->id) {
+ case snd_soc_dapm_dai_out:
+ rtd->cpu_dai->capture_widget = w;
+ dai->name = rtd->dai_link->name;
+ dev_dbg(sdev->dev, "tplg: connected widget %s -> DAI link %s\n",
+ w->name, rtd->dai_link->name);
+ break;
+ case snd_soc_dapm_dai_in:
+ rtd->cpu_dai->playback_widget = w;
+ dai->name = rtd->dai_link->name;
+ dev_dbg(sdev->dev, "tplg: connected widget %s -> DAI link %s\n",
+ w->name, rtd->dai_link->name);
+ break;
+ default:
+ break;
+ }
+ }
+
+ /* check we have a connection */
+ if (!dai->name) {
+ dev_err(sdev->dev, "error: can't connect DAI %s stream %s\n",
+ w->name, w->sname);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int sof_widget_load_dai(struct snd_soc_component *scomp, int index,
+ struct snd_sof_widget *swidget,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r,
+ struct snd_sof_dai *dai)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct sof_ipc_comp_dai comp_dai;
+ int ret;
+
+ /* configure dai IPC message */
+ memset(&comp_dai, 0, sizeof(comp_dai));
+ comp_dai.comp.hdr.size = sizeof(comp_dai);
+ comp_dai.comp.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_COMP_NEW;
+ comp_dai.comp.id = swidget->comp_id;
+ comp_dai.comp.type = SOF_COMP_DAI;
+ comp_dai.comp.pipeline_id = index;
+ comp_dai.config.hdr.size = sizeof(comp_dai.config);
+
+ ret = sof_parse_tokens(scomp, &comp_dai, dai_tokens,
+ ARRAY_SIZE(dai_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse dai tokens failed %d\n",
+ le32_to_cpu(private->size));
+ return ret;
+ }
+
+ ret = sof_parse_tokens(scomp, &comp_dai.config, comp_tokens,
+ ARRAY_SIZE(comp_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse dai.cfg tokens failed %d\n",
+ private->size);
+ return ret;
+ }
+
+ dev_dbg(sdev->dev, "dai %s: type %d index %d\n",
+ swidget->widget->name, comp_dai.type, comp_dai.dai_index);
+ sof_dbg_comp_config(scomp, &comp_dai.config);
+
+ ret = sof_ipc_tx_message(sdev->ipc, comp_dai.comp.hdr.cmd,
+ &comp_dai, sizeof(comp_dai), r, sizeof(*r));
+
+ if (ret == 0 && dai) {
+ dai->sdev = sdev;
+ memcpy(&dai->comp_dai, &comp_dai, sizeof(comp_dai));
+ }
+
+ return ret;
+}
+
+/*
+ * Buffer topology
+ */
+
+static int sof_widget_load_buffer(struct snd_soc_component *scomp, int index,
+ struct snd_sof_widget *swidget,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct sof_ipc_buffer *buffer;
+ int ret;
+
+ buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
+ if (!buffer)
+ return -ENOMEM;
+
+ /* configure dai IPC message */
+ buffer->comp.hdr.size = sizeof(*buffer);
+ buffer->comp.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_BUFFER_NEW;
+ buffer->comp.id = swidget->comp_id;
+ buffer->comp.type = SOF_COMP_BUFFER;
+ buffer->comp.pipeline_id = index;
+
+ ret = sof_parse_tokens(scomp, buffer, buffer_tokens,
+ ARRAY_SIZE(buffer_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse buffer tokens failed %d\n",
+ private->size);
+ kfree(buffer);
+ return ret;
+ }
+
+ dev_dbg(sdev->dev, "buffer %s: size %d caps 0x%x\n",
+ swidget->widget->name, buffer->size, buffer->caps);
+
+ swidget->private = buffer;
+
+ ret = sof_ipc_tx_message(sdev->ipc, buffer->comp.hdr.cmd, buffer,
+ sizeof(*buffer), r, sizeof(*r));
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: buffer %s load failed\n",
+ swidget->widget->name);
+ kfree(buffer);
+ }
+
+ return ret;
+}
+
+/* bind PCM ID to host component ID */
+static int spcm_bind(struct snd_sof_dev *sdev, struct snd_sof_pcm *spcm,
+ int dir)
+{
+ struct snd_sof_widget *host_widget;
+
+ host_widget = snd_sof_find_swidget_sname(sdev,
+ spcm->pcm.caps[dir].name,
+ dir);
+ if (!host_widget) {
+ dev_err(sdev->dev, "can't find host comp to bind pcm\n");
+ return -EINVAL;
+ }
+
+ spcm->stream[dir].comp_id = host_widget->comp_id;
+
+ return 0;
+}
+
+/*
+ * PCM Topology
+ */
+
+static int sof_widget_load_pcm(struct snd_soc_component *scomp, int index,
+ struct snd_sof_widget *swidget,
+ enum sof_ipc_stream_direction dir,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct sof_ipc_comp_host *host;
+ int ret;
+
+ host = kzalloc(sizeof(*host), GFP_KERNEL);
+ if (!host)
+ return -ENOMEM;
+
+ /* configure host comp IPC message */
+ host->comp.hdr.size = sizeof(*host);
+ host->comp.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_COMP_NEW;
+ host->comp.id = swidget->comp_id;
+ host->comp.type = SOF_COMP_HOST;
+ host->comp.pipeline_id = index;
+ host->direction = dir;
+ host->config.hdr.size = sizeof(host->config);
+
+ ret = sof_parse_tokens(scomp, host, pcm_tokens,
+ ARRAY_SIZE(pcm_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse host tokens failed %d\n",
+ private->size);
+ goto err;
+ }
+
+ ret = sof_parse_tokens(scomp, &host->config, comp_tokens,
+ ARRAY_SIZE(comp_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse host.cfg tokens failed %d\n",
+ le32_to_cpu(private->size));
+ goto err;
+ }
+
+ dev_dbg(sdev->dev, "loaded host %s\n", swidget->widget->name);
+ sof_dbg_comp_config(scomp, &host->config);
+
+ swidget->private = host;
+
+ ret = sof_ipc_tx_message(sdev->ipc, host->comp.hdr.cmd, host,
+ sizeof(*host), r, sizeof(*r));
+ if (ret >= 0)
+ return ret;
+err:
+ kfree(host);
+ return ret;
+}
+
+/*
+ * Pipeline Topology
+ */
+int sof_load_pipeline_ipc(struct snd_sof_dev *sdev,
+ struct sof_ipc_pipe_new *pipeline,
+ struct sof_ipc_comp_reply *r)
+{
+ struct sof_ipc_pm_core_config pm_core_config;
+ int ret;
+
+ ret = sof_ipc_tx_message(sdev->ipc, pipeline->hdr.cmd, pipeline,
+ sizeof(*pipeline), r, sizeof(*r));
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: load pipeline ipc failure\n");
+ return ret;
+ }
+
+ /* power up the core that this pipeline is scheduled on */
+ ret = snd_sof_dsp_core_power_up(sdev, 1 << pipeline->core);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: powering up pipeline schedule core %d\n",
+ pipeline->core);
+ return ret;
+ }
+
+ /* update enabled cores mask */
+ sdev->enabled_cores_mask |= 1 << pipeline->core;
+
+ /*
+ * Now notify DSP that the core that this pipeline is scheduled on
+ * has been powered up
+ */
+ memset(&pm_core_config, 0, sizeof(pm_core_config));
+ pm_core_config.enable_mask = sdev->enabled_cores_mask;
+
+ /* configure CORE_ENABLE ipc message */
+ pm_core_config.hdr.size = sizeof(pm_core_config);
+ pm_core_config.hdr.cmd = SOF_IPC_GLB_PM_MSG | SOF_IPC_PM_CORE_ENABLE;
+
+ /* send ipc */
+ ret = sof_ipc_tx_message(sdev->ipc, pm_core_config.hdr.cmd,
+ &pm_core_config, sizeof(pm_core_config),
+ &pm_core_config, sizeof(pm_core_config));
+ if (ret < 0)
+ dev_err(sdev->dev, "error: core enable ipc failure\n");
+
+ return ret;
+}
+
+static int sof_widget_load_pipeline(struct snd_soc_component *scomp,
+ int index, struct snd_sof_widget *swidget,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct sof_ipc_pipe_new *pipeline;
+ struct snd_sof_widget *comp_swidget;
+ int ret;
+
+ pipeline = kzalloc(sizeof(*pipeline), GFP_KERNEL);
+ if (!pipeline)
+ return -ENOMEM;
+
+ /* configure dai IPC message */
+ pipeline->hdr.size = sizeof(*pipeline);
+ pipeline->hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_PIPE_NEW;
+ pipeline->pipeline_id = index;
+ pipeline->comp_id = swidget->comp_id;
+
+ /* component at start of pipeline is our stream id */
+ comp_swidget = snd_sof_find_swidget(sdev, tw->sname);
+ if (!comp_swidget) {
+ dev_err(sdev->dev, "error: widget %s refers to non existent widget %s\n",
+ tw->name, tw->sname);
+ ret = -EINVAL;
+ goto err;
+ }
+
+ pipeline->sched_id = comp_swidget->comp_id;
+
+ dev_dbg(sdev->dev, "tplg: pipeline id %d comp %d scheduling comp id %d\n",
+ pipeline->pipeline_id, pipeline->comp_id, pipeline->sched_id);
+
+ ret = sof_parse_tokens(scomp, pipeline, sched_tokens,
+ ARRAY_SIZE(sched_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse pipeline tokens failed %d\n",
+ private->size);
+ goto err;
+ }
+
+ dev_dbg(sdev->dev, "pipeline %s: period %d pri %d mips %d core %d frames %d\n",
+ swidget->widget->name, pipeline->period, pipeline->priority,
+ pipeline->period_mips, pipeline->core, pipeline->frames_per_sched);
+
+ swidget->private = pipeline;
+
+ /* send ipc's to create pipeline comp and power up schedule core */
+ ret = sof_load_pipeline_ipc(sdev, pipeline, r);
+ if (ret >= 0)
+ return ret;
+err:
+ kfree(pipeline);
+ return ret;
+}
+
+/*
+ * Mixer topology
+ */
+
+static int sof_widget_load_mixer(struct snd_soc_component *scomp, int index,
+ struct snd_sof_widget *swidget,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct sof_ipc_comp_mixer *mixer;
+ int ret;
+
+ mixer = kzalloc(sizeof(*mixer), GFP_KERNEL);
+ if (!mixer)
+ return -ENOMEM;
+
+ /* configure mixer IPC message */
+ mixer->comp.hdr.size = sizeof(*mixer);
+ mixer->comp.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_COMP_NEW;
+ mixer->comp.id = swidget->comp_id;
+ mixer->comp.type = SOF_COMP_MIXER;
+ mixer->comp.pipeline_id = index;
+ mixer->config.hdr.size = sizeof(mixer->config);
+
+ ret = sof_parse_tokens(scomp, &mixer->config, comp_tokens,
+ ARRAY_SIZE(comp_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse mixer.cfg tokens failed %d\n",
+ private->size);
+ kfree(mixer);
+ return ret;
+ }
+
+ sof_dbg_comp_config(scomp, &mixer->config);
+
+ swidget->private = mixer;
+
+ ret = sof_ipc_tx_message(sdev->ipc, mixer->comp.hdr.cmd, mixer,
+ sizeof(*mixer), r, sizeof(*r));
+ if (ret < 0)
+ kfree(mixer);
+
+ return ret;
+}
+
+/*
+ * Mux topology
+ */
+static int sof_widget_load_mux(struct snd_soc_component *scomp, int index,
+ struct snd_sof_widget *swidget,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct sof_ipc_comp_mux *mux;
+ int ret;
+
+ mux = kzalloc(sizeof(*mux), GFP_KERNEL);
+ if (!mux)
+ return -ENOMEM;
+
+ /* configure mux IPC message */
+ mux->comp.hdr.size = sizeof(*mux);
+ mux->comp.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_COMP_NEW;
+ mux->comp.id = swidget->comp_id;
+ mux->comp.type = SOF_COMP_MUX;
+ mux->comp.pipeline_id = index;
+ mux->config.hdr.size = sizeof(mux->config);
+
+ ret = sof_parse_tokens(scomp, &mux->config, comp_tokens,
+ ARRAY_SIZE(comp_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse mux.cfg tokens failed %d\n",
+ private->size);
+ kfree(mux);
+ return ret;
+ }
+
+ sof_dbg_comp_config(scomp, &mux->config);
+
+ swidget->private = mux;
+
+ ret = sof_ipc_tx_message(sdev->ipc, mux->comp.hdr.cmd, mux,
+ sizeof(*mux), r, sizeof(*r));
+ if (ret < 0)
+ kfree(mux);
+
+ return ret;
+}
+
+/*
+ * PGA Topology
+ */
+
+static int sof_widget_load_pga(struct snd_soc_component *scomp, int index,
+ struct snd_sof_widget *swidget,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct sof_ipc_comp_volume *volume;
+ int ret;
+
+ volume = kzalloc(sizeof(*volume), GFP_KERNEL);
+ if (!volume)
+ return -ENOMEM;
+
+ if (le32_to_cpu(tw->num_kcontrols) != 1) {
+ dev_err(sdev->dev, "error: invalid kcontrol count %d for volume\n",
+ tw->num_kcontrols);
+ ret = -EINVAL;
+ goto err;
+ }
+
+ /* configure volume IPC message */
+ volume->comp.hdr.size = sizeof(*volume);
+ volume->comp.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_COMP_NEW;
+ volume->comp.id = swidget->comp_id;
+ volume->comp.type = SOF_COMP_VOLUME;
+ volume->comp.pipeline_id = index;
+ volume->config.hdr.size = sizeof(volume->config);
+
+ ret = sof_parse_tokens(scomp, volume, volume_tokens,
+ ARRAY_SIZE(volume_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse volume tokens failed %d\n",
+ private->size);
+ goto err;
+ }
+ ret = sof_parse_tokens(scomp, &volume->config, comp_tokens,
+ ARRAY_SIZE(comp_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse volume.cfg tokens failed %d\n",
+ le32_to_cpu(private->size));
+ goto err;
+ }
+
+ sof_dbg_comp_config(scomp, &volume->config);
+
+ swidget->private = volume;
+
+ ret = sof_ipc_tx_message(sdev->ipc, volume->comp.hdr.cmd, volume,
+ sizeof(*volume), r, sizeof(*r));
+ if (ret >= 0)
+ return ret;
+err:
+ kfree(volume);
+ return ret;
+}
+
+/*
+ * SRC Topology
+ */
+
+static int sof_widget_load_src(struct snd_soc_component *scomp, int index,
+ struct snd_sof_widget *swidget,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct sof_ipc_comp_src *src;
+ int ret;
+
+ src = kzalloc(sizeof(*src), GFP_KERNEL);
+ if (!src)
+ return -ENOMEM;
+
+ /* configure src IPC message */
+ src->comp.hdr.size = sizeof(*src);
+ src->comp.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_COMP_NEW;
+ src->comp.id = swidget->comp_id;
+ src->comp.type = SOF_COMP_SRC;
+ src->comp.pipeline_id = index;
+ src->config.hdr.size = sizeof(src->config);
+
+ ret = sof_parse_tokens(scomp, src, src_tokens,
+ ARRAY_SIZE(src_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse src tokens failed %d\n",
+ private->size);
+ goto err;
+ }
+
+ ret = sof_parse_tokens(scomp, &src->config, comp_tokens,
+ ARRAY_SIZE(comp_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse src.cfg tokens failed %d\n",
+ le32_to_cpu(private->size));
+ goto err;
+ }
+
+ dev_dbg(sdev->dev, "src %s: source rate %d sink rate %d\n",
+ swidget->widget->name, src->source_rate, src->sink_rate);
+ sof_dbg_comp_config(scomp, &src->config);
+
+ swidget->private = src;
+
+ ret = sof_ipc_tx_message(sdev->ipc, src->comp.hdr.cmd, src,
+ sizeof(*src), r, sizeof(*r));
+ if (ret >= 0)
+ return ret;
+err:
+ kfree(src);
+ return ret;
+}
+
+/*
+ * Signal Generator Topology
+ */
+
+static int sof_widget_load_siggen(struct snd_soc_component *scomp, int index,
+ struct snd_sof_widget *swidget,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct sof_ipc_comp_tone *tone;
+ int ret;
+
+ tone = kzalloc(sizeof(*tone), GFP_KERNEL);
+ if (!tone)
+ return -ENOMEM;
+
+ /* configure siggen IPC message */
+ tone->comp.hdr.size = sizeof(*tone);
+ tone->comp.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_COMP_NEW;
+ tone->comp.id = swidget->comp_id;
+ tone->comp.type = SOF_COMP_TONE;
+ tone->comp.pipeline_id = index;
+ tone->config.hdr.size = sizeof(tone->config);
+
+ ret = sof_parse_tokens(scomp, tone, tone_tokens,
+ ARRAY_SIZE(tone_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse tone tokens failed %d\n",
+ le32_to_cpu(private->size));
+ goto err;
+ }
+
+ ret = sof_parse_tokens(scomp, &tone->config, comp_tokens,
+ ARRAY_SIZE(comp_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse tone.cfg tokens failed %d\n",
+ le32_to_cpu(private->size));
+ goto err;
+ }
+
+ dev_dbg(sdev->dev, "tone %s: frequency %d amplitude %d\n",
+ swidget->widget->name, tone->frequency, tone->amplitude);
+ sof_dbg_comp_config(scomp, &tone->config);
+
+ swidget->private = tone;
+
+ ret = sof_ipc_tx_message(sdev->ipc, tone->comp.hdr.cmd, tone,
+ sizeof(*tone), r, sizeof(*r));
+ if (ret >= 0)
+ return ret;
+err:
+ kfree(tone);
+ return ret;
+}
+
+static int sof_process_load(struct snd_soc_component *scomp, int index,
+ struct snd_sof_widget *swidget,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r,
+ int type)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct snd_soc_dapm_widget *widget = swidget->widget;
+ const struct snd_kcontrol_new *kc;
+ struct soc_bytes_ext *sbe;
+ struct soc_mixer_control *sm;
+ struct soc_enum *se;
+ struct snd_sof_control *scontrol = NULL;
+ struct sof_abi_hdr *pdata = NULL;
+ struct sof_ipc_comp_process *process;
+ size_t ipc_size, ipc_data_size = 0;
+ int ret, i, offset = 0;
+
+ if (type == SOF_COMP_NONE) {
+ dev_err(sdev->dev, "error: invalid process comp type %d\n",
+ type);
+ return -EINVAL;
+ }
+
+ /*
+ * get possible component controls - get size of all pdata,
+ * then memcpy with headers
+ */
+ for (i = 0; i < widget->num_kcontrols; i++) {
+
+ kc = &widget->kcontrol_news[i];
+
+ switch (widget->dobj.widget.kcontrol_type) {
+ case SND_SOC_TPLG_TYPE_MIXER:
+ sm = (struct soc_mixer_control *)kc->private_value;
+ scontrol = sm->dobj.private;
+ break;
+ case SND_SOC_TPLG_TYPE_BYTES:
+ sbe = (struct soc_bytes_ext *)kc->private_value;
+ scontrol = sbe->dobj.private;
+ break;
+ case SND_SOC_TPLG_TYPE_ENUM:
+ se = (struct soc_enum *)kc->private_value;
+ scontrol = se->dobj.private;
+ break;
+ default:
+ dev_err(sdev->dev, "error: unknown kcontrol type %d in widget %s\n",
+ widget->dobj.widget.kcontrol_type,
+ widget->name);
+ return -EINVAL;
+ }
+
+ if (!scontrol) {
+ dev_err(sdev->dev, "error: no scontrol for widget %s\n",
+ widget->name);
+ return -EINVAL;
+ }
+
+ /* don't include if no private data */
+ pdata = scontrol->control_data->data;
+ if (!pdata)
+ continue;
+
+ /* make sure data is valid - data can be updated at runtime */
+ if (pdata->magic != SOF_ABI_MAGIC)
+ continue;
+
+ ipc_data_size += pdata->size;
+ }
+
+ ipc_size = sizeof(struct sof_ipc_comp_process) +
+ le32_to_cpu(private->size) +
+ ipc_data_size;
+
+ process = kzalloc(ipc_size, GFP_KERNEL);
+ if (!process)
+ return -ENOMEM;
+
+ /* configure iir IPC message */
+ process->comp.hdr.size = ipc_size;
+ process->comp.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_COMP_NEW;
+ process->comp.id = swidget->comp_id;
+ process->comp.type = type;
+ process->comp.pipeline_id = index;
+ process->config.hdr.size = sizeof(process->config);
+
+ ret = sof_parse_tokens(scomp, &process->config, comp_tokens,
+ ARRAY_SIZE(comp_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse process.cfg tokens failed %d\n",
+ le32_to_cpu(private->size));
+ goto err;
+ }
+
+ sof_dbg_comp_config(scomp, &process->config);
+
+ /*
+ * found private data in control, so copy it.
+ * get possible component controls - get size of all pdata,
+ * then memcpy with headers
+ */
+ for (i = 0; i < widget->num_kcontrols; i++) {
+ kc = &widget->kcontrol_news[i];
+
+ switch (widget->dobj.widget.kcontrol_type) {
+ case SND_SOC_TPLG_TYPE_MIXER:
+ sm = (struct soc_mixer_control *)kc->private_value;
+ scontrol = sm->dobj.private;
+ break;
+ case SND_SOC_TPLG_TYPE_BYTES:
+ sbe = (struct soc_bytes_ext *)kc->private_value;
+ scontrol = sbe->dobj.private;
+ break;
+ case SND_SOC_TPLG_TYPE_ENUM:
+ se = (struct soc_enum *)kc->private_value;
+ scontrol = se->dobj.private;
+ break;
+ default:
+ dev_err(sdev->dev, "error: unknown kcontrol type %d in widget %s\n",
+ widget->dobj.widget.kcontrol_type,
+ widget->name);
+ return -EINVAL;
+ }
+
+ /* don't include if no private data */
+ pdata = scontrol->control_data->data;
+ if (!pdata)
+ continue;
+
+ /* make sure data is valid - data can be updated at runtime */
+ if (pdata->magic != SOF_ABI_MAGIC)
+ continue;
+
+ memcpy(&process->data + offset, pdata->data, pdata->size);
+ offset += pdata->size;
+ }
+
+ process->size = ipc_data_size;
+ swidget->private = process;
+
+ ret = sof_ipc_tx_message(sdev->ipc, process->comp.hdr.cmd, process,
+ ipc_size, r, sizeof(*r));
+ if (ret >= 0)
+ return ret;
+err:
+ kfree(process);
+ return ret;
+}
+
+/*
+ * Processing Component Topology - can be "effect", "codec", or general
+ * "processing".
+ */
+
+static int sof_widget_load_process(struct snd_soc_component *scomp, int index,
+ struct snd_sof_widget *swidget,
+ struct snd_soc_tplg_dapm_widget *tw,
+ struct sof_ipc_comp_reply *r)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &tw->priv;
+ struct sof_ipc_comp_process config;
+ int ret;
+
+ /* check we have some tokens - we need at least process type */
+ if (le32_to_cpu(private->size) == 0) {
+ dev_err(sdev->dev, "error: process tokens not found\n");
+ return -EINVAL;
+ }
+
+ memset(&config, 0, sizeof(config));
+
+ /* get the process token */
+ ret = sof_parse_tokens(scomp, &config, process_tokens,
+ ARRAY_SIZE(process_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse process tokens failed %d\n",
+ le32_to_cpu(private->size));
+ return ret;
+ }
+
+ /* now load process specific data and send IPC */
+ ret = sof_process_load(scomp, index, swidget, tw, r,
+ find_process_comp_type(config.type));
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: process loading failed\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int sof_widget_bind_event(struct snd_sof_dev *sdev,
+ struct snd_sof_widget *swidget,
+ u16 event_type)
+{
+ struct sof_ipc_comp *ipc_comp;
+
+ /* validate widget event type */
+ switch (event_type) {
+ case SOF_KEYWORD_DETECT_DAPM_EVENT:
+ /* only KEYWORD_DETECT comps should handle this */
+ if (swidget->id != snd_soc_dapm_effect)
+ break;
+
+ ipc_comp = swidget->private;
+ if (ipc_comp && ipc_comp->type != SOF_COMP_KEYWORD_DETECT)
+ break;
+
+ /* bind event to keyword detect comp */
+ return snd_soc_tplg_widget_bind_event(swidget->widget,
+ sof_kwd_events,
+ ARRAY_SIZE(sof_kwd_events),
+ event_type);
+ default:
+ break;
+ }
+
+ dev_err(sdev->dev,
+ "error: invalid event type %d for widget %s\n",
+ event_type, swidget->widget->name);
+ return -EINVAL;
+}
+
+/* external widget init - used for any driver specific init */
+static int sof_widget_ready(struct snd_soc_component *scomp, int index,
+ struct snd_soc_dapm_widget *w,
+ struct snd_soc_tplg_dapm_widget *tw)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_sof_widget *swidget;
+ struct snd_sof_dai *dai;
+ struct sof_ipc_comp_reply reply;
+ struct snd_sof_control *scontrol;
+ int ret = 0;
+
+ swidget = kzalloc(sizeof(*swidget), GFP_KERNEL);
+ if (!swidget)
+ return -ENOMEM;
+
+ swidget->sdev = sdev;
+ swidget->widget = w;
+ swidget->comp_id = sdev->next_comp_id++;
+ swidget->complete = 0;
+ swidget->id = w->id;
+ swidget->pipeline_id = index;
+ swidget->private = NULL;
+ memset(&reply, 0, sizeof(reply));
+
+ dev_dbg(sdev->dev, "tplg: ready widget id %d pipe %d type %d name : %s stream %s\n",
+ swidget->comp_id, index, swidget->id, tw->name,
+ strnlen(tw->sname, SNDRV_CTL_ELEM_ID_NAME_MAXLEN) > 0
+ ? tw->sname : "none");
+
+ /* handle any special case widgets */
+ switch (w->id) {
+ case snd_soc_dapm_dai_in:
+ case snd_soc_dapm_dai_out:
+ dai = kzalloc(sizeof(*dai), GFP_KERNEL);
+ if (!dai) {
+ kfree(swidget);
+ return -ENOMEM;
+ }
+
+ ret = sof_widget_load_dai(scomp, index, swidget, tw, &reply,
+ dai);
+ if (ret == 0) {
+ sof_connect_dai_widget(scomp, w, tw, dai);
+ list_add(&dai->list, &sdev->dai_list);
+ swidget->private = dai;
+ } else {
+ kfree(dai);
+ }
+ break;
+ case snd_soc_dapm_mixer:
+ ret = sof_widget_load_mixer(scomp, index, swidget, tw, &reply);
+ break;
+ case snd_soc_dapm_pga:
+ ret = sof_widget_load_pga(scomp, index, swidget, tw, &reply);
+ /* Find scontrol for this pga and set readback offset*/
+ list_for_each_entry(scontrol, &sdev->kcontrol_list, list) {
+ if (scontrol->comp_id == swidget->comp_id) {
+ scontrol->readback_offset = reply.offset;
+ break;
+ }
+ }
+ break;
+ case snd_soc_dapm_buffer:
+ ret = sof_widget_load_buffer(scomp, index, swidget, tw, &reply);
+ break;
+ case snd_soc_dapm_scheduler:
+ ret = sof_widget_load_pipeline(scomp, index, swidget, tw,
+ &reply);
+ break;
+ case snd_soc_dapm_aif_out:
+ ret = sof_widget_load_pcm(scomp, index, swidget,
+ SOF_IPC_STREAM_CAPTURE, tw, &reply);
+ break;
+ case snd_soc_dapm_aif_in:
+ ret = sof_widget_load_pcm(scomp, index, swidget,
+ SOF_IPC_STREAM_PLAYBACK, tw, &reply);
+ break;
+ case snd_soc_dapm_src:
+ ret = sof_widget_load_src(scomp, index, swidget, tw, &reply);
+ break;
+ case snd_soc_dapm_siggen:
+ ret = sof_widget_load_siggen(scomp, index, swidget, tw, &reply);
+ break;
+ case snd_soc_dapm_effect:
+ ret = sof_widget_load_process(scomp, index, swidget, tw,
+ &reply);
+ break;
+ case snd_soc_dapm_mux:
+ case snd_soc_dapm_demux:
+ ret = sof_widget_load_mux(scomp, index, swidget, tw, &reply);
+ break;
+ case snd_soc_dapm_switch:
+ case snd_soc_dapm_dai_link:
+ case snd_soc_dapm_kcontrol:
+ default:
+ dev_warn(sdev->dev, "warning: widget type %d name %s not handled\n",
+ swidget->id, tw->name);
+ break;
+ }
+
+ /* check IPC reply */
+ if (ret < 0 || reply.rhdr.error < 0) {
+ dev_err(sdev->dev,
+ "error: DSP failed to add widget id %d type %d name : %s stream %s reply %d\n",
+ tw->shift, swidget->id, tw->name,
+ strnlen(tw->sname, SNDRV_CTL_ELEM_ID_NAME_MAXLEN) > 0
+ ? tw->sname : "none", reply.rhdr.error);
+ kfree(swidget);
+ return ret;
+ }
+
+ /* bind widget to external event */
+ if (tw->event_type) {
+ ret = sof_widget_bind_event(sdev, swidget,
+ le16_to_cpu(tw->event_type));
+ if (ret) {
+ dev_err(sdev->dev, "error: widget event binding failed\n");
+ kfree(swidget->private);
+ kfree(swidget);
+ return ret;
+ }
+ }
+
+ w->dobj.private = swidget;
+ list_add(&swidget->list, &sdev->widget_list);
+ return ret;
+}
+
+static int sof_route_unload(struct snd_soc_component *scomp,
+ struct snd_soc_dobj *dobj)
+{
+ struct snd_sof_route *sroute;
+
+ sroute = dobj->private;
+ if (!sroute)
+ return 0;
+
+ /* free sroute and its private data */
+ kfree(sroute->private);
+ list_del(&sroute->list);
+ kfree(sroute);
+
+ return 0;
+}
+
+static int sof_widget_unload(struct snd_soc_component *scomp,
+ struct snd_soc_dobj *dobj)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ const struct snd_kcontrol_new *kc;
+ struct snd_soc_dapm_widget *widget;
+ struct sof_ipc_pipe_new *pipeline;
+ struct snd_sof_control *scontrol;
+ struct snd_sof_widget *swidget;
+ struct soc_mixer_control *sm;
+ struct soc_bytes_ext *sbe;
+ struct snd_sof_dai *dai;
+ struct soc_enum *se;
+ int ret = 0;
+ int i;
+
+ swidget = dobj->private;
+ if (!swidget)
+ return 0;
+
+ widget = swidget->widget;
+
+ switch (swidget->id) {
+ case snd_soc_dapm_dai_in:
+ case snd_soc_dapm_dai_out:
+ dai = swidget->private;
+
+ if (dai) {
+ /* free dai config */
+ kfree(dai->dai_config);
+ list_del(&dai->list);
+ }
+ break;
+ case snd_soc_dapm_scheduler:
+
+ /* power down the pipeline schedule core */
+ pipeline = swidget->private;
+ ret = snd_sof_dsp_core_power_down(sdev, 1 << pipeline->core);
+ if (ret < 0)
+ dev_err(sdev->dev, "error: powering down pipeline schedule core %d\n",
+ pipeline->core);
+
+ /* update enabled cores mask */
+ sdev->enabled_cores_mask &= ~(1 << pipeline->core);
+
+ break;
+ default:
+ break;
+ }
+ for (i = 0; i < widget->num_kcontrols; i++) {
+ kc = &widget->kcontrol_news[i];
+ switch (dobj->widget.kcontrol_type) {
+ case SND_SOC_TPLG_TYPE_MIXER:
+ sm = (struct soc_mixer_control *)kc->private_value;
+ scontrol = sm->dobj.private;
+ if (sm->max > 1)
+ kfree(scontrol->volume_table);
+ break;
+ case SND_SOC_TPLG_TYPE_ENUM:
+ se = (struct soc_enum *)kc->private_value;
+ scontrol = se->dobj.private;
+ break;
+ case SND_SOC_TPLG_TYPE_BYTES:
+ sbe = (struct soc_bytes_ext *)kc->private_value;
+ scontrol = sbe->dobj.private;
+ break;
+ default:
+ dev_warn(sdev->dev, "unsupported kcontrol_type\n");
+ goto out;
+ }
+ kfree(scontrol->control_data);
+ list_del(&scontrol->list);
+ kfree(scontrol);
+ }
+
+out:
+ /* free private value */
+ kfree(swidget->private);
+
+ /* remove and free swidget object */
+ list_del(&swidget->list);
+ kfree(swidget);
+
+ return ret;
+}
+
+/*
+ * DAI HW configuration.
+ */
+
+/* FE DAI - used for any driver specific init */
+static int sof_dai_load(struct snd_soc_component *scomp, int index,
+ struct snd_soc_dai_driver *dai_drv,
+ struct snd_soc_tplg_pcm *pcm, struct snd_soc_dai *dai)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_stream_caps *caps;
+ struct snd_sof_pcm *spcm;
+ int stream = SNDRV_PCM_STREAM_PLAYBACK;
+ int ret = 0;
+
+ /* nothing to do for BEs atm */
+ if (!pcm)
+ return 0;
+
+ spcm = kzalloc(sizeof(*spcm), GFP_KERNEL);
+ if (!spcm)
+ return -ENOMEM;
+
+ spcm->sdev = sdev;
+ spcm->stream[SNDRV_PCM_STREAM_PLAYBACK].comp_id = COMP_ID_UNASSIGNED;
+ spcm->stream[SNDRV_PCM_STREAM_CAPTURE].comp_id = COMP_ID_UNASSIGNED;
+
+ if (pcm) {
+ spcm->pcm = *pcm;
+ dev_dbg(sdev->dev, "tplg: load pcm %s\n", pcm->dai_name);
+ }
+ dai_drv->dobj.private = spcm;
+ list_add(&spcm->list, &sdev->pcm_list);
+
+ /* do we need to allocate playback PCM DMA pages */
+ if (!spcm->pcm.playback)
+ goto capture;
+
+ caps = &spcm->pcm.caps[stream];
+
+ /* allocate playback page table buffer */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV, sdev->dev,
+ PAGE_SIZE, &spcm->stream[stream].page_table);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: can't alloc page table for %s %d\n",
+ caps->name, ret);
+
+ return ret;
+ }
+
+ /* bind pcm to host comp */
+ ret = spcm_bind(sdev, spcm, stream);
+ if (ret) {
+ dev_err(sdev->dev,
+ "error: can't bind pcm to host\n");
+ goto free_playback_tables;
+ }
+
+capture:
+ stream = SNDRV_PCM_STREAM_CAPTURE;
+
+ /* do we need to allocate capture PCM DMA pages */
+ if (!spcm->pcm.capture)
+ return ret;
+
+ caps = &spcm->pcm.caps[stream];
+
+ /* allocate capture page table buffer */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV, sdev->dev,
+ PAGE_SIZE, &spcm->stream[stream].page_table);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: can't alloc page table for %s %d\n",
+ caps->name, ret);
+ goto free_playback_tables;
+ }
+
+ /* bind pcm to host comp */
+ ret = spcm_bind(sdev, spcm, stream);
+ if (ret) {
+ dev_err(sdev->dev,
+ "error: can't bind pcm to host\n");
+ snd_dma_free_pages(&spcm->stream[stream].page_table);
+ goto free_playback_tables;
+ }
+
+ return ret;
+
+free_playback_tables:
+ if (spcm->pcm.playback)
+ snd_dma_free_pages(&spcm->stream[SNDRV_PCM_STREAM_PLAYBACK].page_table);
+
+ return ret;
+}
+
+static int sof_dai_unload(struct snd_soc_component *scomp,
+ struct snd_soc_dobj *dobj)
+{
+ struct snd_sof_pcm *spcm = dobj->private;
+
+ /* free PCM DMA pages */
+ if (spcm->pcm.playback)
+ snd_dma_free_pages(&spcm->stream[SNDRV_PCM_STREAM_PLAYBACK].page_table);
+
+ if (spcm->pcm.capture)
+ snd_dma_free_pages(&spcm->stream[SNDRV_PCM_STREAM_CAPTURE].page_table);
+
+ /* remove from list and free spcm */
+ list_del(&spcm->list);
+ kfree(spcm);
+
+ return 0;
+}
+
+static void sof_dai_set_format(struct snd_soc_tplg_hw_config *hw_config,
+ struct sof_ipc_dai_config *config)
+{
+ /* clock directions wrt codec */
+ if (hw_config->bclk_master == SND_SOC_TPLG_BCLK_CM) {
+ /* codec is bclk master */
+ if (hw_config->fsync_master == SND_SOC_TPLG_FSYNC_CM)
+ config->format |= SOF_DAI_FMT_CBM_CFM;
+ else
+ config->format |= SOF_DAI_FMT_CBM_CFS;
+ } else {
+ /* codec is bclk slave */
+ if (hw_config->fsync_master == SND_SOC_TPLG_FSYNC_CM)
+ config->format |= SOF_DAI_FMT_CBS_CFM;
+ else
+ config->format |= SOF_DAI_FMT_CBS_CFS;
+ }
+
+ /* inverted clocks ? */
+ if (hw_config->invert_bclk) {
+ if (hw_config->invert_fsync)
+ config->format |= SOF_DAI_FMT_IB_IF;
+ else
+ config->format |= SOF_DAI_FMT_IB_NF;
+ } else {
+ if (hw_config->invert_fsync)
+ config->format |= SOF_DAI_FMT_NB_IF;
+ else
+ config->format |= SOF_DAI_FMT_NB_NF;
+ }
+}
+
+/* set config for all DAI's with name matching the link name */
+static int sof_set_dai_config(struct snd_sof_dev *sdev, u32 size,
+ struct snd_soc_dai_link *link,
+ struct sof_ipc_dai_config *config)
+{
+ struct snd_sof_dai *dai;
+ int found = 0;
+
+ list_for_each_entry(dai, &sdev->dai_list, list) {
+ if (!dai->name)
+ continue;
+
+ if (strcmp(link->name, dai->name) == 0) {
+ dai->dai_config = kmemdup(config, size, GFP_KERNEL);
+ if (!dai->dai_config)
+ return -ENOMEM;
+
+ found = 1;
+ }
+ }
+
+ /*
+ * machine driver may define a dai link with playback and capture
+ * dai enabled, but the dai link in topology would support both, one
+ * or none of them. Here print a warning message to notify user
+ */
+ if (!found) {
+ dev_warn(sdev->dev, "warning: failed to find dai for dai link %s",
+ link->name);
+ }
+
+ return 0;
+}
+
+static int sof_link_ssp_load(struct snd_soc_component *scomp, int index,
+ struct snd_soc_dai_link *link,
+ struct snd_soc_tplg_link_config *cfg,
+ struct snd_soc_tplg_hw_config *hw_config,
+ struct sof_ipc_dai_config *config)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &cfg->priv;
+ struct sof_ipc_reply reply;
+ u32 size = sizeof(*config);
+ int ret;
+
+ /* handle master/slave and inverted clocks */
+ sof_dai_set_format(hw_config, config);
+
+ /* init IPC */
+ memset(&config->ssp, 0, sizeof(struct sof_ipc_dai_ssp_params));
+ config->hdr.size = size;
+
+ ret = sof_parse_tokens(scomp, &config->ssp, ssp_tokens,
+ ARRAY_SIZE(ssp_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse ssp tokens failed %d\n",
+ le32_to_cpu(private->size));
+ return ret;
+ }
+
+ config->ssp.mclk_rate = le32_to_cpu(hw_config->mclk_rate);
+ config->ssp.bclk_rate = le32_to_cpu(hw_config->bclk_rate);
+ config->ssp.fsync_rate = le32_to_cpu(hw_config->fsync_rate);
+ config->ssp.tdm_slots = le32_to_cpu(hw_config->tdm_slots);
+ config->ssp.tdm_slot_width = le32_to_cpu(hw_config->tdm_slot_width);
+ config->ssp.mclk_direction = hw_config->mclk_direction;
+ config->ssp.rx_slots = le32_to_cpu(hw_config->rx_slots);
+ config->ssp.tx_slots = le32_to_cpu(hw_config->tx_slots);
+
+ dev_dbg(sdev->dev, "tplg: config SSP%d fmt 0x%x mclk %d bclk %d fclk %d width (%d)%d slots %d mclk id %d quirks %d\n",
+ config->dai_index, config->format,
+ config->ssp.mclk_rate, config->ssp.bclk_rate,
+ config->ssp.fsync_rate, config->ssp.sample_valid_bits,
+ config->ssp.tdm_slot_width, config->ssp.tdm_slots,
+ config->ssp.mclk_id, config->ssp.quirks);
+
+ /* validate SSP fsync rate and channel count */
+ if (config->ssp.fsync_rate < 8000 || config->ssp.fsync_rate > 192000) {
+ dev_err(sdev->dev, "error: invalid fsync rate for SSP%d\n",
+ config->dai_index);
+ return -EINVAL;
+ }
+
+ if (config->ssp.tdm_slots < 1 || config->ssp.tdm_slots > 8) {
+ dev_err(sdev->dev, "error: invalid channel count for SSP%d\n",
+ config->dai_index);
+ return -EINVAL;
+ }
+
+ /* send message to DSP */
+ ret = sof_ipc_tx_message(sdev->ipc,
+ config->hdr.cmd, config, size, &reply,
+ sizeof(reply));
+
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to set DAI config for SSP%d\n",
+ config->dai_index);
+ return ret;
+ }
+
+ /* set config for all DAI's with name matching the link name */
+ ret = sof_set_dai_config(sdev, size, link, config);
+ if (ret < 0)
+ dev_err(sdev->dev, "error: failed to save DAI config for SSP%d\n",
+ config->dai_index);
+
+ return ret;
+}
+
+static int sof_link_dmic_load(struct snd_soc_component *scomp, int index,
+ struct snd_soc_dai_link *link,
+ struct snd_soc_tplg_link_config *cfg,
+ struct snd_soc_tplg_hw_config *hw_config,
+ struct sof_ipc_dai_config *config)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &cfg->priv;
+ struct sof_ipc_dai_config *ipc_config;
+ struct sof_ipc_reply reply;
+ struct sof_ipc_fw_ready *ready = &sdev->fw_ready;
+ struct sof_ipc_fw_version *v = &ready->version;
+ u32 size;
+ int ret, j;
+
+ /*
+ * config is only used for the common params in dmic_params structure
+ * that does not include the PDM controller config array
+ * Set the common params to 0.
+ */
+ memset(&config->dmic, 0, sizeof(struct sof_ipc_dai_dmic_params));
+
+ /* get DMIC tokens */
+ ret = sof_parse_tokens(scomp, &config->dmic, dmic_tokens,
+ ARRAY_SIZE(dmic_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse dmic tokens failed %d\n",
+ le32_to_cpu(private->size));
+ return ret;
+ }
+
+ /*
+ * allocate memory for dmic dai config accounting for the
+ * variable number of active pdm controllers
+ * This will be the ipc payload for setting dai config
+ */
+ size = sizeof(*config) + sizeof(struct sof_ipc_dai_dmic_pdm_ctrl) *
+ config->dmic.num_pdm_active;
+
+ ipc_config = kzalloc(size, GFP_KERNEL);
+ if (!ipc_config)
+ return -ENOMEM;
+
+ /* copy the common dai config and dmic params */
+ memcpy(ipc_config, config, sizeof(*config));
+
+ /*
+ * alloc memory for private member
+ * Used to track the pdm config array index currently being parsed
+ */
+ sdev->private = kzalloc(sizeof(u32), GFP_KERNEL);
+ if (!sdev->private) {
+ kfree(ipc_config);
+ return -ENOMEM;
+ }
+
+ /* get DMIC PDM tokens */
+ ret = sof_parse_tokens(scomp, &ipc_config->dmic.pdm[0], dmic_pdm_tokens,
+ ARRAY_SIZE(dmic_pdm_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse dmic pdm tokens failed %d\n",
+ le32_to_cpu(private->size));
+ goto err;
+ }
+
+ /* set IPC header size */
+ ipc_config->hdr.size = size;
+
+ /* debug messages */
+ dev_dbg(sdev->dev, "tplg: config DMIC%d driver version %d\n",
+ ipc_config->dai_index, ipc_config->dmic.driver_ipc_version);
+ dev_dbg(sdev->dev, "pdmclk_min %d pdm_clkmax %d duty_min %hd\n",
+ ipc_config->dmic.pdmclk_min, ipc_config->dmic.pdmclk_max,
+ ipc_config->dmic.duty_min);
+ dev_dbg(sdev->dev, "duty_max %hd fifo_fs %d num_pdms active %d\n",
+ ipc_config->dmic.duty_max, ipc_config->dmic.fifo_fs,
+ ipc_config->dmic.num_pdm_active);
+ dev_dbg(sdev->dev, "fifo word length %hd\n",
+ ipc_config->dmic.fifo_bits);
+
+ for (j = 0; j < ipc_config->dmic.num_pdm_active; j++) {
+ dev_dbg(sdev->dev, "pdm %hd mic a %hd mic b %hd\n",
+ ipc_config->dmic.pdm[j].id,
+ ipc_config->dmic.pdm[j].enable_mic_a,
+ ipc_config->dmic.pdm[j].enable_mic_b);
+ dev_dbg(sdev->dev, "pdm %hd polarity a %hd polarity b %hd\n",
+ ipc_config->dmic.pdm[j].id,
+ ipc_config->dmic.pdm[j].polarity_mic_a,
+ ipc_config->dmic.pdm[j].polarity_mic_b);
+ dev_dbg(sdev->dev, "pdm %hd clk_edge %hd skew %hd\n",
+ ipc_config->dmic.pdm[j].id,
+ ipc_config->dmic.pdm[j].clk_edge,
+ ipc_config->dmic.pdm[j].skew);
+ }
+
+ if (SOF_ABI_VER(v->major, v->minor, v->micro) < SOF_ABI_VER(3, 0, 1)) {
+ /* this takes care of backwards compatible handling of fifo_bits_b */
+ ipc_config->dmic.reserved_2 = ipc_config->dmic.fifo_bits;
+ }
+
+ /* send message to DSP */
+ ret = sof_ipc_tx_message(sdev->ipc,
+ ipc_config->hdr.cmd, ipc_config, size, &reply,
+ sizeof(reply));
+
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: failed to set DAI config for DMIC%d\n",
+ config->dai_index);
+ goto err;
+ }
+
+ /* set config for all DAI's with name matching the link name */
+ ret = sof_set_dai_config(sdev, size, link, ipc_config);
+ if (ret < 0)
+ dev_err(sdev->dev, "error: failed to save DAI config for DMIC%d\n",
+ config->dai_index);
+
+err:
+ kfree(sdev->private);
+ kfree(ipc_config);
+
+ return ret;
+}
+
+/*
+ * for hda link, playback and capture are supported by different dai
+ * in FW. Here get the dai_index, set dma channel of each dai
+ * and send config to FW. In FW, each dai sets config by dai_index
+ */
+static int sof_link_hda_process(struct snd_sof_dev *sdev,
+ struct snd_soc_dai_link *link,
+ struct sof_ipc_dai_config *config,
+ int tx_slot,
+ int rx_slot)
+{
+ struct sof_ipc_reply reply;
+ u32 size = sizeof(*config);
+ struct snd_sof_dai *sof_dai;
+ int found = 0;
+ int ret;
+
+ list_for_each_entry(sof_dai, &sdev->dai_list, list) {
+ if (!sof_dai->name)
+ continue;
+
+ if (strcmp(link->name, sof_dai->name) == 0) {
+ if (sof_dai->comp_dai.direction ==
+ SNDRV_PCM_STREAM_PLAYBACK) {
+ if (!link->dpcm_playback)
+ return -EINVAL;
+
+ config->hda.link_dma_ch = tx_slot;
+ } else {
+ if (!link->dpcm_capture)
+ return -EINVAL;
+
+ config->hda.link_dma_ch = rx_slot;
+ }
+
+ config->dai_index = sof_dai->comp_dai.dai_index;
+ found = 1;
+
+ /* save config in dai component */
+ sof_dai->dai_config = kmemdup(config, size, GFP_KERNEL);
+ if (!sof_dai->dai_config)
+ return -ENOMEM;
+
+ /* send message to DSP */
+ ret = sof_ipc_tx_message(sdev->ipc,
+ config->hdr.cmd, config, size,
+ &reply, sizeof(reply));
+
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to set DAI config for direction:%d of HDA dai %d\n",
+ sof_dai->comp_dai.direction,
+ config->dai_index);
+
+ return ret;
+ }
+ }
+ }
+
+ /*
+ * machine driver may define a dai link with playback and capture
+ * dai enabled, but the dai link in topology would support both, one
+ * or none of them. Here print a warning message to notify user
+ */
+ if (!found) {
+ dev_warn(sdev->dev, "warning: failed to find dai for dai link %s",
+ link->name);
+ }
+
+ return 0;
+}
+
+static int sof_link_hda_load(struct snd_soc_component *scomp, int index,
+ struct snd_soc_dai_link *link,
+ struct snd_soc_tplg_link_config *cfg,
+ struct snd_soc_tplg_hw_config *hw_config,
+ struct sof_ipc_dai_config *config)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_dai_link_component dai_component;
+ struct snd_soc_tplg_private *private = &cfg->priv;
+ struct snd_soc_dai *dai;
+ u32 size = sizeof(*config);
+ u32 tx_num = 0;
+ u32 tx_slot = 0;
+ u32 rx_num = 0;
+ u32 rx_slot = 0;
+ int ret;
+
+ /* init IPC */
+ memset(&dai_component, 0, sizeof(dai_component));
+ memset(&config->hda, 0, sizeof(struct sof_ipc_dai_hda_params));
+ config->hdr.size = size;
+
+ /* get any bespoke DAI tokens */
+ ret = sof_parse_tokens(scomp, config, hda_tokens,
+ ARRAY_SIZE(hda_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse hda tokens failed %d\n",
+ le32_to_cpu(private->size));
+ return ret;
+ }
+
+ dai_component.dai_name = link->cpu_dai_name;
+ dai = snd_soc_find_dai(&dai_component);
+ if (!dai) {
+ dev_err(sdev->dev, "error: failed to find dai %s in %s",
+ dai_component.dai_name, __func__);
+ return -EINVAL;
+ }
+
+ if (link->dpcm_playback)
+ tx_num = 1;
+
+ if (link->dpcm_capture)
+ rx_num = 1;
+
+ ret = snd_soc_dai_get_channel_map(dai, &tx_num, &tx_slot,
+ &rx_num, &rx_slot);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to get dma channel for HDA%d\n",
+ config->dai_index);
+
+ return ret;
+ }
+
+ ret = sof_link_hda_process(sdev, link, config, tx_slot, rx_slot);
+ if (ret < 0)
+ dev_err(sdev->dev, "error: failed to process hda dai link %s",
+ link->name);
+
+ return ret;
+}
+
+/* DAI link - used for any driver specific init */
+static int sof_link_load(struct snd_soc_component *scomp, int index,
+ struct snd_soc_dai_link *link,
+ struct snd_soc_tplg_link_config *cfg)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_tplg_private *private = &cfg->priv;
+ struct sof_ipc_dai_config config;
+ struct snd_soc_tplg_hw_config *hw_config;
+ int num_hw_configs;
+ int ret;
+ int i = 0;
+
+ link->platform_name = dev_name(sdev->dev);
+
+ /*
+ * Set nonatomic property for FE dai links as their trigger action
+ * involves IPC's.
+ */
+ if (!link->no_pcm) {
+ link->nonatomic = true;
+
+ /* nothing more to do for FE dai links */
+ return 0;
+ }
+
+ /* check we have some tokens - we need at least DAI type */
+ if (le32_to_cpu(private->size) == 0) {
+ dev_err(sdev->dev, "error: expected tokens for DAI, none found\n");
+ return -EINVAL;
+ }
+
+ /* Send BE DAI link configurations to DSP */
+ memset(&config, 0, sizeof(config));
+
+ /* get any common DAI tokens */
+ ret = sof_parse_tokens(scomp, &config, dai_link_tokens,
+ ARRAY_SIZE(dai_link_tokens), private->array,
+ le32_to_cpu(private->size));
+ if (ret != 0) {
+ dev_err(sdev->dev, "error: parse link tokens failed %d\n",
+ le32_to_cpu(private->size));
+ return ret;
+ }
+
+ /*
+ * DAI links are expected to have at least 1 hw_config.
+ * But some older topologies might have no hw_config for HDA dai links.
+ */
+ num_hw_configs = le32_to_cpu(cfg->num_hw_configs);
+ if (!num_hw_configs) {
+ if (config.type != SOF_DAI_INTEL_HDA) {
+ dev_err(sdev->dev, "error: unexpected DAI config count %d!\n",
+ le32_to_cpu(cfg->num_hw_configs));
+ return -EINVAL;
+ }
+ } else {
+ dev_dbg(sdev->dev, "tplg: %d hw_configs found, default id: %d!\n",
+ cfg->num_hw_configs, le32_to_cpu(cfg->default_hw_config_id));
+
+ for (i = 0; i < num_hw_configs; i++) {
+ if (cfg->hw_config[i].id == cfg->default_hw_config_id)
+ break;
+ }
+
+ if (i == num_hw_configs) {
+ dev_err(sdev->dev, "error: default hw_config id: %d not found!\n",
+ le32_to_cpu(cfg->default_hw_config_id));
+ return -EINVAL;
+ }
+ }
+
+ /* configure dai IPC message */
+ hw_config = &cfg->hw_config[i];
+
+ config.hdr.cmd = SOF_IPC_GLB_DAI_MSG | SOF_IPC_DAI_CONFIG;
+ config.format = le32_to_cpu(hw_config->fmt);
+
+ /* now load DAI specific data and send IPC - type comes from token */
+ switch (config.type) {
+ case SOF_DAI_INTEL_SSP:
+ ret = sof_link_ssp_load(scomp, index, link, cfg, hw_config,
+ &config);
+ break;
+ case SOF_DAI_INTEL_DMIC:
+ ret = sof_link_dmic_load(scomp, index, link, cfg, hw_config,
+ &config);
+ break;
+ case SOF_DAI_INTEL_HDA:
+ ret = sof_link_hda_load(scomp, index, link, cfg, hw_config,
+ &config);
+ break;
+ default:
+ dev_err(sdev->dev, "error: invalid DAI type %d\n", config.type);
+ ret = -EINVAL;
+ break;
+ }
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static int sof_link_hda_unload(struct snd_sof_dev *sdev,
+ struct snd_soc_dai_link *link)
+{
+ struct snd_soc_dai_link_component dai_component;
+ struct snd_soc_dai *dai;
+ int ret = 0;
+
+ memset(&dai_component, 0, sizeof(dai_component));
+ dai_component.dai_name = link->cpu_dai_name;
+ dai = snd_soc_find_dai(&dai_component);
+ if (!dai) {
+ dev_err(sdev->dev, "error: failed to find dai %s in %s",
+ dai_component.dai_name, __func__);
+ return -EINVAL;
+ }
+
+ /*
+ * FIXME: this call to hw_free is mainly to release the link DMA ID.
+ * This is abusing the API and handling SOC internals is not
+ * recommended. This part will be reworked.
+ */
+ if (dai->driver->ops->hw_free)
+ ret = dai->driver->ops->hw_free(NULL, dai);
+ if (ret < 0)
+ dev_err(sdev->dev, "error: failed to free hda resource for %s\n",
+ link->name);
+
+ return ret;
+}
+
+static int sof_link_unload(struct snd_soc_component *scomp,
+ struct snd_soc_dobj *dobj)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_soc_dai_link *link =
+ container_of(dobj, struct snd_soc_dai_link, dobj);
+
+ struct snd_sof_dai *sof_dai;
+ int ret = 0;
+
+ /* only BE link is loaded by sof */
+ if (!link->no_pcm)
+ return 0;
+
+ list_for_each_entry(sof_dai, &sdev->dai_list, list) {
+ if (!sof_dai->name)
+ continue;
+
+ if (strcmp(link->name, sof_dai->name) == 0)
+ goto found;
+ }
+
+ dev_err(sdev->dev, "error: failed to find dai %s in %s",
+ link->name, __func__);
+ return -EINVAL;
+found:
+
+ switch (sof_dai->dai_config->type) {
+ case SOF_DAI_INTEL_SSP:
+ case SOF_DAI_INTEL_DMIC:
+ /* no resource needs to be released for SSP and DMIC */
+ break;
+ case SOF_DAI_INTEL_HDA:
+ ret = sof_link_hda_unload(sdev, link);
+ break;
+ default:
+ dev_err(sdev->dev, "error: invalid DAI type %d\n",
+ sof_dai->dai_config->type);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+/* DAI link - used for any driver specific init */
+static int sof_route_load(struct snd_soc_component *scomp, int index,
+ struct snd_soc_dapm_route *route)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct sof_ipc_pipe_comp_connect *connect;
+ struct snd_sof_widget *source_swidget, *sink_swidget;
+ struct snd_soc_dobj *dobj = &route->dobj;
+ struct snd_sof_route *sroute;
+ struct sof_ipc_reply reply;
+ int ret = 0;
+
+ /* allocate memory for sroute and connect */
+ sroute = kzalloc(sizeof(*sroute), GFP_KERNEL);
+ if (!sroute)
+ return -ENOMEM;
+
+ sroute->sdev = sdev;
+
+ connect = kzalloc(sizeof(*connect), GFP_KERNEL);
+ if (!connect) {
+ kfree(sroute);
+ return -ENOMEM;
+ }
+
+ connect->hdr.size = sizeof(*connect);
+ connect->hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_COMP_CONNECT;
+
+ dev_dbg(sdev->dev, "sink %s control %s source %s\n",
+ route->sink, route->control ? route->control : "none",
+ route->source);
+
+ /* source component */
+ source_swidget = snd_sof_find_swidget(sdev, (char *)route->source);
+ if (!source_swidget) {
+ dev_err(sdev->dev, "error: source %s not found\n",
+ route->source);
+ ret = -EINVAL;
+ goto err;
+ }
+
+ /*
+ * Virtual widgets of type output/out_drv may be added in topology
+ * for compatibility. These are not handled by the FW.
+ * So, don't send routes whose source/sink widget is of such types
+ * to the DSP.
+ */
+ if (source_swidget->id == snd_soc_dapm_out_drv ||
+ source_swidget->id == snd_soc_dapm_output)
+ goto err;
+
+ connect->source_id = source_swidget->comp_id;
+
+ /* sink component */
+ sink_swidget = snd_sof_find_swidget(sdev, (char *)route->sink);
+ if (!sink_swidget) {
+ dev_err(sdev->dev, "error: sink %s not found\n",
+ route->sink);
+ ret = -EINVAL;
+ goto err;
+ }
+
+ /*
+ * Don't send routes whose sink widget is of type
+ * output or out_drv to the DSP
+ */
+ if (sink_swidget->id == snd_soc_dapm_out_drv ||
+ sink_swidget->id == snd_soc_dapm_output)
+ goto err;
+
+ connect->sink_id = sink_swidget->comp_id;
+
+ /*
+ * For virtual routes, both sink and source are not
+ * buffer. Since only buffer linked to component is supported by
+ * FW, others are reported as error, add check in route function,
+ * do not send it to FW when both source and sink are not buffer
+ */
+ if (source_swidget->id != snd_soc_dapm_buffer &&
+ sink_swidget->id != snd_soc_dapm_buffer) {
+ dev_dbg(sdev->dev, "warning: neither Linked source component %s nor sink component %s is of buffer type, ignoring link\n",
+ route->source, route->sink);
+ ret = 0;
+ goto err;
+ } else {
+ ret = sof_ipc_tx_message(sdev->ipc,
+ connect->hdr.cmd,
+ connect, sizeof(*connect),
+ &reply, sizeof(reply));
+
+ /* check IPC return value */
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: failed to add route sink %s control %s source %s\n",
+ route->sink,
+ route->control ? route->control : "none",
+ route->source);
+ goto err;
+ }
+
+ /* check IPC reply */
+ if (reply.error < 0) {
+ dev_err(sdev->dev, "error: DSP failed to add route sink %s control %s source %s result %d\n",
+ route->sink,
+ route->control ? route->control : "none",
+ route->source, reply.error);
+ ret = reply.error;
+ goto err;
+ }
+
+ sroute->route = route;
+ dobj->private = sroute;
+ sroute->private = connect;
+
+ /* add route to route list */
+ list_add(&sroute->list, &sdev->route_list);
+
+ return ret;
+ }
+
+err:
+ kfree(connect);
+ kfree(sroute);
+ return ret;
+}
+
+int snd_sof_complete_pipeline(struct snd_sof_dev *sdev,
+ struct snd_sof_widget *swidget)
+{
+ struct sof_ipc_pipe_ready ready;
+ struct sof_ipc_reply reply;
+ int ret;
+
+ dev_dbg(sdev->dev, "tplg: complete pipeline %s id %d\n",
+ swidget->widget->name, swidget->comp_id);
+
+ memset(&ready, 0, sizeof(ready));
+ ready.hdr.size = sizeof(ready);
+ ready.hdr.cmd = SOF_IPC_GLB_TPLG_MSG | SOF_IPC_TPLG_PIPE_COMPLETE;
+ ready.comp_id = swidget->comp_id;
+
+ ret = sof_ipc_tx_message(sdev->ipc,
+ ready.hdr.cmd, &ready, sizeof(ready), &reply,
+ sizeof(reply));
+ if (ret < 0)
+ return ret;
+ return 1;
+}
+
+/* completion - called at completion of firmware loading */
+static void sof_complete(struct snd_soc_component *scomp)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ struct snd_sof_widget *swidget;
+
+ /* some widget types require completion notificattion */
+ list_for_each_entry(swidget, &sdev->widget_list, list) {
+ if (swidget->complete)
+ continue;
+
+ switch (swidget->id) {
+ case snd_soc_dapm_scheduler:
+ swidget->complete =
+ snd_sof_complete_pipeline(sdev, swidget);
+ break;
+ default:
+ break;
+ }
+ }
+}
+
+/* manifest - optional to inform component of manifest */
+static int sof_manifest(struct snd_soc_component *scomp, int index,
+ struct snd_soc_tplg_manifest *man)
+{
+ struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp);
+ u32 size;
+ u32 abi_version;
+
+ size = le32_to_cpu(man->priv.size);
+
+ /* backward compatible with tplg without ABI info */
+ if (!size) {
+ dev_dbg(sdev->dev, "No topology ABI info\n");
+ return 0;
+ }
+
+ if (size != SOF_TPLG_ABI_SIZE) {
+ dev_err(sdev->dev, "error: invalid topology ABI size\n");
+ return -EINVAL;
+ }
+
+ dev_info(sdev->dev,
+ "Topology: ABI %d:%d:%d Kernel ABI %d:%d:%d\n",
+ man->priv.data[0], man->priv.data[1],
+ man->priv.data[2], SOF_ABI_MAJOR, SOF_ABI_MINOR,
+ SOF_ABI_PATCH);
+
+ abi_version = SOF_ABI_VER(man->priv.data[0],
+ man->priv.data[1],
+ man->priv.data[2]);
+
+ if (SOF_ABI_VERSION_INCOMPATIBLE(SOF_ABI_VERSION, abi_version)) {
+ dev_err(sdev->dev, "error: incompatible topology ABI version\n");
+ return -EINVAL;
+ }
+
+ if (abi_version > SOF_ABI_VERSION) {
+ if (!IS_ENABLED(CONFIG_SND_SOC_SOF_STRICT_ABI_CHECKS)) {
+ dev_warn(sdev->dev, "warn: topology ABI is more recent than kernel\n");
+ } else {
+ dev_err(sdev->dev, "error: topology ABI is more recent than kernel\n");
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+/* vendor specific kcontrol handlers available for binding */
+static const struct snd_soc_tplg_kcontrol_ops sof_io_ops[] = {
+ {SOF_TPLG_KCTL_VOL_ID, snd_sof_volume_get, snd_sof_volume_put},
+ {SOF_TPLG_KCTL_BYTES_ID, snd_sof_bytes_get, snd_sof_bytes_put},
+ {SOF_TPLG_KCTL_ENUM_ID, snd_sof_enum_get, snd_sof_enum_put},
+ {SOF_TPLG_KCTL_SWITCH_ID, snd_sof_switch_get, snd_sof_switch_put},
+};
+
+/* vendor specific bytes ext handlers available for binding */
+static const struct snd_soc_tplg_bytes_ext_ops sof_bytes_ext_ops[] = {
+ {SOF_TPLG_KCTL_BYTES_ID, snd_sof_bytes_ext_get, snd_sof_bytes_ext_put},
+};
+
+static struct snd_soc_tplg_ops sof_tplg_ops = {
+ /* external kcontrol init - used for any driver specific init */
+ .control_load = sof_control_load,
+ .control_unload = sof_control_unload,
+
+ /* external kcontrol init - used for any driver specific init */
+ .dapm_route_load = sof_route_load,
+ .dapm_route_unload = sof_route_unload,
+
+ /* external widget init - used for any driver specific init */
+ /* .widget_load is not currently used */
+ .widget_ready = sof_widget_ready,
+ .widget_unload = sof_widget_unload,
+
+ /* FE DAI - used for any driver specific init */
+ .dai_load = sof_dai_load,
+ .dai_unload = sof_dai_unload,
+
+ /* DAI link - used for any driver specific init */
+ .link_load = sof_link_load,
+ .link_unload = sof_link_unload,
+
+ /* completion - called at completion of firmware loading */
+ .complete = sof_complete,
+
+ /* manifest - optional to inform component of manifest */
+ .manifest = sof_manifest,
+
+ /* vendor specific kcontrol handlers available for binding */
+ .io_ops = sof_io_ops,
+ .io_ops_count = ARRAY_SIZE(sof_io_ops),
+
+ /* vendor specific bytes ext handlers available for binding */
+ .bytes_ext_ops = sof_bytes_ext_ops,
+ .bytes_ext_ops_count = ARRAY_SIZE(sof_bytes_ext_ops),
+};
+
+int snd_sof_init_topology(struct snd_sof_dev *sdev,
+ struct snd_soc_tplg_ops *ops)
+{
+ /* TODO: support linked list of topologies */
+ sdev->tplg_ops = ops;
+ return 0;
+}
+EXPORT_SYMBOL(snd_sof_init_topology);
+
+int snd_sof_load_topology(struct snd_sof_dev *sdev, const char *file)
+{
+ const struct firmware *fw;
+ int ret;
+
+ dev_dbg(sdev->dev, "loading topology:%s\n", file);
+
+ ret = request_firmware(&fw, file, sdev->dev);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: tplg request firmware %s failed err: %d\n",
+ file, ret);
+ return ret;
+ }
+
+ ret = snd_soc_tplg_component_load(sdev->component,
+ &sof_tplg_ops, fw,
+ SND_SOC_TPLG_INDEX_ALL);
+ if (ret < 0) {
+ dev_err(sdev->dev, "error: tplg component load failed %d\n",
+ ret);
+ ret = -EINVAL;
+ }
+
+ release_firmware(fw);
+ return ret;
+}
+EXPORT_SYMBOL(snd_sof_load_topology);
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Liam Girdwood <liam.r.girdwood@linux.intel.com>
+//
+
+#include <linux/debugfs.h>
+#include <linux/sched/signal.h>
+#include "sof-priv.h"
+#include "ops.h"
+
+static size_t sof_wait_trace_avail(struct snd_sof_dev *sdev,
+ loff_t pos, size_t buffer_size)
+{
+ wait_queue_entry_t wait;
+ loff_t host_offset = READ_ONCE(sdev->host_offset);
+
+ /*
+ * If host offset is less than local pos, it means write pointer of
+ * host DMA buffer has been wrapped. We should output the trace data
+ * at the end of host DMA buffer at first.
+ */
+ if (host_offset < pos)
+ return buffer_size - pos;
+
+ /* If there is available trace data now, it is unnecessary to wait. */
+ if (host_offset > pos)
+ return host_offset - pos;
+
+ /* wait for available trace data from FW */
+ init_waitqueue_entry(&wait, current);
+ set_current_state(TASK_INTERRUPTIBLE);
+ add_wait_queue(&sdev->trace_sleep, &wait);
+
+ if (!signal_pending(current)) {
+ /* set timeout to max value, no error code */
+ schedule_timeout(MAX_SCHEDULE_TIMEOUT);
+ }
+ remove_wait_queue(&sdev->trace_sleep, &wait);
+
+ /* return bytes available for copy */
+ host_offset = READ_ONCE(sdev->host_offset);
+ if (host_offset < pos)
+ return buffer_size - pos;
+
+ return host_offset - pos;
+}
+
+static ssize_t sof_dfsentry_trace_read(struct file *file, char __user *buffer,
+ size_t count, loff_t *ppos)
+{
+ struct snd_sof_dfsentry *dfse = file->private_data;
+ struct snd_sof_dev *sdev = dfse->sdev;
+ unsigned long rem;
+ loff_t lpos = *ppos;
+ size_t avail, buffer_size = dfse->size;
+ u64 lpos_64;
+
+ /* make sure we know about any failures on the DSP side */
+ sdev->dtrace_error = false;
+
+ /* check pos and count */
+ if (lpos < 0)
+ return -EINVAL;
+ if (!count)
+ return 0;
+
+ /* check for buffer wrap and count overflow */
+ lpos_64 = lpos;
+ lpos = do_div(lpos_64, buffer_size);
+
+ if (count > buffer_size - lpos) /* min() not used to avoid sparse warnings */
+ count = buffer_size - lpos;
+
+ /* get available count based on current host offset */
+ avail = sof_wait_trace_avail(sdev, lpos, buffer_size);
+ if (sdev->dtrace_error) {
+ dev_err(sdev->dev, "error: trace IO error\n");
+ return -EIO;
+ }
+
+ /* make sure count is <= avail */
+ count = avail > count ? count : avail;
+
+ /* copy available trace data to debugfs */
+ rem = copy_to_user(buffer, ((u8 *)(dfse->buf) + lpos), count);
+ if (rem)
+ return -EFAULT;
+
+ *ppos += count;
+
+ /* move debugfs reading position */
+ return count;
+}
+
+static const struct file_operations sof_dfs_trace_fops = {
+ .open = simple_open,
+ .read = sof_dfsentry_trace_read,
+ .llseek = default_llseek,
+};
+
+static int trace_debugfs_create(struct snd_sof_dev *sdev)
+{
+ struct snd_sof_dfsentry *dfse;
+
+ if (!sdev)
+ return -EINVAL;
+
+ dfse = devm_kzalloc(sdev->dev, sizeof(*dfse), GFP_KERNEL);
+ if (!dfse)
+ return -ENOMEM;
+
+ dfse->type = SOF_DFSENTRY_TYPE_BUF;
+ dfse->buf = sdev->dmatb.area;
+ dfse->size = sdev->dmatb.bytes;
+ dfse->sdev = sdev;
+
+ dfse->dfsentry = debugfs_create_file("trace", 0444, sdev->debugfs_root,
+ dfse, &sof_dfs_trace_fops);
+ if (!dfse->dfsentry) {
+ /* can't rely on debugfs, only log error and keep going */
+ dev_err(sdev->dev,
+ "error: cannot create debugfs entry for trace\n");
+ }
+
+ return 0;
+}
+
+int snd_sof_init_trace_ipc(struct snd_sof_dev *sdev)
+{
+ struct sof_ipc_dma_trace_params params;
+ struct sof_ipc_reply ipc_reply;
+ int ret;
+
+ if (sdev->dtrace_is_enabled || !sdev->dma_trace_pages)
+ return -EINVAL;
+
+ /* set IPC parameters */
+ params.hdr.size = sizeof(params);
+ params.hdr.cmd = SOF_IPC_GLB_TRACE_MSG | SOF_IPC_TRACE_DMA_PARAMS;
+ params.buffer.phy_addr = sdev->dmatp.addr;
+ params.buffer.size = sdev->dmatb.bytes;
+ params.buffer.pages = sdev->dma_trace_pages;
+ params.stream_tag = 0;
+
+ sdev->host_offset = 0;
+
+ ret = snd_sof_dma_trace_init(sdev, ¶ms.stream_tag);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: fail in snd_sof_dma_trace_init %d\n", ret);
+ return ret;
+ }
+ dev_dbg(sdev->dev, "stream_tag: %d\n", params.stream_tag);
+
+ /* send IPC to the DSP */
+ ret = sof_ipc_tx_message(sdev->ipc,
+ params.hdr.cmd, ¶ms, sizeof(params),
+ &ipc_reply, sizeof(ipc_reply));
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: can't set params for DMA for trace %d\n", ret);
+ goto trace_release;
+ }
+
+ ret = snd_sof_dma_trace_trigger(sdev, SNDRV_PCM_TRIGGER_START);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: snd_sof_dma_trace_trigger: start: %d\n", ret);
+ goto trace_release;
+ }
+
+ sdev->dtrace_is_enabled = true;
+
+ return 0;
+
+trace_release:
+ snd_sof_dma_trace_release(sdev);
+ return ret;
+}
+
+int snd_sof_init_trace(struct snd_sof_dev *sdev)
+{
+ int ret;
+
+ /* set false before start initialization */
+ sdev->dtrace_is_enabled = false;
+
+ /* allocate trace page table buffer */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV, sdev->dev,
+ PAGE_SIZE, &sdev->dmatp);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: can't alloc page table for trace %d\n", ret);
+ return ret;
+ }
+
+ /* allocate trace data buffer */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV_SG, sdev->dev,
+ DMA_BUF_SIZE_FOR_TRACE, &sdev->dmatb);
+ if (ret < 0) {
+ dev_err(sdev->dev,
+ "error: can't alloc buffer for trace %d\n", ret);
+ goto page_err;
+ }
+
+ /* create compressed page table for audio firmware */
+ ret = snd_sof_create_page_table(sdev, &sdev->dmatb, sdev->dmatp.area,
+ sdev->dmatb.bytes);
+ if (ret < 0)
+ goto table_err;
+
+ sdev->dma_trace_pages = ret;
+ dev_dbg(sdev->dev, "dma_trace_pages: %d\n", sdev->dma_trace_pages);
+
+ if (sdev->first_boot) {
+ ret = trace_debugfs_create(sdev);
+ if (ret < 0)
+ goto table_err;
+ }
+
+ init_waitqueue_head(&sdev->trace_sleep);
+
+ ret = snd_sof_init_trace_ipc(sdev);
+ if (ret < 0)
+ goto table_err;
+
+ return 0;
+table_err:
+ sdev->dma_trace_pages = 0;
+ snd_dma_free_pages(&sdev->dmatb);
+page_err:
+ snd_dma_free_pages(&sdev->dmatp);
+ return ret;
+}
+EXPORT_SYMBOL(snd_sof_init_trace);
+
+int snd_sof_trace_update_pos(struct snd_sof_dev *sdev,
+ struct sof_ipc_dma_trace_posn *posn)
+{
+ if (sdev->dtrace_is_enabled && sdev->host_offset != posn->host_offset) {
+ sdev->host_offset = posn->host_offset;
+ wake_up(&sdev->trace_sleep);
+ }
+
+ if (posn->overflow != 0)
+ dev_err(sdev->dev,
+ "error: DSP trace buffer overflow %u bytes. Total messages %d\n",
+ posn->overflow, posn->messages);
+
+ return 0;
+}
+
+/* an error has occurred within the DSP that prevents further trace */
+void snd_sof_trace_notify_for_error(struct snd_sof_dev *sdev)
+{
+ if (sdev->dtrace_is_enabled) {
+ dev_err(sdev->dev, "error: waking up any trace sleepers\n");
+ sdev->dtrace_error = true;
+ wake_up(&sdev->trace_sleep);
+ }
+}
+EXPORT_SYMBOL(snd_sof_trace_notify_for_error);
+
+void snd_sof_release_trace(struct snd_sof_dev *sdev)
+{
+ int ret;
+
+ if (!sdev->dtrace_is_enabled)
+ return;
+
+ ret = snd_sof_dma_trace_trigger(sdev, SNDRV_PCM_TRIGGER_STOP);
+ if (ret < 0)
+ dev_err(sdev->dev,
+ "error: snd_sof_dma_trace_trigger: stop: %d\n", ret);
+
+ ret = snd_sof_dma_trace_release(sdev);
+ if (ret < 0)
+ dev_err(sdev->dev,
+ "error: fail in snd_sof_dma_trace_release %d\n", ret);
+
+ sdev->dtrace_is_enabled = false;
+}
+EXPORT_SYMBOL(snd_sof_release_trace);
+
+void snd_sof_free_trace(struct snd_sof_dev *sdev)
+{
+ snd_sof_release_trace(sdev);
+
+ snd_dma_free_pages(&sdev->dmatb);
+ snd_dma_free_pages(&sdev->dmatp);
+}
+EXPORT_SYMBOL(snd_sof_free_trace);
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Keyon Jie <yang.jie@linux.intel.com>
+//
+
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/platform_device.h>
+#include <sound/soc.h>
+#include <sound/sof.h>
+#include "sof-priv.h"
+
+/*
+ * Register IO
+ *
+ * The sof_io_xyz() wrappers are typically referenced in snd_sof_dsp_ops
+ * structures and cannot be inlined.
+ */
+
+void sof_io_write(struct snd_sof_dev *sdev, void __iomem *addr, u32 value)
+{
+ writel(value, addr);
+}
+EXPORT_SYMBOL(sof_io_write);
+
+u32 sof_io_read(struct snd_sof_dev *sdev, void __iomem *addr)
+{
+ return readl(addr);
+}
+EXPORT_SYMBOL(sof_io_read);
+
+void sof_io_write64(struct snd_sof_dev *sdev, void __iomem *addr, u64 value)
+{
+ writeq(value, addr);
+}
+EXPORT_SYMBOL(sof_io_write64);
+
+u64 sof_io_read64(struct snd_sof_dev *sdev, void __iomem *addr)
+{
+ return readq(addr);
+}
+EXPORT_SYMBOL(sof_io_read64);
+
+/*
+ * IPC Mailbox IO
+ */
+
+void sof_mailbox_write(struct snd_sof_dev *sdev, u32 offset,
+ void *message, size_t bytes)
+{
+ void __iomem *dest = sdev->bar[sdev->mailbox_bar] + offset;
+
+ memcpy_toio(dest, message, bytes);
+}
+EXPORT_SYMBOL(sof_mailbox_write);
+
+void sof_mailbox_read(struct snd_sof_dev *sdev, u32 offset,
+ void *message, size_t bytes)
+{
+ void __iomem *src = sdev->bar[sdev->mailbox_bar] + offset;
+
+ memcpy_fromio(message, src, bytes);
+}
+EXPORT_SYMBOL(sof_mailbox_read);
+
+/*
+ * Memory copy.
+ */
+
+void sof_block_write(struct snd_sof_dev *sdev, u32 bar, u32 offset, void *src,
+ size_t size)
+{
+ void __iomem *dest = sdev->bar[bar] + offset;
+ const u8 *src_byte = src;
+ u32 affected_mask;
+ u32 tmp;
+ int m, n;
+
+ m = size / 4;
+ n = size % 4;
+
+ /* __iowrite32_copy use 32bit size values so divide by 4 */
+ __iowrite32_copy(dest, src, m);
+
+ if (n) {
+ affected_mask = (1 << (8 * n)) - 1;
+
+ /* first read the 32bit data of dest, then change affected
+ * bytes, and write back to dest. For unaffected bytes, it
+ * should not be changed
+ */
+ tmp = ioread32(dest + m * 4);
+ tmp &= ~affected_mask;
+
+ tmp |= *(u32 *)(src_byte + m * 4) & affected_mask;
+ iowrite32(tmp, dest + m * 4);
+ }
+}
+EXPORT_SYMBOL(sof_block_write);
+
+void sof_block_read(struct snd_sof_dev *sdev, u32 bar, u32 offset, void *dest,
+ size_t size)
+{
+ void __iomem *src = sdev->bar[bar] + offset;
+
+ memcpy_fromio(dest, src, size);
+}
+EXPORT_SYMBOL(sof_block_read);
--- /dev/null
+config SND_SOC_SOF_XTENSA
+ tristate
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+
+snd-sof-xtensa-dsp-objs := core.o
+
+obj-$(CONFIG_SND_SOC_SOF_XTENSA) += snd-sof-xtensa-dsp.o
--- /dev/null
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license. When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018 Intel Corporation. All rights reserved.
+//
+// Author: Pan Xiuli <xiuli.pan@linux.intel.com>
+//
+
+#include <linux/module.h>
+#include <sound/sof.h>
+#include <sound/sof/xtensa.h>
+#include "../sof-priv.h"
+
+struct xtensa_exception_cause {
+ u32 id;
+ const char *msg;
+ const char *description;
+};
+
+/*
+ * From 4.4.1.5 table 4-64 Exception Causes of Xtensa
+ * Instruction Set Architecture (ISA) Reference Manual
+ */
+static const struct xtensa_exception_cause xtensa_exception_causes[] = {
+ {0, "IllegalInstructionCause", "Illegal instruction"},
+ {1, "SyscallCause", "SYSCALL instruction"},
+ {2, "InstructionFetchErrorCause",
+ "Processor internal physical address or data error during instruction fetch"},
+ {3, "LoadStoreErrorCause",
+ "Processor internal physical address or data error during load or store"},
+ {4, "Level1InterruptCause",
+ "Level-1 interrupt as indicated by set level-1 bits in the INTERRUPT register"},
+ {5, "AllocaCause",
+ "MOVSP instruction, if caller’s registers are not in the register file"},
+ {6, "IntegerDivideByZeroCause",
+ "QUOS, QUOU, REMS, or REMU divisor operand is zero"},
+ {8, "PrivilegedCause",
+ "Attempt to execute a privileged operation when CRING ? 0"},
+ {9, "LoadStoreAlignmentCause", "Load or store to an unaligned address"},
+ {12, "InstrPIFDataErrorCause",
+ "PIF data error during instruction fetch"},
+ {13, "LoadStorePIFDataErrorCause",
+ "Synchronous PIF data error during LoadStore access"},
+ {14, "InstrPIFAddrErrorCause",
+ "PIF address error during instruction fetch"},
+ {15, "LoadStorePIFAddrErrorCause",
+ "Synchronous PIF address error during LoadStore access"},
+ {16, "InstTLBMissCause", "Error during Instruction TLB refill"},
+ {17, "InstTLBMultiHitCause",
+ "Multiple instruction TLB entries matched"},
+ {18, "InstFetchPrivilegeCause",
+ "An instruction fetch referenced a virtual address at a ring level less than CRING"},
+ {20, "InstFetchProhibitedCause",
+ "An instruction fetch referenced a page mapped with an attribute that does not permit instruction fetch"},
+ {24, "LoadStoreTLBMissCause",
+ "Error during TLB refill for a load or store"},
+ {25, "LoadStoreTLBMultiHitCause",
+ "Multiple TLB entries matched for a load or store"},
+ {26, "LoadStorePrivilegeCause",
+ "A load or store referenced a virtual address at a ring level less than CRING"},
+ {28, "LoadProhibitedCause",
+ "A load referenced a page mapped with an attribute that does not permit loads"},
+ {32, "Coprocessor0Disabled",
+ "Coprocessor 0 instruction when cp0 disabled"},
+ {33, "Coprocessor1Disabled",
+ "Coprocessor 1 instruction when cp1 disabled"},
+ {34, "Coprocessor2Disabled",
+ "Coprocessor 2 instruction when cp2 disabled"},
+ {35, "Coprocessor3Disabled",
+ "Coprocessor 3 instruction when cp3 disabled"},
+ {36, "Coprocessor4Disabled",
+ "Coprocessor 4 instruction when cp4 disabled"},
+ {37, "Coprocessor5Disabled",
+ "Coprocessor 5 instruction when cp5 disabled"},
+ {38, "Coprocessor6Disabled",
+ "Coprocessor 6 instruction when cp6 disabled"},
+ {39, "Coprocessor7Disabled",
+ "Coprocessor 7 instruction when cp7 disabled"},
+};
+
+/* only need xtensa atm */
+static void xtensa_dsp_oops(struct snd_sof_dev *sdev, void *oops)
+{
+ struct sof_ipc_dsp_oops_xtensa *xoops = oops;
+ int i;
+
+ dev_err(sdev->dev, "error: DSP Firmware Oops\n");
+ for (i = 0; i < ARRAY_SIZE(xtensa_exception_causes); i++) {
+ if (xtensa_exception_causes[i].id == xoops->exccause) {
+ dev_err(sdev->dev, "error: Exception Cause: %s, %s\n",
+ xtensa_exception_causes[i].msg,
+ xtensa_exception_causes[i].description);
+ }
+ }
+ dev_err(sdev->dev, "EXCCAUSE 0x%8.8x EXCVADDR 0x%8.8x PS 0x%8.8x SAR 0x%8.8x\n",
+ xoops->exccause, xoops->excvaddr, xoops->ps, xoops->sar);
+ dev_err(sdev->dev, "EPC1 0x%8.8x EPC2 0x%8.8x EPC3 0x%8.8x EPC4 0x%8.8x",
+ xoops->epc1, xoops->epc2, xoops->epc3, xoops->epc4);
+ dev_err(sdev->dev, "EPC5 0x%8.8x EPC6 0x%8.8x EPC7 0x%8.8x DEPC 0x%8.8x",
+ xoops->epc5, xoops->epc6, xoops->epc7, xoops->depc);
+ dev_err(sdev->dev, "EPS2 0x%8.8x EPS3 0x%8.8x EPS4 0x%8.8x EPS5 0x%8.8x",
+ xoops->eps2, xoops->eps3, xoops->eps4, xoops->eps5);
+ dev_err(sdev->dev, "EPS6 0x%8.8x EPS7 0x%8.8x INTENABL 0x%8.8x INTERRU 0x%8.8x",
+ xoops->eps6, xoops->eps7, xoops->intenable, xoops->interrupt);
+}
+
+static void xtensa_stack(struct snd_sof_dev *sdev, void *oops, u32 *stack,
+ u32 stack_words)
+{
+ struct sof_ipc_dsp_oops_xtensa *xoops = oops;
+ u32 stack_ptr = xoops->stack;
+ /* 4 * 8chars + 3 ws + 1 terminating NUL */
+ unsigned char buf[4 * 8 + 3 + 1];
+ int i;
+
+ dev_err(sdev->dev, "stack dump from 0x%8.8x\n", stack_ptr);
+
+ /*
+ * example output:
+ * 0x0049fbb0: 8000f2d0 0049fc00 6f6c6c61 00632e63
+ */
+ for (i = 0; i < stack_words; i += 4) {
+ hex_dump_to_buffer(stack + i * 4, 16, 16, 4,
+ buf, sizeof(buf), false);
+ dev_err(sdev->dev, "0x%08x: %s\n", stack_ptr + i, buf);
+ }
+}
+
+const struct sof_arch_ops sof_xtensa_arch_ops = {
+ .dsp_oops = xtensa_dsp_oops,
+ .dsp_stack = xtensa_stack,
+};
+EXPORT_SYMBOL(sof_xtensa_arch_ops);
+
+MODULE_DESCRIPTION("SOF Xtensa DSP support");
+MODULE_LICENSE("Dual BSD/GPL");
config SND_SOC_SPRD
tristate "SoC Audio for the Spreadtrum SoC chips"
depends on ARCH_SPRD || COMPILE_TEST
+ select SND_SOC_COMPRESS
help
Say Y or M if you want to add support for codecs attached to
the Spreadtrum SoCs' Audio interfaces.
+
+config SND_SOC_SPRD_MCDT
+ bool "Spreadtrum multi-channel data transfer support"
+ depends on SND_SOC_SPRD
+ help
+ Say y here to enable multi-channel data transfer support. It
+ is used for sound stream transmission between audio subsystem
+ and other AP/CP subsystem.
# SPDX-License-Identifier: GPL-2.0
# Spreadtrum Audio Support
-obj-$(CONFIG_SND_SOC_SPRD) += sprd-pcm-dma.o
+snd-soc-sprd-platform-objs := sprd-pcm-dma.o sprd-pcm-compress.o
+
+obj-$(CONFIG_SND_SOC_SPRD) += snd-soc-sprd-platform.o
+
+obj-$(CONFIG_SND_SOC_SPRD_MCDT) += sprd-mcdt.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2019 Spreadtrum Communications Inc.
+
+#include <linux/errno.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/spinlock.h>
+
+#include "sprd-mcdt.h"
+
+/* MCDT registers definition */
+#define MCDT_CH0_TXD 0x0
+#define MCDT_CH0_RXD 0x28
+#define MCDT_DAC0_WTMK 0x60
+#define MCDT_ADC0_WTMK 0x88
+#define MCDT_DMA_EN 0xb0
+
+#define MCDT_INT_EN0 0xb4
+#define MCDT_INT_EN1 0xb8
+#define MCDT_INT_EN2 0xbc
+
+#define MCDT_INT_CLR0 0xc0
+#define MCDT_INT_CLR1 0xc4
+#define MCDT_INT_CLR2 0xc8
+
+#define MCDT_INT_RAW1 0xcc
+#define MCDT_INT_RAW2 0xd0
+#define MCDT_INT_RAW3 0xd4
+
+#define MCDT_INT_MSK1 0xd8
+#define MCDT_INT_MSK2 0xdc
+#define MCDT_INT_MSK3 0xe0
+
+#define MCDT_DAC0_FIFO_ADDR_ST 0xe4
+#define MCDT_ADC0_FIFO_ADDR_ST 0xe8
+
+#define MCDT_CH_FIFO_ST0 0x134
+#define MCDT_CH_FIFO_ST1 0x138
+#define MCDT_CH_FIFO_ST2 0x13c
+
+#define MCDT_INT_MSK_CFG0 0x140
+#define MCDT_INT_MSK_CFG1 0x144
+
+#define MCDT_DMA_CFG0 0x148
+#define MCDT_FIFO_CLR 0x14c
+#define MCDT_DMA_CFG1 0x150
+#define MCDT_DMA_CFG2 0x154
+#define MCDT_DMA_CFG3 0x158
+#define MCDT_DMA_CFG4 0x15c
+#define MCDT_DMA_CFG5 0x160
+
+/* Channel water mark definition */
+#define MCDT_CH_FIFO_AE_SHIFT 16
+#define MCDT_CH_FIFO_AE_MASK GENMASK(24, 16)
+#define MCDT_CH_FIFO_AF_MASK GENMASK(8, 0)
+
+/* DMA channel select definition */
+#define MCDT_DMA_CH0_SEL_MASK GENMASK(3, 0)
+#define MCDT_DMA_CH0_SEL_SHIFT 0
+#define MCDT_DMA_CH1_SEL_MASK GENMASK(7, 4)
+#define MCDT_DMA_CH1_SEL_SHIFT 4
+#define MCDT_DMA_CH2_SEL_MASK GENMASK(11, 8)
+#define MCDT_DMA_CH2_SEL_SHIFT 8
+#define MCDT_DMA_CH3_SEL_MASK GENMASK(15, 12)
+#define MCDT_DMA_CH3_SEL_SHIFT 12
+#define MCDT_DMA_CH4_SEL_MASK GENMASK(19, 16)
+#define MCDT_DMA_CH4_SEL_SHIFT 16
+#define MCDT_DAC_DMA_SHIFT 16
+
+/* DMA channel ACK select definition */
+#define MCDT_DMA_ACK_SEL_MASK GENMASK(3, 0)
+
+/* Channel FIFO definition */
+#define MCDT_CH_FIFO_ADDR_SHIFT 16
+#define MCDT_CH_FIFO_ADDR_MASK GENMASK(9, 0)
+#define MCDT_ADC_FIFO_SHIFT 16
+#define MCDT_FIFO_LENGTH 512
+
+#define MCDT_ADC_CHANNEL_NUM 10
+#define MCDT_DAC_CHANNEL_NUM 10
+#define MCDT_CHANNEL_NUM (MCDT_ADC_CHANNEL_NUM + MCDT_DAC_CHANNEL_NUM)
+
+enum sprd_mcdt_fifo_int {
+ MCDT_ADC_FIFO_AE_INT,
+ MCDT_ADC_FIFO_AF_INT,
+ MCDT_DAC_FIFO_AE_INT,
+ MCDT_DAC_FIFO_AF_INT,
+ MCDT_ADC_FIFO_OV_INT,
+ MCDT_DAC_FIFO_OV_INT
+};
+
+enum sprd_mcdt_fifo_sts {
+ MCDT_ADC_FIFO_REAL_FULL,
+ MCDT_ADC_FIFO_REAL_EMPTY,
+ MCDT_ADC_FIFO_AF,
+ MCDT_ADC_FIFO_AE,
+ MCDT_DAC_FIFO_REAL_FULL,
+ MCDT_DAC_FIFO_REAL_EMPTY,
+ MCDT_DAC_FIFO_AF,
+ MCDT_DAC_FIFO_AE
+};
+
+struct sprd_mcdt_dev {
+ struct device *dev;
+ void __iomem *base;
+ spinlock_t lock;
+ struct sprd_mcdt_chan chan[MCDT_CHANNEL_NUM];
+};
+
+static LIST_HEAD(sprd_mcdt_chan_list);
+static DEFINE_MUTEX(sprd_mcdt_list_mutex);
+
+static void sprd_mcdt_update(struct sprd_mcdt_dev *mcdt, u32 reg, u32 val,
+ u32 mask)
+{
+ u32 orig = readl_relaxed(mcdt->base + reg);
+ u32 tmp;
+
+ tmp = (orig & ~mask) | val;
+ writel_relaxed(tmp, mcdt->base + reg);
+}
+
+static void sprd_mcdt_dac_set_watermark(struct sprd_mcdt_dev *mcdt, u8 channel,
+ u32 full, u32 empty)
+{
+ u32 reg = MCDT_DAC0_WTMK + channel * 4;
+ u32 water_mark =
+ (empty << MCDT_CH_FIFO_AE_SHIFT) & MCDT_CH_FIFO_AE_MASK;
+
+ water_mark |= full & MCDT_CH_FIFO_AF_MASK;
+ sprd_mcdt_update(mcdt, reg, water_mark,
+ MCDT_CH_FIFO_AE_MASK | MCDT_CH_FIFO_AF_MASK);
+}
+
+static void sprd_mcdt_adc_set_watermark(struct sprd_mcdt_dev *mcdt, u8 channel,
+ u32 full, u32 empty)
+{
+ u32 reg = MCDT_ADC0_WTMK + channel * 4;
+ u32 water_mark =
+ (empty << MCDT_CH_FIFO_AE_SHIFT) & MCDT_CH_FIFO_AE_MASK;
+
+ water_mark |= full & MCDT_CH_FIFO_AF_MASK;
+ sprd_mcdt_update(mcdt, reg, water_mark,
+ MCDT_CH_FIFO_AE_MASK | MCDT_CH_FIFO_AF_MASK);
+}
+
+static void sprd_mcdt_dac_dma_enable(struct sprd_mcdt_dev *mcdt, u8 channel,
+ bool enable)
+{
+ u32 shift = MCDT_DAC_DMA_SHIFT + channel;
+
+ if (enable)
+ sprd_mcdt_update(mcdt, MCDT_DMA_EN, BIT(shift), BIT(shift));
+ else
+ sprd_mcdt_update(mcdt, MCDT_DMA_EN, 0, BIT(shift));
+}
+
+static void sprd_mcdt_adc_dma_enable(struct sprd_mcdt_dev *mcdt, u8 channel,
+ bool enable)
+{
+ if (enable)
+ sprd_mcdt_update(mcdt, MCDT_DMA_EN, BIT(channel), BIT(channel));
+ else
+ sprd_mcdt_update(mcdt, MCDT_DMA_EN, 0, BIT(channel));
+}
+
+static void sprd_mcdt_ap_int_enable(struct sprd_mcdt_dev *mcdt, u8 channel,
+ bool enable)
+{
+ if (enable)
+ sprd_mcdt_update(mcdt, MCDT_INT_MSK_CFG0, BIT(channel),
+ BIT(channel));
+ else
+ sprd_mcdt_update(mcdt, MCDT_INT_MSK_CFG0, 0, BIT(channel));
+}
+
+static void sprd_mcdt_dac_write_fifo(struct sprd_mcdt_dev *mcdt, u8 channel,
+ u32 val)
+{
+ u32 reg = MCDT_CH0_TXD + channel * 4;
+
+ writel_relaxed(val, mcdt->base + reg);
+}
+
+static void sprd_mcdt_adc_read_fifo(struct sprd_mcdt_dev *mcdt, u8 channel,
+ u32 *val)
+{
+ u32 reg = MCDT_CH0_RXD + channel * 4;
+
+ *val = readl_relaxed(mcdt->base + reg);
+}
+
+static void sprd_mcdt_dac_dma_chn_select(struct sprd_mcdt_dev *mcdt, u8 channel,
+ enum sprd_mcdt_dma_chan dma_chan)
+{
+ switch (dma_chan) {
+ case SPRD_MCDT_DMA_CH0:
+ sprd_mcdt_update(mcdt, MCDT_DMA_CFG0,
+ channel << MCDT_DMA_CH0_SEL_SHIFT,
+ MCDT_DMA_CH0_SEL_MASK);
+ break;
+
+ case SPRD_MCDT_DMA_CH1:
+ sprd_mcdt_update(mcdt, MCDT_DMA_CFG0,
+ channel << MCDT_DMA_CH1_SEL_SHIFT,
+ MCDT_DMA_CH1_SEL_MASK);
+ break;
+
+ case SPRD_MCDT_DMA_CH2:
+ sprd_mcdt_update(mcdt, MCDT_DMA_CFG0,
+ channel << MCDT_DMA_CH2_SEL_SHIFT,
+ MCDT_DMA_CH2_SEL_MASK);
+ break;
+
+ case SPRD_MCDT_DMA_CH3:
+ sprd_mcdt_update(mcdt, MCDT_DMA_CFG0,
+ channel << MCDT_DMA_CH3_SEL_SHIFT,
+ MCDT_DMA_CH3_SEL_MASK);
+ break;
+
+ case SPRD_MCDT_DMA_CH4:
+ sprd_mcdt_update(mcdt, MCDT_DMA_CFG0,
+ channel << MCDT_DMA_CH4_SEL_SHIFT,
+ MCDT_DMA_CH4_SEL_MASK);
+ break;
+ }
+}
+
+static void sprd_mcdt_adc_dma_chn_select(struct sprd_mcdt_dev *mcdt, u8 channel,
+ enum sprd_mcdt_dma_chan dma_chan)
+{
+ switch (dma_chan) {
+ case SPRD_MCDT_DMA_CH0:
+ sprd_mcdt_update(mcdt, MCDT_DMA_CFG1,
+ channel << MCDT_DMA_CH0_SEL_SHIFT,
+ MCDT_DMA_CH0_SEL_MASK);
+ break;
+
+ case SPRD_MCDT_DMA_CH1:
+ sprd_mcdt_update(mcdt, MCDT_DMA_CFG1,
+ channel << MCDT_DMA_CH1_SEL_SHIFT,
+ MCDT_DMA_CH1_SEL_MASK);
+ break;
+
+ case SPRD_MCDT_DMA_CH2:
+ sprd_mcdt_update(mcdt, MCDT_DMA_CFG1,
+ channel << MCDT_DMA_CH2_SEL_SHIFT,
+ MCDT_DMA_CH2_SEL_MASK);
+ break;
+
+ case SPRD_MCDT_DMA_CH3:
+ sprd_mcdt_update(mcdt, MCDT_DMA_CFG1,
+ channel << MCDT_DMA_CH3_SEL_SHIFT,
+ MCDT_DMA_CH3_SEL_MASK);
+ break;
+
+ case SPRD_MCDT_DMA_CH4:
+ sprd_mcdt_update(mcdt, MCDT_DMA_CFG1,
+ channel << MCDT_DMA_CH4_SEL_SHIFT,
+ MCDT_DMA_CH4_SEL_MASK);
+ break;
+ }
+}
+
+static u32 sprd_mcdt_dma_ack_shift(u8 channel)
+{
+ switch (channel) {
+ default:
+ case 0:
+ case 8:
+ return 0;
+ case 1:
+ case 9:
+ return 4;
+ case 2:
+ return 8;
+ case 3:
+ return 12;
+ case 4:
+ return 16;
+ case 5:
+ return 20;
+ case 6:
+ return 24;
+ case 7:
+ return 28;
+ }
+}
+
+static void sprd_mcdt_dac_dma_ack_select(struct sprd_mcdt_dev *mcdt, u8 channel,
+ enum sprd_mcdt_dma_chan dma_chan)
+{
+ u32 reg, shift = sprd_mcdt_dma_ack_shift(channel), ack = dma_chan;
+
+ switch (channel) {
+ case 0 ... 7:
+ reg = MCDT_DMA_CFG2;
+ break;
+
+ case 8 ... 9:
+ reg = MCDT_DMA_CFG3;
+ break;
+
+ default:
+ return;
+ }
+
+ sprd_mcdt_update(mcdt, reg, ack << shift,
+ MCDT_DMA_ACK_SEL_MASK << shift);
+}
+
+static void sprd_mcdt_adc_dma_ack_select(struct sprd_mcdt_dev *mcdt, u8 channel,
+ enum sprd_mcdt_dma_chan dma_chan)
+{
+ u32 reg, shift = sprd_mcdt_dma_ack_shift(channel), ack = dma_chan;
+
+ switch (channel) {
+ case 0 ... 7:
+ reg = MCDT_DMA_CFG4;
+ break;
+
+ case 8 ... 9:
+ reg = MCDT_DMA_CFG5;
+ break;
+
+ default:
+ return;
+ }
+
+ sprd_mcdt_update(mcdt, reg, ack << shift,
+ MCDT_DMA_ACK_SEL_MASK << shift);
+}
+
+static bool sprd_mcdt_chan_fifo_sts(struct sprd_mcdt_dev *mcdt, u8 channel,
+ enum sprd_mcdt_fifo_sts fifo_sts)
+{
+ u32 reg, shift;
+
+ switch (channel) {
+ case 0 ... 3:
+ reg = MCDT_CH_FIFO_ST0;
+ break;
+ case 4 ... 7:
+ reg = MCDT_CH_FIFO_ST1;
+ break;
+ case 8 ... 9:
+ reg = MCDT_CH_FIFO_ST2;
+ break;
+ default:
+ return false;
+ }
+
+ switch (channel) {
+ case 0:
+ case 4:
+ case 8:
+ shift = fifo_sts;
+ break;
+
+ case 1:
+ case 5:
+ case 9:
+ shift = 8 + fifo_sts;
+ break;
+
+ case 2:
+ case 6:
+ shift = 16 + fifo_sts;
+ break;
+
+ case 3:
+ case 7:
+ shift = 24 + fifo_sts;
+ break;
+
+ default:
+ return false;
+ }
+
+ return !!(readl_relaxed(mcdt->base + reg) & BIT(shift));
+}
+
+static void sprd_mcdt_dac_fifo_clear(struct sprd_mcdt_dev *mcdt, u8 channel)
+{
+ sprd_mcdt_update(mcdt, MCDT_FIFO_CLR, BIT(channel), BIT(channel));
+}
+
+static void sprd_mcdt_adc_fifo_clear(struct sprd_mcdt_dev *mcdt, u8 channel)
+{
+ u32 shift = MCDT_ADC_FIFO_SHIFT + channel;
+
+ sprd_mcdt_update(mcdt, MCDT_FIFO_CLR, BIT(shift), BIT(shift));
+}
+
+static u32 sprd_mcdt_dac_fifo_avail(struct sprd_mcdt_dev *mcdt, u8 channel)
+{
+ u32 reg = MCDT_DAC0_FIFO_ADDR_ST + channel * 8;
+ u32 r_addr = (readl_relaxed(mcdt->base + reg) >>
+ MCDT_CH_FIFO_ADDR_SHIFT) & MCDT_CH_FIFO_ADDR_MASK;
+ u32 w_addr = readl_relaxed(mcdt->base + reg) & MCDT_CH_FIFO_ADDR_MASK;
+
+ if (w_addr >= r_addr)
+ return 4 * (MCDT_FIFO_LENGTH - w_addr + r_addr);
+ else
+ return 4 * (r_addr - w_addr);
+}
+
+static u32 sprd_mcdt_adc_fifo_avail(struct sprd_mcdt_dev *mcdt, u8 channel)
+{
+ u32 reg = MCDT_ADC0_FIFO_ADDR_ST + channel * 8;
+ u32 r_addr = (readl_relaxed(mcdt->base + reg) >>
+ MCDT_CH_FIFO_ADDR_SHIFT) & MCDT_CH_FIFO_ADDR_MASK;
+ u32 w_addr = readl_relaxed(mcdt->base + reg) & MCDT_CH_FIFO_ADDR_MASK;
+
+ if (w_addr >= r_addr)
+ return 4 * (w_addr - r_addr);
+ else
+ return 4 * (MCDT_FIFO_LENGTH - r_addr + w_addr);
+}
+
+static u32 sprd_mcdt_int_type_shift(u8 channel,
+ enum sprd_mcdt_fifo_int int_type)
+{
+ switch (channel) {
+ case 0:
+ case 4:
+ case 8:
+ return int_type;
+
+ case 1:
+ case 5:
+ case 9:
+ return 8 + int_type;
+
+ case 2:
+ case 6:
+ return 16 + int_type;
+
+ case 3:
+ case 7:
+ return 24 + int_type;
+
+ default:
+ return 0;
+ }
+}
+
+static void sprd_mcdt_chan_int_en(struct sprd_mcdt_dev *mcdt, u8 channel,
+ enum sprd_mcdt_fifo_int int_type, bool enable)
+{
+ u32 reg, shift = sprd_mcdt_int_type_shift(channel, int_type);
+
+ switch (channel) {
+ case 0 ... 3:
+ reg = MCDT_INT_EN0;
+ break;
+ case 4 ... 7:
+ reg = MCDT_INT_EN1;
+ break;
+ case 8 ... 9:
+ reg = MCDT_INT_EN2;
+ break;
+ default:
+ return;
+ }
+
+ if (enable)
+ sprd_mcdt_update(mcdt, reg, BIT(shift), BIT(shift));
+ else
+ sprd_mcdt_update(mcdt, reg, 0, BIT(shift));
+}
+
+static void sprd_mcdt_chan_int_clear(struct sprd_mcdt_dev *mcdt, u8 channel,
+ enum sprd_mcdt_fifo_int int_type)
+{
+ u32 reg, shift = sprd_mcdt_int_type_shift(channel, int_type);
+
+ switch (channel) {
+ case 0 ... 3:
+ reg = MCDT_INT_CLR0;
+ break;
+ case 4 ... 7:
+ reg = MCDT_INT_CLR1;
+ break;
+ case 8 ... 9:
+ reg = MCDT_INT_CLR2;
+ break;
+ default:
+ return;
+ }
+
+ sprd_mcdt_update(mcdt, reg, BIT(shift), BIT(shift));
+}
+
+static bool sprd_mcdt_chan_int_sts(struct sprd_mcdt_dev *mcdt, u8 channel,
+ enum sprd_mcdt_fifo_int int_type)
+{
+ u32 reg, shift = sprd_mcdt_int_type_shift(channel, int_type);
+
+ switch (channel) {
+ case 0 ... 3:
+ reg = MCDT_INT_MSK1;
+ break;
+ case 4 ... 7:
+ reg = MCDT_INT_MSK2;
+ break;
+ case 8 ... 9:
+ reg = MCDT_INT_MSK3;
+ break;
+ default:
+ return false;
+ }
+
+ return !!(readl_relaxed(mcdt->base + reg) & BIT(shift));
+}
+
+static irqreturn_t sprd_mcdt_irq_handler(int irq, void *dev_id)
+{
+ struct sprd_mcdt_dev *mcdt = (struct sprd_mcdt_dev *)dev_id;
+ int i;
+
+ spin_lock(&mcdt->lock);
+
+ for (i = 0; i < MCDT_ADC_CHANNEL_NUM; i++) {
+ if (sprd_mcdt_chan_int_sts(mcdt, i, MCDT_ADC_FIFO_AF_INT)) {
+ struct sprd_mcdt_chan *chan = &mcdt->chan[i];
+
+ sprd_mcdt_chan_int_clear(mcdt, i, MCDT_ADC_FIFO_AF_INT);
+ if (chan->cb)
+ chan->cb->notify(chan->cb->data);
+ }
+ }
+
+ for (i = 0; i < MCDT_DAC_CHANNEL_NUM; i++) {
+ if (sprd_mcdt_chan_int_sts(mcdt, i, MCDT_DAC_FIFO_AE_INT)) {
+ struct sprd_mcdt_chan *chan =
+ &mcdt->chan[i + MCDT_ADC_CHANNEL_NUM];
+
+ sprd_mcdt_chan_int_clear(mcdt, i, MCDT_DAC_FIFO_AE_INT);
+ if (chan->cb)
+ chan->cb->notify(chan->cb->data);
+ }
+ }
+
+ spin_unlock(&mcdt->lock);
+
+ return IRQ_HANDLED;
+}
+
+/**
+ * sprd_mcdt_chan_write - write data to the MCDT channel's fifo
+ * @chan: the MCDT channel
+ * @tx_buf: send buffer
+ * @size: data size
+ *
+ * Note: We can not write data to the channel fifo when enabling the DMA mode,
+ * otherwise the channel fifo data will be invalid.
+ *
+ * If there are not enough space of the channel fifo, it will return errors
+ * to users.
+ *
+ * Returns 0 on success, or an appropriate error code on failure.
+ */
+int sprd_mcdt_chan_write(struct sprd_mcdt_chan *chan, char *tx_buf, u32 size)
+{
+ struct sprd_mcdt_dev *mcdt = chan->mcdt;
+ unsigned long flags;
+ int avail, i = 0, words = size / 4;
+ u32 *buf = (u32 *)tx_buf;
+
+ spin_lock_irqsave(&mcdt->lock, flags);
+
+ if (chan->dma_enable) {
+ dev_err(mcdt->dev,
+ "Can not write data when DMA mode enabled\n");
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return -EINVAL;
+ }
+
+ if (sprd_mcdt_chan_fifo_sts(mcdt, chan->id, MCDT_DAC_FIFO_REAL_FULL)) {
+ dev_err(mcdt->dev, "Channel fifo is full now\n");
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return -EBUSY;
+ }
+
+ avail = sprd_mcdt_dac_fifo_avail(mcdt, chan->id);
+ if (size > avail) {
+ dev_err(mcdt->dev,
+ "Data size is larger than the available fifo size\n");
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return -EBUSY;
+ }
+
+ while (i++ < words)
+ sprd_mcdt_dac_write_fifo(mcdt, chan->id, *buf++);
+
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(sprd_mcdt_chan_write);
+
+/**
+ * sprd_mcdt_chan_read - read data from the MCDT channel's fifo
+ * @chan: the MCDT channel
+ * @rx_buf: receive buffer
+ * @size: data size
+ *
+ * Note: We can not read data from the channel fifo when enabling the DMA mode,
+ * otherwise the reading data will be invalid.
+ *
+ * Usually user need start to read data once receiving the fifo full interrupt.
+ *
+ * Returns data size of reading successfully, or an error code on failure.
+ */
+int sprd_mcdt_chan_read(struct sprd_mcdt_chan *chan, char *rx_buf, u32 size)
+{
+ struct sprd_mcdt_dev *mcdt = chan->mcdt;
+ unsigned long flags;
+ int i = 0, avail, words = size / 4;
+ u32 *buf = (u32 *)rx_buf;
+
+ spin_lock_irqsave(&mcdt->lock, flags);
+
+ if (chan->dma_enable) {
+ dev_err(mcdt->dev, "Can not read data when DMA mode enabled\n");
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return -EINVAL;
+ }
+
+ if (sprd_mcdt_chan_fifo_sts(mcdt, chan->id, MCDT_ADC_FIFO_REAL_EMPTY)) {
+ dev_err(mcdt->dev, "Channel fifo is empty\n");
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return -EBUSY;
+ }
+
+ avail = sprd_mcdt_adc_fifo_avail(mcdt, chan->id);
+ if (size > avail)
+ words = avail / 4;
+
+ while (i++ < words)
+ sprd_mcdt_adc_read_fifo(mcdt, chan->id, buf++);
+
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return words * 4;
+}
+EXPORT_SYMBOL_GPL(sprd_mcdt_chan_read);
+
+/**
+ * sprd_mcdt_chan_int_enable - enable the interrupt mode for the MCDT channel
+ * @chan: the MCDT channel
+ * @water_mark: water mark to trigger a interrupt
+ * @cb: callback when a interrupt happened
+ *
+ * Now it only can enable fifo almost full interrupt for ADC channel and fifo
+ * almost empty interrupt for DAC channel. Morevoer for interrupt mode, user
+ * should use sprd_mcdt_chan_read() or sprd_mcdt_chan_write() to read or write
+ * data manually.
+ *
+ * For ADC channel, user can start to read data once receiving one fifo full
+ * interrupt. For DAC channel, user can start to write data once receiving one
+ * fifo empty interrupt or just call sprd_mcdt_chan_write() to write data
+ * directly.
+ *
+ * Returns 0 on success, or an error code on failure.
+ */
+int sprd_mcdt_chan_int_enable(struct sprd_mcdt_chan *chan, u32 water_mark,
+ struct sprd_mcdt_chan_callback *cb)
+{
+ struct sprd_mcdt_dev *mcdt = chan->mcdt;
+ unsigned long flags;
+ int ret = 0;
+
+ spin_lock_irqsave(&mcdt->lock, flags);
+
+ if (chan->dma_enable || chan->int_enable) {
+ dev_err(mcdt->dev, "Failed to set interrupt mode.\n");
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return -EINVAL;
+ }
+
+ switch (chan->type) {
+ case SPRD_MCDT_ADC_CHAN:
+ sprd_mcdt_adc_fifo_clear(mcdt, chan->id);
+ sprd_mcdt_adc_set_watermark(mcdt, chan->id, water_mark,
+ MCDT_FIFO_LENGTH - 1);
+ sprd_mcdt_chan_int_en(mcdt, chan->id,
+ MCDT_ADC_FIFO_AF_INT, true);
+ sprd_mcdt_ap_int_enable(mcdt, chan->id, true);
+ break;
+
+ case SPRD_MCDT_DAC_CHAN:
+ sprd_mcdt_dac_fifo_clear(mcdt, chan->id);
+ sprd_mcdt_dac_set_watermark(mcdt, chan->id,
+ MCDT_FIFO_LENGTH - 1, water_mark);
+ sprd_mcdt_chan_int_en(mcdt, chan->id,
+ MCDT_DAC_FIFO_AE_INT, true);
+ sprd_mcdt_ap_int_enable(mcdt, chan->id, true);
+ break;
+
+ default:
+ dev_err(mcdt->dev, "Unsupported channel type\n");
+ ret = -EINVAL;
+ }
+
+ if (!ret) {
+ chan->cb = cb;
+ chan->int_enable = true;
+ }
+
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(sprd_mcdt_chan_int_enable);
+
+/**
+ * sprd_mcdt_chan_int_disable - disable the interrupt mode for the MCDT channel
+ * @chan: the MCDT channel
+ */
+void sprd_mcdt_chan_int_disable(struct sprd_mcdt_chan *chan)
+{
+ struct sprd_mcdt_dev *mcdt = chan->mcdt;
+ unsigned long flags;
+
+ spin_lock_irqsave(&mcdt->lock, flags);
+
+ if (!chan->int_enable) {
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return;
+ }
+
+ switch (chan->type) {
+ case SPRD_MCDT_ADC_CHAN:
+ sprd_mcdt_chan_int_en(mcdt, chan->id,
+ MCDT_ADC_FIFO_AF_INT, false);
+ sprd_mcdt_chan_int_clear(mcdt, chan->id, MCDT_ADC_FIFO_AF_INT);
+ sprd_mcdt_ap_int_enable(mcdt, chan->id, false);
+ break;
+
+ case SPRD_MCDT_DAC_CHAN:
+ sprd_mcdt_chan_int_en(mcdt, chan->id,
+ MCDT_DAC_FIFO_AE_INT, false);
+ sprd_mcdt_chan_int_clear(mcdt, chan->id, MCDT_DAC_FIFO_AE_INT);
+ sprd_mcdt_ap_int_enable(mcdt, chan->id, false);
+ break;
+
+ default:
+ break;
+ }
+
+ chan->int_enable = false;
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+}
+EXPORT_SYMBOL_GPL(sprd_mcdt_chan_int_disable);
+
+/**
+ * sprd_mcdt_chan_dma_enable - enable the DMA mode for the MCDT channel
+ * @chan: the MCDT channel
+ * @dma_chan: specify which DMA channel will be used for this MCDT channel
+ * @water_mark: water mark to trigger a DMA request
+ *
+ * Enable the DMA mode for the MCDT channel, that means we can use DMA to
+ * transfer data to the channel fifo and do not need reading/writing data
+ * manually.
+ *
+ * Returns 0 on success, or an error code on failure.
+ */
+int sprd_mcdt_chan_dma_enable(struct sprd_mcdt_chan *chan,
+ enum sprd_mcdt_dma_chan dma_chan,
+ u32 water_mark)
+{
+ struct sprd_mcdt_dev *mcdt = chan->mcdt;
+ unsigned long flags;
+ int ret = 0;
+
+ spin_lock_irqsave(&mcdt->lock, flags);
+
+ if (chan->dma_enable || chan->int_enable ||
+ dma_chan > SPRD_MCDT_DMA_CH4) {
+ dev_err(mcdt->dev, "Failed to set DMA mode\n");
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return -EINVAL;
+ }
+
+ switch (chan->type) {
+ case SPRD_MCDT_ADC_CHAN:
+ sprd_mcdt_adc_fifo_clear(mcdt, chan->id);
+ sprd_mcdt_adc_set_watermark(mcdt, chan->id,
+ water_mark, MCDT_FIFO_LENGTH - 1);
+ sprd_mcdt_adc_dma_enable(mcdt, chan->id, true);
+ sprd_mcdt_adc_dma_chn_select(mcdt, chan->id, dma_chan);
+ sprd_mcdt_adc_dma_ack_select(mcdt, chan->id, dma_chan);
+ break;
+
+ case SPRD_MCDT_DAC_CHAN:
+ sprd_mcdt_dac_fifo_clear(mcdt, chan->id);
+ sprd_mcdt_dac_set_watermark(mcdt, chan->id,
+ MCDT_FIFO_LENGTH - 1, water_mark);
+ sprd_mcdt_dac_dma_enable(mcdt, chan->id, true);
+ sprd_mcdt_dac_dma_chn_select(mcdt, chan->id, dma_chan);
+ sprd_mcdt_dac_dma_ack_select(mcdt, chan->id, dma_chan);
+ break;
+
+ default:
+ dev_err(mcdt->dev, "Unsupported channel type\n");
+ ret = -EINVAL;
+ }
+
+ if (!ret)
+ chan->dma_enable = true;
+
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(sprd_mcdt_chan_dma_enable);
+
+/**
+ * sprd_mcdt_chan_dma_disable - disable the DMA mode for the MCDT channel
+ * @chan: the MCDT channel
+ */
+void sprd_mcdt_chan_dma_disable(struct sprd_mcdt_chan *chan)
+{
+ struct sprd_mcdt_dev *mcdt = chan->mcdt;
+ unsigned long flags;
+
+ spin_lock_irqsave(&mcdt->lock, flags);
+
+ if (!chan->dma_enable) {
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+ return;
+ }
+
+ switch (chan->type) {
+ case SPRD_MCDT_ADC_CHAN:
+ sprd_mcdt_adc_dma_enable(mcdt, chan->id, false);
+ sprd_mcdt_adc_fifo_clear(mcdt, chan->id);
+ break;
+
+ case SPRD_MCDT_DAC_CHAN:
+ sprd_mcdt_dac_dma_enable(mcdt, chan->id, false);
+ sprd_mcdt_dac_fifo_clear(mcdt, chan->id);
+ break;
+
+ default:
+ break;
+ }
+
+ chan->dma_enable = false;
+ spin_unlock_irqrestore(&mcdt->lock, flags);
+}
+EXPORT_SYMBOL_GPL(sprd_mcdt_chan_dma_disable);
+
+/**
+ * sprd_mcdt_request_chan - request one MCDT channel
+ * @channel: channel id
+ * @type: channel type, it can be one ADC channel or DAC channel
+ *
+ * Rreturn NULL if no available channel.
+ */
+struct sprd_mcdt_chan *sprd_mcdt_request_chan(u8 channel,
+ enum sprd_mcdt_channel_type type)
+{
+ struct sprd_mcdt_chan *temp, *chan = NULL;
+
+ mutex_lock(&sprd_mcdt_list_mutex);
+
+ list_for_each_entry(temp, &sprd_mcdt_chan_list, list) {
+ if (temp->type == type && temp->id == channel) {
+ chan = temp;
+ break;
+ }
+ }
+
+ if (chan)
+ list_del(&chan->list);
+
+ mutex_unlock(&sprd_mcdt_list_mutex);
+
+ return chan;
+}
+EXPORT_SYMBOL_GPL(sprd_mcdt_request_chan);
+
+/**
+ * sprd_mcdt_free_chan - free one MCDT channel
+ * @chan: the channel to be freed
+ */
+void sprd_mcdt_free_chan(struct sprd_mcdt_chan *chan)
+{
+ struct sprd_mcdt_chan *temp;
+
+ sprd_mcdt_chan_dma_disable(chan);
+ sprd_mcdt_chan_int_disable(chan);
+
+ mutex_lock(&sprd_mcdt_list_mutex);
+
+ list_for_each_entry(temp, &sprd_mcdt_chan_list, list) {
+ if (temp == chan) {
+ mutex_unlock(&sprd_mcdt_list_mutex);
+ return;
+ }
+ }
+
+ list_add_tail(&chan->list, &sprd_mcdt_chan_list);
+ mutex_unlock(&sprd_mcdt_list_mutex);
+}
+EXPORT_SYMBOL_GPL(sprd_mcdt_free_chan);
+
+static void sprd_mcdt_init_chans(struct sprd_mcdt_dev *mcdt,
+ struct resource *res)
+{
+ int i;
+
+ for (i = 0; i < MCDT_CHANNEL_NUM; i++) {
+ struct sprd_mcdt_chan *chan = &mcdt->chan[i];
+
+ if (i < MCDT_ADC_CHANNEL_NUM) {
+ chan->id = i;
+ chan->type = SPRD_MCDT_ADC_CHAN;
+ chan->fifo_phys = res->start + MCDT_CH0_RXD + i * 4;
+ } else {
+ chan->id = i - MCDT_ADC_CHANNEL_NUM;
+ chan->type = SPRD_MCDT_DAC_CHAN;
+ chan->fifo_phys = res->start + MCDT_CH0_TXD +
+ (i - MCDT_ADC_CHANNEL_NUM) * 4;
+ }
+
+ chan->mcdt = mcdt;
+ INIT_LIST_HEAD(&chan->list);
+
+ mutex_lock(&sprd_mcdt_list_mutex);
+ list_add_tail(&chan->list, &sprd_mcdt_chan_list);
+ mutex_unlock(&sprd_mcdt_list_mutex);
+ }
+}
+
+static int sprd_mcdt_probe(struct platform_device *pdev)
+{
+ struct sprd_mcdt_dev *mcdt;
+ struct resource *res;
+ int ret, irq;
+
+ mcdt = devm_kzalloc(&pdev->dev, sizeof(*mcdt), GFP_KERNEL);
+ if (!mcdt)
+ return -ENOMEM;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ mcdt->base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(mcdt->base))
+ return PTR_ERR(mcdt->base);
+
+ mcdt->dev = &pdev->dev;
+ spin_lock_init(&mcdt->lock);
+ platform_set_drvdata(pdev, mcdt);
+
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0) {
+ dev_err(&pdev->dev, "Failed to get MCDT interrupt\n");
+ return irq;
+ }
+
+ ret = devm_request_irq(&pdev->dev, irq, sprd_mcdt_irq_handler,
+ 0, "sprd-mcdt", mcdt);
+ if (ret) {
+ dev_err(&pdev->dev, "Failed to request MCDT IRQ\n");
+ return ret;
+ }
+
+ sprd_mcdt_init_chans(mcdt, res);
+
+ return 0;
+}
+
+static int sprd_mcdt_remove(struct platform_device *pdev)
+{
+ struct sprd_mcdt_chan *chan, *temp;
+
+ mutex_lock(&sprd_mcdt_list_mutex);
+
+ list_for_each_entry_safe(chan, temp, &sprd_mcdt_chan_list, list)
+ list_del(&chan->list);
+
+ mutex_unlock(&sprd_mcdt_list_mutex);
+
+ return 0;
+}
+
+static const struct of_device_id sprd_mcdt_of_match[] = {
+ { .compatible = "sprd,sc9860-mcdt", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, sprd_mcdt_of_match);
+
+static struct platform_driver sprd_mcdt_driver = {
+ .probe = sprd_mcdt_probe,
+ .remove = sprd_mcdt_remove,
+ .driver = {
+ .name = "sprd-mcdt",
+ .of_match_table = sprd_mcdt_of_match,
+ },
+};
+
+module_platform_driver(sprd_mcdt_driver);
+
+MODULE_DESCRIPTION("Spreadtrum Multi-Channel Data Transfer Driver");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+
+#ifndef __SPRD_MCDT_H
+#define __SPRD_MCDT_H
+
+enum sprd_mcdt_channel_type {
+ SPRD_MCDT_DAC_CHAN,
+ SPRD_MCDT_ADC_CHAN,
+ SPRD_MCDT_UNKNOWN_CHAN,
+};
+
+enum sprd_mcdt_dma_chan {
+ SPRD_MCDT_DMA_CH0,
+ SPRD_MCDT_DMA_CH1,
+ SPRD_MCDT_DMA_CH2,
+ SPRD_MCDT_DMA_CH3,
+ SPRD_MCDT_DMA_CH4,
+};
+
+struct sprd_mcdt_chan_callback {
+ void (*notify)(void *data);
+ void *data;
+};
+
+/**
+ * struct sprd_mcdt_chan - this struct represents a single channel instance
+ * @mcdt: the mcdt controller
+ * @id: channel id
+ * @fifo_phys: channel fifo physical address which is used for DMA transfer
+ * @type: channel type
+ * @cb: channel fifo interrupt's callback interface to notify the fifo events
+ * @dma_enable: indicate if use DMA mode to transfer data
+ * @int_enable: indicate if use interrupt mode to notify users to read or
+ * write data manually
+ * @list: used to link into the global list
+ *
+ * Note: users should not modify any members of this structure.
+ */
+struct sprd_mcdt_chan {
+ struct sprd_mcdt_dev *mcdt;
+ u8 id;
+ unsigned long fifo_phys;
+ enum sprd_mcdt_channel_type type;
+ enum sprd_mcdt_dma_chan dma_chan;
+ struct sprd_mcdt_chan_callback *cb;
+ bool dma_enable;
+ bool int_enable;
+ struct list_head list;
+};
+
+#ifdef CONFIG_SND_SOC_SPRD_MCDT
+struct sprd_mcdt_chan *sprd_mcdt_request_chan(u8 channel,
+ enum sprd_mcdt_channel_type type);
+void sprd_mcdt_free_chan(struct sprd_mcdt_chan *chan);
+
+int sprd_mcdt_chan_write(struct sprd_mcdt_chan *chan, char *tx_buf, u32 size);
+int sprd_mcdt_chan_read(struct sprd_mcdt_chan *chan, char *rx_buf, u32 size);
+int sprd_mcdt_chan_int_enable(struct sprd_mcdt_chan *chan, u32 water_mark,
+ struct sprd_mcdt_chan_callback *cb);
+void sprd_mcdt_chan_int_disable(struct sprd_mcdt_chan *chan);
+
+int sprd_mcdt_chan_dma_enable(struct sprd_mcdt_chan *chan,
+ enum sprd_mcdt_dma_chan dma_chan, u32 water_mark);
+void sprd_mcdt_chan_dma_disable(struct sprd_mcdt_chan *chan);
+
+#else
+
+struct sprd_mcdt_chan *sprd_mcdt_request_chan(u8 channel,
+ enum sprd_mcdt_channel_type type)
+{
+ return NULL;
+}
+
+void sprd_mcdt_free_chan(struct sprd_mcdt_chan *chan)
+{ }
+
+int sprd_mcdt_chan_write(struct sprd_mcdt_chan *chan, char *tx_buf, u32 size)
+{
+ return -EINVAL;
+}
+
+int sprd_mcdt_chan_read(struct sprd_mcdt_chan *chan, char *rx_buf, u32 size)
+{
+ return 0;
+}
+
+int sprd_mcdt_chan_int_enable(struct sprd_mcdt_chan *chan, u32 water_mark,
+ struct sprd_mcdt_chan_callback *cb)
+{
+ return -EINVAL;
+}
+
+void sprd_mcdt_chan_int_disable(struct sprd_mcdt_chan *chan)
+{ }
+
+int sprd_mcdt_chan_dma_enable(struct sprd_mcdt_chan *chan,
+ enum sprd_mcdt_dma_chan dma_chan, u32 water_mark)
+{
+ return -EINVAL;
+}
+
+void sprd_mcdt_chan_dma_disable(struct sprd_mcdt_chan *chan)
+{ }
+
+#endif
+
+#endif /* __SPRD_MCDT_H */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2019 Spreadtrum Communications Inc.
+
+#include <linux/dma-mapping.h>
+#include <linux/dmaengine.h>
+#include <linux/dma/sprd-dma.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <sound/pcm.h>
+#include <sound/pcm_params.h>
+#include <sound/soc.h>
+#include <sound/compress_driver.h>
+
+#include "sprd-pcm-dma.h"
+
+#define SPRD_COMPR_DMA_CHANS 2
+
+/* Default values if userspace does not set */
+#define SPRD_COMPR_MIN_FRAGMENT_SIZE SZ_8K
+#define SPRD_COMPR_MAX_FRAGMENT_SIZE SZ_128K
+#define SPRD_COMPR_MIN_NUM_FRAGMENTS 4
+#define SPRD_COMPR_MAX_NUM_FRAGMENTS 64
+
+/* DSP FIFO size */
+#define SPRD_COMPR_MCDT_EMPTY_WMK 0
+#define SPRD_COMPR_MCDT_FIFO_SIZE 512
+
+/* Stage 0 IRAM buffer size definition */
+#define SPRD_COMPR_IRAM_BUF_SIZE SZ_32K
+#define SPRD_COMPR_IRAM_INFO_SIZE (sizeof(struct sprd_compr_playinfo))
+#define SPRD_COMPR_IRAM_LINKLIST_SIZE (1024 - SPRD_COMPR_IRAM_INFO_SIZE)
+#define SPRD_COMPR_IRAM_SIZE (SPRD_COMPR_IRAM_BUF_SIZE + \
+ SPRD_COMPR_IRAM_INFO_SIZE + \
+ SPRD_COMPR_IRAM_LINKLIST_SIZE)
+
+/* Stage 1 DDR buffer size definition */
+#define SPRD_COMPR_AREA_BUF_SIZE SZ_2M
+#define SPRD_COMPR_AREA_LINKLIST_SIZE 1024
+#define SPRD_COMPR_AREA_SIZE (SPRD_COMPR_AREA_BUF_SIZE + \
+ SPRD_COMPR_AREA_LINKLIST_SIZE)
+
+struct sprd_compr_dma {
+ struct dma_chan *chan;
+ struct dma_async_tx_descriptor *desc;
+ dma_cookie_t cookie;
+ dma_addr_t phys;
+ void *virt;
+ int trans_len;
+};
+
+/*
+ * The Spreadtrum Audio compress offload mode will use 2-stage DMA transfer to
+ * save power. That means we can request 2 dma channels, one for source channel,
+ * and another one for destination channel. Once the source channel's transaction
+ * is done, it will trigger the destination channel's transaction automatically
+ * by hardware signal.
+ *
+ * For 2-stage DMA transfer, we can allocate 2 buffers: IRAM buffer (always
+ * power-on) and DDR buffer. The source channel will transfer data from IRAM
+ * buffer to the DSP fifo to decoding/encoding, once IRAM buffer is empty by
+ * transferring done, the destination channel will start to transfer data from
+ * DDR buffer to IRAM buffer.
+ *
+ * Since the DSP fifo is only 512B, IRAM buffer is allocated by 32K, and DDR
+ * buffer is larger to 2M. That means only the IRAM 32k data is transferred
+ * done, we can wake up the AP system to transfer data from DDR to IRAM, and
+ * other time the AP system can be suspended to save power.
+ */
+struct sprd_compr_stream {
+ struct snd_compr_stream *cstream;
+ struct sprd_compr_ops *compr_ops;
+ struct sprd_compr_dma dma[SPRD_COMPR_DMA_CHANS];
+
+ /* DMA engine channel number */
+ int num_channels;
+
+ /* Stage 0 IRAM buffer */
+ struct snd_dma_buffer iram_buffer;
+ /* Stage 1 DDR buffer */
+ struct snd_dma_buffer compr_buffer;
+
+ /* DSP play information IRAM buffer */
+ dma_addr_t info_phys;
+ void *info_area;
+ int info_size;
+
+ /* Data size copied to IRAM buffer */
+ int copied_total;
+ /* Total received data size from userspace */
+ int received_total;
+ /* Stage 0 IRAM buffer received data size */
+ int received_stage0;
+ /* Stage 1 DDR buffer received data size */
+ int received_stage1;
+ /* Stage 1 DDR buffer pointer */
+ int stage1_pointer;
+};
+
+static int sprd_platform_compr_trigger(struct snd_compr_stream *cstream,
+ int cmd);
+
+static void sprd_platform_compr_drain_notify(void *arg)
+{
+ struct snd_compr_stream *cstream = arg;
+ struct snd_compr_runtime *runtime = cstream->runtime;
+ struct sprd_compr_stream *stream = runtime->private_data;
+
+ memset(stream->info_area, 0, sizeof(struct sprd_compr_playinfo));
+
+ snd_compr_drain_notify(cstream);
+}
+
+static void sprd_platform_compr_dma_complete(void *data)
+{
+ struct snd_compr_stream *cstream = data;
+ struct snd_compr_runtime *runtime = cstream->runtime;
+ struct sprd_compr_stream *stream = runtime->private_data;
+ struct sprd_compr_dma *dma = &stream->dma[1];
+
+ /* Update data size copied to IRAM buffer */
+ stream->copied_total += dma->trans_len;
+ if (stream->copied_total > stream->received_total)
+ stream->copied_total = stream->received_total;
+
+ snd_compr_fragment_elapsed(cstream);
+}
+
+static int sprd_platform_compr_dma_config(struct snd_compr_stream *cstream,
+ struct snd_compr_params *params,
+ int channel)
+{
+ struct snd_compr_runtime *runtime = cstream->runtime;
+ struct sprd_compr_stream *stream = runtime->private_data;
+ struct snd_soc_pcm_runtime *rtd = cstream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct device *dev = component->dev;
+ struct sprd_compr_data *data = snd_soc_dai_get_drvdata(rtd->cpu_dai);
+ struct sprd_pcm_dma_params *dma_params = data->dma_params;
+ struct sprd_compr_dma *dma = &stream->dma[channel];
+ struct dma_slave_config config = { };
+ struct sprd_dma_linklist link = { };
+ enum dma_transfer_direction dir;
+ struct scatterlist *sg, *sgt;
+ enum dma_slave_buswidth bus_width;
+ int period, period_cnt, sg_num = 2;
+ dma_addr_t src_addr, dst_addr;
+ unsigned long flags;
+ int ret, j;
+
+ if (!dma_params) {
+ dev_err(dev, "no dma parameters setting\n");
+ return -EINVAL;
+ }
+
+ dma->chan = dma_request_slave_channel(dev,
+ dma_params->chan_name[channel]);
+ if (!dma->chan) {
+ dev_err(dev, "failed to request dma channel\n");
+ return -ENODEV;
+ }
+
+ sgt = sg = devm_kcalloc(dev, sg_num, sizeof(*sg), GFP_KERNEL);
+ if (!sg) {
+ ret = -ENOMEM;
+ goto sg_err;
+ }
+
+ switch (channel) {
+ case 0:
+ bus_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
+ period = (SPRD_COMPR_MCDT_FIFO_SIZE - SPRD_COMPR_MCDT_EMPTY_WMK) * 4;
+ period_cnt = params->buffer.fragment_size / period;
+ src_addr = stream->iram_buffer.addr;
+ dst_addr = dma_params->dev_phys[channel];
+ flags = SPRD_DMA_FLAGS(SPRD_DMA_SRC_CHN1,
+ SPRD_DMA_TRANS_DONE_TRG,
+ SPRD_DMA_FRAG_REQ,
+ SPRD_DMA_TRANS_INT);
+ break;
+
+ case 1:
+ bus_width = DMA_SLAVE_BUSWIDTH_2_BYTES;
+ period = params->buffer.fragment_size;
+ period_cnt = params->buffer.fragments;
+ src_addr = stream->compr_buffer.addr;
+ dst_addr = stream->iram_buffer.addr;
+ flags = SPRD_DMA_FLAGS(SPRD_DMA_DST_CHN1,
+ SPRD_DMA_TRANS_DONE_TRG,
+ SPRD_DMA_FRAG_REQ,
+ SPRD_DMA_TRANS_INT);
+ break;
+
+ default:
+ ret = -EINVAL;
+ goto config_err;
+ }
+
+ dma->trans_len = period * period_cnt;
+
+ config.src_maxburst = period;
+ config.src_addr_width = bus_width;
+ config.dst_addr_width = bus_width;
+ if (cstream->direction == SND_COMPRESS_PLAYBACK) {
+ config.src_addr = src_addr;
+ config.dst_addr = dst_addr;
+ dir = DMA_MEM_TO_DEV;
+ } else {
+ config.src_addr = dst_addr;
+ config.dst_addr = src_addr;
+ dir = DMA_DEV_TO_MEM;
+ }
+
+ sg_init_table(sgt, sg_num);
+ for (j = 0; j < sg_num; j++, sgt++) {
+ sg_dma_len(sgt) = dma->trans_len;
+ sg_dma_address(sgt) = dst_addr;
+ }
+
+ /*
+ * Configure the link-list address for the DMA engine link-list
+ * mode.
+ */
+ link.virt_addr = (unsigned long)dma->virt;
+ link.phy_addr = dma->phys;
+
+ ret = dmaengine_slave_config(dma->chan, &config);
+ if (ret) {
+ dev_err(dev,
+ "failed to set slave configuration: %d\n", ret);
+ goto config_err;
+ }
+
+ /*
+ * We configure the DMA request mode, interrupt mode, channel
+ * mode and channel trigger mode by the flags.
+ */
+ dma->desc = dma->chan->device->device_prep_slave_sg(dma->chan, sg,
+ sg_num, dir,
+ flags, &link);
+ if (!dma->desc) {
+ dev_err(dev, "failed to prepare slave sg\n");
+ ret = -ENOMEM;
+ goto config_err;
+ }
+
+ /* Only channel 1 transfer can wake up the AP system. */
+ if (!params->no_wake_mode && channel == 1) {
+ dma->desc->callback = sprd_platform_compr_dma_complete;
+ dma->desc->callback_param = cstream;
+ }
+
+ devm_kfree(dev, sg);
+
+ return 0;
+
+config_err:
+ devm_kfree(dev, sg);
+sg_err:
+ dma_release_channel(dma->chan);
+ return ret;
+}
+
+static int sprd_platform_compr_set_params(struct snd_compr_stream *cstream,
+ struct snd_compr_params *params)
+{
+ struct snd_compr_runtime *runtime = cstream->runtime;
+ struct sprd_compr_stream *stream = runtime->private_data;
+ struct snd_soc_pcm_runtime *rtd = cstream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct device *dev = component->dev;
+ struct sprd_compr_params compr_params = { };
+ int ret;
+
+ /*
+ * Configure the DMA engine 2-stage transfer mode. Channel 1 set as the
+ * destination channel, and channel 0 set as the source channel, that
+ * means once the source channel's transaction is done, it will trigger
+ * the destination channel's transaction automatically.
+ */
+ ret = sprd_platform_compr_dma_config(cstream, params, 1);
+ if (ret) {
+ dev_err(dev, "failed to config stage 1 DMA: %d\n", ret);
+ return ret;
+ }
+
+ ret = sprd_platform_compr_dma_config(cstream, params, 0);
+ if (ret) {
+ dev_err(dev, "failed to config stage 0 DMA: %d\n", ret);
+ goto config_err;
+ }
+
+ compr_params.direction = cstream->direction;
+ compr_params.sample_rate = params->codec.sample_rate;
+ compr_params.channels = stream->num_channels;
+ compr_params.info_phys = stream->info_phys;
+ compr_params.info_size = stream->info_size;
+ compr_params.rate = params->codec.bit_rate;
+ compr_params.format = params->codec.id;
+
+ ret = stream->compr_ops->set_params(cstream->direction, &compr_params);
+ if (ret) {
+ dev_err(dev, "failed to set parameters: %d\n", ret);
+ goto params_err;
+ }
+
+ return 0;
+
+params_err:
+ dma_release_channel(stream->dma[0].chan);
+config_err:
+ dma_release_channel(stream->dma[1].chan);
+ return ret;
+}
+
+static int sprd_platform_compr_open(struct snd_compr_stream *cstream)
+{
+ struct snd_compr_runtime *runtime = cstream->runtime;
+ struct snd_soc_pcm_runtime *rtd = cstream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct device *dev = component->dev;
+ struct sprd_compr_data *data = snd_soc_dai_get_drvdata(rtd->cpu_dai);
+ struct sprd_compr_stream *stream;
+ struct sprd_compr_callback cb;
+ int stream_id = cstream->direction, ret;
+
+ ret = dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(32));
+ if (ret)
+ return ret;
+
+ stream = devm_kzalloc(dev, sizeof(*stream), GFP_KERNEL);
+ if (!stream)
+ return -ENOMEM;
+
+ stream->cstream = cstream;
+ stream->num_channels = 2;
+ stream->compr_ops = data->ops;
+
+ /*
+ * Allocate the stage 0 IRAM buffer size, including the DMA 0
+ * link-list size and play information of DSP address size.
+ */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV_IRAM, dev,
+ SPRD_COMPR_IRAM_SIZE, &stream->iram_buffer);
+ if (ret < 0)
+ goto err_iram;
+
+ /* Use to save link-list configuration for DMA 0. */
+ stream->dma[0].virt = stream->iram_buffer.area + SPRD_COMPR_IRAM_SIZE;
+ stream->dma[0].phys = stream->iram_buffer.addr + SPRD_COMPR_IRAM_SIZE;
+
+ /* Use to update the current data offset of DSP. */
+ stream->info_phys = stream->iram_buffer.addr + SPRD_COMPR_IRAM_SIZE +
+ SPRD_COMPR_IRAM_LINKLIST_SIZE;
+ stream->info_area = stream->iram_buffer.area + SPRD_COMPR_IRAM_SIZE +
+ SPRD_COMPR_IRAM_LINKLIST_SIZE;
+ stream->info_size = SPRD_COMPR_IRAM_INFO_SIZE;
+
+ /*
+ * Allocate the stage 1 DDR buffer size, including the DMA 1 link-list
+ * size.
+ */
+ ret = snd_dma_alloc_pages(SNDRV_DMA_TYPE_DEV, dev,
+ SPRD_COMPR_AREA_SIZE, &stream->compr_buffer);
+ if (ret < 0)
+ goto err_compr;
+
+ /* Use to save link-list configuration for DMA 1. */
+ stream->dma[1].virt = stream->compr_buffer.area + SPRD_COMPR_AREA_SIZE;
+ stream->dma[1].phys = stream->compr_buffer.addr + SPRD_COMPR_AREA_SIZE;
+
+ cb.drain_notify = sprd_platform_compr_drain_notify;
+ cb.drain_data = cstream;
+ ret = stream->compr_ops->open(stream_id, &cb);
+ if (ret) {
+ dev_err(dev, "failed to open compress platform: %d\n", ret);
+ goto err_open;
+ }
+
+ runtime->private_data = stream;
+ return 0;
+
+err_open:
+ snd_dma_free_pages(&stream->compr_buffer);
+err_compr:
+ snd_dma_free_pages(&stream->iram_buffer);
+err_iram:
+ devm_kfree(dev, stream);
+
+ return ret;
+}
+
+static int sprd_platform_compr_free(struct snd_compr_stream *cstream)
+{
+ struct snd_compr_runtime *runtime = cstream->runtime;
+ struct sprd_compr_stream *stream = runtime->private_data;
+ struct snd_soc_pcm_runtime *rtd = cstream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct device *dev = component->dev;
+ int stream_id = cstream->direction, i;
+
+ for (i = 0; i < stream->num_channels; i++) {
+ struct sprd_compr_dma *dma = &stream->dma[i];
+
+ if (dma->chan) {
+ dma_release_channel(dma->chan);
+ dma->chan = NULL;
+ }
+ }
+
+ snd_dma_free_pages(&stream->compr_buffer);
+ snd_dma_free_pages(&stream->iram_buffer);
+
+ stream->compr_ops->close(stream_id);
+
+ devm_kfree(dev, stream);
+ return 0;
+}
+
+static int sprd_platform_compr_trigger(struct snd_compr_stream *cstream,
+ int cmd)
+{
+ struct snd_compr_runtime *runtime = cstream->runtime;
+ struct sprd_compr_stream *stream = runtime->private_data;
+ struct snd_soc_pcm_runtime *rtd = cstream->private_data;
+ struct snd_soc_component *component =
+ snd_soc_rtdcom_lookup(rtd, DRV_NAME);
+ struct device *dev = component->dev;
+ int channels = stream->num_channels, ret = 0, i;
+ int stream_id = cstream->direction;
+
+ if (cstream->direction != SND_COMPRESS_PLAYBACK) {
+ dev_err(dev, "unsupported compress direction\n");
+ return -EINVAL;
+ }
+
+ switch (cmd) {
+ case SNDRV_PCM_TRIGGER_START:
+ for (i = channels - 1; i >= 0; i--) {
+ struct sprd_compr_dma *dma = &stream->dma[i];
+
+ if (!dma->desc)
+ continue;
+
+ dma->cookie = dmaengine_submit(dma->desc);
+ ret = dma_submit_error(dma->cookie);
+ if (ret) {
+ dev_err(dev, "failed to submit request: %d\n",
+ ret);
+ return ret;
+ }
+ }
+
+ for (i = channels - 1; i >= 0; i--) {
+ struct sprd_compr_dma *dma = &stream->dma[i];
+
+ if (dma->chan)
+ dma_async_issue_pending(dma->chan);
+ }
+
+ ret = stream->compr_ops->start(stream_id);
+ break;
+
+ case SNDRV_PCM_TRIGGER_STOP:
+ for (i = channels - 1; i >= 0; i--) {
+ struct sprd_compr_dma *dma = &stream->dma[i];
+
+ if (dma->chan)
+ dmaengine_terminate_async(dma->chan);
+ }
+
+ stream->copied_total = 0;
+ stream->stage1_pointer = 0;
+ stream->received_total = 0;
+ stream->received_stage0 = 0;
+ stream->received_stage1 = 0;
+
+ ret = stream->compr_ops->stop(stream_id);
+ break;
+
+ case SNDRV_PCM_TRIGGER_SUSPEND:
+ case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
+ for (i = channels - 1; i >= 0; i--) {
+ struct sprd_compr_dma *dma = &stream->dma[i];
+
+ if (dma->chan)
+ dmaengine_pause(dma->chan);
+ }
+
+ ret = stream->compr_ops->pause(stream_id);
+ break;
+
+ case SNDRV_PCM_TRIGGER_RESUME:
+ case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
+ for (i = channels - 1; i >= 0; i--) {
+ struct sprd_compr_dma *dma = &stream->dma[i];
+
+ if (dma->chan)
+ dmaengine_resume(dma->chan);
+ }
+
+ ret = stream->compr_ops->pause_release(stream_id);
+ break;
+
+ case SND_COMPR_TRIGGER_PARTIAL_DRAIN:
+ case SND_COMPR_TRIGGER_DRAIN:
+ ret = stream->compr_ops->drain(stream->received_total);
+ break;
+
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static int sprd_platform_compr_pointer(struct snd_compr_stream *cstream,
+ struct snd_compr_tstamp *tstamp)
+{
+ struct snd_compr_runtime *runtime = cstream->runtime;
+ struct sprd_compr_stream *stream = runtime->private_data;
+ struct sprd_compr_playinfo *info =
+ (struct sprd_compr_playinfo *)stream->info_area;
+
+ tstamp->copied_total = stream->copied_total;
+ tstamp->pcm_io_frames = info->current_data_offset;
+
+ return 0;
+}
+
+static int sprd_platform_compr_copy(struct snd_compr_stream *cstream,
+ char __user *buf, size_t count)
+{
+ struct snd_compr_runtime *runtime = cstream->runtime;
+ struct sprd_compr_stream *stream = runtime->private_data;
+ int avail_bytes, data_count = count;
+ void *dst;
+
+ /*
+ * We usually set fragment size as 32K, and the stage 0 IRAM buffer
+ * size is 32K too. So if now the received data size of the stage 0
+ * IRAM buffer is less than 32K, that means we have some available
+ * spaces for the stage 0 IRAM buffer.
+ */
+ if (stream->received_stage0 < runtime->fragment_size) {
+ avail_bytes = runtime->fragment_size - stream->received_stage0;
+ dst = stream->iram_buffer.area + stream->received_stage0;
+
+ if (avail_bytes >= data_count) {
+ /*
+ * Copy data to the stage 0 IRAM buffer directly if
+ * spaces are enough.
+ */
+ if (copy_from_user(dst, buf, data_count))
+ return -EFAULT;
+
+ stream->received_stage0 += data_count;
+ stream->copied_total += data_count;
+ goto copy_done;
+ } else {
+ /*
+ * If the data count is larger than the available spaces
+ * of the the stage 0 IRAM buffer, we should copy one
+ * partial data to the stage 0 IRAM buffer, and copy
+ * the left to the stage 1 DDR buffer.
+ */
+ if (copy_from_user(dst, buf, avail_bytes))
+ return -EFAULT;
+
+ data_count -= avail_bytes;
+ stream->received_stage0 += avail_bytes;
+ stream->copied_total += avail_bytes;
+ buf += avail_bytes;
+ }
+ }
+
+ /*
+ * Copy data to the stage 1 DDR buffer if no spaces for the stage 0 IRAM
+ * buffer.
+ */
+ dst = stream->compr_buffer.area + stream->stage1_pointer;
+ if (data_count < stream->compr_buffer.bytes - stream->stage1_pointer) {
+ if (copy_from_user(dst, buf, data_count))
+ return -EFAULT;
+
+ stream->stage1_pointer += data_count;
+ } else {
+ avail_bytes = stream->compr_buffer.bytes - stream->stage1_pointer;
+
+ if (copy_from_user(dst, buf, avail_bytes))
+ return -EFAULT;
+
+ if (copy_from_user(stream->compr_buffer.area, buf + avail_bytes,
+ data_count - avail_bytes))
+ return -EFAULT;
+
+ stream->stage1_pointer = data_count - avail_bytes;
+ }
+
+ stream->received_stage1 += data_count;
+
+copy_done:
+ /* Update the copied data size. */
+ stream->received_total += count;
+ return count;
+}
+
+static int sprd_platform_compr_get_caps(struct snd_compr_stream *cstream,
+ struct snd_compr_caps *caps)
+{
+ caps->direction = cstream->direction;
+ caps->min_fragment_size = SPRD_COMPR_MIN_FRAGMENT_SIZE;
+ caps->max_fragment_size = SPRD_COMPR_MAX_FRAGMENT_SIZE;
+ caps->min_fragments = SPRD_COMPR_MIN_NUM_FRAGMENTS;
+ caps->max_fragments = SPRD_COMPR_MAX_NUM_FRAGMENTS;
+ caps->num_codecs = 2;
+ caps->codecs[0] = SND_AUDIOCODEC_MP3;
+ caps->codecs[1] = SND_AUDIOCODEC_AAC;
+
+ return 0;
+}
+
+static int
+sprd_platform_compr_get_codec_caps(struct snd_compr_stream *cstream,
+ struct snd_compr_codec_caps *codec)
+{
+ switch (codec->codec) {
+ case SND_AUDIOCODEC_MP3:
+ codec->num_descriptors = 2;
+ codec->descriptor[0].max_ch = 2;
+ codec->descriptor[0].bit_rate[0] = 320;
+ codec->descriptor[0].bit_rate[1] = 128;
+ codec->descriptor[0].num_bitrates = 2;
+ codec->descriptor[0].profiles = 0;
+ codec->descriptor[0].modes = SND_AUDIOCHANMODE_MP3_STEREO;
+ codec->descriptor[0].formats = 0;
+ break;
+
+ case SND_AUDIOCODEC_AAC:
+ codec->num_descriptors = 2;
+ codec->descriptor[1].max_ch = 2;
+ codec->descriptor[1].bit_rate[0] = 320;
+ codec->descriptor[1].bit_rate[1] = 128;
+ codec->descriptor[1].num_bitrates = 2;
+ codec->descriptor[1].profiles = 0;
+ codec->descriptor[1].modes = 0;
+ codec->descriptor[1].formats = 0;
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+const struct snd_compr_ops sprd_platform_compr_ops = {
+ .open = sprd_platform_compr_open,
+ .free = sprd_platform_compr_free,
+ .set_params = sprd_platform_compr_set_params,
+ .trigger = sprd_platform_compr_trigger,
+ .pointer = sprd_platform_compr_pointer,
+ .copy = sprd_platform_compr_copy,
+ .get_caps = sprd_platform_compr_get_caps,
+ .get_codec_caps = sprd_platform_compr_get_codec_caps,
+};
+
+MODULE_DESCRIPTION("Spreadtrum ASoC Compress Platform Driver");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:compress-platform");
#include <linux/dma/sprd-dma.h>
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/of_reserved_mem.h>
#include <linux/platform_device.h>
#include <sound/pcm.h>
#include <sound/pcm_params.h>
#include "sprd-pcm-dma.h"
-#define DRV_NAME "sprd_pcm_dma"
#define SPRD_PCM_DMA_LINKLIST_SIZE 64
#define SPRD_PCM_DMA_BRUST_LEN 640
static const struct snd_soc_component_driver sprd_soc_component = {
.name = DRV_NAME,
.ops = &sprd_pcm_ops,
+ .compr_ops = &sprd_platform_compr_ops,
.pcm_new = sprd_pcm_new,
.pcm_free = sprd_pcm_free,
};
static int sprd_soc_platform_probe(struct platform_device *pdev)
{
+ struct device_node *np = pdev->dev.of_node;
int ret;
+ ret = of_reserved_mem_device_init_by_idx(&pdev->dev, np, 0);
+ if (ret)
+ dev_warn(&pdev->dev,
+ "no reserved DMA memory for audio platform device\n");
+
ret = devm_snd_soc_register_component(&pdev->dev, &sprd_soc_component,
NULL, 0);
if (ret)
#ifndef __SPRD_PCM_DMA_H
#define __SPRD_PCM_DMA_H
+#define DRV_NAME "sprd_pcm_dma"
#define SPRD_PCM_CHANNEL_MAX 2
+extern const struct snd_compr_ops sprd_platform_compr_ops;
+
struct sprd_pcm_dma_params {
dma_addr_t dev_phys[SPRD_PCM_CHANNEL_MAX];
u32 datawidth[SPRD_PCM_CHANNEL_MAX];
const char *chan_name[SPRD_PCM_CHANNEL_MAX];
};
+struct sprd_compr_playinfo {
+ int total_time;
+ int current_time;
+ int total_data_length;
+ int current_data_offset;
+};
+
+struct sprd_compr_params {
+ u32 direction;
+ u32 rate;
+ u32 sample_rate;
+ u32 channels;
+ u32 format;
+ u32 period;
+ u32 periods;
+ u32 info_phys;
+ u32 info_size;
+};
+
+struct sprd_compr_callback {
+ void (*drain_notify)(void *data);
+ void *drain_data;
+};
+
+struct sprd_compr_ops {
+ int (*open)(int str_id, struct sprd_compr_callback *cb);
+ int (*close)(int str_id);
+ int (*start)(int str_id);
+ int (*stop)(int str_id);
+ int (*pause)(int str_id);
+ int (*pause_release)(int str_id);
+ int (*drain)(int received_total);
+ int (*set_params)(int str_id, struct sprd_compr_params *params);
+};
+
+struct sprd_compr_data {
+ struct sprd_compr_ops *ops;
+ struct sprd_pcm_dma_params *dma_params;
+};
+
#endif /* __SPRD_PCM_DMA_H */
static const struct snd_pcm_hardware stm32_adfsdm_pcm_hw = {
.info = SNDRV_PCM_INFO_INTERLEAVED | SNDRV_PCM_INFO_BLOCK_TRANSFER |
- SNDRV_PCM_INFO_PAUSE,
+ SNDRV_PCM_INFO_MMAP | SNDRV_PCM_INFO_PAUSE,
.formats = SNDRV_PCM_FMTBIT_S32_LE,
.rate_min = 8000,
I2S_I2SMOD_DATLEN_32,
};
-#define STM32_I2S_DAI_NAME_SIZE 20
#define STM32_I2S_FIFO_SIZE 16
#define STM32_I2S_IS_MASTER(x) ((x)->ms_flg == I2S_MS_MASTER)
* @phys_addr: I2S registers physical base address
* @lock_fd: lock to manage race conditions in full duplex mode
* @irq_lock: prevent race condition with IRQ
- * @dais_name: DAI name
* @mclk_rate: master clock frequency (Hz)
* @fmt: DAI protocol
* @refcount: keep count of opened streams on I2S
dma_addr_t phys_addr;
spinlock_t lock_fd; /* Manage race conditions for full duplex */
spinlock_t irq_lock; /* used to prevent race condition with IRQ */
- char dais_name[STM32_I2S_DAI_NAME_SIZE];
unsigned int mclk_rate;
unsigned int fmt;
int refcount;
unsigned int fthlv;
int ret;
- if ((params_channels(params) == 1) &&
- ((i2s->fmt & SND_SOC_DAIFMT_FORMAT_MASK) != SND_SOC_DAIFMT_DSP_A)) {
- dev_err(cpu_dai->dev, "Mono mode supported only by DSP_A\n");
- return -EINVAL;
- }
-
switch (format) {
case 16:
cfgr = I2S_CGFR_DATLEN_SET(I2S_I2SMOD_DATLEN_16);
i2s->substream = substream;
spin_unlock_irqrestore(&i2s->irq_lock, flags);
+ if ((i2s->fmt & SND_SOC_DAIFMT_FORMAT_MASK) != SND_SOC_DAIFMT_DSP_A)
+ snd_pcm_hw_constraint_single(substream->runtime,
+ SNDRV_PCM_HW_PARAM_CHANNELS, 2);
+
ret = clk_prepare_enable(i2s->i2sclk);
if (ret < 0) {
dev_err(cpu_dai->dev, "Failed to enable clock: %d\n", ret);
case SNDRV_PCM_TRIGGER_RESUME:
case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
/* Enable i2s */
- dev_dbg(cpu_dai->dev, "start I2S\n");
+ dev_dbg(cpu_dai->dev, "start I2S %s\n",
+ playback_flg ? "playback" : "capture");
cfg1_mask = I2S_CFG1_RXDMAEN | I2S_CFG1_TXDMAEN;
regmap_update_bits(i2s->regmap, STM32_I2S_CFG1_REG,
case SNDRV_PCM_TRIGGER_STOP:
case SNDRV_PCM_TRIGGER_SUSPEND:
case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
+ dev_dbg(cpu_dai->dev, "stop I2S %s\n",
+ playback_flg ? "playback" : "capture");
+
if (playback_flg)
regmap_update_bits(i2s->regmap, STM32_I2S_IER_REG,
I2S_IER_UDRIE,
break;
}
- dev_dbg(cpu_dai->dev, "stop I2S\n");
-
ret = regmap_update_bits(i2s->regmap, STM32_I2S_CR1_REG,
I2S_CR1_SPE, 0);
if (ret < 0) {
static const struct snd_pcm_hardware stm32_i2s_pcm_hw = {
.info = SNDRV_PCM_INFO_INTERLEAVED | SNDRV_PCM_INFO_MMAP,
.buffer_bytes_max = 8 * PAGE_SIZE,
- .period_bytes_max = 2048,
+ .period_bytes_min = 1024,
+ .period_bytes_max = 4 * PAGE_SIZE,
.periods_min = 2,
.periods_max = 8,
};
if (!dai_ptr)
return -ENOMEM;
- snprintf(i2s->dais_name, STM32_I2S_DAI_NAME_SIZE,
- "%s", dev_name(&pdev->dev));
-
dai_ptr->probe = stm32_i2s_dai_probe;
dai_ptr->ops = &stm32_i2s_pcm_dai_ops;
- dai_ptr->name = i2s->dais_name;
dai_ptr->id = 1;
stm32_i2s_dai_init(&dai_ptr->playback, "playback");
stm32_i2s_dai_init(&dai_ptr->capture, "capture");
/* Get irqs */
irq = platform_get_irq(pdev, 0);
if (irq < 0) {
- dev_err(&pdev->dev, "no irq for node %s\n", pdev->name);
- return -ENOENT;
+ if (irq != -EPROBE_DEFER)
+ dev_err(&pdev->dev, "no irq for node %s\n", pdev->name);
+ return irq;
}
ret = devm_request_irq(&pdev->dev, irq, stm32_i2s_isr, IRQF_ONESHOT,
#include <linux/delay.h>
#include <linux/module.h>
#include <linux/of_platform.h>
+#include <linux/pinctrl/consumer.h>
#include <linux/reset.h>
#include <sound/dmaengine_pcm.h>
{}
};
-static int stm32_sai_sync_conf_client(struct stm32_sai_data *sai, int synci)
+static int stm32_sai_pclk_disable(struct device *dev)
+{
+ struct stm32_sai_data *sai = dev_get_drvdata(dev);
+
+ clk_disable_unprepare(sai->pclk);
+
+ return 0;
+}
+
+static int stm32_sai_pclk_enable(struct device *dev)
{
+ struct stm32_sai_data *sai = dev_get_drvdata(dev);
int ret;
- /* Enable peripheral clock to allow GCR register access */
ret = clk_prepare_enable(sai->pclk);
if (ret) {
dev_err(&sai->pdev->dev, "failed to enable clock: %d\n", ret);
return ret;
}
+ return 0;
+}
+
+static int stm32_sai_sync_conf_client(struct stm32_sai_data *sai, int synci)
+{
+ int ret;
+
+ /* Enable peripheral clock to allow GCR register access */
+ ret = stm32_sai_pclk_enable(&sai->pdev->dev);
+ if (ret)
+ return ret;
+
writel_relaxed(FIELD_PREP(SAI_GCR_SYNCIN_MASK, (synci - 1)), sai->base);
- clk_disable_unprepare(sai->pclk);
+ stm32_sai_pclk_disable(&sai->pdev->dev);
return 0;
}
int ret;
/* Enable peripheral clock to allow GCR register access */
- ret = clk_prepare_enable(sai->pclk);
- if (ret) {
- dev_err(&sai->pdev->dev, "failed to enable clock: %d\n", ret);
+ ret = stm32_sai_pclk_enable(&sai->pdev->dev);
+ if (ret)
return ret;
- }
dev_dbg(&sai->pdev->dev, "Set %pOFn%s as synchro provider\n",
sai->pdev->dev.of_node,
dev_err(&sai->pdev->dev, "%pOFn%s already set as sync provider\n",
sai->pdev->dev.of_node,
prev_synco == STM_SAI_SYNC_OUT_A ? "A" : "B");
- clk_disable_unprepare(sai->pclk);
+ stm32_sai_pclk_disable(&sai->pdev->dev);
return -EINVAL;
}
writel_relaxed(FIELD_PREP(SAI_GCR_SYNCOUT_MASK, synco), sai->base);
- clk_disable_unprepare(sai->pclk);
+ stm32_sai_pclk_disable(&sai->pdev->dev);
return 0;
}
return devm_of_platform_populate(&pdev->dev);
}
+#ifdef CONFIG_PM_SLEEP
+/*
+ * When pins are shared by two sai sub instances, pins have to be defined
+ * in sai parent node. In this case, pins state is not managed by alsa fw.
+ * These pins are managed in suspend/resume callbacks.
+ */
+static int stm32_sai_suspend(struct device *dev)
+{
+ struct stm32_sai_data *sai = dev_get_drvdata(dev);
+ int ret;
+
+ ret = stm32_sai_pclk_enable(dev);
+ if (ret)
+ return ret;
+
+ sai->gcr = readl_relaxed(sai->base);
+ stm32_sai_pclk_disable(dev);
+
+ return pinctrl_pm_select_sleep_state(dev);
+}
+
+static int stm32_sai_resume(struct device *dev)
+{
+ struct stm32_sai_data *sai = dev_get_drvdata(dev);
+ int ret;
+
+ ret = stm32_sai_pclk_enable(dev);
+ if (ret)
+ return ret;
+
+ writel_relaxed(sai->gcr, sai->base);
+ stm32_sai_pclk_disable(dev);
+
+ return pinctrl_pm_select_default_state(dev);
+}
+#endif /* CONFIG_PM_SLEEP */
+
+static const struct dev_pm_ops stm32_sai_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(stm32_sai_suspend, stm32_sai_resume)
+};
+
MODULE_DEVICE_TABLE(of, stm32_sai_ids);
static struct platform_driver stm32_sai_driver = {
.driver = {
.name = "st,stm32-sai",
.of_match_table = stm32_sai_ids,
+ .pm = &stm32_sai_pm_ops,
},
.probe = stm32_sai_probe,
};
* @version: SOC version
* @irq: SAI interrupt line
* @set_sync: pointer to synchro mode configuration callback
+ * @gcr: SAI Global Configuration Register
*/
struct stm32_sai_data {
struct platform_device *pdev;
int irq;
int (*set_sync)(struct stm32_sai_data *sai,
struct device_node *np_provider, int synco, int synci);
+ u32 gcr;
};
struct regmap *regmap;
const struct regmap_config *regmap_config;
struct snd_dmaengine_dai_dma_data dma_params;
- struct snd_soc_dai_driver *cpu_dai_drv;
+ struct snd_soc_dai_driver cpu_dai_drv;
struct snd_soc_dai *cpu_dai;
struct snd_pcm_substream *substream;
struct stm32_sai_data *pdata;
{
switch (reg) {
case STM_SAI_DR_REGX:
+ case STM_SAI_SR_REGX:
return true;
default:
return false;
case STM_SAI_FRCR_REGX:
case STM_SAI_SLOTR_REGX:
case STM_SAI_IMR_REGX:
- case STM_SAI_SR_REGX:
case STM_SAI_CLRFR_REGX:
case STM_SAI_DR_REGX:
case STM_SAI_PDMCR_REGX:
.volatile_reg = stm32_sai_sub_volatile_reg,
.writeable_reg = stm32_sai_sub_writeable_reg,
.fast_io = true,
+ .cache_type = REGCACHE_FLAT,
};
static const struct regmap_config stm32_sai_sub_regmap_config_h7 = {
.volatile_reg = stm32_sai_sub_volatile_reg,
.writeable_reg = stm32_sai_sub_writeable_reg,
.fast_io = true,
+ .cache_type = REGCACHE_FLAT,
};
static int snd_pcm_iec958_info(struct snd_kcontrol *kcontrol,
if (!flags)
return IRQ_NONE;
- regmap_update_bits(sai->regmap, STM_SAI_CLRFR_REGX, SAI_XCLRFR_MASK,
- SAI_XCLRFR_MASK);
+ regmap_write_bits(sai->regmap, STM_SAI_CLRFR_REGX, SAI_XCLRFR_MASK,
+ SAI_XCLRFR_MASK);
if (!sai->substream) {
dev_err(&pdev->dev, "Device stopped. Spurious IRQ 0x%x\n", sr);
}
/* Enable ITs */
-
- regmap_update_bits(sai->regmap, STM_SAI_CLRFR_REGX,
- SAI_XCLRFR_MASK, SAI_XCLRFR_MASK);
+ regmap_write_bits(sai->regmap, STM_SAI_CLRFR_REGX,
+ SAI_XCLRFR_MASK, SAI_XCLRFR_MASK);
imr = SAI_XIMR_OVRUDRIE;
if (STM_SAI_IS_CAPTURE(sai)) {
* SAI fifo threshold is set to half fifo, to keep enough space
* for DMA incoming bursts.
*/
- regmap_update_bits(sai->regmap, STM_SAI_CR2_REGX,
- SAI_XCR2_FFLUSH | SAI_XCR2_FTH_MASK,
- SAI_XCR2_FFLUSH |
- SAI_XCR2_FTH_SET(STM_SAI_FIFO_TH_HALF));
+ regmap_write_bits(sai->regmap, STM_SAI_CR2_REGX,
+ SAI_XCR2_FFLUSH | SAI_XCR2_FTH_MASK,
+ SAI_XCR2_FFLUSH |
+ SAI_XCR2_FTH_SET(STM_SAI_FIFO_TH_HALF));
/* DS bits in CR1 not set for SPDIF (size forced to 24 bits).*/
if (STM_SAI_PROTOCOL_IS_SPDIF(sai)) {
.periods_max = 8,
};
-static struct snd_soc_dai_driver stm32_sai_playback_dai[] = {
-{
+static struct snd_soc_dai_driver stm32_sai_playback_dai = {
.probe = stm32_sai_dai_probe,
.pcm_new = stm32_sai_pcm_new,
.id = 1, /* avoid call to fmt_single_name() */
SNDRV_PCM_FMTBIT_S32_LE,
},
.ops = &stm32_sai_pcm_dai_ops,
- }
};
-static struct snd_soc_dai_driver stm32_sai_capture_dai[] = {
-{
+static struct snd_soc_dai_driver stm32_sai_capture_dai = {
.probe = stm32_sai_dai_probe,
.id = 1, /* avoid call to fmt_single_name() */
.capture = {
SNDRV_PCM_FMTBIT_S32_LE,
},
.ops = &stm32_sai_pcm_dai_ops,
- }
};
static const struct snd_dmaengine_pcm_config stm32_sai_pcm_config = {
return 0;
}
-static int stm32_sai_sub_dais_init(struct platform_device *pdev,
- struct stm32_sai_sub_data *sai)
-{
- sai->cpu_dai_drv = devm_kzalloc(&pdev->dev,
- sizeof(struct snd_soc_dai_driver),
- GFP_KERNEL);
- if (!sai->cpu_dai_drv)
- return -ENOMEM;
-
- if (STM_SAI_IS_PLAYBACK(sai)) {
- memcpy(sai->cpu_dai_drv, &stm32_sai_playback_dai,
- sizeof(stm32_sai_playback_dai));
- sai->cpu_dai_drv->playback.stream_name = sai->cpu_dai_drv->name;
- } else {
- memcpy(sai->cpu_dai_drv, &stm32_sai_capture_dai,
- sizeof(stm32_sai_capture_dai));
- sai->cpu_dai_drv->capture.stream_name = sai->cpu_dai_drv->name;
- }
- sai->cpu_dai_drv->name = dev_name(&pdev->dev);
-
- return 0;
-}
-
static int stm32_sai_sub_probe(struct platform_device *pdev)
{
struct stm32_sai_sub_data *sai;
if (ret)
return ret;
- ret = stm32_sai_sub_dais_init(pdev, sai);
- if (ret)
- return ret;
+ if (STM_SAI_IS_PLAYBACK(sai))
+ sai->cpu_dai_drv = stm32_sai_playback_dai;
+ else
+ sai->cpu_dai_drv = stm32_sai_capture_dai;
+ sai->cpu_dai_drv.name = dev_name(&pdev->dev);
ret = devm_request_irq(&pdev->dev, sai->pdata->irq, stm32_sai_isr,
IRQF_SHARED, dev_name(&pdev->dev), sai);
}
ret = devm_snd_soc_register_component(&pdev->dev, &stm32_component,
- sai->cpu_dai_drv, 1);
+ &sai->cpu_dai_drv, 1);
if (ret)
return ret;
return 0;
}
+#ifdef CONFIG_PM_SLEEP
+static int stm32_sai_sub_suspend(struct device *dev)
+{
+ struct stm32_sai_sub_data *sai = dev_get_drvdata(dev);
+
+ regcache_cache_only(sai->regmap, true);
+ regcache_mark_dirty(sai->regmap);
+ return 0;
+}
+
+static int stm32_sai_sub_resume(struct device *dev)
+{
+ struct stm32_sai_sub_data *sai = dev_get_drvdata(dev);
+
+ regcache_cache_only(sai->regmap, false);
+ return regcache_sync(sai->regmap);
+}
+#endif /* CONFIG_PM_SLEEP */
+
+static const struct dev_pm_ops stm32_sai_sub_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(stm32_sai_sub_suspend, stm32_sai_sub_resume)
+};
+
static struct platform_driver stm32_sai_sub_driver = {
.driver = {
.name = "st,stm32-sai-sub",
.of_match_table = stm32_sai_sub_ids,
+ .pm = &stm32_sai_sub_pm_ops,
},
.probe = stm32_sai_sub_probe,
};
#include <linux/delay.h>
#include <linux/module.h>
#include <linux/of_platform.h>
+#include <linux/pinctrl/consumer.h>
#include <linux/regmap.h>
#include <linux/reset.h>
memset(spdifrx->cs, 0, SPDIFRX_CS_BYTES_NB);
memset(spdifrx->ub, 0, SPDIFRX_UB_BYTES_NB);
+ pinctrl_pm_select_default_state(&spdifrx->pdev->dev);
+
ret = stm32_spdifrx_dma_ctrl_start(spdifrx);
if (ret < 0)
return ret;
if (wait_for_completion_interruptible_timeout(&spdifrx->cs_completion,
msecs_to_jiffies(100))
<= 0) {
- dev_err(&spdifrx->pdev->dev, "Failed to get control data\n");
+ dev_dbg(&spdifrx->pdev->dev, "Failed to get control data\n");
ret = -EAGAIN;
}
end:
clk_disable_unprepare(spdifrx->kclk);
+ pinctrl_pm_select_sleep_state(&spdifrx->pdev->dev);
return ret;
}
static bool stm32_spdifrx_volatile_reg(struct device *dev, unsigned int reg)
{
- if (reg == STM32_SPDIFRX_DR)
+ switch (reg) {
+ case STM32_SPDIFRX_DR:
+ case STM32_SPDIFRX_CSR:
+ case STM32_SPDIFRX_SR:
+ case STM32_SPDIFRX_DIR:
return true;
-
- return false;
+ default:
+ return false;
+ }
}
static bool stm32_spdifrx_writeable_reg(struct device *dev, unsigned int reg)
.volatile_reg = stm32_spdifrx_volatile_reg,
.writeable_reg = stm32_spdifrx_writeable_reg,
.fast_io = true,
+ .cache_type = REGCACHE_FLAT,
};
static irqreturn_t stm32_spdifrx_isr(int irq, void *devid)
static const struct snd_pcm_hardware stm32_spdifrx_pcm_hw = {
.info = SNDRV_PCM_INFO_INTERLEAVED | SNDRV_PCM_INFO_MMAP,
.buffer_bytes_max = 8 * PAGE_SIZE,
- .period_bytes_max = 2048, /* MDMA constraint */
+ .period_bytes_min = 1024,
+ .period_bytes_max = 4 * PAGE_SIZE,
.periods_min = 2,
.periods_max = 8,
};
MODULE_DEVICE_TABLE(of, stm32_spdifrx_ids);
+#ifdef CONFIG_PM_SLEEP
+static int stm32_spdifrx_suspend(struct device *dev)
+{
+ struct stm32_spdifrx_data *spdifrx = dev_get_drvdata(dev);
+
+ regcache_cache_only(spdifrx->regmap, true);
+ regcache_mark_dirty(spdifrx->regmap);
+
+ return 0;
+}
+
+static int stm32_spdifrx_resume(struct device *dev)
+{
+ struct stm32_spdifrx_data *spdifrx = dev_get_drvdata(dev);
+
+ regcache_cache_only(spdifrx->regmap, false);
+
+ return regcache_sync(spdifrx->regmap);
+}
+#endif /* CONFIG_PM_SLEEP */
+
+static const struct dev_pm_ops stm32_spdifrx_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(stm32_spdifrx_suspend, stm32_spdifrx_resume)
+};
+
static struct platform_driver stm32_spdifrx_driver = {
.driver = {
.name = "st,stm32-spdifrx",
.of_match_table = stm32_spdifrx_ids,
+ .pm = &stm32_spdifrx_pm_ops,
},
.probe = stm32_spdifrx_probe,
.remove = stm32_spdifrx_remove,
config SND_SOC_DAVINCI_MCASP
tristate "Multichannel Audio Serial Port (McASP) support"
- select SND_SOC_TI_EDMA_PCM if TI_EDMA
- select SND_SOC_TI_SDMA_PCM if DMA_OMAP
+ select SND_SOC_TI_EDMA_PCM
+ select SND_SOC_TI_SDMA_PCM
help
Say Y or M here if you want to have support for McASP IP found in
various Texas Instruments SoCs like:
return 0;
}
-static const SOC_ENUM_SINGLE_EXT_DECL(ams_delta_audio_enum,
+static SOC_ENUM_SINGLE_EXT_DECL(ams_delta_audio_enum,
ams_delta_audio_mode);
static const struct snd_kcontrol_new ams_delta_audio_controls[] = {
#define MCASP_MAX_AFIFO_DEPTH 64
+#ifdef CONFIG_PM
static u32 context_regs[] = {
DAVINCI_MCASP_TXFMCTL_REG,
DAVINCI_MCASP_RXFMCTL_REG,
u32 *xrsr_regs; /* for serializer configuration */
bool pm_state;
};
+#endif
struct davinci_mcasp_ruledata {
struct davinci_mcasp *mcasp;
#include <sound/pcm_params.h>
#include <sound/soc.h>
#include <sound/dmaengine_pcm.h>
-#include <linux/edma.h>
#include "edma-pcm.h"
static const struct snd_dmaengine_pcm_config edma_dmaengine_pcm_config = {
.pcm_hardware = &edma_pcm_hardware,
.prepare_slave_config = snd_dmaengine_pcm_prepare_slave_config,
- .compat_filter_fn = edma_filter_fn,
.prealloc_buffer_size = 128 * 1024,
};
int edma_pcm_platform_register(struct device *dev)
{
- return devm_snd_dmaengine_pcm_register(dev, &edma_dmaengine_pcm_config,
- SND_DMAENGINE_PCM_FLAG_COMPAT);
+ return devm_snd_dmaengine_pcm_register(dev, &edma_dmaengine_pcm_config, 0);
}
EXPORT_SYMBOL_GPL(edma_pcm_platform_register);
#include <sound/pcm_params.h>
#include <sound/soc.h>
#include <sound/dmaengine_pcm.h>
-#include <linux/omap-dmaengine.h>
#include "sdma-pcm.h"
static const struct snd_dmaengine_pcm_config sdma_dmaengine_pcm_config = {
.pcm_hardware = &sdma_pcm_hardware,
.prepare_slave_config = snd_dmaengine_pcm_prepare_slave_config,
- .compat_filter_fn = omap_dma_filter_fn,
.prealloc_buffer_size = 128 * 1024,
};
char *txdmachan, char *rxdmachan)
{
struct snd_dmaengine_pcm_config *config;
- unsigned int flags = SND_DMAENGINE_PCM_FLAG_COMPAT;
+ unsigned int flags = 0;
/* Standard names for the directions: 'tx' and 'rx' */
if (!txdmachan && !rxdmachan)
return devm_snd_dmaengine_pcm_register(dev,
- &sdma_dmaengine_pcm_config,
- flags);
+ &sdma_dmaengine_pcm_config, 0);
config = devm_kzalloc(dev, sizeof(*config), GFP_KERNEL);
if (!config)
config->chan_names[0] = txdmachan;
config->chan_names[1] = rxdmachan;
- return devm_snd_dmaengine_pcm_register(dev, config, flags);
+ return devm_snd_dmaengine_pcm_register(dev, config, 0);
}
EXPORT_SYMBOL_GPL(sdma_pcm_platform_register);
if (copy_from_user(&patch, arg, sizeof(patch)))
return -EFAULT;
+ if (patch.key == GUS_PATCH)
+ return snd_soundfont_load_guspatch(emu->sflist, arg,
+ patch.len + sizeof(patch),
+ TMP_CLIENT_ID);
+
if (patch.type >= SNDRV_SFNT_LOAD_INFO &&
patch.type <= SNDRV_SFNT_PROBE_DATA) {
err = snd_soundfont_load(emu->sflist, arg, patch.len + sizeof(patch), TMP_CLIENT_ID);
int r, p, t;
r = (3 - ((rate >> 6) & 3)) * 3;
p = rate & 0x3f;
+ if (!p)
+ p = 1;
t = end - start;
if (t < 0) t = -t;
if (13 > r)
/* Firmware version (x 100) */
u8 firmware_version;
- /* Timer for delayed PCM startup */
- struct timer_list timer;
+ /* Work for delayed PCM startup */
+ struct delayed_work pcm_work;
/* Device type */
enum line6_device_type type;
return 1;
}
-static void toneport_start_pcm(struct timer_list *t)
+static void toneport_start_pcm(struct work_struct *work)
{
- struct usb_line6_toneport *toneport = from_timer(toneport, t, timer);
+ struct usb_line6_toneport *toneport =
+ container_of(work, struct usb_line6_toneport, pcm_work.work);
struct usb_line6 *line6 = &toneport->line6;
line6_pcm_acquire(line6->line6pcm, LINE6_STREAM_MONITOR, true);
if (toneport_has_led(toneport))
toneport_update_led(toneport);
- mod_timer(&toneport->timer, jiffies + TONEPORT_PCM_DELAY * HZ);
+ schedule_delayed_work(&toneport->pcm_work,
+ msecs_to_jiffies(TONEPORT_PCM_DELAY * 1000));
return 0;
}
struct usb_line6_toneport *toneport =
(struct usb_line6_toneport *)line6;
- del_timer_sync(&toneport->timer);
+ cancel_delayed_work_sync(&toneport->pcm_work);
if (toneport_has_led(toneport))
toneport_remove_leds(toneport);
struct usb_line6_toneport *toneport = (struct usb_line6_toneport *) line6;
toneport->type = id->driver_info;
- timer_setup(&toneport->timer, toneport_start_pcm, 0);
+ INIT_DELAYED_WORK(&toneport->pcm_work, toneport_start_pcm);
line6->disconnect = line6_toneport_disconnect;
kctl = snd_ctl_new1(&mixer_selectunit_ctl, cval);
if (! kctl) {
usb_audio_err(state->chip, "cannot malloc kcontrol\n");
+ for (i = 0; i < desc->bNrInPins; i++)
+ kfree(namelist[i]);
kfree(namelist);
kfree(cval);
return -ENOMEM;
if (err < 0)
goto _error;
- snd_usb_mixer_apply_create_quirk(mixer);
+ err = snd_usb_mixer_apply_create_quirk(mixer);
+ if (err < 0)
+ goto _error;
err = snd_device_new(chip->card, SNDRV_DEV_CODEC, mixer, &dev_ops);
if (err < 0)
.type = QUIRK_MIDI_NOVATION
}
},
+{
+ /*
+ * Focusrite Scarlett Solo 2nd generation
+ * Reports that playback should use Synch: Synchronous
+ * while still providing a feedback endpoint. Synchronous causes
+ * snapping on some sample rates.
+ * Force it to use Synch: Asynchronous.
+ */
+ USB_DEVICE(0x1235, 0x8205),
+ .driver_info = (unsigned long) & (const struct snd_usb_audio_quirk) {
+ .ifnum = QUIRK_ANY_INTERFACE,
+ .type = QUIRK_COMPOSITE,
+ .data = (const struct snd_usb_audio_quirk[]) {
+ {
+ .ifnum = 1,
+ .type = QUIRK_AUDIO_FIXED_ENDPOINT,
+ .data = & (const struct audioformat) {
+ .formats = SNDRV_PCM_FMTBIT_S32_LE,
+ .channels = 2,
+ .iface = 1,
+ .altsetting = 1,
+ .altset_idx = 1,
+ .attributes = 0,
+ .endpoint = 0x01,
+ .ep_attr = USB_ENDPOINT_XFER_ISOC |
+ USB_ENDPOINT_SYNC_ASYNC,
+ .protocol = UAC_VERSION_2,
+ .rates = SNDRV_PCM_RATE_44100 |
+ SNDRV_PCM_RATE_48000 |
+ SNDRV_PCM_RATE_88200 |
+ SNDRV_PCM_RATE_96000 |
+ SNDRV_PCM_RATE_176400 |
+ SNDRV_PCM_RATE_192000,
+ .rate_min = 44100,
+ .rate_max = 192000,
+ .nr_rates = 6,
+ .rate_table = (unsigned int[]) {
+ 44100, 48000, 88200,
+ 96000, 176400, 192000
+ },
+ .clock = 41
+ }
+ },
+ {
+ .ifnum = 2,
+ .type = QUIRK_AUDIO_FIXED_ENDPOINT,
+ .data = & (const struct audioformat) {
+ .formats = SNDRV_PCM_FMTBIT_S32_LE,
+ .channels = 2,
+ .iface = 2,
+ .altsetting = 1,
+ .altset_idx = 1,
+ .attributes = 0,
+ .endpoint = 0x82,
+ .ep_attr = USB_ENDPOINT_XFER_ISOC |
+ USB_ENDPOINT_SYNC_ASYNC |
+ USB_ENDPOINT_USAGE_IMPLICIT_FB,
+ .protocol = UAC_VERSION_2,
+ .rates = SNDRV_PCM_RATE_44100 |
+ SNDRV_PCM_RATE_48000 |
+ SNDRV_PCM_RATE_88200 |
+ SNDRV_PCM_RATE_96000 |
+ SNDRV_PCM_RATE_176400 |
+ SNDRV_PCM_RATE_192000,
+ .rate_min = 44100,
+ .rate_max = 192000,
+ .nr_rates = 6,
+ .rate_table = (unsigned int[]) {
+ 44100, 48000, 88200,
+ 96000, 176400, 192000
+ },
+ .clock = 41
+ }
+ },
+ {
+ .ifnum = 3,
+ .type = QUIRK_IGNORE_INTERFACE
+ },
+ {
+ .ifnum = -1
+ }
+ }
+ }
+},
/* Access Music devices */
{
if (!us428->us428ctls_sharedmem) {
init_waitqueue_head(&us428->us428ctls_wait_queue_head);
- if(!(us428->us428ctls_sharedmem = snd_malloc_pages(sizeof(struct us428ctls_sharedmem), GFP_KERNEL)))
+ us428->us428ctls_sharedmem = alloc_pages_exact(sizeof(struct us428ctls_sharedmem), GFP_KERNEL);
+ if (!us428->us428ctls_sharedmem)
return -ENOMEM;
memset(us428->us428ctls_sharedmem, -1, sizeof(struct us428ctls_sharedmem));
us428->us428ctls_sharedmem->CtlSnapShotLast = -2;
if (!s)
return;
- free_pages((unsigned long)sk->write_page, get_order(s->write_size));
+ free_pages_exact(sk->write_page, s->write_size);
sk->write_page = NULL;
- free_pages((unsigned long)s, get_order(s->read_size));
+ free_pages_exact(s, s->read_size);
sk->s = NULL;
}
int read_size = sizeof(struct usb_stream);
int write_size;
int usb_frames = dev->speed == USB_SPEED_HIGH ? 8000 : 1000;
- int pg;
in_pipe = usb_rcvisocpipe(dev, in_endpoint);
out_pipe = usb_sndisocpipe(dev, out_endpoint);
goto out;
}
- pg = get_order(read_size);
- sk->s = (void *) __get_free_pages(GFP_KERNEL|__GFP_COMP|__GFP_ZERO|
- __GFP_NOWARN, pg);
+ sk->s = alloc_pages_exact(read_size,
+ GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN);
if (!sk->s) {
- snd_printk(KERN_WARNING "couldn't __get_free_pages()\n");
+ pr_warn("us122l: couldn't allocate read buffer\n");
goto out;
}
sk->s->cfg.version = USB_STREAM_INTERFACE_VERSION;
sk->s->period_size = frame_size * period_frames;
sk->s->write_size = write_size;
- pg = get_order(write_size);
- sk->write_page =
- (void *)__get_free_pages(GFP_KERNEL|__GFP_COMP|__GFP_ZERO|
- __GFP_NOWARN, pg);
+ sk->write_page = alloc_pages_exact(write_size,
+ GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN);
if (!sk->write_page) {
- snd_printk(KERN_WARNING "couldn't __get_free_pages()\n");
+ pr_warn("us122l: couldn't allocate write buffer\n");
usb_stream_free(sk);
return NULL;
}
if (! (usX2Y->In04urb = usb_alloc_urb(0, GFP_KERNEL)))
return -ENOMEM;
- if (! (usX2Y->In04Buf = kmalloc(21, GFP_KERNEL))) {
- usb_free_urb(usX2Y->In04urb);
+ if (! (usX2Y->In04Buf = kmalloc(21, GFP_KERNEL)))
return -ENOMEM;
- }
init_waitqueue_head(&usX2Y->In04WaitQueue);
usb_fill_int_urb(usX2Y->In04urb, usX2Y->dev, usb_rcvintpipe(usX2Y->dev, 0x4),
kfree(usX2Y(card)->In04Buf);
usb_free_urb(usX2Y(card)->In04urb);
if (usX2Y(card)->us428ctls_sharedmem)
- snd_free_pages(usX2Y(card)->us428ctls_sharedmem, sizeof(*usX2Y(card)->us428ctls_sharedmem));
+ free_pages_exact(usX2Y(card)->us428ctls_sharedmem,
+ sizeof(*usX2Y(card)->us428ctls_sharedmem));
if (usX2Y(card)->card_index >= 0 && usX2Y(card)->card_index < SNDRV_CARDS)
snd_usX2Y_card_used[usX2Y(card)->card_index] = 0;
}
snd_printdd("snd_usX2Y_pcm_prepare(%p)\n", substream);
if (NULL == usX2Y->hwdep_pcm_shm) {
- if (NULL == (usX2Y->hwdep_pcm_shm = snd_malloc_pages(sizeof(struct snd_usX2Y_hwdep_pcm_shm), GFP_KERNEL)))
+ usX2Y->hwdep_pcm_shm = alloc_pages_exact(sizeof(struct snd_usX2Y_hwdep_pcm_shm),
+ GFP_KERNEL);
+ if (!usX2Y->hwdep_pcm_shm)
return -ENOMEM;
memset(usX2Y->hwdep_pcm_shm, 0, sizeof(struct snd_usX2Y_hwdep_pcm_shm));
}
{
struct usX2Ydev *usX2Y = hwdep->private_data;
if (NULL != usX2Y->hwdep_pcm_shm)
- snd_free_pages(usX2Y->hwdep_pcm_shm, sizeof(struct snd_usX2Y_hwdep_pcm_shm));
+ free_pages_exact(usX2Y->hwdep_pcm_shm, sizeof(struct snd_usX2Y_hwdep_pcm_shm));
}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+// Copyright (C) 2019 Hangzhou C-SKY Microsystems co.,ltd.
+
+#ifndef _ASM_CSKY_PERF_REGS_H
+#define _ASM_CSKY_PERF_REGS_H
+
+/* Index of struct pt_regs */
+enum perf_event_csky_regs {
+ PERF_REG_CSKY_TLS,
+ PERF_REG_CSKY_LR,
+ PERF_REG_CSKY_PC,
+ PERF_REG_CSKY_SR,
+ PERF_REG_CSKY_SP,
+ PERF_REG_CSKY_ORIG_A0,
+ PERF_REG_CSKY_A0,
+ PERF_REG_CSKY_A1,
+ PERF_REG_CSKY_A2,
+ PERF_REG_CSKY_A3,
+ PERF_REG_CSKY_REGS0,
+ PERF_REG_CSKY_REGS1,
+ PERF_REG_CSKY_REGS2,
+ PERF_REG_CSKY_REGS3,
+ PERF_REG_CSKY_REGS4,
+ PERF_REG_CSKY_REGS5,
+ PERF_REG_CSKY_REGS6,
+ PERF_REG_CSKY_REGS7,
+ PERF_REG_CSKY_REGS8,
+ PERF_REG_CSKY_REGS9,
+#if defined(__CSKYABIV2__)
+ PERF_REG_CSKY_EXREGS0,
+ PERF_REG_CSKY_EXREGS1,
+ PERF_REG_CSKY_EXREGS2,
+ PERF_REG_CSKY_EXREGS3,
+ PERF_REG_CSKY_EXREGS4,
+ PERF_REG_CSKY_EXREGS5,
+ PERF_REG_CSKY_EXREGS6,
+ PERF_REG_CSKY_EXREGS7,
+ PERF_REG_CSKY_EXREGS8,
+ PERF_REG_CSKY_EXREGS9,
+ PERF_REG_CSKY_EXREGS10,
+ PERF_REG_CSKY_EXREGS11,
+ PERF_REG_CSKY_EXREGS12,
+ PERF_REG_CSKY_EXREGS13,
+ PERF_REG_CSKY_EXREGS14,
+ PERF_REG_CSKY_HI,
+ PERF_REG_CSKY_LO,
+ PERF_REG_CSKY_DCSR,
+#endif
+ PERF_REG_CSKY_MAX,
+};
+#endif /* _ASM_CSKY_PERF_REGS_H */
LIBUNWIND_LIBS = -lunwind -lunwind-aarch64
endif
+ifeq ($(SRCARCH),csky)
+ NO_PERF_REGS := 0
+endif
+
ifeq ($(ARCH),s390)
NO_PERF_REGS := 0
NO_SYSCALL_TABLE := 0
# Disable it on all other architectures in case libdw unwind
# support is detected in system. Add supported architectures
# to the check.
-ifneq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc s390))
+ifneq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc s390 csky))
NO_LIBDW_DWARF_UNWIND := 1
endif
--- /dev/null
+perf-y += util/
--- /dev/null
+ifndef NO_DWARF
+PERF_HAVE_DWARF_REGS := 1
+endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+// Copyright (C) 2019 Hangzhou C-SKY Microsystems co.,ltd.
+
+#ifndef ARCH_PERF_REGS_H
+#define ARCH_PERF_REGS_H
+
+#include <stdlib.h>
+#include <linux/types.h>
+#include <asm/perf_regs.h>
+
+#define PERF_REGS_MASK ((1ULL << PERF_REG_CSKY_MAX) - 1)
+#define PERF_REGS_MAX PERF_REG_CSKY_MAX
+#define PERF_SAMPLE_REGS_ABI PERF_SAMPLE_REGS_ABI_32
+
+#define PERF_REG_IP PERF_REG_CSKY_PC
+#define PERF_REG_SP PERF_REG_CSKY_SP
+
+static inline const char *perf_reg_name(int id)
+{
+ switch (id) {
+ case PERF_REG_CSKY_A0:
+ return "a0";
+ case PERF_REG_CSKY_A1:
+ return "a1";
+ case PERF_REG_CSKY_A2:
+ return "a2";
+ case PERF_REG_CSKY_A3:
+ return "a3";
+ case PERF_REG_CSKY_REGS0:
+ return "regs0";
+ case PERF_REG_CSKY_REGS1:
+ return "regs1";
+ case PERF_REG_CSKY_REGS2:
+ return "regs2";
+ case PERF_REG_CSKY_REGS3:
+ return "regs3";
+ case PERF_REG_CSKY_REGS4:
+ return "regs4";
+ case PERF_REG_CSKY_REGS5:
+ return "regs5";
+ case PERF_REG_CSKY_REGS6:
+ return "regs6";
+ case PERF_REG_CSKY_REGS7:
+ return "regs7";
+ case PERF_REG_CSKY_REGS8:
+ return "regs8";
+ case PERF_REG_CSKY_REGS9:
+ return "regs9";
+ case PERF_REG_CSKY_SP:
+ return "sp";
+ case PERF_REG_CSKY_LR:
+ return "lr";
+ case PERF_REG_CSKY_PC:
+ return "pc";
+#if defined(__CSKYABIV2__)
+ case PERF_REG_CSKY_EXREGS0:
+ return "exregs0";
+ case PERF_REG_CSKY_EXREGS1:
+ return "exregs1";
+ case PERF_REG_CSKY_EXREGS2:
+ return "exregs2";
+ case PERF_REG_CSKY_EXREGS3:
+ return "exregs3";
+ case PERF_REG_CSKY_EXREGS4:
+ return "exregs4";
+ case PERF_REG_CSKY_EXREGS5:
+ return "exregs5";
+ case PERF_REG_CSKY_EXREGS6:
+ return "exregs6";
+ case PERF_REG_CSKY_EXREGS7:
+ return "exregs7";
+ case PERF_REG_CSKY_EXREGS8:
+ return "exregs8";
+ case PERF_REG_CSKY_EXREGS9:
+ return "exregs9";
+ case PERF_REG_CSKY_EXREGS10:
+ return "exregs10";
+ case PERF_REG_CSKY_EXREGS11:
+ return "exregs11";
+ case PERF_REG_CSKY_EXREGS12:
+ return "exregs12";
+ case PERF_REG_CSKY_EXREGS13:
+ return "exregs13";
+ case PERF_REG_CSKY_EXREGS14:
+ return "exregs14";
+ case PERF_REG_CSKY_TLS:
+ return "tls";
+ case PERF_REG_CSKY_HI:
+ return "hi";
+ case PERF_REG_CSKY_LO:
+ return "lo";
+#endif
+ default:
+ return NULL;
+ }
+
+ return NULL;
+}
+
+#endif /* ARCH_PERF_REGS_H */
--- /dev/null
+perf-$(CONFIG_DWARF) += dwarf-regs.o
+perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2019 Hangzhou C-SKY Microsystems co.,ltd.
+// Mapping of DWARF debug register numbers into register names.
+
+#include <stddef.h>
+#include <dwarf-regs.h>
+
+#if defined(__CSKYABIV2__)
+#define CSKY_MAX_REGS 73
+const char *csky_dwarf_regs_table[CSKY_MAX_REGS] = {
+ /* r0 ~ r8 */
+ "%a0", "%a1", "%a2", "%a3", "%regs0", "%regs1", "%regs2", "%regs3",
+ /* r9 ~ r15 */
+ "%regs4", "%regs5", "%regs6", "%regs7", "%regs8", "%regs9", "%sp",
+ "%lr",
+ /* r16 ~ r23 */
+ "%exregs0", "%exregs1", "%exregs2", "%exregs3", "%exregs4",
+ "%exregs5", "%exregs6", "%exregs7",
+ /* r24 ~ r31 */
+ "%exregs8", "%exregs9", "%exregs10", "%exregs11", "%exregs12",
+ "%exregs13", "%exregs14", "%tls",
+ "%pc", NULL, NULL, NULL, "%hi", "%lo", NULL, NULL,
+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
+ "%epc",
+};
+#else
+#define CSKY_MAX_REGS 57
+const char *csky_dwarf_regs_table[CSKY_MAX_REGS] = {
+ /* r0 ~ r8 */
+ "%sp", "%regs9", "%a0", "%a1", "%a2", "%a3", "%regs0", "%regs1",
+ /* r9 ~ r15 */
+ "%regs2", "%regs3", "%regs4", "%regs5", "%regs6", "%regs7", "%regs8",
+ "%lr",
+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
+ "%epc",
+};
+#endif
+
+const char *get_arch_regstr(unsigned int n)
+{
+ return (n < CSKY_MAX_REGS) ? csky_dwarf_regs_table[n] : NULL;
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2019 Hangzhou C-SKY Microsystems co.,ltd.
+
+#include <elfutils/libdwfl.h>
+#include "../../util/unwind-libdw.h"
+#include "../../util/perf_regs.h"
+#include "../../util/event.h"
+
+bool libdw__arch_set_initial_registers(Dwfl_Thread *thread, void *arg)
+{
+ struct unwind_info *ui = arg;
+ struct regs_dump *user_regs = &ui->sample->user_regs;
+ Dwarf_Word dwarf_regs[PERF_REG_CSKY_MAX];
+
+#define REG(r) ({ \
+ Dwarf_Word val = 0; \
+ perf_reg_value(&val, user_regs, PERF_REG_CSKY_##r); \
+ val; \
+})
+
+#if defined(__CSKYABIV2__)
+ dwarf_regs[0] = REG(A0);
+ dwarf_regs[1] = REG(A1);
+ dwarf_regs[2] = REG(A2);
+ dwarf_regs[3] = REG(A3);
+ dwarf_regs[4] = REG(REGS0);
+ dwarf_regs[5] = REG(REGS1);
+ dwarf_regs[6] = REG(REGS2);
+ dwarf_regs[7] = REG(REGS3);
+ dwarf_regs[8] = REG(REGS4);
+ dwarf_regs[9] = REG(REGS5);
+ dwarf_regs[10] = REG(REGS6);
+ dwarf_regs[11] = REG(REGS7);
+ dwarf_regs[12] = REG(REGS8);
+ dwarf_regs[13] = REG(REGS9);
+ dwarf_regs[14] = REG(SP);
+ dwarf_regs[15] = REG(LR);
+ dwarf_regs[16] = REG(EXREGS0);
+ dwarf_regs[17] = REG(EXREGS1);
+ dwarf_regs[18] = REG(EXREGS2);
+ dwarf_regs[19] = REG(EXREGS3);
+ dwarf_regs[20] = REG(EXREGS4);
+ dwarf_regs[21] = REG(EXREGS5);
+ dwarf_regs[22] = REG(EXREGS6);
+ dwarf_regs[23] = REG(EXREGS7);
+ dwarf_regs[24] = REG(EXREGS8);
+ dwarf_regs[25] = REG(EXREGS9);
+ dwarf_regs[26] = REG(EXREGS10);
+ dwarf_regs[27] = REG(EXREGS11);
+ dwarf_regs[28] = REG(EXREGS12);
+ dwarf_regs[29] = REG(EXREGS13);
+ dwarf_regs[30] = REG(EXREGS14);
+ dwarf_regs[31] = REG(TLS);
+ dwarf_regs[32] = REG(PC);
+#else
+ dwarf_regs[0] = REG(SP);
+ dwarf_regs[1] = REG(REGS9);
+ dwarf_regs[2] = REG(A0);
+ dwarf_regs[3] = REG(A1);
+ dwarf_regs[4] = REG(A2);
+ dwarf_regs[5] = REG(A3);
+ dwarf_regs[6] = REG(REGS0);
+ dwarf_regs[7] = REG(REGS1);
+ dwarf_regs[8] = REG(REGS2);
+ dwarf_regs[9] = REG(REGS3);
+ dwarf_regs[10] = REG(REGS4);
+ dwarf_regs[11] = REG(REGS5);
+ dwarf_regs[12] = REG(REGS6);
+ dwarf_regs[13] = REG(REGS7);
+ dwarf_regs[14] = REG(REGS8);
+ dwarf_regs[15] = REG(LR);
+#endif
+ dwfl_thread_state_register_pc(thread, REG(PC));
+
+ return dwfl_thread_state_registers(thread, 0, PERF_REG_CSKY_MAX,
+ dwarf_regs);
+}
test_memcontrol
test_core
+test_freezer
TEST_GEN_PROGS = test_memcontrol
TEST_GEN_PROGS += test_core
+TEST_GEN_PROGS += test_freezer
include ../lib.mk
$(OUTPUT)/test_memcontrol: cgroup_util.c
$(OUTPUT)/test_core: cgroup_util.c
+$(OUTPUT)/test_freezer: cgroup_util.c
return ret;
}
+char *cg_control(const char *cgroup, const char *control)
+{
+ size_t len = strlen(cgroup) + strlen(control) + 2;
+ char *ret = malloc(len);
+
+ snprintf(ret, len, "%s/%s", cgroup, control);
+
+ return ret;
+}
+
int cg_read(const char *cgroup, const char *control, char *buf, size_t len)
{
char path[PATH_MAX];
return mkdir(cgroup, 0644);
}
-static int cg_killall(const char *cgroup)
+int cg_wait_for_proc_count(const char *cgroup, int count)
+{
+ char buf[10 * PAGE_SIZE] = {0};
+ int attempts;
+ char *ptr;
+
+ for (attempts = 10; attempts >= 0; attempts--) {
+ int nr = 0;
+
+ if (cg_read(cgroup, "cgroup.procs", buf, sizeof(buf)))
+ break;
+
+ for (ptr = buf; *ptr; ptr++)
+ if (*ptr == '\n')
+ nr++;
+
+ if (nr >= count)
+ return 0;
+
+ usleep(100000);
+ }
+
+ return -1;
+}
+
+int cg_killall(const char *cgroup)
{
char buf[PAGE_SIZE];
char *ptr = buf;
retry:
ret = rmdir(cgroup);
if (ret && errno == EBUSY) {
- ret = cg_killall(cgroup);
- if (ret)
- return ret;
+ cg_killall(cgroup);
usleep(100);
goto retry;
}
return ret;
}
+int cg_enter(const char *cgroup, int pid)
+{
+ char pidbuf[64];
+
+ snprintf(pidbuf, sizeof(pidbuf), "%d", pid);
+ return cg_write(cgroup, "cgroup.procs", pidbuf);
+}
+
int cg_enter_current(const char *cgroup)
{
char pidbuf[64];
close(fd);
return 0;
}
+
+char proc_read_text(int pid, const char *item, char *buf, size_t size)
+{
+ char path[PATH_MAX];
+
+ snprintf(path, sizeof(path), "/proc/%d/%s", pid, item);
+
+ return read_text(path, buf, size);
+}
extern int cg_find_unified_root(char *root, size_t len);
extern char *cg_name(const char *root, const char *name);
extern char *cg_name_indexed(const char *root, const char *name, int index);
+extern char *cg_control(const char *cgroup, const char *control);
extern int cg_create(const char *cgroup);
extern int cg_destroy(const char *cgroup);
extern int cg_read(const char *cgroup, const char *control,
extern int cg_run(const char *cgroup,
int (*fn)(const char *cgroup, void *arg),
void *arg);
+extern int cg_enter(const char *cgroup, int pid);
extern int cg_enter_current(const char *cgroup);
extern int cg_run_nowait(const char *cgroup,
int (*fn)(const char *cgroup, void *arg),
extern int alloc_anon(const char *cgroup, void *arg);
extern int is_swap_enabled(void);
extern int set_oom_adj_score(int pid, int score);
+extern int cg_wait_for_proc_count(const char *cgroup, int count);
+extern int cg_killall(const char *cgroup);
+extern char proc_read_text(int pid, const char *item, char *buf, size_t size);
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <stdbool.h>
+#include <linux/limits.h>
+#include <sys/ptrace.h>
+#include <sys/types.h>
+#include <sys/mman.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <errno.h>
+#include <poll.h>
+#include <stdlib.h>
+#include <sys/inotify.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+
+#include "../kselftest.h"
+#include "cgroup_util.h"
+
+#define DEBUG
+#ifdef DEBUG
+#define debug(args...) fprintf(stderr, args)
+#else
+#define debug(args...)
+#endif
+
+/*
+ * Check if the cgroup is frozen by looking at the cgroup.events::frozen value.
+ */
+static int cg_check_frozen(const char *cgroup, bool frozen)
+{
+ if (frozen) {
+ if (cg_read_strstr(cgroup, "cgroup.events", "frozen 1") != 0) {
+ debug("Cgroup %s isn't frozen\n", cgroup);
+ return -1;
+ }
+ } else {
+ /*
+ * Check the cgroup.events::frozen value.
+ */
+ if (cg_read_strstr(cgroup, "cgroup.events", "frozen 0") != 0) {
+ debug("Cgroup %s is frozen\n", cgroup);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+/*
+ * Freeze the given cgroup.
+ */
+static int cg_freeze_nowait(const char *cgroup, bool freeze)
+{
+ return cg_write(cgroup, "cgroup.freeze", freeze ? "1" : "0");
+}
+
+/*
+ * Prepare for waiting on cgroup.events file.
+ */
+static int cg_prepare_for_wait(const char *cgroup)
+{
+ int fd, ret = -1;
+
+ fd = inotify_init1(0);
+ if (fd == -1) {
+ debug("Error: inotify_init1() failed\n");
+ return fd;
+ }
+
+ ret = inotify_add_watch(fd, cg_control(cgroup, "cgroup.events"),
+ IN_MODIFY);
+ if (ret == -1) {
+ debug("Error: inotify_add_watch() failed\n");
+ close(fd);
+ }
+
+ return fd;
+}
+
+/*
+ * Wait for an event. If there are no events for 10 seconds,
+ * treat this an error.
+ */
+static int cg_wait_for(int fd)
+{
+ int ret = -1;
+ struct pollfd fds = {
+ .fd = fd,
+ .events = POLLIN,
+ };
+
+ while (true) {
+ ret = poll(&fds, 1, 10000);
+
+ if (ret == -1) {
+ if (errno == EINTR)
+ continue;
+ debug("Error: poll() failed\n");
+ break;
+ }
+
+ if (ret > 0 && fds.revents & POLLIN) {
+ ret = 0;
+ break;
+ }
+ }
+
+ return ret;
+}
+
+/*
+ * Attach a task to the given cgroup and wait for a cgroup frozen event.
+ * All transient events (e.g. populated) are ignored.
+ */
+static int cg_enter_and_wait_for_frozen(const char *cgroup, int pid,
+ bool frozen)
+{
+ int fd, ret = -1;
+ int attempts;
+
+ fd = cg_prepare_for_wait(cgroup);
+ if (fd < 0)
+ return fd;
+
+ ret = cg_enter(cgroup, pid);
+ if (ret)
+ goto out;
+
+ for (attempts = 0; attempts < 10; attempts++) {
+ ret = cg_wait_for(fd);
+ if (ret)
+ break;
+
+ ret = cg_check_frozen(cgroup, frozen);
+ if (ret)
+ continue;
+ }
+
+out:
+ close(fd);
+ return ret;
+}
+
+/*
+ * Freeze the given cgroup and wait for the inotify signal.
+ * If there are no events in 10 seconds, treat this as an error.
+ * Then check that the cgroup is in the desired state.
+ */
+static int cg_freeze_wait(const char *cgroup, bool freeze)
+{
+ int fd, ret = -1;
+
+ fd = cg_prepare_for_wait(cgroup);
+ if (fd < 0)
+ return fd;
+
+ ret = cg_freeze_nowait(cgroup, freeze);
+ if (ret) {
+ debug("Error: cg_freeze_nowait() failed\n");
+ goto out;
+ }
+
+ ret = cg_wait_for(fd);
+ if (ret)
+ goto out;
+
+ ret = cg_check_frozen(cgroup, freeze);
+out:
+ close(fd);
+ return ret;
+}
+
+/*
+ * A simple process running in a sleep loop until being
+ * re-parented.
+ */
+static int child_fn(const char *cgroup, void *arg)
+{
+ int ppid = getppid();
+
+ while (getppid() == ppid)
+ usleep(1000);
+
+ return getppid() == ppid;
+}
+
+/*
+ * A simple test for the cgroup freezer: populated the cgroup with 100
+ * running processes and freeze it. Then unfreeze it. Then it kills all
+ * processes and destroys the cgroup.
+ */
+static int test_cgfreezer_simple(const char *root)
+{
+ int ret = KSFT_FAIL;
+ char *cgroup = NULL;
+ int i;
+
+ cgroup = cg_name(root, "cg_test_simple");
+ if (!cgroup)
+ goto cleanup;
+
+ if (cg_create(cgroup))
+ goto cleanup;
+
+ for (i = 0; i < 100; i++)
+ cg_run_nowait(cgroup, child_fn, NULL);
+
+ if (cg_wait_for_proc_count(cgroup, 100))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup, false))
+ goto cleanup;
+
+ if (cg_freeze_wait(cgroup, true))
+ goto cleanup;
+
+ if (cg_freeze_wait(cgroup, false))
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (cgroup)
+ cg_destroy(cgroup);
+ free(cgroup);
+ return ret;
+}
+
+/*
+ * The test creates the following hierarchy:
+ * A
+ * / / \ \
+ * B E I K
+ * /\ |
+ * C D F
+ * |
+ * G
+ * |
+ * H
+ *
+ * with a process in C, H and 3 processes in K.
+ * Then it tries to freeze and unfreeze the whole tree.
+ */
+static int test_cgfreezer_tree(const char *root)
+{
+ char *cgroup[10] = {0};
+ int ret = KSFT_FAIL;
+ int i;
+
+ cgroup[0] = cg_name(root, "cg_test_tree_A");
+ if (!cgroup[0])
+ goto cleanup;
+
+ cgroup[1] = cg_name(cgroup[0], "B");
+ if (!cgroup[1])
+ goto cleanup;
+
+ cgroup[2] = cg_name(cgroup[1], "C");
+ if (!cgroup[2])
+ goto cleanup;
+
+ cgroup[3] = cg_name(cgroup[1], "D");
+ if (!cgroup[3])
+ goto cleanup;
+
+ cgroup[4] = cg_name(cgroup[0], "E");
+ if (!cgroup[4])
+ goto cleanup;
+
+ cgroup[5] = cg_name(cgroup[4], "F");
+ if (!cgroup[5])
+ goto cleanup;
+
+ cgroup[6] = cg_name(cgroup[5], "G");
+ if (!cgroup[6])
+ goto cleanup;
+
+ cgroup[7] = cg_name(cgroup[6], "H");
+ if (!cgroup[7])
+ goto cleanup;
+
+ cgroup[8] = cg_name(cgroup[0], "I");
+ if (!cgroup[8])
+ goto cleanup;
+
+ cgroup[9] = cg_name(cgroup[0], "K");
+ if (!cgroup[9])
+ goto cleanup;
+
+ for (i = 0; i < 10; i++)
+ if (cg_create(cgroup[i]))
+ goto cleanup;
+
+ cg_run_nowait(cgroup[2], child_fn, NULL);
+ cg_run_nowait(cgroup[7], child_fn, NULL);
+ cg_run_nowait(cgroup[9], child_fn, NULL);
+ cg_run_nowait(cgroup[9], child_fn, NULL);
+ cg_run_nowait(cgroup[9], child_fn, NULL);
+
+ /*
+ * Wait until all child processes will enter
+ * corresponding cgroups.
+ */
+
+ if (cg_wait_for_proc_count(cgroup[2], 1) ||
+ cg_wait_for_proc_count(cgroup[7], 1) ||
+ cg_wait_for_proc_count(cgroup[9], 3))
+ goto cleanup;
+
+ /*
+ * Freeze B.
+ */
+ if (cg_freeze_wait(cgroup[1], true))
+ goto cleanup;
+
+ /*
+ * Freeze F.
+ */
+ if (cg_freeze_wait(cgroup[5], true))
+ goto cleanup;
+
+ /*
+ * Freeze G.
+ */
+ if (cg_freeze_wait(cgroup[6], true))
+ goto cleanup;
+
+ /*
+ * Check that A and E are not frozen.
+ */
+ if (cg_check_frozen(cgroup[0], false))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup[4], false))
+ goto cleanup;
+
+ /*
+ * Freeze A. Check that A, B and E are frozen.
+ */
+ if (cg_freeze_wait(cgroup[0], true))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup[1], true))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup[4], true))
+ goto cleanup;
+
+ /*
+ * Unfreeze B, F and G
+ */
+ if (cg_freeze_nowait(cgroup[1], false))
+ goto cleanup;
+
+ if (cg_freeze_nowait(cgroup[5], false))
+ goto cleanup;
+
+ if (cg_freeze_nowait(cgroup[6], false))
+ goto cleanup;
+
+ /*
+ * Check that C and H are still frozen.
+ */
+ if (cg_check_frozen(cgroup[2], true))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup[7], true))
+ goto cleanup;
+
+ /*
+ * Unfreeze A. Check that A, C and K are not frozen.
+ */
+ if (cg_freeze_wait(cgroup[0], false))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup[2], false))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup[9], false))
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ for (i = 9; i >= 0 && cgroup[i]; i--) {
+ cg_destroy(cgroup[i]);
+ free(cgroup[i]);
+ }
+
+ return ret;
+}
+
+/*
+ * A fork bomb emulator.
+ */
+static int forkbomb_fn(const char *cgroup, void *arg)
+{
+ int ppid;
+
+ fork();
+ fork();
+
+ ppid = getppid();
+
+ while (getppid() == ppid)
+ usleep(1000);
+
+ return getppid() == ppid;
+}
+
+/*
+ * The test runs a fork bomb in a cgroup and tries to freeze it.
+ * Then it kills all processes and checks that cgroup isn't populated
+ * anymore.
+ */
+static int test_cgfreezer_forkbomb(const char *root)
+{
+ int ret = KSFT_FAIL;
+ char *cgroup = NULL;
+
+ cgroup = cg_name(root, "cg_forkbomb_test");
+ if (!cgroup)
+ goto cleanup;
+
+ if (cg_create(cgroup))
+ goto cleanup;
+
+ cg_run_nowait(cgroup, forkbomb_fn, NULL);
+
+ usleep(100000);
+
+ if (cg_freeze_wait(cgroup, true))
+ goto cleanup;
+
+ if (cg_killall(cgroup))
+ goto cleanup;
+
+ if (cg_wait_for_proc_count(cgroup, 0))
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (cgroup)
+ cg_destroy(cgroup);
+ free(cgroup);
+ return ret;
+}
+
+/*
+ * The test creates two nested cgroups, freezes the parent
+ * and removes the child. Then it checks that the parent cgroup
+ * remains frozen and it's possible to create a new child
+ * without unfreezing. The new child is frozen too.
+ */
+static int test_cgfreezer_rmdir(const char *root)
+{
+ int ret = KSFT_FAIL;
+ char *parent, *child = NULL;
+
+ parent = cg_name(root, "cg_test_rmdir_A");
+ if (!parent)
+ goto cleanup;
+
+ child = cg_name(parent, "cg_test_rmdir_B");
+ if (!child)
+ goto cleanup;
+
+ if (cg_create(parent))
+ goto cleanup;
+
+ if (cg_create(child))
+ goto cleanup;
+
+ if (cg_freeze_wait(parent, true))
+ goto cleanup;
+
+ if (cg_destroy(child))
+ goto cleanup;
+
+ if (cg_check_frozen(parent, true))
+ goto cleanup;
+
+ if (cg_create(child))
+ goto cleanup;
+
+ if (cg_check_frozen(child, true))
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (child)
+ cg_destroy(child);
+ free(child);
+ if (parent)
+ cg_destroy(parent);
+ free(parent);
+ return ret;
+}
+
+/*
+ * The test creates two cgroups: A and B, runs a process in A
+ * and performs several migrations:
+ * 1) A (running) -> B (frozen)
+ * 2) B (frozen) -> A (running)
+ * 3) A (frozen) -> B (frozen)
+ *
+ * On each step it checks the actual state of both cgroups.
+ */
+static int test_cgfreezer_migrate(const char *root)
+{
+ int ret = KSFT_FAIL;
+ char *cgroup[2] = {0};
+ int pid;
+
+ cgroup[0] = cg_name(root, "cg_test_migrate_A");
+ if (!cgroup[0])
+ goto cleanup;
+
+ cgroup[1] = cg_name(root, "cg_test_migrate_B");
+ if (!cgroup[1])
+ goto cleanup;
+
+ if (cg_create(cgroup[0]))
+ goto cleanup;
+
+ if (cg_create(cgroup[1]))
+ goto cleanup;
+
+ pid = cg_run_nowait(cgroup[0], child_fn, NULL);
+ if (pid < 0)
+ goto cleanup;
+
+ if (cg_wait_for_proc_count(cgroup[0], 1))
+ goto cleanup;
+
+ /*
+ * Migrate from A (running) to B (frozen)
+ */
+ if (cg_freeze_wait(cgroup[1], true))
+ goto cleanup;
+
+ if (cg_enter_and_wait_for_frozen(cgroup[1], pid, true))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup[0], false))
+ goto cleanup;
+
+ /*
+ * Migrate from B (frozen) to A (running)
+ */
+ if (cg_enter_and_wait_for_frozen(cgroup[0], pid, false))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup[1], true))
+ goto cleanup;
+
+ /*
+ * Migrate from A (frozen) to B (frozen)
+ */
+ if (cg_freeze_wait(cgroup[0], true))
+ goto cleanup;
+
+ if (cg_enter_and_wait_for_frozen(cgroup[1], pid, true))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup[0], true))
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (cgroup[0])
+ cg_destroy(cgroup[0]);
+ free(cgroup[0]);
+ if (cgroup[1])
+ cg_destroy(cgroup[1]);
+ free(cgroup[1]);
+ return ret;
+}
+
+/*
+ * The test checks that ptrace works with a tracing process in a frozen cgroup.
+ */
+static int test_cgfreezer_ptrace(const char *root)
+{
+ int ret = KSFT_FAIL;
+ char *cgroup = NULL;
+ siginfo_t siginfo;
+ int pid;
+
+ cgroup = cg_name(root, "cg_test_ptrace");
+ if (!cgroup)
+ goto cleanup;
+
+ if (cg_create(cgroup))
+ goto cleanup;
+
+ pid = cg_run_nowait(cgroup, child_fn, NULL);
+ if (pid < 0)
+ goto cleanup;
+
+ if (cg_wait_for_proc_count(cgroup, 1))
+ goto cleanup;
+
+ if (cg_freeze_wait(cgroup, true))
+ goto cleanup;
+
+ if (ptrace(PTRACE_SEIZE, pid, NULL, NULL))
+ goto cleanup;
+
+ if (ptrace(PTRACE_INTERRUPT, pid, NULL, NULL))
+ goto cleanup;
+
+ waitpid(pid, NULL, 0);
+
+ /*
+ * Cgroup has to remain frozen, however the test task
+ * is in traced state.
+ */
+ if (cg_check_frozen(cgroup, true))
+ goto cleanup;
+
+ if (ptrace(PTRACE_GETSIGINFO, pid, NULL, &siginfo))
+ goto cleanup;
+
+ if (ptrace(PTRACE_DETACH, pid, NULL, NULL))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup, true))
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (cgroup)
+ cg_destroy(cgroup);
+ free(cgroup);
+ return ret;
+}
+
+/*
+ * Check if the process is stopped.
+ */
+static int proc_check_stopped(int pid)
+{
+ char buf[PAGE_SIZE];
+ int len;
+
+ len = proc_read_text(pid, "stat", buf, sizeof(buf));
+ if (len == -1) {
+ debug("Can't get %d stat\n", pid);
+ return -1;
+ }
+
+ if (strstr(buf, "(test_freezer) T ") == NULL) {
+ debug("Process %d in the unexpected state: %s\n", pid, buf);
+ return -1;
+ }
+
+ return 0;
+}
+
+/*
+ * Test that it's possible to freeze a cgroup with a stopped process.
+ */
+static int test_cgfreezer_stopped(const char *root)
+{
+ int pid, ret = KSFT_FAIL;
+ char *cgroup = NULL;
+
+ cgroup = cg_name(root, "cg_test_stopped");
+ if (!cgroup)
+ goto cleanup;
+
+ if (cg_create(cgroup))
+ goto cleanup;
+
+ pid = cg_run_nowait(cgroup, child_fn, NULL);
+
+ if (cg_wait_for_proc_count(cgroup, 1))
+ goto cleanup;
+
+ if (kill(pid, SIGSTOP))
+ goto cleanup;
+
+ if (cg_check_frozen(cgroup, false))
+ goto cleanup;
+
+ if (cg_freeze_wait(cgroup, true))
+ goto cleanup;
+
+ if (cg_freeze_wait(cgroup, false))
+ goto cleanup;
+
+ if (proc_check_stopped(pid))
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (cgroup)
+ cg_destroy(cgroup);
+ free(cgroup);
+ return ret;
+}
+
+/*
+ * Test that it's possible to freeze a cgroup with a ptraced process.
+ */
+static int test_cgfreezer_ptraced(const char *root)
+{
+ int pid, ret = KSFT_FAIL;
+ char *cgroup = NULL;
+ siginfo_t siginfo;
+
+ cgroup = cg_name(root, "cg_test_ptraced");
+ if (!cgroup)
+ goto cleanup;
+
+ if (cg_create(cgroup))
+ goto cleanup;
+
+ pid = cg_run_nowait(cgroup, child_fn, NULL);
+
+ if (cg_wait_for_proc_count(cgroup, 1))
+ goto cleanup;
+
+ if (ptrace(PTRACE_SEIZE, pid, NULL, NULL))
+ goto cleanup;
+
+ if (ptrace(PTRACE_INTERRUPT, pid, NULL, NULL))
+ goto cleanup;
+
+ waitpid(pid, NULL, 0);
+
+ if (cg_check_frozen(cgroup, false))
+ goto cleanup;
+
+ if (cg_freeze_wait(cgroup, true))
+ goto cleanup;
+
+ /*
+ * cg_check_frozen(cgroup, true) will fail here,
+ * because the task in in the TRACEd state.
+ */
+ if (cg_freeze_wait(cgroup, false))
+ goto cleanup;
+
+ if (ptrace(PTRACE_GETSIGINFO, pid, NULL, &siginfo))
+ goto cleanup;
+
+ if (ptrace(PTRACE_DETACH, pid, NULL, NULL))
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (cgroup)
+ cg_destroy(cgroup);
+ free(cgroup);
+ return ret;
+}
+
+static int vfork_fn(const char *cgroup, void *arg)
+{
+ int pid = vfork();
+
+ if (pid == 0)
+ while (true)
+ sleep(1);
+
+ return pid;
+}
+
+/*
+ * Test that it's possible to freeze a cgroup with a process,
+ * which called vfork() and is waiting for a child.
+ */
+static int test_cgfreezer_vfork(const char *root)
+{
+ int ret = KSFT_FAIL;
+ char *cgroup = NULL;
+
+ cgroup = cg_name(root, "cg_test_vfork");
+ if (!cgroup)
+ goto cleanup;
+
+ if (cg_create(cgroup))
+ goto cleanup;
+
+ cg_run_nowait(cgroup, vfork_fn, NULL);
+
+ if (cg_wait_for_proc_count(cgroup, 2))
+ goto cleanup;
+
+ if (cg_freeze_wait(cgroup, true))
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (cgroup)
+ cg_destroy(cgroup);
+ free(cgroup);
+ return ret;
+}
+
+#define T(x) { x, #x }
+struct cgfreezer_test {
+ int (*fn)(const char *root);
+ const char *name;
+} tests[] = {
+ T(test_cgfreezer_simple),
+ T(test_cgfreezer_tree),
+ T(test_cgfreezer_forkbomb),
+ T(test_cgfreezer_rmdir),
+ T(test_cgfreezer_migrate),
+ T(test_cgfreezer_ptrace),
+ T(test_cgfreezer_stopped),
+ T(test_cgfreezer_ptraced),
+ T(test_cgfreezer_vfork),
+};
+#undef T
+
+int main(int argc, char *argv[])
+{
+ char root[PATH_MAX];
+ int i, ret = EXIT_SUCCESS;
+
+ if (cg_find_unified_root(root, sizeof(root)))
+ ksft_exit_skip("cgroup v2 isn't mounted\n");
+ for (i = 0; i < ARRAY_SIZE(tests); i++) {
+ switch (tests[i].fn(root)) {
+ case KSFT_PASS:
+ ksft_test_result_pass("%s\n", tests[i].name);
+ break;
+ case KSFT_SKIP:
+ ksft_test_result_skip("%s\n", tests[i].name);
+ break;
+ default:
+ ret = EXIT_FAILURE;
+ ksft_test_result_fail("%s\n", tests[i].name);
+ break;
+ }
+ }
+
+ return ret;
+}
projects / linux-2.6-block.git / commitdiff